/home/wangjiarui/AIGV_2025/shell/train_qa.sh: 20: torchrun: not found
/home/wangjiarui/AIGV_2025/shell/train_qa.sh: 20: torchrun: not found
/home/wangjiarui/AIGV_2025/shell/train_qa.sh: 20: torchrun: not found
/home/wangjiarui/AIGV_2025/shell/train_qa.sh: line 21: torchrun: command not found
/home/wangjiarui/.conda/envs/intern25/bin/python: can't open file '/home/wangjiarui/train/train_qa.py': [Errno 2] No such file or directory
E0425 07:41:42.362114 2660716 .conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 2) local_rank: 0 (pid: 2660790) of binary: /home/wangjiarui/.conda/envs/intern25/bin/python
Traceback (most recent call last):
  File "/home/wangjiarui/.conda/envs/intern25/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train/train_qa.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-25_07:41:42
  host      : amax
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 2660790)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[2025-04-25 07:42:22,677] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:42:26,002] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:42:26,002] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
04/25/2025 07:42:26 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:42:26 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-42-26_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:42:26 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:42:26,151 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:42:26,607 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:42:26 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:42:26,747 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:42:26,749 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:42:26 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:42:26,749 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:42:26,750 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:42:26,752 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:42:26,825 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:42:26,825 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Setting backbone: fragments_backbone
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1219, in <module>
[rank0]:     main()
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1064, in main
[rank0]:     model = InternVLChatModel.from_pretrained(
[rank0]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/modeling_utils.py", line 4111, in from_pretrained
[rank0]:     model = cls(config, *model_args, **model_kwargs)
[rank0]:   File "/home/wangjiarui/AIGV_2025/model/internvl_chat_st1_fast/modeling_internvl_chat.py", line 284, in __init__
[rank0]:     self.evaluator.load_state_dict(torch.load('/media/amax/e1efc3d3-8977-4b90-9121-3f956ab56974/huiyu/wjr/method/FAST-VQA-and-FasterVQA-dev/FAST_VQA_B_1_4.pth')["state_dict"], strict=False)
[rank0]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/serialization.py", line 1319, in load
[rank0]:     with _open_file_like(f, "rb") as opened_file:
[rank0]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/serialization.py", line 659, in _open_file_like
[rank0]:     return _open_file(name_or_buffer, mode)
[rank0]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/serialization.py", line 640, in __init__
[rank0]:     super().__init__(open(name, mode))
[rank0]: FileNotFoundError: [Errno 2] No such file or directory: '/media/amax/e1efc3d3-8977-4b90-9121-3f956ab56974/huiyu/wjr/method/FAST-VQA-and-FasterVQA-dev/FAST_VQA_B_1_4.pth'
[rank0]:[W425 07:42:29.164691144 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
E0425 07:42:29.724624 2660996 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 2661075) of binary: /home/wangjiarui/.conda/envs/intern25/bin/python
Traceback (most recent call last):
  File "/home/wangjiarui/.conda/envs/intern25/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train/train_qa.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-25_07:42:29
  host      : amax
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 2661075)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[2025-04-25 07:43:05,225] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:43:09,010] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:43:09,010] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
04/25/2025 07:43:09 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:43:09 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-43-09_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:43:09 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,154 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,154 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,154 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,154 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,154 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:09,155 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:43:09,613 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:43:09 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:43:09,759 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:43:09,761 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:43:09 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:43:09,762 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:43:09,762 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:09,764 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:43:09,837 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:09,837 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.18it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.12it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.82it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.18it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.74it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:43:12,021 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:43:12,022 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:43:12,031 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:12,032 >> Generate config GenerationConfig {}

04/25/2025 07:43:12 - INFO - __main__ - Finished
04/25/2025 07:43:12 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:43:12 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:43:12 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:43:12 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:43:12 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:43:12 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:43:12 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:43:12 - INFO - __main__ - Formatting inputs...Skip in lazy mode
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1219, in <module>
[rank0]:     main()
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1136, in main
[rank0]:     train_dataset = build_datasets(
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 939, in build_datasets
[rank0]:     dataset = LazySupervisedDataset(
[rank0]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 454, in __init__
[rank0]:     with open(meta['annotation_train'], 'r') as f:
[rank0]: FileNotFoundError: [Errno 2] No such file or directory: 'data3/video_qa_train.jsonl'
[rank0]:[W425 07:43:13.472553291 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
E0425 07:43:14.157253 2661523 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 2661602) of binary: /home/wangjiarui/.conda/envs/intern25/bin/python
Traceback (most recent call last):
  File "/home/wangjiarui/.conda/envs/intern25/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train/train_qa.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-25_07:43:14
  host      : amax
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 2661602)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[2025-04-25 07:43:35,983] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:43:39,235] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:43:39,235] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
04/25/2025 07:43:39 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:43:39 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-43-39_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:43:39 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,381 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,381 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,382 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,382 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,382 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:43:39,382 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:43:39,813 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:43:39 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:43:39,947 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:43:39,949 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:43:39 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:43:39,950 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:43:39,950 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:39,952 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:43:40,021 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:40,021 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

motion_mlp.weight1 Parameter containing:
tensor([[ 3.5237e-19,  4.3141e-31,  0.0000e+00,  ...,  1.7323e-07,
         -7.8496e-17,  1.3132e-07],
        [ 0.0000e+00,  6.0000e+00, -1.3173e-31,  ...,  3.8184e-07,
         -1.6693e+09,  2.8871e-07],
        [ 0.0000e+00,  1.2000e+01, -1.3173e-31,  ...,  5.8860e-07,
         -1.1731e+07,  4.4703e-07],
        ...,
        [-1.4412e+17,  2.4448e+04,  5.5460e+29,  ...,  8.5449e-04,
         -4.8884e+20,  6.4468e-04],
        [-2.4179e+24,  2.4448e+04,  1.6161e+35,  ...,  8.5449e-04,
         -1.1799e+27,  6.4468e-04],
        [-4.0565e+31,  2.4448e+04, -4.2906e-37,  ...,  8.5449e-04,
         -2.9815e+33,  6.4468e-04]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.44it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.05it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.04it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:43:42,225 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:43:42,226 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:43:42,235 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:43:42,236 >> Generate config GenerationConfig {}

04/25/2025 07:43:42 - INFO - __main__ - Finished
04/25/2025 07:43:42 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:43:42 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:43:42 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:43:42 - INFO - __main__ - Formatting inputs...Skip in lazy mode
04/25/2025 07:43:42 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:43:42 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:43:42 - INFO - __main__ - Formatting inputs...Skip in lazy mode
04/25/2025 07:43:43 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 9000
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7fd77ec64d30>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/25/2025 07:43:44 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-43-39_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-25 07:43:44,409 >> Using auto half precision backend
[WARNING|trainer.py:803] 2025-04-25 07:43:44,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:43:44,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:43:44,626] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-25 07:43:44,627] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 1
[2025-04-25 07:43:56,887] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.6631324291229248 seconds
[2025-04-25 07:43:57,554] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-25 07:43:57,555] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-25 07:43:57,647] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-25 07:43:57,647] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-25 07:43:57,647] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-25 07:43:57,647] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-25 07:43:57,647] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-25 07:43:57,647] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-25 07:43:57,647] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-25 07:43:58,084] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-25 07:43:58,085] [INFO] [utils.py:782:see_memory_usage] MA 18.18 GB         Max_MA 18.28 GB         CA 18.52 GB         Max_CA 19 GB 
[2025-04-25 07:43:58,085] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 33.54 GB, percent = 13.3%
[2025-04-25 07:43:58,253] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-25 07:43:58,254] [INFO] [utils.py:782:see_memory_usage] MA 18.18 GB         Max_MA 18.37 GB         CA 18.72 GB         Max_CA 19 GB 
[2025-04-25 07:43:58,254] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 33.55 GB, percent = 13.3%
[2025-04-25 07:43:58,254] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-25 07:43:58,426] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-25 07:43:58,426] [INFO] [utils.py:782:see_memory_usage] MA 18.18 GB         Max_MA 18.18 GB         CA 18.72 GB         Max_CA 19 GB 
[2025-04-25 07:43:58,427] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 33.56 GB, percent = 13.3%
[2025-04-25 07:43:58,438] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-25 07:43:58,439] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-25 07:43:58,439] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7fd77489c8b0>
[2025-04-25 07:43:58,439] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-25 07:43:58,449] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fd77489c370>
[2025-04-25 07:43:58,450] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-25 07:43:58,451] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-25 07:43:58,452] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   train_batch_size ............. 4
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   world_size ................... 1
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-25 07:43:58,453] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-25 07:43:58,454] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 4, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-25 07:43:58,455 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-25 07:43:58,455 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-25 07:43:58,455 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-25 07:43:58,455 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-25 07:43:58,455 >>   Total train batch size (w. parallel, distributed & accumulation) = 4
[INFO|trainer.py:2376] 2025-04-25 07:43:58,455 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-25 07:43:58,455 >>   Total optimization steps = 123,750
[INFO|trainer.py:2378] 2025-04-25 07:43:58,462 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/123750 [00:00<?, ?it/s][2025-04-25 07:44:00,777] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:44:06,500] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:44:12,039] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:44:17,463] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
[2025-04-25 07:44:38,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 50.47 | optimizer_step: 11.03
[2025-04-25 07:44:38,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 5362.11 | bwd_microstep: 5684.87 | bwd_inner_microstep: 5681.84 | bwd_allreduce_microstep: 2.98 | step_microstep: 204.15
[2025-04-25 07:44:38,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 5362.08 | bwd: 5684.89 | bwd_inner: 5681.84 | bwd_allreduce: 3.00 | step: 204.15

  0%|          | 1/123750 [00:39<1361:24:17, 39.60s/it]
                                                       
{'loss': 0.9509, 'grad_norm': 19.343244552612305, 'learning_rate': 1.0772959870724482e-08, 'epoch': 0.0}

  0%|          | 1/123750 [00:39<1361:24:17, 39.60s/it][2025-04-25 07:44:46,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 2.65
[2025-04-25 07:44:46,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.96 | bwd_microstep: 5600.43 | bwd_inner_microstep: 5596.97 | bwd_allreduce_microstep: 3.39 | step_microstep: 51.79
[2025-04-25 07:44:46,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.96 | bwd: 5600.46 | bwd_inner: 5596.97 | bwd_allreduce: 3.42 | step: 51.80

  0%|          | 2/123750 [00:48<735:10:52, 21.39s/it] 
                                                      
{'loss': 1.3608, 'grad_norm': 21.4112606048584, 'learning_rate': 2.1545919741448964e-08, 'epoch': 0.0}

  0%|          | 2/123750 [00:48<735:10:52, 21.39s/it][2025-04-25 07:44:55,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 2.65
[2025-04-25 07:44:55,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.01 | bwd_microstep: 5643.04 | bwd_inner_microstep: 5639.52 | bwd_allreduce_microstep: 3.45 | step_microstep: 51.88
[2025-04-25 07:44:55,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.01 | bwd: 5643.06 | bwd_inner: 5639.52 | bwd_allreduce: 3.48 | step: 51.89

  0%|          | 3/123750 [00:56<534:51:43, 15.56s/it]
                                                      
{'loss': 1.6155, 'grad_norm': 16.761287689208984, 'learning_rate': 3.2318879612173446e-08, 'epoch': 0.0}

  0%|          | 3/123750 [00:56<534:51:43, 15.56s/it][2025-04-25 07:45:03,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.28 | optimizer_step: 2.65
[2025-04-25 07:45:03,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.92 | bwd_microstep: 5651.28 | bwd_inner_microstep: 5648.40 | bwd_allreduce_microstep: 2.84 | step_microstep: 30.22
[2025-04-25 07:45:03,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.92 | bwd: 5651.30 | bwd_inner: 5648.40 | bwd_allreduce: 2.86 | step: 30.23

  0%|          | 4/123750 [01:05<440:46:48, 12.82s/it]
                                                      
{'loss': 0.7864, 'grad_norm': 12.798917770385742, 'learning_rate': 4.309183948289793e-08, 'epoch': 0.0}

  0%|          | 4/123750 [01:05<440:46:48, 12.82s/it][2025-04-25 07:45:12,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.30 | optimizer_step: 2.65
[2025-04-25 07:45:12,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3156.21 | bwd_microstep: 5666.70 | bwd_inner_microstep: 5663.72 | bwd_allreduce_microstep: 2.93 | step_microstep: 30.44
[2025-04-25 07:45:12,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3156.21 | bwd: 5666.71 | bwd_inner: 5663.72 | bwd_allreduce: 2.95 | step: 30.45

  0%|          | 5/123750 [01:14<392:15:01, 11.41s/it]
                                                      
{'loss': 1.274, 'grad_norm': 16.308181762695312, 'learning_rate': 5.386479935362241e-08, 'epoch': 0.0}

  0%|          | 5/123750 [01:14<392:15:01, 11.41s/it][2025-04-25 07:45:21,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.65
[2025-04-25 07:45:21,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5680.28 | bwd_inner_microstep: 5677.33 | bwd_allreduce_microstep: 2.90 | step_microstep: 30.89
[2025-04-25 07:45:21,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5680.30 | bwd_inner: 5677.33 | bwd_allreduce: 2.92 | step: 30.90

  0%|          | 6/123750 [01:22<359:21:57, 10.45s/it]
                                                      
{'loss': 0.6914, 'grad_norm': 14.699689865112305, 'learning_rate': 6.463775922434689e-08, 'epoch': 0.0}

  0%|          | 6/123750 [01:23<359:21:57, 10.45s/it][2025-04-25 07:45:29,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.65
[2025-04-25 07:45:29,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2803.43 | bwd_microstep: 5608.49 | bwd_inner_microstep: 5605.58 | bwd_allreduce_microstep: 2.86 | step_microstep: 29.69
[2025-04-25 07:45:29,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2803.43 | bwd: 5608.50 | bwd_inner: 5605.58 | bwd_allreduce: 2.87 | step: 29.69

  0%|          | 7/123750 [01:31<337:21:12,  9.81s/it]
                                                      
{'loss': 1.0178, 'grad_norm': 20.169700622558594, 'learning_rate': 7.541071909507137e-08, 'epoch': 0.0}

  0%|          | 7/123750 [01:31<337:21:12,  9.81s/it][2025-04-25 07:45:38,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.30 | optimizer_step: 2.65
[2025-04-25 07:45:38,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2805.80 | bwd_microstep: 5606.31 | bwd_inner_microstep: 5603.35 | bwd_allreduce_microstep: 2.91 | step_microstep: 29.51
[2025-04-25 07:45:38,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2805.80 | bwd: 5606.32 | bwd_inner: 5603.35 | bwd_allreduce: 2.93 | step: 29.51

  0%|          | 8/123750 [01:39<322:54:26,  9.39s/it]
                                                      
{'loss': 1.0638, 'grad_norm': 21.816726684570312, 'learning_rate': 8.618367896579586e-08, 'epoch': 0.0}

  0%|          | 8/123750 [01:39<322:54:26,  9.39s/it][2025-04-25 07:45:46,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.31 | optimizer_step: 2.64
[2025-04-25 07:45:46,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.25 | bwd_microstep: 5621.85 | bwd_inner_microstep: 5618.87 | bwd_allreduce_microstep: 2.93 | step_microstep: 30.85
[2025-04-25 07:45:46,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.25 | bwd: 5621.86 | bwd_inner: 5618.87 | bwd_allreduce: 2.95 | step: 30.85

  0%|          | 9/123750 [01:48<313:33:12,  9.12s/it]
                                                      
{'loss': 1.4726, 'grad_norm': 18.098291397094727, 'learning_rate': 9.695663883652034e-08, 'epoch': 0.0}

  0%|          | 9/123750 [01:48<313:33:12,  9.12s/it][2025-04-25 07:45:55,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.30 | optimizer_step: 2.64
[2025-04-25 07:45:55,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.66 | bwd_microstep: 5638.52 | bwd_inner_microstep: 5635.57 | bwd_allreduce_microstep: 2.90 | step_microstep: 30.36
[2025-04-25 07:45:55,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.66 | bwd: 5638.54 | bwd_inner: 5635.57 | bwd_allreduce: 2.92 | step: 30.36

  0%|          | 10/123750 [01:57<307:23:09,  8.94s/it]
                                                       
{'loss': 1.1884, 'grad_norm': 19.998252868652344, 'learning_rate': 1.0772959870724482e-07, 'epoch': 0.0}

  0%|          | 10/123750 [01:57<307:23:09,  8.94s/it][2025-04-25 07:46:04,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.31 | optimizer_step: 2.65
[2025-04-25 07:46:04,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.24 | bwd_microstep: 5667.80 | bwd_inner_microstep: 5664.72 | bwd_allreduce_microstep: 3.03 | step_microstep: 30.56
[2025-04-25 07:46:04,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.24 | bwd: 5667.82 | bwd_inner: 5664.72 | bwd_allreduce: 3.05 | step: 30.56

  0%|          | 11/123750 [02:05<303:41:23,  8.84s/it]
                                                       
{'loss': 0.9096, 'grad_norm': 15.783824920654297, 'learning_rate': 1.185025585779693e-07, 'epoch': 0.0}

  0%|          | 11/123750 [02:05<303:41:23,  8.84s/it][2025-04-25 07:46:12,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.32 | optimizer_step: 2.65
[2025-04-25 07:46:12,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.62 | bwd_microstep: 5667.75 | bwd_inner_microstep: 5664.84 | bwd_allreduce_microstep: 2.86 | step_microstep: 30.18
[2025-04-25 07:46:12,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.62 | bwd: 5667.77 | bwd_inner: 5664.84 | bwd_allreduce: 2.88 | step: 30.18

  0%|          | 12/123750 [02:14<301:05:43,  8.76s/it]
                                                       
{'loss': 0.9709, 'grad_norm': 19.62442970275879, 'learning_rate': 1.2927551844869378e-07, 'epoch': 0.0}

  0%|          | 12/123750 [02:14<301:05:43,  8.76s/it][2025-04-25 07:46:21,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.28 | optimizer_step: 2.65
[2025-04-25 07:46:21,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.83 | bwd_microstep: 5632.83 | bwd_inner_microstep: 5629.96 | bwd_allreduce_microstep: 2.83 | step_microstep: 29.21
[2025-04-25 07:46:21,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.83 | bwd: 5632.85 | bwd_inner: 5629.96 | bwd_allreduce: 2.85 | step: 29.22

  0%|          | 13/123750 [02:22<298:42:40,  8.69s/it]
                                                       
{'loss': 0.9428, 'grad_norm': 13.95112133026123, 'learning_rate': 1.4004847831941827e-07, 'epoch': 0.0}

  0%|          | 13/123750 [02:22<298:42:40,  8.69s/it][2025-04-25 07:46:29,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.64
[2025-04-25 07:46:29,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.02 | bwd_microstep: 5630.47 | bwd_inner_microstep: 5627.56 | bwd_allreduce_microstep: 2.87 | step_microstep: 29.42
[2025-04-25 07:46:29,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.02 | bwd: 5630.48 | bwd_inner: 5627.56 | bwd_allreduce: 2.88 | step: 29.43

  0%|          | 14/123750 [02:31<297:01:13,  8.64s/it]
                                                       
{'loss': 1.108, 'grad_norm': 20.227598190307617, 'learning_rate': 1.5082143819014275e-07, 'epoch': 0.0}

  0%|          | 14/123750 [02:31<297:01:13,  8.64s/it][2025-04-25 07:46:38,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.31 | optimizer_step: 2.64
[2025-04-25 07:46:38,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.25 | bwd_microstep: 5643.22 | bwd_inner_microstep: 5640.20 | bwd_allreduce_microstep: 2.98 | step_microstep: 30.23
[2025-04-25 07:46:38,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.25 | bwd: 5643.24 | bwd_inner: 5640.20 | bwd_allreduce: 3.00 | step: 30.23

  0%|          | 15/123750 [02:39<296:05:06,  8.61s/it]
                                                       
{'loss': 1.6169, 'grad_norm': 21.02290916442871, 'learning_rate': 1.6159439806086723e-07, 'epoch': 0.0}

  0%|          | 15/123750 [02:39<296:05:06,  8.61s/it][2025-04-25 07:46:47,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.64
[2025-04-25 07:46:47,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3194.46 | bwd_microstep: 5748.24 | bwd_inner_microstep: 5745.26 | bwd_allreduce_microstep: 2.93 | step_microstep: 29.18
[2025-04-25 07:46:47,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3194.46 | bwd: 5748.26 | bwd_inner: 5745.26 | bwd_allreduce: 2.95 | step: 29.18

  0%|          | 16/123750 [02:48<300:19:32,  8.74s/it]
                                                       
{'loss': 0.833, 'grad_norm': 11.836997985839844, 'learning_rate': 1.723673579315917e-07, 'epoch': 0.0}

  0%|          | 16/123750 [02:48<300:19:32,  8.74s/it][2025-04-25 07:46:55,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.65
[2025-04-25 07:46:55,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.35 | bwd_microstep: 5645.59 | bwd_inner_microstep: 5642.68 | bwd_allreduce_microstep: 2.87 | step_microstep: 30.12
[2025-04-25 07:46:55,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.35 | bwd: 5645.61 | bwd_inner: 5642.68 | bwd_allreduce: 2.89 | step: 30.13

  0%|          | 17/123750 [02:57<298:22:28,  8.68s/it]
                                                       
{'loss': 0.6944, 'grad_norm': 16.387479782104492, 'learning_rate': 1.8314031780231622e-07, 'epoch': 0.0}

  0%|          | 17/123750 [02:57<298:22:28,  8.68s/it][2025-04-25 07:47:04,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.65
[2025-04-25 07:47:04,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.15 | bwd_microstep: 5682.15 | bwd_inner_microstep: 5679.25 | bwd_allreduce_microstep: 2.85 | step_microstep: 28.96
[2025-04-25 07:47:04,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.15 | bwd: 5682.16 | bwd_inner: 5679.25 | bwd_allreduce: 2.87 | step: 28.96

  0%|          | 18/123750 [03:06<297:33:20,  8.66s/it]
                                                       
{'loss': 1.4572, 'grad_norm': 18.660022735595703, 'learning_rate': 1.9391327767304068e-07, 'epoch': 0.0}

  0%|          | 18/123750 [03:06<297:33:20,  8.66s/it][2025-04-25 07:47:13,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.65
[2025-04-25 07:47:13,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.54 | bwd_microstep: 5646.75 | bwd_inner_microstep: 5643.72 | bwd_allreduce_microstep: 2.98 | step_microstep: 29.00
[2025-04-25 07:47:13,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.55 | bwd: 5646.76 | bwd_inner: 5643.72 | bwd_allreduce: 3.00 | step: 29.00

  0%|          | 19/123750 [03:14<296:25:58,  8.62s/it]
                                                       
{'loss': 0.8718, 'grad_norm': 17.391035079956055, 'learning_rate': 2.0468623754376516e-07, 'epoch': 0.0}

  0%|          | 19/123750 [03:14<296:25:58,  8.62s/it][2025-04-25 07:47:21,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.29 | optimizer_step: 2.64
[2025-04-25 07:47:21,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5699.70 | bwd_inner_microstep: 5696.78 | bwd_allreduce_microstep: 2.88 | step_microstep: 29.64
[2025-04-25 07:47:21,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5699.71 | bwd_inner: 5696.78 | bwd_allreduce: 2.90 | step: 29.65

  0%|          | 20/123750 [03:23<296:31:10,  8.63s/it]
                                                       
{'loss': 1.8805, 'grad_norm': 21.6248779296875, 'learning_rate': 2.1545919741448964e-07, 'epoch': 0.0}

  0%|          | 20/123750 [03:23<296:31:10,  8.63s/it][2025-04-25 07:47:30,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.27 | optimizer_step: 2.65
[2025-04-25 07:47:30,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.43 | bwd_microstep: 5675.50 | bwd_inner_microstep: 5672.60 | bwd_allreduce_microstep: 2.86 | step_microstep: 28.84
[2025-04-25 07:47:30,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.43 | bwd: 5675.52 | bwd_inner: 5672.60 | bwd_allreduce: 2.87 | step: 28.84

  0%|          | 21/123750 [03:31<296:11:40,  8.62s/it]
                                                       
{'loss': 0.7563, 'grad_norm': 26.239229202270508, 'learning_rate': 2.2623215728521415e-07, 'epoch': 0.0}

  0%|          | 21/123750 [03:31<296:11:40,  8.62s/it][2025-04-25 07:47:38,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.30 | optimizer_step: 2.65
[2025-04-25 07:47:38,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.63 | bwd_microstep: 5702.06 | bwd_inner_microstep: 5699.16 | bwd_allreduce_microstep: 2.85 | step_microstep: 29.32
[2025-04-25 07:47:38,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.63 | bwd: 5702.07 | bwd_inner: 5699.16 | bwd_allreduce: 2.87 | step: 29.33

  0%|          | 22/123750 [03:40<296:17:44,  8.62s/it]
                                                       
{'loss': 1.1306, 'grad_norm': 17.30942726135254, 'learning_rate': 2.370051171559386e-07, 'epoch': 0.0}

  0%|          | 22/123750 [03:40<296:17:44,  8.62s/it]W0425 07:47:44.639111 2667406 site-packages/torch/distributed/run.py:793] 
W0425 07:47:44.639111 2667406 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:47:44.639111 2667406 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 07:47:44.639111 2667406 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-25 07:47:46,694] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:47:46,696] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:47:46,697] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:47:50,031] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:47:50,031] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2025-04-25 07:47:50,121] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:47:50,129] [INFO] [comm.py:652:init_distributed] cdb=None
04/25/2025 07:47:50 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:47:50 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-47-50_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:47:50 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:47:50,165 >> loading file chat_template.jinja
04/25/2025 07:47:50 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:47:50 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:47:50,615 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:47:50 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:47:50,758 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:47:50,760 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:47:50 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:47:50,761 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:47:50,761 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:47:50,763 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:47:50,833 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:47:50,833 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

this model
this model
[WARNING|logging.py:328] 2025-04-25 07:47:50,900 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[WARNING|logging.py:328] 2025-04-25 07:47:50,924 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
Parameter containing:
tensor([[-4.0625e-01,  4.9593e-08,  0.0000e+00,  ...,  1.7323e-07,
         -7.8496e-17,  1.3132e-07],
        [ 0.0000e+00,  6.0000e+00, -1.3173e-31,  ...,  3.8184e-07,
         -1.6693e+09,  2.8871e-07],
        [ 0.0000e+00,  1.2000e+01, -1.3173e-31,  ...,  5.8860e-07,
         -1.1731e+07,  4.4703e-07],
        ...,
        [-1.4412e+17,  2.4448e+04,  5.5460e+29,  ...,  8.5449e-04,
         -4.8884e+20,  6.4468e-04],
        [-2.4179e+24,  2.4448e+04,  1.6161e+35,  ...,  8.5449e-04,
         -1.1799e+27,  6.4468e-04],
        [-4.0565e+31,  2.4448e+04, -4.2906e-37,  ...,  8.5449e-04,
         -2.9815e+33,  6.4468e-04]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
None False
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.96it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.89it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.22it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.93it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.91it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.14it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.83it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.43it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:47:56,989 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.76it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.37it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:47:57,061 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.82it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.50it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:47:57,074 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:47:57,074 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:47:57,084 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:47:57,085 >> Generate config GenerationConfig {}

04/25/2025 07:47:57 - INFO - __main__ - Finished
04/25/2025 07:47:57 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:47:57 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:47:57 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:47:57 - INFO - __main__ - Formatting inputs...Skip in lazy mode
04/25/2025 07:47:57 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:47:57 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:47:57 - INFO - __main__ - Formatting inputs...Skip in lazy mode
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7ff173855c70>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f0982d14c40>
04/25/2025 07:47:57 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 9000
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f22f1c4cd30>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=2,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-47-50_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-47-50_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/25/2025 07:47:58 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-47-50_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-25 07:47:58,764 >> Using auto half precision backend
[WARNING|trainer.py:803] 2025-04-25 07:47:58,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:47:58,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:47:58,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:47:58,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:47:58,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:47:58,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:47:59,019] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-25 07:47:59,020] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 3
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
[2025-04-25 07:48:05,449] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.6127653121948242 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6027617454528809 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6032164096832275 seconds
[2025-04-25 07:48:06,506] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-25 07:48:06,506] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-25 07:48:06,595] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-25 07:48:06,595] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-25 07:48:06,595] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-25 07:48:06,595] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-25 07:48:06,595] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-25 07:48:06,595] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-25 07:48:06,595] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-25 07:48:06,998] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-25 07:48:06,999] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.08 GB         CA 18.33 GB         Max_CA 18 GB 
[2025-04-25 07:48:06,999] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.41 GB, percent = 10.9%
[2025-04-25 07:48:07,173] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-25 07:48:07,174] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.11 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:48:07,174] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.42 GB, percent = 10.9%
[2025-04-25 07:48:07,174] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-25 07:48:07,348] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-25 07:48:07,349] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.05 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:48:07,349] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.56 GB, percent = 11.0%
[2025-04-25 07:48:07,355] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-25 07:48:07,355] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-25 07:48:07,355] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f22eb84a8e0>
[2025-04-25 07:48:07,355] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-25 07:48:07,365] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f22eb84a310>
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-25 07:48:07,366] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-25 07:48:07,367] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-25 07:48:07,368] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   train_batch_size ............. 12
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   world_size ................... 3
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-25 07:48:07,369] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-25 07:48:07,370] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 12, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-25 07:48:07,371 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-25 07:48:07,371 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-25 07:48:07,371 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-25 07:48:07,371 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-25 07:48:07,371 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2376] 2025-04-25 07:48:07,371 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-25 07:48:07,371 >>   Total optimization steps = 41,250
[INFO|trainer.py:2378] 2025-04-25 07:48:07,378 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/41250 [00:00<?, ?it/s][2025-04-25 07:48:08,963] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:08,998] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:09,540] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:48:14,496] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:14,518] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:15,164] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:48:20,000] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:20,045] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:20,710] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:48:25,457] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:25,817] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:48:26,237] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
8 7 7
8 7 7
[2025-04-25 07:48:50,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 11.09 | optimizer_gradients: 26.78 | optimizer_step: 5.30
[2025-04-25 07:48:50,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 5990.71 | bwd_microstep: 6050.97 | bwd_inner_microstep: 6037.59 | bwd_allreduce_microstep: 13.32 | step_microstep: 103.15
[2025-04-25 07:48:50,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 5990.69 | bwd: 6050.99 | bwd_inner: 6037.59 | bwd_allreduce: 13.36 | step: 103.15

  0%|          | 1/41250 [00:42<490:50:46, 42.84s/it]
                                                     
{'loss': 1.3896, 'grad_norm': 15.71052360534668, 'learning_rate': 3.231017770597739e-08, 'epoch': 0.0}

  0%|          | 1/41250 [00:42<490:50:46, 42.84s/it][2025-04-25 07:48:59,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 9.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 07:48:59,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2806.22 | bwd_microstep: 5998.41 | bwd_inner_microstep: 5614.88 | bwd_allreduce_microstep: 383.48 | step_microstep: 36.74
[2025-04-25 07:48:59,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2806.22 | bwd: 5998.43 | bwd_inner: 5614.88 | bwd_allreduce: 383.50 | step: 36.75

  0%|          | 2/41250 [00:51<262:56:13, 22.95s/it]
                                                     
{'loss': 0.9697, 'grad_norm': 11.639215469360352, 'learning_rate': 6.462035541195477e-08, 'epoch': 0.0}

  0%|          | 2/41250 [00:51<262:56:13, 22.95s/it][2025-04-25 07:49:07,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 07:49:07,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.67 | bwd_microstep: 5681.16 | bwd_inner_microstep: 5629.45 | bwd_allreduce_microstep: 51.67 | step_microstep: 19.10
[2025-04-25 07:49:07,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.67 | bwd: 5681.18 | bwd_inner: 5629.45 | bwd_allreduce: 51.69 | step: 19.10

  0%|          | 3/41250 [01:00<188:01:09, 16.41s/it]
                                                     
{'loss': 1.193, 'grad_norm': 17.614702224731445, 'learning_rate': 9.693053311793216e-08, 'epoch': 0.0}

  0%|          | 3/41250 [01:00<188:01:09, 16.41s/it][2025-04-25 07:49:16,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.63 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 07:49:16,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.01 | bwd_microstep: 5669.82 | bwd_inner_microstep: 5650.79 | bwd_allreduce_microstep: 18.99 | step_microstep: 18.99
[2025-04-25 07:49:16,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.01 | bwd: 5669.83 | bwd_inner: 5650.79 | bwd_allreduce: 19.01 | step: 19.00

  0%|          | 4/41250 [01:09<152:43:22, 13.33s/it]evaluate!
evaluate!

                                                     
{'loss': 0.9854, 'grad_norm': 15.179015159606934, 'learning_rate': 1.2924071082390955e-07, 'epoch': 0.0}

  0%|          | 4/41250 [01:09<152:43:22, 13.33s/it]evaluate!
1
1
1
[WARNING|trainer.py:803] 2025-04-25 07:49:19,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:19,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:19,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2
2
2
[WARNING|trainer.py:803] 2025-04-25 07:49:21,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:22,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:22,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3
3
3
[WARNING|trainer.py:803] 2025-04-25 07:49:24,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-25 07:49:24,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-25 07:49:24,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4
4
4
[WARNING|trainer.py:803] 2025-04-25 07:49:27,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-25 07:49:27,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-25 07:49:27,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5
5
5
[WARNING|trainer.py:803] 2025-04-25 07:49:29,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:29,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:49:29,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6
6
6
[WARNING|trainer.py:803] 2025-04-25 07:49:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 No
[WARNING|trainer.py:803] 2025-04-25 07:49:32,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 No
[WARNING|trainer.py:803] 2025-04-25 07:49:32,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 No
7
7
7
[WARNING|trainer.py:803] 2025-04-25 07:49:34,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
.Yes
[WARNING|trainer.py:803] 2025-04-25 07:49:34,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
.Yes
[WARNING|trainer.py:803] 2025-04-25 07:49:35,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
.Yes
8
8
[WARNING|trainer.py:803] 2025-04-25 07:49:37,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-25 07:49:37,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
8
W0425 07:49:41.769487 2673708 site-packages/torch/distributed/run.py:793] 
W0425 07:49:41.769487 2673708 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:49:41.769487 2673708 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 07:49:41.769487 2673708 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-25 07:49:43,797] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:49:43,800] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:49:43,801] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:49:47,009] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:49:47,009] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
04/25/2025 07:49:47 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:49:47 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-49-47_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:49:47 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:49:47,255 >> loading file chat_template.jinja
[2025-04-25 07:49:47,302] [INFO] [comm.py:652:init_distributed] cdb=None
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
04/25/2025 07:49:47 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
[2025-04-25 07:49:47,461] [INFO] [comm.py:652:init_distributed] cdb=None
04/25/2025 07:49:47 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:49:47,734 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:49:47 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:49:47,875 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:49:47,877 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:49:47 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:49:47,877 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:49:47,878 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:49:47,880 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:49:47,928 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:49:47,929 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

this model
[WARNING|logging.py:328] 2025-04-25 07:49:48,146 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
this model
[WARNING|logging.py:328] 2025-04-25 07:49:48,253 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
None False
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 None False
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.90it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.06it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.93it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.60it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.17it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.87it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.97it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.53it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:49:53,901 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:49:53,901 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:49:53,911 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:49:53,912 >> Generate config GenerationConfig {}

04/25/2025 07:49:53 - INFO - __main__ - Finished
04/25/2025 07:49:53 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:49:53 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:49:53 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:49:53 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:49:53 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:49:53 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:49:53 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:49:53 - INFO - __main__ - Formatting inputs...Skip in lazy mode

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.99it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.28it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.31it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:49:54,231 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.41it/s]04/25/2025 07:49:54 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/25/2025 07:49:54 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:49:54 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:49:54 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:49:54 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:49:54 - INFO - __main__ - Formatting inputs...Skip in lazy mode

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.71it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.40it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:49:54,435 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
04/25/2025 07:49:54 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 3
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f0f15162d30>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7efde1394c70>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f9a3e5d4c40>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/25/2025 07:49:55 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-49-47_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-25 07:49:55,566 >> Using auto half precision backend
[WARNING|trainer.py:803] 2025-04-25 07:49:55,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:49:55,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:49:55,769] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-25 07:49:55,770] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 3
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=2,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-49-47_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-49-47_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[WARNING|trainer.py:803] 2025-04-25 07:49:56,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:49:56,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:49:56,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:49:56,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:50:02,030] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.640312910079956 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.7031729221343994 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.7033886909484863 seconds
[2025-04-25 07:50:03,095] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-25 07:50:03,096] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-25 07:50:03,184] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-25 07:50:03,184] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-25 07:50:03,184] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-25 07:50:03,184] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-25 07:50:03,184] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-25 07:50:03,184] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-25 07:50:03,185] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-25 07:50:03,608] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-25 07:50:03,609] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.08 GB         CA 18.33 GB         Max_CA 18 GB 
[2025-04-25 07:50:03,609] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 26.36 GB, percent = 10.5%
[2025-04-25 07:50:03,781] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-25 07:50:03,782] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.11 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:50:03,782] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 26.3 GB, percent = 10.4%
[2025-04-25 07:50:03,782] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-25 07:50:03,957] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-25 07:50:03,958] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.05 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:50:03,958] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 26.45 GB, percent = 10.5%
[2025-04-25 07:50:03,964] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-25 07:50:03,964] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-25 07:50:03,964] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f0f0f069910>
[2025-04-25 07:50:03,964] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-25 07:50:03,975] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-25 07:50:03,975] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-25 07:50:03,975] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-25 07:50:03,975] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-25 07:50:03,975] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-25 07:50:03,975] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f0f0f069340>
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-25 07:50:03,976] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-25 07:50:03,977] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   train_batch_size ............. 12
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-25 07:50:03,978] [INFO] [config.py:1003:print]   world_size ................... 3
[2025-04-25 07:50:03,979] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-25 07:50:03,979] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-25 07:50:03,979] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-25 07:50:03,979] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-25 07:50:03,979] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-25 07:50:03,979] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 12, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-25 07:50:03,980 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-25 07:50:03,980 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-25 07:50:03,980 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-25 07:50:03,980 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-25 07:50:03,980 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2376] 2025-04-25 07:50:03,980 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-25 07:50:03,980 >>   Total optimization steps = 41,250
[INFO|trainer.py:2378] 2025-04-25 07:50:03,987 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/41250 [00:00<?, ?it/s][2025-04-25 07:50:05,537] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:05,557] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:06,189] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:50:11,022] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:11,052] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:12,132] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:50:16,599] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:16,623] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:17,725] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:50:22,305] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:22,312] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:50:23,221] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
8 7 7
8 7 7
[2025-04-25 07:50:46,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.12 | optimizer_gradients: 43.95 | optimizer_step: 8.13
[2025-04-25 07:50:46,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 7725.58 | bwd_microstep: 5754.09 | bwd_inner_microstep: 5740.37 | bwd_allreduce_microstep: 13.64 | step_microstep: 168.75
[2025-04-25 07:50:46,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 7725.56 | bwd: 5754.11 | bwd_inner: 5740.37 | bwd_allreduce: 13.68 | step: 168.75

  0%|          | 1/41250 [00:42<489:39:29, 42.73s/it]
                                                     
{'loss': 1.3896, 'grad_norm': 15.71048355102539, 'learning_rate': 3.231017770597739e-08, 'epoch': 0.0}

  0%|          | 1/41250 [00:42<489:39:29, 42.73s/it][2025-04-25 07:50:55,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 1.20 | optimizer_step: 0.89
[2025-04-25 07:50:55,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2809.43 | bwd_microstep: 6006.33 | bwd_inner_microstep: 5628.33 | bwd_allreduce_microstep: 377.95 | step_microstep: 20.10
[2025-04-25 07:50:55,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2809.43 | bwd: 6006.34 | bwd_inner: 5628.33 | bwd_allreduce: 377.97 | step: 20.10

  0%|          | 2/41250 [00:51<262:28:13, 22.91s/it]
                                                     
{'loss': 0.9697, 'grad_norm': 11.639888763427734, 'learning_rate': 6.462035541195477e-08, 'epoch': 0.0}

  0%|          | 2/41250 [00:51<262:28:13, 22.91s/it][2025-04-25 07:51:04,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 07:51:04,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.97 | bwd_microstep: 5740.55 | bwd_inner_microstep: 5625.40 | bwd_allreduce_microstep: 115.10 | step_microstep: 19.14
[2025-04-25 07:51:04,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.97 | bwd: 5740.56 | bwd_inner: 5625.40 | bwd_allreduce: 115.12 | step: 19.14

  0%|          | 3/41250 [01:00<187:55:43, 16.40s/it]
                                                     
{'loss': 1.1894, 'grad_norm': 17.531147003173828, 'learning_rate': 9.693053311793216e-08, 'epoch': 0.0}

  0%|          | 3/41250 [01:00<187:55:43, 16.40s/it][2025-04-25 07:51:12,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.84 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 07:51:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.39 | bwd_microstep: 5681.74 | bwd_inner_microstep: 5611.38 | bwd_allreduce_microstep: 70.32 | step_microstep: 23.66
[2025-04-25 07:51:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.39 | bwd: 5681.76 | bwd_inner: 5611.38 | bwd_allreduce: 70.34 | step: 23.66

  0%|          | 4/41250 [01:09<152:35:25, 13.32s/it]evaluate!
evaluate!

                                                     
{'loss': 0.9921, 'grad_norm': 15.2952880859375, 'learning_rate': 1.2924071082390955e-07, 'epoch': 0.0}

  0%|          | 4/41250 [01:09<152:35:25, 13.32s/it]evaluate!
1
1
[WARNING|trainer.py:803] 2025-04-25 07:51:15,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1
[WARNING|trainer.py:803] 2025-04-25 07:51:15,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:51:16,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2
2
[WARNING|trainer.py:803] 2025-04-25 07:51:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2
[WARNING|trainer.py:803] 2025-04-25 07:51:18,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:51:19,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3
3
3
[WARNING|trainer.py:803] 2025-04-25 07:51:20,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[WARNING|trainer.py:803] 2025-04-25 07:51:20,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[WARNING|trainer.py:803] 2025-04-25 07:51:21,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[INFO|trainer.py:3910] 2025-04-25 07:51:32,907 >> Saving model checkpoint to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast
[INFO|configuration_utils.py:420] 2025-04-25 07:51:32,912 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/config.json
[INFO|configuration_utils.py:909] 2025-04-25 07:51:32,913 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/generation_config.json
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1219, in <module>
[rank1]:     main()
[rank1]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1214, in main
[rank1]:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
[rank1]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 2171, in train
[rank1]:     return inner_training_loop(
[rank1]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 2598, in _inner_training_loop
[rank1]:     self._maybe_log_save_evaluate(
[rank1]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 3071, in _maybe_log_save_evaluate
[rank1]:     metrics = self._evaluate(trial, ignore_keys_for_eval)
[rank1]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 3025, in _evaluate
[rank1]:     metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
[rank1]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 282, in evaluate
[rank1]:     self.save_lora_weights(self.model, self.args.output_dir)
[rank1]: AttributeError: 'CustomTrainer' object has no attribute 'save_lora_weights'
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1219, in <module>
[rank2]:     main()
[rank2]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 1214, in main
[rank2]:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
[rank2]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 2171, in train
[rank2]:     return inner_training_loop(
[rank2]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 2598, in _inner_training_loop
[rank2]:     self._maybe_log_save_evaluate(
[rank2]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 3071, in _maybe_log_save_evaluate
[rank2]:     metrics = self._evaluate(trial, ignore_keys_for_eval)
[rank2]:   File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/transformers/trainer.py", line 3025, in _evaluate
[rank2]:     metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
[rank2]:   File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 282, in evaluate
[rank2]:     self.save_lora_weights(self.model, self.args.output_dir)
[rank2]: AttributeError: 'CustomTrainer' object has no attribute 'save_lora_weights'
W0425 07:51:40.631898 2673708 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2673802 closing signal SIGTERM
W0425 07:51:40.634155 2673708 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2673804 closing signal SIGTERM
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 21 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
E0425 07:51:45.660335 2673708 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 2673803) of binary: /home/wangjiarui/.conda/envs/intern25/bin/python
Traceback (most recent call last):
  File "/home/wangjiarui/.conda/envs/intern25/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train/train_qa.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-25_07:51:40
  host      : amax
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 2673803)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
W0425 07:53:36.100325 2679929 site-packages/torch/distributed/run.py:793] 
W0425 07:53:36.100325 2679929 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:53:36.100325 2679929 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 07:53:36.100325 2679929 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-25 07:53:38,044] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:53:38,046] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:53:38,053] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
Replace train sampler!!Replace train sampler!!

petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.petrel_client is not installed. Using PIL to load images.

[2025-04-25 07:53:41,686] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:53:41,686] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
04/25/2025 07:53:41 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:53:41 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-53-41_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:53:41 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:53:41,862 >> loading file chat_template.jinja
[2025-04-25 07:53:41,863] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:53:41,872] [INFO] [comm.py:652:init_distributed] cdb=None
04/25/2025 07:53:41 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:53:41 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:53:42,322 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:53:42 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:53:42,459 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:53:42,462 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:53:42 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:53:42,462 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:53:42,463 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:53:42,465 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:53:42,533 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:53:42,534 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

this model
[WARNING|logging.py:328] 2025-04-25 07:53:42,611 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
this model
[WARNING|logging.py:328] 2025-04-25 07:53:42,684 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 None False
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
None False
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.24it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.36it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.89it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.16it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.31it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.83it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.57it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.68it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.32it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.85it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.54it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:53:48,556 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:53:48,556 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:53:48,567 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:53:48,567 >> Generate config GenerationConfig {}

04/25/2025 07:53:48 - INFO - __main__ - Finished
04/25/2025 07:53:48 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:53:48 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:53:48 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:53:48 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:53:48 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:53:48 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:53:48 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:53:48 - INFO - __main__ - Formatting inputs...Skip in lazy mode

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.98it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.67it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:53:48,573 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.67it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.29it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:53:48,686 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
04/25/2025 07:53:49 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/25/2025 07:53:49 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:53:49 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:53:49 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:53:49 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:53:49 - INFO - __main__ - Formatting inputs...Skip in lazy mode
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f4561718c40>
04/25/2025 07:53:49 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 3
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f4dcf09ad30>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f0a44158c40>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=2,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-53-41_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/25/2025 07:53:50 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-53-41_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-25 07:53:50,250 >> Using auto half precision backend
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-53-41_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[WARNING|trainer.py:803] 2025-04-25 07:53:50,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:53:50,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:53:50,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:53:50,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:53:50,502] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-25 07:53:50,502] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 3
[WARNING|trainer.py:803] 2025-04-25 07:53:50,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:53:50,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
[2025-04-25 07:53:57,168] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.6174883842468262 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6031739711761475 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6032898426055908 seconds
[2025-04-25 07:53:58,221] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-25 07:53:58,221] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-25 07:53:58,307] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-25 07:53:58,308] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-25 07:53:58,308] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-25 07:53:58,308] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-25 07:53:58,308] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-25 07:53:58,308] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-25 07:53:58,308] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-25 07:53:58,700] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-25 07:53:58,701] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.08 GB         CA 18.33 GB         Max_CA 18 GB 
[2025-04-25 07:53:58,702] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.44 GB, percent = 10.9%
[2025-04-25 07:53:58,887] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-25 07:53:58,888] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.11 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:53:58,888] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.46 GB, percent = 10.9%
[2025-04-25 07:53:58,888] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-25 07:53:59,063] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-25 07:53:59,064] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.05 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:53:59,064] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.6 GB, percent = 11.0%
[2025-04-25 07:53:59,070] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-25 07:53:59,070] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-25 07:53:59,070] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f4dcc647100>
[2025-04-25 07:53:59,070] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-25 07:53:59,080] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f4dccfa1070>
[2025-04-25 07:53:59,081] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-25 07:53:59,082] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-25 07:53:59,083] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-25 07:53:59,084] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   train_batch_size ............. 12
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   world_size ................... 3
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-25 07:53:59,085] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-25 07:53:59,086] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 12, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-25 07:53:59,087 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-25 07:53:59,087 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-25 07:53:59,087 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-25 07:53:59,087 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-25 07:53:59,087 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2376] 2025-04-25 07:53:59,087 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-25 07:53:59,087 >>   Total optimization steps = 41,250
[INFO|trainer.py:2378] 2025-04-25 07:53:59,094 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/41250 [00:00<?, ?it/s][2025-04-25 07:54:00,604] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:00,622] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:01,285] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:54:06,117] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:06,184] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:07,189] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:54:11,534] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:11,714] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:12,708] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:54:16,913] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:17,450] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:54:18,257] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
8 7 7
8 7 7
[2025-04-25 07:54:41,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.73 | optimizer_gradients: 21.32 | optimizer_step: 4.41
[2025-04-25 07:54:41,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 6730.12 | bwd_microstep: 5659.91 | bwd_inner_microstep: 5647.17 | bwd_allreduce_microstep: 12.71 | step_microstep: 84.09
[2025-04-25 07:54:41,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 6730.09 | bwd: 5659.93 | bwd_inner: 5647.17 | bwd_allreduce: 12.72 | step: 84.09

  0%|          | 1/41250 [00:42<489:13:28, 42.70s/it]
                                                     
{'loss': 1.3896, 'grad_norm': 15.716039657592773, 'learning_rate': 3.231017770597739e-08, 'epoch': 0.0}

  0%|          | 1/41250 [00:42<489:13:28, 42.70s/it][2025-04-25 07:54:50,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.79 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 07:54:50,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2788.89 | bwd_microstep: 5943.48 | bwd_inner_microstep: 5588.06 | bwd_allreduce_microstep: 355.37 | step_microstep: 20.21
[2025-04-25 07:54:50,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2788.89 | bwd: 5943.49 | bwd_inner: 5588.06 | bwd_allreduce: 355.39 | step: 20.21

  0%|          | 2/41250 [00:51<261:42:49, 22.84s/it]
                                                     
{'loss': 0.9697, 'grad_norm': 11.641993522644043, 'learning_rate': 6.462035541195477e-08, 'epoch': 0.0}

  0%|          | 2/41250 [00:51<261:42:49, 22.84s/it][2025-04-25 07:54:59,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 07:54:59,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2799.31 | bwd_microstep: 5671.04 | bwd_inner_microstep: 5604.12 | bwd_allreduce_microstep: 66.88 | step_microstep: 18.89
[2025-04-25 07:54:59,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2799.31 | bwd: 5671.05 | bwd_inner: 5604.12 | bwd_allreduce: 66.89 | step: 18.90

  0%|          | 3/41250 [01:00<186:56:29, 16.32s/it]
                                                     
{'loss': 1.1738, 'grad_norm': 17.41244125366211, 'learning_rate': 9.693053311793216e-08, 'epoch': 0.0}

  0%|          | 3/41250 [01:00<186:56:29, 16.32s/it][2025-04-25 07:55:07,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 07:55:07,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2807.00 | bwd_microstep: 5683.64 | bwd_inner_microstep: 5626.74 | bwd_allreduce_microstep: 56.85 | step_microstep: 18.40
[2025-04-25 07:55:07,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2807.00 | bwd: 5683.65 | bwd_inner: 5626.74 | bwd_allreduce: 56.87 | step: 18.40

  0%|          | 4/41250 [01:08<151:53:56, 13.26s/it]evaluate!
evaluate!

                                                     
{'loss': 0.9953, 'grad_norm': 15.2977876663208, 'learning_rate': 1.2924071082390955e-07, 'epoch': 0.0}

  0%|          | 4/41250 [01:08<151:53:56, 13.26s/it]evaluate!
1
1
1
[WARNING|trainer.py:803] 2025-04-25 07:55:10,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:55:10,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:55:10,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2
2
2
[WARNING|trainer.py:803] 2025-04-25 07:55:12,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:55:12,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-25 07:55:13,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3
3
3
[WARNING|trainer.py:803] 2025-04-25 07:55:15,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[WARNING|trainer.py:803] 2025-04-25 07:55:15,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[WARNING|trainer.py:803] 2025-04-25 07:55:15,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.0
New best accuracy: 0.0. Saving model...
[INFO|trainer.py:3910] 2025-04-25 07:55:27,160 >> Saving model checkpoint to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast
[INFO|configuration_utils.py:420] 2025-04-25 07:55:27,165 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/config.json
[INFO|configuration_utils.py:909] 2025-04-25 07:55:27,166 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/generation_config.json
[INFO|modeling_utils.py:2996] 2025-04-25 07:56:12,162 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2491] 2025-04-25 07:56:12,164 >> tokenizer config file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/tokenizer_config.json
[INFO|tokenization_utils_base.py:2500] 2025-04-25 07:56:12,164 >> Special tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/special_tokens_map.json
[INFO|tokenization_utils_base.py:2553] 2025-04-25 07:56:12,164 >> added tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/added_tokens.json
04/25/2025 07:56:14 - INFO - __main__ - Saved LoRA weights to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/lora_weights.pth
[2025-04-25 07:56:22,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 07:56:22,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2790.32 | bwd_microstep: 5607.11 | bwd_inner_microstep: 5594.78 | bwd_allreduce_microstep: 12.28 | step_microstep: 19.10
[2025-04-25 07:56:22,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2790.32 | bwd: 5607.12 | bwd_inner: 5594.78 | bwd_allreduce: 12.30 | step: 19.11

  0%|          | 5/41250 [02:23<406:15:59, 35.46s/it]
                                                     
{'loss': 1.2025, 'grad_norm': 15.128987312316895, 'learning_rate': 1.6155088852988693e-07, 'epoch': 0.0}

  0%|          | 5/41250 [02:23<406:15:59, 35.46s/it][2025-04-25 07:56:31,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 07:56:31,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3370.26 | bwd_microstep: 5730.02 | bwd_inner_microstep: 5717.05 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.91
[2025-04-25 07:56:31,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3370.26 | bwd: 5730.03 | bwd_inner: 5717.05 | bwd_allreduce: 12.94 | step: 18.92

  0%|          | 6/41250 [02:32<303:54:16, 26.53s/it]
                                                     
{'loss': 1.0311, 'grad_norm': 12.863167762756348, 'learning_rate': 1.938610662358643e-07, 'epoch': 0.0}

  0%|          | 6/41250 [02:32<303:54:16, 26.53s/it]W0425 07:57:00.173141 2687168 site-packages/torch/distributed/run.py:793] 
W0425 07:57:00.173141 2687168 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:57:00.173141 2687168 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 07:57:00.173141 2687168 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:57:12.391524 2687306 site-packages/torch/distributed/run.py:793] 
W0425 07:57:12.391524 2687306 site-packages/torch/distributed/run.py:793] *****************************************
W0425 07:57:12.391524 2687306 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 07:57:12.391524 2687306 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-25 07:57:14,346] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:14,349] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:14,349] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 07:57:17,584] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:57:17,584] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
04/25/2025 07:57:17 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:57:17 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-57-17_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:57:17 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-25 07:57:17,769 >> loading file chat_template.jinja
[2025-04-25 07:57:17,853] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-25 07:57:17,866] [INFO] [comm.py:652:init_distributed] cdb=None
04/25/2025 07:57:17 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
04/25/2025 07:57:17 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
[INFO|tokenization_utils_base.py:2304] 2025-04-25 07:57:18,232 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/25/2025 07:57:18 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-25 07:57:18,377 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-25 07:57:18,379 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/25/2025 07:57:18 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-25 07:57:18,379 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-25 07:57:18,380 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-25 07:57:18,382 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-25 07:57:18,449 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-25 07:57:18,449 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

this model
this model
[WARNING|logging.py:328] 2025-04-25 07:57:18,620 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[WARNING|logging.py:328] 2025-04-25 07:57:18,640 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight2 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight2 None False
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.93it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.24it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.26it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.91it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.16it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.23it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.32it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.67it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.67it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.31it/s]
[INFO|modeling_utils.py:4888] 2025-04-25 07:57:24,509 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-25 07:57:24,509 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-25 07:57:24,519 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-25 07:57:24,519 >> Generate config GenerationConfig {}

04/25/2025 07:57:24 - INFO - __main__ - Finished
04/25/2025 07:57:24 - INFO - __main__ - model.config.force_image_size: 448
04/25/2025 07:57:24 - INFO - __main__ - data_args.force_image_size: 448
04/25/2025 07:57:24 - INFO - __main__ - model.config.vision_config.image_size: 448
04/25/2025 07:57:24 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:57:24 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:57:24 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:57:24 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:57:24 - INFO - __main__ - Formatting inputs...Skip in lazy mode

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.01it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.66it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:57:24,555 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.83it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.52it/s]
[WARNING|modeling_utils.py:4890] 2025-04-25 07:57:24,573 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
04/25/2025 07:57:25 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/25/2025 07:57:25 - INFO - __main__ - [Dataset] num_image_token: 256
04/25/2025 07:57:25 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/25/2025 07:57:25 - INFO - __main__ - [Dataset] use_thumbnail: True
04/25/2025 07:57:25 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/25/2025 07:57:25 - INFO - __main__ - Formatting inputs...Skip in lazy mode
04/25/2025 07:57:25 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 3
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7faad5bb4eb0>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f4a4b41b190>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7fead4058cd0>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
training_args 04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=2,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-57-17_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight

04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
training_args 04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-57-17_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/25/2025 07:57:26 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr25_07-57-17_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-25 07:57:26,197 >> Using auto half precision backend
[WARNING|trainer.py:803] 2025-04-25 07:57:26,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:57:26,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:57:26,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:57:26,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:57:26,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-25 07:57:26,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-25 07:57:26,431] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-25 07:57:26,431] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 3
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
[2025-04-25 07:57:32,749] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.5859417915344238 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6033613681793213 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.6033060550689697 seconds
[2025-04-25 07:57:33,803] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-25 07:57:33,803] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-25 07:57:33,895] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-25 07:57:33,895] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-25 07:57:33,895] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-25 07:57:33,895] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-25 07:57:33,895] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-25 07:57:33,895] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-25 07:57:33,895] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-25 07:57:34,310] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-25 07:57:34,310] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.08 GB         CA 18.33 GB         Max_CA 18 GB 
[2025-04-25 07:57:34,311] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.16 GB, percent = 10.8%
[2025-04-25 07:57:34,484] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-25 07:57:34,485] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.11 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:57:34,485] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.15 GB, percent = 10.8%
[2025-04-25 07:57:34,485] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-25 07:57:34,656] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-25 07:57:34,657] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.05 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-25 07:57:34,657] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 27.26 GB, percent = 10.8%
[2025-04-25 07:57:34,663] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-25 07:57:34,663] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-25 07:57:34,663] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7faaba4ef190>
[2025-04-25 07:57:34,664] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-25 07:57:34,674] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-25 07:57:34,674] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7faaba4e30a0>
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-25 07:57:34,675] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-25 07:57:34,676] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   train_batch_size ............. 12
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   world_size ................... 3
[2025-04-25 07:57:34,677] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-25 07:57:34,678] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-25 07:57:34,678] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-25 07:57:34,678] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-25 07:57:34,678] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-25 07:57:34,678] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 12, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-25 07:57:34,679 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-25 07:57:34,679 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-25 07:57:34,679 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-25 07:57:34,679 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-25 07:57:34,679 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2376] 2025-04-25 07:57:34,679 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-25 07:57:34,679 >>   Total optimization steps = 41,250
[INFO|trainer.py:2378] 2025-04-25 07:57:34,686 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/41250 [00:00<?, ?it/s][2025-04-25 07:57:36,205] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:36,325] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:36,905] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:57:41,729] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:42,100] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:42,929] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:57:47,205] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:47,538] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:48,568] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 07:57:52,615] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:53,079] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 07:57:54,097] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
8 7 7
8 7 7
[2025-04-25 07:58:16,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.82 | optimizer_gradients: 23.44 | optimizer_step: 4.55
[2025-04-25 07:58:16,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 7091.15 | bwd_microstep: 5689.96 | bwd_inner_microstep: 5677.24 | bwd_allreduce_microstep: 12.68 | step_microstep: 88.34
[2025-04-25 07:58:16,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 7091.12 | bwd: 5689.97 | bwd_inner: 5677.24 | bwd_allreduce: 12.69 | step: 88.34

  0%|          | 1/41250 [00:42<481:59:51, 42.07s/it]
                                                     
{'loss': 1.3896, 'grad_norm': 15.71754264831543, 'learning_rate': 3.231017770597739e-08, 'epoch': 0.0}

  0%|          | 1/41250 [00:42<481:59:51, 42.07s/it][2025-04-25 07:58:25,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.23 | optimizer_step: 1.08
[2025-04-25 07:58:25,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2808.95 | bwd_microstep: 5985.55 | bwd_inner_microstep: 5603.11 | bwd_allreduce_microstep: 382.37 | step_microstep: 20.29
[2025-04-25 07:58:25,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2808.95 | bwd: 5985.56 | bwd_inner: 5603.11 | bwd_allreduce: 382.40 | step: 20.29

  0%|          | 2/41250 [00:51<259:08:13, 22.62s/it]
                                                     
{'loss': 0.9697, 'grad_norm': 11.641520500183105, 'learning_rate': 6.462035541195477e-08, 'epoch': 0.0}

  0%|          | 2/41250 [00:51<259:08:13, 22.62s/it][2025-04-25 07:58:34,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 07:58:34,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.89 | bwd_microstep: 5700.88 | bwd_inner_microstep: 5626.25 | bwd_allreduce_microstep: 74.58 | step_microstep: 19.80
[2025-04-25 07:58:34,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.89 | bwd: 5700.89 | bwd_inner: 5626.25 | bwd_allreduce: 74.59 | step: 19.80

  0%|          | 3/41250 [00:59<185:49:28, 16.22s/it]
                                                     
{'loss': 1.1689, 'grad_norm': 17.227046966552734, 'learning_rate': 9.693053311793216e-08, 'epoch': 0.0}

  0%|          | 3/41250 [00:59<185:49:28, 16.22s/it][2025-04-25 07:58:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.20
[2025-04-25 07:58:42,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.25 | bwd_microstep: 5679.58 | bwd_inner_microstep: 5635.48 | bwd_allreduce_microstep: 44.04 | step_microstep: 19.72
[2025-04-25 07:58:42,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.25 | bwd: 5679.59 | bwd_inner: 5635.48 | bwd_allreduce: 44.06 | step: 19.72

  0%|          | 4/41250 [01:08<151:17:50, 13.21s/it]
                                                     
{'loss': 1.0071, 'grad_norm': 15.48046875, 'learning_rate': 1.2924071082390955e-07, 'epoch': 0.0}

  0%|          | 4/41250 [01:08<151:17:50, 13.21s/it][2025-04-25 07:58:51,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.15
[2025-04-25 07:58:51,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2811.70 | bwd_microstep: 5659.41 | bwd_inner_microstep: 5623.94 | bwd_allreduce_microstep: 35.42 | step_microstep: 19.14
[2025-04-25 07:58:51,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2811.70 | bwd: 5659.42 | bwd_inner: 5623.94 | bwd_allreduce: 35.43 | step: 19.14

  0%|          | 5/41250 [01:16<132:05:22, 11.53s/it]
                                                     
{'loss': 1.2079, 'grad_norm': 15.139225006103516, 'learning_rate': 1.6155088852988693e-07, 'epoch': 0.0}

  0%|          | 5/41250 [01:16<132:05:22, 11.53s/it][2025-04-25 07:59:00,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 07:59:00,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3387.19 | bwd_microstep: 5740.51 | bwd_inner_microstep: 5727.69 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.56
[2025-04-25 07:59:00,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3387.19 | bwd: 5740.52 | bwd_inner: 5727.69 | bwd_allreduce: 12.79 | step: 19.56

  0%|          | 6/41250 [01:26<123:03:32, 10.74s/it]
                                                     
{'loss': 1.0338, 'grad_norm': 12.511625289916992, 'learning_rate': 1.938610662358643e-07, 'epoch': 0.0}

  0%|          | 6/41250 [01:26<123:03:32, 10.74s/it][2025-04-25 07:59:09,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 07:59:09,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.86 | bwd_microstep: 5692.32 | bwd_inner_microstep: 5679.40 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.82
[2025-04-25 07:59:09,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.86 | bwd: 5692.34 | bwd_inner: 5679.40 | bwd_allreduce: 12.90 | step: 18.82

  0%|          | 7/41250 [01:34<115:04:57, 10.05s/it]
                                                     
{'loss': 1.147, 'grad_norm': 14.809407234191895, 'learning_rate': 2.261712439418417e-07, 'epoch': 0.0}

  0%|          | 7/41250 [01:34<115:04:57, 10.05s/it][2025-04-25 07:59:17,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-25 07:59:17,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.52 | bwd_microstep: 5738.44 | bwd_inner_microstep: 5624.28 | bwd_allreduce_microstep: 114.12 | step_microstep: 19.65
[2025-04-25 07:59:17,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.52 | bwd: 5738.46 | bwd_inner: 5624.28 | bwd_allreduce: 114.14 | step: 19.66

  0%|          | 8/41250 [01:43<109:58:02,  9.60s/it]
                                                     
{'loss': 1.128, 'grad_norm': 11.448230743408203, 'learning_rate': 2.584814216478191e-07, 'epoch': 0.0}

  0%|          | 8/41250 [01:43<109:58:02,  9.60s/it][2025-04-25 07:59:26,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-25 07:59:26,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.16 | bwd_microstep: 5672.22 | bwd_inner_microstep: 5637.71 | bwd_allreduce_microstep: 34.46 | step_microstep: 19.56
[2025-04-25 07:59:26,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.16 | bwd: 5672.24 | bwd_inner: 5637.71 | bwd_allreduce: 34.48 | step: 19.56

  0%|          | 9/41250 [01:51<106:18:14,  9.28s/it]
                                                     
{'loss': 0.8527, 'grad_norm': 12.677689552307129, 'learning_rate': 2.9079159935379645e-07, 'epoch': 0.0}

  0%|          | 9/41250 [01:51<106:18:14,  9.28s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x2d607280] moov atom not found
[07:59:26] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4...
[2025-04-25 07:59:35,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 07:59:35,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.20 | bwd_microstep: 5688.96 | bwd_inner_microstep: 5676.15 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.87
[2025-04-25 07:59:35,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.20 | bwd: 5688.97 | bwd_inner: 5676.15 | bwd_allreduce: 12.78 | step: 18.88

  0%|          | 10/41250 [02:00<103:57:43,  9.08s/it]
                                                      
{'loss': 1.1827, 'grad_norm': 15.625365257263184, 'learning_rate': 3.2310177705977386e-07, 'epoch': 0.0}

  0%|          | 10/41250 [02:00<103:57:43,  9.08s/it][2025-04-25 07:59:44,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.19 | optimizer_step: 1.00
[2025-04-25 07:59:44,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.44 | bwd_microstep: 6058.31 | bwd_inner_microstep: 5688.50 | bwd_allreduce_microstep: 369.76 | step_microstep: 19.36
[2025-04-25 07:59:44,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.44 | bwd: 6058.33 | bwd_inner: 5688.50 | bwd_allreduce: 369.78 | step: 19.36

  0%|          | 11/41250 [02:09<103:39:35,  9.05s/it]
                                                      
{'loss': 1.1381, 'grad_norm': 16.2872314453125, 'learning_rate': 3.5541195476575127e-07, 'epoch': 0.0}

  0%|          | 11/41250 [02:09<103:39:35,  9.05s/it][2025-04-25 07:59:52,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.25 | optimizer_step: 1.04
[2025-04-25 07:59:52,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.33 | bwd_microstep: 5693.86 | bwd_inner_microstep: 5632.33 | bwd_allreduce_microstep: 61.49 | step_microstep: 19.49
[2025-04-25 07:59:52,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.33 | bwd: 5693.88 | bwd_inner: 5632.33 | bwd_allreduce: 61.51 | step: 19.50

  0%|          | 12/41250 [02:18<102:07:34,  8.92s/it]
                                                      
{'loss': 0.8199, 'grad_norm': 15.752564430236816, 'learning_rate': 3.877221324717286e-07, 'epoch': 0.0}

  0%|          | 12/41250 [02:18<102:07:34,  8.92s/it][2025-04-25 08:00:01,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:00:01,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.53 | bwd_microstep: 5701.59 | bwd_inner_microstep: 5688.78 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.52
[2025-04-25 08:00:01,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.53 | bwd: 5701.61 | bwd_inner: 5688.78 | bwd_allreduce: 12.78 | step: 18.52

  0%|          | 13/41250 [02:26<101:09:05,  8.83s/it]
                                                      
{'loss': 1.1353, 'grad_norm': 15.49112606048584, 'learning_rate': 4.2003231017770603e-07, 'epoch': 0.0}

  0%|          | 13/41250 [02:26<101:09:05,  8.83s/it][2025-04-25 08:00:09,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.20 | optimizer_step: 1.09
[2025-04-25 08:00:09,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.89 | bwd_microstep: 5703.42 | bwd_inner_microstep: 5642.39 | bwd_allreduce_microstep: 60.98 | step_microstep: 19.91
[2025-04-25 08:00:09,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.89 | bwd: 5703.43 | bwd_inner: 5642.39 | bwd_allreduce: 61.00 | step: 19.92

  0%|          | 14/41250 [02:35<100:23:32,  8.76s/it]
                                                      
{'loss': 1.1384, 'grad_norm': 14.890922546386719, 'learning_rate': 4.523424878836834e-07, 'epoch': 0.0}

  0%|          | 14/41250 [02:35<100:23:32,  8.76s/it][2025-04-25 08:00:18,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 08:00:18,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.73 | bwd_microstep: 5762.89 | bwd_inner_microstep: 5667.86 | bwd_allreduce_microstep: 94.98 | step_microstep: 19.73
[2025-04-25 08:00:18,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.73 | bwd: 5762.90 | bwd_inner: 5667.86 | bwd_allreduce: 95.00 | step: 19.73

  0%|          | 15/41250 [02:44<100:07:24,  8.74s/it]
                                                      
{'loss': 0.8912, 'grad_norm': 11.876169204711914, 'learning_rate': 4.846526655896607e-07, 'epoch': 0.0}

  0%|          | 15/41250 [02:44<100:07:24,  8.74s/it][2025-04-25 08:00:27,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-25 08:00:27,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.78 | bwd_microstep: 5693.63 | bwd_inner_microstep: 5671.41 | bwd_allreduce_microstep: 22.16 | step_microstep: 19.51
[2025-04-25 08:00:27,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.78 | bwd: 5693.65 | bwd_inner: 5671.41 | bwd_allreduce: 22.19 | step: 19.51

  0%|          | 16/41250 [02:52<99:42:21,  8.70s/it] 
                                                     
{'loss': 0.8388, 'grad_norm': 8.377857208251953, 'learning_rate': 5.169628432956382e-07, 'epoch': 0.0}

  0%|          | 16/41250 [02:52<99:42:21,  8.70s/it][2025-04-25 08:00:35,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:00:35,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.67 | bwd_microstep: 5700.66 | bwd_inner_microstep: 5638.17 | bwd_allreduce_microstep: 62.44 | step_microstep: 18.76
[2025-04-25 08:00:35,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.67 | bwd: 5700.67 | bwd_inner: 5638.17 | bwd_allreduce: 62.46 | step: 18.76

  0%|          | 17/41250 [03:01<99:21:41,  8.68s/it]
                                                     
{'loss': 0.9398, 'grad_norm': 14.038540840148926, 'learning_rate': 5.492730210016156e-07, 'epoch': 0.0}

  0%|          | 17/41250 [03:01<99:21:41,  8.68s/it][2025-04-25 08:00:44,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.10 | optimizer_step: 1.07
[2025-04-25 08:00:44,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.30 | bwd_microstep: 5691.65 | bwd_inner_microstep: 5669.18 | bwd_allreduce_microstep: 22.42 | step_microstep: 19.90
[2025-04-25 08:00:44,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.30 | bwd: 5691.67 | bwd_inner: 5669.18 | bwd_allreduce: 22.45 | step: 19.91

  0%|          | 18/41250 [03:09<99:09:29,  8.66s/it]
                                                     
{'loss': 1.1166, 'grad_norm': 13.930654525756836, 'learning_rate': 5.815831987075929e-07, 'epoch': 0.0}

  0%|          | 18/41250 [03:09<99:09:29,  8.66s/it][2025-04-25 08:00:53,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 08:00:53,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.31 | bwd_microstep: 5712.19 | bwd_inner_microstep: 5637.84 | bwd_allreduce_microstep: 74.30 | step_microstep: 19.52
[2025-04-25 08:00:53,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.31 | bwd: 5712.20 | bwd_inner: 5637.84 | bwd_allreduce: 74.32 | step: 19.53

  0%|          | 19/41250 [03:18<99:01:49,  8.65s/it]
                                                     
{'loss': 0.9856, 'grad_norm': 12.493513107299805, 'learning_rate': 6.138933764135703e-07, 'epoch': 0.0}

  0%|          | 19/41250 [03:18<99:01:49,  8.65s/it][2025-04-25 08:01:01,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 08:01:01,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.44 | bwd_microstep: 5703.81 | bwd_inner_microstep: 5642.58 | bwd_allreduce_microstep: 61.18 | step_microstep: 19.35
[2025-04-25 08:01:01,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.44 | bwd: 5703.82 | bwd_inner: 5642.58 | bwd_allreduce: 61.21 | step: 19.35

  0%|          | 20/41250 [03:27<98:55:42,  8.64s/it]
                                                     
{'loss': 0.7093, 'grad_norm': 13.370922088623047, 'learning_rate': 6.462035541195477e-07, 'epoch': 0.0}

  0%|          | 20/41250 [03:27<98:55:42,  8.64s/it][2025-04-25 08:01:10,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 08:01:10,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.62 | bwd_microstep: 5766.24 | bwd_inner_microstep: 5632.45 | bwd_allreduce_microstep: 133.74 | step_microstep: 19.22
[2025-04-25 08:01:10,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.62 | bwd: 5766.26 | bwd_inner: 5632.45 | bwd_allreduce: 133.77 | step: 19.22

  0%|          | 21/41250 [03:35<99:02:55,  8.65s/it]
                                                     
{'loss': 0.9888, 'grad_norm': 13.096196174621582, 'learning_rate': 6.785137318255252e-07, 'epoch': 0.01}

  0%|          | 21/41250 [03:35<99:02:55,  8.65s/it][2025-04-25 08:01:19,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 08:01:19,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.16 | bwd_microstep: 6071.26 | bwd_inner_microstep: 5641.58 | bwd_allreduce_microstep: 429.63 | step_microstep: 19.36
[2025-04-25 08:01:19,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.16 | bwd: 6071.28 | bwd_inner: 5641.58 | bwd_allreduce: 429.66 | step: 19.36

  0%|          | 22/41250 [03:44<100:11:32,  8.75s/it]
                                                      
{'loss': 1.3353, 'grad_norm': 15.138091087341309, 'learning_rate': 7.108239095315025e-07, 'epoch': 0.01}

  0%|          | 22/41250 [03:44<100:11:32,  8.75s/it][2025-04-25 08:01:28,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 08:01:28,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5697.24 | bwd_inner_microstep: 5684.42 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.27
[2025-04-25 08:01:28,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.94 | bwd: 5697.26 | bwd_inner: 5684.42 | bwd_allreduce: 12.80 | step: 19.28

  0%|          | 23/41250 [03:53<99:47:40,  8.71s/it] 
                                                     
{'loss': 0.9033, 'grad_norm': 11.826258659362793, 'learning_rate': 7.431340872374798e-07, 'epoch': 0.01}

  0%|          | 23/41250 [03:53<99:47:40,  8.71s/it][2025-04-25 08:01:36,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 08:01:36,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.33 | bwd_microstep: 5692.98 | bwd_inner_microstep: 5680.27 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.23
[2025-04-25 08:01:36,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.33 | bwd: 5693.00 | bwd_inner: 5680.27 | bwd_allreduce: 12.68 | step: 19.23

  0%|          | 24/41250 [04:02<99:28:43,  8.69s/it]
                                                     
{'loss': 1.1965, 'grad_norm': 10.768841743469238, 'learning_rate': 7.754442649434572e-07, 'epoch': 0.01}

  0%|          | 24/41250 [04:02<99:28:43,  8.69s/it][2025-04-25 08:01:45,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 08:01:45,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 5679.72 | bwd_inner_microstep: 5637.23 | bwd_allreduce_microstep: 42.45 | step_microstep: 19.72
[2025-04-25 08:01:45,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 5679.74 | bwd_inner: 5637.23 | bwd_allreduce: 42.47 | step: 19.72

  0%|          | 25/41250 [04:10<99:08:07,  8.66s/it]
                                                     
{'loss': 1.0032, 'grad_norm': 14.346658706665039, 'learning_rate': 8.077544426494346e-07, 'epoch': 0.01}

  0%|          | 25/41250 [04:10<99:08:07,  8.66s/it][2025-04-25 08:01:54,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:01:54,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 6091.88 | bwd_inner_microstep: 5685.47 | bwd_allreduce_microstep: 406.36 | step_microstep: 18.49
[2025-04-25 08:01:54,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 6091.89 | bwd_inner: 5685.47 | bwd_allreduce: 406.38 | step: 18.50

  0%|          | 26/41250 [04:19<100:22:36,  8.77s/it]
                                                      
{'loss': 1.0644, 'grad_norm': 13.056071281433105, 'learning_rate': 8.400646203554121e-07, 'epoch': 0.01}

  0%|          | 26/41250 [04:19<100:22:36,  8.77s/it][2025-04-25 08:02:02,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 08:02:02,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.10 | bwd_microstep: 5749.66 | bwd_inner_microstep: 5671.09 | bwd_allreduce_microstep: 78.52 | step_microstep: 19.22
[2025-04-25 08:02:02,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.10 | bwd: 5749.67 | bwd_inner: 5671.08 | bwd_allreduce: 78.55 | step: 19.22

  0%|          | 27/41250 [04:28<100:03:11,  8.74s/it]
                                                      
{'loss': 1.0556, 'grad_norm': 11.639249801635742, 'learning_rate': 8.723747980613895e-07, 'epoch': 0.01}

  0%|          | 27/41250 [04:28<100:03:11,  8.74s/it][2025-04-25 08:02:11,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.24
[2025-04-25 08:02:11,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5676.41 | bwd_inner_microstep: 5663.48 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.96
[2025-04-25 08:02:11,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5676.42 | bwd_inner: 5663.48 | bwd_allreduce: 12.90 | step: 19.96

  0%|          | 28/41250 [04:36<99:35:23,  8.70s/it] 
                                                     
{'loss': 1.0484, 'grad_norm': 16.2164249420166, 'learning_rate': 9.046849757673668e-07, 'epoch': 0.01}

  0%|          | 28/41250 [04:36<99:35:23,  8.70s/it][2025-04-25 08:02:20,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 1.14
[2025-04-25 08:02:20,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.44 | bwd_microstep: 5732.67 | bwd_inner_microstep: 5683.89 | bwd_allreduce_microstep: 48.73 | step_microstep: 20.09
[2025-04-25 08:02:20,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.44 | bwd: 5732.68 | bwd_inner: 5683.89 | bwd_allreduce: 48.75 | step: 20.09

  0%|          | 29/41250 [04:45<99:29:23,  8.69s/it]
                                                     
{'loss': 0.6663, 'grad_norm': 9.069692611694336, 'learning_rate': 9.369951534733441e-07, 'epoch': 0.01}

  0%|          | 29/41250 [04:45<99:29:23,  8.69s/it][2025-04-25 08:02:28,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:02:28,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.14 | bwd_microstep: 5691.66 | bwd_inner_microstep: 5678.95 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.78
[2025-04-25 08:02:28,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.14 | bwd: 5691.68 | bwd_inner: 5678.95 | bwd_allreduce: 12.69 | step: 18.78

  0%|          | 30/41250 [04:54<99:15:10,  8.67s/it]
                                                     
{'loss': 1.2061, 'grad_norm': 15.919360160827637, 'learning_rate': 9.693053311793215e-07, 'epoch': 0.01}

  0%|          | 30/41250 [04:54<99:15:10,  8.67s/it][2025-04-25 08:02:37,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 08:02:37,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.68 | bwd_microstep: 5686.99 | bwd_inner_microstep: 5674.17 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.15
[2025-04-25 08:02:37,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.68 | bwd: 5687.00 | bwd_inner: 5674.17 | bwd_allreduce: 12.80 | step: 19.15

  0%|          | 31/41250 [05:02<99:02:20,  8.65s/it]
                                                     
{'loss': 0.9534, 'grad_norm': 13.733417510986328, 'learning_rate': 1.001615508885299e-06, 'epoch': 0.01}

  0%|          | 31/41250 [05:02<99:02:20,  8.65s/it][2025-04-25 08:02:46,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 08:02:46,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.00 | bwd_microstep: 5701.36 | bwd_inner_microstep: 5628.49 | bwd_allreduce_microstep: 72.83 | step_microstep: 19.43
[2025-04-25 08:02:46,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.00 | bwd: 5701.37 | bwd_inner: 5628.49 | bwd_allreduce: 72.84 | step: 19.44

  0%|          | 32/41250 [05:11<98:52:09,  8.64s/it]
                                                     
{'loss': 1.3693, 'grad_norm': 16.116548538208008, 'learning_rate': 1.0339256865912764e-06, 'epoch': 0.01}

  0%|          | 32/41250 [05:11<98:52:09,  8.64s/it][2025-04-25 08:02:54,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:02:54,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.06 | bwd_microstep: 5703.02 | bwd_inner_microstep: 5690.04 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.57
[2025-04-25 08:02:54,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.06 | bwd: 5703.03 | bwd_inner: 5690.04 | bwd_allreduce: 12.94 | step: 18.58

  0%|          | 33/41250 [05:20<98:51:44,  8.63s/it]
                                                     
{'loss': 0.9522, 'grad_norm': 12.828269004821777, 'learning_rate': 1.0662358642972536e-06, 'epoch': 0.01}

  0%|          | 33/41250 [05:20<98:51:44,  8.63s/it][2025-04-25 08:03:03,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.23 | optimizer_step: 1.07
[2025-04-25 08:03:03,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5767.23 | bwd_inner_microstep: 5629.68 | bwd_allreduce_microstep: 137.50 | step_microstep: 19.97
[2025-04-25 08:03:03,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5767.24 | bwd_inner: 5629.68 | bwd_allreduce: 137.52 | step: 19.97

  0%|          | 34/41250 [05:28<99:00:13,  8.65s/it]
                                                     
{'loss': 0.7928, 'grad_norm': 16.245237350463867, 'learning_rate': 1.098546042003231e-06, 'epoch': 0.01}

  0%|          | 34/41250 [05:28<99:00:13,  8.65s/it][2025-04-25 08:03:11,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-25 08:03:11,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.35 | bwd_microstep: 5687.66 | bwd_inner_microstep: 5674.52 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.22
[2025-04-25 08:03:11,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.35 | bwd: 5687.67 | bwd_inner: 5674.52 | bwd_allreduce: 13.10 | step: 19.22

  0%|          | 35/41250 [05:37<98:54:10,  8.64s/it]
                                                     
{'loss': 0.7122, 'grad_norm': 12.461237907409668, 'learning_rate': 1.1308562197092086e-06, 'epoch': 0.01}

  0%|          | 35/41250 [05:37<98:54:10,  8.64s/it][2025-04-25 08:03:20,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 08:03:20,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.34 | bwd_microstep: 5693.06 | bwd_inner_microstep: 5637.40 | bwd_allreduce_microstep: 55.60 | step_microstep: 19.00
[2025-04-25 08:03:20,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.34 | bwd: 5693.07 | bwd_inner: 5637.40 | bwd_allreduce: 55.62 | step: 19.00

  0%|          | 36/41250 [05:45<98:45:40,  8.63s/it]
                                                     
{'loss': 1.1494, 'grad_norm': 16.0876522064209, 'learning_rate': 1.1631663974151858e-06, 'epoch': 0.01}

  0%|          | 36/41250 [05:45<98:45:40,  8.63s/it][2025-04-25 08:03:29,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 08:03:29,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 6087.71 | bwd_inner_microstep: 5638.18 | bwd_allreduce_microstep: 449.48 | step_microstep: 19.20
[2025-04-25 08:03:29,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 6087.72 | bwd_inner: 5638.18 | bwd_allreduce: 449.50 | step: 19.20

  0%|          | 37/41250 [05:54<100:02:57,  8.74s/it]
                                                      
{'loss': 0.974, 'grad_norm': 12.466655731201172, 'learning_rate': 1.1954765751211633e-06, 'epoch': 0.01}

  0%|          | 37/41250 [05:54<100:02:57,  8.74s/it][2025-04-25 08:03:38,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:03:38,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.32 | bwd_microstep: 5754.17 | bwd_inner_microstep: 5634.37 | bwd_allreduce_microstep: 119.76 | step_microstep: 18.37
[2025-04-25 08:03:38,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.32 | bwd: 5754.18 | bwd_inner: 5634.37 | bwd_allreduce: 119.78 | step: 18.37

  0%|          | 38/41250 [06:03<99:46:53,  8.72s/it] 
                                                     
{'loss': 0.8905, 'grad_norm': 14.074410438537598, 'learning_rate': 1.2277867528271405e-06, 'epoch': 0.01}

  0%|          | 38/41250 [06:03<99:46:53,  8.72s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x3e5f1540] moov atom not found
[08:03:38] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4...
[2025-04-25 08:03:46,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 08:03:46,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.55 | bwd_microstep: 5699.41 | bwd_inner_microstep: 5672.61 | bwd_allreduce_microstep: 26.76 | step_microstep: 19.25
[2025-04-25 08:03:46,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.55 | bwd: 5699.43 | bwd_inner: 5672.60 | bwd_allreduce: 26.78 | step: 19.25

  0%|          | 39/41250 [06:12<99:31:03,  8.69s/it]
                                                     
{'loss': 1.1409, 'grad_norm': 12.480310440063477, 'learning_rate': 1.2600969305331182e-06, 'epoch': 0.01}

  0%|          | 39/41250 [06:12<99:31:03,  8.69s/it][2025-04-25 08:03:55,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 08:03:55,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5686.46 | bwd_inner_microstep: 5673.58 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.27
[2025-04-25 08:03:55,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5686.48 | bwd_inner: 5673.58 | bwd_allreduce: 12.86 | step: 19.27

  0%|          | 40/41250 [06:20<99:16:37,  8.67s/it]
                                                     
{'loss': 1.0517, 'grad_norm': 12.125739097595215, 'learning_rate': 1.2924071082390954e-06, 'epoch': 0.01}

  0%|          | 40/41250 [06:20<99:16:37,  8.67s/it][2025-04-25 08:04:04,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 08:04:04,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 6172.87 | bwd_inner_microstep: 5679.31 | bwd_allreduce_microstep: 493.49 | step_microstep: 18.70
[2025-04-25 08:04:04,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 6172.88 | bwd_inner: 5679.31 | bwd_allreduce: 493.52 | step: 18.70

  0%|          | 41/41250 [06:29<100:44:34,  8.80s/it]
                                                      
{'loss': 1.2772, 'grad_norm': 16.8228759765625, 'learning_rate': 1.3247172859450727e-06, 'epoch': 0.01}

  0%|          | 41/41250 [06:29<100:44:34,  8.80s/it][2025-04-25 08:04:13,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 08:04:13,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.64 | bwd_microstep: 5771.49 | bwd_inner_microstep: 5637.79 | bwd_allreduce_microstep: 133.65 | step_microstep: 19.09
[2025-04-25 08:04:13,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.64 | bwd: 5771.50 | bwd_inner: 5637.79 | bwd_allreduce: 133.67 | step: 19.10

  0%|          | 42/41250 [06:38<100:22:45,  8.77s/it]
                                                      
{'loss': 1.2497, 'grad_norm': 17.457456588745117, 'learning_rate': 1.3570274636510504e-06, 'epoch': 0.01}

  0%|          | 42/41250 [06:38<100:22:45,  8.77s/it][2025-04-25 08:04:21,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 08:04:21,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.48 | bwd_microstep: 5762.81 | bwd_inner_microstep: 5636.16 | bwd_allreduce_microstep: 126.59 | step_microstep: 19.53
[2025-04-25 08:04:21,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.48 | bwd: 5762.82 | bwd_inner: 5636.16 | bwd_allreduce: 126.61 | step: 19.53

  0%|          | 43/41250 [06:47<100:04:25,  8.74s/it]
                                                      
{'loss': 0.8133, 'grad_norm': 12.180554389953613, 'learning_rate': 1.3893376413570276e-06, 'epoch': 0.01}

  0%|          | 43/41250 [06:47<100:04:25,  8.74s/it][2025-04-25 08:04:30,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 08:04:30,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.21 | bwd_microstep: 5685.25 | bwd_inner_microstep: 5672.16 | bwd_allreduce_microstep: 13.04 | step_microstep: 19.14
[2025-04-25 08:04:30,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.21 | bwd: 5685.26 | bwd_inner: 5672.16 | bwd_allreduce: 13.06 | step: 19.14

  0%|          | 44/41250 [06:55<99:36:40,  8.70s/it] 
                                                     
{'loss': 1.0541, 'grad_norm': 17.766786575317383, 'learning_rate': 1.421647819063005e-06, 'epoch': 0.01}

  0%|          | 44/41250 [06:55<99:36:40,  8.70s/it][2025-04-25 08:04:39,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 08:04:39,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.26 | bwd_microstep: 5691.84 | bwd_inner_microstep: 5638.42 | bwd_allreduce_microstep: 53.37 | step_microstep: 19.01
[2025-04-25 08:04:39,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.26 | bwd: 5691.85 | bwd_inner: 5638.42 | bwd_allreduce: 53.39 | step: 19.01

  0%|          | 45/41250 [07:04<99:16:16,  8.67s/it]
                                                     
{'loss': 0.9278, 'grad_norm': 16.693262100219727, 'learning_rate': 1.4539579967689823e-06, 'epoch': 0.01}

  0%|          | 45/41250 [07:04<99:16:16,  8.67s/it][2025-04-25 08:04:47,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:04:47,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.85 | bwd_microstep: 5688.06 | bwd_inner_microstep: 5642.94 | bwd_allreduce_microstep: 45.09 | step_microstep: 18.58
[2025-04-25 08:04:47,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.85 | bwd: 5688.08 | bwd_inner: 5642.93 | bwd_allreduce: 45.10 | step: 18.59

  0%|          | 46/41250 [07:13<99:01:13,  8.65s/it]
                                                     
{'loss': 1.1462, 'grad_norm': 13.948007583618164, 'learning_rate': 1.4862681744749596e-06, 'epoch': 0.01}

  0%|          | 46/41250 [07:13<99:01:13,  8.65s/it][2025-04-25 08:04:56,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 08:04:56,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.24 | bwd_microstep: 5729.52 | bwd_inner_microstep: 5698.76 | bwd_allreduce_microstep: 30.72 | step_microstep: 19.23
[2025-04-25 08:04:56,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.24 | bwd: 5729.53 | bwd_inner: 5698.76 | bwd_allreduce: 30.74 | step: 19.24

  0%|          | 47/41250 [07:21<99:05:03,  8.66s/it]
                                                     
{'loss': 0.6516, 'grad_norm': 11.372976303100586, 'learning_rate': 1.5185783521809372e-06, 'epoch': 0.01}

  0%|          | 47/41250 [07:21<99:05:03,  8.66s/it][2025-04-25 08:05:05,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:05:05,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.09 | bwd_microstep: 6102.87 | bwd_inner_microstep: 5652.61 | bwd_allreduce_microstep: 450.22 | step_microstep: 18.31
[2025-04-25 08:05:05,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.09 | bwd: 6102.88 | bwd_inner: 5652.61 | bwd_allreduce: 450.23 | step: 18.31

  0%|          | 48/41250 [07:30<100:18:04,  8.76s/it]
                                                      
{'loss': 0.5583, 'grad_norm': 7.870979309082031, 'learning_rate': 1.5508885298869145e-06, 'epoch': 0.01}

  0%|          | 48/41250 [07:30<100:18:04,  8.76s/it][2025-04-25 08:05:14,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:05:14,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.07 | bwd_microstep: 5719.85 | bwd_inner_microstep: 5707.18 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.33
[2025-04-25 08:05:14,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.07 | bwd: 5719.86 | bwd_inner: 5707.18 | bwd_allreduce: 12.64 | step: 18.34

  0%|          | 49/41250 [07:39<99:56:39,  8.73s/it] 
                                                     
{'loss': 0.762, 'grad_norm': 12.458499908447266, 'learning_rate': 1.583198707592892e-06, 'epoch': 0.01}

  0%|          | 49/41250 [07:39<99:56:39,  8.73s/it][2025-04-25 08:05:22,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:05:22,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.42 | bwd_microstep: 5743.04 | bwd_inner_microstep: 5687.22 | bwd_allreduce_microstep: 55.76 | step_microstep: 17.81
[2025-04-25 08:05:22,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.42 | bwd: 5743.05 | bwd_inner: 5687.22 | bwd_allreduce: 55.78 | step: 17.81

  0%|          | 50/41250 [07:48<99:46:45,  8.72s/it]
                                                     
{'loss': 0.8173, 'grad_norm': 12.867875099182129, 'learning_rate': 1.6155088852988692e-06, 'epoch': 0.01}

  0%|          | 50/41250 [07:48<99:46:45,  8.72s/it][2025-04-25 08:05:31,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 08:05:31,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5713.17 | bwd_inner_microstep: 5647.28 | bwd_allreduce_microstep: 65.85 | step_microstep: 18.55
[2025-04-25 08:05:31,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5713.19 | bwd_inner: 5647.28 | bwd_allreduce: 65.87 | step: 18.56

  0%|          | 51/41250 [07:56<99:29:03,  8.69s/it]
                                                     
{'loss': 0.9351, 'grad_norm': 14.26898193359375, 'learning_rate': 1.6478190630048467e-06, 'epoch': 0.01}

  0%|          | 51/41250 [07:56<99:29:03,  8.69s/it][2025-04-25 08:05:40,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:05:40,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.37 | bwd_microstep: 5706.32 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.73
[2025-04-25 08:05:40,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5706.33 | bwd_inner: 5693.53 | bwd_allreduce: 12.76 | step: 18.73

  0%|          | 52/41250 [08:05<99:18:37,  8.68s/it]
                                                     
{'loss': 1.0894, 'grad_norm': 17.679706573486328, 'learning_rate': 1.6801292407108241e-06, 'epoch': 0.01}

  0%|          | 52/41250 [08:05<99:18:37,  8.68s/it][2025-04-25 08:05:48,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 08:05:48,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.14 | bwd_microstep: 5761.65 | bwd_inner_microstep: 5657.04 | bwd_allreduce_microstep: 104.56 | step_microstep: 19.55
[2025-04-25 08:05:48,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.14 | bwd: 5761.67 | bwd_inner: 5657.04 | bwd_allreduce: 104.58 | step: 19.55

  0%|          | 53/41250 [08:14<99:21:21,  8.68s/it]
                                                     
{'loss': 1.1213, 'grad_norm': 13.874848365783691, 'learning_rate': 1.7124394184168014e-06, 'epoch': 0.01}

  0%|          | 53/41250 [08:14<99:21:21,  8.68s/it][2025-04-25 08:05:57,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 08:05:57,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.14 | bwd_microstep: 5786.48 | bwd_inner_microstep: 5658.77 | bwd_allreduce_microstep: 127.67 | step_microstep: 19.10
[2025-04-25 08:05:57,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.14 | bwd: 5786.49 | bwd_inner: 5658.77 | bwd_allreduce: 127.68 | step: 19.11

  0%|          | 54/41250 [08:22<99:26:24,  8.69s/it]
                                                     
{'loss': 1.1512, 'grad_norm': 17.30350685119629, 'learning_rate': 1.744749596122779e-06, 'epoch': 0.01}

  0%|          | 54/41250 [08:22<99:26:24,  8.69s/it][2025-04-25 08:06:06,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 08:06:06,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.86 | bwd_microstep: 6373.33 | bwd_inner_microstep: 5664.91 | bwd_allreduce_microstep: 708.37 | step_microstep: 18.71
[2025-04-25 08:06:06,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.86 | bwd: 6373.34 | bwd_inner: 5664.91 | bwd_allreduce: 708.39 | step: 18.71

  0%|          | 55/41250 [08:32<101:31:31,  8.87s/it]
                                                      
{'loss': 0.7862, 'grad_norm': 11.710603713989258, 'learning_rate': 1.7770597738287563e-06, 'epoch': 0.01}

  0%|          | 55/41250 [08:32<101:31:31,  8.87s/it][2025-04-25 08:06:15,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:06:15,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.08 | bwd_microstep: 5773.51 | bwd_inner_microstep: 5701.19 | bwd_allreduce_microstep: 72.28 | step_microstep: 18.48
[2025-04-25 08:06:15,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.08 | bwd: 5773.52 | bwd_inner: 5701.19 | bwd_allreduce: 72.30 | step: 18.48

  0%|          | 56/41250 [08:40<100:58:03,  8.82s/it]
                                                      
{'loss': 0.6813, 'grad_norm': 9.257482528686523, 'learning_rate': 1.8093699515347335e-06, 'epoch': 0.01}

  0%|          | 56/41250 [08:40<100:58:03,  8.82s/it][2025-04-25 08:06:24,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.94 | optimizer_step: 1.06
[2025-04-25 08:06:24,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5753.91 | bwd_inner_microstep: 5697.99 | bwd_allreduce_microstep: 55.88 | step_microstep: 18.24
[2025-04-25 08:06:24,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5753.92 | bwd_inner: 5697.98 | bwd_allreduce: 55.89 | step: 18.25

  0%|          | 57/41250 [08:49<100:32:48,  8.79s/it]
                                                      
{'loss': 0.69, 'grad_norm': 11.029779434204102, 'learning_rate': 1.841680129240711e-06, 'epoch': 0.01}

  0%|          | 57/41250 [08:49<100:32:48,  8.79s/it][2025-04-25 08:06:32,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 08:06:32,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.91 | bwd_microstep: 5753.20 | bwd_inner_microstep: 5652.27 | bwd_allreduce_microstep: 100.90 | step_microstep: 18.58
[2025-04-25 08:06:32,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.91 | bwd: 5753.21 | bwd_inner: 5652.27 | bwd_allreduce: 100.91 | step: 18.58

  0%|          | 58/41250 [08:58<100:11:51,  8.76s/it]
                                                      
{'loss': 0.8989, 'grad_norm': 15.873826026916504, 'learning_rate': 1.8739903069466882e-06, 'epoch': 0.01}

  0%|          | 58/41250 [08:58<100:11:51,  8.76s/it][2025-04-25 08:06:41,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:06:41,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.12 | bwd_microstep: 5712.17 | bwd_inner_microstep: 5699.50 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.30
[2025-04-25 08:06:41,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.12 | bwd: 5712.18 | bwd_inner: 5699.50 | bwd_allreduce: 12.64 | step: 18.30

  0%|          | 59/41250 [09:06<99:49:25,  8.72s/it] 
                                                     
{'loss': 0.9131, 'grad_norm': 12.888813972473145, 'learning_rate': 1.906300484652666e-06, 'epoch': 0.01}

  0%|          | 59/41250 [09:06<99:49:25,  8.72s/it][2025-04-25 08:06:50,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:06:50,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5700.40 | bwd_inner_microstep: 5649.16 | bwd_allreduce_microstep: 51.19 | step_microstep: 18.42
[2025-04-25 08:06:50,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5700.41 | bwd_inner: 5649.16 | bwd_allreduce: 51.21 | step: 18.42

  0%|          | 60/41250 [09:15<99:30:17,  8.70s/it]
                                                     
{'loss': 0.6133, 'grad_norm': 10.374539375305176, 'learning_rate': 1.938610662358643e-06, 'epoch': 0.01}

  0%|          | 60/41250 [09:15<99:30:17,  8.70s/it][2025-04-25 08:06:58,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 1.23
[2025-04-25 08:06:58,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.14 | bwd_microstep: 5731.60 | bwd_inner_microstep: 5718.88 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.66
[2025-04-25 08:06:58,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.14 | bwd: 5731.61 | bwd_inner: 5718.88 | bwd_allreduce: 12.69 | step: 19.66

  0%|          | 61/41250 [09:24<99:27:18,  8.69s/it]
                                                     
{'loss': 0.9621, 'grad_norm': 17.616010665893555, 'learning_rate': 1.9709208400646204e-06, 'epoch': 0.01}

  0%|          | 61/41250 [09:24<99:27:18,  8.69s/it][2025-04-25 08:07:07,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.17
[2025-04-25 08:07:07,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5713.50 | bwd_inner_microstep: 5700.87 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.51
[2025-04-25 08:07:07,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5713.51 | bwd_inner: 5700.87 | bwd_allreduce: 12.60 | step: 19.52

  0%|          | 62/41250 [09:32<99:18:31,  8.68s/it]
                                                     
{'loss': 0.829, 'grad_norm': 11.63230037689209, 'learning_rate': 2.003231017770598e-06, 'epoch': 0.02}

  0%|          | 62/41250 [09:32<99:18:31,  8.68s/it][2025-04-25 08:07:16,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.12
[2025-04-25 08:07:16,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.31 | bwd_microstep: 5855.58 | bwd_inner_microstep: 5705.63 | bwd_allreduce_microstep: 149.90 | step_microstep: 19.70
[2025-04-25 08:07:16,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.31 | bwd: 5855.59 | bwd_inner: 5705.63 | bwd_allreduce: 149.92 | step: 19.70

  0%|          | 63/41250 [09:41<99:43:03,  8.72s/it]
                                                     
{'loss': 0.5741, 'grad_norm': 9.350339889526367, 'learning_rate': 2.0355411954765753e-06, 'epoch': 0.02}

  0%|          | 63/41250 [09:41<99:43:03,  8.72s/it][2025-04-25 08:07:24,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:07:24,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 5727.63 | bwd_inner_microstep: 5657.77 | bwd_allreduce_microstep: 69.82 | step_microstep: 18.97
[2025-04-25 08:07:24,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 5727.64 | bwd_inner: 5657.77 | bwd_allreduce: 69.83 | step: 18.97

  0%|          | 64/41250 [09:50<99:26:47,  8.69s/it]
                                                     
{'loss': 0.7968, 'grad_norm': 11.275052070617676, 'learning_rate': 2.0678513731825528e-06, 'epoch': 0.02}

  0%|          | 64/41250 [09:50<99:26:47,  8.69s/it][2025-04-25 08:07:33,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 08:07:33,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.23 | bwd_microstep: 5707.92 | bwd_inner_microstep: 5662.65 | bwd_allreduce_microstep: 45.22 | step_microstep: 19.39
[2025-04-25 08:07:33,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.23 | bwd: 5707.93 | bwd_inner: 5662.65 | bwd_allreduce: 45.24 | step: 19.39

  0%|          | 65/41250 [09:58<99:13:48,  8.67s/it]
                                                     
{'loss': 0.746, 'grad_norm': 13.440441131591797, 'learning_rate': 2.1001615508885302e-06, 'epoch': 0.02}

  0%|          | 65/41250 [09:58<99:13:48,  8.67s/it][2025-04-25 08:07:42,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:07:42,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.87 | bwd_microstep: 5725.17 | bwd_inner_microstep: 5671.33 | bwd_allreduce_microstep: 53.79 | step_microstep: 18.89
[2025-04-25 08:07:42,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.87 | bwd: 5725.18 | bwd_inner: 5671.33 | bwd_allreduce: 53.81 | step: 18.89

  0%|          | 66/41250 [10:07<99:06:41,  8.66s/it]
                                                     
{'loss': 0.4907, 'grad_norm': 10.163981437683105, 'learning_rate': 2.1324717285945073e-06, 'epoch': 0.02}

  0%|          | 66/41250 [10:07<99:06:41,  8.66s/it][2025-04-25 08:07:50,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.08
[2025-04-25 08:07:50,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.37 | bwd_microstep: 5774.89 | bwd_inner_microstep: 5706.00 | bwd_allreduce_microstep: 68.84 | step_microstep: 19.87
[2025-04-25 08:07:50,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.37 | bwd: 5774.91 | bwd_inner: 5706.00 | bwd_allreduce: 68.86 | step: 19.87

  0%|          | 67/41250 [10:16<99:16:19,  8.68s/it]
                                                     
{'loss': 0.4877, 'grad_norm': 9.41242504119873, 'learning_rate': 2.1647819063004847e-06, 'epoch': 0.02}

  0%|          | 67/41250 [10:16<99:16:19,  8.68s/it][2025-04-25 08:07:59,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 08:07:59,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5788.51 | bwd_inner_microstep: 5650.89 | bwd_allreduce_microstep: 137.57 | step_microstep: 19.72
[2025-04-25 08:07:59,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5788.52 | bwd_inner: 5650.89 | bwd_allreduce: 137.59 | step: 19.73

  0%|          | 68/41250 [10:24<99:21:06,  8.69s/it]
                                                     
{'loss': 0.6348, 'grad_norm': 9.42882251739502, 'learning_rate': 2.197092084006462e-06, 'epoch': 0.02}

  0%|          | 68/41250 [10:24<99:21:06,  8.69s/it][2025-04-25 08:08:08,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:08:08,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5766.80 | bwd_inner_microstep: 5642.49 | bwd_allreduce_microstep: 124.26 | step_microstep: 18.72
[2025-04-25 08:08:08,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5766.81 | bwd_inner: 5642.49 | bwd_allreduce: 124.28 | step: 18.72

  0%|          | 69/41250 [10:33<99:20:00,  8.68s/it]
                                                     
{'loss': 0.7292, 'grad_norm': 8.772915840148926, 'learning_rate': 2.2294022617124397e-06, 'epoch': 0.02}

  0%|          | 69/41250 [10:33<99:20:00,  8.68s/it][2025-04-25 08:08:16,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 08:08:16,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.93 | bwd_microstep: 5709.04 | bwd_inner_microstep: 5663.55 | bwd_allreduce_microstep: 45.44 | step_microstep: 19.04
[2025-04-25 08:08:16,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.93 | bwd: 5709.05 | bwd_inner: 5663.55 | bwd_allreduce: 45.46 | step: 19.04

  0%|          | 70/41250 [10:42<99:07:19,  8.67s/it]
                                                     
{'loss': 0.6063, 'grad_norm': 10.447288513183594, 'learning_rate': 2.261712439418417e-06, 'epoch': 0.02}

  0%|          | 70/41250 [10:42<99:07:19,  8.67s/it][2025-04-25 08:08:25,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:08:25,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.20 | bwd_microstep: 5715.64 | bwd_inner_microstep: 5645.39 | bwd_allreduce_microstep: 70.21 | step_microstep: 18.29
[2025-04-25 08:08:25,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.20 | bwd: 5715.66 | bwd_inner: 5645.39 | bwd_allreduce: 70.23 | step: 18.30

  0%|          | 71/41250 [10:50<98:59:37,  8.65s/it]
                                                     
{'loss': 0.7098, 'grad_norm': 9.446019172668457, 'learning_rate': 2.294022617124394e-06, 'epoch': 0.02}

  0%|          | 71/41250 [10:50<98:59:37,  8.65s/it][2025-04-25 08:08:34,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:08:34,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.15 | bwd_microstep: 5745.44 | bwd_inner_microstep: 5680.97 | bwd_allreduce_microstep: 64.42 | step_microstep: 18.34
[2025-04-25 08:08:34,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.15 | bwd: 5745.45 | bwd_inner: 5680.97 | bwd_allreduce: 64.44 | step: 18.34

  0%|          | 72/41250 [10:59<99:04:28,  8.66s/it]
                                                     
{'loss': 0.5431, 'grad_norm': 6.203885078430176, 'learning_rate': 2.3263327948303716e-06, 'epoch': 0.02}

  0%|          | 72/41250 [10:59<99:04:28,  8.66s/it][2025-04-25 08:08:42,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 08:08:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.50 | bwd_microstep: 5722.26 | bwd_inner_microstep: 5709.65 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.67
[2025-04-25 08:08:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.50 | bwd: 5722.27 | bwd_inner: 5709.65 | bwd_allreduce: 12.58 | step: 18.67

  0%|          | 73/41250 [11:08<99:06:54,  8.67s/it]
                                                     
{'loss': 0.4201, 'grad_norm': 5.685744285583496, 'learning_rate': 2.358642972536349e-06, 'epoch': 0.02}

  0%|          | 73/41250 [11:08<99:06:54,  8.67s/it][2025-04-25 08:08:51,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:08:51,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.76 | bwd_microstep: 5750.57 | bwd_inner_microstep: 5647.11 | bwd_allreduce_microstep: 103.42 | step_microstep: 18.38
[2025-04-25 08:08:51,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.76 | bwd: 5750.58 | bwd_inner: 5647.11 | bwd_allreduce: 103.44 | step: 18.38

  0%|          | 74/41250 [11:16<99:07:06,  8.67s/it]
                                                     
{'loss': 0.3997, 'grad_norm': 14.444547653198242, 'learning_rate': 2.3909531502423265e-06, 'epoch': 0.02}

  0%|          | 74/41250 [11:16<99:07:06,  8.67s/it][2025-04-25 08:09:00,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:09:00,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.43 | bwd_microstep: 5747.84 | bwd_inner_microstep: 5686.58 | bwd_allreduce_microstep: 61.22 | step_microstep: 17.92
[2025-04-25 08:09:00,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.43 | bwd: 5747.85 | bwd_inner: 5686.58 | bwd_allreduce: 61.24 | step: 17.92

  0%|          | 75/41250 [11:25<99:09:14,  8.67s/it]
                                                     
{'loss': 0.5989, 'grad_norm': 7.078418254852295, 'learning_rate': 2.423263327948304e-06, 'epoch': 0.02}

  0%|          | 75/41250 [11:25<99:09:14,  8.67s/it][2025-04-25 08:09:08,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:09:08,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5694.62 | bwd_inner_microstep: 5662.69 | bwd_allreduce_microstep: 31.89 | step_microstep: 18.44
[2025-04-25 08:09:08,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5694.63 | bwd_inner: 5662.69 | bwd_allreduce: 31.91 | step: 18.44

  0%|          | 76/41250 [11:34<98:56:35,  8.65s/it]
                                                     
{'loss': 0.4956, 'grad_norm': 7.2668304443359375, 'learning_rate': 2.455573505654281e-06, 'epoch': 0.02}

  0%|          | 76/41250 [11:34<98:56:35,  8.65s/it][2025-04-25 08:09:17,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 08:09:17,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.60 | bwd_microstep: 5729.95 | bwd_inner_microstep: 5660.45 | bwd_allreduce_microstep: 69.46 | step_microstep: 18.30
[2025-04-25 08:09:17,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.59 | bwd: 5729.96 | bwd_inner: 5660.45 | bwd_allreduce: 69.47 | step: 18.31

  0%|          | 77/41250 [11:42<98:58:00,  8.65s/it]
                                                     
{'loss': 0.4594, 'grad_norm': 5.886804103851318, 'learning_rate': 2.487883683360259e-06, 'epoch': 0.02}

  0%|          | 77/41250 [11:42<98:58:00,  8.65s/it][2025-04-25 08:09:26,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:09:26,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3267.76 | bwd_microstep: 5888.80 | bwd_inner_microstep: 5875.86 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.04
[2025-04-25 08:09:26,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3267.76 | bwd: 5888.81 | bwd_inner: 5875.86 | bwd_allreduce: 12.91 | step: 19.04

  0%|          | 78/41250 [11:52<100:58:35,  8.83s/it]
                                                      
{'loss': 0.3275, 'grad_norm': 4.388795852661133, 'learning_rate': 2.5201938610662364e-06, 'epoch': 0.02}

  0%|          | 78/41250 [11:52<100:58:35,  8.83s/it][2025-04-25 08:09:35,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:09:35,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 6024.87 | bwd_inner_microstep: 5697.82 | bwd_allreduce_microstep: 327.01 | step_microstep: 18.40
[2025-04-25 08:09:35,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 6024.89 | bwd_inner: 5697.82 | bwd_allreduce: 327.02 | step: 18.40

  0%|          | 79/41250 [12:01<101:25:01,  8.87s/it]
                                                      
{'loss': 0.3206, 'grad_norm': 5.982944011688232, 'learning_rate': 2.5525040387722134e-06, 'epoch': 0.02}

  0%|          | 79/41250 [12:01<101:25:01,  8.87s/it][2025-04-25 08:09:44,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:09:44,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.07 | bwd_microstep: 5729.86 | bwd_inner_microstep: 5687.93 | bwd_allreduce_microstep: 41.88 | step_microstep: 17.94
[2025-04-25 08:09:44,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.07 | bwd: 5729.87 | bwd_inner: 5687.93 | bwd_allreduce: 41.90 | step: 17.95

  0%|          | 80/41250 [12:09<100:44:43,  8.81s/it]
                                                      
{'loss': 0.4795, 'grad_norm': 11.101036071777344, 'learning_rate': 2.584814216478191e-06, 'epoch': 0.02}

  0%|          | 80/41250 [12:09<100:44:43,  8.81s/it][2025-04-25 08:09:53,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:09:53,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.65 | bwd_microstep: 5788.96 | bwd_inner_microstep: 5776.35 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.27
[2025-04-25 08:09:53,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.65 | bwd: 5788.98 | bwd_inner: 5776.35 | bwd_allreduce: 12.59 | step: 18.27

  0%|          | 81/41250 [12:18<100:36:03,  8.80s/it]
                                                      
{'loss': 0.6141, 'grad_norm': 3.9772861003875732, 'learning_rate': 2.6171243941841683e-06, 'epoch': 0.02}

  0%|          | 81/41250 [12:18<100:36:03,  8.80s/it][2025-04-25 08:10:01,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:10:01,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.83 | bwd_microstep: 5708.93 | bwd_inner_microstep: 5696.31 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.20
[2025-04-25 08:10:01,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.83 | bwd: 5708.94 | bwd_inner: 5696.31 | bwd_allreduce: 12.60 | step: 18.20

  0%|          | 82/41250 [12:27<100:05:10,  8.75s/it]
                                                      
{'loss': 0.3842, 'grad_norm': 3.477598190307617, 'learning_rate': 2.6494345718901454e-06, 'epoch': 0.02}

  0%|          | 82/41250 [12:27<100:05:10,  8.75s/it][2025-04-25 08:10:10,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:10:10,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.24 | bwd_microstep: 5744.50 | bwd_inner_microstep: 5656.65 | bwd_allreduce_microstep: 87.81 | step_microstep: 18.25
[2025-04-25 08:10:10,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.24 | bwd: 5744.52 | bwd_inner: 5656.65 | bwd_allreduce: 87.83 | step: 18.25

  0%|          | 83/41250 [12:35<99:46:46,  8.73s/it] 
                                                     
{'loss': 0.3521, 'grad_norm': 3.6246745586395264, 'learning_rate': 2.681744749596123e-06, 'epoch': 0.02}

  0%|          | 83/41250 [12:35<99:46:46,  8.73s/it][2025-04-25 08:10:19,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:10:19,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.08 | bwd_microstep: 5699.69 | bwd_inner_microstep: 5686.96 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.77
[2025-04-25 08:10:19,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.08 | bwd: 5699.70 | bwd_inner: 5686.96 | bwd_allreduce: 12.70 | step: 17.78

  0%|          | 84/41250 [12:44<99:29:29,  8.70s/it]
                                                     
{'loss': 0.2592, 'grad_norm': 4.202627658843994, 'learning_rate': 2.7140549273021007e-06, 'epoch': 0.02}

  0%|          | 84/41250 [12:44<99:29:29,  8.70s/it][2025-04-25 08:10:27,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 08:10:27,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5761.19 | bwd_inner_microstep: 5650.02 | bwd_allreduce_microstep: 111.13 | step_microstep: 18.16
[2025-04-25 08:10:27,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5761.21 | bwd_inner: 5650.02 | bwd_allreduce: 111.14 | step: 18.16

  0%|          | 85/41250 [12:53<99:23:51,  8.69s/it]
                                                     
{'loss': 0.2977, 'grad_norm': 3.0204412937164307, 'learning_rate': 2.7463651050080773e-06, 'epoch': 0.02}

  0%|          | 85/41250 [12:53<99:23:51,  8.69s/it][2025-04-25 08:10:36,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:10:36,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5710.16 | bwd_inner_microstep: 5698.01 | bwd_allreduce_microstep: 12.10 | step_microstep: 18.40
[2025-04-25 08:10:36,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5710.17 | bwd_inner: 5698.01 | bwd_allreduce: 12.12 | step: 18.40

  0%|          | 86/41250 [13:01<99:12:20,  8.68s/it]
                                                     
{'loss': 0.5778, 'grad_norm': 3.828176259994507, 'learning_rate': 2.7786752827140552e-06, 'epoch': 0.02}

  0%|          | 86/41250 [13:01<99:12:20,  8.68s/it][2025-04-25 08:10:45,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:10:45,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.51 | bwd_microstep: 5720.04 | bwd_inner_microstep: 5693.55 | bwd_allreduce_microstep: 26.45 | step_microstep: 18.10
[2025-04-25 08:10:45,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.51 | bwd: 5720.05 | bwd_inner: 5693.55 | bwd_allreduce: 26.47 | step: 18.11

  0%|          | 87/41250 [13:10<99:09:21,  8.67s/it]
                                                     
{'loss': 0.596, 'grad_norm': 4.867366790771484, 'learning_rate': 2.8109854604200327e-06, 'epoch': 0.02}

  0%|          | 87/41250 [13:10<99:09:21,  8.67s/it][2025-04-25 08:10:53,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:10:53,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.63 | bwd_microstep: 5678.45 | bwd_inner_microstep: 5649.39 | bwd_allreduce_microstep: 29.02 | step_microstep: 18.42
[2025-04-25 08:10:53,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.63 | bwd: 5678.47 | bwd_inner: 5649.39 | bwd_allreduce: 29.04 | step: 18.42

  0%|          | 88/41250 [13:19<98:53:36,  8.65s/it]
                                                     
{'loss': 0.3217, 'grad_norm': 3.9697470664978027, 'learning_rate': 2.84329563812601e-06, 'epoch': 0.02}

  0%|          | 88/41250 [13:19<98:53:36,  8.65s/it][2025-04-25 08:11:02,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:11:02,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.30 | bwd_microstep: 5759.05 | bwd_inner_microstep: 5652.36 | bwd_allreduce_microstep: 106.64 | step_microstep: 17.85
[2025-04-25 08:11:02,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.30 | bwd: 5759.06 | bwd_inner: 5652.36 | bwd_allreduce: 106.66 | step: 17.86

  0%|          | 89/41250 [13:27<98:57:13,  8.65s/it]
                                                     
{'loss': 0.6085, 'grad_norm': 13.929429054260254, 'learning_rate': 2.875605815831987e-06, 'epoch': 0.02}

  0%|          | 89/41250 [13:27<98:57:13,  8.65s/it][2025-04-25 08:11:11,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.92
[2025-04-25 08:11:11,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.97 | bwd_microstep: 5743.21 | bwd_inner_microstep: 5646.94 | bwd_allreduce_microstep: 96.23 | step_microstep: 18.41
[2025-04-25 08:11:11,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.97 | bwd: 5743.23 | bwd_inner: 5646.94 | bwd_allreduce: 96.25 | step: 18.42

  0%|          | 90/41250 [13:36<98:55:30,  8.65s/it]
                                                     
{'loss': 0.5499, 'grad_norm': 9.005071640014648, 'learning_rate': 2.9079159935379646e-06, 'epoch': 0.02}

  0%|          | 90/41250 [13:36<98:55:30,  8.65s/it][2025-04-25 08:11:19,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.96
[2025-04-25 08:11:19,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.08 | bwd_microstep: 5715.13 | bwd_inner_microstep: 5687.01 | bwd_allreduce_microstep: 28.07 | step_microstep: 18.30
[2025-04-25 08:11:19,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.08 | bwd: 5715.14 | bwd_inner: 5687.01 | bwd_allreduce: 28.09 | step: 18.30

  0%|          | 91/41250 [13:44<98:53:55,  8.65s/it]
                                                     
{'loss': 0.5267, 'grad_norm': 3.676863670349121, 'learning_rate': 2.940226171243942e-06, 'epoch': 0.02}

  0%|          | 91/41250 [13:44<98:53:55,  8.65s/it][2025-04-25 08:11:28,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:11:28,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.09 | bwd_microstep: 5705.22 | bwd_inner_microstep: 5652.60 | bwd_allreduce_microstep: 52.57 | step_microstep: 18.64
[2025-04-25 08:11:28,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.09 | bwd: 5705.23 | bwd_inner: 5652.60 | bwd_allreduce: 52.59 | step: 18.64

  0%|          | 92/41250 [13:53<98:46:22,  8.64s/it]
                                                     
{'loss': 0.2882, 'grad_norm': 2.634801149368286, 'learning_rate': 2.972536348949919e-06, 'epoch': 0.02}

  0%|          | 92/41250 [13:53<98:46:22,  8.64s/it][2025-04-25 08:11:36,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.13
[2025-04-25 08:11:36,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5723.73 | bwd_inner_microstep: 5711.02 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.20
[2025-04-25 08:11:36,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5723.74 | bwd_inner: 5711.02 | bwd_allreduce: 12.68 | step: 19.20

  0%|          | 93/41250 [14:02<98:51:11,  8.65s/it]
                                                     
{'loss': 0.4497, 'grad_norm': 9.868090629577637, 'learning_rate': 3.004846526655897e-06, 'epoch': 0.02}

  0%|          | 93/41250 [14:02<98:51:11,  8.65s/it][2025-04-25 08:11:45,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 08:11:45,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.37 | bwd_microstep: 5704.70 | bwd_inner_microstep: 5691.81 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.86
[2025-04-25 08:11:45,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.37 | bwd: 5704.71 | bwd_inner: 5691.81 | bwd_allreduce: 12.85 | step: 19.86

  0%|          | 94/41250 [14:10<98:48:37,  8.64s/it]
                                                     
{'loss': 0.2637, 'grad_norm': 2.5703532695770264, 'learning_rate': 3.0371567043618745e-06, 'epoch': 0.02}

  0%|          | 94/41250 [14:10<98:48:37,  8.64s/it][2025-04-25 08:11:54,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 08:11:54,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5695.09 | bwd_inner_microstep: 5682.41 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.38
[2025-04-25 08:11:54,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5695.10 | bwd_inner: 5682.41 | bwd_allreduce: 12.65 | step: 19.38

  0%|          | 95/41250 [14:19<98:45:37,  8.64s/it]
                                                     
{'loss': 0.233, 'grad_norm': 2.9262046813964844, 'learning_rate': 3.0694668820678515e-06, 'epoch': 0.02}

  0%|          | 95/41250 [14:19<98:45:37,  8.64s/it][2025-04-25 08:12:02,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 08:12:02,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.07 | bwd_microstep: 5741.04 | bwd_inner_microstep: 5658.96 | bwd_allreduce_microstep: 82.03 | step_microstep: 19.38
[2025-04-25 08:12:02,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.07 | bwd: 5741.06 | bwd_inner: 5658.96 | bwd_allreduce: 82.05 | step: 19.38

  0%|          | 96/41250 [14:28<98:49:22,  8.64s/it]
                                                     
{'loss': 0.5171, 'grad_norm': 3.808654308319092, 'learning_rate': 3.101777059773829e-06, 'epoch': 0.02}

  0%|          | 96/41250 [14:28<98:49:22,  8.64s/it][2025-04-25 08:12:11,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-25 08:12:11,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.89 | bwd_microstep: 5782.67 | bwd_inner_microstep: 5769.78 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.45
[2025-04-25 08:12:11,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.89 | bwd: 5782.68 | bwd_inner: 5769.78 | bwd_allreduce: 12.86 | step: 19.45

  0%|          | 97/41250 [14:36<99:14:04,  8.68s/it]
                                                     
{'loss': 0.3426, 'grad_norm': 2.5142178535461426, 'learning_rate': 3.1340872374798064e-06, 'epoch': 0.02}

  0%|          | 97/41250 [14:36<99:14:04,  8.68s/it][2025-04-25 08:12:20,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 08:12:20,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.70 | bwd_microstep: 5745.23 | bwd_inner_microstep: 5697.12 | bwd_allreduce_microstep: 48.06 | step_microstep: 18.98
[2025-04-25 08:12:20,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.70 | bwd: 5745.24 | bwd_inner: 5697.12 | bwd_allreduce: 48.08 | step: 18.99

  0%|          | 98/41250 [14:45<99:12:42,  8.68s/it]
                                                     
{'loss': 0.4276, 'grad_norm': 3.998213052749634, 'learning_rate': 3.166397415185784e-06, 'epoch': 0.02}

  0%|          | 98/41250 [14:45<99:12:42,  8.68s/it][2025-04-25 08:12:28,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:12:28,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.11 | bwd_microstep: 5765.21 | bwd_inner_microstep: 5649.83 | bwd_allreduce_microstep: 115.33 | step_microstep: 18.83
[2025-04-25 08:12:28,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.11 | bwd: 5765.22 | bwd_inner: 5649.83 | bwd_allreduce: 115.35 | step: 18.84

  0%|          | 99/41250 [14:54<99:11:15,  8.68s/it]
                                                     
{'loss': 0.602, 'grad_norm': 4.295295238494873, 'learning_rate': 3.198707592891761e-06, 'epoch': 0.02}

  0%|          | 99/41250 [14:54<99:11:15,  8.68s/it][2025-04-25 08:12:37,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 08:12:37,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5762.68 | bwd_inner_microstep: 5694.51 | bwd_allreduce_microstep: 68.11 | step_microstep: 19.22
[2025-04-25 08:12:37,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5762.69 | bwd_inner: 5694.51 | bwd_allreduce: 68.13 | step: 19.22

  0%|          | 100/41250 [15:02<99:14:47,  8.68s/it]
                                                      
{'loss': 0.2256, 'grad_norm': 1.7661924362182617, 'learning_rate': 3.2310177705977384e-06, 'epoch': 0.02}

  0%|          | 100/41250 [15:02<99:14:47,  8.68s/it][2025-04-25 08:12:46,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 08:12:46,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.75 | bwd_microstep: 5729.18 | bwd_inner_microstep: 5687.37 | bwd_allreduce_microstep: 41.76 | step_microstep: 19.26
[2025-04-25 08:12:46,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.75 | bwd: 5729.19 | bwd_inner: 5687.37 | bwd_allreduce: 41.78 | step: 19.26

  0%|          | 101/41250 [15:11<99:09:33,  8.68s/it]
                                                      
{'loss': 0.3672, 'grad_norm': 6.41527795791626, 'learning_rate': 3.2633279483037163e-06, 'epoch': 0.02}

  0%|          | 101/41250 [15:11<99:09:33,  8.68s/it][2025-04-25 08:12:54,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 08:12:54,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.17 | bwd_microstep: 5722.54 | bwd_inner_microstep: 5709.51 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.32
[2025-04-25 08:12:54,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.17 | bwd: 5722.56 | bwd_inner: 5709.51 | bwd_allreduce: 13.00 | step: 19.32

  0%|          | 102/41250 [15:20<99:06:21,  8.67s/it]
                                                      
{'loss': 0.35, 'grad_norm': 2.614314556121826, 'learning_rate': 3.2956381260096933e-06, 'epoch': 0.02}

  0%|          | 102/41250 [15:20<99:06:21,  8.67s/it][2025-04-25 08:13:03,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 08:13:03,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.90 | bwd_microstep: 5709.66 | bwd_inner_microstep: 5696.94 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.77
[2025-04-25 08:13:03,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.90 | bwd: 5709.67 | bwd_inner: 5696.94 | bwd_allreduce: 12.69 | step: 18.78

  0%|          | 103/41250 [15:28<98:59:37,  8.66s/it]
                                                      
{'loss': 0.216, 'grad_norm': 1.9662525653839111, 'learning_rate': 3.3279483037156708e-06, 'epoch': 0.02}

  0%|          | 103/41250 [15:28<98:59:37,  8.66s/it][2025-04-25 08:13:12,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 08:13:12,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5732.49 | bwd_inner_microstep: 5688.11 | bwd_allreduce_microstep: 44.32 | step_microstep: 19.39
[2025-04-25 08:13:12,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5732.50 | bwd_inner: 5688.11 | bwd_allreduce: 44.35 | step: 19.39

  0%|          | 104/41250 [15:37<99:00:10,  8.66s/it]
                                                      
{'loss': 0.4454, 'grad_norm': 4.090278625488281, 'learning_rate': 3.3602584814216482e-06, 'epoch': 0.03}

  0%|          | 104/41250 [15:37<99:00:10,  8.66s/it][2025-04-25 08:13:20,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:13:20,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.02 | bwd_microstep: 5699.33 | bwd_inner_microstep: 5686.55 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.73
[2025-04-25 08:13:20,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.02 | bwd: 5699.34 | bwd_inner: 5686.55 | bwd_allreduce: 12.76 | step: 18.73

  0%|          | 105/41250 [15:46<98:52:15,  8.65s/it]
                                                      
{'loss': 0.3512, 'grad_norm': 5.4300360679626465, 'learning_rate': 3.3925686591276253e-06, 'epoch': 0.03}

  0%|          | 105/41250 [15:46<98:52:15,  8.65s/it][2025-04-25 08:13:29,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.19 | optimizer_step: 1.13
[2025-04-25 08:13:29,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.56 | bwd_microstep: 5723.05 | bwd_inner_microstep: 5707.85 | bwd_allreduce_microstep: 15.12 | step_microstep: 20.97
[2025-04-25 08:13:29,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.56 | bwd: 5723.07 | bwd_inner: 5707.85 | bwd_allreduce: 15.16 | step: 20.96

  0%|          | 106/41250 [15:54<98:56:55,  8.66s/it]
                                                      
{'loss': 0.4106, 'grad_norm': 3.398583173751831, 'learning_rate': 3.4248788368336027e-06, 'epoch': 0.03}

  0%|          | 106/41250 [15:54<98:56:55,  8.66s/it][2025-04-25 08:13:38,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 08:13:38,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.21 | bwd_microstep: 5689.35 | bwd_inner_microstep: 5654.42 | bwd_allreduce_microstep: 34.89 | step_microstep: 18.90
[2025-04-25 08:13:38,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.21 | bwd: 5689.36 | bwd_inner: 5654.42 | bwd_allreduce: 34.90 | step: 18.91

  0%|          | 107/41250 [16:03<98:45:37,  8.64s/it]
                                                      
{'loss': 0.4134, 'grad_norm': 3.820371627807617, 'learning_rate': 3.45718901453958e-06, 'epoch': 0.03}

  0%|          | 107/41250 [16:03<98:45:37,  8.64s/it][2025-04-25 08:13:46,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.71 | optimizer_step: 1.21
[2025-04-25 08:13:46,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.69 | bwd_microstep: 5702.71 | bwd_inner_microstep: 5642.05 | bwd_allreduce_microstep: 60.61 | step_microstep: 20.22
[2025-04-25 08:13:46,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5702.72 | bwd_inner: 5642.05 | bwd_allreduce: 60.63 | step: 20.22

  0%|          | 108/41250 [16:12<98:39:19,  8.63s/it]
                                                      
{'loss': 0.2249, 'grad_norm': 1.9855748414993286, 'learning_rate': 3.489499192245558e-06, 'epoch': 0.03}

  0%|          | 108/41250 [16:12<98:39:19,  8.63s/it][2025-04-25 08:13:55,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 08:13:55,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5709.04 | bwd_inner_microstep: 5696.22 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.22
[2025-04-25 08:13:55,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5709.06 | bwd_inner: 5696.22 | bwd_allreduce: 12.80 | step: 19.22

  0%|          | 109/41250 [16:20<98:43:39,  8.64s/it]
                                                      
{'loss': 0.4163, 'grad_norm': 3.9239253997802734, 'learning_rate': 3.5218093699515347e-06, 'epoch': 0.03}

  0%|          | 109/41250 [16:20<98:43:39,  8.64s/it][2025-04-25 08:14:04,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:14:04,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.36 | bwd_microstep: 5765.79 | bwd_inner_microstep: 5690.68 | bwd_allreduce_microstep: 75.07 | step_microstep: 18.85
[2025-04-25 08:14:04,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.36 | bwd: 5765.80 | bwd_inner: 5690.68 | bwd_allreduce: 75.08 | step: 18.86

  0%|          | 110/41250 [16:29<98:56:21,  8.66s/it]
                                                      
{'loss': 0.2966, 'grad_norm': 2.118825912475586, 'learning_rate': 3.5541195476575126e-06, 'epoch': 0.03}

  0%|          | 110/41250 [16:29<98:56:21,  8.66s/it][2025-04-25 08:14:12,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.18 | optimizer_step: 1.13
[2025-04-25 08:14:12,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.15 | bwd_microstep: 5750.42 | bwd_inner_microstep: 5709.92 | bwd_allreduce_microstep: 40.44 | step_microstep: 20.18
[2025-04-25 08:14:12,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.15 | bwd: 5750.44 | bwd_inner: 5709.92 | bwd_allreduce: 40.47 | step: 20.17

  0%|          | 111/41250 [16:38<99:03:30,  8.67s/it]
                                                      
{'loss': 0.4535, 'grad_norm': 3.4304542541503906, 'learning_rate': 3.58642972536349e-06, 'epoch': 0.03}

  0%|          | 111/41250 [16:38<99:03:30,  8.67s/it][2025-04-25 08:14:21,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 08:14:21,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5772.92 | bwd_inner_microstep: 5666.62 | bwd_allreduce_microstep: 106.26 | step_microstep: 18.93
[2025-04-25 08:14:21,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5772.93 | bwd_inner: 5666.62 | bwd_allreduce: 106.28 | step: 18.93

  0%|          | 112/41250 [16:46<99:07:32,  8.67s/it]
                                                      
{'loss': 0.2288, 'grad_norm': 2.8234429359436035, 'learning_rate': 3.618739903069467e-06, 'epoch': 0.03}

  0%|          | 112/41250 [16:46<99:07:32,  8.67s/it][2025-04-25 08:14:30,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.17 | optimizer_step: 1.15
[2025-04-25 08:14:30,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.65 | bwd_microstep: 5706.77 | bwd_inner_microstep: 5665.15 | bwd_allreduce_microstep: 41.56 | step_microstep: 19.68
[2025-04-25 08:14:30,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.65 | bwd: 5706.78 | bwd_inner: 5665.15 | bwd_allreduce: 41.58 | step: 19.68

  0%|          | 113/41250 [16:55<98:58:32,  8.66s/it]
                                                      
{'loss': 0.3653, 'grad_norm': 3.044584274291992, 'learning_rate': 3.6510500807754445e-06, 'epoch': 0.03}

  0%|          | 113/41250 [16:55<98:58:32,  8.66s/it][2025-04-25 08:14:38,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 08:14:38,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.20 | bwd_microstep: 5763.08 | bwd_inner_microstep: 5701.33 | bwd_allreduce_microstep: 61.70 | step_microstep: 18.97
[2025-04-25 08:14:38,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.20 | bwd: 5763.09 | bwd_inner: 5701.33 | bwd_allreduce: 61.72 | step: 18.97

  0%|          | 114/41250 [17:04<99:05:57,  8.67s/it]
                                                      
{'loss': 0.633, 'grad_norm': 7.459475994110107, 'learning_rate': 3.683360258481422e-06, 'epoch': 0.03}

  0%|          | 114/41250 [17:04<99:05:57,  8.67s/it][2025-04-25 08:14:48,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 08:14:48,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.77 | bwd_microstep: 6255.88 | bwd_inner_microstep: 5662.96 | bwd_allreduce_microstep: 592.87 | step_microstep: 18.77
[2025-04-25 08:14:48,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.77 | bwd: 6255.89 | bwd_inner: 5662.96 | bwd_allreduce: 592.89 | step: 18.77

  0%|          | 115/41250 [17:13<100:50:10,  8.82s/it]
                                                       
{'loss': 0.3047, 'grad_norm': 6.511588096618652, 'learning_rate': 3.715670436187399e-06, 'epoch': 0.03}

  0%|          | 115/41250 [17:13<100:50:10,  8.82s/it][2025-04-25 08:14:56,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 08:14:56,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.01 | bwd_microstep: 5773.51 | bwd_inner_microstep: 5709.65 | bwd_allreduce_microstep: 63.82 | step_microstep: 18.96
[2025-04-25 08:14:56,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.01 | bwd: 5773.52 | bwd_inner: 5709.65 | bwd_allreduce: 63.83 | step: 18.96

  0%|          | 116/41250 [17:22<100:27:56,  8.79s/it]
                                                       
{'loss': 0.2241, 'grad_norm': 2.181046485900879, 'learning_rate': 3.7479806138933765e-06, 'epoch': 0.03}

  0%|          | 116/41250 [17:22<100:27:56,  8.79s/it][2025-04-25 08:15:05,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:15:05,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.43 | bwd_microstep: 5773.78 | bwd_inner_microstep: 5715.62 | bwd_allreduce_microstep: 58.12 | step_microstep: 18.42
[2025-04-25 08:15:05,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.43 | bwd: 5773.80 | bwd_inner: 5715.62 | bwd_allreduce: 58.14 | step: 18.43

  0%|          | 117/41250 [17:30<100:10:17,  8.77s/it]
                                                       
{'loss': 0.246, 'grad_norm': 2.035851240158081, 'learning_rate': 3.7802907915993544e-06, 'epoch': 0.03}

  0%|          | 117/41250 [17:30<100:10:17,  8.77s/it][2025-04-25 08:15:14,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:15:14,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5731.45 | bwd_inner_microstep: 5662.28 | bwd_allreduce_microstep: 69.13 | step_microstep: 18.40
[2025-04-25 08:15:14,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5731.46 | bwd_inner: 5662.28 | bwd_allreduce: 69.14 | step: 18.40

  0%|          | 118/41250 [17:39<99:43:50,  8.73s/it] 
                                                      
{'loss': 0.4445, 'grad_norm': 3.3834128379821777, 'learning_rate': 3.812600969305332e-06, 'epoch': 0.03}

  0%|          | 118/41250 [17:39<99:43:50,  8.73s/it][2025-04-25 08:15:22,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:15:22,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.36 | bwd_microstep: 5768.04 | bwd_inner_microstep: 5702.31 | bwd_allreduce_microstep: 65.68 | step_microstep: 18.42
[2025-04-25 08:15:22,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.36 | bwd: 5768.05 | bwd_inner: 5702.31 | bwd_allreduce: 65.70 | step: 18.42

  0%|          | 119/41250 [17:48<99:42:03,  8.73s/it]
                                                      
{'loss': 0.2015, 'grad_norm': 1.606475830078125, 'learning_rate': 3.844911147011309e-06, 'epoch': 0.03}

  0%|          | 119/41250 [17:48<99:42:03,  8.73s/it][2025-04-25 08:15:31,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 08:15:31,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.08 | bwd_microstep: 5741.48 | bwd_inner_microstep: 5703.09 | bwd_allreduce_microstep: 38.34 | step_microstep: 19.28
[2025-04-25 08:15:31,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.08 | bwd: 5741.49 | bwd_inner: 5703.09 | bwd_allreduce: 38.37 | step: 19.28

  0%|          | 120/41250 [17:56<99:33:13,  8.71s/it]
                                                      
{'loss': 0.3626, 'grad_norm': 3.7633020877838135, 'learning_rate': 3.877221324717286e-06, 'epoch': 0.03}

  0%|          | 120/41250 [17:56<99:33:13,  8.71s/it][2025-04-25 08:15:40,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:15:40,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.07 | bwd_microstep: 5716.76 | bwd_inner_microstep: 5699.10 | bwd_allreduce_microstep: 17.61 | step_microstep: 18.54
[2025-04-25 08:15:40,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.07 | bwd: 5716.77 | bwd_inner: 5699.10 | bwd_allreduce: 17.63 | step: 18.55

  0%|          | 121/41250 [18:05<99:21:30,  8.70s/it]
                                                      
{'loss': 0.3389, 'grad_norm': 7.4238200187683105, 'learning_rate': 3.909531502423264e-06, 'epoch': 0.03}

  0%|          | 121/41250 [18:05<99:21:30,  8.70s/it][2025-04-25 08:15:48,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 08:15:48,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.52 | bwd_microstep: 5731.02 | bwd_inner_microstep: 5718.35 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.15
[2025-04-25 08:15:48,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.52 | bwd: 5731.03 | bwd_inner: 5718.35 | bwd_allreduce: 12.64 | step: 19.15

  0%|          | 122/41250 [18:14<99:16:15,  8.69s/it]
                                                      
{'loss': 0.6462, 'grad_norm': 3.5198416709899902, 'learning_rate': 3.941841680129241e-06, 'epoch': 0.03}

  0%|          | 122/41250 [18:14<99:16:15,  8.69s/it][2025-04-25 08:15:57,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:15:57,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.67 | bwd_microstep: 5883.26 | bwd_inner_microstep: 5716.51 | bwd_allreduce_microstep: 166.71 | step_microstep: 18.68
[2025-04-25 08:15:57,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.67 | bwd: 5883.27 | bwd_inner: 5716.51 | bwd_allreduce: 166.73 | step: 18.68

  0%|          | 123/41250 [18:23<99:44:26,  8.73s/it]
                                                      
{'loss': 0.353, 'grad_norm': 3.218257427215576, 'learning_rate': 3.974151857835219e-06, 'epoch': 0.03}

  0%|          | 123/41250 [18:23<99:44:26,  8.73s/it][2025-04-25 08:16:06,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-25 08:16:06,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.15 | bwd_microstep: 5725.72 | bwd_inner_microstep: 5712.42 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.22
[2025-04-25 08:16:06,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.15 | bwd: 5725.73 | bwd_inner: 5712.42 | bwd_allreduce: 13.27 | step: 19.22

  0%|          | 124/41250 [18:31<99:32:41,  8.71s/it]
                                                      
{'loss': 0.4553, 'grad_norm': 2.851754665374756, 'learning_rate': 4.006462035541196e-06, 'epoch': 0.03}

  0%|          | 124/41250 [18:31<99:32:41,  8.71s/it][2025-04-25 08:16:15,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 08:16:15,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5776.04 | bwd_inner_microstep: 5671.35 | bwd_allreduce_microstep: 104.64 | step_microstep: 19.66
[2025-04-25 08:16:15,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5776.06 | bwd_inner: 5671.35 | bwd_allreduce: 104.66 | step: 19.66

  0%|          | 125/41250 [18:40<99:32:39,  8.71s/it]
                                                      
{'loss': 0.2758, 'grad_norm': 1.82973051071167, 'learning_rate': 4.038772213247173e-06, 'epoch': 0.03}

  0%|          | 125/41250 [18:40<99:32:39,  8.71s/it][2025-04-25 08:16:23,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:16:23,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.46 | bwd_microstep: 5784.04 | bwd_inner_microstep: 5771.42 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.67
[2025-04-25 08:16:23,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.46 | bwd: 5784.05 | bwd_inner: 5771.42 | bwd_allreduce: 12.60 | step: 18.67

  0%|          | 126/41250 [18:49<99:40:44,  8.73s/it]
                                                      
{'loss': 0.285, 'grad_norm': 3.2290825843811035, 'learning_rate': 4.071082390953151e-06, 'epoch': 0.03}

  0%|          | 126/41250 [18:49<99:40:44,  8.73s/it][2025-04-25 08:16:32,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.25
[2025-04-25 08:16:32,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5760.92 | bwd_inner_microstep: 5697.16 | bwd_allreduce_microstep: 63.71 | step_microstep: 19.74
[2025-04-25 08:16:32,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5760.93 | bwd_inner: 5697.16 | bwd_allreduce: 63.73 | step: 19.74

  0%|          | 127/41250 [18:57<99:34:36,  8.72s/it]
                                                      
{'loss': 0.2949, 'grad_norm': 2.609118700027466, 'learning_rate': 4.103392568659128e-06, 'epoch': 0.03}

  0%|          | 127/41250 [18:57<99:34:36,  8.72s/it][2025-04-25 08:16:41,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.34 | optimizer_step: 1.08
[2025-04-25 08:16:41,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.54 | bwd_microstep: 5722.05 | bwd_inner_microstep: 5671.04 | bwd_allreduce_microstep: 50.94 | step_microstep: 21.44
[2025-04-25 08:16:41,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.54 | bwd: 5722.07 | bwd_inner: 5671.04 | bwd_allreduce: 50.97 | step: 21.44

  0%|          | 128/41250 [19:06<99:18:47,  8.69s/it]
                                                      
{'loss': 0.3605, 'grad_norm': 2.8143439292907715, 'learning_rate': 4.1357027463651056e-06, 'epoch': 0.03}

  0%|          | 128/41250 [19:06<99:18:47,  8.69s/it][2025-04-25 08:16:49,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:16:49,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.66 | bwd_microstep: 5719.49 | bwd_inner_microstep: 5706.74 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.79
[2025-04-25 08:16:49,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.66 | bwd: 5719.50 | bwd_inner: 5706.74 | bwd_allreduce: 12.72 | step: 18.79

  0%|          | 129/41250 [19:15<99:11:29,  8.68s/it]
                                                      
{'loss': 0.39, 'grad_norm': 3.6089932918548584, 'learning_rate': 4.168012924071083e-06, 'epoch': 0.03}

  0%|          | 129/41250 [19:15<99:11:29,  8.68s/it][2025-04-25 08:16:58,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-25 08:16:58,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.42 | bwd_microstep: 5780.91 | bwd_inner_microstep: 5661.73 | bwd_allreduce_microstep: 119.14 | step_microstep: 18.96
[2025-04-25 08:16:58,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.42 | bwd: 5780.92 | bwd_inner: 5661.73 | bwd_allreduce: 119.16 | step: 18.96

  0%|          | 130/41250 [19:23<99:13:14,  8.69s/it]
                                                      
{'loss': 0.337, 'grad_norm': 2.725152015686035, 'learning_rate': 4.2003231017770605e-06, 'epoch': 0.03}

  0%|          | 130/41250 [19:23<99:13:14,  8.69s/it][2025-04-25 08:17:07,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:17:07,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.57 | bwd_microstep: 5744.69 | bwd_inner_microstep: 5694.08 | bwd_allreduce_microstep: 50.57 | step_microstep: 18.67
[2025-04-25 08:17:07,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.57 | bwd: 5744.71 | bwd_inner: 5694.08 | bwd_allreduce: 50.59 | step: 18.68

  0%|          | 131/41250 [19:32<99:12:03,  8.69s/it]
                                                      
{'loss': 0.2914, 'grad_norm': 2.5961499214172363, 'learning_rate': 4.2326332794830375e-06, 'epoch': 0.03}

  0%|          | 131/41250 [19:32<99:12:03,  8.69s/it][2025-04-25 08:17:15,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 08:17:15,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.43 | bwd_microstep: 5797.11 | bwd_inner_microstep: 5784.40 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.81
[2025-04-25 08:17:15,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.43 | bwd: 5797.12 | bwd_inner: 5784.40 | bwd_allreduce: 12.68 | step: 18.81

  0%|          | 132/41250 [19:41<99:30:18,  8.71s/it]
                                                      
{'loss': 0.394, 'grad_norm': 2.5039634704589844, 'learning_rate': 4.2649434571890146e-06, 'epoch': 0.03}

  0%|          | 132/41250 [19:41<99:30:18,  8.71s/it][2025-04-25 08:17:24,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 08:17:24,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5739.04 | bwd_inner_microstep: 5689.44 | bwd_allreduce_microstep: 49.55 | step_microstep: 18.69
[2025-04-25 08:17:24,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5739.06 | bwd_inner: 5689.44 | bwd_allreduce: 49.57 | step: 18.70

  0%|          | 133/41250 [19:49<99:21:29,  8.70s/it]
                                                      
{'loss': 0.4447, 'grad_norm': 3.1140286922454834, 'learning_rate': 4.2972536348949925e-06, 'epoch': 0.03}

  0%|          | 133/41250 [19:49<99:21:29,  8.70s/it][2025-04-25 08:17:33,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:17:33,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.54 | bwd_microstep: 5790.25 | bwd_inner_microstep: 5777.64 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.51
[2025-04-25 08:17:33,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.54 | bwd: 5790.26 | bwd_inner: 5777.64 | bwd_allreduce: 12.58 | step: 18.51

  0%|          | 134/41250 [19:58<99:35:01,  8.72s/it]
                                                      
{'loss': 0.2055, 'grad_norm': 1.407902717590332, 'learning_rate': 4.3295638126009695e-06, 'epoch': 0.03}

  0%|          | 134/41250 [19:58<99:35:01,  8.72s/it][2025-04-25 08:17:42,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:17:42,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.68 | bwd_microstep: 5762.97 | bwd_inner_microstep: 5662.78 | bwd_allreduce_microstep: 100.14 | step_microstep: 18.29
[2025-04-25 08:17:42,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.68 | bwd: 5762.98 | bwd_inner: 5662.78 | bwd_allreduce: 100.15 | step: 18.30

  0%|          | 135/41250 [20:07<99:26:30,  8.71s/it]
                                                      
{'loss': 0.3386, 'grad_norm': 2.1194591522216797, 'learning_rate': 4.3618739903069465e-06, 'epoch': 0.03}

  0%|          | 135/41250 [20:07<99:26:30,  8.71s/it][2025-04-25 08:17:50,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:17:50,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.42 | bwd_microstep: 5742.64 | bwd_inner_microstep: 5688.92 | bwd_allreduce_microstep: 53.68 | step_microstep: 17.89
[2025-04-25 08:17:50,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.42 | bwd: 5742.65 | bwd_inner: 5688.92 | bwd_allreduce: 53.69 | step: 17.89

  0%|          | 136/41250 [20:16<99:18:19,  8.70s/it]
                                                      
{'loss': 0.2429, 'grad_norm': 2.081285238265991, 'learning_rate': 4.394184168012924e-06, 'epoch': 0.03}

  0%|          | 136/41250 [20:16<99:18:19,  8.70s/it][2025-04-25 08:17:59,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.25 | optimizer_step: 1.01
[2025-04-25 08:17:59,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5776.86 | bwd_inner_microstep: 5647.23 | bwd_allreduce_microstep: 129.57 | step_microstep: 19.84
[2025-04-25 08:17:59,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5776.87 | bwd_inner: 5647.23 | bwd_allreduce: 129.60 | step: 19.84

  0%|          | 137/41250 [20:24<99:17:41,  8.69s/it]
                                                      
{'loss': 0.522, 'grad_norm': 2.9264843463897705, 'learning_rate': 4.426494345718902e-06, 'epoch': 0.03}

  0%|          | 137/41250 [20:24<99:17:41,  8.69s/it][2025-04-25 08:18:08,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.20 | optimizer_step: 0.93
[2025-04-25 08:18:08,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.76 | bwd_microstep: 5755.16 | bwd_inner_microstep: 5696.78 | bwd_allreduce_microstep: 58.33 | step_microstep: 19.36
[2025-04-25 08:18:08,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.76 | bwd: 5755.17 | bwd_inner: 5696.78 | bwd_allreduce: 58.35 | step: 19.36

  0%|          | 138/41250 [20:33<99:16:25,  8.69s/it]
                                                      
{'loss': 0.2275, 'grad_norm': 1.9566043615341187, 'learning_rate': 4.458804523424879e-06, 'epoch': 0.03}

  0%|          | 138/41250 [20:33<99:16:25,  8.69s/it][2025-04-25 08:18:16,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:18:16,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.28 | bwd_microstep: 5775.18 | bwd_inner_microstep: 5762.54 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.30
[2025-04-25 08:18:16,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.28 | bwd: 5775.19 | bwd_inner: 5762.54 | bwd_allreduce: 12.62 | step: 18.30

  0%|          | 139/41250 [20:42<99:27:47,  8.71s/it]
                                                      
{'loss': 0.2264, 'grad_norm': 2.5214128494262695, 'learning_rate': 4.491114701130856e-06, 'epoch': 0.03}

  0%|          | 139/41250 [20:42<99:27:47,  8.71s/it][2025-04-25 08:18:25,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:18:25,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.07 | bwd_microstep: 5760.80 | bwd_inner_microstep: 5654.95 | bwd_allreduce_microstep: 105.81 | step_microstep: 18.25
[2025-04-25 08:18:25,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.07 | bwd: 5760.81 | bwd_inner: 5654.95 | bwd_allreduce: 105.83 | step: 18.25

  0%|          | 140/41250 [20:50<99:20:50,  8.70s/it]
                                                      
{'loss': 0.1599, 'grad_norm': 1.2807502746582031, 'learning_rate': 4.523424878836834e-06, 'epoch': 0.03}

  0%|          | 140/41250 [20:50<99:20:50,  8.70s/it][2025-04-25 08:18:34,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:18:34,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.78 | bwd_microstep: 5707.34 | bwd_inner_microstep: 5647.55 | bwd_allreduce_microstep: 59.75 | step_microstep: 18.07
[2025-04-25 08:18:34,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.78 | bwd: 5707.35 | bwd_inner: 5647.55 | bwd_allreduce: 59.76 | step: 18.07

  0%|          | 141/41250 [20:59<99:03:34,  8.67s/it]
                                                      
{'loss': 0.2384, 'grad_norm': 2.9901115894317627, 'learning_rate': 4.555735056542811e-06, 'epoch': 0.03}

  0%|          | 141/41250 [20:59<99:03:34,  8.67s/it][2025-04-25 08:18:42,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:18:42,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5709.80 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.55
[2025-04-25 08:18:42,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5709.81 | bwd_inner: 5696.95 | bwd_allreduce: 12.83 | step: 18.55

  0%|          | 142/41250 [21:08<98:57:11,  8.67s/it]
                                                      
{'loss': 0.3086, 'grad_norm': 3.50030779838562, 'learning_rate': 4.588045234248788e-06, 'epoch': 0.03}

  0%|          | 142/41250 [21:08<98:57:11,  8.67s/it][2025-04-25 08:18:51,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:18:51,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.40 | bwd_microstep: 5744.77 | bwd_inner_microstep: 5685.16 | bwd_allreduce_microstep: 59.56 | step_microstep: 18.46
[2025-04-25 08:18:51,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.40 | bwd: 5744.78 | bwd_inner: 5685.16 | bwd_allreduce: 59.58 | step: 18.46

  0%|          | 143/41250 [21:16<98:58:42,  8.67s/it]
                                                      
{'loss': 0.3459, 'grad_norm': 2.418071985244751, 'learning_rate': 4.620355411954766e-06, 'epoch': 0.03}

  0%|          | 143/41250 [21:16<98:58:42,  8.67s/it][2025-04-25 08:19:00,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 08:19:00,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5749.28 | bwd_inner_microstep: 5652.32 | bwd_allreduce_microstep: 96.90 | step_microstep: 19.62
[2025-04-25 08:19:00,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5749.29 | bwd_inner: 5652.32 | bwd_allreduce: 96.92 | step: 19.63

  0%|          | 144/41250 [21:25<98:58:29,  8.67s/it]
                                                      
{'loss': 0.3807, 'grad_norm': 2.9553723335266113, 'learning_rate': 4.652665589660743e-06, 'epoch': 0.03}

  0%|          | 144/41250 [21:25<98:58:29,  8.67s/it][2025-04-25 08:19:08,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:19:08,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5749.14 | bwd_inner_microstep: 5700.88 | bwd_allreduce_microstep: 48.22 | step_microstep: 18.58
[2025-04-25 08:19:08,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5749.15 | bwd_inner: 5700.88 | bwd_allreduce: 48.23 | step: 18.58

  0%|          | 145/41250 [21:34<99:00:52,  8.67s/it]
                                                      
{'loss': 0.212, 'grad_norm': 3.5518269538879395, 'learning_rate': 4.68497576736672e-06, 'epoch': 0.04}

  0%|          | 145/41250 [21:34<99:00:52,  8.67s/it][2025-04-25 08:19:17,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.07 | optimizer_step: 1.05
[2025-04-25 08:19:17,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.78 | bwd_microstep: 5746.95 | bwd_inner_microstep: 5696.36 | bwd_allreduce_microstep: 50.54 | step_microstep: 19.59
[2025-04-25 08:19:17,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.78 | bwd: 5746.97 | bwd_inner: 5696.36 | bwd_allreduce: 50.56 | step: 19.60

  0%|          | 146/41250 [21:42<99:04:17,  8.68s/it]
                                                      
{'loss': 0.2484, 'grad_norm': 3.5950777530670166, 'learning_rate': 4.717285945072698e-06, 'epoch': 0.04}

  0%|          | 146/41250 [21:42<99:04:17,  8.68s/it][2025-04-25 08:19:26,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:19:26,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.82 | bwd_microstep: 5754.76 | bwd_inner_microstep: 5681.31 | bwd_allreduce_microstep: 73.41 | step_microstep: 18.50
[2025-04-25 08:19:26,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.82 | bwd: 5754.77 | bwd_inner: 5681.31 | bwd_allreduce: 73.42 | step: 18.51

  0%|          | 147/41250 [21:51<99:05:01,  8.68s/it]
                                                      
{'loss': 0.2864, 'grad_norm': 1.9516124725341797, 'learning_rate': 4.749596122778676e-06, 'epoch': 0.04}

  0%|          | 147/41250 [21:51<99:05:01,  8.68s/it][2025-04-25 08:19:34,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 08:19:34,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5708.43 | bwd_inner_microstep: 5695.76 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.44
[2025-04-25 08:19:34,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5708.45 | bwd_inner: 5695.76 | bwd_allreduce: 12.64 | step: 18.45

  0%|          | 148/41250 [22:00<98:57:04,  8.67s/it]
                                                      
{'loss': 0.2623, 'grad_norm': 1.9669657945632935, 'learning_rate': 4.781906300484653e-06, 'epoch': 0.04}

  0%|          | 148/41250 [22:00<98:57:04,  8.67s/it][2025-04-25 08:19:43,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:19:43,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.01 | bwd_microstep: 5685.51 | bwd_inner_microstep: 5648.81 | bwd_allreduce_microstep: 36.66 | step_microstep: 18.59
[2025-04-25 08:19:43,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.01 | bwd: 5685.53 | bwd_inner: 5648.81 | bwd_allreduce: 36.67 | step: 18.60

  0%|          | 149/41250 [22:08<98:40:22,  8.64s/it]
                                                      
{'loss': 0.232, 'grad_norm': 1.4762310981750488, 'learning_rate': 4.81421647819063e-06, 'epoch': 0.04}

  0%|          | 149/41250 [22:08<98:40:22,  8.64s/it][2025-04-25 08:19:52,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 08:19:52,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.92 | bwd_microstep: 5692.98 | bwd_inner_microstep: 5680.07 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.31
[2025-04-25 08:19:52,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.92 | bwd: 5692.99 | bwd_inner: 5680.07 | bwd_allreduce: 12.87 | step: 19.31

  0%|          | 150/41250 [22:17<98:38:39,  8.64s/it]
                                                      
{'loss': 0.5966, 'grad_norm': 5.268546104431152, 'learning_rate': 4.846526655896608e-06, 'epoch': 0.04}

  0%|          | 150/41250 [22:17<98:38:39,  8.64s/it][2025-04-25 08:20:00,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:20:00,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.80 | bwd_microstep: 5732.43 | bwd_inner_microstep: 5657.67 | bwd_allreduce_microstep: 74.72 | step_microstep: 18.75
[2025-04-25 08:20:00,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.80 | bwd: 5732.44 | bwd_inner: 5657.67 | bwd_allreduce: 74.74 | step: 18.76

  0%|          | 151/41250 [22:26<98:38:15,  8.64s/it]
                                                      
{'loss': 0.3563, 'grad_norm': 3.7761318683624268, 'learning_rate': 4.878836833602585e-06, 'epoch': 0.04}

  0%|          | 151/41250 [22:26<98:38:15,  8.64s/it][2025-04-25 08:20:09,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:20:09,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.18 | bwd_microstep: 5741.58 | bwd_inner_microstep: 5639.18 | bwd_allreduce_microstep: 102.36 | step_microstep: 18.26
[2025-04-25 08:20:09,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.18 | bwd: 5741.60 | bwd_inner: 5639.18 | bwd_allreduce: 102.38 | step: 18.26

  0%|          | 152/41250 [22:34<98:43:12,  8.65s/it]
                                                      
{'loss': 0.2437, 'grad_norm': 2.1894242763519287, 'learning_rate': 4.911147011308562e-06, 'epoch': 0.04}

  0%|          | 152/41250 [22:34<98:43:12,  8.65s/it][2025-04-25 08:20:18,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:20:18,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5712.83 | bwd_inner_microstep: 5678.76 | bwd_allreduce_microstep: 34.03 | step_microstep: 18.32
[2025-04-25 08:20:18,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5712.84 | bwd_inner: 5678.76 | bwd_allreduce: 34.04 | step: 18.32

  0%|          | 153/41250 [22:43<98:43:00,  8.65s/it]
                                                      
{'loss': 0.2795, 'grad_norm': 2.2487986087799072, 'learning_rate': 4.94345718901454e-06, 'epoch': 0.04}

  0%|          | 153/41250 [22:43<98:43:00,  8.65s/it][2025-04-25 08:20:26,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:20:26,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.19 | bwd_microstep: 5700.15 | bwd_inner_microstep: 5687.52 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.29
[2025-04-25 08:20:26,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.19 | bwd: 5700.16 | bwd_inner: 5687.52 | bwd_allreduce: 12.60 | step: 18.29

  0%|          | 154/41250 [22:51<98:40:24,  8.64s/it]
                                                      
{'loss': 0.3936, 'grad_norm': 2.1463818550109863, 'learning_rate': 4.975767366720518e-06, 'epoch': 0.04}

  0%|          | 154/41250 [22:51<98:40:24,  8.64s/it][2025-04-25 08:20:35,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:20:35,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.92 | bwd_microstep: 5691.93 | bwd_inner_microstep: 5679.30 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.55
[2025-04-25 08:20:35,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.92 | bwd: 5691.95 | bwd_inner: 5679.30 | bwd_allreduce: 12.61 | step: 18.55

  0%|          | 155/41250 [23:00<98:37:26,  8.64s/it]
                                                      
{'loss': 0.2737, 'grad_norm': 2.757199764251709, 'learning_rate': 5.008077544426495e-06, 'epoch': 0.04}

  0%|          | 155/41250 [23:00<98:37:26,  8.64s/it][2025-04-25 08:20:43,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:20:43,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.05 | bwd_microstep: 5738.82 | bwd_inner_microstep: 5681.24 | bwd_allreduce_microstep: 57.53 | step_microstep: 18.32
[2025-04-25 08:20:43,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.05 | bwd: 5738.83 | bwd_inner: 5681.24 | bwd_allreduce: 57.55 | step: 18.32

  0%|          | 156/41250 [23:09<98:45:24,  8.65s/it]
                                                      
{'loss': 0.3904, 'grad_norm': 3.949371099472046, 'learning_rate': 5.040387722132473e-06, 'epoch': 0.04}

  0%|          | 156/41250 [23:09<98:45:24,  8.65s/it][2025-04-25 08:20:52,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-25 08:20:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.77 | bwd_microstep: 5714.56 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.33
[2025-04-25 08:20:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.77 | bwd: 5714.57 | bwd_inner: 5701.94 | bwd_allreduce: 12.59 | step: 18.33

  0%|          | 157/41250 [23:17<98:43:14,  8.65s/it]
                                                      
{'loss': 0.2287, 'grad_norm': 1.1969586610794067, 'learning_rate': 5.072697899838449e-06, 'epoch': 0.04}

  0%|          | 157/41250 [23:17<98:43:14,  8.65s/it][2025-04-25 08:21:01,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 08:21:01,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.18 | bwd_microstep: 5750.60 | bwd_inner_microstep: 5650.68 | bwd_allreduce_microstep: 99.87 | step_microstep: 18.02
[2025-04-25 08:21:01,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.18 | bwd: 5750.61 | bwd_inner: 5650.68 | bwd_allreduce: 99.89 | step: 18.02

  0%|          | 158/41250 [23:26<98:45:47,  8.65s/it]
                                                      
{'loss': 0.4646, 'grad_norm': 2.679903268814087, 'learning_rate': 5.105008077544427e-06, 'epoch': 0.04}

  0%|          | 158/41250 [23:26<98:45:47,  8.65s/it][2025-04-25 08:21:09,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:21:09,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.84 | bwd_microstep: 5742.40 | bwd_inner_microstep: 5648.89 | bwd_allreduce_microstep: 93.46 | step_microstep: 18.53
[2025-04-25 08:21:09,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.84 | bwd: 5742.41 | bwd_inner: 5648.89 | bwd_allreduce: 93.48 | step: 18.53

  0%|          | 159/41250 [23:35<98:48:04,  8.66s/it]
                                                      
{'loss': 0.213, 'grad_norm': 1.7379741668701172, 'learning_rate': 5.137318255250404e-06, 'epoch': 0.04}

  0%|          | 159/41250 [23:35<98:48:04,  8.66s/it][2025-04-25 08:21:18,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:21:18,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.04 | bwd_microstep: 5702.85 | bwd_inner_microstep: 5690.25 | bwd_allreduce_microstep: 12.56 | step_microstep: 17.93
[2025-04-25 08:21:18,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5702.86 | bwd_inner: 5690.25 | bwd_allreduce: 12.58 | step: 17.93

  0%|          | 160/41250 [23:43<98:44:02,  8.65s/it]
                                                      
{'loss': 0.1216, 'grad_norm': 1.1707361936569214, 'learning_rate': 5.169628432956382e-06, 'epoch': 0.04}

  0%|          | 160/41250 [23:43<98:44:02,  8.65s/it][2025-04-25 08:21:27,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-25 08:21:27,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5830.42 | bwd_inner_microstep: 5691.13 | bwd_allreduce_microstep: 139.25 | step_microstep: 18.02
[2025-04-25 08:21:27,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5830.43 | bwd_inner: 5691.13 | bwd_allreduce: 139.26 | step: 18.02

  0%|          | 161/41250 [23:52<99:06:47,  8.68s/it]
                                                      
{'loss': 0.2536, 'grad_norm': 2.3161497116088867, 'learning_rate': 5.20193861066236e-06, 'epoch': 0.04}

  0%|          | 161/41250 [23:52<99:06:47,  8.68s/it][2025-04-25 08:21:35,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:21:35,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.07 | bwd_microstep: 5693.33 | bwd_inner_microstep: 5644.62 | bwd_allreduce_microstep: 48.67 | step_microstep: 18.40
[2025-04-25 08:21:35,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.07 | bwd: 5693.34 | bwd_inner: 5644.62 | bwd_allreduce: 48.68 | step: 18.40

  0%|          | 162/41250 [24:01<98:50:57,  8.66s/it]
                                                      
{'loss': 0.2761, 'grad_norm': 1.6566516160964966, 'learning_rate': 5.234248788368337e-06, 'epoch': 0.04}

  0%|          | 162/41250 [24:01<98:50:57,  8.66s/it][2025-04-25 08:21:44,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:21:44,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.44 | bwd_microstep: 5722.48 | bwd_inner_microstep: 5709.80 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.56
[2025-04-25 08:21:44,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.44 | bwd: 5722.49 | bwd_inner: 5709.80 | bwd_allreduce: 12.65 | step: 18.56

  0%|          | 163/41250 [24:09<98:51:53,  8.66s/it]
                                                      
{'loss': 0.2931, 'grad_norm': 2.5471081733703613, 'learning_rate': 5.2665589660743146e-06, 'epoch': 0.04}

  0%|          | 163/41250 [24:09<98:51:53,  8.66s/it][2025-04-25 08:21:53,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:21:53,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.10 | bwd_microstep: 5727.59 | bwd_inner_microstep: 5641.33 | bwd_allreduce_microstep: 86.21 | step_microstep: 18.33
[2025-04-25 08:21:53,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.10 | bwd: 5727.61 | bwd_inner: 5641.33 | bwd_allreduce: 86.23 | step: 18.33

  0%|          | 164/41250 [24:18<98:50:08,  8.66s/it]
                                                      
{'loss': 0.3574, 'grad_norm': 3.5321054458618164, 'learning_rate': 5.298869143780291e-06, 'epoch': 0.04}

  0%|          | 164/41250 [24:18<98:50:08,  8.66s/it][2025-04-25 08:22:01,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:22:01,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.77 | bwd_microstep: 5708.27 | bwd_inner_microstep: 5695.47 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.36
[2025-04-25 08:22:01,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.77 | bwd: 5708.28 | bwd_inner: 5695.47 | bwd_allreduce: 12.77 | step: 18.36

  0%|          | 165/41250 [24:27<98:47:32,  8.66s/it]
                                                      
{'loss': 0.2442, 'grad_norm': 1.9053012132644653, 'learning_rate': 5.331179321486269e-06, 'epoch': 0.04}

  0%|          | 165/41250 [24:27<98:47:32,  8.66s/it][2025-04-25 08:22:10,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:22:10,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.12 | bwd_microstep: 5755.19 | bwd_inner_microstep: 5649.52 | bwd_allreduce_microstep: 105.63 | step_microstep: 18.08
[2025-04-25 08:22:10,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.12 | bwd: 5755.20 | bwd_inner: 5649.51 | bwd_allreduce: 105.65 | step: 18.08

  0%|          | 166/41250 [24:35<98:49:50,  8.66s/it]
                                                      
{'loss': 0.3848, 'grad_norm': 2.1766130924224854, 'learning_rate': 5.363489499192246e-06, 'epoch': 0.04}

  0%|          | 166/41250 [24:35<98:49:50,  8.66s/it][2025-04-25 08:22:19,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 08:22:19,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.30 | bwd_microstep: 5744.74 | bwd_inner_microstep: 5706.64 | bwd_allreduce_microstep: 38.06 | step_microstep: 18.58
[2025-04-25 08:22:19,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.30 | bwd: 5744.76 | bwd_inner: 5706.64 | bwd_allreduce: 38.08 | step: 18.58

  0%|          | 167/41250 [24:44<98:53:27,  8.67s/it]
                                                      
{'loss': 0.2999, 'grad_norm': 2.8695788383483887, 'learning_rate': 5.3957996768982236e-06, 'epoch': 0.04}

  0%|          | 167/41250 [24:44<98:53:27,  8.67s/it][2025-04-25 08:22:27,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:22:27,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.01 | bwd_microstep: 5706.47 | bwd_inner_microstep: 5650.53 | bwd_allreduce_microstep: 55.90 | step_microstep: 18.12
[2025-04-25 08:22:27,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.01 | bwd: 5706.48 | bwd_inner: 5650.53 | bwd_allreduce: 55.91 | step: 18.12

  0%|          | 168/41250 [24:53<98:44:09,  8.65s/it]
                                                      
{'loss': 0.2916, 'grad_norm': 1.999558925628662, 'learning_rate': 5.4281098546042014e-06, 'epoch': 0.04}

  0%|          | 168/41250 [24:53<98:44:09,  8.65s/it][2025-04-25 08:22:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:22:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5720.11 | bwd_inner_microstep: 5707.41 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.51
[2025-04-25 08:22:36,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5720.13 | bwd_inner: 5707.41 | bwd_allreduce: 12.68 | step: 18.51

  0%|          | 169/41250 [25:01<98:45:10,  8.65s/it]
                                                      
{'loss': 0.2453, 'grad_norm': 1.8276561498641968, 'learning_rate': 5.4604200323101785e-06, 'epoch': 0.04}

  0%|          | 169/41250 [25:01<98:45:10,  8.65s/it][2025-04-25 08:22:45,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:22:45,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.00 | bwd_microstep: 5790.79 | bwd_inner_microstep: 5778.02 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.00
[2025-04-25 08:22:45,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.00 | bwd: 5790.80 | bwd_inner: 5778.02 | bwd_allreduce: 12.75 | step: 18.00

  0%|          | 170/41250 [25:10<99:08:35,  8.69s/it]
                                                      
{'loss': 0.3318, 'grad_norm': 1.5441128015518188, 'learning_rate': 5.492730210016155e-06, 'epoch': 0.04}

  0%|          | 170/41250 [25:10<99:08:35,  8.69s/it][2025-04-25 08:22:53,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.94 | optimizer_step: 0.96
[2025-04-25 08:22:53,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.24 | bwd_microstep: 5756.07 | bwd_inner_microstep: 5659.38 | bwd_allreduce_microstep: 96.65 | step_microstep: 17.93
[2025-04-25 08:22:53,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.24 | bwd: 5756.08 | bwd_inner: 5659.38 | bwd_allreduce: 96.67 | step: 17.94

  0%|          | 171/41250 [25:19<99:06:13,  8.69s/it]
                                                      
{'loss': 0.2901, 'grad_norm': 3.9108798503875732, 'learning_rate': 5.5250403877221325e-06, 'epoch': 0.04}

  0%|          | 171/41250 [25:19<99:06:13,  8.69s/it][2025-04-25 08:23:02,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:23:02,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.20 | bwd_microstep: 5763.50 | bwd_inner_microstep: 5659.14 | bwd_allreduce_microstep: 104.32 | step_microstep: 18.29
[2025-04-25 08:23:02,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.20 | bwd: 5763.51 | bwd_inner: 5659.14 | bwd_allreduce: 104.33 | step: 18.30

  0%|          | 172/41250 [25:27<99:04:15,  8.68s/it]
                                                      
{'loss': 0.3375, 'grad_norm': 2.2783660888671875, 'learning_rate': 5.5573505654281104e-06, 'epoch': 0.04}

  0%|          | 172/41250 [25:27<99:04:15,  8.68s/it][2025-04-25 08:23:11,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:23:11,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.22 | bwd_microstep: 5744.16 | bwd_inner_microstep: 5689.48 | bwd_allreduce_microstep: 54.63 | step_microstep: 18.04
[2025-04-25 08:23:11,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.22 | bwd: 5744.17 | bwd_inner: 5689.48 | bwd_allreduce: 54.65 | step: 18.04

  0%|          | 173/41250 [25:36<99:02:03,  8.68s/it]
                                                      
{'loss': 0.2292, 'grad_norm': 1.5509450435638428, 'learning_rate': 5.5896607431340875e-06, 'epoch': 0.04}

  0%|          | 173/41250 [25:36<99:02:03,  8.68s/it][2025-04-25 08:23:20,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.93 | optimizer_step: 0.90
[2025-04-25 08:23:20,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.98 | bwd_microstep: 5892.86 | bwd_inner_microstep: 5649.99 | bwd_allreduce_microstep: 242.83 | step_microstep: 18.35
[2025-04-25 08:23:20,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.98 | bwd: 5892.87 | bwd_inner: 5649.99 | bwd_allreduce: 242.84 | step: 18.35

  0%|          | 174/41250 [25:45<99:28:34,  8.72s/it]
                                                      
{'loss': 0.2127, 'grad_norm': 1.4063142538070679, 'learning_rate': 5.621970920840065e-06, 'epoch': 0.04}

  0%|          | 174/41250 [25:45<99:28:34,  8.72s/it][2025-04-25 08:23:28,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:23:28,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.53 | bwd_microstep: 5714.37 | bwd_inner_microstep: 5651.92 | bwd_allreduce_microstep: 62.41 | step_microstep: 18.21
[2025-04-25 08:23:28,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.53 | bwd: 5714.38 | bwd_inner: 5651.92 | bwd_allreduce: 62.42 | step: 18.21

  0%|          | 175/41250 [25:54<99:08:51,  8.69s/it]
                                                      
{'loss': 0.2889, 'grad_norm': 2.0951836109161377, 'learning_rate': 5.654281098546042e-06, 'epoch': 0.04}

  0%|          | 175/41250 [25:54<99:08:51,  8.69s/it][2025-04-25 08:23:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:23:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.51 | bwd_microstep: 5765.33 | bwd_inner_microstep: 5664.63 | bwd_allreduce_microstep: 100.65 | step_microstep: 17.70
[2025-04-25 08:23:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.51 | bwd: 5765.34 | bwd_inner: 5664.63 | bwd_allreduce: 100.67 | step: 17.70

  0%|          | 176/41250 [26:02<99:09:51,  8.69s/it]
                                                      
{'loss': 0.3367, 'grad_norm': 1.6981961727142334, 'learning_rate': 5.68659127625202e-06, 'epoch': 0.04}

  0%|          | 176/41250 [26:02<99:09:51,  8.69s/it][2025-04-25 08:23:46,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 08:23:46,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.75 | bwd_microstep: 5866.72 | bwd_inner_microstep: 5703.39 | bwd_allreduce_microstep: 163.29 | step_microstep: 18.77
[2025-04-25 08:23:46,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.75 | bwd: 5866.73 | bwd_inner: 5703.39 | bwd_allreduce: 163.31 | step: 18.78

  0%|          | 177/41250 [26:11<99:32:32,  8.72s/it]
                                                      
{'loss': 0.1767, 'grad_norm': 1.4252803325653076, 'learning_rate': 5.7189014539579965e-06, 'epoch': 0.04}

  0%|          | 177/41250 [26:11<99:32:32,  8.72s/it][2025-04-25 08:23:54,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:23:54,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.18 | bwd_microstep: 5783.96 | bwd_inner_microstep: 5665.72 | bwd_allreduce_microstep: 118.20 | step_microstep: 17.81
[2025-04-25 08:23:54,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.18 | bwd: 5783.98 | bwd_inner: 5665.72 | bwd_allreduce: 118.22 | step: 17.81

  0%|          | 178/41250 [26:20<99:26:55,  8.72s/it]
                                                      
{'loss': 0.2114, 'grad_norm': 1.4929949045181274, 'learning_rate': 5.751211631663974e-06, 'epoch': 0.04}

  0%|          | 178/41250 [26:20<99:26:55,  8.72s/it][2025-04-25 08:24:03,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:24:03,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.13 | bwd_microstep: 5725.17 | bwd_inner_microstep: 5712.44 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.58
[2025-04-25 08:24:03,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.13 | bwd: 5725.18 | bwd_inner: 5712.44 | bwd_allreduce: 12.71 | step: 18.58

  0%|          | 179/41250 [26:28<99:16:55,  8.70s/it]
                                                      
{'loss': 0.3849, 'grad_norm': 2.040151596069336, 'learning_rate': 5.783521809369952e-06, 'epoch': 0.04}

  0%|          | 179/41250 [26:28<99:16:55,  8.70s/it][2025-04-25 08:24:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:24:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.90 | bwd_microstep: 5717.11 | bwd_inner_microstep: 5663.28 | bwd_allreduce_microstep: 53.78 | step_microstep: 18.35
[2025-04-25 08:24:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.90 | bwd: 5717.12 | bwd_inner: 5663.28 | bwd_allreduce: 53.80 | step: 18.36

  0%|          | 180/41250 [26:37<99:02:41,  8.68s/it]
                                                      
{'loss': 0.3642, 'grad_norm': 2.721205234527588, 'learning_rate': 5.815831987075929e-06, 'epoch': 0.04}

  0%|          | 180/41250 [26:37<99:02:41,  8.68s/it][2025-04-25 08:24:20,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.76 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:24:20,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.25 | bwd_microstep: 5727.59 | bwd_inner_microstep: 5670.11 | bwd_allreduce_microstep: 57.44 | step_microstep: 17.60
[2025-04-25 08:24:20,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.25 | bwd: 5727.60 | bwd_inner: 5670.11 | bwd_allreduce: 57.45 | step: 17.60

  0%|          | 181/41250 [26:46<98:53:58,  8.67s/it]
                                                      
{'loss': 0.1971, 'grad_norm': 1.452427625656128, 'learning_rate': 5.848142164781907e-06, 'epoch': 0.04}

  0%|          | 181/41250 [26:46<98:53:58,  8.67s/it][2025-04-25 08:24:29,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:24:29,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.77 | bwd_microstep: 5704.83 | bwd_inner_microstep: 5664.71 | bwd_allreduce_microstep: 40.08 | step_microstep: 17.83
[2025-04-25 08:24:29,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.77 | bwd: 5704.84 | bwd_inner: 5664.71 | bwd_allreduce: 40.09 | step: 17.83

  0%|          | 182/41250 [26:54<98:43:03,  8.65s/it]
                                                      
{'loss': 0.2725, 'grad_norm': 1.92888605594635, 'learning_rate': 5.880452342487884e-06, 'epoch': 0.04}

  0%|          | 182/41250 [26:54<98:43:03,  8.65s/it][2025-04-25 08:24:38,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.94 | optimizer_step: 0.99
[2025-04-25 08:24:38,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.45 | bwd_microstep: 5783.10 | bwd_inner_microstep: 5726.00 | bwd_allreduce_microstep: 57.06 | step_microstep: 18.08
[2025-04-25 08:24:38,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.45 | bwd: 5783.12 | bwd_inner: 5726.00 | bwd_allreduce: 57.08 | step: 18.08

  0%|          | 183/41250 [27:03<98:58:02,  8.68s/it]
                                                      
{'loss': 0.2548, 'grad_norm': 2.0660057067871094, 'learning_rate': 5.912762520193862e-06, 'epoch': 0.04}

  0%|          | 183/41250 [27:03<98:58:02,  8.68s/it][2025-04-25 08:24:46,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.94 | optimizer_step: 0.95
[2025-04-25 08:24:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.78 | bwd_microstep: 5717.37 | bwd_inner_microstep: 5670.96 | bwd_allreduce_microstep: 46.36 | step_microstep: 18.37
[2025-04-25 08:24:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.78 | bwd: 5717.38 | bwd_inner: 5670.96 | bwd_allreduce: 46.38 | step: 18.38

  0%|          | 184/41250 [27:12<98:48:51,  8.66s/it]
                                                      
{'loss': 0.3515, 'grad_norm': 2.4447379112243652, 'learning_rate': 5.945072697899838e-06, 'epoch': 0.04}

  0%|          | 184/41250 [27:12<98:48:51,  8.66s/it][2025-04-25 08:24:55,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:24:55,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.08 | bwd_microstep: 5798.81 | bwd_inner_microstep: 5786.13 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.37
[2025-04-25 08:24:55,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.08 | bwd: 5798.82 | bwd_inner: 5786.13 | bwd_allreduce: 12.66 | step: 18.37

  0%|          | 185/41250 [27:20<99:12:10,  8.70s/it]
                                                      
{'loss': 0.3207, 'grad_norm': 1.6176937818527222, 'learning_rate': 5.977382875605816e-06, 'epoch': 0.04}

  0%|          | 185/41250 [27:20<99:12:10,  8.70s/it][2025-04-25 08:25:04,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:25:04,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.48 | bwd_microstep: 5744.92 | bwd_inner_microstep: 5707.76 | bwd_allreduce_microstep: 37.12 | step_microstep: 18.33
[2025-04-25 08:25:04,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.48 | bwd: 5744.93 | bwd_inner: 5707.75 | bwd_allreduce: 37.14 | step: 18.33

  0%|          | 186/41250 [27:29<99:08:18,  8.69s/it]
                                                      
{'loss': 0.4035, 'grad_norm': 2.9320404529571533, 'learning_rate': 6.009693053311794e-06, 'epoch': 0.05}

  0%|          | 186/41250 [27:29<99:08:18,  8.69s/it][2025-04-25 08:25:13,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:25:13,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.02 | bwd_microstep: 5775.58 | bwd_inner_microstep: 5666.90 | bwd_allreduce_microstep: 108.64 | step_microstep: 18.35
[2025-04-25 08:25:13,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.02 | bwd: 5775.59 | bwd_inner: 5666.90 | bwd_allreduce: 108.65 | step: 18.35

  0%|          | 187/41250 [27:38<99:11:20,  8.70s/it]
                                                      
{'loss': 0.3182, 'grad_norm': 1.7718267440795898, 'learning_rate': 6.042003231017771e-06, 'epoch': 0.05}

  0%|          | 187/41250 [27:38<99:11:20,  8.70s/it][2025-04-25 08:25:21,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:25:21,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.38 | bwd_microstep: 5749.39 | bwd_inner_microstep: 5711.20 | bwd_allreduce_microstep: 38.14 | step_microstep: 18.35
[2025-04-25 08:25:21,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.38 | bwd: 5749.40 | bwd_inner: 5711.20 | bwd_allreduce: 38.15 | step: 18.36

  0%|          | 188/41250 [27:47<99:09:58,  8.69s/it]
                                                      
{'loss': 0.3317, 'grad_norm': 3.3080625534057617, 'learning_rate': 6.074313408723749e-06, 'epoch': 0.05}

  0%|          | 188/41250 [27:47<99:09:58,  8.69s/it][2025-04-25 08:25:30,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 08:25:30,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5776.78 | bwd_inner_microstep: 5661.11 | bwd_allreduce_microstep: 115.62 | step_microstep: 18.55
[2025-04-25 08:25:30,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5776.79 | bwd_inner: 5661.11 | bwd_allreduce: 115.64 | step: 18.55

  0%|          | 189/41250 [27:55<99:09:14,  8.69s/it]
                                                      
{'loss': 0.2497, 'grad_norm': 1.4691170454025269, 'learning_rate': 6.106623586429726e-06, 'epoch': 0.05}

  0%|          | 189/41250 [27:55<99:09:14,  8.69s/it][2025-04-25 08:25:39,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:25:39,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.26 | bwd_microstep: 5722.45 | bwd_inner_microstep: 5657.38 | bwd_allreduce_microstep: 65.03 | step_microstep: 18.16
[2025-04-25 08:25:39,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.26 | bwd: 5722.46 | bwd_inner: 5657.38 | bwd_allreduce: 65.04 | step: 18.17

  0%|          | 190/41250 [28:04<98:55:42,  8.67s/it]
                                                      
{'loss': 0.4357, 'grad_norm': 1.97160005569458, 'learning_rate': 6.138933764135703e-06, 'epoch': 0.05}

  0%|          | 190/41250 [28:04<98:55:42,  8.67s/it][2025-04-25 08:25:47,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 08:25:47,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.49 | bwd_microstep: 5720.07 | bwd_inner_microstep: 5707.34 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.26
[2025-04-25 08:25:47,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.49 | bwd: 5720.08 | bwd_inner: 5707.34 | bwd_allreduce: 12.70 | step: 18.26

  0%|          | 191/41250 [28:13<98:50:41,  8.67s/it]
                                                      
{'loss': 0.3787, 'grad_norm': 2.080832004547119, 'learning_rate': 6.17124394184168e-06, 'epoch': 0.05}

  0%|          | 191/41250 [28:13<98:50:41,  8.67s/it][2025-04-25 08:25:56,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:25:56,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2939.21 | bwd_microstep: 5869.50 | bwd_inner_microstep: 5857.02 | bwd_allreduce_microstep: 12.44 | step_microstep: 17.71
[2025-04-25 08:25:56,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2939.21 | bwd: 5869.52 | bwd_inner: 5857.02 | bwd_allreduce: 12.46 | step: 17.72

  0%|          | 192/41250 [28:21<99:36:38,  8.73s/it]
                                                      
{'loss': 0.2088, 'grad_norm': 1.1542068719863892, 'learning_rate': 6.203554119547658e-06, 'epoch': 0.05}

  0%|          | 192/41250 [28:21<99:36:38,  8.73s/it][2025-04-25 08:26:05,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:26:05,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.87 | bwd_microstep: 5703.82 | bwd_inner_microstep: 5668.68 | bwd_allreduce_microstep: 35.10 | step_microstep: 17.84
[2025-04-25 08:26:05,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.87 | bwd: 5703.83 | bwd_inner: 5668.68 | bwd_allreduce: 35.12 | step: 17.85

  0%|          | 193/41250 [28:30<99:13:27,  8.70s/it]
                                                      
{'loss': 0.2567, 'grad_norm': 1.598849892616272, 'learning_rate': 6.235864297253636e-06, 'epoch': 0.05}

  0%|          | 193/41250 [28:30<99:13:27,  8.70s/it][2025-04-25 08:26:13,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-25 08:26:13,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.42 | bwd_microstep: 5689.25 | bwd_inner_microstep: 5649.87 | bwd_allreduce_microstep: 39.33 | step_microstep: 18.13
[2025-04-25 08:26:13,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.42 | bwd: 5689.26 | bwd_inner: 5649.87 | bwd_allreduce: 39.35 | step: 18.14

  0%|          | 194/41250 [28:39<98:52:56,  8.67s/it]
                                                      
{'loss': 0.1789, 'grad_norm': 1.3840177059173584, 'learning_rate': 6.268174474959613e-06, 'epoch': 0.05}

  0%|          | 194/41250 [28:39<98:52:56,  8.67s/it][2025-04-25 08:26:22,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:26:22,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.08 | bwd_microstep: 5721.46 | bwd_inner_microstep: 5708.49 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.38
[2025-04-25 08:26:22,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.08 | bwd: 5721.47 | bwd_inner: 5708.49 | bwd_allreduce: 12.94 | step: 18.38

  0%|          | 195/41250 [28:47<98:52:05,  8.67s/it]
                                                      
{'loss': 0.3851, 'grad_norm': 1.9259555339813232, 'learning_rate': 6.300484652665591e-06, 'epoch': 0.05}

  0%|          | 195/41250 [28:47<98:52:05,  8.67s/it][2025-04-25 08:26:31,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:26:31,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.00 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5697.00 | bwd_allreduce_microstep: 12.53 | step_microstep: 17.72
[2025-04-25 08:26:31,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.00 | bwd: 5709.58 | bwd_inner: 5697.00 | bwd_allreduce: 12.54 | step: 17.72

  0%|          | 196/41250 [28:56<98:47:36,  8.66s/it]
                                                      
{'loss': 0.3305, 'grad_norm': 1.743261456489563, 'learning_rate': 6.332794830371568e-06, 'epoch': 0.05}

  0%|          | 196/41250 [28:56<98:47:36,  8.66s/it][2025-04-25 08:26:39,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:26:39,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.46 | bwd_microstep: 5751.18 | bwd_inner_microstep: 5661.49 | bwd_allreduce_microstep: 89.64 | step_microstep: 17.98
[2025-04-25 08:26:39,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.46 | bwd: 5751.19 | bwd_inner: 5661.49 | bwd_allreduce: 89.66 | step: 17.98

  0%|          | 197/41250 [29:05<98:49:19,  8.67s/it]
                                                      
{'loss': 0.3331, 'grad_norm': 1.9405657052993774, 'learning_rate': 6.365105008077545e-06, 'epoch': 0.05}

  0%|          | 197/41250 [29:05<98:49:19,  8.67s/it][2025-04-25 08:26:48,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:26:48,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.37 | bwd_microstep: 5691.63 | bwd_inner_microstep: 5679.11 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.37
[2025-04-25 08:26:48,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.37 | bwd: 5691.64 | bwd_inner: 5679.11 | bwd_allreduce: 12.50 | step: 18.37

  0%|          | 198/41250 [29:13<98:40:26,  8.65s/it]
                                                      
{'loss': 0.2661, 'grad_norm': 1.6784251928329468, 'learning_rate': 6.397415185783522e-06, 'epoch': 0.05}

  0%|          | 198/41250 [29:13<98:40:26,  8.65s/it][2025-04-25 08:26:57,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.95 | optimizer_step: 1.01
[2025-04-25 08:26:57,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.03 | bwd_microstep: 5728.24 | bwd_inner_microstep: 5695.30 | bwd_allreduce_microstep: 32.90 | step_microstep: 18.02
[2025-04-25 08:26:57,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.03 | bwd: 5728.25 | bwd_inner: 5695.29 | bwd_allreduce: 32.92 | step: 18.02

  0%|          | 199/41250 [29:22<98:44:32,  8.66s/it]
                                                      
{'loss': 0.3972, 'grad_norm': 6.0914692878723145, 'learning_rate': 6.4297253634895e-06, 'epoch': 0.05}

  0%|          | 199/41250 [29:22<98:44:32,  8.66s/it][2025-04-25 08:27:05,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:27:05,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5696.24 | bwd_inner_microstep: 5683.49 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.35
[2025-04-25 08:27:05,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5696.26 | bwd_inner: 5683.49 | bwd_allreduce: 12.73 | step: 18.35

  0%|          | 200/41250 [29:31<98:38:02,  8.65s/it]
                                                      
{'loss': 0.1465, 'grad_norm': 4.008931636810303, 'learning_rate': 6.462035541195477e-06, 'epoch': 0.05}

  0%|          | 200/41250 [29:31<98:38:02,  8.65s/it][2025-04-25 08:27:14,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-25 08:27:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.13 | bwd_microstep: 5729.21 | bwd_inner_microstep: 5693.31 | bwd_allreduce_microstep: 35.84 | step_microstep: 18.81
[2025-04-25 08:27:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.13 | bwd: 5729.22 | bwd_inner: 5693.31 | bwd_allreduce: 35.87 | step: 18.81

  0%|          | 201/41250 [29:39<98:43:12,  8.66s/it]
                                                      
{'loss': 0.2925, 'grad_norm': 1.8300992250442505, 'learning_rate': 6.494345718901455e-06, 'epoch': 0.05}

  0%|          | 201/41250 [29:39<98:43:12,  8.66s/it][2025-04-25 08:27:23,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:27:23,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.64 | bwd_microstep: 5864.13 | bwd_inner_microstep: 5686.00 | bwd_allreduce_microstep: 178.08 | step_microstep: 18.58
[2025-04-25 08:27:23,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.64 | bwd: 5864.14 | bwd_inner: 5686.00 | bwd_allreduce: 178.10 | step: 18.58

  0%|          | 202/41250 [29:48<99:10:27,  8.70s/it]
                                                      
{'loss': 0.3617, 'grad_norm': 1.9004766941070557, 'learning_rate': 6.5266558966074325e-06, 'epoch': 0.05}

  0%|          | 202/41250 [29:48<99:10:27,  8.70s/it][2025-04-25 08:27:31,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:27:31,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5700.89 | bwd_inner_microstep: 5640.70 | bwd_allreduce_microstep: 60.15 | step_microstep: 18.34
[2025-04-25 08:27:31,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5700.91 | bwd_inner: 5640.70 | bwd_allreduce: 60.17 | step: 18.34

  0%|          | 203/41250 [29:57<98:52:26,  8.67s/it]
                                                      
{'loss': 0.2801, 'grad_norm': 1.848279595375061, 'learning_rate': 6.5589660743134096e-06, 'epoch': 0.05}

  0%|          | 203/41250 [29:57<98:52:26,  8.67s/it][2025-04-25 08:27:40,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:27:40,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.83 | bwd_microstep: 5769.35 | bwd_inner_microstep: 5652.79 | bwd_allreduce_microstep: 116.51 | step_microstep: 18.60
[2025-04-25 08:27:40,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.83 | bwd: 5769.36 | bwd_inner: 5652.79 | bwd_allreduce: 116.53 | step: 18.60

  0%|          | 204/41250 [30:05<98:54:02,  8.67s/it]
                                                      
{'loss': 0.1568, 'grad_norm': 0.99672532081604, 'learning_rate': 6.591276252019387e-06, 'epoch': 0.05}

  0%|          | 204/41250 [30:05<98:54:02,  8.67s/it][2025-04-25 08:27:49,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:27:49,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.42 | bwd_microstep: 5765.34 | bwd_inner_microstep: 5631.73 | bwd_allreduce_microstep: 133.57 | step_microstep: 17.84
[2025-04-25 08:27:49,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.42 | bwd: 5765.35 | bwd_inner: 5631.72 | bwd_allreduce: 133.59 | step: 17.84

  0%|          | 205/41250 [30:14<98:53:59,  8.67s/it]
                                                      
{'loss': 0.2492, 'grad_norm': 1.5170283317565918, 'learning_rate': 6.623586429725364e-06, 'epoch': 0.05}

  0%|          | 205/41250 [30:14<98:53:59,  8.67s/it][2025-04-25 08:27:57,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:27:57,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.17 | bwd_microstep: 5751.45 | bwd_inner_microstep: 5646.31 | bwd_allreduce_microstep: 105.11 | step_microstep: 18.42
[2025-04-25 08:27:57,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5751.47 | bwd_inner: 5646.31 | bwd_allreduce: 105.12 | step: 18.42

  0%|          | 206/41250 [30:23<98:51:25,  8.67s/it]
                                                      
{'loss': 0.18, 'grad_norm': 1.3070298433303833, 'learning_rate': 6.6558966074313415e-06, 'epoch': 0.05}

  0%|          | 206/41250 [30:23<98:51:25,  8.67s/it][2025-04-25 08:28:06,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 08:28:06,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.27 | bwd_microstep: 5709.04 | bwd_inner_microstep: 5696.30 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.66
[2025-04-25 08:28:06,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.27 | bwd: 5709.05 | bwd_inner: 5696.30 | bwd_allreduce: 12.72 | step: 18.66

  1%|          | 207/41250 [30:31<98:46:32,  8.66s/it]
                                                      
{'loss': 0.2128, 'grad_norm': 1.345746397972107, 'learning_rate': 6.6882067851373186e-06, 'epoch': 0.05}

  1%|          | 207/41250 [30:31<98:46:32,  8.66s/it][2025-04-25 08:28:15,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:28:15,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.58 | bwd_microstep: 5756.29 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 60.44 | step_microstep: 18.43
[2025-04-25 08:28:15,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.58 | bwd: 5756.31 | bwd_inner: 5695.81 | bwd_allreduce: 60.46 | step: 18.44

  1%|          | 208/41250 [30:40<98:50:17,  8.67s/it]
                                                      
{'loss': 0.2778, 'grad_norm': 1.8986737728118896, 'learning_rate': 6.7205169628432965e-06, 'epoch': 0.05}

  1%|          | 208/41250 [30:40<98:50:17,  8.67s/it][2025-04-25 08:28:23,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 08:28:23,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.35 | bwd_microstep: 5750.12 | bwd_inner_microstep: 5632.92 | bwd_allreduce_microstep: 117.16 | step_microstep: 18.96
[2025-04-25 08:28:23,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.35 | bwd: 5750.14 | bwd_inner: 5632.92 | bwd_allreduce: 117.18 | step: 18.96

  1%|          | 209/41250 [30:49<98:47:42,  8.67s/it]
                                                      
{'loss': 0.1783, 'grad_norm': 1.5668219327926636, 'learning_rate': 6.752827140549274e-06, 'epoch': 0.05}

  1%|          | 209/41250 [30:49<98:47:42,  8.67s/it][2025-04-25 08:28:32,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 08:28:32,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.48 | bwd_microstep: 5705.36 | bwd_inner_microstep: 5692.62 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.67
[2025-04-25 08:28:32,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.48 | bwd: 5705.37 | bwd_inner: 5692.62 | bwd_allreduce: 12.71 | step: 18.67

  1%|          | 210/41250 [30:57<98:41:52,  8.66s/it]
                                                      
{'loss': 0.1509, 'grad_norm': 1.6673667430877686, 'learning_rate': 6.7851373182552505e-06, 'epoch': 0.05}

  1%|          | 210/41250 [30:57<98:41:52,  8.66s/it][2025-04-25 08:28:41,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.29 | optimizer_step: 1.02
[2025-04-25 08:28:41,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.09 | bwd_microstep: 5766.39 | bwd_inner_microstep: 5643.09 | bwd_allreduce_microstep: 123.24 | step_microstep: 19.77
[2025-04-25 08:28:41,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.09 | bwd: 5766.41 | bwd_inner: 5643.09 | bwd_allreduce: 123.27 | step: 19.77

  1%|          | 211/41250 [31:06<98:44:54,  8.66s/it]
                                                      
{'loss': 0.2, 'grad_norm': 2.6164867877960205, 'learning_rate': 6.817447495961228e-06, 'epoch': 0.05}

  1%|          | 211/41250 [31:06<98:44:54,  8.66s/it][2025-04-25 08:28:49,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:28:49,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.12 | bwd_microstep: 5850.12 | bwd_inner_microstep: 5643.25 | bwd_allreduce_microstep: 206.83 | step_microstep: 18.33
[2025-04-25 08:28:49,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.12 | bwd: 5850.13 | bwd_inner: 5643.25 | bwd_allreduce: 206.85 | step: 18.33

  1%|          | 212/41250 [31:15<99:05:06,  8.69s/it]
                                                      
{'loss': 0.162, 'grad_norm': 1.1862845420837402, 'learning_rate': 6.8497576736672054e-06, 'epoch': 0.05}

  1%|          | 212/41250 [31:15<99:05:06,  8.69s/it][2025-04-25 08:28:58,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:28:58,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.49 | bwd_microstep: 5747.12 | bwd_inner_microstep: 5625.38 | bwd_allreduce_microstep: 121.69 | step_microstep: 18.44
[2025-04-25 08:28:58,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.49 | bwd: 5747.13 | bwd_inner: 5625.38 | bwd_allreduce: 121.71 | step: 18.44

  1%|          | 213/41250 [31:23<98:55:41,  8.68s/it]
                                                      
{'loss': 0.2677, 'grad_norm': 1.6326392889022827, 'learning_rate': 6.882067851373183e-06, 'epoch': 0.05}

  1%|          | 213/41250 [31:23<98:55:41,  8.68s/it][2025-04-25 08:29:07,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:29:07,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.40 | bwd_microstep: 5688.08 | bwd_inner_microstep: 5628.65 | bwd_allreduce_microstep: 59.39 | step_microstep: 17.81
[2025-04-25 08:29:07,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.40 | bwd: 5688.09 | bwd_inner: 5628.65 | bwd_allreduce: 59.40 | step: 17.81

  1%|          | 214/41250 [31:32<98:40:20,  8.66s/it]
                                                      
{'loss': 0.3174, 'grad_norm': 2.231602907180786, 'learning_rate': 6.91437802907916e-06, 'epoch': 0.05}

  1%|          | 214/41250 [31:32<98:40:20,  8.66s/it][2025-04-25 08:29:15,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:29:15,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.94 | bwd_microstep: 5848.49 | bwd_inner_microstep: 5677.69 | bwd_allreduce_microstep: 170.76 | step_microstep: 18.17
[2025-04-25 08:29:15,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.94 | bwd: 5848.50 | bwd_inner: 5677.69 | bwd_allreduce: 170.77 | step: 18.17

  1%|          | 215/41250 [31:41<99:06:05,  8.69s/it]
                                                      
{'loss': 0.2207, 'grad_norm': 2.066828489303589, 'learning_rate': 6.946688206785138e-06, 'epoch': 0.05}

  1%|          | 215/41250 [31:41<99:06:05,  8.69s/it][2025-04-25 08:29:24,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:29:24,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.04 | bwd_microstep: 5799.47 | bwd_inner_microstep: 5641.53 | bwd_allreduce_microstep: 157.89 | step_microstep: 17.83
[2025-04-25 08:29:24,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.04 | bwd: 5799.48 | bwd_inner: 5641.53 | bwd_allreduce: 157.91 | step: 17.84

  1%|          | 216/41250 [31:49<99:07:58,  8.70s/it]
                                                      
{'loss': 0.2204, 'grad_norm': 1.821297526359558, 'learning_rate': 6.978998384491116e-06, 'epoch': 0.05}

  1%|          | 216/41250 [31:49<99:07:58,  8.70s/it][2025-04-25 08:29:33,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 1.10
[2025-04-25 08:29:33,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.63 | bwd_microstep: 5730.56 | bwd_inner_microstep: 5641.89 | bwd_allreduce_microstep: 88.62 | step_microstep: 18.35
[2025-04-25 08:29:33,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.63 | bwd: 5730.57 | bwd_inner: 5641.89 | bwd_allreduce: 88.64 | step: 18.36

  1%|          | 217/41250 [31:58<98:55:46,  8.68s/it]
                                                      
{'loss': 0.3777, 'grad_norm': 2.037876844406128, 'learning_rate': 7.011308562197092e-06, 'epoch': 0.05}

  1%|          | 217/41250 [31:58<98:55:46,  8.68s/it][2025-04-25 08:29:41,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:29:41,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5704.93 | bwd_inner_microstep: 5677.39 | bwd_allreduce_microstep: 27.50 | step_microstep: 18.46
[2025-04-25 08:29:41,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5704.94 | bwd_inner: 5677.39 | bwd_allreduce: 27.52 | step: 18.47

  1%|          | 218/41250 [32:07<98:46:11,  8.67s/it]
                                                      
{'loss': 0.3517, 'grad_norm': 2.6871497631073, 'learning_rate': 7.043618739903069e-06, 'epoch': 0.05}

  1%|          | 218/41250 [32:07<98:46:11,  8.67s/it][2025-04-25 08:29:50,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:29:50,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5689.17 | bwd_inner_microstep: 5676.76 | bwd_allreduce_microstep: 12.36 | step_microstep: 18.72
[2025-04-25 08:29:50,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5689.18 | bwd_inner: 5676.76 | bwd_allreduce: 12.38 | step: 18.73

  1%|          | 219/41250 [32:15<98:36:43,  8.65s/it]
                                                      
{'loss': 0.3691, 'grad_norm': 2.2316458225250244, 'learning_rate': 7.075928917609047e-06, 'epoch': 0.05}

  1%|          | 219/41250 [32:15<98:36:43,  8.65s/it][2025-04-25 08:29:59,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:29:59,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.61 | bwd_microstep: 5681.53 | bwd_inner_microstep: 5669.01 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.32
[2025-04-25 08:29:59,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.61 | bwd: 5681.54 | bwd_inner: 5669.01 | bwd_allreduce: 12.49 | step: 18.33

  1%|          | 220/41250 [32:24<98:28:33,  8.64s/it]
                                                      
{'loss': 0.2968, 'grad_norm': 1.44822096824646, 'learning_rate': 7.108239095315025e-06, 'epoch': 0.05}

  1%|          | 220/41250 [32:24<98:28:33,  8.64s/it][2025-04-25 08:30:07,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.06 | optimizer_step: 1.12
[2025-04-25 08:30:07,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.33 | bwd_microstep: 5918.57 | bwd_inner_microstep: 5652.87 | bwd_allreduce_microstep: 265.65 | step_microstep: 19.33
[2025-04-25 08:30:07,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.33 | bwd: 5918.58 | bwd_inner: 5652.87 | bwd_allreduce: 265.67 | step: 19.34

  1%|          | 221/41250 [32:33<99:06:22,  8.70s/it]
                                                      
{'loss': 0.1941, 'grad_norm': 1.3828657865524292, 'learning_rate': 7.140549273021002e-06, 'epoch': 0.05}

  1%|          | 221/41250 [32:33<99:06:22,  8.70s/it][2025-04-25 08:30:16,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-25 08:30:16,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.97 | bwd_microstep: 5763.43 | bwd_inner_microstep: 5644.69 | bwd_allreduce_microstep: 118.69 | step_microstep: 19.50
[2025-04-25 08:30:16,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.97 | bwd: 5763.45 | bwd_inner: 5644.69 | bwd_allreduce: 118.71 | step: 19.50

  1%|          | 222/41250 [32:41<99:02:31,  8.69s/it]
                                                      
{'loss': 0.242, 'grad_norm': 1.8628588914871216, 'learning_rate': 7.17285945072698e-06, 'epoch': 0.05}

  1%|          | 222/41250 [32:41<99:02:31,  8.69s/it][2025-04-25 08:30:25,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:30:25,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2924.33 | bwd_microstep: 5846.29 | bwd_inner_microstep: 5833.45 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.53
[2025-04-25 08:30:25,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2924.33 | bwd: 5846.31 | bwd_inner: 5833.45 | bwd_allreduce: 12.82 | step: 18.53

  1%|          | 223/41250 [32:50<99:37:16,  8.74s/it]
                                                      
{'loss': 0.3037, 'grad_norm': 1.683786153793335, 'learning_rate': 7.205169628432956e-06, 'epoch': 0.05}

  1%|          | 223/41250 [32:50<99:37:16,  8.74s/it][2025-04-25 08:30:34,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:30:34,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.89 | bwd_microstep: 5717.06 | bwd_inner_microstep: 5704.25 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.48
[2025-04-25 08:30:34,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.89 | bwd: 5717.08 | bwd_inner: 5704.25 | bwd_allreduce: 12.79 | step: 18.48

  1%|          | 224/41250 [32:59<99:21:08,  8.72s/it]
                                                      
{'loss': 0.2195, 'grad_norm': 1.5721461772918701, 'learning_rate': 7.237479806138934e-06, 'epoch': 0.05}

  1%|          | 224/41250 [32:59<99:21:08,  8.72s/it][2025-04-25 08:30:42,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.18
[2025-04-25 08:30:42,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.30 | bwd_microstep: 5696.00 | bwd_inner_microstep: 5641.87 | bwd_allreduce_microstep: 54.08 | step_microstep: 19.26
[2025-04-25 08:30:42,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.30 | bwd: 5696.01 | bwd_inner: 5641.87 | bwd_allreduce: 54.10 | step: 19.26

  1%|          | 225/41250 [33:08<98:57:15,  8.68s/it]
                                                      
{'loss': 0.2099, 'grad_norm': 1.8083606958389282, 'learning_rate': 7.269789983844911e-06, 'epoch': 0.05}

  1%|          | 225/41250 [33:08<98:57:15,  8.68s/it][2025-04-25 08:30:51,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:30:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.42 | bwd_microstep: 5711.88 | bwd_inner_microstep: 5699.30 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.24
[2025-04-25 08:30:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.42 | bwd: 5711.89 | bwd_inner: 5699.30 | bwd_allreduce: 12.55 | step: 18.24

  1%|          | 226/41250 [33:16<98:48:23,  8.67s/it]
                                                      
{'loss': 0.6068, 'grad_norm': 3.468975782394409, 'learning_rate': 7.302100161550889e-06, 'epoch': 0.05}

  1%|          | 226/41250 [33:16<98:48:23,  8.67s/it][2025-04-25 08:31:00,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:31:00,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.61 | bwd_microstep: 5753.16 | bwd_inner_microstep: 5701.38 | bwd_allreduce_microstep: 51.73 | step_microstep: 18.50
[2025-04-25 08:31:00,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.61 | bwd: 5753.17 | bwd_inner: 5701.38 | bwd_allreduce: 51.75 | step: 18.50

  1%|          | 227/41250 [33:25<98:52:20,  8.68s/it]
                                                      
{'loss': 0.2442, 'grad_norm': 1.954226016998291, 'learning_rate': 7.334410339256867e-06, 'epoch': 0.06}

  1%|          | 227/41250 [33:25<98:52:20,  8.68s/it][2025-04-25 08:31:08,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:31:08,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.24 | bwd_microstep: 5745.96 | bwd_inner_microstep: 5694.84 | bwd_allreduce_microstep: 51.07 | step_microstep: 18.50
[2025-04-25 08:31:08,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.24 | bwd: 5745.97 | bwd_inner: 5694.84 | bwd_allreduce: 51.09 | step: 18.50

  1%|          | 228/41250 [33:34<98:52:45,  8.68s/it]
                                                      
{'loss': 0.1953, 'grad_norm': 1.8845200538635254, 'learning_rate': 7.366720516962844e-06, 'epoch': 0.06}

  1%|          | 228/41250 [33:34<98:52:45,  8.68s/it][2025-04-25 08:31:17,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:31:17,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.17 | bwd_microstep: 5710.04 | bwd_inner_microstep: 5697.35 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.19
[2025-04-25 08:31:17,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.17 | bwd: 5710.06 | bwd_inner: 5697.35 | bwd_allreduce: 12.66 | step: 18.20

  1%|          | 229/41250 [33:42<98:45:11,  8.67s/it]
                                                      
{'loss': 0.2165, 'grad_norm': 1.4440144300460815, 'learning_rate': 7.399030694668822e-06, 'epoch': 0.06}

  1%|          | 229/41250 [33:42<98:45:11,  8.67s/it][2025-04-25 08:31:26,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 08:31:26,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.68 | bwd_microstep: 5774.93 | bwd_inner_microstep: 5657.32 | bwd_allreduce_microstep: 117.56 | step_microstep: 18.68
[2025-04-25 08:31:26,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.68 | bwd: 5774.94 | bwd_inner: 5657.32 | bwd_allreduce: 117.58 | step: 18.68

  1%|          | 230/41250 [33:51<98:49:09,  8.67s/it]
                                                      
{'loss': 0.3062, 'grad_norm': 1.8528461456298828, 'learning_rate': 7.431340872374798e-06, 'epoch': 0.06}

  1%|          | 230/41250 [33:51<98:49:09,  8.67s/it][2025-04-25 08:31:34,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 08:31:34,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.91 | bwd_microstep: 5792.36 | bwd_inner_microstep: 5659.89 | bwd_allreduce_microstep: 132.43 | step_microstep: 18.72
[2025-04-25 08:31:34,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.91 | bwd: 5792.37 | bwd_inner: 5659.89 | bwd_allreduce: 132.45 | step: 18.72

  1%|          | 231/41250 [34:00<98:54:33,  8.68s/it]
                                                      
{'loss': 0.3305, 'grad_norm': 2.1878178119659424, 'learning_rate': 7.463651050080776e-06, 'epoch': 0.06}

  1%|          | 231/41250 [34:00<98:54:33,  8.68s/it][2025-04-25 08:31:43,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 08:31:43,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.23 | bwd_microstep: 5754.20 | bwd_inner_microstep: 5666.21 | bwd_allreduce_microstep: 87.95 | step_microstep: 18.35
[2025-04-25 08:31:43,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.23 | bwd: 5754.21 | bwd_inner: 5666.21 | bwd_allreduce: 87.97 | step: 18.35

  1%|          | 232/41250 [34:08<98:58:21,  8.69s/it]
                                                      
{'loss': 0.3551, 'grad_norm': 2.416841745376587, 'learning_rate': 7.495961227786753e-06, 'epoch': 0.06}

  1%|          | 232/41250 [34:08<98:58:21,  8.69s/it][2025-04-25 08:31:52,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.98
[2025-04-25 08:31:52,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.52 | bwd_microstep: 5758.02 | bwd_inner_microstep: 5708.14 | bwd_allreduce_microstep: 49.83 | step_microstep: 18.44
[2025-04-25 08:31:52,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.52 | bwd: 5758.03 | bwd_inner: 5708.14 | bwd_allreduce: 49.85 | step: 18.44

  1%|          | 233/41250 [34:17<99:00:19,  8.69s/it]
                                                      
{'loss': 0.3569, 'grad_norm': 2.781856060028076, 'learning_rate': 7.528271405492731e-06, 'epoch': 0.06}

  1%|          | 233/41250 [34:17<99:00:19,  8.69s/it][2025-04-25 08:32:00,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:32:00,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5803.33 | bwd_inner_microstep: 5697.75 | bwd_allreduce_microstep: 105.54 | step_microstep: 18.42
[2025-04-25 08:32:00,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5803.34 | bwd_inner: 5697.75 | bwd_allreduce: 105.56 | step: 18.43

  1%|          | 234/41250 [34:26<99:09:49,  8.70s/it]
                                                      
{'loss': 0.1923, 'grad_norm': 1.160584807395935, 'learning_rate': 7.560581583198709e-06, 'epoch': 0.06}

  1%|          | 234/41250 [34:26<99:09:49,  8.70s/it][2025-04-25 08:32:09,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:32:09,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.87 | bwd_microstep: 5715.53 | bwd_inner_microstep: 5702.73 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.51
[2025-04-25 08:32:09,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.87 | bwd: 5715.54 | bwd_inner: 5702.73 | bwd_allreduce: 12.78 | step: 18.51

  1%|          | 235/41250 [34:34<98:58:31,  8.69s/it]
                                                      
{'loss': 0.3697, 'grad_norm': 2.5747945308685303, 'learning_rate': 7.592891760904686e-06, 'epoch': 0.06}

  1%|          | 235/41250 [34:34<98:58:31,  8.69s/it][2025-04-25 08:32:18,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:32:18,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.68 | bwd_microstep: 5749.22 | bwd_inner_microstep: 5703.69 | bwd_allreduce_microstep: 45.48 | step_microstep: 18.48
[2025-04-25 08:32:18,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.68 | bwd: 5749.23 | bwd_inner: 5703.69 | bwd_allreduce: 45.50 | step: 18.48

  1%|          | 236/41250 [34:43<98:58:12,  8.69s/it]
                                                      
{'loss': 0.1901, 'grad_norm': 1.5400182008743286, 'learning_rate': 7.625201938610664e-06, 'epoch': 0.06}

  1%|          | 236/41250 [34:43<98:58:12,  8.69s/it][2025-04-25 08:32:26,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 08:32:26,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.92 | bwd_microstep: 5746.51 | bwd_inner_microstep: 5697.46 | bwd_allreduce_microstep: 49.00 | step_microstep: 18.88
[2025-04-25 08:32:26,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.92 | bwd: 5746.53 | bwd_inner: 5697.46 | bwd_allreduce: 49.02 | step: 18.88

  1%|          | 237/41250 [34:52<98:57:54,  8.69s/it]
                                                      
{'loss': 0.2459, 'grad_norm': 2.1314585208892822, 'learning_rate': 7.65751211631664e-06, 'epoch': 0.06}

  1%|          | 237/41250 [34:52<98:57:54,  8.69s/it][2025-04-25 08:32:35,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:32:35,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.56 | bwd_microstep: 5734.76 | bwd_inner_microstep: 5709.42 | bwd_allreduce_microstep: 25.29 | step_microstep: 18.48
[2025-04-25 08:32:35,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.56 | bwd: 5734.77 | bwd_inner: 5709.42 | bwd_allreduce: 25.31 | step: 18.48

  1%|          | 238/41250 [35:00<98:57:27,  8.69s/it]
                                                      
{'loss': 0.3772, 'grad_norm': 1.9182919263839722, 'learning_rate': 7.689822294022618e-06, 'epoch': 0.06}

  1%|          | 238/41250 [35:00<98:57:27,  8.69s/it][2025-04-25 08:32:44,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:32:44,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.98 | bwd_microstep: 5721.20 | bwd_inner_microstep: 5708.48 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.23
[2025-04-25 08:32:44,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.98 | bwd: 5721.21 | bwd_inner: 5708.48 | bwd_allreduce: 12.69 | step: 18.23

  1%|          | 239/41250 [35:09<98:50:39,  8.68s/it]
                                                      
{'loss': 0.3023, 'grad_norm': 1.6732912063598633, 'learning_rate': 7.722132471728596e-06, 'epoch': 0.06}

  1%|          | 239/41250 [35:09<98:50:39,  8.68s/it][2025-04-25 08:32:52,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:32:52,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5793.90 | bwd_inner_microstep: 5659.62 | bwd_allreduce_microstep: 134.24 | step_microstep: 18.35
[2025-04-25 08:32:52,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5793.92 | bwd_inner: 5659.62 | bwd_allreduce: 134.26 | step: 18.36

  1%|          | 240/41250 [35:18<98:56:15,  8.69s/it]
                                                      
{'loss': 0.2658, 'grad_norm': 2.1097795963287354, 'learning_rate': 7.754442649434572e-06, 'epoch': 0.06}

  1%|          | 240/41250 [35:18<98:56:15,  8.69s/it][2025-04-25 08:33:01,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:33:01,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.61 | bwd_microstep: 5777.26 | bwd_inner_microstep: 5695.98 | bwd_allreduce_microstep: 81.24 | step_microstep: 18.19
[2025-04-25 08:33:01,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.61 | bwd: 5777.27 | bwd_inner: 5695.98 | bwd_allreduce: 81.26 | step: 18.19

  1%|          | 241/41250 [35:27<99:03:19,  8.70s/it]
                                                      
{'loss': 0.3732, 'grad_norm': 2.542490243911743, 'learning_rate': 7.78675282714055e-06, 'epoch': 0.06}

  1%|          | 241/41250 [35:27<99:03:19,  8.70s/it][2025-04-25 08:33:10,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 08:33:10,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.45 | bwd_microstep: 5730.13 | bwd_inner_microstep: 5651.40 | bwd_allreduce_microstep: 78.68 | step_microstep: 18.30
[2025-04-25 08:33:10,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.45 | bwd: 5730.14 | bwd_inner: 5651.40 | bwd_allreduce: 78.70 | step: 18.30

  1%|          | 242/41250 [35:35<98:54:09,  8.68s/it]
                                                      
{'loss': 0.0764, 'grad_norm': 0.8372846245765686, 'learning_rate': 7.819063004846528e-06, 'epoch': 0.06}

  1%|          | 242/41250 [35:35<98:54:09,  8.68s/it][2025-04-25 08:33:19,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:33:19,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5761.13 | bwd_inner_microstep: 5711.08 | bwd_allreduce_microstep: 50.00 | step_microstep: 18.24
[2025-04-25 08:33:19,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5761.14 | bwd_inner: 5711.08 | bwd_allreduce: 50.02 | step: 18.24

  1%|          | 243/41250 [35:44<98:58:03,  8.69s/it]
                                                      
{'loss': 0.2396, 'grad_norm': 1.7706310749053955, 'learning_rate': 7.851373182552504e-06, 'epoch': 0.06}

  1%|          | 243/41250 [35:44<98:58:03,  8.69s/it][2025-04-25 08:33:27,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.20
[2025-04-25 08:33:27,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.75 | bwd_microstep: 5723.05 | bwd_inner_microstep: 5709.80 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.36
[2025-04-25 08:33:27,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.75 | bwd: 5723.07 | bwd_inner: 5709.80 | bwd_allreduce: 13.21 | step: 19.36

  1%|          | 244/41250 [35:53<98:53:04,  8.68s/it]
                                                      
{'loss': 0.1215, 'grad_norm': 1.1220550537109375, 'learning_rate': 7.883683360258482e-06, 'epoch': 0.06}

  1%|          | 244/41250 [35:53<98:53:04,  8.68s/it][2025-04-25 08:33:36,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:33:36,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.35 | bwd_microstep: 5770.65 | bwd_inner_microstep: 5717.15 | bwd_allreduce_microstep: 53.45 | step_microstep: 18.31
[2025-04-25 08:33:36,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.35 | bwd: 5770.66 | bwd_inner: 5717.15 | bwd_allreduce: 53.47 | step: 18.31

  1%|          | 245/41250 [36:01<98:59:28,  8.69s/it]
                                                      
{'loss': 0.1751, 'grad_norm': 1.3934494256973267, 'learning_rate': 7.91599353796446e-06, 'epoch': 0.06}

  1%|          | 245/41250 [36:01<98:59:28,  8.69s/it][2025-04-25 08:33:45,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 08:33:45,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.84 | bwd_microstep: 5728.45 | bwd_inner_microstep: 5715.76 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.88
[2025-04-25 08:33:45,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.84 | bwd: 5728.46 | bwd_inner: 5715.76 | bwd_allreduce: 12.66 | step: 18.88

  1%|          | 246/41250 [36:10<98:56:44,  8.69s/it]
                                                      
{'loss': 0.1951, 'grad_norm': 1.4871145486831665, 'learning_rate': 7.948303715670437e-06, 'epoch': 0.06}

  1%|          | 246/41250 [36:10<98:56:44,  8.69s/it][2025-04-25 08:33:53,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:33:53,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.07 | bwd_microstep: 5744.72 | bwd_inner_microstep: 5708.20 | bwd_allreduce_microstep: 36.48 | step_microstep: 18.45
[2025-04-25 08:33:53,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.07 | bwd: 5744.73 | bwd_inner: 5708.20 | bwd_allreduce: 36.49 | step: 18.45

  1%|          | 247/41250 [36:19<98:55:57,  8.69s/it]
                                                      
{'loss': 0.2239, 'grad_norm': 2.002671003341675, 'learning_rate': 7.980613893376414e-06, 'epoch': 0.06}

  1%|          | 247/41250 [36:19<98:55:57,  8.69s/it][2025-04-25 08:34:02,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:34:02,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.65 | bwd_microstep: 5768.43 | bwd_inner_microstep: 5703.55 | bwd_allreduce_microstep: 64.83 | step_microstep: 18.47
[2025-04-25 08:34:02,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.65 | bwd: 5768.44 | bwd_inner: 5703.55 | bwd_allreduce: 64.85 | step: 18.47

  1%|          | 248/41250 [36:27<98:58:31,  8.69s/it]
                                                      
{'loss': 0.3686, 'grad_norm': 2.6197543144226074, 'learning_rate': 8.012924071082391e-06, 'epoch': 0.06}

  1%|          | 248/41250 [36:27<98:58:31,  8.69s/it][2025-04-25 08:34:11,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 08:34:11,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.29 | bwd_microstep: 5776.38 | bwd_inner_microstep: 5763.75 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.88
[2025-04-25 08:34:11,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.29 | bwd: 5776.39 | bwd_inner: 5763.75 | bwd_allreduce: 12.60 | step: 18.88

  1%|          | 249/41250 [36:36<99:11:28,  8.71s/it]
                                                      
{'loss': 0.3061, 'grad_norm': 1.7631888389587402, 'learning_rate': 8.04523424878837e-06, 'epoch': 0.06}

  1%|          | 249/41250 [36:36<99:11:28,  8.71s/it][2025-04-25 08:34:19,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:34:19,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.19 | bwd_microstep: 5751.08 | bwd_inner_microstep: 5717.02 | bwd_allreduce_microstep: 34.02 | step_microstep: 18.15
[2025-04-25 08:34:19,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.19 | bwd: 5751.09 | bwd_inner: 5717.02 | bwd_allreduce: 34.04 | step: 18.15

  1%|          | 250/41250 [36:45<99:09:27,  8.71s/it]
                                                      
{'loss': 0.3087, 'grad_norm': 2.7893669605255127, 'learning_rate': 8.077544426494346e-06, 'epoch': 0.06}

  1%|          | 250/41250 [36:45<99:09:27,  8.71s/it][2025-04-25 08:34:28,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:34:28,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.33 | bwd_microstep: 5702.15 | bwd_inner_microstep: 5689.39 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.06
[2025-04-25 08:34:28,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.33 | bwd: 5702.16 | bwd_inner: 5689.39 | bwd_allreduce: 12.73 | step: 18.06

  1%|          | 251/41250 [36:53<98:55:25,  8.69s/it]
                                                      
{'loss': 0.179, 'grad_norm': 1.6316828727722168, 'learning_rate': 8.109854604200323e-06, 'epoch': 0.06}

  1%|          | 251/41250 [36:53<98:55:25,  8.69s/it][2025-04-25 08:34:37,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.94 | optimizer_step: 1.07
[2025-04-25 08:34:37,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.95 | bwd_microstep: 5717.25 | bwd_inner_microstep: 5695.25 | bwd_allreduce_microstep: 21.96 | step_microstep: 18.29
[2025-04-25 08:34:37,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.95 | bwd: 5717.26 | bwd_inner: 5695.25 | bwd_allreduce: 21.98 | step: 18.29

  1%|          | 252/41250 [37:02<98:49:31,  8.68s/it]
                                                      
{'loss': 0.3129, 'grad_norm': 2.7633490562438965, 'learning_rate': 8.142164781906301e-06, 'epoch': 0.06}

  1%|          | 252/41250 [37:02<98:49:31,  8.68s/it][2025-04-25 08:34:45,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:34:45,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.79 | bwd_microstep: 5802.22 | bwd_inner_microstep: 5640.24 | bwd_allreduce_microstep: 161.94 | step_microstep: 18.26
[2025-04-25 08:34:45,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5802.23 | bwd_inner: 5640.24 | bwd_allreduce: 161.96 | step: 18.26

  1%|          | 253/41250 [37:11<98:57:28,  8.69s/it]
                                                      
{'loss': 0.2726, 'grad_norm': 2.2230560779571533, 'learning_rate': 8.17447495961228e-06, 'epoch': 0.06}

  1%|          | 253/41250 [37:11<98:57:28,  8.69s/it][2025-04-25 08:34:54,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 08:34:54,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.64 | bwd_microstep: 5744.96 | bwd_inner_microstep: 5678.05 | bwd_allreduce_microstep: 66.87 | step_microstep: 18.70
[2025-04-25 08:34:54,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.65 | bwd: 5744.97 | bwd_inner: 5678.05 | bwd_allreduce: 66.88 | step: 18.70

  1%|          | 254/41250 [37:19<98:53:46,  8.68s/it]
                                                      
{'loss': 0.1928, 'grad_norm': 1.6533745527267456, 'learning_rate': 8.206785137318255e-06, 'epoch': 0.06}

  1%|          | 254/41250 [37:19<98:53:46,  8.68s/it][2025-04-25 08:35:03,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:35:03,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.09 | bwd_microstep: 5760.73 | bwd_inner_microstep: 5662.61 | bwd_allreduce_microstep: 98.08 | step_microstep: 17.89
[2025-04-25 08:35:03,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.09 | bwd: 5760.75 | bwd_inner: 5662.61 | bwd_allreduce: 98.10 | step: 17.89

  1%|          | 255/41250 [37:28<98:53:04,  8.68s/it]
                                                      
{'loss': 0.29, 'grad_norm': 1.916360855102539, 'learning_rate': 8.239095315024233e-06, 'epoch': 0.06}

  1%|          | 255/41250 [37:28<98:53:04,  8.68s/it][2025-04-25 08:35:12,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:35:12,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.65 | bwd_microstep: 5775.88 | bwd_inner_microstep: 5763.13 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.38
[2025-04-25 08:35:12,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.65 | bwd: 5775.89 | bwd_inner: 5763.13 | bwd_allreduce: 12.72 | step: 18.39

  1%|          | 256/41250 [37:37<99:05:07,  8.70s/it]
                                                      
{'loss': 0.1884, 'grad_norm': 1.620060920715332, 'learning_rate': 8.271405492730211e-06, 'epoch': 0.06}

  1%|          | 256/41250 [37:37<99:05:07,  8.70s/it][2025-04-25 08:35:20,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:35:20,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5741.36 | bwd_inner_microstep: 5693.08 | bwd_allreduce_microstep: 48.23 | step_microstep: 18.10
[2025-04-25 08:35:20,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5741.37 | bwd_inner: 5693.08 | bwd_allreduce: 48.25 | step: 18.10

  1%|          | 257/41250 [37:46<99:00:02,  8.69s/it]
                                                      
{'loss': 0.1689, 'grad_norm': 1.7414363622665405, 'learning_rate': 8.303715670436187e-06, 'epoch': 0.06}

  1%|          | 257/41250 [37:46<99:00:02,  8.69s/it][2025-04-25 08:35:29,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:35:29,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5705.43 | bwd_inner_microstep: 5654.76 | bwd_allreduce_microstep: 50.62 | step_microstep: 18.00
[2025-04-25 08:35:29,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5705.44 | bwd_inner: 5654.76 | bwd_allreduce: 50.64 | step: 18.00

  1%|          | 258/41250 [37:54<98:44:15,  8.67s/it]
                                                      
{'loss': 0.1751, 'grad_norm': 1.536134958267212, 'learning_rate': 8.336025848142165e-06, 'epoch': 0.06}

  1%|          | 258/41250 [37:54<98:44:15,  8.67s/it][2025-04-25 08:35:38,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 08:35:38,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.90 | bwd_microstep: 5736.21 | bwd_inner_microstep: 5683.08 | bwd_allreduce_microstep: 53.09 | step_microstep: 17.94
[2025-04-25 08:35:38,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.90 | bwd: 5736.22 | bwd_inner: 5683.08 | bwd_allreduce: 53.10 | step: 17.94

  1%|          | 259/41250 [38:03<98:45:26,  8.67s/it]
                                                      
{'loss': 0.0783, 'grad_norm': 1.30600905418396, 'learning_rate': 8.368336025848143e-06, 'epoch': 0.06}

  1%|          | 259/41250 [38:03<98:45:26,  8.67s/it][2025-04-25 08:35:46,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:35:46,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5699.65 | bwd_inner_microstep: 5646.92 | bwd_allreduce_microstep: 52.69 | step_microstep: 18.45
[2025-04-25 08:35:46,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.75 | bwd: 5699.67 | bwd_inner: 5646.92 | bwd_allreduce: 52.71 | step: 18.45

  1%|          | 260/41250 [38:11<98:32:06,  8.65s/it]
                                                      
{'loss': 0.2328, 'grad_norm': 2.113016128540039, 'learning_rate': 8.400646203554121e-06, 'epoch': 0.06}

  1%|          | 260/41250 [38:11<98:32:06,  8.65s/it][2025-04-25 08:35:55,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.84 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:35:55,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.90 | bwd_microstep: 5745.74 | bwd_inner_microstep: 5675.31 | bwd_allreduce_microstep: 70.38 | step_microstep: 17.75
[2025-04-25 08:35:55,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.90 | bwd: 5745.75 | bwd_inner: 5675.31 | bwd_allreduce: 70.40 | step: 17.75

  1%|          | 261/41250 [38:20<98:36:37,  8.66s/it]
                                                      
{'loss': 0.1766, 'grad_norm': 1.4529616832733154, 'learning_rate': 8.432956381260097e-06, 'epoch': 0.06}

  1%|          | 261/41250 [38:20<98:36:37,  8.66s/it][2025-04-25 08:36:03,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 08:36:03,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5705.68 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.53
[2025-04-25 08:36:03,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5705.70 | bwd_inner: 5692.86 | bwd_allreduce: 12.80 | step: 18.54

  1%|          | 262/41250 [38:29<98:32:57,  8.66s/it]
                                                      
{'loss': 0.3037, 'grad_norm': 4.187695026397705, 'learning_rate': 8.465266558966075e-06, 'epoch': 0.06}

  1%|          | 262/41250 [38:29<98:32:57,  8.66s/it][2025-04-25 08:36:12,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:36:12,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.21 | bwd_microstep: 5751.62 | bwd_inner_microstep: 5677.74 | bwd_allreduce_microstep: 73.85 | step_microstep: 18.50
[2025-04-25 08:36:12,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.21 | bwd: 5751.63 | bwd_inner: 5677.74 | bwd_allreduce: 73.86 | step: 18.50

  1%|          | 263/41250 [38:37<98:39:04,  8.66s/it]
                                                      
{'loss': 0.3264, 'grad_norm': 3.0465848445892334, 'learning_rate': 8.497576736672051e-06, 'epoch': 0.06}

  1%|          | 263/41250 [38:37<98:39:04,  8.66s/it][2025-04-25 08:36:21,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:36:21,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.32 | bwd_microstep: 5690.79 | bwd_inner_microstep: 5678.10 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.34
[2025-04-25 08:36:21,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.32 | bwd: 5690.80 | bwd_inner: 5678.10 | bwd_allreduce: 12.66 | step: 18.34

  1%|          | 264/41250 [38:46<98:30:02,  8.65s/it]
                                                      
{'loss': 0.2332, 'grad_norm': 3.6961328983306885, 'learning_rate': 8.529886914378029e-06, 'epoch': 0.06}

  1%|          | 264/41250 [38:46<98:30:02,  8.65s/it][2025-04-25 08:36:29,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:36:29,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5699.01 | bwd_inner_microstep: 5643.01 | bwd_allreduce_microstep: 55.96 | step_microstep: 18.23
[2025-04-25 08:36:29,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5699.03 | bwd_inner: 5643.01 | bwd_allreduce: 55.97 | step: 18.23

  1%|          | 265/41250 [38:55<98:21:59,  8.64s/it]
                                                      
{'loss': 0.3011, 'grad_norm': 2.198544502258301, 'learning_rate': 8.562197092084007e-06, 'epoch': 0.06}

  1%|          | 265/41250 [38:55<98:21:59,  8.64s/it][2025-04-25 08:36:38,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:36:38,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5757.01 | bwd_inner_microstep: 5689.27 | bwd_allreduce_microstep: 67.70 | step_microstep: 18.36
[2025-04-25 08:36:38,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5757.02 | bwd_inner: 5689.27 | bwd_allreduce: 67.71 | step: 18.36

  1%|          | 266/41250 [39:03<98:32:15,  8.66s/it]
                                                      
{'loss': 0.2954, 'grad_norm': 2.495978832244873, 'learning_rate': 8.594507269789985e-06, 'epoch': 0.06}

  1%|          | 266/41250 [39:03<98:32:15,  8.66s/it][2025-04-25 08:36:47,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.95 | optimizer_step: 0.98
[2025-04-25 08:36:47,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.64 | bwd_microstep: 5772.53 | bwd_inner_microstep: 5641.92 | bwd_allreduce_microstep: 130.56 | step_microstep: 18.00
[2025-04-25 08:36:47,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5772.54 | bwd_inner: 5641.92 | bwd_allreduce: 130.58 | step: 18.00

  1%|          | 267/41250 [39:12<98:37:53,  8.66s/it]
                                                      
{'loss': 0.1972, 'grad_norm': 2.6422014236450195, 'learning_rate': 8.626817447495963e-06, 'epoch': 0.06}

  1%|          | 267/41250 [39:12<98:37:53,  8.66s/it][2025-04-25 08:36:55,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:36:55,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.96 | bwd_microstep: 5734.74 | bwd_inner_microstep: 5698.10 | bwd_allreduce_microstep: 36.60 | step_microstep: 18.49
[2025-04-25 08:36:55,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.96 | bwd: 5734.75 | bwd_inner: 5698.10 | bwd_allreduce: 36.62 | step: 18.49

  1%|          | 268/41250 [39:21<98:38:27,  8.66s/it]
                                                      
{'loss': 0.129, 'grad_norm': 1.5052878856658936, 'learning_rate': 8.659127625201939e-06, 'epoch': 0.06}

  1%|          | 268/41250 [39:21<98:38:27,  8.66s/it][2025-04-25 08:37:04,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 08:37:04,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5743.97 | bwd_inner_microstep: 5673.35 | bwd_allreduce_microstep: 70.58 | step_microstep: 18.61
[2025-04-25 08:37:04,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5743.99 | bwd_inner: 5673.35 | bwd_allreduce: 70.60 | step: 18.61

  1%|          | 269/41250 [39:29<98:40:46,  8.67s/it]
                                                      
{'loss': 0.2574, 'grad_norm': 2.035139799118042, 'learning_rate': 8.691437802907917e-06, 'epoch': 0.07}

  1%|          | 269/41250 [39:29<98:40:46,  8.67s/it][2025-04-25 08:37:13,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:37:13,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.64 | bwd_microstep: 5732.06 | bwd_inner_microstep: 5637.49 | bwd_allreduce_microstep: 94.53 | step_microstep: 18.23
[2025-04-25 08:37:13,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.64 | bwd: 5732.07 | bwd_inner: 5637.49 | bwd_allreduce: 94.54 | step: 18.23

  1%|          | 270/41250 [39:38<98:35:31,  8.66s/it]
                                                      
{'loss': 0.36, 'grad_norm': 2.6091108322143555, 'learning_rate': 8.723747980613893e-06, 'epoch': 0.07}

  1%|          | 270/41250 [39:38<98:35:31,  8.66s/it][2025-04-25 08:37:21,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:37:21,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5694.96 | bwd_inner_microstep: 5628.67 | bwd_allreduce_microstep: 66.25 | step_microstep: 17.91
[2025-04-25 08:37:21,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5694.97 | bwd_inner: 5628.67 | bwd_allreduce: 66.26 | step: 17.92

  1%|          | 271/41250 [39:47<98:25:20,  8.65s/it]
                                                      
{'loss': 0.1493, 'grad_norm': 1.9501588344573975, 'learning_rate': 8.756058158319871e-06, 'epoch': 0.07}

  1%|          | 271/41250 [39:47<98:25:20,  8.65s/it][2025-04-25 08:37:30,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:37:30,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.72 | bwd_microstep: 5775.42 | bwd_inner_microstep: 5639.45 | bwd_allreduce_microstep: 135.93 | step_microstep: 18.05
[2025-04-25 08:37:30,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.72 | bwd: 5775.43 | bwd_inner: 5639.45 | bwd_allreduce: 135.94 | step: 18.06

  1%|          | 272/41250 [39:55<98:31:39,  8.66s/it]
                                                      
{'loss': 0.0727, 'grad_norm': 0.904013991355896, 'learning_rate': 8.788368336025849e-06, 'epoch': 0.07}

  1%|          | 272/41250 [39:55<98:31:39,  8.66s/it][2025-04-25 08:37:39,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:37:39,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5728.63 | bwd_inner_microstep: 5691.12 | bwd_allreduce_microstep: 37.47 | step_microstep: 18.26
[2025-04-25 08:37:39,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5728.65 | bwd_inner: 5691.12 | bwd_allreduce: 37.48 | step: 18.26

  1%|          | 273/41250 [40:04<98:33:42,  8.66s/it]
                                                      
{'loss': 0.3058, 'grad_norm': 2.245638608932495, 'learning_rate': 8.820678513731827e-06, 'epoch': 0.07}

  1%|          | 273/41250 [40:04<98:33:42,  8.66s/it][2025-04-25 08:37:47,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:37:47,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.01 | bwd_microstep: 5759.52 | bwd_inner_microstep: 5652.52 | bwd_allreduce_microstep: 106.96 | step_microstep: 17.97
[2025-04-25 08:37:47,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.01 | bwd: 5759.53 | bwd_inner: 5652.52 | bwd_allreduce: 106.97 | step: 17.97

  1%|          | 274/41250 [40:13<98:38:34,  8.67s/it]
                                                      
{'loss': 0.2939, 'grad_norm': 3.8655683994293213, 'learning_rate': 8.852988691437805e-06, 'epoch': 0.07}

  1%|          | 274/41250 [40:13<98:38:34,  8.67s/it][2025-04-25 08:37:56,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.94 | optimizer_step: 0.95
[2025-04-25 08:37:56,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.15 | bwd_microstep: 5740.76 | bwd_inner_microstep: 5672.54 | bwd_allreduce_microstep: 68.18 | step_microstep: 18.16
[2025-04-25 08:37:56,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.15 | bwd: 5740.77 | bwd_inner: 5672.54 | bwd_allreduce: 68.19 | step: 18.16

  1%|          | 275/41250 [40:21<98:37:20,  8.66s/it]
                                                      
{'loss': 0.2524, 'grad_norm': 2.928022623062134, 'learning_rate': 8.88529886914378e-06, 'epoch': 0.07}

  1%|          | 275/41250 [40:21<98:37:20,  8.66s/it][2025-04-25 08:38:05,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:38:05,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.43 | bwd_microstep: 5732.85 | bwd_inner_microstep: 5650.28 | bwd_allreduce_microstep: 82.53 | step_microstep: 17.99
[2025-04-25 08:38:05,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.43 | bwd: 5732.87 | bwd_inner: 5650.28 | bwd_allreduce: 82.54 | step: 17.99

  1%|          | 276/41250 [40:30<98:32:59,  8.66s/it]
                                                      
{'loss': 0.2924, 'grad_norm': 4.588143825531006, 'learning_rate': 8.917609046849759e-06, 'epoch': 0.07}

  1%|          | 276/41250 [40:30<98:32:59,  8.66s/it][2025-04-25 08:38:13,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:38:13,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 5690.27 | bwd_inner_microstep: 5677.63 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.35
[2025-04-25 08:38:13,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 5690.29 | bwd_inner: 5677.63 | bwd_allreduce: 12.62 | step: 18.35

  1%|          | 277/41250 [40:39<98:25:14,  8.65s/it]
                                                      
{'loss': 0.1671, 'grad_norm': 2.516690731048584, 'learning_rate': 8.949919224555735e-06, 'epoch': 0.07}

  1%|          | 277/41250 [40:39<98:25:14,  8.65s/it][2025-04-25 08:38:22,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-25 08:38:22,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.03 | bwd_microstep: 5730.46 | bwd_inner_microstep: 5648.88 | bwd_allreduce_microstep: 81.53 | step_microstep: 18.66
[2025-04-25 08:38:22,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.03 | bwd: 5730.47 | bwd_inner: 5648.88 | bwd_allreduce: 81.55 | step: 18.66

  1%|          | 278/41250 [40:47<98:25:49,  8.65s/it]
                                                      
{'loss': 0.2685, 'grad_norm': 2.260939359664917, 'learning_rate': 8.982229402261713e-06, 'epoch': 0.07}

  1%|          | 278/41250 [40:47<98:25:49,  8.65s/it][2025-04-25 08:38:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:38:31,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5705.67 | bwd_inner_microstep: 5693.04 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.22
[2025-04-25 08:38:31,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5705.68 | bwd_inner: 5693.03 | bwd_allreduce: 12.60 | step: 18.23

  1%|          | 279/41250 [40:56<98:23:39,  8.65s/it]
                                                      
{'loss': 0.2301, 'grad_norm': 2.0884525775909424, 'learning_rate': 9.01453957996769e-06, 'epoch': 0.07}

  1%|          | 279/41250 [40:56<98:23:39,  8.65s/it][2025-04-25 08:38:39,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:38:39,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.03 | bwd_microstep: 5720.56 | bwd_inner_microstep: 5683.45 | bwd_allreduce_microstep: 37.06 | step_microstep: 18.06
[2025-04-25 08:38:39,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.03 | bwd: 5720.57 | bwd_inner: 5683.45 | bwd_allreduce: 37.08 | step: 18.07

  1%|          | 280/41250 [41:05<98:25:51,  8.65s/it]
                                                      
{'loss': 0.324, 'grad_norm': 2.5513992309570312, 'learning_rate': 9.046849757673668e-06, 'epoch': 0.07}

  1%|          | 280/41250 [41:05<98:25:51,  8.65s/it][2025-04-25 08:38:48,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:38:48,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.75 | bwd_microstep: 5763.17 | bwd_inner_microstep: 5651.98 | bwd_allreduce_microstep: 111.14 | step_microstep: 18.88
[2025-04-25 08:38:48,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.75 | bwd: 5763.18 | bwd_inner: 5651.98 | bwd_allreduce: 111.16 | step: 18.88

  1%|          | 281/41250 [41:13<98:33:29,  8.66s/it]
                                                      
{'loss': 0.2621, 'grad_norm': 2.3424322605133057, 'learning_rate': 9.079159935379646e-06, 'epoch': 0.07}

  1%|          | 281/41250 [41:13<98:33:29,  8.66s/it][2025-04-25 08:38:57,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:38:57,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.90 | bwd_microstep: 5711.78 | bwd_inner_microstep: 5686.63 | bwd_allreduce_microstep: 25.10 | step_microstep: 18.55
[2025-04-25 08:38:57,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.90 | bwd: 5711.79 | bwd_inner: 5686.63 | bwd_allreduce: 25.12 | step: 18.55

  1%|          | 282/41250 [41:22<98:33:05,  8.66s/it]
                                                      
{'loss': 0.1555, 'grad_norm': 1.2164027690887451, 'learning_rate': 9.111470113085623e-06, 'epoch': 0.07}

  1%|          | 282/41250 [41:22<98:33:05,  8.66s/it][2025-04-25 08:39:05,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:39:05,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.10 | bwd_microstep: 5695.40 | bwd_inner_microstep: 5682.70 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.42
[2025-04-25 08:39:05,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.10 | bwd: 5695.41 | bwd_inner: 5682.70 | bwd_allreduce: 12.67 | step: 18.43

  1%|          | 283/41250 [41:31<98:26:12,  8.65s/it]
                                                      
{'loss': 0.3526, 'grad_norm': 3.420158863067627, 'learning_rate': 9.143780290791599e-06, 'epoch': 0.07}

  1%|          | 283/41250 [41:31<98:26:12,  8.65s/it][2025-04-25 08:39:14,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 08:39:14,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.31 | bwd_microstep: 5755.36 | bwd_inner_microstep: 5678.92 | bwd_allreduce_microstep: 76.40 | step_microstep: 18.88
[2025-04-25 08:39:14,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.31 | bwd: 5755.37 | bwd_inner: 5678.92 | bwd_allreduce: 76.42 | step: 18.88

  1%|          | 284/41250 [41:39<98:33:32,  8.66s/it]
                                                      
{'loss': 0.1893, 'grad_norm': 1.3922197818756104, 'learning_rate': 9.176090468497577e-06, 'epoch': 0.07}

  1%|          | 284/41250 [41:39<98:33:32,  8.66s/it][2025-04-25 08:39:23,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:39:23,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.99 | bwd_microstep: 5672.74 | bwd_inner_microstep: 5653.02 | bwd_allreduce_microstep: 19.68 | step_microstep: 18.81
[2025-04-25 08:39:23,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.99 | bwd: 5672.75 | bwd_inner: 5653.01 | bwd_allreduce: 19.70 | step: 18.81

  1%|          | 285/41250 [41:48<98:19:38,  8.64s/it]
                                                      
{'loss': 0.2336, 'grad_norm': 2.454723358154297, 'learning_rate': 9.208400646203555e-06, 'epoch': 0.07}

  1%|          | 285/41250 [41:48<98:19:38,  8.64s/it][2025-04-25 08:39:31,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:39:31,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.16 | bwd_microstep: 5734.75 | bwd_inner_microstep: 5694.14 | bwd_allreduce_microstep: 40.56 | step_microstep: 18.30
[2025-04-25 08:39:31,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.16 | bwd: 5734.77 | bwd_inner: 5694.14 | bwd_allreduce: 40.58 | step: 18.31

  1%|          | 286/41250 [41:57<98:26:54,  8.65s/it]
                                                      
{'loss': 0.1133, 'grad_norm': 1.2954466342926025, 'learning_rate': 9.240710823909532e-06, 'epoch': 0.07}

  1%|          | 286/41250 [41:57<98:26:54,  8.65s/it][2025-04-25 08:39:40,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 08:39:40,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5738.97 | bwd_inner_microstep: 5699.52 | bwd_allreduce_microstep: 39.39 | step_microstep: 18.81
[2025-04-25 08:39:40,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.07 | bwd: 5738.98 | bwd_inner: 5699.52 | bwd_allreduce: 39.41 | step: 18.81

  1%|          | 287/41250 [42:05<98:31:23,  8.66s/it]
                                                      
{'loss': 0.2727, 'grad_norm': 3.664973020553589, 'learning_rate': 9.27302100161551e-06, 'epoch': 0.07}

  1%|          | 287/41250 [42:05<98:31:23,  8.66s/it][2025-04-25 08:39:49,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 08:39:49,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5787.02 | bwd_inner_microstep: 5662.22 | bwd_allreduce_microstep: 124.74 | step_microstep: 19.13
[2025-04-25 08:39:49,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5787.03 | bwd_inner: 5662.22 | bwd_allreduce: 124.77 | step: 19.13

  1%|          | 288/41250 [42:14<98:40:01,  8.67s/it]
                                                      
{'loss': 0.234, 'grad_norm': 1.606065273284912, 'learning_rate': 9.305331179321486e-06, 'epoch': 0.07}

  1%|          | 288/41250 [42:14<98:40:01,  8.67s/it][2025-04-25 08:39:57,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 08:39:57,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.37 | bwd_microstep: 5727.19 | bwd_inner_microstep: 5656.08 | bwd_allreduce_microstep: 71.07 | step_microstep: 18.53
[2025-04-25 08:39:57,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.37 | bwd: 5727.21 | bwd_inner: 5656.08 | bwd_allreduce: 71.09 | step: 18.53

  1%|          | 289/41250 [42:23<98:33:49,  8.66s/it]
                                                      
{'loss': 0.3322, 'grad_norm': 2.7249722480773926, 'learning_rate': 9.337641357027464e-06, 'epoch': 0.07}

  1%|          | 289/41250 [42:23<98:33:49,  8.66s/it][2025-04-25 08:40:06,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:40:06,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.62 | bwd_microstep: 5773.68 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 128.30 | step_microstep: 18.35
[2025-04-25 08:40:06,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.62 | bwd: 5773.69 | bwd_inner: 5645.34 | bwd_allreduce: 128.32 | step: 18.35

  1%|          | 290/41250 [42:31<98:39:06,  8.67s/it]
                                                      
{'loss': 0.1991, 'grad_norm': 1.8206764459609985, 'learning_rate': 9.36995153473344e-06, 'epoch': 0.07}

  1%|          | 290/41250 [42:31<98:39:06,  8.67s/it][2025-04-25 08:40:15,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:40:15,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.89 | bwd_microstep: 5721.03 | bwd_inner_microstep: 5708.29 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.31
[2025-04-25 08:40:15,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.89 | bwd: 5721.04 | bwd_inner: 5708.29 | bwd_allreduce: 12.72 | step: 18.32

  1%|          | 291/41250 [42:40<98:37:44,  8.67s/it]
                                                      
{'loss': 0.2017, 'grad_norm': 1.602535605430603, 'learning_rate': 9.402261712439418e-06, 'epoch': 0.07}

  1%|          | 291/41250 [42:40<98:37:44,  8.67s/it][2025-04-25 08:40:23,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:40:23,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.28 | bwd_microstep: 5769.49 | bwd_inner_microstep: 5703.92 | bwd_allreduce_microstep: 65.53 | step_microstep: 18.59
[2025-04-25 08:40:23,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5769.51 | bwd_inner: 5703.92 | bwd_allreduce: 65.55 | step: 18.59

  1%|          | 292/41250 [42:49<98:43:40,  8.68s/it]
                                                      
{'loss': 0.4105, 'grad_norm': 3.181896209716797, 'learning_rate': 9.434571890145396e-06, 'epoch': 0.07}

  1%|          | 292/41250 [42:49<98:43:40,  8.68s/it][2025-04-25 08:40:32,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 1.16
[2025-04-25 08:40:32,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.20 | bwd_microstep: 5715.85 | bwd_inner_microstep: 5667.76 | bwd_allreduce_microstep: 48.05 | step_microstep: 18.87
[2025-04-25 08:40:32,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.20 | bwd: 5715.86 | bwd_inner: 5667.76 | bwd_allreduce: 48.06 | step: 18.87

  1%|          | 293/41250 [42:57<98:38:07,  8.67s/it]
                                                      
{'loss': 0.1816, 'grad_norm': 1.8712999820709229, 'learning_rate': 9.466882067851374e-06, 'epoch': 0.07}

  1%|          | 293/41250 [42:57<98:38:07,  8.67s/it][2025-04-25 08:40:41,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:40:41,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.59 | bwd_microstep: 5787.38 | bwd_inner_microstep: 5660.70 | bwd_allreduce_microstep: 126.63 | step_microstep: 18.51
[2025-04-25 08:40:41,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.60 | bwd: 5787.39 | bwd_inner: 5660.70 | bwd_allreduce: 126.65 | step: 18.51

  1%|          | 294/41250 [43:06<98:45:53,  8.68s/it]
                                                      
{'loss': 0.1479, 'grad_norm': 1.4222699403762817, 'learning_rate': 9.499192245557352e-06, 'epoch': 0.07}

  1%|          | 294/41250 [43:06<98:45:53,  8.68s/it][2025-04-25 08:40:49,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:40:49,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5796.24 | bwd_inner_microstep: 5699.00 | bwd_allreduce_microstep: 97.19 | step_microstep: 18.46
[2025-04-25 08:40:49,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5796.26 | bwd_inner: 5699.00 | bwd_allreduce: 97.21 | step: 18.47

  1%|          | 295/41250 [43:15<98:55:11,  8.70s/it]
                                                      
{'loss': 0.1145, 'grad_norm': 1.3910000324249268, 'learning_rate': 9.531502423263328e-06, 'epoch': 0.07}

  1%|          | 295/41250 [43:15<98:55:11,  8.70s/it][2025-04-25 08:40:58,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:40:58,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.57 | bwd_microstep: 5728.58 | bwd_inner_microstep: 5682.46 | bwd_allreduce_microstep: 46.07 | step_microstep: 18.61
[2025-04-25 08:40:58,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.57 | bwd: 5728.59 | bwd_inner: 5682.46 | bwd_allreduce: 46.09 | step: 18.61

  1%|          | 296/41250 [43:23<98:47:32,  8.68s/it]
                                                      
{'loss': 0.1578, 'grad_norm': 1.2876454591751099, 'learning_rate': 9.563812600969306e-06, 'epoch': 0.07}

  1%|          | 296/41250 [43:23<98:47:32,  8.68s/it][2025-04-25 08:41:07,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:41:07,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.93 | bwd_microstep: 5719.10 | bwd_inner_microstep: 5651.26 | bwd_allreduce_microstep: 67.80 | step_microstep: 17.69
[2025-04-25 08:41:07,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.93 | bwd: 5719.11 | bwd_inner: 5651.26 | bwd_allreduce: 67.81 | step: 17.69

  1%|          | 297/41250 [43:32<98:38:12,  8.67s/it]
                                                      
{'loss': 0.3676, 'grad_norm': 3.8930094242095947, 'learning_rate': 9.596122778675282e-06, 'epoch': 0.07}

  1%|          | 297/41250 [43:32<98:38:12,  8.67s/it][2025-04-25 08:41:15,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:41:15,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.94 | bwd_microstep: 5727.32 | bwd_inner_microstep: 5714.74 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.20
[2025-04-25 08:41:15,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.94 | bwd: 5727.34 | bwd_inner: 5714.74 | bwd_allreduce: 12.56 | step: 18.20

  1%|          | 298/41250 [43:41<98:37:29,  8.67s/it]
                                                      
{'loss': 0.252, 'grad_norm': 2.3680317401885986, 'learning_rate': 9.62843295638126e-06, 'epoch': 0.07}

  1%|          | 298/41250 [43:41<98:37:29,  8.67s/it][2025-04-25 08:41:24,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 08:41:24,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.54 | bwd_microstep: 5713.64 | bwd_inner_microstep: 5701.09 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.81
[2025-04-25 08:41:24,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.54 | bwd: 5713.65 | bwd_inner: 5701.09 | bwd_allreduce: 12.53 | step: 18.81

  1%|          | 299/41250 [43:49<98:31:37,  8.66s/it]
                                                      
{'loss': 0.1883, 'grad_norm': 2.7250137329101562, 'learning_rate': 9.660743134087238e-06, 'epoch': 0.07}

  1%|          | 299/41250 [43:49<98:31:37,  8.66s/it][2025-04-25 08:41:33,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:41:33,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5725.58 | bwd_inner_microstep: 5657.14 | bwd_allreduce_microstep: 68.39 | step_microstep: 17.96
[2025-04-25 08:41:33,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5725.59 | bwd_inner: 5657.14 | bwd_allreduce: 68.41 | step: 17.96

  1%|          | 300/41250 [43:58<98:26:15,  8.65s/it]
                                                      
{'loss': 0.152, 'grad_norm': 1.9413031339645386, 'learning_rate': 9.693053311793216e-06, 'epoch': 0.07}

  1%|          | 300/41250 [43:58<98:26:15,  8.65s/it][2025-04-25 08:41:41,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:41:41,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.00 | bwd_microstep: 5788.09 | bwd_inner_microstep: 5659.40 | bwd_allreduce_microstep: 128.65 | step_microstep: 18.19
[2025-04-25 08:41:41,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.00 | bwd: 5788.11 | bwd_inner: 5659.40 | bwd_allreduce: 128.67 | step: 18.19

  1%|          | 301/41250 [44:07<98:37:49,  8.67s/it]
                                                      
{'loss': 0.2595, 'grad_norm': 1.9450353384017944, 'learning_rate': 9.725363489499194e-06, 'epoch': 0.07}

  1%|          | 301/41250 [44:07<98:37:49,  8.67s/it][2025-04-25 08:41:50,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 08:41:50,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.80 | bwd_microstep: 5797.07 | bwd_inner_microstep: 5658.45 | bwd_allreduce_microstep: 138.58 | step_microstep: 18.41
[2025-04-25 08:41:50,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.80 | bwd: 5797.08 | bwd_inner: 5658.45 | bwd_allreduce: 138.60 | step: 18.42

  1%|          | 302/41250 [44:15<98:45:31,  8.68s/it]
                                                      
{'loss': 0.2365, 'grad_norm': 3.872612476348877, 'learning_rate': 9.75767366720517e-06, 'epoch': 0.07}

  1%|          | 302/41250 [44:15<98:45:31,  8.68s/it][2025-04-25 08:41:59,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 08:41:59,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.81 | bwd_microstep: 5719.14 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 58.16 | step_microstep: 18.21
[2025-04-25 08:41:59,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.81 | bwd: 5719.15 | bwd_inner: 5660.93 | bwd_allreduce: 58.18 | step: 18.21

  1%|          | 303/41250 [44:24<98:37:41,  8.67s/it]
                                                      
{'loss': 0.1139, 'grad_norm': 1.337911605834961, 'learning_rate': 9.789983844911148e-06, 'epoch': 0.07}

  1%|          | 303/41250 [44:24<98:37:41,  8.67s/it][2025-04-25 08:42:07,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:42:07,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.29 | bwd_microstep: 5770.42 | bwd_inner_microstep: 5653.23 | bwd_allreduce_microstep: 117.15 | step_microstep: 18.04
[2025-04-25 08:42:07,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.29 | bwd: 5770.44 | bwd_inner: 5653.23 | bwd_allreduce: 117.17 | step: 18.04

  1%|          | 304/41250 [44:33<98:44:11,  8.68s/it]
                                                      
{'loss': 0.1687, 'grad_norm': 1.6806294918060303, 'learning_rate': 9.822294022617124e-06, 'epoch': 0.07}

  1%|          | 304/41250 [44:33<98:44:11,  8.68s/it][2025-04-25 08:42:16,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 08:42:16,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.65 | bwd_microstep: 5779.46 | bwd_inner_microstep: 5664.81 | bwd_allreduce_microstep: 114.60 | step_microstep: 19.35
[2025-04-25 08:42:16,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.65 | bwd: 5779.47 | bwd_inner: 5664.81 | bwd_allreduce: 114.62 | step: 19.35

  1%|          | 305/41250 [44:41<98:47:34,  8.69s/it]
                                                      
{'loss': 0.2397, 'grad_norm': 3.6594831943511963, 'learning_rate': 9.854604200323102e-06, 'epoch': 0.07}

  1%|          | 305/41250 [44:41<98:47:34,  8.69s/it][2025-04-25 08:42:25,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:42:25,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.03 | bwd_microstep: 5701.50 | bwd_inner_microstep: 5653.37 | bwd_allreduce_microstep: 48.09 | step_microstep: 18.47
[2025-04-25 08:42:25,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.03 | bwd: 5701.51 | bwd_inner: 5653.37 | bwd_allreduce: 48.10 | step: 18.48

  1%|          | 306/41250 [44:50<98:32:22,  8.66s/it]
                                                      
{'loss': 0.2508, 'grad_norm': 2.929117202758789, 'learning_rate': 9.88691437802908e-06, 'epoch': 0.07}

  1%|          | 306/41250 [44:50<98:32:22,  8.66s/it][2025-04-25 08:42:33,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 08:42:33,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.76 | bwd_microstep: 5763.58 | bwd_inner_microstep: 5645.90 | bwd_allreduce_microstep: 117.63 | step_microstep: 18.74
[2025-04-25 08:42:33,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.76 | bwd: 5763.59 | bwd_inner: 5645.90 | bwd_allreduce: 117.65 | step: 18.74

  1%|          | 307/41250 [44:59<98:34:20,  8.67s/it]
                                                      
{'loss': 0.1323, 'grad_norm': 2.5508415699005127, 'learning_rate': 9.919224555735058e-06, 'epoch': 0.07}

  1%|          | 307/41250 [44:59<98:34:20,  8.67s/it][2025-04-25 08:42:42,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:42:42,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.22 | bwd_microstep: 5730.49 | bwd_inner_microstep: 5713.79 | bwd_allreduce_microstep: 16.65 | step_microstep: 18.55
[2025-04-25 08:42:42,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.22 | bwd: 5730.50 | bwd_inner: 5713.79 | bwd_allreduce: 16.67 | step: 18.55

  1%|          | 308/41250 [45:07<98:35:23,  8.67s/it]
                                                      
{'loss': 0.1954, 'grad_norm': 2.0964128971099854, 'learning_rate': 9.951534733441036e-06, 'epoch': 0.07}

  1%|          | 308/41250 [45:07<98:35:23,  8.67s/it][2025-04-25 08:42:51,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 08:42:51,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5790.05 | bwd_inner_microstep: 5650.25 | bwd_allreduce_microstep: 139.74 | step_microstep: 18.51
[2025-04-25 08:42:51,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5790.06 | bwd_inner: 5650.26 | bwd_allreduce: 139.76 | step: 18.52

  1%|          | 309/41250 [45:16<98:41:28,  8.68s/it]
                                                      
{'loss': 0.3474, 'grad_norm': 3.6415343284606934, 'learning_rate': 9.983844911147012e-06, 'epoch': 0.07}

  1%|          | 309/41250 [45:16<98:41:28,  8.68s/it][2025-04-25 08:42:59,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 08:42:59,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.74 | bwd_microstep: 5793.34 | bwd_inner_microstep: 5654.42 | bwd_allreduce_microstep: 138.87 | step_microstep: 19.30
[2025-04-25 08:42:59,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.74 | bwd: 5793.36 | bwd_inner: 5654.42 | bwd_allreduce: 138.89 | step: 19.30

  1%|          | 310/41250 [45:25<98:49:00,  8.69s/it]
                                                      
{'loss': 0.3649, 'grad_norm': 3.7102036476135254, 'learning_rate': 1.001615508885299e-05, 'epoch': 0.08}

  1%|          | 310/41250 [45:25<98:49:00,  8.69s/it][2025-04-25 08:43:08,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.91
[2025-04-25 08:43:08,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.36 | bwd_microstep: 5763.48 | bwd_inner_microstep: 5695.38 | bwd_allreduce_microstep: 68.06 | step_microstep: 18.67
[2025-04-25 08:43:08,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.36 | bwd: 5763.50 | bwd_inner: 5695.38 | bwd_allreduce: 68.08 | step: 18.67

  1%|          | 311/41250 [45:33<98:50:01,  8.69s/it]
                                                      
{'loss': 0.2293, 'grad_norm': 3.310459613800049, 'learning_rate': 1.0048465266558966e-05, 'epoch': 0.08}

  1%|          | 311/41250 [45:33<98:50:01,  8.69s/it][2025-04-25 08:43:17,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 08:43:17,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.97 | bwd_microstep: 5758.96 | bwd_inner_microstep: 5644.53 | bwd_allreduce_microstep: 114.38 | step_microstep: 18.58
[2025-04-25 08:43:17,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.97 | bwd: 5758.97 | bwd_inner: 5644.54 | bwd_allreduce: 114.40 | step: 18.58

  1%|          | 312/41250 [45:42<98:44:16,  8.68s/it]
                                                      
{'loss': 0.1681, 'grad_norm': 3.5736336708068848, 'learning_rate': 1.0080775444264946e-05, 'epoch': 0.08}

  1%|          | 312/41250 [45:42<98:44:16,  8.68s/it][2025-04-25 08:43:25,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 08:43:25,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.90 | bwd_microstep: 5732.76 | bwd_inner_microstep: 5706.71 | bwd_allreduce_microstep: 26.00 | step_microstep: 18.90
[2025-04-25 08:43:25,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.90 | bwd: 5732.77 | bwd_inner: 5706.71 | bwd_allreduce: 26.02 | step: 18.90

  1%|          | 313/41250 [45:51<98:44:15,  8.68s/it]
                                                      
{'loss': 0.1873, 'grad_norm': 2.009171962738037, 'learning_rate': 1.0113085621970922e-05, 'epoch': 0.08}

  1%|          | 313/41250 [45:51<98:44:15,  8.68s/it][2025-04-25 08:43:34,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 08:43:34,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5756.08 | bwd_inner_microstep: 5660.35 | bwd_allreduce_microstep: 95.68 | step_microstep: 19.38
[2025-04-25 08:43:34,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5756.10 | bwd_inner: 5660.35 | bwd_allreduce: 95.70 | step: 19.38

  1%|          | 314/41250 [45:59<98:41:13,  8.68s/it]
                                                      
{'loss': 0.3639, 'grad_norm': 3.239358425140381, 'learning_rate': 1.0145395799676898e-05, 'epoch': 0.08}

  1%|          | 314/41250 [45:59<98:41:13,  8.68s/it][2025-04-25 08:43:43,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 08:43:43,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.17 | bwd_microstep: 5750.46 | bwd_inner_microstep: 5655.59 | bwd_allreduce_microstep: 94.82 | step_microstep: 19.08
[2025-04-25 08:43:43,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.17 | bwd: 5750.47 | bwd_inner: 5655.59 | bwd_allreduce: 94.84 | step: 19.08

  1%|          | 315/41250 [46:08<98:37:54,  8.67s/it]
                                                      
{'loss': 0.269, 'grad_norm': 2.95432186126709, 'learning_rate': 1.0177705977382877e-05, 'epoch': 0.08}

  1%|          | 315/41250 [46:08<98:37:54,  8.67s/it][2025-04-25 08:43:51,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:43:51,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.70 | bwd_microstep: 5684.97 | bwd_inner_microstep: 5656.12 | bwd_allreduce_microstep: 28.81 | step_microstep: 18.51
[2025-04-25 08:43:51,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.70 | bwd: 5684.99 | bwd_inner: 5656.12 | bwd_allreduce: 28.82 | step: 18.51

  1%|          | 316/41250 [46:17<98:22:30,  8.65s/it]
                                                      
{'loss': 0.2745, 'grad_norm': 3.16935133934021, 'learning_rate': 1.0210016155088854e-05, 'epoch': 0.08}

  1%|          | 316/41250 [46:17<98:22:30,  8.65s/it][2025-04-25 08:44:00,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:44:00,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.15 | bwd_microstep: 5733.32 | bwd_inner_microstep: 5655.75 | bwd_allreduce_microstep: 77.52 | step_microstep: 18.35
[2025-04-25 08:44:00,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.15 | bwd: 5733.33 | bwd_inner: 5655.75 | bwd_allreduce: 77.54 | step: 18.35

  1%|          | 317/41250 [46:25<98:22:25,  8.65s/it]
                                                      
{'loss': 0.0793, 'grad_norm': 1.1880398988723755, 'learning_rate': 1.0242326332794832e-05, 'epoch': 0.08}

  1%|          | 317/41250 [46:25<98:22:25,  8.65s/it][2025-04-25 08:44:09,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 08:44:09,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.63 | bwd_microstep: 5716.07 | bwd_inner_microstep: 5703.41 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.79
[2025-04-25 08:44:09,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.63 | bwd: 5716.08 | bwd_inner: 5703.41 | bwd_allreduce: 12.63 | step: 18.79

  1%|          | 318/41250 [46:34<98:22:22,  8.65s/it]
                                                      
{'loss': 0.3486, 'grad_norm': 3.5819900035858154, 'learning_rate': 1.0274636510500808e-05, 'epoch': 0.08}

  1%|          | 318/41250 [46:34<98:22:22,  8.65s/it][2025-04-25 08:44:17,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:44:17,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.38 | bwd_microstep: 5712.74 | bwd_inner_microstep: 5647.02 | bwd_allreduce_microstep: 65.67 | step_microstep: 18.30
[2025-04-25 08:44:17,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.38 | bwd: 5712.75 | bwd_inner: 5647.02 | bwd_allreduce: 65.69 | step: 18.31

  1%|          | 319/41250 [46:43<98:16:47,  8.64s/it]
                                                      
{'loss': 0.3125, 'grad_norm': 3.346698760986328, 'learning_rate': 1.0306946688206787e-05, 'epoch': 0.08}

  1%|          | 319/41250 [46:43<98:16:47,  8.64s/it][2025-04-25 08:44:26,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:44:26,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.13 | bwd_microstep: 5880.74 | bwd_inner_microstep: 5649.60 | bwd_allreduce_microstep: 231.10 | step_microstep: 18.25
[2025-04-25 08:44:26,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.13 | bwd: 5880.75 | bwd_inner: 5649.60 | bwd_allreduce: 231.11 | step: 18.25

  1%|          | 320/41250 [46:51<98:46:44,  8.69s/it]
                                                      
{'loss': 0.1173, 'grad_norm': 1.2808520793914795, 'learning_rate': 1.0339256865912764e-05, 'epoch': 0.08}

  1%|          | 320/41250 [46:51<98:46:44,  8.69s/it][2025-04-25 08:44:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:44:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.76 | bwd_microstep: 5699.52 | bwd_inner_microstep: 5686.86 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.47
[2025-04-25 08:44:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.76 | bwd: 5699.53 | bwd_inner: 5686.86 | bwd_allreduce: 12.63 | step: 18.48

  1%|          | 321/41250 [47:00<98:34:43,  8.67s/it]
                                                      
{'loss': 0.3306, 'grad_norm': 3.5957932472229004, 'learning_rate': 1.037156704361874e-05, 'epoch': 0.08}

  1%|          | 321/41250 [47:00<98:34:43,  8.67s/it][2025-04-25 08:44:43,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.14 | optimizer_step: 1.12
[2025-04-25 08:44:43,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.71 | bwd_microstep: 5677.83 | bwd_inner_microstep: 5639.10 | bwd_allreduce_microstep: 38.67 | step_microstep: 19.58
[2025-04-25 08:44:43,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.71 | bwd: 5677.84 | bwd_inner: 5639.10 | bwd_allreduce: 38.70 | step: 19.58

  1%|          | 322/41250 [47:09<98:22:51,  8.65s/it]
                                                      
{'loss': 0.2298, 'grad_norm': 2.623976230621338, 'learning_rate': 1.040387722132472e-05, 'epoch': 0.08}

  1%|          | 322/41250 [47:09<98:22:51,  8.65s/it][2025-04-25 08:44:52,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:44:52,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.58 | bwd_microstep: 5777.28 | bwd_inner_microstep: 5764.28 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.32
[2025-04-25 08:44:52,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.58 | bwd: 5777.29 | bwd_inner: 5764.28 | bwd_allreduce: 12.98 | step: 18.32

  1%|          | 323/41250 [47:17<98:42:38,  8.68s/it]
                                                      
{'loss': 0.4515, 'grad_norm': 2.8755877017974854, 'learning_rate': 1.0436187399030695e-05, 'epoch': 0.08}

  1%|          | 323/41250 [47:17<98:42:38,  8.68s/it][2025-04-25 08:45:01,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:45:01,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5758.10 | bwd_inner_microstep: 5680.89 | bwd_allreduce_microstep: 77.17 | step_microstep: 18.44
[2025-04-25 08:45:01,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5758.11 | bwd_inner: 5680.89 | bwd_allreduce: 77.19 | step: 18.44

  1%|          | 324/41250 [47:26<98:44:07,  8.69s/it]
                                                      
{'loss': 0.4121, 'grad_norm': 3.709523916244507, 'learning_rate': 1.0468497576736673e-05, 'epoch': 0.08}

  1%|          | 324/41250 [47:26<98:44:07,  8.69s/it][2025-04-25 08:45:09,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 08:45:09,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5705.60 | bwd_inner_microstep: 5692.60 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.36
[2025-04-25 08:45:09,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5705.62 | bwd_inner: 5692.59 | bwd_allreduce: 12.98 | step: 19.37

  1%|          | 325/41250 [47:35<98:34:10,  8.67s/it]
                                                      
{'loss': 0.2458, 'grad_norm': 2.272737503051758, 'learning_rate': 1.050080775444265e-05, 'epoch': 0.08}

  1%|          | 325/41250 [47:35<98:34:10,  8.67s/it][2025-04-25 08:45:18,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 08:45:18,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.95 | bwd_microstep: 5781.43 | bwd_inner_microstep: 5768.77 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.26
[2025-04-25 08:45:18,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.95 | bwd: 5781.44 | bwd_inner: 5768.77 | bwd_allreduce: 12.63 | step: 18.26

  1%|          | 326/41250 [47:44<98:50:09,  8.69s/it]
                                                      
{'loss': 0.3152, 'grad_norm': 2.6152379512786865, 'learning_rate': 1.0533117932148629e-05, 'epoch': 0.08}

  1%|          | 326/41250 [47:44<98:50:09,  8.69s/it][2025-04-25 08:45:27,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:45:27,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.41 | bwd_microstep: 5680.51 | bwd_inner_microstep: 5652.69 | bwd_allreduce_microstep: 27.78 | step_microstep: 18.65
[2025-04-25 08:45:27,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.41 | bwd: 5680.53 | bwd_inner: 5652.69 | bwd_allreduce: 27.79 | step: 18.65

  1%|          | 327/41250 [47:52<98:27:59,  8.66s/it]
                                                      
{'loss': 0.3296, 'grad_norm': 3.9284965991973877, 'learning_rate': 1.0565428109854605e-05, 'epoch': 0.08}

  1%|          | 327/41250 [47:52<98:27:59,  8.66s/it][2025-04-25 08:45:35,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:45:35,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5727.32 | bwd_inner_microstep: 5678.14 | bwd_allreduce_microstep: 49.14 | step_microstep: 18.49
[2025-04-25 08:45:35,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5727.33 | bwd_inner: 5678.14 | bwd_allreduce: 49.15 | step: 18.49

  1%|          | 328/41250 [48:01<98:27:36,  8.66s/it]
                                                      
{'loss': 0.1346, 'grad_norm': 1.4345204830169678, 'learning_rate': 1.0597738287560582e-05, 'epoch': 0.08}

  1%|          | 328/41250 [48:01<98:27:36,  8.66s/it][2025-04-25 08:45:44,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 08:45:44,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.77 | bwd_microstep: 5734.44 | bwd_inner_microstep: 5655.53 | bwd_allreduce_microstep: 78.86 | step_microstep: 18.66
[2025-04-25 08:45:44,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.77 | bwd: 5734.45 | bwd_inner: 5655.53 | bwd_allreduce: 78.88 | step: 18.67

  1%|          | 329/41250 [48:09<98:24:09,  8.66s/it]
                                                      
{'loss': 0.0941, 'grad_norm': 1.2351467609405518, 'learning_rate': 1.0630048465266561e-05, 'epoch': 0.08}

  1%|          | 329/41250 [48:09<98:24:09,  8.66s/it][2025-04-25 08:45:53,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:45:53,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.16 | bwd_microstep: 5747.88 | bwd_inner_microstep: 5688.00 | bwd_allreduce_microstep: 59.83 | step_microstep: 18.67
[2025-04-25 08:45:53,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.16 | bwd: 5747.89 | bwd_inner: 5688.00 | bwd_allreduce: 59.85 | step: 18.67

  1%|          | 330/41250 [48:18<98:29:32,  8.67s/it]
                                                      
{'loss': 0.2256, 'grad_norm': 3.1854708194732666, 'learning_rate': 1.0662358642972537e-05, 'epoch': 0.08}

  1%|          | 330/41250 [48:18<98:29:32,  8.67s/it][2025-04-25 08:46:01,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:46:01,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.85 | bwd_microstep: 5705.37 | bwd_inner_microstep: 5686.97 | bwd_allreduce_microstep: 18.35 | step_microstep: 18.32
[2025-04-25 08:46:01,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.85 | bwd: 5705.38 | bwd_inner: 5686.97 | bwd_allreduce: 18.37 | step: 18.32

  1%|          | 331/41250 [48:27<98:23:50,  8.66s/it]
                                                      
{'loss': 0.1638, 'grad_norm': 2.1656885147094727, 'learning_rate': 1.0694668820678515e-05, 'epoch': 0.08}

  1%|          | 331/41250 [48:27<98:23:50,  8.66s/it][2025-04-25 08:46:10,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:46:10,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.85 | bwd_microstep: 5736.12 | bwd_inner_microstep: 5644.93 | bwd_allreduce_microstep: 91.14 | step_microstep: 18.52
[2025-04-25 08:46:10,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.85 | bwd: 5736.13 | bwd_inner: 5644.93 | bwd_allreduce: 91.16 | step: 18.52

  1%|          | 332/41250 [48:35<98:20:05,  8.65s/it]
                                                      
{'loss': 0.2197, 'grad_norm': 2.4923768043518066, 'learning_rate': 1.0726978998384491e-05, 'epoch': 0.08}

  1%|          | 332/41250 [48:35<98:20:05,  8.65s/it][2025-04-25 08:46:19,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:46:19,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5694.13 | bwd_inner_microstep: 5681.24 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.00
[2025-04-25 08:46:19,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5694.14 | bwd_inner: 5681.24 | bwd_allreduce: 12.87 | step: 18.01

  1%|          | 333/41250 [48:44<98:15:05,  8.64s/it]
                                                      
{'loss': 0.2012, 'grad_norm': 2.1899566650390625, 'learning_rate': 1.0759289176090468e-05, 'epoch': 0.08}

  1%|          | 333/41250 [48:44<98:15:05,  8.64s/it][2025-04-25 08:46:27,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:46:27,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.12 | bwd_microstep: 5778.97 | bwd_inner_microstep: 5766.43 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.69
[2025-04-25 08:46:27,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.12 | bwd: 5778.98 | bwd_inner: 5766.43 | bwd_allreduce: 12.51 | step: 18.69

  1%|          | 334/41250 [48:53<98:35:45,  8.67s/it]
                                                      
{'loss': 0.1037, 'grad_norm': 1.6826133728027344, 'learning_rate': 1.0791599353796447e-05, 'epoch': 0.08}

  1%|          | 334/41250 [48:53<98:35:45,  8.67s/it][2025-04-25 08:46:36,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:46:36,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.00 | bwd_microstep: 5695.36 | bwd_inner_microstep: 5682.59 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.41
[2025-04-25 08:46:36,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.00 | bwd: 5695.37 | bwd_inner: 5682.59 | bwd_allreduce: 12.75 | step: 18.41

  1%|          | 335/41250 [49:01<98:26:35,  8.66s/it]
                                                      
{'loss': 0.1707, 'grad_norm': 1.8774166107177734, 'learning_rate': 1.0823909531502423e-05, 'epoch': 0.08}

  1%|          | 335/41250 [49:01<98:26:35,  8.66s/it][2025-04-25 08:46:45,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:46:45,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.33 | bwd_microstep: 5731.48 | bwd_inner_microstep: 5674.75 | bwd_allreduce_microstep: 56.69 | step_microstep: 18.26
[2025-04-25 08:46:45,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.33 | bwd: 5731.49 | bwd_inner: 5674.75 | bwd_allreduce: 56.71 | step: 18.26

  1%|          | 336/41250 [49:10<98:28:30,  8.66s/it]
                                                      
{'loss': 0.2195, 'grad_norm': 2.221560478210449, 'learning_rate': 1.0856219709208403e-05, 'epoch': 0.08}

  1%|          | 336/41250 [49:10<98:28:30,  8.66s/it][2025-04-25 08:46:53,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:46:53,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.22 | bwd_microstep: 5756.62 | bwd_inner_microstep: 5637.90 | bwd_allreduce_microstep: 118.67 | step_microstep: 18.08
[2025-04-25 08:46:53,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.22 | bwd: 5756.63 | bwd_inner: 5637.90 | bwd_allreduce: 118.69 | step: 18.08

  1%|          | 337/41250 [49:19<98:30:44,  8.67s/it]
                                                      
{'loss': 0.1631, 'grad_norm': 2.7114570140838623, 'learning_rate': 1.0888529886914379e-05, 'epoch': 0.08}

  1%|          | 337/41250 [49:19<98:30:44,  8.67s/it][2025-04-25 08:47:02,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 08:47:02,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.11 | bwd_microstep: 5746.85 | bwd_inner_microstep: 5643.18 | bwd_allreduce_microstep: 103.62 | step_microstep: 18.83
[2025-04-25 08:47:02,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.11 | bwd: 5746.86 | bwd_inner: 5643.18 | bwd_allreduce: 103.63 | step: 18.83

  1%|          | 338/41250 [49:27<98:27:17,  8.66s/it]
                                                      
{'loss': 0.0972, 'grad_norm': 1.2424529790878296, 'learning_rate': 1.0920840064620357e-05, 'epoch': 0.08}

  1%|          | 338/41250 [49:27<98:27:17,  8.66s/it][2025-04-25 08:47:11,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:47:11,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.87 | bwd_microstep: 5857.17 | bwd_inner_microstep: 5650.86 | bwd_allreduce_microstep: 206.26 | step_microstep: 18.87
[2025-04-25 08:47:11,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.87 | bwd: 5857.18 | bwd_inner: 5650.86 | bwd_allreduce: 206.28 | step: 18.87

  1%|          | 339/41250 [49:36<98:51:25,  8.70s/it]
                                                      
{'loss': 0.5307, 'grad_norm': 4.257749557495117, 'learning_rate': 1.0953150242326333e-05, 'epoch': 0.08}

  1%|          | 339/41250 [49:36<98:51:25,  8.70s/it][2025-04-25 08:47:19,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:47:19,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.06 | bwd_microstep: 5668.48 | bwd_inner_microstep: 5622.89 | bwd_allreduce_microstep: 45.54 | step_microstep: 18.55
[2025-04-25 08:47:19,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.06 | bwd: 5668.49 | bwd_inner: 5622.89 | bwd_allreduce: 45.56 | step: 18.55

  1%|          | 340/41250 [49:45<98:26:18,  8.66s/it]
                                                      
{'loss': 0.1749, 'grad_norm': 2.703179121017456, 'learning_rate': 1.098546042003231e-05, 'epoch': 0.08}

  1%|          | 340/41250 [49:45<98:26:18,  8.66s/it][2025-04-25 08:47:28,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:47:28,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.62 | bwd_microstep: 5734.26 | bwd_inner_microstep: 5696.36 | bwd_allreduce_microstep: 37.85 | step_microstep: 18.32
[2025-04-25 08:47:28,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.63 | bwd: 5734.27 | bwd_inner: 5696.36 | bwd_allreduce: 37.87 | step: 18.32

  1%|          | 341/41250 [49:53<98:24:26,  8.66s/it]
                                                      
{'loss': 0.1502, 'grad_norm': 3.443694829940796, 'learning_rate': 1.1017770597738289e-05, 'epoch': 0.08}

  1%|          | 341/41250 [49:53<98:24:26,  8.66s/it][2025-04-25 08:47:37,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:47:37,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.43 | bwd_microstep: 5678.29 | bwd_inner_microstep: 5631.89 | bwd_allreduce_microstep: 46.35 | step_microstep: 18.69
[2025-04-25 08:47:37,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.43 | bwd: 5678.30 | bwd_inner: 5631.89 | bwd_allreduce: 46.37 | step: 18.69

  1%|          | 342/41250 [50:02<98:08:07,  8.64s/it]
                                                      
{'loss': 0.1873, 'grad_norm': 4.547693729400635, 'learning_rate': 1.1050080775444265e-05, 'epoch': 0.08}

  1%|          | 342/41250 [50:02<98:08:07,  8.64s/it][2025-04-25 08:47:45,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:47:45,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.40 | bwd_microstep: 5752.37 | bwd_inner_microstep: 5634.05 | bwd_allreduce_microstep: 118.27 | step_microstep: 18.37
[2025-04-25 08:47:45,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.40 | bwd: 5752.38 | bwd_inner: 5634.05 | bwd_allreduce: 118.29 | step: 18.37

  1%|          | 343/41250 [50:11<98:13:24,  8.64s/it]
                                                      
{'loss': 0.3427, 'grad_norm': 5.582585334777832, 'learning_rate': 1.1082390953150243e-05, 'epoch': 0.08}

  1%|          | 343/41250 [50:11<98:13:24,  8.64s/it][2025-04-25 08:47:54,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 08:47:54,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.30 | bwd_microstep: 5731.53 | bwd_inner_microstep: 5704.59 | bwd_allreduce_microstep: 26.90 | step_microstep: 18.51
[2025-04-25 08:47:54,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.30 | bwd: 5731.54 | bwd_inner: 5704.59 | bwd_allreduce: 26.91 | step: 18.52

  1%|          | 344/41250 [50:19<98:19:27,  8.65s/it]
                                                      
{'loss': 0.1717, 'grad_norm': 1.264743685722351, 'learning_rate': 1.1114701130856221e-05, 'epoch': 0.08}

  1%|          | 344/41250 [50:19<98:19:27,  8.65s/it][2025-04-25 08:48:03,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:48:03,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.12 | bwd_microstep: 5780.44 | bwd_inner_microstep: 5767.70 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.39
[2025-04-25 08:48:03,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.12 | bwd: 5780.45 | bwd_inner: 5767.70 | bwd_allreduce: 12.71 | step: 18.40

  1%|          | 345/41250 [50:28<98:38:07,  8.68s/it]
                                                      
{'loss': 0.142, 'grad_norm': 1.5437431335449219, 'learning_rate': 1.1147011308562199e-05, 'epoch': 0.08}

  1%|          | 345/41250 [50:28<98:38:07,  8.68s/it][2025-04-25 08:48:11,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:48:11,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.83 | bwd_microstep: 5775.66 | bwd_inner_microstep: 5650.37 | bwd_allreduce_microstep: 125.24 | step_microstep: 18.56
[2025-04-25 08:48:11,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.83 | bwd: 5775.67 | bwd_inner: 5650.37 | bwd_allreduce: 125.26 | step: 18.56

  1%|          | 346/41250 [50:37<98:39:53,  8.68s/it]
                                                      
{'loss': 0.1593, 'grad_norm': 2.523664951324463, 'learning_rate': 1.1179321486268175e-05, 'epoch': 0.08}

  1%|          | 346/41250 [50:37<98:39:53,  8.68s/it][2025-04-25 08:48:20,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:48:20,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.45 | bwd_microstep: 5740.17 | bwd_inner_microstep: 5691.78 | bwd_allreduce_microstep: 48.34 | step_microstep: 18.06
[2025-04-25 08:48:20,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.45 | bwd: 5740.19 | bwd_inner: 5691.78 | bwd_allreduce: 48.36 | step: 18.06

  1%|          | 347/41250 [50:45<98:37:01,  8.68s/it]
                                                      
{'loss': 0.3555, 'grad_norm': 3.7335424423217773, 'learning_rate': 1.1211631663974151e-05, 'epoch': 0.08}

  1%|          | 347/41250 [50:45<98:37:01,  8.68s/it][2025-04-25 08:48:29,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 08:48:29,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.09 | bwd_microstep: 5741.85 | bwd_inner_microstep: 5694.43 | bwd_allreduce_microstep: 47.38 | step_microstep: 18.61
[2025-04-25 08:48:29,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.09 | bwd: 5741.87 | bwd_inner: 5694.43 | bwd_allreduce: 47.39 | step: 18.61

  1%|          | 348/41250 [50:54<98:36:03,  8.68s/it]
                                                      
{'loss': 0.3093, 'grad_norm': 2.4196650981903076, 'learning_rate': 1.124394184168013e-05, 'epoch': 0.08}

  1%|          | 348/41250 [50:54<98:36:03,  8.68s/it][2025-04-25 08:48:37,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:48:37,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.79 | bwd_microstep: 5760.07 | bwd_inner_microstep: 5705.96 | bwd_allreduce_microstep: 54.07 | step_microstep: 18.38
[2025-04-25 08:48:37,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.79 | bwd: 5760.08 | bwd_inner: 5705.96 | bwd_allreduce: 54.08 | step: 18.38

  1%|          | 349/41250 [51:03<98:39:40,  8.68s/it]
                                                      
{'loss': 0.2066, 'grad_norm': 1.9032713174819946, 'learning_rate': 1.1276252019386107e-05, 'epoch': 0.08}

  1%|          | 349/41250 [51:03<98:39:40,  8.68s/it][2025-04-25 08:48:46,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:48:46,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.50 | bwd_microstep: 5856.78 | bwd_inner_microstep: 5696.58 | bwd_allreduce_microstep: 160.16 | step_microstep: 18.53
[2025-04-25 08:48:46,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5856.79 | bwd_inner: 5696.58 | bwd_allreduce: 160.18 | step: 18.53

  1%|          | 350/41250 [51:12<99:02:04,  8.72s/it]
                                                      
{'loss': 0.2844, 'grad_norm': 2.335712194442749, 'learning_rate': 1.1308562197092085e-05, 'epoch': 0.08}

  1%|          | 350/41250 [51:12<99:02:04,  8.72s/it][2025-04-25 08:48:55,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:48:55,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.02 | bwd_microstep: 5776.27 | bwd_inner_microstep: 5682.22 | bwd_allreduce_microstep: 94.00 | step_microstep: 18.36
[2025-04-25 08:48:55,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.02 | bwd: 5776.28 | bwd_inner: 5682.22 | bwd_allreduce: 94.02 | step: 18.36

  1%|          | 351/41250 [51:20<99:00:47,  8.72s/it]
                                                      
{'loss': 0.3248, 'grad_norm': 2.3688087463378906, 'learning_rate': 1.1340872374798063e-05, 'epoch': 0.09}

  1%|          | 351/41250 [51:20<99:00:47,  8.72s/it][2025-04-25 08:49:04,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:49:04,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.20 | bwd_microstep: 5747.60 | bwd_inner_microstep: 5709.62 | bwd_allreduce_microstep: 37.93 | step_microstep: 18.30
[2025-04-25 08:49:04,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.20 | bwd: 5747.61 | bwd_inner: 5709.62 | bwd_allreduce: 37.95 | step: 18.30

  1%|          | 352/41250 [51:29<98:55:30,  8.71s/it]
                                                      
{'loss': 0.287, 'grad_norm': 2.7517712116241455, 'learning_rate': 1.137318255250404e-05, 'epoch': 0.09}

  1%|          | 352/41250 [51:29<98:55:30,  8.71s/it][2025-04-25 08:49:12,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:49:12,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.07 | bwd_microstep: 5807.88 | bwd_inner_microstep: 5672.64 | bwd_allreduce_microstep: 135.19 | step_microstep: 18.25
[2025-04-25 08:49:12,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.07 | bwd: 5807.89 | bwd_inner: 5672.64 | bwd_allreduce: 135.21 | step: 18.25

  1%|          | 353/41250 [51:38<98:59:56,  8.71s/it]
                                                      
{'loss': 0.1628, 'grad_norm': 1.3649340867996216, 'learning_rate': 1.1405492730210017e-05, 'epoch': 0.09}

  1%|          | 353/41250 [51:38<98:59:56,  8.71s/it][2025-04-25 08:49:21,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:49:21,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.45 | bwd_microstep: 5773.06 | bwd_inner_microstep: 5709.81 | bwd_allreduce_microstep: 63.20 | step_microstep: 17.89
[2025-04-25 08:49:21,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.45 | bwd: 5773.07 | bwd_inner: 5709.81 | bwd_allreduce: 63.22 | step: 17.89

  1%|          | 354/41250 [51:46<98:57:30,  8.71s/it]
                                                      
{'loss': 0.185, 'grad_norm': 1.949427604675293, 'learning_rate': 1.1437802907915993e-05, 'epoch': 0.09}

  1%|          | 354/41250 [51:46<98:57:30,  8.71s/it][2025-04-25 08:49:30,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:49:30,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.09 | bwd_microstep: 5914.41 | bwd_inner_microstep: 5669.23 | bwd_allreduce_microstep: 245.13 | step_microstep: 18.61
[2025-04-25 08:49:30,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.09 | bwd: 5914.42 | bwd_inner: 5669.23 | bwd_allreduce: 245.15 | step: 18.61

  1%|          | 355/41250 [51:55<99:23:46,  8.75s/it]
                                                      
{'loss': 0.1803, 'grad_norm': 1.5891748666763306, 'learning_rate': 1.1470113085621972e-05, 'epoch': 0.09}

  1%|          | 355/41250 [51:55<99:23:46,  8.75s/it][2025-04-25 08:49:39,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:49:39,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.79 | bwd_microstep: 5794.83 | bwd_inner_microstep: 5656.81 | bwd_allreduce_microstep: 137.97 | step_microstep: 18.43
[2025-04-25 08:49:39,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.79 | bwd: 5794.84 | bwd_inner: 5656.81 | bwd_allreduce: 137.99 | step: 18.43

  1%|          | 356/41250 [52:04<99:14:46,  8.74s/it]
                                                      
{'loss': 0.4423, 'grad_norm': 3.8944942951202393, 'learning_rate': 1.1502423263327949e-05, 'epoch': 0.09}

  1%|          | 356/41250 [52:04<99:14:46,  8.74s/it][2025-04-25 08:49:47,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-25 08:49:47,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5723.19 | bwd_inner_microstep: 5651.73 | bwd_allreduce_microstep: 71.42 | step_microstep: 18.47
[2025-04-25 08:49:47,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5723.20 | bwd_inner: 5651.73 | bwd_allreduce: 71.44 | step: 18.47

  1%|          | 357/41250 [52:13<98:54:07,  8.71s/it]
                                                      
{'loss': 0.2036, 'grad_norm': 1.956114649772644, 'learning_rate': 1.1534733441033927e-05, 'epoch': 0.09}

  1%|          | 357/41250 [52:13<98:54:07,  8.71s/it][2025-04-25 08:49:56,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.92 | optimizer_gradients: 0.96 | optimizer_step: 1.06
[2025-04-25 08:49:56,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5751.66 | bwd_inner_microstep: 5693.07 | bwd_allreduce_microstep: 58.55 | step_microstep: 18.98
[2025-04-25 08:49:56,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5751.67 | bwd_inner: 5693.07 | bwd_allreduce: 58.56 | step: 18.98

  1%|          | 358/41250 [52:21<98:51:11,  8.70s/it]
                                                      
{'loss': 0.1628, 'grad_norm': 1.6464790105819702, 'learning_rate': 1.1567043618739904e-05, 'epoch': 0.09}

  1%|          | 358/41250 [52:21<98:51:11,  8.70s/it][2025-04-25 08:50:05,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:50:05,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.75 | bwd_microstep: 5720.07 | bwd_inner_microstep: 5658.45 | bwd_allreduce_microstep: 61.57 | step_microstep: 18.09
[2025-04-25 08:50:05,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.75 | bwd: 5720.08 | bwd_inner: 5658.45 | bwd_allreduce: 61.59 | step: 18.10

  1%|          | 359/41250 [52:30<98:38:52,  8.68s/it]
                                                      
{'loss': 0.1975, 'grad_norm': 2.0733754634857178, 'learning_rate': 1.1599353796445882e-05, 'epoch': 0.09}

  1%|          | 359/41250 [52:30<98:38:52,  8.68s/it][2025-04-25 08:50:13,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 08:50:13,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.13 | bwd_microstep: 5768.37 | bwd_inner_microstep: 5666.62 | bwd_allreduce_microstep: 101.71 | step_microstep: 18.60
[2025-04-25 08:50:13,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.13 | bwd: 5768.39 | bwd_inner: 5666.62 | bwd_allreduce: 101.73 | step: 18.60

  1%|          | 360/41250 [52:39<98:39:18,  8.69s/it]
                                                      
{'loss': 0.3449, 'grad_norm': 2.8266944885253906, 'learning_rate': 1.1631663974151859e-05, 'epoch': 0.09}

  1%|          | 360/41250 [52:39<98:39:18,  8.69s/it][2025-04-25 08:50:22,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 08:50:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.76 | bwd_microstep: 5775.60 | bwd_inner_microstep: 5653.46 | bwd_allreduce_microstep: 122.10 | step_microstep: 18.71
[2025-04-25 08:50:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.76 | bwd: 5775.62 | bwd_inner: 5653.46 | bwd_allreduce: 122.12 | step: 18.72

  1%|          | 361/41250 [52:47<98:40:39,  8.69s/it]
                                                      
{'loss': 0.3754, 'grad_norm': 2.775932788848877, 'learning_rate': 1.1663974151857835e-05, 'epoch': 0.09}

  1%|          | 361/41250 [52:47<98:40:39,  8.69s/it][2025-04-25 08:50:31,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 08:50:31,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.18 | bwd_microstep: 5739.94 | bwd_inner_microstep: 5726.85 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.79
[2025-04-25 08:50:31,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.18 | bwd: 5739.95 | bwd_inner: 5726.85 | bwd_allreduce: 13.06 | step: 18.80

  1%|          | 362/41250 [52:56<98:41:33,  8.69s/it]
                                                      
{'loss': 0.3487, 'grad_norm': 3.6606173515319824, 'learning_rate': 1.1696284329563814e-05, 'epoch': 0.09}

  1%|          | 362/41250 [52:56<98:41:33,  8.69s/it][2025-04-25 08:50:39,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:50:39,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.20 | bwd_microstep: 5783.33 | bwd_inner_microstep: 5665.90 | bwd_allreduce_microstep: 117.39 | step_microstep: 18.56
[2025-04-25 08:50:39,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.20 | bwd: 5783.34 | bwd_inner: 5665.90 | bwd_allreduce: 117.40 | step: 18.56

  1%|          | 363/41250 [53:05<98:45:28,  8.70s/it]
                                                      
{'loss': 0.1478, 'grad_norm': 1.4771850109100342, 'learning_rate': 1.172859450726979e-05, 'epoch': 0.09}

  1%|          | 363/41250 [53:05<98:45:28,  8.70s/it][2025-04-25 08:50:48,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 08:50:48,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.20 | bwd_microstep: 5741.31 | bwd_inner_microstep: 5711.78 | bwd_allreduce_microstep: 29.48 | step_microstep: 18.82
[2025-04-25 08:50:48,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.19 | bwd: 5741.32 | bwd_inner: 5711.78 | bwd_allreduce: 29.50 | step: 18.82

  1%|          | 364/41250 [53:13<98:43:50,  8.69s/it]
                                                      
{'loss': 0.2938, 'grad_norm': 2.0274691581726074, 'learning_rate': 1.1760904684975768e-05, 'epoch': 0.09}

  1%|          | 364/41250 [53:13<98:43:50,  8.69s/it][2025-04-25 08:50:57,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:50:57,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.14 | bwd_microstep: 5714.96 | bwd_inner_microstep: 5702.17 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.58
[2025-04-25 08:50:57,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.14 | bwd: 5714.97 | bwd_inner: 5702.17 | bwd_allreduce: 12.77 | step: 18.59

  1%|          | 365/41250 [53:22<98:35:23,  8.68s/it]
                                                      
{'loss': 0.091, 'grad_norm': 1.0110985040664673, 'learning_rate': 1.1793214862681746e-05, 'epoch': 0.09}

  1%|          | 365/41250 [53:22<98:35:23,  8.68s/it][2025-04-25 08:51:05,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 08:51:05,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.84 | bwd_microstep: 5777.66 | bwd_inner_microstep: 5699.67 | bwd_allreduce_microstep: 77.95 | step_microstep: 18.99
[2025-04-25 08:51:05,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.84 | bwd: 5777.67 | bwd_inner: 5699.67 | bwd_allreduce: 77.97 | step: 18.99

  1%|          | 366/41250 [53:31<98:43:27,  8.69s/it]
                                                      
{'loss': 0.2098, 'grad_norm': 1.784301996231079, 'learning_rate': 1.1825525040387724e-05, 'epoch': 0.09}

  1%|          | 366/41250 [53:31<98:43:27,  8.69s/it][2025-04-25 08:51:14,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:51:14,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.84 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5660.66 | bwd_allreduce_microstep: 44.18 | step_microstep: 18.11
[2025-04-25 08:51:14,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.84 | bwd: 5704.89 | bwd_inner: 5660.66 | bwd_allreduce: 44.19 | step: 18.11

  1%|          | 367/41250 [53:39<98:29:50,  8.67s/it]
                                                      
{'loss': 0.3343, 'grad_norm': 3.0777130126953125, 'learning_rate': 1.18578352180937e-05, 'epoch': 0.09}

  1%|          | 367/41250 [53:39<98:29:50,  8.67s/it][2025-04-25 08:51:23,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:51:23,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.30 | bwd_microstep: 5760.06 | bwd_inner_microstep: 5662.43 | bwd_allreduce_microstep: 97.58 | step_microstep: 18.45
[2025-04-25 08:51:23,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.30 | bwd: 5760.07 | bwd_inner: 5662.43 | bwd_allreduce: 97.60 | step: 18.45

  1%|          | 368/41250 [53:48<98:33:13,  8.68s/it]
                                                      
{'loss': 0.0559, 'grad_norm': 0.8943654298782349, 'learning_rate': 1.1890145395799677e-05, 'epoch': 0.09}

  1%|          | 368/41250 [53:48<98:33:13,  8.68s/it][2025-04-25 08:51:32,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 08:51:32,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.79 | bwd_microstep: 5791.84 | bwd_inner_microstep: 5779.18 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.62
[2025-04-25 08:51:32,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.79 | bwd: 5791.85 | bwd_inner: 5779.18 | bwd_allreduce: 12.63 | step: 18.62

  1%|          | 369/41250 [53:57<98:53:01,  8.71s/it]
                                                      
{'loss': 0.2788, 'grad_norm': 1.7539759874343872, 'learning_rate': 1.1922455573505656e-05, 'epoch': 0.09}

  1%|          | 369/41250 [53:57<98:53:01,  8.71s/it][2025-04-25 08:51:40,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:51:40,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5703.14 | bwd_inner_microstep: 5690.46 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.21
[2025-04-25 08:51:40,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5703.15 | bwd_inner: 5690.46 | bwd_allreduce: 12.65 | step: 18.21

  1%|          | 370/41250 [54:06<98:39:14,  8.69s/it]
                                                      
{'loss': 0.2487, 'grad_norm': 2.5114471912384033, 'learning_rate': 1.1954765751211632e-05, 'epoch': 0.09}

  1%|          | 370/41250 [54:06<98:39:14,  8.69s/it][2025-04-25 08:51:49,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:51:49,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.65 | bwd_microstep: 5717.73 | bwd_inner_microstep: 5705.19 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.34
[2025-04-25 08:51:49,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.65 | bwd: 5717.74 | bwd_inner: 5705.19 | bwd_allreduce: 12.51 | step: 18.35

  1%|          | 371/41250 [54:14<98:32:44,  8.68s/it]
                                                      
{'loss': 0.2142, 'grad_norm': 2.0766735076904297, 'learning_rate': 1.198707592891761e-05, 'epoch': 0.09}

  1%|          | 371/41250 [54:14<98:32:44,  8.68s/it][2025-04-25 08:51:58,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 08:51:58,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5749.32 | bwd_inner_microstep: 5683.20 | bwd_allreduce_microstep: 66.08 | step_microstep: 18.22
[2025-04-25 08:51:58,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5749.34 | bwd_inner: 5683.20 | bwd_allreduce: 66.10 | step: 18.23

  1%|          | 372/41250 [54:23<98:34:23,  8.68s/it]
                                                      
{'loss': 0.2421, 'grad_norm': 1.8331204652786255, 'learning_rate': 1.2019386106623588e-05, 'epoch': 0.09}

  1%|          | 372/41250 [54:23<98:34:23,  8.68s/it][2025-04-25 08:52:06,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-25 08:52:06,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.91 | bwd_microstep: 5715.29 | bwd_inner_microstep: 5702.47 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.46
[2025-04-25 08:52:06,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.91 | bwd: 5715.30 | bwd_inner: 5702.47 | bwd_allreduce: 12.79 | step: 19.46

  1%|          | 373/41250 [54:32<98:30:47,  8.68s/it]
                                                      
{'loss': 0.1401, 'grad_norm': 1.4058576822280884, 'learning_rate': 1.2051696284329564e-05, 'epoch': 0.09}

  1%|          | 373/41250 [54:32<98:30:47,  8.68s/it][2025-04-25 08:52:15,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 08:52:15,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.07 | bwd_microstep: 5679.48 | bwd_inner_microstep: 5654.49 | bwd_allreduce_microstep: 24.94 | step_microstep: 18.72
[2025-04-25 08:52:15,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.07 | bwd: 5679.50 | bwd_inner: 5654.49 | bwd_allreduce: 24.96 | step: 18.72

  1%|          | 374/41250 [54:40<98:13:41,  8.65s/it]
                                                      
{'loss': 0.2284, 'grad_norm': 2.743295192718506, 'learning_rate': 1.2084006462035542e-05, 'epoch': 0.09}

  1%|          | 374/41250 [54:40<98:13:41,  8.65s/it][2025-04-25 08:52:24,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:52:24,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.64 | bwd_microstep: 5879.81 | bwd_inner_microstep: 5867.20 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.72
[2025-04-25 08:52:24,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.64 | bwd: 5879.83 | bwd_inner: 5867.19 | bwd_allreduce: 12.59 | step: 18.72

  1%|          | 375/41250 [54:49<99:03:35,  8.72s/it]
                                                      
{'loss': 0.3815, 'grad_norm': 4.104799747467041, 'learning_rate': 1.2116316639741518e-05, 'epoch': 0.09}

  1%|          | 375/41250 [54:49<99:03:35,  8.72s/it][2025-04-25 08:52:32,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:52:32,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.18 | bwd_microstep: 5703.90 | bwd_inner_microstep: 5691.13 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.47
[2025-04-25 08:52:32,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.18 | bwd: 5703.91 | bwd_inner: 5691.12 | bwd_allreduce: 12.75 | step: 18.47

  1%|          | 376/41250 [54:58<98:46:36,  8.70s/it]
                                                      
{'loss': 0.2248, 'grad_norm': 2.1003775596618652, 'learning_rate': 1.2148626817447498e-05, 'epoch': 0.09}

  1%|          | 376/41250 [54:58<98:46:36,  8.70s/it][2025-04-25 08:52:41,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 08:52:41,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.84 | bwd_microstep: 5746.39 | bwd_inner_microstep: 5646.75 | bwd_allreduce_microstep: 99.59 | step_microstep: 18.50
[2025-04-25 08:52:41,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.84 | bwd: 5746.40 | bwd_inner: 5646.75 | bwd_allreduce: 99.61 | step: 18.51

  1%|          | 377/41250 [55:06<98:39:41,  8.69s/it]
                                                      
{'loss': 0.1343, 'grad_norm': 1.8369051218032837, 'learning_rate': 1.2180936995153474e-05, 'epoch': 0.09}

  1%|          | 377/41250 [55:06<98:39:41,  8.69s/it][2025-04-25 08:52:50,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 08:52:50,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.48 | bwd_microstep: 5761.94 | bwd_inner_microstep: 5648.99 | bwd_allreduce_microstep: 112.90 | step_microstep: 19.04
[2025-04-25 08:52:50,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.48 | bwd: 5761.96 | bwd_inner: 5648.99 | bwd_allreduce: 112.92 | step: 19.04

  1%|          | 378/41250 [55:15<98:37:15,  8.69s/it]
                                                      
{'loss': 0.2103, 'grad_norm': 2.510881185531616, 'learning_rate': 1.2213247172859452e-05, 'epoch': 0.09}

  1%|          | 378/41250 [55:15<98:37:15,  8.69s/it][2025-04-25 08:52:58,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:52:58,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.88 | bwd_microstep: 5713.99 | bwd_inner_microstep: 5696.51 | bwd_allreduce_microstep: 17.43 | step_microstep: 18.40
[2025-04-25 08:52:58,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.88 | bwd: 5714.00 | bwd_inner: 5696.51 | bwd_allreduce: 17.45 | step: 18.40

  1%|          | 379/41250 [55:24<98:32:19,  8.68s/it]
                                                      
{'loss': 0.2698, 'grad_norm': 2.9790151119232178, 'learning_rate': 1.224555735056543e-05, 'epoch': 0.09}

  1%|          | 379/41250 [55:24<98:32:19,  8.68s/it][2025-04-25 08:53:07,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 08:53:07,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.42 | bwd_microstep: 5762.43 | bwd_inner_microstep: 5696.80 | bwd_allreduce_microstep: 65.58 | step_microstep: 19.38
[2025-04-25 08:53:07,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.42 | bwd: 5762.44 | bwd_inner: 5696.80 | bwd_allreduce: 65.60 | step: 19.39

  1%|          | 380/41250 [55:32<98:35:46,  8.68s/it]
                                                      
{'loss': 0.1995, 'grad_norm': 1.893347978591919, 'learning_rate': 1.2277867528271406e-05, 'epoch': 0.09}

  1%|          | 380/41250 [55:32<98:35:46,  8.68s/it][2025-04-25 08:53:16,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 08:53:16,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.52 | bwd_microstep: 5750.94 | bwd_inner_microstep: 5679.99 | bwd_allreduce_microstep: 70.91 | step_microstep: 18.73
[2025-04-25 08:53:16,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.52 | bwd: 5750.95 | bwd_inner: 5679.99 | bwd_allreduce: 70.93 | step: 18.74

  1%|          | 381/41250 [55:41<98:34:20,  8.68s/it]
                                                      
{'loss': 0.1558, 'grad_norm': 1.8937638998031616, 'learning_rate': 1.2310177705977384e-05, 'epoch': 0.09}

  1%|          | 381/41250 [55:41<98:34:20,  8.68s/it][2025-04-25 08:53:24,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 08:53:24,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.61 | bwd_microstep: 5719.70 | bwd_inner_microstep: 5706.99 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.83
[2025-04-25 08:53:24,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.61 | bwd: 5719.71 | bwd_inner: 5706.99 | bwd_allreduce: 12.68 | step: 18.83

  1%|          | 382/41250 [55:50<98:29:15,  8.68s/it]
                                                      
{'loss': 0.3622, 'grad_norm': 2.773750066757202, 'learning_rate': 1.234248788368336e-05, 'epoch': 0.09}

  1%|          | 382/41250 [55:50<98:29:15,  8.68s/it][2025-04-25 08:53:33,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 08:53:33,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5692.75 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.94
[2025-04-25 08:53:33,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5692.77 | bwd_inner: 5679.98 | bwd_allreduce: 12.75 | step: 18.94

  1%|          | 383/41250 [55:58<98:18:46,  8.66s/it]
                                                      
{'loss': 0.138, 'grad_norm': 1.443631649017334, 'learning_rate': 1.237479806138934e-05, 'epoch': 0.09}

  1%|          | 383/41250 [55:58<98:18:46,  8.66s/it][2025-04-25 08:53:42,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 08:53:42,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.81 | bwd_microstep: 5720.58 | bwd_inner_microstep: 5707.98 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.95
[2025-04-25 08:53:42,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.81 | bwd: 5720.59 | bwd_inner: 5707.98 | bwd_allreduce: 12.57 | step: 18.95

  1%|          | 384/41250 [56:07<98:17:24,  8.66s/it]
                                                      
{'loss': 0.1502, 'grad_norm': 2.960838794708252, 'learning_rate': 1.2407108239095316e-05, 'epoch': 0.09}

  1%|          | 384/41250 [56:07<98:17:24,  8.66s/it][2025-04-25 08:53:50,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 08:53:50,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.98 | bwd_microstep: 5852.46 | bwd_inner_microstep: 5702.16 | bwd_allreduce_microstep: 150.25 | step_microstep: 18.73
[2025-04-25 08:53:50,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.98 | bwd: 5852.47 | bwd_inner: 5702.16 | bwd_allreduce: 150.27 | step: 18.73

  1%|          | 385/41250 [56:16<98:44:15,  8.70s/it]
                                                      
{'loss': 0.196, 'grad_norm': 2.2636611461639404, 'learning_rate': 1.2439418416801294e-05, 'epoch': 0.09}

  1%|          | 385/41250 [56:16<98:44:15,  8.70s/it][2025-04-25 08:53:59,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 08:53:59,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.66 | bwd_microstep: 5676.69 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 32.96 | step_microstep: 18.27
[2025-04-25 08:53:59,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.66 | bwd: 5676.70 | bwd_inner: 5643.69 | bwd_allreduce: 32.98 | step: 18.27

  1%|          | 386/41250 [56:24<98:24:04,  8.67s/it]
                                                      
{'loss': 0.2012, 'grad_norm': 3.576580762863159, 'learning_rate': 1.2471728594507272e-05, 'epoch': 0.09}

  1%|          | 386/41250 [56:24<98:24:04,  8.67s/it][2025-04-25 08:54:08,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 08:54:08,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.33 | bwd_microstep: 5709.58 | bwd_inner_microstep: 5696.65 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.43
[2025-04-25 08:54:08,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.33 | bwd: 5709.59 | bwd_inner: 5696.65 | bwd_allreduce: 12.90 | step: 19.43

  1%|          | 387/41250 [56:33<98:20:36,  8.66s/it]
                                                      
{'loss': 0.1759, 'grad_norm': 2.226358413696289, 'learning_rate': 1.2504038772213248e-05, 'epoch': 0.09}

  1%|          | 387/41250 [56:33<98:20:36,  8.66s/it][2025-04-25 08:54:16,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 08:54:16,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5718.52 | bwd_inner_microstep: 5682.24 | bwd_allreduce_microstep: 36.24 | step_microstep: 18.83
[2025-04-25 08:54:16,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.83 | bwd: 5718.53 | bwd_inner: 5682.24 | bwd_allreduce: 36.25 | step: 18.83

  1%|          | 388/41250 [56:42<98:17:32,  8.66s/it]
                                                      
{'loss': 0.2692, 'grad_norm': 2.704127073287964, 'learning_rate': 1.2536348949919226e-05, 'epoch': 0.09}

  1%|          | 388/41250 [56:42<98:17:32,  8.66s/it][2025-04-25 08:54:25,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 08:54:25,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5725.43 | bwd_inner_microstep: 5681.53 | bwd_allreduce_microstep: 43.85 | step_microstep: 19.21
[2025-04-25 08:54:25,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5725.44 | bwd_inner: 5681.53 | bwd_allreduce: 43.87 | step: 19.21

  1%|          | 389/41250 [56:50<98:16:30,  8.66s/it]
                                                      
{'loss': 0.3237, 'grad_norm': 4.254935264587402, 'learning_rate': 1.2568659127625202e-05, 'epoch': 0.09}

  1%|          | 389/41250 [56:50<98:16:30,  8.66s/it][2025-04-25 08:54:34,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 08:54:34,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.06 | bwd_microstep: 5715.79 | bwd_inner_microstep: 5703.13 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.66
[2025-04-25 08:54:34,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.06 | bwd: 5715.81 | bwd_inner: 5703.13 | bwd_allreduce: 12.64 | step: 18.66

  1%|          | 390/41250 [56:59<98:18:43,  8.66s/it]
                                                      
{'loss': 0.0596, 'grad_norm': 1.5235176086425781, 'learning_rate': 1.2600969305331181e-05, 'epoch': 0.09}

  1%|          | 390/41250 [56:59<98:18:43,  8.66s/it][2025-04-25 08:54:42,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 08:54:42,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.56 | bwd_microstep: 5698.56 | bwd_inner_microstep: 5685.87 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.81
[2025-04-25 08:54:42,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.56 | bwd: 5698.57 | bwd_inner: 5685.87 | bwd_allreduce: 12.65 | step: 18.81

  1%|          | 391/41250 [57:08<98:13:09,  8.65s/it]
                                                      
{'loss': 0.0694, 'grad_norm': 1.2384824752807617, 'learning_rate': 1.2633279483037158e-05, 'epoch': 0.09}

  1%|          | 391/41250 [57:08<98:13:09,  8.65s/it][2025-04-25 08:54:51,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 08:54:51,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.48 | bwd_microstep: 5727.21 | bwd_inner_microstep: 5660.51 | bwd_allreduce_microstep: 66.65 | step_microstep: 19.01
[2025-04-25 08:54:51,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.48 | bwd: 5727.22 | bwd_inner: 5660.51 | bwd_allreduce: 66.68 | step: 19.01

  1%|          | 392/41250 [57:16<98:11:09,  8.65s/it]
                                                      
{'loss': 0.0934, 'grad_norm': 1.564616084098816, 'learning_rate': 1.2665589660743136e-05, 'epoch': 0.1}

  1%|          | 392/41250 [57:16<98:11:09,  8.65s/it][2025-04-25 08:55:00,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 08:55:00,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.56 | bwd_microstep: 5762.95 | bwd_inner_microstep: 5750.20 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.22
[2025-04-25 08:55:00,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.56 | bwd: 5762.96 | bwd_inner: 5750.20 | bwd_allreduce: 12.72 | step: 19.22

  1%|          | 393/41250 [57:25<98:26:03,  8.67s/it]
                                                      
{'loss': 0.1309, 'grad_norm': 1.8955953121185303, 'learning_rate': 1.2697899838449112e-05, 'epoch': 0.1}

  1%|          | 393/41250 [57:25<98:26:03,  8.67s/it][2025-04-25 08:55:08,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 08:55:08,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.52 | bwd_microstep: 5689.90 | bwd_inner_microstep: 5646.83 | bwd_allreduce_microstep: 43.03 | step_microstep: 18.83
[2025-04-25 08:55:08,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.52 | bwd: 5689.91 | bwd_inner: 5646.83 | bwd_allreduce: 43.04 | step: 18.83

  1%|          | 394/41250 [57:34<98:10:29,  8.65s/it]
                                                      
{'loss': 0.1239, 'grad_norm': 1.4713858366012573, 'learning_rate': 1.273021001615509e-05, 'epoch': 0.1}

  1%|          | 394/41250 [57:34<98:10:29,  8.65s/it][2025-04-25 08:55:17,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 08:55:17,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.98 | bwd_microstep: 5731.79 | bwd_inner_microstep: 5698.13 | bwd_allreduce_microstep: 33.61 | step_microstep: 18.67
[2025-04-25 08:55:17,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.98 | bwd: 5731.80 | bwd_inner: 5698.13 | bwd_allreduce: 33.63 | step: 18.67

  1%|          | 395/41250 [57:42<98:13:04,  8.65s/it]
                                                      
{'loss': 0.3697, 'grad_norm': 4.167075157165527, 'learning_rate': 1.2762520193861068e-05, 'epoch': 0.1}

  1%|          | 395/41250 [57:42<98:13:04,  8.65s/it][2025-04-25 08:55:26,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 08:55:26,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.62 | bwd_microstep: 5724.23 | bwd_inner_microstep: 5647.12 | bwd_allreduce_microstep: 77.06 | step_microstep: 18.32
[2025-04-25 08:55:26,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.62 | bwd: 5724.24 | bwd_inner: 5647.12 | bwd_allreduce: 77.08 | step: 18.33

  1%|          | 396/41250 [57:51<98:10:18,  8.65s/it]
                                                      
{'loss': 0.3058, 'grad_norm': 5.7155890464782715, 'learning_rate': 1.2794830371567044e-05, 'epoch': 0.1}

  1%|          | 396/41250 [57:51<98:10:18,  8.65s/it][2025-04-25 08:55:34,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 08:55:34,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.49 | bwd_microstep: 5715.49 | bwd_inner_microstep: 5702.70 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.01
[2025-04-25 08:55:34,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.49 | bwd: 5715.50 | bwd_inner: 5702.70 | bwd_allreduce: 12.76 | step: 19.01

  1%|          | 397/41250 [58:00<98:09:28,  8.65s/it]
                                                      
{'loss': 0.1085, 'grad_norm': 2.27651047706604, 'learning_rate': 1.2827140549273023e-05, 'epoch': 0.1}

  1%|          | 397/41250 [58:00<98:09:28,  8.65s/it][2025-04-25 08:55:43,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:55:43,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.95 | bwd_microstep: 5711.15 | bwd_inner_microstep: 5698.49 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.53
[2025-04-25 08:55:43,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5711.16 | bwd_inner: 5698.49 | bwd_allreduce: 12.63 | step: 18.53

  1%|          | 398/41250 [58:08<98:08:35,  8.65s/it]
                                                      
{'loss': 0.2115, 'grad_norm': 2.7824172973632812, 'learning_rate': 1.2859450726979e-05, 'epoch': 0.1}

  1%|          | 398/41250 [58:08<98:08:35,  8.65s/it][2025-04-25 08:55:52,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 08:55:52,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.92 | bwd_microstep: 5749.00 | bwd_inner_microstep: 5659.09 | bwd_allreduce_microstep: 89.86 | step_microstep: 17.98
[2025-04-25 08:55:52,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5749.01 | bwd_inner: 5659.09 | bwd_allreduce: 89.88 | step: 17.98

  1%|          | 399/41250 [58:17<98:10:39,  8.65s/it]
                                                      
{'loss': 0.1685, 'grad_norm': 2.5020205974578857, 'learning_rate': 1.2891760904684977e-05, 'epoch': 0.1}

  1%|          | 399/41250 [58:17<98:10:39,  8.65s/it][2025-04-25 08:56:00,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:56:00,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.53 | bwd_microstep: 5753.31 | bwd_inner_microstep: 5643.24 | bwd_allreduce_microstep: 110.02 | step_microstep: 18.62
[2025-04-25 08:56:00,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.53 | bwd: 5753.32 | bwd_inner: 5643.24 | bwd_allreduce: 110.04 | step: 18.62

  1%|          | 400/41250 [58:26<98:12:41,  8.66s/it]
                                                      
{'loss': 0.2683, 'grad_norm': 3.243835687637329, 'learning_rate': 1.2924071082390954e-05, 'epoch': 0.1}

  1%|          | 400/41250 [58:26<98:12:41,  8.66s/it][2025-04-25 08:56:09,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 08:56:09,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.54 | bwd_microstep: 5765.68 | bwd_inner_microstep: 5648.51 | bwd_allreduce_microstep: 117.12 | step_microstep: 19.00
[2025-04-25 08:56:09,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5765.69 | bwd_inner: 5648.51 | bwd_allreduce: 117.14 | step: 19.00

  1%|          | 401/41250 [58:34<98:18:31,  8.66s/it]
                                                      
{'loss': 0.3193, 'grad_norm': 3.4743235111236572, 'learning_rate': 1.2956381260096931e-05, 'epoch': 0.1}

  1%|          | 401/41250 [58:34<98:18:31,  8.66s/it][2025-04-25 08:56:18,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 08:56:18,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.15 | bwd_microstep: 5693.59 | bwd_inner_microstep: 5642.55 | bwd_allreduce_microstep: 50.99 | step_microstep: 18.70
[2025-04-25 08:56:18,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.15 | bwd: 5693.60 | bwd_inner: 5642.55 | bwd_allreduce: 51.01 | step: 18.71

  1%|          | 402/41250 [58:43<98:07:40,  8.65s/it]
                                                      
{'loss': 0.2212, 'grad_norm': 3.0073463916778564, 'learning_rate': 1.298869143780291e-05, 'epoch': 0.1}

  1%|          | 402/41250 [58:43<98:07:40,  8.65s/it][2025-04-25 08:56:26,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:56:26,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.84 | bwd_microstep: 5725.74 | bwd_inner_microstep: 5694.66 | bwd_allreduce_microstep: 31.04 | step_microstep: 18.69
[2025-04-25 08:56:26,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.84 | bwd: 5725.75 | bwd_inner: 5694.66 | bwd_allreduce: 31.05 | step: 18.69

  1%|          | 403/41250 [58:52<98:11:18,  8.65s/it]
                                                      
{'loss': 0.2302, 'grad_norm': 2.101890802383423, 'learning_rate': 1.3021001615508886e-05, 'epoch': 0.1}

  1%|          | 403/41250 [58:52<98:11:18,  8.65s/it][2025-04-25 08:56:35,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 08:56:35,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.71 | bwd_microstep: 5725.61 | bwd_inner_microstep: 5705.85 | bwd_allreduce_microstep: 19.71 | step_microstep: 18.14
[2025-04-25 08:56:35,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.71 | bwd: 5725.63 | bwd_inner: 5705.85 | bwd_allreduce: 19.73 | step: 18.14

  1%|          | 404/41250 [59:00<98:12:26,  8.66s/it]
                                                      
{'loss': 0.2608, 'grad_norm': 3.8084540367126465, 'learning_rate': 1.3053311793214865e-05, 'epoch': 0.1}

  1%|          | 404/41250 [59:00<98:12:26,  8.66s/it][2025-04-25 08:56:44,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:56:44,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.66 | bwd_microstep: 5785.66 | bwd_inner_microstep: 5655.50 | bwd_allreduce_microstep: 130.12 | step_microstep: 18.38
[2025-04-25 08:56:44,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.66 | bwd: 5785.67 | bwd_inner: 5655.50 | bwd_allreduce: 130.14 | step: 18.39

  1%|          | 405/41250 [59:09<98:20:53,  8.67s/it]
                                                      
{'loss': 0.2437, 'grad_norm': 5.0321431159973145, 'learning_rate': 1.3085621970920841e-05, 'epoch': 0.1}

  1%|          | 405/41250 [59:09<98:20:53,  8.67s/it][2025-04-25 08:56:52,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 08:56:52,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.79 | bwd_microstep: 5708.05 | bwd_inner_microstep: 5695.24 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.21
[2025-04-25 08:56:52,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.79 | bwd: 5708.06 | bwd_inner: 5695.24 | bwd_allreduce: 12.78 | step: 19.21

  1%|          | 406/41250 [59:18<98:19:13,  8.67s/it]
                                                      
{'loss': 0.2498, 'grad_norm': 3.5778744220733643, 'learning_rate': 1.3117932148626819e-05, 'epoch': 0.1}

  1%|          | 406/41250 [59:18<98:19:13,  8.67s/it][2025-04-25 08:57:01,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 08:57:01,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5743.02 | bwd_inner_microstep: 5688.23 | bwd_allreduce_microstep: 54.75 | step_microstep: 19.08
[2025-04-25 08:57:01,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5743.03 | bwd_inner: 5688.23 | bwd_allreduce: 54.76 | step: 19.08

  1%|          | 407/41250 [59:26<98:23:58,  8.67s/it]
                                                      
{'loss': 0.2211, 'grad_norm': 2.5515456199645996, 'learning_rate': 1.3150242326332795e-05, 'epoch': 0.1}

  1%|          | 407/41250 [59:26<98:23:58,  8.67s/it][2025-04-25 08:57:10,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 08:57:10,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.03 | bwd_microstep: 5774.85 | bwd_inner_microstep: 5650.28 | bwd_allreduce_microstep: 124.52 | step_microstep: 19.14
[2025-04-25 08:57:10,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.03 | bwd: 5774.87 | bwd_inner: 5650.28 | bwd_allreduce: 124.54 | step: 19.15

  1%|          | 408/41250 [59:35<98:28:17,  8.68s/it]
                                                      
{'loss': 0.2219, 'grad_norm': 2.246213436126709, 'learning_rate': 1.3182552504038773e-05, 'epoch': 0.1}

  1%|          | 408/41250 [59:35<98:28:17,  8.68s/it][2025-04-25 08:57:18,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 08:57:18,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.78 | bwd_microstep: 5758.15 | bwd_inner_microstep: 5662.11 | bwd_allreduce_microstep: 96.00 | step_microstep: 18.51
[2025-04-25 08:57:18,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.78 | bwd: 5758.16 | bwd_inner: 5662.11 | bwd_allreduce: 96.01 | step: 18.52

  1%|          | 409/41250 [59:44<98:28:17,  8.68s/it]
                                                      
{'loss': 0.0888, 'grad_norm': 1.7160539627075195, 'learning_rate': 1.3214862681744751e-05, 'epoch': 0.1}

  1%|          | 409/41250 [59:44<98:28:17,  8.68s/it][2025-04-25 08:57:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:57:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.30 | bwd_microstep: 5872.19 | bwd_inner_microstep: 5672.34 | bwd_allreduce_microstep: 199.81 | step_microstep: 18.41
[2025-04-25 08:57:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.30 | bwd: 5872.21 | bwd_inner: 5672.34 | bwd_allreduce: 199.83 | step: 18.41

  1%|          | 410/41250 [59:52<98:52:39,  8.72s/it]
                                                      
{'loss': 0.0784, 'grad_norm': 1.4604074954986572, 'learning_rate': 1.3247172859450727e-05, 'epoch': 0.1}

  1%|          | 410/41250 [59:52<98:52:39,  8.72s/it][2025-04-25 08:57:36,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 08:57:36,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5689.97 | bwd_inner_microstep: 5661.23 | bwd_allreduce_microstep: 28.69 | step_microstep: 18.99
[2025-04-25 08:57:36,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.59 | bwd: 5689.98 | bwd_inner: 5661.23 | bwd_allreduce: 28.71 | step: 19.00

  1%|          | 411/41250 [1:00:01<98:32:02,  8.69s/it]
                                                        
{'loss': 0.1242, 'grad_norm': 2.4625163078308105, 'learning_rate': 1.3279483037156707e-05, 'epoch': 0.1}

  1%|          | 411/41250 [1:00:01<98:32:02,  8.69s/it][2025-04-25 08:57:44,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 08:57:44,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.76 | bwd_microstep: 5735.13 | bwd_inner_microstep: 5722.24 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.75
[2025-04-25 08:57:44,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.76 | bwd: 5735.14 | bwd_inner: 5722.24 | bwd_allreduce: 12.86 | step: 18.75

  1%|          | 412/41250 [1:00:10<98:31:45,  8.69s/it]
                                                        
{'loss': 0.1019, 'grad_norm': 1.5462982654571533, 'learning_rate': 1.3311793214862683e-05, 'epoch': 0.1}

  1%|          | 412/41250 [1:00:10<98:31:45,  8.69s/it][2025-04-25 08:57:53,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:57:53,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.20 | bwd_microstep: 5785.63 | bwd_inner_microstep: 5653.05 | bwd_allreduce_microstep: 132.54 | step_microstep: 18.37
[2025-04-25 08:57:53,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.20 | bwd: 5785.64 | bwd_inner: 5653.05 | bwd_allreduce: 132.55 | step: 18.38

  1%|          | 413/41250 [1:00:18<98:35:42,  8.69s/it]
                                                        
{'loss': 0.3363, 'grad_norm': 3.1905808448791504, 'learning_rate': 1.334410339256866e-05, 'epoch': 0.1}

  1%|          | 413/41250 [1:00:18<98:35:42,  8.69s/it][2025-04-25 08:58:02,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 1.03
[2025-04-25 08:58:02,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5704.71 | bwd_inner_microstep: 5667.13 | bwd_allreduce_microstep: 37.53 | step_microstep: 18.12
[2025-04-25 08:58:02,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5704.72 | bwd_inner: 5667.13 | bwd_allreduce: 37.55 | step: 18.12

  1%|          | 414/41250 [1:00:27<98:22:55,  8.67s/it]
                                                        
{'loss': 0.2221, 'grad_norm': 3.7060141563415527, 'learning_rate': 1.3376413570274637e-05, 'epoch': 0.1}

  1%|          | 414/41250 [1:00:27<98:22:55,  8.67s/it][2025-04-25 08:58:10,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.20 | optimizer_step: 0.95
[2025-04-25 08:58:10,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.95 | bwd_microstep: 5701.03 | bwd_inner_microstep: 5672.11 | bwd_allreduce_microstep: 28.87 | step_microstep: 18.94
[2025-04-25 08:58:10,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.95 | bwd: 5701.04 | bwd_inner: 5672.11 | bwd_allreduce: 28.89 | step: 18.94

  1%|          | 415/41250 [1:00:36<98:12:58,  8.66s/it]
                                                        
{'loss': 0.2395, 'grad_norm': 5.540739059448242, 'learning_rate': 1.3408723747980615e-05, 'epoch': 0.1}

  1%|          | 415/41250 [1:00:36<98:12:58,  8.66s/it][2025-04-25 08:58:19,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 08:58:19,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5750.87 | bwd_inner_microstep: 5654.16 | bwd_allreduce_microstep: 96.65 | step_microstep: 18.73
[2025-04-25 08:58:19,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5750.88 | bwd_inner: 5654.16 | bwd_allreduce: 96.68 | step: 18.73

  1%|          | 416/41250 [1:00:44<98:18:29,  8.67s/it]
                                                        
{'loss': 0.1579, 'grad_norm': 3.2139060497283936, 'learning_rate': 1.3441033925686593e-05, 'epoch': 0.1}

  1%|          | 416/41250 [1:00:44<98:18:29,  8.67s/it][2025-04-25 08:58:28,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:58:28,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.61 | bwd_microstep: 5729.29 | bwd_inner_microstep: 5716.69 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.56
[2025-04-25 08:58:28,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.61 | bwd: 5729.31 | bwd_inner: 5716.69 | bwd_allreduce: 12.58 | step: 18.56

  1%|          | 417/41250 [1:00:53<98:18:14,  8.67s/it]
                                                        
{'loss': 0.5805, 'grad_norm': 6.002415180206299, 'learning_rate': 1.3473344103392569e-05, 'epoch': 0.1}

  1%|          | 417/41250 [1:00:53<98:18:14,  8.67s/it][2025-04-25 08:58:36,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:58:36,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.44 | bwd_microstep: 5862.63 | bwd_inner_microstep: 5696.20 | bwd_allreduce_microstep: 166.39 | step_microstep: 18.48
[2025-04-25 08:58:36,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.44 | bwd: 5862.65 | bwd_inner: 5696.20 | bwd_allreduce: 166.41 | step: 18.49

  1%|          | 418/41250 [1:01:02<98:46:28,  8.71s/it]
                                                        
{'loss': 0.1429, 'grad_norm': 2.408806085586548, 'learning_rate': 1.3505654281098549e-05, 'epoch': 0.1}

  1%|          | 418/41250 [1:01:02<98:46:28,  8.71s/it][2025-04-25 08:58:45,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 08:58:45,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.31 | bwd_microstep: 5809.70 | bwd_inner_microstep: 5671.32 | bwd_allreduce_microstep: 138.33 | step_microstep: 18.53
[2025-04-25 08:58:45,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.31 | bwd: 5809.71 | bwd_inner: 5671.32 | bwd_allreduce: 138.35 | step: 18.54

  1%|          | 419/41250 [1:01:11<98:51:32,  8.72s/it]
                                                        
{'loss': 0.3372, 'grad_norm': 6.489299297332764, 'learning_rate': 1.3537964458804525e-05, 'epoch': 0.1}

  1%|          | 419/41250 [1:01:11<98:51:32,  8.72s/it][2025-04-25 08:58:54,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 08:58:54,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.29 | bwd_microstep: 5785.69 | bwd_inner_microstep: 5657.37 | bwd_allreduce_microstep: 128.27 | step_microstep: 18.23
[2025-04-25 08:58:54,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.29 | bwd: 5785.70 | bwd_inner: 5657.37 | bwd_allreduce: 128.29 | step: 18.23

  1%|          | 420/41250 [1:01:19<98:48:27,  8.71s/it]
                                                        
{'loss': 0.2524, 'grad_norm': 3.9416024684906006, 'learning_rate': 1.3570274636510501e-05, 'epoch': 0.1}

  1%|          | 420/41250 [1:01:19<98:48:27,  8.71s/it][2025-04-25 08:59:03,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 08:59:03,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.19 | bwd_microstep: 5728.53 | bwd_inner_microstep: 5667.39 | bwd_allreduce_microstep: 61.08 | step_microstep: 18.61
[2025-04-25 08:59:03,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.19 | bwd: 5728.54 | bwd_inner: 5667.39 | bwd_allreduce: 61.11 | step: 18.61

  1%|          | 421/41250 [1:01:28<98:35:18,  8.69s/it]
                                                        
{'loss': 0.272, 'grad_norm': 4.099250793457031, 'learning_rate': 1.3602584814216479e-05, 'epoch': 0.1}

  1%|          | 421/41250 [1:01:28<98:35:18,  8.69s/it][2025-04-25 08:59:11,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 08:59:11,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.76 | bwd_microstep: 5897.82 | bwd_inner_microstep: 5703.57 | bwd_allreduce_microstep: 194.19 | step_microstep: 19.01
[2025-04-25 08:59:11,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.76 | bwd: 5897.83 | bwd_inner: 5703.57 | bwd_allreduce: 194.22 | step: 19.01

  1%|          | 422/41250 [1:01:37<99:05:39,  8.74s/it]
                                                        
{'loss': 0.3735, 'grad_norm': 3.251192569732666, 'learning_rate': 1.3634894991922457e-05, 'epoch': 0.1}

  1%|          | 422/41250 [1:01:37<99:05:39,  8.74s/it][2025-04-25 08:59:20,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 08:59:20,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5709.44 | bwd_inner_microstep: 5696.68 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.49
[2025-04-25 08:59:20,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5709.46 | bwd_inner: 5696.68 | bwd_allreduce: 12.74 | step: 18.49

  1%|          | 423/41250 [1:01:45<98:46:48,  8.71s/it]
                                                        
{'loss': 0.1606, 'grad_norm': 3.0936343669891357, 'learning_rate': 1.3667205169628435e-05, 'epoch': 0.1}

  1%|          | 423/41250 [1:01:45<98:46:48,  8.71s/it][2025-04-25 08:59:29,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 08:59:29,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.21 | bwd_microstep: 5770.71 | bwd_inner_microstep: 5716.25 | bwd_allreduce_microstep: 54.41 | step_microstep: 18.41
[2025-04-25 08:59:29,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.21 | bwd: 5770.72 | bwd_inner: 5716.26 | bwd_allreduce: 54.43 | step: 18.41

  1%|          | 424/41250 [1:01:54<98:47:26,  8.71s/it]
                                                        
{'loss': 0.3358, 'grad_norm': 3.0226404666900635, 'learning_rate': 1.3699515347334411e-05, 'epoch': 0.1}

  1%|          | 424/41250 [1:01:54<98:47:26,  8.71s/it][2025-04-25 08:59:37,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 08:59:37,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.27 | bwd_microstep: 5720.15 | bwd_inner_microstep: 5663.40 | bwd_allreduce_microstep: 56.71 | step_microstep: 18.63
[2025-04-25 08:59:37,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.27 | bwd: 5720.17 | bwd_inner: 5663.40 | bwd_allreduce: 56.73 | step: 18.63

  1%|          | 425/41250 [1:02:03<98:31:21,  8.69s/it]
                                                        
{'loss': 0.2579, 'grad_norm': 2.6838793754577637, 'learning_rate': 1.373182552504039e-05, 'epoch': 0.1}

  1%|          | 425/41250 [1:02:03<98:31:21,  8.69s/it][2025-04-25 08:59:46,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 08:59:46,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.42 | bwd_microstep: 5749.82 | bwd_inner_microstep: 5700.49 | bwd_allreduce_microstep: 49.28 | step_microstep: 18.87
[2025-04-25 08:59:46,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.42 | bwd: 5749.83 | bwd_inner: 5700.49 | bwd_allreduce: 49.30 | step: 18.87

  1%|          | 426/41250 [1:02:11<98:34:25,  8.69s/it]
                                                        
{'loss': 0.258, 'grad_norm': 2.206369638442993, 'learning_rate': 1.3764135702746367e-05, 'epoch': 0.1}

  1%|          | 426/41250 [1:02:11<98:34:25,  8.69s/it][2025-04-25 08:59:55,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 08:59:55,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.07 | bwd_microstep: 5713.36 | bwd_inner_microstep: 5700.43 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.34
[2025-04-25 08:59:55,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.07 | bwd: 5713.37 | bwd_inner: 5700.43 | bwd_allreduce: 12.90 | step: 18.34

  1%|          | 427/41250 [1:02:20<98:28:06,  8.68s/it]
                                                        
{'loss': 0.1546, 'grad_norm': 2.441969871520996, 'learning_rate': 1.3796445880452343e-05, 'epoch': 0.1}

  1%|          | 427/41250 [1:02:20<98:28:06,  8.68s/it][2025-04-25 09:00:03,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 09:00:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5774.60 | bwd_inner_microstep: 5692.82 | bwd_allreduce_microstep: 81.72 | step_microstep: 19.23
[2025-04-25 09:00:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5774.61 | bwd_inner: 5692.83 | bwd_allreduce: 81.75 | step: 19.24

  1%|          | 428/41250 [1:02:29<98:31:37,  8.69s/it]
                                                        
{'loss': 0.2056, 'grad_norm': 2.2503559589385986, 'learning_rate': 1.382875605815832e-05, 'epoch': 0.1}

  1%|          | 428/41250 [1:02:29<98:31:37,  8.69s/it][2025-04-25 09:00:12,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:00:12,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.75 | bwd_microstep: 5702.79 | bwd_inner_microstep: 5661.00 | bwd_allreduce_microstep: 41.74 | step_microstep: 18.59
[2025-04-25 09:00:12,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.75 | bwd: 5702.80 | bwd_inner: 5661.00 | bwd_allreduce: 41.76 | step: 18.60

  1%|          | 429/41250 [1:02:37<98:17:50,  8.67s/it]
                                                        
{'loss': 0.4271, 'grad_norm': 3.9879555702209473, 'learning_rate': 1.3861066235864299e-05, 'epoch': 0.1}

  1%|          | 429/41250 [1:02:37<98:17:50,  8.67s/it][2025-04-25 09:00:21,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:00:21,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5694.50 | bwd_inner_microstep: 5663.24 | bwd_allreduce_microstep: 31.22 | step_microstep: 18.41
[2025-04-25 09:00:21,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5694.52 | bwd_inner: 5663.24 | bwd_allreduce: 31.24 | step: 18.42

  1%|          | 430/41250 [1:02:46<98:08:23,  8.66s/it]
                                                        
{'loss': 0.1889, 'grad_norm': 1.858852505683899, 'learning_rate': 1.3893376413570277e-05, 'epoch': 0.1}

  1%|          | 430/41250 [1:02:46<98:08:23,  8.66s/it][2025-04-25 09:00:29,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:00:29,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.50 | bwd_microstep: 5699.71 | bwd_inner_microstep: 5687.00 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.31
[2025-04-25 09:00:29,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5699.73 | bwd_inner: 5687.00 | bwd_allreduce: 12.69 | step: 18.31

  1%|          | 431/41250 [1:02:55<98:05:47,  8.65s/it]
                                                        
{'loss': 0.0973, 'grad_norm': 1.348991870880127, 'learning_rate': 1.3925686591276253e-05, 'epoch': 0.1}

  1%|          | 431/41250 [1:02:55<98:05:47,  8.65s/it][2025-04-25 09:00:38,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-25 09:00:38,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5712.08 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.92
[2025-04-25 09:00:38,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5712.09 | bwd_inner: 5699.18 | bwd_allreduce: 12.87 | step: 18.92

  1%|          | 432/41250 [1:03:03<98:04:21,  8.65s/it]
                                                        
{'loss': 0.2435, 'grad_norm': 2.758455991744995, 'learning_rate': 1.3957996768982232e-05, 'epoch': 0.1}

  1%|          | 432/41250 [1:03:03<98:04:21,  8.65s/it][2025-04-25 09:00:47,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:00:47,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.68 | bwd_microstep: 5711.01 | bwd_inner_microstep: 5698.19 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.57
[2025-04-25 09:00:47,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.68 | bwd: 5711.02 | bwd_inner: 5698.19 | bwd_allreduce: 12.79 | step: 18.58

  1%|          | 433/41250 [1:03:12<98:03:07,  8.65s/it]
                                                        
{'loss': 0.263, 'grad_norm': 3.2347962856292725, 'learning_rate': 1.3990306946688208e-05, 'epoch': 0.1}

  1%|          | 433/41250 [1:03:12<98:03:07,  8.65s/it][2025-04-25 09:00:55,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 09:00:55,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.54 | bwd_microstep: 5765.43 | bwd_inner_microstep: 5651.59 | bwd_allreduce_microstep: 113.79 | step_microstep: 19.76
[2025-04-25 09:00:55,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.54 | bwd: 5765.44 | bwd_inner: 5651.59 | bwd_allreduce: 113.81 | step: 19.76

  1%|          | 434/41250 [1:03:21<98:10:52,  8.66s/it]
                                                        
{'loss': 0.213, 'grad_norm': 1.952966570854187, 'learning_rate': 1.4022617124394185e-05, 'epoch': 0.11}

  1%|          | 434/41250 [1:03:21<98:10:52,  8.66s/it][2025-04-25 09:01:04,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:01:04,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.38 | bwd_microstep: 5767.97 | bwd_inner_microstep: 5640.38 | bwd_allreduce_microstep: 127.55 | step_microstep: 18.31
[2025-04-25 09:01:04,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.38 | bwd: 5767.98 | bwd_inner: 5640.38 | bwd_allreduce: 127.56 | step: 18.31

  1%|          | 435/41250 [1:03:29<98:13:39,  8.66s/it]
                                                        
{'loss': 0.3052, 'grad_norm': 3.9901416301727295, 'learning_rate': 1.4054927302100163e-05, 'epoch': 0.11}

  1%|          | 435/41250 [1:03:29<98:13:39,  8.66s/it][2025-04-25 09:01:13,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:01:13,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.60 | bwd_microstep: 5750.85 | bwd_inner_microstep: 5649.96 | bwd_allreduce_microstep: 100.85 | step_microstep: 17.95
[2025-04-25 09:01:13,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.60 | bwd: 5750.86 | bwd_inner: 5649.96 | bwd_allreduce: 100.87 | step: 17.95

  1%|          | 436/41250 [1:03:38<98:12:56,  8.66s/it]
                                                        
{'loss': 0.0879, 'grad_norm': 0.9459832906723022, 'learning_rate': 1.4087237479806139e-05, 'epoch': 0.11}

  1%|          | 436/41250 [1:03:38<98:12:56,  8.66s/it][2025-04-25 09:01:21,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:01:21,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.62 | bwd_microstep: 5765.09 | bwd_inner_microstep: 5644.24 | bwd_allreduce_microstep: 120.81 | step_microstep: 18.71
[2025-04-25 09:01:21,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.62 | bwd: 5765.10 | bwd_inner: 5644.24 | bwd_allreduce: 120.83 | step: 18.71

  1%|          | 437/41250 [1:03:47<98:14:49,  8.67s/it]
                                                        
{'loss': 0.3469, 'grad_norm': 3.779057025909424, 'learning_rate': 1.4119547657512118e-05, 'epoch': 0.11}

  1%|          | 437/41250 [1:03:47<98:14:49,  8.67s/it][2025-04-25 09:01:30,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 09:01:30,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.59 | bwd_microstep: 5698.00 | bwd_inner_microstep: 5647.34 | bwd_allreduce_microstep: 50.61 | step_microstep: 19.19
[2025-04-25 09:01:30,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.59 | bwd: 5698.02 | bwd_inner: 5647.34 | bwd_allreduce: 50.63 | step: 19.20

  1%|          | 438/41250 [1:03:55<98:02:00,  8.65s/it]
                                                        
{'loss': 0.0967, 'grad_norm': 1.0667577981948853, 'learning_rate': 1.4151857835218094e-05, 'epoch': 0.11}

  1%|          | 438/41250 [1:03:55<98:02:00,  8.65s/it][2025-04-25 09:01:39,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 09:01:39,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.68 | bwd_microstep: 5703.34 | bwd_inner_microstep: 5690.57 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.31
[2025-04-25 09:01:39,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.68 | bwd: 5703.36 | bwd_inner: 5690.57 | bwd_allreduce: 12.75 | step: 19.31

  1%|          | 439/41250 [1:04:04<97:59:31,  8.64s/it]
                                                        
{'loss': 0.4231, 'grad_norm': 5.075942516326904, 'learning_rate': 1.4184168012924074e-05, 'epoch': 0.11}

  1%|          | 439/41250 [1:04:04<97:59:31,  8.64s/it][2025-04-25 09:01:47,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:01:47,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.36 | bwd_microstep: 5784.81 | bwd_inner_microstep: 5772.09 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.56
[2025-04-25 09:01:47,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.36 | bwd: 5784.83 | bwd_inner: 5772.09 | bwd_allreduce: 12.69 | step: 18.56

  1%|          | 440/41250 [1:04:13<98:21:52,  8.68s/it]
                                                        
{'loss': 0.1564, 'grad_norm': 1.869921088218689, 'learning_rate': 1.421647819063005e-05, 'epoch': 0.11}

  1%|          | 440/41250 [1:04:13<98:21:52,  8.68s/it][2025-04-25 09:01:56,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-25 09:01:56,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.69 | bwd_microstep: 5698.98 | bwd_inner_microstep: 5685.96 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.59
[2025-04-25 09:01:56,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.69 | bwd: 5698.99 | bwd_inner: 5685.96 | bwd_allreduce: 12.99 | step: 18.60

  1%|          | 441/41250 [1:04:21<98:12:36,  8.66s/it]
                                                        
{'loss': 0.3488, 'grad_norm': 4.5731706619262695, 'learning_rate': 1.4248788368336026e-05, 'epoch': 0.11}

  1%|          | 441/41250 [1:04:21<98:12:36,  8.66s/it][2025-04-25 09:02:05,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 09:02:05,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.93 | bwd_microstep: 5701.02 | bwd_inner_microstep: 5633.88 | bwd_allreduce_microstep: 67.09 | step_microstep: 18.74
[2025-04-25 09:02:05,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.93 | bwd: 5701.03 | bwd_inner: 5633.88 | bwd_allreduce: 67.11 | step: 18.74

  1%|          | 442/41250 [1:04:30<98:00:33,  8.65s/it]
                                                        
{'loss': 0.3449, 'grad_norm': 5.116671562194824, 'learning_rate': 1.4281098546042004e-05, 'epoch': 0.11}

  1%|          | 442/41250 [1:04:30<98:00:33,  8.65s/it][2025-04-25 09:02:13,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 09:02:13,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.03 | bwd_microstep: 5736.07 | bwd_inner_microstep: 5676.13 | bwd_allreduce_microstep: 59.89 | step_microstep: 18.46
[2025-04-25 09:02:13,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.03 | bwd: 5736.08 | bwd_inner: 5676.13 | bwd_allreduce: 59.91 | step: 18.46

  1%|          | 443/41250 [1:04:39<98:06:14,  8.65s/it]
                                                        
{'loss': 0.1241, 'grad_norm': 1.357592225074768, 'learning_rate': 1.431340872374798e-05, 'epoch': 0.11}

  1%|          | 443/41250 [1:04:39<98:06:14,  8.65s/it][2025-04-25 09:02:22,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:02:22,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.15 | bwd_microstep: 5694.20 | bwd_inner_microstep: 5639.77 | bwd_allreduce_microstep: 54.38 | step_microstep: 18.54
[2025-04-25 09:02:22,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.15 | bwd: 5694.21 | bwd_inner: 5639.77 | bwd_allreduce: 54.40 | step: 18.54

  1%|          | 444/41250 [1:04:47<97:54:46,  8.64s/it]
                                                        
{'loss': 0.2825, 'grad_norm': 2.813062906265259, 'learning_rate': 1.434571890145396e-05, 'epoch': 0.11}

  1%|          | 444/41250 [1:04:47<97:54:46,  8.64s/it][2025-04-25 09:02:31,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:02:31,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.74 | bwd_microstep: 5728.35 | bwd_inner_microstep: 5691.41 | bwd_allreduce_microstep: 36.89 | step_microstep: 18.80
[2025-04-25 09:02:31,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.74 | bwd: 5728.36 | bwd_inner: 5691.41 | bwd_allreduce: 36.91 | step: 18.81

  1%|          | 445/41250 [1:04:56<97:59:05,  8.64s/it]
                                                        
{'loss': 0.207, 'grad_norm': 2.115051746368408, 'learning_rate': 1.4378029079159936e-05, 'epoch': 0.11}

  1%|          | 445/41250 [1:04:56<97:59:05,  8.64s/it][2025-04-25 09:02:39,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 09:02:39,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.78 | bwd_microstep: 5763.90 | bwd_inner_microstep: 5643.04 | bwd_allreduce_microstep: 120.80 | step_microstep: 18.76
[2025-04-25 09:02:39,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.78 | bwd: 5763.91 | bwd_inner: 5643.04 | bwd_allreduce: 120.83 | step: 18.76

  1%|          | 446/41250 [1:05:05<98:05:38,  8.65s/it]
                                                        
{'loss': 0.3544, 'grad_norm': 2.3990962505340576, 'learning_rate': 1.4410339256865912e-05, 'epoch': 0.11}

  1%|          | 446/41250 [1:05:05<98:05:38,  8.65s/it][2025-04-25 09:02:48,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 09:02:48,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.93 | bwd_microstep: 5692.98 | bwd_inner_microstep: 5679.82 | bwd_allreduce_microstep: 13.11 | step_microstep: 18.96
[2025-04-25 09:02:48,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5693.00 | bwd_inner: 5679.82 | bwd_allreduce: 13.13 | step: 18.97

  1%|          | 447/41250 [1:05:13<97:58:44,  8.64s/it]
                                                        
{'loss': 0.194, 'grad_norm': 2.0921080112457275, 'learning_rate': 1.4442649434571892e-05, 'epoch': 0.11}

  1%|          | 447/41250 [1:05:13<97:58:44,  8.64s/it][2025-04-25 09:02:57,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 09:02:57,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.82 | bwd_microstep: 6215.24 | bwd_inner_microstep: 5677.52 | bwd_allreduce_microstep: 537.67 | step_microstep: 19.04
[2025-04-25 09:02:57,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 6215.25 | bwd_inner: 5677.52 | bwd_allreduce: 537.69 | step: 19.04

  1%|          | 448/41250 [1:05:22<99:43:59,  8.80s/it]
                                                        
{'loss': 0.1764, 'grad_norm': 2.9061763286590576, 'learning_rate': 1.4474959612277868e-05, 'epoch': 0.11}

  1%|          | 448/41250 [1:05:22<99:43:59,  8.80s/it][2025-04-25 09:03:06,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 09:03:06,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.82 | bwd_microstep: 5703.22 | bwd_inner_microstep: 5690.25 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.15
[2025-04-25 09:03:06,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.82 | bwd: 5703.23 | bwd_inner: 5690.25 | bwd_allreduce: 12.94 | step: 19.15

  1%|          | 449/41250 [1:05:31<99:12:07,  8.75s/it]
                                                        
{'loss': 0.1908, 'grad_norm': 2.4543473720550537, 'learning_rate': 1.4507269789983846e-05, 'epoch': 0.11}

  1%|          | 449/41250 [1:05:31<99:12:07,  8.75s/it][2025-04-25 09:03:14,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.30 | optimizer_step: 1.07
[2025-04-25 09:03:14,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.76 | bwd_microstep: 5712.52 | bwd_inner_microstep: 5698.70 | bwd_allreduce_microstep: 13.76 | step_microstep: 20.16
[2025-04-25 09:03:14,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.76 | bwd: 5712.53 | bwd_inner: 5698.70 | bwd_allreduce: 13.78 | step: 20.17

  1%|          | 450/41250 [1:05:40<98:52:59,  8.72s/it]
                                                        
{'loss': 0.316, 'grad_norm': 3.4788622856140137, 'learning_rate': 1.4539579967689822e-05, 'epoch': 0.11}

  1%|          | 450/41250 [1:05:40<98:52:59,  8.72s/it][2025-04-25 09:03:23,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 09:03:23,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5743.80 | bwd_inner_microstep: 5634.31 | bwd_allreduce_microstep: 109.45 | step_microstep: 18.61
[2025-04-25 09:03:23,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5743.81 | bwd_inner: 5634.31 | bwd_allreduce: 109.46 | step: 18.61

  1%|          | 451/41250 [1:05:48<98:39:24,  8.71s/it]
                                                        
{'loss': 0.144, 'grad_norm': 1.6443341970443726, 'learning_rate': 1.4571890145395802e-05, 'epoch': 0.11}

  1%|          | 451/41250 [1:05:48<98:39:24,  8.71s/it][2025-04-25 09:03:32,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 09:03:32,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.29 | bwd_microstep: 5790.80 | bwd_inner_microstep: 5764.87 | bwd_allreduce_microstep: 25.87 | step_microstep: 19.28
[2025-04-25 09:03:32,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.29 | bwd: 5790.81 | bwd_inner: 5764.87 | bwd_allreduce: 25.89 | step: 19.28

  1%|          | 452/41250 [1:05:57<98:50:11,  8.72s/it]
                                                        
{'loss': 0.1874, 'grad_norm': 2.1618521213531494, 'learning_rate': 1.4604200323101778e-05, 'epoch': 0.11}

  1%|          | 452/41250 [1:05:57<98:50:11,  8.72s/it][2025-04-25 09:03:40,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 09:03:40,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.61 | bwd_microstep: 5706.10 | bwd_inner_microstep: 5693.37 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.64
[2025-04-25 09:03:40,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.61 | bwd: 5706.12 | bwd_inner: 5693.37 | bwd_allreduce: 12.70 | step: 18.65

  1%|          | 453/41250 [1:06:06<98:35:37,  8.70s/it]
                                                        
{'loss': 0.209, 'grad_norm': 2.4104268550872803, 'learning_rate': 1.4636510500807754e-05, 'epoch': 0.11}

  1%|          | 453/41250 [1:06:06<98:35:37,  8.70s/it][2025-04-25 09:03:49,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 09:03:49,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.24 | bwd_microstep: 5760.02 | bwd_inner_microstep: 5632.74 | bwd_allreduce_microstep: 127.23 | step_microstep: 18.86
[2025-04-25 09:03:49,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.24 | bwd: 5760.03 | bwd_inner: 5632.74 | bwd_allreduce: 127.25 | step: 18.86

  1%|          | 454/41250 [1:06:14<98:28:20,  8.69s/it]
                                                        
{'loss': 0.3176, 'grad_norm': 3.4543204307556152, 'learning_rate': 1.4668820678513734e-05, 'epoch': 0.11}

  1%|          | 454/41250 [1:06:14<98:28:20,  8.69s/it][2025-04-25 09:03:58,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.18 | optimizer_step: 0.91
[2025-04-25 09:03:58,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.29 | bwd_microstep: 5719.93 | bwd_inner_microstep: 5677.15 | bwd_allreduce_microstep: 42.72 | step_microstep: 18.89
[2025-04-25 09:03:58,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.29 | bwd: 5719.94 | bwd_inner: 5677.15 | bwd_allreduce: 42.74 | step: 18.89

  1%|          | 455/41250 [1:06:23<98:19:10,  8.68s/it]
                                                        
{'loss': 0.1697, 'grad_norm': 1.8467174768447876, 'learning_rate': 1.470113085621971e-05, 'epoch': 0.11}

  1%|          | 455/41250 [1:06:23<98:19:10,  8.68s/it][2025-04-25 09:04:06,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 09:04:06,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5766.21 | bwd_inner_microstep: 5635.90 | bwd_allreduce_microstep: 130.25 | step_microstep: 19.01
[2025-04-25 09:04:06,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5766.22 | bwd_inner: 5635.90 | bwd_allreduce: 130.27 | step: 19.01

  1%|          | 456/41250 [1:06:32<98:19:28,  8.68s/it]
                                                        
{'loss': 0.3328, 'grad_norm': 2.5532612800598145, 'learning_rate': 1.4733441033925688e-05, 'epoch': 0.11}

  1%|          | 456/41250 [1:06:32<98:19:28,  8.68s/it][2025-04-25 09:04:15,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 09:04:15,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.04 | bwd_microstep: 5689.34 | bwd_inner_microstep: 5676.75 | bwd_allreduce_microstep: 12.54 | step_microstep: 19.08
[2025-04-25 09:04:15,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.04 | bwd: 5689.35 | bwd_inner: 5676.75 | bwd_allreduce: 12.56 | step: 19.08

  1%|          | 457/41250 [1:06:40<98:06:04,  8.66s/it]
                                                        
{'loss': 0.2005, 'grad_norm': 1.6659650802612305, 'learning_rate': 1.4765751211631664e-05, 'epoch': 0.11}

  1%|          | 457/41250 [1:06:40<98:06:04,  8.66s/it][2025-04-25 09:04:24,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 09:04:24,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.27 | bwd_microstep: 5733.45 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 39.38 | step_microstep: 18.81
[2025-04-25 09:04:24,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.27 | bwd: 5733.47 | bwd_inner: 5694.02 | bwd_allreduce: 39.40 | step: 18.81

  1%|          | 458/41250 [1:06:49<98:06:09,  8.66s/it]
                                                        
{'loss': 0.4898, 'grad_norm': 3.5957930088043213, 'learning_rate': 1.4798061389337644e-05, 'epoch': 0.11}

  1%|          | 458/41250 [1:06:49<98:06:09,  8.66s/it][2025-04-25 09:04:32,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:04:32,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.61 | bwd_microstep: 5703.61 | bwd_inner_microstep: 5690.83 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.78
[2025-04-25 09:04:32,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.61 | bwd: 5703.62 | bwd_inner: 5690.83 | bwd_allreduce: 12.74 | step: 18.78

  1%|          | 459/41250 [1:06:58<98:00:14,  8.65s/it]
                                                        
{'loss': 0.2806, 'grad_norm': 4.183459758758545, 'learning_rate': 1.483037156704362e-05, 'epoch': 0.11}

  1%|          | 459/41250 [1:06:58<98:00:14,  8.65s/it][2025-04-25 09:04:41,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 09:04:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5749.32 | bwd_inner_microstep: 5683.27 | bwd_allreduce_microstep: 66.01 | step_microstep: 18.86
[2025-04-25 09:04:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5749.34 | bwd_inner: 5683.26 | bwd_allreduce: 66.03 | step: 18.86

  1%|          | 460/41250 [1:07:06<98:05:45,  8.66s/it]
                                                        
{'loss': 0.3615, 'grad_norm': 4.548552989959717, 'learning_rate': 1.4862681744749596e-05, 'epoch': 0.11}

  1%|          | 460/41250 [1:07:06<98:05:45,  8.66s/it][2025-04-25 09:04:50,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.97 | optimizer_step: 1.10
[2025-04-25 09:04:50,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.10 | bwd_microstep: 5715.59 | bwd_inner_microstep: 5645.87 | bwd_allreduce_microstep: 69.68 | step_microstep: 18.80
[2025-04-25 09:04:50,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.10 | bwd: 5715.61 | bwd_inner: 5645.87 | bwd_allreduce: 69.70 | step: 18.80

  1%|          | 461/41250 [1:07:15<97:57:03,  8.65s/it]
                                                        
{'loss': 0.1292, 'grad_norm': 1.399004340171814, 'learning_rate': 1.4894991922455576e-05, 'epoch': 0.11}

  1%|          | 461/41250 [1:07:15<97:57:03,  8.65s/it][2025-04-25 09:04:58,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.95
[2025-04-25 09:04:58,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.51 | bwd_microstep: 5900.20 | bwd_inner_microstep: 5708.99 | bwd_allreduce_microstep: 191.17 | step_microstep: 18.42
[2025-04-25 09:04:58,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.51 | bwd: 5900.22 | bwd_inner: 5708.99 | bwd_allreduce: 191.19 | step: 18.42

  1%|          | 462/41250 [1:07:24<98:34:44,  8.70s/it]
                                                        
{'loss': 0.2848, 'grad_norm': 3.194340229034424, 'learning_rate': 1.4927302100161552e-05, 'epoch': 0.11}

  1%|          | 462/41250 [1:07:24<98:34:44,  8.70s/it][2025-04-25 09:05:07,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:05:07,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.34 | bwd_microstep: 5690.26 | bwd_inner_microstep: 5652.09 | bwd_allreduce_microstep: 38.13 | step_microstep: 18.58
[2025-04-25 09:05:07,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.34 | bwd: 5690.27 | bwd_inner: 5652.09 | bwd_allreduce: 38.15 | step: 18.59

  1%|          | 463/41250 [1:07:32<98:13:17,  8.67s/it]
                                                        
{'loss': 0.2448, 'grad_norm': 1.6615581512451172, 'learning_rate': 1.495961227786753e-05, 'epoch': 0.11}

  1%|          | 463/41250 [1:07:32<98:13:17,  8.67s/it][2025-04-25 09:05:16,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 09:05:16,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.10 | bwd_microstep: 5698.25 | bwd_inner_microstep: 5655.32 | bwd_allreduce_microstep: 42.89 | step_microstep: 19.18
[2025-04-25 09:05:16,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.10 | bwd: 5698.27 | bwd_inner: 5655.32 | bwd_allreduce: 42.91 | step: 19.18

  1%|          | 464/41250 [1:07:41<98:00:41,  8.65s/it]
                                                        
{'loss': 0.056, 'grad_norm': 1.0240745544433594, 'learning_rate': 1.4991922455573506e-05, 'epoch': 0.11}

  1%|          | 464/41250 [1:07:41<98:00:41,  8.65s/it][2025-04-25 09:05:24,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 09:05:24,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.07 | bwd_microstep: 5773.01 | bwd_inner_microstep: 5690.84 | bwd_allreduce_microstep: 82.13 | step_microstep: 18.70
[2025-04-25 09:05:24,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.07 | bwd: 5773.03 | bwd_inner: 5690.84 | bwd_allreduce: 82.15 | step: 18.71

  1%|          | 465/41250 [1:07:50<98:10:22,  8.67s/it]
                                                        
{'loss': 0.2067, 'grad_norm': 1.7652089595794678, 'learning_rate': 1.5024232633279485e-05, 'epoch': 0.11}

  1%|          | 465/41250 [1:07:50<98:10:22,  8.67s/it][2025-04-25 09:05:33,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:05:33,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.64 | bwd_microstep: 5706.31 | bwd_inner_microstep: 5661.27 | bwd_allreduce_microstep: 45.00 | step_microstep: 18.38
[2025-04-25 09:05:33,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.64 | bwd: 5706.33 | bwd_inner: 5661.27 | bwd_allreduce: 45.01 | step: 18.39

  1%|          | 466/41250 [1:07:58<98:00:34,  8.65s/it]
                                                        
{'loss': 0.5218, 'grad_norm': 3.8589439392089844, 'learning_rate': 1.5056542810985462e-05, 'epoch': 0.11}

  1%|          | 466/41250 [1:07:58<98:00:34,  8.65s/it][2025-04-25 09:05:42,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 09:05:42,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.73 | bwd_microstep: 5774.03 | bwd_inner_microstep: 5684.79 | bwd_allreduce_microstep: 89.20 | step_microstep: 18.62
[2025-04-25 09:05:42,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.73 | bwd: 5774.05 | bwd_inner: 5684.79 | bwd_allreduce: 89.21 | step: 18.63

  1%|          | 467/41250 [1:08:07<98:09:56,  8.67s/it]
                                                        
{'loss': 0.094, 'grad_norm': 1.3190721273422241, 'learning_rate': 1.5088852988691438e-05, 'epoch': 0.11}

  1%|          | 467/41250 [1:08:07<98:09:56,  8.67s/it][2025-04-25 09:05:50,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:05:50,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.92 | bwd_microstep: 5774.97 | bwd_inner_microstep: 5653.77 | bwd_allreduce_microstep: 121.15 | step_microstep: 18.62
[2025-04-25 09:05:50,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.92 | bwd: 5774.99 | bwd_inner: 5653.77 | bwd_allreduce: 121.17 | step: 18.62

  1%|          | 468/41250 [1:08:16<98:13:50,  8.67s/it]
                                                        
{'loss': 0.2641, 'grad_norm': 2.2348265647888184, 'learning_rate': 1.5121163166397417e-05, 'epoch': 0.11}

  1%|          | 468/41250 [1:08:16<98:13:50,  8.67s/it][2025-04-25 09:05:59,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 0.99
[2025-04-25 09:05:59,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.06 | bwd_microstep: 5712.54 | bwd_inner_microstep: 5664.67 | bwd_allreduce_microstep: 47.82 | step_microstep: 19.38
[2025-04-25 09:05:59,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.06 | bwd: 5712.56 | bwd_inner: 5664.67 | bwd_allreduce: 47.84 | step: 19.38

  1%|          | 469/41250 [1:08:24<98:03:35,  8.66s/it]
                                                        
{'loss': 0.2032, 'grad_norm': 1.6232118606567383, 'learning_rate': 1.5153473344103394e-05, 'epoch': 0.11}

  1%|          | 469/41250 [1:08:24<98:03:35,  8.66s/it][2025-04-25 09:06:08,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:06:08,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.13 | bwd_microstep: 5755.49 | bwd_inner_microstep: 5659.09 | bwd_allreduce_microstep: 96.35 | step_microstep: 18.06
[2025-04-25 09:06:08,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.13 | bwd: 5755.50 | bwd_inner: 5659.09 | bwd_allreduce: 96.37 | step: 18.07

  1%|          | 470/41250 [1:08:33<98:08:25,  8.66s/it]
                                                        
{'loss': 0.1963, 'grad_norm': 2.1764233112335205, 'learning_rate': 1.5185783521809372e-05, 'epoch': 0.11}

  1%|          | 470/41250 [1:08:33<98:08:25,  8.66s/it][2025-04-25 09:06:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:06:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.91 | bwd_microstep: 6029.78 | bwd_inner_microstep: 5708.72 | bwd_allreduce_microstep: 321.01 | step_microstep: 18.50
[2025-04-25 09:06:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.91 | bwd: 6029.79 | bwd_inner: 5708.72 | bwd_allreduce: 321.03 | step: 18.50

  1%|          | 471/41250 [1:08:42<99:10:13,  8.75s/it]
                                                        
{'loss': 0.2935, 'grad_norm': 2.8113489151000977, 'learning_rate': 1.5218093699515348e-05, 'epoch': 0.11}

  1%|          | 471/41250 [1:08:42<99:10:13,  8.75s/it][2025-04-25 09:06:25,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:06:25,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.48 | bwd_microstep: 5780.06 | bwd_inner_microstep: 5715.20 | bwd_allreduce_microstep: 64.81 | step_microstep: 18.37
[2025-04-25 09:06:25,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.48 | bwd: 5780.07 | bwd_inner: 5715.20 | bwd_allreduce: 64.83 | step: 18.37

  1%|          | 472/41250 [1:08:51<99:02:41,  8.74s/it]
                                                        
{'loss': 0.1883, 'grad_norm': 2.1008920669555664, 'learning_rate': 1.5250403877221327e-05, 'epoch': 0.11}

  1%|          | 472/41250 [1:08:51<99:02:41,  8.74s/it][2025-04-25 09:06:34,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.97
[2025-04-25 09:06:34,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.64 | bwd_microstep: 5795.37 | bwd_inner_microstep: 5782.75 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.62
[2025-04-25 09:06:34,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.64 | bwd: 5795.39 | bwd_inner: 5782.75 | bwd_allreduce: 12.60 | step: 18.63

  1%|          | 473/41250 [1:08:59<99:07:03,  8.75s/it]
                                                        
{'loss': 0.2544, 'grad_norm': 2.1022391319274902, 'learning_rate': 1.5282714054927302e-05, 'epoch': 0.11}

  1%|          | 473/41250 [1:08:59<99:07:03,  8.75s/it][2025-04-25 09:06:43,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 1.06
[2025-04-25 09:06:43,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5697.04 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.56
[2025-04-25 09:06:43,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5709.57 | bwd_inner: 5697.04 | bwd_allreduce: 12.49 | step: 18.56

  1%|          | 474/41250 [1:09:08<98:45:53,  8.72s/it]
                                                        
{'loss': 0.2852, 'grad_norm': 2.856140375137329, 'learning_rate': 1.531502423263328e-05, 'epoch': 0.11}

  1%|          | 474/41250 [1:09:08<98:45:53,  8.72s/it][2025-04-25 09:06:51,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.96 | optimizer_step: 0.97
[2025-04-25 09:06:51,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5731.56 | bwd_inner_microstep: 5698.67 | bwd_allreduce_microstep: 32.85 | step_microstep: 18.20
[2025-04-25 09:06:51,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.08 | bwd: 5731.58 | bwd_inner: 5698.67 | bwd_allreduce: 32.87 | step: 18.20

  1%|          | 475/41250 [1:09:17<98:34:06,  8.70s/it]
                                                        
{'loss': 0.2411, 'grad_norm': 2.3695273399353027, 'learning_rate': 1.5347334410339258e-05, 'epoch': 0.12}

  1%|          | 475/41250 [1:09:17<98:34:06,  8.70s/it][2025-04-25 09:07:00,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 09:07:00,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.49 | bwd_microstep: 5716.75 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.10
[2025-04-25 09:07:00,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.49 | bwd: 5716.76 | bwd_inner: 5704.05 | bwd_allreduce: 12.67 | step: 19.10

  1%|          | 476/41250 [1:09:25<98:23:12,  8.69s/it]
                                                        
{'loss': 0.3021, 'grad_norm': 3.720696449279785, 'learning_rate': 1.5379644588045235e-05, 'epoch': 0.12}

  1%|          | 476/41250 [1:09:25<98:23:12,  8.69s/it][2025-04-25 09:07:09,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 09:07:09,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5704.58 | bwd_inner_microstep: 5690.32 | bwd_allreduce_microstep: 14.22 | step_microstep: 19.25
[2025-04-25 09:07:09,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.26 | bwd: 5704.60 | bwd_inner: 5690.32 | bwd_allreduce: 14.24 | step: 19.25

  1%|          | 477/41250 [1:09:34<98:13:48,  8.67s/it]
                                                        
{'loss': 0.1996, 'grad_norm': 1.775390863418579, 'learning_rate': 1.5411954765751213e-05, 'epoch': 0.12}

  1%|          | 477/41250 [1:09:34<98:13:48,  8.67s/it][2025-04-25 09:07:17,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:07:17,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.75 | bwd_microstep: 5804.36 | bwd_inner_microstep: 5672.63 | bwd_allreduce_microstep: 131.69 | step_microstep: 18.64
[2025-04-25 09:07:17,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.75 | bwd: 5804.37 | bwd_inner: 5672.63 | bwd_allreduce: 131.71 | step: 18.65

  1%|          | 478/41250 [1:09:43<98:24:10,  8.69s/it]
                                                        
{'loss': 0.0904, 'grad_norm': 1.121603012084961, 'learning_rate': 1.544426494345719e-05, 'epoch': 0.12}

  1%|          | 478/41250 [1:09:43<98:24:10,  8.69s/it][2025-04-25 09:07:26,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:07:26,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.11 | bwd_microstep: 5925.44 | bwd_inner_microstep: 5668.05 | bwd_allreduce_microstep: 257.35 | step_microstep: 18.58
[2025-04-25 09:07:26,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.11 | bwd: 5925.45 | bwd_inner: 5668.05 | bwd_allreduce: 257.36 | step: 18.58

  1%|          | 479/41250 [1:09:52<98:56:00,  8.74s/it]
                                                        
{'loss': 0.2919, 'grad_norm': 3.0386178493499756, 'learning_rate': 1.547657512116317e-05, 'epoch': 0.12}

  1%|          | 479/41250 [1:09:52<98:56:00,  8.74s/it][2025-04-25 09:07:35,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:07:35,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.52 | bwd_microstep: 5789.03 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 88.81 | step_microstep: 18.67
[2025-04-25 09:07:35,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5789.04 | bwd_inner: 5700.17 | bwd_allreduce: 88.83 | step: 18.67

  1%|          | 480/41250 [1:10:00<98:52:08,  8.73s/it]
                                                        
{'loss': 0.2548, 'grad_norm': 2.7705416679382324, 'learning_rate': 1.5508885298869144e-05, 'epoch': 0.12}

  1%|          | 480/41250 [1:10:00<98:52:08,  8.73s/it][2025-04-25 09:07:44,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 09:07:44,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5780.78 | bwd_inner_microstep: 5659.33 | bwd_allreduce_microstep: 121.40 | step_microstep: 18.43
[2025-04-25 09:07:44,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5780.79 | bwd_inner: 5659.33 | bwd_allreduce: 121.42 | step: 18.44

  1%|          | 481/41250 [1:10:09<98:43:57,  8.72s/it]
                                                        
{'loss': 0.2956, 'grad_norm': 1.6936837434768677, 'learning_rate': 1.554119547657512e-05, 'epoch': 0.12}

  1%|          | 481/41250 [1:10:09<98:43:57,  8.72s/it][2025-04-25 09:07:52,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:07:52,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.62 | bwd_microstep: 5780.47 | bwd_inner_microstep: 5663.75 | bwd_allreduce_microstep: 116.68 | step_microstep: 18.83
[2025-04-25 09:07:52,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.62 | bwd: 5780.48 | bwd_inner: 5663.75 | bwd_allreduce: 116.69 | step: 18.83

  1%|          | 482/41250 [1:10:18<98:41:13,  8.71s/it]
                                                        
{'loss': 0.1398, 'grad_norm': 1.1259955167770386, 'learning_rate': 1.55735056542811e-05, 'epoch': 0.12}

  1%|          | 482/41250 [1:10:18<98:41:13,  8.71s/it][2025-04-25 09:08:01,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:08:01,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.59 | bwd_microstep: 5722.44 | bwd_inner_microstep: 5660.89 | bwd_allreduce_microstep: 61.50 | step_microstep: 18.47
[2025-04-25 09:08:01,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.59 | bwd: 5722.46 | bwd_inner: 5660.89 | bwd_allreduce: 61.52 | step: 18.48

  1%|          | 483/41250 [1:10:26<98:26:05,  8.69s/it]
                                                        
{'loss': 0.1427, 'grad_norm': 1.8154278993606567, 'learning_rate': 1.5605815831987077e-05, 'epoch': 0.12}

  1%|          | 483/41250 [1:10:26<98:26:05,  8.69s/it][2025-04-25 09:08:10,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.08 | optimizer_step: 0.99
[2025-04-25 09:08:10,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.30 | bwd_microstep: 5723.76 | bwd_inner_microstep: 5710.15 | bwd_allreduce_microstep: 13.55 | step_microstep: 19.30
[2025-04-25 09:08:10,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.30 | bwd: 5723.77 | bwd_inner: 5710.15 | bwd_allreduce: 13.58 | step: 19.30

  1%|          | 484/41250 [1:10:35<98:19:55,  8.68s/it]
                                                        
{'loss': 0.2202, 'grad_norm': 1.810075283050537, 'learning_rate': 1.5638126009693055e-05, 'epoch': 0.12}

  1%|          | 484/41250 [1:10:35<98:19:55,  8.68s/it][2025-04-25 09:08:18,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.18 | optimizer_step: 0.92
[2025-04-25 09:08:18,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.91 | bwd_microstep: 5742.14 | bwd_inner_microstep: 5697.87 | bwd_allreduce_microstep: 44.21 | step_microstep: 19.21
[2025-04-25 09:08:18,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.91 | bwd: 5742.15 | bwd_inner: 5697.87 | bwd_allreduce: 44.23 | step: 19.21

  1%|          | 485/41250 [1:10:44<98:17:29,  8.68s/it]
                                                        
{'loss': 0.162, 'grad_norm': 1.8692598342895508, 'learning_rate': 1.5670436187399033e-05, 'epoch': 0.12}

  1%|          | 485/41250 [1:10:44<98:17:29,  8.68s/it][2025-04-25 09:08:27,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 09:08:27,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.90 | bwd_microstep: 5743.71 | bwd_inner_microstep: 5716.26 | bwd_allreduce_microstep: 27.39 | step_microstep: 19.39
[2025-04-25 09:08:27,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.90 | bwd: 5743.72 | bwd_inner: 5716.26 | bwd_allreduce: 27.41 | step: 19.39

  1%|          | 486/41250 [1:10:52<98:18:51,  8.68s/it]
                                                        
{'loss': 0.1048, 'grad_norm': 1.676831603050232, 'learning_rate': 1.5702746365105007e-05, 'epoch': 0.12}

  1%|          | 486/41250 [1:10:52<98:18:51,  8.68s/it][2025-04-25 09:08:36,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 09:08:36,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5760.14 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 113.63 | step_microstep: 18.96
[2025-04-25 09:08:36,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5760.16 | bwd_inner: 5646.46 | bwd_allreduce: 113.65 | step: 18.96

  1%|          | 487/41250 [1:11:01<98:16:52,  8.68s/it]
                                                        
{'loss': 0.2041, 'grad_norm': 2.5517899990081787, 'learning_rate': 1.5735056542810985e-05, 'epoch': 0.12}

  1%|          | 487/41250 [1:11:01<98:16:52,  8.68s/it][2025-04-25 09:08:44,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:08:44,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5708.50 | bwd_inner_microstep: 5651.90 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.57
[2025-04-25 09:08:44,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5708.52 | bwd_inner: 5651.90 | bwd_allreduce: 56.58 | step: 18.57

  1%|          | 488/41250 [1:11:10<98:04:37,  8.66s/it]
                                                        
{'loss': 0.2392, 'grad_norm': 1.7806185483932495, 'learning_rate': 1.5767366720516963e-05, 'epoch': 0.12}

  1%|          | 488/41250 [1:11:10<98:04:37,  8.66s/it][2025-04-25 09:08:53,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:08:53,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.83 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5705.14 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.62
[2025-04-25 09:08:53,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.83 | bwd: 5717.95 | bwd_inner: 5705.14 | bwd_allreduce: 12.77 | step: 18.62

  1%|          | 489/41250 [1:11:18<98:06:33,  8.66s/it]
                                                        
{'loss': 0.1753, 'grad_norm': 1.6160722970962524, 'learning_rate': 1.579967689822294e-05, 'epoch': 0.12}

  1%|          | 489/41250 [1:11:18<98:06:33,  8.66s/it][2025-04-25 09:09:02,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 09:09:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.28 | bwd_microstep: 5893.23 | bwd_inner_microstep: 5648.26 | bwd_allreduce_microstep: 244.93 | step_microstep: 19.41
[2025-04-25 09:09:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.28 | bwd: 5893.25 | bwd_inner: 5648.26 | bwd_allreduce: 244.95 | step: 19.41

  1%|          | 490/41250 [1:11:27<98:35:16,  8.71s/it]
                                                        
{'loss': 0.2388, 'grad_norm': 2.5003981590270996, 'learning_rate': 1.583198707592892e-05, 'epoch': 0.12}

  1%|          | 490/41250 [1:11:27<98:35:16,  8.71s/it][2025-04-25 09:09:10,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:09:10,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.94 | bwd_microstep: 5748.39 | bwd_inner_microstep: 5698.44 | bwd_allreduce_microstep: 49.91 | step_microstep: 18.41
[2025-04-25 09:09:10,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.94 | bwd: 5748.40 | bwd_inner: 5698.44 | bwd_allreduce: 49.93 | step: 18.41

  1%|          | 491/41250 [1:11:36<98:30:31,  8.70s/it]
                                                        
{'loss': 0.2264, 'grad_norm': 2.361895799636841, 'learning_rate': 1.5864297253634897e-05, 'epoch': 0.12}

  1%|          | 491/41250 [1:11:36<98:30:31,  8.70s/it][2025-04-25 09:09:19,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:09:19,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.21 | bwd_microstep: 5709.14 | bwd_inner_microstep: 5696.39 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.61
[2025-04-25 09:09:19,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.21 | bwd: 5709.15 | bwd_inner: 5696.39 | bwd_allreduce: 12.72 | step: 18.61

  1%|          | 492/41250 [1:11:44<98:19:09,  8.68s/it]
                                                        
{'loss': 0.3008, 'grad_norm': 3.498396635055542, 'learning_rate': 1.5896607431340875e-05, 'epoch': 0.12}

  1%|          | 492/41250 [1:11:44<98:19:09,  8.68s/it][2025-04-25 09:09:28,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 09:09:28,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.08 | bwd_microstep: 5781.22 | bwd_inner_microstep: 5645.78 | bwd_allreduce_microstep: 135.39 | step_microstep: 19.02
[2025-04-25 09:09:28,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.08 | bwd: 5781.23 | bwd_inner: 5645.78 | bwd_allreduce: 135.41 | step: 19.02

  1%|          | 493/41250 [1:11:53<98:19:59,  8.69s/it]
                                                        
{'loss': 0.1066, 'grad_norm': 1.315002202987671, 'learning_rate': 1.592891760904685e-05, 'epoch': 0.12}

  1%|          | 493/41250 [1:11:53<98:19:59,  8.69s/it][2025-04-25 09:09:36,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 09:09:36,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5773.37 | bwd_inner_microstep: 5638.68 | bwd_allreduce_microstep: 134.65 | step_microstep: 18.72
[2025-04-25 09:09:36,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5773.38 | bwd_inner: 5638.68 | bwd_allreduce: 134.66 | step: 18.72

  1%|          | 494/41250 [1:12:02<98:19:47,  8.69s/it]
                                                        
{'loss': 0.2787, 'grad_norm': 2.794351100921631, 'learning_rate': 1.5961227786752827e-05, 'epoch': 0.12}

  1%|          | 494/41250 [1:12:02<98:19:47,  8.69s/it][2025-04-25 09:09:45,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 09:09:45,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.40 | bwd_microstep: 5779.56 | bwd_inner_microstep: 5647.93 | bwd_allreduce_microstep: 131.58 | step_microstep: 18.62
[2025-04-25 09:09:45,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.40 | bwd: 5779.57 | bwd_inner: 5647.93 | bwd_allreduce: 131.60 | step: 18.62

  1%|          | 495/41250 [1:12:10<98:20:12,  8.69s/it]
                                                        
{'loss': 0.1118, 'grad_norm': 1.5879484415054321, 'learning_rate': 1.5993537964458805e-05, 'epoch': 0.12}

  1%|          | 495/41250 [1:12:10<98:20:12,  8.69s/it][2025-04-25 09:09:54,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 09:09:54,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.89 | bwd_microstep: 5705.50 | bwd_inner_microstep: 5692.21 | bwd_allreduce_microstep: 13.24 | step_microstep: 18.49
[2025-04-25 09:09:54,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.89 | bwd: 5705.51 | bwd_inner: 5692.21 | bwd_allreduce: 13.26 | step: 18.50

  1%|          | 496/41250 [1:12:19<98:09:57,  8.67s/it]
                                                        
{'loss': 0.1549, 'grad_norm': 1.3220628499984741, 'learning_rate': 1.6025848142164783e-05, 'epoch': 0.12}

  1%|          | 496/41250 [1:12:19<98:09:57,  8.67s/it][2025-04-25 09:10:02,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:10:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.15 | bwd_microstep: 5787.12 | bwd_inner_microstep: 5683.46 | bwd_allreduce_microstep: 103.60 | step_microstep: 18.42
[2025-04-25 09:10:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.15 | bwd: 5787.13 | bwd_inner: 5683.46 | bwd_allreduce: 103.63 | step: 18.42

  1%|          | 497/41250 [1:12:28<98:20:32,  8.69s/it]
                                                        
{'loss': 0.2657, 'grad_norm': 2.319065570831299, 'learning_rate': 1.605815831987076e-05, 'epoch': 0.12}

  1%|          | 497/41250 [1:12:28<98:20:32,  8.69s/it][2025-04-25 09:10:11,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 09:10:11,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5741.09 | bwd_inner_microstep: 5699.73 | bwd_allreduce_microstep: 41.32 | step_microstep: 18.43
[2025-04-25 09:10:11,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5741.10 | bwd_inner: 5699.73 | bwd_allreduce: 41.34 | step: 18.44

  1%|          | 498/41250 [1:12:36<98:18:36,  8.68s/it]
                                                        
{'loss': 0.4058, 'grad_norm': 3.1739416122436523, 'learning_rate': 1.609046849757674e-05, 'epoch': 0.12}

  1%|          | 498/41250 [1:12:37<98:18:36,  8.68s/it][2025-04-25 09:10:20,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:10:20,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.73 | bwd_microstep: 5714.60 | bwd_inner_microstep: 5701.89 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.58
[2025-04-25 09:10:20,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.73 | bwd: 5714.61 | bwd_inner: 5701.89 | bwd_allreduce: 12.69 | step: 18.59

  1%|          | 499/41250 [1:12:45<98:40:46,  8.72s/it]
                                                        
{'loss': 0.1536, 'grad_norm': 1.5637847185134888, 'learning_rate': 1.6122778675282717e-05, 'epoch': 0.12}

  1%|          | 499/41250 [1:12:45<98:40:46,  8.72s/it][2025-04-25 09:10:29,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 09:10:29,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.29 | bwd_microstep: 5753.95 | bwd_inner_microstep: 5633.52 | bwd_allreduce_microstep: 120.38 | step_microstep: 18.39
[2025-04-25 09:10:29,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.29 | bwd: 5753.96 | bwd_inner: 5633.52 | bwd_allreduce: 120.40 | step: 18.39

  1%|          | 500/41250 [1:12:54<98:30:14,  8.70s/it]
                                                        
{'loss': 0.1407, 'grad_norm': 1.5131889581680298, 'learning_rate': 1.615508885298869e-05, 'epoch': 0.12}

  1%|          | 500/41250 [1:12:54<98:30:14,  8.70s/it][2025-04-25 09:10:37,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 09:10:37,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.39 | bwd_microstep: 5722.73 | bwd_inner_microstep: 5688.43 | bwd_allreduce_microstep: 34.25 | step_microstep: 18.82
[2025-04-25 09:10:37,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.39 | bwd: 5722.74 | bwd_inner: 5688.43 | bwd_allreduce: 34.27 | step: 18.82

  1%|          | 501/41250 [1:13:03<98:19:01,  8.69s/it]
                                                        
{'loss': 0.1319, 'grad_norm': 1.4151042699813843, 'learning_rate': 1.618739903069467e-05, 'epoch': 0.12}

  1%|          | 501/41250 [1:13:03<98:19:01,  8.69s/it][2025-04-25 09:10:46,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:10:46,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.34 | bwd_microstep: 5874.45 | bwd_inner_microstep: 5684.26 | bwd_allreduce_microstep: 190.14 | step_microstep: 18.29
[2025-04-25 09:10:46,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.34 | bwd: 5874.46 | bwd_inner: 5684.26 | bwd_allreduce: 190.16 | step: 18.30

  1%|          | 502/41250 [1:13:11<98:43:20,  8.72s/it]
                                                        
{'loss': 0.0652, 'grad_norm': 1.0309462547302246, 'learning_rate': 1.6219709208400647e-05, 'epoch': 0.12}

  1%|          | 502/41250 [1:13:11<98:43:20,  8.72s/it][2025-04-25 09:10:55,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:10:55,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.54 | bwd_microstep: 5679.91 | bwd_inner_microstep: 5667.28 | bwd_allreduce_microstep: 12.59 | step_microstep: 17.97
[2025-04-25 09:10:55,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.54 | bwd: 5679.93 | bwd_inner: 5667.28 | bwd_allreduce: 12.61 | step: 17.97

  1%|          | 503/41250 [1:13:20<98:20:22,  8.69s/it]
                                                        
{'loss': 0.3081, 'grad_norm': 3.442473888397217, 'learning_rate': 1.6252019386106625e-05, 'epoch': 0.12}

  1%|          | 503/41250 [1:13:20<98:20:22,  8.69s/it][2025-04-25 09:11:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.82 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:11:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.73 | bwd_microstep: 5692.74 | bwd_inner_microstep: 5635.98 | bwd_allreduce_microstep: 56.72 | step_microstep: 17.86
[2025-04-25 09:11:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.73 | bwd: 5692.75 | bwd_inner: 5635.98 | bwd_allreduce: 56.73 | step: 17.87

  1%|          | 504/41250 [1:13:29<98:01:25,  8.66s/it]
                                                        
{'loss': 0.2209, 'grad_norm': 2.307652473449707, 'learning_rate': 1.6284329563812603e-05, 'epoch': 0.12}

  1%|          | 504/41250 [1:13:29<98:01:25,  8.66s/it][2025-04-25 09:11:12,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 09:11:12,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.48 | bwd_microstep: 5689.42 | bwd_inner_microstep: 5642.31 | bwd_allreduce_microstep: 47.06 | step_microstep: 19.26
[2025-04-25 09:11:12,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.48 | bwd: 5689.44 | bwd_inner: 5642.31 | bwd_allreduce: 47.08 | step: 19.27

  1%|          | 505/41250 [1:13:37<97:49:58,  8.64s/it]
                                                        
{'loss': 0.2675, 'grad_norm': 2.267515182495117, 'learning_rate': 1.631663974151858e-05, 'epoch': 0.12}

  1%|          | 505/41250 [1:13:37<97:49:58,  8.64s/it][2025-04-25 09:11:21,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 09:11:21,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.12 | bwd_microstep: 5703.77 | bwd_inner_microstep: 5626.17 | bwd_allreduce_microstep: 77.55 | step_microstep: 18.74
[2025-04-25 09:11:21,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.12 | bwd: 5703.78 | bwd_inner: 5626.17 | bwd_allreduce: 77.57 | step: 18.75

  1%|          | 506/41250 [1:13:46<97:42:35,  8.63s/it]
                                                        
{'loss': 0.1957, 'grad_norm': 2.872994899749756, 'learning_rate': 1.634894991922456e-05, 'epoch': 0.12}

  1%|          | 506/41250 [1:13:46<97:42:35,  8.63s/it][2025-04-25 09:11:29,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 09:11:29,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.36 | bwd_microstep: 5704.55 | bwd_inner_microstep: 5691.81 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.56
[2025-04-25 09:11:29,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.37 | bwd: 5704.56 | bwd_inner: 5691.81 | bwd_allreduce: 12.71 | step: 18.57

  1%|          | 507/41250 [1:13:54<97:43:20,  8.63s/it]
                                                        
{'loss': 0.1233, 'grad_norm': 2.1023988723754883, 'learning_rate': 1.6381260096930533e-05, 'epoch': 0.12}

  1%|          | 507/41250 [1:13:54<97:43:20,  8.63s/it][2025-04-25 09:11:38,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 09:11:38,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5746.76 | bwd_inner_microstep: 5636.26 | bwd_allreduce_microstep: 110.45 | step_microstep: 18.16
[2025-04-25 09:11:38,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5746.77 | bwd_inner: 5636.26 | bwd_allreduce: 110.47 | step: 18.16

  1%|          | 508/41250 [1:14:03<97:47:40,  8.64s/it]
                                                        
{'loss': 0.2927, 'grad_norm': 3.4855504035949707, 'learning_rate': 1.641357027463651e-05, 'epoch': 0.12}

  1%|          | 508/41250 [1:14:03<97:47:40,  8.64s/it][2025-04-25 09:11:46,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 09:11:46,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.05 | bwd_microstep: 5735.12 | bwd_inner_microstep: 5678.67 | bwd_allreduce_microstep: 56.41 | step_microstep: 18.75
[2025-04-25 09:11:46,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.05 | bwd: 5735.13 | bwd_inner: 5678.67 | bwd_allreduce: 56.43 | step: 18.75

  1%|          | 509/41250 [1:14:12<97:52:41,  8.65s/it]
                                                        
{'loss': 0.2848, 'grad_norm': 4.841415882110596, 'learning_rate': 1.644588045234249e-05, 'epoch': 0.12}

  1%|          | 509/41250 [1:14:12<97:52:41,  8.65s/it][2025-04-25 09:11:55,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 09:11:55,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.23 | bwd_microstep: 5758.39 | bwd_inner_microstep: 5632.93 | bwd_allreduce_microstep: 125.42 | step_microstep: 18.56
[2025-04-25 09:11:55,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.23 | bwd: 5758.40 | bwd_inner: 5632.93 | bwd_allreduce: 125.43 | step: 18.57

  1%|          | 510/41250 [1:14:20<97:55:33,  8.65s/it]
                                                        
{'loss': 0.1245, 'grad_norm': 2.5508363246917725, 'learning_rate': 1.6478190630048467e-05, 'epoch': 0.12}

  1%|          | 510/41250 [1:14:20<97:55:33,  8.65s/it][2025-04-25 09:12:04,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 09:12:04,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.80 | bwd_microstep: 5782.80 | bwd_inner_microstep: 5764.08 | bwd_allreduce_microstep: 18.67 | step_microstep: 18.51
[2025-04-25 09:12:04,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.80 | bwd: 5782.82 | bwd_inner: 5764.08 | bwd_allreduce: 18.69 | step: 18.53

  1%|          | 511/41250 [1:14:29<98:16:29,  8.68s/it]
                                                        
{'loss': 0.2945, 'grad_norm': 2.553386926651001, 'learning_rate': 1.6510500807754444e-05, 'epoch': 0.12}

  1%|          | 511/41250 [1:14:29<98:16:29,  8.68s/it][2025-04-25 09:12:12,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 09:12:12,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.24 | bwd_microstep: 5688.10 | bwd_inner_microstep: 5633.96 | bwd_allreduce_microstep: 54.09 | step_microstep: 18.72
[2025-04-25 09:12:12,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.24 | bwd: 5688.11 | bwd_inner: 5633.96 | bwd_allreduce: 54.11 | step: 18.72

  1%|          | 512/41250 [1:14:38<97:59:54,  8.66s/it]
                                                        
{'loss': 0.1323, 'grad_norm': 1.6257163286209106, 'learning_rate': 1.6542810985460422e-05, 'epoch': 0.12}

  1%|          | 512/41250 [1:14:38<97:59:54,  8.66s/it][2025-04-25 09:12:21,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 09:12:21,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5687.28 | bwd_inner_microstep: 5647.46 | bwd_allreduce_microstep: 39.78 | step_microstep: 19.12
[2025-04-25 09:12:21,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5687.30 | bwd_inner: 5647.46 | bwd_allreduce: 39.80 | step: 19.13

  1%|          | 513/41250 [1:14:46<97:46:28,  8.64s/it]
                                                        
{'loss': 0.2488, 'grad_norm': 2.3870317935943604, 'learning_rate': 1.65751211631664e-05, 'epoch': 0.12}

  1%|          | 513/41250 [1:14:46<97:46:28,  8.64s/it][2025-04-25 09:12:30,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 09:12:30,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.89 | bwd_microstep: 5695.19 | bwd_inner_microstep: 5682.32 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.95
[2025-04-25 09:12:30,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.89 | bwd: 5695.20 | bwd_inner: 5682.32 | bwd_allreduce: 12.84 | step: 18.96

  1%|          | 514/41250 [1:14:55<97:45:42,  8.64s/it]
                                                        
{'loss': 0.2707, 'grad_norm': 4.007335662841797, 'learning_rate': 1.6607431340872375e-05, 'epoch': 0.12}

  1%|          | 514/41250 [1:14:55<97:45:42,  8.64s/it][2025-04-25 09:12:38,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:12:38,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5715.96 | bwd_inner_microstep: 5703.13 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.61
[2025-04-25 09:12:38,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5715.98 | bwd_inner: 5703.12 | bwd_allreduce: 12.81 | step: 18.62

  1%|          | 515/41250 [1:15:04<97:47:42,  8.64s/it]
                                                        
{'loss': 0.1886, 'grad_norm': 1.9333988428115845, 'learning_rate': 1.6639741518578353e-05, 'epoch': 0.12}

  1%|          | 515/41250 [1:15:04<97:47:42,  8.64s/it][2025-04-25 09:12:47,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:12:47,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.74 | bwd_microstep: 5704.04 | bwd_inner_microstep: 5660.51 | bwd_allreduce_microstep: 43.49 | step_microstep: 18.45
[2025-04-25 09:12:47,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.74 | bwd: 5704.05 | bwd_inner: 5660.51 | bwd_allreduce: 43.51 | step: 18.45

  1%|▏         | 516/41250 [1:15:12<97:41:11,  8.63s/it]
                                                        
{'loss': 0.1336, 'grad_norm': 1.701223611831665, 'learning_rate': 1.667205169628433e-05, 'epoch': 0.13}

  1%|▏         | 516/41250 [1:15:12<97:41:11,  8.63s/it][2025-04-25 09:12:56,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:12:56,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5987.96 | bwd_inner_microstep: 5701.85 | bwd_allreduce_microstep: 286.07 | step_microstep: 18.33
[2025-04-25 09:12:56,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5987.98 | bwd_inner: 5701.85 | bwd_allreduce: 286.09 | step: 18.34

  1%|▏         | 517/41250 [1:15:21<98:39:21,  8.72s/it]
                                                        
{'loss': 0.3113, 'grad_norm': 2.679239511489868, 'learning_rate': 1.670436187399031e-05, 'epoch': 0.13}

  1%|▏         | 517/41250 [1:15:21<98:39:21,  8.72s/it][2025-04-25 09:13:05,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:13:05,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5762.63 | bwd_inner_microstep: 5648.65 | bwd_allreduce_microstep: 113.93 | step_microstep: 18.18
[2025-04-25 09:13:05,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5762.64 | bwd_inner: 5648.65 | bwd_allreduce: 113.95 | step: 18.19

  1%|▏         | 518/41250 [1:15:30<98:28:35,  8.70s/it]
                                                        
{'loss': 0.1587, 'grad_norm': 2.412672519683838, 'learning_rate': 1.6736672051696286e-05, 'epoch': 0.13}

  1%|▏         | 518/41250 [1:15:30<98:28:35,  8.70s/it][2025-04-25 09:13:13,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 09:13:13,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.81 | bwd_microstep: 5714.46 | bwd_inner_microstep: 5702.23 | bwd_allreduce_microstep: 12.19 | step_microstep: 18.51
[2025-04-25 09:13:13,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.81 | bwd: 5714.47 | bwd_inner: 5702.23 | bwd_allreduce: 12.20 | step: 18.51

  1%|▏         | 519/41250 [1:15:39<98:19:54,  8.69s/it]
                                                        
{'loss': 0.0963, 'grad_norm': 1.488110899925232, 'learning_rate': 1.6768982229402264e-05, 'epoch': 0.13}

  1%|▏         | 519/41250 [1:15:39<98:19:54,  8.69s/it][2025-04-25 09:13:22,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:13:22,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.98 | bwd_microstep: 5700.40 | bwd_inner_microstep: 5656.74 | bwd_allreduce_microstep: 43.61 | step_microstep: 18.37
[2025-04-25 09:13:22,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.98 | bwd: 5700.41 | bwd_inner: 5656.74 | bwd_allreduce: 43.63 | step: 18.37

  1%|▏         | 520/41250 [1:15:47<98:03:40,  8.67s/it]
                                                        
{'loss': 0.208, 'grad_norm': 3.090118408203125, 'learning_rate': 1.6801292407108242e-05, 'epoch': 0.13}

  1%|▏         | 520/41250 [1:15:47<98:03:40,  8.67s/it][2025-04-25 09:13:30,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 09:13:30,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5692.68 | bwd_inner_microstep: 5680.01 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.13
[2025-04-25 09:13:30,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5692.69 | bwd_inner: 5680.01 | bwd_allreduce: 12.64 | step: 18.13

  1%|▏         | 521/41250 [1:15:56<97:54:12,  8.65s/it]
                                                        
{'loss': 0.1779, 'grad_norm': 2.5582494735717773, 'learning_rate': 1.6833602584814216e-05, 'epoch': 0.13}

  1%|▏         | 521/41250 [1:15:56<97:54:12,  8.65s/it][2025-04-25 09:13:39,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:13:39,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.70 | bwd_microstep: 5771.61 | bwd_inner_microstep: 5658.35 | bwd_allreduce_microstep: 113.20 | step_microstep: 18.61
[2025-04-25 09:13:39,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.70 | bwd: 5771.62 | bwd_inner: 5658.35 | bwd_allreduce: 113.22 | step: 18.62

  1%|▏         | 522/41250 [1:16:04<98:02:15,  8.67s/it]
                                                        
{'loss': 0.4145, 'grad_norm': 4.254329681396484, 'learning_rate': 1.6865912762520194e-05, 'epoch': 0.13}

  1%|▏         | 522/41250 [1:16:04<98:02:15,  8.67s/it][2025-04-25 09:13:48,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 09:13:48,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.25 | bwd_microstep: 5708.04 | bwd_inner_microstep: 5695.15 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.18
[2025-04-25 09:13:48,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.25 | bwd: 5708.05 | bwd_inner: 5695.15 | bwd_allreduce: 12.86 | step: 19.18

  1%|▏         | 523/41250 [1:16:13<97:59:30,  8.66s/it]
                                                        
{'loss': 0.2322, 'grad_norm': 2.2072315216064453, 'learning_rate': 1.6898222940226172e-05, 'epoch': 0.13}

  1%|▏         | 523/41250 [1:16:13<97:59:30,  8.66s/it][2025-04-25 09:13:57,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 09:13:57,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 5788.42 | bwd_inner_microstep: 5672.73 | bwd_allreduce_microstep: 115.64 | step_microstep: 18.51
[2025-04-25 09:13:57,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.17 | bwd: 5788.43 | bwd_inner: 5672.73 | bwd_allreduce: 115.66 | step: 18.52

  1%|▏         | 524/41250 [1:16:22<98:10:52,  8.68s/it]
                                                        
{'loss': 0.2051, 'grad_norm': 2.943916082382202, 'learning_rate': 1.693053311793215e-05, 'epoch': 0.13}

  1%|▏         | 524/41250 [1:16:22<98:10:52,  8.68s/it][2025-04-25 09:14:05,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:14:05,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3099.93 | bwd_microstep: 5717.65 | bwd_inner_microstep: 5704.79 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.62
[2025-04-25 09:14:05,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3099.93 | bwd: 5717.67 | bwd_inner: 5704.79 | bwd_allreduce: 12.84 | step: 18.62

  1%|▏         | 525/41250 [1:16:31<98:57:49,  8.75s/it]
                                                        
{'loss': 0.1842, 'grad_norm': 2.1290364265441895, 'learning_rate': 1.6962843295638128e-05, 'epoch': 0.13}

  1%|▏         | 525/41250 [1:16:31<98:57:49,  8.75s/it][2025-04-25 09:14:14,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 09:14:14,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.29 | bwd_microstep: 5763.79 | bwd_inner_microstep: 5717.94 | bwd_allreduce_microstep: 45.81 | step_microstep: 18.82
[2025-04-25 09:14:14,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.29 | bwd: 5763.81 | bwd_inner: 5717.94 | bwd_allreduce: 45.83 | step: 18.82

  1%|▏         | 526/41250 [1:16:39<98:49:18,  8.74s/it]
                                                        
{'loss': 0.1132, 'grad_norm': 1.7899640798568726, 'learning_rate': 1.6995153473344103e-05, 'epoch': 0.13}

  1%|▏         | 526/41250 [1:16:39<98:49:18,  8.74s/it][2025-04-25 09:14:23,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:14:23,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.59 | bwd_microstep: 5715.39 | bwd_inner_microstep: 5702.62 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.87
[2025-04-25 09:14:23,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.59 | bwd: 5715.40 | bwd_inner: 5702.62 | bwd_allreduce: 12.74 | step: 18.87

  1%|▏         | 527/41250 [1:16:48<98:31:42,  8.71s/it]
                                                        
{'loss': 0.0821, 'grad_norm': 1.4052395820617676, 'learning_rate': 1.7027463651050084e-05, 'epoch': 0.13}

  1%|▏         | 527/41250 [1:16:48<98:31:42,  8.71s/it][2025-04-25 09:14:31,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.09 | optimizer_step: 1.02
[2025-04-25 09:14:31,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.53 | bwd_microstep: 5716.03 | bwd_inner_microstep: 5702.52 | bwd_allreduce_microstep: 13.45 | step_microstep: 19.70
[2025-04-25 09:14:31,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.53 | bwd: 5716.04 | bwd_inner: 5702.52 | bwd_allreduce: 13.47 | step: 19.70

  1%|▏         | 528/41250 [1:16:57<98:20:14,  8.69s/it]
                                                        
{'loss': 0.1115, 'grad_norm': 2.3207192420959473, 'learning_rate': 1.7059773828756058e-05, 'epoch': 0.13}

  1%|▏         | 528/41250 [1:16:57<98:20:14,  8.69s/it][2025-04-25 09:14:40,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.18 | optimizer_step: 0.96
[2025-04-25 09:14:40,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.31 | bwd_microstep: 5778.05 | bwd_inner_microstep: 5688.01 | bwd_allreduce_microstep: 90.00 | step_microstep: 19.02
[2025-04-25 09:14:40,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.31 | bwd: 5778.06 | bwd_inner: 5688.01 | bwd_allreduce: 90.01 | step: 19.02

  1%|▏         | 529/41250 [1:17:05<98:23:17,  8.70s/it]
                                                        
{'loss': 0.1948, 'grad_norm': 2.1915886402130127, 'learning_rate': 1.7092084006462036e-05, 'epoch': 0.13}

  1%|▏         | 529/41250 [1:17:05<98:23:17,  8.70s/it][2025-04-25 09:14:49,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.17 | optimizer_step: 1.07
[2025-04-25 09:14:49,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.85 | bwd_microstep: 5776.96 | bwd_inner_microstep: 5716.67 | bwd_allreduce_microstep: 60.21 | step_microstep: 20.64
[2025-04-25 09:14:49,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.85 | bwd: 5776.98 | bwd_inner: 5716.67 | bwd_allreduce: 60.25 | step: 20.64

  1%|▏         | 530/41250 [1:17:14<98:29:26,  8.71s/it]
                                                        
{'loss': 0.4272, 'grad_norm': 4.161288738250732, 'learning_rate': 1.7124394184168014e-05, 'epoch': 0.13}

  1%|▏         | 530/41250 [1:17:14<98:29:26,  8.71s/it][2025-04-25 09:14:58,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 09:14:58,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.94 | bwd_microstep: 5724.96 | bwd_inner_microstep: 5712.22 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.91
[2025-04-25 09:14:58,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.94 | bwd: 5724.98 | bwd_inner: 5712.22 | bwd_allreduce: 12.72 | step: 18.92

  1%|▏         | 531/41250 [1:17:23<98:21:14,  8.70s/it]
                                                        
{'loss': 0.0822, 'grad_norm': 1.457287311553955, 'learning_rate': 1.7156704361873992e-05, 'epoch': 0.13}

  1%|▏         | 531/41250 [1:17:23<98:21:14,  8.70s/it][2025-04-25 09:15:06,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.30 | optimizer_step: 1.06
[2025-04-25 09:15:06,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.16 | bwd_microstep: 5714.22 | bwd_inner_microstep: 5668.91 | bwd_allreduce_microstep: 45.25 | step_microstep: 19.91
[2025-04-25 09:15:06,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.16 | bwd: 5714.24 | bwd_inner: 5668.91 | bwd_allreduce: 45.28 | step: 19.90

  1%|▏         | 532/41250 [1:17:32<98:09:29,  8.68s/it]
                                                        
{'loss': 0.1716, 'grad_norm': 1.72068190574646, 'learning_rate': 1.718901453957997e-05, 'epoch': 0.13}

  1%|▏         | 532/41250 [1:17:32<98:09:29,  8.68s/it][2025-04-25 09:15:15,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:15:15,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.21 | bwd_microstep: 5742.98 | bwd_inner_microstep: 5653.39 | bwd_allreduce_microstep: 89.55 | step_microstep: 18.56
[2025-04-25 09:15:15,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.21 | bwd: 5743.00 | bwd_inner: 5653.39 | bwd_allreduce: 89.57 | step: 18.57

  1%|▏         | 533/41250 [1:17:40<98:04:48,  8.67s/it]
                                                        
{'loss': 0.3627, 'grad_norm': 5.439889430999756, 'learning_rate': 1.7221324717285944e-05, 'epoch': 0.13}

  1%|▏         | 533/41250 [1:17:40<98:04:48,  8.67s/it][2025-04-25 09:15:24,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 09:15:24,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.49 | bwd_microstep: 5792.17 | bwd_inner_microstep: 5702.99 | bwd_allreduce_microstep: 89.14 | step_microstep: 19.40
[2025-04-25 09:15:24,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.49 | bwd: 5792.18 | bwd_inner: 5702.99 | bwd_allreduce: 89.15 | step: 19.40

  1%|▏         | 534/41250 [1:17:49<98:18:07,  8.69s/it]
                                                        
{'loss': 0.1801, 'grad_norm': 2.335252523422241, 'learning_rate': 1.7253634894991926e-05, 'epoch': 0.13}

  1%|▏         | 534/41250 [1:17:49<98:18:07,  8.69s/it][2025-04-25 09:15:32,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:15:32,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5779.35 | bwd_inner_microstep: 5652.50 | bwd_allreduce_microstep: 126.81 | step_microstep: 18.60
[2025-04-25 09:15:32,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5779.36 | bwd_inner: 5652.50 | bwd_allreduce: 126.82 | step: 18.60

  1%|▏         | 535/41250 [1:17:58<98:17:30,  8.69s/it]
                                                        
{'loss': 0.299, 'grad_norm': 2.701549530029297, 'learning_rate': 1.72859450726979e-05, 'epoch': 0.13}

  1%|▏         | 535/41250 [1:17:58<98:17:30,  8.69s/it][2025-04-25 09:15:41,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 09:15:41,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.07 | bwd_microstep: 5799.97 | bwd_inner_microstep: 5661.27 | bwd_allreduce_microstep: 138.65 | step_microstep: 18.77
[2025-04-25 09:15:41,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.07 | bwd: 5799.98 | bwd_inner: 5661.27 | bwd_allreduce: 138.67 | step: 18.78

  1%|▏         | 536/41250 [1:18:06<98:22:23,  8.70s/it]
                                                        
{'loss': 0.2041, 'grad_norm': 1.8519319295883179, 'learning_rate': 1.7318255250403878e-05, 'epoch': 0.13}

  1%|▏         | 536/41250 [1:18:06<98:22:23,  8.70s/it][2025-04-25 09:15:50,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:15:50,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.36 | bwd_microstep: 5701.27 | bwd_inner_microstep: 5688.46 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.88
[2025-04-25 09:15:50,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.36 | bwd: 5701.28 | bwd_inner: 5688.46 | bwd_allreduce: 12.78 | step: 18.89

  1%|▏         | 537/41250 [1:18:15<98:09:24,  8.68s/it]
                                                        
{'loss': 0.2669, 'grad_norm': 2.3746774196624756, 'learning_rate': 1.7350565428109856e-05, 'epoch': 0.13}

  1%|▏         | 537/41250 [1:18:15<98:09:24,  8.68s/it][2025-04-25 09:15:58,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 09:15:58,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.76 | bwd_microstep: 5758.02 | bwd_inner_microstep: 5687.22 | bwd_allreduce_microstep: 70.76 | step_microstep: 18.96
[2025-04-25 09:15:58,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.76 | bwd: 5758.03 | bwd_inner: 5687.22 | bwd_allreduce: 70.78 | step: 18.96

  1%|▏         | 538/41250 [1:18:24<98:11:17,  8.68s/it]
                                                        
{'loss': 0.1907, 'grad_norm': 3.316389560699463, 'learning_rate': 1.7382875605815834e-05, 'epoch': 0.13}

  1%|▏         | 538/41250 [1:18:24<98:11:17,  8.68s/it][2025-04-25 09:16:07,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.83 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 09:16:07,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.55 | bwd_microstep: 5734.10 | bwd_inner_microstep: 5712.10 | bwd_allreduce_microstep: 21.96 | step_microstep: 18.11
[2025-04-25 09:16:07,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.55 | bwd: 5734.12 | bwd_inner: 5712.10 | bwd_allreduce: 21.98 | step: 18.11

  1%|▏         | 539/41250 [1:18:32<98:11:29,  8.68s/it]
                                                        
{'loss': 0.1301, 'grad_norm': 1.6586731672286987, 'learning_rate': 1.741518578352181e-05, 'epoch': 0.13}

  1%|▏         | 539/41250 [1:18:32<98:11:29,  8.68s/it][2025-04-25 09:16:16,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:16:16,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5753.20 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 55.25 | step_microstep: 18.66
[2025-04-25 09:16:16,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5753.22 | bwd_inner: 5697.91 | bwd_allreduce: 55.27 | step: 18.67

  1%|▏         | 540/41250 [1:18:41<98:12:58,  8.69s/it]
                                                        
{'loss': 0.1258, 'grad_norm': 2.3430707454681396, 'learning_rate': 1.7447495961227786e-05, 'epoch': 0.13}

  1%|▏         | 540/41250 [1:18:41<98:12:58,  8.69s/it][2025-04-25 09:16:24,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:16:24,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5762.24 | bwd_inner_microstep: 5655.07 | bwd_allreduce_microstep: 107.12 | step_microstep: 18.37
[2025-04-25 09:16:24,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5762.26 | bwd_inner: 5655.07 | bwd_allreduce: 107.15 | step: 18.37

  1%|▏         | 541/41250 [1:18:50<98:12:30,  8.68s/it]
                                                        
{'loss': 0.2822, 'grad_norm': 2.462879180908203, 'learning_rate': 1.7479806138933767e-05, 'epoch': 0.13}

  1%|▏         | 541/41250 [1:18:50<98:12:30,  8.68s/it][2025-04-25 09:16:33,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 1.04
[2025-04-25 09:16:33,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.99 | bwd_microstep: 5709.87 | bwd_inner_microstep: 5649.95 | bwd_allreduce_microstep: 59.88 | step_microstep: 18.26
[2025-04-25 09:16:33,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.98 | bwd: 5709.88 | bwd_inner: 5649.94 | bwd_allreduce: 59.90 | step: 18.27

  1%|▏         | 542/41250 [1:18:58<97:59:02,  8.67s/it]
                                                        
{'loss': 0.2247, 'grad_norm': 1.9068013429641724, 'learning_rate': 1.7512116316639742e-05, 'epoch': 0.13}

  1%|▏         | 542/41250 [1:18:58<97:59:02,  8.67s/it][2025-04-25 09:16:42,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 09:16:42,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5755.60 | bwd_inner_microstep: 5650.56 | bwd_allreduce_microstep: 104.98 | step_microstep: 18.78
[2025-04-25 09:16:42,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5755.62 | bwd_inner: 5650.56 | bwd_allreduce: 105.01 | step: 18.78

  1%|▏         | 543/41250 [1:19:07<98:00:49,  8.67s/it]
                                                        
{'loss': 0.5193, 'grad_norm': 5.75699520111084, 'learning_rate': 1.754442649434572e-05, 'epoch': 0.13}

  1%|▏         | 543/41250 [1:19:07<98:00:49,  8.67s/it][2025-04-25 09:16:50,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 09:16:50,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5758.36 | bwd_inner_microstep: 5695.82 | bwd_allreduce_microstep: 62.50 | step_microstep: 19.19
[2025-04-25 09:16:50,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5758.38 | bwd_inner: 5695.82 | bwd_allreduce: 62.52 | step: 19.19

  1%|▏         | 544/41250 [1:19:16<98:04:59,  8.67s/it]
                                                        
{'loss': 0.2009, 'grad_norm': 2.7244813442230225, 'learning_rate': 1.7576736672051698e-05, 'epoch': 0.13}

  1%|▏         | 544/41250 [1:19:16<98:04:59,  8.67s/it][2025-04-25 09:16:59,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:16:59,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.48 | bwd_microstep: 5755.68 | bwd_inner_microstep: 5688.29 | bwd_allreduce_microstep: 67.35 | step_microstep: 18.77
[2025-04-25 09:16:59,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.48 | bwd: 5755.70 | bwd_inner: 5688.29 | bwd_allreduce: 67.36 | step: 18.77

  1%|▏         | 545/41250 [1:19:24<98:08:28,  8.68s/it]
                                                        
{'loss': 0.2663, 'grad_norm': 3.1295061111450195, 'learning_rate': 1.7609046849757676e-05, 'epoch': 0.13}

  1%|▏         | 545/41250 [1:19:24<98:08:28,  8.68s/it][2025-04-25 09:17:08,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:17:08,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.58 | bwd_microstep: 5770.46 | bwd_inner_microstep: 5657.85 | bwd_allreduce_microstep: 112.57 | step_microstep: 18.30
[2025-04-25 09:17:08,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.58 | bwd: 5770.48 | bwd_inner: 5657.85 | bwd_allreduce: 112.58 | step: 18.30

  1%|▏         | 546/41250 [1:19:33<98:09:36,  8.68s/it]
                                                        
{'loss': 0.22, 'grad_norm': 3.0360257625579834, 'learning_rate': 1.7641357027463653e-05, 'epoch': 0.13}

  1%|▏         | 546/41250 [1:19:33<98:09:36,  8.68s/it][2025-04-25 09:17:16,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:17:16,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.06 | bwd_microstep: 5756.03 | bwd_inner_microstep: 5651.44 | bwd_allreduce_microstep: 104.55 | step_microstep: 18.40
[2025-04-25 09:17:16,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.06 | bwd: 5756.04 | bwd_inner: 5651.44 | bwd_allreduce: 104.57 | step: 18.40

  1%|▏         | 547/41250 [1:19:42<98:07:37,  8.68s/it]
                                                        
{'loss': 0.3083, 'grad_norm': 2.577228546142578, 'learning_rate': 1.7673667205169628e-05, 'epoch': 0.13}

  1%|▏         | 547/41250 [1:19:42<98:07:37,  8.68s/it][2025-04-25 09:17:25,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:17:25,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.31 | bwd_microstep: 5778.43 | bwd_inner_microstep: 5641.99 | bwd_allreduce_microstep: 136.39 | step_microstep: 18.50
[2025-04-25 09:17:25,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.31 | bwd: 5778.44 | bwd_inner: 5641.99 | bwd_allreduce: 136.40 | step: 18.50

  1%|▏         | 548/41250 [1:19:50<98:09:15,  8.68s/it]
                                                        
{'loss': 0.3515, 'grad_norm': 3.311626434326172, 'learning_rate': 1.770597738287561e-05, 'epoch': 0.13}

  1%|▏         | 548/41250 [1:19:50<98:09:15,  8.68s/it][2025-04-25 09:17:34,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 09:17:34,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.61 | bwd_microstep: 5804.13 | bwd_inner_microstep: 5791.35 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.90
[2025-04-25 09:17:34,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.61 | bwd: 5804.15 | bwd_inner: 5791.35 | bwd_allreduce: 12.74 | step: 18.90

  1%|▏         | 549/41250 [1:19:59<98:29:54,  8.71s/it]
                                                        
{'loss': 0.1206, 'grad_norm': 1.1978572607040405, 'learning_rate': 1.7738287560581584e-05, 'epoch': 0.13}

  1%|▏         | 549/41250 [1:19:59<98:29:54,  8.71s/it][2025-04-25 09:17:42,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:17:42,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.49 | bwd_microstep: 5692.13 | bwd_inner_microstep: 5650.46 | bwd_allreduce_microstep: 41.63 | step_microstep: 17.91
[2025-04-25 09:17:42,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.49 | bwd: 5692.15 | bwd_inner: 5650.46 | bwd_allreduce: 41.65 | step: 17.91

  1%|▏         | 550/41250 [1:20:08<98:10:24,  8.68s/it]
                                                        
{'loss': 0.1588, 'grad_norm': 2.2148265838623047, 'learning_rate': 1.777059773828756e-05, 'epoch': 0.13}

  1%|▏         | 550/41250 [1:20:08<98:10:24,  8.68s/it][2025-04-25 09:17:51,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 09:17:51,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.88 | bwd_microstep: 5716.20 | bwd_inner_microstep: 5638.61 | bwd_allreduce_microstep: 77.55 | step_microstep: 17.79
[2025-04-25 09:17:51,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.88 | bwd: 5716.21 | bwd_inner: 5638.61 | bwd_allreduce: 77.56 | step: 17.79

  1%|▏         | 551/41250 [1:20:16<97:58:52,  8.67s/it]
                                                        
{'loss': 0.0913, 'grad_norm': 1.2263234853744507, 'learning_rate': 1.780290791599354e-05, 'epoch': 0.13}

  1%|▏         | 551/41250 [1:20:16<97:58:52,  8.67s/it][2025-04-25 09:18:00,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 09:18:00,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.58 | bwd_microstep: 5679.98 | bwd_inner_microstep: 5663.74 | bwd_allreduce_microstep: 16.20 | step_microstep: 18.58
[2025-04-25 09:18:00,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.58 | bwd: 5680.00 | bwd_inner: 5663.74 | bwd_allreduce: 16.22 | step: 18.58

  1%|▏         | 552/41250 [1:20:25<97:44:34,  8.65s/it]
                                                        
{'loss': 0.2106, 'grad_norm': 1.976173758506775, 'learning_rate': 1.7835218093699517e-05, 'epoch': 0.13}

  1%|▏         | 552/41250 [1:20:25<97:44:34,  8.65s/it][2025-04-25 09:18:08,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:18:08,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5704.89 | bwd_inner_microstep: 5691.90 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.36
[2025-04-25 09:18:08,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5704.90 | bwd_inner: 5691.90 | bwd_allreduce: 12.96 | step: 18.36

  1%|▏         | 553/41250 [1:20:34<97:40:59,  8.64s/it]
                                                        
{'loss': 0.4107, 'grad_norm': 3.0804953575134277, 'learning_rate': 1.7867528271405495e-05, 'epoch': 0.13}

  1%|▏         | 553/41250 [1:20:34<97:40:59,  8.64s/it][2025-04-25 09:18:17,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 09:18:17,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.27 | bwd_microstep: 5728.80 | bwd_inner_microstep: 5699.08 | bwd_allreduce_microstep: 29.68 | step_microstep: 18.70
[2025-04-25 09:18:17,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.27 | bwd: 5728.82 | bwd_inner: 5699.08 | bwd_allreduce: 29.70 | step: 18.70

  1%|▏         | 554/41250 [1:20:42<97:47:28,  8.65s/it]
                                                        
{'loss': 0.1294, 'grad_norm': 1.813137412071228, 'learning_rate': 1.789983844911147e-05, 'epoch': 0.13}

  1%|▏         | 554/41250 [1:20:42<97:47:28,  8.65s/it][2025-04-25 09:18:26,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:18:26,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5768.96 | bwd_inner_microstep: 5693.54 | bwd_allreduce_microstep: 75.38 | step_microstep: 18.29
[2025-04-25 09:18:26,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5768.97 | bwd_inner: 5693.54 | bwd_allreduce: 75.40 | step: 18.29

  1%|▏         | 555/41250 [1:20:51<97:56:53,  8.66s/it]
                                                        
{'loss': 0.0949, 'grad_norm': 1.7510513067245483, 'learning_rate': 1.793214862681745e-05, 'epoch': 0.13}

  1%|▏         | 555/41250 [1:20:51<97:56:53,  8.66s/it][2025-04-25 09:18:34,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.15 | optimizer_step: 0.98
[2025-04-25 09:18:34,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.79 | bwd_microstep: 5754.49 | bwd_inner_microstep: 5657.54 | bwd_allreduce_microstep: 96.89 | step_microstep: 19.21
[2025-04-25 09:18:34,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.79 | bwd: 5754.50 | bwd_inner: 5657.54 | bwd_allreduce: 96.91 | step: 19.21

  1%|▏         | 556/41250 [1:21:00<97:59:16,  8.67s/it]
                                                        
{'loss': 0.1976, 'grad_norm': 1.6600050926208496, 'learning_rate': 1.7964458804523425e-05, 'epoch': 0.13}

  1%|▏         | 556/41250 [1:21:00<97:59:16,  8.67s/it][2025-04-25 09:18:43,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 09:18:43,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.09 | bwd_microstep: 5725.90 | bwd_inner_microstep: 5654.99 | bwd_allreduce_microstep: 70.86 | step_microstep: 18.89
[2025-04-25 09:18:43,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.09 | bwd: 5725.91 | bwd_inner: 5654.99 | bwd_allreduce: 70.88 | step: 18.90

  1%|▏         | 557/41250 [1:21:08<97:55:15,  8.66s/it]
                                                        
{'loss': 0.2577, 'grad_norm': 2.2516233921051025, 'learning_rate': 1.7996768982229403e-05, 'epoch': 0.14}

  1%|▏         | 557/41250 [1:21:08<97:55:15,  8.66s/it][2025-04-25 09:18:52,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:18:52,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.95 | bwd_microstep: 5714.92 | bwd_inner_microstep: 5682.49 | bwd_allreduce_microstep: 32.38 | step_microstep: 18.98
[2025-04-25 09:18:52,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.95 | bwd: 5714.94 | bwd_inner: 5682.49 | bwd_allreduce: 32.40 | step: 18.98

  1%|▏         | 558/41250 [1:21:17<97:53:59,  8.66s/it]
                                                        
{'loss': 0.1886, 'grad_norm': 1.7850261926651, 'learning_rate': 1.802907915993538e-05, 'epoch': 0.14}

  1%|▏         | 558/41250 [1:21:17<97:53:59,  8.66s/it][2025-04-25 09:19:00,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 09:19:00,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.66 | bwd_microstep: 5684.16 | bwd_inner_microstep: 5652.84 | bwd_allreduce_microstep: 31.29 | step_microstep: 19.23
[2025-04-25 09:19:00,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.66 | bwd: 5684.18 | bwd_inner: 5652.84 | bwd_allreduce: 31.30 | step: 19.23

  1%|▏         | 559/41250 [1:21:26<97:42:04,  8.64s/it]
                                                        
{'loss': 0.2121, 'grad_norm': 3.2889750003814697, 'learning_rate': 1.806138933764136e-05, 'epoch': 0.14}

  1%|▏         | 559/41250 [1:21:26<97:42:04,  8.64s/it][2025-04-25 09:19:09,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:19:09,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.95 | bwd_microstep: 5721.44 | bwd_inner_microstep: 5683.99 | bwd_allreduce_microstep: 37.40 | step_microstep: 18.57
[2025-04-25 09:19:09,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.95 | bwd: 5721.45 | bwd_inner: 5683.99 | bwd_allreduce: 37.42 | step: 18.57

  1%|▏         | 560/41250 [1:21:34<97:45:29,  8.65s/it]
                                                        
{'loss': 0.3698, 'grad_norm': 2.989825963973999, 'learning_rate': 1.8093699515347337e-05, 'epoch': 0.14}

  1%|▏         | 560/41250 [1:21:34<97:45:29,  8.65s/it][2025-04-25 09:19:18,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:19:18,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.03 | bwd_microstep: 5776.74 | bwd_inner_microstep: 5646.74 | bwd_allreduce_microstep: 129.95 | step_microstep: 18.56
[2025-04-25 09:19:18,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.03 | bwd: 5776.75 | bwd_inner: 5646.74 | bwd_allreduce: 129.97 | step: 18.57

  1%|▏         | 561/41250 [1:21:43<97:51:32,  8.66s/it]
                                                        
{'loss': 0.1335, 'grad_norm': 1.4639878273010254, 'learning_rate': 1.812600969305331e-05, 'epoch': 0.14}

  1%|▏         | 561/41250 [1:21:43<97:51:32,  8.66s/it][2025-04-25 09:19:26,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:19:26,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.41 | bwd_microstep: 5704.06 | bwd_inner_microstep: 5691.22 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.54
[2025-04-25 09:19:26,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.41 | bwd: 5704.08 | bwd_inner: 5691.22 | bwd_allreduce: 12.82 | step: 18.54

  1%|▏         | 562/41250 [1:21:52<97:47:26,  8.65s/it]
                                                        
{'loss': 0.2555, 'grad_norm': 2.429466485977173, 'learning_rate': 1.8158319870759293e-05, 'epoch': 0.14}

  1%|▏         | 562/41250 [1:21:52<97:47:26,  8.65s/it][2025-04-25 09:19:35,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.83 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:19:35,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.72 | bwd_microstep: 5852.61 | bwd_inner_microstep: 5689.76 | bwd_allreduce_microstep: 162.80 | step_microstep: 17.89
[2025-04-25 09:19:35,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.72 | bwd: 5852.62 | bwd_inner: 5689.76 | bwd_allreduce: 162.82 | step: 17.89

  1%|▏         | 563/41250 [1:22:00<98:14:39,  8.69s/it]
                                                        
{'loss': 0.1106, 'grad_norm': 1.2847602367401123, 'learning_rate': 1.8190630048465267e-05, 'epoch': 0.14}

  1%|▏         | 563/41250 [1:22:00<98:14:39,  8.69s/it][2025-04-25 09:19:44,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 1.01
[2025-04-25 09:19:44,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5713.86 | bwd_inner_microstep: 5678.84 | bwd_allreduce_microstep: 34.97 | step_microstep: 19.48
[2025-04-25 09:19:44,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5713.88 | bwd_inner: 5678.84 | bwd_allreduce: 34.99 | step: 19.48

  1%|▏         | 564/41250 [1:22:09<98:06:08,  8.68s/it]
                                                        
{'loss': 0.2859, 'grad_norm': 3.03867244720459, 'learning_rate': 1.8222940226171245e-05, 'epoch': 0.14}

  1%|▏         | 564/41250 [1:22:09<98:06:08,  8.68s/it][2025-04-25 09:19:52,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:19:52,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.30 | bwd_microstep: 5744.13 | bwd_inner_microstep: 5676.57 | bwd_allreduce_microstep: 67.51 | step_microstep: 18.52
[2025-04-25 09:19:52,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.30 | bwd: 5744.14 | bwd_inner: 5676.57 | bwd_allreduce: 67.53 | step: 18.53

  1%|▏         | 565/41250 [1:22:18<98:04:07,  8.68s/it]
                                                        
{'loss': 0.1941, 'grad_norm': 2.1766676902770996, 'learning_rate': 1.8255250403877223e-05, 'epoch': 0.14}

  1%|▏         | 565/41250 [1:22:18<98:04:07,  8.68s/it][2025-04-25 09:20:01,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 09:20:01,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.69 | bwd_microstep: 5758.51 | bwd_inner_microstep: 5673.00 | bwd_allreduce_microstep: 85.47 | step_microstep: 19.40
[2025-04-25 09:20:01,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.69 | bwd: 5758.53 | bwd_inner: 5673.00 | bwd_allreduce: 85.49 | step: 19.40

  1%|▏         | 566/41250 [1:22:26<98:04:54,  8.68s/it]
                                                        
{'loss': 0.0633, 'grad_norm': 1.153053879737854, 'learning_rate': 1.8287560581583198e-05, 'epoch': 0.14}

  1%|▏         | 566/41250 [1:22:26<98:04:54,  8.68s/it][2025-04-25 09:20:10,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 09:20:10,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.79 | bwd_microstep: 5738.30 | bwd_inner_microstep: 5684.64 | bwd_allreduce_microstep: 53.61 | step_microstep: 18.86
[2025-04-25 09:20:10,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.79 | bwd: 5738.31 | bwd_inner: 5684.64 | bwd_allreduce: 53.63 | step: 18.86

  1%|▏         | 567/41250 [1:22:35<98:00:43,  8.67s/it]
                                                        
{'loss': 0.3176, 'grad_norm': 2.368730306625366, 'learning_rate': 1.831987075928918e-05, 'epoch': 0.14}

  1%|▏         | 567/41250 [1:22:35<98:00:43,  8.67s/it][2025-04-25 09:20:18,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:20:18,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.72 | bwd_microstep: 5690.90 | bwd_inner_microstep: 5639.94 | bwd_allreduce_microstep: 50.92 | step_microstep: 18.68
[2025-04-25 09:20:18,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.72 | bwd: 5690.92 | bwd_inner: 5639.94 | bwd_allreduce: 50.94 | step: 18.69

  1%|▏         | 568/41250 [1:22:44<97:43:58,  8.65s/it]
                                                        
{'loss': 0.2131, 'grad_norm': 2.1590161323547363, 'learning_rate': 1.8352180936995153e-05, 'epoch': 0.14}

  1%|▏         | 568/41250 [1:22:44<97:43:58,  8.65s/it][2025-04-25 09:20:27,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:20:27,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.61 | bwd_microstep: 5690.12 | bwd_inner_microstep: 5642.49 | bwd_allreduce_microstep: 47.59 | step_microstep: 18.58
[2025-04-25 09:20:27,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.61 | bwd: 5690.14 | bwd_inner: 5642.49 | bwd_allreduce: 47.61 | step: 18.58

  1%|▏         | 569/41250 [1:22:52<97:32:45,  8.63s/it]
                                                        
{'loss': 0.2336, 'grad_norm': 2.9073336124420166, 'learning_rate': 1.8384491114701135e-05, 'epoch': 0.14}

  1%|▏         | 569/41250 [1:22:52<97:32:45,  8.63s/it][2025-04-25 09:20:36,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.12 | optimizer_step: 1.01
[2025-04-25 09:20:36,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.86 | bwd_microstep: 5808.95 | bwd_inner_microstep: 5764.26 | bwd_allreduce_microstep: 44.59 | step_microstep: 19.73
[2025-04-25 09:20:36,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.86 | bwd: 5808.98 | bwd_inner: 5764.26 | bwd_allreduce: 44.64 | step: 19.72

  1%|▏         | 570/41250 [1:23:01<98:01:45,  8.68s/it]
                                                        
{'loss': 0.269, 'grad_norm': 2.224431037902832, 'learning_rate': 1.841680129240711e-05, 'epoch': 0.14}

  1%|▏         | 570/41250 [1:23:01<98:01:45,  8.68s/it][2025-04-25 09:20:44,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:20:44,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.32 | bwd_microstep: 5767.38 | bwd_inner_microstep: 5636.16 | bwd_allreduce_microstep: 131.18 | step_microstep: 18.76
[2025-04-25 09:20:44,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.32 | bwd: 5767.39 | bwd_inner: 5636.16 | bwd_allreduce: 131.20 | step: 18.76

  1%|▏         | 571/41250 [1:23:10<98:01:10,  8.67s/it]
                                                        
{'loss': 0.3047, 'grad_norm': 2.8957407474517822, 'learning_rate': 1.8449111470113087e-05, 'epoch': 0.14}

  1%|▏         | 571/41250 [1:23:10<98:01:10,  8.67s/it][2025-04-25 09:20:53,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 09:20:53,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.03 | bwd_microstep: 5752.86 | bwd_inner_microstep: 5667.29 | bwd_allreduce_microstep: 85.53 | step_microstep: 18.76
[2025-04-25 09:20:53,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.03 | bwd: 5752.87 | bwd_inner: 5667.29 | bwd_allreduce: 85.54 | step: 18.76

  1%|▏         | 572/41250 [1:23:18<98:00:21,  8.67s/it]
                                                        
{'loss': 0.2877, 'grad_norm': 2.939384698867798, 'learning_rate': 1.8481421647819065e-05, 'epoch': 0.14}

  1%|▏         | 572/41250 [1:23:18<98:00:21,  8.67s/it][2025-04-25 09:21:02,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 09:21:02,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.96 | bwd_microstep: 5864.05 | bwd_inner_microstep: 5626.06 | bwd_allreduce_microstep: 237.94 | step_microstep: 18.75
[2025-04-25 09:21:02,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.96 | bwd: 5864.06 | bwd_inner: 5626.06 | bwd_allreduce: 237.96 | step: 18.76

  1%|▏         | 573/41250 [1:23:27<98:18:04,  8.70s/it]
                                                        
{'loss': 0.4767, 'grad_norm': 3.649439573287964, 'learning_rate': 1.851373182552504e-05, 'epoch': 0.14}

  1%|▏         | 573/41250 [1:23:27<98:18:04,  8.70s/it][2025-04-25 09:21:10,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.19
[2025-04-25 09:21:10,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.50 | bwd_microstep: 5732.24 | bwd_inner_microstep: 5680.07 | bwd_allreduce_microstep: 52.12 | step_microstep: 19.62
[2025-04-25 09:21:10,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.50 | bwd: 5732.25 | bwd_inner: 5680.07 | bwd_allreduce: 52.14 | step: 19.63

  1%|▏         | 574/41250 [1:23:36<98:11:18,  8.69s/it]
                                                        
{'loss': 0.2894, 'grad_norm': 3.1653270721435547, 'learning_rate': 1.854604200323102e-05, 'epoch': 0.14}

  1%|▏         | 574/41250 [1:23:36<98:11:18,  8.69s/it][2025-04-25 09:21:19,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:21:19,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5707.21 | bwd_inner_microstep: 5694.48 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.94
[2025-04-25 09:21:19,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5707.23 | bwd_inner: 5694.48 | bwd_allreduce: 12.70 | step: 18.95

  1%|▏         | 575/41250 [1:23:44<98:01:17,  8.68s/it]
                                                        
{'loss': 0.123, 'grad_norm': 1.9757139682769775, 'learning_rate': 1.8578352180936995e-05, 'epoch': 0.14}

  1%|▏         | 575/41250 [1:23:44<98:01:17,  8.68s/it][2025-04-25 09:21:28,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 09:21:28,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.27 | bwd_microstep: 5747.06 | bwd_inner_microstep: 5636.96 | bwd_allreduce_microstep: 110.05 | step_microstep: 18.94
[2025-04-25 09:21:28,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.27 | bwd: 5747.07 | bwd_inner: 5636.96 | bwd_allreduce: 110.07 | step: 18.94

  1%|▏         | 576/41250 [1:23:53<97:57:00,  8.67s/it]
                                                        
{'loss': 0.2255, 'grad_norm': 2.646036148071289, 'learning_rate': 1.8610662358642973e-05, 'epoch': 0.14}

  1%|▏         | 576/41250 [1:23:53<97:57:00,  8.67s/it][2025-04-25 09:21:36,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 09:21:36,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.19 | bwd_microstep: 5759.63 | bwd_inner_microstep: 5681.54 | bwd_allreduce_microstep: 78.05 | step_microstep: 18.14
[2025-04-25 09:21:36,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.19 | bwd: 5759.64 | bwd_inner: 5681.54 | bwd_allreduce: 78.06 | step: 18.15

  1%|▏         | 577/41250 [1:24:02<98:00:03,  8.67s/it]
                                                        
{'loss': 0.2642, 'grad_norm': 2.8449933528900146, 'learning_rate': 1.864297253634895e-05, 'epoch': 0.14}

  1%|▏         | 577/41250 [1:24:02<98:00:03,  8.67s/it][2025-04-25 09:21:45,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:21:45,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.97 | bwd_microstep: 5687.77 | bwd_inner_microstep: 5675.08 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.94
[2025-04-25 09:21:45,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.97 | bwd: 5687.79 | bwd_inner: 5675.08 | bwd_allreduce: 12.66 | step: 18.94

  1%|▏         | 578/41250 [1:24:10<97:48:31,  8.66s/it]
                                                        
{'loss': 0.4099, 'grad_norm': 2.814663887023926, 'learning_rate': 1.867528271405493e-05, 'epoch': 0.14}

  1%|▏         | 578/41250 [1:24:10<97:48:31,  8.66s/it][2025-04-25 09:21:54,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:21:54,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.87 | bwd_microstep: 5784.68 | bwd_inner_microstep: 5650.68 | bwd_allreduce_microstep: 133.94 | step_microstep: 18.85
[2025-04-25 09:21:54,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.87 | bwd: 5784.69 | bwd_inner: 5650.68 | bwd_allreduce: 133.96 | step: 18.85

  1%|▏         | 579/41250 [1:24:19<97:56:03,  8.67s/it]
                                                        
{'loss': 0.1639, 'grad_norm': 1.7080665826797485, 'learning_rate': 1.8707592891760907e-05, 'epoch': 0.14}

  1%|▏         | 579/41250 [1:24:19<97:56:03,  8.67s/it][2025-04-25 09:22:02,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.24
[2025-04-25 09:22:02,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5760.17 | bwd_inner_microstep: 5689.44 | bwd_allreduce_microstep: 70.69 | step_microstep: 19.66
[2025-04-25 09:22:02,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5760.18 | bwd_inner: 5689.44 | bwd_allreduce: 70.71 | step: 19.66

  1%|▏         | 580/41250 [1:24:28<98:00:13,  8.68s/it]
                                                        
{'loss': 0.1414, 'grad_norm': 1.268687129020691, 'learning_rate': 1.873990306946688e-05, 'epoch': 0.14}

  1%|▏         | 580/41250 [1:24:28<98:00:13,  8.68s/it][2025-04-25 09:22:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.16 | optimizer_step: 1.09
[2025-04-25 09:22:11,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.50 | bwd_microstep: 5760.59 | bwd_inner_microstep: 5691.47 | bwd_allreduce_microstep: 69.06 | step_microstep: 20.87
[2025-04-25 09:22:11,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.50 | bwd: 5760.60 | bwd_inner: 5691.47 | bwd_allreduce: 69.08 | step: 20.87

  1%|▏         | 581/41250 [1:24:36<98:03:24,  8.68s/it]
                                                        
{'loss': 0.3783, 'grad_norm': 2.589238405227661, 'learning_rate': 1.8772213247172862e-05, 'epoch': 0.14}

  1%|▏         | 581/41250 [1:24:36<98:03:24,  8.68s/it][2025-04-25 09:22:20,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-25 09:22:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.09 | bwd_microstep: 5772.60 | bwd_inner_microstep: 5661.73 | bwd_allreduce_microstep: 110.80 | step_microstep: 19.45
[2025-04-25 09:22:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.09 | bwd: 5772.61 | bwd_inner: 5661.73 | bwd_allreduce: 110.83 | step: 19.45

  1%|▏         | 582/41250 [1:24:45<98:05:34,  8.68s/it]
                                                        
{'loss': 0.1556, 'grad_norm': 4.293356418609619, 'learning_rate': 1.8804523424878837e-05, 'epoch': 0.14}

  1%|▏         | 582/41250 [1:24:45<98:05:34,  8.68s/it][2025-04-25 09:22:28,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 09:22:28,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.83 | bwd_microstep: 5705.03 | bwd_inner_microstep: 5692.17 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.91
[2025-04-25 09:22:28,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.83 | bwd: 5705.04 | bwd_inner: 5692.17 | bwd_allreduce: 12.83 | step: 18.91

  1%|▏         | 583/41250 [1:24:54<97:58:45,  8.67s/it]
                                                        
{'loss': 0.11, 'grad_norm': 1.3321988582611084, 'learning_rate': 1.8836833602584815e-05, 'epoch': 0.14}

  1%|▏         | 583/41250 [1:24:54<97:58:45,  8.67s/it][2025-04-25 09:22:37,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 09:22:37,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2939.42 | bwd_microstep: 5884.93 | bwd_inner_microstep: 5871.94 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.40
[2025-04-25 09:22:37,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2939.42 | bwd: 5884.95 | bwd_inner: 5871.94 | bwd_allreduce: 12.96 | step: 19.41

  1%|▏         | 584/41250 [1:25:03<98:47:26,  8.75s/it]
                                                        
{'loss': 0.1485, 'grad_norm': 1.937215805053711, 'learning_rate': 1.8869143780290793e-05, 'epoch': 0.14}

  1%|▏         | 584/41250 [1:25:03<98:47:26,  8.75s/it][2025-04-25 09:22:46,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 09:22:46,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.41 | bwd_microstep: 5760.28 | bwd_inner_microstep: 5652.87 | bwd_allreduce_microstep: 107.36 | step_microstep: 18.91
[2025-04-25 09:22:46,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5760.29 | bwd_inner: 5652.87 | bwd_allreduce: 107.38 | step: 18.91

  1%|▏         | 585/41250 [1:25:11<98:34:46,  8.73s/it]
                                                        
{'loss': 0.1319, 'grad_norm': 1.5506888628005981, 'learning_rate': 1.890145395799677e-05, 'epoch': 0.14}

  1%|▏         | 585/41250 [1:25:11<98:34:46,  8.73s/it][2025-04-25 09:22:55,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:22:55,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.53 | bwd_microstep: 5727.85 | bwd_inner_microstep: 5692.60 | bwd_allreduce_microstep: 35.20 | step_microstep: 18.64
[2025-04-25 09:22:55,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.53 | bwd: 5727.86 | bwd_inner: 5692.60 | bwd_allreduce: 35.21 | step: 18.65

  1%|▏         | 586/41250 [1:25:20<98:22:38,  8.71s/it]
                                                        
{'loss': 0.3969, 'grad_norm': 3.020979404449463, 'learning_rate': 1.893376413570275e-05, 'epoch': 0.14}

  1%|▏         | 586/41250 [1:25:20<98:22:38,  8.71s/it][2025-04-25 09:23:03,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:23:03,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5782.50 | bwd_inner_microstep: 5658.08 | bwd_allreduce_microstep: 124.38 | step_microstep: 18.17
[2025-04-25 09:23:03,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5782.51 | bwd_inner: 5658.08 | bwd_allreduce: 124.40 | step: 18.17

  1%|▏         | 587/41250 [1:25:29<98:20:16,  8.71s/it]
                                                        
{'loss': 0.2215, 'grad_norm': 2.030726671218872, 'learning_rate': 1.8966074313408723e-05, 'epoch': 0.14}

  1%|▏         | 587/41250 [1:25:29<98:20:16,  8.71s/it][2025-04-25 09:23:12,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:23:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.10 | bwd_microstep: 5718.51 | bwd_inner_microstep: 5668.27 | bwd_allreduce_microstep: 50.19 | step_microstep: 18.57
[2025-04-25 09:23:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.10 | bwd: 5718.52 | bwd_inner: 5668.27 | bwd_allreduce: 50.21 | step: 18.57

  1%|▏         | 588/41250 [1:25:37<98:06:58,  8.69s/it]
                                                        
{'loss': 0.3327, 'grad_norm': 2.430981159210205, 'learning_rate': 1.8998384491114704e-05, 'epoch': 0.14}

  1%|▏         | 588/41250 [1:25:37<98:06:58,  8.69s/it][2025-04-25 09:23:21,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 09:23:21,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.95 | bwd_microstep: 5709.42 | bwd_inner_microstep: 5695.61 | bwd_allreduce_microstep: 13.76 | step_microstep: 18.72
[2025-04-25 09:23:21,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.95 | bwd: 5709.43 | bwd_inner: 5695.61 | bwd_allreduce: 13.78 | step: 18.72

  1%|▏         | 589/41250 [1:25:46<97:58:08,  8.67s/it]
                                                        
{'loss': 0.2832, 'grad_norm': 2.590346097946167, 'learning_rate': 1.903069466882068e-05, 'epoch': 0.14}

  1%|▏         | 589/41250 [1:25:46<97:58:08,  8.67s/it][2025-04-25 09:23:29,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:23:29,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.14 | bwd_microstep: 5734.29 | bwd_inner_microstep: 5721.44 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.45
[2025-04-25 09:23:29,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.14 | bwd: 5734.30 | bwd_inner: 5721.44 | bwd_allreduce: 12.82 | step: 18.46

  1%|▏         | 590/41250 [1:25:55<98:00:01,  8.68s/it]
                                                        
{'loss': 0.2497, 'grad_norm': 2.5519840717315674, 'learning_rate': 1.9063004846526657e-05, 'epoch': 0.14}

  1%|▏         | 590/41250 [1:25:55<98:00:01,  8.68s/it][2025-04-25 09:23:38,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:23:38,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.69 | bwd_microstep: 5784.79 | bwd_inner_microstep: 5655.67 | bwd_allreduce_microstep: 129.07 | step_microstep: 18.56
[2025-04-25 09:23:38,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.69 | bwd: 5784.80 | bwd_inner: 5655.67 | bwd_allreduce: 129.09 | step: 18.57

  1%|▏         | 591/41250 [1:26:03<98:03:55,  8.68s/it]
                                                        
{'loss': 0.2462, 'grad_norm': 1.6961398124694824, 'learning_rate': 1.9095315024232634e-05, 'epoch': 0.14}

  1%|▏         | 591/41250 [1:26:03<98:03:55,  8.68s/it][2025-04-25 09:23:47,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-25 09:23:47,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.15 | bwd_microstep: 5729.46 | bwd_inner_microstep: 5670.64 | bwd_allreduce_microstep: 58.77 | step_microstep: 18.60
[2025-04-25 09:23:47,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.15 | bwd: 5729.47 | bwd_inner: 5670.65 | bwd_allreduce: 58.79 | step: 18.60

  1%|▏         | 592/41250 [1:26:12<97:56:41,  8.67s/it]
                                                        
{'loss': 0.2079, 'grad_norm': 1.4445550441741943, 'learning_rate': 1.9127625201938612e-05, 'epoch': 0.14}

  1%|▏         | 592/41250 [1:26:12<97:56:41,  8.67s/it][2025-04-25 09:23:55,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:23:55,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.25 | bwd_microstep: 5760.55 | bwd_inner_microstep: 5697.43 | bwd_allreduce_microstep: 63.08 | step_microstep: 18.57
[2025-04-25 09:23:55,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.25 | bwd: 5760.57 | bwd_inner: 5697.43 | bwd_allreduce: 63.10 | step: 18.57

  1%|▏         | 593/41250 [1:26:21<98:03:10,  8.68s/it]
                                                        
{'loss': 0.2569, 'grad_norm': 2.865089178085327, 'learning_rate': 1.915993537964459e-05, 'epoch': 0.14}

  1%|▏         | 593/41250 [1:26:21<98:03:10,  8.68s/it][2025-04-25 09:24:04,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:24:04,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.82 | bwd_microstep: 5767.18 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 70.18 | step_microstep: 18.35
[2025-04-25 09:24:04,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5767.20 | bwd_inner: 5696.95 | bwd_allreduce: 70.21 | step: 18.35

  1%|▏         | 594/41250 [1:26:30<98:07:49,  8.69s/it]
                                                        
{'loss': 0.2907, 'grad_norm': 2.034558057785034, 'learning_rate': 1.9192245557350565e-05, 'epoch': 0.14}

  1%|▏         | 594/41250 [1:26:30<98:07:49,  8.69s/it][2025-04-25 09:24:13,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 09:24:13,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.15 | bwd_microstep: 5786.07 | bwd_inner_microstep: 5709.39 | bwd_allreduce_microstep: 76.63 | step_microstep: 19.26
[2025-04-25 09:24:13,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.15 | bwd: 5786.08 | bwd_inner: 5709.39 | bwd_allreduce: 76.65 | step: 19.26

  1%|▏         | 595/41250 [1:26:38<98:13:25,  8.70s/it]
                                                        
{'loss': 0.1922, 'grad_norm': 1.4510350227355957, 'learning_rate': 1.9224555735056546e-05, 'epoch': 0.14}

  1%|▏         | 595/41250 [1:26:38<98:13:25,  8.70s/it][2025-04-25 09:24:22,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:24:22,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5706.62 | bwd_inner_microstep: 5662.58 | bwd_allreduce_microstep: 44.00 | step_microstep: 18.69
[2025-04-25 09:24:22,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5706.63 | bwd_inner: 5662.58 | bwd_allreduce: 44.01 | step: 18.69

  1%|▏         | 596/41250 [1:26:47<97:58:11,  8.68s/it]
                                                        
{'loss': 0.2481, 'grad_norm': 1.2925522327423096, 'learning_rate': 1.925686591276252e-05, 'epoch': 0.14}

  1%|▏         | 596/41250 [1:26:47<97:58:11,  8.68s/it][2025-04-25 09:24:30,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:24:30,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.37 | bwd_microstep: 5742.58 | bwd_inner_microstep: 5645.73 | bwd_allreduce_microstep: 96.80 | step_microstep: 18.48
[2025-04-25 09:24:30,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.37 | bwd: 5742.59 | bwd_inner: 5645.73 | bwd_allreduce: 96.82 | step: 18.49

  1%|▏         | 597/41250 [1:26:56<97:53:41,  8.67s/it]
                                                        
{'loss': 0.1651, 'grad_norm': 1.405949592590332, 'learning_rate': 1.92891760904685e-05, 'epoch': 0.14}

  1%|▏         | 597/41250 [1:26:56<97:53:41,  8.67s/it][2025-04-25 09:24:39,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 09:24:39,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.29 | bwd_microstep: 5721.60 | bwd_inner_microstep: 5658.15 | bwd_allreduce_microstep: 63.41 | step_microstep: 18.69
[2025-04-25 09:24:39,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.29 | bwd: 5721.61 | bwd_inner: 5658.15 | bwd_allreduce: 63.42 | step: 18.69

  1%|▏         | 598/41250 [1:27:04<97:48:11,  8.66s/it]
                                                        
{'loss': 0.2391, 'grad_norm': 2.0907163619995117, 'learning_rate': 1.9321486268174476e-05, 'epoch': 0.14}

  1%|▏         | 598/41250 [1:27:04<97:48:11,  8.66s/it][2025-04-25 09:24:47,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 09:24:47,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.17 | bwd_microstep: 5714.19 | bwd_inner_microstep: 5690.94 | bwd_allreduce_microstep: 23.20 | step_microstep: 18.75
[2025-04-25 09:24:47,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.17 | bwd: 5714.21 | bwd_inner: 5690.94 | bwd_allreduce: 23.22 | step: 18.76

  1%|▏         | 599/41250 [1:27:13<97:45:06,  8.66s/it]
                                                        
{'loss': 0.1739, 'grad_norm': 1.1663668155670166, 'learning_rate': 1.9353796445880454e-05, 'epoch': 0.15}

  1%|▏         | 599/41250 [1:27:13<97:45:06,  8.66s/it][2025-04-25 09:24:56,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 09:24:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5780.31 | bwd_inner_microstep: 5644.44 | bwd_allreduce_microstep: 135.82 | step_microstep: 19.17
[2025-04-25 09:24:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5780.32 | bwd_inner: 5644.44 | bwd_allreduce: 135.84 | step: 19.17

  1%|▏         | 600/41250 [1:27:21<97:52:57,  8.67s/it]
                                                        
{'loss': 0.0877, 'grad_norm': 1.063796877861023, 'learning_rate': 1.9386106623586432e-05, 'epoch': 0.15}

  1%|▏         | 600/41250 [1:27:21<97:52:57,  8.67s/it][2025-04-25 09:25:05,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 09:25:05,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.62 | bwd_microstep: 5725.65 | bwd_inner_microstep: 5712.93 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.52
[2025-04-25 09:25:05,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.62 | bwd: 5725.66 | bwd_inner: 5712.93 | bwd_allreduce: 12.69 | step: 18.53

  1%|▏         | 601/41250 [1:27:30<97:51:17,  8.67s/it]
                                                        
{'loss': 0.2764, 'grad_norm': 1.992734432220459, 'learning_rate': 1.9418416801292407e-05, 'epoch': 0.15}

  1%|▏         | 601/41250 [1:27:30<97:51:17,  8.67s/it][2025-04-25 09:25:13,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 09:25:13,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.29 | bwd_microstep: 5721.62 | bwd_inner_microstep: 5708.80 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.41
[2025-04-25 09:25:13,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.29 | bwd: 5721.63 | bwd_inner: 5708.80 | bwd_allreduce: 12.79 | step: 19.42

  1%|▏         | 602/41250 [1:27:39<97:52:05,  8.67s/it]
                                                        
{'loss': 0.2531, 'grad_norm': 1.6214159727096558, 'learning_rate': 1.9450726978998388e-05, 'epoch': 0.15}

  1%|▏         | 602/41250 [1:27:39<97:52:05,  8.67s/it][2025-04-25 09:25:22,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:25:22,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.88 | bwd_microstep: 5860.79 | bwd_inner_microstep: 5692.25 | bwd_allreduce_microstep: 168.49 | step_microstep: 18.62
[2025-04-25 09:25:22,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.88 | bwd: 5860.80 | bwd_inner: 5692.25 | bwd_allreduce: 168.51 | step: 18.62

  1%|▏         | 603/41250 [1:27:48<98:18:31,  8.71s/it]
                                                        
{'loss': 0.1661, 'grad_norm': 1.1834681034088135, 'learning_rate': 1.9483037156704362e-05, 'epoch': 0.15}

  1%|▏         | 603/41250 [1:27:48<98:18:31,  8.71s/it][2025-04-25 09:25:31,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:25:31,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.46 | bwd_microstep: 5725.46 | bwd_inner_microstep: 5694.56 | bwd_allreduce_microstep: 30.85 | step_microstep: 18.67
[2025-04-25 09:25:31,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.46 | bwd: 5725.47 | bwd_inner: 5694.56 | bwd_allreduce: 30.87 | step: 18.68

  1%|▏         | 604/41250 [1:27:56<98:13:10,  8.70s/it]
                                                        
{'loss': 0.1075, 'grad_norm': 1.0740162134170532, 'learning_rate': 1.951534733441034e-05, 'epoch': 0.15}

  1%|▏         | 604/41250 [1:27:56<98:13:10,  8.70s/it][2025-04-25 09:25:40,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 09:25:40,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.73 | bwd_microstep: 5762.56 | bwd_inner_microstep: 5703.59 | bwd_allreduce_microstep: 58.91 | step_microstep: 18.96
[2025-04-25 09:25:40,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.73 | bwd: 5762.57 | bwd_inner: 5703.59 | bwd_allreduce: 58.94 | step: 18.96

  1%|▏         | 605/41250 [1:28:05<98:14:36,  8.70s/it]
                                                        
{'loss': 0.2308, 'grad_norm': 2.3398547172546387, 'learning_rate': 1.9547657512116318e-05, 'epoch': 0.15}

  1%|▏         | 605/41250 [1:28:05<98:14:36,  8.70s/it][2025-04-25 09:25:48,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 1.03
[2025-04-25 09:25:48,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.27 | bwd_microstep: 5729.22 | bwd_inner_microstep: 5686.39 | bwd_allreduce_microstep: 42.79 | step_microstep: 19.54
[2025-04-25 09:25:48,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.27 | bwd: 5729.24 | bwd_inner: 5686.38 | bwd_allreduce: 42.81 | step: 19.55

  1%|▏         | 606/41250 [1:28:14<98:08:02,  8.69s/it]
                                                        
{'loss': 0.1995, 'grad_norm': 2.349421501159668, 'learning_rate': 1.9579967689822296e-05, 'epoch': 0.15}

  1%|▏         | 606/41250 [1:28:14<98:08:02,  8.69s/it][2025-04-25 09:25:57,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:25:57,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.10 | bwd_microstep: 5753.85 | bwd_inner_microstep: 5708.04 | bwd_allreduce_microstep: 45.77 | step_microstep: 18.55
[2025-04-25 09:25:57,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.10 | bwd: 5753.87 | bwd_inner: 5708.04 | bwd_allreduce: 45.79 | step: 18.55

  1%|▏         | 607/41250 [1:28:22<98:08:28,  8.69s/it]
                                                        
{'loss': 0.2536, 'grad_norm': 1.7774935960769653, 'learning_rate': 1.9612277867528274e-05, 'epoch': 0.15}

  1%|▏         | 607/41250 [1:28:22<98:08:28,  8.69s/it][2025-04-25 09:26:06,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:26:06,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5709.64 | bwd_inner_microstep: 5696.83 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.68
[2025-04-25 09:26:06,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5709.65 | bwd_inner: 5696.83 | bwd_allreduce: 12.78 | step: 18.68

  1%|▏         | 608/41250 [1:28:31<97:58:08,  8.68s/it]
                                                        
{'loss': 0.245, 'grad_norm': 3.260779619216919, 'learning_rate': 1.9644588045234248e-05, 'epoch': 0.15}

  1%|▏         | 608/41250 [1:28:31<97:58:08,  8.68s/it][2025-04-25 09:26:14,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:26:14,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.75 | bwd_microstep: 5720.37 | bwd_inner_microstep: 5645.85 | bwd_allreduce_microstep: 74.48 | step_microstep: 18.76
[2025-04-25 09:26:14,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.75 | bwd: 5720.38 | bwd_inner: 5645.85 | bwd_allreduce: 74.50 | step: 18.76

  1%|▏         | 609/41250 [1:28:40<97:48:01,  8.66s/it]
                                                        
{'loss': 0.0961, 'grad_norm': 1.3223541975021362, 'learning_rate': 1.967689822294023e-05, 'epoch': 0.15}

  1%|▏         | 609/41250 [1:28:40<97:48:01,  8.66s/it][2025-04-25 09:26:23,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:26:23,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.91 | bwd_microstep: 5703.88 | bwd_inner_microstep: 5691.50 | bwd_allreduce_microstep: 12.34 | step_microstep: 18.51
[2025-04-25 09:26:23,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.91 | bwd: 5703.89 | bwd_inner: 5691.50 | bwd_allreduce: 12.36 | step: 18.52

  1%|▏         | 610/41250 [1:28:48<97:42:30,  8.66s/it]
                                                        
{'loss': 0.1362, 'grad_norm': 1.907104253768921, 'learning_rate': 1.9709208400646204e-05, 'epoch': 0.15}

  1%|▏         | 610/41250 [1:28:48<97:42:30,  8.66s/it][2025-04-25 09:26:32,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:26:32,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.99 | bwd_microstep: 5772.12 | bwd_inner_microstep: 5642.42 | bwd_allreduce_microstep: 129.64 | step_microstep: 18.09
[2025-04-25 09:26:32,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.99 | bwd: 5772.13 | bwd_inner: 5642.42 | bwd_allreduce: 129.66 | step: 18.09

  1%|▏         | 611/41250 [1:28:57<97:46:44,  8.66s/it]
                                                        
{'loss': 0.2778, 'grad_norm': 2.0201733112335205, 'learning_rate': 1.9741518578352182e-05, 'epoch': 0.15}

  1%|▏         | 611/41250 [1:28:57<97:46:44,  8.66s/it][2025-04-25 09:26:40,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 09:26:40,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.91 | bwd_microstep: 5714.37 | bwd_inner_microstep: 5697.26 | bwd_allreduce_microstep: 17.07 | step_microstep: 19.04
[2025-04-25 09:26:40,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.91 | bwd: 5714.39 | bwd_inner: 5697.26 | bwd_allreduce: 17.09 | step: 19.04

  1%|▏         | 612/41250 [1:29:06<97:46:27,  8.66s/it]
                                                        
{'loss': 0.2556, 'grad_norm': 3.570216178894043, 'learning_rate': 1.977382875605816e-05, 'epoch': 0.15}

  1%|▏         | 612/41250 [1:29:06<97:46:27,  8.66s/it][2025-04-25 09:26:49,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:26:49,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.94 | bwd_microstep: 5748.73 | bwd_inner_microstep: 5640.52 | bwd_allreduce_microstep: 108.17 | step_microstep: 18.10
[2025-04-25 09:26:49,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.94 | bwd: 5748.74 | bwd_inner: 5640.52 | bwd_allreduce: 108.19 | step: 18.10

  1%|▏         | 613/41250 [1:29:14<97:45:39,  8.66s/it]
                                                        
{'loss': 0.3227, 'grad_norm': 3.2593343257904053, 'learning_rate': 1.9806138933764138e-05, 'epoch': 0.15}

  1%|▏         | 613/41250 [1:29:14<97:45:39,  8.66s/it][2025-04-25 09:26:58,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 09:26:58,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.00 | bwd_microstep: 5698.05 | bwd_inner_microstep: 5685.24 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.77
[2025-04-25 09:26:58,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.00 | bwd: 5698.06 | bwd_inner: 5685.24 | bwd_allreduce: 12.78 | step: 18.78

  1%|▏         | 614/41250 [1:29:23<97:39:22,  8.65s/it]
                                                        
{'loss': 0.2087, 'grad_norm': 1.9740065336227417, 'learning_rate': 1.9838449111470116e-05, 'epoch': 0.15}

  1%|▏         | 614/41250 [1:29:23<97:39:22,  8.65s/it][2025-04-25 09:27:06,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:27:06,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5740.11 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 38.13 | step_microstep: 18.67
[2025-04-25 09:27:06,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5740.12 | bwd_inner: 5701.94 | bwd_allreduce: 38.15 | step: 18.67

  1%|▏         | 615/41250 [1:29:32<97:43:12,  8.66s/it]
                                                        
{'loss': 0.1331, 'grad_norm': 1.3191324472427368, 'learning_rate': 1.987075928917609e-05, 'epoch': 0.15}

  1%|▏         | 615/41250 [1:29:32<97:43:12,  8.66s/it][2025-04-25 09:27:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:27:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.55 | bwd_microstep: 5723.02 | bwd_inner_microstep: 5693.97 | bwd_allreduce_microstep: 29.00 | step_microstep: 18.52
[2025-04-25 09:27:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.55 | bwd: 5723.03 | bwd_inner: 5693.97 | bwd_allreduce: 29.02 | step: 18.52

  1%|▏         | 616/41250 [1:29:40<97:43:17,  8.66s/it]
                                                        
{'loss': 0.3023, 'grad_norm': 2.529425859451294, 'learning_rate': 1.990306946688207e-05, 'epoch': 0.15}

  1%|▏         | 616/41250 [1:29:40<97:43:17,  8.66s/it][2025-04-25 09:27:24,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.26 | optimizer_step: 0.92
[2025-04-25 09:27:24,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.49 | bwd_microstep: 5786.64 | bwd_inner_microstep: 5772.83 | bwd_allreduce_microstep: 13.74 | step_microstep: 19.34
[2025-04-25 09:27:24,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.49 | bwd: 5786.65 | bwd_inner: 5772.83 | bwd_allreduce: 13.77 | step: 19.34

  1%|▏         | 617/41250 [1:29:49<98:03:37,  8.69s/it]
                                                        
{'loss': 0.1997, 'grad_norm': 2.2781484127044678, 'learning_rate': 1.9935379644588046e-05, 'epoch': 0.15}

  1%|▏         | 617/41250 [1:29:49<98:03:37,  8.69s/it][2025-04-25 09:27:32,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 09:27:32,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.15 | bwd_microstep: 5747.09 | bwd_inner_microstep: 5654.01 | bwd_allreduce_microstep: 93.04 | step_microstep: 19.48
[2025-04-25 09:27:32,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.15 | bwd: 5747.11 | bwd_inner: 5654.01 | bwd_allreduce: 93.06 | step: 19.48

  1%|▏         | 618/41250 [1:29:58<97:57:57,  8.68s/it]
                                                        
{'loss': 0.1376, 'grad_norm': 1.5413538217544556, 'learning_rate': 1.9967689822294024e-05, 'epoch': 0.15}

  1%|▏         | 618/41250 [1:29:58<97:57:57,  8.68s/it][2025-04-25 09:27:41,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:27:41,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.29 | bwd_microstep: 5766.92 | bwd_inner_microstep: 5640.71 | bwd_allreduce_microstep: 126.16 | step_microstep: 18.83
[2025-04-25 09:27:41,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.29 | bwd: 5766.93 | bwd_inner: 5640.71 | bwd_allreduce: 126.18 | step: 18.83

  2%|▏         | 619/41250 [1:30:06<97:56:24,  8.68s/it]
                                                        
{'loss': 0.4091, 'grad_norm': 4.799709796905518, 'learning_rate': 2e-05, 'epoch': 0.15}

  2%|▏         | 619/41250 [1:30:06<97:56:24,  8.68s/it][2025-04-25 09:27:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-25 09:27:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5715.73 | bwd_inner_microstep: 5702.21 | bwd_allreduce_microstep: 13.45 | step_microstep: 19.07
[2025-04-25 09:27:50,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5715.74 | bwd_inner: 5702.21 | bwd_allreduce: 13.48 | step: 19.07

  2%|▏         | 620/41250 [1:30:15<97:51:03,  8.67s/it]
                                                        
{'loss': 0.0668, 'grad_norm': 1.416301965713501, 'learning_rate': 2.003231017770598e-05, 'epoch': 0.15}

  2%|▏         | 620/41250 [1:30:15<97:51:03,  8.67s/it][2025-04-25 09:27:58,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:27:58,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.66 | bwd_microstep: 5751.83 | bwd_inner_microstep: 5640.76 | bwd_allreduce_microstep: 111.02 | step_microstep: 18.46
[2025-04-25 09:27:58,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.66 | bwd: 5751.84 | bwd_inner: 5640.77 | bwd_allreduce: 111.03 | step: 18.47

  2%|▏         | 621/41250 [1:30:24<97:49:39,  8.67s/it]
                                                        
{'loss': 0.3472, 'grad_norm': 4.385878086090088, 'learning_rate': 2.0064620355411954e-05, 'epoch': 0.15}

  2%|▏         | 621/41250 [1:30:24<97:49:39,  8.67s/it][2025-04-25 09:28:07,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 09:28:07,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5700.47 | bwd_inner_microstep: 5687.55 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.36
[2025-04-25 09:28:07,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5700.49 | bwd_inner: 5687.55 | bwd_allreduce: 12.89 | step: 19.36

  2%|▏         | 622/41250 [1:30:32<97:44:31,  8.66s/it]
                                                        
{'loss': 0.216, 'grad_norm': 4.696005344390869, 'learning_rate': 2.0096930533117932e-05, 'epoch': 0.15}

  2%|▏         | 622/41250 [1:30:32<97:44:31,  8.66s/it][2025-04-25 09:28:16,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-25 09:28:16,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.04 | bwd_microstep: 5745.27 | bwd_inner_microstep: 5654.17 | bwd_allreduce_microstep: 91.05 | step_microstep: 19.07
[2025-04-25 09:28:16,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.04 | bwd: 5745.29 | bwd_inner: 5654.17 | bwd_allreduce: 91.07 | step: 19.07

  2%|▏         | 623/41250 [1:30:41<97:46:59,  8.66s/it]
                                                        
{'loss': 0.1578, 'grad_norm': 2.528069496154785, 'learning_rate': 2.0129240710823913e-05, 'epoch': 0.15}

  2%|▏         | 623/41250 [1:30:41<97:46:59,  8.66s/it][2025-04-25 09:28:24,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.28 | optimizer_step: 0.90
[2025-04-25 09:28:24,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.35 | bwd_microstep: 5685.22 | bwd_inner_microstep: 5646.56 | bwd_allreduce_microstep: 38.60 | step_microstep: 19.69
[2025-04-25 09:28:24,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.35 | bwd: 5685.24 | bwd_inner: 5646.56 | bwd_allreduce: 38.63 | step: 19.69

  2%|▏         | 624/41250 [1:30:50<97:33:59,  8.65s/it]
                                                        
{'loss': 0.2512, 'grad_norm': 2.8846547603607178, 'learning_rate': 2.016155088852989e-05, 'epoch': 0.15}

  2%|▏         | 624/41250 [1:30:50<97:33:59,  8.65s/it][2025-04-25 09:28:33,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.09 | optimizer_step: 1.08
[2025-04-25 09:28:33,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.77 | bwd_microstep: 5712.09 | bwd_inner_microstep: 5698.81 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.20
[2025-04-25 09:28:33,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.77 | bwd: 5712.11 | bwd_inner: 5698.81 | bwd_allreduce: 13.25 | step: 19.20

  2%|▏         | 625/41250 [1:30:58<97:35:13,  8.65s/it]
                                                        
{'loss': 0.4483, 'grad_norm': 3.4273626804351807, 'learning_rate': 2.0193861066235866e-05, 'epoch': 0.15}

  2%|▏         | 625/41250 [1:30:58<97:35:13,  8.65s/it][2025-04-25 09:28:42,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.19 | optimizer_step: 0.97
[2025-04-25 09:28:42,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.55 | bwd_microstep: 5712.63 | bwd_inner_microstep: 5699.64 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.91
[2025-04-25 09:28:42,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.55 | bwd: 5712.65 | bwd_inner: 5699.64 | bwd_allreduce: 12.96 | step: 19.92

  2%|▏         | 626/41250 [1:31:07<97:36:00,  8.65s/it]
                                                        
{'loss': 0.2017, 'grad_norm': 2.330902099609375, 'learning_rate': 2.0226171243941843e-05, 'epoch': 0.15}

  2%|▏         | 626/41250 [1:31:07<97:36:00,  8.65s/it][2025-04-25 09:28:50,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:28:50,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.30 | bwd_microstep: 5676.21 | bwd_inner_microstep: 5643.38 | bwd_allreduce_microstep: 32.78 | step_microstep: 18.61
[2025-04-25 09:28:50,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.30 | bwd: 5676.22 | bwd_inner: 5643.38 | bwd_allreduce: 32.80 | step: 18.62

  2%|▏         | 627/41250 [1:31:15<97:22:49,  8.63s/it]
                                                        
{'loss': 0.1534, 'grad_norm': 2.9448907375335693, 'learning_rate': 2.025848142164782e-05, 'epoch': 0.15}

  2%|▏         | 627/41250 [1:31:15<97:22:49,  8.63s/it][2025-04-25 09:28:59,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:28:59,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.56 | bwd_microstep: 5747.61 | bwd_inner_microstep: 5649.03 | bwd_allreduce_microstep: 98.54 | step_microstep: 18.56
[2025-04-25 09:28:59,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.56 | bwd: 5747.62 | bwd_inner: 5649.03 | bwd_allreduce: 98.55 | step: 18.56

  2%|▏         | 628/41250 [1:31:24<97:27:18,  8.64s/it]
                                                        
{'loss': 0.3224, 'grad_norm': 4.71328592300415, 'learning_rate': 2.0290791599353796e-05, 'epoch': 0.15}

  2%|▏         | 628/41250 [1:31:24<97:27:18,  8.64s/it][2025-04-25 09:29:08,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:29:08,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.77 | bwd_microstep: 5830.08 | bwd_inner_microstep: 5684.75 | bwd_allreduce_microstep: 145.29 | step_microstep: 18.51
[2025-04-25 09:29:08,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.77 | bwd: 5830.09 | bwd_inner: 5684.75 | bwd_allreduce: 145.30 | step: 18.51

  2%|▏         | 629/41250 [1:31:33<97:50:45,  8.67s/it]
                                                        
{'loss': 0.3117, 'grad_norm': 3.2770774364471436, 'learning_rate': 2.0323101777059774e-05, 'epoch': 0.15}

  2%|▏         | 629/41250 [1:31:33<97:50:45,  8.67s/it][2025-04-25 09:29:16,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 09:29:16,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.33 | bwd_microstep: 5746.72 | bwd_inner_microstep: 5676.09 | bwd_allreduce_microstep: 70.59 | step_microstep: 18.96
[2025-04-25 09:29:16,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.33 | bwd: 5746.74 | bwd_inner: 5676.09 | bwd_allreduce: 70.61 | step: 18.97

  2%|▏         | 630/41250 [1:31:42<97:50:19,  8.67s/it]
                                                        
{'loss': 0.2789, 'grad_norm': 1.9525328874588013, 'learning_rate': 2.0355411954765755e-05, 'epoch': 0.15}

  2%|▏         | 630/41250 [1:31:42<97:50:19,  8.67s/it][2025-04-25 09:29:25,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 09:29:25,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.23 | bwd_microstep: 5738.69 | bwd_inner_microstep: 5634.99 | bwd_allreduce_microstep: 103.66 | step_microstep: 19.15
[2025-04-25 09:29:25,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.23 | bwd: 5738.71 | bwd_inner: 5634.99 | bwd_allreduce: 103.68 | step: 19.15

  2%|▏         | 631/41250 [1:31:50<97:43:55,  8.66s/it]
                                                        
{'loss': 0.386, 'grad_norm': 3.2639682292938232, 'learning_rate': 2.0387722132471733e-05, 'epoch': 0.15}

  2%|▏         | 631/41250 [1:31:50<97:43:55,  8.66s/it][2025-04-25 09:29:33,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 09:29:33,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.17 | bwd_microstep: 5748.99 | bwd_inner_microstep: 5632.65 | bwd_allreduce_microstep: 116.29 | step_microstep: 18.49
[2025-04-25 09:29:33,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.17 | bwd: 5749.00 | bwd_inner: 5632.65 | bwd_allreduce: 116.31 | step: 18.49

  2%|▏         | 632/41250 [1:31:59<97:40:50,  8.66s/it]
                                                        
{'loss': 0.1953, 'grad_norm': 4.68179988861084, 'learning_rate': 2.0420032310177707e-05, 'epoch': 0.15}

  2%|▏         | 632/41250 [1:31:59<97:40:50,  8.66s/it][2025-04-25 09:29:42,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 09:29:42,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.88 | bwd_microstep: 5666.71 | bwd_inner_microstep: 5631.61 | bwd_allreduce_microstep: 35.06 | step_microstep: 18.75
[2025-04-25 09:29:42,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.88 | bwd: 5666.72 | bwd_inner: 5631.60 | bwd_allreduce: 35.08 | step: 18.75

  2%|▏         | 633/41250 [1:32:07<97:25:55,  8.64s/it]
                                                        
{'loss': 0.1466, 'grad_norm': 1.3573616743087769, 'learning_rate': 2.0452342487883685e-05, 'epoch': 0.15}

  2%|▏         | 633/41250 [1:32:07<97:25:55,  8.64s/it][2025-04-25 09:29:51,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.05 | optimizer_step: 1.18
[2025-04-25 09:29:51,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.07 | bwd_microstep: 5675.65 | bwd_inner_microstep: 5641.00 | bwd_allreduce_microstep: 34.59 | step_microstep: 19.18
[2025-04-25 09:29:51,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.07 | bwd: 5675.66 | bwd_inner: 5641.00 | bwd_allreduce: 34.61 | step: 19.18

  2%|▏         | 634/41250 [1:32:16<97:16:58,  8.62s/it]
                                                        
{'loss': 0.3131, 'grad_norm': 3.423495292663574, 'learning_rate': 2.0484652665589663e-05, 'epoch': 0.15}

  2%|▏         | 634/41250 [1:32:16<97:16:58,  8.62s/it][2025-04-25 09:29:59,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 09:29:59,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.14 | bwd_microstep: 5674.29 | bwd_inner_microstep: 5639.01 | bwd_allreduce_microstep: 35.24 | step_microstep: 18.78
[2025-04-25 09:29:59,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.14 | bwd: 5674.30 | bwd_inner: 5639.01 | bwd_allreduce: 35.25 | step: 18.79

  2%|▏         | 635/41250 [1:32:25<97:09:55,  8.61s/it]
                                                        
{'loss': 0.1552, 'grad_norm': 2.185556411743164, 'learning_rate': 2.0516962843295638e-05, 'epoch': 0.15}

  2%|▏         | 635/41250 [1:32:25<97:09:55,  8.61s/it][2025-04-25 09:30:08,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 09:30:08,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.08 | bwd_microstep: 5737.06 | bwd_inner_microstep: 5664.77 | bwd_allreduce_microstep: 72.24 | step_microstep: 18.80
[2025-04-25 09:30:08,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.08 | bwd: 5737.08 | bwd_inner: 5664.77 | bwd_allreduce: 72.26 | step: 18.80

  2%|▏         | 636/41250 [1:32:33<97:20:58,  8.63s/it]
                                                        
{'loss': 0.156, 'grad_norm': 2.7122318744659424, 'learning_rate': 2.0549273021001615e-05, 'epoch': 0.15}

  2%|▏         | 636/41250 [1:32:33<97:20:58,  8.63s/it][2025-04-25 09:30:17,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 09:30:17,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.76 | bwd_microstep: 5715.79 | bwd_inner_microstep: 5677.34 | bwd_allreduce_microstep: 38.41 | step_microstep: 18.21
[2025-04-25 09:30:17,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.76 | bwd: 5715.80 | bwd_inner: 5677.34 | bwd_allreduce: 38.42 | step: 18.21

  2%|▏         | 637/41250 [1:32:42<97:24:11,  8.63s/it]
                                                        
{'loss': 0.1843, 'grad_norm': 2.1694934368133545, 'learning_rate': 2.0581583198707597e-05, 'epoch': 0.15}

  2%|▏         | 637/41250 [1:32:42<97:24:11,  8.63s/it][2025-04-25 09:30:25,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:30:25,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.96 | bwd_microstep: 5659.75 | bwd_inner_microstep: 5637.94 | bwd_allreduce_microstep: 21.76 | step_microstep: 18.51
[2025-04-25 09:30:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.96 | bwd: 5659.76 | bwd_inner: 5637.94 | bwd_allreduce: 21.78 | step: 18.51

  2%|▏         | 638/41250 [1:32:50<97:10:55,  8.61s/it]
                                                        
{'loss': 0.0913, 'grad_norm': 1.4332666397094727, 'learning_rate': 2.0613893376413575e-05, 'epoch': 0.15}

  2%|▏         | 638/41250 [1:32:50<97:10:55,  8.61s/it][2025-04-25 09:30:34,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 09:30:34,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.62 | bwd_microstep: 5736.34 | bwd_inner_microstep: 5639.31 | bwd_allreduce_microstep: 96.99 | step_microstep: 19.03
[2025-04-25 09:30:34,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.62 | bwd: 5736.35 | bwd_inner: 5639.31 | bwd_allreduce: 97.01 | step: 19.03

  2%|▏         | 639/41250 [1:32:59<97:18:49,  8.63s/it]
                                                        
{'loss': 0.2992, 'grad_norm': 3.264282464981079, 'learning_rate': 2.064620355411955e-05, 'epoch': 0.15}

  2%|▏         | 639/41250 [1:32:59<97:18:49,  8.63s/it][2025-04-25 09:30:43,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 09:30:43,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.55 | bwd_microstep: 5848.46 | bwd_inner_microstep: 5682.12 | bwd_allreduce_microstep: 166.30 | step_microstep: 18.85
[2025-04-25 09:30:43,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5848.47 | bwd_inner: 5682.12 | bwd_allreduce: 166.32 | step: 18.85

  2%|▏         | 640/41250 [1:33:08<97:51:20,  8.67s/it]
                                                        
{'loss': 0.2995, 'grad_norm': 3.3905038833618164, 'learning_rate': 2.0678513731825527e-05, 'epoch': 0.16}

  2%|▏         | 640/41250 [1:33:08<97:51:20,  8.67s/it][2025-04-25 09:30:51,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:30:51,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.51 | bwd_microstep: 5771.83 | bwd_inner_microstep: 5626.49 | bwd_allreduce_microstep: 145.30 | step_microstep: 18.64
[2025-04-25 09:30:51,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.51 | bwd: 5771.85 | bwd_inner: 5626.49 | bwd_allreduce: 145.32 | step: 18.65

  2%|▏         | 641/41250 [1:33:17<97:53:03,  8.68s/it]
                                                        
{'loss': 0.1279, 'grad_norm': 1.7851625680923462, 'learning_rate': 2.0710823909531505e-05, 'epoch': 0.16}

  2%|▏         | 641/41250 [1:33:17<97:53:03,  8.68s/it][2025-04-25 09:31:00,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:31:00,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.08 | bwd_microstep: 5750.33 | bwd_inner_microstep: 5688.32 | bwd_allreduce_microstep: 61.96 | step_microstep: 18.52
[2025-04-25 09:31:00,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.08 | bwd: 5750.34 | bwd_inner: 5688.32 | bwd_allreduce: 61.98 | step: 18.52

  2%|▏         | 642/41250 [1:33:25<97:52:47,  8.68s/it]
                                                        
{'loss': 0.3937, 'grad_norm': 3.25321102142334, 'learning_rate': 2.074313408723748e-05, 'epoch': 0.16}

  2%|▏         | 642/41250 [1:33:25<97:52:47,  8.68s/it][2025-04-25 09:31:09,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:31:09,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.23 | bwd_microstep: 5770.26 | bwd_inner_microstep: 5647.11 | bwd_allreduce_microstep: 123.10 | step_microstep: 18.37
[2025-04-25 09:31:09,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.23 | bwd: 5770.27 | bwd_inner: 5647.11 | bwd_allreduce: 123.12 | step: 18.37

  2%|▏         | 643/41250 [1:33:34<97:52:50,  8.68s/it]
                                                        
{'loss': 0.0963, 'grad_norm': 1.2782528400421143, 'learning_rate': 2.0775444264943457e-05, 'epoch': 0.16}

  2%|▏         | 643/41250 [1:33:34<97:52:50,  8.68s/it][2025-04-25 09:31:17,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:31:17,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.13 | bwd_microstep: 5703.51 | bwd_inner_microstep: 5636.64 | bwd_allreduce_microstep: 66.83 | step_microstep: 18.42
[2025-04-25 09:31:17,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.13 | bwd: 5703.53 | bwd_inner: 5636.63 | bwd_allreduce: 66.85 | step: 18.42

  2%|▏         | 644/41250 [1:33:43<97:38:06,  8.66s/it]
                                                        
{'loss': 0.0984, 'grad_norm': 1.6216808557510376, 'learning_rate': 2.080775444264944e-05, 'epoch': 0.16}

  2%|▏         | 644/41250 [1:33:43<97:38:06,  8.66s/it][2025-04-25 09:31:26,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:31:26,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5711.30 | bwd_inner_microstep: 5698.85 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.31
[2025-04-25 09:31:26,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5711.31 | bwd_inner: 5698.85 | bwd_allreduce: 12.42 | step: 18.31

  2%|▏         | 645/41250 [1:33:51<97:35:25,  8.65s/it]
                                                        
{'loss': 0.273, 'grad_norm': 1.8639237880706787, 'learning_rate': 2.0840064620355416e-05, 'epoch': 0.16}

  2%|▏         | 645/41250 [1:33:51<97:35:25,  8.65s/it][2025-04-25 09:31:34,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 09:31:34,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.88 | bwd_microstep: 5702.83 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 42.85 | step_microstep: 18.79
[2025-04-25 09:31:34,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.88 | bwd: 5702.84 | bwd_inner: 5659.93 | bwd_allreduce: 42.87 | step: 18.80

  2%|▏         | 646/41250 [1:34:00<97:29:14,  8.64s/it]
                                                        
{'loss': 0.2588, 'grad_norm': 2.4081945419311523, 'learning_rate': 2.087237479806139e-05, 'epoch': 0.16}

  2%|▏         | 646/41250 [1:34:00<97:29:14,  8.64s/it][2025-04-25 09:31:43,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:31:43,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.60 | bwd_microstep: 5762.42 | bwd_inner_microstep: 5658.64 | bwd_allreduce_microstep: 103.75 | step_microstep: 18.27
[2025-04-25 09:31:43,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.60 | bwd: 5762.44 | bwd_inner: 5658.64 | bwd_allreduce: 103.76 | step: 18.28

  2%|▏         | 647/41250 [1:34:08<97:37:10,  8.66s/it]
                                                        
{'loss': 0.0967, 'grad_norm': 1.5468850135803223, 'learning_rate': 2.090468497576737e-05, 'epoch': 0.16}

  2%|▏         | 647/41250 [1:34:09<97:37:10,  8.66s/it][2025-04-25 09:31:52,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 09:31:52,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.25 | bwd_microstep: 5774.25 | bwd_inner_microstep: 5658.94 | bwd_allreduce_microstep: 115.27 | step_microstep: 18.63
[2025-04-25 09:31:52,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.25 | bwd: 5774.26 | bwd_inner: 5658.94 | bwd_allreduce: 115.28 | step: 18.64

  2%|▏         | 648/41250 [1:34:17<97:45:27,  8.67s/it]
                                                        
{'loss': 0.3011, 'grad_norm': 2.5236825942993164, 'learning_rate': 2.0936995153473347e-05, 'epoch': 0.16}

  2%|▏         | 648/41250 [1:34:17<97:45:27,  8.67s/it][2025-04-25 09:32:01,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:32:01,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.58 | bwd_microstep: 5729.84 | bwd_inner_microstep: 5717.13 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.44
[2025-04-25 09:32:01,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.58 | bwd: 5729.85 | bwd_inner: 5717.13 | bwd_allreduce: 12.68 | step: 18.44

  2%|▏         | 649/41250 [1:34:26<97:45:54,  8.67s/it]
                                                        
{'loss': 0.2657, 'grad_norm': 2.7028563022613525, 'learning_rate': 2.096930533117932e-05, 'epoch': 0.16}

  2%|▏         | 649/41250 [1:34:26<97:45:54,  8.67s/it][2025-04-25 09:32:09,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:32:09,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.25 | bwd_microstep: 5711.66 | bwd_inner_microstep: 5698.94 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.61
[2025-04-25 09:32:09,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.25 | bwd: 5711.67 | bwd_inner: 5698.94 | bwd_allreduce: 12.69 | step: 18.62

  2%|▏         | 650/41250 [1:34:35<97:45:15,  8.67s/it]
                                                        
{'loss': 0.2116, 'grad_norm': 3.3580846786499023, 'learning_rate': 2.10016155088853e-05, 'epoch': 0.16}

  2%|▏         | 650/41250 [1:34:35<97:45:15,  8.67s/it][2025-04-25 09:32:18,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:32:18,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5696.39 | bwd_inner_microstep: 5683.93 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.47
[2025-04-25 09:32:18,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.00 | bwd: 5696.40 | bwd_inner: 5683.93 | bwd_allreduce: 12.43 | step: 18.47

  2%|▏         | 651/41250 [1:34:43<97:38:50,  8.66s/it]
                                                        
{'loss': 0.2427, 'grad_norm': 2.07120680809021, 'learning_rate': 2.103392568659128e-05, 'epoch': 0.16}

  2%|▏         | 651/41250 [1:34:43<97:38:50,  8.66s/it][2025-04-25 09:32:27,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 09:32:27,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.86 | bwd_microstep: 5883.33 | bwd_inner_microstep: 5870.57 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.64
[2025-04-25 09:32:27,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.86 | bwd: 5883.35 | bwd_inner: 5870.57 | bwd_allreduce: 12.73 | step: 18.64

  2%|▏         | 652/41250 [1:34:52<98:28:45,  8.73s/it]
                                                        
{'loss': 0.2009, 'grad_norm': 1.6533195972442627, 'learning_rate': 2.1066235864297258e-05, 'epoch': 0.16}

  2%|▏         | 652/41250 [1:34:52<98:28:45,  8.73s/it][2025-04-25 09:32:35,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:32:35,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.13 | bwd_microstep: 5751.32 | bwd_inner_microstep: 5704.67 | bwd_allreduce_microstep: 46.60 | step_microstep: 18.48
[2025-04-25 09:32:35,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.13 | bwd: 5751.33 | bwd_inner: 5704.67 | bwd_allreduce: 46.62 | step: 18.48

  2%|▏         | 653/41250 [1:35:01<98:20:40,  8.72s/it]
                                                        
{'loss': 0.1723, 'grad_norm': 3.7816097736358643, 'learning_rate': 2.1098546042003233e-05, 'epoch': 0.16}

  2%|▏         | 653/41250 [1:35:01<98:20:40,  8.72s/it][2025-04-25 09:32:44,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:32:44,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.04 | bwd_microstep: 5702.60 | bwd_inner_microstep: 5689.71 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.61
[2025-04-25 09:32:44,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.04 | bwd: 5702.62 | bwd_inner: 5689.71 | bwd_allreduce: 12.87 | step: 18.61

  2%|▏         | 654/41250 [1:35:09<98:06:23,  8.70s/it]
                                                        
{'loss': 0.2677, 'grad_norm': 2.2561681270599365, 'learning_rate': 2.113085621970921e-05, 'epoch': 0.16}

  2%|▏         | 654/41250 [1:35:09<98:06:23,  8.70s/it][2025-04-25 09:32:53,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:32:53,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2904.18 | bwd_microstep: 5800.69 | bwd_inner_microstep: 5787.92 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.36
[2025-04-25 09:32:53,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2904.18 | bwd: 5800.71 | bwd_inner: 5787.93 | bwd_allreduce: 12.74 | step: 18.37

  2%|▏         | 655/41250 [1:35:18<98:24:11,  8.73s/it]
                                                        
{'loss': 0.276, 'grad_norm': 2.4120302200317383, 'learning_rate': 2.116316639741519e-05, 'epoch': 0.16}

  2%|▏         | 655/41250 [1:35:18<98:24:11,  8.73s/it][2025-04-25 09:33:02,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:33:02,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.30 | bwd_microstep: 5764.44 | bwd_inner_microstep: 5696.29 | bwd_allreduce_microstep: 68.11 | step_microstep: 18.15
[2025-04-25 09:33:02,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.30 | bwd: 5764.46 | bwd_inner: 5696.29 | bwd_allreduce: 68.12 | step: 18.15

  2%|▏         | 656/41250 [1:35:27<98:19:03,  8.72s/it]
                                                        
{'loss': 0.1425, 'grad_norm': 2.3189382553100586, 'learning_rate': 2.1195476575121163e-05, 'epoch': 0.16}

  2%|▏         | 656/41250 [1:35:27<98:19:03,  8.72s/it][2025-04-25 09:33:10,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:33:10,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.36 | bwd_microstep: 5716.23 | bwd_inner_microstep: 5703.45 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.52
[2025-04-25 09:33:10,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.36 | bwd: 5716.24 | bwd_inner: 5703.45 | bwd_allreduce: 12.75 | step: 18.52

  2%|▏         | 657/41250 [1:35:36<98:06:38,  8.70s/it]
                                                        
{'loss': 0.3243, 'grad_norm': 2.9623618125915527, 'learning_rate': 2.122778675282714e-05, 'epoch': 0.16}

  2%|▏         | 657/41250 [1:35:36<98:06:38,  8.70s/it][2025-04-25 09:33:19,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:33:19,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.97 | bwd_microstep: 5715.51 | bwd_inner_microstep: 5661.94 | bwd_allreduce_microstep: 53.53 | step_microstep: 18.46
[2025-04-25 09:33:19,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.97 | bwd: 5715.53 | bwd_inner: 5661.94 | bwd_allreduce: 53.55 | step: 18.46

  2%|▏         | 658/41250 [1:35:44<97:52:51,  8.68s/it]
                                                        
{'loss': 0.0583, 'grad_norm': 0.8837235569953918, 'learning_rate': 2.1260096930533122e-05, 'epoch': 0.16}

  2%|▏         | 658/41250 [1:35:44<97:52:51,  8.68s/it][2025-04-25 09:33:28,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 09:33:28,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.56 | bwd_microstep: 5754.19 | bwd_inner_microstep: 5692.85 | bwd_allreduce_microstep: 61.30 | step_microstep: 18.88
[2025-04-25 09:33:28,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5754.20 | bwd_inner: 5692.84 | bwd_allreduce: 61.32 | step: 18.89

  2%|▏         | 659/41250 [1:35:53<97:54:11,  8.68s/it]
                                                        
{'loss': 0.1356, 'grad_norm': 1.0483766794204712, 'learning_rate': 2.1292407108239093e-05, 'epoch': 0.16}

  2%|▏         | 659/41250 [1:35:53<97:54:11,  8.68s/it][2025-04-25 09:33:36,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 09:33:36,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.76 | bwd_microstep: 5886.36 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 185.48 | step_microstep: 18.48
[2025-04-25 09:33:36,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.76 | bwd: 5886.37 | bwd_inner: 5700.83 | bwd_allreduce: 185.50 | step: 18.48

  2%|▏         | 660/41250 [1:36:02<98:22:06,  8.72s/it]
                                                        
{'loss': 0.2945, 'grad_norm': 2.277325391769409, 'learning_rate': 2.1324717285945075e-05, 'epoch': 0.16}

  2%|▏         | 660/41250 [1:36:02<98:22:06,  8.72s/it][2025-04-25 09:33:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:33:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5881.24 | bwd_inner_microstep: 5714.95 | bwd_allreduce_microstep: 166.25 | step_microstep: 18.72
[2025-04-25 09:33:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5881.25 | bwd_inner: 5714.95 | bwd_allreduce: 166.27 | step: 18.72

  2%|▏         | 661/41250 [1:36:11<98:40:47,  8.75s/it]
                                                        
{'loss': 0.3728, 'grad_norm': 3.502739429473877, 'learning_rate': 2.1357027463651052e-05, 'epoch': 0.16}

  2%|▏         | 661/41250 [1:36:11<98:40:47,  8.75s/it][2025-04-25 09:33:54,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:33:54,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.66 | bwd_microstep: 5718.82 | bwd_inner_microstep: 5653.34 | bwd_allreduce_microstep: 65.44 | step_microstep: 18.53
[2025-04-25 09:33:54,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.66 | bwd: 5718.84 | bwd_inner: 5653.34 | bwd_allreduce: 65.46 | step: 18.53

  2%|▏         | 662/41250 [1:36:19<98:15:31,  8.72s/it]
                                                        
{'loss': 0.1461, 'grad_norm': 3.265307903289795, 'learning_rate': 2.138933764135703e-05, 'epoch': 0.16}

  2%|▏         | 662/41250 [1:36:19<98:15:31,  8.72s/it][2025-04-25 09:34:03,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:34:03,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.91 | bwd_microstep: 5755.07 | bwd_inner_microstep: 5663.26 | bwd_allreduce_microstep: 91.77 | step_microstep: 18.50
[2025-04-25 09:34:03,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.91 | bwd: 5755.08 | bwd_inner: 5663.26 | bwd_allreduce: 91.79 | step: 18.50

  2%|▏         | 663/41250 [1:36:28<98:09:28,  8.71s/it]
                                                        
{'loss': 0.1752, 'grad_norm': 2.1580700874328613, 'learning_rate': 2.1421647819063005e-05, 'epoch': 0.16}

  2%|▏         | 663/41250 [1:36:28<98:09:28,  8.71s/it][2025-04-25 09:34:11,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:34:11,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.76 | bwd_microstep: 5722.58 | bwd_inner_microstep: 5709.75 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.57
[2025-04-25 09:34:11,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.76 | bwd: 5722.60 | bwd_inner: 5709.75 | bwd_allreduce: 12.81 | step: 18.57

  2%|▏         | 664/41250 [1:36:37<98:01:21,  8.69s/it]
                                                        
{'loss': 0.1064, 'grad_norm': 1.2846715450286865, 'learning_rate': 2.1453957996768983e-05, 'epoch': 0.16}

  2%|▏         | 664/41250 [1:36:37<98:01:21,  8.69s/it][2025-04-25 09:34:20,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:34:20,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.76 | bwd_microstep: 5712.06 | bwd_inner_microstep: 5699.28 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.39
[2025-04-25 09:34:20,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.76 | bwd: 5712.07 | bwd_inner: 5699.28 | bwd_allreduce: 12.75 | step: 18.39

  2%|▏         | 665/41250 [1:36:45<97:51:14,  8.68s/it]
                                                        
{'loss': 0.2088, 'grad_norm': 2.369192361831665, 'learning_rate': 2.1486268174474964e-05, 'epoch': 0.16}

  2%|▏         | 665/41250 [1:36:45<97:51:14,  8.68s/it][2025-04-25 09:34:29,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:34:29,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.21 | bwd_microstep: 5779.01 | bwd_inner_microstep: 5664.39 | bwd_allreduce_microstep: 114.58 | step_microstep: 18.44
[2025-04-25 09:34:29,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.21 | bwd: 5779.02 | bwd_inner: 5664.39 | bwd_allreduce: 114.59 | step: 18.44

  2%|▏         | 666/41250 [1:36:54<97:53:58,  8.68s/it]
                                                        
{'loss': 0.2724, 'grad_norm': 2.6410739421844482, 'learning_rate': 2.1518578352180935e-05, 'epoch': 0.16}

  2%|▏         | 666/41250 [1:36:54<97:53:58,  8.68s/it][2025-04-25 09:34:37,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:34:37,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.93 | bwd_microstep: 5686.99 | bwd_inner_microstep: 5651.92 | bwd_allreduce_microstep: 35.03 | step_microstep: 18.60
[2025-04-25 09:34:37,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.93 | bwd: 5687.00 | bwd_inner: 5651.92 | bwd_allreduce: 35.04 | step: 18.60

  2%|▏         | 667/41250 [1:37:02<97:37:09,  8.66s/it]
                                                        
{'loss': 0.1346, 'grad_norm': 2.3709752559661865, 'learning_rate': 2.1550888529886916e-05, 'epoch': 0.16}

  2%|▏         | 667/41250 [1:37:02<97:37:09,  8.66s/it][2025-04-25 09:34:46,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 09:34:46,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.06 | bwd_microstep: 5716.89 | bwd_inner_microstep: 5657.19 | bwd_allreduce_microstep: 59.66 | step_microstep: 18.50
[2025-04-25 09:34:46,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.06 | bwd: 5716.90 | bwd_inner: 5657.19 | bwd_allreduce: 59.67 | step: 18.50

  2%|▏         | 668/41250 [1:37:11<97:32:51,  8.65s/it]
                                                        
{'loss': 0.4595, 'grad_norm': 3.5824873447418213, 'learning_rate': 2.1583198707592894e-05, 'epoch': 0.16}

  2%|▏         | 668/41250 [1:37:11<97:32:51,  8.65s/it][2025-04-25 09:34:55,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:34:55,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.10 | bwd_microstep: 5802.00 | bwd_inner_microstep: 5789.01 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.37
[2025-04-25 09:34:55,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.10 | bwd: 5802.02 | bwd_inner: 5789.01 | bwd_allreduce: 12.97 | step: 18.38

  2%|▏         | 669/41250 [1:37:20<97:59:44,  8.69s/it]
                                                        
{'loss': 0.2912, 'grad_norm': 2.2559292316436768, 'learning_rate': 2.1615508885298872e-05, 'epoch': 0.16}

  2%|▏         | 669/41250 [1:37:20<97:59:44,  8.69s/it][2025-04-25 09:35:03,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 09:35:03,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.59 | bwd_microstep: 5709.48 | bwd_inner_microstep: 5696.73 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.28
[2025-04-25 09:35:03,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.59 | bwd: 5709.49 | bwd_inner: 5696.73 | bwd_allreduce: 12.72 | step: 18.29

  2%|▏         | 670/41250 [1:37:29<97:49:51,  8.68s/it]
                                                        
{'loss': 0.0435, 'grad_norm': 0.6132776737213135, 'learning_rate': 2.1647819063004847e-05, 'epoch': 0.16}

  2%|▏         | 670/41250 [1:37:29<97:49:51,  8.68s/it][2025-04-25 09:35:12,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:35:12,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.54 | bwd_microstep: 5722.70 | bwd_inner_microstep: 5709.85 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.54
[2025-04-25 09:35:12,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.54 | bwd: 5722.71 | bwd_inner: 5709.85 | bwd_allreduce: 12.82 | step: 18.54

  2%|▏         | 671/41250 [1:37:37<97:46:40,  8.67s/it]
                                                        
{'loss': 0.4241, 'grad_norm': 9.71865177154541, 'learning_rate': 2.1680129240710824e-05, 'epoch': 0.16}

  2%|▏         | 671/41250 [1:37:37<97:46:40,  8.67s/it][2025-04-25 09:35:20,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 09:35:20,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.81 | bwd_microstep: 5685.70 | bwd_inner_microstep: 5645.83 | bwd_allreduce_microstep: 39.83 | step_microstep: 19.22
[2025-04-25 09:35:20,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.81 | bwd: 5685.72 | bwd_inner: 5645.83 | bwd_allreduce: 39.85 | step: 19.22

  2%|▏         | 672/41250 [1:37:46<97:34:41,  8.66s/it]
                                                        
{'loss': 0.3208, 'grad_norm': 2.378729820251465, 'learning_rate': 2.1712439418416806e-05, 'epoch': 0.16}

  2%|▏         | 672/41250 [1:37:46<97:34:41,  8.66s/it][2025-04-25 09:35:29,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 09:35:29,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.60 | bwd_microstep: 5716.49 | bwd_inner_microstep: 5703.74 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.96
[2025-04-25 09:35:29,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.60 | bwd: 5716.50 | bwd_inner: 5703.74 | bwd_allreduce: 12.73 | step: 18.96

  2%|▏         | 673/41250 [1:37:54<97:36:20,  8.66s/it]
                                                        
{'loss': 0.243, 'grad_norm': 2.2597811222076416, 'learning_rate': 2.1744749596122777e-05, 'epoch': 0.16}

  2%|▏         | 673/41250 [1:37:54<97:36:20,  8.66s/it][2025-04-25 09:35:38,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:35:38,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.15 | bwd_microstep: 5722.13 | bwd_inner_microstep: 5702.11 | bwd_allreduce_microstep: 19.97 | step_microstep: 18.36
[2025-04-25 09:35:38,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.15 | bwd: 5722.14 | bwd_inner: 5702.11 | bwd_allreduce: 19.99 | step: 18.37

  2%|▏         | 674/41250 [1:38:03<97:38:51,  8.66s/it]
                                                        
{'loss': 0.3258, 'grad_norm': 4.289728164672852, 'learning_rate': 2.1777059773828758e-05, 'epoch': 0.16}

  2%|▏         | 674/41250 [1:38:03<97:38:51,  8.66s/it][2025-04-25 09:35:46,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:35:46,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.78 | bwd_microstep: 5688.81 | bwd_inner_microstep: 5649.49 | bwd_allreduce_microstep: 39.28 | step_microstep: 18.56
[2025-04-25 09:35:46,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.78 | bwd: 5688.82 | bwd_inner: 5649.49 | bwd_allreduce: 39.30 | step: 18.56

  2%|▏         | 675/41250 [1:38:12<97:26:03,  8.64s/it]
                                                        
{'loss': 0.1587, 'grad_norm': 2.9379830360412598, 'learning_rate': 2.1809369951534736e-05, 'epoch': 0.16}

  2%|▏         | 675/41250 [1:38:12<97:26:03,  8.64s/it][2025-04-25 09:35:55,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 09:35:55,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.67 | bwd_microstep: 5751.67 | bwd_inner_microstep: 5689.27 | bwd_allreduce_microstep: 62.35 | step_microstep: 18.72
[2025-04-25 09:35:55,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.67 | bwd: 5751.68 | bwd_inner: 5689.27 | bwd_allreduce: 62.37 | step: 18.72

  2%|▏         | 676/41250 [1:38:20<97:34:20,  8.66s/it]
                                                        
{'loss': 0.3629, 'grad_norm': 3.741389036178589, 'learning_rate': 2.1841680129240714e-05, 'epoch': 0.16}

  2%|▏         | 676/41250 [1:38:20<97:34:20,  8.66s/it][2025-04-25 09:36:04,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:36:04,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.13 | bwd_microstep: 5750.37 | bwd_inner_microstep: 5653.80 | bwd_allreduce_microstep: 96.53 | step_microstep: 18.61
[2025-04-25 09:36:04,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.13 | bwd: 5750.39 | bwd_inner: 5653.80 | bwd_allreduce: 96.54 | step: 18.62

  2%|▏         | 677/41250 [1:38:29<97:36:52,  8.66s/it]
                                                        
{'loss': 0.1352, 'grad_norm': 1.734032154083252, 'learning_rate': 2.187399030694669e-05, 'epoch': 0.16}

  2%|▏         | 677/41250 [1:38:29<97:36:52,  8.66s/it][2025-04-25 09:36:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:36:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.93 | bwd_microstep: 5742.01 | bwd_inner_microstep: 5694.55 | bwd_allreduce_microstep: 47.42 | step_microstep: 17.98
[2025-04-25 09:36:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.93 | bwd: 5742.03 | bwd_inner: 5694.55 | bwd_allreduce: 47.43 | step: 17.98

  2%|▏         | 678/41250 [1:38:38<97:39:55,  8.67s/it]
                                                        
{'loss': 0.1455, 'grad_norm': 2.1251025199890137, 'learning_rate': 2.1906300484652666e-05, 'epoch': 0.16}

  2%|▏         | 678/41250 [1:38:38<97:39:55,  8.67s/it][2025-04-25 09:36:21,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 09:36:21,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.08 | bwd_microstep: 5668.62 | bwd_inner_microstep: 5649.00 | bwd_allreduce_microstep: 19.58 | step_microstep: 18.62
[2025-04-25 09:36:21,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.08 | bwd: 5668.63 | bwd_inner: 5649.00 | bwd_allreduce: 19.59 | step: 18.62

  2%|▏         | 679/41250 [1:38:46<97:24:48,  8.64s/it]
                                                        
{'loss': 0.1319, 'grad_norm': 2.0395991802215576, 'learning_rate': 2.1938610662358648e-05, 'epoch': 0.16}

  2%|▏         | 679/41250 [1:38:46<97:24:48,  8.64s/it][2025-04-25 09:36:30,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 09:36:30,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.62 | bwd_microstep: 5743.83 | bwd_inner_microstep: 5646.50 | bwd_allreduce_microstep: 97.29 | step_microstep: 18.63
[2025-04-25 09:36:30,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.62 | bwd: 5743.85 | bwd_inner: 5646.50 | bwd_allreduce: 97.30 | step: 18.64

  2%|▏         | 680/41250 [1:38:55<97:27:42,  8.65s/it]
                                                        
{'loss': 0.1257, 'grad_norm': 2.0104005336761475, 'learning_rate': 2.197092084006462e-05, 'epoch': 0.16}

  2%|▏         | 680/41250 [1:38:55<97:27:42,  8.65s/it][2025-04-25 09:36:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:36:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5706.69 | bwd_inner_microstep: 5682.31 | bwd_allreduce_microstep: 24.34 | step_microstep: 18.58
[2025-04-25 09:36:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5706.70 | bwd_inner: 5682.31 | bwd_allreduce: 24.35 | step: 18.58

  2%|▏         | 681/41250 [1:39:04<97:27:54,  8.65s/it]
                                                        
{'loss': 0.3163, 'grad_norm': 2.8478469848632812, 'learning_rate': 2.20032310177706e-05, 'epoch': 0.17}

  2%|▏         | 681/41250 [1:39:04<97:27:54,  8.65s/it][2025-04-25 09:36:47,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 09:36:47,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.43 | bwd_microstep: 5667.96 | bwd_inner_microstep: 5655.35 | bwd_allreduce_microstep: 12.57 | step_microstep: 19.21
[2025-04-25 09:36:47,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.43 | bwd: 5667.98 | bwd_inner: 5655.35 | bwd_allreduce: 12.59 | step: 19.21

  2%|▏         | 682/41250 [1:39:12<97:15:26,  8.63s/it]
                                                        
{'loss': 0.4254, 'grad_norm': 3.4347989559173584, 'learning_rate': 2.2035541195476578e-05, 'epoch': 0.17}

  2%|▏         | 682/41250 [1:39:12<97:15:26,  8.63s/it][2025-04-25 09:36:56,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:36:56,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.02 | bwd_microstep: 5767.91 | bwd_inner_microstep: 5755.22 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.59
[2025-04-25 09:36:56,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.02 | bwd: 5767.92 | bwd_inner: 5755.22 | bwd_allreduce: 12.66 | step: 18.59

  2%|▏         | 683/41250 [1:39:21<97:37:04,  8.66s/it]
                                                        
{'loss': 0.1737, 'grad_norm': 1.9102048873901367, 'learning_rate': 2.2067851373182556e-05, 'epoch': 0.17}

  2%|▏         | 683/41250 [1:39:21<97:37:04,  8.66s/it][2025-04-25 09:37:04,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 09:37:04,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.00 | bwd_microstep: 5746.40 | bwd_inner_microstep: 5690.48 | bwd_allreduce_microstep: 55.87 | step_microstep: 19.04
[2025-04-25 09:37:04,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.00 | bwd: 5746.41 | bwd_inner: 5690.47 | bwd_allreduce: 55.89 | step: 19.04

  2%|▏         | 684/41250 [1:39:30<97:40:47,  8.67s/it]
                                                        
{'loss': 0.2745, 'grad_norm': 2.2028024196624756, 'learning_rate': 2.210016155088853e-05, 'epoch': 0.17}

  2%|▏         | 684/41250 [1:39:30<97:40:47,  8.67s/it][2025-04-25 09:37:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 09:37:13,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.88 | bwd_microstep: 5697.87 | bwd_inner_microstep: 5683.34 | bwd_allreduce_microstep: 14.48 | step_microstep: 18.75
[2025-04-25 09:37:13,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.88 | bwd: 5697.88 | bwd_inner: 5683.34 | bwd_allreduce: 14.50 | step: 18.75

  2%|▏         | 685/41250 [1:39:38<97:34:07,  8.66s/it]
                                                        
{'loss': 0.3753, 'grad_norm': 3.0644662380218506, 'learning_rate': 2.2132471728594508e-05, 'epoch': 0.17}

  2%|▏         | 685/41250 [1:39:38<97:34:07,  8.66s/it][2025-04-25 09:37:22,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 09:37:22,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.97 | bwd_microstep: 5699.65 | bwd_inner_microstep: 5686.91 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.61
[2025-04-25 09:37:22,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.97 | bwd: 5699.66 | bwd_inner: 5686.91 | bwd_allreduce: 12.72 | step: 18.61

  2%|▏         | 686/41250 [1:39:47<97:29:37,  8.65s/it]
                                                        
{'loss': 0.0855, 'grad_norm': 1.0040618181228638, 'learning_rate': 2.2164781906300486e-05, 'epoch': 0.17}

  2%|▏         | 686/41250 [1:39:47<97:29:37,  8.65s/it][2025-04-25 09:37:30,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:37:30,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.68 | bwd_microstep: 5695.71 | bwd_inner_microstep: 5682.59 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.97
[2025-04-25 09:37:30,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.68 | bwd: 5695.72 | bwd_inner: 5682.59 | bwd_allreduce: 13.08 | step: 18.97

  2%|▏         | 687/41250 [1:39:56<97:23:24,  8.64s/it]
                                                        
{'loss': 0.218, 'grad_norm': 1.9173561334609985, 'learning_rate': 2.219709208400646e-05, 'epoch': 0.17}

  2%|▏         | 687/41250 [1:39:56<97:23:24,  8.64s/it][2025-04-25 09:37:39,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 09:37:39,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.49 | bwd_microstep: 5675.64 | bwd_inner_microstep: 5649.57 | bwd_allreduce_microstep: 26.03 | step_microstep: 19.01
[2025-04-25 09:37:39,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.49 | bwd: 5675.66 | bwd_inner: 5649.57 | bwd_allreduce: 26.04 | step: 19.01

  2%|▏         | 688/41250 [1:40:04<97:12:04,  8.63s/it]
                                                        
{'loss': 0.2511, 'grad_norm': 2.13466215133667, 'learning_rate': 2.2229402261712442e-05, 'epoch': 0.17}

  2%|▏         | 688/41250 [1:40:04<97:12:04,  8.63s/it][2025-04-25 09:37:48,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:37:48,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.61 | bwd_microstep: 5743.86 | bwd_inner_microstep: 5681.92 | bwd_allreduce_microstep: 61.89 | step_microstep: 18.42
[2025-04-25 09:37:48,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.61 | bwd: 5743.88 | bwd_inner: 5681.92 | bwd_allreduce: 61.91 | step: 18.43

  2%|▏         | 689/41250 [1:40:13<97:21:25,  8.64s/it]
                                                        
{'loss': 0.2105, 'grad_norm': 1.6592820882797241, 'learning_rate': 2.226171243941842e-05, 'epoch': 0.17}

  2%|▏         | 689/41250 [1:40:13<97:21:25,  8.64s/it][2025-04-25 09:37:56,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 09:37:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.07 | bwd_microstep: 5695.93 | bwd_inner_microstep: 5683.03 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.23
[2025-04-25 09:37:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.07 | bwd: 5695.95 | bwd_inner: 5683.03 | bwd_allreduce: 12.87 | step: 19.23

  2%|▏         | 690/41250 [1:40:21<97:21:38,  8.64s/it]
                                                        
{'loss': 0.2909, 'grad_norm': 2.2514209747314453, 'learning_rate': 2.2294022617124397e-05, 'epoch': 0.17}

  2%|▏         | 690/41250 [1:40:21<97:21:38,  8.64s/it][2025-04-25 09:38:05,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:38:05,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.83 | bwd_microstep: 5680.69 | bwd_inner_microstep: 5637.38 | bwd_allreduce_microstep: 43.27 | step_microstep: 18.38
[2025-04-25 09:38:05,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.83 | bwd: 5680.70 | bwd_inner: 5637.38 | bwd_allreduce: 43.28 | step: 18.39

  2%|▏         | 691/41250 [1:40:30<97:10:40,  8.63s/it]
                                                        
{'loss': 0.1774, 'grad_norm': 2.312990188598633, 'learning_rate': 2.2326332794830372e-05, 'epoch': 0.17}

  2%|▏         | 691/41250 [1:40:30<97:10:40,  8.63s/it][2025-04-25 09:38:13,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:38:13,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.67 | bwd_microstep: 5747.11 | bwd_inner_microstep: 5641.13 | bwd_allreduce_microstep: 105.93 | step_microstep: 18.66
[2025-04-25 09:38:13,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.67 | bwd: 5747.12 | bwd_inner: 5641.13 | bwd_allreduce: 105.95 | step: 18.66

  2%|▏         | 692/41250 [1:40:39<97:15:59,  8.63s/it]
                                                        
{'loss': 0.2449, 'grad_norm': 1.6527985334396362, 'learning_rate': 2.235864297253635e-05, 'epoch': 0.17}

  2%|▏         | 692/41250 [1:40:39<97:15:59,  8.63s/it][2025-04-25 09:38:22,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 1.05
[2025-04-25 09:38:22,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5737.00 | bwd_inner_microstep: 5677.44 | bwd_allreduce_microstep: 59.50 | step_microstep: 19.69
[2025-04-25 09:38:22,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5737.01 | bwd_inner: 5677.44 | bwd_allreduce: 59.53 | step: 19.70

  2%|▏         | 693/41250 [1:40:47<97:22:57,  8.64s/it]
                                                        
{'loss': 0.21, 'grad_norm': 1.9310829639434814, 'learning_rate': 2.2390953150242328e-05, 'epoch': 0.17}

  2%|▏         | 693/41250 [1:40:47<97:22:57,  8.64s/it][2025-04-25 09:38:31,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 09:38:31,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.78 | bwd_microstep: 5743.29 | bwd_inner_microstep: 5687.20 | bwd_allreduce_microstep: 56.05 | step_microstep: 18.80
[2025-04-25 09:38:31,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.78 | bwd: 5743.30 | bwd_inner: 5687.20 | bwd_allreduce: 56.07 | step: 18.81

  2%|▏         | 694/41250 [1:40:56<97:28:55,  8.65s/it]
                                                        
{'loss': 0.2042, 'grad_norm': 2.4250073432922363, 'learning_rate': 2.2423263327948302e-05, 'epoch': 0.17}

  2%|▏         | 694/41250 [1:40:56<97:28:55,  8.65s/it][2025-04-25 09:38:39,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:38:39,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.23 | bwd_microstep: 5685.29 | bwd_inner_microstep: 5659.83 | bwd_allreduce_microstep: 25.42 | step_microstep: 18.87
[2025-04-25 09:38:39,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.23 | bwd: 5685.31 | bwd_inner: 5659.83 | bwd_allreduce: 25.44 | step: 18.88

  2%|▏         | 695/41250 [1:41:05<97:17:29,  8.64s/it]
                                                        
{'loss': 0.2282, 'grad_norm': 2.2744898796081543, 'learning_rate': 2.2455573505654284e-05, 'epoch': 0.17}

  2%|▏         | 695/41250 [1:41:05<97:17:29,  8.64s/it][2025-04-25 09:38:48,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:38:48,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5683.33 | bwd_inner_microstep: 5670.69 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.50
[2025-04-25 09:38:48,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5683.34 | bwd_inner: 5670.69 | bwd_allreduce: 12.61 | step: 18.50

  2%|▏         | 696/41250 [1:41:13<97:11:37,  8.63s/it]
                                                        
{'loss': 0.1373, 'grad_norm': 1.5039232969284058, 'learning_rate': 2.248788368336026e-05, 'epoch': 0.17}

  2%|▏         | 696/41250 [1:41:13<97:11:37,  8.63s/it][2025-04-25 09:38:57,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 09:38:57,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.76 | bwd_microstep: 5745.65 | bwd_inner_microstep: 5650.39 | bwd_allreduce_microstep: 95.22 | step_microstep: 19.06
[2025-04-25 09:38:57,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.76 | bwd: 5745.66 | bwd_inner: 5650.39 | bwd_allreduce: 95.24 | step: 19.06

  2%|▏         | 697/41250 [1:41:22<97:17:40,  8.64s/it]
                                                        
{'loss': 0.2285, 'grad_norm': 2.2447121143341064, 'learning_rate': 2.252019386106624e-05, 'epoch': 0.17}

  2%|▏         | 697/41250 [1:41:22<97:17:40,  8.64s/it][2025-04-25 09:39:05,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 09:39:05,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2812.86 | bwd_microstep: 5694.41 | bwd_inner_microstep: 5634.10 | bwd_allreduce_microstep: 60.25 | step_microstep: 18.54
[2025-04-25 09:39:05,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2812.86 | bwd: 5694.42 | bwd_inner: 5634.10 | bwd_allreduce: 60.27 | step: 18.54

  2%|▏         | 698/41250 [1:41:31<97:07:56,  8.62s/it]
                                                        
{'loss': 0.4393, 'grad_norm': 2.477206230163574, 'learning_rate': 2.2552504038772214e-05, 'epoch': 0.17}

  2%|▏         | 698/41250 [1:41:31<97:07:56,  8.62s/it][2025-04-25 09:39:14,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 09:39:14,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.33 | bwd_microstep: 5718.72 | bwd_inner_microstep: 5674.25 | bwd_allreduce_microstep: 44.40 | step_microstep: 18.86
[2025-04-25 09:39:14,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.33 | bwd: 5718.73 | bwd_inner: 5674.25 | bwd_allreduce: 44.43 | step: 18.86

  2%|▏         | 699/41250 [1:41:39<97:12:49,  8.63s/it]
                                                        
{'loss': 0.1378, 'grad_norm': 1.2673381567001343, 'learning_rate': 2.258481421647819e-05, 'epoch': 0.17}

  2%|▏         | 699/41250 [1:41:39<97:12:49,  8.63s/it][2025-04-25 09:39:23,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 09:39:23,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.72 | bwd_microstep: 5759.57 | bwd_inner_microstep: 5641.67 | bwd_allreduce_microstep: 117.85 | step_microstep: 18.88
[2025-04-25 09:39:23,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.72 | bwd: 5759.58 | bwd_inner: 5641.67 | bwd_allreduce: 117.87 | step: 18.88

  2%|▏         | 700/41250 [1:41:48<97:21:50,  8.64s/it]
                                                        
{'loss': 0.165, 'grad_norm': 1.9782073497772217, 'learning_rate': 2.261712439418417e-05, 'epoch': 0.17}

  2%|▏         | 700/41250 [1:41:48<97:21:50,  8.64s/it][2025-04-25 09:39:31,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 09:39:31,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.75 | bwd_microstep: 5682.93 | bwd_inner_microstep: 5632.38 | bwd_allreduce_microstep: 50.49 | step_microstep: 18.99
[2025-04-25 09:39:31,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.75 | bwd: 5682.94 | bwd_inner: 5632.38 | bwd_allreduce: 50.51 | step: 18.99

  2%|▏         | 701/41250 [1:41:56<97:10:28,  8.63s/it]
                                                        
{'loss': 0.2073, 'grad_norm': 1.3545387983322144, 'learning_rate': 2.2649434571890144e-05, 'epoch': 0.17}

  2%|▏         | 701/41250 [1:41:56<97:10:28,  8.63s/it][2025-04-25 09:39:40,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 09:39:40,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5719.59 | bwd_inner_microstep: 5706.59 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.03
[2025-04-25 09:39:40,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5719.61 | bwd_inner: 5706.59 | bwd_allreduce: 12.97 | step: 19.03

  2%|▏         | 702/41250 [1:42:05<97:16:42,  8.64s/it]
                                                        
{'loss': 0.1709, 'grad_norm': 1.3697270154953003, 'learning_rate': 2.2681744749596125e-05, 'epoch': 0.17}

  2%|▏         | 702/41250 [1:42:05<97:16:42,  8.64s/it][2025-04-25 09:39:48,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:39:48,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.78 | bwd_microstep: 5700.11 | bwd_inner_microstep: 5687.15 | bwd_allreduce_microstep: 12.91 | step_microstep: 17.78
[2025-04-25 09:39:48,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.78 | bwd: 5700.12 | bwd_inner: 5687.15 | bwd_allreduce: 12.93 | step: 17.78

  2%|▏         | 703/41250 [1:42:14<97:15:56,  8.64s/it]
                                                        
{'loss': 0.2226, 'grad_norm': 1.7853517532348633, 'learning_rate': 2.2714054927302103e-05, 'epoch': 0.17}

  2%|▏         | 703/41250 [1:42:14<97:15:56,  8.64s/it][2025-04-25 09:39:57,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:39:57,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5727.87 | bwd_inner_microstep: 5679.06 | bwd_allreduce_microstep: 48.77 | step_microstep: 18.50
[2025-04-25 09:39:57,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5727.88 | bwd_inner: 5679.06 | bwd_allreduce: 48.78 | step: 18.50

  2%|▏         | 704/41250 [1:42:22<97:21:00,  8.64s/it]
                                                        
{'loss': 0.1978, 'grad_norm': 2.6107358932495117, 'learning_rate': 2.274636510500808e-05, 'epoch': 0.17}

  2%|▏         | 704/41250 [1:42:22<97:21:00,  8.64s/it][2025-04-25 09:40:06,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 09:40:06,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.34 | bwd_microstep: 5905.44 | bwd_inner_microstep: 5644.70 | bwd_allreduce_microstep: 260.69 | step_microstep: 18.51
[2025-04-25 09:40:06,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.34 | bwd: 5905.45 | bwd_inner: 5644.70 | bwd_allreduce: 260.71 | step: 18.52

  2%|▏         | 705/41250 [1:42:31<97:55:11,  8.69s/it]
                                                        
{'loss': 0.1955, 'grad_norm': 1.4150750637054443, 'learning_rate': 2.2778675282714056e-05, 'epoch': 0.17}

  2%|▏         | 705/41250 [1:42:31<97:55:11,  8.69s/it][2025-04-25 09:40:15,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.11 | optimizer_step: 1.16
[2025-04-25 09:40:15,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.76 | bwd_microstep: 5782.52 | bwd_inner_microstep: 5632.66 | bwd_allreduce_microstep: 149.80 | step_microstep: 19.56
[2025-04-25 09:40:15,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.76 | bwd: 5782.53 | bwd_inner: 5632.66 | bwd_allreduce: 149.83 | step: 19.56

  2%|▏         | 706/41250 [1:42:40<97:54:48,  8.69s/it]
                                                        
{'loss': 0.2998, 'grad_norm': 2.1429591178894043, 'learning_rate': 2.2810985460420033e-05, 'epoch': 0.17}

  2%|▏         | 706/41250 [1:42:40<97:54:48,  8.69s/it][2025-04-25 09:40:23,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 09:40:23,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.14 | bwd_microstep: 5726.64 | bwd_inner_microstep: 5671.49 | bwd_allreduce_microstep: 55.10 | step_microstep: 19.16
[2025-04-25 09:40:23,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.14 | bwd: 5726.65 | bwd_inner: 5671.49 | bwd_allreduce: 55.12 | step: 19.16

  2%|▏         | 707/41250 [1:42:49<97:47:12,  8.68s/it]
                                                        
{'loss': 0.1401, 'grad_norm': 2.0259194374084473, 'learning_rate': 2.284329563812601e-05, 'epoch': 0.17}

  2%|▏         | 707/41250 [1:42:49<97:47:12,  8.68s/it][2025-04-25 09:40:32,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 09:40:32,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.80 | bwd_microstep: 5747.22 | bwd_inner_microstep: 5700.79 | bwd_allreduce_microstep: 46.37 | step_microstep: 19.27
[2025-04-25 09:40:32,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.80 | bwd: 5747.23 | bwd_inner: 5700.79 | bwd_allreduce: 46.40 | step: 19.27

  2%|▏         | 708/41250 [1:42:57<97:46:53,  8.68s/it]
                                                        
{'loss': 0.2167, 'grad_norm': 2.454564332962036, 'learning_rate': 2.2875605815831986e-05, 'epoch': 0.17}

  2%|▏         | 708/41250 [1:42:57<97:46:53,  8.68s/it][2025-04-25 09:40:41,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:40:41,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.94 | bwd_microstep: 5765.95 | bwd_inner_microstep: 5674.99 | bwd_allreduce_microstep: 90.92 | step_microstep: 18.62
[2025-04-25 09:40:41,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.94 | bwd: 5765.97 | bwd_inner: 5674.99 | bwd_allreduce: 90.94 | step: 18.63

  2%|▏         | 709/41250 [1:43:06<97:49:29,  8.69s/it]
                                                        
{'loss': 0.1866, 'grad_norm': 2.530689239501953, 'learning_rate': 2.2907915993537967e-05, 'epoch': 0.17}

  2%|▏         | 709/41250 [1:43:06<97:49:29,  8.69s/it][2025-04-25 09:40:49,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 09:40:49,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5718.17 | bwd_inner_microstep: 5705.41 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.38
[2025-04-25 09:40:49,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5718.18 | bwd_inner: 5705.41 | bwd_allreduce: 12.73 | step: 19.38

  2%|▏         | 710/41250 [1:43:15<97:43:19,  8.68s/it]
                                                        
{'loss': 0.1807, 'grad_norm': 1.7770295143127441, 'learning_rate': 2.2940226171243945e-05, 'epoch': 0.17}

  2%|▏         | 710/41250 [1:43:15<97:43:19,  8.68s/it][2025-04-25 09:40:58,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:40:58,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.68 | bwd_microstep: 5772.54 | bwd_inner_microstep: 5662.28 | bwd_allreduce_microstep: 110.22 | step_microstep: 18.62
[2025-04-25 09:40:58,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.68 | bwd: 5772.56 | bwd_inner: 5662.28 | bwd_allreduce: 110.24 | step: 18.62

  2%|▏         | 711/41250 [1:43:23<97:45:13,  8.68s/it]
                                                        
{'loss': 0.2729, 'grad_norm': 2.1079068183898926, 'learning_rate': 2.2972536348949923e-05, 'epoch': 0.17}

  2%|▏         | 711/41250 [1:43:23<97:45:13,  8.68s/it][2025-04-25 09:41:07,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 09:41:07,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.30 | bwd_microstep: 5702.69 | bwd_inner_microstep: 5658.71 | bwd_allreduce_microstep: 43.93 | step_microstep: 19.49
[2025-04-25 09:41:07,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5702.70 | bwd_inner: 5658.71 | bwd_allreduce: 43.95 | step: 19.49

  2%|▏         | 712/41250 [1:43:32<97:31:56,  8.66s/it]
                                                        
{'loss': 0.4444, 'grad_norm': 2.6019952297210693, 'learning_rate': 2.3004846526655897e-05, 'epoch': 0.17}

  2%|▏         | 712/41250 [1:43:32<97:31:56,  8.66s/it][2025-04-25 09:41:15,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:41:15,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.80 | bwd_microstep: 5797.53 | bwd_inner_microstep: 5651.43 | bwd_allreduce_microstep: 146.06 | step_microstep: 18.72
[2025-04-25 09:41:15,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.80 | bwd: 5797.54 | bwd_inner: 5651.43 | bwd_allreduce: 146.07 | step: 18.72

  2%|▏         | 713/41250 [1:43:41<97:41:53,  8.68s/it]
                                                        
{'loss': 0.0814, 'grad_norm': 0.7544764876365662, 'learning_rate': 2.3037156704361875e-05, 'epoch': 0.17}

  2%|▏         | 713/41250 [1:43:41<97:41:53,  8.68s/it][2025-04-25 09:41:24,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 09:41:24,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.20 | bwd_microstep: 5686.55 | bwd_inner_microstep: 5660.69 | bwd_allreduce_microstep: 25.81 | step_microstep: 18.68
[2025-04-25 09:41:24,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.20 | bwd: 5686.56 | bwd_inner: 5660.69 | bwd_allreduce: 25.83 | step: 18.68

  2%|▏         | 714/41250 [1:43:49<97:28:40,  8.66s/it]
                                                        
{'loss': 0.313, 'grad_norm': 2.460009813308716, 'learning_rate': 2.3069466882067853e-05, 'epoch': 0.17}

  2%|▏         | 714/41250 [1:43:49<97:28:40,  8.66s/it][2025-04-25 09:41:33,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.21
[2025-04-25 09:41:33,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.13 | bwd_microstep: 5704.65 | bwd_inner_microstep: 5652.74 | bwd_allreduce_microstep: 51.86 | step_microstep: 19.42
[2025-04-25 09:41:33,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.13 | bwd: 5704.66 | bwd_inner: 5652.75 | bwd_allreduce: 51.88 | step: 19.42

  2%|▏         | 715/41250 [1:43:58<97:24:59,  8.65s/it]
                                                        
{'loss': 0.3377, 'grad_norm': 2.882868528366089, 'learning_rate': 2.3101777059773828e-05, 'epoch': 0.17}

  2%|▏         | 715/41250 [1:43:58<97:24:59,  8.65s/it][2025-04-25 09:41:41,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-25 09:41:41,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.54 | bwd_microstep: 5752.64 | bwd_inner_microstep: 5698.44 | bwd_allreduce_microstep: 54.15 | step_microstep: 18.81
[2025-04-25 09:41:41,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.54 | bwd: 5752.65 | bwd_inner: 5698.44 | bwd_allreduce: 54.16 | step: 18.81

  2%|▏         | 716/41250 [1:44:07<97:34:01,  8.67s/it]
                                                        
{'loss': 0.2598, 'grad_norm': 2.0482985973358154, 'learning_rate': 2.313408723747981e-05, 'epoch': 0.17}

  2%|▏         | 716/41250 [1:44:07<97:34:01,  8.67s/it][2025-04-25 09:41:50,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.12 | optimizer_step: 1.12
[2025-04-25 09:41:50,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5707.52 | bwd_inner_microstep: 5656.64 | bwd_allreduce_microstep: 50.83 | step_microstep: 19.57
[2025-04-25 09:41:50,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5707.53 | bwd_inner: 5656.64 | bwd_allreduce: 50.85 | step: 19.58

  2%|▏         | 717/41250 [1:44:15<97:26:31,  8.65s/it]
                                                        
{'loss': 0.2451, 'grad_norm': 3.87748122215271, 'learning_rate': 2.3166397415185787e-05, 'epoch': 0.17}

  2%|▏         | 717/41250 [1:44:15<97:26:31,  8.65s/it][2025-04-25 09:41:59,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 09:41:59,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5752.38 | bwd_inner_microstep: 5703.86 | bwd_allreduce_microstep: 48.47 | step_microstep: 18.62
[2025-04-25 09:41:59,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.66 | bwd: 5752.39 | bwd_inner: 5703.86 | bwd_allreduce: 48.49 | step: 18.62

  2%|▏         | 718/41250 [1:44:24<97:33:58,  8.67s/it]
                                                        
{'loss': 0.1642, 'grad_norm': 2.704887866973877, 'learning_rate': 2.3198707592891765e-05, 'epoch': 0.17}

  2%|▏         | 718/41250 [1:44:24<97:33:58,  8.67s/it][2025-04-25 09:42:07,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-25 09:42:07,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.52 | bwd_microstep: 5828.52 | bwd_inner_microstep: 5786.50 | bwd_allreduce_microstep: 41.96 | step_microstep: 19.01
[2025-04-25 09:42:07,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.52 | bwd: 5828.54 | bwd_inner: 5786.50 | bwd_allreduce: 41.99 | step: 19.01

  2%|▏         | 719/41250 [1:44:33<98:03:27,  8.71s/it]
                                                        
{'loss': 0.2119, 'grad_norm': 2.4427404403686523, 'learning_rate': 2.323101777059774e-05, 'epoch': 0.17}

  2%|▏         | 719/41250 [1:44:33<98:03:27,  8.71s/it][2025-04-25 09:42:16,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 09:42:16,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.75 | bwd_microstep: 5717.63 | bwd_inner_microstep: 5658.99 | bwd_allreduce_microstep: 58.59 | step_microstep: 18.81
[2025-04-25 09:42:16,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.75 | bwd: 5717.64 | bwd_inner: 5658.99 | bwd_allreduce: 58.61 | step: 18.82

  2%|▏         | 720/41250 [1:44:41<97:46:43,  8.69s/it]
                                                        
{'loss': 0.1725, 'grad_norm': 1.9413927793502808, 'learning_rate': 2.3263327948303717e-05, 'epoch': 0.17}

  2%|▏         | 720/41250 [1:44:41<97:46:43,  8.69s/it][2025-04-25 09:42:25,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 09:42:25,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.43 | bwd_microstep: 5757.10 | bwd_inner_microstep: 5702.16 | bwd_allreduce_microstep: 54.90 | step_microstep: 18.58
[2025-04-25 09:42:25,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.43 | bwd: 5757.12 | bwd_inner: 5702.16 | bwd_allreduce: 54.92 | step: 18.58

  2%|▏         | 721/41250 [1:44:50<97:50:43,  8.69s/it]
                                                        
{'loss': 0.2694, 'grad_norm': 2.0481793880462646, 'learning_rate': 2.3295638126009695e-05, 'epoch': 0.17}

  2%|▏         | 721/41250 [1:44:50<97:50:43,  8.69s/it][2025-04-25 09:42:33,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-25 09:42:33,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.57 | bwd_microstep: 5726.29 | bwd_inner_microstep: 5713.48 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.49
[2025-04-25 09:42:33,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.57 | bwd: 5726.30 | bwd_inner: 5713.48 | bwd_allreduce: 12.78 | step: 19.50

  2%|▏         | 722/41250 [1:44:59<97:47:00,  8.69s/it]
                                                        
{'loss': 0.1633, 'grad_norm': 1.2786457538604736, 'learning_rate': 2.332794830371567e-05, 'epoch': 0.18}

  2%|▏         | 722/41250 [1:44:59<97:47:00,  8.69s/it][2025-04-25 09:42:42,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 09:42:42,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.96 | bwd_microstep: 5894.42 | bwd_inner_microstep: 5655.65 | bwd_allreduce_microstep: 238.71 | step_microstep: 18.93
[2025-04-25 09:42:42,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.96 | bwd: 5894.43 | bwd_inner: 5655.65 | bwd_allreduce: 238.73 | step: 18.93

  2%|▏         | 723/41250 [1:45:08<98:13:24,  8.73s/it]
                                                        
{'loss': 0.195, 'grad_norm': 1.2689636945724487, 'learning_rate': 2.336025848142165e-05, 'epoch': 0.18}

  2%|▏         | 723/41250 [1:45:08<98:13:24,  8.73s/it][2025-04-25 09:42:51,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 09:42:51,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.44 | bwd_microstep: 5700.51 | bwd_inner_microstep: 5661.23 | bwd_allreduce_microstep: 39.22 | step_microstep: 18.86
[2025-04-25 09:42:51,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.44 | bwd: 5700.52 | bwd_inner: 5661.23 | bwd_allreduce: 39.24 | step: 18.86

  2%|▏         | 724/41250 [1:45:16<97:54:49,  8.70s/it]
                                                        
{'loss': 0.324, 'grad_norm': 2.4771008491516113, 'learning_rate': 2.339256865912763e-05, 'epoch': 0.18}

  2%|▏         | 724/41250 [1:45:16<97:54:49,  8.70s/it][2025-04-25 09:43:00,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:43:00,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.27 | bwd_microstep: 5795.88 | bwd_inner_microstep: 5783.12 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.64
[2025-04-25 09:43:00,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.27 | bwd: 5795.89 | bwd_inner: 5783.12 | bwd_allreduce: 12.73 | step: 18.64

  2%|▏         | 725/41250 [1:45:25<98:11:08,  8.72s/it]
                                                        
{'loss': 0.2068, 'grad_norm': 1.7309054136276245, 'learning_rate': 2.3424878836833606e-05, 'epoch': 0.18}

  2%|▏         | 725/41250 [1:45:25<98:11:08,  8.72s/it][2025-04-25 09:43:08,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 09:43:08,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.70 | bwd_microstep: 5743.60 | bwd_inner_microstep: 5710.41 | bwd_allreduce_microstep: 33.15 | step_microstep: 19.00
[2025-04-25 09:43:08,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.70 | bwd: 5743.61 | bwd_inner: 5710.41 | bwd_allreduce: 33.16 | step: 19.00

  2%|▏         | 726/41250 [1:45:34<98:05:48,  8.71s/it]
                                                        
{'loss': 0.0888, 'grad_norm': 1.5147120952606201, 'learning_rate': 2.345718901453958e-05, 'epoch': 0.18}

  2%|▏         | 726/41250 [1:45:34<98:05:48,  8.71s/it][2025-04-25 09:43:17,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 09:43:17,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.37 | bwd_microstep: 5771.11 | bwd_inner_microstep: 5655.25 | bwd_allreduce_microstep: 115.80 | step_microstep: 18.84
[2025-04-25 09:43:17,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.37 | bwd: 5771.12 | bwd_inner: 5655.25 | bwd_allreduce: 115.82 | step: 18.85

  2%|▏         | 727/41250 [1:45:42<98:00:10,  8.71s/it]
                                                        
{'loss': 0.1494, 'grad_norm': 1.2953273057937622, 'learning_rate': 2.348949919224556e-05, 'epoch': 0.18}

  2%|▏         | 727/41250 [1:45:42<98:00:10,  8.71s/it][2025-04-25 09:43:26,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:43:26,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.63 | bwd_microstep: 5761.17 | bwd_inner_microstep: 5662.29 | bwd_allreduce_microstep: 98.84 | step_microstep: 18.48
[2025-04-25 09:43:26,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.64 | bwd: 5761.19 | bwd_inner: 5662.29 | bwd_allreduce: 98.85 | step: 18.48

  2%|▏         | 728/41250 [1:45:51<97:55:21,  8.70s/it]
                                                        
{'loss': 0.2138, 'grad_norm': 2.479856491088867, 'learning_rate': 2.3521809369951537e-05, 'epoch': 0.18}

  2%|▏         | 728/41250 [1:45:51<97:55:21,  8.70s/it][2025-04-25 09:43:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 09:43:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.36 | bwd_microstep: 5787.67 | bwd_inner_microstep: 5774.73 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.87
[2025-04-25 09:43:34,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.36 | bwd: 5787.68 | bwd_inner: 5774.73 | bwd_allreduce: 12.91 | step: 18.88

  2%|▏         | 729/41250 [1:46:00<98:07:37,  8.72s/it]
                                                        
{'loss': 0.2355, 'grad_norm': 1.83456552028656, 'learning_rate': 2.355411954765751e-05, 'epoch': 0.18}

  2%|▏         | 729/41250 [1:46:00<98:07:37,  8.72s/it][2025-04-25 09:43:43,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.18
[2025-04-25 09:43:43,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.55 | bwd_microstep: 5700.99 | bwd_inner_microstep: 5688.63 | bwd_allreduce_microstep: 12.31 | step_microstep: 19.37
[2025-04-25 09:43:43,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.55 | bwd: 5701.00 | bwd_inner: 5688.63 | bwd_allreduce: 12.33 | step: 19.37

  2%|▏         | 730/41250 [1:46:08<97:50:22,  8.69s/it]
                                                        
{'loss': 0.1006, 'grad_norm': 1.1216849088668823, 'learning_rate': 2.3586429725363493e-05, 'epoch': 0.18}

  2%|▏         | 730/41250 [1:46:08<97:50:22,  8.69s/it][2025-04-25 09:43:52,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 09:43:52,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.11 | bwd_microstep: 5710.08 | bwd_inner_microstep: 5666.11 | bwd_allreduce_microstep: 43.93 | step_microstep: 18.82
[2025-04-25 09:43:52,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.11 | bwd: 5710.09 | bwd_inner: 5666.11 | bwd_allreduce: 43.94 | step: 18.82

  2%|▏         | 731/41250 [1:46:17<97:36:17,  8.67s/it]
                                                        
{'loss': 0.2774, 'grad_norm': 2.163316488265991, 'learning_rate': 2.361873990306947e-05, 'epoch': 0.18}

  2%|▏         | 731/41250 [1:46:17<97:36:17,  8.67s/it][2025-04-25 09:44:00,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 09:44:00,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.33 | bwd_microstep: 5710.43 | bwd_inner_microstep: 5697.59 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.46
[2025-04-25 09:44:00,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.33 | bwd: 5710.44 | bwd_inner: 5697.59 | bwd_allreduce: 12.82 | step: 18.46

  2%|▏         | 732/41250 [1:46:26<97:33:13,  8.67s/it]
                                                        
{'loss': 0.1453, 'grad_norm': 1.9766299724578857, 'learning_rate': 2.3651050080775448e-05, 'epoch': 0.18}

  2%|▏         | 732/41250 [1:46:26<97:33:13,  8.67s/it][2025-04-25 09:44:09,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:44:09,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.83 | bwd_microstep: 5681.82 | bwd_inner_microstep: 5649.89 | bwd_allreduce_microstep: 31.88 | step_microstep: 18.72
[2025-04-25 09:44:09,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.83 | bwd: 5681.83 | bwd_inner: 5649.89 | bwd_allreduce: 31.90 | step: 18.72

  2%|▏         | 733/41250 [1:46:34<97:16:49,  8.64s/it]
                                                        
{'loss': 0.1735, 'grad_norm': 1.3865104913711548, 'learning_rate': 2.3683360258481423e-05, 'epoch': 0.18}

  2%|▏         | 733/41250 [1:46:34<97:16:49,  8.64s/it][2025-04-25 09:44:18,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:44:18,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.76 | bwd_microstep: 5787.07 | bwd_inner_microstep: 5678.05 | bwd_allreduce_microstep: 108.98 | step_microstep: 18.43
[2025-04-25 09:44:18,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.76 | bwd: 5787.09 | bwd_inner: 5678.05 | bwd_allreduce: 109.00 | step: 18.43

  2%|▏         | 734/41250 [1:46:43<97:31:50,  8.67s/it]
                                                        
{'loss': 0.1833, 'grad_norm': 1.8163331747055054, 'learning_rate': 2.37156704361874e-05, 'epoch': 0.18}

  2%|▏         | 734/41250 [1:46:43<97:31:50,  8.67s/it][2025-04-25 09:44:26,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 09:44:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5740.81 | bwd_inner_microstep: 5693.64 | bwd_allreduce_microstep: 47.12 | step_microstep: 18.88
[2025-04-25 09:44:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5740.82 | bwd_inner: 5693.64 | bwd_allreduce: 47.14 | step: 18.88

  2%|▏         | 735/41250 [1:46:52<97:34:03,  8.67s/it]
                                                        
{'loss': 0.1959, 'grad_norm': 1.571664571762085, 'learning_rate': 2.374798061389338e-05, 'epoch': 0.18}

  2%|▏         | 735/41250 [1:46:52<97:34:03,  8.67s/it][2025-04-25 09:44:35,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:44:35,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.97 | bwd_microstep: 5738.16 | bwd_inner_microstep: 5696.91 | bwd_allreduce_microstep: 41.21 | step_microstep: 17.87
[2025-04-25 09:44:35,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.97 | bwd: 5738.17 | bwd_inner: 5696.91 | bwd_allreduce: 41.23 | step: 17.87

  2%|▏         | 736/41250 [1:47:00<97:35:03,  8.67s/it]
                                                        
{'loss': 0.2859, 'grad_norm': 1.911751627922058, 'learning_rate': 2.3780290791599353e-05, 'epoch': 0.18}

  2%|▏         | 736/41250 [1:47:00<97:35:03,  8.67s/it][2025-04-25 09:44:44,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:44:44,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.44 | bwd_microstep: 5737.98 | bwd_inner_microstep: 5701.81 | bwd_allreduce_microstep: 36.12 | step_microstep: 18.48
[2025-04-25 09:44:44,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.44 | bwd: 5737.99 | bwd_inner: 5701.81 | bwd_allreduce: 36.13 | step: 18.48

  2%|▏         | 737/41250 [1:47:09<97:33:29,  8.67s/it]
                                                        
{'loss': 0.2531, 'grad_norm': 2.278707981109619, 'learning_rate': 2.3812600969305334e-05, 'epoch': 0.18}

  2%|▏         | 737/41250 [1:47:09<97:33:29,  8.67s/it][2025-04-25 09:44:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 09:44:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5705.29 | bwd_inner_microstep: 5692.46 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.58
[2025-04-25 09:44:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5705.31 | bwd_inner: 5692.46 | bwd_allreduce: 12.81 | step: 18.58

  2%|▏         | 738/41250 [1:47:18<97:28:27,  8.66s/it]
                                                        
{'loss': 0.1151, 'grad_norm': 1.6759947538375854, 'learning_rate': 2.3844911147011312e-05, 'epoch': 0.18}

  2%|▏         | 738/41250 [1:47:18<97:28:27,  8.66s/it][2025-04-25 09:45:01,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:45:01,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5724.76 | bwd_inner_microstep: 5712.06 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.87
[2025-04-25 09:45:01,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5724.77 | bwd_inner: 5712.06 | bwd_allreduce: 12.67 | step: 18.87

  2%|▏         | 739/41250 [1:47:26<97:27:40,  8.66s/it]
                                                        
{'loss': 0.1279, 'grad_norm': 1.4517807960510254, 'learning_rate': 2.3877221324717287e-05, 'epoch': 0.18}

  2%|▏         | 739/41250 [1:47:26<97:27:40,  8.66s/it][2025-04-25 09:45:10,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 09:45:10,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.58 | bwd_microstep: 5758.82 | bwd_inner_microstep: 5644.44 | bwd_allreduce_microstep: 114.33 | step_microstep: 19.20
[2025-04-25 09:45:10,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.58 | bwd: 5758.83 | bwd_inner: 5644.44 | bwd_allreduce: 114.35 | step: 19.20

  2%|▏         | 740/41250 [1:47:35<97:28:35,  8.66s/it]
                                                        
{'loss': 0.4589, 'grad_norm': 3.357461452484131, 'learning_rate': 2.3909531502423265e-05, 'epoch': 0.18}

  2%|▏         | 740/41250 [1:47:35<97:28:35,  8.66s/it][2025-04-25 09:45:18,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:45:18,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.00 | bwd_microstep: 5741.35 | bwd_inner_microstep: 5694.49 | bwd_allreduce_microstep: 46.81 | step_microstep: 19.10
[2025-04-25 09:45:18,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.00 | bwd: 5741.36 | bwd_inner: 5694.49 | bwd_allreduce: 46.83 | step: 19.10

  2%|▏         | 741/41250 [1:47:44<97:30:52,  8.67s/it]
                                                        
{'loss': 0.5228, 'grad_norm': 2.472233533859253, 'learning_rate': 2.3941841680129242e-05, 'epoch': 0.18}

  2%|▏         | 741/41250 [1:47:44<97:30:52,  8.67s/it][2025-04-25 09:45:27,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 09:45:27,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.65 | bwd_microstep: 5673.61 | bwd_inner_microstep: 5656.44 | bwd_allreduce_microstep: 17.13 | step_microstep: 18.81
[2025-04-25 09:45:27,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.65 | bwd: 5673.62 | bwd_inner: 5656.44 | bwd_allreduce: 17.14 | step: 18.81

  2%|▏         | 742/41250 [1:47:52<97:15:19,  8.64s/it]
                                                        
{'loss': 0.222, 'grad_norm': 2.769200563430786, 'learning_rate': 2.397415185783522e-05, 'epoch': 0.18}

  2%|▏         | 742/41250 [1:47:52<97:15:19,  8.64s/it][2025-04-25 09:45:36,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 09:45:36,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5861.57 | bwd_inner_microstep: 5692.06 | bwd_allreduce_microstep: 169.47 | step_microstep: 18.70
[2025-04-25 09:45:36,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.49 | bwd: 5861.59 | bwd_inner: 5692.06 | bwd_allreduce: 169.49 | step: 18.70

  2%|▏         | 743/41250 [1:48:01<97:45:27,  8.69s/it]
                                                        
{'loss': 0.1886, 'grad_norm': 1.573460340499878, 'learning_rate': 2.4006462035541195e-05, 'epoch': 0.18}

  2%|▏         | 743/41250 [1:48:01<97:45:27,  8.69s/it][2025-04-25 09:45:44,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:45:44,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.02 | bwd_microstep: 5737.22 | bwd_inner_microstep: 5703.67 | bwd_allreduce_microstep: 33.50 | step_microstep: 18.83
[2025-04-25 09:45:44,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.02 | bwd: 5737.23 | bwd_inner: 5703.67 | bwd_allreduce: 33.52 | step: 18.83

  2%|▏         | 744/41250 [1:48:10<97:42:24,  8.68s/it]
                                                        
{'loss': 0.226, 'grad_norm': 1.531967282295227, 'learning_rate': 2.4038772213247176e-05, 'epoch': 0.18}

  2%|▏         | 744/41250 [1:48:10<97:42:24,  8.68s/it][2025-04-25 09:45:53,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 09:45:53,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.81 | bwd_microstep: 5745.19 | bwd_inner_microstep: 5649.74 | bwd_allreduce_microstep: 95.40 | step_microstep: 19.15
[2025-04-25 09:45:53,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.81 | bwd: 5745.21 | bwd_inner: 5649.74 | bwd_allreduce: 95.43 | step: 19.15

  2%|▏         | 745/41250 [1:48:18<97:35:50,  8.67s/it]
                                                        
{'loss': 0.146, 'grad_norm': 1.726293921470642, 'learning_rate': 2.4071082390953154e-05, 'epoch': 0.18}

  2%|▏         | 745/41250 [1:48:18<97:35:50,  8.67s/it][2025-04-25 09:46:02,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 09:46:02,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.26 | bwd_microstep: 6012.50 | bwd_inner_microstep: 5640.80 | bwd_allreduce_microstep: 371.65 | step_microstep: 18.94
[2025-04-25 09:46:02,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.26 | bwd: 6012.52 | bwd_inner: 5640.80 | bwd_allreduce: 371.67 | step: 18.95

  2%|▏         | 746/41250 [1:48:27<98:26:30,  8.75s/it]
                                                        
{'loss': 0.2746, 'grad_norm': 1.7987594604492188, 'learning_rate': 2.410339256865913e-05, 'epoch': 0.18}

  2%|▏         | 746/41250 [1:48:27<98:26:30,  8.75s/it][2025-04-25 09:46:11,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:46:11,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.80 | bwd_microstep: 5747.30 | bwd_inner_microstep: 5633.78 | bwd_allreduce_microstep: 113.48 | step_microstep: 17.94
[2025-04-25 09:46:11,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.80 | bwd: 5747.31 | bwd_inner: 5633.78 | bwd_allreduce: 113.50 | step: 17.94

  2%|▏         | 747/41250 [1:48:36<98:08:32,  8.72s/it]
                                                        
{'loss': 0.4046, 'grad_norm': 2.5501925945281982, 'learning_rate': 2.4135702746365106e-05, 'epoch': 0.18}

  2%|▏         | 747/41250 [1:48:36<98:08:32,  8.72s/it][2025-04-25 09:46:19,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:46:19,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.92 | bwd_microstep: 5751.29 | bwd_inner_microstep: 5631.47 | bwd_allreduce_microstep: 119.77 | step_microstep: 18.12
[2025-04-25 09:46:19,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.92 | bwd: 5751.30 | bwd_inner: 5631.47 | bwd_allreduce: 119.78 | step: 18.12

  2%|▏         | 748/41250 [1:48:45<97:55:08,  8.70s/it]
                                                        
{'loss': 0.1855, 'grad_norm': 1.5553454160690308, 'learning_rate': 2.4168012924071084e-05, 'epoch': 0.18}

  2%|▏         | 748/41250 [1:48:45<97:55:08,  8.70s/it][2025-04-25 09:46:28,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:46:28,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.32 | bwd_microstep: 5729.75 | bwd_inner_microstep: 5688.35 | bwd_allreduce_microstep: 41.35 | step_microstep: 18.20
[2025-04-25 09:46:28,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.32 | bwd: 5729.76 | bwd_inner: 5688.35 | bwd_allreduce: 41.37 | step: 18.20

  2%|▏         | 749/41250 [1:48:53<97:50:55,  8.70s/it]
                                                        
{'loss': 0.2117, 'grad_norm': 1.387158989906311, 'learning_rate': 2.4200323101777062e-05, 'epoch': 0.18}

  2%|▏         | 749/41250 [1:48:53<97:50:55,  8.70s/it][2025-04-25 09:46:37,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 09:46:37,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.77 | bwd_microstep: 5899.33 | bwd_inner_microstep: 5665.71 | bwd_allreduce_microstep: 233.57 | step_microstep: 18.68
[2025-04-25 09:46:37,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.77 | bwd: 5899.35 | bwd_inner: 5665.71 | bwd_allreduce: 233.59 | step: 18.69

  2%|▏         | 750/41250 [1:49:02<98:16:25,  8.74s/it]
                                                        
{'loss': 0.1931, 'grad_norm': 1.6266144514083862, 'learning_rate': 2.4232633279483037e-05, 'epoch': 0.18}

  2%|▏         | 750/41250 [1:49:02<98:16:25,  8.74s/it][2025-04-25 09:46:45,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 09:46:45,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.83 | bwd_microstep: 5708.48 | bwd_inner_microstep: 5695.54 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.87
[2025-04-25 09:46:45,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.83 | bwd: 5708.49 | bwd_inner: 5695.54 | bwd_allreduce: 12.91 | step: 18.87

  2%|▏         | 751/41250 [1:49:11<97:58:45,  8.71s/it]
                                                        
{'loss': 0.2877, 'grad_norm': 2.6782305240631104, 'learning_rate': 2.4264943457189018e-05, 'epoch': 0.18}

  2%|▏         | 751/41250 [1:49:11<97:58:45,  8.71s/it][2025-04-25 09:46:54,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.56 | optimizer_step: 0.90
[2025-04-25 09:46:54,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.77 | bwd_microstep: 5698.57 | bwd_inner_microstep: 5685.97 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.70
[2025-04-25 09:46:54,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.78 | bwd: 5698.58 | bwd_inner: 5685.97 | bwd_allreduce: 12.57 | step: 18.70

  2%|▏         | 752/41250 [1:49:19<97:40:31,  8.68s/it]
                                                        
{'loss': 0.2994, 'grad_norm': 2.184847831726074, 'learning_rate': 2.4297253634894996e-05, 'epoch': 0.18}

  2%|▏         | 752/41250 [1:49:19<97:40:31,  8.68s/it][2025-04-25 09:47:03,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:47:03,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.46 | bwd_microstep: 5772.13 | bwd_inner_microstep: 5759.66 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.41
[2025-04-25 09:47:03,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.46 | bwd: 5772.15 | bwd_inner: 5759.65 | bwd_allreduce: 12.45 | step: 18.41

  2%|▏         | 753/41250 [1:49:28<97:53:16,  8.70s/it]
                                                        
{'loss': 0.1272, 'grad_norm': 1.3395894765853882, 'learning_rate': 2.432956381260097e-05, 'epoch': 0.18}

  2%|▏         | 753/41250 [1:49:28<97:53:16,  8.70s/it][2025-04-25 09:47:11,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:47:11,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.17 | bwd_microstep: 5779.35 | bwd_inner_microstep: 5625.75 | bwd_allreduce_microstep: 153.55 | step_microstep: 18.40
[2025-04-25 09:47:11,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.17 | bwd: 5779.36 | bwd_inner: 5625.75 | bwd_allreduce: 153.57 | step: 18.40

  2%|▏         | 754/41250 [1:49:37<97:51:02,  8.70s/it]
                                                        
{'loss': 0.1796, 'grad_norm': 1.6399855613708496, 'learning_rate': 2.4361873990306948e-05, 'epoch': 0.18}

  2%|▏         | 754/41250 [1:49:37<97:51:02,  8.70s/it][2025-04-25 09:47:20,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:47:20,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.46 | bwd_microstep: 5762.73 | bwd_inner_microstep: 5692.40 | bwd_allreduce_microstep: 70.28 | step_microstep: 18.36
[2025-04-25 09:47:20,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.46 | bwd: 5762.74 | bwd_inner: 5692.40 | bwd_allreduce: 70.29 | step: 18.37

  2%|▏         | 755/41250 [1:49:45<97:50:10,  8.70s/it]
                                                        
{'loss': 0.2576, 'grad_norm': 1.4177905321121216, 'learning_rate': 2.4394184168012926e-05, 'epoch': 0.18}

  2%|▏         | 755/41250 [1:49:45<97:50:10,  8.70s/it][2025-04-25 09:47:29,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 09:47:29,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5757.83 | bwd_inner_microstep: 5684.79 | bwd_allreduce_microstep: 73.00 | step_microstep: 18.84
[2025-04-25 09:47:29,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5757.84 | bwd_inner: 5684.79 | bwd_allreduce: 73.01 | step: 18.84

  2%|▏         | 756/41250 [1:49:54<97:50:36,  8.70s/it]
                                                        
{'loss': 0.4025, 'grad_norm': 2.822040319442749, 'learning_rate': 2.4426494345718904e-05, 'epoch': 0.18}

  2%|▏         | 756/41250 [1:49:54<97:50:36,  8.70s/it][2025-04-25 09:47:37,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:47:37,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5689.11 | bwd_inner_microstep: 5676.57 | bwd_allreduce_microstep: 12.50 | step_microstep: 17.83
[2025-04-25 09:47:37,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5689.13 | bwd_inner: 5676.57 | bwd_allreduce: 12.51 | step: 17.84

  2%|▏         | 757/41250 [1:50:03<97:34:29,  8.67s/it]
                                                        
{'loss': 0.2009, 'grad_norm': 2.586228132247925, 'learning_rate': 2.445880452342488e-05, 'epoch': 0.18}

  2%|▏         | 757/41250 [1:50:03<97:34:29,  8.67s/it][2025-04-25 09:47:46,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.05 | optimizer_step: 1.13
[2025-04-25 09:47:46,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.47 | bwd_microstep: 5720.30 | bwd_inner_microstep: 5658.12 | bwd_allreduce_microstep: 62.12 | step_microstep: 19.27
[2025-04-25 09:47:46,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.48 | bwd: 5720.31 | bwd_inner: 5658.12 | bwd_allreduce: 62.14 | step: 19.27

  2%|▏         | 758/41250 [1:50:11<97:27:19,  8.66s/it]
                                                        
{'loss': 0.1563, 'grad_norm': 1.2119499444961548, 'learning_rate': 2.449111470113086e-05, 'epoch': 0.18}

  2%|▏         | 758/41250 [1:50:11<97:27:19,  8.66s/it][2025-04-25 09:47:55,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:47:55,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.50 | bwd_microstep: 5785.10 | bwd_inner_microstep: 5647.45 | bwd_allreduce_microstep: 137.61 | step_microstep: 18.58
[2025-04-25 09:47:55,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.50 | bwd: 5785.11 | bwd_inner: 5647.45 | bwd_allreduce: 137.62 | step: 18.59

  2%|▏         | 759/41250 [1:50:20<97:33:37,  8.67s/it]
                                                        
{'loss': 0.2337, 'grad_norm': 2.073179006576538, 'learning_rate': 2.4523424878836838e-05, 'epoch': 0.18}

  2%|▏         | 759/41250 [1:50:20<97:33:37,  8.67s/it][2025-04-25 09:48:04,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 09:48:04,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.74 | bwd_microstep: 5765.98 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 73.08 | step_microstep: 18.44
[2025-04-25 09:48:04,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.75 | bwd: 5766.00 | bwd_inner: 5692.86 | bwd_allreduce: 73.10 | step: 18.44

  2%|▏         | 760/41250 [1:50:29<97:40:26,  8.68s/it]
                                                        
{'loss': 0.3392, 'grad_norm': 2.826591730117798, 'learning_rate': 2.4555735056542812e-05, 'epoch': 0.18}

  2%|▏         | 760/41250 [1:50:29<97:40:26,  8.68s/it][2025-04-25 09:48:12,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 09:48:12,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.74 | bwd_microstep: 5762.86 | bwd_inner_microstep: 5657.52 | bwd_allreduce_microstep: 105.30 | step_microstep: 18.79
[2025-04-25 09:48:12,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.74 | bwd: 5762.88 | bwd_inner: 5657.52 | bwd_allreduce: 105.32 | step: 18.80

  2%|▏         | 761/41250 [1:50:38<97:39:44,  8.68s/it]
                                                        
{'loss': 0.3794, 'grad_norm': 2.5536179542541504, 'learning_rate': 2.458804523424879e-05, 'epoch': 0.18}

  2%|▏         | 761/41250 [1:50:38<97:39:44,  8.68s/it][2025-04-25 09:48:21,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 09:48:21,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.52 | bwd_microstep: 5746.57 | bwd_inner_microstep: 5705.56 | bwd_allreduce_microstep: 40.96 | step_microstep: 18.77
[2025-04-25 09:48:21,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.52 | bwd: 5746.58 | bwd_inner: 5705.56 | bwd_allreduce: 40.98 | step: 18.77

  2%|▏         | 762/41250 [1:50:46<97:40:55,  8.69s/it]
                                                        
{'loss': 0.2021, 'grad_norm': 1.5970779657363892, 'learning_rate': 2.4620355411954768e-05, 'epoch': 0.18}

  2%|▏         | 762/41250 [1:50:46<97:40:55,  8.69s/it][2025-04-25 09:48:30,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:48:30,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.44 | bwd_microstep: 5761.64 | bwd_inner_microstep: 5650.31 | bwd_allreduce_microstep: 111.29 | step_microstep: 17.84
[2025-04-25 09:48:30,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.44 | bwd: 5761.65 | bwd_inner: 5650.31 | bwd_allreduce: 111.31 | step: 17.85

  2%|▏         | 763/41250 [1:50:55<97:38:27,  8.68s/it]
                                                        
{'loss': 0.2351, 'grad_norm': 1.7224986553192139, 'learning_rate': 2.4652665589660746e-05, 'epoch': 0.18}

  2%|▏         | 763/41250 [1:50:55<97:38:27,  8.68s/it][2025-04-25 09:48:38,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 09:48:38,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.65 | bwd_microstep: 5768.23 | bwd_inner_microstep: 5665.90 | bwd_allreduce_microstep: 102.28 | step_microstep: 18.45
[2025-04-25 09:48:38,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.65 | bwd: 5768.24 | bwd_inner: 5665.90 | bwd_allreduce: 102.30 | step: 18.45

  2%|▏         | 764/41250 [1:51:04<97:39:16,  8.68s/it]
                                                        
{'loss': 0.1639, 'grad_norm': 1.2905538082122803, 'learning_rate': 2.468497576736672e-05, 'epoch': 0.19}

  2%|▏         | 764/41250 [1:51:04<97:39:16,  8.68s/it][2025-04-25 09:48:47,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:48:47,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.23 | bwd_microstep: 5749.88 | bwd_inner_microstep: 5665.32 | bwd_allreduce_microstep: 84.52 | step_microstep: 17.94
[2025-04-25 09:48:47,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.23 | bwd: 5749.89 | bwd_inner: 5665.32 | bwd_allreduce: 84.53 | step: 17.94

  2%|▏         | 765/41250 [1:51:12<97:38:57,  8.68s/it]
                                                        
{'loss': 0.1599, 'grad_norm': 1.4438012838363647, 'learning_rate': 2.47172859450727e-05, 'epoch': 0.19}

  2%|▏         | 765/41250 [1:51:12<97:38:57,  8.68s/it][2025-04-25 09:48:56,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 09:48:56,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.76 | bwd_microstep: 5781.64 | bwd_inner_microstep: 5665.46 | bwd_allreduce_microstep: 116.13 | step_microstep: 18.57
[2025-04-25 09:48:56,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.76 | bwd: 5781.65 | bwd_inner: 5665.46 | bwd_allreduce: 116.15 | step: 18.57

  2%|▏         | 766/41250 [1:51:21<97:42:12,  8.69s/it]
                                                        
{'loss': 0.1392, 'grad_norm': 1.2054691314697266, 'learning_rate': 2.474959612277868e-05, 'epoch': 0.19}

  2%|▏         | 766/41250 [1:51:21<97:42:12,  8.69s/it][2025-04-25 09:49:04,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:49:04,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5717.55 | bwd_inner_microstep: 5659.78 | bwd_allreduce_microstep: 57.73 | step_microstep: 18.43
[2025-04-25 09:49:04,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5717.56 | bwd_inner: 5659.78 | bwd_allreduce: 57.75 | step: 18.44

  2%|▏         | 767/41250 [1:51:30<97:30:11,  8.67s/it]
                                                        
{'loss': 0.191, 'grad_norm': 1.70234215259552, 'learning_rate': 2.4781906300484654e-05, 'epoch': 0.19}

  2%|▏         | 767/41250 [1:51:30<97:30:11,  8.67s/it][2025-04-25 09:49:13,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.09 | optimizer_step: 1.13
[2025-04-25 09:49:13,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.06 | bwd_microstep: 5751.83 | bwd_inner_microstep: 5711.06 | bwd_allreduce_microstep: 40.71 | step_microstep: 19.59
[2025-04-25 09:49:13,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.06 | bwd: 5751.84 | bwd_inner: 5711.06 | bwd_allreduce: 40.73 | step: 19.59

  2%|▏         | 768/41250 [1:51:38<97:34:33,  8.68s/it]
                                                        
{'loss': 0.2842, 'grad_norm': 1.9306784868240356, 'learning_rate': 2.4814216478190632e-05, 'epoch': 0.19}

  2%|▏         | 768/41250 [1:51:38<97:34:33,  8.68s/it][2025-04-25 09:49:22,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:49:22,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5732.37 | bwd_inner_microstep: 5650.32 | bwd_allreduce_microstep: 82.01 | step_microstep: 18.83
[2025-04-25 09:49:22,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5732.39 | bwd_inner: 5650.32 | bwd_allreduce: 82.03 | step: 18.83

  2%|▏         | 769/41250 [1:51:47<97:28:34,  8.67s/it]
                                                        
{'loss': 0.2301, 'grad_norm': 2.4937796592712402, 'learning_rate': 2.484652665589661e-05, 'epoch': 0.19}

  2%|▏         | 769/41250 [1:51:47<97:28:34,  8.67s/it][2025-04-25 09:49:30,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:49:30,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.73 | bwd_microstep: 5731.86 | bwd_inner_microstep: 5658.59 | bwd_allreduce_microstep: 73.23 | step_microstep: 18.48
[2025-04-25 09:49:30,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.73 | bwd: 5731.88 | bwd_inner: 5658.59 | bwd_allreduce: 73.25 | step: 18.48

  2%|▏         | 770/41250 [1:51:56<97:22:38,  8.66s/it]
                                                        
{'loss': 0.105, 'grad_norm': 1.0520209074020386, 'learning_rate': 2.4878836833602588e-05, 'epoch': 0.19}

  2%|▏         | 770/41250 [1:51:56<97:22:38,  8.66s/it][2025-04-25 09:49:39,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:49:39,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.86 | bwd_microstep: 5788.27 | bwd_inner_microstep: 5648.62 | bwd_allreduce_microstep: 139.61 | step_microstep: 18.64
[2025-04-25 09:49:39,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.86 | bwd: 5788.29 | bwd_inner: 5648.62 | bwd_allreduce: 139.63 | step: 18.64

  2%|▏         | 771/41250 [1:52:04<97:30:31,  8.67s/it]
                                                        
{'loss': 0.148, 'grad_norm': 1.7386631965637207, 'learning_rate': 2.4911147011308562e-05, 'epoch': 0.19}

  2%|▏         | 771/41250 [1:52:04<97:30:31,  8.67s/it][2025-04-25 09:49:48,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:49:48,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.49 | bwd_microstep: 5748.87 | bwd_inner_microstep: 5709.87 | bwd_allreduce_microstep: 38.94 | step_microstep: 18.56
[2025-04-25 09:49:48,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.49 | bwd: 5748.88 | bwd_inner: 5709.87 | bwd_allreduce: 38.97 | step: 18.56

  2%|▏         | 772/41250 [1:52:13<97:35:56,  8.68s/it]
                                                        
{'loss': 0.2517, 'grad_norm': 2.987323045730591, 'learning_rate': 2.4943457189014543e-05, 'epoch': 0.19}

  2%|▏         | 772/41250 [1:52:13<97:35:56,  8.68s/it][2025-04-25 09:49:56,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.25
[2025-04-25 09:49:56,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.10 | bwd_microstep: 5784.90 | bwd_inner_microstep: 5662.94 | bwd_allreduce_microstep: 121.91 | step_microstep: 19.77
[2025-04-25 09:49:56,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.10 | bwd: 5784.91 | bwd_inner: 5662.94 | bwd_allreduce: 121.93 | step: 19.77

  2%|▏         | 773/41250 [1:52:22<97:42:00,  8.69s/it]
                                                        
{'loss': 0.2691, 'grad_norm': 1.9578548669815063, 'learning_rate': 2.497576736672052e-05, 'epoch': 0.19}

  2%|▏         | 773/41250 [1:52:22<97:42:00,  8.69s/it][2025-04-25 09:50:05,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 09:50:05,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5699.83 | bwd_inner_microstep: 5665.48 | bwd_allreduce_microstep: 34.31 | step_microstep: 18.54
[2025-04-25 09:50:05,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5699.85 | bwd_inner: 5665.48 | bwd_allreduce: 34.33 | step: 18.55

  2%|▏         | 774/41250 [1:52:30<97:30:55,  8.67s/it]
                                                        
{'loss': 0.2201, 'grad_norm': 1.8825174570083618, 'learning_rate': 2.5008077544426496e-05, 'epoch': 0.19}

  2%|▏         | 774/41250 [1:52:30<97:30:55,  8.67s/it][2025-04-25 09:50:14,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 09:50:14,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.37 | bwd_microstep: 5786.55 | bwd_inner_microstep: 5662.68 | bwd_allreduce_microstep: 123.82 | step_microstep: 18.53
[2025-04-25 09:50:14,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.37 | bwd: 5786.56 | bwd_inner: 5662.68 | bwd_allreduce: 123.84 | step: 18.53

  2%|▏         | 775/41250 [1:52:39<97:35:56,  8.68s/it]
                                                        
{'loss': 0.2234, 'grad_norm': 2.2910685539245605, 'learning_rate': 2.5040387722132474e-05, 'epoch': 0.19}

  2%|▏         | 775/41250 [1:52:39<97:35:56,  8.68s/it][2025-04-25 09:50:22,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:50:22,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5712.43 | bwd_inner_microstep: 5661.36 | bwd_allreduce_microstep: 51.03 | step_microstep: 18.54
[2025-04-25 09:50:22,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.43 | bwd: 5712.44 | bwd_inner: 5661.36 | bwd_allreduce: 51.04 | step: 18.54

  2%|▏         | 776/41250 [1:52:48<97:28:36,  8.67s/it]
                                                        
{'loss': 0.1883, 'grad_norm': 2.093724489212036, 'learning_rate': 2.507269789983845e-05, 'epoch': 0.19}

  2%|▏         | 776/41250 [1:52:48<97:28:36,  8.67s/it][2025-04-25 09:50:31,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 09:50:31,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.82 | bwd_microstep: 5728.47 | bwd_inner_microstep: 5715.52 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.92
[2025-04-25 09:50:31,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.82 | bwd: 5728.49 | bwd_inner: 5715.52 | bwd_allreduce: 12.92 | step: 18.92

  2%|▏         | 777/41250 [1:52:56<97:28:50,  8.67s/it]
                                                        
{'loss': 0.2121, 'grad_norm': 1.823278784751892, 'learning_rate': 2.510500807754443e-05, 'epoch': 0.19}

  2%|▏         | 777/41250 [1:52:56<97:28:50,  8.67s/it][2025-04-25 09:50:40,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.12
[2025-04-25 09:50:40,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5714.30 | bwd_inner_microstep: 5701.43 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.64
[2025-04-25 09:50:40,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5714.31 | bwd_inner: 5701.43 | bwd_allreduce: 12.84 | step: 19.65

  2%|▏         | 778/41250 [1:53:05<97:24:01,  8.66s/it]
                                                        
{'loss': 0.1158, 'grad_norm': 0.9058712720870972, 'learning_rate': 2.5137318255250404e-05, 'epoch': 0.19}

  2%|▏         | 778/41250 [1:53:05<97:24:01,  8.66s/it][2025-04-25 09:50:48,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:50:48,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.95 | bwd_microstep: 5708.40 | bwd_inner_microstep: 5662.71 | bwd_allreduce_microstep: 45.64 | step_microstep: 18.61
[2025-04-25 09:50:48,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.95 | bwd: 5708.41 | bwd_inner: 5662.71 | bwd_allreduce: 45.66 | step: 18.61

  2%|▏         | 779/41250 [1:53:14<97:15:07,  8.65s/it]
                                                        
{'loss': 0.1457, 'grad_norm': 1.6989718675613403, 'learning_rate': 2.5169628432956382e-05, 'epoch': 0.19}

  2%|▏         | 779/41250 [1:53:14<97:15:07,  8.65s/it][2025-04-25 09:50:57,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:50:57,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.40 | bwd_microstep: 5774.68 | bwd_inner_microstep: 5659.86 | bwd_allreduce_microstep: 114.78 | step_microstep: 18.64
[2025-04-25 09:50:57,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.40 | bwd: 5774.70 | bwd_inner: 5659.86 | bwd_allreduce: 114.80 | step: 18.64

  2%|▏         | 780/41250 [1:53:22<97:22:41,  8.66s/it]
                                                        
{'loss': 0.1385, 'grad_norm': 1.5364631414413452, 'learning_rate': 2.5201938610662363e-05, 'epoch': 0.19}

  2%|▏         | 780/41250 [1:53:22<97:22:41,  8.66s/it][2025-04-25 09:51:06,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 09:51:06,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.32 | bwd_microstep: 5763.08 | bwd_inner_microstep: 5716.14 | bwd_allreduce_microstep: 46.89 | step_microstep: 18.58
[2025-04-25 09:51:06,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.32 | bwd: 5763.09 | bwd_inner: 5716.14 | bwd_allreduce: 46.91 | step: 18.58

  2%|▏         | 781/41250 [1:53:31<97:30:29,  8.67s/it]
                                                        
{'loss': 0.1852, 'grad_norm': 2.1978702545166016, 'learning_rate': 2.5234248788368337e-05, 'epoch': 0.19}

  2%|▏         | 781/41250 [1:53:31<97:30:29,  8.67s/it][2025-04-25 09:51:14,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:51:14,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.63 | bwd_microstep: 5747.62 | bwd_inner_microstep: 5691.35 | bwd_allreduce_microstep: 56.22 | step_microstep: 18.65
[2025-04-25 09:51:14,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.63 | bwd: 5747.64 | bwd_inner: 5691.35 | bwd_allreduce: 56.24 | step: 18.66

  2%|▏         | 782/41250 [1:53:40<97:31:09,  8.68s/it]
                                                        
{'loss': 0.2892, 'grad_norm': 1.9974335432052612, 'learning_rate': 2.5266558966074315e-05, 'epoch': 0.19}

  2%|▏         | 782/41250 [1:53:40<97:31:09,  8.68s/it][2025-04-25 09:51:23,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 1.21
[2025-04-25 09:51:23,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.71 | bwd_microstep: 5752.41 | bwd_inner_microstep: 5696.80 | bwd_allreduce_microstep: 55.56 | step_microstep: 19.19
[2025-04-25 09:51:23,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.71 | bwd: 5752.42 | bwd_inner: 5696.79 | bwd_allreduce: 55.58 | step: 19.20

  2%|▏         | 783/41250 [1:53:48<97:33:46,  8.68s/it]
                                                        
{'loss': 0.2245, 'grad_norm': 2.056865930557251, 'learning_rate': 2.5298869143780293e-05, 'epoch': 0.19}

  2%|▏         | 783/41250 [1:53:48<97:33:46,  8.68s/it][2025-04-25 09:51:32,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.17 | optimizer_step: 0.90
[2025-04-25 09:51:32,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.22 | bwd_microstep: 5778.02 | bwd_inner_microstep: 5651.94 | bwd_allreduce_microstep: 126.04 | step_microstep: 19.23
[2025-04-25 09:51:32,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.22 | bwd: 5778.03 | bwd_inner: 5651.93 | bwd_allreduce: 126.06 | step: 19.23

  2%|▏         | 784/41250 [1:53:57<97:35:42,  8.68s/it]
                                                        
{'loss': 0.2616, 'grad_norm': 2.5331454277038574, 'learning_rate': 2.533117932148627e-05, 'epoch': 0.19}

  2%|▏         | 784/41250 [1:53:57<97:35:42,  8.68s/it][2025-04-25 09:51:40,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:51:40,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5729.73 | bwd_inner_microstep: 5702.02 | bwd_allreduce_microstep: 27.67 | step_microstep: 18.83
[2025-04-25 09:51:40,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.18 | bwd: 5729.75 | bwd_inner: 5702.02 | bwd_allreduce: 27.68 | step: 18.83

  2%|▏         | 785/41250 [1:54:06<97:32:25,  8.68s/it]
                                                        
{'loss': 0.2786, 'grad_norm': 2.5144143104553223, 'learning_rate': 2.5363489499192246e-05, 'epoch': 0.19}

  2%|▏         | 785/41250 [1:54:06<97:32:25,  8.68s/it][2025-04-25 09:51:49,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 09:51:49,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5729.12 | bwd_inner_microstep: 5704.92 | bwd_allreduce_microstep: 24.15 | step_microstep: 18.79
[2025-04-25 09:51:49,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5729.13 | bwd_inner: 5704.92 | bwd_allreduce: 24.17 | step: 18.79

  2%|▏         | 786/41250 [1:54:14<97:29:48,  8.67s/it]
                                                        
{'loss': 0.1838, 'grad_norm': 2.1696043014526367, 'learning_rate': 2.5395799676898223e-05, 'epoch': 0.19}

  2%|▏         | 786/41250 [1:54:14<97:29:48,  8.67s/it][2025-04-25 09:51:58,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 09:51:58,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.79 | bwd_microstep: 5775.88 | bwd_inner_microstep: 5647.53 | bwd_allreduce_microstep: 128.29 | step_microstep: 18.75
[2025-04-25 09:51:58,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.79 | bwd: 5775.89 | bwd_inner: 5647.53 | bwd_allreduce: 128.31 | step: 18.75

  2%|▏         | 787/41250 [1:54:23<97:31:20,  8.68s/it]
                                                        
{'loss': 0.2614, 'grad_norm': 2.5665860176086426, 'learning_rate': 2.5428109854604205e-05, 'epoch': 0.19}

  2%|▏         | 787/41250 [1:54:23<97:31:20,  8.68s/it][2025-04-25 09:52:07,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 09:52:07,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.72 | bwd_microstep: 5912.08 | bwd_inner_microstep: 5650.88 | bwd_allreduce_microstep: 261.16 | step_microstep: 19.12
[2025-04-25 09:52:07,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.72 | bwd: 5912.09 | bwd_inner: 5650.88 | bwd_allreduce: 261.17 | step: 19.13

  2%|▏         | 788/41250 [1:54:32<97:59:17,  8.72s/it]
                                                        
{'loss': 0.2753, 'grad_norm': 2.5714638233184814, 'learning_rate': 2.546042003231018e-05, 'epoch': 0.19}

  2%|▏         | 788/41250 [1:54:32<97:59:17,  8.72s/it][2025-04-25 09:52:15,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 09:52:15,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.95 | bwd_microstep: 5696.49 | bwd_inner_microstep: 5683.64 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.93
[2025-04-25 09:52:15,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.95 | bwd: 5696.50 | bwd_inner: 5683.64 | bwd_allreduce: 12.82 | step: 18.93

  2%|▏         | 789/41250 [1:54:41<97:39:13,  8.69s/it]
                                                        
{'loss': 0.2969, 'grad_norm': 2.4755945205688477, 'learning_rate': 2.5492730210016157e-05, 'epoch': 0.19}

  2%|▏         | 789/41250 [1:54:41<97:39:13,  8.69s/it][2025-04-25 09:52:24,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 09:52:24,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5739.69 | bwd_inner_microstep: 5701.16 | bwd_allreduce_microstep: 38.48 | step_microstep: 18.64
[2025-04-25 09:52:24,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5739.70 | bwd_inner: 5701.16 | bwd_allreduce: 38.50 | step: 18.64

  2%|▏         | 790/41250 [1:54:49<97:35:53,  8.68s/it]
                                                        
{'loss': 0.2987, 'grad_norm': 4.191558837890625, 'learning_rate': 2.5525040387722135e-05, 'epoch': 0.19}

  2%|▏         | 790/41250 [1:54:49<97:35:53,  8.68s/it][2025-04-25 09:52:33,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 09:52:33,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.01 | bwd_microstep: 5760.97 | bwd_inner_microstep: 5659.45 | bwd_allreduce_microstep: 101.48 | step_microstep: 18.70
[2025-04-25 09:52:33,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.01 | bwd: 5760.98 | bwd_inner: 5659.45 | bwd_allreduce: 101.50 | step: 18.71

  2%|▏         | 791/41250 [1:54:58<97:34:28,  8.68s/it]
                                                        
{'loss': 0.1673, 'grad_norm': 2.7965102195739746, 'learning_rate': 2.5557350565428113e-05, 'epoch': 0.19}

  2%|▏         | 791/41250 [1:54:58<97:34:28,  8.68s/it][2025-04-25 09:52:41,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:52:41,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.77 | bwd_microstep: 5694.29 | bwd_inner_microstep: 5681.53 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.62
[2025-04-25 09:52:41,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.77 | bwd: 5694.30 | bwd_inner: 5681.53 | bwd_allreduce: 12.73 | step: 18.62

  2%|▏         | 792/41250 [1:55:06<97:23:21,  8.67s/it]
                                                        
{'loss': 0.0662, 'grad_norm': 1.318744421005249, 'learning_rate': 2.5589660743134087e-05, 'epoch': 0.19}

  2%|▏         | 792/41250 [1:55:06<97:23:21,  8.67s/it][2025-04-25 09:52:50,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:52:50,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.16 | bwd_microstep: 5735.50 | bwd_inner_microstep: 5658.14 | bwd_allreduce_microstep: 77.31 | step_microstep: 18.45
[2025-04-25 09:52:50,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.16 | bwd: 5735.52 | bwd_inner: 5658.14 | bwd_allreduce: 77.33 | step: 18.46

  2%|▏         | 793/41250 [1:55:15<97:21:26,  8.66s/it]
                                                        
{'loss': 0.2503, 'grad_norm': 4.370097637176514, 'learning_rate': 2.5621970920840065e-05, 'epoch': 0.19}

  2%|▏         | 793/41250 [1:55:15<97:21:26,  8.66s/it][2025-04-25 09:52:58,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 09:52:58,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.98 | bwd_microstep: 5740.56 | bwd_inner_microstep: 5705.15 | bwd_allreduce_microstep: 35.36 | step_microstep: 19.20
[2025-04-25 09:52:58,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.98 | bwd: 5740.57 | bwd_inner: 5705.15 | bwd_allreduce: 35.38 | step: 19.21

  2%|▏         | 794/41250 [1:55:24<97:23:59,  8.67s/it]
                                                        
{'loss': 0.2951, 'grad_norm': 2.982945203781128, 'learning_rate': 2.5654281098546047e-05, 'epoch': 0.19}

  2%|▏         | 794/41250 [1:55:24<97:23:59,  8.67s/it][2025-04-25 09:53:07,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 09:53:07,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.18 | bwd_microstep: 5808.56 | bwd_inner_microstep: 5632.48 | bwd_allreduce_microstep: 176.04 | step_microstep: 18.82
[2025-04-25 09:53:07,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.18 | bwd: 5808.57 | bwd_inner: 5632.48 | bwd_allreduce: 176.06 | step: 18.83

  2%|▏         | 795/41250 [1:55:33<97:35:23,  8.68s/it]
                                                        
{'loss': 0.2228, 'grad_norm': 2.470210552215576, 'learning_rate': 2.568659127625202e-05, 'epoch': 0.19}

  2%|▏         | 795/41250 [1:55:33<97:35:23,  8.68s/it][2025-04-25 09:53:16,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:53:16,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.55 | bwd_microstep: 5705.25 | bwd_inner_microstep: 5656.15 | bwd_allreduce_microstep: 49.06 | step_microstep: 18.74
[2025-04-25 09:53:16,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.55 | bwd: 5705.26 | bwd_inner: 5656.15 | bwd_allreduce: 49.07 | step: 18.74

  2%|▏         | 796/41250 [1:55:41<97:20:51,  8.66s/it]
                                                        
{'loss': 0.202, 'grad_norm': 2.1147592067718506, 'learning_rate': 2.5718901453958e-05, 'epoch': 0.19}

  2%|▏         | 796/41250 [1:55:41<97:20:51,  8.66s/it][2025-04-25 09:53:24,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 09:53:24,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5730.48 | bwd_inner_microstep: 5696.10 | bwd_allreduce_microstep: 34.34 | step_microstep: 18.31
[2025-04-25 09:53:24,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5730.50 | bwd_inner: 5696.10 | bwd_allreduce: 34.36 | step: 18.31

  2%|▏         | 797/41250 [1:55:50<97:19:03,  8.66s/it]
                                                        
{'loss': 0.4071, 'grad_norm': 3.8513710498809814, 'learning_rate': 2.5751211631663977e-05, 'epoch': 0.19}

  2%|▏         | 797/41250 [1:55:50<97:19:03,  8.66s/it][2025-04-25 09:53:33,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 09:53:33,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.78 | bwd_microstep: 5745.83 | bwd_inner_microstep: 5691.96 | bwd_allreduce_microstep: 53.82 | step_microstep: 19.04
[2025-04-25 09:53:33,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.78 | bwd: 5745.84 | bwd_inner: 5691.96 | bwd_allreduce: 53.84 | step: 19.04

  2%|▏         | 798/41250 [1:55:58<97:21:25,  8.66s/it]
                                                        
{'loss': 0.2318, 'grad_norm': 3.037811517715454, 'learning_rate': 2.5783521809369955e-05, 'epoch': 0.19}

  2%|▏         | 798/41250 [1:55:58<97:21:25,  8.66s/it][2025-04-25 09:53:42,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 09:53:42,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.69 | bwd_microstep: 5757.93 | bwd_inner_microstep: 5708.21 | bwd_allreduce_microstep: 49.68 | step_microstep: 18.70
[2025-04-25 09:53:42,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.69 | bwd: 5757.95 | bwd_inner: 5708.20 | bwd_allreduce: 49.70 | step: 18.71

  2%|▏         | 799/41250 [1:56:07<97:27:00,  8.67s/it]
                                                        
{'loss': 0.1954, 'grad_norm': 1.8578085899353027, 'learning_rate': 2.581583198707593e-05, 'epoch': 0.19}

  2%|▏         | 799/41250 [1:56:07<97:27:00,  8.67s/it][2025-04-25 09:53:51,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 09:53:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.20 | bwd_microstep: 5742.78 | bwd_inner_microstep: 5656.98 | bwd_allreduce_microstep: 85.74 | step_microstep: 18.79
[2025-04-25 09:53:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.20 | bwd: 5742.79 | bwd_inner: 5656.98 | bwd_allreduce: 85.76 | step: 18.79

  2%|▏         | 800/41250 [1:56:16<97:23:14,  8.67s/it]
                                                        
{'loss': 0.0766, 'grad_norm': 0.9776834845542908, 'learning_rate': 2.5848142164781907e-05, 'epoch': 0.19}

  2%|▏         | 800/41250 [1:56:16<97:23:14,  8.67s/it][2025-04-25 09:53:59,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.30 | optimizer_step: 1.04
[2025-04-25 09:53:59,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.01 | bwd_microstep: 5730.02 | bwd_inner_microstep: 5679.26 | bwd_allreduce_microstep: 50.70 | step_microstep: 20.07
[2025-04-25 09:53:59,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.01 | bwd: 5730.03 | bwd_inner: 5679.26 | bwd_allreduce: 50.73 | step: 20.07

  2%|▏         | 801/41250 [1:56:24<97:20:20,  8.66s/it]
                                                        
{'loss': 0.1461, 'grad_norm': 1.5232465267181396, 'learning_rate': 2.588045234248789e-05, 'epoch': 0.19}

  2%|▏         | 801/41250 [1:56:24<97:20:20,  8.66s/it][2025-04-25 09:54:08,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:54:08,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.64 | bwd_microstep: 5695.81 | bwd_inner_microstep: 5683.09 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.65
[2025-04-25 09:54:08,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.64 | bwd: 5695.83 | bwd_inner: 5683.09 | bwd_allreduce: 12.70 | step: 18.65

  2%|▏         | 802/41250 [1:56:33<97:12:51,  8.65s/it]
                                                        
{'loss': 0.1655, 'grad_norm': 1.9471324682235718, 'learning_rate': 2.5912762520193863e-05, 'epoch': 0.19}

  2%|▏         | 802/41250 [1:56:33<97:12:51,  8.65s/it][2025-04-25 09:54:16,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 09:54:16,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.43 | bwd_microstep: 5698.60 | bwd_inner_microstep: 5654.23 | bwd_allreduce_microstep: 44.32 | step_microstep: 18.52
[2025-04-25 09:54:16,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.43 | bwd: 5698.61 | bwd_inner: 5654.23 | bwd_allreduce: 44.34 | step: 18.52

  2%|▏         | 803/41250 [1:56:42<97:04:20,  8.64s/it]
                                                        
{'loss': 0.1856, 'grad_norm': 2.755859136581421, 'learning_rate': 2.594507269789984e-05, 'epoch': 0.19}

  2%|▏         | 803/41250 [1:56:42<97:04:20,  8.64s/it][2025-04-25 09:54:25,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:54:25,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.07 | bwd_microstep: 5782.80 | bwd_inner_microstep: 5769.80 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.95
[2025-04-25 09:54:25,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.07 | bwd: 5782.82 | bwd_inner: 5769.80 | bwd_allreduce: 12.98 | step: 18.95

  2%|▏         | 804/41250 [1:56:50<97:27:01,  8.67s/it]
                                                        
{'loss': 0.1359, 'grad_norm': 1.6183750629425049, 'learning_rate': 2.597738287560582e-05, 'epoch': 0.19}

  2%|▏         | 804/41250 [1:56:50<97:27:01,  8.67s/it][2025-04-25 09:54:34,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 09:54:34,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.36 | bwd_microstep: 5711.27 | bwd_inner_microstep: 5693.70 | bwd_allreduce_microstep: 17.52 | step_microstep: 18.83
[2025-04-25 09:54:34,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.36 | bwd: 5711.28 | bwd_inner: 5693.70 | bwd_allreduce: 17.54 | step: 18.84

  2%|▏         | 805/41250 [1:56:59<97:20:08,  8.66s/it]
                                                        
{'loss': 0.2142, 'grad_norm': 2.2474703788757324, 'learning_rate': 2.6009693053311797e-05, 'epoch': 0.2}

  2%|▏         | 805/41250 [1:56:59<97:20:08,  8.66s/it][2025-04-25 09:54:42,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 09:54:42,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.33 | bwd_microstep: 5755.96 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 110.91 | step_microstep: 18.25
[2025-04-25 09:54:42,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.34 | bwd: 5755.97 | bwd_inner: 5645.00 | bwd_allreduce: 110.93 | step: 18.25

  2%|▏         | 806/41250 [1:57:08<97:19:27,  8.66s/it]
                                                        
{'loss': 0.1797, 'grad_norm': 3.1502578258514404, 'learning_rate': 2.604200323101777e-05, 'epoch': 0.2}

  2%|▏         | 806/41250 [1:57:08<97:19:27,  8.66s/it][2025-04-25 09:54:51,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 09:54:51,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.99 | bwd_microstep: 5718.08 | bwd_inner_microstep: 5697.80 | bwd_allreduce_microstep: 20.24 | step_microstep: 18.29
[2025-04-25 09:54:51,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.99 | bwd: 5718.10 | bwd_inner: 5697.80 | bwd_allreduce: 20.26 | step: 18.29

  2%|▏         | 807/41250 [1:57:16<97:18:59,  8.66s/it]
                                                        
{'loss': 0.2425, 'grad_norm': 2.1606361865997314, 'learning_rate': 2.607431340872375e-05, 'epoch': 0.2}

  2%|▏         | 807/41250 [1:57:16<97:18:59,  8.66s/it][2025-04-25 09:55:00,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:55:00,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.22 | bwd_microstep: 5713.07 | bwd_inner_microstep: 5700.04 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.73
[2025-04-25 09:55:00,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.22 | bwd: 5713.08 | bwd_inner: 5700.04 | bwd_allreduce: 13.00 | step: 18.73

  2%|▏         | 808/41250 [1:57:25<97:15:41,  8.66s/it]
                                                        
{'loss': 0.2023, 'grad_norm': 2.025643825531006, 'learning_rate': 2.610662358642973e-05, 'epoch': 0.2}

  2%|▏         | 808/41250 [1:57:25<97:15:41,  8.66s/it][2025-04-25 09:55:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 1.12
[2025-04-25 09:55:08,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5759.30 | bwd_inner_microstep: 5636.04 | bwd_allreduce_microstep: 123.21 | step_microstep: 18.87
[2025-04-25 09:55:08,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5759.31 | bwd_inner: 5636.04 | bwd_allreduce: 123.23 | step: 18.87

  2%|▏         | 809/41250 [1:57:34<97:19:54,  8.66s/it]
                                                        
{'loss': 0.122, 'grad_norm': 1.4195736646652222, 'learning_rate': 2.6138933764135705e-05, 'epoch': 0.2}

  2%|▏         | 809/41250 [1:57:34<97:19:54,  8.66s/it][2025-04-25 09:55:17,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 09:55:17,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.07 | bwd_microstep: 5752.34 | bwd_inner_microstep: 5676.23 | bwd_allreduce_microstep: 76.06 | step_microstep: 18.59
[2025-04-25 09:55:17,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.07 | bwd: 5752.35 | bwd_inner: 5676.23 | bwd_allreduce: 76.08 | step: 18.59

  2%|▏         | 810/41250 [1:57:42<97:22:52,  8.67s/it]
                                                        
{'loss': 0.1521, 'grad_norm': 1.898363709449768, 'learning_rate': 2.6171243941841683e-05, 'epoch': 0.2}

  2%|▏         | 810/41250 [1:57:42<97:22:52,  8.67s/it][2025-04-25 09:55:26,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 09:55:26,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.85 | bwd_microstep: 5673.48 | bwd_inner_microstep: 5641.84 | bwd_allreduce_microstep: 31.58 | step_microstep: 19.42
[2025-04-25 09:55:26,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.85 | bwd: 5673.49 | bwd_inner: 5641.84 | bwd_allreduce: 31.60 | step: 19.42

  2%|▏         | 811/41250 [1:57:51<97:04:49,  8.64s/it]
                                                        
{'loss': 0.2252, 'grad_norm': 2.021947145462036, 'learning_rate': 2.620355411954766e-05, 'epoch': 0.2}

  2%|▏         | 811/41250 [1:57:51<97:04:49,  8.64s/it][2025-04-25 09:55:34,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:55:34,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.41 | bwd_microstep: 5682.44 | bwd_inner_microstep: 5641.76 | bwd_allreduce_microstep: 40.64 | step_microstep: 19.02
[2025-04-25 09:55:34,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.41 | bwd: 5682.45 | bwd_inner: 5641.76 | bwd_allreduce: 40.65 | step: 19.02

  2%|▏         | 812/41250 [1:58:00<96:53:55,  8.63s/it]
                                                        
{'loss': 0.173, 'grad_norm': 1.3806159496307373, 'learning_rate': 2.6235864297253638e-05, 'epoch': 0.2}

  2%|▏         | 812/41250 [1:58:00<96:53:55,  8.63s/it][2025-04-25 09:55:43,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.20
[2025-04-25 09:55:43,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.03 | bwd_microstep: 5701.01 | bwd_inner_microstep: 5688.37 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.50
[2025-04-25 09:55:43,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.04 | bwd: 5701.02 | bwd_inner: 5688.37 | bwd_allreduce: 12.62 | step: 19.51

  2%|▏         | 813/41250 [1:58:08<96:55:49,  8.63s/it]
                                                        
{'loss': 0.2215, 'grad_norm': 1.6714674234390259, 'learning_rate': 2.6268174474959613e-05, 'epoch': 0.2}

  2%|▏         | 813/41250 [1:58:08<96:55:49,  8.63s/it][2025-04-25 09:55:52,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:55:52,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.49 | bwd_microstep: 5764.80 | bwd_inner_microstep: 5623.48 | bwd_allreduce_microstep: 141.28 | step_microstep: 18.94
[2025-04-25 09:55:52,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.49 | bwd: 5764.82 | bwd_inner: 5623.48 | bwd_allreduce: 141.30 | step: 18.94

  2%|▏         | 814/41250 [1:58:17<97:02:17,  8.64s/it]
                                                        
{'loss': 0.3599, 'grad_norm': 3.339108467102051, 'learning_rate': 2.630048465266559e-05, 'epoch': 0.2}

  2%|▏         | 814/41250 [1:58:17<97:02:17,  8.64s/it][2025-04-25 09:56:00,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.16
[2025-04-25 09:56:00,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.28 | bwd_microstep: 5741.11 | bwd_inner_microstep: 5681.68 | bwd_allreduce_microstep: 59.38 | step_microstep: 19.51
[2025-04-25 09:56:00,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.28 | bwd: 5741.13 | bwd_inner: 5681.68 | bwd_allreduce: 59.41 | step: 19.51

  2%|▏         | 815/41250 [1:58:26<97:08:02,  8.65s/it]
                                                        
{'loss': 0.1615, 'grad_norm': 1.2435839176177979, 'learning_rate': 2.6332794830371572e-05, 'epoch': 0.2}

  2%|▏         | 815/41250 [1:58:26<97:08:02,  8.65s/it][2025-04-25 09:56:09,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 09:56:09,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.42 | bwd_microstep: 5696.88 | bwd_inner_microstep: 5684.42 | bwd_allreduce_microstep: 12.42 | step_microstep: 18.83
[2025-04-25 09:56:09,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.42 | bwd: 5696.89 | bwd_inner: 5684.42 | bwd_allreduce: 12.43 | step: 18.83

  2%|▏         | 816/41250 [1:58:34<97:02:57,  8.64s/it]
                                                        
{'loss': 0.1942, 'grad_norm': 3.87872576713562, 'learning_rate': 2.6365105008077546e-05, 'epoch': 0.2}

  2%|▏         | 816/41250 [1:58:34<97:02:57,  8.64s/it][2025-04-25 09:56:18,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:56:18,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.71 | bwd_microstep: 5697.45 | bwd_inner_microstep: 5684.68 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.55
[2025-04-25 09:56:18,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.71 | bwd: 5697.47 | bwd_inner: 5684.68 | bwd_allreduce: 12.74 | step: 18.55

  2%|▏         | 817/41250 [1:58:43<96:58:01,  8.63s/it]
                                                        
{'loss': 0.2723, 'grad_norm': 3.6017684936523438, 'learning_rate': 2.6397415185783524e-05, 'epoch': 0.2}

  2%|▏         | 817/41250 [1:58:43<96:58:01,  8.63s/it][2025-04-25 09:56:26,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 1.17
[2025-04-25 09:56:26,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.76 | bwd_microstep: 5791.78 | bwd_inner_microstep: 5636.79 | bwd_allreduce_microstep: 154.95 | step_microstep: 18.98
[2025-04-25 09:56:26,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.76 | bwd: 5791.80 | bwd_inner: 5636.79 | bwd_allreduce: 154.96 | step: 18.98

  2%|▏         | 818/41250 [1:58:52<97:10:08,  8.65s/it]
                                                        
{'loss': 0.2307, 'grad_norm': 2.066636323928833, 'learning_rate': 2.6429725363489502e-05, 'epoch': 0.2}

  2%|▏         | 818/41250 [1:58:52<97:10:08,  8.65s/it][2025-04-25 09:56:35,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.26 | optimizer_step: 0.94
[2025-04-25 09:56:35,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.32 | bwd_microstep: 5698.92 | bwd_inner_microstep: 5685.63 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.44
[2025-04-25 09:56:35,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.33 | bwd: 5698.93 | bwd_inner: 5685.63 | bwd_allreduce: 13.26 | step: 19.44

  2%|▏         | 819/41250 [1:59:00<97:05:30,  8.65s/it]
                                                        
{'loss': 0.2285, 'grad_norm': 2.804603338241577, 'learning_rate': 2.6462035541195477e-05, 'epoch': 0.2}

  2%|▏         | 819/41250 [1:59:00<97:05:30,  8.65s/it][2025-04-25 09:56:43,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:56:43,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.37 | bwd_microstep: 5682.09 | bwd_inner_microstep: 5654.57 | bwd_allreduce_microstep: 27.47 | step_microstep: 18.73
[2025-04-25 09:56:43,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.37 | bwd: 5682.10 | bwd_inner: 5654.57 | bwd_allreduce: 27.49 | step: 18.74

  2%|▏         | 820/41250 [1:59:09<96:54:08,  8.63s/it]
                                                        
{'loss': 0.0879, 'grad_norm': 1.0450140237808228, 'learning_rate': 2.6494345718901455e-05, 'epoch': 0.2}

  2%|▏         | 820/41250 [1:59:09<96:54:08,  8.63s/it][2025-04-25 09:56:52,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:56:52,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.30 | bwd_microstep: 5707.77 | bwd_inner_microstep: 5694.92 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.95
[2025-04-25 09:56:52,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.30 | bwd: 5707.79 | bwd_inner: 5694.92 | bwd_allreduce: 12.82 | step: 18.96

  2%|▏         | 821/41250 [1:59:17<96:55:57,  8.63s/it]
                                                        
{'loss': 0.3856, 'grad_norm': 2.608020305633545, 'learning_rate': 2.6526655896607432e-05, 'epoch': 0.2}

  2%|▏         | 821/41250 [1:59:17<96:55:57,  8.63s/it][2025-04-25 09:57:01,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 09:57:01,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5750.92 | bwd_inner_microstep: 5647.70 | bwd_allreduce_microstep: 103.18 | step_microstep: 18.61
[2025-04-25 09:57:01,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5750.93 | bwd_inner: 5647.70 | bwd_allreduce: 103.19 | step: 18.61

  2%|▏         | 822/41250 [1:59:26<97:01:49,  8.64s/it]
                                                        
{'loss': 0.2362, 'grad_norm': 2.194720506668091, 'learning_rate': 2.6558966074313414e-05, 'epoch': 0.2}

  2%|▏         | 822/41250 [1:59:26<97:01:49,  8.64s/it][2025-04-25 09:57:09,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:57:09,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.71 | bwd_microstep: 5756.56 | bwd_inner_microstep: 5695.57 | bwd_allreduce_microstep: 60.95 | step_microstep: 18.56
[2025-04-25 09:57:09,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.71 | bwd: 5756.58 | bwd_inner: 5695.57 | bwd_allreduce: 60.97 | step: 18.57

  2%|▏         | 823/41250 [1:59:35<97:13:17,  8.66s/it]
                                                        
{'loss': 0.2532, 'grad_norm': 2.4322266578674316, 'learning_rate': 2.6591276252019388e-05, 'epoch': 0.2}

  2%|▏         | 823/41250 [1:59:35<97:13:17,  8.66s/it][2025-04-25 09:57:18,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 09:57:18,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.98 | bwd_microstep: 5696.18 | bwd_inner_microstep: 5642.88 | bwd_allreduce_microstep: 53.26 | step_microstep: 18.93
[2025-04-25 09:57:18,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.98 | bwd: 5696.20 | bwd_inner: 5642.88 | bwd_allreduce: 53.28 | step: 18.93

  2%|▏         | 824/41250 [1:59:43<97:04:12,  8.64s/it]
                                                        
{'loss': 0.0882, 'grad_norm': 1.448294997215271, 'learning_rate': 2.6623586429725366e-05, 'epoch': 0.2}

  2%|▏         | 824/41250 [1:59:43<97:04:12,  8.64s/it][2025-04-25 09:57:27,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 09:57:27,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.23 | bwd_microstep: 5762.04 | bwd_inner_microstep: 5698.98 | bwd_allreduce_microstep: 63.01 | step_microstep: 18.45
[2025-04-25 09:57:27,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.23 | bwd: 5762.05 | bwd_inner: 5698.98 | bwd_allreduce: 63.03 | step: 18.46

  2%|▏         | 825/41250 [1:59:52<97:16:18,  8.66s/it]
                                                        
{'loss': 0.2367, 'grad_norm': 1.666137933731079, 'learning_rate': 2.6655896607431344e-05, 'epoch': 0.2}

  2%|▏         | 825/41250 [1:59:52<97:16:18,  8.66s/it][2025-04-25 09:57:35,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:57:35,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.96 | bwd_microstep: 5754.30 | bwd_inner_microstep: 5685.81 | bwd_allreduce_microstep: 68.45 | step_microstep: 18.09
[2025-04-25 09:57:35,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.96 | bwd: 5754.32 | bwd_inner: 5685.81 | bwd_allreduce: 68.46 | step: 18.09

  2%|▏         | 826/41250 [2:00:01<97:20:44,  8.67s/it]
                                                        
{'loss': 0.2827, 'grad_norm': 1.5274578332901, 'learning_rate': 2.668820678513732e-05, 'epoch': 0.2}

  2%|▏         | 826/41250 [2:00:01<97:20:44,  8.67s/it][2025-04-25 09:57:44,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 09:57:44,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.02 | bwd_microstep: 5692.17 | bwd_inner_microstep: 5649.32 | bwd_allreduce_microstep: 42.81 | step_microstep: 18.26
[2025-04-25 09:57:44,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.02 | bwd: 5692.19 | bwd_inner: 5649.32 | bwd_allreduce: 42.83 | step: 18.26

  2%|▏         | 827/41250 [2:00:09<97:09:16,  8.65s/it]
                                                        
{'loss': 0.3717, 'grad_norm': 2.970271110534668, 'learning_rate': 2.6720516962843296e-05, 'epoch': 0.2}

  2%|▏         | 827/41250 [2:00:09<97:09:16,  8.65s/it][2025-04-25 09:57:53,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 09:57:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.70 | bwd_microstep: 5715.35 | bwd_inner_microstep: 5702.43 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.04
[2025-04-25 09:57:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.70 | bwd: 5715.36 | bwd_inner: 5702.43 | bwd_allreduce: 12.89 | step: 19.04

  2%|▏         | 828/41250 [2:00:18<97:10:43,  8.65s/it]
                                                        
{'loss': 0.1577, 'grad_norm': 1.2691800594329834, 'learning_rate': 2.6752827140549274e-05, 'epoch': 0.2}

  2%|▏         | 828/41250 [2:00:18<97:10:43,  8.65s/it][2025-04-25 09:58:01,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 09:58:01,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.82 | bwd_microstep: 5690.96 | bwd_inner_microstep: 5659.38 | bwd_allreduce_microstep: 31.53 | step_microstep: 18.97
[2025-04-25 09:58:01,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.82 | bwd: 5690.97 | bwd_inner: 5659.38 | bwd_allreduce: 31.55 | step: 18.97

  2%|▏         | 829/41250 [2:00:27<97:02:59,  8.64s/it]
                                                        
{'loss': 0.3617, 'grad_norm': 2.831509828567505, 'learning_rate': 2.6785137318255256e-05, 'epoch': 0.2}

  2%|▏         | 829/41250 [2:00:27<97:02:59,  8.64s/it][2025-04-25 09:58:10,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 09:58:10,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.43 | bwd_microstep: 5761.78 | bwd_inner_microstep: 5693.94 | bwd_allreduce_microstep: 67.80 | step_microstep: 18.39
[2025-04-25 09:58:10,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.43 | bwd: 5761.79 | bwd_inner: 5693.94 | bwd_allreduce: 67.81 | step: 18.40

  2%|▏         | 830/41250 [2:00:35<97:15:43,  8.66s/it]
                                                        
{'loss': 0.2687, 'grad_norm': 1.8534153699874878, 'learning_rate': 2.681744749596123e-05, 'epoch': 0.2}

  2%|▏         | 830/41250 [2:00:35<97:15:43,  8.66s/it][2025-04-25 09:58:19,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:58:19,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.47 | bwd_microstep: 5782.35 | bwd_inner_microstep: 5707.07 | bwd_allreduce_microstep: 75.24 | step_microstep: 18.52
[2025-04-25 09:58:19,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.47 | bwd: 5782.36 | bwd_inner: 5707.07 | bwd_allreduce: 75.25 | step: 18.53

  2%|▏         | 831/41250 [2:00:44<97:27:11,  8.68s/it]
                                                        
{'loss': 0.1407, 'grad_norm': 1.1870375871658325, 'learning_rate': 2.6849757673667208e-05, 'epoch': 0.2}

  2%|▏         | 831/41250 [2:00:44<97:27:11,  8.68s/it][2025-04-25 09:58:27,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:58:27,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.66 | bwd_microstep: 5729.46 | bwd_inner_microstep: 5716.21 | bwd_allreduce_microstep: 13.21 | step_microstep: 18.62
[2025-04-25 09:58:27,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.66 | bwd: 5729.47 | bwd_inner: 5716.21 | bwd_allreduce: 13.22 | step: 18.62

  2%|▏         | 832/41250 [2:00:53<97:27:01,  8.68s/it]
                                                        
{'loss': 0.3745, 'grad_norm': 2.816781759262085, 'learning_rate': 2.6882067851373186e-05, 'epoch': 0.2}

  2%|▏         | 832/41250 [2:00:53<97:27:01,  8.68s/it][2025-04-25 09:58:36,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 09:58:36,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.92 | bwd_microstep: 5770.82 | bwd_inner_microstep: 5649.94 | bwd_allreduce_microstep: 120.84 | step_microstep: 18.34
[2025-04-25 09:58:36,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.92 | bwd: 5770.83 | bwd_inner: 5649.94 | bwd_allreduce: 120.85 | step: 18.34

  2%|▏         | 833/41250 [2:01:01<97:28:53,  8.68s/it]
                                                        
{'loss': 0.1742, 'grad_norm': 2.24346923828125, 'learning_rate': 2.691437802907916e-05, 'epoch': 0.2}

  2%|▏         | 833/41250 [2:01:01<97:28:53,  8.68s/it][2025-04-25 09:58:45,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:58:45,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.32 | bwd_microstep: 5788.34 | bwd_inner_microstep: 5671.57 | bwd_allreduce_microstep: 116.73 | step_microstep: 18.40
[2025-04-25 09:58:45,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.32 | bwd: 5788.35 | bwd_inner: 5671.57 | bwd_allreduce: 116.74 | step: 18.41

  2%|▏         | 834/41250 [2:01:10<97:34:03,  8.69s/it]
                                                        
{'loss': 0.1297, 'grad_norm': 1.336548924446106, 'learning_rate': 2.6946688206785138e-05, 'epoch': 0.2}

  2%|▏         | 834/41250 [2:01:10<97:34:03,  8.69s/it][2025-04-25 09:58:53,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 09:58:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.87 | bwd_microstep: 5720.11 | bwd_inner_microstep: 5707.05 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.64
[2025-04-25 09:58:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.87 | bwd: 5720.13 | bwd_inner: 5707.05 | bwd_allreduce: 13.03 | step: 18.64

  2%|▏         | 835/41250 [2:01:19<97:27:32,  8.68s/it]
                                                        
{'loss': 0.1214, 'grad_norm': 1.2121790647506714, 'learning_rate': 2.6978998384491116e-05, 'epoch': 0.2}

  2%|▏         | 835/41250 [2:01:19<97:27:32,  8.68s/it][2025-04-25 09:59:02,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 09:59:02,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.51 | bwd_microstep: 5698.19 | bwd_inner_microstep: 5657.86 | bwd_allreduce_microstep: 40.28 | step_microstep: 18.89
[2025-04-25 09:59:02,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.51 | bwd: 5698.20 | bwd_inner: 5657.86 | bwd_allreduce: 40.30 | step: 18.90

  2%|▏         | 836/41250 [2:01:27<97:16:25,  8.66s/it]
                                                        
{'loss': 0.3887, 'grad_norm': 2.202378511428833, 'learning_rate': 2.7011308562197097e-05, 'epoch': 0.2}

  2%|▏         | 836/41250 [2:01:27<97:16:25,  8.66s/it][2025-04-25 09:59:11,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 09:59:11,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.84 | bwd_microstep: 5808.42 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 150.73 | step_microstep: 18.06
[2025-04-25 09:59:11,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.85 | bwd: 5808.43 | bwd_inner: 5657.65 | bwd_allreduce: 150.74 | step: 18.07

  2%|▏         | 837/41250 [2:01:36<97:28:11,  8.68s/it]
                                                        
{'loss': 0.1748, 'grad_norm': 1.7749004364013672, 'learning_rate': 2.7043618739903072e-05, 'epoch': 0.2}

  2%|▏         | 837/41250 [2:01:36<97:28:11,  8.68s/it][2025-04-25 09:59:19,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 09:59:19,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.91 | bwd_microstep: 5716.75 | bwd_inner_microstep: 5703.87 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.07
[2025-04-25 09:59:19,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.92 | bwd: 5716.77 | bwd_inner: 5703.87 | bwd_allreduce: 12.85 | step: 19.07

  2%|▏         | 838/41250 [2:01:45<97:22:25,  8.67s/it]
                                                        
{'loss': 0.1929, 'grad_norm': 1.5533359050750732, 'learning_rate': 2.707592891760905e-05, 'epoch': 0.2}

  2%|▏         | 838/41250 [2:01:45<97:22:25,  8.67s/it][2025-04-25 09:59:28,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 09:59:28,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.70 | bwd_microstep: 6065.77 | bwd_inner_microstep: 5671.22 | bwd_allreduce_microstep: 394.50 | step_microstep: 18.94
[2025-04-25 09:59:28,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.70 | bwd: 6065.78 | bwd_inner: 5671.22 | bwd_allreduce: 394.52 | step: 18.94

  2%|▏         | 839/41250 [2:01:54<98:24:12,  8.77s/it]
                                                        
{'loss': 0.087, 'grad_norm': 0.8583626747131348, 'learning_rate': 2.7108239095315028e-05, 'epoch': 0.2}

  2%|▏         | 839/41250 [2:01:54<98:24:12,  8.77s/it][2025-04-25 09:59:37,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 09:59:37,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.53 | bwd_microstep: 5794.78 | bwd_inner_microstep: 5782.01 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.88
[2025-04-25 09:59:37,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.53 | bwd: 5794.80 | bwd_inner: 5782.01 | bwd_allreduce: 12.75 | step: 18.88

  2%|▏         | 840/41250 [2:02:03<98:25:02,  8.77s/it]
                                                        
{'loss': 0.1302, 'grad_norm': 1.3211696147918701, 'learning_rate': 2.7140549273021002e-05, 'epoch': 0.2}

  2%|▏         | 840/41250 [2:02:03<98:25:02,  8.77s/it][2025-04-25 09:59:46,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.10 | optimizer_step: 0.93
[2025-04-25 09:59:46,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.93 | bwd_microstep: 5712.96 | bwd_inner_microstep: 5667.61 | bwd_allreduce_microstep: 45.31 | step_microstep: 18.82
[2025-04-25 09:59:46,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.93 | bwd: 5712.98 | bwd_inner: 5667.61 | bwd_allreduce: 45.33 | step: 18.82

  2%|▏         | 841/41250 [2:02:11<97:56:59,  8.73s/it]
                                                        
{'loss': 0.2746, 'grad_norm': 3.8059585094451904, 'learning_rate': 2.717285945072698e-05, 'epoch': 0.2}

  2%|▏         | 841/41250 [2:02:11<97:56:59,  8.73s/it][2025-04-25 09:59:55,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 09:59:55,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5715.85 | bwd_inner_microstep: 5703.11 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.78
[2025-04-25 09:59:55,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5715.86 | bwd_inner: 5703.11 | bwd_allreduce: 12.71 | step: 18.78

  2%|▏         | 842/41250 [2:02:20<97:40:46,  8.70s/it]
                                                        
{'loss': 0.2926, 'grad_norm': 2.0382132530212402, 'learning_rate': 2.7205169628432958e-05, 'epoch': 0.2}

  2%|▏         | 842/41250 [2:02:20<97:40:46,  8.70s/it][2025-04-25 10:00:03,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 1.19
[2025-04-25 10:00:03,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.00 | bwd_microstep: 5713.40 | bwd_inner_microstep: 5700.73 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.00
[2025-04-25 10:00:03,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.00 | bwd: 5713.41 | bwd_inner: 5700.73 | bwd_allreduce: 12.64 | step: 19.00

  2%|▏         | 843/41250 [2:02:28<97:30:26,  8.69s/it]
                                                        
{'loss': 0.3077, 'grad_norm': 2.3278138637542725, 'learning_rate': 2.723747980613894e-05, 'epoch': 0.2}

  2%|▏         | 843/41250 [2:02:28<97:30:26,  8.69s/it][2025-04-25 10:00:12,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 10:00:12,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5702.97 | bwd_inner_microstep: 5690.39 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.17
[2025-04-25 10:00:12,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.98 | bwd: 5702.99 | bwd_inner: 5690.39 | bwd_allreduce: 12.56 | step: 18.18

  2%|▏         | 844/41250 [2:02:37<97:21:06,  8.67s/it]
                                                        
{'loss': 0.2908, 'grad_norm': 1.4217004776000977, 'learning_rate': 2.7269789983844914e-05, 'epoch': 0.2}

  2%|▏         | 844/41250 [2:02:37<97:21:06,  8.67s/it][2025-04-25 10:00:20,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 10:00:20,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.14 | bwd_microstep: 5698.89 | bwd_inner_microstep: 5686.31 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.42
[2025-04-25 10:00:20,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.14 | bwd: 5698.91 | bwd_inner: 5686.31 | bwd_allreduce: 12.56 | step: 18.42

  2%|▏         | 845/41250 [2:02:46<97:15:07,  8.66s/it]
                                                        
{'loss': 0.231, 'grad_norm': 5.30738639831543, 'learning_rate': 2.730210016155089e-05, 'epoch': 0.2}

  2%|▏         | 845/41250 [2:02:46<97:15:07,  8.66s/it][2025-04-25 10:00:29,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.18
[2025-04-25 10:00:29,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.35 | bwd_microstep: 5773.45 | bwd_inner_microstep: 5721.21 | bwd_allreduce_microstep: 52.19 | step_microstep: 19.34
[2025-04-25 10:00:29,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.35 | bwd: 5773.47 | bwd_inner: 5721.21 | bwd_allreduce: 52.21 | step: 19.36

  2%|▏         | 846/41250 [2:02:54<97:25:56,  8.68s/it]
                                                        
{'loss': 0.2592, 'grad_norm': 2.5888314247131348, 'learning_rate': 2.733441033925687e-05, 'epoch': 0.21}

  2%|▏         | 846/41250 [2:02:54<97:25:56,  8.68s/it][2025-04-25 10:00:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:00:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.63 | bwd_microstep: 5780.21 | bwd_inner_microstep: 5644.72 | bwd_allreduce_microstep: 135.44 | step_microstep: 18.43
[2025-04-25 10:00:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.63 | bwd: 5780.23 | bwd_inner: 5644.72 | bwd_allreduce: 135.46 | step: 18.43

  2%|▏         | 847/41250 [2:03:03<97:27:05,  8.68s/it]
                                                        
{'loss': 0.3754, 'grad_norm': 2.601055145263672, 'learning_rate': 2.7366720516962844e-05, 'epoch': 0.21}

  2%|▏         | 847/41250 [2:03:03<97:27:05,  8.68s/it][2025-04-25 10:00:47,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:00:47,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.52 | bwd_microstep: 5775.86 | bwd_inner_microstep: 5654.15 | bwd_allreduce_microstep: 121.67 | step_microstep: 18.56
[2025-04-25 10:00:47,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.52 | bwd: 5775.88 | bwd_inner: 5654.15 | bwd_allreduce: 121.68 | step: 18.56

  2%|▏         | 848/41250 [2:03:12<97:28:08,  8.68s/it]
                                                        
{'loss': 0.1174, 'grad_norm': 0.9527285695075989, 'learning_rate': 2.7399030694668822e-05, 'epoch': 0.21}

  2%|▏         | 848/41250 [2:03:12<97:28:08,  8.68s/it][2025-04-25 10:00:55,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:00:55,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.57 | bwd_microstep: 5737.31 | bwd_inner_microstep: 5714.80 | bwd_allreduce_microstep: 22.47 | step_microstep: 18.46
[2025-04-25 10:00:55,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.57 | bwd: 5737.33 | bwd_inner: 5714.80 | bwd_allreduce: 22.49 | step: 18.47

  2%|▏         | 849/41250 [2:03:21<97:28:30,  8.69s/it]
                                                        
{'loss': 0.1622, 'grad_norm': 2.101231098175049, 'learning_rate': 2.74313408723748e-05, 'epoch': 0.21}

  2%|▏         | 849/41250 [2:03:21<97:28:30,  8.69s/it][2025-04-25 10:01:04,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.16
[2025-04-25 10:01:04,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.88 | bwd_microstep: 5700.32 | bwd_inner_microstep: 5660.94 | bwd_allreduce_microstep: 39.33 | step_microstep: 19.26
[2025-04-25 10:01:04,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.88 | bwd: 5700.33 | bwd_inner: 5660.94 | bwd_allreduce: 39.35 | step: 19.27

  2%|▏         | 850/41250 [2:03:29<97:15:10,  8.67s/it]
                                                        
{'loss': 0.2071, 'grad_norm': 2.4356343746185303, 'learning_rate': 2.746365105008078e-05, 'epoch': 0.21}

  2%|▏         | 850/41250 [2:03:29<97:15:10,  8.67s/it][2025-04-25 10:01:12,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:01:12,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.32 | bwd_microstep: 5706.13 | bwd_inner_microstep: 5658.93 | bwd_allreduce_microstep: 47.16 | step_microstep: 18.05
[2025-04-25 10:01:12,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.32 | bwd: 5706.15 | bwd_inner: 5658.93 | bwd_allreduce: 47.17 | step: 18.06

  2%|▏         | 851/41250 [2:03:38<97:05:50,  8.65s/it]
                                                        
{'loss': 0.4093, 'grad_norm': 2.7936158180236816, 'learning_rate': 2.7495961227786755e-05, 'epoch': 0.21}

  2%|▏         | 851/41250 [2:03:38<97:05:50,  8.65s/it][2025-04-25 10:01:21,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:01:21,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5765.90 | bwd_inner_microstep: 5641.57 | bwd_allreduce_microstep: 124.28 | step_microstep: 18.45
[2025-04-25 10:01:21,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5765.91 | bwd_inner: 5641.57 | bwd_allreduce: 124.30 | step: 18.45

  2%|▏         | 852/41250 [2:03:46<97:09:55,  8.66s/it]
                                                        
{'loss': 0.2086, 'grad_norm': 2.0488693714141846, 'learning_rate': 2.7528271405492733e-05, 'epoch': 0.21}

  2%|▏         | 852/41250 [2:03:46<97:09:55,  8.66s/it][2025-04-25 10:01:30,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 10:01:30,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.18 | bwd_microstep: 5699.35 | bwd_inner_microstep: 5685.78 | bwd_allreduce_microstep: 13.52 | step_microstep: 18.63
[2025-04-25 10:01:30,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.18 | bwd: 5699.36 | bwd_inner: 5685.78 | bwd_allreduce: 13.54 | step: 18.63

  2%|▏         | 853/41250 [2:03:55<97:07:27,  8.66s/it]
                                                        
{'loss': 0.2173, 'grad_norm': 1.827099323272705, 'learning_rate': 2.756058158319871e-05, 'epoch': 0.21}

  2%|▏         | 853/41250 [2:03:55<97:07:27,  8.66s/it][2025-04-25 10:01:38,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 10:01:38,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.98 | bwd_microstep: 5697.00 | bwd_inner_microstep: 5684.05 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.63
[2025-04-25 10:01:38,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.98 | bwd: 5697.01 | bwd_inner: 5684.05 | bwd_allreduce: 12.92 | step: 18.64

  2%|▏         | 854/41250 [2:04:04<97:02:04,  8.65s/it]
                                                        
{'loss': 0.3275, 'grad_norm': 2.928266763687134, 'learning_rate': 2.7592891760904686e-05, 'epoch': 0.21}

  2%|▏         | 854/41250 [2:04:04<97:02:04,  8.65s/it][2025-04-25 10:01:47,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 10:01:47,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5710.09 | bwd_inner_microstep: 5697.36 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.73
[2025-04-25 10:01:47,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5710.11 | bwd_inner: 5697.36 | bwd_allreduce: 12.70 | step: 18.73

  2%|▏         | 855/41250 [2:04:12<97:01:27,  8.65s/it]
                                                        
{'loss': 0.1633, 'grad_norm': 1.260807991027832, 'learning_rate': 2.7625201938610664e-05, 'epoch': 0.21}

  2%|▏         | 855/41250 [2:04:12<97:01:27,  8.65s/it][2025-04-25 10:01:56,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:01:56,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.09 | bwd_microstep: 5678.69 | bwd_inner_microstep: 5644.54 | bwd_allreduce_microstep: 34.11 | step_microstep: 18.39
[2025-04-25 10:01:56,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.09 | bwd: 5678.71 | bwd_inner: 5644.54 | bwd_allreduce: 34.12 | step: 18.39

  2%|▏         | 856/41250 [2:04:21<96:51:12,  8.63s/it]
                                                        
{'loss': 0.1064, 'grad_norm': 0.9857742786407471, 'learning_rate': 2.765751211631664e-05, 'epoch': 0.21}

  2%|▏         | 856/41250 [2:04:21<96:51:12,  8.63s/it][2025-04-25 10:02:04,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:02:04,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.51 | bwd_microstep: 5726.30 | bwd_inner_microstep: 5695.90 | bwd_allreduce_microstep: 30.35 | step_microstep: 18.68
[2025-04-25 10:02:04,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.51 | bwd: 5726.32 | bwd_inner: 5695.90 | bwd_allreduce: 30.37 | step: 18.68

  2%|▏         | 857/41250 [2:04:30<96:56:52,  8.64s/it]
                                                        
{'loss': 0.2746, 'grad_norm': 2.0062928199768066, 'learning_rate': 2.7689822294022623e-05, 'epoch': 0.21}

  2%|▏         | 857/41250 [2:04:30<96:56:52,  8.64s/it][2025-04-25 10:02:13,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 10:02:13,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5755.97 | bwd_inner_microstep: 5657.13 | bwd_allreduce_microstep: 98.78 | step_microstep: 19.40
[2025-04-25 10:02:13,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5755.98 | bwd_inner: 5657.13 | bwd_allreduce: 98.80 | step: 19.40

  2%|▏         | 858/41250 [2:04:38<97:02:28,  8.65s/it]
                                                        
{'loss': 0.2277, 'grad_norm': 1.83975088596344, 'learning_rate': 2.7722132471728597e-05, 'epoch': 0.21}

  2%|▏         | 858/41250 [2:04:38<97:02:28,  8.65s/it][2025-04-25 10:02:22,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:02:22,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5854.58 | bwd_inner_microstep: 5641.26 | bwd_allreduce_microstep: 213.28 | step_microstep: 18.71
[2025-04-25 10:02:22,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5854.60 | bwd_inner: 5641.26 | bwd_allreduce: 213.30 | step: 18.71

  2%|▏         | 859/41250 [2:04:47<97:26:04,  8.68s/it]
                                                        
{'loss': 0.1715, 'grad_norm': 2.059441566467285, 'learning_rate': 2.7754442649434575e-05, 'epoch': 0.21}

  2%|▏         | 859/41250 [2:04:47<97:26:04,  8.68s/it][2025-04-25 10:02:30,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 10:02:30,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.95 | bwd_microstep: 5693.67 | bwd_inner_microstep: 5680.92 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.83
[2025-04-25 10:02:30,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.95 | bwd: 5693.69 | bwd_inner: 5680.92 | bwd_allreduce: 12.73 | step: 18.83

  2%|▏         | 860/41250 [2:04:56<97:11:43,  8.66s/it]
                                                        
{'loss': 0.1189, 'grad_norm': 1.2096221446990967, 'learning_rate': 2.7786752827140553e-05, 'epoch': 0.21}

  2%|▏         | 860/41250 [2:04:56<97:11:43,  8.66s/it][2025-04-25 10:02:39,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:02:39,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.50 | bwd_microstep: 5692.59 | bwd_inner_microstep: 5679.72 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.08
[2025-04-25 10:02:39,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.50 | bwd: 5692.61 | bwd_inner: 5679.72 | bwd_allreduce: 12.84 | step: 19.08

  2%|▏         | 861/41250 [2:05:04<97:02:09,  8.65s/it]
                                                        
{'loss': 0.1482, 'grad_norm': 1.2033683061599731, 'learning_rate': 2.7819063004846527e-05, 'epoch': 0.21}

  2%|▏         | 861/41250 [2:05:04<97:02:09,  8.65s/it][2025-04-25 10:02:48,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.13
[2025-04-25 10:02:48,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.02 | bwd_microstep: 5714.17 | bwd_inner_microstep: 5696.22 | bwd_allreduce_microstep: 17.91 | step_microstep: 19.28
[2025-04-25 10:02:48,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.02 | bwd: 5714.18 | bwd_inner: 5696.22 | bwd_allreduce: 17.92 | step: 19.29

  2%|▏         | 862/41250 [2:05:13<97:01:35,  8.65s/it]
                                                        
{'loss': 0.3339, 'grad_norm': 3.7129757404327393, 'learning_rate': 2.7851373182552505e-05, 'epoch': 0.21}

  2%|▏         | 862/41250 [2:05:13<97:01:35,  8.65s/it][2025-04-25 10:02:56,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 10:02:56,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.58 | bwd_microstep: 5747.57 | bwd_inner_microstep: 5651.84 | bwd_allreduce_microstep: 95.68 | step_microstep: 19.52
[2025-04-25 10:02:56,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.58 | bwd: 5747.59 | bwd_inner: 5651.84 | bwd_allreduce: 95.71 | step: 19.52

  2%|▏         | 863/41250 [2:05:22<97:04:58,  8.65s/it]
                                                        
{'loss': 0.2095, 'grad_norm': 2.512493371963501, 'learning_rate': 2.7883683360258483e-05, 'epoch': 0.21}

  2%|▏         | 863/41250 [2:05:22<97:04:58,  8.65s/it][2025-04-25 10:03:05,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 10:03:05,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.49 | bwd_microstep: 5782.23 | bwd_inner_microstep: 5769.26 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.44
[2025-04-25 10:03:05,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.49 | bwd: 5782.25 | bwd_inner: 5769.26 | bwd_allreduce: 12.94 | step: 19.44

  2%|▏         | 864/41250 [2:05:30<97:25:18,  8.68s/it]
                                                        
{'loss': 0.3227, 'grad_norm': 2.457449436187744, 'learning_rate': 2.7915993537964465e-05, 'epoch': 0.21}

  2%|▏         | 864/41250 [2:05:30<97:25:18,  8.68s/it][2025-04-25 10:03:14,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:03:14,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.92 | bwd_microstep: 5731.33 | bwd_inner_microstep: 5686.17 | bwd_allreduce_microstep: 45.12 | step_microstep: 18.55
[2025-04-25 10:03:14,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.92 | bwd: 5731.34 | bwd_inner: 5686.17 | bwd_allreduce: 45.14 | step: 18.55

  2%|▏         | 865/41250 [2:05:39<97:25:04,  8.68s/it]
                                                        
{'loss': 0.2272, 'grad_norm': 1.865816354751587, 'learning_rate': 2.7948303715670436e-05, 'epoch': 0.21}

  2%|▏         | 865/41250 [2:05:39<97:25:04,  8.68s/it][2025-04-25 10:03:22,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 10:03:22,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.90 | bwd_microstep: 5766.37 | bwd_inner_microstep: 5637.38 | bwd_allreduce_microstep: 128.94 | step_microstep: 18.96
[2025-04-25 10:03:22,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.90 | bwd: 5766.38 | bwd_inner: 5637.38 | bwd_allreduce: 128.97 | step: 18.96

  2%|▏         | 866/41250 [2:05:48<97:22:04,  8.68s/it]
                                                        
{'loss': 0.1966, 'grad_norm': 1.5682495832443237, 'learning_rate': 2.7980613893376417e-05, 'epoch': 0.21}

  2%|▏         | 866/41250 [2:05:48<97:22:04,  8.68s/it][2025-04-25 10:03:31,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:03:31,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.99 | bwd_microstep: 5721.34 | bwd_inner_microstep: 5702.89 | bwd_allreduce_microstep: 18.41 | step_microstep: 18.66
[2025-04-25 10:03:31,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.99 | bwd: 5721.35 | bwd_inner: 5702.89 | bwd_allreduce: 18.42 | step: 18.67

  2%|▏         | 867/41250 [2:05:56<97:19:27,  8.68s/it]
                                                        
{'loss': 0.1189, 'grad_norm': 1.301019549369812, 'learning_rate': 2.8012924071082395e-05, 'epoch': 0.21}

  2%|▏         | 867/41250 [2:05:56<97:19:27,  8.68s/it][2025-04-25 10:03:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 10:03:40,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5769.47 | bwd_inner_microstep: 5643.80 | bwd_allreduce_microstep: 125.62 | step_microstep: 18.59
[2025-04-25 10:03:40,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.99 | bwd: 5769.49 | bwd_inner: 5643.80 | bwd_allreduce: 125.64 | step: 18.59

  2%|▏         | 868/41250 [2:06:05<97:20:42,  8.68s/it]
                                                        
{'loss': 0.198, 'grad_norm': 2.042497158050537, 'learning_rate': 2.804523424878837e-05, 'epoch': 0.21}

  2%|▏         | 868/41250 [2:06:05<97:20:42,  8.68s/it][2025-04-25 10:03:48,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 10:03:48,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.48 | bwd_microstep: 5715.95 | bwd_inner_microstep: 5703.32 | bwd_allreduce_microstep: 12.58 | step_microstep: 19.38
[2025-04-25 10:03:48,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.48 | bwd: 5715.96 | bwd_inner: 5703.32 | bwd_allreduce: 12.60 | step: 19.38

  2%|▏         | 869/41250 [2:06:14<97:18:50,  8.68s/it]
                                                        
{'loss': 0.1291, 'grad_norm': 1.336362361907959, 'learning_rate': 2.8077544426494347e-05, 'epoch': 0.21}

  2%|▏         | 869/41250 [2:06:14<97:18:50,  8.68s/it][2025-04-25 10:03:57,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-25 10:03:57,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.68 | bwd_microstep: 5770.75 | bwd_inner_microstep: 5635.84 | bwd_allreduce_microstep: 134.86 | step_microstep: 19.00
[2025-04-25 10:03:57,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.68 | bwd: 5770.76 | bwd_inner: 5635.84 | bwd_allreduce: 134.88 | step: 19.00

  2%|▏         | 870/41250 [2:06:22<97:17:56,  8.67s/it]
                                                        
{'loss': 0.0644, 'grad_norm': 0.8135383129119873, 'learning_rate': 2.8109854604200325e-05, 'epoch': 0.21}

  2%|▏         | 870/41250 [2:06:22<97:17:56,  8.67s/it][2025-04-25 10:04:06,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:04:06,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.35 | bwd_microstep: 5744.91 | bwd_inner_microstep: 5691.67 | bwd_allreduce_microstep: 53.19 | step_microstep: 18.62
[2025-04-25 10:04:06,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.35 | bwd: 5744.92 | bwd_inner: 5691.67 | bwd_allreduce: 53.21 | step: 18.63

  2%|▏         | 871/41250 [2:06:31<97:18:37,  8.68s/it]
                                                        
{'loss': 0.0448, 'grad_norm': 0.46248674392700195, 'learning_rate': 2.8142164781906306e-05, 'epoch': 0.21}

  2%|▏         | 871/41250 [2:06:31<97:18:37,  8.68s/it][2025-04-25 10:04:14,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:04:14,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.85 | bwd_microstep: 5773.30 | bwd_inner_microstep: 5650.83 | bwd_allreduce_microstep: 122.40 | step_microstep: 18.77
[2025-04-25 10:04:14,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.85 | bwd: 5773.31 | bwd_inner: 5650.83 | bwd_allreduce: 122.43 | step: 18.77

  2%|▏         | 872/41250 [2:06:40<97:20:24,  8.68s/it]
                                                        
{'loss': 0.2043, 'grad_norm': 2.2761902809143066, 'learning_rate': 2.8174474959612277e-05, 'epoch': 0.21}

  2%|▏         | 872/41250 [2:06:40<97:20:24,  8.68s/it][2025-04-25 10:04:23,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-25 10:04:23,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.11 | bwd_microstep: 5736.59 | bwd_inner_microstep: 5684.29 | bwd_allreduce_microstep: 52.25 | step_microstep: 19.12
[2025-04-25 10:04:23,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.11 | bwd: 5736.60 | bwd_inner: 5684.29 | bwd_allreduce: 52.28 | step: 19.12

  2%|▏         | 873/41250 [2:06:48<97:17:43,  8.67s/it]
                                                        
{'loss': 0.3205, 'grad_norm': 2.7930569648742676, 'learning_rate': 2.820678513731826e-05, 'epoch': 0.21}

  2%|▏         | 873/41250 [2:06:48<97:17:43,  8.67s/it][2025-04-25 10:04:32,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.10 | optimizer_step: 1.04
[2025-04-25 10:04:32,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.10 | bwd_microstep: 5903.79 | bwd_inner_microstep: 5649.86 | bwd_allreduce_microstep: 253.88 | step_microstep: 19.26
[2025-04-25 10:04:32,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.10 | bwd: 5903.80 | bwd_inner: 5649.86 | bwd_allreduce: 253.90 | step: 19.27

  2%|▏         | 874/41250 [2:06:57<97:47:17,  8.72s/it]
                                                        
{'loss': 0.0697, 'grad_norm': 0.9233564734458923, 'learning_rate': 2.8239095315024237e-05, 'epoch': 0.21}

  2%|▏         | 874/41250 [2:06:57<97:47:17,  8.72s/it][2025-04-25 10:04:41,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:04:41,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.67 | bwd_microstep: 5707.39 | bwd_inner_microstep: 5678.73 | bwd_allreduce_microstep: 28.62 | step_microstep: 18.41
[2025-04-25 10:04:41,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.67 | bwd: 5707.40 | bwd_inner: 5678.73 | bwd_allreduce: 28.63 | step: 18.42

  2%|▏         | 875/41250 [2:07:06<97:31:03,  8.70s/it]
                                                        
{'loss': 0.2082, 'grad_norm': 2.6988751888275146, 'learning_rate': 2.827140549273021e-05, 'epoch': 0.21}

  2%|▏         | 875/41250 [2:07:06<97:31:03,  8.70s/it][2025-04-25 10:04:49,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:04:49,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.34 | bwd_microstep: 5709.06 | bwd_inner_microstep: 5663.77 | bwd_allreduce_microstep: 45.24 | step_microstep: 18.65
[2025-04-25 10:04:49,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.34 | bwd: 5709.07 | bwd_inner: 5663.77 | bwd_allreduce: 45.26 | step: 18.65

  2%|▏         | 876/41250 [2:07:15<97:18:58,  8.68s/it]
                                                        
{'loss': 0.1101, 'grad_norm': 1.4866151809692383, 'learning_rate': 2.830371567043619e-05, 'epoch': 0.21}

  2%|▏         | 876/41250 [2:07:15<97:18:58,  8.68s/it][2025-04-25 10:04:58,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:04:58,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.96 | bwd_microstep: 5719.51 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 28.62 | step_microstep: 18.48
[2025-04-25 10:04:58,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.96 | bwd: 5719.52 | bwd_inner: 5690.85 | bwd_allreduce: 28.64 | step: 18.48

  2%|▏         | 877/41250 [2:07:23<97:13:15,  8.67s/it]
                                                        
{'loss': 0.2017, 'grad_norm': 2.7597758769989014, 'learning_rate': 2.8336025848142167e-05, 'epoch': 0.21}

  2%|▏         | 877/41250 [2:07:23<97:13:15,  8.67s/it][2025-04-25 10:05:07,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 10:05:07,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5719.35 | bwd_inner_microstep: 5687.07 | bwd_allreduce_microstep: 32.25 | step_microstep: 19.15
[2025-04-25 10:05:07,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5719.37 | bwd_inner: 5687.07 | bwd_allreduce: 32.26 | step: 19.15

  2%|▏         | 878/41250 [2:07:32<97:09:21,  8.66s/it]
                                                        
{'loss': 0.1252, 'grad_norm': 1.7828994989395142, 'learning_rate': 2.8368336025848148e-05, 'epoch': 0.21}

  2%|▏         | 878/41250 [2:07:32<97:09:21,  8.66s/it][2025-04-25 10:05:15,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:05:15,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.85 | bwd_microstep: 5728.36 | bwd_inner_microstep: 5657.98 | bwd_allreduce_microstep: 70.34 | step_microstep: 18.54
[2025-04-25 10:05:15,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.85 | bwd: 5728.38 | bwd_inner: 5657.98 | bwd_allreduce: 70.35 | step: 18.55

  2%|▏         | 879/41250 [2:07:41<97:09:17,  8.66s/it]
                                                        
{'loss': 0.1335, 'grad_norm': 2.0721256732940674, 'learning_rate': 2.840064620355412e-05, 'epoch': 0.21}

  2%|▏         | 879/41250 [2:07:41<97:09:17,  8.66s/it][2025-04-25 10:05:24,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:05:24,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.19 | bwd_microstep: 5693.67 | bwd_inner_microstep: 5680.80 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.57
[2025-04-25 10:05:24,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.19 | bwd: 5693.68 | bwd_inner: 5680.80 | bwd_allreduce: 12.85 | step: 18.57

  2%|▏         | 880/41250 [2:07:49<97:01:46,  8.65s/it]
                                                        
{'loss': 0.3608, 'grad_norm': 2.6647520065307617, 'learning_rate': 2.84329563812601e-05, 'epoch': 0.21}

  2%|▏         | 880/41250 [2:07:49<97:01:46,  8.65s/it][2025-04-25 10:05:33,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:05:33,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.72 | bwd_microstep: 5878.56 | bwd_inner_microstep: 5640.64 | bwd_allreduce_microstep: 237.88 | step_microstep: 18.24
[2025-04-25 10:05:33,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.72 | bwd: 5878.58 | bwd_inner: 5640.64 | bwd_allreduce: 237.90 | step: 18.24

  2%|▏         | 881/41250 [2:07:58<97:29:04,  8.69s/it]
                                                        
{'loss': 0.1004, 'grad_norm': 1.8886300325393677, 'learning_rate': 2.846526655896608e-05, 'epoch': 0.21}

  2%|▏         | 881/41250 [2:07:58<97:29:04,  8.69s/it][2025-04-25 10:05:41,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 10:05:41,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.49 | bwd_microstep: 5778.60 | bwd_inner_microstep: 5765.79 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.72
[2025-04-25 10:05:41,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.49 | bwd: 5778.61 | bwd_inner: 5765.79 | bwd_allreduce: 12.78 | step: 18.72

  2%|▏         | 882/41250 [2:08:07<97:40:31,  8.71s/it]
                                                        
{'loss': 0.1959, 'grad_norm': 2.833757162094116, 'learning_rate': 2.8497576736672053e-05, 'epoch': 0.21}

  2%|▏         | 882/41250 [2:08:07<97:40:31,  8.71s/it][2025-04-25 10:05:50,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:05:50,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.08 | bwd_microstep: 5765.46 | bwd_inner_microstep: 5638.79 | bwd_allreduce_microstep: 126.62 | step_microstep: 18.47
[2025-04-25 10:05:50,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.08 | bwd: 5765.48 | bwd_inner: 5638.79 | bwd_allreduce: 126.64 | step: 18.47

  2%|▏         | 883/41250 [2:08:15<97:33:45,  8.70s/it]
                                                        
{'loss': 0.294, 'grad_norm': 2.8418891429901123, 'learning_rate': 2.852988691437803e-05, 'epoch': 0.21}

  2%|▏         | 883/41250 [2:08:15<97:33:45,  8.70s/it][2025-04-25 10:05:59,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 10:05:59,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.44 | bwd_microstep: 5772.39 | bwd_inner_microstep: 5642.04 | bwd_allreduce_microstep: 130.31 | step_microstep: 18.88
[2025-04-25 10:05:59,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.44 | bwd: 5772.40 | bwd_inner: 5642.04 | bwd_allreduce: 130.32 | step: 18.88

  2%|▏         | 884/41250 [2:08:24<97:30:45,  8.70s/it]
                                                        
{'loss': 0.4042, 'grad_norm': 2.683314323425293, 'learning_rate': 2.856219709208401e-05, 'epoch': 0.21}

  2%|▏         | 884/41250 [2:08:24<97:30:45,  8.70s/it][2025-04-25 10:06:07,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:06:07,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.00 | bwd_microstep: 5725.76 | bwd_inner_microstep: 5692.77 | bwd_allreduce_microstep: 32.95 | step_microstep: 18.55
[2025-04-25 10:06:07,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.00 | bwd: 5725.78 | bwd_inner: 5692.77 | bwd_allreduce: 32.96 | step: 18.55

  2%|▏         | 885/41250 [2:08:33<97:26:09,  8.69s/it]
                                                        
{'loss': 0.2315, 'grad_norm': 2.770115852355957, 'learning_rate': 2.859450726978999e-05, 'epoch': 0.21}

  2%|▏         | 885/41250 [2:08:33<97:26:09,  8.69s/it][2025-04-25 10:06:16,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 10:06:16,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.76 | bwd_microstep: 6014.64 | bwd_inner_microstep: 5657.57 | bwd_allreduce_microstep: 357.03 | step_microstep: 18.56
[2025-04-25 10:06:16,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.76 | bwd: 6014.66 | bwd_inner: 5657.56 | bwd_allreduce: 357.05 | step: 18.56

  2%|▏         | 886/41250 [2:08:42<98:12:01,  8.76s/it]
                                                        
{'loss': 0.2303, 'grad_norm': 2.8355584144592285, 'learning_rate': 2.862681744749596e-05, 'epoch': 0.21}

  2%|▏         | 886/41250 [2:08:42<98:12:01,  8.76s/it][2025-04-25 10:06:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 10:06:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.58 | bwd_microstep: 5774.42 | bwd_inner_microstep: 5761.66 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.39
[2025-04-25 10:06:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.58 | bwd: 5774.43 | bwd_inner: 5761.66 | bwd_allreduce: 12.72 | step: 18.40

  2%|▏         | 887/41250 [2:08:50<98:08:20,  8.75s/it]
                                                        
{'loss': 0.2024, 'grad_norm': 1.6341688632965088, 'learning_rate': 2.8659127625201942e-05, 'epoch': 0.22}

  2%|▏         | 887/41250 [2:08:50<98:08:20,  8.75s/it][2025-04-25 10:06:34,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:06:34,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5693.31 | bwd_inner_microstep: 5653.77 | bwd_allreduce_microstep: 39.50 | step_microstep: 18.70
[2025-04-25 10:06:34,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5693.33 | bwd_inner: 5653.77 | bwd_allreduce: 39.52 | step: 18.70

  2%|▏         | 888/41250 [2:08:59<97:38:12,  8.71s/it]
                                                        
{'loss': 0.3584, 'grad_norm': 4.426094055175781, 'learning_rate': 2.869143780290792e-05, 'epoch': 0.22}

  2%|▏         | 888/41250 [2:08:59<97:38:12,  8.71s/it][2025-04-25 10:06:42,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 10:06:42,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5712.39 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.61
[2025-04-25 10:06:42,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5712.40 | bwd_inner: 5699.69 | bwd_allreduce: 12.68 | step: 18.61

  2%|▏         | 889/41250 [2:09:08<97:25:44,  8.69s/it]
                                                        
{'loss': 0.2808, 'grad_norm': 2.5264272689819336, 'learning_rate': 2.8723747980613895e-05, 'epoch': 0.22}

  2%|▏         | 889/41250 [2:09:08<97:25:44,  8.69s/it][2025-04-25 10:06:51,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:06:51,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.28 | bwd_microstep: 5750.42 | bwd_inner_microstep: 5687.38 | bwd_allreduce_microstep: 62.99 | step_microstep: 18.41
[2025-04-25 10:06:51,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.28 | bwd: 5750.43 | bwd_inner: 5687.38 | bwd_allreduce: 63.01 | step: 18.41

  2%|▏         | 890/41250 [2:09:16<97:24:28,  8.69s/it]
                                                        
{'loss': 0.2516, 'grad_norm': 2.26692795753479, 'learning_rate': 2.8756058158319873e-05, 'epoch': 0.22}

  2%|▏         | 890/41250 [2:09:16<97:24:28,  8.69s/it][2025-04-25 10:07:00,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:07:00,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.02 | bwd_microstep: 5707.22 | bwd_inner_microstep: 5661.58 | bwd_allreduce_microstep: 45.60 | step_microstep: 18.38
[2025-04-25 10:07:00,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.02 | bwd: 5707.24 | bwd_inner: 5661.58 | bwd_allreduce: 45.61 | step: 18.38

  2%|▏         | 891/41250 [2:09:25<97:12:00,  8.67s/it]
                                                        
{'loss': 0.2692, 'grad_norm': 1.958299994468689, 'learning_rate': 2.878836833602585e-05, 'epoch': 0.22}

  2%|▏         | 891/41250 [2:09:25<97:12:00,  8.67s/it][2025-04-25 10:07:08,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:07:08,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.06 | bwd_microstep: 5687.22 | bwd_inner_microstep: 5656.59 | bwd_allreduce_microstep: 30.58 | step_microstep: 18.92
[2025-04-25 10:07:08,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.06 | bwd: 5687.24 | bwd_inner: 5656.59 | bwd_allreduce: 30.60 | step: 18.92

  2%|▏         | 892/41250 [2:09:34<97:04:09,  8.66s/it]
                                                        
{'loss': 0.2793, 'grad_norm': 2.147800922393799, 'learning_rate': 2.8820678513731825e-05, 'epoch': 0.22}

  2%|▏         | 892/41250 [2:09:34<97:04:09,  8.66s/it][2025-04-25 10:07:17,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 10:07:17,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.62 | bwd_microstep: 5730.40 | bwd_inner_microstep: 5717.42 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.98
[2025-04-25 10:07:17,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.62 | bwd: 5730.41 | bwd_inner: 5717.42 | bwd_allreduce: 12.95 | step: 18.99

  2%|▏         | 893/41250 [2:09:42<97:08:10,  8.66s/it]
                                                        
{'loss': 0.0735, 'grad_norm': 0.9472250938415527, 'learning_rate': 2.8852988691437803e-05, 'epoch': 0.22}

  2%|▏         | 893/41250 [2:09:42<97:08:10,  8.66s/it][2025-04-25 10:07:26,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 10:07:26,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.06 | bwd_microstep: 5743.97 | bwd_inner_microstep: 5692.33 | bwd_allreduce_microstep: 51.59 | step_microstep: 18.72
[2025-04-25 10:07:26,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.06 | bwd: 5743.98 | bwd_inner: 5692.33 | bwd_allreduce: 51.61 | step: 18.73

  2%|▏         | 894/41250 [2:09:51<97:12:26,  8.67s/it]
                                                        
{'loss': 0.2561, 'grad_norm': 2.568652629852295, 'learning_rate': 2.8885298869143784e-05, 'epoch': 0.22}

  2%|▏         | 894/41250 [2:09:51<97:12:26,  8.67s/it][2025-04-25 10:07:34,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 10:07:34,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.33 | bwd_microstep: 5711.44 | bwd_inner_microstep: 5698.80 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.75
[2025-04-25 10:07:34,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.33 | bwd: 5711.45 | bwd_inner: 5698.80 | bwd_allreduce: 12.62 | step: 18.75

  2%|▏         | 895/41250 [2:10:00<97:06:51,  8.66s/it]
                                                        
{'loss': 0.3281, 'grad_norm': 3.1283457279205322, 'learning_rate': 2.8917609046849762e-05, 'epoch': 0.22}

  2%|▏         | 895/41250 [2:10:00<97:06:51,  8.66s/it][2025-04-25 10:07:43,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:07:43,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.51 | bwd_microstep: 5712.44 | bwd_inner_microstep: 5699.56 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.80
[2025-04-25 10:07:43,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.51 | bwd: 5712.45 | bwd_inner: 5699.56 | bwd_allreduce: 12.86 | step: 18.80

  2%|▏         | 896/41250 [2:10:08<97:04:03,  8.66s/it]
                                                        
{'loss': 0.1517, 'grad_norm': 1.9949305057525635, 'learning_rate': 2.8949919224555736e-05, 'epoch': 0.22}

  2%|▏         | 896/41250 [2:10:08<97:04:03,  8.66s/it][2025-04-25 10:07:52,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:07:52,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.09 | bwd_microstep: 5921.84 | bwd_inner_microstep: 5667.15 | bwd_allreduce_microstep: 254.64 | step_microstep: 18.68
[2025-04-25 10:07:52,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.09 | bwd: 5921.85 | bwd_inner: 5667.15 | bwd_allreduce: 254.66 | step: 18.68

  2%|▏         | 897/41250 [2:10:17<97:40:19,  8.71s/it]
                                                        
{'loss': 0.2421, 'grad_norm': 2.11702299118042, 'learning_rate': 2.8982229402261714e-05, 'epoch': 0.22}

  2%|▏         | 897/41250 [2:10:17<97:40:19,  8.71s/it][2025-04-25 10:08:00,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.18
[2025-04-25 10:08:00,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.74 | bwd_microstep: 5709.86 | bwd_inner_microstep: 5696.91 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.38
[2025-04-25 10:08:00,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.74 | bwd: 5709.88 | bwd_inner: 5696.91 | bwd_allreduce: 12.92 | step: 19.38

  2%|▏         | 898/41250 [2:10:26<97:28:44,  8.70s/it]
                                                        
{'loss': 0.17, 'grad_norm': 1.4209370613098145, 'learning_rate': 2.9014539579967692e-05, 'epoch': 0.22}

  2%|▏         | 898/41250 [2:10:26<97:28:44,  8.70s/it][2025-04-25 10:08:09,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 10:08:09,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.86 | bwd_microstep: 5776.54 | bwd_inner_microstep: 5687.95 | bwd_allreduce_microstep: 88.56 | step_microstep: 18.53
[2025-04-25 10:08:09,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.86 | bwd: 5776.56 | bwd_inner: 5687.94 | bwd_allreduce: 88.57 | step: 18.54

  2%|▏         | 899/41250 [2:10:34<97:32:09,  8.70s/it]
                                                        
{'loss': 0.1245, 'grad_norm': 6.9203782081604, 'learning_rate': 2.9046849757673667e-05, 'epoch': 0.22}

  2%|▏         | 899/41250 [2:10:34<97:32:09,  8.70s/it][2025-04-25 10:08:18,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 10:08:18,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.49 | bwd_microstep: 5710.67 | bwd_inner_microstep: 5664.55 | bwd_allreduce_microstep: 46.07 | step_microstep: 19.01
[2025-04-25 10:08:18,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.49 | bwd: 5710.68 | bwd_inner: 5664.55 | bwd_allreduce: 46.09 | step: 19.02

  2%|▏         | 900/41250 [2:10:43<97:17:16,  8.68s/it]
                                                        
{'loss': 0.2506, 'grad_norm': 2.3496196269989014, 'learning_rate': 2.9079159935379645e-05, 'epoch': 0.22}

  2%|▏         | 900/41250 [2:10:43<97:17:16,  8.68s/it][2025-04-25 10:08:26,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 10:08:26,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.49 | bwd_microstep: 5793.82 | bwd_inner_microstep: 5718.00 | bwd_allreduce_microstep: 75.77 | step_microstep: 18.60
[2025-04-25 10:08:26,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.49 | bwd: 5793.83 | bwd_inner: 5718.00 | bwd_allreduce: 75.79 | step: 18.60

  2%|▏         | 901/41250 [2:10:52<97:27:29,  8.70s/it]
                                                        
{'loss': 0.401, 'grad_norm': 2.363328218460083, 'learning_rate': 2.9111470113085626e-05, 'epoch': 0.22}

  2%|▏         | 901/41250 [2:10:52<97:27:29,  8.70s/it][2025-04-25 10:08:35,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:08:35,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.35 | bwd_microstep: 5703.62 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 45.92 | step_microstep: 18.38
[2025-04-25 10:08:35,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.35 | bwd: 5703.63 | bwd_inner: 5657.65 | bwd_allreduce: 45.94 | step: 18.38

  2%|▏         | 902/41250 [2:11:00<97:14:59,  8.68s/it]
                                                        
{'loss': 0.211, 'grad_norm': 2.544459819793701, 'learning_rate': 2.9143780290791604e-05, 'epoch': 0.22}

  2%|▏         | 902/41250 [2:11:00<97:14:59,  8.68s/it][2025-04-25 10:08:44,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 10:08:44,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.08 | bwd_microstep: 5739.17 | bwd_inner_microstep: 5707.47 | bwd_allreduce_microstep: 31.66 | step_microstep: 18.76
[2025-04-25 10:08:44,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.08 | bwd: 5739.18 | bwd_inner: 5707.46 | bwd_allreduce: 31.68 | step: 18.76

  2%|▏         | 903/41250 [2:11:09<97:16:57,  8.68s/it]
                                                        
{'loss': 0.1395, 'grad_norm': 2.0115387439727783, 'learning_rate': 2.9176090468497578e-05, 'epoch': 0.22}

  2%|▏         | 903/41250 [2:11:09<97:16:57,  8.68s/it][2025-04-25 10:08:53,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 10:08:53,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.75 | bwd_microstep: 5789.98 | bwd_inner_microstep: 5662.28 | bwd_allreduce_microstep: 127.65 | step_microstep: 18.98
[2025-04-25 10:08:53,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.75 | bwd: 5789.99 | bwd_inner: 5662.28 | bwd_allreduce: 127.67 | step: 18.99

  2%|▏         | 904/41250 [2:11:18<97:21:45,  8.69s/it]
                                                        
{'loss': 0.2428, 'grad_norm': 2.316753387451172, 'learning_rate': 2.9208400646203556e-05, 'epoch': 0.22}

  2%|▏         | 904/41250 [2:11:18<97:21:45,  8.69s/it][2025-04-25 10:09:01,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:09:01,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.10 | bwd_microstep: 5776.58 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 88.99 | step_microstep: 18.40
[2025-04-25 10:09:01,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.10 | bwd: 5776.59 | bwd_inner: 5687.54 | bwd_allreduce: 89.01 | step: 18.41

  2%|▏         | 905/41250 [2:11:27<97:25:25,  8.69s/it]
                                                        
{'loss': 0.2541, 'grad_norm': 1.3992267847061157, 'learning_rate': 2.9240710823909534e-05, 'epoch': 0.22}

  2%|▏         | 905/41250 [2:11:27<97:25:25,  8.69s/it][2025-04-25 10:09:10,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 10:09:10,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.38 | bwd_microstep: 5796.71 | bwd_inner_microstep: 5661.47 | bwd_allreduce_microstep: 135.20 | step_microstep: 18.58
[2025-04-25 10:09:10,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.38 | bwd: 5796.72 | bwd_inner: 5661.47 | bwd_allreduce: 135.22 | step: 18.59

  2%|▏         | 906/41250 [2:11:35<97:30:26,  8.70s/it]
                                                        
{'loss': 0.2029, 'grad_norm': 1.6473952531814575, 'learning_rate': 2.927302100161551e-05, 'epoch': 0.22}

  2%|▏         | 906/41250 [2:11:35<97:30:26,  8.70s/it][2025-04-25 10:09:19,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:09:19,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.59 | bwd_microstep: 5774.03 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 125.00 | step_microstep: 18.63
[2025-04-25 10:09:19,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.59 | bwd: 5774.04 | bwd_inner: 5648.98 | bwd_allreduce: 125.02 | step: 18.63

  2%|▏         | 907/41250 [2:11:44<97:30:32,  8.70s/it]
                                                        
{'loss': 0.1584, 'grad_norm': 2.159752607345581, 'learning_rate': 2.9305331179321486e-05, 'epoch': 0.22}

  2%|▏         | 907/41250 [2:11:44<97:30:32,  8.70s/it][2025-04-25 10:09:27,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:09:27,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.65 | bwd_microstep: 5716.55 | bwd_inner_microstep: 5703.57 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.47
[2025-04-25 10:09:27,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.65 | bwd: 5716.56 | bwd_inner: 5703.57 | bwd_allreduce: 12.94 | step: 18.47

  2%|▏         | 908/41250 [2:11:53<97:23:49,  8.69s/it]
                                                        
{'loss': 0.2576, 'grad_norm': 2.1635360717773438, 'learning_rate': 2.9337641357027468e-05, 'epoch': 0.22}

  2%|▏         | 908/41250 [2:11:53<97:23:49,  8.69s/it][2025-04-25 10:09:36,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 10:09:36,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.87 | bwd_microstep: 5708.69 | bwd_inner_microstep: 5695.89 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.88
[2025-04-25 10:09:36,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.87 | bwd: 5708.70 | bwd_inner: 5695.89 | bwd_allreduce: 12.77 | step: 18.89

  2%|▏         | 909/41250 [2:12:01<97:14:39,  8.68s/it]
                                                        
{'loss': 0.2292, 'grad_norm': 2.2435827255249023, 'learning_rate': 2.9369951534733446e-05, 'epoch': 0.22}

  2%|▏         | 909/41250 [2:12:01<97:14:39,  8.68s/it][2025-04-25 10:09:45,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:09:45,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.58 | bwd_microstep: 5694.61 | bwd_inner_microstep: 5681.85 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.71
[2025-04-25 10:09:45,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.58 | bwd: 5694.63 | bwd_inner: 5681.85 | bwd_allreduce: 12.74 | step: 18.71

  2%|▏         | 910/41250 [2:12:10<97:03:24,  8.66s/it]
                                                        
{'loss': 0.2218, 'grad_norm': 1.7926338911056519, 'learning_rate': 2.940226171243942e-05, 'epoch': 0.22}

  2%|▏         | 910/41250 [2:12:10<97:03:24,  8.66s/it][2025-04-25 10:09:53,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:09:53,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5699.70 | bwd_inner_microstep: 5687.04 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.57
[2025-04-25 10:09:53,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5699.72 | bwd_inner: 5687.04 | bwd_allreduce: 12.64 | step: 18.57

  2%|▏         | 911/41250 [2:12:19<96:58:58,  8.66s/it]
                                                        
{'loss': 0.181, 'grad_norm': 2.200561761856079, 'learning_rate': 2.9434571890145398e-05, 'epoch': 0.22}

  2%|▏         | 911/41250 [2:12:19<96:58:58,  8.66s/it][2025-04-25 10:10:02,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.26 | optimizer_step: 1.08
[2025-04-25 10:10:02,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.93 | bwd_microstep: 5743.78 | bwd_inner_microstep: 5648.06 | bwd_allreduce_microstep: 95.67 | step_microstep: 19.88
[2025-04-25 10:10:02,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.93 | bwd: 5743.79 | bwd_inner: 5648.06 | bwd_allreduce: 95.69 | step: 19.89

  2%|▏         | 912/41250 [2:12:27<97:00:45,  8.66s/it]
                                                        
{'loss': 0.2112, 'grad_norm': 1.3878501653671265, 'learning_rate': 2.9466882067851376e-05, 'epoch': 0.22}

  2%|▏         | 912/41250 [2:12:27<97:00:45,  8.66s/it][2025-04-25 10:10:11,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-25 10:10:11,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5737.77 | bwd_inner_microstep: 5681.62 | bwd_allreduce_microstep: 56.09 | step_microstep: 19.21
[2025-04-25 10:10:11,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.97 | bwd: 5737.78 | bwd_inner: 5681.62 | bwd_allreduce: 56.11 | step: 19.21

  2%|▏         | 913/41250 [2:12:36<97:05:15,  8.66s/it]
                                                        
{'loss': 0.2384, 'grad_norm': 3.1879141330718994, 'learning_rate': 2.949919224555735e-05, 'epoch': 0.22}

  2%|▏         | 913/41250 [2:12:36<97:05:15,  8.66s/it][2025-04-25 10:10:19,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:10:19,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.00 | bwd_microstep: 5795.86 | bwd_inner_microstep: 5782.96 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.00
[2025-04-25 10:10:19,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.00 | bwd: 5795.88 | bwd_inner: 5782.96 | bwd_allreduce: 12.88 | step: 19.01

  2%|▏         | 914/41250 [2:12:45<97:26:45,  8.70s/it]
                                                        
{'loss': 0.2743, 'grad_norm': 2.1745693683624268, 'learning_rate': 2.9531502423263328e-05, 'epoch': 0.22}

  2%|▏         | 914/41250 [2:12:45<97:26:45,  8.70s/it][2025-04-25 10:10:28,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-25 10:10:28,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.44 | bwd_microstep: 5786.22 | bwd_inner_microstep: 5773.19 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.03
[2025-04-25 10:10:28,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.44 | bwd: 5786.24 | bwd_inner: 5773.19 | bwd_allreduce: 13.00 | step: 19.03

  2%|▏         | 915/41250 [2:12:53<97:41:16,  8.72s/it]
                                                        
{'loss': 0.2294, 'grad_norm': 1.6186083555221558, 'learning_rate': 2.956381260096931e-05, 'epoch': 0.22}

  2%|▏         | 915/41250 [2:12:53<97:41:16,  8.72s/it][2025-04-25 10:10:37,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 10:10:37,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.25 | bwd_microstep: 5903.37 | bwd_inner_microstep: 5649.07 | bwd_allreduce_microstep: 254.25 | step_microstep: 19.12
[2025-04-25 10:10:37,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.25 | bwd: 5903.39 | bwd_inner: 5649.07 | bwd_allreduce: 254.28 | step: 19.12

  2%|▏         | 916/41250 [2:13:02<98:03:57,  8.75s/it]
                                                        
{'loss': 0.3329, 'grad_norm': 4.342732906341553, 'learning_rate': 2.9596122778675287e-05, 'epoch': 0.22}

  2%|▏         | 916/41250 [2:13:02<98:03:57,  8.75s/it][2025-04-25 10:10:46,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:10:46,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.23 | bwd_microstep: 5726.52 | bwd_inner_microstep: 5680.14 | bwd_allreduce_microstep: 46.32 | step_microstep: 19.30
[2025-04-25 10:10:46,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.23 | bwd: 5726.53 | bwd_inner: 5680.14 | bwd_allreduce: 46.35 | step: 19.30

  2%|▏         | 917/41250 [2:13:11<97:45:55,  8.73s/it]
                                                        
{'loss': 0.2461, 'grad_norm': 1.2836147546768188, 'learning_rate': 2.9628432956381262e-05, 'epoch': 0.22}

  2%|▏         | 917/41250 [2:13:11<97:45:55,  8.73s/it][2025-04-25 10:10:54,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:10:54,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.81 | bwd_microstep: 5876.26 | bwd_inner_microstep: 5863.42 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.75
[2025-04-25 10:10:54,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.81 | bwd: 5876.27 | bwd_inner: 5863.42 | bwd_allreduce: 12.82 | step: 18.75

  2%|▏         | 918/41250 [2:13:20<98:19:27,  8.78s/it]
                                                        
{'loss': 0.2322, 'grad_norm': 1.653706669807434, 'learning_rate': 2.966074313408724e-05, 'epoch': 0.22}

  2%|▏         | 918/41250 [2:13:20<98:19:27,  8.78s/it][2025-04-25 10:11:03,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 10:11:03,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.64 | bwd_microstep: 5893.57 | bwd_inner_microstep: 5652.02 | bwd_allreduce_microstep: 241.51 | step_microstep: 18.55
[2025-04-25 10:11:03,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.64 | bwd: 5893.58 | bwd_inner: 5652.02 | bwd_allreduce: 241.53 | step: 18.55

  2%|▏         | 919/41250 [2:13:29<98:26:21,  8.79s/it]
                                                        
{'loss': 0.1932, 'grad_norm': 2.8898510932922363, 'learning_rate': 2.9693053311793218e-05, 'epoch': 0.22}

  2%|▏         | 919/41250 [2:13:29<98:26:21,  8.79s/it][2025-04-25 10:11:12,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.07 | optimizer_step: 1.01
[2025-04-25 10:11:12,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.51 | bwd_microstep: 5701.67 | bwd_inner_microstep: 5689.20 | bwd_allreduce_microstep: 12.41 | step_microstep: 19.79
[2025-04-25 10:11:12,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.51 | bwd: 5701.68 | bwd_inner: 5689.20 | bwd_allreduce: 12.43 | step: 19.79

  2%|▏         | 920/41250 [2:13:37<97:55:17,  8.74s/it]
                                                        
{'loss': 0.2155, 'grad_norm': 1.6311732530593872, 'learning_rate': 2.9725363489499192e-05, 'epoch': 0.22}

  2%|▏         | 920/41250 [2:13:37<97:55:17,  8.74s/it][2025-04-25 10:11:21,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.07 | optimizer_step: 0.92
[2025-04-25 10:11:21,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.31 | bwd_microstep: 5859.38 | bwd_inner_microstep: 5637.74 | bwd_allreduce_microstep: 221.58 | step_microstep: 19.09
[2025-04-25 10:11:21,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.31 | bwd: 5859.39 | bwd_inner: 5637.74 | bwd_allreduce: 221.61 | step: 19.09

  2%|▏         | 921/41250 [2:13:46<98:03:03,  8.75s/it]
                                                        
{'loss': 0.2044, 'grad_norm': 1.8642090559005737, 'learning_rate': 2.975767366720517e-05, 'epoch': 0.22}

  2%|▏         | 921/41250 [2:13:46<98:03:03,  8.75s/it][2025-04-25 10:11:29,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.18 | optimizer_step: 0.93
[2025-04-25 10:11:29,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.35 | bwd_microstep: 5691.28 | bwd_inner_microstep: 5642.32 | bwd_allreduce_microstep: 48.89 | step_microstep: 19.04
[2025-04-25 10:11:29,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.35 | bwd: 5691.29 | bwd_inner: 5642.32 | bwd_allreduce: 48.92 | step: 19.03

  2%|▏         | 922/41250 [2:13:55<97:33:09,  8.71s/it]
                                                        
{'loss': 0.0814, 'grad_norm': 0.849540114402771, 'learning_rate': 2.978998384491115e-05, 'epoch': 0.22}

  2%|▏         | 922/41250 [2:13:55<97:33:09,  8.71s/it][2025-04-25 10:11:38,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-25 10:11:38,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.19 | bwd_microstep: 5742.17 | bwd_inner_microstep: 5640.11 | bwd_allreduce_microstep: 102.00 | step_microstep: 19.35
[2025-04-25 10:11:38,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.19 | bwd: 5742.18 | bwd_inner: 5640.11 | bwd_allreduce: 102.02 | step: 19.35

  2%|▏         | 923/41250 [2:14:03<97:20:44,  8.69s/it]
                                                        
{'loss': 0.2277, 'grad_norm': 2.1518049240112305, 'learning_rate': 2.982229402261713e-05, 'epoch': 0.22}

  2%|▏         | 923/41250 [2:14:03<97:20:44,  8.69s/it][2025-04-25 10:11:47,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 10:11:47,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.01 | bwd_microstep: 5755.26 | bwd_inner_microstep: 5693.96 | bwd_allreduce_microstep: 61.25 | step_microstep: 19.05
[2025-04-25 10:11:47,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.01 | bwd: 5755.28 | bwd_inner: 5693.96 | bwd_allreduce: 61.28 | step: 19.05

  2%|▏         | 924/41250 [2:14:12<97:19:26,  8.69s/it]
                                                        
{'loss': 0.1135, 'grad_norm': 1.169637680053711, 'learning_rate': 2.9854604200323104e-05, 'epoch': 0.22}

  2%|▏         | 924/41250 [2:14:12<97:19:26,  8.69s/it][2025-04-25 10:11:55,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 10:11:55,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.42 | bwd_microstep: 5686.00 | bwd_inner_microstep: 5631.24 | bwd_allreduce_microstep: 54.71 | step_microstep: 18.88
[2025-04-25 10:11:55,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.42 | bwd: 5686.02 | bwd_inner: 5631.24 | bwd_allreduce: 54.73 | step: 18.89

  2%|▏         | 925/41250 [2:14:21<97:00:24,  8.66s/it]
                                                        
{'loss': 0.2969, 'grad_norm': 2.2881572246551514, 'learning_rate': 2.988691437802908e-05, 'epoch': 0.22}

  2%|▏         | 925/41250 [2:14:21<97:00:24,  8.66s/it][2025-04-25 10:12:04,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:12:04,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.29 | bwd_microstep: 5733.39 | bwd_inner_microstep: 5661.81 | bwd_allreduce_microstep: 71.53 | step_microstep: 18.87
[2025-04-25 10:12:04,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.29 | bwd: 5733.40 | bwd_inner: 5661.81 | bwd_allreduce: 71.55 | step: 18.88

  2%|▏         | 926/41250 [2:14:29<96:59:17,  8.66s/it]
                                                        
{'loss': 0.1745, 'grad_norm': 2.4910998344421387, 'learning_rate': 2.991922455573506e-05, 'epoch': 0.22}

  2%|▏         | 926/41250 [2:14:29<96:59:17,  8.66s/it][2025-04-25 10:12:13,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 1.18
[2025-04-25 10:12:13,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.06 | bwd_microstep: 5751.37 | bwd_inner_microstep: 5669.74 | bwd_allreduce_microstep: 81.58 | step_microstep: 19.33
[2025-04-25 10:12:13,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.06 | bwd: 5751.39 | bwd_inner: 5669.74 | bwd_allreduce: 81.60 | step: 19.33

  2%|▏         | 927/41250 [2:14:38<97:02:56,  8.66s/it]
                                                        
{'loss': 0.1695, 'grad_norm': 0.9861545562744141, 'learning_rate': 2.9951534733441034e-05, 'epoch': 0.22}

  2%|▏         | 927/41250 [2:14:38<97:02:56,  8.66s/it][2025-04-25 10:12:21,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:12:21,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.35 | bwd_microstep: 5688.32 | bwd_inner_microstep: 5675.61 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.68
[2025-04-25 10:12:21,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.35 | bwd: 5688.33 | bwd_inner: 5675.61 | bwd_allreduce: 12.68 | step: 18.68

  2%|▏         | 928/41250 [2:14:47<96:51:55,  8.65s/it]
                                                        
{'loss': 0.0639, 'grad_norm': 0.6342306137084961, 'learning_rate': 2.9983844911147012e-05, 'epoch': 0.22}

  2%|▏         | 928/41250 [2:14:47<96:51:55,  8.65s/it][2025-04-25 10:12:30,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 10:12:30,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.91 | bwd_microstep: 5746.30 | bwd_inner_microstep: 5642.20 | bwd_allreduce_microstep: 104.05 | step_microstep: 19.14
[2025-04-25 10:12:30,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.91 | bwd: 5746.31 | bwd_inner: 5642.20 | bwd_allreduce: 104.07 | step: 19.14

  2%|▏         | 929/41250 [2:14:55<96:54:14,  8.65s/it]
                                                        
{'loss': 0.2358, 'grad_norm': 3.369234561920166, 'learning_rate': 3.0016155088852993e-05, 'epoch': 0.23}

  2%|▏         | 929/41250 [2:14:55<96:54:14,  8.65s/it][2025-04-25 10:12:39,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 10:12:39,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.61 | bwd_microstep: 5768.63 | bwd_inner_microstep: 5615.55 | bwd_allreduce_microstep: 153.04 | step_microstep: 19.25
[2025-04-25 10:12:39,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.61 | bwd: 5768.65 | bwd_inner: 5615.55 | bwd_allreduce: 153.06 | step: 19.25

  2%|▏         | 930/41250 [2:15:04<96:58:53,  8.66s/it]
                                                        
{'loss': 0.1836, 'grad_norm': 1.6587426662445068, 'learning_rate': 3.004846526655897e-05, 'epoch': 0.23}

  2%|▏         | 930/41250 [2:15:04<96:58:53,  8.66s/it][2025-04-25 10:12:47,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 10:12:47,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.62 | bwd_microstep: 5682.88 | bwd_inner_microstep: 5670.01 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.07
[2025-04-25 10:12:47,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.62 | bwd: 5682.89 | bwd_inner: 5670.01 | bwd_allreduce: 12.84 | step: 19.08

  2%|▏         | 931/41250 [2:15:12<96:47:20,  8.64s/it]
                                                        
{'loss': 0.2159, 'grad_norm': 2.6150219440460205, 'learning_rate': 3.0080775444264945e-05, 'epoch': 0.23}

  2%|▏         | 931/41250 [2:15:12<96:47:20,  8.64s/it][2025-04-25 10:12:56,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.15
[2025-04-25 10:12:56,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.30 | bwd_microstep: 5767.77 | bwd_inner_microstep: 5637.68 | bwd_allreduce_microstep: 130.04 | step_microstep: 19.38
[2025-04-25 10:12:56,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.30 | bwd: 5767.78 | bwd_inner: 5637.68 | bwd_allreduce: 130.06 | step: 19.38

  2%|▏         | 932/41250 [2:15:21<96:53:30,  8.65s/it]
                                                        
{'loss': 0.1057, 'grad_norm': 1.1937893629074097, 'learning_rate': 3.0113085621970923e-05, 'epoch': 0.23}

  2%|▏         | 932/41250 [2:15:21<96:53:30,  8.65s/it][2025-04-25 10:13:04,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 10:13:04,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.45 | bwd_microstep: 5737.61 | bwd_inner_microstep: 5690.58 | bwd_allreduce_microstep: 46.99 | step_microstep: 18.58
[2025-04-25 10:13:04,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.45 | bwd: 5737.63 | bwd_inner: 5690.57 | bwd_allreduce: 47.01 | step: 18.58

  2%|▏         | 933/41250 [2:15:30<96:56:02,  8.66s/it]
                                                        
{'loss': 0.1423, 'grad_norm': 2.2816994190216064, 'learning_rate': 3.01453957996769e-05, 'epoch': 0.23}

  2%|▏         | 933/41250 [2:15:30<96:56:02,  8.66s/it][2025-04-25 10:13:13,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.23 | optimizer_step: 0.96
[2025-04-25 10:13:13,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.07 | bwd_microstep: 5734.97 | bwd_inner_microstep: 5677.15 | bwd_allreduce_microstep: 57.76 | step_microstep: 20.05
[2025-04-25 10:13:13,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.07 | bwd: 5734.98 | bwd_inner: 5677.15 | bwd_allreduce: 57.78 | step: 20.06

  2%|▏         | 934/41250 [2:15:38<96:57:24,  8.66s/it]
                                                        
{'loss': 0.2729, 'grad_norm': 2.5945842266082764, 'learning_rate': 3.0177705977382876e-05, 'epoch': 0.23}

  2%|▏         | 934/41250 [2:15:38<96:57:24,  8.66s/it][2025-04-25 10:13:22,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:13:22,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.76 | bwd_microstep: 5711.29 | bwd_inner_microstep: 5630.45 | bwd_allreduce_microstep: 80.80 | step_microstep: 18.58
[2025-04-25 10:13:22,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.76 | bwd: 5711.31 | bwd_inner: 5630.45 | bwd_allreduce: 80.82 | step: 18.58

  2%|▏         | 935/41250 [2:15:47<96:47:44,  8.64s/it]
                                                        
{'loss': 0.0852, 'grad_norm': 1.6771278381347656, 'learning_rate': 3.0210016155088854e-05, 'epoch': 0.23}

  2%|▏         | 935/41250 [2:15:47<96:47:44,  8.64s/it][2025-04-25 10:13:30,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:13:30,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.51 | bwd_microstep: 5770.75 | bwd_inner_microstep: 5628.27 | bwd_allreduce_microstep: 142.43 | step_microstep: 18.40
[2025-04-25 10:13:30,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.51 | bwd: 5770.77 | bwd_inner: 5628.27 | bwd_allreduce: 142.45 | step: 18.41

  2%|▏         | 936/41250 [2:15:56<96:53:05,  8.65s/it]
                                                        
{'loss': 0.0722, 'grad_norm': 0.7101629972457886, 'learning_rate': 3.0242326332794835e-05, 'epoch': 0.23}

  2%|▏         | 936/41250 [2:15:56<96:53:05,  8.65s/it][2025-04-25 10:13:39,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.12
[2025-04-25 10:13:39,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.64 | bwd_microstep: 5685.71 | bwd_inner_microstep: 5673.15 | bwd_allreduce_microstep: 12.52 | step_microstep: 19.07
[2025-04-25 10:13:39,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.64 | bwd: 5685.73 | bwd_inner: 5673.15 | bwd_allreduce: 12.53 | step: 19.07

  2%|▏         | 937/41250 [2:16:04<96:46:17,  8.64s/it]
                                                        
{'loss': 0.2496, 'grad_norm': 1.9838300943374634, 'learning_rate': 3.0274636510500813e-05, 'epoch': 0.23}

  2%|▏         | 937/41250 [2:16:04<96:46:17,  8.64s/it][2025-04-25 10:13:48,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:13:48,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.26 | bwd_microstep: 5751.45 | bwd_inner_microstep: 5635.20 | bwd_allreduce_microstep: 116.21 | step_microstep: 18.79
[2025-04-25 10:13:48,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.26 | bwd: 5751.46 | bwd_inner: 5635.20 | bwd_allreduce: 116.22 | step: 18.79

  2%|▏         | 938/41250 [2:16:13<96:48:07,  8.64s/it]
                                                        
{'loss': 0.1297, 'grad_norm': 1.5779703855514526, 'learning_rate': 3.0306946688206787e-05, 'epoch': 0.23}

  2%|▏         | 938/41250 [2:16:13<96:48:07,  8.64s/it][2025-04-25 10:13:56,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 10:13:56,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.75 | bwd_microstep: 5774.85 | bwd_inner_microstep: 5635.69 | bwd_allreduce_microstep: 139.12 | step_microstep: 18.62
[2025-04-25 10:13:56,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.75 | bwd: 5774.86 | bwd_inner: 5635.69 | bwd_allreduce: 139.13 | step: 18.63

  2%|▏         | 939/41250 [2:16:22<96:53:53,  8.65s/it]
                                                        
{'loss': 0.26, 'grad_norm': 2.4957268238067627, 'learning_rate': 3.0339256865912765e-05, 'epoch': 0.23}

  2%|▏         | 939/41250 [2:16:22<96:53:53,  8.65s/it][2025-04-25 10:14:05,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:14:05,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.90 | bwd_microstep: 5708.01 | bwd_inner_microstep: 5695.11 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.65
[2025-04-25 10:14:05,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.90 | bwd: 5708.02 | bwd_inner: 5695.11 | bwd_allreduce: 12.87 | step: 18.66

  2%|▏         | 940/41250 [2:16:30<96:50:57,  8.65s/it]
                                                        
{'loss': 0.5266, 'grad_norm': 4.148018836975098, 'learning_rate': 3.0371567043618743e-05, 'epoch': 0.23}

  2%|▏         | 940/41250 [2:16:30<96:50:57,  8.65s/it][2025-04-25 10:14:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:14:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.24 | bwd_microstep: 5796.99 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 141.39 | step_microstep: 18.29
[2025-04-25 10:14:14,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.24 | bwd: 5797.00 | bwd_inner: 5655.56 | bwd_allreduce: 141.40 | step: 18.29

  2%|▏         | 941/41250 [2:16:39<96:59:19,  8.66s/it]
                                                        
{'loss': 0.2278, 'grad_norm': 2.7107744216918945, 'learning_rate': 3.0403877221324718e-05, 'epoch': 0.23}

  2%|▏         | 941/41250 [2:16:39<96:59:19,  8.66s/it][2025-04-25 10:14:22,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 10:14:22,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.15 | bwd_microstep: 5776.78 | bwd_inner_microstep: 5656.39 | bwd_allreduce_microstep: 120.35 | step_microstep: 18.89
[2025-04-25 10:14:22,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.15 | bwd: 5776.79 | bwd_inner: 5656.39 | bwd_allreduce: 120.37 | step: 18.89

  2%|▏         | 942/41250 [2:16:48<97:03:44,  8.67s/it]
                                                        
{'loss': 0.0857, 'grad_norm': 1.1157106161117554, 'learning_rate': 3.0436187399030695e-05, 'epoch': 0.23}

  2%|▏         | 942/41250 [2:16:48<97:03:44,  8.67s/it][2025-04-25 10:14:31,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:14:31,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.93 | bwd_microstep: 5787.16 | bwd_inner_microstep: 5773.71 | bwd_allreduce_microstep: 13.39 | step_microstep: 19.32
[2025-04-25 10:14:31,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.93 | bwd: 5787.17 | bwd_inner: 5773.71 | bwd_allreduce: 13.41 | step: 19.33

  2%|▏         | 943/41250 [2:16:56<97:22:08,  8.70s/it]
                                                        
{'loss': 0.301, 'grad_norm': 1.6927330493927002, 'learning_rate': 3.0468497576736677e-05, 'epoch': 0.23}

  2%|▏         | 943/41250 [2:16:56<97:22:08,  8.70s/it][2025-04-25 10:14:40,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:14:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.60 | bwd_microstep: 5714.83 | bwd_inner_microstep: 5652.16 | bwd_allreduce_microstep: 62.63 | step_microstep: 18.61
[2025-04-25 10:14:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.60 | bwd: 5714.84 | bwd_inner: 5652.16 | bwd_allreduce: 62.64 | step: 18.61

  2%|▏         | 944/41250 [2:17:05<97:06:33,  8.67s/it]
                                                        
{'loss': 0.1419, 'grad_norm': 1.6574114561080933, 'learning_rate': 3.0500807754442655e-05, 'epoch': 0.23}

  2%|▏         | 944/41250 [2:17:05<97:06:33,  8.67s/it][2025-04-25 10:14:48,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 10:14:48,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.74 | bwd_microstep: 5719.44 | bwd_inner_microstep: 5643.23 | bwd_allreduce_microstep: 76.16 | step_microstep: 18.75
[2025-04-25 10:14:48,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.74 | bwd: 5719.45 | bwd_inner: 5643.23 | bwd_allreduce: 76.18 | step: 18.75

  2%|▏         | 945/41250 [2:17:14<96:56:38,  8.66s/it]
                                                        
{'loss': 0.1172, 'grad_norm': 1.468883991241455, 'learning_rate': 3.0533117932148626e-05, 'epoch': 0.23}

  2%|▏         | 945/41250 [2:17:14<96:56:38,  8.66s/it][2025-04-25 10:14:57,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:14:57,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.74 | bwd_microstep: 5716.43 | bwd_inner_microstep: 5651.95 | bwd_allreduce_microstep: 64.43 | step_microstep: 19.10
[2025-04-25 10:14:57,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.74 | bwd: 5716.44 | bwd_inner: 5651.95 | bwd_allreduce: 64.45 | step: 19.11

  2%|▏         | 946/41250 [2:17:22<96:50:14,  8.65s/it]
                                                        
{'loss': 0.1912, 'grad_norm': 2.0411150455474854, 'learning_rate': 3.0565428109854604e-05, 'epoch': 0.23}

  2%|▏         | 946/41250 [2:17:22<96:50:14,  8.65s/it][2025-04-25 10:15:06,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:15:06,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.40 | bwd_microstep: 5691.73 | bwd_inner_microstep: 5665.38 | bwd_allreduce_microstep: 26.30 | step_microstep: 18.71
[2025-04-25 10:15:06,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.40 | bwd: 5691.74 | bwd_inner: 5665.38 | bwd_allreduce: 26.32 | step: 18.71

  2%|▏         | 947/41250 [2:17:31<96:43:43,  8.64s/it]
                                                        
{'loss': 0.1726, 'grad_norm': 1.651259422302246, 'learning_rate': 3.059773828756059e-05, 'epoch': 0.23}

  2%|▏         | 947/41250 [2:17:31<96:43:43,  8.64s/it][2025-04-25 10:15:14,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 10:15:14,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.57 | bwd_microstep: 5761.79 | bwd_inner_microstep: 5699.48 | bwd_allreduce_microstep: 62.26 | step_microstep: 18.97
[2025-04-25 10:15:14,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.57 | bwd: 5761.80 | bwd_inner: 5699.48 | bwd_allreduce: 62.28 | step: 18.98

  2%|▏         | 948/41250 [2:17:40<96:55:37,  8.66s/it]
                                                        
{'loss': 0.371, 'grad_norm': 2.3787097930908203, 'learning_rate': 3.063004846526656e-05, 'epoch': 0.23}

  2%|▏         | 948/41250 [2:17:40<96:55:37,  8.66s/it][2025-04-25 10:15:23,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 10:15:23,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.20 | bwd_microstep: 5715.49 | bwd_inner_microstep: 5702.95 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.73
[2025-04-25 10:15:23,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.20 | bwd: 5715.51 | bwd_inner: 5702.95 | bwd_allreduce: 12.51 | step: 18.73

  2%|▏         | 949/41250 [2:17:48<96:55:10,  8.66s/it]
                                                        
{'loss': 0.1521, 'grad_norm': 2.2988688945770264, 'learning_rate': 3.066235864297254e-05, 'epoch': 0.23}

  2%|▏         | 949/41250 [2:17:48<96:55:10,  8.66s/it][2025-04-25 10:15:32,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:15:32,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.75 | bwd_microstep: 5782.81 | bwd_inner_microstep: 5770.03 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.00
[2025-04-25 10:15:32,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.75 | bwd: 5782.82 | bwd_inner: 5770.03 | bwd_allreduce: 12.75 | step: 19.00

  2%|▏         | 950/41250 [2:17:57<97:13:14,  8.68s/it]
                                                        
{'loss': 0.1663, 'grad_norm': 1.718657374382019, 'learning_rate': 3.0694668820678515e-05, 'epoch': 0.23}

  2%|▏         | 950/41250 [2:17:57<97:13:14,  8.68s/it][2025-04-25 10:15:40,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:15:40,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.96 | bwd_microstep: 5790.95 | bwd_inner_microstep: 5663.44 | bwd_allreduce_microstep: 127.47 | step_microstep: 18.69
[2025-04-25 10:15:40,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5790.97 | bwd_inner: 5663.44 | bwd_allreduce: 127.48 | step: 18.70

  2%|▏         | 951/41250 [2:18:06<97:17:19,  8.69s/it]
                                                        
{'loss': 0.0976, 'grad_norm': 1.5579770803451538, 'learning_rate': 3.072697899838449e-05, 'epoch': 0.23}

  2%|▏         | 951/41250 [2:18:06<97:17:19,  8.69s/it][2025-04-25 10:15:49,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:15:49,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.84 | bwd_microstep: 5768.57 | bwd_inner_microstep: 5723.38 | bwd_allreduce_microstep: 45.14 | step_microstep: 18.41
[2025-04-25 10:15:49,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.84 | bwd: 5768.58 | bwd_inner: 5723.38 | bwd_allreduce: 45.16 | step: 18.41

  2%|▏         | 952/41250 [2:18:14<97:19:45,  8.69s/it]
                                                        
{'loss': 0.1738, 'grad_norm': 5.862417697906494, 'learning_rate': 3.075928917609047e-05, 'epoch': 0.23}

  2%|▏         | 952/41250 [2:18:14<97:19:45,  8.69s/it][2025-04-25 10:15:58,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 10:15:58,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.41 | bwd_microstep: 5768.96 | bwd_inner_microstep: 5689.96 | bwd_allreduce_microstep: 78.95 | step_microstep: 19.49
[2025-04-25 10:15:58,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.41 | bwd: 5768.98 | bwd_inner: 5689.96 | bwd_allreduce: 78.98 | step: 19.50

  2%|▏         | 953/41250 [2:18:23<97:22:55,  8.70s/it]
                                                        
{'loss': 0.1408, 'grad_norm': 2.800262451171875, 'learning_rate': 3.079159935379645e-05, 'epoch': 0.23}

  2%|▏         | 953/41250 [2:18:23<97:22:55,  8.70s/it][2025-04-25 10:16:07,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:16:07,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.98 | bwd_microstep: 5762.21 | bwd_inner_microstep: 5694.77 | bwd_allreduce_microstep: 67.40 | step_microstep: 18.77
[2025-04-25 10:16:07,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.98 | bwd: 5762.22 | bwd_inner: 5694.77 | bwd_allreduce: 67.41 | step: 18.77

  2%|▏         | 954/41250 [2:18:32<97:21:23,  8.70s/it]
                                                        
{'loss': 0.3824, 'grad_norm': 3.3181867599487305, 'learning_rate': 3.082390953150243e-05, 'epoch': 0.23}

  2%|▏         | 954/41250 [2:18:32<97:21:23,  8.70s/it][2025-04-25 10:16:15,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:16:15,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.66 | bwd_microstep: 5715.48 | bwd_inner_microstep: 5691.73 | bwd_allreduce_microstep: 23.70 | step_microstep: 18.68
[2025-04-25 10:16:15,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.66 | bwd: 5715.49 | bwd_inner: 5691.73 | bwd_allreduce: 23.71 | step: 18.68

  2%|▏         | 955/41250 [2:18:41<97:11:02,  8.68s/it]
                                                        
{'loss': 0.0499, 'grad_norm': 1.0890108346939087, 'learning_rate': 3.0856219709208405e-05, 'epoch': 0.23}

  2%|▏         | 955/41250 [2:18:41<97:11:02,  8.68s/it][2025-04-25 10:16:24,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 10:16:24,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.53 | bwd_microstep: 5712.50 | bwd_inner_microstep: 5699.77 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.48
[2025-04-25 10:16:24,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.54 | bwd: 5712.52 | bwd_inner: 5699.77 | bwd_allreduce: 12.71 | step: 18.48

  2%|▏         | 956/41250 [2:18:49<97:03:34,  8.67s/it]
                                                        
{'loss': 0.5379, 'grad_norm': 3.804003953933716, 'learning_rate': 3.088852988691438e-05, 'epoch': 0.23}

  2%|▏         | 956/41250 [2:18:49<97:03:34,  8.67s/it][2025-04-25 10:16:33,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 10:16:33,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.92 | bwd_microstep: 5780.33 | bwd_inner_microstep: 5767.30 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.84
[2025-04-25 10:16:33,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.92 | bwd: 5780.34 | bwd_inner: 5767.30 | bwd_allreduce: 13.00 | step: 18.84

  2%|▏         | 957/41250 [2:18:58<97:19:32,  8.70s/it]
                                                        
{'loss': 0.1288, 'grad_norm': 2.7628371715545654, 'learning_rate': 3.092084006462036e-05, 'epoch': 0.23}

  2%|▏         | 957/41250 [2:18:58<97:19:32,  8.70s/it][2025-04-25 10:16:41,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 10:16:41,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.11 | bwd_microstep: 5713.43 | bwd_inner_microstep: 5700.41 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.82
[2025-04-25 10:16:41,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.11 | bwd: 5713.44 | bwd_inner: 5700.41 | bwd_allreduce: 12.99 | step: 18.83

  2%|▏         | 958/41250 [2:19:07<97:10:32,  8.68s/it]
                                                        
{'loss': 0.17, 'grad_norm': 1.920711874961853, 'learning_rate': 3.095315024232634e-05, 'epoch': 0.23}

  2%|▏         | 958/41250 [2:19:07<97:10:32,  8.68s/it][2025-04-25 10:16:50,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 10:16:50,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.80 | bwd_microstep: 5751.60 | bwd_inner_microstep: 5721.59 | bwd_allreduce_microstep: 29.96 | step_microstep: 18.99
[2025-04-25 10:16:50,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.80 | bwd: 5751.61 | bwd_inner: 5721.59 | bwd_allreduce: 29.98 | step: 18.99

  2%|▏         | 959/41250 [2:19:15<97:12:38,  8.69s/it]
                                                        
{'loss': 0.0564, 'grad_norm': 0.9113385081291199, 'learning_rate': 3.098546042003231e-05, 'epoch': 0.23}

  2%|▏         | 959/41250 [2:19:15<97:12:38,  8.69s/it][2025-04-25 10:16:59,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 10:16:59,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.43 | bwd_microstep: 5696.28 | bwd_inner_microstep: 5667.40 | bwd_allreduce_microstep: 28.84 | step_microstep: 18.83
[2025-04-25 10:16:59,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.43 | bwd: 5696.30 | bwd_inner: 5667.40 | bwd_allreduce: 28.86 | step: 18.84

  2%|▏         | 960/41250 [2:19:24<96:59:08,  8.67s/it]
                                                        
{'loss': 0.0596, 'grad_norm': 0.9469261765480042, 'learning_rate': 3.101777059773829e-05, 'epoch': 0.23}

  2%|▏         | 960/41250 [2:19:24<96:59:08,  8.67s/it][2025-04-25 10:17:07,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.27 | optimizer_step: 0.93
[2025-04-25 10:17:07,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.16 | bwd_microstep: 5730.97 | bwd_inner_microstep: 5658.73 | bwd_allreduce_microstep: 72.18 | step_microstep: 19.66
[2025-04-25 10:17:07,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.16 | bwd: 5730.98 | bwd_inner: 5658.73 | bwd_allreduce: 72.21 | step: 19.66

  2%|▏         | 961/41250 [2:19:33<96:56:00,  8.66s/it]
                                                        
{'loss': 0.3421, 'grad_norm': 3.464069128036499, 'learning_rate': 3.105008077544427e-05, 'epoch': 0.23}

  2%|▏         | 961/41250 [2:19:33<96:56:00,  8.66s/it][2025-04-25 10:17:16,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:17:16,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.69 | bwd_microstep: 5718.69 | bwd_inner_microstep: 5651.82 | bwd_allreduce_microstep: 66.82 | step_microstep: 18.57
[2025-04-25 10:17:16,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.69 | bwd: 5718.70 | bwd_inner: 5651.83 | bwd_allreduce: 66.84 | step: 18.58

  2%|▏         | 962/41250 [2:19:41<96:50:16,  8.65s/it]
                                                        
{'loss': 0.3063, 'grad_norm': 1.889841079711914, 'learning_rate': 3.108239095315024e-05, 'epoch': 0.23}

  2%|▏         | 962/41250 [2:19:41<96:50:16,  8.65s/it][2025-04-25 10:17:24,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:17:24,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.46 | bwd_microstep: 5718.66 | bwd_inner_microstep: 5705.81 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.89
[2025-04-25 10:17:24,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.46 | bwd: 5718.67 | bwd_inner: 5705.81 | bwd_allreduce: 12.82 | step: 18.89

  2%|▏         | 963/41250 [2:19:50<96:50:18,  8.65s/it]
                                                        
{'loss': 0.2104, 'grad_norm': 1.4800403118133545, 'learning_rate': 3.111470113085622e-05, 'epoch': 0.23}

  2%|▏         | 963/41250 [2:19:50<96:50:18,  8.65s/it][2025-04-25 10:17:33,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:17:33,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.90 | bwd_microstep: 5885.62 | bwd_inner_microstep: 5660.17 | bwd_allreduce_microstep: 225.40 | step_microstep: 19.39
[2025-04-25 10:17:33,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.90 | bwd: 5885.63 | bwd_inner: 5660.17 | bwd_allreduce: 225.42 | step: 19.39

  2%|▏         | 964/41250 [2:19:59<97:20:56,  8.70s/it]
                                                        
{'loss': 0.2456, 'grad_norm': 2.033440589904785, 'learning_rate': 3.11470113085622e-05, 'epoch': 0.23}

  2%|▏         | 964/41250 [2:19:59<97:20:56,  8.70s/it][2025-04-25 10:17:42,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:17:42,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5763.92 | bwd_inner_microstep: 5656.68 | bwd_allreduce_microstep: 107.19 | step_microstep: 18.77
[2025-04-25 10:17:42,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5763.93 | bwd_inner: 5656.68 | bwd_allreduce: 107.20 | step: 18.77

  2%|▏         | 965/41250 [2:20:07<97:16:50,  8.69s/it]
                                                        
{'loss': 0.1263, 'grad_norm': 1.4040714502334595, 'learning_rate': 3.1179321486268177e-05, 'epoch': 0.23}

  2%|▏         | 965/41250 [2:20:07<97:16:50,  8.69s/it][2025-04-25 10:17:51,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:17:51,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.22 | bwd_microstep: 5693.46 | bwd_inner_microstep: 5680.81 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.30
[2025-04-25 10:17:51,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.23 | bwd: 5693.48 | bwd_inner: 5680.81 | bwd_allreduce: 12.62 | step: 18.31

  2%|▏         | 966/41250 [2:20:16<97:02:24,  8.67s/it]
                                                        
{'loss': 0.4273, 'grad_norm': 2.661936044692993, 'learning_rate': 3.1211631663974154e-05, 'epoch': 0.23}

  2%|▏         | 966/41250 [2:20:16<97:02:24,  8.67s/it][2025-04-25 10:17:59,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 10:17:59,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5763.71 | bwd_inner_microstep: 5665.26 | bwd_allreduce_microstep: 98.40 | step_microstep: 18.67
[2025-04-25 10:17:59,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.16 | bwd: 5763.72 | bwd_inner: 5665.26 | bwd_allreduce: 98.42 | step: 18.67

  2%|▏         | 967/41250 [2:20:25<97:02:38,  8.67s/it]
                                                        
{'loss': 0.1493, 'grad_norm': 2.126112222671509, 'learning_rate': 3.124394184168013e-05, 'epoch': 0.23}

  2%|▏         | 967/41250 [2:20:25<97:02:38,  8.67s/it][2025-04-25 10:18:08,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.39 | optimizer_step: 0.94
[2025-04-25 10:18:08,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.04 | bwd_microstep: 5765.94 | bwd_inner_microstep: 5658.08 | bwd_allreduce_microstep: 107.81 | step_microstep: 19.01
[2025-04-25 10:18:08,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.04 | bwd: 5765.95 | bwd_inner: 5658.08 | bwd_allreduce: 107.83 | step: 19.01

  2%|▏         | 968/41250 [2:20:33<97:03:31,  8.67s/it]
                                                        
{'loss': 0.2351, 'grad_norm': 1.7604880332946777, 'learning_rate': 3.127625201938611e-05, 'epoch': 0.23}

  2%|▏         | 968/41250 [2:20:33<97:03:31,  8.67s/it][2025-04-25 10:18:17,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 10:18:17,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.76 | bwd_microstep: 5863.54 | bwd_inner_microstep: 5641.80 | bwd_allreduce_microstep: 221.69 | step_microstep: 19.00
[2025-04-25 10:18:17,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.76 | bwd: 5863.55 | bwd_inner: 5641.80 | bwd_allreduce: 221.71 | step: 19.00

  2%|▏         | 969/41250 [2:20:42<97:24:01,  8.70s/it]
                                                        
{'loss': 0.1069, 'grad_norm': 1.3738977909088135, 'learning_rate': 3.130856219709209e-05, 'epoch': 0.23}

  2%|▏         | 969/41250 [2:20:42<97:24:01,  8.70s/it][2025-04-25 10:18:25,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 10:18:25,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.10 | bwd_microstep: 5762.13 | bwd_inner_microstep: 5648.68 | bwd_allreduce_microstep: 113.41 | step_microstep: 18.88
[2025-04-25 10:18:25,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.10 | bwd: 5762.14 | bwd_inner: 5648.68 | bwd_allreduce: 113.42 | step: 18.89

  2%|▏         | 970/41250 [2:20:51<97:17:34,  8.70s/it]
                                                        
{'loss': 0.1411, 'grad_norm': 1.4708679914474487, 'learning_rate': 3.1340872374798066e-05, 'epoch': 0.24}

  2%|▏         | 970/41250 [2:20:51<97:17:34,  8.70s/it][2025-04-25 10:18:34,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[2025-04-25 10:18:34,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.70 | bwd_microstep: 5760.12 | bwd_inner_microstep: 5676.67 | bwd_allreduce_microstep: 83.40 | step_microstep: 19.13
[2025-04-25 10:18:34,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.70 | bwd: 5760.13 | bwd_inner: 5676.67 | bwd_allreduce: 83.42 | step: 19.13

  2%|▏         | 971/41250 [2:20:59<97:16:21,  8.69s/it]
                                                        
{'loss': 0.0562, 'grad_norm': 1.3265211582183838, 'learning_rate': 3.1373182552504044e-05, 'epoch': 0.24}

  2%|▏         | 971/41250 [2:20:59<97:16:21,  8.69s/it][2025-04-25 10:18:43,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:18:43,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.40 | bwd_microstep: 5737.41 | bwd_inner_microstep: 5653.33 | bwd_allreduce_microstep: 84.04 | step_microstep: 18.32
[2025-04-25 10:18:43,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.40 | bwd: 5737.42 | bwd_inner: 5653.33 | bwd_allreduce: 84.06 | step: 18.33

  2%|▏         | 972/41250 [2:21:08<97:06:59,  8.68s/it]
                                                        
{'loss': 0.1182, 'grad_norm': 1.5984224081039429, 'learning_rate': 3.1405492730210015e-05, 'epoch': 0.24}

  2%|▏         | 972/41250 [2:21:08<97:06:59,  8.68s/it][2025-04-25 10:18:51,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:18:51,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.52 | bwd_microstep: 5757.17 | bwd_inner_microstep: 5697.43 | bwd_allreduce_microstep: 59.70 | step_microstep: 18.66
[2025-04-25 10:18:51,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.52 | bwd: 5757.19 | bwd_inner: 5697.43 | bwd_allreduce: 59.72 | step: 18.66

  2%|▏         | 973/41250 [2:21:17<97:08:48,  8.68s/it]
                                                        
{'loss': 0.0848, 'grad_norm': 0.8819294571876526, 'learning_rate': 3.143780290791599e-05, 'epoch': 0.24}

  2%|▏         | 973/41250 [2:21:17<97:08:48,  8.68s/it][2025-04-25 10:19:00,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 10:19:00,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 5882.39 | bwd_inner_microstep: 5655.62 | bwd_allreduce_microstep: 226.72 | step_microstep: 18.72
[2025-04-25 10:19:00,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 5882.40 | bwd_inner: 5655.62 | bwd_allreduce: 226.74 | step: 18.72

  2%|▏         | 974/41250 [2:21:26<97:31:45,  8.72s/it]
                                                        
{'loss': 0.1737, 'grad_norm': 2.025632381439209, 'learning_rate': 3.147011308562197e-05, 'epoch': 0.24}

  2%|▏         | 974/41250 [2:21:26<97:31:45,  8.72s/it][2025-04-25 10:19:09,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:19:09,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.62 | bwd_microstep: 5785.77 | bwd_inner_microstep: 5773.12 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.62
[2025-04-25 10:19:09,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.62 | bwd: 5785.78 | bwd_inner: 5773.12 | bwd_allreduce: 12.62 | step: 18.62

  2%|▏         | 975/41250 [2:21:34<97:39:01,  8.73s/it]
                                                        
{'loss': 0.4207, 'grad_norm': 2.494709014892578, 'learning_rate': 3.1502423263327955e-05, 'epoch': 0.24}

  2%|▏         | 975/41250 [2:21:34<97:39:01,  8.73s/it][2025-04-25 10:19:18,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-25 10:19:18,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5865.15 | bwd_inner_microstep: 5678.69 | bwd_allreduce_microstep: 186.40 | step_microstep: 19.32
[2025-04-25 10:19:18,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5865.16 | bwd_inner: 5678.69 | bwd_allreduce: 186.43 | step: 19.32

  2%|▏         | 976/41250 [2:21:43<97:51:58,  8.75s/it]
                                                        
{'loss': 0.1555, 'grad_norm': 1.2703819274902344, 'learning_rate': 3.1534733441033927e-05, 'epoch': 0.24}

  2%|▏         | 976/41250 [2:21:43<97:51:58,  8.75s/it][2025-04-25 10:19:26,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 10:19:26,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5709.44 | bwd_inner_microstep: 5696.57 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.81
[2025-04-25 10:19:26,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5709.45 | bwd_inner: 5696.57 | bwd_allreduce: 12.84 | step: 18.81

  2%|▏         | 977/41250 [2:21:52<97:31:14,  8.72s/it]
                                                        
{'loss': 0.1416, 'grad_norm': 1.2437199354171753, 'learning_rate': 3.1567043618739904e-05, 'epoch': 0.24}

  2%|▏         | 977/41250 [2:21:52<97:31:14,  8.72s/it][2025-04-25 10:19:35,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:19:35,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5758.25 | bwd_inner_microstep: 5652.15 | bwd_allreduce_microstep: 106.06 | step_microstep: 18.70
[2025-04-25 10:19:35,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5758.27 | bwd_inner: 5652.15 | bwd_allreduce: 106.07 | step: 18.70

  2%|▏         | 978/41250 [2:22:00<97:21:20,  8.70s/it]
                                                        
{'loss': 0.1484, 'grad_norm': 2.3906607627868652, 'learning_rate': 3.159935379644588e-05, 'epoch': 0.24}

  2%|▏         | 978/41250 [2:22:00<97:21:20,  8.70s/it][2025-04-25 10:19:44,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:19:44,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5708.86 | bwd_inner_microstep: 5651.82 | bwd_allreduce_microstep: 56.99 | step_microstep: 18.57
[2025-04-25 10:19:44,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.48 | bwd: 5708.87 | bwd_inner: 5651.82 | bwd_allreduce: 57.00 | step: 18.58

  2%|▏         | 979/41250 [2:22:09<97:04:00,  8.68s/it]
                                                        
{'loss': 0.1726, 'grad_norm': 1.3994580507278442, 'learning_rate': 3.163166397415186e-05, 'epoch': 0.24}

  2%|▏         | 979/41250 [2:22:09<97:04:00,  8.68s/it][2025-04-25 10:19:52,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:19:52,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5682.95 | bwd_inner_microstep: 5654.76 | bwd_allreduce_microstep: 28.14 | step_microstep: 18.86
[2025-04-25 10:19:52,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5682.96 | bwd_inner: 5654.76 | bwd_allreduce: 28.16 | step: 18.86

  2%|▏         | 980/41250 [2:22:18<96:46:32,  8.65s/it]
                                                        
{'loss': 0.2207, 'grad_norm': 2.003108263015747, 'learning_rate': 3.166397415185784e-05, 'epoch': 0.24}

  2%|▏         | 980/41250 [2:22:18<96:46:32,  8.65s/it][2025-04-25 10:20:01,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-25 10:20:01,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5696.84 | bwd_inner_microstep: 5684.17 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.39
[2025-04-25 10:20:01,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5696.85 | bwd_inner: 5684.17 | bwd_allreduce: 12.64 | step: 18.39

  2%|▏         | 981/41250 [2:22:26<96:41:42,  8.64s/it]
                                                        
{'loss': 0.1242, 'grad_norm': 1.1705868244171143, 'learning_rate': 3.1696284329563816e-05, 'epoch': 0.24}

  2%|▏         | 981/41250 [2:22:26<96:41:42,  8.64s/it][2025-04-25 10:20:10,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 10:20:10,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.59 | bwd_microstep: 5748.46 | bwd_inner_microstep: 5647.82 | bwd_allreduce_microstep: 100.59 | step_microstep: 19.11
[2025-04-25 10:20:10,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.59 | bwd: 5748.47 | bwd_inner: 5647.82 | bwd_allreduce: 100.60 | step: 19.12

  2%|▏         | 982/41250 [2:22:35<96:43:23,  8.65s/it]
                                                        
{'loss': 0.0389, 'grad_norm': 0.4773292541503906, 'learning_rate': 3.1728594507269794e-05, 'epoch': 0.24}

  2%|▏         | 982/41250 [2:22:35<96:43:23,  8.65s/it][2025-04-25 10:20:18,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:20:18,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.78 | bwd_microstep: 5769.13 | bwd_inner_microstep: 5641.45 | bwd_allreduce_microstep: 127.63 | step_microstep: 18.75
[2025-04-25 10:20:18,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.78 | bwd: 5769.14 | bwd_inner: 5641.45 | bwd_allreduce: 127.65 | step: 18.75

  2%|▏         | 983/41250 [2:22:44<96:48:15,  8.65s/it]
                                                        
{'loss': 0.3771, 'grad_norm': 2.8501999378204346, 'learning_rate': 3.176090468497577e-05, 'epoch': 0.24}

  2%|▏         | 983/41250 [2:22:44<96:48:15,  8.65s/it][2025-04-25 10:20:27,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:20:27,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.16 | bwd_microstep: 5688.01 | bwd_inner_microstep: 5636.43 | bwd_allreduce_microstep: 51.53 | step_microstep: 18.63
[2025-04-25 10:20:27,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.16 | bwd: 5688.03 | bwd_inner: 5636.43 | bwd_allreduce: 51.56 | step: 18.64

  2%|▏         | 984/41250 [2:22:52<96:35:11,  8.64s/it]
                                                        
{'loss': 0.2459, 'grad_norm': 2.5997071266174316, 'learning_rate': 3.179321486268175e-05, 'epoch': 0.24}

  2%|▏         | 984/41250 [2:22:52<96:35:11,  8.64s/it][2025-04-25 10:20:35,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:20:35,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.85 | bwd_microstep: 5701.41 | bwd_inner_microstep: 5640.02 | bwd_allreduce_microstep: 61.35 | step_microstep: 18.57
[2025-04-25 10:20:35,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.85 | bwd: 5701.42 | bwd_inner: 5640.02 | bwd_allreduce: 61.36 | step: 18.57

  2%|▏         | 985/41250 [2:23:01<96:28:29,  8.63s/it]
                                                        
{'loss': 0.279, 'grad_norm': 2.631514310836792, 'learning_rate': 3.182552504038773e-05, 'epoch': 0.24}

  2%|▏         | 985/41250 [2:23:01<96:28:29,  8.63s/it][2025-04-25 10:20:44,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 10:20:44,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.88 | bwd_microstep: 5770.28 | bwd_inner_microstep: 5654.35 | bwd_allreduce_microstep: 115.89 | step_microstep: 18.62
[2025-04-25 10:20:44,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.88 | bwd: 5770.29 | bwd_inner: 5654.35 | bwd_allreduce: 115.91 | step: 18.63

  2%|▏         | 986/41250 [2:23:09<96:38:15,  8.64s/it]
                                                        
{'loss': 0.1875, 'grad_norm': 1.5361591577529907, 'learning_rate': 3.18578352180937e-05, 'epoch': 0.24}

  2%|▏         | 986/41250 [2:23:09<96:38:15,  8.64s/it][2025-04-25 10:20:53,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 10:20:53,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.52 | bwd_microstep: 5673.74 | bwd_inner_microstep: 5651.88 | bwd_allreduce_microstep: 21.81 | step_microstep: 19.39
[2025-04-25 10:20:53,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.52 | bwd: 5673.75 | bwd_inner: 5651.88 | bwd_allreduce: 21.83 | step: 19.39

  2%|▏         | 987/41250 [2:23:18<96:29:11,  8.63s/it]
                                                        
{'loss': 0.2807, 'grad_norm': 1.8435057401657104, 'learning_rate': 3.1890145395799676e-05, 'epoch': 0.24}

  2%|▏         | 987/41250 [2:23:18<96:29:11,  8.63s/it][2025-04-25 10:21:01,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 10:21:01,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 5740.11 | bwd_inner_microstep: 5640.62 | bwd_allreduce_microstep: 99.44 | step_microstep: 19.16
[2025-04-25 10:21:01,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 5740.12 | bwd_inner: 5640.62 | bwd_allreduce: 99.46 | step: 19.17

  2%|▏         | 988/41250 [2:23:27<96:33:26,  8.63s/it]
                                                        
{'loss': 0.2066, 'grad_norm': 1.2377676963806152, 'learning_rate': 3.1922455573505654e-05, 'epoch': 0.24}

  2%|▏         | 988/41250 [2:23:27<96:33:26,  8.63s/it][2025-04-25 10:21:10,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 10:21:10,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.44 | bwd_microstep: 5667.18 | bwd_inner_microstep: 5633.25 | bwd_allreduce_microstep: 33.88 | step_microstep: 18.82
[2025-04-25 10:21:10,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.44 | bwd: 5667.19 | bwd_inner: 5633.25 | bwd_allreduce: 33.90 | step: 18.82

  2%|▏         | 989/41250 [2:23:35<96:22:57,  8.62s/it]
                                                        
{'loss': 0.0802, 'grad_norm': 0.957713782787323, 'learning_rate': 3.195476575121164e-05, 'epoch': 0.24}

  2%|▏         | 989/41250 [2:23:35<96:22:57,  8.62s/it][2025-04-25 10:21:19,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 10:21:19,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.79 | bwd_microstep: 5744.08 | bwd_inner_microstep: 5664.26 | bwd_allreduce_microstep: 79.77 | step_microstep: 19.42
[2025-04-25 10:21:19,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.79 | bwd: 5744.09 | bwd_inner: 5664.26 | bwd_allreduce: 79.79 | step: 19.43

  2%|▏         | 990/41250 [2:23:44<96:34:01,  8.63s/it]
                                                        
{'loss': 0.1569, 'grad_norm': 1.789522647857666, 'learning_rate': 3.198707592891761e-05, 'epoch': 0.24}

  2%|▏         | 990/41250 [2:23:44<96:34:01,  8.63s/it][2025-04-25 10:21:27,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 10:21:27,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.24 | bwd_microstep: 5748.81 | bwd_inner_microstep: 5635.88 | bwd_allreduce_microstep: 112.90 | step_microstep: 18.33
[2025-04-25 10:21:27,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.25 | bwd: 5748.83 | bwd_inner: 5635.88 | bwd_allreduce: 112.91 | step: 18.33

  2%|▏         | 991/41250 [2:23:53<96:37:08,  8.64s/it]
                                                        
{'loss': 0.3417, 'grad_norm': 2.611970901489258, 'learning_rate': 3.201938610662359e-05, 'epoch': 0.24}

  2%|▏         | 991/41250 [2:23:53<96:37:08,  8.64s/it][2025-04-25 10:21:36,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 10:21:36,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.91 | bwd_microstep: 5706.46 | bwd_inner_microstep: 5693.40 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.92
[2025-04-25 10:21:36,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.91 | bwd: 5706.47 | bwd_inner: 5693.40 | bwd_allreduce: 13.03 | step: 18.92

  2%|▏         | 992/41250 [2:24:01<96:36:46,  8.64s/it]
                                                        
{'loss': 0.1907, 'grad_norm': 3.4074432849884033, 'learning_rate': 3.2051696284329566e-05, 'epoch': 0.24}

  2%|▏         | 992/41250 [2:24:01<96:36:46,  8.64s/it][2025-04-25 10:21:45,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:21:45,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.56 | bwd_microstep: 5679.61 | bwd_inner_microstep: 5666.08 | bwd_allreduce_microstep: 13.47 | step_microstep: 18.85
[2025-04-25 10:21:45,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5679.62 | bwd_inner: 5666.08 | bwd_allreduce: 13.49 | step: 18.85

  2%|▏         | 993/41250 [2:24:10<96:30:27,  8.63s/it]
                                                        
{'loss': 0.2998, 'grad_norm': 2.271270990371704, 'learning_rate': 3.2084006462035544e-05, 'epoch': 0.24}

  2%|▏         | 993/41250 [2:24:10<96:30:27,  8.63s/it][2025-04-25 10:21:53,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 10:21:53,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5840.90 | bwd_inner_microstep: 5635.42 | bwd_allreduce_microstep: 205.42 | step_microstep: 19.18
[2025-04-25 10:21:53,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5840.91 | bwd_inner: 5635.42 | bwd_allreduce: 205.44 | step: 19.18

  2%|▏         | 994/41250 [2:24:19<96:54:23,  8.67s/it]
                                                        
{'loss': 0.1731, 'grad_norm': 2.3010053634643555, 'learning_rate': 3.211631663974152e-05, 'epoch': 0.24}

  2%|▏         | 994/41250 [2:24:19<96:54:23,  8.67s/it][2025-04-25 10:22:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 10:22:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.32 | bwd_microstep: 5977.01 | bwd_inner_microstep: 5675.25 | bwd_allreduce_microstep: 301.71 | step_microstep: 18.68
[2025-04-25 10:22:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.32 | bwd: 5977.02 | bwd_inner: 5675.25 | bwd_allreduce: 301.73 | step: 18.69

  2%|▏         | 995/41250 [2:24:28<97:43:34,  8.74s/it]
                                                        
{'loss': 0.1537, 'grad_norm': 1.294623851776123, 'learning_rate': 3.21486268174475e-05, 'epoch': 0.24}

  2%|▏         | 995/41250 [2:24:28<97:43:34,  8.74s/it][2025-04-25 10:22:11,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:22:11,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.52 | bwd_microstep: 5714.02 | bwd_inner_microstep: 5668.49 | bwd_allreduce_microstep: 45.49 | step_microstep: 18.71
[2025-04-25 10:22:11,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.52 | bwd: 5714.03 | bwd_inner: 5668.49 | bwd_allreduce: 45.51 | step: 18.71

  2%|▏         | 996/41250 [2:24:36<97:24:19,  8.71s/it]
                                                        
{'loss': 0.3639, 'grad_norm': 2.4140231609344482, 'learning_rate': 3.218093699515348e-05, 'epoch': 0.24}

  2%|▏         | 996/41250 [2:24:36<97:24:19,  8.71s/it][2025-04-25 10:22:20,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:22:20,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.11 | bwd_microstep: 5781.37 | bwd_inner_microstep: 5768.93 | bwd_allreduce_microstep: 12.40 | step_microstep: 18.48
[2025-04-25 10:22:20,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.11 | bwd: 5781.39 | bwd_inner: 5768.93 | bwd_allreduce: 12.41 | step: 18.49

  2%|▏         | 997/41250 [2:24:45<97:34:03,  8.73s/it]
                                                        
{'loss': 0.2713, 'grad_norm': 2.736741781234741, 'learning_rate': 3.2213247172859455e-05, 'epoch': 0.24}

  2%|▏         | 997/41250 [2:24:45<97:34:03,  8.73s/it][2025-04-25 10:22:28,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.06 | optimizer_step: 1.18
[2025-04-25 10:22:28,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.85 | bwd_microstep: 5758.69 | bwd_inner_microstep: 5745.31 | bwd_allreduce_microstep: 13.31 | step_microstep: 19.54
[2025-04-25 10:22:28,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.85 | bwd: 5758.70 | bwd_inner: 5745.31 | bwd_allreduce: 13.34 | step: 19.55

  2%|▏         | 998/41250 [2:24:54<97:33:23,  8.73s/it]
                                                        
{'loss': 0.1418, 'grad_norm': 1.213571310043335, 'learning_rate': 3.224555735056543e-05, 'epoch': 0.24}

  2%|▏         | 998/41250 [2:24:54<97:33:23,  8.73s/it][2025-04-25 10:22:37,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 10:22:37,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2871.54 | bwd_microstep: 5753.92 | bwd_inner_microstep: 5740.83 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.13
[2025-04-25 10:22:37,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2871.54 | bwd: 5753.94 | bwd_inner: 5740.83 | bwd_allreduce: 13.07 | step: 19.13

  2%|▏         | 999/41250 [2:25:02<97:31:03,  8.72s/it]
                                                        
{'loss': 0.1932, 'grad_norm': 1.759372353553772, 'learning_rate': 3.227786752827141e-05, 'epoch': 0.24}

  2%|▏         | 999/41250 [2:25:02<97:31:03,  8.72s/it][2025-04-25 10:22:46,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:22:46,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.13 | bwd_microstep: 5756.17 | bwd_inner_microstep: 5631.83 | bwd_allreduce_microstep: 124.30 | step_microstep: 18.73
[2025-04-25 10:22:46,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.13 | bwd: 5756.18 | bwd_inner: 5631.83 | bwd_allreduce: 124.32 | step: 18.73

  2%|▏         | 1000/41250 [2:25:11<97:19:24,  8.70s/it]
                                                         
{'loss': 0.2227, 'grad_norm': 2.440979242324829, 'learning_rate': 3.231017770597738e-05, 'epoch': 0.24}

  2%|▏         | 1000/41250 [2:25:11<97:19:24,  8.70s/it][2025-04-25 10:22:54,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 10:22:54,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.00 | bwd_microstep: 5766.77 | bwd_inner_microstep: 5632.55 | bwd_allreduce_microstep: 134.17 | step_microstep: 18.68
[2025-04-25 10:22:54,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.00 | bwd: 5766.78 | bwd_inner: 5632.55 | bwd_allreduce: 134.19 | step: 18.68

  2%|▏         | 1001/41250 [2:25:20<97:12:25,  8.69s/it]
                                                         
{'loss': 0.1422, 'grad_norm': 1.2733523845672607, 'learning_rate': 3.234248788368336e-05, 'epoch': 0.24}

  2%|▏         | 1001/41250 [2:25:20<97:12:25,  8.69s/it][2025-04-25 10:23:03,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:23:03,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.05 | bwd_microstep: 5704.76 | bwd_inner_microstep: 5638.99 | bwd_allreduce_microstep: 65.73 | step_microstep: 18.80
[2025-04-25 10:23:03,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.05 | bwd: 5704.78 | bwd_inner: 5638.99 | bwd_allreduce: 65.75 | step: 18.80

  2%|▏         | 1002/41250 [2:25:28<96:56:11,  8.67s/it]
                                                         
{'loss': 0.1779, 'grad_norm': 2.131108522415161, 'learning_rate': 3.237479806138934e-05, 'epoch': 0.24}

  2%|▏         | 1002/41250 [2:25:28<96:56:11,  8.67s/it][2025-04-25 10:23:12,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:23:12,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.77 | bwd_microstep: 5696.40 | bwd_inner_microstep: 5683.62 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.42
[2025-04-25 10:23:12,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.77 | bwd: 5696.41 | bwd_inner: 5683.63 | bwd_allreduce: 12.75 | step: 18.42

  2%|▏         | 1003/41250 [2:25:37<96:48:16,  8.66s/it]
                                                         
{'loss': 0.134, 'grad_norm': 2.308905839920044, 'learning_rate': 3.240710823909532e-05, 'epoch': 0.24}

  2%|▏         | 1003/41250 [2:25:37<96:48:16,  8.66s/it][2025-04-25 10:23:20,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 10:23:20,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5705.41 | bwd_inner_microstep: 5692.38 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.21
[2025-04-25 10:23:20,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5705.43 | bwd_inner: 5692.38 | bwd_allreduce: 13.01 | step: 19.21

  2%|▏         | 1004/41250 [2:25:46<96:44:59,  8.65s/it]
                                                         
{'loss': 0.1788, 'grad_norm': 1.7081879377365112, 'learning_rate': 3.2439418416801294e-05, 'epoch': 0.24}

  2%|▏         | 1004/41250 [2:25:46<96:44:59,  8.65s/it][2025-04-25 10:23:29,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 10:23:29,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.43 | bwd_microstep: 5703.24 | bwd_inner_microstep: 5690.50 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.67
[2025-04-25 10:23:29,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.43 | bwd: 5703.25 | bwd_inner: 5690.50 | bwd_allreduce: 12.71 | step: 18.68

  2%|▏         | 1005/41250 [2:25:54<96:39:50,  8.65s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 2.565828323364258, 'learning_rate': 3.247172859450727e-05, 'epoch': 0.24}

  2%|▏         | 1005/41250 [2:25:54<96:39:50,  8.65s/it][2025-04-25 10:23:38,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 10:23:38,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.63 | bwd_microstep: 5738.89 | bwd_inner_microstep: 5687.21 | bwd_allreduce_microstep: 51.64 | step_microstep: 18.83
[2025-04-25 10:23:38,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.63 | bwd: 5738.90 | bwd_inner: 5687.21 | bwd_allreduce: 51.65 | step: 18.83

  2%|▏         | 1006/41250 [2:26:03<96:43:53,  8.65s/it]
                                                         
{'loss': 0.2702, 'grad_norm': 2.86849308013916, 'learning_rate': 3.250403877221325e-05, 'epoch': 0.24}

  2%|▏         | 1006/41250 [2:26:03<96:43:53,  8.65s/it][2025-04-25 10:23:46,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:23:46,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.66 | bwd_microstep: 5699.52 | bwd_inner_microstep: 5681.96 | bwd_allreduce_microstep: 17.52 | step_microstep: 18.70
[2025-04-25 10:23:46,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.66 | bwd: 5699.53 | bwd_inner: 5681.96 | bwd_allreduce: 17.53 | step: 18.71

  2%|▏         | 1007/41250 [2:26:11<96:38:56,  8.65s/it]
                                                         
{'loss': 0.1789, 'grad_norm': 2.6342475414276123, 'learning_rate': 3.253634894991923e-05, 'epoch': 0.24}

  2%|▏         | 1007/41250 [2:26:12<96:38:56,  8.65s/it][2025-04-25 10:23:55,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:23:55,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.78 | bwd_microstep: 5689.57 | bwd_inner_microstep: 5649.22 | bwd_allreduce_microstep: 40.31 | step_microstep: 18.86
[2025-04-25 10:23:55,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.78 | bwd: 5689.59 | bwd_inner: 5649.22 | bwd_allreduce: 40.33 | step: 18.86

  2%|▏         | 1008/41250 [2:26:20<96:29:26,  8.63s/it]
                                                         
{'loss': 0.1365, 'grad_norm': 1.4921343326568604, 'learning_rate': 3.2568659127625205e-05, 'epoch': 0.24}

  2%|▏         | 1008/41250 [2:26:20<96:29:26,  8.63s/it][2025-04-25 10:24:03,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.29 | optimizer_step: 1.06
[2025-04-25 10:24:03,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.79 | bwd_microstep: 5705.80 | bwd_inner_microstep: 5691.90 | bwd_allreduce_microstep: 13.83 | step_microstep: 20.11
[2025-04-25 10:24:03,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.79 | bwd: 5705.82 | bwd_inner: 5691.90 | bwd_allreduce: 13.86 | step: 20.11

  2%|▏         | 1009/41250 [2:26:29<96:30:44,  8.63s/it]
                                                         
{'loss': 0.1209, 'grad_norm': 2.0615317821502686, 'learning_rate': 3.260096930533118e-05, 'epoch': 0.24}

  2%|▏         | 1009/41250 [2:26:29<96:30:44,  8.63s/it][2025-04-25 10:24:12,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.09 | optimizer_step: 1.08
[2025-04-25 10:24:12,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.57 | bwd_microstep: 5892.12 | bwd_inner_microstep: 5653.49 | bwd_allreduce_microstep: 238.57 | step_microstep: 19.83
[2025-04-25 10:24:12,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.57 | bwd: 5892.13 | bwd_inner: 5653.49 | bwd_allreduce: 238.60 | step: 19.83

  2%|▏         | 1010/41250 [2:26:38<97:07:00,  8.69s/it]
                                                         
{'loss': 0.435, 'grad_norm': 2.761955738067627, 'learning_rate': 3.263327948303716e-05, 'epoch': 0.24}

  2%|▏         | 1010/41250 [2:26:38<97:07:00,  8.69s/it][2025-04-25 10:24:21,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.19 | optimizer_step: 0.96
[2025-04-25 10:24:21,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.42 | bwd_microstep: 5709.60 | bwd_inner_microstep: 5696.42 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.55
[2025-04-25 10:24:21,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.42 | bwd: 5709.61 | bwd_inner: 5696.42 | bwd_allreduce: 13.15 | step: 19.55

  2%|▏         | 1011/41250 [2:26:46<96:58:36,  8.68s/it]
                                                         
{'loss': 0.2535, 'grad_norm': 2.7865381240844727, 'learning_rate': 3.266558966074314e-05, 'epoch': 0.25}

  2%|▏         | 1011/41250 [2:26:46<96:58:36,  8.68s/it][2025-04-25 10:24:30,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 10:24:30,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.96 | bwd_microstep: 5797.12 | bwd_inner_microstep: 5784.19 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.95
[2025-04-25 10:24:30,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.96 | bwd: 5797.13 | bwd_inner: 5784.19 | bwd_allreduce: 12.90 | step: 18.95

  2%|▏         | 1012/41250 [2:26:55<97:18:02,  8.71s/it]
                                                         
{'loss': 0.2148, 'grad_norm': 1.8631478548049927, 'learning_rate': 3.269789983844912e-05, 'epoch': 0.25}

  2%|▏         | 1012/41250 [2:26:55<97:18:02,  8.71s/it][2025-04-25 10:24:38,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 10:24:38,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.38 | bwd_microstep: 5722.55 | bwd_inner_microstep: 5709.62 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.02
[2025-04-25 10:24:38,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.38 | bwd: 5722.56 | bwd_inner: 5709.62 | bwd_allreduce: 12.90 | step: 19.02

  2%|▏         | 1013/41250 [2:27:04<97:08:04,  8.69s/it]
                                                         
{'loss': 0.108, 'grad_norm': 1.633943796157837, 'learning_rate': 3.2730210016155095e-05, 'epoch': 0.25}

  2%|▏         | 1013/41250 [2:27:04<97:08:04,  8.69s/it][2025-04-25 10:24:47,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:24:47,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5732.81 | bwd_inner_microstep: 5657.54 | bwd_allreduce_microstep: 75.21 | step_microstep: 19.07
[2025-04-25 10:24:47,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.69 | bwd: 5732.82 | bwd_inner: 5657.54 | bwd_allreduce: 75.24 | step: 19.07

  2%|▏         | 1014/41250 [2:27:12<96:59:05,  8.68s/it]
                                                         
{'loss': 0.2098, 'grad_norm': 2.4737348556518555, 'learning_rate': 3.2762520193861066e-05, 'epoch': 0.25}

  2%|▏         | 1014/41250 [2:27:12<96:59:05,  8.68s/it][2025-04-25 10:24:56,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 10:24:56,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.15 | bwd_microstep: 5728.91 | bwd_inner_microstep: 5655.25 | bwd_allreduce_microstep: 73.62 | step_microstep: 19.44
[2025-04-25 10:24:56,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.15 | bwd: 5728.93 | bwd_inner: 5655.25 | bwd_allreduce: 73.64 | step: 19.45

  2%|▏         | 1015/41250 [2:27:21<96:53:11,  8.67s/it]
                                                         
{'loss': 0.1768, 'grad_norm': 2.3316123485565186, 'learning_rate': 3.2794830371567044e-05, 'epoch': 0.25}

  2%|▏         | 1015/41250 [2:27:21<96:53:11,  8.67s/it][2025-04-25 10:25:04,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 10:25:04,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.55 | bwd_microstep: 5773.85 | bwd_inner_microstep: 5672.04 | bwd_allreduce_microstep: 101.76 | step_microstep: 19.06
[2025-04-25 10:25:04,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.55 | bwd: 5773.86 | bwd_inner: 5672.04 | bwd_allreduce: 101.79 | step: 19.07

  2%|▏         | 1016/41250 [2:27:30<96:58:06,  8.68s/it]
                                                         
{'loss': 0.2144, 'grad_norm': 1.6224710941314697, 'learning_rate': 3.282714054927302e-05, 'epoch': 0.25}

  2%|▏         | 1016/41250 [2:27:30<96:58:06,  8.68s/it][2025-04-25 10:25:13,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 10:25:13,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.42 | bwd_microstep: 5772.37 | bwd_inner_microstep: 5702.94 | bwd_allreduce_microstep: 69.38 | step_microstep: 18.43
[2025-04-25 10:25:13,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.42 | bwd: 5772.38 | bwd_inner: 5702.94 | bwd_allreduce: 69.40 | step: 18.43

  2%|▏         | 1017/41250 [2:27:38<97:05:59,  8.69s/it]
                                                         
{'loss': 0.1705, 'grad_norm': 1.2432886362075806, 'learning_rate': 3.2859450726979006e-05, 'epoch': 0.25}

  2%|▏         | 1017/41250 [2:27:38<97:05:59,  8.69s/it][2025-04-25 10:25:22,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:25:22,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.33 | bwd_microstep: 5798.19 | bwd_inner_microstep: 5655.84 | bwd_allreduce_microstep: 142.30 | step_microstep: 18.60
[2025-04-25 10:25:22,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.33 | bwd: 5798.20 | bwd_inner: 5655.84 | bwd_allreduce: 142.32 | step: 18.60

  2%|▏         | 1018/41250 [2:27:47<97:12:16,  8.70s/it]
                                                         
{'loss': 0.2479, 'grad_norm': 2.1745550632476807, 'learning_rate': 3.289176090468498e-05, 'epoch': 0.25}

  2%|▏         | 1018/41250 [2:27:47<97:12:16,  8.70s/it][2025-04-25 10:25:30,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:25:30,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.80 | bwd_microstep: 5728.87 | bwd_inner_microstep: 5645.67 | bwd_allreduce_microstep: 83.16 | step_microstep: 18.69
[2025-04-25 10:25:30,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.80 | bwd: 5728.88 | bwd_inner: 5645.67 | bwd_allreduce: 83.17 | step: 18.69

  2%|▏         | 1019/41250 [2:27:56<96:59:51,  8.68s/it]
                                                         
{'loss': 0.1666, 'grad_norm': 1.9618817567825317, 'learning_rate': 3.2924071082390955e-05, 'epoch': 0.25}

  2%|▏         | 1019/41250 [2:27:56<96:59:51,  8.68s/it][2025-04-25 10:25:39,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:25:39,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.23 | bwd_microstep: 5910.62 | bwd_inner_microstep: 5659.45 | bwd_allreduce_microstep: 251.13 | step_microstep: 18.48
[2025-04-25 10:25:39,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.23 | bwd: 5910.64 | bwd_inner: 5659.45 | bwd_allreduce: 251.14 | step: 18.49

  2%|▏         | 1020/41250 [2:28:05<97:28:03,  8.72s/it]
                                                         
{'loss': 0.3574, 'grad_norm': 1.8820040225982666, 'learning_rate': 3.295638126009693e-05, 'epoch': 0.25}

  2%|▏         | 1020/41250 [2:28:05<97:28:03,  8.72s/it][2025-04-25 10:25:48,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:25:48,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.78 | bwd_microstep: 5782.72 | bwd_inner_microstep: 5695.18 | bwd_allreduce_microstep: 87.50 | step_microstep: 18.58
[2025-04-25 10:25:48,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.78 | bwd: 5782.74 | bwd_inner: 5695.18 | bwd_allreduce: 87.52 | step: 18.58

  2%|▏         | 1021/41250 [2:28:13<97:26:18,  8.72s/it]
                                                         
{'loss': 0.2389, 'grad_norm': 1.9981598854064941, 'learning_rate': 3.298869143780291e-05, 'epoch': 0.25}

  2%|▏         | 1021/41250 [2:28:13<97:26:18,  8.72s/it][2025-04-25 10:25:57,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:25:57,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.27 | bwd_microstep: 5751.37 | bwd_inner_microstep: 5705.66 | bwd_allreduce_microstep: 45.67 | step_microstep: 18.89
[2025-04-25 10:25:57,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.27 | bwd: 5751.39 | bwd_inner: 5705.66 | bwd_allreduce: 45.69 | step: 18.90

  2%|▏         | 1022/41250 [2:28:22<97:19:06,  8.71s/it]
                                                         
{'loss': 0.1863, 'grad_norm': 2.1237192153930664, 'learning_rate': 3.302100161550889e-05, 'epoch': 0.25}

  2%|▏         | 1022/41250 [2:28:22<97:19:06,  8.71s/it][2025-04-25 10:26:05,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 10:26:05,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.50 | bwd_microstep: 5771.00 | bwd_inner_microstep: 5658.88 | bwd_allreduce_microstep: 112.07 | step_microstep: 18.69
[2025-04-25 10:26:05,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.50 | bwd: 5771.01 | bwd_inner: 5658.88 | bwd_allreduce: 112.09 | step: 18.70

  2%|▏         | 1023/41250 [2:28:31<97:14:05,  8.70s/it]
                                                         
{'loss': 0.3213, 'grad_norm': 3.364278554916382, 'learning_rate': 3.305331179321487e-05, 'epoch': 0.25}

  2%|▏         | 1023/41250 [2:28:31<97:14:05,  8.70s/it][2025-04-25 10:26:14,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:26:14,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.20 | bwd_microstep: 5708.44 | bwd_inner_microstep: 5651.65 | bwd_allreduce_microstep: 56.73 | step_microstep: 18.77
[2025-04-25 10:26:14,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.20 | bwd: 5708.46 | bwd_inner: 5651.65 | bwd_allreduce: 56.75 | step: 18.77

  2%|▏         | 1024/41250 [2:28:39<97:01:47,  8.68s/it]
                                                         
{'loss': 0.346, 'grad_norm': 2.143535614013672, 'learning_rate': 3.3085621970920845e-05, 'epoch': 0.25}

  2%|▏         | 1024/41250 [2:28:39<97:01:47,  8.68s/it][2025-04-25 10:26:23,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 10:26:23,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.75 | bwd_microstep: 5712.52 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 51.52 | step_microstep: 19.32
[2025-04-25 10:26:23,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.75 | bwd: 5712.53 | bwd_inner: 5660.93 | bwd_allreduce: 51.56 | step: 19.32

  2%|▏         | 1025/41250 [2:28:48<96:50:01,  8.67s/it]
                                                         
{'loss': 0.2717, 'grad_norm': 2.835803508758545, 'learning_rate': 3.3117932148626816e-05, 'epoch': 0.25}

  2%|▏         | 1025/41250 [2:28:48<96:50:01,  8.67s/it][2025-04-25 10:26:31,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 10:26:31,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.79 | bwd_microstep: 5744.49 | bwd_inner_microstep: 5694.70 | bwd_allreduce_microstep: 49.74 | step_microstep: 18.58
[2025-04-25 10:26:31,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.79 | bwd: 5744.50 | bwd_inner: 5694.70 | bwd_allreduce: 49.76 | step: 18.58

  2%|▏         | 1026/41250 [2:28:57<96:52:45,  8.67s/it]
                                                         
{'loss': 0.1067, 'grad_norm': 1.0753639936447144, 'learning_rate': 3.31502423263328e-05, 'epoch': 0.25}

  2%|▏         | 1026/41250 [2:28:57<96:52:45,  8.67s/it][2025-04-25 10:26:40,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:26:40,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.91 | bwd_microstep: 5760.90 | bwd_inner_microstep: 5687.90 | bwd_allreduce_microstep: 72.95 | step_microstep: 18.16
[2025-04-25 10:26:40,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.91 | bwd: 5760.91 | bwd_inner: 5687.90 | bwd_allreduce: 72.96 | step: 18.16

  2%|▏         | 1027/41250 [2:29:05<96:58:07,  8.68s/it]
                                                         
{'loss': 0.3269, 'grad_norm': 3.417356252670288, 'learning_rate': 3.318255250403878e-05, 'epoch': 0.25}

  2%|▏         | 1027/41250 [2:29:05<96:58:07,  8.68s/it][2025-04-25 10:26:49,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-25 10:26:49,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.46 | bwd_microstep: 5737.36 | bwd_inner_microstep: 5708.13 | bwd_allreduce_microstep: 29.18 | step_microstep: 19.01
[2025-04-25 10:26:49,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.46 | bwd: 5737.37 | bwd_inner: 5708.13 | bwd_allreduce: 29.20 | step: 19.02

  2%|▏         | 1028/41250 [2:29:14<97:01:02,  8.68s/it]
                                                         
{'loss': 0.1916, 'grad_norm': 2.3798978328704834, 'learning_rate': 3.321486268174475e-05, 'epoch': 0.25}

  2%|▏         | 1028/41250 [2:29:14<97:01:02,  8.68s/it][2025-04-25 10:26:57,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:26:57,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.03 | bwd_microstep: 5791.73 | bwd_inner_microstep: 5779.02 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.72
[2025-04-25 10:26:57,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.03 | bwd: 5791.74 | bwd_inner: 5779.02 | bwd_allreduce: 12.68 | step: 18.72

  2%|▏         | 1029/41250 [2:29:23<97:18:57,  8.71s/it]
                                                         
{'loss': 0.083, 'grad_norm': 1.0060118436813354, 'learning_rate': 3.324717285945073e-05, 'epoch': 0.25}

  2%|▏         | 1029/41250 [2:29:23<97:18:57,  8.71s/it][2025-04-25 10:27:06,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 10:27:06,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.17 | bwd_microstep: 5714.91 | bwd_inner_microstep: 5649.73 | bwd_allreduce_microstep: 65.13 | step_microstep: 19.30
[2025-04-25 10:27:06,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.17 | bwd: 5714.92 | bwd_inner: 5649.73 | bwd_allreduce: 65.14 | step: 19.30

  2%|▏         | 1030/41250 [2:29:31<97:01:01,  8.68s/it]
                                                         
{'loss': 0.0977, 'grad_norm': 1.0909626483917236, 'learning_rate': 3.3279483037156705e-05, 'epoch': 0.25}

  2%|▏         | 1030/41250 [2:29:31<97:01:01,  8.68s/it][2025-04-25 10:27:15,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 0.98
[2025-04-25 10:27:15,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.59 | bwd_microstep: 5849.49 | bwd_inner_microstep: 5694.81 | bwd_allreduce_microstep: 154.63 | step_microstep: 19.13
[2025-04-25 10:27:15,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.59 | bwd: 5849.50 | bwd_inner: 5694.81 | bwd_allreduce: 154.65 | step: 19.13

  2%|▏         | 1031/41250 [2:29:40<97:21:53,  8.72s/it]
                                                         
{'loss': 0.1264, 'grad_norm': 1.4769201278686523, 'learning_rate': 3.331179321486269e-05, 'epoch': 0.25}

  2%|▏         | 1031/41250 [2:29:40<97:21:53,  8.72s/it][2025-04-25 10:27:23,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:27:23,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.26 | bwd_microstep: 5714.48 | bwd_inner_microstep: 5648.90 | bwd_allreduce_microstep: 65.53 | step_microstep: 18.55
[2025-04-25 10:27:23,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.26 | bwd: 5714.49 | bwd_inner: 5648.90 | bwd_allreduce: 65.55 | step: 18.55

  3%|▎         | 1032/41250 [2:29:49<97:03:05,  8.69s/it]
                                                         
{'loss': 0.1669, 'grad_norm': 1.0954012870788574, 'learning_rate': 3.334410339256866e-05, 'epoch': 0.25}

  3%|▎         | 1032/41250 [2:29:49<97:03:05,  8.69s/it][2025-04-25 10:27:32,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:27:32,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.40 | bwd_microstep: 5703.95 | bwd_inner_microstep: 5644.62 | bwd_allreduce_microstep: 59.28 | step_microstep: 18.63
[2025-04-25 10:27:32,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.40 | bwd: 5703.96 | bwd_inner: 5644.62 | bwd_allreduce: 59.30 | step: 18.63

  3%|▎         | 1033/41250 [2:29:57<96:47:59,  8.66s/it]
                                                         
{'loss': 0.318, 'grad_norm': 1.7487379312515259, 'learning_rate': 3.337641357027464e-05, 'epoch': 0.25}

  3%|▎         | 1033/41250 [2:29:57<96:47:59,  8.66s/it][2025-04-25 10:27:41,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 10:27:41,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.63 | bwd_microstep: 5788.37 | bwd_inner_microstep: 5651.57 | bwd_allreduce_microstep: 136.75 | step_microstep: 17.96
[2025-04-25 10:27:41,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.63 | bwd: 5788.38 | bwd_inner: 5651.57 | bwd_allreduce: 136.77 | step: 17.96

  3%|▎         | 1034/41250 [2:30:06<96:53:18,  8.67s/it]
                                                         
{'loss': 0.2319, 'grad_norm': 1.4856220483779907, 'learning_rate': 3.340872374798062e-05, 'epoch': 0.25}

  3%|▎         | 1034/41250 [2:30:06<96:53:18,  8.67s/it][2025-04-25 10:27:49,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:27:49,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.42 | bwd_microstep: 5710.92 | bwd_inner_microstep: 5641.54 | bwd_allreduce_microstep: 69.34 | step_microstep: 18.73
[2025-04-25 10:27:49,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.42 | bwd: 5710.94 | bwd_inner: 5641.54 | bwd_allreduce: 69.35 | step: 18.73

  3%|▎         | 1035/41250 [2:30:15<96:41:10,  8.66s/it]
                                                         
{'loss': 0.2222, 'grad_norm': 1.9415364265441895, 'learning_rate': 3.3441033925686595e-05, 'epoch': 0.25}

  3%|▎         | 1035/41250 [2:30:15<96:41:10,  8.66s/it][2025-04-25 10:27:58,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 10:27:58,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5699.58 | bwd_inner_microstep: 5686.70 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.32
[2025-04-25 10:27:58,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5699.59 | bwd_inner: 5686.69 | bwd_allreduce: 12.85 | step: 19.32

  3%|▎         | 1036/41250 [2:30:23<96:35:50,  8.65s/it]
                                                         
{'loss': 0.2741, 'grad_norm': 1.9332399368286133, 'learning_rate': 3.347334410339257e-05, 'epoch': 0.25}

  3%|▎         | 1036/41250 [2:30:23<96:35:50,  8.65s/it][2025-04-25 10:28:07,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.10 | optimizer_step: 0.96
[2025-04-25 10:28:07,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.97 | bwd_microstep: 5747.30 | bwd_inner_microstep: 5705.08 | bwd_allreduce_microstep: 42.16 | step_microstep: 19.65
[2025-04-25 10:28:07,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.97 | bwd: 5747.31 | bwd_inner: 5705.08 | bwd_allreduce: 42.19 | step: 19.65

  3%|▎         | 1037/41250 [2:30:32<96:42:32,  8.66s/it]
                                                         
{'loss': 0.138, 'grad_norm': 1.2530990839004517, 'learning_rate': 3.350565428109855e-05, 'epoch': 0.25}

  3%|▎         | 1037/41250 [2:30:32<96:42:32,  8.66s/it][2025-04-25 10:28:15,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:28:15,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.11 | bwd_microstep: 5766.10 | bwd_inner_microstep: 5753.08 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.81
[2025-04-25 10:28:15,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.11 | bwd: 5766.12 | bwd_inner: 5753.08 | bwd_allreduce: 12.99 | step: 18.81

  3%|▎         | 1038/41250 [2:30:41<96:56:41,  8.68s/it]
                                                         
{'loss': 0.2149, 'grad_norm': 1.5119980573654175, 'learning_rate': 3.353796445880453e-05, 'epoch': 0.25}

  3%|▎         | 1038/41250 [2:30:41<96:56:41,  8.68s/it][2025-04-25 10:28:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:28:24,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.27 | bwd_microstep: 5686.91 | bwd_inner_microstep: 5646.61 | bwd_allreduce_microstep: 40.25 | step_microstep: 18.81
[2025-04-25 10:28:24,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.28 | bwd: 5686.93 | bwd_inner: 5646.61 | bwd_allreduce: 40.27 | step: 18.81

  3%|▎         | 1039/41250 [2:30:49<96:39:53,  8.65s/it]
                                                         
{'loss': 0.1978, 'grad_norm': 2.3185646533966064, 'learning_rate': 3.35702746365105e-05, 'epoch': 0.25}

  3%|▎         | 1039/41250 [2:30:49<96:39:53,  8.65s/it][2025-04-25 10:28:33,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 10:28:33,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.69 | bwd_microstep: 5707.28 | bwd_inner_microstep: 5694.54 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.73
[2025-04-25 10:28:33,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.69 | bwd: 5707.30 | bwd_inner: 5694.54 | bwd_allreduce: 12.72 | step: 18.73

  3%|▎         | 1040/41250 [2:30:58<96:36:04,  8.65s/it]
                                                         
{'loss': 0.1074, 'grad_norm': 1.5685625076293945, 'learning_rate': 3.3602584814216484e-05, 'epoch': 0.25}

  3%|▎         | 1040/41250 [2:30:58<96:36:04,  8.65s/it][2025-04-25 10:28:41,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 10:28:41,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.25 | bwd_microstep: 5747.11 | bwd_inner_microstep: 5653.19 | bwd_allreduce_microstep: 93.86 | step_microstep: 19.33
[2025-04-25 10:28:41,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.25 | bwd: 5747.12 | bwd_inner: 5653.19 | bwd_allreduce: 93.88 | step: 19.33

  3%|▎         | 1041/41250 [2:31:07<96:37:52,  8.65s/it]
                                                         
{'loss': 0.2726, 'grad_norm': 2.0834271907806396, 'learning_rate': 3.363489499192246e-05, 'epoch': 0.25}

  3%|▎         | 1041/41250 [2:31:07<96:37:52,  8.65s/it][2025-04-25 10:28:50,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:28:50,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5707.68 | bwd_inner_microstep: 5695.12 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.74
[2025-04-25 10:28:50,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5707.69 | bwd_inner: 5695.12 | bwd_allreduce: 12.53 | step: 18.75

  3%|▎         | 1042/41250 [2:31:15<96:34:37,  8.65s/it]
                                                         
{'loss': 0.1619, 'grad_norm': 1.2955734729766846, 'learning_rate': 3.366720516962843e-05, 'epoch': 0.25}

  3%|▎         | 1042/41250 [2:31:15<96:34:37,  8.65s/it][2025-04-25 10:28:59,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:28:59,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.35 | bwd_microstep: 5742.64 | bwd_inner_microstep: 5685.00 | bwd_allreduce_microstep: 57.60 | step_microstep: 18.75
[2025-04-25 10:28:59,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.35 | bwd: 5742.66 | bwd_inner: 5685.00 | bwd_allreduce: 57.62 | step: 18.75

  3%|▎         | 1043/41250 [2:31:24<96:38:24,  8.65s/it]
                                                         
{'loss': 0.194, 'grad_norm': 1.5903793573379517, 'learning_rate': 3.369951534733441e-05, 'epoch': 0.25}

  3%|▎         | 1043/41250 [2:31:24<96:38:24,  8.65s/it][2025-04-25 10:29:07,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:29:07,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5702.17 | bwd_inner_microstep: 5689.52 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.57
[2025-04-25 10:29:07,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5702.19 | bwd_inner: 5689.52 | bwd_allreduce: 12.63 | step: 18.57

  3%|▎         | 1044/41250 [2:31:33<96:33:57,  8.65s/it]
                                                         
{'loss': 0.2887, 'grad_norm': 2.31343674659729, 'learning_rate': 3.373182552504039e-05, 'epoch': 0.25}

  3%|▎         | 1044/41250 [2:31:33<96:33:57,  8.65s/it][2025-04-25 10:29:16,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:29:16,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.20 | bwd_microstep: 5694.87 | bwd_inner_microstep: 5682.19 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.64
[2025-04-25 10:29:16,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.20 | bwd: 5694.88 | bwd_inner: 5682.19 | bwd_allreduce: 12.65 | step: 18.64

  3%|▎         | 1045/41250 [2:31:41<96:28:50,  8.64s/it]
                                                         
{'loss': 0.3742, 'grad_norm': 2.695979118347168, 'learning_rate': 3.3764135702746367e-05, 'epoch': 0.25}

  3%|▎         | 1045/41250 [2:31:41<96:28:50,  8.64s/it][2025-04-25 10:29:24,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:29:24,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.52 | bwd_microstep: 5688.85 | bwd_inner_microstep: 5650.33 | bwd_allreduce_microstep: 38.47 | step_microstep: 19.39
[2025-04-25 10:29:24,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.52 | bwd: 5688.86 | bwd_inner: 5650.33 | bwd_allreduce: 38.49 | step: 19.40

  3%|▎         | 1046/41250 [2:31:50<96:21:21,  8.63s/it]
                                                         
{'loss': 0.1512, 'grad_norm': 1.1887387037277222, 'learning_rate': 3.3796445880452344e-05, 'epoch': 0.25}

  3%|▎         | 1046/41250 [2:31:50<96:21:21,  8.63s/it][2025-04-25 10:29:33,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:29:33,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.02 | bwd_microstep: 5692.00 | bwd_inner_microstep: 5643.13 | bwd_allreduce_microstep: 48.83 | step_microstep: 18.71
[2025-04-25 10:29:33,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.02 | bwd: 5692.02 | bwd_inner: 5643.13 | bwd_allreduce: 48.84 | step: 18.71

  3%|▎         | 1047/41250 [2:31:58<96:14:58,  8.62s/it]
                                                         
{'loss': 0.1404, 'grad_norm': 1.347064733505249, 'learning_rate': 3.382875605815832e-05, 'epoch': 0.25}

  3%|▎         | 1047/41250 [2:31:58<96:14:58,  8.62s/it][2025-04-25 10:29:42,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:29:42,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.21 | bwd_microstep: 5694.96 | bwd_inner_microstep: 5643.23 | bwd_allreduce_microstep: 51.68 | step_microstep: 18.65
[2025-04-25 10:29:42,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.21 | bwd: 5694.97 | bwd_inner: 5643.23 | bwd_allreduce: 51.70 | step: 18.65

  3%|▎         | 1048/41250 [2:32:07<96:10:54,  8.61s/it]
                                                         
{'loss': 0.4177, 'grad_norm': 2.8555469512939453, 'learning_rate': 3.38610662358643e-05, 'epoch': 0.25}

  3%|▎         | 1048/41250 [2:32:07<96:10:54,  8.61s/it][2025-04-25 10:29:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 10:29:50,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.11 | bwd_microstep: 5836.96 | bwd_inner_microstep: 5648.65 | bwd_allreduce_microstep: 188.26 | step_microstep: 17.70
[2025-04-25 10:29:50,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.11 | bwd: 5836.97 | bwd_inner: 5648.65 | bwd_allreduce: 188.28 | step: 17.70

  3%|▎         | 1049/41250 [2:32:16<96:39:52,  8.66s/it]
                                                         
{'loss': 0.2698, 'grad_norm': 1.6824724674224854, 'learning_rate': 3.389337641357028e-05, 'epoch': 0.25}

  3%|▎         | 1049/41250 [2:32:16<96:39:52,  8.66s/it][2025-04-25 10:29:59,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 10:29:59,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5743.33 | bwd_inner_microstep: 5673.96 | bwd_allreduce_microstep: 69.32 | step_microstep: 18.71
[2025-04-25 10:29:59,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.48 | bwd: 5743.34 | bwd_inner: 5673.96 | bwd_allreduce: 69.34 | step: 18.71

  3%|▎         | 1050/41250 [2:32:24<96:43:16,  8.66s/it]
                                                         
{'loss': 0.2392, 'grad_norm': 1.9433820247650146, 'learning_rate': 3.3925686591276256e-05, 'epoch': 0.25}

  3%|▎         | 1050/41250 [2:32:24<96:43:16,  8.66s/it][2025-04-25 10:30:08,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 10:30:08,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5685.64 | bwd_inner_microstep: 5651.87 | bwd_allreduce_microstep: 33.73 | step_microstep: 18.82
[2025-04-25 10:30:08,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5685.66 | bwd_inner: 5651.87 | bwd_allreduce: 33.74 | step: 18.83

  3%|▎         | 1051/41250 [2:32:33<96:30:42,  8.64s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.0347288846969604, 'learning_rate': 3.3957996768982234e-05, 'epoch': 0.25}

  3%|▎         | 1051/41250 [2:32:33<96:30:42,  8.64s/it][2025-04-25 10:30:16,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:30:16,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.31 | bwd_microstep: 5740.25 | bwd_inner_microstep: 5670.67 | bwd_allreduce_microstep: 69.54 | step_microstep: 18.56
[2025-04-25 10:30:16,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.31 | bwd: 5740.26 | bwd_inner: 5670.66 | bwd_allreduce: 69.56 | step: 18.56

  3%|▎         | 1052/41250 [2:32:42<96:34:33,  8.65s/it]
                                                         
{'loss': 0.273, 'grad_norm': 1.6483328342437744, 'learning_rate': 3.3990306946688205e-05, 'epoch': 0.26}

  3%|▎         | 1052/41250 [2:32:42<96:34:33,  8.65s/it][2025-04-25 10:30:25,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:30:25,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.68 | bwd_microstep: 5686.20 | bwd_inner_microstep: 5650.65 | bwd_allreduce_microstep: 35.45 | step_microstep: 18.57
[2025-04-25 10:30:25,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.68 | bwd: 5686.21 | bwd_inner: 5650.65 | bwd_allreduce: 35.52 | step: 18.57

  3%|▎         | 1053/41250 [2:32:50<96:24:04,  8.63s/it]
                                                         
{'loss': 0.2591, 'grad_norm': 1.3172669410705566, 'learning_rate': 3.402261712439418e-05, 'epoch': 0.26}

  3%|▎         | 1053/41250 [2:32:50<96:24:04,  8.63s/it][2025-04-25 10:30:34,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:30:34,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.47 | bwd_microstep: 5684.60 | bwd_inner_microstep: 5628.90 | bwd_allreduce_microstep: 55.65 | step_microstep: 18.45
[2025-04-25 10:30:34,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.47 | bwd: 5684.61 | bwd_inner: 5628.90 | bwd_allreduce: 55.67 | step: 18.45

  3%|▎         | 1054/41250 [2:32:59<96:15:02,  8.62s/it]
                                                         
{'loss': 0.1474, 'grad_norm': 1.4958374500274658, 'learning_rate': 3.405492730210017e-05, 'epoch': 0.26}

  3%|▎         | 1054/41250 [2:32:59<96:15:02,  8.62s/it][2025-04-25 10:30:42,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.05 | optimizer_step: 1.18
[2025-04-25 10:30:42,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.92 | bwd_microstep: 6028.77 | bwd_inner_microstep: 5635.97 | bwd_allreduce_microstep: 392.75 | step_microstep: 19.66
[2025-04-25 10:30:42,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.92 | bwd: 6028.79 | bwd_inner: 5635.97 | bwd_allreduce: 392.77 | step: 19.66

  3%|▎         | 1055/41250 [2:33:08<97:18:29,  8.72s/it]
                                                         
{'loss': 0.3009, 'grad_norm': 2.7583632469177246, 'learning_rate': 3.4087237479806145e-05, 'epoch': 0.26}

  3%|▎         | 1055/41250 [2:33:08<97:18:29,  8.72s/it][2025-04-25 10:30:51,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 10:30:51,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.98 | bwd_microstep: 5745.47 | bwd_inner_microstep: 5678.39 | bwd_allreduce_microstep: 67.03 | step_microstep: 18.93
[2025-04-25 10:30:51,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.98 | bwd: 5745.49 | bwd_inner: 5678.39 | bwd_allreduce: 67.05 | step: 18.93

  3%|▎         | 1056/41250 [2:33:16<97:10:07,  8.70s/it]
                                                         
{'loss': 0.2157, 'grad_norm': 1.323136329650879, 'learning_rate': 3.4119547657512117e-05, 'epoch': 0.26}

  3%|▎         | 1056/41250 [2:33:16<97:10:07,  8.70s/it][2025-04-25 10:31:00,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:31:00,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5732.35 | bwd_inner_microstep: 5683.23 | bwd_allreduce_microstep: 49.07 | step_microstep: 18.86
[2025-04-25 10:31:00,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5732.37 | bwd_inner: 5683.23 | bwd_allreduce: 49.09 | step: 18.86

  3%|▎         | 1057/41250 [2:33:25<97:02:51,  8.69s/it]
                                                         
{'loss': 0.1946, 'grad_norm': 1.1439675092697144, 'learning_rate': 3.4151857835218094e-05, 'epoch': 0.26}

  3%|▎         | 1057/41250 [2:33:25<97:02:51,  8.69s/it][2025-04-25 10:31:09,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 10:31:09,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.78 | bwd_microstep: 5854.43 | bwd_inner_microstep: 5675.10 | bwd_allreduce_microstep: 179.28 | step_microstep: 18.93
[2025-04-25 10:31:09,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.78 | bwd: 5854.45 | bwd_inner: 5675.10 | bwd_allreduce: 179.30 | step: 18.93

  3%|▎         | 1058/41250 [2:33:34<97:19:54,  8.72s/it]
                                                         
{'loss': 0.3089, 'grad_norm': 2.8410537242889404, 'learning_rate': 3.418416801292407e-05, 'epoch': 0.26}

  3%|▎         | 1058/41250 [2:33:34<97:19:54,  8.72s/it][2025-04-25 10:31:17,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:31:17,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.59 | bwd_microstep: 5690.48 | bwd_inner_microstep: 5677.63 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.66
[2025-04-25 10:31:17,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.59 | bwd: 5690.49 | bwd_inner: 5677.63 | bwd_allreduce: 12.82 | step: 18.67

  3%|▎         | 1059/41250 [2:33:42<96:59:08,  8.69s/it]
                                                         
{'loss': 0.2237, 'grad_norm': 1.3422431945800781, 'learning_rate': 3.421647819063005e-05, 'epoch': 0.26}

  3%|▎         | 1059/41250 [2:33:43<96:59:08,  8.69s/it][2025-04-25 10:31:26,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.19 | optimizer_step: 1.12
[2025-04-25 10:31:26,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5692.69 | bwd_inner_microstep: 5638.22 | bwd_allreduce_microstep: 54.40 | step_microstep: 19.79
[2025-04-25 10:31:26,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5692.70 | bwd_inner: 5638.22 | bwd_allreduce: 54.44 | step: 19.79

  3%|▎         | 1060/41250 [2:33:51<96:43:39,  8.66s/it]
                                                         
{'loss': 0.1801, 'grad_norm': 1.7197390794754028, 'learning_rate': 3.424878836833603e-05, 'epoch': 0.26}

  3%|▎         | 1060/41250 [2:33:51<96:43:39,  8.66s/it][2025-04-25 10:31:34,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 10:31:34,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.09 | bwd_microstep: 5669.54 | bwd_inner_microstep: 5627.92 | bwd_allreduce_microstep: 41.56 | step_microstep: 18.93
[2025-04-25 10:31:34,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.09 | bwd: 5669.55 | bwd_inner: 5627.92 | bwd_allreduce: 41.58 | step: 18.93

  3%|▎         | 1061/41250 [2:34:00<96:26:11,  8.64s/it]
                                                         
{'loss': 0.1957, 'grad_norm': 1.7295178174972534, 'learning_rate': 3.4281098546042006e-05, 'epoch': 0.26}

  3%|▎         | 1061/41250 [2:34:00<96:26:11,  8.64s/it][2025-04-25 10:31:43,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 10:31:43,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.03 | bwd_microstep: 5731.49 | bwd_inner_microstep: 5680.08 | bwd_allreduce_microstep: 51.37 | step_microstep: 18.59
[2025-04-25 10:31:43,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.03 | bwd: 5731.50 | bwd_inner: 5680.08 | bwd_allreduce: 51.39 | step: 18.60

  3%|▎         | 1062/41250 [2:34:08<96:30:08,  8.64s/it]
                                                         
{'loss': 0.1522, 'grad_norm': 2.588329553604126, 'learning_rate': 3.4313408723747984e-05, 'epoch': 0.26}

  3%|▎         | 1062/41250 [2:34:08<96:30:08,  8.64s/it][2025-04-25 10:31:52,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:31:52,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.19 | bwd_microstep: 5697.55 | bwd_inner_microstep: 5632.94 | bwd_allreduce_microstep: 64.55 | step_microstep: 18.52
[2025-04-25 10:31:52,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.19 | bwd: 5697.56 | bwd_inner: 5632.94 | bwd_allreduce: 64.58 | step: 18.52

  3%|▎         | 1063/41250 [2:34:17<96:21:14,  8.63s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 1.3538447618484497, 'learning_rate': 3.434571890145396e-05, 'epoch': 0.26}

  3%|▎         | 1063/41250 [2:34:17<96:21:14,  8.63s/it][2025-04-25 10:32:00,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 10:32:00,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.59 | bwd_microstep: 5690.25 | bwd_inner_microstep: 5677.42 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.66
[2025-04-25 10:32:00,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.59 | bwd: 5690.26 | bwd_inner: 5677.42 | bwd_allreduce: 12.80 | step: 18.66

  3%|▎         | 1064/41250 [2:34:26<96:18:45,  8.63s/it]
                                                         
{'loss': 0.2696, 'grad_norm': 2.854586362838745, 'learning_rate': 3.437802907915994e-05, 'epoch': 0.26}

  3%|▎         | 1064/41250 [2:34:26<96:18:45,  8.63s/it][2025-04-25 10:32:09,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 10:32:09,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.24 | bwd_microstep: 5729.22 | bwd_inner_microstep: 5694.19 | bwd_allreduce_microstep: 34.99 | step_microstep: 19.05
[2025-04-25 10:32:09,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.24 | bwd: 5729.24 | bwd_inner: 5694.19 | bwd_allreduce: 35.01 | step: 19.05

  3%|▎         | 1065/41250 [2:34:34<96:26:03,  8.64s/it]
                                                         
{'loss': 0.0915, 'grad_norm': 0.892871618270874, 'learning_rate': 3.441033925686592e-05, 'epoch': 0.26}

  3%|▎         | 1065/41250 [2:34:34<96:26:03,  8.64s/it][2025-04-25 10:32:18,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 10:32:18,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5892.73 | bwd_inner_microstep: 5639.57 | bwd_allreduce_microstep: 253.11 | step_microstep: 18.70
[2025-04-25 10:32:18,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5892.75 | bwd_inner: 5639.57 | bwd_allreduce: 253.13 | step: 18.71

  3%|▎         | 1066/41250 [2:34:43<96:57:48,  8.69s/it]
                                                         
{'loss': 0.0775, 'grad_norm': 1.4624066352844238, 'learning_rate': 3.444264943457189e-05, 'epoch': 0.26}

  3%|▎         | 1066/41250 [2:34:43<96:57:48,  8.69s/it][2025-04-25 10:32:26,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:32:26,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.83 | bwd_microstep: 5699.90 | bwd_inner_microstep: 5650.87 | bwd_allreduce_microstep: 48.98 | step_microstep: 18.58
[2025-04-25 10:32:26,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.83 | bwd: 5699.91 | bwd_inner: 5650.87 | bwd_allreduce: 49.01 | step: 18.58

  3%|▎         | 1067/41250 [2:34:52<96:40:58,  8.66s/it]
                                                         
{'loss': 0.0828, 'grad_norm': 1.0442423820495605, 'learning_rate': 3.4474959612277866e-05, 'epoch': 0.26}

  3%|▎         | 1067/41250 [2:34:52<96:40:58,  8.66s/it][2025-04-25 10:32:35,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 10:32:35,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.79 | bwd_microstep: 5704.68 | bwd_inner_microstep: 5636.35 | bwd_allreduce_microstep: 68.28 | step_microstep: 18.72
[2025-04-25 10:32:35,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.79 | bwd: 5704.69 | bwd_inner: 5636.35 | bwd_allreduce: 68.30 | step: 18.73

  3%|▎         | 1068/41250 [2:35:00<96:30:49,  8.65s/it]
                                                         
{'loss': 0.3482, 'grad_norm': 2.652894973754883, 'learning_rate': 3.450726978998385e-05, 'epoch': 0.26}

  3%|▎         | 1068/41250 [2:35:00<96:30:49,  8.65s/it][2025-04-25 10:32:44,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:32:44,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.70 | bwd_microstep: 5753.04 | bwd_inner_microstep: 5679.44 | bwd_allreduce_microstep: 73.55 | step_microstep: 18.90
[2025-04-25 10:32:44,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.70 | bwd: 5753.05 | bwd_inner: 5679.44 | bwd_allreduce: 73.57 | step: 18.91

  3%|▎         | 1069/41250 [2:35:09<96:37:10,  8.66s/it]
                                                         
{'loss': 0.0362, 'grad_norm': 0.5205958485603333, 'learning_rate': 3.453957996768983e-05, 'epoch': 0.26}

  3%|▎         | 1069/41250 [2:35:09<96:37:10,  8.66s/it][2025-04-25 10:32:52,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:32:52,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5720.52 | bwd_inner_microstep: 5707.64 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.40
[2025-04-25 10:32:52,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5720.54 | bwd_inner: 5707.64 | bwd_allreduce: 12.85 | step: 19.40

  3%|▎         | 1070/41250 [2:35:18<96:36:09,  8.66s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.5077391862869263, 'learning_rate': 3.45718901453958e-05, 'epoch': 0.26}

  3%|▎         | 1070/41250 [2:35:18<96:36:09,  8.66s/it][2025-04-25 10:33:01,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:33:01,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.82 | bwd_microstep: 5733.71 | bwd_inner_microstep: 5702.26 | bwd_allreduce_microstep: 31.40 | step_microstep: 18.53
[2025-04-25 10:33:01,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.82 | bwd: 5733.72 | bwd_inner: 5702.26 | bwd_allreduce: 31.42 | step: 18.53

  3%|▎         | 1071/41250 [2:35:26<96:39:24,  8.66s/it]
                                                         
{'loss': 0.1078, 'grad_norm': 1.327995777130127, 'learning_rate': 3.460420032310178e-05, 'epoch': 0.26}

  3%|▎         | 1071/41250 [2:35:26<96:39:24,  8.66s/it][2025-04-25 10:33:10,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:33:10,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.17 | bwd_microstep: 5721.32 | bwd_inner_microstep: 5708.34 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.75
[2025-04-25 10:33:10,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.17 | bwd: 5721.33 | bwd_inner: 5708.34 | bwd_allreduce: 12.95 | step: 18.75

  3%|▎         | 1072/41250 [2:35:35<96:38:24,  8.66s/it]
                                                         
{'loss': 0.1195, 'grad_norm': 1.8809726238250732, 'learning_rate': 3.4636510500807756e-05, 'epoch': 0.26}

  3%|▎         | 1072/41250 [2:35:35<96:38:24,  8.66s/it][2025-04-25 10:33:18,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.26 | optimizer_step: 0.93
[2025-04-25 10:33:18,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.53 | bwd_microstep: 5763.78 | bwd_inner_microstep: 5696.18 | bwd_allreduce_microstep: 67.55 | step_microstep: 19.21
[2025-04-25 10:33:18,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.53 | bwd: 5763.79 | bwd_inner: 5696.18 | bwd_allreduce: 67.56 | step: 19.21

  3%|▎         | 1073/41250 [2:35:44<96:46:03,  8.67s/it]
                                                         
{'loss': 0.3251, 'grad_norm': 2.869719982147217, 'learning_rate': 3.4668820678513734e-05, 'epoch': 0.26}

  3%|▎         | 1073/41250 [2:35:44<96:46:03,  8.67s/it][2025-04-25 10:33:27,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.07 | optimizer_step: 1.22
[2025-04-25 10:33:27,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.53 | bwd_microstep: 5791.07 | bwd_inner_microstep: 5777.47 | bwd_allreduce_microstep: 13.54 | step_microstep: 19.69
[2025-04-25 10:33:27,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.53 | bwd: 5791.09 | bwd_inner: 5777.47 | bwd_allreduce: 13.57 | step: 19.69

  3%|▎         | 1074/41250 [2:35:52<97:06:14,  8.70s/it]
                                                         
{'loss': 0.2416, 'grad_norm': 2.129809617996216, 'learning_rate': 3.470113085621971e-05, 'epoch': 0.26}

  3%|▎         | 1074/41250 [2:35:52<97:06:14,  8.70s/it][2025-04-25 10:33:36,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:33:36,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5939.12 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 285.58 | step_microstep: 18.35
[2025-04-25 10:33:36,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5939.14 | bwd_inner: 5653.50 | bwd_allreduce: 285.59 | step: 18.36

  3%|▎         | 1075/41250 [2:36:01<97:35:56,  8.75s/it]
                                                         
{'loss': 0.2939, 'grad_norm': 2.6599671840667725, 'learning_rate': 3.473344103392569e-05, 'epoch': 0.26}

  3%|▎         | 1075/41250 [2:36:01<97:35:56,  8.75s/it][2025-04-25 10:33:45,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:33:45,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.99 | bwd_microstep: 5765.92 | bwd_inner_microstep: 5707.93 | bwd_allreduce_microstep: 57.95 | step_microstep: 18.51
[2025-04-25 10:33:45,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.99 | bwd: 5765.94 | bwd_inner: 5707.93 | bwd_allreduce: 57.96 | step: 18.51

  3%|▎         | 1076/41250 [2:36:10<97:27:52,  8.73s/it]
                                                         
{'loss': 0.4676, 'grad_norm': 3.4569358825683594, 'learning_rate': 3.476575121163167e-05, 'epoch': 0.26}

  3%|▎         | 1076/41250 [2:36:10<97:27:52,  8.73s/it][2025-04-25 10:33:53,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-25 10:33:53,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.54 | bwd_microstep: 5724.73 | bwd_inner_microstep: 5711.97 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.01
[2025-04-25 10:33:53,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.54 | bwd: 5724.74 | bwd_inner: 5711.97 | bwd_allreduce: 12.73 | step: 19.01

  3%|▎         | 1077/41250 [2:36:19<97:13:54,  8.71s/it]
                                                         
{'loss': 0.1682, 'grad_norm': 1.9713504314422607, 'learning_rate': 3.4798061389337645e-05, 'epoch': 0.26}

  3%|▎         | 1077/41250 [2:36:19<97:13:54,  8.71s/it][2025-04-25 10:34:02,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 10:34:02,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.16 | bwd_microstep: 5747.44 | bwd_inner_microstep: 5709.19 | bwd_allreduce_microstep: 38.20 | step_microstep: 19.27
[2025-04-25 10:34:02,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.16 | bwd: 5747.46 | bwd_inner: 5709.19 | bwd_allreduce: 38.22 | step: 19.27

  3%|▎         | 1078/41250 [2:36:27<97:08:28,  8.71s/it]
                                                         
{'loss': 0.1837, 'grad_norm': 1.5012181997299194, 'learning_rate': 3.483037156704362e-05, 'epoch': 0.26}

  3%|▎         | 1078/41250 [2:36:27<97:08:28,  8.71s/it][2025-04-25 10:34:11,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:34:11,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.90 | bwd_microstep: 5713.86 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.51
[2025-04-25 10:34:11,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.90 | bwd: 5713.87 | bwd_inner: 5701.15 | bwd_allreduce: 12.68 | step: 18.52

  3%|▎         | 1079/41250 [2:36:36<96:56:51,  8.69s/it]
                                                         
{'loss': 0.2703, 'grad_norm': 3.7087340354919434, 'learning_rate': 3.48626817447496e-05, 'epoch': 0.26}

  3%|▎         | 1079/41250 [2:36:36<96:56:51,  8.69s/it][2025-04-25 10:34:19,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:34:19,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.08 | bwd_microstep: 5950.67 | bwd_inner_microstep: 5669.46 | bwd_allreduce_microstep: 281.17 | step_microstep: 18.73
[2025-04-25 10:34:19,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.08 | bwd: 5950.69 | bwd_inner: 5669.46 | bwd_allreduce: 281.19 | step: 18.74

  3%|▎         | 1080/41250 [2:36:45<97:33:27,  8.74s/it]
                                                         
{'loss': 0.0732, 'grad_norm': 1.1310386657714844, 'learning_rate': 3.489499192245557e-05, 'epoch': 0.26}

  3%|▎         | 1080/41250 [2:36:45<97:33:27,  8.74s/it][2025-04-25 10:34:28,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 10:34:28,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.75 | bwd_microstep: 5717.90 | bwd_inner_microstep: 5656.67 | bwd_allreduce_microstep: 61.18 | step_microstep: 19.37
[2025-04-25 10:34:28,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.75 | bwd: 5717.92 | bwd_inner: 5656.67 | bwd_allreduce: 61.20 | step: 19.38

  3%|▎         | 1081/41250 [2:36:53<97:10:56,  8.71s/it]
                                                         
{'loss': 0.1261, 'grad_norm': 2.4534504413604736, 'learning_rate': 3.492730210016155e-05, 'epoch': 0.26}

  3%|▎         | 1081/41250 [2:36:53<97:10:56,  8.71s/it][2025-04-25 10:34:37,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:34:37,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5701.77 | bwd_inner_microstep: 5661.47 | bwd_allreduce_microstep: 40.26 | step_microstep: 18.43
[2025-04-25 10:34:37,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5701.78 | bwd_inner: 5661.47 | bwd_allreduce: 40.28 | step: 18.43

  3%|▎         | 1082/41250 [2:37:02<96:52:05,  8.68s/it]
                                                         
{'loss': 0.1185, 'grad_norm': 1.819409966468811, 'learning_rate': 3.4959612277867535e-05, 'epoch': 0.26}

  3%|▎         | 1082/41250 [2:37:02<96:52:05,  8.68s/it][2025-04-25 10:34:45,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 10:34:45,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.73 | bwd_microstep: 5766.31 | bwd_inner_microstep: 5667.50 | bwd_allreduce_microstep: 98.76 | step_microstep: 19.08
[2025-04-25 10:34:45,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.73 | bwd: 5766.32 | bwd_inner: 5667.49 | bwd_allreduce: 98.78 | step: 19.09

  3%|▎         | 1083/41250 [2:37:11<96:52:20,  8.68s/it]
                                                         
{'loss': 0.2802, 'grad_norm': 2.4105868339538574, 'learning_rate': 3.499192245557351e-05, 'epoch': 0.26}

  3%|▎         | 1083/41250 [2:37:11<96:52:20,  8.68s/it][2025-04-25 10:34:54,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:34:54,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.68 | bwd_microstep: 5808.90 | bwd_inner_microstep: 5668.67 | bwd_allreduce_microstep: 140.19 | step_microstep: 18.43
[2025-04-25 10:34:54,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5808.92 | bwd_inner: 5668.67 | bwd_allreduce: 140.21 | step: 18.44

  3%|▎         | 1084/41250 [2:37:19<97:00:27,  8.69s/it]
                                                         
{'loss': 0.0658, 'grad_norm': 1.4359705448150635, 'learning_rate': 3.5024232633279484e-05, 'epoch': 0.26}

  3%|▎         | 1084/41250 [2:37:19<97:00:27,  8.69s/it][2025-04-25 10:35:03,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 10:35:03,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5776.03 | bwd_inner_microstep: 5659.98 | bwd_allreduce_microstep: 116.01 | step_microstep: 18.52
[2025-04-25 10:35:03,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5776.05 | bwd_inner: 5659.98 | bwd_allreduce: 116.03 | step: 18.52

  3%|▎         | 1085/41250 [2:37:28<96:59:13,  8.69s/it]
                                                         
{'loss': 0.1459, 'grad_norm': 2.6330392360687256, 'learning_rate': 3.505654281098546e-05, 'epoch': 0.26}

  3%|▎         | 1085/41250 [2:37:28<96:59:13,  8.69s/it][2025-04-25 10:35:11,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 10:35:11,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.93 | bwd_microstep: 5714.46 | bwd_inner_microstep: 5701.66 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.92
[2025-04-25 10:35:11,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.93 | bwd: 5714.47 | bwd_inner: 5701.66 | bwd_allreduce: 12.77 | step: 18.92

  3%|▎         | 1086/41250 [2:37:37<96:49:45,  8.68s/it]
                                                         
{'loss': 0.3649, 'grad_norm': 3.242356061935425, 'learning_rate': 3.508885298869144e-05, 'epoch': 0.26}

  3%|▎         | 1086/41250 [2:37:37<96:49:45,  8.68s/it][2025-04-25 10:35:20,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 10:35:20,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.75 | bwd_microstep: 5714.89 | bwd_inner_microstep: 5664.86 | bwd_allreduce_microstep: 49.98 | step_microstep: 19.19
[2025-04-25 10:35:20,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.75 | bwd: 5714.90 | bwd_inner: 5664.86 | bwd_allreduce: 50.00 | step: 19.19

  3%|▎         | 1087/41250 [2:37:45<96:40:34,  8.67s/it]
                                                         
{'loss': 0.2246, 'grad_norm': 2.338526725769043, 'learning_rate': 3.512116316639742e-05, 'epoch': 0.26}

  3%|▎         | 1087/41250 [2:37:45<96:40:34,  8.67s/it][2025-04-25 10:35:29,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-25 10:35:29,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5694.54 | bwd_inner_microstep: 5681.35 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.26
[2025-04-25 10:35:29,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5694.56 | bwd_inner: 5681.35 | bwd_allreduce: 13.16 | step: 19.26

  3%|▎         | 1088/41250 [2:37:54<96:34:03,  8.66s/it]
                                                         
{'loss': 0.0797, 'grad_norm': 0.9921442866325378, 'learning_rate': 3.5153473344103395e-05, 'epoch': 0.26}

  3%|▎         | 1088/41250 [2:37:54<96:34:03,  8.66s/it][2025-04-25 10:35:37,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 10:35:37,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.23 | bwd_microstep: 5704.84 | bwd_inner_microstep: 5650.46 | bwd_allreduce_microstep: 54.34 | step_microstep: 18.48
[2025-04-25 10:35:37,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.23 | bwd: 5704.85 | bwd_inner: 5650.46 | bwd_allreduce: 54.36 | step: 18.48

  3%|▎         | 1089/41250 [2:38:03<96:25:50,  8.64s/it]
                                                         
{'loss': 0.2071, 'grad_norm': 1.877063274383545, 'learning_rate': 3.518578352180937e-05, 'epoch': 0.26}

  3%|▎         | 1089/41250 [2:38:03<96:25:50,  8.64s/it][2025-04-25 10:35:46,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:35:46,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3082.62 | bwd_microstep: 5699.64 | bwd_inner_microstep: 5686.92 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.70
[2025-04-25 10:35:46,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3082.62 | bwd: 5699.66 | bwd_inner: 5686.92 | bwd_allreduce: 12.69 | step: 18.70

  3%|▎         | 1090/41250 [2:38:12<97:09:49,  8.71s/it]
                                                         
{'loss': 0.1789, 'grad_norm': 1.4864877462387085, 'learning_rate': 3.521809369951535e-05, 'epoch': 0.26}

  3%|▎         | 1090/41250 [2:38:12<97:09:49,  8.71s/it][2025-04-25 10:35:55,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:35:55,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.40 | bwd_microstep: 5783.25 | bwd_inner_microstep: 5647.24 | bwd_allreduce_microstep: 135.97 | step_microstep: 18.84
[2025-04-25 10:35:55,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.40 | bwd: 5783.27 | bwd_inner: 5647.24 | bwd_allreduce: 135.99 | step: 18.85

  3%|▎         | 1091/41250 [2:38:20<97:05:41,  8.70s/it]
                                                         
{'loss': 0.1764, 'grad_norm': 2.238420248031616, 'learning_rate': 3.525040387722133e-05, 'epoch': 0.26}

  3%|▎         | 1091/41250 [2:38:20<97:05:41,  8.70s/it][2025-04-25 10:36:04,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:36:04,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.34 | bwd_microstep: 5702.88 | bwd_inner_microstep: 5649.42 | bwd_allreduce_microstep: 53.41 | step_microstep: 18.81
[2025-04-25 10:36:04,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.34 | bwd: 5702.90 | bwd_inner: 5649.42 | bwd_allreduce: 53.43 | step: 18.82

  3%|▎         | 1092/41250 [2:38:29<96:47:11,  8.68s/it]
                                                         
{'loss': 0.052, 'grad_norm': 0.7181000709533691, 'learning_rate': 3.528271405492731e-05, 'epoch': 0.26}

  3%|▎         | 1092/41250 [2:38:29<96:47:11,  8.68s/it][2025-04-25 10:36:12,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:36:12,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.23 | bwd_microstep: 5851.22 | bwd_inner_microstep: 5682.14 | bwd_allreduce_microstep: 169.04 | step_microstep: 18.83
[2025-04-25 10:36:12,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.23 | bwd: 5851.23 | bwd_inner: 5682.14 | bwd_allreduce: 169.05 | step: 18.83

  3%|▎         | 1093/41250 [2:38:38<97:08:06,  8.71s/it]
                                                         
{'loss': 0.065, 'grad_norm': 1.2457457780838013, 'learning_rate': 3.5315024232633285e-05, 'epoch': 0.26}

  3%|▎         | 1093/41250 [2:38:38<97:08:06,  8.71s/it][2025-04-25 10:36:21,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:36:21,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.59 | bwd_microstep: 5747.86 | bwd_inner_microstep: 5709.97 | bwd_allreduce_microstep: 37.84 | step_microstep: 18.80
[2025-04-25 10:36:21,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.59 | bwd: 5747.87 | bwd_inner: 5709.97 | bwd_allreduce: 37.86 | step: 18.81

  3%|▎         | 1094/41250 [2:38:46<97:02:36,  8.70s/it]
                                                         
{'loss': 0.3282, 'grad_norm': 3.582066059112549, 'learning_rate': 3.5347334410339256e-05, 'epoch': 0.27}

  3%|▎         | 1094/41250 [2:38:46<97:02:36,  8.70s/it][2025-04-25 10:36:30,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 10:36:30,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.85 | bwd_microstep: 5763.81 | bwd_inner_microstep: 5652.61 | bwd_allreduce_microstep: 111.16 | step_microstep: 19.16
[2025-04-25 10:36:30,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.85 | bwd: 5763.82 | bwd_inner: 5652.61 | bwd_allreduce: 111.18 | step: 19.17

  3%|▎         | 1095/41250 [2:38:55<96:57:11,  8.69s/it]
                                                         
{'loss': 0.0219, 'grad_norm': 0.451226145029068, 'learning_rate': 3.5379644588045234e-05, 'epoch': 0.27}

  3%|▎         | 1095/41250 [2:38:55<96:57:11,  8.69s/it][2025-04-25 10:36:38,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 10:36:38,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.18 | bwd_microstep: 5707.18 | bwd_inner_microstep: 5694.05 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.51
[2025-04-25 10:36:38,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.18 | bwd: 5707.20 | bwd_inner: 5694.05 | bwd_allreduce: 13.10 | step: 19.51

  3%|▎         | 1096/41250 [2:39:04<96:47:34,  8.68s/it]
                                                         
{'loss': 0.2033, 'grad_norm': 3.903264045715332, 'learning_rate': 3.541195476575122e-05, 'epoch': 0.27}

  3%|▎         | 1096/41250 [2:39:04<96:47:34,  8.68s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x3342e8c0] moov atom not found
[10:36:38] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4...
[2025-04-25 10:36:47,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 10:36:47,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.16 | bwd_microstep: 5770.03 | bwd_inner_microstep: 5651.27 | bwd_allreduce_microstep: 118.72 | step_microstep: 19.55
[2025-04-25 10:36:47,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.16 | bwd: 5770.05 | bwd_inner: 5651.27 | bwd_allreduce: 118.74 | step: 19.55

  3%|▎         | 1097/41250 [2:39:12<96:48:01,  8.68s/it]
                                                         
{'loss': 0.628, 'grad_norm': 5.401479721069336, 'learning_rate': 3.5444264943457196e-05, 'epoch': 0.27}

  3%|▎         | 1097/41250 [2:39:12<96:48:01,  8.68s/it][2025-04-25 10:36:56,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:36:56,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.80 | bwd_microstep: 5782.35 | bwd_inner_microstep: 5769.72 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.61
[2025-04-25 10:36:56,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.80 | bwd: 5782.37 | bwd_inner: 5769.72 | bwd_allreduce: 12.60 | step: 18.61

  3%|▎         | 1098/41250 [2:39:21<97:02:32,  8.70s/it]
                                                         
{'loss': 0.1567, 'grad_norm': 1.736297369003296, 'learning_rate': 3.547657512116317e-05, 'epoch': 0.27}

  3%|▎         | 1098/41250 [2:39:21<97:02:32,  8.70s/it][2025-04-25 10:37:04,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 1.12
[2025-04-25 10:37:04,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.07 | bwd_microstep: 5745.82 | bwd_inner_microstep: 5690.94 | bwd_allreduce_microstep: 54.83 | step_microstep: 19.18
[2025-04-25 10:37:04,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.07 | bwd: 5745.83 | bwd_inner: 5690.94 | bwd_allreduce: 54.85 | step: 19.18

  3%|▎         | 1099/41250 [2:39:30<96:59:28,  8.70s/it]
                                                         
{'loss': 0.1842, 'grad_norm': 3.2896926403045654, 'learning_rate': 3.5508885298869145e-05, 'epoch': 0.27}

  3%|▎         | 1099/41250 [2:39:30<96:59:28,  8.70s/it][2025-04-25 10:37:13,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:37:13,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.71 | bwd_microstep: 5746.52 | bwd_inner_microstep: 5645.35 | bwd_allreduce_microstep: 101.13 | step_microstep: 18.66
[2025-04-25 10:37:13,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.71 | bwd: 5746.54 | bwd_inner: 5645.35 | bwd_allreduce: 101.14 | step: 18.66

  3%|▎         | 1100/41250 [2:39:38<96:50:44,  8.68s/it]
                                                         
{'loss': 0.358, 'grad_norm': 4.039144992828369, 'learning_rate': 3.554119547657512e-05, 'epoch': 0.27}

  3%|▎         | 1100/41250 [2:39:38<96:50:44,  8.68s/it][2025-04-25 10:37:22,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 10:37:22,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5707.64 | bwd_inner_microstep: 5694.87 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.66
[2025-04-25 10:37:22,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5707.65 | bwd_inner: 5694.87 | bwd_allreduce: 12.75 | step: 18.66

  3%|▎         | 1101/41250 [2:39:47<96:42:25,  8.67s/it]
                                                         
{'loss': 0.3753, 'grad_norm': 7.455212116241455, 'learning_rate': 3.55735056542811e-05, 'epoch': 0.27}

  3%|▎         | 1101/41250 [2:39:47<96:42:25,  8.67s/it][2025-04-25 10:37:30,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:37:30,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.71 | bwd_microstep: 5699.67 | bwd_inner_microstep: 5655.80 | bwd_allreduce_microstep: 43.81 | step_microstep: 18.98
[2025-04-25 10:37:30,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.71 | bwd: 5699.68 | bwd_inner: 5655.80 | bwd_allreduce: 43.83 | step: 18.99

  3%|▎         | 1102/41250 [2:39:56<96:32:28,  8.66s/it]
                                                         
{'loss': 0.1488, 'grad_norm': 2.356940269470215, 'learning_rate': 3.560581583198708e-05, 'epoch': 0.27}

  3%|▎         | 1102/41250 [2:39:56<96:32:28,  8.66s/it][2025-04-25 10:37:39,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 10:37:39,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.51 | bwd_microstep: 5733.92 | bwd_inner_microstep: 5707.79 | bwd_allreduce_microstep: 26.08 | step_microstep: 19.13
[2025-04-25 10:37:39,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.51 | bwd: 5733.93 | bwd_inner: 5707.79 | bwd_allreduce: 26.11 | step: 19.13

  3%|▎         | 1103/41250 [2:40:04<96:35:30,  8.66s/it]
                                                         
{'loss': 0.2108, 'grad_norm': 3.7590861320495605, 'learning_rate': 3.563812600969306e-05, 'epoch': 0.27}

  3%|▎         | 1103/41250 [2:40:04<96:35:30,  8.66s/it][2025-04-25 10:37:48,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 10:37:48,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.73 | bwd_microstep: 5741.12 | bwd_inner_microstep: 5691.10 | bwd_allreduce_microstep: 49.97 | step_microstep: 18.89
[2025-04-25 10:37:48,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.73 | bwd: 5741.13 | bwd_inner: 5691.10 | bwd_allreduce: 49.99 | step: 18.89

  3%|▎         | 1104/41250 [2:40:13<96:36:55,  8.66s/it]
                                                         
{'loss': 0.2915, 'grad_norm': 1.6401230096817017, 'learning_rate': 3.5670436187399035e-05, 'epoch': 0.27}

  3%|▎         | 1104/41250 [2:40:13<96:36:55,  8.66s/it][2025-04-25 10:37:56,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:37:56,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5692.75 | bwd_inner_microstep: 5679.74 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.74
[2025-04-25 10:37:56,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5692.76 | bwd_inner: 5679.74 | bwd_allreduce: 12.98 | step: 18.74

  3%|▎         | 1105/41250 [2:40:22<96:30:55,  8.66s/it]
                                                         
{'loss': 0.4171, 'grad_norm': 3.934905529022217, 'learning_rate': 3.570274636510501e-05, 'epoch': 0.27}

  3%|▎         | 1105/41250 [2:40:22<96:30:55,  8.66s/it][2025-04-25 10:38:05,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 10:38:05,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.26 | bwd_microstep: 5744.35 | bwd_inner_microstep: 5686.12 | bwd_allreduce_microstep: 58.19 | step_microstep: 18.03
[2025-04-25 10:38:05,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.26 | bwd: 5744.37 | bwd_inner: 5686.12 | bwd_allreduce: 58.21 | step: 18.03

  3%|▎         | 1106/41250 [2:40:30<96:37:17,  8.66s/it]
                                                         
{'loss': 0.1547, 'grad_norm': 2.1674249172210693, 'learning_rate': 3.573505654281099e-05, 'epoch': 0.27}

  3%|▎         | 1106/41250 [2:40:30<96:37:17,  8.66s/it][2025-04-25 10:38:14,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 10:38:14,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.44 | bwd_microstep: 5710.55 | bwd_inner_microstep: 5688.29 | bwd_allreduce_microstep: 22.21 | step_microstep: 19.23
[2025-04-25 10:38:14,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.44 | bwd: 5710.56 | bwd_inner: 5688.29 | bwd_allreduce: 22.23 | step: 19.23

  3%|▎         | 1107/41250 [2:40:39<96:33:16,  8.66s/it]
                                                         
{'loss': 0.1532, 'grad_norm': 1.4755498170852661, 'learning_rate': 3.576736672051697e-05, 'epoch': 0.27}

  3%|▎         | 1107/41250 [2:40:39<96:33:16,  8.66s/it][2025-04-25 10:38:22,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.15 | optimizer_step: 1.03
[2025-04-25 10:38:22,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.70 | bwd_microstep: 5864.72 | bwd_inner_microstep: 5694.81 | bwd_allreduce_microstep: 169.87 | step_microstep: 19.35
[2025-04-25 10:38:22,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.70 | bwd: 5864.74 | bwd_inner: 5694.81 | bwd_allreduce: 169.89 | step: 19.36

  3%|▎         | 1108/41250 [2:40:48<97:03:54,  8.70s/it]
                                                         
{'loss': 0.4553, 'grad_norm': 3.4975106716156006, 'learning_rate': 3.579967689822294e-05, 'epoch': 0.27}

  3%|▎         | 1108/41250 [2:40:48<97:03:54,  8.70s/it][2025-04-25 10:38:31,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:38:31,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.97 | bwd_microstep: 5743.99 | bwd_inner_microstep: 5649.47 | bwd_allreduce_microstep: 94.47 | step_microstep: 18.76
[2025-04-25 10:38:31,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.97 | bwd: 5744.00 | bwd_inner: 5649.47 | bwd_allreduce: 94.49 | step: 18.76

  3%|▎         | 1109/41250 [2:40:56<96:53:23,  8.69s/it]
                                                         
{'loss': 0.2102, 'grad_norm': 2.1804440021514893, 'learning_rate': 3.583198707592892e-05, 'epoch': 0.27}

  3%|▎         | 1109/41250 [2:40:56<96:53:23,  8.69s/it][2025-04-25 10:38:40,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-25 10:38:40,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.96 | bwd_microstep: 5691.82 | bwd_inner_microstep: 5679.10 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.57
[2025-04-25 10:38:40,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.96 | bwd: 5691.83 | bwd_inner: 5679.10 | bwd_allreduce: 12.69 | step: 18.58

  3%|▎         | 1110/41250 [2:41:05<96:39:15,  8.67s/it]
                                                         
{'loss': 0.1786, 'grad_norm': 3.1182808876037598, 'learning_rate': 3.58642972536349e-05, 'epoch': 0.27}

  3%|▎         | 1110/41250 [2:41:05<96:39:15,  8.67s/it][2025-04-25 10:38:48,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 10:38:48,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.78 | bwd_microstep: 5740.02 | bwd_inner_microstep: 5707.42 | bwd_allreduce_microstep: 32.54 | step_microstep: 18.41
[2025-04-25 10:38:48,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.78 | bwd: 5740.03 | bwd_inner: 5707.42 | bwd_allreduce: 32.56 | step: 18.41

  3%|▎         | 1111/41250 [2:41:14<96:39:04,  8.67s/it]
                                                         
{'loss': 0.1445, 'grad_norm': 1.9453083276748657, 'learning_rate': 3.589660743134088e-05, 'epoch': 0.27}

  3%|▎         | 1111/41250 [2:41:14<96:39:04,  8.67s/it][2025-04-25 10:38:57,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 10:38:57,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.86 | bwd_microstep: 5888.56 | bwd_inner_microstep: 5638.36 | bwd_allreduce_microstep: 250.15 | step_microstep: 18.84
[2025-04-25 10:38:57,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.86 | bwd: 5888.57 | bwd_inner: 5638.36 | bwd_allreduce: 250.17 | step: 18.84

  3%|▎         | 1112/41250 [2:41:23<97:03:26,  8.71s/it]
                                                         
{'loss': 0.3014, 'grad_norm': 2.5262365341186523, 'learning_rate': 3.592891760904685e-05, 'epoch': 0.27}

  3%|▎         | 1112/41250 [2:41:23<97:03:26,  8.71s/it][2025-04-25 10:39:06,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 10:39:06,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.43 | bwd_microstep: 5700.50 | bwd_inner_microstep: 5634.04 | bwd_allreduce_microstep: 66.41 | step_microstep: 19.25
[2025-04-25 10:39:06,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.43 | bwd: 5700.51 | bwd_inner: 5634.04 | bwd_allreduce: 66.43 | step: 19.26

  3%|▎         | 1113/41250 [2:41:31<96:42:42,  8.67s/it]
                                                         
{'loss': 0.1141, 'grad_norm': 0.9870232343673706, 'learning_rate': 3.596122778675283e-05, 'epoch': 0.27}

  3%|▎         | 1113/41250 [2:41:31<96:42:42,  8.67s/it][2025-04-25 10:39:14,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:39:14,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.95 | bwd_microstep: 5748.87 | bwd_inner_microstep: 5668.14 | bwd_allreduce_microstep: 80.68 | step_microstep: 18.69
[2025-04-25 10:39:14,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.95 | bwd: 5748.89 | bwd_inner: 5668.14 | bwd_allreduce: 80.70 | step: 18.70

  3%|▎         | 1114/41250 [2:41:40<96:45:24,  8.68s/it]
                                                         
{'loss': 0.181, 'grad_norm': 1.13093900680542, 'learning_rate': 3.599353796445881e-05, 'epoch': 0.27}

  3%|▎         | 1114/41250 [2:41:40<96:45:24,  8.68s/it][2025-04-25 10:39:23,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 10:39:23,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.79 | bwd_microstep: 5747.14 | bwd_inner_microstep: 5637.19 | bwd_allreduce_microstep: 109.90 | step_microstep: 19.19
[2025-04-25 10:39:23,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.79 | bwd: 5747.15 | bwd_inner: 5637.19 | bwd_allreduce: 109.92 | step: 19.19

  3%|▎         | 1115/41250 [2:41:48<96:40:53,  8.67s/it]
                                                         
{'loss': 0.2117, 'grad_norm': 2.165060043334961, 'learning_rate': 3.6025848142164785e-05, 'epoch': 0.27}

  3%|▎         | 1115/41250 [2:41:48<96:40:53,  8.67s/it][2025-04-25 10:39:32,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 10:39:32,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.80 | bwd_microstep: 5860.68 | bwd_inner_microstep: 5681.16 | bwd_allreduce_microstep: 179.46 | step_microstep: 18.80
[2025-04-25 10:39:32,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.80 | bwd: 5860.69 | bwd_inner: 5681.16 | bwd_allreduce: 179.49 | step: 18.80

  3%|▎         | 1116/41250 [2:41:57<97:07:09,  8.71s/it]
                                                         
{'loss': 0.256, 'grad_norm': 2.4036052227020264, 'learning_rate': 3.605815831987076e-05, 'epoch': 0.27}

  3%|▎         | 1116/41250 [2:41:57<97:07:09,  8.71s/it][2025-04-25 10:39:41,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:39:41,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5702.62 | bwd_inner_microstep: 5680.34 | bwd_allreduce_microstep: 22.24 | step_microstep: 18.67
[2025-04-25 10:39:41,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5702.63 | bwd_inner: 5680.34 | bwd_allreduce: 22.25 | step: 18.67

  3%|▎         | 1117/41250 [2:42:06<96:53:22,  8.69s/it]
                                                         
{'loss': 0.2326, 'grad_norm': 1.543562412261963, 'learning_rate': 3.609046849757674e-05, 'epoch': 0.27}

  3%|▎         | 1117/41250 [2:42:06<96:53:22,  8.69s/it][2025-04-25 10:39:49,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:39:49,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5748.75 | bwd_inner_microstep: 5629.38 | bwd_allreduce_microstep: 119.33 | step_microstep: 18.59
[2025-04-25 10:39:49,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5748.76 | bwd_inner: 5629.38 | bwd_allreduce: 119.35 | step: 18.59

  3%|▎         | 1118/41250 [2:42:15<96:46:33,  8.68s/it]
                                                         
{'loss': 0.1914, 'grad_norm': 1.5532735586166382, 'learning_rate': 3.612277867528272e-05, 'epoch': 0.27}

  3%|▎         | 1118/41250 [2:42:15<96:46:33,  8.68s/it][2025-04-25 10:39:58,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 10:39:58,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.77 | bwd_microstep: 5692.21 | bwd_inner_microstep: 5636.95 | bwd_allreduce_microstep: 55.20 | step_microstep: 18.68
[2025-04-25 10:39:58,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.77 | bwd: 5692.22 | bwd_inner: 5636.95 | bwd_allreduce: 55.23 | step: 18.68

  3%|▎         | 1119/41250 [2:42:23<96:32:21,  8.66s/it]
                                                         
{'loss': 0.1639, 'grad_norm': 1.6149827241897583, 'learning_rate': 3.6155088852988696e-05, 'epoch': 0.27}

  3%|▎         | 1119/41250 [2:42:23<96:32:21,  8.66s/it][2025-04-25 10:40:07,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:40:07,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.33 | bwd_microstep: 5743.45 | bwd_inner_microstep: 5646.15 | bwd_allreduce_microstep: 97.26 | step_microstep: 18.74
[2025-04-25 10:40:07,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.33 | bwd: 5743.46 | bwd_inner: 5646.15 | bwd_allreduce: 97.27 | step: 18.74

  3%|▎         | 1120/41250 [2:42:32<96:31:28,  8.66s/it]
                                                         
{'loss': 0.1694, 'grad_norm': 1.4379078149795532, 'learning_rate': 3.6187399030694674e-05, 'epoch': 0.27}

  3%|▎         | 1120/41250 [2:42:32<96:31:28,  8.66s/it][2025-04-25 10:40:15,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 10:40:15,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.46 | bwd_microstep: 5737.25 | bwd_inner_microstep: 5676.20 | bwd_allreduce_microstep: 60.99 | step_microstep: 18.51
[2025-04-25 10:40:15,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.46 | bwd: 5737.26 | bwd_inner: 5676.20 | bwd_allreduce: 61.02 | step: 18.51

  3%|▎         | 1121/41250 [2:42:41<96:33:22,  8.66s/it]
                                                         
{'loss': 0.1429, 'grad_norm': 2.152726888656616, 'learning_rate': 3.621970920840065e-05, 'epoch': 0.27}

  3%|▎         | 1121/41250 [2:42:41<96:33:22,  8.66s/it][2025-04-25 10:40:24,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:40:24,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.13 | bwd_microstep: 5754.31 | bwd_inner_microstep: 5636.08 | bwd_allreduce_microstep: 118.19 | step_microstep: 18.66
[2025-04-25 10:40:24,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.13 | bwd: 5754.33 | bwd_inner: 5636.08 | bwd_allreduce: 118.21 | step: 18.66

  3%|▎         | 1122/41250 [2:42:49<96:35:08,  8.66s/it]
                                                         
{'loss': 0.2154, 'grad_norm': 2.2439441680908203, 'learning_rate': 3.625201938610662e-05, 'epoch': 0.27}

  3%|▎         | 1122/41250 [2:42:49<96:35:08,  8.66s/it][2025-04-25 10:40:33,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 10:40:33,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5727.68 | bwd_inner_microstep: 5672.24 | bwd_allreduce_microstep: 55.39 | step_microstep: 18.47
[2025-04-25 10:40:33,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5727.69 | bwd_inner: 5672.24 | bwd_allreduce: 55.41 | step: 18.48

  3%|▎         | 1123/41250 [2:42:58<96:34:26,  8.66s/it]
                                                         
{'loss': 0.1936, 'grad_norm': 2.3448596000671387, 'learning_rate': 3.62843295638126e-05, 'epoch': 0.27}

  3%|▎         | 1123/41250 [2:42:58<96:34:26,  8.66s/it][2025-04-25 10:40:41,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:40:41,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.41 | bwd_microstep: 5729.28 | bwd_inner_microstep: 5704.59 | bwd_allreduce_microstep: 24.65 | step_microstep: 18.37
[2025-04-25 10:40:41,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.41 | bwd: 5729.29 | bwd_inner: 5704.59 | bwd_allreduce: 24.67 | step: 18.37

  3%|▎         | 1124/41250 [2:43:07<96:33:58,  8.66s/it]
                                                         
{'loss': 0.1969, 'grad_norm': 1.7810128927230835, 'learning_rate': 3.6316639741518586e-05, 'epoch': 0.27}

  3%|▎         | 1124/41250 [2:43:07<96:33:58,  8.66s/it][2025-04-25 10:40:50,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.69 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 10:40:50,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5741.41 | bwd_inner_microstep: 5676.06 | bwd_allreduce_microstep: 65.31 | step_microstep: 19.27
[2025-04-25 10:40:50,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5741.43 | bwd_inner: 5676.06 | bwd_allreduce: 65.33 | step: 19.27

  3%|▎         | 1125/41250 [2:43:15<96:37:03,  8.67s/it]
                                                         
{'loss': 0.1962, 'grad_norm': 1.7229952812194824, 'learning_rate': 3.634894991922456e-05, 'epoch': 0.27}

  3%|▎         | 1125/41250 [2:43:15<96:37:03,  8.67s/it][2025-04-25 10:40:59,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:40:59,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2941.27 | bwd_microstep: 5885.33 | bwd_inner_microstep: 5872.55 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.63
[2025-04-25 10:40:59,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2941.27 | bwd: 5885.34 | bwd_inner: 5872.55 | bwd_allreduce: 12.75 | step: 18.63

  3%|▎         | 1126/41250 [2:43:24<97:26:05,  8.74s/it]
                                                         
{'loss': 0.2209, 'grad_norm': 1.7211014032363892, 'learning_rate': 3.6381260096930535e-05, 'epoch': 0.27}

  3%|▎         | 1126/41250 [2:43:24<97:26:05,  8.74s/it][2025-04-25 10:41:07,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:41:07,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.82 | bwd_microstep: 5767.46 | bwd_inner_microstep: 5659.82 | bwd_allreduce_microstep: 107.59 | step_microstep: 18.30
[2025-04-25 10:41:07,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.82 | bwd: 5767.48 | bwd_inner: 5659.82 | bwd_allreduce: 107.61 | step: 18.30

  3%|▎         | 1127/41250 [2:43:33<97:15:24,  8.73s/it]
                                                         
{'loss': 0.1899, 'grad_norm': 1.6856377124786377, 'learning_rate': 3.641357027463651e-05, 'epoch': 0.27}

  3%|▎         | 1127/41250 [2:43:33<97:15:24,  8.73s/it][2025-04-25 10:41:16,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 10:41:16,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.43 | bwd_microstep: 5887.15 | bwd_inner_microstep: 5702.18 | bwd_allreduce_microstep: 184.93 | step_microstep: 18.73
[2025-04-25 10:41:16,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.43 | bwd: 5887.17 | bwd_inner: 5702.18 | bwd_allreduce: 184.95 | step: 18.74

  3%|▎         | 1128/41250 [2:43:42<97:33:43,  8.75s/it]
                                                         
{'loss': 0.2388, 'grad_norm': 2.7965238094329834, 'learning_rate': 3.644588045234249e-05, 'epoch': 0.27}

  3%|▎         | 1128/41250 [2:43:42<97:33:43,  8.75s/it][2025-04-25 10:41:25,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 10:41:25,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.60 | bwd_microstep: 5744.22 | bwd_inner_microstep: 5717.97 | bwd_allreduce_microstep: 26.20 | step_microstep: 19.65
[2025-04-25 10:41:25,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.60 | bwd: 5744.24 | bwd_inner: 5717.97 | bwd_allreduce: 26.22 | step: 19.65

  3%|▎         | 1129/41250 [2:43:50<97:20:36,  8.73s/it]
                                                         
{'loss': 0.1311, 'grad_norm': 2.1114389896392822, 'learning_rate': 3.647819063004847e-05, 'epoch': 0.27}

  3%|▎         | 1129/41250 [2:43:50<97:20:36,  8.73s/it][2025-04-25 10:41:34,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 10:41:34,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.76 | bwd_microstep: 5775.23 | bwd_inner_microstep: 5705.60 | bwd_allreduce_microstep: 69.56 | step_microstep: 19.16
[2025-04-25 10:41:34,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.76 | bwd: 5775.24 | bwd_inner: 5705.60 | bwd_allreduce: 69.59 | step: 19.16

  3%|▎         | 1130/41250 [2:43:59<97:16:42,  8.73s/it]
                                                         
{'loss': 0.2972, 'grad_norm': 2.2030999660491943, 'learning_rate': 3.6510500807754446e-05, 'epoch': 0.27}

  3%|▎         | 1130/41250 [2:43:59<97:16:42,  8.73s/it][2025-04-25 10:41:43,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.27 | optimizer_step: 1.01
[2025-04-25 10:41:43,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.41 | bwd_microstep: 5896.20 | bwd_inner_microstep: 5719.98 | bwd_allreduce_microstep: 176.16 | step_microstep: 19.48
[2025-04-25 10:41:43,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.41 | bwd: 5896.21 | bwd_inner: 5719.98 | bwd_allreduce: 176.19 | step: 19.49

  3%|▎         | 1131/41250 [2:44:08<97:39:28,  8.76s/it]
                                                         
{'loss': 0.2225, 'grad_norm': 2.071122407913208, 'learning_rate': 3.6542810985460424e-05, 'epoch': 0.27}

  3%|▎         | 1131/41250 [2:44:08<97:39:28,  8.76s/it][2025-04-25 10:41:51,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 10:41:51,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.32 | bwd_microstep: 5778.38 | bwd_inner_microstep: 5671.62 | bwd_allreduce_microstep: 106.72 | step_microstep: 18.50
[2025-04-25 10:41:51,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.32 | bwd: 5778.40 | bwd_inner: 5671.62 | bwd_allreduce: 106.73 | step: 18.51

  3%|▎         | 1132/41250 [2:44:17<97:26:03,  8.74s/it]
                                                         
{'loss': 0.4332, 'grad_norm': 2.6088836193084717, 'learning_rate': 3.6575121163166395e-05, 'epoch': 0.27}

  3%|▎         | 1132/41250 [2:44:17<97:26:03,  8.74s/it][2025-04-25 10:42:00,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:42:00,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.47 | bwd_microstep: 5785.62 | bwd_inner_microstep: 5772.78 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.80
[2025-04-25 10:42:00,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.47 | bwd: 5785.63 | bwd_inner: 5772.78 | bwd_allreduce: 12.81 | step: 18.80

  3%|▎         | 1133/41250 [2:44:25<97:29:48,  8.75s/it]
                                                         
{'loss': 0.2602, 'grad_norm': 2.2078499794006348, 'learning_rate': 3.660743134087238e-05, 'epoch': 0.27}

  3%|▎         | 1133/41250 [2:44:25<97:29:48,  8.75s/it][2025-04-25 10:42:09,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.18 | optimizer_step: 1.06
[2025-04-25 10:42:09,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.28 | bwd_microstep: 5904.45 | bwd_inner_microstep: 5662.70 | bwd_allreduce_microstep: 241.69 | step_microstep: 19.47
[2025-04-25 10:42:09,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.28 | bwd: 5904.47 | bwd_inner: 5662.70 | bwd_allreduce: 241.72 | step: 19.47

  3%|▎         | 1134/41250 [2:44:34<97:44:58,  8.77s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 1.2984918355941772, 'learning_rate': 3.663974151857836e-05, 'epoch': 0.27}

  3%|▎         | 1134/41250 [2:44:34<97:44:58,  8.77s/it][2025-04-25 10:42:18,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-25 10:42:18,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.70 | bwd_microstep: 5898.92 | bwd_inner_microstep: 5885.57 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.49
[2025-04-25 10:42:18,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.70 | bwd: 5898.94 | bwd_inner: 5885.57 | bwd_allreduce: 13.32 | step: 19.49

  3%|▎         | 1135/41250 [2:44:43<98:14:19,  8.82s/it]
                                                         
{'loss': 0.2048, 'grad_norm': 1.8252794742584229, 'learning_rate': 3.6672051696284335e-05, 'epoch': 0.28}

  3%|▎         | 1135/41250 [2:44:43<98:14:19,  8.82s/it][2025-04-25 10:42:26,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:42:26,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.93 | bwd_microstep: 5703.74 | bwd_inner_microstep: 5691.11 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.61
[2025-04-25 10:42:26,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.93 | bwd: 5703.76 | bwd_inner: 5691.11 | bwd_allreduce: 12.61 | step: 18.61

  3%|▎         | 1136/41250 [2:44:52<97:40:11,  8.77s/it]
                                                         
{'loss': 0.2286, 'grad_norm': 1.8825595378875732, 'learning_rate': 3.6704361873990307e-05, 'epoch': 0.28}

  3%|▎         | 1136/41250 [2:44:52<97:40:11,  8.77s/it][2025-04-25 10:42:35,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 10:42:35,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.79 | bwd_microstep: 5778.25 | bwd_inner_microstep: 5663.43 | bwd_allreduce_microstep: 114.78 | step_microstep: 19.28
[2025-04-25 10:42:35,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.79 | bwd: 5778.27 | bwd_inner: 5663.43 | bwd_allreduce: 114.80 | step: 19.28

  3%|▎         | 1137/41250 [2:45:00<97:26:30,  8.75s/it]
                                                         
{'loss': 0.0719, 'grad_norm': 1.231157660484314, 'learning_rate': 3.6736672051696284e-05, 'epoch': 0.28}

  3%|▎         | 1137/41250 [2:45:00<97:26:30,  8.75s/it][2025-04-25 10:42:44,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 10:42:44,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5771.60 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 80.66 | step_microstep: 18.71
[2025-04-25 10:42:44,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5771.61 | bwd_inner: 5690.89 | bwd_allreduce: 80.68 | step: 18.71

  3%|▎         | 1138/41250 [2:45:09<97:18:16,  8.73s/it]
                                                         
{'loss': 0.1849, 'grad_norm': 1.7044806480407715, 'learning_rate': 3.676898222940227e-05, 'epoch': 0.28}

  3%|▎         | 1138/41250 [2:45:09<97:18:16,  8.73s/it][2025-04-25 10:42:53,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:42:53,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.27 | bwd_microstep: 5822.70 | bwd_inner_microstep: 5670.50 | bwd_allreduce_microstep: 152.16 | step_microstep: 18.62
[2025-04-25 10:42:53,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.27 | bwd: 5822.71 | bwd_inner: 5670.50 | bwd_allreduce: 152.17 | step: 18.62

  3%|▎         | 1139/41250 [2:45:18<97:18:55,  8.73s/it]
                                                         
{'loss': 0.3969, 'grad_norm': 2.149003744125366, 'learning_rate': 3.680129240710824e-05, 'epoch': 0.28}

  3%|▎         | 1139/41250 [2:45:18<97:18:55,  8.73s/it][2025-04-25 10:43:01,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:43:01,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.42 | bwd_microstep: 5782.83 | bwd_inner_microstep: 5697.23 | bwd_allreduce_microstep: 85.55 | step_microstep: 18.79
[2025-04-25 10:43:01,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.42 | bwd: 5782.85 | bwd_inner: 5697.23 | bwd_allreduce: 85.57 | step: 18.80

  3%|▎         | 1140/41250 [2:45:27<97:15:01,  8.73s/it]
                                                         
{'loss': 0.1214, 'grad_norm': 1.05787992477417, 'learning_rate': 3.683360258481422e-05, 'epoch': 0.28}

  3%|▎         | 1140/41250 [2:45:27<97:15:01,  8.73s/it][2025-04-25 10:43:10,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 10:43:10,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.14 | bwd_microstep: 5904.18 | bwd_inner_microstep: 5662.52 | bwd_allreduce_microstep: 241.61 | step_microstep: 18.96
[2025-04-25 10:43:10,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.14 | bwd: 5904.19 | bwd_inner: 5662.52 | bwd_allreduce: 241.63 | step: 18.97

  3%|▎         | 1141/41250 [2:45:35<97:33:34,  8.76s/it]
                                                         
{'loss': 0.2273, 'grad_norm': 2.6498055458068848, 'learning_rate': 3.6865912762520196e-05, 'epoch': 0.28}

  3%|▎         | 1141/41250 [2:45:35<97:33:34,  8.76s/it][2025-04-25 10:43:19,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 10:43:19,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.70 | bwd_microstep: 5715.52 | bwd_inner_microstep: 5664.81 | bwd_allreduce_microstep: 50.66 | step_microstep: 18.53
[2025-04-25 10:43:19,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.70 | bwd: 5715.53 | bwd_inner: 5664.81 | bwd_allreduce: 50.68 | step: 18.53

  3%|▎         | 1142/41250 [2:45:44<97:08:12,  8.72s/it]
                                                         
{'loss': 0.2311, 'grad_norm': 2.1903939247131348, 'learning_rate': 3.6898222940226174e-05, 'epoch': 0.28}

  3%|▎         | 1142/41250 [2:45:44<97:08:12,  8.72s/it][2025-04-25 10:43:27,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.24 | optimizer_step: 1.01
[2025-04-25 10:43:27,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.58 | bwd_microstep: 5784.99 | bwd_inner_microstep: 5656.18 | bwd_allreduce_microstep: 128.74 | step_microstep: 19.79
[2025-04-25 10:43:27,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.58 | bwd: 5785.00 | bwd_inner: 5656.18 | bwd_allreduce: 128.77 | step: 19.79

  3%|▎         | 1143/41250 [2:45:53<97:05:10,  8.71s/it]
                                                         
{'loss': 0.1704, 'grad_norm': 2.8184237480163574, 'learning_rate': 3.693053311793215e-05, 'epoch': 0.28}

  3%|▎         | 1143/41250 [2:45:53<97:05:10,  8.71s/it][2025-04-25 10:43:36,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:43:36,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5797.68 | bwd_inner_microstep: 5649.30 | bwd_allreduce_microstep: 148.34 | step_microstep: 18.45
[2025-04-25 10:43:36,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.16 | bwd: 5797.69 | bwd_inner: 5649.30 | bwd_allreduce: 148.36 | step: 18.45

  3%|▎         | 1144/41250 [2:46:01<97:04:17,  8.71s/it]
                                                         
{'loss': 0.1403, 'grad_norm': 1.465018391609192, 'learning_rate': 3.696284329563813e-05, 'epoch': 0.28}

  3%|▎         | 1144/41250 [2:46:01<97:04:17,  8.71s/it][2025-04-25 10:43:45,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 10:43:45,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.91 | bwd_microstep: 5719.26 | bwd_inner_microstep: 5706.50 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.60
[2025-04-25 10:43:45,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.91 | bwd: 5719.27 | bwd_inner: 5706.50 | bwd_allreduce: 12.73 | step: 18.60

  3%|▎         | 1145/41250 [2:46:10<96:52:39,  8.70s/it]
                                                         
{'loss': 0.3553, 'grad_norm': 2.0373218059539795, 'learning_rate': 3.699515347334411e-05, 'epoch': 0.28}

  3%|▎         | 1145/41250 [2:46:10<96:52:39,  8.70s/it][2025-04-25 10:43:53,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:43:53,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.62 | bwd_microstep: 5703.34 | bwd_inner_microstep: 5690.60 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.74
[2025-04-25 10:43:53,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.62 | bwd: 5703.35 | bwd_inner: 5690.60 | bwd_allreduce: 12.71 | step: 18.74

  3%|▎         | 1146/41250 [2:46:19<96:39:35,  8.68s/it]
                                                         
{'loss': 0.251, 'grad_norm': 1.4094890356063843, 'learning_rate': 3.702746365105008e-05, 'epoch': 0.28}

  3%|▎         | 1146/41250 [2:46:19<96:39:35,  8.68s/it][2025-04-25 10:44:02,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:44:02,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.21 | bwd_microstep: 5713.29 | bwd_inner_microstep: 5700.53 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.77
[2025-04-25 10:44:02,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.21 | bwd: 5713.31 | bwd_inner: 5700.53 | bwd_allreduce: 12.74 | step: 18.78

  3%|▎         | 1147/41250 [2:46:27<96:32:55,  8.67s/it]
                                                         
{'loss': 0.1578, 'grad_norm': 1.4644330739974976, 'learning_rate': 3.705977382875606e-05, 'epoch': 0.28}

  3%|▎         | 1147/41250 [2:46:27<96:32:55,  8.67s/it][2025-04-25 10:44:11,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 10:44:11,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5704.05 | bwd_inner_microstep: 5657.26 | bwd_allreduce_microstep: 46.74 | step_microstep: 19.17
[2025-04-25 10:44:11,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.79 | bwd: 5704.06 | bwd_inner: 5657.26 | bwd_allreduce: 46.76 | step: 19.18

  3%|▎         | 1148/41250 [2:46:36<96:21:50,  8.65s/it]
                                                         
{'loss': 0.1571, 'grad_norm': 1.586372971534729, 'learning_rate': 3.709208400646204e-05, 'epoch': 0.28}

  3%|▎         | 1148/41250 [2:46:36<96:21:50,  8.65s/it][2025-04-25 10:44:19,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 10:44:19,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.66 | bwd_microstep: 5784.89 | bwd_inner_microstep: 5662.01 | bwd_allreduce_microstep: 122.83 | step_microstep: 18.79
[2025-04-25 10:44:19,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.66 | bwd: 5784.91 | bwd_inner: 5662.01 | bwd_allreduce: 122.85 | step: 18.79

  3%|▎         | 1149/41250 [2:46:45<96:30:33,  8.66s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.7250757217407227, 'learning_rate': 3.712439418416802e-05, 'epoch': 0.28}

  3%|▎         | 1149/41250 [2:46:45<96:30:33,  8.66s/it][2025-04-25 10:44:28,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:44:28,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.95 | bwd_microstep: 5746.57 | bwd_inner_microstep: 5693.42 | bwd_allreduce_microstep: 53.10 | step_microstep: 18.75
[2025-04-25 10:44:28,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.95 | bwd: 5746.58 | bwd_inner: 5693.42 | bwd_allreduce: 53.12 | step: 18.76

  3%|▎         | 1150/41250 [2:46:53<96:33:05,  8.67s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 1.0394526720046997, 'learning_rate': 3.715670436187399e-05, 'epoch': 0.28}

  3%|▎         | 1150/41250 [2:46:53<96:33:05,  8.67s/it][2025-04-25 10:44:37,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:44:37,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5714.84 | bwd_inner_microstep: 5652.80 | bwd_allreduce_microstep: 62.00 | step_microstep: 18.19
[2025-04-25 10:44:37,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5714.85 | bwd_inner: 5652.80 | bwd_allreduce: 62.01 | step: 18.19

  3%|▎         | 1151/41250 [2:47:02<96:23:29,  8.65s/it]
                                                         
{'loss': 0.0435, 'grad_norm': 0.4616537392139435, 'learning_rate': 3.718901453957997e-05, 'epoch': 0.28}

  3%|▎         | 1151/41250 [2:47:02<96:23:29,  8.65s/it][2025-04-25 10:44:46,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 10:44:46,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.30 | bwd_microstep: 6012.80 | bwd_inner_microstep: 5703.74 | bwd_allreduce_microstep: 309.01 | step_microstep: 17.97
[2025-04-25 10:44:46,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.30 | bwd: 6012.81 | bwd_inner: 5703.74 | bwd_allreduce: 309.03 | step: 17.97

  3%|▎         | 1152/41250 [2:47:11<97:21:01,  8.74s/it]
                                                         
{'loss': 0.3901, 'grad_norm': 3.519619941711426, 'learning_rate': 3.7221324717285946e-05, 'epoch': 0.28}

  3%|▎         | 1152/41250 [2:47:11<97:21:01,  8.74s/it][2025-04-25 10:44:54,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:44:54,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5707.01 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.66
[2025-04-25 10:44:54,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5707.03 | bwd_inner: 5694.02 | bwd_allreduce: 12.96 | step: 18.66

  3%|▎         | 1153/41250 [2:47:20<96:58:43,  8.71s/it]
                                                         
{'loss': 0.3808, 'grad_norm': 2.357451915740967, 'learning_rate': 3.7253634894991924e-05, 'epoch': 0.28}

  3%|▎         | 1153/41250 [2:47:20<96:58:43,  8.71s/it][2025-04-25 10:45:03,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-25 10:45:03,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5719.96 | bwd_inner_microstep: 5701.06 | bwd_allreduce_microstep: 18.84 | step_microstep: 18.37
[2025-04-25 10:45:03,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5719.97 | bwd_inner: 5701.06 | bwd_allreduce: 18.86 | step: 18.38

  3%|▎         | 1154/41250 [2:47:28<96:48:13,  8.69s/it]
                                                         
{'loss': 0.129, 'grad_norm': 2.011857032775879, 'learning_rate': 3.72859450726979e-05, 'epoch': 0.28}

  3%|▎         | 1154/41250 [2:47:28<96:48:13,  8.69s/it][2025-04-25 10:45:12,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.13 | optimizer_step: 1.04
[2025-04-25 10:45:12,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5716.79 | bwd_inner_microstep: 5703.25 | bwd_allreduce_microstep: 13.47 | step_microstep: 20.41
[2025-04-25 10:45:12,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.69 | bwd: 5716.80 | bwd_inner: 5703.25 | bwd_allreduce: 13.50 | step: 20.42

  3%|▎         | 1155/41250 [2:47:37<96:40:47,  8.68s/it]
                                                         
{'loss': 0.2422, 'grad_norm': 2.0999996662139893, 'learning_rate': 3.731825525040388e-05, 'epoch': 0.28}

  3%|▎         | 1155/41250 [2:47:37<96:40:47,  8.68s/it][2025-04-25 10:45:20,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 10:45:20,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.02 | bwd_microstep: 5752.05 | bwd_inner_microstep: 5682.37 | bwd_allreduce_microstep: 69.63 | step_microstep: 18.49
[2025-04-25 10:45:20,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.01 | bwd: 5752.06 | bwd_inner: 5682.37 | bwd_allreduce: 69.64 | step: 18.49

  3%|▎         | 1156/41250 [2:47:46<96:39:51,  8.68s/it]
                                                         
{'loss': 0.4554, 'grad_norm': 2.5560619831085205, 'learning_rate': 3.735056542810986e-05, 'epoch': 0.28}

  3%|▎         | 1156/41250 [2:47:46<96:39:51,  8.68s/it][2025-04-25 10:45:29,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 10:45:29,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.71 | bwd_microstep: 5882.11 | bwd_inner_microstep: 5681.50 | bwd_allreduce_microstep: 200.57 | step_microstep: 18.92
[2025-04-25 10:45:29,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.71 | bwd: 5882.12 | bwd_inner: 5681.50 | bwd_allreduce: 200.59 | step: 18.92

  3%|▎         | 1157/41250 [2:47:54<97:04:09,  8.72s/it]
                                                         
{'loss': 0.2182, 'grad_norm': 1.4310576915740967, 'learning_rate': 3.7382875605815835e-05, 'epoch': 0.28}

  3%|▎         | 1157/41250 [2:47:54<97:04:09,  8.72s/it][2025-04-25 10:45:38,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:45:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.42 | bwd_microstep: 5702.81 | bwd_inner_microstep: 5689.86 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.83
[2025-04-25 10:45:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.42 | bwd: 5702.82 | bwd_inner: 5689.86 | bwd_allreduce: 12.92 | step: 18.83

  3%|▎         | 1158/41250 [2:48:03<96:47:24,  8.69s/it]
                                                         
{'loss': 0.0574, 'grad_norm': 0.6010886430740356, 'learning_rate': 3.741518578352181e-05, 'epoch': 0.28}

  3%|▎         | 1158/41250 [2:48:03<96:47:24,  8.69s/it][2025-04-25 10:45:46,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:45:46,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.73 | bwd_microstep: 5724.50 | bwd_inner_microstep: 5711.59 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.76
[2025-04-25 10:45:46,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.73 | bwd: 5724.52 | bwd_inner: 5711.59 | bwd_allreduce: 12.89 | step: 18.76

  3%|▎         | 1159/41250 [2:48:12<96:41:34,  8.68s/it]
                                                         
{'loss': 0.1985, 'grad_norm': 1.5237067937850952, 'learning_rate': 3.744749596122779e-05, 'epoch': 0.28}

  3%|▎         | 1159/41250 [2:48:12<96:41:34,  8.68s/it][2025-04-25 10:45:55,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:45:55,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.99 | bwd_microstep: 5748.23 | bwd_inner_microstep: 5643.45 | bwd_allreduce_microstep: 104.73 | step_microstep: 18.74
[2025-04-25 10:45:55,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.99 | bwd: 5748.24 | bwd_inner: 5643.45 | bwd_allreduce: 104.75 | step: 18.74

  3%|▎         | 1160/41250 [2:48:20<96:36:33,  8.68s/it]
                                                         
{'loss': 0.1918, 'grad_norm': 1.3954755067825317, 'learning_rate': 3.747980613893376e-05, 'epoch': 0.28}

  3%|▎         | 1160/41250 [2:48:20<96:36:33,  8.68s/it][2025-04-25 10:46:04,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 10:46:04,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.22 | bwd_microstep: 5718.28 | bwd_inner_microstep: 5705.44 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.28
[2025-04-25 10:46:04,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.22 | bwd: 5718.29 | bwd_inner: 5705.43 | bwd_allreduce: 12.82 | step: 19.28

  3%|▎         | 1161/41250 [2:48:29<96:31:20,  8.67s/it]
                                                         
{'loss': 0.2278, 'grad_norm': 1.1648441553115845, 'learning_rate': 3.751211631663975e-05, 'epoch': 0.28}

  3%|▎         | 1161/41250 [2:48:29<96:31:20,  8.67s/it][2025-04-25 10:46:12,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:46:12,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.43 | bwd_microstep: 5755.88 | bwd_inner_microstep: 5643.03 | bwd_allreduce_microstep: 112.80 | step_microstep: 18.52
[2025-04-25 10:46:12,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.43 | bwd: 5755.89 | bwd_inner: 5643.03 | bwd_allreduce: 112.82 | step: 18.52

  3%|▎         | 1162/41250 [2:48:38<96:28:53,  8.66s/it]
                                                         
{'loss': 0.1932, 'grad_norm': 2.019878625869751, 'learning_rate': 3.7544426494345725e-05, 'epoch': 0.28}

  3%|▎         | 1162/41250 [2:48:38<96:28:53,  8.66s/it][2025-04-25 10:46:21,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:46:21,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.23 | bwd_microstep: 5709.38 | bwd_inner_microstep: 5696.57 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.72
[2025-04-25 10:46:21,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.23 | bwd: 5709.39 | bwd_inner: 5696.57 | bwd_allreduce: 12.78 | step: 18.73

  3%|▎         | 1163/41250 [2:48:46<96:23:06,  8.66s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 0.7942833304405212, 'learning_rate': 3.75767366720517e-05, 'epoch': 0.28}

  3%|▎         | 1163/41250 [2:48:46<96:23:06,  8.66s/it][2025-04-25 10:46:30,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 10:46:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5712.36 | bwd_inner_microstep: 5699.66 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.87
[2025-04-25 10:46:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5712.38 | bwd_inner: 5699.66 | bwd_allreduce: 12.68 | step: 18.87

  3%|▎         | 1164/41250 [2:48:55<96:19:53,  8.65s/it]
                                                         
{'loss': 0.3052, 'grad_norm': 2.0065395832061768, 'learning_rate': 3.7609046849757674e-05, 'epoch': 0.28}

  3%|▎         | 1164/41250 [2:48:55<96:19:53,  8.65s/it][2025-04-25 10:46:38,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:46:38,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5721.84 | bwd_inner_microstep: 5704.45 | bwd_allreduce_microstep: 17.35 | step_microstep: 18.62
[2025-04-25 10:46:38,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5721.85 | bwd_inner: 5704.45 | bwd_allreduce: 17.37 | step: 18.62

  3%|▎         | 1165/41250 [2:49:04<96:19:56,  8.65s/it]
                                                         
{'loss': 0.1571, 'grad_norm': 1.3904242515563965, 'learning_rate': 3.764135702746365e-05, 'epoch': 0.28}

  3%|▎         | 1165/41250 [2:49:04<96:19:56,  8.65s/it][2025-04-25 10:46:47,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:46:47,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.09 | bwd_microstep: 5699.60 | bwd_inner_microstep: 5686.59 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.79
[2025-04-25 10:46:47,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.09 | bwd: 5699.62 | bwd_inner: 5686.59 | bwd_allreduce: 12.99 | step: 18.79

  3%|▎         | 1166/41250 [2:49:12<96:14:42,  8.64s/it]
                                                         
{'loss': 0.2128, 'grad_norm': 1.5736366510391235, 'learning_rate': 3.767366720516963e-05, 'epoch': 0.28}

  3%|▎         | 1166/41250 [2:49:12<96:14:42,  8.64s/it][2025-04-25 10:46:55,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 10:46:55,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.97 | bwd_microstep: 5747.53 | bwd_inner_microstep: 5641.08 | bwd_allreduce_microstep: 106.41 | step_microstep: 18.72
[2025-04-25 10:46:55,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.97 | bwd: 5747.55 | bwd_inner: 5641.08 | bwd_allreduce: 106.43 | step: 18.72

  3%|▎         | 1167/41250 [2:49:21<96:17:45,  8.65s/it]
                                                         
{'loss': 0.2516, 'grad_norm': 1.292633056640625, 'learning_rate': 3.770597738287561e-05, 'epoch': 0.28}

  3%|▎         | 1167/41250 [2:49:21<96:17:45,  8.65s/it][2025-04-25 10:47:04,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:47:04,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.25 | bwd_microstep: 5723.28 | bwd_inner_microstep: 5689.69 | bwd_allreduce_microstep: 33.55 | step_microstep: 18.60
[2025-04-25 10:47:04,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.25 | bwd: 5723.29 | bwd_inner: 5689.69 | bwd_allreduce: 33.57 | step: 18.61

  3%|▎         | 1168/41250 [2:49:29<96:17:47,  8.65s/it]
                                                         
{'loss': 0.208, 'grad_norm': 1.6532641649246216, 'learning_rate': 3.7738287560581585e-05, 'epoch': 0.28}

  3%|▎         | 1168/41250 [2:49:29<96:17:47,  8.65s/it][2025-04-25 10:47:13,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:47:13,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.15 | bwd_microstep: 5783.87 | bwd_inner_microstep: 5635.76 | bwd_allreduce_microstep: 148.06 | step_microstep: 18.57
[2025-04-25 10:47:13,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.15 | bwd: 5783.88 | bwd_inner: 5635.76 | bwd_allreduce: 148.08 | step: 18.57

  3%|▎         | 1169/41250 [2:49:38<96:25:42,  8.66s/it]
                                                         
{'loss': 0.3593, 'grad_norm': 2.2784221172332764, 'learning_rate': 3.777059773828756e-05, 'epoch': 0.28}

  3%|▎         | 1169/41250 [2:49:38<96:25:42,  8.66s/it][2025-04-25 10:47:21,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 10:47:21,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.60 | bwd_microstep: 5686.73 | bwd_inner_microstep: 5659.25 | bwd_allreduce_microstep: 27.43 | step_microstep: 19.08
[2025-04-25 10:47:21,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.60 | bwd: 5686.75 | bwd_inner: 5659.25 | bwd_allreduce: 27.46 | step: 19.08

  3%|▎         | 1170/41250 [2:49:47<96:14:49,  8.64s/it]
                                                         
{'loss': 0.2868, 'grad_norm': 1.9918721914291382, 'learning_rate': 3.780290791599354e-05, 'epoch': 0.28}

  3%|▎         | 1170/41250 [2:49:47<96:14:49,  8.64s/it][2025-04-25 10:47:30,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 10:47:30,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2812.65 | bwd_microstep: 5885.80 | bwd_inner_microstep: 5622.21 | bwd_allreduce_microstep: 263.54 | step_microstep: 19.07
[2025-04-25 10:47:30,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2812.66 | bwd: 5885.82 | bwd_inner: 5622.21 | bwd_allreduce: 263.56 | step: 19.07

  3%|▎         | 1171/41250 [2:49:56<96:42:22,  8.69s/it]
                                                         
{'loss': 0.2144, 'grad_norm': 1.5634918212890625, 'learning_rate': 3.783521809369952e-05, 'epoch': 0.28}

  3%|▎         | 1171/41250 [2:49:56<96:42:22,  8.69s/it][2025-04-25 10:47:39,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:47:39,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.10 | bwd_microstep: 5712.51 | bwd_inner_microstep: 5673.64 | bwd_allreduce_microstep: 38.82 | step_microstep: 18.39
[2025-04-25 10:47:39,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.10 | bwd: 5712.52 | bwd_inner: 5673.64 | bwd_allreduce: 38.84 | step: 18.39

  3%|▎         | 1172/41250 [2:50:04<96:33:37,  8.67s/it]
                                                         
{'loss': 0.2318, 'grad_norm': 2.49172043800354, 'learning_rate': 3.78675282714055e-05, 'epoch': 0.28}

  3%|▎         | 1172/41250 [2:50:04<96:33:37,  8.67s/it][2025-04-25 10:47:48,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:47:48,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.65 | bwd_microstep: 5761.97 | bwd_inner_microstep: 5643.08 | bwd_allreduce_microstep: 118.85 | step_microstep: 18.80
[2025-04-25 10:47:48,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.65 | bwd: 5761.99 | bwd_inner: 5643.08 | bwd_allreduce: 118.86 | step: 18.80

  3%|▎         | 1173/41250 [2:50:13<96:34:33,  8.68s/it]
                                                         
{'loss': 0.3206, 'grad_norm': 2.1127982139587402, 'learning_rate': 3.7899838449111475e-05, 'epoch': 0.28}

  3%|▎         | 1173/41250 [2:50:13<96:34:33,  8.68s/it][2025-04-25 10:47:56,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:47:56,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.92 | bwd_microstep: 5746.78 | bwd_inner_microstep: 5668.49 | bwd_allreduce_microstep: 78.24 | step_microstep: 18.79
[2025-04-25 10:47:56,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.92 | bwd: 5746.79 | bwd_inner: 5668.49 | bwd_allreduce: 78.26 | step: 18.79

  3%|▎         | 1174/41250 [2:50:22<96:33:14,  8.67s/it]
                                                         
{'loss': 0.2188, 'grad_norm': 1.3996633291244507, 'learning_rate': 3.7932148626817446e-05, 'epoch': 0.28}

  3%|▎         | 1174/41250 [2:50:22<96:33:14,  8.67s/it][2025-04-25 10:48:05,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 10:48:05,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.22 | bwd_microstep: 5697.96 | bwd_inner_microstep: 5628.64 | bwd_allreduce_microstep: 69.26 | step_microstep: 19.11
[2025-04-25 10:48:05,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.22 | bwd: 5697.97 | bwd_inner: 5628.64 | bwd_allreduce: 69.29 | step: 19.12

  3%|▎         | 1175/41250 [2:50:30<96:18:43,  8.65s/it]
                                                         
{'loss': 0.1838, 'grad_norm': 1.2835122346878052, 'learning_rate': 3.796445880452343e-05, 'epoch': 0.28}

  3%|▎         | 1175/41250 [2:50:30<96:18:43,  8.65s/it][2025-04-25 10:48:13,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 10:48:13,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.12 | bwd_microstep: 5714.05 | bwd_inner_microstep: 5690.92 | bwd_allreduce_microstep: 23.08 | step_microstep: 18.54
[2025-04-25 10:48:13,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.12 | bwd: 5714.06 | bwd_inner: 5690.92 | bwd_allreduce: 23.10 | step: 18.54

  3%|▎         | 1176/41250 [2:50:39<96:16:48,  8.65s/it]
                                                         
{'loss': 0.3785, 'grad_norm': 2.925647020339966, 'learning_rate': 3.799676898222941e-05, 'epoch': 0.29}

  3%|▎         | 1176/41250 [2:50:39<96:16:48,  8.65s/it][2025-04-25 10:48:22,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 10:48:22,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.84 | bwd_microstep: 5692.84 | bwd_inner_microstep: 5679.99 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.47
[2025-04-25 10:48:22,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.84 | bwd: 5692.86 | bwd_inner: 5679.99 | bwd_allreduce: 12.83 | step: 18.48

  3%|▎         | 1177/41250 [2:50:47<96:11:50,  8.64s/it]
                                                         
{'loss': 0.2611, 'grad_norm': 1.4750663042068481, 'learning_rate': 3.8029079159935386e-05, 'epoch': 0.29}

  3%|▎         | 1177/41250 [2:50:47<96:11:50,  8.64s/it][2025-04-25 10:48:31,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:48:31,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.92 | bwd_microstep: 5732.81 | bwd_inner_microstep: 5683.19 | bwd_allreduce_microstep: 49.57 | step_microstep: 18.62
[2025-04-25 10:48:31,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.92 | bwd: 5732.83 | bwd_inner: 5683.19 | bwd_allreduce: 49.59 | step: 18.62

  3%|▎         | 1178/41250 [2:50:56<96:16:08,  8.65s/it]
                                                         
{'loss': 0.0826, 'grad_norm': 1.0264400243759155, 'learning_rate': 3.806138933764136e-05, 'epoch': 0.29}

  3%|▎         | 1178/41250 [2:50:56<96:16:08,  8.65s/it][2025-04-25 10:48:39,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:48:39,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.99 | bwd_microstep: 5746.86 | bwd_inner_microstep: 5646.43 | bwd_allreduce_microstep: 100.39 | step_microstep: 18.29
[2025-04-25 10:48:39,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.99 | bwd: 5746.88 | bwd_inner: 5646.43 | bwd_allreduce: 100.41 | step: 18.29

  3%|▎         | 1179/41250 [2:51:05<96:16:11,  8.65s/it]
                                                         
{'loss': 0.1186, 'grad_norm': 0.5146782994270325, 'learning_rate': 3.8093699515347335e-05, 'epoch': 0.29}

  3%|▎         | 1179/41250 [2:51:05<96:16:11,  8.65s/it][2025-04-25 10:48:48,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 10:48:48,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.87 | bwd_microstep: 5772.84 | bwd_inner_microstep: 5632.43 | bwd_allreduce_microstep: 140.35 | step_microstep: 18.83
[2025-04-25 10:48:48,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.87 | bwd: 5772.86 | bwd_inner: 5632.43 | bwd_allreduce: 140.38 | step: 18.83

  3%|▎         | 1180/41250 [2:51:13<96:21:15,  8.66s/it]
                                                         
{'loss': 0.2768, 'grad_norm': 2.2471234798431396, 'learning_rate': 3.812600969305331e-05, 'epoch': 0.29}

  3%|▎         | 1180/41250 [2:51:13<96:21:15,  8.66s/it][2025-04-25 10:48:57,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 10:48:57,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.51 | bwd_microstep: 5920.49 | bwd_inner_microstep: 5640.02 | bwd_allreduce_microstep: 280.42 | step_microstep: 18.41
[2025-04-25 10:48:57,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.51 | bwd: 5920.50 | bwd_inner: 5640.02 | bwd_allreduce: 280.44 | step: 18.41

  3%|▎         | 1181/41250 [2:51:22<96:55:35,  8.71s/it]
                                                         
{'loss': 0.0885, 'grad_norm': 0.6261923313140869, 'learning_rate': 3.815831987075929e-05, 'epoch': 0.29}

  3%|▎         | 1181/41250 [2:51:22<96:55:35,  8.71s/it][2025-04-25 10:49:06,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:49:06,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.12 | bwd_microstep: 5748.64 | bwd_inner_microstep: 5638.11 | bwd_allreduce_microstep: 110.48 | step_microstep: 18.29
[2025-04-25 10:49:06,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.12 | bwd: 5748.65 | bwd_inner: 5638.11 | bwd_allreduce: 110.50 | step: 18.29

  3%|▎         | 1182/41250 [2:51:31<96:45:25,  8.69s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 1.2095040082931519, 'learning_rate': 3.819063004846527e-05, 'epoch': 0.29}

  3%|▎         | 1182/41250 [2:51:31<96:45:25,  8.69s/it][2025-04-25 10:49:14,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.81 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:49:14,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.49 | bwd_microstep: 5992.65 | bwd_inner_microstep: 5678.76 | bwd_allreduce_microstep: 313.84 | step_microstep: 17.76
[2025-04-25 10:49:14,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.49 | bwd: 5992.66 | bwd_inner: 5678.76 | bwd_allreduce: 313.86 | step: 17.76

  3%|▎         | 1183/41250 [2:51:40<97:30:11,  8.76s/it]
                                                         
{'loss': 0.2193, 'grad_norm': 1.5941625833511353, 'learning_rate': 3.822294022617125e-05, 'epoch': 0.29}

  3%|▎         | 1183/41250 [2:51:40<97:30:11,  8.76s/it][2025-04-25 10:49:23,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:49:23,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2927.27 | bwd_microstep: 5866.94 | bwd_inner_microstep: 5854.06 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.59
[2025-04-25 10:49:23,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2927.27 | bwd: 5866.96 | bwd_inner: 5854.06 | bwd_allreduce: 12.86 | step: 18.59

  3%|▎         | 1184/41250 [2:51:49<97:53:20,  8.80s/it]
                                                         
{'loss': 0.3263, 'grad_norm': 1.8604501485824585, 'learning_rate': 3.8255250403877225e-05, 'epoch': 0.29}

  3%|▎         | 1184/41250 [2:51:49<97:53:20,  8.80s/it][2025-04-25 10:49:32,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 10:49:32,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.19 | bwd_microstep: 5764.69 | bwd_inner_microstep: 5638.89 | bwd_allreduce_microstep: 125.75 | step_microstep: 18.90
[2025-04-25 10:49:32,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.19 | bwd: 5764.71 | bwd_inner: 5638.89 | bwd_allreduce: 125.77 | step: 18.90

  3%|▎         | 1185/41250 [2:51:57<97:30:17,  8.76s/it]
                                                         
{'loss': 0.0566, 'grad_norm': 0.5943048000335693, 'learning_rate': 3.82875605815832e-05, 'epoch': 0.29}

  3%|▎         | 1185/41250 [2:51:57<97:30:17,  8.76s/it][2025-04-25 10:49:41,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 10:49:41,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.83 | bwd_microstep: 5757.36 | bwd_inner_microstep: 5632.70 | bwd_allreduce_microstep: 124.61 | step_microstep: 19.11
[2025-04-25 10:49:41,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.83 | bwd: 5757.37 | bwd_inner: 5632.70 | bwd_allreduce: 124.63 | step: 19.12

  3%|▎         | 1186/41250 [2:52:06<97:09:26,  8.73s/it]
                                                         
{'loss': 0.3785, 'grad_norm': 2.94537615776062, 'learning_rate': 3.831987075928918e-05, 'epoch': 0.29}

  3%|▎         | 1186/41250 [2:52:06<97:09:26,  8.73s/it][2025-04-25 10:49:49,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 10:49:49,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.78 | bwd_microstep: 5790.36 | bwd_inner_microstep: 5640.61 | bwd_allreduce_microstep: 149.70 | step_microstep: 18.43
[2025-04-25 10:49:49,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.78 | bwd: 5790.37 | bwd_inner: 5640.61 | bwd_allreduce: 149.72 | step: 18.43

  3%|▎         | 1187/41250 [2:52:15<97:01:10,  8.72s/it]
                                                         
{'loss': 0.0868, 'grad_norm': 1.012959599494934, 'learning_rate': 3.835218093699516e-05, 'epoch': 0.29}

  3%|▎         | 1187/41250 [2:52:15<97:01:10,  8.72s/it][2025-04-25 10:49:58,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:49:58,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.85 | bwd_microstep: 5698.15 | bwd_inner_microstep: 5685.06 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.61
[2025-04-25 10:49:58,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.85 | bwd: 5698.16 | bwd_inner: 5685.06 | bwd_allreduce: 13.06 | step: 18.61

  3%|▎         | 1188/41250 [2:52:23<96:41:53,  8.69s/it]
                                                         
{'loss': 0.1712, 'grad_norm': 1.9347361326217651, 'learning_rate': 3.838449111470113e-05, 'epoch': 0.29}

  3%|▎         | 1188/41250 [2:52:23<96:41:53,  8.69s/it][2025-04-25 10:50:07,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:50:07,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.65 | bwd_microstep: 5676.00 | bwd_inner_microstep: 5659.36 | bwd_allreduce_microstep: 16.59 | step_microstep: 18.90
[2025-04-25 10:50:07,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.65 | bwd: 5676.01 | bwd_inner: 5659.36 | bwd_allreduce: 16.61 | step: 18.90

  3%|▎         | 1189/41250 [2:52:32<96:25:14,  8.66s/it]
                                                         
{'loss': 0.1586, 'grad_norm': 1.7920300960540771, 'learning_rate': 3.8416801292407114e-05, 'epoch': 0.29}

  3%|▎         | 1189/41250 [2:52:32<96:25:14,  8.66s/it][2025-04-25 10:50:15,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 10:50:15,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.87 | bwd_microstep: 5714.51 | bwd_inner_microstep: 5701.74 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.13
[2025-04-25 10:50:15,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.87 | bwd: 5714.52 | bwd_inner: 5701.74 | bwd_allreduce: 12.74 | step: 19.13

  3%|▎         | 1190/41250 [2:52:41<96:21:25,  8.66s/it]
                                                         
{'loss': 0.1622, 'grad_norm': 1.6290000677108765, 'learning_rate': 3.844911147011309e-05, 'epoch': 0.29}

  3%|▎         | 1190/41250 [2:52:41<96:21:25,  8.66s/it][2025-04-25 10:50:24,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:50:24,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.51 | bwd_microstep: 5715.08 | bwd_inner_microstep: 5674.69 | bwd_allreduce_microstep: 40.35 | step_microstep: 18.44
[2025-04-25 10:50:24,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.51 | bwd: 5715.09 | bwd_inner: 5674.69 | bwd_allreduce: 40.37 | step: 18.45

  3%|▎         | 1191/41250 [2:52:49<96:15:26,  8.65s/it]
                                                         
{'loss': 0.2853, 'grad_norm': 1.9955472946166992, 'learning_rate': 3.848142164781907e-05, 'epoch': 0.29}

  3%|▎         | 1191/41250 [2:52:49<96:15:26,  8.65s/it][2025-04-25 10:50:33,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 10:50:33,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.14 | bwd_microstep: 5759.68 | bwd_inner_microstep: 5657.80 | bwd_allreduce_microstep: 101.84 | step_microstep: 18.84
[2025-04-25 10:50:33,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.14 | bwd: 5759.70 | bwd_inner: 5657.80 | bwd_allreduce: 101.86 | step: 18.84

  3%|▎         | 1192/41250 [2:52:58<96:20:52,  8.66s/it]
                                                         
{'loss': 0.0993, 'grad_norm': 1.0346168279647827, 'learning_rate': 3.851373182552504e-05, 'epoch': 0.29}

  3%|▎         | 1192/41250 [2:52:58<96:20:52,  8.66s/it][2025-04-25 10:50:41,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 10:50:41,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2938.42 | bwd_microstep: 5888.99 | bwd_inner_microstep: 5876.36 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.48
[2025-04-25 10:50:41,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2938.42 | bwd: 5889.00 | bwd_inner: 5876.36 | bwd_allreduce: 12.60 | step: 18.48

  3%|▎         | 1193/41250 [2:53:07<97:10:27,  8.73s/it]
                                                         
{'loss': 0.2722, 'grad_norm': 1.8428137302398682, 'learning_rate': 3.854604200323102e-05, 'epoch': 0.29}

  3%|▎         | 1193/41250 [2:53:07<97:10:27,  8.73s/it][2025-04-25 10:50:50,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:50:50,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5770.36 | bwd_inner_microstep: 5661.60 | bwd_allreduce_microstep: 108.72 | step_microstep: 18.62
[2025-04-25 10:50:50,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5770.37 | bwd_inner: 5661.60 | bwd_allreduce: 108.73 | step: 18.63

  3%|▎         | 1194/41250 [2:53:15<97:00:58,  8.72s/it]
                                                         
{'loss': 0.1869, 'grad_norm': 2.0920722484588623, 'learning_rate': 3.8578352180937e-05, 'epoch': 0.29}

  3%|▎         | 1194/41250 [2:53:15<97:00:58,  8.72s/it][2025-04-25 10:50:59,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 10:50:59,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.09 | bwd_microstep: 5753.16 | bwd_inner_microstep: 5719.01 | bwd_allreduce_microstep: 34.11 | step_microstep: 18.24
[2025-04-25 10:50:59,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.09 | bwd: 5753.17 | bwd_inner: 5719.01 | bwd_allreduce: 34.12 | step: 18.24

  3%|▎         | 1195/41250 [2:53:24<96:55:36,  8.71s/it]
                                                         
{'loss': 0.2428, 'grad_norm': 2.275728464126587, 'learning_rate': 3.8610662358642975e-05, 'epoch': 0.29}

  3%|▎         | 1195/41250 [2:53:24<96:55:36,  8.71s/it][2025-04-25 10:51:08,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:51:08,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.40 | bwd_microstep: 5755.69 | bwd_inner_microstep: 5705.81 | bwd_allreduce_microstep: 49.83 | step_microstep: 18.61
[2025-04-25 10:51:08,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.40 | bwd: 5755.70 | bwd_inner: 5705.81 | bwd_allreduce: 49.85 | step: 18.61

  3%|▎         | 1196/41250 [2:53:33<96:51:37,  8.71s/it]
                                                         
{'loss': 0.1771, 'grad_norm': 1.2695800065994263, 'learning_rate': 3.864297253634895e-05, 'epoch': 0.29}

  3%|▎         | 1196/41250 [2:53:33<96:51:37,  8.71s/it][2025-04-25 10:51:16,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 10:51:16,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5727.79 | bwd_inner_microstep: 5650.73 | bwd_allreduce_microstep: 77.01 | step_microstep: 18.11
[2025-04-25 10:51:16,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5727.81 | bwd_inner: 5650.74 | bwd_allreduce: 77.03 | step: 18.11

  3%|▎         | 1197/41250 [2:53:41<96:37:59,  8.69s/it]
                                                         
{'loss': 0.1154, 'grad_norm': 1.5258227586746216, 'learning_rate': 3.867528271405493e-05, 'epoch': 0.29}

  3%|▎         | 1197/41250 [2:53:41<96:37:59,  8.69s/it][2025-04-25 10:51:25,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:51:25,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.09 | bwd_microstep: 5793.09 | bwd_inner_microstep: 5669.97 | bwd_allreduce_microstep: 123.08 | step_microstep: 18.12
[2025-04-25 10:51:25,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.08 | bwd: 5793.11 | bwd_inner: 5669.97 | bwd_allreduce: 123.10 | step: 18.12

  3%|▎         | 1198/41250 [2:53:50<96:44:06,  8.69s/it]
                                                         
{'loss': 0.0717, 'grad_norm': 1.213075041770935, 'learning_rate': 3.870759289176091e-05, 'epoch': 0.29}

  3%|▎         | 1198/41250 [2:53:50<96:44:06,  8.69s/it][2025-04-25 10:51:34,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-25 10:51:34,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.61 | bwd_microstep: 5707.78 | bwd_inner_microstep: 5658.19 | bwd_allreduce_microstep: 49.55 | step_microstep: 18.39
[2025-04-25 10:51:34,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.61 | bwd: 5707.80 | bwd_inner: 5658.19 | bwd_allreduce: 49.57 | step: 18.39

  3%|▎         | 1199/41250 [2:53:59<96:28:57,  8.67s/it]
                                                         
{'loss': 0.1206, 'grad_norm': 1.5785163640975952, 'learning_rate': 3.8739903069466886e-05, 'epoch': 0.29}

  3%|▎         | 1199/41250 [2:53:59<96:28:57,  8.67s/it][2025-04-25 10:51:42,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 10:51:42,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.34 | bwd_microstep: 5761.65 | bwd_inner_microstep: 5699.43 | bwd_allreduce_microstep: 62.18 | step_microstep: 18.36
[2025-04-25 10:51:42,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.34 | bwd: 5761.67 | bwd_inner: 5699.43 | bwd_allreduce: 62.20 | step: 18.36

  3%|▎         | 1200/41250 [2:54:08<96:34:01,  8.68s/it]
                                                         
{'loss': 0.2565, 'grad_norm': 2.056305408477783, 'learning_rate': 3.8772213247172864e-05, 'epoch': 0.29}

  3%|▎         | 1200/41250 [2:54:08<96:34:01,  8.68s/it][2025-04-25 10:51:51,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:51:51,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5772.97 | bwd_inner_microstep: 5689.59 | bwd_allreduce_microstep: 83.33 | step_microstep: 18.29
[2025-04-25 10:51:51,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5772.98 | bwd_inner: 5689.59 | bwd_allreduce: 83.35 | step: 18.29

  3%|▎         | 1201/41250 [2:54:16<96:39:36,  8.69s/it]
                                                         
{'loss': 0.4509, 'grad_norm': 3.8163092136383057, 'learning_rate': 3.880452342487884e-05, 'epoch': 0.29}

  3%|▎         | 1201/41250 [2:54:16<96:39:36,  8.69s/it][2025-04-25 10:52:00,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 10:52:00,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.19 | bwd_microstep: 5723.34 | bwd_inner_microstep: 5710.74 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.37
[2025-04-25 10:52:00,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.19 | bwd: 5723.35 | bwd_inner: 5710.74 | bwd_allreduce: 12.57 | step: 18.38

  3%|▎         | 1202/41250 [2:54:25<96:33:32,  8.68s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 1.3728392124176025, 'learning_rate': 3.883683360258481e-05, 'epoch': 0.29}

  3%|▎         | 1202/41250 [2:54:25<96:33:32,  8.68s/it][2025-04-25 10:52:08,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:52:08,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.08 | bwd_microstep: 5710.98 | bwd_inner_microstep: 5698.21 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.50
[2025-04-25 10:52:08,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.08 | bwd: 5710.99 | bwd_inner: 5698.21 | bwd_allreduce: 12.73 | step: 18.50

  3%|▎         | 1203/41250 [2:54:34<96:27:36,  8.67s/it]
                                                         
{'loss': 0.1016, 'grad_norm': 1.025420904159546, 'learning_rate': 3.88691437802908e-05, 'epoch': 0.29}

  3%|▎         | 1203/41250 [2:54:34<96:27:36,  8.67s/it][2025-04-25 10:52:17,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:52:17,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.40 | bwd_microstep: 5730.30 | bwd_inner_microstep: 5665.98 | bwd_allreduce_microstep: 64.28 | step_microstep: 18.31
[2025-04-25 10:52:17,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.40 | bwd: 5730.32 | bwd_inner: 5665.98 | bwd_allreduce: 64.29 | step: 18.31

  3%|▎         | 1204/41250 [2:54:42<96:21:58,  8.66s/it]
                                                         
{'loss': 0.2647, 'grad_norm': 1.5391104221343994, 'learning_rate': 3.8901453957996776e-05, 'epoch': 0.29}

  3%|▎         | 1204/41250 [2:54:42<96:21:58,  8.66s/it][2025-04-25 10:52:26,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:52:26,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.61 | bwd_microstep: 5776.13 | bwd_inner_microstep: 5660.32 | bwd_allreduce_microstep: 115.77 | step_microstep: 18.29
[2025-04-25 10:52:26,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.61 | bwd: 5776.14 | bwd_inner: 5660.32 | bwd_allreduce: 115.79 | step: 18.29

  3%|▎         | 1205/41250 [2:54:51<96:25:45,  8.67s/it]
                                                         
{'loss': 0.4408, 'grad_norm': 2.3165154457092285, 'learning_rate': 3.893376413570275e-05, 'epoch': 0.29}

  3%|▎         | 1205/41250 [2:54:51<96:25:45,  8.67s/it][2025-04-25 10:52:34,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:52:34,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.37 | bwd_microstep: 5717.50 | bwd_inner_microstep: 5704.48 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.63
[2025-04-25 10:52:34,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.37 | bwd: 5717.51 | bwd_inner: 5704.48 | bwd_allreduce: 12.99 | step: 18.63

  3%|▎         | 1206/41250 [2:55:00<96:23:45,  8.67s/it]
                                                         
{'loss': 0.1749, 'grad_norm': 1.8235136270523071, 'learning_rate': 3.8966074313408725e-05, 'epoch': 0.29}

  3%|▎         | 1206/41250 [2:55:00<96:23:45,  8.67s/it][2025-04-25 10:52:43,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 10:52:43,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.40 | bwd_microstep: 5711.77 | bwd_inner_microstep: 5698.96 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.47
[2025-04-25 10:52:43,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.40 | bwd: 5711.78 | bwd_inner: 5698.96 | bwd_allreduce: 12.78 | step: 19.47

  3%|▎         | 1207/41250 [2:55:08<96:19:25,  8.66s/it]
                                                         
{'loss': 0.2742, 'grad_norm': 2.3197085857391357, 'learning_rate': 3.89983844911147e-05, 'epoch': 0.29}

  3%|▎         | 1207/41250 [2:55:08<96:19:25,  8.66s/it][2025-04-25 10:52:52,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 10:52:52,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.41 | bwd_microstep: 5755.78 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 65.53 | step_microstep: 18.67
[2025-04-25 10:52:52,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.41 | bwd: 5755.80 | bwd_inner: 5690.21 | bwd_allreduce: 65.54 | step: 18.68

  3%|▎         | 1208/41250 [2:55:17<96:23:52,  8.67s/it]
                                                         
{'loss': 0.2665, 'grad_norm': 1.8146010637283325, 'learning_rate': 3.903069466882068e-05, 'epoch': 0.29}

  3%|▎         | 1208/41250 [2:55:17<96:23:52,  8.67s/it][2025-04-25 10:53:00,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:53:00,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.36 | bwd_microstep: 5770.70 | bwd_inner_microstep: 5663.00 | bwd_allreduce_microstep: 107.65 | step_microstep: 18.52
[2025-04-25 10:53:00,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.36 | bwd: 5770.71 | bwd_inner: 5663.00 | bwd_allreduce: 107.67 | step: 18.53

  3%|▎         | 1209/41250 [2:55:26<96:27:02,  8.67s/it]
                                                         
{'loss': 0.3463, 'grad_norm': 3.132262945175171, 'learning_rate': 3.906300484652666e-05, 'epoch': 0.29}

  3%|▎         | 1209/41250 [2:55:26<96:27:02,  8.67s/it][2025-04-25 10:53:09,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:53:09,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5714.04 | bwd_inner_microstep: 5701.19 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.89
[2025-04-25 10:53:09,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5714.05 | bwd_inner: 5701.19 | bwd_allreduce: 12.82 | step: 18.89

  3%|▎         | 1210/41250 [2:55:34<96:22:22,  8.66s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.556178331375122, 'learning_rate': 3.9095315024232636e-05, 'epoch': 0.29}

  3%|▎         | 1210/41250 [2:55:34<96:22:22,  8.66s/it][2025-04-25 10:53:18,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:53:18,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.50 | bwd_microstep: 5771.92 | bwd_inner_microstep: 5653.14 | bwd_allreduce_microstep: 118.73 | step_microstep: 18.44
[2025-04-25 10:53:18,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.50 | bwd: 5771.93 | bwd_inner: 5653.14 | bwd_allreduce: 118.75 | step: 18.44

  3%|▎         | 1211/41250 [2:55:43<96:24:11,  8.67s/it]
                                                         
{'loss': 0.3555, 'grad_norm': 2.3545873165130615, 'learning_rate': 3.9127625201938614e-05, 'epoch': 0.29}

  3%|▎         | 1211/41250 [2:55:43<96:24:11,  8.67s/it][2025-04-25 10:53:26,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:53:26,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.78 | bwd_microstep: 5792.85 | bwd_inner_microstep: 5780.08 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.75
[2025-04-25 10:53:26,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.78 | bwd: 5792.87 | bwd_inner: 5780.08 | bwd_allreduce: 12.74 | step: 18.75

  3%|▎         | 1212/41250 [2:55:52<96:43:50,  8.70s/it]
                                                         
{'loss': 0.1862, 'grad_norm': 1.2032980918884277, 'learning_rate': 3.915993537964459e-05, 'epoch': 0.29}

  3%|▎         | 1212/41250 [2:55:52<96:43:50,  8.70s/it][2025-04-25 10:53:35,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 10:53:35,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.83 | bwd_microstep: 5717.72 | bwd_inner_microstep: 5705.11 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.49
[2025-04-25 10:53:35,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.83 | bwd: 5717.74 | bwd_inner: 5705.11 | bwd_allreduce: 12.59 | step: 18.49

  3%|▎         | 1213/41250 [2:56:00<96:35:58,  8.69s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 0.7458154559135437, 'learning_rate': 3.919224555735057e-05, 'epoch': 0.29}

  3%|▎         | 1213/41250 [2:56:00<96:35:58,  8.69s/it][2025-04-25 10:53:44,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-25 10:53:44,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.04 | bwd_microstep: 5721.81 | bwd_inner_microstep: 5708.73 | bwd_allreduce_microstep: 13.04 | step_microstep: 19.13
[2025-04-25 10:53:44,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.04 | bwd: 5721.83 | bwd_inner: 5708.73 | bwd_allreduce: 13.06 | step: 19.13

  3%|▎         | 1214/41250 [2:56:09<96:31:34,  8.68s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 1.7477484941482544, 'learning_rate': 3.922455573505655e-05, 'epoch': 0.29}

  3%|▎         | 1214/41250 [2:56:09<96:31:34,  8.68s/it][2025-04-25 10:53:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:53:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.56 | bwd_microstep: 5754.47 | bwd_inner_microstep: 5697.01 | bwd_allreduce_microstep: 57.42 | step_microstep: 18.79
[2025-04-25 10:53:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.56 | bwd: 5754.48 | bwd_inner: 5697.01 | bwd_allreduce: 57.44 | step: 18.79

  3%|▎         | 1215/41250 [2:56:18<96:34:11,  8.68s/it]
                                                         
{'loss': 0.1663, 'grad_norm': 2.7133102416992188, 'learning_rate': 3.9256865912762526e-05, 'epoch': 0.29}

  3%|▎         | 1215/41250 [2:56:18<96:34:11,  8.68s/it][2025-04-25 10:54:01,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 10:54:01,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.00 | bwd_microstep: 5709.63 | bwd_inner_microstep: 5697.01 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.43
[2025-04-25 10:54:01,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.00 | bwd: 5709.64 | bwd_inner: 5697.01 | bwd_allreduce: 12.60 | step: 18.44

  3%|▎         | 1216/41250 [2:56:26<96:26:26,  8.67s/it]
                                                         
{'loss': 0.0746, 'grad_norm': 0.9623020887374878, 'learning_rate': 3.9289176090468497e-05, 'epoch': 0.29}

  3%|▎         | 1216/41250 [2:56:26<96:26:26,  8.67s/it][2025-04-25 10:54:10,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:54:10,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.58 | bwd_microstep: 5695.80 | bwd_inner_microstep: 5682.88 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.53
[2025-04-25 10:54:10,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.58 | bwd: 5695.81 | bwd_inner: 5682.88 | bwd_allreduce: 12.90 | step: 18.53

  3%|▎         | 1217/41250 [2:56:35<96:17:45,  8.66s/it]
                                                         
{'loss': 0.0934, 'grad_norm': 1.6966259479522705, 'learning_rate': 3.932148626817448e-05, 'epoch': 0.3}

  3%|▎         | 1217/41250 [2:56:35<96:17:45,  8.66s/it][2025-04-25 10:54:18,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 10:54:18,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.28 | bwd_microstep: 5746.40 | bwd_inner_microstep: 5654.55 | bwd_allreduce_microstep: 91.81 | step_microstep: 18.81
[2025-04-25 10:54:18,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.28 | bwd: 5746.41 | bwd_inner: 5654.54 | bwd_allreduce: 91.83 | step: 18.82

  3%|▎         | 1218/41250 [2:56:44<96:18:16,  8.66s/it]
                                                         
{'loss': 0.256, 'grad_norm': 1.6956042051315308, 'learning_rate': 3.935379644588046e-05, 'epoch': 0.3}

  3%|▎         | 1218/41250 [2:56:44<96:18:16,  8.66s/it][2025-04-25 10:54:27,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 10:54:27,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.17 | bwd_microstep: 5761.57 | bwd_inner_microstep: 5636.96 | bwd_allreduce_microstep: 124.57 | step_microstep: 18.70
[2025-04-25 10:54:27,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.17 | bwd: 5761.58 | bwd_inner: 5636.96 | bwd_allreduce: 124.58 | step: 18.70

  3%|▎         | 1219/41250 [2:56:52<96:21:40,  8.67s/it]
                                                         
{'loss': 0.2567, 'grad_norm': 2.130669593811035, 'learning_rate': 3.938610662358643e-05, 'epoch': 0.3}

  3%|▎         | 1219/41250 [2:56:52<96:21:40,  8.67s/it][2025-04-25 10:54:36,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:54:36,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.61 | bwd_microstep: 5737.02 | bwd_inner_microstep: 5684.58 | bwd_allreduce_microstep: 52.39 | step_microstep: 18.52
[2025-04-25 10:54:36,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.61 | bwd: 5737.03 | bwd_inner: 5684.59 | bwd_allreduce: 52.41 | step: 18.53

  3%|▎         | 1220/41250 [2:57:01<96:23:15,  8.67s/it]
                                                         
{'loss': 0.2493, 'grad_norm': 2.3609495162963867, 'learning_rate': 3.941841680129241e-05, 'epoch': 0.3}

  3%|▎         | 1220/41250 [2:57:01<96:23:15,  8.67s/it][2025-04-25 10:54:44,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:54:44,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.26 | bwd_microstep: 5769.31 | bwd_inner_microstep: 5700.53 | bwd_allreduce_microstep: 68.74 | step_microstep: 18.49
[2025-04-25 10:54:44,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.26 | bwd: 5769.33 | bwd_inner: 5700.52 | bwd_allreduce: 68.76 | step: 18.49

  3%|▎         | 1221/41250 [2:57:10<96:28:51,  8.68s/it]
                                                         
{'loss': 0.3903, 'grad_norm': 2.3570356369018555, 'learning_rate': 3.9450726978998386e-05, 'epoch': 0.3}

  3%|▎         | 1221/41250 [2:57:10<96:28:51,  8.68s/it][2025-04-25 10:54:53,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:54:53,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.69 | bwd_microstep: 5701.38 | bwd_inner_microstep: 5688.68 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.28
[2025-04-25 10:54:53,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.69 | bwd: 5701.40 | bwd_inner: 5688.68 | bwd_allreduce: 12.68 | step: 18.28

  3%|▎         | 1222/41250 [2:57:18<96:19:19,  8.66s/it]
                                                         
{'loss': 0.1847, 'grad_norm': 2.413745164871216, 'learning_rate': 3.9483037156704364e-05, 'epoch': 0.3}

  3%|▎         | 1222/41250 [2:57:18<96:19:19,  8.66s/it][2025-04-25 10:55:02,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:55:02,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.80 | bwd_microstep: 5773.99 | bwd_inner_microstep: 5644.06 | bwd_allreduce_microstep: 129.88 | step_microstep: 18.40
[2025-04-25 10:55:02,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.80 | bwd: 5774.01 | bwd_inner: 5644.06 | bwd_allreduce: 129.90 | step: 18.41

  3%|▎         | 1223/41250 [2:57:27<96:22:39,  8.67s/it]
                                                         
{'loss': 0.301, 'grad_norm': 1.9698879718780518, 'learning_rate': 3.951534733441034e-05, 'epoch': 0.3}

  3%|▎         | 1223/41250 [2:57:27<96:22:39,  8.67s/it][2025-04-25 10:55:10,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:55:10,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.00 | bwd_microstep: 5684.31 | bwd_inner_microstep: 5648.10 | bwd_allreduce_microstep: 36.16 | step_microstep: 18.70
[2025-04-25 10:55:10,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.00 | bwd: 5684.33 | bwd_inner: 5648.10 | bwd_allreduce: 36.18 | step: 18.71

  3%|▎         | 1224/41250 [2:57:36<96:08:46,  8.65s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 0.8723323941230774, 'learning_rate': 3.954765751211632e-05, 'epoch': 0.3}

  3%|▎         | 1224/41250 [2:57:36<96:08:46,  8.65s/it][2025-04-25 10:55:19,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:55:19,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.60 | bwd_microstep: 5768.52 | bwd_inner_microstep: 5642.45 | bwd_allreduce_microstep: 126.02 | step_microstep: 18.54
[2025-04-25 10:55:19,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.60 | bwd: 5768.53 | bwd_inner: 5642.45 | bwd_allreduce: 126.04 | step: 18.54

  3%|▎         | 1225/41250 [2:57:44<96:16:10,  8.66s/it]
                                                         
{'loss': 0.1614, 'grad_norm': 1.2569396495819092, 'learning_rate': 3.95799676898223e-05, 'epoch': 0.3}

  3%|▎         | 1225/41250 [2:57:44<96:16:10,  8.66s/it][2025-04-25 10:55:28,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 10:55:28,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.38 | bwd_microstep: 5726.24 | bwd_inner_microstep: 5671.53 | bwd_allreduce_microstep: 54.67 | step_microstep: 18.72
[2025-04-25 10:55:28,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.38 | bwd: 5726.25 | bwd_inner: 5671.53 | bwd_allreduce: 54.69 | step: 18.73

  3%|▎         | 1226/41250 [2:57:53<96:14:22,  8.66s/it]
                                                         
{'loss': 0.3023, 'grad_norm': 2.1252007484436035, 'learning_rate': 3.9612277867528275e-05, 'epoch': 0.3}

  3%|▎         | 1226/41250 [2:57:53<96:14:22,  8.66s/it][2025-04-25 10:55:36,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 10:55:36,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5774.79 | bwd_inner_microstep: 5638.36 | bwd_allreduce_microstep: 136.38 | step_microstep: 18.89
[2025-04-25 10:55:36,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.92 | bwd: 5774.80 | bwd_inner: 5638.36 | bwd_allreduce: 136.40 | step: 18.89

  3%|▎         | 1227/41250 [2:58:02<96:20:28,  8.67s/it]
                                                         
{'loss': 0.3284, 'grad_norm': 1.9728513956069946, 'learning_rate': 3.964458804523425e-05, 'epoch': 0.3}

  3%|▎         | 1227/41250 [2:58:02<96:20:28,  8.67s/it][2025-04-25 10:55:45,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 10:55:45,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5736.73 | bwd_inner_microstep: 5675.75 | bwd_allreduce_microstep: 60.93 | step_microstep: 18.78
[2025-04-25 10:55:45,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5736.74 | bwd_inner: 5675.75 | bwd_allreduce: 60.95 | step: 18.78

  3%|▎         | 1228/41250 [2:58:10<96:20:44,  8.67s/it]
                                                         
{'loss': 0.1644, 'grad_norm': 1.5039201974868774, 'learning_rate': 3.967689822294023e-05, 'epoch': 0.3}

  3%|▎         | 1228/41250 [2:58:10<96:20:44,  8.67s/it][2025-04-25 10:55:54,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 10:55:54,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.78 | bwd_microstep: 5684.07 | bwd_inner_microstep: 5634.24 | bwd_allreduce_microstep: 49.79 | step_microstep: 19.19
[2025-04-25 10:55:54,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.78 | bwd: 5684.09 | bwd_inner: 5634.24 | bwd_allreduce: 49.81 | step: 19.20

  3%|▎         | 1229/41250 [2:58:19<96:05:43,  8.64s/it]
                                                         
{'loss': 0.1221, 'grad_norm': 0.9012359380722046, 'learning_rate': 3.970920840064621e-05, 'epoch': 0.3}

  3%|▎         | 1229/41250 [2:58:19<96:05:43,  8.64s/it][2025-04-25 10:56:02,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 10:56:02,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.91 | bwd_microstep: 5724.61 | bwd_inner_microstep: 5664.30 | bwd_allreduce_microstep: 60.27 | step_microstep: 19.04
[2025-04-25 10:56:02,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.91 | bwd: 5724.63 | bwd_inner: 5664.30 | bwd_allreduce: 60.29 | step: 19.05

  3%|▎         | 1230/41250 [2:58:27<96:05:49,  8.64s/it]
                                                         
{'loss': 0.2654, 'grad_norm': 1.7321062088012695, 'learning_rate': 3.974151857835218e-05, 'epoch': 0.3}

  3%|▎         | 1230/41250 [2:58:27<96:05:49,  8.64s/it][2025-04-25 10:56:11,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:56:11,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.43 | bwd_microstep: 5758.10 | bwd_inner_microstep: 5692.31 | bwd_allreduce_microstep: 65.74 | step_microstep: 18.71
[2025-04-25 10:56:11,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.43 | bwd: 5758.11 | bwd_inner: 5692.31 | bwd_allreduce: 65.76 | step: 18.72

  3%|▎         | 1231/41250 [2:58:36<96:13:23,  8.66s/it]
                                                         
{'loss': 0.1379, 'grad_norm': 1.025160551071167, 'learning_rate': 3.9773828756058165e-05, 'epoch': 0.3}

  3%|▎         | 1231/41250 [2:58:36<96:13:23,  8.66s/it][2025-04-25 10:56:20,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.12 | optimizer_step: 1.06
[2025-04-25 10:56:20,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.95 | bwd_microstep: 5876.36 | bwd_inner_microstep: 5642.49 | bwd_allreduce_microstep: 233.82 | step_microstep: 19.87
[2025-04-25 10:56:20,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.95 | bwd: 5876.37 | bwd_inner: 5642.49 | bwd_allreduce: 233.84 | step: 19.87

  3%|▎         | 1232/41250 [2:58:45<96:39:18,  8.70s/it]
                                                         
{'loss': 0.3518, 'grad_norm': 2.0048465728759766, 'learning_rate': 3.980613893376414e-05, 'epoch': 0.3}

  3%|▎         | 1232/41250 [2:58:45<96:39:18,  8.70s/it][2025-04-25 10:56:28,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.16 | optimizer_step: 1.06
[2025-04-25 10:56:28,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.01 | bwd_microstep: 5664.71 | bwd_inner_microstep: 5627.94 | bwd_allreduce_microstep: 36.70 | step_microstep: 20.17
[2025-04-25 10:56:28,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.02 | bwd: 5664.73 | bwd_inner: 5627.94 | bwd_allreduce: 36.73 | step: 20.17

  3%|▎         | 1233/41250 [2:58:54<96:15:12,  8.66s/it]
                                                         
{'loss': 0.3128, 'grad_norm': 1.8603851795196533, 'learning_rate': 3.9838449111470114e-05, 'epoch': 0.3}

  3%|▎         | 1233/41250 [2:58:54<96:15:12,  8.66s/it][2025-04-25 10:56:37,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 10:56:37,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.46 | bwd_microstep: 5874.78 | bwd_inner_microstep: 5640.08 | bwd_allreduce_microstep: 234.64 | step_microstep: 19.11
[2025-04-25 10:56:37,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.46 | bwd: 5874.79 | bwd_inner: 5640.08 | bwd_allreduce: 234.66 | step: 19.11

  3%|▎         | 1234/41250 [2:59:02<96:42:18,  8.70s/it]
                                                         
{'loss': 0.2979, 'grad_norm': 1.4034662246704102, 'learning_rate': 3.987075928917609e-05, 'epoch': 0.3}

  3%|▎         | 1234/41250 [2:59:02<96:42:18,  8.70s/it][2025-04-25 10:56:46,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 10:56:46,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.36 | bwd_microstep: 5745.64 | bwd_inner_microstep: 5648.10 | bwd_allreduce_microstep: 97.49 | step_microstep: 18.14
[2025-04-25 10:56:46,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.36 | bwd: 5745.65 | bwd_inner: 5648.10 | bwd_allreduce: 97.51 | step: 18.14

  3%|▎         | 1235/41250 [2:59:11<96:33:14,  8.69s/it]
                                                         
{'loss': 0.158, 'grad_norm': 1.1693651676177979, 'learning_rate': 3.990306946688207e-05, 'epoch': 0.3}

  3%|▎         | 1235/41250 [2:59:11<96:33:14,  8.69s/it][2025-04-25 10:56:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 10:56:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.32 | bwd_microstep: 5874.75 | bwd_inner_microstep: 5861.91 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.98
[2025-04-25 10:56:55,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.32 | bwd: 5874.76 | bwd_inner: 5861.91 | bwd_allreduce: 12.82 | step: 18.98

  3%|▎         | 1236/41250 [2:59:20<97:14:40,  8.75s/it]
                                                         
{'loss': 0.2721, 'grad_norm': 1.890738606452942, 'learning_rate': 3.993537964458805e-05, 'epoch': 0.3}

  3%|▎         | 1236/41250 [2:59:20<97:14:40,  8.75s/it][2025-04-25 10:57:03,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 10:57:03,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.47 | bwd_microstep: 5687.27 | bwd_inner_microstep: 5674.57 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.68
[2025-04-25 10:57:03,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.47 | bwd: 5687.29 | bwd_inner: 5674.57 | bwd_allreduce: 12.68 | step: 18.68

  3%|▎         | 1237/41250 [2:59:28<96:46:54,  8.71s/it]
                                                         
{'loss': 0.2277, 'grad_norm': 1.4087952375411987, 'learning_rate': 3.9967689822294025e-05, 'epoch': 0.3}

  3%|▎         | 1237/41250 [2:59:28<96:46:54,  8.71s/it][2025-04-25 10:57:12,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 10:57:12,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5765.85 | bwd_inner_microstep: 5663.40 | bwd_allreduce_microstep: 102.41 | step_microstep: 18.27
[2025-04-25 10:57:12,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.59 | bwd: 5765.87 | bwd_inner: 5663.40 | bwd_allreduce: 102.43 | step: 18.27

  3%|▎         | 1238/41250 [2:59:37<96:42:20,  8.70s/it]
                                                         
{'loss': 0.1073, 'grad_norm': 1.1152762174606323, 'learning_rate': 4e-05, 'epoch': 0.3}

  3%|▎         | 1238/41250 [2:59:37<96:42:20,  8.70s/it][2025-04-25 10:57:21,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 10:57:21,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.64 | bwd_microstep: 5737.34 | bwd_inner_microstep: 5688.61 | bwd_allreduce_microstep: 48.68 | step_microstep: 19.03
[2025-04-25 10:57:21,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.65 | bwd: 5737.35 | bwd_inner: 5688.61 | bwd_allreduce: 48.70 | step: 19.03

  3%|▎         | 1239/41250 [2:59:46<96:35:58,  8.69s/it]
                                                         
{'loss': 0.369, 'grad_norm': 2.0997936725616455, 'learning_rate': 3.999999993835197e-05, 'epoch': 0.3}

  3%|▎         | 1239/41250 [2:59:46<96:35:58,  8.69s/it][2025-04-25 10:57:29,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 10:57:29,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5690.41 | bwd_inner_microstep: 5677.67 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.81
[2025-04-25 10:57:29,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5690.42 | bwd_inner: 5677.67 | bwd_allreduce: 12.71 | step: 18.82

  3%|▎         | 1240/41250 [2:59:54<96:22:41,  8.67s/it]
                                                         
{'loss': 0.0332, 'grad_norm': 0.3427841067314148, 'learning_rate': 3.999999975340787e-05, 'epoch': 0.3}

  3%|▎         | 1240/41250 [2:59:54<96:22:41,  8.67s/it][2025-04-25 10:57:38,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 10:57:38,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5729.75 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 29.53 | step_microstep: 18.63
[2025-04-25 10:57:38,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5729.76 | bwd_inner: 5700.17 | bwd_allreduce: 29.55 | step: 18.63

  3%|▎         | 1241/41250 [3:00:03<96:21:19,  8.67s/it]
                                                         
{'loss': 0.3124, 'grad_norm': 1.9742058515548706, 'learning_rate': 3.999999944516771e-05, 'epoch': 0.3}

  3%|▎         | 1241/41250 [3:00:03<96:21:19,  8.67s/it][2025-04-25 10:57:47,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 10:57:47,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.45 | bwd_microstep: 5775.72 | bwd_inner_microstep: 5762.85 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.33
[2025-04-25 10:57:47,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.45 | bwd: 5775.74 | bwd_inner: 5762.85 | bwd_allreduce: 12.85 | step: 18.34

  3%|▎         | 1242/41250 [3:00:12<96:35:49,  8.69s/it]
                                                         
{'loss': 0.1714, 'grad_norm': 2.037680149078369, 'learning_rate': 3.999999901363148e-05, 'epoch': 0.3}

  3%|▎         | 1242/41250 [3:00:12<96:35:49,  8.69s/it][2025-04-25 10:57:55,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 10:57:55,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.98 | bwd_microstep: 5868.15 | bwd_inner_microstep: 5702.68 | bwd_allreduce_microstep: 165.42 | step_microstep: 18.43
[2025-04-25 10:57:55,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.98 | bwd: 5868.16 | bwd_inner: 5702.68 | bwd_allreduce: 165.44 | step: 18.43

  3%|▎         | 1243/41250 [3:00:21<96:59:52,  8.73s/it]
                                                         
{'loss': 0.1991, 'grad_norm': 1.4045244455337524, 'learning_rate': 3.9999998458799195e-05, 'epoch': 0.3}

  3%|▎         | 1243/41250 [3:00:21<96:59:52,  8.73s/it][2025-04-25 10:58:04,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 10:58:04,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5661.12 | bwd_allreduce_microstep: 107.93 | step_microstep: 19.72
[2025-04-25 10:58:04,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5769.11 | bwd_inner: 5661.12 | bwd_allreduce: 107.95 | step: 19.73

  3%|▎         | 1244/41250 [3:00:29<96:51:06,  8.72s/it]
                                                         
{'loss': 0.2151, 'grad_norm': 1.3832427263259888, 'learning_rate': 3.999999778067085e-05, 'epoch': 0.3}

  3%|▎         | 1244/41250 [3:00:29<96:51:06,  8.72s/it][2025-04-25 10:58:13,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:58:13,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5764.54 | bwd_inner_microstep: 5663.31 | bwd_allreduce_microstep: 101.18 | step_microstep: 18.65
[2025-04-25 10:58:13,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5764.55 | bwd_inner: 5663.31 | bwd_allreduce: 101.20 | step: 18.65

  3%|▎         | 1245/41250 [3:00:38<96:44:13,  8.71s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.4135241508483887, 'learning_rate': 3.9999996979246454e-05, 'epoch': 0.3}

  3%|▎         | 1245/41250 [3:00:38<96:44:13,  8.71s/it][2025-04-25 10:58:21,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 10:58:21,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.44 | bwd_microstep: 5768.09 | bwd_inner_microstep: 5649.91 | bwd_allreduce_microstep: 118.13 | step_microstep: 18.75
[2025-04-25 10:58:21,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.44 | bwd: 5768.11 | bwd_inner: 5649.91 | bwd_allreduce: 118.15 | step: 18.75

  3%|▎         | 1246/41250 [3:00:47<96:38:09,  8.70s/it]
                                                         
{'loss': 0.2283, 'grad_norm': 1.8647748231887817, 'learning_rate': 3.999999605452601e-05, 'epoch': 0.3}

  3%|▎         | 1246/41250 [3:00:47<96:38:09,  8.70s/it][2025-04-25 10:58:30,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 10:58:30,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.99 | bwd_microstep: 5747.88 | bwd_inner_microstep: 5698.10 | bwd_allreduce_microstep: 49.73 | step_microstep: 18.90
[2025-04-25 10:58:30,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.99 | bwd: 5747.90 | bwd_inner: 5698.10 | bwd_allreduce: 49.75 | step: 18.91

  3%|▎         | 1247/41250 [3:00:55<96:34:43,  8.69s/it]
                                                         
{'loss': 0.1959, 'grad_norm': 1.3334007263183594, 'learning_rate': 3.9999995006509526e-05, 'epoch': 0.3}

  3%|▎         | 1247/41250 [3:00:55<96:34:43,  8.69s/it][2025-04-25 10:58:39,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 10:58:39,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.98 | bwd_microstep: 5703.84 | bwd_inner_microstep: 5691.06 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.70
[2025-04-25 10:58:39,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.98 | bwd: 5703.85 | bwd_inner: 5691.06 | bwd_allreduce: 12.75 | step: 18.71

  3%|▎         | 1248/41250 [3:01:04<96:22:37,  8.67s/it]
                                                         
{'loss': 0.3172, 'grad_norm': 1.992933988571167, 'learning_rate': 3.999999383519701e-05, 'epoch': 0.3}

  3%|▎         | 1248/41250 [3:01:04<96:22:37,  8.67s/it][2025-04-25 10:58:47,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 10:58:47,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.59 | bwd_microstep: 5715.05 | bwd_inner_microstep: 5662.26 | bwd_allreduce_microstep: 52.75 | step_microstep: 18.83
[2025-04-25 10:58:47,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.59 | bwd: 5715.06 | bwd_inner: 5662.26 | bwd_allreduce: 52.76 | step: 18.83

  3%|▎         | 1249/41250 [3:01:13<96:13:17,  8.66s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.216387152671814, 'learning_rate': 3.9999992540588456e-05, 'epoch': 0.3}

  3%|▎         | 1249/41250 [3:01:13<96:13:17,  8.66s/it][2025-04-25 10:58:56,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 10:58:56,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.43 | bwd_microstep: 5718.67 | bwd_inner_microstep: 5705.87 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.33
[2025-04-25 10:58:56,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.43 | bwd: 5718.68 | bwd_inner: 5705.87 | bwd_allreduce: 12.77 | step: 19.33

  3%|▎         | 1250/41250 [3:01:21<96:13:16,  8.66s/it]
                                                         
{'loss': 0.1621, 'grad_norm': 1.7790578603744507, 'learning_rate': 3.9999991122683885e-05, 'epoch': 0.3}

  3%|▎         | 1250/41250 [3:01:21<96:13:16,  8.66s/it][2025-04-25 10:59:05,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 10:59:05,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.58 | bwd_microstep: 5704.05 | bwd_inner_microstep: 5664.86 | bwd_allreduce_microstep: 39.13 | step_microstep: 18.89
[2025-04-25 10:59:05,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.58 | bwd: 5704.06 | bwd_inner: 5664.86 | bwd_allreduce: 39.15 | step: 18.89

  3%|▎         | 1251/41250 [3:01:30<96:04:29,  8.65s/it]
                                                         
{'loss': 0.2396, 'grad_norm': 2.4311277866363525, 'learning_rate': 3.999998958148331e-05, 'epoch': 0.3}

  3%|▎         | 1251/41250 [3:01:30<96:04:29,  8.65s/it][2025-04-25 10:59:13,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 10:59:13,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.74 | bwd_microstep: 5709.81 | bwd_inner_microstep: 5696.98 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.84
[2025-04-25 10:59:13,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.74 | bwd: 5709.83 | bwd_inner: 5696.98 | bwd_allreduce: 12.81 | step: 18.84

  3%|▎         | 1252/41250 [3:01:39<96:05:46,  8.65s/it]
                                                         
{'loss': 0.0863, 'grad_norm': 1.149512529373169, 'learning_rate': 3.999998791698672e-05, 'epoch': 0.3}

  3%|▎         | 1252/41250 [3:01:39<96:05:46,  8.65s/it][2025-04-25 10:59:22,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.17 | optimizer_step: 1.07
[2025-04-25 10:59:22,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.01 | bwd_microstep: 5801.14 | bwd_inner_microstep: 5663.47 | bwd_allreduce_microstep: 137.59 | step_microstep: 20.40
[2025-04-25 10:59:22,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.01 | bwd: 5801.15 | bwd_inner: 5663.47 | bwd_allreduce: 137.62 | step: 20.40

  3%|▎         | 1253/41250 [3:01:47<96:20:13,  8.67s/it]
                                                         
{'loss': 0.2658, 'grad_norm': 2.3831608295440674, 'learning_rate': 3.9999986129194146e-05, 'epoch': 0.3}

  3%|▎         | 1253/41250 [3:01:47<96:20:13,  8.67s/it][2025-04-25 10:59:31,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 10:59:31,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.96 | bwd_microstep: 5732.02 | bwd_inner_microstep: 5659.37 | bwd_allreduce_microstep: 72.61 | step_microstep: 19.04
[2025-04-25 10:59:31,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.96 | bwd: 5732.04 | bwd_inner: 5659.37 | bwd_allreduce: 72.63 | step: 19.04

  3%|▎         | 1254/41250 [3:01:56<96:14:45,  8.66s/it]
                                                         
{'loss': 0.1683, 'grad_norm': 1.4123804569244385, 'learning_rate': 3.9999984218105596e-05, 'epoch': 0.3}

  3%|▎         | 1254/41250 [3:01:56<96:14:45,  8.66s/it][2025-04-25 10:59:39,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:59:39,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.22 | bwd_microstep: 5915.64 | bwd_inner_microstep: 5667.48 | bwd_allreduce_microstep: 248.11 | step_microstep: 18.79
[2025-04-25 10:59:39,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.22 | bwd: 5915.65 | bwd_inner: 5667.48 | bwd_allreduce: 248.13 | step: 18.79

  3%|▎         | 1255/41250 [3:02:05<96:48:39,  8.71s/it]
                                                         
{'loss': 0.2634, 'grad_norm': 1.7019845247268677, 'learning_rate': 3.9999982183721074e-05, 'epoch': 0.3}

  3%|▎         | 1255/41250 [3:02:05<96:48:39,  8.71s/it][2025-04-25 10:59:48,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 10:59:48,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.70 | bwd_microstep: 5785.53 | bwd_inner_microstep: 5707.11 | bwd_allreduce_microstep: 78.38 | step_microstep: 18.93
[2025-04-25 10:59:48,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.70 | bwd: 5785.54 | bwd_inner: 5707.11 | bwd_allreduce: 78.40 | step: 18.94

  3%|▎         | 1256/41250 [3:02:14<96:50:08,  8.72s/it]
                                                         
{'loss': 0.0734, 'grad_norm': 0.718959629535675, 'learning_rate': 3.999998002604059e-05, 'epoch': 0.3}

  3%|▎         | 1256/41250 [3:02:14<96:50:08,  8.72s/it][2025-04-25 10:59:57,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 10:59:57,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.63 | bwd_microstep: 5716.38 | bwd_inner_microstep: 5659.42 | bwd_allreduce_microstep: 56.91 | step_microstep: 18.42
[2025-04-25 10:59:57,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.63 | bwd: 5716.39 | bwd_inner: 5659.42 | bwd_allreduce: 56.93 | step: 18.43

  3%|▎         | 1257/41250 [3:02:22<96:33:11,  8.69s/it]
                                                         
{'loss': 0.1866, 'grad_norm': 2.644481897354126, 'learning_rate': 3.999997774506417e-05, 'epoch': 0.3}

  3%|▎         | 1257/41250 [3:02:22<96:33:11,  8.69s/it][2025-04-25 11:00:06,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 1.18
[2025-04-25 11:00:06,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.18 | bwd_microstep: 5772.37 | bwd_inner_microstep: 5667.03 | bwd_allreduce_microstep: 105.28 | step_microstep: 19.23
[2025-04-25 11:00:06,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.18 | bwd: 5772.38 | bwd_inner: 5667.03 | bwd_allreduce: 105.31 | step: 19.23

  3%|▎         | 1258/41250 [3:02:31<96:32:53,  8.69s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 0.8833099007606506, 'learning_rate': 3.9999975340791816e-05, 'epoch': 0.3}

  3%|▎         | 1258/41250 [3:02:31<96:32:53,  8.69s/it][2025-04-25 11:00:14,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-25 11:00:14,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.71 | bwd_microstep: 5777.52 | bwd_inner_microstep: 5764.76 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.81
[2025-04-25 11:00:14,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.71 | bwd: 5777.54 | bwd_inner: 5764.76 | bwd_allreduce: 12.74 | step: 19.81

  3%|▎         | 1259/41250 [3:02:40<96:44:22,  8.71s/it]
                                                         
{'loss': 0.2811, 'grad_norm': 2.1358346939086914, 'learning_rate': 3.9999972813223546e-05, 'epoch': 0.31}

  3%|▎         | 1259/41250 [3:02:40<96:44:22,  8.71s/it][2025-04-25 11:00:23,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-25 11:00:23,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.13 | bwd_microstep: 5782.03 | bwd_inner_microstep: 5660.15 | bwd_allreduce_microstep: 121.82 | step_microstep: 19.02
[2025-04-25 11:00:23,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.13 | bwd: 5782.04 | bwd_inner: 5660.15 | bwd_allreduce: 121.84 | step: 19.02

  3%|▎         | 1260/41250 [3:02:48<96:43:27,  8.71s/it]
                                                         
{'loss': 0.2124, 'grad_norm': 1.6205638647079468, 'learning_rate': 3.9999970162359385e-05, 'epoch': 0.31}

  3%|▎         | 1260/41250 [3:02:48<96:43:27,  8.71s/it][2025-04-25 11:00:32,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 11:00:32,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5765.37 | bwd_inner_microstep: 5670.50 | bwd_allreduce_microstep: 94.83 | step_microstep: 19.11
[2025-04-25 11:00:32,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5765.38 | bwd_inner: 5670.50 | bwd_allreduce: 94.85 | step: 19.11

  3%|▎         | 1261/41250 [3:02:57<96:38:23,  8.70s/it]
                                                         
{'loss': 0.1597, 'grad_norm': 1.659406065940857, 'learning_rate': 3.999996738819934e-05, 'epoch': 0.31}

  3%|▎         | 1261/41250 [3:02:57<96:38:23,  8.70s/it][2025-04-25 11:00:40,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 11:00:40,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.77 | bwd_microstep: 5763.24 | bwd_inner_microstep: 5658.19 | bwd_allreduce_microstep: 104.99 | step_microstep: 19.23
[2025-04-25 11:00:40,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.77 | bwd: 5763.26 | bwd_inner: 5658.19 | bwd_allreduce: 105.01 | step: 19.23

  3%|▎         | 1262/41250 [3:03:06<96:34:52,  8.69s/it]
                                                         
{'loss': 0.2273, 'grad_norm': 1.5823872089385986, 'learning_rate': 3.9999964490743423e-05, 'epoch': 0.31}

  3%|▎         | 1262/41250 [3:03:06<96:34:52,  8.69s/it][2025-04-25 11:00:49,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:00:49,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.26 | bwd_microstep: 5783.01 | bwd_inner_microstep: 5646.98 | bwd_allreduce_microstep: 135.98 | step_microstep: 18.92
[2025-04-25 11:00:49,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.26 | bwd: 5783.03 | bwd_inner: 5646.98 | bwd_allreduce: 136.00 | step: 18.92

  3%|▎         | 1263/41250 [3:03:14<96:36:06,  8.70s/it]
                                                         
{'loss': 0.1048, 'grad_norm': 1.3850374221801758, 'learning_rate': 3.9999961469991664e-05, 'epoch': 0.31}

  3%|▎         | 1263/41250 [3:03:14<96:36:06,  8.70s/it][2025-04-25 11:00:58,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:00:58,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.28 | bwd_microstep: 5778.39 | bwd_inner_microstep: 5690.67 | bwd_allreduce_microstep: 87.67 | step_microstep: 18.76
[2025-04-25 11:00:58,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.28 | bwd: 5778.41 | bwd_inner: 5690.67 | bwd_allreduce: 87.69 | step: 18.77

  3%|▎         | 1264/41250 [3:03:23<96:40:12,  8.70s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 2.3041000366210938, 'learning_rate': 3.9999958325944073e-05, 'epoch': 0.31}

  3%|▎         | 1264/41250 [3:03:23<96:40:12,  8.70s/it][2025-04-25 11:01:06,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 11:01:06,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.27 | bwd_microstep: 5785.03 | bwd_inner_microstep: 5689.92 | bwd_allreduce_microstep: 95.07 | step_microstep: 19.45
[2025-04-25 11:01:06,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.27 | bwd: 5785.05 | bwd_inner: 5689.92 | bwd_allreduce: 95.09 | step: 19.45

  3%|▎         | 1265/41250 [3:03:32<96:42:23,  8.71s/it]
                                                         
{'loss': 0.3105, 'grad_norm': 1.9774171113967896, 'learning_rate': 3.9999955058600674e-05, 'epoch': 0.31}

  3%|▎         | 1265/41250 [3:03:32<96:42:23,  8.71s/it][2025-04-25 11:01:15,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:01:15,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.25 | bwd_microstep: 5880.95 | bwd_inner_microstep: 5648.38 | bwd_allreduce_microstep: 232.52 | step_microstep: 18.93
[2025-04-25 11:01:15,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.25 | bwd: 5880.97 | bwd_inner: 5648.38 | bwd_allreduce: 232.54 | step: 18.93

  3%|▎         | 1266/41250 [3:03:41<97:00:11,  8.73s/it]
                                                         
{'loss': 0.1376, 'grad_norm': 1.7748068571090698, 'learning_rate': 3.999995166796149e-05, 'epoch': 0.31}

  3%|▎         | 1266/41250 [3:03:41<97:00:11,  8.73s/it][2025-04-25 11:01:24,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:01:24,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.55 | bwd_microstep: 5727.80 | bwd_inner_microstep: 5714.88 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.96
[2025-04-25 11:01:24,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.55 | bwd: 5727.81 | bwd_inner: 5714.88 | bwd_allreduce: 12.88 | step: 18.97

  3%|▎         | 1267/41250 [3:03:49<96:46:58,  8.71s/it]
                                                         
{'loss': 0.0582, 'grad_norm': 0.7325924634933472, 'learning_rate': 3.999994815402653e-05, 'epoch': 0.31}

  3%|▎         | 1267/41250 [3:03:49<96:46:58,  8.71s/it][2025-04-25 11:01:33,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.14 | optimizer_step: 1.07
[2025-04-25 11:01:33,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.11 | bwd_microstep: 5741.04 | bwd_inner_microstep: 5696.97 | bwd_allreduce_microstep: 44.02 | step_microstep: 20.30
[2025-04-25 11:01:33,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.11 | bwd: 5741.06 | bwd_inner: 5696.97 | bwd_allreduce: 44.04 | step: 20.31

  3%|▎         | 1268/41250 [3:03:58<96:39:14,  8.70s/it]
                                                         
{'loss': 0.1679, 'grad_norm': 1.725770115852356, 'learning_rate': 3.999994451679583e-05, 'epoch': 0.31}

  3%|▎         | 1268/41250 [3:03:58<96:39:14,  8.70s/it][2025-04-25 11:01:41,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 11:01:41,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.82 | bwd_microstep: 5722.38 | bwd_inner_microstep: 5657.88 | bwd_allreduce_microstep: 64.45 | step_microstep: 18.95
[2025-04-25 11:01:41,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.82 | bwd: 5722.40 | bwd_inner: 5657.88 | bwd_allreduce: 64.47 | step: 18.96

  3%|▎         | 1269/41250 [3:04:07<96:25:51,  8.68s/it]
                                                         
{'loss': 0.3648, 'grad_norm': 2.2606208324432373, 'learning_rate': 3.999994075626941e-05, 'epoch': 0.31}

  3%|▎         | 1269/41250 [3:04:07<96:25:51,  8.68s/it][2025-04-25 11:01:50,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 11:01:50,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.32 | bwd_microstep: 5754.66 | bwd_inner_microstep: 5643.38 | bwd_allreduce_microstep: 111.23 | step_microstep: 19.28
[2025-04-25 11:01:50,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.32 | bwd: 5754.68 | bwd_inner: 5643.38 | bwd_allreduce: 111.25 | step: 19.29

  3%|▎         | 1270/41250 [3:04:15<96:21:54,  8.68s/it]
                                                         
{'loss': 0.0735, 'grad_norm': 1.0562984943389893, 'learning_rate': 3.999993687244728e-05, 'epoch': 0.31}

  3%|▎         | 1270/41250 [3:04:15<96:21:54,  8.68s/it][2025-04-25 11:01:59,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 11:01:59,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.00 | bwd_microstep: 5716.70 | bwd_inner_microstep: 5701.44 | bwd_allreduce_microstep: 15.22 | step_microstep: 18.85
[2025-04-25 11:01:59,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.00 | bwd: 5716.72 | bwd_inner: 5701.44 | bwd_allreduce: 15.24 | step: 18.86

  3%|▎         | 1271/41250 [3:04:24<96:18:41,  8.67s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 1.8935807943344116, 'learning_rate': 3.999993286532947e-05, 'epoch': 0.31}

  3%|▎         | 1271/41250 [3:04:24<96:18:41,  8.67s/it][2025-04-25 11:02:07,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:02:07,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.54 | bwd_microstep: 5729.17 | bwd_inner_microstep: 5691.08 | bwd_allreduce_microstep: 38.04 | step_microstep: 18.83
[2025-04-25 11:02:07,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.54 | bwd: 5729.18 | bwd_inner: 5691.08 | bwd_allreduce: 38.06 | step: 18.83

  3%|▎         | 1272/41250 [3:04:33<96:15:35,  8.67s/it]
                                                         
{'loss': 0.14, 'grad_norm': 1.0257924795150757, 'learning_rate': 3.999992873491602e-05, 'epoch': 0.31}

  3%|▎         | 1272/41250 [3:04:33<96:15:35,  8.67s/it][2025-04-25 11:02:16,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 11:02:16,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.44 | bwd_microstep: 5693.76 | bwd_inner_microstep: 5680.16 | bwd_allreduce_microstep: 13.56 | step_microstep: 18.92
[2025-04-25 11:02:16,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.44 | bwd: 5693.77 | bwd_inner: 5680.15 | bwd_allreduce: 13.57 | step: 18.92

  3%|▎         | 1273/41250 [3:04:41<96:06:42,  8.66s/it]
                                                         
{'loss': 0.36, 'grad_norm': 2.3885345458984375, 'learning_rate': 3.9999924481206934e-05, 'epoch': 0.31}

  3%|▎         | 1273/41250 [3:04:41<96:06:42,  8.66s/it][2025-04-25 11:02:25,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 11:02:25,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5753.64 | bwd_inner_microstep: 5690.99 | bwd_allreduce_microstep: 62.60 | step_microstep: 19.38
[2025-04-25 11:02:25,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.00 | bwd: 5753.65 | bwd_inner: 5690.99 | bwd_allreduce: 62.62 | step: 19.39

  3%|▎         | 1274/41250 [3:04:50<96:13:52,  8.67s/it]
                                                         
{'loss': 0.4092, 'grad_norm': 2.143672466278076, 'learning_rate': 3.999992010420225e-05, 'epoch': 0.31}

  3%|▎         | 1274/41250 [3:04:50<96:13:52,  8.67s/it][2025-04-25 11:02:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:02:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.46 | bwd_microstep: 5690.09 | bwd_inner_microstep: 5653.10 | bwd_allreduce_microstep: 36.95 | step_microstep: 18.89
[2025-04-25 11:02:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.46 | bwd: 5690.10 | bwd_inner: 5653.10 | bwd_allreduce: 36.96 | step: 18.89

  3%|▎         | 1275/41250 [3:04:58<96:01:15,  8.65s/it]
                                                         
{'loss': 0.1655, 'grad_norm': 1.1950899362564087, 'learning_rate': 3.999991560390199e-05, 'epoch': 0.31}

  3%|▎         | 1275/41250 [3:04:58<96:01:15,  8.65s/it][2025-04-25 11:02:42,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 11:02:42,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.71 | bwd_microstep: 5779.96 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 121.04 | step_microstep: 19.25
[2025-04-25 11:02:42,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.71 | bwd: 5779.97 | bwd_inner: 5658.86 | bwd_allreduce: 121.06 | step: 19.25

  3%|▎         | 1276/41250 [3:05:07<96:09:36,  8.66s/it]
                                                         
{'loss': 0.2008, 'grad_norm': 1.5891315937042236, 'learning_rate': 3.999991098030619e-05, 'epoch': 0.31}

  3%|▎         | 1276/41250 [3:05:07<96:09:36,  8.66s/it][2025-04-25 11:02:51,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:02:51,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.36 | bwd_microstep: 5743.10 | bwd_inner_microstep: 5643.45 | bwd_allreduce_microstep: 99.60 | step_microstep: 18.94
[2025-04-25 11:02:51,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.36 | bwd: 5743.11 | bwd_inner: 5643.45 | bwd_allreduce: 99.62 | step: 18.94

  3%|▎         | 1277/41250 [3:05:16<96:10:45,  8.66s/it]
                                                         
{'loss': 0.1818, 'grad_norm': 1.554761528968811, 'learning_rate': 3.999990623341487e-05, 'epoch': 0.31}

  3%|▎         | 1277/41250 [3:05:16<96:10:45,  8.66s/it][2025-04-25 11:02:59,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 11:02:59,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5712.09 | bwd_inner_microstep: 5699.54 | bwd_allreduce_microstep: 12.49 | step_microstep: 19.62
[2025-04-25 11:02:59,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5712.10 | bwd_inner: 5699.54 | bwd_allreduce: 12.51 | step: 19.63

  3%|▎         | 1278/41250 [3:05:24<96:07:56,  8.66s/it]
                                                         
{'loss': 0.1606, 'grad_norm': 1.3437589406967163, 'learning_rate': 3.9999901363228056e-05, 'epoch': 0.31}

  3%|▎         | 1278/41250 [3:05:24<96:07:56,  8.66s/it][2025-04-25 11:03:08,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 11:03:08,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.28 | bwd_microstep: 5728.61 | bwd_inner_microstep: 5696.92 | bwd_allreduce_microstep: 31.65 | step_microstep: 18.90
[2025-04-25 11:03:08,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.28 | bwd: 5728.63 | bwd_inner: 5696.92 | bwd_allreduce: 31.67 | step: 18.90

  3%|▎         | 1279/41250 [3:05:33<96:10:44,  8.66s/it]
                                                         
{'loss': 0.0509, 'grad_norm': 0.7116105556488037, 'learning_rate': 3.99998963697458e-05, 'epoch': 0.31}

  3%|▎         | 1279/41250 [3:05:33<96:10:44,  8.66s/it][2025-04-25 11:03:17,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:03:17,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.81 | bwd_microstep: 5774.15 | bwd_inner_microstep: 5642.19 | bwd_allreduce_microstep: 131.91 | step_microstep: 18.66
[2025-04-25 11:03:17,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.81 | bwd: 5774.16 | bwd_inner: 5642.19 | bwd_allreduce: 131.93 | step: 18.66

  3%|▎         | 1280/41250 [3:05:42<96:14:16,  8.67s/it]
                                                         
{'loss': 0.1467, 'grad_norm': 1.36723792552948, 'learning_rate': 3.99998912529681e-05, 'epoch': 0.31}

  3%|▎         | 1280/41250 [3:05:42<96:14:16,  8.67s/it][2025-04-25 11:03:25,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.10 | optimizer_step: 1.19
[2025-04-25 11:03:25,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.57 | bwd_microstep: 5771.85 | bwd_inner_microstep: 5634.56 | bwd_allreduce_microstep: 137.23 | step_microstep: 19.66
[2025-04-25 11:03:25,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.57 | bwd: 5771.86 | bwd_inner: 5634.56 | bwd_allreduce: 137.25 | step: 19.66

  3%|▎         | 1281/41250 [3:05:51<96:16:45,  8.67s/it]
                                                         
{'loss': 0.2551, 'grad_norm': 2.0970406532287598, 'learning_rate': 3.9999886012895004e-05, 'epoch': 0.31}

  3%|▎         | 1281/41250 [3:05:51<96:16:45,  8.67s/it][2025-04-25 11:03:34,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 11:03:34,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.58 | bwd_microstep: 5738.45 | bwd_inner_microstep: 5681.35 | bwd_allreduce_microstep: 57.06 | step_microstep: 19.12
[2025-04-25 11:03:34,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.58 | bwd: 5738.47 | bwd_inner: 5681.35 | bwd_allreduce: 57.07 | step: 19.12

  3%|▎         | 1282/41250 [3:05:59<96:16:12,  8.67s/it]
                                                         
{'loss': 0.105, 'grad_norm': 1.2410205602645874, 'learning_rate': 3.999988064952655e-05, 'epoch': 0.31}

  3%|▎         | 1282/41250 [3:05:59<96:16:12,  8.67s/it][2025-04-25 11:03:43,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 11:03:43,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.68 | bwd_microstep: 5751.75 | bwd_inner_microstep: 5641.97 | bwd_allreduce_microstep: 109.72 | step_microstep: 19.36
[2025-04-25 11:03:43,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.68 | bwd: 5751.76 | bwd_inner: 5641.97 | bwd_allreduce: 109.75 | step: 19.37

  3%|▎         | 1283/41250 [3:06:08<96:13:33,  8.67s/it]
                                                         
{'loss': 0.0985, 'grad_norm': 0.9045676589012146, 'learning_rate': 3.9999875162862767e-05, 'epoch': 0.31}

  3%|▎         | 1283/41250 [3:06:08<96:13:33,  8.67s/it][2025-04-25 11:03:51,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 11:03:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.34 | bwd_microstep: 5751.31 | bwd_inner_microstep: 5678.55 | bwd_allreduce_microstep: 72.71 | step_microstep: 18.95
[2025-04-25 11:03:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.34 | bwd: 5751.33 | bwd_inner: 5678.55 | bwd_allreduce: 72.73 | step: 18.96

  3%|▎         | 1284/41250 [3:06:17<96:15:10,  8.67s/it]
                                                         
{'loss': 0.089, 'grad_norm': 0.9403590559959412, 'learning_rate': 3.999986955290369e-05, 'epoch': 0.31}

  3%|▎         | 1284/41250 [3:06:17<96:15:10,  8.67s/it][2025-04-25 11:04:00,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:04:00,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.73 | bwd_microstep: 5692.55 | bwd_inner_microstep: 5679.88 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.39
[2025-04-25 11:04:00,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.73 | bwd: 5692.56 | bwd_inner: 5679.88 | bwd_allreduce: 12.64 | step: 19.39

  3%|▎         | 1285/41250 [3:06:25<96:05:06,  8.66s/it]
                                                         
{'loss': 0.2711, 'grad_norm': 3.266547203063965, 'learning_rate': 3.999986381964934e-05, 'epoch': 0.31}

  3%|▎         | 1285/41250 [3:06:25<96:05:06,  8.66s/it][2025-04-25 11:04:08,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:04:08,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.92 | bwd_microstep: 5750.47 | bwd_inner_microstep: 5672.67 | bwd_allreduce_microstep: 77.76 | step_microstep: 18.49
[2025-04-25 11:04:08,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.92 | bwd: 5750.49 | bwd_inner: 5672.67 | bwd_allreduce: 77.78 | step: 18.49

  3%|▎         | 1286/41250 [3:06:34<96:08:15,  8.66s/it]
                                                         
{'loss': 0.1239, 'grad_norm': 1.4126347303390503, 'learning_rate': 3.999985796309977e-05, 'epoch': 0.31}

  3%|▎         | 1286/41250 [3:06:34<96:08:15,  8.66s/it][2025-04-25 11:04:17,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 11:04:17,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.04 | bwd_microstep: 5727.35 | bwd_inner_microstep: 5682.09 | bwd_allreduce_microstep: 45.22 | step_microstep: 18.79
[2025-04-25 11:04:17,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.04 | bwd: 5727.37 | bwd_inner: 5682.09 | bwd_allreduce: 45.24 | step: 18.79

  3%|▎         | 1287/41250 [3:06:42<96:06:29,  8.66s/it]
                                                         
{'loss': 0.1528, 'grad_norm': 1.403557538986206, 'learning_rate': 3.999985198325501e-05, 'epoch': 0.31}

  3%|▎         | 1287/41250 [3:06:42<96:06:29,  8.66s/it][2025-04-25 11:04:26,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:04:26,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.81 | bwd_microstep: 5758.08 | bwd_inner_microstep: 5634.70 | bwd_allreduce_microstep: 123.34 | step_microstep: 18.71
[2025-04-25 11:04:26,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.81 | bwd: 5758.09 | bwd_inner: 5634.70 | bwd_allreduce: 123.36 | step: 18.72

  3%|▎         | 1288/41250 [3:06:51<96:07:43,  8.66s/it]
                                                         
{'loss': 0.5585, 'grad_norm': 3.335237979888916, 'learning_rate': 3.99998458801151e-05, 'epoch': 0.31}

  3%|▎         | 1288/41250 [3:06:51<96:07:43,  8.66s/it][2025-04-25 11:04:34,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 11:04:34,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.03 | bwd_microstep: 5766.00 | bwd_inner_microstep: 5676.81 | bwd_allreduce_microstep: 89.14 | step_microstep: 19.49
[2025-04-25 11:04:34,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.03 | bwd: 5766.01 | bwd_inner: 5676.81 | bwd_allreduce: 89.16 | step: 19.49

  3%|▎         | 1289/41250 [3:07:00<96:13:20,  8.67s/it]
                                                         
{'loss': 0.2321, 'grad_norm': 2.464484214782715, 'learning_rate': 3.999983965368007e-05, 'epoch': 0.31}

  3%|▎         | 1289/41250 [3:07:00<96:13:20,  8.67s/it][2025-04-25 11:04:43,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 11:04:43,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.24 | bwd_microstep: 5734.83 | bwd_inner_microstep: 5689.24 | bwd_allreduce_microstep: 45.54 | step_microstep: 19.00
[2025-04-25 11:04:43,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.25 | bwd: 5734.84 | bwd_inner: 5689.24 | bwd_allreduce: 45.56 | step: 19.00

  3%|▎         | 1290/41250 [3:07:08<96:12:55,  8.67s/it]
                                                         
{'loss': 0.1541, 'grad_norm': 1.6081129312515259, 'learning_rate': 3.999983330394996e-05, 'epoch': 0.31}

  3%|▎         | 1290/41250 [3:07:08<96:12:55,  8.67s/it][2025-04-25 11:04:52,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:04:52,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5695.96 | bwd_inner_microstep: 5636.61 | bwd_allreduce_microstep: 59.31 | step_microstep: 18.53
[2025-04-25 11:04:52,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5695.97 | bwd_inner: 5636.61 | bwd_allreduce: 59.32 | step: 18.54

  3%|▎         | 1291/41250 [3:07:17<96:00:23,  8.65s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 2.937805414199829, 'learning_rate': 3.9999826830924814e-05, 'epoch': 0.31}

  3%|▎         | 1291/41250 [3:07:17<96:00:23,  8.65s/it][2025-04-25 11:05:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 11:05:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.54 | bwd_microstep: 5692.91 | bwd_inner_microstep: 5679.79 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.94
[2025-04-25 11:05:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.54 | bwd: 5692.93 | bwd_inner: 5679.79 | bwd_allreduce: 13.10 | step: 18.94

  3%|▎         | 1292/41250 [3:07:26<95:54:22,  8.64s/it]
                                                         
{'loss': 0.3733, 'grad_norm': 2.5629236698150635, 'learning_rate': 3.999982023460467e-05, 'epoch': 0.31}

  3%|▎         | 1292/41250 [3:07:26<95:54:22,  8.64s/it][2025-04-25 11:05:09,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.07 | optimizer_step: 1.03
[2025-04-25 11:05:09,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.52 | bwd_microstep: 5706.16 | bwd_inner_microstep: 5693.38 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.22
[2025-04-25 11:05:09,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5706.17 | bwd_inner: 5693.38 | bwd_allreduce: 12.75 | step: 19.22

  3%|▎         | 1293/41250 [3:07:34<95:53:47,  8.64s/it]
                                                         
{'loss': 0.1939, 'grad_norm': 2.0665860176086426, 'learning_rate': 3.9999813514989566e-05, 'epoch': 0.31}

  3%|▎         | 1293/41250 [3:07:34<95:53:47,  8.64s/it][2025-04-25 11:05:18,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:05:18,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.48 | bwd_microstep: 5771.78 | bwd_inner_microstep: 5759.09 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.62
[2025-04-25 11:05:18,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.48 | bwd: 5771.80 | bwd_inner: 5759.09 | bwd_allreduce: 12.66 | step: 18.63

  3%|▎         | 1294/41250 [3:07:43<96:13:56,  8.67s/it]
                                                         
{'loss': 0.1555, 'grad_norm': 1.4199310541152954, 'learning_rate': 3.999980667207955e-05, 'epoch': 0.31}

  3%|▎         | 1294/41250 [3:07:43<96:13:56,  8.67s/it][2025-04-25 11:05:26,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:05:26,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.88 | bwd_microstep: 5744.57 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 64.55 | step_microstep: 18.31
[2025-04-25 11:05:26,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.88 | bwd: 5744.58 | bwd_inner: 5679.98 | bwd_allreduce: 64.56 | step: 18.31

  3%|▎         | 1295/41250 [3:07:52<96:13:32,  8.67s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 1.4832082986831665, 'learning_rate': 3.999979970587465e-05, 'epoch': 0.31}

  3%|▎         | 1295/41250 [3:07:52<96:13:32,  8.67s/it][2025-04-25 11:05:35,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:05:35,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.70 | bwd_microstep: 5710.33 | bwd_inner_microstep: 5644.10 | bwd_allreduce_microstep: 66.18 | step_microstep: 18.43
[2025-04-25 11:05:35,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.70 | bwd: 5710.34 | bwd_inner: 5644.10 | bwd_allreduce: 66.20 | step: 18.44

  3%|▎         | 1296/41250 [3:08:00<96:01:40,  8.65s/it]
                                                         
{'loss': 0.3955, 'grad_norm': 2.7779033184051514, 'learning_rate': 3.9999792616374926e-05, 'epoch': 0.31}

  3%|▎         | 1296/41250 [3:08:00<96:01:40,  8.65s/it][2025-04-25 11:05:44,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 11:05:44,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.87 | bwd_microstep: 5681.81 | bwd_inner_microstep: 5666.48 | bwd_allreduce_microstep: 15.28 | step_microstep: 19.56
[2025-04-25 11:05:44,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.87 | bwd: 5681.83 | bwd_inner: 5666.48 | bwd_allreduce: 15.30 | step: 19.56

  3%|▎         | 1297/41250 [3:08:09<95:52:05,  8.64s/it]
                                                         
{'loss': 0.0756, 'grad_norm': 0.8522654175758362, 'learning_rate': 3.9999785403580417e-05, 'epoch': 0.31}

  3%|▎         | 1297/41250 [3:08:09<95:52:05,  8.64s/it][2025-04-25 11:05:52,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 11:05:52,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.32 | bwd_microstep: 5759.74 | bwd_inner_microstep: 5669.82 | bwd_allreduce_microstep: 89.87 | step_microstep: 18.90
[2025-04-25 11:05:52,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.32 | bwd: 5759.75 | bwd_inner: 5669.82 | bwd_allreduce: 89.89 | step: 18.91

  3%|▎         | 1298/41250 [3:08:18<96:01:04,  8.65s/it]
                                                         
{'loss': 0.2441, 'grad_norm': 1.6908618211746216, 'learning_rate': 3.9999778067491154e-05, 'epoch': 0.31}

  3%|▎         | 1298/41250 [3:08:18<96:01:04,  8.65s/it][2025-04-25 11:06:01,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 11:06:01,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.78 | bwd_microstep: 5759.47 | bwd_inner_microstep: 5623.06 | bwd_allreduce_microstep: 136.35 | step_microstep: 18.76
[2025-04-25 11:06:01,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.78 | bwd: 5759.48 | bwd_inner: 5623.06 | bwd_allreduce: 136.38 | step: 18.76

  3%|▎         | 1299/41250 [3:08:26<96:02:06,  8.65s/it]
                                                         
{'loss': 0.1104, 'grad_norm': 0.9620935916900635, 'learning_rate': 3.99997706081072e-05, 'epoch': 0.31}

  3%|▎         | 1299/41250 [3:08:26<96:02:06,  8.65s/it][2025-04-25 11:06:10,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:06:10,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.31 | bwd_microstep: 5702.29 | bwd_inner_microstep: 5677.57 | bwd_allreduce_microstep: 24.67 | step_microstep: 18.90
[2025-04-25 11:06:10,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.31 | bwd: 5702.30 | bwd_inner: 5677.57 | bwd_allreduce: 24.69 | step: 18.90

  3%|▎         | 1300/41250 [3:08:35<95:56:54,  8.65s/it]
                                                         
{'loss': 0.144, 'grad_norm': 1.7171885967254639, 'learning_rate': 3.9999763025428595e-05, 'epoch': 0.32}

  3%|▎         | 1300/41250 [3:08:35<95:56:54,  8.65s/it][2025-04-25 11:06:18,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 11:06:18,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.17 | bwd_microstep: 5746.67 | bwd_inner_microstep: 5637.18 | bwd_allreduce_microstep: 109.44 | step_microstep: 19.16
[2025-04-25 11:06:18,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.17 | bwd: 5746.69 | bwd_inner: 5637.18 | bwd_allreduce: 109.46 | step: 19.16

  3%|▎         | 1301/41250 [3:08:44<95:58:42,  8.65s/it]
                                                         
{'loss': 0.2312, 'grad_norm': 1.7949193716049194, 'learning_rate': 3.999975531945538e-05, 'epoch': 0.32}

  3%|▎         | 1301/41250 [3:08:44<95:58:42,  8.65s/it][2025-04-25 11:06:27,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.23 | optimizer_step: 0.94
[2025-04-25 11:06:27,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.05 | bwd_microstep: 5746.51 | bwd_inner_microstep: 5681.58 | bwd_allreduce_microstep: 64.87 | step_microstep: 19.65
[2025-04-25 11:06:27,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.05 | bwd: 5746.52 | bwd_inner: 5681.58 | bwd_allreduce: 64.89 | step: 19.65

  3%|▎         | 1302/41250 [3:08:52<96:03:38,  8.66s/it]
                                                         
{'loss': 0.2518, 'grad_norm': 2.131556272506714, 'learning_rate': 3.9999747490187614e-05, 'epoch': 0.32}

  3%|▎         | 1302/41250 [3:08:52<96:03:38,  8.66s/it][2025-04-25 11:06:36,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-25 11:06:36,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.18 | bwd_microstep: 5710.15 | bwd_inner_microstep: 5638.27 | bwd_allreduce_microstep: 71.84 | step_microstep: 18.61
[2025-04-25 11:06:36,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.18 | bwd: 5710.16 | bwd_inner: 5638.27 | bwd_allreduce: 71.86 | step: 18.62

  3%|▎         | 1303/41250 [3:09:01<95:55:14,  8.64s/it]
                                                         
{'loss': 0.2675, 'grad_norm': 2.662431001663208, 'learning_rate': 3.999973953762533e-05, 'epoch': 0.32}

  3%|▎         | 1303/41250 [3:09:01<95:55:14,  8.64s/it][2025-04-25 11:06:44,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:06:44,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.84 | bwd_microstep: 5712.69 | bwd_inner_microstep: 5700.11 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.33
[2025-04-25 11:06:44,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.84 | bwd: 5712.70 | bwd_inner: 5700.11 | bwd_allreduce: 12.55 | step: 18.33

  3%|▎         | 1304/41250 [3:09:10<95:53:35,  8.64s/it]
                                                         
{'loss': 0.4262, 'grad_norm': 7.493181228637695, 'learning_rate': 3.999973146176859e-05, 'epoch': 0.32}

  3%|▎         | 1304/41250 [3:09:10<95:53:35,  8.64s/it][2025-04-25 11:06:53,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.15 | optimizer_step: 1.07
[2025-04-25 11:06:53,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5787.57 | bwd_inner_microstep: 5648.25 | bwd_allreduce_microstep: 139.27 | step_microstep: 19.54
[2025-04-25 11:06:53,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5787.59 | bwd_inner: 5648.25 | bwd_allreduce: 139.29 | step: 19.54

  3%|▎         | 1305/41250 [3:09:18<96:04:17,  8.66s/it]
                                                         
{'loss': 0.3269, 'grad_norm': 1.7970319986343384, 'learning_rate': 3.999972326261744e-05, 'epoch': 0.32}

  3%|▎         | 1305/41250 [3:09:18<96:04:17,  8.66s/it][2025-04-25 11:07:02,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.16 | optimizer_step: 0.91
[2025-04-25 11:07:02,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.63 | bwd_microstep: 5766.70 | bwd_inner_microstep: 5658.42 | bwd_allreduce_microstep: 108.23 | step_microstep: 19.20
[2025-04-25 11:07:02,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.63 | bwd: 5766.71 | bwd_inner: 5658.42 | bwd_allreduce: 108.25 | step: 19.20

  3%|▎         | 1306/41250 [3:09:27<96:08:32,  8.66s/it]
                                                         
{'loss': 0.0359, 'grad_norm': 0.387334942817688, 'learning_rate': 3.999971494017193e-05, 'epoch': 0.32}

  3%|▎         | 1306/41250 [3:09:27<96:08:32,  8.66s/it][2025-04-25 11:07:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 11:07:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.37 | bwd_microstep: 5711.34 | bwd_inner_microstep: 5649.76 | bwd_allreduce_microstep: 61.53 | step_microstep: 18.92
[2025-04-25 11:07:10,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.37 | bwd: 5711.35 | bwd_inner: 5649.76 | bwd_allreduce: 61.55 | step: 18.92

  3%|▎         | 1307/41250 [3:09:36<95:59:08,  8.65s/it]
                                                         
{'loss': 0.1536, 'grad_norm': 1.0795871019363403, 'learning_rate': 3.999970649443211e-05, 'epoch': 0.32}

  3%|▎         | 1307/41250 [3:09:36<95:59:08,  8.65s/it][2025-04-25 11:07:19,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:07:19,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.49 | bwd_microstep: 5754.45 | bwd_inner_microstep: 5690.37 | bwd_allreduce_microstep: 64.03 | step_microstep: 18.60
[2025-04-25 11:07:19,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.48 | bwd: 5754.46 | bwd_inner: 5690.37 | bwd_allreduce: 64.05 | step: 18.61

  3%|▎         | 1308/41250 [3:09:44<96:05:29,  8.66s/it]
                                                         
{'loss': 0.144, 'grad_norm': 1.2079415321350098, 'learning_rate': 3.999969792539804e-05, 'epoch': 0.32}

  3%|▎         | 1308/41250 [3:09:44<96:05:29,  8.66s/it][2025-04-25 11:07:28,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 11:07:28,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5770.75 | bwd_inner_microstep: 5692.83 | bwd_allreduce_microstep: 77.88 | step_microstep: 18.72
[2025-04-25 11:07:28,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5770.76 | bwd_inner: 5692.83 | bwd_allreduce: 77.89 | step: 18.72

  3%|▎         | 1309/41250 [3:09:53<96:13:17,  8.67s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.3159321546554565, 'learning_rate': 3.9999689233069754e-05, 'epoch': 0.32}

  3%|▎         | 1309/41250 [3:09:53<96:13:17,  8.67s/it][2025-04-25 11:07:36,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:07:36,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.43 | bwd_microstep: 5727.26 | bwd_inner_microstep: 5714.43 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.89
[2025-04-25 11:07:36,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.43 | bwd: 5727.27 | bwd_inner: 5714.43 | bwd_allreduce: 12.80 | step: 18.89

  3%|▎         | 1310/41250 [3:10:02<96:13:27,  8.67s/it]
                                                         
{'loss': 0.1453, 'grad_norm': 2.4141016006469727, 'learning_rate': 3.999968041744732e-05, 'epoch': 0.32}

  3%|▎         | 1310/41250 [3:10:02<96:13:27,  8.67s/it][2025-04-25 11:07:45,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 11:07:45,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.91 | bwd_microstep: 5910.95 | bwd_inner_microstep: 5669.55 | bwd_allreduce_microstep: 241.36 | step_microstep: 18.66
[2025-04-25 11:07:45,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.91 | bwd: 5910.97 | bwd_inner: 5669.55 | bwd_allreduce: 241.38 | step: 18.66

  3%|▎         | 1311/41250 [3:10:10<96:45:00,  8.72s/it]
                                                         
{'loss': 0.0559, 'grad_norm': 1.2319542169570923, 'learning_rate': 3.99996714785308e-05, 'epoch': 0.32}

  3%|▎         | 1311/41250 [3:10:10<96:45:00,  8.72s/it][2025-04-25 11:07:54,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 11:07:54,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.22 | bwd_microstep: 5703.09 | bwd_inner_microstep: 5690.10 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.07
[2025-04-25 11:07:54,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.22 | bwd: 5703.11 | bwd_inner: 5690.10 | bwd_allreduce: 12.96 | step: 19.07

  3%|▎         | 1312/41250 [3:10:19<96:30:55,  8.70s/it]
                                                         
{'loss': 0.2921, 'grad_norm': 2.6725547313690186, 'learning_rate': 3.999966241632023e-05, 'epoch': 0.32}

  3%|▎         | 1312/41250 [3:10:19<96:30:55,  8.70s/it][2025-04-25 11:08:02,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 11:08:02,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.12 | bwd_microstep: 5771.07 | bwd_inner_microstep: 5670.47 | bwd_allreduce_microstep: 100.56 | step_microstep: 19.27
[2025-04-25 11:08:02,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.12 | bwd: 5771.09 | bwd_inner: 5670.47 | bwd_allreduce: 100.58 | step: 19.27

  3%|▎         | 1313/41250 [3:10:28<96:28:39,  8.70s/it]
                                                         
{'loss': 0.1045, 'grad_norm': 1.6586017608642578, 'learning_rate': 3.999965323081568e-05, 'epoch': 0.32}

  3%|▎         | 1313/41250 [3:10:28<96:28:39,  8.70s/it][2025-04-25 11:08:11,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 0.93
[2025-04-25 11:08:11,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.04 | bwd_microstep: 5763.71 | bwd_inner_microstep: 5690.47 | bwd_allreduce_microstep: 73.18 | step_microstep: 19.57
[2025-04-25 11:08:11,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5763.72 | bwd_inner: 5690.47 | bwd_allreduce: 73.20 | step: 19.57

  3%|▎         | 1314/41250 [3:10:36<96:29:14,  8.70s/it]
                                                         
{'loss': 0.4185, 'grad_norm': 3.8792073726654053, 'learning_rate': 3.9999643922017196e-05, 'epoch': 0.32}

  3%|▎         | 1314/41250 [3:10:36<96:29:14,  8.70s/it][2025-04-25 11:08:20,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:08:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.58 | bwd_microstep: 5770.01 | bwd_inner_microstep: 5705.57 | bwd_allreduce_microstep: 64.39 | step_microstep: 19.06
[2025-04-25 11:08:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.58 | bwd: 5770.02 | bwd_inner: 5705.57 | bwd_allreduce: 64.41 | step: 19.06

  3%|▎         | 1315/41250 [3:10:45<96:31:34,  8.70s/it]
                                                         
{'loss': 0.1941, 'grad_norm': 1.7228940725326538, 'learning_rate': 3.999963448992485e-05, 'epoch': 0.32}

  3%|▎         | 1315/41250 [3:10:45<96:31:34,  8.70s/it][2025-04-25 11:08:28,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:08:28,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.52 | bwd_microstep: 5725.02 | bwd_inner_microstep: 5712.19 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.75
[2025-04-25 11:08:28,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.52 | bwd: 5725.03 | bwd_inner: 5712.19 | bwd_allreduce: 12.80 | step: 18.75

  3%|▎         | 1316/41250 [3:10:54<96:23:02,  8.69s/it]
                                                         
{'loss': 0.1324, 'grad_norm': 1.7759312391281128, 'learning_rate': 3.9999624934538676e-05, 'epoch': 0.32}

  3%|▎         | 1316/41250 [3:10:54<96:23:02,  8.69s/it][2025-04-25 11:08:37,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 11:08:37,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.79 | bwd_microstep: 5804.40 | bwd_inner_microstep: 5664.75 | bwd_allreduce_microstep: 139.61 | step_microstep: 18.63
[2025-04-25 11:08:37,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.79 | bwd: 5804.42 | bwd_inner: 5664.75 | bwd_allreduce: 139.63 | step: 18.64

  3%|▎         | 1317/41250 [3:11:03<96:29:23,  8.70s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.7776367664337158, 'learning_rate': 3.999961525585876e-05, 'epoch': 0.32}

  3%|▎         | 1317/41250 [3:11:03<96:29:23,  8.70s/it][2025-04-25 11:08:46,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 11:08:46,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.09 | bwd_microstep: 5779.23 | bwd_inner_microstep: 5714.24 | bwd_allreduce_microstep: 64.94 | step_microstep: 19.23
[2025-04-25 11:08:46,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.09 | bwd: 5779.24 | bwd_inner: 5714.24 | bwd_allreduce: 64.96 | step: 19.23

  3%|▎         | 1318/41250 [3:11:11<96:33:15,  8.70s/it]
                                                         
{'loss': 0.3499, 'grad_norm': 1.7349251508712769, 'learning_rate': 3.999960545388514e-05, 'epoch': 0.32}

  3%|▎         | 1318/41250 [3:11:11<96:33:15,  8.70s/it][2025-04-25 11:08:55,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 11:08:55,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.82 | bwd_microstep: 5711.85 | bwd_inner_microstep: 5699.16 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.48
[2025-04-25 11:08:55,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.82 | bwd: 5711.86 | bwd_inner: 5699.16 | bwd_allreduce: 12.66 | step: 18.49

  3%|▎         | 1319/41250 [3:11:20<96:20:17,  8.69s/it]
                                                         
{'loss': 0.3803, 'grad_norm': 3.3877227306365967, 'learning_rate': 3.99995955286179e-05, 'epoch': 0.32}

  3%|▎         | 1319/41250 [3:11:20<96:20:17,  8.69s/it][2025-04-25 11:09:03,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:09:03,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.77 | bwd_microstep: 5765.22 | bwd_inner_microstep: 5708.54 | bwd_allreduce_microstep: 56.64 | step_microstep: 18.50
[2025-04-25 11:09:03,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.77 | bwd: 5765.24 | bwd_inner: 5708.54 | bwd_allreduce: 56.66 | step: 18.50

  3%|▎         | 1320/41250 [3:11:29<96:24:08,  8.69s/it]
                                                         
{'loss': 0.1236, 'grad_norm': 1.5651432275772095, 'learning_rate': 3.9999585480057076e-05, 'epoch': 0.32}

  3%|▎         | 1320/41250 [3:11:29<96:24:08,  8.69s/it][2025-04-25 11:09:12,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:09:12,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.69 | bwd_microstep: 5792.06 | bwd_inner_microstep: 5696.40 | bwd_allreduce_microstep: 95.61 | step_microstep: 18.56
[2025-04-25 11:09:12,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.69 | bwd: 5792.07 | bwd_inner: 5696.40 | bwd_allreduce: 95.62 | step: 18.56

  3%|▎         | 1321/41250 [3:11:37<96:30:09,  8.70s/it]
                                                         
{'loss': 0.095, 'grad_norm': 1.0153892040252686, 'learning_rate': 3.9999575308202744e-05, 'epoch': 0.32}

  3%|▎         | 1321/41250 [3:11:37<96:30:09,  8.70s/it][2025-04-25 11:09:21,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:09:21,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.17 | bwd_microstep: 5719.64 | bwd_inner_microstep: 5662.79 | bwd_allreduce_microstep: 56.80 | step_microstep: 18.43
[2025-04-25 11:09:21,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5719.65 | bwd_inner: 5662.79 | bwd_allreduce: 56.82 | step: 18.44

  3%|▎         | 1322/41250 [3:11:46<96:16:19,  8.68s/it]
                                                         
{'loss': 0.2411, 'grad_norm': 1.7305529117584229, 'learning_rate': 3.9999565013054966e-05, 'epoch': 0.32}

  3%|▎         | 1322/41250 [3:11:46<96:16:19,  8.68s/it][2025-04-25 11:09:29,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-25 11:09:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5747.81 | bwd_inner_microstep: 5715.58 | bwd_allreduce_microstep: 32.17 | step_microstep: 19.81
[2025-04-25 11:09:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5747.83 | bwd_inner: 5715.58 | bwd_allreduce: 32.20 | step: 19.81

  3%|▎         | 1323/41250 [3:11:55<96:17:26,  8.68s/it]
                                                         
{'loss': 0.1964, 'grad_norm': 2.101811170578003, 'learning_rate': 3.9999554594613805e-05, 'epoch': 0.32}

  3%|▎         | 1323/41250 [3:11:55<96:17:26,  8.68s/it][2025-04-25 11:09:38,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:09:38,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.38 | bwd_microstep: 5764.89 | bwd_inner_microstep: 5700.35 | bwd_allreduce_microstep: 64.50 | step_microstep: 18.72
[2025-04-25 11:09:38,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.38 | bwd: 5764.91 | bwd_inner: 5700.35 | bwd_allreduce: 64.52 | step: 18.72

  3%|▎         | 1324/41250 [3:12:03<96:19:35,  8.69s/it]
                                                         
{'loss': 0.0561, 'grad_norm': 0.6326274871826172, 'learning_rate': 3.999954405287932e-05, 'epoch': 0.32}

  3%|▎         | 1324/41250 [3:12:03<96:19:35,  8.69s/it][2025-04-25 11:09:47,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:09:47,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5707.52 | bwd_inner_microstep: 5661.45 | bwd_allreduce_microstep: 46.03 | step_microstep: 18.63
[2025-04-25 11:09:47,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5707.53 | bwd_inner: 5661.45 | bwd_allreduce: 46.04 | step: 18.63

  3%|▎         | 1325/41250 [3:12:12<96:06:39,  8.67s/it]
                                                         
{'loss': 0.2753, 'grad_norm': 1.7329570055007935, 'learning_rate': 3.999953338785158e-05, 'epoch': 0.32}

  3%|▎         | 1325/41250 [3:12:12<96:06:39,  8.67s/it][2025-04-25 11:09:55,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 11:09:55,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.78 | bwd_microstep: 5730.45 | bwd_inner_microstep: 5664.05 | bwd_allreduce_microstep: 66.36 | step_microstep: 18.51
[2025-04-25 11:09:55,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.78 | bwd: 5730.46 | bwd_inner: 5664.05 | bwd_allreduce: 66.37 | step: 18.51

  3%|▎         | 1326/41250 [3:12:21<96:00:47,  8.66s/it]
                                                         
{'loss': 0.2108, 'grad_norm': 2.237399101257324, 'learning_rate': 3.9999522599530654e-05, 'epoch': 0.32}

  3%|▎         | 1326/41250 [3:12:21<96:00:47,  8.66s/it][2025-04-25 11:10:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 11:10:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5742.30 | bwd_inner_microstep: 5710.64 | bwd_allreduce_microstep: 31.62 | step_microstep: 18.73
[2025-04-25 11:10:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5742.32 | bwd_inner: 5710.64 | bwd_allreduce: 31.63 | step: 18.74

  3%|▎         | 1327/41250 [3:12:29<96:03:55,  8.66s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 1.8097002506256104, 'learning_rate': 3.9999511687916594e-05, 'epoch': 0.32}

  3%|▎         | 1327/41250 [3:12:29<96:03:55,  8.66s/it][2025-04-25 11:10:13,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.27
[2025-04-25 11:10:13,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.62 | bwd_microstep: 5711.37 | bwd_inner_microstep: 5652.21 | bwd_allreduce_microstep: 59.11 | step_microstep: 20.06
[2025-04-25 11:10:13,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.62 | bwd: 5711.38 | bwd_inner: 5652.21 | bwd_allreduce: 59.13 | step: 20.06

  3%|▎         | 1328/41250 [3:12:38<95:56:21,  8.65s/it]
                                                         
{'loss': 0.1669, 'grad_norm': 1.431931972503662, 'learning_rate': 3.999950065300949e-05, 'epoch': 0.32}

  3%|▎         | 1328/41250 [3:12:38<95:56:21,  8.65s/it][2025-04-25 11:10:21,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:10:21,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.98 | bwd_microstep: 5785.13 | bwd_inner_microstep: 5662.10 | bwd_allreduce_microstep: 122.99 | step_microstep: 18.64
[2025-04-25 11:10:21,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.98 | bwd: 5785.15 | bwd_inner: 5662.10 | bwd_allreduce: 123.00 | step: 18.65

  3%|▎         | 1329/41250 [3:12:47<96:05:25,  8.67s/it]
                                                         
{'loss': 0.2178, 'grad_norm': 1.3017315864562988, 'learning_rate': 3.999948949480939e-05, 'epoch': 0.32}

  3%|▎         | 1329/41250 [3:12:47<96:05:25,  8.67s/it][2025-04-25 11:10:30,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-25 11:10:30,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.13 | bwd_microstep: 5721.48 | bwd_inner_microstep: 5666.24 | bwd_allreduce_microstep: 55.18 | step_microstep: 19.27
[2025-04-25 11:10:30,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.13 | bwd: 5721.49 | bwd_inner: 5666.24 | bwd_allreduce: 55.21 | step: 19.28

  3%|▎         | 1330/41250 [3:12:55<95:58:51,  8.66s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 1.5327470302581787, 'learning_rate': 3.999947821331638e-05, 'epoch': 0.32}

  3%|▎         | 1330/41250 [3:12:55<95:58:51,  8.66s/it][2025-04-25 11:10:39,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:10:39,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.56 | bwd_microstep: 5726.06 | bwd_inner_microstep: 5697.73 | bwd_allreduce_microstep: 28.28 | step_microstep: 18.64
[2025-04-25 11:10:39,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5726.08 | bwd_inner: 5697.73 | bwd_allreduce: 28.30 | step: 18.64

  3%|▎         | 1331/41250 [3:13:04<96:00:09,  8.66s/it]
                                                         
{'loss': 0.0831, 'grad_norm': 0.8758832216262817, 'learning_rate': 3.999946680853051e-05, 'epoch': 0.32}

  3%|▎         | 1331/41250 [3:13:04<96:00:09,  8.66s/it][2025-04-25 11:10:47,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 11:10:47,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.54 | bwd_microstep: 5707.07 | bwd_inner_microstep: 5694.19 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.29
[2025-04-25 11:10:47,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.54 | bwd: 5707.09 | bwd_inner: 5694.19 | bwd_allreduce: 12.86 | step: 19.30

  3%|▎         | 1332/41250 [3:13:13<95:56:26,  8.65s/it]
                                                         
{'loss': 0.0878, 'grad_norm': 0.86513352394104, 'learning_rate': 3.999945528045187e-05, 'epoch': 0.32}

  3%|▎         | 1332/41250 [3:13:13<95:56:26,  8.65s/it][2025-04-25 11:10:56,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:10:56,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.35 | bwd_microstep: 5706.18 | bwd_inner_microstep: 5646.95 | bwd_allreduce_microstep: 59.19 | step_microstep: 18.60
[2025-04-25 11:10:56,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.35 | bwd: 5706.19 | bwd_inner: 5646.95 | bwd_allreduce: 59.20 | step: 18.61

  3%|▎         | 1333/41250 [3:13:21<95:49:44,  8.64s/it]
                                                         
{'loss': 0.228, 'grad_norm': 3.1830363273620605, 'learning_rate': 3.9999443629080514e-05, 'epoch': 0.32}

  3%|▎         | 1333/41250 [3:13:21<95:49:44,  8.64s/it][2025-04-25 11:11:04,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:11:04,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.97 | bwd_microstep: 5709.15 | bwd_inner_microstep: 5696.34 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.88
[2025-04-25 11:11:04,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.97 | bwd: 5709.16 | bwd_inner: 5696.34 | bwd_allreduce: 12.78 | step: 18.88

  3%|▎         | 1334/41250 [3:13:30<95:49:54,  8.64s/it]
                                                         
{'loss': 0.2127, 'grad_norm': 2.002861738204956, 'learning_rate': 3.9999431854416534e-05, 'epoch': 0.32}

  3%|▎         | 1334/41250 [3:13:30<95:49:54,  8.64s/it][2025-04-25 11:11:13,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 11:11:13,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.27 | bwd_microstep: 5769.34 | bwd_inner_microstep: 5707.64 | bwd_allreduce_microstep: 61.65 | step_microstep: 19.45
[2025-04-25 11:11:13,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.27 | bwd: 5769.36 | bwd_inner: 5707.64 | bwd_allreduce: 61.67 | step: 19.46

  3%|▎         | 1335/41250 [3:13:39<96:02:02,  8.66s/it]
                                                         
{'loss': 0.1315, 'grad_norm': 0.9286666512489319, 'learning_rate': 3.9999419956459976e-05, 'epoch': 0.32}

  3%|▎         | 1335/41250 [3:13:39<96:02:02,  8.66s/it][2025-04-25 11:11:22,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 11:11:22,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.08 | bwd_microstep: 5789.07 | bwd_inner_microstep: 5775.87 | bwd_allreduce_microstep: 13.15 | step_microstep: 18.54
[2025-04-25 11:11:22,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.08 | bwd: 5789.08 | bwd_inner: 5775.87 | bwd_allreduce: 13.17 | step: 18.54

  3%|▎         | 1336/41250 [3:13:47<96:23:22,  8.69s/it]
                                                         
{'loss': 0.1587, 'grad_norm': 1.9801154136657715, 'learning_rate': 3.9999407935210935e-05, 'epoch': 0.32}

  3%|▎         | 1336/41250 [3:13:47<96:23:22,  8.69s/it][2025-04-25 11:11:31,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 11:11:31,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.78 | bwd_microstep: 5698.08 | bwd_inner_microstep: 5685.31 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.09
[2025-04-25 11:11:31,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.78 | bwd: 5698.10 | bwd_inner: 5685.31 | bwd_allreduce: 12.75 | step: 19.09

  3%|▎         | 1337/41250 [3:13:56<96:09:38,  8.67s/it]
                                                         
{'loss': 0.1274, 'grad_norm': 1.0482627153396606, 'learning_rate': 3.999939579066948e-05, 'epoch': 0.32}

  3%|▎         | 1337/41250 [3:13:56<96:09:38,  8.67s/it][2025-04-25 11:11:39,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 11:11:39,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.84 | bwd_microstep: 5704.31 | bwd_inner_microstep: 5691.06 | bwd_allreduce_microstep: 13.19 | step_microstep: 18.95
[2025-04-25 11:11:39,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.84 | bwd: 5704.33 | bwd_inner: 5691.06 | bwd_allreduce: 13.22 | step: 18.96

  3%|▎         | 1338/41250 [3:14:05<96:04:43,  8.67s/it]
                                                         
{'loss': 0.1381, 'grad_norm': 0.6548963785171509, 'learning_rate': 3.9999383522835676e-05, 'epoch': 0.32}

  3%|▎         | 1338/41250 [3:14:05<96:04:43,  8.67s/it][2025-04-25 11:11:48,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 11:11:48,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.77 | bwd_microstep: 5714.33 | bwd_inner_microstep: 5693.91 | bwd_allreduce_microstep: 20.37 | step_microstep: 19.12
[2025-04-25 11:11:48,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.77 | bwd: 5714.35 | bwd_inner: 5693.91 | bwd_allreduce: 20.39 | step: 19.12

  3%|▎         | 1339/41250 [3:14:13<96:03:18,  8.66s/it]
                                                         
{'loss': 0.0704, 'grad_norm': 0.7287519574165344, 'learning_rate': 3.999937113170961e-05, 'epoch': 0.32}

  3%|▎         | 1339/41250 [3:14:13<96:03:18,  8.66s/it][2025-04-25 11:11:57,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:11:57,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.12 | bwd_microstep: 5760.45 | bwd_inner_microstep: 5648.33 | bwd_allreduce_microstep: 112.08 | step_microstep: 18.37
[2025-04-25 11:11:57,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.12 | bwd: 5760.47 | bwd_inner: 5648.33 | bwd_allreduce: 112.10 | step: 18.37

  3%|▎         | 1340/41250 [3:14:22<96:05:22,  8.67s/it]
                                                         
{'loss': 0.2376, 'grad_norm': 2.3139536380767822, 'learning_rate': 3.9999358617291354e-05, 'epoch': 0.32}

  3%|▎         | 1340/41250 [3:14:22<96:05:22,  8.67s/it][2025-04-25 11:12:05,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 11:12:05,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.75 | bwd_microstep: 5706.23 | bwd_inner_microstep: 5681.88 | bwd_allreduce_microstep: 24.29 | step_microstep: 18.90
[2025-04-25 11:12:05,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.75 | bwd: 5706.24 | bwd_inner: 5681.88 | bwd_allreduce: 24.32 | step: 18.90

  3%|▎         | 1341/41250 [3:14:31<96:01:34,  8.66s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 1.1332520246505737, 'learning_rate': 3.999934597958099e-05, 'epoch': 0.33}

  3%|▎         | 1341/41250 [3:14:31<96:01:34,  8.66s/it][2025-04-25 11:12:14,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.10 | optimizer_step: 1.09
[2025-04-25 11:12:14,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.44 | bwd_microstep: 5721.49 | bwd_inner_microstep: 5708.07 | bwd_allreduce_microstep: 13.36 | step_microstep: 19.79
[2025-04-25 11:12:14,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.44 | bwd: 5721.50 | bwd_inner: 5708.07 | bwd_allreduce: 13.38 | step: 19.79

  3%|▎         | 1342/41250 [3:14:39<96:01:44,  8.66s/it]
                                                         
{'loss': 0.647, 'grad_norm': 3.3535726070404053, 'learning_rate': 3.9999333218578585e-05, 'epoch': 0.33}

  3%|▎         | 1342/41250 [3:14:39<96:01:44,  8.66s/it][2025-04-25 11:12:22,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 11:12:22,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5684.34 | bwd_inner_microstep: 5648.32 | bwd_allreduce_microstep: 35.96 | step_microstep: 19.45
[2025-04-25 11:12:22,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5684.36 | bwd_inner: 5648.32 | bwd_allreduce: 35.99 | step: 19.45

  3%|▎         | 1343/41250 [3:14:48<95:48:35,  8.64s/it]
                                                         
{'loss': 0.4962, 'grad_norm': 3.364589214324951, 'learning_rate': 3.999932033428423e-05, 'epoch': 0.33}

  3%|▎         | 1343/41250 [3:14:48<95:48:35,  8.64s/it][2025-04-25 11:12:31,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 11:12:31,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.64 | bwd_microstep: 5692.43 | bwd_inner_microstep: 5650.57 | bwd_allreduce_microstep: 41.81 | step_microstep: 19.24
[2025-04-25 11:12:31,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.64 | bwd: 5692.44 | bwd_inner: 5650.57 | bwd_allreduce: 41.83 | step: 19.25

  3%|▎         | 1344/41250 [3:14:56<95:41:22,  8.63s/it]
                                                         
{'loss': 0.2154, 'grad_norm': 1.3153609037399292, 'learning_rate': 3.9999307326698e-05, 'epoch': 0.33}

  3%|▎         | 1344/41250 [3:14:56<95:41:22,  8.63s/it][2025-04-25 11:12:40,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-25 11:12:40,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.60 | bwd_microstep: 5708.66 | bwd_inner_microstep: 5695.49 | bwd_allreduce_microstep: 13.12 | step_microstep: 18.75
[2025-04-25 11:12:40,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.60 | bwd: 5708.68 | bwd_inner: 5695.49 | bwd_allreduce: 13.14 | step: 18.75

  3%|▎         | 1345/41250 [3:15:05<95:44:12,  8.64s/it]
                                                         
{'loss': 0.1268, 'grad_norm': 1.024200439453125, 'learning_rate': 3.999929419581997e-05, 'epoch': 0.33}

  3%|▎         | 1345/41250 [3:15:05<95:44:12,  8.64s/it][2025-04-25 11:12:48,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.16
[2025-04-25 11:12:48,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.10 | bwd_microstep: 5693.81 | bwd_inner_microstep: 5680.83 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.52
[2025-04-25 11:12:48,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.10 | bwd: 5693.83 | bwd_inner: 5680.83 | bwd_allreduce: 12.96 | step: 19.53

  3%|▎         | 1346/41250 [3:15:14<95:42:55,  8.64s/it]
                                                         
{'loss': 0.0745, 'grad_norm': 1.0968343019485474, 'learning_rate': 3.999928094165023e-05, 'epoch': 0.33}

  3%|▎         | 1346/41250 [3:15:14<95:42:55,  8.64s/it][2025-04-25 11:12:57,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.29 | optimizer_step: 1.07
[2025-04-25 11:12:57,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.14 | bwd_microstep: 5769.35 | bwd_inner_microstep: 5755.57 | bwd_allreduce_microstep: 13.71 | step_microstep: 20.48
[2025-04-25 11:12:57,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.14 | bwd: 5769.36 | bwd_inner: 5755.57 | bwd_allreduce: 13.74 | step: 20.48

  3%|▎         | 1347/41250 [3:15:22<96:02:33,  8.66s/it]
                                                         
{'loss': 0.3235, 'grad_norm': 1.436380386352539, 'learning_rate': 3.9999267564188844e-05, 'epoch': 0.33}

  3%|▎         | 1347/41250 [3:15:22<96:02:33,  8.66s/it][2025-04-25 11:13:06,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 11:13:06,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.57 | bwd_microstep: 5748.23 | bwd_inner_microstep: 5698.91 | bwd_allreduce_microstep: 49.28 | step_microstep: 19.08
[2025-04-25 11:13:06,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.57 | bwd: 5748.25 | bwd_inner: 5698.91 | bwd_allreduce: 49.30 | step: 19.09

  3%|▎         | 1348/41250 [3:15:31<96:05:41,  8.67s/it]
                                                         
{'loss': 0.1993, 'grad_norm': 1.8864206075668335, 'learning_rate': 3.999925406343592e-05, 'epoch': 0.33}

  3%|▎         | 1348/41250 [3:15:31<96:05:41,  8.67s/it][2025-04-25 11:13:14,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 11:13:14,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.14 | bwd_microstep: 5738.46 | bwd_inner_microstep: 5639.43 | bwd_allreduce_microstep: 98.97 | step_microstep: 19.04
[2025-04-25 11:13:14,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.14 | bwd: 5738.48 | bwd_inner: 5639.43 | bwd_allreduce: 99.00 | step: 19.04

  3%|▎         | 1349/41250 [3:15:40<96:01:24,  8.66s/it]
                                                         
{'loss': 0.3561, 'grad_norm': 1.2808197736740112, 'learning_rate': 3.999924043939152e-05, 'epoch': 0.33}

  3%|▎         | 1349/41250 [3:15:40<96:01:24,  8.66s/it][2025-04-25 11:13:23,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 11:13:23,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.35 | bwd_microstep: 5699.96 | bwd_inner_microstep: 5639.31 | bwd_allreduce_microstep: 60.61 | step_microstep: 19.19
[2025-04-25 11:13:23,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.35 | bwd: 5699.98 | bwd_inner: 5639.31 | bwd_allreduce: 60.63 | step: 19.19

  3%|▎         | 1350/41250 [3:15:48<95:48:57,  8.65s/it]
                                                         
{'loss': 0.1269, 'grad_norm': 1.172644853591919, 'learning_rate': 3.999922669205574e-05, 'epoch': 0.33}

  3%|▎         | 1350/41250 [3:15:48<95:48:57,  8.65s/it][2025-04-25 11:13:32,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:13:32,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.59 | bwd_microstep: 5727.11 | bwd_inner_microstep: 5672.98 | bwd_allreduce_microstep: 54.08 | step_microstep: 19.02
[2025-04-25 11:13:32,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.59 | bwd: 5727.12 | bwd_inner: 5672.98 | bwd_allreduce: 54.10 | step: 19.02

  3%|▎         | 1351/41250 [3:15:57<95:51:17,  8.65s/it]
                                                         
{'loss': 0.3457, 'grad_norm': 2.6678719520568848, 'learning_rate': 3.999921282142866e-05, 'epoch': 0.33}

  3%|▎         | 1351/41250 [3:15:57<95:51:17,  8.65s/it][2025-04-25 11:13:40,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 11:13:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.99 | bwd_microstep: 5691.60 | bwd_inner_microstep: 5678.94 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.63
[2025-04-25 11:13:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.99 | bwd: 5691.61 | bwd_inner: 5678.94 | bwd_allreduce: 12.63 | step: 18.64

  3%|▎         | 1352/41250 [3:16:06<95:44:40,  8.64s/it]
                                                         
{'loss': 0.2288, 'grad_norm': 1.3995318412780762, 'learning_rate': 3.9999198827510365e-05, 'epoch': 0.33}

  3%|▎         | 1352/41250 [3:16:06<95:44:40,  8.64s/it][2025-04-25 11:13:49,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:13:49,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2942.54 | bwd_microstep: 5868.69 | bwd_inner_microstep: 5855.81 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.84
[2025-04-25 11:13:49,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2942.54 | bwd: 5868.71 | bwd_inner: 5855.81 | bwd_allreduce: 12.86 | step: 18.85

  3%|▎         | 1353/41250 [3:16:15<96:35:23,  8.72s/it]
                                                         
{'loss': 0.219, 'grad_norm': 1.7674041986465454, 'learning_rate': 3.999918471030095e-05, 'epoch': 0.33}

  3%|▎         | 1353/41250 [3:16:15<96:35:23,  8.72s/it][2025-04-25 11:13:58,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:13:58,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.76 | bwd_microstep: 5729.60 | bwd_inner_microstep: 5690.58 | bwd_allreduce_microstep: 38.97 | step_microstep: 19.06
[2025-04-25 11:13:58,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.76 | bwd: 5729.61 | bwd_inner: 5690.58 | bwd_allreduce: 38.99 | step: 19.07

  3%|▎         | 1354/41250 [3:16:23<96:24:48,  8.70s/it]
                                                         
{'loss': 0.163, 'grad_norm': 1.0584920644760132, 'learning_rate': 3.999917046980049e-05, 'epoch': 0.33}

  3%|▎         | 1354/41250 [3:16:23<96:24:48,  8.70s/it][2025-04-25 11:14:07,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 11:14:07,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.44 | bwd_microstep: 5752.23 | bwd_inner_microstep: 5677.95 | bwd_allreduce_microstep: 74.23 | step_microstep: 19.08
[2025-04-25 11:14:07,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.44 | bwd: 5752.24 | bwd_inner: 5677.95 | bwd_allreduce: 74.25 | step: 19.08

  3%|▎         | 1355/41250 [3:16:32<96:20:16,  8.69s/it]
                                                         
{'loss': 0.1425, 'grad_norm': 1.1420727968215942, 'learning_rate': 3.9999156106009075e-05, 'epoch': 0.33}

  3%|▎         | 1355/41250 [3:16:32<96:20:16,  8.69s/it][2025-04-25 11:14:15,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:14:15,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.84 | bwd_microstep: 5662.47 | bwd_inner_microstep: 5637.92 | bwd_allreduce_microstep: 24.50 | step_microstep: 18.79
[2025-04-25 11:14:15,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.84 | bwd: 5662.49 | bwd_inner: 5637.92 | bwd_allreduce: 24.52 | step: 18.79

  3%|▎         | 1356/41250 [3:16:40<95:56:37,  8.66s/it]
                                                         
{'loss': 0.3638, 'grad_norm': 2.819391965866089, 'learning_rate': 3.9999141618926805e-05, 'epoch': 0.33}

  3%|▎         | 1356/41250 [3:16:40<95:56:37,  8.66s/it][2025-04-25 11:14:24,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:14:24,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.34 | bwd_microstep: 5734.10 | bwd_inner_microstep: 5684.45 | bwd_allreduce_microstep: 49.60 | step_microstep: 18.58
[2025-04-25 11:14:24,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.34 | bwd: 5734.11 | bwd_inner: 5684.45 | bwd_allreduce: 49.62 | step: 18.59

  3%|▎         | 1357/41250 [3:16:49<95:57:02,  8.66s/it]
                                                         
{'loss': 0.2536, 'grad_norm': 1.330527901649475, 'learning_rate': 3.9999127008553754e-05, 'epoch': 0.33}

  3%|▎         | 1357/41250 [3:16:49<95:57:02,  8.66s/it][2025-04-25 11:14:32,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:14:32,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.93 | bwd_microstep: 5723.61 | bwd_inner_microstep: 5696.21 | bwd_allreduce_microstep: 27.35 | step_microstep: 18.81
[2025-04-25 11:14:32,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.93 | bwd: 5723.62 | bwd_inner: 5696.21 | bwd_allreduce: 27.37 | step: 18.81

  3%|▎         | 1358/41250 [3:16:58<95:56:00,  8.66s/it]
                                                         
{'loss': 0.1924, 'grad_norm': 1.445257544517517, 'learning_rate': 3.9999112274890016e-05, 'epoch': 0.33}

  3%|▎         | 1358/41250 [3:16:58<95:56:00,  8.66s/it][2025-04-25 11:14:41,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 11:14:41,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.60 | bwd_microstep: 5725.46 | bwd_inner_microstep: 5677.20 | bwd_allreduce_microstep: 48.21 | step_microstep: 19.17
[2025-04-25 11:14:41,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.60 | bwd: 5725.47 | bwd_inner: 5677.20 | bwd_allreduce: 48.23 | step: 19.17

  3%|▎         | 1359/41250 [3:17:06<95:55:01,  8.66s/it]
                                                         
{'loss': 0.3387, 'grad_norm': 1.5207901000976562, 'learning_rate': 3.9999097417935685e-05, 'epoch': 0.33}

  3%|▎         | 1359/41250 [3:17:06<95:55:01,  8.66s/it][2025-04-25 11:14:50,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:14:50,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.42 | bwd_microstep: 5766.92 | bwd_inner_microstep: 5754.02 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.31
[2025-04-25 11:14:50,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.42 | bwd: 5766.94 | bwd_inner: 5754.02 | bwd_allreduce: 12.88 | step: 18.31

  3%|▎         | 1360/41250 [3:17:15<96:09:23,  8.68s/it]
                                                         
{'loss': 0.3759, 'grad_norm': 2.114044427871704, 'learning_rate': 3.999908243769086e-05, 'epoch': 0.33}

  3%|▎         | 1360/41250 [3:17:15<96:09:23,  8.68s/it][2025-04-25 11:14:59,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:14:59,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5976.00 | bwd_inner_microstep: 5692.05 | bwd_allreduce_microstep: 283.91 | step_microstep: 18.41
[2025-04-25 11:14:59,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5976.01 | bwd_inner: 5692.05 | bwd_allreduce: 283.93 | step: 18.41

  3%|▎         | 1361/41250 [3:17:24<96:54:36,  8.75s/it]
                                                         
{'loss': 0.1967, 'grad_norm': 1.556275725364685, 'learning_rate': 3.999906733415562e-05, 'epoch': 0.33}

  3%|▎         | 1361/41250 [3:17:24<96:54:36,  8.75s/it][2025-04-25 11:15:07,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:15:07,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.25 | bwd_microstep: 5741.61 | bwd_inner_microstep: 5683.62 | bwd_allreduce_microstep: 57.95 | step_microstep: 18.45
[2025-04-25 11:15:07,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.25 | bwd: 5741.62 | bwd_inner: 5683.62 | bwd_allreduce: 57.97 | step: 18.46

  3%|▎         | 1362/41250 [3:17:33<96:37:15,  8.72s/it]
                                                         
{'loss': 0.2502, 'grad_norm': 2.2080869674682617, 'learning_rate': 3.999905210733006e-05, 'epoch': 0.33}

  3%|▎         | 1362/41250 [3:17:33<96:37:15,  8.72s/it][2025-04-25 11:15:16,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:15:16,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.10 | bwd_microstep: 5761.33 | bwd_inner_microstep: 5640.15 | bwd_allreduce_microstep: 121.14 | step_microstep: 18.53
[2025-04-25 11:15:16,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.10 | bwd: 5761.34 | bwd_inner: 5640.15 | bwd_allreduce: 121.15 | step: 18.53

  3%|▎         | 1363/41250 [3:17:41<96:25:13,  8.70s/it]
                                                         
{'loss': 0.1509, 'grad_norm': 1.230355978012085, 'learning_rate': 3.9999036757214285e-05, 'epoch': 0.33}

  3%|▎         | 1363/41250 [3:17:41<96:25:13,  8.70s/it][2025-04-25 11:15:25,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:15:25,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.99 | bwd_microstep: 5689.90 | bwd_inner_microstep: 5677.30 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.63
[2025-04-25 11:15:25,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.99 | bwd: 5689.91 | bwd_inner: 5677.30 | bwd_allreduce: 12.57 | step: 18.63

  3%|▎         | 1364/41250 [3:17:50<96:06:16,  8.67s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 1.0311882495880127, 'learning_rate': 3.9999021283808376e-05, 'epoch': 0.33}

  3%|▎         | 1364/41250 [3:17:50<96:06:16,  8.67s/it][2025-04-25 11:15:33,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 11:15:33,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.05 | bwd_microstep: 5699.01 | bwd_inner_microstep: 5641.67 | bwd_allreduce_microstep: 57.30 | step_microstep: 18.76
[2025-04-25 11:15:33,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.05 | bwd: 5699.02 | bwd_inner: 5641.67 | bwd_allreduce: 57.32 | step: 18.76

  3%|▎         | 1365/41250 [3:17:59<95:54:43,  8.66s/it]
                                                         
{'loss': 0.2819, 'grad_norm': 1.449847936630249, 'learning_rate': 3.999900568711244e-05, 'epoch': 0.33}

  3%|▎         | 1365/41250 [3:17:59<95:54:43,  8.66s/it][2025-04-25 11:15:42,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 11:15:42,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.83 | bwd_microstep: 5784.01 | bwd_inner_microstep: 5633.03 | bwd_allreduce_microstep: 150.94 | step_microstep: 18.54
[2025-04-25 11:15:42,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.83 | bwd: 5784.03 | bwd_inner: 5633.03 | bwd_allreduce: 150.96 | step: 18.55

  3%|▎         | 1366/41250 [3:18:07<96:01:32,  8.67s/it]
                                                         
{'loss': 0.1462, 'grad_norm': 0.9609647989273071, 'learning_rate': 3.9998989967126564e-05, 'epoch': 0.33}

  3%|▎         | 1366/41250 [3:18:07<96:01:32,  8.67s/it][2025-04-25 11:15:51,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 11:15:51,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.80 | bwd_microstep: 5761.77 | bwd_inner_microstep: 5648.78 | bwd_allreduce_microstep: 112.93 | step_microstep: 19.13
[2025-04-25 11:15:51,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.80 | bwd: 5761.78 | bwd_inner: 5648.78 | bwd_allreduce: 112.96 | step: 19.13

  3%|▎         | 1367/41250 [3:18:16<96:04:07,  8.67s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.3011419773101807, 'learning_rate': 3.9998974123850843e-05, 'epoch': 0.33}

  3%|▎         | 1367/41250 [3:18:16<96:04:07,  8.67s/it][2025-04-25 11:15:59,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:15:59,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.63 | bwd_microstep: 5707.44 | bwd_inner_microstep: 5646.64 | bwd_allreduce_microstep: 60.75 | step_microstep: 18.54
[2025-04-25 11:15:59,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.63 | bwd: 5707.45 | bwd_inner: 5646.64 | bwd_allreduce: 60.77 | step: 18.54

  3%|▎         | 1368/41250 [3:18:25<95:52:32,  8.65s/it]
                                                         
{'loss': 0.2282, 'grad_norm': 2.430117607116699, 'learning_rate': 3.9998958157285385e-05, 'epoch': 0.33}

  3%|▎         | 1368/41250 [3:18:25<95:52:32,  8.65s/it][2025-04-25 11:16:08,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:16:08,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.52 | bwd_microstep: 5694.35 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 42.01 | step_microstep: 18.66
[2025-04-25 11:16:08,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.52 | bwd: 5694.36 | bwd_inner: 5652.29 | bwd_allreduce: 42.02 | step: 18.66

  3%|▎         | 1369/41250 [3:18:33<95:43:54,  8.64s/it]
                                                         
{'loss': 0.222, 'grad_norm': 1.2889940738677979, 'learning_rate': 3.999894206743029e-05, 'epoch': 0.33}

  3%|▎         | 1369/41250 [3:18:33<95:43:54,  8.64s/it][2025-04-25 11:16:16,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:16:16,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.82 | bwd_microstep: 5703.77 | bwd_inner_microstep: 5663.94 | bwd_allreduce_microstep: 39.79 | step_microstep: 18.54
[2025-04-25 11:16:16,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.82 | bwd: 5703.79 | bwd_inner: 5663.94 | bwd_allreduce: 39.81 | step: 18.55

  3%|▎         | 1370/41250 [3:18:42<95:39:53,  8.64s/it]
                                                         
{'loss': 0.2489, 'grad_norm': 1.9798486232757568, 'learning_rate': 3.999892585428564e-05, 'epoch': 0.33}

  3%|▎         | 1370/41250 [3:18:42<95:39:53,  8.64s/it][2025-04-25 11:16:25,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:16:25,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5775.95 | bwd_inner_microstep: 5719.03 | bwd_allreduce_microstep: 56.88 | step_microstep: 18.72
[2025-04-25 11:16:25,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5775.97 | bwd_inner: 5719.03 | bwd_allreduce: 56.90 | step: 18.72

  3%|▎         | 1371/41250 [3:18:51<95:55:08,  8.66s/it]
                                                         
{'loss': 0.2226, 'grad_norm': 1.0848548412322998, 'learning_rate': 3.9998909517851555e-05, 'epoch': 0.33}

  3%|▎         | 1371/41250 [3:18:51<95:55:08,  8.66s/it][2025-04-25 11:16:34,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 11:16:34,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.55 | bwd_microstep: 5808.76 | bwd_inner_microstep: 5653.85 | bwd_allreduce_microstep: 154.85 | step_microstep: 18.89
[2025-04-25 11:16:34,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.55 | bwd: 5808.78 | bwd_inner: 5653.85 | bwd_allreduce: 154.88 | step: 18.89

  3%|▎         | 1372/41250 [3:18:59<96:06:27,  8.68s/it]
                                                         
{'loss': 0.2855, 'grad_norm': 2.0616719722747803, 'learning_rate': 3.999889305812812e-05, 'epoch': 0.33}

  3%|▎         | 1372/41250 [3:18:59<96:06:27,  8.68s/it][2025-04-25 11:16:43,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.08 | optimizer_step: 1.15
[2025-04-25 11:16:43,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.31 | bwd_microstep: 5776.92 | bwd_inner_microstep: 5664.82 | bwd_allreduce_microstep: 112.04 | step_microstep: 19.49
[2025-04-25 11:16:43,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.31 | bwd: 5776.93 | bwd_inner: 5664.82 | bwd_allreduce: 112.07 | step: 19.49

  3%|▎         | 1373/41250 [3:19:08<96:11:11,  8.68s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 1.2427140474319458, 'learning_rate': 3.9998876475115446e-05, 'epoch': 0.33}

  3%|▎         | 1373/41250 [3:19:08<96:11:11,  8.68s/it][2025-04-25 11:16:52,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-25 11:16:52,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.71 | bwd_microstep: 6005.92 | bwd_inner_microstep: 5709.99 | bwd_allreduce_microstep: 295.88 | step_microstep: 18.80
[2025-04-25 11:16:52,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.71 | bwd: 6005.94 | bwd_inner: 5709.99 | bwd_allreduce: 295.91 | step: 18.80

  3%|▎         | 1374/41250 [3:19:17<97:05:31,  8.77s/it]
                                                         
{'loss': 0.1317, 'grad_norm': 1.0766936540603638, 'learning_rate': 3.9998859768813624e-05, 'epoch': 0.33}

  3%|▎         | 1374/41250 [3:19:17<97:05:31,  8.77s/it][2025-04-25 11:17:00,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:17:00,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.69 | bwd_microstep: 5748.56 | bwd_inner_microstep: 5704.91 | bwd_allreduce_microstep: 43.60 | step_microstep: 18.56
[2025-04-25 11:17:00,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.69 | bwd: 5748.57 | bwd_inner: 5704.91 | bwd_allreduce: 43.62 | step: 18.56

  3%|▎         | 1375/41250 [3:19:26<96:52:36,  8.75s/it]
                                                         
{'loss': 0.2586, 'grad_norm': 1.0358937978744507, 'learning_rate': 3.9998842939222775e-05, 'epoch': 0.33}

  3%|▎         | 1375/41250 [3:19:26<96:52:36,  8.75s/it][2025-04-25 11:17:09,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 11:17:09,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.54 | bwd_microstep: 5726.40 | bwd_inner_microstep: 5713.49 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.57
[2025-04-25 11:17:09,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.54 | bwd: 5726.42 | bwd_inner: 5713.49 | bwd_allreduce: 12.88 | step: 19.57

  3%|▎         | 1376/41250 [3:19:34<96:36:58,  8.72s/it]
                                                         
{'loss': 0.4179, 'grad_norm': 2.5689709186553955, 'learning_rate': 3.9998825986342995e-05, 'epoch': 0.33}

  3%|▎         | 1376/41250 [3:19:34<96:36:58,  8.72s/it][2025-04-25 11:17:18,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 11:17:18,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.06 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5665.08 | bwd_allreduce_microstep: 52.81 | step_microstep: 19.08
[2025-04-25 11:17:18,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.06 | bwd: 5717.96 | bwd_inner: 5665.08 | bwd_allreduce: 52.83 | step: 19.08

  3%|▎         | 1377/41250 [3:19:43<96:19:27,  8.70s/it]
                                                         
{'loss': 0.236, 'grad_norm': 1.6562987565994263, 'learning_rate': 3.999880891017438e-05, 'epoch': 0.33}

  3%|▎         | 1377/41250 [3:19:43<96:19:27,  8.70s/it][2025-04-25 11:17:26,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.10 | optimizer_step: 1.08
[2025-04-25 11:17:26,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.53 | bwd_microstep: 5773.68 | bwd_inner_microstep: 5702.10 | bwd_allreduce_microstep: 71.53 | step_microstep: 19.48
[2025-04-25 11:17:26,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.53 | bwd: 5773.70 | bwd_inner: 5702.10 | bwd_allreduce: 71.55 | step: 19.48

  3%|▎         | 1378/41250 [3:19:52<96:22:42,  8.70s/it]
                                                         
{'loss': 0.1488, 'grad_norm': 1.9922369718551636, 'learning_rate': 3.9998791710717044e-05, 'epoch': 0.33}

  3%|▎         | 1378/41250 [3:19:52<96:22:42,  8.70s/it][2025-04-25 11:17:35,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:17:35,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.24 | bwd_microstep: 6021.91 | bwd_inner_microstep: 6009.31 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.85
[2025-04-25 11:17:35,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.24 | bwd: 6021.92 | bwd_inner: 6009.31 | bwd_allreduce: 12.57 | step: 18.86

  3%|▎         | 1379/41250 [3:20:01<97:22:09,  8.79s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 1.4864434003829956, 'learning_rate': 3.999877438797109e-05, 'epoch': 0.33}

  3%|▎         | 1379/41250 [3:20:01<97:22:09,  8.79s/it][2025-04-25 11:17:44,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:17:44,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.65 | bwd_microstep: 5768.92 | bwd_inner_microstep: 5698.85 | bwd_allreduce_microstep: 70.02 | step_microstep: 18.89
[2025-04-25 11:17:44,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.65 | bwd: 5768.94 | bwd_inner: 5698.85 | bwd_allreduce: 70.04 | step: 18.89

  3%|▎         | 1380/41250 [3:20:09<97:03:41,  8.76s/it]
                                                         
{'loss': 0.341, 'grad_norm': 2.896867036819458, 'learning_rate': 3.999875694193662e-05, 'epoch': 0.33}

  3%|▎         | 1380/41250 [3:20:09<97:03:41,  8.76s/it][2025-04-25 11:17:53,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 11:17:53,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.37 | bwd_microstep: 5726.96 | bwd_inner_microstep: 5714.02 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.44
[2025-04-25 11:17:53,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.37 | bwd: 5726.97 | bwd_inner: 5714.02 | bwd_allreduce: 12.91 | step: 19.44

  3%|▎         | 1381/41250 [3:20:18<96:45:06,  8.74s/it]
                                                         
{'loss': 0.2676, 'grad_norm': 1.5069801807403564, 'learning_rate': 3.999873937261376e-05, 'epoch': 0.33}

  3%|▎         | 1381/41250 [3:20:18<96:45:06,  8.74s/it][2025-04-25 11:18:01,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:18:01,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5721.09 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 27.84 | step_microstep: 18.94
[2025-04-25 11:18:01,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5721.10 | bwd_inner: 5693.20 | bwd_allreduce: 27.86 | step: 18.94

  3%|▎         | 1382/41250 [3:20:27<96:28:08,  8.71s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.2118775844573975, 'learning_rate': 3.9998721680002595e-05, 'epoch': 0.34}

  3%|▎         | 1382/41250 [3:20:27<96:28:08,  8.71s/it][2025-04-25 11:18:10,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:18:10,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.78 | bwd_microstep: 5767.48 | bwd_inner_microstep: 5684.92 | bwd_allreduce_microstep: 82.51 | step_microstep: 18.58
[2025-04-25 11:18:10,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.78 | bwd: 5767.49 | bwd_inner: 5684.92 | bwd_allreduce: 82.53 | step: 18.58

  3%|▎         | 1383/41250 [3:20:35<96:26:27,  8.71s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 0.7933467626571655, 'learning_rate': 3.999870386410324e-05, 'epoch': 0.34}

  3%|▎         | 1383/41250 [3:20:35<96:26:27,  8.71s/it][2025-04-25 11:18:19,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:18:19,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.94 | bwd_microstep: 5737.36 | bwd_inner_microstep: 5724.45 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.67
[2025-04-25 11:18:19,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.94 | bwd: 5737.38 | bwd_inner: 5724.45 | bwd_allreduce: 12.89 | step: 18.67

  3%|▎         | 1384/41250 [3:20:44<96:19:30,  8.70s/it]
                                                         
{'loss': 0.1772, 'grad_norm': 0.9700731039047241, 'learning_rate': 3.999868592491582e-05, 'epoch': 0.34}

  3%|▎         | 1384/41250 [3:20:44<96:19:30,  8.70s/it][2025-04-25 11:18:27,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:18:27,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.37 | bwd_microstep: 5727.11 | bwd_inner_microstep: 5660.78 | bwd_allreduce_microstep: 66.28 | step_microstep: 18.30
[2025-04-25 11:18:27,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.37 | bwd: 5727.12 | bwd_inner: 5660.78 | bwd_allreduce: 66.29 | step: 18.30

  3%|▎         | 1385/41250 [3:20:53<96:08:02,  8.68s/it]
                                                         
{'loss': 0.3681, 'grad_norm': 2.185584545135498, 'learning_rate': 3.9998667862440426e-05, 'epoch': 0.34}

  3%|▎         | 1385/41250 [3:20:53<96:08:02,  8.68s/it][2025-04-25 11:18:36,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:18:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5804.07 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 153.88 | step_microstep: 18.62
[2025-04-25 11:18:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5804.08 | bwd_inner: 5650.14 | bwd_allreduce: 153.90 | step: 18.62

  3%|▎         | 1386/41250 [3:21:01<96:14:24,  8.69s/it]
                                                         
{'loss': 0.1591, 'grad_norm': 1.77129065990448, 'learning_rate': 3.999864967667718e-05, 'epoch': 0.34}

  3%|▎         | 1386/41250 [3:21:01<96:14:24,  8.69s/it][2025-04-25 11:18:45,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 11:18:45,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.89 | bwd_microstep: 5781.52 | bwd_inner_microstep: 5685.57 | bwd_allreduce_microstep: 95.91 | step_microstep: 18.59
[2025-04-25 11:18:45,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5781.53 | bwd_inner: 5685.57 | bwd_allreduce: 95.92 | step: 18.59

  3%|▎         | 1387/41250 [3:21:10<96:18:23,  8.70s/it]
                                                         
{'loss': 0.303, 'grad_norm': 2.0996463298797607, 'learning_rate': 3.9998631367626196e-05, 'epoch': 0.34}

  3%|▎         | 1387/41250 [3:21:10<96:18:23,  8.70s/it][2025-04-25 11:18:53,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.33 | optimizer_step: 1.06
[2025-04-25 11:18:53,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.13 | bwd_microstep: 5771.57 | bwd_inner_microstep: 5671.17 | bwd_allreduce_microstep: 100.33 | step_microstep: 20.37
[2025-04-25 11:18:53,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.13 | bwd: 5771.58 | bwd_inner: 5671.17 | bwd_allreduce: 100.36 | step: 20.37

  3%|▎         | 1388/41250 [3:21:19<96:16:05,  8.69s/it]
                                                         
{'loss': 0.0539, 'grad_norm': 0.49403780698776245, 'learning_rate': 3.9998612935287587e-05, 'epoch': 0.34}

  3%|▎         | 1388/41250 [3:21:19<96:16:05,  8.69s/it][2025-04-25 11:19:02,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 11:19:02,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.73 | bwd_microstep: 5774.75 | bwd_inner_microstep: 5762.04 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.99
[2025-04-25 11:19:02,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.73 | bwd: 5774.76 | bwd_inner: 5762.04 | bwd_allreduce: 12.68 | step: 19.00

  3%|▎         | 1389/41250 [3:21:28<96:25:04,  8.71s/it]
                                                         
{'loss': 0.182, 'grad_norm': 1.4437061548233032, 'learning_rate': 3.9998594379661455e-05, 'epoch': 0.34}

  3%|▎         | 1389/41250 [3:21:28<96:25:04,  8.71s/it][2025-04-25 11:19:11,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:19:11,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5764.52 | bwd_inner_microstep: 5705.53 | bwd_allreduce_microstep: 58.94 | step_microstep: 18.80
[2025-04-25 11:19:11,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5764.53 | bwd_inner: 5705.53 | bwd_allreduce: 58.96 | step: 18.81

  3%|▎         | 1390/41250 [3:21:36<96:23:46,  8.71s/it]
                                                         
{'loss': 0.3139, 'grad_norm': 2.0039710998535156, 'learning_rate': 3.999857570074792e-05, 'epoch': 0.34}

  3%|▎         | 1390/41250 [3:21:36<96:23:46,  8.71s/it][2025-04-25 11:19:20,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:19:20,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.33 | bwd_microstep: 5795.21 | bwd_inner_microstep: 5782.51 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.62
[2025-04-25 11:19:20,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.33 | bwd: 5795.22 | bwd_inner: 5782.51 | bwd_allreduce: 12.67 | step: 18.62

  3%|▎         | 1391/41250 [3:21:45<96:35:59,  8.72s/it]
                                                         
{'loss': 0.2115, 'grad_norm': 1.750623106956482, 'learning_rate': 3.999855689854711e-05, 'epoch': 0.34}

  3%|▎         | 1391/41250 [3:21:45<96:35:59,  8.72s/it][2025-04-25 11:19:28,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-25 11:19:28,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.17 | bwd_microstep: 5744.72 | bwd_inner_microstep: 5683.98 | bwd_allreduce_microstep: 60.69 | step_microstep: 18.97
[2025-04-25 11:19:28,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.17 | bwd: 5744.74 | bwd_inner: 5683.98 | bwd_allreduce: 60.71 | step: 18.97

  3%|▎         | 1392/41250 [3:21:54<96:27:15,  8.71s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.2518447637557983, 'learning_rate': 3.9998537973059125e-05, 'epoch': 0.34}

  3%|▎         | 1392/41250 [3:21:54<96:27:15,  8.71s/it][2025-04-25 11:19:37,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-25 11:19:37,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.49 | bwd_microstep: 5708.12 | bwd_inner_microstep: 5695.36 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.24
[2025-04-25 11:19:37,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.49 | bwd: 5708.14 | bwd_inner: 5695.36 | bwd_allreduce: 12.73 | step: 19.24

  3%|▎         | 1393/41250 [3:22:02<96:12:44,  8.69s/it]
                                                         
{'loss': 0.1057, 'grad_norm': 1.2836254835128784, 'learning_rate': 3.999851892428408e-05, 'epoch': 0.34}

  3%|▎         | 1393/41250 [3:22:02<96:12:44,  8.69s/it][2025-04-25 11:19:46,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:19:46,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.73 | bwd_microstep: 5781.40 | bwd_inner_microstep: 5650.36 | bwd_allreduce_microstep: 131.00 | step_microstep: 18.65
[2025-04-25 11:19:46,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.73 | bwd: 5781.41 | bwd_inner: 5650.36 | bwd_allreduce: 131.01 | step: 18.65

  3%|▎         | 1394/41250 [3:22:11<96:12:54,  8.69s/it]
                                                         
{'loss': 0.2168, 'grad_norm': 1.8005447387695312, 'learning_rate': 3.9998499752222114e-05, 'epoch': 0.34}

  3%|▎         | 1394/41250 [3:22:11<96:12:54,  8.69s/it][2025-04-25 11:19:54,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:19:54,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.47 | bwd_microstep: 5730.32 | bwd_inner_microstep: 5681.34 | bwd_allreduce_microstep: 48.93 | step_microstep: 18.34
[2025-04-25 11:19:54,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.47 | bwd: 5730.33 | bwd_inner: 5681.34 | bwd_allreduce: 48.95 | step: 18.34

  3%|▎         | 1395/41250 [3:22:20<96:06:55,  8.68s/it]
                                                         
{'loss': 0.1646, 'grad_norm': 2.119445323944092, 'learning_rate': 3.999848045687332e-05, 'epoch': 0.34}

  3%|▎         | 1395/41250 [3:22:20<96:06:55,  8.68s/it][2025-04-25 11:20:03,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 11:20:03,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.74 | bwd_microstep: 5756.76 | bwd_inner_microstep: 5692.59 | bwd_allreduce_microstep: 64.12 | step_microstep: 19.02
[2025-04-25 11:20:03,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.74 | bwd: 5756.77 | bwd_inner: 5692.59 | bwd_allreduce: 64.14 | step: 19.03

  3%|▎         | 1396/41250 [3:22:28<96:07:14,  8.68s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 1.3675205707550049, 'learning_rate': 3.999846103823783e-05, 'epoch': 0.34}

  3%|▎         | 1396/41250 [3:22:28<96:07:14,  8.68s/it][2025-04-25 11:20:12,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:20:12,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.64 | bwd_microstep: 5693.86 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 36.26 | step_microstep: 18.55
[2025-04-25 11:20:12,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.64 | bwd: 5693.87 | bwd_inner: 5657.55 | bwd_allreduce: 36.28 | step: 18.56

  3%|▎         | 1397/41250 [3:22:37<95:52:02,  8.66s/it]
                                                         
{'loss': 0.3393, 'grad_norm': 2.6781556606292725, 'learning_rate': 3.999844149631576e-05, 'epoch': 0.34}

  3%|▎         | 1397/41250 [3:22:37<95:52:02,  8.66s/it][2025-04-25 11:20:20,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:20:20,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.50 | bwd_microstep: 5690.53 | bwd_inner_microstep: 5656.43 | bwd_allreduce_microstep: 34.06 | step_microstep: 18.59
[2025-04-25 11:20:20,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.50 | bwd: 5690.54 | bwd_inner: 5656.43 | bwd_allreduce: 34.08 | step: 18.59

  3%|▎         | 1398/41250 [3:22:46<95:40:27,  8.64s/it]
                                                         
{'loss': 0.189, 'grad_norm': 1.3871299028396606, 'learning_rate': 3.9998421831107234e-05, 'epoch': 0.34}

  3%|▎         | 1398/41250 [3:22:46<95:40:27,  8.64s/it][2025-04-25 11:20:29,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 11:20:29,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.17 | bwd_microstep: 5719.33 | bwd_inner_microstep: 5698.78 | bwd_allreduce_microstep: 20.51 | step_microstep: 18.79
[2025-04-25 11:20:29,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.17 | bwd: 5719.35 | bwd_inner: 5698.78 | bwd_allreduce: 20.53 | step: 18.80

  3%|▎         | 1399/41250 [3:22:54<95:43:42,  8.65s/it]
                                                         
{'loss': 0.0921, 'grad_norm': 1.394959568977356, 'learning_rate': 3.999840204261237e-05, 'epoch': 0.34}

  3%|▎         | 1399/41250 [3:22:54<95:43:42,  8.65s/it][2025-04-25 11:20:38,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:20:38,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.20 | bwd_microstep: 5768.27 | bwd_inner_microstep: 5755.61 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.64
[2025-04-25 11:20:38,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.20 | bwd: 5768.29 | bwd_inner: 5755.61 | bwd_allreduce: 12.64 | step: 18.64

  3%|▎         | 1400/41250 [3:23:03<95:59:39,  8.67s/it]
                                                         
{'loss': 0.2748, 'grad_norm': 3.362046003341675, 'learning_rate': 3.999838213083129e-05, 'epoch': 0.34}

  3%|▎         | 1400/41250 [3:23:03<95:59:39,  8.67s/it][2025-04-25 11:20:46,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:20:46,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.34 | bwd_microstep: 5716.16 | bwd_inner_microstep: 5703.55 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.65
[2025-04-25 11:20:46,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.34 | bwd: 5716.17 | bwd_inner: 5703.55 | bwd_allreduce: 12.59 | step: 18.65

  3%|▎         | 1401/41250 [3:23:12<95:54:18,  8.66s/it]
                                                         
{'loss': 0.1004, 'grad_norm': 0.9084187150001526, 'learning_rate': 3.999836209576412e-05, 'epoch': 0.34}

  3%|▎         | 1401/41250 [3:23:12<95:54:18,  8.66s/it][2025-04-25 11:20:55,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 11:20:55,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.45 | bwd_microstep: 5728.96 | bwd_inner_microstep: 5700.07 | bwd_allreduce_microstep: 28.85 | step_microstep: 19.29
[2025-04-25 11:20:55,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.45 | bwd: 5728.97 | bwd_inner: 5700.07 | bwd_allreduce: 28.87 | step: 19.29

  3%|▎         | 1402/41250 [3:23:20<95:53:56,  8.66s/it]
                                                         
{'loss': 0.2265, 'grad_norm': 2.892106533050537, 'learning_rate': 3.9998341937410975e-05, 'epoch': 0.34}

  3%|▎         | 1402/41250 [3:23:20<95:53:56,  8.66s/it][2025-04-25 11:21:04,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:21:04,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5734.23 | bwd_inner_microstep: 5702.46 | bwd_allreduce_microstep: 31.72 | step_microstep: 18.67
[2025-04-25 11:21:04,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5734.24 | bwd_inner: 5702.46 | bwd_allreduce: 31.74 | step: 18.67

  3%|▎         | 1403/41250 [3:23:29<95:54:33,  8.66s/it]
                                                         
{'loss': 0.4095, 'grad_norm': 1.9956073760986328, 'learning_rate': 3.999832165577199e-05, 'epoch': 0.34}

  3%|▎         | 1403/41250 [3:23:29<95:54:33,  8.66s/it][2025-04-25 11:21:12,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:21:12,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5751.91 | bwd_inner_microstep: 5649.02 | bwd_allreduce_microstep: 102.85 | step_microstep: 18.86
[2025-04-25 11:21:12,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5751.93 | bwd_inner: 5649.02 | bwd_allreduce: 102.87 | step: 18.87

  3%|▎         | 1404/41250 [3:23:38<95:53:48,  8.66s/it]
                                                         
{'loss': 0.0929, 'grad_norm': 1.1313141584396362, 'learning_rate': 3.999830125084729e-05, 'epoch': 0.34}

  3%|▎         | 1404/41250 [3:23:38<95:53:48,  8.66s/it][2025-04-25 11:21:21,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:21:21,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5716.38 | bwd_inner_microstep: 5693.92 | bwd_allreduce_microstep: 22.42 | step_microstep: 18.80
[2025-04-25 11:21:21,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5716.40 | bwd_inner: 5693.92 | bwd_allreduce: 22.44 | step: 18.80

  3%|▎         | 1405/41250 [3:23:46<95:50:46,  8.66s/it]
                                                         
{'loss': 0.1434, 'grad_norm': 1.3431155681610107, 'learning_rate': 3.999828072263699e-05, 'epoch': 0.34}

  3%|▎         | 1405/41250 [3:23:46<95:50:46,  8.66s/it][2025-04-25 11:21:30,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.69 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:21:30,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.31 | bwd_microstep: 5685.47 | bwd_inner_microstep: 5672.75 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.84
[2025-04-25 11:21:30,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.31 | bwd: 5685.49 | bwd_inner: 5672.75 | bwd_allreduce: 12.70 | step: 18.85

  3%|▎         | 1406/41250 [3:23:55<95:40:23,  8.64s/it]
                                                         
{'loss': 0.2496, 'grad_norm': 3.6272995471954346, 'learning_rate': 3.999826007114122e-05, 'epoch': 0.34}

  3%|▎         | 1406/41250 [3:23:55<95:40:23,  8.64s/it][2025-04-25 11:21:38,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:21:38,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.77 | bwd_microstep: 5735.35 | bwd_inner_microstep: 5646.95 | bwd_allreduce_microstep: 88.35 | step_microstep: 18.65
[2025-04-25 11:21:38,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.77 | bwd: 5735.36 | bwd_inner: 5646.95 | bwd_allreduce: 88.37 | step: 18.66

  3%|▎         | 1407/41250 [3:24:03<95:41:47,  8.65s/it]
                                                         
{'loss': 0.2465, 'grad_norm': 1.824353575706482, 'learning_rate': 3.999823929636011e-05, 'epoch': 0.34}

  3%|▎         | 1407/41250 [3:24:03<95:41:47,  8.65s/it][2025-04-25 11:21:47,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 1.05
[2025-04-25 11:21:47,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.79 | bwd_microstep: 5749.70 | bwd_inner_microstep: 5702.17 | bwd_allreduce_microstep: 47.48 | step_microstep: 18.49
[2025-04-25 11:21:47,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.79 | bwd: 5749.72 | bwd_inner: 5702.17 | bwd_allreduce: 47.50 | step: 18.49

  3%|▎         | 1408/41250 [3:24:12<95:48:38,  8.66s/it]
                                                         
{'loss': 0.1371, 'grad_norm': 4.293257236480713, 'learning_rate': 3.999821839829379e-05, 'epoch': 0.34}

  3%|▎         | 1408/41250 [3:24:12<95:48:38,  8.66s/it][2025-04-25 11:21:55,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 11:21:55,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.59 | bwd_microstep: 5759.14 | bwd_inner_microstep: 5650.39 | bwd_allreduce_microstep: 108.70 | step_microstep: 18.98
[2025-04-25 11:21:55,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.59 | bwd: 5759.15 | bwd_inner: 5650.39 | bwd_allreduce: 108.72 | step: 18.98

  3%|▎         | 1409/41250 [3:24:21<95:49:58,  8.66s/it]
                                                         
{'loss': 0.0726, 'grad_norm': 0.6458218097686768, 'learning_rate': 3.99981973769424e-05, 'epoch': 0.34}

  3%|▎         | 1409/41250 [3:24:21<95:49:58,  8.66s/it][2025-04-25 11:22:04,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 11:22:04,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5770.90 | bwd_inner_microstep: 5633.24 | bwd_allreduce_microstep: 137.62 | step_microstep: 18.65
[2025-04-25 11:22:04,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5770.91 | bwd_inner: 5633.24 | bwd_allreduce: 137.64 | step: 18.65

  3%|▎         | 1410/41250 [3:24:30<95:54:36,  8.67s/it]
                                                         
{'loss': 0.0849, 'grad_norm': 1.053733468055725, 'learning_rate': 3.9998176232306044e-05, 'epoch': 0.34}

  3%|▎         | 1410/41250 [3:24:30<95:54:36,  8.67s/it][2025-04-25 11:22:13,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:22:13,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.32 | bwd_microstep: 5691.51 | bwd_inner_microstep: 5678.86 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.50
[2025-04-25 11:22:13,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.32 | bwd: 5691.52 | bwd_inner: 5678.86 | bwd_allreduce: 12.62 | step: 18.50

  3%|▎         | 1411/41250 [3:24:38<95:44:34,  8.65s/it]
                                                         
{'loss': 0.3443, 'grad_norm': 3.012908458709717, 'learning_rate': 3.999815496438486e-05, 'epoch': 0.34}

  3%|▎         | 1411/41250 [3:24:38<95:44:34,  8.65s/it][2025-04-25 11:22:21,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:22:21,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.70 | bwd_microstep: 5771.20 | bwd_inner_microstep: 5645.57 | bwd_allreduce_microstep: 125.59 | step_microstep: 18.71
[2025-04-25 11:22:21,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.70 | bwd: 5771.21 | bwd_inner: 5645.57 | bwd_allreduce: 125.60 | step: 18.71

  3%|▎         | 1412/41250 [3:24:47<95:49:24,  8.66s/it]
                                                         
{'loss': 0.2642, 'grad_norm': 1.6395816802978516, 'learning_rate': 3.9998133573179e-05, 'epoch': 0.34}

  3%|▎         | 1412/41250 [3:24:47<95:49:24,  8.66s/it][2025-04-25 11:22:30,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:22:30,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.67 | bwd_microstep: 5684.04 | bwd_inner_microstep: 5636.91 | bwd_allreduce_microstep: 47.08 | step_microstep: 18.32
[2025-04-25 11:22:30,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.67 | bwd: 5684.05 | bwd_inner: 5636.92 | bwd_allreduce: 47.09 | step: 18.32

  3%|▎         | 1413/41250 [3:24:55<95:34:59,  8.64s/it]
                                                         
{'loss': 0.175, 'grad_norm': 1.7686216831207275, 'learning_rate': 3.999811205868857e-05, 'epoch': 0.34}

  3%|▎         | 1413/41250 [3:24:55<95:34:59,  8.64s/it][2025-04-25 11:22:39,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:22:39,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5696.10 | bwd_inner_microstep: 5683.52 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.53
[2025-04-25 11:22:39,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5696.12 | bwd_inner: 5683.52 | bwd_allreduce: 12.56 | step: 18.54

  3%|▎         | 1414/41250 [3:25:04<95:33:42,  8.64s/it]
                                                         
{'loss': 0.1541, 'grad_norm': 2.098625659942627, 'learning_rate': 3.999809042091371e-05, 'epoch': 0.34}

  3%|▎         | 1414/41250 [3:25:04<95:33:42,  8.64s/it][2025-04-25 11:22:48,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:22:48,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2926.81 | bwd_microstep: 5859.50 | bwd_inner_microstep: 5846.72 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.40
[2025-04-25 11:22:48,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2926.81 | bwd: 5859.51 | bwd_inner: 5846.72 | bwd_allreduce: 12.75 | step: 18.40

  3%|▎         | 1415/41250 [3:25:13<96:20:06,  8.71s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 1.0518525838851929, 'learning_rate': 3.999806865985456e-05, 'epoch': 0.34}

  3%|▎         | 1415/41250 [3:25:13<96:20:06,  8.71s/it][2025-04-25 11:22:56,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:22:56,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.42 | bwd_microstep: 5761.15 | bwd_inner_microstep: 5635.54 | bwd_allreduce_microstep: 125.56 | step_microstep: 18.67
[2025-04-25 11:22:56,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.42 | bwd: 5761.16 | bwd_inner: 5635.53 | bwd_allreduce: 125.58 | step: 18.67

  3%|▎         | 1416/41250 [3:25:22<96:11:13,  8.69s/it]
                                                         
{'loss': 0.1435, 'grad_norm': 1.733988642692566, 'learning_rate': 3.999804677551125e-05, 'epoch': 0.34}

  3%|▎         | 1416/41250 [3:25:22<96:11:13,  8.69s/it][2025-04-25 11:23:05,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 11:23:05,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5701.66 | bwd_inner_microstep: 5638.67 | bwd_allreduce_microstep: 62.95 | step_microstep: 18.98
[2025-04-25 11:23:05,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5701.67 | bwd_inner: 5638.67 | bwd_allreduce: 62.97 | step: 18.99

  3%|▎         | 1417/41250 [3:25:30<95:54:13,  8.67s/it]
                                                         
{'loss': 0.2358, 'grad_norm': 1.232284665107727, 'learning_rate': 3.999802476788391e-05, 'epoch': 0.34}

  3%|▎         | 1417/41250 [3:25:30<95:54:13,  8.67s/it][2025-04-25 11:23:13,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.15 | optimizer_step: 0.91
[2025-04-25 11:23:13,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5710.09 | bwd_inner_microstep: 5673.77 | bwd_allreduce_microstep: 36.27 | step_microstep: 19.31
[2025-04-25 11:23:13,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5710.10 | bwd_inner: 5673.77 | bwd_allreduce: 36.29 | step: 19.31

  3%|▎         | 1418/41250 [3:25:39<95:48:25,  8.66s/it]
                                                         
{'loss': 0.2635, 'grad_norm': 2.417421579360962, 'learning_rate': 3.999800263697269e-05, 'epoch': 0.34}

  3%|▎         | 1418/41250 [3:25:39<95:48:25,  8.66s/it][2025-04-25 11:23:22,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 11:23:22,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.12 | bwd_microstep: 5765.34 | bwd_inner_microstep: 5638.58 | bwd_allreduce_microstep: 126.72 | step_microstep: 19.04
[2025-04-25 11:23:22,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.12 | bwd: 5765.35 | bwd_inner: 5638.58 | bwd_allreduce: 126.74 | step: 19.04

  3%|▎         | 1419/41250 [3:25:47<95:51:56,  8.66s/it]
                                                         
{'loss': 0.0979, 'grad_norm': 0.8288367986679077, 'learning_rate': 3.999798038277771e-05, 'epoch': 0.34}

  3%|▎         | 1419/41250 [3:25:47<95:51:56,  8.66s/it][2025-04-25 11:23:31,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.21 | optimizer_step: 0.89
[2025-04-25 11:23:31,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.18 | bwd_microstep: 5694.52 | bwd_inner_microstep: 5632.49 | bwd_allreduce_microstep: 61.99 | step_microstep: 18.83
[2025-04-25 11:23:31,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.18 | bwd: 5694.54 | bwd_inner: 5632.49 | bwd_allreduce: 62.01 | step: 18.83

  3%|▎         | 1420/41250 [3:25:56<95:38:45,  8.64s/it]
                                                         
{'loss': 0.2293, 'grad_norm': 1.5402086973190308, 'learning_rate': 3.999795800529912e-05, 'epoch': 0.34}

  3%|▎         | 1420/41250 [3:25:56<95:38:45,  8.64s/it][2025-04-25 11:23:39,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:23:39,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.95 | bwd_microstep: 5753.76 | bwd_inner_microstep: 5630.13 | bwd_allreduce_microstep: 123.57 | step_microstep: 18.66
[2025-04-25 11:23:39,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.95 | bwd: 5753.77 | bwd_inner: 5630.13 | bwd_allreduce: 123.59 | step: 18.67

  3%|▎         | 1421/41250 [3:26:05<95:39:53,  8.65s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 1.5712441205978394, 'learning_rate': 3.9997935504537046e-05, 'epoch': 0.34}

  3%|▎         | 1421/41250 [3:26:05<95:39:53,  8.65s/it][2025-04-25 11:23:48,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 11:23:48,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5667.89 | bwd_inner_microstep: 5654.83 | bwd_allreduce_microstep: 13.02 | step_microstep: 19.22
[2025-04-25 11:23:48,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5667.91 | bwd_inner: 5654.83 | bwd_allreduce: 13.03 | step: 19.22

  3%|▎         | 1422/41250 [3:26:13<95:30:31,  8.63s/it]
                                                         
{'loss': 0.3226, 'grad_norm': 1.669744849205017, 'learning_rate': 3.9997912880491635e-05, 'epoch': 0.34}

  3%|▎         | 1422/41250 [3:26:13<95:30:31,  8.63s/it][2025-04-25 11:23:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.18 | optimizer_step: 0.93
[2025-04-25 11:23:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.60 | bwd_microstep: 5732.41 | bwd_inner_microstep: 5636.89 | bwd_allreduce_microstep: 95.47 | step_microstep: 19.46
[2025-04-25 11:23:57,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.60 | bwd: 5732.42 | bwd_inner: 5636.89 | bwd_allreduce: 95.49 | step: 19.46

  3%|▎         | 1423/41250 [3:26:22<95:31:00,  8.63s/it]
                                                         
{'loss': 0.2974, 'grad_norm': 3.115917682647705, 'learning_rate': 3.999789013316303e-05, 'epoch': 0.34}

  3%|▎         | 1423/41250 [3:26:22<95:31:00,  8.63s/it][2025-04-25 11:24:05,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:24:05,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2871.83 | bwd_microstep: 5755.52 | bwd_inner_microstep: 5742.85 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.81
[2025-04-25 11:24:05,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2871.83 | bwd: 5755.54 | bwd_inner: 5742.85 | bwd_allreduce: 12.64 | step: 18.82

  3%|▎         | 1424/41250 [3:26:31<95:46:28,  8.66s/it]
                                                         
{'loss': 0.1266, 'grad_norm': 1.5184566974639893, 'learning_rate': 3.9997867262551355e-05, 'epoch': 0.35}

  3%|▎         | 1424/41250 [3:26:31<95:46:28,  8.66s/it][2025-04-25 11:24:14,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:24:14,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.47 | bwd_microstep: 5690.89 | bwd_inner_microstep: 5678.19 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.49
[2025-04-25 11:24:14,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.47 | bwd: 5690.90 | bwd_inner: 5678.19 | bwd_allreduce: 12.68 | step: 18.49

  3%|▎         | 1425/41250 [3:26:39<95:37:57,  8.64s/it]
                                                         
{'loss': 0.3523, 'grad_norm': 2.708148241043091, 'learning_rate': 3.999784426865677e-05, 'epoch': 0.35}

  3%|▎         | 1425/41250 [3:26:39<95:37:57,  8.64s/it][2025-04-25 11:24:23,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:24:23,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.63 | bwd_microstep: 5773.56 | bwd_inner_microstep: 5640.83 | bwd_allreduce_microstep: 132.68 | step_microstep: 18.68
[2025-04-25 11:24:23,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.63 | bwd: 5773.57 | bwd_inner: 5640.83 | bwd_allreduce: 132.70 | step: 18.68

  3%|▎         | 1426/41250 [3:26:48<95:43:57,  8.65s/it]
                                                         
{'loss': 0.2165, 'grad_norm': 3.4262826442718506, 'learning_rate': 3.999782115147941e-05, 'epoch': 0.35}

  3%|▎         | 1426/41250 [3:26:48<95:43:57,  8.65s/it][2025-04-25 11:24:31,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 11:24:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.88 | bwd_microstep: 5722.95 | bwd_inner_microstep: 5681.20 | bwd_allreduce_microstep: 41.70 | step_microstep: 19.33
[2025-04-25 11:24:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.88 | bwd: 5722.96 | bwd_inner: 5681.20 | bwd_allreduce: 41.72 | step: 19.33

  3%|▎         | 1427/41250 [3:26:57<95:43:07,  8.65s/it]
                                                         
{'loss': 0.1583, 'grad_norm': 2.0162224769592285, 'learning_rate': 3.999779791101941e-05, 'epoch': 0.35}

  3%|▎         | 1427/41250 [3:26:57<95:43:07,  8.65s/it][2025-04-25 11:24:40,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:24:40,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.66 | bwd_microstep: 5745.60 | bwd_inner_microstep: 5628.57 | bwd_allreduce_microstep: 116.98 | step_microstep: 19.19
[2025-04-25 11:24:40,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.66 | bwd: 5745.61 | bwd_inner: 5628.57 | bwd_allreduce: 117.00 | step: 19.20

  3%|▎         | 1428/41250 [3:27:05<95:41:37,  8.65s/it]
                                                         
{'loss': 0.0443, 'grad_norm': 0.3816164433956146, 'learning_rate': 3.999777454727692e-05, 'epoch': 0.35}

  3%|▎         | 1428/41250 [3:27:05<95:41:37,  8.65s/it][2025-04-25 11:24:49,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:24:49,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.37 | bwd_microstep: 5701.77 | bwd_inner_microstep: 5624.05 | bwd_allreduce_microstep: 77.68 | step_microstep: 18.80
[2025-04-25 11:24:49,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.37 | bwd: 5701.78 | bwd_inner: 5624.05 | bwd_allreduce: 77.69 | step: 18.80

  3%|▎         | 1429/41250 [3:27:14<95:31:56,  8.64s/it]
                                                         
{'loss': 0.2254, 'grad_norm': 1.4334303140640259, 'learning_rate': 3.999775106025209e-05, 'epoch': 0.35}

  3%|▎         | 1429/41250 [3:27:14<95:31:56,  8.64s/it][2025-04-25 11:24:57,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:24:57,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.77 | bwd_microstep: 5726.45 | bwd_inner_microstep: 5688.49 | bwd_allreduce_microstep: 37.92 | step_microstep: 18.33
[2025-04-25 11:24:57,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.77 | bwd: 5726.47 | bwd_inner: 5688.49 | bwd_allreduce: 37.94 | step: 18.33

  3%|▎         | 1430/41250 [3:27:23<95:34:39,  8.64s/it]
                                                         
{'loss': 0.2416, 'grad_norm': 1.2980341911315918, 'learning_rate': 3.9997727449945046e-05, 'epoch': 0.35}

  3%|▎         | 1430/41250 [3:27:23<95:34:39,  8.64s/it][2025-04-25 11:25:06,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:25:06,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.66 | bwd_microstep: 5785.60 | bwd_inner_microstep: 5630.81 | bwd_allreduce_microstep: 154.75 | step_microstep: 18.34
[2025-04-25 11:25:06,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.66 | bwd: 5785.62 | bwd_inner: 5630.81 | bwd_allreduce: 154.76 | step: 18.34

  3%|▎         | 1431/41250 [3:27:31<95:43:02,  8.65s/it]
                                                         
{'loss': 0.0622, 'grad_norm': 0.56046462059021, 'learning_rate': 3.9997703716355957e-05, 'epoch': 0.35}

  3%|▎         | 1431/41250 [3:27:31<95:43:02,  8.65s/it][2025-04-25 11:25:15,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-25 11:25:15,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5730.44 | bwd_inner_microstep: 5686.66 | bwd_allreduce_microstep: 43.72 | step_microstep: 18.73
[2025-04-25 11:25:15,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5730.46 | bwd_inner: 5686.66 | bwd_allreduce: 43.74 | step: 18.73

  3%|▎         | 1432/41250 [3:27:40<95:44:26,  8.66s/it]
                                                         
{'loss': 0.2442, 'grad_norm': 2.771162748336792, 'learning_rate': 3.999767985948495e-05, 'epoch': 0.35}

  3%|▎         | 1432/41250 [3:27:40<95:44:26,  8.66s/it][2025-04-25 11:25:23,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 11:25:23,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.09 | bwd_microstep: 5700.75 | bwd_inner_microstep: 5640.69 | bwd_allreduce_microstep: 60.01 | step_microstep: 19.24
[2025-04-25 11:25:23,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.10 | bwd: 5700.76 | bwd_inner: 5640.69 | bwd_allreduce: 60.03 | step: 19.25

  3%|▎         | 1433/41250 [3:27:48<95:33:32,  8.64s/it]
                                                         
{'loss': 0.2292, 'grad_norm': 1.1043473482131958, 'learning_rate': 3.9997655879332186e-05, 'epoch': 0.35}

  3%|▎         | 1433/41250 [3:27:48<95:33:32,  8.64s/it][2025-04-25 11:25:32,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 11:25:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5735.81 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 34.62 | step_microstep: 18.93
[2025-04-25 11:25:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5735.83 | bwd_inner: 5701.15 | bwd_allreduce: 34.64 | step: 18.93

  3%|▎         | 1434/41250 [3:27:57<95:38:54,  8.65s/it]
                                                         
{'loss': 0.1443, 'grad_norm': 1.1113600730895996, 'learning_rate': 3.99976317758978e-05, 'epoch': 0.35}

  3%|▎         | 1434/41250 [3:27:57<95:38:54,  8.65s/it][2025-04-25 11:25:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:25:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.69 | bwd_microstep: 6031.16 | bwd_inner_microstep: 5658.69 | bwd_allreduce_microstep: 372.43 | step_microstep: 18.65
[2025-04-25 11:25:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.69 | bwd: 6031.17 | bwd_inner: 5658.69 | bwd_allreduce: 372.45 | step: 18.66

  3%|▎         | 1435/41250 [3:28:06<96:38:03,  8.74s/it]
                                                         
{'loss': 0.2656, 'grad_norm': 2.1798603534698486, 'learning_rate': 3.9997607549181954e-05, 'epoch': 0.35}

  3%|▎         | 1435/41250 [3:28:06<96:38:03,  8.74s/it][2025-04-25 11:25:49,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:25:49,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.88 | bwd_microstep: 5724.55 | bwd_inner_microstep: 5646.45 | bwd_allreduce_microstep: 78.05 | step_microstep: 18.50
[2025-04-25 11:25:49,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.88 | bwd: 5724.56 | bwd_inner: 5646.45 | bwd_allreduce: 78.06 | step: 18.50

  3%|▎         | 1436/41250 [3:28:15<96:15:56,  8.70s/it]
                                                         
{'loss': 0.3787, 'grad_norm': 4.490288257598877, 'learning_rate': 3.999758319918479e-05, 'epoch': 0.35}

  3%|▎         | 1436/41250 [3:28:15<96:15:56,  8.70s/it][2025-04-25 11:25:58,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 11:25:58,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.59 | bwd_microstep: 5782.31 | bwd_inner_microstep: 5769.58 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.85
[2025-04-25 11:25:58,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.59 | bwd: 5782.32 | bwd_inner: 5769.58 | bwd_allreduce: 12.70 | step: 18.86

  3%|▎         | 1437/41250 [3:28:23<96:24:20,  8.72s/it]
                                                         
{'loss': 0.298, 'grad_norm': 2.392674446105957, 'learning_rate': 3.999755872590646e-05, 'epoch': 0.35}

  3%|▎         | 1437/41250 [3:28:23<96:24:20,  8.72s/it][2025-04-25 11:26:07,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:26:07,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5776.93 | bwd_inner_microstep: 5652.72 | bwd_allreduce_microstep: 124.16 | step_microstep: 18.79
[2025-04-25 11:26:07,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5776.95 | bwd_inner: 5652.72 | bwd_allreduce: 124.18 | step: 18.79

  3%|▎         | 1438/41250 [3:28:32<96:17:47,  8.71s/it]
                                                         
{'loss': 0.3224, 'grad_norm': 1.5690569877624512, 'learning_rate': 3.9997534129347106e-05, 'epoch': 0.35}

  3%|▎         | 1438/41250 [3:28:32<96:17:47,  8.71s/it][2025-04-25 11:26:15,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 11:26:15,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.45 | bwd_microstep: 5705.25 | bwd_inner_microstep: 5692.33 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.17
[2025-04-25 11:26:15,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.45 | bwd: 5705.26 | bwd_inner: 5692.33 | bwd_allreduce: 12.89 | step: 19.17

  3%|▎         | 1439/41250 [3:28:41<96:04:07,  8.69s/it]
                                                         
{'loss': 0.3279, 'grad_norm': 2.016545057296753, 'learning_rate': 3.99975094095069e-05, 'epoch': 0.35}

  3%|▎         | 1439/41250 [3:28:41<96:04:07,  8.69s/it][2025-04-25 11:26:24,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:26:24,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.03 | bwd_microstep: 5710.79 | bwd_inner_microstep: 5698.10 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.82
[2025-04-25 11:26:24,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.03 | bwd: 5710.81 | bwd_inner: 5698.10 | bwd_allreduce: 12.67 | step: 18.83

  3%|▎         | 1440/41250 [3:28:49<95:55:23,  8.67s/it]
                                                         
{'loss': 0.101, 'grad_norm': 0.9569809436798096, 'learning_rate': 3.999748456638597e-05, 'epoch': 0.35}

  3%|▎         | 1440/41250 [3:28:49<95:55:23,  8.67s/it][2025-04-25 11:26:33,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.97
[2025-04-25 11:26:33,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.82 | bwd_microstep: 5720.44 | bwd_inner_microstep: 5659.96 | bwd_allreduce_microstep: 60.44 | step_microstep: 18.46
[2025-04-25 11:26:33,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.82 | bwd: 5720.45 | bwd_inner: 5659.96 | bwd_allreduce: 60.45 | step: 18.47

  3%|▎         | 1441/41250 [3:28:58<95:46:55,  8.66s/it]
                                                         
{'loss': 0.087, 'grad_norm': 1.3831425905227661, 'learning_rate': 3.9997459599984484e-05, 'epoch': 0.35}

  3%|▎         | 1441/41250 [3:28:58<95:46:55,  8.66s/it][2025-04-25 11:26:41,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:26:41,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.45 | bwd_microstep: 5722.64 | bwd_inner_microstep: 5650.68 | bwd_allreduce_microstep: 71.91 | step_microstep: 18.68
[2025-04-25 11:26:41,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.45 | bwd: 5722.65 | bwd_inner: 5650.68 | bwd_allreduce: 71.93 | step: 18.68

  3%|▎         | 1442/41250 [3:29:07<95:42:05,  8.65s/it]
                                                         
{'loss': 0.2176, 'grad_norm': 1.858638882637024, 'learning_rate': 3.999743451030259e-05, 'epoch': 0.35}

  3%|▎         | 1442/41250 [3:29:07<95:42:05,  8.65s/it][2025-04-25 11:26:50,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:26:50,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.23 | bwd_microstep: 5777.78 | bwd_inner_microstep: 5662.47 | bwd_allreduce_microstep: 115.26 | step_microstep: 18.71
[2025-04-25 11:26:50,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.23 | bwd: 5777.79 | bwd_inner: 5662.47 | bwd_allreduce: 115.28 | step: 18.72

  3%|▎         | 1443/41250 [3:29:15<95:53:02,  8.67s/it]
                                                         
{'loss': 0.337, 'grad_norm': 2.546213150024414, 'learning_rate': 3.9997409297340456e-05, 'epoch': 0.35}

  3%|▎         | 1443/41250 [3:29:15<95:53:02,  8.67s/it][2025-04-25 11:26:59,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-25 11:26:59,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.64 | bwd_microstep: 5729.51 | bwd_inner_microstep: 5716.57 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.17
[2025-04-25 11:26:59,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.64 | bwd: 5729.52 | bwd_inner: 5716.57 | bwd_allreduce: 12.91 | step: 19.17

  4%|▎         | 1444/41250 [3:29:24<95:52:50,  8.67s/it]
                                                         
{'loss': 0.1374, 'grad_norm': 1.1221134662628174, 'learning_rate': 3.999738396109822e-05, 'epoch': 0.35}

  4%|▎         | 1444/41250 [3:29:24<95:52:50,  8.67s/it][2025-04-25 11:27:07,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 11:27:07,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.90 | bwd_microstep: 5795.32 | bwd_inner_microstep: 5668.34 | bwd_allreduce_microstep: 126.93 | step_microstep: 18.72
[2025-04-25 11:27:07,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.90 | bwd: 5795.34 | bwd_inner: 5668.34 | bwd_allreduce: 126.95 | step: 18.72

  4%|▎         | 1445/41250 [3:29:33<96:02:17,  8.69s/it]
                                                         
{'loss': 0.2695, 'grad_norm': 1.344070315361023, 'learning_rate': 3.9997358501576044e-05, 'epoch': 0.35}

  4%|▎         | 1445/41250 [3:29:33<96:02:17,  8.69s/it][2025-04-25 11:27:16,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-25 11:27:16,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.12 | bwd_microstep: 5726.59 | bwd_inner_microstep: 5661.77 | bwd_allreduce_microstep: 64.77 | step_microstep: 18.49
[2025-04-25 11:27:16,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.12 | bwd: 5726.60 | bwd_inner: 5661.77 | bwd_allreduce: 64.79 | step: 18.49

  4%|▎         | 1446/41250 [3:29:41<95:55:51,  8.68s/it]
                                                         
{'loss': 0.0853, 'grad_norm': 0.9817301034927368, 'learning_rate': 3.999733291877409e-05, 'epoch': 0.35}

  4%|▎         | 1446/41250 [3:29:41<95:55:51,  8.68s/it][2025-04-25 11:27:25,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:27:25,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.70 | bwd_microstep: 5747.88 | bwd_inner_microstep: 5690.61 | bwd_allreduce_microstep: 57.22 | step_microstep: 18.57
[2025-04-25 11:27:25,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.70 | bwd: 5747.89 | bwd_inner: 5690.61 | bwd_allreduce: 57.24 | step: 18.57

  4%|▎         | 1447/41250 [3:29:50<95:59:04,  8.68s/it]
                                                         
{'loss': 0.2504, 'grad_norm': 1.3940808773040771, 'learning_rate': 3.999730721269251e-05, 'epoch': 0.35}

  4%|▎         | 1447/41250 [3:29:50<95:59:04,  8.68s/it][2025-04-25 11:27:34,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 11:27:34,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.26 | bwd_microstep: 5776.36 | bwd_inner_microstep: 5700.32 | bwd_allreduce_microstep: 75.99 | step_microstep: 19.24
[2025-04-25 11:27:34,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.26 | bwd: 5776.38 | bwd_inner: 5700.32 | bwd_allreduce: 76.01 | step: 19.24

  4%|▎         | 1448/41250 [3:29:59<96:05:31,  8.69s/it]
                                                         
{'loss': 0.2475, 'grad_norm': 1.4721763134002686, 'learning_rate': 3.999728138333146e-05, 'epoch': 0.35}

  4%|▎         | 1448/41250 [3:29:59<96:05:31,  8.69s/it][2025-04-25 11:27:42,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:27:42,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5731.90 | bwd_inner_microstep: 5718.90 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.04
[2025-04-25 11:27:42,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5731.92 | bwd_inner: 5718.90 | bwd_allreduce: 12.98 | step: 19.04

  4%|▎         | 1449/41250 [3:30:08<96:02:03,  8.69s/it]
                                                         
{'loss': 0.2153, 'grad_norm': 1.7400996685028076, 'learning_rate': 3.999725543069111e-05, 'epoch': 0.35}

  4%|▎         | 1449/41250 [3:30:08<96:02:03,  8.69s/it][2025-04-25 11:27:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 11:27:51,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.51 | bwd_microstep: 5882.25 | bwd_inner_microstep: 5699.14 | bwd_allreduce_microstep: 183.06 | step_microstep: 18.78
[2025-04-25 11:27:51,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.51 | bwd: 5882.27 | bwd_inner: 5699.14 | bwd_allreduce: 183.08 | step: 18.78

  4%|▎         | 1450/41250 [3:30:16<96:29:55,  8.73s/it]
                                                         
{'loss': 0.2487, 'grad_norm': 1.717944860458374, 'learning_rate': 3.9997229354771615e-05, 'epoch': 0.35}

  4%|▎         | 1450/41250 [3:30:16<96:29:55,  8.73s/it][2025-04-25 11:28:00,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 11:28:00,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.32 | bwd_microstep: 5719.50 | bwd_inner_microstep: 5651.80 | bwd_allreduce_microstep: 67.66 | step_microstep: 18.66
[2025-04-25 11:28:00,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.32 | bwd: 5719.52 | bwd_inner: 5651.80 | bwd_allreduce: 67.67 | step: 18.66

  4%|▎         | 1451/41250 [3:30:25<96:11:10,  8.70s/it]
                                                         
{'loss': 0.2166, 'grad_norm': 1.491942048072815, 'learning_rate': 3.9997203155573124e-05, 'epoch': 0.35}

  4%|▎         | 1451/41250 [3:30:25<96:11:10,  8.70s/it][2025-04-25 11:28:08,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 11:28:08,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.46 | bwd_microstep: 5722.41 | bwd_inner_microstep: 5660.16 | bwd_allreduce_microstep: 62.20 | step_microstep: 18.85
[2025-04-25 11:28:08,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.46 | bwd: 5722.43 | bwd_inner: 5660.16 | bwd_allreduce: 62.22 | step: 18.85

  4%|▎         | 1452/41250 [3:30:34<95:59:16,  8.68s/it]
                                                         
{'loss': 0.1703, 'grad_norm': 1.0099616050720215, 'learning_rate': 3.999717683309582e-05, 'epoch': 0.35}

  4%|▎         | 1452/41250 [3:30:34<95:59:16,  8.68s/it][2025-04-25 11:28:17,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 11:28:17,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.00 | bwd_microstep: 5765.79 | bwd_inner_microstep: 5674.96 | bwd_allreduce_microstep: 90.79 | step_microstep: 19.23
[2025-04-25 11:28:17,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.00 | bwd: 5765.81 | bwd_inner: 5674.96 | bwd_allreduce: 90.81 | step: 19.23

  4%|▎         | 1453/41250 [3:30:42<96:01:43,  8.69s/it]
                                                         
{'loss': 0.2077, 'grad_norm': 1.3489964008331299, 'learning_rate': 3.9997150387339846e-05, 'epoch': 0.35}

  4%|▎         | 1453/41250 [3:30:42<96:01:43,  8.69s/it][2025-04-25 11:28:26,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 11:28:26,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.92 | bwd_microstep: 5715.32 | bwd_inner_microstep: 5656.15 | bwd_allreduce_microstep: 59.13 | step_microstep: 18.70
[2025-04-25 11:28:26,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.92 | bwd: 5715.34 | bwd_inner: 5656.15 | bwd_allreduce: 59.15 | step: 18.70

  4%|▎         | 1454/41250 [3:30:51<95:49:21,  8.67s/it]
                                                         
{'loss': 0.3126, 'grad_norm': 1.287708044052124, 'learning_rate': 3.999712381830538e-05, 'epoch': 0.35}

  4%|▎         | 1454/41250 [3:30:51<95:49:21,  8.67s/it][2025-04-25 11:28:34,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 11:28:34,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.00 | bwd_microstep: 5736.17 | bwd_inner_microstep: 5700.10 | bwd_allreduce_microstep: 36.03 | step_microstep: 18.45
[2025-04-25 11:28:34,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.00 | bwd: 5736.19 | bwd_inner: 5700.10 | bwd_allreduce: 36.05 | step: 18.45

  4%|▎         | 1455/41250 [3:31:00<95:50:31,  8.67s/it]
                                                         
{'loss': 0.2074, 'grad_norm': 0.9167425036430359, 'learning_rate': 3.999709712599257e-05, 'epoch': 0.35}

  4%|▎         | 1455/41250 [3:31:00<95:50:31,  8.67s/it][2025-04-25 11:28:43,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 11:28:43,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.79 | bwd_microstep: 5698.04 | bwd_inner_microstep: 5685.34 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.51
[2025-04-25 11:28:43,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.79 | bwd: 5698.06 | bwd_inner: 5685.34 | bwd_allreduce: 12.67 | step: 19.51

  4%|▎         | 1456/41250 [3:31:08<95:45:21,  8.66s/it]
                                                         
{'loss': 0.1639, 'grad_norm': 1.2690480947494507, 'learning_rate': 3.9997070310401595e-05, 'epoch': 0.35}

  4%|▎         | 1456/41250 [3:31:08<95:45:21,  8.66s/it][2025-04-25 11:28:52,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:28:52,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5691.06 | bwd_inner_microstep: 5678.38 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.19
[2025-04-25 11:28:52,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5691.08 | bwd_inner: 5678.38 | bwd_allreduce: 12.66 | step: 19.19

  4%|▎         | 1457/41250 [3:31:17<95:38:09,  8.65s/it]
                                                         
{'loss': 0.282, 'grad_norm': 2.2990882396698, 'learning_rate': 3.999704337153261e-05, 'epoch': 0.35}

  4%|▎         | 1457/41250 [3:31:17<95:38:09,  8.65s/it][2025-04-25 11:29:00,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 11:29:00,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.87 | bwd_microstep: 5698.23 | bwd_inner_microstep: 5685.39 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.85
[2025-04-25 11:29:00,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.87 | bwd: 5698.24 | bwd_inner: 5685.39 | bwd_allreduce: 12.81 | step: 18.86

  4%|▎         | 1458/41250 [3:31:26<95:32:08,  8.64s/it]
                                                         
{'loss': 0.2086, 'grad_norm': 1.83919358253479, 'learning_rate': 3.999701630938579e-05, 'epoch': 0.35}

  4%|▎         | 1458/41250 [3:31:26<95:32:08,  8.64s/it][2025-04-25 11:29:09,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:29:09,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5733.47 | bwd_inner_microstep: 5677.41 | bwd_allreduce_microstep: 56.02 | step_microstep: 18.30
[2025-04-25 11:29:09,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5733.48 | bwd_inner: 5677.41 | bwd_allreduce: 56.04 | step: 18.32

  4%|▎         | 1459/41250 [3:31:34<95:36:22,  8.65s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 1.271579384803772, 'learning_rate': 3.99969891239613e-05, 'epoch': 0.35}

  4%|▎         | 1459/41250 [3:31:34<95:36:22,  8.65s/it][2025-04-25 11:29:18,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:29:18,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.69 | bwd_microstep: 5747.59 | bwd_inner_microstep: 5682.23 | bwd_allreduce_microstep: 65.32 | step_microstep: 19.15
[2025-04-25 11:29:18,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.69 | bwd: 5747.60 | bwd_inner: 5682.23 | bwd_allreduce: 65.33 | step: 19.16

  4%|▎         | 1460/41250 [3:31:43<95:42:42,  8.66s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 0.5629228949546814, 'learning_rate': 3.999696181525931e-05, 'epoch': 0.35}

  4%|▎         | 1460/41250 [3:31:43<95:42:42,  8.66s/it][2025-04-25 11:29:26,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.10 | optimizer_step: 1.05
[2025-04-25 11:29:26,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.84 | bwd_microstep: 5693.72 | bwd_inner_microstep: 5680.42 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.76
[2025-04-25 11:29:26,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.84 | bwd: 5693.73 | bwd_inner: 5680.42 | bwd_allreduce: 13.26 | step: 19.76

  4%|▎         | 1461/41250 [3:31:51<95:36:07,  8.65s/it]
                                                         
{'loss': 0.3581, 'grad_norm': 1.9827930927276611, 'learning_rate': 3.999693438327997e-05, 'epoch': 0.35}

  4%|▎         | 1461/41250 [3:31:51<95:36:07,  8.65s/it][2025-04-25 11:29:35,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:29:35,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.81 | bwd_microstep: 5744.38 | bwd_inner_microstep: 5692.76 | bwd_allreduce_microstep: 51.58 | step_microstep: 18.95
[2025-04-25 11:29:35,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.81 | bwd: 5744.39 | bwd_inner: 5692.76 | bwd_allreduce: 51.59 | step: 18.95

  4%|▎         | 1462/41250 [3:32:00<95:40:45,  8.66s/it]
                                                         
{'loss': 0.1275, 'grad_norm': 0.9259446263313293, 'learning_rate': 3.999690682802347e-05, 'epoch': 0.35}

  4%|▎         | 1462/41250 [3:32:00<95:40:45,  8.66s/it][2025-04-25 11:29:44,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.07 | optimizer_step: 0.98
[2025-04-25 11:29:44,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.78 | bwd_microstep: 5779.95 | bwd_inner_microstep: 5633.91 | bwd_allreduce_microstep: 146.00 | step_microstep: 18.95
[2025-04-25 11:29:44,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.78 | bwd: 5779.97 | bwd_inner: 5633.90 | bwd_allreduce: 146.02 | step: 18.95

  4%|▎         | 1463/41250 [3:32:09<95:45:19,  8.66s/it]
                                                         
{'loss': 0.2227, 'grad_norm': 1.3728442192077637, 'learning_rate': 3.999687914948997e-05, 'epoch': 0.35}

  4%|▎         | 1463/41250 [3:32:09<95:45:19,  8.66s/it][2025-04-25 11:29:52,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 11:29:52,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5733.50 | bwd_inner_microstep: 5688.80 | bwd_allreduce_microstep: 44.65 | step_microstep: 19.10
[2025-04-25 11:29:52,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5733.52 | bwd_inner: 5688.80 | bwd_allreduce: 44.67 | step: 19.10

  4%|▎         | 1464/41250 [3:32:18<95:44:58,  8.66s/it]
                                                         
{'loss': 0.248, 'grad_norm': 1.0950376987457275, 'learning_rate': 3.9996851347679645e-05, 'epoch': 0.35}

  4%|▎         | 1464/41250 [3:32:18<95:44:58,  8.66s/it][2025-04-25 11:30:01,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.07 | optimizer_step: 0.95
[2025-04-25 11:30:01,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.92 | bwd_microstep: 5882.45 | bwd_inner_microstep: 5868.84 | bwd_allreduce_microstep: 13.55 | step_microstep: 20.07
[2025-04-25 11:30:01,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.92 | bwd: 5882.46 | bwd_inner: 5868.84 | bwd_allreduce: 13.58 | step: 20.07

  4%|▎         | 1465/41250 [3:32:26<96:32:27,  8.74s/it]
                                                         
{'loss': 0.0943, 'grad_norm': 0.6835848689079285, 'learning_rate': 3.9996823422592665e-05, 'epoch': 0.36}

  4%|▎         | 1465/41250 [3:32:26<96:32:27,  8.74s/it][2025-04-25 11:30:10,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 11:30:10,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5693.43 | bwd_inner_microstep: 5680.10 | bwd_allreduce_microstep: 13.27 | step_microstep: 18.98
[2025-04-25 11:30:10,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5693.44 | bwd_inner: 5680.10 | bwd_allreduce: 13.30 | step: 18.98

  4%|▎         | 1466/41250 [3:32:35<96:08:32,  8.70s/it]
                                                         
{'loss': 0.1813, 'grad_norm': 1.0921329259872437, 'learning_rate': 3.99967953742292e-05, 'epoch': 0.36}

  4%|▎         | 1466/41250 [3:32:35<96:08:32,  8.70s/it][2025-04-25 11:30:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:30:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.72 | bwd_microstep: 5789.37 | bwd_inner_microstep: 5776.64 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.98
[2025-04-25 11:30:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.72 | bwd: 5789.39 | bwd_inner: 5776.64 | bwd_allreduce: 12.70 | step: 18.98

  4%|▎         | 1467/41250 [3:32:44<96:19:39,  8.72s/it]
                                                         
{'loss': 0.1978, 'grad_norm': 1.2799443006515503, 'learning_rate': 3.999676720258942e-05, 'epoch': 0.36}

  4%|▎         | 1467/41250 [3:32:44<96:19:39,  8.72s/it][2025-04-25 11:30:27,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:30:27,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5680.27 | bwd_inner_microstep: 5648.28 | bwd_allreduce_microstep: 31.95 | step_microstep: 18.78
[2025-04-25 11:30:27,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5680.28 | bwd_inner: 5648.28 | bwd_allreduce: 31.96 | step: 18.78

  4%|▎         | 1468/41250 [3:32:52<95:54:27,  8.68s/it]
                                                         
{'loss': 0.1809, 'grad_norm': 1.5309407711029053, 'learning_rate': 3.999673890767352e-05, 'epoch': 0.36}

  4%|▎         | 1468/41250 [3:32:52<95:54:27,  8.68s/it][2025-04-25 11:30:36,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 11:30:36,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.12 | bwd_microstep: 5695.81 | bwd_inner_microstep: 5683.04 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.21
[2025-04-25 11:30:36,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.12 | bwd: 5695.82 | bwd_inner: 5683.04 | bwd_allreduce: 12.74 | step: 19.22

  4%|▎         | 1469/41250 [3:33:01<95:43:59,  8.66s/it]
                                                         
{'loss': 0.2089, 'grad_norm': 1.569126009941101, 'learning_rate': 3.999671048948164e-05, 'epoch': 0.36}

  4%|▎         | 1469/41250 [3:33:01<95:43:59,  8.66s/it][2025-04-25 11:30:44,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 11:30:44,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.61 | bwd_microstep: 5722.86 | bwd_inner_microstep: 5688.60 | bwd_allreduce_microstep: 34.22 | step_microstep: 18.92
[2025-04-25 11:30:44,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.61 | bwd: 5722.88 | bwd_inner: 5688.60 | bwd_allreduce: 34.24 | step: 18.92

  4%|▎         | 1470/41250 [3:33:10<95:40:26,  8.66s/it]
                                                         
{'loss': 0.195, 'grad_norm': 1.9436713457107544, 'learning_rate': 3.999668194801398e-05, 'epoch': 0.36}

  4%|▎         | 1470/41250 [3:33:10<95:40:26,  8.66s/it][2025-04-25 11:30:53,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 11:30:53,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.28 | bwd_microstep: 5691.73 | bwd_inner_microstep: 5678.94 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.68
[2025-04-25 11:30:53,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.28 | bwd: 5691.75 | bwd_inner: 5678.94 | bwd_allreduce: 12.77 | step: 18.68

  4%|▎         | 1471/41250 [3:33:18<95:32:26,  8.65s/it]
                                                         
{'loss': 0.2714, 'grad_norm': 1.7814537286758423, 'learning_rate': 3.9996653283270715e-05, 'epoch': 0.36}

  4%|▎         | 1471/41250 [3:33:18<95:32:26,  8.65s/it][2025-04-25 11:31:02,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:31:02,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.25 | bwd_microstep: 5724.94 | bwd_inner_microstep: 5686.99 | bwd_allreduce_microstep: 37.90 | step_microstep: 18.60
[2025-04-25 11:31:02,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.25 | bwd: 5724.95 | bwd_inner: 5686.99 | bwd_allreduce: 37.92 | step: 18.60

  4%|▎         | 1472/41250 [3:33:27<95:33:50,  8.65s/it]
                                                         
{'loss': 0.1617, 'grad_norm': 1.3717758655548096, 'learning_rate': 3.999662449525201e-05, 'epoch': 0.36}

  4%|▎         | 1472/41250 [3:33:27<95:33:50,  8.65s/it][2025-04-25 11:31:10,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:31:10,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.42 | bwd_microstep: 5692.92 | bwd_inner_microstep: 5680.05 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.86
[2025-04-25 11:31:10,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.42 | bwd: 5692.93 | bwd_inner: 5680.05 | bwd_allreduce: 12.84 | step: 18.87

  4%|▎         | 1473/41250 [3:33:36<95:27:57,  8.64s/it]
                                                         
{'loss': 0.1666, 'grad_norm': 0.7361555099487305, 'learning_rate': 3.999659558395805e-05, 'epoch': 0.36}

  4%|▎         | 1473/41250 [3:33:36<95:27:57,  8.64s/it][2025-04-25 11:31:19,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:31:19,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.14 | bwd_microstep: 5984.23 | bwd_inner_microstep: 5692.79 | bwd_allreduce_microstep: 291.39 | step_microstep: 18.77
[2025-04-25 11:31:19,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.14 | bwd: 5984.25 | bwd_inner: 5692.79 | bwd_allreduce: 291.41 | step: 18.77

  4%|▎         | 1474/41250 [3:33:44<96:21:52,  8.72s/it]
                                                         
{'loss': 0.2251, 'grad_norm': 1.664690375328064, 'learning_rate': 3.999656654938901e-05, 'epoch': 0.36}

  4%|▎         | 1474/41250 [3:33:44<96:21:52,  8.72s/it][2025-04-25 11:31:28,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.05 | optimizer_step: 1.18
[2025-04-25 11:31:28,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.38 | bwd_microstep: 5728.18 | bwd_inner_microstep: 5642.32 | bwd_allreduce_microstep: 85.81 | step_microstep: 19.20
[2025-04-25 11:31:28,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.38 | bwd: 5728.19 | bwd_inner: 5642.32 | bwd_allreduce: 85.83 | step: 19.20

  4%|▎         | 1475/41250 [3:33:53<96:06:54,  8.70s/it]
                                                         
{'loss': 0.2399, 'grad_norm': 1.8294272422790527, 'learning_rate': 3.999653739154506e-05, 'epoch': 0.36}

  4%|▎         | 1475/41250 [3:33:53<96:06:54,  8.70s/it][2025-04-25 11:31:36,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 11:31:36,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.53 | bwd_microstep: 5696.39 | bwd_inner_microstep: 5633.37 | bwd_allreduce_microstep: 62.96 | step_microstep: 19.68
[2025-04-25 11:31:36,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.53 | bwd: 5696.40 | bwd_inner: 5633.37 | bwd_allreduce: 62.98 | step: 19.68

  4%|▎         | 1476/41250 [3:34:02<95:47:27,  8.67s/it]
                                                         
{'loss': 0.2258, 'grad_norm': 1.6948145627975464, 'learning_rate': 3.99965081104264e-05, 'epoch': 0.36}

  4%|▎         | 1476/41250 [3:34:02<95:47:27,  8.67s/it][2025-04-25 11:31:45,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-25 11:31:45,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.41 | bwd_microstep: 5728.41 | bwd_inner_microstep: 5685.07 | bwd_allreduce_microstep: 43.29 | step_microstep: 19.11
[2025-04-25 11:31:45,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5728.42 | bwd_inner: 5685.07 | bwd_allreduce: 43.31 | step: 19.11

  4%|▎         | 1477/41250 [3:34:10<95:44:23,  8.67s/it]
                                                         
{'loss': 0.3746, 'grad_norm': 2.1340646743774414, 'learning_rate': 3.99964787060332e-05, 'epoch': 0.36}

  4%|▎         | 1477/41250 [3:34:10<95:44:23,  8.67s/it][2025-04-25 11:31:54,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.16 | optimizer_step: 1.10
[2025-04-25 11:31:54,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.12 | bwd_microstep: 5739.56 | bwd_inner_microstep: 5637.42 | bwd_allreduce_microstep: 102.08 | step_microstep: 19.82
[2025-04-25 11:31:54,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.12 | bwd: 5739.57 | bwd_inner: 5637.42 | bwd_allreduce: 102.11 | step: 19.82

  4%|▎         | 1478/41250 [3:34:19<95:40:11,  8.66s/it]
                                                         
{'loss': 0.3442, 'grad_norm': 1.578275203704834, 'learning_rate': 3.9996449178365646e-05, 'epoch': 0.36}

  4%|▎         | 1478/41250 [3:34:19<95:40:11,  8.66s/it][2025-04-25 11:32:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 11:32:03,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.10 | bwd_microstep: 5883.41 | bwd_inner_microstep: 5676.90 | bwd_allreduce_microstep: 206.47 | step_microstep: 18.74
[2025-04-25 11:32:03,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.10 | bwd: 5883.43 | bwd_inner: 5676.90 | bwd_allreduce: 206.49 | step: 18.74

  4%|▎         | 1479/41250 [3:34:28<96:10:45,  8.71s/it]
                                                         
{'loss': 0.2363, 'grad_norm': 2.3024656772613525, 'learning_rate': 3.999641952742392e-05, 'epoch': 0.36}

  4%|▎         | 1479/41250 [3:34:28<96:10:45,  8.71s/it][2025-04-25 11:32:11,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 11:32:11,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.71 | bwd_microstep: 5698.08 | bwd_inner_microstep: 5685.35 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.40
[2025-04-25 11:32:11,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.71 | bwd: 5698.09 | bwd_inner: 5685.35 | bwd_allreduce: 12.70 | step: 19.40

  4%|▎         | 1480/41250 [3:34:36<95:54:28,  8.68s/it]
                                                         
{'loss': 0.2529, 'grad_norm': 1.6134721040725708, 'learning_rate': 3.9996389753208184e-05, 'epoch': 0.36}

  4%|▎         | 1480/41250 [3:34:36<95:54:28,  8.68s/it][2025-04-25 11:32:20,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.20 | optimizer_step: 0.92
[2025-04-25 11:32:20,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.10 | bwd_microstep: 5742.83 | bwd_inner_microstep: 5677.68 | bwd_allreduce_microstep: 65.10 | step_microstep: 19.17
[2025-04-25 11:32:20,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.11 | bwd: 5742.84 | bwd_inner: 5677.68 | bwd_allreduce: 65.12 | step: 19.17

  4%|▎         | 1481/41250 [3:34:45<95:50:27,  8.68s/it]
                                                         
{'loss': 0.2683, 'grad_norm': 1.5966987609863281, 'learning_rate': 3.999635985571865e-05, 'epoch': 0.36}

  4%|▎         | 1481/41250 [3:34:45<95:50:27,  8.68s/it][2025-04-25 11:32:28,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 11:32:28,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.87 | bwd_microstep: 5744.61 | bwd_inner_microstep: 5638.46 | bwd_allreduce_microstep: 106.10 | step_microstep: 18.62
[2025-04-25 11:32:28,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.87 | bwd: 5744.62 | bwd_inner: 5638.46 | bwd_allreduce: 106.12 | step: 18.63

  4%|▎         | 1482/41250 [3:34:54<95:45:18,  8.67s/it]
                                                         
{'loss': 0.3328, 'grad_norm': 2.5869252681732178, 'learning_rate': 3.9996329834955485e-05, 'epoch': 0.36}

  4%|▎         | 1482/41250 [3:34:54<95:45:18,  8.67s/it][2025-04-25 11:32:37,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:32:37,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.89 | bwd_microstep: 5725.13 | bwd_inner_microstep: 5670.88 | bwd_allreduce_microstep: 54.21 | step_microstep: 18.85
[2025-04-25 11:32:37,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.89 | bwd: 5725.15 | bwd_inner: 5670.88 | bwd_allreduce: 54.23 | step: 18.86

  4%|▎         | 1483/41250 [3:35:02<95:40:33,  8.66s/it]
                                                         
{'loss': 0.3293, 'grad_norm': 2.187288284301758, 'learning_rate': 3.9996299690918886e-05, 'epoch': 0.36}

  4%|▎         | 1483/41250 [3:35:02<95:40:33,  8.66s/it][2025-04-25 11:32:46,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:32:46,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.93 | bwd_microstep: 5787.09 | bwd_inner_microstep: 5641.04 | bwd_allreduce_microstep: 146.00 | step_microstep: 18.82
[2025-04-25 11:32:46,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.93 | bwd: 5787.10 | bwd_inner: 5641.04 | bwd_allreduce: 146.02 | step: 18.82

  4%|▎         | 1484/41250 [3:35:11<95:45:42,  8.67s/it]
                                                         
{'loss': 0.1729, 'grad_norm': 0.9630893468856812, 'learning_rate': 3.999626942360903e-05, 'epoch': 0.36}

  4%|▎         | 1484/41250 [3:35:11<95:45:42,  8.67s/it][2025-04-25 11:32:54,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:32:54,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.80 | bwd_microstep: 5756.36 | bwd_inner_microstep: 5637.81 | bwd_allreduce_microstep: 118.51 | step_microstep: 18.68
[2025-04-25 11:32:54,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.80 | bwd: 5756.38 | bwd_inner: 5637.80 | bwd_allreduce: 118.53 | step: 18.68

  4%|▎         | 1485/41250 [3:35:20<95:43:29,  8.67s/it]
                                                         
{'loss': 0.2047, 'grad_norm': 2.000322103500366, 'learning_rate': 3.99962390330261e-05, 'epoch': 0.36}

  4%|▎         | 1485/41250 [3:35:20<95:43:29,  8.67s/it][2025-04-25 11:33:03,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 11:33:03,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.01 | bwd_microstep: 5721.94 | bwd_inner_microstep: 5632.86 | bwd_allreduce_microstep: 89.04 | step_microstep: 19.61
[2025-04-25 11:33:03,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.01 | bwd: 5721.95 | bwd_inner: 5632.86 | bwd_allreduce: 89.05 | step: 19.61

  4%|▎         | 1486/41250 [3:35:28<95:34:31,  8.65s/it]
                                                         
{'loss': 0.1398, 'grad_norm': 1.0892010927200317, 'learning_rate': 3.9996208519170295e-05, 'epoch': 0.36}

  4%|▎         | 1486/41250 [3:35:28<95:34:31,  8.65s/it][2025-04-25 11:33:12,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 11:33:12,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.29 | bwd_microstep: 5706.99 | bwd_inner_microstep: 5636.60 | bwd_allreduce_microstep: 70.34 | step_microstep: 18.87
[2025-04-25 11:33:12,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.29 | bwd: 5707.00 | bwd_inner: 5636.60 | bwd_allreduce: 70.36 | step: 18.87

  4%|▎         | 1487/41250 [3:35:37<95:25:31,  8.64s/it]
                                                         
{'loss': 0.3106, 'grad_norm': 1.6558690071105957, 'learning_rate': 3.9996177882041793e-05, 'epoch': 0.36}

  4%|▎         | 1487/41250 [3:35:37<95:25:31,  8.64s/it][2025-04-25 11:33:20,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 11:33:20,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5754.00 | bwd_inner_microstep: 5685.09 | bwd_allreduce_microstep: 68.87 | step_microstep: 18.56
[2025-04-25 11:33:20,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5754.02 | bwd_inner: 5685.09 | bwd_allreduce: 68.89 | step: 18.56

  4%|▎         | 1488/41250 [3:35:46<95:34:28,  8.65s/it]
                                                         
{'loss': 0.135, 'grad_norm': 1.0271251201629639, 'learning_rate': 3.999614712164079e-05, 'epoch': 0.36}

  4%|▎         | 1488/41250 [3:35:46<95:34:28,  8.65s/it][2025-04-25 11:33:29,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:33:29,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.42 | bwd_microstep: 5761.77 | bwd_inner_microstep: 5698.34 | bwd_allreduce_microstep: 63.38 | step_microstep: 18.70
[2025-04-25 11:33:29,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.42 | bwd: 5761.78 | bwd_inner: 5698.34 | bwd_allreduce: 63.39 | step: 18.70

  4%|▎         | 1489/41250 [3:35:54<95:42:52,  8.67s/it]
                                                         
{'loss': 0.2738, 'grad_norm': 1.2910022735595703, 'learning_rate': 3.9996116237967474e-05, 'epoch': 0.36}

  4%|▎         | 1489/41250 [3:35:54<95:42:52,  8.67s/it][2025-04-25 11:33:38,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 11:33:38,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.90 | bwd_microstep: 5803.26 | bwd_inner_microstep: 5647.45 | bwd_allreduce_microstep: 155.76 | step_microstep: 19.24
[2025-04-25 11:33:38,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.90 | bwd: 5803.27 | bwd_inner: 5647.45 | bwd_allreduce: 155.78 | step: 19.24

  4%|▎         | 1490/41250 [3:36:03<95:51:04,  8.68s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.2232944965362549, 'learning_rate': 3.9996085231022037e-05, 'epoch': 0.36}

  4%|▎         | 1490/41250 [3:36:03<95:51:04,  8.68s/it][2025-04-25 11:33:46,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 11:33:46,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.50 | bwd_microstep: 5693.68 | bwd_inner_microstep: 5680.94 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.53
[2025-04-25 11:33:46,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.50 | bwd: 5693.69 | bwd_inner: 5680.94 | bwd_allreduce: 12.71 | step: 18.53

  4%|▎         | 1491/41250 [3:36:12<95:41:14,  8.66s/it]
                                                         
{'loss': 0.1554, 'grad_norm': 1.2094262838363647, 'learning_rate': 3.999605410080467e-05, 'epoch': 0.36}

  4%|▎         | 1491/41250 [3:36:12<95:41:14,  8.66s/it][2025-04-25 11:33:55,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 11:33:55,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.88 | bwd_microstep: 5711.01 | bwd_inner_microstep: 5659.52 | bwd_allreduce_microstep: 51.45 | step_microstep: 18.66
[2025-04-25 11:33:55,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.88 | bwd: 5711.03 | bwd_inner: 5659.52 | bwd_allreduce: 51.46 | step: 18.67

  4%|▎         | 1492/41250 [3:36:20<95:33:04,  8.65s/it]
                                                         
{'loss': 0.1242, 'grad_norm': 1.0161741971969604, 'learning_rate': 3.999602284731555e-05, 'epoch': 0.36}

  4%|▎         | 1492/41250 [3:36:20<95:33:04,  8.65s/it][2025-04-25 11:34:04,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 11:34:04,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.73 | bwd_microstep: 5717.47 | bwd_inner_microstep: 5704.75 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.70
[2025-04-25 11:34:04,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.73 | bwd: 5717.48 | bwd_inner: 5704.75 | bwd_allreduce: 12.69 | step: 18.70

  4%|▎         | 1493/41250 [3:36:29<95:32:39,  8.65s/it]
                                                         
{'loss': 0.2787, 'grad_norm': 1.5358771085739136, 'learning_rate': 3.9995991470554886e-05, 'epoch': 0.36}

  4%|▎         | 1493/41250 [3:36:29<95:32:39,  8.65s/it][2025-04-25 11:34:12,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:34:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.08 | bwd_microstep: 5776.71 | bwd_inner_microstep: 5763.97 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.84
[2025-04-25 11:34:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.08 | bwd: 5776.72 | bwd_inner: 5763.97 | bwd_allreduce: 12.71 | step: 18.84

  4%|▎         | 1494/41250 [3:36:38<95:52:31,  8.68s/it]
                                                         
{'loss': 0.198, 'grad_norm': 1.4143602848052979, 'learning_rate': 3.999595997052287e-05, 'epoch': 0.36}

  4%|▎         | 1494/41250 [3:36:38<95:52:31,  8.68s/it][2025-04-25 11:34:21,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 11:34:21,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.61 | bwd_microstep: 5713.48 | bwd_inner_microstep: 5668.35 | bwd_allreduce_microstep: 45.08 | step_microstep: 18.68
[2025-04-25 11:34:21,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.61 | bwd: 5713.49 | bwd_inner: 5668.35 | bwd_allreduce: 45.10 | step: 18.68

  4%|▎         | 1495/41250 [3:36:46<95:42:03,  8.67s/it]
                                                         
{'loss': 0.2378, 'grad_norm': 1.2001817226409912, 'learning_rate': 3.99959283472197e-05, 'epoch': 0.36}

  4%|▎         | 1495/41250 [3:36:46<95:42:03,  8.67s/it][2025-04-25 11:34:30,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:34:30,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5711.58 | bwd_inner_microstep: 5649.39 | bwd_allreduce_microstep: 62.14 | step_microstep: 18.91
[2025-04-25 11:34:30,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5711.60 | bwd_inner: 5649.39 | bwd_allreduce: 62.16 | step: 18.91

  4%|▎         | 1496/41250 [3:36:55<95:33:11,  8.65s/it]
                                                         
{'loss': 0.1642, 'grad_norm': 0.745045006275177, 'learning_rate': 3.999589660064556e-05, 'epoch': 0.36}

  4%|▎         | 1496/41250 [3:36:55<95:33:11,  8.65s/it][2025-04-25 11:34:38,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-25 11:34:38,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5720.27 | bwd_inner_microstep: 5654.93 | bwd_allreduce_microstep: 65.29 | step_microstep: 19.47
[2025-04-25 11:34:38,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5720.29 | bwd_inner: 5654.93 | bwd_allreduce: 65.31 | step: 19.47

  4%|▎         | 1497/41250 [3:37:04<95:28:57,  8.65s/it]
                                                         
{'loss': 0.0971, 'grad_norm': 1.1850236654281616, 'learning_rate': 3.999586473080065e-05, 'epoch': 0.36}

  4%|▎         | 1497/41250 [3:37:04<95:28:57,  8.65s/it][2025-04-25 11:34:47,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:34:47,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.76 | bwd_microstep: 5722.41 | bwd_inner_microstep: 5666.93 | bwd_allreduce_microstep: 55.44 | step_microstep: 18.09
[2025-04-25 11:34:47,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.76 | bwd: 5722.42 | bwd_inner: 5666.93 | bwd_allreduce: 55.45 | step: 18.09

  4%|▎         | 1498/41250 [3:37:12<95:26:04,  8.64s/it]
                                                         
{'loss': 0.2399, 'grad_norm': 1.903902530670166, 'learning_rate': 3.999583273768517e-05, 'epoch': 0.36}

  4%|▎         | 1498/41250 [3:37:12<95:26:04,  8.64s/it][2025-04-25 11:34:56,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 11:34:56,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.49 | bwd_microstep: 5775.51 | bwd_inner_microstep: 5701.82 | bwd_allreduce_microstep: 73.64 | step_microstep: 18.91
[2025-04-25 11:34:56,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.49 | bwd: 5775.52 | bwd_inner: 5701.82 | bwd_allreduce: 73.66 | step: 18.91

  4%|▎         | 1499/41250 [3:37:21<95:39:05,  8.66s/it]
                                                         
{'loss': 0.0747, 'grad_norm': 0.8907859325408936, 'learning_rate': 3.9995800621299305e-05, 'epoch': 0.36}

  4%|▎         | 1499/41250 [3:37:21<95:39:05,  8.66s/it][2025-04-25 11:35:04,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:35:04,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.57 | bwd_microstep: 5801.22 | bwd_inner_microstep: 5788.59 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.75
[2025-04-25 11:35:04,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.57 | bwd: 5801.23 | bwd_inner: 5788.59 | bwd_allreduce: 12.60 | step: 18.75

  4%|▎         | 1500/41250 [3:37:30<96:02:38,  8.70s/it]
                                                         
{'loss': 0.2068, 'grad_norm': 1.1687324047088623, 'learning_rate': 3.999576838164327e-05, 'epoch': 0.36}

  4%|▎         | 1500/41250 [3:37:30<96:02:38,  8.70s/it][2025-04-25 11:35:13,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-25 11:35:13,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.41 | bwd_microstep: 5764.63 | bwd_inner_microstep: 5674.64 | bwd_allreduce_microstep: 89.95 | step_microstep: 18.47
[2025-04-25 11:35:13,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.42 | bwd: 5764.64 | bwd_inner: 5674.63 | bwd_allreduce: 89.97 | step: 18.47

  4%|▎         | 1501/41250 [3:37:38<95:59:52,  8.69s/it]
                                                         
{'loss': 0.0552, 'grad_norm': 0.5439493060112, 'learning_rate': 3.999573601871726e-05, 'epoch': 0.36}

  4%|▎         | 1501/41250 [3:37:38<95:59:52,  8.69s/it][2025-04-25 11:35:22,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:35:22,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.89 | bwd_microstep: 5706.54 | bwd_inner_microstep: 5664.99 | bwd_allreduce_microstep: 41.51 | step_microstep: 18.39
[2025-04-25 11:35:22,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.89 | bwd: 5706.56 | bwd_inner: 5664.99 | bwd_allreduce: 41.52 | step: 18.39

  4%|▎         | 1502/41250 [3:37:47<95:45:24,  8.67s/it]
                                                         
{'loss': 0.3895, 'grad_norm': 2.2883167266845703, 'learning_rate': 3.9995703532521464e-05, 'epoch': 0.36}

  4%|▎         | 1502/41250 [3:37:47<95:45:24,  8.67s/it][2025-04-25 11:35:30,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-25 11:35:30,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5748.21 | bwd_inner_microstep: 5705.41 | bwd_allreduce_microstep: 42.76 | step_microstep: 18.75
[2025-04-25 11:35:30,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5748.23 | bwd_inner: 5705.41 | bwd_allreduce: 42.78 | step: 18.76

  4%|▎         | 1503/41250 [3:37:56<95:46:57,  8.68s/it]
                                                         
{'loss': 0.2762, 'grad_norm': 1.8810715675354004, 'learning_rate': 3.9995670923056084e-05, 'epoch': 0.36}

  4%|▎         | 1503/41250 [3:37:56<95:46:57,  8.68s/it][2025-04-25 11:35:39,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 11:35:39,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.65 | bwd_microstep: 5810.32 | bwd_inner_microstep: 5651.06 | bwd_allreduce_microstep: 159.21 | step_microstep: 18.47
[2025-04-25 11:35:39,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.65 | bwd: 5810.33 | bwd_inner: 5651.05 | bwd_allreduce: 159.23 | step: 18.47

  4%|▎         | 1504/41250 [3:38:04<95:55:55,  8.69s/it]
                                                         
{'loss': 0.1039, 'grad_norm': 0.9476083517074585, 'learning_rate': 3.9995638190321335e-05, 'epoch': 0.36}

  4%|▎         | 1504/41250 [3:38:04<95:55:55,  8.69s/it][2025-04-25 11:35:48,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 11:35:48,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5705.25 | bwd_inner_microstep: 5690.57 | bwd_allreduce_microstep: 14.64 | step_microstep: 19.09
[2025-04-25 11:35:48,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5705.26 | bwd_inner: 5690.57 | bwd_allreduce: 14.65 | step: 19.09

  4%|▎         | 1505/41250 [3:38:13<95:45:07,  8.67s/it]
                                                         
{'loss': 0.1407, 'grad_norm': 1.123234510421753, 'learning_rate': 3.9995605334317407e-05, 'epoch': 0.36}

  4%|▎         | 1505/41250 [3:38:13<95:45:07,  8.67s/it][2025-04-25 11:35:56,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 11:35:56,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5699.89 | bwd_allreduce_microstep: 65.20 | step_microstep: 19.48
[2025-04-25 11:35:56,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5765.15 | bwd_inner: 5699.89 | bwd_allreduce: 65.22 | step: 19.48

  4%|▎         | 1506/41250 [3:38:22<95:50:39,  8.68s/it]
                                                         
{'loss': 0.0987, 'grad_norm': 1.2549697160720825, 'learning_rate': 3.99955723550445e-05, 'epoch': 0.37}

  4%|▎         | 1506/41250 [3:38:22<95:50:39,  8.68s/it][2025-04-25 11:36:05,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 11:36:05,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.88 | bwd_microstep: 5741.98 | bwd_inner_microstep: 5661.56 | bwd_allreduce_microstep: 80.37 | step_microstep: 18.94
[2025-04-25 11:36:05,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.88 | bwd: 5742.00 | bwd_inner: 5661.56 | bwd_allreduce: 80.39 | step: 18.95

  4%|▎         | 1507/41250 [3:38:30<95:47:35,  8.68s/it]
                                                         
{'loss': 0.1927, 'grad_norm': 0.9928983449935913, 'learning_rate': 3.9995539252502826e-05, 'epoch': 0.37}

  4%|▎         | 1507/41250 [3:38:30<95:47:35,  8.68s/it][2025-04-25 11:36:14,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.24
[2025-04-25 11:36:14,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.32 | bwd_microstep: 5885.36 | bwd_inner_microstep: 5656.56 | bwd_allreduce_microstep: 228.74 | step_microstep: 19.77
[2025-04-25 11:36:14,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.32 | bwd: 5885.38 | bwd_inner: 5656.56 | bwd_allreduce: 228.77 | step: 19.77

  4%|▎         | 1508/41250 [3:38:39<96:11:55,  8.71s/it]
                                                         
{'loss': 0.2113, 'grad_norm': 1.1427925825119019, 'learning_rate': 3.9995506026692586e-05, 'epoch': 0.37}

  4%|▎         | 1508/41250 [3:38:39<96:11:55,  8.71s/it][2025-04-25 11:36:23,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 11:36:23,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5695.08 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 32.96 | step_microstep: 19.17
[2025-04-25 11:36:23,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5695.10 | bwd_inner: 5662.08 | bwd_allreduce: 32.98 | step: 19.17

  4%|▎         | 1509/41250 [3:38:48<95:52:10,  8.68s/it]
                                                         
{'loss': 0.0786, 'grad_norm': 0.6380332708358765, 'learning_rate': 3.999547267761398e-05, 'epoch': 0.37}

  4%|▎         | 1509/41250 [3:38:48<95:52:10,  8.68s/it][2025-04-25 11:36:31,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:36:31,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.05 | bwd_microstep: 5799.86 | bwd_inner_microstep: 5786.98 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.56
[2025-04-25 11:36:31,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.05 | bwd: 5799.87 | bwd_inner: 5786.98 | bwd_allreduce: 12.85 | step: 18.56

  4%|▎         | 1510/41250 [3:38:57<96:10:05,  8.71s/it]
                                                         
{'loss': 0.109, 'grad_norm': 1.0931955575942993, 'learning_rate': 3.999543920526723e-05, 'epoch': 0.37}

  4%|▎         | 1510/41250 [3:38:57<96:10:05,  8.71s/it][2025-04-25 11:36:40,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:36:40,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5707.32 | bwd_inner_microstep: 5651.91 | bwd_allreduce_microstep: 55.36 | step_microstep: 18.48
[2025-04-25 11:36:40,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.82 | bwd: 5707.33 | bwd_inner: 5651.91 | bwd_allreduce: 55.38 | step: 18.48

  4%|▎         | 1511/41250 [3:39:05<95:52:02,  8.68s/it]
                                                         
{'loss': 0.0621, 'grad_norm': 0.7406649589538574, 'learning_rate': 3.999540560965251e-05, 'epoch': 0.37}

  4%|▎         | 1511/41250 [3:39:05<95:52:02,  8.68s/it][2025-04-25 11:36:49,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:36:49,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5771.36 | bwd_inner_microstep: 5654.10 | bwd_allreduce_microstep: 117.22 | step_microstep: 18.31
[2025-04-25 11:36:49,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5771.37 | bwd_inner: 5654.10 | bwd_allreduce: 117.23 | step: 18.32

  4%|▎         | 1512/41250 [3:39:14<95:51:06,  8.68s/it]
                                                         
{'loss': 0.1876, 'grad_norm': 2.166109323501587, 'learning_rate': 3.999537189077006e-05, 'epoch': 0.37}

  4%|▎         | 1512/41250 [3:39:14<95:51:06,  8.68s/it][2025-04-25 11:36:57,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 1.10
[2025-04-25 11:36:57,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.17 | bwd_microstep: 5723.01 | bwd_inner_microstep: 5686.42 | bwd_allreduce_microstep: 36.55 | step_microstep: 18.67
[2025-04-25 11:36:57,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.17 | bwd: 5723.02 | bwd_inner: 5686.42 | bwd_allreduce: 36.57 | step: 18.67

  4%|▎         | 1513/41250 [3:39:23<95:46:33,  8.68s/it]
                                                         
{'loss': 0.0338, 'grad_norm': 0.5121782422065735, 'learning_rate': 3.999533804862008e-05, 'epoch': 0.37}

  4%|▎         | 1513/41250 [3:39:23<95:46:33,  8.68s/it][2025-04-25 11:37:06,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:37:06,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5718.34 | bwd_inner_microstep: 5705.64 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.35
[2025-04-25 11:37:06,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5718.35 | bwd_inner: 5705.64 | bwd_allreduce: 12.67 | step: 18.35

  4%|▎         | 1514/41250 [3:39:31<95:41:36,  8.67s/it]
                                                         
{'loss': 0.2214, 'grad_norm': 1.7894107103347778, 'learning_rate': 3.999530408320276e-05, 'epoch': 0.37}

  4%|▎         | 1514/41250 [3:39:31<95:41:36,  8.67s/it][2025-04-25 11:37:15,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.23 | optimizer_step: 0.93
[2025-04-25 11:37:15,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.09 | bwd_microstep: 5819.74 | bwd_inner_microstep: 5699.88 | bwd_allreduce_microstep: 119.81 | step_microstep: 19.66
[2025-04-25 11:37:15,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.09 | bwd: 5819.75 | bwd_inner: 5699.88 | bwd_allreduce: 119.83 | step: 19.66

  4%|▎         | 1515/41250 [3:39:40<96:01:35,  8.70s/it]
                                                         
{'loss': 0.1755, 'grad_norm': 1.2402101755142212, 'learning_rate': 3.999526999451833e-05, 'epoch': 0.37}

  4%|▎         | 1515/41250 [3:39:40<96:01:35,  8.70s/it][2025-04-25 11:37:23,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 11:37:23,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.97 | bwd_microstep: 5743.20 | bwd_inner_microstep: 5692.74 | bwd_allreduce_microstep: 50.41 | step_microstep: 18.99
[2025-04-25 11:37:23,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.97 | bwd: 5743.22 | bwd_inner: 5692.74 | bwd_allreduce: 50.43 | step: 18.99

  4%|▎         | 1516/41250 [3:39:49<95:57:36,  8.69s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.1191000938415527, 'learning_rate': 3.9995235782567e-05, 'epoch': 0.37}

  4%|▎         | 1516/41250 [3:39:49<95:57:36,  8.69s/it][2025-04-25 11:37:32,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 11:37:32,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.93 | bwd_microstep: 5699.17 | bwd_inner_microstep: 5656.21 | bwd_allreduce_microstep: 42.92 | step_microstep: 19.05
[2025-04-25 11:37:32,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.93 | bwd: 5699.19 | bwd_inner: 5656.21 | bwd_allreduce: 42.93 | step: 19.05

  4%|▎         | 1517/41250 [3:39:57<95:39:48,  8.67s/it]
                                                         
{'loss': 0.1044, 'grad_norm': 1.3449269533157349, 'learning_rate': 3.999520144734897e-05, 'epoch': 0.37}

  4%|▎         | 1517/41250 [3:39:57<95:39:48,  8.67s/it][2025-04-25 11:37:41,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 11:37:41,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.79 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5650.00 | bwd_allreduce_microstep: 56.74 | step_microstep: 19.19
[2025-04-25 11:37:41,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.79 | bwd: 5706.79 | bwd_inner: 5650.00 | bwd_allreduce: 56.76 | step: 19.19

  4%|▎         | 1518/41250 [3:40:06<95:29:40,  8.65s/it]
                                                         
{'loss': 0.262, 'grad_norm': 1.1215733289718628, 'learning_rate': 3.9995166988864454e-05, 'epoch': 0.37}

  4%|▎         | 1518/41250 [3:40:06<95:29:40,  8.65s/it][2025-04-25 11:37:49,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 11:37:49,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.96 | bwd_microstep: 5697.84 | bwd_inner_microstep: 5685.09 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.22
[2025-04-25 11:37:49,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.96 | bwd: 5697.86 | bwd_inner: 5685.09 | bwd_allreduce: 12.73 | step: 19.22

  4%|▎         | 1519/41250 [3:40:15<95:24:01,  8.64s/it]
                                                         
{'loss': 0.0988, 'grad_norm': 1.9441617727279663, 'learning_rate': 3.999513240711367e-05, 'epoch': 0.37}

  4%|▎         | 1519/41250 [3:40:15<95:24:01,  8.64s/it][2025-04-25 11:37:58,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:37:58,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.35 | bwd_microstep: 5724.73 | bwd_inner_microstep: 5703.54 | bwd_allreduce_microstep: 21.15 | step_microstep: 18.76
[2025-04-25 11:37:58,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.35 | bwd: 5724.75 | bwd_inner: 5703.54 | bwd_allreduce: 21.17 | step: 18.76

  4%|▎         | 1520/41250 [3:40:23<95:26:47,  8.65s/it]
                                                         
{'loss': 0.1711, 'grad_norm': 1.3352266550064087, 'learning_rate': 3.999509770209682e-05, 'epoch': 0.37}

  4%|▎         | 1520/41250 [3:40:23<95:26:47,  8.65s/it][2025-04-25 11:38:07,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:38:07,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.17 | bwd_microstep: 5720.23 | bwd_inner_microstep: 5689.89 | bwd_allreduce_microstep: 30.29 | step_microstep: 18.98
[2025-04-25 11:38:07,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.17 | bwd: 5720.24 | bwd_inner: 5689.89 | bwd_allreduce: 30.31 | step: 18.98

  4%|▎         | 1521/41250 [3:40:32<95:28:01,  8.65s/it]
                                                         
{'loss': 0.2119, 'grad_norm': 2.4776594638824463, 'learning_rate': 3.999506287381413e-05, 'epoch': 0.37}

  4%|▎         | 1521/41250 [3:40:32<95:28:01,  8.65s/it][2025-04-25 11:38:15,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 11:38:15,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.44 | bwd_microstep: 5729.39 | bwd_inner_microstep: 5648.92 | bwd_allreduce_microstep: 80.42 | step_microstep: 18.73
[2025-04-25 11:38:15,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.44 | bwd: 5729.40 | bwd_inner: 5648.92 | bwd_allreduce: 80.44 | step: 18.74

  4%|▎         | 1522/41250 [3:40:41<95:25:47,  8.65s/it]
                                                         
{'loss': 0.2288, 'grad_norm': 2.568678617477417, 'learning_rate': 3.999502792226581e-05, 'epoch': 0.37}

  4%|▎         | 1522/41250 [3:40:41<95:25:47,  8.65s/it][2025-04-25 11:38:24,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 11:38:24,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.34 | bwd_microstep: 5702.28 | bwd_inner_microstep: 5689.54 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.32
[2025-04-25 11:38:24,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.34 | bwd: 5702.29 | bwd_inner: 5689.54 | bwd_allreduce: 12.71 | step: 19.32

  4%|▎         | 1523/41250 [3:40:49<95:23:14,  8.64s/it]
                                                         
{'loss': 0.1451, 'grad_norm': 1.7233197689056396, 'learning_rate': 3.999499284745208e-05, 'epoch': 0.37}

  4%|▎         | 1523/41250 [3:40:49<95:23:14,  8.64s/it][2025-04-25 11:38:32,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.21 | optimizer_step: 0.93
[2025-04-25 11:38:32,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5724.95 | bwd_inner_microstep: 5692.68 | bwd_allreduce_microstep: 32.23 | step_microstep: 19.52
[2025-04-25 11:38:32,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5724.97 | bwd_inner: 5692.68 | bwd_allreduce: 32.25 | step: 19.52

  4%|▎         | 1524/41250 [3:40:58<95:25:40,  8.65s/it]
                                                         
{'loss': 0.4876, 'grad_norm': 3.7893364429473877, 'learning_rate': 3.999495764937314e-05, 'epoch': 0.37}

  4%|▎         | 1524/41250 [3:40:58<95:25:40,  8.65s/it][2025-04-25 11:38:41,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 11:38:41,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.34 | bwd_microstep: 5766.26 | bwd_inner_microstep: 5637.97 | bwd_allreduce_microstep: 128.25 | step_microstep: 19.25
[2025-04-25 11:38:41,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.34 | bwd: 5766.28 | bwd_inner: 5637.97 | bwd_allreduce: 128.27 | step: 19.26

  4%|▎         | 1525/41250 [3:41:06<95:29:56,  8.65s/it]
                                                         
{'loss': 0.1687, 'grad_norm': 2.084993362426758, 'learning_rate': 3.999492232802923e-05, 'epoch': 0.37}

  4%|▎         | 1525/41250 [3:41:06<95:29:56,  8.65s/it][2025-04-25 11:38:50,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 11:38:50,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.98 | bwd_microstep: 5730.56 | bwd_inner_microstep: 5653.65 | bwd_allreduce_microstep: 76.86 | step_microstep: 18.51
[2025-04-25 11:38:50,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.98 | bwd: 5730.57 | bwd_inner: 5653.65 | bwd_allreduce: 76.88 | step: 18.51

  4%|▎         | 1526/41250 [3:41:15<95:26:38,  8.65s/it]
                                                         
{'loss': 0.054, 'grad_norm': 0.754253625869751, 'learning_rate': 3.9994886883420555e-05, 'epoch': 0.37}

  4%|▎         | 1526/41250 [3:41:15<95:26:38,  8.65s/it][2025-04-25 11:38:58,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:38:58,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.19 | bwd_microstep: 5688.54 | bwd_inner_microstep: 5647.31 | bwd_allreduce_microstep: 41.19 | step_microstep: 18.54
[2025-04-25 11:38:58,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.19 | bwd: 5688.56 | bwd_inner: 5647.31 | bwd_allreduce: 41.21 | step: 18.55

  4%|▎         | 1527/41250 [3:41:24<95:16:36,  8.63s/it]
                                                         
{'loss': 0.2389, 'grad_norm': 3.2456750869750977, 'learning_rate': 3.999485131554733e-05, 'epoch': 0.37}

  4%|▎         | 1527/41250 [3:41:24<95:16:36,  8.63s/it][2025-04-25 11:39:07,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:39:07,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.86 | bwd_microstep: 5698.67 | bwd_inner_microstep: 5685.99 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.04
[2025-04-25 11:39:07,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.86 | bwd: 5698.69 | bwd_inner: 5685.99 | bwd_allreduce: 12.65 | step: 19.04

  4%|▎         | 1528/41250 [3:41:32<95:14:54,  8.63s/it]
                                                         
{'loss': 0.0994, 'grad_norm': 1.0038203001022339, 'learning_rate': 3.9994815624409785e-05, 'epoch': 0.37}

  4%|▎         | 1528/41250 [3:41:32<95:14:54,  8.63s/it][2025-04-25 11:39:16,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:39:16,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.49 | bwd_microstep: 5733.36 | bwd_inner_microstep: 5700.26 | bwd_allreduce_microstep: 33.05 | step_microstep: 19.38
[2025-04-25 11:39:16,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.49 | bwd: 5733.37 | bwd_inner: 5700.26 | bwd_allreduce: 33.07 | step: 19.38

  4%|▎         | 1529/41250 [3:41:41<95:22:40,  8.64s/it]
                                                         
{'loss': 0.3397, 'grad_norm': 3.6536428928375244, 'learning_rate': 3.999477981000813e-05, 'epoch': 0.37}

  4%|▎         | 1529/41250 [3:41:41<95:22:40,  8.64s/it][2025-04-25 11:39:24,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 11:39:24,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.59 | bwd_microstep: 5865.83 | bwd_inner_microstep: 5649.82 | bwd_allreduce_microstep: 215.96 | step_microstep: 19.09
[2025-04-25 11:39:24,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.59 | bwd: 5865.84 | bwd_inner: 5649.82 | bwd_allreduce: 215.98 | step: 19.09

  4%|▎         | 1530/41250 [3:41:50<95:48:36,  8.68s/it]
                                                         
{'loss': 0.1729, 'grad_norm': 1.52655029296875, 'learning_rate': 3.999474387234259e-05, 'epoch': 0.37}

  4%|▎         | 1530/41250 [3:41:50<95:48:36,  8.68s/it][2025-04-25 11:39:33,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:39:33,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.24 | bwd_microstep: 5757.10 | bwd_inner_microstep: 5634.05 | bwd_allreduce_microstep: 123.01 | step_microstep: 18.57
[2025-04-25 11:39:33,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.24 | bwd: 5757.12 | bwd_inner: 5634.05 | bwd_allreduce: 123.03 | step: 18.58

  4%|▎         | 1531/41250 [3:41:58<95:44:30,  8.68s/it]
                                                         
{'loss': 0.3087, 'grad_norm': 3.8370487689971924, 'learning_rate': 3.999470781141339e-05, 'epoch': 0.37}

  4%|▎         | 1531/41250 [3:41:58<95:44:30,  8.68s/it][2025-04-25 11:39:42,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 11:39:42,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.13 | bwd_microstep: 5769.59 | bwd_inner_microstep: 5756.72 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.07
[2025-04-25 11:39:42,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.13 | bwd: 5769.60 | bwd_inner: 5756.72 | bwd_allreduce: 12.84 | step: 19.08

  4%|▎         | 1532/41250 [3:42:07<95:56:35,  8.70s/it]
                                                         
{'loss': 0.2225, 'grad_norm': 2.9614336490631104, 'learning_rate': 3.999467162722075e-05, 'epoch': 0.37}

  4%|▎         | 1532/41250 [3:42:07<95:56:35,  8.70s/it][2025-04-25 11:39:51,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-25 11:39:51,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.69 | bwd_microstep: 5709.12 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.39
[2025-04-25 11:39:51,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.69 | bwd: 5709.14 | bwd_inner: 5695.81 | bwd_allreduce: 13.28 | step: 19.39

  4%|▎         | 1533/41250 [3:42:16<95:45:14,  8.68s/it]
                                                         
{'loss': 0.089, 'grad_norm': 1.66795015335083, 'learning_rate': 3.999463531976489e-05, 'epoch': 0.37}

  4%|▎         | 1533/41250 [3:42:16<95:45:14,  8.68s/it][2025-04-25 11:39:59,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 11:39:59,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.45 | bwd_microstep: 5694.20 | bwd_inner_microstep: 5681.02 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.05
[2025-04-25 11:39:59,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.45 | bwd: 5694.21 | bwd_inner: 5681.02 | bwd_allreduce: 13.15 | step: 19.05

  4%|▎         | 1534/41250 [3:42:24<95:32:43,  8.66s/it]
                                                         
{'loss': 0.1902, 'grad_norm': 0.6642282605171204, 'learning_rate': 3.999459888904603e-05, 'epoch': 0.37}

  4%|▎         | 1534/41250 [3:42:24<95:32:43,  8.66s/it][2025-04-25 11:40:08,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:40:08,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.93 | bwd_microstep: 5685.42 | bwd_inner_microstep: 5645.62 | bwd_allreduce_microstep: 39.75 | step_microstep: 18.38
[2025-04-25 11:40:08,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.93 | bwd: 5685.43 | bwd_inner: 5645.62 | bwd_allreduce: 39.77 | step: 18.38

  4%|▎         | 1535/41250 [3:42:33<95:18:09,  8.64s/it]
                                                         
{'loss': 0.237, 'grad_norm': 1.3791329860687256, 'learning_rate': 3.9994562335064415e-05, 'epoch': 0.37}

  4%|▎         | 1535/41250 [3:42:33<95:18:09,  8.64s/it][2025-04-25 11:40:17,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:40:17,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.79 | bwd_microstep: 6001.94 | bwd_inner_microstep: 5641.06 | bwd_allreduce_microstep: 360.83 | step_microstep: 18.37
[2025-04-25 11:40:17,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.79 | bwd: 6001.95 | bwd_inner: 5641.06 | bwd_allreduce: 360.85 | step: 18.37

  4%|▎         | 1536/41250 [3:42:42<96:11:10,  8.72s/it]
                                                         
{'loss': 0.2549, 'grad_norm': 2.020273447036743, 'learning_rate': 3.999452565782025e-05, 'epoch': 0.37}

  4%|▎         | 1536/41250 [3:42:42<96:11:10,  8.72s/it][2025-04-25 11:40:25,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.15 | optimizer_step: 1.07
[2025-04-25 11:40:25,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.46 | bwd_microstep: 5743.62 | bwd_inner_microstep: 5679.92 | bwd_allreduce_microstep: 63.65 | step_microstep: 19.47
[2025-04-25 11:40:25,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.46 | bwd: 5743.63 | bwd_inner: 5679.92 | bwd_allreduce: 63.67 | step: 19.47

  4%|▎         | 1537/41250 [3:42:51<96:01:20,  8.70s/it]
                                                         
{'loss': 0.2091, 'grad_norm': 1.526887059211731, 'learning_rate': 3.999448885731376e-05, 'epoch': 0.37}

  4%|▎         | 1537/41250 [3:42:51<96:01:20,  8.70s/it][2025-04-25 11:40:34,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-25 11:40:34,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.84 | bwd_microstep: 5738.39 | bwd_inner_microstep: 5669.33 | bwd_allreduce_microstep: 69.00 | step_microstep: 19.43
[2025-04-25 11:40:34,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.84 | bwd: 5738.40 | bwd_inner: 5669.33 | bwd_allreduce: 69.02 | step: 19.43

  4%|▎         | 1538/41250 [3:42:59<95:52:28,  8.69s/it]
                                                         
{'loss': 0.0411, 'grad_norm': 0.4531441926956177, 'learning_rate': 3.999445193354518e-05, 'epoch': 0.37}

  4%|▎         | 1538/41250 [3:42:59<95:52:28,  8.69s/it][2025-04-25 11:40:43,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 11:40:43,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.83 | bwd_microstep: 5741.96 | bwd_inner_microstep: 5680.18 | bwd_allreduce_microstep: 61.73 | step_microstep: 18.74
[2025-04-25 11:40:43,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.83 | bwd: 5741.97 | bwd_inner: 5680.18 | bwd_allreduce: 61.75 | step: 18.75

  4%|▎         | 1539/41250 [3:43:08<95:50:36,  8.69s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 1.7256642580032349, 'learning_rate': 3.999441488651474e-05, 'epoch': 0.37}

  4%|▎         | 1539/41250 [3:43:08<95:50:36,  8.69s/it][2025-04-25 11:40:51,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:40:51,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.98 | bwd_microstep: 5748.58 | bwd_inner_microstep: 5663.50 | bwd_allreduce_microstep: 85.04 | step_microstep: 18.69
[2025-04-25 11:40:51,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.98 | bwd: 5748.60 | bwd_inner: 5663.50 | bwd_allreduce: 85.06 | step: 18.70

  4%|▎         | 1540/41250 [3:43:17<95:47:02,  8.68s/it]
                                                         
{'loss': 0.1599, 'grad_norm': 1.7077654600143433, 'learning_rate': 3.999437771622267e-05, 'epoch': 0.37}

  4%|▎         | 1540/41250 [3:43:17<95:47:02,  8.68s/it][2025-04-25 11:41:00,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 11:41:00,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.45 | bwd_microstep: 5743.55 | bwd_inner_microstep: 5639.41 | bwd_allreduce_microstep: 104.09 | step_microstep: 18.76
[2025-04-25 11:41:00,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.45 | bwd: 5743.56 | bwd_inner: 5639.41 | bwd_allreduce: 104.11 | step: 18.76

  4%|▎         | 1541/41250 [3:43:25<95:39:25,  8.67s/it]
                                                         
{'loss': 0.2266, 'grad_norm': 1.603114128112793, 'learning_rate': 3.999434042266919e-05, 'epoch': 0.37}

  4%|▎         | 1541/41250 [3:43:25<95:39:25,  8.67s/it][2025-04-25 11:41:09,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:41:09,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.53 | bwd_microstep: 5857.92 | bwd_inner_microstep: 5668.10 | bwd_allreduce_microstep: 189.77 | step_microstep: 18.87
[2025-04-25 11:41:09,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.53 | bwd: 5857.93 | bwd_inner: 5668.10 | bwd_allreduce: 189.79 | step: 18.87

  4%|▎         | 1542/41250 [3:43:34<96:00:57,  8.70s/it]
                                                         
{'loss': 0.2575, 'grad_norm': 1.6573184728622437, 'learning_rate': 3.999430300585453e-05, 'epoch': 0.37}

  4%|▎         | 1542/41250 [3:43:34<96:00:57,  8.70s/it][2025-04-25 11:41:17,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:41:17,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.17 | bwd_microstep: 5693.78 | bwd_inner_microstep: 5634.89 | bwd_allreduce_microstep: 58.83 | step_microstep: 18.77
[2025-04-25 11:41:17,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.17 | bwd: 5693.79 | bwd_inner: 5634.89 | bwd_allreduce: 58.86 | step: 18.77

  4%|▎         | 1543/41250 [3:43:43<95:39:45,  8.67s/it]
                                                         
{'loss': 0.218, 'grad_norm': 1.7597390413284302, 'learning_rate': 3.9994265465778934e-05, 'epoch': 0.37}

  4%|▎         | 1543/41250 [3:43:43<95:39:45,  8.67s/it][2025-04-25 11:41:26,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 11:41:26,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.44 | bwd_microstep: 5780.84 | bwd_inner_microstep: 5636.03 | bwd_allreduce_microstep: 144.76 | step_microstep: 19.06
[2025-04-25 11:41:26,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.44 | bwd: 5780.86 | bwd_inner: 5636.03 | bwd_allreduce: 144.78 | step: 19.06

  4%|▎         | 1544/41250 [3:43:51<95:41:44,  8.68s/it]
                                                         
{'loss': 0.237, 'grad_norm': 1.8757612705230713, 'learning_rate': 3.999422780244262e-05, 'epoch': 0.37}

  4%|▎         | 1544/41250 [3:43:51<95:41:44,  8.68s/it][2025-04-25 11:41:35,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:41:35,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.30 | bwd_microstep: 5778.83 | bwd_inner_microstep: 5766.23 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.46
[2025-04-25 11:41:35,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.30 | bwd: 5778.84 | bwd_inner: 5766.23 | bwd_allreduce: 12.57 | step: 18.47

  4%|▎         | 1545/41250 [3:44:00<95:56:38,  8.70s/it]
                                                         
{'loss': 0.2328, 'grad_norm': 1.9808874130249023, 'learning_rate': 3.9994190015845815e-05, 'epoch': 0.37}

  4%|▎         | 1545/41250 [3:44:00<95:56:38,  8.70s/it][2025-04-25 11:41:43,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 11:41:43,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.05 | bwd_microstep: 5763.36 | bwd_inner_microstep: 5626.45 | bwd_allreduce_microstep: 136.86 | step_microstep: 18.78
[2025-04-25 11:41:43,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.05 | bwd: 5763.38 | bwd_inner: 5626.45 | bwd_allreduce: 136.88 | step: 18.78

  4%|▎         | 1546/41250 [3:44:09<95:50:52,  8.69s/it]
                                                         
{'loss': 0.3112, 'grad_norm': 1.4906306266784668, 'learning_rate': 3.999415210598877e-05, 'epoch': 0.37}

  4%|▎         | 1546/41250 [3:44:09<95:50:52,  8.69s/it][2025-04-25 11:41:52,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:41:52,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.93 | bwd_microstep: 5767.51 | bwd_inner_microstep: 5628.50 | bwd_allreduce_microstep: 138.97 | step_microstep: 18.35
[2025-04-25 11:41:52,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.93 | bwd: 5767.52 | bwd_inner: 5628.50 | bwd_allreduce: 138.99 | step: 18.35

  4%|▍         | 1547/41250 [3:44:17<95:47:37,  8.69s/it]
                                                         
{'loss': 0.0837, 'grad_norm': 0.917434811592102, 'learning_rate': 3.999411407287171e-05, 'epoch': 0.38}

  4%|▍         | 1547/41250 [3:44:17<95:47:37,  8.69s/it][2025-04-25 11:42:01,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:42:01,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.97 | bwd_microstep: 5701.91 | bwd_inner_microstep: 5689.32 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.33
[2025-04-25 11:42:01,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.97 | bwd: 5701.92 | bwd_inner: 5689.32 | bwd_allreduce: 12.57 | step: 18.34

  4%|▍         | 1548/41250 [3:44:26<95:38:14,  8.67s/it]
                                                         
{'loss': 0.0851, 'grad_norm': 1.1403930187225342, 'learning_rate': 3.999407591649486e-05, 'epoch': 0.38}

  4%|▍         | 1548/41250 [3:44:26<95:38:14,  8.67s/it][2025-04-25 11:42:09,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 11:42:09,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5701.43 | bwd_inner_microstep: 5627.21 | bwd_allreduce_microstep: 74.18 | step_microstep: 18.93
[2025-04-25 11:42:09,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5701.44 | bwd_inner: 5627.21 | bwd_allreduce: 74.20 | step: 18.93

  4%|▍         | 1549/41250 [3:44:35<95:27:54,  8.66s/it]
                                                         
{'loss': 0.1413, 'grad_norm': 2.4068241119384766, 'learning_rate': 3.999403763685847e-05, 'epoch': 0.38}

  4%|▍         | 1549/41250 [3:44:35<95:27:54,  8.66s/it][2025-04-25 11:42:18,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 11:42:18,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.40 | bwd_microstep: 5865.96 | bwd_inner_microstep: 5684.08 | bwd_allreduce_microstep: 181.84 | step_microstep: 18.72
[2025-04-25 11:42:18,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.40 | bwd: 5865.98 | bwd_inner: 5684.08 | bwd_allreduce: 181.86 | step: 18.73

  4%|▍         | 1550/41250 [3:44:43<95:55:59,  8.70s/it]
                                                         
{'loss': 0.3275, 'grad_norm': 1.757887363433838, 'learning_rate': 3.999399923396278e-05, 'epoch': 0.38}

  4%|▍         | 1550/41250 [3:44:43<95:55:59,  8.70s/it][2025-04-25 11:42:27,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:42:27,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5735.02 | bwd_inner_microstep: 5678.90 | bwd_allreduce_microstep: 56.08 | step_microstep: 18.87
[2025-04-25 11:42:27,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5735.04 | bwd_inner: 5678.90 | bwd_allreduce: 56.10 | step: 18.87

  4%|▍         | 1551/41250 [3:44:52<95:49:03,  8.69s/it]
                                                         
{'loss': 0.1642, 'grad_norm': 1.8592894077301025, 'learning_rate': 3.999396070780801e-05, 'epoch': 0.38}

  4%|▍         | 1551/41250 [3:44:52<95:49:03,  8.69s/it][2025-04-25 11:42:35,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:42:35,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5707.30 | bwd_inner_microstep: 5694.59 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.71
[2025-04-25 11:42:35,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5707.31 | bwd_inner: 5694.59 | bwd_allreduce: 12.69 | step: 18.71

  4%|▍         | 1552/41250 [3:45:01<95:38:27,  8.67s/it]
                                                         
{'loss': 0.4002, 'grad_norm': 2.7902472019195557, 'learning_rate': 3.99939220583944e-05, 'epoch': 0.38}

  4%|▍         | 1552/41250 [3:45:01<95:38:27,  8.67s/it][2025-04-25 11:42:44,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 11:42:44,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5723.09 | bwd_inner_microstep: 5644.45 | bwd_allreduce_microstep: 78.60 | step_microstep: 19.22
[2025-04-25 11:42:44,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5723.10 | bwd_inner: 5644.45 | bwd_allreduce: 78.61 | step: 19.22

  4%|▍         | 1553/41250 [3:45:09<95:30:32,  8.66s/it]
                                                         
{'loss': 0.2717, 'grad_norm': 1.1349719762802124, 'learning_rate': 3.99938832857222e-05, 'epoch': 0.38}

  4%|▍         | 1553/41250 [3:45:09<95:30:32,  8.66s/it][2025-04-25 11:42:53,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.21
[2025-04-25 11:42:53,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.62 | bwd_microstep: 5804.67 | bwd_inner_microstep: 5774.19 | bwd_allreduce_microstep: 30.24 | step_microstep: 19.83
[2025-04-25 11:42:53,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.62 | bwd: 5804.49 | bwd_inner: 5774.19 | bwd_allreduce: 30.26 | step: 19.83

  4%|▍         | 1554/41250 [3:45:18<95:53:54,  8.70s/it]
                                                         
{'loss': 0.4103, 'grad_norm': 2.239284038543701, 'learning_rate': 3.9993844389791644e-05, 'epoch': 0.38}

  4%|▍         | 1554/41250 [3:45:18<95:53:54,  8.70s/it][2025-04-25 11:43:02,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:43:02,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.72 | bwd_microstep: 5876.10 | bwd_inner_microstep: 5658.81 | bwd_allreduce_microstep: 217.24 | step_microstep: 18.87
[2025-04-25 11:43:02,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.72 | bwd: 5876.11 | bwd_inner: 5658.81 | bwd_allreduce: 217.26 | step: 18.87

  4%|▍         | 1555/41250 [3:45:27<96:11:53,  8.72s/it]
                                                         
{'loss': 0.2608, 'grad_norm': 1.5169453620910645, 'learning_rate': 3.9993805370602966e-05, 'epoch': 0.38}

  4%|▍         | 1555/41250 [3:45:27<96:11:53,  8.72s/it][2025-04-25 11:43:10,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:43:10,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.97 | bwd_microstep: 5716.43 | bwd_inner_microstep: 5658.66 | bwd_allreduce_microstep: 57.72 | step_microstep: 18.72
[2025-04-25 11:43:10,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.97 | bwd: 5716.45 | bwd_inner: 5658.66 | bwd_allreduce: 57.74 | step: 18.72

  4%|▍         | 1556/41250 [3:45:36<95:52:38,  8.70s/it]
                                                         
{'loss': 0.1353, 'grad_norm': 0.8497984409332275, 'learning_rate': 3.999376622815641e-05, 'epoch': 0.38}

  4%|▍         | 1556/41250 [3:45:36<95:52:38,  8.70s/it][2025-04-25 11:43:19,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 11:43:19,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.58 | bwd_microstep: 5729.80 | bwd_inner_microstep: 5716.95 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.23
[2025-04-25 11:43:19,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.58 | bwd: 5729.81 | bwd_inner: 5716.95 | bwd_allreduce: 12.82 | step: 19.23

  4%|▍         | 1557/41250 [3:45:44<95:47:16,  8.69s/it]
                                                         
{'loss': 0.4014, 'grad_norm': 2.4302191734313965, 'learning_rate': 3.999372696245222e-05, 'epoch': 0.38}

  4%|▍         | 1557/41250 [3:45:44<95:47:16,  8.69s/it][2025-04-25 11:43:28,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.39 | optimizer_step: 0.90
[2025-04-25 11:43:28,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5778.11 | bwd_inner_microstep: 5667.94 | bwd_allreduce_microstep: 110.12 | step_microstep: 20.04
[2025-04-25 11:43:28,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5778.12 | bwd_inner: 5667.94 | bwd_allreduce: 110.14 | step: 20.04

  4%|▍         | 1558/41250 [3:45:53<95:48:04,  8.69s/it]
                                                         
{'loss': 0.3544, 'grad_norm': 2.009505033493042, 'learning_rate': 3.999368757349063e-05, 'epoch': 0.38}

  4%|▍         | 1558/41250 [3:45:53<95:48:04,  8.69s/it][2025-04-25 11:43:36,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 11:43:36,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5780.50 | bwd_inner_microstep: 5698.30 | bwd_allreduce_microstep: 82.16 | step_microstep: 19.55
[2025-04-25 11:43:36,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5780.52 | bwd_inner: 5698.30 | bwd_allreduce: 82.18 | step: 19.55

  4%|▍         | 1559/41250 [3:46:02<95:54:12,  8.70s/it]
                                                         
{'loss': 0.1125, 'grad_norm': 1.1301692724227905, 'learning_rate': 3.999364806127189e-05, 'epoch': 0.38}

  4%|▍         | 1559/41250 [3:46:02<95:54:12,  8.70s/it][2025-04-25 11:43:45,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 11:43:45,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5786.10 | bwd_inner_microstep: 5653.37 | bwd_allreduce_microstep: 132.68 | step_microstep: 18.82
[2025-04-25 11:43:45,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5786.12 | bwd_inner: 5653.37 | bwd_allreduce: 132.70 | step: 18.82

  4%|▍         | 1560/41250 [3:46:10<95:53:50,  8.70s/it]
                                                         
{'loss': 0.1077, 'grad_norm': 0.7639368772506714, 'learning_rate': 3.999360842579625e-05, 'epoch': 0.38}

  4%|▍         | 1560/41250 [3:46:10<95:53:50,  8.70s/it][2025-04-25 11:43:54,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 11:43:54,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5809.30 | bwd_inner_microstep: 5699.26 | bwd_allreduce_microstep: 109.99 | step_microstep: 18.40
[2025-04-25 11:43:54,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5809.31 | bwd_inner: 5699.26 | bwd_allreduce: 110.01 | step: 18.40

  4%|▍         | 1561/41250 [3:46:19<96:02:15,  8.71s/it]
                                                         
{'loss': 0.169, 'grad_norm': 1.5484851598739624, 'learning_rate': 3.999356866706394e-05, 'epoch': 0.38}

  4%|▍         | 1561/41250 [3:46:19<96:02:15,  8.71s/it][2025-04-25 11:44:03,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.25 | optimizer_step: 1.06
[2025-04-25 11:44:03,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.58 | bwd_microstep: 5889.11 | bwd_inner_microstep: 5717.07 | bwd_allreduce_microstep: 171.99 | step_microstep: 19.93
[2025-04-25 11:44:03,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.58 | bwd: 5889.12 | bwd_inner: 5717.07 | bwd_allreduce: 172.00 | step: 19.93

  4%|▍         | 1562/41250 [3:46:28<96:26:21,  8.75s/it]
                                                         
{'loss': 0.1649, 'grad_norm': 3.0956740379333496, 'learning_rate': 3.999352878507521e-05, 'epoch': 0.38}

  4%|▍         | 1562/41250 [3:46:28<96:26:21,  8.75s/it][2025-04-25 11:44:11,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:44:11,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.04 | bwd_microstep: 5778.27 | bwd_inner_microstep: 5693.86 | bwd_allreduce_microstep: 84.37 | step_microstep: 18.65
[2025-04-25 11:44:11,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.04 | bwd: 5778.29 | bwd_inner: 5693.86 | bwd_allreduce: 84.39 | step: 18.65

  4%|▍         | 1563/41250 [3:46:37<96:18:52,  8.74s/it]
                                                         
{'loss': 0.2361, 'grad_norm': 1.4433127641677856, 'learning_rate': 3.999348877983031e-05, 'epoch': 0.38}

  4%|▍         | 1563/41250 [3:46:37<96:18:52,  8.74s/it][2025-04-25 11:44:20,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 11:44:20,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.46 | bwd_microstep: 5698.12 | bwd_inner_microstep: 5654.47 | bwd_allreduce_microstep: 43.61 | step_microstep: 19.65
[2025-04-25 11:44:20,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.46 | bwd: 5698.14 | bwd_inner: 5654.47 | bwd_allreduce: 43.63 | step: 19.65

  4%|▍         | 1564/41250 [3:46:45<95:54:53,  8.70s/it]
                                                         
{'loss': 0.209, 'grad_norm': 1.850227952003479, 'learning_rate': 3.999344865132949e-05, 'epoch': 0.38}

  4%|▍         | 1564/41250 [3:46:45<95:54:53,  8.70s/it][2025-04-25 11:44:29,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 11:44:29,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.09 | bwd_microstep: 5699.44 | bwd_inner_microstep: 5683.28 | bwd_allreduce_microstep: 16.12 | step_microstep: 19.11
[2025-04-25 11:44:29,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.09 | bwd: 5699.46 | bwd_inner: 5683.28 | bwd_allreduce: 16.14 | step: 19.11

  4%|▍         | 1565/41250 [3:46:54<95:43:19,  8.68s/it]
                                                         
{'loss': 0.283, 'grad_norm': 1.6545565128326416, 'learning_rate': 3.999340839957299e-05, 'epoch': 0.38}

  4%|▍         | 1565/41250 [3:46:54<95:43:19,  8.68s/it][2025-04-25 11:44:37,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 11:44:37,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.23 | bwd_microstep: 5714.15 | bwd_inner_microstep: 5653.85 | bwd_allreduce_microstep: 60.24 | step_microstep: 19.06
[2025-04-25 11:44:37,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.23 | bwd: 5714.17 | bwd_inner: 5653.85 | bwd_allreduce: 60.26 | step: 19.06

  4%|▍         | 1566/41250 [3:47:03<95:33:33,  8.67s/it]
                                                         
{'loss': 0.1175, 'grad_norm': 0.9658346176147461, 'learning_rate': 3.999336802456105e-05, 'epoch': 0.38}

  4%|▍         | 1566/41250 [3:47:03<95:33:33,  8.67s/it][2025-04-25 11:44:46,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-25 11:44:46,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.06 | bwd_microstep: 5712.12 | bwd_inner_microstep: 5689.47 | bwd_allreduce_microstep: 22.60 | step_microstep: 19.39
[2025-04-25 11:44:46,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.06 | bwd: 5712.13 | bwd_inner: 5689.47 | bwd_allreduce: 22.62 | step: 19.39

  4%|▍         | 1567/41250 [3:47:11<95:27:47,  8.66s/it]
                                                         
{'loss': 0.2249, 'grad_norm': 1.2645481824874878, 'learning_rate': 3.9993327526293936e-05, 'epoch': 0.38}

  4%|▍         | 1567/41250 [3:47:11<95:27:47,  8.66s/it][2025-04-25 11:44:55,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.15 | optimizer_step: 1.00
[2025-04-25 11:44:55,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5753.11 | bwd_inner_microstep: 5705.50 | bwd_allreduce_microstep: 47.56 | step_microstep: 19.34
[2025-04-25 11:44:55,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5753.12 | bwd_inner: 5705.50 | bwd_allreduce: 47.58 | step: 19.34

  4%|▍         | 1568/41250 [3:47:20<95:34:36,  8.67s/it]
                                                         
{'loss': 0.1259, 'grad_norm': 0.9213043451309204, 'learning_rate': 3.999328690477189e-05, 'epoch': 0.38}

  4%|▍         | 1568/41250 [3:47:20<95:34:36,  8.67s/it][2025-04-25 11:45:03,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 11:45:03,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.84 | bwd_microstep: 5759.92 | bwd_inner_microstep: 5710.27 | bwd_allreduce_microstep: 49.60 | step_microstep: 18.54
[2025-04-25 11:45:03,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.84 | bwd: 5759.93 | bwd_inner: 5710.27 | bwd_allreduce: 49.62 | step: 18.54

  4%|▍         | 1569/41250 [3:47:29<95:40:31,  8.68s/it]
                                                         
{'loss': 0.1096, 'grad_norm': 0.9344080090522766, 'learning_rate': 3.999324615999516e-05, 'epoch': 0.38}

  4%|▍         | 1569/41250 [3:47:29<95:40:31,  8.68s/it][2025-04-25 11:45:12,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:45:12,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.13 | bwd_microstep: 5801.93 | bwd_inner_microstep: 5789.38 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.79
[2025-04-25 11:45:12,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.13 | bwd: 5801.94 | bwd_inner: 5789.38 | bwd_allreduce: 12.52 | step: 18.80

  4%|▍         | 1570/41250 [3:47:37<96:00:16,  8.71s/it]
                                                         
{'loss': 0.2193, 'grad_norm': 1.2410224676132202, 'learning_rate': 3.9993205291964006e-05, 'epoch': 0.38}

  4%|▍         | 1570/41250 [3:47:37<96:00:16,  8.71s/it][2025-04-25 11:45:21,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 11:45:21,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 6018.21 | bwd_inner_microstep: 5704.55 | bwd_allreduce_microstep: 313.61 | step_microstep: 18.97
[2025-04-25 11:45:21,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 6018.23 | bwd_inner: 5704.55 | bwd_allreduce: 313.63 | step: 18.97

  4%|▍         | 1571/41250 [3:47:46<96:47:18,  8.78s/it]
                                                         
{'loss': 0.2371, 'grad_norm': 2.0199689865112305, 'learning_rate': 3.999316430067867e-05, 'epoch': 0.38}

  4%|▍         | 1571/41250 [3:47:46<96:47:18,  8.78s/it][2025-04-25 11:45:30,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:45:30,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.76 | bwd_microstep: 5771.49 | bwd_inner_microstep: 5650.09 | bwd_allreduce_microstep: 121.36 | step_microstep: 18.48
[2025-04-25 11:45:30,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.76 | bwd: 5771.50 | bwd_inner: 5650.09 | bwd_allreduce: 121.37 | step: 18.49

  4%|▍         | 1572/41250 [3:47:55<96:28:37,  8.75s/it]
                                                         
{'loss': 0.2006, 'grad_norm': 0.9910517334938049, 'learning_rate': 3.9993123186139405e-05, 'epoch': 0.38}

  4%|▍         | 1572/41250 [3:47:55<96:28:37,  8.75s/it][2025-04-25 11:45:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:45:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.14 | bwd_microstep: 5708.82 | bwd_inner_microstep: 5696.02 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.43
[2025-04-25 11:45:38,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.14 | bwd: 5708.84 | bwd_inner: 5696.02 | bwd_allreduce: 12.78 | step: 18.43

  4%|▍         | 1573/41250 [3:48:04<96:05:36,  8.72s/it]
                                                         
{'loss': 0.2295, 'grad_norm': 1.4510124921798706, 'learning_rate': 3.999308194834647e-05, 'epoch': 0.38}

  4%|▍         | 1573/41250 [3:48:04<96:05:36,  8.72s/it][2025-04-25 11:45:47,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-25 11:45:47,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.19 | bwd_microstep: 5871.46 | bwd_inner_microstep: 5687.86 | bwd_allreduce_microstep: 183.55 | step_microstep: 19.58
[2025-04-25 11:45:47,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.19 | bwd: 5871.48 | bwd_inner: 5687.85 | bwd_allreduce: 183.57 | step: 19.59

  4%|▍         | 1574/41250 [3:48:12<96:21:35,  8.74s/it]
                                                         
{'loss': 0.1915, 'grad_norm': 2.0279154777526855, 'learning_rate': 3.999304058730012e-05, 'epoch': 0.38}

  4%|▍         | 1574/41250 [3:48:12<96:21:35,  8.74s/it][2025-04-25 11:45:56,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 11:45:56,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5708.09 | bwd_inner_microstep: 5646.06 | bwd_allreduce_microstep: 61.98 | step_microstep: 18.63
[2025-04-25 11:45:56,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5708.10 | bwd_inner: 5646.06 | bwd_allreduce: 62.00 | step: 18.64

  4%|▍         | 1575/41250 [3:48:21<95:57:18,  8.71s/it]
                                                         
{'loss': 0.2808, 'grad_norm': 1.2059682607650757, 'learning_rate': 3.99929991030006e-05, 'epoch': 0.38}

  4%|▍         | 1575/41250 [3:48:21<95:57:18,  8.71s/it][2025-04-25 11:46:04,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 11:46:04,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.28 | bwd_microstep: 5739.77 | bwd_inner_microstep: 5698.99 | bwd_allreduce_microstep: 40.73 | step_microstep: 19.27
[2025-04-25 11:46:04,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.28 | bwd: 5739.79 | bwd_inner: 5698.99 | bwd_allreduce: 40.75 | step: 19.27

  4%|▍         | 1576/41250 [3:48:30<95:50:48,  8.70s/it]
                                                         
{'loss': 0.1145, 'grad_norm': 1.3857945203781128, 'learning_rate': 3.999295749544819e-05, 'epoch': 0.38}

  4%|▍         | 1576/41250 [3:48:30<95:50:48,  8.70s/it][2025-04-25 11:46:13,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 11:46:13,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.23 | bwd_microstep: 5862.83 | bwd_inner_microstep: 5681.63 | bwd_allreduce_microstep: 181.15 | step_microstep: 18.94
[2025-04-25 11:46:13,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.23 | bwd: 5862.85 | bwd_inner: 5681.63 | bwd_allreduce: 181.17 | step: 18.94

  4%|▍         | 1577/41250 [3:48:39<96:10:06,  8.73s/it]
                                                         
{'loss': 0.0575, 'grad_norm': 0.47422781586647034, 'learning_rate': 3.999291576464311e-05, 'epoch': 0.38}

  4%|▍         | 1577/41250 [3:48:39<96:10:06,  8.73s/it][2025-04-25 11:46:22,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 11:46:22,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.09 | bwd_microstep: 5719.19 | bwd_inner_microstep: 5686.00 | bwd_allreduce_microstep: 33.13 | step_microstep: 19.18
[2025-04-25 11:46:22,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.09 | bwd: 5719.21 | bwd_inner: 5686.00 | bwd_allreduce: 33.16 | step: 19.18

  4%|▍         | 1578/41250 [3:48:47<95:54:34,  8.70s/it]
                                                         
{'loss': 0.1548, 'grad_norm': 1.175406813621521, 'learning_rate': 3.999287391058564e-05, 'epoch': 0.38}

  4%|▍         | 1578/41250 [3:48:47<95:54:34,  8.70s/it][2025-04-25 11:46:31,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:46:31,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.56 | bwd_microstep: 5725.90 | bwd_inner_microstep: 5689.32 | bwd_allreduce_microstep: 36.54 | step_microstep: 18.93
[2025-04-25 11:46:31,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5725.91 | bwd_inner: 5689.32 | bwd_allreduce: 36.56 | step: 18.93

  4%|▍         | 1579/41250 [3:48:56<95:45:51,  8.69s/it]
                                                         
{'loss': 0.2255, 'grad_norm': 1.447347640991211, 'learning_rate': 3.9992831933276045e-05, 'epoch': 0.38}

  4%|▍         | 1579/41250 [3:48:56<95:45:51,  8.69s/it][2025-04-25 11:46:39,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.17 | optimizer_step: 0.91
[2025-04-25 11:46:39,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5743.51 | bwd_inner_microstep: 5676.92 | bwd_allreduce_microstep: 66.54 | step_microstep: 18.89
[2025-04-25 11:46:39,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5743.53 | bwd_inner: 5676.92 | bwd_allreduce: 66.57 | step: 18.89

  4%|▍         | 1580/41250 [3:49:05<95:42:15,  8.69s/it]
                                                         
{'loss': 0.1155, 'grad_norm': 1.0151011943817139, 'learning_rate': 3.999278983271457e-05, 'epoch': 0.38}

  4%|▍         | 1580/41250 [3:49:05<95:42:15,  8.69s/it][2025-04-25 11:46:48,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 1.13
[2025-04-25 11:46:48,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5706.16 | bwd_inner_microstep: 5693.46 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.34
[2025-04-25 11:46:48,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5706.18 | bwd_inner: 5693.46 | bwd_allreduce: 12.67 | step: 19.34

  4%|▍         | 1581/41250 [3:49:13<95:33:31,  8.67s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.2132169008255005, 'learning_rate': 3.9992747608901475e-05, 'epoch': 0.38}

  4%|▍         | 1581/41250 [3:49:13<95:33:31,  8.67s/it][2025-04-25 11:46:56,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:46:56,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5696.29 | bwd_inner_microstep: 5655.03 | bwd_allreduce_microstep: 41.22 | step_microstep: 18.84
[2025-04-25 11:46:56,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5696.31 | bwd_inner: 5655.03 | bwd_allreduce: 41.24 | step: 18.84

  4%|▍         | 1582/41250 [3:49:22<95:19:59,  8.65s/it]
                                                         
{'loss': 0.1362, 'grad_norm': 1.2754241228103638, 'learning_rate': 3.999270526183702e-05, 'epoch': 0.38}

  4%|▍         | 1582/41250 [3:49:22<95:19:59,  8.65s/it][2025-04-25 11:47:05,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 11:47:05,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.66 | bwd_microstep: 5699.71 | bwd_inner_microstep: 5686.99 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.27
[2025-04-25 11:47:05,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.66 | bwd: 5699.72 | bwd_inner: 5686.99 | bwd_allreduce: 12.70 | step: 19.27

  4%|▍         | 1583/41250 [3:49:30<95:15:01,  8.64s/it]
                                                         
{'loss': 0.0966, 'grad_norm': 0.8304641246795654, 'learning_rate': 3.9992662791521474e-05, 'epoch': 0.38}

  4%|▍         | 1583/41250 [3:49:30<95:15:01,  8.64s/it][2025-04-25 11:47:14,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:47:14,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.45 | bwd_microstep: 5699.53 | bwd_inner_microstep: 5686.84 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.91
[2025-04-25 11:47:14,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.45 | bwd: 5699.55 | bwd_inner: 5686.84 | bwd_allreduce: 12.67 | step: 18.91

  4%|▍         | 1584/41250 [3:49:39<95:12:01,  8.64s/it]
                                                         
{'loss': 0.0666, 'grad_norm': 0.9794690608978271, 'learning_rate': 3.999262019795509e-05, 'epoch': 0.38}

  4%|▍         | 1584/41250 [3:49:39<95:12:01,  8.64s/it][2025-04-25 11:47:23,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:47:23,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.02 | bwd_microstep: 5864.25 | bwd_inner_microstep: 5681.49 | bwd_allreduce_microstep: 182.72 | step_microstep: 18.88
[2025-04-25 11:47:23,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.02 | bwd: 5864.26 | bwd_inner: 5681.49 | bwd_allreduce: 182.73 | step: 18.88

  4%|▍         | 1585/41250 [3:49:48<95:42:18,  8.69s/it]
                                                         
{'loss': 0.0622, 'grad_norm': 0.7639535069465637, 'learning_rate': 3.999257748113815e-05, 'epoch': 0.38}

  4%|▍         | 1585/41250 [3:49:48<95:42:18,  8.69s/it][2025-04-25 11:47:31,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 11:47:31,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.22 | bwd_microstep: 5723.63 | bwd_inner_microstep: 5675.83 | bwd_allreduce_microstep: 47.76 | step_microstep: 19.12
[2025-04-25 11:47:31,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.22 | bwd: 5723.65 | bwd_inner: 5675.83 | bwd_allreduce: 47.78 | step: 19.12

  4%|▍         | 1586/41250 [3:49:57<95:35:54,  8.68s/it]
                                                         
{'loss': 0.1001, 'grad_norm': 0.9830851554870605, 'learning_rate': 3.999253464107089e-05, 'epoch': 0.38}

  4%|▍         | 1586/41250 [3:49:57<95:35:54,  8.68s/it][2025-04-25 11:47:40,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:47:40,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.17 | bwd_microstep: 5690.09 | bwd_inner_microstep: 5634.30 | bwd_allreduce_microstep: 55.75 | step_microstep: 18.13
[2025-04-25 11:47:40,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.17 | bwd: 5690.10 | bwd_inner: 5634.30 | bwd_allreduce: 55.77 | step: 18.13

  4%|▍         | 1587/41250 [3:50:05<95:21:01,  8.65s/it]
                                                         
{'loss': 0.0777, 'grad_norm': 0.8143350481987, 'learning_rate': 3.9992491677753585e-05, 'epoch': 0.38}

  4%|▍         | 1587/41250 [3:50:05<95:21:01,  8.65s/it][2025-04-25 11:47:48,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:47:48,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.20 | bwd_microstep: 5705.72 | bwd_inner_microstep: 5690.20 | bwd_allreduce_microstep: 15.48 | step_microstep: 18.60
[2025-04-25 11:47:48,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.20 | bwd: 5705.73 | bwd_inner: 5690.20 | bwd_allreduce: 15.49 | step: 18.61

  4%|▍         | 1588/41250 [3:50:14<95:19:32,  8.65s/it]
                                                         
{'loss': 0.3216, 'grad_norm': 1.8433016538619995, 'learning_rate': 3.99924485911865e-05, 'epoch': 0.38}

  4%|▍         | 1588/41250 [3:50:14<95:19:32,  8.65s/it][2025-04-25 11:47:57,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:47:57,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.01 | bwd_microstep: 5722.80 | bwd_inner_microstep: 5676.19 | bwd_allreduce_microstep: 46.57 | step_microstep: 18.98
[2025-04-25 11:47:57,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.01 | bwd: 5722.82 | bwd_inner: 5676.19 | bwd_allreduce: 46.58 | step: 18.99

  4%|▍         | 1589/41250 [3:50:22<95:19:08,  8.65s/it]
                                                         
{'loss': 0.3061, 'grad_norm': 2.6893255710601807, 'learning_rate': 3.999240538136991e-05, 'epoch': 0.39}

  4%|▍         | 1589/41250 [3:50:22<95:19:08,  8.65s/it][2025-04-25 11:48:06,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:48:06,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.90 | bwd_microstep: 5744.77 | bwd_inner_microstep: 5635.95 | bwd_allreduce_microstep: 108.77 | step_microstep: 18.88
[2025-04-25 11:48:06,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.90 | bwd: 5744.78 | bwd_inner: 5635.95 | bwd_allreduce: 108.79 | step: 18.88

  4%|▍         | 1590/41250 [3:50:31<95:17:47,  8.65s/it]
                                                         
{'loss': 0.3887, 'grad_norm': 1.8864761590957642, 'learning_rate': 3.999236204830406e-05, 'epoch': 0.39}

  4%|▍         | 1590/41250 [3:50:31<95:17:47,  8.65s/it][2025-04-25 11:48:14,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 11:48:14,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.32 | bwd_microstep: 5758.54 | bwd_inner_microstep: 5677.85 | bwd_allreduce_microstep: 80.65 | step_microstep: 18.56
[2025-04-25 11:48:14,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.32 | bwd: 5758.55 | bwd_inner: 5677.85 | bwd_allreduce: 80.66 | step: 18.56

  4%|▍         | 1591/41250 [3:50:40<95:24:14,  8.66s/it]
                                                         
{'loss': 0.1698, 'grad_norm': 2.218549966812134, 'learning_rate': 3.999231859198924e-05, 'epoch': 0.39}

  4%|▍         | 1591/41250 [3:50:40<95:24:14,  8.66s/it][2025-04-25 11:48:23,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 11:48:23,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.58 | bwd_microstep: 5782.07 | bwd_inner_microstep: 5769.23 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.79
[2025-04-25 11:48:23,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.58 | bwd: 5782.09 | bwd_inner: 5769.23 | bwd_allreduce: 12.81 | step: 18.80

  4%|▍         | 1592/41250 [3:50:48<95:42:41,  8.69s/it]
                                                         
{'loss': 0.2936, 'grad_norm': 1.6439919471740723, 'learning_rate': 3.999227501242571e-05, 'epoch': 0.39}

  4%|▍         | 1592/41250 [3:50:48<95:42:41,  8.69s/it][2025-04-25 11:48:32,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 11:48:32,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5672.31 | bwd_inner_microstep: 5659.60 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.43
[2025-04-25 11:48:32,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5672.32 | bwd_inner: 5659.60 | bwd_allreduce: 12.69 | step: 18.43

  4%|▍         | 1593/41250 [3:50:57<95:27:36,  8.67s/it]
                                                         
{'loss': 0.0937, 'grad_norm': 1.4316694736480713, 'learning_rate': 3.9992231309613735e-05, 'epoch': 0.39}

  4%|▍         | 1593/41250 [3:50:57<95:27:36,  8.67s/it][2025-04-25 11:48:40,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 11:48:40,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.47 | bwd_microstep: 5716.76 | bwd_inner_microstep: 5695.28 | bwd_allreduce_microstep: 21.43 | step_microstep: 18.74
[2025-04-25 11:48:40,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.47 | bwd: 5716.77 | bwd_inner: 5695.28 | bwd_allreduce: 21.45 | step: 18.74

  4%|▍         | 1594/41250 [3:51:06<95:24:41,  8.66s/it]
                                                         
{'loss': 0.2629, 'grad_norm': 2.8928160667419434, 'learning_rate': 3.9992187483553584e-05, 'epoch': 0.39}

  4%|▍         | 1594/41250 [3:51:06<95:24:41,  8.66s/it][2025-04-25 11:48:49,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:48:49,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.91 | bwd_microstep: 5698.23 | bwd_inner_microstep: 5685.62 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.57
[2025-04-25 11:48:49,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.91 | bwd: 5698.25 | bwd_inner: 5685.62 | bwd_allreduce: 12.58 | step: 18.57

  4%|▍         | 1595/41250 [3:51:14<95:17:47,  8.65s/it]
                                                         
{'loss': 0.2805, 'grad_norm': 2.507333755493164, 'learning_rate': 3.9992143534245524e-05, 'epoch': 0.39}

  4%|▍         | 1595/41250 [3:51:14<95:17:47,  8.65s/it][2025-04-25 11:48:58,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.14 | optimizer_step: 1.01
[2025-04-25 11:48:58,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.42 | bwd_microstep: 5689.31 | bwd_inner_microstep: 5629.97 | bwd_allreduce_microstep: 59.26 | step_microstep: 18.92
[2025-04-25 11:48:58,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.42 | bwd: 5689.33 | bwd_inner: 5629.97 | bwd_allreduce: 59.30 | step: 18.92

  4%|▍         | 1596/41250 [3:51:23<95:06:14,  8.63s/it]
                                                         
{'loss': 0.272, 'grad_norm': 3.280416488647461, 'learning_rate': 3.999209946168984e-05, 'epoch': 0.39}

  4%|▍         | 1596/41250 [3:51:23<95:06:14,  8.63s/it][2025-04-25 11:49:06,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 11:49:06,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.15 | bwd_microstep: 5682.96 | bwd_inner_microstep: 5670.29 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.40
[2025-04-25 11:49:06,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.15 | bwd: 5682.98 | bwd_inner: 5670.29 | bwd_allreduce: 12.64 | step: 19.40

  4%|▍         | 1597/41250 [3:51:32<95:01:07,  8.63s/it]
                                                         
{'loss': 0.1506, 'grad_norm': 2.2930727005004883, 'learning_rate': 3.9992055265886796e-05, 'epoch': 0.39}

  4%|▍         | 1597/41250 [3:51:32<95:01:07,  8.63s/it][2025-04-25 11:49:15,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 0.93
[2025-04-25 11:49:15,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.07 | bwd_microstep: 5756.70 | bwd_inner_microstep: 5664.36 | bwd_allreduce_microstep: 92.30 | step_microstep: 19.46
[2025-04-25 11:49:15,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.07 | bwd: 5756.72 | bwd_inner: 5664.36 | bwd_allreduce: 92.31 | step: 19.46

  4%|▍         | 1598/41250 [3:51:40<95:10:30,  8.64s/it]
                                                         
{'loss': 0.2089, 'grad_norm': 1.8200085163116455, 'learning_rate': 3.999201094683666e-05, 'epoch': 0.39}

  4%|▍         | 1598/41250 [3:51:40<95:10:30,  8.64s/it][2025-04-25 11:49:24,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:49:24,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.82 | bwd_microstep: 5693.39 | bwd_inner_microstep: 5680.52 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.82
[2025-04-25 11:49:24,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.82 | bwd: 5693.41 | bwd_inner: 5680.52 | bwd_allreduce: 12.85 | step: 18.82

  4%|▍         | 1599/41250 [3:51:49<95:06:15,  8.63s/it]
                                                         
{'loss': 0.054, 'grad_norm': 1.0575039386749268, 'learning_rate': 3.999196650453971e-05, 'epoch': 0.39}

  4%|▍         | 1599/41250 [3:51:49<95:06:15,  8.63s/it][2025-04-25 11:49:32,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:49:32,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.60 | bwd_microstep: 5740.24 | bwd_inner_microstep: 5634.27 | bwd_allreduce_microstep: 105.92 | step_microstep: 18.91
[2025-04-25 11:49:32,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.60 | bwd: 5740.25 | bwd_inner: 5634.27 | bwd_allreduce: 105.93 | step: 18.92

  4%|▍         | 1600/41250 [3:51:58<95:07:38,  8.64s/it]
                                                         
{'loss': 0.3687, 'grad_norm': 2.8633251190185547, 'learning_rate': 3.999192193899621e-05, 'epoch': 0.39}

  4%|▍         | 1600/41250 [3:51:58<95:07:38,  8.64s/it][2025-04-25 11:49:41,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:49:41,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.46 | bwd_microstep: 5756.43 | bwd_inner_microstep: 5634.97 | bwd_allreduce_microstep: 121.42 | step_microstep: 18.64
[2025-04-25 11:49:41,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.46 | bwd: 5756.44 | bwd_inner: 5634.97 | bwd_allreduce: 121.43 | step: 18.65

  4%|▍         | 1601/41250 [3:52:06<95:12:04,  8.64s/it]
                                                         
{'loss': 0.3905, 'grad_norm': 2.6136951446533203, 'learning_rate': 3.9991877250206454e-05, 'epoch': 0.39}

  4%|▍         | 1601/41250 [3:52:06<95:12:04,  8.64s/it][2025-04-25 11:49:50,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-25 11:49:50,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5743.25 | bwd_inner_microstep: 5687.75 | bwd_allreduce_microstep: 55.45 | step_microstep: 19.25
[2025-04-25 11:49:50,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5743.26 | bwd_inner: 5687.75 | bwd_allreduce: 55.47 | step: 19.25

  4%|▍         | 1602/41250 [3:52:15<95:17:45,  8.65s/it]
                                                         
{'loss': 0.2614, 'grad_norm': 1.5512330532073975, 'learning_rate': 3.9991832438170706e-05, 'epoch': 0.39}

  4%|▍         | 1602/41250 [3:52:15<95:17:45,  8.65s/it][2025-04-25 11:49:58,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.20 | optimizer_step: 0.92
[2025-04-25 11:49:58,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.87 | bwd_microstep: 5679.17 | bwd_inner_microstep: 5665.75 | bwd_allreduce_microstep: 13.38 | step_microstep: 19.23
[2025-04-25 11:49:58,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.87 | bwd: 5679.19 | bwd_inner: 5665.75 | bwd_allreduce: 13.40 | step: 19.24

  4%|▍         | 1603/41250 [3:52:23<95:07:39,  8.64s/it]
                                                         
{'loss': 0.1502, 'grad_norm': 1.1709187030792236, 'learning_rate': 3.999178750288924e-05, 'epoch': 0.39}

  4%|▍         | 1603/41250 [3:52:23<95:07:39,  8.64s/it][2025-04-25 11:50:07,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:50:07,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.77 | bwd_microstep: 5741.14 | bwd_inner_microstep: 5638.46 | bwd_allreduce_microstep: 102.63 | step_microstep: 18.74
[2025-04-25 11:50:07,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.77 | bwd: 5741.16 | bwd_inner: 5638.46 | bwd_allreduce: 102.65 | step: 18.74

  4%|▍         | 1604/41250 [3:52:32<95:08:21,  8.64s/it]
                                                         
{'loss': 0.237, 'grad_norm': 2.380575180053711, 'learning_rate': 3.9991742444362334e-05, 'epoch': 0.39}

  4%|▍         | 1604/41250 [3:52:32<95:08:21,  8.64s/it][2025-04-25 11:50:15,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 11:50:15,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.78 | bwd_microstep: 5727.83 | bwd_inner_microstep: 5673.48 | bwd_allreduce_microstep: 54.31 | step_microstep: 18.52
[2025-04-25 11:50:15,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.78 | bwd: 5727.84 | bwd_inner: 5673.48 | bwd_allreduce: 54.33 | step: 18.52

  4%|▍         | 1605/41250 [3:52:41<95:09:57,  8.64s/it]
                                                         
{'loss': 0.1706, 'grad_norm': 1.0646014213562012, 'learning_rate': 3.9991697262590273e-05, 'epoch': 0.39}

  4%|▍         | 1605/41250 [3:52:41<95:09:57,  8.64s/it][2025-04-25 11:50:24,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:50:24,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5747.20 | bwd_inner_microstep: 5687.89 | bwd_allreduce_microstep: 59.27 | step_microstep: 18.92
[2025-04-25 11:50:24,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5747.21 | bwd_inner: 5687.89 | bwd_allreduce: 59.29 | step: 18.92

  4%|▍         | 1606/41250 [3:52:49<95:17:53,  8.65s/it]
                                                         
{'loss': 0.2327, 'grad_norm': 1.7743453979492188, 'learning_rate': 3.9991651957573327e-05, 'epoch': 0.39}

  4%|▍         | 1606/41250 [3:52:49<95:17:53,  8.65s/it][2025-04-25 11:50:33,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.19
[2025-04-25 11:50:33,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.89 | bwd_microstep: 5737.46 | bwd_inner_microstep: 5642.71 | bwd_allreduce_microstep: 94.71 | step_microstep: 19.03
[2025-04-25 11:50:33,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.89 | bwd: 5737.47 | bwd_inner: 5642.71 | bwd_allreduce: 94.73 | step: 19.03

  4%|▍         | 1607/41250 [3:52:58<95:17:52,  8.65s/it]
                                                         
{'loss': 0.1928, 'grad_norm': 1.6312782764434814, 'learning_rate': 3.9991606529311785e-05, 'epoch': 0.39}

  4%|▍         | 1607/41250 [3:52:58<95:17:52,  8.65s/it][2025-04-25 11:50:41,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:50:41,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.82 | bwd_microstep: 5685.17 | bwd_inner_microstep: 5635.26 | bwd_allreduce_microstep: 49.86 | step_microstep: 18.49
[2025-04-25 11:50:41,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.82 | bwd: 5685.18 | bwd_inner: 5635.26 | bwd_allreduce: 49.88 | step: 18.50

  4%|▍         | 1608/41250 [3:53:07<95:05:23,  8.64s/it]
                                                         
{'loss': 0.072, 'grad_norm': 1.1187372207641602, 'learning_rate': 3.999156097780591e-05, 'epoch': 0.39}

  4%|▍         | 1608/41250 [3:53:07<95:05:23,  8.64s/it][2025-04-25 11:50:50,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 11:50:50,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5711.45 | bwd_inner_microstep: 5698.77 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.49
[2025-04-25 11:50:50,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5711.46 | bwd_inner: 5698.77 | bwd_allreduce: 12.65 | step: 18.49

  4%|▍         | 1609/41250 [3:53:15<95:06:55,  8.64s/it]
                                                         
{'loss': 0.2293, 'grad_norm': 2.8406805992126465, 'learning_rate': 3.9991515303056e-05, 'epoch': 0.39}

  4%|▍         | 1609/41250 [3:53:15<95:06:55,  8.64s/it][2025-04-25 11:50:59,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:50:59,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.26 | bwd_microstep: 5737.88 | bwd_inner_microstep: 5687.83 | bwd_allreduce_microstep: 50.00 | step_microstep: 17.97
[2025-04-25 11:50:59,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.26 | bwd: 5737.89 | bwd_inner: 5687.83 | bwd_allreduce: 50.02 | step: 17.98

  4%|▍         | 1610/41250 [3:53:24<95:14:29,  8.65s/it]
                                                         
{'loss': 0.1415, 'grad_norm': 1.5333821773529053, 'learning_rate': 3.9991469505062335e-05, 'epoch': 0.39}

  4%|▍         | 1610/41250 [3:53:24<95:14:29,  8.65s/it][2025-04-25 11:51:07,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:51:07,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.58 | bwd_microstep: 5785.41 | bwd_inner_microstep: 5772.79 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.33
[2025-04-25 11:51:07,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.58 | bwd: 5785.42 | bwd_inner: 5772.79 | bwd_allreduce: 12.59 | step: 18.33

  4%|▍         | 1611/41250 [3:53:33<95:36:11,  8.68s/it]
                                                         
{'loss': 0.2559, 'grad_norm': 1.870612621307373, 'learning_rate': 3.9991423583825185e-05, 'epoch': 0.39}

  4%|▍         | 1611/41250 [3:53:33<95:36:11,  8.68s/it][2025-04-25 11:51:16,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 11:51:16,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.67 | bwd_microstep: 5738.68 | bwd_inner_microstep: 5707.96 | bwd_allreduce_microstep: 30.68 | step_microstep: 18.53
[2025-04-25 11:51:16,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.67 | bwd: 5738.70 | bwd_inner: 5707.96 | bwd_allreduce: 30.69 | step: 18.53

  4%|▍         | 1612/41250 [3:53:41<95:34:26,  8.68s/it]
                                                         
{'loss': 0.1063, 'grad_norm': 1.2002270221710205, 'learning_rate': 3.999137753934484e-05, 'epoch': 0.39}

  4%|▍         | 1612/41250 [3:53:41<95:34:26,  8.68s/it][2025-04-25 11:51:25,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 11:51:25,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.31 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5696.83 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.96
[2025-04-25 11:51:25,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.31 | bwd: 5709.57 | bwd_inner: 5696.83 | bwd_allreduce: 12.70 | step: 18.96

  4%|▍         | 1613/41250 [3:53:50<95:29:47,  8.67s/it]
                                                         
{'loss': 0.1706, 'grad_norm': 1.5892360210418701, 'learning_rate': 3.9991331371621587e-05, 'epoch': 0.39}

  4%|▍         | 1613/41250 [3:53:50<95:29:47,  8.67s/it][2025-04-25 11:51:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:51:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.18 | bwd_microstep: 5728.58 | bwd_inner_microstep: 5709.76 | bwd_allreduce_microstep: 18.78 | step_microstep: 18.58
[2025-04-25 11:51:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.18 | bwd: 5728.59 | bwd_inner: 5709.76 | bwd_allreduce: 18.79 | step: 18.58

  4%|▍         | 1614/41250 [3:53:59<95:29:15,  8.67s/it]
                                                         
{'loss': 0.2376, 'grad_norm': 1.6710749864578247, 'learning_rate': 3.9991285080655714e-05, 'epoch': 0.39}

  4%|▍         | 1614/41250 [3:53:59<95:29:15,  8.67s/it][2025-04-25 11:51:42,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:51:42,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.14 | bwd_microstep: 5728.90 | bwd_inner_microstep: 5716.05 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.16
[2025-04-25 11:51:42,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.14 | bwd: 5728.91 | bwd_inner: 5716.05 | bwd_allreduce: 12.82 | step: 18.17

  4%|▍         | 1615/41250 [3:54:07<95:28:31,  8.67s/it]
                                                         
{'loss': 0.0392, 'grad_norm': 0.34792736172676086, 'learning_rate': 3.9991238666447496e-05, 'epoch': 0.39}

  4%|▍         | 1615/41250 [3:54:07<95:28:31,  8.67s/it][2025-04-25 11:51:51,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-25 11:51:51,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.89 | bwd_microstep: 5773.93 | bwd_inner_microstep: 5670.10 | bwd_allreduce_microstep: 103.78 | step_microstep: 18.51
[2025-04-25 11:51:51,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.89 | bwd: 5773.94 | bwd_inner: 5670.10 | bwd_allreduce: 103.79 | step: 18.52

  4%|▍         | 1616/41250 [3:54:16<95:32:16,  8.68s/it]
                                                         
{'loss': 0.2275, 'grad_norm': 1.788496732711792, 'learning_rate': 3.9991192128997225e-05, 'epoch': 0.39}

  4%|▍         | 1616/41250 [3:54:16<95:32:16,  8.68s/it][2025-04-25 11:51:59,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 11:51:59,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5686.01 | bwd_inner_microstep: 5667.26 | bwd_allreduce_microstep: 18.70 | step_microstep: 18.16
[2025-04-25 11:51:59,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.59 | bwd: 5686.02 | bwd_inner: 5667.26 | bwd_allreduce: 18.72 | step: 18.16

  4%|▍         | 1617/41250 [3:54:25<95:17:30,  8.66s/it]
                                                         
{'loss': 0.3039, 'grad_norm': 2.4700074195861816, 'learning_rate': 3.9991145468305184e-05, 'epoch': 0.39}

  4%|▍         | 1617/41250 [3:54:25<95:17:30,  8.66s/it][2025-04-25 11:52:08,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-25 11:52:08,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.34 | bwd_microstep: 5731.48 | bwd_inner_microstep: 5718.84 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.42
[2025-04-25 11:52:08,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.34 | bwd: 5731.50 | bwd_inner: 5718.84 | bwd_allreduce: 12.62 | step: 18.42

  4%|▍         | 1618/41250 [3:54:33<95:21:26,  8.66s/it]
                                                         
{'loss': 0.1753, 'grad_norm': 1.5120759010314941, 'learning_rate': 3.9991098684371666e-05, 'epoch': 0.39}

  4%|▍         | 1618/41250 [3:54:33<95:21:26,  8.66s/it][2025-04-25 11:52:17,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 11:52:17,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.64 | bwd_microstep: 5716.56 | bwd_inner_microstep: 5703.97 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.40
[2025-04-25 11:52:17,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.64 | bwd: 5716.57 | bwd_inner: 5703.97 | bwd_allreduce: 12.56 | step: 18.40

  4%|▍         | 1619/41250 [3:54:42<95:18:33,  8.66s/it]
                                                         
{'loss': 0.3783, 'grad_norm': 2.1091582775115967, 'learning_rate': 3.9991051777196954e-05, 'epoch': 0.39}

  4%|▍         | 1619/41250 [3:54:42<95:18:33,  8.66s/it][2025-04-25 11:52:25,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:52:25,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.66 | bwd_microstep: 5751.81 | bwd_inner_microstep: 5712.77 | bwd_allreduce_microstep: 39.00 | step_microstep: 18.45
[2025-04-25 11:52:25,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.66 | bwd: 5751.82 | bwd_inner: 5712.77 | bwd_allreduce: 39.01 | step: 18.45

  4%|▍         | 1620/41250 [3:54:51<95:24:12,  8.67s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 1.4435621500015259, 'learning_rate': 3.999100474678134e-05, 'epoch': 0.39}

  4%|▍         | 1620/41250 [3:54:51<95:24:12,  8.67s/it][2025-04-25 11:52:34,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 11:52:34,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.21 | bwd_microstep: 5712.53 | bwd_inner_microstep: 5663.94 | bwd_allreduce_microstep: 48.55 | step_microstep: 18.24
[2025-04-25 11:52:34,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.21 | bwd: 5712.54 | bwd_inner: 5663.94 | bwd_allreduce: 48.56 | step: 18.24

  4%|▍         | 1621/41250 [3:54:59<95:17:00,  8.66s/it]
                                                         
{'loss': 0.1065, 'grad_norm': 1.2231148481369019, 'learning_rate': 3.999095759312512e-05, 'epoch': 0.39}

  4%|▍         | 1621/41250 [3:54:59<95:17:00,  8.66s/it][2025-04-25 11:52:43,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 11:52:43,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.51 | bwd_microstep: 5743.64 | bwd_inner_microstep: 5669.15 | bwd_allreduce_microstep: 74.45 | step_microstep: 18.88
[2025-04-25 11:52:43,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.51 | bwd: 5743.65 | bwd_inner: 5669.15 | bwd_allreduce: 74.47 | step: 18.88

  4%|▍         | 1622/41250 [3:55:08<95:20:48,  8.66s/it]
                                                         
{'loss': 0.2044, 'grad_norm': 1.2171348333358765, 'learning_rate': 3.999091031622857e-05, 'epoch': 0.39}

  4%|▍         | 1622/41250 [3:55:08<95:20:48,  8.66s/it][2025-04-25 11:52:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:52:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.04 | bwd_microstep: 5726.43 | bwd_inner_microstep: 5660.65 | bwd_allreduce_microstep: 65.74 | step_microstep: 17.93
[2025-04-25 11:52:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.04 | bwd: 5726.44 | bwd_inner: 5660.65 | bwd_allreduce: 65.75 | step: 17.94

  4%|▍         | 1623/41250 [3:55:17<95:15:19,  8.65s/it]
                                                         
{'loss': 0.3179, 'grad_norm': 2.5723495483398438, 'learning_rate': 3.999086291609199e-05, 'epoch': 0.39}

  4%|▍         | 1623/41250 [3:55:17<95:15:19,  8.65s/it][2025-04-25 11:53:00,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.76 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:53:00,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.46 | bwd_microstep: 5793.36 | bwd_inner_microstep: 5661.53 | bwd_allreduce_microstep: 131.79 | step_microstep: 17.51
[2025-04-25 11:53:00,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.46 | bwd: 5793.38 | bwd_inner: 5661.53 | bwd_allreduce: 131.81 | step: 17.51

  4%|▍         | 1624/41250 [3:55:25<95:26:16,  8.67s/it]
                                                         
{'loss': 0.3203, 'grad_norm': 2.085782051086426, 'learning_rate': 3.999081539271568e-05, 'epoch': 0.39}

  4%|▍         | 1624/41250 [3:55:25<95:26:16,  8.67s/it][2025-04-25 11:53:09,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:53:09,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.31 | bwd_microstep: 5804.73 | bwd_inner_microstep: 5655.15 | bwd_allreduce_microstep: 149.53 | step_microstep: 17.63
[2025-04-25 11:53:09,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.31 | bwd: 5804.74 | bwd_inner: 5655.15 | bwd_allreduce: 149.55 | step: 17.63

  4%|▍         | 1625/41250 [3:55:34<95:33:17,  8.68s/it]
                                                         
{'loss': 0.2626, 'grad_norm': 2.8035879135131836, 'learning_rate': 3.999076774609992e-05, 'epoch': 0.39}

  4%|▍         | 1625/41250 [3:55:34<95:33:17,  8.68s/it][2025-04-25 11:53:17,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 11:53:17,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.44 | bwd_microstep: 5696.86 | bwd_inner_microstep: 5667.54 | bwd_allreduce_microstep: 29.27 | step_microstep: 18.70
[2025-04-25 11:53:17,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.44 | bwd: 5696.87 | bwd_inner: 5667.54 | bwd_allreduce: 29.29 | step: 18.70

  4%|▍         | 1626/41250 [3:55:43<95:23:16,  8.67s/it]
                                                         
{'loss': 0.2556, 'grad_norm': 1.5261019468307495, 'learning_rate': 3.9990719976245006e-05, 'epoch': 0.39}

  4%|▍         | 1626/41250 [3:55:43<95:23:16,  8.67s/it][2025-04-25 11:53:26,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 11:53:26,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.12 | bwd_microstep: 5773.40 | bwd_inner_microstep: 5661.66 | bwd_allreduce_microstep: 111.70 | step_microstep: 18.38
[2025-04-25 11:53:26,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.12 | bwd: 5773.42 | bwd_inner: 5661.66 | bwd_allreduce: 111.72 | step: 18.38

  4%|▍         | 1627/41250 [3:55:51<95:29:24,  8.68s/it]
                                                         
{'loss': 0.1726, 'grad_norm': 1.5310454368591309, 'learning_rate': 3.999067208315124e-05, 'epoch': 0.39}

  4%|▍         | 1627/41250 [3:55:51<95:29:24,  8.68s/it][2025-04-25 11:53:35,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:53:35,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.01 | bwd_microstep: 5731.20 | bwd_inner_microstep: 5718.63 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.41
[2025-04-25 11:53:35,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.01 | bwd: 5731.21 | bwd_inner: 5718.63 | bwd_allreduce: 12.54 | step: 18.41

  4%|▍         | 1628/41250 [3:56:00<95:28:06,  8.67s/it]
                                                         
{'loss': 0.1818, 'grad_norm': 1.0297139883041382, 'learning_rate': 3.999062406681891e-05, 'epoch': 0.39}

  4%|▍         | 1628/41250 [3:56:00<95:28:06,  8.67s/it][2025-04-25 11:53:44,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:53:44,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.76 | bwd_microstep: 5881.14 | bwd_inner_microstep: 5868.48 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.38
[2025-04-25 11:53:44,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.76 | bwd: 5881.15 | bwd_inner: 5868.48 | bwd_allreduce: 12.64 | step: 18.38

  4%|▍         | 1629/41250 [3:56:09<96:12:26,  8.74s/it]
                                                         
{'loss': 0.155, 'grad_norm': 3.9571826457977295, 'learning_rate': 3.999057592724831e-05, 'epoch': 0.39}

  4%|▍         | 1629/41250 [3:56:09<96:12:26,  8.74s/it][2025-04-25 11:53:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.81 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 11:53:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.25 | bwd_microstep: 5701.69 | bwd_inner_microstep: 5657.44 | bwd_allreduce_microstep: 44.21 | step_microstep: 17.61
[2025-04-25 11:53:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.25 | bwd: 5701.70 | bwd_inner: 5657.44 | bwd_allreduce: 44.22 | step: 17.61

  4%|▍         | 1630/41250 [3:56:18<95:47:18,  8.70s/it]
                                                         
{'loss': 0.2957, 'grad_norm': 2.204514503479004, 'learning_rate': 3.999052766443975e-05, 'epoch': 0.4}

  4%|▍         | 1630/41250 [3:56:18<95:47:18,  8.70s/it][2025-04-25 11:54:01,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 11:54:01,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.77 | bwd_microstep: 5699.88 | bwd_inner_microstep: 5670.24 | bwd_allreduce_microstep: 29.60 | step_microstep: 18.20
[2025-04-25 11:54:01,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.77 | bwd: 5699.89 | bwd_inner: 5670.24 | bwd_allreduce: 29.61 | step: 18.20

  4%|▍         | 1631/41250 [3:56:26<95:30:32,  8.68s/it]
                                                         
{'loss': 0.2005, 'grad_norm': 1.2533000707626343, 'learning_rate': 3.999047927839352e-05, 'epoch': 0.4}

  4%|▍         | 1631/41250 [3:56:26<95:30:32,  8.68s/it][2025-04-25 11:54:10,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 11:54:10,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.70 | bwd_microstep: 5757.50 | bwd_inner_microstep: 5664.84 | bwd_allreduce_microstep: 92.62 | step_microstep: 19.10
[2025-04-25 11:54:10,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.70 | bwd: 5757.51 | bwd_inner: 5664.84 | bwd_allreduce: 92.64 | step: 19.10

  4%|▍         | 1632/41250 [3:56:35<95:29:02,  8.68s/it]
                                                         
{'loss': 0.1607, 'grad_norm': 0.9569926857948303, 'learning_rate': 3.9990430769109905e-05, 'epoch': 0.4}

  4%|▍         | 1632/41250 [3:56:35<95:29:02,  8.68s/it][2025-04-25 11:54:18,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:54:18,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.27 | bwd_microstep: 5759.01 | bwd_inner_microstep: 5712.73 | bwd_allreduce_microstep: 46.24 | step_microstep: 18.65
[2025-04-25 11:54:18,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.27 | bwd: 5759.03 | bwd_inner: 5712.73 | bwd_allreduce: 46.26 | step: 18.65

  4%|▍         | 1633/41250 [3:56:44<95:33:25,  8.68s/it]
                                                         
{'loss': 0.1897, 'grad_norm': 1.5492284297943115, 'learning_rate': 3.9990382136589226e-05, 'epoch': 0.4}

  4%|▍         | 1633/41250 [3:56:44<95:33:25,  8.68s/it][2025-04-25 11:54:27,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 11:54:27,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5712.01 | bwd_inner_microstep: 5699.38 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.64
[2025-04-25 11:54:27,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5712.02 | bwd_inner: 5699.38 | bwd_allreduce: 12.61 | step: 18.64

  4%|▍         | 1634/41250 [3:56:52<95:27:15,  8.67s/it]
                                                         
{'loss': 0.2096, 'grad_norm': 2.215270757675171, 'learning_rate': 3.999033338083177e-05, 'epoch': 0.4}

  4%|▍         | 1634/41250 [3:56:52<95:27:15,  8.67s/it][2025-04-25 11:54:36,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 11:54:36,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.32 | bwd_microstep: 5749.35 | bwd_inner_microstep: 5651.54 | bwd_allreduce_microstep: 97.76 | step_microstep: 18.89
[2025-04-25 11:54:36,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.32 | bwd: 5749.36 | bwd_inner: 5651.54 | bwd_allreduce: 97.78 | step: 18.89

  4%|▍         | 1635/41250 [3:57:01<95:26:20,  8.67s/it]
                                                         
{'loss': 0.2324, 'grad_norm': 1.5125588178634644, 'learning_rate': 3.999028450183784e-05, 'epoch': 0.4}

  4%|▍         | 1635/41250 [3:57:01<95:26:20,  8.67s/it][2025-04-25 11:54:44,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 11:54:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5721.30 | bwd_inner_microstep: 5708.05 | bwd_allreduce_microstep: 13.20 | step_microstep: 18.92
[2025-04-25 11:54:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5721.31 | bwd_inner: 5708.05 | bwd_allreduce: 13.22 | step: 18.93

  4%|▍         | 1636/41250 [3:57:10<95:23:27,  8.67s/it]
                                                         
{'loss': 0.3539, 'grad_norm': 1.9461058378219604, 'learning_rate': 3.9990235499607745e-05, 'epoch': 0.4}

  4%|▍         | 1636/41250 [3:57:10<95:23:27,  8.67s/it][2025-04-25 11:54:53,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 11:54:53,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.69 | bwd_microstep: 5768.18 | bwd_inner_microstep: 5653.05 | bwd_allreduce_microstep: 115.08 | step_microstep: 18.79
[2025-04-25 11:54:53,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.69 | bwd: 5768.19 | bwd_inner: 5653.05 | bwd_allreduce: 115.10 | step: 18.79

  4%|▍         | 1637/41250 [3:57:18<95:25:33,  8.67s/it]
                                                         
{'loss': 0.2312, 'grad_norm': 2.5530905723571777, 'learning_rate': 3.9990186374141775e-05, 'epoch': 0.4}

  4%|▍         | 1637/41250 [3:57:18<95:25:33,  8.67s/it][2025-04-25 11:55:02,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 11:55:02,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5710.45 | bwd_inner_microstep: 5697.22 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.26
[2025-04-25 11:55:02,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5710.46 | bwd_inner: 5697.22 | bwd_allreduce: 13.20 | step: 19.26

  4%|▍         | 1638/41250 [3:57:27<95:20:29,  8.66s/it]
                                                         
{'loss': 0.3029, 'grad_norm': 1.5815430879592896, 'learning_rate': 3.9990137125440236e-05, 'epoch': 0.4}

  4%|▍         | 1638/41250 [3:57:27<95:20:29,  8.66s/it][2025-04-25 11:55:10,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 11:55:10,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.36 | bwd_microstep: 5767.26 | bwd_inner_microstep: 5657.12 | bwd_allreduce_microstep: 110.09 | step_microstep: 19.36
[2025-04-25 11:55:10,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.36 | bwd: 5767.27 | bwd_inner: 5657.12 | bwd_allreduce: 110.11 | step: 19.36

  4%|▍         | 1639/41250 [3:57:36<95:22:22,  8.67s/it]
                                                         
{'loss': 0.1858, 'grad_norm': 0.988877534866333, 'learning_rate': 3.999008775350343e-05, 'epoch': 0.4}

  4%|▍         | 1639/41250 [3:57:36<95:22:22,  8.67s/it][2025-04-25 11:55:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.07 | optimizer_step: 1.01
[2025-04-25 11:55:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.87 | bwd_microstep: 5763.35 | bwd_inner_microstep: 5648.48 | bwd_allreduce_microstep: 114.81 | step_microstep: 19.16
[2025-04-25 11:55:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.87 | bwd: 5763.36 | bwd_inner: 5648.48 | bwd_allreduce: 114.84 | step: 19.16

  4%|▍         | 1640/41250 [3:57:44<95:22:27,  8.67s/it]
                                                         
{'loss': 0.1346, 'grad_norm': 1.2385263442993164, 'learning_rate': 3.999003825833168e-05, 'epoch': 0.4}

  4%|▍         | 1640/41250 [3:57:44<95:22:27,  8.67s/it][2025-04-25 11:55:28,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:55:28,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5759.40 | bwd_inner_microstep: 5658.79 | bwd_allreduce_microstep: 100.57 | step_microstep: 18.80
[2025-04-25 11:55:28,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5759.41 | bwd_inner: 5658.78 | bwd_allreduce: 100.59 | step: 18.80

  4%|▍         | 1641/41250 [3:57:53<95:23:28,  8.67s/it]
                                                         
{'loss': 0.1178, 'grad_norm': 1.335290789604187, 'learning_rate': 3.998998863992526e-05, 'epoch': 0.4}

  4%|▍         | 1641/41250 [3:57:53<95:23:28,  8.67s/it][2025-04-25 11:55:36,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.28 | optimizer_step: 1.00
[2025-04-25 11:55:36,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.04 | bwd_microstep: 5851.48 | bwd_inner_microstep: 5643.53 | bwd_allreduce_microstep: 207.90 | step_microstep: 19.62
[2025-04-25 11:55:36,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.04 | bwd: 5851.49 | bwd_inner: 5643.53 | bwd_allreduce: 207.92 | step: 19.62

  4%|▍         | 1642/41250 [3:58:02<95:41:59,  8.70s/it]
                                                         
{'loss': 0.1259, 'grad_norm': 0.8972151875495911, 'learning_rate': 3.99899388982845e-05, 'epoch': 0.4}

  4%|▍         | 1642/41250 [3:58:02<95:41:59,  8.70s/it][2025-04-25 11:55:45,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.94
[2025-04-25 11:55:45,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5703.07 | bwd_inner_microstep: 5647.61 | bwd_allreduce_microstep: 55.40 | step_microstep: 19.95
[2025-04-25 11:55:45,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5703.09 | bwd_inner: 5647.61 | bwd_allreduce: 55.43 | step: 19.95

  4%|▍         | 1643/41250 [3:58:10<95:24:46,  8.67s/it]
                                                         
{'loss': 0.1209, 'grad_norm': 1.0210131406784058, 'learning_rate': 3.9989889033409695e-05, 'epoch': 0.4}

  4%|▍         | 1643/41250 [3:58:10<95:24:46,  8.67s/it][2025-04-25 11:55:54,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 11:55:54,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.78 | bwd_microstep: 5698.54 | bwd_inner_microstep: 5640.85 | bwd_allreduce_microstep: 57.65 | step_microstep: 19.05
[2025-04-25 11:55:54,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.78 | bwd: 5698.55 | bwd_inner: 5640.85 | bwd_allreduce: 57.66 | step: 19.05

  4%|▍         | 1644/41250 [3:58:19<95:11:13,  8.65s/it]
                                                         
{'loss': 0.1246, 'grad_norm': 1.0707736015319824, 'learning_rate': 3.998983904530115e-05, 'epoch': 0.4}

  4%|▍         | 1644/41250 [3:58:19<95:11:13,  8.65s/it][2025-04-25 11:56:02,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-25 11:56:02,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.01 | bwd_microstep: 5893.14 | bwd_inner_microstep: 5644.02 | bwd_allreduce_microstep: 249.07 | step_microstep: 18.93
[2025-04-25 11:56:02,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.01 | bwd: 5893.16 | bwd_inner: 5644.02 | bwd_allreduce: 249.09 | step: 18.93

  4%|▍         | 1645/41250 [3:58:28<95:40:56,  8.70s/it]
                                                         
{'loss': 0.2755, 'grad_norm': 1.9328038692474365, 'learning_rate': 3.998978893395919e-05, 'epoch': 0.4}

  4%|▍         | 1645/41250 [3:58:28<95:40:56,  8.70s/it][2025-04-25 11:56:11,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 11:56:11,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.12 | bwd_microstep: 5750.77 | bwd_inner_microstep: 5687.09 | bwd_allreduce_microstep: 63.63 | step_microstep: 18.86
[2025-04-25 11:56:11,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.12 | bwd: 5750.78 | bwd_inner: 5687.09 | bwd_allreduce: 63.65 | step: 18.86

  4%|▍         | 1646/41250 [3:58:36<95:37:08,  8.69s/it]
                                                         
{'loss': 0.1009, 'grad_norm': 0.6481313109397888, 'learning_rate': 3.998973869938411e-05, 'epoch': 0.4}

  4%|▍         | 1646/41250 [3:58:36<95:37:08,  8.69s/it][2025-04-25 11:56:20,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 11:56:20,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 5713.34 | bwd_inner_microstep: 5699.96 | bwd_allreduce_microstep: 13.33 | step_microstep: 19.69
[2025-04-25 11:56:20,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 5713.36 | bwd_inner: 5699.96 | bwd_allreduce: 13.36 | step: 19.70

  4%|▍         | 1647/41250 [3:58:45<95:27:44,  8.68s/it]
                                                         
{'loss': 0.0552, 'grad_norm': 0.44300204515457153, 'learning_rate': 3.998968834157622e-05, 'epoch': 0.4}

  4%|▍         | 1647/41250 [3:58:45<95:27:44,  8.68s/it][2025-04-25 11:56:28,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:56:28,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.45 | bwd_microstep: 5780.34 | bwd_inner_microstep: 5637.92 | bwd_allreduce_microstep: 142.38 | step_microstep: 18.88
[2025-04-25 11:56:28,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.45 | bwd: 5780.36 | bwd_inner: 5637.92 | bwd_allreduce: 142.40 | step: 18.88

  4%|▍         | 1648/41250 [3:58:54<95:28:43,  8.68s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 1.758458137512207, 'learning_rate': 3.9989637860535834e-05, 'epoch': 0.4}

  4%|▍         | 1648/41250 [3:58:54<95:28:43,  8.68s/it][2025-04-25 11:56:37,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:56:37,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.78 | bwd_microstep: 6012.98 | bwd_inner_microstep: 5625.19 | bwd_allreduce_microstep: 387.75 | step_microstep: 18.51
[2025-04-25 11:56:37,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.78 | bwd: 6013.00 | bwd_inner: 5625.19 | bwd_allreduce: 387.77 | step: 18.52

  4%|▍         | 1649/41250 [3:59:03<96:14:00,  8.75s/it]
                                                         
{'loss': 0.124, 'grad_norm': 0.8159072399139404, 'learning_rate': 3.9989587256263264e-05, 'epoch': 0.4}

  4%|▍         | 1649/41250 [3:59:03<96:14:00,  8.75s/it][2025-04-25 11:56:46,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:56:46,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.38 | bwd_microstep: 5771.44 | bwd_inner_microstep: 5758.60 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.06
[2025-04-25 11:56:46,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.39 | bwd: 5771.45 | bwd_inner: 5758.60 | bwd_allreduce: 12.81 | step: 19.06

  4%|▍         | 1650/41250 [3:59:11<96:10:47,  8.74s/it]
                                                         
{'loss': 0.3367, 'grad_norm': 1.6547013521194458, 'learning_rate': 3.998953652875882e-05, 'epoch': 0.4}

  4%|▍         | 1650/41250 [3:59:11<96:10:47,  8.74s/it][2025-04-25 11:56:55,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 11:56:55,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.73 | bwd_microstep: 5750.33 | bwd_inner_microstep: 5672.47 | bwd_allreduce_microstep: 77.81 | step_microstep: 19.25
[2025-04-25 11:56:55,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.73 | bwd: 5750.34 | bwd_inner: 5672.47 | bwd_allreduce: 77.83 | step: 19.25

  4%|▍         | 1651/41250 [3:59:20<95:56:53,  8.72s/it]
                                                         
{'loss': 0.0965, 'grad_norm': 0.8862279653549194, 'learning_rate': 3.9989485678022816e-05, 'epoch': 0.4}

  4%|▍         | 1651/41250 [3:59:20<95:56:53,  8.72s/it][2025-04-25 11:57:03,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.22 | optimizer_step: 0.93
[2025-04-25 11:57:03,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.75 | bwd_microstep: 5665.51 | bwd_inner_microstep: 5643.73 | bwd_allreduce_microstep: 21.72 | step_microstep: 19.23
[2025-04-25 11:57:03,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.75 | bwd: 5665.52 | bwd_inner: 5643.73 | bwd_allreduce: 21.75 | step: 19.23

  4%|▍         | 1652/41250 [3:59:29<95:26:42,  8.68s/it]
                                                         
{'loss': 0.2576, 'grad_norm': 2.7827653884887695, 'learning_rate': 3.998943470405556e-05, 'epoch': 0.4}

  4%|▍         | 1652/41250 [3:59:29<95:26:42,  8.68s/it][2025-04-25 11:57:12,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 11:57:12,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.41 | bwd_microstep: 5769.77 | bwd_inner_microstep: 5640.68 | bwd_allreduce_microstep: 129.04 | step_microstep: 19.02
[2025-04-25 11:57:12,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.41 | bwd: 5769.78 | bwd_inner: 5640.68 | bwd_allreduce: 129.06 | step: 19.02

  4%|▍         | 1653/41250 [3:59:37<95:27:44,  8.68s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 3.7175474166870117, 'learning_rate': 3.998938360685737e-05, 'epoch': 0.4}

  4%|▍         | 1653/41250 [3:59:37<95:27:44,  8.68s/it][2025-04-25 11:57:21,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:57:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5694.06 | bwd_inner_microstep: 5681.48 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.84
[2025-04-25 11:57:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5694.07 | bwd_inner: 5681.48 | bwd_allreduce: 12.56 | step: 18.84

  4%|▍         | 1654/41250 [3:59:46<95:17:19,  8.66s/it]
                                                         
{'loss': 0.0325, 'grad_norm': 0.35638922452926636, 'learning_rate': 3.998933238642857e-05, 'epoch': 0.4}

  4%|▍         | 1654/41250 [3:59:46<95:17:19,  8.66s/it][2025-04-25 11:57:29,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:57:29,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.54 | bwd_microstep: 5734.65 | bwd_inner_microstep: 5626.29 | bwd_allreduce_microstep: 108.31 | step_microstep: 18.71
[2025-04-25 11:57:29,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.54 | bwd: 5734.67 | bwd_inner: 5626.29 | bwd_allreduce: 108.33 | step: 18.72

  4%|▍         | 1655/41250 [3:59:55<95:12:11,  8.66s/it]
                                                         
{'loss': 0.1371, 'grad_norm': 0.9546326398849487, 'learning_rate': 3.9989281042769467e-05, 'epoch': 0.4}

  4%|▍         | 1655/41250 [3:59:55<95:12:11,  8.66s/it][2025-04-25 11:57:38,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 11:57:38,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5734.39 | bwd_inner_microstep: 5678.31 | bwd_allreduce_microstep: 56.03 | step_microstep: 18.75
[2025-04-25 11:57:38,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5734.40 | bwd_inner: 5678.31 | bwd_allreduce: 56.05 | step: 18.75

  4%|▍         | 1656/41250 [4:00:03<95:12:38,  8.66s/it]
                                                         
{'loss': 0.3554, 'grad_norm': 2.057478666305542, 'learning_rate': 3.998922957588037e-05, 'epoch': 0.4}

  4%|▍         | 1656/41250 [4:00:03<95:12:38,  8.66s/it][2025-04-25 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.93 | bwd_microstep: 5898.50 | bwd_inner_microstep: 5634.77 | bwd_allreduce_microstep: 263.68 | step_microstep: 18.80
[2025-04-25 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.93 | bwd: 5898.52 | bwd_inner: 5634.77 | bwd_allreduce: 263.70 | step: 18.81

  4%|▍         | 1657/41250 [4:00:12<95:41:37,  8.70s/it]
                                                         
{'loss': 0.0469, 'grad_norm': 0.4197196066379547, 'learning_rate': 3.9989177985761606e-05, 'epoch': 0.4}

  4%|▍         | 1657/41250 [4:00:12<95:41:37,  8.70s/it][2025-04-25 11:57:55,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.30 | optimizer_step: 1.02
[2025-04-25 11:57:55,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.26 | bwd_microstep: 5686.15 | bwd_inner_microstep: 5672.28 | bwd_allreduce_microstep: 13.81 | step_microstep: 20.17
[2025-04-25 11:57:55,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.26 | bwd: 5686.17 | bwd_inner: 5672.28 | bwd_allreduce: 13.84 | step: 20.18

  4%|▍         | 1658/41250 [4:00:21<95:23:28,  8.67s/it]
                                                         
{'loss': 0.2866, 'grad_norm': 2.6705360412597656, 'learning_rate': 3.99891262724135e-05, 'epoch': 0.4}

  4%|▍         | 1658/41250 [4:00:21<95:23:28,  8.67s/it][2025-04-25 11:58:04,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:58:04,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.60 | bwd_microstep: 5676.36 | bwd_inner_microstep: 5634.78 | bwd_allreduce_microstep: 41.53 | step_microstep: 18.79
[2025-04-25 11:58:04,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.60 | bwd: 5676.38 | bwd_inner: 5634.78 | bwd_allreduce: 41.55 | step: 18.80

  4%|▍         | 1659/41250 [4:00:29<95:06:02,  8.65s/it]
                                                         
{'loss': 0.1776, 'grad_norm': 2.0814197063446045, 'learning_rate': 3.998907443583635e-05, 'epoch': 0.4}

  4%|▍         | 1659/41250 [4:00:29<95:06:02,  8.65s/it][2025-04-25 11:58:12,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 0.95
[2025-04-25 11:58:12,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.68 | bwd_microstep: 5693.20 | bwd_inner_microstep: 5624.07 | bwd_allreduce_microstep: 69.08 | step_microstep: 19.86
[2025-04-25 11:58:12,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.69 | bwd: 5693.22 | bwd_inner: 5624.07 | bwd_allreduce: 69.10 | step: 19.86

  4%|▍         | 1660/41250 [4:00:38<94:55:34,  8.63s/it]
                                                         
{'loss': 0.1827, 'grad_norm': 3.7622017860412598, 'learning_rate': 3.99890224760305e-05, 'epoch': 0.4}

  4%|▍         | 1660/41250 [4:00:38<94:55:34,  8.63s/it][2025-04-25 11:58:21,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 11:58:21,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.90 | bwd_microstep: 5751.75 | bwd_inner_microstep: 5680.31 | bwd_allreduce_microstep: 71.39 | step_microstep: 18.85
[2025-04-25 11:58:21,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.90 | bwd: 5751.77 | bwd_inner: 5680.31 | bwd_allreduce: 71.41 | step: 18.85

  4%|▍         | 1661/41250 [4:00:46<95:03:32,  8.64s/it]
                                                         
{'loss': 0.1548, 'grad_norm': 1.4058146476745605, 'learning_rate': 3.998897039299625e-05, 'epoch': 0.4}

  4%|▍         | 1661/41250 [4:00:46<95:03:32,  8.64s/it][2025-04-25 11:58:30,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:58:30,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.63 | bwd_microstep: 5706.63 | bwd_inner_microstep: 5693.90 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.52
[2025-04-25 11:58:30,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.64 | bwd: 5706.64 | bwd_inner: 5693.90 | bwd_allreduce: 12.70 | step: 18.53

  4%|▍         | 1662/41250 [4:00:55<95:02:42,  8.64s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 2.2319822311401367, 'learning_rate': 3.998891818673393e-05, 'epoch': 0.4}

  4%|▍         | 1662/41250 [4:00:55<95:02:42,  8.64s/it][2025-04-25 11:58:38,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-25 11:58:38,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.73 | bwd_microstep: 5679.71 | bwd_inner_microstep: 5635.26 | bwd_allreduce_microstep: 44.40 | step_microstep: 19.47
[2025-04-25 11:58:38,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.73 | bwd: 5679.73 | bwd_inner: 5635.26 | bwd_allreduce: 44.42 | step: 19.47

  4%|▍         | 1663/41250 [4:01:04<94:50:40,  8.63s/it]
                                                         
{'loss': 0.1218, 'grad_norm': 1.4095778465270996, 'learning_rate': 3.9988865857243865e-05, 'epoch': 0.4}

  4%|▍         | 1663/41250 [4:01:04<94:50:40,  8.63s/it][2025-04-25 11:58:47,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 11:58:47,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.13 | bwd_microstep: 5680.14 | bwd_inner_microstep: 5639.83 | bwd_allreduce_microstep: 40.27 | step_microstep: 18.86
[2025-04-25 11:58:47,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.13 | bwd: 5680.16 | bwd_inner: 5639.83 | bwd_allreduce: 40.28 | step: 18.86

  4%|▍         | 1664/41250 [4:01:12<94:43:04,  8.61s/it]
                                                         
{'loss': 0.194, 'grad_norm': 1.9009194374084473, 'learning_rate': 3.998881340452636e-05, 'epoch': 0.4}

  4%|▍         | 1664/41250 [4:01:12<94:43:04,  8.61s/it][2025-04-25 11:58:56,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-25 11:58:56,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.82 | bwd_microstep: 5698.11 | bwd_inner_microstep: 5685.40 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.29
[2025-04-25 11:58:56,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.82 | bwd: 5698.12 | bwd_inner: 5685.40 | bwd_allreduce: 12.68 | step: 19.29

  4%|▍         | 1665/41250 [4:01:21<94:45:44,  8.62s/it]
                                                         
{'loss': 0.4088, 'grad_norm': 6.469527721405029, 'learning_rate': 3.998876082858177e-05, 'epoch': 0.4}

  4%|▍         | 1665/41250 [4:01:21<94:45:44,  8.62s/it][2025-04-25 11:59:04,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 11:59:04,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.86 | bwd_microstep: 5682.37 | bwd_inner_microstep: 5639.40 | bwd_allreduce_microstep: 42.93 | step_microstep: 19.01
[2025-04-25 11:59:04,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.86 | bwd: 5682.39 | bwd_inner: 5639.40 | bwd_allreduce: 42.95 | step: 19.01

  4%|▍         | 1666/41250 [4:01:30<94:40:09,  8.61s/it]
                                                         
{'loss': 0.219, 'grad_norm': 1.2038882970809937, 'learning_rate': 3.998870812941039e-05, 'epoch': 0.4}

  4%|▍         | 1666/41250 [4:01:30<94:40:09,  8.61s/it][2025-04-25 11:59:13,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 11:59:13,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.06 | bwd_microstep: 5737.90 | bwd_inner_microstep: 5672.95 | bwd_allreduce_microstep: 64.90 | step_microstep: 19.59
[2025-04-25 11:59:13,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.06 | bwd: 5737.92 | bwd_inner: 5672.95 | bwd_allreduce: 64.92 | step: 19.59

  4%|▍         | 1667/41250 [4:01:38<94:51:07,  8.63s/it]
                                                         
{'loss': 0.2725, 'grad_norm': 1.8095908164978027, 'learning_rate': 3.998865530701256e-05, 'epoch': 0.4}

  4%|▍         | 1667/41250 [4:01:38<94:51:07,  8.63s/it][2025-04-25 11:59:22,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:59:22,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.99 | bwd_microstep: 5703.33 | bwd_inner_microstep: 5670.70 | bwd_allreduce_microstep: 32.59 | step_microstep: 19.02
[2025-04-25 11:59:22,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.99 | bwd: 5703.34 | bwd_inner: 5670.70 | bwd_allreduce: 32.60 | step: 19.03

  4%|▍         | 1668/41250 [4:01:47<94:56:15,  8.63s/it]
                                                         
{'loss': 0.1031, 'grad_norm': 0.9932811856269836, 'learning_rate': 3.99886023613886e-05, 'epoch': 0.4}

  4%|▍         | 1668/41250 [4:01:47<94:56:15,  8.63s/it][2025-04-25 11:59:30,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 11:59:30,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.58 | bwd_microstep: 5745.57 | bwd_inner_microstep: 5673.81 | bwd_allreduce_microstep: 71.72 | step_microstep: 18.86
[2025-04-25 11:59:30,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.58 | bwd: 5745.59 | bwd_inner: 5673.81 | bwd_allreduce: 71.74 | step: 18.87

  4%|▍         | 1669/41250 [4:01:55<95:02:38,  8.64s/it]
                                                         
{'loss': 0.2169, 'grad_norm': 1.2009011507034302, 'learning_rate': 3.998854929253884e-05, 'epoch': 0.4}

  4%|▍         | 1669/41250 [4:01:55<95:02:38,  8.64s/it][2025-04-25 11:59:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 11:59:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.62 | bwd_microstep: 5747.04 | bwd_inner_microstep: 5647.71 | bwd_allreduce_microstep: 99.28 | step_microstep: 19.21
[2025-04-25 11:59:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.62 | bwd: 5747.06 | bwd_inner: 5647.71 | bwd_allreduce: 99.30 | step: 19.22

  4%|▍         | 1670/41250 [4:02:04<95:04:28,  8.65s/it]
                                                         
{'loss': 0.1896, 'grad_norm': 2.3064920902252197, 'learning_rate': 3.9988496100463604e-05, 'epoch': 0.4}

  4%|▍         | 1670/41250 [4:02:04<95:04:28,  8.65s/it][2025-04-25 11:59:48,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 11:59:48,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.17 | bwd_microstep: 5761.93 | bwd_inner_microstep: 5748.97 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.87
[2025-04-25 11:59:48,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.17 | bwd: 5761.95 | bwd_inner: 5748.97 | bwd_allreduce: 12.94 | step: 18.87

  4%|▍         | 1671/41250 [4:02:13<95:19:00,  8.67s/it]
                                                         
{'loss': 0.2231, 'grad_norm': 1.4140232801437378, 'learning_rate': 3.998844278516322e-05, 'epoch': 0.41}

  4%|▍         | 1671/41250 [4:02:13<95:19:00,  8.67s/it][2025-04-25 11:59:56,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 11:59:56,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.29 | bwd_microstep: 5835.59 | bwd_inner_microstep: 5676.65 | bwd_allreduce_microstep: 158.89 | step_microstep: 18.97
[2025-04-25 11:59:56,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.29 | bwd: 5835.60 | bwd_inner: 5676.65 | bwd_allreduce: 158.91 | step: 18.98

  4%|▍         | 1672/41250 [4:02:22<95:37:33,  8.70s/it]
                                                         
{'loss': 0.2995, 'grad_norm': 1.8728537559509277, 'learning_rate': 3.9988389346638017e-05, 'epoch': 0.41}

  4%|▍         | 1672/41250 [4:02:22<95:37:33,  8.70s/it][2025-04-25 12:00:05,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 12:00:05,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.76 | bwd_microstep: 5698.47 | bwd_inner_microstep: 5652.44 | bwd_allreduce_microstep: 45.97 | step_microstep: 19.24
[2025-04-25 12:00:05,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.76 | bwd: 5698.48 | bwd_inner: 5652.44 | bwd_allreduce: 45.99 | step: 19.24

  4%|▍         | 1673/41250 [4:02:30<95:19:44,  8.67s/it]
                                                         
{'loss': 0.196, 'grad_norm': 1.825141429901123, 'learning_rate': 3.9988335784888336e-05, 'epoch': 0.41}

  4%|▍         | 1673/41250 [4:02:30<95:19:44,  8.67s/it][2025-04-25 12:00:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 12:00:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5744.33 | bwd_inner_microstep: 5649.43 | bwd_allreduce_microstep: 94.86 | step_microstep: 18.73
[2025-04-25 12:00:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5744.35 | bwd_inner: 5649.43 | bwd_allreduce: 94.88 | step: 18.74

  4%|▍         | 1674/41250 [4:02:39<95:15:05,  8.66s/it]
                                                         
{'loss': 0.0925, 'grad_norm': 1.1236300468444824, 'learning_rate': 3.9988282099914484e-05, 'epoch': 0.41}

  4%|▍         | 1674/41250 [4:02:39<95:15:05,  8.66s/it][2025-04-25 12:00:22,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:00:22,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.67 | bwd_microstep: 5737.93 | bwd_inner_microstep: 5692.69 | bwd_allreduce_microstep: 45.20 | step_microstep: 18.72
[2025-04-25 12:00:22,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.67 | bwd: 5737.95 | bwd_inner: 5692.69 | bwd_allreduce: 45.21 | step: 18.72

  4%|▍         | 1675/41250 [4:02:48<95:16:34,  8.67s/it]
                                                         
{'loss': 0.11, 'grad_norm': 1.0415914058685303, 'learning_rate': 3.9988228291716806e-05, 'epoch': 0.41}

  4%|▍         | 1675/41250 [4:02:48<95:16:34,  8.67s/it][2025-04-25 12:00:31,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:00:31,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.01 | bwd_microstep: 5751.83 | bwd_inner_microstep: 5705.23 | bwd_allreduce_microstep: 46.55 | step_microstep: 18.79
[2025-04-25 12:00:31,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.01 | bwd: 5751.84 | bwd_inner: 5705.23 | bwd_allreduce: 46.57 | step: 18.79

  4%|▍         | 1676/41250 [4:02:56<95:20:04,  8.67s/it]
                                                         
{'loss': 0.1142, 'grad_norm': 1.8316290378570557, 'learning_rate': 3.9988174360295635e-05, 'epoch': 0.41}

  4%|▍         | 1676/41250 [4:02:56<95:20:04,  8.67s/it][2025-04-25 12:00:40,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:00:40,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.13 | bwd_microstep: 5705.43 | bwd_inner_microstep: 5692.69 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.40
[2025-04-25 12:00:40,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.13 | bwd: 5705.44 | bwd_inner: 5692.69 | bwd_allreduce: 12.71 | step: 18.40

  4%|▍         | 1677/41250 [4:03:05<95:16:29,  8.67s/it]
                                                         
{'loss': 0.1633, 'grad_norm': 1.799304485321045, 'learning_rate': 3.9988120305651296e-05, 'epoch': 0.41}

  4%|▍         | 1677/41250 [4:03:05<95:16:29,  8.67s/it][2025-04-25 12:00:48,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:00:48,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.71 | bwd_microstep: 5756.69 | bwd_inner_microstep: 5655.22 | bwd_allreduce_microstep: 101.42 | step_microstep: 18.67
[2025-04-25 12:00:48,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.71 | bwd: 5756.70 | bwd_inner: 5655.22 | bwd_allreduce: 101.44 | step: 18.68

  4%|▍         | 1678/41250 [4:03:14<95:19:54,  8.67s/it]
                                                         
{'loss': 0.2854, 'grad_norm': 2.1304008960723877, 'learning_rate': 3.9988066127784134e-05, 'epoch': 0.41}

  4%|▍         | 1678/41250 [4:03:14<95:19:54,  8.67s/it][2025-04-25 12:00:57,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 12:00:57,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.41 | bwd_microstep: 5742.81 | bwd_inner_microstep: 5705.44 | bwd_allreduce_microstep: 37.32 | step_microstep: 18.60
[2025-04-25 12:00:57,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.41 | bwd: 5742.82 | bwd_inner: 5705.44 | bwd_allreduce: 37.34 | step: 18.60

  4%|▍         | 1679/41250 [4:03:22<95:21:41,  8.68s/it]
                                                         
{'loss': 0.0942, 'grad_norm': 0.8112426996231079, 'learning_rate': 3.9988011826694474e-05, 'epoch': 0.41}

  4%|▍         | 1679/41250 [4:03:22<95:21:41,  8.68s/it][2025-04-25 12:01:06,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:01:06,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.66 | bwd_microstep: 5705.19 | bwd_inner_microstep: 5692.54 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.74
[2025-04-25 12:01:06,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.66 | bwd: 5705.20 | bwd_inner: 5692.54 | bwd_allreduce: 12.62 | step: 18.74

  4%|▍         | 1680/41250 [4:03:31<95:15:49,  8.67s/it]
                                                         
{'loss': 0.2935, 'grad_norm': 2.021088123321533, 'learning_rate': 3.998795740238265e-05, 'epoch': 0.41}

  4%|▍         | 1680/41250 [4:03:31<95:15:49,  8.67s/it][2025-04-25 12:01:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:01:14,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5704.10 | bwd_inner_microstep: 5665.38 | bwd_allreduce_microstep: 38.68 | step_microstep: 18.80
[2025-04-25 12:01:14,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5704.11 | bwd_inner: 5665.38 | bwd_allreduce: 38.70 | step: 18.80

  4%|▍         | 1681/41250 [4:03:40<95:09:48,  8.66s/it]
                                                         
{'loss': 0.1864, 'grad_norm': 1.9176158905029297, 'learning_rate': 3.9987902854849e-05, 'epoch': 0.41}

  4%|▍         | 1681/41250 [4:03:40<95:09:48,  8.66s/it][2025-04-25 12:01:23,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:01:23,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.03 | bwd_microstep: 5711.53 | bwd_inner_microstep: 5698.68 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.73
[2025-04-25 12:01:23,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.03 | bwd: 5711.54 | bwd_inner: 5698.68 | bwd_allreduce: 12.82 | step: 18.74

  4%|▍         | 1682/41250 [4:03:48<95:08:16,  8.66s/it]
                                                         
{'loss': 0.1599, 'grad_norm': 1.389796257019043, 'learning_rate': 3.998784818409386e-05, 'epoch': 0.41}

  4%|▍         | 1682/41250 [4:03:48<95:08:16,  8.66s/it][2025-04-25 12:01:32,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:01:32,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2871.92 | bwd_microstep: 5729.53 | bwd_inner_microstep: 5709.81 | bwd_allreduce_microstep: 19.67 | step_microstep: 18.68
[2025-04-25 12:01:32,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2871.92 | bwd: 5729.55 | bwd_inner: 5709.81 | bwd_allreduce: 19.69 | step: 18.68

  4%|▍         | 1683/41250 [4:03:57<95:14:45,  8.67s/it]
                                                         
{'loss': 0.2375, 'grad_norm': 1.006899118423462, 'learning_rate': 3.998779339011757e-05, 'epoch': 0.41}

  4%|▍         | 1683/41250 [4:03:57<95:14:45,  8.67s/it][2025-04-25 12:01:40,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 12:01:40,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.11 | bwd_microstep: 5718.88 | bwd_inner_microstep: 5705.90 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.10
[2025-04-25 12:01:40,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.11 | bwd: 5718.89 | bwd_inner: 5705.90 | bwd_allreduce: 12.95 | step: 19.11

  4%|▍         | 1684/41250 [4:04:06<95:12:52,  8.66s/it]
                                                         
{'loss': 0.2782, 'grad_norm': 1.3501018285751343, 'learning_rate': 3.9987738472920464e-05, 'epoch': 0.41}

  4%|▍         | 1684/41250 [4:04:06<95:12:52,  8.66s/it][2025-04-25 12:01:49,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:01:49,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.64 | bwd_microstep: 5793.47 | bwd_inner_microstep: 5659.44 | bwd_allreduce_microstep: 133.98 | step_microstep: 18.71
[2025-04-25 12:01:49,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.64 | bwd: 5793.48 | bwd_inner: 5659.44 | bwd_allreduce: 134.00 | step: 18.71

  4%|▍         | 1685/41250 [4:04:14<95:21:44,  8.68s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 1.666998028755188, 'learning_rate': 3.998768343250288e-05, 'epoch': 0.41}

  4%|▍         | 1685/41250 [4:04:14<95:21:44,  8.68s/it][2025-04-25 12:01:58,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:01:58,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.81 | bwd_microstep: 5714.11 | bwd_inner_microstep: 5661.75 | bwd_allreduce_microstep: 52.31 | step_microstep: 18.60
[2025-04-25 12:01:58,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.81 | bwd: 5714.12 | bwd_inner: 5661.75 | bwd_allreduce: 52.33 | step: 18.60

  4%|▍         | 1686/41250 [4:04:23<95:11:47,  8.66s/it]
                                                         
{'loss': 0.4044, 'grad_norm': 2.3902788162231445, 'learning_rate': 3.998762826886516e-05, 'epoch': 0.41}

  4%|▍         | 1686/41250 [4:04:23<95:11:47,  8.66s/it][2025-04-25 12:02:06,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:02:06,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.65 | bwd_microstep: 5730.48 | bwd_inner_microstep: 5667.76 | bwd_allreduce_microstep: 62.68 | step_microstep: 18.65
[2025-04-25 12:02:06,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.65 | bwd: 5730.49 | bwd_inner: 5667.76 | bwd_allreduce: 62.69 | step: 18.65

  4%|▍         | 1687/41250 [4:04:32<95:08:13,  8.66s/it]
                                                         
{'loss': 0.0812, 'grad_norm': 0.7122156023979187, 'learning_rate': 3.998757298200764e-05, 'epoch': 0.41}

  4%|▍         | 1687/41250 [4:04:32<95:08:13,  8.66s/it][2025-04-25 12:02:15,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:02:15,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.45 | bwd_microstep: 5739.23 | bwd_inner_microstep: 5709.02 | bwd_allreduce_microstep: 30.16 | step_microstep: 18.93
[2025-04-25 12:02:15,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.45 | bwd: 5739.25 | bwd_inner: 5709.02 | bwd_allreduce: 30.18 | step: 18.93

  4%|▍         | 1688/41250 [4:04:40<95:12:00,  8.66s/it]
                                                         
{'loss': 0.1622, 'grad_norm': 2.591745615005493, 'learning_rate': 3.998751757193067e-05, 'epoch': 0.41}

  4%|▍         | 1688/41250 [4:04:40<95:12:00,  8.66s/it][2025-04-25 12:02:24,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 12:02:24,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.55 | bwd_microstep: 5764.37 | bwd_inner_microstep: 5727.84 | bwd_allreduce_microstep: 36.48 | step_microstep: 19.43
[2025-04-25 12:02:24,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.55 | bwd: 5764.38 | bwd_inner: 5727.84 | bwd_allreduce: 36.50 | step: 19.44

  4%|▍         | 1689/41250 [4:04:49<95:19:58,  8.68s/it]
                                                         
{'loss': 0.3933, 'grad_norm': 2.0593979358673096, 'learning_rate': 3.9987462038634574e-05, 'epoch': 0.41}

  4%|▍         | 1689/41250 [4:04:49<95:19:58,  8.68s/it][2025-04-25 12:02:33,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 12:02:33,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.62 | bwd_microstep: 5905.08 | bwd_inner_microstep: 5892.25 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.16
[2025-04-25 12:02:33,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.62 | bwd: 5905.09 | bwd_inner: 5892.25 | bwd_allreduce: 12.80 | step: 19.17

  4%|▍         | 1690/41250 [4:04:58<96:09:57,  8.75s/it]
                                                         
{'loss': 0.168, 'grad_norm': 1.6420844793319702, 'learning_rate': 3.998740638211972e-05, 'epoch': 0.41}

  4%|▍         | 1690/41250 [4:04:58<96:09:57,  8.75s/it][2025-04-25 12:02:41,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 12:02:41,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5712.14 | bwd_inner_microstep: 5699.39 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.85
[2025-04-25 12:02:41,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5712.15 | bwd_inner: 5699.39 | bwd_allreduce: 12.71 | step: 18.85

  4%|▍         | 1691/41250 [4:05:06<95:49:48,  8.72s/it]
                                                         
{'loss': 0.1863, 'grad_norm': 1.6671583652496338, 'learning_rate': 3.998735060238643e-05, 'epoch': 0.41}

  4%|▍         | 1691/41250 [4:05:06<95:49:48,  8.72s/it][2025-04-25 12:02:50,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:02:50,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5716.75 | bwd_inner_microstep: 5657.85 | bwd_allreduce_microstep: 58.84 | step_microstep: 18.85
[2025-04-25 12:02:50,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5716.76 | bwd_inner: 5657.85 | bwd_allreduce: 58.86 | step: 18.85

  4%|▍         | 1692/41250 [4:05:15<95:31:36,  8.69s/it]
                                                         
{'loss': 0.1878, 'grad_norm': 2.1137776374816895, 'learning_rate': 3.998729469943506e-05, 'epoch': 0.41}

  4%|▍         | 1692/41250 [4:05:15<95:31:36,  8.69s/it][2025-04-25 12:02:58,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 12:02:58,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.97 | bwd_microstep: 5745.05 | bwd_inner_microstep: 5708.51 | bwd_allreduce_microstep: 36.49 | step_microstep: 18.90
[2025-04-25 12:02:58,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.98 | bwd: 5745.06 | bwd_inner: 5708.51 | bwd_allreduce: 36.51 | step: 18.90

  4%|▍         | 1693/41250 [4:05:24<95:29:29,  8.69s/it]
                                                         
{'loss': 0.2169, 'grad_norm': 1.4452670812606812, 'learning_rate': 3.998723867326594e-05, 'epoch': 0.41}

  4%|▍         | 1693/41250 [4:05:24<95:29:29,  8.69s/it][2025-04-25 12:03:07,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:03:07,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.19 | bwd_microstep: 5765.52 | bwd_inner_microstep: 5656.10 | bwd_allreduce_microstep: 109.37 | step_microstep: 18.71
[2025-04-25 12:03:07,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.19 | bwd: 5765.53 | bwd_inner: 5656.10 | bwd_allreduce: 109.39 | step: 18.71

  4%|▍         | 1694/41250 [4:05:32<95:26:18,  8.69s/it]
                                                         
{'loss': 0.1329, 'grad_norm': 1.1192903518676758, 'learning_rate': 3.998718252387943e-05, 'epoch': 0.41}

  4%|▍         | 1694/41250 [4:05:32<95:26:18,  8.69s/it][2025-04-25 12:03:16,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 12:03:16,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.69 | bwd_microstep: 5782.38 | bwd_inner_microstep: 5769.58 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.80
[2025-04-25 12:03:16,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.69 | bwd: 5782.39 | bwd_inner: 5769.58 | bwd_allreduce: 12.78 | step: 18.80

  4%|▍         | 1695/41250 [4:05:41<95:39:07,  8.71s/it]
                                                         
{'loss': 0.2961, 'grad_norm': 2.4796042442321777, 'learning_rate': 3.9987126251275874e-05, 'epoch': 0.41}

  4%|▍         | 1695/41250 [4:05:41<95:39:07,  8.71s/it][2025-04-25 12:03:25,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:03:25,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.38 | bwd_microstep: 5788.97 | bwd_inner_microstep: 5662.86 | bwd_allreduce_microstep: 126.06 | step_microstep: 18.72
[2025-04-25 12:03:25,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.38 | bwd: 5788.99 | bwd_inner: 5662.86 | bwd_allreduce: 126.08 | step: 18.72

  4%|▍         | 1696/41250 [4:05:50<95:38:09,  8.70s/it]
                                                         
{'loss': 0.2517, 'grad_norm': 1.3301070928573608, 'learning_rate': 3.9987069855455616e-05, 'epoch': 0.41}

  4%|▍         | 1696/41250 [4:05:50<95:38:09,  8.70s/it][2025-04-25 12:03:33,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 12:03:33,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.47 | bwd_microstep: 5732.11 | bwd_inner_microstep: 5705.59 | bwd_allreduce_microstep: 26.47 | step_microstep: 18.82
[2025-04-25 12:03:33,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.47 | bwd: 5732.13 | bwd_inner: 5705.59 | bwd_allreduce: 26.49 | step: 18.82

  4%|▍         | 1697/41250 [4:05:59<95:31:52,  8.69s/it]
                                                         
{'loss': 0.3622, 'grad_norm': 2.980397939682007, 'learning_rate': 3.9987013336419e-05, 'epoch': 0.41}

  4%|▍         | 1697/41250 [4:05:59<95:31:52,  8.69s/it][2025-04-25 12:03:42,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 12:03:42,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.70 | bwd_microstep: 5762.56 | bwd_inner_microstep: 5667.42 | bwd_allreduce_microstep: 95.09 | step_microstep: 18.89
[2025-04-25 12:03:42,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.70 | bwd: 5762.57 | bwd_inner: 5667.42 | bwd_allreduce: 95.11 | step: 18.89

  4%|▍         | 1698/41250 [4:06:07<95:29:43,  8.69s/it]
                                                         
{'loss': 0.1002, 'grad_norm': 0.8029299378395081, 'learning_rate': 3.998695669416638e-05, 'epoch': 0.41}

  4%|▍         | 1698/41250 [4:06:07<95:29:43,  8.69s/it][2025-04-25 12:03:51,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 12:03:51,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5828.86 | bwd_inner_microstep: 5707.89 | bwd_allreduce_microstep: 120.91 | step_microstep: 19.12
[2025-04-25 12:03:51,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.78 | bwd: 5828.87 | bwd_inner: 5707.89 | bwd_allreduce: 120.93 | step: 19.12

  4%|▍         | 1699/41250 [4:06:16<95:43:42,  8.71s/it]
                                                         
{'loss': 0.2385, 'grad_norm': 3.3263697624206543, 'learning_rate': 3.9986899928698096e-05, 'epoch': 0.41}

  4%|▍         | 1699/41250 [4:06:16<95:43:42,  8.71s/it][2025-04-25 12:03:59,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 12:03:59,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5696.22 | bwd_inner_microstep: 5683.48 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.80
[2025-04-25 12:03:59,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5696.24 | bwd_inner: 5683.48 | bwd_allreduce: 12.72 | step: 18.80

  4%|▍         | 1700/41250 [4:06:25<95:25:56,  8.69s/it]
                                                         
{'loss': 0.1338, 'grad_norm': 1.1728922128677368, 'learning_rate': 3.998684304001452e-05, 'epoch': 0.41}

  4%|▍         | 1700/41250 [4:06:25<95:25:56,  8.69s/it][2025-04-25 12:04:08,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 12:04:08,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5717.12 | bwd_inner_microstep: 5704.43 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.61
[2025-04-25 12:04:08,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5717.13 | bwd_inner: 5704.43 | bwd_allreduce: 12.66 | step: 18.61

  4%|▍         | 1701/41250 [4:06:33<95:19:13,  8.68s/it]
                                                         
{'loss': 0.1866, 'grad_norm': 1.263051986694336, 'learning_rate': 3.9986786028115974e-05, 'epoch': 0.41}

  4%|▍         | 1701/41250 [4:06:33<95:19:13,  8.68s/it][2025-04-25 12:04:17,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 12:04:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.00 | bwd_microstep: 5769.53 | bwd_inner_microstep: 5656.26 | bwd_allreduce_microstep: 113.23 | step_microstep: 18.88
[2025-04-25 12:04:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.01 | bwd: 5769.54 | bwd_inner: 5656.26 | bwd_allreduce: 113.24 | step: 18.88

  4%|▍         | 1702/41250 [4:06:42<95:19:17,  8.68s/it]
                                                         
{'loss': 0.1398, 'grad_norm': 1.0317684412002563, 'learning_rate': 3.9986728893002826e-05, 'epoch': 0.41}

  4%|▍         | 1702/41250 [4:06:42<95:19:17,  8.68s/it][2025-04-25 12:04:25,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 12:04:25,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5698.05 | bwd_inner_microstep: 5685.26 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.24
[2025-04-25 12:04:25,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5698.06 | bwd_inner: 5685.26 | bwd_allreduce: 12.76 | step: 19.24

  4%|▍         | 1703/41250 [4:06:51<95:09:29,  8.66s/it]
                                                         
{'loss': 0.0667, 'grad_norm': 0.666670560836792, 'learning_rate': 3.998667163467543e-05, 'epoch': 0.41}

  4%|▍         | 1703/41250 [4:06:51<95:09:29,  8.66s/it][2025-04-25 12:04:34,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:04:34,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.94 | bwd_microstep: 5723.95 | bwd_inner_microstep: 5682.28 | bwd_allreduce_microstep: 41.63 | step_microstep: 18.90
[2025-04-25 12:04:34,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.94 | bwd: 5723.96 | bwd_inner: 5682.28 | bwd_allreduce: 41.65 | step: 18.91

  4%|▍         | 1704/41250 [4:06:59<95:07:27,  8.66s/it]
                                                         
{'loss': 0.0821, 'grad_norm': 0.7836632132530212, 'learning_rate': 3.9986614253134135e-05, 'epoch': 0.41}

  4%|▍         | 1704/41250 [4:06:59<95:07:27,  8.66s/it][2025-04-25 12:04:43,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:04:43,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5751.89 | bwd_inner_microstep: 5668.78 | bwd_allreduce_microstep: 83.07 | step_microstep: 18.84
[2025-04-25 12:04:43,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5751.90 | bwd_inner: 5668.78 | bwd_allreduce: 83.09 | step: 18.84

  4%|▍         | 1705/41250 [4:07:08<95:07:48,  8.66s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 0.796453595161438, 'learning_rate': 3.998655674837929e-05, 'epoch': 0.41}

  4%|▍         | 1705/41250 [4:07:08<95:07:48,  8.66s/it][2025-04-25 12:04:51,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 12:04:51,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.55 | bwd_microstep: 5742.53 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 41.66 | step_microstep: 19.02
[2025-04-25 12:04:51,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.55 | bwd: 5742.54 | bwd_inner: 5700.82 | bwd_allreduce: 41.68 | step: 19.02

  4%|▍         | 1706/41250 [4:07:17<95:11:17,  8.67s/it]
                                                         
{'loss': 0.2737, 'grad_norm': 1.8198564052581787, 'learning_rate': 3.9986499120411254e-05, 'epoch': 0.41}

  4%|▍         | 1706/41250 [4:07:17<95:11:17,  8.67s/it][2025-04-25 12:05:00,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 12:05:00,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.98 | bwd_microstep: 5695.87 | bwd_inner_microstep: 5683.05 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.70
[2025-04-25 12:05:00,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.98 | bwd: 5695.88 | bwd_inner: 5683.05 | bwd_allreduce: 12.79 | step: 18.70

  4%|▍         | 1707/41250 [4:07:25<95:02:21,  8.65s/it]
                                                         
{'loss': 0.1671, 'grad_norm': 1.2856764793395996, 'learning_rate': 3.998644136923038e-05, 'epoch': 0.41}

  4%|▍         | 1707/41250 [4:07:25<95:02:21,  8.65s/it][2025-04-25 12:05:09,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:05:09,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.59 | bwd_microstep: 5718.17 | bwd_inner_microstep: 5654.89 | bwd_allreduce_microstep: 63.23 | step_microstep: 18.57
[2025-04-25 12:05:09,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.59 | bwd: 5718.18 | bwd_inner: 5654.89 | bwd_allreduce: 63.25 | step: 18.57

  4%|▍         | 1708/41250 [4:07:34<94:58:58,  8.65s/it]
                                                         
{'loss': 0.3218, 'grad_norm': 2.079216241836548, 'learning_rate': 3.9986383494837035e-05, 'epoch': 0.41}

  4%|▍         | 1708/41250 [4:07:34<94:58:58,  8.65s/it][2025-04-25 12:05:17,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:05:17,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.38 | bwd_microstep: 5763.44 | bwd_inner_microstep: 5652.62 | bwd_allreduce_microstep: 110.77 | step_microstep: 18.49
[2025-04-25 12:05:17,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.38 | bwd: 5763.45 | bwd_inner: 5652.62 | bwd_allreduce: 110.79 | step: 18.49

  4%|▍         | 1709/41250 [4:07:43<95:03:46,  8.65s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 1.8682290315628052, 'learning_rate': 3.998632549723156e-05, 'epoch': 0.41}

  4%|▍         | 1709/41250 [4:07:43<95:03:46,  8.65s/it][2025-04-25 12:05:26,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:05:26,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.32 | bwd_microstep: 5736.41 | bwd_inner_microstep: 5695.41 | bwd_allreduce_microstep: 40.95 | step_microstep: 18.58
[2025-04-25 12:05:26,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.32 | bwd: 5736.42 | bwd_inner: 5695.41 | bwd_allreduce: 40.97 | step: 18.58

  4%|▍         | 1710/41250 [4:07:51<95:06:15,  8.66s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 1.1169272661209106, 'learning_rate': 3.998626737641432e-05, 'epoch': 0.41}

  4%|▍         | 1710/41250 [4:07:51<95:06:15,  8.66s/it][2025-04-25 12:05:35,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:05:35,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5766.16 | bwd_inner_microstep: 5641.88 | bwd_allreduce_microstep: 124.23 | step_microstep: 18.56
[2025-04-25 12:05:35,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5766.17 | bwd_inner: 5641.88 | bwd_allreduce: 124.25 | step: 18.56

  4%|▍         | 1711/41250 [4:08:00<95:08:29,  8.66s/it]
                                                         
{'loss': 0.2281, 'grad_norm': 0.9821304678916931, 'learning_rate': 3.998620913238568e-05, 'epoch': 0.41}

  4%|▍         | 1711/41250 [4:08:00<95:08:29,  8.66s/it][2025-04-25 12:05:43,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 12:05:43,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.27 | bwd_microstep: 5719.72 | bwd_inner_microstep: 5688.97 | bwd_allreduce_microstep: 30.70 | step_microstep: 19.33
[2025-04-25 12:05:43,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.27 | bwd: 5719.74 | bwd_inner: 5688.97 | bwd_allreduce: 30.73 | step: 19.34

  4%|▍         | 1712/41250 [4:08:09<95:05:18,  8.66s/it]
                                                         
{'loss': 0.3401, 'grad_norm': 1.6340969800949097, 'learning_rate': 3.998615076514598e-05, 'epoch': 0.42}

  4%|▍         | 1712/41250 [4:08:09<95:05:18,  8.66s/it][2025-04-25 12:05:52,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 12:05:52,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.96 | bwd_microstep: 5723.13 | bwd_inner_microstep: 5678.24 | bwd_allreduce_microstep: 44.85 | step_microstep: 18.77
[2025-04-25 12:05:52,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.96 | bwd: 5723.15 | bwd_inner: 5678.24 | bwd_allreduce: 44.87 | step: 18.78

  4%|▍         | 1713/41250 [4:08:17<95:03:50,  8.66s/it]
                                                         
{'loss': 0.132, 'grad_norm': 1.5985121726989746, 'learning_rate': 3.99860922746956e-05, 'epoch': 0.42}

  4%|▍         | 1713/41250 [4:08:17<95:03:50,  8.66s/it][2025-04-25 12:06:01,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:06:01,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5736.42 | bwd_inner_microstep: 5638.54 | bwd_allreduce_microstep: 97.84 | step_microstep: 18.61
[2025-04-25 12:06:01,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5736.43 | bwd_inner: 5638.54 | bwd_allreduce: 97.86 | step: 18.61

  4%|▍         | 1714/41250 [4:08:26<95:02:13,  8.65s/it]
                                                         
{'loss': 0.1872, 'grad_norm': 1.6519792079925537, 'learning_rate': 3.998603366103489e-05, 'epoch': 0.42}

  4%|▍         | 1714/41250 [4:08:26<95:02:13,  8.65s/it][2025-04-25 12:06:09,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:06:09,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.44 | bwd_microstep: 5758.37 | bwd_inner_microstep: 5639.79 | bwd_allreduce_microstep: 118.54 | step_microstep: 18.74
[2025-04-25 12:06:09,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.44 | bwd: 5758.38 | bwd_inner: 5639.79 | bwd_allreduce: 118.55 | step: 18.74

  4%|▍         | 1715/41250 [4:08:35<95:05:29,  8.66s/it]
                                                         
{'loss': 0.1704, 'grad_norm': 1.1396491527557373, 'learning_rate': 3.998597492416421e-05, 'epoch': 0.42}

  4%|▍         | 1715/41250 [4:08:35<95:05:29,  8.66s/it][2025-04-25 12:06:18,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:06:18,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5752.42 | bwd_inner_microstep: 5672.76 | bwd_allreduce_microstep: 79.61 | step_microstep: 18.86
[2025-04-25 12:06:18,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5752.43 | bwd_inner: 5672.76 | bwd_allreduce: 79.63 | step: 18.86

  4%|▍         | 1716/41250 [4:08:43<95:09:16,  8.66s/it]
                                                         
{'loss': 0.3892, 'grad_norm': 2.779451370239258, 'learning_rate': 3.998591606408393e-05, 'epoch': 0.42}

  4%|▍         | 1716/41250 [4:08:43<95:09:16,  8.66s/it][2025-04-25 12:06:26,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-25 12:06:26,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.43 | bwd_microstep: 5700.14 | bwd_inner_microstep: 5632.47 | bwd_allreduce_microstep: 67.62 | step_microstep: 18.68
[2025-04-25 12:06:26,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.43 | bwd: 5700.15 | bwd_inner: 5632.47 | bwd_allreduce: 67.64 | step: 18.68

  4%|▍         | 1717/41250 [4:08:52<94:56:13,  8.65s/it]
                                                         
{'loss': 0.2113, 'grad_norm': 1.2385706901550293, 'learning_rate': 3.998585708079441e-05, 'epoch': 0.42}

  4%|▍         | 1717/41250 [4:08:52<94:56:13,  8.65s/it][2025-04-25 12:06:35,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:06:35,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5691.13 | bwd_inner_microstep: 5678.27 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.03
[2025-04-25 12:06:35,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5691.15 | bwd_inner: 5678.27 | bwd_allreduce: 12.83 | step: 19.03

  4%|▍         | 1718/41250 [4:09:00<94:52:02,  8.64s/it]
                                                         
{'loss': 0.0586, 'grad_norm': 1.1278382539749146, 'learning_rate': 3.9985797974296006e-05, 'epoch': 0.42}

  4%|▍         | 1718/41250 [4:09:00<94:52:02,  8.64s/it][2025-04-25 12:06:44,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 12:06:44,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.46 | bwd_microstep: 5742.44 | bwd_inner_microstep: 5637.32 | bwd_allreduce_microstep: 105.08 | step_microstep: 18.51
[2025-04-25 12:06:44,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.46 | bwd: 5742.45 | bwd_inner: 5637.32 | bwd_allreduce: 105.09 | step: 18.51

  4%|▍         | 1719/41250 [4:09:09<94:52:36,  8.64s/it]
                                                         
{'loss': 0.222, 'grad_norm': 3.2540316581726074, 'learning_rate': 3.998573874458909e-05, 'epoch': 0.42}

  4%|▍         | 1719/41250 [4:09:09<94:52:36,  8.64s/it][2025-04-25 12:06:52,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:06:52,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5734.57 | bwd_inner_microstep: 5683.87 | bwd_allreduce_microstep: 50.66 | step_microstep: 18.43
[2025-04-25 12:06:52,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5734.58 | bwd_inner: 5683.87 | bwd_allreduce: 50.68 | step: 18.43

  4%|▍         | 1720/41250 [4:09:18<94:58:06,  8.65s/it]
                                                         
{'loss': 0.2536, 'grad_norm': 2.205854654312134, 'learning_rate': 3.998567939167403e-05, 'epoch': 0.42}

  4%|▍         | 1720/41250 [4:09:18<94:58:06,  8.65s/it][2025-04-25 12:07:01,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:07:01,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.65 | bwd_microstep: 5692.20 | bwd_inner_microstep: 5679.45 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.39
[2025-04-25 12:07:01,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.65 | bwd: 5692.21 | bwd_inner: 5679.45 | bwd_allreduce: 12.72 | step: 18.39

  4%|▍         | 1721/41250 [4:09:26<94:52:07,  8.64s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.4952064752578735, 'learning_rate': 3.998561991555118e-05, 'epoch': 0.42}

  4%|▍         | 1721/41250 [4:09:26<94:52:07,  8.64s/it][2025-04-25 12:07:10,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 12:07:10,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.02 | bwd_microstep: 5717.66 | bwd_inner_microstep: 5705.00 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.65
[2025-04-25 12:07:10,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.02 | bwd: 5717.67 | bwd_inner: 5705.00 | bwd_allreduce: 12.63 | step: 18.66

  4%|▍         | 1722/41250 [4:09:35<94:54:26,  8.64s/it]
                                                         
{'loss': 0.1655, 'grad_norm': 0.8970580697059631, 'learning_rate': 3.998556031622092e-05, 'epoch': 0.42}

  4%|▍         | 1722/41250 [4:09:35<94:54:26,  8.64s/it][2025-04-25 12:07:18,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:07:18,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.99 | bwd_microstep: 5763.08 | bwd_inner_microstep: 5646.39 | bwd_allreduce_microstep: 116.65 | step_microstep: 18.41
[2025-04-25 12:07:18,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.99 | bwd: 5763.09 | bwd_inner: 5646.39 | bwd_allreduce: 116.67 | step: 18.41

  4%|▍         | 1723/41250 [4:09:44<94:59:07,  8.65s/it]
                                                         
{'loss': 0.1188, 'grad_norm': 1.0423614978790283, 'learning_rate': 3.998550059368361e-05, 'epoch': 0.42}

  4%|▍         | 1723/41250 [4:09:44<94:59:07,  8.65s/it][2025-04-25 12:07:27,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:07:27,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.46 | bwd_microstep: 5691.06 | bwd_inner_microstep: 5678.29 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.70
[2025-04-25 12:07:27,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.46 | bwd: 5691.08 | bwd_inner: 5678.29 | bwd_allreduce: 12.75 | step: 18.70

  4%|▍         | 1724/41250 [4:09:52<94:52:40,  8.64s/it]
                                                         
{'loss': 0.3086, 'grad_norm': 1.4824306964874268, 'learning_rate': 3.998544074793962e-05, 'epoch': 0.42}

  4%|▍         | 1724/41250 [4:09:52<94:52:40,  8.64s/it][2025-04-25 12:07:36,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:07:36,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.00 | bwd_microstep: 5691.95 | bwd_inner_microstep: 5674.88 | bwd_allreduce_microstep: 17.03 | step_microstep: 18.54
[2025-04-25 12:07:36,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.00 | bwd: 5691.96 | bwd_inner: 5674.88 | bwd_allreduce: 17.05 | step: 18.54

  4%|▍         | 1725/41250 [4:10:01<94:46:51,  8.63s/it]
                                                         
{'loss': 0.0948, 'grad_norm': 1.270930528640747, 'learning_rate': 3.998538077898932e-05, 'epoch': 0.42}

  4%|▍         | 1725/41250 [4:10:01<94:46:51,  8.63s/it][2025-04-25 12:07:44,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:07:44,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.93 | bwd_microstep: 5692.86 | bwd_inner_microstep: 5651.38 | bwd_allreduce_microstep: 41.43 | step_microstep: 18.49
[2025-04-25 12:07:44,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.93 | bwd: 5692.87 | bwd_inner: 5651.38 | bwd_allreduce: 41.45 | step: 18.49

  4%|▍         | 1726/41250 [4:10:10<94:41:50,  8.63s/it]
                                                         
{'loss': 0.1204, 'grad_norm': 1.086916446685791, 'learning_rate': 3.998532068683308e-05, 'epoch': 0.42}

  4%|▍         | 1726/41250 [4:10:10<94:41:50,  8.63s/it][2025-04-25 12:07:53,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:07:53,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.65 | bwd_microstep: 5766.19 | bwd_inner_microstep: 5650.57 | bwd_allreduce_microstep: 115.57 | step_microstep: 18.69
[2025-04-25 12:07:53,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.65 | bwd: 5766.20 | bwd_inner: 5650.57 | bwd_allreduce: 115.59 | step: 18.70

  4%|▍         | 1727/41250 [4:10:18<94:52:17,  8.64s/it]
                                                         
{'loss': 0.2567, 'grad_norm': 2.742708921432495, 'learning_rate': 3.9985260471471266e-05, 'epoch': 0.42}

  4%|▍         | 1727/41250 [4:10:18<94:52:17,  8.64s/it][2025-04-25 12:08:01,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 12:08:01,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.07 | bwd_microstep: 5702.82 | bwd_inner_microstep: 5648.95 | bwd_allreduce_microstep: 53.83 | step_microstep: 19.24
[2025-04-25 12:08:01,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5702.83 | bwd_inner: 5648.95 | bwd_allreduce: 53.85 | step: 19.24

  4%|▍         | 1728/41250 [4:10:27<94:46:45,  8.63s/it]
                                                         
{'loss': 0.0886, 'grad_norm': 1.560005784034729, 'learning_rate': 3.998520013290425e-05, 'epoch': 0.42}

  4%|▍         | 1728/41250 [4:10:27<94:46:45,  8.63s/it][2025-04-25 12:08:10,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:08:10,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5765.82 | bwd_inner_microstep: 5655.39 | bwd_allreduce_microstep: 110.38 | step_microstep: 18.26
[2025-04-25 12:08:10,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5765.83 | bwd_inner: 5655.39 | bwd_allreduce: 110.40 | step: 18.26

  4%|▍         | 1729/41250 [4:10:35<94:56:45,  8.65s/it]
                                                         
{'loss': 0.2633, 'grad_norm': 1.7781896591186523, 'learning_rate': 3.998513967113241e-05, 'epoch': 0.42}

  4%|▍         | 1729/41250 [4:10:35<94:56:45,  8.65s/it][2025-04-25 12:08:19,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:08:19,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5752.67 | bwd_inner_microstep: 5711.77 | bwd_allreduce_microstep: 40.86 | step_microstep: 18.44
[2025-04-25 12:08:19,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5752.68 | bwd_inner: 5711.77 | bwd_allreduce: 40.87 | step: 18.44

  4%|▍         | 1730/41250 [4:10:44<95:03:49,  8.66s/it]
                                                         
{'loss': 0.1533, 'grad_norm': 1.6270853281021118, 'learning_rate': 3.998507908615611e-05, 'epoch': 0.42}

  4%|▍         | 1730/41250 [4:10:44<95:03:49,  8.66s/it][2025-04-25 12:08:28,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:08:28,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.60 | bwd_microstep: 5802.41 | bwd_inner_microstep: 5654.53 | bwd_allreduce_microstep: 147.83 | step_microstep: 18.36
[2025-04-25 12:08:28,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.60 | bwd: 5802.42 | bwd_inner: 5654.53 | bwd_allreduce: 147.85 | step: 18.36

  4%|▍         | 1731/41250 [4:10:53<95:12:58,  8.67s/it]
                                                         
{'loss': 0.1565, 'grad_norm': 1.1080257892608643, 'learning_rate': 3.9985018377975734e-05, 'epoch': 0.42}

  4%|▍         | 1731/41250 [4:10:53<95:12:58,  8.67s/it][2025-04-25 12:08:36,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:08:36,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.65 | bwd_microstep: 5798.73 | bwd_inner_microstep: 5666.31 | bwd_allreduce_microstep: 132.37 | step_microstep: 18.64
[2025-04-25 12:08:36,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.65 | bwd: 5798.74 | bwd_inner: 5666.31 | bwd_allreduce: 132.39 | step: 18.64

  4%|▍         | 1732/41250 [4:11:02<95:20:30,  8.69s/it]
                                                         
{'loss': 0.1545, 'grad_norm': 1.8991039991378784, 'learning_rate': 3.998495754659165e-05, 'epoch': 0.42}

  4%|▍         | 1732/41250 [4:11:02<95:20:30,  8.69s/it][2025-04-25 12:08:45,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 12:08:45,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.69 | bwd_microstep: 5715.30 | bwd_inner_microstep: 5655.78 | bwd_allreduce_microstep: 59.48 | step_microstep: 18.62
[2025-04-25 12:08:45,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.69 | bwd: 5715.31 | bwd_inner: 5655.78 | bwd_allreduce: 59.50 | step: 18.62

  4%|▍         | 1733/41250 [4:11:10<95:09:37,  8.67s/it]
                                                         
{'loss': 0.1372, 'grad_norm': 1.1224311590194702, 'learning_rate': 3.998489659200423e-05, 'epoch': 0.42}

  4%|▍         | 1733/41250 [4:11:10<95:09:37,  8.67s/it][2025-04-25 12:08:54,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:08:54,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5701.42 | bwd_inner_microstep: 5688.69 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.44
[2025-04-25 12:08:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5701.43 | bwd_inner: 5688.69 | bwd_allreduce: 12.71 | step: 18.45

  4%|▍         | 1734/41250 [4:11:19<95:03:20,  8.66s/it]
                                                         
{'loss': 0.252, 'grad_norm': 2.9557321071624756, 'learning_rate': 3.9984835514213854e-05, 'epoch': 0.42}

  4%|▍         | 1734/41250 [4:11:19<95:03:20,  8.66s/it][2025-04-25 12:09:02,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:09:02,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.25 | bwd_microstep: 5813.55 | bwd_inner_microstep: 5667.88 | bwd_allreduce_microstep: 145.62 | step_microstep: 18.37
[2025-04-25 12:09:02,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.25 | bwd: 5813.56 | bwd_inner: 5667.88 | bwd_allreduce: 145.64 | step: 18.37

  4%|▍         | 1735/41250 [4:11:28<95:17:36,  8.68s/it]
                                                         
{'loss': 0.17, 'grad_norm': 1.5865764617919922, 'learning_rate': 3.99847743132209e-05, 'epoch': 0.42}

  4%|▍         | 1735/41250 [4:11:28<95:17:36,  8.68s/it][2025-04-25 12:09:11,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:09:11,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.80 | bwd_microstep: 5774.11 | bwd_inner_microstep: 5667.79 | bwd_allreduce_microstep: 106.29 | step_microstep: 18.41
[2025-04-25 12:09:11,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.80 | bwd: 5774.13 | bwd_inner: 5667.78 | bwd_allreduce: 106.30 | step: 18.42

  4%|▍         | 1736/41250 [4:11:36<95:21:45,  8.69s/it]
                                                         
{'loss': 0.0662, 'grad_norm': 0.768080472946167, 'learning_rate': 3.998471298902574e-05, 'epoch': 0.42}

  4%|▍         | 1736/41250 [4:11:36<95:21:45,  8.69s/it][2025-04-25 12:09:20,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 12:09:20,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5719.76 | bwd_inner_microstep: 5670.55 | bwd_allreduce_microstep: 49.16 | step_microstep: 19.09
[2025-04-25 12:09:20,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.49 | bwd: 5719.77 | bwd_inner: 5670.55 | bwd_allreduce: 49.18 | step: 19.09

  4%|▍         | 1737/41250 [4:11:45<95:11:07,  8.67s/it]
                                                         
{'loss': 0.2002, 'grad_norm': 1.9593596458435059, 'learning_rate': 3.9984651541628756e-05, 'epoch': 0.42}

  4%|▍         | 1737/41250 [4:11:45<95:11:07,  8.67s/it][2025-04-25 12:09:28,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:09:28,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.28 | bwd_microstep: 5743.51 | bwd_inner_microstep: 5709.87 | bwd_allreduce_microstep: 33.59 | step_microstep: 18.04
[2025-04-25 12:09:28,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.28 | bwd: 5743.52 | bwd_inner: 5709.87 | bwd_allreduce: 33.61 | step: 18.05

  4%|▍         | 1738/41250 [4:11:54<95:16:13,  8.68s/it]
                                                         
{'loss': 0.199, 'grad_norm': 2.1039931774139404, 'learning_rate': 3.998458997103033e-05, 'epoch': 0.42}

  4%|▍         | 1738/41250 [4:11:54<95:16:13,  8.68s/it][2025-04-25 12:09:37,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:09:37,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.89 | bwd_microstep: 5710.94 | bwd_inner_microstep: 5666.77 | bwd_allreduce_microstep: 44.13 | step_microstep: 18.38
[2025-04-25 12:09:37,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.89 | bwd: 5710.96 | bwd_inner: 5666.77 | bwd_allreduce: 44.15 | step: 18.39

  4%|▍         | 1739/41250 [4:12:02<95:06:47,  8.67s/it]
                                                         
{'loss': 0.1335, 'grad_norm': 1.997782588005066, 'learning_rate': 3.9984528277230834e-05, 'epoch': 0.42}

  4%|▍         | 1739/41250 [4:12:02<95:06:47,  8.67s/it][2025-04-25 12:09:46,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:09:46,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.82 | bwd_microstep: 5730.23 | bwd_inner_microstep: 5651.46 | bwd_allreduce_microstep: 78.72 | step_microstep: 18.06
[2025-04-25 12:09:46,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.82 | bwd: 5730.24 | bwd_inner: 5651.46 | bwd_allreduce: 78.74 | step: 18.06

  4%|▍         | 1740/41250 [4:12:11<95:00:22,  8.66s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 0.9570052027702332, 'learning_rate': 3.998446646023065e-05, 'epoch': 0.42}

  4%|▍         | 1740/41250 [4:12:11<95:00:22,  8.66s/it][2025-04-25 12:09:54,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:09:54,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.85 | bwd_microstep: 5748.88 | bwd_inner_microstep: 5712.94 | bwd_allreduce_microstep: 35.89 | step_microstep: 18.80
[2025-04-25 12:09:54,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.85 | bwd: 5748.89 | bwd_inner: 5712.94 | bwd_allreduce: 35.91 | step: 18.80

  4%|▍         | 1741/41250 [4:12:20<95:07:21,  8.67s/it]
                                                         
{'loss': 0.0524, 'grad_norm': 0.7701418399810791, 'learning_rate': 3.998440452003016e-05, 'epoch': 0.42}

  4%|▍         | 1741/41250 [4:12:20<95:07:21,  8.67s/it][2025-04-25 12:10:03,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 12:10:03,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.21 | bwd_microstep: 5707.56 | bwd_inner_microstep: 5694.57 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.08
[2025-04-25 12:10:03,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.21 | bwd: 5707.58 | bwd_inner: 5694.57 | bwd_allreduce: 12.96 | step: 19.09

  4%|▍         | 1742/41250 [4:12:28<95:04:48,  8.66s/it]
                                                         
{'loss': 0.2179, 'grad_norm': 1.7209458351135254, 'learning_rate': 3.9984342456629754e-05, 'epoch': 0.42}

  4%|▍         | 1742/41250 [4:12:28<95:04:48,  8.66s/it][2025-04-25 12:10:12,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 12:10:12,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.07 | bwd_microstep: 5724.38 | bwd_inner_microstep: 5653.46 | bwd_allreduce_microstep: 70.88 | step_microstep: 18.52
[2025-04-25 12:10:12,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.07 | bwd: 5724.39 | bwd_inner: 5653.46 | bwd_allreduce: 70.90 | step: 18.52

  4%|▍         | 1743/41250 [4:12:37<94:59:33,  8.66s/it]
                                                         
{'loss': 0.2252, 'grad_norm': 2.0321104526519775, 'learning_rate': 3.99842802700298e-05, 'epoch': 0.42}

  4%|▍         | 1743/41250 [4:12:37<94:59:33,  8.66s/it][2025-04-25 12:10:20,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:10:20,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5702.61 | bwd_inner_microstep: 5653.39 | bwd_allreduce_microstep: 49.18 | step_microstep: 18.27
[2025-04-25 12:10:20,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5702.63 | bwd_inner: 5653.39 | bwd_allreduce: 49.20 | step: 18.27

  4%|▍         | 1744/41250 [4:12:46<94:50:51,  8.64s/it]
                                                         
{'loss': 0.2574, 'grad_norm': 1.9609593152999878, 'learning_rate': 3.99842179602307e-05, 'epoch': 0.42}

  4%|▍         | 1744/41250 [4:12:46<94:50:51,  8.64s/it][2025-04-25 12:10:29,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:10:29,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5787.55 | bwd_inner_microstep: 5662.97 | bwd_allreduce_microstep: 124.54 | step_microstep: 18.41
[2025-04-25 12:10:29,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5787.56 | bwd_inner: 5662.97 | bwd_allreduce: 124.56 | step: 18.41

  4%|▍         | 1745/41250 [4:12:54<95:02:03,  8.66s/it]
                                                         
{'loss': 0.177, 'grad_norm': 1.4117306470870972, 'learning_rate': 3.998415552723282e-05, 'epoch': 0.42}

  4%|▍         | 1745/41250 [4:12:54<95:02:03,  8.66s/it][2025-04-25 12:10:37,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:10:37,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5690.06 | bwd_inner_microstep: 5660.80 | bwd_allreduce_microstep: 29.22 | step_microstep: 18.49
[2025-04-25 12:10:37,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5690.08 | bwd_inner: 5660.80 | bwd_allreduce: 29.24 | step: 18.50

  4%|▍         | 1746/41250 [4:13:03<94:53:46,  8.65s/it]
                                                         
{'loss': 0.1641, 'grad_norm': 1.4890344142913818, 'learning_rate': 3.9984092971036545e-05, 'epoch': 0.42}

  4%|▍         | 1746/41250 [4:13:03<94:53:46,  8.65s/it][2025-04-25 12:10:46,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:10:46,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5686.87 | bwd_inner_microstep: 5649.06 | bwd_allreduce_microstep: 37.76 | step_microstep: 18.34
[2025-04-25 12:10:46,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.10 | bwd: 5686.88 | bwd_inner: 5649.06 | bwd_allreduce: 37.78 | step: 18.35

  4%|▍         | 1747/41250 [4:13:11<94:43:42,  8.63s/it]
                                                         
{'loss': 0.1377, 'grad_norm': 1.3772329092025757, 'learning_rate': 3.998403029164228e-05, 'epoch': 0.42}

  4%|▍         | 1747/41250 [4:13:11<94:43:42,  8.63s/it][2025-04-25 12:10:55,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 12:10:55,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.02 | bwd_microstep: 5733.00 | bwd_inner_microstep: 5720.24 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.87
[2025-04-25 12:10:55,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.02 | bwd: 5733.01 | bwd_inner: 5720.24 | bwd_allreduce: 12.73 | step: 18.87

  4%|▍         | 1748/41250 [4:13:20<94:54:38,  8.65s/it]
                                                         
{'loss': 0.1527, 'grad_norm': 0.9181888103485107, 'learning_rate': 3.9983967489050395e-05, 'epoch': 0.42}

  4%|▍         | 1748/41250 [4:13:20<94:54:38,  8.65s/it][2025-04-25 12:11:04,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:11:04,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.88 | bwd_microstep: 5803.75 | bwd_inner_microstep: 5790.97 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.61
[2025-04-25 12:11:04,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.88 | bwd: 5803.77 | bwd_inner: 5790.97 | bwd_allreduce: 12.75 | step: 18.61

  4%|▍         | 1749/41250 [4:13:29<95:22:20,  8.69s/it]
                                                         
{'loss': 0.2587, 'grad_norm': 2.146486759185791, 'learning_rate': 3.998390456326127e-05, 'epoch': 0.42}

  4%|▍         | 1749/41250 [4:13:29<95:22:20,  8.69s/it][2025-04-25 12:11:12,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:11:12,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.80 | bwd_microstep: 5753.99 | bwd_inner_microstep: 5660.24 | bwd_allreduce_microstep: 93.70 | step_microstep: 18.49
[2025-04-25 12:11:12,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.80 | bwd: 5754.00 | bwd_inner: 5660.24 | bwd_allreduce: 93.72 | step: 18.49

  4%|▍         | 1750/41250 [4:13:38<95:19:07,  8.69s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 1.4047082662582397, 'learning_rate': 3.9983841514275316e-05, 'epoch': 0.42}

  4%|▍         | 1750/41250 [4:13:38<95:19:07,  8.69s/it][2025-04-25 12:11:21,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:11:21,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5704.46 | bwd_inner_microstep: 5639.69 | bwd_allreduce_microstep: 64.73 | step_microstep: 18.28
[2025-04-25 12:11:21,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5704.48 | bwd_inner: 5639.69 | bwd_allreduce: 64.74 | step: 18.28

  4%|▍         | 1751/41250 [4:13:46<95:06:15,  8.67s/it]
                                                         
{'loss': 0.1926, 'grad_norm': 2.117692708969116, 'learning_rate': 3.998377834209291e-05, 'epoch': 0.42}

  4%|▍         | 1751/41250 [4:13:46<95:06:15,  8.67s/it][2025-04-25 12:11:29,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:11:29,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.09 | bwd_microstep: 5693.61 | bwd_inner_microstep: 5650.63 | bwd_allreduce_microstep: 42.93 | step_microstep: 17.91
[2025-04-25 12:11:29,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.09 | bwd: 5693.62 | bwd_inner: 5650.63 | bwd_allreduce: 42.95 | step: 17.92

  4%|▍         | 1752/41250 [4:13:55<94:54:49,  8.65s/it]
                                                         
{'loss': 0.34, 'grad_norm': 1.9547715187072754, 'learning_rate': 3.998371504671443e-05, 'epoch': 0.42}

  4%|▍         | 1752/41250 [4:13:55<94:54:49,  8.65s/it][2025-04-25 12:11:38,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:11:38,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.80 | bwd_microstep: 5793.19 | bwd_inner_microstep: 5780.26 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.52
[2025-04-25 12:11:38,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.80 | bwd: 5793.20 | bwd_inner: 5780.26 | bwd_allreduce: 12.91 | step: 18.52

  4%|▍         | 1753/41250 [4:14:04<95:19:55,  8.69s/it]
                                                         
{'loss': 0.3094, 'grad_norm': 3.332390069961548, 'learning_rate': 3.998365162814029e-05, 'epoch': 0.42}

  4%|▍         | 1753/41250 [4:14:04<95:19:55,  8.69s/it][2025-04-25 12:11:47,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 12:11:47,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.83 | bwd_microstep: 5834.90 | bwd_inner_microstep: 5685.85 | bwd_allreduce_microstep: 149.01 | step_microstep: 18.80
[2025-04-25 12:11:47,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.83 | bwd: 5834.91 | bwd_inner: 5685.85 | bwd_allreduce: 149.02 | step: 18.80

  4%|▍         | 1754/41250 [4:14:12<95:36:24,  8.71s/it]
                                                         
{'loss': 0.2665, 'grad_norm': 2.5586938858032227, 'learning_rate': 3.998358808637086e-05, 'epoch': 0.43}

  4%|▍         | 1754/41250 [4:14:12<95:36:24,  8.71s/it][2025-04-25 12:11:56,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:11:56,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.81 | bwd_microstep: 5729.60 | bwd_inner_microstep: 5695.18 | bwd_allreduce_microstep: 34.38 | step_microstep: 18.64
[2025-04-25 12:11:56,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.81 | bwd: 5729.61 | bwd_inner: 5695.18 | bwd_allreduce: 34.39 | step: 18.64

  4%|▍         | 1755/41250 [4:14:21<95:25:33,  8.70s/it]
                                                         
{'loss': 0.4543, 'grad_norm': 2.91725492477417, 'learning_rate': 3.998352442140655e-05, 'epoch': 0.43}

  4%|▍         | 1755/41250 [4:14:21<95:25:33,  8.70s/it][2025-04-25 12:12:04,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.98
[2025-04-25 12:12:04,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5714.98 | bwd_inner_microstep: 5702.37 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.39
[2025-04-25 12:12:04,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5714.99 | bwd_inner: 5702.37 | bwd_allreduce: 12.58 | step: 18.39

  4%|▍         | 1756/41250 [4:14:30<95:15:04,  8.68s/it]
                                                         
{'loss': 0.1816, 'grad_norm': 1.4468305110931396, 'learning_rate': 3.998346063324773e-05, 'epoch': 0.43}

  4%|▍         | 1756/41250 [4:14:30<95:15:04,  8.68s/it][2025-04-25 12:12:13,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:12:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.38 | bwd_microstep: 5746.25 | bwd_inner_microstep: 5647.64 | bwd_allreduce_microstep: 98.57 | step_microstep: 18.13
[2025-04-25 12:12:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.38 | bwd: 5746.26 | bwd_inner: 5647.64 | bwd_allreduce: 98.58 | step: 18.13

  4%|▍         | 1757/41250 [4:14:38<95:08:59,  8.67s/it]
                                                         
{'loss': 0.0717, 'grad_norm': 0.883254885673523, 'learning_rate': 3.998339672189482e-05, 'epoch': 0.43}

  4%|▍         | 1757/41250 [4:14:38<95:08:59,  8.67s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x116851c0] moov atom not found
[12:12:17] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4...
[2025-04-25 12:12:22,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:12:22,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.53 | bwd_microstep: 5688.96 | bwd_inner_microstep: 5653.47 | bwd_allreduce_microstep: 35.44 | step_microstep: 18.49
[2025-04-25 12:12:22,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.53 | bwd: 5688.97 | bwd_inner: 5653.47 | bwd_allreduce: 35.46 | step: 18.49

  4%|▍         | 1758/41250 [4:14:47<94:55:29,  8.65s/it]
                                                         
{'loss': 0.4119, 'grad_norm': 2.2075390815734863, 'learning_rate': 3.998333268734818e-05, 'epoch': 0.43}

  4%|▍         | 1758/41250 [4:14:47<94:55:29,  8.65s/it][2025-04-25 12:12:30,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 12:12:30,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.01 | bwd_microstep: 5747.46 | bwd_inner_microstep: 5706.38 | bwd_allreduce_microstep: 41.03 | step_microstep: 19.36
[2025-04-25 12:12:30,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.01 | bwd: 5747.47 | bwd_inner: 5706.38 | bwd_allreduce: 41.05 | step: 19.36

  4%|▍         | 1759/41250 [4:14:56<95:00:44,  8.66s/it]
                                                         
{'loss': 0.587, 'grad_norm': 3.1396234035491943, 'learning_rate': 3.9983268529608244e-05, 'epoch': 0.43}

  4%|▍         | 1759/41250 [4:14:56<95:00:44,  8.66s/it][2025-04-25 12:12:39,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:12:39,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.05 | bwd_microstep: 5693.18 | bwd_inner_microstep: 5680.55 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.40
[2025-04-25 12:12:39,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.05 | bwd: 5693.20 | bwd_inner: 5680.55 | bwd_allreduce: 12.61 | step: 18.41

  4%|▍         | 1760/41250 [4:15:04<94:52:16,  8.65s/it]
                                                         
{'loss': 0.0858, 'grad_norm': 1.1422741413116455, 'learning_rate': 3.998320424867537e-05, 'epoch': 0.43}

  4%|▍         | 1760/41250 [4:15:04<94:52:16,  8.65s/it][2025-04-25 12:12:48,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 12:12:48,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.12 | bwd_microstep: 5726.41 | bwd_inner_microstep: 5688.30 | bwd_allreduce_microstep: 38.07 | step_microstep: 18.37
[2025-04-25 12:12:48,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.12 | bwd: 5726.43 | bwd_inner: 5688.30 | bwd_allreduce: 38.09 | step: 18.38

  4%|▍         | 1761/41250 [4:15:13<94:52:30,  8.65s/it]
                                                         
{'loss': 0.1142, 'grad_norm': 1.0001604557037354, 'learning_rate': 3.998313984454999e-05, 'epoch': 0.43}

  4%|▍         | 1761/41250 [4:15:13<94:52:30,  8.65s/it][2025-04-25 12:12:56,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:12:56,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5721.38 | bwd_inner_microstep: 5700.05 | bwd_allreduce_microstep: 21.29 | step_microstep: 18.21
[2025-04-25 12:12:56,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5721.39 | bwd_inner: 5700.04 | bwd_allreduce: 21.30 | step: 18.22

  4%|▍         | 1762/41250 [4:15:22<94:52:59,  8.65s/it]
                                                         
{'loss': 0.2054, 'grad_norm': 2.5045454502105713, 'learning_rate': 3.998307531723248e-05, 'epoch': 0.43}

  4%|▍         | 1762/41250 [4:15:22<94:52:59,  8.65s/it][2025-04-25 12:13:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:13:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5729.43 | bwd_inner_microstep: 5716.65 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.37
[2025-04-25 12:13:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5729.44 | bwd_inner: 5716.65 | bwd_allreduce: 12.76 | step: 18.38

  4%|▍         | 1763/41250 [4:15:30<94:56:48,  8.66s/it]
                                                         
{'loss': 0.1152, 'grad_norm': 1.1400014162063599, 'learning_rate': 3.998301066672323e-05, 'epoch': 0.43}

  4%|▍         | 1763/41250 [4:15:30<94:56:48,  8.66s/it][2025-04-25 12:13:14,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:13:14,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5998.50 | bwd_inner_microstep: 5689.58 | bwd_allreduce_microstep: 308.87 | step_microstep: 18.16
[2025-04-25 12:13:14,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5998.51 | bwd_inner: 5689.58 | bwd_allreduce: 308.89 | step: 18.16

  4%|▍         | 1764/41250 [4:15:39<95:52:36,  8.74s/it]
                                                         
{'loss': 0.2191, 'grad_norm': 2.360313653945923, 'learning_rate': 3.998294589302266e-05, 'epoch': 0.43}

  4%|▍         | 1764/41250 [4:15:39<95:52:36,  8.74s/it][2025-04-25 12:13:22,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:13:22,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.78 | bwd_microstep: 5744.52 | bwd_inner_microstep: 5635.79 | bwd_allreduce_microstep: 108.69 | step_microstep: 18.07
[2025-04-25 12:13:22,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.79 | bwd: 5744.53 | bwd_inner: 5635.79 | bwd_allreduce: 108.70 | step: 18.07

  4%|▍         | 1765/41250 [4:15:48<95:33:59,  8.71s/it]
                                                         
{'loss': 0.1254, 'grad_norm': 1.0733882188796997, 'learning_rate': 3.998288099613115e-05, 'epoch': 0.43}

  4%|▍         | 1765/41250 [4:15:48<95:33:59,  8.71s/it][2025-04-25 12:13:31,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 1.18
[2025-04-25 12:13:31,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.98 | bwd_microstep: 5691.78 | bwd_inner_microstep: 5679.05 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.23
[2025-04-25 12:13:31,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.98 | bwd: 5691.80 | bwd_inner: 5679.06 | bwd_allreduce: 12.70 | step: 19.23

  4%|▍         | 1766/41250 [4:15:56<95:17:06,  8.69s/it]
                                                         
{'loss': 0.0595, 'grad_norm': 0.5719195604324341, 'learning_rate': 3.998281597604912e-05, 'epoch': 0.43}

  4%|▍         | 1766/41250 [4:15:56<95:17:06,  8.69s/it][2025-04-25 12:13:40,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:13:40,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.23 | bwd_microstep: 5742.79 | bwd_inner_microstep: 5631.21 | bwd_allreduce_microstep: 111.54 | step_microstep: 18.11
[2025-04-25 12:13:40,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.23 | bwd: 5742.80 | bwd_inner: 5631.21 | bwd_allreduce: 111.55 | step: 18.12

  4%|▍         | 1767/41250 [4:16:05<95:08:36,  8.68s/it]
                                                         
{'loss': 0.1324, 'grad_norm': 0.992599368095398, 'learning_rate': 3.998275083277695e-05, 'epoch': 0.43}

  4%|▍         | 1767/41250 [4:16:05<95:08:36,  8.68s/it][2025-04-25 12:13:48,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:13:48,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5680.20 | bwd_inner_microstep: 5667.37 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.35
[2025-04-25 12:13:48,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5680.22 | bwd_inner: 5667.37 | bwd_allreduce: 12.81 | step: 18.35

  4%|▍         | 1768/41250 [4:16:14<94:54:29,  8.65s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 0.7775296568870544, 'learning_rate': 3.998268556631505e-05, 'epoch': 0.43}

  4%|▍         | 1768/41250 [4:16:14<94:54:29,  8.65s/it][2025-04-25 12:13:57,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 12:13:57,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.75 | bwd_microstep: 5721.11 | bwd_inner_microstep: 5677.13 | bwd_allreduce_microstep: 43.93 | step_microstep: 18.29
[2025-04-25 12:13:57,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.75 | bwd: 5721.12 | bwd_inner: 5677.13 | bwd_allreduce: 43.95 | step: 18.29

  4%|▍         | 1769/41250 [4:16:22<94:51:53,  8.65s/it]
                                                         
{'loss': 0.3031, 'grad_norm': 1.8835335969924927, 'learning_rate': 3.998262017666384e-05, 'epoch': 0.43}

  4%|▍         | 1769/41250 [4:16:22<94:51:53,  8.65s/it][2025-04-25 12:14:06,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:14:06,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5695.54 | bwd_inner_microstep: 5682.90 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.96
[2025-04-25 12:14:06,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5695.55 | bwd_inner: 5682.90 | bwd_allreduce: 12.61 | step: 17.96

  4%|▍         | 1770/41250 [4:16:31<94:48:19,  8.64s/it]
                                                         
{'loss': 0.2516, 'grad_norm': 1.8398187160491943, 'learning_rate': 3.998255466382369e-05, 'epoch': 0.43}

  4%|▍         | 1770/41250 [4:16:31<94:48:19,  8.64s/it][2025-04-25 12:14:14,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 12:14:14,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.55 | bwd_microstep: 5683.22 | bwd_inner_microstep: 5670.60 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.88
[2025-04-25 12:14:14,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.55 | bwd: 5683.23 | bwd_inner: 5670.60 | bwd_allreduce: 12.59 | step: 18.89

  4%|▍         | 1771/41250 [4:16:40<94:42:59,  8.64s/it]
                                                         
{'loss': 0.3571, 'grad_norm': 3.1475419998168945, 'learning_rate': 3.998248902779503e-05, 'epoch': 0.43}

  4%|▍         | 1771/41250 [4:16:40<94:42:59,  8.64s/it][2025-04-25 12:14:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:14:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.93 | bwd_microstep: 5760.29 | bwd_inner_microstep: 5649.74 | bwd_allreduce_microstep: 110.51 | step_microstep: 18.22
[2025-04-25 12:14:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.93 | bwd: 5760.30 | bwd_inner: 5649.74 | bwd_allreduce: 110.53 | step: 18.23

  4%|▍         | 1772/41250 [4:16:48<94:48:24,  8.65s/it]
                                                         
{'loss': 0.4128, 'grad_norm': 2.9278159141540527, 'learning_rate': 3.998242326857825e-05, 'epoch': 0.43}

  4%|▍         | 1772/41250 [4:16:48<94:48:24,  8.65s/it][2025-04-25 12:14:32,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:14:32,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.99 | bwd_microstep: 5691.20 | bwd_inner_microstep: 5678.42 | bwd_allreduce_microstep: 12.73 | step_microstep: 17.84
[2025-04-25 12:14:32,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.99 | bwd: 5691.21 | bwd_inner: 5678.42 | bwd_allreduce: 12.75 | step: 17.84

  4%|▍         | 1773/41250 [4:16:57<94:43:12,  8.64s/it]
                                                         
{'loss': 0.1159, 'grad_norm': 1.1090995073318481, 'learning_rate': 3.998235738617377e-05, 'epoch': 0.43}

  4%|▍         | 1773/41250 [4:16:57<94:43:12,  8.64s/it][2025-04-25 12:14:40,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:14:40,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.93 | bwd_microstep: 5671.49 | bwd_inner_microstep: 5638.40 | bwd_allreduce_microstep: 33.04 | step_microstep: 18.42
[2025-04-25 12:14:40,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.93 | bwd: 5671.50 | bwd_inner: 5638.40 | bwd_allreduce: 33.06 | step: 18.43

  4%|▍         | 1774/41250 [4:17:05<94:33:51,  8.62s/it]
                                                         
{'loss': 0.1699, 'grad_norm': 1.8540624380111694, 'learning_rate': 3.998229138058197e-05, 'epoch': 0.43}

  4%|▍         | 1774/41250 [4:17:05<94:33:51,  8.62s/it][2025-04-25 12:14:49,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 12:14:49,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.14 | bwd_microstep: 5746.26 | bwd_inner_microstep: 5636.09 | bwd_allreduce_microstep: 110.13 | step_microstep: 18.84
[2025-04-25 12:14:49,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.14 | bwd: 5746.27 | bwd_inner: 5636.09 | bwd_allreduce: 110.14 | step: 18.85

  4%|▍         | 1775/41250 [4:17:14<94:38:09,  8.63s/it]
                                                         
{'loss': 0.3418, 'grad_norm': 1.5619525909423828, 'learning_rate': 3.998222525180329e-05, 'epoch': 0.43}

  4%|▍         | 1775/41250 [4:17:14<94:38:09,  8.63s/it][2025-04-25 12:14:57,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 12:14:57,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.67 | bwd_microstep: 5738.11 | bwd_inner_microstep: 5632.73 | bwd_allreduce_microstep: 105.33 | step_microstep: 18.31
[2025-04-25 12:14:57,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.67 | bwd: 5738.12 | bwd_inner: 5632.73 | bwd_allreduce: 105.35 | step: 18.32

  4%|▍         | 1776/41250 [4:17:23<94:40:44,  8.63s/it]
                                                         
{'loss': 0.2584, 'grad_norm': 1.3720626831054688, 'learning_rate': 3.9982158999838114e-05, 'epoch': 0.43}

  4%|▍         | 1776/41250 [4:17:23<94:40:44,  8.63s/it][2025-04-25 12:15:06,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.92 | optimizer_step: 0.89
[2025-04-25 12:15:06,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.28 | bwd_microstep: 5729.74 | bwd_inner_microstep: 5704.81 | bwd_allreduce_microstep: 24.89 | step_microstep: 18.48
[2025-04-25 12:15:06,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.28 | bwd: 5729.75 | bwd_inner: 5704.81 | bwd_allreduce: 24.91 | step: 18.49

  4%|▍         | 1777/41250 [4:17:31<94:46:33,  8.64s/it]
                                                         
{'loss': 0.3407, 'grad_norm': 1.277040958404541, 'learning_rate': 3.998209262468686e-05, 'epoch': 0.43}

  4%|▍         | 1777/41250 [4:17:31<94:46:33,  8.64s/it][2025-04-25 12:15:15,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-25 12:15:15,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.34 | bwd_microstep: 5708.39 | bwd_inner_microstep: 5690.43 | bwd_allreduce_microstep: 17.90 | step_microstep: 19.27
[2025-04-25 12:15:15,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.34 | bwd: 5708.40 | bwd_inner: 5690.43 | bwd_allreduce: 17.92 | step: 19.27

  4%|▍         | 1778/41250 [4:17:40<94:47:12,  8.64s/it]
                                                         
{'loss': 0.0508, 'grad_norm': 0.5221976041793823, 'learning_rate': 3.9982026126349935e-05, 'epoch': 0.43}

  4%|▍         | 1778/41250 [4:17:40<94:47:12,  8.64s/it][2025-04-25 12:15:23,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:15:23,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5706.79 | bwd_inner_microstep: 5691.91 | bwd_allreduce_microstep: 14.83 | step_microstep: 18.16
[2025-04-25 12:15:23,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5706.80 | bwd_inner: 5691.91 | bwd_allreduce: 14.85 | step: 18.16

  4%|▍         | 1779/41250 [4:17:49<94:46:28,  8.64s/it]
                                                         
{'loss': 0.166, 'grad_norm': 1.6689057350158691, 'learning_rate': 3.998195950482775e-05, 'epoch': 0.43}

  4%|▍         | 1779/41250 [4:17:49<94:46:28,  8.64s/it][2025-04-25 12:15:32,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:15:32,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.04 | bwd_microstep: 5693.90 | bwd_inner_microstep: 5680.77 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.46
[2025-04-25 12:15:32,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.04 | bwd: 5693.92 | bwd_inner: 5680.77 | bwd_allreduce: 13.11 | step: 18.46

  4%|▍         | 1780/41250 [4:17:57<94:40:06,  8.63s/it]
                                                         
{'loss': 0.1224, 'grad_norm': 1.3992600440979004, 'learning_rate': 3.9981892760120715e-05, 'epoch': 0.43}

  4%|▍         | 1780/41250 [4:17:57<94:40:06,  8.63s/it][2025-04-25 12:15:41,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-25 12:15:41,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.40 | bwd_microstep: 5683.18 | bwd_inner_microstep: 5628.57 | bwd_allreduce_microstep: 54.56 | step_microstep: 19.23
[2025-04-25 12:15:41,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.40 | bwd: 5683.19 | bwd_inner: 5628.57 | bwd_allreduce: 54.58 | step: 19.23

  4%|▍         | 1781/41250 [4:18:06<94:33:31,  8.62s/it]
                                                         
{'loss': 0.1727, 'grad_norm': 1.5723297595977783, 'learning_rate': 3.998182589222924e-05, 'epoch': 0.43}

  4%|▍         | 1781/41250 [4:18:06<94:33:31,  8.62s/it][2025-04-25 12:15:49,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 12:15:49,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.65 | bwd_microstep: 5762.65 | bwd_inner_microstep: 5749.78 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.94
[2025-04-25 12:15:49,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.65 | bwd: 5762.67 | bwd_inner: 5749.78 | bwd_allreduce: 12.85 | step: 18.94

  4%|▍         | 1782/41250 [4:18:15<94:55:46,  8.66s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.4239929914474487, 'learning_rate': 3.998175890115375e-05, 'epoch': 0.43}

  4%|▍         | 1782/41250 [4:18:15<94:55:46,  8.66s/it][2025-04-25 12:15:58,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:15:58,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.43 | bwd_microstep: 5715.44 | bwd_inner_microstep: 5692.29 | bwd_allreduce_microstep: 23.10 | step_microstep: 18.16
[2025-04-25 12:15:58,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.43 | bwd: 5715.45 | bwd_inner: 5692.29 | bwd_allreduce: 23.12 | step: 18.17

  4%|▍         | 1783/41250 [4:18:23<94:54:12,  8.66s/it]
                                                         
{'loss': 0.2025, 'grad_norm': 1.8080236911773682, 'learning_rate': 3.998169178689463e-05, 'epoch': 0.43}

  4%|▍         | 1783/41250 [4:18:23<94:54:12,  8.66s/it][2025-04-25 12:16:07,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:16:07,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5689.82 | bwd_inner_microstep: 5636.84 | bwd_allreduce_microstep: 52.93 | step_microstep: 18.67
[2025-04-25 12:16:07,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5689.83 | bwd_inner: 5636.84 | bwd_allreduce: 52.95 | step: 18.68

  4%|▍         | 1784/41250 [4:18:32<94:42:52,  8.64s/it]
                                                         
{'loss': 0.2292, 'grad_norm': 1.732201337814331, 'learning_rate': 3.998162454945232e-05, 'epoch': 0.43}

  4%|▍         | 1784/41250 [4:18:32<94:42:52,  8.64s/it][2025-04-25 12:16:15,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:16:15,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.65 | bwd_microstep: 5691.84 | bwd_inner_microstep: 5679.24 | bwd_allreduce_microstep: 12.55 | step_microstep: 19.10
[2025-04-25 12:16:15,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.65 | bwd: 5691.85 | bwd_inner: 5679.24 | bwd_allreduce: 12.57 | step: 19.09

  4%|▍         | 1785/41250 [4:18:41<94:40:25,  8.64s/it]
                                                         
{'loss': 0.1419, 'grad_norm': 1.548810601234436, 'learning_rate': 3.998155718882722e-05, 'epoch': 0.43}

  4%|▍         | 1785/41250 [4:18:41<94:40:25,  8.64s/it][2025-04-25 12:16:24,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 12:16:24,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.22 | bwd_microstep: 5671.80 | bwd_inner_microstep: 5632.88 | bwd_allreduce_microstep: 38.87 | step_microstep: 19.04
[2025-04-25 12:16:24,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.22 | bwd: 5671.81 | bwd_inner: 5632.88 | bwd_allreduce: 38.89 | step: 19.04

  4%|▍         | 1786/41250 [4:18:49<94:32:21,  8.62s/it]
                                                         
{'loss': 0.1134, 'grad_norm': 1.2098183631896973, 'learning_rate': 3.998148970501975e-05, 'epoch': 0.43}

  4%|▍         | 1786/41250 [4:18:49<94:32:21,  8.62s/it][2025-04-25 12:16:32,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:16:32,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.10 | bwd_microstep: 5687.85 | bwd_inner_microstep: 5631.19 | bwd_allreduce_microstep: 56.62 | step_microstep: 18.67
[2025-04-25 12:16:32,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.10 | bwd: 5687.87 | bwd_inner: 5631.19 | bwd_allreduce: 56.64 | step: 18.67

  4%|▍         | 1787/41250 [4:18:58<94:25:49,  8.61s/it]
                                                         
{'loss': 0.5343, 'grad_norm': 2.3160018920898438, 'learning_rate': 3.9981422098030325e-05, 'epoch': 0.43}

  4%|▍         | 1787/41250 [4:18:58<94:25:49,  8.61s/it][2025-04-25 12:16:41,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:16:41,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.91 | bwd_microstep: 5760.05 | bwd_inner_microstep: 5642.07 | bwd_allreduce_microstep: 117.93 | step_microstep: 18.41
[2025-04-25 12:16:41,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.91 | bwd: 5760.06 | bwd_inner: 5642.07 | bwd_allreduce: 117.95 | step: 18.41

  4%|▍         | 1788/41250 [4:19:06<94:39:36,  8.64s/it]
                                                         
{'loss': 0.1626, 'grad_norm': 1.7431782484054565, 'learning_rate': 3.9981354367859366e-05, 'epoch': 0.43}

  4%|▍         | 1788/41250 [4:19:06<94:39:36,  8.64s/it][2025-04-25 12:16:50,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:16:50,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.67 | bwd_microstep: 5776.61 | bwd_inner_microstep: 5763.85 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.80
[2025-04-25 12:16:50,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.67 | bwd: 5776.62 | bwd_inner: 5763.85 | bwd_allreduce: 12.73 | step: 18.80

  4%|▍         | 1789/41250 [4:19:15<95:03:27,  8.67s/it]
                                                         
{'loss': 0.1048, 'grad_norm': 0.7461344599723816, 'learning_rate': 3.9981286514507285e-05, 'epoch': 0.43}

  4%|▍         | 1789/41250 [4:19:15<95:03:27,  8.67s/it][2025-04-25 12:16:58,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:16:58,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.16 | bwd_microstep: 5701.25 | bwd_inner_microstep: 5646.84 | bwd_allreduce_microstep: 54.37 | step_microstep: 18.37
[2025-04-25 12:16:58,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.16 | bwd: 5701.27 | bwd_inner: 5646.84 | bwd_allreduce: 54.38 | step: 18.38

  4%|▍         | 1790/41250 [4:19:24<94:52:24,  8.66s/it]
                                                         
{'loss': 0.0921, 'grad_norm': 1.0858123302459717, 'learning_rate': 3.998121853797451e-05, 'epoch': 0.43}

  4%|▍         | 1790/41250 [4:19:24<94:52:24,  8.66s/it][2025-04-25 12:17:07,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:17:07,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5702.20 | bwd_inner_microstep: 5689.99 | bwd_allreduce_microstep: 12.18 | step_microstep: 18.35
[2025-04-25 12:17:07,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5702.22 | bwd_inner: 5689.99 | bwd_allreduce: 12.19 | step: 18.35

  4%|▍         | 1791/41250 [4:19:32<94:48:01,  8.65s/it]
                                                         
{'loss': 0.1377, 'grad_norm': 0.78407222032547, 'learning_rate': 3.998115043826144e-05, 'epoch': 0.43}

  4%|▍         | 1791/41250 [4:19:32<94:48:01,  8.65s/it][2025-04-25 12:17:16,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 12:17:16,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5707.31 | bwd_inner_microstep: 5651.23 | bwd_allreduce_microstep: 56.03 | step_microstep: 19.15
[2025-04-25 12:17:16,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5707.32 | bwd_inner: 5651.23 | bwd_allreduce: 56.05 | step: 19.16

  4%|▍         | 1792/41250 [4:19:41<94:42:47,  8.64s/it]
                                                         
{'loss': 0.3611, 'grad_norm': 2.018559217453003, 'learning_rate': 3.998108221536852e-05, 'epoch': 0.43}

  4%|▍         | 1792/41250 [4:19:41<94:42:47,  8.64s/it][2025-04-25 12:17:24,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 12:17:24,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.65 | bwd_microstep: 5733.44 | bwd_inner_microstep: 5720.74 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.85
[2025-04-25 12:17:24,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.65 | bwd: 5733.45 | bwd_inner: 5720.74 | bwd_allreduce: 12.67 | step: 18.84

  4%|▍         | 1793/41250 [4:19:50<94:50:24,  8.65s/it]
                                                         
{'loss': 0.2088, 'grad_norm': 2.106872320175171, 'learning_rate': 3.998101386929615e-05, 'epoch': 0.43}

  4%|▍         | 1793/41250 [4:19:50<94:50:24,  8.65s/it][2025-04-25 12:17:33,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.29 | optimizer_step: 1.03
[2025-04-25 12:17:33,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5722.19 | bwd_inner_microstep: 5708.07 | bwd_allreduce_microstep: 14.06 | step_microstep: 19.90
[2025-04-25 12:17:33,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5722.21 | bwd_inner: 5708.07 | bwd_allreduce: 14.09 | step: 19.90

  4%|▍         | 1794/41250 [4:19:58<94:52:06,  8.66s/it]
                                                         
{'loss': 0.1129, 'grad_norm': 1.3873789310455322, 'learning_rate': 3.998094540004477e-05, 'epoch': 0.43}

  4%|▍         | 1794/41250 [4:19:58<94:52:06,  8.66s/it][2025-04-25 12:17:42,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 12:17:42,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.85 | bwd_microstep: 5874.88 | bwd_inner_microstep: 5861.72 | bwd_allreduce_microstep: 13.12 | step_microstep: 18.95
[2025-04-25 12:17:42,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.85 | bwd: 5874.90 | bwd_inner: 5861.71 | bwd_allreduce: 13.14 | step: 18.96

  4%|▍         | 1795/41250 [4:20:07<95:38:57,  8.73s/it]
                                                         
{'loss': 0.2301, 'grad_norm': 1.4704076051712036, 'learning_rate': 3.998087680761478e-05, 'epoch': 0.44}

  4%|▍         | 1795/41250 [4:20:07<95:38:57,  8.73s/it][2025-04-25 12:17:51,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:17:51,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.29 | bwd_microstep: 5851.54 | bwd_inner_microstep: 5714.10 | bwd_allreduce_microstep: 137.38 | step_microstep: 18.78
[2025-04-25 12:17:51,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.29 | bwd: 5851.55 | bwd_inner: 5714.10 | bwd_allreduce: 137.40 | step: 18.78

  4%|▍         | 1796/41250 [4:20:16<95:53:03,  8.75s/it]
                                                         
{'loss': 0.1262, 'grad_norm': 0.7945087552070618, 'learning_rate': 3.998080809200662e-05, 'epoch': 0.44}

  4%|▍         | 1796/41250 [4:20:16<95:53:03,  8.75s/it][2025-04-25 12:17:59,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:17:59,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.49 | bwd_microstep: 5774.21 | bwd_inner_microstep: 5697.00 | bwd_allreduce_microstep: 77.17 | step_microstep: 18.54
[2025-04-25 12:17:59,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.49 | bwd: 5774.23 | bwd_inner: 5697.00 | bwd_allreduce: 77.18 | step: 18.54

  4%|▍         | 1797/41250 [4:20:25<95:46:11,  8.74s/it]
                                                         
{'loss': 0.1799, 'grad_norm': 2.6839911937713623, 'learning_rate': 3.9980739253220704e-05, 'epoch': 0.44}

  4%|▍         | 1797/41250 [4:20:25<95:46:11,  8.74s/it][2025-04-25 12:18:08,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:18:08,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.27 | bwd_microstep: 5694.50 | bwd_inner_microstep: 5660.32 | bwd_allreduce_microstep: 34.14 | step_microstep: 18.73
[2025-04-25 12:18:08,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.27 | bwd: 5694.51 | bwd_inner: 5660.32 | bwd_allreduce: 34.15 | step: 18.74

  4%|▍         | 1798/41250 [4:20:33<95:21:40,  8.70s/it]
                                                         
{'loss': 0.2816, 'grad_norm': 1.252672791481018, 'learning_rate': 3.998067029125746e-05, 'epoch': 0.44}

  4%|▍         | 1798/41250 [4:20:33<95:21:40,  8.70s/it][2025-04-25 12:18:17,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 12:18:17,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.26 | bwd_microstep: 5801.91 | bwd_inner_microstep: 5657.11 | bwd_allreduce_microstep: 144.75 | step_microstep: 18.91
[2025-04-25 12:18:17,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5801.92 | bwd_inner: 5657.11 | bwd_allreduce: 144.78 | step: 18.91

  4%|▍         | 1799/41250 [4:20:42<95:25:31,  8.71s/it]
                                                         
{'loss': 0.1368, 'grad_norm': 1.2835451364517212, 'learning_rate': 3.998060120611732e-05, 'epoch': 0.44}

  4%|▍         | 1799/41250 [4:20:42<95:25:31,  8.71s/it][2025-04-25 12:18:25,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 12:18:25,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.42 | bwd_microstep: 5726.60 | bwd_inner_microstep: 5713.86 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.87
[2025-04-25 12:18:25,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.42 | bwd: 5726.61 | bwd_inner: 5713.86 | bwd_allreduce: 12.71 | step: 18.87

  4%|▍         | 1800/41250 [4:20:51<95:19:40,  8.70s/it]
                                                         
{'loss': 0.1356, 'grad_norm': 2.348191976547241, 'learning_rate': 3.9980531997800705e-05, 'epoch': 0.44}

  4%|▍         | 1800/41250 [4:20:51<95:19:40,  8.70s/it][2025-04-25 12:18:34,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 12:18:34,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.70 | bwd_microstep: 5737.04 | bwd_inner_microstep: 5724.28 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.47
[2025-04-25 12:18:34,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.70 | bwd: 5737.05 | bwd_inner: 5724.28 | bwd_allreduce: 12.73 | step: 18.47

  4%|▍         | 1801/41250 [4:20:59<95:17:30,  8.70s/it]
                                                         
{'loss': 0.1493, 'grad_norm': 0.8744640946388245, 'learning_rate': 3.9980462666308036e-05, 'epoch': 0.44}

  4%|▍         | 1801/41250 [4:20:59<95:17:30,  8.70s/it][2025-04-25 12:18:43,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:18:43,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.90 | bwd_microstep: 5763.91 | bwd_inner_microstep: 5666.78 | bwd_allreduce_microstep: 97.09 | step_microstep: 18.51
[2025-04-25 12:18:43,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.90 | bwd: 5763.93 | bwd_inner: 5666.78 | bwd_allreduce: 97.11 | step: 18.51

  4%|▍         | 1802/41250 [4:21:08<95:15:33,  8.69s/it]
                                                         
{'loss': 0.292, 'grad_norm': 1.707207202911377, 'learning_rate': 3.998039321163974e-05, 'epoch': 0.44}

  4%|▍         | 1802/41250 [4:21:08<95:15:33,  8.69s/it][2025-04-25 12:18:52,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:18:52,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.87 | bwd_microstep: 5903.41 | bwd_inner_microstep: 5662.96 | bwd_allreduce_microstep: 240.40 | step_microstep: 18.22
[2025-04-25 12:18:52,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.87 | bwd: 5903.43 | bwd_inner: 5662.96 | bwd_allreduce: 240.42 | step: 18.22

  4%|▍         | 1803/41250 [4:21:17<95:42:14,  8.73s/it]
                                                         
{'loss': 0.231, 'grad_norm': 2.136040210723877, 'learning_rate': 3.9980323633796255e-05, 'epoch': 0.44}

  4%|▍         | 1803/41250 [4:21:17<95:42:14,  8.73s/it][2025-04-25 12:19:00,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 12:19:00,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.74 | bwd_microstep: 5772.07 | bwd_inner_microstep: 5661.08 | bwd_allreduce_microstep: 110.94 | step_microstep: 18.98
[2025-04-25 12:19:00,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.74 | bwd: 5772.08 | bwd_inner: 5661.08 | bwd_allreduce: 110.96 | step: 18.98

  4%|▍         | 1804/41250 [4:21:26<95:33:52,  8.72s/it]
                                                         
{'loss': 0.2301, 'grad_norm': 2.3376286029815674, 'learning_rate': 3.9980253932778e-05, 'epoch': 0.44}

  4%|▍         | 1804/41250 [4:21:26<95:33:52,  8.72s/it][2025-04-25 12:19:09,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 12:19:09,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.79 | bwd_microstep: 5776.02 | bwd_inner_microstep: 5718.82 | bwd_allreduce_microstep: 57.15 | step_microstep: 18.87
[2025-04-25 12:19:09,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.79 | bwd: 5776.03 | bwd_inner: 5718.82 | bwd_allreduce: 57.17 | step: 18.87

  4%|▍         | 1805/41250 [4:21:34<95:32:13,  8.72s/it]
                                                         
{'loss': 0.2725, 'grad_norm': 5.561544895172119, 'learning_rate': 3.998018410858542e-05, 'epoch': 0.44}

  4%|▍         | 1805/41250 [4:21:34<95:32:13,  8.72s/it][2025-04-25 12:19:18,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 12:19:18,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5756.65 | bwd_inner_microstep: 5662.79 | bwd_allreduce_microstep: 93.81 | step_microstep: 19.00
[2025-04-25 12:19:18,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5756.66 | bwd_inner: 5662.79 | bwd_allreduce: 93.83 | step: 19.01

  4%|▍         | 1806/41250 [4:21:43<95:26:16,  8.71s/it]
                                                         
{'loss': 0.1292, 'grad_norm': 0.9645683765411377, 'learning_rate': 3.998011416121893e-05, 'epoch': 0.44}

  4%|▍         | 1806/41250 [4:21:43<95:26:16,  8.71s/it][2025-04-25 12:19:26,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:19:26,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.83 | bwd_microstep: 5717.66 | bwd_inner_microstep: 5664.95 | bwd_allreduce_microstep: 52.66 | step_microstep: 18.98
[2025-04-25 12:19:26,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.83 | bwd: 5717.68 | bwd_inner: 5664.95 | bwd_allreduce: 52.68 | step: 18.98

  4%|▍         | 1807/41250 [4:21:52<95:10:52,  8.69s/it]
                                                         
{'loss': 0.0874, 'grad_norm': 1.1047557592391968, 'learning_rate': 3.9980044090678964e-05, 'epoch': 0.44}

  4%|▍         | 1807/41250 [4:21:52<95:10:52,  8.69s/it][2025-04-25 12:19:35,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 12:19:35,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.54 | bwd_microstep: 5799.31 | bwd_inner_microstep: 5786.39 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.01
[2025-04-25 12:19:35,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.54 | bwd: 5799.32 | bwd_inner: 5786.39 | bwd_allreduce: 12.89 | step: 19.01

  4%|▍         | 1808/41250 [4:22:00<95:28:53,  8.71s/it]
                                                         
{'loss': 0.142, 'grad_norm': 1.2231513261795044, 'learning_rate': 3.997997389696595e-05, 'epoch': 0.44}

  4%|▍         | 1808/41250 [4:22:00<95:28:53,  8.71s/it][2025-04-25 12:19:44,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:19:44,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.60 | bwd_microstep: 5773.34 | bwd_inner_microstep: 5669.64 | bwd_allreduce_microstep: 103.66 | step_microstep: 18.89
[2025-04-25 12:19:44,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.60 | bwd: 5773.35 | bwd_inner: 5669.64 | bwd_allreduce: 103.68 | step: 18.89

  4%|▍         | 1809/41250 [4:22:09<95:23:55,  8.71s/it]
                                                         
{'loss': 0.0866, 'grad_norm': 0.9257456660270691, 'learning_rate': 3.997990358008034e-05, 'epoch': 0.44}

  4%|▍         | 1809/41250 [4:22:09<95:23:55,  8.71s/it][2025-04-25 12:19:53,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.15
[2025-04-25 12:19:53,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.56 | bwd_microstep: 5716.80 | bwd_inner_microstep: 5660.78 | bwd_allreduce_microstep: 55.98 | step_microstep: 19.70
[2025-04-25 12:19:53,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.56 | bwd: 5716.81 | bwd_inner: 5660.78 | bwd_allreduce: 56.00 | step: 19.70

  4%|▍         | 1810/41250 [4:22:18<95:10:25,  8.69s/it]
                                                         
{'loss': 0.226, 'grad_norm': 2.255260944366455, 'learning_rate': 3.9979833140022545e-05, 'epoch': 0.44}

  4%|▍         | 1810/41250 [4:22:18<95:10:25,  8.69s/it][2025-04-25 12:20:01,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.14 | optimizer_step: 0.92
[2025-04-25 12:20:01,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.38 | bwd_microstep: 5753.63 | bwd_inner_microstep: 5685.90 | bwd_allreduce_microstep: 67.69 | step_microstep: 18.78
[2025-04-25 12:20:01,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.38 | bwd: 5753.65 | bwd_inner: 5685.90 | bwd_allreduce: 67.71 | step: 18.78

  4%|▍         | 1811/41250 [4:22:27<95:09:23,  8.69s/it]
                                                         
{'loss': 0.0749, 'grad_norm': 0.8469100594520569, 'learning_rate': 3.997976257679301e-05, 'epoch': 0.44}

  4%|▍         | 1811/41250 [4:22:27<95:09:23,  8.69s/it][2025-04-25 12:20:10,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:20:10,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.06 | bwd_microstep: 5715.08 | bwd_inner_microstep: 5702.37 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.03
[2025-04-25 12:20:10,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.06 | bwd: 5715.10 | bwd_inner: 5702.37 | bwd_allreduce: 12.69 | step: 19.04

  4%|▍         | 1812/41250 [4:22:35<95:02:04,  8.67s/it]
                                                         
{'loss': 0.1637, 'grad_norm': 1.6126806735992432, 'learning_rate': 3.997969189039217e-05, 'epoch': 0.44}

  4%|▍         | 1812/41250 [4:22:35<95:02:04,  8.67s/it][2025-04-25 12:20:18,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:20:18,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.16 | bwd_microstep: 5693.54 | bwd_inner_microstep: 5680.53 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.61
[2025-04-25 12:20:18,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.17 | bwd: 5693.56 | bwd_inner: 5680.53 | bwd_allreduce: 12.99 | step: 18.62

  4%|▍         | 1813/41250 [4:22:44<94:51:25,  8.66s/it]
                                                         
{'loss': 0.3315, 'grad_norm': 2.341780185699463, 'learning_rate': 3.997962108082046e-05, 'epoch': 0.44}

  4%|▍         | 1813/41250 [4:22:44<94:51:25,  8.66s/it][2025-04-25 12:20:27,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 12:20:27,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.81 | bwd_microstep: 5797.88 | bwd_inner_microstep: 5785.19 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.80
[2025-04-25 12:20:27,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.81 | bwd: 5797.90 | bwd_inner: 5785.19 | bwd_allreduce: 12.67 | step: 18.80

  4%|▍         | 1814/41250 [4:22:53<95:13:13,  8.69s/it]
                                                         
{'loss': 0.1048, 'grad_norm': 1.0038981437683105, 'learning_rate': 3.9979550148078316e-05, 'epoch': 0.44}

  4%|▍         | 1814/41250 [4:22:53<95:13:13,  8.69s/it][2025-04-25 12:20:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:20:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.71 | bwd_microstep: 5769.71 | bwd_inner_microstep: 5675.64 | bwd_allreduce_microstep: 94.02 | step_microstep: 18.62
[2025-04-25 12:20:36,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.71 | bwd: 5769.73 | bwd_inner: 5675.64 | bwd_allreduce: 94.04 | step: 18.62

  4%|▍         | 1815/41250 [4:23:01<95:14:40,  8.69s/it]
                                                         
{'loss': 0.258, 'grad_norm': 2.020796298980713, 'learning_rate': 3.997947909216617e-05, 'epoch': 0.44}

  4%|▍         | 1815/41250 [4:23:01<95:14:40,  8.69s/it][2025-04-25 12:20:45,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 12:20:45,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.74 | bwd_microstep: 5704.53 | bwd_inner_microstep: 5657.50 | bwd_allreduce_microstep: 46.99 | step_microstep: 19.41
[2025-04-25 12:20:45,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.74 | bwd: 5704.55 | bwd_inner: 5657.50 | bwd_allreduce: 47.01 | step: 19.41

  4%|▍         | 1816/41250 [4:23:10<94:58:59,  8.67s/it]
                                                         
{'loss': 0.1951, 'grad_norm': 3.1462996006011963, 'learning_rate': 3.997940791308447e-05, 'epoch': 0.44}

  4%|▍         | 1816/41250 [4:23:10<94:58:59,  8.67s/it][2025-04-25 12:20:53,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-25 12:20:53,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.04 | bwd_microstep: 5710.97 | bwd_inner_microstep: 5697.51 | bwd_allreduce_microstep: 13.41 | step_microstep: 19.27
[2025-04-25 12:20:53,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.04 | bwd: 5710.98 | bwd_inner: 5697.51 | bwd_allreduce: 13.43 | step: 19.28

  4%|▍         | 1817/41250 [4:23:19<94:53:05,  8.66s/it]
                                                         
{'loss': 0.08, 'grad_norm': 0.7764653563499451, 'learning_rate': 3.997933661083365e-05, 'epoch': 0.44}

  4%|▍         | 1817/41250 [4:23:19<94:53:05,  8.66s/it][2025-04-25 12:21:02,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 12:21:02,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.31 | bwd_microstep: 5694.71 | bwd_inner_microstep: 5651.07 | bwd_allreduce_microstep: 43.59 | step_microstep: 18.98
[2025-04-25 12:21:02,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.31 | bwd: 5694.72 | bwd_inner: 5651.07 | bwd_allreduce: 43.61 | step: 18.99

  4%|▍         | 1818/41250 [4:23:27<94:41:17,  8.64s/it]
                                                         
{'loss': 0.193, 'grad_norm': 2.142444133758545, 'learning_rate': 3.997926518541415e-05, 'epoch': 0.44}

  4%|▍         | 1818/41250 [4:23:27<94:41:17,  8.64s/it][2025-04-25 12:21:10,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:21:10,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.95 | bwd_microstep: 5703.99 | bwd_inner_microstep: 5691.33 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.87
[2025-04-25 12:21:10,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.94 | bwd: 5704.00 | bwd_inner: 5691.33 | bwd_allreduce: 12.63 | step: 18.87

  4%|▍         | 1819/41250 [4:23:36<94:40:03,  8.64s/it]
                                                         
{'loss': 0.2511, 'grad_norm': 2.1101748943328857, 'learning_rate': 3.9979193636826406e-05, 'epoch': 0.44}

  4%|▍         | 1819/41250 [4:23:36<94:40:03,  8.64s/it][2025-04-25 12:21:19,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 12:21:19,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5668.73 | bwd_inner_microstep: 5649.76 | bwd_allreduce_microstep: 18.92 | step_microstep: 19.11
[2025-04-25 12:21:19,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5668.75 | bwd_inner: 5649.76 | bwd_allreduce: 18.94 | step: 19.11

  4%|▍         | 1820/41250 [4:23:44<94:27:56,  8.62s/it]
                                                         
{'loss': 0.0839, 'grad_norm': 1.6953098773956299, 'learning_rate': 3.997912196507087e-05, 'epoch': 0.44}

  4%|▍         | 1820/41250 [4:23:44<94:27:56,  8.62s/it][2025-04-25 12:21:28,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 12:21:28,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.80 | bwd_microstep: 5715.63 | bwd_inner_microstep: 5702.86 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.98
[2025-04-25 12:21:28,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.80 | bwd: 5715.65 | bwd_inner: 5702.86 | bwd_allreduce: 12.75 | step: 18.99

  4%|▍         | 1821/41250 [4:23:53<94:33:04,  8.63s/it]
                                                         
{'loss': 0.3361, 'grad_norm': 2.5164053440093994, 'learning_rate': 3.997905017014796e-05, 'epoch': 0.44}

  4%|▍         | 1821/41250 [4:23:53<94:33:04,  8.63s/it][2025-04-25 12:21:36,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 12:21:36,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.37 | bwd_microstep: 5717.07 | bwd_inner_microstep: 5704.15 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.26
[2025-04-25 12:21:36,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.37 | bwd: 5717.09 | bwd_inner: 5704.15 | bwd_allreduce: 12.90 | step: 19.27

  4%|▍         | 1822/41250 [4:24:02<94:35:48,  8.64s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 1.8443629741668701, 'learning_rate': 3.997897825205815e-05, 'epoch': 0.44}

  4%|▍         | 1822/41250 [4:24:02<94:35:48,  8.64s/it][2025-04-25 12:21:45,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:21:45,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.70 | bwd_microstep: 5731.80 | bwd_inner_microstep: 5677.07 | bwd_allreduce_microstep: 54.68 | step_microstep: 18.59
[2025-04-25 12:21:45,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.70 | bwd: 5731.82 | bwd_inner: 5677.07 | bwd_allreduce: 54.70 | step: 18.59

  4%|▍         | 1823/41250 [4:24:10<94:40:20,  8.64s/it]
                                                         
{'loss': 0.0867, 'grad_norm': 2.314929723739624, 'learning_rate': 3.997890621080187e-05, 'epoch': 0.44}

  4%|▍         | 1823/41250 [4:24:10<94:40:20,  8.64s/it][2025-04-25 12:21:54,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:21:54,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5676.43 | bwd_inner_microstep: 5652.13 | bwd_allreduce_microstep: 24.26 | step_microstep: 18.94
[2025-04-25 12:21:54,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5676.45 | bwd_inner: 5652.13 | bwd_allreduce: 24.28 | step: 18.94

  4%|▍         | 1824/41250 [4:24:19<94:28:16,  8.63s/it]
                                                         
{'loss': 0.1506, 'grad_norm': 0.6504583358764648, 'learning_rate': 3.9978834046379564e-05, 'epoch': 0.44}

  4%|▍         | 1824/41250 [4:24:19<94:28:16,  8.63s/it][2025-04-25 12:22:02,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:22:02,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.13 | bwd_microstep: 5687.75 | bwd_inner_microstep: 5651.82 | bwd_allreduce_microstep: 35.89 | step_microstep: 18.66
[2025-04-25 12:22:02,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.13 | bwd: 5687.77 | bwd_inner: 5651.82 | bwd_allreduce: 35.90 | step: 18.66

  4%|▍         | 1825/41250 [4:24:27<94:22:18,  8.62s/it]
                                                         
{'loss': 0.35, 'grad_norm': 1.8234566450119019, 'learning_rate': 3.997876175879166e-05, 'epoch': 0.44}

  4%|▍         | 1825/41250 [4:24:27<94:22:18,  8.62s/it][2025-04-25 12:22:11,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:22:11,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.00 | bwd_microstep: 5712.56 | bwd_inner_microstep: 5699.73 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.94
[2025-04-25 12:22:11,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.00 | bwd: 5712.57 | bwd_inner: 5699.73 | bwd_allreduce: 12.80 | step: 18.94

  4%|▍         | 1826/41250 [4:24:36<94:27:35,  8.63s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.2190628051757812, 'learning_rate': 3.9978689348038635e-05, 'epoch': 0.44}

  4%|▍         | 1826/41250 [4:24:36<94:27:35,  8.63s/it][2025-04-25 12:22:19,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:22:19,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.55 | bwd_microstep: 5735.93 | bwd_inner_microstep: 5643.67 | bwd_allreduce_microstep: 92.21 | step_microstep: 18.68
[2025-04-25 12:22:19,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.55 | bwd: 5735.94 | bwd_inner: 5643.67 | bwd_allreduce: 92.23 | step: 18.69

  4%|▍         | 1827/41250 [4:24:45<94:33:23,  8.63s/it]
                                                         
{'loss': 0.1133, 'grad_norm': 1.6208903789520264, 'learning_rate': 3.9978616814120914e-05, 'epoch': 0.44}

  4%|▍         | 1827/41250 [4:24:45<94:33:23,  8.63s/it][2025-04-25 12:22:28,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 12:22:28,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.83 | bwd_microstep: 5757.45 | bwd_inner_microstep: 5645.93 | bwd_allreduce_microstep: 111.47 | step_microstep: 19.34
[2025-04-25 12:22:28,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.83 | bwd: 5757.46 | bwd_inner: 5645.93 | bwd_allreduce: 111.49 | step: 19.34

  4%|▍         | 1828/41250 [4:24:53<94:39:27,  8.64s/it]
                                                         
{'loss': 0.1231, 'grad_norm': 1.5208165645599365, 'learning_rate': 3.997854415703895e-05, 'epoch': 0.44}

  4%|▍         | 1828/41250 [4:24:53<94:39:27,  8.64s/it][2025-04-25 12:22:37,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 12:22:37,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5738.14 | bwd_inner_microstep: 5640.15 | bwd_allreduce_microstep: 97.95 | step_microstep: 19.13
[2025-04-25 12:22:37,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5738.16 | bwd_inner: 5640.15 | bwd_allreduce: 97.96 | step: 19.13

  4%|▍         | 1829/41250 [4:25:02<94:39:44,  8.64s/it]
                                                         
{'loss': 0.1112, 'grad_norm': 1.1522502899169922, 'learning_rate': 3.997847137679319e-05, 'epoch': 0.44}

  4%|▍         | 1829/41250 [4:25:02<94:39:44,  8.64s/it][2025-04-25 12:22:45,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 12:22:45,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.67 | bwd_microstep: 5729.45 | bwd_inner_microstep: 5670.00 | bwd_allreduce_microstep: 59.40 | step_microstep: 18.75
[2025-04-25 12:22:45,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.67 | bwd: 5729.46 | bwd_inner: 5670.00 | bwd_allreduce: 59.41 | step: 18.75

  4%|▍         | 1830/41250 [4:25:11<94:41:15,  8.65s/it]
                                                         
{'loss': 0.0861, 'grad_norm': 0.8652684092521667, 'learning_rate': 3.9978398473384086e-05, 'epoch': 0.44}

  4%|▍         | 1830/41250 [4:25:11<94:41:15,  8.65s/it][2025-04-25 12:22:54,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.09 | optimizer_step: 1.06
[2025-04-25 12:22:54,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.90 | bwd_microstep: 5690.47 | bwd_inner_microstep: 5643.59 | bwd_allreduce_microstep: 46.82 | step_microstep: 19.77
[2025-04-25 12:22:54,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.90 | bwd: 5690.49 | bwd_inner: 5643.59 | bwd_allreduce: 46.85 | step: 19.77

  4%|▍         | 1831/41250 [4:25:19<94:32:27,  8.63s/it]
                                                         
{'loss': 0.0627, 'grad_norm': 1.056227207183838, 'learning_rate': 3.997832544681209e-05, 'epoch': 0.44}

  4%|▍         | 1831/41250 [4:25:19<94:32:27,  8.63s/it][2025-04-25 12:23:03,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 12:23:03,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5698.05 | bwd_inner_microstep: 5684.98 | bwd_allreduce_microstep: 13.03 | step_microstep: 19.06
[2025-04-25 12:23:03,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5698.07 | bwd_inner: 5684.98 | bwd_allreduce: 13.05 | step: 19.06

  4%|▍         | 1832/41250 [4:25:28<94:31:04,  8.63s/it]
                                                         
{'loss': 0.3791, 'grad_norm': 2.312340021133423, 'learning_rate': 3.9978252297077635e-05, 'epoch': 0.44}

  4%|▍         | 1832/41250 [4:25:28<94:31:04,  8.63s/it][2025-04-25 12:23:11,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:23:11,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.68 | bwd_microstep: 5689.01 | bwd_inner_microstep: 5640.11 | bwd_allreduce_microstep: 48.86 | step_microstep: 18.80
[2025-04-25 12:23:11,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.69 | bwd: 5689.02 | bwd_inner: 5640.11 | bwd_allreduce: 48.87 | step: 18.80

  4%|▍         | 1833/41250 [4:25:37<94:22:58,  8.62s/it]
                                                         
{'loss': 0.0898, 'grad_norm': 1.441986322402954, 'learning_rate': 3.997817902418119e-05, 'epoch': 0.44}

  4%|▍         | 1833/41250 [4:25:37<94:22:58,  8.62s/it][2025-04-25 12:23:20,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.15 | optimizer_step: 0.90
[2025-04-25 12:23:20,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5771.03 | bwd_inner_microstep: 5639.04 | bwd_allreduce_microstep: 131.94 | step_microstep: 18.72
[2025-04-25 12:23:20,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5771.04 | bwd_inner: 5639.04 | bwd_allreduce: 131.96 | step: 18.72

  4%|▍         | 1834/41250 [4:25:45<94:34:15,  8.64s/it]
                                                         
{'loss': 0.2976, 'grad_norm': 1.7613255977630615, 'learning_rate': 3.99781056281232e-05, 'epoch': 0.44}

  4%|▍         | 1834/41250 [4:25:45<94:34:15,  8.64s/it][2025-04-25 12:23:29,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 12:23:29,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.44 | bwd_microstep: 5718.22 | bwd_inner_microstep: 5676.98 | bwd_allreduce_microstep: 41.19 | step_microstep: 18.50
[2025-04-25 12:23:29,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.44 | bwd: 5718.23 | bwd_inner: 5676.98 | bwd_allreduce: 41.21 | step: 18.50

  4%|▍         | 1835/41250 [4:25:54<94:35:06,  8.64s/it]
                                                         
{'loss': 0.0345, 'grad_norm': 0.3319243788719177, 'learning_rate': 3.997803210890412e-05, 'epoch': 0.44}

  4%|▍         | 1835/41250 [4:25:54<94:35:06,  8.64s/it][2025-04-25 12:23:37,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 12:23:37,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.90 | bwd_microstep: 5746.56 | bwd_inner_microstep: 5672.70 | bwd_allreduce_microstep: 73.81 | step_microstep: 19.03
[2025-04-25 12:23:37,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.90 | bwd: 5746.57 | bwd_inner: 5672.70 | bwd_allreduce: 73.83 | step: 19.03

  4%|▍         | 1836/41250 [4:26:03<94:40:44,  8.65s/it]
                                                         
{'loss': 0.206, 'grad_norm': 2.4808506965637207, 'learning_rate': 3.99779584665244e-05, 'epoch': 0.45}

  4%|▍         | 1836/41250 [4:26:03<94:40:44,  8.65s/it][2025-04-25 12:23:46,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.20 | optimizer_step: 0.94
[2025-04-25 12:23:46,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.02 | bwd_microstep: 5704.71 | bwd_inner_microstep: 5691.36 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.62
[2025-04-25 12:23:46,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.02 | bwd: 5704.72 | bwd_inner: 5691.36 | bwd_allreduce: 13.32 | step: 19.63

  4%|▍         | 1837/41250 [4:26:11<94:38:06,  8.64s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 2.6149160861968994, 'learning_rate': 3.99778847009845e-05, 'epoch': 0.45}

  4%|▍         | 1837/41250 [4:26:11<94:38:06,  8.64s/it][2025-04-25 12:23:54,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:23:54,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 5683.37 | bwd_inner_microstep: 5670.38 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.96
[2025-04-25 12:23:54,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 5683.39 | bwd_inner: 5670.38 | bwd_allreduce: 12.96 | step: 18.96

  4%|▍         | 1838/41250 [4:26:20<94:32:16,  8.64s/it]
                                                         
{'loss': 0.0613, 'grad_norm': 0.5082060098648071, 'learning_rate': 3.997781081228486e-05, 'epoch': 0.45}

  4%|▍         | 1838/41250 [4:26:20<94:32:16,  8.64s/it][2025-04-25 12:24:03,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:24:03,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5704.81 | bwd_inner_microstep: 5691.86 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.84
[2025-04-25 12:24:03,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5704.82 | bwd_inner: 5691.86 | bwd_allreduce: 12.91 | step: 18.85

  4%|▍         | 1839/41250 [4:26:28<94:33:23,  8.64s/it]
                                                         
{'loss': 0.1679, 'grad_norm': 2.0656018257141113, 'learning_rate': 3.9977736800425956e-05, 'epoch': 0.45}

  4%|▍         | 1839/41250 [4:26:28<94:33:23,  8.64s/it][2025-04-25 12:24:12,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 12:24:12,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.94 | bwd_microstep: 5718.94 | bwd_inner_microstep: 5686.73 | bwd_allreduce_microstep: 32.17 | step_microstep: 19.08
[2025-04-25 12:24:12,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.94 | bwd: 5718.96 | bwd_inner: 5686.73 | bwd_allreduce: 32.19 | step: 19.08

  4%|▍         | 1840/41250 [4:26:37<94:35:49,  8.64s/it]
                                                         
{'loss': 0.1645, 'grad_norm': 1.8667738437652588, 'learning_rate': 3.997766266540823e-05, 'epoch': 0.45}

  4%|▍         | 1840/41250 [4:26:37<94:35:49,  8.64s/it][2025-04-25 12:24:20,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.04 | optimizer_step: 1.13
[2025-04-25 12:24:20,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.80 | bwd_microstep: 5768.01 | bwd_inner_microstep: 5640.51 | bwd_allreduce_microstep: 127.45 | step_microstep: 19.33
[2025-04-25 12:24:20,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.80 | bwd: 5768.03 | bwd_inner: 5640.51 | bwd_allreduce: 127.47 | step: 19.33

  4%|▍         | 1841/41250 [4:26:46<94:42:21,  8.65s/it]
                                                         
{'loss': 0.0514, 'grad_norm': 0.756107747554779, 'learning_rate': 3.997758840723214e-05, 'epoch': 0.45}

  4%|▍         | 1841/41250 [4:26:46<94:42:21,  8.65s/it][2025-04-25 12:24:29,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:24:29,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5730.24 | bwd_inner_microstep: 5675.46 | bwd_allreduce_microstep: 54.73 | step_microstep: 18.92
[2025-04-25 12:24:29,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5730.26 | bwd_inner: 5675.46 | bwd_allreduce: 54.75 | step: 18.92

  4%|▍         | 1842/41250 [4:26:54<94:43:04,  8.65s/it]
                                                         
{'loss': 0.1231, 'grad_norm': 1.9389944076538086, 'learning_rate': 3.9977514025898154e-05, 'epoch': 0.45}

  4%|▍         | 1842/41250 [4:26:54<94:43:04,  8.65s/it][2025-04-25 12:24:38,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 12:24:38,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.34 | bwd_microstep: 5682.55 | bwd_inner_microstep: 5669.50 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.43
[2025-04-25 12:24:38,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.34 | bwd: 5682.56 | bwd_inner: 5669.50 | bwd_allreduce: 13.02 | step: 19.43

  4%|▍         | 1843/41250 [4:27:03<94:35:33,  8.64s/it]
                                                         
{'loss': 0.1585, 'grad_norm': 0.7552403807640076, 'learning_rate': 3.9977439521406724e-05, 'epoch': 0.45}

  4%|▍         | 1843/41250 [4:27:03<94:35:33,  8.64s/it][2025-04-25 12:24:46,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:24:46,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.72 | bwd_microstep: 5733.46 | bwd_inner_microstep: 5642.98 | bwd_allreduce_microstep: 90.44 | step_microstep: 18.84
[2025-04-25 12:24:46,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.72 | bwd: 5733.47 | bwd_inner: 5642.98 | bwd_allreduce: 90.45 | step: 18.84

  4%|▍         | 1844/41250 [4:27:12<94:35:11,  8.64s/it]
                                                         
{'loss': 0.278, 'grad_norm': 1.2152512073516846, 'learning_rate': 3.99773648937583e-05, 'epoch': 0.45}

  4%|▍         | 1844/41250 [4:27:12<94:35:11,  8.64s/it][2025-04-25 12:24:55,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:24:55,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.39 | bwd_microstep: 5762.91 | bwd_inner_microstep: 5627.51 | bwd_allreduce_microstep: 135.36 | step_microstep: 18.52
[2025-04-25 12:24:55,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.39 | bwd: 5762.93 | bwd_inner: 5627.51 | bwd_allreduce: 135.37 | step: 18.52

  4%|▍         | 1845/41250 [4:27:20<94:40:10,  8.65s/it]
                                                         
{'loss': 0.3751, 'grad_norm': 2.383204698562622, 'learning_rate': 3.997729014295336e-05, 'epoch': 0.45}

  4%|▍         | 1845/41250 [4:27:20<94:40:10,  8.65s/it][2025-04-25 12:25:04,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 12:25:04,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.42 | bwd_microstep: 5685.60 | bwd_inner_microstep: 5672.62 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.86
[2025-04-25 12:25:04,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.42 | bwd: 5685.61 | bwd_inner: 5672.62 | bwd_allreduce: 12.95 | step: 18.86

  4%|▍         | 1846/41250 [4:27:29<94:32:20,  8.64s/it]
                                                         
{'loss': 0.2922, 'grad_norm': 1.9225304126739502, 'learning_rate': 3.997721526899235e-05, 'epoch': 0.45}

  4%|▍         | 1846/41250 [4:27:29<94:32:20,  8.64s/it][2025-04-25 12:25:12,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:25:12,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5703.10 | bwd_inner_microstep: 5690.44 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.30
[2025-04-25 12:25:12,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5703.12 | bwd_inner: 5690.44 | bwd_allreduce: 12.64 | step: 18.30

  4%|▍         | 1847/41250 [4:27:38<94:33:32,  8.64s/it]
                                                         
{'loss': 0.1498, 'grad_norm': 1.180381417274475, 'learning_rate': 3.997714027187574e-05, 'epoch': 0.45}

  4%|▍         | 1847/41250 [4:27:38<94:33:32,  8.64s/it][2025-04-25 12:25:21,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:25:21,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5727.06 | bwd_inner_microstep: 5683.68 | bwd_allreduce_microstep: 43.33 | step_microstep: 18.60
[2025-04-25 12:25:21,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5727.07 | bwd_inner: 5683.68 | bwd_allreduce: 43.35 | step: 18.60

  4%|▍         | 1848/41250 [4:27:46<94:37:56,  8.65s/it]
                                                         
{'loss': 0.208, 'grad_norm': 2.555131435394287, 'learning_rate': 3.997706515160398e-05, 'epoch': 0.45}

  4%|▍         | 1848/41250 [4:27:46<94:37:56,  8.65s/it][2025-04-25 12:25:30,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:25:30,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.71 | bwd_microstep: 5691.80 | bwd_inner_microstep: 5679.19 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.45
[2025-04-25 12:25:30,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.71 | bwd: 5691.82 | bwd_inner: 5679.19 | bwd_allreduce: 12.59 | step: 18.46

  4%|▍         | 1849/41250 [4:27:55<94:32:32,  8.64s/it]
                                                         
{'loss': 0.0754, 'grad_norm': 0.6911908388137817, 'learning_rate': 3.997698990817755e-05, 'epoch': 0.45}

  4%|▍         | 1849/41250 [4:27:55<94:32:32,  8.64s/it][2025-04-25 12:25:38,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:25:38,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.98 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5692.19 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.21
[2025-04-25 12:25:38,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.98 | bwd: 5704.89 | bwd_inner: 5692.19 | bwd_allreduce: 12.66 | step: 18.21

  4%|▍         | 1850/41250 [4:28:04<94:34:58,  8.64s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 2.1782054901123047, 'learning_rate': 3.997691454159691e-05, 'epoch': 0.45}

  4%|▍         | 1850/41250 [4:28:04<94:34:58,  8.64s/it][2025-04-25 12:25:47,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 12:25:47,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.11 | bwd_microstep: 5779.28 | bwd_inner_microstep: 5644.81 | bwd_allreduce_microstep: 134.42 | step_microstep: 18.65
[2025-04-25 12:25:47,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.11 | bwd: 5779.29 | bwd_inner: 5644.81 | bwd_allreduce: 134.44 | step: 18.66

  4%|▍         | 1851/41250 [4:28:12<94:44:28,  8.66s/it]
                                                         
{'loss': 0.0491, 'grad_norm': 1.2788387537002563, 'learning_rate': 3.997683905186251e-05, 'epoch': 0.45}

  4%|▍         | 1851/41250 [4:28:12<94:44:28,  8.66s/it][2025-04-25 12:25:56,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:25:56,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.02 | bwd_microstep: 5772.03 | bwd_inner_microstep: 5759.18 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.76
[2025-04-25 12:25:56,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.02 | bwd: 5772.05 | bwd_inner: 5759.18 | bwd_allreduce: 12.82 | step: 18.77

  4%|▍         | 1852/41250 [4:28:21<95:00:27,  8.68s/it]
                                                         
{'loss': 0.0974, 'grad_norm': 0.9891206622123718, 'learning_rate': 3.997676343897484e-05, 'epoch': 0.45}

  4%|▍         | 1852/41250 [4:28:21<95:00:27,  8.68s/it][2025-04-25 12:26:04,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:26:04,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.24 | bwd_microstep: 5733.18 | bwd_inner_microstep: 5700.85 | bwd_allreduce_microstep: 32.28 | step_microstep: 17.92
[2025-04-25 12:26:04,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.24 | bwd: 5733.19 | bwd_inner: 5700.85 | bwd_allreduce: 32.30 | step: 17.92

  4%|▍         | 1853/41250 [4:28:30<94:57:13,  8.68s/it]
                                                         
{'loss': 0.2459, 'grad_norm': 2.3146402835845947, 'learning_rate': 3.9976687702934344e-05, 'epoch': 0.45}

  4%|▍         | 1853/41250 [4:28:30<94:57:13,  8.68s/it][2025-04-25 12:26:13,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:26:13,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.58 | bwd_microstep: 5752.00 | bwd_inner_microstep: 5694.28 | bwd_allreduce_microstep: 57.67 | step_microstep: 18.81
[2025-04-25 12:26:13,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.58 | bwd: 5752.01 | bwd_inner: 5694.28 | bwd_allreduce: 57.69 | step: 18.82

  4%|▍         | 1854/41250 [4:28:38<94:56:53,  8.68s/it]
                                                         
{'loss': 0.1494, 'grad_norm': 2.012328624725342, 'learning_rate': 3.99766118437415e-05, 'epoch': 0.45}

  4%|▍         | 1854/41250 [4:28:38<94:56:53,  8.68s/it][2025-04-25 12:26:22,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:26:22,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5761.62 | bwd_inner_microstep: 5703.54 | bwd_allreduce_microstep: 58.04 | step_microstep: 18.35
[2025-04-25 12:26:22,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.48 | bwd: 5761.63 | bwd_inner: 5703.53 | bwd_allreduce: 58.05 | step: 18.35

  4%|▍         | 1855/41250 [4:28:47<94:59:22,  8.68s/it]
                                                         
{'loss': 0.0696, 'grad_norm': 0.5150580406188965, 'learning_rate': 3.997653586139677e-05, 'epoch': 0.45}

  4%|▍         | 1855/41250 [4:28:47<94:59:22,  8.68s/it][2025-04-25 12:26:30,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:26:30,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.15 | bwd_microstep: 5836.06 | bwd_inner_microstep: 5696.05 | bwd_allreduce_microstep: 139.96 | step_microstep: 18.75
[2025-04-25 12:26:30,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.15 | bwd: 5836.07 | bwd_inner: 5696.05 | bwd_allreduce: 139.98 | step: 18.76

  4%|▍         | 1856/41250 [4:28:56<95:20:14,  8.71s/it]
                                                         
{'loss': 0.2433, 'grad_norm': 1.5966686010360718, 'learning_rate': 3.997645975590064e-05, 'epoch': 0.45}

  4%|▍         | 1856/41250 [4:28:56<95:20:14,  8.71s/it][2025-04-25 12:26:39,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 12:26:39,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.36 | bwd_microstep: 5724.25 | bwd_inner_microstep: 5711.75 | bwd_allreduce_microstep: 12.45 | step_microstep: 18.76
[2025-04-25 12:26:39,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.36 | bwd: 5724.27 | bwd_inner: 5711.75 | bwd_allreduce: 12.47 | step: 18.76

  5%|▍         | 1857/41250 [4:29:04<95:11:29,  8.70s/it]
                                                         
{'loss': 0.2018, 'grad_norm': 1.7757755517959595, 'learning_rate': 3.9976383527253546e-05, 'epoch': 0.45}

  5%|▍         | 1857/41250 [4:29:04<95:11:29,  8.70s/it][2025-04-25 12:26:48,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.15 | optimizer_step: 1.04
[2025-04-25 12:26:48,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5744.47 | bwd_inner_microstep: 5712.89 | bwd_allreduce_microstep: 31.48 | step_microstep: 20.66
[2025-04-25 12:26:48,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5744.51 | bwd_inner: 5712.89 | bwd_allreduce: 31.53 | step: 20.64

  5%|▍         | 1858/41250 [4:29:13<95:07:49,  8.69s/it]
                                                         
{'loss': 0.0685, 'grad_norm': 0.9496169090270996, 'learning_rate': 3.997630717545598e-05, 'epoch': 0.45}

  5%|▍         | 1858/41250 [4:29:13<95:07:49,  8.69s/it][2025-04-25 12:26:56,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 12:26:56,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.04 | bwd_microstep: 5725.59 | bwd_inner_microstep: 5712.93 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.79
[2025-04-25 12:26:56,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.04 | bwd: 5725.60 | bwd_inner: 5712.93 | bwd_allreduce: 12.64 | step: 18.79

  5%|▍         | 1859/41250 [4:29:22<95:02:44,  8.69s/it]
                                                         
{'loss': 0.1518, 'grad_norm': 1.256903886795044, 'learning_rate': 3.997623070050842e-05, 'epoch': 0.45}

  5%|▍         | 1859/41250 [4:29:22<95:02:44,  8.69s/it][2025-04-25 12:27:05,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 0.91
[2025-04-25 12:27:05,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5743.29 | bwd_inner_microstep: 5689.41 | bwd_allreduce_microstep: 53.81 | step_microstep: 19.67
[2025-04-25 12:27:05,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5743.31 | bwd_inner: 5689.41 | bwd_allreduce: 53.84 | step: 19.67

  5%|▍         | 1860/41250 [4:29:30<95:02:15,  8.69s/it]
                                                         
{'loss': 0.3081, 'grad_norm': 1.9191886186599731, 'learning_rate': 3.997615410241132e-05, 'epoch': 0.45}

  5%|▍         | 1860/41250 [4:29:30<95:02:15,  8.69s/it][2025-04-25 12:27:14,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 12:27:14,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.14 | bwd_microstep: 5725.25 | bwd_inner_microstep: 5712.53 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.03
[2025-04-25 12:27:14,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.14 | bwd: 5725.27 | bwd_inner: 5712.53 | bwd_allreduce: 12.70 | step: 19.03

  5%|▍         | 1861/41250 [4:29:39<94:59:14,  8.68s/it]
                                                         
{'loss': 0.1043, 'grad_norm': 1.0060359239578247, 'learning_rate': 3.997607738116516e-05, 'epoch': 0.45}

  5%|▍         | 1861/41250 [4:29:39<94:59:14,  8.68s/it][2025-04-25 12:27:22,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 12:27:23,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.65 | bwd_microstep: 5725.50 | bwd_inner_microstep: 5712.67 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.89
[2025-04-25 12:27:23,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.65 | bwd: 5725.51 | bwd_inner: 5712.67 | bwd_allreduce: 12.80 | step: 18.89

  5%|▍         | 1862/41250 [4:29:48<94:55:56,  8.68s/it]
                                                         
{'loss': 0.0845, 'grad_norm': 0.8670909404754639, 'learning_rate': 3.9976000536770415e-05, 'epoch': 0.45}

  5%|▍         | 1862/41250 [4:29:48<94:55:56,  8.68s/it][2025-04-25 12:27:31,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 12:27:31,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.92 | bwd_microstep: 5810.03 | bwd_inner_microstep: 5669.39 | bwd_allreduce_microstep: 140.60 | step_microstep: 19.32
[2025-04-25 12:27:31,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.92 | bwd: 5810.05 | bwd_inner: 5669.39 | bwd_allreduce: 140.61 | step: 19.33

  5%|▍         | 1863/41250 [4:29:57<95:06:54,  8.69s/it]
                                                         
{'loss': 0.1876, 'grad_norm': 2.178309202194214, 'learning_rate': 3.997592356922755e-05, 'epoch': 0.45}

  5%|▍         | 1863/41250 [4:29:57<95:06:54,  8.69s/it][2025-04-25 12:27:40,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.17 | optimizer_step: 0.91
[2025-04-25 12:27:40,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.00 | bwd_microstep: 5767.51 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 76.57 | step_microstep: 19.13
[2025-04-25 12:27:40,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.00 | bwd: 5767.52 | bwd_inner: 5690.89 | bwd_allreduce: 76.59 | step: 19.13

  5%|▍         | 1864/41250 [4:30:05<95:08:18,  8.70s/it]
                                                         
{'loss': 0.1299, 'grad_norm': 1.2496016025543213, 'learning_rate': 3.997584647853705e-05, 'epoch': 0.45}

  5%|▍         | 1864/41250 [4:30:05<95:08:18,  8.70s/it][2025-04-25 12:27:49,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:27:49,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.28 | bwd_microstep: 5718.76 | bwd_inner_microstep: 5705.86 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.97
[2025-04-25 12:27:49,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.28 | bwd: 5718.77 | bwd_inner: 5705.86 | bwd_allreduce: 12.87 | step: 18.97

  5%|▍         | 1865/41250 [4:30:14<95:01:49,  8.69s/it]
                                                         
{'loss': 0.109, 'grad_norm': 1.4485833644866943, 'learning_rate': 3.997576926469938e-05, 'epoch': 0.45}

  5%|▍         | 1865/41250 [4:30:14<95:01:49,  8.69s/it][2025-04-25 12:27:57,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.30 | optimizer_step: 1.03
[2025-04-25 12:27:57,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.00 | bwd_microstep: 5714.06 | bwd_inner_microstep: 5659.92 | bwd_allreduce_microstep: 54.08 | step_microstep: 20.10
[2025-04-25 12:27:57,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.00 | bwd: 5714.07 | bwd_inner: 5659.92 | bwd_allreduce: 54.11 | step: 20.10

  5%|▍         | 1866/41250 [4:30:23<94:49:57,  8.67s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.9978834390640259, 'learning_rate': 3.9975691927715024e-05, 'epoch': 0.45}

  5%|▍         | 1866/41250 [4:30:23<94:49:57,  8.67s/it][2025-04-25 12:28:06,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 12:28:06,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.77 | bwd_microstep: 5809.84 | bwd_inner_microstep: 5657.36 | bwd_allreduce_microstep: 152.44 | step_microstep: 18.85
[2025-04-25 12:28:06,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.77 | bwd: 5809.85 | bwd_inner: 5657.35 | bwd_allreduce: 152.45 | step: 18.85

  5%|▍         | 1867/41250 [4:30:31<95:00:55,  8.69s/it]
                                                         
{'loss': 0.3636, 'grad_norm': 2.5767459869384766, 'learning_rate': 3.9975614467584456e-05, 'epoch': 0.45}

  5%|▍         | 1867/41250 [4:30:31<95:00:55,  8.69s/it][2025-04-25 12:28:15,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:28:15,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.76 | bwd_microstep: 5788.53 | bwd_inner_microstep: 5652.85 | bwd_allreduce_microstep: 135.64 | step_microstep: 18.80
[2025-04-25 12:28:15,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.76 | bwd: 5788.55 | bwd_inner: 5652.85 | bwd_allreduce: 135.65 | step: 18.80

  5%|▍         | 1868/41250 [4:30:40<95:03:25,  8.69s/it]
                                                         
{'loss': 0.4021, 'grad_norm': 2.842984199523926, 'learning_rate': 3.997553688430816e-05, 'epoch': 0.45}

  5%|▍         | 1868/41250 [4:30:40<95:03:25,  8.69s/it][2025-04-25 12:28:23,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 12:28:23,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.47 | bwd_microstep: 5803.63 | bwd_inner_microstep: 5790.75 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.86
[2025-04-25 12:28:23,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.47 | bwd: 5803.65 | bwd_inner: 5790.75 | bwd_allreduce: 12.85 | step: 18.86

  5%|▍         | 1869/41250 [4:30:49<95:20:21,  8.72s/it]
                                                         
{'loss': 0.0961, 'grad_norm': 0.9640570878982544, 'learning_rate': 3.99754591778866e-05, 'epoch': 0.45}

  5%|▍         | 1869/41250 [4:30:49<95:20:21,  8.72s/it][2025-04-25 12:28:32,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 12:28:32,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5718.34 | bwd_inner_microstep: 5705.29 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.32
[2025-04-25 12:28:32,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5718.36 | bwd_inner: 5705.29 | bwd_allreduce: 13.02 | step: 19.32

  5%|▍         | 1870/41250 [4:30:57<95:07:05,  8.70s/it]
                                                         
{'loss': 0.1957, 'grad_norm': 1.944496512413025, 'learning_rate': 3.997538134832026e-05, 'epoch': 0.45}

  5%|▍         | 1870/41250 [4:30:57<95:07:05,  8.70s/it][2025-04-25 12:28:41,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 12:28:41,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.98 | bwd_microstep: 5719.87 | bwd_inner_microstep: 5660.21 | bwd_allreduce_microstep: 59.62 | step_microstep: 19.35
[2025-04-25 12:28:41,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.98 | bwd: 5719.88 | bwd_inner: 5660.21 | bwd_allreduce: 59.64 | step: 19.35

  5%|▍         | 1871/41250 [4:31:06<94:55:22,  8.68s/it]
                                                         
{'loss': 0.1875, 'grad_norm': 1.0258233547210693, 'learning_rate': 3.997530339560963e-05, 'epoch': 0.45}

  5%|▍         | 1871/41250 [4:31:06<94:55:22,  8.68s/it][2025-04-25 12:28:49,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 12:28:49,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.21 | bwd_microstep: 5709.48 | bwd_inner_microstep: 5662.73 | bwd_allreduce_microstep: 46.70 | step_microstep: 18.86
[2025-04-25 12:28:49,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.21 | bwd: 5709.49 | bwd_inner: 5662.73 | bwd_allreduce: 46.72 | step: 18.86

  5%|▍         | 1872/41250 [4:31:15<94:44:22,  8.66s/it]
                                                         
{'loss': 0.1765, 'grad_norm': 1.6231489181518555, 'learning_rate': 3.997522531975518e-05, 'epoch': 0.45}

  5%|▍         | 1872/41250 [4:31:15<94:44:22,  8.66s/it][2025-04-25 12:28:58,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:28:58,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.59 | bwd_microstep: 5700.53 | bwd_inner_microstep: 5687.77 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.91
[2025-04-25 12:28:58,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.59 | bwd: 5700.54 | bwd_inner: 5687.77 | bwd_allreduce: 12.73 | step: 18.92

  5%|▍         | 1873/41250 [4:31:23<94:39:02,  8.65s/it]
                                                         
{'loss': 0.0337, 'grad_norm': 0.35617247223854065, 'learning_rate': 3.9975147120757406e-05, 'epoch': 0.45}

  5%|▍         | 1873/41250 [4:31:23<94:39:02,  8.65s/it][2025-04-25 12:29:07,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 12:29:07,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5714.71 | bwd_inner_microstep: 5701.95 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.59
[2025-04-25 12:29:07,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.00 | bwd: 5714.72 | bwd_inner: 5701.95 | bwd_allreduce: 12.73 | step: 18.59

  5%|▍         | 1874/41250 [4:31:32<94:39:02,  8.65s/it]
                                                         
{'loss': 0.3053, 'grad_norm': 2.3957858085632324, 'learning_rate': 3.997506879861677e-05, 'epoch': 0.45}

  5%|▍         | 1874/41250 [4:31:32<94:39:02,  8.65s/it][2025-04-25 12:29:15,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 12:29:15,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.23 | bwd_microstep: 5799.52 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 137.39 | step_microstep: 18.70
[2025-04-25 12:29:15,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.23 | bwd: 5799.53 | bwd_inner: 5662.08 | bwd_allreduce: 137.41 | step: 18.71

  5%|▍         | 1875/41250 [4:31:41<94:49:45,  8.67s/it]
                                                         
{'loss': 0.2188, 'grad_norm': 1.7111369371414185, 'learning_rate': 3.997499035333376e-05, 'epoch': 0.45}

  5%|▍         | 1875/41250 [4:31:41<94:49:45,  8.67s/it][2025-04-25 12:29:24,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 12:29:24,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.22 | bwd_microstep: 5707.18 | bwd_inner_microstep: 5694.48 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.81
[2025-04-25 12:29:24,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.22 | bwd: 5707.20 | bwd_inner: 5694.48 | bwd_allreduce: 12.68 | step: 18.81

  5%|▍         | 1876/41250 [4:31:49<94:42:04,  8.66s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.8957067728042603, 'learning_rate': 3.997491178490887e-05, 'epoch': 0.45}

  5%|▍         | 1876/41250 [4:31:49<94:42:04,  8.66s/it][2025-04-25 12:29:33,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 12:29:33,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.68 | bwd_microstep: 5686.23 | bwd_inner_microstep: 5656.00 | bwd_allreduce_microstep: 30.18 | step_microstep: 18.69
[2025-04-25 12:29:33,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.68 | bwd: 5686.24 | bwd_inner: 5656.00 | bwd_allreduce: 30.20 | step: 18.69

  5%|▍         | 1877/41250 [4:31:58<94:31:07,  8.64s/it]
                                                         
{'loss': 0.0842, 'grad_norm': 1.2696765661239624, 'learning_rate': 3.997483309334257e-05, 'epoch': 0.46}

  5%|▍         | 1877/41250 [4:31:58<94:31:07,  8.64s/it][2025-04-25 12:29:41,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 12:29:41,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5753.64 | bwd_inner_microstep: 5653.09 | bwd_allreduce_microstep: 100.50 | step_microstep: 19.49
[2025-04-25 12:29:41,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5753.65 | bwd_inner: 5653.09 | bwd_allreduce: 100.52 | step: 19.49

  5%|▍         | 1878/41250 [4:32:07<94:36:06,  8.65s/it]
                                                         
{'loss': 0.3035, 'grad_norm': 3.245912551879883, 'learning_rate': 3.997475427863536e-05, 'epoch': 0.46}

  5%|▍         | 1878/41250 [4:32:07<94:36:06,  8.65s/it][2025-04-25 12:29:50,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:29:50,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.64 | bwd_microstep: 5755.52 | bwd_inner_microstep: 5702.82 | bwd_allreduce_microstep: 52.66 | step_microstep: 18.70
[2025-04-25 12:29:50,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.64 | bwd: 5755.53 | bwd_inner: 5702.82 | bwd_allreduce: 52.67 | step: 18.70

  5%|▍         | 1879/41250 [4:32:15<94:43:47,  8.66s/it]
                                                         
{'loss': 0.0723, 'grad_norm': 1.5231690406799316, 'learning_rate': 3.9974675340787715e-05, 'epoch': 0.46}

  5%|▍         | 1879/41250 [4:32:15<94:43:47,  8.66s/it][2025-04-25 12:29:59,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:29:59,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.55 | bwd_microstep: 5763.37 | bwd_inner_microstep: 5654.69 | bwd_allreduce_microstep: 108.63 | step_microstep: 18.73
[2025-04-25 12:29:59,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.55 | bwd: 5763.38 | bwd_inner: 5654.69 | bwd_allreduce: 108.65 | step: 18.74

  5%|▍         | 1880/41250 [4:32:24<94:47:28,  8.67s/it]
                                                         
{'loss': 0.1282, 'grad_norm': 1.2944350242614746, 'learning_rate': 3.997459627980013e-05, 'epoch': 0.46}

  5%|▍         | 1880/41250 [4:32:24<94:47:28,  8.67s/it][2025-04-25 12:30:07,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:30:07,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.90 | bwd_microstep: 5762.09 | bwd_inner_microstep: 5647.41 | bwd_allreduce_microstep: 114.62 | step_microstep: 18.75
[2025-04-25 12:30:07,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.90 | bwd: 5762.10 | bwd_inner: 5647.41 | bwd_allreduce: 114.64 | step: 18.75

  5%|▍         | 1881/41250 [4:32:33<94:48:22,  8.67s/it]
                                                         
{'loss': 0.1623, 'grad_norm': 2.546250581741333, 'learning_rate': 3.9974517095673084e-05, 'epoch': 0.46}

  5%|▍         | 1881/41250 [4:32:33<94:48:22,  8.67s/it][2025-04-25 12:30:16,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 12:30:16,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.21 | bwd_microstep: 5715.33 | bwd_inner_microstep: 5702.52 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.34
[2025-04-25 12:30:16,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.21 | bwd: 5715.34 | bwd_inner: 5702.52 | bwd_allreduce: 12.78 | step: 19.34

  5%|▍         | 1882/41250 [4:32:41<94:45:16,  8.66s/it]
                                                         
{'loss': 0.1926, 'grad_norm': 1.763068675994873, 'learning_rate': 3.997443778840707e-05, 'epoch': 0.46}

  5%|▍         | 1882/41250 [4:32:41<94:45:16,  8.66s/it][2025-04-25 12:30:25,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:30:25,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5696.54 | bwd_inner_microstep: 5657.13 | bwd_allreduce_microstep: 39.36 | step_microstep: 18.62
[2025-04-25 12:30:25,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5696.55 | bwd_inner: 5657.13 | bwd_allreduce: 39.38 | step: 18.62

  5%|▍         | 1883/41250 [4:32:50<94:34:21,  8.65s/it]
                                                         
{'loss': 0.0867, 'grad_norm': 1.7682836055755615, 'learning_rate': 3.9974358358002575e-05, 'epoch': 0.46}

  5%|▍         | 1883/41250 [4:32:50<94:34:21,  8.65s/it][2025-04-25 12:30:33,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 12:30:33,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.93 | bwd_microstep: 5772.85 | bwd_inner_microstep: 5759.76 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.21
[2025-04-25 12:30:33,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.93 | bwd: 5772.86 | bwd_inner: 5759.76 | bwd_allreduce: 13.07 | step: 19.22

  5%|▍         | 1884/41250 [4:32:59<94:52:03,  8.68s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 0.9385153651237488, 'learning_rate': 3.997427880446008e-05, 'epoch': 0.46}

  5%|▍         | 1884/41250 [4:32:59<94:52:03,  8.68s/it][2025-04-25 12:30:42,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 12:30:42,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.95 | bwd_microstep: 5736.21 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 32.12 | step_microstep: 18.80
[2025-04-25 12:30:42,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.95 | bwd: 5736.23 | bwd_inner: 5704.05 | bwd_allreduce: 32.13 | step: 18.81

  5%|▍         | 1885/41250 [4:33:07<94:50:27,  8.67s/it]
                                                         
{'loss': 0.2931, 'grad_norm': 1.695229172706604, 'learning_rate': 3.9974199127780095e-05, 'epoch': 0.46}

  5%|▍         | 1885/41250 [4:33:07<94:50:27,  8.67s/it][2025-04-25 12:30:51,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 12:30:51,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.18 | bwd_microstep: 5698.95 | bwd_inner_microstep: 5630.47 | bwd_allreduce_microstep: 68.43 | step_microstep: 18.83
[2025-04-25 12:30:51,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.18 | bwd: 5698.96 | bwd_inner: 5630.47 | bwd_allreduce: 68.45 | step: 18.83

  5%|▍         | 1886/41250 [4:33:16<94:35:27,  8.65s/it]
                                                         
{'loss': 0.2395, 'grad_norm': 2.668485641479492, 'learning_rate': 3.9974119327963106e-05, 'epoch': 0.46}

  5%|▍         | 1886/41250 [4:33:16<94:35:27,  8.65s/it][2025-04-25 12:30:59,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 12:30:59,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.86 | bwd_microstep: 5766.87 | bwd_inner_microstep: 5648.89 | bwd_allreduce_microstep: 117.92 | step_microstep: 18.86
[2025-04-25 12:30:59,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.86 | bwd: 5766.88 | bwd_inner: 5648.89 | bwd_allreduce: 117.95 | step: 18.86

  5%|▍         | 1887/41250 [4:33:25<94:41:08,  8.66s/it]
                                                         
{'loss': 0.3452, 'grad_norm': 3.769028425216675, 'learning_rate': 3.99740394050096e-05, 'epoch': 0.46}

  5%|▍         | 1887/41250 [4:33:25<94:41:08,  8.66s/it][2025-04-25 12:31:08,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:31:08,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.40 | bwd_microstep: 5714.92 | bwd_inner_microstep: 5688.59 | bwd_allreduce_microstep: 26.28 | step_microstep: 18.40
[2025-04-25 12:31:08,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.40 | bwd: 5714.93 | bwd_inner: 5688.59 | bwd_allreduce: 26.30 | step: 18.40

  5%|▍         | 1888/41250 [4:33:33<94:40:53,  8.66s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.5440870523452759, 'learning_rate': 3.997395935892006e-05, 'epoch': 0.46}

  5%|▍         | 1888/41250 [4:33:33<94:40:53,  8.66s/it][2025-04-25 12:31:17,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-25 12:31:17,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.25 | bwd_microstep: 5854.83 | bwd_inner_microstep: 5693.72 | bwd_allreduce_microstep: 161.05 | step_microstep: 18.78
[2025-04-25 12:31:17,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.25 | bwd: 5854.84 | bwd_inner: 5693.72 | bwd_allreduce: 161.08 | step: 18.78

  5%|▍         | 1889/41250 [4:33:42<95:07:00,  8.70s/it]
                                                         
{'loss': 0.1315, 'grad_norm': 1.4872504472732544, 'learning_rate': 3.997387918969499e-05, 'epoch': 0.46}

  5%|▍         | 1889/41250 [4:33:42<95:07:00,  8.70s/it][2025-04-25 12:31:25,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:31:25,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.33 | bwd_microstep: 5679.22 | bwd_inner_microstep: 5636.95 | bwd_allreduce_microstep: 42.23 | step_microstep: 18.46
[2025-04-25 12:31:25,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.33 | bwd: 5679.23 | bwd_inner: 5636.95 | bwd_allreduce: 42.24 | step: 18.46

  5%|▍         | 1890/41250 [4:33:51<94:45:22,  8.67s/it]
                                                         
{'loss': 0.0694, 'grad_norm': 0.889743983745575, 'learning_rate': 3.997379889733489e-05, 'epoch': 0.46}

  5%|▍         | 1890/41250 [4:33:51<94:45:22,  8.67s/it][2025-04-25 12:31:34,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:31:34,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.17 | bwd_microstep: 5694.43 | bwd_inner_microstep: 5641.73 | bwd_allreduce_microstep: 52.66 | step_microstep: 18.90
[2025-04-25 12:31:34,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.17 | bwd: 5694.45 | bwd_inner: 5641.73 | bwd_allreduce: 52.68 | step: 18.91

  5%|▍         | 1891/41250 [4:33:59<94:32:49,  8.65s/it]
                                                         
{'loss': 0.2309, 'grad_norm': 1.8745677471160889, 'learning_rate': 3.9973718481840244e-05, 'epoch': 0.46}

  5%|▍         | 1891/41250 [4:33:59<94:32:49,  8.65s/it][2025-04-25 12:31:42,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:31:42,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.73 | bwd_microstep: 5673.49 | bwd_inner_microstep: 5660.86 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.63
[2025-04-25 12:31:42,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.73 | bwd: 5673.50 | bwd_inner: 5660.86 | bwd_allreduce: 12.60 | step: 18.64

  5%|▍         | 1892/41250 [4:34:08<94:23:27,  8.63s/it]
                                                         
{'loss': 0.2214, 'grad_norm': 1.765601396560669, 'learning_rate': 3.9973637943211554e-05, 'epoch': 0.46}

  5%|▍         | 1892/41250 [4:34:08<94:23:27,  8.63s/it][2025-04-25 12:31:51,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.23 | optimizer_step: 1.01
[2025-04-25 12:31:51,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5764.58 | bwd_inner_microstep: 5641.81 | bwd_allreduce_microstep: 122.71 | step_microstep: 19.29
[2025-04-25 12:31:51,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5764.60 | bwd_inner: 5641.81 | bwd_allreduce: 122.74 | step: 19.29

  5%|▍         | 1893/41250 [4:34:16<94:33:04,  8.65s/it]
                                                         
{'loss': 0.1502, 'grad_norm': 3.733947277069092, 'learning_rate': 3.997355728144931e-05, 'epoch': 0.46}

  5%|▍         | 1893/41250 [4:34:16<94:33:04,  8.65s/it][2025-04-25 12:32:00,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:32:00,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.56 | bwd_microstep: 5695.02 | bwd_inner_microstep: 5682.27 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.27
[2025-04-25 12:32:00,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.56 | bwd: 5695.03 | bwd_inner: 5682.27 | bwd_allreduce: 12.72 | step: 18.27

  5%|▍         | 1894/41250 [4:34:25<94:29:34,  8.64s/it]
                                                         
{'loss': 0.2026, 'grad_norm': 2.4831342697143555, 'learning_rate': 3.9973476496554024e-05, 'epoch': 0.46}

  5%|▍         | 1894/41250 [4:34:25<94:29:34,  8.64s/it][2025-04-25 12:32:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:32:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.44 | bwd_microstep: 5739.29 | bwd_inner_microstep: 5639.22 | bwd_allreduce_microstep: 100.02 | step_microstep: 18.26
[2025-04-25 12:32:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.44 | bwd: 5739.30 | bwd_inner: 5639.22 | bwd_allreduce: 100.04 | step: 18.26

  5%|▍         | 1895/41250 [4:34:34<94:31:30,  8.65s/it]
                                                         
{'loss': 0.1474, 'grad_norm': 1.6344434022903442, 'learning_rate': 3.9973395588526175e-05, 'epoch': 0.46}

  5%|▍         | 1895/41250 [4:34:34<94:31:30,  8.65s/it][2025-04-25 12:32:17,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 12:32:17,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.65 | bwd_microstep: 5724.18 | bwd_inner_microstep: 5635.88 | bwd_allreduce_microstep: 88.25 | step_microstep: 18.61
[2025-04-25 12:32:17,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.65 | bwd: 5724.19 | bwd_inner: 5635.88 | bwd_allreduce: 88.27 | step: 18.62

  5%|▍         | 1896/41250 [4:34:42<94:31:48,  8.65s/it]
                                                         
{'loss': 0.2262, 'grad_norm': 2.412409543991089, 'learning_rate': 3.997331455736628e-05, 'epoch': 0.46}

  5%|▍         | 1896/41250 [4:34:42<94:31:48,  8.65s/it][2025-04-25 12:32:26,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 12:32:26,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.59 | bwd_microstep: 5766.84 | bwd_inner_microstep: 5753.86 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.57
[2025-04-25 12:32:26,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.59 | bwd: 5766.85 | bwd_inner: 5753.86 | bwd_allreduce: 12.96 | step: 18.57

  5%|▍         | 1897/41250 [4:34:51<94:47:28,  8.67s/it]
                                                         
{'loss': 0.1564, 'grad_norm': 1.5228248834609985, 'learning_rate': 3.997323340307482e-05, 'epoch': 0.46}

  5%|▍         | 1897/41250 [4:34:51<94:47:28,  8.67s/it][2025-04-25 12:32:34,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:32:34,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5682.20 | bwd_inner_microstep: 5669.76 | bwd_allreduce_microstep: 12.40 | step_microstep: 18.69
[2025-04-25 12:32:34,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5682.22 | bwd_inner: 5669.76 | bwd_allreduce: 12.42 | step: 18.69

  5%|▍         | 1898/41250 [4:35:00<94:36:02,  8.65s/it]
                                                         
{'loss': 0.1372, 'grad_norm': 1.9229459762573242, 'learning_rate': 3.9973152125652304e-05, 'epoch': 0.46}

  5%|▍         | 1898/41250 [4:35:00<94:36:02,  8.65s/it][2025-04-25 12:32:43,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:32:43,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.25 | bwd_microstep: 5838.45 | bwd_inner_microstep: 5637.08 | bwd_allreduce_microstep: 201.33 | step_microstep: 18.61
[2025-04-25 12:32:43,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.25 | bwd: 5838.46 | bwd_inner: 5637.08 | bwd_allreduce: 201.35 | step: 18.61

  5%|▍         | 1899/41250 [4:35:09<94:55:33,  8.68s/it]
                                                         
{'loss': 0.204, 'grad_norm': 1.8045772314071655, 'learning_rate': 3.997307072509924e-05, 'epoch': 0.46}

  5%|▍         | 1899/41250 [4:35:09<94:55:33,  8.68s/it][2025-04-25 12:32:52,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:32:52,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.63 | bwd_microstep: 5705.53 | bwd_inner_microstep: 5692.69 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.41
[2025-04-25 12:32:52,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.63 | bwd: 5705.54 | bwd_inner: 5692.69 | bwd_allreduce: 12.81 | step: 18.41

  5%|▍         | 1900/41250 [4:35:17<94:47:53,  8.67s/it]
                                                         
{'loss': 0.1255, 'grad_norm': 1.5721091032028198, 'learning_rate': 3.997298920141612e-05, 'epoch': 0.46}

  5%|▍         | 1900/41250 [4:35:17<94:47:53,  8.67s/it][2025-04-25 12:33:00,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 12:33:00,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.67 | bwd_microstep: 5692.27 | bwd_inner_microstep: 5679.73 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.45
[2025-04-25 12:33:00,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.67 | bwd: 5692.28 | bwd_inner: 5679.73 | bwd_allreduce: 12.52 | step: 18.45

  5%|▍         | 1901/41250 [4:35:26<94:40:38,  8.66s/it]
                                                         
{'loss': 0.0896, 'grad_norm': 0.838032603263855, 'learning_rate': 3.9972907554603446e-05, 'epoch': 0.46}

  5%|▍         | 1901/41250 [4:35:26<94:40:38,  8.66s/it][2025-04-25 12:33:09,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:33:09,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.85 | bwd_microstep: 5759.16 | bwd_inner_microstep: 5641.10 | bwd_allreduce_microstep: 118.01 | step_microstep: 18.39
[2025-04-25 12:33:09,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.85 | bwd: 5759.17 | bwd_inner: 5641.10 | bwd_allreduce: 118.03 | step: 18.40

  5%|▍         | 1902/41250 [4:35:34<94:40:56,  8.66s/it]
                                                         
{'loss': 0.2496, 'grad_norm': 2.7622289657592773, 'learning_rate': 3.997282578466173e-05, 'epoch': 0.46}

  5%|▍         | 1902/41250 [4:35:34<94:40:56,  8.66s/it][2025-04-25 12:33:18,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 12:33:18,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.67 | bwd_microstep: 5675.21 | bwd_inner_microstep: 5634.88 | bwd_allreduce_microstep: 40.28 | step_microstep: 19.29
[2025-04-25 12:33:18,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.67 | bwd: 5675.22 | bwd_inner: 5634.88 | bwd_allreduce: 40.30 | step: 19.29

  5%|▍         | 1903/41250 [4:35:43<94:25:47,  8.64s/it]
                                                         
{'loss': 0.1763, 'grad_norm': 2.4240002632141113, 'learning_rate': 3.9972743891591477e-05, 'epoch': 0.46}

  5%|▍         | 1903/41250 [4:35:43<94:25:47,  8.64s/it][2025-04-25 12:33:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:33:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.32 | bwd_microstep: 5697.24 | bwd_inner_microstep: 5636.24 | bwd_allreduce_microstep: 60.96 | step_microstep: 18.85
[2025-04-25 12:33:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.32 | bwd: 5697.25 | bwd_inner: 5636.24 | bwd_allreduce: 60.97 | step: 18.85

  5%|▍         | 1904/41250 [4:35:52<94:18:14,  8.63s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 3.3921053409576416, 'learning_rate': 3.997266187539319e-05, 'epoch': 0.46}

  5%|▍         | 1904/41250 [4:35:52<94:18:14,  8.63s/it][2025-04-25 12:33:35,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.14 | optimizer_step: 0.97
[2025-04-25 12:33:35,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.15 | bwd_microstep: 5786.45 | bwd_inner_microstep: 5635.05 | bwd_allreduce_microstep: 151.34 | step_microstep: 19.32
[2025-04-25 12:33:35,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.15 | bwd: 5786.47 | bwd_inner: 5635.05 | bwd_allreduce: 151.37 | step: 19.32

  5%|▍         | 1905/41250 [4:36:00<94:29:16,  8.65s/it]
                                                         
{'loss': 0.3245, 'grad_norm': 2.1784939765930176, 'learning_rate': 3.9972579736067355e-05, 'epoch': 0.46}

  5%|▍         | 1905/41250 [4:36:00<94:29:16,  8.65s/it][2025-04-25 12:33:44,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 12:33:44,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.82 | bwd_microstep: 5846.33 | bwd_inner_microstep: 5833.53 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.03
[2025-04-25 12:33:44,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.82 | bwd: 5846.34 | bwd_inner: 5833.53 | bwd_allreduce: 12.77 | step: 19.04

  5%|▍         | 1906/41250 [4:36:09<95:12:51,  8.71s/it]
                                                         
{'loss': 0.2657, 'grad_norm': 3.4332590103149414, 'learning_rate': 3.99724974736145e-05, 'epoch': 0.46}

  5%|▍         | 1906/41250 [4:36:09<95:12:51,  8.71s/it][2025-04-25 12:33:53,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 12:33:53,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.49 | bwd_microstep: 5755.85 | bwd_inner_microstep: 5631.35 | bwd_allreduce_microstep: 124.46 | step_microstep: 18.89
[2025-04-25 12:33:53,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.49 | bwd: 5755.87 | bwd_inner: 5631.34 | bwd_allreduce: 124.48 | step: 18.89

  5%|▍         | 1907/41250 [4:36:18<95:04:36,  8.70s/it]
                                                         
{'loss': 0.1644, 'grad_norm': 1.72757089138031, 'learning_rate': 3.997241508803513e-05, 'epoch': 0.46}

  5%|▍         | 1907/41250 [4:36:18<95:04:36,  8.70s/it][2025-04-25 12:34:01,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:34:01,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.48 | bwd_microstep: 5682.31 | bwd_inner_microstep: 5669.43 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.75
[2025-04-25 12:34:01,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.48 | bwd: 5682.32 | bwd_inner: 5669.43 | bwd_allreduce: 12.85 | step: 18.75

  5%|▍         | 1908/41250 [4:36:26<94:48:41,  8.68s/it]
                                                         
{'loss': 0.2262, 'grad_norm': 1.608115553855896, 'learning_rate': 3.997233257932975e-05, 'epoch': 0.46}

  5%|▍         | 1908/41250 [4:36:26<94:48:41,  8.68s/it][2025-04-25 12:34:10,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:34:10,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.51 | bwd_microstep: 5775.04 | bwd_inner_microstep: 5640.05 | bwd_allreduce_microstep: 134.95 | step_microstep: 18.28
[2025-04-25 12:34:10,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.51 | bwd: 5775.05 | bwd_inner: 5640.05 | bwd_allreduce: 134.96 | step: 18.29

  5%|▍         | 1909/41250 [4:36:35<94:50:22,  8.68s/it]
                                                         
{'loss': 0.0388, 'grad_norm': 0.6560807824134827, 'learning_rate': 3.997224994749886e-05, 'epoch': 0.46}

  5%|▍         | 1909/41250 [4:36:35<94:50:22,  8.68s/it][2025-04-25 12:34:19,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:34:19,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5735.76 | bwd_inner_microstep: 5673.50 | bwd_allreduce_microstep: 62.22 | step_microstep: 18.43
[2025-04-25 12:34:19,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5735.77 | bwd_inner: 5673.50 | bwd_allreduce: 62.23 | step: 18.44

  5%|▍         | 1910/41250 [4:36:44<94:48:42,  8.68s/it]
                                                         
{'loss': 0.1416, 'grad_norm': 2.711060047149658, 'learning_rate': 3.997216719254298e-05, 'epoch': 0.46}

  5%|▍         | 1910/41250 [4:36:44<94:48:42,  8.68s/it][2025-04-25 12:34:27,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:34:27,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.26 | bwd_microstep: 5786.56 | bwd_inner_microstep: 5639.76 | bwd_allreduce_microstep: 146.76 | step_microstep: 18.39
[2025-04-25 12:34:27,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.26 | bwd: 5786.57 | bwd_inner: 5639.76 | bwd_allreduce: 146.77 | step: 18.40

  5%|▍         | 1911/41250 [4:36:53<94:51:31,  8.68s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 1.4802038669586182, 'learning_rate': 3.997208431446262e-05, 'epoch': 0.46}

  5%|▍         | 1911/41250 [4:36:53<94:51:31,  8.68s/it][2025-04-25 12:34:36,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:34:36,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.06 | bwd_microstep: 5698.39 | bwd_inner_microstep: 5685.66 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.59
[2025-04-25 12:34:36,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.06 | bwd: 5698.40 | bwd_inner: 5685.66 | bwd_allreduce: 12.71 | step: 18.59

  5%|▍         | 1912/41250 [4:37:01<94:42:47,  8.67s/it]
                                                         
{'loss': 0.2899, 'grad_norm': 1.6469272375106812, 'learning_rate': 3.997200131325829e-05, 'epoch': 0.46}

  5%|▍         | 1912/41250 [4:37:01<94:42:47,  8.67s/it][2025-04-25 12:34:44,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 12:34:44,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.00 | bwd_microstep: 5691.94 | bwd_inner_microstep: 5657.32 | bwd_allreduce_microstep: 34.58 | step_microstep: 18.34
[2025-04-25 12:34:44,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.00 | bwd: 5691.96 | bwd_inner: 5657.32 | bwd_allreduce: 34.60 | step: 18.34

  5%|▍         | 1913/41250 [4:37:10<94:31:07,  8.65s/it]
                                                         
{'loss': 0.1082, 'grad_norm': 2.072213649749756, 'learning_rate': 3.99719181889305e-05, 'epoch': 0.46}

  5%|▍         | 1913/41250 [4:37:10<94:31:07,  8.65s/it][2025-04-25 12:34:53,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:34:53,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.41 | bwd_microstep: 5750.19 | bwd_inner_microstep: 5713.42 | bwd_allreduce_microstep: 36.73 | step_microstep: 17.90
[2025-04-25 12:34:53,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.41 | bwd: 5750.20 | bwd_inner: 5713.42 | bwd_allreduce: 36.75 | step: 17.90

  5%|▍         | 1914/41250 [4:37:18<94:36:44,  8.66s/it]
                                                         
{'loss': 0.1651, 'grad_norm': 1.4309744834899902, 'learning_rate': 3.997183494147976e-05, 'epoch': 0.46}

  5%|▍         | 1914/41250 [4:37:18<94:36:44,  8.66s/it][2025-04-25 12:35:02,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 12:35:02,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.57 | bwd_microstep: 5755.17 | bwd_inner_microstep: 5686.03 | bwd_allreduce_microstep: 69.09 | step_microstep: 18.83
[2025-04-25 12:35:02,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.57 | bwd: 5755.19 | bwd_inner: 5686.03 | bwd_allreduce: 69.11 | step: 18.84

  5%|▍         | 1915/41250 [4:37:27<94:44:23,  8.67s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 1.1262321472167969, 'learning_rate': 3.997175157090659e-05, 'epoch': 0.46}

  5%|▍         | 1915/41250 [4:37:27<94:44:23,  8.67s/it][2025-04-25 12:35:10,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:35:10,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5706.32 | bwd_inner_microstep: 5693.54 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.48
[2025-04-25 12:35:10,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5706.33 | bwd_inner: 5693.54 | bwd_allreduce: 12.75 | step: 18.48

  5%|▍         | 1916/41250 [4:37:36<94:38:18,  8.66s/it]
                                                         
{'loss': 0.1428, 'grad_norm': 1.398017168045044, 'learning_rate': 3.99716680772115e-05, 'epoch': 0.46}

  5%|▍         | 1916/41250 [4:37:36<94:38:18,  8.66s/it][2025-04-25 12:35:19,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:35:19,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.49 | bwd_microstep: 5770.61 | bwd_inner_microstep: 5661.26 | bwd_allreduce_microstep: 109.30 | step_microstep: 18.78
[2025-04-25 12:35:19,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.49 | bwd: 5770.62 | bwd_inner: 5661.26 | bwd_allreduce: 109.32 | step: 18.78

  5%|▍         | 1917/41250 [4:37:44<94:42:32,  8.67s/it]
                                                         
{'loss': 0.2322, 'grad_norm': 2.338526964187622, 'learning_rate': 3.9971584460395e-05, 'epoch': 0.46}

  5%|▍         | 1917/41250 [4:37:44<94:42:32,  8.67s/it][2025-04-25 12:35:28,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.14 | optimizer_step: 0.96
[2025-04-25 12:35:28,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.60 | bwd_microstep: 5775.25 | bwd_inner_microstep: 5667.03 | bwd_allreduce_microstep: 108.17 | step_microstep: 19.01
[2025-04-25 12:35:28,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.60 | bwd: 5775.27 | bwd_inner: 5667.03 | bwd_allreduce: 108.19 | step: 19.01

  5%|▍         | 1918/41250 [4:37:53<94:48:25,  8.68s/it]
                                                         
{'loss': 0.125, 'grad_norm': 1.1435123682022095, 'learning_rate': 3.997150072045761e-05, 'epoch': 0.46}

  5%|▍         | 1918/41250 [4:37:53<94:48:25,  8.68s/it][2025-04-25 12:35:37,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 12:35:37,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.38 | bwd_microstep: 5862.26 | bwd_inner_microstep: 5712.30 | bwd_allreduce_microstep: 149.92 | step_microstep: 18.80
[2025-04-25 12:35:37,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.38 | bwd: 5862.27 | bwd_inner: 5712.30 | bwd_allreduce: 149.94 | step: 18.80

  5%|▍         | 1919/41250 [4:38:02<95:12:49,  8.72s/it]
                                                         
{'loss': 0.3482, 'grad_norm': 2.605591058731079, 'learning_rate': 3.997141685739985e-05, 'epoch': 0.47}

  5%|▍         | 1919/41250 [4:38:02<95:12:49,  8.72s/it][2025-04-25 12:35:45,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 0.92
[2025-04-25 12:35:45,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.75 | bwd_microstep: 5715.74 | bwd_inner_microstep: 5662.39 | bwd_allreduce_microstep: 53.30 | step_microstep: 19.66
[2025-04-25 12:35:45,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.75 | bwd: 5715.75 | bwd_inner: 5662.39 | bwd_allreduce: 53.32 | step: 19.67

  5%|▍         | 1920/41250 [4:38:11<94:56:11,  8.69s/it]
                                                         
{'loss': 0.1394, 'grad_norm': 1.076276421546936, 'learning_rate': 3.997133287122223e-05, 'epoch': 0.47}

  5%|▍         | 1920/41250 [4:38:11<94:56:11,  8.69s/it][2025-04-25 12:35:54,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:35:54,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.80 | bwd_microstep: 5772.22 | bwd_inner_microstep: 5677.15 | bwd_allreduce_microstep: 95.03 | step_microstep: 17.75
[2025-04-25 12:35:54,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.80 | bwd: 5772.23 | bwd_inner: 5677.15 | bwd_allreduce: 95.04 | step: 17.75

  5%|▍         | 1921/41250 [4:38:19<94:55:53,  8.69s/it]
                                                         
{'loss': 0.0765, 'grad_norm': 1.3676832914352417, 'learning_rate': 3.997124876192527e-05, 'epoch': 0.47}

  5%|▍         | 1921/41250 [4:38:19<94:55:53,  8.69s/it][2025-04-25 12:36:03,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:36:03,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.97 | bwd_microstep: 5793.55 | bwd_inner_microstep: 5668.24 | bwd_allreduce_microstep: 125.27 | step_microstep: 18.86
[2025-04-25 12:36:03,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.97 | bwd: 5793.56 | bwd_inner: 5668.24 | bwd_allreduce: 125.29 | step: 18.86

  5%|▍         | 1922/41250 [4:38:28<94:59:46,  8.70s/it]
                                                         
{'loss': 0.4116, 'grad_norm': 3.489433765411377, 'learning_rate': 3.997116452950949e-05, 'epoch': 0.47}

  5%|▍         | 1922/41250 [4:38:28<94:59:46,  8.70s/it][2025-04-25 12:36:11,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:36:11,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.42 | bwd_microstep: 5733.91 | bwd_inner_microstep: 5721.20 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.07
[2025-04-25 12:36:11,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.42 | bwd: 5733.92 | bwd_inner: 5721.20 | bwd_allreduce: 12.68 | step: 18.07

  5%|▍         | 1923/41250 [4:38:37<94:57:50,  8.69s/it]
                                                         
{'loss': 0.5053, 'grad_norm': 3.4594674110412598, 'learning_rate': 3.997108017397542e-05, 'epoch': 0.47}

  5%|▍         | 1923/41250 [4:38:37<94:57:50,  8.69s/it][2025-04-25 12:36:20,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:36:20,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.29 | bwd_microstep: 5695.00 | bwd_inner_microstep: 5674.56 | bwd_allreduce_microstep: 20.40 | step_microstep: 18.95
[2025-04-25 12:36:20,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.29 | bwd: 5695.01 | bwd_inner: 5674.56 | bwd_allreduce: 20.41 | step: 18.95

  5%|▍         | 1924/41250 [4:38:45<94:44:31,  8.67s/it]
                                                         
{'loss': 0.2528, 'grad_norm': 1.6329442262649536, 'learning_rate': 3.997099569532355e-05, 'epoch': 0.47}

  5%|▍         | 1924/41250 [4:38:45<94:44:31,  8.67s/it][2025-04-25 12:36:29,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.10 | optimizer_step: 1.18
[2025-04-25 12:36:29,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.12 | bwd_microstep: 5753.93 | bwd_inner_microstep: 5714.88 | bwd_allreduce_microstep: 38.98 | step_microstep: 19.74
[2025-04-25 12:36:29,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.12 | bwd: 5753.94 | bwd_inner: 5714.88 | bwd_allreduce: 39.01 | step: 19.74

  5%|▍         | 1925/41250 [4:38:54<94:48:26,  8.68s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 1.6445331573486328, 'learning_rate': 3.997091109355443e-05, 'epoch': 0.47}

  5%|▍         | 1925/41250 [4:38:54<94:48:26,  8.68s/it][2025-04-25 12:36:37,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:36:37,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.64 | bwd_microstep: 5729.22 | bwd_inner_microstep: 5716.54 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.34
[2025-04-25 12:36:37,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.64 | bwd: 5729.23 | bwd_inner: 5716.54 | bwd_allreduce: 12.65 | step: 18.35

  5%|▍         | 1926/41250 [4:39:03<94:46:54,  8.68s/it]
                                                         
{'loss': 0.2825, 'grad_norm': 1.732927918434143, 'learning_rate': 3.9970826368668574e-05, 'epoch': 0.47}

  5%|▍         | 1926/41250 [4:39:03<94:46:54,  8.68s/it][2025-04-25 12:36:46,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:36:46,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.56 | bwd_microstep: 5752.01 | bwd_inner_microstep: 5704.96 | bwd_allreduce_microstep: 47.00 | step_microstep: 18.25
[2025-04-25 12:36:46,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.56 | bwd: 5752.02 | bwd_inner: 5704.96 | bwd_allreduce: 47.02 | step: 18.26

  5%|▍         | 1927/41250 [4:39:11<94:52:23,  8.69s/it]
                                                         
{'loss': 0.1092, 'grad_norm': 1.4300307035446167, 'learning_rate': 3.9970741520666496e-05, 'epoch': 0.47}

  5%|▍         | 1927/41250 [4:39:11<94:52:23,  8.69s/it][2025-04-25 12:36:55,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:36:55,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.22 | bwd_microstep: 5744.71 | bwd_inner_microstep: 5712.90 | bwd_allreduce_microstep: 31.76 | step_microstep: 18.42
[2025-04-25 12:36:55,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.22 | bwd: 5744.72 | bwd_inner: 5712.90 | bwd_allreduce: 31.78 | step: 18.42

  5%|▍         | 1928/41250 [4:39:20<94:52:26,  8.69s/it]
                                                         
{'loss': 0.2638, 'grad_norm': 2.1112310886383057, 'learning_rate': 3.997065654954872e-05, 'epoch': 0.47}

  5%|▍         | 1928/41250 [4:39:20<94:52:26,  8.69s/it][2025-04-25 12:37:03,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:37:03,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5779.68 | bwd_inner_microstep: 5666.91 | bwd_allreduce_microstep: 112.73 | step_microstep: 18.14
[2025-04-25 12:37:03,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5779.69 | bwd_inner: 5666.91 | bwd_allreduce: 112.74 | step: 18.14

  5%|▍         | 1929/41250 [4:39:29<94:53:49,  8.69s/it]
                                                         
{'loss': 0.0309, 'grad_norm': 1.9369864463806152, 'learning_rate': 3.997057145531578e-05, 'epoch': 0.47}

  5%|▍         | 1929/41250 [4:39:29<94:53:49,  8.69s/it][2025-04-25 12:37:12,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:37:12,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5696.52 | bwd_inner_microstep: 5663.56 | bwd_allreduce_microstep: 32.92 | step_microstep: 18.37
[2025-04-25 12:37:12,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.49 | bwd: 5696.54 | bwd_inner: 5663.56 | bwd_allreduce: 32.94 | step: 18.38

  5%|▍         | 1930/41250 [4:39:37<94:38:11,  8.66s/it]
                                                         
{'loss': 0.3095, 'grad_norm': 2.009230375289917, 'learning_rate': 3.99704862379682e-05, 'epoch': 0.47}

  5%|▍         | 1930/41250 [4:39:37<94:38:11,  8.66s/it][2025-04-25 12:37:21,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:37:21,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.06 | bwd_microstep: 5709.33 | bwd_inner_microstep: 5696.67 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.36
[2025-04-25 12:37:21,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.06 | bwd: 5709.34 | bwd_inner: 5696.67 | bwd_allreduce: 12.63 | step: 18.36

  5%|▍         | 1931/41250 [4:39:46<94:33:44,  8.66s/it]
                                                         
{'loss': 0.0583, 'grad_norm': 0.7715359330177307, 'learning_rate': 3.9970400897506495e-05, 'epoch': 0.47}

  5%|▍         | 1931/41250 [4:39:46<94:33:44,  8.66s/it][2025-04-25 12:37:29,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 1.09
[2025-04-25 12:37:29,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.27 | bwd_microstep: 5691.95 | bwd_inner_microstep: 5657.91 | bwd_allreduce_microstep: 34.00 | step_microstep: 18.50
[2025-04-25 12:37:29,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.27 | bwd: 5691.97 | bwd_inner: 5657.91 | bwd_allreduce: 34.02 | step: 18.50

  5%|▍         | 1932/41250 [4:39:55<94:25:42,  8.65s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 2.4166195392608643, 'learning_rate': 3.99703154339312e-05, 'epoch': 0.47}

  5%|▍         | 1932/41250 [4:39:55<94:25:42,  8.65s/it][2025-04-25 12:37:38,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 12:37:38,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.55 | bwd_microstep: 5685.18 | bwd_inner_microstep: 5648.90 | bwd_allreduce_microstep: 36.23 | step_microstep: 18.85
[2025-04-25 12:37:38,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.55 | bwd: 5685.19 | bwd_inner: 5648.90 | bwd_allreduce: 36.25 | step: 18.85

  5%|▍         | 1933/41250 [4:40:03<94:17:21,  8.63s/it]
                                                         
{'loss': 0.2293, 'grad_norm': 1.507620096206665, 'learning_rate': 3.9970229847242833e-05, 'epoch': 0.47}

  5%|▍         | 1933/41250 [4:40:03<94:17:21,  8.63s/it][2025-04-25 12:37:47,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:37:47,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5696.59 | bwd_inner_microstep: 5683.86 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.48
[2025-04-25 12:37:47,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.03 | bwd: 5696.60 | bwd_inner: 5683.86 | bwd_allreduce: 12.71 | step: 18.48

  5%|▍         | 1934/41250 [4:40:12<94:17:37,  8.63s/it]
                                                         
{'loss': 0.3849, 'grad_norm': 2.3523926734924316, 'learning_rate': 3.997014413744193e-05, 'epoch': 0.47}

  5%|▍         | 1934/41250 [4:40:12<94:17:37,  8.63s/it][2025-04-25 12:37:55,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:37:55,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.86 | bwd_microstep: 5723.68 | bwd_inner_microstep: 5710.93 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.51
[2025-04-25 12:37:55,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.86 | bwd: 5723.69 | bwd_inner: 5710.93 | bwd_allreduce: 12.72 | step: 18.52

  5%|▍         | 1935/41250 [4:40:21<94:24:43,  8.65s/it]
                                                         
{'loss': 0.2771, 'grad_norm': 3.315139055252075, 'learning_rate': 3.997005830452901e-05, 'epoch': 0.47}

  5%|▍         | 1935/41250 [4:40:21<94:24:43,  8.65s/it][2025-04-25 12:38:04,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 12:38:04,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.67 | bwd_microstep: 5689.07 | bwd_inner_microstep: 5651.24 | bwd_allreduce_microstep: 37.78 | step_microstep: 19.19
[2025-04-25 12:38:04,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.67 | bwd: 5689.08 | bwd_inner: 5651.24 | bwd_allreduce: 37.80 | step: 19.19

  5%|▍         | 1936/41250 [4:40:29<94:18:31,  8.64s/it]
                                                         
{'loss': 0.1471, 'grad_norm': 1.558579444885254, 'learning_rate': 3.996997234850461e-05, 'epoch': 0.47}

  5%|▍         | 1936/41250 [4:40:29<94:18:31,  8.64s/it][2025-04-25 12:38:13,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:38:13,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5713.47 | bwd_inner_microstep: 5700.59 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.86
[2025-04-25 12:38:13,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5713.48 | bwd_inner: 5700.59 | bwd_allreduce: 12.86 | step: 18.87

  5%|▍         | 1937/41250 [4:40:38<94:20:30,  8.64s/it]
                                                         
{'loss': 0.1449, 'grad_norm': 1.596052885055542, 'learning_rate': 3.996988626936927e-05, 'epoch': 0.47}

  5%|▍         | 1937/41250 [4:40:38<94:20:30,  8.64s/it][2025-04-25 12:38:21,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:38:21,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5705.83 | bwd_inner_microstep: 5692.97 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.87
[2025-04-25 12:38:21,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5705.85 | bwd_inner: 5692.97 | bwd_allreduce: 12.83 | step: 18.87

  5%|▍         | 1938/41250 [4:40:46<94:19:51,  8.64s/it]
                                                         
{'loss': 0.3321, 'grad_norm': 2.056281566619873, 'learning_rate': 3.9969800067123503e-05, 'epoch': 0.47}

  5%|▍         | 1938/41250 [4:40:46<94:19:51,  8.64s/it][2025-04-25 12:38:30,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 12:38:30,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.63 | bwd_microstep: 5878.06 | bwd_inner_microstep: 5643.31 | bwd_allreduce_microstep: 234.70 | step_microstep: 18.94
[2025-04-25 12:38:30,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.63 | bwd: 5878.07 | bwd_inner: 5643.31 | bwd_allreduce: 234.72 | step: 18.94

  5%|▍         | 1939/41250 [4:40:55<94:50:56,  8.69s/it]
                                                         
{'loss': 0.1132, 'grad_norm': 0.801078200340271, 'learning_rate': 3.9969713741767847e-05, 'epoch': 0.47}

  5%|▍         | 1939/41250 [4:40:55<94:50:56,  8.69s/it][2025-04-25 12:38:39,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:38:39,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.72 | bwd_microstep: 5714.50 | bwd_inner_microstep: 5701.81 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.62
[2025-04-25 12:38:39,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.72 | bwd: 5714.52 | bwd_inner: 5701.81 | bwd_allreduce: 12.66 | step: 18.62

  5%|▍         | 1940/41250 [4:41:04<94:44:57,  8.68s/it]
                                                         
{'loss': 0.0948, 'grad_norm': 1.703294277191162, 'learning_rate': 3.996962729330283e-05, 'epoch': 0.47}

  5%|▍         | 1940/41250 [4:41:04<94:44:57,  8.68s/it][2025-04-25 12:38:47,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.14 | optimizer_step: 1.01
[2025-04-25 12:38:47,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.80 | bwd_microstep: 5684.71 | bwd_inner_microstep: 5658.39 | bwd_allreduce_microstep: 26.27 | step_microstep: 19.53
[2025-04-25 12:38:47,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.80 | bwd: 5684.72 | bwd_inner: 5658.39 | bwd_allreduce: 26.30 | step: 19.54

  5%|▍         | 1941/41250 [4:41:13<94:30:18,  8.65s/it]
                                                         
{'loss': 0.3987, 'grad_norm': 2.511631727218628, 'learning_rate': 3.9969540721728985e-05, 'epoch': 0.47}

  5%|▍         | 1941/41250 [4:41:13<94:30:18,  8.65s/it][2025-04-25 12:38:56,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.14 | optimizer_step: 0.93
[2025-04-25 12:38:56,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5732.35 | bwd_inner_microstep: 5697.29 | bwd_allreduce_microstep: 35.01 | step_microstep: 19.10
[2025-04-25 12:38:56,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5732.37 | bwd_inner: 5697.29 | bwd_allreduce: 35.03 | step: 19.10

  5%|▍         | 1942/41250 [4:41:21<94:32:36,  8.66s/it]
                                                         
{'loss': 0.145, 'grad_norm': 1.0408008098602295, 'learning_rate': 3.996945402704686e-05, 'epoch': 0.47}

  5%|▍         | 1942/41250 [4:41:21<94:32:36,  8.66s/it][2025-04-25 12:39:04,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:39:04,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.29 | bwd_microstep: 5696.80 | bwd_inner_microstep: 5633.09 | bwd_allreduce_microstep: 63.67 | step_microstep: 18.60
[2025-04-25 12:39:04,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.29 | bwd: 5696.82 | bwd_inner: 5633.09 | bwd_allreduce: 63.69 | step: 18.61

  5%|▍         | 1943/41250 [4:41:30<94:20:11,  8.64s/it]
                                                         
{'loss': 0.4788, 'grad_norm': 2.0861103534698486, 'learning_rate': 3.996936720925697e-05, 'epoch': 0.47}

  5%|▍         | 1943/41250 [4:41:30<94:20:11,  8.64s/it][2025-04-25 12:39:13,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:39:13,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5713.23 | bwd_inner_microstep: 5700.29 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.05
[2025-04-25 12:39:13,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5713.25 | bwd_inner: 5700.29 | bwd_allreduce: 12.91 | step: 19.05

  5%|▍         | 1944/41250 [4:41:38<94:20:49,  8.64s/it]
                                                         
{'loss': 0.2538, 'grad_norm': 2.0408411026000977, 'learning_rate': 3.996928026835986e-05, 'epoch': 0.47}

  5%|▍         | 1944/41250 [4:41:38<94:20:49,  8.64s/it][2025-04-25 12:39:22,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 12:39:22,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5947.26 | bwd_inner_microstep: 5699.54 | bwd_allreduce_microstep: 247.68 | step_microstep: 18.89
[2025-04-25 12:39:22,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5947.28 | bwd_inner: 5699.54 | bwd_allreduce: 247.70 | step: 18.89

  5%|▍         | 1945/41250 [4:41:47<95:07:39,  8.71s/it]
                                                         
{'loss': 0.3028, 'grad_norm': 2.673358201980591, 'learning_rate': 3.996919320435607e-05, 'epoch': 0.47}

  5%|▍         | 1945/41250 [4:41:47<95:07:39,  8.71s/it][2025-04-25 12:39:31,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:39:31,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.18 | bwd_microstep: 5698.70 | bwd_inner_microstep: 5685.92 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.38
[2025-04-25 12:39:31,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.18 | bwd: 5698.71 | bwd_inner: 5685.92 | bwd_allreduce: 12.75 | step: 18.38

  5%|▍         | 1946/41250 [4:41:56<94:49:17,  8.69s/it]
                                                         
{'loss': 0.2476, 'grad_norm': 0.9208199977874756, 'learning_rate': 3.9969106017246126e-05, 'epoch': 0.47}

  5%|▍         | 1946/41250 [4:41:56<94:49:17,  8.69s/it][2025-04-25 12:39:39,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 12:39:39,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.87 | bwd_microstep: 5719.12 | bwd_inner_microstep: 5683.68 | bwd_allreduce_microstep: 35.39 | step_microstep: 18.86
[2025-04-25 12:39:39,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.87 | bwd: 5719.14 | bwd_inner: 5683.68 | bwd_allreduce: 35.41 | step: 18.87

  5%|▍         | 1947/41250 [4:42:05<94:42:11,  8.67s/it]
                                                         
{'loss': 0.3666, 'grad_norm': 2.26572585105896, 'learning_rate': 3.996901870703058e-05, 'epoch': 0.47}

  5%|▍         | 1947/41250 [4:42:05<94:42:11,  8.67s/it][2025-04-25 12:39:48,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:39:48,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.22 | bwd_microstep: 5727.32 | bwd_inner_microstep: 5636.16 | bwd_allreduce_microstep: 91.11 | step_microstep: 18.70
[2025-04-25 12:39:48,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.22 | bwd: 5727.34 | bwd_inner: 5636.16 | bwd_allreduce: 91.13 | step: 18.70

  5%|▍         | 1948/41250 [4:42:13<94:35:24,  8.66s/it]
                                                         
{'loss': 0.2274, 'grad_norm': 1.45513117313385, 'learning_rate': 3.996893127370995e-05, 'epoch': 0.47}

  5%|▍         | 1948/41250 [4:42:13<94:35:24,  8.66s/it][2025-04-25 12:39:57,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:39:57,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.82 | bwd_microstep: 5698.27 | bwd_inner_microstep: 5685.40 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.53
[2025-04-25 12:39:57,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.82 | bwd: 5698.28 | bwd_inner: 5685.40 | bwd_allreduce: 12.84 | step: 18.53

  5%|▍         | 1949/41250 [4:42:22<94:28:09,  8.65s/it]
                                                         
{'loss': 0.2995, 'grad_norm': 1.4507622718811035, 'learning_rate': 3.996884371728479e-05, 'epoch': 0.47}

  5%|▍         | 1949/41250 [4:42:22<94:28:09,  8.65s/it][2025-04-25 12:40:05,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:40:05,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.05 | bwd_microstep: 5692.63 | bwd_inner_microstep: 5631.33 | bwd_allreduce_microstep: 61.26 | step_microstep: 18.30
[2025-04-25 12:40:05,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.05 | bwd: 5692.64 | bwd_inner: 5631.33 | bwd_allreduce: 61.27 | step: 18.31

  5%|▍         | 1950/41250 [4:42:30<94:18:47,  8.64s/it]
                                                         
{'loss': 0.1691, 'grad_norm': 1.5402271747589111, 'learning_rate': 3.996875603775564e-05, 'epoch': 0.47}

  5%|▍         | 1950/41250 [4:42:30<94:18:47,  8.64s/it][2025-04-25 12:40:14,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 12:40:14,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.22 | bwd_microstep: 5738.13 | bwd_inner_microstep: 5633.17 | bwd_allreduce_microstep: 104.92 | step_microstep: 18.47
[2025-04-25 12:40:14,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.22 | bwd: 5738.15 | bwd_inner: 5633.17 | bwd_allreduce: 104.94 | step: 18.47

  5%|▍         | 1951/41250 [4:42:39<94:20:03,  8.64s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 1.3550423383712769, 'learning_rate': 3.996866823512303e-05, 'epoch': 0.47}

  5%|▍         | 1951/41250 [4:42:39<94:20:03,  8.64s/it][2025-04-25 12:40:22,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:40:22,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.35 | bwd_microstep: 5745.56 | bwd_inner_microstep: 5628.47 | bwd_allreduce_microstep: 117.05 | step_microstep: 18.44
[2025-04-25 12:40:22,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.35 | bwd: 5745.57 | bwd_inner: 5628.47 | bwd_allreduce: 117.06 | step: 18.44

  5%|▍         | 1952/41250 [4:42:48<94:24:18,  8.65s/it]
                                                         
{'loss': 0.2265, 'grad_norm': 1.2793281078338623, 'learning_rate': 3.996858030938752e-05, 'epoch': 0.47}

  5%|▍         | 1952/41250 [4:42:48<94:24:18,  8.65s/it][2025-04-25 12:40:31,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 12:40:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5712.75 | bwd_inner_microstep: 5700.31 | bwd_allreduce_microstep: 12.39 | step_microstep: 18.84
[2025-04-25 12:40:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.67 | bwd: 5712.76 | bwd_inner: 5700.31 | bwd_allreduce: 12.41 | step: 18.85

  5%|▍         | 1953/41250 [4:42:56<94:23:12,  8.65s/it]
                                                         
{'loss': 0.3548, 'grad_norm': 1.8909658193588257, 'learning_rate': 3.9968492260549625e-05, 'epoch': 0.47}

  5%|▍         | 1953/41250 [4:42:56<94:23:12,  8.65s/it][2025-04-25 12:40:40,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-25 12:40:40,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.87 | bwd_microstep: 5736.45 | bwd_inner_microstep: 5639.10 | bwd_allreduce_microstep: 97.31 | step_microstep: 19.08
[2025-04-25 12:40:40,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.87 | bwd: 5736.47 | bwd_inner: 5639.10 | bwd_allreduce: 97.33 | step: 19.08

  5%|▍         | 1954/41250 [4:43:05<94:23:30,  8.65s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 1.5520168542861938, 'learning_rate': 3.996840408860991e-05, 'epoch': 0.47}

  5%|▍         | 1954/41250 [4:43:05<94:23:30,  8.65s/it][2025-04-25 12:40:48,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 12:40:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.71 | bwd_microstep: 5747.81 | bwd_inner_microstep: 5692.55 | bwd_allreduce_microstep: 55.21 | step_microstep: 18.73
[2025-04-25 12:40:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.71 | bwd: 5747.83 | bwd_inner: 5692.55 | bwd_allreduce: 55.23 | step: 18.73

  5%|▍         | 1955/41250 [4:43:14<94:28:41,  8.66s/it]
                                                         
{'loss': 0.1956, 'grad_norm': 1.4170116186141968, 'learning_rate': 3.9968315793568915e-05, 'epoch': 0.47}

  5%|▍         | 1955/41250 [4:43:14<94:28:41,  8.66s/it][2025-04-25 12:40:57,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 12:40:57,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3053.10 | bwd_microstep: 5695.85 | bwd_inner_microstep: 5683.04 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.36
[2025-04-25 12:40:57,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3053.10 | bwd: 5695.86 | bwd_inner: 5683.04 | bwd_allreduce: 12.78 | step: 18.37

  5%|▍         | 1956/41250 [4:43:23<95:04:35,  8.71s/it]
                                                         
{'loss': 0.2148, 'grad_norm': 1.5788754224777222, 'learning_rate': 3.9968227375427176e-05, 'epoch': 0.47}

  5%|▍         | 1956/41250 [4:43:23<95:04:35,  8.71s/it][2025-04-25 12:41:06,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 12:41:06,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.10 | bwd_microstep: 5711.11 | bwd_inner_microstep: 5673.70 | bwd_allreduce_microstep: 37.37 | step_microstep: 18.49
[2025-04-25 12:41:06,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.10 | bwd: 5711.12 | bwd_inner: 5673.70 | bwd_allreduce: 37.38 | step: 18.49

  5%|▍         | 1957/41250 [4:43:31<94:52:24,  8.69s/it]
                                                         
{'loss': 0.1657, 'grad_norm': 1.4730521440505981, 'learning_rate': 3.9968138834185245e-05, 'epoch': 0.47}

  5%|▍         | 1957/41250 [4:43:31<94:52:24,  8.69s/it][2025-04-25 12:41:14,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:41:14,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.38 | bwd_microstep: 5657.49 | bwd_inner_microstep: 5631.83 | bwd_allreduce_microstep: 25.61 | step_microstep: 18.32
[2025-04-25 12:41:14,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.38 | bwd: 5657.50 | bwd_inner: 5631.83 | bwd_allreduce: 25.63 | step: 18.32

  5%|▍         | 1958/41250 [4:43:40<94:27:55,  8.66s/it]
                                                         
{'loss': 0.2503, 'grad_norm': 1.2448222637176514, 'learning_rate': 3.996805016984366e-05, 'epoch': 0.47}

  5%|▍         | 1958/41250 [4:43:40<94:27:55,  8.66s/it][2025-04-25 12:41:23,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:41:23,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.12 | bwd_microstep: 5741.55 | bwd_inner_microstep: 5645.40 | bwd_allreduce_microstep: 96.10 | step_microstep: 18.75
[2025-04-25 12:41:23,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.12 | bwd: 5741.57 | bwd_inner: 5645.40 | bwd_allreduce: 96.12 | step: 18.75

  5%|▍         | 1959/41250 [4:43:48<94:27:14,  8.65s/it]
                                                         
{'loss': 0.2877, 'grad_norm': 2.2546188831329346, 'learning_rate': 3.996796138240298e-05, 'epoch': 0.47}

  5%|▍         | 1959/41250 [4:43:48<94:27:14,  8.65s/it][2025-04-25 12:41:32,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 12:41:32,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.95 | bwd_microstep: 5711.56 | bwd_inner_microstep: 5668.97 | bwd_allreduce_microstep: 42.55 | step_microstep: 18.93
[2025-04-25 12:41:32,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.95 | bwd: 5711.57 | bwd_inner: 5668.97 | bwd_allreduce: 42.56 | step: 18.93

  5%|▍         | 1960/41250 [4:43:57<94:25:45,  8.65s/it]
                                                         
{'loss': 0.1428, 'grad_norm': 0.8485822081565857, 'learning_rate': 3.996787247186374e-05, 'epoch': 0.48}

  5%|▍         | 1960/41250 [4:43:57<94:25:45,  8.65s/it][2025-04-25 12:41:40,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:41:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5688.35 | bwd_inner_microstep: 5675.79 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.43
[2025-04-25 12:41:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5688.37 | bwd_inner: 5675.79 | bwd_allreduce: 12.54 | step: 18.44

  5%|▍         | 1961/41250 [4:44:06<94:20:33,  8.64s/it]
                                                         
{'loss': 0.2386, 'grad_norm': 1.7503352165222168, 'learning_rate': 3.9967783438226496e-05, 'epoch': 0.48}

  5%|▍         | 1961/41250 [4:44:06<94:20:33,  8.64s/it][2025-04-25 12:41:49,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 12:41:49,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.61 | bwd_microstep: 5714.02 | bwd_inner_microstep: 5701.44 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.59
[2025-04-25 12:41:49,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.61 | bwd: 5714.03 | bwd_inner: 5701.44 | bwd_allreduce: 12.56 | step: 18.60

  5%|▍         | 1962/41250 [4:44:14<94:25:12,  8.65s/it]
                                                         
{'loss': 0.2283, 'grad_norm': 1.1009578704833984, 'learning_rate': 3.99676942814918e-05, 'epoch': 0.48}

  5%|▍         | 1962/41250 [4:44:14<94:25:12,  8.65s/it][2025-04-25 12:41:58,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:41:58,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.99 | bwd_microstep: 5766.05 | bwd_inner_microstep: 5655.32 | bwd_allreduce_microstep: 110.67 | step_microstep: 18.42
[2025-04-25 12:41:58,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.99 | bwd: 5766.06 | bwd_inner: 5655.32 | bwd_allreduce: 110.69 | step: 18.42

  5%|▍         | 1963/41250 [4:44:23<94:29:44,  8.66s/it]
                                                         
{'loss': 0.1594, 'grad_norm': 1.0215516090393066, 'learning_rate': 3.996760500166019e-05, 'epoch': 0.48}

  5%|▍         | 1963/41250 [4:44:23<94:29:44,  8.66s/it][2025-04-25 12:42:06,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 12:42:06,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.95 | bwd_microstep: 5699.24 | bwd_inner_microstep: 5653.75 | bwd_allreduce_microstep: 45.45 | step_microstep: 18.98
[2025-04-25 12:42:06,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.95 | bwd: 5699.25 | bwd_inner: 5653.74 | bwd_allreduce: 45.47 | step: 18.98

  5%|▍         | 1964/41250 [4:44:32<94:20:38,  8.65s/it]
                                                         
{'loss': 0.101, 'grad_norm': 0.7155169248580933, 'learning_rate': 3.9967515598732226e-05, 'epoch': 0.48}

  5%|▍         | 1964/41250 [4:44:32<94:20:38,  8.65s/it][2025-04-25 12:42:15,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 12:42:15,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.56 | bwd_microstep: 5740.99 | bwd_inner_microstep: 5710.57 | bwd_allreduce_microstep: 30.37 | step_microstep: 18.69
[2025-04-25 12:42:15,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.56 | bwd: 5741.00 | bwd_inner: 5710.57 | bwd_allreduce: 30.39 | step: 18.69

  5%|▍         | 1965/41250 [4:44:40<94:27:50,  8.66s/it]
                                                         
{'loss': 0.0651, 'grad_norm': 0.5783810615539551, 'learning_rate': 3.996742607270845e-05, 'epoch': 0.48}

  5%|▍         | 1965/41250 [4:44:40<94:27:50,  8.66s/it][2025-04-25 12:42:24,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:42:24,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5753.93 | bwd_inner_microstep: 5652.78 | bwd_allreduce_microstep: 101.10 | step_microstep: 18.12
[2025-04-25 12:42:24,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5753.94 | bwd_inner: 5652.78 | bwd_allreduce: 101.12 | step: 18.12

  5%|▍         | 1966/41250 [4:44:49<94:29:26,  8.66s/it]
                                                         
{'loss': 0.1906, 'grad_norm': 1.035409688949585, 'learning_rate': 3.9967336423589425e-05, 'epoch': 0.48}

  5%|▍         | 1966/41250 [4:44:49<94:29:26,  8.66s/it][2025-04-25 12:42:32,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:42:32,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.76 | bwd_microstep: 5722.68 | bwd_inner_microstep: 5709.86 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.69
[2025-04-25 12:42:32,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.76 | bwd: 5722.70 | bwd_inner: 5709.86 | bwd_allreduce: 12.79 | step: 18.69

  5%|▍         | 1967/41250 [4:44:58<94:30:02,  8.66s/it]
                                                         
{'loss': 0.1708, 'grad_norm': 1.6583422422409058, 'learning_rate': 3.99672466513757e-05, 'epoch': 0.48}

  5%|▍         | 1967/41250 [4:44:58<94:30:02,  8.66s/it][2025-04-25 12:42:41,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:42:41,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.55 | bwd_microstep: 5737.11 | bwd_inner_microstep: 5707.85 | bwd_allreduce_microstep: 29.21 | step_microstep: 18.76
[2025-04-25 12:42:41,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.55 | bwd: 5737.12 | bwd_inner: 5707.85 | bwd_allreduce: 29.23 | step: 18.76

  5%|▍         | 1968/41250 [4:45:06<94:34:02,  8.67s/it]
                                                         
{'loss': 0.2478, 'grad_norm': 1.7309564352035522, 'learning_rate': 3.996715675606782e-05, 'epoch': 0.48}

  5%|▍         | 1968/41250 [4:45:06<94:34:02,  8.67s/it][2025-04-25 12:42:50,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 12:42:50,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.88 | bwd_microstep: 5698.20 | bwd_inner_microstep: 5649.52 | bwd_allreduce_microstep: 48.63 | step_microstep: 18.78
[2025-04-25 12:42:50,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.88 | bwd: 5698.21 | bwd_inner: 5649.52 | bwd_allreduce: 48.65 | step: 18.79

  5%|▍         | 1969/41250 [4:45:15<94:23:01,  8.65s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 2.218172788619995, 'learning_rate': 3.9967066737666356e-05, 'epoch': 0.48}

  5%|▍         | 1969/41250 [4:45:15<94:23:01,  8.65s/it][2025-04-25 12:42:58,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:42:58,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.34 | bwd_microstep: 5710.75 | bwd_inner_microstep: 5659.12 | bwd_allreduce_microstep: 51.59 | step_microstep: 18.90
[2025-04-25 12:42:58,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.34 | bwd: 5710.77 | bwd_inner: 5659.12 | bwd_allreduce: 51.60 | step: 18.90

  5%|▍         | 1970/41250 [4:45:24<94:18:03,  8.64s/it]
                                                         
{'loss': 0.1987, 'grad_norm': 1.5427768230438232, 'learning_rate': 3.996697659617184e-05, 'epoch': 0.48}

  5%|▍         | 1970/41250 [4:45:24<94:18:03,  8.64s/it][2025-04-25 12:43:07,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:43:07,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.81 | bwd_microstep: 5764.63 | bwd_inner_microstep: 5704.44 | bwd_allreduce_microstep: 60.15 | step_microstep: 18.73
[2025-04-25 12:43:07,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5764.64 | bwd_inner: 5704.44 | bwd_allreduce: 60.17 | step: 18.74

  5%|▍         | 1971/41250 [4:45:32<94:29:16,  8.66s/it]
                                                         
{'loss': 0.1346, 'grad_norm': 1.3801230192184448, 'learning_rate': 3.996688633158485e-05, 'epoch': 0.48}

  5%|▍         | 1971/41250 [4:45:32<94:29:16,  8.66s/it][2025-04-25 12:43:16,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:43:16,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.70 | bwd_microstep: 5795.65 | bwd_inner_microstep: 5672.56 | bwd_allreduce_microstep: 123.04 | step_microstep: 19.10
[2025-04-25 12:43:16,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.70 | bwd: 5795.67 | bwd_inner: 5672.56 | bwd_allreduce: 123.06 | step: 19.10

  5%|▍         | 1972/41250 [4:45:41<94:38:21,  8.67s/it]
                                                         
{'loss': 0.299, 'grad_norm': 1.8062719106674194, 'learning_rate': 3.996679594390593e-05, 'epoch': 0.48}

  5%|▍         | 1972/41250 [4:45:41<94:38:21,  8.67s/it][2025-04-25 12:43:24,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 1.23 | optimizer_step: 0.99
[2025-04-25 12:43:24,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.52 | bwd_microstep: 5758.26 | bwd_inner_microstep: 5724.12 | bwd_allreduce_microstep: 34.10 | step_microstep: 19.25
[2025-04-25 12:43:24,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.52 | bwd: 5758.28 | bwd_inner: 5724.12 | bwd_allreduce: 34.12 | step: 19.25

  5%|▍         | 1973/41250 [4:45:50<94:43:49,  8.68s/it]
                                                         
{'loss': 0.243, 'grad_norm': 1.3278441429138184, 'learning_rate': 3.9966705433135636e-05, 'epoch': 0.48}

  5%|▍         | 1973/41250 [4:45:50<94:43:49,  8.68s/it][2025-04-25 12:43:33,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-25 12:43:33,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5713.29 | bwd_inner_microstep: 5665.76 | bwd_allreduce_microstep: 47.47 | step_microstep: 19.63
[2025-04-25 12:43:33,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5713.30 | bwd_inner: 5665.76 | bwd_allreduce: 47.49 | step: 19.64

  5%|▍         | 1974/41250 [4:45:58<94:32:59,  8.67s/it]
                                                         
{'loss': 0.1312, 'grad_norm': 0.9564940333366394, 'learning_rate': 3.9966614799274536e-05, 'epoch': 0.48}

  5%|▍         | 1974/41250 [4:45:58<94:32:59,  8.67s/it][2025-04-25 12:43:42,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:43:42,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.81 | bwd_microstep: 5717.65 | bwd_inner_microstep: 5704.97 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.74
[2025-04-25 12:43:42,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 5717.67 | bwd_inner: 5704.97 | bwd_allreduce: 12.66 | step: 18.74

  5%|▍         | 1975/41250 [4:46:07<94:30:52,  8.66s/it]
                                                         
{'loss': 0.2742, 'grad_norm': 1.7766650915145874, 'learning_rate': 3.9966524042323184e-05, 'epoch': 0.48}

  5%|▍         | 1975/41250 [4:46:07<94:30:52,  8.66s/it][2025-04-25 12:43:50,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:43:50,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.38 | bwd_microstep: 5765.54 | bwd_inner_microstep: 5694.61 | bwd_allreduce_microstep: 70.88 | step_microstep: 18.76
[2025-04-25 12:43:50,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.38 | bwd: 5765.56 | bwd_inner: 5694.61 | bwd_allreduce: 70.90 | step: 18.77

  5%|▍         | 1976/41250 [4:46:16<94:37:56,  8.67s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 1.538033366203308, 'learning_rate': 3.9966433162282135e-05, 'epoch': 0.48}

  5%|▍         | 1976/41250 [4:46:16<94:37:56,  8.67s/it][2025-04-25 12:43:59,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 12:43:59,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.60 | bwd_microstep: 5910.90 | bwd_inner_microstep: 5664.69 | bwd_allreduce_microstep: 246.16 | step_microstep: 18.97
[2025-04-25 12:43:59,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.60 | bwd: 5910.91 | bwd_inner: 5664.69 | bwd_allreduce: 246.18 | step: 18.98

  5%|▍         | 1977/41250 [4:46:25<95:08:34,  8.72s/it]
                                                         
{'loss': 0.2549, 'grad_norm': 1.1086399555206299, 'learning_rate': 3.996634215915195e-05, 'epoch': 0.48}

  5%|▍         | 1977/41250 [4:46:25<95:08:34,  8.72s/it][2025-04-25 12:44:08,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 12:44:08,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.27 | bwd_microstep: 5777.36 | bwd_inner_microstep: 5710.46 | bwd_allreduce_microstep: 66.85 | step_microstep: 18.92
[2025-04-25 12:44:08,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.27 | bwd: 5777.37 | bwd_inner: 5710.46 | bwd_allreduce: 66.87 | step: 18.92

  5%|▍         | 1978/41250 [4:46:33<95:06:56,  8.72s/it]
                                                         
{'loss': 0.1232, 'grad_norm': 0.9530488848686218, 'learning_rate': 3.99662510329332e-05, 'epoch': 0.48}

  5%|▍         | 1978/41250 [4:46:33<95:06:56,  8.72s/it][2025-04-25 12:44:17,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:44:17,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.48 | bwd_microstep: 5876.64 | bwd_inner_microstep: 5661.65 | bwd_allreduce_microstep: 214.94 | step_microstep: 18.03
[2025-04-25 12:44:17,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.48 | bwd: 5876.65 | bwd_inner: 5661.65 | bwd_allreduce: 214.96 | step: 18.03

  5%|▍         | 1979/41250 [4:46:42<95:23:26,  8.74s/it]
                                                         
{'loss': 0.1896, 'grad_norm': 1.2942005395889282, 'learning_rate': 3.996615978362644e-05, 'epoch': 0.48}

  5%|▍         | 1979/41250 [4:46:42<95:23:26,  8.74s/it][2025-04-25 12:44:25,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 12:44:25,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.33 | bwd_microstep: 5719.45 | bwd_inner_microstep: 5671.91 | bwd_allreduce_microstep: 47.49 | step_microstep: 18.69
[2025-04-25 12:44:25,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.33 | bwd: 5719.46 | bwd_inner: 5671.91 | bwd_allreduce: 47.51 | step: 18.70

  5%|▍         | 1980/41250 [4:46:51<95:03:13,  8.71s/it]
                                                         
{'loss': 0.2774, 'grad_norm': 1.7288461923599243, 'learning_rate': 3.996606841123222e-05, 'epoch': 0.48}

  5%|▍         | 1980/41250 [4:46:51<95:03:13,  8.71s/it][2025-04-25 12:44:34,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:44:34,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.04 | bwd_microstep: 5725.43 | bwd_inner_microstep: 5712.54 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.50
[2025-04-25 12:44:34,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.04 | bwd: 5725.45 | bwd_inner: 5712.54 | bwd_allreduce: 12.86 | step: 18.50

  5%|▍         | 1981/41250 [4:46:59<94:53:53,  8.70s/it]
                                                         
{'loss': 0.1758, 'grad_norm': 1.617620587348938, 'learning_rate': 3.996597691575113e-05, 'epoch': 0.48}

  5%|▍         | 1981/41250 [4:46:59<94:53:53,  8.70s/it][2025-04-25 12:44:43,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:44:43,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.17 | bwd_microstep: 5741.19 | bwd_inner_microstep: 5706.51 | bwd_allreduce_microstep: 34.64 | step_microstep: 18.59
[2025-04-25 12:44:43,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.17 | bwd: 5741.21 | bwd_inner: 5706.51 | bwd_allreduce: 34.65 | step: 18.59

  5%|▍         | 1982/41250 [4:47:08<94:50:46,  8.70s/it]
                                                         
{'loss': 0.2359, 'grad_norm': 1.9556840658187866, 'learning_rate': 3.996588529718371e-05, 'epoch': 0.48}

  5%|▍         | 1982/41250 [4:47:08<94:50:46,  8.70s/it][2025-04-25 12:44:51,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 12:44:51,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.55 | bwd_microstep: 5795.19 | bwd_inner_microstep: 5782.10 | bwd_allreduce_microstep: 13.03 | step_microstep: 19.17
[2025-04-25 12:44:51,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.55 | bwd: 5795.20 | bwd_inner: 5782.10 | bwd_allreduce: 13.06 | step: 19.18

  5%|▍         | 1983/41250 [4:47:17<95:05:25,  8.72s/it]
                                                         
{'loss': 0.1122, 'grad_norm': 0.8445799350738525, 'learning_rate': 3.996579355553054e-05, 'epoch': 0.48}

  5%|▍         | 1983/41250 [4:47:17<95:05:25,  8.72s/it][2025-04-25 12:45:00,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 12:45:00,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.18 | bwd_microstep: 5694.00 | bwd_inner_microstep: 5669.82 | bwd_allreduce_microstep: 24.13 | step_microstep: 19.10
[2025-04-25 12:45:00,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.18 | bwd: 5694.01 | bwd_inner: 5669.82 | bwd_allreduce: 24.15 | step: 19.10

  5%|▍         | 1984/41250 [4:47:25<94:45:58,  8.69s/it]
                                                         
{'loss': 0.359, 'grad_norm': 1.469940185546875, 'learning_rate': 3.996570169079217e-05, 'epoch': 0.48}

  5%|▍         | 1984/41250 [4:47:25<94:45:58,  8.69s/it][2025-04-25 12:45:09,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:45:09,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.35 | bwd_microstep: 5786.23 | bwd_inner_microstep: 5666.90 | bwd_allreduce_microstep: 119.29 | step_microstep: 18.81
[2025-04-25 12:45:09,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.35 | bwd: 5786.24 | bwd_inner: 5666.89 | bwd_allreduce: 119.31 | step: 18.82

  5%|▍         | 1985/41250 [4:47:34<94:46:53,  8.69s/it]
                                                         
{'loss': 0.1342, 'grad_norm': 1.1554274559020996, 'learning_rate': 3.9965609702969184e-05, 'epoch': 0.48}

  5%|▍         | 1985/41250 [4:47:34<94:46:53,  8.69s/it][2025-04-25 12:45:17,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:45:17,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.44 | bwd_microstep: 5772.91 | bwd_inner_microstep: 5655.66 | bwd_allreduce_microstep: 117.20 | step_microstep: 18.54
[2025-04-25 12:45:17,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.44 | bwd: 5772.92 | bwd_inner: 5655.66 | bwd_allreduce: 117.22 | step: 18.54

  5%|▍         | 1986/41250 [4:47:43<94:46:27,  8.69s/it]
                                                         
{'loss': 0.1277, 'grad_norm': 1.1149094104766846, 'learning_rate': 3.996551759206214e-05, 'epoch': 0.48}

  5%|▍         | 1986/41250 [4:47:43<94:46:27,  8.69s/it][2025-04-25 12:45:26,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:45:26,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5717.80 | bwd_inner_microstep: 5704.95 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.36
[2025-04-25 12:45:26,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5717.81 | bwd_inner: 5704.95 | bwd_allreduce: 12.82 | step: 18.37

  5%|▍         | 1987/41250 [4:47:51<94:38:54,  8.68s/it]
                                                         
{'loss': 0.0749, 'grad_norm': 1.2717597484588623, 'learning_rate': 3.996542535807161e-05, 'epoch': 0.48}

  5%|▍         | 1987/41250 [4:47:51<94:38:54,  8.68s/it][2025-04-25 12:45:35,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-25 12:45:35,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.79 | bwd_microstep: 5685.26 | bwd_inner_microstep: 5650.03 | bwd_allreduce_microstep: 35.18 | step_microstep: 18.63
[2025-04-25 12:45:35,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5685.27 | bwd_inner: 5650.03 | bwd_allreduce: 35.20 | step: 18.63

  5%|▍         | 1988/41250 [4:48:00<94:23:11,  8.65s/it]
                                                         
{'loss': 0.1444, 'grad_norm': 1.1764289140701294, 'learning_rate': 3.9965333000998156e-05, 'epoch': 0.48}

  5%|▍         | 1988/41250 [4:48:00<94:23:11,  8.65s/it][2025-04-25 12:45:43,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 12:45:43,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.48 | bwd_microstep: 5779.79 | bwd_inner_microstep: 5690.76 | bwd_allreduce_microstep: 88.98 | step_microstep: 19.23
[2025-04-25 12:45:43,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.48 | bwd: 5779.81 | bwd_inner: 5690.76 | bwd_allreduce: 89.01 | step: 19.23

  5%|▍         | 1989/41250 [4:48:09<94:33:09,  8.67s/it]
                                                         
{'loss': 0.1782, 'grad_norm': 0.9273252487182617, 'learning_rate': 3.996524052084235e-05, 'epoch': 0.48}

  5%|▍         | 1989/41250 [4:48:09<94:33:09,  8.67s/it][2025-04-25 12:45:52,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:45:52,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.56 | bwd_microstep: 5712.16 | bwd_inner_microstep: 5699.52 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.79
[2025-04-25 12:45:52,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.56 | bwd: 5712.18 | bwd_inner: 5699.52 | bwd_allreduce: 12.61 | step: 18.79

  5%|▍         | 1990/41250 [4:48:17<94:29:50,  8.67s/it]
                                                         
{'loss': 0.1495, 'grad_norm': 1.5362534523010254, 'learning_rate': 3.996514791760477e-05, 'epoch': 0.48}

  5%|▍         | 1990/41250 [4:48:17<94:29:50,  8.67s/it][2025-04-25 12:46:01,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:46:01,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.23 | bwd_microstep: 5761.23 | bwd_inner_microstep: 5663.27 | bwd_allreduce_microstep: 97.92 | step_microstep: 18.80
[2025-04-25 12:46:01,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.23 | bwd: 5761.24 | bwd_inner: 5663.27 | bwd_allreduce: 97.94 | step: 18.80

  5%|▍         | 1991/41250 [4:48:26<94:31:59,  8.67s/it]
                                                         
{'loss': 0.3609, 'grad_norm': 1.8061532974243164, 'learning_rate': 3.996505519128597e-05, 'epoch': 0.48}

  5%|▍         | 1991/41250 [4:48:26<94:31:59,  8.67s/it][2025-04-25 12:46:10,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:46:10,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.77 | bwd_microstep: 5787.75 | bwd_inner_microstep: 5775.05 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.44
[2025-04-25 12:46:10,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.77 | bwd: 5787.76 | bwd_inner: 5775.05 | bwd_allreduce: 12.67 | step: 18.45

  5%|▍         | 1992/41250 [4:48:35<94:51:12,  8.70s/it]
                                                         
{'loss': 0.4097, 'grad_norm': 2.1812660694122314, 'learning_rate': 3.996496234188654e-05, 'epoch': 0.48}

  5%|▍         | 1992/41250 [4:48:35<94:51:12,  8.70s/it][2025-04-25 12:46:18,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:46:18,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5747.11 | bwd_inner_microstep: 5647.94 | bwd_allreduce_microstep: 99.13 | step_microstep: 18.70
[2025-04-25 12:46:18,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5747.13 | bwd_inner: 5647.94 | bwd_allreduce: 99.14 | step: 18.70

  5%|▍         | 1993/41250 [4:48:44<94:43:24,  8.69s/it]
                                                         
{'loss': 0.1335, 'grad_norm': 1.3125442266464233, 'learning_rate': 3.996486936940704e-05, 'epoch': 0.48}

  5%|▍         | 1993/41250 [4:48:44<94:43:24,  8.69s/it][2025-04-25 12:46:27,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.86 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 12:46:27,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.49 | bwd_microstep: 5877.81 | bwd_inner_microstep: 5651.46 | bwd_allreduce_microstep: 226.31 | step_microstep: 19.65
[2025-04-25 12:46:27,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.49 | bwd: 5877.82 | bwd_inner: 5651.46 | bwd_allreduce: 226.32 | step: 19.66

  5%|▍         | 1994/41250 [4:48:52<95:02:48,  8.72s/it]
                                                         
{'loss': 0.1928, 'grad_norm': 1.2851825952529907, 'learning_rate': 3.9964776273848044e-05, 'epoch': 0.48}

  5%|▍         | 1994/41250 [4:48:52<95:02:48,  8.72s/it][2025-04-25 12:46:36,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:46:36,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.57 | bwd_microstep: 5699.24 | bwd_inner_microstep: 5662.24 | bwd_allreduce_microstep: 36.96 | step_microstep: 18.34
[2025-04-25 12:46:36,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.57 | bwd: 5699.25 | bwd_inner: 5662.24 | bwd_allreduce: 36.98 | step: 18.34

  5%|▍         | 1995/41250 [4:49:01<94:42:21,  8.69s/it]
                                                         
{'loss': 0.2152, 'grad_norm': 1.3836605548858643, 'learning_rate': 3.996468305521014e-05, 'epoch': 0.48}

  5%|▍         | 1995/41250 [4:49:01<94:42:21,  8.69s/it][2025-04-25 12:46:44,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.22 | optimizer_step: 0.91
[2025-04-25 12:46:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.69 | bwd_microstep: 5720.76 | bwd_inner_microstep: 5686.69 | bwd_allreduce_microstep: 34.02 | step_microstep: 19.01
[2025-04-25 12:46:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.69 | bwd: 5720.77 | bwd_inner: 5686.69 | bwd_allreduce: 34.04 | step: 19.02

  5%|▍         | 1996/41250 [4:49:10<94:36:19,  8.68s/it]
                                                         
{'loss': 0.1083, 'grad_norm': 1.2612465620040894, 'learning_rate': 3.9964589713493877e-05, 'epoch': 0.48}

  5%|▍         | 1996/41250 [4:49:10<94:36:19,  8.68s/it][2025-04-25 12:46:53,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:46:53,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5679.89 | bwd_inner_microstep: 5640.45 | bwd_allreduce_microstep: 39.39 | step_microstep: 18.70
[2025-04-25 12:46:53,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5679.90 | bwd_inner: 5640.45 | bwd_allreduce: 39.41 | step: 18.70

  5%|▍         | 1997/41250 [4:49:18<94:20:30,  8.65s/it]
                                                         
{'loss': 0.2968, 'grad_norm': 1.6390832662582397, 'learning_rate': 3.996449624869985e-05, 'epoch': 0.48}

  5%|▍         | 1997/41250 [4:49:18<94:20:30,  8.65s/it][2025-04-25 12:47:02,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:47:02,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.73 | bwd_microstep: 5729.23 | bwd_inner_microstep: 5695.26 | bwd_allreduce_microstep: 33.92 | step_microstep: 18.79
[2025-04-25 12:47:02,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.73 | bwd: 5729.24 | bwd_inner: 5695.26 | bwd_allreduce: 33.94 | step: 18.80

  5%|▍         | 1998/41250 [4:49:27<94:22:28,  8.66s/it]
                                                         
{'loss': 0.2361, 'grad_norm': 1.9209423065185547, 'learning_rate': 3.9964402660828636e-05, 'epoch': 0.48}

  5%|▍         | 1998/41250 [4:49:27<94:22:28,  8.66s/it][2025-04-25 12:47:10,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-25 12:47:10,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.85 | bwd_microstep: 5730.70 | bwd_inner_microstep: 5687.78 | bwd_allreduce_microstep: 42.85 | step_microstep: 19.08
[2025-04-25 12:47:10,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.85 | bwd: 5730.72 | bwd_inner: 5687.78 | bwd_allreduce: 42.88 | step: 19.08

  5%|▍         | 1999/41250 [4:49:35<94:23:33,  8.66s/it]
                                                         
{'loss': 0.1095, 'grad_norm': 1.08293879032135, 'learning_rate': 3.996430894988081e-05, 'epoch': 0.48}

  5%|▍         | 1999/41250 [4:49:36<94:23:33,  8.66s/it][2025-04-25 12:47:19,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:47:19,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.66 | bwd_microstep: 5749.95 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 97.08 | step_microstep: 19.10
[2025-04-25 12:47:19,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.66 | bwd: 5749.97 | bwd_inner: 5652.83 | bwd_allreduce: 97.10 | step: 19.11

  5%|▍         | 2000/41250 [4:49:44<94:24:12,  8.66s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 3.2898075580596924, 'learning_rate': 3.9964215115856936e-05, 'epoch': 0.48}

  5%|▍         | 2000/41250 [4:49:44<94:24:12,  8.66s/it][2025-04-25 12:47:28,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 12:47:28,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.86 | bwd_microstep: 5753.25 | bwd_inner_microstep: 5691.30 | bwd_allreduce_microstep: 61.90 | step_microstep: 18.89
[2025-04-25 12:47:28,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.86 | bwd: 5753.26 | bwd_inner: 5691.30 | bwd_allreduce: 61.92 | step: 18.89

  5%|▍         | 2001/41250 [4:49:53<94:28:39,  8.67s/it]
                                                         
{'loss': 0.2006, 'grad_norm': 1.9291878938674927, 'learning_rate': 3.9964121158757606e-05, 'epoch': 0.49}

  5%|▍         | 2001/41250 [4:49:53<94:28:39,  8.67s/it][2025-04-25 12:47:36,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-25 12:47:36,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.75 | bwd_microstep: 5691.16 | bwd_inner_microstep: 5652.96 | bwd_allreduce_microstep: 38.16 | step_microstep: 18.96
[2025-04-25 12:47:36,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.75 | bwd: 5691.17 | bwd_inner: 5652.96 | bwd_allreduce: 38.17 | step: 18.97

  5%|▍         | 2002/41250 [4:50:01<94:16:23,  8.65s/it]
                                                         
{'loss': 0.2114, 'grad_norm': 1.971951961517334, 'learning_rate': 3.996402707858339e-05, 'epoch': 0.49}

  5%|▍         | 2002/41250 [4:50:01<94:16:23,  8.65s/it][2025-04-25 12:47:45,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:47:45,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5750.92 | bwd_inner_microstep: 5657.77 | bwd_allreduce_microstep: 93.11 | step_microstep: 18.66
[2025-04-25 12:47:45,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5750.93 | bwd_inner: 5657.77 | bwd_allreduce: 93.12 | step: 18.67

  5%|▍         | 2003/41250 [4:50:10<94:19:46,  8.65s/it]
                                                         
{'loss': 0.1169, 'grad_norm': 0.713338315486908, 'learning_rate': 3.9963932875334886e-05, 'epoch': 0.49}

  5%|▍         | 2003/41250 [4:50:10<94:19:46,  8.65s/it][2025-04-25 12:47:53,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:47:53,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5736.66 | bwd_inner_microstep: 5636.65 | bwd_allreduce_microstep: 99.95 | step_microstep: 19.27
[2025-04-25 12:47:53,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5736.68 | bwd_inner: 5636.65 | bwd_allreduce: 99.98 | step: 19.27

  5%|▍         | 2004/41250 [4:50:19<94:18:01,  8.65s/it]
                                                         
{'loss': 0.4056, 'grad_norm': 3.116132974624634, 'learning_rate': 3.9963838549012654e-05, 'epoch': 0.49}

  5%|▍         | 2004/41250 [4:50:19<94:18:01,  8.65s/it][2025-04-25 12:48:02,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.04 | optimizer_step: 1.19
[2025-04-25 12:48:02,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.44 | bwd_microstep: 5702.13 | bwd_inner_microstep: 5689.17 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.64
[2025-04-25 12:48:02,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.44 | bwd: 5702.14 | bwd_inner: 5689.17 | bwd_allreduce: 12.93 | step: 19.65

  5%|▍         | 2005/41250 [4:50:27<94:13:55,  8.64s/it]
                                                         
{'loss': 0.2297, 'grad_norm': 1.9624279737472534, 'learning_rate': 3.9963744099617294e-05, 'epoch': 0.49}

  5%|▍         | 2005/41250 [4:50:27<94:13:55,  8.64s/it][2025-04-25 12:48:11,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 12:48:11,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.58 | bwd_microstep: 5759.40 | bwd_inner_microstep: 5642.60 | bwd_allreduce_microstep: 116.76 | step_microstep: 19.18
[2025-04-25 12:48:11,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.58 | bwd: 5759.42 | bwd_inner: 5642.60 | bwd_allreduce: 116.78 | step: 19.18

  5%|▍         | 2006/41250 [4:50:36<94:17:15,  8.65s/it]
                                                         
{'loss': 0.3028, 'grad_norm': 1.3398910760879517, 'learning_rate': 3.996364952714936e-05, 'epoch': 0.49}

  5%|▍         | 2006/41250 [4:50:36<94:17:15,  8.65s/it][2025-04-25 12:48:20,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:48:20,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.91 | bwd_microstep: 5861.32 | bwd_inner_microstep: 5683.62 | bwd_allreduce_microstep: 177.66 | step_microstep: 18.71
[2025-04-25 12:48:20,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.91 | bwd: 5861.33 | bwd_inner: 5683.62 | bwd_allreduce: 177.67 | step: 18.72

  5%|▍         | 2007/41250 [4:50:45<94:43:55,  8.69s/it]
                                                         
{'loss': 0.1261, 'grad_norm': 1.2939832210540771, 'learning_rate': 3.996355483160947e-05, 'epoch': 0.49}

  5%|▍         | 2007/41250 [4:50:45<94:43:55,  8.69s/it][2025-04-25 12:48:28,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:48:28,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.88 | bwd_microstep: 5710.25 | bwd_inner_microstep: 5639.29 | bwd_allreduce_microstep: 70.91 | step_microstep: 18.61
[2025-04-25 12:48:28,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.88 | bwd: 5710.26 | bwd_inner: 5639.29 | bwd_allreduce: 70.93 | step: 18.62

  5%|▍         | 2008/41250 [4:50:53<94:28:46,  8.67s/it]
                                                         
{'loss': 0.1594, 'grad_norm': 1.9769943952560425, 'learning_rate': 3.996346001299818e-05, 'epoch': 0.49}

  5%|▍         | 2008/41250 [4:50:53<94:28:46,  8.67s/it][2025-04-25 12:48:37,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:48:37,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 5705.09 | bwd_inner_microstep: 5692.40 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.71
[2025-04-25 12:48:37,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 5705.10 | bwd_inner: 5692.40 | bwd_allreduce: 12.66 | step: 18.71

  5%|▍         | 2009/41250 [4:51:02<94:22:02,  8.66s/it]
                                                         
{'loss': 0.1989, 'grad_norm': 1.2354986667633057, 'learning_rate': 3.99633650713161e-05, 'epoch': 0.49}

  5%|▍         | 2009/41250 [4:51:02<94:22:02,  8.66s/it][2025-04-25 12:48:45,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 12:48:45,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.63 | bwd_microstep: 5692.25 | bwd_inner_microstep: 5642.84 | bwd_allreduce_microstep: 49.35 | step_microstep: 18.84
[2025-04-25 12:48:45,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.63 | bwd: 5692.26 | bwd_inner: 5642.84 | bwd_allreduce: 49.37 | step: 18.84

  5%|▍         | 2010/41250 [4:51:11<94:09:05,  8.64s/it]
                                                         
{'loss': 0.1171, 'grad_norm': 1.19873046875, 'learning_rate': 3.996327000656379e-05, 'epoch': 0.49}

  5%|▍         | 2010/41250 [4:51:11<94:09:05,  8.64s/it][2025-04-25 12:48:54,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 12:48:54,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.16 | bwd_microstep: 5709.12 | bwd_inner_microstep: 5696.21 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.89
[2025-04-25 12:48:54,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.16 | bwd: 5709.14 | bwd_inner: 5696.21 | bwd_allreduce: 12.88 | step: 18.89

  5%|▍         | 2011/41250 [4:51:19<94:08:48,  8.64s/it]
                                                         
{'loss': 0.1386, 'grad_norm': 1.1396777629852295, 'learning_rate': 3.996317481874185e-05, 'epoch': 0.49}

  5%|▍         | 2011/41250 [4:51:19<94:08:48,  8.64s/it][2025-04-25 12:49:03,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.09 | optimizer_step: 1.27
[2025-04-25 12:49:03,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.63 | bwd_microstep: 5716.45 | bwd_inner_microstep: 5677.07 | bwd_allreduce_microstep: 39.33 | step_microstep: 19.99
[2025-04-25 12:49:03,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.63 | bwd: 5716.47 | bwd_inner: 5677.07 | bwd_allreduce: 39.36 | step: 19.99

  5%|▍         | 2012/41250 [4:51:28<94:09:05,  8.64s/it]
                                                         
{'loss': 0.2321, 'grad_norm': 1.4383187294006348, 'learning_rate': 3.996307950785086e-05, 'epoch': 0.49}

  5%|▍         | 2012/41250 [4:51:28<94:09:05,  8.64s/it][2025-04-25 12:49:11,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 12:49:11,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.60 | bwd_microstep: 5738.56 | bwd_inner_microstep: 5677.41 | bwd_allreduce_microstep: 61.11 | step_microstep: 18.99
[2025-04-25 12:49:11,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.60 | bwd: 5738.58 | bwd_inner: 5677.41 | bwd_allreduce: 61.13 | step: 19.00

  5%|▍         | 2013/41250 [4:51:37<94:13:16,  8.64s/it]
                                                         
{'loss': 0.2412, 'grad_norm': 1.927188515663147, 'learning_rate': 3.9962984073891413e-05, 'epoch': 0.49}

  5%|▍         | 2013/41250 [4:51:37<94:13:16,  8.64s/it][2025-04-25 12:49:20,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 12:49:20,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.71 | bwd_microstep: 5715.77 | bwd_inner_microstep: 5694.05 | bwd_allreduce_microstep: 21.67 | step_microstep: 19.01
[2025-04-25 12:49:20,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.71 | bwd: 5715.78 | bwd_inner: 5694.05 | bwd_allreduce: 21.69 | step: 19.01

  5%|▍         | 2014/41250 [4:51:45<94:12:42,  8.64s/it]
                                                         
{'loss': 0.3657, 'grad_norm': 2.1911795139312744, 'learning_rate': 3.9962888516864096e-05, 'epoch': 0.49}

  5%|▍         | 2014/41250 [4:51:45<94:12:42,  8.64s/it][2025-04-25 12:49:29,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:49:29,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.62 | bwd_microstep: 5667.51 | bwd_inner_microstep: 5645.83 | bwd_allreduce_microstep: 21.63 | step_microstep: 18.89
[2025-04-25 12:49:29,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.62 | bwd: 5667.52 | bwd_inner: 5645.83 | bwd_allreduce: 21.65 | step: 18.89

  5%|▍         | 2015/41250 [4:51:54<93:58:46,  8.62s/it]
                                                         
{'loss': 0.1583, 'grad_norm': 1.3240641355514526, 'learning_rate': 3.9962792836769504e-05, 'epoch': 0.49}

  5%|▍         | 2015/41250 [4:51:54<93:58:46,  8.62s/it][2025-04-25 12:49:37,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:49:37,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.06 | bwd_microstep: 5687.54 | bwd_inner_microstep: 5631.00 | bwd_allreduce_microstep: 56.49 | step_microstep: 18.55
[2025-04-25 12:49:37,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.06 | bwd: 5687.55 | bwd_inner: 5631.00 | bwd_allreduce: 56.51 | step: 18.55

  5%|▍         | 2016/41250 [4:52:02<93:51:07,  8.61s/it]
                                                         
{'loss': 0.3421, 'grad_norm': 2.563917636871338, 'learning_rate': 3.9962697033608215e-05, 'epoch': 0.49}

  5%|▍         | 2016/41250 [4:52:02<93:51:07,  8.61s/it][2025-04-25 12:49:46,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:49:46,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5719.62 | bwd_inner_microstep: 5675.14 | bwd_allreduce_microstep: 44.43 | step_microstep: 18.34
[2025-04-25 12:49:46,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5719.63 | bwd_inner: 5675.14 | bwd_allreduce: 44.45 | step: 18.34

  5%|▍         | 2017/41250 [4:52:11<93:58:51,  8.62s/it]
                                                         
{'loss': 0.2025, 'grad_norm': 1.0165852308273315, 'learning_rate': 3.996260110738082e-05, 'epoch': 0.49}

  5%|▍         | 2017/41250 [4:52:11<93:58:51,  8.62s/it][2025-04-25 12:49:54,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:49:54,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.39 | bwd_microstep: 5743.08 | bwd_inner_microstep: 5635.38 | bwd_allreduce_microstep: 107.66 | step_microstep: 18.80
[2025-04-25 12:49:54,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.39 | bwd: 5743.10 | bwd_inner: 5635.38 | bwd_allreduce: 107.68 | step: 18.80

  5%|▍         | 2018/41250 [4:52:20<94:04:55,  8.63s/it]
                                                         
{'loss': 0.1722, 'grad_norm': 2.1955759525299072, 'learning_rate': 3.996250505808792e-05, 'epoch': 0.49}

  5%|▍         | 2018/41250 [4:52:20<94:04:55,  8.63s/it][2025-04-25 12:50:03,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 12:50:03,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.15 | bwd_microstep: 5777.27 | bwd_inner_microstep: 5646.85 | bwd_allreduce_microstep: 130.37 | step_microstep: 18.46
[2025-04-25 12:50:03,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.15 | bwd: 5777.28 | bwd_inner: 5646.85 | bwd_allreduce: 130.39 | step: 18.46

  5%|▍         | 2019/41250 [4:52:28<94:15:06,  8.65s/it]
                                                         
{'loss': 0.218, 'grad_norm': 1.420371651649475, 'learning_rate': 3.9962408885730106e-05, 'epoch': 0.49}

  5%|▍         | 2019/41250 [4:52:28<94:15:06,  8.65s/it][2025-04-25 12:50:12,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:50:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.09 | bwd_microstep: 5778.24 | bwd_inner_microstep: 5644.13 | bwd_allreduce_microstep: 134.06 | step_microstep: 18.63
[2025-04-25 12:50:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.09 | bwd: 5778.26 | bwd_inner: 5644.13 | bwd_allreduce: 134.09 | step: 18.64

  5%|▍         | 2020/41250 [4:52:37<94:22:28,  8.66s/it]
                                                         
{'loss': 0.1073, 'grad_norm': 1.2703251838684082, 'learning_rate': 3.996231259030797e-05, 'epoch': 0.49}

  5%|▍         | 2020/41250 [4:52:37<94:22:28,  8.66s/it][2025-04-25 12:50:20,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 12:50:20,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.18 | bwd_microstep: 5696.49 | bwd_inner_microstep: 5683.52 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.02
[2025-04-25 12:50:20,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.18 | bwd: 5696.50 | bwd_inner: 5683.52 | bwd_allreduce: 12.93 | step: 19.02

  5%|▍         | 2021/41250 [4:52:46<94:13:38,  8.65s/it]
                                                         
{'loss': 0.0633, 'grad_norm': 0.5158149003982544, 'learning_rate': 3.996221617182209e-05, 'epoch': 0.49}

  5%|▍         | 2021/41250 [4:52:46<94:13:38,  8.65s/it][2025-04-25 12:50:29,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:50:29,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.96 | bwd_microstep: 5698.16 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 40.67 | step_microstep: 18.80
[2025-04-25 12:50:29,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5698.17 | bwd_inner: 5657.45 | bwd_allreduce: 40.69 | step: 18.80

  5%|▍         | 2022/41250 [4:52:54<94:06:22,  8.64s/it]
                                                         
{'loss': 0.2271, 'grad_norm': 2.3434760570526123, 'learning_rate': 3.996211963027309e-05, 'epoch': 0.49}

  5%|▍         | 2022/41250 [4:52:54<94:06:22,  8.64s/it][2025-04-25 12:50:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 12:50:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.63 | bwd_microstep: 5692.01 | bwd_inner_microstep: 5679.42 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.56
[2025-04-25 12:50:38,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.63 | bwd: 5692.02 | bwd_inner: 5679.42 | bwd_allreduce: 12.56 | step: 18.57

  5%|▍         | 2023/41250 [4:53:03<94:03:02,  8.63s/it]
                                                         
{'loss': 0.2095, 'grad_norm': 1.3611773252487183, 'learning_rate': 3.996202296566153e-05, 'epoch': 0.49}

  5%|▍         | 2023/41250 [4:53:03<94:03:02,  8.63s/it][2025-04-25 12:50:46,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:50:46,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.85 | bwd_microstep: 5700.67 | bwd_inner_microstep: 5645.62 | bwd_allreduce_microstep: 55.00 | step_microstep: 18.36
[2025-04-25 12:50:46,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.85 | bwd: 5700.68 | bwd_inner: 5645.62 | bwd_allreduce: 55.02 | step: 18.37

  5%|▍         | 2024/41250 [4:53:12<94:00:22,  8.63s/it]
                                                         
{'loss': 0.1692, 'grad_norm': 1.8558027744293213, 'learning_rate': 3.996192617798804e-05, 'epoch': 0.49}

  5%|▍         | 2024/41250 [4:53:12<94:00:22,  8.63s/it][2025-04-25 12:50:55,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-25 12:50:55,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5780.23 | bwd_inner_microstep: 5654.47 | bwd_allreduce_microstep: 125.72 | step_microstep: 18.29
[2025-04-25 12:50:55,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5780.25 | bwd_inner: 5654.47 | bwd_allreduce: 125.74 | step: 18.29

  5%|▍         | 2025/41250 [4:53:20<94:13:16,  8.65s/it]
                                                         
{'loss': 0.095, 'grad_norm': 1.050134301185608, 'learning_rate': 3.99618292672532e-05, 'epoch': 0.49}

  5%|▍         | 2025/41250 [4:53:20<94:13:16,  8.65s/it][2025-04-25 12:51:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:51:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.92 | bwd_microstep: 5688.48 | bwd_inner_microstep: 5666.98 | bwd_allreduce_microstep: 21.45 | step_microstep: 18.47
[2025-04-25 12:51:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.92 | bwd: 5688.50 | bwd_inner: 5666.98 | bwd_allreduce: 21.48 | step: 18.48

  5%|▍         | 2026/41250 [4:53:29<94:05:22,  8.64s/it]
                                                         
{'loss': 0.0443, 'grad_norm': 0.5943440794944763, 'learning_rate': 3.996173223345761e-05, 'epoch': 0.49}

  5%|▍         | 2026/41250 [4:53:29<94:05:22,  8.64s/it][2025-04-25 12:51:12,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:51:12,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.25 | bwd_microstep: 5728.97 | bwd_inner_microstep: 5716.22 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.68
[2025-04-25 12:51:12,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.25 | bwd: 5728.98 | bwd_inner: 5716.22 | bwd_allreduce: 12.73 | step: 18.68

  5%|▍         | 2027/41250 [4:53:38<94:13:47,  8.65s/it]
                                                         
{'loss': 0.07, 'grad_norm': 0.5773112773895264, 'learning_rate': 3.9961635076601854e-05, 'epoch': 0.49}

  5%|▍         | 2027/41250 [4:53:38<94:13:47,  8.65s/it][2025-04-25 12:51:21,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:51:21,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.04 | bwd_microstep: 5776.05 | bwd_inner_microstep: 5763.30 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.35
[2025-04-25 12:51:21,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.04 | bwd: 5776.06 | bwd_inner: 5763.30 | bwd_allreduce: 12.72 | step: 18.35

  5%|▍         | 2028/41250 [4:53:46<94:33:23,  8.68s/it]
                                                         
{'loss': 0.1202, 'grad_norm': 0.7065326571464539, 'learning_rate': 3.996153779668656e-05, 'epoch': 0.49}

  5%|▍         | 2028/41250 [4:53:46<94:33:23,  8.68s/it][2025-04-25 12:51:30,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:51:30,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.46 | bwd_microstep: 5751.78 | bwd_inner_microstep: 5730.28 | bwd_allreduce_microstep: 21.46 | step_microstep: 18.12
[2025-04-25 12:51:30,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.46 | bwd: 5751.79 | bwd_inner: 5730.28 | bwd_allreduce: 21.47 | step: 18.12

  5%|▍         | 2029/41250 [4:53:55<94:38:05,  8.69s/it]
                                                         
{'loss': 0.1358, 'grad_norm': 1.5586822032928467, 'learning_rate': 3.99614403937123e-05, 'epoch': 0.49}

  5%|▍         | 2029/41250 [4:53:55<94:38:05,  8.69s/it][2025-04-25 12:51:38,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:51:38,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.44 | bwd_microstep: 5723.13 | bwd_inner_microstep: 5710.45 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.91
[2025-04-25 12:51:38,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.44 | bwd: 5723.15 | bwd_inner: 5710.45 | bwd_allreduce: 12.66 | step: 17.91

  5%|▍         | 2030/41250 [4:54:04<94:33:55,  8.68s/it]
                                                         
{'loss': 0.1216, 'grad_norm': 0.9930499792098999, 'learning_rate': 3.9961342867679695e-05, 'epoch': 0.49}

  5%|▍         | 2030/41250 [4:54:04<94:33:55,  8.68s/it][2025-04-25 12:51:47,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 12:51:47,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.94 | bwd_microstep: 5713.73 | bwd_inner_microstep: 5663.82 | bwd_allreduce_microstep: 49.86 | step_microstep: 18.82
[2025-04-25 12:51:47,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.94 | bwd: 5713.75 | bwd_inner: 5663.82 | bwd_allreduce: 49.88 | step: 18.82

  5%|▍         | 2031/41250 [4:54:12<94:23:51,  8.66s/it]
                                                         
{'loss': 0.0914, 'grad_norm': 1.248939871788025, 'learning_rate': 3.996124521858933e-05, 'epoch': 0.49}

  5%|▍         | 2031/41250 [4:54:12<94:23:51,  8.66s/it][2025-04-25 12:51:56,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 12:51:56,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.50 | bwd_microstep: 5694.38 | bwd_inner_microstep: 5648.09 | bwd_allreduce_microstep: 46.24 | step_microstep: 18.96
[2025-04-25 12:51:56,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.51 | bwd: 5694.39 | bwd_inner: 5648.09 | bwd_allreduce: 46.26 | step: 18.97

  5%|▍         | 2032/41250 [4:54:21<94:15:07,  8.65s/it]
                                                         
{'loss': 0.2033, 'grad_norm': 0.9481157660484314, 'learning_rate': 3.9961147446441815e-05, 'epoch': 0.49}

  5%|▍         | 2032/41250 [4:54:21<94:15:07,  8.65s/it][2025-04-25 12:52:04,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:52:04,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.12 | bwd_microstep: 5750.38 | bwd_inner_microstep: 5715.98 | bwd_allreduce_microstep: 34.36 | step_microstep: 18.61
[2025-04-25 12:52:04,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.12 | bwd: 5750.40 | bwd_inner: 5715.98 | bwd_allreduce: 34.38 | step: 18.62

  5%|▍         | 2033/41250 [4:54:30<94:25:24,  8.67s/it]
                                                         
{'loss': 0.1296, 'grad_norm': 1.3895175457000732, 'learning_rate': 3.996104955123776e-05, 'epoch': 0.49}

  5%|▍         | 2033/41250 [4:54:30<94:25:24,  8.67s/it][2025-04-25 12:52:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:52:13,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.83 | bwd_microstep: 5771.18 | bwd_inner_microstep: 5722.22 | bwd_allreduce_microstep: 48.91 | step_microstep: 18.30
[2025-04-25 12:52:13,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 5771.19 | bwd_inner: 5722.22 | bwd_allreduce: 48.93 | step: 18.31

  5%|▍         | 2034/41250 [4:54:38<94:33:10,  8.68s/it]
                                                         
{'loss': 0.2363, 'grad_norm': 1.9560754299163818, 'learning_rate': 3.9960951532977754e-05, 'epoch': 0.49}

  5%|▍         | 2034/41250 [4:54:38<94:33:10,  8.68s/it][2025-04-25 12:52:22,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:52:22,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.67 | bwd_microstep: 5731.70 | bwd_inner_microstep: 5719.01 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.21
[2025-04-25 12:52:22,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.67 | bwd: 5731.71 | bwd_inner: 5719.01 | bwd_allreduce: 12.66 | step: 18.22

  5%|▍         | 2035/41250 [4:54:47<94:34:56,  8.68s/it]
                                                         
{'loss': 0.1567, 'grad_norm': 1.8866674900054932, 'learning_rate': 3.9960853391662414e-05, 'epoch': 0.49}

  5%|▍         | 2035/41250 [4:54:47<94:34:56,  8.68s/it][2025-04-25 12:52:30,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:52:30,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5719.00 | bwd_inner_microstep: 5674.41 | bwd_allreduce_microstep: 44.54 | step_microstep: 18.46
[2025-04-25 12:52:30,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5719.01 | bwd_inner: 5674.41 | bwd_allreduce: 44.56 | step: 18.47

  5%|▍         | 2036/41250 [4:54:56<94:28:17,  8.67s/it]
                                                         
{'loss': 0.2662, 'grad_norm': 4.010341644287109, 'learning_rate': 3.996075512729233e-05, 'epoch': 0.49}

  5%|▍         | 2036/41250 [4:54:56<94:28:17,  8.67s/it][2025-04-25 12:52:39,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 12:52:39,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.81 | bwd_microstep: 5789.06 | bwd_inner_microstep: 5701.42 | bwd_allreduce_microstep: 87.59 | step_microstep: 18.87
[2025-04-25 12:52:39,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.81 | bwd: 5789.07 | bwd_inner: 5701.42 | bwd_allreduce: 87.61 | step: 18.87

  5%|▍         | 2037/41250 [4:55:04<94:38:58,  8.69s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.9664347171783447, 'learning_rate': 3.996065673986812e-05, 'epoch': 0.49}

  5%|▍         | 2037/41250 [4:55:04<94:38:58,  8.69s/it][2025-04-25 12:52:48,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:52:48,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.84 | bwd_microstep: 5741.12 | bwd_inner_microstep: 5662.86 | bwd_allreduce_microstep: 78.22 | step_microstep: 18.35
[2025-04-25 12:52:48,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.84 | bwd: 5741.14 | bwd_inner: 5662.86 | bwd_allreduce: 78.24 | step: 18.36

  5%|▍         | 2038/41250 [4:55:13<94:31:29,  8.68s/it]
                                                         
{'loss': 0.3474, 'grad_norm': 1.750045895576477, 'learning_rate': 3.996055822939039e-05, 'epoch': 0.49}

  5%|▍         | 2038/41250 [4:55:13<94:31:29,  8.68s/it][2025-04-25 12:52:57,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:52:57,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.57 | bwd_microstep: 5886.16 | bwd_inner_microstep: 5716.30 | bwd_allreduce_microstep: 169.82 | step_microstep: 18.29
[2025-04-25 12:52:57,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.57 | bwd: 5886.17 | bwd_inner: 5716.30 | bwd_allreduce: 169.83 | step: 18.30

  5%|▍         | 2039/41250 [4:55:22<95:01:32,  8.72s/it]
                                                         
{'loss': 0.1382, 'grad_norm': 1.2629024982452393, 'learning_rate': 3.996045959585974e-05, 'epoch': 0.49}

  5%|▍         | 2039/41250 [4:55:22<95:01:32,  8.72s/it][2025-04-25 12:53:05,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 12:53:05,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.39 | bwd_microstep: 5804.64 | bwd_inner_microstep: 5658.39 | bwd_allreduce_microstep: 146.20 | step_microstep: 19.27
[2025-04-25 12:53:05,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.39 | bwd: 5804.65 | bwd_inner: 5658.39 | bwd_allreduce: 146.22 | step: 19.27

  5%|▍         | 2040/41250 [4:55:31<95:02:21,  8.73s/it]
                                                         
{'loss': 0.2895, 'grad_norm': 2.663506507873535, 'learning_rate': 3.9960360839276783e-05, 'epoch': 0.49}

  5%|▍         | 2040/41250 [4:55:31<95:02:21,  8.73s/it][2025-04-25 12:53:14,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.01
[2025-04-25 12:53:14,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.61 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5697.71 | bwd_allreduce_microstep: 65.64 | step_microstep: 19.62
[2025-04-25 12:53:14,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.61 | bwd: 5763.41 | bwd_inner: 5697.71 | bwd_allreduce: 65.65 | step: 19.63

  5%|▍         | 2041/41250 [4:55:39<94:58:12,  8.72s/it]
                                                         
{'loss': 0.2988, 'grad_norm': 3.364306688308716, 'learning_rate': 3.996026195964213e-05, 'epoch': 0.49}

  5%|▍         | 2041/41250 [4:55:39<94:58:12,  8.72s/it][2025-04-25 12:53:23,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 12:53:23,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.84 | bwd_microstep: 5706.17 | bwd_inner_microstep: 5672.06 | bwd_allreduce_microstep: 34.06 | step_microstep: 19.15
[2025-04-25 12:53:23,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.84 | bwd: 5706.18 | bwd_inner: 5672.06 | bwd_allreduce: 34.08 | step: 19.15

  5%|▍         | 2042/41250 [4:55:48<94:39:57,  8.69s/it]
                                                         
{'loss': 0.1637, 'grad_norm': 1.010572910308838, 'learning_rate': 3.996016295695639e-05, 'epoch': 0.5}

  5%|▍         | 2042/41250 [4:55:48<94:39:57,  8.69s/it][2025-04-25 12:53:31,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.22 | optimizer_step: 0.92
[2025-04-25 12:53:31,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.72 | bwd_microstep: 5749.17 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 51.21 | step_microstep: 19.57
[2025-04-25 12:53:31,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.72 | bwd: 5749.18 | bwd_inner: 5697.91 | bwd_allreduce: 51.23 | step: 19.57

  5%|▍         | 2043/41250 [4:55:57<94:39:27,  8.69s/it]
                                                         
{'loss': 0.3717, 'grad_norm': 2.4787063598632812, 'learning_rate': 3.9960063831220166e-05, 'epoch': 0.5}

  5%|▍         | 2043/41250 [4:55:57<94:39:27,  8.69s/it][2025-04-25 12:53:40,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 12:53:40,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.84 | bwd_microstep: 5700.00 | bwd_inner_microstep: 5668.09 | bwd_allreduce_microstep: 31.86 | step_microstep: 19.45
[2025-04-25 12:53:40,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.84 | bwd: 5700.01 | bwd_inner: 5668.08 | bwd_allreduce: 31.88 | step: 19.45

  5%|▍         | 2044/41250 [4:56:05<94:24:18,  8.67s/it]
                                                         
{'loss': 0.2951, 'grad_norm': 2.971889019012451, 'learning_rate': 3.9959964582434076e-05, 'epoch': 0.5}

  5%|▍         | 2044/41250 [4:56:05<94:24:18,  8.67s/it][2025-04-25 12:53:49,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:53:49,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.39 | bwd_microstep: 5723.08 | bwd_inner_microstep: 5710.27 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.99
[2025-04-25 12:53:49,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.39 | bwd: 5723.10 | bwd_inner: 5710.27 | bwd_allreduce: 12.79 | step: 19.00

  5%|▍         | 2045/41250 [4:56:14<94:22:11,  8.67s/it]
                                                         
{'loss': 0.1367, 'grad_norm': 1.5779380798339844, 'learning_rate': 3.995986521059874e-05, 'epoch': 0.5}

  5%|▍         | 2045/41250 [4:56:14<94:22:11,  8.67s/it][2025-04-25 12:53:57,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-25 12:53:57,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.59 | bwd_microstep: 5738.36 | bwd_inner_microstep: 5665.56 | bwd_allreduce_microstep: 72.76 | step_microstep: 19.54
[2025-04-25 12:53:57,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.59 | bwd: 5738.38 | bwd_inner: 5665.56 | bwd_allreduce: 72.78 | step: 19.54

  5%|▍         | 2046/41250 [4:56:23<94:22:44,  8.67s/it]
                                                         
{'loss': 0.1261, 'grad_norm': 1.4341769218444824, 'learning_rate': 3.995976571571475e-05, 'epoch': 0.5}

  5%|▍         | 2046/41250 [4:56:23<94:22:44,  8.67s/it][2025-04-25 12:54:06,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 12:54:06,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5713.74 | bwd_inner_microstep: 5701.16 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.56
[2025-04-25 12:54:06,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5713.75 | bwd_inner: 5701.16 | bwd_allreduce: 12.55 | step: 18.56

  5%|▍         | 2047/41250 [4:56:31<94:19:58,  8.66s/it]
                                                         
{'loss': 0.1412, 'grad_norm': 1.4573675394058228, 'learning_rate': 3.995966609778273e-05, 'epoch': 0.5}

  5%|▍         | 2047/41250 [4:56:31<94:19:58,  8.66s/it][2025-04-25 12:54:15,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:54:15,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.65 | bwd_microstep: 5737.73 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 44.40 | step_microstep: 19.05
[2025-04-25 12:54:15,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.65 | bwd: 5737.75 | bwd_inner: 5693.28 | bwd_allreduce: 44.42 | step: 19.05

  5%|▍         | 2048/41250 [4:56:40<94:21:05,  8.66s/it]
                                                         
{'loss': 0.053, 'grad_norm': 0.5802299976348877, 'learning_rate': 3.9959566356803294e-05, 'epoch': 0.5}

  5%|▍         | 2048/41250 [4:56:40<94:21:05,  8.66s/it][2025-04-25 12:54:23,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:54:23,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.33 | bwd_microstep: 5769.18 | bwd_inner_microstep: 5654.40 | bwd_allreduce_microstep: 114.74 | step_microstep: 18.25
[2025-04-25 12:54:23,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.33 | bwd: 5769.19 | bwd_inner: 5654.40 | bwd_allreduce: 114.76 | step: 18.25

  5%|▍         | 2049/41250 [4:56:49<94:26:47,  8.67s/it]
                                                         
{'loss': 0.1056, 'grad_norm': 1.4052115678787231, 'learning_rate': 3.9959466492777066e-05, 'epoch': 0.5}

  5%|▍         | 2049/41250 [4:56:49<94:26:47,  8.67s/it][2025-04-25 12:54:32,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:54:32,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.69 | bwd_microstep: 5697.20 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 38.18 | step_microstep: 18.49
[2025-04-25 12:54:32,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.69 | bwd: 5697.22 | bwd_inner: 5658.97 | bwd_allreduce: 38.20 | step: 18.49

  5%|▍         | 2050/41250 [4:56:57<94:15:47,  8.66s/it]
                                                         
{'loss': 0.2878, 'grad_norm': 2.512564182281494, 'learning_rate': 3.9959366505704646e-05, 'epoch': 0.5}

  5%|▍         | 2050/41250 [4:56:57<94:15:47,  8.66s/it][2025-04-25 12:54:41,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 12:54:41,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.24 | bwd_microstep: 5716.10 | bwd_inner_microstep: 5703.39 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.18
[2025-04-25 12:54:41,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.24 | bwd: 5716.12 | bwd_inner: 5703.39 | bwd_allreduce: 12.69 | step: 19.19

  5%|▍         | 2051/41250 [4:57:06<94:15:20,  8.66s/it]
                                                         
{'loss': 0.145, 'grad_norm': 1.4657329320907593, 'learning_rate': 3.9959266395586656e-05, 'epoch': 0.5}

  5%|▍         | 2051/41250 [4:57:06<94:15:20,  8.66s/it][2025-04-25 12:54:49,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:54:49,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.26 | bwd_microstep: 5691.33 | bwd_inner_microstep: 5654.31 | bwd_allreduce_microstep: 36.98 | step_microstep: 18.82
[2025-04-25 12:54:49,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.26 | bwd: 5691.35 | bwd_inner: 5654.31 | bwd_allreduce: 37.00 | step: 18.82

  5%|▍         | 2052/41250 [4:57:14<94:05:53,  8.64s/it]
                                                         
{'loss': 0.2089, 'grad_norm': 3.4442808628082275, 'learning_rate': 3.995916616242372e-05, 'epoch': 0.5}

  5%|▍         | 2052/41250 [4:57:14<94:05:53,  8.64s/it][2025-04-25 12:54:58,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:54:58,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.28 | bwd_microstep: 5766.14 | bwd_inner_microstep: 5685.89 | bwd_allreduce_microstep: 80.20 | step_microstep: 18.44
[2025-04-25 12:54:58,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.28 | bwd: 5766.16 | bwd_inner: 5685.89 | bwd_allreduce: 80.22 | step: 18.45

  5%|▍         | 2053/41250 [4:57:23<94:15:02,  8.66s/it]
                                                         
{'loss': 0.3892, 'grad_norm': 3.508124351501465, 'learning_rate': 3.995906580621645e-05, 'epoch': 0.5}

  5%|▍         | 2053/41250 [4:57:23<94:15:02,  8.66s/it][2025-04-25 12:55:06,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:55:06,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.26 | bwd_microstep: 5714.14 | bwd_inner_microstep: 5701.53 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.52
[2025-04-25 12:55:06,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.26 | bwd: 5714.15 | bwd_inner: 5701.53 | bwd_allreduce: 12.59 | step: 18.53

  5%|▍         | 2054/41250 [4:57:32<94:12:56,  8.65s/it]
                                                         
{'loss': 0.1462, 'grad_norm': 1.1467552185058594, 'learning_rate': 3.995896532696546e-05, 'epoch': 0.5}

  5%|▍         | 2054/41250 [4:57:32<94:12:56,  8.65s/it][2025-04-25 12:55:15,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 1.09
[2025-04-25 12:55:15,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.44 | bwd_microstep: 5757.58 | bwd_inner_microstep: 5708.17 | bwd_allreduce_microstep: 49.37 | step_microstep: 18.41
[2025-04-25 12:55:15,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.44 | bwd: 5757.59 | bwd_inner: 5708.17 | bwd_allreduce: 49.38 | step: 18.42

  5%|▍         | 2055/41250 [4:57:40<94:19:30,  8.66s/it]
                                                         
{'loss': 0.2592, 'grad_norm': 2.1948139667510986, 'learning_rate': 3.995886472467137e-05, 'epoch': 0.5}

  5%|▍         | 2055/41250 [4:57:40<94:19:30,  8.66s/it][2025-04-25 12:55:24,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 12:55:24,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.27 | bwd_microstep: 5724.46 | bwd_inner_microstep: 5682.04 | bwd_allreduce_microstep: 42.37 | step_microstep: 18.91
[2025-04-25 12:55:24,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.27 | bwd: 5724.47 | bwd_inner: 5682.03 | bwd_allreduce: 42.39 | step: 18.91

  5%|▍         | 2056/41250 [4:57:49<94:17:24,  8.66s/it]
                                                         
{'loss': 0.3371, 'grad_norm': 6.633668422698975, 'learning_rate': 3.995876399933482e-05, 'epoch': 0.5}

  5%|▍         | 2056/41250 [4:57:49<94:17:24,  8.66s/it][2025-04-25 12:55:32,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:55:32,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.51 | bwd_microstep: 5756.39 | bwd_inner_microstep: 5652.48 | bwd_allreduce_microstep: 103.87 | step_microstep: 18.48
[2025-04-25 12:55:32,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.51 | bwd: 5756.40 | bwd_inner: 5652.48 | bwd_allreduce: 103.89 | step: 18.48

  5%|▍         | 2057/41250 [4:57:58<94:18:51,  8.66s/it]
                                                         
{'loss': 0.3206, 'grad_norm': 3.186187982559204, 'learning_rate': 3.99586631509564e-05, 'epoch': 0.5}

  5%|▍         | 2057/41250 [4:57:58<94:18:51,  8.66s/it][2025-04-25 12:55:41,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 12:55:41,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.52 | bwd_microstep: 5717.82 | bwd_inner_microstep: 5689.14 | bwd_allreduce_microstep: 28.64 | step_microstep: 18.53
[2025-04-25 12:55:41,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.52 | bwd: 5717.83 | bwd_inner: 5689.14 | bwd_allreduce: 28.65 | step: 18.53

  5%|▍         | 2058/41250 [4:58:06<94:17:41,  8.66s/it]
                                                         
{'loss': 0.0867, 'grad_norm': 1.0649446249008179, 'learning_rate': 3.9958562179536754e-05, 'epoch': 0.5}

  5%|▍         | 2058/41250 [4:58:06<94:17:41,  8.66s/it][2025-04-25 12:55:50,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:55:50,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.75 | bwd_microstep: 5741.08 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 92.06 | step_microstep: 18.29
[2025-04-25 12:55:50,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.75 | bwd: 5741.09 | bwd_inner: 5648.98 | bwd_allreduce: 92.08 | step: 18.29

  5%|▍         | 2059/41250 [4:58:15<94:16:06,  8.66s/it]
                                                         
{'loss': 0.2996, 'grad_norm': 2.606330394744873, 'learning_rate': 3.9958461085076505e-05, 'epoch': 0.5}

  5%|▍         | 2059/41250 [4:58:15<94:16:06,  8.66s/it][2025-04-25 12:55:59,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.85 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:55:59,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.98 | bwd_microstep: 5788.47 | bwd_inner_microstep: 5775.84 | bwd_allreduce_microstep: 12.59 | step_microstep: 17.68
[2025-04-25 12:55:59,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.98 | bwd: 5788.48 | bwd_inner: 5775.84 | bwd_allreduce: 12.61 | step: 17.68

  5%|▍         | 2060/41250 [4:58:24<94:35:19,  8.69s/it]
                                                         
{'loss': 0.3564, 'grad_norm': 1.6368298530578613, 'learning_rate': 3.9958359867576256e-05, 'epoch': 0.5}

  5%|▍         | 2060/41250 [4:58:24<94:35:19,  8.69s/it][2025-04-25 12:56:07,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:56:07,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.22 | bwd_microstep: 5675.98 | bwd_inner_microstep: 5638.43 | bwd_allreduce_microstep: 37.51 | step_microstep: 18.30
[2025-04-25 12:56:07,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.22 | bwd: 5675.99 | bwd_inner: 5638.43 | bwd_allreduce: 37.52 | step: 18.30

  5%|▍         | 2061/41250 [4:58:32<94:15:04,  8.66s/it]
                                                         
{'loss': 0.2772, 'grad_norm': 2.373802661895752, 'learning_rate': 3.995825852703665e-05, 'epoch': 0.5}

  5%|▍         | 2061/41250 [4:58:32<94:15:04,  8.66s/it][2025-04-25 12:56:16,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 12:56:16,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5668.49 | bwd_inner_microstep: 5645.97 | bwd_allreduce_microstep: 22.47 | step_microstep: 18.56
[2025-04-25 12:56:16,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5668.50 | bwd_inner: 5645.97 | bwd_allreduce: 22.49 | step: 18.56

  5%|▍         | 2062/41250 [4:58:41<93:58:55,  8.63s/it]
                                                         
{'loss': 0.2011, 'grad_norm': 1.1623815298080444, 'learning_rate': 3.9958157063458315e-05, 'epoch': 0.5}

  5%|▍         | 2062/41250 [4:58:41<93:58:55,  8.63s/it][2025-04-25 12:56:24,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:56:24,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.52 | bwd_microstep: 5695.05 | bwd_inner_microstep: 5653.04 | bwd_allreduce_microstep: 41.97 | step_microstep: 17.91
[2025-04-25 12:56:24,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.52 | bwd: 5695.06 | bwd_inner: 5653.03 | bwd_allreduce: 41.99 | step: 17.92

  5%|▌         | 2063/41250 [4:58:50<93:51:54,  8.62s/it]
                                                         
{'loss': 0.0738, 'grad_norm': 0.8325732350349426, 'learning_rate': 3.9958055476841856e-05, 'epoch': 0.5}

  5%|▌         | 2063/41250 [4:58:50<93:51:54,  8.62s/it][2025-04-25 12:56:33,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 12:56:33,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5709.95 | bwd_inner_microstep: 5677.90 | bwd_allreduce_microstep: 32.01 | step_microstep: 19.17
[2025-04-25 12:56:33,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5709.96 | bwd_inner: 5677.90 | bwd_allreduce: 32.03 | step: 19.17

  5%|▌         | 2064/41250 [4:58:58<93:55:34,  8.63s/it]
                                                         
{'loss': 0.2501, 'grad_norm': 2.5584003925323486, 'learning_rate': 3.995795376718791e-05, 'epoch': 0.5}

  5%|▌         | 2064/41250 [4:58:58<93:55:34,  8.63s/it][2025-04-25 12:56:42,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:56:42,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.03 | bwd_microstep: 5694.77 | bwd_inner_microstep: 5682.17 | bwd_allreduce_microstep: 12.55 | step_microstep: 17.93
[2025-04-25 12:56:42,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.03 | bwd: 5694.78 | bwd_inner: 5682.17 | bwd_allreduce: 12.57 | step: 17.93

  5%|▌         | 2065/41250 [4:59:07<93:55:19,  8.63s/it]
                                                         
{'loss': 0.3276, 'grad_norm': 3.8959150314331055, 'learning_rate': 3.995785193449711e-05, 'epoch': 0.5}

  5%|▌         | 2065/41250 [4:59:07<93:55:19,  8.63s/it][2025-04-25 12:56:50,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 12:56:50,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.71 | bwd_microstep: 5732.53 | bwd_inner_microstep: 5684.28 | bwd_allreduce_microstep: 48.20 | step_microstep: 18.58
[2025-04-25 12:56:50,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.71 | bwd: 5732.54 | bwd_inner: 5684.28 | bwd_allreduce: 48.23 | step: 18.58

  5%|▌         | 2066/41250 [4:59:16<93:59:56,  8.64s/it]
                                                         
{'loss': 0.2087, 'grad_norm': 1.9350131750106812, 'learning_rate': 3.995774997877008e-05, 'epoch': 0.5}

  5%|▌         | 2066/41250 [4:59:16<93:59:56,  8.64s/it][2025-04-25 12:56:59,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:56:59,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.48 | bwd_microstep: 5665.51 | bwd_inner_microstep: 5645.60 | bwd_allreduce_microstep: 19.87 | step_microstep: 18.28
[2025-04-25 12:56:59,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.48 | bwd: 5665.52 | bwd_inner: 5645.60 | bwd_allreduce: 19.89 | step: 18.28

  5%|▌         | 2067/41250 [4:59:24<93:48:48,  8.62s/it]
                                                         
{'loss': 0.4297, 'grad_norm': 2.402545690536499, 'learning_rate': 3.995764790000744e-05, 'epoch': 0.5}

  5%|▌         | 2067/41250 [4:59:24<93:48:48,  8.62s/it][2025-04-25 12:57:07,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:57:07,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.62 | bwd_microstep: 5744.99 | bwd_inner_microstep: 5642.87 | bwd_allreduce_microstep: 102.07 | step_microstep: 18.46
[2025-04-25 12:57:07,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.62 | bwd: 5745.00 | bwd_inner: 5642.87 | bwd_allreduce: 102.09 | step: 18.47

  5%|▌         | 2068/41250 [4:59:33<93:56:17,  8.63s/it]
                                                         
{'loss': 0.0806, 'grad_norm': 1.493162989616394, 'learning_rate': 3.995754569820982e-05, 'epoch': 0.5}

  5%|▌         | 2068/41250 [4:59:33<93:56:17,  8.63s/it][2025-04-25 12:57:16,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:57:16,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.77 | bwd_microstep: 5728.88 | bwd_inner_microstep: 5636.56 | bwd_allreduce_microstep: 92.28 | step_microstep: 18.05
[2025-04-25 12:57:16,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.77 | bwd: 5728.90 | bwd_inner: 5636.56 | bwd_allreduce: 92.30 | step: 18.05

  5%|▌         | 2069/41250 [4:59:41<93:59:22,  8.64s/it]
                                                         
{'loss': 0.2027, 'grad_norm': 1.685293436050415, 'learning_rate': 3.995744337337787e-05, 'epoch': 0.5}

  5%|▌         | 2069/41250 [4:59:41<93:59:22,  8.64s/it][2025-04-25 12:57:25,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 12:57:25,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5751.27 | bwd_inner_microstep: 5637.30 | bwd_allreduce_microstep: 113.93 | step_microstep: 18.60
[2025-04-25 12:57:25,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5751.29 | bwd_inner: 5637.30 | bwd_allreduce: 113.95 | step: 18.61

  5%|▌         | 2070/41250 [4:59:50<94:05:23,  8.65s/it]
                                                         
{'loss': 0.215, 'grad_norm': 1.7019606828689575, 'learning_rate': 3.99573409255122e-05, 'epoch': 0.5}

  5%|▌         | 2070/41250 [4:59:50<94:05:23,  8.65s/it][2025-04-25 12:57:33,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 12:57:33,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.93 | bwd_microstep: 5725.33 | bwd_inner_microstep: 5641.51 | bwd_allreduce_microstep: 83.78 | step_microstep: 18.25
[2025-04-25 12:57:33,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.93 | bwd: 5725.35 | bwd_inner: 5641.51 | bwd_allreduce: 83.79 | step: 18.26

  5%|▌         | 2071/41250 [4:59:59<94:05:48,  8.65s/it]
                                                         
{'loss': 0.2146, 'grad_norm': 1.9806808233261108, 'learning_rate': 3.9957238354613455e-05, 'epoch': 0.5}

  5%|▌         | 2071/41250 [4:59:59<94:05:48,  8.65s/it][2025-04-25 12:57:42,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.10 | optimizer_step: 0.90
[2025-04-25 12:57:42,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.29 | bwd_microstep: 5709.01 | bwd_inner_microstep: 5695.55 | bwd_allreduce_microstep: 13.41 | step_microstep: 19.31
[2025-04-25 12:57:42,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.29 | bwd: 5709.03 | bwd_inner: 5695.55 | bwd_allreduce: 13.43 | step: 19.31

  5%|▌         | 2072/41250 [5:00:07<94:05:20,  8.65s/it]
                                                         
{'loss': 0.1999, 'grad_norm': 1.7702255249023438, 'learning_rate': 3.9957135660682256e-05, 'epoch': 0.5}

  5%|▌         | 2072/41250 [5:00:07<94:05:20,  8.65s/it][2025-04-25 12:57:51,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-25 12:57:51,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.66 | bwd_microstep: 5737.05 | bwd_inner_microstep: 5681.62 | bwd_allreduce_microstep: 55.38 | step_microstep: 19.32
[2025-04-25 12:57:51,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.66 | bwd: 5737.06 | bwd_inner: 5681.62 | bwd_allreduce: 55.40 | step: 19.32

  5%|▌         | 2073/41250 [5:00:16<94:08:41,  8.65s/it]
                                                         
{'loss': 0.3342, 'grad_norm': 2.1663413047790527, 'learning_rate': 3.995703284371924e-05, 'epoch': 0.5}

  5%|▌         | 2073/41250 [5:00:16<94:08:41,  8.65s/it][2025-04-25 12:57:59,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 12:57:59,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.38 | bwd_microstep: 5735.25 | bwd_inner_microstep: 5689.00 | bwd_allreduce_microstep: 46.20 | step_microstep: 19.00
[2025-04-25 12:57:59,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.38 | bwd: 5735.26 | bwd_inner: 5689.00 | bwd_allreduce: 46.22 | step: 19.00

  5%|▌         | 2074/41250 [5:00:25<94:13:48,  8.66s/it]
                                                         
{'loss': 0.1367, 'grad_norm': 1.372663974761963, 'learning_rate': 3.995692990372504e-05, 'epoch': 0.5}

  5%|▌         | 2074/41250 [5:00:25<94:13:48,  8.66s/it][2025-04-25 12:58:08,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.27 | optimizer_step: 1.02
[2025-04-25 12:58:08,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.93 | bwd_microstep: 5773.86 | bwd_inner_microstep: 5661.08 | bwd_allreduce_microstep: 112.72 | step_microstep: 19.72
[2025-04-25 12:58:08,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.93 | bwd: 5773.88 | bwd_inner: 5661.08 | bwd_allreduce: 112.75 | step: 19.72

  5%|▌         | 2075/41250 [5:00:33<94:18:26,  8.67s/it]
                                                         
{'loss': 0.2152, 'grad_norm': 2.9173123836517334, 'learning_rate': 3.99568268407003e-05, 'epoch': 0.5}

  5%|▌         | 2075/41250 [5:00:33<94:18:26,  8.67s/it][2025-04-25 12:58:17,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 12:58:17,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.85 | bwd_microstep: 5679.87 | bwd_inner_microstep: 5658.68 | bwd_allreduce_microstep: 21.15 | step_microstep: 18.02
[2025-04-25 12:58:17,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.85 | bwd: 5679.88 | bwd_inner: 5658.68 | bwd_allreduce: 21.16 | step: 18.02

  5%|▌         | 2076/41250 [5:00:42<94:02:58,  8.64s/it]
                                                         
{'loss': 0.1968, 'grad_norm': 1.157960057258606, 'learning_rate': 3.995672365464564e-05, 'epoch': 0.5}

  5%|▌         | 2076/41250 [5:00:42<94:02:58,  8.64s/it][2025-04-25 12:58:25,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 12:58:25,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.54 | bwd_microstep: 5723.58 | bwd_inner_microstep: 5689.44 | bwd_allreduce_microstep: 34.09 | step_microstep: 18.00
[2025-04-25 12:58:25,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5723.59 | bwd_inner: 5689.44 | bwd_allreduce: 34.11 | step: 18.01

  5%|▌         | 2077/41250 [5:00:51<94:05:30,  8.65s/it]
                                                         
{'loss': 0.1557, 'grad_norm': 0.8962106704711914, 'learning_rate': 3.995662034556171e-05, 'epoch': 0.5}

  5%|▌         | 2077/41250 [5:00:51<94:05:30,  8.65s/it][2025-04-25 12:58:34,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 12:58:34,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.53 | bwd_microstep: 5754.77 | bwd_inner_microstep: 5704.93 | bwd_allreduce_microstep: 49.79 | step_microstep: 18.39
[2025-04-25 12:58:34,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.53 | bwd: 5754.78 | bwd_inner: 5704.93 | bwd_allreduce: 49.81 | step: 18.40

  5%|▌         | 2078/41250 [5:00:59<94:12:29,  8.66s/it]
                                                         
{'loss': 0.1689, 'grad_norm': 1.3917365074157715, 'learning_rate': 3.995651691344914e-05, 'epoch': 0.5}

  5%|▌         | 2078/41250 [5:00:59<94:12:29,  8.66s/it][2025-04-25 12:58:43,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:58:43,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.31 | bwd_microstep: 5718.84 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.35
[2025-04-25 12:58:43,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.31 | bwd: 5718.85 | bwd_inner: 5706.20 | bwd_allreduce: 12.61 | step: 18.36

  5%|▌         | 2079/41250 [5:01:08<94:11:31,  8.66s/it]
                                                         
{'loss': 0.2906, 'grad_norm': 1.6591037511825562, 'learning_rate': 3.995641335830857e-05, 'epoch': 0.5}

  5%|▌         | 2079/41250 [5:01:08<94:11:31,  8.66s/it][2025-04-25 12:58:51,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 12:58:51,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.61 | bwd_microstep: 5708.60 | bwd_inner_microstep: 5695.95 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.37
[2025-04-25 12:58:51,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.61 | bwd: 5708.61 | bwd_inner: 5695.95 | bwd_allreduce: 12.62 | step: 18.38

  5%|▌         | 2080/41250 [5:01:17<94:07:36,  8.65s/it]
                                                         
{'loss': 0.1939, 'grad_norm': 1.599577784538269, 'learning_rate': 3.995630968014063e-05, 'epoch': 0.5}

  5%|▌         | 2080/41250 [5:01:17<94:07:36,  8.65s/it][2025-04-25 12:59:00,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 12:59:00,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.55 | bwd_microstep: 5728.74 | bwd_inner_microstep: 5694.95 | bwd_allreduce_microstep: 33.75 | step_microstep: 18.20
[2025-04-25 12:59:00,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.55 | bwd: 5728.76 | bwd_inner: 5694.95 | bwd_allreduce: 33.77 | step: 18.20

  5%|▌         | 2081/41250 [5:01:25<94:11:13,  8.66s/it]
                                                         
{'loss': 0.2454, 'grad_norm': 2.3855888843536377, 'learning_rate': 3.995620587894597e-05, 'epoch': 0.5}

  5%|▌         | 2081/41250 [5:01:25<94:11:13,  8.66s/it][2025-04-25 12:59:09,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 12:59:09,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.63 | bwd_microstep: 5706.30 | bwd_inner_microstep: 5693.67 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.12
[2025-04-25 12:59:09,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.63 | bwd: 5706.31 | bwd_inner: 5693.67 | bwd_allreduce: 12.61 | step: 18.13

  5%|▌         | 2082/41250 [5:01:34<94:05:45,  8.65s/it]
                                                         
{'loss': 0.1764, 'grad_norm': 1.2183910608291626, 'learning_rate': 3.995610195472522e-05, 'epoch': 0.5}

  5%|▌         | 2082/41250 [5:01:34<94:05:45,  8.65s/it][2025-04-25 12:59:17,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 12:59:17,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5707.46 | bwd_inner_microstep: 5694.74 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.53
[2025-04-25 12:59:17,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5707.47 | bwd_inner: 5694.74 | bwd_allreduce: 12.69 | step: 18.54

  5%|▌         | 2083/41250 [5:01:43<94:03:52,  8.65s/it]
                                                         
{'loss': 0.2169, 'grad_norm': 1.6302655935287476, 'learning_rate': 3.9955997907479035e-05, 'epoch': 0.5}

  5%|▌         | 2083/41250 [5:01:43<94:03:52,  8.65s/it][2025-04-25 12:59:26,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 12:59:26,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.80 | bwd_microstep: 5713.00 | bwd_inner_microstep: 5700.46 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.26
[2025-04-25 12:59:26,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.80 | bwd: 5713.01 | bwd_inner: 5700.46 | bwd_allreduce: 12.51 | step: 18.26

  5%|▌         | 2084/41250 [5:01:51<94:02:26,  8.64s/it]
                                                         
{'loss': 0.0895, 'grad_norm': 1.5756571292877197, 'learning_rate': 3.995589373720805e-05, 'epoch': 0.51}

  5%|▌         | 2084/41250 [5:01:51<94:02:26,  8.64s/it][2025-04-25 12:59:35,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 12:59:35,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.78 | bwd_microstep: 5774.58 | bwd_inner_microstep: 5661.15 | bwd_allreduce_microstep: 113.39 | step_microstep: 18.20
[2025-04-25 12:59:35,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.78 | bwd: 5774.60 | bwd_inner: 5661.15 | bwd_allreduce: 113.41 | step: 18.21

  5%|▌         | 2085/41250 [5:02:00<94:10:56,  8.66s/it]
                                                         
{'loss': 0.1381, 'grad_norm': 1.312493920326233, 'learning_rate': 3.99557894439129e-05, 'epoch': 0.51}

  5%|▌         | 2085/41250 [5:02:00<94:10:56,  8.66s/it][2025-04-25 12:59:43,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 12:59:43,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.34 | bwd_microstep: 5696.59 | bwd_inner_microstep: 5667.87 | bwd_allreduce_microstep: 28.68 | step_microstep: 19.25
[2025-04-25 12:59:43,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.34 | bwd: 5696.61 | bwd_inner: 5667.87 | bwd_allreduce: 28.70 | step: 19.26

  5%|▌         | 2086/41250 [5:02:09<94:03:55,  8.65s/it]
                                                         
{'loss': 0.1206, 'grad_norm': 1.661210298538208, 'learning_rate': 3.995568502759423e-05, 'epoch': 0.51}

  5%|▌         | 2086/41250 [5:02:09<94:03:55,  8.65s/it][2025-04-25 12:59:52,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 12:59:52,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.40 | bwd_microstep: 5762.75 | bwd_inner_microstep: 5711.44 | bwd_allreduce_microstep: 51.26 | step_microstep: 18.36
[2025-04-25 12:59:52,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.40 | bwd: 5762.76 | bwd_inner: 5711.44 | bwd_allreduce: 51.28 | step: 18.37

  5%|▌         | 2087/41250 [5:02:17<94:14:53,  8.66s/it]
                                                         
{'loss': 0.0847, 'grad_norm': 0.8235859274864197, 'learning_rate': 3.9955580488252694e-05, 'epoch': 0.51}

  5%|▌         | 2087/41250 [5:02:17<94:14:53,  8.66s/it][2025-04-25 13:00:01,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:00:01,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.33 | bwd_microstep: 5774.06 | bwd_inner_microstep: 5668.61 | bwd_allreduce_microstep: 105.40 | step_microstep: 18.32
[2025-04-25 13:00:01,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.33 | bwd: 5774.07 | bwd_inner: 5668.61 | bwd_allreduce: 105.42 | step: 18.33

  5%|▌         | 2088/41250 [5:02:26<94:23:05,  8.68s/it]
                                                         
{'loss': 0.2322, 'grad_norm': 1.2210437059402466, 'learning_rate': 3.9955475825888923e-05, 'epoch': 0.51}

  5%|▌         | 2088/41250 [5:02:26<94:23:05,  8.68s/it][2025-04-25 13:00:09,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:00:09,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.55 | bwd_microstep: 5761.61 | bwd_inner_microstep: 5660.98 | bwd_allreduce_microstep: 100.58 | step_microstep: 18.53
[2025-04-25 13:00:09,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.55 | bwd: 5761.62 | bwd_inner: 5660.98 | bwd_allreduce: 100.60 | step: 18.53

  5%|▌         | 2089/41250 [5:02:35<94:23:45,  8.68s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 0.7561390995979309, 'learning_rate': 3.995537104050357e-05, 'epoch': 0.51}

  5%|▌         | 2089/41250 [5:02:35<94:23:45,  8.68s/it][2025-04-25 13:00:18,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 13:00:18,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.17 | bwd_microstep: 5800.02 | bwd_inner_microstep: 5787.05 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.29
[2025-04-25 13:00:18,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.17 | bwd: 5800.04 | bwd_inner: 5787.05 | bwd_allreduce: 12.95 | step: 18.30

  5%|▌         | 2090/41250 [5:02:43<94:43:49,  8.71s/it]
                                                         
{'loss': 0.3189, 'grad_norm': 1.5580612421035767, 'learning_rate': 3.995526613209729e-05, 'epoch': 0.51}

  5%|▌         | 2090/41250 [5:02:43<94:43:49,  8.71s/it][2025-04-25 13:00:27,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 13:00:27,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2899.38 | bwd_microstep: 5801.02 | bwd_inner_microstep: 5787.97 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.03
[2025-04-25 13:00:27,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2899.38 | bwd: 5801.03 | bwd_inner: 5787.97 | bwd_allreduce: 13.02 | step: 19.04

  5%|▌         | 2091/41250 [5:02:52<94:58:47,  8.73s/it]
                                                         
{'loss': 0.1638, 'grad_norm': 2.102071762084961, 'learning_rate': 3.9955161100670706e-05, 'epoch': 0.51}

  5%|▌         | 2091/41250 [5:02:52<94:58:47,  8.73s/it][2025-04-25 13:00:36,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:00:36,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.95 | bwd_microstep: 5787.27 | bwd_inner_microstep: 5673.81 | bwd_allreduce_microstep: 113.40 | step_microstep: 18.77
[2025-04-25 13:00:36,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.95 | bwd: 5787.28 | bwd_inner: 5673.81 | bwd_allreduce: 113.42 | step: 18.77

  5%|▌         | 2092/41250 [5:03:01<94:53:24,  8.72s/it]
                                                         
{'loss': 0.1522, 'grad_norm': 1.4591456651687622, 'learning_rate': 3.995505594622448e-05, 'epoch': 0.51}

  5%|▌         | 2092/41250 [5:03:01<94:53:24,  8.72s/it][2025-04-25 13:00:44,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 13:00:44,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.16 | bwd_microstep: 5799.49 | bwd_inner_microstep: 5663.14 | bwd_allreduce_microstep: 136.30 | step_microstep: 18.79
[2025-04-25 13:00:44,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.16 | bwd: 5799.50 | bwd_inner: 5663.14 | bwd_allreduce: 136.32 | step: 18.79

  5%|▌         | 2093/41250 [5:03:10<94:51:59,  8.72s/it]
                                                         
{'loss': 0.1709, 'grad_norm': 2.4952609539031982, 'learning_rate': 3.995495066875927e-05, 'epoch': 0.51}

  5%|▌         | 2093/41250 [5:03:10<94:51:59,  8.72s/it][2025-04-25 13:00:53,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 13:00:53,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.11 | bwd_microstep: 5782.25 | bwd_inner_microstep: 5666.28 | bwd_allreduce_microstep: 115.93 | step_microstep: 18.86
[2025-04-25 13:00:53,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.11 | bwd: 5782.26 | bwd_inner: 5666.28 | bwd_allreduce: 115.94 | step: 18.87

  5%|▌         | 2094/41250 [5:03:18<94:48:10,  8.72s/it]
                                                         
{'loss': 0.224, 'grad_norm': 2.133096218109131, 'learning_rate': 3.99548452682757e-05, 'epoch': 0.51}

  5%|▌         | 2094/41250 [5:03:18<94:48:10,  8.72s/it][2025-04-25 13:01:02,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:01:02,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.31 | bwd_microstep: 5763.38 | bwd_inner_microstep: 5723.76 | bwd_allreduce_microstep: 39.57 | step_microstep: 19.06
[2025-04-25 13:01:02,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.31 | bwd: 5763.39 | bwd_inner: 5723.76 | bwd_allreduce: 39.59 | step: 19.06

  5%|▌         | 2095/41250 [5:03:27<94:46:53,  8.71s/it]
                                                         
{'loss': 0.2827, 'grad_norm': 1.6148546934127808, 'learning_rate': 3.995473974477443e-05, 'epoch': 0.51}

  5%|▌         | 2095/41250 [5:03:27<94:46:53,  8.71s/it][2025-04-25 13:01:10,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:01:10,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.11 | bwd_microstep: 5725.25 | bwd_inner_microstep: 5712.47 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.86
[2025-04-25 13:01:10,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.11 | bwd: 5725.26 | bwd_inner: 5712.47 | bwd_allreduce: 12.75 | step: 18.86

  5%|▌         | 2096/41250 [5:03:36<94:37:23,  8.70s/it]
                                                         
{'loss': 0.1631, 'grad_norm': 2.225241184234619, 'learning_rate': 3.995463409825613e-05, 'epoch': 0.51}

  5%|▌         | 2096/41250 [5:03:36<94:37:23,  8.70s/it][2025-04-25 13:01:19,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:01:19,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.41 | bwd_microstep: 5710.63 | bwd_inner_microstep: 5663.89 | bwd_allreduce_microstep: 46.69 | step_microstep: 19.08
[2025-04-25 13:01:19,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.41 | bwd: 5710.65 | bwd_inner: 5663.89 | bwd_allreduce: 46.71 | step: 19.08

  5%|▌         | 2097/41250 [5:03:44<94:23:08,  8.68s/it]
                                                         
{'loss': 0.2588, 'grad_norm': 1.7690958976745605, 'learning_rate': 3.9954528328721434e-05, 'epoch': 0.51}

  5%|▌         | 2097/41250 [5:03:44<94:23:08,  8.68s/it][2025-04-25 13:01:28,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.32 | optimizer_step: 1.01
[2025-04-25 13:01:28,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.23 | bwd_microstep: 5699.92 | bwd_inner_microstep: 5686.38 | bwd_allreduce_microstep: 13.48 | step_microstep: 20.07
[2025-04-25 13:01:28,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.23 | bwd: 5699.93 | bwd_inner: 5686.38 | bwd_allreduce: 13.51 | step: 20.07

  5%|▌         | 2098/41250 [5:03:53<94:14:16,  8.67s/it]
                                                         
{'loss': 0.1049, 'grad_norm': 2.348621368408203, 'learning_rate': 3.995442243617099e-05, 'epoch': 0.51}

  5%|▌         | 2098/41250 [5:03:53<94:14:16,  8.67s/it][2025-04-25 13:01:36,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 13:01:36,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.25 | bwd_microstep: 5717.65 | bwd_inner_microstep: 5704.88 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.80
[2025-04-25 13:01:36,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.25 | bwd: 5717.67 | bwd_inner: 5704.88 | bwd_allreduce: 12.75 | step: 18.80

  5%|▌         | 2099/41250 [5:04:02<94:15:20,  8.67s/it]
                                                         
{'loss': 0.1741, 'grad_norm': 1.4242901802062988, 'learning_rate': 3.995431642060546e-05, 'epoch': 0.51}

  5%|▌         | 2099/41250 [5:04:02<94:15:20,  8.67s/it][2025-04-25 13:01:45,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 13:01:45,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.98 | bwd_microstep: 5779.26 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 85.93 | step_microstep: 18.60
[2025-04-25 13:01:45,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.98 | bwd: 5779.27 | bwd_inner: 5693.28 | bwd_allreduce: 85.95 | step: 18.61

  5%|▌         | 2100/41250 [5:04:10<94:26:56,  8.68s/it]
                                                         
{'loss': 0.2777, 'grad_norm': 2.2205257415771484, 'learning_rate': 3.995421028202549e-05, 'epoch': 0.51}

  5%|▌         | 2100/41250 [5:04:10<94:26:56,  8.68s/it][2025-04-25 13:01:54,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:01:54,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.93 | bwd_microstep: 5704.78 | bwd_inner_microstep: 5652.91 | bwd_allreduce_microstep: 51.83 | step_microstep: 18.78
[2025-04-25 13:01:54,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.93 | bwd: 5704.80 | bwd_inner: 5652.91 | bwd_allreduce: 51.85 | step: 18.79

  5%|▌         | 2101/41250 [5:04:19<94:16:10,  8.67s/it]
                                                         
{'loss': 0.1677, 'grad_norm': 2.35071063041687, 'learning_rate': 3.995410402043174e-05, 'epoch': 0.51}

  5%|▌         | 2101/41250 [5:04:19<94:16:10,  8.67s/it][2025-04-25 13:02:02,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:02:02,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.94 | bwd_microstep: 5752.34 | bwd_inner_microstep: 5699.96 | bwd_allreduce_microstep: 52.33 | step_microstep: 18.21
[2025-04-25 13:02:02,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.94 | bwd: 5752.35 | bwd_inner: 5699.96 | bwd_allreduce: 52.35 | step: 18.21

  5%|▌         | 2102/41250 [5:04:28<94:18:04,  8.67s/it]
                                                         
{'loss': 0.127, 'grad_norm': 1.383833646774292, 'learning_rate': 3.9953997635824864e-05, 'epoch': 0.51}

  5%|▌         | 2102/41250 [5:04:28<94:18:04,  8.67s/it][2025-04-25 13:02:11,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 13:02:11,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.01 | bwd_microstep: 5714.97 | bwd_inner_microstep: 5702.28 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.50
[2025-04-25 13:02:11,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.01 | bwd: 5714.99 | bwd_inner: 5702.28 | bwd_allreduce: 12.67 | step: 18.50

  5%|▌         | 2103/41250 [5:04:36<94:15:10,  8.67s/it]
                                                         
{'loss': 0.2691, 'grad_norm': 2.1111631393432617, 'learning_rate': 3.995389112820552e-05, 'epoch': 0.51}

  5%|▌         | 2103/41250 [5:04:36<94:15:10,  8.67s/it][2025-04-25 13:02:20,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.20
[2025-04-25 13:02:20,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.89 | bwd_microstep: 5690.71 | bwd_inner_microstep: 5653.82 | bwd_allreduce_microstep: 36.85 | step_microstep: 19.26
[2025-04-25 13:02:20,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.89 | bwd: 5690.73 | bwd_inner: 5653.82 | bwd_allreduce: 36.86 | step: 19.26

  5%|▌         | 2104/41250 [5:04:45<94:04:19,  8.65s/it]
                                                         
{'loss': 0.3068, 'grad_norm': 1.712180495262146, 'learning_rate': 3.995378449757436e-05, 'epoch': 0.51}

  5%|▌         | 2104/41250 [5:04:45<94:04:19,  8.65s/it][2025-04-25 13:02:29,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:02:29,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.97 | bwd_microstep: 6005.12 | bwd_inner_microstep: 5693.83 | bwd_allreduce_microstep: 311.24 | step_microstep: 18.82
[2025-04-25 13:02:29,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.97 | bwd: 6005.14 | bwd_inner: 5693.83 | bwd_allreduce: 311.26 | step: 18.83

  5%|▌         | 2105/41250 [5:04:54<95:00:42,  8.74s/it]
                                                         
{'loss': 0.0913, 'grad_norm': 0.9111629128456116, 'learning_rate': 3.995367774393204e-05, 'epoch': 0.51}

  5%|▌         | 2105/41250 [5:04:54<95:00:42,  8.74s/it][2025-04-25 13:02:37,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:02:37,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.07 | bwd_microstep: 5709.16 | bwd_inner_microstep: 5696.45 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.23
[2025-04-25 13:02:37,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.07 | bwd: 5709.18 | bwd_inner: 5696.45 | bwd_allreduce: 12.69 | step: 18.23

  5%|▌         | 2106/41250 [5:05:03<94:41:40,  8.71s/it]
                                                         
{'loss': 0.1848, 'grad_norm': 1.089030146598816, 'learning_rate': 3.9953570867279225e-05, 'epoch': 0.51}

  5%|▌         | 2106/41250 [5:05:03<94:41:40,  8.71s/it][2025-04-25 13:02:46,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:02:46,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.33 | bwd_microstep: 5762.15 | bwd_inner_microstep: 5645.06 | bwd_allreduce_microstep: 117.05 | step_microstep: 18.22
[2025-04-25 13:02:46,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.33 | bwd: 5762.16 | bwd_inner: 5645.06 | bwd_allreduce: 117.07 | step: 18.23

  5%|▌         | 2107/41250 [5:05:11<94:36:13,  8.70s/it]
                                                         
{'loss': 0.0553, 'grad_norm': 0.44068798422813416, 'learning_rate': 3.995346386761657e-05, 'epoch': 0.51}

  5%|▌         | 2107/41250 [5:05:11<94:36:13,  8.70s/it][2025-04-25 13:02:55,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:02:55,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.73 | bwd_microstep: 5753.27 | bwd_inner_microstep: 5689.85 | bwd_allreduce_microstep: 63.37 | step_microstep: 18.53
[2025-04-25 13:02:55,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.73 | bwd: 5753.28 | bwd_inner: 5689.85 | bwd_allreduce: 63.39 | step: 18.54

  5%|▌         | 2108/41250 [5:05:20<94:32:13,  8.69s/it]
                                                         
{'loss': 0.1552, 'grad_norm': 1.3742159605026245, 'learning_rate': 3.995335674494473e-05, 'epoch': 0.51}

  5%|▌         | 2108/41250 [5:05:20<94:32:13,  8.69s/it][2025-04-25 13:03:03,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 13:03:03,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.54 | bwd_microstep: 5688.31 | bwd_inner_microstep: 5662.39 | bwd_allreduce_microstep: 25.88 | step_microstep: 18.79
[2025-04-25 13:03:03,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.54 | bwd: 5688.33 | bwd_inner: 5662.39 | bwd_allreduce: 25.90 | step: 18.80

  5%|▌         | 2109/41250 [5:05:28<94:13:30,  8.67s/it]
                                                         
{'loss': 0.2212, 'grad_norm': 1.815258264541626, 'learning_rate': 3.995324949926438e-05, 'epoch': 0.51}

  5%|▌         | 2109/41250 [5:05:28<94:13:30,  8.67s/it][2025-04-25 13:03:12,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:03:12,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5755.03 | bwd_inner_microstep: 5654.90 | bwd_allreduce_microstep: 100.09 | step_microstep: 18.84
[2025-04-25 13:03:12,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5755.05 | bwd_inner: 5654.90 | bwd_allreduce: 100.11 | step: 18.85

  5%|▌         | 2110/41250 [5:05:37<94:12:20,  8.66s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 2.1753172874450684, 'learning_rate': 3.995314213057616e-05, 'epoch': 0.51}

  5%|▌         | 2110/41250 [5:05:37<94:12:20,  8.66s/it][2025-04-25 13:03:21,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:03:21,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.72 | bwd_microstep: 5759.52 | bwd_inner_microstep: 5640.42 | bwd_allreduce_microstep: 119.05 | step_microstep: 18.32
[2025-04-25 13:03:21,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.73 | bwd: 5759.53 | bwd_inner: 5640.42 | bwd_allreduce: 119.07 | step: 18.32

  5%|▌         | 2111/41250 [5:05:46<94:14:38,  8.67s/it]
                                                         
{'loss': 0.1996, 'grad_norm': 1.4969826936721802, 'learning_rate': 3.9953034638880755e-05, 'epoch': 0.51}

  5%|▌         | 2111/41250 [5:05:46<94:14:38,  8.67s/it][2025-04-25 13:03:29,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 1.08
[2025-04-25 13:03:29,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5739.03 | bwd_inner_microstep: 5703.91 | bwd_allreduce_microstep: 35.08 | step_microstep: 18.29
[2025-04-25 13:03:29,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5739.05 | bwd_inner: 5703.91 | bwd_allreduce: 35.09 | step: 18.29

  5%|▌         | 2112/41250 [5:05:55<94:15:19,  8.67s/it]
                                                         
{'loss': 0.1722, 'grad_norm': 2.285198926925659, 'learning_rate': 3.995292702417881e-05, 'epoch': 0.51}

  5%|▌         | 2112/41250 [5:05:55<94:15:19,  8.67s/it][2025-04-25 13:03:38,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:03:38,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.15 | bwd_microstep: 5691.16 | bwd_inner_microstep: 5678.61 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.37
[2025-04-25 13:03:38,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5691.17 | bwd_inner: 5678.61 | bwd_allreduce: 12.52 | step: 18.38

  5%|▌         | 2113/41250 [5:06:03<94:04:38,  8.65s/it]
                                                         
{'loss': 0.3123, 'grad_norm': 2.0368616580963135, 'learning_rate': 3.9952819286470995e-05, 'epoch': 0.51}

  5%|▌         | 2113/41250 [5:06:03<94:04:38,  8.65s/it][2025-04-25 13:03:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 13:03:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.70 | bwd_microstep: 5703.53 | bwd_inner_microstep: 5690.54 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.69
[2025-04-25 13:03:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.70 | bwd: 5703.54 | bwd_inner: 5690.54 | bwd_allreduce: 12.96 | step: 18.69

  5%|▌         | 2114/41250 [5:06:12<94:02:27,  8.65s/it]
                                                         
{'loss': 0.1959, 'grad_norm': 1.5545647144317627, 'learning_rate': 3.995271142575797e-05, 'epoch': 0.51}

  5%|▌         | 2114/41250 [5:06:12<94:02:27,  8.65s/it][2025-04-25 13:03:55,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:03:55,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5756.43 | bwd_inner_microstep: 5654.67 | bwd_allreduce_microstep: 101.71 | step_microstep: 18.51
[2025-04-25 13:03:55,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5756.44 | bwd_inner: 5654.67 | bwd_allreduce: 101.73 | step: 18.51

  5%|▌         | 2115/41250 [5:06:20<94:07:42,  8.66s/it]
                                                         
{'loss': 0.0706, 'grad_norm': 0.7378288507461548, 'learning_rate': 3.995260344204041e-05, 'epoch': 0.51}

  5%|▌         | 2115/41250 [5:06:20<94:07:42,  8.66s/it][2025-04-25 13:04:04,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-25 13:04:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5693.83 | bwd_inner_microstep: 5657.75 | bwd_allreduce_microstep: 36.03 | step_microstep: 19.32
[2025-04-25 13:04:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5693.84 | bwd_inner: 5657.75 | bwd_allreduce: 36.05 | step: 19.32

  5%|▌         | 2116/41250 [5:06:29<93:56:52,  8.64s/it]
                                                         
{'loss': 0.0866, 'grad_norm': 0.9764262437820435, 'learning_rate': 3.9952495335318965e-05, 'epoch': 0.51}

  5%|▌         | 2116/41250 [5:06:29<93:56:52,  8.64s/it][2025-04-25 13:04:12,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 13:04:12,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.62 | bwd_microstep: 5689.07 | bwd_inner_microstep: 5676.60 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.48
[2025-04-25 13:04:12,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.62 | bwd: 5689.08 | bwd_inner: 5676.60 | bwd_allreduce: 12.44 | step: 18.49

  5%|▌         | 2117/41250 [5:06:38<93:55:35,  8.64s/it]
                                                         
{'loss': 0.2786, 'grad_norm': 2.7809813022613525, 'learning_rate': 3.995238710559432e-05, 'epoch': 0.51}

  5%|▌         | 2117/41250 [5:06:38<93:55:35,  8.64s/it][2025-04-25 13:04:21,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 13:04:21,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.82 | bwd_microstep: 5698.78 | bwd_inner_microstep: 5686.27 | bwd_allreduce_microstep: 12.47 | step_microstep: 17.92
[2025-04-25 13:04:21,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5698.79 | bwd_inner: 5686.27 | bwd_allreduce: 12.48 | step: 17.93

  5%|▌         | 2118/41250 [5:06:46<93:53:02,  8.64s/it]
                                                         
{'loss': 0.1131, 'grad_norm': 1.348546028137207, 'learning_rate': 3.995227875286713e-05, 'epoch': 0.51}

  5%|▌         | 2118/41250 [5:06:46<93:53:02,  8.64s/it][2025-04-25 13:04:30,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:04:30,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.18 | bwd_microstep: 5700.66 | bwd_inner_microstep: 5687.97 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.36
[2025-04-25 13:04:30,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.18 | bwd: 5700.67 | bwd_inner: 5687.97 | bwd_allreduce: 12.67 | step: 18.36

  5%|▌         | 2119/41250 [5:06:55<93:52:35,  8.64s/it]
                                                         
{'loss': 0.2527, 'grad_norm': 1.910918951034546, 'learning_rate': 3.9952170277138065e-05, 'epoch': 0.51}

  5%|▌         | 2119/41250 [5:06:55<93:52:35,  8.64s/it][2025-04-25 13:04:38,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:04:38,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.63 | bwd_microstep: 5746.45 | bwd_inner_microstep: 5652.34 | bwd_allreduce_microstep: 94.07 | step_microstep: 18.35
[2025-04-25 13:04:38,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.62 | bwd: 5746.46 | bwd_inner: 5652.34 | bwd_allreduce: 94.09 | step: 18.36

  5%|▌         | 2120/41250 [5:07:04<93:55:44,  8.64s/it]
                                                         
{'loss': 0.2228, 'grad_norm': 1.2341843843460083, 'learning_rate': 3.9952061678407796e-05, 'epoch': 0.51}

  5%|▌         | 2120/41250 [5:07:04<93:55:44,  8.64s/it][2025-04-25 13:04:47,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:04:47,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.51 | bwd_microstep: 5786.98 | bwd_inner_microstep: 5774.26 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.46
[2025-04-25 13:04:47,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.51 | bwd: 5786.99 | bwd_inner: 5774.26 | bwd_allreduce: 12.69 | step: 18.47

  5%|▌         | 2121/41250 [5:07:12<94:17:46,  8.68s/it]
                                                         
{'loss': 0.1103, 'grad_norm': 1.307373046875, 'learning_rate': 3.9951952956676987e-05, 'epoch': 0.51}

  5%|▌         | 2121/41250 [5:07:12<94:17:46,  8.68s/it][2025-04-25 13:04:56,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.62 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 13:04:56,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.01 | bwd_microstep: 5884.57 | bwd_inner_microstep: 5656.56 | bwd_allreduce_microstep: 227.97 | step_microstep: 18.81
[2025-04-25 13:04:56,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.01 | bwd: 5884.59 | bwd_inner: 5656.56 | bwd_allreduce: 227.98 | step: 18.81

  5%|▌         | 2122/41250 [5:07:21<94:40:58,  8.71s/it]
                                                         
{'loss': 0.0905, 'grad_norm': 1.2063394784927368, 'learning_rate': 3.995184411194631e-05, 'epoch': 0.51}

  5%|▌         | 2122/41250 [5:07:21<94:40:58,  8.71s/it][2025-04-25 13:05:05,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:05:05,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.05 | bwd_microstep: 5763.44 | bwd_inner_microstep: 5701.29 | bwd_allreduce_microstep: 62.11 | step_microstep: 18.25
[2025-04-25 13:05:05,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.05 | bwd: 5763.46 | bwd_inner: 5701.29 | bwd_allreduce: 62.13 | step: 18.26

  5%|▌         | 2123/41250 [5:07:30<94:37:31,  8.71s/it]
                                                         
{'loss': 0.1275, 'grad_norm': 1.230657935142517, 'learning_rate': 3.9951735144216454e-05, 'epoch': 0.51}

  5%|▌         | 2123/41250 [5:07:30<94:37:31,  8.71s/it][2025-04-25 13:05:13,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 13:05:13,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5741.69 | bwd_inner_microstep: 5682.25 | bwd_allreduce_microstep: 59.40 | step_microstep: 18.23
[2025-04-25 13:05:13,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5741.70 | bwd_inner: 5682.25 | bwd_allreduce: 59.42 | step: 18.23

  5%|▌         | 2124/41250 [5:07:39<94:31:47,  8.70s/it]
                                                         
{'loss': 0.223, 'grad_norm': 2.2285828590393066, 'learning_rate': 3.995162605348807e-05, 'epoch': 0.51}

  5%|▌         | 2124/41250 [5:07:39<94:31:47,  8.70s/it][2025-04-25 13:05:22,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 13:05:22,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.96 | bwd_microstep: 5718.95 | bwd_inner_microstep: 5692.23 | bwd_allreduce_microstep: 26.68 | step_microstep: 18.77
[2025-04-25 13:05:22,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.96 | bwd: 5718.97 | bwd_inner: 5692.23 | bwd_allreduce: 26.69 | step: 18.77

  5%|▌         | 2125/41250 [5:07:47<94:23:08,  8.68s/it]
                                                         
{'loss': 0.1488, 'grad_norm': 1.431380271911621, 'learning_rate': 3.995151683976183e-05, 'epoch': 0.52}

  5%|▌         | 2125/41250 [5:07:47<94:23:08,  8.68s/it][2025-04-25 13:05:31,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:05:31,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5759.70 | bwd_inner_microstep: 5647.78 | bwd_allreduce_microstep: 111.88 | step_microstep: 18.19
[2025-04-25 13:05:31,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5759.72 | bwd_inner: 5647.78 | bwd_allreduce: 111.90 | step: 18.19

  5%|▌         | 2126/41250 [5:07:56<94:20:16,  8.68s/it]
                                                         
{'loss': 0.1818, 'grad_norm': 2.538635492324829, 'learning_rate': 3.995140750303841e-05, 'epoch': 0.52}

  5%|▌         | 2126/41250 [5:07:56<94:20:16,  8.68s/it][2025-04-25 13:05:39,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 13:05:39,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.38 | bwd_microstep: 5716.22 | bwd_inner_microstep: 5696.58 | bwd_allreduce_microstep: 19.59 | step_microstep: 18.79
[2025-04-25 13:05:39,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.38 | bwd: 5716.23 | bwd_inner: 5696.58 | bwd_allreduce: 19.61 | step: 18.79

  5%|▌         | 2127/41250 [5:08:04<94:15:00,  8.67s/it]
                                                         
{'loss': 0.119, 'grad_norm': 1.0827077627182007, 'learning_rate': 3.995129804331849e-05, 'epoch': 0.52}

  5%|▌         | 2127/41250 [5:08:05<94:15:00,  8.67s/it][2025-04-25 13:05:48,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 13:05:48,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 6012.19 | bwd_inner_microstep: 5641.09 | bwd_allreduce_microstep: 371.06 | step_microstep: 18.70
[2025-04-25 13:05:48,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 6012.21 | bwd_inner: 5641.09 | bwd_allreduce: 371.08 | step: 18.70

  5%|▌         | 2128/41250 [5:08:13<95:02:51,  8.75s/it]
                                                         
{'loss': 0.2255, 'grad_norm': 1.591853380203247, 'learning_rate': 3.9951188460602746e-05, 'epoch': 0.52}

  5%|▌         | 2128/41250 [5:08:13<95:02:51,  8.75s/it][2025-04-25 13:05:57,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:05:57,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5750.90 | bwd_inner_microstep: 5635.12 | bwd_allreduce_microstep: 115.74 | step_microstep: 18.66
[2025-04-25 13:05:57,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5750.91 | bwd_inner: 5635.12 | bwd_allreduce: 115.76 | step: 18.66

  5%|▌         | 2129/41250 [5:08:22<94:45:27,  8.72s/it]
                                                         
{'loss': 0.1251, 'grad_norm': 1.3221302032470703, 'learning_rate': 3.995107875489184e-05, 'epoch': 0.52}

  5%|▌         | 2129/41250 [5:08:22<94:45:27,  8.72s/it][2025-04-25 13:06:05,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:06:05,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5732.99 | bwd_inner_microstep: 5636.13 | bwd_allreduce_microstep: 96.82 | step_microstep: 18.64
[2025-04-25 13:06:05,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5733.01 | bwd_inner: 5636.13 | bwd_allreduce: 96.83 | step: 18.65

  5%|▌         | 2130/41250 [5:08:31<94:31:06,  8.70s/it]
                                                         
{'loss': 0.368, 'grad_norm': 2.838590621948242, 'learning_rate': 3.9950968926186465e-05, 'epoch': 0.52}

  5%|▌         | 2130/41250 [5:08:31<94:31:06,  8.70s/it][2025-04-25 13:06:14,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:06:14,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.80 | bwd_microstep: 5701.93 | bwd_inner_microstep: 5689.27 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.29
[2025-04-25 13:06:14,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.80 | bwd: 5701.94 | bwd_inner: 5689.27 | bwd_allreduce: 12.64 | step: 18.29

  5%|▌         | 2131/41250 [5:08:39<94:18:01,  8.68s/it]
                                                         
{'loss': 0.2308, 'grad_norm': 1.9845305681228638, 'learning_rate': 3.9950858974487285e-05, 'epoch': 0.52}

  5%|▌         | 2131/41250 [5:08:39<94:18:01,  8.68s/it][2025-04-25 13:06:23,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:06:23,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5691.98 | bwd_inner_microstep: 5678.82 | bwd_allreduce_microstep: 13.11 | step_microstep: 18.64
[2025-04-25 13:06:23,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5692.00 | bwd_inner: 5678.82 | bwd_allreduce: 13.13 | step: 18.64

  5%|▌         | 2132/41250 [5:08:48<94:07:46,  8.66s/it]
                                                         
{'loss': 0.1856, 'grad_norm': 1.8101916313171387, 'learning_rate': 3.995074889979499e-05, 'epoch': 0.52}

  5%|▌         | 2132/41250 [5:08:48<94:07:46,  8.66s/it][2025-04-25 13:06:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:06:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.57 | bwd_microstep: 5696.15 | bwd_inner_microstep: 5683.48 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.11
[2025-04-25 13:06:31,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.57 | bwd: 5696.16 | bwd_inner: 5683.48 | bwd_allreduce: 12.65 | step: 18.12

  5%|▌         | 2133/41250 [5:08:57<94:02:11,  8.65s/it]
                                                         
{'loss': 0.1827, 'grad_norm': 1.9457123279571533, 'learning_rate': 3.995063870211025e-05, 'epoch': 0.52}

  5%|▌         | 2133/41250 [5:08:57<94:02:11,  8.65s/it][2025-04-25 13:06:40,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.92
[2025-04-25 13:06:40,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.86 | bwd_microstep: 5677.76 | bwd_inner_microstep: 5652.71 | bwd_allreduce_microstep: 25.00 | step_microstep: 18.10
[2025-04-25 13:06:40,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.87 | bwd: 5677.77 | bwd_inner: 5652.71 | bwd_allreduce: 25.02 | step: 18.10

  5%|▌         | 2134/41250 [5:09:05<93:50:05,  8.64s/it]
                                                         
{'loss': 0.1189, 'grad_norm': 1.1389583349227905, 'learning_rate': 3.995052838143375e-05, 'epoch': 0.52}

  5%|▌         | 2134/41250 [5:09:05<93:50:05,  8.64s/it][2025-04-25 13:06:49,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:06:49,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.75 | bwd_microstep: 5876.06 | bwd_inner_microstep: 5863.42 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.32
[2025-04-25 13:06:49,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.75 | bwd: 5876.07 | bwd_inner: 5863.42 | bwd_allreduce: 12.60 | step: 18.32

  5%|▌         | 2135/41250 [5:09:14<94:40:32,  8.71s/it]
                                                         
{'loss': 0.3176, 'grad_norm': 4.329315185546875, 'learning_rate': 3.995041793776616e-05, 'epoch': 0.52}

  5%|▌         | 2135/41250 [5:09:14<94:40:32,  8.71s/it][2025-04-25 13:06:57,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 13:06:57,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.70 | bwd_microstep: 5730.98 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 70.00 | step_microstep: 18.69
[2025-04-25 13:06:57,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.70 | bwd: 5730.99 | bwd_inner: 5660.93 | bwd_allreduce: 70.02 | step: 18.69

  5%|▌         | 2136/41250 [5:09:23<94:26:52,  8.69s/it]
                                                         
{'loss': 0.2027, 'grad_norm': 2.2590649127960205, 'learning_rate': 3.995030737110817e-05, 'epoch': 0.52}

  5%|▌         | 2136/41250 [5:09:23<94:26:52,  8.69s/it][2025-04-25 13:07:06,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:07:06,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.01 | bwd_microstep: 5716.60 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.41
[2025-04-25 13:07:06,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.01 | bwd: 5716.62 | bwd_inner: 5704.05 | bwd_allreduce: 12.53 | step: 18.41

  5%|▌         | 2137/41250 [5:09:31<94:21:17,  8.68s/it]
                                                         
{'loss': 0.3518, 'grad_norm': 1.926207184791565, 'learning_rate': 3.9950196681460454e-05, 'epoch': 0.52}

  5%|▌         | 2137/41250 [5:09:31<94:21:17,  8.68s/it][2025-04-25 13:07:15,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:07:15,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.34 | bwd_microstep: 5729.53 | bwd_inner_microstep: 5691.14 | bwd_allreduce_microstep: 38.34 | step_microstep: 18.62
[2025-04-25 13:07:15,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5729.55 | bwd_inner: 5691.14 | bwd_allreduce: 38.36 | step: 18.62

  5%|▌         | 2138/41250 [5:09:40<94:16:36,  8.68s/it]
                                                         
{'loss': 0.1244, 'grad_norm': 1.5939298868179321, 'learning_rate': 3.995008586882371e-05, 'epoch': 0.52}

  5%|▌         | 2138/41250 [5:09:40<94:16:36,  8.68s/it][2025-04-25 13:07:23,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:07:23,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5710.04 | bwd_inner_microstep: 5697.34 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.32
[2025-04-25 13:07:23,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5710.05 | bwd_inner: 5697.34 | bwd_allreduce: 12.67 | step: 18.33

  5%|▌         | 2139/41250 [5:09:49<94:10:07,  8.67s/it]
                                                         
{'loss': 0.3823, 'grad_norm': 2.674893856048584, 'learning_rate': 3.9949974933198605e-05, 'epoch': 0.52}

  5%|▌         | 2139/41250 [5:09:49<94:10:07,  8.67s/it][2025-04-25 13:07:32,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:07:32,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.80 | bwd_microstep: 5789.98 | bwd_inner_microstep: 5777.25 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.65
[2025-04-25 13:07:32,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.80 | bwd: 5789.99 | bwd_inner: 5777.25 | bwd_allreduce: 12.70 | step: 18.66

  5%|▌         | 2140/41250 [5:09:57<94:29:49,  8.70s/it]
                                                         
{'loss': 0.1674, 'grad_norm': 1.7818549871444702, 'learning_rate': 3.9949863874585825e-05, 'epoch': 0.52}

  5%|▌         | 2140/41250 [5:09:57<94:29:49,  8.70s/it][2025-04-25 13:07:41,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 13:07:41,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.54 | bwd_microstep: 5702.20 | bwd_inner_microstep: 5689.47 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.72
[2025-04-25 13:07:41,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.54 | bwd: 5702.21 | bwd_inner: 5689.47 | bwd_allreduce: 12.70 | step: 18.72

  5%|▌         | 2141/41250 [5:10:06<94:17:43,  8.68s/it]
                                                         
{'loss': 0.1657, 'grad_norm': 1.8490608930587769, 'learning_rate': 3.9949752692986064e-05, 'epoch': 0.52}

  5%|▌         | 2141/41250 [5:10:06<94:17:43,  8.68s/it][2025-04-25 13:07:49,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:07:49,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5727.64 | bwd_inner_microstep: 5691.61 | bwd_allreduce_microstep: 35.98 | step_microstep: 19.09
[2025-04-25 13:07:49,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.77 | bwd: 5727.65 | bwd_inner: 5691.61 | bwd_allreduce: 36.00 | step: 19.09

  5%|▌         | 2142/41250 [5:10:15<94:14:19,  8.67s/it]
                                                         
{'loss': 0.3972, 'grad_norm': 3.0746679306030273, 'learning_rate': 3.9949641388400005e-05, 'epoch': 0.52}

  5%|▌         | 2142/41250 [5:10:15<94:14:19,  8.67s/it][2025-04-25 13:07:58,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.93 | optimizer_step: 0.90
[2025-04-25 13:07:58,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.30 | bwd_microstep: 5690.62 | bwd_inner_microstep: 5660.45 | bwd_allreduce_microstep: 30.13 | step_microstep: 18.38
[2025-04-25 13:07:58,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.30 | bwd: 5690.64 | bwd_inner: 5660.45 | bwd_allreduce: 30.15 | step: 18.38

  5%|▌         | 2143/41250 [5:10:23<93:59:50,  8.65s/it]
                                                         
{'loss': 0.3094, 'grad_norm': 3.308708906173706, 'learning_rate': 3.994952996082832e-05, 'epoch': 0.52}

  5%|▌         | 2143/41250 [5:10:23<93:59:50,  8.65s/it][2025-04-25 13:08:07,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:08:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.05 | bwd_microstep: 5793.88 | bwd_inner_microstep: 5781.05 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.73
[2025-04-25 13:08:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.05 | bwd: 5793.89 | bwd_inner: 5781.05 | bwd_allreduce: 12.80 | step: 18.73

  5%|▌         | 2144/41250 [5:10:32<94:23:30,  8.69s/it]
                                                         
{'loss': 0.1735, 'grad_norm': 2.106186866760254, 'learning_rate': 3.9949418410271714e-05, 'epoch': 0.52}

  5%|▌         | 2144/41250 [5:10:32<94:23:30,  8.69s/it][2025-04-25 13:08:15,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 13:08:15,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.24 | bwd_microstep: 5701.68 | bwd_inner_microstep: 5671.45 | bwd_allreduce_microstep: 30.18 | step_microstep: 18.54
[2025-04-25 13:08:15,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.24 | bwd: 5701.70 | bwd_inner: 5671.45 | bwd_allreduce: 30.20 | step: 18.54

  5%|▌         | 2145/41250 [5:10:41<94:09:52,  8.67s/it]
                                                         
{'loss': 0.1078, 'grad_norm': 0.7622596621513367, 'learning_rate': 3.9949306736730864e-05, 'epoch': 0.52}

  5%|▌         | 2145/41250 [5:10:41<94:09:52,  8.67s/it][2025-04-25 13:08:24,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 13:08:24,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.56 | bwd_microstep: 5709.88 | bwd_inner_microstep: 5697.27 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.39
[2025-04-25 13:08:24,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5709.89 | bwd_inner: 5697.27 | bwd_allreduce: 12.57 | step: 18.39

  5%|▌         | 2146/41250 [5:10:49<94:05:00,  8.66s/it]
                                                         
{'loss': 0.2194, 'grad_norm': 2.181159257888794, 'learning_rate': 3.994919494020647e-05, 'epoch': 0.52}

  5%|▌         | 2146/41250 [5:10:49<94:05:00,  8.66s/it][2025-04-25 13:08:33,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:08:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.15 | bwd_microstep: 5796.46 | bwd_inner_microstep: 5783.46 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.67
[2025-04-25 13:08:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.15 | bwd: 5796.47 | bwd_inner: 5783.46 | bwd_allreduce: 12.97 | step: 18.67

  5%|▌         | 2147/41250 [5:10:58<94:25:12,  8.69s/it]
                                                         
{'loss': 0.2282, 'grad_norm': 4.086971282958984, 'learning_rate': 3.99490830206992e-05, 'epoch': 0.52}

  5%|▌         | 2147/41250 [5:10:58<94:25:12,  8.69s/it][2025-04-25 13:08:41,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:08:41,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.47 | bwd_microstep: 5704.28 | bwd_inner_microstep: 5659.76 | bwd_allreduce_microstep: 44.48 | step_microstep: 18.88
[2025-04-25 13:08:41,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.47 | bwd: 5704.29 | bwd_inner: 5659.76 | bwd_allreduce: 44.50 | step: 18.88

  5%|▌         | 2148/41250 [5:11:07<94:12:13,  8.67s/it]
                                                         
{'loss': 0.2208, 'grad_norm': 1.2155625820159912, 'learning_rate': 3.9948970978209766e-05, 'epoch': 0.52}

  5%|▌         | 2148/41250 [5:11:07<94:12:13,  8.67s/it][2025-04-25 13:08:50,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 13:08:50,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5760.05 | bwd_inner_microstep: 5711.38 | bwd_allreduce_microstep: 48.63 | step_microstep: 18.59
[2025-04-25 13:08:50,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5760.06 | bwd_inner: 5711.37 | bwd_allreduce: 48.64 | step: 18.60

  5%|▌         | 2149/41250 [5:11:16<94:16:25,  8.68s/it]
                                                         
{'loss': 0.1078, 'grad_norm': 1.0430809259414673, 'learning_rate': 3.9948858812738845e-05, 'epoch': 0.52}

  5%|▌         | 2149/41250 [5:11:16<94:16:25,  8.68s/it][2025-04-25 13:08:59,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:08:59,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5773.08 | bwd_inner_microstep: 5657.41 | bwd_allreduce_microstep: 115.62 | step_microstep: 19.04
[2025-04-25 13:08:59,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5773.10 | bwd_inner: 5657.41 | bwd_allreduce: 115.65 | step: 19.05

  5%|▌         | 2150/41250 [5:11:24<94:19:02,  8.68s/it]
                                                         
{'loss': 0.2607, 'grad_norm': 1.8444979190826416, 'learning_rate': 3.9948746524287135e-05, 'epoch': 0.52}

  5%|▌         | 2150/41250 [5:11:24<94:19:02,  8.68s/it][2025-04-25 13:09:08,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.06 | optimizer_step: 1.25
[2025-04-25 13:09:08,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5722.60 | bwd_inner_microstep: 5654.81 | bwd_allreduce_microstep: 67.74 | step_microstep: 19.76
[2025-04-25 13:09:08,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5722.61 | bwd_inner: 5654.81 | bwd_allreduce: 67.76 | step: 19.76

  5%|▌         | 2151/41250 [5:11:33<94:09:52,  8.67s/it]
                                                         
{'loss': 0.1377, 'grad_norm': 2.490962266921997, 'learning_rate': 3.994863411285533e-05, 'epoch': 0.52}

  5%|▌         | 2151/41250 [5:11:33<94:09:52,  8.67s/it][2025-04-25 13:09:16,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:09:16,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.09 | bwd_microstep: 5749.13 | bwd_inner_microstep: 5708.19 | bwd_allreduce_microstep: 40.89 | step_microstep: 18.60
[2025-04-25 13:09:16,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.09 | bwd: 5749.14 | bwd_inner: 5708.19 | bwd_allreduce: 40.91 | step: 18.60

  5%|▌         | 2152/41250 [5:11:42<94:12:03,  8.67s/it]
                                                         
{'loss': 0.1292, 'grad_norm': 2.1441402435302734, 'learning_rate': 3.994852157844411e-05, 'epoch': 0.52}

  5%|▌         | 2152/41250 [5:11:42<94:12:03,  8.67s/it][2025-04-25 13:09:25,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:09:25,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.98 | bwd_microstep: 5748.12 | bwd_inner_microstep: 5715.67 | bwd_allreduce_microstep: 32.41 | step_microstep: 18.75
[2025-04-25 13:09:25,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.98 | bwd: 5748.13 | bwd_inner: 5715.67 | bwd_allreduce: 32.42 | step: 18.76

  5%|▌         | 2153/41250 [5:11:50<94:14:02,  8.68s/it]
                                                         
{'loss': 0.2726, 'grad_norm': 1.1879558563232422, 'learning_rate': 3.994840892105418e-05, 'epoch': 0.52}

  5%|▌         | 2153/41250 [5:11:50<94:14:02,  8.68s/it][2025-04-25 13:09:34,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:09:34,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.28 | bwd_microstep: 5739.81 | bwd_inner_microstep: 5726.94 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.91
[2025-04-25 13:09:34,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.28 | bwd: 5739.82 | bwd_inner: 5726.94 | bwd_allreduce: 12.84 | step: 18.92

  5%|▌         | 2154/41250 [5:11:59<94:18:18,  8.68s/it]
                                                         
{'loss': 0.2753, 'grad_norm': 2.6572093963623047, 'learning_rate': 3.994829614068624e-05, 'epoch': 0.52}

  5%|▌         | 2154/41250 [5:11:59<94:18:18,  8.68s/it][2025-04-25 13:09:42,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 13:09:42,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.67 | bwd_microstep: 5729.71 | bwd_inner_microstep: 5657.30 | bwd_allreduce_microstep: 72.36 | step_microstep: 19.43
[2025-04-25 13:09:42,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.67 | bwd: 5729.73 | bwd_inner: 5657.30 | bwd_allreduce: 72.39 | step: 19.43

  5%|▌         | 2155/41250 [5:12:08<94:11:19,  8.67s/it]
                                                         
{'loss': 0.173, 'grad_norm': 1.0395476818084717, 'learning_rate': 3.9948183237340975e-05, 'epoch': 0.52}

  5%|▌         | 2155/41250 [5:12:08<94:11:19,  8.67s/it][2025-04-25 13:09:51,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 13:09:51,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.55 | bwd_microstep: 5719.68 | bwd_inner_microstep: 5663.78 | bwd_allreduce_microstep: 55.85 | step_microstep: 19.02
[2025-04-25 13:09:51,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.55 | bwd: 5719.69 | bwd_inner: 5663.78 | bwd_allreduce: 55.87 | step: 19.02

  5%|▌         | 2156/41250 [5:12:16<94:05:17,  8.66s/it]
                                                         
{'loss': 0.1126, 'grad_norm': 1.0025818347930908, 'learning_rate': 3.9948070211019085e-05, 'epoch': 0.52}

  5%|▌         | 2156/41250 [5:12:16<94:05:17,  8.66s/it][2025-04-25 13:10:00,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 13:10:00,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5698.96 | bwd_inner_microstep: 5671.78 | bwd_allreduce_microstep: 27.13 | step_microstep: 18.80
[2025-04-25 13:10:00,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5698.98 | bwd_inner: 5671.78 | bwd_allreduce: 27.15 | step: 18.80

  5%|▌         | 2157/41250 [5:12:25<93:58:35,  8.65s/it]
                                                         
{'loss': 0.1995, 'grad_norm': 1.761860966682434, 'learning_rate': 3.9947957061721265e-05, 'epoch': 0.52}

  5%|▌         | 2157/41250 [5:12:25<93:58:35,  8.65s/it][2025-04-25 13:10:08,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:10:08,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.52 | bwd_microstep: 5775.97 | bwd_inner_microstep: 5651.50 | bwd_allreduce_microstep: 124.42 | step_microstep: 18.95
[2025-04-25 13:10:08,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.52 | bwd: 5775.98 | bwd_inner: 5651.50 | bwd_allreduce: 124.44 | step: 18.95

  5%|▌         | 2158/41250 [5:12:34<94:06:43,  8.67s/it]
                                                         
{'loss': 0.1887, 'grad_norm': 1.8954272270202637, 'learning_rate': 3.994784378944821e-05, 'epoch': 0.52}

  5%|▌         | 2158/41250 [5:12:34<94:06:43,  8.67s/it][2025-04-25 13:10:17,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:10:17,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.71 | bwd_microstep: 5748.78 | bwd_inner_microstep: 5691.51 | bwd_allreduce_microstep: 57.23 | step_microstep: 18.55
[2025-04-25 13:10:17,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.71 | bwd: 5748.80 | bwd_inner: 5691.51 | bwd_allreduce: 57.25 | step: 18.56

  5%|▌         | 2159/41250 [5:12:42<94:09:54,  8.67s/it]
                                                         
{'loss': 0.17, 'grad_norm': 1.6077464818954468, 'learning_rate': 3.994773039420062e-05, 'epoch': 0.52}

  5%|▌         | 2159/41250 [5:12:42<94:09:54,  8.67s/it][2025-04-25 13:10:26,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:10:26,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.37 | bwd_microstep: 5792.58 | bwd_inner_microstep: 5664.86 | bwd_allreduce_microstep: 127.67 | step_microstep: 18.65
[2025-04-25 13:10:26,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.37 | bwd: 5792.59 | bwd_inner: 5664.86 | bwd_allreduce: 127.69 | step: 18.66

  5%|▌         | 2160/41250 [5:12:51<94:16:46,  8.68s/it]
                                                         
{'loss': 0.1323, 'grad_norm': 1.0951834917068481, 'learning_rate': 3.99476168759792e-05, 'epoch': 0.52}

  5%|▌         | 2160/41250 [5:12:51<94:16:46,  8.68s/it][2025-04-25 13:10:34,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:10:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.32 | bwd_microstep: 5875.10 | bwd_inner_microstep: 5709.32 | bwd_allreduce_microstep: 165.73 | step_microstep: 18.47
[2025-04-25 13:10:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.32 | bwd: 5875.11 | bwd_inner: 5709.32 | bwd_allreduce: 165.75 | step: 18.47

  5%|▌         | 2161/41250 [5:13:00<94:44:16,  8.73s/it]
                                                         
{'loss': 0.2698, 'grad_norm': 2.041710376739502, 'learning_rate': 3.994750323478464e-05, 'epoch': 0.52}

  5%|▌         | 2161/41250 [5:13:00<94:44:16,  8.73s/it][2025-04-25 13:10:43,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 13:10:43,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.05 | bwd_microstep: 5730.91 | bwd_inner_microstep: 5702.91 | bwd_allreduce_microstep: 27.96 | step_microstep: 19.08
[2025-04-25 13:10:43,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.05 | bwd: 5730.92 | bwd_inner: 5702.91 | bwd_allreduce: 27.98 | step: 19.08

  5%|▌         | 2162/41250 [5:13:08<94:34:47,  8.71s/it]
                                                         
{'loss': 0.1478, 'grad_norm': 1.402092456817627, 'learning_rate': 3.994738947061766e-05, 'epoch': 0.52}

  5%|▌         | 2162/41250 [5:13:08<94:34:47,  8.71s/it][2025-04-25 13:10:52,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 13:10:52,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.71 | bwd_microstep: 5764.61 | bwd_inner_microstep: 5693.13 | bwd_allreduce_microstep: 71.44 | step_microstep: 18.49
[2025-04-25 13:10:52,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.71 | bwd: 5764.63 | bwd_inner: 5693.13 | bwd_allreduce: 71.46 | step: 18.49

  5%|▌         | 2163/41250 [5:13:17<94:32:22,  8.71s/it]
                                                         
{'loss': 0.1391, 'grad_norm': 1.6843793392181396, 'learning_rate': 3.994727558347894e-05, 'epoch': 0.52}

  5%|▌         | 2163/41250 [5:13:17<94:32:22,  8.71s/it][2025-04-25 13:11:00,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 13:11:00,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.55 | bwd_microstep: 5771.90 | bwd_inner_microstep: 5647.56 | bwd_allreduce_microstep: 124.30 | step_microstep: 19.05
[2025-04-25 13:11:00,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5771.91 | bwd_inner: 5647.56 | bwd_allreduce: 124.32 | step: 19.05

  5%|▌         | 2164/41250 [5:13:26<94:27:43,  8.70s/it]
                                                         
{'loss': 0.2397, 'grad_norm': 1.6255593299865723, 'learning_rate': 3.994716157336918e-05, 'epoch': 0.52}

  5%|▌         | 2164/41250 [5:13:26<94:27:43,  8.70s/it][2025-04-25 13:11:09,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:11:09,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.90 | bwd_microstep: 5708.40 | bwd_inner_microstep: 5652.49 | bwd_allreduce_microstep: 55.87 | step_microstep: 18.71
[2025-04-25 13:11:09,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.90 | bwd: 5708.41 | bwd_inner: 5652.49 | bwd_allreduce: 55.88 | step: 18.71

  5%|▌         | 2165/41250 [5:13:34<94:10:30,  8.67s/it]
                                                         
{'loss': 0.475, 'grad_norm': 3.0983450412750244, 'learning_rate': 3.9947047440289106e-05, 'epoch': 0.52}

  5%|▌         | 2165/41250 [5:13:34<94:10:30,  8.67s/it][2025-04-25 13:11:18,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:11:18,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5697.39 | bwd_inner_microstep: 5655.01 | bwd_allreduce_microstep: 42.34 | step_microstep: 18.68
[2025-04-25 13:11:18,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5697.40 | bwd_inner: 5655.01 | bwd_allreduce: 42.35 | step: 18.68

  5%|▌         | 2166/41250 [5:13:43<93:58:47,  8.66s/it]
                                                         
{'loss': 0.1091, 'grad_norm': 1.2424017190933228, 'learning_rate': 3.9946933184239396e-05, 'epoch': 0.53}

  5%|▌         | 2166/41250 [5:13:43<93:58:47,  8.66s/it][2025-04-25 13:11:26,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:11:26,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.08 | bwd_microstep: 5720.24 | bwd_inner_microstep: 5707.37 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.75
[2025-04-25 13:11:26,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.08 | bwd: 5720.25 | bwd_inner: 5707.37 | bwd_allreduce: 12.84 | step: 18.75

  5%|▌         | 2167/41250 [5:13:52<93:59:46,  8.66s/it]
                                                         
{'loss': 0.1587, 'grad_norm': 1.1321868896484375, 'learning_rate': 3.9946818805220776e-05, 'epoch': 0.53}

  5%|▌         | 2167/41250 [5:13:52<93:59:46,  8.66s/it][2025-04-25 13:11:35,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:11:35,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.12 | bwd_microstep: 5711.10 | bwd_inner_microstep: 5698.20 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.77
[2025-04-25 13:11:35,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.12 | bwd: 5711.11 | bwd_inner: 5698.20 | bwd_allreduce: 12.87 | step: 18.78

  5%|▌         | 2168/41250 [5:14:00<93:59:10,  8.66s/it]
                                                         
{'loss': 0.0456, 'grad_norm': 0.46171918511390686, 'learning_rate': 3.994670430323393e-05, 'epoch': 0.53}

  5%|▌         | 2168/41250 [5:14:00<93:59:10,  8.66s/it][2025-04-25 13:11:44,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 13:11:44,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.44 | bwd_microstep: 5764.30 | bwd_inner_microstep: 5680.52 | bwd_allreduce_microstep: 83.73 | step_microstep: 18.72
[2025-04-25 13:11:44,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.44 | bwd: 5764.32 | bwd_inner: 5680.52 | bwd_allreduce: 83.75 | step: 18.73

  5%|▌         | 2169/41250 [5:14:09<94:06:28,  8.67s/it]
                                                         
{'loss': 0.2215, 'grad_norm': 3.4300711154937744, 'learning_rate': 3.994658967827959e-05, 'epoch': 0.53}

  5%|▌         | 2169/41250 [5:14:09<94:06:28,  8.67s/it][2025-04-25 13:11:52,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:11:52,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.75 | bwd_microstep: 5742.03 | bwd_inner_microstep: 5658.12 | bwd_allreduce_microstep: 83.87 | step_microstep: 18.54
[2025-04-25 13:11:52,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.75 | bwd: 5742.05 | bwd_inner: 5658.12 | bwd_allreduce: 83.89 | step: 18.55

  5%|▌         | 2170/41250 [5:14:18<94:04:31,  8.67s/it]
                                                         
{'loss': 0.2334, 'grad_norm': 1.3502910137176514, 'learning_rate': 3.994647493035844e-05, 'epoch': 0.53}

  5%|▌         | 2170/41250 [5:14:18<94:04:31,  8.67s/it][2025-04-25 13:12:01,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-25 13:12:01,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.59 | bwd_microstep: 5747.48 | bwd_inner_microstep: 5641.64 | bwd_allreduce_microstep: 105.80 | step_microstep: 19.12
[2025-04-25 13:12:01,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.59 | bwd: 5747.50 | bwd_inner: 5641.64 | bwd_allreduce: 105.82 | step: 19.12

  5%|▌         | 2171/41250 [5:14:26<94:04:22,  8.67s/it]
                                                         
{'loss': 0.1968, 'grad_norm': 1.6393142938613892, 'learning_rate': 3.994636005947119e-05, 'epoch': 0.53}

  5%|▌         | 2171/41250 [5:14:26<94:04:22,  8.67s/it][2025-04-25 13:12:10,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:12:10,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.74 | bwd_microstep: 5720.83 | bwd_inner_microstep: 5708.06 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.58
[2025-04-25 13:12:10,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.74 | bwd: 5720.84 | bwd_inner: 5708.06 | bwd_allreduce: 12.74 | step: 18.59

  5%|▌         | 2172/41250 [5:14:35<94:03:24,  8.66s/it]
                                                         
{'loss': 0.2022, 'grad_norm': 1.4276964664459229, 'learning_rate': 3.994624506561857e-05, 'epoch': 0.53}

  5%|▌         | 2172/41250 [5:14:35<94:03:24,  8.66s/it][2025-04-25 13:12:18,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 13:12:18,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5739.59 | bwd_inner_microstep: 5651.56 | bwd_allreduce_microstep: 87.98 | step_microstep: 18.85
[2025-04-25 13:12:18,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5739.60 | bwd_inner: 5651.56 | bwd_allreduce: 88.00 | step: 18.85

  5%|▌         | 2173/41250 [5:14:44<94:00:44,  8.66s/it]
                                                         
{'loss': 0.1284, 'grad_norm': 1.3052520751953125, 'learning_rate': 3.994612994880125e-05, 'epoch': 0.53}

  5%|▌         | 2173/41250 [5:14:44<94:00:44,  8.66s/it][2025-04-25 13:12:27,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:12:27,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.39 | bwd_microstep: 5718.60 | bwd_inner_microstep: 5684.63 | bwd_allreduce_microstep: 33.92 | step_microstep: 18.86
[2025-04-25 13:12:27,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.39 | bwd: 5718.61 | bwd_inner: 5684.63 | bwd_allreduce: 33.94 | step: 18.86

  5%|▌         | 2174/41250 [5:14:52<93:56:50,  8.66s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 1.9416300058364868, 'learning_rate': 3.994601470901998e-05, 'epoch': 0.53}

  5%|▌         | 2174/41250 [5:14:52<93:56:50,  8.66s/it][2025-04-25 13:12:36,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 13:12:36,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5721.65 | bwd_inner_microstep: 5708.75 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.64
[2025-04-25 13:12:36,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5721.67 | bwd_inner: 5708.75 | bwd_allreduce: 12.87 | step: 19.64

  5%|▌         | 2175/41250 [5:15:01<93:58:28,  8.66s/it]
                                                         
{'loss': 0.3007, 'grad_norm': 1.8865346908569336, 'learning_rate': 3.994589934627545e-05, 'epoch': 0.53}

  5%|▌         | 2175/41250 [5:15:01<93:58:28,  8.66s/it][2025-04-25 13:12:44,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.26 | optimizer_step: 1.08
[2025-04-25 13:12:44,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.86 | bwd_microstep: 5686.60 | bwd_inner_microstep: 5655.65 | bwd_allreduce_microstep: 30.89 | step_microstep: 20.21
[2025-04-25 13:12:44,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.86 | bwd: 5686.61 | bwd_inner: 5655.65 | bwd_allreduce: 30.92 | step: 20.21

  5%|▌         | 2176/41250 [5:15:10<93:49:27,  8.64s/it]
                                                         
{'loss': 0.2836, 'grad_norm': 1.558293342590332, 'learning_rate': 3.9945783860568364e-05, 'epoch': 0.53}

  5%|▌         | 2176/41250 [5:15:10<93:49:27,  8.64s/it][2025-04-25 13:12:53,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.63 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 13:12:53,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5751.94 | bwd_inner_microstep: 5652.84 | bwd_allreduce_microstep: 99.04 | step_microstep: 19.38
[2025-04-25 13:12:53,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5751.95 | bwd_inner: 5652.84 | bwd_allreduce: 99.07 | step: 19.38

  5%|▌         | 2177/41250 [5:15:18<93:52:41,  8.65s/it]
                                                         
{'loss': 0.0802, 'grad_norm': 0.7436225414276123, 'learning_rate': 3.994566825189945e-05, 'epoch': 0.53}

  5%|▌         | 2177/41250 [5:15:18<93:52:41,  8.65s/it][2025-04-25 13:13:02,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:13:02,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.76 | bwd_microstep: 5723.56 | bwd_inner_microstep: 5700.46 | bwd_allreduce_microstep: 23.05 | step_microstep: 19.09
[2025-04-25 13:13:02,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.76 | bwd: 5723.57 | bwd_inner: 5700.46 | bwd_allreduce: 23.07 | step: 19.09

  5%|▌         | 2178/41250 [5:15:27<93:54:22,  8.65s/it]
                                                         
{'loss': 0.1665, 'grad_norm': 0.9703426957130432, 'learning_rate': 3.9945552520269416e-05, 'epoch': 0.53}

  5%|▌         | 2178/41250 [5:15:27<93:54:22,  8.65s/it][2025-04-25 13:13:10,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-25 13:13:10,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.61 | bwd_microstep: 5751.45 | bwd_inner_microstep: 5651.72 | bwd_allreduce_microstep: 99.68 | step_microstep: 19.46
[2025-04-25 13:13:10,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.61 | bwd: 5751.46 | bwd_inner: 5651.72 | bwd_allreduce: 99.69 | step: 19.46

  5%|▌         | 2179/41250 [5:15:36<93:56:00,  8.66s/it]
                                                         
{'loss': 0.1485, 'grad_norm': 1.5423526763916016, 'learning_rate': 3.9945436665678976e-05, 'epoch': 0.53}

  5%|▌         | 2179/41250 [5:15:36<93:56:00,  8.66s/it][2025-04-25 13:13:19,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 13:13:19,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.93 | bwd_microstep: 5743.70 | bwd_inner_microstep: 5687.42 | bwd_allreduce_microstep: 56.24 | step_microstep: 19.42
[2025-04-25 13:13:19,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.93 | bwd: 5743.72 | bwd_inner: 5687.42 | bwd_allreduce: 56.26 | step: 19.43

  5%|▌         | 2180/41250 [5:15:44<93:59:17,  8.66s/it]
                                                         
{'loss': 0.2749, 'grad_norm': 1.6489295959472656, 'learning_rate': 3.9945320688128834e-05, 'epoch': 0.53}

  5%|▌         | 2180/41250 [5:15:44<93:59:17,  8.66s/it][2025-04-25 13:13:28,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.25 | optimizer_step: 1.01
[2025-04-25 13:13:28,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.06 | bwd_microstep: 5744.67 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 92.33 | step_microstep: 19.77
[2025-04-25 13:13:28,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.06 | bwd: 5744.68 | bwd_inner: 5652.29 | bwd_allreduce: 92.35 | step: 19.77

  5%|▌         | 2181/41250 [5:15:53<93:59:47,  8.66s/it]
                                                         
{'loss': 0.2454, 'grad_norm': 1.8858675956726074, 'learning_rate': 3.994520458761972e-05, 'epoch': 0.53}

  5%|▌         | 2181/41250 [5:15:53<93:59:47,  8.66s/it][2025-04-25 13:13:36,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 13:13:36,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.25 | bwd_microstep: 5724.83 | bwd_inner_microstep: 5685.12 | bwd_allreduce_microstep: 39.66 | step_microstep: 18.94
[2025-04-25 13:13:36,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.25 | bwd: 5724.84 | bwd_inner: 5685.12 | bwd_allreduce: 39.68 | step: 18.95

  5%|▌         | 2182/41250 [5:16:02<93:57:03,  8.66s/it]
                                                         
{'loss': 0.2218, 'grad_norm': 1.4876950979232788, 'learning_rate': 3.994508836415234e-05, 'epoch': 0.53}

  5%|▌         | 2182/41250 [5:16:02<93:57:03,  8.66s/it][2025-04-25 13:13:45,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:13:45,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.97 | bwd_microstep: 5671.37 | bwd_inner_microstep: 5658.42 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.83
[2025-04-25 13:13:45,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.97 | bwd: 5671.39 | bwd_inner: 5658.42 | bwd_allreduce: 12.92 | step: 18.83

  5%|▌         | 2183/41250 [5:16:10<93:43:32,  8.64s/it]
                                                         
{'loss': 0.1105, 'grad_norm': 1.7895225286483765, 'learning_rate': 3.994497201772741e-05, 'epoch': 0.53}

  5%|▌         | 2183/41250 [5:16:10<93:43:32,  8.64s/it][2025-04-25 13:13:53,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 13:13:53,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.38 | bwd_microstep: 5683.51 | bwd_inner_microstep: 5657.17 | bwd_allreduce_microstep: 26.29 | step_microstep: 19.01
[2025-04-25 13:13:53,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.38 | bwd: 5683.52 | bwd_inner: 5657.17 | bwd_allreduce: 26.31 | step: 19.02

  5%|▌         | 2184/41250 [5:16:19<93:34:46,  8.62s/it]
                                                         
{'loss': 0.0923, 'grad_norm': 0.6906712055206299, 'learning_rate': 3.994485554834565e-05, 'epoch': 0.53}

  5%|▌         | 2184/41250 [5:16:19<93:34:46,  8.62s/it][2025-04-25 13:14:02,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 13:14:02,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.09 | bwd_microstep: 5688.37 | bwd_inner_microstep: 5646.00 | bwd_allreduce_microstep: 42.32 | step_microstep: 18.74
[2025-04-25 13:14:02,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.09 | bwd: 5688.39 | bwd_inner: 5646.00 | bwd_allreduce: 42.34 | step: 18.74

  5%|▌         | 2185/41250 [5:16:27<93:31:12,  8.62s/it]
                                                         
{'loss': 0.3278, 'grad_norm': 2.4911179542541504, 'learning_rate': 3.994473895600778e-05, 'epoch': 0.53}

  5%|▌         | 2185/41250 [5:16:27<93:31:12,  8.62s/it][2025-04-25 13:14:11,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:14:11,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.98 | bwd_microstep: 5710.14 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 54.85 | step_microstep: 18.77
[2025-04-25 13:14:11,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.98 | bwd: 5710.16 | bwd_inner: 5655.24 | bwd_allreduce: 54.87 | step: 18.77

  5%|▌         | 2186/41250 [5:16:36<93:35:09,  8.62s/it]
                                                         
{'loss': 0.121, 'grad_norm': 1.5977004766464233, 'learning_rate': 3.9944622240714516e-05, 'epoch': 0.53}

  5%|▌         | 2186/41250 [5:16:36<93:35:09,  8.62s/it][2025-04-25 13:14:19,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 13:14:19,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5726.43 | bwd_inner_microstep: 5682.55 | bwd_allreduce_microstep: 43.84 | step_microstep: 19.21
[2025-04-25 13:14:19,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5726.44 | bwd_inner: 5682.55 | bwd_allreduce: 43.85 | step: 19.22

  5%|▌         | 2187/41250 [5:16:45<93:42:32,  8.64s/it]
                                                         
{'loss': 0.2654, 'grad_norm': 1.646677851676941, 'learning_rate': 3.994450540246659e-05, 'epoch': 0.53}

  5%|▌         | 2187/41250 [5:16:45<93:42:32,  8.64s/it][2025-04-25 13:14:28,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:14:28,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5678.46 | bwd_inner_microstep: 5651.56 | bwd_allreduce_microstep: 26.86 | step_microstep: 18.53
[2025-04-25 13:14:28,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5678.48 | bwd_inner: 5651.56 | bwd_allreduce: 26.88 | step: 18.53

  5%|▌         | 2188/41250 [5:16:53<93:33:46,  8.62s/it]
                                                         
{'loss': 0.2249, 'grad_norm': 1.586862564086914, 'learning_rate': 3.99443884412647e-05, 'epoch': 0.53}

  5%|▌         | 2188/41250 [5:16:53<93:33:46,  8.62s/it][2025-04-25 13:14:37,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 13:14:37,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.49 | bwd_microstep: 5724.57 | bwd_inner_microstep: 5685.92 | bwd_allreduce_microstep: 38.61 | step_microstep: 18.63
[2025-04-25 13:14:37,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.49 | bwd: 5724.59 | bwd_inner: 5685.92 | bwd_allreduce: 38.63 | step: 18.64

  5%|▌         | 2189/41250 [5:17:02<93:40:50,  8.63s/it]
                                                         
{'loss': 0.1922, 'grad_norm': 1.267816185951233, 'learning_rate': 3.994427135710958e-05, 'epoch': 0.53}

  5%|▌         | 2189/41250 [5:17:02<93:40:50,  8.63s/it][2025-04-25 13:14:45,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:14:45,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.92 | bwd_microstep: 5704.71 | bwd_inner_microstep: 5691.89 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.05
[2025-04-25 13:14:45,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.92 | bwd: 5704.72 | bwd_inner: 5691.89 | bwd_allreduce: 12.79 | step: 18.05

  5%|▌         | 2190/41250 [5:17:11<93:42:13,  8.64s/it]
                                                         
{'loss': 0.2503, 'grad_norm': 2.0509541034698486, 'learning_rate': 3.9944154150001956e-05, 'epoch': 0.53}

  5%|▌         | 2190/41250 [5:17:11<93:42:13,  8.64s/it][2025-04-25 13:14:54,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:14:54,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.87 | bwd_microstep: 5887.52 | bwd_inner_microstep: 5875.01 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.75
[2025-04-25 13:14:54,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.87 | bwd: 5887.53 | bwd_inner: 5875.01 | bwd_allreduce: 12.48 | step: 18.75

  5%|▌         | 2191/41250 [5:17:19<94:34:10,  8.72s/it]
                                                         
{'loss': 0.1592, 'grad_norm': 1.236387848854065, 'learning_rate': 3.994403681994254e-05, 'epoch': 0.53}

  5%|▌         | 2191/41250 [5:17:19<94:34:10,  8.72s/it][2025-04-25 13:15:03,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 13:15:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.35 | bwd_microstep: 5786.74 | bwd_inner_microstep: 5773.62 | bwd_allreduce_microstep: 13.07 | step_microstep: 19.52
[2025-04-25 13:15:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.36 | bwd: 5786.76 | bwd_inner: 5773.62 | bwd_allreduce: 13.09 | step: 19.52

  5%|▌         | 2192/41250 [5:17:28<94:42:06,  8.73s/it]
                                                         
{'loss': 0.1249, 'grad_norm': 1.271354079246521, 'learning_rate': 3.994391936693205e-05, 'epoch': 0.53}

  5%|▌         | 2192/41250 [5:17:28<94:42:06,  8.73s/it][2025-04-25 13:15:12,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:15:12,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5882.38 | bwd_inner_microstep: 5681.30 | bwd_allreduce_microstep: 201.03 | step_microstep: 18.75
[2025-04-25 13:15:12,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5882.39 | bwd_inner: 5681.30 | bwd_allreduce: 201.05 | step: 18.75

  5%|▌         | 2193/41250 [5:17:37<94:58:29,  8.75s/it]
                                                         
{'loss': 0.2002, 'grad_norm': 1.958289384841919, 'learning_rate': 3.994380179097123e-05, 'epoch': 0.53}

  5%|▌         | 2193/41250 [5:17:37<94:58:29,  8.75s/it][2025-04-25 13:15:20,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:15:20,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.65 | bwd_microstep: 5754.60 | bwd_inner_microstep: 5681.14 | bwd_allreduce_microstep: 73.42 | step_microstep: 18.59
[2025-04-25 13:15:20,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.65 | bwd: 5754.62 | bwd_inner: 5681.14 | bwd_allreduce: 73.44 | step: 18.60

  5%|▌         | 2194/41250 [5:17:46<94:43:57,  8.73s/it]
                                                         
{'loss': 0.1592, 'grad_norm': 1.4308326244354248, 'learning_rate': 3.99436840920608e-05, 'epoch': 0.53}

  5%|▌         | 2194/41250 [5:17:46<94:43:57,  8.73s/it][2025-04-25 13:15:29,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 13:15:29,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.35 | bwd_microstep: 5754.47 | bwd_inner_microstep: 5653.06 | bwd_allreduce_microstep: 101.37 | step_microstep: 18.51
[2025-04-25 13:15:29,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.35 | bwd: 5754.48 | bwd_inner: 5653.06 | bwd_allreduce: 101.38 | step: 18.51

  5%|▌         | 2195/41250 [5:17:54<94:30:20,  8.71s/it]
                                                         
{'loss': 0.1469, 'grad_norm': 2.7417755126953125, 'learning_rate': 3.994356627020147e-05, 'epoch': 0.53}

  5%|▌         | 2195/41250 [5:17:54<94:30:20,  8.71s/it][2025-04-25 13:15:38,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 13:15:38,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.27 | bwd_microstep: 5704.50 | bwd_inner_microstep: 5650.28 | bwd_allreduce_microstep: 54.18 | step_microstep: 19.18
[2025-04-25 13:15:38,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.27 | bwd: 5704.51 | bwd_inner: 5650.28 | bwd_allreduce: 54.19 | step: 19.19

  5%|▌         | 2196/41250 [5:18:03<94:10:55,  8.68s/it]
                                                         
{'loss': 0.159, 'grad_norm': 2.017324447631836, 'learning_rate': 3.994344832539398e-05, 'epoch': 0.53}

  5%|▌         | 2196/41250 [5:18:03<94:10:55,  8.68s/it][2025-04-25 13:15:46,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 13:15:46,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.46 | bwd_microstep: 5694.44 | bwd_inner_microstep: 5639.71 | bwd_allreduce_microstep: 54.68 | step_microstep: 18.69
[2025-04-25 13:15:46,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.46 | bwd: 5694.45 | bwd_inner: 5639.71 | bwd_allreduce: 54.70 | step: 18.70

  5%|▌         | 2197/41250 [5:18:12<93:55:55,  8.66s/it]
                                                         
{'loss': 0.2767, 'grad_norm': 2.202413558959961, 'learning_rate': 3.994333025763905e-05, 'epoch': 0.53}

  5%|▌         | 2197/41250 [5:18:12<93:55:55,  8.66s/it][2025-04-25 13:15:55,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:15:55,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.37 | bwd_microstep: 5721.93 | bwd_inner_microstep: 5709.05 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.89
[2025-04-25 13:15:55,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.37 | bwd: 5721.94 | bwd_inner: 5709.05 | bwd_allreduce: 12.85 | step: 18.89

  5%|▌         | 2198/41250 [5:18:20<93:56:30,  8.66s/it]
                                                         
{'loss': 0.1267, 'grad_norm': 1.441326379776001, 'learning_rate': 3.994321206693742e-05, 'epoch': 0.53}

  5%|▌         | 2198/41250 [5:18:20<93:56:30,  8.66s/it][2025-04-25 13:16:04,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:16:04,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5697.18 | bwd_inner_microstep: 5652.70 | bwd_allreduce_microstep: 44.44 | step_microstep: 18.59
[2025-04-25 13:16:04,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5697.19 | bwd_inner: 5652.70 | bwd_allreduce: 44.46 | step: 18.59

  5%|▌         | 2199/41250 [5:18:29<93:47:12,  8.65s/it]
                                                         
{'loss': 0.1331, 'grad_norm': 1.3396261930465698, 'learning_rate': 3.994309375328981e-05, 'epoch': 0.53}

  5%|▌         | 2199/41250 [5:18:29<93:47:12,  8.65s/it][2025-04-25 13:16:12,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:16:12,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.56 | bwd_microstep: 5757.83 | bwd_inner_microstep: 5645.86 | bwd_allreduce_microstep: 111.93 | step_microstep: 18.38
[2025-04-25 13:16:12,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.56 | bwd: 5757.84 | bwd_inner: 5645.86 | bwd_allreduce: 111.94 | step: 18.39

  5%|▌         | 2200/41250 [5:18:38<93:50:44,  8.65s/it]
                                                         
{'loss': 0.0988, 'grad_norm': 1.1735280752182007, 'learning_rate': 3.9942975316696944e-05, 'epoch': 0.53}

  5%|▌         | 2200/41250 [5:18:38<93:50:44,  8.65s/it][2025-04-25 13:16:21,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 1.15
[2025-04-25 13:16:21,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.89 | bwd_microstep: 5753.75 | bwd_inner_microstep: 5646.18 | bwd_allreduce_microstep: 107.51 | step_microstep: 19.67
[2025-04-25 13:16:21,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.89 | bwd: 5753.76 | bwd_inner: 5646.18 | bwd_allreduce: 107.53 | step: 19.67

  5%|▌         | 2201/41250 [5:18:46<93:54:45,  8.66s/it]
                                                         
{'loss': 0.1419, 'grad_norm': 1.3474186658859253, 'learning_rate': 3.994285675715956e-05, 'epoch': 0.53}

  5%|▌         | 2201/41250 [5:18:46<93:54:45,  8.66s/it][2025-04-25 13:16:30,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:16:30,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5739.44 | bwd_inner_microstep: 5695.49 | bwd_allreduce_microstep: 43.90 | step_microstep: 18.87
[2025-04-25 13:16:30,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5739.45 | bwd_inner: 5695.49 | bwd_allreduce: 43.92 | step: 18.87

  5%|▌         | 2202/41250 [5:18:55<93:57:26,  8.66s/it]
                                                         
{'loss': 0.1565, 'grad_norm': 2.0573318004608154, 'learning_rate': 3.9942738074678386e-05, 'epoch': 0.53}

  5%|▌         | 2202/41250 [5:18:55<93:57:26,  8.66s/it][2025-04-25 13:16:38,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 13:16:38,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.51 | bwd_microstep: 5706.81 | bwd_inner_microstep: 5693.86 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.06
[2025-04-25 13:16:38,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.51 | bwd: 5706.82 | bwd_inner: 5693.86 | bwd_allreduce: 12.93 | step: 19.06

  5%|▌         | 2203/41250 [5:19:04<93:52:51,  8.66s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 1.4057198762893677, 'learning_rate': 3.9942619269254154e-05, 'epoch': 0.53}

  5%|▌         | 2203/41250 [5:19:04<93:52:51,  8.66s/it][2025-04-25 13:16:47,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:16:47,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5746.01 | bwd_inner_microstep: 5695.44 | bwd_allreduce_microstep: 50.52 | step_microstep: 18.77
[2025-04-25 13:16:47,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5746.02 | bwd_inner: 5695.44 | bwd_allreduce: 50.54 | step: 18.77

  5%|▌         | 2204/41250 [5:19:12<93:58:05,  8.66s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.6716833114624023, 'learning_rate': 3.9942500340887596e-05, 'epoch': 0.53}

  5%|▌         | 2204/41250 [5:19:12<93:58:05,  8.66s/it][2025-04-25 13:16:56,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 13:16:56,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.23 | bwd_microstep: 5744.20 | bwd_inner_microstep: 5711.45 | bwd_allreduce_microstep: 32.71 | step_microstep: 19.04
[2025-04-25 13:16:56,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.23 | bwd: 5744.22 | bwd_inner: 5711.45 | bwd_allreduce: 32.73 | step: 19.04

  5%|▌         | 2205/41250 [5:19:21<94:01:39,  8.67s/it]
                                                         
{'loss': 0.1922, 'grad_norm': 1.9905352592468262, 'learning_rate': 3.994238128957944e-05, 'epoch': 0.53}

  5%|▌         | 2205/41250 [5:19:21<94:01:39,  8.67s/it][2025-04-25 13:17:04,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.29 | optimizer_step: 1.05
[2025-04-25 13:17:04,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5783.08 | bwd_inner_microstep: 5662.12 | bwd_allreduce_microstep: 120.89 | step_microstep: 20.13
[2025-04-25 13:17:04,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5783.09 | bwd_inner: 5662.12 | bwd_allreduce: 120.92 | step: 20.14

  5%|▌         | 2206/41250 [5:19:30<94:07:31,  8.68s/it]
                                                         
{'loss': 0.0985, 'grad_norm': 0.9158285856246948, 'learning_rate': 3.994226211533044e-05, 'epoch': 0.53}

  5%|▌         | 2206/41250 [5:19:30<94:07:31,  8.68s/it][2025-04-25 13:17:13,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 13:17:13,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.21 | bwd_microstep: 5713.22 | bwd_inner_microstep: 5663.17 | bwd_allreduce_microstep: 50.01 | step_microstep: 19.02
[2025-04-25 13:17:13,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.21 | bwd: 5713.24 | bwd_inner: 5663.17 | bwd_allreduce: 50.03 | step: 19.02

  5%|▌         | 2207/41250 [5:19:38<93:57:51,  8.66s/it]
                                                         
{'loss': 0.3343, 'grad_norm': 2.4599897861480713, 'learning_rate': 3.994214281814131e-05, 'epoch': 0.54}

  5%|▌         | 2207/41250 [5:19:38<93:57:51,  8.66s/it][2025-04-25 13:17:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:17:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.29 | bwd_microstep: 5782.51 | bwd_inner_microstep: 5769.67 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.94
[2025-04-25 13:17:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.29 | bwd: 5782.52 | bwd_inner: 5769.67 | bwd_allreduce: 12.81 | step: 18.95

  5%|▌         | 2208/41250 [5:19:47<94:14:47,  8.69s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 2.325143337249756, 'learning_rate': 3.994202339801279e-05, 'epoch': 0.54}

  5%|▌         | 2208/41250 [5:19:47<94:14:47,  8.69s/it][2025-04-25 13:17:30,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 13:17:30,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5817.96 | bwd_inner_microstep: 5665.34 | bwd_allreduce_microstep: 152.56 | step_microstep: 18.77
[2025-04-25 13:17:30,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5817.97 | bwd_inner: 5665.34 | bwd_allreduce: 152.59 | step: 18.78

  5%|▌         | 2209/41250 [5:19:56<94:21:43,  8.70s/it]
                                                         
{'loss': 0.2148, 'grad_norm': 1.7935824394226074, 'learning_rate': 3.994190385494562e-05, 'epoch': 0.54}

  5%|▌         | 2209/41250 [5:19:56<94:21:43,  8.70s/it][2025-04-25 13:17:39,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 13:17:39,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.69 | bwd_microstep: 5704.62 | bwd_inner_microstep: 5672.78 | bwd_allreduce_microstep: 31.79 | step_microstep: 19.47
[2025-04-25 13:17:39,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.69 | bwd: 5704.63 | bwd_inner: 5672.78 | bwd_allreduce: 31.80 | step: 19.47

  5%|▌         | 2210/41250 [5:20:04<94:06:41,  8.68s/it]
                                                         
{'loss': 0.0425, 'grad_norm': 1.1480684280395508, 'learning_rate': 3.994178418894053e-05, 'epoch': 0.54}

  5%|▌         | 2210/41250 [5:20:04<94:06:41,  8.68s/it][2025-04-25 13:17:48,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:17:48,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.94 | bwd_microstep: 5789.84 | bwd_inner_microstep: 5673.41 | bwd_allreduce_microstep: 116.39 | step_microstep: 18.49
[2025-04-25 13:17:48,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.94 | bwd: 5789.85 | bwd_inner: 5673.41 | bwd_allreduce: 116.40 | step: 18.50

  5%|▌         | 2211/41250 [5:20:13<94:12:27,  8.69s/it]
                                                         
{'loss': 0.2116, 'grad_norm': 2.380162477493286, 'learning_rate': 3.994166439999827e-05, 'epoch': 0.54}

  5%|▌         | 2211/41250 [5:20:13<94:12:27,  8.69s/it][2025-04-25 13:17:56,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 13:17:56,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.30 | bwd_microstep: 5730.55 | bwd_inner_microstep: 5668.01 | bwd_allreduce_microstep: 62.49 | step_microstep: 18.87
[2025-04-25 13:17:56,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.30 | bwd: 5730.56 | bwd_inner: 5668.01 | bwd_allreduce: 62.51 | step: 18.87

  5%|▌         | 2212/41250 [5:20:22<94:04:08,  8.67s/it]
                                                         
{'loss': 0.3776, 'grad_norm': 2.4132561683654785, 'learning_rate': 3.9941544488119565e-05, 'epoch': 0.54}

  5%|▌         | 2212/41250 [5:20:22<94:04:08,  8.67s/it][2025-04-25 13:18:05,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 13:18:05,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5764.28 | bwd_inner_microstep: 5697.94 | bwd_allreduce_microstep: 66.29 | step_microstep: 19.05
[2025-04-25 13:18:05,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5764.30 | bwd_inner: 5697.94 | bwd_allreduce: 66.31 | step: 19.05

  5%|▌         | 2213/41250 [5:20:30<94:08:17,  8.68s/it]
                                                         
{'loss': 0.3021, 'grad_norm': 2.6687428951263428, 'learning_rate': 3.994142445330517e-05, 'epoch': 0.54}

  5%|▌         | 2213/41250 [5:20:30<94:08:17,  8.68s/it][2025-04-25 13:18:14,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:18:14,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.09 | bwd_microstep: 5799.31 | bwd_inner_microstep: 5666.74 | bwd_allreduce_microstep: 132.53 | step_microstep: 18.74
[2025-04-25 13:18:14,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.09 | bwd: 5799.33 | bwd_inner: 5666.74 | bwd_allreduce: 132.55 | step: 18.74

  5%|▌         | 2214/41250 [5:20:39<94:14:44,  8.69s/it]
                                                         
{'loss': 0.2021, 'grad_norm': 1.0792773962020874, 'learning_rate': 3.994130429555581e-05, 'epoch': 0.54}

  5%|▌         | 2214/41250 [5:20:39<94:14:44,  8.69s/it][2025-04-25 13:18:23,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:18:23,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.63 | bwd_microstep: 5883.94 | bwd_inner_microstep: 5694.78 | bwd_allreduce_microstep: 189.12 | step_microstep: 18.66
[2025-04-25 13:18:23,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.63 | bwd: 5883.95 | bwd_inner: 5694.78 | bwd_allreduce: 189.13 | step: 18.67

  5%|▌         | 2215/41250 [5:20:48<94:40:30,  8.73s/it]
                                                         
{'loss': 0.1896, 'grad_norm': 1.3574001789093018, 'learning_rate': 3.9941184014872224e-05, 'epoch': 0.54}

  5%|▌         | 2215/41250 [5:20:48<94:40:30,  8.73s/it][2025-04-25 13:18:31,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 13:18:31,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.81 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5705.21 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.94
[2025-04-25 13:18:31,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.81 | bwd: 5717.95 | bwd_inner: 5705.21 | bwd_allreduce: 12.70 | step: 18.94

  5%|▌         | 2216/41250 [5:20:57<94:23:57,  8.71s/it]
                                                         
{'loss': 0.1599, 'grad_norm': 1.3442550897598267, 'learning_rate': 3.994106361125517e-05, 'epoch': 0.54}

  5%|▌         | 2216/41250 [5:20:57<94:23:57,  8.71s/it][2025-04-25 13:18:40,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.32 | optimizer_step: 1.02
[2025-04-25 13:18:40,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.31 | bwd_microstep: 5735.56 | bwd_inner_microstep: 5722.11 | bwd_allreduce_microstep: 13.40 | step_microstep: 19.67
[2025-04-25 13:18:40,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.31 | bwd: 5735.57 | bwd_inner: 5722.11 | bwd_allreduce: 13.42 | step: 19.67

  5%|▌         | 2217/41250 [5:21:05<94:18:36,  8.70s/it]
                                                         
{'loss': 0.1535, 'grad_norm': 1.2530639171600342, 'learning_rate': 3.994094308470538e-05, 'epoch': 0.54}

  5%|▌         | 2217/41250 [5:21:05<94:18:36,  8.70s/it][2025-04-25 13:18:49,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 13:18:49,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.12 | bwd_microstep: 5714.80 | bwd_inner_microstep: 5677.53 | bwd_allreduce_microstep: 37.23 | step_microstep: 18.97
[2025-04-25 13:18:49,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.12 | bwd: 5714.82 | bwd_inner: 5677.53 | bwd_allreduce: 37.25 | step: 18.97

  5%|▌         | 2218/41250 [5:21:14<94:05:07,  8.68s/it]
                                                         
{'loss': 0.1209, 'grad_norm': 1.6500709056854248, 'learning_rate': 3.9940822435223596e-05, 'epoch': 0.54}

  5%|▌         | 2218/41250 [5:21:14<94:05:07,  8.68s/it][2025-04-25 13:18:57,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:18:57,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.58 | bwd_microstep: 5873.27 | bwd_inner_microstep: 5663.92 | bwd_allreduce_microstep: 209.30 | step_microstep: 18.48
[2025-04-25 13:18:57,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.58 | bwd: 5873.28 | bwd_inner: 5663.92 | bwd_allreduce: 209.32 | step: 18.48

  5%|▌         | 2219/41250 [5:21:23<94:26:16,  8.71s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.8205188512802124, 'learning_rate': 3.9940701662810564e-05, 'epoch': 0.54}

  5%|▌         | 2219/41250 [5:21:23<94:26:16,  8.71s/it][2025-04-25 13:19:06,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:19:06,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.93 | bwd_microstep: 5708.46 | bwd_inner_microstep: 5695.83 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.53
[2025-04-25 13:19:06,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.93 | bwd: 5708.47 | bwd_inner: 5695.83 | bwd_allreduce: 12.60 | step: 18.53

  5%|▌         | 2220/41250 [5:21:31<94:12:44,  8.69s/it]
                                                         
{'loss': 0.3056, 'grad_norm': 2.1613025665283203, 'learning_rate': 3.994058076746703e-05, 'epoch': 0.54}

  5%|▌         | 2220/41250 [5:21:31<94:12:44,  8.69s/it][2025-04-25 13:19:15,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 13:19:15,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.12 | bwd_microstep: 5706.71 | bwd_inner_microstep: 5693.98 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.83
[2025-04-25 13:19:15,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.12 | bwd: 5706.73 | bwd_inner: 5693.98 | bwd_allreduce: 12.71 | step: 18.83

  5%|▌         | 2221/41250 [5:21:40<94:02:03,  8.67s/it]
                                                         
{'loss': 0.1796, 'grad_norm': 1.955916404724121, 'learning_rate': 3.994045974919374e-05, 'epoch': 0.54}

  5%|▌         | 2221/41250 [5:21:40<94:02:03,  8.67s/it][2025-04-25 13:19:23,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:19:23,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5752.73 | bwd_inner_microstep: 5694.89 | bwd_allreduce_microstep: 57.80 | step_microstep: 18.72
[2025-04-25 13:19:23,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5752.74 | bwd_inner: 5694.89 | bwd_allreduce: 57.82 | step: 18.72

  5%|▌         | 2222/41250 [5:21:49<94:04:27,  8.68s/it]
                                                         
{'loss': 0.1777, 'grad_norm': 2.098773717880249, 'learning_rate': 3.994033860799143e-05, 'epoch': 0.54}

  5%|▌         | 2222/41250 [5:21:49<94:04:27,  8.68s/it][2025-04-25 13:19:32,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:19:32,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.22 | bwd_microstep: 5785.25 | bwd_inner_microstep: 5649.85 | bwd_allreduce_microstep: 135.33 | step_microstep: 18.84
[2025-04-25 13:19:32,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.22 | bwd: 5785.26 | bwd_inner: 5649.85 | bwd_allreduce: 135.36 | step: 18.84

  5%|▌         | 2223/41250 [5:21:57<94:08:18,  8.68s/it]
                                                         
{'loss': 0.1971, 'grad_norm': 1.5364928245544434, 'learning_rate': 3.994021734386086e-05, 'epoch': 0.54}

  5%|▌         | 2223/41250 [5:21:57<94:08:18,  8.68s/it][2025-04-25 13:19:41,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-25 13:19:41,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.04 | bwd_microstep: 5756.94 | bwd_inner_microstep: 5686.41 | bwd_allreduce_microstep: 70.47 | step_microstep: 20.22
[2025-04-25 13:19:41,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.04 | bwd: 5756.96 | bwd_inner: 5686.41 | bwd_allreduce: 70.50 | step: 20.22

  5%|▌         | 2224/41250 [5:22:06<94:08:34,  8.68s/it]
                                                         
{'loss': 0.2318, 'grad_norm': 1.529250144958496, 'learning_rate': 3.994009595680277e-05, 'epoch': 0.54}

  5%|▌         | 2224/41250 [5:22:06<94:08:34,  8.68s/it][2025-04-25 13:19:49,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:19:49,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.32 | bwd_microstep: 5744.12 | bwd_inner_microstep: 5686.76 | bwd_allreduce_microstep: 57.31 | step_microstep: 18.85
[2025-04-25 13:19:49,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.32 | bwd: 5744.13 | bwd_inner: 5686.76 | bwd_allreduce: 57.32 | step: 18.86

  5%|▌         | 2225/41250 [5:22:15<94:06:46,  8.68s/it]
                                                         
{'loss': 0.1405, 'grad_norm': 1.0934579372406006, 'learning_rate': 3.993997444681791e-05, 'epoch': 0.54}

  5%|▌         | 2225/41250 [5:22:15<94:06:46,  8.68s/it][2025-04-25 13:19:58,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:19:58,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.39 | bwd_microstep: 5746.31 | bwd_inner_microstep: 5651.31 | bwd_allreduce_microstep: 94.96 | step_microstep: 18.58
[2025-04-25 13:19:58,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.39 | bwd: 5746.33 | bwd_inner: 5651.31 | bwd_allreduce: 94.98 | step: 18.58

  5%|▌         | 2226/41250 [5:22:23<94:00:55,  8.67s/it]
                                                         
{'loss': 0.1011, 'grad_norm': 0.9940142631530762, 'learning_rate': 3.9939852813907036e-05, 'epoch': 0.54}

  5%|▌         | 2226/41250 [5:22:23<94:00:55,  8.67s/it][2025-04-25 13:20:07,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 13:20:07,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5721.39 | bwd_inner_microstep: 5708.66 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.74
[2025-04-25 13:20:07,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5721.40 | bwd_inner: 5708.65 | bwd_allreduce: 12.71 | step: 18.74

  5%|▌         | 2227/41250 [5:22:32<93:57:14,  8.67s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.210028052330017, 'learning_rate': 3.993973105807088e-05, 'epoch': 0.54}

  5%|▌         | 2227/41250 [5:22:32<93:57:14,  8.67s/it][2025-04-25 13:20:15,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:20:15,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.45 | bwd_microstep: 5707.18 | bwd_inner_microstep: 5694.19 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.78
[2025-04-25 13:20:15,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.45 | bwd: 5707.19 | bwd_inner: 5694.19 | bwd_allreduce: 12.96 | step: 18.77

  5%|▌         | 2228/41250 [5:22:41<93:51:17,  8.66s/it]
                                                         
{'loss': 0.1005, 'grad_norm': 1.328773856163025, 'learning_rate': 3.993960917931021e-05, 'epoch': 0.54}

  5%|▌         | 2228/41250 [5:22:41<93:51:17,  8.66s/it][2025-04-25 13:20:24,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:20:24,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.66 | bwd_microstep: 5680.53 | bwd_inner_microstep: 5650.87 | bwd_allreduce_microstep: 29.61 | step_microstep: 18.69
[2025-04-25 13:20:24,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.66 | bwd: 5680.55 | bwd_inner: 5650.87 | bwd_allreduce: 29.63 | step: 18.70

  5%|▌         | 2229/41250 [5:22:49<93:37:16,  8.64s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 1.210121750831604, 'learning_rate': 3.9939487177625774e-05, 'epoch': 0.54}

  5%|▌         | 2229/41250 [5:22:49<93:37:16,  8.64s/it][2025-04-25 13:20:33,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:20:33,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.41 | bwd_microstep: 5777.50 | bwd_inner_microstep: 5653.35 | bwd_allreduce_microstep: 124.10 | step_microstep: 18.98
[2025-04-25 13:20:33,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.41 | bwd: 5777.51 | bwd_inner: 5653.35 | bwd_allreduce: 124.11 | step: 18.98

  5%|▌         | 2230/41250 [5:22:58<93:46:57,  8.65s/it]
                                                         
{'loss': 0.1082, 'grad_norm': 1.0886125564575195, 'learning_rate': 3.9939365053018316e-05, 'epoch': 0.54}

  5%|▌         | 2230/41250 [5:22:58<93:46:57,  8.65s/it][2025-04-25 13:20:41,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 13:20:41,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.12 | bwd_microstep: 5723.83 | bwd_inner_microstep: 5711.10 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.33
[2025-04-25 13:20:41,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.12 | bwd: 5723.84 | bwd_inner: 5711.10 | bwd_allreduce: 12.70 | step: 19.34

  5%|▌         | 2231/41250 [5:23:07<93:49:02,  8.66s/it]
                                                         
{'loss': 0.2102, 'grad_norm': 1.9329288005828857, 'learning_rate': 3.99392428054886e-05, 'epoch': 0.54}

  5%|▌         | 2231/41250 [5:23:07<93:49:02,  8.66s/it][2025-04-25 13:20:50,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:20:50,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.72 | bwd_microstep: 5702.05 | bwd_inner_microstep: 5643.26 | bwd_allreduce_microstep: 58.74 | step_microstep: 18.57
[2025-04-25 13:20:50,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.72 | bwd: 5702.06 | bwd_inner: 5643.26 | bwd_allreduce: 58.76 | step: 18.57

  5%|▌         | 2232/41250 [5:23:15<93:42:11,  8.65s/it]
                                                         
{'loss': 0.0617, 'grad_norm': 0.649806797504425, 'learning_rate': 3.993912043503737e-05, 'epoch': 0.54}

  5%|▌         | 2232/41250 [5:23:15<93:42:11,  8.65s/it][2025-04-25 13:20:58,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:20:58,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5701.72 | bwd_inner_microstep: 5649.17 | bwd_allreduce_microstep: 52.50 | step_microstep: 18.68
[2025-04-25 13:20:58,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5701.73 | bwd_inner: 5649.17 | bwd_allreduce: 52.52 | step: 18.68

  5%|▌         | 2233/41250 [5:23:24<93:35:28,  8.64s/it]
                                                         
{'loss': 0.1078, 'grad_norm': 0.9148414134979248, 'learning_rate': 3.993899794166539e-05, 'epoch': 0.54}

  5%|▌         | 2233/41250 [5:23:24<93:35:28,  8.64s/it][2025-04-25 13:21:07,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:21:07,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5701.00 | bwd_inner_microstep: 5688.10 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.90
[2025-04-25 13:21:07,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5701.01 | bwd_inner: 5688.10 | bwd_allreduce: 12.87 | step: 18.91

  5%|▌         | 2234/41250 [5:23:32<93:34:47,  8.63s/it]
                                                         
{'loss': 0.1272, 'grad_norm': 1.2828272581100464, 'learning_rate': 3.99388753253734e-05, 'epoch': 0.54}

  5%|▌         | 2234/41250 [5:23:32<93:34:47,  8.63s/it][2025-04-25 13:21:16,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 13:21:16,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5861.90 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 201.91 | step_microstep: 19.25
[2025-04-25 13:21:16,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.45 | bwd: 5861.91 | bwd_inner: 5659.93 | bwd_allreduce: 201.93 | step: 19.25

  5%|▌         | 2235/41250 [5:23:41<94:03:13,  8.68s/it]
                                                         
{'loss': 0.3742, 'grad_norm': 2.5670571327209473, 'learning_rate': 3.993875258616217e-05, 'epoch': 0.54}

  5%|▌         | 2235/41250 [5:23:41<94:03:13,  8.68s/it][2025-04-25 13:21:25,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.29 | optimizer_step: 1.03
[2025-04-25 13:21:25,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.66 | bwd_microstep: 5759.18 | bwd_inner_microstep: 5645.49 | bwd_allreduce_microstep: 113.63 | step_microstep: 19.79
[2025-04-25 13:21:25,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.66 | bwd: 5759.19 | bwd_inner: 5645.49 | bwd_allreduce: 113.66 | step: 19.79

  5%|▌         | 2236/41250 [5:23:50<94:00:19,  8.67s/it]
                                                         
{'loss': 0.4654, 'grad_norm': 2.3831474781036377, 'learning_rate': 3.993862972403246e-05, 'epoch': 0.54}

  5%|▌         | 2236/41250 [5:23:50<94:00:19,  8.67s/it][2025-04-25 13:21:33,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 13:21:33,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.74 | bwd_microstep: 5764.45 | bwd_inner_microstep: 5648.73 | bwd_allreduce_microstep: 115.67 | step_microstep: 18.87
[2025-04-25 13:21:33,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.74 | bwd: 5764.46 | bwd_inner: 5648.73 | bwd_allreduce: 115.69 | step: 18.88

  5%|▌         | 2237/41250 [5:23:59<93:59:30,  8.67s/it]
                                                         
{'loss': 0.1663, 'grad_norm': 1.5716248750686646, 'learning_rate': 3.9938506738985005e-05, 'epoch': 0.54}

  5%|▌         | 2237/41250 [5:23:59<93:59:30,  8.67s/it][2025-04-25 13:21:42,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:21:42,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5705.72 | bwd_inner_microstep: 5690.67 | bwd_allreduce_microstep: 15.01 | step_microstep: 18.65
[2025-04-25 13:21:42,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5705.73 | bwd_inner: 5690.67 | bwd_allreduce: 15.02 | step: 18.65

  5%|▌         | 2238/41250 [5:24:07<93:53:39,  8.66s/it]
                                                         
{'loss': 0.0395, 'grad_norm': 0.6360436081886292, 'learning_rate': 3.993838363102058e-05, 'epoch': 0.54}

  5%|▌         | 2238/41250 [5:24:07<93:53:39,  8.66s/it][2025-04-25 13:21:51,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:21:51,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.63 | bwd_microstep: 5882.80 | bwd_inner_microstep: 5870.08 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.58
[2025-04-25 13:21:51,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.63 | bwd: 5882.81 | bwd_inner: 5870.08 | bwd_allreduce: 12.69 | step: 18.59

  5%|▌         | 2239/41250 [5:24:16<94:40:02,  8.74s/it]
                                                         
{'loss': 0.2375, 'grad_norm': 3.6999120712280273, 'learning_rate': 3.993826040013996e-05, 'epoch': 0.54}

  5%|▌         | 2239/41250 [5:24:16<94:40:02,  8.74s/it][2025-04-25 13:21:59,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 13:21:59,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.03 | bwd_microstep: 5692.04 | bwd_inner_microstep: 5650.53 | bwd_allreduce_microstep: 41.47 | step_microstep: 18.71
[2025-04-25 13:21:59,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.03 | bwd: 5692.05 | bwd_inner: 5650.53 | bwd_allreduce: 41.49 | step: 18.71

  5%|▌         | 2240/41250 [5:24:25<94:13:12,  8.70s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.1452610492706299, 'learning_rate': 3.993813704634386e-05, 'epoch': 0.54}

  5%|▌         | 2240/41250 [5:24:25<94:13:12,  8.70s/it][2025-04-25 13:22:08,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:22:08,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5738.85 | bwd_inner_microstep: 5700.37 | bwd_allreduce_microstep: 38.44 | step_microstep: 18.62
[2025-04-25 13:22:08,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5738.87 | bwd_inner: 5700.37 | bwd_allreduce: 38.46 | step: 18.62

  5%|▌         | 2241/41250 [5:24:33<94:08:43,  8.69s/it]
                                                         
{'loss': 0.2096, 'grad_norm': 2.2755792140960693, 'learning_rate': 3.9938013569633084e-05, 'epoch': 0.54}

  5%|▌         | 2241/41250 [5:24:33<94:08:43,  8.69s/it][2025-04-25 13:22:17,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:22:17,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.53 | bwd_microstep: 5756.55 | bwd_inner_microstep: 5645.73 | bwd_allreduce_microstep: 110.78 | step_microstep: 18.73
[2025-04-25 13:22:17,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.53 | bwd: 5756.57 | bwd_inner: 5645.73 | bwd_allreduce: 110.80 | step: 18.73

  5%|▌         | 2242/41250 [5:24:42<94:03:49,  8.68s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.9548428058624268, 'learning_rate': 3.9937889970008375e-05, 'epoch': 0.54}

  5%|▌         | 2242/41250 [5:24:42<94:03:49,  8.68s/it][2025-04-25 13:22:25,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 13:22:25,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.79 | bwd_microstep: 5703.55 | bwd_inner_microstep: 5691.14 | bwd_allreduce_microstep: 12.36 | step_microstep: 19.43
[2025-04-25 13:22:25,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.79 | bwd: 5703.56 | bwd_inner: 5691.14 | bwd_allreduce: 12.38 | step: 19.43

  5%|▌         | 2243/41250 [5:24:51<93:54:08,  8.67s/it]
                                                         
{'loss': 0.2242, 'grad_norm': 3.8792548179626465, 'learning_rate': 3.9937766247470484e-05, 'epoch': 0.54}

  5%|▌         | 2243/41250 [5:24:51<93:54:08,  8.67s/it][2025-04-25 13:22:34,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:22:34,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.04 | bwd_microstep: 5693.38 | bwd_inner_microstep: 5680.39 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.91
[2025-04-25 13:22:34,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.04 | bwd: 5693.40 | bwd_inner: 5680.39 | bwd_allreduce: 12.97 | step: 18.91

  5%|▌         | 2244/41250 [5:24:59<93:51:13,  8.66s/it]
                                                         
{'loss': 0.1871, 'grad_norm': 1.2970905303955078, 'learning_rate': 3.99376424020202e-05, 'epoch': 0.54}

  5%|▌         | 2244/41250 [5:24:59<93:51:13,  8.66s/it][2025-04-25 13:22:43,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 13:22:43,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.80 | bwd_microstep: 5784.36 | bwd_inner_microstep: 5771.45 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.76
[2025-04-25 13:22:43,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.80 | bwd: 5784.37 | bwd_inner: 5771.45 | bwd_allreduce: 12.88 | step: 18.76

  5%|▌         | 2245/41250 [5:25:08<94:08:10,  8.69s/it]
                                                         
{'loss': 0.2452, 'grad_norm': 1.548913598060608, 'learning_rate': 3.993751843365826e-05, 'epoch': 0.54}

  5%|▌         | 2245/41250 [5:25:08<94:08:10,  8.69s/it][2025-04-25 13:22:51,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:22:51,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5749.81 | bwd_inner_microstep: 5647.56 | bwd_allreduce_microstep: 102.21 | step_microstep: 18.78
[2025-04-25 13:22:51,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5749.82 | bwd_inner: 5647.56 | bwd_allreduce: 102.23 | step: 18.78

  5%|▌         | 2246/41250 [5:25:17<94:02:19,  8.68s/it]
                                                         
{'loss': 0.1721, 'grad_norm': 1.3215762376785278, 'learning_rate': 3.993739434238545e-05, 'epoch': 0.54}

  5%|▌         | 2246/41250 [5:25:17<94:02:19,  8.68s/it][2025-04-25 13:23:00,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:23:00,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.83 | bwd_microstep: 5695.68 | bwd_inner_microstep: 5659.55 | bwd_allreduce_microstep: 36.08 | step_microstep: 18.63
[2025-04-25 13:23:00,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.83 | bwd: 5695.69 | bwd_inner: 5659.55 | bwd_allreduce: 36.09 | step: 18.63

  5%|▌         | 2247/41250 [5:25:25<93:47:32,  8.66s/it]
                                                         
{'loss': 0.2481, 'grad_norm': 2.324841260910034, 'learning_rate': 3.9937270128202515e-05, 'epoch': 0.54}

  5%|▌         | 2247/41250 [5:25:25<93:47:32,  8.66s/it][2025-04-25 13:23:09,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:23:09,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.65 | bwd_microstep: 5731.88 | bwd_inner_microstep: 5649.35 | bwd_allreduce_microstep: 82.48 | step_microstep: 18.46
[2025-04-25 13:23:09,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.65 | bwd: 5731.89 | bwd_inner: 5649.35 | bwd_allreduce: 82.50 | step: 18.46

  5%|▌         | 2248/41250 [5:25:34<93:43:51,  8.65s/it]
                                                         
{'loss': 0.1751, 'grad_norm': 2.408862352371216, 'learning_rate': 3.9937145791110244e-05, 'epoch': 0.54}

  5%|▌         | 2248/41250 [5:25:34<93:43:51,  8.65s/it][2025-04-25 13:23:17,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:23:17,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.15 | bwd_microstep: 5684.54 | bwd_inner_microstep: 5646.09 | bwd_allreduce_microstep: 38.40 | step_microstep: 18.81
[2025-04-25 13:23:17,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.15 | bwd: 5684.55 | bwd_inner: 5646.09 | bwd_allreduce: 38.42 | step: 18.81

  5%|▌         | 2249/41250 [5:25:43<93:31:26,  8.63s/it]
                                                         
{'loss': 0.0723, 'grad_norm': 1.6452423334121704, 'learning_rate': 3.993702133110938e-05, 'epoch': 0.55}

  5%|▌         | 2249/41250 [5:25:43<93:31:26,  8.63s/it][2025-04-25 13:23:26,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-25 13:23:26,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.78 | bwd_microstep: 5731.74 | bwd_inner_microstep: 5686.12 | bwd_allreduce_microstep: 45.57 | step_microstep: 19.06
[2025-04-25 13:23:26,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.78 | bwd: 5731.75 | bwd_inner: 5686.12 | bwd_allreduce: 45.59 | step: 19.06

  5%|▌         | 2250/41250 [5:25:51<93:37:05,  8.64s/it]
                                                         
{'loss': 0.1288, 'grad_norm': 1.5748385190963745, 'learning_rate': 3.9936896748200705e-05, 'epoch': 0.55}

  5%|▌         | 2250/41250 [5:25:51<93:37:05,  8.64s/it][2025-04-25 13:23:34,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 13:23:34,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5670.10 | bwd_inner_microstep: 5637.83 | bwd_allreduce_microstep: 32.23 | step_microstep: 18.66
[2025-04-25 13:23:34,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5670.12 | bwd_inner: 5637.83 | bwd_allreduce: 32.25 | step: 18.67

  5%|▌         | 2251/41250 [5:26:00<93:24:20,  8.62s/it]
                                                         
{'loss': 0.2816, 'grad_norm': 2.287097692489624, 'learning_rate': 3.9936772042384983e-05, 'epoch': 0.55}

  5%|▌         | 2251/41250 [5:26:00<93:24:20,  8.62s/it][2025-04-25 13:23:43,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.13 | optimizer_step: 1.05
[2025-04-25 13:23:43,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.61 | bwd_microstep: 5728.81 | bwd_inner_microstep: 5639.35 | bwd_allreduce_microstep: 89.40 | step_microstep: 19.49
[2025-04-25 13:23:43,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.61 | bwd: 5728.82 | bwd_inner: 5639.35 | bwd_allreduce: 89.42 | step: 19.50

  5%|▌         | 2252/41250 [5:26:08<93:28:17,  8.63s/it]
                                                         
{'loss': 0.1134, 'grad_norm': 1.3255467414855957, 'learning_rate': 3.9936647213662985e-05, 'epoch': 0.55}

  5%|▌         | 2252/41250 [5:26:08<93:28:17,  8.63s/it][2025-04-25 13:23:52,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:23:52,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.67 | bwd_microstep: 5689.04 | bwd_inner_microstep: 5662.19 | bwd_allreduce_microstep: 26.81 | step_microstep: 18.87
[2025-04-25 13:23:52,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.67 | bwd: 5689.05 | bwd_inner: 5662.19 | bwd_allreduce: 26.82 | step: 18.87

  5%|▌         | 2253/41250 [5:26:17<93:23:59,  8.62s/it]
                                                         
{'loss': 0.2215, 'grad_norm': 1.502625823020935, 'learning_rate': 3.9936522262035476e-05, 'epoch': 0.55}

  5%|▌         | 2253/41250 [5:26:17<93:23:59,  8.62s/it][2025-04-25 13:24:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:24:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.91 | bwd_microstep: 5837.27 | bwd_inner_microstep: 5651.75 | bwd_allreduce_microstep: 185.48 | step_microstep: 18.87
[2025-04-25 13:24:00,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.91 | bwd: 5837.29 | bwd_inner: 5651.75 | bwd_allreduce: 185.50 | step: 18.88

  5%|▌         | 2254/41250 [5:26:26<93:48:41,  8.66s/it]
                                                         
{'loss': 0.0908, 'grad_norm': 0.9084054827690125, 'learning_rate': 3.993639718750323e-05, 'epoch': 0.55}

  5%|▌         | 2254/41250 [5:26:26<93:48:41,  8.66s/it][2025-04-25 13:24:09,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 13:24:09,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.31 | bwd_microstep: 5739.14 | bwd_inner_microstep: 5686.39 | bwd_allreduce_microstep: 52.70 | step_microstep: 19.56
[2025-04-25 13:24:09,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.31 | bwd: 5739.16 | bwd_inner: 5686.39 | bwd_allreduce: 52.72 | step: 19.56

  5%|▌         | 2255/41250 [5:26:34<93:49:21,  8.66s/it]
                                                         
{'loss': 0.1032, 'grad_norm': 1.264907717704773, 'learning_rate': 3.993627199006701e-05, 'epoch': 0.55}

  5%|▌         | 2255/41250 [5:26:34<93:49:21,  8.66s/it][2025-04-25 13:24:18,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 13:24:18,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.32 | bwd_microstep: 5717.47 | bwd_inner_microstep: 5687.41 | bwd_allreduce_microstep: 30.02 | step_microstep: 18.70
[2025-04-25 13:24:18,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.32 | bwd: 5717.48 | bwd_inner: 5687.41 | bwd_allreduce: 30.03 | step: 18.71

  5%|▌         | 2256/41250 [5:26:43<93:46:13,  8.66s/it]
                                                         
{'loss': 0.205, 'grad_norm': 1.7098525762557983, 'learning_rate': 3.99361466697276e-05, 'epoch': 0.55}

  5%|▌         | 2256/41250 [5:26:43<93:46:13,  8.66s/it][2025-04-25 13:24:26,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 13:24:26,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.75 | bwd_microstep: 5745.26 | bwd_inner_microstep: 5694.66 | bwd_allreduce_microstep: 50.56 | step_microstep: 18.87
[2025-04-25 13:24:26,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.75 | bwd: 5745.28 | bwd_inner: 5694.66 | bwd_allreduce: 50.58 | step: 18.87

  5%|▌         | 2257/41250 [5:26:52<93:49:46,  8.66s/it]
                                                         
{'loss': 0.0564, 'grad_norm': 0.9259118437767029, 'learning_rate': 3.993602122648577e-05, 'epoch': 0.55}

  5%|▌         | 2257/41250 [5:26:52<93:49:46,  8.66s/it][2025-04-25 13:24:35,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:24:35,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.86 | bwd_microstep: 5665.61 | bwd_inner_microstep: 5652.97 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.93
[2025-04-25 13:24:35,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.86 | bwd: 5665.63 | bwd_inner: 5652.97 | bwd_allreduce: 12.62 | step: 18.93

  5%|▌         | 2258/41250 [5:27:00<93:34:50,  8.64s/it]
                                                         
{'loss': 0.174, 'grad_norm': 1.3596240282058716, 'learning_rate': 3.993589566034228e-05, 'epoch': 0.55}

  5%|▌         | 2258/41250 [5:27:00<93:34:50,  8.64s/it][2025-04-25 13:24:44,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.07 | optimizer_step: 0.91
[2025-04-25 13:24:44,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.44 | bwd_microstep: 5694.38 | bwd_inner_microstep: 5659.48 | bwd_allreduce_microstep: 34.84 | step_microstep: 18.90
[2025-04-25 13:24:44,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.44 | bwd: 5694.40 | bwd_inner: 5659.48 | bwd_allreduce: 34.86 | step: 18.90

  5%|▌         | 2259/41250 [5:27:09<93:29:26,  8.63s/it]
                                                         
{'loss': 0.4356, 'grad_norm': 2.616136074066162, 'learning_rate': 3.9935769971297925e-05, 'epoch': 0.55}

  5%|▌         | 2259/41250 [5:27:09<93:29:26,  8.63s/it][2025-04-25 13:24:52,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 13:24:52,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.67 | bwd_microstep: 5771.84 | bwd_inner_microstep: 5652.00 | bwd_allreduce_microstep: 119.79 | step_microstep: 19.28
[2025-04-25 13:24:52,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.67 | bwd: 5771.85 | bwd_inner: 5652.00 | bwd_allreduce: 119.80 | step: 19.28

  5%|▌         | 2260/41250 [5:27:18<93:40:07,  8.65s/it]
                                                         
{'loss': 0.1284, 'grad_norm': 1.4541881084442139, 'learning_rate': 3.9935644159353465e-05, 'epoch': 0.55}

  5%|▌         | 2260/41250 [5:27:18<93:40:07,  8.65s/it][2025-04-25 13:25:01,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:25:01,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5760.42 | bwd_inner_microstep: 5656.07 | bwd_allreduce_microstep: 104.31 | step_microstep: 19.06
[2025-04-25 13:25:01,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5760.43 | bwd_inner: 5656.07 | bwd_allreduce: 104.32 | step: 19.06

  5%|▌         | 2261/41250 [5:27:26<93:45:23,  8.66s/it]
                                                         
{'loss': 0.0274, 'grad_norm': 0.48041394352912903, 'learning_rate': 3.993551822450968e-05, 'epoch': 0.55}

  5%|▌         | 2261/41250 [5:27:26<93:45:23,  8.66s/it][2025-04-25 13:25:10,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:25:10,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5765.67 | bwd_inner_microstep: 5712.49 | bwd_allreduce_microstep: 53.13 | step_microstep: 18.33
[2025-04-25 13:25:10,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5765.68 | bwd_inner: 5712.49 | bwd_allreduce: 53.15 | step: 18.33

  5%|▌         | 2262/41250 [5:27:35<93:54:23,  8.67s/it]
                                                         
{'loss': 0.2905, 'grad_norm': 1.8314660787582397, 'learning_rate': 3.9935392166767345e-05, 'epoch': 0.55}

  5%|▌         | 2262/41250 [5:27:35<93:54:23,  8.67s/it][2025-04-25 13:25:19,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:25:19,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.59 | bwd_microstep: 5839.40 | bwd_inner_microstep: 5790.15 | bwd_allreduce_microstep: 49.21 | step_microstep: 18.93
[2025-04-25 13:25:19,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.59 | bwd: 5839.42 | bwd_inner: 5790.15 | bwd_allreduce: 49.22 | step: 18.93

  5%|▌         | 2263/41250 [5:27:44<94:21:34,  8.71s/it]
                                                         
{'loss': 0.1986, 'grad_norm': 1.6681957244873047, 'learning_rate': 3.9935265986127244e-05, 'epoch': 0.55}

  5%|▌         | 2263/41250 [5:27:44<94:21:34,  8.71s/it][2025-04-25 13:25:27,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 13:25:27,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.09 | bwd_microstep: 5768.13 | bwd_inner_microstep: 5701.38 | bwd_allreduce_microstep: 66.71 | step_microstep: 19.30
[2025-04-25 13:25:27,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.09 | bwd: 5768.14 | bwd_inner: 5701.38 | bwd_allreduce: 66.72 | step: 19.30

  5%|▌         | 2264/41250 [5:27:53<94:18:31,  8.71s/it]
                                                         
{'loss': 0.1452, 'grad_norm': 1.1858474016189575, 'learning_rate': 3.993513968259015e-05, 'epoch': 0.55}

  5%|▌         | 2264/41250 [5:27:53<94:18:31,  8.71s/it][2025-04-25 13:25:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-25 13:25:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.93 | bwd_microstep: 5775.74 | bwd_inner_microstep: 5658.26 | bwd_allreduce_microstep: 117.44 | step_microstep: 18.42
[2025-04-25 13:25:36,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.93 | bwd: 5775.76 | bwd_inner: 5658.26 | bwd_allreduce: 117.45 | step: 18.43

  5%|▌         | 2265/41250 [5:28:01<94:14:59,  8.70s/it]
                                                         
{'loss': 0.2653, 'grad_norm': 1.8501144647598267, 'learning_rate': 3.993501325615683e-05, 'epoch': 0.55}

  5%|▌         | 2265/41250 [5:28:01<94:14:59,  8.70s/it][2025-04-25 13:25:45,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 13:25:45,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.25 | bwd_microstep: 5731.23 | bwd_inner_microstep: 5717.99 | bwd_allreduce_microstep: 13.20 | step_microstep: 19.03
[2025-04-25 13:25:45,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.25 | bwd: 5731.25 | bwd_inner: 5717.99 | bwd_allreduce: 13.22 | step: 19.03

  5%|▌         | 2266/41250 [5:28:10<94:09:43,  8.70s/it]
                                                         
{'loss': 0.3228, 'grad_norm': 1.9248874187469482, 'learning_rate': 3.993488670682808e-05, 'epoch': 0.55}

  5%|▌         | 2266/41250 [5:28:10<94:09:43,  8.70s/it][2025-04-25 13:25:53,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:25:53,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.24 | bwd_microstep: 5858.65 | bwd_inner_microstep: 5721.03 | bwd_allreduce_microstep: 137.57 | step_microstep: 18.66
[2025-04-25 13:25:53,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.24 | bwd: 5858.66 | bwd_inner: 5721.03 | bwd_allreduce: 137.59 | step: 18.67

  5%|▌         | 2267/41250 [5:28:19<94:32:13,  8.73s/it]
                                                         
{'loss': 0.2415, 'grad_norm': 3.585487127304077, 'learning_rate': 3.993476003460468e-05, 'epoch': 0.55}

  5%|▌         | 2267/41250 [5:28:19<94:32:13,  8.73s/it][2025-04-25 13:26:02,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 13:26:02,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.67 | bwd_microstep: 5735.48 | bwd_inner_microstep: 5710.12 | bwd_allreduce_microstep: 25.31 | step_microstep: 18.75
[2025-04-25 13:26:02,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.67 | bwd: 5735.49 | bwd_inner: 5710.12 | bwd_allreduce: 25.33 | step: 18.75

  5%|▌         | 2268/41250 [5:28:27<94:22:26,  8.72s/it]
                                                         
{'loss': 0.093, 'grad_norm': 0.9417563080787659, 'learning_rate': 3.99346332394874e-05, 'epoch': 0.55}

  5%|▌         | 2268/41250 [5:28:27<94:22:26,  8.72s/it][2025-04-25 13:26:11,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 13:26:11,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.63 | bwd_microstep: 5866.98 | bwd_inner_microstep: 5675.06 | bwd_allreduce_microstep: 191.88 | step_microstep: 19.14
[2025-04-25 13:26:11,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.63 | bwd: 5867.00 | bwd_inner: 5675.06 | bwd_allreduce: 191.90 | step: 19.14

  6%|▌         | 2269/41250 [5:28:36<94:37:24,  8.74s/it]
                                                         
{'loss': 0.0795, 'grad_norm': 0.8831248879432678, 'learning_rate': 3.993450632147702e-05, 'epoch': 0.55}

  6%|▌         | 2269/41250 [5:28:36<94:37:24,  8.74s/it][2025-04-25 13:26:20,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:26:20,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.45 | bwd_microstep: 5912.68 | bwd_inner_microstep: 5649.27 | bwd_allreduce_microstep: 263.36 | step_microstep: 18.82
[2025-04-25 13:26:20,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.45 | bwd: 5912.69 | bwd_inner: 5649.27 | bwd_allreduce: 263.38 | step: 18.82

  6%|▌         | 2270/41250 [5:28:45<94:53:39,  8.76s/it]
                                                         
{'loss': 0.1264, 'grad_norm': 1.3279759883880615, 'learning_rate': 3.993437928057434e-05, 'epoch': 0.55}

  6%|▌         | 2270/41250 [5:28:45<94:53:39,  8.76s/it][2025-04-25 13:26:28,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 13:26:28,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.80 | bwd_microstep: 5727.57 | bwd_inner_microstep: 5714.60 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.21
[2025-04-25 13:26:28,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.80 | bwd: 5727.59 | bwd_inner: 5714.60 | bwd_allreduce: 12.95 | step: 19.21

  6%|▌         | 2271/41250 [5:28:54<94:34:50,  8.74s/it]
                                                         
{'loss': 0.2039, 'grad_norm': 2.175703763961792, 'learning_rate': 3.9934252116780124e-05, 'epoch': 0.55}

  6%|▌         | 2271/41250 [5:28:54<94:34:50,  8.74s/it][2025-04-25 13:26:37,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-25 13:26:37,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.52 | bwd_microstep: 5720.52 | bwd_inner_microstep: 5707.63 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.35
[2025-04-25 13:26:37,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.52 | bwd: 5720.54 | bwd_inner: 5707.63 | bwd_allreduce: 12.86 | step: 19.35

  6%|▌         | 2272/41250 [5:29:02<94:20:04,  8.71s/it]
                                                         
{'loss': 0.0519, 'grad_norm': 1.2546074390411377, 'learning_rate': 3.9934124830095175e-05, 'epoch': 0.55}

  6%|▌         | 2272/41250 [5:29:02<94:20:04,  8.71s/it][2025-04-25 13:26:46,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 13:26:46,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.56 | bwd_microstep: 6048.62 | bwd_inner_microstep: 5671.42 | bwd_allreduce_microstep: 377.15 | step_microstep: 18.68
[2025-04-25 13:26:46,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.56 | bwd: 6048.64 | bwd_inner: 5671.42 | bwd_allreduce: 377.17 | step: 18.68

  6%|▌         | 2273/41250 [5:29:11<95:08:33,  8.79s/it]
                                                         
{'loss': 0.2195, 'grad_norm': 2.388190746307373, 'learning_rate': 3.993399742052026e-05, 'epoch': 0.55}

  6%|▌         | 2273/41250 [5:29:11<95:08:33,  8.79s/it][2025-04-25 13:26:55,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:26:55,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5718.43 | bwd_inner_microstep: 5705.57 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.87
[2025-04-25 13:26:55,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5718.44 | bwd_inner: 5705.57 | bwd_allreduce: 12.83 | step: 18.87

  6%|▌         | 2274/41250 [5:29:20<94:42:28,  8.75s/it]
                                                         
{'loss': 0.0853, 'grad_norm': 0.7573590278625488, 'learning_rate': 3.993386988805617e-05, 'epoch': 0.55}

  6%|▌         | 2274/41250 [5:29:20<94:42:28,  8.75s/it][2025-04-25 13:27:03,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:27:03,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.39 | bwd_microstep: 5791.73 | bwd_inner_microstep: 5659.38 | bwd_allreduce_microstep: 132.30 | step_microstep: 18.52
[2025-04-25 13:27:03,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.39 | bwd: 5791.75 | bwd_inner: 5659.38 | bwd_allreduce: 132.32 | step: 18.52

  6%|▌         | 2275/41250 [5:29:29<94:33:01,  8.73s/it]
                                                         
{'loss': 0.5143, 'grad_norm': 2.873652935028076, 'learning_rate': 3.9933742232703696e-05, 'epoch': 0.55}

  6%|▌         | 2275/41250 [5:29:29<94:33:01,  8.73s/it][2025-04-25 13:27:12,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:27:12,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5719.12 | bwd_inner_microstep: 5706.44 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.73
[2025-04-25 13:27:12,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5719.13 | bwd_inner: 5706.44 | bwd_allreduce: 12.65 | step: 18.73

  6%|▌         | 2276/41250 [5:29:37<94:17:52,  8.71s/it]
                                                         
{'loss': 0.2671, 'grad_norm': 2.053055763244629, 'learning_rate': 3.993361445446363e-05, 'epoch': 0.55}

  6%|▌         | 2276/41250 [5:29:37<94:17:52,  8.71s/it][2025-04-25 13:27:21,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 13:27:21,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5720.01 | bwd_inner_microstep: 5663.40 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.69
[2025-04-25 13:27:21,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5720.02 | bwd_inner: 5663.40 | bwd_allreduce: 56.58 | step: 18.69

  6%|▌         | 2277/41250 [5:29:46<94:02:10,  8.69s/it]
                                                         
{'loss': 0.1755, 'grad_norm': 1.2468461990356445, 'learning_rate': 3.993348655333674e-05, 'epoch': 0.55}

  6%|▌         | 2277/41250 [5:29:46<94:02:10,  8.69s/it][2025-04-25 13:27:29,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.07 | optimizer_step: 1.00
[2025-04-25 13:27:29,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.16 | bwd_microstep: 5718.18 | bwd_inner_microstep: 5705.27 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.61
[2025-04-25 13:27:29,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.17 | bwd: 5718.19 | bwd_inner: 5705.27 | bwd_allreduce: 12.88 | step: 19.61

  6%|▌         | 2278/41250 [5:29:55<93:55:48,  8.68s/it]
                                                         
{'loss': 0.1906, 'grad_norm': 2.4198434352874756, 'learning_rate': 3.9933358529323836e-05, 'epoch': 0.55}

  6%|▌         | 2278/41250 [5:29:55<93:55:48,  8.68s/it][2025-04-25 13:27:38,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:27:38,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.04 | bwd_microstep: 5704.67 | bwd_inner_microstep: 5648.43 | bwd_allreduce_microstep: 56.20 | step_microstep: 18.41
[2025-04-25 13:27:38,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.04 | bwd: 5704.69 | bwd_inner: 5648.43 | bwd_allreduce: 56.21 | step: 18.41

  6%|▌         | 2279/41250 [5:30:03<93:43:49,  8.66s/it]
                                                         
{'loss': 0.2014, 'grad_norm': 0.9489507675170898, 'learning_rate': 3.993323038242569e-05, 'epoch': 0.55}

  6%|▌         | 2279/41250 [5:30:03<93:43:49,  8.66s/it][2025-04-25 13:27:47,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 13:27:47,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.58 | bwd_microstep: 5785.90 | bwd_inner_microstep: 5711.18 | bwd_allreduce_microstep: 74.68 | step_microstep: 18.54
[2025-04-25 13:27:47,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.58 | bwd: 5785.91 | bwd_inner: 5711.18 | bwd_allreduce: 74.69 | step: 18.54

  6%|▌         | 2280/41250 [5:30:12<93:56:13,  8.68s/it]
                                                         
{'loss': 0.3409, 'grad_norm': 2.233158826828003, 'learning_rate': 3.9933102112643094e-05, 'epoch': 0.55}

  6%|▌         | 2280/41250 [5:30:12<93:56:13,  8.68s/it][2025-04-25 13:27:55,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:27:55,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.09 | bwd_microstep: 5726.26 | bwd_inner_microstep: 5713.62 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.83
[2025-04-25 13:27:55,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.09 | bwd: 5726.27 | bwd_inner: 5713.62 | bwd_allreduce: 12.61 | step: 18.83

  6%|▌         | 2281/41250 [5:30:21<93:53:22,  8.67s/it]
                                                         
{'loss': 0.0333, 'grad_norm': 0.45617109537124634, 'learning_rate': 3.993297371997686e-05, 'epoch': 0.55}

  6%|▌         | 2281/41250 [5:30:21<93:53:22,  8.67s/it][2025-04-25 13:28:04,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:28:04,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5741.82 | bwd_inner_microstep: 5689.57 | bwd_allreduce_microstep: 52.20 | step_microstep: 18.83
[2025-04-25 13:28:04,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5741.84 | bwd_inner: 5689.57 | bwd_allreduce: 52.22 | step: 18.83

  6%|▌         | 2282/41250 [5:30:29<93:51:46,  8.67s/it]
                                                         
{'loss': 0.1699, 'grad_norm': 2.0914902687072754, 'learning_rate': 3.993284520442775e-05, 'epoch': 0.55}

  6%|▌         | 2282/41250 [5:30:29<93:51:46,  8.67s/it][2025-04-25 13:28:13,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 13:28:13,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.52 | bwd_microstep: 5714.25 | bwd_inner_microstep: 5701.35 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.14
[2025-04-25 13:28:13,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.52 | bwd: 5714.27 | bwd_inner: 5701.35 | bwd_allreduce: 12.88 | step: 19.14

  6%|▌         | 2283/41250 [5:30:38<93:50:32,  8.67s/it]
                                                         
{'loss': 0.258, 'grad_norm': 1.66726815700531, 'learning_rate': 3.993271656599658e-05, 'epoch': 0.55}

  6%|▌         | 2283/41250 [5:30:38<93:50:32,  8.67s/it][2025-04-25 13:28:21,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:28:21,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.00 | bwd_microstep: 5674.27 | bwd_inner_microstep: 5661.69 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.88
[2025-04-25 13:28:21,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.00 | bwd: 5674.29 | bwd_inner: 5661.69 | bwd_allreduce: 12.55 | step: 18.88

  6%|▌         | 2284/41250 [5:30:47<93:38:39,  8.65s/it]
                                                         
{'loss': 0.0956, 'grad_norm': 1.039298176765442, 'learning_rate': 3.993258780468412e-05, 'epoch': 0.55}

  6%|▌         | 2284/41250 [5:30:47<93:38:39,  8.65s/it][2025-04-25 13:28:30,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.22 | optimizer_step: 1.01
[2025-04-25 13:28:30,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.93 | bwd_microstep: 5778.30 | bwd_inner_microstep: 5694.46 | bwd_allreduce_microstep: 83.80 | step_microstep: 19.23
[2025-04-25 13:28:30,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.93 | bwd: 5778.31 | bwd_inner: 5694.46 | bwd_allreduce: 83.82 | step: 19.23

  6%|▌         | 2285/41250 [5:30:55<93:52:32,  8.67s/it]
                                                         
{'loss': 0.1079, 'grad_norm': 1.1241439580917358, 'learning_rate': 3.9932458920491184e-05, 'epoch': 0.55}

  6%|▌         | 2285/41250 [5:30:55<93:52:32,  8.67s/it][2025-04-25 13:28:39,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:28:39,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.10 | bwd_microstep: 5767.99 | bwd_inner_microstep: 5641.11 | bwd_allreduce_microstep: 126.83 | step_microstep: 18.56
[2025-04-25 13:28:39,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.10 | bwd: 5768.00 | bwd_inner: 5641.11 | bwd_allreduce: 126.85 | step: 18.56

  6%|▌         | 2286/41250 [5:31:04<93:52:38,  8.67s/it]
                                                         
{'loss': 0.1155, 'grad_norm': 1.1215683221817017, 'learning_rate': 3.993232991341856e-05, 'epoch': 0.55}

  6%|▌         | 2286/41250 [5:31:04<93:52:38,  8.67s/it][2025-04-25 13:28:47,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 13:28:47,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.27 | bwd_microstep: 5768.78 | bwd_inner_microstep: 5660.82 | bwd_allreduce_microstep: 107.92 | step_microstep: 18.55
[2025-04-25 13:28:47,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.27 | bwd: 5768.80 | bwd_inner: 5660.82 | bwd_allreduce: 107.94 | step: 18.55

  6%|▌         | 2287/41250 [5:31:13<93:55:26,  8.68s/it]
                                                         
{'loss': 0.2476, 'grad_norm': 1.3858131170272827, 'learning_rate': 3.9932200783467045e-05, 'epoch': 0.55}

  6%|▌         | 2287/41250 [5:31:13<93:55:26,  8.68s/it][2025-04-25 13:28:56,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:28:56,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.84 | bwd_microstep: 5733.89 | bwd_inner_microstep: 5706.41 | bwd_allreduce_microstep: 27.44 | step_microstep: 18.70
[2025-04-25 13:28:56,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.84 | bwd: 5733.91 | bwd_inner: 5706.41 | bwd_allreduce: 27.46 | step: 18.70

  6%|▌         | 2288/41250 [5:31:21<93:55:05,  8.68s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 1.2807127237319946, 'learning_rate': 3.993207153063743e-05, 'epoch': 0.55}

  6%|▌         | 2288/41250 [5:31:21<93:55:05,  8.68s/it][2025-04-25 13:29:05,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 13:29:05,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.93 | bwd_microstep: 5675.08 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 22.58 | step_microstep: 18.63
[2025-04-25 13:29:05,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.93 | bwd: 5675.09 | bwd_inner: 5652.46 | bwd_allreduce: 22.59 | step: 18.63

  6%|▌         | 2289/41250 [5:31:30<93:38:25,  8.65s/it]
                                                         
{'loss': 0.1936, 'grad_norm': 2.687016248703003, 'learning_rate': 3.993194215493051e-05, 'epoch': 0.55}

  6%|▌         | 2289/41250 [5:31:30<93:38:25,  8.65s/it][2025-04-25 13:29:13,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 13:29:13,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.70 | bwd_microstep: 5829.26 | bwd_inner_microstep: 5656.82 | bwd_allreduce_microstep: 172.40 | step_microstep: 18.41
[2025-04-25 13:29:13,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.70 | bwd: 5829.28 | bwd_inner: 5656.82 | bwd_allreduce: 172.41 | step: 18.42

  6%|▌         | 2290/41250 [5:31:39<93:57:58,  8.68s/it]
                                                         
{'loss': 0.1811, 'grad_norm': 1.189864158630371, 'learning_rate': 3.99318126563471e-05, 'epoch': 0.56}

  6%|▌         | 2290/41250 [5:31:39<93:57:58,  8.68s/it][2025-04-25 13:29:22,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:29:22,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.56 | bwd_microstep: 5672.54 | bwd_inner_microstep: 5656.81 | bwd_allreduce_microstep: 15.69 | step_microstep: 18.62
[2025-04-25 13:29:22,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.56 | bwd: 5672.55 | bwd_inner: 5656.81 | bwd_allreduce: 15.70 | step: 18.62

  6%|▌         | 2291/41250 [5:31:47<93:38:59,  8.65s/it]
                                                         
{'loss': 0.2223, 'grad_norm': 1.9331995248794556, 'learning_rate': 3.993168303488797e-05, 'epoch': 0.56}

  6%|▌         | 2291/41250 [5:31:47<93:38:59,  8.65s/it][2025-04-25 13:29:31,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 13:29:31,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.06 | bwd_microstep: 5735.98 | bwd_inner_microstep: 5691.37 | bwd_allreduce_microstep: 44.57 | step_microstep: 19.30
[2025-04-25 13:29:31,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.06 | bwd: 5735.99 | bwd_inner: 5691.37 | bwd_allreduce: 44.58 | step: 19.30

  6%|▌         | 2292/41250 [5:31:56<93:43:48,  8.66s/it]
                                                         
{'loss': 0.3419, 'grad_norm': 1.815209150314331, 'learning_rate': 3.9931553290553945e-05, 'epoch': 0.56}

  6%|▌         | 2292/41250 [5:31:56<93:43:48,  8.66s/it][2025-04-25 13:29:39,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:29:39,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.84 | bwd_microstep: 5711.54 | bwd_inner_microstep: 5698.77 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.57
[2025-04-25 13:29:39,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.84 | bwd: 5711.55 | bwd_inner: 5698.77 | bwd_allreduce: 12.75 | step: 18.58

  6%|▌         | 2293/41250 [5:32:05<93:40:11,  8.66s/it]
                                                         
{'loss': 0.2236, 'grad_norm': 1.9375834465026855, 'learning_rate': 3.9931423423345805e-05, 'epoch': 0.56}

  6%|▌         | 2293/41250 [5:32:05<93:40:11,  8.66s/it][2025-04-25 13:29:48,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:29:48,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.92 | bwd_microstep: 5696.62 | bwd_inner_microstep: 5683.66 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.00
[2025-04-25 13:29:48,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.92 | bwd: 5696.63 | bwd_inner: 5683.66 | bwd_allreduce: 12.93 | step: 19.00

  6%|▌         | 2294/41250 [5:32:13<93:35:08,  8.65s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.2596348524093628, 'learning_rate': 3.993129343326437e-05, 'epoch': 0.56}

  6%|▌         | 2294/41250 [5:32:13<93:35:08,  8.65s/it][2025-04-25 13:29:56,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:29:56,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.01 | bwd_microstep: 5690.96 | bwd_inner_microstep: 5648.93 | bwd_allreduce_microstep: 41.99 | step_microstep: 18.34
[2025-04-25 13:29:56,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.01 | bwd: 5690.97 | bwd_inner: 5648.93 | bwd_allreduce: 42.00 | step: 18.34

  6%|▌         | 2295/41250 [5:32:22<93:26:38,  8.64s/it]
                                                         
{'loss': 0.2003, 'grad_norm': 1.4178597927093506, 'learning_rate': 3.993116332031043e-05, 'epoch': 0.56}

  6%|▌         | 2295/41250 [5:32:22<93:26:38,  8.64s/it][2025-04-25 13:30:05,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:30:05,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.45 | bwd_microstep: 5710.11 | bwd_inner_microstep: 5697.40 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.29
[2025-04-25 13:30:05,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.45 | bwd: 5710.13 | bwd_inner: 5697.40 | bwd_allreduce: 12.68 | step: 18.29

  6%|▌         | 2296/41250 [5:32:30<93:29:09,  8.64s/it]
                                                         
{'loss': 0.1734, 'grad_norm': 2.069836378097534, 'learning_rate': 3.993103308448478e-05, 'epoch': 0.56}

  6%|▌         | 2296/41250 [5:32:30<93:29:09,  8.64s/it][2025-04-25 13:30:14,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:30:14,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.82 | bwd_microstep: 5747.23 | bwd_inner_microstep: 5642.97 | bwd_allreduce_microstep: 104.22 | step_microstep: 18.41
[2025-04-25 13:30:14,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.82 | bwd: 5747.25 | bwd_inner: 5642.96 | bwd_allreduce: 104.24 | step: 18.42

  6%|▌         | 2297/41250 [5:32:39<93:32:08,  8.64s/it]
                                                         
{'loss': 0.1269, 'grad_norm': 2.6620376110076904, 'learning_rate': 3.993090272578824e-05, 'epoch': 0.56}

  6%|▌         | 2297/41250 [5:32:39<93:32:08,  8.64s/it][2025-04-25 13:30:22,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:30:22,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.95 | bwd_microstep: 5712.41 | bwd_inner_microstep: 5699.61 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.27
[2025-04-25 13:30:22,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.95 | bwd: 5712.42 | bwd_inner: 5699.61 | bwd_allreduce: 12.77 | step: 19.27

  6%|▌         | 2298/41250 [5:32:48<93:32:52,  8.65s/it]
                                                         
{'loss': 0.066, 'grad_norm': 0.9245588183403015, 'learning_rate': 3.993077224422159e-05, 'epoch': 0.56}

  6%|▌         | 2298/41250 [5:32:48<93:32:52,  8.65s/it][2025-04-25 13:30:31,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 13:30:31,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.58 | bwd_microstep: 5736.25 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 36.51 | step_microstep: 18.55
[2025-04-25 13:30:31,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.58 | bwd: 5736.27 | bwd_inner: 5699.69 | bwd_allreduce: 36.53 | step: 18.55

  6%|▌         | 2299/41250 [5:32:56<93:39:31,  8.66s/it]
                                                         
{'loss': 0.1615, 'grad_norm': 0.8816692233085632, 'learning_rate': 3.993064163978567e-05, 'epoch': 0.56}

  6%|▌         | 2299/41250 [5:32:56<93:39:31,  8.66s/it][2025-04-25 13:30:40,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 13:30:40,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.89 | bwd_microstep: 5756.76 | bwd_inner_microstep: 5674.67 | bwd_allreduce_microstep: 82.05 | step_microstep: 18.55
[2025-04-25 13:30:40,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.89 | bwd: 5756.77 | bwd_inner: 5674.67 | bwd_allreduce: 82.06 | step: 18.56

  6%|▌         | 2300/41250 [5:33:05<93:44:06,  8.66s/it]
                                                         
{'loss': 0.0928, 'grad_norm': 0.7389233112335205, 'learning_rate': 3.9930510912481246e-05, 'epoch': 0.56}

  6%|▌         | 2300/41250 [5:33:05<93:44:06,  8.66s/it][2025-04-25 13:30:48,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 13:30:48,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.18 | bwd_microstep: 5708.08 | bwd_inner_microstep: 5695.48 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.79
[2025-04-25 13:30:48,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.18 | bwd: 5708.09 | bwd_inner: 5695.48 | bwd_allreduce: 12.57 | step: 18.79

  6%|▌         | 2301/41250 [5:33:14<93:41:02,  8.66s/it]
                                                         
{'loss': 0.3258, 'grad_norm': 1.862121820449829, 'learning_rate': 3.993038006230916e-05, 'epoch': 0.56}

  6%|▌         | 2301/41250 [5:33:14<93:41:02,  8.66s/it][2025-04-25 13:30:57,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:30:57,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.45 | bwd_microstep: 5663.85 | bwd_inner_microstep: 5651.11 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.55
[2025-04-25 13:30:57,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.45 | bwd: 5663.86 | bwd_inner: 5651.11 | bwd_allreduce: 12.72 | step: 18.55

  6%|▌         | 2302/41250 [5:33:22<93:27:07,  8.64s/it]
                                                         
{'loss': 0.2969, 'grad_norm': 2.0849862098693848, 'learning_rate': 3.9930249089270185e-05, 'epoch': 0.56}

  6%|▌         | 2302/41250 [5:33:22<93:27:07,  8.64s/it][2025-04-25 13:31:06,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:31:06,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.91 | bwd_microstep: 5732.44 | bwd_inner_microstep: 5652.99 | bwd_allreduce_microstep: 79.41 | step_microstep: 18.47
[2025-04-25 13:31:06,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.91 | bwd: 5732.45 | bwd_inner: 5652.99 | bwd_allreduce: 79.43 | step: 18.47

  6%|▌         | 2303/41250 [5:33:31<93:29:04,  8.64s/it]
                                                         
{'loss': 0.1758, 'grad_norm': 1.1819026470184326, 'learning_rate': 3.993011799336515e-05, 'epoch': 0.56}

  6%|▌         | 2303/41250 [5:33:31<93:29:04,  8.64s/it][2025-04-25 13:31:14,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 13:31:14,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.24 | bwd_microstep: 5700.34 | bwd_inner_microstep: 5687.59 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.91
[2025-04-25 13:31:14,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.24 | bwd: 5700.36 | bwd_inner: 5687.59 | bwd_allreduce: 12.73 | step: 18.91

  6%|▌         | 2304/41250 [5:33:40<93:28:29,  8.64s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.7375041246414185, 'learning_rate': 3.992998677459486e-05, 'epoch': 0.56}

  6%|▌         | 2304/41250 [5:33:40<93:28:29,  8.64s/it][2025-04-25 13:31:23,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:31:23,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.86 | bwd_microstep: 5857.11 | bwd_inner_microstep: 5662.66 | bwd_allreduce_microstep: 194.40 | step_microstep: 18.58
[2025-04-25 13:31:23,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.86 | bwd: 5857.12 | bwd_inner: 5662.66 | bwd_allreduce: 194.42 | step: 18.58

  6%|▌         | 2305/41250 [5:33:48<93:53:37,  8.68s/it]
                                                         
{'loss': 0.102, 'grad_norm': 1.2998546361923218, 'learning_rate': 3.992985543296012e-05, 'epoch': 0.56}

  6%|▌         | 2305/41250 [5:33:48<93:53:37,  8.68s/it][2025-04-25 13:31:32,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:31:32,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.77 | bwd_microstep: 5740.80 | bwd_inner_microstep: 5676.55 | bwd_allreduce_microstep: 64.20 | step_microstep: 18.65
[2025-04-25 13:31:32,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.77 | bwd: 5740.81 | bwd_inner: 5676.55 | bwd_allreduce: 64.22 | step: 18.66

  6%|▌         | 2306/41250 [5:33:57<93:50:53,  8.68s/it]
                                                         
{'loss': 0.1732, 'grad_norm': 1.712808609008789, 'learning_rate': 3.992972396846174e-05, 'epoch': 0.56}

  6%|▌         | 2306/41250 [5:33:57<93:50:53,  8.68s/it][2025-04-25 13:31:40,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:31:40,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.16 | bwd_microstep: 5704.33 | bwd_inner_microstep: 5691.42 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.64
[2025-04-25 13:31:40,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.16 | bwd: 5704.35 | bwd_inner: 5691.42 | bwd_allreduce: 12.89 | step: 18.64

  6%|▌         | 2307/41250 [5:34:06<93:44:14,  8.67s/it]
                                                         
{'loss': 0.2267, 'grad_norm': 2.1907670497894287, 'learning_rate': 3.9929592381100536e-05, 'epoch': 0.56}

  6%|▌         | 2307/41250 [5:34:06<93:44:14,  8.67s/it][2025-04-25 13:31:49,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:31:49,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5724.62 | bwd_inner_microstep: 5711.86 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.73
[2025-04-25 13:31:49,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5724.63 | bwd_inner: 5711.86 | bwd_allreduce: 12.73 | step: 18.73

  6%|▌         | 2308/41250 [5:34:14<93:41:28,  8.66s/it]
                                                         
{'loss': 0.2335, 'grad_norm': 1.8210331201553345, 'learning_rate': 3.992946067087731e-05, 'epoch': 0.56}

  6%|▌         | 2308/41250 [5:34:14<93:41:28,  8.66s/it][2025-04-25 13:31:58,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:31:58,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5753.79 | bwd_inner_microstep: 5655.65 | bwd_allreduce_microstep: 98.10 | step_microstep: 18.38
[2025-04-25 13:31:58,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.09 | bwd: 5753.81 | bwd_inner: 5655.65 | bwd_allreduce: 98.12 | step: 18.39

  6%|▌         | 2309/41250 [5:34:23<93:40:45,  8.66s/it]
                                                         
{'loss': 0.1448, 'grad_norm': 1.189778208732605, 'learning_rate': 3.992932883779288e-05, 'epoch': 0.56}

  6%|▌         | 2309/41250 [5:34:23<93:40:45,  8.66s/it][2025-04-25 13:32:06,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:32:06,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.89 | bwd_microstep: 5778.40 | bwd_inner_microstep: 5657.16 | bwd_allreduce_microstep: 121.20 | step_microstep: 18.42
[2025-04-25 13:32:06,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.89 | bwd: 5778.42 | bwd_inner: 5657.16 | bwd_allreduce: 121.22 | step: 18.42

  6%|▌         | 2310/41250 [5:34:32<93:45:38,  8.67s/it]
                                                         
{'loss': 0.2811, 'grad_norm': 3.8059825897216797, 'learning_rate': 3.992919688184807e-05, 'epoch': 0.56}

  6%|▌         | 2310/41250 [5:34:32<93:45:38,  8.67s/it][2025-04-25 13:32:15,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 13:32:15,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.53 | bwd_microstep: 5692.98 | bwd_inner_microstep: 5649.26 | bwd_allreduce_microstep: 43.69 | step_microstep: 18.55
[2025-04-25 13:32:15,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.53 | bwd: 5693.00 | bwd_inner: 5649.26 | bwd_allreduce: 43.70 | step: 18.55

  6%|▌         | 2311/41250 [5:34:40<93:31:08,  8.65s/it]
                                                         
{'loss': 0.1842, 'grad_norm': 1.3603450059890747, 'learning_rate': 3.9929064803043665e-05, 'epoch': 0.56}

  6%|▌         | 2311/41250 [5:34:40<93:31:08,  8.65s/it][2025-04-25 13:32:24,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:32:24,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5682.06 | bwd_inner_microstep: 5647.63 | bwd_allreduce_microstep: 34.39 | step_microstep: 18.42
[2025-04-25 13:32:24,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5682.08 | bwd_inner: 5647.63 | bwd_allreduce: 34.41 | step: 18.42

  6%|▌         | 2312/41250 [5:34:49<93:20:36,  8.63s/it]
                                                         
{'loss': 0.2266, 'grad_norm': 2.1649537086486816, 'learning_rate': 3.9928932601380495e-05, 'epoch': 0.56}

  6%|▌         | 2312/41250 [5:34:49<93:20:36,  8.63s/it][2025-04-25 13:32:32,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 13:32:32,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.78 | bwd_microstep: 5674.06 | bwd_inner_microstep: 5640.15 | bwd_allreduce_microstep: 33.86 | step_microstep: 18.35
[2025-04-25 13:32:32,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.78 | bwd: 5674.07 | bwd_inner: 5640.15 | bwd_allreduce: 33.88 | step: 18.36

  6%|▌         | 2313/41250 [5:34:58<93:11:29,  8.62s/it]
                                                         
{'loss': 0.2097, 'grad_norm': 1.7099299430847168, 'learning_rate': 3.992880027685939e-05, 'epoch': 0.56}

  6%|▌         | 2313/41250 [5:34:58<93:11:29,  8.62s/it][2025-04-25 13:32:41,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.08 | optimizer_step: 1.00
[2025-04-25 13:32:41,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.69 | bwd_microstep: 5685.08 | bwd_inner_microstep: 5671.75 | bwd_allreduce_microstep: 13.28 | step_microstep: 19.53
[2025-04-25 13:32:41,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.69 | bwd: 5685.10 | bwd_inner: 5671.75 | bwd_allreduce: 13.30 | step: 19.53

  6%|▌         | 2314/41250 [5:35:06<93:11:24,  8.62s/it]
                                                         
{'loss': 0.1564, 'grad_norm': 1.2328451871871948, 'learning_rate': 3.9928667829481136e-05, 'epoch': 0.56}

  6%|▌         | 2314/41250 [5:35:06<93:11:24,  8.62s/it][2025-04-25 13:32:49,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 13:32:49,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5734.74 | bwd_inner_microstep: 5677.61 | bwd_allreduce_microstep: 57.09 | step_microstep: 18.66
[2025-04-25 13:32:49,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5734.75 | bwd_inner: 5677.61 | bwd_allreduce: 57.10 | step: 18.66

  6%|▌         | 2315/41250 [5:35:15<93:21:13,  8.63s/it]
                                                         
{'loss': 0.2013, 'grad_norm': 1.467764139175415, 'learning_rate': 3.9928535259246575e-05, 'epoch': 0.56}

  6%|▌         | 2315/41250 [5:35:15<93:21:13,  8.63s/it][2025-04-25 13:32:58,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 13:32:58,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5705.66 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.14
[2025-04-25 13:32:58,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5705.68 | bwd_inner: 5692.86 | bwd_allreduce: 12.77 | step: 19.15

  6%|▌         | 2316/41250 [5:35:23<93:22:58,  8.63s/it]
                                                         
{'loss': 0.1837, 'grad_norm': 1.445773959159851, 'learning_rate': 3.9928402566156504e-05, 'epoch': 0.56}

  6%|▌         | 2316/41250 [5:35:23<93:22:58,  8.63s/it][2025-04-25 13:33:07,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:33:07,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.79 | bwd_microstep: 5758.44 | bwd_inner_microstep: 5647.62 | bwd_allreduce_microstep: 110.77 | step_microstep: 19.34
[2025-04-25 13:33:07,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.79 | bwd: 5758.45 | bwd_inner: 5647.62 | bwd_allreduce: 110.79 | step: 19.34

  6%|▌         | 2317/41250 [5:35:32<93:30:51,  8.65s/it]
                                                         
{'loss': 0.2541, 'grad_norm': 1.6536792516708374, 'learning_rate': 3.992826975021176e-05, 'epoch': 0.56}

  6%|▌         | 2317/41250 [5:35:32<93:30:51,  8.65s/it][2025-04-25 13:33:15,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:33:15,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.53 | bwd_microstep: 5751.96 | bwd_inner_microstep: 5647.05 | bwd_allreduce_microstep: 104.86 | step_microstep: 18.80
[2025-04-25 13:33:15,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.53 | bwd: 5751.97 | bwd_inner: 5647.05 | bwd_allreduce: 104.88 | step: 18.80

  6%|▌         | 2318/41250 [5:35:41<93:31:16,  8.65s/it]
                                                         
{'loss': 0.2224, 'grad_norm': 1.954593539237976, 'learning_rate': 3.9928136811413145e-05, 'epoch': 0.56}

  6%|▌         | 2318/41250 [5:35:41<93:31:16,  8.65s/it][2025-04-25 13:33:24,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:33:24,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.91 | bwd_microstep: 5727.80 | bwd_inner_microstep: 5698.65 | bwd_allreduce_microstep: 29.10 | step_microstep: 18.67
[2025-04-25 13:33:24,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.91 | bwd: 5727.81 | bwd_inner: 5698.65 | bwd_allreduce: 29.12 | step: 18.67

  6%|▌         | 2319/41250 [5:35:49<93:32:55,  8.65s/it]
                                                         
{'loss': 0.1979, 'grad_norm': 1.8501460552215576, 'learning_rate': 3.992800374976149e-05, 'epoch': 0.56}

  6%|▌         | 2319/41250 [5:35:49<93:32:55,  8.65s/it][2025-04-25 13:33:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:33:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.89 | bwd_microstep: 5804.73 | bwd_inner_microstep: 5791.98 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.65
[2025-04-25 13:33:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.89 | bwd: 5804.74 | bwd_inner: 5791.98 | bwd_allreduce: 12.71 | step: 18.66

  6%|▌         | 2320/41250 [5:35:58<93:59:12,  8.69s/it]
                                                         
{'loss': 0.1351, 'grad_norm': 1.1510494947433472, 'learning_rate': 3.992787056525761e-05, 'epoch': 0.56}

  6%|▌         | 2320/41250 [5:35:58<93:59:12,  8.69s/it][2025-04-25 13:33:42,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.14 | optimizer_step: 1.03
[2025-04-25 13:33:42,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.52 | bwd_microstep: 5728.27 | bwd_inner_microstep: 5698.02 | bwd_allreduce_microstep: 30.19 | step_microstep: 20.00
[2025-04-25 13:33:42,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.52 | bwd: 5728.28 | bwd_inner: 5698.02 | bwd_allreduce: 30.22 | step: 20.00

  6%|▌         | 2321/41250 [5:36:07<93:52:59,  8.68s/it]
                                                         
{'loss': 0.073, 'grad_norm': 1.0685030221939087, 'learning_rate': 3.992773725790233e-05, 'epoch': 0.56}

  6%|▌         | 2321/41250 [5:36:07<93:52:59,  8.68s/it][2025-04-25 13:33:50,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:33:50,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.67 | bwd_microstep: 5764.99 | bwd_inner_microstep: 5658.09 | bwd_allreduce_microstep: 106.85 | step_microstep: 18.78
[2025-04-25 13:33:50,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.67 | bwd: 5765.00 | bwd_inner: 5658.09 | bwd_allreduce: 106.87 | step: 18.78

  6%|▌         | 2322/41250 [5:36:16<93:52:30,  8.68s/it]
                                                         
{'loss': 0.3462, 'grad_norm': 1.9794758558273315, 'learning_rate': 3.992760382769647e-05, 'epoch': 0.56}

  6%|▌         | 2322/41250 [5:36:16<93:52:30,  8.68s/it][2025-04-25 13:33:59,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.21 | optimizer_step: 0.93
[2025-04-25 13:33:59,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.29 | bwd_microstep: 5791.48 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 133.79 | step_microstep: 19.04
[2025-04-25 13:33:59,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.29 | bwd: 5791.50 | bwd_inner: 5657.65 | bwd_allreduce: 133.81 | step: 19.05

  6%|▌         | 2323/41250 [5:36:24<93:57:27,  8.69s/it]
                                                         
{'loss': 0.2354, 'grad_norm': 1.2093942165374756, 'learning_rate': 3.992747027464085e-05, 'epoch': 0.56}

  6%|▌         | 2323/41250 [5:36:24<93:57:27,  8.69s/it][2025-04-25 13:34:08,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 1.00
[2025-04-25 13:34:08,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.85 | bwd_microstep: 5738.82 | bwd_inner_microstep: 5693.69 | bwd_allreduce_microstep: 45.08 | step_microstep: 19.50
[2025-04-25 13:34:08,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.85 | bwd: 5738.84 | bwd_inner: 5693.69 | bwd_allreduce: 45.10 | step: 19.50

  6%|▌         | 2324/41250 [5:36:33<93:54:37,  8.69s/it]
                                                         
{'loss': 0.1562, 'grad_norm': 1.186671257019043, 'learning_rate': 3.9927336598736296e-05, 'epoch': 0.56}

  6%|▌         | 2324/41250 [5:36:33<93:54:37,  8.69s/it][2025-04-25 13:34:16,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:34:16,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5761.60 | bwd_inner_microstep: 5647.06 | bwd_allreduce_microstep: 114.48 | step_microstep: 18.94
[2025-04-25 13:34:16,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5761.61 | bwd_inner: 5647.06 | bwd_allreduce: 114.50 | step: 18.94

  6%|▌         | 2325/41250 [5:36:42<93:52:39,  8.68s/it]
                                                         
{'loss': 0.058, 'grad_norm': 0.6707267165184021, 'learning_rate': 3.9927202799983626e-05, 'epoch': 0.56}

  6%|▌         | 2325/41250 [5:36:42<93:52:39,  8.68s/it][2025-04-25 13:34:25,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.99
[2025-04-25 13:34:25,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.61 | bwd_microstep: 5742.66 | bwd_inner_microstep: 5661.84 | bwd_allreduce_microstep: 80.78 | step_microstep: 19.41
[2025-04-25 13:34:25,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.61 | bwd: 5742.68 | bwd_inner: 5661.84 | bwd_allreduce: 80.80 | step: 19.41

  6%|▌         | 2326/41250 [5:36:50<93:48:28,  8.68s/it]
                                                         
{'loss': 0.2189, 'grad_norm': 1.552760362625122, 'learning_rate': 3.9927068878383675e-05, 'epoch': 0.56}

  6%|▌         | 2326/41250 [5:36:50<93:48:28,  8.68s/it][2025-04-25 13:34:34,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 13:34:34,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.78 | bwd_microstep: 5742.14 | bwd_inner_microstep: 5709.77 | bwd_allreduce_microstep: 32.32 | step_microstep: 19.22
[2025-04-25 13:34:34,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.78 | bwd: 5742.16 | bwd_inner: 5709.77 | bwd_allreduce: 32.34 | step: 19.22

  6%|▌         | 2327/41250 [5:36:59<93:48:55,  8.68s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.3223497867584229, 'learning_rate': 3.992693483393727e-05, 'epoch': 0.56}

  6%|▌         | 2327/41250 [5:36:59<93:48:55,  8.68s/it][2025-04-25 13:34:42,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:34:42,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.36 | bwd_microstep: 5775.42 | bwd_inner_microstep: 5673.98 | bwd_allreduce_microstep: 101.40 | step_microstep: 18.61
[2025-04-25 13:34:42,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.36 | bwd: 5775.44 | bwd_inner: 5673.98 | bwd_allreduce: 101.42 | step: 18.61

  6%|▌         | 2328/41250 [5:37:08<93:51:11,  8.68s/it]
                                                         
{'loss': 0.2947, 'grad_norm': 2.7273335456848145, 'learning_rate': 3.9926800666645225e-05, 'epoch': 0.56}

  6%|▌         | 2328/41250 [5:37:08<93:51:11,  8.68s/it][2025-04-25 13:34:51,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:34:51,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.76 | bwd_microstep: 5788.21 | bwd_inner_microstep: 5666.06 | bwd_allreduce_microstep: 122.10 | step_microstep: 18.97
[2025-04-25 13:34:51,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.76 | bwd: 5788.22 | bwd_inner: 5666.06 | bwd_allreduce: 122.12 | step: 18.97

  6%|▌         | 2329/41250 [5:37:16<93:56:24,  8.69s/it]
                                                         
{'loss': 0.0786, 'grad_norm': 0.5829521417617798, 'learning_rate': 3.992666637650837e-05, 'epoch': 0.56}

  6%|▌         | 2329/41250 [5:37:16<93:56:24,  8.69s/it][2025-04-25 13:35:00,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:35:00,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.50 | bwd_microstep: 5706.60 | bwd_inner_microstep: 5664.15 | bwd_allreduce_microstep: 42.40 | step_microstep: 18.89
[2025-04-25 13:35:00,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.50 | bwd: 5706.61 | bwd_inner: 5664.15 | bwd_allreduce: 42.42 | step: 18.89

  6%|▌         | 2330/41250 [5:37:25<93:43:47,  8.67s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 1.4201661348342896, 'learning_rate': 3.992653196352754e-05, 'epoch': 0.56}

  6%|▌         | 2330/41250 [5:37:25<93:43:47,  8.67s/it][2025-04-25 13:35:08,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.23 | optimizer_step: 0.96
[2025-04-25 13:35:08,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.25 | bwd_microstep: 5763.37 | bwd_inner_microstep: 5669.26 | bwd_allreduce_microstep: 94.05 | step_microstep: 19.19
[2025-04-25 13:35:08,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.25 | bwd: 5763.38 | bwd_inner: 5669.26 | bwd_allreduce: 94.08 | step: 19.20

  6%|▌         | 2331/41250 [5:37:34<93:45:59,  8.67s/it]
                                                         
{'loss': 0.1572, 'grad_norm': 1.0563305616378784, 'learning_rate': 3.992639742770356e-05, 'epoch': 0.57}

  6%|▌         | 2331/41250 [5:37:34<93:45:59,  8.67s/it][2025-04-25 13:35:17,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.06 | optimizer_step: 1.10
[2025-04-25 13:35:17,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5718.61 | bwd_inner_microstep: 5705.47 | bwd_allreduce_microstep: 13.09 | step_microstep: 19.25
[2025-04-25 13:35:17,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5718.63 | bwd_inner: 5705.47 | bwd_allreduce: 13.11 | step: 19.26

  6%|▌         | 2332/41250 [5:37:42<93:41:24,  8.67s/it]
                                                         
{'loss': 0.0978, 'grad_norm': 0.8275149464607239, 'learning_rate': 3.992626276903726e-05, 'epoch': 0.57}

  6%|▌         | 2332/41250 [5:37:42<93:41:24,  8.67s/it][2025-04-25 13:35:26,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 13:35:26,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5785.82 | bwd_inner_microstep: 5661.04 | bwd_allreduce_microstep: 124.74 | step_microstep: 18.59
[2025-04-25 13:35:26,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5785.83 | bwd_inner: 5661.04 | bwd_allreduce: 124.75 | step: 18.60

  6%|▌         | 2333/41250 [5:37:51<93:47:27,  8.68s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 1.517319679260254, 'learning_rate': 3.9926127987529466e-05, 'epoch': 0.57}

  6%|▌         | 2333/41250 [5:37:51<93:47:27,  8.68s/it][2025-04-25 13:35:34,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:35:34,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.28 | bwd_microstep: 5778.00 | bwd_inner_microstep: 5675.70 | bwd_allreduce_microstep: 102.26 | step_microstep: 18.79
[2025-04-25 13:35:34,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.28 | bwd: 5778.02 | bwd_inner: 5675.70 | bwd_allreduce: 102.28 | step: 18.80

  6%|▌         | 2334/41250 [5:38:00<93:51:08,  8.68s/it]
                                                         
{'loss': 0.07, 'grad_norm': 1.1899027824401855, 'learning_rate': 3.992599308318101e-05, 'epoch': 0.57}

  6%|▌         | 2334/41250 [5:38:00<93:51:08,  8.68s/it][2025-04-25 13:35:43,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 13:35:43,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.47 | bwd_microstep: 6022.37 | bwd_inner_microstep: 5699.75 | bwd_allreduce_microstep: 322.58 | step_microstep: 18.87
[2025-04-25 13:35:43,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.47 | bwd: 6022.39 | bwd_inner: 5699.75 | bwd_allreduce: 322.59 | step: 18.88

  6%|▌         | 2335/41250 [5:38:09<94:44:02,  8.76s/it]
                                                         
{'loss': 0.111, 'grad_norm': 1.0225976705551147, 'learning_rate': 3.992585805599273e-05, 'epoch': 0.57}

  6%|▌         | 2335/41250 [5:38:09<94:44:02,  8.76s/it][2025-04-25 13:35:52,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:35:52,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.60 | bwd_microstep: 5720.16 | bwd_inner_microstep: 5707.39 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.02
[2025-04-25 13:35:52,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.60 | bwd: 5720.17 | bwd_inner: 5707.39 | bwd_allreduce: 12.74 | step: 19.02

  6%|▌         | 2336/41250 [5:38:17<94:21:59,  8.73s/it]
                                                         
{'loss': 0.2224, 'grad_norm': 1.7320868968963623, 'learning_rate': 3.992572290596545e-05, 'epoch': 0.57}

  6%|▌         | 2336/41250 [5:38:17<94:21:59,  8.73s/it][2025-04-25 13:36:01,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 13:36:01,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.78 | bwd_microstep: 5773.33 | bwd_inner_microstep: 5713.14 | bwd_allreduce_microstep: 60.14 | step_microstep: 19.58
[2025-04-25 13:36:01,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.78 | bwd: 5773.34 | bwd_inner: 5713.14 | bwd_allreduce: 60.16 | step: 19.58

  6%|▌         | 2337/41250 [5:38:26<94:18:02,  8.72s/it]
                                                         
{'loss': 0.1382, 'grad_norm': 1.2593286037445068, 'learning_rate': 3.992558763310002e-05, 'epoch': 0.57}

  6%|▌         | 2337/41250 [5:38:26<94:18:02,  8.72s/it][2025-04-25 13:36:09,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:36:09,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.83 | bwd_microstep: 5801.95 | bwd_inner_microstep: 5789.35 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.81
[2025-04-25 13:36:09,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.83 | bwd: 5801.97 | bwd_inner: 5789.35 | bwd_allreduce: 12.57 | step: 18.82

  6%|▌         | 2338/41250 [5:38:35<94:28:39,  8.74s/it]
                                                         
{'loss': 0.0945, 'grad_norm': 1.1277929544448853, 'learning_rate': 3.992545223739725e-05, 'epoch': 0.57}

  6%|▌         | 2338/41250 [5:38:35<94:28:39,  8.74s/it][2025-04-25 13:36:18,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:36:18,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.39 | bwd_microstep: 5734.51 | bwd_inner_microstep: 5720.06 | bwd_allreduce_microstep: 14.40 | step_microstep: 18.76
[2025-04-25 13:36:18,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.39 | bwd: 5734.52 | bwd_inner: 5720.06 | bwd_allreduce: 14.42 | step: 18.76

  6%|▌         | 2339/41250 [5:38:43<94:15:24,  8.72s/it]
                                                         
{'loss': 0.2901, 'grad_norm': 3.7280633449554443, 'learning_rate': 3.9925316718857996e-05, 'epoch': 0.57}

  6%|▌         | 2339/41250 [5:38:43<94:15:24,  8.72s/it][2025-04-25 13:36:27,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 13:36:27,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5760.97 | bwd_inner_microstep: 5700.74 | bwd_allreduce_microstep: 60.18 | step_microstep: 18.89
[2025-04-25 13:36:27,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5760.99 | bwd_inner: 5700.74 | bwd_allreduce: 60.21 | step: 18.89

  6%|▌         | 2340/41250 [5:38:52<94:09:58,  8.71s/it]
                                                         
{'loss': 0.0685, 'grad_norm': 0.8503223657608032, 'learning_rate': 3.9925181077483074e-05, 'epoch': 0.57}

  6%|▌         | 2340/41250 [5:38:52<94:09:58,  8.71s/it][2025-04-25 13:36:36,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:36:36,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5758.63 | bwd_inner_microstep: 5700.64 | bwd_allreduce_microstep: 57.94 | step_microstep: 18.79
[2025-04-25 13:36:36,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5758.65 | bwd_inner: 5700.64 | bwd_allreduce: 57.96 | step: 18.79

  6%|▌         | 2341/41250 [5:39:01<94:06:07,  8.71s/it]
                                                         
{'loss': 0.1808, 'grad_norm': 2.018305778503418, 'learning_rate': 3.992504531327333e-05, 'epoch': 0.57}

  6%|▌         | 2341/41250 [5:39:01<94:06:07,  8.71s/it][2025-04-25 13:36:44,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:36:44,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.65 | bwd_microstep: 5783.57 | bwd_inner_microstep: 5672.70 | bwd_allreduce_microstep: 110.82 | step_microstep: 18.66
[2025-04-25 13:36:44,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.65 | bwd: 5783.58 | bwd_inner: 5672.70 | bwd_allreduce: 110.84 | step: 18.67

  6%|▌         | 2342/41250 [5:39:10<94:04:23,  8.70s/it]
                                                         
{'loss': 0.088, 'grad_norm': 1.0274206399917603, 'learning_rate': 3.992490942622961e-05, 'epoch': 0.57}

  6%|▌         | 2342/41250 [5:39:10<94:04:23,  8.70s/it][2025-04-25 13:36:53,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-25 13:36:53,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.10 | bwd_microstep: 5889.20 | bwd_inner_microstep: 5875.84 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.34
[2025-04-25 13:36:53,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.10 | bwd: 5889.21 | bwd_inner: 5875.84 | bwd_allreduce: 13.32 | step: 19.33

  6%|▌         | 2343/41250 [5:39:18<94:44:30,  8.77s/it]
                                                         
{'loss': 0.3573, 'grad_norm': 3.573239326477051, 'learning_rate': 3.9924773416352723e-05, 'epoch': 0.57}

  6%|▌         | 2343/41250 [5:39:18<94:44:30,  8.77s/it][2025-04-25 13:37:02,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 13:37:02,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.37 | bwd_microstep: 5856.10 | bwd_inner_microstep: 5704.08 | bwd_allreduce_microstep: 151.97 | step_microstep: 19.44
[2025-04-25 13:37:02,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5856.11 | bwd_inner: 5704.08 | bwd_allreduce: 151.99 | step: 19.44

  6%|▌         | 2344/41250 [5:39:27<94:49:29,  8.77s/it]
                                                         
{'loss': 0.3637, 'grad_norm': 5.14973258972168, 'learning_rate': 3.9924637283643546e-05, 'epoch': 0.57}

  6%|▌         | 2344/41250 [5:39:27<94:49:29,  8.77s/it][2025-04-25 13:37:11,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 13:37:11,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.48 | bwd_microstep: 5782.50 | bwd_inner_microstep: 5703.38 | bwd_allreduce_microstep: 79.08 | step_microstep: 18.96
[2025-04-25 13:37:11,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.48 | bwd: 5782.52 | bwd_inner: 5703.38 | bwd_allreduce: 79.10 | step: 18.96

  6%|▌         | 2345/41250 [5:39:36<94:36:46,  8.75s/it]
                                                         
{'loss': 0.1228, 'grad_norm': 1.0988134145736694, 'learning_rate': 3.9924501028102896e-05, 'epoch': 0.57}

  6%|▌         | 2345/41250 [5:39:36<94:36:46,  8.75s/it][2025-04-25 13:37:19,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:37:19,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.77 | bwd_microstep: 5795.37 | bwd_inner_microstep: 5782.49 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.80
[2025-04-25 13:37:19,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.77 | bwd: 5795.38 | bwd_inner: 5782.48 | bwd_allreduce: 12.86 | step: 18.81

  6%|▌         | 2346/41250 [5:39:45<94:39:35,  8.76s/it]
                                                         
{'loss': 0.1141, 'grad_norm': 1.5015206336975098, 'learning_rate': 3.9924364649731606e-05, 'epoch': 0.57}

  6%|▌         | 2346/41250 [5:39:45<94:39:35,  8.76s/it][2025-04-25 13:37:28,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:37:28,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5866.57 | bwd_inner_microstep: 5694.06 | bwd_allreduce_microstep: 172.47 | step_microstep: 18.69
[2025-04-25 13:37:28,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5866.58 | bwd_inner: 5694.06 | bwd_allreduce: 172.48 | step: 18.69

  6%|▌         | 2347/41250 [5:39:54<94:47:06,  8.77s/it]
                                                         
{'loss': 0.3919, 'grad_norm': 4.573256492614746, 'learning_rate': 3.992422814853053e-05, 'epoch': 0.57}

  6%|▌         | 2347/41250 [5:39:54<94:47:06,  8.77s/it][2025-04-25 13:37:37,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 13:37:37,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.02 | bwd_microstep: 5746.09 | bwd_inner_microstep: 5651.98 | bwd_allreduce_microstep: 94.07 | step_microstep: 19.32
[2025-04-25 13:37:37,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.02 | bwd: 5746.11 | bwd_inner: 5651.98 | bwd_allreduce: 94.09 | step: 19.32

  6%|▌         | 2348/41250 [5:40:02<94:27:11,  8.74s/it]
                                                         
{'loss': 0.2106, 'grad_norm': 1.4985992908477783, 'learning_rate': 3.9924091524500516e-05, 'epoch': 0.57}

  6%|▌         | 2348/41250 [5:40:02<94:27:11,  8.74s/it][2025-04-25 13:37:46,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:37:46,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.15 | bwd_microstep: 5992.02 | bwd_inner_microstep: 5704.52 | bwd_allreduce_microstep: 287.46 | step_microstep: 18.41
[2025-04-25 13:37:46,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.15 | bwd: 5992.03 | bwd_inner: 5704.51 | bwd_allreduce: 287.48 | step: 18.42

  6%|▌         | 2349/41250 [5:40:11<95:03:59,  8.80s/it]
                                                         
{'loss': 0.2318, 'grad_norm': 4.439553737640381, 'learning_rate': 3.992395477764239e-05, 'epoch': 0.57}

  6%|▌         | 2349/41250 [5:40:11<95:03:59,  8.80s/it][2025-04-25 13:37:54,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:37:54,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.18 | bwd_microstep: 5719.21 | bwd_inner_microstep: 5680.08 | bwd_allreduce_microstep: 39.09 | step_microstep: 18.60
[2025-04-25 13:37:54,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.18 | bwd: 5719.22 | bwd_inner: 5680.08 | bwd_allreduce: 39.10 | step: 18.60

  6%|▌         | 2350/41250 [5:40:20<94:37:47,  8.76s/it]
                                                         
{'loss': 0.0908, 'grad_norm': 1.404505968093872, 'learning_rate': 3.9923817907957004e-05, 'epoch': 0.57}

  6%|▌         | 2350/41250 [5:40:20<94:37:47,  8.76s/it][2025-04-25 13:38:03,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 13:38:03,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5743.29 | bwd_inner_microstep: 5649.92 | bwd_allreduce_microstep: 93.32 | step_microstep: 19.22
[2025-04-25 13:38:03,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5743.31 | bwd_inner: 5649.92 | bwd_allreduce: 93.34 | step: 19.22

  6%|▌         | 2351/41250 [5:40:28<94:19:30,  8.73s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 1.544191837310791, 'learning_rate': 3.9923680915445194e-05, 'epoch': 0.57}

  6%|▌         | 2351/41250 [5:40:28<94:19:30,  8.73s/it][2025-04-25 13:38:12,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:38:12,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.58 | bwd_microstep: 5699.55 | bwd_inner_microstep: 5686.37 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.78
[2025-04-25 13:38:12,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.58 | bwd: 5699.56 | bwd_inner: 5686.37 | bwd_allreduce: 13.15 | step: 18.79

  6%|▌         | 2352/41250 [5:40:37<94:00:41,  8.70s/it]
                                                         
{'loss': 0.0859, 'grad_norm': 0.7676007151603699, 'learning_rate': 3.992354380010781e-05, 'epoch': 0.57}

  6%|▌         | 2352/41250 [5:40:37<94:00:41,  8.70s/it][2025-04-25 13:38:20,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:38:20,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.20 | bwd_microstep: 5698.76 | bwd_inner_microstep: 5641.88 | bwd_allreduce_microstep: 56.84 | step_microstep: 18.67
[2025-04-25 13:38:20,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.20 | bwd: 5698.77 | bwd_inner: 5641.88 | bwd_allreduce: 56.86 | step: 18.68

  6%|▌         | 2353/41250 [5:40:46<93:42:34,  8.67s/it]
                                                         
{'loss': 0.2392, 'grad_norm': 1.7216777801513672, 'learning_rate': 3.992340656194571e-05, 'epoch': 0.57}

  6%|▌         | 2353/41250 [5:40:46<93:42:34,  8.67s/it][2025-04-25 13:38:29,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:38:29,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5730.81 | bwd_inner_microstep: 5688.22 | bwd_allreduce_microstep: 42.54 | step_microstep: 18.49
[2025-04-25 13:38:29,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5730.82 | bwd_inner: 5688.22 | bwd_allreduce: 42.55 | step: 18.49

  6%|▌         | 2354/41250 [5:40:54<93:41:18,  8.67s/it]
                                                         
{'loss': 0.0835, 'grad_norm': 1.278191328048706, 'learning_rate': 3.992326920095971e-05, 'epoch': 0.57}

  6%|▌         | 2354/41250 [5:40:54<93:41:18,  8.67s/it][2025-04-25 13:38:38,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:38:38,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.25 | bwd_microstep: 5722.39 | bwd_inner_microstep: 5703.90 | bwd_allreduce_microstep: 18.45 | step_microstep: 18.76
[2025-04-25 13:38:38,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.25 | bwd: 5722.41 | bwd_inner: 5703.90 | bwd_allreduce: 18.46 | step: 18.76

  6%|▌         | 2355/41250 [5:41:03<93:40:14,  8.67s/it]
                                                         
{'loss': 0.1407, 'grad_norm': 2.9878878593444824, 'learning_rate': 3.9923131717150686e-05, 'epoch': 0.57}

  6%|▌         | 2355/41250 [5:41:03<93:40:14,  8.67s/it][2025-04-25 13:38:46,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:38:46,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.77 | bwd_microstep: 5824.99 | bwd_inner_microstep: 5683.31 | bwd_allreduce_microstep: 141.63 | step_microstep: 18.64
[2025-04-25 13:38:46,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.77 | bwd: 5825.00 | bwd_inner: 5683.31 | bwd_allreduce: 141.65 | step: 18.64

  6%|▌         | 2356/41250 [5:41:12<94:00:02,  8.70s/it]
                                                         
{'loss': 0.4933, 'grad_norm': 3.4760565757751465, 'learning_rate': 3.9922994110519464e-05, 'epoch': 0.57}

  6%|▌         | 2356/41250 [5:41:12<94:00:02,  8.70s/it][2025-04-25 13:38:55,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 13:38:55,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.58 | bwd_microstep: 5689.83 | bwd_inner_microstep: 5652.88 | bwd_allreduce_microstep: 36.90 | step_microstep: 19.11
[2025-04-25 13:38:55,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.58 | bwd: 5689.84 | bwd_inner: 5652.88 | bwd_allreduce: 36.93 | step: 19.11

  6%|▌         | 2357/41250 [5:41:20<93:40:22,  8.67s/it]
                                                         
{'loss': 0.1666, 'grad_norm': 1.2218834161758423, 'learning_rate': 3.992285638106691e-05, 'epoch': 0.57}

  6%|▌         | 2357/41250 [5:41:20<93:40:22,  8.67s/it][2025-04-25 13:39:04,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:39:04,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5760.08 | bwd_inner_microstep: 5654.88 | bwd_allreduce_microstep: 105.15 | step_microstep: 18.85
[2025-04-25 13:39:04,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.09 | bwd: 5760.09 | bwd_inner: 5654.88 | bwd_allreduce: 105.17 | step: 18.85

  6%|▌         | 2358/41250 [5:41:29<93:39:33,  8.67s/it]
                                                         
{'loss': 0.1864, 'grad_norm': 2.170083522796631, 'learning_rate': 3.9922718528793866e-05, 'epoch': 0.57}

  6%|▌         | 2358/41250 [5:41:29<93:39:33,  8.67s/it][2025-04-25 13:39:12,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 13:39:12,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.28 | bwd_microstep: 5714.75 | bwd_inner_microstep: 5702.24 | bwd_allreduce_microstep: 12.46 | step_microstep: 18.91
[2025-04-25 13:39:12,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.28 | bwd: 5714.76 | bwd_inner: 5702.24 | bwd_allreduce: 12.48 | step: 18.92

  6%|▌         | 2359/41250 [5:41:38<93:36:01,  8.66s/it]
                                                         
{'loss': 0.0788, 'grad_norm': 1.3191770315170288, 'learning_rate': 3.992258055370117e-05, 'epoch': 0.57}

  6%|▌         | 2359/41250 [5:41:38<93:36:01,  8.66s/it][2025-04-25 13:39:21,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:39:21,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5677.01 | bwd_inner_microstep: 5653.11 | bwd_allreduce_microstep: 23.85 | step_microstep: 18.84
[2025-04-25 13:39:21,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5677.03 | bwd_inner: 5653.11 | bwd_allreduce: 23.87 | step: 18.84

  6%|▌         | 2360/41250 [5:41:46<93:20:50,  8.64s/it]
                                                         
{'loss': 0.1303, 'grad_norm': 1.4609766006469727, 'learning_rate': 3.9922442455789693e-05, 'epoch': 0.57}

  6%|▌         | 2360/41250 [5:41:46<93:20:50,  8.64s/it][2025-04-25 13:39:30,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 13:39:30,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.75 | bwd_microstep: 5711.40 | bwd_inner_microstep: 5698.71 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.58
[2025-04-25 13:39:30,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.75 | bwd: 5711.41 | bwd_inner: 5698.71 | bwd_allreduce: 12.66 | step: 18.58

  6%|▌         | 2361/41250 [5:41:55<93:21:09,  8.64s/it]
                                                         
{'loss': 0.1651, 'grad_norm': 1.674394130706787, 'learning_rate': 3.992230423506027e-05, 'epoch': 0.57}

  6%|▌         | 2361/41250 [5:41:55<93:21:09,  8.64s/it][2025-04-25 13:39:38,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-25 13:39:38,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.76 | bwd_microstep: 5721.67 | bwd_inner_microstep: 5689.09 | bwd_allreduce_microstep: 32.53 | step_microstep: 19.34
[2025-04-25 13:39:38,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.76 | bwd: 5721.68 | bwd_inner: 5689.09 | bwd_allreduce: 32.55 | step: 19.34

  6%|▌         | 2362/41250 [5:42:04<93:22:06,  8.64s/it]
                                                         
{'loss': 0.0785, 'grad_norm': 0.6492615342140198, 'learning_rate': 3.992216589151377e-05, 'epoch': 0.57}

  6%|▌         | 2362/41250 [5:42:04<93:22:06,  8.64s/it][2025-04-25 13:39:47,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 13:39:47,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.02 | bwd_microstep: 5717.47 | bwd_inner_microstep: 5692.53 | bwd_allreduce_microstep: 24.89 | step_microstep: 18.83
[2025-04-25 13:39:47,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.02 | bwd: 5717.48 | bwd_inner: 5692.53 | bwd_allreduce: 24.91 | step: 18.83

  6%|▌         | 2363/41250 [5:42:12<93:24:53,  8.65s/it]
                                                         
{'loss': 0.2969, 'grad_norm': 3.437812328338623, 'learning_rate': 3.9922027425151026e-05, 'epoch': 0.57}

  6%|▌         | 2363/41250 [5:42:12<93:24:53,  8.65s/it][2025-04-25 13:39:56,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:39:56,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.78 | bwd_microstep: 5676.88 | bwd_inner_microstep: 5655.57 | bwd_allreduce_microstep: 21.26 | step_microstep: 19.29
[2025-04-25 13:39:56,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.78 | bwd: 5676.90 | bwd_inner: 5655.57 | bwd_allreduce: 21.28 | step: 19.29

  6%|▌         | 2364/41250 [5:42:21<93:14:19,  8.63s/it]
                                                         
{'loss': 0.1759, 'grad_norm': 3.1286087036132812, 'learning_rate': 3.992188883597291e-05, 'epoch': 0.57}

  6%|▌         | 2364/41250 [5:42:21<93:14:19,  8.63s/it][2025-04-25 13:40:04,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:40:04,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5715.53 | bwd_inner_microstep: 5702.78 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.81
[2025-04-25 13:40:04,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5715.55 | bwd_inner: 5702.78 | bwd_allreduce: 12.73 | step: 18.82

  6%|▌         | 2365/41250 [5:42:29<93:17:09,  8.64s/it]
                                                         
{'loss': 0.0788, 'grad_norm': 0.8603729605674744, 'learning_rate': 3.9921750123980264e-05, 'epoch': 0.57}

  6%|▌         | 2365/41250 [5:42:30<93:17:09,  8.64s/it][2025-04-25 13:40:13,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:40:13,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.88 | bwd_microstep: 5686.53 | bwd_inner_microstep: 5654.90 | bwd_allreduce_microstep: 31.59 | step_microstep: 18.51
[2025-04-25 13:40:13,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.88 | bwd: 5686.55 | bwd_inner: 5654.90 | bwd_allreduce: 31.60 | step: 18.51

  6%|▌         | 2366/41250 [5:42:38<93:09:29,  8.62s/it]
                                                         
{'loss': 0.1961, 'grad_norm': 1.723020315170288, 'learning_rate': 3.992161128917395e-05, 'epoch': 0.57}

  6%|▌         | 2366/41250 [5:42:38<93:09:29,  8.62s/it][2025-04-25 13:40:21,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 13:40:21,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5707.48 | bwd_inner_microstep: 5694.68 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.90
[2025-04-25 13:40:21,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5707.49 | bwd_inner: 5694.68 | bwd_allreduce: 12.78 | step: 18.90

  6%|▌         | 2367/41250 [5:42:47<93:12:58,  8.63s/it]
                                                         
{'loss': 0.106, 'grad_norm': 0.893842339515686, 'learning_rate': 3.992147233155481e-05, 'epoch': 0.57}

  6%|▌         | 2367/41250 [5:42:47<93:12:58,  8.63s/it][2025-04-25 13:40:30,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 13:40:30,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.37 | bwd_microstep: 5789.41 | bwd_inner_microstep: 5776.49 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.45
[2025-04-25 13:40:30,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.37 | bwd: 5789.43 | bwd_inner: 5776.49 | bwd_allreduce: 12.89 | step: 19.45

  6%|▌         | 2368/41250 [5:42:55<93:37:56,  8.67s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 1.5961862802505493, 'learning_rate': 3.992133325112372e-05, 'epoch': 0.57}

  6%|▌         | 2368/41250 [5:42:56<93:37:56,  8.67s/it][2025-04-25 13:40:39,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:40:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.98 | bwd_microstep: 5688.78 | bwd_inner_microstep: 5645.96 | bwd_allreduce_microstep: 42.78 | step_microstep: 18.37
[2025-04-25 13:40:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.98 | bwd: 5688.80 | bwd_inner: 5645.96 | bwd_allreduce: 42.80 | step: 18.38

  6%|▌         | 2369/41250 [5:43:04<93:23:29,  8.65s/it]
                                                         
{'loss': 0.0743, 'grad_norm': 0.981176495552063, 'learning_rate': 3.992119404788152e-05, 'epoch': 0.57}

  6%|▌         | 2369/41250 [5:43:04<93:23:29,  8.65s/it][2025-04-25 13:40:47,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 13:40:47,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.60 | bwd_microstep: 5762.22 | bwd_inner_microstep: 5642.29 | bwd_allreduce_microstep: 119.88 | step_microstep: 18.85
[2025-04-25 13:40:47,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.60 | bwd: 5762.23 | bwd_inner: 5642.29 | bwd_allreduce: 119.90 | step: 18.85

  6%|▌         | 2370/41250 [5:43:13<93:27:20,  8.65s/it]
                                                         
{'loss': 0.0873, 'grad_norm': 1.9162167310714722, 'learning_rate': 3.992105472182909e-05, 'epoch': 0.57}

  6%|▌         | 2370/41250 [5:43:13<93:27:20,  8.65s/it][2025-04-25 13:40:56,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 13:40:56,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.13 | bwd_microstep: 5840.95 | bwd_inner_microstep: 5701.92 | bwd_allreduce_microstep: 138.98 | step_microstep: 18.74
[2025-04-25 13:40:56,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.13 | bwd: 5840.96 | bwd_inner: 5701.92 | bwd_allreduce: 139.00 | step: 18.74

  6%|▌         | 2371/41250 [5:43:22<93:50:05,  8.69s/it]
                                                         
{'loss': 0.2822, 'grad_norm': 2.4462969303131104, 'learning_rate': 3.992091527296727e-05, 'epoch': 0.57}

  6%|▌         | 2371/41250 [5:43:22<93:50:05,  8.69s/it][2025-04-25 13:41:05,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 13:41:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.77 | bwd_microstep: 5740.35 | bwd_inner_microstep: 5693.68 | bwd_allreduce_microstep: 46.61 | step_microstep: 18.96
[2025-04-25 13:41:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.77 | bwd: 5740.36 | bwd_inner: 5693.68 | bwd_allreduce: 46.63 | step: 18.96

  6%|▌         | 2372/41250 [5:43:30<93:46:02,  8.68s/it]
                                                         
{'loss': 0.3907, 'grad_norm': 1.7628604173660278, 'learning_rate': 3.992077570129692e-05, 'epoch': 0.58}

  6%|▌         | 2372/41250 [5:43:30<93:46:02,  8.68s/it][2025-04-25 13:41:14,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-25 13:41:14,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.99 | bwd_microstep: 5740.13 | bwd_inner_microstep: 5642.70 | bwd_allreduce_microstep: 97.38 | step_microstep: 18.98
[2025-04-25 13:41:14,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.99 | bwd: 5740.14 | bwd_inner: 5642.70 | bwd_allreduce: 97.40 | step: 18.98

  6%|▌         | 2373/41250 [5:43:39<93:39:03,  8.67s/it]
                                                         
{'loss': 0.0828, 'grad_norm': 1.1019631624221802, 'learning_rate': 3.992063600681892e-05, 'epoch': 0.58}

  6%|▌         | 2373/41250 [5:43:39<93:39:03,  8.67s/it][2025-04-25 13:41:22,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:41:22,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5727.89 | bwd_inner_microstep: 5678.30 | bwd_allreduce_microstep: 49.54 | step_microstep: 18.45
[2025-04-25 13:41:22,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5727.90 | bwd_inner: 5678.30 | bwd_allreduce: 49.56 | step: 18.46

  6%|▌         | 2374/41250 [5:43:48<93:37:22,  8.67s/it]
                                                         
{'loss': 0.1005, 'grad_norm': 1.386563777923584, 'learning_rate': 3.9920496189534106e-05, 'epoch': 0.58}

  6%|▌         | 2374/41250 [5:43:48<93:37:22,  8.67s/it][2025-04-25 13:41:31,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:41:31,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5738.43 | bwd_inner_microstep: 5647.73 | bwd_allreduce_microstep: 90.65 | step_microstep: 18.93
[2025-04-25 13:41:31,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5738.44 | bwd_inner: 5647.73 | bwd_allreduce: 90.67 | step: 18.94

  6%|▌         | 2375/41250 [5:43:56<93:33:24,  8.66s/it]
                                                         
{'loss': 0.4715, 'grad_norm': 2.0343399047851562, 'learning_rate': 3.9920356249443354e-05, 'epoch': 0.58}

  6%|▌         | 2375/41250 [5:43:56<93:33:24,  8.66s/it][2025-04-25 13:41:40,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:41:40,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.05 | bwd_microstep: 5719.31 | bwd_inner_microstep: 5699.98 | bwd_allreduce_microstep: 19.28 | step_microstep: 18.83
[2025-04-25 13:41:40,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.05 | bwd: 5719.32 | bwd_inner: 5699.98 | bwd_allreduce: 19.30 | step: 18.83

  6%|▌         | 2376/41250 [5:44:05<93:35:14,  8.67s/it]
                                                         
{'loss': 0.1626, 'grad_norm': 1.7637101411819458, 'learning_rate': 3.992021618654752e-05, 'epoch': 0.58}

  6%|▌         | 2376/41250 [5:44:05<93:35:14,  8.67s/it][2025-04-25 13:41:48,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:41:48,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.69 | bwd_microstep: 5865.51 | bwd_inner_microstep: 5656.32 | bwd_allreduce_microstep: 209.14 | step_microstep: 18.50
[2025-04-25 13:41:48,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.69 | bwd: 5865.52 | bwd_inner: 5656.32 | bwd_allreduce: 209.16 | step: 18.50

  6%|▌         | 2377/41250 [5:44:14<93:56:34,  8.70s/it]
                                                         
{'loss': 0.1935, 'grad_norm': 1.7748945951461792, 'learning_rate': 3.992007600084748e-05, 'epoch': 0.58}

  6%|▌         | 2377/41250 [5:44:14<93:56:34,  8.70s/it][2025-04-25 13:41:57,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 13:41:57,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.80 | bwd_microstep: 5723.83 | bwd_inner_microstep: 5638.88 | bwd_allreduce_microstep: 84.89 | step_microstep: 19.25
[2025-04-25 13:41:57,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.80 | bwd: 5723.85 | bwd_inner: 5638.88 | bwd_allreduce: 84.92 | step: 19.25

  6%|▌         | 2378/41250 [5:44:22<93:45:11,  8.68s/it]
                                                         
{'loss': 0.2355, 'grad_norm': 2.4731338024139404, 'learning_rate': 3.991993569234408e-05, 'epoch': 0.58}

  6%|▌         | 2378/41250 [5:44:22<93:45:11,  8.68s/it][2025-04-25 13:42:06,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 13:42:06,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.09 | bwd_microstep: 5721.99 | bwd_inner_microstep: 5680.59 | bwd_allreduce_microstep: 41.34 | step_microstep: 18.92
[2025-04-25 13:42:06,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.09 | bwd: 5722.00 | bwd_inner: 5680.59 | bwd_allreduce: 41.36 | step: 18.92

  6%|▌         | 2379/41250 [5:44:31<93:39:08,  8.67s/it]
                                                         
{'loss': 0.085, 'grad_norm': 1.014892339706421, 'learning_rate': 3.9919795261038204e-05, 'epoch': 0.58}

  6%|▌         | 2379/41250 [5:44:31<93:39:08,  8.67s/it][2025-04-25 13:42:14,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 13:42:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5738.50 | bwd_inner_microstep: 5656.43 | bwd_allreduce_microstep: 82.03 | step_microstep: 19.38
[2025-04-25 13:42:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5738.51 | bwd_inner: 5656.43 | bwd_allreduce: 82.05 | step: 19.38

  6%|▌         | 2380/41250 [5:44:40<93:33:30,  8.67s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.6781564950942993, 'learning_rate': 3.99196547069307e-05, 'epoch': 0.58}

  6%|▌         | 2380/41250 [5:44:40<93:33:30,  8.67s/it][2025-04-25 13:42:23,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:42:23,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5710.44 | bwd_inner_microstep: 5697.58 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.99
[2025-04-25 13:42:23,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5710.45 | bwd_inner: 5697.58 | bwd_allreduce: 12.83 | step: 18.99

  6%|▌         | 2381/41250 [5:44:48<93:28:55,  8.66s/it]
                                                         
{'loss': 0.5055, 'grad_norm': 3.0212666988372803, 'learning_rate': 3.991951403002245e-05, 'epoch': 0.58}

  6%|▌         | 2381/41250 [5:44:48<93:28:55,  8.66s/it][2025-04-25 13:42:32,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 13:42:32,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.81 | bwd_microstep: 5750.17 | bwd_inner_microstep: 5688.66 | bwd_allreduce_microstep: 61.47 | step_microstep: 18.82
[2025-04-25 13:42:32,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.81 | bwd: 5750.19 | bwd_inner: 5688.66 | bwd_allreduce: 61.49 | step: 18.82

  6%|▌         | 2382/41250 [5:44:57<93:32:38,  8.66s/it]
                                                         
{'loss': 0.6171, 'grad_norm': 3.208852767944336, 'learning_rate': 3.991937323031431e-05, 'epoch': 0.58}

  6%|▌         | 2382/41250 [5:44:57<93:32:38,  8.66s/it][2025-04-25 13:42:40,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 13:42:40,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.93 | bwd_microstep: 5727.95 | bwd_inner_microstep: 5692.83 | bwd_allreduce_microstep: 35.07 | step_microstep: 18.70
[2025-04-25 13:42:40,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.93 | bwd: 5727.96 | bwd_inner: 5692.83 | bwd_allreduce: 35.09 | step: 18.70

  6%|▌         | 2383/41250 [5:45:06<93:31:22,  8.66s/it]
                                                         
{'loss': 0.255, 'grad_norm': 1.7252925634384155, 'learning_rate': 3.991923230780715e-05, 'epoch': 0.58}

  6%|▌         | 2383/41250 [5:45:06<93:31:22,  8.66s/it][2025-04-25 13:42:49,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 13:42:49,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5756.20 | bwd_inner_microstep: 5649.22 | bwd_allreduce_microstep: 106.92 | step_microstep: 18.57
[2025-04-25 13:42:49,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5756.21 | bwd_inner: 5649.22 | bwd_allreduce: 106.95 | step: 18.57

  6%|▌         | 2384/41250 [5:45:14<93:31:44,  8.66s/it]
                                                         
{'loss': 0.2449, 'grad_norm': 1.4244364500045776, 'learning_rate': 3.9919091262501844e-05, 'epoch': 0.58}

  6%|▌         | 2384/41250 [5:45:14<93:31:44,  8.66s/it][2025-04-25 13:42:58,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:42:58,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.54 | bwd_microstep: 5726.33 | bwd_inner_microstep: 5714.14 | bwd_allreduce_microstep: 12.15 | step_microstep: 18.70
[2025-04-25 13:42:58,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.54 | bwd: 5726.34 | bwd_inner: 5714.14 | bwd_allreduce: 12.16 | step: 18.71

  6%|▌         | 2385/41250 [5:45:23<93:32:12,  8.66s/it]
                                                         
{'loss': 0.1158, 'grad_norm': 1.0720109939575195, 'learning_rate': 3.991895009439926e-05, 'epoch': 0.58}

  6%|▌         | 2385/41250 [5:45:23<93:32:12,  8.66s/it][2025-04-25 13:43:06,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 13:43:06,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5740.77 | bwd_inner_microstep: 5711.21 | bwd_allreduce_microstep: 29.52 | step_microstep: 19.48
[2025-04-25 13:43:06,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5740.79 | bwd_inner: 5711.21 | bwd_allreduce: 29.54 | step: 19.49

  6%|▌         | 2386/41250 [5:45:32<93:34:41,  8.67s/it]
                                                         
{'loss': 0.2806, 'grad_norm': 1.4478918313980103, 'learning_rate': 3.991880880350026e-05, 'epoch': 0.58}

  6%|▌         | 2386/41250 [5:45:32<93:34:41,  8.67s/it][2025-04-25 13:43:15,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:43:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5771.35 | bwd_inner_microstep: 5655.62 | bwd_allreduce_microstep: 115.68 | step_microstep: 18.86
[2025-04-25 13:43:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5771.36 | bwd_inner: 5655.62 | bwd_allreduce: 115.70 | step: 18.87

  6%|▌         | 2387/41250 [5:45:40<93:38:58,  8.68s/it]
                                                         
{'loss': 0.0969, 'grad_norm': 0.9307635426521301, 'learning_rate': 3.9918667389805725e-05, 'epoch': 0.58}

  6%|▌         | 2387/41250 [5:45:40<93:38:58,  8.68s/it][2025-04-25 13:43:24,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:43:24,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5860.50 | bwd_inner_microstep: 5704.39 | bwd_allreduce_microstep: 156.06 | step_microstep: 19.21
[2025-04-25 13:43:24,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5860.52 | bwd_inner: 5704.39 | bwd_allreduce: 156.08 | step: 19.21

  6%|▌         | 2388/41250 [5:45:49<94:02:34,  8.71s/it]
                                                         
{'loss': 0.275, 'grad_norm': 1.5208839178085327, 'learning_rate': 3.9918525853316525e-05, 'epoch': 0.58}

  6%|▌         | 2388/41250 [5:45:49<94:02:34,  8.71s/it][2025-04-25 13:43:32,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 13:43:32,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.97 | bwd_microstep: 5761.37 | bwd_inner_microstep: 5695.17 | bwd_allreduce_microstep: 66.15 | step_microstep: 18.76
[2025-04-25 13:43:32,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.97 | bwd: 5761.38 | bwd_inner: 5695.17 | bwd_allreduce: 66.17 | step: 18.76

  6%|▌         | 2389/41250 [5:45:58<93:58:45,  8.71s/it]
                                                         
{'loss': 0.1624, 'grad_norm': 1.3756321668624878, 'learning_rate': 3.991838419403353e-05, 'epoch': 0.58}

  6%|▌         | 2389/41250 [5:45:58<93:58:45,  8.71s/it][2025-04-25 13:43:41,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:43:41,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.88 | bwd_microstep: 5746.32 | bwd_inner_microstep: 5715.55 | bwd_allreduce_microstep: 30.72 | step_microstep: 18.75
[2025-04-25 13:43:41,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.88 | bwd: 5746.33 | bwd_inner: 5715.55 | bwd_allreduce: 30.74 | step: 18.76

  6%|▌         | 2390/41250 [5:46:06<93:54:35,  8.70s/it]
                                                         
{'loss': 0.2806, 'grad_norm': 5.383550643920898, 'learning_rate': 3.9918242411957606e-05, 'epoch': 0.58}

  6%|▌         | 2390/41250 [5:46:06<93:54:35,  8.70s/it][2025-04-25 13:43:50,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:43:50,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.34 | bwd_microstep: 5725.01 | bwd_inner_microstep: 5712.34 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.73
[2025-04-25 13:43:50,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.34 | bwd: 5725.02 | bwd_inner: 5712.34 | bwd_allreduce: 12.64 | step: 18.73

  6%|▌         | 2391/41250 [5:46:15<93:48:39,  8.69s/it]
                                                         
{'loss': 0.1003, 'grad_norm': 1.5713204145431519, 'learning_rate': 3.991810050708964e-05, 'epoch': 0.58}

  6%|▌         | 2391/41250 [5:46:15<93:48:39,  8.69s/it][2025-04-25 13:43:58,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 13:43:58,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.78 | bwd_microstep: 5702.42 | bwd_inner_microstep: 5648.50 | bwd_allreduce_microstep: 53.87 | step_microstep: 18.89
[2025-04-25 13:43:58,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.78 | bwd: 5702.43 | bwd_inner: 5648.50 | bwd_allreduce: 53.89 | step: 18.89

  6%|▌         | 2392/41250 [5:46:24<93:33:27,  8.67s/it]
                                                         
{'loss': 0.2765, 'grad_norm': 2.1989052295684814, 'learning_rate': 3.991795847943051e-05, 'epoch': 0.58}

  6%|▌         | 2392/41250 [5:46:24<93:33:27,  8.67s/it][2025-04-25 13:44:07,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 13:44:07,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.88 | bwd_microstep: 5757.92 | bwd_inner_microstep: 5723.58 | bwd_allreduce_microstep: 34.30 | step_microstep: 18.81
[2025-04-25 13:44:07,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.88 | bwd: 5757.93 | bwd_inner: 5723.58 | bwd_allreduce: 34.31 | step: 18.81

  6%|▌         | 2393/41250 [5:46:32<93:39:51,  8.68s/it]
                                                         
{'loss': 0.0536, 'grad_norm': 0.6231924891471863, 'learning_rate': 3.991781632898108e-05, 'epoch': 0.58}

  6%|▌         | 2393/41250 [5:46:32<93:39:51,  8.68s/it][2025-04-25 13:44:16,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-25 13:44:16,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5718.71 | bwd_inner_microstep: 5705.39 | bwd_allreduce_microstep: 13.26 | step_microstep: 19.57
[2025-04-25 13:44:16,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5718.72 | bwd_inner: 5705.39 | bwd_allreduce: 13.28 | step: 19.57

  6%|▌         | 2394/41250 [5:46:41<93:36:02,  8.67s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.4028143882751465, 'learning_rate': 3.991767405574223e-05, 'epoch': 0.58}

  6%|▌         | 2394/41250 [5:46:41<93:36:02,  8.67s/it][2025-04-25 13:44:24,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 13:44:24,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.93 | bwd_microstep: 5708.12 | bwd_inner_microstep: 5666.90 | bwd_allreduce_microstep: 41.17 | step_microstep: 19.16
[2025-04-25 13:44:24,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.93 | bwd: 5708.13 | bwd_inner: 5666.90 | bwd_allreduce: 41.19 | step: 19.16

  6%|▌         | 2395/41250 [5:46:50<93:27:15,  8.66s/it]
                                                         
{'loss': 0.1092, 'grad_norm': 1.3138973712921143, 'learning_rate': 3.991753165971483e-05, 'epoch': 0.58}

  6%|▌         | 2395/41250 [5:46:50<93:27:15,  8.66s/it][2025-04-25 13:44:33,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 13:44:33,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.25 | bwd_microstep: 5708.34 | bwd_inner_microstep: 5695.57 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.87
[2025-04-25 13:44:33,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.25 | bwd: 5708.35 | bwd_inner: 5695.57 | bwd_allreduce: 12.75 | step: 18.87

  6%|▌         | 2396/41250 [5:46:58<93:24:21,  8.65s/it]
                                                         
{'loss': 0.0566, 'grad_norm': 0.9980050325393677, 'learning_rate': 3.991738914089977e-05, 'epoch': 0.58}

  6%|▌         | 2396/41250 [5:46:58<93:24:21,  8.65s/it][2025-04-25 13:44:42,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 13:44:42,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.06 | bwd_microstep: 5754.73 | bwd_inner_microstep: 5702.45 | bwd_allreduce_microstep: 52.24 | step_microstep: 18.83
[2025-04-25 13:44:42,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.06 | bwd: 5754.74 | bwd_inner: 5702.45 | bwd_allreduce: 52.26 | step: 18.83

  6%|▌         | 2397/41250 [5:47:07<93:31:20,  8.67s/it]
                                                         
{'loss': 0.1988, 'grad_norm': 2.4223201274871826, 'learning_rate': 3.991724649929793e-05, 'epoch': 0.58}

  6%|▌         | 2397/41250 [5:47:07<93:31:20,  8.67s/it][2025-04-25 13:44:50,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:44:50,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.33 | bwd_microstep: 5693.17 | bwd_inner_microstep: 5663.17 | bwd_allreduce_microstep: 29.95 | step_microstep: 18.99
[2025-04-25 13:44:50,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.33 | bwd: 5693.18 | bwd_inner: 5663.17 | bwd_allreduce: 29.97 | step: 18.99

  6%|▌         | 2398/41250 [5:47:16<93:19:49,  8.65s/it]
                                                         
{'loss': 0.376, 'grad_norm': 2.0230016708374023, 'learning_rate': 3.991710373491017e-05, 'epoch': 0.58}

  6%|▌         | 2398/41250 [5:47:16<93:19:49,  8.65s/it][2025-04-25 13:44:59,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 13:44:59,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.26 | bwd_microstep: 5709.19 | bwd_inner_microstep: 5666.65 | bwd_allreduce_microstep: 42.49 | step_microstep: 19.24
[2025-04-25 13:44:59,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.26 | bwd: 5709.21 | bwd_inner: 5666.65 | bwd_allreduce: 42.52 | step: 19.24

  6%|▌         | 2399/41250 [5:47:24<93:14:59,  8.64s/it]
                                                         
{'loss': 0.2671, 'grad_norm': 1.8599878549575806, 'learning_rate': 3.9916960847737396e-05, 'epoch': 0.58}

  6%|▌         | 2399/41250 [5:47:24<93:14:59,  8.64s/it][2025-04-25 13:45:08,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:45:08,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.17 | bwd_microstep: 5705.84 | bwd_inner_microstep: 5666.29 | bwd_allreduce_microstep: 39.50 | step_microstep: 18.54
[2025-04-25 13:45:08,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.17 | bwd: 5705.85 | bwd_inner: 5666.29 | bwd_allreduce: 39.52 | step: 18.55

  6%|▌         | 2400/41250 [5:47:33<93:12:45,  8.64s/it]
                                                         
{'loss': 0.3634, 'grad_norm': 1.9943703413009644, 'learning_rate': 3.9916817837780464e-05, 'epoch': 0.58}

  6%|▌         | 2400/41250 [5:47:33<93:12:45,  8.64s/it][2025-04-25 13:45:16,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 13:45:16,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.68 | bwd_microstep: 5764.49 | bwd_inner_microstep: 5668.07 | bwd_allreduce_microstep: 96.38 | step_microstep: 18.65
[2025-04-25 13:45:16,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.68 | bwd: 5764.50 | bwd_inner: 5668.07 | bwd_allreduce: 96.40 | step: 18.65

  6%|▌         | 2401/41250 [5:47:42<93:23:32,  8.65s/it]
                                                         
{'loss': 0.0805, 'grad_norm': 1.6513996124267578, 'learning_rate': 3.991667470504028e-05, 'epoch': 0.58}

  6%|▌         | 2401/41250 [5:47:42<93:23:32,  8.65s/it][2025-04-25 13:45:25,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 13:45:25,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.26 | bwd_microstep: 5723.89 | bwd_inner_microstep: 5711.05 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.80
[2025-04-25 13:45:25,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.26 | bwd: 5723.90 | bwd_inner: 5711.05 | bwd_allreduce: 12.81 | step: 18.80

  6%|▌         | 2402/41250 [5:47:50<93:25:12,  8.66s/it]
                                                         
{'loss': 0.1253, 'grad_norm': 1.4813151359558105, 'learning_rate': 3.99165314495177e-05, 'epoch': 0.58}

  6%|▌         | 2402/41250 [5:47:50<93:25:12,  8.66s/it][2025-04-25 13:45:34,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-25 13:45:34,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.15 | bwd_microstep: 5773.38 | bwd_inner_microstep: 5664.74 | bwd_allreduce_microstep: 108.59 | step_microstep: 19.54
[2025-04-25 13:45:34,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.15 | bwd: 5773.39 | bwd_inner: 5664.74 | bwd_allreduce: 108.61 | step: 19.56

  6%|▌         | 2403/41250 [5:47:59<93:32:20,  8.67s/it]
                                                         
{'loss': 0.2617, 'grad_norm': 2.223092555999756, 'learning_rate': 3.991638807121363e-05, 'epoch': 0.58}

  6%|▌         | 2403/41250 [5:47:59<93:32:20,  8.67s/it][2025-04-25 13:45:42,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 13:45:42,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5752.62 | bwd_inner_microstep: 5712.19 | bwd_allreduce_microstep: 40.39 | step_microstep: 19.04
[2025-04-25 13:45:42,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.69 | bwd: 5752.63 | bwd_inner: 5712.19 | bwd_allreduce: 40.41 | step: 19.04

  6%|▌         | 2404/41250 [5:48:08<93:36:14,  8.67s/it]
                                                         
{'loss': 0.1674, 'grad_norm': 1.2320533990859985, 'learning_rate': 3.991624457012894e-05, 'epoch': 0.58}

  6%|▌         | 2404/41250 [5:48:08<93:36:14,  8.67s/it][2025-04-25 13:45:51,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:45:51,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.93 | bwd_microstep: 5714.57 | bwd_inner_microstep: 5643.37 | bwd_allreduce_microstep: 71.16 | step_microstep: 18.95
[2025-04-25 13:45:51,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.93 | bwd: 5714.59 | bwd_inner: 5643.37 | bwd_allreduce: 71.18 | step: 18.95

  6%|▌         | 2405/41250 [5:48:16<93:26:28,  8.66s/it]
                                                         
{'loss': 0.054, 'grad_norm': 0.5092527866363525, 'learning_rate': 3.9916100946264526e-05, 'epoch': 0.58}

  6%|▌         | 2405/41250 [5:48:16<93:26:28,  8.66s/it][2025-04-25 13:46:00,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 13:46:00,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.99 | bwd_microstep: 5696.54 | bwd_inner_microstep: 5640.32 | bwd_allreduce_microstep: 56.17 | step_microstep: 18.55
[2025-04-25 13:46:00,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.99 | bwd: 5696.55 | bwd_inner: 5640.32 | bwd_allreduce: 56.19 | step: 18.56

  6%|▌         | 2406/41250 [5:48:25<93:16:04,  8.64s/it]
                                                         
{'loss': 0.1479, 'grad_norm': 1.014237880706787, 'learning_rate': 3.991595719962126e-05, 'epoch': 0.58}

  6%|▌         | 2406/41250 [5:48:25<93:16:04,  8.64s/it][2025-04-25 13:46:08,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 13:46:08,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.29 | bwd_microstep: 5711.51 | bwd_inner_microstep: 5698.59 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.41
[2025-04-25 13:46:08,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.29 | bwd: 5711.52 | bwd_inner: 5698.59 | bwd_allreduce: 12.89 | step: 18.41

  6%|▌         | 2407/41250 [5:48:33<93:17:06,  8.65s/it]
                                                         
{'loss': 0.115, 'grad_norm': 1.4492853879928589, 'learning_rate': 3.991581333020004e-05, 'epoch': 0.58}

  6%|▌         | 2407/41250 [5:48:34<93:17:06,  8.65s/it][2025-04-25 13:46:17,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 13:46:17,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.73 | bwd_microstep: 5704.00 | bwd_inner_microstep: 5647.29 | bwd_allreduce_microstep: 56.67 | step_microstep: 19.27
[2025-04-25 13:46:17,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.73 | bwd: 5704.02 | bwd_inner: 5647.29 | bwd_allreduce: 56.69 | step: 19.27

  6%|▌         | 2408/41250 [5:48:42<93:10:57,  8.64s/it]
                                                         
{'loss': 0.2526, 'grad_norm': 1.8219034671783447, 'learning_rate': 3.9915669338001745e-05, 'epoch': 0.58}

  6%|▌         | 2408/41250 [5:48:42<93:10:57,  8.64s/it][2025-04-25 13:46:25,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:46:25,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.13 | bwd_microstep: 5717.78 | bwd_inner_microstep: 5698.67 | bwd_allreduce_microstep: 19.05 | step_microstep: 18.65
[2025-04-25 13:46:25,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.13 | bwd: 5717.79 | bwd_inner: 5698.67 | bwd_allreduce: 19.07 | step: 18.66

  6%|▌         | 2409/41250 [5:48:51<93:15:15,  8.64s/it]
                                                         
{'loss': 0.3532, 'grad_norm': 2.3642401695251465, 'learning_rate': 3.991552522302727e-05, 'epoch': 0.58}

  6%|▌         | 2409/41250 [5:48:51<93:15:15,  8.64s/it][2025-04-25 13:46:34,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-25 13:46:34,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.87 | bwd_microstep: 5734.44 | bwd_inner_microstep: 5695.50 | bwd_allreduce_microstep: 38.89 | step_microstep: 19.77
[2025-04-25 13:46:34,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.87 | bwd: 5734.46 | bwd_inner: 5695.50 | bwd_allreduce: 38.92 | step: 19.78

  6%|▌         | 2410/41250 [5:48:59<93:20:12,  8.65s/it]
                                                         
{'loss': 0.2174, 'grad_norm': 2.0186755657196045, 'learning_rate': 3.991538098527749e-05, 'epoch': 0.58}

  6%|▌         | 2410/41250 [5:48:59<93:20:12,  8.65s/it][2025-04-25 13:46:43,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-25 13:46:43,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.49 | bwd_microstep: 5715.37 | bwd_inner_microstep: 5701.90 | bwd_allreduce_microstep: 13.42 | step_microstep: 19.07
[2025-04-25 13:46:43,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.49 | bwd: 5715.39 | bwd_inner: 5701.90 | bwd_allreduce: 13.45 | step: 19.08

  6%|▌         | 2411/41250 [5:49:08<93:19:46,  8.65s/it]
                                                         
{'loss': 0.2705, 'grad_norm': 1.1362894773483276, 'learning_rate': 3.9915236624753306e-05, 'epoch': 0.58}

  6%|▌         | 2411/41250 [5:49:08<93:19:46,  8.65s/it][2025-04-25 13:46:51,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 13:46:51,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.47 | bwd_microstep: 5720.80 | bwd_inner_microstep: 5707.63 | bwd_allreduce_microstep: 13.10 | step_microstep: 19.01
[2025-04-25 13:46:51,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.47 | bwd: 5720.81 | bwd_inner: 5707.63 | bwd_allreduce: 13.12 | step: 19.01

  6%|▌         | 2412/41250 [5:49:17<93:23:45,  8.66s/it]
                                                         
{'loss': 0.0445, 'grad_norm': 0.4411924481391907, 'learning_rate': 3.991509214145561e-05, 'epoch': 0.58}

  6%|▌         | 2412/41250 [5:49:17<93:23:45,  8.66s/it][2025-04-25 13:47:00,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 13:47:00,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.55 | bwd_microstep: 5691.15 | bwd_inner_microstep: 5638.70 | bwd_allreduce_microstep: 52.40 | step_microstep: 18.94
[2025-04-25 13:47:00,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.55 | bwd: 5691.16 | bwd_inner: 5638.70 | bwd_allreduce: 52.42 | step: 18.94

  6%|▌         | 2413/41250 [5:49:25<93:12:10,  8.64s/it]
                                                         
{'loss': 0.1575, 'grad_norm': 1.188953161239624, 'learning_rate': 3.991494753538529e-05, 'epoch': 0.58}

  6%|▌         | 2413/41250 [5:49:25<93:12:10,  8.64s/it][2025-04-25 13:47:09,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 13:47:09,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5691.12 | bwd_inner_microstep: 5654.66 | bwd_allreduce_microstep: 36.42 | step_microstep: 18.80
[2025-04-25 13:47:09,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5691.14 | bwd_inner: 5654.66 | bwd_allreduce: 36.43 | step: 18.80

  6%|▌         | 2414/41250 [5:49:34<93:05:08,  8.63s/it]
                                                         
{'loss': 0.1543, 'grad_norm': 1.0242865085601807, 'learning_rate': 3.991480280654323e-05, 'epoch': 0.59}

  6%|▌         | 2414/41250 [5:49:34<93:05:08,  8.63s/it][2025-04-25 13:47:17,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 13:47:17,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.44 | bwd_microstep: 5701.36 | bwd_inner_microstep: 5688.65 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.62
[2025-04-25 13:47:17,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.44 | bwd: 5701.38 | bwd_inner: 5688.65 | bwd_allreduce: 12.68 | step: 18.62

  6%|▌         | 2415/41250 [5:49:43<93:04:59,  8.63s/it]
                                                         
{'loss': 0.0783, 'grad_norm': 0.7225725054740906, 'learning_rate': 3.991465795493034e-05, 'epoch': 0.59}

  6%|▌         | 2415/41250 [5:49:43<93:04:59,  8.63s/it][2025-04-25 13:47:26,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:47:26,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.55 | bwd_microstep: 5711.86 | bwd_inner_microstep: 5698.61 | bwd_allreduce_microstep: 13.20 | step_microstep: 19.00
[2025-04-25 13:47:26,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.55 | bwd: 5711.87 | bwd_inner: 5698.61 | bwd_allreduce: 13.21 | step: 19.00

  6%|▌         | 2416/41250 [5:49:51<93:09:17,  8.64s/it]
                                                         
{'loss': 0.2524, 'grad_norm': 1.6437240839004517, 'learning_rate': 3.991451298054748e-05, 'epoch': 0.59}

  6%|▌         | 2416/41250 [5:49:51<93:09:17,  8.64s/it][2025-04-25 13:47:35,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 13:47:35,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.87 | bwd_microstep: 5755.25 | bwd_inner_microstep: 5654.72 | bwd_allreduce_microstep: 100.48 | step_microstep: 18.66
[2025-04-25 13:47:35,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.87 | bwd: 5755.26 | bwd_inner: 5654.72 | bwd_allreduce: 100.50 | step: 18.67

  6%|▌         | 2417/41250 [5:50:00<93:14:17,  8.64s/it]
                                                         
{'loss': 0.1932, 'grad_norm': 0.9269722104072571, 'learning_rate': 3.991436788339558e-05, 'epoch': 0.59}

  6%|▌         | 2417/41250 [5:50:00<93:14:17,  8.64s/it][2025-04-25 13:47:43,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:47:43,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5840.58 | bwd_inner_microstep: 5647.87 | bwd_allreduce_microstep: 192.66 | step_microstep: 18.72
[2025-04-25 13:47:43,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5840.59 | bwd_inner: 5647.87 | bwd_allreduce: 192.68 | step: 18.72

  6%|▌         | 2418/41250 [5:50:09<93:34:38,  8.68s/it]
                                                         
{'loss': 0.1992, 'grad_norm': 1.2404060363769531, 'learning_rate': 3.991422266347551e-05, 'epoch': 0.59}

  6%|▌         | 2418/41250 [5:50:09<93:34:38,  8.68s/it][2025-04-25 13:47:52,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 13:47:52,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.58 | bwd_microstep: 5717.92 | bwd_inner_microstep: 5705.05 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.00
[2025-04-25 13:47:52,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.58 | bwd: 5717.93 | bwd_inner: 5705.05 | bwd_allreduce: 12.84 | step: 19.00

  6%|▌         | 2419/41250 [5:50:17<93:30:59,  8.67s/it]
                                                         
{'loss': 0.122, 'grad_norm': 1.1352120637893677, 'learning_rate': 3.991407732078818e-05, 'epoch': 0.59}

  6%|▌         | 2419/41250 [5:50:17<93:30:59,  8.67s/it][2025-04-25 13:48:01,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:48:01,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.10 | bwd_microstep: 5662.05 | bwd_inner_microstep: 5649.12 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.58
[2025-04-25 13:48:01,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.10 | bwd: 5662.07 | bwd_inner: 5649.12 | bwd_allreduce: 12.91 | step: 18.58

  6%|▌         | 2420/41250 [5:50:26<93:12:54,  8.64s/it]
                                                         
{'loss': 0.1615, 'grad_norm': 1.9922857284545898, 'learning_rate': 3.991393185533448e-05, 'epoch': 0.59}

  6%|▌         | 2420/41250 [5:50:26<93:12:54,  8.64s/it][2025-04-25 13:48:09,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.33 | optimizer_step: 1.03
[2025-04-25 13:48:09,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.30 | bwd_microstep: 6004.71 | bwd_inner_microstep: 5651.88 | bwd_allreduce_microstep: 352.76 | step_microstep: 20.06
[2025-04-25 13:48:09,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.30 | bwd: 6004.72 | bwd_inner: 5651.88 | bwd_allreduce: 352.79 | step: 20.06

  6%|▌         | 2421/41250 [5:50:35<94:05:22,  8.72s/it]
                                                         
{'loss': 0.5022, 'grad_norm': 2.188178300857544, 'learning_rate': 3.99137862671153e-05, 'epoch': 0.59}

  6%|▌         | 2421/41250 [5:50:35<94:05:22,  8.72s/it][2025-04-25 13:48:18,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:48:18,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.20 | bwd_microstep: 5719.44 | bwd_inner_microstep: 5706.77 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.83
[2025-04-25 13:48:18,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.20 | bwd: 5719.46 | bwd_inner: 5706.77 | bwd_allreduce: 12.65 | step: 18.83

  6%|▌         | 2422/41250 [5:50:43<93:51:04,  8.70s/it]
                                                         
{'loss': 0.1635, 'grad_norm': 1.2846513986587524, 'learning_rate': 3.991364055613155e-05, 'epoch': 0.59}

  6%|▌         | 2422/41250 [5:50:43<93:51:04,  8.70s/it][2025-04-25 13:48:27,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 13:48:27,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.50 | bwd_microstep: 5711.68 | bwd_inner_microstep: 5698.94 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.29
[2025-04-25 13:48:27,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5711.69 | bwd_inner: 5698.94 | bwd_allreduce: 12.71 | step: 19.29

  6%|▌         | 2423/41250 [5:50:52<93:41:22,  8.69s/it]
                                                         
{'loss': 0.1098, 'grad_norm': 1.1789331436157227, 'learning_rate': 3.991349472238413e-05, 'epoch': 0.59}

  6%|▌         | 2423/41250 [5:50:52<93:41:22,  8.69s/it][2025-04-25 13:48:35,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:48:35,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5713.42 | bwd_inner_microstep: 5696.06 | bwd_allreduce_microstep: 17.32 | step_microstep: 18.66
[2025-04-25 13:48:35,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5713.44 | bwd_inner: 5696.06 | bwd_allreduce: 17.34 | step: 18.66

  6%|▌         | 2424/41250 [5:51:01<93:34:24,  8.68s/it]
                                                         
{'loss': 0.0865, 'grad_norm': 0.7592039704322815, 'learning_rate': 3.9913348765873924e-05, 'epoch': 0.59}

  6%|▌         | 2424/41250 [5:51:01<93:34:24,  8.68s/it][2025-04-25 13:48:44,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 13:48:44,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5708.20 | bwd_inner_microstep: 5686.13 | bwd_allreduce_microstep: 22.03 | step_microstep: 18.58
[2025-04-25 13:48:44,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5708.21 | bwd_inner: 5686.13 | bwd_allreduce: 22.04 | step: 18.59

  6%|▌         | 2425/41250 [5:51:09<93:26:43,  8.66s/it]
                                                         
{'loss': 0.3189, 'grad_norm': 1.973188877105713, 'learning_rate': 3.991320268660184e-05, 'epoch': 0.59}

  6%|▌         | 2425/41250 [5:51:09<93:26:43,  8.66s/it][2025-04-25 13:48:53,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 13:48:53,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.78 | bwd_microstep: 5842.54 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 144.85 | step_microstep: 19.26
[2025-04-25 13:48:53,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.78 | bwd: 5842.56 | bwd_inner: 5697.65 | bwd_allreduce: 144.87 | step: 19.26

  6%|▌         | 2426/41250 [5:51:18<93:47:35,  8.70s/it]
                                                         
{'loss': 0.1413, 'grad_norm': 1.298264503479004, 'learning_rate': 3.9913056484568774e-05, 'epoch': 0.59}

  6%|▌         | 2426/41250 [5:51:18<93:47:35,  8.70s/it][2025-04-25 13:49:02,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:49:02,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.86 | bwd_microstep: 5782.52 | bwd_inner_microstep: 5769.77 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.83
[2025-04-25 13:49:02,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.86 | bwd: 5782.53 | bwd_inner: 5769.77 | bwd_allreduce: 12.72 | step: 18.84

  6%|▌         | 2427/41250 [5:51:27<93:57:20,  8.71s/it]
                                                         
{'loss': 0.1941, 'grad_norm': 1.6589820384979248, 'learning_rate': 3.991291015977564e-05, 'epoch': 0.59}

  6%|▌         | 2427/41250 [5:51:27<93:57:20,  8.71s/it][2025-04-25 13:49:10,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.11 | optimizer_step: 1.04
[2025-04-25 13:49:10,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.30 | bwd_microstep: 5722.70 | bwd_inner_microstep: 5693.92 | bwd_allreduce_microstep: 28.73 | step_microstep: 19.50
[2025-04-25 13:49:10,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.30 | bwd: 5722.71 | bwd_inner: 5693.92 | bwd_allreduce: 28.75 | step: 19.50

  6%|▌         | 2428/41250 [5:51:36<93:45:05,  8.69s/it]
                                                         
{'loss': 0.1959, 'grad_norm': 1.5211663246154785, 'learning_rate': 3.9912763712223324e-05, 'epoch': 0.59}

  6%|▌         | 2428/41250 [5:51:36<93:45:05,  8.69s/it][2025-04-25 13:49:19,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:49:19,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.02 | bwd_microstep: 5736.65 | bwd_inner_microstep: 5649.63 | bwd_allreduce_microstep: 86.98 | step_microstep: 18.71
[2025-04-25 13:49:19,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.02 | bwd: 5736.66 | bwd_inner: 5649.63 | bwd_allreduce: 86.99 | step: 18.72

  6%|▌         | 2429/41250 [5:51:44<93:35:43,  8.68s/it]
                                                         
{'loss': 0.2248, 'grad_norm': 1.3864943981170654, 'learning_rate': 3.991261714191274e-05, 'epoch': 0.59}

  6%|▌         | 2429/41250 [5:51:44<93:35:43,  8.68s/it][2025-04-25 13:49:28,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:49:28,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.49 | bwd_microstep: 5764.81 | bwd_inner_microstep: 5648.25 | bwd_allreduce_microstep: 116.51 | step_microstep: 18.97
[2025-04-25 13:49:28,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.49 | bwd: 5764.84 | bwd_inner: 5648.25 | bwd_allreduce: 116.53 | step: 18.97

  6%|▌         | 2430/41250 [5:51:53<93:33:31,  8.68s/it]
                                                         
{'loss': 0.123, 'grad_norm': 1.008357048034668, 'learning_rate': 3.991247044884479e-05, 'epoch': 0.59}

  6%|▌         | 2430/41250 [5:51:53<93:33:31,  8.68s/it][2025-04-25 13:49:36,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 13:49:36,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.05 | bwd_microstep: 5728.91 | bwd_inner_microstep: 5685.73 | bwd_allreduce_microstep: 43.13 | step_microstep: 19.03
[2025-04-25 13:49:36,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.05 | bwd: 5728.92 | bwd_inner: 5685.73 | bwd_allreduce: 43.14 | step: 19.03

  6%|▌         | 2431/41250 [5:52:02<93:29:24,  8.67s/it]
                                                         
{'loss': 0.0835, 'grad_norm': 1.1304683685302734, 'learning_rate': 3.9912323633020366e-05, 'epoch': 0.59}

  6%|▌         | 2431/41250 [5:52:02<93:29:24,  8.67s/it][2025-04-25 13:49:45,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-25 13:49:45,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.49 | bwd_microstep: 5744.63 | bwd_inner_microstep: 5641.78 | bwd_allreduce_microstep: 102.79 | step_microstep: 20.18
[2025-04-25 13:49:45,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.49 | bwd: 5744.65 | bwd_inner: 5641.78 | bwd_allreduce: 102.81 | step: 20.18

  6%|▌         | 2432/41250 [5:52:10<93:25:21,  8.66s/it]
                                                         
{'loss': 0.2793, 'grad_norm': 1.9851680994033813, 'learning_rate': 3.99121766944404e-05, 'epoch': 0.59}

  6%|▌         | 2432/41250 [5:52:10<93:25:21,  8.66s/it][2025-04-25 13:49:54,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 13:49:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.74 | bwd_microstep: 5765.31 | bwd_inner_microstep: 5652.71 | bwd_allreduce_microstep: 112.55 | step_microstep: 19.21
[2025-04-25 13:49:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.74 | bwd: 5765.33 | bwd_inner: 5652.71 | bwd_allreduce: 112.57 | step: 19.21

  6%|▌         | 2433/41250 [5:52:19<93:27:45,  8.67s/it]
                                                         
{'loss': 0.1015, 'grad_norm': 0.7413845062255859, 'learning_rate': 3.991202963310577e-05, 'epoch': 0.59}

  6%|▌         | 2433/41250 [5:52:19<93:27:45,  8.67s/it][2025-04-25 13:50:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 13:50:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5725.79 | bwd_inner_microstep: 5633.41 | bwd_allreduce_microstep: 92.34 | step_microstep: 19.31
[2025-04-25 13:50:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5725.80 | bwd_inner: 5633.41 | bwd_allreduce: 92.36 | step: 19.32

  6%|▌         | 2434/41250 [5:52:28<93:21:48,  8.66s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 1.180144190788269, 'learning_rate': 3.9911882449017394e-05, 'epoch': 0.59}

  6%|▌         | 2434/41250 [5:52:28<93:21:48,  8.66s/it][2025-04-25 13:50:11,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:50:11,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.85 | bwd_microstep: 5717.90 | bwd_inner_microstep: 5705.15 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.71
[2025-04-25 13:50:11,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.85 | bwd: 5717.91 | bwd_inner: 5705.15 | bwd_allreduce: 12.72 | step: 18.71

  6%|▌         | 2435/41250 [5:52:36<93:19:37,  8.66s/it]
                                                         
{'loss': 0.1421, 'grad_norm': 0.976792573928833, 'learning_rate': 3.991173514217618e-05, 'epoch': 0.59}

  6%|▌         | 2435/41250 [5:52:36<93:19:37,  8.66s/it][2025-04-25 13:50:19,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:50:19,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5752.26 | bwd_inner_microstep: 5649.10 | bwd_allreduce_microstep: 103.11 | step_microstep: 18.63
[2025-04-25 13:50:19,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.69 | bwd: 5752.27 | bwd_inner: 5649.10 | bwd_allreduce: 103.13 | step: 18.63

  6%|▌         | 2436/41250 [5:52:45<93:20:06,  8.66s/it]
                                                         
{'loss': 0.0583, 'grad_norm': 0.7184491157531738, 'learning_rate': 3.991158771258304e-05, 'epoch': 0.59}

  6%|▌         | 2436/41250 [5:52:45<93:20:06,  8.66s/it][2025-04-25 13:50:28,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 13:50:28,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.81 | bwd_microstep: 5702.31 | bwd_inner_microstep: 5633.10 | bwd_allreduce_microstep: 69.15 | step_microstep: 18.88
[2025-04-25 13:50:28,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.81 | bwd: 5702.32 | bwd_inner: 5633.10 | bwd_allreduce: 69.17 | step: 18.88

  6%|▌         | 2437/41250 [5:52:53<93:10:18,  8.64s/it]
                                                         
{'loss': 0.14, 'grad_norm': 1.481755018234253, 'learning_rate': 3.9911440160238876e-05, 'epoch': 0.59}

  6%|▌         | 2437/41250 [5:52:53<93:10:18,  8.64s/it][2025-04-25 13:50:37,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 13:50:37,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.04 | bwd_microstep: 5699.06 | bwd_inner_microstep: 5686.21 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.65
[2025-04-25 13:50:37,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.04 | bwd: 5699.08 | bwd_inner: 5686.21 | bwd_allreduce: 12.83 | step: 19.66

  6%|▌         | 2438/41250 [5:53:02<93:07:20,  8.64s/it]
                                                         
{'loss': 0.0981, 'grad_norm': 0.7187312841415405, 'learning_rate': 3.9911292485144595e-05, 'epoch': 0.59}

  6%|▌         | 2438/41250 [5:53:02<93:07:20,  8.64s/it][2025-04-25 13:50:45,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.27 | optimizer_step: 1.03
[2025-04-25 13:50:45,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.25 | bwd_microstep: 5717.87 | bwd_inner_microstep: 5680.85 | bwd_allreduce_microstep: 36.96 | step_microstep: 19.80
[2025-04-25 13:50:45,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.25 | bwd: 5717.89 | bwd_inner: 5680.85 | bwd_allreduce: 36.99 | step: 19.80

  6%|▌         | 2439/41250 [5:53:11<93:08:07,  8.64s/it]
                                                         
{'loss': 0.1582, 'grad_norm': 2.329261302947998, 'learning_rate': 3.991114468730111e-05, 'epoch': 0.59}

  6%|▌         | 2439/41250 [5:53:11<93:08:07,  8.64s/it][2025-04-25 13:50:54,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:50:54,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.09 | bwd_microstep: 5687.15 | bwd_inner_microstep: 5674.47 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.82
[2025-04-25 13:50:54,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.09 | bwd: 5687.16 | bwd_inner: 5674.47 | bwd_allreduce: 12.65 | step: 18.82

  6%|▌         | 2440/41250 [5:53:19<93:04:05,  8.63s/it]
                                                         
{'loss': 0.0975, 'grad_norm': 1.1804845333099365, 'learning_rate': 3.991099676670935e-05, 'epoch': 0.59}

  6%|▌         | 2440/41250 [5:53:19<93:04:05,  8.63s/it][2025-04-25 13:51:03,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.25 | optimizer_step: 1.23
[2025-04-25 13:51:03,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.39 | bwd_microstep: 5766.82 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 71.10 | step_microstep: 20.15
[2025-04-25 13:51:03,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.39 | bwd: 5766.83 | bwd_inner: 5695.67 | bwd_allreduce: 71.12 | step: 20.15

  6%|▌         | 2441/41250 [5:53:28<93:18:31,  8.66s/it]
                                                         
{'loss': 0.0853, 'grad_norm': 1.0507405996322632, 'learning_rate': 3.99108487233702e-05, 'epoch': 0.59}

  6%|▌         | 2441/41250 [5:53:28<93:18:31,  8.66s/it][2025-04-25 13:51:11,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:51:11,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.75 | bwd_microstep: 5739.83 | bwd_inner_microstep: 5658.21 | bwd_allreduce_microstep: 81.58 | step_microstep: 18.88
[2025-04-25 13:51:11,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.75 | bwd: 5739.84 | bwd_inner: 5658.21 | bwd_allreduce: 81.59 | step: 18.88

  6%|▌         | 2442/41250 [5:53:37<93:19:49,  8.66s/it]
                                                         
{'loss': 0.0758, 'grad_norm': 0.8203034400939941, 'learning_rate': 3.991070055728458e-05, 'epoch': 0.59}

  6%|▌         | 2442/41250 [5:53:37<93:19:49,  8.66s/it][2025-04-25 13:51:20,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 13:51:20,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.23 | bwd_microstep: 5776.83 | bwd_inner_microstep: 5763.86 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.09
[2025-04-25 13:51:20,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.23 | bwd: 5776.84 | bwd_inner: 5763.86 | bwd_allreduce: 12.94 | step: 19.09

  6%|▌         | 2443/41250 [5:53:45<93:38:19,  8.69s/it]
                                                         
{'loss': 0.2137, 'grad_norm': 1.6909428834915161, 'learning_rate': 3.991055226845341e-05, 'epoch': 0.59}

  6%|▌         | 2443/41250 [5:53:45<93:38:19,  8.69s/it][2025-04-25 13:51:29,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 13:51:29,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.55 | bwd_microstep: 5759.80 | bwd_inner_microstep: 5660.54 | bwd_allreduce_microstep: 99.21 | step_microstep: 19.01
[2025-04-25 13:51:29,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.55 | bwd: 5759.81 | bwd_inner: 5660.54 | bwd_allreduce: 99.22 | step: 19.01

  6%|▌         | 2444/41250 [5:53:54<93:36:04,  8.68s/it]
                                                         
{'loss': 0.1403, 'grad_norm': 1.6376066207885742, 'learning_rate': 3.991040385687761e-05, 'epoch': 0.59}

  6%|▌         | 2444/41250 [5:53:54<93:36:04,  8.68s/it][2025-04-25 13:51:37,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:51:37,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5693.94 | bwd_inner_microstep: 5656.35 | bwd_allreduce_microstep: 37.55 | step_microstep: 18.89
[2025-04-25 13:51:37,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5693.95 | bwd_inner: 5656.35 | bwd_allreduce: 37.56 | step: 18.89

  6%|▌         | 2445/41250 [5:54:03<93:20:21,  8.66s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 1.2880942821502686, 'learning_rate': 3.991025532255808e-05, 'epoch': 0.59}

  6%|▌         | 2445/41250 [5:54:03<93:20:21,  8.66s/it][2025-04-25 13:51:46,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.16
[2025-04-25 13:51:46,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5700.01 | bwd_allreduce_microstep: 12.85 | step_microstep: 20.09
[2025-04-25 13:51:46,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5712.92 | bwd_inner: 5700.01 | bwd_allreduce: 12.87 | step: 20.09

  6%|▌         | 2446/41250 [5:54:11<93:18:58,  8.66s/it]
                                                         
{'loss': 0.1539, 'grad_norm': 3.151322841644287, 'learning_rate': 3.991010666549574e-05, 'epoch': 0.59}

  6%|▌         | 2446/41250 [5:54:11<93:18:58,  8.66s/it][2025-04-25 13:51:55,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 13:51:55,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.46 | bwd_microstep: 5677.68 | bwd_inner_microstep: 5655.84 | bwd_allreduce_microstep: 21.79 | step_microstep: 18.92
[2025-04-25 13:51:55,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.46 | bwd: 5677.69 | bwd_inner: 5655.84 | bwd_allreduce: 21.81 | step: 18.93

  6%|▌         | 2447/41250 [5:54:20<93:07:06,  8.64s/it]
                                                         
{'loss': 0.1106, 'grad_norm': 1.0791404247283936, 'learning_rate': 3.990995788569151e-05, 'epoch': 0.59}

  6%|▌         | 2447/41250 [5:54:20<93:07:06,  8.64s/it][2025-04-25 13:52:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:52:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.67 | bwd_microstep: 5711.11 | bwd_inner_microstep: 5698.49 | bwd_allreduce_microstep: 12.58 | step_microstep: 19.12
[2025-04-25 13:52:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.67 | bwd: 5711.13 | bwd_inner: 5698.49 | bwd_allreduce: 12.59 | step: 19.13

  6%|▌         | 2448/41250 [5:54:29<93:09:22,  8.64s/it]
                                                         
{'loss': 0.3781, 'grad_norm': 2.4926538467407227, 'learning_rate': 3.990980898314631e-05, 'epoch': 0.59}

  6%|▌         | 2448/41250 [5:54:29<93:09:22,  8.64s/it][2025-04-25 13:52:12,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 13:52:12,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5764.15 | bwd_inner_microstep: 5697.96 | bwd_allreduce_microstep: 66.14 | step_microstep: 19.05
[2025-04-25 13:52:12,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5764.16 | bwd_inner: 5697.96 | bwd_allreduce: 66.16 | step: 19.05

  6%|▌         | 2449/41250 [5:54:37<93:19:49,  8.66s/it]
                                                         
{'loss': 0.2099, 'grad_norm': 2.2964260578155518, 'learning_rate': 3.9909659957861045e-05, 'epoch': 0.59}

  6%|▌         | 2449/41250 [5:54:37<93:19:49,  8.66s/it][2025-04-25 13:52:21,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:52:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.03 | bwd_microstep: 5726.85 | bwd_inner_microstep: 5713.91 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.08
[2025-04-25 13:52:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.03 | bwd: 5726.86 | bwd_inner: 5713.91 | bwd_allreduce: 12.92 | step: 19.08

  6%|▌         | 2450/41250 [5:54:46<93:21:52,  8.66s/it]
                                                         
{'loss': 0.278, 'grad_norm': 2.010481357574463, 'learning_rate': 3.990951080983665e-05, 'epoch': 0.59}

  6%|▌         | 2450/41250 [5:54:46<93:21:52,  8.66s/it][2025-04-25 13:52:30,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-25 13:52:30,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2947.73 | bwd_microstep: 5899.33 | bwd_inner_microstep: 5886.36 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.06
[2025-04-25 13:52:30,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2947.73 | bwd: 5899.35 | bwd_inner: 5886.36 | bwd_allreduce: 12.94 | step: 19.06

  6%|▌         | 2451/41250 [5:54:55<94:14:30,  8.74s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 1.8661658763885498, 'learning_rate': 3.990936153907403e-05, 'epoch': 0.59}

  6%|▌         | 2451/41250 [5:54:55<94:14:30,  8.74s/it][2025-04-25 13:52:38,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.05 | optimizer_step: 1.21
[2025-04-25 13:52:38,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.43 | bwd_microstep: 5752.12 | bwd_inner_microstep: 5669.41 | bwd_allreduce_microstep: 82.67 | step_microstep: 19.27
[2025-04-25 13:52:38,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.43 | bwd: 5752.13 | bwd_inner: 5669.41 | bwd_allreduce: 82.68 | step: 19.27

  6%|▌         | 2452/41250 [5:55:04<94:01:30,  8.72s/it]
                                                         
{'loss': 0.1091, 'grad_norm': 2.08454966545105, 'learning_rate': 3.9909212145574125e-05, 'epoch': 0.59}

  6%|▌         | 2452/41250 [5:55:04<94:01:30,  8.72s/it][2025-04-25 13:52:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-25 13:52:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.52 | bwd_microstep: 5758.20 | bwd_inner_microstep: 5720.96 | bwd_allreduce_microstep: 37.19 | step_microstep: 18.91
[2025-04-25 13:52:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.52 | bwd: 5758.22 | bwd_inner: 5720.96 | bwd_allreduce: 37.22 | step: 18.91

  6%|▌         | 2453/41250 [5:55:12<93:56:43,  8.72s/it]
                                                         
{'loss': 0.0673, 'grad_norm': 0.6723281145095825, 'learning_rate': 3.9909062629337826e-05, 'epoch': 0.59}

  6%|▌         | 2453/41250 [5:55:12<93:56:43,  8.72s/it][2025-04-25 13:52:56,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:52:56,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.15 | bwd_microstep: 5796.48 | bwd_inner_microstep: 5700.04 | bwd_allreduce_microstep: 96.39 | step_microstep: 18.43
[2025-04-25 13:52:56,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.15 | bwd: 5796.49 | bwd_inner: 5700.04 | bwd_allreduce: 96.41 | step: 18.43

  6%|▌         | 2454/41250 [5:55:21<93:59:08,  8.72s/it]
                                                         
{'loss': 0.1214, 'grad_norm': 2.3056631088256836, 'learning_rate': 3.990891299036609e-05, 'epoch': 0.59}

  6%|▌         | 2454/41250 [5:55:21<93:59:08,  8.72s/it][2025-04-25 13:53:04,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 13:53:04,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.80 | bwd_microstep: 5782.49 | bwd_inner_microstep: 5713.13 | bwd_allreduce_microstep: 69.32 | step_microstep: 18.33
[2025-04-25 13:53:04,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.80 | bwd: 5782.50 | bwd_inner: 5713.13 | bwd_allreduce: 69.33 | step: 18.33

  6%|▌         | 2455/41250 [5:55:30<94:01:20,  8.72s/it]
                                                         
{'loss': 0.1755, 'grad_norm': 1.7508891820907593, 'learning_rate': 3.990876322865981e-05, 'epoch': 0.6}

  6%|▌         | 2455/41250 [5:55:30<94:01:20,  8.72s/it][2025-04-25 13:53:13,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:53:13,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 5766.83 | bwd_inner_microstep: 5665.82 | bwd_allreduce_microstep: 100.97 | step_microstep: 18.81
[2025-04-25 13:53:13,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5766.85 | bwd_inner: 5665.82 | bwd_allreduce: 100.99 | step: 18.82

  6%|▌         | 2456/41250 [5:55:38<93:54:58,  8.72s/it]
                                                         
{'loss': 0.1943, 'grad_norm': 2.0890183448791504, 'learning_rate': 3.9908613344219914e-05, 'epoch': 0.6}

  6%|▌         | 2456/41250 [5:55:38<93:54:58,  8.72s/it][2025-04-25 13:53:22,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-25 13:53:22,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.09 | bwd_microstep: 5775.04 | bwd_inner_microstep: 5666.81 | bwd_allreduce_microstep: 108.18 | step_microstep: 19.26
[2025-04-25 13:53:22,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.09 | bwd: 5775.06 | bwd_inner: 5666.81 | bwd_allreduce: 108.20 | step: 19.26

  6%|▌         | 2457/41250 [5:55:47<93:51:36,  8.71s/it]
                                                         
{'loss': 0.0416, 'grad_norm': 0.35042959451675415, 'learning_rate': 3.9908463337047344e-05, 'epoch': 0.6}

  6%|▌         | 2457/41250 [5:55:47<93:51:36,  8.71s/it][2025-04-25 13:53:30,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 13:53:30,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.86 | bwd_microstep: 5706.97 | bwd_inner_microstep: 5694.23 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.75
[2025-04-25 13:53:30,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.86 | bwd: 5706.98 | bwd_inner: 5694.23 | bwd_allreduce: 12.70 | step: 18.76

  6%|▌         | 2458/41250 [5:55:56<93:38:04,  8.69s/it]
                                                         
{'loss': 0.2765, 'grad_norm': 2.144167900085449, 'learning_rate': 3.9908313207143006e-05, 'epoch': 0.6}

  6%|▌         | 2458/41250 [5:55:56<93:38:04,  8.69s/it][2025-04-25 13:53:39,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 13:53:39,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.22 | bwd_microstep: 5719.94 | bwd_inner_microstep: 5707.19 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.71
[2025-04-25 13:53:39,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.23 | bwd: 5719.95 | bwd_inner: 5707.19 | bwd_allreduce: 12.73 | step: 18.71

  6%|▌         | 2459/41250 [5:56:04<93:33:16,  8.68s/it]
                                                         
{'loss': 0.1806, 'grad_norm': 1.3919543027877808, 'learning_rate': 3.9908162954507836e-05, 'epoch': 0.6}

  6%|▌         | 2459/41250 [5:56:04<93:33:16,  8.68s/it][2025-04-25 13:53:48,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 13:53:48,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.06 | bwd_microstep: 5747.90 | bwd_inner_microstep: 5667.19 | bwd_allreduce_microstep: 80.67 | step_microstep: 18.60
[2025-04-25 13:53:48,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.06 | bwd: 5747.92 | bwd_inner: 5667.19 | bwd_allreduce: 80.69 | step: 18.60

  6%|▌         | 2460/41250 [5:56:13<93:31:12,  8.68s/it]
                                                         
{'loss': 0.1994, 'grad_norm': 2.050422191619873, 'learning_rate': 3.990801257914276e-05, 'epoch': 0.6}

  6%|▌         | 2460/41250 [5:56:13<93:31:12,  8.68s/it][2025-04-25 13:53:56,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:53:56,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.93 | bwd_microstep: 5744.47 | bwd_inner_microstep: 5708.05 | bwd_allreduce_microstep: 36.37 | step_microstep: 18.97
[2025-04-25 13:53:56,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.93 | bwd: 5744.49 | bwd_inner: 5708.05 | bwd_allreduce: 36.39 | step: 18.97

  6%|▌         | 2461/41250 [5:56:22<93:31:32,  8.68s/it]
                                                         
{'loss': 0.3149, 'grad_norm': 2.302178382873535, 'learning_rate': 3.990786208104871e-05, 'epoch': 0.6}

  6%|▌         | 2461/41250 [5:56:22<93:31:32,  8.68s/it][2025-04-25 13:54:05,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 13:54:05,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5730.58 | bwd_inner_microstep: 5699.78 | bwd_allreduce_microstep: 30.75 | step_microstep: 18.73
[2025-04-25 13:54:05,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5730.59 | bwd_inner: 5699.78 | bwd_allreduce: 30.76 | step: 18.73

  6%|▌         | 2462/41250 [5:56:30<93:28:33,  8.68s/it]
                                                         
{'loss': 0.1508, 'grad_norm': 3.041459560394287, 'learning_rate': 3.990771146022659e-05, 'epoch': 0.6}

  6%|▌         | 2462/41250 [5:56:30<93:28:33,  8.68s/it][2025-04-25 13:54:14,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 13:54:14,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.21 | bwd_microstep: 5896.44 | bwd_inner_microstep: 5659.19 | bwd_allreduce_microstep: 237.20 | step_microstep: 19.03
[2025-04-25 13:54:14,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.21 | bwd: 5896.46 | bwd_inner: 5659.19 | bwd_allreduce: 237.22 | step: 19.03

  6%|▌         | 2463/41250 [5:56:39<93:54:31,  8.72s/it]
                                                         
{'loss': 0.0716, 'grad_norm': 0.9994687438011169, 'learning_rate': 3.990756071667735e-05, 'epoch': 0.6}

  6%|▌         | 2463/41250 [5:56:39<93:54:31,  8.72s/it][2025-04-25 13:54:23,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:54:23,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5763.97 | bwd_inner_microstep: 5659.57 | bwd_allreduce_microstep: 104.36 | step_microstep: 18.91
[2025-04-25 13:54:23,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5763.99 | bwd_inner: 5659.57 | bwd_allreduce: 104.38 | step: 18.91

  6%|▌         | 2464/41250 [5:56:48<93:46:30,  8.70s/it]
                                                         
{'loss': 0.4407, 'grad_norm': 2.967395305633545, 'learning_rate': 3.990740985040191e-05, 'epoch': 0.6}

  6%|▌         | 2464/41250 [5:56:48<93:46:30,  8.70s/it][2025-04-25 13:54:31,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:54:31,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5699.49 | bwd_inner_microstep: 5658.67 | bwd_allreduce_microstep: 40.78 | step_microstep: 18.77
[2025-04-25 13:54:31,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5699.50 | bwd_inner: 5658.66 | bwd_allreduce: 40.79 | step: 18.78

  6%|▌         | 2465/41250 [5:56:57<93:28:31,  8.68s/it]
                                                         
{'loss': 0.274, 'grad_norm': 1.8081398010253906, 'learning_rate': 3.990725886140121e-05, 'epoch': 0.6}

  6%|▌         | 2465/41250 [5:56:57<93:28:31,  8.68s/it][2025-04-25 13:54:40,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 13:54:40,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.87 | bwd_microstep: 5762.87 | bwd_inner_microstep: 5696.15 | bwd_allreduce_microstep: 66.67 | step_microstep: 18.80
[2025-04-25 13:54:40,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.87 | bwd: 5762.88 | bwd_inner: 5696.15 | bwd_allreduce: 66.69 | step: 18.81

  6%|▌         | 2466/41250 [5:57:05<93:31:19,  8.68s/it]
                                                         
{'loss': 0.1051, 'grad_norm': 1.0294452905654907, 'learning_rate': 3.9907107749676175e-05, 'epoch': 0.6}

  6%|▌         | 2466/41250 [5:57:05<93:31:19,  8.68s/it][2025-04-25 13:54:49,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-25 13:54:49,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.38 | bwd_microstep: 5709.14 | bwd_inner_microstep: 5695.61 | bwd_allreduce_microstep: 13.48 | step_microstep: 19.34
[2025-04-25 13:54:49,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.39 | bwd: 5709.15 | bwd_inner: 5695.61 | bwd_allreduce: 13.50 | step: 19.34

  6%|▌         | 2467/41250 [5:57:14<93:23:25,  8.67s/it]
                                                         
{'loss': 0.2232, 'grad_norm': 2.6327123641967773, 'learning_rate': 3.9906956515227735e-05, 'epoch': 0.6}

  6%|▌         | 2467/41250 [5:57:14<93:23:25,  8.67s/it][2025-04-25 13:54:57,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-25 13:54:57,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5699.31 | bwd_inner_microstep: 5685.86 | bwd_allreduce_microstep: 13.40 | step_microstep: 19.79
[2025-04-25 13:54:57,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5699.32 | bwd_inner: 5685.86 | bwd_allreduce: 13.42 | step: 19.79

  6%|▌         | 2468/41250 [5:57:23<93:17:52,  8.66s/it]
                                                         
{'loss': 0.2959, 'grad_norm': 2.8317811489105225, 'learning_rate': 3.9906805158056825e-05, 'epoch': 0.6}

  6%|▌         | 2468/41250 [5:57:23<93:17:52,  8.66s/it][2025-04-25 13:55:06,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:55:06,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5722.95 | bwd_inner_microstep: 5710.08 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.74
[2025-04-25 13:55:06,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5722.96 | bwd_inner: 5710.08 | bwd_allreduce: 12.84 | step: 18.74

  6%|▌         | 2469/41250 [5:57:31<93:17:21,  8.66s/it]
                                                         
{'loss': 0.3911, 'grad_norm': 1.6679328680038452, 'learning_rate': 3.990665367816437e-05, 'epoch': 0.6}

  6%|▌         | 2469/41250 [5:57:31<93:17:21,  8.66s/it][2025-04-25 13:55:14,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 13:55:14,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.15 | bwd_microstep: 5687.09 | bwd_inner_microstep: 5662.74 | bwd_allreduce_microstep: 24.30 | step_microstep: 19.36
[2025-04-25 13:55:14,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.15 | bwd: 5687.11 | bwd_inner: 5662.74 | bwd_allreduce: 24.32 | step: 19.36

  6%|▌         | 2470/41250 [5:57:40<93:07:25,  8.64s/it]
                                                         
{'loss': 0.1782, 'grad_norm': 1.403028130531311, 'learning_rate': 3.9906502075551314e-05, 'epoch': 0.6}

  6%|▌         | 2470/41250 [5:57:40<93:07:25,  8.64s/it][2025-04-25 13:55:23,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 13:55:23,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.42 | bwd_microstep: 5702.62 | bwd_inner_microstep: 5642.08 | bwd_allreduce_microstep: 60.49 | step_microstep: 19.44
[2025-04-25 13:55:23,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.42 | bwd: 5702.63 | bwd_inner: 5642.08 | bwd_allreduce: 60.51 | step: 19.44

  6%|▌         | 2471/41250 [5:57:48<93:00:43,  8.63s/it]
                                                         
{'loss': 0.0864, 'grad_norm': 0.9739531874656677, 'learning_rate': 3.990635035021859e-05, 'epoch': 0.6}

  6%|▌         | 2471/41250 [5:57:48<93:00:43,  8.63s/it][2025-04-25 13:55:32,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.21 | optimizer_step: 0.93
[2025-04-25 13:55:32,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.36 | bwd_microstep: 5737.53 | bwd_inner_microstep: 5647.29 | bwd_allreduce_microstep: 90.20 | step_microstep: 19.62
[2025-04-25 13:55:32,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.36 | bwd: 5737.55 | bwd_inner: 5647.29 | bwd_allreduce: 90.22 | step: 19.62

  6%|▌         | 2472/41250 [5:57:57<93:04:23,  8.64s/it]
                                                         
{'loss': 0.0596, 'grad_norm': 0.5969729423522949, 'learning_rate': 3.990619850216713e-05, 'epoch': 0.6}

  6%|▌         | 2472/41250 [5:57:57<93:04:23,  8.64s/it][2025-04-25 13:55:40,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.17 | optimizer_step: 0.90
[2025-04-25 13:55:40,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.57 | bwd_microstep: 5717.43 | bwd_inner_microstep: 5704.58 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.79
[2025-04-25 13:55:40,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.57 | bwd: 5717.44 | bwd_inner: 5704.58 | bwd_allreduce: 12.81 | step: 18.79

  6%|▌         | 2473/41250 [5:58:06<93:06:18,  8.64s/it]
                                                         
{'loss': 0.2937, 'grad_norm': 2.2267496585845947, 'learning_rate': 3.9906046531397876e-05, 'epoch': 0.6}

  6%|▌         | 2473/41250 [5:58:06<93:06:18,  8.64s/it][2025-04-25 13:55:49,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:55:49,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5721.17 | bwd_inner_microstep: 5677.22 | bwd_allreduce_microstep: 43.89 | step_microstep: 18.69
[2025-04-25 13:55:49,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5721.18 | bwd_inner: 5677.22 | bwd_allreduce: 43.91 | step: 18.69

  6%|▌         | 2474/41250 [5:58:14<93:07:44,  8.65s/it]
                                                         
{'loss': 0.0287, 'grad_norm': 0.24678920209407806, 'learning_rate': 3.990589443791175e-05, 'epoch': 0.6}

  6%|▌         | 2474/41250 [5:58:14<93:07:44,  8.65s/it][2025-04-25 13:55:58,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-25 13:55:58,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.30 | bwd_microstep: 5713.18 | bwd_inner_microstep: 5700.08 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.07
[2025-04-25 13:55:58,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.30 | bwd: 5713.20 | bwd_inner: 5700.08 | bwd_allreduce: 13.07 | step: 19.07

  6%|▌         | 2475/41250 [5:58:23<93:06:37,  8.64s/it]
                                                         
{'loss': 0.1756, 'grad_norm': 1.4194209575653076, 'learning_rate': 3.99057422217097e-05, 'epoch': 0.6}

  6%|▌         | 2475/41250 [5:58:23<93:06:37,  8.64s/it][2025-04-25 13:56:06,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 13:56:06,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.01 | bwd_microstep: 5740.80 | bwd_inner_microstep: 5649.85 | bwd_allreduce_microstep: 90.90 | step_microstep: 18.74
[2025-04-25 13:56:06,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.01 | bwd: 5740.81 | bwd_inner: 5649.85 | bwd_allreduce: 90.92 | step: 18.74

  6%|▌         | 2476/41250 [5:58:32<93:06:51,  8.65s/it]
                                                         
{'loss': 0.2891, 'grad_norm': 1.8469451665878296, 'learning_rate': 3.9905589882792675e-05, 'epoch': 0.6}

  6%|▌         | 2476/41250 [5:58:32<93:06:51,  8.65s/it][2025-04-25 13:56:15,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 13:56:15,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.71 | bwd_microstep: 5724.20 | bwd_inner_microstep: 5711.45 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.90
[2025-04-25 13:56:15,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.71 | bwd: 5724.21 | bwd_inner: 5711.45 | bwd_allreduce: 12.73 | step: 18.90

  6%|▌         | 2477/41250 [5:58:40<93:08:50,  8.65s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.748831868171692, 'learning_rate': 3.99054374211616e-05, 'epoch': 0.6}

  6%|▌         | 2477/41250 [5:58:40<93:08:50,  8.65s/it][2025-04-25 13:56:24,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:56:24,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.39 | bwd_microstep: 5789.77 | bwd_inner_microstep: 5776.84 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.08
[2025-04-25 13:56:24,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.39 | bwd: 5789.78 | bwd_inner: 5776.84 | bwd_allreduce: 12.90 | step: 19.08

  6%|▌         | 2478/41250 [5:58:49<93:31:47,  8.68s/it]
                                                         
{'loss': 0.1953, 'grad_norm': 1.7550350427627563, 'learning_rate': 3.990528483681742e-05, 'epoch': 0.6}

  6%|▌         | 2478/41250 [5:58:49<93:31:47,  8.68s/it][2025-04-25 13:56:32,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.07 | optimizer_step: 1.03
[2025-04-25 13:56:32,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5751.89 | bwd_inner_microstep: 5654.84 | bwd_allreduce_microstep: 97.00 | step_microstep: 19.93
[2025-04-25 13:56:32,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5751.91 | bwd_inner: 5654.84 | bwd_allreduce: 97.02 | step: 19.93

  6%|▌         | 2479/41250 [5:58:58<93:28:54,  8.68s/it]
                                                         
{'loss': 0.1207, 'grad_norm': 1.2325702905654907, 'learning_rate': 3.990513212976107e-05, 'epoch': 0.6}

  6%|▌         | 2479/41250 [5:58:58<93:28:54,  8.68s/it][2025-04-25 13:56:41,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.15 | optimizer_step: 0.90
[2025-04-25 13:56:41,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.57 | bwd_microstep: 5691.48 | bwd_inner_microstep: 5678.10 | bwd_allreduce_microstep: 13.31 | step_microstep: 19.22
[2025-04-25 13:56:41,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.57 | bwd: 5691.49 | bwd_inner: 5678.11 | bwd_allreduce: 13.33 | step: 19.22

  6%|▌         | 2480/41250 [5:59:06<93:17:51,  8.66s/it]
                                                         
{'loss': 0.0994, 'grad_norm': 1.210254430770874, 'learning_rate': 3.99049792999935e-05, 'epoch': 0.6}

  6%|▌         | 2480/41250 [5:59:06<93:17:51,  8.66s/it][2025-04-25 13:56:50,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 13:56:50,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.08 | bwd_microstep: 5675.81 | bwd_inner_microstep: 5655.50 | bwd_allreduce_microstep: 20.26 | step_microstep: 19.12
[2025-04-25 13:56:50,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.08 | bwd: 5675.83 | bwd_inner: 5655.50 | bwd_allreduce: 20.28 | step: 19.12

  6%|▌         | 2481/41250 [5:59:15<93:02:32,  8.64s/it]
                                                         
{'loss': 0.1638, 'grad_norm': 1.8394567966461182, 'learning_rate': 3.9904826347515645e-05, 'epoch': 0.6}

  6%|▌         | 2481/41250 [5:59:15<93:02:32,  8.64s/it][2025-04-25 13:56:58,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 13:56:58,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5727.06 | bwd_inner_microstep: 5693.11 | bwd_allreduce_microstep: 33.91 | step_microstep: 18.78
[2025-04-25 13:56:58,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.74 | bwd: 5727.08 | bwd_inner: 5693.11 | bwd_allreduce: 33.92 | step: 18.78

  6%|▌         | 2482/41250 [5:59:24<93:06:17,  8.65s/it]
                                                         
{'loss': 0.3941, 'grad_norm': 2.225271224975586, 'learning_rate': 3.990467327232845e-05, 'epoch': 0.6}

  6%|▌         | 2482/41250 [5:59:24<93:06:17,  8.65s/it][2025-04-25 13:57:07,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 13:57:07,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.67 | bwd_microstep: 5708.48 | bwd_inner_microstep: 5690.50 | bwd_allreduce_microstep: 17.93 | step_microstep: 19.10
[2025-04-25 13:57:07,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.67 | bwd: 5708.49 | bwd_inner: 5690.50 | bwd_allreduce: 17.95 | step: 19.09

  6%|▌         | 2483/41250 [5:59:32<93:05:02,  8.64s/it]
                                                         
{'loss': 0.1122, 'grad_norm': 2.398887872695923, 'learning_rate': 3.9904520074432856e-05, 'epoch': 0.6}

  6%|▌         | 2483/41250 [5:59:32<93:05:02,  8.64s/it][2025-04-25 13:57:16,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 13:57:16,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.69 | bwd_microstep: 5707.45 | bwd_inner_microstep: 5680.30 | bwd_allreduce_microstep: 27.11 | step_microstep: 18.72
[2025-04-25 13:57:16,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.69 | bwd: 5707.46 | bwd_inner: 5680.30 | bwd_allreduce: 27.12 | step: 18.73

  6%|▌         | 2484/41250 [5:59:41<93:02:57,  8.64s/it]
                                                         
{'loss': 0.1602, 'grad_norm': 1.5354640483856201, 'learning_rate': 3.9904366753829816e-05, 'epoch': 0.6}

  6%|▌         | 2484/41250 [5:59:41<93:02:57,  8.64s/it][2025-04-25 13:57:24,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 13:57:24,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5712.23 | bwd_inner_microstep: 5699.67 | bwd_allreduce_microstep: 12.52 | step_microstep: 19.03
[2025-04-25 13:57:24,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5712.25 | bwd_inner: 5699.67 | bwd_allreduce: 12.54 | step: 19.04

  6%|▌         | 2485/41250 [5:59:50<93:03:16,  8.64s/it]
                                                         
{'loss': 0.058, 'grad_norm': 0.6832596659660339, 'learning_rate': 3.990421331052027e-05, 'epoch': 0.6}

  6%|▌         | 2485/41250 [5:59:50<93:03:16,  8.64s/it][2025-04-25 13:57:33,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 13:57:33,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.25 | bwd_microstep: 5788.26 | bwd_inner_microstep: 5775.40 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.92
[2025-04-25 13:57:33,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.25 | bwd: 5788.27 | bwd_inner: 5775.40 | bwd_allreduce: 12.82 | step: 18.92

  6%|▌         | 2486/41250 [5:59:58<93:26:06,  8.68s/it]
                                                         
{'loss': 0.0893, 'grad_norm': 1.2072445154190063, 'learning_rate': 3.990405974450516e-05, 'epoch': 0.6}

  6%|▌         | 2486/41250 [5:59:58<93:26:06,  8.68s/it][2025-04-25 13:57:42,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.27 | optimizer_step: 1.02
[2025-04-25 13:57:42,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.14 | bwd_microstep: 5744.38 | bwd_inner_microstep: 5649.70 | bwd_allreduce_microstep: 94.62 | step_microstep: 19.65
[2025-04-25 13:57:42,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.14 | bwd: 5744.40 | bwd_inner: 5649.70 | bwd_allreduce: 94.65 | step: 19.66

  6%|▌         | 2487/41250 [6:00:07<93:21:08,  8.67s/it]
                                                         
{'loss': 0.1648, 'grad_norm': 1.2884351015090942, 'learning_rate': 3.9903906055785447e-05, 'epoch': 0.6}

  6%|▌         | 2487/41250 [6:00:07<93:21:08,  8.67s/it][2025-04-25 13:57:50,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 13:57:50,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.39 | bwd_microstep: 5700.54 | bwd_inner_microstep: 5687.75 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.18
[2025-04-25 13:57:50,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.39 | bwd: 5700.55 | bwd_inner: 5687.75 | bwd_allreduce: 12.76 | step: 19.18

  6%|▌         | 2488/41250 [6:00:16<93:13:07,  8.66s/it]
                                                         
{'loss': 0.1578, 'grad_norm': 1.7677371501922607, 'learning_rate': 3.990375224436206e-05, 'epoch': 0.6}

  6%|▌         | 2488/41250 [6:00:16<93:13:07,  8.66s/it][2025-04-25 13:57:59,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 13:57:59,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.10 | bwd_microstep: 5731.00 | bwd_inner_microstep: 5684.64 | bwd_allreduce_microstep: 46.32 | step_microstep: 18.74
[2025-04-25 13:57:59,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.10 | bwd: 5731.01 | bwd_inner: 5684.64 | bwd_allreduce: 46.34 | step: 18.74

  6%|▌         | 2489/41250 [6:00:24<93:14:26,  8.66s/it]
                                                         
{'loss': 0.1763, 'grad_norm': 1.7236415147781372, 'learning_rate': 3.990359831023595e-05, 'epoch': 0.6}

  6%|▌         | 2489/41250 [6:00:24<93:14:26,  8.66s/it][2025-04-25 13:58:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 13:58:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.74 | bwd_microstep: 5712.56 | bwd_inner_microstep: 5691.82 | bwd_allreduce_microstep: 20.69 | step_microstep: 18.70
[2025-04-25 13:58:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.74 | bwd: 5712.57 | bwd_inner: 5691.82 | bwd_allreduce: 20.71 | step: 18.70

  6%|▌         | 2490/41250 [6:00:33<93:10:43,  8.65s/it]
                                                         
{'loss': 0.2413, 'grad_norm': 0.9015266299247742, 'learning_rate': 3.9903444253408075e-05, 'epoch': 0.6}

  6%|▌         | 2490/41250 [6:00:33<93:10:43,  8.65s/it][2025-04-25 13:58:16,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 13:58:16,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.76 | bwd_microstep: 5766.89 | bwd_inner_microstep: 5651.07 | bwd_allreduce_microstep: 115.77 | step_microstep: 18.80
[2025-04-25 13:58:16,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.76 | bwd: 5766.90 | bwd_inner: 5651.07 | bwd_allreduce: 115.79 | step: 18.81

  6%|▌         | 2491/41250 [6:00:42<93:13:58,  8.66s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.9726805686950684, 'learning_rate': 3.9903290073879387e-05, 'epoch': 0.6}

  6%|▌         | 2491/41250 [6:00:42<93:13:58,  8.66s/it][2025-04-25 13:58:25,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.11
[2025-04-25 13:58:25,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.83 | bwd_microstep: 5711.21 | bwd_inner_microstep: 5682.28 | bwd_allreduce_microstep: 28.88 | step_microstep: 19.39
[2025-04-25 13:58:25,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.83 | bwd: 5711.22 | bwd_inner: 5682.28 | bwd_allreduce: 28.90 | step: 19.39

  6%|▌         | 2492/41250 [6:00:50<93:12:20,  8.66s/it]
                                                         
{'loss': 0.058, 'grad_norm': 0.7578909993171692, 'learning_rate': 3.9903135771650827e-05, 'epoch': 0.6}

  6%|▌         | 2492/41250 [6:00:50<93:12:20,  8.66s/it][2025-04-25 13:58:34,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 13:58:34,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.18 | bwd_microstep: 5715.52 | bwd_inner_microstep: 5675.73 | bwd_allreduce_microstep: 39.73 | step_microstep: 19.02
[2025-04-25 13:58:34,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.18 | bwd: 5715.53 | bwd_inner: 5675.73 | bwd_allreduce: 39.75 | step: 19.02

  6%|▌         | 2493/41250 [6:00:59<93:11:00,  8.66s/it]
                                                         
{'loss': 0.3419, 'grad_norm': 1.647923469543457, 'learning_rate': 3.990298134672335e-05, 'epoch': 0.6}

  6%|▌         | 2493/41250 [6:00:59<93:11:00,  8.66s/it][2025-04-25 13:58:42,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:58:42,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.34 | bwd_microstep: 5722.59 | bwd_inner_microstep: 5681.07 | bwd_allreduce_microstep: 41.49 | step_microstep: 18.65
[2025-04-25 13:58:42,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.34 | bwd: 5722.61 | bwd_inner: 5681.07 | bwd_allreduce: 41.50 | step: 18.65

  6%|▌         | 2494/41250 [6:01:08<93:09:23,  8.65s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.1686993837356567, 'learning_rate': 3.990282679909791e-05, 'epoch': 0.6}

  6%|▌         | 2494/41250 [6:01:08<93:09:23,  8.65s/it][2025-04-25 13:58:51,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 13:58:51,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 5722.78 | bwd_inner_microstep: 5694.92 | bwd_allreduce_microstep: 27.82 | step_microstep: 18.63
[2025-04-25 13:58:51,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 5722.80 | bwd_inner: 5694.92 | bwd_allreduce: 27.84 | step: 18.63

  6%|▌         | 2495/41250 [6:01:16<93:09:21,  8.65s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 1.481630802154541, 'learning_rate': 3.9902672128775446e-05, 'epoch': 0.6}

  6%|▌         | 2495/41250 [6:01:16<93:09:21,  8.65s/it][2025-04-25 13:58:59,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 13:58:59,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.92 | bwd_microstep: 5670.38 | bwd_inner_microstep: 5651.25 | bwd_allreduce_microstep: 19.09 | step_microstep: 18.49
[2025-04-25 13:58:59,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.92 | bwd: 5670.40 | bwd_inner: 5651.25 | bwd_allreduce: 19.11 | step: 18.49

  6%|▌         | 2496/41250 [6:01:25<92:56:54,  8.63s/it]
                                                         
{'loss': 0.2202, 'grad_norm': 1.4227616786956787, 'learning_rate': 3.9902517335756934e-05, 'epoch': 0.61}

  6%|▌         | 2496/41250 [6:01:25<92:56:54,  8.63s/it][2025-04-25 13:59:08,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 13:59:08,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.02 | bwd_microstep: 5719.44 | bwd_inner_microstep: 5706.55 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.94
[2025-04-25 13:59:08,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.02 | bwd: 5719.45 | bwd_inner: 5706.55 | bwd_allreduce: 12.86 | step: 18.94

  6%|▌         | 2497/41250 [6:01:33<93:01:12,  8.64s/it]
                                                         
{'loss': 0.3112, 'grad_norm': 1.657508134841919, 'learning_rate': 3.9902362420043315e-05, 'epoch': 0.61}

  6%|▌         | 2497/41250 [6:01:33<93:01:12,  8.64s/it][2025-04-25 13:59:17,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 13:59:17,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.63 | bwd_microstep: 5681.09 | bwd_inner_microstep: 5643.41 | bwd_allreduce_microstep: 37.63 | step_microstep: 19.01
[2025-04-25 13:59:17,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.63 | bwd: 5681.10 | bwd_inner: 5643.41 | bwd_allreduce: 37.65 | step: 19.01

  6%|▌         | 2498/41250 [6:01:42<92:49:34,  8.62s/it]
                                                         
{'loss': 0.0788, 'grad_norm': 0.7144931554794312, 'learning_rate': 3.9902207381635544e-05, 'epoch': 0.61}

  6%|▌         | 2498/41250 [6:01:42<92:49:34,  8.62s/it][2025-04-25 13:59:25,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:59:25,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.51 | bwd_microstep: 5745.53 | bwd_inner_microstep: 5657.78 | bwd_allreduce_microstep: 87.71 | step_microstep: 18.43
[2025-04-25 13:59:25,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.51 | bwd: 5745.54 | bwd_inner: 5657.78 | bwd_allreduce: 87.73 | step: 18.44

  6%|▌         | 2499/41250 [6:01:51<92:57:58,  8.64s/it]
                                                         
{'loss': 0.2763, 'grad_norm': 1.689048171043396, 'learning_rate': 3.990205222053458e-05, 'epoch': 0.61}

  6%|▌         | 2499/41250 [6:01:51<92:57:58,  8.64s/it][2025-04-25 13:59:34,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 13:59:34,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.36 | bwd_microstep: 5726.15 | bwd_inner_microstep: 5713.31 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.83
[2025-04-25 13:59:34,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.36 | bwd: 5726.16 | bwd_inner: 5713.31 | bwd_allreduce: 12.81 | step: 18.83

  6%|▌         | 2500/41250 [6:01:59<93:04:59,  8.65s/it]
                                                         
{'loss': 0.0539, 'grad_norm': 0.5713549852371216, 'learning_rate': 3.990189693674138e-05, 'epoch': 0.61}

  6%|▌         | 2500/41250 [6:01:59<93:04:59,  8.65s/it][2025-04-25 13:59:43,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 13:59:43,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.58 | bwd_microstep: 5847.59 | bwd_inner_microstep: 5700.14 | bwd_allreduce_microstep: 147.40 | step_microstep: 18.90
[2025-04-25 13:59:43,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.58 | bwd: 5847.60 | bwd_inner: 5700.14 | bwd_allreduce: 147.42 | step: 18.90

  6%|▌         | 2501/41250 [6:02:08<93:32:25,  8.69s/it]
                                                         
{'loss': 0.2406, 'grad_norm': 1.8692998886108398, 'learning_rate': 3.99017415302569e-05, 'epoch': 0.61}

  6%|▌         | 2501/41250 [6:02:08<93:32:25,  8.69s/it][2025-04-25 13:59:51,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 13:59:51,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.26 | bwd_microstep: 5715.06 | bwd_inner_microstep: 5702.14 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.95
[2025-04-25 13:59:51,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.26 | bwd: 5715.07 | bwd_inner: 5702.14 | bwd_allreduce: 12.89 | step: 18.95

  6%|▌         | 2502/41250 [6:02:17<93:26:09,  8.68s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.1717636585235596, 'learning_rate': 3.99015860010821e-05, 'epoch': 0.61}

  6%|▌         | 2502/41250 [6:02:17<93:26:09,  8.68s/it][2025-04-25 14:00:00,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 14:00:00,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.14 | bwd_microstep: 5724.95 | bwd_inner_microstep: 5712.02 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.46
[2025-04-25 14:00:00,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.14 | bwd: 5724.97 | bwd_inner: 5712.02 | bwd_allreduce: 12.90 | step: 19.47

  6%|▌         | 2503/41250 [6:02:25<93:23:28,  8.68s/it]
                                                         
{'loss': 0.2732, 'grad_norm': 2.3983304500579834, 'learning_rate': 3.9901430349217934e-05, 'epoch': 0.61}

  6%|▌         | 2503/41250 [6:02:25<93:23:28,  8.68s/it][2025-04-25 14:00:09,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:00:09,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5736.07 | bwd_inner_microstep: 5704.42 | bwd_allreduce_microstep: 31.61 | step_microstep: 18.74
[2025-04-25 14:00:09,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.03 | bwd: 5736.08 | bwd_inner: 5704.42 | bwd_allreduce: 31.63 | step: 18.74

  6%|▌         | 2504/41250 [6:02:34<93:22:27,  8.68s/it]
                                                         
{'loss': 0.3832, 'grad_norm': 3.102602958679199, 'learning_rate': 3.990127457466536e-05, 'epoch': 0.61}

  6%|▌         | 2504/41250 [6:02:34<93:22:27,  8.68s/it][2025-04-25 14:00:17,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:00:17,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5739.35 | bwd_inner_microstep: 5692.71 | bwd_allreduce_microstep: 46.60 | step_microstep: 18.72
[2025-04-25 14:00:17,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5739.37 | bwd_inner: 5692.71 | bwd_allreduce: 46.61 | step: 18.72

  6%|▌         | 2505/41250 [6:02:43<93:22:15,  8.68s/it]
                                                         
{'loss': 0.2564, 'grad_norm': 2.0462021827697754, 'learning_rate': 3.9901118677425346e-05, 'epoch': 0.61}

  6%|▌         | 2505/41250 [6:02:43<93:22:15,  8.68s/it][2025-04-25 14:00:26,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 14:00:26,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.47 | bwd_microstep: 5743.86 | bwd_inner_microstep: 5660.61 | bwd_allreduce_microstep: 83.20 | step_microstep: 18.62
[2025-04-25 14:00:26,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.48 | bwd: 5743.87 | bwd_inner: 5660.61 | bwd_allreduce: 83.22 | step: 18.62

  6%|▌         | 2506/41250 [6:02:51<93:19:59,  8.67s/it]
                                                         
{'loss': 0.1341, 'grad_norm': 1.7751272916793823, 'learning_rate': 3.990096265749885e-05, 'epoch': 0.61}

  6%|▌         | 2506/41250 [6:02:51<93:19:59,  8.67s/it][2025-04-25 14:00:35,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 14:00:35,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5732.12 | bwd_inner_microstep: 5718.37 | bwd_allreduce_microstep: 13.70 | step_microstep: 18.99
[2025-04-25 14:00:35,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5732.13 | bwd_inner: 5718.37 | bwd_allreduce: 13.72 | step: 19.00

  6%|▌         | 2507/41250 [6:03:00<93:18:19,  8.67s/it]
                                                         
{'loss': 0.0658, 'grad_norm': 0.9125375151634216, 'learning_rate': 3.990080651488684e-05, 'epoch': 0.61}

  6%|▌         | 2507/41250 [6:03:00<93:18:19,  8.67s/it][2025-04-25 14:00:44,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 14:00:44,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.65 | bwd_microstep: 5856.44 | bwd_inner_microstep: 5648.50 | bwd_allreduce_microstep: 207.89 | step_microstep: 18.85
[2025-04-25 14:00:44,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.65 | bwd: 5856.45 | bwd_inner: 5648.50 | bwd_allreduce: 207.91 | step: 18.86

  6%|▌         | 2508/41250 [6:03:09<93:40:01,  8.70s/it]
                                                         
{'loss': 0.1778, 'grad_norm': 1.2770007848739624, 'learning_rate': 3.990065024959026e-05, 'epoch': 0.61}

  6%|▌         | 2508/41250 [6:03:09<93:40:01,  8.70s/it][2025-04-25 14:00:52,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 14:00:52,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.15 | bwd_microstep: 5774.79 | bwd_inner_microstep: 5659.94 | bwd_allreduce_microstep: 114.80 | step_microstep: 18.68
[2025-04-25 14:00:52,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.15 | bwd: 5774.81 | bwd_inner: 5659.94 | bwd_allreduce: 114.82 | step: 18.68

  6%|▌         | 2509/41250 [6:03:18<93:36:16,  8.70s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 0.8700135350227356, 'learning_rate': 3.990049386161009e-05, 'epoch': 0.61}

  6%|▌         | 2509/41250 [6:03:18<93:36:16,  8.70s/it][2025-04-25 14:01:01,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.12
[2025-04-25 14:01:01,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.71 | bwd_microstep: 5705.75 | bwd_inner_microstep: 5666.27 | bwd_allreduce_microstep: 39.43 | step_microstep: 19.44
[2025-04-25 14:01:01,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.71 | bwd: 5705.76 | bwd_inner: 5666.27 | bwd_allreduce: 39.45 | step: 19.44

  6%|▌         | 2510/41250 [6:03:26<93:22:21,  8.68s/it]
                                                         
{'loss': 0.1686, 'grad_norm': 2.4117813110351562, 'learning_rate': 3.9900337350947295e-05, 'epoch': 0.61}

  6%|▌         | 2510/41250 [6:03:26<93:22:21,  8.68s/it][2025-04-25 14:01:10,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 14:01:10,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.08 | bwd_microstep: 5745.37 | bwd_inner_microstep: 5690.33 | bwd_allreduce_microstep: 54.99 | step_microstep: 18.85
[2025-04-25 14:01:10,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.08 | bwd: 5745.38 | bwd_inner: 5690.33 | bwd_allreduce: 55.01 | step: 18.86

  6%|▌         | 2511/41250 [6:03:35<93:22:59,  8.68s/it]
                                                         
{'loss': 0.1476, 'grad_norm': 2.6098222732543945, 'learning_rate': 3.990018071760283e-05, 'epoch': 0.61}

  6%|▌         | 2511/41250 [6:03:35<93:22:59,  8.68s/it][2025-04-25 14:01:18,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:01:18,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.36 | bwd_microstep: 5730.08 | bwd_inner_microstep: 5717.19 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.66
[2025-04-25 14:01:18,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.36 | bwd: 5730.09 | bwd_inner: 5717.19 | bwd_allreduce: 12.85 | step: 18.66

  6%|▌         | 2512/41250 [6:03:44<93:23:51,  8.68s/it]
                                                         
{'loss': 0.3403, 'grad_norm': 2.2646501064300537, 'learning_rate': 3.9900023961577664e-05, 'epoch': 0.61}

  6%|▌         | 2512/41250 [6:03:44<93:23:51,  8.68s/it][2025-04-25 14:01:27,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:01:27,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.76 | bwd_microstep: 5758.41 | bwd_inner_microstep: 5716.31 | bwd_allreduce_microstep: 42.06 | step_microstep: 18.57
[2025-04-25 14:01:27,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.76 | bwd: 5758.42 | bwd_inner: 5716.31 | bwd_allreduce: 42.07 | step: 18.57

  6%|▌         | 2513/41250 [6:03:52<93:28:01,  8.69s/it]
                                                         
{'loss': 0.0841, 'grad_norm': 0.5450565218925476, 'learning_rate': 3.9899867082872765e-05, 'epoch': 0.61}

  6%|▌         | 2513/41250 [6:03:52<93:28:01,  8.69s/it][2025-04-25 14:01:36,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:01:36,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.51 | bwd_microstep: 5755.85 | bwd_inner_microstep: 5710.59 | bwd_allreduce_microstep: 45.22 | step_microstep: 18.77
[2025-04-25 14:01:36,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.51 | bwd: 5755.87 | bwd_inner: 5710.59 | bwd_allreduce: 45.23 | step: 18.77

  6%|▌         | 2514/41250 [6:04:01<93:31:20,  8.69s/it]
                                                         
{'loss': 0.2165, 'grad_norm': 3.6095869541168213, 'learning_rate': 3.98997100814891e-05, 'epoch': 0.61}

  6%|▌         | 2514/41250 [6:04:01<93:31:20,  8.69s/it][2025-04-25 14:01:44,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 14:01:44,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.20 | bwd_microstep: 5807.67 | bwd_inner_microstep: 5656.50 | bwd_allreduce_microstep: 151.12 | step_microstep: 19.01
[2025-04-25 14:01:44,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.20 | bwd: 5807.68 | bwd_inner: 5656.50 | bwd_allreduce: 151.14 | step: 19.01

  6%|▌         | 2515/41250 [6:04:10<93:37:57,  8.70s/it]
                                                         
{'loss': 0.0246, 'grad_norm': 0.19845730066299438, 'learning_rate': 3.989955295742764e-05, 'epoch': 0.61}

  6%|▌         | 2515/41250 [6:04:10<93:37:57,  8.70s/it][2025-04-25 14:01:53,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 14:01:53,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.55 | bwd_microstep: 5786.41 | bwd_inner_microstep: 5773.82 | bwd_allreduce_microstep: 12.54 | step_microstep: 19.10
[2025-04-25 14:01:53,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.55 | bwd: 5786.43 | bwd_inner: 5773.82 | bwd_allreduce: 12.56 | step: 19.11

  6%|▌         | 2516/41250 [6:04:18<93:49:37,  8.72s/it]
                                                         
{'loss': 0.1221, 'grad_norm': 1.1408536434173584, 'learning_rate': 3.9899395710689346e-05, 'epoch': 0.61}

  6%|▌         | 2516/41250 [6:04:18<93:49:37,  8.72s/it][2025-04-25 14:02:02,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:02:02,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2872.23 | bwd_microstep: 5735.07 | bwd_inner_microstep: 5722.08 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.17
[2025-04-25 14:02:02,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2872.23 | bwd: 5735.09 | bwd_inner: 5722.08 | bwd_allreduce: 12.97 | step: 18.17

  6%|▌         | 2517/41250 [6:04:27<93:43:49,  8.71s/it]
                                                         
{'loss': 0.3117, 'grad_norm': 2.749375343322754, 'learning_rate': 3.9899238341275195e-05, 'epoch': 0.61}

  6%|▌         | 2517/41250 [6:04:27<93:43:49,  8.71s/it][2025-04-25 14:02:11,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 14:02:11,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.90 | bwd_microstep: 5787.00 | bwd_inner_microstep: 5668.36 | bwd_allreduce_microstep: 118.60 | step_microstep: 18.51
[2025-04-25 14:02:11,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.90 | bwd: 5787.02 | bwd_inner: 5668.36 | bwd_allreduce: 118.61 | step: 18.51

  6%|▌         | 2518/41250 [6:04:36<93:41:20,  8.71s/it]
                                                         
{'loss': 0.171, 'grad_norm': 2.715599775314331, 'learning_rate': 3.989908084918614e-05, 'epoch': 0.61}

  6%|▌         | 2518/41250 [6:04:36<93:41:20,  8.71s/it][2025-04-25 14:02:19,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 14:02:19,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.42 | bwd_microstep: 5717.65 | bwd_inner_microstep: 5704.25 | bwd_allreduce_microstep: 13.35 | step_microstep: 18.62
[2025-04-25 14:02:19,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.42 | bwd: 5717.66 | bwd_inner: 5704.25 | bwd_allreduce: 13.37 | step: 18.62

  6%|▌         | 2519/41250 [6:04:45<93:32:14,  8.69s/it]
                                                         
{'loss': 0.1126, 'grad_norm': 1.8377857208251953, 'learning_rate': 3.9898923234423186e-05, 'epoch': 0.61}

  6%|▌         | 2519/41250 [6:04:45<93:32:14,  8.69s/it][2025-04-25 14:02:28,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 14:02:28,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.45 | bwd_microstep: 5708.25 | bwd_inner_microstep: 5694.94 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.39
[2025-04-25 14:02:28,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.45 | bwd: 5708.26 | bwd_inner: 5694.94 | bwd_allreduce: 13.27 | step: 19.39

  6%|▌         | 2520/41250 [6:04:53<93:23:05,  8.68s/it]
                                                         
{'loss': 0.2192, 'grad_norm': 2.2602341175079346, 'learning_rate': 3.9898765496987274e-05, 'epoch': 0.61}

  6%|▌         | 2520/41250 [6:04:53<93:23:05,  8.68s/it][2025-04-25 14:02:37,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 14:02:37,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.86 | bwd_microstep: 5747.84 | bwd_inner_microstep: 5710.41 | bwd_allreduce_microstep: 37.38 | step_microstep: 19.14
[2025-04-25 14:02:37,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.86 | bwd: 5747.85 | bwd_inner: 5710.41 | bwd_allreduce: 37.40 | step: 19.14

  6%|▌         | 2521/41250 [6:05:02<93:25:17,  8.68s/it]
                                                         
{'loss': 0.1997, 'grad_norm': 1.5869404077529907, 'learning_rate': 3.989860763687938e-05, 'epoch': 0.61}

  6%|▌         | 2521/41250 [6:05:02<93:25:17,  8.68s/it][2025-04-25 14:02:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:02:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5703.26 | bwd_inner_microstep: 5659.28 | bwd_allreduce_microstep: 43.93 | step_microstep: 18.85
[2025-04-25 14:02:45,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5703.27 | bwd_inner: 5659.28 | bwd_allreduce: 43.95 | step: 18.85

  6%|▌         | 2522/41250 [6:05:11<93:12:36,  8.66s/it]
                                                         
{'loss': 0.0763, 'grad_norm': 1.054758906364441, 'learning_rate': 3.98984496541005e-05, 'epoch': 0.61}

  6%|▌         | 2522/41250 [6:05:11<93:12:36,  8.66s/it][2025-04-25 14:02:54,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:02:54,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5715.81 | bwd_inner_microstep: 5664.70 | bwd_allreduce_microstep: 51.07 | step_microstep: 18.67
[2025-04-25 14:02:54,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5715.83 | bwd_inner: 5664.70 | bwd_allreduce: 51.09 | step: 18.68

  6%|▌         | 2523/41250 [6:05:19<93:06:03,  8.65s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 1.3123716115951538, 'learning_rate': 3.989829154865158e-05, 'epoch': 0.61}

  6%|▌         | 2523/41250 [6:05:19<93:06:03,  8.65s/it][2025-04-25 14:03:02,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:03:02,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.85 | bwd_microstep: 5718.48 | bwd_inner_microstep: 5705.87 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.60
[2025-04-25 14:03:02,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.86 | bwd: 5718.50 | bwd_inner: 5705.87 | bwd_allreduce: 12.59 | step: 18.60

  6%|▌         | 2524/41250 [6:05:28<93:07:13,  8.66s/it]
                                                         
{'loss': 0.0484, 'grad_norm': 0.5785480737686157, 'learning_rate': 3.989813332053361e-05, 'epoch': 0.61}

  6%|▌         | 2524/41250 [6:05:28<93:07:13,  8.66s/it][2025-04-25 14:03:11,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 14:03:11,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5744.96 | bwd_inner_microstep: 5651.78 | bwd_allreduce_microstep: 93.13 | step_microstep: 19.02
[2025-04-25 14:03:11,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5744.97 | bwd_inner: 5651.78 | bwd_allreduce: 93.15 | step: 19.03

  6%|▌         | 2525/41250 [6:05:36<93:08:41,  8.66s/it]
                                                         
{'loss': 0.1002, 'grad_norm': 1.3307583332061768, 'learning_rate': 3.989797496974756e-05, 'epoch': 0.61}

  6%|▌         | 2525/41250 [6:05:36<93:08:41,  8.66s/it][2025-04-25 14:03:20,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 14:03:20,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.26 | bwd_microstep: 5870.36 | bwd_inner_microstep: 5660.32 | bwd_allreduce_microstep: 209.99 | step_microstep: 19.16
[2025-04-25 14:03:20,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.26 | bwd: 5870.37 | bwd_inner: 5660.32 | bwd_allreduce: 210.01 | step: 19.16

  6%|▌         | 2526/41250 [6:05:45<93:32:24,  8.70s/it]
                                                         
{'loss': 0.0891, 'grad_norm': 1.0196032524108887, 'learning_rate': 3.989781649629441e-05, 'epoch': 0.61}

  6%|▌         | 2526/41250 [6:05:45<93:32:24,  8.70s/it][2025-04-25 14:03:29,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:03:29,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.41 | bwd_microstep: 5775.36 | bwd_inner_microstep: 5651.57 | bwd_allreduce_microstep: 123.75 | step_microstep: 18.70
[2025-04-25 14:03:29,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.41 | bwd: 5775.37 | bwd_inner: 5651.57 | bwd_allreduce: 123.76 | step: 18.70

  6%|▌         | 2527/41250 [6:05:54<93:31:01,  8.69s/it]
                                                         
{'loss': 0.0984, 'grad_norm': 1.422196626663208, 'learning_rate': 3.9897657900175135e-05, 'epoch': 0.61}

  6%|▌         | 2527/41250 [6:05:54<93:31:01,  8.69s/it][2025-04-25 14:03:37,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 14:03:37,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5690.62 | bwd_inner_microstep: 5677.54 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.94
[2025-04-25 14:03:37,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5690.64 | bwd_inner: 5677.54 | bwd_allreduce: 13.05 | step: 18.94

  6%|▌         | 2528/41250 [6:06:03<93:18:34,  8.68s/it]
                                                         
{'loss': 0.1434, 'grad_norm': 1.4498131275177002, 'learning_rate': 3.989749918139071e-05, 'epoch': 0.61}

  6%|▌         | 2528/41250 [6:06:03<93:18:34,  8.68s/it][2025-04-25 14:03:46,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-25 14:03:46,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.95 | bwd_microstep: 5743.84 | bwd_inner_microstep: 5685.15 | bwd_allreduce_microstep: 58.64 | step_microstep: 19.01
[2025-04-25 14:03:46,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.95 | bwd: 5743.85 | bwd_inner: 5685.15 | bwd_allreduce: 58.66 | step: 19.01

  6%|▌         | 2529/41250 [6:06:11<93:18:39,  8.68s/it]
                                                         
{'loss': 0.16, 'grad_norm': 1.8154709339141846, 'learning_rate': 3.9897340339942116e-05, 'epoch': 0.61}

  6%|▌         | 2529/41250 [6:06:11<93:18:39,  8.68s/it][2025-04-25 14:03:55,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-25 14:03:55,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.69 | bwd_microstep: 5693.40 | bwd_inner_microstep: 5680.63 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.29
[2025-04-25 14:03:55,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.69 | bwd: 5693.42 | bwd_inner: 5680.63 | bwd_allreduce: 12.74 | step: 19.29

  6%|▌         | 2530/41250 [6:06:20<93:08:54,  8.66s/it]
                                                         
{'loss': 0.2269, 'grad_norm': 1.9374902248382568, 'learning_rate': 3.989718137583033e-05, 'epoch': 0.61}

  6%|▌         | 2530/41250 [6:06:20<93:08:54,  8.66s/it][2025-04-25 14:04:03,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.23 | optimizer_step: 0.91
[2025-04-25 14:04:03,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.31 | bwd_microstep: 5763.62 | bwd_inner_microstep: 5655.98 | bwd_allreduce_microstep: 107.59 | step_microstep: 19.10
[2025-04-25 14:04:03,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.31 | bwd: 5763.64 | bwd_inner: 5655.98 | bwd_allreduce: 107.62 | step: 19.10

  6%|▌         | 2531/41250 [6:06:29<93:11:32,  8.66s/it]
                                                         
{'loss': 0.233, 'grad_norm': 1.2506901025772095, 'learning_rate': 3.9897022289056344e-05, 'epoch': 0.61}

  6%|▌         | 2531/41250 [6:06:29<93:11:32,  8.66s/it][2025-04-25 14:04:12,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:04:12,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.48 | bwd_microstep: 6016.03 | bwd_inner_microstep: 5647.73 | bwd_allreduce_microstep: 368.25 | step_microstep: 18.88
[2025-04-25 14:04:12,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.48 | bwd: 6016.04 | bwd_inner: 5647.73 | bwd_allreduce: 368.27 | step: 18.88

  6%|▌         | 2532/41250 [6:06:37<94:01:52,  8.74s/it]
                                                         
{'loss': 0.1914, 'grad_norm': 2.6855833530426025, 'learning_rate': 3.989686307962112e-05, 'epoch': 0.61}

  6%|▌         | 2532/41250 [6:06:37<94:01:52,  8.74s/it][2025-04-25 14:04:21,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 14:04:21,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.37 | bwd_microstep: 5745.99 | bwd_inner_microstep: 5693.83 | bwd_allreduce_microstep: 52.10 | step_microstep: 18.64
[2025-04-25 14:04:21,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.37 | bwd: 5746.00 | bwd_inner: 5693.83 | bwd_allreduce: 52.12 | step: 18.64

  6%|▌         | 2533/41250 [6:06:46<93:47:52,  8.72s/it]
                                                         
{'loss': 0.2447, 'grad_norm': 2.177919387817383, 'learning_rate': 3.989670374752565e-05, 'epoch': 0.61}

  6%|▌         | 2533/41250 [6:06:46<93:47:52,  8.72s/it][2025-04-25 14:04:29,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 14:04:29,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.75 | bwd_microstep: 5724.51 | bwd_inner_microstep: 5704.73 | bwd_allreduce_microstep: 19.73 | step_microstep: 19.51
[2025-04-25 14:04:29,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.75 | bwd: 5724.52 | bwd_inner: 5704.73 | bwd_allreduce: 19.75 | step: 19.51

  6%|▌         | 2534/41250 [6:06:55<93:37:43,  8.71s/it]
                                                         
{'loss': 0.141, 'grad_norm': 1.284930944442749, 'learning_rate': 3.989654429277092e-05, 'epoch': 0.61}

  6%|▌         | 2534/41250 [6:06:55<93:37:43,  8.71s/it][2025-04-25 14:04:38,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 14:04:38,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5672.86 | bwd_inner_microstep: 5649.19 | bwd_allreduce_microstep: 23.62 | step_microstep: 18.96
[2025-04-25 14:04:38,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5672.87 | bwd_inner: 5649.19 | bwd_allreduce: 23.64 | step: 18.97

  6%|▌         | 2535/41250 [6:07:03<93:17:55,  8.68s/it]
                                                         
{'loss': 0.3172, 'grad_norm': 4.039185523986816, 'learning_rate': 3.9896384715357904e-05, 'epoch': 0.61}

  6%|▌         | 2535/41250 [6:07:03<93:17:55,  8.68s/it][2025-04-25 14:04:47,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 14:04:47,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.83 | bwd_microstep: 5685.02 | bwd_inner_microstep: 5650.11 | bwd_allreduce_microstep: 34.87 | step_microstep: 18.84
[2025-04-25 14:04:47,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.83 | bwd: 5685.03 | bwd_inner: 5650.11 | bwd_allreduce: 34.88 | step: 18.85

  6%|▌         | 2536/41250 [6:07:12<93:02:12,  8.65s/it]
                                                         
{'loss': 0.148, 'grad_norm': 1.6129014492034912, 'learning_rate': 3.989622501528759e-05, 'epoch': 0.61}

  6%|▌         | 2536/41250 [6:07:12<93:02:12,  8.65s/it][2025-04-25 14:04:55,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:04:55,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.99 | bwd_microstep: 5754.85 | bwd_inner_microstep: 5651.58 | bwd_allreduce_microstep: 103.22 | step_microstep: 18.62
[2025-04-25 14:04:55,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.99 | bwd: 5754.87 | bwd_inner: 5651.58 | bwd_allreduce: 103.24 | step: 18.63

  6%|▌         | 2537/41250 [6:07:21<93:04:27,  8.66s/it]
                                                         
{'loss': 0.6881, 'grad_norm': 4.851731777191162, 'learning_rate': 3.9896065192560964e-05, 'epoch': 0.62}

  6%|▌         | 2537/41250 [6:07:21<93:04:27,  8.66s/it][2025-04-25 14:05:04,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 14:05:04,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5862.61 | bwd_inner_microstep: 5653.23 | bwd_allreduce_microstep: 209.34 | step_microstep: 19.18
[2025-04-25 14:05:04,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5862.62 | bwd_inner: 5653.23 | bwd_allreduce: 209.36 | step: 19.18

  6%|▌         | 2538/41250 [6:07:29<93:26:33,  8.69s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 1.0049731731414795, 'learning_rate': 3.989590524717901e-05, 'epoch': 0.62}

  6%|▌         | 2538/41250 [6:07:29<93:26:33,  8.69s/it][2025-04-25 14:05:13,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.22 | optimizer_step: 0.92
[2025-04-25 14:05:13,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5758.28 | bwd_inner_microstep: 5648.09 | bwd_allreduce_microstep: 110.14 | step_microstep: 19.21
[2025-04-25 14:05:13,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5758.29 | bwd_inner: 5648.09 | bwd_allreduce: 110.15 | step: 19.22

  6%|▌         | 2539/41250 [6:07:38<93:22:46,  8.68s/it]
                                                         
{'loss': 0.2333, 'grad_norm': 1.7743412256240845, 'learning_rate': 3.989574517914271e-05, 'epoch': 0.62}

  6%|▌         | 2539/41250 [6:07:38<93:22:46,  8.68s/it][2025-04-25 14:05:21,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.13 | optimizer_step: 0.92
[2025-04-25 14:05:21,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.53 | bwd_microstep: 5752.72 | bwd_inner_microstep: 5641.74 | bwd_allreduce_microstep: 110.93 | step_microstep: 18.87
[2025-04-25 14:05:21,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.53 | bwd: 5752.73 | bwd_inner: 5641.74 | bwd_allreduce: 110.95 | step: 18.87

  6%|▌         | 2540/41250 [6:07:47<93:17:48,  8.68s/it]
                                                         
{'loss': 0.1567, 'grad_norm': 1.3738200664520264, 'learning_rate': 3.9895584988453054e-05, 'epoch': 0.62}

  6%|▌         | 2540/41250 [6:07:47<93:17:48,  8.68s/it][2025-04-25 14:05:30,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:05:30,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.41 | bwd_microstep: 5707.34 | bwd_inner_microstep: 5694.67 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.74
[2025-04-25 14:05:30,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.41 | bwd: 5707.36 | bwd_inner: 5694.67 | bwd_allreduce: 12.65 | step: 18.74

  6%|▌         | 2541/41250 [6:07:55<93:11:11,  8.67s/it]
                                                         
{'loss': 0.3219, 'grad_norm': 1.932781457901001, 'learning_rate': 3.989542467511103e-05, 'epoch': 0.62}

  6%|▌         | 2541/41250 [6:07:55<93:11:11,  8.67s/it][2025-04-25 14:05:39,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:05:39,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.08 | bwd_microstep: 5746.73 | bwd_inner_microstep: 5681.34 | bwd_allreduce_microstep: 65.35 | step_microstep: 18.62
[2025-04-25 14:05:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.08 | bwd: 5746.74 | bwd_inner: 5681.34 | bwd_allreduce: 65.36 | step: 18.62

  6%|▌         | 2542/41250 [6:08:04<93:12:45,  8.67s/it]
                                                         
{'loss': 0.2555, 'grad_norm': 2.6891086101531982, 'learning_rate': 3.989526423911764e-05, 'epoch': 0.62}

  6%|▌         | 2542/41250 [6:08:04<93:12:45,  8.67s/it][2025-04-25 14:05:47,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-25 14:05:47,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.99 | bwd_microstep: 5727.57 | bwd_inner_microstep: 5683.85 | bwd_allreduce_microstep: 43.67 | step_microstep: 19.57
[2025-04-25 14:05:47,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.99 | bwd: 5727.58 | bwd_inner: 5683.85 | bwd_allreduce: 43.69 | step: 19.57

  6%|▌         | 2543/41250 [6:08:13<93:10:14,  8.67s/it]
                                                         
{'loss': 0.1351, 'grad_norm': 1.4277206659317017, 'learning_rate': 3.9895103680473844e-05, 'epoch': 0.62}

  6%|▌         | 2543/41250 [6:08:13<93:10:14,  8.67s/it][2025-04-25 14:05:56,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:05:56,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.08 | bwd_microstep: 5692.57 | bwd_inner_microstep: 5652.24 | bwd_allreduce_microstep: 40.29 | step_microstep: 18.77
[2025-04-25 14:05:56,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.08 | bwd: 5692.59 | bwd_inner: 5652.24 | bwd_allreduce: 40.31 | step: 18.77

  6%|▌         | 2544/41250 [6:08:21<92:57:29,  8.65s/it]
                                                         
{'loss': 0.1958, 'grad_norm': 0.8469149470329285, 'learning_rate': 3.989494299918065e-05, 'epoch': 0.62}

  6%|▌         | 2544/41250 [6:08:21<92:57:29,  8.65s/it][2025-04-25 14:06:05,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:06:05,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.58 | bwd_microstep: 5722.71 | bwd_inner_microstep: 5700.50 | bwd_allreduce_microstep: 22.17 | step_microstep: 18.70
[2025-04-25 14:06:05,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.58 | bwd: 5722.72 | bwd_inner: 5700.50 | bwd_allreduce: 22.18 | step: 18.70

  6%|▌         | 2545/41250 [6:08:30<92:59:40,  8.65s/it]
                                                         
{'loss': 0.1745, 'grad_norm': 1.7927039861679077, 'learning_rate': 3.989478219523904e-05, 'epoch': 0.62}

  6%|▌         | 2545/41250 [6:08:30<92:59:40,  8.65s/it][2025-04-25 14:06:13,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 14:06:13,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.21 | bwd_microstep: 5713.58 | bwd_inner_microstep: 5678.64 | bwd_allreduce_microstep: 34.89 | step_microstep: 19.04
[2025-04-25 14:06:13,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.22 | bwd: 5713.59 | bwd_inner: 5678.64 | bwd_allreduce: 34.91 | step: 19.04

  6%|▌         | 2546/41250 [6:08:39<92:57:25,  8.65s/it]
                                                         
{'loss': 0.2431, 'grad_norm': 1.7088457345962524, 'learning_rate': 3.989462126865002e-05, 'epoch': 0.62}

  6%|▌         | 2546/41250 [6:08:39<92:57:25,  8.65s/it][2025-04-25 14:06:22,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.37 | optimizer_step: 1.08
[2025-04-25 14:06:22,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.76 | bwd_microstep: 5771.29 | bwd_inner_microstep: 5644.05 | bwd_allreduce_microstep: 127.16 | step_microstep: 20.49
[2025-04-25 14:06:22,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.76 | bwd: 5771.31 | bwd_inner: 5644.05 | bwd_allreduce: 127.20 | step: 20.49

  6%|▌         | 2547/41250 [6:08:47<93:03:46,  8.66s/it]
                                                         
{'loss': 0.2014, 'grad_norm': 1.5735877752304077, 'learning_rate': 3.989446021941457e-05, 'epoch': 0.62}

  6%|▌         | 2547/41250 [6:08:47<93:03:46,  8.66s/it][2025-04-25 14:06:31,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-25 14:06:31,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.43 | bwd_microstep: 5728.64 | bwd_inner_microstep: 5651.49 | bwd_allreduce_microstep: 77.10 | step_microstep: 19.04
[2025-04-25 14:06:31,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.43 | bwd: 5728.65 | bwd_inner: 5651.49 | bwd_allreduce: 77.12 | step: 19.04

  6%|▌         | 2548/41250 [6:08:56<93:00:32,  8.65s/it]
                                                         
{'loss': 0.1733, 'grad_norm': 1.6540511846542358, 'learning_rate': 3.989429904753368e-05, 'epoch': 0.62}

  6%|▌         | 2548/41250 [6:08:56<93:00:32,  8.65s/it][2025-04-25 14:06:39,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 14:06:39,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.23 | bwd_microstep: 5675.49 | bwd_inner_microstep: 5647.58 | bwd_allreduce_microstep: 27.85 | step_microstep: 19.78
[2025-04-25 14:06:39,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.23 | bwd: 5675.50 | bwd_inner: 5647.58 | bwd_allreduce: 27.88 | step: 19.78

  6%|▌         | 2549/41250 [6:09:05<92:47:23,  8.63s/it]
                                                         
{'loss': 0.2013, 'grad_norm': 3.7168757915496826, 'learning_rate': 3.9894137753008356e-05, 'epoch': 0.62}

  6%|▌         | 2549/41250 [6:09:05<92:47:23,  8.63s/it][2025-04-25 14:06:48,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:06:48,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.11 | bwd_microstep: 5733.38 | bwd_inner_microstep: 5644.42 | bwd_allreduce_microstep: 88.92 | step_microstep: 18.13
[2025-04-25 14:06:48,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.12 | bwd: 5733.40 | bwd_inner: 5644.42 | bwd_allreduce: 88.94 | step: 18.14

  6%|▌         | 2550/41250 [6:09:13<92:49:36,  8.64s/it]
                                                         
{'loss': 0.2033, 'grad_norm': 1.5889506340026855, 'learning_rate': 3.989397633583958e-05, 'epoch': 0.62}

  6%|▌         | 2550/41250 [6:09:13<92:49:36,  8.64s/it][2025-04-25 14:06:57,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:06:57,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.44 | bwd_microstep: 5787.68 | bwd_inner_microstep: 5774.63 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.98
[2025-04-25 14:06:57,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.44 | bwd: 5787.70 | bwd_inner: 5774.63 | bwd_allreduce: 13.03 | step: 18.98

  6%|▌         | 2551/41250 [6:09:22<93:14:24,  8.67s/it]
                                                         
{'loss': 0.3454, 'grad_norm': 2.7804219722747803, 'learning_rate': 3.9893814796028356e-05, 'epoch': 0.62}

  6%|▌         | 2551/41250 [6:09:22<93:14:24,  8.67s/it][2025-04-25 14:07:05,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:07:05,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5717.34 | bwd_inner_microstep: 5703.52 | bwd_allreduce_microstep: 13.76 | step_microstep: 18.86
[2025-04-25 14:07:05,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.70 | bwd: 5717.35 | bwd_inner: 5703.52 | bwd_allreduce: 13.78 | step: 18.86

  6%|▌         | 2552/41250 [6:09:31<93:10:14,  8.67s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 1.2763196229934692, 'learning_rate': 3.989365313357568e-05, 'epoch': 0.62}

  6%|▌         | 2552/41250 [6:09:31<93:10:14,  8.67s/it][2025-04-25 14:07:14,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-25 14:07:14,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.82 | bwd_microstep: 5691.10 | bwd_inner_microstep: 5650.06 | bwd_allreduce_microstep: 40.99 | step_microstep: 19.41
[2025-04-25 14:07:14,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.82 | bwd: 5691.11 | bwd_inner: 5650.06 | bwd_allreduce: 41.01 | step: 19.41

  6%|▌         | 2553/41250 [6:09:39<92:57:24,  8.65s/it]
                                                         
{'loss': 0.4871, 'grad_norm': 2.391679048538208, 'learning_rate': 3.989349134848254e-05, 'epoch': 0.62}

  6%|▌         | 2553/41250 [6:09:39<92:57:24,  8.65s/it][2025-04-25 14:07:22,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:07:22,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5683.77 | bwd_inner_microstep: 5655.29 | bwd_allreduce_microstep: 28.44 | step_microstep: 18.69
[2025-04-25 14:07:22,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5683.79 | bwd_inner: 5655.29 | bwd_allreduce: 28.46 | step: 18.69

  6%|▌         | 2554/41250 [6:09:48<92:46:54,  8.63s/it]
                                                         
{'loss': 0.3771, 'grad_norm': 2.05556583404541, 'learning_rate': 3.989332944074995e-05, 'epoch': 0.62}

  6%|▌         | 2554/41250 [6:09:48<92:46:54,  8.63s/it][2025-04-25 14:07:31,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 14:07:31,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.60 | bwd_microstep: 5696.00 | bwd_inner_microstep: 5683.63 | bwd_allreduce_microstep: 12.32 | step_microstep: 18.71
[2025-04-25 14:07:31,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.60 | bwd: 5696.01 | bwd_inner: 5683.63 | bwd_allreduce: 12.34 | step: 18.71

  6%|▌         | 2555/41250 [6:09:56<92:45:46,  8.63s/it]
                                                         
{'loss': 0.0746, 'grad_norm': 0.4967111647129059, 'learning_rate': 3.989316741037889e-05, 'epoch': 0.62}

  6%|▌         | 2555/41250 [6:09:56<92:45:46,  8.63s/it][2025-04-25 14:07:40,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:07:40,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.92 | bwd_microstep: 5713.71 | bwd_inner_microstep: 5700.95 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.76
[2025-04-25 14:07:40,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.92 | bwd: 5713.72 | bwd_inner: 5700.95 | bwd_allreduce: 12.73 | step: 18.76

  6%|▌         | 2556/41250 [6:10:05<92:50:21,  8.64s/it]
                                                         
{'loss': 0.2333, 'grad_norm': 1.378614902496338, 'learning_rate': 3.989300525737036e-05, 'epoch': 0.62}

  6%|▌         | 2556/41250 [6:10:05<92:50:21,  8.64s/it][2025-04-25 14:07:48,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:07:48,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.14 | bwd_microstep: 5767.31 | bwd_inner_microstep: 5647.57 | bwd_allreduce_microstep: 119.69 | step_microstep: 18.63
[2025-04-25 14:07:48,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.14 | bwd: 5767.33 | bwd_inner: 5647.57 | bwd_allreduce: 119.71 | step: 18.63

  6%|▌         | 2557/41250 [6:10:14<92:58:05,  8.65s/it]
                                                         
{'loss': 0.1171, 'grad_norm': 0.8188771605491638, 'learning_rate': 3.989284298172537e-05, 'epoch': 0.62}

  6%|▌         | 2557/41250 [6:10:14<92:58:05,  8.65s/it][2025-04-25 14:07:57,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 14:07:57,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.95 | bwd_microstep: 5751.59 | bwd_inner_microstep: 5656.24 | bwd_allreduce_microstep: 95.30 | step_microstep: 19.01
[2025-04-25 14:07:57,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.95 | bwd: 5751.60 | bwd_inner: 5656.24 | bwd_allreduce: 95.32 | step: 19.02

  6%|▌         | 2558/41250 [6:10:22<93:00:34,  8.65s/it]
                                                         
{'loss': 0.2909, 'grad_norm': 1.6461312770843506, 'learning_rate': 3.989268058344491e-05, 'epoch': 0.62}

  6%|▌         | 2558/41250 [6:10:22<93:00:34,  8.65s/it][2025-04-25 14:08:06,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 14:08:06,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.57 | bwd_microstep: 5682.97 | bwd_inner_microstep: 5655.63 | bwd_allreduce_microstep: 27.29 | step_microstep: 19.24
[2025-04-25 14:08:06,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.57 | bwd: 5682.99 | bwd_inner: 5655.63 | bwd_allreduce: 27.31 | step: 19.24

  6%|▌         | 2559/41250 [6:10:31<92:48:41,  8.64s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 1.2291975021362305, 'learning_rate': 3.989251806252999e-05, 'epoch': 0.62}

  6%|▌         | 2559/41250 [6:10:31<92:48:41,  8.64s/it][2025-04-25 14:08:14,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:08:14,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.22 | bwd_microstep: 5786.00 | bwd_inner_microstep: 5773.10 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.67
[2025-04-25 14:08:14,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.22 | bwd: 5786.01 | bwd_inner: 5773.10 | bwd_allreduce: 12.87 | step: 18.67

  6%|▌         | 2560/41250 [6:10:40<93:11:03,  8.67s/it]
                                                         
{'loss': 0.1693, 'grad_norm': 1.4232126474380493, 'learning_rate': 3.9892355418981616e-05, 'epoch': 0.62}

  6%|▌         | 2560/41250 [6:10:40<93:11:03,  8.67s/it][2025-04-25 14:08:23,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:08:23,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5757.98 | bwd_inner_microstep: 5698.59 | bwd_allreduce_microstep: 59.35 | step_microstep: 18.66
[2025-04-25 14:08:23,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.94 | bwd: 5758.00 | bwd_inner: 5698.59 | bwd_allreduce: 59.37 | step: 18.67

  6%|▌         | 2561/41250 [6:10:48<93:13:42,  8.67s/it]
                                                         
{'loss': 0.0854, 'grad_norm': 0.6359310150146484, 'learning_rate': 3.989219265280078e-05, 'epoch': 0.62}

  6%|▌         | 2561/41250 [6:10:48<93:13:42,  8.67s/it][2025-04-25 14:08:32,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:08:32,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5846.25 | bwd_inner_microstep: 5699.94 | bwd_allreduce_microstep: 146.26 | step_microstep: 18.57
[2025-04-25 14:08:32,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5846.26 | bwd_inner: 5699.94 | bwd_allreduce: 146.28 | step: 18.57

  6%|▌         | 2562/41250 [6:10:57<93:35:06,  8.71s/it]
                                                         
{'loss': 0.2307, 'grad_norm': 1.0411145687103271, 'learning_rate': 3.9892029763988485e-05, 'epoch': 0.62}

  6%|▌         | 2562/41250 [6:10:57<93:35:06,  8.71s/it][2025-04-25 14:08:41,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:08:41,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5768.34 | bwd_inner_microstep: 5652.56 | bwd_allreduce_microstep: 115.73 | step_microstep: 19.00
[2025-04-25 14:08:41,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5768.35 | bwd_inner: 5652.56 | bwd_allreduce: 115.75 | step: 19.00

  6%|▌         | 2563/41250 [6:11:06<93:29:42,  8.70s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 1.279496192932129, 'learning_rate': 3.989186675254574e-05, 'epoch': 0.62}

  6%|▌         | 2563/41250 [6:11:06<93:29:42,  8.70s/it][2025-04-25 14:08:49,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 14:08:49,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.26 | bwd_microstep: 5717.76 | bwd_inner_microstep: 5661.79 | bwd_allreduce_microstep: 55.92 | step_microstep: 19.25
[2025-04-25 14:08:49,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.26 | bwd: 5717.78 | bwd_inner: 5661.79 | bwd_allreduce: 55.94 | step: 19.25

  6%|▌         | 2564/41250 [6:11:15<93:16:54,  8.68s/it]
                                                         
{'loss': 0.0926, 'grad_norm': 1.0699102878570557, 'learning_rate': 3.989170361847355e-05, 'epoch': 0.62}

  6%|▌         | 2564/41250 [6:11:15<93:16:54,  8.68s/it][2025-04-25 14:08:58,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 14:08:58,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.58 | bwd_microstep: 5759.40 | bwd_inner_microstep: 5694.96 | bwd_allreduce_microstep: 64.39 | step_microstep: 18.81
[2025-04-25 14:08:58,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.58 | bwd: 5759.41 | bwd_inner: 5694.96 | bwd_allreduce: 64.41 | step: 18.81

  6%|▌         | 2565/41250 [6:11:23<93:19:19,  8.68s/it]
                                                         
{'loss': 0.1625, 'grad_norm': 1.0723075866699219, 'learning_rate': 3.989154036177292e-05, 'epoch': 0.62}

  6%|▌         | 2565/41250 [6:11:23<93:19:19,  8.68s/it][2025-04-25 14:09:07,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 14:09:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.61 | bwd_microstep: 5909.75 | bwd_inner_microstep: 5659.44 | bwd_allreduce_microstep: 250.26 | step_microstep: 19.21
[2025-04-25 14:09:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.61 | bwd: 5909.76 | bwd_inner: 5659.44 | bwd_allreduce: 250.28 | step: 19.21

  6%|▌         | 2566/41250 [6:11:32<93:45:42,  8.73s/it]
                                                         
{'loss': 0.117, 'grad_norm': 1.111782431602478, 'learning_rate': 3.9891376982444855e-05, 'epoch': 0.62}

  6%|▌         | 2566/41250 [6:11:32<93:45:42,  8.73s/it][2025-04-25 14:09:16,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 14:09:16,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5878.68 | bwd_inner_microstep: 5702.75 | bwd_allreduce_microstep: 175.88 | step_microstep: 18.83
[2025-04-25 14:09:16,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5878.70 | bwd_inner: 5702.75 | bwd_allreduce: 175.90 | step: 18.83

  6%|▌         | 2567/41250 [6:11:41<94:02:53,  8.75s/it]
                                                         
{'loss': 0.2424, 'grad_norm': 1.992887020111084, 'learning_rate': 3.989121348049037e-05, 'epoch': 0.62}

  6%|▌         | 2567/41250 [6:11:41<94:02:53,  8.75s/it][2025-04-25 14:09:24,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:09:24,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.95 | bwd_microstep: 5802.94 | bwd_inner_microstep: 5658.48 | bwd_allreduce_microstep: 144.41 | step_microstep: 18.50
[2025-04-25 14:09:24,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.95 | bwd: 5802.95 | bwd_inner: 5658.48 | bwd_allreduce: 144.43 | step: 18.50

  6%|▌         | 2568/41250 [6:11:50<93:55:04,  8.74s/it]
                                                         
{'loss': 0.0499, 'grad_norm': 0.48651018738746643, 'learning_rate': 3.989104985591046e-05, 'epoch': 0.62}

  6%|▌         | 2568/41250 [6:11:50<93:55:04,  8.74s/it][2025-04-25 14:09:33,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:09:33,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.66 | bwd_microstep: 5730.63 | bwd_inner_microstep: 5717.88 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.93
[2025-04-25 14:09:33,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.66 | bwd: 5730.64 | bwd_inner: 5717.88 | bwd_allreduce: 12.72 | step: 18.94

  6%|▌         | 2569/41250 [6:11:58<93:42:14,  8.72s/it]
                                                         
{'loss': 0.3149, 'grad_norm': 2.673996925354004, 'learning_rate': 3.989088610870614e-05, 'epoch': 0.62}

  6%|▌         | 2569/41250 [6:11:58<93:42:14,  8.72s/it][2025-04-25 14:09:42,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:09:42,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5793.16 | bwd_inner_microstep: 5695.91 | bwd_allreduce_microstep: 97.21 | step_microstep: 18.59
[2025-04-25 14:09:42,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5793.17 | bwd_inner: 5695.91 | bwd_allreduce: 97.23 | step: 18.59

  6%|▌         | 2570/41250 [6:12:07<93:42:06,  8.72s/it]
                                                         
{'loss': 0.3655, 'grad_norm': 3.181227445602417, 'learning_rate': 3.989072223887842e-05, 'epoch': 0.62}

  6%|▌         | 2570/41250 [6:12:07<93:42:06,  8.72s/it][2025-04-25 14:09:50,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:09:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5761.88 | bwd_inner_microstep: 5704.53 | bwd_allreduce_microstep: 57.31 | step_microstep: 18.72
[2025-04-25 14:09:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5761.89 | bwd_inner: 5704.53 | bwd_allreduce: 57.33 | step: 18.72

  6%|▌         | 2571/41250 [6:12:16<93:38:13,  8.72s/it]
                                                         
{'loss': 0.1612, 'grad_norm': 1.3458205461502075, 'learning_rate': 3.9890558246428304e-05, 'epoch': 0.62}

  6%|▌         | 2571/41250 [6:12:16<93:38:13,  8.72s/it][2025-04-25 14:09:59,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 14:09:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.74 | bwd_microstep: 5778.45 | bwd_inner_microstep: 5704.03 | bwd_allreduce_microstep: 74.38 | step_microstep: 19.66
[2025-04-25 14:09:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.74 | bwd: 5778.47 | bwd_inner: 5704.03 | bwd_allreduce: 74.39 | step: 19.66

  6%|▌         | 2572/41250 [6:12:24<93:38:20,  8.72s/it]
                                                         
{'loss': 0.0768, 'grad_norm': 0.6165193915367126, 'learning_rate': 3.9890394131356816e-05, 'epoch': 0.62}

  6%|▌         | 2572/41250 [6:12:24<93:38:20,  8.72s/it][2025-04-25 14:10:08,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 14:10:08,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.37 | bwd_microstep: 5714.38 | bwd_inner_microstep: 5654.56 | bwd_allreduce_microstep: 59.77 | step_microstep: 19.08
[2025-04-25 14:10:08,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.38 | bwd: 5714.40 | bwd_inner: 5654.56 | bwd_allreduce: 59.79 | step: 19.08

  6%|▌         | 2573/41250 [6:12:33<93:21:19,  8.69s/it]
                                                         
{'loss': 0.2407, 'grad_norm': 1.1648281812667847, 'learning_rate': 3.989022989366496e-05, 'epoch': 0.62}

  6%|▌         | 2573/41250 [6:12:33<93:21:19,  8.69s/it][2025-04-25 14:10:16,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 14:10:16,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.20 | bwd_microstep: 5780.08 | bwd_inner_microstep: 5673.32 | bwd_allreduce_microstep: 106.72 | step_microstep: 19.11
[2025-04-25 14:10:16,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.20 | bwd: 5780.10 | bwd_inner: 5673.32 | bwd_allreduce: 106.74 | step: 19.12

  6%|▌         | 2574/41250 [6:12:42<93:23:54,  8.69s/it]
                                                         
{'loss': 0.201, 'grad_norm': 1.7427090406417847, 'learning_rate': 3.989006553335375e-05, 'epoch': 0.62}

  6%|▌         | 2574/41250 [6:12:42<93:23:54,  8.69s/it][2025-04-25 14:10:25,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.21 | optimizer_step: 0.89
[2025-04-25 14:10:25,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.67 | bwd_microstep: 5757.89 | bwd_inner_microstep: 5725.48 | bwd_allreduce_microstep: 32.35 | step_microstep: 19.01
[2025-04-25 14:10:25,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.67 | bwd: 5757.91 | bwd_inner: 5725.48 | bwd_allreduce: 32.38 | step: 19.01

  6%|▌         | 2575/41250 [6:12:50<93:26:04,  8.70s/it]
                                                         
{'loss': 0.1, 'grad_norm': 0.8338434100151062, 'learning_rate': 3.988990105042419e-05, 'epoch': 0.62}

  6%|▌         | 2575/41250 [6:12:50<93:26:04,  8.70s/it][2025-04-25 14:10:34,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:10:34,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.70 | bwd_microstep: 5709.71 | bwd_inner_microstep: 5644.83 | bwd_allreduce_microstep: 64.83 | step_microstep: 18.64
[2025-04-25 14:10:34,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.70 | bwd: 5709.72 | bwd_inner: 5644.83 | bwd_allreduce: 64.86 | step: 18.64

  6%|▌         | 2576/41250 [6:12:59<93:10:30,  8.67s/it]
                                                         
{'loss': 0.0657, 'grad_norm': 0.6331603527069092, 'learning_rate': 3.9889736444877313e-05, 'epoch': 0.62}

  6%|▌         | 2576/41250 [6:12:59<93:10:30,  8.67s/it][2025-04-25 14:10:43,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.36 | optimizer_step: 1.03
[2025-04-25 14:10:43,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.93 | bwd_microstep: 5847.11 | bwd_inner_microstep: 5718.38 | bwd_allreduce_microstep: 128.67 | step_microstep: 20.09
[2025-04-25 14:10:43,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.93 | bwd: 5847.12 | bwd_inner: 5718.38 | bwd_allreduce: 128.69 | step: 20.10

  6%|▌         | 2577/41250 [6:13:08<93:32:39,  8.71s/it]
                                                         
{'loss': 0.1309, 'grad_norm': 1.2475676536560059, 'learning_rate': 3.9889571716714115e-05, 'epoch': 0.62}

  6%|▌         | 2577/41250 [6:13:08<93:32:39,  8.71s/it][2025-04-25 14:10:51,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:10:51,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.78 | bwd_microstep: 5697.59 | bwd_inner_microstep: 5668.63 | bwd_allreduce_microstep: 28.91 | step_microstep: 19.29
[2025-04-25 14:10:51,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.78 | bwd: 5697.61 | bwd_inner: 5668.62 | bwd_allreduce: 28.93 | step: 19.30

  6%|▌         | 2578/41250 [6:13:16<93:14:45,  8.68s/it]
                                                         
{'loss': 0.1681, 'grad_norm': 2.1109588146209717, 'learning_rate': 3.988940686593562e-05, 'epoch': 0.62}

  6%|▌         | 2578/41250 [6:13:16<93:14:45,  8.68s/it][2025-04-25 14:11:00,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 14:11:00,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.05 | bwd_microstep: 5765.64 | bwd_inner_microstep: 5666.96 | bwd_allreduce_microstep: 98.63 | step_microstep: 18.45
[2025-04-25 14:11:00,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.05 | bwd: 5765.65 | bwd_inner: 5666.96 | bwd_allreduce: 98.65 | step: 18.46

  6%|▋         | 2579/41250 [6:13:25<93:14:02,  8.68s/it]
                                                         
{'loss': 0.1692, 'grad_norm': 1.9706251621246338, 'learning_rate': 3.988924189254285e-05, 'epoch': 0.63}

  6%|▋         | 2579/41250 [6:13:25<93:14:02,  8.68s/it][2025-04-25 14:11:09,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:11:09,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5752.47 | bwd_inner_microstep: 5664.50 | bwd_allreduce_microstep: 87.92 | step_microstep: 18.62
[2025-04-25 14:11:09,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5752.48 | bwd_inner: 5664.50 | bwd_allreduce: 87.94 | step: 18.62

  6%|▋         | 2580/41250 [6:13:34<93:11:06,  8.68s/it]
                                                         
{'loss': 0.2587, 'grad_norm': 1.7544610500335693, 'learning_rate': 3.988907679653681e-05, 'epoch': 0.63}

  6%|▋         | 2580/41250 [6:13:34<93:11:06,  8.68s/it][2025-04-25 14:11:17,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:11:17,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5858.11 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 163.27 | step_microstep: 18.99
[2025-04-25 14:11:17,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5858.12 | bwd_inner: 5694.79 | bwd_allreduce: 163.29 | step: 18.99

  6%|▋         | 2581/41250 [6:13:43<93:33:17,  8.71s/it]
                                                         
{'loss': 0.3096, 'grad_norm': 1.5247656106948853, 'learning_rate': 3.9888911577918525e-05, 'epoch': 0.63}

  6%|▋         | 2581/41250 [6:13:43<93:33:17,  8.71s/it][2025-04-25 14:11:26,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.30 | optimizer_step: 1.05
[2025-04-25 14:11:26,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.98 | bwd_microstep: 5757.63 | bwd_inner_microstep: 5659.15 | bwd_allreduce_microstep: 98.42 | step_microstep: 20.19
[2025-04-25 14:11:26,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.98 | bwd: 5757.64 | bwd_inner: 5659.14 | bwd_allreduce: 98.45 | step: 20.19

  6%|▋         | 2582/41250 [6:13:51<93:29:08,  8.70s/it]
                                                         
{'loss': 0.1723, 'grad_norm': 2.3200318813323975, 'learning_rate': 3.9888746236689014e-05, 'epoch': 0.63}

  6%|▋         | 2582/41250 [6:13:51<93:29:08,  8.70s/it][2025-04-25 14:11:35,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.26 | optimizer_step: 0.94
[2025-04-25 14:11:35,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.79 | bwd_microstep: 5737.62 | bwd_inner_microstep: 5646.47 | bwd_allreduce_microstep: 91.10 | step_microstep: 19.74
[2025-04-25 14:11:35,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.79 | bwd: 5737.64 | bwd_inner: 5646.47 | bwd_allreduce: 91.12 | step: 19.74

  6%|▋         | 2583/41250 [6:14:00<93:18:48,  8.69s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 2.1723954677581787, 'learning_rate': 3.9888580772849295e-05, 'epoch': 0.63}

  6%|▋         | 2583/41250 [6:14:00<93:18:48,  8.69s/it][2025-04-25 14:11:43,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 14:11:43,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5770.62 | bwd_inner_microstep: 5654.81 | bwd_allreduce_microstep: 115.76 | step_microstep: 19.35
[2025-04-25 14:11:43,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5770.63 | bwd_inner: 5654.81 | bwd_allreduce: 115.78 | step: 19.35

  6%|▋         | 2584/41250 [6:14:09<93:18:45,  8.69s/it]
                                                         
{'loss': 0.0748, 'grad_norm': 0.9281598925590515, 'learning_rate': 3.9888415186400385e-05, 'epoch': 0.63}

  6%|▋         | 2584/41250 [6:14:09<93:18:45,  8.69s/it][2025-04-25 14:11:52,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:11:52,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5715.63 | bwd_inner_microstep: 5702.95 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.85
[2025-04-25 14:11:52,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.03 | bwd: 5715.64 | bwd_inner: 5702.95 | bwd_allreduce: 12.64 | step: 18.85

  6%|▋         | 2585/41250 [6:14:17<93:11:36,  8.68s/it]
                                                         
{'loss': 0.1113, 'grad_norm': 1.0224114656448364, 'learning_rate': 3.9888249477343306e-05, 'epoch': 0.63}

  6%|▋         | 2585/41250 [6:14:17<93:11:36,  8.68s/it][2025-04-25 14:12:01,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.14 | optimizer_step: 0.90
[2025-04-25 14:12:01,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.31 | bwd_microstep: 5914.72 | bwd_inner_microstep: 5685.97 | bwd_allreduce_microstep: 228.71 | step_microstep: 19.21
[2025-04-25 14:12:01,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.31 | bwd: 5914.73 | bwd_inner: 5685.97 | bwd_allreduce: 228.72 | step: 19.22

  6%|▋         | 2586/41250 [6:14:26<93:44:16,  8.73s/it]
                                                         
{'loss': 0.1504, 'grad_norm': 2.5610504150390625, 'learning_rate': 3.988808364567908e-05, 'epoch': 0.63}

  6%|▋         | 2586/41250 [6:14:26<93:44:16,  8.73s/it][2025-04-25 14:12:10,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.32 | optimizer_step: 1.05
[2025-04-25 14:12:10,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.99 | bwd_microstep: 5781.48 | bwd_inner_microstep: 5767.27 | bwd_allreduce_microstep: 14.14 | step_microstep: 20.04
[2025-04-25 14:12:10,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.99 | bwd: 5781.49 | bwd_inner: 5767.27 | bwd_allreduce: 14.18 | step: 20.04

  6%|▋         | 2587/41250 [6:14:35<93:48:25,  8.73s/it]
                                                         
{'loss': 0.1159, 'grad_norm': 0.9887465834617615, 'learning_rate': 3.988791769140873e-05, 'epoch': 0.63}

  6%|▋         | 2587/41250 [6:14:35<93:48:25,  8.73s/it][2025-04-25 14:12:18,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.25 | optimizer_step: 0.96
[2025-04-25 14:12:18,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.91 | bwd_microstep: 5689.66 | bwd_inner_microstep: 5646.49 | bwd_allreduce_microstep: 43.11 | step_microstep: 19.71
[2025-04-25 14:12:18,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.91 | bwd: 5689.67 | bwd_inner: 5646.49 | bwd_allreduce: 43.14 | step: 19.72

  6%|▋         | 2588/41250 [6:14:43<93:21:42,  8.69s/it]
                                                         
{'loss': 0.2499, 'grad_norm': 1.5701398849487305, 'learning_rate': 3.9887751614533286e-05, 'epoch': 0.63}

  6%|▋         | 2588/41250 [6:14:43<93:21:42,  8.69s/it][2025-04-25 14:12:27,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 14:12:27,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5714.87 | bwd_inner_microstep: 5698.60 | bwd_allreduce_microstep: 16.22 | step_microstep: 19.53
[2025-04-25 14:12:27,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5714.89 | bwd_inner: 5698.60 | bwd_allreduce: 16.24 | step: 19.53

  6%|▋         | 2589/41250 [6:14:52<93:13:38,  8.68s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 0.9630138874053955, 'learning_rate': 3.988758541505376e-05, 'epoch': 0.63}

  6%|▋         | 2589/41250 [6:14:52<93:13:38,  8.68s/it][2025-04-25 14:12:36,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.30 | optimizer_step: 1.07
[2025-04-25 14:12:36,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.51 | bwd_microstep: 5862.59 | bwd_inner_microstep: 5654.02 | bwd_allreduce_microstep: 208.51 | step_microstep: 20.04
[2025-04-25 14:12:36,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.51 | bwd: 5862.61 | bwd_inner: 5654.02 | bwd_allreduce: 208.54 | step: 20.04

  6%|▋         | 2590/41250 [6:15:01<93:32:31,  8.71s/it]
                                                         
{'loss': 0.3523, 'grad_norm': 2.320077657699585, 'learning_rate': 3.988741909297118e-05, 'epoch': 0.63}

  6%|▋         | 2590/41250 [6:15:01<93:32:31,  8.71s/it][2025-04-25 14:12:44,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 14:12:44,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.97 | bwd_microstep: 5684.79 | bwd_inner_microstep: 5648.13 | bwd_allreduce_microstep: 36.61 | step_microstep: 18.81
[2025-04-25 14:12:44,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.97 | bwd: 5684.80 | bwd_inner: 5648.13 | bwd_allreduce: 36.63 | step: 18.81

  6%|▋         | 2591/41250 [6:15:10<93:11:31,  8.68s/it]
                                                         
{'loss': 0.0702, 'grad_norm': 0.8101088404655457, 'learning_rate': 3.9887252648286575e-05, 'epoch': 0.63}

  6%|▋         | 2591/41250 [6:15:10<93:11:31,  8.68s/it][2025-04-25 14:12:53,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:12:53,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.04 | bwd_microstep: 5753.85 | bwd_inner_microstep: 5649.94 | bwd_allreduce_microstep: 103.87 | step_microstep: 18.68
[2025-04-25 14:12:53,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.04 | bwd: 5753.87 | bwd_inner: 5649.94 | bwd_allreduce: 103.88 | step: 18.68

  6%|▋         | 2592/41250 [6:15:18<93:10:41,  8.68s/it]
                                                         
{'loss': 0.139, 'grad_norm': 2.0260610580444336, 'learning_rate': 3.988708608100097e-05, 'epoch': 0.63}

  6%|▋         | 2592/41250 [6:15:18<93:10:41,  8.68s/it][2025-04-25 14:13:02,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:13:02,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5750.77 | bwd_inner_microstep: 5679.57 | bwd_allreduce_microstep: 71.16 | step_microstep: 18.33
[2025-04-25 14:13:02,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5750.79 | bwd_inner: 5679.57 | bwd_allreduce: 71.18 | step: 18.33

  6%|▋         | 2593/41250 [6:15:27<93:12:58,  8.68s/it]
                                                         
{'loss': 0.3287, 'grad_norm': 2.041576385498047, 'learning_rate': 3.9886919391115394e-05, 'epoch': 0.63}

  6%|▋         | 2593/41250 [6:15:27<93:12:58,  8.68s/it][2025-04-25 14:13:10,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 14:13:10,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.26 | bwd_microstep: 5714.97 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.51 | step_microstep: 19.22
[2025-04-25 14:13:10,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.26 | bwd: 5714.98 | bwd_inner: 5702.41 | bwd_allreduce: 12.52 | step: 19.22

  6%|▋         | 2594/41250 [6:15:36<93:07:04,  8.67s/it]
                                                         
{'loss': 0.1167, 'grad_norm': 1.754439115524292, 'learning_rate': 3.988675257863087e-05, 'epoch': 0.63}

  6%|▋         | 2594/41250 [6:15:36<93:07:04,  8.67s/it][2025-04-25 14:13:19,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:13:19,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5876.18 | bwd_inner_microstep: 5715.76 | bwd_allreduce_microstep: 160.38 | step_microstep: 18.59
[2025-04-25 14:13:19,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5876.19 | bwd_inner: 5715.76 | bwd_allreduce: 160.39 | step: 18.60

  6%|▋         | 2595/41250 [6:15:44<93:34:20,  8.71s/it]
                                                         
{'loss': 0.0419, 'grad_norm': 1.1849825382232666, 'learning_rate': 3.988658564354842e-05, 'epoch': 0.63}

  6%|▋         | 2595/41250 [6:15:44<93:34:20,  8.71s/it][2025-04-25 14:13:28,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 14:13:28,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.23 | bwd_microstep: 5710.95 | bwd_inner_microstep: 5697.86 | bwd_allreduce_microstep: 13.04 | step_microstep: 19.15
[2025-04-25 14:13:28,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.23 | bwd: 5710.96 | bwd_inner: 5697.86 | bwd_allreduce: 13.06 | step: 19.15

  6%|▋         | 2596/41250 [6:15:53<93:22:38,  8.70s/it]
                                                         
{'loss': 0.1732, 'grad_norm': 1.7161813974380493, 'learning_rate': 3.988641858586908e-05, 'epoch': 0.63}

  6%|▋         | 2596/41250 [6:15:53<93:22:38,  8.70s/it][2025-04-25 14:13:36,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:13:36,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.59 | bwd_microstep: 5814.50 | bwd_inner_microstep: 5703.03 | bwd_allreduce_microstep: 111.42 | step_microstep: 18.70
[2025-04-25 14:13:36,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.60 | bwd: 5814.51 | bwd_inner: 5703.03 | bwd_allreduce: 111.44 | step: 18.70

  6%|▋         | 2597/41250 [6:16:02<93:32:52,  8.71s/it]
                                                         
{'loss': 0.0563, 'grad_norm': 0.6920148134231567, 'learning_rate': 3.988625140559389e-05, 'epoch': 0.63}

  6%|▋         | 2597/41250 [6:16:02<93:32:52,  8.71s/it][2025-04-25 14:13:45,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 14:13:45,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5703.69 | bwd_inner_microstep: 5690.65 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.97
[2025-04-25 14:13:45,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.73 | bwd: 5703.70 | bwd_inner: 5690.64 | bwd_allreduce: 13.02 | step: 18.97

  6%|▋         | 2598/41250 [6:16:10<93:18:39,  8.69s/it]
                                                         
{'loss': 0.0907, 'grad_norm': 1.058284044265747, 'learning_rate': 3.9886084102723874e-05, 'epoch': 0.63}

  6%|▋         | 2598/41250 [6:16:10<93:18:39,  8.69s/it][2025-04-25 14:13:54,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.16 | optimizer_step: 1.07
[2025-04-25 14:13:54,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.89 | bwd_microstep: 5727.51 | bwd_inner_microstep: 5655.21 | bwd_allreduce_microstep: 72.23 | step_microstep: 20.27
[2025-04-25 14:13:54,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.89 | bwd: 5727.53 | bwd_inner: 5655.21 | bwd_allreduce: 72.27 | step: 20.27

  6%|▋         | 2599/41250 [6:16:19<93:11:53,  8.68s/it]
                                                         
{'loss': 0.668, 'grad_norm': 3.0420925617218018, 'learning_rate': 3.988591667726005e-05, 'epoch': 0.63}

  6%|▋         | 2599/41250 [6:16:19<93:11:53,  8.68s/it][2025-04-25 14:14:02,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 14:14:02,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.43 | bwd_microstep: 5758.84 | bwd_inner_microstep: 5643.15 | bwd_allreduce_microstep: 115.64 | step_microstep: 18.69
[2025-04-25 14:14:02,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.43 | bwd: 5758.85 | bwd_inner: 5643.15 | bwd_allreduce: 115.66 | step: 18.69

  6%|▋         | 2600/41250 [6:16:28<93:08:59,  8.68s/it]
                                                         
{'loss': 0.2768, 'grad_norm': 2.2254390716552734, 'learning_rate': 3.988574912920347e-05, 'epoch': 0.63}

  6%|▋         | 2600/41250 [6:16:28<93:08:59,  8.68s/it][2025-04-25 14:14:11,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 14:14:11,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.42 | bwd_microstep: 5748.51 | bwd_inner_microstep: 5644.78 | bwd_allreduce_microstep: 103.67 | step_microstep: 18.89
[2025-04-25 14:14:11,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.42 | bwd: 5748.52 | bwd_inner: 5644.78 | bwd_allreduce: 103.70 | step: 18.89

  6%|▋         | 2601/41250 [6:16:36<93:05:50,  8.67s/it]
                                                         
{'loss': 0.05, 'grad_norm': 1.027912974357605, 'learning_rate': 3.988558145855515e-05, 'epoch': 0.63}

  6%|▋         | 2601/41250 [6:16:36<93:05:50,  8.67s/it][2025-04-25 14:14:20,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.27 | optimizer_step: 1.04
[2025-04-25 14:14:20,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5707.15 | bwd_inner_microstep: 5693.12 | bwd_allreduce_microstep: 13.97 | step_microstep: 20.21
[2025-04-25 14:14:20,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5707.17 | bwd_inner: 5693.12 | bwd_allreduce: 14.00 | step: 20.21

  6%|▋         | 2602/41250 [6:16:45<92:59:15,  8.66s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 1.4250680208206177, 'learning_rate': 3.988541366531614e-05, 'epoch': 0.63}

  6%|▋         | 2602/41250 [6:16:45<92:59:15,  8.66s/it][2025-04-25 14:14:28,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 14:14:28,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.45 | bwd_microstep: 5705.94 | bwd_inner_microstep: 5693.33 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.90
[2025-04-25 14:14:28,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.45 | bwd: 5705.95 | bwd_inner: 5693.33 | bwd_allreduce: 12.58 | step: 18.90

  6%|▋         | 2603/41250 [6:16:54<92:55:37,  8.66s/it]
                                                         
{'loss': 0.2981, 'grad_norm': 2.3698675632476807, 'learning_rate': 3.988524574948746e-05, 'epoch': 0.63}

  6%|▋         | 2603/41250 [6:16:54<92:55:37,  8.66s/it][2025-04-25 14:14:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:14:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.03 | bwd_microstep: 5697.83 | bwd_inner_microstep: 5685.11 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.65
[2025-04-25 14:14:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.03 | bwd: 5697.84 | bwd_inner: 5685.11 | bwd_allreduce: 12.69 | step: 18.65

  6%|▋         | 2604/41250 [6:17:02<92:48:41,  8.65s/it]
                                                         
{'loss': 0.1472, 'grad_norm': 3.198838710784912, 'learning_rate': 3.988507771107015e-05, 'epoch': 0.63}

  6%|▋         | 2604/41250 [6:17:02<92:48:41,  8.65s/it][2025-04-25 14:14:46,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:14:46,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5714.98 | bwd_inner_microstep: 5696.71 | bwd_allreduce_microstep: 18.23 | step_microstep: 18.79
[2025-04-25 14:14:46,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5715.00 | bwd_inner: 5696.71 | bwd_allreduce: 18.25 | step: 18.79

  6%|▋         | 2605/41250 [6:17:11<92:49:52,  8.65s/it]
                                                         
{'loss': 0.1454, 'grad_norm': 1.2128796577453613, 'learning_rate': 3.9884909550065255e-05, 'epoch': 0.63}

  6%|▋         | 2605/41250 [6:17:11<92:49:52,  8.65s/it][2025-04-25 14:14:54,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:14:54,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5740.44 | bwd_inner_microstep: 5648.38 | bwd_allreduce_microstep: 92.01 | step_microstep: 18.93
[2025-04-25 14:14:54,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.21 | bwd: 5740.45 | bwd_inner: 5648.38 | bwd_allreduce: 92.03 | step: 18.93

  6%|▋         | 2606/41250 [6:17:20<92:49:41,  8.65s/it]
                                                         
{'loss': 0.1563, 'grad_norm': 1.2206307649612427, 'learning_rate': 3.9884741266473794e-05, 'epoch': 0.63}

  6%|▋         | 2606/41250 [6:17:20<92:49:41,  8.65s/it][2025-04-25 14:15:03,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 14:15:03,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5741.89 | bwd_inner_microstep: 5648.75 | bwd_allreduce_microstep: 93.09 | step_microstep: 19.08
[2025-04-25 14:15:03,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5741.90 | bwd_inner: 5648.75 | bwd_allreduce: 93.11 | step: 19.08

  6%|▋         | 2607/41250 [6:17:28<92:52:42,  8.65s/it]
                                                         
{'loss': 0.123, 'grad_norm': 0.8300061821937561, 'learning_rate': 3.9884572860296824e-05, 'epoch': 0.63}

  6%|▋         | 2607/41250 [6:17:28<92:52:42,  8.65s/it][2025-04-25 14:15:12,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 14:15:12,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.42 | bwd_microstep: 5671.94 | bwd_inner_microstep: 5651.80 | bwd_allreduce_microstep: 20.10 | step_microstep: 18.67
[2025-04-25 14:15:12,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.42 | bwd: 5671.95 | bwd_inner: 5651.80 | bwd_allreduce: 20.11 | step: 18.68

  6%|▋         | 2608/41250 [6:17:37<92:38:17,  8.63s/it]
                                                         
{'loss': 0.076, 'grad_norm': 0.771482527256012, 'learning_rate': 3.988440433153537e-05, 'epoch': 0.63}

  6%|▋         | 2608/41250 [6:17:37<92:38:17,  8.63s/it][2025-04-25 14:15:20,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:15:20,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5699.03 | bwd_inner_microstep: 5686.26 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.20
[2025-04-25 14:15:20,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5699.04 | bwd_inner: 5686.26 | bwd_allreduce: 12.74 | step: 18.20

  6%|▋         | 2609/41250 [6:17:45<92:38:14,  8.63s/it]
                                                         
{'loss': 0.1597, 'grad_norm': 1.0923256874084473, 'learning_rate': 3.988423568019047e-05, 'epoch': 0.63}

  6%|▋         | 2609/41250 [6:17:45<92:38:14,  8.63s/it][2025-04-25 14:15:29,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.10 | optimizer_step: 0.99
[2025-04-25 14:15:29,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.38 | bwd_microstep: 5745.18 | bwd_inner_microstep: 5639.99 | bwd_allreduce_microstep: 105.14 | step_microstep: 18.85
[2025-04-25 14:15:29,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.38 | bwd: 5745.19 | bwd_inner: 5639.99 | bwd_allreduce: 105.16 | step: 18.86

  6%|▋         | 2610/41250 [6:17:54<92:43:37,  8.64s/it]
                                                         
{'loss': 0.1109, 'grad_norm': 0.7937102913856506, 'learning_rate': 3.988406690626317e-05, 'epoch': 0.63}

  6%|▋         | 2610/41250 [6:17:54<92:43:37,  8.64s/it][2025-04-25 14:15:37,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:15:37,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.66 | bwd_microstep: 5743.34 | bwd_inner_microstep: 5681.52 | bwd_allreduce_microstep: 61.76 | step_microstep: 18.73
[2025-04-25 14:15:37,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.66 | bwd: 5743.35 | bwd_inner: 5681.52 | bwd_allreduce: 61.78 | step: 18.73

  6%|▋         | 2611/41250 [6:18:03<92:49:31,  8.65s/it]
                                                         
{'loss': 0.4297, 'grad_norm': 2.248690128326416, 'learning_rate': 3.988389800975451e-05, 'epoch': 0.63}

  6%|▋         | 2611/41250 [6:18:03<92:49:31,  8.65s/it][2025-04-25 14:15:46,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:15:46,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5715.26 | bwd_inner_microstep: 5672.88 | bwd_allreduce_microstep: 42.34 | step_microstep: 18.80
[2025-04-25 14:15:46,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.42 | bwd: 5715.28 | bwd_inner: 5672.88 | bwd_allreduce: 42.35 | step: 18.80

  6%|▋         | 2612/41250 [6:18:11<92:47:35,  8.65s/it]
                                                         
{'loss': 0.4411, 'grad_norm': 2.9325644969940186, 'learning_rate': 3.988372899066553e-05, 'epoch': 0.63}

  6%|▋         | 2612/41250 [6:18:11<92:47:35,  8.65s/it][2025-04-25 14:15:55,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.29 | optimizer_step: 1.06
[2025-04-25 14:15:55,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.40 | bwd_microstep: 5694.33 | bwd_inner_microstep: 5680.95 | bwd_allreduce_microstep: 13.32 | step_microstep: 20.06
[2025-04-25 14:15:55,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.40 | bwd: 5694.35 | bwd_inner: 5680.95 | bwd_allreduce: 13.35 | step: 20.06

  6%|▋         | 2613/41250 [6:18:20<92:43:42,  8.64s/it]
                                                         
{'loss': 0.1833, 'grad_norm': 2.247032642364502, 'learning_rate': 3.988355984899727e-05, 'epoch': 0.63}

  6%|▋         | 2613/41250 [6:18:20<92:43:42,  8.64s/it][2025-04-25 14:16:03,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 14:16:03,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5707.75 | bwd_inner_microstep: 5691.18 | bwd_allreduce_microstep: 16.53 | step_microstep: 19.15
[2025-04-25 14:16:03,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5707.76 | bwd_inner: 5691.18 | bwd_allreduce: 16.55 | step: 19.15

  6%|▋         | 2614/41250 [6:18:29<92:43:47,  8.64s/it]
                                                         
{'loss': 0.2067, 'grad_norm': 1.7252748012542725, 'learning_rate': 3.988339058475077e-05, 'epoch': 0.63}

  6%|▋         | 2614/41250 [6:18:29<92:43:47,  8.64s/it][2025-04-25 14:16:12,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.18 | optimizer_step: 0.89
[2025-04-25 14:16:12,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.89 | bwd_microstep: 5783.87 | bwd_inner_microstep: 5651.93 | bwd_allreduce_microstep: 131.89 | step_microstep: 18.99
[2025-04-25 14:16:12,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.89 | bwd: 5783.88 | bwd_inner: 5651.93 | bwd_allreduce: 131.91 | step: 19.00

  6%|▋         | 2615/41250 [6:18:37<92:53:13,  8.66s/it]
                                                         
{'loss': 0.1111, 'grad_norm': 0.7403745651245117, 'learning_rate': 3.9883221197927085e-05, 'epoch': 0.63}

  6%|▋         | 2615/41250 [6:18:37<92:53:13,  8.66s/it][2025-04-25 14:16:21,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:16:21,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.37 | bwd_microstep: 5728.85 | bwd_inner_microstep: 5698.06 | bwd_allreduce_microstep: 30.74 | step_microstep: 19.01
[2025-04-25 14:16:21,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.37 | bwd: 5728.87 | bwd_inner: 5698.06 | bwd_allreduce: 30.76 | step: 19.01

  6%|▋         | 2616/41250 [6:18:46<92:54:53,  8.66s/it]
                                                         
{'loss': 0.0431, 'grad_norm': 0.44707486033439636, 'learning_rate': 3.9883051688527255e-05, 'epoch': 0.63}

  6%|▋         | 2616/41250 [6:18:46<92:54:53,  8.66s/it][2025-04-25 14:16:29,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:16:29,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5717.52 | bwd_inner_microstep: 5704.83 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.69
[2025-04-25 14:16:29,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5717.53 | bwd_inner: 5704.83 | bwd_allreduce: 12.66 | step: 18.70

  6%|▋         | 2617/41250 [6:18:55<92:53:47,  8.66s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.9299042224884033, 'learning_rate': 3.988288205655232e-05, 'epoch': 0.63}

  6%|▋         | 2617/41250 [6:18:55<92:53:47,  8.66s/it][2025-04-25 14:16:38,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:16:38,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.08 | bwd_microstep: 5748.92 | bwd_inner_microstep: 5644.23 | bwd_allreduce_microstep: 104.64 | step_microstep: 18.79
[2025-04-25 14:16:38,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.08 | bwd: 5748.93 | bwd_inner: 5644.23 | bwd_allreduce: 104.66 | step: 18.80

  6%|▋         | 2618/41250 [6:19:03<92:53:38,  8.66s/it]
                                                         
{'loss': 0.1471, 'grad_norm': 0.9278730154037476, 'learning_rate': 3.988271230200332e-05, 'epoch': 0.63}

  6%|▋         | 2618/41250 [6:19:03<92:53:38,  8.66s/it][2025-04-25 14:16:47,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 14:16:47,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.18 | bwd_microstep: 5711.54 | bwd_inner_microstep: 5698.87 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.49
[2025-04-25 14:16:47,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.18 | bwd: 5711.56 | bwd_inner: 5698.87 | bwd_allreduce: 12.65 | step: 19.49

  6%|▋         | 2619/41250 [6:19:12<92:51:52,  8.65s/it]
                                                         
{'loss': 0.2711, 'grad_norm': 2.2659523487091064, 'learning_rate': 3.988254242488132e-05, 'epoch': 0.63}

  6%|▋         | 2619/41250 [6:19:12<92:51:52,  8.65s/it][2025-04-25 14:16:55,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.24 | optimizer_step: 0.93
[2025-04-25 14:16:55,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5704.93 | bwd_inner_microstep: 5649.89 | bwd_allreduce_microstep: 54.99 | step_microstep: 19.30
[2025-04-25 14:16:55,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5704.94 | bwd_inner: 5649.89 | bwd_allreduce: 55.01 | step: 19.30

  6%|▋         | 2620/41250 [6:19:21<92:45:20,  8.64s/it]
                                                         
{'loss': 0.034, 'grad_norm': 0.5984195470809937, 'learning_rate': 3.988237242518735e-05, 'epoch': 0.64}

  6%|▋         | 2620/41250 [6:19:21<92:45:20,  8.64s/it][2025-04-25 14:17:04,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:17:04,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.96 | bwd_microstep: 5743.33 | bwd_inner_microstep: 5646.57 | bwd_allreduce_microstep: 96.72 | step_microstep: 18.74
[2025-04-25 14:17:04,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.96 | bwd: 5743.35 | bwd_inner: 5646.57 | bwd_allreduce: 96.74 | step: 18.74

  6%|▋         | 2621/41250 [6:19:29<92:49:13,  8.65s/it]
                                                         
{'loss': 0.2577, 'grad_norm': 2.2374019622802734, 'learning_rate': 3.988220230292246e-05, 'epoch': 0.64}

  6%|▋         | 2621/41250 [6:19:29<92:49:13,  8.65s/it][2025-04-25 14:17:13,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 14:17:13,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.05 | bwd_microstep: 5762.98 | bwd_inner_microstep: 5702.75 | bwd_allreduce_microstep: 60.19 | step_microstep: 18.68
[2025-04-25 14:17:13,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.05 | bwd: 5762.99 | bwd_inner: 5702.75 | bwd_allreduce: 60.20 | step: 18.68

  6%|▋         | 2622/41250 [6:19:38<92:58:06,  8.66s/it]
                                                         
{'loss': 0.1564, 'grad_norm': 1.8861031532287598, 'learning_rate': 3.988203205808771e-05, 'epoch': 0.64}

  6%|▋         | 2622/41250 [6:19:38<92:58:06,  8.66s/it][2025-04-25 14:17:21,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-25 14:17:21,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5700.59 | bwd_inner_microstep: 5658.12 | bwd_allreduce_microstep: 42.43 | step_microstep: 18.64
[2025-04-25 14:17:21,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.82 | bwd: 5700.60 | bwd_inner: 5658.12 | bwd_allreduce: 42.44 | step: 18.64

  6%|▋         | 2623/41250 [6:19:47<92:48:35,  8.65s/it]
                                                         
{'loss': 0.2466, 'grad_norm': 2.9497594833374023, 'learning_rate': 3.9881861690684145e-05, 'epoch': 0.64}

  6%|▋         | 2623/41250 [6:19:47<92:48:35,  8.65s/it][2025-04-25 14:17:30,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 14:17:30,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.38 | bwd_microstep: 5744.16 | bwd_inner_microstep: 5710.82 | bwd_allreduce_microstep: 33.29 | step_microstep: 19.20
[2025-04-25 14:17:30,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.38 | bwd: 5744.17 | bwd_inner: 5710.82 | bwd_allreduce: 33.31 | step: 19.20

  6%|▋         | 2624/41250 [6:19:55<92:54:48,  8.66s/it]
                                                         
{'loss': 0.1635, 'grad_norm': 1.8955583572387695, 'learning_rate': 3.9881691200712815e-05, 'epoch': 0.64}

  6%|▋         | 2624/41250 [6:19:55<92:54:48,  8.66s/it][2025-04-25 14:17:39,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 14:17:39,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5716.11 | bwd_inner_microstep: 5703.34 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.68
[2025-04-25 14:17:39,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5716.12 | bwd_inner: 5703.34 | bwd_allreduce: 12.74 | step: 18.68

  6%|▋         | 2625/41250 [6:20:04<92:52:43,  8.66s/it]
                                                         
{'loss': 0.1062, 'grad_norm': 2.002882957458496, 'learning_rate': 3.988152058817476e-05, 'epoch': 0.64}

  6%|▋         | 2625/41250 [6:20:04<92:52:43,  8.66s/it][2025-04-25 14:17:47,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:17:47,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.07 | bwd_microstep: 5697.77 | bwd_inner_microstep: 5653.97 | bwd_allreduce_microstep: 43.75 | step_microstep: 19.24
[2025-04-25 14:17:47,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.07 | bwd: 5697.78 | bwd_inner: 5653.97 | bwd_allreduce: 43.77 | step: 19.26

  6%|▋         | 2626/41250 [6:20:13<92:44:19,  8.64s/it]
                                                         
{'loss': 0.2854, 'grad_norm': 2.5656919479370117, 'learning_rate': 3.9881349853071047e-05, 'epoch': 0.64}

  6%|▋         | 2626/41250 [6:20:13<92:44:19,  8.64s/it][2025-04-25 14:17:56,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:17:56,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.74 | bwd_microstep: 5733.14 | bwd_inner_microstep: 5693.04 | bwd_allreduce_microstep: 40.06 | step_microstep: 18.88
[2025-04-25 14:17:56,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.74 | bwd: 5733.16 | bwd_inner: 5693.04 | bwd_allreduce: 40.07 | step: 18.88

  6%|▋         | 2627/41250 [6:20:21<92:48:35,  8.65s/it]
                                                         
{'loss': 0.2293, 'grad_norm': 1.6288597583770752, 'learning_rate': 3.988117899540272e-05, 'epoch': 0.64}

  6%|▋         | 2627/41250 [6:20:21<92:48:35,  8.65s/it][2025-04-25 14:18:05,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 14:18:05,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.93 | bwd_microstep: 5766.06 | bwd_inner_microstep: 5709.53 | bwd_allreduce_microstep: 56.49 | step_microstep: 18.97
[2025-04-25 14:18:05,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.93 | bwd: 5766.08 | bwd_inner: 5709.53 | bwd_allreduce: 56.51 | step: 18.97

  6%|▋         | 2628/41250 [6:20:30<92:58:30,  8.67s/it]
                                                         
{'loss': 0.1369, 'grad_norm': 1.1141911745071411, 'learning_rate': 3.9881008015170836e-05, 'epoch': 0.64}

  6%|▋         | 2628/41250 [6:20:30<92:58:30,  8.67s/it][2025-04-25 14:18:13,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:18:13,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.02 | bwd_microstep: 5757.56 | bwd_inner_microstep: 5709.78 | bwd_allreduce_microstep: 47.73 | step_microstep: 18.78
[2025-04-25 14:18:13,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.02 | bwd: 5757.57 | bwd_inner: 5709.78 | bwd_allreduce: 47.75 | step: 18.78

  6%|▋         | 2629/41250 [6:20:39<93:03:28,  8.67s/it]
                                                         
{'loss': 0.1047, 'grad_norm': 1.0585132837295532, 'learning_rate': 3.988083691237645e-05, 'epoch': 0.64}

  6%|▋         | 2629/41250 [6:20:39<93:03:28,  8.67s/it][2025-04-25 14:18:22,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:18:22,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5775.96 | bwd_inner_microstep: 5656.03 | bwd_allreduce_microstep: 119.88 | step_microstep: 18.91
[2025-04-25 14:18:22,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5775.97 | bwd_inner: 5656.03 | bwd_allreduce: 119.90 | step: 18.91

  6%|▋         | 2630/41250 [6:20:47<93:05:31,  8.68s/it]
                                                         
{'loss': 0.4423, 'grad_norm': 2.8267312049865723, 'learning_rate': 3.9880665687020614e-05, 'epoch': 0.64}

  6%|▋         | 2630/41250 [6:20:47<93:05:31,  8.68s/it][2025-04-25 14:18:31,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:18:31,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.95 | bwd_microstep: 5756.38 | bwd_inner_microstep: 5686.71 | bwd_allreduce_microstep: 69.62 | step_microstep: 18.64
[2025-04-25 14:18:31,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.95 | bwd: 5756.40 | bwd_inner: 5686.71 | bwd_allreduce: 69.64 | step: 18.64

  6%|▋         | 2631/41250 [6:20:56<93:07:02,  8.68s/it]
                                                         
{'loss': 0.1757, 'grad_norm': 2.3051955699920654, 'learning_rate': 3.9880494339104384e-05, 'epoch': 0.64}

  6%|▋         | 2631/41250 [6:20:56<93:07:02,  8.68s/it][2025-04-25 14:18:39,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 14:18:39,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5732.76 | bwd_inner_microstep: 5719.98 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.31
[2025-04-25 14:18:39,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5732.77 | bwd_inner: 5719.98 | bwd_allreduce: 12.75 | step: 19.32

  6%|▋         | 2632/41250 [6:21:05<93:05:06,  8.68s/it]
                                                         
{'loss': 0.2112, 'grad_norm': 2.0848307609558105, 'learning_rate': 3.9880322868628816e-05, 'epoch': 0.64}

  6%|▋         | 2632/41250 [6:21:05<93:05:06,  8.68s/it][2025-04-25 14:18:48,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 14:18:48,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.79 | bwd_microstep: 5773.06 | bwd_inner_microstep: 5666.45 | bwd_allreduce_microstep: 106.56 | step_microstep: 18.80
[2025-04-25 14:18:48,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.79 | bwd: 5773.07 | bwd_inner: 5666.45 | bwd_allreduce: 106.58 | step: 18.80

  6%|▋         | 2633/41250 [6:21:13<93:08:17,  8.68s/it]
                                                         
{'loss': 0.1329, 'grad_norm': 1.7659423351287842, 'learning_rate': 3.988015127559497e-05, 'epoch': 0.64}

  6%|▋         | 2633/41250 [6:21:13<93:08:17,  8.68s/it][2025-04-25 14:18:57,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-25 14:18:57,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.93 | bwd_microstep: 5802.82 | bwd_inner_microstep: 5662.99 | bwd_allreduce_microstep: 139.79 | step_microstep: 18.41
[2025-04-25 14:18:57,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.93 | bwd: 5802.83 | bwd_inner: 5662.99 | bwd_allreduce: 139.81 | step: 18.41

  6%|▋         | 2634/41250 [6:21:22<93:14:45,  8.69s/it]
                                                         
{'loss': 0.2304, 'grad_norm': 2.569734573364258, 'learning_rate': 3.98799795600039e-05, 'epoch': 0.64}

  6%|▋         | 2634/41250 [6:21:22<93:14:45,  8.69s/it][2025-04-25 14:19:05,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:19:05,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.66 | bwd_microstep: 5774.20 | bwd_inner_microstep: 5687.09 | bwd_allreduce_microstep: 87.07 | step_microstep: 18.34
[2025-04-25 14:19:05,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.66 | bwd: 5774.21 | bwd_inner: 5687.09 | bwd_allreduce: 87.09 | step: 18.34

  6%|▋         | 2635/41250 [6:21:31<93:19:46,  8.70s/it]
                                                         
{'loss': 0.3199, 'grad_norm': 1.525107502937317, 'learning_rate': 3.987980772185666e-05, 'epoch': 0.64}

  6%|▋         | 2635/41250 [6:21:31<93:19:46,  8.70s/it][2025-04-25 14:19:14,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:19:14,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.18 | bwd_microstep: 5774.01 | bwd_inner_microstep: 5708.37 | bwd_allreduce_microstep: 65.59 | step_microstep: 18.73
[2025-04-25 14:19:14,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.18 | bwd: 5774.03 | bwd_inner: 5708.37 | bwd_allreduce: 65.61 | step: 18.73

  6%|▋         | 2636/41250 [6:21:40<93:22:51,  8.71s/it]
                                                         
{'loss': 0.1943, 'grad_norm': 1.9510704278945923, 'learning_rate': 3.987963576115433e-05, 'epoch': 0.64}

  6%|▋         | 2636/41250 [6:21:40<93:22:51,  8.71s/it][2025-04-25 14:19:23,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.13
[2025-04-25 14:19:23,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.17 | bwd_microstep: 5783.07 | bwd_inner_microstep: 5651.32 | bwd_allreduce_microstep: 131.70 | step_microstep: 19.54
[2025-04-25 14:19:23,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.17 | bwd: 5783.08 | bwd_inner: 5651.32 | bwd_allreduce: 131.72 | step: 19.54

  6%|▋         | 2637/41250 [6:21:48<93:22:27,  8.71s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.4221223592758179, 'learning_rate': 3.9879463677897946e-05, 'epoch': 0.64}

  6%|▋         | 2637/41250 [6:21:48<93:22:27,  8.71s/it][2025-04-25 14:19:32,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:19:32,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.01 | bwd_microstep: 5711.31 | bwd_inner_microstep: 5698.49 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.71
[2025-04-25 14:19:32,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.01 | bwd: 5711.32 | bwd_inner: 5698.49 | bwd_allreduce: 12.78 | step: 18.71

  6%|▋         | 2638/41250 [6:21:57<93:11:54,  8.69s/it]
                                                         
{'loss': 0.2122, 'grad_norm': 1.6602813005447388, 'learning_rate': 3.987929147208857e-05, 'epoch': 0.64}

  6%|▋         | 2638/41250 [6:21:57<93:11:54,  8.69s/it][2025-04-25 14:19:40,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:19:40,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.81 | bwd_microstep: 5737.06 | bwd_inner_microstep: 5693.71 | bwd_allreduce_microstep: 43.31 | step_microstep: 18.90
[2025-04-25 14:19:40,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.81 | bwd: 5737.07 | bwd_inner: 5693.71 | bwd_allreduce: 43.33 | step: 18.90

  6%|▋         | 2639/41250 [6:22:06<93:10:00,  8.69s/it]
                                                         
{'loss': 0.3196, 'grad_norm': 1.7301119565963745, 'learning_rate': 3.987911914372728e-05, 'epoch': 0.64}

  6%|▋         | 2639/41250 [6:22:06<93:10:00,  8.69s/it][2025-04-25 14:19:49,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:19:49,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.78 | bwd_microstep: 5712.38 | bwd_inner_microstep: 5654.42 | bwd_allreduce_microstep: 57.91 | step_microstep: 18.87
[2025-04-25 14:19:49,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.78 | bwd: 5712.39 | bwd_inner: 5654.42 | bwd_allreduce: 57.92 | step: 18.87

  6%|▋         | 2640/41250 [6:22:14<93:00:10,  8.67s/it]
                                                         
{'loss': 0.2945, 'grad_norm': 2.069246530532837, 'learning_rate': 3.987894669281514e-05, 'epoch': 0.64}

  6%|▋         | 2640/41250 [6:22:14<93:00:10,  8.67s/it][2025-04-25 14:19:58,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 14:19:58,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5709.03 | bwd_inner_microstep: 5696.20 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.04
[2025-04-25 14:19:58,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.65 | bwd: 5709.04 | bwd_inner: 5696.20 | bwd_allreduce: 12.80 | step: 19.04

  6%|▋         | 2641/41250 [6:22:23<92:55:10,  8.66s/it]
                                                         
{'loss': 0.1922, 'grad_norm': 0.8116616010665894, 'learning_rate': 3.9878774119353194e-05, 'epoch': 0.64}

  6%|▋         | 2641/41250 [6:22:23<92:55:10,  8.66s/it][2025-04-25 14:20:06,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 14:20:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.32 | bwd_microstep: 5773.10 | bwd_inner_microstep: 5663.16 | bwd_allreduce_microstep: 109.89 | step_microstep: 19.11
[2025-04-25 14:20:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.32 | bwd: 5773.12 | bwd_inner: 5663.16 | bwd_allreduce: 109.91 | step: 19.11

  6%|▋         | 2642/41250 [6:22:32<92:58:42,  8.67s/it]
                                                         
{'loss': 0.1824, 'grad_norm': 2.5713729858398438, 'learning_rate': 3.987860142334252e-05, 'epoch': 0.64}

  6%|▋         | 2642/41250 [6:22:32<92:58:42,  8.67s/it][2025-04-25 14:20:15,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 14:20:15,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.43 | bwd_microstep: 5740.61 | bwd_inner_microstep: 5710.57 | bwd_allreduce_microstep: 29.98 | step_microstep: 19.08
[2025-04-25 14:20:15,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.43 | bwd: 5740.63 | bwd_inner: 5710.57 | bwd_allreduce: 30.00 | step: 19.08

  6%|▋         | 2643/41250 [6:22:40<93:00:19,  8.67s/it]
                                                         
{'loss': 0.1079, 'grad_norm': 0.746266782283783, 'learning_rate': 3.987842860478417e-05, 'epoch': 0.64}

  6%|▋         | 2643/41250 [6:22:40<93:00:19,  8.67s/it][2025-04-25 14:20:24,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:20:24,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.62 | bwd_microstep: 5761.98 | bwd_inner_microstep: 5657.76 | bwd_allreduce_microstep: 104.18 | step_microstep: 18.28
[2025-04-25 14:20:24,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5761.99 | bwd_inner: 5657.76 | bwd_allreduce: 104.19 | step: 18.28

  6%|▋         | 2644/41250 [6:22:49<93:01:11,  8.67s/it]
                                                         
{'loss': 0.1916, 'grad_norm': 1.7519505023956299, 'learning_rate': 3.9878255663679226e-05, 'epoch': 0.64}

  6%|▋         | 2644/41250 [6:22:49<93:01:11,  8.67s/it][2025-04-25 14:20:32,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:20:32,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5698.01 | bwd_inner_microstep: 5685.55 | bwd_allreduce_microstep: 12.42 | step_microstep: 18.36
[2025-04-25 14:20:32,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5698.02 | bwd_inner: 5685.55 | bwd_allreduce: 12.43 | step: 18.37

  6%|▋         | 2645/41250 [6:22:57<92:52:36,  8.66s/it]
                                                         
{'loss': 0.2024, 'grad_norm': 1.9712709188461304, 'learning_rate': 3.987808260002874e-05, 'epoch': 0.64}

  6%|▋         | 2645/41250 [6:22:57<92:52:36,  8.66s/it][2025-04-25 14:20:41,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:20:41,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.50 | bwd_microstep: 5695.56 | bwd_inner_microstep: 5658.94 | bwd_allreduce_microstep: 36.56 | step_microstep: 18.42
[2025-04-25 14:20:41,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.50 | bwd: 5695.58 | bwd_inner: 5658.94 | bwd_allreduce: 36.58 | step: 18.43

  6%|▋         | 2646/41250 [6:23:06<92:44:37,  8.65s/it]
                                                         
{'loss': 0.0889, 'grad_norm': 0.842947781085968, 'learning_rate': 3.9877909413833794e-05, 'epoch': 0.64}

  6%|▋         | 2646/41250 [6:23:06<92:44:37,  8.65s/it][2025-04-25 14:20:50,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 14:20:50,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.73 | bwd_microstep: 5782.52 | bwd_inner_microstep: 5769.88 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.92
[2025-04-25 14:20:50,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.73 | bwd: 5782.54 | bwd_inner: 5769.88 | bwd_allreduce: 12.61 | step: 18.93

  6%|▋         | 2647/41250 [6:23:15<93:04:42,  8.68s/it]
                                                         
{'loss': 0.3792, 'grad_norm': 2.108879327774048, 'learning_rate': 3.9877736105095435e-05, 'epoch': 0.64}

  6%|▋         | 2647/41250 [6:23:15<93:04:42,  8.68s/it][2025-04-25 14:20:58,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:20:58,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.65 | bwd_microstep: 5721.71 | bwd_inner_microstep: 5697.51 | bwd_allreduce_microstep: 24.15 | step_microstep: 18.78
[2025-04-25 14:20:58,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.65 | bwd: 5721.73 | bwd_inner: 5697.51 | bwd_allreduce: 24.17 | step: 18.78

  6%|▋         | 2648/41250 [6:23:24<93:00:26,  8.67s/it]
                                                         
{'loss': 0.1075, 'grad_norm': 0.9025543332099915, 'learning_rate': 3.9877562673814745e-05, 'epoch': 0.64}

  6%|▋         | 2648/41250 [6:23:24<93:00:26,  8.67s/it][2025-04-25 14:21:07,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:21:07,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.95 | bwd_microstep: 5692.43 | bwd_inner_microstep: 5653.67 | bwd_allreduce_microstep: 38.72 | step_microstep: 17.99
[2025-04-25 14:21:07,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.95 | bwd: 5692.44 | bwd_inner: 5653.67 | bwd_allreduce: 38.74 | step: 17.99

  6%|▋         | 2649/41250 [6:23:32<92:48:35,  8.66s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 1.2338746786117554, 'learning_rate': 3.987738911999279e-05, 'epoch': 0.64}

  6%|▋         | 2649/41250 [6:23:32<92:48:35,  8.66s/it][2025-04-25 14:21:16,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:21:16,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.68 | bwd_microstep: 5877.36 | bwd_inner_microstep: 5650.99 | bwd_allreduce_microstep: 226.33 | step_microstep: 18.56
[2025-04-25 14:21:16,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.68 | bwd: 5877.38 | bwd_inner: 5650.99 | bwd_allreduce: 226.35 | step: 18.56

  6%|▋         | 2650/41250 [6:23:41<93:13:51,  8.70s/it]
                                                         
{'loss': 0.176, 'grad_norm': 1.5720934867858887, 'learning_rate': 3.9877215443630646e-05, 'epoch': 0.64}

  6%|▋         | 2650/41250 [6:23:41<93:13:51,  8.70s/it][2025-04-25 14:21:24,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 14:21:24,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.30 | bwd_microstep: 5714.57 | bwd_inner_microstep: 5685.41 | bwd_allreduce_microstep: 29.11 | step_microstep: 18.86
[2025-04-25 14:21:24,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.30 | bwd: 5714.58 | bwd_inner: 5685.41 | bwd_allreduce: 29.13 | step: 18.86

  6%|▋         | 2651/41250 [6:23:50<93:04:55,  8.68s/it]
                                                         
{'loss': 0.2787, 'grad_norm': 1.9183120727539062, 'learning_rate': 3.987704164472938e-05, 'epoch': 0.64}

  6%|▋         | 2651/41250 [6:23:50<93:04:55,  8.68s/it][2025-04-25 14:21:33,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:21:33,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5715.05 | bwd_inner_microstep: 5702.04 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.47
[2025-04-25 14:21:33,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5715.06 | bwd_inner: 5702.04 | bwd_allreduce: 12.98 | step: 18.47

  6%|▋         | 2652/41250 [6:23:58<92:59:08,  8.67s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 1.0205150842666626, 'learning_rate': 3.9876867723290054e-05, 'epoch': 0.64}

  6%|▋         | 2652/41250 [6:23:58<92:59:08,  8.67s/it][2025-04-25 14:21:41,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 14:21:41,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.72 | bwd_microstep: 5659.07 | bwd_inner_microstep: 5642.78 | bwd_allreduce_microstep: 16.25 | step_microstep: 17.73
[2025-04-25 14:21:41,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.72 | bwd: 5659.08 | bwd_inner: 5642.78 | bwd_allreduce: 16.27 | step: 17.73

  6%|▋         | 2653/41250 [6:24:07<92:41:39,  8.65s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 1.5062096118927002, 'learning_rate': 3.9876693679313754e-05, 'epoch': 0.64}

  6%|▋         | 2653/41250 [6:24:07<92:41:39,  8.65s/it][2025-04-25 14:21:50,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 14:21:50,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.87 | bwd_microstep: 5785.38 | bwd_inner_microstep: 5772.75 | bwd_allreduce_microstep: 12.58 | step_microstep: 17.98
[2025-04-25 14:21:50,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.87 | bwd: 5785.39 | bwd_inner: 5772.75 | bwd_allreduce: 12.60 | step: 17.98

  6%|▋         | 2654/41250 [6:24:16<93:02:08,  8.68s/it]
                                                         
{'loss': 0.1249, 'grad_norm': 0.7584514617919922, 'learning_rate': 3.987651951280154e-05, 'epoch': 0.64}

  6%|▋         | 2654/41250 [6:24:16<93:02:08,  8.68s/it][2025-04-25 14:21:59,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:21:59,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.18 | bwd_microstep: 5731.41 | bwd_inner_microstep: 5691.80 | bwd_allreduce_microstep: 39.57 | step_microstep: 18.23
[2025-04-25 14:21:59,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.18 | bwd: 5731.42 | bwd_inner: 5691.80 | bwd_allreduce: 39.59 | step: 18.23

  6%|▋         | 2655/41250 [6:24:24<92:59:42,  8.67s/it]
                                                         
{'loss': 0.1042, 'grad_norm': 0.7845240235328674, 'learning_rate': 3.98763452237545e-05, 'epoch': 0.64}

  6%|▋         | 2655/41250 [6:24:24<92:59:42,  8.67s/it][2025-04-25 14:22:08,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:22:08,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.09 | bwd_microstep: 5715.59 | bwd_inner_microstep: 5702.68 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.22
[2025-04-25 14:22:08,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.09 | bwd: 5715.60 | bwd_inner: 5702.68 | bwd_allreduce: 12.88 | step: 18.22

  6%|▋         | 2656/41250 [6:24:33<92:56:51,  8.67s/it]
                                                         
{'loss': 0.2608, 'grad_norm': 2.2864794731140137, 'learning_rate': 3.98761708121737e-05, 'epoch': 0.64}

  6%|▋         | 2656/41250 [6:24:33<92:56:51,  8.67s/it][2025-04-25 14:22:16,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 14:22:16,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.68 | bwd_microstep: 5691.62 | bwd_inner_microstep: 5678.62 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.73
[2025-04-25 14:22:16,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.68 | bwd: 5691.63 | bwd_inner: 5678.62 | bwd_allreduce: 12.96 | step: 18.73

  6%|▋         | 2657/41250 [6:24:42<92:48:28,  8.66s/it]
                                                         
{'loss': 0.1628, 'grad_norm': 0.8544734716415405, 'learning_rate': 3.987599627806021e-05, 'epoch': 0.64}

  6%|▋         | 2657/41250 [6:24:42<92:48:28,  8.66s/it][2025-04-25 14:22:25,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:22:25,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.60 | bwd_microstep: 5680.25 | bwd_inner_microstep: 5650.98 | bwd_allreduce_microstep: 29.23 | step_microstep: 18.32
[2025-04-25 14:22:25,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.60 | bwd: 5680.27 | bwd_inner: 5650.98 | bwd_allreduce: 29.25 | step: 18.33

  6%|▋         | 2658/41250 [6:24:50<92:35:09,  8.64s/it]
                                                         
{'loss': 0.1998, 'grad_norm': 1.3847135305404663, 'learning_rate': 3.987582162141512e-05, 'epoch': 0.64}

  6%|▋         | 2658/41250 [6:24:50<92:35:09,  8.64s/it][2025-04-25 14:22:33,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 14:22:33,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.53 | bwd_microstep: 5711.91 | bwd_inner_microstep: 5699.36 | bwd_allreduce_microstep: 12.51 | step_microstep: 19.03
[2025-04-25 14:22:33,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.53 | bwd: 5711.92 | bwd_inner: 5699.36 | bwd_allreduce: 12.53 | step: 19.03

  6%|▋         | 2659/41250 [6:24:59<92:37:42,  8.64s/it]
                                                         
{'loss': 0.3684, 'grad_norm': 1.6503543853759766, 'learning_rate': 3.98756468422395e-05, 'epoch': 0.64}

  6%|▋         | 2659/41250 [6:24:59<92:37:42,  8.64s/it][2025-04-25 14:22:42,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:22:42,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5677.86 | bwd_inner_microstep: 5643.11 | bwd_allreduce_microstep: 34.70 | step_microstep: 18.90
[2025-04-25 14:22:42,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5677.87 | bwd_inner: 5643.11 | bwd_allreduce: 34.72 | step: 18.90

  6%|▋         | 2660/41250 [6:25:07<92:27:26,  8.63s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 1.1956664323806763, 'learning_rate': 3.987547194053442e-05, 'epoch': 0.64}

  6%|▋         | 2660/41250 [6:25:07<92:27:26,  8.63s/it][2025-04-25 14:22:51,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:22:51,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.95 | bwd_microstep: 5742.41 | bwd_inner_microstep: 5641.47 | bwd_allreduce_microstep: 100.90 | step_microstep: 17.79
[2025-04-25 14:22:51,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.95 | bwd: 5742.43 | bwd_inner: 5641.47 | bwd_allreduce: 100.91 | step: 17.80

  6%|▋         | 2661/41250 [6:25:16<92:31:52,  8.63s/it]
                                                         
{'loss': 0.1883, 'grad_norm': 1.6944693326950073, 'learning_rate': 3.987529691630097e-05, 'epoch': 0.65}

  6%|▋         | 2661/41250 [6:25:16<92:31:52,  8.63s/it][2025-04-25 14:22:59,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:22:59,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.33 | bwd_microstep: 5675.43 | bwd_inner_microstep: 5634.14 | bwd_allreduce_microstep: 41.25 | step_microstep: 18.72
[2025-04-25 14:22:59,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.33 | bwd: 5675.44 | bwd_inner: 5634.14 | bwd_allreduce: 41.27 | step: 18.72

  6%|▋         | 2662/41250 [6:25:25<92:20:51,  8.62s/it]
                                                         
{'loss': 0.1958, 'grad_norm': 1.793348789215088, 'learning_rate': 3.987512176954022e-05, 'epoch': 0.65}

  6%|▋         | 2662/41250 [6:25:25<92:20:51,  8.62s/it][2025-04-25 14:23:08,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:23:08,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.62 | bwd_microstep: 5660.55 | bwd_inner_microstep: 5647.63 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.28
[2025-04-25 14:23:08,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.62 | bwd: 5660.56 | bwd_inner: 5647.63 | bwd_allreduce: 12.89 | step: 18.28

  6%|▋         | 2663/41250 [6:25:33<92:12:40,  8.60s/it]
                                                         
{'loss': 0.3147, 'grad_norm': 1.3791714906692505, 'learning_rate': 3.987494650025326e-05, 'epoch': 0.65}

  6%|▋         | 2663/41250 [6:25:33<92:12:40,  8.60s/it][2025-04-25 14:23:16,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 14:23:16,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.30 | bwd_microstep: 5692.27 | bwd_inner_microstep: 5679.63 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.30
[2025-04-25 14:23:16,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.30 | bwd: 5692.28 | bwd_inner: 5679.63 | bwd_allreduce: 12.61 | step: 18.31

  6%|▋         | 2664/41250 [6:25:42<92:15:19,  8.61s/it]
                                                         
{'loss': 0.0785, 'grad_norm': 0.6329085230827332, 'learning_rate': 3.9874771108441156e-05, 'epoch': 0.65}

  6%|▋         | 2664/41250 [6:25:42<92:15:19,  8.61s/it][2025-04-25 14:23:25,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.85 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:23:25,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.23 | bwd_microstep: 5688.42 | bwd_inner_microstep: 5642.97 | bwd_allreduce_microstep: 45.40 | step_microstep: 17.65
[2025-04-25 14:23:25,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.23 | bwd: 5688.43 | bwd_inner: 5642.97 | bwd_allreduce: 45.42 | step: 17.66

  6%|▋         | 2665/41250 [6:25:50<92:12:10,  8.60s/it]
                                                         
{'loss': 0.2344, 'grad_norm': 2.208930015563965, 'learning_rate': 3.9874595594105e-05, 'epoch': 0.65}

  6%|▋         | 2665/41250 [6:25:50<92:12:10,  8.60s/it][2025-04-25 14:23:34,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 14:23:34,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5691.40 | bwd_inner_microstep: 5678.84 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.38
[2025-04-25 14:23:34,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5691.41 | bwd_inner: 5678.84 | bwd_allreduce: 12.53 | step: 18.38

  6%|▋         | 2666/41250 [6:25:59<92:14:55,  8.61s/it]
                                                         
{'loss': 0.2793, 'grad_norm': 1.4213746786117554, 'learning_rate': 3.987441995724587e-05, 'epoch': 0.65}

  6%|▋         | 2666/41250 [6:25:59<92:14:55,  8.61s/it][2025-04-25 14:23:42,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:23:42,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.17 | bwd_microstep: 5693.55 | bwd_inner_microstep: 5680.84 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.27
[2025-04-25 14:23:42,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.17 | bwd: 5693.57 | bwd_inner: 5680.84 | bwd_allreduce: 12.69 | step: 18.27

  6%|▋         | 2667/41250 [6:26:08<92:17:30,  8.61s/it]
                                                         
{'loss': 0.1153, 'grad_norm': 0.8359075784683228, 'learning_rate': 3.9874244197864856e-05, 'epoch': 0.65}

  6%|▋         | 2667/41250 [6:26:08<92:17:30,  8.61s/it][2025-04-25 14:23:51,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:23:51,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.16 | bwd_microstep: 5744.88 | bwd_inner_microstep: 5645.64 | bwd_allreduce_microstep: 99.19 | step_microstep: 18.31
[2025-04-25 14:23:51,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.16 | bwd: 5744.89 | bwd_inner: 5645.64 | bwd_allreduce: 99.21 | step: 18.31

  6%|▋         | 2668/41250 [6:26:16<92:25:14,  8.62s/it]
                                                         
{'loss': 0.13, 'grad_norm': 0.7193832397460938, 'learning_rate': 3.987406831596303e-05, 'epoch': 0.65}

  6%|▋         | 2668/41250 [6:26:16<92:25:14,  8.62s/it][2025-04-25 14:24:00,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:24:00,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.75 | bwd_microstep: 5701.21 | bwd_inner_microstep: 5652.02 | bwd_allreduce_microstep: 49.14 | step_microstep: 18.37
[2025-04-25 14:24:00,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.75 | bwd: 5701.23 | bwd_inner: 5652.03 | bwd_allreduce: 49.16 | step: 18.37

  6%|▋         | 2669/41250 [6:26:25<92:21:49,  8.62s/it]
                                                         
{'loss': 0.2076, 'grad_norm': 1.096750259399414, 'learning_rate': 3.9873892311541485e-05, 'epoch': 0.65}

  6%|▋         | 2669/41250 [6:26:25<92:21:49,  8.62s/it][2025-04-25 14:24:08,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 14:24:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.76 | bwd_microstep: 5715.42 | bwd_inner_microstep: 5702.84 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.88
[2025-04-25 14:24:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.76 | bwd: 5715.43 | bwd_inner: 5702.84 | bwd_allreduce: 12.55 | step: 18.88

  6%|▋         | 2670/41250 [6:26:34<92:31:29,  8.63s/it]
                                                         
{'loss': 0.1112, 'grad_norm': 0.93131023645401, 'learning_rate': 3.98737161846013e-05, 'epoch': 0.65}

  6%|▋         | 2670/41250 [6:26:34<92:31:29,  8.63s/it][2025-04-25 14:24:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:24:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5757.44 | bwd_inner_microstep: 5654.29 | bwd_allreduce_microstep: 103.11 | step_microstep: 18.76
[2025-04-25 14:24:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5757.45 | bwd_inner: 5654.29 | bwd_allreduce: 103.13 | step: 18.76

  6%|▋         | 2671/41250 [6:26:42<92:37:57,  8.64s/it]
                                                         
{'loss': 0.1724, 'grad_norm': 1.1257293224334717, 'learning_rate': 3.987353993514356e-05, 'epoch': 0.65}

  6%|▋         | 2671/41250 [6:26:42<92:37:57,  8.64s/it][2025-04-25 14:24:25,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 14:24:25,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.03 | bwd_microstep: 5685.65 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 37.15 | step_microstep: 18.38
[2025-04-25 14:24:25,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.03 | bwd: 5685.66 | bwd_inner: 5648.45 | bwd_allreduce: 37.16 | step: 18.38

  6%|▋         | 2672/41250 [6:26:51<92:28:58,  8.63s/it]
                                                         
{'loss': 0.2384, 'grad_norm': 1.2790807485580444, 'learning_rate': 3.9873363563169366e-05, 'epoch': 0.65}

  6%|▋         | 2672/41250 [6:26:51<92:28:58,  8.63s/it][2025-04-25 14:24:34,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:24:34,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5739.55 | bwd_inner_microstep: 5648.64 | bwd_allreduce_microstep: 90.86 | step_microstep: 18.68
[2025-04-25 14:24:34,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5739.56 | bwd_inner: 5648.64 | bwd_allreduce: 90.88 | step: 18.68

  6%|▋         | 2673/41250 [6:26:59<92:34:27,  8.64s/it]
                                                         
{'loss': 0.0698, 'grad_norm': 0.7578369379043579, 'learning_rate': 3.987318706867979e-05, 'epoch': 0.65}

  6%|▋         | 2673/41250 [6:26:59<92:34:27,  8.64s/it][2025-04-25 14:24:43,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:24:43,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.08 | bwd_microstep: 5719.49 | bwd_inner_microstep: 5706.87 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.50
[2025-04-25 14:24:43,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.08 | bwd: 5719.50 | bwd_inner: 5706.87 | bwd_allreduce: 12.60 | step: 18.51

  6%|▋         | 2674/41250 [6:27:08<92:39:50,  8.65s/it]
                                                         
{'loss': 0.1215, 'grad_norm': 1.3566738367080688, 'learning_rate': 3.9873010451675925e-05, 'epoch': 0.65}

  6%|▋         | 2674/41250 [6:27:08<92:39:50,  8.65s/it][2025-04-25 14:24:51,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 14:24:51,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.95 | bwd_microstep: 5729.74 | bwd_inner_microstep: 5717.04 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.12
[2025-04-25 14:24:51,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.95 | bwd: 5729.75 | bwd_inner: 5717.04 | bwd_allreduce: 12.67 | step: 19.12

  6%|▋         | 2675/41250 [6:27:17<92:46:54,  8.66s/it]
                                                         
{'loss': 0.2116, 'grad_norm': 1.4139442443847656, 'learning_rate': 3.987283371215885e-05, 'epoch': 0.65}

  6%|▋         | 2675/41250 [6:27:17<92:46:54,  8.66s/it][2025-04-25 14:25:00,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 14:25:00,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.24 | bwd_microstep: 5733.16 | bwd_inner_microstep: 5700.93 | bwd_allreduce_microstep: 32.18 | step_microstep: 19.03
[2025-04-25 14:25:00,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.24 | bwd: 5733.17 | bwd_inner: 5700.93 | bwd_allreduce: 32.20 | step: 19.03

  6%|▋         | 2676/41250 [6:27:25<92:49:20,  8.66s/it]
                                                         
{'loss': 0.4178, 'grad_norm': 1.9783085584640503, 'learning_rate': 3.9872656850129674e-05, 'epoch': 0.65}

  6%|▋         | 2676/41250 [6:27:25<92:49:20,  8.66s/it][2025-04-25 14:25:09,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:25:09,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.15 | bwd_microstep: 5713.21 | bwd_inner_microstep: 5700.68 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.37
[2025-04-25 14:25:09,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.15 | bwd: 5713.22 | bwd_inner: 5700.68 | bwd_allreduce: 12.50 | step: 18.37

  6%|▋         | 2677/41250 [6:27:34<92:47:10,  8.66s/it]
                                                         
{'loss': 0.2646, 'grad_norm': 1.3259174823760986, 'learning_rate': 3.987247986558947e-05, 'epoch': 0.65}

  6%|▋         | 2677/41250 [6:27:34<92:47:10,  8.66s/it][2025-04-25 14:25:17,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:25:17,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.13 | bwd_microstep: 5771.00 | bwd_inner_microstep: 5663.88 | bwd_allreduce_microstep: 107.07 | step_microstep: 18.55
[2025-04-25 14:25:17,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.13 | bwd: 5771.01 | bwd_inner: 5663.88 | bwd_allreduce: 107.09 | step: 18.55

  6%|▋         | 2678/41250 [6:27:43<92:53:37,  8.67s/it]
                                                         
{'loss': 0.1096, 'grad_norm': 1.2753136157989502, 'learning_rate': 3.987230275853934e-05, 'epoch': 0.65}

  6%|▋         | 2678/41250 [6:27:43<92:53:37,  8.67s/it][2025-04-25 14:25:26,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 14:25:26,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.30 | bwd_microstep: 5704.64 | bwd_inner_microstep: 5660.00 | bwd_allreduce_microstep: 44.60 | step_microstep: 18.51
[2025-04-25 14:25:26,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.30 | bwd: 5704.66 | bwd_inner: 5660.00 | bwd_allreduce: 44.61 | step: 18.52

  6%|▋         | 2679/41250 [6:27:51<92:44:44,  8.66s/it]
                                                         
{'loss': 0.1547, 'grad_norm': 0.9297436475753784, 'learning_rate': 3.987212552898038e-05, 'epoch': 0.65}

  6%|▋         | 2679/41250 [6:27:51<92:44:44,  8.66s/it][2025-04-25 14:25:35,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 14:25:35,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.03 | bwd_microstep: 5726.01 | bwd_inner_microstep: 5713.29 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.74
[2025-04-25 14:25:35,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.03 | bwd: 5726.02 | bwd_inner: 5713.29 | bwd_allreduce: 12.70 | step: 18.75

  6%|▋         | 2680/41250 [6:28:00<92:46:45,  8.66s/it]
                                                         
{'loss': 0.212, 'grad_norm': 1.6250441074371338, 'learning_rate': 3.9871948176913664e-05, 'epoch': 0.65}

  6%|▋         | 2680/41250 [6:28:00<92:46:45,  8.66s/it][2025-04-25 14:25:43,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:25:43,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5738.57 | bwd_inner_microstep: 5696.64 | bwd_allreduce_microstep: 41.89 | step_microstep: 18.35
[2025-04-25 14:25:43,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5738.58 | bwd_inner: 5696.64 | bwd_allreduce: 41.90 | step: 18.36

  6%|▋         | 2681/41250 [6:28:09<92:50:22,  8.67s/it]
                                                         
{'loss': 0.0438, 'grad_norm': 0.46169036626815796, 'learning_rate': 3.9871770702340304e-05, 'epoch': 0.65}

  6%|▋         | 2681/41250 [6:28:09<92:50:22,  8.67s/it][2025-04-25 14:25:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:25:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.24 | bwd_microstep: 5706.61 | bwd_inner_microstep: 5694.11 | bwd_allreduce_microstep: 12.47 | step_microstep: 17.99
[2025-04-25 14:25:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.24 | bwd: 5706.63 | bwd_inner: 5694.11 | bwd_allreduce: 12.48 | step: 18.00

  7%|▋         | 2682/41250 [6:28:17<92:46:19,  8.66s/it]
                                                         
{'loss': 0.1569, 'grad_norm': 1.4010146856307983, 'learning_rate': 3.9871593105261374e-05, 'epoch': 0.65}

  7%|▋         | 2682/41250 [6:28:17<92:46:19,  8.66s/it][2025-04-25 14:26:01,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 14:26:01,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.03 | bwd_microstep: 5761.91 | bwd_inner_microstep: 5685.42 | bwd_allreduce_microstep: 76.44 | step_microstep: 18.84
[2025-04-25 14:26:01,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.03 | bwd: 5761.93 | bwd_inner: 5685.42 | bwd_allreduce: 76.46 | step: 18.84

  7%|▋         | 2683/41250 [6:28:26<92:53:53,  8.67s/it]
                                                         
{'loss': 0.1893, 'grad_norm': 0.9513264298439026, 'learning_rate': 3.9871415385677994e-05, 'epoch': 0.65}

  7%|▋         | 2683/41250 [6:28:26<92:53:53,  8.67s/it][2025-04-25 14:26:09,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.08 | optimizer_step: 0.99
[2025-04-25 14:26:09,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.24 | bwd_microstep: 5704.21 | bwd_inner_microstep: 5652.76 | bwd_allreduce_microstep: 51.39 | step_microstep: 19.35
[2025-04-25 14:26:09,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.24 | bwd: 5704.22 | bwd_inner: 5652.76 | bwd_allreduce: 51.42 | step: 19.35

  7%|▋         | 2684/41250 [6:28:35<92:45:11,  8.66s/it]
                                                         
{'loss': 0.3784, 'grad_norm': 1.767198085784912, 'learning_rate': 3.987123754359124e-05, 'epoch': 0.65}

  7%|▋         | 2684/41250 [6:28:35<92:45:11,  8.66s/it][2025-04-25 14:26:18,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:26:18,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.89 | bwd_microstep: 5715.96 | bwd_inner_microstep: 5703.19 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.42
[2025-04-25 14:26:18,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5715.97 | bwd_inner: 5703.19 | bwd_allreduce: 12.75 | step: 18.42

  7%|▋         | 2685/41250 [6:28:43<92:43:29,  8.66s/it]
                                                         
{'loss': 0.2576, 'grad_norm': 1.7164008617401123, 'learning_rate': 3.9871059579002225e-05, 'epoch': 0.65}

  7%|▋         | 2685/41250 [6:28:43<92:43:29,  8.66s/it][2025-04-25 14:26:27,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:26:27,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.99 | bwd_microstep: 5774.57 | bwd_inner_microstep: 5658.28 | bwd_allreduce_microstep: 116.24 | step_microstep: 18.97
[2025-04-25 14:26:27,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.99 | bwd: 5774.58 | bwd_inner: 5658.28 | bwd_allreduce: 116.26 | step: 18.97

  7%|▋         | 2686/41250 [6:28:52<92:50:31,  8.67s/it]
                                                         
{'loss': 0.3114, 'grad_norm': 1.425450325012207, 'learning_rate': 3.987088149191202e-05, 'epoch': 0.65}

  7%|▋         | 2686/41250 [6:28:52<92:50:31,  8.67s/it][2025-04-25 14:26:35,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:26:35,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5759.16 | bwd_inner_microstep: 5673.10 | bwd_allreduce_microstep: 86.01 | step_microstep: 18.81
[2025-04-25 14:26:35,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5759.17 | bwd_inner: 5673.09 | bwd_allreduce: 86.03 | step: 18.81

  7%|▋         | 2687/41250 [6:29:01<92:52:36,  8.67s/it]
                                                         
{'loss': 0.3226, 'grad_norm': 3.2235403060913086, 'learning_rate': 3.987070328232175e-05, 'epoch': 0.65}

  7%|▋         | 2687/41250 [6:29:01<92:52:36,  8.67s/it][2025-04-25 14:26:44,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-25 14:26:44,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5880.89 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 179.69 | step_microstep: 19.35
[2025-04-25 14:26:44,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.48 | bwd: 5880.90 | bwd_inner: 5701.15 | bwd_allreduce: 179.72 | step: 19.35

  7%|▋         | 2688/41250 [6:29:10<93:19:37,  8.71s/it]
                                                         
{'loss': 0.328, 'grad_norm': 2.678544521331787, 'learning_rate': 3.98705249502325e-05, 'epoch': 0.65}

  7%|▋         | 2688/41250 [6:29:10<93:19:37,  8.71s/it][2025-04-25 14:26:53,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 14:26:53,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5715.94 | bwd_inner_microstep: 5702.38 | bwd_allreduce_microstep: 13.51 | step_microstep: 19.09
[2025-04-25 14:26:53,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5715.95 | bwd_inner: 5702.38 | bwd_allreduce: 13.53 | step: 19.09

  7%|▋         | 2689/41250 [6:29:18<93:07:19,  8.69s/it]
                                                         
{'loss': 0.2901, 'grad_norm': 1.7218443155288696, 'learning_rate': 3.987034649564537e-05, 'epoch': 0.65}

  7%|▋         | 2689/41250 [6:29:18<93:07:19,  8.69s/it][2025-04-25 14:27:02,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.84 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:27:02,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.92 | bwd_microstep: 5760.73 | bwd_inner_microstep: 5712.17 | bwd_allreduce_microstep: 48.51 | step_microstep: 17.59
[2025-04-25 14:27:02,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.92 | bwd: 5760.74 | bwd_inner: 5712.17 | bwd_allreduce: 48.53 | step: 17.60

  7%|▋         | 2690/41250 [6:29:27<93:07:45,  8.69s/it]
                                                         
{'loss': 0.4702, 'grad_norm': 2.1158952713012695, 'learning_rate': 3.9870167918561464e-05, 'epoch': 0.65}

  7%|▋         | 2690/41250 [6:29:27<93:07:45,  8.69s/it][2025-04-25 14:27:10,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:27:10,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.26 | bwd_microstep: 5761.10 | bwd_inner_microstep: 5712.12 | bwd_allreduce_microstep: 48.93 | step_microstep: 18.80
[2025-04-25 14:27:10,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.26 | bwd: 5761.12 | bwd_inner: 5712.12 | bwd_allreduce: 48.96 | step: 18.81

  7%|▋         | 2691/41250 [6:29:36<93:12:09,  8.70s/it]
                                                         
{'loss': 0.4242, 'grad_norm': 1.672475814819336, 'learning_rate': 3.986998921898188e-05, 'epoch': 0.65}

  7%|▋         | 2691/41250 [6:29:36<93:12:09,  8.70s/it][2025-04-25 14:27:19,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 14:27:19,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.66 | bwd_microstep: 5767.30 | bwd_inner_microstep: 5711.36 | bwd_allreduce_microstep: 55.89 | step_microstep: 18.72
[2025-04-25 14:27:19,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.66 | bwd: 5767.31 | bwd_inner: 5711.36 | bwd_allreduce: 55.91 | step: 18.72

  7%|▋         | 2692/41250 [6:29:44<93:11:56,  8.70s/it]
                                                         
{'loss': 0.0386, 'grad_norm': 0.30198532342910767, 'learning_rate': 3.986981039690772e-05, 'epoch': 0.65}

  7%|▋         | 2692/41250 [6:29:44<93:11:56,  8.70s/it][2025-04-25 14:27:28,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 14:27:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.97 | bwd_microstep: 5725.51 | bwd_inner_microstep: 5701.12 | bwd_allreduce_microstep: 24.34 | step_microstep: 18.13
[2025-04-25 14:27:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.97 | bwd: 5725.52 | bwd_inner: 5701.12 | bwd_allreduce: 24.36 | step: 18.13

  7%|▋         | 2693/41250 [6:29:53<93:04:25,  8.69s/it]
                                                         
{'loss': 0.185, 'grad_norm': 2.6360363960266113, 'learning_rate': 3.986963145234009e-05, 'epoch': 0.65}

  7%|▋         | 2693/41250 [6:29:53<93:04:25,  8.69s/it][2025-04-25 14:27:36,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:27:36,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.30 | bwd_microstep: 5672.82 | bwd_inner_microstep: 5660.08 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.45
[2025-04-25 14:27:36,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.30 | bwd: 5672.83 | bwd_inner: 5660.08 | bwd_allreduce: 12.71 | step: 18.45

  7%|▋         | 2694/41250 [6:30:02<92:48:40,  8.67s/it]
                                                         
{'loss': 0.2034, 'grad_norm': 1.320900321006775, 'learning_rate': 3.986945238528009e-05, 'epoch': 0.65}

  7%|▋         | 2694/41250 [6:30:02<92:48:40,  8.67s/it][2025-04-25 14:27:45,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:27:45,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5703.20 | bwd_inner_microstep: 5690.42 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.30
[2025-04-25 14:27:45,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5703.21 | bwd_inner: 5690.42 | bwd_allreduce: 12.75 | step: 18.30

  7%|▋         | 2695/41250 [6:30:10<92:44:34,  8.66s/it]
                                                         
{'loss': 0.2505, 'grad_norm': 2.072429895401001, 'learning_rate': 3.986927319572882e-05, 'epoch': 0.65}

  7%|▋         | 2695/41250 [6:30:10<92:44:34,  8.66s/it][2025-04-25 14:27:54,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:27:54,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.92 | bwd_microstep: 5712.01 | bwd_inner_microstep: 5699.32 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.42
[2025-04-25 14:27:54,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.92 | bwd: 5712.02 | bwd_inner: 5699.32 | bwd_allreduce: 12.66 | step: 18.42

  7%|▋         | 2696/41250 [6:30:19<92:43:24,  8.66s/it]
                                                         
{'loss': 0.151, 'grad_norm': 1.2004468441009521, 'learning_rate': 3.986909388368739e-05, 'epoch': 0.65}

  7%|▋         | 2696/41250 [6:30:19<92:43:24,  8.66s/it][2025-04-25 14:28:02,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 14:28:02,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.64 | bwd_microstep: 5719.58 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 62.07 | step_microstep: 19.45
[2025-04-25 14:28:02,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.64 | bwd: 5719.60 | bwd_inner: 5657.45 | bwd_allreduce: 62.10 | step: 19.45

  7%|▋         | 2697/41250 [6:30:28<92:39:15,  8.65s/it]
                                                         
{'loss': 0.1119, 'grad_norm': 0.7072372436523438, 'learning_rate': 3.986891444915691e-05, 'epoch': 0.65}

  7%|▋         | 2697/41250 [6:30:28<92:39:15,  8.65s/it][2025-04-25 14:28:11,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:28:11,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.96 | bwd_microstep: 5782.26 | bwd_inner_microstep: 5769.56 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.46
[2025-04-25 14:28:11,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.96 | bwd: 5782.28 | bwd_inner: 5769.56 | bwd_allreduce: 12.68 | step: 18.46

  7%|▋         | 2698/41250 [6:30:36<92:57:59,  8.68s/it]
                                                         
{'loss': 0.1857, 'grad_norm': 1.0170522928237915, 'learning_rate': 3.9868734892138475e-05, 'epoch': 0.65}

  7%|▋         | 2698/41250 [6:30:36<92:57:59,  8.68s/it][2025-04-25 14:28:20,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:28:20,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5712.06 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.52
[2025-04-25 14:28:20,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5712.07 | bwd_inner: 5699.18 | bwd_allreduce: 12.85 | step: 18.53

  7%|▋         | 2699/41250 [6:30:45<92:50:43,  8.67s/it]
                                                         
{'loss': 0.1041, 'grad_norm': 0.9274555444717407, 'learning_rate': 3.9868555212633205e-05, 'epoch': 0.65}

  7%|▋         | 2699/41250 [6:30:45<92:50:43,  8.67s/it][2025-04-25 14:28:28,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:28:28,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5745.94 | bwd_inner_microstep: 5697.02 | bwd_allreduce_microstep: 48.88 | step_microstep: 18.03
[2025-04-25 14:28:28,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5745.96 | bwd_inner: 5697.02 | bwd_allreduce: 48.89 | step: 18.03

  7%|▋         | 2700/41250 [6:30:54<92:51:53,  8.67s/it]
                                                         
{'loss': 0.1916, 'grad_norm': 1.1080245971679688, 'learning_rate': 3.9868375410642194e-05, 'epoch': 0.65}

  7%|▋         | 2700/41250 [6:30:54<92:51:53,  8.67s/it][2025-04-25 14:28:37,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:28:37,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.85 | bwd_microstep: 5710.97 | bwd_inner_microstep: 5698.26 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.40
[2025-04-25 14:28:37,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.85 | bwd: 5710.98 | bwd_inner: 5698.26 | bwd_allreduce: 12.68 | step: 18.40

  7%|▋         | 2701/41250 [6:31:02<92:48:00,  8.67s/it]
                                                         
{'loss': 0.2289, 'grad_norm': 1.4179420471191406, 'learning_rate': 3.986819548616655e-05, 'epoch': 0.65}

  7%|▋         | 2701/41250 [6:31:02<92:48:00,  8.67s/it][2025-04-25 14:28:46,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:28:46,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5712.10 | bwd_inner_microstep: 5699.43 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.54
[2025-04-25 14:28:46,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5712.11 | bwd_inner: 5699.43 | bwd_allreduce: 12.64 | step: 18.55

  7%|▋         | 2702/41250 [6:31:11<92:43:50,  8.66s/it]
                                                         
{'loss': 0.081, 'grad_norm': 0.676996648311615, 'learning_rate': 3.98680154392074e-05, 'epoch': 0.66}

  7%|▋         | 2702/41250 [6:31:11<92:43:50,  8.66s/it][2025-04-25 14:28:54,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:28:54,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.35 | bwd_microstep: 5708.43 | bwd_inner_microstep: 5695.69 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.45
[2025-04-25 14:28:54,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.36 | bwd: 5708.45 | bwd_inner: 5695.69 | bwd_allreduce: 12.72 | step: 18.46

  7%|▋         | 2703/41250 [6:31:20<92:41:28,  8.66s/it]
                                                         
{'loss': 0.1047, 'grad_norm': 0.9111818671226501, 'learning_rate': 3.986783526976584e-05, 'epoch': 0.66}

  7%|▋         | 2703/41250 [6:31:20<92:41:28,  8.66s/it][2025-04-25 14:29:03,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:29:03,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.40 | bwd_microstep: 5772.84 | bwd_inner_microstep: 5760.06 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.50
[2025-04-25 14:29:03,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.40 | bwd: 5772.85 | bwd_inner: 5760.06 | bwd_allreduce: 12.75 | step: 18.51

  7%|▋         | 2704/41250 [6:31:28<92:57:00,  8.68s/it]
                                                         
{'loss': 0.2461, 'grad_norm': 1.11396062374115, 'learning_rate': 3.9867654977842975e-05, 'epoch': 0.66}

  7%|▋         | 2704/41250 [6:31:28<92:57:00,  8.68s/it][2025-04-25 14:29:12,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 14:29:12,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5755.01 | bwd_inner_microstep: 5642.54 | bwd_allreduce_microstep: 112.42 | step_microstep: 18.50
[2025-04-25 14:29:12,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5755.02 | bwd_inner: 5642.54 | bwd_allreduce: 112.44 | step: 18.50

  7%|▋         | 2705/41250 [6:31:37<92:53:41,  8.68s/it]
                                                         
{'loss': 0.2101, 'grad_norm': 1.5500096082687378, 'learning_rate': 3.986747456343993e-05, 'epoch': 0.66}

  7%|▋         | 2705/41250 [6:31:37<92:53:41,  8.68s/it][2025-04-25 14:29:20,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:29:20,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.74 | bwd_microstep: 5744.23 | bwd_inner_microstep: 5641.03 | bwd_allreduce_microstep: 103.16 | step_microstep: 18.32
[2025-04-25 14:29:20,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.74 | bwd: 5744.25 | bwd_inner: 5641.03 | bwd_allreduce: 103.18 | step: 18.32

  7%|▋         | 2706/41250 [6:31:46<92:49:01,  8.67s/it]
                                                         
{'loss': 0.2856, 'grad_norm': 1.4390311241149902, 'learning_rate': 3.986729402655782e-05, 'epoch': 0.66}

  7%|▋         | 2706/41250 [6:31:46<92:49:01,  8.67s/it][2025-04-25 14:29:29,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:29:29,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.71 | bwd_microstep: 5761.06 | bwd_inner_microstep: 5642.91 | bwd_allreduce_microstep: 118.10 | step_microstep: 18.28
[2025-04-25 14:29:29,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.71 | bwd: 5761.07 | bwd_inner: 5642.91 | bwd_allreduce: 118.12 | step: 18.28

  7%|▋         | 2707/41250 [6:31:54<92:48:13,  8.67s/it]
                                                         
{'loss': 0.1372, 'grad_norm': 0.7127693295478821, 'learning_rate': 3.986711336719774e-05, 'epoch': 0.66}

  7%|▋         | 2707/41250 [6:31:54<92:48:13,  8.67s/it][2025-04-25 14:29:38,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 14:29:38,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.97 | bwd_microstep: 5744.17 | bwd_inner_microstep: 5643.91 | bwd_allreduce_microstep: 100.21 | step_microstep: 19.03
[2025-04-25 14:29:38,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.97 | bwd: 5744.18 | bwd_inner: 5643.91 | bwd_allreduce: 100.23 | step: 19.04

  7%|▋         | 2708/41250 [6:32:03<92:45:00,  8.66s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 1.8608944416046143, 'learning_rate': 3.986693258536081e-05, 'epoch': 0.66}

  7%|▋         | 2708/41250 [6:32:03<92:45:00,  8.66s/it][2025-04-25 14:29:46,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 14:29:46,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.48 | bwd_microstep: 5748.74 | bwd_inner_microstep: 5654.59 | bwd_allreduce_microstep: 94.11 | step_microstep: 17.98
[2025-04-25 14:29:46,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.49 | bwd: 5748.76 | bwd_inner: 5654.59 | bwd_allreduce: 94.12 | step: 17.98

  7%|▋         | 2709/41250 [6:32:12<92:42:55,  8.66s/it]
                                                         
{'loss': 0.1413, 'grad_norm': 1.067541241645813, 'learning_rate': 3.9866751681048156e-05, 'epoch': 0.66}

  7%|▋         | 2709/41250 [6:32:12<92:42:55,  8.66s/it][2025-04-25 14:29:55,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:29:55,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5688.61 | bwd_inner_microstep: 5657.87 | bwd_allreduce_microstep: 30.70 | step_microstep: 17.93
[2025-04-25 14:29:55,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5688.62 | bwd_inner: 5657.87 | bwd_allreduce: 30.71 | step: 17.93

  7%|▋         | 2710/41250 [6:32:20<92:30:59,  8.64s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 0.9893290400505066, 'learning_rate': 3.9866570654260886e-05, 'epoch': 0.66}

  7%|▋         | 2710/41250 [6:32:20<92:30:59,  8.64s/it][2025-04-25 14:30:04,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:30:04,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.58 | bwd_microstep: 5708.83 | bwd_inner_microstep: 5695.83 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.97
[2025-04-25 14:30:04,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.58 | bwd: 5708.84 | bwd_inner: 5695.83 | bwd_allreduce: 12.97 | step: 18.98

  7%|▋         | 2711/41250 [6:32:29<92:33:05,  8.65s/it]
                                                         
{'loss': 0.1529, 'grad_norm': 1.4359877109527588, 'learning_rate': 3.986638950500011e-05, 'epoch': 0.66}

  7%|▋         | 2711/41250 [6:32:29<92:33:05,  8.65s/it][2025-04-25 14:30:12,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.83 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 14:30:12,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.19 | bwd_microstep: 5747.49 | bwd_inner_microstep: 5648.84 | bwd_allreduce_microstep: 98.60 | step_microstep: 17.58
[2025-04-25 14:30:12,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.19 | bwd: 5747.50 | bwd_inner: 5648.84 | bwd_allreduce: 98.62 | step: 17.58

  7%|▋         | 2712/41250 [6:32:38<92:35:15,  8.65s/it]
                                                         
{'loss': 0.3213, 'grad_norm': 1.7145969867706299, 'learning_rate': 3.9866208233266954e-05, 'epoch': 0.66}

  7%|▋         | 2712/41250 [6:32:38<92:35:15,  8.65s/it][2025-04-25 14:30:21,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:30:21,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.51 | bwd_microstep: 5759.09 | bwd_inner_microstep: 5647.33 | bwd_allreduce_microstep: 111.72 | step_microstep: 18.42
[2025-04-25 14:30:21,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.51 | bwd: 5759.11 | bwd_inner: 5647.33 | bwd_allreduce: 111.73 | step: 18.42

  7%|▋         | 2713/41250 [6:32:46<92:39:16,  8.66s/it]
                                                         
{'loss': 0.1293, 'grad_norm': 1.4345499277114868, 'learning_rate': 3.9866026839062535e-05, 'epoch': 0.66}

  7%|▋         | 2713/41250 [6:32:46<92:39:16,  8.66s/it][2025-04-25 14:30:30,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 14:30:30,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.98 | bwd_microstep: 5731.74 | bwd_inner_microstep: 5643.40 | bwd_allreduce_microstep: 88.30 | step_microstep: 18.04
[2025-04-25 14:30:30,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.98 | bwd: 5731.75 | bwd_inner: 5643.40 | bwd_allreduce: 88.32 | step: 18.04

  7%|▋         | 2714/41250 [6:32:55<92:36:30,  8.65s/it]
                                                         
{'loss': 0.0907, 'grad_norm': 1.9243566989898682, 'learning_rate': 3.986584532238796e-05, 'epoch': 0.66}

  7%|▋         | 2714/41250 [6:32:55<92:36:30,  8.65s/it][2025-04-25 14:30:38,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:30:38,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.92 | bwd_microstep: 5765.66 | bwd_inner_microstep: 5643.09 | bwd_allreduce_microstep: 122.53 | step_microstep: 18.36
[2025-04-25 14:30:38,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.92 | bwd: 5765.67 | bwd_inner: 5643.09 | bwd_allreduce: 122.55 | step: 18.37

  7%|▋         | 2715/41250 [6:33:04<92:39:26,  8.66s/it]
                                                         
{'loss': 0.0837, 'grad_norm': 0.7405200600624084, 'learning_rate': 3.986566368324437e-05, 'epoch': 0.66}

  7%|▋         | 2715/41250 [6:33:04<92:39:26,  8.66s/it][2025-04-25 14:30:47,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:30:47,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.19 | bwd_microstep: 5762.68 | bwd_inner_microstep: 5644.06 | bwd_allreduce_microstep: 118.58 | step_microstep: 18.56
[2025-04-25 14:30:47,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.19 | bwd: 5762.69 | bwd_inner: 5644.06 | bwd_allreduce: 118.59 | step: 18.57

  7%|▋         | 2716/41250 [6:33:12<92:41:21,  8.66s/it]
                                                         
{'loss': 0.0627, 'grad_norm': 0.5653150081634521, 'learning_rate': 3.9865481921632854e-05, 'epoch': 0.66}

  7%|▋         | 2716/41250 [6:33:12<92:41:21,  8.66s/it][2025-04-25 14:30:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:30:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.04 | bwd_microstep: 5663.94 | bwd_inner_microstep: 5651.06 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.67
[2025-04-25 14:30:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.04 | bwd: 5663.95 | bwd_inner: 5651.06 | bwd_allreduce: 12.86 | step: 18.67

  7%|▋         | 2717/41250 [6:33:21<92:24:16,  8.63s/it]
                                                         
{'loss': 0.0717, 'grad_norm': 0.7816376090049744, 'learning_rate': 3.986530003755456e-05, 'epoch': 0.66}

  7%|▋         | 2717/41250 [6:33:21<92:24:16,  8.63s/it][2025-04-25 14:31:04,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:31:04,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.13 | bwd_microstep: 5770.54 | bwd_inner_microstep: 5757.34 | bwd_allreduce_microstep: 13.15 | step_microstep: 19.08
[2025-04-25 14:31:04,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.13 | bwd: 5770.55 | bwd_inner: 5757.34 | bwd_allreduce: 13.17 | step: 19.08

  7%|▋         | 2718/41250 [6:33:29<92:44:10,  8.66s/it]
                                                         
{'loss': 0.1798, 'grad_norm': 1.3568917512893677, 'learning_rate': 3.986511803101059e-05, 'epoch': 0.66}

  7%|▋         | 2718/41250 [6:33:29<92:44:10,  8.66s/it][2025-04-25 14:31:13,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 14:31:13,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.60 | bwd_microstep: 5665.63 | bwd_inner_microstep: 5652.81 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.20
[2025-04-25 14:31:13,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.60 | bwd: 5665.64 | bwd_inner: 5652.81 | bwd_allreduce: 12.79 | step: 19.20

  7%|▋         | 2719/41250 [6:33:38<92:28:30,  8.64s/it]
                                                         
{'loss': 0.2573, 'grad_norm': 1.9370719194412231, 'learning_rate': 3.9864935902002086e-05, 'epoch': 0.66}

  7%|▋         | 2719/41250 [6:33:38<92:28:30,  8.64s/it][2025-04-25 14:31:21,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:31:21,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.88 | bwd_microstep: 5705.97 | bwd_inner_microstep: 5680.34 | bwd_allreduce_microstep: 25.58 | step_microstep: 18.91
[2025-04-25 14:31:21,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.88 | bwd: 5705.98 | bwd_inner: 5680.34 | bwd_allreduce: 25.60 | step: 18.91

  7%|▋         | 2720/41250 [6:33:47<92:29:06,  8.64s/it]
                                                         
{'loss': 0.1793, 'grad_norm': 1.4656785726547241, 'learning_rate': 3.986475365053015e-05, 'epoch': 0.66}

  7%|▋         | 2720/41250 [6:33:47<92:29:06,  8.64s/it][2025-04-25 14:31:30,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:31:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.01 | bwd_microstep: 5703.56 | bwd_inner_microstep: 5690.82 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.59
[2025-04-25 14:31:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.01 | bwd: 5703.58 | bwd_inner: 5690.82 | bwd_allreduce: 12.72 | step: 18.60

  7%|▋         | 2721/41250 [6:33:55<92:26:53,  8.64s/it]
                                                         
{'loss': 0.2234, 'grad_norm': 2.254300594329834, 'learning_rate': 3.986457127659592e-05, 'epoch': 0.66}

  7%|▋         | 2721/41250 [6:33:55<92:26:53,  8.64s/it][2025-04-25 14:31:39,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:31:39,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5703.35 | bwd_inner_microstep: 5690.72 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.78
[2025-04-25 14:31:39,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.48 | bwd: 5703.37 | bwd_inner: 5690.72 | bwd_allreduce: 12.61 | step: 18.78

  7%|▋         | 2722/41250 [6:34:04<92:26:17,  8.64s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 1.3872379064559937, 'learning_rate': 3.9864388780200514e-05, 'epoch': 0.66}

  7%|▋         | 2722/41250 [6:34:04<92:26:17,  8.64s/it][2025-04-25 14:31:47,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 14:31:47,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.61 | bwd_microstep: 5779.35 | bwd_inner_microstep: 5766.59 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.27
[2025-04-25 14:31:47,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.61 | bwd: 5779.36 | bwd_inner: 5766.59 | bwd_allreduce: 12.73 | step: 19.28

  7%|▋         | 2723/41250 [6:34:13<92:47:02,  8.67s/it]
                                                         
{'loss': 0.1553, 'grad_norm': 1.8986696004867554, 'learning_rate': 3.986420616134505e-05, 'epoch': 0.66}

  7%|▋         | 2723/41250 [6:34:13<92:47:02,  8.67s/it][2025-04-25 14:31:56,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.44 | optimizer_step: 0.90
[2025-04-25 14:31:56,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.45 | bwd_microstep: 5714.55 | bwd_inner_microstep: 5655.21 | bwd_allreduce_microstep: 59.29 | step_microstep: 19.78
[2025-04-25 14:31:56,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.45 | bwd: 5714.56 | bwd_inner: 5655.21 | bwd_allreduce: 59.31 | step: 19.78

  7%|▋         | 2724/41250 [6:34:21<92:40:05,  8.66s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.6694256067276, 'learning_rate': 3.9864023420030675e-05, 'epoch': 0.66}

  7%|▋         | 2724/41250 [6:34:21<92:40:05,  8.66s/it][2025-04-25 14:32:05,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 14:32:05,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5669.40 | bwd_inner_microstep: 5656.63 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.21
[2025-04-25 14:32:05,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.74 | bwd: 5669.42 | bwd_inner: 5656.63 | bwd_allreduce: 12.75 | step: 19.21

  7%|▋         | 2725/41250 [6:34:30<92:25:02,  8.64s/it]
                                                         
{'loss': 0.3448, 'grad_norm': 3.1835403442382812, 'learning_rate': 3.9863840556258495e-05, 'epoch': 0.66}

  7%|▋         | 2725/41250 [6:34:30<92:25:02,  8.64s/it][2025-04-25 14:32:13,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 14:32:13,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5726.53 | bwd_inner_microstep: 5652.22 | bwd_allreduce_microstep: 74.26 | step_microstep: 19.09
[2025-04-25 14:32:13,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5726.54 | bwd_inner: 5652.22 | bwd_allreduce: 74.29 | step: 19.09

  7%|▋         | 2726/41250 [6:34:39<92:24:37,  8.64s/it]
                                                         
{'loss': 0.3637, 'grad_norm': 2.2488889694213867, 'learning_rate': 3.986365757002965e-05, 'epoch': 0.66}

  7%|▋         | 2726/41250 [6:34:39<92:24:37,  8.64s/it][2025-04-25 14:32:22,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 14:32:22,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.33 | bwd_microstep: 5764.09 | bwd_inner_microstep: 5647.67 | bwd_allreduce_microstep: 116.37 | step_microstep: 18.64
[2025-04-25 14:32:22,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.33 | bwd: 5764.11 | bwd_inner: 5647.67 | bwd_allreduce: 116.39 | step: 18.65

  7%|▋         | 2727/41250 [6:34:47<92:31:38,  8.65s/it]
                                                         
{'loss': 0.2106, 'grad_norm': 1.5930067300796509, 'learning_rate': 3.986347446134526e-05, 'epoch': 0.66}

  7%|▋         | 2727/41250 [6:34:47<92:31:38,  8.65s/it][2025-04-25 14:32:31,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:32:31,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.30 | bwd_microstep: 5722.28 | bwd_inner_microstep: 5685.23 | bwd_allreduce_microstep: 37.01 | step_microstep: 18.95
[2025-04-25 14:32:31,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.30 | bwd: 5722.30 | bwd_inner: 5685.23 | bwd_allreduce: 37.02 | step: 18.95

  7%|▋         | 2728/41250 [6:34:56<92:31:40,  8.65s/it]
                                                         
{'loss': 0.1692, 'grad_norm': 1.2103198766708374, 'learning_rate': 3.9863291230206456e-05, 'epoch': 0.66}

  7%|▋         | 2728/41250 [6:34:56<92:31:40,  8.65s/it][2025-04-25 14:32:39,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:32:39,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5678.28 | bwd_inner_microstep: 5656.66 | bwd_allreduce_microstep: 21.57 | step_microstep: 18.48
[2025-04-25 14:32:39,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5678.30 | bwd_inner: 5656.66 | bwd_allreduce: 21.59 | step: 18.48

  7%|▋         | 2729/41250 [6:35:04<92:20:04,  8.63s/it]
                                                         
{'loss': 0.0658, 'grad_norm': 0.9480593204498291, 'learning_rate': 3.9863107876614374e-05, 'epoch': 0.66}

  7%|▋         | 2729/41250 [6:35:04<92:20:04,  8.63s/it][2025-04-25 14:32:48,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:32:48,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.47 | bwd_microstep: 5713.77 | bwd_inner_microstep: 5700.93 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.68
[2025-04-25 14:32:48,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.47 | bwd: 5713.79 | bwd_inner: 5700.93 | bwd_allreduce: 12.81 | step: 18.68

  7%|▋         | 2730/41250 [6:35:13<92:24:07,  8.64s/it]
                                                         
{'loss': 0.0817, 'grad_norm': 0.7806858420372009, 'learning_rate': 3.9862924400570134e-05, 'epoch': 0.66}

  7%|▋         | 2730/41250 [6:35:13<92:24:07,  8.64s/it][2025-04-25 14:32:56,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 14:32:56,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.26 | bwd_microstep: 5736.54 | bwd_inner_microstep: 5638.37 | bwd_allreduce_microstep: 98.12 | step_microstep: 19.23
[2025-04-25 14:32:56,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.26 | bwd: 5736.55 | bwd_inner: 5638.37 | bwd_allreduce: 98.13 | step: 19.23

  7%|▋         | 2731/41250 [6:35:22<92:25:21,  8.64s/it]
                                                         
{'loss': 0.5348, 'grad_norm': 4.029847621917725, 'learning_rate': 3.986274080207487e-05, 'epoch': 0.66}

  7%|▋         | 2731/41250 [6:35:22<92:25:21,  8.64s/it][2025-04-25 14:33:05,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:33:05,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.29 | bwd_microstep: 5670.71 | bwd_inner_microstep: 5657.91 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.84
[2025-04-25 14:33:05,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.29 | bwd: 5670.72 | bwd_inner: 5657.91 | bwd_allreduce: 12.77 | step: 18.84

  7%|▋         | 2732/41250 [6:35:30<92:13:14,  8.62s/it]
                                                         
{'loss': 0.0527, 'grad_norm': 0.47300612926483154, 'learning_rate': 3.986255708112972e-05, 'epoch': 0.66}

  7%|▋         | 2732/41250 [6:35:30<92:13:14,  8.62s/it][2025-04-25 14:33:14,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 14:33:14,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.76 | bwd_microstep: 5755.37 | bwd_inner_microstep: 5655.16 | bwd_allreduce_microstep: 100.16 | step_microstep: 17.83
[2025-04-25 14:33:14,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.76 | bwd: 5755.38 | bwd_inner: 5655.16 | bwd_allreduce: 100.18 | step: 17.83

  7%|▋         | 2733/41250 [6:35:39<92:22:14,  8.63s/it]
                                                         
{'loss': 0.1732, 'grad_norm': 0.9426328539848328, 'learning_rate': 3.986237323773581e-05, 'epoch': 0.66}

  7%|▋         | 2733/41250 [6:35:39<92:22:14,  8.63s/it][2025-04-25 14:33:22,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:33:22,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.85 | bwd_microstep: 5787.60 | bwd_inner_microstep: 5774.70 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.30
[2025-04-25 14:33:22,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.85 | bwd: 5787.61 | bwd_inner: 5774.70 | bwd_allreduce: 12.87 | step: 19.30

  7%|▋         | 2734/41250 [6:35:48<92:47:34,  8.67s/it]
                                                         
{'loss': 0.2554, 'grad_norm': 2.444337844848633, 'learning_rate': 3.986218927189427e-05, 'epoch': 0.66}

  7%|▋         | 2734/41250 [6:35:48<92:47:34,  8.67s/it][2025-04-25 14:33:31,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:33:31,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.01 | bwd_microstep: 5757.91 | bwd_inner_microstep: 5710.67 | bwd_allreduce_microstep: 47.19 | step_microstep: 18.60
[2025-04-25 14:33:31,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.01 | bwd: 5757.93 | bwd_inner: 5710.67 | bwd_allreduce: 47.21 | step: 18.60

  7%|▋         | 2735/41250 [6:35:56<92:52:48,  8.68s/it]
                                                         
{'loss': 0.084, 'grad_norm': 0.8545451760292053, 'learning_rate': 3.986200518360625e-05, 'epoch': 0.66}

  7%|▋         | 2735/41250 [6:35:56<92:52:48,  8.68s/it][2025-04-25 14:33:40,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 14:33:40,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.06 | bwd_microstep: 5714.04 | bwd_inner_microstep: 5701.07 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.07
[2025-04-25 14:33:40,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.06 | bwd: 5714.05 | bwd_inner: 5701.07 | bwd_allreduce: 12.93 | step: 19.07

  7%|▋         | 2736/41250 [6:36:05<92:47:52,  8.67s/it]
                                                         
{'loss': 0.2912, 'grad_norm': 1.927364468574524, 'learning_rate': 3.986182097287287e-05, 'epoch': 0.66}

  7%|▋         | 2736/41250 [6:36:05<92:47:52,  8.67s/it][2025-04-25 14:33:48,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 14:33:48,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.14 | bwd_microstep: 5716.54 | bwd_inner_microstep: 5700.88 | bwd_allreduce_microstep: 15.61 | step_microstep: 18.89
[2025-04-25 14:33:48,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.14 | bwd: 5716.55 | bwd_inner: 5700.88 | bwd_allreduce: 15.63 | step: 18.89

  7%|▋         | 2737/41250 [6:36:14<92:46:04,  8.67s/it]
                                                         
{'loss': 0.3991, 'grad_norm': 1.8864622116088867, 'learning_rate': 3.986163663969527e-05, 'epoch': 0.66}

  7%|▋         | 2737/41250 [6:36:14<92:46:04,  8.67s/it][2025-04-25 14:33:57,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:33:57,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.56 | bwd_microstep: 5761.06 | bwd_inner_microstep: 5644.21 | bwd_allreduce_microstep: 116.80 | step_microstep: 19.15
[2025-04-25 14:33:57,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.56 | bwd: 5761.07 | bwd_inner: 5644.21 | bwd_allreduce: 116.83 | step: 19.15

  7%|▋         | 2738/41250 [6:36:22<92:45:43,  8.67s/it]
                                                         
{'loss': 0.1962, 'grad_norm': 1.961250901222229, 'learning_rate': 3.986145218407459e-05, 'epoch': 0.66}

  7%|▋         | 2738/41250 [6:36:22<92:45:43,  8.67s/it][2025-04-25 14:34:06,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 14:34:06,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5703.07 | bwd_inner_microstep: 5689.28 | bwd_allreduce_microstep: 13.74 | step_microstep: 19.14
[2025-04-25 14:34:06,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5703.08 | bwd_inner: 5689.28 | bwd_allreduce: 13.77 | step: 19.14

  7%|▋         | 2739/41250 [6:36:31<92:37:38,  8.66s/it]
                                                         
{'loss': 0.3128, 'grad_norm': 1.7351489067077637, 'learning_rate': 3.986126760601196e-05, 'epoch': 0.66}

  7%|▋         | 2739/41250 [6:36:31<92:37:38,  8.66s/it][2025-04-25 14:34:14,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 14:34:14,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.54 | bwd_microstep: 5702.82 | bwd_inner_microstep: 5643.64 | bwd_allreduce_microstep: 59.12 | step_microstep: 19.44
[2025-04-25 14:34:14,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.54 | bwd: 5702.83 | bwd_inner: 5643.64 | bwd_allreduce: 59.15 | step: 19.44

  7%|▋         | 2740/41250 [6:36:40<92:28:28,  8.64s/it]
                                                         
{'loss': 0.1371, 'grad_norm': 1.6687668561935425, 'learning_rate': 3.9861082905508525e-05, 'epoch': 0.66}

  7%|▋         | 2740/41250 [6:36:40<92:28:28,  8.64s/it][2025-04-25 14:34:23,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:34:23,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.62 | bwd_microstep: 5733.89 | bwd_inner_microstep: 5721.21 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.45
[2025-04-25 14:34:23,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.62 | bwd: 5733.90 | bwd_inner: 5721.21 | bwd_allreduce: 12.65 | step: 18.45

  7%|▋         | 2741/41250 [6:36:48<92:32:37,  8.65s/it]
                                                         
{'loss': 0.2643, 'grad_norm': 1.6789684295654297, 'learning_rate': 3.9860898082565415e-05, 'epoch': 0.66}

  7%|▋         | 2741/41250 [6:36:48<92:32:37,  8.65s/it][2025-04-25 14:34:32,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:34:32,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.40 | bwd_microstep: 5732.91 | bwd_inner_microstep: 5693.59 | bwd_allreduce_microstep: 39.28 | step_microstep: 18.86
[2025-04-25 14:34:32,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.40 | bwd: 5732.93 | bwd_inner: 5693.59 | bwd_allreduce: 39.29 | step: 18.86

  7%|▋         | 2742/41250 [6:36:57<92:34:18,  8.65s/it]
                                                         
{'loss': 0.1441, 'grad_norm': 0.9746026992797852, 'learning_rate': 3.986071313718378e-05, 'epoch': 0.66}

  7%|▋         | 2742/41250 [6:36:57<92:34:18,  8.65s/it][2025-04-25 14:34:40,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 14:34:40,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.47 | bwd_microstep: 5720.37 | bwd_inner_microstep: 5707.73 | bwd_allreduce_microstep: 12.58 | step_microstep: 19.03
[2025-04-25 14:34:40,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.47 | bwd: 5720.38 | bwd_inner: 5707.73 | bwd_allreduce: 12.61 | step: 19.03

  7%|▋         | 2743/41250 [6:37:06<92:33:33,  8.65s/it]
                                                         
{'loss': 0.5292, 'grad_norm': 4.486796855926514, 'learning_rate': 3.9860528069364753e-05, 'epoch': 0.66}

  7%|▋         | 2743/41250 [6:37:06<92:33:33,  8.65s/it][2025-04-25 14:34:49,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 14:34:49,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.82 | bwd_microstep: 5709.84 | bwd_inner_microstep: 5658.89 | bwd_allreduce_microstep: 50.91 | step_microstep: 19.04
[2025-04-25 14:34:49,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.82 | bwd: 5709.86 | bwd_inner: 5658.89 | bwd_allreduce: 50.93 | step: 19.04

  7%|▋         | 2744/41250 [6:37:14<92:28:22,  8.65s/it]
                                                         
{'loss': 0.2459, 'grad_norm': 2.121729612350464, 'learning_rate': 3.986034287910948e-05, 'epoch': 0.67}

  7%|▋         | 2744/41250 [6:37:14<92:28:22,  8.65s/it][2025-04-25 14:34:58,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 14:34:58,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5700.89 | bwd_inner_microstep: 5685.87 | bwd_allreduce_microstep: 14.98 | step_microstep: 19.02
[2025-04-25 14:34:58,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.85 | bwd: 5700.91 | bwd_inner: 5685.87 | bwd_allreduce: 15.00 | step: 19.02

  7%|▋         | 2745/41250 [6:37:23<92:25:00,  8.64s/it]
                                                         
{'loss': 0.1605, 'grad_norm': 0.7414129376411438, 'learning_rate': 3.9860157566419095e-05, 'epoch': 0.67}

  7%|▋         | 2745/41250 [6:37:23<92:25:00,  8.64s/it][2025-04-25 14:35:06,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:35:06,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5718.90 | bwd_inner_microstep: 5706.05 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.00
[2025-04-25 14:35:06,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5718.92 | bwd_inner: 5706.05 | bwd_allreduce: 12.82 | step: 19.00

  7%|▋         | 2746/41250 [6:37:32<92:26:52,  8.64s/it]
                                                         
{'loss': 0.1501, 'grad_norm': 1.8117703199386597, 'learning_rate': 3.9859972131294746e-05, 'epoch': 0.67}

  7%|▋         | 2746/41250 [6:37:32<92:26:52,  8.64s/it][2025-04-25 14:35:15,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 14:35:15,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.96 | bwd_microstep: 5884.20 | bwd_inner_microstep: 5706.15 | bwd_allreduce_microstep: 178.00 | step_microstep: 18.75
[2025-04-25 14:35:15,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.96 | bwd: 5884.21 | bwd_inner: 5706.15 | bwd_allreduce: 178.02 | step: 18.75

  7%|▋         | 2747/41250 [6:37:40<93:00:39,  8.70s/it]
                                                         
{'loss': 0.2083, 'grad_norm': 1.8234877586364746, 'learning_rate': 3.985978657373758e-05, 'epoch': 0.67}

  7%|▋         | 2747/41250 [6:37:40<93:00:39,  8.70s/it][2025-04-25 14:35:24,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.31 | optimizer_step: 1.03
[2025-04-25 14:35:24,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.05 | bwd_microstep: 5893.59 | bwd_inner_microstep: 5670.41 | bwd_allreduce_microstep: 223.12 | step_microstep: 19.95
[2025-04-25 14:35:24,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.05 | bwd: 5893.61 | bwd_inner: 5670.41 | bwd_allreduce: 223.15 | step: 19.95

  7%|▋         | 2748/41250 [6:37:49<93:23:13,  8.73s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 1.2092970609664917, 'learning_rate': 3.9859600893748726e-05, 'epoch': 0.67}

  7%|▋         | 2748/41250 [6:37:49<93:23:13,  8.73s/it][2025-04-25 14:35:33,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:35:33,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5759.35 | bwd_inner_microstep: 5648.32 | bwd_allreduce_microstep: 110.99 | step_microstep: 18.45
[2025-04-25 14:35:33,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5759.37 | bwd_inner: 5648.32 | bwd_allreduce: 111.01 | step: 18.45

  7%|▋         | 2749/41250 [6:37:58<93:11:14,  8.71s/it]
                                                         
{'loss': 0.2341, 'grad_norm': 1.6964855194091797, 'learning_rate': 3.985941509132935e-05, 'epoch': 0.67}

  7%|▋         | 2749/41250 [6:37:58<93:11:14,  8.71s/it][2025-04-25 14:35:41,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 14:35:41,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5729.25 | bwd_inner_microstep: 5712.65 | bwd_allreduce_microstep: 16.55 | step_microstep: 19.10
[2025-04-25 14:35:41,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5729.26 | bwd_inner: 5712.65 | bwd_allreduce: 16.57 | step: 19.10

  7%|▋         | 2750/41250 [6:38:07<93:01:48,  8.70s/it]
                                                         
{'loss': 0.1304, 'grad_norm': 1.1700968742370605, 'learning_rate': 3.985922916648058e-05, 'epoch': 0.67}

  7%|▋         | 2750/41250 [6:38:07<93:01:48,  8.70s/it][2025-04-25 14:35:50,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:35:50,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.39 | bwd_microstep: 5737.88 | bwd_inner_microstep: 5725.15 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.95
[2025-04-25 14:35:50,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.39 | bwd: 5737.90 | bwd_inner: 5725.15 | bwd_allreduce: 12.71 | step: 18.95

  7%|▋         | 2751/41250 [6:38:15<92:58:21,  8.69s/it]
                                                         
{'loss': 0.1672, 'grad_norm': 1.8540127277374268, 'learning_rate': 3.985904311920356e-05, 'epoch': 0.67}

  7%|▋         | 2751/41250 [6:38:15<92:58:21,  8.69s/it][2025-04-25 14:35:59,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:35:59,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.08 | bwd_microstep: 5785.13 | bwd_inner_microstep: 5772.31 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.82
[2025-04-25 14:35:59,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.08 | bwd: 5785.14 | bwd_inner: 5772.31 | bwd_allreduce: 12.79 | step: 18.82

  7%|▋         | 2752/41250 [6:38:24<93:09:53,  8.71s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.3715546131134033, 'learning_rate': 3.9858856949499456e-05, 'epoch': 0.67}

  7%|▋         | 2752/41250 [6:38:24<93:09:53,  8.71s/it][2025-04-25 14:36:07,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:36:07,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.08 | bwd_microstep: 5855.57 | bwd_inner_microstep: 5713.53 | bwd_allreduce_microstep: 141.99 | step_microstep: 18.75
[2025-04-25 14:36:07,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.08 | bwd: 5855.58 | bwd_inner: 5713.53 | bwd_allreduce: 142.01 | step: 18.75

  7%|▋         | 2753/41250 [6:38:33<93:25:04,  8.74s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.1569563150405884, 'learning_rate': 3.9858670657369405e-05, 'epoch': 0.67}

  7%|▋         | 2753/41250 [6:38:33<93:25:04,  8.74s/it][2025-04-25 14:36:16,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:36:16,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.51 | bwd_microstep: 5730.66 | bwd_inner_microstep: 5698.49 | bwd_allreduce_microstep: 32.12 | step_microstep: 18.86
[2025-04-25 14:36:16,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.51 | bwd: 5730.67 | bwd_inner: 5698.49 | bwd_allreduce: 32.14 | step: 18.86

  7%|▋         | 2754/41250 [6:38:41<93:12:38,  8.72s/it]
                                                         
{'loss': 0.07, 'grad_norm': 0.6648049354553223, 'learning_rate': 3.985848424281455e-05, 'epoch': 0.67}

  7%|▋         | 2754/41250 [6:38:41<93:12:38,  8.72s/it][2025-04-25 14:36:25,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 14:36:25,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.43 | bwd_microstep: 5736.41 | bwd_inner_microstep: 5723.41 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.06
[2025-04-25 14:36:25,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.43 | bwd: 5736.43 | bwd_inner: 5723.41 | bwd_allreduce: 12.98 | step: 19.07

  7%|▋         | 2755/41250 [6:38:50<93:06:22,  8.71s/it]
                                                         
{'loss': 0.0951, 'grad_norm': 1.0838656425476074, 'learning_rate': 3.9858297705836054e-05, 'epoch': 0.67}

  7%|▋         | 2755/41250 [6:38:50<93:06:22,  8.71s/it][2025-04-25 14:36:33,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:36:33,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.52 | bwd_microstep: 5712.10 | bwd_inner_microstep: 5658.29 | bwd_allreduce_microstep: 53.77 | step_microstep: 18.47
[2025-04-25 14:36:33,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.52 | bwd: 5712.11 | bwd_inner: 5658.29 | bwd_allreduce: 53.79 | step: 18.48

  7%|▋         | 2756/41250 [6:38:59<92:49:37,  8.68s/it]
                                                         
{'loss': 0.0872, 'grad_norm': 0.9098773002624512, 'learning_rate': 3.985811104643505e-05, 'epoch': 0.67}

  7%|▋         | 2756/41250 [6:38:59<92:49:37,  8.68s/it][2025-04-25 14:36:42,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:36:42,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.57 | bwd_microstep: 5775.56 | bwd_inner_microstep: 5713.42 | bwd_allreduce_microstep: 62.09 | step_microstep: 18.81
[2025-04-25 14:36:42,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.57 | bwd: 5775.57 | bwd_inner: 5713.42 | bwd_allreduce: 62.11 | step: 18.81

  7%|▋         | 2757/41250 [6:39:07<92:56:35,  8.69s/it]
                                                         
{'loss': 0.1933, 'grad_norm': 2.0034210681915283, 'learning_rate': 3.98579242646127e-05, 'epoch': 0.67}

  7%|▋         | 2757/41250 [6:39:07<92:56:35,  8.69s/it][2025-04-25 14:36:51,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 1.11
[2025-04-25 14:36:51,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.95 | bwd_microstep: 5744.10 | bwd_inner_microstep: 5699.65 | bwd_allreduce_microstep: 44.40 | step_microstep: 19.13
[2025-04-25 14:36:51,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.95 | bwd: 5744.12 | bwd_inner: 5699.65 | bwd_allreduce: 44.43 | step: 19.13

  7%|▋         | 2758/41250 [6:39:16<92:53:23,  8.69s/it]
                                                         
{'loss': 0.1233, 'grad_norm': 0.8496196269989014, 'learning_rate': 3.985773736037015e-05, 'epoch': 0.67}

  7%|▋         | 2758/41250 [6:39:16<92:53:23,  8.69s/it][2025-04-25 14:36:59,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 14:36:59,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.80 | bwd_microstep: 5717.64 | bwd_inner_microstep: 5664.90 | bwd_allreduce_microstep: 52.69 | step_microstep: 18.84
[2025-04-25 14:36:59,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.80 | bwd: 5717.65 | bwd_inner: 5664.90 | bwd_allreduce: 52.71 | step: 18.84

  7%|▋         | 2759/41250 [6:39:25<92:42:02,  8.67s/it]
                                                         
{'loss': 0.1403, 'grad_norm': 1.462900996208191, 'learning_rate': 3.985755033370856e-05, 'epoch': 0.67}

  7%|▋         | 2759/41250 [6:39:25<92:42:02,  8.67s/it][2025-04-25 14:37:08,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 14:37:08,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5726.66 | bwd_inner_microstep: 5714.02 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.69
[2025-04-25 14:37:08,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5726.68 | bwd_inner: 5714.02 | bwd_allreduce: 12.62 | step: 18.69

  7%|▋         | 2760/41250 [6:39:33<92:40:30,  8.67s/it]
                                                         
{'loss': 0.1046, 'grad_norm': 1.1250665187835693, 'learning_rate': 3.9857363184629074e-05, 'epoch': 0.67}

  7%|▋         | 2760/41250 [6:39:33<92:40:30,  8.67s/it][2025-04-25 14:37:17,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:37:17,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5717.77 | bwd_inner_microstep: 5704.94 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.61
[2025-04-25 14:37:17,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5717.79 | bwd_inner: 5704.94 | bwd_allreduce: 12.80 | step: 18.61

  7%|▋         | 2761/41250 [6:39:42<92:37:18,  8.66s/it]
                                                         
{'loss': 0.1787, 'grad_norm': 1.4652416706085205, 'learning_rate': 3.985717591313286e-05, 'epoch': 0.67}

  7%|▋         | 2761/41250 [6:39:42<92:37:18,  8.66s/it][2025-04-25 14:37:25,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:37:25,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.82 | bwd_microstep: 5757.51 | bwd_inner_microstep: 5690.39 | bwd_allreduce_microstep: 67.06 | step_microstep: 18.65
[2025-04-25 14:37:25,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.82 | bwd: 5757.52 | bwd_inner: 5690.39 | bwd_allreduce: 67.08 | step: 18.65

  7%|▋         | 2762/41250 [6:39:51<92:41:44,  8.67s/it]
                                                         
{'loss': 0.158, 'grad_norm': 3.642975091934204, 'learning_rate': 3.985698851922105e-05, 'epoch': 0.67}

  7%|▋         | 2762/41250 [6:39:51<92:41:44,  8.67s/it][2025-04-25 14:37:34,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 14:37:34,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.79 | bwd_microstep: 5787.25 | bwd_inner_microstep: 5703.70 | bwd_allreduce_microstep: 83.51 | step_microstep: 19.05
[2025-04-25 14:37:34,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.79 | bwd: 5787.27 | bwd_inner: 5703.70 | bwd_allreduce: 83.53 | step: 19.05

  7%|▋         | 2763/41250 [6:40:00<92:51:00,  8.69s/it]
                                                         
{'loss': 0.1959, 'grad_norm': 0.8699867725372314, 'learning_rate': 3.985680100289482e-05, 'epoch': 0.67}

  7%|▋         | 2763/41250 [6:40:00<92:51:00,  8.69s/it][2025-04-25 14:37:43,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:37:43,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.92 | bwd_microstep: 5700.96 | bwd_inner_microstep: 5688.37 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.34
[2025-04-25 14:37:43,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.92 | bwd: 5700.97 | bwd_inner: 5688.37 | bwd_allreduce: 12.57 | step: 18.34

  7%|▋         | 2764/41250 [6:40:08<92:41:25,  8.67s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 0.9862379431724548, 'learning_rate': 3.985661336415531e-05, 'epoch': 0.67}

  7%|▋         | 2764/41250 [6:40:08<92:41:25,  8.67s/it][2025-04-25 14:37:52,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:37:52,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.70 | bwd_microstep: 5750.92 | bwd_inner_microstep: 5648.73 | bwd_allreduce_microstep: 102.14 | step_microstep: 18.22
[2025-04-25 14:37:52,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.70 | bwd: 5750.93 | bwd_inner: 5648.73 | bwd_allreduce: 102.16 | step: 18.22

  7%|▋         | 2765/41250 [6:40:17<92:42:08,  8.67s/it]
                                                         
{'loss': 0.2059, 'grad_norm': 2.7803494930267334, 'learning_rate': 3.9856425603003685e-05, 'epoch': 0.67}

  7%|▋         | 2765/41250 [6:40:17<92:42:08,  8.67s/it][2025-04-25 14:38:00,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:38:00,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.09 | bwd_microstep: 5697.29 | bwd_inner_microstep: 5649.82 | bwd_allreduce_microstep: 47.42 | step_microstep: 18.33
[2025-04-25 14:38:00,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.09 | bwd: 5697.30 | bwd_inner: 5649.82 | bwd_allreduce: 47.44 | step: 18.33

  7%|▋         | 2766/41250 [6:40:25<92:30:13,  8.65s/it]
                                                         
{'loss': 0.1209, 'grad_norm': 1.7539175748825073, 'learning_rate': 3.98562377194411e-05, 'epoch': 0.67}

  7%|▋         | 2766/41250 [6:40:25<92:30:13,  8.65s/it][2025-04-25 14:38:09,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-25 14:38:09,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.57 | bwd_microstep: 5723.62 | bwd_inner_microstep: 5710.96 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.83
[2025-04-25 14:38:09,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.57 | bwd: 5723.64 | bwd_inner: 5710.96 | bwd_allreduce: 12.64 | step: 18.83

  7%|▋         | 2767/41250 [6:40:34<92:31:51,  8.66s/it]
                                                         
{'loss': 0.0435, 'grad_norm': 0.41430234909057617, 'learning_rate': 3.985604971346872e-05, 'epoch': 0.67}

  7%|▋         | 2767/41250 [6:40:34<92:31:51,  8.66s/it][2025-04-25 14:38:17,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:38:17,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5702.32 | bwd_inner_microstep: 5654.53 | bwd_allreduce_microstep: 47.75 | step_microstep: 17.96
[2025-04-25 14:38:17,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5702.33 | bwd_inner: 5654.53 | bwd_allreduce: 47.76 | step: 17.96

  7%|▋         | 2768/41250 [6:40:43<92:21:51,  8.64s/it]
                                                         
{'loss': 0.3549, 'grad_norm': 1.7365883588790894, 'learning_rate': 3.9855861585087696e-05, 'epoch': 0.67}

  7%|▋         | 2768/41250 [6:40:43<92:21:51,  8.64s/it][2025-04-25 14:38:26,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:38:26,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.01 | bwd_microstep: 5682.74 | bwd_inner_microstep: 5651.70 | bwd_allreduce_microstep: 30.99 | step_microstep: 18.27
[2025-04-25 14:38:26,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.01 | bwd: 5682.75 | bwd_inner: 5651.70 | bwd_allreduce: 31.01 | step: 18.27

  7%|▋         | 2769/41250 [6:40:51<92:14:51,  8.63s/it]
                                                         
{'loss': 0.1513, 'grad_norm': 1.5975009202957153, 'learning_rate': 3.9855673334299196e-05, 'epoch': 0.67}

  7%|▋         | 2769/41250 [6:40:51<92:14:51,  8.63s/it][2025-04-25 14:38:35,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 14:38:35,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.21 | bwd_microstep: 5715.45 | bwd_inner_microstep: 5691.02 | bwd_allreduce_microstep: 24.38 | step_microstep: 18.09
[2025-04-25 14:38:35,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.21 | bwd: 5715.46 | bwd_inner: 5691.02 | bwd_allreduce: 24.40 | step: 18.09

  7%|▋         | 2770/41250 [6:41:00<92:19:05,  8.64s/it]
                                                         
{'loss': 0.2766, 'grad_norm': 1.9164276123046875, 'learning_rate': 3.985548496110437e-05, 'epoch': 0.67}

  7%|▋         | 2770/41250 [6:41:00<92:19:05,  8.64s/it][2025-04-25 14:38:43,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-25 14:38:43,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.57 | bwd_microstep: 5849.15 | bwd_inner_microstep: 5694.47 | bwd_allreduce_microstep: 154.64 | step_microstep: 18.22
[2025-04-25 14:38:43,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.57 | bwd: 5849.16 | bwd_inner: 5694.47 | bwd_allreduce: 154.65 | step: 18.22

  7%|▋         | 2771/41250 [6:41:09<92:45:14,  8.68s/it]
                                                         
{'loss': 0.0486, 'grad_norm': 0.7583678364753723, 'learning_rate': 3.985529646550439e-05, 'epoch': 0.67}

  7%|▋         | 2771/41250 [6:41:09<92:45:14,  8.68s/it][2025-04-25 14:38:52,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 14:38:52,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.78 | bwd_microstep: 5747.57 | bwd_inner_microstep: 5635.99 | bwd_allreduce_microstep: 111.54 | step_microstep: 18.49
[2025-04-25 14:38:52,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.78 | bwd: 5747.58 | bwd_inner: 5635.98 | bwd_allreduce: 111.55 | step: 18.49

  7%|▋         | 2772/41250 [6:41:17<92:41:15,  8.67s/it]
                                                         
{'loss': 0.2037, 'grad_norm': 1.9022771120071411, 'learning_rate': 3.985510784750041e-05, 'epoch': 0.67}

  7%|▋         | 2772/41250 [6:41:17<92:41:15,  8.67s/it][2025-04-25 14:39:01,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:39:01,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.69 | bwd_microstep: 5739.50 | bwd_inner_microstep: 5690.44 | bwd_allreduce_microstep: 49.01 | step_microstep: 18.24
[2025-04-25 14:39:01,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.69 | bwd: 5739.51 | bwd_inner: 5690.44 | bwd_allreduce: 49.03 | step: 18.24

  7%|▋         | 2773/41250 [6:41:26<92:39:21,  8.67s/it]
                                                         
{'loss': 0.0765, 'grad_norm': 0.9470141530036926, 'learning_rate': 3.985491910709359e-05, 'epoch': 0.67}

  7%|▋         | 2773/41250 [6:41:26<92:39:21,  8.67s/it][2025-04-25 14:39:09,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 14:39:09,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5701.34 | bwd_inner_microstep: 5688.57 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.92
[2025-04-25 14:39:09,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5701.35 | bwd_inner: 5688.57 | bwd_allreduce: 12.74 | step: 18.93

  7%|▋         | 2774/41250 [6:41:35<92:34:22,  8.66s/it]
                                                         
{'loss': 0.1993, 'grad_norm': 2.569950580596924, 'learning_rate': 3.985473024428511e-05, 'epoch': 0.67}

  7%|▋         | 2774/41250 [6:41:35<92:34:22,  8.66s/it][2025-04-25 14:39:18,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 14:39:18,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5697.02 | bwd_inner_microstep: 5684.20 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.83
[2025-04-25 14:39:18,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5697.04 | bwd_inner: 5684.20 | bwd_allreduce: 12.79 | step: 18.83

  7%|▋         | 2775/41250 [6:41:43<92:29:44,  8.65s/it]
                                                         
{'loss': 0.3306, 'grad_norm': 1.5464730262756348, 'learning_rate': 3.9854541259076114e-05, 'epoch': 0.67}

  7%|▋         | 2775/41250 [6:41:43<92:29:44,  8.65s/it][2025-04-25 14:39:27,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:39:27,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.07 | bwd_microstep: 5709.71 | bwd_inner_microstep: 5696.87 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.83
[2025-04-25 14:39:27,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.07 | bwd: 5709.72 | bwd_inner: 5696.87 | bwd_allreduce: 12.81 | step: 18.83

  7%|▋         | 2776/41250 [6:41:52<92:27:44,  8.65s/it]
                                                         
{'loss': 0.1262, 'grad_norm': 1.1528774499893188, 'learning_rate': 3.985435215146778e-05, 'epoch': 0.67}

  7%|▋         | 2776/41250 [6:41:52<92:27:44,  8.65s/it][2025-04-25 14:39:35,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:39:35,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.66 | bwd_microstep: 5743.99 | bwd_inner_microstep: 5648.87 | bwd_allreduce_microstep: 95.07 | step_microstep: 18.72
[2025-04-25 14:39:35,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.66 | bwd: 5744.00 | bwd_inner: 5648.87 | bwd_allreduce: 95.09 | step: 18.72

  7%|▋         | 2777/41250 [6:42:01<92:28:15,  8.65s/it]
                                                         
{'loss': 0.191, 'grad_norm': 2.2541890144348145, 'learning_rate': 3.985416292146126e-05, 'epoch': 0.67}

  7%|▋         | 2777/41250 [6:42:01<92:28:15,  8.65s/it][2025-04-25 14:39:44,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:39:44,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.81 | bwd_microstep: 5733.87 | bwd_inner_microstep: 5685.94 | bwd_allreduce_microstep: 47.88 | step_microstep: 18.72
[2025-04-25 14:39:44,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.81 | bwd: 5733.88 | bwd_inner: 5685.94 | bwd_allreduce: 47.90 | step: 18.72

  7%|▋         | 2778/41250 [6:42:09<92:29:09,  8.65s/it]
                                                         
{'loss': 0.2011, 'grad_norm': 1.773501992225647, 'learning_rate': 3.985397356905774e-05, 'epoch': 0.67}

  7%|▋         | 2778/41250 [6:42:09<92:29:09,  8.65s/it][2025-04-25 14:39:53,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:39:53,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.88 | bwd_microstep: 5764.77 | bwd_inner_microstep: 5638.47 | bwd_allreduce_microstep: 126.26 | step_microstep: 18.73
[2025-04-25 14:39:53,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.88 | bwd: 5764.78 | bwd_inner: 5638.46 | bwd_allreduce: 126.28 | step: 18.74

  7%|▋         | 2779/41250 [6:42:18<92:30:59,  8.66s/it]
                                                         
{'loss': 0.0901, 'grad_norm': 2.021394968032837, 'learning_rate': 3.9853784094258386e-05, 'epoch': 0.67}

  7%|▋         | 2779/41250 [6:42:18<92:30:59,  8.66s/it][2025-04-25 14:40:02,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:40:02,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.45 | bwd_microstep: 6000.10 | bwd_inner_microstep: 5681.70 | bwd_allreduce_microstep: 318.36 | step_microstep: 18.82
[2025-04-25 14:40:02,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.45 | bwd: 6000.12 | bwd_inner: 5681.70 | bwd_allreduce: 318.38 | step: 18.82

  7%|▋         | 2780/41250 [6:42:27<93:22:00,  8.74s/it]
                                                         
{'loss': 0.2436, 'grad_norm': 2.406944751739502, 'learning_rate': 3.985359449706435e-05, 'epoch': 0.67}

  7%|▋         | 2780/41250 [6:42:27<93:22:00,  8.74s/it][2025-04-25 14:40:10,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:40:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.60 | bwd_microstep: 5700.88 | bwd_inner_microstep: 5688.08 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.66
[2025-04-25 14:40:10,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.60 | bwd: 5700.90 | bwd_inner: 5688.08 | bwd_allreduce: 12.78 | step: 18.67

  7%|▋         | 2781/41250 [6:42:36<93:00:57,  8.70s/it]
                                                         
{'loss': 0.427, 'grad_norm': 3.0711395740509033, 'learning_rate': 3.985340477747681e-05, 'epoch': 0.67}

  7%|▋         | 2781/41250 [6:42:36<93:00:57,  8.70s/it][2025-04-25 14:40:19,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:40:19,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.34 | bwd_microstep: 5723.57 | bwd_inner_microstep: 5680.97 | bwd_allreduce_microstep: 42.54 | step_microstep: 18.84
[2025-04-25 14:40:19,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.34 | bwd: 5723.58 | bwd_inner: 5680.97 | bwd_allreduce: 42.56 | step: 18.84

  7%|▋         | 2782/41250 [6:42:44<92:49:46,  8.69s/it]
                                                         
{'loss': 0.1074, 'grad_norm': 1.4256144762039185, 'learning_rate': 3.985321493549693e-05, 'epoch': 0.67}

  7%|▋         | 2782/41250 [6:42:44<92:49:46,  8.69s/it][2025-04-25 14:40:28,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 14:40:28,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.29 | bwd_microstep: 5779.97 | bwd_inner_microstep: 5767.25 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.75
[2025-04-25 14:40:28,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.29 | bwd: 5779.98 | bwd_inner: 5767.25 | bwd_allreduce: 12.69 | step: 18.76

  7%|▋         | 2783/41250 [6:42:53<93:00:24,  8.70s/it]
                                                         
{'loss': 0.1918, 'grad_norm': 2.0071372985839844, 'learning_rate': 3.985302497112589e-05, 'epoch': 0.67}

  7%|▋         | 2783/41250 [6:42:53<93:00:24,  8.70s/it][2025-04-25 14:40:36,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 14:40:36,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.21 | bwd_microstep: 5684.42 | bwd_inner_microstep: 5640.56 | bwd_allreduce_microstep: 43.81 | step_microstep: 18.92
[2025-04-25 14:40:36,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.21 | bwd: 5684.43 | bwd_inner: 5640.56 | bwd_allreduce: 43.83 | step: 18.92

  7%|▋         | 2784/41250 [6:43:01<92:37:14,  8.67s/it]
                                                         
{'loss': 0.1477, 'grad_norm': 1.2912324666976929, 'learning_rate': 3.985283488436485e-05, 'epoch': 0.67}

  7%|▋         | 2784/41250 [6:43:01<92:37:14,  8.67s/it][2025-04-25 14:40:45,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:40:45,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.94 | bwd_microstep: 5715.22 | bwd_inner_microstep: 5702.46 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.73
[2025-04-25 14:40:45,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.94 | bwd: 5715.24 | bwd_inner: 5702.46 | bwd_allreduce: 12.74 | step: 18.73

  7%|▋         | 2785/41250 [6:43:10<92:33:29,  8.66s/it]
                                                         
{'loss': 0.2932, 'grad_norm': 1.9899871349334717, 'learning_rate': 3.985264467521499e-05, 'epoch': 0.68}

  7%|▋         | 2785/41250 [6:43:10<92:33:29,  8.66s/it][2025-04-25 14:40:53,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 14:40:53,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.30 | bwd_microstep: 5692.55 | bwd_inner_microstep: 5679.86 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.10
[2025-04-25 14:40:53,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.30 | bwd: 5692.56 | bwd_inner: 5679.86 | bwd_allreduce: 12.67 | step: 19.10

  7%|▋         | 2786/41250 [6:43:19<92:25:16,  8.65s/it]
                                                         
{'loss': 0.0963, 'grad_norm': 0.9820659160614014, 'learning_rate': 3.9852454343677485e-05, 'epoch': 0.68}

  7%|▋         | 2786/41250 [6:43:19<92:25:16,  8.65s/it][2025-04-25 14:41:02,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:41:02,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.09 | bwd_microstep: 5677.12 | bwd_inner_microstep: 5663.22 | bwd_allreduce_microstep: 13.85 | step_microstep: 18.78
[2025-04-25 14:41:02,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.09 | bwd: 5677.13 | bwd_inner: 5663.22 | bwd_allreduce: 13.86 | step: 18.79

  7%|▋         | 2787/41250 [6:43:27<92:14:40,  8.63s/it]
                                                         
{'loss': 0.0949, 'grad_norm': 0.8311523199081421, 'learning_rate': 3.98522638897535e-05, 'epoch': 0.68}

  7%|▋         | 2787/41250 [6:43:27<92:14:40,  8.63s/it][2025-04-25 14:41:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:41:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.61 | bwd_microstep: 5669.12 | bwd_inner_microstep: 5636.88 | bwd_allreduce_microstep: 32.19 | step_microstep: 18.82
[2025-04-25 14:41:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.61 | bwd: 5669.13 | bwd_inner: 5636.88 | bwd_allreduce: 32.21 | step: 18.83

  7%|▋         | 2788/41250 [6:43:36<92:03:36,  8.62s/it]
                                                         
{'loss': 0.1697, 'grad_norm': 1.2606066465377808, 'learning_rate': 3.9852073313444214e-05, 'epoch': 0.68}

  7%|▋         | 2788/41250 [6:43:36<92:03:36,  8.62s/it][2025-04-25 14:41:19,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:41:19,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.84 | bwd_microstep: 5688.60 | bwd_inner_microstep: 5675.85 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.72
[2025-04-25 14:41:19,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5688.61 | bwd_inner: 5675.85 | bwd_allreduce: 12.72 | step: 18.72

  7%|▋         | 2789/41250 [6:43:45<92:02:42,  8.62s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 2.3131375312805176, 'learning_rate': 3.98518826147508e-05, 'epoch': 0.68}

  7%|▋         | 2789/41250 [6:43:45<92:02:42,  8.62s/it][2025-04-25 14:41:28,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:41:28,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5675.33 | bwd_inner_microstep: 5645.96 | bwd_allreduce_microstep: 29.32 | step_microstep: 18.96
[2025-04-25 14:41:28,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5675.34 | bwd_inner: 5645.96 | bwd_allreduce: 29.34 | step: 18.96

  7%|▋         | 2790/41250 [6:43:53<91:56:59,  8.61s/it]
                                                         
{'loss': 0.0763, 'grad_norm': 0.7009345889091492, 'learning_rate': 3.985169179367444e-05, 'epoch': 0.68}

  7%|▋         | 2790/41250 [6:43:53<91:56:59,  8.61s/it][2025-04-25 14:41:36,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-25 14:41:36,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.08 | bwd_microstep: 5702.77 | bwd_inner_microstep: 5638.99 | bwd_allreduce_microstep: 63.72 | step_microstep: 19.04
[2025-04-25 14:41:36,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.08 | bwd: 5702.79 | bwd_inner: 5638.99 | bwd_allreduce: 63.74 | step: 19.04

  7%|▋         | 2791/41250 [6:44:02<91:56:23,  8.61s/it]
                                                         
{'loss': 0.0335, 'grad_norm': 0.5097163319587708, 'learning_rate': 3.9851500850216296e-05, 'epoch': 0.68}

  7%|▋         | 2791/41250 [6:44:02<91:56:23,  8.61s/it][2025-04-25 14:41:45,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:41:45,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.16 | bwd_microstep: 5723.35 | bwd_inner_microstep: 5710.68 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.94
[2025-04-25 14:41:45,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.16 | bwd: 5723.36 | bwd_inner: 5710.68 | bwd_allreduce: 12.64 | step: 18.95

  7%|▋         | 2792/41250 [6:44:10<92:07:34,  8.62s/it]
                                                         
{'loss': 0.2218, 'grad_norm': 1.6628856658935547, 'learning_rate': 3.985130978437756e-05, 'epoch': 0.68}

  7%|▋         | 2792/41250 [6:44:10<92:07:34,  8.62s/it][2025-04-25 14:41:54,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 14:41:54,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5744.12 | bwd_inner_microstep: 5705.74 | bwd_allreduce_microstep: 38.33 | step_microstep: 19.17
[2025-04-25 14:41:54,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5744.13 | bwd_inner: 5705.74 | bwd_allreduce: 38.35 | step: 19.17

  7%|▋         | 2793/41250 [6:44:19<92:18:45,  8.64s/it]
                                                         
{'loss': 0.1781, 'grad_norm': 1.6429498195648193, 'learning_rate': 3.98511185961594e-05, 'epoch': 0.68}

  7%|▋         | 2793/41250 [6:44:19<92:18:45,  8.64s/it][2025-04-25 14:42:03,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 14:42:03,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5887.92 | bwd_inner_microstep: 5646.27 | bwd_allreduce_microstep: 241.59 | step_microstep: 18.97
[2025-04-25 14:42:03,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5887.94 | bwd_inner: 5646.27 | bwd_allreduce: 241.62 | step: 18.97

  7%|▋         | 2794/41250 [6:44:28<92:47:58,  8.69s/it]
                                                         
{'loss': 0.1624, 'grad_norm': 1.1907634735107422, 'learning_rate': 3.9850927285563e-05, 'epoch': 0.68}

  7%|▋         | 2794/41250 [6:44:28<92:47:58,  8.69s/it][2025-04-25 14:42:11,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 14:42:11,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.94 | bwd_microstep: 5761.28 | bwd_inner_microstep: 5652.67 | bwd_allreduce_microstep: 108.56 | step_microstep: 18.82
[2025-04-25 14:42:11,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.94 | bwd: 5761.29 | bwd_inner: 5652.67 | bwd_allreduce: 108.58 | step: 18.83

  7%|▋         | 2795/41250 [6:44:37<92:44:44,  8.68s/it]
                                                         
{'loss': 0.1582, 'grad_norm': 1.304907202720642, 'learning_rate': 3.9850735852589534e-05, 'epoch': 0.68}

  7%|▋         | 2795/41250 [6:44:37<92:44:44,  8.68s/it][2025-04-25 14:42:20,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 14:42:20,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.49 | bwd_microstep: 5748.96 | bwd_inner_microstep: 5659.62 | bwd_allreduce_microstep: 89.31 | step_microstep: 18.84
[2025-04-25 14:42:20,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.49 | bwd: 5748.98 | bwd_inner: 5659.62 | bwd_allreduce: 89.32 | step: 18.85

  7%|▋         | 2796/41250 [6:44:45<92:41:51,  8.68s/it]
                                                         
{'loss': 0.0522, 'grad_norm': 1.0539839267730713, 'learning_rate': 3.9850544297240186e-05, 'epoch': 0.68}

  7%|▋         | 2796/41250 [6:44:45<92:41:51,  8.68s/it][2025-04-25 14:42:29,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 14:42:29,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.28 | bwd_microstep: 5766.98 | bwd_inner_microstep: 5661.67 | bwd_allreduce_microstep: 105.27 | step_microstep: 18.44
[2025-04-25 14:42:29,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.28 | bwd: 5766.99 | bwd_inner: 5661.67 | bwd_allreduce: 105.28 | step: 18.45

  7%|▋         | 2797/41250 [6:44:54<92:41:37,  8.68s/it]
                                                         
{'loss': 0.1628, 'grad_norm': 1.1400264501571655, 'learning_rate': 3.9850352619516146e-05, 'epoch': 0.68}

  7%|▋         | 2797/41250 [6:44:54<92:41:37,  8.68s/it][2025-04-25 14:42:37,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:42:37,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.45 | bwd_microstep: 5727.00 | bwd_inner_microstep: 5693.08 | bwd_allreduce_microstep: 33.88 | step_microstep: 18.81
[2025-04-25 14:42:37,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.45 | bwd: 5727.02 | bwd_inner: 5693.08 | bwd_allreduce: 33.90 | step: 18.81

  7%|▋         | 2798/41250 [6:45:03<92:38:45,  8.67s/it]
                                                         
{'loss': 0.0594, 'grad_norm': 0.5577142834663391, 'learning_rate': 3.985016081941858e-05, 'epoch': 0.68}

  7%|▋         | 2798/41250 [6:45:03<92:38:45,  8.67s/it][2025-04-25 14:42:46,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:42:46,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.71 | bwd_microstep: 5755.31 | bwd_inner_microstep: 5667.72 | bwd_allreduce_microstep: 87.55 | step_microstep: 18.31
[2025-04-25 14:42:46,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.71 | bwd: 5755.33 | bwd_inner: 5667.72 | bwd_allreduce: 87.57 | step: 18.32

  7%|▋         | 2799/41250 [6:45:11<92:37:15,  8.67s/it]
                                                         
{'loss': 0.169, 'grad_norm': 2.3866872787475586, 'learning_rate': 3.984996889694867e-05, 'epoch': 0.68}

  7%|▋         | 2799/41250 [6:45:11<92:37:15,  8.67s/it][2025-04-25 14:42:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:42:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5702.26 | bwd_inner_microstep: 5689.45 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.75
[2025-04-25 14:42:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5702.27 | bwd_inner: 5689.45 | bwd_allreduce: 12.78 | step: 18.75

  7%|▋         | 2800/41250 [6:45:20<92:30:04,  8.66s/it]
                                                         
{'loss': 0.2008, 'grad_norm': 1.71660315990448, 'learning_rate': 3.984977685210762e-05, 'epoch': 0.68}

  7%|▋         | 2800/41250 [6:45:20<92:30:04,  8.66s/it][2025-04-25 14:43:03,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:43:03,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.79 | bwd_microstep: 5828.91 | bwd_inner_microstep: 5647.24 | bwd_allreduce_microstep: 181.63 | step_microstep: 18.22
[2025-04-25 14:43:03,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.79 | bwd: 5828.93 | bwd_inner: 5647.24 | bwd_allreduce: 181.64 | step: 18.22

  7%|▋         | 2801/41250 [6:45:29<92:43:41,  8.68s/it]
                                                         
{'loss': 0.2729, 'grad_norm': 2.416877031326294, 'learning_rate': 3.9849584684896586e-05, 'epoch': 0.68}

  7%|▋         | 2801/41250 [6:45:29<92:43:41,  8.68s/it][2025-04-25 14:43:12,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 14:43:12,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.44 | bwd_microstep: 5781.36 | bwd_inner_microstep: 5666.04 | bwd_allreduce_microstep: 115.28 | step_microstep: 19.00
[2025-04-25 14:43:12,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.44 | bwd: 5781.37 | bwd_inner: 5666.04 | bwd_allreduce: 115.29 | step: 19.01

  7%|▋         | 2802/41250 [6:45:37<92:48:22,  8.69s/it]
                                                         
{'loss': 0.154, 'grad_norm': 1.694880485534668, 'learning_rate': 3.9849392395316785e-05, 'epoch': 0.68}

  7%|▋         | 2802/41250 [6:45:37<92:48:22,  8.69s/it][2025-04-25 14:43:21,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.80 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:43:21,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.70 | bwd_microstep: 5687.37 | bwd_inner_microstep: 5659.87 | bwd_allreduce_microstep: 27.45 | step_microstep: 18.15
[2025-04-25 14:43:21,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.70 | bwd: 5687.38 | bwd_inner: 5659.87 | bwd_allreduce: 27.47 | step: 18.15

  7%|▋         | 2803/41250 [6:45:46<92:32:50,  8.67s/it]
                                                         
{'loss': 0.1348, 'grad_norm': 1.5019491910934448, 'learning_rate': 3.9849199983369367e-05, 'epoch': 0.68}

  7%|▋         | 2803/41250 [6:45:46<92:32:50,  8.67s/it][2025-04-25 14:43:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-25 14:43:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.01 | bwd_microstep: 5745.62 | bwd_inner_microstep: 5713.86 | bwd_allreduce_microstep: 31.69 | step_microstep: 19.32
[2025-04-25 14:43:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.01 | bwd: 5745.64 | bwd_inner: 5713.86 | bwd_allreduce: 31.72 | step: 19.32

  7%|▋         | 2804/41250 [6:45:55<92:35:17,  8.67s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.143344759941101, 'learning_rate': 3.9849007449055544e-05, 'epoch': 0.68}

  7%|▋         | 2804/41250 [6:45:55<92:35:17,  8.67s/it][2025-04-25 14:43:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.09 | optimizer_step: 1.05
[2025-04-25 14:43:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.89 | bwd_microstep: 5696.60 | bwd_inner_microstep: 5652.19 | bwd_allreduce_microstep: 44.37 | step_microstep: 19.07
[2025-04-25 14:43:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.89 | bwd: 5696.62 | bwd_inner: 5652.19 | bwd_allreduce: 44.38 | step: 19.07

  7%|▋         | 2805/41250 [6:46:03<92:24:02,  8.65s/it]
                                                         
{'loss': 0.0637, 'grad_norm': 0.6335078477859497, 'learning_rate': 3.984881479237649e-05, 'epoch': 0.68}

  7%|▋         | 2805/41250 [6:46:03<92:24:02,  8.65s/it][2025-04-25 14:43:47,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:43:47,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.71 | bwd_microstep: 5698.35 | bwd_inner_microstep: 5664.40 | bwd_allreduce_microstep: 33.91 | step_microstep: 18.80
[2025-04-25 14:43:47,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.71 | bwd: 5698.37 | bwd_inner: 5664.40 | bwd_allreduce: 33.93 | step: 18.80

  7%|▋         | 2806/41250 [6:46:12<92:18:56,  8.64s/it]
                                                         
{'loss': 0.1202, 'grad_norm': 1.221228003501892, 'learning_rate': 3.984862201333339e-05, 'epoch': 0.68}

  7%|▋         | 2806/41250 [6:46:12<92:18:56,  8.64s/it][2025-04-25 14:43:55,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 14:43:55,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.83 | bwd_microstep: 5767.20 | bwd_inner_microstep: 5710.60 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.51
[2025-04-25 14:43:55,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.83 | bwd: 5767.21 | bwd_inner: 5710.60 | bwd_allreduce: 56.57 | step: 18.51

  7%|▋         | 2807/41250 [6:46:21<92:30:42,  8.66s/it]
                                                         
{'loss': 0.2887, 'grad_norm': 2.6525931358337402, 'learning_rate': 3.984842911192745e-05, 'epoch': 0.68}

  7%|▋         | 2807/41250 [6:46:21<92:30:42,  8.66s/it][2025-04-25 14:44:04,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 14:44:04,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.06 | bwd_microstep: 5803.87 | bwd_inner_microstep: 5791.14 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.52
[2025-04-25 14:44:04,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.07 | bwd: 5803.89 | bwd_inner: 5791.14 | bwd_allreduce: 12.70 | step: 18.53

  7%|▋         | 2808/41250 [6:46:29<92:53:11,  8.70s/it]
                                                         
{'loss': 0.1141, 'grad_norm': 1.227823257446289, 'learning_rate': 3.984823608815985e-05, 'epoch': 0.68}

  7%|▋         | 2808/41250 [6:46:29<92:53:11,  8.70s/it][2025-04-25 14:44:13,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:44:13,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.38 | bwd_microstep: 5787.00 | bwd_inner_microstep: 5689.30 | bwd_allreduce_microstep: 97.65 | step_microstep: 18.71
[2025-04-25 14:44:13,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.38 | bwd: 5787.02 | bwd_inner: 5689.30 | bwd_allreduce: 97.67 | step: 18.71

  7%|▋         | 2809/41250 [6:46:38<92:56:40,  8.70s/it]
                                                         
{'loss': 0.1428, 'grad_norm': 1.5773802995681763, 'learning_rate': 3.9848042942031775e-05, 'epoch': 0.68}

  7%|▋         | 2809/41250 [6:46:38<92:56:40,  8.70s/it][2025-04-25 14:44:21,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.25 | optimizer_step: 0.98
[2025-04-25 14:44:21,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.60 | bwd_microstep: 5758.53 | bwd_inner_microstep: 5671.96 | bwd_allreduce_microstep: 86.52 | step_microstep: 19.36
[2025-04-25 14:44:21,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.60 | bwd: 5758.55 | bwd_inner: 5671.96 | bwd_allreduce: 86.55 | step: 19.36

  7%|▋         | 2810/41250 [6:46:47<92:51:05,  8.70s/it]
                                                         
{'loss': 0.1498, 'grad_norm': 1.5070381164550781, 'learning_rate': 3.9847849673544413e-05, 'epoch': 0.68}

  7%|▋         | 2810/41250 [6:46:47<92:51:05,  8.70s/it][2025-04-25 14:44:30,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.84 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 14:44:30,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5753.73 | bwd_inner_microstep: 5704.36 | bwd_allreduce_microstep: 49.32 | step_microstep: 18.61
[2025-04-25 14:44:30,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5753.74 | bwd_inner: 5704.36 | bwd_allreduce: 49.34 | step: 18.61

  7%|▋         | 2811/41250 [6:46:55<92:50:43,  8.70s/it]
                                                         
{'loss': 0.3047, 'grad_norm': 2.3221893310546875, 'learning_rate': 3.9847656282698975e-05, 'epoch': 0.68}

  7%|▋         | 2811/41250 [6:46:55<92:50:43,  8.70s/it][2025-04-25 14:44:39,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 14:44:39,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.91 | bwd_microstep: 5861.29 | bwd_inner_microstep: 5716.64 | bwd_allreduce_microstep: 144.41 | step_microstep: 18.83
[2025-04-25 14:44:39,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.91 | bwd: 5861.11 | bwd_inner: 5716.64 | bwd_allreduce: 144.43 | step: 18.84

  7%|▋         | 2812/41250 [6:47:04<93:11:05,  8.73s/it]
                                                         
{'loss': 0.1512, 'grad_norm': 1.7022202014923096, 'learning_rate': 3.984746276949663e-05, 'epoch': 0.68}

  7%|▋         | 2812/41250 [6:47:04<93:11:05,  8.73s/it][2025-04-25 14:44:48,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 14:44:48,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.96 | bwd_microstep: 5730.28 | bwd_inner_microstep: 5717.38 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.97
[2025-04-25 14:44:48,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.97 | bwd: 5730.30 | bwd_inner: 5717.38 | bwd_allreduce: 12.87 | step: 18.98

  7%|▋         | 2813/41250 [6:47:13<93:01:03,  8.71s/it]
                                                         
{'loss': 0.2437, 'grad_norm': 2.312983274459839, 'learning_rate': 3.984726913393858e-05, 'epoch': 0.68}

  7%|▋         | 2813/41250 [6:47:13<93:01:03,  8.71s/it][2025-04-25 14:44:56,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:44:56,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.34 | bwd_microstep: 5780.75 | bwd_inner_microstep: 5767.85 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.92
[2025-04-25 14:44:56,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.34 | bwd: 5780.77 | bwd_inner: 5767.85 | bwd_allreduce: 12.88 | step: 18.92

  7%|▋         | 2814/41250 [6:47:22<93:08:08,  8.72s/it]
                                                         
{'loss': 0.1358, 'grad_norm': 2.195505142211914, 'learning_rate': 3.984707537602602e-05, 'epoch': 0.68}

  7%|▋         | 2814/41250 [6:47:22<93:08:08,  8.72s/it][2025-04-25 14:45:05,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-25 14:45:05,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.66 | bwd_microstep: 5710.02 | bwd_inner_microstep: 5697.03 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.22
[2025-04-25 14:45:05,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.66 | bwd: 5710.04 | bwd_inner: 5697.04 | bwd_allreduce: 12.96 | step: 19.23

  7%|▋         | 2815/41250 [6:47:30<92:52:55,  8.70s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 0.8116154074668884, 'learning_rate': 3.9846881495760145e-05, 'epoch': 0.68}

  7%|▋         | 2815/41250 [6:47:30<92:52:55,  8.70s/it][2025-04-25 14:45:14,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:45:14,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.34 | bwd_microstep: 5769.64 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 68.44 | step_microstep: 18.96
[2025-04-25 14:45:14,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.34 | bwd: 5769.65 | bwd_inner: 5701.15 | bwd_allreduce: 68.46 | step: 18.97

  7%|▋         | 2816/41250 [6:47:39<92:52:59,  8.70s/it]
                                                         
{'loss': 0.3309, 'grad_norm': 1.9381389617919922, 'learning_rate': 3.984668749314216e-05, 'epoch': 0.68}

  7%|▋         | 2816/41250 [6:47:39<92:52:59,  8.70s/it][2025-04-25 14:45:22,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:45:22,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.10 | bwd_microstep: 5726.35 | bwd_inner_microstep: 5705.58 | bwd_allreduce_microstep: 20.72 | step_microstep: 18.82
[2025-04-25 14:45:22,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.10 | bwd: 5726.36 | bwd_inner: 5705.58 | bwd_allreduce: 20.74 | step: 18.82

  7%|▋         | 2817/41250 [6:47:48<92:47:06,  8.69s/it]
                                                         
{'loss': 0.2803, 'grad_norm': 14.635138511657715, 'learning_rate': 3.984649336817324e-05, 'epoch': 0.68}

  7%|▋         | 2817/41250 [6:47:48<92:47:06,  8.69s/it][2025-04-25 14:45:31,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.27 | optimizer_step: 1.04
[2025-04-25 14:45:31,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.54 | bwd_microstep: 5710.21 | bwd_inner_microstep: 5696.70 | bwd_allreduce_microstep: 13.46 | step_microstep: 19.82
[2025-04-25 14:45:31,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.54 | bwd: 5710.23 | bwd_inner: 5696.70 | bwd_allreduce: 13.48 | step: 19.82

  7%|▋         | 2818/41250 [6:47:56<92:42:32,  8.68s/it]
                                                         
{'loss': 0.4664, 'grad_norm': 3.139453172683716, 'learning_rate': 3.984629912085459e-05, 'epoch': 0.68}

  7%|▋         | 2818/41250 [6:47:56<92:42:32,  8.68s/it][2025-04-25 14:45:40,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 14:45:40,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5721.46 | bwd_inner_microstep: 5654.67 | bwd_allreduce_microstep: 66.74 | step_microstep: 18.86
[2025-04-25 14:45:40,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5721.48 | bwd_inner: 5654.67 | bwd_allreduce: 66.76 | step: 18.86

  7%|▋         | 2819/41250 [6:48:05<92:31:45,  8.67s/it]
                                                         
{'loss': 0.1371, 'grad_norm': 0.8237631320953369, 'learning_rate': 3.984610475118741e-05, 'epoch': 0.68}

  7%|▋         | 2819/41250 [6:48:05<92:31:45,  8.67s/it][2025-04-25 14:45:48,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:45:48,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5700.95 | bwd_inner_microstep: 5688.20 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.68
[2025-04-25 14:45:48,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5700.96 | bwd_inner: 5688.20 | bwd_allreduce: 12.72 | step: 18.68

  7%|▋         | 2820/41250 [6:48:14<92:24:26,  8.66s/it]
                                                         
{'loss': 0.1753, 'grad_norm': 1.1204419136047363, 'learning_rate': 3.9845910259172896e-05, 'epoch': 0.68}

  7%|▋         | 2820/41250 [6:48:14<92:24:26,  8.66s/it][2025-04-25 14:45:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:45:57,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.41 | bwd_microstep: 5687.08 | bwd_inner_microstep: 5665.26 | bwd_allreduce_microstep: 21.78 | step_microstep: 18.50
[2025-04-25 14:45:57,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.41 | bwd: 5687.10 | bwd_inner: 5665.26 | bwd_allreduce: 21.80 | step: 18.51

  7%|▋         | 2821/41250 [6:48:22<92:12:47,  8.64s/it]
                                                         
{'loss': 0.1676, 'grad_norm': 1.5561223030090332, 'learning_rate': 3.984571564481225e-05, 'epoch': 0.68}

  7%|▋         | 2821/41250 [6:48:22<92:12:47,  8.64s/it][2025-04-25 14:46:06,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.08 | optimizer_step: 1.00
[2025-04-25 14:46:06,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.03 | bwd_microstep: 5749.62 | bwd_inner_microstep: 5657.89 | bwd_allreduce_microstep: 91.66 | step_microstep: 19.33
[2025-04-25 14:46:06,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.03 | bwd: 5749.63 | bwd_inner: 5657.90 | bwd_allreduce: 91.69 | step: 19.34

  7%|▋         | 2822/41250 [6:48:31<92:15:42,  8.64s/it]
                                                         
{'loss': 0.1469, 'grad_norm': 2.0716640949249268, 'learning_rate': 3.9845520908106684e-05, 'epoch': 0.68}

  7%|▋         | 2822/41250 [6:48:31<92:15:42,  8.64s/it][2025-04-25 14:46:14,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 14:46:14,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.81 | bwd_microstep: 5685.53 | bwd_inner_microstep: 5656.04 | bwd_allreduce_microstep: 29.44 | step_microstep: 19.11
[2025-04-25 14:46:14,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.81 | bwd: 5685.54 | bwd_inner: 5656.04 | bwd_allreduce: 29.46 | step: 19.11

  7%|▋         | 2823/41250 [6:48:39<92:07:52,  8.63s/it]
                                                         
{'loss': 0.2279, 'grad_norm': 2.1716768741607666, 'learning_rate': 3.984532604905737e-05, 'epoch': 0.68}

  7%|▋         | 2823/41250 [6:48:39<92:07:52,  8.63s/it][2025-04-25 14:46:23,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:46:23,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.88 | bwd_microstep: 5733.42 | bwd_inner_microstep: 5697.09 | bwd_allreduce_microstep: 36.28 | step_microstep: 19.02
[2025-04-25 14:46:23,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.88 | bwd: 5733.44 | bwd_inner: 5697.09 | bwd_allreduce: 36.30 | step: 19.02

  7%|▋         | 2824/41250 [6:48:48<92:14:39,  8.64s/it]
                                                         
{'loss': 0.1922, 'grad_norm': 1.4783252477645874, 'learning_rate': 3.984513106766553e-05, 'epoch': 0.68}

  7%|▋         | 2824/41250 [6:48:48<92:14:39,  8.64s/it][2025-04-25 14:46:32,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:46:32,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.87 | bwd_microstep: 6062.12 | bwd_inner_microstep: 5699.78 | bwd_allreduce_microstep: 362.30 | step_microstep: 18.89
[2025-04-25 14:46:32,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.87 | bwd: 6062.13 | bwd_inner: 5699.78 | bwd_allreduce: 362.31 | step: 18.89

  7%|▋         | 2825/41250 [6:48:57<93:21:50,  8.75s/it]
                                                         
{'loss': 0.2342, 'grad_norm': 2.4718000888824463, 'learning_rate': 3.9844935963932355e-05, 'epoch': 0.68}

  7%|▋         | 2825/41250 [6:48:57<93:21:50,  8.75s/it][2025-04-25 14:46:40,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:46:40,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.28 | bwd_microstep: 5711.32 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.85
[2025-04-25 14:46:40,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.28 | bwd: 5711.33 | bwd_inner: 5698.45 | bwd_allreduce: 12.84 | step: 18.85

  7%|▋         | 2826/41250 [6:49:06<93:06:21,  8.72s/it]
                                                         
{'loss': 0.1153, 'grad_norm': 0.9483859539031982, 'learning_rate': 3.984474073785906e-05, 'epoch': 0.69}

  7%|▋         | 2826/41250 [6:49:06<93:06:21,  8.72s/it][2025-04-25 14:46:49,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:46:49,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.55 | bwd_microstep: 5719.30 | bwd_inner_microstep: 5706.41 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.78
[2025-04-25 14:46:49,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.55 | bwd: 5719.31 | bwd_inner: 5706.41 | bwd_allreduce: 12.85 | step: 18.79

  7%|▋         | 2827/41250 [6:49:14<92:52:57,  8.70s/it]
                                                         
{'loss': 0.4727, 'grad_norm': 5.624706745147705, 'learning_rate': 3.984454538944684e-05, 'epoch': 0.69}

  7%|▋         | 2827/41250 [6:49:14<92:52:57,  8.70s/it][2025-04-25 14:46:58,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.19 | optimizer_step: 1.01
[2025-04-25 14:46:58,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5722.30 | bwd_inner_microstep: 5680.58 | bwd_allreduce_microstep: 41.67 | step_microstep: 19.35
[2025-04-25 14:46:58,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5722.31 | bwd_inner: 5680.58 | bwd_allreduce: 41.69 | step: 19.35

  7%|▋         | 2828/41250 [6:49:23<92:43:08,  8.69s/it]
                                                         
{'loss': 0.0491, 'grad_norm': 0.33110249042510986, 'learning_rate': 3.9844349918696896e-05, 'epoch': 0.69}

  7%|▋         | 2828/41250 [6:49:23<92:43:08,  8.69s/it][2025-04-25 14:47:06,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:47:06,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.86 | bwd_microstep: 5690.59 | bwd_inner_microstep: 5677.79 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.93
[2025-04-25 14:47:06,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.86 | bwd: 5690.61 | bwd_inner: 5677.79 | bwd_allreduce: 12.78 | step: 18.93

  7%|▋         | 2829/41250 [6:49:32<92:29:12,  8.67s/it]
                                                         
{'loss': 0.1351, 'grad_norm': 1.1803282499313354, 'learning_rate': 3.9844154325610444e-05, 'epoch': 0.69}

  7%|▋         | 2829/41250 [6:49:32<92:29:12,  8.67s/it][2025-04-25 14:47:15,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 14:47:15,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.78 | bwd_microstep: 5691.95 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 45.44 | step_microstep: 19.29
[2025-04-25 14:47:15,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.78 | bwd: 5691.97 | bwd_inner: 5646.46 | bwd_allreduce: 45.46 | step: 19.29

  7%|▋         | 2830/41250 [6:49:40<92:15:46,  8.65s/it]
                                                         
{'loss': 0.2473, 'grad_norm': 1.5268588066101074, 'learning_rate': 3.9843958610188676e-05, 'epoch': 0.69}

  7%|▋         | 2830/41250 [6:49:40<92:15:46,  8.65s/it][2025-04-25 14:47:24,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 14:47:24,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.17 | bwd_microstep: 5761.27 | bwd_inner_microstep: 5650.08 | bwd_allreduce_microstep: 111.14 | step_microstep: 18.47
[2025-04-25 14:47:24,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.17 | bwd: 5761.29 | bwd_inner: 5650.08 | bwd_allreduce: 111.16 | step: 18.47

  7%|▋         | 2831/41250 [6:49:49<92:19:52,  8.65s/it]
                                                         
{'loss': 0.254, 'grad_norm': 1.3724898099899292, 'learning_rate': 3.984376277243281e-05, 'epoch': 0.69}

  7%|▋         | 2831/41250 [6:49:49<92:19:52,  8.65s/it][2025-04-25 14:47:32,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:47:32,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.20 | bwd_microstep: 5687.34 | bwd_inner_microstep: 5651.79 | bwd_allreduce_microstep: 35.51 | step_microstep: 18.72
[2025-04-25 14:47:32,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.20 | bwd: 5687.35 | bwd_inner: 5651.79 | bwd_allreduce: 35.52 | step: 18.72

  7%|▋         | 2832/41250 [6:49:58<92:09:57,  8.64s/it]
                                                         
{'loss': 0.1693, 'grad_norm': 1.1339062452316284, 'learning_rate': 3.9843566812344054e-05, 'epoch': 0.69}

  7%|▋         | 2832/41250 [6:49:58<92:09:57,  8.64s/it][2025-04-25 14:47:41,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.20 | optimizer_step: 1.05
[2025-04-25 14:47:41,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.06 | bwd_microstep: 5738.57 | bwd_inner_microstep: 5704.06 | bwd_allreduce_microstep: 34.46 | step_microstep: 19.72
[2025-04-25 14:47:41,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.06 | bwd: 5738.58 | bwd_inner: 5704.06 | bwd_allreduce: 34.48 | step: 19.72

  7%|▋         | 2833/41250 [6:50:06<92:17:02,  8.65s/it]
                                                         
{'loss': 0.1424, 'grad_norm': 1.2354239225387573, 'learning_rate': 3.984337072992361e-05, 'epoch': 0.69}

  7%|▋         | 2833/41250 [6:50:06<92:17:02,  8.65s/it][2025-04-25 14:47:50,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:47:50,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.93 | bwd_microstep: 5705.50 | bwd_inner_microstep: 5687.41 | bwd_allreduce_microstep: 18.04 | step_microstep: 18.64
[2025-04-25 14:47:50,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.93 | bwd: 5705.52 | bwd_inner: 5687.41 | bwd_allreduce: 18.06 | step: 18.65

  7%|▋         | 2834/41250 [6:50:15<92:17:17,  8.65s/it]
                                                         
{'loss': 0.2315, 'grad_norm': 1.1160027980804443, 'learning_rate': 3.9843174525172686e-05, 'epoch': 0.69}

  7%|▋         | 2834/41250 [6:50:15<92:17:17,  8.65s/it][2025-04-25 14:47:58,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:47:58,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.32 | bwd_microstep: 5710.88 | bwd_inner_microstep: 5697.81 | bwd_allreduce_microstep: 13.02 | step_microstep: 19.13
[2025-04-25 14:47:58,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.32 | bwd: 5710.90 | bwd_inner: 5697.81 | bwd_allreduce: 13.04 | step: 19.13

  7%|▋         | 2835/41250 [6:50:24<92:16:43,  8.65s/it]
                                                         
{'loss': 0.1473, 'grad_norm': 2.9880809783935547, 'learning_rate': 3.9842978198092496e-05, 'epoch': 0.69}

  7%|▋         | 2835/41250 [6:50:24<92:16:43,  8.65s/it][2025-04-25 14:48:07,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 14:48:07,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.45 | bwd_microstep: 5719.67 | bwd_inner_microstep: 5650.43 | bwd_allreduce_microstep: 69.20 | step_microstep: 18.72
[2025-04-25 14:48:07,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.45 | bwd: 5719.69 | bwd_inner: 5650.43 | bwd_allreduce: 69.22 | step: 18.72

  7%|▋         | 2836/41250 [6:50:32<92:16:18,  8.65s/it]
                                                         
{'loss': 0.2353, 'grad_norm': 4.62299108505249, 'learning_rate': 3.984278174868425e-05, 'epoch': 0.69}

  7%|▋         | 2836/41250 [6:50:32<92:16:18,  8.65s/it][2025-04-25 14:48:15,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 14:48:15,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.93 | bwd_microstep: 5680.88 | bwd_inner_microstep: 5648.93 | bwd_allreduce_microstep: 31.91 | step_microstep: 18.60
[2025-04-25 14:48:15,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.93 | bwd: 5680.90 | bwd_inner: 5648.93 | bwd_allreduce: 31.93 | step: 18.60

  7%|▋         | 2837/41250 [6:50:41<92:04:48,  8.63s/it]
                                                         
{'loss': 0.0732, 'grad_norm': 0.7207311987876892, 'learning_rate': 3.9842585176949156e-05, 'epoch': 0.69}

  7%|▋         | 2837/41250 [6:50:41<92:04:48,  8.63s/it][2025-04-25 14:48:24,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:48:24,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.88 | bwd_microstep: 5738.79 | bwd_inner_microstep: 5684.83 | bwd_allreduce_microstep: 53.91 | step_microstep: 18.68
[2025-04-25 14:48:24,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.88 | bwd: 5738.80 | bwd_inner: 5684.83 | bwd_allreduce: 53.93 | step: 18.68

  7%|▋         | 2838/41250 [6:50:49<92:10:46,  8.64s/it]
                                                         
{'loss': 0.1804, 'grad_norm': 2.4428634643554688, 'learning_rate': 3.984238848288842e-05, 'epoch': 0.69}

  7%|▋         | 2838/41250 [6:50:49<92:10:46,  8.64s/it][2025-04-25 14:48:33,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:48:33,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.67 | bwd_microstep: 5752.38 | bwd_inner_microstep: 5647.23 | bwd_allreduce_microstep: 105.11 | step_microstep: 18.84
[2025-04-25 14:48:33,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.67 | bwd: 5752.39 | bwd_inner: 5647.23 | bwd_allreduce: 105.12 | step: 18.84

  7%|▋         | 2839/41250 [6:50:58<92:15:27,  8.65s/it]
                                                         
{'loss': 0.2771, 'grad_norm': 2.4185264110565186, 'learning_rate': 3.984219166650327e-05, 'epoch': 0.69}

  7%|▋         | 2839/41250 [6:50:58<92:15:27,  8.65s/it][2025-04-25 14:48:41,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 14:48:41,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.47 | bwd_microstep: 5733.77 | bwd_inner_microstep: 5643.53 | bwd_allreduce_microstep: 90.19 | step_microstep: 18.81
[2025-04-25 14:48:41,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.47 | bwd: 5733.78 | bwd_inner: 5643.53 | bwd_allreduce: 90.21 | step: 18.81

  7%|▋         | 2840/41250 [6:51:07<92:16:18,  8.65s/it]
                                                         
{'loss': 0.1455, 'grad_norm': 1.0006866455078125, 'learning_rate': 3.9841994727794906e-05, 'epoch': 0.69}

  7%|▋         | 2840/41250 [6:51:07<92:16:18,  8.65s/it][2025-04-25 14:48:50,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 14:48:50,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.43 | bwd_microstep: 5717.18 | bwd_inner_microstep: 5654.71 | bwd_allreduce_microstep: 62.42 | step_microstep: 18.70
[2025-04-25 14:48:50,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.43 | bwd: 5717.19 | bwd_inner: 5654.71 | bwd_allreduce: 62.44 | step: 18.70

  7%|▋         | 2841/41250 [6:51:15<92:14:04,  8.64s/it]
                                                         
{'loss': 0.102, 'grad_norm': 1.9716126918792725, 'learning_rate': 3.984179766676455e-05, 'epoch': 0.69}

  7%|▋         | 2841/41250 [6:51:15<92:14:04,  8.64s/it][2025-04-25 14:48:59,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 14:48:59,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.65 | bwd_microstep: 5736.24 | bwd_inner_microstep: 5691.92 | bwd_allreduce_microstep: 44.27 | step_microstep: 18.46
[2025-04-25 14:48:59,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.65 | bwd: 5736.25 | bwd_inner: 5691.92 | bwd_allreduce: 44.29 | step: 18.46

  7%|▋         | 2842/41250 [6:51:24<92:17:20,  8.65s/it]
                                                         
{'loss': 0.2701, 'grad_norm': 2.635775327682495, 'learning_rate': 3.9841600483413414e-05, 'epoch': 0.69}

  7%|▋         | 2842/41250 [6:51:24<92:17:20,  8.65s/it][2025-04-25 14:49:07,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 14:49:07,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.50 | bwd_microstep: 5737.86 | bwd_inner_microstep: 5638.47 | bwd_allreduce_microstep: 99.34 | step_microstep: 18.66
[2025-04-25 14:49:07,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.50 | bwd: 5737.87 | bwd_inner: 5638.47 | bwd_allreduce: 99.36 | step: 18.66

  7%|▋         | 2843/41250 [6:51:33<92:18:26,  8.65s/it]
                                                         
{'loss': 0.1469, 'grad_norm': 1.2333812713623047, 'learning_rate': 3.984140317774271e-05, 'epoch': 0.69}

  7%|▋         | 2843/41250 [6:51:33<92:18:26,  8.65s/it][2025-04-25 14:49:16,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:49:16,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.18 | bwd_microstep: 5722.67 | bwd_inner_microstep: 5646.36 | bwd_allreduce_microstep: 76.26 | step_microstep: 18.68
[2025-04-25 14:49:16,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.18 | bwd: 5722.68 | bwd_inner: 5646.36 | bwd_allreduce: 76.28 | step: 18.68

  7%|▋         | 2844/41250 [6:51:41<92:15:43,  8.65s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.7848726511001587, 'learning_rate': 3.984120574975366e-05, 'epoch': 0.69}

  7%|▋         | 2844/41250 [6:51:41<92:15:43,  8.65s/it][2025-04-25 14:49:25,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.28 | optimizer_step: 1.00
[2025-04-25 14:49:25,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.08 | bwd_microstep: 5654.94 | bwd_inner_microstep: 5633.97 | bwd_allreduce_microstep: 20.92 | step_microstep: 21.14
[2025-04-25 14:49:25,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.08 | bwd: 5654.95 | bwd_inner: 5633.97 | bwd_allreduce: 20.94 | step: 21.15

  7%|▋         | 2845/41250 [6:51:50<92:01:49,  8.63s/it]
                                                         
{'loss': 0.2952, 'grad_norm': 1.4675390720367432, 'learning_rate': 3.984100819944749e-05, 'epoch': 0.69}

  7%|▋         | 2845/41250 [6:51:50<92:01:49,  8.63s/it][2025-04-25 14:49:33,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:49:33,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.50 | bwd_microstep: 5688.27 | bwd_inner_microstep: 5675.47 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.35
[2025-04-25 14:49:33,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.50 | bwd: 5688.28 | bwd_inner: 5675.47 | bwd_allreduce: 12.77 | step: 18.35

  7%|▋         | 2846/41250 [6:51:59<91:59:09,  8.62s/it]
                                                         
{'loss': 0.14, 'grad_norm': 1.320219874382019, 'learning_rate': 3.9840810526825396e-05, 'epoch': 0.69}

  7%|▋         | 2846/41250 [6:51:59<91:59:09,  8.62s/it][2025-04-25 14:49:42,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:49:42,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.11 | bwd_microstep: 5734.16 | bwd_inner_microstep: 5643.42 | bwd_allreduce_microstep: 90.70 | step_microstep: 18.61
[2025-04-25 14:49:42,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.11 | bwd: 5734.17 | bwd_inner: 5643.42 | bwd_allreduce: 90.71 | step: 18.61

  7%|▋         | 2847/41250 [6:52:07<92:05:28,  8.63s/it]
                                                         
{'loss': 0.0931, 'grad_norm': 1.2691131830215454, 'learning_rate': 3.984061273188861e-05, 'epoch': 0.69}

  7%|▋         | 2847/41250 [6:52:07<92:05:28,  8.63s/it][2025-04-25 14:49:51,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 14:49:51,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.00 | bwd_microstep: 5736.61 | bwd_inner_microstep: 5689.51 | bwd_allreduce_microstep: 47.05 | step_microstep: 18.91
[2025-04-25 14:49:51,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.00 | bwd: 5736.62 | bwd_inner: 5689.51 | bwd_allreduce: 47.07 | step: 18.91

  7%|▋         | 2848/41250 [6:52:16<92:13:29,  8.65s/it]
                                                         
{'loss': 0.2063, 'grad_norm': 2.640296697616577, 'learning_rate': 3.9840414814638346e-05, 'epoch': 0.69}

  7%|▋         | 2848/41250 [6:52:16<92:13:29,  8.65s/it][2025-04-25 14:49:59,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:49:59,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.31 | bwd_microstep: 5758.48 | bwd_inner_microstep: 5675.67 | bwd_allreduce_microstep: 82.77 | step_microstep: 18.79
[2025-04-25 14:49:59,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.31 | bwd: 5758.49 | bwd_inner: 5675.67 | bwd_allreduce: 82.78 | step: 18.80

  7%|▋         | 2849/41250 [6:52:25<92:20:14,  8.66s/it]
                                                         
{'loss': 0.1819, 'grad_norm': 1.8241841793060303, 'learning_rate': 3.9840216775075834e-05, 'epoch': 0.69}

  7%|▋         | 2849/41250 [6:52:25<92:20:14,  8.66s/it][2025-04-25 14:50:08,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:50:08,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5749.47 | bwd_inner_microstep: 5646.68 | bwd_allreduce_microstep: 102.75 | step_microstep: 18.53
[2025-04-25 14:50:08,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5749.49 | bwd_inner: 5646.68 | bwd_allreduce: 102.77 | step: 18.53

  7%|▋         | 2850/41250 [6:52:33<92:20:05,  8.66s/it]
                                                         
{'loss': 0.235, 'grad_norm': 2.246034622192383, 'learning_rate': 3.984001861320228e-05, 'epoch': 0.69}

  7%|▋         | 2850/41250 [6:52:33<92:20:05,  8.66s/it][2025-04-25 14:50:17,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 14:50:17,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.75 | bwd_microstep: 5796.98 | bwd_inner_microstep: 5783.92 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.87
[2025-04-25 14:50:17,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.75 | bwd: 5797.00 | bwd_inner: 5783.92 | bwd_allreduce: 13.04 | step: 18.87

  7%|▋         | 2851/41250 [6:52:42<92:42:34,  8.69s/it]
                                                         
{'loss': 0.276, 'grad_norm': 1.4021292924880981, 'learning_rate': 3.9839820329018916e-05, 'epoch': 0.69}

  7%|▋         | 2851/41250 [6:52:42<92:42:34,  8.69s/it][2025-04-25 14:50:25,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:50:25,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.36 | bwd_microstep: 5734.37 | bwd_inner_microstep: 5644.85 | bwd_allreduce_microstep: 89.48 | step_microstep: 18.27
[2025-04-25 14:50:25,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.36 | bwd: 5734.38 | bwd_inner: 5644.85 | bwd_allreduce: 89.49 | step: 18.28

  7%|▋         | 2852/41250 [6:52:51<92:32:37,  8.68s/it]
                                                         
{'loss': 0.4233, 'grad_norm': 2.5422160625457764, 'learning_rate': 3.9839621922526956e-05, 'epoch': 0.69}

  7%|▋         | 2852/41250 [6:52:51<92:32:37,  8.68s/it][2025-04-25 14:50:34,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:50:34,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.61 | bwd_microstep: 5680.56 | bwd_inner_microstep: 5643.44 | bwd_allreduce_microstep: 37.08 | step_microstep: 18.63
[2025-04-25 14:50:34,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.61 | bwd: 5680.58 | bwd_inner: 5643.44 | bwd_allreduce: 37.09 | step: 18.63

  7%|▋         | 2853/41250 [6:52:59<92:16:30,  8.65s/it]
                                                         
{'loss': 0.1053, 'grad_norm': 1.7492374181747437, 'learning_rate': 3.983942339372764e-05, 'epoch': 0.69}

  7%|▋         | 2853/41250 [6:52:59<92:16:30,  8.65s/it][2025-04-25 14:50:43,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 14:50:43,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.64 | bwd_microstep: 5707.29 | bwd_inner_microstep: 5694.48 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.51
[2025-04-25 14:50:43,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.64 | bwd: 5707.31 | bwd_inner: 5694.48 | bwd_allreduce: 12.78 | step: 18.52

  7%|▋         | 2854/41250 [6:53:08<92:17:05,  8.65s/it]
                                                         
{'loss': 0.2751, 'grad_norm': 1.3146493434906006, 'learning_rate': 3.983922474262217e-05, 'epoch': 0.69}

  7%|▋         | 2854/41250 [6:53:08<92:17:05,  8.65s/it][2025-04-25 14:50:51,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 14:50:51,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.31 | bwd_microstep: 5711.61 | bwd_inner_microstep: 5698.74 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.59
[2025-04-25 14:50:51,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.31 | bwd: 5711.62 | bwd_inner: 5698.74 | bwd_allreduce: 12.84 | step: 18.59

  7%|▋         | 2855/41250 [6:53:16<92:15:57,  8.65s/it]
                                                         
{'loss': 0.3005, 'grad_norm': 2.276930809020996, 'learning_rate': 3.9839025969211796e-05, 'epoch': 0.69}

  7%|▋         | 2855/41250 [6:53:16<92:15:57,  8.65s/it][2025-04-25 14:51:00,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 14:51:00,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2903.51 | bwd_microstep: 5794.77 | bwd_inner_microstep: 5781.80 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.33
[2025-04-25 14:51:00,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2903.51 | bwd: 5794.79 | bwd_inner: 5781.80 | bwd_allreduce: 12.95 | step: 19.33

  7%|▋         | 2856/41250 [6:53:25<92:42:08,  8.69s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 2.323823928833008, 'learning_rate': 3.9838827073497716e-05, 'epoch': 0.69}

  7%|▋         | 2856/41250 [6:53:25<92:42:08,  8.69s/it][2025-04-25 14:51:09,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 14:51:09,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5747.70 | bwd_inner_microstep: 5683.96 | bwd_allreduce_microstep: 63.70 | step_microstep: 18.54
[2025-04-25 14:51:09,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5747.72 | bwd_inner: 5683.96 | bwd_allreduce: 63.72 | step: 18.55

  7%|▋         | 2857/41250 [6:53:34<92:39:03,  8.69s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 0.8665159940719604, 'learning_rate': 3.983862805548118e-05, 'epoch': 0.69}

  7%|▋         | 2857/41250 [6:53:34<92:39:03,  8.69s/it][2025-04-25 14:51:17,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:51:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.12 | bwd_microstep: 5715.27 | bwd_inner_microstep: 5702.50 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.44
[2025-04-25 14:51:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.12 | bwd: 5715.29 | bwd_inner: 5702.50 | bwd_allreduce: 12.74 | step: 18.44

  7%|▋         | 2858/41250 [6:53:43<92:33:02,  8.68s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.2041431665420532, 'learning_rate': 3.9838428915163396e-05, 'epoch': 0.69}

  7%|▋         | 2858/41250 [6:53:43<92:33:02,  8.68s/it][2025-04-25 14:51:26,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:51:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.81 | bwd_microstep: 5762.74 | bwd_inner_microstep: 5656.42 | bwd_allreduce_microstep: 106.27 | step_microstep: 18.55
[2025-04-25 14:51:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.81 | bwd: 5762.75 | bwd_inner: 5656.42 | bwd_allreduce: 106.29 | step: 18.56

  7%|▋         | 2859/41250 [6:53:51<92:33:56,  8.68s/it]
                                                         
{'loss': 0.2011, 'grad_norm': 1.84835684299469, 'learning_rate': 3.9838229652545604e-05, 'epoch': 0.69}

  7%|▋         | 2859/41250 [6:53:51<92:33:56,  8.68s/it][2025-04-25 14:51:35,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:51:35,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.97 | bwd_microstep: 5881.25 | bwd_inner_microstep: 5703.44 | bwd_allreduce_microstep: 177.77 | step_microstep: 18.69
[2025-04-25 14:51:35,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.97 | bwd: 5881.27 | bwd_inner: 5703.44 | bwd_allreduce: 177.79 | step: 18.70

  7%|▋         | 2860/41250 [6:54:00<92:59:43,  8.72s/it]
                                                         
{'loss': 0.2263, 'grad_norm': 2.6115734577178955, 'learning_rate': 3.983803026762903e-05, 'epoch': 0.69}

  7%|▋         | 2860/41250 [6:54:00<92:59:43,  8.72s/it][2025-04-25 14:51:43,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:51:43,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.97 | bwd_microstep: 5692.15 | bwd_inner_microstep: 5665.86 | bwd_allreduce_microstep: 26.24 | step_microstep: 18.71
[2025-04-25 14:51:43,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.97 | bwd: 5692.16 | bwd_inner: 5665.86 | bwd_allreduce: 26.26 | step: 18.72

  7%|▋         | 2861/41250 [6:54:09<92:37:16,  8.69s/it]
                                                         
{'loss': 0.2613, 'grad_norm': 1.4248700141906738, 'learning_rate': 3.983783076041489e-05, 'epoch': 0.69}

  7%|▋         | 2861/41250 [6:54:09<92:37:16,  8.69s/it][2025-04-25 14:51:52,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:51:52,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.12 | bwd_microstep: 5777.04 | bwd_inner_microstep: 5655.55 | bwd_allreduce_microstep: 121.44 | step_microstep: 19.07
[2025-04-25 14:51:52,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.12 | bwd: 5777.06 | bwd_inner: 5655.55 | bwd_allreduce: 121.46 | step: 19.07

  7%|▋         | 2862/41250 [6:54:17<92:36:57,  8.69s/it]
                                                         
{'loss': 0.2449, 'grad_norm': 1.8150804042816162, 'learning_rate': 3.983763113090444e-05, 'epoch': 0.69}

  7%|▋         | 2862/41250 [6:54:17<92:36:57,  8.69s/it][2025-04-25 14:52:01,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 14:52:01,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5705.82 | bwd_inner_microstep: 5693.09 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.66
[2025-04-25 14:52:01,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5705.83 | bwd_inner: 5693.09 | bwd_allreduce: 12.70 | step: 18.66

  7%|▋         | 2863/41250 [6:54:26<92:27:28,  8.67s/it]
                                                         
{'loss': 0.2869, 'grad_norm': 2.5737717151641846, 'learning_rate': 3.983743137909889e-05, 'epoch': 0.69}

  7%|▋         | 2863/41250 [6:54:26<92:27:28,  8.67s/it][2025-04-25 14:52:09,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:52:09,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5788.36 | bwd_inner_microstep: 5696.70 | bwd_allreduce_microstep: 91.61 | step_microstep: 18.89
[2025-04-25 14:52:09,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5788.37 | bwd_inner: 5696.70 | bwd_allreduce: 91.63 | step: 18.89

  7%|▋         | 2864/41250 [6:54:35<92:36:42,  8.69s/it]
                                                         
{'loss': 0.2126, 'grad_norm': 2.1259570121765137, 'learning_rate': 3.9837231504999475e-05, 'epoch': 0.69}

  7%|▋         | 2864/41250 [6:54:35<92:36:42,  8.69s/it][2025-04-25 14:52:18,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:52:18,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.85 | bwd_microstep: 5704.67 | bwd_inner_microstep: 5654.86 | bwd_allreduce_microstep: 49.76 | step_microstep: 18.91
[2025-04-25 14:52:18,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.85 | bwd: 5704.68 | bwd_inner: 5654.86 | bwd_allreduce: 49.78 | step: 18.91

  7%|▋         | 2865/41250 [6:54:43<92:24:10,  8.67s/it]
                                                         
{'loss': 0.1278, 'grad_norm': 1.3701355457305908, 'learning_rate': 3.983703150860743e-05, 'epoch': 0.69}

  7%|▋         | 2865/41250 [6:54:43<92:24:10,  8.67s/it][2025-04-25 14:52:27,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.07 | optimizer_step: 1.19
[2025-04-25 14:52:27,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.98 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5668.87 | bwd_allreduce_microstep: 37.86 | step_microstep: 19.77
[2025-04-25 14:52:27,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.98 | bwd: 5706.79 | bwd_inner: 5668.87 | bwd_allreduce: 37.88 | step: 19.78

  7%|▋         | 2866/41250 [6:54:52<92:17:29,  8.66s/it]
                                                         
{'loss': 0.2647, 'grad_norm': 1.2487337589263916, 'learning_rate': 3.983683138992398e-05, 'epoch': 0.69}

  7%|▋         | 2866/41250 [6:54:52<92:17:29,  8.66s/it][2025-04-25 14:52:35,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 14:52:35,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.56 | bwd_microstep: 5737.86 | bwd_inner_microstep: 5715.59 | bwd_allreduce_microstep: 22.23 | step_microstep: 18.96
[2025-04-25 14:52:35,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.56 | bwd: 5737.87 | bwd_inner: 5715.59 | bwd_allreduce: 22.24 | step: 18.96

  7%|▋         | 2867/41250 [6:55:01<92:23:15,  8.67s/it]
                                                         
{'loss': 0.0835, 'grad_norm': 0.8335726261138916, 'learning_rate': 3.983663114895038e-05, 'epoch': 0.7}

  7%|▋         | 2867/41250 [6:55:01<92:23:15,  8.67s/it][2025-04-25 14:52:44,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:52:44,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.80 | bwd_microstep: 5777.48 | bwd_inner_microstep: 5764.66 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.34
[2025-04-25 14:52:44,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.80 | bwd: 5777.50 | bwd_inner: 5764.66 | bwd_allreduce: 12.80 | step: 19.34

  7%|▋         | 2868/41250 [6:55:09<92:40:55,  8.69s/it]
                                                         
{'loss': 0.1311, 'grad_norm': 1.072408676147461, 'learning_rate': 3.9836430785687845e-05, 'epoch': 0.7}

  7%|▋         | 2868/41250 [6:55:09<92:40:55,  8.69s/it][2025-04-25 14:52:53,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-25 14:52:53,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.73 | bwd_microstep: 5694.04 | bwd_inner_microstep: 5660.41 | bwd_allreduce_microstep: 33.59 | step_microstep: 19.20
[2025-04-25 14:52:53,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.73 | bwd: 5694.06 | bwd_inner: 5660.41 | bwd_allreduce: 33.60 | step: 19.20

  7%|▋         | 2869/41250 [6:55:18<92:25:20,  8.67s/it]
                                                         
{'loss': 0.1333, 'grad_norm': 1.1755385398864746, 'learning_rate': 3.983623030013762e-05, 'epoch': 0.7}

  7%|▋         | 2869/41250 [6:55:18<92:25:20,  8.67s/it][2025-04-25 14:53:02,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 14:53:02,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.38 | bwd_microstep: 5784.94 | bwd_inner_microstep: 5771.65 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.49
[2025-04-25 14:53:02,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.38 | bwd: 5784.95 | bwd_inner: 5771.64 | bwd_allreduce: 13.26 | step: 19.49

  7%|▋         | 2870/41250 [6:55:27<92:44:33,  8.70s/it]
                                                         
{'loss': 0.2134, 'grad_norm': 2.5867905616760254, 'learning_rate': 3.9836029692300935e-05, 'epoch': 0.7}

  7%|▋         | 2870/41250 [6:55:27<92:44:33,  8.70s/it][2025-04-25 14:53:10,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 14:53:10,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.34 | bwd_microstep: 5796.01 | bwd_inner_microstep: 5653.63 | bwd_allreduce_microstep: 142.34 | step_microstep: 19.04
[2025-04-25 14:53:10,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.34 | bwd: 5796.03 | bwd_inner: 5653.63 | bwd_allreduce: 142.36 | step: 19.04

  7%|▋         | 2871/41250 [6:55:36<92:46:00,  8.70s/it]
                                                         
{'loss': 0.1947, 'grad_norm': 1.0670808553695679, 'learning_rate': 3.983582896217902e-05, 'epoch': 0.7}

  7%|▋         | 2871/41250 [6:55:36<92:46:00,  8.70s/it][2025-04-25 14:53:19,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 14:53:19,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.12 | bwd_microstep: 5719.95 | bwd_inner_microstep: 5653.30 | bwd_allreduce_microstep: 66.61 | step_microstep: 19.46
[2025-04-25 14:53:19,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.12 | bwd: 5719.97 | bwd_inner: 5653.30 | bwd_allreduce: 66.63 | step: 19.46

  7%|▋         | 2872/41250 [6:55:44<92:32:45,  8.68s/it]
                                                         
{'loss': 0.1657, 'grad_norm': 1.5751194953918457, 'learning_rate': 3.983562810977313e-05, 'epoch': 0.7}

  7%|▋         | 2872/41250 [6:55:44<92:32:45,  8.68s/it][2025-04-25 14:53:28,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 14:53:28,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5761.76 | bwd_inner_microstep: 5712.23 | bwd_allreduce_microstep: 49.47 | step_microstep: 19.04
[2025-04-25 14:53:28,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5761.77 | bwd_inner: 5712.23 | bwd_allreduce: 49.49 | step: 19.04

  7%|▋         | 2873/41250 [6:55:53<92:36:41,  8.69s/it]
                                                         
{'loss': 0.1093, 'grad_norm': 0.930862545967102, 'learning_rate': 3.983542713508449e-05, 'epoch': 0.7}

  7%|▋         | 2873/41250 [6:55:53<92:36:41,  8.69s/it][2025-04-25 14:53:36,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.19 | optimizer_step: 0.91
[2025-04-25 14:53:36,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5726.17 | bwd_inner_microstep: 5703.62 | bwd_allreduce_microstep: 22.49 | step_microstep: 19.31
[2025-04-25 14:53:36,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5726.18 | bwd_inner: 5703.62 | bwd_allreduce: 22.52 | step: 19.31

  7%|▋         | 2874/41250 [6:56:02<92:32:16,  8.68s/it]
                                                         
{'loss': 0.0588, 'grad_norm': 0.49551934003829956, 'learning_rate': 3.983522603811435e-05, 'epoch': 0.7}

  7%|▋         | 2874/41250 [6:56:02<92:32:16,  8.68s/it][2025-04-25 14:53:45,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.19 | optimizer_step: 1.06
[2025-04-25 14:53:45,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.30 | bwd_microstep: 5702.66 | bwd_inner_microstep: 5666.00 | bwd_allreduce_microstep: 36.59 | step_microstep: 20.55
[2025-04-25 14:53:45,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.30 | bwd: 5702.68 | bwd_inner: 5666.00 | bwd_allreduce: 36.63 | step: 20.55

  7%|▋         | 2875/41250 [6:56:10<92:21:23,  8.66s/it]
                                                         
{'loss': 0.2785, 'grad_norm': 1.5680948495864868, 'learning_rate': 3.983502481886393e-05, 'epoch': 0.7}

  7%|▋         | 2875/41250 [6:56:10<92:21:23,  8.66s/it][2025-04-25 14:53:53,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 14:53:53,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5694.21 | bwd_inner_microstep: 5651.75 | bwd_allreduce_microstep: 42.42 | step_microstep: 18.82
[2025-04-25 14:53:53,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5694.22 | bwd_inner: 5651.75 | bwd_allreduce: 42.43 | step: 18.82

  7%|▋         | 2876/41250 [6:56:19<92:09:46,  8.65s/it]
                                                         
{'loss': 0.4744, 'grad_norm': 3.3760907649993896, 'learning_rate': 3.983482347733449e-05, 'epoch': 0.7}

  7%|▋         | 2876/41250 [6:56:19<92:09:46,  8.65s/it][2025-04-25 14:54:02,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 14:54:02,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.50 | bwd_microstep: 5692.93 | bwd_inner_microstep: 5658.68 | bwd_allreduce_microstep: 34.20 | step_microstep: 18.48
[2025-04-25 14:54:02,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.50 | bwd: 5692.94 | bwd_inner: 5658.68 | bwd_allreduce: 34.22 | step: 18.49

  7%|▋         | 2877/41250 [6:56:27<92:02:00,  8.63s/it]
                                                         
{'loss': 0.0854, 'grad_norm': 1.3435289859771729, 'learning_rate': 3.983462201352727e-05, 'epoch': 0.7}

  7%|▋         | 2877/41250 [6:56:27<92:02:00,  8.63s/it][2025-04-25 14:54:11,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:54:11,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.66 | bwd_microstep: 5734.56 | bwd_inner_microstep: 5721.39 | bwd_allreduce_microstep: 13.12 | step_microstep: 19.03
[2025-04-25 14:54:11,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.66 | bwd: 5734.58 | bwd_inner: 5721.39 | bwd_allreduce: 13.15 | step: 19.03

  7%|▋         | 2878/41250 [6:56:36<92:10:35,  8.65s/it]
                                                         
{'loss': 0.1777, 'grad_norm': 1.4106751680374146, 'learning_rate': 3.98344204274435e-05, 'epoch': 0.7}

  7%|▋         | 2878/41250 [6:56:36<92:10:35,  8.65s/it][2025-04-25 14:54:20,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:54:20,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.73 | bwd_microstep: 5883.35 | bwd_inner_microstep: 5705.40 | bwd_allreduce_microstep: 177.91 | step_microstep: 18.61
[2025-04-25 14:54:20,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.73 | bwd: 5883.36 | bwd_inner: 5705.40 | bwd_allreduce: 177.93 | step: 18.61

  7%|▋         | 2879/41250 [6:56:45<92:44:40,  8.70s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 1.1065586805343628, 'learning_rate': 3.9834218719084434e-05, 'epoch': 0.7}

  7%|▋         | 2879/41250 [6:56:45<92:44:40,  8.70s/it][2025-04-25 14:54:28,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:54:28,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.36 | bwd_microstep: 5754.07 | bwd_inner_microstep: 5687.60 | bwd_allreduce_microstep: 66.42 | step_microstep: 18.58
[2025-04-25 14:54:28,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.36 | bwd: 5754.08 | bwd_inner: 5687.60 | bwd_allreduce: 66.44 | step: 18.58

  7%|▋         | 2880/41250 [6:56:54<92:41:54,  8.70s/it]
                                                         
{'loss': 0.1666, 'grad_norm': 1.0017973184585571, 'learning_rate': 3.98340168884513e-05, 'epoch': 0.7}

  7%|▋         | 2880/41250 [6:56:54<92:41:54,  8.70s/it][2025-04-25 14:54:37,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 14:54:37,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.63 | bwd_microstep: 5698.00 | bwd_inner_microstep: 5646.96 | bwd_allreduce_microstep: 50.99 | step_microstep: 18.91
[2025-04-25 14:54:37,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.64 | bwd: 5698.01 | bwd_inner: 5646.96 | bwd_allreduce: 51.01 | step: 18.91

  7%|▋         | 2881/41250 [6:57:02<92:27:07,  8.67s/it]
                                                         
{'loss': 0.1516, 'grad_norm': 1.0813941955566406, 'learning_rate': 3.9833814935545366e-05, 'epoch': 0.7}

  7%|▋         | 2881/41250 [6:57:02<92:27:07,  8.67s/it][2025-04-25 14:54:46,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-25 14:54:46,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.24 | bwd_microstep: 5941.73 | bwd_inner_microstep: 5657.57 | bwd_allreduce_microstep: 284.12 | step_microstep: 18.86
[2025-04-25 14:54:46,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.24 | bwd: 5941.74 | bwd_inner: 5657.57 | bwd_allreduce: 284.14 | step: 18.86

  7%|▋         | 2882/41250 [6:57:11<93:01:43,  8.73s/it]
                                                         
{'loss': 0.1449, 'grad_norm': 2.162557363510132, 'learning_rate': 3.983361286036786e-05, 'epoch': 0.7}

  7%|▋         | 2882/41250 [6:57:11<93:01:43,  8.73s/it][2025-04-25 14:54:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:54:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.83 | bwd_microstep: 5690.68 | bwd_inner_microstep: 5650.69 | bwd_allreduce_microstep: 39.95 | step_microstep: 18.70
[2025-04-25 14:54:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.83 | bwd: 5690.70 | bwd_inner: 5650.69 | bwd_allreduce: 39.97 | step: 18.70

  7%|▋         | 2883/41250 [6:57:20<92:39:05,  8.69s/it]
                                                         
{'loss': 0.3264, 'grad_norm': 2.4076497554779053, 'learning_rate': 3.983341066292003e-05, 'epoch': 0.7}

  7%|▋         | 2883/41250 [6:57:20<92:39:05,  8.69s/it][2025-04-25 14:55:03,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 14:55:03,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.96 | bwd_microstep: 5709.53 | bwd_inner_microstep: 5690.15 | bwd_allreduce_microstep: 19.33 | step_microstep: 18.70
[2025-04-25 14:55:03,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.96 | bwd: 5709.54 | bwd_inner: 5690.15 | bwd_allreduce: 19.35 | step: 18.70

  7%|▋         | 2884/41250 [6:57:28<92:29:56,  8.68s/it]
                                                         
{'loss': 0.1608, 'grad_norm': 1.1794469356536865, 'learning_rate': 3.9833208343203126e-05, 'epoch': 0.7}

  7%|▋         | 2884/41250 [6:57:28<92:29:56,  8.68s/it][2025-04-25 14:55:12,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 14:55:12,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.56 | bwd_microstep: 5710.02 | bwd_inner_microstep: 5697.05 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.68
[2025-04-25 14:55:12,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.56 | bwd: 5710.03 | bwd_inner: 5697.04 | bwd_allreduce: 12.95 | step: 18.68

  7%|▋         | 2885/41250 [6:57:37<92:21:47,  8.67s/it]
                                                         
{'loss': 0.2066, 'grad_norm': 1.798948049545288, 'learning_rate': 3.9833005901218394e-05, 'epoch': 0.7}

  7%|▋         | 2885/41250 [6:57:37<92:21:47,  8.67s/it][2025-04-25 14:55:20,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:55:20,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5737.74 | bwd_inner_microstep: 5651.43 | bwd_allreduce_microstep: 86.27 | step_microstep: 18.45
[2025-04-25 14:55:20,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5737.75 | bwd_inner: 5651.43 | bwd_allreduce: 86.29 | step: 18.46

  7%|▋         | 2886/41250 [6:57:46<92:18:52,  8.66s/it]
                                                         
{'loss': 0.1143, 'grad_norm': 1.010790228843689, 'learning_rate': 3.983280333696708e-05, 'epoch': 0.7}

  7%|▋         | 2886/41250 [6:57:46<92:18:52,  8.66s/it][2025-04-25 14:55:29,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 14:55:29,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5767.39 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 115.05 | step_microstep: 18.75
[2025-04-25 14:55:29,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5767.40 | bwd_inner: 5652.29 | bwd_allreduce: 115.07 | step: 18.75

  7%|▋         | 2887/41250 [6:57:54<92:21:49,  8.67s/it]
                                                         
{'loss': 0.1508, 'grad_norm': 1.0413086414337158, 'learning_rate': 3.983260065045043e-05, 'epoch': 0.7}

  7%|▋         | 2887/41250 [6:57:54<92:21:49,  8.67s/it][2025-04-25 14:55:38,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:55:38,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5745.41 | bwd_inner_microstep: 5665.97 | bwd_allreduce_microstep: 79.39 | step_microstep: 18.63
[2025-04-25 14:55:38,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5745.42 | bwd_inner: 5665.97 | bwd_allreduce: 79.41 | step: 18.63

  7%|▋         | 2888/41250 [6:58:03<92:20:10,  8.67s/it]
                                                         
{'loss': 0.0803, 'grad_norm': 1.9983997344970703, 'learning_rate': 3.9832397841669704e-05, 'epoch': 0.7}

  7%|▋         | 2888/41250 [6:58:03<92:20:10,  8.67s/it][2025-04-25 14:55:46,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 14:55:46,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5733.31 | bwd_inner_microstep: 5694.20 | bwd_allreduce_microstep: 39.07 | step_microstep: 18.70
[2025-04-25 14:55:46,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5733.33 | bwd_inner: 5694.19 | bwd_allreduce: 39.09 | step: 18.70

  7%|▋         | 2889/41250 [6:58:12<92:19:27,  8.66s/it]
                                                         
{'loss': 0.0872, 'grad_norm': 0.7268747687339783, 'learning_rate': 3.983219491062614e-05, 'epoch': 0.7}

  7%|▋         | 2889/41250 [6:58:12<92:19:27,  8.66s/it][2025-04-25 14:55:55,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.16
[2025-04-25 14:55:55,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.89 | bwd_microstep: 5690.86 | bwd_inner_microstep: 5637.27 | bwd_allreduce_microstep: 53.55 | step_microstep: 19.46
[2025-04-25 14:55:55,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.89 | bwd: 5690.88 | bwd_inner: 5637.27 | bwd_allreduce: 53.57 | step: 19.47

  7%|▋         | 2890/41250 [6:58:20<92:06:29,  8.64s/it]
                                                         
{'loss': 0.1191, 'grad_norm': 1.9877766370773315, 'learning_rate': 3.9831991857321e-05, 'epoch': 0.7}

  7%|▋         | 2890/41250 [6:58:20<92:06:29,  8.64s/it][2025-04-25 14:56:04,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:56:04,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.23 | bwd_microstep: 5754.99 | bwd_inner_microstep: 5653.45 | bwd_allreduce_microstep: 101.50 | step_microstep: 18.38
[2025-04-25 14:56:04,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.23 | bwd: 5755.01 | bwd_inner: 5653.45 | bwd_allreduce: 101.51 | step: 18.38

  7%|▋         | 2891/41250 [6:58:29<92:10:38,  8.65s/it]
                                                         
{'loss': 0.1657, 'grad_norm': 0.9395366311073303, 'learning_rate': 3.9831788681755536e-05, 'epoch': 0.7}

  7%|▋         | 2891/41250 [6:58:29<92:10:38,  8.65s/it][2025-04-25 14:56:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 14:56:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.31 | bwd_microstep: 5698.51 | bwd_inner_microstep: 5648.47 | bwd_allreduce_microstep: 49.99 | step_microstep: 18.84
[2025-04-25 14:56:12,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.31 | bwd: 5698.53 | bwd_inner: 5648.47 | bwd_allreduce: 50.01 | step: 18.84

  7%|▋         | 2892/41250 [6:58:37<92:03:13,  8.64s/it]
                                                         
{'loss': 0.1469, 'grad_norm': 1.6083990335464478, 'learning_rate': 3.983158538393099e-05, 'epoch': 0.7}

  7%|▋         | 2892/41250 [6:58:37<92:03:13,  8.64s/it][2025-04-25 14:56:21,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 14:56:21,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.39 | bwd_microstep: 5759.33 | bwd_inner_microstep: 5650.92 | bwd_allreduce_microstep: 108.37 | step_microstep: 18.65
[2025-04-25 14:56:21,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.39 | bwd: 5759.34 | bwd_inner: 5650.92 | bwd_allreduce: 108.38 | step: 18.65

  7%|▋         | 2893/41250 [6:58:46<92:08:11,  8.65s/it]
                                                         
{'loss': 0.1313, 'grad_norm': 1.4044957160949707, 'learning_rate': 3.983138196384862e-05, 'epoch': 0.7}

  7%|▋         | 2893/41250 [6:58:46<92:08:11,  8.65s/it][2025-04-25 14:56:29,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:56:29,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.70 | bwd_microstep: 5697.45 | bwd_inner_microstep: 5644.73 | bwd_allreduce_microstep: 52.67 | step_microstep: 18.87
[2025-04-25 14:56:29,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.70 | bwd: 5697.46 | bwd_inner: 5644.73 | bwd_allreduce: 52.69 | step: 18.88

  7%|▋         | 2894/41250 [6:58:55<92:00:29,  8.64s/it]
                                                         
{'loss': 0.1696, 'grad_norm': 2.61868953704834, 'learning_rate': 3.983117842150968e-05, 'epoch': 0.7}

  7%|▋         | 2894/41250 [6:58:55<92:00:29,  8.64s/it][2025-04-25 14:56:38,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 14:56:38,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.27 | bwd_microstep: 5729.90 | bwd_inner_microstep: 5687.76 | bwd_allreduce_microstep: 42.10 | step_microstep: 18.70
[2025-04-25 14:56:38,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.27 | bwd: 5729.92 | bwd_inner: 5687.76 | bwd_allreduce: 42.12 | step: 18.70

  7%|▋         | 2895/41250 [6:59:03<92:04:28,  8.64s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 3.0172276496887207, 'learning_rate': 3.9830974756915426e-05, 'epoch': 0.7}

  7%|▋         | 2895/41250 [6:59:03<92:04:28,  8.64s/it][2025-04-25 14:56:47,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 14:56:47,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.55 | bwd_microstep: 5700.53 | bwd_inner_microstep: 5687.66 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.76
[2025-04-25 14:56:47,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.55 | bwd: 5700.55 | bwd_inner: 5687.66 | bwd_allreduce: 12.84 | step: 18.76

  7%|▋         | 2896/41250 [6:59:12<92:02:16,  8.64s/it]
                                                         
{'loss': 0.2086, 'grad_norm': 1.330439805984497, 'learning_rate': 3.9830770970067115e-05, 'epoch': 0.7}

  7%|▋         | 2896/41250 [6:59:12<92:02:16,  8.64s/it][2025-04-25 14:56:55,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:56:55,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.12 | bwd_microstep: 5750.95 | bwd_inner_microstep: 5648.80 | bwd_allreduce_microstep: 102.11 | step_microstep: 18.70
[2025-04-25 14:56:55,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.12 | bwd: 5750.96 | bwd_inner: 5648.80 | bwd_allreduce: 102.13 | step: 18.70

  7%|▋         | 2897/41250 [6:59:21<92:06:11,  8.65s/it]
                                                         
{'loss': 0.1282, 'grad_norm': 1.7252486944198608, 'learning_rate': 3.9830567060966004e-05, 'epoch': 0.7}

  7%|▋         | 2897/41250 [6:59:21<92:06:11,  8.65s/it][2025-04-25 14:57:04,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 14:57:04,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.04 | bwd_microstep: 5744.48 | bwd_inner_microstep: 5703.90 | bwd_allreduce_microstep: 40.54 | step_microstep: 18.57
[2025-04-25 14:57:04,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.04 | bwd: 5744.50 | bwd_inner: 5703.89 | bwd_allreduce: 40.56 | step: 18.58

  7%|▋         | 2898/41250 [6:59:29<92:14:14,  8.66s/it]
                                                         
{'loss': 0.1419, 'grad_norm': 1.1743109226226807, 'learning_rate': 3.983036302961334e-05, 'epoch': 0.7}

  7%|▋         | 2898/41250 [6:59:29<92:14:14,  8.66s/it][2025-04-25 14:57:13,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 14:57:13,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.02 | bwd_microstep: 5692.99 | bwd_inner_microstep: 5680.16 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.61
[2025-04-25 14:57:13,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.02 | bwd: 5693.00 | bwd_inner: 5680.17 | bwd_allreduce: 12.80 | step: 18.62

  7%|▋         | 2899/41250 [6:59:38<92:06:33,  8.65s/it]
                                                         
{'loss': 0.1595, 'grad_norm': 1.8893202543258667, 'learning_rate': 3.9830158876010394e-05, 'epoch': 0.7}

  7%|▋         | 2899/41250 [6:59:38<92:06:33,  8.65s/it][2025-04-25 14:57:21,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-25 14:57:21,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.46 | bwd_microstep: 5768.11 | bwd_inner_microstep: 5754.99 | bwd_allreduce_microstep: 13.07 | step_microstep: 19.31
[2025-04-25 14:57:21,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.46 | bwd: 5768.12 | bwd_inner: 5754.99 | bwd_allreduce: 13.09 | step: 19.31

  7%|▋         | 2900/41250 [6:59:47<92:23:12,  8.67s/it]
                                                         
{'loss': 0.1803, 'grad_norm': 2.5407183170318604, 'learning_rate': 3.9829954600158416e-05, 'epoch': 0.7}

  7%|▋         | 2900/41250 [6:59:47<92:23:12,  8.67s/it][2025-04-25 14:57:30,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 14:57:30,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.88 | bwd_microstep: 5758.81 | bwd_inner_microstep: 5639.48 | bwd_allreduce_microstep: 119.28 | step_microstep: 18.71
[2025-04-25 14:57:30,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.88 | bwd: 5758.82 | bwd_inner: 5639.48 | bwd_allreduce: 119.30 | step: 18.72

  7%|▋         | 2901/41250 [6:59:55<92:21:05,  8.67s/it]
                                                         
{'loss': 0.2218, 'grad_norm': 2.451941967010498, 'learning_rate': 3.982975020205867e-05, 'epoch': 0.7}

  7%|▋         | 2901/41250 [6:59:55<92:21:05,  8.67s/it][2025-04-25 14:57:39,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 14:57:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.64 | bwd_microstep: 5743.99 | bwd_inner_microstep: 5688.14 | bwd_allreduce_microstep: 55.80 | step_microstep: 18.86
[2025-04-25 14:57:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.64 | bwd: 5744.00 | bwd_inner: 5688.14 | bwd_allreduce: 55.82 | step: 18.87

  7%|▋         | 2902/41250 [7:00:04<92:23:47,  8.67s/it]
                                                         
{'loss': 0.1792, 'grad_norm': 1.9436376094818115, 'learning_rate': 3.9829545681712416e-05, 'epoch': 0.7}

  7%|▋         | 2902/41250 [7:00:04<92:23:47,  8.67s/it][2025-04-25 14:57:47,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 14:57:47,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.65 | bwd_microstep: 5711.51 | bwd_inner_microstep: 5698.39 | bwd_allreduce_microstep: 13.07 | step_microstep: 19.16
[2025-04-25 14:57:47,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.65 | bwd: 5711.53 | bwd_inner: 5698.39 | bwd_allreduce: 13.09 | step: 19.16

  7%|▋         | 2903/41250 [7:00:13<92:18:00,  8.67s/it]
                                                         
{'loss': 0.2942, 'grad_norm': 2.11310076713562, 'learning_rate': 3.982934103912091e-05, 'epoch': 0.7}

  7%|▋         | 2903/41250 [7:00:13<92:18:00,  8.67s/it][2025-04-25 14:57:56,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 14:57:56,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.86 | bwd_microstep: 5746.36 | bwd_inner_microstep: 5687.99 | bwd_allreduce_microstep: 58.31 | step_microstep: 18.84
[2025-04-25 14:57:56,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.86 | bwd: 5746.37 | bwd_inner: 5687.99 | bwd_allreduce: 58.33 | step: 18.85

  7%|▋         | 2904/41250 [7:00:21<92:19:13,  8.67s/it]
                                                         
{'loss': 0.2576, 'grad_norm': 1.8824388980865479, 'learning_rate': 3.9829136274285416e-05, 'epoch': 0.7}

  7%|▋         | 2904/41250 [7:00:21<92:19:13,  8.67s/it][2025-04-25 14:58:05,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 14:58:05,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.53 | bwd_microstep: 5699.95 | bwd_inner_microstep: 5686.95 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.78
[2025-04-25 14:58:05,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.53 | bwd: 5699.96 | bwd_inner: 5686.95 | bwd_allreduce: 12.97 | step: 18.78

  7%|▋         | 2905/41250 [7:00:30<92:12:41,  8.66s/it]
                                                         
{'loss': 0.483, 'grad_norm': 3.771726131439209, 'learning_rate': 3.9828931387207206e-05, 'epoch': 0.7}

  7%|▋         | 2905/41250 [7:00:30<92:12:41,  8.66s/it][2025-04-25 14:58:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 14:58:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.20 | bwd_microstep: 5745.24 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 100.19 | step_microstep: 18.67
[2025-04-25 14:58:13,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.20 | bwd: 5745.25 | bwd_inner: 5645.00 | bwd_allreduce: 100.21 | step: 18.67

  7%|▋         | 2906/41250 [7:00:39<92:12:04,  8.66s/it]
                                                         
{'loss': 0.1877, 'grad_norm': 1.4455227851867676, 'learning_rate': 3.9828726377887525e-05, 'epoch': 0.7}

  7%|▋         | 2906/41250 [7:00:39<92:12:04,  8.66s/it][2025-04-25 14:58:22,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 14:58:22,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.24 | bwd_microstep: 5725.33 | bwd_inner_microstep: 5683.70 | bwd_allreduce_microstep: 41.58 | step_microstep: 19.08
[2025-04-25 14:58:22,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.24 | bwd: 5725.35 | bwd_inner: 5683.70 | bwd_allreduce: 41.60 | step: 19.08

  7%|▋         | 2907/41250 [7:00:47<92:11:21,  8.66s/it]
                                                         
{'loss': 0.169, 'grad_norm': 1.633386254310608, 'learning_rate': 3.982852124632764e-05, 'epoch': 0.7}

  7%|▋         | 2907/41250 [7:00:47<92:11:21,  8.66s/it][2025-04-25 14:58:31,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.07 | optimizer_step: 1.01
[2025-04-25 14:58:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5670.52 | bwd_inner_microstep: 5656.78 | bwd_allreduce_microstep: 13.69 | step_microstep: 19.68
[2025-04-25 14:58:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5670.53 | bwd_inner: 5656.78 | bwd_allreduce: 13.71 | step: 19.68

  7%|▋         | 2908/41250 [7:00:56<91:58:05,  8.64s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.8975396752357483, 'learning_rate': 3.9828315992528834e-05, 'epoch': 0.7}

  7%|▋         | 2908/41250 [7:00:56<91:58:05,  8.64s/it][2025-04-25 14:58:39,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 14:58:39,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.61 | bwd_microstep: 5770.06 | bwd_inner_microstep: 5646.16 | bwd_allreduce_microstep: 123.86 | step_microstep: 18.75
[2025-04-25 14:58:39,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.61 | bwd: 5770.07 | bwd_inner: 5646.16 | bwd_allreduce: 123.87 | step: 18.75

  7%|▋         | 2909/41250 [7:01:05<92:07:37,  8.65s/it]
                                                         
{'loss': 0.2246, 'grad_norm': 2.237243413925171, 'learning_rate': 3.982811061649236e-05, 'epoch': 0.71}

  7%|▋         | 2909/41250 [7:01:05<92:07:37,  8.65s/it][2025-04-25 14:58:48,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:58:48,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5744.04 | bwd_inner_microstep: 5707.40 | bwd_allreduce_microstep: 36.60 | step_microstep: 18.78
[2025-04-25 14:58:48,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5744.05 | bwd_inner: 5707.40 | bwd_allreduce: 36.61 | step: 18.78

  7%|▋         | 2910/41250 [7:01:13<92:12:15,  8.66s/it]
                                                         
{'loss': 0.0728, 'grad_norm': 0.6962196826934814, 'learning_rate': 3.982790511821947e-05, 'epoch': 0.71}

  7%|▋         | 2910/41250 [7:01:13<92:12:15,  8.66s/it][2025-04-25 14:58:57,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 14:58:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.42 | bwd_microstep: 5694.50 | bwd_inner_microstep: 5652.00 | bwd_allreduce_microstep: 42.45 | step_microstep: 18.62
[2025-04-25 14:58:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.42 | bwd: 5694.51 | bwd_inner: 5652.00 | bwd_allreduce: 42.47 | step: 18.63

  7%|▋         | 2911/41250 [7:01:22<92:01:32,  8.64s/it]
                                                         
{'loss': 0.1444, 'grad_norm': 2.4193344116210938, 'learning_rate': 3.982769949771146e-05, 'epoch': 0.71}

  7%|▋         | 2911/41250 [7:01:22<92:01:32,  8.64s/it][2025-04-25 14:59:05,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 14:59:05,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5712.36 | bwd_inner_microstep: 5699.46 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.94
[2025-04-25 14:59:05,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5712.37 | bwd_inner: 5699.46 | bwd_allreduce: 12.87 | step: 18.94

  7%|▋         | 2912/41250 [7:01:31<92:02:08,  8.64s/it]
                                                         
{'loss': 0.2741, 'grad_norm': 1.498673439025879, 'learning_rate': 3.982749375496957e-05, 'epoch': 0.71}

  7%|▋         | 2912/41250 [7:01:31<92:02:08,  8.64s/it][2025-04-25 14:59:14,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 14:59:14,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.00 | bwd_microstep: 5896.57 | bwd_inner_microstep: 5883.92 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.44
[2025-04-25 14:59:14,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.00 | bwd: 5896.59 | bwd_inner: 5883.92 | bwd_allreduce: 12.62 | step: 18.44

  7%|▋         | 2913/41250 [7:01:39<92:54:30,  8.72s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.0443509817123413, 'learning_rate': 3.98272878899951e-05, 'epoch': 0.71}

  7%|▋         | 2913/41250 [7:01:39<92:54:30,  8.72s/it][2025-04-25 14:59:23,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 14:59:23,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.15 | bwd_microstep: 5727.62 | bwd_inner_microstep: 5701.11 | bwd_allreduce_microstep: 26.46 | step_microstep: 19.57
[2025-04-25 14:59:23,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.15 | bwd: 5727.63 | bwd_inner: 5701.11 | bwd_allreduce: 26.48 | step: 19.58

  7%|▋         | 2914/41250 [7:01:48<92:44:06,  8.71s/it]
                                                         
{'loss': 0.0429, 'grad_norm': 0.49507513642311096, 'learning_rate': 3.982708190278929e-05, 'epoch': 0.71}

  7%|▋         | 2914/41250 [7:01:48<92:44:06,  8.71s/it][2025-04-25 14:59:31,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 14:59:31,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5747.43 | bwd_inner_microstep: 5712.12 | bwd_allreduce_microstep: 35.26 | step_microstep: 18.57
[2025-04-25 14:59:31,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5747.44 | bwd_inner: 5712.12 | bwd_allreduce: 35.28 | step: 18.57

  7%|▋         | 2915/41250 [7:01:57<92:39:03,  8.70s/it]
                                                         
{'loss': 0.3267, 'grad_norm': 2.89860463142395, 'learning_rate': 3.9826875793353415e-05, 'epoch': 0.71}

  7%|▋         | 2915/41250 [7:01:57<92:39:03,  8.70s/it][2025-04-25 14:59:40,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 14:59:40,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.67 | bwd_microstep: 5766.30 | bwd_inner_microstep: 5658.30 | bwd_allreduce_microstep: 107.95 | step_microstep: 19.11
[2025-04-25 14:59:40,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.67 | bwd: 5766.31 | bwd_inner: 5658.30 | bwd_allreduce: 107.97 | step: 19.11

  7%|▋         | 2916/41250 [7:02:06<92:35:59,  8.70s/it]
                                                         
{'loss': 0.0561, 'grad_norm': 0.49287405610084534, 'learning_rate': 3.982666956168875e-05, 'epoch': 0.71}

  7%|▋         | 2916/41250 [7:02:06<92:35:59,  8.70s/it][2025-04-25 14:59:49,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 14:59:49,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.67 | bwd_microstep: 5698.35 | bwd_inner_microstep: 5662.89 | bwd_allreduce_microstep: 35.42 | step_microstep: 18.79
[2025-04-25 14:59:49,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.67 | bwd: 5698.37 | bwd_inner: 5662.70 | bwd_allreduce: 35.43 | step: 18.79

  7%|▋         | 2917/41250 [7:02:14<92:19:52,  8.67s/it]
                                                         
{'loss': 0.2412, 'grad_norm': 1.490285038948059, 'learning_rate': 3.982646320779657e-05, 'epoch': 0.71}

  7%|▋         | 2917/41250 [7:02:14<92:19:52,  8.67s/it][2025-04-25 14:59:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 14:59:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.23 | bwd_microstep: 5789.95 | bwd_inner_microstep: 5651.84 | bwd_allreduce_microstep: 138.06 | step_microstep: 18.47
[2025-04-25 14:59:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.23 | bwd: 5789.96 | bwd_inner: 5651.84 | bwd_allreduce: 138.08 | step: 18.47

  7%|▋         | 2918/41250 [7:02:23<92:25:52,  8.68s/it]
                                                         
{'loss': 0.1802, 'grad_norm': 1.5839613676071167, 'learning_rate': 3.982625673167814e-05, 'epoch': 0.71}

  7%|▋         | 2918/41250 [7:02:23<92:25:52,  8.68s/it][2025-04-25 15:00:06,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.32 | optimizer_step: 1.07
[2025-04-25 15:00:06,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.47 | bwd_microstep: 5696.33 | bwd_inner_microstep: 5660.81 | bwd_allreduce_microstep: 35.46 | step_microstep: 20.41
[2025-04-25 15:00:06,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.47 | bwd: 5696.35 | bwd_inner: 5660.81 | bwd_allreduce: 35.49 | step: 20.41

  7%|▋         | 2919/41250 [7:02:31<92:12:38,  8.66s/it]
                                                         
{'loss': 0.1745, 'grad_norm': 1.5215904712677002, 'learning_rate': 3.982605013333475e-05, 'epoch': 0.71}

  7%|▋         | 2919/41250 [7:02:31<92:12:38,  8.66s/it][2025-04-25 15:00:15,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:00:15,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.62 | bwd_microstep: 5681.63 | bwd_inner_microstep: 5657.23 | bwd_allreduce_microstep: 24.35 | step_microstep: 18.27
[2025-04-25 15:00:15,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.62 | bwd: 5681.64 | bwd_inner: 5657.23 | bwd_allreduce: 24.37 | step: 18.28

  7%|▋         | 2920/41250 [7:02:40<91:59:39,  8.64s/it]
                                                         
{'loss': 0.1002, 'grad_norm': 1.1683741807937622, 'learning_rate': 3.9825843412767644e-05, 'epoch': 0.71}

  7%|▋         | 2920/41250 [7:02:40<91:59:39,  8.64s/it][2025-04-25 15:00:23,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 15:00:23,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5796.30 | bwd_inner_microstep: 5659.90 | bwd_allreduce_microstep: 136.35 | step_microstep: 18.49
[2025-04-25 15:00:23,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5796.31 | bwd_inner: 5659.90 | bwd_allreduce: 136.37 | step: 18.50

  7%|▋         | 2921/41250 [7:02:49<92:12:26,  8.66s/it]
                                                         
{'loss': 0.3645, 'grad_norm': 2.1640377044677734, 'learning_rate': 3.982563656997812e-05, 'epoch': 0.71}

  7%|▋         | 2921/41250 [7:02:49<92:12:26,  8.66s/it][2025-04-25 15:00:32,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 15:00:32,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.17 | bwd_microstep: 5749.90 | bwd_inner_microstep: 5682.65 | bwd_allreduce_microstep: 67.20 | step_microstep: 18.64
[2025-04-25 15:00:32,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.17 | bwd: 5749.92 | bwd_inner: 5682.65 | bwd_allreduce: 67.22 | step: 18.64

  7%|▋         | 2922/41250 [7:02:57<92:17:32,  8.67s/it]
                                                         
{'loss': 0.4459, 'grad_norm': 2.6499576568603516, 'learning_rate': 3.982542960496745e-05, 'epoch': 0.71}

  7%|▋         | 2922/41250 [7:02:57<92:17:32,  8.67s/it][2025-04-25 15:00:41,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:00:41,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5706.30 | bwd_inner_microstep: 5656.41 | bwd_allreduce_microstep: 49.85 | step_microstep: 18.63
[2025-04-25 15:00:41,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5706.31 | bwd_inner: 5656.41 | bwd_allreduce: 49.86 | step: 18.63

  7%|▋         | 2923/41250 [7:03:06<92:08:29,  8.65s/it]
                                                         
{'loss': 0.064, 'grad_norm': 0.6714498400688171, 'learning_rate': 3.98252225177369e-05, 'epoch': 0.71}

  7%|▋         | 2923/41250 [7:03:06<92:08:29,  8.65s/it][2025-04-25 15:00:49,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 15:00:49,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.36 | bwd_microstep: 5736.03 | bwd_inner_microstep: 5723.31 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.44
[2025-04-25 15:00:49,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.36 | bwd: 5736.04 | bwd_inner: 5723.31 | bwd_allreduce: 12.68 | step: 19.43

  7%|▋         | 2924/41250 [7:03:15<92:12:21,  8.66s/it]
                                                         
{'loss': 0.2524, 'grad_norm': 2.4396891593933105, 'learning_rate': 3.982501530828776e-05, 'epoch': 0.71}

  7%|▋         | 2924/41250 [7:03:15<92:12:21,  8.66s/it][2025-04-25 15:00:58,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:00:58,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.90 | bwd_microstep: 5718.99 | bwd_inner_microstep: 5706.23 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.62
[2025-04-25 15:00:58,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.90 | bwd: 5719.01 | bwd_inner: 5706.23 | bwd_allreduce: 12.74 | step: 18.62

  7%|▋         | 2925/41250 [7:03:23<92:11:19,  8.66s/it]
                                                         
{'loss': 0.176, 'grad_norm': 1.4064191579818726, 'learning_rate': 3.982480797662129e-05, 'epoch': 0.71}

  7%|▋         | 2925/41250 [7:03:23<92:11:19,  8.66s/it][2025-04-25 15:01:07,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-25 15:01:07,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.38 | bwd_microstep: 5783.36 | bwd_inner_microstep: 5669.75 | bwd_allreduce_microstep: 113.56 | step_microstep: 18.53
[2025-04-25 15:01:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.38 | bwd: 5783.37 | bwd_inner: 5669.75 | bwd_allreduce: 113.58 | step: 18.53

  7%|▋         | 2926/41250 [7:03:32<92:19:26,  8.67s/it]
                                                         
{'loss': 0.2607, 'grad_norm': 1.7839678525924683, 'learning_rate': 3.9824600522738785e-05, 'epoch': 0.71}

  7%|▋         | 2926/41250 [7:03:32<92:19:26,  8.67s/it][2025-04-25 15:01:15,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:01:15,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.96 | bwd_microstep: 5729.72 | bwd_inner_microstep: 5716.94 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.84
[2025-04-25 15:01:15,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.96 | bwd: 5729.74 | bwd_inner: 5716.94 | bwd_allreduce: 12.75 | step: 18.84

  7%|▋         | 2927/41250 [7:03:41<92:18:14,  8.67s/it]
                                                         
{'loss': 0.2875, 'grad_norm': 2.0127923488616943, 'learning_rate': 3.982439294664152e-05, 'epoch': 0.71}

  7%|▋         | 2927/41250 [7:03:41<92:18:14,  8.67s/it][2025-04-25 15:01:24,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 15:01:24,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2902.00 | bwd_microstep: 5825.43 | bwd_inner_microstep: 5792.96 | bwd_allreduce_microstep: 32.42 | step_microstep: 18.86
[2025-04-25 15:01:24,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2902.00 | bwd: 5825.44 | bwd_inner: 5792.96 | bwd_allreduce: 32.44 | step: 18.87

  7%|▋         | 2928/41250 [7:03:50<92:44:35,  8.71s/it]
                                                         
{'loss': 0.1429, 'grad_norm': 1.211323857307434, 'learning_rate': 3.982418524833076e-05, 'epoch': 0.71}

  7%|▋         | 2928/41250 [7:03:50<92:44:35,  8.71s/it][2025-04-25 15:01:33,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:01:33,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5775.14 | bwd_inner_microstep: 5706.38 | bwd_allreduce_microstep: 68.71 | step_microstep: 19.05
[2025-04-25 15:01:33,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5775.16 | bwd_inner: 5706.38 | bwd_allreduce: 68.73 | step: 19.05

  7%|▋         | 2929/41250 [7:03:58<92:43:28,  8.71s/it]
                                                         
{'loss': 0.138, 'grad_norm': 0.8851692080497742, 'learning_rate': 3.98239774278078e-05, 'epoch': 0.71}

  7%|▋         | 2929/41250 [7:03:58<92:43:28,  8.71s/it][2025-04-25 15:01:42,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 15:01:42,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.58 | bwd_microstep: 5698.88 | bwd_inner_microstep: 5677.37 | bwd_allreduce_microstep: 21.46 | step_microstep: 19.50
[2025-04-25 15:01:42,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.58 | bwd: 5698.89 | bwd_inner: 5677.37 | bwd_allreduce: 21.48 | step: 19.50

  7%|▋         | 2930/41250 [7:04:07<92:25:08,  8.68s/it]
                                                         
{'loss': 0.2163, 'grad_norm': 1.4269944429397583, 'learning_rate': 3.982376948507393e-05, 'epoch': 0.71}

  7%|▋         | 2930/41250 [7:04:07<92:25:08,  8.68s/it][2025-04-25 15:01:50,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 15:01:50,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.13 | bwd_microstep: 5709.23 | bwd_inner_microstep: 5696.25 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.78
[2025-04-25 15:01:50,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.13 | bwd: 5709.24 | bwd_inner: 5696.25 | bwd_allreduce: 12.95 | step: 18.78

  7%|▋         | 2931/41250 [7:04:16<92:17:43,  8.67s/it]
                                                         
{'loss': 0.2067, 'grad_norm': 1.2355514764785767, 'learning_rate': 3.982356142013041e-05, 'epoch': 0.71}

  7%|▋         | 2931/41250 [7:04:16<92:17:43,  8.67s/it][2025-04-25 15:01:59,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 15:01:59,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.53 | bwd_microstep: 5794.76 | bwd_inner_microstep: 5782.00 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.74
[2025-04-25 15:01:59,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.53 | bwd: 5794.77 | bwd_inner: 5782.00 | bwd_allreduce: 12.74 | step: 18.75

  7%|▋         | 2932/41250 [7:04:24<92:36:08,  8.70s/it]
                                                         
{'loss': 0.1054, 'grad_norm': 1.2387255430221558, 'learning_rate': 3.9823353232978533e-05, 'epoch': 0.71}

  7%|▋         | 2932/41250 [7:04:24<92:36:08,  8.70s/it][2025-04-25 15:02:08,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 15:02:08,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.09 | bwd_microstep: 5888.53 | bwd_inner_microstep: 5875.75 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.04
[2025-04-25 15:02:08,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.09 | bwd: 5888.54 | bwd_inner: 5875.75 | bwd_allreduce: 12.75 | step: 19.04

  7%|▋         | 2933/41250 [7:04:33<93:16:58,  8.76s/it]
                                                         
{'loss': 0.0703, 'grad_norm': 0.6090761423110962, 'learning_rate': 3.982314492361958e-05, 'epoch': 0.71}

  7%|▋         | 2933/41250 [7:04:33<93:16:58,  8.76s/it][2025-04-25 15:02:17,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 15:02:17,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.14 | bwd_microstep: 5754.27 | bwd_inner_microstep: 5701.45 | bwd_allreduce_microstep: 52.76 | step_microstep: 19.09
[2025-04-25 15:02:17,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.14 | bwd: 5754.29 | bwd_inner: 5701.45 | bwd_allreduce: 52.78 | step: 19.09

  7%|▋         | 2934/41250 [7:04:42<93:03:14,  8.74s/it]
                                                         
{'loss': 0.2303, 'grad_norm': 2.8568851947784424, 'learning_rate': 3.982293649205485e-05, 'epoch': 0.71}

  7%|▋         | 2934/41250 [7:04:42<93:03:14,  8.74s/it][2025-04-25 15:02:25,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 15:02:25,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.60 | bwd_microstep: 5785.17 | bwd_inner_microstep: 5668.73 | bwd_allreduce_microstep: 116.40 | step_microstep: 19.23
[2025-04-25 15:02:25,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.60 | bwd: 5785.19 | bwd_inner: 5668.73 | bwd_allreduce: 116.41 | step: 19.23

  7%|▋         | 2935/41250 [7:04:51<92:55:18,  8.73s/it]
                                                         
{'loss': 0.1364, 'grad_norm': 0.8913183808326721, 'learning_rate': 3.9822727938285604e-05, 'epoch': 0.71}

  7%|▋         | 2935/41250 [7:04:51<92:55:18,  8.73s/it][2025-04-25 15:02:34,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 15:02:34,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5707.52 | bwd_inner_microstep: 5694.66 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.00
[2025-04-25 15:02:34,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5707.53 | bwd_inner: 5694.66 | bwd_allreduce: 12.83 | step: 19.00

  7%|▋         | 2936/41250 [7:04:59<92:38:35,  8.70s/it]
                                                         
{'loss': 0.0719, 'grad_norm': 0.8690807819366455, 'learning_rate': 3.9822519262313146e-05, 'epoch': 0.71}

  7%|▋         | 2936/41250 [7:04:59<92:38:35,  8.70s/it][2025-04-25 15:02:43,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:02:43,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.44 | bwd_microstep: 5750.54 | bwd_inner_microstep: 5704.26 | bwd_allreduce_microstep: 46.24 | step_microstep: 18.82
[2025-04-25 15:02:43,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.44 | bwd: 5750.55 | bwd_inner: 5704.26 | bwd_allreduce: 46.26 | step: 18.83

  7%|▋         | 2937/41250 [7:05:08<92:36:18,  8.70s/it]
                                                         
{'loss': 0.1858, 'grad_norm': 2.1122117042541504, 'learning_rate': 3.982231046413875e-05, 'epoch': 0.71}

  7%|▋         | 2937/41250 [7:05:08<92:36:18,  8.70s/it][2025-04-25 15:02:51,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:02:51,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.86 | bwd_microstep: 5708.21 | bwd_inner_microstep: 5695.37 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.57
[2025-04-25 15:02:51,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.86 | bwd: 5708.23 | bwd_inner: 5695.37 | bwd_allreduce: 12.82 | step: 18.57

  7%|▋         | 2938/41250 [7:05:17<92:24:47,  8.68s/it]
                                                         
{'loss': 0.3913, 'grad_norm': 3.0548644065856934, 'learning_rate': 3.982210154376371e-05, 'epoch': 0.71}

  7%|▋         | 2938/41250 [7:05:17<92:24:47,  8.68s/it][2025-04-25 15:03:00,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:03:00,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.77 | bwd_microstep: 5760.77 | bwd_inner_microstep: 5708.63 | bwd_allreduce_microstep: 52.10 | step_microstep: 18.86
[2025-04-25 15:03:00,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.77 | bwd: 5760.78 | bwd_inner: 5708.63 | bwd_allreduce: 52.11 | step: 18.86

  7%|▋         | 2939/41250 [7:05:25<92:26:00,  8.69s/it]
                                                         
{'loss': 0.1829, 'grad_norm': 0.9054433107376099, 'learning_rate': 3.982189250118932e-05, 'epoch': 0.71}

  7%|▋         | 2939/41250 [7:05:25<92:26:00,  8.69s/it][2025-04-25 15:03:09,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.91 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 15:03:09,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.28 | bwd_microstep: 5742.33 | bwd_inner_microstep: 5692.21 | bwd_allreduce_microstep: 50.06 | step_microstep: 19.68
[2025-04-25 15:03:09,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.28 | bwd: 5742.35 | bwd_inner: 5692.21 | bwd_allreduce: 50.09 | step: 19.68

  7%|▋         | 2940/41250 [7:05:34<92:23:41,  8.68s/it]
                                                         
{'loss': 0.1073, 'grad_norm': 1.9432748556137085, 'learning_rate': 3.982168333641685e-05, 'epoch': 0.71}

  7%|▋         | 2940/41250 [7:05:34<92:23:41,  8.68s/it][2025-04-25 15:03:18,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 15:03:18,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.65 | bwd_microstep: 5873.40 | bwd_inner_microstep: 5860.47 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.15
[2025-04-25 15:03:18,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.65 | bwd: 5873.41 | bwd_inner: 5860.47 | bwd_allreduce: 12.90 | step: 19.16

  7%|▋         | 2941/41250 [7:05:43<93:02:53,  8.74s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 2.1309115886688232, 'learning_rate': 3.982147404944761e-05, 'epoch': 0.71}

  7%|▋         | 2941/41250 [7:05:43<93:02:53,  8.74s/it][2025-04-25 15:03:26,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-25 15:03:26,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.43 | bwd_microstep: 5735.49 | bwd_inner_microstep: 5653.45 | bwd_allreduce_microstep: 81.99 | step_microstep: 19.13
[2025-04-25 15:03:26,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.43 | bwd: 5735.50 | bwd_inner: 5653.45 | bwd_allreduce: 82.01 | step: 19.13

  7%|▋         | 2942/41250 [7:05:51<92:46:17,  8.72s/it]
                                                         
{'loss': 0.1936, 'grad_norm': 1.514477252960205, 'learning_rate': 3.982126464028288e-05, 'epoch': 0.71}

  7%|▋         | 2942/41250 [7:05:51<92:46:17,  8.72s/it][2025-04-25 15:03:35,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 15:03:35,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5702.33 | bwd_inner_microstep: 5656.16 | bwd_allreduce_microstep: 46.11 | step_microstep: 19.25
[2025-04-25 15:03:35,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5702.35 | bwd_inner: 5656.16 | bwd_allreduce: 46.13 | step: 19.25

  7%|▋         | 2943/41250 [7:06:00<92:26:35,  8.69s/it]
                                                         
{'loss': 0.2897, 'grad_norm': 1.5703799724578857, 'learning_rate': 3.9821055108923945e-05, 'epoch': 0.71}

  7%|▋         | 2943/41250 [7:06:00<92:26:35,  8.69s/it][2025-04-25 15:03:43,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:03:43,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5720.15 | bwd_inner_microstep: 5707.25 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.88
[2025-04-25 15:03:43,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5720.16 | bwd_inner: 5707.25 | bwd_allreduce: 12.87 | step: 18.88

  7%|▋         | 2944/41250 [7:06:09<92:20:44,  8.68s/it]
                                                         
{'loss': 0.1224, 'grad_norm': 1.627408742904663, 'learning_rate': 3.982084545537211e-05, 'epoch': 0.71}

  7%|▋         | 2944/41250 [7:06:09<92:20:44,  8.68s/it][2025-04-25 15:03:52,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:03:52,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.84 | bwd_microstep: 5859.02 | bwd_inner_microstep: 5663.55 | bwd_allreduce_microstep: 195.42 | step_microstep: 18.96
[2025-04-25 15:03:52,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.84 | bwd: 5859.03 | bwd_inner: 5663.55 | bwd_allreduce: 195.44 | step: 18.96

  7%|▋         | 2945/41250 [7:06:18<92:40:43,  8.71s/it]
                                                         
{'loss': 0.1839, 'grad_norm': 1.9287307262420654, 'learning_rate': 3.982063567962866e-05, 'epoch': 0.71}

  7%|▋         | 2945/41250 [7:06:18<92:40:43,  8.71s/it][2025-04-25 15:04:01,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 15:04:01,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.74 | bwd_microstep: 5684.65 | bwd_inner_microstep: 5646.22 | bwd_allreduce_microstep: 38.38 | step_microstep: 19.29
[2025-04-25 15:04:01,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.74 | bwd: 5684.66 | bwd_inner: 5646.22 | bwd_allreduce: 38.40 | step: 19.29

  7%|▋         | 2946/41250 [7:06:26<92:19:20,  8.68s/it]
                                                         
{'loss': 0.3203, 'grad_norm': 2.384270191192627, 'learning_rate': 3.982042578169489e-05, 'epoch': 0.71}

  7%|▋         | 2946/41250 [7:06:26<92:19:20,  8.68s/it][2025-04-25 15:04:09,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 15:04:09,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5707.09 | bwd_inner_microstep: 5693.80 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.31
[2025-04-25 15:04:09,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5707.11 | bwd_inner: 5693.80 | bwd_allreduce: 13.26 | step: 19.32

  7%|▋         | 2947/41250 [7:06:35<92:12:24,  8.67s/it]
                                                         
{'loss': 0.2205, 'grad_norm': 2.055342674255371, 'learning_rate': 3.982021576157209e-05, 'epoch': 0.71}

  7%|▋         | 2947/41250 [7:06:35<92:12:24,  8.67s/it][2025-04-25 15:04:18,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 15:04:18,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5860.88 | bwd_inner_microstep: 5685.75 | bwd_allreduce_microstep: 175.08 | step_microstep: 18.75
[2025-04-25 15:04:18,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5860.89 | bwd_inner: 5685.75 | bwd_allreduce: 175.10 | step: 18.76

  7%|▋         | 2948/41250 [7:06:44<92:36:02,  8.70s/it]
                                                         
{'loss': 0.1424, 'grad_norm': 2.003546714782715, 'learning_rate': 3.982000561926156e-05, 'epoch': 0.71}

  7%|▋         | 2948/41250 [7:06:44<92:36:02,  8.70s/it][2025-04-25 15:04:27,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:04:27,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.36 | bwd_microstep: 5746.45 | bwd_inner_microstep: 5652.98 | bwd_allreduce_microstep: 93.42 | step_microstep: 18.73
[2025-04-25 15:04:27,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.37 | bwd: 5746.46 | bwd_inner: 5652.98 | bwd_allreduce: 93.45 | step: 18.73

  7%|▋         | 2949/41250 [7:06:52<92:27:25,  8.69s/it]
                                                         
{'loss': 0.1443, 'grad_norm': 1.1924240589141846, 'learning_rate': 3.98197953547646e-05, 'epoch': 0.71}

  7%|▋         | 2949/41250 [7:06:52<92:27:25,  8.69s/it][2025-04-25 15:04:36,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:04:36,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5684.28 | bwd_inner_microstep: 5643.91 | bwd_allreduce_microstep: 40.33 | step_microstep: 18.41
[2025-04-25 15:04:36,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5684.30 | bwd_inner: 5643.91 | bwd_allreduce: 40.35 | step: 18.42

  7%|▋         | 2950/41250 [7:07:01<92:09:19,  8.66s/it]
                                                         
{'loss': 0.1841, 'grad_norm': 1.244462013244629, 'learning_rate': 3.98195849680825e-05, 'epoch': 0.72}

  7%|▋         | 2950/41250 [7:07:01<92:09:19,  8.66s/it][2025-04-25 15:04:44,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:04:44,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5708.35 | bwd_inner_microstep: 5695.65 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.78
[2025-04-25 15:04:44,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5708.36 | bwd_inner: 5695.65 | bwd_allreduce: 12.67 | step: 18.78

  7%|▋         | 2951/41250 [7:07:09<92:05:23,  8.66s/it]
                                                         
{'loss': 0.179, 'grad_norm': 2.619075059890747, 'learning_rate': 3.981937445921655e-05, 'epoch': 0.72}

  7%|▋         | 2951/41250 [7:07:09<92:05:23,  8.66s/it][2025-04-25 15:04:53,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 15:04:53,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5717.65 | bwd_inner_microstep: 5688.01 | bwd_allreduce_microstep: 29.59 | step_microstep: 19.26
[2025-04-25 15:04:53,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5717.66 | bwd_inner: 5688.01 | bwd_allreduce: 29.61 | step: 19.27

  7%|▋         | 2952/41250 [7:07:18<92:04:13,  8.65s/it]
                                                         
{'loss': 0.2942, 'grad_norm': 2.6352059841156006, 'learning_rate': 3.981916382816805e-05, 'epoch': 0.72}

  7%|▋         | 2952/41250 [7:07:18<92:04:13,  8.65s/it][2025-04-25 15:05:01,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:05:01,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5710.00 | bwd_inner_microstep: 5697.31 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.07
[2025-04-25 15:05:01,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5710.01 | bwd_inner: 5697.31 | bwd_allreduce: 12.66 | step: 19.07

  7%|▋         | 2953/41250 [7:07:27<92:02:37,  8.65s/it]
                                                         
{'loss': 0.1787, 'grad_norm': 1.3027772903442383, 'learning_rate': 3.981895307493831e-05, 'epoch': 0.72}

  7%|▋         | 2953/41250 [7:07:27<92:02:37,  8.65s/it][2025-04-25 15:05:10,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:05:10,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.77 | bwd_microstep: 5731.07 | bwd_inner_microstep: 5674.47 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.99
[2025-04-25 15:05:10,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.77 | bwd: 5731.09 | bwd_inner: 5674.47 | bwd_allreduce: 56.58 | step: 18.99

  7%|▋         | 2954/41250 [7:07:35<92:04:23,  8.66s/it]
                                                         
{'loss': 0.0667, 'grad_norm': 1.5306333303451538, 'learning_rate': 3.9818742199528626e-05, 'epoch': 0.72}

  7%|▋         | 2954/41250 [7:07:35<92:04:23,  8.66s/it][2025-04-25 15:05:19,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 15:05:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.54 | bwd_microstep: 5737.22 | bwd_inner_microstep: 5696.25 | bwd_allreduce_microstep: 40.93 | step_microstep: 18.19
[2025-04-25 15:05:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.54 | bwd: 5737.24 | bwd_inner: 5696.25 | bwd_allreduce: 40.94 | step: 18.19

  7%|▋         | 2955/41250 [7:07:44<92:07:22,  8.66s/it]
                                                         
{'loss': 0.2155, 'grad_norm': 1.5343900918960571, 'learning_rate': 3.9818531201940295e-05, 'epoch': 0.72}

  7%|▋         | 2955/41250 [7:07:44<92:07:22,  8.66s/it][2025-04-25 15:05:27,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:05:27,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.15 | bwd_microstep: 5698.26 | bwd_inner_microstep: 5685.53 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.46
[2025-04-25 15:05:27,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.15 | bwd: 5698.28 | bwd_inner: 5685.53 | bwd_allreduce: 12.70 | step: 18.46

  7%|▋         | 2956/41250 [7:07:53<92:00:10,  8.65s/it]
                                                         
{'loss': 0.2753, 'grad_norm': 2.354431390762329, 'learning_rate': 3.981832008217461e-05, 'epoch': 0.72}

  7%|▋         | 2956/41250 [7:07:53<92:00:10,  8.65s/it][2025-04-25 15:05:36,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:05:36,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.99 | bwd_microstep: 5678.36 | bwd_inner_microstep: 5647.13 | bwd_allreduce_microstep: 31.18 | step_microstep: 18.36
[2025-04-25 15:05:36,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.99 | bwd: 5678.37 | bwd_inner: 5647.13 | bwd_allreduce: 31.20 | step: 18.36

  7%|▋         | 2957/41250 [7:08:01<91:47:37,  8.63s/it]
                                                         
{'loss': 0.1155, 'grad_norm': 1.140820860862732, 'learning_rate': 3.981810884023288e-05, 'epoch': 0.72}

  7%|▋         | 2957/41250 [7:08:01<91:47:37,  8.63s/it][2025-04-25 15:05:45,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:05:45,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.26 | bwd_microstep: 5740.84 | bwd_inner_microstep: 5681.93 | bwd_allreduce_microstep: 58.87 | step_microstep: 18.58
[2025-04-25 15:05:45,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.27 | bwd: 5740.85 | bwd_inner: 5681.93 | bwd_allreduce: 58.89 | step: 18.58

  7%|▋         | 2958/41250 [7:08:10<91:53:26,  8.64s/it]
                                                         
{'loss': 0.3686, 'grad_norm': 2.100323438644409, 'learning_rate': 3.9817897476116415e-05, 'epoch': 0.72}

  7%|▋         | 2958/41250 [7:08:10<91:53:26,  8.64s/it][2025-04-25 15:05:53,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:05:53,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.02 | bwd_microstep: 5690.47 | bwd_inner_microstep: 5643.22 | bwd_allreduce_microstep: 47.20 | step_microstep: 18.79
[2025-04-25 15:05:53,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.02 | bwd: 5690.48 | bwd_inner: 5643.22 | bwd_allreduce: 47.22 | step: 18.79

  7%|▋         | 2959/41250 [7:08:19<91:44:37,  8.63s/it]
                                                         
{'loss': 0.4166, 'grad_norm': 2.246344566345215, 'learning_rate': 3.98176859898265e-05, 'epoch': 0.72}

  7%|▋         | 2959/41250 [7:08:19<91:44:37,  8.63s/it][2025-04-25 15:06:02,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.18 | optimizer_step: 1.04
[2025-04-25 15:06:02,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.75 | bwd_microstep: 5696.60 | bwd_inner_microstep: 5683.53 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.46
[2025-04-25 15:06:02,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.75 | bwd: 5696.61 | bwd_inner: 5683.53 | bwd_allreduce: 13.04 | step: 19.47

  7%|▋         | 2960/41250 [7:08:27<91:43:27,  8.62s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 1.0961142778396606, 'learning_rate': 3.981747438136445e-05, 'epoch': 0.72}

  7%|▋         | 2960/41250 [7:08:27<91:43:27,  8.62s/it][2025-04-25 15:06:11,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:06:11,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.10 | bwd_microstep: 5780.45 | bwd_inner_microstep: 5767.86 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.49
[2025-04-25 15:06:11,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.10 | bwd: 5780.46 | bwd_inner: 5767.86 | bwd_allreduce: 12.56 | step: 18.49

  7%|▋         | 2961/41250 [7:08:36<92:06:37,  8.66s/it]
                                                         
{'loss': 0.0671, 'grad_norm': 0.521877110004425, 'learning_rate': 3.9817262650731576e-05, 'epoch': 0.72}

  7%|▋         | 2961/41250 [7:08:36<92:06:37,  8.66s/it][2025-04-25 15:06:19,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 15:06:19,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.56 | bwd_microstep: 5699.58 | bwd_inner_microstep: 5686.39 | bwd_allreduce_microstep: 13.14 | step_microstep: 18.99
[2025-04-25 15:06:19,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.56 | bwd: 5699.60 | bwd_inner: 5686.39 | bwd_allreduce: 13.16 | step: 18.98

  7%|▋         | 2962/41250 [7:08:45<91:59:33,  8.65s/it]
                                                         
{'loss': 0.1368, 'grad_norm': 1.1244908571243286, 'learning_rate': 3.981705079792917e-05, 'epoch': 0.72}

  7%|▋         | 2962/41250 [7:08:45<91:59:33,  8.65s/it][2025-04-25 15:06:28,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:06:28,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.10 | bwd_microstep: 5754.58 | bwd_inner_microstep: 5650.26 | bwd_allreduce_microstep: 104.27 | step_microstep: 18.87
[2025-04-25 15:06:28,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.10 | bwd: 5754.60 | bwd_inner: 5650.26 | bwd_allreduce: 104.29 | step: 18.87

  7%|▋         | 2963/41250 [7:08:53<92:01:45,  8.65s/it]
                                                         
{'loss': 0.1678, 'grad_norm': 1.4001164436340332, 'learning_rate': 3.981683882295855e-05, 'epoch': 0.72}

  7%|▋         | 2963/41250 [7:08:53<92:01:45,  8.65s/it][2025-04-25 15:06:36,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:06:36,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.16 | bwd_microstep: 5691.72 | bwd_inner_microstep: 5645.32 | bwd_allreduce_microstep: 46.35 | step_microstep: 19.00
[2025-04-25 15:06:36,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.16 | bwd: 5691.73 | bwd_inner: 5645.32 | bwd_allreduce: 46.37 | step: 19.00

  7%|▋         | 2964/41250 [7:09:02<91:51:00,  8.64s/it]
                                                         
{'loss': 0.1187, 'grad_norm': 0.983153223991394, 'learning_rate': 3.981662672582101e-05, 'epoch': 0.72}

  7%|▋         | 2964/41250 [7:09:02<91:51:00,  8.64s/it][2025-04-25 15:06:45,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:06:45,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5710.04 | bwd_inner_microstep: 5697.11 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.87
[2025-04-25 15:06:45,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.76 | bwd: 5710.05 | bwd_inner: 5697.10 | bwd_allreduce: 12.90 | step: 18.87

  7%|▋         | 2965/41250 [7:09:10<91:53:00,  8.64s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.2688336372375488, 'learning_rate': 3.981641450651787e-05, 'epoch': 0.72}

  7%|▋         | 2965/41250 [7:09:10<91:53:00,  8.64s/it][2025-04-25 15:06:54,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:06:54,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5680.28 | bwd_inner_microstep: 5662.47 | bwd_allreduce_microstep: 17.76 | step_microstep: 18.97
[2025-04-25 15:06:54,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5680.29 | bwd_inner: 5662.47 | bwd_allreduce: 17.78 | step: 18.97

  7%|▋         | 2966/41250 [7:09:19<91:43:44,  8.63s/it]
                                                         
{'loss': 0.2324, 'grad_norm': 1.7521053552627563, 'learning_rate': 3.981620216505042e-05, 'epoch': 0.72}

  7%|▋         | 2966/41250 [7:09:19<91:43:44,  8.63s/it][2025-04-25 15:07:02,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.21 | optimizer_step: 1.08
[2025-04-25 15:07:02,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.87 | bwd_microstep: 5762.57 | bwd_inner_microstep: 5644.57 | bwd_allreduce_microstep: 117.95 | step_microstep: 19.73
[2025-04-25 15:07:02,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.87 | bwd: 5762.59 | bwd_inner: 5644.57 | bwd_allreduce: 117.97 | step: 19.73

  7%|▋         | 2967/41250 [7:09:28<91:53:28,  8.64s/it]
                                                         
{'loss': 0.1062, 'grad_norm': 1.5573574304580688, 'learning_rate': 3.981598970141999e-05, 'epoch': 0.72}

  7%|▋         | 2967/41250 [7:09:28<91:53:28,  8.64s/it][2025-04-25 15:07:11,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:07:11,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5761.01 | bwd_inner_microstep: 5656.69 | bwd_allreduce_microstep: 104.28 | step_microstep: 18.83
[2025-04-25 15:07:11,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5761.03 | bwd_inner: 5656.69 | bwd_allreduce: 104.30 | step: 18.83

  7%|▋         | 2968/41250 [7:09:36<91:58:28,  8.65s/it]
                                                         
{'loss': 0.2555, 'grad_norm': 1.5406414270401, 'learning_rate': 3.981577711562788e-05, 'epoch': 0.72}

  7%|▋         | 2968/41250 [7:09:36<91:58:28,  8.65s/it][2025-04-25 15:07:20,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 15:07:20,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.62 | bwd_microstep: 5682.43 | bwd_inner_microstep: 5649.63 | bwd_allreduce_microstep: 32.76 | step_microstep: 19.54
[2025-04-25 15:07:20,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.62 | bwd: 5682.45 | bwd_inner: 5649.63 | bwd_allreduce: 32.78 | step: 19.54

  7%|▋         | 2969/41250 [7:09:45<91:47:03,  8.63s/it]
                                                         
{'loss': 0.198, 'grad_norm': 1.4377949237823486, 'learning_rate': 3.9815564407675406e-05, 'epoch': 0.72}

  7%|▋         | 2969/41250 [7:09:45<91:47:03,  8.63s/it][2025-04-25 15:07:28,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:07:28,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.51 | bwd_microstep: 5689.85 | bwd_inner_microstep: 5649.66 | bwd_allreduce_microstep: 40.14 | step_microstep: 18.54
[2025-04-25 15:07:28,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.51 | bwd: 5689.86 | bwd_inner: 5649.66 | bwd_allreduce: 40.16 | step: 18.55

  7%|▋         | 2970/41250 [7:09:54<91:41:02,  8.62s/it]
                                                         
{'loss': 0.1049, 'grad_norm': 1.2518858909606934, 'learning_rate': 3.981535157756387e-05, 'epoch': 0.72}

  7%|▋         | 2970/41250 [7:09:54<91:41:02,  8.62s/it][2025-04-25 15:07:37,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:07:37,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.64 | bwd_microstep: 5849.59 | bwd_inner_microstep: 5684.81 | bwd_allreduce_microstep: 164.73 | step_microstep: 18.57
[2025-04-25 15:07:37,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.64 | bwd: 5849.60 | bwd_inner: 5684.81 | bwd_allreduce: 164.75 | step: 18.58

  7%|▋         | 2971/41250 [7:10:02<92:09:42,  8.67s/it]
                                                         
{'loss': 0.2471, 'grad_norm': 1.0727593898773193, 'learning_rate': 3.981513862529459e-05, 'epoch': 0.72}

  7%|▋         | 2971/41250 [7:10:02<92:09:42,  8.67s/it][2025-04-25 15:07:46,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:07:46,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5718.14 | bwd_inner_microstep: 5705.58 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.40
[2025-04-25 15:07:46,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5718.16 | bwd_inner: 5705.58 | bwd_allreduce: 12.54 | step: 18.40

  7%|▋         | 2972/41250 [7:10:11<92:05:54,  8.66s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.0204825401306152, 'learning_rate': 3.981492555086888e-05, 'epoch': 0.72}

  7%|▋         | 2972/41250 [7:10:11<92:05:54,  8.66s/it][2025-04-25 15:07:54,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 15:07:54,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.79 | bwd_microstep: 5772.01 | bwd_inner_microstep: 5655.69 | bwd_allreduce_microstep: 116.27 | step_microstep: 19.16
[2025-04-25 15:07:54,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.79 | bwd: 5772.03 | bwd_inner: 5655.69 | bwd_allreduce: 116.29 | step: 19.16

  7%|▋         | 2973/41250 [7:10:20<92:10:14,  8.67s/it]
                                                         
{'loss': 0.1017, 'grad_norm': 1.1891005039215088, 'learning_rate': 3.981471235428805e-05, 'epoch': 0.72}

  7%|▋         | 2973/41250 [7:10:20<92:10:14,  8.67s/it][2025-04-25 15:08:03,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 15:08:03,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5890.70 | bwd_inner_microstep: 5659.70 | bwd_allreduce_microstep: 230.95 | step_microstep: 19.09
[2025-04-25 15:08:03,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5890.71 | bwd_inner: 5659.70 | bwd_allreduce: 230.97 | step: 19.10

  7%|▋         | 2974/41250 [7:10:29<92:35:58,  8.71s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 0.7671331763267517, 'learning_rate': 3.981449903555341e-05, 'epoch': 0.72}

  7%|▋         | 2974/41250 [7:10:29<92:35:58,  8.71s/it][2025-04-25 15:08:12,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 15:08:12,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5745.35 | bwd_inner_microstep: 5712.20 | bwd_allreduce_microstep: 33.10 | step_microstep: 18.66
[2025-04-25 15:08:12,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5745.36 | bwd_inner: 5712.20 | bwd_allreduce: 33.11 | step: 18.66

  7%|▋         | 2975/41250 [7:10:37<92:30:50,  8.70s/it]
                                                         
{'loss': 0.2283, 'grad_norm': 1.3492178916931152, 'learning_rate': 3.9814285594666286e-05, 'epoch': 0.72}

  7%|▋         | 2975/41250 [7:10:37<92:30:50,  8.70s/it][2025-04-25 15:08:21,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:08:21,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.82 | bwd_microstep: 5768.91 | bwd_inner_microstep: 5756.37 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.69
[2025-04-25 15:08:21,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.82 | bwd: 5768.92 | bwd_inner: 5756.37 | bwd_allreduce: 12.51 | step: 18.70

  7%|▋         | 2976/41250 [7:10:46<92:36:21,  8.71s/it]
                                                         
{'loss': 0.1363, 'grad_norm': 1.3831192255020142, 'learning_rate': 3.9814072031627986e-05, 'epoch': 0.72}

  7%|▋         | 2976/41250 [7:10:46<92:36:21,  8.71s/it][2025-04-25 15:08:29,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:08:29,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.89 | bwd_microstep: 5729.50 | bwd_inner_microstep: 5716.78 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.61
[2025-04-25 15:08:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.89 | bwd: 5729.52 | bwd_inner: 5716.78 | bwd_allreduce: 12.69 | step: 18.61

  7%|▋         | 2977/41250 [7:10:55<92:28:33,  8.70s/it]
                                                         
{'loss': 0.114, 'grad_norm': 1.6690294742584229, 'learning_rate': 3.981385834643983e-05, 'epoch': 0.72}

  7%|▋         | 2977/41250 [7:10:55<92:28:33,  8.70s/it][2025-04-25 15:08:38,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 15:08:38,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.14 | bwd_microstep: 5708.48 | bwd_inner_microstep: 5666.99 | bwd_allreduce_microstep: 41.44 | step_microstep: 19.28
[2025-04-25 15:08:38,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.14 | bwd: 5708.49 | bwd_inner: 5666.99 | bwd_allreduce: 41.46 | step: 19.28

  7%|▋         | 2978/41250 [7:11:03<92:15:00,  8.68s/it]
                                                         
{'loss': 0.3186, 'grad_norm': 3.477177143096924, 'learning_rate': 3.981364453910314e-05, 'epoch': 0.72}

  7%|▋         | 2978/41250 [7:11:03<92:15:00,  8.68s/it][2025-04-25 15:08:47,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:08:47,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5713.98 | bwd_inner_microstep: 5657.30 | bwd_allreduce_microstep: 56.64 | step_microstep: 18.55
[2025-04-25 15:08:47,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5714.00 | bwd_inner: 5657.30 | bwd_allreduce: 56.65 | step: 18.55

  7%|▋         | 2979/41250 [7:11:12<92:04:42,  8.66s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 1.6344516277313232, 'learning_rate': 3.9813430609619224e-05, 'epoch': 0.72}

  7%|▋         | 2979/41250 [7:11:12<92:04:42,  8.66s/it][2025-04-25 15:08:55,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 15:08:55,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.05 | bwd_microstep: 5754.71 | bwd_inner_microstep: 5665.25 | bwd_allreduce_microstep: 89.41 | step_microstep: 18.79
[2025-04-25 15:08:55,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.05 | bwd: 5754.72 | bwd_inner: 5665.25 | bwd_allreduce: 89.43 | step: 18.79

  7%|▋         | 2980/41250 [7:11:21<92:08:04,  8.67s/it]
                                                         
{'loss': 0.1702, 'grad_norm': 2.0565383434295654, 'learning_rate': 3.98132165579894e-05, 'epoch': 0.72}

  7%|▋         | 2980/41250 [7:11:21<92:08:04,  8.67s/it][2025-04-25 15:09:04,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 15:09:04,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.01 | bwd_microstep: 5732.77 | bwd_inner_microstep: 5717.55 | bwd_allreduce_microstep: 15.18 | step_microstep: 18.64
[2025-04-25 15:09:04,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.01 | bwd: 5732.79 | bwd_inner: 5717.55 | bwd_allreduce: 15.20 | step: 18.65

  7%|▋         | 2981/41250 [7:11:29<92:10:10,  8.67s/it]
                                                         
{'loss': 0.2424, 'grad_norm': 2.3790061473846436, 'learning_rate': 3.9813002384215004e-05, 'epoch': 0.72}

  7%|▋         | 2981/41250 [7:11:29<92:10:10,  8.67s/it][2025-04-25 15:09:13,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:09:13,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.77 | bwd_microstep: 5740.37 | bwd_inner_microstep: 5702.72 | bwd_allreduce_microstep: 37.61 | step_microstep: 18.66
[2025-04-25 15:09:13,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.77 | bwd: 5740.38 | bwd_inner: 5702.72 | bwd_allreduce: 37.62 | step: 18.66

  7%|▋         | 2982/41250 [7:11:38<92:13:11,  8.68s/it]
                                                         
{'loss': 0.2799, 'grad_norm': 2.1911659240722656, 'learning_rate': 3.981278808829734e-05, 'epoch': 0.72}

  7%|▋         | 2982/41250 [7:11:38<92:13:11,  8.68s/it][2025-04-25 15:09:21,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-25 15:09:21,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.95 | bwd_microstep: 5722.38 | bwd_inner_microstep: 5709.61 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.98
[2025-04-25 15:09:21,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.95 | bwd: 5722.39 | bwd_inner: 5709.61 | bwd_allreduce: 12.74 | step: 18.98

  7%|▋         | 2983/41250 [7:11:47<92:12:46,  8.67s/it]
                                                         
{'loss': 0.2076, 'grad_norm': 1.8857325315475464, 'learning_rate': 3.9812573670237735e-05, 'epoch': 0.72}

  7%|▋         | 2983/41250 [7:11:47<92:12:46,  8.67s/it][2025-04-25 15:09:30,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 15:09:30,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.56 | bwd_microstep: 5721.06 | bwd_inner_microstep: 5666.47 | bwd_allreduce_microstep: 54.54 | step_microstep: 19.01
[2025-04-25 15:09:30,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.56 | bwd: 5721.07 | bwd_inner: 5666.47 | bwd_allreduce: 54.56 | step: 19.01

  7%|▋         | 2984/41250 [7:11:55<92:05:17,  8.66s/it]
                                                         
{'loss': 0.2052, 'grad_norm': 2.781087875366211, 'learning_rate': 3.981235913003751e-05, 'epoch': 0.72}

  7%|▋         | 2984/41250 [7:11:55<92:05:17,  8.66s/it][2025-04-25 15:09:39,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:09:39,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.60 | bwd_microstep: 5782.90 | bwd_inner_microstep: 5769.95 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.72
[2025-04-25 15:09:39,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.61 | bwd: 5782.92 | bwd_inner: 5769.95 | bwd_allreduce: 12.93 | step: 18.72

  7%|▋         | 2985/41250 [7:12:04<92:22:08,  8.69s/it]
                                                         
{'loss': 0.1217, 'grad_norm': 1.2877782583236694, 'learning_rate': 3.981214446769798e-05, 'epoch': 0.72}

  7%|▋         | 2985/41250 [7:12:04<92:22:08,  8.69s/it][2025-04-25 15:09:47,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:09:47,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.54 | bwd_microstep: 5754.03 | bwd_inner_microstep: 5659.92 | bwd_allreduce_microstep: 94.07 | step_microstep: 18.56
[2025-04-25 15:09:47,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.54 | bwd: 5754.04 | bwd_inner: 5659.92 | bwd_allreduce: 94.08 | step: 18.56

  7%|▋         | 2986/41250 [7:12:13<92:19:20,  8.69s/it]
                                                         
{'loss': 0.1827, 'grad_norm': 2.1621053218841553, 'learning_rate': 3.9811929683220485e-05, 'epoch': 0.72}

  7%|▋         | 2986/41250 [7:12:13<92:19:20,  8.69s/it][2025-04-25 15:09:56,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:09:56,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.75 | bwd_microstep: 5756.05 | bwd_inner_microstep: 5714.68 | bwd_allreduce_microstep: 41.33 | step_microstep: 18.75
[2025-04-25 15:09:56,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.75 | bwd: 5756.06 | bwd_inner: 5714.68 | bwd_allreduce: 41.35 | step: 18.75

  7%|▋         | 2987/41250 [7:12:21<92:21:02,  8.69s/it]
                                                         
{'loss': 0.1575, 'grad_norm': 2.5688583850860596, 'learning_rate': 3.981171477660634e-05, 'epoch': 0.72}

  7%|▋         | 2987/41250 [7:12:21<92:21:02,  8.69s/it][2025-04-25 15:10:05,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:10:05,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.42 | bwd_microstep: 5704.55 | bwd_inner_microstep: 5659.71 | bwd_allreduce_microstep: 44.81 | step_microstep: 18.60
[2025-04-25 15:10:05,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.42 | bwd: 5704.57 | bwd_inner: 5659.71 | bwd_allreduce: 44.82 | step: 18.60

  7%|▋         | 2988/41250 [7:12:30<92:06:41,  8.67s/it]
                                                         
{'loss': 0.2258, 'grad_norm': 1.3109369277954102, 'learning_rate': 3.9811499747856875e-05, 'epoch': 0.72}

  7%|▋         | 2988/41250 [7:12:30<92:06:41,  8.67s/it][2025-04-25 15:10:13,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:10:13,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5721.89 | bwd_inner_microstep: 5708.97 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.65
[2025-04-25 15:10:13,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5721.90 | bwd_inner: 5708.97 | bwd_allreduce: 12.89 | step: 18.65

  7%|▋         | 2989/41250 [7:12:39<92:04:58,  8.66s/it]
                                                         
{'loss': 0.2354, 'grad_norm': 1.5153729915618896, 'learning_rate': 3.98112845969734e-05, 'epoch': 0.72}

  7%|▋         | 2989/41250 [7:12:39<92:04:58,  8.66s/it][2025-04-25 15:10:22,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 15:10:22,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.31 | bwd_microstep: 5734.71 | bwd_inner_microstep: 5721.37 | bwd_allreduce_microstep: 13.29 | step_microstep: 18.81
[2025-04-25 15:10:22,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.31 | bwd: 5734.73 | bwd_inner: 5721.37 | bwd_allreduce: 13.31 | step: 18.81

  7%|▋         | 2990/41250 [7:12:47<92:08:08,  8.67s/it]
                                                         
{'loss': 0.3751, 'grad_norm': 2.1730871200561523, 'learning_rate': 3.981106932395726e-05, 'epoch': 0.72}

  7%|▋         | 2990/41250 [7:12:47<92:08:08,  8.67s/it][2025-04-25 15:10:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 15:10:31,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.95 | bwd_microstep: 5716.77 | bwd_inner_microstep: 5703.97 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.07
[2025-04-25 15:10:31,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.95 | bwd: 5716.78 | bwd_inner: 5703.97 | bwd_allreduce: 12.77 | step: 19.07

  7%|▋         | 2991/41250 [7:12:56<92:06:32,  8.67s/it]
                                                         
{'loss': 0.1003, 'grad_norm': 1.2027134895324707, 'learning_rate': 3.981085392880977e-05, 'epoch': 0.73}

  7%|▋         | 2991/41250 [7:12:56<92:06:32,  8.67s/it][2025-04-25 15:10:39,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:10:39,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.31 | bwd_microstep: 5756.40 | bwd_inner_microstep: 5722.09 | bwd_allreduce_microstep: 34.27 | step_microstep: 18.70
[2025-04-25 15:10:39,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.31 | bwd: 5756.41 | bwd_inner: 5722.09 | bwd_allreduce: 34.29 | step: 18.70

  7%|▋         | 2992/41250 [7:13:05<92:11:55,  8.68s/it]
                                                         
{'loss': 0.201, 'grad_norm': 1.3017234802246094, 'learning_rate': 3.9810638411532264e-05, 'epoch': 0.73}

  7%|▋         | 2992/41250 [7:13:05<92:11:55,  8.68s/it][2025-04-25 15:10:48,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:10:48,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.56 | bwd_microstep: 5694.32 | bwd_inner_microstep: 5662.55 | bwd_allreduce_microstep: 31.73 | step_microstep: 18.55
[2025-04-25 15:10:48,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.56 | bwd: 5694.34 | bwd_inner: 5662.55 | bwd_allreduce: 31.75 | step: 18.56

  7%|▋         | 2993/41250 [7:13:13<92:00:24,  8.66s/it]
                                                         
{'loss': 0.1346, 'grad_norm': 1.422821044921875, 'learning_rate': 3.9810422772126064e-05, 'epoch': 0.73}

  7%|▋         | 2993/41250 [7:13:13<92:00:24,  8.66s/it][2025-04-25 15:10:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 15:10:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.06 | bwd_microstep: 5733.14 | bwd_inner_microstep: 5711.63 | bwd_allreduce_microstep: 21.46 | step_microstep: 18.88
[2025-04-25 15:10:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.06 | bwd: 5733.15 | bwd_inner: 5711.63 | bwd_allreduce: 21.48 | step: 18.88

  7%|▋         | 2994/41250 [7:13:22<92:04:35,  8.66s/it]
                                                         
{'loss': 0.1535, 'grad_norm': 1.0198227167129517, 'learning_rate': 3.981020701059251e-05, 'epoch': 0.73}

  7%|▋         | 2994/41250 [7:13:22<92:04:35,  8.66s/it][2025-04-25 15:11:05,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:11:05,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.91 | bwd_microstep: 5726.06 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 29.07 | step_microstep: 18.66
[2025-04-25 15:11:05,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.91 | bwd: 5726.07 | bwd_inner: 5696.95 | bwd_allreduce: 29.08 | step: 18.66

  7%|▋         | 2995/41250 [7:13:31<92:05:42,  8.67s/it]
                                                         
{'loss': 0.1531, 'grad_norm': 1.2878947257995605, 'learning_rate': 3.9809991126932924e-05, 'epoch': 0.73}

  7%|▋         | 2995/41250 [7:13:31<92:05:42,  8.67s/it][2025-04-25 15:11:14,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:11:14,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.47 | bwd_microstep: 5759.31 | bwd_inner_microstep: 5702.89 | bwd_allreduce_microstep: 56.38 | step_microstep: 18.63
[2025-04-25 15:11:14,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.47 | bwd: 5759.32 | bwd_inner: 5702.89 | bwd_allreduce: 56.39 | step: 18.63

  7%|▋         | 2996/41250 [7:13:39<92:10:17,  8.67s/it]
                                                         
{'loss': 0.2603, 'grad_norm': 1.6627157926559448, 'learning_rate': 3.980977512114864e-05, 'epoch': 0.73}

  7%|▋         | 2996/41250 [7:13:39<92:10:17,  8.67s/it][2025-04-25 15:11:23,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:11:23,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5873.99 | bwd_inner_microstep: 5646.99 | bwd_allreduce_microstep: 226.95 | step_microstep: 18.55
[2025-04-25 15:11:23,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5874.01 | bwd_inner: 5646.99 | bwd_allreduce: 226.97 | step: 18.55

  7%|▋         | 2997/41250 [7:13:48<92:30:19,  8.71s/it]
                                                         
{'loss': 0.2723, 'grad_norm': 1.5253629684448242, 'learning_rate': 3.980955899324099e-05, 'epoch': 0.73}

  7%|▋         | 2997/41250 [7:13:48<92:30:19,  8.71s/it][2025-04-25 15:11:31,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:11:31,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5748.82 | bwd_inner_microstep: 5675.98 | bwd_allreduce_microstep: 72.79 | step_microstep: 18.44
[2025-04-25 15:11:31,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5748.84 | bwd_inner: 5675.98 | bwd_allreduce: 72.81 | step: 18.44

  7%|▋         | 2998/41250 [7:13:57<92:25:24,  8.70s/it]
                                                         
{'loss': 0.1417, 'grad_norm': 1.4323267936706543, 'learning_rate': 3.98093427432113e-05, 'epoch': 0.73}

  7%|▋         | 2998/41250 [7:13:57<92:25:24,  8.70s/it][2025-04-25 15:11:40,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 15:11:40,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.59 | bwd_microstep: 5743.61 | bwd_inner_microstep: 5638.62 | bwd_allreduce_microstep: 104.93 | step_microstep: 18.61
[2025-04-25 15:11:40,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.59 | bwd: 5743.63 | bwd_inner: 5638.62 | bwd_allreduce: 104.96 | step: 18.61

  7%|▋         | 2999/41250 [7:14:05<92:18:35,  8.69s/it]
                                                         
{'loss': 0.4163, 'grad_norm': 1.9904303550720215, 'learning_rate': 3.980912637106091e-05, 'epoch': 0.73}

  7%|▋         | 2999/41250 [7:14:05<92:18:35,  8.69s/it][2025-04-25 15:11:49,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.34 | optimizer_step: 1.04
[2025-04-25 15:11:49,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.80 | bwd_microstep: 5667.94 | bwd_inner_microstep: 5653.24 | bwd_allreduce_microstep: 14.64 | step_microstep: 20.39
[2025-04-25 15:11:49,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.80 | bwd: 5667.96 | bwd_inner: 5653.24 | bwd_allreduce: 14.67 | step: 20.39

  7%|▋         | 3000/41250 [7:14:14<92:02:18,  8.66s/it]
                                                         
{'loss': 0.2492, 'grad_norm': 1.2772743701934814, 'learning_rate': 3.9808909876791155e-05, 'epoch': 0.73}

  7%|▋         | 3000/41250 [7:14:14<92:02:18,  8.66s/it][2025-04-25 15:11:58,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 15:11:58,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 6017.63 | bwd_inner_microstep: 5689.04 | bwd_allreduce_microstep: 328.55 | step_microstep: 18.77
[2025-04-25 15:11:58,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 6017.65 | bwd_inner: 5689.04 | bwd_allreduce: 328.57 | step: 18.77

  7%|▋         | 3001/41250 [7:14:23<92:57:39,  8.75s/it]
                                                         
{'loss': 0.163, 'grad_norm': 1.5422122478485107, 'learning_rate': 3.9808693260403364e-05, 'epoch': 0.73}

  7%|▋         | 3001/41250 [7:14:23<92:57:39,  8.75s/it][2025-04-25 15:12:06,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:12:06,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.01 | bwd_microstep: 5675.02 | bwd_inner_microstep: 5656.55 | bwd_allreduce_microstep: 18.42 | step_microstep: 18.83
[2025-04-25 15:12:06,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.01 | bwd: 5675.03 | bwd_inner: 5656.55 | bwd_allreduce: 18.44 | step: 18.83

  7%|▋         | 3002/41250 [7:14:32<92:25:07,  8.70s/it]
                                                         
{'loss': 0.238, 'grad_norm': 1.3734122514724731, 'learning_rate': 3.980847652189888e-05, 'epoch': 0.73}

  7%|▋         | 3002/41250 [7:14:32<92:25:07,  8.70s/it][2025-04-25 15:12:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:12:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5715.63 | bwd_inner_microstep: 5702.80 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.62
[2025-04-25 15:12:15,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5715.65 | bwd_inner: 5702.80 | bwd_allreduce: 12.81 | step: 18.62

  7%|▋         | 3003/41250 [7:14:40<92:15:09,  8.68s/it]
                                                         
{'loss': 0.2501, 'grad_norm': 1.3782426118850708, 'learning_rate': 3.9808259661279026e-05, 'epoch': 0.73}

  7%|▋         | 3003/41250 [7:14:40<92:15:09,  8.68s/it][2025-04-25 15:12:24,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 15:12:24,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.33 | bwd_microstep: 5702.75 | bwd_inner_microstep: 5690.02 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.92
[2025-04-25 15:12:24,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.33 | bwd: 5702.76 | bwd_inner: 5690.02 | bwd_allreduce: 12.70 | step: 18.92

  7%|▋         | 3004/41250 [7:14:49<92:05:13,  8.67s/it]
                                                         
{'loss': 0.125, 'grad_norm': 0.8497679829597473, 'learning_rate': 3.9808042678545155e-05, 'epoch': 0.73}

  7%|▋         | 3004/41250 [7:14:49<92:05:13,  8.67s/it][2025-04-25 15:12:32,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:12:32,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.91 | bwd_microstep: 5789.70 | bwd_inner_microstep: 5776.98 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.70
[2025-04-25 15:12:32,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.91 | bwd: 5789.72 | bwd_inner: 5776.97 | bwd_allreduce: 12.70 | step: 18.71

  7%|▋         | 3005/41250 [7:14:58<92:25:04,  8.70s/it]
                                                         
{'loss': 0.1963, 'grad_norm': 1.310036063194275, 'learning_rate': 3.98078255736986e-05, 'epoch': 0.73}

  7%|▋         | 3005/41250 [7:14:58<92:25:04,  8.70s/it][2025-04-25 15:12:41,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 15:12:41,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.87 | bwd_microstep: 5669.88 | bwd_inner_microstep: 5656.98 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.65
[2025-04-25 15:12:41,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.87 | bwd: 5669.90 | bwd_inner: 5656.98 | bwd_allreduce: 12.88 | step: 18.65

  7%|▋         | 3006/41250 [7:15:06<92:04:49,  8.67s/it]
                                                         
{'loss': 0.1833, 'grad_norm': 1.6856027841567993, 'learning_rate': 3.980760834674069e-05, 'epoch': 0.73}

  7%|▋         | 3006/41250 [7:15:06<92:04:49,  8.67s/it][2025-04-25 15:12:50,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 15:12:50,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 5732.97 | bwd_inner_microstep: 5673.01 | bwd_allreduce_microstep: 59.89 | step_microstep: 18.72
[2025-04-25 15:12:50,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5732.99 | bwd_inner: 5673.01 | bwd_allreduce: 59.92 | step: 18.72

  7%|▋         | 3007/41250 [7:15:15<92:06:04,  8.67s/it]
                                                         
{'loss': 0.2938, 'grad_norm': 1.4659333229064941, 'learning_rate': 3.980739099767277e-05, 'epoch': 0.73}

  7%|▋         | 3007/41250 [7:15:15<92:06:04,  8.67s/it][2025-04-25 15:12:58,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 15:12:58,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.02 | bwd_microstep: 5755.10 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 108.59 | step_microstep: 18.74
[2025-04-25 15:12:58,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.02 | bwd: 5755.11 | bwd_inner: 5646.46 | bwd_allreduce: 108.61 | step: 18.74

  7%|▋         | 3008/41250 [7:15:24<92:04:27,  8.67s/it]
                                                         
{'loss': 0.2482, 'grad_norm': 1.4555790424346924, 'learning_rate': 3.980717352649618e-05, 'epoch': 0.73}

  7%|▋         | 3008/41250 [7:15:24<92:04:27,  8.67s/it][2025-04-25 15:13:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 15:13:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5691.71 | bwd_inner_microstep: 5678.87 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.94
[2025-04-25 15:13:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5691.73 | bwd_inner: 5678.87 | bwd_allreduce: 12.81 | step: 18.94

  7%|▋         | 3009/41250 [7:15:32<91:55:19,  8.65s/it]
                                                         
{'loss': 0.0908, 'grad_norm': 0.8946712613105774, 'learning_rate': 3.980695593321226e-05, 'epoch': 0.73}

  7%|▋         | 3009/41250 [7:15:32<91:55:19,  8.65s/it][2025-04-25 15:13:15,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:13:15,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5700.59 | bwd_inner_microstep: 5687.83 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.75
[2025-04-25 15:13:15,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5700.60 | bwd_inner: 5687.83 | bwd_allreduce: 12.73 | step: 18.76

  7%|▋         | 3010/41250 [7:15:41<91:51:02,  8.65s/it]
                                                         
{'loss': 0.1139, 'grad_norm': 0.8909087777137756, 'learning_rate': 3.980673821782236e-05, 'epoch': 0.73}

  7%|▋         | 3010/41250 [7:15:41<91:51:02,  8.65s/it][2025-04-25 15:13:24,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:13:24,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.13 | bwd_microstep: 5746.21 | bwd_inner_microstep: 5632.32 | bwd_allreduce_microstep: 113.84 | step_microstep: 18.55
[2025-04-25 15:13:24,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.13 | bwd: 5746.22 | bwd_inner: 5632.32 | bwd_allreduce: 113.86 | step: 18.55

  7%|▋         | 3011/41250 [7:15:49<91:50:17,  8.65s/it]
                                                         
{'loss': 0.1337, 'grad_norm': 1.0342649221420288, 'learning_rate': 3.980652038032781e-05, 'epoch': 0.73}

  7%|▋         | 3011/41250 [7:15:49<91:50:17,  8.65s/it][2025-04-25 15:13:33,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 15:13:33,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.71 | bwd_microstep: 5732.71 | bwd_inner_microstep: 5685.30 | bwd_allreduce_microstep: 47.36 | step_microstep: 18.69
[2025-04-25 15:13:33,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.71 | bwd: 5732.72 | bwd_inner: 5685.30 | bwd_allreduce: 47.38 | step: 18.69

  7%|▋         | 3012/41250 [7:15:58<91:54:35,  8.65s/it]
                                                         
{'loss': 0.2298, 'grad_norm': 1.1758345365524292, 'learning_rate': 3.9806302420729956e-05, 'epoch': 0.73}

  7%|▋         | 3012/41250 [7:15:58<91:54:35,  8.65s/it][2025-04-25 15:13:41,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:13:41,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.33 | bwd_microstep: 5739.53 | bwd_inner_microstep: 5637.48 | bwd_allreduce_microstep: 102.00 | step_microstep: 18.92
[2025-04-25 15:13:41,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.33 | bwd: 5739.54 | bwd_inner: 5637.48 | bwd_allreduce: 102.02 | step: 18.93

  7%|▋         | 3013/41250 [7:16:07<91:52:49,  8.65s/it]
                                                         
{'loss': 0.1627, 'grad_norm': 0.8664552569389343, 'learning_rate': 3.980608433903014e-05, 'epoch': 0.73}

  7%|▋         | 3013/41250 [7:16:07<91:52:49,  8.65s/it][2025-04-25 15:13:50,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:13:50,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.71 | bwd_microstep: 5674.17 | bwd_inner_microstep: 5655.64 | bwd_allreduce_microstep: 18.49 | step_microstep: 18.94
[2025-04-25 15:13:50,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.71 | bwd: 5674.18 | bwd_inner: 5655.64 | bwd_allreduce: 18.50 | step: 18.94

  7%|▋         | 3014/41250 [7:16:15<91:39:46,  8.63s/it]
                                                         
{'loss': 0.136, 'grad_norm': 0.9330837726593018, 'learning_rate': 3.9805866135229716e-05, 'epoch': 0.73}

  7%|▋         | 3014/41250 [7:16:15<91:39:46,  8.63s/it][2025-04-25 15:13:59,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:13:59,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.64 | bwd_microstep: 5714.23 | bwd_inner_microstep: 5701.43 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.45
[2025-04-25 15:13:59,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.64 | bwd: 5714.24 | bwd_inner: 5701.43 | bwd_allreduce: 12.77 | step: 18.45

  7%|▋         | 3015/41250 [7:16:24<91:43:52,  8.64s/it]
                                                         
{'loss': 0.073, 'grad_norm': 0.5579280257225037, 'learning_rate': 3.9805647809330026e-05, 'epoch': 0.73}

  7%|▋         | 3015/41250 [7:16:24<91:43:52,  8.64s/it][2025-04-25 15:14:07,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:14:07,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 5754.61 | bwd_inner_microstep: 5645.99 | bwd_allreduce_microstep: 108.57 | step_microstep: 18.42
[2025-04-25 15:14:07,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 5754.63 | bwd_inner: 5645.99 | bwd_allreduce: 108.60 | step: 18.42

  7%|▋         | 3016/41250 [7:16:33<91:48:09,  8.64s/it]
                                                         
{'loss': 0.2543, 'grad_norm': 1.8683035373687744, 'learning_rate': 3.9805429361332406e-05, 'epoch': 0.73}

  7%|▋         | 3016/41250 [7:16:33<91:48:09,  8.64s/it][2025-04-25 15:14:16,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 15:14:16,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.50 | bwd_microstep: 5676.51 | bwd_inner_microstep: 5643.35 | bwd_allreduce_microstep: 33.12 | step_microstep: 18.79
[2025-04-25 15:14:16,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.50 | bwd: 5676.52 | bwd_inner: 5643.35 | bwd_allreduce: 33.13 | step: 18.79

  7%|▋         | 3017/41250 [7:16:41<91:37:11,  8.63s/it]
                                                         
{'loss': 0.2682, 'grad_norm': 1.385556697845459, 'learning_rate': 3.980521079123821e-05, 'epoch': 0.73}

  7%|▋         | 3017/41250 [7:16:41<91:37:11,  8.63s/it][2025-04-25 15:14:24,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:14:24,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.78 | bwd_microstep: 5669.56 | bwd_inner_microstep: 5633.34 | bwd_allreduce_microstep: 36.17 | step_microstep: 18.54
[2025-04-25 15:14:24,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.78 | bwd: 5669.58 | bwd_inner: 5633.34 | bwd_allreduce: 36.19 | step: 18.55

  7%|▋         | 3018/41250 [7:16:50<91:27:21,  8.61s/it]
                                                         
{'loss': 0.1097, 'grad_norm': 1.38275945186615, 'learning_rate': 3.980499209904879e-05, 'epoch': 0.73}

  7%|▋         | 3018/41250 [7:16:50<91:27:21,  8.61s/it][2025-04-25 15:14:33,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:14:33,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5694.55 | bwd_inner_microstep: 5681.69 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.61
[2025-04-25 15:14:33,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5694.56 | bwd_inner: 5681.69 | bwd_allreduce: 12.83 | step: 18.61

  7%|▋         | 3019/41250 [7:16:58<91:30:12,  8.62s/it]
                                                         
{'loss': 0.2398, 'grad_norm': 2.296011209487915, 'learning_rate': 3.980477328476548e-05, 'epoch': 0.73}

  7%|▋         | 3019/41250 [7:16:58<91:30:12,  8.62s/it][2025-04-25 15:14:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:14:42,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.14 | bwd_microstep: 5718.80 | bwd_inner_microstep: 5680.73 | bwd_allreduce_microstep: 38.04 | step_microstep: 18.60
[2025-04-25 15:14:42,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.14 | bwd: 5718.81 | bwd_inner: 5680.73 | bwd_allreduce: 38.05 | step: 18.60

  7%|▋         | 3020/41250 [7:17:07<91:34:31,  8.62s/it]
                                                         
{'loss': 0.1796, 'grad_norm': 1.2071179151535034, 'learning_rate': 3.980455434838964e-05, 'epoch': 0.73}

  7%|▋         | 3020/41250 [7:17:07<91:34:31,  8.62s/it][2025-04-25 15:14:50,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:14:50,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.73 | bwd_microstep: 5698.86 | bwd_inner_microstep: 5686.11 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.57
[2025-04-25 15:14:50,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.73 | bwd: 5698.87 | bwd_inner: 5686.11 | bwd_allreduce: 12.72 | step: 18.57

  7%|▋         | 3021/41250 [7:17:16<91:37:26,  8.63s/it]
                                                         
{'loss': 0.1552, 'grad_norm': 1.5497206449508667, 'learning_rate': 3.9804335289922624e-05, 'epoch': 0.73}

  7%|▋         | 3021/41250 [7:17:16<91:37:26,  8.63s/it][2025-04-25 15:14:59,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:14:59,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.84 | bwd_microstep: 5722.40 | bwd_inner_microstep: 5639.07 | bwd_allreduce_microstep: 83.29 | step_microstep: 18.68
[2025-04-25 15:14:59,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.84 | bwd: 5722.41 | bwd_inner: 5639.07 | bwd_allreduce: 83.31 | step: 18.68

  7%|▋         | 3022/41250 [7:17:24<91:38:55,  8.63s/it]
                                                         
{'loss': 0.1874, 'grad_norm': 1.7122057676315308, 'learning_rate': 3.9804116109365775e-05, 'epoch': 0.73}

  7%|▋         | 3022/41250 [7:17:24<91:38:55,  8.63s/it][2025-04-25 15:15:08,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:15:08,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.12 | bwd_microstep: 5726.08 | bwd_inner_microstep: 5702.26 | bwd_allreduce_microstep: 23.78 | step_microstep: 18.99
[2025-04-25 15:15:08,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.12 | bwd: 5726.10 | bwd_inner: 5702.26 | bwd_allreduce: 23.80 | step: 18.99

  7%|▋         | 3023/41250 [7:17:33<91:43:34,  8.64s/it]
                                                         
{'loss': 0.2325, 'grad_norm': 1.2694684267044067, 'learning_rate': 3.980389680672044e-05, 'epoch': 0.73}

  7%|▋         | 3023/41250 [7:17:33<91:43:34,  8.64s/it][2025-04-25 15:15:16,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:15:16,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5736.85 | bwd_inner_microstep: 5679.94 | bwd_allreduce_microstep: 56.87 | step_microstep: 18.79
[2025-04-25 15:15:16,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.42 | bwd: 5736.87 | bwd_inner: 5679.94 | bwd_allreduce: 56.89 | step: 18.79

  7%|▋         | 3024/41250 [7:17:42<91:48:56,  8.65s/it]
                                                         
{'loss': 0.1465, 'grad_norm': 1.676259160041809, 'learning_rate': 3.980367738198798e-05, 'epoch': 0.73}

  7%|▋         | 3024/41250 [7:17:42<91:48:56,  8.65s/it][2025-04-25 15:15:25,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 15:15:25,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.31 | bwd_microstep: 5899.41 | bwd_inner_microstep: 5683.19 | bwd_allreduce_microstep: 216.16 | step_microstep: 20.23
[2025-04-25 15:15:25,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.31 | bwd: 5899.43 | bwd_inner: 5683.19 | bwd_allreduce: 216.19 | step: 20.23

  7%|▋         | 3025/41250 [7:17:51<92:23:59,  8.70s/it]
                                                         
{'loss': 0.1105, 'grad_norm': 1.0174919366836548, 'learning_rate': 3.980345783516975e-05, 'epoch': 0.73}

  7%|▋         | 3025/41250 [7:17:51<92:23:59,  8.70s/it][2025-04-25 15:15:34,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 15:15:34,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5719.28 | bwd_inner_microstep: 5706.56 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.50
[2025-04-25 15:15:34,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5719.29 | bwd_inner: 5706.56 | bwd_allreduce: 12.69 | step: 18.50

  7%|▋         | 3026/41250 [7:17:59<92:14:37,  8.69s/it]
                                                         
{'loss': 0.242, 'grad_norm': 1.4906516075134277, 'learning_rate': 3.980323816626709e-05, 'epoch': 0.73}

  7%|▋         | 3026/41250 [7:17:59<92:14:37,  8.69s/it][2025-04-25 15:15:42,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:15:42,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.45 | bwd_microstep: 5696.92 | bwd_inner_microstep: 5684.25 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.64
[2025-04-25 15:15:42,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.45 | bwd: 5696.93 | bwd_inner: 5684.25 | bwd_allreduce: 12.64 | step: 18.64

  7%|▋         | 3027/41250 [7:18:08<92:01:53,  8.67s/it]
                                                         
{'loss': 0.2013, 'grad_norm': 2.08191180229187, 'learning_rate': 3.980301837528137e-05, 'epoch': 0.73}

  7%|▋         | 3027/41250 [7:18:08<92:01:53,  8.67s/it][2025-04-25 15:15:51,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 15:15:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5682.46 | bwd_inner_microstep: 5654.70 | bwd_allreduce_microstep: 27.71 | step_microstep: 18.83
[2025-04-25 15:15:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5682.47 | bwd_inner: 5654.70 | bwd_allreduce: 27.73 | step: 18.83

  7%|▋         | 3028/41250 [7:18:16<91:48:51,  8.65s/it]
                                                         
{'loss': 0.0943, 'grad_norm': 0.9292616844177246, 'learning_rate': 3.9802798462213925e-05, 'epoch': 0.73}

  7%|▋         | 3028/41250 [7:18:16<91:48:51,  8.65s/it][2025-04-25 15:16:00,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:16:00,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.48 | bwd_microstep: 5711.11 | bwd_inner_microstep: 5698.51 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.38
[2025-04-25 15:16:00,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.48 | bwd: 5711.12 | bwd_inner: 5698.51 | bwd_allreduce: 12.57 | step: 18.38

  7%|▋         | 3029/41250 [7:18:25<91:48:06,  8.65s/it]
                                                         
{'loss': 0.2233, 'grad_norm': 1.8603909015655518, 'learning_rate': 3.9802578427066135e-05, 'epoch': 0.73}

  7%|▋         | 3029/41250 [7:18:25<91:48:06,  8.65s/it][2025-04-25 15:16:08,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:16:08,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.87 | bwd_microstep: 5724.45 | bwd_inner_microstep: 5702.80 | bwd_allreduce_microstep: 21.60 | step_microstep: 18.93
[2025-04-25 15:16:08,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.87 | bwd: 5724.46 | bwd_inner: 5702.80 | bwd_allreduce: 21.62 | step: 18.94

  7%|▋         | 3030/41250 [7:18:34<91:49:57,  8.65s/it]
                                                         
{'loss': 0.0602, 'grad_norm': 1.4786407947540283, 'learning_rate': 3.980235826983933e-05, 'epoch': 0.73}

  7%|▋         | 3030/41250 [7:18:34<91:49:57,  8.65s/it][2025-04-25 15:16:17,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:16:17,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.10 | bwd_microstep: 5751.55 | bwd_inner_microstep: 5663.16 | bwd_allreduce_microstep: 88.35 | step_microstep: 18.76
[2025-04-25 15:16:17,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.10 | bwd: 5751.57 | bwd_inner: 5663.16 | bwd_allreduce: 88.37 | step: 18.76

  7%|▋         | 3031/41250 [7:18:42<91:54:08,  8.66s/it]
                                                         
{'loss': 0.1642, 'grad_norm': 1.3250428438186646, 'learning_rate': 3.980213799053489e-05, 'epoch': 0.73}

  7%|▋         | 3031/41250 [7:18:42<91:54:08,  8.66s/it][2025-04-25 15:16:26,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:16:26,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.86 | bwd_microstep: 5710.74 | bwd_inner_microstep: 5662.25 | bwd_allreduce_microstep: 48.44 | step_microstep: 18.85
[2025-04-25 15:16:26,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.86 | bwd: 5710.75 | bwd_inner: 5662.25 | bwd_allreduce: 48.46 | step: 18.85

  7%|▋         | 3032/41250 [7:18:51<91:47:51,  8.65s/it]
                                                         
{'loss': 0.1262, 'grad_norm': 1.3386458158493042, 'learning_rate': 3.980191758915416e-05, 'epoch': 0.74}

  7%|▋         | 3032/41250 [7:18:51<91:47:51,  8.65s/it][2025-04-25 15:16:34,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:16:34,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.57 | bwd_microstep: 5709.84 | bwd_inner_microstep: 5697.33 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.83
[2025-04-25 15:16:34,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.57 | bwd: 5709.86 | bwd_inner: 5697.33 | bwd_allreduce: 12.49 | step: 18.83

  7%|▋         | 3033/41250 [7:19:00<91:48:52,  8.65s/it]
                                                         
{'loss': 0.1606, 'grad_norm': 1.5876498222351074, 'learning_rate': 3.980169706569851e-05, 'epoch': 0.74}

  7%|▋         | 3033/41250 [7:19:00<91:48:52,  8.65s/it][2025-04-25 15:16:43,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:16:43,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.03 | bwd_microstep: 5762.50 | bwd_inner_microstep: 5709.41 | bwd_allreduce_microstep: 53.04 | step_microstep: 18.67
[2025-04-25 15:16:43,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.03 | bwd: 5762.52 | bwd_inner: 5709.41 | bwd_allreduce: 53.06 | step: 18.67

  7%|▋         | 3034/41250 [7:19:08<91:57:24,  8.66s/it]
                                                         
{'loss': 0.2306, 'grad_norm': 2.0355100631713867, 'learning_rate': 3.980147642016929e-05, 'epoch': 0.74}

  7%|▋         | 3034/41250 [7:19:08<91:57:24,  8.66s/it][2025-04-25 15:16:52,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-25 15:16:52,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.63 | bwd_microstep: 5734.77 | bwd_inner_microstep: 5718.94 | bwd_allreduce_microstep: 15.76 | step_microstep: 20.05
[2025-04-25 15:16:52,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.63 | bwd: 5734.78 | bwd_inner: 5718.94 | bwd_allreduce: 15.79 | step: 20.05

  7%|▋         | 3035/41250 [7:19:17<91:59:11,  8.67s/it]
                                                         
{'loss': 0.2604, 'grad_norm': 1.7042796611785889, 'learning_rate': 3.9801255652567855e-05, 'epoch': 0.74}

  7%|▋         | 3035/41250 [7:19:17<91:59:11,  8.67s/it][2025-04-25 15:17:00,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 15:17:00,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5784.98 | bwd_inner_microstep: 5664.14 | bwd_allreduce_microstep: 120.80 | step_microstep: 18.70
[2025-04-25 15:17:00,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5785.00 | bwd_inner: 5664.14 | bwd_allreduce: 120.82 | step: 18.70

  7%|▋         | 3036/41250 [7:19:26<92:04:44,  8.67s/it]
                                                         
{'loss': 0.0288, 'grad_norm': 0.45716822147369385, 'learning_rate': 3.980103476289558e-05, 'epoch': 0.74}

  7%|▋         | 3036/41250 [7:19:26<92:04:44,  8.67s/it][2025-04-25 15:17:09,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 15:17:09,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.90 | bwd_microstep: 5710.87 | bwd_inner_microstep: 5659.32 | bwd_allreduce_microstep: 51.50 | step_microstep: 18.39
[2025-04-25 15:17:09,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.90 | bwd: 5710.88 | bwd_inner: 5659.32 | bwd_allreduce: 51.52 | step: 18.39

  7%|▋         | 3037/41250 [7:19:34<91:53:50,  8.66s/it]
                                                         
{'loss': 0.1647, 'grad_norm': 0.6710938811302185, 'learning_rate': 3.980081375115382e-05, 'epoch': 0.74}

  7%|▋         | 3037/41250 [7:19:34<91:53:50,  8.66s/it][2025-04-25 15:17:18,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 15:17:18,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.66 | bwd_microstep: 5702.70 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 45.20 | step_microstep: 18.45
[2025-04-25 15:17:18,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.66 | bwd: 5702.71 | bwd_inner: 5657.45 | bwd_allreduce: 45.22 | step: 18.46

  7%|▋         | 3038/41250 [7:19:43<91:44:17,  8.64s/it]
                                                         
{'loss': 0.2188, 'grad_norm': 2.8270561695098877, 'learning_rate': 3.9800592617343936e-05, 'epoch': 0.74}

  7%|▋         | 3038/41250 [7:19:43<91:44:17,  8.64s/it][2025-04-25 15:17:26,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:17:26,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.76 | bwd_microstep: 5785.19 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 141.45 | step_microstep: 18.62
[2025-04-25 15:17:26,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.76 | bwd: 5785.20 | bwd_inner: 5643.69 | bwd_allreduce: 141.46 | step: 18.62

  7%|▋         | 3039/41250 [7:19:52<91:53:21,  8.66s/it]
                                                         
{'loss': 0.2318, 'grad_norm': 2.3682146072387695, 'learning_rate': 3.98003713614673e-05, 'epoch': 0.74}

  7%|▋         | 3039/41250 [7:19:52<91:53:21,  8.66s/it][2025-04-25 15:17:35,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-25 15:17:35,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.12 | bwd_microstep: 5766.96 | bwd_inner_microstep: 5664.57 | bwd_allreduce_microstep: 102.35 | step_microstep: 18.64
[2025-04-25 15:17:35,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.12 | bwd: 5766.98 | bwd_inner: 5664.57 | bwd_allreduce: 102.36 | step: 18.64

  7%|▋         | 3040/41250 [7:20:00<91:56:48,  8.66s/it]
                                                         
{'loss': 0.0653, 'grad_norm': 1.2310254573822021, 'learning_rate': 3.980014998352526e-05, 'epoch': 0.74}

  7%|▋         | 3040/41250 [7:20:00<91:56:48,  8.66s/it][2025-04-25 15:17:44,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:17:44,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.37 | bwd_microstep: 5691.29 | bwd_inner_microstep: 5667.83 | bwd_allreduce_microstep: 23.42 | step_microstep: 18.57
[2025-04-25 15:17:44,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.37 | bwd: 5691.30 | bwd_inner: 5667.83 | bwd_allreduce: 23.44 | step: 18.57

  7%|▋         | 3041/41250 [7:20:09<91:45:42,  8.65s/it]
                                                         
{'loss': 0.2873, 'grad_norm': 1.8924007415771484, 'learning_rate': 3.979992848351919e-05, 'epoch': 0.74}

  7%|▋         | 3041/41250 [7:20:09<91:45:42,  8.65s/it][2025-04-25 15:17:52,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 15:17:52,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.30 | bwd_microstep: 5774.84 | bwd_inner_microstep: 5657.60 | bwd_allreduce_microstep: 117.20 | step_microstep: 18.98
[2025-04-25 15:17:52,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.30 | bwd: 5774.85 | bwd_inner: 5657.59 | bwd_allreduce: 117.22 | step: 18.98

  7%|▋         | 3042/41250 [7:20:18<91:53:12,  8.66s/it]
                                                         
{'loss': 0.1889, 'grad_norm': 1.6736246347427368, 'learning_rate': 3.979970686145046e-05, 'epoch': 0.74}

  7%|▋         | 3042/41250 [7:20:18<91:53:12,  8.66s/it][2025-04-25 15:18:01,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:18:01,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5795.08 | bwd_inner_microstep: 5692.13 | bwd_allreduce_microstep: 102.91 | step_microstep: 18.64
[2025-04-25 15:18:01,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5795.09 | bwd_inner: 5692.13 | bwd_allreduce: 102.92 | step: 18.65

  7%|▋         | 3043/41250 [7:20:26<92:06:06,  8.68s/it]
                                                         
{'loss': 0.3013, 'grad_norm': 2.918947219848633, 'learning_rate': 3.979948511732043e-05, 'epoch': 0.74}

  7%|▋         | 3043/41250 [7:20:26<92:06:06,  8.68s/it][2025-04-25 15:18:10,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:18:10,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5773.88 | bwd_inner_microstep: 5664.73 | bwd_allreduce_microstep: 109.09 | step_microstep: 18.88
[2025-04-25 15:18:10,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5773.89 | bwd_inner: 5664.73 | bwd_allreduce: 109.11 | step: 18.88

  7%|▋         | 3044/41250 [7:20:35<92:07:06,  8.68s/it]
                                                         
{'loss': 0.1396, 'grad_norm': 0.8704910278320312, 'learning_rate': 3.979926325113047e-05, 'epoch': 0.74}

  7%|▋         | 3044/41250 [7:20:35<92:07:06,  8.68s/it][2025-04-25 15:18:18,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 15:18:18,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.70 | bwd_microstep: 5709.70 | bwd_inner_microstep: 5666.47 | bwd_allreduce_microstep: 43.19 | step_microstep: 18.78
[2025-04-25 15:18:18,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.70 | bwd: 5709.72 | bwd_inner: 5666.47 | bwd_allreduce: 43.21 | step: 18.78

  7%|▋         | 3045/41250 [7:20:44<91:56:05,  8.66s/it]
                                                         
{'loss': 0.0435, 'grad_norm': 0.4969581663608551, 'learning_rate': 3.979904126288194e-05, 'epoch': 0.74}

  7%|▋         | 3045/41250 [7:20:44<91:56:05,  8.66s/it][2025-04-25 15:18:27,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 15:18:27,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.47 | bwd_microstep: 5783.07 | bwd_inner_microstep: 5701.52 | bwd_allreduce_microstep: 81.50 | step_microstep: 18.87
[2025-04-25 15:18:27,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.47 | bwd: 5783.08 | bwd_inner: 5701.52 | bwd_allreduce: 81.52 | step: 18.87

  7%|▋         | 3046/41250 [7:20:52<92:06:17,  8.68s/it]
                                                         
{'loss': 0.1934, 'grad_norm': 2.2045862674713135, 'learning_rate': 3.979881915257622e-05, 'epoch': 0.74}

  7%|▋         | 3046/41250 [7:20:52<92:06:17,  8.68s/it][2025-04-25 15:18:36,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:18:36,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.01 | bwd_microstep: 5704.96 | bwd_inner_microstep: 5692.13 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.95
[2025-04-25 15:18:36,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.01 | bwd: 5704.98 | bwd_inner: 5692.13 | bwd_allreduce: 12.81 | step: 18.95

  7%|▋         | 3047/41250 [7:21:01<91:57:27,  8.67s/it]
                                                         
{'loss': 0.0945, 'grad_norm': 1.1531838178634644, 'learning_rate': 3.979859692021468e-05, 'epoch': 0.74}

  7%|▋         | 3047/41250 [7:21:01<91:57:27,  8.67s/it][2025-04-25 15:18:44,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 15:18:44,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.38 | bwd_microstep: 5722.79 | bwd_inner_microstep: 5704.65 | bwd_allreduce_microstep: 18.09 | step_microstep: 18.63
[2025-04-25 15:18:44,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.39 | bwd: 5722.81 | bwd_inner: 5704.65 | bwd_allreduce: 18.11 | step: 18.63

  7%|▋         | 3048/41250 [7:21:10<91:56:47,  8.66s/it]
                                                         
{'loss': 0.1384, 'grad_norm': 1.7456849813461304, 'learning_rate': 3.979837456579867e-05, 'epoch': 0.74}

  7%|▋         | 3048/41250 [7:21:10<91:56:47,  8.66s/it][2025-04-25 15:18:53,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:18:53,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.47 | bwd_microstep: 5706.85 | bwd_inner_microstep: 5660.34 | bwd_allreduce_microstep: 46.47 | step_microstep: 18.56
[2025-04-25 15:18:53,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.47 | bwd: 5706.86 | bwd_inner: 5660.34 | bwd_allreduce: 46.48 | step: 18.56

  7%|▋         | 3049/41250 [7:21:18<91:47:38,  8.65s/it]
                                                         
{'loss': 0.0739, 'grad_norm': 1.150856614112854, 'learning_rate': 3.979815208932959e-05, 'epoch': 0.74}

  7%|▋         | 3049/41250 [7:21:18<91:47:38,  8.65s/it][2025-04-25 15:19:02,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:19:02,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.96 | bwd_microstep: 5859.54 | bwd_inner_microstep: 5702.45 | bwd_allreduce_microstep: 157.04 | step_microstep: 18.92
[2025-04-25 15:19:02,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.96 | bwd: 5859.55 | bwd_inner: 5702.45 | bwd_allreduce: 157.06 | step: 18.92

  7%|▋         | 3050/41250 [7:21:27<92:15:31,  8.69s/it]
                                                         
{'loss': 0.2989, 'grad_norm': 1.5606586933135986, 'learning_rate': 3.979792949080879e-05, 'epoch': 0.74}

  7%|▋         | 3050/41250 [7:21:27<92:15:31,  8.69s/it][2025-04-25 15:19:10,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 15:19:10,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5704.80 | bwd_inner_microstep: 5692.12 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.35
[2025-04-25 15:19:10,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5704.81 | bwd_inner: 5692.12 | bwd_allreduce: 12.65 | step: 18.35

  7%|▋         | 3051/41250 [7:21:36<92:04:17,  8.68s/it]
                                                         
{'loss': 0.2713, 'grad_norm': 1.3399227857589722, 'learning_rate': 3.979770677023765e-05, 'epoch': 0.74}

  7%|▋         | 3051/41250 [7:21:36<92:04:17,  8.68s/it][2025-04-25 15:19:19,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:19:19,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.19 | bwd_microstep: 5737.69 | bwd_inner_microstep: 5681.09 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.56
[2025-04-25 15:19:19,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.19 | bwd: 5737.71 | bwd_inner: 5681.09 | bwd_allreduce: 56.57 | step: 18.57

  7%|▋         | 3052/41250 [7:21:44<92:03:00,  8.68s/it]
                                                         
{'loss': 0.0602, 'grad_norm': 0.6167299151420593, 'learning_rate': 3.979748392761754e-05, 'epoch': 0.74}

  7%|▋         | 3052/41250 [7:21:44<92:03:00,  8.68s/it][2025-04-25 15:19:28,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:19:28,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.10 | bwd_microstep: 5687.91 | bwd_inner_microstep: 5652.35 | bwd_allreduce_microstep: 35.51 | step_microstep: 18.51
[2025-04-25 15:19:28,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.10 | bwd: 5687.92 | bwd_inner: 5652.35 | bwd_allreduce: 35.53 | step: 18.51

  7%|▋         | 3053/41250 [7:21:53<91:47:54,  8.65s/it]
                                                         
{'loss': 0.2146, 'grad_norm': 1.3967922925949097, 'learning_rate': 3.9797260962949836e-05, 'epoch': 0.74}

  7%|▋         | 3053/41250 [7:21:53<91:47:54,  8.65s/it][2025-04-25 15:19:36,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:19:36,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.19 | bwd_microstep: 5710.76 | bwd_inner_microstep: 5698.08 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.72
[2025-04-25 15:19:36,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.19 | bwd: 5710.78 | bwd_inner: 5698.08 | bwd_allreduce: 12.65 | step: 18.72

  7%|▋         | 3054/41250 [7:22:02<91:47:53,  8.65s/it]
                                                         
{'loss': 0.3382, 'grad_norm': 1.9533581733703613, 'learning_rate': 3.9797037876235906e-05, 'epoch': 0.74}

  7%|▋         | 3054/41250 [7:22:02<91:47:53,  8.65s/it][2025-04-25 15:19:45,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 15:19:45,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.24 | bwd_microstep: 5711.79 | bwd_inner_microstep: 5698.51 | bwd_allreduce_microstep: 13.23 | step_microstep: 18.84
[2025-04-25 15:19:45,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.24 | bwd: 5711.80 | bwd_inner: 5698.51 | bwd_allreduce: 13.25 | step: 18.84

  7%|▋         | 3055/41250 [7:22:10<91:46:11,  8.65s/it]
                                                         
{'loss': 0.2368, 'grad_norm': 1.4020731449127197, 'learning_rate': 3.979681466747714e-05, 'epoch': 0.74}

  7%|▋         | 3055/41250 [7:22:10<91:46:11,  8.65s/it][2025-04-25 15:19:54,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:19:54,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.21 | bwd_microstep: 5725.46 | bwd_inner_microstep: 5712.61 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.93
[2025-04-25 15:19:54,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.21 | bwd: 5725.48 | bwd_inner: 5712.61 | bwd_allreduce: 12.83 | step: 18.93

  7%|▋         | 3056/41250 [7:22:19<91:49:17,  8.65s/it]
                                                         
{'loss': 0.1101, 'grad_norm': 0.9528692960739136, 'learning_rate': 3.97965913366749e-05, 'epoch': 0.74}

  7%|▋         | 3056/41250 [7:22:19<91:49:17,  8.65s/it][2025-04-25 15:20:02,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.85 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:20:02,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.84 | bwd_microstep: 5723.08 | bwd_inner_microstep: 5699.89 | bwd_allreduce_microstep: 23.15 | step_microstep: 19.06
[2025-04-25 15:20:02,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.84 | bwd: 5723.10 | bwd_inner: 5699.88 | bwd_allreduce: 23.17 | step: 19.06

  7%|▋         | 3057/41250 [7:22:28<91:49:52,  8.66s/it]
                                                         
{'loss': 0.2332, 'grad_norm': 1.0337269306182861, 'learning_rate': 3.979636788383057e-05, 'epoch': 0.74}

  7%|▋         | 3057/41250 [7:22:28<91:49:52,  8.66s/it][2025-04-25 15:20:11,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 15:20:11,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.45 | bwd_microstep: 5733.92 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 36.88 | step_microstep: 19.77
[2025-04-25 15:20:11,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.45 | bwd: 5733.94 | bwd_inner: 5696.99 | bwd_allreduce: 36.90 | step: 19.77

  7%|▋         | 3058/41250 [7:22:36<91:53:52,  8.66s/it]
                                                         
{'loss': 0.1465, 'grad_norm': 1.4548829793930054, 'learning_rate': 3.979614430894553e-05, 'epoch': 0.74}

  7%|▋         | 3058/41250 [7:22:36<91:53:52,  8.66s/it][2025-04-25 15:20:20,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 15:20:20,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.56 | bwd_microstep: 5698.99 | bwd_inner_microstep: 5686.01 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.74
[2025-04-25 15:20:20,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5699.01 | bwd_inner: 5686.01 | bwd_allreduce: 12.95 | step: 18.74

  7%|▋         | 3059/41250 [7:22:45<91:47:57,  8.65s/it]
                                                         
{'loss': 0.1101, 'grad_norm': 1.381820797920227, 'learning_rate': 3.979592061202115e-05, 'epoch': 0.74}

  7%|▋         | 3059/41250 [7:22:45<91:47:57,  8.65s/it][2025-04-25 15:20:28,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:20:28,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.68 | bwd_microstep: 5734.88 | bwd_inner_microstep: 5690.17 | bwd_allreduce_microstep: 44.66 | step_microstep: 18.65
[2025-04-25 15:20:28,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.68 | bwd: 5734.90 | bwd_inner: 5690.17 | bwd_allreduce: 44.68 | step: 18.65

  7%|▋         | 3060/41250 [7:22:54<91:50:08,  8.66s/it]
                                                         
{'loss': 0.0791, 'grad_norm': 0.8169711232185364, 'learning_rate': 3.979569679305881e-05, 'epoch': 0.74}

  7%|▋         | 3060/41250 [7:22:54<91:50:08,  8.66s/it][2025-04-25 15:20:37,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:20:37,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5694.30 | bwd_inner_microstep: 5681.44 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.62
[2025-04-25 15:20:37,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5694.31 | bwd_inner: 5681.44 | bwd_allreduce: 12.83 | step: 18.62

  7%|▋         | 3061/41250 [7:23:02<91:42:46,  8.65s/it]
                                                         
{'loss': 0.1003, 'grad_norm': 0.9776933789253235, 'learning_rate': 3.97954728520599e-05, 'epoch': 0.74}

  7%|▋         | 3061/41250 [7:23:02<91:42:46,  8.65s/it][2025-04-25 15:20:45,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.26 | optimizer_step: 1.06
[2025-04-25 15:20:45,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.82 | bwd_microstep: 5701.57 | bwd_inner_microstep: 5687.68 | bwd_allreduce_microstep: 13.82 | step_microstep: 19.88
[2025-04-25 15:20:45,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.82 | bwd: 5701.58 | bwd_inner: 5687.68 | bwd_allreduce: 13.86 | step: 19.88

  7%|▋         | 3062/41250 [7:23:11<91:42:06,  8.64s/it]
                                                         
{'loss': 0.1685, 'grad_norm': 1.0470207929611206, 'learning_rate': 3.9795248789025786e-05, 'epoch': 0.74}

  7%|▋         | 3062/41250 [7:23:11<91:42:06,  8.64s/it][2025-04-25 15:20:54,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 15:20:54,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5714.68 | bwd_inner_microstep: 5702.09 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.79
[2025-04-25 15:20:54,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.65 | bwd: 5714.69 | bwd_inner: 5702.09 | bwd_allreduce: 12.56 | step: 18.79

  7%|▋         | 3063/41250 [7:23:19<91:43:18,  8.65s/it]
                                                         
{'loss': 0.1124, 'grad_norm': 0.9544084668159485, 'learning_rate': 3.9795024603957856e-05, 'epoch': 0.74}

  7%|▋         | 3063/41250 [7:23:19<91:43:18,  8.65s/it][2025-04-25 15:21:03,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 15:21:03,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.51 | bwd_microstep: 5688.69 | bwd_inner_microstep: 5644.35 | bwd_allreduce_microstep: 44.29 | step_microstep: 19.14
[2025-04-25 15:21:03,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.51 | bwd: 5688.71 | bwd_inner: 5644.35 | bwd_allreduce: 44.31 | step: 19.14

  7%|▋         | 3064/41250 [7:23:28<91:33:38,  8.63s/it]
                                                         
{'loss': 0.2814, 'grad_norm': 2.7402477264404297, 'learning_rate': 3.97948002968575e-05, 'epoch': 0.74}

  7%|▋         | 3064/41250 [7:23:28<91:33:38,  8.63s/it][2025-04-25 15:21:11,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:21:11,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.04 | bwd_microstep: 5708.56 | bwd_inner_microstep: 5695.99 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.44
[2025-04-25 15:21:11,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.04 | bwd: 5708.58 | bwd_inner: 5695.99 | bwd_allreduce: 12.55 | step: 18.44

  7%|▋         | 3065/41250 [7:23:37<91:34:33,  8.63s/it]
                                                         
{'loss': 0.088, 'grad_norm': 1.121572494506836, 'learning_rate': 3.9794575867726096e-05, 'epoch': 0.74}

  7%|▋         | 3065/41250 [7:23:37<91:34:33,  8.63s/it][2025-04-25 15:21:20,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:21:20,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5726.44 | bwd_inner_microstep: 5694.97 | bwd_allreduce_microstep: 31.43 | step_microstep: 18.73
[2025-04-25 15:21:20,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5726.46 | bwd_inner: 5694.97 | bwd_allreduce: 31.45 | step: 18.73

  7%|▋         | 3066/41250 [7:23:45<91:39:22,  8.64s/it]
                                                         
{'loss': 0.347, 'grad_norm': 1.688904881477356, 'learning_rate': 3.979435131656502e-05, 'epoch': 0.74}

  7%|▋         | 3066/41250 [7:23:45<91:39:22,  8.64s/it][2025-04-25 15:21:29,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 15:21:29,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.79 | bwd_microstep: 5735.57 | bwd_inner_microstep: 5650.12 | bwd_allreduce_microstep: 85.39 | step_microstep: 18.85
[2025-04-25 15:21:29,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.79 | bwd: 5735.59 | bwd_inner: 5650.12 | bwd_allreduce: 85.43 | step: 18.85

  7%|▋         | 3067/41250 [7:23:54<91:39:53,  8.64s/it]
                                                         
{'loss': 0.0857, 'grad_norm': 0.7395830154418945, 'learning_rate': 3.9794126643375656e-05, 'epoch': 0.74}

  7%|▋         | 3067/41250 [7:23:54<91:39:53,  8.64s/it][2025-04-25 15:21:37,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 15:21:37,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.00 | bwd_microstep: 5736.87 | bwd_inner_microstep: 5688.06 | bwd_allreduce_microstep: 48.76 | step_microstep: 19.30
[2025-04-25 15:21:37,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.00 | bwd: 5736.88 | bwd_inner: 5688.06 | bwd_allreduce: 48.78 | step: 19.31

  7%|▋         | 3068/41250 [7:24:03<91:45:31,  8.65s/it]
                                                         
{'loss': 0.0988, 'grad_norm': 0.8612889051437378, 'learning_rate': 3.97939018481594e-05, 'epoch': 0.74}

  7%|▋         | 3068/41250 [7:24:03<91:45:31,  8.65s/it][2025-04-25 15:21:46,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.14 | optimizer_step: 0.95
[2025-04-25 15:21:46,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.01 | bwd_microstep: 5724.05 | bwd_inner_microstep: 5657.68 | bwd_allreduce_microstep: 66.32 | step_microstep: 18.96
[2025-04-25 15:21:46,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.01 | bwd: 5724.06 | bwd_inner: 5657.68 | bwd_allreduce: 66.34 | step: 18.97

  7%|▋         | 3069/41250 [7:24:11<91:42:36,  8.65s/it]
                                                         
{'loss': 0.4174, 'grad_norm': 2.3260233402252197, 'learning_rate': 3.9793676930917635e-05, 'epoch': 0.74}

  7%|▋         | 3069/41250 [7:24:11<91:42:36,  8.65s/it][2025-04-25 15:21:55,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.12 | optimizer_step: 0.94
[2025-04-25 15:21:55,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.99 | bwd_microstep: 5716.95 | bwd_inner_microstep: 5693.18 | bwd_allreduce_microstep: 23.72 | step_microstep: 19.65
[2025-04-25 15:21:55,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.99 | bwd: 5716.96 | bwd_inner: 5693.18 | bwd_allreduce: 23.74 | step: 19.66

  7%|▋         | 3070/41250 [7:24:20<91:43:42,  8.65s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 1.495554804801941, 'learning_rate': 3.979345189165174e-05, 'epoch': 0.74}

  7%|▋         | 3070/41250 [7:24:20<91:43:42,  8.65s/it][2025-04-25 15:22:03,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 15:22:03,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.45 | bwd_microstep: 5695.45 | bwd_inner_microstep: 5682.49 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.04
[2025-04-25 15:22:03,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.45 | bwd: 5695.46 | bwd_inner: 5682.49 | bwd_allreduce: 12.93 | step: 19.04

  7%|▋         | 3071/41250 [7:24:29<91:38:04,  8.64s/it]
                                                         
{'loss': 0.1341, 'grad_norm': 1.8726637363433838, 'learning_rate': 3.979322673036311e-05, 'epoch': 0.74}

  7%|▋         | 3071/41250 [7:24:29<91:38:04,  8.64s/it][2025-04-25 15:22:12,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 15:22:12,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.51 | bwd_microstep: 5872.71 | bwd_inner_microstep: 5638.43 | bwd_allreduce_microstep: 234.23 | step_microstep: 19.26
[2025-04-25 15:22:12,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.51 | bwd: 5872.72 | bwd_inner: 5638.43 | bwd_allreduce: 234.25 | step: 19.26

  7%|▋         | 3072/41250 [7:24:37<92:03:45,  8.68s/it]
                                                         
{'loss': 0.107, 'grad_norm': 1.6944940090179443, 'learning_rate': 3.979300144705313e-05, 'epoch': 0.74}

  7%|▋         | 3072/41250 [7:24:37<92:03:45,  8.68s/it][2025-04-25 15:22:21,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 15:22:21,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.31 | bwd_microstep: 5745.55 | bwd_inner_microstep: 5643.27 | bwd_allreduce_microstep: 102.23 | step_microstep: 18.76
[2025-04-25 15:22:21,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.31 | bwd: 5745.57 | bwd_inner: 5643.27 | bwd_allreduce: 102.25 | step: 18.76

  7%|▋         | 3073/41250 [7:24:46<92:00:05,  8.68s/it]
                                                         
{'loss': 0.1316, 'grad_norm': 1.231881856918335, 'learning_rate': 3.9792776041723194e-05, 'epoch': 0.74}

  7%|▋         | 3073/41250 [7:24:46<92:00:05,  8.68s/it][2025-04-25 15:22:29,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-25 15:22:29,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5718.47 | bwd_inner_microstep: 5703.77 | bwd_allreduce_microstep: 14.64 | step_microstep: 18.96
[2025-04-25 15:22:29,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5718.49 | bwd_inner: 5703.77 | bwd_allreduce: 14.67 | step: 18.95

  7%|▋         | 3074/41250 [7:24:55<91:57:46,  8.67s/it]
                                                         
{'loss': 0.0963, 'grad_norm': 0.7969035506248474, 'learning_rate': 3.979255051437468e-05, 'epoch': 0.75}

  7%|▋         | 3074/41250 [7:24:55<91:57:46,  8.67s/it][2025-04-25 15:22:38,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:22:38,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.48 | bwd_microstep: 5708.45 | bwd_inner_microstep: 5695.73 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.68
[2025-04-25 15:22:38,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.48 | bwd: 5708.47 | bwd_inner: 5695.73 | bwd_allreduce: 12.70 | step: 18.68

  7%|▋         | 3075/41250 [7:25:03<91:52:34,  8.66s/it]
                                                         
{'loss': 0.0554, 'grad_norm': 0.6023015975952148, 'learning_rate': 3.9792324865008986e-05, 'epoch': 0.75}

  7%|▋         | 3075/41250 [7:25:03<91:52:34,  8.66s/it][2025-04-25 15:22:47,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 15:22:47,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5714.56 | bwd_inner_microstep: 5699.45 | bwd_allreduce_microstep: 15.07 | step_microstep: 18.93
[2025-04-25 15:22:47,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5714.58 | bwd_inner: 5699.45 | bwd_allreduce: 15.08 | step: 18.94

  7%|▋         | 3076/41250 [7:25:12<91:49:10,  8.66s/it]
                                                         
{'loss': 0.1803, 'grad_norm': 1.1445218324661255, 'learning_rate': 3.9792099093627504e-05, 'epoch': 0.75}

  7%|▋         | 3076/41250 [7:25:12<91:49:10,  8.66s/it][2025-04-25 15:22:55,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 15:22:55,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.36 | bwd_microstep: 5714.30 | bwd_inner_microstep: 5678.06 | bwd_allreduce_microstep: 36.19 | step_microstep: 18.90
[2025-04-25 15:22:55,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.36 | bwd: 5714.31 | bwd_inner: 5678.06 | bwd_allreduce: 36.21 | step: 18.91

  7%|▋         | 3077/41250 [7:25:21<91:45:33,  8.65s/it]
                                                         
{'loss': 0.091, 'grad_norm': 1.3829362392425537, 'learning_rate': 3.979187320023162e-05, 'epoch': 0.75}

  7%|▋         | 3077/41250 [7:25:21<91:45:33,  8.65s/it][2025-04-25 15:23:04,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 15:23:04,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.15 | bwd_microstep: 5683.42 | bwd_inner_microstep: 5648.94 | bwd_allreduce_microstep: 34.43 | step_microstep: 19.10
[2025-04-25 15:23:04,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.15 | bwd: 5683.43 | bwd_inner: 5648.94 | bwd_allreduce: 34.45 | step: 19.10

  7%|▋         | 3078/41250 [7:25:29<91:33:25,  8.63s/it]
                                                         
{'loss': 0.2336, 'grad_norm': 2.4433889389038086, 'learning_rate': 3.9791647184822734e-05, 'epoch': 0.75}

  7%|▋         | 3078/41250 [7:25:29<91:33:25,  8.63s/it][2025-04-25 15:23:13,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:23:13,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5710.08 | bwd_inner_microstep: 5697.64 | bwd_allreduce_microstep: 12.39 | step_microstep: 18.88
[2025-04-25 15:23:13,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5710.10 | bwd_inner: 5697.64 | bwd_allreduce: 12.41 | step: 18.88

  7%|▋         | 3079/41250 [7:25:38<91:34:19,  8.64s/it]
                                                         
{'loss': 0.0918, 'grad_norm': 0.9332054257392883, 'learning_rate': 3.979142104740223e-05, 'epoch': 0.75}

  7%|▋         | 3079/41250 [7:25:38<91:34:19,  8.64s/it][2025-04-25 15:23:21,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:23:21,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5693.33 | bwd_inner_microstep: 5680.52 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.82
[2025-04-25 15:23:21,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5693.34 | bwd_inner: 5680.52 | bwd_allreduce: 12.78 | step: 18.82

  7%|▋         | 3080/41250 [7:25:46<91:30:45,  8.63s/it]
                                                         
{'loss': 0.1866, 'grad_norm': 2.2860445976257324, 'learning_rate': 3.9791194787971525e-05, 'epoch': 0.75}

  7%|▋         | 3080/41250 [7:25:46<91:30:45,  8.63s/it][2025-04-25 15:23:30,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:23:30,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5701.14 | bwd_inner_microstep: 5688.46 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.46
[2025-04-25 15:23:30,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5701.16 | bwd_inner: 5688.46 | bwd_allreduce: 12.66 | step: 18.46

  7%|▋         | 3081/41250 [7:25:55<91:31:09,  8.63s/it]
                                                         
{'loss': 0.1774, 'grad_norm': 1.765561819076538, 'learning_rate': 3.9790968406531984e-05, 'epoch': 0.75}

  7%|▋         | 3081/41250 [7:25:55<91:31:09,  8.63s/it][2025-04-25 15:23:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:23:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.24 | bwd_microstep: 5723.72 | bwd_inner_microstep: 5711.06 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.64
[2025-04-25 15:23:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.24 | bwd: 5723.73 | bwd_inner: 5711.06 | bwd_allreduce: 12.64 | step: 18.65

  7%|▋         | 3082/41250 [7:26:04<91:35:26,  8.64s/it]
                                                         
{'loss': 0.0692, 'grad_norm': 0.7687124013900757, 'learning_rate': 3.979074190308502e-05, 'epoch': 0.75}

  7%|▋         | 3082/41250 [7:26:04<91:35:26,  8.64s/it][2025-04-25 15:23:47,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.88
[2025-04-25 15:23:47,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5699.06 | bwd_inner_microstep: 5655.01 | bwd_allreduce_microstep: 44.01 | step_microstep: 18.48
[2025-04-25 15:23:47,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.92 | bwd: 5699.08 | bwd_inner: 5655.01 | bwd_allreduce: 44.02 | step: 18.49

  7%|▋         | 3083/41250 [7:26:12<91:28:24,  8.63s/it]
                                                         
{'loss': 0.2299, 'grad_norm': 1.812608003616333, 'learning_rate': 3.9790515277632015e-05, 'epoch': 0.75}

  7%|▋         | 3083/41250 [7:26:12<91:28:24,  8.63s/it][2025-04-25 15:23:56,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 15:23:56,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5730.53 | bwd_inner_microstep: 5697.51 | bwd_allreduce_microstep: 32.98 | step_microstep: 18.58
[2025-04-25 15:23:56,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5730.54 | bwd_inner: 5697.51 | bwd_allreduce: 32.99 | step: 18.58

  7%|▋         | 3084/41250 [7:26:21<91:35:55,  8.64s/it]
                                                         
{'loss': 0.0594, 'grad_norm': 0.7353885173797607, 'learning_rate': 3.979028853017438e-05, 'epoch': 0.75}

  7%|▋         | 3084/41250 [7:26:21<91:35:55,  8.64s/it][2025-04-25 15:24:04,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:24:04,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.91 | bwd_microstep: 5756.59 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 111.20 | step_microstep: 18.29
[2025-04-25 15:24:04,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.91 | bwd: 5756.60 | bwd_inner: 5645.34 | bwd_allreduce: 111.22 | step: 18.30

  7%|▋         | 3085/41250 [7:26:30<91:43:58,  8.65s/it]
                                                         
{'loss': 0.132, 'grad_norm': 1.5509345531463623, 'learning_rate': 3.979006166071351e-05, 'epoch': 0.75}

  7%|▋         | 3085/41250 [7:26:30<91:43:58,  8.65s/it][2025-04-25 15:24:13,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 15:24:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.54 | bwd_microstep: 5676.30 | bwd_inner_microstep: 5663.62 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.82
[2025-04-25 15:24:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.54 | bwd: 5676.31 | bwd_inner: 5663.62 | bwd_allreduce: 12.65 | step: 18.83

  7%|▋         | 3086/41250 [7:26:38<91:35:39,  8.64s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 1.5162556171417236, 'learning_rate': 3.9789834669250804e-05, 'epoch': 0.75}

  7%|▋         | 3086/41250 [7:26:38<91:35:39,  8.64s/it][2025-04-25 15:24:22,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:24:22,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.25 | bwd_microstep: 5717.71 | bwd_inner_microstep: 5704.91 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.36
[2025-04-25 15:24:22,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.25 | bwd: 5717.72 | bwd_inner: 5704.91 | bwd_allreduce: 12.77 | step: 18.36

  7%|▋         | 3087/41250 [7:26:47<91:38:36,  8.64s/it]
                                                         
{'loss': 0.0547, 'grad_norm': 1.0783045291900635, 'learning_rate': 3.9789607555787657e-05, 'epoch': 0.75}

  7%|▋         | 3087/41250 [7:26:47<91:38:36,  8.64s/it][2025-04-25 15:24:30,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:24:30,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.29 | bwd_microstep: 5733.58 | bwd_inner_microstep: 5720.50 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.37
[2025-04-25 15:24:30,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.29 | bwd: 5733.59 | bwd_inner: 5720.50 | bwd_allreduce: 13.05 | step: 18.37

  7%|▋         | 3088/41250 [7:26:56<91:44:00,  8.65s/it]
                                                         
{'loss': 0.3458, 'grad_norm': 1.6794404983520508, 'learning_rate': 3.978938032032548e-05, 'epoch': 0.75}

  7%|▋         | 3088/41250 [7:26:56<91:44:00,  8.65s/it][2025-04-25 15:24:39,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:24:39,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5700.99 | bwd_inner_microstep: 5647.80 | bwd_allreduce_microstep: 53.14 | step_microstep: 18.33
[2025-04-25 15:24:39,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.82 | bwd: 5701.00 | bwd_inner: 5647.80 | bwd_allreduce: 53.16 | step: 18.33

  7%|▋         | 3089/41250 [7:27:04<91:36:18,  8.64s/it]
                                                         
{'loss': 0.125, 'grad_norm': 0.8178572058677673, 'learning_rate': 3.978915296286566e-05, 'epoch': 0.75}

  7%|▋         | 3089/41250 [7:27:04<91:36:18,  8.64s/it][2025-04-25 15:24:48,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 15:24:48,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.56 | bwd_microstep: 5737.65 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 47.40 | step_microstep: 18.70
[2025-04-25 15:24:48,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.56 | bwd: 5737.66 | bwd_inner: 5690.21 | bwd_allreduce: 47.42 | step: 18.70

  7%|▋         | 3090/41250 [7:27:13<91:45:09,  8.66s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.1427961587905884, 'learning_rate': 3.97889254834096e-05, 'epoch': 0.75}

  7%|▋         | 3090/41250 [7:27:13<91:45:09,  8.66s/it][2025-04-25 15:24:56,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:24:56,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.08 | bwd_microstep: 5733.85 | bwd_inner_microstep: 5711.45 | bwd_allreduce_microstep: 22.36 | step_microstep: 18.74
[2025-04-25 15:24:56,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.08 | bwd: 5733.87 | bwd_inner: 5711.45 | bwd_allreduce: 22.37 | step: 18.75

  7%|▋         | 3091/41250 [7:27:22<91:49:53,  8.66s/it]
                                                         
{'loss': 0.2131, 'grad_norm': 1.8509958982467651, 'learning_rate': 3.978869788195871e-05, 'epoch': 0.75}

  7%|▋         | 3091/41250 [7:27:22<91:49:53,  8.66s/it][2025-04-25 15:25:05,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:25:05,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.53 | bwd_microstep: 5743.36 | bwd_inner_microstep: 5706.86 | bwd_allreduce_microstep: 36.45 | step_microstep: 19.07
[2025-04-25 15:25:05,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.53 | bwd: 5743.37 | bwd_inner: 5706.86 | bwd_allreduce: 36.47 | step: 19.07

  7%|▋         | 3092/41250 [7:27:30<91:52:37,  8.67s/it]
                                                         
{'loss': 0.2446, 'grad_norm': 1.3427503108978271, 'learning_rate': 3.9788470158514394e-05, 'epoch': 0.75}

  7%|▋         | 3092/41250 [7:27:30<91:52:37,  8.67s/it][2025-04-25 15:25:14,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 15:25:14,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.07 | bwd_microstep: 5753.69 | bwd_inner_microstep: 5713.12 | bwd_allreduce_microstep: 40.52 | step_microstep: 19.20
[2025-04-25 15:25:14,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.07 | bwd: 5753.70 | bwd_inner: 5713.12 | bwd_allreduce: 40.54 | step: 19.21

  7%|▋         | 3093/41250 [7:27:39<91:56:49,  8.67s/it]
                                                         
{'loss': 0.4232, 'grad_norm': 2.430056095123291, 'learning_rate': 3.978824231307805e-05, 'epoch': 0.75}

  7%|▋         | 3093/41250 [7:27:39<91:56:49,  8.67s/it][2025-04-25 15:25:22,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.20 | optimizer_step: 0.93
[2025-04-25 15:25:22,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.38 | bwd_microstep: 5792.95 | bwd_inner_microstep: 5658.40 | bwd_allreduce_microstep: 134.50 | step_microstep: 19.40
[2025-04-25 15:25:22,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.38 | bwd: 5792.97 | bwd_inner: 5658.40 | bwd_allreduce: 134.52 | step: 19.40

  8%|▊         | 3094/41250 [7:27:48<92:01:28,  8.68s/it]
                                                         
{'loss': 0.3358, 'grad_norm': 2.2440078258514404, 'learning_rate': 3.9788014345651084e-05, 'epoch': 0.75}

  8%|▊         | 3094/41250 [7:27:48<92:01:28,  8.68s/it][2025-04-25 15:25:31,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 15:25:31,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.05 | bwd_microstep: 5774.77 | bwd_inner_microstep: 5651.91 | bwd_allreduce_microstep: 122.81 | step_microstep: 18.61
[2025-04-25 15:25:31,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.05 | bwd: 5774.78 | bwd_inner: 5651.91 | bwd_allreduce: 122.82 | step: 18.61

  8%|▊         | 3095/41250 [7:27:56<92:02:06,  8.68s/it]
                                                         
{'loss': 0.3493, 'grad_norm': 3.452038049697876, 'learning_rate': 3.9787786256234894e-05, 'epoch': 0.75}

  8%|▊         | 3095/41250 [7:27:56<92:02:06,  8.68s/it][2025-04-25 15:25:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 15:25:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5771.73 | bwd_inner_microstep: 5668.52 | bwd_allreduce_microstep: 103.16 | step_microstep: 19.34
[2025-04-25 15:25:40,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5771.75 | bwd_inner: 5668.52 | bwd_allreduce: 103.19 | step: 19.34

  8%|▊         | 3096/41250 [7:28:05<92:02:54,  8.69s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 1.2586193084716797, 'learning_rate': 3.97875580448309e-05, 'epoch': 0.75}

  8%|▊         | 3096/41250 [7:28:05<92:02:54,  8.69s/it][2025-04-25 15:25:48,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 15:25:48,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.25 | bwd_microstep: 5759.13 | bwd_inner_microstep: 5658.58 | bwd_allreduce_microstep: 100.50 | step_microstep: 18.59
[2025-04-25 15:25:48,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.25 | bwd: 5759.15 | bwd_inner: 5658.58 | bwd_allreduce: 100.52 | step: 18.59

  8%|▊         | 3097/41250 [7:28:14<92:03:11,  8.69s/it]
                                                         
{'loss': 0.3396, 'grad_norm': 1.772788166999817, 'learning_rate': 3.9787329711440506e-05, 'epoch': 0.75}

  8%|▊         | 3097/41250 [7:28:14<92:03:11,  8.69s/it][2025-04-25 15:25:57,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 15:25:57,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.46 | bwd_microstep: 5735.57 | bwd_inner_microstep: 5722.90 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.72
[2025-04-25 15:25:57,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.46 | bwd: 5735.58 | bwd_inner: 5722.90 | bwd_allreduce: 12.64 | step: 18.72

  8%|▊         | 3098/41250 [7:28:22<92:01:36,  8.68s/it]
                                                         
{'loss': 0.0291, 'grad_norm': 0.40491676330566406, 'learning_rate': 3.978710125606511e-05, 'epoch': 0.75}

  8%|▊         | 3098/41250 [7:28:22<92:01:36,  8.68s/it][2025-04-25 15:26:06,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:26:06,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.79 | bwd_microstep: 5680.18 | bwd_inner_microstep: 5667.58 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.62
[2025-04-25 15:26:06,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.79 | bwd: 5680.20 | bwd_inner: 5667.58 | bwd_allreduce: 12.57 | step: 18.62

  8%|▊         | 3099/41250 [7:28:31<91:48:56,  8.66s/it]
                                                         
{'loss': 0.1725, 'grad_norm': 2.227027654647827, 'learning_rate': 3.978687267870613e-05, 'epoch': 0.75}

  8%|▊         | 3099/41250 [7:28:31<91:48:56,  8.66s/it][2025-04-25 15:26:14,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 15:26:14,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.35 | bwd_microstep: 5712.57 | bwd_inner_microstep: 5699.31 | bwd_allreduce_microstep: 13.21 | step_microstep: 18.84
[2025-04-25 15:26:14,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.35 | bwd: 5712.59 | bwd_inner: 5699.31 | bwd_allreduce: 13.23 | step: 18.84

  8%|▊         | 3100/41250 [7:28:40<91:47:54,  8.66s/it]
                                                         
{'loss': 0.2296, 'grad_norm': 2.15553879737854, 'learning_rate': 3.978664397936497e-05, 'epoch': 0.75}

  8%|▊         | 3100/41250 [7:28:40<91:47:54,  8.66s/it][2025-04-25 15:26:23,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.29 | optimizer_step: 0.90
[2025-04-25 15:26:23,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.80 | bwd_microstep: 5696.20 | bwd_inner_microstep: 5652.99 | bwd_allreduce_microstep: 43.15 | step_microstep: 19.94
[2025-04-25 15:26:23,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.80 | bwd: 5696.22 | bwd_inner: 5652.99 | bwd_allreduce: 43.18 | step: 19.94

  8%|▊         | 3101/41250 [7:28:48<91:37:48,  8.65s/it]
                                                         
{'loss': 0.1154, 'grad_norm': 1.1634225845336914, 'learning_rate': 3.978641515804304e-05, 'epoch': 0.75}

  8%|▊         | 3101/41250 [7:28:48<91:37:48,  8.65s/it][2025-04-25 15:26:32,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 15:26:32,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5877.56 | bwd_inner_microstep: 5654.96 | bwd_allreduce_microstep: 222.56 | step_microstep: 19.25
[2025-04-25 15:26:32,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5877.58 | bwd_inner: 5654.96 | bwd_allreduce: 222.58 | step: 19.26

  8%|▊         | 3102/41250 [7:28:57<92:04:30,  8.69s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 1.5297013521194458, 'learning_rate': 3.978618621474176e-05, 'epoch': 0.75}

  8%|▊         | 3102/41250 [7:28:57<92:04:30,  8.69s/it][2025-04-25 15:26:41,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:26:41,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5875.87 | bwd_inner_microstep: 5663.32 | bwd_allreduce_microstep: 212.50 | step_microstep: 19.17
[2025-04-25 15:26:41,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5875.88 | bwd_inner: 5663.32 | bwd_allreduce: 212.52 | step: 19.17

  8%|▊         | 3103/41250 [7:29:06<92:23:32,  8.72s/it]
                                                         
{'loss': 0.0825, 'grad_norm': 0.9004992246627808, 'learning_rate': 3.978595714946253e-05, 'epoch': 0.75}

  8%|▊         | 3103/41250 [7:29:06<92:23:32,  8.72s/it][2025-04-25 15:26:49,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 15:26:49,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.55 | bwd_microstep: 5773.07 | bwd_inner_microstep: 5665.71 | bwd_allreduce_microstep: 107.32 | step_microstep: 19.19
[2025-04-25 15:26:49,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.55 | bwd: 5773.09 | bwd_inner: 5665.71 | bwd_allreduce: 107.33 | step: 19.19

  8%|▊         | 3104/41250 [7:29:15<92:17:17,  8.71s/it]
                                                         
{'loss': 0.1894, 'grad_norm': 1.9001038074493408, 'learning_rate': 3.978572796220676e-05, 'epoch': 0.75}

  8%|▊         | 3104/41250 [7:29:15<92:17:17,  8.71s/it][2025-04-25 15:26:58,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 15:26:58,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.78 | bwd_microstep: 5765.19 | bwd_inner_microstep: 5711.00 | bwd_allreduce_microstep: 54.15 | step_microstep: 19.01
[2025-04-25 15:26:58,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.78 | bwd: 5765.21 | bwd_inner: 5711.00 | bwd_allreduce: 54.17 | step: 19.01

  8%|▊         | 3105/41250 [7:29:23<92:14:56,  8.71s/it]
                                                         
{'loss': 0.1507, 'grad_norm': 1.1926575899124146, 'learning_rate': 3.978549865297588e-05, 'epoch': 0.75}

  8%|▊         | 3105/41250 [7:29:23<92:14:56,  8.71s/it][2025-04-25 15:27:07,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 15:27:07,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.37 | bwd_microstep: 5735.47 | bwd_inner_microstep: 5703.91 | bwd_allreduce_microstep: 31.52 | step_microstep: 18.60
[2025-04-25 15:27:07,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5735.48 | bwd_inner: 5703.91 | bwd_allreduce: 31.53 | step: 18.61

  8%|▊         | 3106/41250 [7:29:32<92:08:04,  8.70s/it]
                                                         
{'loss': 0.2412, 'grad_norm': 2.671532392501831, 'learning_rate': 3.9785269221771286e-05, 'epoch': 0.75}

  8%|▊         | 3106/41250 [7:29:32<92:08:04,  8.70s/it][2025-04-25 15:27:15,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:27:15,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.00 | bwd_microstep: 5787.45 | bwd_inner_microstep: 5649.83 | bwd_allreduce_microstep: 137.57 | step_microstep: 18.54
[2025-04-25 15:27:15,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.00 | bwd: 5787.46 | bwd_inner: 5649.83 | bwd_allreduce: 137.59 | step: 18.54

  8%|▊         | 3107/41250 [7:29:41<92:08:02,  8.70s/it]
                                                         
{'loss': 0.1381, 'grad_norm': 1.864870548248291, 'learning_rate': 3.97850396685944e-05, 'epoch': 0.75}

  8%|▊         | 3107/41250 [7:29:41<92:08:02,  8.70s/it][2025-04-25 15:27:24,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 15:27:24,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.55 | bwd_microstep: 5698.83 | bwd_inner_microstep: 5686.18 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.74
[2025-04-25 15:27:24,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.55 | bwd: 5698.84 | bwd_inner: 5686.18 | bwd_allreduce: 12.62 | step: 18.75

  8%|▊         | 3108/41250 [7:29:49<91:54:38,  8.67s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.8163638710975647, 'learning_rate': 3.9784809993446634e-05, 'epoch': 0.75}

  8%|▊         | 3108/41250 [7:29:49<91:54:38,  8.67s/it][2025-04-25 15:27:33,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:27:33,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.27 | bwd_microstep: 5715.04 | bwd_inner_microstep: 5702.10 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.13
[2025-04-25 15:27:33,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.27 | bwd: 5715.05 | bwd_inner: 5702.10 | bwd_allreduce: 12.91 | step: 19.13

  8%|▊         | 3109/41250 [7:29:58<91:49:18,  8.67s/it]
                                                         
{'loss': 0.1382, 'grad_norm': 1.2797154188156128, 'learning_rate': 3.97845801963294e-05, 'epoch': 0.75}

  8%|▊         | 3109/41250 [7:29:58<91:49:18,  8.67s/it][2025-04-25 15:27:41,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:27:41,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5739.16 | bwd_inner_microstep: 5706.76 | bwd_allreduce_microstep: 32.35 | step_microstep: 18.74
[2025-04-25 15:27:41,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5739.18 | bwd_inner: 5706.76 | bwd_allreduce: 32.37 | step: 18.74

  8%|▊         | 3110/41250 [7:30:07<91:49:39,  8.67s/it]
                                                         
{'loss': 0.0579, 'grad_norm': 0.5591437816619873, 'learning_rate': 3.978435027724413e-05, 'epoch': 0.75}

  8%|▊         | 3110/41250 [7:30:07<91:49:39,  8.67s/it][2025-04-25 15:27:50,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:27:50,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5881.49 | bwd_inner_microstep: 5868.61 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.62
[2025-04-25 15:27:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5881.51 | bwd_inner: 5868.61 | bwd_allreduce: 12.86 | step: 18.62

  8%|▊         | 3111/41250 [7:30:15<92:12:59,  8.70s/it]
                                                         
{'loss': 0.1342, 'grad_norm': 0.8264490962028503, 'learning_rate': 3.978412023619223e-05, 'epoch': 0.75}

  8%|▊         | 3111/41250 [7:30:15<92:12:59,  8.70s/it][2025-04-25 15:27:59,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-25 15:27:59,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.51 | bwd_microstep: 5872.27 | bwd_inner_microstep: 5644.09 | bwd_allreduce_microstep: 228.13 | step_microstep: 19.77
[2025-04-25 15:27:59,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.51 | bwd: 5872.29 | bwd_inner: 5644.09 | bwd_allreduce: 228.16 | step: 19.77

  8%|▊         | 3112/41250 [7:30:24<92:26:21,  8.73s/it]
                                                         
{'loss': 0.1795, 'grad_norm': 1.6784074306488037, 'learning_rate': 3.978389007317512e-05, 'epoch': 0.75}

  8%|▊         | 3112/41250 [7:30:24<92:26:21,  8.73s/it][2025-04-25 15:28:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:28:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.67 | bwd_microstep: 5774.94 | bwd_inner_microstep: 5762.13 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.68
[2025-04-25 15:28:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.67 | bwd: 5774.96 | bwd_inner: 5762.13 | bwd_allreduce: 12.79 | step: 18.68

  8%|▊         | 3113/41250 [7:30:33<92:28:51,  8.73s/it]
                                                         
{'loss': 0.1129, 'grad_norm': 1.171912670135498, 'learning_rate': 3.9783659788194215e-05, 'epoch': 0.75}

  8%|▊         | 3113/41250 [7:30:33<92:28:51,  8.73s/it][2025-04-25 15:28:16,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:28:16,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.62 | bwd_microstep: 5694.30 | bwd_inner_microstep: 5681.72 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.51
[2025-04-25 15:28:16,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.62 | bwd: 5694.31 | bwd_inner: 5681.72 | bwd_allreduce: 12.55 | step: 18.51

  8%|▊         | 3114/41250 [7:30:42<92:09:00,  8.70s/it]
                                                         
{'loss': 0.1726, 'grad_norm': 1.6221072673797607, 'learning_rate': 3.978342938125094e-05, 'epoch': 0.75}

  8%|▊         | 3114/41250 [7:30:42<92:09:00,  8.70s/it][2025-04-25 15:28:25,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-25 15:28:25,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.00 | bwd_microstep: 5707.79 | bwd_inner_microstep: 5693.72 | bwd_allreduce_microstep: 14.01 | step_microstep: 20.31
[2025-04-25 15:28:25,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.00 | bwd: 5707.81 | bwd_inner: 5693.72 | bwd_allreduce: 14.04 | step: 20.31

  8%|▊         | 3115/41250 [7:30:50<91:56:46,  8.68s/it]
                                                         
{'loss': 0.1569, 'grad_norm': 1.303067922592163, 'learning_rate': 3.9783198852346714e-05, 'epoch': 0.76}

  8%|▊         | 3115/41250 [7:30:50<91:56:46,  8.68s/it][2025-04-25 15:28:34,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:28:34,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.53 | bwd_microstep: 5724.49 | bwd_inner_microstep: 5711.66 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.70
[2025-04-25 15:28:34,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.53 | bwd: 5724.51 | bwd_inner: 5711.66 | bwd_allreduce: 12.81 | step: 18.70

  8%|▊         | 3116/41250 [7:30:59<91:52:24,  8.67s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.5087124109268188, 'learning_rate': 3.978296820148296e-05, 'epoch': 0.76}

  8%|▊         | 3116/41250 [7:30:59<91:52:24,  8.67s/it][2025-04-25 15:28:42,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:28:42,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.98 | bwd_microstep: 5769.26 | bwd_inner_microstep: 5756.74 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.66
[2025-04-25 15:28:42,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.98 | bwd: 5769.27 | bwd_inner: 5756.74 | bwd_allreduce: 12.49 | step: 18.67

  8%|▊         | 3117/41250 [7:31:08<92:02:02,  8.69s/it]
                                                         
{'loss': 0.1884, 'grad_norm': 1.2056461572647095, 'learning_rate': 3.9782737428661103e-05, 'epoch': 0.76}

  8%|▊         | 3117/41250 [7:31:08<92:02:02,  8.69s/it][2025-04-25 15:28:51,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 1.10
[2025-04-25 15:28:51,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.81 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5643.71 | bwd_allreduce_microstep: 121.37 | step_microstep: 18.55
[2025-04-25 15:28:51,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.81 | bwd: 5765.14 | bwd_inner: 5643.71 | bwd_allreduce: 121.39 | step: 18.55

  8%|▊         | 3118/41250 [7:31:16<91:58:06,  8.68s/it]
                                                         
{'loss': 0.0737, 'grad_norm': 1.7159582376480103, 'learning_rate': 3.978250653388256e-05, 'epoch': 0.76}

  8%|▊         | 3118/41250 [7:31:16<91:58:06,  8.68s/it][2025-04-25 15:29:00,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:29:00,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.34 | bwd_microstep: 5772.71 | bwd_inner_microstep: 5638.54 | bwd_allreduce_microstep: 134.12 | step_microstep: 17.93
[2025-04-25 15:29:00,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.34 | bwd: 5772.72 | bwd_inner: 5638.54 | bwd_allreduce: 134.14 | step: 17.93

  8%|▊         | 3119/41250 [7:31:25<91:55:28,  8.68s/it]
                                                         
{'loss': 0.2411, 'grad_norm': 1.9746410846710205, 'learning_rate': 3.978227551714875e-05, 'epoch': 0.76}

  8%|▊         | 3119/41250 [7:31:25<91:55:28,  8.68s/it][2025-04-25 15:29:08,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 15:29:08,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.47 | bwd_microstep: 5667.38 | bwd_inner_microstep: 5644.91 | bwd_allreduce_microstep: 22.43 | step_microstep: 18.84
[2025-04-25 15:29:08,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.47 | bwd: 5667.40 | bwd_inner: 5644.91 | bwd_allreduce: 22.45 | step: 18.84

  8%|▊         | 3120/41250 [7:31:33<91:38:04,  8.65s/it]
                                                         
{'loss': 0.0926, 'grad_norm': 1.1867457628250122, 'learning_rate': 3.97820443784611e-05, 'epoch': 0.76}

  8%|▊         | 3120/41250 [7:31:33<91:38:04,  8.65s/it][2025-04-25 15:29:17,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 15:29:17,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.44 | bwd_microstep: 6083.64 | bwd_inner_microstep: 5701.34 | bwd_allreduce_microstep: 382.25 | step_microstep: 18.46
[2025-04-25 15:29:17,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.44 | bwd: 6083.65 | bwd_inner: 5701.34 | bwd_allreduce: 382.27 | step: 18.46

  8%|▊         | 3121/41250 [7:31:43<92:49:14,  8.76s/it]
                                                         
{'loss': 0.3775, 'grad_norm': 1.2883027791976929, 'learning_rate': 3.9781813117821044e-05, 'epoch': 0.76}

  8%|▊         | 3121/41250 [7:31:43<92:49:14,  8.76s/it][2025-04-25 15:29:26,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 15:29:26,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5712.11 | bwd_inner_microstep: 5699.33 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.92
[2025-04-25 15:29:26,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5712.13 | bwd_inner: 5699.33 | bwd_allreduce: 12.76 | step: 18.92

  8%|▊         | 3122/41250 [7:31:51<92:27:08,  8.73s/it]
                                                         
{'loss': 0.1577, 'grad_norm': 2.4852921962738037, 'learning_rate': 3.978158173522999e-05, 'epoch': 0.76}

  8%|▊         | 3122/41250 [7:31:51<92:27:08,  8.73s/it][2025-04-25 15:29:35,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:29:35,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.51 | bwd_microstep: 5776.59 | bwd_inner_microstep: 5630.83 | bwd_allreduce_microstep: 145.72 | step_microstep: 18.98
[2025-04-25 15:29:35,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.51 | bwd: 5776.61 | bwd_inner: 5630.83 | bwd_allreduce: 145.74 | step: 18.99

  8%|▊         | 3123/41250 [7:32:00<92:17:53,  8.71s/it]
                                                         
{'loss': 0.1285, 'grad_norm': 1.864575743675232, 'learning_rate': 3.978135023068939e-05, 'epoch': 0.76}

  8%|▊         | 3123/41250 [7:32:00<92:17:53,  8.71s/it][2025-04-25 15:29:43,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:29:43,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.71 | bwd_microstep: 5745.68 | bwd_inner_microstep: 5688.29 | bwd_allreduce_microstep: 57.35 | step_microstep: 18.59
[2025-04-25 15:29:43,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.71 | bwd: 5745.69 | bwd_inner: 5688.29 | bwd_allreduce: 57.36 | step: 18.59

  8%|▊         | 3124/41250 [7:32:09<92:09:01,  8.70s/it]
                                                         
{'loss': 0.4303, 'grad_norm': 2.5618770122528076, 'learning_rate': 3.978111860420064e-05, 'epoch': 0.76}

  8%|▊         | 3124/41250 [7:32:09<92:09:01,  8.70s/it][2025-04-25 15:29:52,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:29:52,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.05 | bwd_microstep: 5691.80 | bwd_inner_microstep: 5678.99 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.73
[2025-04-25 15:29:52,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.05 | bwd: 5691.81 | bwd_inner: 5678.99 | bwd_allreduce: 12.78 | step: 18.73

  8%|▊         | 3125/41250 [7:32:17<91:52:46,  8.68s/it]
                                                         
{'loss': 0.0914, 'grad_norm': 1.1802898645401, 'learning_rate': 3.97808868557652e-05, 'epoch': 0.76}

  8%|▊         | 3125/41250 [7:32:17<91:52:46,  8.68s/it][2025-04-25 15:30:00,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 15:30:00,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.86 | bwd_microstep: 5723.75 | bwd_inner_microstep: 5691.62 | bwd_allreduce_microstep: 32.08 | step_microstep: 18.96
[2025-04-25 15:30:00,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.86 | bwd: 5723.76 | bwd_inner: 5691.62 | bwd_allreduce: 32.10 | step: 18.97

  8%|▊         | 3126/41250 [7:32:26<91:48:20,  8.67s/it]
                                                         
{'loss': 0.1651, 'grad_norm': 1.0958765745162964, 'learning_rate': 3.978065498538448e-05, 'epoch': 0.76}

  8%|▊         | 3126/41250 [7:32:26<91:48:20,  8.67s/it][2025-04-25 15:30:09,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:30:09,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.41 | bwd_microstep: 5880.22 | bwd_inner_microstep: 5867.34 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.57
[2025-04-25 15:30:09,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.41 | bwd: 5880.24 | bwd_inner: 5867.34 | bwd_allreduce: 12.86 | step: 18.57

  8%|▊         | 3127/41250 [7:32:35<92:31:33,  8.74s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.071864366531372, 'learning_rate': 3.978042299305991e-05, 'epoch': 0.76}

  8%|▊         | 3127/41250 [7:32:35<92:31:33,  8.74s/it][2025-04-25 15:30:18,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:30:18,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.77 | bwd_microstep: 5759.06 | bwd_inner_microstep: 5655.18 | bwd_allreduce_microstep: 103.84 | step_microstep: 18.09
[2025-04-25 15:30:18,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.77 | bwd: 5759.08 | bwd_inner: 5655.18 | bwd_allreduce: 103.86 | step: 18.09

  8%|▊         | 3128/41250 [7:32:43<92:17:29,  8.72s/it]
                                                         
{'loss': 0.1323, 'grad_norm': 0.9172049164772034, 'learning_rate': 3.9780190878792924e-05, 'epoch': 0.76}

  8%|▊         | 3128/41250 [7:32:43<92:17:29,  8.72s/it][2025-04-25 15:30:27,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:30:27,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.56 | bwd_microstep: 5737.19 | bwd_inner_microstep: 5701.75 | bwd_allreduce_microstep: 35.39 | step_microstep: 18.62
[2025-04-25 15:30:27,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5737.21 | bwd_inner: 5701.75 | bwd_allreduce: 35.41 | step: 18.62

  8%|▊         | 3129/41250 [7:32:52<92:07:53,  8.70s/it]
                                                         
{'loss': 0.0875, 'grad_norm': 1.0273330211639404, 'learning_rate': 3.977995864258495e-05, 'epoch': 0.76}

  8%|▊         | 3129/41250 [7:32:52<92:07:53,  8.70s/it][2025-04-25 15:30:35,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.94 | optimizer_step: 1.04
[2025-04-25 15:30:35,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.96 | bwd_microstep: 5694.36 | bwd_inner_microstep: 5649.98 | bwd_allreduce_microstep: 44.33 | step_microstep: 18.51
[2025-04-25 15:30:35,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.96 | bwd: 5694.37 | bwd_inner: 5649.98 | bwd_allreduce: 44.35 | step: 18.51

  8%|▊         | 3130/41250 [7:33:01<91:48:12,  8.67s/it]
                                                         
{'loss': 0.3426, 'grad_norm': 1.4151018857955933, 'learning_rate': 3.9779726284437424e-05, 'epoch': 0.76}

  8%|▊         | 3130/41250 [7:33:01<91:48:12,  8.67s/it][2025-04-25 15:30:44,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:30:44,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.85 | bwd_microstep: 5867.44 | bwd_inner_microstep: 5854.96 | bwd_allreduce_microstep: 12.44 | step_microstep: 18.59
[2025-04-25 15:30:44,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.85 | bwd: 5867.45 | bwd_inner: 5854.96 | bwd_allreduce: 12.45 | step: 18.59

  8%|▊         | 3131/41250 [7:33:09<92:28:05,  8.73s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 3.047210454940796, 'learning_rate': 3.977949380435177e-05, 'epoch': 0.76}

  8%|▊         | 3131/41250 [7:33:09<92:28:05,  8.73s/it][2025-04-25 15:30:53,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:30:53,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.28 | bwd_microstep: 5760.96 | bwd_inner_microstep: 5748.23 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.59
[2025-04-25 15:30:53,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.28 | bwd: 5760.97 | bwd_inner: 5748.23 | bwd_allreduce: 12.71 | step: 18.59

  8%|▊         | 3132/41250 [7:33:18<92:26:09,  8.73s/it]
                                                         
{'loss': 0.1232, 'grad_norm': 1.9013679027557373, 'learning_rate': 3.977926120232943e-05, 'epoch': 0.76}

  8%|▊         | 3132/41250 [7:33:18<92:26:09,  8.73s/it][2025-04-25 15:31:01,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:31:01,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.26 | bwd_microstep: 5661.92 | bwd_inner_microstep: 5644.65 | bwd_allreduce_microstep: 17.23 | step_microstep: 18.44
[2025-04-25 15:31:01,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.26 | bwd: 5661.93 | bwd_inner: 5644.65 | bwd_allreduce: 17.24 | step: 18.44

  8%|▊         | 3133/41250 [7:33:27<91:55:34,  8.68s/it]
                                                         
{'loss': 0.1686, 'grad_norm': 1.790013313293457, 'learning_rate': 3.977902847837184e-05, 'epoch': 0.76}

  8%|▊         | 3133/41250 [7:33:27<91:55:34,  8.68s/it][2025-04-25 15:31:10,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:31:10,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.26 | bwd_microstep: 5735.09 | bwd_inner_microstep: 5641.32 | bwd_allreduce_microstep: 93.73 | step_microstep: 18.67
[2025-04-25 15:31:10,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.26 | bwd: 5735.11 | bwd_inner: 5641.32 | bwd_allreduce: 93.74 | step: 18.67

  8%|▊         | 3134/41250 [7:33:35<91:48:43,  8.67s/it]
                                                         
{'loss': 0.0573, 'grad_norm': 0.5264139175415039, 'learning_rate': 3.977879563248043e-05, 'epoch': 0.76}

  8%|▊         | 3134/41250 [7:33:35<91:48:43,  8.67s/it][2025-04-25 15:31:19,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:31:19,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.73 | bwd_microstep: 5728.66 | bwd_inner_microstep: 5640.34 | bwd_allreduce_microstep: 88.28 | step_microstep: 18.47
[2025-04-25 15:31:19,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.73 | bwd: 5728.67 | bwd_inner: 5640.34 | bwd_allreduce: 88.29 | step: 18.48

  8%|▊         | 3135/41250 [7:33:44<91:42:05,  8.66s/it]
                                                         
{'loss': 0.1264, 'grad_norm': 0.9605157971382141, 'learning_rate': 3.9778562664656625e-05, 'epoch': 0.76}

  8%|▊         | 3135/41250 [7:33:44<91:42:05,  8.66s/it][2025-04-25 15:31:27,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:31:27,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.07 | bwd_microstep: 5757.21 | bwd_inner_microstep: 5653.08 | bwd_allreduce_microstep: 104.09 | step_microstep: 18.26
[2025-04-25 15:31:27,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.07 | bwd: 5757.22 | bwd_inner: 5653.08 | bwd_allreduce: 104.10 | step: 18.26

  8%|▊         | 3136/41250 [7:33:53<91:42:57,  8.66s/it]
                                                         
{'loss': 0.105, 'grad_norm': 1.5462546348571777, 'learning_rate': 3.977832957490188e-05, 'epoch': 0.76}

  8%|▊         | 3136/41250 [7:33:53<91:42:57,  8.66s/it][2025-04-25 15:31:36,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:31:36,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.82 | bwd_microstep: 5698.02 | bwd_inner_microstep: 5652.05 | bwd_allreduce_microstep: 45.93 | step_microstep: 18.29
[2025-04-25 15:31:36,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.82 | bwd: 5698.04 | bwd_inner: 5652.05 | bwd_allreduce: 45.95 | step: 18.30

  8%|▊         | 3137/41250 [7:34:01<91:30:43,  8.64s/it]
                                                         
{'loss': 0.055, 'grad_norm': 1.1358174085617065, 'learning_rate': 3.977809636321762e-05, 'epoch': 0.76}

  8%|▊         | 3137/41250 [7:34:01<91:30:43,  8.64s/it][2025-04-25 15:31:45,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:31:45,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.60 | bwd_microstep: 5775.69 | bwd_inner_microstep: 5644.81 | bwd_allreduce_microstep: 130.84 | step_microstep: 18.78
[2025-04-25 15:31:45,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.59 | bwd: 5775.70 | bwd_inner: 5644.80 | bwd_allreduce: 130.85 | step: 18.78

  8%|▊         | 3138/41250 [7:34:10<91:37:56,  8.66s/it]
                                                         
{'loss': 0.0979, 'grad_norm': 2.11698055267334, 'learning_rate': 3.977786302960529e-05, 'epoch': 0.76}

  8%|▊         | 3138/41250 [7:34:10<91:37:56,  8.66s/it][2025-04-25 15:31:53,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:31:53,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5803.56 | bwd_inner_microstep: 5691.71 | bwd_allreduce_microstep: 111.80 | step_microstep: 18.60
[2025-04-25 15:31:53,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5803.58 | bwd_inner: 5691.71 | bwd_allreduce: 111.82 | step: 18.60

  8%|▊         | 3139/41250 [7:34:19<91:53:49,  8.68s/it]
                                                         
{'loss': 0.0669, 'grad_norm': 0.7376388907432556, 'learning_rate': 3.977762957406632e-05, 'epoch': 0.76}

  8%|▊         | 3139/41250 [7:34:19<91:53:49,  8.68s/it][2025-04-25 15:32:02,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.10 | optimizer_step: 1.14
[2025-04-25 15:32:02,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5666.68 | bwd_inner_microstep: 5647.78 | bwd_allreduce_microstep: 18.85 | step_microstep: 19.29
[2025-04-25 15:32:02,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5666.69 | bwd_inner: 5647.78 | bwd_allreduce: 18.87 | step: 19.30

  8%|▊         | 3140/41250 [7:34:27<91:34:56,  8.65s/it]
                                                         
{'loss': 0.1205, 'grad_norm': 1.368004560470581, 'learning_rate': 3.9777395996602157e-05, 'epoch': 0.76}

  8%|▊         | 3140/41250 [7:34:27<91:34:56,  8.65s/it][2025-04-25 15:32:11,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 15:32:11,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.30 | bwd_microstep: 5726.96 | bwd_inner_microstep: 5650.18 | bwd_allreduce_microstep: 76.73 | step_microstep: 18.32
[2025-04-25 15:32:11,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.30 | bwd: 5726.98 | bwd_inner: 5650.18 | bwd_allreduce: 76.75 | step: 18.32

  8%|▊         | 3141/41250 [7:34:36<91:34:20,  8.65s/it]
                                                         
{'loss': 0.0589, 'grad_norm': 1.3981653451919556, 'learning_rate': 3.977716229721424e-05, 'epoch': 0.76}

  8%|▊         | 3141/41250 [7:34:36<91:34:20,  8.65s/it][2025-04-25 15:32:19,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 15:32:19,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.93 | bwd_microstep: 5697.16 | bwd_inner_microstep: 5684.37 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.87
[2025-04-25 15:32:19,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.93 | bwd: 5697.17 | bwd_inner: 5684.37 | bwd_allreduce: 12.76 | step: 18.87

  8%|▊         | 3142/41250 [7:34:45<91:30:49,  8.65s/it]
                                                         
{'loss': 0.2528, 'grad_norm': 1.8163349628448486, 'learning_rate': 3.9776928475904e-05, 'epoch': 0.76}

  8%|▊         | 3142/41250 [7:34:45<91:30:49,  8.65s/it][2025-04-25 15:32:28,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:32:28,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.05 | bwd_microstep: 5693.05 | bwd_inner_microstep: 5680.31 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.70
[2025-04-25 15:32:28,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.05 | bwd: 5693.06 | bwd_inner: 5680.31 | bwd_allreduce: 12.71 | step: 18.70

  8%|▊         | 3143/41250 [7:34:53<91:27:54,  8.64s/it]
                                                         
{'loss': 0.0931, 'grad_norm': 1.5145158767700195, 'learning_rate': 3.977669453267289e-05, 'epoch': 0.76}

  8%|▊         | 3143/41250 [7:34:53<91:27:54,  8.64s/it][2025-04-25 15:32:37,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:32:37,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.23 | bwd_microstep: 5676.00 | bwd_inner_microstep: 5648.39 | bwd_allreduce_microstep: 27.57 | step_microstep: 18.57
[2025-04-25 15:32:37,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.23 | bwd: 5676.02 | bwd_inner: 5648.39 | bwd_allreduce: 27.59 | step: 18.57

  8%|▊         | 3144/41250 [7:35:02<91:17:43,  8.62s/it]
                                                         
{'loss': 0.5104, 'grad_norm': 3.730126142501831, 'learning_rate': 3.977646046752235e-05, 'epoch': 0.76}

  8%|▊         | 3144/41250 [7:35:02<91:17:43,  8.62s/it][2025-04-25 15:32:45,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:32:45,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.03 | bwd_microstep: 5883.48 | bwd_inner_microstep: 5644.14 | bwd_allreduce_microstep: 239.30 | step_microstep: 18.70
[2025-04-25 15:32:45,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.03 | bwd: 5883.49 | bwd_inner: 5644.14 | bwd_allreduce: 239.32 | step: 18.70

  8%|▊         | 3145/41250 [7:35:11<91:48:52,  8.67s/it]
                                                         
{'loss': 0.3405, 'grad_norm': 2.4973952770233154, 'learning_rate': 3.977622628045381e-05, 'epoch': 0.76}

  8%|▊         | 3145/41250 [7:35:11<91:48:52,  8.67s/it][2025-04-25 15:32:54,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:32:54,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5693.24 | bwd_inner_microstep: 5680.62 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.61
[2025-04-25 15:32:54,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5693.25 | bwd_inner: 5680.62 | bwd_allreduce: 12.59 | step: 18.62

  8%|▊         | 3146/41250 [7:35:19<91:40:03,  8.66s/it]
                                                         
{'loss': 0.3804, 'grad_norm': 1.8891432285308838, 'learning_rate': 3.977599197146874e-05, 'epoch': 0.76}

  8%|▊         | 3146/41250 [7:35:19<91:40:03,  8.66s/it][2025-04-25 15:33:03,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:33:03,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2938.43 | bwd_microstep: 5869.48 | bwd_inner_microstep: 5856.76 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.49
[2025-04-25 15:33:03,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2938.43 | bwd: 5869.49 | bwd_inner: 5856.76 | bwd_allreduce: 12.70 | step: 18.49

  8%|▊         | 3147/41250 [7:35:28<92:23:34,  8.73s/it]
                                                         
{'loss': 0.2329, 'grad_norm': 1.874371886253357, 'learning_rate': 3.977575754056856e-05, 'epoch': 0.76}

  8%|▊         | 3147/41250 [7:35:28<92:23:34,  8.73s/it][2025-04-25 15:33:11,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 15:33:11,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.29 | bwd_microstep: 5711.98 | bwd_inner_microstep: 5699.24 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.99
[2025-04-25 15:33:11,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5711.99 | bwd_inner: 5699.24 | bwd_allreduce: 12.71 | step: 18.99

  8%|▊         | 3148/41250 [7:35:37<92:07:47,  8.70s/it]
                                                         
{'loss': 0.05, 'grad_norm': 0.9574958086013794, 'learning_rate': 3.977552298775473e-05, 'epoch': 0.76}

  8%|▊         | 3148/41250 [7:35:37<92:07:47,  8.70s/it][2025-04-25 15:33:20,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 15:33:20,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.09 | bwd_microstep: 5702.77 | bwd_inner_microstep: 5689.98 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.59
[2025-04-25 15:33:20,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.09 | bwd: 5702.78 | bwd_inner: 5689.98 | bwd_allreduce: 12.76 | step: 18.60

  8%|▊         | 3149/41250 [7:35:45<91:53:10,  8.68s/it]
                                                         
{'loss': 0.1335, 'grad_norm': 1.624308466911316, 'learning_rate': 3.977528831302869e-05, 'epoch': 0.76}

  8%|▊         | 3149/41250 [7:35:45<91:53:10,  8.68s/it][2025-04-25 15:33:29,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:33:29,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 5792.63 | bwd_inner_microstep: 5661.20 | bwd_allreduce_microstep: 131.38 | step_microstep: 18.85
[2025-04-25 15:33:29,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 5792.65 | bwd_inner: 5661.20 | bwd_allreduce: 131.40 | step: 18.84

  8%|▊         | 3150/41250 [7:35:54<91:56:58,  8.69s/it]
                                                         
{'loss': 0.0649, 'grad_norm': 1.0830050706863403, 'learning_rate': 3.9775053516391884e-05, 'epoch': 0.76}

  8%|▊         | 3150/41250 [7:35:54<91:56:58,  8.69s/it][2025-04-25 15:33:37,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 15:33:37,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.75 | bwd_microstep: 5749.08 | bwd_inner_microstep: 5647.42 | bwd_allreduce_microstep: 101.61 | step_microstep: 18.68
[2025-04-25 15:33:37,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.75 | bwd: 5749.10 | bwd_inner: 5647.42 | bwd_allreduce: 101.63 | step: 18.69

  8%|▊         | 3151/41250 [7:36:03<91:53:22,  8.68s/it]
                                                         
{'loss': 0.2891, 'grad_norm': 4.077704906463623, 'learning_rate': 3.977481859784576e-05, 'epoch': 0.76}

  8%|▊         | 3151/41250 [7:36:03<91:53:22,  8.68s/it][2025-04-25 15:33:46,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 15:33:46,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.24 | bwd_microstep: 5774.82 | bwd_inner_microstep: 5660.30 | bwd_allreduce_microstep: 114.47 | step_microstep: 18.97
[2025-04-25 15:33:46,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.24 | bwd: 5774.83 | bwd_inner: 5660.30 | bwd_allreduce: 114.49 | step: 18.97

  8%|▊         | 3152/41250 [7:36:11<91:54:12,  8.68s/it]
                                                         
{'loss': 0.4306, 'grad_norm': 2.2169382572174072, 'learning_rate': 3.977458355739177e-05, 'epoch': 0.76}

  8%|▊         | 3152/41250 [7:36:11<91:54:12,  8.68s/it][2025-04-25 15:33:55,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 15:33:55,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.97 | bwd_microstep: 5757.87 | bwd_inner_microstep: 5707.86 | bwd_allreduce_microstep: 49.97 | step_microstep: 18.59
[2025-04-25 15:33:55,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.97 | bwd: 5757.89 | bwd_inner: 5707.86 | bwd_allreduce: 49.99 | step: 18.59

  8%|▊         | 3153/41250 [7:36:20<91:54:55,  8.69s/it]
                                                         
{'loss': 0.3404, 'grad_norm': 1.5687274932861328, 'learning_rate': 3.9774348395031366e-05, 'epoch': 0.76}

  8%|▊         | 3153/41250 [7:36:20<91:54:55,  8.69s/it][2025-04-25 15:34:04,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.16 | optimizer_step: 0.92
[2025-04-25 15:34:04,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.03 | bwd_microstep: 5773.44 | bwd_inner_microstep: 5656.42 | bwd_allreduce_microstep: 116.97 | step_microstep: 19.33
[2025-04-25 15:34:04,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.03 | bwd: 5773.45 | bwd_inner: 5656.42 | bwd_allreduce: 116.99 | step: 19.34

  8%|▊         | 3154/41250 [7:36:29<91:54:47,  8.69s/it]
                                                         
{'loss': 0.2528, 'grad_norm': 2.311396360397339, 'learning_rate': 3.977411311076599e-05, 'epoch': 0.76}

  8%|▊         | 3154/41250 [7:36:29<91:54:47,  8.69s/it][2025-04-25 15:34:12,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:34:12,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.15 | bwd_microstep: 5748.34 | bwd_inner_microstep: 5695.49 | bwd_allreduce_microstep: 52.81 | step_microstep: 18.49
[2025-04-25 15:34:12,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5748.36 | bwd_inner: 5695.49 | bwd_allreduce: 52.83 | step: 18.49

  8%|▊         | 3155/41250 [7:36:38<91:53:18,  8.68s/it]
                                                         
{'loss': 0.2169, 'grad_norm': 3.037686586380005, 'learning_rate': 3.97738777045971e-05, 'epoch': 0.76}

  8%|▊         | 3155/41250 [7:36:38<91:53:18,  8.68s/it][2025-04-25 15:34:21,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:34:21,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.89 | bwd_microstep: 5761.62 | bwd_inner_microstep: 5655.90 | bwd_allreduce_microstep: 105.68 | step_microstep: 18.70
[2025-04-25 15:34:21,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.89 | bwd: 5761.64 | bwd_inner: 5655.90 | bwd_allreduce: 105.69 | step: 18.70

  8%|▊         | 3156/41250 [7:36:46<91:52:15,  8.68s/it]
                                                         
{'loss': 0.3067, 'grad_norm': 1.9338356256484985, 'learning_rate': 3.9773642176526135e-05, 'epoch': 0.77}

  8%|▊         | 3156/41250 [7:36:46<91:52:15,  8.68s/it][2025-04-25 15:34:30,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 15:34:30,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.56 | bwd_microstep: 5783.22 | bwd_inner_microstep: 5695.23 | bwd_allreduce_microstep: 87.94 | step_microstep: 18.72
[2025-04-25 15:34:30,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.56 | bwd: 5783.23 | bwd_inner: 5695.23 | bwd_allreduce: 87.96 | step: 18.72

  8%|▊         | 3157/41250 [7:36:55<91:57:42,  8.69s/it]
                                                         
{'loss': 0.0376, 'grad_norm': 0.3081294298171997, 'learning_rate': 3.977340652655456e-05, 'epoch': 0.77}

  8%|▊         | 3157/41250 [7:36:55<91:57:42,  8.69s/it][2025-04-25 15:34:38,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:34:38,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.35 | bwd_microstep: 5735.84 | bwd_inner_microstep: 5722.94 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.65
[2025-04-25 15:34:38,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.35 | bwd: 5735.85 | bwd_inner: 5722.94 | bwd_allreduce: 12.87 | step: 18.65

  8%|▊         | 3158/41250 [7:37:04<91:56:27,  8.69s/it]
                                                         
{'loss': 0.1301, 'grad_norm': 0.9817281365394592, 'learning_rate': 3.977317075468383e-05, 'epoch': 0.77}

  8%|▊         | 3158/41250 [7:37:04<91:56:27,  8.69s/it][2025-04-25 15:34:47,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.08 | optimizer_step: 1.00
[2025-04-25 15:34:47,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.01 | bwd_microstep: 6047.18 | bwd_inner_microstep: 5712.45 | bwd_allreduce_microstep: 334.67 | step_microstep: 19.31
[2025-04-25 15:34:47,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.01 | bwd: 6047.19 | bwd_inner: 5712.45 | bwd_allreduce: 334.69 | step: 19.31

  8%|▊         | 3159/41250 [7:37:13<92:52:35,  8.78s/it]
                                                         
{'loss': 0.1263, 'grad_norm': 1.450108528137207, 'learning_rate': 3.977293486091538e-05, 'epoch': 0.77}

  8%|▊         | 3159/41250 [7:37:13<92:52:35,  8.78s/it][2025-04-25 15:34:56,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.97
[2025-04-25 15:34:56,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5762.29 | bwd_inner_microstep: 5692.66 | bwd_allreduce_microstep: 69.58 | step_microstep: 19.13
[2025-04-25 15:34:56,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5762.30 | bwd_inner: 5692.66 | bwd_allreduce: 69.60 | step: 19.13

  8%|▊         | 3160/41250 [7:37:21<92:36:36,  8.75s/it]
                                                         
{'loss': 0.143, 'grad_norm': 1.2515017986297607, 'learning_rate': 3.977269884525068e-05, 'epoch': 0.77}

  8%|▊         | 3160/41250 [7:37:21<92:36:36,  8.75s/it][2025-04-25 15:35:05,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.82 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:35:05,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.47 | bwd_microstep: 5732.63 | bwd_inner_microstep: 5719.74 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.95
[2025-04-25 15:35:05,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.47 | bwd: 5732.64 | bwd_inner: 5719.74 | bwd_allreduce: 12.86 | step: 18.95

  8%|▊         | 3161/41250 [7:37:30<92:22:06,  8.73s/it]
                                                         
{'loss': 0.0524, 'grad_norm': 0.45154622197151184, 'learning_rate': 3.9772462707691184e-05, 'epoch': 0.77}

  8%|▊         | 3161/41250 [7:37:30<92:22:06,  8.73s/it][2025-04-25 15:35:13,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:35:13,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.98 | bwd_microstep: 5751.41 | bwd_inner_microstep: 5709.13 | bwd_allreduce_microstep: 42.23 | step_microstep: 19.00
[2025-04-25 15:35:13,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.98 | bwd: 5751.43 | bwd_inner: 5709.13 | bwd_allreduce: 42.25 | step: 19.00

  8%|▊         | 3162/41250 [7:37:39<92:14:41,  8.72s/it]
                                                         
{'loss': 0.4114, 'grad_norm': 2.8472774028778076, 'learning_rate': 3.9772226448238346e-05, 'epoch': 0.77}

  8%|▊         | 3162/41250 [7:37:39<92:14:41,  8.72s/it][2025-04-25 15:35:22,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:35:22,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.98 | bwd_microstep: 5749.17 | bwd_inner_microstep: 5706.52 | bwd_allreduce_microstep: 42.60 | step_microstep: 18.41
[2025-04-25 15:35:22,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.98 | bwd: 5749.18 | bwd_inner: 5706.52 | bwd_allreduce: 42.62 | step: 18.41

  8%|▊         | 3163/41250 [7:37:47<92:07:11,  8.71s/it]
                                                         
{'loss': 0.3571, 'grad_norm': 2.2689976692199707, 'learning_rate': 3.977199006689362e-05, 'epoch': 0.77}

  8%|▊         | 3163/41250 [7:37:47<92:07:11,  8.71s/it][2025-04-25 15:35:31,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 15:35:31,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.60 | bwd_microstep: 5749.76 | bwd_inner_microstep: 5708.04 | bwd_allreduce_microstep: 41.68 | step_microstep: 18.79
[2025-04-25 15:35:31,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.60 | bwd: 5749.78 | bwd_inner: 5708.04 | bwd_allreduce: 41.70 | step: 18.80

  8%|▊         | 3164/41250 [7:37:56<92:04:11,  8.70s/it]
                                                         
{'loss': 0.0904, 'grad_norm': 1.3062080144882202, 'learning_rate': 3.977175356365846e-05, 'epoch': 0.77}

  8%|▊         | 3164/41250 [7:37:56<92:04:11,  8.70s/it][2025-04-25 15:35:39,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 15:35:39,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5754.23 | bwd_inner_microstep: 5705.77 | bwd_allreduce_microstep: 48.41 | step_microstep: 18.66
[2025-04-25 15:35:39,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5754.25 | bwd_inner: 5705.77 | bwd_allreduce: 48.43 | step: 18.66

  8%|▊         | 3165/41250 [7:38:05<92:01:42,  8.70s/it]
                                                         
{'loss': 0.2171, 'grad_norm': 1.500569224357605, 'learning_rate': 3.9771516938534326e-05, 'epoch': 0.77}

  8%|▊         | 3165/41250 [7:38:05<92:01:42,  8.70s/it][2025-04-25 15:35:48,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:35:48,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.79 | bwd_microstep: 5801.38 | bwd_inner_microstep: 5788.59 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.70
[2025-04-25 15:35:48,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.79 | bwd: 5801.39 | bwd_inner: 5788.59 | bwd_allreduce: 12.76 | step: 18.70

  8%|▊         | 3166/41250 [7:38:13<92:16:50,  8.72s/it]
                                                         
{'loss': 0.1365, 'grad_norm': 1.0854443311691284, 'learning_rate': 3.977128019152268e-05, 'epoch': 0.77}

  8%|▊         | 3166/41250 [7:38:13<92:16:50,  8.72s/it][2025-04-25 15:35:57,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:35:57,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5679.81 | bwd_inner_microstep: 5658.65 | bwd_allreduce_microstep: 21.12 | step_microstep: 18.67
[2025-04-25 15:35:57,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5679.82 | bwd_inner: 5658.65 | bwd_allreduce: 21.13 | step: 18.68

  8%|▊         | 3167/41250 [7:38:22<91:52:00,  8.68s/it]
                                                         
{'loss': 0.3155, 'grad_norm': 1.7108629941940308, 'learning_rate': 3.9771043322624984e-05, 'epoch': 0.77}

  8%|▊         | 3167/41250 [7:38:22<91:52:00,  8.68s/it][2025-04-25 15:36:05,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 15:36:05,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.80 | bwd_microstep: 5698.09 | bwd_inner_microstep: 5667.67 | bwd_allreduce_microstep: 30.37 | step_microstep: 18.51
[2025-04-25 15:36:05,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.80 | bwd: 5698.10 | bwd_inner: 5667.67 | bwd_allreduce: 30.39 | step: 18.51

  8%|▊         | 3168/41250 [7:38:31<91:38:29,  8.66s/it]
                                                         
{'loss': 0.2392, 'grad_norm': 3.3426058292388916, 'learning_rate': 3.97708063318427e-05, 'epoch': 0.77}

  8%|▊         | 3168/41250 [7:38:31<91:38:29,  8.66s/it][2025-04-25 15:36:14,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:36:14,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.06 | bwd_microstep: 5777.72 | bwd_inner_microstep: 5660.52 | bwd_allreduce_microstep: 117.15 | step_microstep: 18.92
[2025-04-25 15:36:14,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.06 | bwd: 5777.73 | bwd_inner: 5660.52 | bwd_allreduce: 117.17 | step: 18.92

  8%|▊         | 3169/41250 [7:38:39<91:44:42,  8.67s/it]
                                                         
{'loss': 0.1893, 'grad_norm': 1.553285837173462, 'learning_rate': 3.977056921917727e-05, 'epoch': 0.77}

  8%|▊         | 3169/41250 [7:38:39<91:44:42,  8.67s/it][2025-04-25 15:36:23,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.33 | optimizer_step: 1.04
[2025-04-25 15:36:23,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.68 | bwd_microstep: 5724.04 | bwd_inner_microstep: 5710.03 | bwd_allreduce_microstep: 13.95 | step_microstep: 20.23
[2025-04-25 15:36:23,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.68 | bwd: 5724.06 | bwd_inner: 5710.03 | bwd_allreduce: 13.98 | step: 20.23

  8%|▊         | 3170/41250 [7:38:48<91:42:36,  8.67s/it]
                                                         
{'loss': 0.0806, 'grad_norm': 0.6651335954666138, 'learning_rate': 3.9770331984630176e-05, 'epoch': 0.77}

  8%|▊         | 3170/41250 [7:38:48<91:42:36,  8.67s/it][2025-04-25 15:36:31,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 15:36:31,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.59 | bwd_microstep: 5733.20 | bwd_inner_microstep: 5720.57 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.73
[2025-04-25 15:36:31,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.59 | bwd: 5733.22 | bwd_inner: 5720.57 | bwd_allreduce: 12.61 | step: 18.73

  8%|▊         | 3171/41250 [7:38:57<91:43:57,  8.67s/it]
                                                         
{'loss': 0.0807, 'grad_norm': 1.2795062065124512, 'learning_rate': 3.977009462820288e-05, 'epoch': 0.77}

  8%|▊         | 3171/41250 [7:38:57<91:43:57,  8.67s/it][2025-04-25 15:36:40,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:36:40,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.82 | bwd_microstep: 5754.26 | bwd_inner_microstep: 5693.49 | bwd_allreduce_microstep: 60.72 | step_microstep: 18.75
[2025-04-25 15:36:40,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.82 | bwd: 5754.27 | bwd_inner: 5693.49 | bwd_allreduce: 60.74 | step: 18.75

  8%|▊         | 3172/41250 [7:39:05<91:45:37,  8.68s/it]
                                                         
{'loss': 0.0777, 'grad_norm': 1.1170803308486938, 'learning_rate': 3.976985714989683e-05, 'epoch': 0.77}

  8%|▊         | 3172/41250 [7:39:05<91:45:37,  8.68s/it][2025-04-25 15:36:49,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:36:49,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.88 | bwd_microstep: 5743.29 | bwd_inner_microstep: 5661.61 | bwd_allreduce_microstep: 81.63 | step_microstep: 18.57
[2025-04-25 15:36:49,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.88 | bwd: 5743.30 | bwd_inner: 5661.61 | bwd_allreduce: 81.64 | step: 18.57

  8%|▊         | 3173/41250 [7:39:14<91:42:51,  8.67s/it]
                                                         
{'loss': 0.1643, 'grad_norm': 1.135951042175293, 'learning_rate': 3.9769619549713505e-05, 'epoch': 0.77}

  8%|▊         | 3173/41250 [7:39:14<91:42:51,  8.67s/it][2025-04-25 15:36:57,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.06 | optimizer_step: 1.06
[2025-04-25 15:36:57,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.52 | bwd_microstep: 5734.30 | bwd_inner_microstep: 5708.85 | bwd_allreduce_microstep: 25.40 | step_microstep: 19.04
[2025-04-25 15:36:57,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.52 | bwd: 5734.31 | bwd_inner: 5708.85 | bwd_allreduce: 25.42 | step: 19.04

  8%|▊         | 3174/41250 [7:39:23<91:43:16,  8.67s/it]
                                                         
{'loss': 0.2455, 'grad_norm': 1.3087680339813232, 'learning_rate': 3.976938182765436e-05, 'epoch': 0.77}

  8%|▊         | 3174/41250 [7:39:23<91:43:16,  8.67s/it][2025-04-25 15:37:06,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 15:37:06,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.92 | bwd_microstep: 5746.91 | bwd_inner_microstep: 5649.18 | bwd_allreduce_microstep: 97.68 | step_microstep: 18.74
[2025-04-25 15:37:06,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.92 | bwd: 5746.92 | bwd_inner: 5649.18 | bwd_allreduce: 97.70 | step: 18.74

  8%|▊         | 3175/41250 [7:39:31<91:42:27,  8.67s/it]
                                                         
{'loss': 0.2662, 'grad_norm': 1.5118439197540283, 'learning_rate': 3.9769143983720866e-05, 'epoch': 0.77}

  8%|▊         | 3175/41250 [7:39:31<91:42:27,  8.67s/it][2025-04-25 15:37:15,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 1.00
[2025-04-25 15:37:15,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.19 | bwd_microstep: 5703.47 | bwd_inner_microstep: 5659.86 | bwd_allreduce_microstep: 43.56 | step_microstep: 19.47
[2025-04-25 15:37:15,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.19 | bwd: 5703.49 | bwd_inner: 5659.86 | bwd_allreduce: 43.58 | step: 19.47

  8%|▊         | 3176/41250 [7:39:40<91:32:01,  8.65s/it]
                                                         
{'loss': 0.0832, 'grad_norm': 0.644264817237854, 'learning_rate': 3.976890601791449e-05, 'epoch': 0.77}

  8%|▊         | 3176/41250 [7:39:40<91:32:01,  8.65s/it][2025-04-25 15:37:23,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 15:37:23,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.54 | bwd_microstep: 5736.65 | bwd_inner_microstep: 5703.04 | bwd_allreduce_microstep: 33.56 | step_microstep: 18.82
[2025-04-25 15:37:23,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.54 | bwd: 5736.66 | bwd_inner: 5703.04 | bwd_allreduce: 33.58 | step: 18.83

  8%|▊         | 3177/41250 [7:39:49<91:36:08,  8.66s/it]
                                                         
{'loss': 0.1948, 'grad_norm': 1.2391754388809204, 'learning_rate': 3.9768667930236696e-05, 'epoch': 0.77}

  8%|▊         | 3177/41250 [7:39:49<91:36:08,  8.66s/it][2025-04-25 15:37:32,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 15:37:32,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5711.16 | bwd_inner_microstep: 5698.37 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.98
[2025-04-25 15:37:32,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5711.18 | bwd_inner: 5698.37 | bwd_allreduce: 12.77 | step: 18.98

  8%|▊         | 3178/41250 [7:39:57<91:32:33,  8.66s/it]
                                                         
{'loss': 0.1875, 'grad_norm': 1.5557173490524292, 'learning_rate': 3.976842972068895e-05, 'epoch': 0.77}

  8%|▊         | 3178/41250 [7:39:57<91:32:33,  8.66s/it][2025-04-25 15:37:41,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 15:37:41,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5729.98 | bwd_inner_microstep: 5689.75 | bwd_allreduce_microstep: 40.17 | step_microstep: 18.93
[2025-04-25 15:37:41,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5729.99 | bwd_inner: 5689.75 | bwd_allreduce: 40.19 | step: 18.94

  8%|▊         | 3179/41250 [7:40:06<91:33:17,  8.66s/it]
                                                         
{'loss': 0.2622, 'grad_norm': 1.6178206205368042, 'learning_rate': 3.976819138927272e-05, 'epoch': 0.77}

  8%|▊         | 3179/41250 [7:40:06<91:33:17,  8.66s/it][2025-04-25 15:37:49,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:37:49,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.99 | bwd_microstep: 5735.62 | bwd_inner_microstep: 5700.70 | bwd_allreduce_microstep: 34.88 | step_microstep: 18.85
[2025-04-25 15:37:49,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.99 | bwd: 5735.64 | bwd_inner: 5700.70 | bwd_allreduce: 34.89 | step: 18.85

  8%|▊         | 3180/41250 [7:40:15<91:35:29,  8.66s/it]
                                                         
{'loss': 0.1375, 'grad_norm': 1.3367573022842407, 'learning_rate': 3.9767952935989484e-05, 'epoch': 0.77}

  8%|▊         | 3180/41250 [7:40:15<91:35:29,  8.66s/it][2025-04-25 15:37:58,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 15:37:58,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.36 | bwd_microstep: 5758.35 | bwd_inner_microstep: 5661.24 | bwd_allreduce_microstep: 97.07 | step_microstep: 18.75
[2025-04-25 15:37:58,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.36 | bwd: 5758.36 | bwd_inner: 5661.24 | bwd_allreduce: 97.08 | step: 18.75

  8%|▊         | 3181/41250 [7:40:23<91:36:58,  8.66s/it]
                                                         
{'loss': 0.1405, 'grad_norm': 1.4647594690322876, 'learning_rate': 3.9767714360840705e-05, 'epoch': 0.77}

  8%|▊         | 3181/41250 [7:40:23<91:36:58,  8.66s/it][2025-04-25 15:38:07,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 15:38:07,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.99 | bwd_microstep: 5718.53 | bwd_inner_microstep: 5694.69 | bwd_allreduce_microstep: 23.79 | step_microstep: 19.08
[2025-04-25 15:38:07,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.99 | bwd: 5718.54 | bwd_inner: 5694.69 | bwd_allreduce: 23.81 | step: 19.09

  8%|▊         | 3182/41250 [7:40:32<91:34:29,  8.66s/it]
                                                         
{'loss': 0.1697, 'grad_norm': 1.8783577680587769, 'learning_rate': 3.976747566382785e-05, 'epoch': 0.77}

  8%|▊         | 3182/41250 [7:40:32<91:34:29,  8.66s/it][2025-04-25 15:38:15,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.11 | optimizer_step: 0.95
[2025-04-25 15:38:15,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5712.95 | bwd_inner_microstep: 5698.88 | bwd_allreduce_microstep: 14.01 | step_microstep: 20.09
[2025-04-25 15:38:15,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5712.97 | bwd_inner: 5698.88 | bwd_allreduce: 14.04 | step: 20.09

  8%|▊         | 3183/41250 [7:40:41<91:32:12,  8.66s/it]
                                                         
{'loss': 0.0737, 'grad_norm': 1.2456011772155762, 'learning_rate': 3.9767236844952403e-05, 'epoch': 0.77}

  8%|▊         | 3183/41250 [7:40:41<91:32:12,  8.66s/it][2025-04-25 15:38:24,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:38:24,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5710.89 | bwd_inner_microstep: 5697.95 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.10
[2025-04-25 15:38:24,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5710.91 | bwd_inner: 5697.95 | bwd_allreduce: 12.91 | step: 19.10

  8%|▊         | 3184/41250 [7:40:49<91:30:20,  8.65s/it]
                                                         
{'loss': 0.3106, 'grad_norm': 2.9557366371154785, 'learning_rate': 3.976699790421583e-05, 'epoch': 0.77}

  8%|▊         | 3184/41250 [7:40:49<91:30:20,  8.65s/it][2025-04-25 15:38:33,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 15:38:33,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.30 | bwd_microstep: 5722.94 | bwd_inner_microstep: 5679.02 | bwd_allreduce_microstep: 43.88 | step_microstep: 18.99
[2025-04-25 15:38:33,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.30 | bwd: 5722.95 | bwd_inner: 5679.02 | bwd_allreduce: 43.90 | step: 18.99

  8%|▊         | 3185/41250 [7:40:58<91:31:59,  8.66s/it]
                                                         
{'loss': 0.0796, 'grad_norm': 0.8156681060791016, 'learning_rate': 3.97667588416196e-05, 'epoch': 0.77}

  8%|▊         | 3185/41250 [7:40:58<91:31:59,  8.66s/it][2025-04-25 15:38:41,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:38:41,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.63 | bwd_microstep: 5789.39 | bwd_inner_microstep: 5776.54 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.05
[2025-04-25 15:38:41,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.63 | bwd: 5789.40 | bwd_inner: 5776.54 | bwd_allreduce: 12.82 | step: 19.05

  8%|▊         | 3186/41250 [7:41:07<91:52:53,  8.69s/it]
                                                         
{'loss': 0.1357, 'grad_norm': 0.9283541440963745, 'learning_rate': 3.976651965716519e-05, 'epoch': 0.77}

  8%|▊         | 3186/41250 [7:41:07<91:52:53,  8.69s/it][2025-04-25 15:38:50,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.20 | optimizer_step: 1.07
[2025-04-25 15:38:50,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.95 | bwd_microstep: 5709.89 | bwd_inner_microstep: 5695.04 | bwd_allreduce_microstep: 14.79 | step_microstep: 20.70
[2025-04-25 15:38:50,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.95 | bwd: 5709.91 | bwd_inner: 5695.04 | bwd_allreduce: 14.82 | step: 20.70

  8%|▊         | 3187/41250 [7:41:15<91:44:54,  8.68s/it]
                                                         
{'loss': 0.2865, 'grad_norm': 2.0396652221679688, 'learning_rate': 3.976628035085407e-05, 'epoch': 0.77}

  8%|▊         | 3187/41250 [7:41:15<91:44:54,  8.68s/it][2025-04-25 15:38:59,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:38:59,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5734.79 | bwd_inner_microstep: 5651.99 | bwd_allreduce_microstep: 82.76 | step_microstep: 18.33
[2025-04-25 15:38:59,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5734.81 | bwd_inner: 5651.99 | bwd_allreduce: 82.78 | step: 18.34

  8%|▊         | 3188/41250 [7:41:24<91:38:53,  8.67s/it]
                                                         
{'loss': 0.1827, 'grad_norm': 1.15781569480896, 'learning_rate': 3.976604092268772e-05, 'epoch': 0.77}

  8%|▊         | 3188/41250 [7:41:24<91:38:53,  8.67s/it][2025-04-25 15:39:07,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.06 | optimizer_step: 1.06
[2025-04-25 15:39:07,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.96 | bwd_microstep: 5751.95 | bwd_inner_microstep: 5649.12 | bwd_allreduce_microstep: 102.78 | step_microstep: 19.22
[2025-04-25 15:39:07,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.96 | bwd: 5751.96 | bwd_inner: 5649.12 | bwd_allreduce: 102.80 | step: 19.23

  8%|▊         | 3189/41250 [7:41:33<91:39:55,  8.67s/it]
                                                         
{'loss': 0.244, 'grad_norm': 1.435263991355896, 'learning_rate': 3.976580137266762e-05, 'epoch': 0.77}

  8%|▊         | 3189/41250 [7:41:33<91:39:55,  8.67s/it][2025-04-25 15:39:16,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:39:16,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.29 | bwd_microstep: 5697.23 | bwd_inner_microstep: 5684.19 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.15
[2025-04-25 15:39:16,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.29 | bwd: 5697.25 | bwd_inner: 5684.19 | bwd_allreduce: 13.01 | step: 19.15

  8%|▊         | 3190/41250 [7:41:41<91:31:33,  8.66s/it]
                                                         
{'loss': 0.224, 'grad_norm': 1.3247566223144531, 'learning_rate': 3.976556170079524e-05, 'epoch': 0.77}

  8%|▊         | 3190/41250 [7:41:41<91:31:33,  8.66s/it][2025-04-25 15:39:25,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.15 | optimizer_step: 1.09
[2025-04-25 15:39:25,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.36 | bwd_microstep: 5702.20 | bwd_inner_microstep: 5688.21 | bwd_allreduce_microstep: 13.92 | step_microstep: 20.41
[2025-04-25 15:39:25,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.36 | bwd: 5702.22 | bwd_inner: 5688.21 | bwd_allreduce: 13.95 | step: 20.41

  8%|▊         | 3191/41250 [7:41:50<91:25:46,  8.65s/it]
                                                         
{'loss': 0.2544, 'grad_norm': 1.3558309078216553, 'learning_rate': 3.976532190707206e-05, 'epoch': 0.77}

  8%|▊         | 3191/41250 [7:41:50<91:25:46,  8.65s/it][2025-04-25 15:39:33,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-25 15:39:33,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.95 | bwd_microstep: 5733.47 | bwd_inner_microstep: 5686.43 | bwd_allreduce_microstep: 47.00 | step_microstep: 18.80
[2025-04-25 15:39:33,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.95 | bwd: 5733.48 | bwd_inner: 5686.43 | bwd_allreduce: 47.01 | step: 18.80

  8%|▊         | 3192/41250 [7:41:59<91:27:59,  8.65s/it]
                                                         
{'loss': 0.036, 'grad_norm': 0.4897346794605255, 'learning_rate': 3.976508199149956e-05, 'epoch': 0.77}

  8%|▊         | 3192/41250 [7:41:59<91:27:59,  8.65s/it][2025-04-25 15:39:42,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-25 15:39:42,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5783.32 | bwd_inner_microstep: 5648.12 | bwd_allreduce_microstep: 135.16 | step_microstep: 19.26
[2025-04-25 15:39:42,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5783.34 | bwd_inner: 5648.12 | bwd_allreduce: 135.18 | step: 19.27

  8%|▊         | 3193/41250 [7:42:07<91:35:34,  8.66s/it]
                                                         
{'loss': 0.2694, 'grad_norm': 2.266106367111206, 'learning_rate': 3.976484195407922e-05, 'epoch': 0.77}

  8%|▊         | 3193/41250 [7:42:07<91:35:34,  8.66s/it][2025-04-25 15:39:51,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 15:39:51,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.10 | bwd_microstep: 5817.67 | bwd_inner_microstep: 5685.56 | bwd_allreduce_microstep: 132.07 | step_microstep: 19.08
[2025-04-25 15:39:51,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.10 | bwd: 5817.69 | bwd_inner: 5685.56 | bwd_allreduce: 132.08 | step: 19.08

  8%|▊         | 3194/41250 [7:42:16<91:50:40,  8.69s/it]
                                                         
{'loss': 0.147, 'grad_norm': 1.205476999282837, 'learning_rate': 3.9764601794812504e-05, 'epoch': 0.77}

  8%|▊         | 3194/41250 [7:42:16<91:50:40,  8.69s/it][2025-04-25 15:39:59,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:39:59,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5748.15 | bwd_inner_microstep: 5652.92 | bwd_allreduce_microstep: 95.18 | step_microstep: 18.96
[2025-04-25 15:39:59,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5748.16 | bwd_inner: 5652.93 | bwd_allreduce: 95.19 | step: 18.96

  8%|▊         | 3195/41250 [7:42:25<91:45:01,  8.68s/it]
                                                         
{'loss': 0.1607, 'grad_norm': 1.1796376705169678, 'learning_rate': 3.9764361513700913e-05, 'epoch': 0.77}

  8%|▊         | 3195/41250 [7:42:25<91:45:01,  8.68s/it][2025-04-25 15:40:08,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:40:08,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5704.16 | bwd_inner_microstep: 5691.32 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.96
[2025-04-25 15:40:08,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5704.18 | bwd_inner: 5691.32 | bwd_allreduce: 12.81 | step: 18.96

  8%|▊         | 3196/41250 [7:42:33<91:38:11,  8.67s/it]
                                                         
{'loss': 0.0837, 'grad_norm': 0.9457208514213562, 'learning_rate': 3.9764121110745924e-05, 'epoch': 0.77}

  8%|▊         | 3196/41250 [7:42:33<91:38:11,  8.67s/it][2025-04-25 15:40:17,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:40:17,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.70 | bwd_microstep: 5684.10 | bwd_inner_microstep: 5639.46 | bwd_allreduce_microstep: 44.60 | step_microstep: 18.87
[2025-04-25 15:40:17,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.70 | bwd: 5684.12 | bwd_inner: 5639.46 | bwd_allreduce: 44.62 | step: 18.87

  8%|▊         | 3197/41250 [7:42:42<91:25:10,  8.65s/it]
                                                         
{'loss': 0.0951, 'grad_norm': 0.8233208060264587, 'learning_rate': 3.976388058594901e-05, 'epoch': 0.78}

  8%|▊         | 3197/41250 [7:42:42<91:25:10,  8.65s/it][2025-04-25 15:40:25,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.14 | optimizer_step: 1.02
[2025-04-25 15:40:25,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5696.49 | bwd_inner_microstep: 5649.55 | bwd_allreduce_microstep: 46.88 | step_microstep: 20.12
[2025-04-25 15:40:25,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5696.50 | bwd_inner: 5649.55 | bwd_allreduce: 46.91 | step: 20.12

  8%|▊         | 3198/41250 [7:42:51<91:17:15,  8.64s/it]
                                                         
{'loss': 0.3107, 'grad_norm': 1.7386927604675293, 'learning_rate': 3.9763639939311664e-05, 'epoch': 0.78}

  8%|▊         | 3198/41250 [7:42:51<91:17:15,  8.64s/it][2025-04-25 15:40:34,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 15:40:34,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.39 | bwd_microstep: 5721.04 | bwd_inner_microstep: 5700.46 | bwd_allreduce_microstep: 20.53 | step_microstep: 18.96
[2025-04-25 15:40:34,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.39 | bwd: 5721.05 | bwd_inner: 5700.46 | bwd_allreduce: 20.54 | step: 18.97

  8%|▊         | 3199/41250 [7:42:59<91:21:23,  8.64s/it]
                                                         
{'loss': 0.1986, 'grad_norm': 1.587723970413208, 'learning_rate': 3.976339917083536e-05, 'epoch': 0.78}

  8%|▊         | 3199/41250 [7:42:59<91:21:23,  8.64s/it][2025-04-25 15:40:43,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:40:43,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.08 | bwd_microstep: 5733.34 | bwd_inner_microstep: 5651.22 | bwd_allreduce_microstep: 82.08 | step_microstep: 19.03
[2025-04-25 15:40:43,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.08 | bwd: 5733.36 | bwd_inner: 5651.22 | bwd_allreduce: 82.10 | step: 19.03

  8%|▊         | 3200/41250 [7:43:08<91:20:54,  8.64s/it]
                                                         
{'loss': 0.2686, 'grad_norm': 2.34718656539917, 'learning_rate': 3.9763158280521595e-05, 'epoch': 0.78}

  8%|▊         | 3200/41250 [7:43:08<91:20:54,  8.64s/it][2025-04-25 15:40:51,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:40:51,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.38 | bwd_microstep: 5748.12 | bwd_inner_microstep: 5676.77 | bwd_allreduce_microstep: 71.30 | step_microstep: 18.98
[2025-04-25 15:40:51,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.38 | bwd: 5748.13 | bwd_inner: 5676.77 | bwd_allreduce: 71.32 | step: 18.98

  8%|▊         | 3201/41250 [7:43:17<91:27:19,  8.65s/it]
                                                         
{'loss': 0.0997, 'grad_norm': 0.7319098711013794, 'learning_rate': 3.9762917268371844e-05, 'epoch': 0.78}

  8%|▊         | 3201/41250 [7:43:17<91:27:19,  8.65s/it][2025-04-25 15:41:00,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:41:00,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.25 | bwd_microstep: 5707.97 | bwd_inner_microstep: 5687.60 | bwd_allreduce_microstep: 20.32 | step_microstep: 19.04
[2025-04-25 15:41:00,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.25 | bwd: 5707.98 | bwd_inner: 5687.60 | bwd_allreduce: 20.34 | step: 19.05

  8%|▊         | 3202/41250 [7:43:25<91:24:27,  8.65s/it]
                                                         
{'loss': 0.2643, 'grad_norm': 1.3041229248046875, 'learning_rate': 3.976267613438759e-05, 'epoch': 0.78}

  8%|▊         | 3202/41250 [7:43:25<91:24:27,  8.65s/it][2025-04-25 15:41:09,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.16
[2025-04-25 15:41:09,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.41 | bwd_microstep: 5719.61 | bwd_inner_microstep: 5679.79 | bwd_allreduce_microstep: 39.77 | step_microstep: 19.40
[2025-04-25 15:41:09,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.41 | bwd: 5719.62 | bwd_inner: 5679.79 | bwd_allreduce: 39.79 | step: 19.40

  8%|▊         | 3203/41250 [7:43:34<91:24:24,  8.65s/it]
                                                         
{'loss': 0.0714, 'grad_norm': 0.6391189098358154, 'learning_rate': 3.976243487857033e-05, 'epoch': 0.78}

  8%|▊         | 3203/41250 [7:43:34<91:24:24,  8.65s/it][2025-04-25 15:41:17,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:41:17,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5775.69 | bwd_inner_microstep: 5651.34 | bwd_allreduce_microstep: 124.30 | step_microstep: 18.51
[2025-04-25 15:41:17,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5775.70 | bwd_inner: 5651.34 | bwd_allreduce: 124.32 | step: 18.52

  8%|▊         | 3204/41250 [7:43:43<91:30:46,  8.66s/it]
                                                         
{'loss': 0.1915, 'grad_norm': 1.8477296829223633, 'learning_rate': 3.976219350092154e-05, 'epoch': 0.78}

  8%|▊         | 3204/41250 [7:43:43<91:30:46,  8.66s/it][2025-04-25 15:41:26,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 15:41:26,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.52 | bwd_microstep: 5750.78 | bwd_inner_microstep: 5639.75 | bwd_allreduce_microstep: 110.98 | step_microstep: 19.40
[2025-04-25 15:41:26,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.52 | bwd: 5750.80 | bwd_inner: 5639.75 | bwd_allreduce: 111.00 | step: 19.41

  8%|▊         | 3205/41250 [7:43:51<91:31:08,  8.66s/it]
                                                         
{'loss': 0.226, 'grad_norm': 1.1944340467453003, 'learning_rate': 3.9761952001442726e-05, 'epoch': 0.78}

  8%|▊         | 3205/41250 [7:43:51<91:31:08,  8.66s/it][2025-04-25 15:41:35,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 15:41:35,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5751.49 | bwd_inner_microstep: 5686.21 | bwd_allreduce_microstep: 65.24 | step_microstep: 19.22
[2025-04-25 15:41:35,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5751.51 | bwd_inner: 5686.21 | bwd_allreduce: 65.26 | step: 19.22

  8%|▊         | 3206/41250 [7:44:00<91:35:36,  8.67s/it]
                                                         
{'loss': 0.081, 'grad_norm': 0.6096249222755432, 'learning_rate': 3.976171038013536e-05, 'epoch': 0.78}

  8%|▊         | 3206/41250 [7:44:00<91:35:36,  8.67s/it][2025-04-25 15:41:43,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 15:41:43,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.04 | bwd_microstep: 5747.65 | bwd_inner_microstep: 5654.51 | bwd_allreduce_microstep: 93.09 | step_microstep: 18.55
[2025-04-25 15:41:43,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5747.67 | bwd_inner: 5654.51 | bwd_allreduce: 93.11 | step: 18.56

  8%|▊         | 3207/41250 [7:44:09<91:34:14,  8.67s/it]
                                                         
{'loss': 0.1334, 'grad_norm': 1.3158279657363892, 'learning_rate': 3.9761468637000924e-05, 'epoch': 0.78}

  8%|▊         | 3207/41250 [7:44:09<91:34:14,  8.67s/it][2025-04-25 15:41:52,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:41:52,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.08 | bwd_microstep: 5714.53 | bwd_inner_microstep: 5701.71 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.86
[2025-04-25 15:41:52,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.08 | bwd: 5714.54 | bwd_inner: 5701.71 | bwd_allreduce: 12.79 | step: 18.86

  8%|▊         | 3208/41250 [7:44:17<91:31:53,  8.66s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 1.2913799285888672, 'learning_rate': 3.976122677204093e-05, 'epoch': 0.78}

  8%|▊         | 3208/41250 [7:44:17<91:31:53,  8.66s/it][2025-04-25 15:42:01,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 15:42:01,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.07 | bwd_microstep: 5744.00 | bwd_inner_microstep: 5655.23 | bwd_allreduce_microstep: 88.71 | step_microstep: 19.05
[2025-04-25 15:42:01,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.07 | bwd: 5744.01 | bwd_inner: 5655.23 | bwd_allreduce: 88.74 | step: 19.05

  8%|▊         | 3209/41250 [7:44:26<91:32:07,  8.66s/it]
                                                         
{'loss': 0.2772, 'grad_norm': 1.484734296798706, 'learning_rate': 3.9760984785256866e-05, 'epoch': 0.78}

  8%|▊         | 3209/41250 [7:44:26<91:32:07,  8.66s/it][2025-04-25 15:42:09,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.15 | optimizer_step: 1.06
[2025-04-25 15:42:09,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5702.13 | bwd_inner_microstep: 5688.30 | bwd_allreduce_microstep: 13.76 | step_microstep: 20.29
[2025-04-25 15:42:09,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5702.15 | bwd_inner: 5688.30 | bwd_allreduce: 13.79 | step: 20.29

  8%|▊         | 3210/41250 [7:44:34<91:27:04,  8.65s/it]
                                                         
{'loss': 0.1958, 'grad_norm': 1.323895812034607, 'learning_rate': 3.9760742676650205e-05, 'epoch': 0.78}

  8%|▊         | 3210/41250 [7:44:34<91:27:04,  8.65s/it][2025-04-25 15:42:18,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:42:18,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.29 | bwd_microstep: 5720.67 | bwd_inner_microstep: 5692.36 | bwd_allreduce_microstep: 28.27 | step_microstep: 18.78
[2025-04-25 15:42:18,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.29 | bwd: 5720.68 | bwd_inner: 5692.36 | bwd_allreduce: 28.29 | step: 18.78

  8%|▊         | 3211/41250 [7:44:43<91:28:57,  8.66s/it]
                                                         
{'loss': 0.1101, 'grad_norm': 0.9167370796203613, 'learning_rate': 3.9760500446222466e-05, 'epoch': 0.78}

  8%|▊         | 3211/41250 [7:44:43<91:28:57,  8.66s/it][2025-04-25 15:42:27,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 15:42:27,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.11 | bwd_microstep: 5752.11 | bwd_inner_microstep: 5711.00 | bwd_allreduce_microstep: 41.05 | step_microstep: 19.33
[2025-04-25 15:42:27,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.11 | bwd: 5752.12 | bwd_inner: 5711.00 | bwd_allreduce: 41.08 | step: 19.33

  8%|▊         | 3212/41250 [7:44:52<91:34:46,  8.67s/it]
                                                         
{'loss': 0.2976, 'grad_norm': 1.7630701065063477, 'learning_rate': 3.976025809397512e-05, 'epoch': 0.78}

  8%|▊         | 3212/41250 [7:44:52<91:34:46,  8.67s/it][2025-04-25 15:42:35,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 15:42:35,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5751.40 | bwd_inner_microstep: 5689.14 | bwd_allreduce_microstep: 62.22 | step_microstep: 18.72
[2025-04-25 15:42:35,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5751.42 | bwd_inner: 5689.14 | bwd_allreduce: 62.23 | step: 18.72

  8%|▊         | 3213/41250 [7:45:01<91:37:33,  8.67s/it]
                                                         
{'loss': 0.136, 'grad_norm': 1.2471948862075806, 'learning_rate': 3.976001561990967e-05, 'epoch': 0.78}

  8%|▊         | 3213/41250 [7:45:01<91:37:33,  8.67s/it][2025-04-25 15:42:44,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:42:44,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5732.75 | bwd_inner_microstep: 5701.28 | bwd_allreduce_microstep: 31.43 | step_microstep: 18.72
[2025-04-25 15:42:44,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5732.77 | bwd_inner: 5701.28 | bwd_allreduce: 31.45 | step: 18.72

  8%|▊         | 3214/41250 [7:45:09<91:37:01,  8.67s/it]
                                                         
{'loss': 0.1727, 'grad_norm': 1.2804231643676758, 'learning_rate': 3.975977302402761e-05, 'epoch': 0.78}

  8%|▊         | 3214/41250 [7:45:09<91:37:01,  8.67s/it][2025-04-25 15:42:53,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 15:42:53,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.30 | bwd_microstep: 5704.24 | bwd_inner_microstep: 5691.54 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.43
[2025-04-25 15:42:53,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.30 | bwd: 5704.25 | bwd_inner: 5691.54 | bwd_allreduce: 12.67 | step: 19.43

  8%|▊         | 3215/41250 [7:45:18<91:32:10,  8.66s/it]
                                                         
{'loss': 0.1952, 'grad_norm': 1.3844404220581055, 'learning_rate': 3.9759530306330434e-05, 'epoch': 0.78}

  8%|▊         | 3215/41250 [7:45:18<91:32:10,  8.66s/it][2025-04-25 15:43:01,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 15:43:01,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5758.26 | bwd_inner_microstep: 5687.57 | bwd_allreduce_microstep: 70.64 | step_microstep: 19.12
[2025-04-25 15:43:01,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5758.27 | bwd_inner: 5687.57 | bwd_allreduce: 70.66 | step: 19.12

  8%|▊         | 3216/41250 [7:45:27<91:36:39,  8.67s/it]
                                                         
{'loss': 0.2831, 'grad_norm': 1.665919542312622, 'learning_rate': 3.9759287466819646e-05, 'epoch': 0.78}

  8%|▊         | 3216/41250 [7:45:27<91:36:39,  8.67s/it][2025-04-25 15:43:10,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:43:10,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.68 | bwd_microstep: 5729.99 | bwd_inner_microstep: 5715.12 | bwd_allreduce_microstep: 14.83 | step_microstep: 18.73
[2025-04-25 15:43:10,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.68 | bwd: 5730.00 | bwd_inner: 5715.12 | bwd_allreduce: 14.84 | step: 18.74

  8%|▊         | 3217/41250 [7:45:35<91:37:13,  8.67s/it]
                                                         
{'loss': 0.2217, 'grad_norm': 1.506636142730713, 'learning_rate': 3.975904450549673e-05, 'epoch': 0.78}

  8%|▊         | 3217/41250 [7:45:35<91:37:13,  8.67s/it][2025-04-25 15:43:19,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 15:43:19,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.17 | bwd_microstep: 5743.45 | bwd_inner_microstep: 5699.06 | bwd_allreduce_microstep: 44.35 | step_microstep: 18.59
[2025-04-25 15:43:19,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.17 | bwd: 5743.47 | bwd_inner: 5699.06 | bwd_allreduce: 44.37 | step: 18.59

  8%|▊         | 3218/41250 [7:45:44<91:37:03,  8.67s/it]
                                                         
{'loss': 0.1934, 'grad_norm': 0.9614657163619995, 'learning_rate': 3.97588014223632e-05, 'epoch': 0.78}

  8%|▊         | 3218/41250 [7:45:44<91:37:03,  8.67s/it][2025-04-25 15:43:27,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 15:43:27,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.05 | bwd_microstep: 5768.08 | bwd_inner_microstep: 5654.13 | bwd_allreduce_microstep: 113.89 | step_microstep: 18.83
[2025-04-25 15:43:27,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.05 | bwd: 5768.09 | bwd_inner: 5654.13 | bwd_allreduce: 113.91 | step: 18.83

  8%|▊         | 3219/41250 [7:45:53<91:38:04,  8.67s/it]
                                                         
{'loss': 0.2845, 'grad_norm': 2.025287389755249, 'learning_rate': 3.975855821742054e-05, 'epoch': 0.78}

  8%|▊         | 3219/41250 [7:45:53<91:38:04,  8.67s/it][2025-04-25 15:43:36,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 15:43:36,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.21 | bwd_microstep: 5700.18 | bwd_inner_microstep: 5652.89 | bwd_allreduce_microstep: 47.25 | step_microstep: 19.49
[2025-04-25 15:43:36,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.21 | bwd: 5700.20 | bwd_inner: 5652.89 | bwd_allreduce: 47.26 | step: 19.49

  8%|▊         | 3220/41250 [7:46:01<91:27:07,  8.66s/it]
                                                         
{'loss': 0.0974, 'grad_norm': 1.222656011581421, 'learning_rate': 3.9758314890670255e-05, 'epoch': 0.78}

  8%|▊         | 3220/41250 [7:46:01<91:27:07,  8.66s/it][2025-04-25 15:43:45,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:43:45,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.52 | bwd_microstep: 5758.47 | bwd_inner_microstep: 5717.54 | bwd_allreduce_microstep: 40.89 | step_microstep: 18.75
[2025-04-25 15:43:45,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.52 | bwd: 5758.48 | bwd_inner: 5717.54 | bwd_allreduce: 40.90 | step: 18.76

  8%|▊         | 3221/41250 [7:46:10<91:33:58,  8.67s/it]
                                                         
{'loss': 0.0888, 'grad_norm': 1.0605440139770508, 'learning_rate': 3.975807144211384e-05, 'epoch': 0.78}

  8%|▊         | 3221/41250 [7:46:10<91:33:58,  8.67s/it][2025-04-25 15:43:53,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:43:53,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.38 | bwd_microstep: 5721.15 | bwd_inner_microstep: 5660.94 | bwd_allreduce_microstep: 60.16 | step_microstep: 18.60
[2025-04-25 15:43:53,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.38 | bwd: 5721.16 | bwd_inner: 5660.94 | bwd_allreduce: 60.18 | step: 18.60

  8%|▊         | 3222/41250 [7:46:19<91:27:42,  8.66s/it]
                                                         
{'loss': 0.1954, 'grad_norm': 1.0230343341827393, 'learning_rate': 3.975782787175281e-05, 'epoch': 0.78}

  8%|▊         | 3222/41250 [7:46:19<91:27:42,  8.66s/it][2025-04-25 15:44:02,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 15:44:02,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.79 | bwd_microstep: 5819.44 | bwd_inner_microstep: 5792.29 | bwd_allreduce_microstep: 27.10 | step_microstep: 18.96
[2025-04-25 15:44:02,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.79 | bwd: 5819.45 | bwd_inner: 5792.29 | bwd_allreduce: 27.12 | step: 18.96

  8%|▊         | 3223/41250 [7:46:27<91:54:35,  8.70s/it]
                                                         
{'loss': 0.368, 'grad_norm': 2.437575340270996, 'learning_rate': 3.9757584179588654e-05, 'epoch': 0.78}

  8%|▊         | 3223/41250 [7:46:27<91:54:35,  8.70s/it][2025-04-25 15:44:11,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.14 | optimizer_step: 1.06
[2025-04-25 15:44:11,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.14 | bwd_microstep: 5721.19 | bwd_inner_microstep: 5668.87 | bwd_allreduce_microstep: 52.25 | step_microstep: 20.38
[2025-04-25 15:44:11,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.14 | bwd: 5721.20 | bwd_inner: 5668.88 | bwd_allreduce: 52.28 | step: 20.38

  8%|▊         | 3224/41250 [7:46:36<91:43:23,  8.68s/it]
                                                         
{'loss': 0.3633, 'grad_norm': 1.5040985345840454, 'learning_rate': 3.975734036562288e-05, 'epoch': 0.78}

  8%|▊         | 3224/41250 [7:46:36<91:43:23,  8.68s/it][2025-04-25 15:44:19,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:44:19,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5752.95 | bwd_inner_microstep: 5690.98 | bwd_allreduce_microstep: 61.93 | step_microstep: 18.71
[2025-04-25 15:44:19,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5752.97 | bwd_inner: 5690.98 | bwd_allreduce: 61.95 | step: 18.71

  8%|▊         | 3225/41250 [7:46:45<91:44:12,  8.69s/it]
                                                         
{'loss': 0.1129, 'grad_norm': 1.144919753074646, 'learning_rate': 3.9757096429856984e-05, 'epoch': 0.78}

  8%|▊         | 3225/41250 [7:46:45<91:44:12,  8.69s/it][2025-04-25 15:44:28,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 15:44:28,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5758.68 | bwd_inner_microstep: 5704.58 | bwd_allreduce_microstep: 54.06 | step_microstep: 18.98
[2025-04-25 15:44:28,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.76 | bwd: 5758.70 | bwd_inner: 5704.58 | bwd_allreduce: 54.08 | step: 18.98

  8%|▊         | 3226/41250 [7:46:53<91:46:01,  8.69s/it]
                                                         
{'loss': 0.245, 'grad_norm': 1.7320245504379272, 'learning_rate': 3.9756852372292475e-05, 'epoch': 0.78}

  8%|▊         | 3226/41250 [7:46:53<91:46:01,  8.69s/it][2025-04-25 15:44:37,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:44:37,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.99 | bwd_microstep: 5797.73 | bwd_inner_microstep: 5785.05 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.80
[2025-04-25 15:44:37,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.99 | bwd: 5797.74 | bwd_inner: 5785.05 | bwd_allreduce: 12.64 | step: 18.80

  8%|▊         | 3227/41250 [7:47:02<92:02:28,  8.71s/it]
                                                         
{'loss': 0.2872, 'grad_norm': 1.8511673212051392, 'learning_rate': 3.975660819293086e-05, 'epoch': 0.78}

  8%|▊         | 3227/41250 [7:47:02<92:02:28,  8.71s/it][2025-04-25 15:44:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:44:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.33 | bwd_microstep: 5742.65 | bwd_inner_microstep: 5704.48 | bwd_allreduce_microstep: 38.11 | step_microstep: 19.02
[2025-04-25 15:44:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.33 | bwd: 5742.66 | bwd_inner: 5704.48 | bwd_allreduce: 38.13 | step: 19.03

  8%|▊         | 3228/41250 [7:47:11<91:55:18,  8.70s/it]
                                                         
{'loss': 0.1636, 'grad_norm': 1.6315975189208984, 'learning_rate': 3.9756363891773644e-05, 'epoch': 0.78}

  8%|▊         | 3228/41250 [7:47:11<91:55:18,  8.70s/it][2025-04-25 15:44:54,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:44:54,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.69 | bwd_microstep: 5744.16 | bwd_inner_microstep: 5710.60 | bwd_allreduce_microstep: 33.51 | step_microstep: 18.77
[2025-04-25 15:44:54,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.69 | bwd: 5744.18 | bwd_inner: 5710.60 | bwd_allreduce: 33.52 | step: 18.77

  8%|▊         | 3229/41250 [7:47:19<91:51:30,  8.70s/it]
                                                         
{'loss': 0.1528, 'grad_norm': 2.3241524696350098, 'learning_rate': 3.9756119468822324e-05, 'epoch': 0.78}

  8%|▊         | 3229/41250 [7:47:19<91:51:30,  8.70s/it][2025-04-25 15:45:03,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 15:45:03,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.74 | bwd_microstep: 5726.44 | bwd_inner_microstep: 5713.67 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.29
[2025-04-25 15:45:03,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.74 | bwd: 5726.46 | bwd_inner: 5713.67 | bwd_allreduce: 12.75 | step: 19.29

  8%|▊         | 3230/41250 [7:47:28<91:45:00,  8.69s/it]
                                                         
{'loss': 0.0745, 'grad_norm': 0.4684620201587677, 'learning_rate': 3.975587492407842e-05, 'epoch': 0.78}

  8%|▊         | 3230/41250 [7:47:28<91:45:00,  8.69s/it][2025-04-25 15:45:11,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 15:45:11,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.76 | bwd_microstep: 5777.38 | bwd_inner_microstep: 5659.18 | bwd_allreduce_microstep: 118.16 | step_microstep: 18.65
[2025-04-25 15:45:11,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.76 | bwd: 5777.39 | bwd_inner: 5659.18 | bwd_allreduce: 118.17 | step: 18.66

  8%|▊         | 3231/41250 [7:47:37<91:45:22,  8.69s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 1.1781032085418701, 'learning_rate': 3.975563025754342e-05, 'epoch': 0.78}

  8%|▊         | 3231/41250 [7:47:37<91:45:22,  8.69s/it][2025-04-25 15:45:20,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:45:20,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.71 | bwd_microstep: 5786.18 | bwd_inner_microstep: 5773.14 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.58
[2025-04-25 15:45:20,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.71 | bwd: 5786.19 | bwd_inner: 5773.14 | bwd_allreduce: 13.01 | step: 18.58

  8%|▊         | 3232/41250 [7:47:46<91:57:45,  8.71s/it]
                                                         
{'loss': 0.084, 'grad_norm': 0.6878122091293335, 'learning_rate': 3.975538546921886e-05, 'epoch': 0.78}

  8%|▊         | 3232/41250 [7:47:46<91:57:45,  8.71s/it][2025-04-25 15:45:29,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:45:29,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.03 | bwd_microstep: 5746.97 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 42.87 | step_microstep: 19.01
[2025-04-25 15:45:29,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.03 | bwd: 5746.98 | bwd_inner: 5704.05 | bwd_allreduce: 42.89 | step: 19.01

  8%|▊         | 3233/41250 [7:47:54<91:52:59,  8.70s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 1.3986204862594604, 'learning_rate': 3.9755140559106226e-05, 'epoch': 0.78}

  8%|▊         | 3233/41250 [7:47:54<91:52:59,  8.70s/it][2025-04-25 15:45:38,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 15:45:38,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5765.67 | bwd_inner_microstep: 5661.17 | bwd_allreduce_microstep: 104.45 | step_microstep: 19.15
[2025-04-25 15:45:38,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5765.68 | bwd_inner: 5661.17 | bwd_allreduce: 104.47 | step: 19.15

  8%|▊         | 3234/41250 [7:48:03<91:48:41,  8.69s/it]
                                                         
{'loss': 0.1292, 'grad_norm': 0.8474724888801575, 'learning_rate': 3.9754895527207036e-05, 'epoch': 0.78}

  8%|▊         | 3234/41250 [7:48:03<91:48:41,  8.69s/it][2025-04-25 15:45:46,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 15:45:46,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5698.06 | bwd_inner_microstep: 5685.25 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.99
[2025-04-25 15:45:46,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5698.08 | bwd_inner: 5685.24 | bwd_allreduce: 12.79 | step: 19.00

  8%|▊         | 3235/41250 [7:48:12<91:37:43,  8.68s/it]
                                                         
{'loss': 0.1098, 'grad_norm': 2.2557194232940674, 'learning_rate': 3.9754650373522804e-05, 'epoch': 0.78}

  8%|▊         | 3235/41250 [7:48:12<91:37:43,  8.68s/it][2025-04-25 15:45:55,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:45:55,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.08 | bwd_microstep: 5710.62 | bwd_inner_microstep: 5697.90 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.88
[2025-04-25 15:45:55,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.09 | bwd: 5710.63 | bwd_inner: 5697.90 | bwd_allreduce: 12.69 | step: 18.89

  8%|▊         | 3236/41250 [7:48:20<91:30:42,  8.67s/it]
                                                         
{'loss': 0.24, 'grad_norm': 1.8715301752090454, 'learning_rate': 3.9754405098055035e-05, 'epoch': 0.78}

  8%|▊         | 3236/41250 [7:48:20<91:30:42,  8.67s/it][2025-04-25 15:46:04,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 15:46:04,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5710.18 | bwd_inner_microstep: 5659.34 | bwd_allreduce_microstep: 50.79 | step_microstep: 18.92
[2025-04-25 15:46:04,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5710.19 | bwd_inner: 5659.34 | bwd_allreduce: 50.81 | step: 18.92

  8%|▊         | 3237/41250 [7:48:29<91:22:15,  8.65s/it]
                                                         
{'loss': 0.17, 'grad_norm': 1.6708656549453735, 'learning_rate': 3.975415970080525e-05, 'epoch': 0.78}

  8%|▊         | 3237/41250 [7:48:29<91:22:15,  8.65s/it][2025-04-25 15:46:12,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.28 | optimizer_step: 1.06
[2025-04-25 15:46:12,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.20 | bwd_microstep: 5733.17 | bwd_inner_microstep: 5703.28 | bwd_allreduce_microstep: 29.83 | step_microstep: 21.52
[2025-04-25 15:46:12,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.20 | bwd: 5733.18 | bwd_inner: 5703.28 | bwd_allreduce: 29.85 | step: 21.52

  8%|▊         | 3238/41250 [7:48:38<91:27:26,  8.66s/it]
                                                         
{'loss': 0.1965, 'grad_norm': 1.0792951583862305, 'learning_rate': 3.975391418177495e-05, 'epoch': 0.78}

  8%|▊         | 3238/41250 [7:48:38<91:27:26,  8.66s/it][2025-04-25 15:46:21,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:46:21,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.70 | bwd_microstep: 5742.75 | bwd_inner_microstep: 5654.65 | bwd_allreduce_microstep: 88.05 | step_microstep: 18.45
[2025-04-25 15:46:21,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.71 | bwd: 5742.76 | bwd_inner: 5654.65 | bwd_allreduce: 88.07 | step: 18.45

  8%|▊         | 3239/41250 [7:48:46<91:26:04,  8.66s/it]
                                                         
{'loss': 0.2727, 'grad_norm': 1.9765671491622925, 'learning_rate': 3.975366854096565e-05, 'epoch': 0.79}

  8%|▊         | 3239/41250 [7:48:46<91:26:04,  8.66s/it][2025-04-25 15:46:30,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 15:46:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.81 | bwd_microstep: 5756.04 | bwd_inner_microstep: 5695.78 | bwd_allreduce_microstep: 60.21 | step_microstep: 18.81
[2025-04-25 15:46:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.81 | bwd: 5756.05 | bwd_inner: 5695.78 | bwd_allreduce: 60.23 | step: 18.81

  8%|▊         | 3240/41250 [7:48:55<91:30:54,  8.67s/it]
                                                         
{'loss': 0.1122, 'grad_norm': 0.8918008804321289, 'learning_rate': 3.9753422778378885e-05, 'epoch': 0.79}

  8%|▊         | 3240/41250 [7:48:55<91:30:54,  8.67s/it][2025-04-25 15:46:38,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:46:38,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.17 | bwd_microstep: 5866.28 | bwd_inner_microstep: 5679.48 | bwd_allreduce_microstep: 186.75 | step_microstep: 18.96
[2025-04-25 15:46:38,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.17 | bwd: 5866.29 | bwd_inner: 5679.48 | bwd_allreduce: 186.77 | step: 18.97

  8%|▊         | 3241/41250 [7:49:04<91:53:50,  8.70s/it]
                                                         
{'loss': 0.2022, 'grad_norm': 1.1589324474334717, 'learning_rate': 3.975317689401614e-05, 'epoch': 0.79}

  8%|▊         | 3241/41250 [7:49:04<91:53:50,  8.70s/it][2025-04-25 15:46:47,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:46:47,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.63 | bwd_microstep: 5689.89 | bwd_inner_microstep: 5653.34 | bwd_allreduce_microstep: 36.50 | step_microstep: 18.79
[2025-04-25 15:46:47,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.63 | bwd: 5689.90 | bwd_inner: 5653.34 | bwd_allreduce: 36.52 | step: 18.79

  8%|▊         | 3242/41250 [7:49:12<91:34:19,  8.67s/it]
                                                         
{'loss': 0.1401, 'grad_norm': 1.249153971672058, 'learning_rate': 3.9752930887878955e-05, 'epoch': 0.79}

  8%|▊         | 3242/41250 [7:49:12<91:34:19,  8.67s/it][2025-04-25 15:46:56,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:46:56,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.22 | bwd_microstep: 5692.95 | bwd_inner_microstep: 5654.81 | bwd_allreduce_microstep: 38.09 | step_microstep: 18.83
[2025-04-25 15:46:56,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.22 | bwd: 5692.96 | bwd_inner: 5654.81 | bwd_allreduce: 38.11 | step: 18.84

  8%|▊         | 3243/41250 [7:49:21<91:21:01,  8.65s/it]
                                                         
{'loss': 0.1042, 'grad_norm': 0.8949598073959351, 'learning_rate': 3.975268475996882e-05, 'epoch': 0.79}

  8%|▊         | 3243/41250 [7:49:21<91:21:01,  8.65s/it][2025-04-25 15:47:04,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.09 | optimizer_step: 1.00
[2025-04-25 15:47:04,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.73 | bwd_microstep: 5688.36 | bwd_inner_microstep: 5650.18 | bwd_allreduce_microstep: 38.13 | step_microstep: 18.86
[2025-04-25 15:47:04,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.73 | bwd: 5688.38 | bwd_inner: 5650.18 | bwd_allreduce: 38.15 | step: 18.86

  8%|▊         | 3244/41250 [7:49:29<91:13:04,  8.64s/it]
                                                         
{'loss': 0.1787, 'grad_norm': 1.5274438858032227, 'learning_rate': 3.975243851028729e-05, 'epoch': 0.79}

  8%|▊         | 3244/41250 [7:49:29<91:13:04,  8.64s/it][2025-04-25 15:47:13,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 15:47:13,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.14 | bwd_microstep: 5731.20 | bwd_inner_microstep: 5685.16 | bwd_allreduce_microstep: 45.99 | step_microstep: 19.04
[2025-04-25 15:47:13,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.14 | bwd: 5731.21 | bwd_inner: 5685.16 | bwd_allreduce: 46.01 | step: 19.05

  8%|▊         | 3245/41250 [7:49:38<91:16:45,  8.65s/it]
                                                         
{'loss': 0.2001, 'grad_norm': 1.2097711563110352, 'learning_rate': 3.975219213883584e-05, 'epoch': 0.79}

  8%|▊         | 3245/41250 [7:49:38<91:16:45,  8.65s/it][2025-04-25 15:47:21,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:47:21,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.82 | bwd_microstep: 5697.62 | bwd_inner_microstep: 5684.91 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.67
[2025-04-25 15:47:21,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.82 | bwd: 5697.63 | bwd_inner: 5684.91 | bwd_allreduce: 12.68 | step: 18.67

  8%|▊         | 3246/41250 [7:49:47<91:12:10,  8.64s/it]
                                                         
{'loss': 0.1588, 'grad_norm': 2.365947723388672, 'learning_rate': 3.9751945645616025e-05, 'epoch': 0.79}

  8%|▊         | 3246/41250 [7:49:47<91:12:10,  8.64s/it][2025-04-25 15:47:30,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 15:47:30,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.23 | bwd_microstep: 5701.62 | bwd_inner_microstep: 5688.63 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.46
[2025-04-25 15:47:30,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.23 | bwd: 5701.63 | bwd_inner: 5688.63 | bwd_allreduce: 12.96 | step: 19.46

  8%|▊         | 3247/41250 [7:49:55<91:10:55,  8.64s/it]
                                                         
{'loss': 0.126, 'grad_norm': 1.003074288368225, 'learning_rate': 3.975169903062934e-05, 'epoch': 0.79}

  8%|▊         | 3247/41250 [7:49:55<91:10:55,  8.64s/it][2025-04-25 15:47:39,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 15:47:39,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.78 | bwd_microstep: 5703.91 | bwd_inner_microstep: 5687.25 | bwd_allreduce_microstep: 16.61 | step_microstep: 19.00
[2025-04-25 15:47:39,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.78 | bwd: 5703.92 | bwd_inner: 5687.25 | bwd_allreduce: 16.63 | step: 19.01

  8%|▊         | 3248/41250 [7:50:04<91:11:57,  8.64s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.1649590730667114, 'learning_rate': 3.9751452293877325e-05, 'epoch': 0.79}

  8%|▊         | 3248/41250 [7:50:04<91:11:57,  8.64s/it][2025-04-25 15:47:47,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:47:47,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.32 | bwd_microstep: 5715.14 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.80
[2025-04-25 15:47:47,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.32 | bwd: 5715.15 | bwd_inner: 5702.41 | bwd_allreduce: 12.70 | step: 18.80

  8%|▊         | 3249/41250 [7:50:13<91:14:16,  8.64s/it]
                                                         
{'loss': 0.2044, 'grad_norm': 1.0631667375564575, 'learning_rate': 3.975120543536149e-05, 'epoch': 0.79}

  8%|▊         | 3249/41250 [7:50:13<91:14:16,  8.64s/it][2025-04-25 15:47:56,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 15:47:56,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5679.11 | bwd_inner_microstep: 5656.58 | bwd_allreduce_microstep: 22.49 | step_microstep: 18.95
[2025-04-25 15:47:56,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5679.12 | bwd_inner: 5656.58 | bwd_allreduce: 22.51 | step: 18.95

  8%|▊         | 3250/41250 [7:50:21<91:04:23,  8.63s/it]
                                                         
{'loss': 0.131, 'grad_norm': 1.0143486261367798, 'learning_rate': 3.975095845508335e-05, 'epoch': 0.79}

  8%|▊         | 3250/41250 [7:50:21<91:04:23,  8.63s/it][2025-04-25 15:48:05,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 15:48:05,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.97 | bwd_microstep: 5744.18 | bwd_inner_microstep: 5630.23 | bwd_allreduce_microstep: 113.89 | step_microstep: 19.05
[2025-04-25 15:48:05,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.97 | bwd: 5744.19 | bwd_inner: 5630.23 | bwd_allreduce: 113.91 | step: 19.05

  8%|▊         | 3251/41250 [7:50:30<91:07:37,  8.63s/it]
                                                         
{'loss': 0.1589, 'grad_norm': 1.2449390888214111, 'learning_rate': 3.9750711353044446e-05, 'epoch': 0.79}

  8%|▊         | 3251/41250 [7:50:30<91:07:37,  8.63s/it][2025-04-25 15:48:13,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:48:13,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5847.60 | bwd_inner_microstep: 5648.65 | bwd_allreduce_microstep: 198.90 | step_microstep: 18.79
[2025-04-25 15:48:13,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5847.61 | bwd_inner: 5648.65 | bwd_allreduce: 198.92 | step: 18.79

  8%|▊         | 3252/41250 [7:50:39<91:30:45,  8.67s/it]
                                                         
{'loss': 0.3265, 'grad_norm': 2.0815157890319824, 'learning_rate': 3.975046412924629e-05, 'epoch': 0.79}

  8%|▊         | 3252/41250 [7:50:39<91:30:45,  8.67s/it][2025-04-25 15:48:22,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:48:22,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.76 | bwd_microstep: 5682.19 | bwd_inner_microstep: 5651.65 | bwd_allreduce_microstep: 30.49 | step_microstep: 18.48
[2025-04-25 15:48:22,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.76 | bwd: 5682.20 | bwd_inner: 5651.65 | bwd_allreduce: 30.51 | step: 18.48

  8%|▊         | 3253/41250 [7:50:47<91:15:20,  8.65s/it]
                                                         
{'loss': 0.1301, 'grad_norm': 1.068247675895691, 'learning_rate': 3.97502167836904e-05, 'epoch': 0.79}

  8%|▊         | 3253/41250 [7:50:47<91:15:20,  8.65s/it][2025-04-25 15:48:31,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:48:31,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.36 | bwd_microstep: 5706.11 | bwd_inner_microstep: 5693.69 | bwd_allreduce_microstep: 12.38 | step_microstep: 18.70
[2025-04-25 15:48:31,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.36 | bwd: 5706.12 | bwd_inner: 5693.69 | bwd_allreduce: 12.39 | step: 18.71

  8%|▊         | 3254/41250 [7:50:56<91:13:36,  8.64s/it]
                                                         
{'loss': 0.1244, 'grad_norm': 0.9807857275009155, 'learning_rate': 3.974996931637831e-05, 'epoch': 0.79}

  8%|▊         | 3254/41250 [7:50:56<91:13:36,  8.64s/it][2025-04-25 15:48:39,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:48:39,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.97 | bwd_microstep: 5791.60 | bwd_inner_microstep: 5778.72 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.84
[2025-04-25 15:48:39,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.97 | bwd: 5791.61 | bwd_inner: 5778.72 | bwd_allreduce: 12.85 | step: 18.85

  8%|▊         | 3255/41250 [7:51:05<91:36:22,  8.68s/it]
                                                         
{'loss': 0.2037, 'grad_norm': 1.739943027496338, 'learning_rate': 3.974972172731155e-05, 'epoch': 0.79}

  8%|▊         | 3255/41250 [7:51:05<91:36:22,  8.68s/it][2025-04-25 15:48:48,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:48:48,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.27 | bwd_microstep: 5836.66 | bwd_inner_microstep: 5702.29 | bwd_allreduce_microstep: 134.32 | step_microstep: 18.51
[2025-04-25 15:48:48,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.27 | bwd: 5836.67 | bwd_inner: 5702.29 | bwd_allreduce: 134.34 | step: 18.51

  8%|▊         | 3256/41250 [7:51:13<91:54:09,  8.71s/it]
                                                         
{'loss': 0.2151, 'grad_norm': 2.5662240982055664, 'learning_rate': 3.9749474016491635e-05, 'epoch': 0.79}

  8%|▊         | 3256/41250 [7:51:13<91:54:09,  8.71s/it][2025-04-25 15:48:57,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:48:57,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.35 | bwd_microstep: 5688.19 | bwd_inner_microstep: 5641.61 | bwd_allreduce_microstep: 46.54 | step_microstep: 18.47
[2025-04-25 15:48:57,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.35 | bwd: 5688.20 | bwd_inner: 5641.60 | bwd_allreduce: 46.56 | step: 18.47

  8%|▊         | 3257/41250 [7:51:22<91:31:25,  8.67s/it]
                                                         
{'loss': 0.277, 'grad_norm': 1.2936270236968994, 'learning_rate': 3.97492261839201e-05, 'epoch': 0.79}

  8%|▊         | 3257/41250 [7:51:22<91:31:25,  8.67s/it][2025-04-25 15:49:05,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:49:05,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.44 | bwd_microstep: 5700.08 | bwd_inner_microstep: 5687.31 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.84
[2025-04-25 15:49:05,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.44 | bwd: 5700.10 | bwd_inner: 5687.31 | bwd_allreduce: 12.74 | step: 18.84

  8%|▊         | 3258/41250 [7:51:31<91:22:26,  8.66s/it]
                                                         
{'loss': 0.114, 'grad_norm': 1.1636788845062256, 'learning_rate': 3.974897822959847e-05, 'epoch': 0.79}

  8%|▊         | 3258/41250 [7:51:31<91:22:26,  8.66s/it][2025-04-25 15:49:14,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 15:49:14,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.05 | bwd_microstep: 5683.66 | bwd_inner_microstep: 5649.89 | bwd_allreduce_microstep: 33.72 | step_microstep: 18.55
[2025-04-25 15:49:14,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.05 | bwd: 5683.67 | bwd_inner: 5649.89 | bwd_allreduce: 33.73 | step: 18.55

  8%|▊         | 3259/41250 [7:51:39<91:09:18,  8.64s/it]
                                                         
{'loss': 0.222, 'grad_norm': 1.7928417921066284, 'learning_rate': 3.9748730153528276e-05, 'epoch': 0.79}

  8%|▊         | 3259/41250 [7:51:39<91:09:18,  8.64s/it][2025-04-25 15:49:23,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:49:23,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.10 | bwd_microstep: 5735.54 | bwd_inner_microstep: 5637.67 | bwd_allreduce_microstep: 97.82 | step_microstep: 18.57
[2025-04-25 15:49:23,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.10 | bwd: 5735.56 | bwd_inner: 5637.67 | bwd_allreduce: 97.84 | step: 18.57

  8%|▊         | 3260/41250 [7:51:48<91:09:13,  8.64s/it]
                                                         
{'loss': 0.1056, 'grad_norm': 0.9611411690711975, 'learning_rate': 3.974848195571104e-05, 'epoch': 0.79}

  8%|▊         | 3260/41250 [7:51:48<91:09:13,  8.64s/it][2025-04-25 15:49:31,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.34 | optimizer_step: 1.02
[2025-04-25 15:49:31,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.44 | bwd_microstep: 5750.96 | bwd_inner_microstep: 5629.36 | bwd_allreduce_microstep: 121.54 | step_microstep: 20.10
[2025-04-25 15:49:31,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.44 | bwd: 5750.97 | bwd_inner: 5629.36 | bwd_allreduce: 121.57 | step: 20.10

  8%|▊         | 3261/41250 [7:51:57<91:12:31,  8.64s/it]
                                                         
{'loss': 0.2385, 'grad_norm': 2.0360541343688965, 'learning_rate': 3.9748233636148296e-05, 'epoch': 0.79}

  8%|▊         | 3261/41250 [7:51:57<91:12:31,  8.64s/it][2025-04-25 15:49:40,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 15:49:40,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.83 | bwd_microstep: 5732.81 | bwd_inner_microstep: 5690.38 | bwd_allreduce_microstep: 42.39 | step_microstep: 18.93
[2025-04-25 15:49:40,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.83 | bwd: 5732.83 | bwd_inner: 5690.38 | bwd_allreduce: 42.40 | step: 18.93

  8%|▊         | 3262/41250 [7:52:05<91:16:05,  8.65s/it]
                                                         
{'loss': 0.2049, 'grad_norm': 1.040726900100708, 'learning_rate': 3.974798519484158e-05, 'epoch': 0.79}

  8%|▊         | 3262/41250 [7:52:05<91:16:05,  8.65s/it][2025-04-25 15:49:49,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:49:49,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.08 | bwd_microstep: 5701.46 | bwd_inner_microstep: 5688.58 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.03
[2025-04-25 15:49:49,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.08 | bwd: 5701.47 | bwd_inner: 5688.58 | bwd_allreduce: 12.85 | step: 19.03

  8%|▊         | 3263/41250 [7:52:14<91:12:19,  8.64s/it]
                                                         
{'loss': 0.1302, 'grad_norm': 0.9017081260681152, 'learning_rate': 3.974773663179241e-05, 'epoch': 0.79}

  8%|▊         | 3263/41250 [7:52:14<91:12:19,  8.64s/it][2025-04-25 15:49:57,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:49:57,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.99 | bwd_microstep: 5743.14 | bwd_inner_microstep: 5674.96 | bwd_allreduce_microstep: 68.14 | step_microstep: 18.98
[2025-04-25 15:49:57,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.99 | bwd: 5743.15 | bwd_inner: 5674.96 | bwd_allreduce: 68.15 | step: 18.98

  8%|▊         | 3264/41250 [7:52:22<91:16:53,  8.65s/it]
                                                         
{'loss': 0.259, 'grad_norm': 1.6110844612121582, 'learning_rate': 3.974748794700234e-05, 'epoch': 0.79}

  8%|▊         | 3264/41250 [7:52:23<91:16:53,  8.65s/it][2025-04-25 15:50:06,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 15:50:06,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.02 | bwd_microstep: 5698.97 | bwd_inner_microstep: 5686.13 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.62
[2025-04-25 15:50:06,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.02 | bwd: 5698.98 | bwd_inner: 5686.13 | bwd_allreduce: 12.81 | step: 18.62

  8%|▊         | 3265/41250 [7:52:31<91:13:32,  8.65s/it]
                                                         
{'loss': 0.2993, 'grad_norm': 2.1899168491363525, 'learning_rate': 3.974723914047289e-05, 'epoch': 0.79}

  8%|▊         | 3265/41250 [7:52:31<91:13:32,  8.65s/it][2025-04-25 15:50:14,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 15:50:14,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5684.17 | bwd_inner_microstep: 5646.83 | bwd_allreduce_microstep: 37.29 | step_microstep: 18.69
[2025-04-25 15:50:14,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5684.19 | bwd_inner: 5646.83 | bwd_allreduce: 37.31 | step: 18.70

  8%|▊         | 3266/41250 [7:52:40<91:04:23,  8.63s/it]
                                                         
{'loss': 0.1333, 'grad_norm': 1.983679175376892, 'learning_rate': 3.974699021220559e-05, 'epoch': 0.79}

  8%|▊         | 3266/41250 [7:52:40<91:04:23,  8.63s/it][2025-04-25 15:50:23,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:50:23,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.94 | bwd_microstep: 5729.63 | bwd_inner_microstep: 5674.52 | bwd_allreduce_microstep: 55.06 | step_microstep: 18.73
[2025-04-25 15:50:23,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.94 | bwd: 5729.64 | bwd_inner: 5674.52 | bwd_allreduce: 55.08 | step: 18.73

  8%|▊         | 3267/41250 [7:52:48<91:08:43,  8.64s/it]
                                                         
{'loss': 0.0917, 'grad_norm': 1.2350627183914185, 'learning_rate': 3.974674116220198e-05, 'epoch': 0.79}

  8%|▊         | 3267/41250 [7:52:48<91:08:43,  8.64s/it][2025-04-25 15:50:32,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.12 | optimizer_step: 0.97
[2025-04-25 15:50:32,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.31 | bwd_microstep: 5857.22 | bwd_inner_microstep: 5646.76 | bwd_allreduce_microstep: 210.41 | step_microstep: 19.00
[2025-04-25 15:50:32,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.31 | bwd: 5857.23 | bwd_inner: 5646.76 | bwd_allreduce: 210.43 | step: 19.00

  8%|▊         | 3268/41250 [7:52:57<91:34:06,  8.68s/it]
                                                         
{'loss': 0.1367, 'grad_norm': 0.9026291370391846, 'learning_rate': 3.9746491990463596e-05, 'epoch': 0.79}

  8%|▊         | 3268/41250 [7:52:57<91:34:06,  8.68s/it][2025-04-25 15:50:40,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:50:40,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5713.03 | bwd_inner_microstep: 5687.78 | bwd_allreduce_microstep: 25.20 | step_microstep: 18.89
[2025-04-25 15:50:40,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5713.04 | bwd_inner: 5687.78 | bwd_allreduce: 25.22 | step: 18.89

  8%|▊         | 3269/41250 [7:53:06<91:26:56,  8.67s/it]
                                                         
{'loss': 0.0747, 'grad_norm': 0.9505876898765564, 'learning_rate': 3.974624269699197e-05, 'epoch': 0.79}

  8%|▊         | 3269/41250 [7:53:06<91:26:56,  8.67s/it][2025-04-25 15:50:49,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 15:50:49,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.14 | bwd_microstep: 5735.04 | bwd_inner_microstep: 5704.64 | bwd_allreduce_microstep: 30.35 | step_microstep: 19.22
[2025-04-25 15:50:49,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.14 | bwd: 5735.06 | bwd_inner: 5704.64 | bwd_allreduce: 30.37 | step: 19.22

  8%|▊         | 3270/41250 [7:53:14<91:28:39,  8.67s/it]
                                                         
{'loss': 0.0705, 'grad_norm': 0.9793677926063538, 'learning_rate': 3.974599328178865e-05, 'epoch': 0.79}

  8%|▊         | 3270/41250 [7:53:14<91:28:39,  8.67s/it][2025-04-25 15:50:58,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 15:50:58,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.53 | bwd_microstep: 5692.72 | bwd_inner_microstep: 5650.63 | bwd_allreduce_microstep: 42.05 | step_microstep: 18.98
[2025-04-25 15:50:58,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.53 | bwd: 5692.74 | bwd_inner: 5650.63 | bwd_allreduce: 42.06 | step: 18.98

  8%|▊         | 3271/41250 [7:53:23<91:16:29,  8.65s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 1.3345701694488525, 'learning_rate': 3.974574374485515e-05, 'epoch': 0.79}

  8%|▊         | 3271/41250 [7:53:23<91:16:29,  8.65s/it][2025-04-25 15:51:07,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:51:07,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.21 | bwd_microstep: 5782.29 | bwd_inner_microstep: 5769.43 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.82
[2025-04-25 15:51:07,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.21 | bwd: 5782.30 | bwd_inner: 5769.43 | bwd_allreduce: 12.83 | step: 18.82

  8%|▊         | 3272/41250 [7:53:32<91:33:17,  8.68s/it]
                                                         
{'loss': 0.1063, 'grad_norm': 0.8107016086578369, 'learning_rate': 3.974549408619304e-05, 'epoch': 0.79}

  8%|▊         | 3272/41250 [7:53:32<91:33:17,  8.68s/it][2025-04-25 15:51:15,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:51:15,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5770.78 | bwd_inner_microstep: 5649.60 | bwd_allreduce_microstep: 121.12 | step_microstep: 18.60
[2025-04-25 15:51:15,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5770.79 | bwd_inner: 5649.60 | bwd_allreduce: 121.14 | step: 18.60

  8%|▊         | 3273/41250 [7:53:41<91:34:07,  8.68s/it]
                                                         
{'loss': 0.235, 'grad_norm': 1.7933427095413208, 'learning_rate': 3.974524430580383e-05, 'epoch': 0.79}

  8%|▊         | 3273/41250 [7:53:41<91:34:07,  8.68s/it][2025-04-25 15:51:24,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:51:24,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.02 | bwd_microstep: 5687.76 | bwd_inner_microstep: 5654.06 | bwd_allreduce_microstep: 33.65 | step_microstep: 18.92
[2025-04-25 15:51:24,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.02 | bwd: 5687.77 | bwd_inner: 5654.06 | bwd_allreduce: 33.67 | step: 18.92

  8%|▊         | 3274/41250 [7:53:49<91:18:25,  8.66s/it]
                                                         
{'loss': 0.2843, 'grad_norm': 2.2888307571411133, 'learning_rate': 3.974499440368908e-05, 'epoch': 0.79}

  8%|▊         | 3274/41250 [7:53:49<91:18:25,  8.66s/it][2025-04-25 15:51:32,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:51:32,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.37 | bwd_microstep: 5733.28 | bwd_inner_microstep: 5712.53 | bwd_allreduce_microstep: 20.71 | step_microstep: 18.79
[2025-04-25 15:51:32,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.38 | bwd: 5733.30 | bwd_inner: 5712.53 | bwd_allreduce: 20.72 | step: 18.79

  8%|▊         | 3275/41250 [7:53:58<91:21:06,  8.66s/it]
                                                         
{'loss': 0.1938, 'grad_norm': 1.1828889846801758, 'learning_rate': 3.9744744379850325e-05, 'epoch': 0.79}

  8%|▊         | 3275/41250 [7:53:58<91:21:06,  8.66s/it][2025-04-25 15:51:41,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 15:51:41,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.05 | bwd_microstep: 5753.80 | bwd_inner_microstep: 5641.53 | bwd_allreduce_microstep: 112.22 | step_microstep: 18.89
[2025-04-25 15:51:41,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.05 | bwd: 5753.82 | bwd_inner: 5641.53 | bwd_allreduce: 112.24 | step: 18.89

  8%|▊         | 3276/41250 [7:54:06<91:21:52,  8.66s/it]
                                                         
{'loss': 0.3682, 'grad_norm': 2.498380661010742, 'learning_rate': 3.97444942342891e-05, 'epoch': 0.79}

  8%|▊         | 3276/41250 [7:54:06<91:21:52,  8.66s/it][2025-04-25 15:51:50,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:51:50,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.78 | bwd_microstep: 5779.12 | bwd_inner_microstep: 5766.18 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.93
[2025-04-25 15:51:50,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.78 | bwd: 5779.14 | bwd_inner: 5766.18 | bwd_allreduce: 12.92 | step: 18.93

  8%|▊         | 3277/41250 [7:54:15<91:37:19,  8.69s/it]
                                                         
{'loss': 0.2186, 'grad_norm': 2.245851516723633, 'learning_rate': 3.974424396700695e-05, 'epoch': 0.79}

  8%|▊         | 3277/41250 [7:54:15<91:37:19,  8.69s/it][2025-04-25 15:51:59,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:51:59,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.30 | bwd_microstep: 5736.88 | bwd_inner_microstep: 5708.59 | bwd_allreduce_microstep: 28.23 | step_microstep: 18.85
[2025-04-25 15:51:59,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.30 | bwd: 5736.89 | bwd_inner: 5708.59 | bwd_allreduce: 28.25 | step: 18.85

  8%|▊         | 3278/41250 [7:54:24<91:34:48,  8.68s/it]
                                                         
{'loss': 0.0952, 'grad_norm': 1.0108580589294434, 'learning_rate': 3.974399357800542e-05, 'epoch': 0.79}

  8%|▊         | 3278/41250 [7:54:24<91:34:48,  8.68s/it][2025-04-25 15:52:07,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:52:07,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.07 | bwd_microstep: 5725.95 | bwd_inner_microstep: 5713.33 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.50
[2025-04-25 15:52:07,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.07 | bwd: 5725.96 | bwd_inner: 5713.33 | bwd_allreduce: 12.59 | step: 18.50

  8%|▊         | 3279/41250 [7:54:33<91:30:46,  8.68s/it]
                                                         
{'loss': 0.2397, 'grad_norm': 2.617396831512451, 'learning_rate': 3.9743743067286056e-05, 'epoch': 0.79}

  8%|▊         | 3279/41250 [7:54:33<91:30:46,  8.68s/it][2025-04-25 15:52:16,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 15:52:16,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.66 | bwd_microstep: 5778.79 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 116.67 | step_microstep: 19.36
[2025-04-25 15:52:16,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.66 | bwd: 5778.80 | bwd_inner: 5662.08 | bwd_allreduce: 116.69 | step: 19.36

  8%|▊         | 3280/41250 [7:54:41<91:34:15,  8.68s/it]
                                                         
{'loss': 0.1273, 'grad_norm': 1.144489049911499, 'learning_rate': 3.97434924348504e-05, 'epoch': 0.8}

  8%|▊         | 3280/41250 [7:54:41<91:34:15,  8.68s/it][2025-04-25 15:52:25,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 15:52:25,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.30 | bwd_microstep: 5712.23 | bwd_inner_microstep: 5699.43 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.63
[2025-04-25 15:52:25,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.30 | bwd: 5712.24 | bwd_inner: 5699.43 | bwd_allreduce: 12.77 | step: 18.63

  8%|▊         | 3281/41250 [7:54:50<91:27:27,  8.67s/it]
                                                         
{'loss': 0.4592, 'grad_norm': 3.6189725399017334, 'learning_rate': 3.974324168069999e-05, 'epoch': 0.8}

  8%|▊         | 3281/41250 [7:54:50<91:27:27,  8.67s/it][2025-04-25 15:52:33,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 15:52:33,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.74 | bwd_microstep: 5712.55 | bwd_inner_microstep: 5655.77 | bwd_allreduce_microstep: 56.74 | step_microstep: 18.53
[2025-04-25 15:52:33,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.74 | bwd: 5712.56 | bwd_inner: 5655.77 | bwd_allreduce: 56.76 | step: 18.53

  8%|▊         | 3282/41250 [7:54:58<91:18:57,  8.66s/it]
                                                         
{'loss': 0.2335, 'grad_norm': 1.4680066108703613, 'learning_rate': 3.974299080483638e-05, 'epoch': 0.8}

  8%|▊         | 3282/41250 [7:54:58<91:18:57,  8.66s/it][2025-04-25 15:52:42,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 15:52:42,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.25 | bwd_microstep: 5764.02 | bwd_inner_microstep: 5711.53 | bwd_allreduce_microstep: 52.45 | step_microstep: 18.46
[2025-04-25 15:52:42,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.25 | bwd: 5764.04 | bwd_inner: 5711.53 | bwd_allreduce: 52.46 | step: 18.46

  8%|▊         | 3283/41250 [7:55:07<91:26:25,  8.67s/it]
                                                         
{'loss': 0.1886, 'grad_norm': 4.4876275062561035, 'learning_rate': 3.974273980726113e-05, 'epoch': 0.8}

  8%|▊         | 3283/41250 [7:55:07<91:26:25,  8.67s/it][2025-04-25 15:52:51,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 15:52:51,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.67 | bwd_microstep: 5856.70 | bwd_inner_microstep: 5709.10 | bwd_allreduce_microstep: 147.55 | step_microstep: 18.75
[2025-04-25 15:52:51,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.67 | bwd: 5856.71 | bwd_inner: 5709.10 | bwd_allreduce: 147.57 | step: 18.75

  8%|▊         | 3284/41250 [7:55:16<91:50:34,  8.71s/it]
                                                         
{'loss': 0.2409, 'grad_norm': 1.8894948959350586, 'learning_rate': 3.9742488687975754e-05, 'epoch': 0.8}

  8%|▊         | 3284/41250 [7:55:16<91:50:34,  8.71s/it][2025-04-25 15:52:59,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 15:52:59,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.77 | bwd_microstep: 5723.25 | bwd_inner_microstep: 5710.52 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.66
[2025-04-25 15:52:59,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.77 | bwd: 5723.26 | bwd_inner: 5710.52 | bwd_allreduce: 12.70 | step: 18.66

  8%|▊         | 3285/41250 [7:55:25<91:43:27,  8.70s/it]
                                                         
{'loss': 0.174, 'grad_norm': 1.1459970474243164, 'learning_rate': 3.974223744698182e-05, 'epoch': 0.8}

  8%|▊         | 3285/41250 [7:55:25<91:43:27,  8.70s/it][2025-04-25 15:53:08,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:53:08,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5742.60 | bwd_inner_microstep: 5687.45 | bwd_allreduce_microstep: 55.11 | step_microstep: 18.48
[2025-04-25 15:53:08,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.94 | bwd: 5742.62 | bwd_inner: 5687.45 | bwd_allreduce: 55.13 | step: 18.48

  8%|▊         | 3286/41250 [7:55:33<91:38:47,  8.69s/it]
                                                         
{'loss': 0.085, 'grad_norm': 0.9358864426612854, 'learning_rate': 3.9741986084280884e-05, 'epoch': 0.8}

  8%|▊         | 3286/41250 [7:55:33<91:38:47,  8.69s/it][2025-04-25 15:53:17,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:53:17,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.46 | bwd_microstep: 5765.84 | bwd_inner_microstep: 5671.45 | bwd_allreduce_microstep: 94.34 | step_microstep: 18.93
[2025-04-25 15:53:17,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.46 | bwd: 5765.85 | bwd_inner: 5671.45 | bwd_allreduce: 94.36 | step: 18.93

  8%|▊         | 3287/41250 [7:55:42<91:37:25,  8.69s/it]
                                                         
{'loss': 0.3133, 'grad_norm': 2.5182535648345947, 'learning_rate': 3.974173459987448e-05, 'epoch': 0.8}

  8%|▊         | 3287/41250 [7:55:42<91:37:25,  8.69s/it][2025-04-25 15:53:25,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:53:25,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.37 | bwd_microstep: 5718.17 | bwd_inner_microstep: 5652.60 | bwd_allreduce_microstep: 65.52 | step_microstep: 18.83
[2025-04-25 15:53:25,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.37 | bwd: 5718.18 | bwd_inner: 5652.60 | bwd_allreduce: 65.54 | step: 18.83

  8%|▊         | 3288/41250 [7:55:51<91:25:25,  8.67s/it]
                                                         
{'loss': 0.2699, 'grad_norm': 1.5605463981628418, 'learning_rate': 3.974148299376417e-05, 'epoch': 0.8}

  8%|▊         | 3288/41250 [7:55:51<91:25:25,  8.67s/it][2025-04-25 15:53:34,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:53:34,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.92 | bwd_microstep: 5735.79 | bwd_inner_microstep: 5722.96 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.69
[2025-04-25 15:53:34,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.92 | bwd: 5735.81 | bwd_inner: 5722.96 | bwd_allreduce: 12.81 | step: 18.70

  8%|▊         | 3289/41250 [7:55:59<91:26:03,  8.67s/it]
                                                         
{'loss': 0.2105, 'grad_norm': 3.6938719749450684, 'learning_rate': 3.9741231265951496e-05, 'epoch': 0.8}

  8%|▊         | 3289/41250 [7:55:59<91:26:03,  8.67s/it][2025-04-25 15:53:43,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 15:53:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.77 | bwd_microstep: 5712.51 | bwd_inner_microstep: 5700.04 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.81
[2025-04-25 15:53:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.77 | bwd: 5712.52 | bwd_inner: 5700.04 | bwd_allreduce: 12.44 | step: 18.81

  8%|▊         | 3290/41250 [7:56:08<91:21:06,  8.66s/it]
                                                         
{'loss': 0.1298, 'grad_norm': 0.842201828956604, 'learning_rate': 3.9740979416438016e-05, 'epoch': 0.8}

  8%|▊         | 3290/41250 [7:56:08<91:21:06,  8.66s/it][2025-04-25 15:53:51,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.14
[2025-04-25 15:53:51,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.01 | bwd_microstep: 5713.38 | bwd_inner_microstep: 5700.27 | bwd_allreduce_microstep: 13.06 | step_microstep: 19.39
[2025-04-25 15:53:51,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.01 | bwd: 5713.39 | bwd_inner: 5700.27 | bwd_allreduce: 13.08 | step: 19.40

  8%|▊         | 3291/41250 [7:56:17<91:18:33,  8.66s/it]
                                                         
{'loss': 0.346, 'grad_norm': 3.4601898193359375, 'learning_rate': 3.974072744522528e-05, 'epoch': 0.8}

  8%|▊         | 3291/41250 [7:56:17<91:18:33,  8.66s/it][2025-04-25 15:54:00,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 15:54:00,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5692.07 | bwd_inner_microstep: 5660.74 | bwd_allreduce_microstep: 31.28 | step_microstep: 18.43
[2025-04-25 15:54:00,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.70 | bwd: 5692.08 | bwd_inner: 5660.74 | bwd_allreduce: 31.29 | step: 18.43

  8%|▊         | 3292/41250 [7:56:25<91:08:19,  8.64s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 1.6737514734268188, 'learning_rate': 3.9740475352314845e-05, 'epoch': 0.8}

  8%|▊         | 3292/41250 [7:56:25<91:08:19,  8.64s/it][2025-04-25 15:54:09,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 15:54:09,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.51 | bwd_microstep: 5760.69 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 61.46 | step_microstep: 18.80
[2025-04-25 15:54:09,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.51 | bwd: 5760.70 | bwd_inner: 5699.18 | bwd_allreduce: 61.48 | step: 18.80

  8%|▊         | 3293/41250 [7:56:34<91:18:36,  8.66s/it]
                                                         
{'loss': 0.0966, 'grad_norm': 1.3791662454605103, 'learning_rate': 3.9740223137708264e-05, 'epoch': 0.8}

  8%|▊         | 3293/41250 [7:56:34<91:18:36,  8.66s/it][2025-04-25 15:54:17,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:54:17,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.57 | bwd_microstep: 5763.30 | bwd_inner_microstep: 5665.38 | bwd_allreduce_microstep: 97.87 | step_microstep: 18.62
[2025-04-25 15:54:17,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.57 | bwd: 5763.31 | bwd_inner: 5665.38 | bwd_allreduce: 97.89 | step: 18.63

  8%|▊         | 3294/41250 [7:56:43<91:21:18,  8.66s/it]
                                                         
{'loss': 0.1944, 'grad_norm': 2.471994400024414, 'learning_rate': 3.9739970801407084e-05, 'epoch': 0.8}

  8%|▊         | 3294/41250 [7:56:43<91:21:18,  8.66s/it][2025-04-25 15:54:26,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 15:54:26,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.02 | bwd_microstep: 5726.81 | bwd_inner_microstep: 5691.07 | bwd_allreduce_microstep: 35.70 | step_microstep: 18.58
[2025-04-25 15:54:26,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.02 | bwd: 5726.82 | bwd_inner: 5691.07 | bwd_allreduce: 35.72 | step: 18.58

  8%|▊         | 3295/41250 [7:56:51<91:22:11,  8.67s/it]
                                                         
{'loss': 0.2182, 'grad_norm': 1.1533052921295166, 'learning_rate': 3.973971834341287e-05, 'epoch': 0.8}

  8%|▊         | 3295/41250 [7:56:51<91:22:11,  8.67s/it][2025-04-25 15:54:35,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 15:54:35,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.61 | bwd_microstep: 5718.87 | bwd_inner_microstep: 5706.19 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.22
[2025-04-25 15:54:35,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.61 | bwd: 5718.88 | bwd_inner: 5706.19 | bwd_allreduce: 12.65 | step: 19.23

  8%|▊         | 3296/41250 [7:57:00<91:21:01,  8.66s/it]
                                                         
{'loss': 0.225, 'grad_norm': 1.4106745719909668, 'learning_rate': 3.973946576372717e-05, 'epoch': 0.8}

  8%|▊         | 3296/41250 [7:57:00<91:21:01,  8.66s/it][2025-04-25 15:54:43,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 15:54:43,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5774.51 | bwd_inner_microstep: 5701.46 | bwd_allreduce_microstep: 73.01 | step_microstep: 18.71
[2025-04-25 15:54:43,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5774.53 | bwd_inner: 5701.46 | bwd_allreduce: 73.03 | step: 18.71

  8%|▊         | 3297/41250 [7:57:09<91:29:42,  8.68s/it]
                                                         
{'loss': 0.1121, 'grad_norm': 2.121515989303589, 'learning_rate': 3.973921306235155e-05, 'epoch': 0.8}

  8%|▊         | 3297/41250 [7:57:09<91:29:42,  8.68s/it][2025-04-25 15:54:52,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:54:52,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.92 | bwd_microstep: 5883.41 | bwd_inner_microstep: 5653.93 | bwd_allreduce_microstep: 229.43 | step_microstep: 18.88
[2025-04-25 15:54:52,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.92 | bwd: 5883.42 | bwd_inner: 5653.93 | bwd_allreduce: 229.45 | step: 18.88

  8%|▊         | 3298/41250 [7:57:17<91:53:12,  8.72s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.110776424407959, 'learning_rate': 3.9738960239287564e-05, 'epoch': 0.8}

  8%|▊         | 3298/41250 [7:57:17<91:53:12,  8.72s/it][2025-04-25 15:55:01,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 15:55:01,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.86 | bwd_microstep: 5786.21 | bwd_inner_microstep: 5645.61 | bwd_allreduce_microstep: 140.55 | step_microstep: 18.89
[2025-04-25 15:55:01,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.86 | bwd: 5786.22 | bwd_inner: 5645.61 | bwd_allreduce: 140.57 | step: 18.89

  8%|▊         | 3299/41250 [7:57:26<91:48:45,  8.71s/it]
                                                         
{'loss': 0.1771, 'grad_norm': 2.021658420562744, 'learning_rate': 3.973870729453677e-05, 'epoch': 0.8}

  8%|▊         | 3299/41250 [7:57:26<91:48:45,  8.71s/it][2025-04-25 15:55:09,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:55:09,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.35 | bwd_microstep: 5750.81 | bwd_inner_microstep: 5680.02 | bwd_allreduce_microstep: 70.74 | step_microstep: 19.04
[2025-04-25 15:55:09,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.35 | bwd: 5750.82 | bwd_inner: 5680.02 | bwd_allreduce: 70.76 | step: 19.04

  8%|▊         | 3300/41250 [7:57:35<91:42:03,  8.70s/it]
                                                         
{'loss': 0.0911, 'grad_norm': 0.7571878433227539, 'learning_rate': 3.973845422810073e-05, 'epoch': 0.8}

  8%|▊         | 3300/41250 [7:57:35<91:42:03,  8.70s/it][2025-04-25 15:55:18,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 15:55:18,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5700.50 | bwd_inner_microstep: 5687.31 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.23
[2025-04-25 15:55:18,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5700.51 | bwd_inner: 5687.31 | bwd_allreduce: 13.15 | step: 19.24

  8%|▊         | 3301/41250 [7:57:43<91:28:54,  8.68s/it]
                                                         
{'loss': 0.1431, 'grad_norm': 0.8714591860771179, 'learning_rate': 3.9738201039981e-05, 'epoch': 0.8}

  8%|▊         | 3301/41250 [7:57:43<91:28:54,  8.68s/it][2025-04-25 15:55:27,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.09 | optimizer_step: 1.13
[2025-04-25 15:55:27,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.90 | bwd_microstep: 5773.17 | bwd_inner_microstep: 5653.83 | bwd_allreduce_microstep: 119.29 | step_microstep: 19.52
[2025-04-25 15:55:27,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.90 | bwd: 5773.18 | bwd_inner: 5653.83 | bwd_allreduce: 119.31 | step: 19.52

  8%|▊         | 3302/41250 [7:57:52<91:30:16,  8.68s/it]
                                                         
{'loss': 0.1118, 'grad_norm': 0.947464644908905, 'learning_rate': 3.973794773017914e-05, 'epoch': 0.8}

  8%|▊         | 3302/41250 [7:57:52<91:30:16,  8.68s/it][2025-04-25 15:55:35,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-25 15:55:35,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.46 | bwd_microstep: 5707.62 | bwd_inner_microstep: 5651.13 | bwd_allreduce_microstep: 56.44 | step_microstep: 18.54
[2025-04-25 15:55:35,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.46 | bwd: 5707.63 | bwd_inner: 5651.13 | bwd_allreduce: 56.46 | step: 18.54

  8%|▊         | 3303/41250 [7:58:01<91:17:26,  8.66s/it]
                                                         
{'loss': 0.2804, 'grad_norm': 1.9207435846328735, 'learning_rate': 3.973769429869672e-05, 'epoch': 0.8}

  8%|▊         | 3303/41250 [7:58:01<91:17:26,  8.66s/it][2025-04-25 15:55:44,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 15:55:44,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.90 | bwd_microstep: 5696.34 | bwd_inner_microstep: 5644.26 | bwd_allreduce_microstep: 52.03 | step_microstep: 18.88
[2025-04-25 15:55:44,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.90 | bwd: 5696.35 | bwd_inner: 5644.26 | bwd_allreduce: 52.05 | step: 18.88

  8%|▊         | 3304/41250 [7:58:09<91:06:39,  8.64s/it]
                                                         
{'loss': 0.1671, 'grad_norm': 0.9244900345802307, 'learning_rate': 3.973744074553529e-05, 'epoch': 0.8}

  8%|▊         | 3304/41250 [7:58:09<91:06:39,  8.64s/it][2025-04-25 15:55:53,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.95 | optimizer_step: 1.04
[2025-04-25 15:55:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5736.34 | bwd_inner_microstep: 5703.63 | bwd_allreduce_microstep: 32.66 | step_microstep: 18.40
[2025-04-25 15:55:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5736.35 | bwd_inner: 5703.63 | bwd_allreduce: 32.68 | step: 18.40

  8%|▊         | 3305/41250 [7:58:18<91:10:13,  8.65s/it]
                                                         
{'loss': 0.0304, 'grad_norm': 0.28662148118019104, 'learning_rate': 3.9737187070696433e-05, 'epoch': 0.8}

  8%|▊         | 3305/41250 [7:58:18<91:10:13,  8.65s/it][2025-04-25 15:56:01,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 15:56:01,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.21 | bwd_microstep: 5692.88 | bwd_inner_microstep: 5680.17 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.49
[2025-04-25 15:56:01,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.21 | bwd: 5692.89 | bwd_inner: 5680.17 | bwd_allreduce: 12.69 | step: 18.50

  8%|▊         | 3306/41250 [7:58:27<91:04:22,  8.64s/it]
                                                         
{'loss': 0.197, 'grad_norm': 1.7618564367294312, 'learning_rate': 3.9736933274181684e-05, 'epoch': 0.8}

  8%|▊         | 3306/41250 [7:58:27<91:04:22,  8.64s/it][2025-04-25 15:56:10,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:56:10,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.47 | bwd_microstep: 5686.66 | bwd_inner_microstep: 5674.00 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.44
[2025-04-25 15:56:10,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.48 | bwd: 5686.67 | bwd_inner: 5674.00 | bwd_allreduce: 12.63 | step: 18.45

  8%|▊         | 3307/41250 [7:58:35<90:58:20,  8.63s/it]
                                                         
{'loss': 0.2319, 'grad_norm': 1.610553503036499, 'learning_rate': 3.973667935599263e-05, 'epoch': 0.8}

  8%|▊         | 3307/41250 [7:58:35<90:58:20,  8.63s/it][2025-04-25 15:56:19,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:56:19,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.51 | bwd_microstep: 5725.50 | bwd_inner_microstep: 5705.94 | bwd_allreduce_microstep: 19.52 | step_microstep: 18.62
[2025-04-25 15:56:19,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.51 | bwd: 5725.51 | bwd_inner: 5705.94 | bwd_allreduce: 19.54 | step: 18.63

  8%|▊         | 3308/41250 [7:58:44<91:03:47,  8.64s/it]
                                                         
{'loss': 0.0241, 'grad_norm': 0.24696700274944305, 'learning_rate': 3.973642531613083e-05, 'epoch': 0.8}

  8%|▊         | 3308/41250 [7:58:44<91:03:47,  8.64s/it][2025-04-25 15:56:27,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 15:56:27,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.34 | bwd_microstep: 5765.86 | bwd_inner_microstep: 5653.22 | bwd_allreduce_microstep: 112.60 | step_microstep: 18.75
[2025-04-25 15:56:27,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.34 | bwd: 5765.87 | bwd_inner: 5653.21 | bwd_allreduce: 112.61 | step: 18.75

  8%|▊         | 3309/41250 [7:58:53<91:10:41,  8.65s/it]
                                                         
{'loss': 0.1628, 'grad_norm': 1.7563551664352417, 'learning_rate': 3.973617115459785e-05, 'epoch': 0.8}

  8%|▊         | 3309/41250 [7:58:53<91:10:41,  8.65s/it][2025-04-25 15:56:36,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 15:56:36,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.51 | bwd_microstep: 5777.22 | bwd_inner_microstep: 5638.35 | bwd_allreduce_microstep: 138.83 | step_microstep: 18.57
[2025-04-25 15:56:36,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.51 | bwd: 5777.23 | bwd_inner: 5638.35 | bwd_allreduce: 138.85 | step: 18.57

  8%|▊         | 3310/41250 [7:59:01<91:15:56,  8.66s/it]
                                                         
{'loss': 0.2135, 'grad_norm': 1.0926618576049805, 'learning_rate': 3.973591687139526e-05, 'epoch': 0.8}

  8%|▊         | 3310/41250 [7:59:01<91:15:56,  8.66s/it][2025-04-25 15:56:45,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 15:56:45,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.13 | bwd_microstep: 5719.08 | bwd_inner_microstep: 5706.17 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.18
[2025-04-25 15:56:45,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.13 | bwd: 5719.10 | bwd_inner: 5706.17 | bwd_allreduce: 12.89 | step: 19.18

  8%|▊         | 3311/41250 [7:59:10<91:15:28,  8.66s/it]
                                                         
{'loss': 0.153, 'grad_norm': 1.6119977235794067, 'learning_rate': 3.973566246652462e-05, 'epoch': 0.8}

  8%|▊         | 3311/41250 [7:59:10<91:15:28,  8.66s/it][2025-04-25 15:56:53,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 15:56:53,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.06 | bwd_microstep: 5749.67 | bwd_inner_microstep: 5631.35 | bwd_allreduce_microstep: 118.27 | step_microstep: 19.09
[2025-04-25 15:56:53,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.06 | bwd: 5749.68 | bwd_inner: 5631.35 | bwd_allreduce: 118.29 | step: 19.09

  8%|▊         | 3312/41250 [7:59:19<91:15:47,  8.66s/it]
                                                         
{'loss': 0.091, 'grad_norm': 0.9539212584495544, 'learning_rate': 3.9735407939987507e-05, 'epoch': 0.8}

  8%|▊         | 3312/41250 [7:59:19<91:15:47,  8.66s/it][2025-04-25 15:57:02,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-25 15:57:02,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.17 | bwd_microstep: 5734.85 | bwd_inner_microstep: 5685.94 | bwd_allreduce_microstep: 48.85 | step_microstep: 19.08
[2025-04-25 15:57:02,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.17 | bwd: 5734.86 | bwd_inner: 5685.94 | bwd_allreduce: 48.88 | step: 19.09

  8%|▊         | 3313/41250 [7:59:27<91:15:29,  8.66s/it]
                                                         
{'loss': 0.367, 'grad_norm': 2.6093671321868896, 'learning_rate': 3.973515329178548e-05, 'epoch': 0.8}

  8%|▊         | 3313/41250 [7:59:27<91:15:29,  8.66s/it][2025-04-25 15:57:11,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:57:11,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.93 | bwd_microstep: 5707.44 | bwd_inner_microstep: 5694.75 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.60
[2025-04-25 15:57:11,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.93 | bwd: 5707.45 | bwd_inner: 5694.75 | bwd_allreduce: 12.66 | step: 18.60

  8%|▊         | 3314/41250 [7:59:36<91:12:17,  8.66s/it]
                                                         
{'loss': 0.0945, 'grad_norm': 1.2044928073883057, 'learning_rate': 3.973489852192012e-05, 'epoch': 0.8}

  8%|▊         | 3314/41250 [7:59:36<91:12:17,  8.66s/it][2025-04-25 15:57:19,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:57:19,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5756.71 | bwd_inner_microstep: 5649.55 | bwd_allreduce_microstep: 107.12 | step_microstep: 18.58
[2025-04-25 15:57:19,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5756.73 | bwd_inner: 5649.55 | bwd_allreduce: 107.14 | step: 18.58

  8%|▊         | 3315/41250 [7:59:45<91:14:04,  8.66s/it]
                                                         
{'loss': 0.14, 'grad_norm': 0.9474058747291565, 'learning_rate': 3.973464363039299e-05, 'epoch': 0.8}

  8%|▊         | 3315/41250 [7:59:45<91:14:04,  8.66s/it][2025-04-25 15:57:28,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 15:57:28,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.28 | bwd_microstep: 5698.32 | bwd_inner_microstep: 5685.73 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.99
[2025-04-25 15:57:28,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.28 | bwd: 5698.34 | bwd_inner: 5685.73 | bwd_allreduce: 12.56 | step: 18.99

  8%|▊         | 3316/41250 [7:59:53<91:08:56,  8.65s/it]
                                                         
{'loss': 0.2989, 'grad_norm': 4.626391410827637, 'learning_rate': 3.973438861720566e-05, 'epoch': 0.8}

  8%|▊         | 3316/41250 [7:59:53<91:08:56,  8.65s/it][2025-04-25 15:57:37,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 15:57:37,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.44 | bwd_microstep: 5720.68 | bwd_inner_microstep: 5688.90 | bwd_allreduce_microstep: 31.74 | step_microstep: 18.93
[2025-04-25 15:57:37,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.44 | bwd: 5720.69 | bwd_inner: 5688.90 | bwd_allreduce: 31.75 | step: 18.93

  8%|▊         | 3317/41250 [8:00:02<91:08:05,  8.65s/it]
                                                         
{'loss': 0.181, 'grad_norm': 1.6389590501785278, 'learning_rate': 3.973413348235971e-05, 'epoch': 0.8}

  8%|▊         | 3317/41250 [8:00:02<91:08:05,  8.65s/it][2025-04-25 15:57:45,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 15:57:45,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.90 | bwd_microstep: 5774.98 | bwd_inner_microstep: 5646.10 | bwd_allreduce_microstep: 128.83 | step_microstep: 19.13
[2025-04-25 15:57:45,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.90 | bwd: 5774.99 | bwd_inner: 5646.10 | bwd_allreduce: 128.85 | step: 19.13

  8%|▊         | 3318/41250 [8:00:11<91:15:04,  8.66s/it]
                                                         
{'loss': 0.1911, 'grad_norm': 1.8030896186828613, 'learning_rate': 3.973387822585671e-05, 'epoch': 0.8}

  8%|▊         | 3318/41250 [8:00:11<91:15:04,  8.66s/it][2025-04-25 15:57:54,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:57:54,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.35 | bwd_microstep: 5721.00 | bwd_inner_microstep: 5708.39 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.82
[2025-04-25 15:57:54,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.35 | bwd: 5721.02 | bwd_inner: 5708.39 | bwd_allreduce: 12.58 | step: 18.82

  8%|▊         | 3319/41250 [8:00:19<91:14:17,  8.66s/it]
                                                         
{'loss': 0.3047, 'grad_norm': 2.2319259643554688, 'learning_rate': 3.973362284769823e-05, 'epoch': 0.8}

  8%|▊         | 3319/41250 [8:00:19<91:14:17,  8.66s/it][2025-04-25 15:58:02,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 15:58:02,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.57 | bwd_microstep: 5710.82 | bwd_inner_microstep: 5657.98 | bwd_allreduce_microstep: 52.80 | step_microstep: 18.76
[2025-04-25 15:58:02,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.57 | bwd: 5710.84 | bwd_inner: 5657.98 | bwd_allreduce: 52.82 | step: 18.77

  8%|▊         | 3320/41250 [8:00:28<91:07:19,  8.65s/it]
                                                         
{'loss': 0.0307, 'grad_norm': 0.4243515431880951, 'learning_rate': 3.973336734788585e-05, 'epoch': 0.8}

  8%|▊         | 3320/41250 [8:00:28<91:07:19,  8.65s/it][2025-04-25 15:58:11,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 15:58:11,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.57 | bwd_microstep: 5905.20 | bwd_inner_microstep: 5644.77 | bwd_allreduce_microstep: 260.38 | step_microstep: 18.77
[2025-04-25 15:58:11,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.57 | bwd: 5905.22 | bwd_inner: 5644.77 | bwd_allreduce: 260.40 | step: 18.77

  8%|▊         | 3321/41250 [8:00:37<91:38:57,  8.70s/it]
                                                         
{'loss': 0.2798, 'grad_norm': 1.8658547401428223, 'learning_rate': 3.973311172642113e-05, 'epoch': 0.81}

  8%|▊         | 3321/41250 [8:00:37<91:38:57,  8.70s/it][2025-04-25 15:58:20,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.30 | optimizer_step: 1.05
[2025-04-25 15:58:20,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.43 | bwd_microstep: 5759.54 | bwd_inner_microstep: 5684.53 | bwd_allreduce_microstep: 74.94 | step_microstep: 20.29
[2025-04-25 15:58:20,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.43 | bwd: 5759.55 | bwd_inner: 5684.53 | bwd_allreduce: 74.97 | step: 20.29

  8%|▊         | 3322/41250 [8:00:45<91:36:59,  8.70s/it]
                                                         
{'loss': 0.1025, 'grad_norm': 0.8980226516723633, 'learning_rate': 3.973285598330567e-05, 'epoch': 0.81}

  8%|▊         | 3322/41250 [8:00:45<91:36:59,  8.70s/it][2025-04-25 15:58:29,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:58:29,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5716.66 | bwd_inner_microstep: 5703.85 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.80
[2025-04-25 15:58:29,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5716.68 | bwd_inner: 5703.85 | bwd_allreduce: 12.78 | step: 18.80

  8%|▊         | 3323/41250 [8:00:54<91:27:54,  8.68s/it]
                                                         
{'loss': 0.3214, 'grad_norm': 2.4904139041900635, 'learning_rate': 3.973260011854103e-05, 'epoch': 0.81}

  8%|▊         | 3323/41250 [8:00:54<91:27:54,  8.68s/it][2025-04-25 15:58:37,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 15:58:37,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.96 | bwd_microstep: 5778.23 | bwd_inner_microstep: 5652.85 | bwd_allreduce_microstep: 125.33 | step_microstep: 18.86
[2025-04-25 15:58:37,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.96 | bwd: 5778.24 | bwd_inner: 5652.85 | bwd_allreduce: 125.35 | step: 18.86

  8%|▊         | 3324/41250 [8:01:03<91:31:20,  8.69s/it]
                                                         
{'loss': 0.0686, 'grad_norm': 1.0423449277877808, 'learning_rate': 3.973234413212879e-05, 'epoch': 0.81}

  8%|▊         | 3324/41250 [8:01:03<91:31:20,  8.69s/it][2025-04-25 15:58:46,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 15:58:46,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.41 | bwd_microstep: 5751.10 | bwd_inner_microstep: 5659.81 | bwd_allreduce_microstep: 91.24 | step_microstep: 18.45
[2025-04-25 15:58:46,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.41 | bwd: 5751.11 | bwd_inner: 5659.81 | bwd_allreduce: 91.26 | step: 18.45

  8%|▊         | 3325/41250 [8:01:11<91:26:04,  8.68s/it]
                                                         
{'loss': 0.0691, 'grad_norm': 0.9849536418914795, 'learning_rate': 3.973208802407053e-05, 'epoch': 0.81}

  8%|▊         | 3325/41250 [8:01:11<91:26:04,  8.68s/it][2025-04-25 15:58:55,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.13 | optimizer_step: 1.03
[2025-04-25 15:58:55,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.96 | bwd_microstep: 5777.21 | bwd_inner_microstep: 5651.54 | bwd_allreduce_microstep: 125.61 | step_microstep: 19.95
[2025-04-25 15:58:55,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.96 | bwd: 5777.23 | bwd_inner: 5651.54 | bwd_allreduce: 125.64 | step: 19.96

  8%|▊         | 3326/41250 [8:01:20<91:28:32,  8.68s/it]
                                                         
{'loss': 0.1918, 'grad_norm': 1.7705222368240356, 'learning_rate': 3.973183179436783e-05, 'epoch': 0.81}

  8%|▊         | 3326/41250 [8:01:20<91:28:32,  8.68s/it][2025-04-25 15:59:03,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 15:59:03,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.96 | bwd_microstep: 5774.29 | bwd_inner_microstep: 5654.71 | bwd_allreduce_microstep: 119.53 | step_microstep: 19.10
[2025-04-25 15:59:03,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.96 | bwd: 5774.31 | bwd_inner: 5654.71 | bwd_allreduce: 119.55 | step: 19.11

  8%|▊         | 3327/41250 [8:01:29<91:31:36,  8.69s/it]
                                                         
{'loss': 0.1511, 'grad_norm': 1.2935914993286133, 'learning_rate': 3.973157544302226e-05, 'epoch': 0.81}

  8%|▊         | 3327/41250 [8:01:29<91:31:36,  8.69s/it][2025-04-25 15:59:12,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.92
[2025-04-25 15:59:12,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5759.86 | bwd_inner_microstep: 5697.82 | bwd_allreduce_microstep: 61.99 | step_microstep: 18.23
[2025-04-25 15:59:12,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5759.87 | bwd_inner: 5697.82 | bwd_allreduce: 62.01 | step: 18.24

  8%|▊         | 3328/41250 [8:01:37<91:32:19,  8.69s/it]
                                                         
{'loss': 0.0994, 'grad_norm': 1.0668230056762695, 'learning_rate': 3.9731318970035426e-05, 'epoch': 0.81}

  8%|▊         | 3328/41250 [8:01:37<91:32:19,  8.69s/it][2025-04-25 15:59:21,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 15:59:21,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5805.91 | bwd_inner_microstep: 5652.64 | bwd_allreduce_microstep: 153.21 | step_microstep: 19.11
[2025-04-25 15:59:21,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5805.92 | bwd_inner: 5652.64 | bwd_allreduce: 153.24 | step: 19.11

  8%|▊         | 3329/41250 [8:01:46<91:37:13,  8.70s/it]
                                                         
{'loss': 0.0713, 'grad_norm': 0.7734088897705078, 'learning_rate': 3.973106237540887e-05, 'epoch': 0.81}

  8%|▊         | 3329/41250 [8:01:46<91:37:13,  8.70s/it][2025-04-25 15:59:29,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 15:59:29,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.09 | bwd_microstep: 5700.95 | bwd_inner_microstep: 5663.61 | bwd_allreduce_microstep: 37.29 | step_microstep: 18.73
[2025-04-25 15:59:29,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.09 | bwd: 5700.96 | bwd_inner: 5663.61 | bwd_allreduce: 37.31 | step: 18.73

  8%|▊         | 3330/41250 [8:01:55<91:22:21,  8.67s/it]
                                                         
{'loss': 0.236, 'grad_norm': 1.4587416648864746, 'learning_rate': 3.973080565914421e-05, 'epoch': 0.81}

  8%|▊         | 3330/41250 [8:01:55<91:22:21,  8.67s/it][2025-04-25 15:59:38,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 15:59:38,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.17 | bwd_microstep: 5779.78 | bwd_inner_microstep: 5766.82 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.25
[2025-04-25 15:59:38,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.17 | bwd: 5779.79 | bwd_inner: 5766.82 | bwd_allreduce: 12.93 | step: 19.26

  8%|▊         | 3331/41250 [8:02:03<91:37:04,  8.70s/it]
                                                         
{'loss': 0.3047, 'grad_norm': 1.637917160987854, 'learning_rate': 3.9730548821243016e-05, 'epoch': 0.81}

  8%|▊         | 3331/41250 [8:02:03<91:37:04,  8.70s/it][2025-04-25 15:59:47,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 15:59:47,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.46 | bwd_microstep: 5768.62 | bwd_inner_microstep: 5670.03 | bwd_allreduce_microstep: 98.54 | step_microstep: 19.20
[2025-04-25 15:59:47,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.46 | bwd: 5768.63 | bwd_inner: 5670.03 | bwd_allreduce: 98.56 | step: 19.20

  8%|▊         | 3332/41250 [8:02:12<91:36:23,  8.70s/it]
                                                         
{'loss': 0.3109, 'grad_norm': 1.912853717803955, 'learning_rate': 3.9730291861706855e-05, 'epoch': 0.81}

  8%|▊         | 3332/41250 [8:02:12<91:36:23,  8.70s/it][2025-04-25 15:59:56,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 15:59:56,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.96 | bwd_microstep: 5762.82 | bwd_inner_microstep: 5670.87 | bwd_allreduce_microstep: 91.90 | step_microstep: 18.92
[2025-04-25 15:59:56,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.96 | bwd: 5762.83 | bwd_inner: 5670.87 | bwd_allreduce: 91.92 | step: 18.92

  8%|▊         | 3333/41250 [8:02:21<91:33:28,  8.69s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 0.989580512046814, 'learning_rate': 3.973003478053734e-05, 'epoch': 0.81}

  8%|▊         | 3333/41250 [8:02:21<91:33:28,  8.69s/it][2025-04-25 16:00:04,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:00:04,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.34 | bwd_microstep: 5735.34 | bwd_inner_microstep: 5722.51 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.01
[2025-04-25 16:00:04,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.34 | bwd: 5735.35 | bwd_inner: 5722.51 | bwd_allreduce: 12.80 | step: 19.01

  8%|▊         | 3334/41250 [8:02:30<91:31:21,  8.69s/it]
                                                         
{'loss': 0.1674, 'grad_norm': 0.7675355076789856, 'learning_rate': 3.972977757773604e-05, 'epoch': 0.81}

  8%|▊         | 3334/41250 [8:02:30<91:31:21,  8.69s/it][2025-04-25 16:00:13,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:00:13,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5753.71 | bwd_inner_microstep: 5702.95 | bwd_allreduce_microstep: 50.71 | step_microstep: 18.90
[2025-04-25 16:00:13,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5753.73 | bwd_inner: 5702.95 | bwd_allreduce: 50.73 | step: 18.90

  8%|▊         | 3335/41250 [8:02:38<91:31:11,  8.69s/it]
                                                         
{'loss': 0.2325, 'grad_norm': 1.8906794786453247, 'learning_rate': 3.972952025330453e-05, 'epoch': 0.81}

  8%|▊         | 3335/41250 [8:02:38<91:31:11,  8.69s/it][2025-04-25 16:00:22,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 16:00:22,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.34 | bwd_microstep: 5711.71 | bwd_inner_microstep: 5698.90 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.21
[2025-04-25 16:00:22,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.34 | bwd: 5711.72 | bwd_inner: 5698.90 | bwd_allreduce: 12.78 | step: 19.21

  8%|▊         | 3336/41250 [8:02:47<91:21:52,  8.68s/it]
                                                         
{'loss': 0.2309, 'grad_norm': 1.8456696271896362, 'learning_rate': 3.972926280724442e-05, 'epoch': 0.81}

  8%|▊         | 3336/41250 [8:02:47<91:21:52,  8.68s/it][2025-04-25 16:00:30,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-25 16:00:30,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.30 | bwd_microstep: 5935.47 | bwd_inner_microstep: 5706.55 | bwd_allreduce_microstep: 228.88 | step_microstep: 18.60
[2025-04-25 16:00:30,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.30 | bwd: 5935.48 | bwd_inner: 5706.55 | bwd_allreduce: 228.89 | step: 18.60

  8%|▊         | 3337/41250 [8:02:56<91:58:47,  8.73s/it]
                                                         
{'loss': 0.2736, 'grad_norm': 1.8061530590057373, 'learning_rate': 3.9729005239557274e-05, 'epoch': 0.81}

  8%|▊         | 3337/41250 [8:02:56<91:58:47,  8.73s/it][2025-04-25 16:00:39,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 16:00:39,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 5703.48 | bwd_inner_microstep: 5670.61 | bwd_allreduce_microstep: 32.83 | step_microstep: 18.98
[2025-04-25 16:00:39,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 5703.50 | bwd_inner: 5670.61 | bwd_allreduce: 32.85 | step: 18.99

  8%|▊         | 3338/41250 [8:03:04<91:36:50,  8.70s/it]
                                                         
{'loss': 0.1836, 'grad_norm': 2.5025546550750732, 'learning_rate': 3.97287475502447e-05, 'epoch': 0.81}

  8%|▊         | 3338/41250 [8:03:04<91:36:50,  8.70s/it][2025-04-25 16:00:48,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:00:48,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.58 | bwd_microstep: 5732.77 | bwd_inner_microstep: 5719.91 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.64
[2025-04-25 16:00:48,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.58 | bwd: 5732.79 | bwd_inner: 5719.91 | bwd_allreduce: 12.83 | step: 18.65

  8%|▊         | 3339/41250 [8:03:13<91:32:34,  8.69s/it]
                                                         
{'loss': 0.0776, 'grad_norm': 1.501950740814209, 'learning_rate': 3.972848973930828e-05, 'epoch': 0.81}

  8%|▊         | 3339/41250 [8:03:13<91:32:34,  8.69s/it][2025-04-25 16:00:57,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.07
[2025-04-25 16:00:57,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.35 | bwd_microstep: 5933.46 | bwd_inner_microstep: 5706.00 | bwd_allreduce_microstep: 227.41 | step_microstep: 19.50
[2025-04-25 16:00:57,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.35 | bwd: 5933.48 | bwd_inner: 5706.00 | bwd_allreduce: 227.43 | step: 19.50

  8%|▊         | 3340/41250 [8:03:22<92:07:07,  8.75s/it]
                                                         
{'loss': 0.0616, 'grad_norm': 0.8241437077522278, 'learning_rate': 3.9728231806749593e-05, 'epoch': 0.81}

  8%|▊         | 3340/41250 [8:03:22<92:07:07,  8.75s/it][2025-04-25 16:01:05,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-25 16:01:05,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.34 | bwd_microstep: 5788.25 | bwd_inner_microstep: 5716.41 | bwd_allreduce_microstep: 71.80 | step_microstep: 19.33
[2025-04-25 16:01:05,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.34 | bwd: 5788.27 | bwd_inner: 5716.41 | bwd_allreduce: 71.82 | step: 19.33

  8%|▊         | 3341/41250 [8:03:31<92:02:47,  8.74s/it]
                                                         
{'loss': 0.0881, 'grad_norm': 1.014096736907959, 'learning_rate': 3.9727973752570236e-05, 'epoch': 0.81}

  8%|▊         | 3341/41250 [8:03:31<92:02:47,  8.74s/it][2025-04-25 16:01:14,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-25 16:01:14,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.78 | bwd_microstep: 5905.61 | bwd_inner_microstep: 5892.26 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.34
[2025-04-25 16:01:14,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.78 | bwd: 5905.62 | bwd_inner: 5892.26 | bwd_allreduce: 13.32 | step: 19.34

  8%|▊         | 3342/41250 [8:03:40<92:37:57,  8.80s/it]
                                                         
{'loss': 0.3168, 'grad_norm': 1.828387975692749, 'learning_rate': 3.972771557677181e-05, 'epoch': 0.81}

  8%|▊         | 3342/41250 [8:03:40<92:37:57,  8.80s/it][2025-04-25 16:01:23,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 16:01:23,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5708.50 | bwd_inner_microstep: 5695.66 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.83
[2025-04-25 16:01:23,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5708.51 | bwd_inner: 5695.66 | bwd_allreduce: 12.81 | step: 18.83

  8%|▊         | 3343/41250 [8:03:48<92:08:11,  8.75s/it]
                                                         
{'loss': 0.0647, 'grad_norm': 0.7211742401123047, 'learning_rate': 3.9727457279355895e-05, 'epoch': 0.81}

  8%|▊         | 3343/41250 [8:03:48<92:08:11,  8.75s/it][2025-04-25 16:01:32,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 16:01:32,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.12 | bwd_microstep: 5721.19 | bwd_inner_microstep: 5664.32 | bwd_allreduce_microstep: 56.82 | step_microstep: 18.95
[2025-04-25 16:01:32,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.12 | bwd: 5721.20 | bwd_inner: 5664.32 | bwd_allreduce: 56.84 | step: 18.96

  8%|▊         | 3344/41250 [8:03:57<91:46:38,  8.72s/it]
                                                         
{'loss': 0.1376, 'grad_norm': 1.5347269773483276, 'learning_rate': 3.972719886032408e-05, 'epoch': 0.81}

  8%|▊         | 3344/41250 [8:03:57<91:46:38,  8.72s/it][2025-04-25 16:01:40,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:01:40,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.19 | bwd_microstep: 5756.68 | bwd_inner_microstep: 5712.09 | bwd_allreduce_microstep: 44.55 | step_microstep: 18.68
[2025-04-25 16:01:40,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.19 | bwd: 5756.70 | bwd_inner: 5712.09 | bwd_allreduce: 44.57 | step: 18.68

  8%|▊         | 3345/41250 [8:04:06<91:42:07,  8.71s/it]
                                                         
{'loss': 0.2084, 'grad_norm': 4.097473621368408, 'learning_rate': 3.9726940319677976e-05, 'epoch': 0.81}

  8%|▊         | 3345/41250 [8:04:06<91:42:07,  8.71s/it][2025-04-25 16:01:49,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:01:49,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5728.39 | bwd_inner_microstep: 5715.41 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.04
[2025-04-25 16:01:49,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5728.40 | bwd_inner: 5715.41 | bwd_allreduce: 12.96 | step: 19.04

  8%|▊         | 3346/41250 [8:04:14<91:33:01,  8.70s/it]
                                                         
{'loss': 0.2795, 'grad_norm': 2.1014342308044434, 'learning_rate': 3.9726681657419155e-05, 'epoch': 0.81}

  8%|▊         | 3346/41250 [8:04:14<91:33:01,  8.70s/it][2025-04-25 16:01:58,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 16:01:58,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.43 | bwd_microstep: 5716.16 | bwd_inner_microstep: 5660.68 | bwd_allreduce_microstep: 55.43 | step_microstep: 19.60
[2025-04-25 16:01:58,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.43 | bwd: 5716.17 | bwd_inner: 5660.68 | bwd_allreduce: 55.44 | step: 19.60

  8%|▊         | 3347/41250 [8:04:23<91:20:40,  8.68s/it]
                                                         
{'loss': 0.0838, 'grad_norm': 0.8465540409088135, 'learning_rate': 3.9726422873549225e-05, 'epoch': 0.81}

  8%|▊         | 3347/41250 [8:04:23<91:20:40,  8.68s/it][2025-04-25 16:02:06,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 16:02:06,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.14 | bwd_microstep: 5874.94 | bwd_inner_microstep: 5650.23 | bwd_allreduce_microstep: 224.67 | step_microstep: 18.67
[2025-04-25 16:02:06,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5874.96 | bwd_inner: 5650.23 | bwd_allreduce: 224.68 | step: 18.67

  8%|▊         | 3348/41250 [8:04:32<91:41:16,  8.71s/it]
                                                         
{'loss': 0.0972, 'grad_norm': 0.7154980897903442, 'learning_rate': 3.972616396806978e-05, 'epoch': 0.81}

  8%|▊         | 3348/41250 [8:04:32<91:41:16,  8.71s/it][2025-04-25 16:02:15,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:02:15,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.32 | bwd_microstep: 5719.50 | bwd_inner_microstep: 5706.44 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.63
[2025-04-25 16:02:15,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.32 | bwd: 5719.51 | bwd_inner: 5706.44 | bwd_allreduce: 13.03 | step: 18.63

  8%|▊         | 3349/41250 [8:04:40<91:33:02,  8.70s/it]
                                                         
{'loss': 0.0303, 'grad_norm': 0.2664172053337097, 'learning_rate': 3.972590494098241e-05, 'epoch': 0.81}

  8%|▊         | 3349/41250 [8:04:40<91:33:02,  8.70s/it][2025-04-25 16:02:24,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:02:24,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.55 | bwd_microstep: 5741.40 | bwd_inner_microstep: 5700.98 | bwd_allreduce_microstep: 40.38 | step_microstep: 18.83
[2025-04-25 16:02:24,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.55 | bwd: 5741.42 | bwd_inner: 5700.98 | bwd_allreduce: 40.40 | step: 18.83

  8%|▊         | 3350/41250 [8:04:49<91:29:23,  8.69s/it]
                                                         
{'loss': 0.1013, 'grad_norm': 1.710277795791626, 'learning_rate': 3.9725645792288714e-05, 'epoch': 0.81}

  8%|▊         | 3350/41250 [8:04:49<91:29:23,  8.69s/it][2025-04-25 16:02:32,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.04 | optimizer_step: 1.26
[2025-04-25 16:02:32,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.44 | bwd_microstep: 5774.65 | bwd_inner_microstep: 5761.93 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.76
[2025-04-25 16:02:32,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.44 | bwd: 5774.67 | bwd_inner: 5761.93 | bwd_allreduce: 12.69 | step: 19.76

  8%|▊         | 3351/41250 [8:04:58<91:38:41,  8.71s/it]
                                                         
{'loss': 0.1813, 'grad_norm': 1.9829518795013428, 'learning_rate': 3.97253865219903e-05, 'epoch': 0.81}

  8%|▊         | 3351/41250 [8:04:58<91:38:41,  8.71s/it][2025-04-25 16:02:41,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:02:41,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5721.62 | bwd_inner_microstep: 5702.09 | bwd_allreduce_microstep: 19.49 | step_microstep: 18.88
[2025-04-25 16:02:41,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5721.63 | bwd_inner: 5702.09 | bwd_allreduce: 19.50 | step: 18.88

  8%|▊         | 3352/41250 [8:05:06<91:28:25,  8.69s/it]
                                                         
{'loss': 0.1728, 'grad_norm': 1.249480128288269, 'learning_rate': 3.9725127130088754e-05, 'epoch': 0.81}

  8%|▊         | 3352/41250 [8:05:06<91:28:25,  8.69s/it][2025-04-25 16:02:50,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:02:50,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.89 | bwd_microstep: 5693.83 | bwd_inner_microstep: 5652.12 | bwd_allreduce_microstep: 41.67 | step_microstep: 18.67
[2025-04-25 16:02:50,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.89 | bwd: 5693.84 | bwd_inner: 5652.12 | bwd_allreduce: 41.68 | step: 18.67

  8%|▊         | 3353/41250 [8:05:15<91:10:45,  8.66s/it]
                                                         
{'loss': 0.2896, 'grad_norm': 2.2260143756866455, 'learning_rate': 3.972486761658567e-05, 'epoch': 0.81}

  8%|▊         | 3353/41250 [8:05:15<91:10:45,  8.66s/it][2025-04-25 16:02:58,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 16:02:58,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.28 | bwd_microstep: 5709.06 | bwd_inner_microstep: 5696.29 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.74
[2025-04-25 16:02:58,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.28 | bwd: 5709.08 | bwd_inner: 5696.28 | bwd_allreduce: 12.75 | step: 18.75

  8%|▊         | 3354/41250 [8:05:24<91:07:13,  8.66s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 2.8718156814575195, 'learning_rate': 3.9724607981482665e-05, 'epoch': 0.81}

  8%|▊         | 3354/41250 [8:05:24<91:07:13,  8.66s/it][2025-04-25 16:03:07,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 16:03:07,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.91 | bwd_microstep: 5763.54 | bwd_inner_microstep: 5652.85 | bwd_allreduce_microstep: 110.64 | step_microstep: 18.47
[2025-04-25 16:03:07,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.91 | bwd: 5763.55 | bwd_inner: 5652.85 | bwd_allreduce: 110.66 | step: 18.47

  8%|▊         | 3355/41250 [8:05:32<91:11:11,  8.66s/it]
                                                         
{'loss': 0.3202, 'grad_norm': 1.7192808389663696, 'learning_rate': 3.9724348224781334e-05, 'epoch': 0.81}

  8%|▊         | 3355/41250 [8:05:32<91:11:11,  8.66s/it][2025-04-25 16:03:16,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 16:03:16,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.60 | bwd_microstep: 5695.60 | bwd_inner_microstep: 5662.05 | bwd_allreduce_microstep: 33.50 | step_microstep: 19.60
[2025-04-25 16:03:16,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.60 | bwd: 5695.62 | bwd_inner: 5662.05 | bwd_allreduce: 33.52 | step: 19.60

  8%|▊         | 3356/41250 [8:05:41<91:01:57,  8.65s/it]
                                                         
{'loss': 0.2667, 'grad_norm': 1.9706331491470337, 'learning_rate': 3.9724088346483274e-05, 'epoch': 0.81}

  8%|▊         | 3356/41250 [8:05:41<91:01:57,  8.65s/it][2025-04-25 16:03:25,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 16:03:25,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.31 | bwd_microstep: 5983.48 | bwd_inner_microstep: 5762.34 | bwd_allreduce_microstep: 221.08 | step_microstep: 19.00
[2025-04-25 16:03:25,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.31 | bwd: 5983.49 | bwd_inner: 5762.34 | bwd_allreduce: 221.10 | step: 19.01

  8%|▊         | 3357/41250 [8:05:50<91:58:37,  8.74s/it]
                                                         
{'loss': 0.1138, 'grad_norm': 1.4236994981765747, 'learning_rate': 3.972382834659009e-05, 'epoch': 0.81}

  8%|▊         | 3357/41250 [8:05:50<91:58:37,  8.74s/it][2025-04-25 16:03:33,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:03:33,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.14 | bwd_microstep: 5696.86 | bwd_inner_microstep: 5684.10 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.85
[2025-04-25 16:03:33,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.14 | bwd: 5696.88 | bwd_inner: 5684.10 | bwd_allreduce: 12.73 | step: 18.86

  8%|▊         | 3358/41250 [8:05:58<91:38:22,  8.71s/it]
                                                         
{'loss': 0.2472, 'grad_norm': 1.5146516561508179, 'learning_rate': 3.972356822510338e-05, 'epoch': 0.81}

  8%|▊         | 3358/41250 [8:05:58<91:38:22,  8.71s/it][2025-04-25 16:03:42,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 16:03:42,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.53 | bwd_microstep: 5784.96 | bwd_inner_microstep: 5772.09 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.76
[2025-04-25 16:03:42,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.53 | bwd: 5784.98 | bwd_inner: 5772.09 | bwd_allreduce: 12.84 | step: 19.76

  8%|▊         | 3359/41250 [8:06:07<91:46:37,  8.72s/it]
                                                         
{'loss': 0.3797, 'grad_norm': 1.9079856872558594, 'learning_rate': 3.9723307982024756e-05, 'epoch': 0.81}

  8%|▊         | 3359/41250 [8:06:07<91:46:37,  8.72s/it][2025-04-25 16:03:51,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 16:03:51,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.07 | bwd_microstep: 5733.38 | bwd_inner_microstep: 5682.77 | bwd_allreduce_microstep: 50.57 | step_microstep: 19.01
[2025-04-25 16:03:51,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.07 | bwd: 5733.40 | bwd_inner: 5682.77 | bwd_allreduce: 50.59 | step: 19.01

  8%|▊         | 3360/41250 [8:06:16<91:35:38,  8.70s/it]
                                                         
{'loss': 0.2517, 'grad_norm': 1.9576761722564697, 'learning_rate': 3.972304761735581e-05, 'epoch': 0.81}

  8%|▊         | 3360/41250 [8:06:16<91:35:38,  8.70s/it][2025-04-25 16:03:59,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-25 16:03:59,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.76 | bwd_microstep: 5692.87 | bwd_inner_microstep: 5680.09 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.17
[2025-04-25 16:03:59,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.76 | bwd: 5692.89 | bwd_inner: 5680.08 | bwd_allreduce: 12.76 | step: 19.17

  8%|▊         | 3361/41250 [8:06:25<91:19:35,  8.68s/it]
                                                         
{'loss': 0.1005, 'grad_norm': 1.4680242538452148, 'learning_rate': 3.972278713109816e-05, 'epoch': 0.81}

  8%|▊         | 3361/41250 [8:06:25<91:19:35,  8.68s/it][2025-04-25 16:04:08,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 16:04:08,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.07 | bwd_microstep: 5755.17 | bwd_inner_microstep: 5690.51 | bwd_allreduce_microstep: 64.62 | step_microstep: 18.80
[2025-04-25 16:04:08,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.07 | bwd: 5755.19 | bwd_inner: 5690.51 | bwd_allreduce: 64.64 | step: 18.80

  8%|▊         | 3362/41250 [8:06:33<91:20:11,  8.68s/it]
                                                         
{'loss': 0.1632, 'grad_norm': 1.4602984189987183, 'learning_rate': 3.9722526523253406e-05, 'epoch': 0.82}

  8%|▊         | 3362/41250 [8:06:33<91:20:11,  8.68s/it][2025-04-25 16:04:17,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:04:17,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.37 | bwd_microstep: 5713.59 | bwd_inner_microstep: 5700.82 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.74
[2025-04-25 16:04:17,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.37 | bwd: 5713.61 | bwd_inner: 5700.82 | bwd_allreduce: 12.74 | step: 18.74

  8%|▊         | 3363/41250 [8:06:42<91:14:59,  8.67s/it]
                                                         
{'loss': 0.2234, 'grad_norm': 1.2513720989227295, 'learning_rate': 3.972226579382316e-05, 'epoch': 0.82}

  8%|▊         | 3363/41250 [8:06:42<91:14:59,  8.67s/it][2025-04-25 16:04:25,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:04:25,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.96 | bwd_microstep: 5739.28 | bwd_inner_microstep: 5646.93 | bwd_allreduce_microstep: 92.30 | step_microstep: 18.83
[2025-04-25 16:04:25,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.96 | bwd: 5739.29 | bwd_inner: 5646.93 | bwd_allreduce: 92.31 | step: 18.84

  8%|▊         | 3364/41250 [8:06:50<91:09:33,  8.66s/it]
                                                         
{'loss': 0.2418, 'grad_norm': 1.6996039152145386, 'learning_rate': 3.972200494280902e-05, 'epoch': 0.82}

  8%|▊         | 3364/41250 [8:06:50<91:09:33,  8.66s/it][2025-04-25 16:04:34,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 16:04:34,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.61 | bwd_microstep: 5703.09 | bwd_inner_microstep: 5646.76 | bwd_allreduce_microstep: 56.29 | step_microstep: 18.88
[2025-04-25 16:04:34,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.61 | bwd: 5703.11 | bwd_inner: 5646.76 | bwd_allreduce: 56.31 | step: 18.89

  8%|▊         | 3365/41250 [8:06:59<90:59:21,  8.65s/it]
                                                         
{'loss': 0.1985, 'grad_norm': 2.185450315475464, 'learning_rate': 3.97217439702126e-05, 'epoch': 0.82}

  8%|▊         | 3365/41250 [8:06:59<90:59:21,  8.65s/it][2025-04-25 16:04:42,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-25 16:04:42,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.17 | bwd_microstep: 5685.40 | bwd_inner_microstep: 5651.67 | bwd_allreduce_microstep: 33.69 | step_microstep: 19.22
[2025-04-25 16:04:42,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.17 | bwd: 5685.42 | bwd_inner: 5651.67 | bwd_allreduce: 33.71 | step: 19.23

  8%|▊         | 3366/41250 [8:07:08<90:49:08,  8.63s/it]
                                                         
{'loss': 0.0969, 'grad_norm': 0.704279363155365, 'learning_rate': 3.97214828760355e-05, 'epoch': 0.82}

  8%|▊         | 3366/41250 [8:07:08<90:49:08,  8.63s/it][2025-04-25 16:04:51,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:04:51,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.42 | bwd_microstep: 5747.67 | bwd_inner_microstep: 5651.49 | bwd_allreduce_microstep: 96.14 | step_microstep: 18.82
[2025-04-25 16:04:51,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.42 | bwd: 5747.69 | bwd_inner: 5651.48 | bwd_allreduce: 96.16 | step: 18.82

  8%|▊         | 3367/41250 [8:07:16<90:55:49,  8.64s/it]
                                                         
{'loss': 0.096, 'grad_norm': 1.4895802736282349, 'learning_rate': 3.972122166027934e-05, 'epoch': 0.82}

  8%|▊         | 3367/41250 [8:07:16<90:55:49,  8.64s/it][2025-04-25 16:05:00,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 16:05:00,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.78 | bwd_microstep: 5678.38 | bwd_inner_microstep: 5647.93 | bwd_allreduce_microstep: 30.41 | step_microstep: 19.30
[2025-04-25 16:05:00,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.78 | bwd: 5678.39 | bwd_inner: 5647.93 | bwd_allreduce: 30.42 | step: 19.30

  8%|▊         | 3368/41250 [8:07:25<90:46:32,  8.63s/it]
                                                         
{'loss': 0.1186, 'grad_norm': 0.8106241822242737, 'learning_rate': 3.972096032294573e-05, 'epoch': 0.82}

  8%|▊         | 3368/41250 [8:07:25<90:46:32,  8.63s/it][2025-04-25 16:05:08,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:05:08,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5706.29 | bwd_inner_microstep: 5693.39 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.72
[2025-04-25 16:05:08,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5706.30 | bwd_inner: 5693.39 | bwd_allreduce: 12.86 | step: 18.73

  8%|▊         | 3369/41250 [8:07:34<90:48:51,  8.63s/it]
                                                         
{'loss': 0.3081, 'grad_norm': 1.5718637704849243, 'learning_rate': 3.972069886403628e-05, 'epoch': 0.82}

  8%|▊         | 3369/41250 [8:07:34<90:48:51,  8.63s/it][2025-04-25 16:05:17,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.18 | optimizer_step: 1.18
[2025-04-25 16:05:17,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 5726.53 | bwd_inner_microstep: 5650.83 | bwd_allreduce_microstep: 75.64 | step_microstep: 20.39
[2025-04-25 16:05:17,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.28 | bwd: 5726.54 | bwd_inner: 5650.83 | bwd_allreduce: 75.67 | step: 20.39

  8%|▊         | 3370/41250 [8:07:42<90:50:36,  8.63s/it]
                                                         
{'loss': 0.0623, 'grad_norm': 0.6302022933959961, 'learning_rate': 3.97204372835526e-05, 'epoch': 0.82}

  8%|▊         | 3370/41250 [8:07:42<90:50:36,  8.63s/it][2025-04-25 16:05:26,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 16:05:26,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.23 | bwd_microstep: 5699.75 | bwd_inner_microstep: 5653.03 | bwd_allreduce_microstep: 46.68 | step_microstep: 19.08
[2025-04-25 16:05:26,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.23 | bwd: 5699.77 | bwd_inner: 5653.03 | bwd_allreduce: 46.70 | step: 19.08

  8%|▊         | 3371/41250 [8:07:51<90:45:48,  8.63s/it]
                                                         
{'loss': 0.1697, 'grad_norm': 1.1973576545715332, 'learning_rate': 3.97201755814963e-05, 'epoch': 0.82}

  8%|▊         | 3371/41250 [8:07:51<90:45:48,  8.63s/it][2025-04-25 16:05:34,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.28 | optimizer_step: 0.90
[2025-04-25 16:05:34,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.71 | bwd_microstep: 5727.94 | bwd_inner_microstep: 5641.86 | bwd_allreduce_microstep: 86.03 | step_microstep: 19.73
[2025-04-25 16:05:34,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.71 | bwd: 5727.96 | bwd_inner: 5641.86 | bwd_allreduce: 86.05 | step: 19.73

  8%|▊         | 3372/41250 [8:07:59<90:48:05,  8.63s/it]
                                                         
{'loss': 0.1387, 'grad_norm': 0.9758133292198181, 'learning_rate': 3.9719913757868995e-05, 'epoch': 0.82}

  8%|▊         | 3372/41250 [8:07:59<90:48:05,  8.63s/it][2025-04-25 16:05:43,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:05:43,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.02 | bwd_microstep: 5761.76 | bwd_inner_microstep: 5748.90 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.99
[2025-04-25 16:05:43,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.02 | bwd: 5761.77 | bwd_inner: 5748.90 | bwd_allreduce: 12.84 | step: 18.99

  8%|▊         | 3373/41250 [8:08:08<91:05:12,  8.66s/it]
                                                         
{'loss': 0.2215, 'grad_norm': 1.9488927125930786, 'learning_rate': 3.971965181267231e-05, 'epoch': 0.82}

  8%|▊         | 3373/41250 [8:08:08<91:05:12,  8.66s/it][2025-04-25 16:05:52,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 16:05:52,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.61 | bwd_microstep: 5740.03 | bwd_inner_microstep: 5649.61 | bwd_allreduce_microstep: 90.37 | step_microstep: 19.15
[2025-04-25 16:05:52,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.61 | bwd: 5740.05 | bwd_inner: 5649.61 | bwd_allreduce: 90.39 | step: 19.15

  8%|▊         | 3374/41250 [8:08:17<91:04:40,  8.66s/it]
                                                         
{'loss': 0.2644, 'grad_norm': 1.9915919303894043, 'learning_rate': 3.971938974590784e-05, 'epoch': 0.82}

  8%|▊         | 3374/41250 [8:08:17<91:04:40,  8.66s/it][2025-04-25 16:06:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 16:06:00,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.64 | bwd_microstep: 5772.65 | bwd_inner_microstep: 5759.81 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.47
[2025-04-25 16:06:00,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.64 | bwd: 5772.66 | bwd_inner: 5759.81 | bwd_allreduce: 12.80 | step: 19.47

  8%|▊         | 3375/41250 [8:08:26<91:19:42,  8.68s/it]
                                                         
{'loss': 0.0741, 'grad_norm': 0.6260166764259338, 'learning_rate': 3.9719127557577216e-05, 'epoch': 0.82}

  8%|▊         | 3375/41250 [8:08:26<91:19:42,  8.68s/it][2025-04-25 16:06:09,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:06:09,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.37 | bwd_microstep: 5673.99 | bwd_inner_microstep: 5648.93 | bwd_allreduce_microstep: 25.02 | step_microstep: 18.66
[2025-04-25 16:06:09,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.38 | bwd: 5674.01 | bwd_inner: 5648.93 | bwd_allreduce: 25.03 | step: 18.67

  8%|▊         | 3376/41250 [8:08:34<91:00:49,  8.65s/it]
                                                         
{'loss': 0.1721, 'grad_norm': 2.1332614421844482, 'learning_rate': 3.971886524768205e-05, 'epoch': 0.82}

  8%|▊         | 3376/41250 [8:08:34<91:00:49,  8.65s/it][2025-04-25 16:06:17,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 16:06:17,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.70 | bwd_microstep: 5756.03 | bwd_inner_microstep: 5643.87 | bwd_allreduce_microstep: 112.12 | step_microstep: 18.91
[2025-04-25 16:06:17,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.70 | bwd: 5756.04 | bwd_inner: 5643.87 | bwd_allreduce: 112.14 | step: 18.91

  8%|▊         | 3377/41250 [8:08:43<91:02:18,  8.65s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.1613707542419434, 'learning_rate': 3.9718602816223955e-05, 'epoch': 0.82}

  8%|▊         | 3377/41250 [8:08:43<91:02:18,  8.65s/it][2025-04-25 16:06:26,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:06:26,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.11 | bwd_microstep: 5763.41 | bwd_inner_microstep: 5638.52 | bwd_allreduce_microstep: 124.83 | step_microstep: 19.00
[2025-04-25 16:06:26,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.11 | bwd: 5763.43 | bwd_inner: 5638.52 | bwd_allreduce: 124.86 | step: 19.00

  8%|▊         | 3378/41250 [8:08:51<91:04:27,  8.66s/it]
                                                         
{'loss': 0.1563, 'grad_norm': 1.943469524383545, 'learning_rate': 3.9718340263204556e-05, 'epoch': 0.82}

  8%|▊         | 3378/41250 [8:08:51<91:04:27,  8.66s/it][2025-04-25 16:06:35,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:06:35,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.77 | bwd_microstep: 5677.29 | bwd_inner_microstep: 5653.65 | bwd_allreduce_microstep: 23.59 | step_microstep: 18.83
[2025-04-25 16:06:35,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.77 | bwd: 5677.30 | bwd_inner: 5653.65 | bwd_allreduce: 23.61 | step: 18.83

  8%|▊         | 3379/41250 [8:09:00<90:53:31,  8.64s/it]
                                                         
{'loss': 0.1242, 'grad_norm': 1.0747737884521484, 'learning_rate': 3.971807758862546e-05, 'epoch': 0.82}

  8%|▊         | 3379/41250 [8:09:00<90:53:31,  8.64s/it][2025-04-25 16:06:43,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:06:43,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.69 | bwd_microstep: 5764.43 | bwd_inner_microstep: 5647.85 | bwd_allreduce_microstep: 116.54 | step_microstep: 18.75
[2025-04-25 16:06:43,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.69 | bwd: 5764.45 | bwd_inner: 5647.85 | bwd_allreduce: 116.56 | step: 18.75

  8%|▊         | 3380/41250 [8:09:09<91:01:25,  8.65s/it]
                                                         
{'loss': 0.2505, 'grad_norm': 1.2937272787094116, 'learning_rate': 3.97178147924883e-05, 'epoch': 0.82}

  8%|▊         | 3380/41250 [8:09:09<91:01:25,  8.65s/it][2025-04-25 16:06:52,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 16:06:52,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.75 | bwd_microstep: 5855.97 | bwd_inner_microstep: 5693.78 | bwd_allreduce_microstep: 162.14 | step_microstep: 18.88
[2025-04-25 16:06:52,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.75 | bwd: 5855.98 | bwd_inner: 5693.78 | bwd_allreduce: 162.16 | step: 18.88

  8%|▊         | 3381/41250 [8:09:18<91:27:07,  8.69s/it]
                                                         
{'loss': 0.1585, 'grad_norm': 0.6584874391555786, 'learning_rate': 3.971755187479469e-05, 'epoch': 0.82}

  8%|▊         | 3381/41250 [8:09:18<91:27:07,  8.69s/it][2025-04-25 16:07:01,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-25 16:07:01,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.92 | bwd_microstep: 5686.90 | bwd_inner_microstep: 5645.16 | bwd_allreduce_microstep: 41.70 | step_microstep: 19.64
[2025-04-25 16:07:01,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.92 | bwd: 5686.92 | bwd_inner: 5645.16 | bwd_allreduce: 41.72 | step: 19.65

  8%|▊         | 3382/41250 [8:09:26<91:12:03,  8.67s/it]
                                                         
{'loss': 0.1726, 'grad_norm': 1.4541610479354858, 'learning_rate': 3.971728883554625e-05, 'epoch': 0.82}

  8%|▊         | 3382/41250 [8:09:26<91:12:03,  8.67s/it][2025-04-25 16:07:10,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 16:07:10,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.80 | bwd_microstep: 5749.62 | bwd_inner_microstep: 5656.21 | bwd_allreduce_microstep: 93.36 | step_microstep: 19.04
[2025-04-25 16:07:10,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.80 | bwd: 5749.63 | bwd_inner: 5656.21 | bwd_allreduce: 93.38 | step: 19.05

  8%|▊         | 3383/41250 [8:09:35<91:12:57,  8.67s/it]
                                                         
{'loss': 0.2759, 'grad_norm': 3.864108085632324, 'learning_rate': 3.9717025674744604e-05, 'epoch': 0.82}

  8%|▊         | 3383/41250 [8:09:35<91:12:57,  8.67s/it][2025-04-25 16:07:18,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 16:07:18,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.52 | bwd_microstep: 5756.04 | bwd_inner_microstep: 5713.44 | bwd_allreduce_microstep: 42.55 | step_microstep: 18.43
[2025-04-25 16:07:18,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.52 | bwd: 5756.05 | bwd_inner: 5713.44 | bwd_allreduce: 42.57 | step: 18.43

  8%|▊         | 3384/41250 [8:09:44<91:17:22,  8.68s/it]
                                                         
{'loss': 0.1104, 'grad_norm': 1.3354086875915527, 'learning_rate': 3.971676239239137e-05, 'epoch': 0.82}

  8%|▊         | 3384/41250 [8:09:44<91:17:22,  8.68s/it][2025-04-25 16:07:27,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:07:27,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.69 | bwd_microstep: 5755.68 | bwd_inner_microstep: 5651.67 | bwd_allreduce_microstep: 103.97 | step_microstep: 18.72
[2025-04-25 16:07:27,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.69 | bwd: 5755.70 | bwd_inner: 5651.67 | bwd_allreduce: 103.99 | step: 18.72

  8%|▊         | 3385/41250 [8:09:52<91:14:58,  8.68s/it]
                                                         
{'loss': 0.1457, 'grad_norm': 1.2941609621047974, 'learning_rate': 3.9716498988488174e-05, 'epoch': 0.82}

  8%|▊         | 3385/41250 [8:09:52<91:14:58,  8.68s/it][2025-04-25 16:07:35,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:07:35,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.29 | bwd_microstep: 5686.84 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 31.24 | step_microstep: 18.72
[2025-04-25 16:07:35,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.29 | bwd: 5686.86 | bwd_inner: 5655.56 | bwd_allreduce: 31.25 | step: 18.73

  8%|▊         | 3386/41250 [8:10:01<91:00:28,  8.65s/it]
                                                         
{'loss': 0.122, 'grad_norm': 1.1906908750534058, 'learning_rate': 3.971623546303664e-05, 'epoch': 0.82}

  8%|▊         | 3386/41250 [8:10:01<91:00:28,  8.65s/it][2025-04-25 16:07:44,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 16:07:44,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.15 | bwd_microstep: 5769.17 | bwd_inner_microstep: 5654.67 | bwd_allreduce_microstep: 114.44 | step_microstep: 18.56
[2025-04-25 16:07:44,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.15 | bwd: 5769.18 | bwd_inner: 5654.67 | bwd_allreduce: 114.47 | step: 18.56

  8%|▊         | 3387/41250 [8:10:10<91:07:12,  8.66s/it]
                                                         
{'loss': 0.2274, 'grad_norm': 3.4730770587921143, 'learning_rate': 3.971597181603839e-05, 'epoch': 0.82}

  8%|▊         | 3387/41250 [8:10:10<91:07:12,  8.66s/it][2025-04-25 16:07:53,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.09 | optimizer_step: 1.00
[2025-04-25 16:07:53,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.12 | bwd_microstep: 5704.55 | bwd_inner_microstep: 5691.20 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.70
[2025-04-25 16:07:53,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.12 | bwd: 5704.57 | bwd_inner: 5691.20 | bwd_allreduce: 13.33 | step: 19.70

  8%|▊         | 3388/41250 [8:10:18<91:03:09,  8.66s/it]
                                                         
{'loss': 0.2145, 'grad_norm': 1.8986073732376099, 'learning_rate': 3.971570804749506e-05, 'epoch': 0.82}

  8%|▊         | 3388/41250 [8:10:18<91:03:09,  8.66s/it][2025-04-25 16:08:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:08:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.90 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5702.57 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.58
[2025-04-25 16:08:01,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.90 | bwd: 5715.48 | bwd_inner: 5702.57 | bwd_allreduce: 12.87 | step: 18.59

  8%|▊         | 3389/41250 [8:10:27<91:04:28,  8.66s/it]
                                                         
{'loss': 0.3054, 'grad_norm': 2.3720803260803223, 'learning_rate': 3.9715444157408256e-05, 'epoch': 0.82}

  8%|▊         | 3389/41250 [8:10:27<91:04:28,  8.66s/it][2025-04-25 16:08:10,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 16:08:10,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5703.98 | bwd_inner_microstep: 5662.79 | bwd_allreduce_microstep: 41.14 | step_microstep: 18.99
[2025-04-25 16:08:10,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.00 | bwd: 5703.99 | bwd_inner: 5662.79 | bwd_allreduce: 41.15 | step: 18.99

  8%|▊         | 3390/41250 [8:10:35<91:01:10,  8.65s/it]
                                                         
{'loss': 0.1431, 'grad_norm': 1.0083551406860352, 'learning_rate': 3.971518014577963e-05, 'epoch': 0.82}

  8%|▊         | 3390/41250 [8:10:35<91:01:10,  8.65s/it][2025-04-25 16:08:19,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 16:08:19,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5728.30 | bwd_inner_microstep: 5692.33 | bwd_allreduce_microstep: 35.92 | step_microstep: 18.97
[2025-04-25 16:08:19,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5728.31 | bwd_inner: 5692.33 | bwd_allreduce: 35.94 | step: 18.97

  8%|▊         | 3391/41250 [8:10:44<91:03:16,  8.66s/it]
                                                         
{'loss': 0.1891, 'grad_norm': 1.6084049940109253, 'learning_rate': 3.971491601261079e-05, 'epoch': 0.82}

  8%|▊         | 3391/41250 [8:10:44<91:03:16,  8.66s/it][2025-04-25 16:08:27,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:08:27,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.12 | bwd_microstep: 5719.94 | bwd_inner_microstep: 5707.15 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.74
[2025-04-25 16:08:27,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.12 | bwd: 5719.96 | bwd_inner: 5707.15 | bwd_allreduce: 12.76 | step: 18.74

  8%|▊         | 3392/41250 [8:10:53<91:04:18,  8.66s/it]
                                                         
{'loss': 0.0678, 'grad_norm': 0.8547126054763794, 'learning_rate': 3.9714651757903364e-05, 'epoch': 0.82}

  8%|▊         | 3392/41250 [8:10:53<91:04:18,  8.66s/it][2025-04-25 16:08:36,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:08:36,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.94 | bwd_microstep: 5783.71 | bwd_inner_microstep: 5770.77 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.00
[2025-04-25 16:08:36,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.94 | bwd: 5783.72 | bwd_inner: 5770.77 | bwd_allreduce: 12.91 | step: 18.00

  8%|▊         | 3393/41250 [8:11:02<91:21:54,  8.69s/it]
                                                         
{'loss': 0.188, 'grad_norm': 1.471179485321045, 'learning_rate': 3.9714387381659e-05, 'epoch': 0.82}

  8%|▊         | 3393/41250 [8:11:02<91:21:54,  8.69s/it][2025-04-25 16:08:45,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 16:08:45,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.45 | bwd_microstep: 5726.15 | bwd_inner_microstep: 5713.57 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.60
[2025-04-25 16:08:45,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.45 | bwd: 5726.16 | bwd_inner: 5713.57 | bwd_allreduce: 12.55 | step: 18.60

  8%|▊         | 3394/41250 [8:11:10<91:19:29,  8.68s/it]
                                                         
{'loss': 0.3709, 'grad_norm': 1.5512747764587402, 'learning_rate': 3.971412288387931e-05, 'epoch': 0.82}

  8%|▊         | 3394/41250 [8:11:10<91:19:29,  8.68s/it][2025-04-25 16:08:54,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.27
[2025-04-25 16:08:54,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.91 | bwd_microstep: 5779.81 | bwd_inner_microstep: 5713.34 | bwd_allreduce_microstep: 66.42 | step_microstep: 20.02
[2025-04-25 16:08:54,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.91 | bwd: 5779.82 | bwd_inner: 5713.34 | bwd_allreduce: 66.44 | step: 20.02

  8%|▊         | 3395/41250 [8:11:19<91:26:19,  8.70s/it]
                                                         
{'loss': 0.1715, 'grad_norm': 1.830405831336975, 'learning_rate': 3.971385826456592e-05, 'epoch': 0.82}

  8%|▊         | 3395/41250 [8:11:19<91:26:19,  8.70s/it][2025-04-25 16:09:02,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.09 | optimizer_step: 0.99
[2025-04-25 16:09:02,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.58 | bwd_microstep: 5800.32 | bwd_inner_microstep: 5663.58 | bwd_allreduce_microstep: 136.69 | step_microstep: 19.10
[2025-04-25 16:09:02,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.58 | bwd: 5800.34 | bwd_inner: 5663.58 | bwd_allreduce: 136.71 | step: 19.10

  8%|▊         | 3396/41250 [8:11:28<91:30:42,  8.70s/it]
                                                         
{'loss': 0.2031, 'grad_norm': 1.7594300508499146, 'learning_rate': 3.9713593523720484e-05, 'epoch': 0.82}

  8%|▊         | 3396/41250 [8:11:28<91:30:42,  8.70s/it][2025-04-25 16:09:11,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:09:11,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5963.99 | bwd_inner_microstep: 5951.30 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.87
[2025-04-25 16:09:11,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.01 | bwd: 5964.00 | bwd_inner: 5951.30 | bwd_allreduce: 12.66 | step: 18.88

  8%|▊         | 3397/41250 [8:11:37<92:08:33,  8.76s/it]
                                                         
{'loss': 0.1594, 'grad_norm': 1.506915807723999, 'learning_rate': 3.971332866134462e-05, 'epoch': 0.82}

  8%|▊         | 3397/41250 [8:11:37<92:08:33,  8.76s/it][2025-04-25 16:09:20,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 16:09:20,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.55 | bwd_microstep: 5734.04 | bwd_inner_microstep: 5721.32 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.91
[2025-04-25 16:09:20,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.55 | bwd: 5734.05 | bwd_inner: 5721.32 | bwd_allreduce: 12.69 | step: 18.91

  8%|▊         | 3398/41250 [8:11:45<91:51:26,  8.74s/it]
                                                         
{'loss': 0.2261, 'grad_norm': 1.4866183996200562, 'learning_rate': 3.971306367743996e-05, 'epoch': 0.82}

  8%|▊         | 3398/41250 [8:11:45<91:51:26,  8.74s/it][2025-04-25 16:09:29,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.07 | optimizer_step: 0.97
[2025-04-25 16:09:29,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5690.33 | bwd_inner_microstep: 5663.78 | bwd_allreduce_microstep: 26.50 | step_microstep: 19.22
[2025-04-25 16:09:29,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5690.34 | bwd_inner: 5663.78 | bwd_allreduce: 26.52 | step: 19.22

  8%|▊         | 3399/41250 [8:11:54<91:27:28,  8.70s/it]
                                                         
{'loss': 0.2268, 'grad_norm': 1.8439022302627563, 'learning_rate': 3.971279857200814e-05, 'epoch': 0.82}

  8%|▊         | 3399/41250 [8:11:54<91:27:28,  8.70s/it][2025-04-25 16:09:37,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:09:37,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5681.16 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 23.27 | step_microstep: 18.96
[2025-04-25 16:09:37,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5681.18 | bwd_inner: 5657.84 | bwd_allreduce: 23.29 | step: 18.96

  8%|▊         | 3400/41250 [8:12:02<91:07:38,  8.67s/it]
                                                         
{'loss': 0.2418, 'grad_norm': 1.3497098684310913, 'learning_rate': 3.971253334505079e-05, 'epoch': 0.82}

  8%|▊         | 3400/41250 [8:12:02<91:07:38,  8.67s/it][2025-04-25 16:09:46,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 16:09:46,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.01 | bwd_microstep: 5760.75 | bwd_inner_microstep: 5666.18 | bwd_allreduce_microstep: 94.52 | step_microstep: 18.53
[2025-04-25 16:09:46,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.01 | bwd: 5760.76 | bwd_inner: 5666.18 | bwd_allreduce: 94.54 | step: 18.53

  8%|▊         | 3401/41250 [8:12:11<91:09:35,  8.67s/it]
                                                         
{'loss': 0.0299, 'grad_norm': 0.2849203050136566, 'learning_rate': 3.971226799656955e-05, 'epoch': 0.82}

  8%|▊         | 3401/41250 [8:12:11<91:09:35,  8.67s/it][2025-04-25 16:09:54,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.12
[2025-04-25 16:09:54,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.85 | bwd_microstep: 5693.62 | bwd_inner_microstep: 5656.19 | bwd_allreduce_microstep: 37.39 | step_microstep: 19.41
[2025-04-25 16:09:54,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.86 | bwd: 5693.64 | bwd_inner: 5656.19 | bwd_allreduce: 37.41 | step: 19.41

  8%|▊         | 3402/41250 [8:12:20<90:57:01,  8.65s/it]
                                                         
{'loss': 0.2145, 'grad_norm': 2.1473209857940674, 'learning_rate': 3.9712002526566056e-05, 'epoch': 0.82}

  8%|▊         | 3402/41250 [8:12:20<90:57:01,  8.65s/it][2025-04-25 16:10:03,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.82 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:10:03,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.83 | bwd_microstep: 5749.97 | bwd_inner_microstep: 5648.73 | bwd_allreduce_microstep: 101.20 | step_microstep: 17.90
[2025-04-25 16:10:03,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.83 | bwd: 5749.99 | bwd_inner: 5648.73 | bwd_allreduce: 101.22 | step: 17.90

  8%|▊         | 3403/41250 [8:12:28<91:01:00,  8.66s/it]
                                                         
{'loss': 0.1401, 'grad_norm': 1.4588325023651123, 'learning_rate': 3.9711736935041954e-05, 'epoch': 0.82}

  8%|▊         | 3403/41250 [8:12:28<91:01:00,  8.66s/it][2025-04-25 16:10:12,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 16:10:12,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.37 | bwd_microstep: 5737.16 | bwd_inner_microstep: 5686.76 | bwd_allreduce_microstep: 50.35 | step_microstep: 19.82
[2025-04-25 16:10:12,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.37 | bwd: 5737.17 | bwd_inner: 5686.76 | bwd_allreduce: 50.37 | step: 19.83

  8%|▊         | 3404/41250 [8:12:37<91:05:46,  8.67s/it]
                                                         
{'loss': 0.1206, 'grad_norm': 1.387634515762329, 'learning_rate': 3.971147122199885e-05, 'epoch': 0.83}

  8%|▊         | 3404/41250 [8:12:37<91:05:46,  8.67s/it][2025-04-25 16:10:20,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 16:10:20,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.46 | bwd_microstep: 5786.71 | bwd_inner_microstep: 5658.68 | bwd_allreduce_microstep: 127.98 | step_microstep: 18.80
[2025-04-25 16:10:20,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.46 | bwd: 5786.72 | bwd_inner: 5658.68 | bwd_allreduce: 128.00 | step: 18.81

  8%|▊         | 3405/41250 [8:12:46<91:12:55,  8.68s/it]
                                                         
{'loss': 0.0875, 'grad_norm': 1.0787583589553833, 'learning_rate': 3.971120538743842e-05, 'epoch': 0.83}

  8%|▊         | 3405/41250 [8:12:46<91:12:55,  8.68s/it][2025-04-25 16:10:29,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 16:10:29,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5708.73 | bwd_inner_microstep: 5695.99 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.85
[2025-04-25 16:10:29,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5708.75 | bwd_inner: 5695.99 | bwd_allreduce: 12.71 | step: 18.86

  8%|▊         | 3406/41250 [8:12:54<91:05:51,  8.67s/it]
                                                         
{'loss': 0.1303, 'grad_norm': 0.6845543384552002, 'learning_rate': 3.971093943136228e-05, 'epoch': 0.83}

  8%|▊         | 3406/41250 [8:12:54<91:05:51,  8.67s/it][2025-04-25 16:10:38,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 16:10:38,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.03 | bwd_microstep: 5683.06 | bwd_inner_microstep: 5657.54 | bwd_allreduce_microstep: 25.48 | step_microstep: 18.90
[2025-04-25 16:10:38,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.03 | bwd: 5683.08 | bwd_inner: 5657.54 | bwd_allreduce: 25.50 | step: 18.90

  8%|▊         | 3407/41250 [8:13:03<90:52:03,  8.64s/it]
                                                         
{'loss': 0.1343, 'grad_norm': 1.0986088514328003, 'learning_rate': 3.971067335377208e-05, 'epoch': 0.83}

  8%|▊         | 3407/41250 [8:13:03<90:52:03,  8.64s/it][2025-04-25 16:10:46,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:10:46,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.87 | bwd_microstep: 5795.98 | bwd_inner_microstep: 5782.99 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.78
[2025-04-25 16:10:46,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.87 | bwd: 5796.00 | bwd_inner: 5782.99 | bwd_allreduce: 12.96 | step: 18.79

  8%|▊         | 3408/41250 [8:13:12<91:15:43,  8.68s/it]
                                                         
{'loss': 0.2429, 'grad_norm': 1.6811673641204834, 'learning_rate': 3.971040715466945e-05, 'epoch': 0.83}

  8%|▊         | 3408/41250 [8:13:12<91:15:43,  8.68s/it][2025-04-25 16:10:55,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:10:55,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5721.80 | bwd_inner_microstep: 5709.17 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.80
[2025-04-25 16:10:55,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5721.81 | bwd_inner: 5709.17 | bwd_allreduce: 12.60 | step: 18.80

  8%|▊         | 3409/41250 [8:13:20<91:11:06,  8.67s/it]
                                                         
{'loss': 0.0651, 'grad_norm': 0.72889244556427, 'learning_rate': 3.971014083405604e-05, 'epoch': 0.83}

  8%|▊         | 3409/41250 [8:13:20<91:11:06,  8.67s/it][2025-04-25 16:11:04,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:11:04,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.75 | bwd_microstep: 5721.63 | bwd_inner_microstep: 5681.13 | bwd_allreduce_microstep: 40.46 | step_microstep: 18.79
[2025-04-25 16:11:04,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.75 | bwd: 5721.65 | bwd_inner: 5681.13 | bwd_allreduce: 40.47 | step: 18.79

  8%|▊         | 3410/41250 [8:13:29<91:06:12,  8.67s/it]
                                                         
{'loss': 0.0839, 'grad_norm': 0.9577512145042419, 'learning_rate': 3.970987439193349e-05, 'epoch': 0.83}

  8%|▊         | 3410/41250 [8:13:29<91:06:12,  8.67s/it][2025-04-25 16:11:12,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 16:11:12,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.90 | bwd_microstep: 5689.53 | bwd_inner_microstep: 5676.88 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.38
[2025-04-25 16:11:12,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.90 | bwd: 5689.54 | bwd_inner: 5676.88 | bwd_allreduce: 12.62 | step: 19.38

  8%|▊         | 3411/41250 [8:13:38<90:56:22,  8.65s/it]
                                                         
{'loss': 0.2101, 'grad_norm': 2.0913941860198975, 'learning_rate': 3.970960782830344e-05, 'epoch': 0.83}

  8%|▊         | 3411/41250 [8:13:38<90:56:22,  8.65s/it][2025-04-25 16:11:21,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:11:21,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.18 | bwd_microstep: 5990.42 | bwd_inner_microstep: 5693.43 | bwd_allreduce_microstep: 296.94 | step_microstep: 18.67
[2025-04-25 16:11:21,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.18 | bwd: 5990.43 | bwd_inner: 5693.43 | bwd_allreduce: 296.96 | step: 18.67

  8%|▊         | 3412/41250 [8:13:47<91:45:56,  8.73s/it]
                                                         
{'loss': 0.1244, 'grad_norm': 1.2510918378829956, 'learning_rate': 3.970934114316752e-05, 'epoch': 0.83}

  8%|▊         | 3412/41250 [8:13:47<91:45:56,  8.73s/it][2025-04-25 16:11:30,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:11:30,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.42 | bwd_microstep: 5735.00 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 35.77 | step_microstep: 19.15
[2025-04-25 16:11:30,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.42 | bwd: 5735.01 | bwd_inner: 5699.18 | bwd_allreduce: 35.79 | step: 19.15

  8%|▊         | 3413/41250 [8:13:55<91:33:11,  8.71s/it]
                                                         
{'loss': 0.3481, 'grad_norm': 2.781582832336426, 'learning_rate': 3.9709074336527414e-05, 'epoch': 0.83}

  8%|▊         | 3413/41250 [8:13:55<91:33:11,  8.71s/it][2025-04-25 16:11:39,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:11:39,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.28 | bwd_microstep: 5707.63 | bwd_inner_microstep: 5653.62 | bwd_allreduce_microstep: 53.96 | step_microstep: 18.81
[2025-04-25 16:11:39,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.28 | bwd: 5707.65 | bwd_inner: 5653.62 | bwd_allreduce: 53.98 | step: 18.81

  8%|▊         | 3414/41250 [8:14:04<91:14:14,  8.68s/it]
                                                         
{'loss': 0.1168, 'grad_norm': 0.9883065819740295, 'learning_rate': 3.9708807408384726e-05, 'epoch': 0.83}

  8%|▊         | 3414/41250 [8:14:04<91:14:14,  8.68s/it][2025-04-25 16:11:47,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:11:47,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.05 | bwd_microstep: 5861.57 | bwd_inner_microstep: 5699.35 | bwd_allreduce_microstep: 162.18 | step_microstep: 18.95
[2025-04-25 16:11:47,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.06 | bwd: 5861.59 | bwd_inner: 5699.35 | bwd_allreduce: 162.19 | step: 18.96

  8%|▊         | 3415/41250 [8:14:13<91:35:24,  8.71s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 1.22129225730896, 'learning_rate': 3.970854035874112e-05, 'epoch': 0.83}

  8%|▊         | 3415/41250 [8:14:13<91:35:24,  8.71s/it][2025-04-25 16:11:56,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 16:11:56,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.11 | bwd_microstep: 5867.30 | bwd_inner_microstep: 5643.71 | bwd_allreduce_microstep: 223.55 | step_microstep: 19.14
[2025-04-25 16:11:56,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.11 | bwd: 5867.32 | bwd_inner: 5643.71 | bwd_allreduce: 223.56 | step: 19.14

  8%|▊         | 3416/41250 [8:14:21<91:49:04,  8.74s/it]
                                                         
{'loss': 0.3361, 'grad_norm': 1.4317656755447388, 'learning_rate': 3.9708273187598235e-05, 'epoch': 0.83}

  8%|▊         | 3416/41250 [8:14:21<91:49:04,  8.74s/it][2025-04-25 16:12:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-25 16:12:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.54 | bwd_microstep: 5755.37 | bwd_inner_microstep: 5658.21 | bwd_allreduce_microstep: 97.11 | step_microstep: 19.03
[2025-04-25 16:12:05,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.54 | bwd: 5755.39 | bwd_inner: 5658.21 | bwd_allreduce: 97.13 | step: 19.03

  8%|▊         | 3417/41250 [8:14:30<91:35:06,  8.71s/it]
                                                         
{'loss': 0.2516, 'grad_norm': 1.9068751335144043, 'learning_rate': 3.9708005894957726e-05, 'epoch': 0.83}

  8%|▊         | 3417/41250 [8:14:30<91:35:06,  8.71s/it][2025-04-25 16:12:13,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 16:12:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.96 | bwd_microstep: 5686.87 | bwd_inner_microstep: 5641.04 | bwd_allreduce_microstep: 45.79 | step_microstep: 18.64
[2025-04-25 16:12:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.96 | bwd: 5686.89 | bwd_inner: 5641.04 | bwd_allreduce: 45.81 | step: 18.64

  8%|▊         | 3418/41250 [8:14:39<91:12:04,  8.68s/it]
                                                         
{'loss': 0.0784, 'grad_norm': 0.7613996863365173, 'learning_rate': 3.970773848082124e-05, 'epoch': 0.83}

  8%|▊         | 3418/41250 [8:14:39<91:12:04,  8.68s/it][2025-04-25 16:12:22,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 16:12:22,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.97 | bwd_microstep: 5693.98 | bwd_inner_microstep: 5653.31 | bwd_allreduce_microstep: 40.63 | step_microstep: 18.78
[2025-04-25 16:12:22,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.97 | bwd: 5693.99 | bwd_inner: 5653.31 | bwd_allreduce: 40.64 | step: 18.78

  8%|▊         | 3419/41250 [8:14:47<90:58:33,  8.66s/it]
                                                         
{'loss': 0.1906, 'grad_norm': 1.2110220193862915, 'learning_rate': 3.970747094519041e-05, 'epoch': 0.83}

  8%|▊         | 3419/41250 [8:14:47<90:58:33,  8.66s/it][2025-04-25 16:12:31,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:12:31,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.46 | bwd_microstep: 5689.94 | bwd_inner_microstep: 5655.46 | bwd_allreduce_microstep: 34.44 | step_microstep: 18.70
[2025-04-25 16:12:31,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.46 | bwd: 5689.95 | bwd_inner: 5655.46 | bwd_allreduce: 34.45 | step: 18.71

  8%|▊         | 3420/41250 [8:14:56<90:47:14,  8.64s/it]
                                                         
{'loss': 0.1876, 'grad_norm': 1.2357045412063599, 'learning_rate': 3.970720328806691e-05, 'epoch': 0.83}

  8%|▊         | 3420/41250 [8:14:56<90:47:14,  8.64s/it][2025-04-25 16:12:39,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:12:39,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.46 | bwd_microstep: 5694.93 | bwd_inner_microstep: 5652.49 | bwd_allreduce_microstep: 42.40 | step_microstep: 18.67
[2025-04-25 16:12:39,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.46 | bwd: 5694.95 | bwd_inner: 5652.49 | bwd_allreduce: 42.42 | step: 18.67

  8%|▊         | 3421/41250 [8:15:05<90:41:05,  8.63s/it]
                                                         
{'loss': 0.114, 'grad_norm': 1.5122389793395996, 'learning_rate': 3.970693550945238e-05, 'epoch': 0.83}

  8%|▊         | 3421/41250 [8:15:05<90:41:05,  8.63s/it][2025-04-25 16:12:48,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 16:12:48,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.36 | bwd_microstep: 5769.52 | bwd_inner_microstep: 5756.82 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.50
[2025-04-25 16:12:48,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.36 | bwd: 5769.53 | bwd_inner: 5756.82 | bwd_allreduce: 12.67 | step: 18.50

  8%|▊         | 3422/41250 [8:15:13<90:59:25,  8.66s/it]
                                                         
{'loss': 0.4843, 'grad_norm': 3.4341673851013184, 'learning_rate': 3.970666760934846e-05, 'epoch': 0.83}

  8%|▊         | 3422/41250 [8:15:13<90:59:25,  8.66s/it][2025-04-25 16:12:57,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:12:57,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.93 | bwd_microstep: 5697.20 | bwd_inner_microstep: 5649.24 | bwd_allreduce_microstep: 47.92 | step_microstep: 18.80
[2025-04-25 16:12:57,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.93 | bwd: 5697.21 | bwd_inner: 5649.24 | bwd_allreduce: 47.93 | step: 18.80

  8%|▊         | 3423/41250 [8:15:22<90:50:52,  8.65s/it]
                                                         
{'loss': 0.2199, 'grad_norm': 1.8536298274993896, 'learning_rate': 3.970639958775682e-05, 'epoch': 0.83}

  8%|▊         | 3423/41250 [8:15:22<90:50:52,  8.65s/it][2025-04-25 16:13:05,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 16:13:05,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.82 | bwd_microstep: 5738.26 | bwd_inner_microstep: 5661.70 | bwd_allreduce_microstep: 76.51 | step_microstep: 18.97
[2025-04-25 16:13:05,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.82 | bwd: 5738.27 | bwd_inner: 5661.70 | bwd_allreduce: 76.53 | step: 18.97

  8%|▊         | 3424/41250 [8:15:31<90:53:36,  8.65s/it]
                                                         
{'loss': 0.0835, 'grad_norm': 0.9099414944648743, 'learning_rate': 3.970613144467909e-05, 'epoch': 0.83}

  8%|▊         | 3424/41250 [8:15:31<90:53:36,  8.65s/it][2025-04-25 16:13:14,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 16:13:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.21 | bwd_microstep: 5745.48 | bwd_inner_microstep: 5659.71 | bwd_allreduce_microstep: 85.72 | step_microstep: 19.05
[2025-04-25 16:13:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.21 | bwd: 5745.49 | bwd_inner: 5659.71 | bwd_allreduce: 85.74 | step: 19.06

  8%|▊         | 3425/41250 [8:15:39<90:56:58,  8.66s/it]
                                                         
{'loss': 0.1246, 'grad_norm': 1.4702656269073486, 'learning_rate': 3.970586318011694e-05, 'epoch': 0.83}

  8%|▊         | 3425/41250 [8:15:39<90:56:58,  8.66s/it][2025-04-25 16:13:23,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:13:23,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5777.14 | bwd_inner_microstep: 5644.67 | bwd_allreduce_microstep: 132.42 | step_microstep: 19.09
[2025-04-25 16:13:23,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5777.16 | bwd_inner: 5644.67 | bwd_allreduce: 132.44 | step: 19.09

  8%|▊         | 3426/41250 [8:15:48<91:03:12,  8.67s/it]
                                                         
{'loss': 0.1599, 'grad_norm': 0.9803867936134338, 'learning_rate': 3.970559479407203e-05, 'epoch': 0.83}

  8%|▊         | 3426/41250 [8:15:48<91:03:12,  8.67s/it][2025-04-25 16:13:31,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.20 | optimizer_step: 1.19
[2025-04-25 16:13:31,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.47 | bwd_microstep: 5771.76 | bwd_inner_microstep: 5699.42 | bwd_allreduce_microstep: 72.28 | step_microstep: 20.00
[2025-04-25 16:13:31,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.47 | bwd: 5771.78 | bwd_inner: 5699.42 | bwd_allreduce: 72.31 | step: 20.01

  8%|▊         | 3427/41250 [8:15:57<91:12:33,  8.68s/it]
                                                         
{'loss': 0.4709, 'grad_norm': 3.0871551036834717, 'learning_rate': 3.970532628654599e-05, 'epoch': 0.83}

  8%|▊         | 3427/41250 [8:15:57<91:12:33,  8.68s/it][2025-04-25 16:13:40,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 16:13:40,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.98 | bwd_microstep: 5722.74 | bwd_inner_microstep: 5709.80 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.75
[2025-04-25 16:13:40,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.98 | bwd: 5722.75 | bwd_inner: 5709.80 | bwd_allreduce: 12.91 | step: 18.75

  8%|▊         | 3428/41250 [8:16:05<91:11:57,  8.68s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 1.1033573150634766, 'learning_rate': 3.97050576575405e-05, 'epoch': 0.83}

  8%|▊         | 3428/41250 [8:16:05<91:11:57,  8.68s/it][2025-04-25 16:13:49,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:13:49,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.69 | bwd_microstep: 5722.11 | bwd_inner_microstep: 5709.48 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.47
[2025-04-25 16:13:49,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.69 | bwd: 5722.12 | bwd_inner: 5709.48 | bwd_allreduce: 12.61 | step: 18.47

  8%|▊         | 3429/41250 [8:16:14<91:07:27,  8.67s/it]
                                                         
{'loss': 0.1343, 'grad_norm': 3.5240566730499268, 'learning_rate': 3.970478890705719e-05, 'epoch': 0.83}

  8%|▊         | 3429/41250 [8:16:14<91:07:27,  8.67s/it][2025-04-25 16:13:57,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:13:57,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.32 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5660.69 | bwd_allreduce_microstep: 108.37 | step_microstep: 18.79
[2025-04-25 16:13:57,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.32 | bwd: 5769.12 | bwd_inner: 5660.69 | bwd_allreduce: 108.38 | step: 18.79

  8%|▊         | 3430/41250 [8:16:23<91:08:39,  8.68s/it]
                                                         
{'loss': 0.155, 'grad_norm': 0.9284704327583313, 'learning_rate': 3.9704520035097746e-05, 'epoch': 0.83}

  8%|▊         | 3430/41250 [8:16:23<91:08:39,  8.68s/it][2025-04-25 16:14:06,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 16:14:06,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.31 | bwd_microstep: 5706.50 | bwd_inner_microstep: 5693.13 | bwd_allreduce_microstep: 13.31 | step_microstep: 19.07
[2025-04-25 16:14:06,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.31 | bwd: 5706.52 | bwd_inner: 5693.13 | bwd_allreduce: 13.33 | step: 19.07

  8%|▊         | 3431/41250 [8:16:31<91:01:54,  8.67s/it]
                                                         
{'loss': 0.3236, 'grad_norm': 2.0719833374023438, 'learning_rate': 3.970425104166381e-05, 'epoch': 0.83}

  8%|▊         | 3431/41250 [8:16:31<91:01:54,  8.67s/it][2025-04-25 16:14:15,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 16:14:15,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.95 | bwd_microstep: 5754.62 | bwd_inner_microstep: 5709.55 | bwd_allreduce_microstep: 45.02 | step_microstep: 18.87
[2025-04-25 16:14:15,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.95 | bwd: 5754.63 | bwd_inner: 5709.55 | bwd_allreduce: 45.04 | step: 18.87

  8%|▊         | 3432/41250 [8:16:40<91:05:58,  8.67s/it]
                                                         
{'loss': 0.0676, 'grad_norm': 0.8944680690765381, 'learning_rate': 3.9703981926757036e-05, 'epoch': 0.83}

  8%|▊         | 3432/41250 [8:16:40<91:05:58,  8.67s/it][2025-04-25 16:14:23,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 16:14:23,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.25 | bwd_microstep: 5707.29 | bwd_inner_microstep: 5694.33 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.97
[2025-04-25 16:14:23,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.25 | bwd: 5707.30 | bwd_inner: 5694.33 | bwd_allreduce: 12.92 | step: 18.98

  8%|▊         | 3433/41250 [8:16:49<91:02:02,  8.67s/it]
                                                         
{'loss': 0.2026, 'grad_norm': 1.472760796546936, 'learning_rate': 3.97037126903791e-05, 'epoch': 0.83}

  8%|▊         | 3433/41250 [8:16:49<91:02:02,  8.67s/it][2025-04-25 16:14:32,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 16:14:32,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.90 | bwd_microstep: 5721.50 | bwd_inner_microstep: 5667.42 | bwd_allreduce_microstep: 54.03 | step_microstep: 18.98
[2025-04-25 16:14:32,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.90 | bwd: 5721.51 | bwd_inner: 5667.42 | bwd_allreduce: 54.05 | step: 18.98

  8%|▊         | 3434/41250 [8:16:57<90:55:44,  8.66s/it]
                                                         
{'loss': 0.0907, 'grad_norm': 0.7766558527946472, 'learning_rate': 3.970344333253164e-05, 'epoch': 0.83}

  8%|▊         | 3434/41250 [8:16:57<90:55:44,  8.66s/it][2025-04-25 16:14:41,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:14:41,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.73 | bwd_microstep: 5941.68 | bwd_inner_microstep: 5665.66 | bwd_allreduce_microstep: 275.97 | step_microstep: 18.88
[2025-04-25 16:14:41,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.73 | bwd: 5941.70 | bwd_inner: 5665.66 | bwd_allreduce: 275.99 | step: 18.89

  8%|▊         | 3435/41250 [8:17:06<91:34:38,  8.72s/it]
                                                         
{'loss': 0.0753, 'grad_norm': 1.0875109434127808, 'learning_rate': 3.970317385321633e-05, 'epoch': 0.83}

  8%|▊         | 3435/41250 [8:17:06<91:34:38,  8.72s/it][2025-04-25 16:14:50,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:14:50,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.60 | bwd_microstep: 5757.13 | bwd_inner_microstep: 5723.49 | bwd_allreduce_microstep: 33.60 | step_microstep: 18.52
[2025-04-25 16:14:50,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.61 | bwd: 5757.15 | bwd_inner: 5723.49 | bwd_allreduce: 33.61 | step: 18.52

  8%|▊         | 3436/41250 [8:17:15<91:30:49,  8.71s/it]
                                                         
{'loss': 0.2037, 'grad_norm': 1.3120791912078857, 'learning_rate': 3.9702904252434826e-05, 'epoch': 0.83}

  8%|▊         | 3436/41250 [8:17:15<91:30:49,  8.71s/it][2025-04-25 16:14:58,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:14:58,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.64 | bwd_microstep: 5764.42 | bwd_inner_microstep: 5655.82 | bwd_allreduce_microstep: 108.56 | step_microstep: 18.74
[2025-04-25 16:14:58,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.64 | bwd: 5764.44 | bwd_inner: 5655.82 | bwd_allreduce: 108.58 | step: 18.74

  8%|▊         | 3437/41250 [8:17:24<91:24:21,  8.70s/it]
                                                         
{'loss': 0.1312, 'grad_norm': 1.4267781972885132, 'learning_rate': 3.9702634530188796e-05, 'epoch': 0.83}

  8%|▊         | 3437/41250 [8:17:24<91:24:21,  8.70s/it][2025-04-25 16:15:07,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:15:07,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.65 | bwd_microstep: 5768.57 | bwd_inner_microstep: 5649.49 | bwd_allreduce_microstep: 119.03 | step_microstep: 18.78
[2025-04-25 16:15:07,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.65 | bwd: 5768.59 | bwd_inner: 5649.49 | bwd_allreduce: 119.06 | step: 18.78

  8%|▊         | 3438/41250 [8:17:32<91:20:53,  8.70s/it]
                                                         
{'loss': 0.0654, 'grad_norm': 0.5708075165748596, 'learning_rate': 3.97023646864799e-05, 'epoch': 0.83}

  8%|▊         | 3438/41250 [8:17:32<91:20:53,  8.70s/it][2025-04-25 16:15:16,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:15:16,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.92 | bwd_microstep: 5751.88 | bwd_inner_microstep: 5712.12 | bwd_allreduce_microstep: 39.71 | step_microstep: 18.56
[2025-04-25 16:15:16,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.92 | bwd: 5751.89 | bwd_inner: 5712.13 | bwd_allreduce: 39.73 | step: 18.56

  8%|▊         | 3439/41250 [8:17:41<91:20:27,  8.70s/it]
                                                         
{'loss': 0.2192, 'grad_norm': 2.905273675918579, 'learning_rate': 3.97020947213098e-05, 'epoch': 0.83}

  8%|▊         | 3439/41250 [8:17:41<91:20:27,  8.70s/it][2025-04-25 16:15:24,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 16:15:24,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.39 | bwd_microstep: 5736.67 | bwd_inner_microstep: 5703.59 | bwd_allreduce_microstep: 33.03 | step_microstep: 18.87
[2025-04-25 16:15:24,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.39 | bwd: 5736.68 | bwd_inner: 5703.59 | bwd_allreduce: 33.05 | step: 18.87

  8%|▊         | 3440/41250 [8:17:50<91:15:43,  8.69s/it]
                                                         
{'loss': 0.3109, 'grad_norm': 2.7190451622009277, 'learning_rate': 3.970182463468017e-05, 'epoch': 0.83}

  8%|▊         | 3440/41250 [8:17:50<91:15:43,  8.69s/it][2025-04-25 16:15:33,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.23 | optimizer_step: 0.98
[2025-04-25 16:15:33,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.21 | bwd_microstep: 5750.19 | bwd_inner_microstep: 5654.60 | bwd_allreduce_microstep: 95.53 | step_microstep: 19.62
[2025-04-25 16:15:33,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.21 | bwd: 5750.20 | bwd_inner: 5654.60 | bwd_allreduce: 95.56 | step: 19.63

  8%|▊         | 3441/41250 [8:17:58<91:13:03,  8.69s/it]
                                                         
{'loss': 0.251, 'grad_norm': 2.4046812057495117, 'learning_rate': 3.9701554426592654e-05, 'epoch': 0.83}

  8%|▊         | 3441/41250 [8:17:58<91:13:03,  8.69s/it][2025-04-25 16:15:42,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 16:15:42,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5791.15 | bwd_inner_microstep: 5666.76 | bwd_allreduce_microstep: 124.35 | step_microstep: 18.52
[2025-04-25 16:15:42,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.74 | bwd: 5791.17 | bwd_inner: 5666.76 | bwd_allreduce: 124.37 | step: 18.52

  8%|▊         | 3442/41250 [8:18:07<91:16:03,  8.69s/it]
                                                         
{'loss': 0.3162, 'grad_norm': 1.600584864616394, 'learning_rate': 3.970128409704893e-05, 'epoch': 0.83}

  8%|▊         | 3442/41250 [8:18:07<91:16:03,  8.69s/it][2025-04-25 16:15:50,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-25 16:15:50,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.23 | bwd_microstep: 5777.00 | bwd_inner_microstep: 5655.91 | bwd_allreduce_microstep: 121.05 | step_microstep: 19.32
[2025-04-25 16:15:50,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.23 | bwd: 5777.02 | bwd_inner: 5655.91 | bwd_allreduce: 121.07 | step: 19.33

  8%|▊         | 3443/41250 [8:18:16<91:15:38,  8.69s/it]
                                                         
{'loss': 0.3092, 'grad_norm': 3.6039717197418213, 'learning_rate': 3.970101364605067e-05, 'epoch': 0.83}

  8%|▊         | 3443/41250 [8:18:16<91:15:38,  8.69s/it][2025-04-25 16:15:59,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 16:15:59,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.25 | bwd_microstep: 5691.98 | bwd_inner_microstep: 5664.39 | bwd_allreduce_microstep: 27.54 | step_microstep: 18.95
[2025-04-25 16:15:59,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.25 | bwd: 5692.00 | bwd_inner: 5664.39 | bwd_allreduce: 27.56 | step: 18.95

  8%|▊         | 3444/41250 [8:18:24<90:59:51,  8.67s/it]
                                                         
{'loss': 0.2387, 'grad_norm': 2.0821194648742676, 'learning_rate': 3.9700743073599535e-05, 'epoch': 0.83}

  8%|▊         | 3444/41250 [8:18:24<90:59:51,  8.67s/it][2025-04-25 16:16:08,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:16:08,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5772.81 | bwd_inner_microstep: 5707.15 | bwd_allreduce_microstep: 65.61 | step_microstep: 18.50
[2025-04-25 16:16:08,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5772.83 | bwd_inner: 5707.15 | bwd_allreduce: 65.63 | step: 18.50

  8%|▊         | 3445/41250 [8:18:33<91:07:32,  8.68s/it]
                                                         
{'loss': 0.1008, 'grad_norm': 1.4730921983718872, 'learning_rate': 3.9700472379697197e-05, 'epoch': 0.84}

  8%|▊         | 3445/41250 [8:18:33<91:07:32,  8.68s/it][2025-04-25 16:16:16,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:16:16,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.39 | bwd_microstep: 5720.26 | bwd_inner_microstep: 5707.57 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.83
[2025-04-25 16:16:16,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.39 | bwd: 5720.27 | bwd_inner: 5707.57 | bwd_allreduce: 12.66 | step: 18.83

  8%|▊         | 3446/41250 [8:18:42<91:05:46,  8.67s/it]
                                                         
{'loss': 0.2899, 'grad_norm': 1.70810067653656, 'learning_rate': 3.9700201564345316e-05, 'epoch': 0.84}

  8%|▊         | 3446/41250 [8:18:42<91:05:46,  8.67s/it][2025-04-25 16:16:25,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 16:16:25,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.27 | bwd_microstep: 5736.44 | bwd_inner_microstep: 5697.52 | bwd_allreduce_microstep: 38.87 | step_microstep: 19.31
[2025-04-25 16:16:25,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.27 | bwd: 5736.46 | bwd_inner: 5697.52 | bwd_allreduce: 38.89 | step: 19.31

  8%|▊         | 3447/41250 [8:18:50<91:04:27,  8.67s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 1.3912651538848877, 'learning_rate': 3.969993062754557e-05, 'epoch': 0.84}

  8%|▊         | 3447/41250 [8:18:50<91:04:27,  8.67s/it][2025-04-25 16:16:34,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:16:34,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.57 | bwd_microstep: 5762.32 | bwd_inner_microstep: 5705.96 | bwd_allreduce_microstep: 56.31 | step_microstep: 18.75
[2025-04-25 16:16:34,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.57 | bwd: 5762.33 | bwd_inner: 5705.96 | bwd_allreduce: 56.33 | step: 18.75

  8%|▊         | 3448/41250 [8:18:59<91:09:12,  8.68s/it]
                                                         
{'loss': 0.0941, 'grad_norm': 0.78697270154953, 'learning_rate': 3.969965956929962e-05, 'epoch': 0.84}

  8%|▊         | 3448/41250 [8:18:59<91:09:12,  8.68s/it][2025-04-25 16:16:42,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:16:42,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.40 | bwd_microstep: 5718.01 | bwd_inner_microstep: 5701.29 | bwd_allreduce_microstep: 16.67 | step_microstep: 18.55
[2025-04-25 16:16:42,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.40 | bwd: 5718.02 | bwd_inner: 5701.29 | bwd_allreduce: 16.69 | step: 18.55

  8%|▊         | 3449/41250 [8:19:08<91:03:16,  8.67s/it]
                                                         
{'loss': 0.5734, 'grad_norm': 2.1488966941833496, 'learning_rate': 3.969938838960915e-05, 'epoch': 0.84}

  8%|▊         | 3449/41250 [8:19:08<91:03:16,  8.67s/it][2025-04-25 16:16:51,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:16:51,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5742.74 | bwd_inner_microstep: 5703.78 | bwd_allreduce_microstep: 38.91 | step_microstep: 18.59
[2025-04-25 16:16:51,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.73 | bwd: 5742.75 | bwd_inner: 5703.78 | bwd_allreduce: 38.93 | step: 18.59

  8%|▊         | 3450/41250 [8:19:16<91:03:36,  8.67s/it]
                                                         
{'loss': 0.3581, 'grad_norm': 2.5138392448425293, 'learning_rate': 3.969911708847583e-05, 'epoch': 0.84}

  8%|▊         | 3450/41250 [8:19:16<91:03:36,  8.67s/it][2025-04-25 16:17:00,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:17:00,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5678.56 | bwd_inner_microstep: 5642.13 | bwd_allreduce_microstep: 36.38 | step_microstep: 18.97
[2025-04-25 16:17:00,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5678.58 | bwd_inner: 5642.13 | bwd_allreduce: 36.40 | step: 18.97

  8%|▊         | 3451/41250 [8:19:25<90:48:26,  8.65s/it]
                                                         
{'loss': 0.0659, 'grad_norm': 0.6974591016769409, 'learning_rate': 3.969884566590132e-05, 'epoch': 0.84}

  8%|▊         | 3451/41250 [8:19:25<90:48:26,  8.65s/it][2025-04-25 16:17:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 16:17:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.50 | bwd_microstep: 5743.08 | bwd_inner_microstep: 5690.47 | bwd_allreduce_microstep: 52.56 | step_microstep: 19.08
[2025-04-25 16:17:08,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.50 | bwd: 5743.10 | bwd_inner: 5690.47 | bwd_allreduce: 52.58 | step: 19.09

  8%|▊         | 3452/41250 [8:19:34<90:53:08,  8.66s/it]
                                                         
{'loss': 0.1178, 'grad_norm': 1.7671250104904175, 'learning_rate': 3.9698574121887304e-05, 'epoch': 0.84}

  8%|▊         | 3452/41250 [8:19:34<90:53:08,  8.66s/it][2025-04-25 16:17:17,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:17:17,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5716.16 | bwd_inner_microstep: 5703.15 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.51
[2025-04-25 16:17:17,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5716.17 | bwd_inner: 5703.15 | bwd_allreduce: 12.98 | step: 18.51

  8%|▊         | 3453/41250 [8:19:42<90:51:24,  8.65s/it]
                                                         
{'loss': 0.1592, 'grad_norm': 2.451308488845825, 'learning_rate': 3.969830245643545e-05, 'epoch': 0.84}

  8%|▊         | 3453/41250 [8:19:42<90:51:24,  8.65s/it][2025-04-25 16:17:26,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:17:26,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5762.93 | bwd_inner_microstep: 5703.16 | bwd_allreduce_microstep: 59.73 | step_microstep: 18.38
[2025-04-25 16:17:26,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5762.95 | bwd_inner: 5703.16 | bwd_allreduce: 59.75 | step: 18.38

  8%|▊         | 3454/41250 [8:19:51<90:59:10,  8.67s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 1.5425835847854614, 'learning_rate': 3.969803066954744e-05, 'epoch': 0.84}

  8%|▊         | 3454/41250 [8:19:51<90:59:10,  8.67s/it][2025-04-25 16:17:34,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:17:34,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5767.78 | bwd_inner_microstep: 5643.08 | bwd_allreduce_microstep: 124.66 | step_microstep: 18.13
[2025-04-25 16:17:34,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5767.79 | bwd_inner: 5643.08 | bwd_allreduce: 124.68 | step: 18.13

  8%|▊         | 3455/41250 [8:20:00<91:00:41,  8.67s/it]
                                                         
{'loss': 0.1496, 'grad_norm': 1.171652913093567, 'learning_rate': 3.969775876122494e-05, 'epoch': 0.84}

  8%|▊         | 3455/41250 [8:20:00<91:00:41,  8.67s/it][2025-04-25 16:17:43,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:17:43,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.27 | bwd_microstep: 5771.22 | bwd_inner_microstep: 5655.53 | bwd_allreduce_microstep: 115.64 | step_microstep: 18.20
[2025-04-25 16:17:43,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.27 | bwd: 5771.23 | bwd_inner: 5655.53 | bwd_allreduce: 115.66 | step: 18.21

  8%|▊         | 3456/41250 [8:20:08<91:02:44,  8.67s/it]
                                                         
{'loss': 0.1342, 'grad_norm': 1.8778384923934937, 'learning_rate': 3.969748673146964e-05, 'epoch': 0.84}

  8%|▊         | 3456/41250 [8:20:08<91:02:44,  8.67s/it][2025-04-25 16:17:52,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 16:17:52,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.96 | bwd_microstep: 5711.76 | bwd_inner_microstep: 5698.97 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.45
[2025-04-25 16:17:52,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.96 | bwd: 5711.77 | bwd_inner: 5698.97 | bwd_allreduce: 12.76 | step: 18.46

  8%|▊         | 3457/41250 [8:20:17<90:56:41,  8.66s/it]
                                                         
{'loss': 0.1727, 'grad_norm': 0.9244842529296875, 'learning_rate': 3.9697214580283205e-05, 'epoch': 0.84}

  8%|▊         | 3457/41250 [8:20:17<90:56:41,  8.66s/it][2025-04-25 16:18:00,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:18:00,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.10 | bwd_microstep: 5708.49 | bwd_inner_microstep: 5652.23 | bwd_allreduce_microstep: 56.21 | step_microstep: 18.59
[2025-04-25 16:18:00,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.10 | bwd: 5708.50 | bwd_inner: 5652.23 | bwd_allreduce: 56.23 | step: 18.59

  8%|▊         | 3458/41250 [8:20:26<90:46:55,  8.65s/it]
                                                         
{'loss': 0.2013, 'grad_norm': 1.15058171749115, 'learning_rate': 3.969694230766731e-05, 'epoch': 0.84}

  8%|▊         | 3458/41250 [8:20:26<90:46:55,  8.65s/it][2025-04-25 16:18:09,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:18:09,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.60 | bwd_microstep: 5719.48 | bwd_inner_microstep: 5676.98 | bwd_allreduce_microstep: 42.45 | step_microstep: 18.54
[2025-04-25 16:18:09,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.60 | bwd: 5719.49 | bwd_inner: 5676.98 | bwd_allreduce: 42.47 | step: 18.54

  8%|▊         | 3459/41250 [8:20:34<90:49:26,  8.65s/it]
                                                         
{'loss': 0.0766, 'grad_norm': 0.6886844038963318, 'learning_rate': 3.9696669913623646e-05, 'epoch': 0.84}

  8%|▊         | 3459/41250 [8:20:34<90:49:26,  8.65s/it][2025-04-25 16:18:17,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:18:17,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.01 | bwd_microstep: 5694.03 | bwd_inner_microstep: 5681.33 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.38
[2025-04-25 16:18:17,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.01 | bwd: 5694.04 | bwd_inner: 5681.33 | bwd_allreduce: 12.67 | step: 18.38

  8%|▊         | 3460/41250 [8:20:43<90:42:58,  8.64s/it]
                                                         
{'loss': 0.1367, 'grad_norm': 0.9624582529067993, 'learning_rate': 3.969639739815389e-05, 'epoch': 0.84}

  8%|▊         | 3460/41250 [8:20:43<90:42:58,  8.64s/it][2025-04-25 16:18:26,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 16:18:26,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.11 | bwd_microstep: 5751.04 | bwd_inner_microstep: 5680.98 | bwd_allreduce_microstep: 70.02 | step_microstep: 18.19
[2025-04-25 16:18:26,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.11 | bwd: 5751.06 | bwd_inner: 5680.98 | bwd_allreduce: 70.04 | step: 18.19

  8%|▊         | 3461/41250 [8:20:51<90:48:24,  8.65s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.2497998476028442, 'learning_rate': 3.969612476125971e-05, 'epoch': 0.84}

  8%|▊         | 3461/41250 [8:20:51<90:48:24,  8.65s/it][2025-04-25 16:18:35,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:18:35,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5685.12 | bwd_inner_microstep: 5672.37 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.30
[2025-04-25 16:18:35,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5685.13 | bwd_inner: 5672.37 | bwd_allreduce: 12.72 | step: 18.30

  8%|▊         | 3462/41250 [8:21:00<90:42:03,  8.64s/it]
                                                         
{'loss': 0.0631, 'grad_norm': 1.0256413221359253, 'learning_rate': 3.96958520029428e-05, 'epoch': 0.84}

  8%|▊         | 3462/41250 [8:21:00<90:42:03,  8.64s/it][2025-04-25 16:18:43,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:18:43,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.09 | bwd_microstep: 5706.56 | bwd_inner_microstep: 5689.50 | bwd_allreduce_microstep: 17.01 | step_microstep: 18.30
[2025-04-25 16:18:43,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.09 | bwd: 5706.57 | bwd_inner: 5689.50 | bwd_allreduce: 17.03 | step: 18.30

  8%|▊         | 3463/41250 [8:21:09<90:43:20,  8.64s/it]
                                                         
{'loss': 0.2827, 'grad_norm': 1.4786890745162964, 'learning_rate': 3.969557912320483e-05, 'epoch': 0.84}

  8%|▊         | 3463/41250 [8:21:09<90:43:20,  8.64s/it][2025-04-25 16:18:52,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:18:52,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.48 | bwd_microstep: 5717.15 | bwd_inner_microstep: 5704.25 | bwd_allreduce_microstep: 12.86 | step_microstep: 17.97
[2025-04-25 16:18:52,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.48 | bwd: 5717.17 | bwd_inner: 5704.25 | bwd_allreduce: 12.88 | step: 17.97

  8%|▊         | 3464/41250 [8:21:17<90:45:06,  8.65s/it]
                                                         
{'loss': 0.215, 'grad_norm': 1.8313120603561401, 'learning_rate': 3.969530612204749e-05, 'epoch': 0.84}

  8%|▊         | 3464/41250 [8:21:17<90:45:06,  8.65s/it][2025-04-25 16:19:01,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:19:01,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.70 | bwd_microstep: 5708.05 | bwd_inner_microstep: 5695.37 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.06
[2025-04-25 16:19:01,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.70 | bwd: 5708.06 | bwd_inner: 5695.37 | bwd_allreduce: 12.65 | step: 18.06

  8%|▊         | 3465/41250 [8:21:26<90:45:27,  8.65s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 1.5080691576004028, 'learning_rate': 3.969503299947246e-05, 'epoch': 0.84}

  8%|▊         | 3465/41250 [8:21:26<90:45:27,  8.65s/it][2025-04-25 16:19:09,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 16:19:09,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.83 | bwd_microstep: 5717.50 | bwd_inner_microstep: 5697.32 | bwd_allreduce_microstep: 20.14 | step_microstep: 18.09
[2025-04-25 16:19:09,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.84 | bwd: 5717.51 | bwd_inner: 5697.32 | bwd_allreduce: 20.16 | step: 18.09

  8%|▊         | 3466/41250 [8:21:35<90:48:30,  8.65s/it]
                                                         
{'loss': 0.1785, 'grad_norm': 1.1158908605575562, 'learning_rate': 3.969475975548144e-05, 'epoch': 0.84}

  8%|▊         | 3466/41250 [8:21:35<90:48:30,  8.65s/it][2025-04-25 16:19:18,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:19:18,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.58 | bwd_microstep: 5716.75 | bwd_inner_microstep: 5704.24 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.19
[2025-04-25 16:19:18,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.58 | bwd: 5716.76 | bwd_inner: 5704.24 | bwd_allreduce: 12.48 | step: 18.19

  8%|▊         | 3467/41250 [8:21:43<90:50:10,  8.65s/it]
                                                         
{'loss': 0.1268, 'grad_norm': 1.3738704919815063, 'learning_rate': 3.9694486390076086e-05, 'epoch': 0.84}

  8%|▊         | 3467/41250 [8:21:43<90:50:10,  8.65s/it][2025-04-25 16:19:27,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:19:27,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.23 | bwd_microstep: 5745.35 | bwd_inner_microstep: 5636.08 | bwd_allreduce_microstep: 109.22 | step_microstep: 18.41
[2025-04-25 16:19:27,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.23 | bwd: 5745.36 | bwd_inner: 5636.08 | bwd_allreduce: 109.24 | step: 18.41

  8%|▊         | 3468/41250 [8:21:52<90:49:02,  8.65s/it]
                                                         
{'loss': 0.0524, 'grad_norm': 0.6574631929397583, 'learning_rate': 3.96942129032581e-05, 'epoch': 0.84}

  8%|▊         | 3468/41250 [8:21:52<90:49:02,  8.65s/it][2025-04-25 16:19:35,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:19:35,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.27 | bwd_microstep: 5744.57 | bwd_inner_microstep: 5682.77 | bwd_allreduce_microstep: 61.76 | step_microstep: 18.15
[2025-04-25 16:19:35,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.27 | bwd: 5744.58 | bwd_inner: 5682.77 | bwd_allreduce: 61.78 | step: 18.16

  8%|▊         | 3469/41250 [8:22:01<90:51:20,  8.66s/it]
                                                         
{'loss': 0.1983, 'grad_norm': 1.2644233703613281, 'learning_rate': 3.969393929502917e-05, 'epoch': 0.84}

  8%|▊         | 3469/41250 [8:22:01<90:51:20,  8.66s/it][2025-04-25 16:19:44,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 16:19:44,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.14 | bwd_microstep: 5757.65 | bwd_inner_microstep: 5655.94 | bwd_allreduce_microstep: 101.67 | step_microstep: 18.73
[2025-04-25 16:19:44,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.14 | bwd: 5757.66 | bwd_inner: 5655.94 | bwd_allreduce: 101.68 | step: 18.74

  8%|▊         | 3470/41250 [8:22:09<90:54:43,  8.66s/it]
                                                         
{'loss': 0.1758, 'grad_norm': 1.4454951286315918, 'learning_rate': 3.969366556539097e-05, 'epoch': 0.84}

  8%|▊         | 3470/41250 [8:22:09<90:54:43,  8.66s/it][2025-04-25 16:19:53,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:19:53,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.54 | bwd_microstep: 5688.16 | bwd_inner_microstep: 5654.44 | bwd_allreduce_microstep: 33.68 | step_microstep: 18.87
[2025-04-25 16:19:53,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.54 | bwd: 5688.18 | bwd_inner: 5654.44 | bwd_allreduce: 33.70 | step: 18.87

  8%|▊         | 3471/41250 [8:22:18<90:43:12,  8.64s/it]
                                                         
{'loss': 0.23, 'grad_norm': 1.599871039390564, 'learning_rate': 3.96933917143452e-05, 'epoch': 0.84}

  8%|▊         | 3471/41250 [8:22:18<90:43:12,  8.64s/it][2025-04-25 16:20:01,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.10 | optimizer_step: 1.20
[2025-04-25 16:20:01,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.07 | bwd_microstep: 5710.38 | bwd_inner_microstep: 5696.66 | bwd_allreduce_microstep: 13.66 | step_microstep: 19.38
[2025-04-25 16:20:01,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.07 | bwd: 5710.39 | bwd_inner: 5696.66 | bwd_allreduce: 13.69 | step: 19.38

  8%|▊         | 3472/41250 [8:22:27<90:42:57,  8.64s/it]
                                                         
{'loss': 0.0793, 'grad_norm': 1.118802547454834, 'learning_rate': 3.969311774189354e-05, 'epoch': 0.84}

  8%|▊         | 3472/41250 [8:22:27<90:42:57,  8.64s/it][2025-04-25 16:20:10,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:20:10,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5736.97 | bwd_inner_microstep: 5707.86 | bwd_allreduce_microstep: 29.07 | step_microstep: 18.21
[2025-04-25 16:20:10,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5736.98 | bwd_inner: 5707.85 | bwd_allreduce: 29.09 | step: 18.21

  8%|▊         | 3473/41250 [8:22:35<90:47:53,  8.65s/it]
                                                         
{'loss': 0.269, 'grad_norm': 1.9373844861984253, 'learning_rate': 3.9692843648037686e-05, 'epoch': 0.84}

  8%|▊         | 3473/41250 [8:22:35<90:47:53,  8.65s/it][2025-04-25 16:20:19,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:20:19,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.33 | bwd_microstep: 5738.54 | bwd_inner_microstep: 5701.08 | bwd_allreduce_microstep: 37.41 | step_microstep: 18.80
[2025-04-25 16:20:19,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.33 | bwd: 5738.55 | bwd_inner: 5701.08 | bwd_allreduce: 37.43 | step: 18.80

  8%|▊         | 3474/41250 [8:22:44<90:52:22,  8.66s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 2.0521109104156494, 'learning_rate': 3.969256943277932e-05, 'epoch': 0.84}

  8%|▊         | 3474/41250 [8:22:44<90:52:22,  8.66s/it][2025-04-25 16:20:27,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 16:20:27,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.93 | bwd_microstep: 5704.68 | bwd_inner_microstep: 5691.93 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.74
[2025-04-25 16:20:27,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.93 | bwd: 5704.69 | bwd_inner: 5691.93 | bwd_allreduce: 12.72 | step: 18.74

  8%|▊         | 3475/41250 [8:22:53<90:48:09,  8.65s/it]
                                                         
{'loss': 0.2484, 'grad_norm': 2.140932321548462, 'learning_rate': 3.9692295096120146e-05, 'epoch': 0.84}

  8%|▊         | 3475/41250 [8:22:53<90:48:09,  8.65s/it][2025-04-25 16:20:36,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:20:36,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.83 | bwd_microstep: 5706.46 | bwd_inner_microstep: 5693.72 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.46
[2025-04-25 16:20:36,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.83 | bwd: 5706.47 | bwd_inner: 5693.72 | bwd_allreduce: 12.71 | step: 18.46

  8%|▊         | 3476/41250 [8:23:01<90:43:39,  8.65s/it]
                                                         
{'loss': 0.2043, 'grad_norm': 2.4486660957336426, 'learning_rate': 3.969202063806184e-05, 'epoch': 0.84}

  8%|▊         | 3476/41250 [8:23:01<90:43:39,  8.65s/it][2025-04-25 16:20:45,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:20:45,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.15 | bwd_microstep: 5781.64 | bwd_inner_microstep: 5657.30 | bwd_allreduce_microstep: 124.29 | step_microstep: 18.55
[2025-04-25 16:20:45,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.15 | bwd: 5781.66 | bwd_inner: 5657.30 | bwd_allreduce: 124.31 | step: 18.55

  8%|▊         | 3477/41250 [8:23:10<90:53:52,  8.66s/it]
                                                         
{'loss': 0.0706, 'grad_norm': 0.6393564939498901, 'learning_rate': 3.96917460586061e-05, 'epoch': 0.84}

  8%|▊         | 3477/41250 [8:23:10<90:53:52,  8.66s/it][2025-04-25 16:20:53,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:20:53,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5769.90 | bwd_inner_microstep: 5685.90 | bwd_allreduce_microstep: 83.96 | step_microstep: 18.68
[2025-04-25 16:20:53,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5769.92 | bwd_inner: 5685.90 | bwd_allreduce: 83.98 | step: 18.68

  8%|▊         | 3478/41250 [8:23:19<90:59:50,  8.67s/it]
                                                         
{'loss': 0.1049, 'grad_norm': 1.0164819955825806, 'learning_rate': 3.9691471357754616e-05, 'epoch': 0.84}

  8%|▊         | 3478/41250 [8:23:19<90:59:50,  8.67s/it][2025-04-25 16:21:02,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:21:02,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.46 | bwd_microstep: 5776.36 | bwd_inner_microstep: 5649.75 | bwd_allreduce_microstep: 126.56 | step_microstep: 18.68
[2025-04-25 16:21:02,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.46 | bwd: 5776.38 | bwd_inner: 5649.75 | bwd_allreduce: 126.58 | step: 18.68

  8%|▊         | 3479/41250 [8:23:27<91:03:22,  8.68s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 0.8202164173126221, 'learning_rate': 3.969119653550909e-05, 'epoch': 0.84}

  8%|▊         | 3479/41250 [8:23:27<91:03:22,  8.68s/it][2025-04-25 16:21:11,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 16:21:11,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.55 | bwd_microstep: 5777.90 | bwd_inner_microstep: 5765.25 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.45
[2025-04-25 16:21:11,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.55 | bwd: 5777.91 | bwd_inner: 5765.25 | bwd_allreduce: 12.62 | step: 18.46

  8%|▊         | 3480/41250 [8:23:36<91:15:20,  8.70s/it]
                                                         
{'loss': 0.2771, 'grad_norm': 1.863684058189392, 'learning_rate': 3.9690921591871205e-05, 'epoch': 0.84}

  8%|▊         | 3480/41250 [8:23:36<91:15:20,  8.70s/it][2025-04-25 16:21:19,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:21:19,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5754.14 | bwd_inner_microstep: 5711.81 | bwd_allreduce_microstep: 42.28 | step_microstep: 18.72
[2025-04-25 16:21:19,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5754.15 | bwd_inner: 5711.81 | bwd_allreduce: 42.30 | step: 18.73

  8%|▊         | 3481/41250 [8:23:45<91:13:55,  8.70s/it]
                                                         
{'loss': 0.0334, 'grad_norm': 0.347367525100708, 'learning_rate': 3.9690646526842663e-05, 'epoch': 0.84}

  8%|▊         | 3481/41250 [8:23:45<91:13:55,  8.70s/it][2025-04-25 16:21:28,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-25 16:21:28,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.30 | bwd_microstep: 5743.23 | bwd_inner_microstep: 5711.64 | bwd_allreduce_microstep: 31.55 | step_microstep: 18.22
[2025-04-25 16:21:28,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.30 | bwd: 5743.25 | bwd_inner: 5711.64 | bwd_allreduce: 31.56 | step: 18.22

  8%|▊         | 3482/41250 [8:23:53<91:12:41,  8.69s/it]
                                                         
{'loss': 0.1494, 'grad_norm': 2.1839113235473633, 'learning_rate': 3.969037134042516e-05, 'epoch': 0.84}

  8%|▊         | 3482/41250 [8:23:53<91:12:41,  8.69s/it][2025-04-25 16:21:37,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 16:21:37,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.45 | bwd_microstep: 5775.96 | bwd_inner_microstep: 5662.30 | bwd_allreduce_microstep: 113.61 | step_microstep: 19.26
[2025-04-25 16:21:37,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.45 | bwd: 5775.98 | bwd_inner: 5662.30 | bwd_allreduce: 113.64 | step: 19.26

  8%|▊         | 3483/41250 [8:24:02<91:12:47,  8.69s/it]
                                                         
{'loss': 0.1906, 'grad_norm': 1.0915569067001343, 'learning_rate': 3.969009603262039e-05, 'epoch': 0.84}

  8%|▊         | 3483/41250 [8:24:02<91:12:47,  8.69s/it][2025-04-25 16:21:46,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-25 16:21:46,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.52 | bwd_microstep: 5805.49 | bwd_inner_microstep: 5666.81 | bwd_allreduce_microstep: 138.62 | step_microstep: 19.21
[2025-04-25 16:21:46,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.52 | bwd: 5805.50 | bwd_inner: 5666.81 | bwd_allreduce: 138.65 | step: 19.21

  8%|▊         | 3484/41250 [8:24:11<91:17:51,  8.70s/it]
                                                         
{'loss': 0.3007, 'grad_norm': 2.4077706336975098, 'learning_rate': 3.968982060343005e-05, 'epoch': 0.84}

  8%|▊         | 3484/41250 [8:24:11<91:17:51,  8.70s/it][2025-04-25 16:21:54,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:21:54,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.21 | bwd_microstep: 5740.30 | bwd_inner_microstep: 5725.67 | bwd_allreduce_microstep: 14.59 | step_microstep: 18.97
[2025-04-25 16:21:54,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.21 | bwd: 5740.32 | bwd_inner: 5725.67 | bwd_allreduce: 14.61 | step: 18.97

  8%|▊         | 3485/41250 [8:24:20<91:15:18,  8.70s/it]
                                                         
{'loss': 0.2419, 'grad_norm': 3.8589491844177246, 'learning_rate': 3.9689545052855835e-05, 'epoch': 0.84}

  8%|▊         | 3485/41250 [8:24:20<91:15:18,  8.70s/it][2025-04-25 16:22:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:22:03,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5746.54 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 44.56 | step_microstep: 18.54
[2025-04-25 16:22:03,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5746.56 | bwd_inner: 5701.94 | bwd_allreduce: 44.58 | step: 18.54

  8%|▊         | 3486/41250 [8:24:28<91:11:17,  8.69s/it]
                                                         
{'loss': 0.1345, 'grad_norm': 1.1305524110794067, 'learning_rate': 3.9689269380899456e-05, 'epoch': 0.85}

  8%|▊         | 3486/41250 [8:24:28<91:11:17,  8.69s/it][2025-04-25 16:22:12,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 16:22:12,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5716.58 | bwd_inner_microstep: 5650.86 | bwd_allreduce_microstep: 65.68 | step_microstep: 18.73
[2025-04-25 16:22:12,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5716.60 | bwd_inner: 5650.86 | bwd_allreduce: 65.70 | step: 18.73

  8%|▊         | 3487/41250 [8:24:37<90:57:53,  8.67s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 3.3610873222351074, 'learning_rate': 3.96889935875626e-05, 'epoch': 0.85}

  8%|▊         | 3487/41250 [8:24:37<90:57:53,  8.67s/it][2025-04-25 16:22:20,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[2025-04-25 16:22:20,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.51 | bwd_microstep: 5744.30 | bwd_inner_microstep: 5713.40 | bwd_allreduce_microstep: 30.85 | step_microstep: 18.74
[2025-04-25 16:22:20,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.51 | bwd: 5744.31 | bwd_inner: 5713.40 | bwd_allreduce: 30.87 | step: 18.73

  8%|▊         | 3488/41250 [8:24:46<90:58:42,  8.67s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 1.9151489734649658, 'learning_rate': 3.968871767284697e-05, 'epoch': 0.85}

  8%|▊         | 3488/41250 [8:24:46<90:58:42,  8.67s/it][2025-04-25 16:22:29,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-25 16:22:29,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.18 | bwd_microstep: 5796.53 | bwd_inner_microstep: 5663.88 | bwd_allreduce_microstep: 132.60 | step_microstep: 19.07
[2025-04-25 16:22:29,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.18 | bwd: 5796.55 | bwd_inner: 5663.88 | bwd_allreduce: 132.62 | step: 19.07

  8%|▊         | 3489/41250 [8:24:54<91:07:10,  8.69s/it]
                                                         
{'loss': 0.1762, 'grad_norm': 1.9332938194274902, 'learning_rate': 3.968844163675426e-05, 'epoch': 0.85}

  8%|▊         | 3489/41250 [8:24:54<91:07:10,  8.69s/it][2025-04-25 16:22:38,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:22:38,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5716.24 | bwd_inner_microstep: 5703.52 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.21
[2025-04-25 16:22:38,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5716.26 | bwd_inner: 5703.52 | bwd_allreduce: 12.69 | step: 18.21

  8%|▊         | 3490/41250 [8:25:03<91:01:19,  8.68s/it]
                                                         
{'loss': 0.0369, 'grad_norm': 0.6040617823600769, 'learning_rate': 3.9688165479286195e-05, 'epoch': 0.85}

  8%|▊         | 3490/41250 [8:25:03<91:01:19,  8.68s/it][2025-04-25 16:22:46,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:22:46,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5698.80 | bwd_inner_microstep: 5671.55 | bwd_allreduce_microstep: 27.20 | step_microstep: 17.77
[2025-04-25 16:22:46,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5698.81 | bwd_inner: 5671.55 | bwd_allreduce: 27.22 | step: 17.78

  8%|▊         | 3491/41250 [8:25:12<90:49:20,  8.66s/it]
                                                         
{'loss': 0.2678, 'grad_norm': 1.5032461881637573, 'learning_rate': 3.968788920044445e-05, 'epoch': 0.85}

  8%|▊         | 3491/41250 [8:25:12<90:49:20,  8.66s/it][2025-04-25 16:22:55,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.18
[2025-04-25 16:22:55,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.56 | bwd_microstep: 5723.83 | bwd_inner_microstep: 5711.20 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.03
[2025-04-25 16:22:55,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.56 | bwd: 5723.85 | bwd_inner: 5711.20 | bwd_allreduce: 12.61 | step: 19.03

  8%|▊         | 3492/41250 [8:25:20<90:51:24,  8.66s/it]
                                                         
{'loss': 0.1447, 'grad_norm': 1.8115060329437256, 'learning_rate': 3.968761280023075e-05, 'epoch': 0.85}

  8%|▊         | 3492/41250 [8:25:20<90:51:24,  8.66s/it][2025-04-25 16:23:03,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:23:03,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.08 | bwd_microstep: 5703.78 | bwd_inner_microstep: 5660.31 | bwd_allreduce_microstep: 43.44 | step_microstep: 18.30
[2025-04-25 16:23:03,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.08 | bwd: 5703.80 | bwd_inner: 5660.30 | bwd_allreduce: 43.45 | step: 18.30

  8%|▊         | 3493/41250 [8:25:29<90:43:27,  8.65s/it]
                                                         
{'loss': 0.3813, 'grad_norm': 2.0150623321533203, 'learning_rate': 3.968733627864679e-05, 'epoch': 0.85}

  8%|▊         | 3493/41250 [8:25:29<90:43:27,  8.65s/it][2025-04-25 16:23:12,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:23:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.85 | bwd_microstep: 5701.21 | bwd_inner_microstep: 5657.10 | bwd_allreduce_microstep: 44.07 | step_microstep: 18.41
[2025-04-25 16:23:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.85 | bwd: 5701.22 | bwd_inner: 5657.10 | bwd_allreduce: 44.09 | step: 18.42

  8%|▊         | 3494/41250 [8:25:37<90:36:59,  8.64s/it]
                                                         
{'loss': 0.0923, 'grad_norm': 0.9446126222610474, 'learning_rate': 3.9687059635694266e-05, 'epoch': 0.85}

  8%|▊         | 3494/41250 [8:25:37<90:36:59,  8.64s/it][2025-04-25 16:23:21,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:23:21,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.57 | bwd_microstep: 5720.56 | bwd_inner_microstep: 5704.51 | bwd_allreduce_microstep: 16.01 | step_microstep: 18.30
[2025-04-25 16:23:21,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.57 | bwd: 5720.57 | bwd_inner: 5704.51 | bwd_allreduce: 16.02 | step: 18.31

  8%|▊         | 3495/41250 [8:25:46<90:42:42,  8.65s/it]
                                                         
{'loss': 0.2909, 'grad_norm': 1.3840045928955078, 'learning_rate': 3.96867828713749e-05, 'epoch': 0.85}

  8%|▊         | 3495/41250 [8:25:46<90:42:42,  8.65s/it][2025-04-25 16:23:29,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:23:29,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.43 | bwd_microstep: 5717.99 | bwd_inner_microstep: 5705.21 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.27
[2025-04-25 16:23:29,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.43 | bwd: 5718.00 | bwd_inner: 5705.21 | bwd_allreduce: 12.75 | step: 18.27

  8%|▊         | 3496/41250 [8:25:55<90:43:47,  8.65s/it]
                                                         
{'loss': 0.2097, 'grad_norm': 1.4475072622299194, 'learning_rate': 3.968650598569038e-05, 'epoch': 0.85}

  8%|▊         | 3496/41250 [8:25:55<90:43:47,  8.65s/it][2025-04-25 16:23:38,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:23:38,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.81 | bwd_microstep: 5876.88 | bwd_inner_microstep: 5676.18 | bwd_allreduce_microstep: 200.66 | step_microstep: 18.97
[2025-04-25 16:23:38,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.81 | bwd: 5876.90 | bwd_inner: 5676.18 | bwd_allreduce: 200.68 | step: 18.98

  8%|▊         | 3497/41250 [8:26:04<91:09:45,  8.69s/it]
                                                         
{'loss': 0.102, 'grad_norm': 1.134477138519287, 'learning_rate': 3.968622897864243e-05, 'epoch': 0.85}

  8%|▊         | 3497/41250 [8:26:04<91:09:45,  8.69s/it][2025-04-25 16:23:47,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:23:47,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.30 | bwd_microstep: 5742.29 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 43.80 | step_microstep: 18.77
[2025-04-25 16:23:47,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.30 | bwd: 5742.30 | bwd_inner: 5698.45 | bwd_allreduce: 43.81 | step: 18.77

  8%|▊         | 3498/41250 [8:26:12<91:07:11,  8.69s/it]
                                                         
{'loss': 0.2151, 'grad_norm': 1.6596734523773193, 'learning_rate': 3.9685951850232745e-05, 'epoch': 0.85}

  8%|▊         | 3498/41250 [8:26:12<91:07:11,  8.69s/it][2025-04-25 16:23:56,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:23:56,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.70 | bwd_microstep: 5750.24 | bwd_inner_microstep: 5692.04 | bwd_allreduce_microstep: 58.16 | step_microstep: 18.46
[2025-04-25 16:23:56,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.70 | bwd: 5750.26 | bwd_inner: 5692.04 | bwd_allreduce: 58.17 | step: 18.47

  8%|▊         | 3499/41250 [8:26:21<91:06:13,  8.69s/it]
                                                         
{'loss': 0.264, 'grad_norm': 1.2154009342193604, 'learning_rate': 3.968567460046304e-05, 'epoch': 0.85}

  8%|▊         | 3499/41250 [8:26:21<91:06:13,  8.69s/it][2025-04-25 16:24:04,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-25 16:24:04,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.85 | bwd_microstep: 5796.08 | bwd_inner_microstep: 5664.60 | bwd_allreduce_microstep: 131.43 | step_microstep: 19.09
[2025-04-25 16:24:04,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.85 | bwd: 5796.10 | bwd_inner: 5664.60 | bwd_allreduce: 131.45 | step: 19.09

  8%|▊         | 3500/41250 [8:26:30<91:09:43,  8.69s/it]
                                                         
{'loss': 0.0391, 'grad_norm': 0.7850825190544128, 'learning_rate': 3.968539722933503e-05, 'epoch': 0.85}

  8%|▊         | 3500/41250 [8:26:30<91:09:43,  8.69s/it][2025-04-25 16:24:13,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.24 | optimizer_step: 0.97
[2025-04-25 16:24:13,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.31 | bwd_microstep: 5716.89 | bwd_inner_microstep: 5649.26 | bwd_allreduce_microstep: 67.57 | step_microstep: 19.31
[2025-04-25 16:24:13,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.31 | bwd: 5716.90 | bwd_inner: 5649.26 | bwd_allreduce: 67.60 | step: 19.32

  8%|▊         | 3501/41250 [8:26:38<90:56:28,  8.67s/it]
                                                         
{'loss': 0.3257, 'grad_norm': 3.6908481121063232, 'learning_rate': 3.968511973685041e-05, 'epoch': 0.85}

  8%|▊         | 3501/41250 [8:26:38<90:56:28,  8.67s/it][2025-04-25 16:24:22,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.13 | optimizer_step: 0.98
[2025-04-25 16:24:22,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.27 | bwd_microstep: 5743.48 | bwd_inner_microstep: 5693.70 | bwd_allreduce_microstep: 49.73 | step_microstep: 19.06
[2025-04-25 16:24:22,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.27 | bwd: 5743.49 | bwd_inner: 5693.70 | bwd_allreduce: 49.75 | step: 19.06

  8%|▊         | 3502/41250 [8:26:47<90:57:23,  8.67s/it]
                                                         
{'loss': 0.2246, 'grad_norm': 2.4390816688537598, 'learning_rate': 3.96848421230109e-05, 'epoch': 0.85}

  8%|▊         | 3502/41250 [8:26:47<90:57:23,  8.67s/it][2025-04-25 16:24:30,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:24:30,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5785.20 | bwd_inner_microstep: 5647.29 | bwd_allreduce_microstep: 137.86 | step_microstep: 18.58
[2025-04-25 16:24:30,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5785.21 | bwd_inner: 5647.29 | bwd_allreduce: 137.88 | step: 18.58

  8%|▊         | 3503/41250 [8:26:56<91:00:19,  8.68s/it]
                                                         
{'loss': 0.099, 'grad_norm': 1.7331732511520386, 'learning_rate': 3.9684564387818213e-05, 'epoch': 0.85}

  8%|▊         | 3503/41250 [8:26:56<91:00:19,  8.68s/it][2025-04-25 16:24:39,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 16:24:39,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5699.25 | bwd_inner_microstep: 5686.68 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.84
[2025-04-25 16:24:39,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5699.26 | bwd_inner: 5686.68 | bwd_allreduce: 12.54 | step: 18.84

  8%|▊         | 3504/41250 [8:27:04<90:50:13,  8.66s/it]
                                                         
{'loss': 0.0571, 'grad_norm': 0.43644633889198303, 'learning_rate': 3.9684286531274064e-05, 'epoch': 0.85}

  8%|▊         | 3504/41250 [8:27:04<90:50:13,  8.66s/it][2025-04-25 16:24:48,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 16:24:48,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.81 | bwd_microstep: 5749.80 | bwd_inner_microstep: 5657.02 | bwd_allreduce_microstep: 92.73 | step_microstep: 18.45
[2025-04-25 16:24:48,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.81 | bwd: 5749.81 | bwd_inner: 5657.02 | bwd_allreduce: 92.75 | step: 18.45

  8%|▊         | 3505/41250 [8:27:13<90:48:56,  8.66s/it]
                                                         
{'loss': 0.1121, 'grad_norm': 0.9467082023620605, 'learning_rate': 3.9684008553380147e-05, 'epoch': 0.85}

  8%|▊         | 3505/41250 [8:27:13<90:48:56,  8.66s/it][2025-04-25 16:24:56,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:24:56,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.82 | bwd_microstep: 5754.74 | bwd_inner_microstep: 5690.54 | bwd_allreduce_microstep: 64.15 | step_microstep: 18.64
[2025-04-25 16:24:56,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.82 | bwd: 5754.75 | bwd_inner: 5690.54 | bwd_allreduce: 64.17 | step: 18.64

  8%|▊         | 3506/41250 [8:27:22<90:52:43,  8.67s/it]
                                                         
{'loss': 0.2941, 'grad_norm': 2.5351502895355225, 'learning_rate': 3.9683730454138195e-05, 'epoch': 0.85}

  8%|▊         | 3506/41250 [8:27:22<90:52:43,  8.67s/it][2025-04-25 16:25:05,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 16:25:05,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.27 | bwd_microstep: 5737.11 | bwd_inner_microstep: 5685.74 | bwd_allreduce_microstep: 51.33 | step_microstep: 18.54
[2025-04-25 16:25:05,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.27 | bwd: 5737.13 | bwd_inner: 5685.74 | bwd_allreduce: 51.34 | step: 18.54

  9%|▊         | 3507/41250 [8:27:30<90:52:28,  8.67s/it]
                                                         
{'loss': 0.2284, 'grad_norm': 2.330214500427246, 'learning_rate': 3.968345223354992e-05, 'epoch': 0.85}

  9%|▊         | 3507/41250 [8:27:30<90:52:28,  8.67s/it][2025-04-25 16:25:14,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:25:14,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.42 | bwd_microstep: 5719.56 | bwd_inner_microstep: 5706.79 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.31
[2025-04-25 16:25:14,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.43 | bwd: 5719.57 | bwd_inner: 5706.79 | bwd_allreduce: 12.74 | step: 18.31

  9%|▊         | 3508/41250 [8:27:39<90:50:08,  8.66s/it]
                                                         
{'loss': 0.1637, 'grad_norm': 1.2169142961502075, 'learning_rate': 3.9683173891617034e-05, 'epoch': 0.85}

  9%|▊         | 3508/41250 [8:27:39<90:50:08,  8.66s/it][2025-04-25 16:25:22,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 16:25:22,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.05 | bwd_microstep: 5694.80 | bwd_inner_microstep: 5682.08 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.99
[2025-04-25 16:25:22,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.06 | bwd: 5694.81 | bwd_inner: 5682.08 | bwd_allreduce: 12.69 | step: 19.00

  9%|▊         | 3509/41250 [8:27:48<90:41:53,  8.65s/it]
                                                         
{'loss': 0.1345, 'grad_norm': 0.5891111493110657, 'learning_rate': 3.968289542834124e-05, 'epoch': 0.85}

  9%|▊         | 3509/41250 [8:27:48<90:41:53,  8.65s/it][2025-04-25 16:25:31,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 16:25:31,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.60 | bwd_microstep: 5689.75 | bwd_inner_microstep: 5677.12 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.24
[2025-04-25 16:25:31,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.60 | bwd: 5689.76 | bwd_inner: 5677.12 | bwd_allreduce: 12.61 | step: 18.25

  9%|▊         | 3510/41250 [8:27:56<90:36:09,  8.64s/it]
                                                         
{'loss': 0.2265, 'grad_norm': 1.3214983940124512, 'learning_rate': 3.9682616843724276e-05, 'epoch': 0.85}

  9%|▊         | 3510/41250 [8:27:56<90:36:09,  8.64s/it][2025-04-25 16:25:39,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:25:39,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.52 | bwd_microstep: 5711.67 | bwd_inner_microstep: 5699.02 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.08
[2025-04-25 16:25:39,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.52 | bwd: 5711.68 | bwd_inner: 5699.02 | bwd_allreduce: 12.63 | step: 18.09

  9%|▊         | 3511/41250 [8:28:05<90:38:56,  8.65s/it]
                                                         
{'loss': 0.238, 'grad_norm': 1.714414358139038, 'learning_rate': 3.968233813776785e-05, 'epoch': 0.85}

  9%|▊         | 3511/41250 [8:28:05<90:38:56,  8.65s/it][2025-04-25 16:25:48,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:25:48,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.51 | bwd_microstep: 5708.38 | bwd_inner_microstep: 5634.87 | bwd_allreduce_microstep: 73.47 | step_microstep: 18.30
[2025-04-25 16:25:48,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.51 | bwd: 5708.39 | bwd_inner: 5634.87 | bwd_allreduce: 73.48 | step: 18.30

  9%|▊         | 3512/41250 [8:28:13<90:33:34,  8.64s/it]
                                                         
{'loss': 0.4273, 'grad_norm': 2.5617542266845703, 'learning_rate': 3.9682059310473684e-05, 'epoch': 0.85}

  9%|▊         | 3512/41250 [8:28:13<90:33:34,  8.64s/it][2025-04-25 16:25:57,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 16:25:57,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.49 | bwd_microstep: 5681.99 | bwd_inner_microstep: 5646.33 | bwd_allreduce_microstep: 35.61 | step_microstep: 18.71
[2025-04-25 16:25:57,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.49 | bwd: 5682.00 | bwd_inner: 5646.33 | bwd_allreduce: 35.64 | step: 18.72

  9%|▊         | 3513/41250 [8:28:22<90:23:14,  8.62s/it]
                                                         
{'loss': 0.0956, 'grad_norm': 1.3560142517089844, 'learning_rate': 3.968178036184349e-05, 'epoch': 0.85}

  9%|▊         | 3513/41250 [8:28:22<90:23:14,  8.62s/it][2025-04-25 16:26:05,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 16:26:05,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5754.92 | bwd_inner_microstep: 5660.46 | bwd_allreduce_microstep: 94.41 | step_microstep: 18.68
[2025-04-25 16:26:05,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5754.93 | bwd_inner: 5660.46 | bwd_allreduce: 94.43 | step: 18.69

  9%|▊         | 3514/41250 [8:28:31<90:31:39,  8.64s/it]
                                                         
{'loss': 0.1318, 'grad_norm': 2.489537000656128, 'learning_rate': 3.968150129187899e-05, 'epoch': 0.85}

  9%|▊         | 3514/41250 [8:28:31<90:31:39,  8.64s/it][2025-04-25 16:26:14,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:26:14,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.91 | bwd_microstep: 5710.06 | bwd_inner_microstep: 5697.42 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.93
[2025-04-25 16:26:14,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.91 | bwd: 5710.07 | bwd_inner: 5697.42 | bwd_allreduce: 12.61 | step: 17.93

  9%|▊         | 3515/41250 [8:28:39<90:34:39,  8.64s/it]
                                                         
{'loss': 0.1762, 'grad_norm': 0.971368134021759, 'learning_rate': 3.9681222100581905e-05, 'epoch': 0.85}

  9%|▊         | 3515/41250 [8:28:39<90:34:39,  8.64s/it][2025-04-25 16:26:23,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:26:23,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5668.23 | bwd_inner_microstep: 5642.21 | bwd_allreduce_microstep: 25.98 | step_microstep: 18.42
[2025-04-25 16:26:23,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.83 | bwd: 5668.24 | bwd_inner: 5642.21 | bwd_allreduce: 26.00 | step: 18.42

  9%|▊         | 3516/41250 [8:28:48<90:23:18,  8.62s/it]
                                                         
{'loss': 0.1041, 'grad_norm': 1.9212381839752197, 'learning_rate': 3.968094278795396e-05, 'epoch': 0.85}

  9%|▊         | 3516/41250 [8:28:48<90:23:18,  8.62s/it][2025-04-25 16:26:31,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:26:31,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.95 | bwd_microstep: 5724.21 | bwd_inner_microstep: 5711.54 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.40
[2025-04-25 16:26:31,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.95 | bwd: 5724.22 | bwd_inner: 5711.54 | bwd_allreduce: 12.64 | step: 18.41

  9%|▊         | 3517/41250 [8:28:57<90:30:47,  8.64s/it]
                                                         
{'loss': 0.4127, 'grad_norm': 3.044206142425537, 'learning_rate': 3.9680663353996875e-05, 'epoch': 0.85}

  9%|▊         | 3517/41250 [8:28:57<90:30:47,  8.64s/it][2025-04-25 16:26:40,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:26:40,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.48 | bwd_microstep: 5721.48 | bwd_inner_microstep: 5642.98 | bwd_allreduce_microstep: 78.44 | step_microstep: 18.73
[2025-04-25 16:26:40,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.48 | bwd: 5721.50 | bwd_inner: 5642.98 | bwd_allreduce: 78.46 | step: 18.73

  9%|▊         | 3518/41250 [8:29:05<90:33:15,  8.64s/it]
                                                         
{'loss': 0.195, 'grad_norm': 1.777541160583496, 'learning_rate': 3.968038379871237e-05, 'epoch': 0.85}

  9%|▊         | 3518/41250 [8:29:05<90:33:15,  8.64s/it][2025-04-25 16:26:49,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:26:49,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.75 | bwd_microstep: 5836.61 | bwd_inner_microstep: 5691.84 | bwd_allreduce_microstep: 144.73 | step_microstep: 18.04
[2025-04-25 16:26:49,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.75 | bwd: 5836.62 | bwd_inner: 5691.84 | bwd_allreduce: 144.75 | step: 18.04

  9%|▊         | 3519/41250 [8:29:14<90:58:42,  8.68s/it]
                                                         
{'loss': 0.1564, 'grad_norm': 1.3200483322143555, 'learning_rate': 3.968010412210217e-05, 'epoch': 0.85}

  9%|▊         | 3519/41250 [8:29:14<90:58:42,  8.68s/it][2025-04-25 16:26:57,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 16:26:57,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.00 | bwd_microstep: 5713.71 | bwd_inner_microstep: 5701.01 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.70
[2025-04-25 16:26:57,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.00 | bwd: 5713.72 | bwd_inner: 5701.01 | bwd_allreduce: 12.67 | step: 18.70

  9%|▊         | 3520/41250 [8:29:23<90:52:50,  8.67s/it]
                                                         
{'loss': 0.1562, 'grad_norm': 1.4165619611740112, 'learning_rate': 3.9679824324168e-05, 'epoch': 0.85}

  9%|▊         | 3520/41250 [8:29:23<90:52:50,  8.67s/it][2025-04-25 16:27:06,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.34 | optimizer_step: 0.97
[2025-04-25 16:27:06,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.97 | bwd_microstep: 5836.18 | bwd_inner_microstep: 5703.84 | bwd_allreduce_microstep: 132.28 | step_microstep: 20.14
[2025-04-25 16:27:06,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.98 | bwd: 5836.19 | bwd_inner: 5703.84 | bwd_allreduce: 132.30 | step: 20.14

  9%|▊         | 3521/41250 [8:29:31<91:12:57,  8.70s/it]
                                                         
{'loss': 0.1532, 'grad_norm': 1.2742916345596313, 'learning_rate': 3.967954440491158e-05, 'epoch': 0.85}

  9%|▊         | 3521/41250 [8:29:31<91:12:57,  8.70s/it][2025-04-25 16:27:15,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.17 | optimizer_step: 0.92
[2025-04-25 16:27:15,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.68 | bwd_microstep: 5691.94 | bwd_inner_microstep: 5678.87 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.30
[2025-04-25 16:27:15,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.68 | bwd: 5691.95 | bwd_inner: 5678.87 | bwd_allreduce: 13.03 | step: 19.31

  9%|▊         | 3522/41250 [8:29:40<90:57:06,  8.68s/it]
                                                         
{'loss': 0.248, 'grad_norm': 1.2562001943588257, 'learning_rate': 3.967926436433464e-05, 'epoch': 0.85}

  9%|▊         | 3522/41250 [8:29:40<90:57:06,  8.68s/it][2025-04-25 16:27:23,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 16:27:23,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.14 | bwd_microstep: 5757.37 | bwd_inner_microstep: 5643.37 | bwd_allreduce_microstep: 113.95 | step_microstep: 19.10
[2025-04-25 16:27:23,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.14 | bwd: 5757.38 | bwd_inner: 5643.37 | bwd_allreduce: 113.97 | step: 19.10

  9%|▊         | 3523/41250 [8:29:49<90:53:30,  8.67s/it]
                                                         
{'loss': 0.1275, 'grad_norm': 1.8603407144546509, 'learning_rate': 3.96789842024389e-05, 'epoch': 0.85}

  9%|▊         | 3523/41250 [8:29:49<90:53:30,  8.67s/it][2025-04-25 16:27:32,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 16:27:32,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5771.65 | bwd_inner_microstep: 5655.34 | bwd_allreduce_microstep: 116.26 | step_microstep: 18.68
[2025-04-25 16:27:32,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5771.66 | bwd_inner: 5655.34 | bwd_allreduce: 116.28 | step: 18.68

  9%|▊         | 3524/41250 [8:29:57<90:54:52,  8.68s/it]
                                                         
{'loss': 0.164, 'grad_norm': 1.8483527898788452, 'learning_rate': 3.9678703919226106e-05, 'epoch': 0.85}

  9%|▊         | 3524/41250 [8:29:57<90:54:52,  8.68s/it][2025-04-25 16:27:41,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 16:27:41,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.64 | bwd_microstep: 5723.44 | bwd_inner_microstep: 5695.28 | bwd_allreduce_microstep: 28.12 | step_microstep: 19.22
[2025-04-25 16:27:41,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.64 | bwd: 5723.45 | bwd_inner: 5695.28 | bwd_allreduce: 28.13 | step: 19.22

  9%|▊         | 3525/41250 [8:30:06<90:51:24,  8.67s/it]
                                                         
{'loss': 0.315, 'grad_norm': 2.206514596939087, 'learning_rate': 3.967842351469797e-05, 'epoch': 0.85}

  9%|▊         | 3525/41250 [8:30:06<90:51:24,  8.67s/it][2025-04-25 16:27:49,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 16:27:49,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5767.32 | bwd_inner_microstep: 5660.57 | bwd_allreduce_microstep: 106.70 | step_microstep: 18.69
[2025-04-25 16:27:49,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5767.33 | bwd_inner: 5660.57 | bwd_allreduce: 106.72 | step: 18.68

  9%|▊         | 3526/41250 [8:30:15<90:53:57,  8.67s/it]
                                                         
{'loss': 0.2685, 'grad_norm': 0.9462171196937561, 'learning_rate': 3.967814298885622e-05, 'epoch': 0.85}

  9%|▊         | 3526/41250 [8:30:15<90:53:57,  8.67s/it][2025-04-25 16:27:58,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:27:58,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.16 | bwd_microstep: 5738.12 | bwd_inner_microstep: 5650.85 | bwd_allreduce_microstep: 87.23 | step_microstep: 18.27
[2025-04-25 16:27:58,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.16 | bwd: 5738.14 | bwd_inner: 5650.85 | bwd_allreduce: 87.25 | step: 18.27

  9%|▊         | 3527/41250 [8:30:23<90:51:11,  8.67s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 1.5411615371704102, 'learning_rate': 3.967786234170259e-05, 'epoch': 0.86}

  9%|▊         | 3527/41250 [8:30:23<90:51:11,  8.67s/it][2025-04-25 16:28:07,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:28:07,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.71 | bwd_microstep: 5747.68 | bwd_inner_microstep: 5701.75 | bwd_allreduce_microstep: 45.88 | step_microstep: 18.15
[2025-04-25 16:28:07,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.71 | bwd: 5747.69 | bwd_inner: 5701.75 | bwd_allreduce: 45.90 | step: 18.15

  9%|▊         | 3528/41250 [8:30:32<90:53:54,  8.67s/it]
                                                         
{'loss': 0.2257, 'grad_norm': 1.6952259540557861, 'learning_rate': 3.9677581573238806e-05, 'epoch': 0.86}

  9%|▊         | 3528/41250 [8:30:32<90:53:54,  8.67s/it][2025-04-25 16:28:15,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 16:28:15,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5679.23 | bwd_inner_microstep: 5664.09 | bwd_allreduce_microstep: 15.10 | step_microstep: 18.53
[2025-04-25 16:28:15,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5679.24 | bwd_inner: 5664.09 | bwd_allreduce: 15.12 | step: 18.54

  9%|▊         | 3529/41250 [8:30:41<90:38:14,  8.65s/it]
                                                         
{'loss': 0.0477, 'grad_norm': 0.6347241401672363, 'learning_rate': 3.967730068346661e-05, 'epoch': 0.86}

  9%|▊         | 3529/41250 [8:30:41<90:38:14,  8.65s/it][2025-04-25 16:28:24,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:28:24,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.63 | bwd_microstep: 5741.36 | bwd_inner_microstep: 5706.93 | bwd_allreduce_microstep: 34.39 | step_microstep: 18.69
[2025-04-25 16:28:24,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.63 | bwd: 5741.37 | bwd_inner: 5706.93 | bwd_allreduce: 34.40 | step: 18.69

  9%|▊         | 3530/41250 [8:30:49<90:44:29,  8.66s/it]
                                                         
{'loss': 0.108, 'grad_norm': 1.0646501779556274, 'learning_rate': 3.967701967238772e-05, 'epoch': 0.86}

  9%|▊         | 3530/41250 [8:30:49<90:44:29,  8.66s/it][2025-04-25 16:28:33,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.05 | optimizer_step: 1.15
[2025-04-25 16:28:33,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.79 | bwd_microstep: 5736.28 | bwd_inner_microstep: 5685.31 | bwd_allreduce_microstep: 50.92 | step_microstep: 19.33
[2025-04-25 16:28:33,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.79 | bwd: 5736.29 | bwd_inner: 5685.30 | bwd_allreduce: 50.94 | step: 19.33

  9%|▊         | 3531/41250 [8:30:58<90:50:09,  8.67s/it]
                                                         
{'loss': 0.2946, 'grad_norm': 2.0565059185028076, 'learning_rate': 3.967673854000387e-05, 'epoch': 0.86}

  9%|▊         | 3531/41250 [8:30:58<90:50:09,  8.67s/it][2025-04-25 16:28:41,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:28:41,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5763.26 | bwd_inner_microstep: 5708.71 | bwd_allreduce_microstep: 54.50 | step_microstep: 18.33
[2025-04-25 16:28:41,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5763.27 | bwd_inner: 5708.71 | bwd_allreduce: 54.52 | step: 18.33

  9%|▊         | 3532/41250 [8:31:07<90:57:37,  8.68s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 1.6253249645233154, 'learning_rate': 3.967645728631681e-05, 'epoch': 0.86}

  9%|▊         | 3532/41250 [8:31:07<90:57:37,  8.68s/it][2025-04-25 16:28:50,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.72 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:28:50,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2901.63 | bwd_microstep: 5801.95 | bwd_inner_microstep: 5789.31 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.50
[2025-04-25 16:28:50,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2901.63 | bwd: 5801.96 | bwd_inner: 5789.31 | bwd_allreduce: 12.60 | step: 18.51

  9%|▊         | 3533/41250 [8:31:16<91:16:44,  8.71s/it]
                                                         
{'loss': 0.1161, 'grad_norm': 1.1715248823165894, 'learning_rate': 3.9676175911328247e-05, 'epoch': 0.86}

  9%|▊         | 3533/41250 [8:31:16<91:16:44,  8.71s/it][2025-04-25 16:28:59,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:28:59,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.40 | bwd_microstep: 5712.85 | bwd_inner_microstep: 5692.35 | bwd_allreduce_microstep: 20.46 | step_microstep: 18.58
[2025-04-25 16:28:59,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.40 | bwd: 5712.86 | bwd_inner: 5692.35 | bwd_allreduce: 20.47 | step: 18.58

  9%|▊         | 3534/41250 [8:31:24<91:03:51,  8.69s/it]
                                                         
{'loss': 0.2158, 'grad_norm': 1.7132998704910278, 'learning_rate': 3.967589441503994e-05, 'epoch': 0.86}

  9%|▊         | 3534/41250 [8:31:24<91:03:51,  8.69s/it][2025-04-25 16:29:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:29:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.19 | bwd_microstep: 5788.05 | bwd_inner_microstep: 5671.77 | bwd_allreduce_microstep: 116.23 | step_microstep: 18.65
[2025-04-25 16:29:08,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.19 | bwd: 5788.06 | bwd_inner: 5671.77 | bwd_allreduce: 116.24 | step: 18.66

  9%|▊         | 3535/41250 [8:31:33<91:09:59,  8.70s/it]
                                                         
{'loss': 0.3017, 'grad_norm': 1.9633748531341553, 'learning_rate': 3.9675612797453605e-05, 'epoch': 0.86}

  9%|▊         | 3535/41250 [8:31:33<91:09:59,  8.70s/it][2025-04-25 16:29:16,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:29:16,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.76 | bwd_microstep: 5708.04 | bwd_inner_microstep: 5695.37 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.58
[2025-04-25 16:29:16,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.76 | bwd: 5708.05 | bwd_inner: 5695.37 | bwd_allreduce: 12.64 | step: 18.58

  9%|▊         | 3536/41250 [8:31:42<91:00:06,  8.69s/it]
                                                         
{'loss': 0.195, 'grad_norm': 0.91036057472229, 'learning_rate': 3.967533105857099e-05, 'epoch': 0.86}

  9%|▊         | 3536/41250 [8:31:42<91:00:06,  8.69s/it][2025-04-25 16:29:25,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:29:25,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.63 | bwd_microstep: 5756.00 | bwd_inner_microstep: 5699.34 | bwd_allreduce_microstep: 56.60 | step_microstep: 18.50
[2025-04-25 16:29:25,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.63 | bwd: 5756.01 | bwd_inner: 5699.34 | bwd_allreduce: 56.62 | step: 18.50

  9%|▊         | 3537/41250 [8:31:50<91:01:06,  8.69s/it]
                                                         
{'loss': 0.1347, 'grad_norm': 1.1750882863998413, 'learning_rate': 3.967504919839383e-05, 'epoch': 0.86}

  9%|▊         | 3537/41250 [8:31:50<91:01:06,  8.69s/it][2025-04-25 16:29:34,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:29:34,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.53 | bwd_microstep: 5757.40 | bwd_inner_microstep: 5692.38 | bwd_allreduce_microstep: 64.97 | step_microstep: 18.56
[2025-04-25 16:29:34,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.53 | bwd: 5757.41 | bwd_inner: 5692.38 | bwd_allreduce: 64.99 | step: 18.56

  9%|▊         | 3538/41250 [8:31:59<91:01:46,  8.69s/it]
                                                         
{'loss': 0.4089, 'grad_norm': 2.3618998527526855, 'learning_rate': 3.967476721692386e-05, 'epoch': 0.86}

  9%|▊         | 3538/41250 [8:31:59<91:01:46,  8.69s/it][2025-04-25 16:29:42,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:29:42,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.17 | bwd_microstep: 5719.01 | bwd_inner_microstep: 5706.30 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.82
[2025-04-25 16:29:42,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.17 | bwd: 5719.02 | bwd_inner: 5706.30 | bwd_allreduce: 12.69 | step: 18.82

  9%|▊         | 3539/41250 [8:32:08<90:57:56,  8.68s/it]
                                                         
{'loss': 0.1774, 'grad_norm': 1.2080897092819214, 'learning_rate': 3.967448511416282e-05, 'epoch': 0.86}

  9%|▊         | 3539/41250 [8:32:08<90:57:56,  8.68s/it][2025-04-25 16:29:51,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:29:51,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.60 | bwd_microstep: 5741.47 | bwd_inner_microstep: 5710.07 | bwd_allreduce_microstep: 31.36 | step_microstep: 18.53
[2025-04-25 16:29:51,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.60 | bwd: 5741.48 | bwd_inner: 5710.07 | bwd_allreduce: 31.37 | step: 18.53

  9%|▊         | 3540/41250 [8:32:16<90:56:57,  8.68s/it]
                                                         
{'loss': 0.1247, 'grad_norm': 1.0301657915115356, 'learning_rate': 3.967420289011245e-05, 'epoch': 0.86}

  9%|▊         | 3540/41250 [8:32:16<90:56:57,  8.68s/it][2025-04-25 16:30:00,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:30:00,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.04 | bwd_microstep: 5766.13 | bwd_inner_microstep: 5663.04 | bwd_allreduce_microstep: 103.04 | step_microstep: 18.87
[2025-04-25 16:30:00,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.04 | bwd: 5766.14 | bwd_inner: 5663.04 | bwd_allreduce: 103.06 | step: 18.87

  9%|▊         | 3541/41250 [8:32:25<90:57:06,  8.68s/it]
                                                         
{'loss': 0.1785, 'grad_norm': 2.142714023590088, 'learning_rate': 3.967392054477449e-05, 'epoch': 0.86}

  9%|▊         | 3541/41250 [8:32:25<90:57:06,  8.68s/it][2025-04-25 16:30:08,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 16:30:08,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.77 | bwd_microstep: 5776.34 | bwd_inner_microstep: 5664.55 | bwd_allreduce_microstep: 111.75 | step_microstep: 18.86
[2025-04-25 16:30:08,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.77 | bwd: 5776.36 | bwd_inner: 5664.55 | bwd_allreduce: 111.77 | step: 18.86

  9%|▊         | 3542/41250 [8:32:34<91:00:03,  8.69s/it]
                                                         
{'loss': 0.2672, 'grad_norm': 1.992012858390808, 'learning_rate': 3.9673638078150685e-05, 'epoch': 0.86}

  9%|▊         | 3542/41250 [8:32:34<91:00:03,  8.69s/it][2025-04-25 16:30:17,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:30:17,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.82 | bwd_microstep: 5732.18 | bwd_inner_microstep: 5719.40 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.52
[2025-04-25 16:30:17,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.82 | bwd: 5732.19 | bwd_inner: 5719.40 | bwd_allreduce: 12.75 | step: 18.52

  9%|▊         | 3543/41250 [8:32:42<90:59:02,  8.69s/it]
                                                         
{'loss': 0.1223, 'grad_norm': 1.307335615158081, 'learning_rate': 3.9673355490242763e-05, 'epoch': 0.86}

  9%|▊         | 3543/41250 [8:32:42<90:59:02,  8.69s/it][2025-04-25 16:30:26,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:30:26,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.78 | bwd_microstep: 5791.32 | bwd_inner_microstep: 5663.25 | bwd_allreduce_microstep: 128.02 | step_microstep: 18.46
[2025-04-25 16:30:26,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.78 | bwd: 5791.33 | bwd_inner: 5663.25 | bwd_allreduce: 128.04 | step: 18.46

  9%|▊         | 3544/41250 [8:32:51<91:04:43,  8.70s/it]
                                                         
{'loss': 0.3064, 'grad_norm': 1.9442740678787231, 'learning_rate': 3.967307278105248e-05, 'epoch': 0.86}

  9%|▊         | 3544/41250 [8:32:51<91:04:43,  8.70s/it][2025-04-25 16:30:34,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 16:30:34,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.66 | bwd_microstep: 5720.35 | bwd_inner_microstep: 5707.47 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.23
[2025-04-25 16:30:34,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.66 | bwd: 5720.37 | bwd_inner: 5707.47 | bwd_allreduce: 12.85 | step: 18.24

  9%|▊         | 3545/41250 [8:33:00<90:57:41,  8.68s/it]
                                                         
{'loss': 0.2855, 'grad_norm': 2.6159095764160156, 'learning_rate': 3.967278995058157e-05, 'epoch': 0.86}

  9%|▊         | 3545/41250 [8:33:00<90:57:41,  8.68s/it][2025-04-25 16:30:43,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 16:30:43,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.91 | bwd_microstep: 5748.90 | bwd_inner_microstep: 5699.19 | bwd_allreduce_microstep: 49.66 | step_microstep: 18.45
[2025-04-25 16:30:43,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.91 | bwd: 5748.91 | bwd_inner: 5699.19 | bwd_allreduce: 49.68 | step: 18.45

  9%|▊         | 3546/41250 [8:33:08<90:58:14,  8.69s/it]
                                                         
{'loss': 0.3396, 'grad_norm': 1.765406608581543, 'learning_rate': 3.9672506998831775e-05, 'epoch': 0.86}

  9%|▊         | 3546/41250 [8:33:08<90:58:14,  8.69s/it][2025-04-25 16:30:52,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:30:52,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.65 | bwd_microstep: 5689.56 | bwd_inner_microstep: 5663.34 | bwd_allreduce_microstep: 26.17 | step_microstep: 18.58
[2025-04-25 16:30:52,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.65 | bwd: 5689.57 | bwd_inner: 5663.34 | bwd_allreduce: 26.19 | step: 18.58

  9%|▊         | 3547/41250 [8:33:17<90:44:56,  8.66s/it]
                                                         
{'loss': 0.2096, 'grad_norm': 1.3961220979690552, 'learning_rate': 3.967222392580484e-05, 'epoch': 0.86}

  9%|▊         | 3547/41250 [8:33:17<90:44:56,  8.66s/it][2025-04-25 16:31:00,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:31:00,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.49 | bwd_microstep: 5747.67 | bwd_inner_microstep: 5665.86 | bwd_allreduce_microstep: 81.78 | step_microstep: 18.67
[2025-04-25 16:31:00,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.49 | bwd: 5747.69 | bwd_inner: 5665.86 | bwd_allreduce: 81.79 | step: 18.67

  9%|▊         | 3548/41250 [8:33:26<90:46:51,  8.67s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 5.328229904174805, 'learning_rate': 3.9671940731502524e-05, 'epoch': 0.86}

  9%|▊         | 3548/41250 [8:33:26<90:46:51,  8.67s/it][2025-04-25 16:31:09,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 16:31:09,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.68 | bwd_microstep: 5761.20 | bwd_inner_microstep: 5658.36 | bwd_allreduce_microstep: 102.80 | step_microstep: 18.57
[2025-04-25 16:31:09,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.68 | bwd: 5761.21 | bwd_inner: 5658.36 | bwd_allreduce: 102.82 | step: 18.57

  9%|▊         | 3549/41250 [8:33:34<90:48:10,  8.67s/it]
                                                         
{'loss': 0.197, 'grad_norm': 1.564500331878662, 'learning_rate': 3.967165741592656e-05, 'epoch': 0.86}

  9%|▊         | 3549/41250 [8:33:34<90:48:10,  8.67s/it][2025-04-25 16:31:18,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:31:18,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.77 | bwd_microstep: 5755.00 | bwd_inner_microstep: 5662.31 | bwd_allreduce_microstep: 92.64 | step_microstep: 18.59
[2025-04-25 16:31:18,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.77 | bwd: 5755.01 | bwd_inner: 5662.31 | bwd_allreduce: 92.66 | step: 18.60

  9%|▊         | 3550/41250 [8:33:43<90:46:20,  8.67s/it]
                                                         
{'loss': 0.28, 'grad_norm': 4.144301414489746, 'learning_rate': 3.9671373979078696e-05, 'epoch': 0.86}

  9%|▊         | 3550/41250 [8:33:43<90:46:20,  8.67s/it][2025-04-25 16:31:26,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:31:26,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.22 | bwd_microstep: 5704.47 | bwd_inner_microstep: 5691.74 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.75
[2025-04-25 16:31:26,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.22 | bwd: 5704.49 | bwd_inner: 5691.74 | bwd_allreduce: 12.71 | step: 18.75

  9%|▊         | 3551/41250 [8:33:52<90:42:16,  8.66s/it]
                                                         
{'loss': 0.2166, 'grad_norm': 1.2406071424484253, 'learning_rate': 3.9671090420960684e-05, 'epoch': 0.86}

  9%|▊         | 3551/41250 [8:33:52<90:42:16,  8.66s/it][2025-04-25 16:31:35,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 16:31:35,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.07 | bwd_microstep: 5713.93 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.70
[2025-04-25 16:31:35,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.07 | bwd: 5713.94 | bwd_inner: 5701.15 | bwd_allreduce: 12.75 | step: 18.70

  9%|▊         | 3552/41250 [8:34:00<90:41:12,  8.66s/it]
                                                         
{'loss': 0.3042, 'grad_norm': 2.333850145339966, 'learning_rate': 3.9670806741574265e-05, 'epoch': 0.86}

  9%|▊         | 3552/41250 [8:34:00<90:41:12,  8.66s/it][2025-04-25 16:31:44,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 16:31:44,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.34 | bwd_microstep: 5760.01 | bwd_inner_microstep: 5691.36 | bwd_allreduce_microstep: 68.60 | step_microstep: 18.83
[2025-04-25 16:31:44,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.34 | bwd: 5760.02 | bwd_inner: 5691.36 | bwd_allreduce: 68.62 | step: 18.83

  9%|▊         | 3553/41250 [8:34:09<90:47:15,  8.67s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 1.066283941268921, 'learning_rate': 3.967052294092119e-05, 'epoch': 0.86}

  9%|▊         | 3553/41250 [8:34:09<90:47:15,  8.67s/it][2025-04-25 16:31:52,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:31:52,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.36 | bwd_microstep: 5725.24 | bwd_inner_microstep: 5707.26 | bwd_allreduce_microstep: 17.94 | step_microstep: 18.10
[2025-04-25 16:31:52,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.36 | bwd: 5725.26 | bwd_inner: 5707.26 | bwd_allreduce: 17.95 | step: 18.10

  9%|▊         | 3554/41250 [8:34:18<90:45:24,  8.67s/it]
                                                         
{'loss': 0.3834, 'grad_norm': 1.5125970840454102, 'learning_rate': 3.967023901900321e-05, 'epoch': 0.86}

  9%|▊         | 3554/41250 [8:34:18<90:45:24,  8.67s/it][2025-04-25 16:32:01,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:32:01,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.21 | bwd_microstep: 5735.40 | bwd_inner_microstep: 5698.97 | bwd_allreduce_microstep: 36.38 | step_microstep: 18.72
[2025-04-25 16:32:01,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.21 | bwd: 5735.41 | bwd_inner: 5698.97 | bwd_allreduce: 36.40 | step: 18.73

  9%|▊         | 3555/41250 [8:34:26<90:47:44,  8.67s/it]
                                                         
{'loss': 0.3148, 'grad_norm': 0.9405214190483093, 'learning_rate': 3.966995497582208e-05, 'epoch': 0.86}

  9%|▊         | 3555/41250 [8:34:26<90:47:44,  8.67s/it][2025-04-25 16:32:10,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 16:32:10,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.26 | bwd_microstep: 5730.21 | bwd_inner_microstep: 5679.22 | bwd_allreduce_microstep: 50.95 | step_microstep: 19.27
[2025-04-25 16:32:10,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.26 | bwd: 5730.23 | bwd_inner: 5679.22 | bwd_allreduce: 50.97 | step: 19.27

  9%|▊         | 3556/41250 [8:34:35<90:47:40,  8.67s/it]
                                                         
{'loss': 0.1393, 'grad_norm': 1.1369140148162842, 'learning_rate': 3.9669670811379544e-05, 'epoch': 0.86}

  9%|▊         | 3556/41250 [8:34:35<90:47:40,  8.67s/it][2025-04-25 16:32:18,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:32:18,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5690.62 | bwd_inner_microstep: 5645.87 | bwd_allreduce_microstep: 44.70 | step_microstep: 18.79
[2025-04-25 16:32:18,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.49 | bwd: 5690.63 | bwd_inner: 5645.87 | bwd_allreduce: 44.71 | step: 18.79

  9%|▊         | 3557/41250 [8:34:44<90:35:44,  8.65s/it]
                                                         
{'loss': 0.1686, 'grad_norm': 1.3883216381072998, 'learning_rate': 3.966938652567735e-05, 'epoch': 0.86}

  9%|▊         | 3557/41250 [8:34:44<90:35:44,  8.65s/it][2025-04-25 16:32:27,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 1.12 | optimizer_step: 1.02
[2025-04-25 16:32:27,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5701.17 | bwd_inner_microstep: 5687.12 | bwd_allreduce_microstep: 13.99 | step_microstep: 19.37
[2025-04-25 16:32:27,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5701.18 | bwd_inner: 5687.11 | bwd_allreduce: 14.02 | step: 19.37

  9%|▊         | 3558/41250 [8:34:52<90:32:18,  8.65s/it]
                                                         
{'loss': 0.1576, 'grad_norm': 1.2605631351470947, 'learning_rate': 3.966910211871727e-05, 'epoch': 0.86}

  9%|▊         | 3558/41250 [8:34:52<90:32:18,  8.65s/it][2025-04-25 16:32:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 16:32:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.85 | bwd_microstep: 5742.80 | bwd_inner_microstep: 5691.45 | bwd_allreduce_microstep: 51.30 | step_microstep: 18.57
[2025-04-25 16:32:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.85 | bwd: 5742.81 | bwd_inner: 5691.45 | bwd_allreduce: 51.32 | step: 18.58

  9%|▊         | 3559/41250 [8:35:01<90:37:28,  8.66s/it]
                                                         
{'loss': 0.1639, 'grad_norm': 0.8912169337272644, 'learning_rate': 3.966881759050104e-05, 'epoch': 0.86}

  9%|▊         | 3559/41250 [8:35:01<90:37:28,  8.66s/it][2025-04-25 16:32:44,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:32:44,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.31 | bwd_microstep: 5712.10 | bwd_inner_microstep: 5695.66 | bwd_allreduce_microstep: 16.39 | step_microstep: 18.56
[2025-04-25 16:32:44,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.31 | bwd: 5712.11 | bwd_inner: 5695.66 | bwd_allreduce: 16.41 | step: 18.56

  9%|▊         | 3560/41250 [8:35:10<90:35:16,  8.65s/it]
                                                         
{'loss': 0.1137, 'grad_norm': 0.7416310906410217, 'learning_rate': 3.966853294103041e-05, 'epoch': 0.86}

  9%|▊         | 3560/41250 [8:35:10<90:35:16,  8.65s/it][2025-04-25 16:32:53,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:32:53,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5700.08 | bwd_inner_microstep: 5647.92 | bwd_allreduce_microstep: 52.11 | step_microstep: 18.21
[2025-04-25 16:32:53,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5700.09 | bwd_inner: 5647.92 | bwd_allreduce: 52.13 | step: 18.21

  9%|▊         | 3561/41250 [8:35:18<90:26:52,  8.64s/it]
                                                         
{'loss': 0.1173, 'grad_norm': 0.8941547870635986, 'learning_rate': 3.9668248170307154e-05, 'epoch': 0.86}

  9%|▊         | 3561/41250 [8:35:18<90:26:52,  8.64s/it][2025-04-25 16:33:02,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 16:33:02,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.83 | bwd_microstep: 5730.28 | bwd_inner_microstep: 5643.79 | bwd_allreduce_microstep: 86.44 | step_microstep: 18.87
[2025-04-25 16:33:02,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.83 | bwd: 5730.29 | bwd_inner: 5643.79 | bwd_allreduce: 86.45 | step: 18.87

  9%|▊         | 3562/41250 [8:35:27<90:27:34,  8.64s/it]
                                                         
{'loss': 0.1595, 'grad_norm': 1.402382731437683, 'learning_rate': 3.9667963278333006e-05, 'epoch': 0.86}

  9%|▊         | 3562/41250 [8:35:27<90:27:34,  8.64s/it][2025-04-25 16:33:10,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 16:33:10,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5697.58 | bwd_inner_microstep: 5653.45 | bwd_allreduce_microstep: 44.08 | step_microstep: 19.01
[2025-04-25 16:33:10,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5697.60 | bwd_inner: 5653.45 | bwd_allreduce: 44.10 | step: 19.01

  9%|▊         | 3563/41250 [8:35:35<90:21:34,  8.63s/it]
                                                         
{'loss': 0.1055, 'grad_norm': 1.650078535079956, 'learning_rate': 3.966767826510974e-05, 'epoch': 0.86}

  9%|▊         | 3563/41250 [8:35:35<90:21:34,  8.63s/it][2025-04-25 16:33:19,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:33:19,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.39 | bwd_microstep: 5724.45 | bwd_inner_microstep: 5684.67 | bwd_allreduce_microstep: 39.74 | step_microstep: 18.94
[2025-04-25 16:33:19,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.39 | bwd: 5724.47 | bwd_inner: 5684.67 | bwd_allreduce: 39.76 | step: 18.94

  9%|▊         | 3564/41250 [8:35:44<90:25:14,  8.64s/it]
                                                         
{'loss': 0.1019, 'grad_norm': 0.8836087584495544, 'learning_rate': 3.9667393130639104e-05, 'epoch': 0.86}

  9%|▊         | 3564/41250 [8:35:44<90:25:14,  8.64s/it][2025-04-25 16:33:27,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 16:33:27,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.69 | bwd_microstep: 5713.33 | bwd_inner_microstep: 5700.48 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.86
[2025-04-25 16:33:27,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.69 | bwd: 5713.34 | bwd_inner: 5700.48 | bwd_allreduce: 12.82 | step: 18.86

  9%|▊         | 3565/41250 [8:35:53<90:27:52,  8.64s/it]
                                                         
{'loss': 0.1687, 'grad_norm': 0.9025037288665771, 'learning_rate': 3.966710787492286e-05, 'epoch': 0.86}

  9%|▊         | 3565/41250 [8:35:53<90:27:52,  8.64s/it][2025-04-25 16:33:36,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 16:33:36,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.70 | bwd_microstep: 5832.91 | bwd_inner_microstep: 5702.67 | bwd_allreduce_microstep: 130.19 | step_microstep: 19.15
[2025-04-25 16:33:36,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.70 | bwd: 5832.92 | bwd_inner: 5702.67 | bwd_allreduce: 130.21 | step: 19.16

  9%|▊         | 3566/41250 [8:36:02<90:51:19,  8.68s/it]
                                                         
{'loss': 0.1224, 'grad_norm': 1.0170551538467407, 'learning_rate': 3.966682249796276e-05, 'epoch': 0.86}

  9%|▊         | 3566/41250 [8:36:02<90:51:19,  8.68s/it][2025-04-25 16:33:45,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 16:33:45,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.07 | bwd_microstep: 5702.83 | bwd_inner_microstep: 5689.86 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.97
[2025-04-25 16:33:45,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.07 | bwd: 5702.84 | bwd_inner: 5689.86 | bwd_allreduce: 12.94 | step: 18.97

  9%|▊         | 3567/41250 [8:36:10<90:44:54,  8.67s/it]
                                                         
{'loss': 0.2486, 'grad_norm': 1.5299443006515503, 'learning_rate': 3.966653699976057e-05, 'epoch': 0.86}

  9%|▊         | 3567/41250 [8:36:10<90:44:54,  8.67s/it][2025-04-25 16:33:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.30 | optimizer_step: 0.89
[2025-04-25 16:33:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.24 | bwd_microstep: 5740.28 | bwd_inner_microstep: 5676.81 | bwd_allreduce_microstep: 63.41 | step_microstep: 19.14
[2025-04-25 16:33:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.24 | bwd: 5740.29 | bwd_inner: 5676.81 | bwd_allreduce: 63.43 | step: 19.14

  9%|▊         | 3568/41250 [8:36:19<90:44:02,  8.67s/it]
                                                         
{'loss': 0.1974, 'grad_norm': 1.7373113632202148, 'learning_rate': 3.966625138031805e-05, 'epoch': 0.86}

  9%|▊         | 3568/41250 [8:36:19<90:44:02,  8.67s/it][2025-04-25 16:34:02,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:34:02,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5753.89 | bwd_inner_microstep: 5658.20 | bwd_allreduce_microstep: 95.64 | step_microstep: 18.27
[2025-04-25 16:34:02,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5753.90 | bwd_inner: 5658.20 | bwd_allreduce: 95.66 | step: 18.27

  9%|▊         | 3569/41250 [8:36:28<90:42:57,  8.67s/it]
                                                         
{'loss': 0.0785, 'grad_norm': 0.6456536054611206, 'learning_rate': 3.966596563963695e-05, 'epoch': 0.87}

  9%|▊         | 3569/41250 [8:36:28<90:42:57,  8.67s/it][2025-04-25 16:34:11,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.20
[2025-04-25 16:34:11,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.61 | bwd_microstep: 5669.89 | bwd_inner_microstep: 5642.80 | bwd_allreduce_microstep: 27.05 | step_microstep: 19.47
[2025-04-25 16:34:11,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.61 | bwd: 5669.91 | bwd_inner: 5642.80 | bwd_allreduce: 27.07 | step: 19.47

  9%|▊         | 3570/41250 [8:36:36<90:26:47,  8.64s/it]
                                                         
{'loss': 0.2314, 'grad_norm': 2.5643904209136963, 'learning_rate': 3.9665679777719046e-05, 'epoch': 0.87}

  9%|▊         | 3570/41250 [8:36:36<90:26:47,  8.64s/it][2025-04-25 16:34:19,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 16:34:19,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.38 | bwd_microstep: 5720.05 | bwd_inner_microstep: 5693.98 | bwd_allreduce_microstep: 26.03 | step_microstep: 18.80
[2025-04-25 16:34:19,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.38 | bwd: 5720.07 | bwd_inner: 5693.98 | bwd_allreduce: 26.05 | step: 18.80

  9%|▊         | 3571/41250 [8:36:45<90:28:17,  8.64s/it]
                                                         
{'loss': 0.4033, 'grad_norm': 3.155406951904297, 'learning_rate': 3.9665393794566096e-05, 'epoch': 0.87}

  9%|▊         | 3571/41250 [8:36:45<90:28:17,  8.64s/it][2025-04-25 16:34:28,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.20 | optimizer_step: 0.93
[2025-04-25 16:34:28,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.94 | bwd_microstep: 5679.33 | bwd_inner_microstep: 5647.08 | bwd_allreduce_microstep: 32.20 | step_microstep: 19.36
[2025-04-25 16:34:28,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.94 | bwd: 5679.34 | bwd_inner: 5647.08 | bwd_allreduce: 32.22 | step: 19.36

  9%|▊         | 3572/41250 [8:36:53<90:17:03,  8.63s/it]
                                                         
{'loss': 0.1219, 'grad_norm': 0.9624172449111938, 'learning_rate': 3.9665107690179854e-05, 'epoch': 0.87}

  9%|▊         | 3572/41250 [8:36:53<90:17:03,  8.63s/it][2025-04-25 16:34:37,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:34:37,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.21 | bwd_microstep: 5746.50 | bwd_inner_microstep: 5643.34 | bwd_allreduce_microstep: 103.11 | step_microstep: 18.73
[2025-04-25 16:34:37,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.22 | bwd: 5746.51 | bwd_inner: 5643.34 | bwd_allreduce: 103.13 | step: 18.73

  9%|▊         | 3573/41250 [8:37:02<90:22:29,  8.64s/it]
                                                         
{'loss': 0.3145, 'grad_norm': 2.1885995864868164, 'learning_rate': 3.9664821464562095e-05, 'epoch': 0.87}

  9%|▊         | 3573/41250 [8:37:02<90:22:29,  8.64s/it][2025-04-25 16:34:45,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:34:45,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5722.04 | bwd_inner_microstep: 5691.69 | bwd_allreduce_microstep: 30.30 | step_microstep: 19.00
[2025-04-25 16:34:45,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5722.05 | bwd_inner: 5691.69 | bwd_allreduce: 30.32 | step: 19.00

  9%|▊         | 3574/41250 [8:37:11<90:27:22,  8.64s/it]
                                                         
{'loss': 0.0704, 'grad_norm': 0.6178162693977356, 'learning_rate': 3.966453511771458e-05, 'epoch': 0.87}

  9%|▊         | 3574/41250 [8:37:11<90:27:22,  8.64s/it][2025-04-25 16:34:54,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.15 | optimizer_step: 1.07
[2025-04-25 16:34:54,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5702.33 | bwd_inner_microstep: 5688.38 | bwd_allreduce_microstep: 13.89 | step_microstep: 20.47
[2025-04-25 16:34:54,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5702.35 | bwd_inner: 5688.38 | bwd_allreduce: 13.92 | step: 20.47

  9%|▊         | 3575/41250 [8:37:19<90:26:59,  8.64s/it]
                                                         
{'loss': 0.2992, 'grad_norm': 1.7641932964324951, 'learning_rate': 3.966424864963908e-05, 'epoch': 0.87}

  9%|▊         | 3575/41250 [8:37:19<90:26:59,  8.64s/it][2025-04-25 16:35:03,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 16:35:03,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.14 | bwd_microstep: 5745.24 | bwd_inner_microstep: 5692.00 | bwd_allreduce_microstep: 53.19 | step_microstep: 19.26
[2025-04-25 16:35:03,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.14 | bwd: 5745.25 | bwd_inner: 5692.00 | bwd_allreduce: 53.21 | step: 19.26

  9%|▊         | 3576/41250 [8:37:28<90:33:15,  8.65s/it]
                                                         
{'loss': 0.0338, 'grad_norm': 0.3722374141216278, 'learning_rate': 3.966396206033734e-05, 'epoch': 0.87}

  9%|▊         | 3576/41250 [8:37:28<90:33:15,  8.65s/it][2025-04-25 16:35:11,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:35:11,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.31 | bwd_microstep: 5682.54 | bwd_inner_microstep: 5649.86 | bwd_allreduce_microstep: 32.63 | step_microstep: 18.65
[2025-04-25 16:35:11,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.31 | bwd: 5682.55 | bwd_inner: 5649.86 | bwd_allreduce: 32.65 | step: 18.65

  9%|▊         | 3577/41250 [8:37:37<90:22:36,  8.64s/it]
                                                         
{'loss': 0.3364, 'grad_norm': 2.3389945030212402, 'learning_rate': 3.966367534981116e-05, 'epoch': 0.87}

  9%|▊         | 3577/41250 [8:37:37<90:22:36,  8.64s/it][2025-04-25 16:35:20,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:35:20,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.19 | bwd_microstep: 5703.84 | bwd_inner_microstep: 5691.08 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.81
[2025-04-25 16:35:20,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.19 | bwd: 5703.85 | bwd_inner: 5691.08 | bwd_allreduce: 12.73 | step: 18.81

  9%|▊         | 3578/41250 [8:37:45<90:22:33,  8.64s/it]
                                                         
{'loss': 0.1824, 'grad_norm': 1.131699562072754, 'learning_rate': 3.9663388518062275e-05, 'epoch': 0.87}

  9%|▊         | 3578/41250 [8:37:45<90:22:33,  8.64s/it][2025-04-25 16:35:29,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:35:29,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.84 | bwd_microstep: 5758.71 | bwd_inner_microstep: 5661.30 | bwd_allreduce_microstep: 97.37 | step_microstep: 18.69
[2025-04-25 16:35:29,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.84 | bwd: 5758.73 | bwd_inner: 5661.30 | bwd_allreduce: 97.39 | step: 18.69

  9%|▊         | 3579/41250 [8:37:54<90:29:16,  8.65s/it]
                                                         
{'loss': 0.0383, 'grad_norm': 0.33503249287605286, 'learning_rate': 3.9663101565092473e-05, 'epoch': 0.87}

  9%|▊         | 3579/41250 [8:37:54<90:29:16,  8.65s/it][2025-04-25 16:35:37,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 16:35:37,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.98 | bwd_microstep: 5859.65 | bwd_inner_microstep: 5705.71 | bwd_allreduce_microstep: 153.90 | step_microstep: 18.82
[2025-04-25 16:35:37,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.98 | bwd: 5859.67 | bwd_inner: 5705.71 | bwd_allreduce: 153.92 | step: 18.83

  9%|▊         | 3580/41250 [8:38:03<90:57:21,  8.69s/it]
                                                         
{'loss': 0.331, 'grad_norm': 3.0370728969573975, 'learning_rate': 3.9662814490903515e-05, 'epoch': 0.87}

  9%|▊         | 3580/41250 [8:38:03<90:57:21,  8.69s/it][2025-04-25 16:35:46,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:35:46,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.61 | bwd_microstep: 5689.38 | bwd_inner_microstep: 5657.01 | bwd_allreduce_microstep: 32.32 | step_microstep: 18.79
[2025-04-25 16:35:46,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.61 | bwd: 5689.39 | bwd_inner: 5657.01 | bwd_allreduce: 32.34 | step: 18.79

  9%|▊         | 3581/41250 [8:38:11<90:41:45,  8.67s/it]
                                                         
{'loss': 0.0987, 'grad_norm': 1.373882532119751, 'learning_rate': 3.966252729549718e-05, 'epoch': 0.87}

  9%|▊         | 3581/41250 [8:38:11<90:41:45,  8.67s/it][2025-04-25 16:35:55,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:35:55,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5693.88 | bwd_inner_microstep: 5652.04 | bwd_allreduce_microstep: 41.80 | step_microstep: 18.81
[2025-04-25 16:35:55,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5693.89 | bwd_inner: 5652.04 | bwd_allreduce: 41.81 | step: 18.82

  9%|▊         | 3582/41250 [8:38:20<90:29:16,  8.65s/it]
                                                         
{'loss': 0.1449, 'grad_norm': 2.2648561000823975, 'learning_rate': 3.966223997887522e-05, 'epoch': 0.87}

  9%|▊         | 3582/41250 [8:38:20<90:29:16,  8.65s/it][2025-04-25 16:36:03,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 16:36:03,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 5762.07 | bwd_inner_microstep: 5698.89 | bwd_allreduce_microstep: 63.14 | step_microstep: 18.83
[2025-04-25 16:36:03,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5762.09 | bwd_inner: 5698.89 | bwd_allreduce: 63.16 | step: 18.83

  9%|▊         | 3583/41250 [8:38:29<90:36:34,  8.66s/it]
                                                         
{'loss': 0.2047, 'grad_norm': 1.2887792587280273, 'learning_rate': 3.9661952541039425e-05, 'epoch': 0.87}

  9%|▊         | 3583/41250 [8:38:29<90:36:34,  8.66s/it][2025-04-25 16:36:12,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 16:36:12,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.90 | bwd_microstep: 5793.52 | bwd_inner_microstep: 5648.87 | bwd_allreduce_microstep: 144.61 | step_microstep: 19.36
[2025-04-25 16:36:12,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.90 | bwd: 5793.53 | bwd_inner: 5648.87 | bwd_allreduce: 144.62 | step: 19.36

  9%|▊         | 3584/41250 [8:38:37<90:45:01,  8.67s/it]
                                                         
{'loss': 0.1496, 'grad_norm': 1.5922105312347412, 'learning_rate': 3.966166498199156e-05, 'epoch': 0.87}

  9%|▊         | 3584/41250 [8:38:37<90:45:01,  8.67s/it][2025-04-25 16:36:21,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 16:36:21,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.11 | bwd_microstep: 5702.20 | bwd_inner_microstep: 5661.26 | bwd_allreduce_microstep: 40.90 | step_microstep: 18.37
[2025-04-25 16:36:21,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.11 | bwd: 5702.22 | bwd_inner: 5661.26 | bwd_allreduce: 40.92 | step: 18.37

  9%|▊         | 3585/41250 [8:38:46<90:32:56,  8.65s/it]
                                                         
{'loss': 0.1329, 'grad_norm': 1.390531063079834, 'learning_rate': 3.96613773017334e-05, 'epoch': 0.87}

  9%|▊         | 3585/41250 [8:38:46<90:32:56,  8.65s/it][2025-04-25 16:36:29,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:36:29,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.86 | bwd_microstep: 5727.38 | bwd_inner_microstep: 5714.61 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.71
[2025-04-25 16:36:29,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.86 | bwd: 5727.39 | bwd_inner: 5714.61 | bwd_allreduce: 12.74 | step: 18.71

  9%|▊         | 3586/41250 [8:38:55<90:33:43,  8.66s/it]
                                                         
{'loss': 0.0614, 'grad_norm': 1.302088975906372, 'learning_rate': 3.966108950026671e-05, 'epoch': 0.87}

  9%|▊         | 3586/41250 [8:38:55<90:33:43,  8.66s/it][2025-04-25 16:36:38,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:36:38,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.78 | bwd_microstep: 5712.18 | bwd_inner_microstep: 5660.27 | bwd_allreduce_microstep: 51.86 | step_microstep: 18.71
[2025-04-25 16:36:38,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.78 | bwd: 5712.20 | bwd_inner: 5660.27 | bwd_allreduce: 51.88 | step: 18.71

  9%|▊         | 3587/41250 [8:39:03<90:27:15,  8.65s/it]
                                                         
{'loss': 0.1576, 'grad_norm': 1.9023678302764893, 'learning_rate': 3.966080157759327e-05, 'epoch': 0.87}

  9%|▊         | 3587/41250 [8:39:03<90:27:15,  8.65s/it][2025-04-25 16:36:47,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 16:36:47,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.74 | bwd_microstep: 5748.31 | bwd_inner_microstep: 5696.07 | bwd_allreduce_microstep: 52.20 | step_microstep: 19.30
[2025-04-25 16:36:47,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.74 | bwd: 5748.33 | bwd_inner: 5696.07 | bwd_allreduce: 52.22 | step: 19.30

  9%|▊         | 3588/41250 [8:39:12<90:34:32,  8.66s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 1.5701773166656494, 'learning_rate': 3.966051353371486e-05, 'epoch': 0.87}

  9%|▊         | 3588/41250 [8:39:12<90:34:32,  8.66s/it][2025-04-25 16:36:55,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:36:55,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.94 | bwd_microstep: 5783.17 | bwd_inner_microstep: 5658.02 | bwd_allreduce_microstep: 125.10 | step_microstep: 18.64
[2025-04-25 16:36:55,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.94 | bwd: 5783.18 | bwd_inner: 5658.02 | bwd_allreduce: 125.12 | step: 18.64

  9%|▊         | 3589/41250 [8:39:21<90:41:41,  8.67s/it]
                                                         
{'loss': 0.0518, 'grad_norm': 0.6492488384246826, 'learning_rate': 3.9660225368633243e-05, 'epoch': 0.87}

  9%|▊         | 3589/41250 [8:39:21<90:41:41,  8.67s/it][2025-04-25 16:37:04,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:37:04,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5769.99 | bwd_inner_microstep: 5662.86 | bwd_allreduce_microstep: 107.08 | step_microstep: 18.76
[2025-04-25 16:37:04,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5770.00 | bwd_inner: 5662.86 | bwd_allreduce: 107.10 | step: 18.76

  9%|▊         | 3590/41250 [8:39:29<90:44:21,  8.67s/it]
                                                         
{'loss': 0.2351, 'grad_norm': 2.2270216941833496, 'learning_rate': 3.9659937082350214e-05, 'epoch': 0.87}

  9%|▊         | 3590/41250 [8:39:29<90:44:21,  8.67s/it][2025-04-25 16:37:13,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.15 | optimizer_step: 0.89
[2025-04-25 16:37:13,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.02 | bwd_microstep: 5785.78 | bwd_inner_microstep: 5689.90 | bwd_allreduce_microstep: 95.83 | step_microstep: 18.97
[2025-04-25 16:37:13,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.02 | bwd: 5785.80 | bwd_inner: 5689.90 | bwd_allreduce: 95.85 | step: 18.97

  9%|▊         | 3591/41250 [8:39:38<90:51:54,  8.69s/it]
                                                         
{'loss': 0.1324, 'grad_norm': 1.5708060264587402, 'learning_rate': 3.965964867486754e-05, 'epoch': 0.87}

  9%|▊         | 3591/41250 [8:39:38<90:51:54,  8.69s/it][2025-04-25 16:37:21,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:37:21,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.30 | bwd_microstep: 5730.21 | bwd_inner_microstep: 5717.55 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.91
[2025-04-25 16:37:21,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.30 | bwd: 5730.22 | bwd_inner: 5717.55 | bwd_allreduce: 12.63 | step: 18.91

  9%|▊         | 3592/41250 [8:39:47<90:49:38,  8.68s/it]
                                                         
{'loss': 0.1344, 'grad_norm': 1.5820410251617432, 'learning_rate': 3.965936014618699e-05, 'epoch': 0.87}

  9%|▊         | 3592/41250 [8:39:47<90:49:38,  8.68s/it][2025-04-25 16:37:30,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 16:37:30,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2940.73 | bwd_microstep: 5881.19 | bwd_inner_microstep: 5868.47 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.41
[2025-04-25 16:37:30,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2940.73 | bwd: 5881.20 | bwd_inner: 5868.47 | bwd_allreduce: 12.68 | step: 19.41

  9%|▊         | 3593/41250 [8:39:56<91:31:31,  8.75s/it]
                                                         
{'loss': 0.189, 'grad_norm': 1.6894451379776, 'learning_rate': 3.965907149631036e-05, 'epoch': 0.87}

  9%|▊         | 3593/41250 [8:39:56<91:31:31,  8.75s/it][2025-04-25 16:37:39,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:37:39,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.12 | bwd_microstep: 5686.08 | bwd_inner_microstep: 5673.04 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.07
[2025-04-25 16:37:39,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.12 | bwd: 5686.09 | bwd_inner: 5673.04 | bwd_allreduce: 13.01 | step: 19.07

  9%|▊         | 3594/41250 [8:40:04<91:03:39,  8.71s/it]
                                                         
{'loss': 0.1029, 'grad_norm': 1.0374729633331299, 'learning_rate': 3.965878272523941e-05, 'epoch': 0.87}

  9%|▊         | 3594/41250 [8:40:04<91:03:39,  8.71s/it][2025-04-25 16:37:47,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:37:47,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.69 | bwd_microstep: 5714.90 | bwd_inner_microstep: 5702.09 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.67
[2025-04-25 16:37:47,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.69 | bwd: 5714.92 | bwd_inner: 5702.09 | bwd_allreduce: 12.78 | step: 18.68

  9%|▊         | 3595/41250 [8:40:13<90:52:33,  8.69s/it]
                                                         
{'loss': 0.1327, 'grad_norm': 1.7738103866577148, 'learning_rate': 3.965849383297594e-05, 'epoch': 0.87}

  9%|▊         | 3595/41250 [8:40:13<90:52:33,  8.69s/it][2025-04-25 16:37:56,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:37:56,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.24 | bwd_microstep: 5783.84 | bwd_inner_microstep: 5770.51 | bwd_allreduce_microstep: 13.28 | step_microstep: 18.93
[2025-04-25 16:37:56,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.24 | bwd: 5783.86 | bwd_inner: 5770.51 | bwd_allreduce: 13.30 | step: 18.93

  9%|▊         | 3596/41250 [8:40:22<91:05:29,  8.71s/it]
                                                         
{'loss': 0.2695, 'grad_norm': 1.9932082891464233, 'learning_rate': 3.965820481952172e-05, 'epoch': 0.87}

  9%|▊         | 3596/41250 [8:40:22<91:05:29,  8.71s/it][2025-04-25 16:38:05,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:38:05,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.27 | bwd_microstep: 5756.05 | bwd_inner_microstep: 5653.87 | bwd_allreduce_microstep: 102.13 | step_microstep: 18.62
[2025-04-25 16:38:05,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.27 | bwd: 5756.06 | bwd_inner: 5653.87 | bwd_allreduce: 102.16 | step: 18.63

  9%|▊         | 3597/41250 [8:40:30<90:57:41,  8.70s/it]
                                                         
{'loss': 0.1556, 'grad_norm': 1.139983892440796, 'learning_rate': 3.965791568487853e-05, 'epoch': 0.87}

  9%|▊         | 3597/41250 [8:40:30<90:57:41,  8.70s/it][2025-04-25 16:38:14,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 16:38:14,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5749.21 | bwd_inner_microstep: 5694.91 | bwd_allreduce_microstep: 54.25 | step_microstep: 18.48
[2025-04-25 16:38:14,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5749.22 | bwd_inner: 5694.91 | bwd_allreduce: 54.27 | step: 18.49

  9%|▊         | 3598/41250 [8:40:39<90:54:31,  8.69s/it]
                                                         
{'loss': 0.2343, 'grad_norm': 2.048961877822876, 'learning_rate': 3.9657626429048164e-05, 'epoch': 0.87}

  9%|▊         | 3598/41250 [8:40:39<90:54:31,  8.69s/it][2025-04-25 16:38:22,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 16:38:22,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.86 | bwd_microstep: 5878.56 | bwd_inner_microstep: 5726.33 | bwd_allreduce_microstep: 152.17 | step_microstep: 19.13
[2025-04-25 16:38:22,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.86 | bwd: 5878.57 | bwd_inner: 5726.33 | bwd_allreduce: 152.19 | step: 19.14

  9%|▊         | 3599/41250 [8:40:48<91:18:01,  8.73s/it]
                                                         
{'loss': 0.2363, 'grad_norm': 1.1887539625167847, 'learning_rate': 3.965733705203239e-05, 'epoch': 0.87}

  9%|▊         | 3599/41250 [8:40:48<91:18:01,  8.73s/it][2025-04-25 16:38:31,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 16:38:31,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5737.67 | bwd_inner_microstep: 5706.43 | bwd_allreduce_microstep: 31.20 | step_microstep: 19.02
[2025-04-25 16:38:31,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5737.69 | bwd_inner: 5706.43 | bwd_allreduce: 31.22 | step: 19.02

  9%|▊         | 3600/41250 [8:40:56<91:06:41,  8.71s/it]
                                                         
{'loss': 0.2682, 'grad_norm': 2.3212687969207764, 'learning_rate': 3.9657047553833006e-05, 'epoch': 0.87}

  9%|▊         | 3600/41250 [8:40:56<91:06:41,  8.71s/it][2025-04-25 16:38:40,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:38:40,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.46 | bwd_microstep: 5711.82 | bwd_inner_microstep: 5699.03 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.14
[2025-04-25 16:38:40,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.46 | bwd: 5711.84 | bwd_inner: 5699.03 | bwd_allreduce: 12.76 | step: 19.14

  9%|▊         | 3601/41250 [8:41:05<90:54:01,  8.69s/it]
                                                         
{'loss': 0.2152, 'grad_norm': 3.5453455448150635, 'learning_rate': 3.9656757934451794e-05, 'epoch': 0.87}

  9%|▊         | 3601/41250 [8:41:05<90:54:01,  8.69s/it][2025-04-25 16:38:48,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:38:48,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.65 | bwd_microstep: 5714.14 | bwd_inner_microstep: 5701.49 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.61
[2025-04-25 16:38:48,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.66 | bwd: 5714.15 | bwd_inner: 5701.49 | bwd_allreduce: 12.62 | step: 18.62

  9%|▊         | 3602/41250 [8:41:14<90:45:26,  8.68s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.62545645236969, 'learning_rate': 3.9656468193890525e-05, 'epoch': 0.87}

  9%|▊         | 3602/41250 [8:41:14<90:45:26,  8.68s/it][2025-04-25 16:38:57,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:38:57,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.19 | bwd_microstep: 5711.26 | bwd_inner_microstep: 5698.28 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.93
[2025-04-25 16:38:57,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.19 | bwd: 5711.28 | bwd_inner: 5698.28 | bwd_allreduce: 12.95 | step: 18.93

  9%|▊         | 3603/41250 [8:41:22<90:39:58,  8.67s/it]
                                                         
{'loss': 0.1332, 'grad_norm': 1.7375733852386475, 'learning_rate': 3.9656178332151e-05, 'epoch': 0.87}

  9%|▊         | 3603/41250 [8:41:22<90:39:58,  8.67s/it][2025-04-25 16:39:06,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.18
[2025-04-25 16:39:06,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5702.98 | bwd_inner_microstep: 5666.29 | bwd_allreduce_microstep: 36.64 | step_microstep: 19.37
[2025-04-25 16:39:06,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5702.99 | bwd_inner: 5666.29 | bwd_allreduce: 36.66 | step: 19.37

  9%|▊         | 3604/41250 [8:41:31<90:29:53,  8.65s/it]
                                                         
{'loss': 0.2128, 'grad_norm': 2.5706417560577393, 'learning_rate': 3.9655888349235e-05, 'epoch': 0.87}

  9%|▊         | 3604/41250 [8:41:31<90:29:53,  8.65s/it][2025-04-25 16:39:14,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 16:39:14,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5765.93 | bwd_inner_microstep: 5706.26 | bwd_allreduce_microstep: 59.63 | step_microstep: 18.48
[2025-04-25 16:39:14,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5765.94 | bwd_inner: 5706.26 | bwd_allreduce: 59.65 | step: 18.49

  9%|▊         | 3605/41250 [8:41:40<90:38:17,  8.67s/it]
                                                         
{'loss': 0.1647, 'grad_norm': 3.33087420463562, 'learning_rate': 3.965559824514432e-05, 'epoch': 0.87}

  9%|▊         | 3605/41250 [8:41:40<90:38:17,  8.67s/it][2025-04-25 16:39:23,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 16:39:23,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.38 | bwd_microstep: 5720.26 | bwd_inner_microstep: 5707.51 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.88
[2025-04-25 16:39:23,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.38 | bwd: 5720.27 | bwd_inner: 5707.51 | bwd_allreduce: 12.72 | step: 18.89

  9%|▊         | 3606/41250 [8:41:48<90:35:16,  8.66s/it]
                                                         
{'loss': 0.0534, 'grad_norm': 0.8552536964416504, 'learning_rate': 3.965530801988074e-05, 'epoch': 0.87}

  9%|▊         | 3606/41250 [8:41:48<90:35:16,  8.66s/it][2025-04-25 16:39:32,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:39:32,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.27 | bwd_microstep: 5747.82 | bwd_inner_microstep: 5695.05 | bwd_allreduce_microstep: 52.72 | step_microstep: 18.87
[2025-04-25 16:39:32,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.27 | bwd: 5747.83 | bwd_inner: 5695.05 | bwd_allreduce: 52.73 | step: 18.88

  9%|▊         | 3607/41250 [8:41:57<90:37:54,  8.67s/it]
                                                         
{'loss': 0.1035, 'grad_norm': 1.0577189922332764, 'learning_rate': 3.9655017673446046e-05, 'epoch': 0.87}

  9%|▊         | 3607/41250 [8:41:57<90:37:54,  8.67s/it][2025-04-25 16:39:40,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 16:39:40,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.56 | bwd_microstep: 5891.15 | bwd_inner_microstep: 5649.94 | bwd_allreduce_microstep: 241.16 | step_microstep: 18.89
[2025-04-25 16:39:40,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.56 | bwd: 5891.16 | bwd_inner: 5649.94 | bwd_allreduce: 241.18 | step: 18.89

  9%|▊         | 3608/41250 [8:42:06<91:04:40,  8.71s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.9998297691345215, 'learning_rate': 3.965472720584204e-05, 'epoch': 0.87}

  9%|▊         | 3608/41250 [8:42:06<91:04:40,  8.71s/it][2025-04-25 16:39:49,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:39:49,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.77 | bwd_microstep: 5707.19 | bwd_inner_microstep: 5655.93 | bwd_allreduce_microstep: 51.21 | step_microstep: 18.85
[2025-04-25 16:39:49,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.77 | bwd: 5707.20 | bwd_inner: 5655.93 | bwd_allreduce: 51.23 | step: 18.86

  9%|▊         | 3609/41250 [8:42:14<90:46:39,  8.68s/it]
                                                         
{'loss': 0.2216, 'grad_norm': 1.2932417392730713, 'learning_rate': 3.965443661707051e-05, 'epoch': 0.87}

  9%|▊         | 3609/41250 [8:42:14<90:46:39,  8.68s/it][2025-04-25 16:39:58,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:39:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.17 | bwd_microstep: 5705.59 | bwd_inner_microstep: 5653.61 | bwd_allreduce_microstep: 51.93 | step_microstep: 18.75
[2025-04-25 16:39:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.17 | bwd: 5705.60 | bwd_inner: 5653.61 | bwd_allreduce: 51.95 | step: 18.76

  9%|▉         | 3610/41250 [8:42:23<90:32:34,  8.66s/it]
                                                         
{'loss': 0.282, 'grad_norm': 1.7565383911132812, 'learning_rate': 3.9654145907133235e-05, 'epoch': 0.88}

  9%|▉         | 3610/41250 [8:42:23<90:32:34,  8.66s/it][2025-04-25 16:40:06,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:40:06,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.34 | bwd_microstep: 5715.43 | bwd_inner_microstep: 5694.58 | bwd_allreduce_microstep: 20.80 | step_microstep: 19.00
[2025-04-25 16:40:06,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.34 | bwd: 5715.44 | bwd_inner: 5694.58 | bwd_allreduce: 20.82 | step: 19.01

  9%|▉         | 3611/41250 [8:42:32<90:31:47,  8.66s/it]
                                                         
{'loss': 0.1066, 'grad_norm': 0.9711330533027649, 'learning_rate': 3.965385507603202e-05, 'epoch': 0.88}

  9%|▉         | 3611/41250 [8:42:32<90:31:47,  8.66s/it][2025-04-25 16:40:15,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:40:15,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5714.61 | bwd_inner_microstep: 5701.69 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.90
[2025-04-25 16:40:15,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.21 | bwd: 5714.62 | bwd_inner: 5701.69 | bwd_allreduce: 12.89 | step: 18.91

  9%|▉         | 3612/41250 [8:42:40<90:30:05,  8.66s/it]
                                                         
{'loss': 0.1529, 'grad_norm': 2.8071541786193848, 'learning_rate': 3.9653564123768655e-05, 'epoch': 0.88}

  9%|▉         | 3612/41250 [8:42:40<90:30:05,  8.66s/it][2025-04-25 16:40:24,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 16:40:24,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.93 | bwd_microstep: 5706.14 | bwd_inner_microstep: 5693.29 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.46
[2025-04-25 16:40:24,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.93 | bwd: 5706.15 | bwd_inner: 5693.29 | bwd_allreduce: 12.82 | step: 19.46

  9%|▉         | 3613/41250 [8:42:49<90:26:00,  8.65s/it]
                                                         
{'loss': 0.2683, 'grad_norm': 2.1968109607696533, 'learning_rate': 3.9653273050344936e-05, 'epoch': 0.88}

  9%|▉         | 3613/41250 [8:42:49<90:26:00,  8.65s/it][2025-04-25 16:40:32,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:40:32,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.67 | bwd_microstep: 5774.68 | bwd_inner_microstep: 5761.95 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.67
[2025-04-25 16:40:32,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.67 | bwd: 5774.69 | bwd_inner: 5761.95 | bwd_allreduce: 12.71 | step: 18.68

  9%|▉         | 3614/41250 [8:42:58<90:43:48,  8.68s/it]
                                                         
{'loss': 0.1542, 'grad_norm': 1.1466267108917236, 'learning_rate': 3.965298185576265e-05, 'epoch': 0.88}

  9%|▉         | 3614/41250 [8:42:58<90:43:48,  8.68s/it][2025-04-25 16:40:41,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 16:40:41,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.67 | bwd_microstep: 5682.44 | bwd_inner_microstep: 5638.93 | bwd_allreduce_microstep: 43.46 | step_microstep: 18.42
[2025-04-25 16:40:41,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.67 | bwd: 5682.45 | bwd_inner: 5638.93 | bwd_allreduce: 43.47 | step: 18.42

  9%|▉         | 3615/41250 [8:43:06<90:26:31,  8.65s/it]
                                                         
{'loss': 0.3064, 'grad_norm': 2.000308036804199, 'learning_rate': 3.965269054002359e-05, 'epoch': 0.88}

  9%|▉         | 3615/41250 [8:43:06<90:26:31,  8.65s/it][2025-04-25 16:40:50,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:40:50,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.47 | bwd_microstep: 5741.59 | bwd_inner_microstep: 5651.47 | bwd_allreduce_microstep: 90.07 | step_microstep: 18.66
[2025-04-25 16:40:50,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.47 | bwd: 5741.60 | bwd_inner: 5651.47 | bwd_allreduce: 90.09 | step: 18.66

  9%|▉         | 3616/41250 [8:43:15<90:25:47,  8.65s/it]
                                                         
{'loss': 0.2279, 'grad_norm': 2.0018467903137207, 'learning_rate': 3.965239910312957e-05, 'epoch': 0.88}

  9%|▉         | 3616/41250 [8:43:15<90:25:47,  8.65s/it][2025-04-25 16:40:58,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:40:58,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5738.45 | bwd_inner_microstep: 5648.49 | bwd_allreduce_microstep: 89.92 | step_microstep: 18.50
[2025-04-25 16:40:58,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.09 | bwd: 5738.47 | bwd_inner: 5648.49 | bwd_allreduce: 89.93 | step: 18.50

  9%|▉         | 3617/41250 [8:43:24<90:24:43,  8.65s/it]
                                                         
{'loss': 0.2971, 'grad_norm': 1.9142403602600098, 'learning_rate': 3.9652107545082376e-05, 'epoch': 0.88}

  9%|▉         | 3617/41250 [8:43:24<90:24:43,  8.65s/it][2025-04-25 16:41:07,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:41:07,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5741.34 | bwd_inner_microstep: 5698.34 | bwd_allreduce_microstep: 42.95 | step_microstep: 18.89
[2025-04-25 16:41:07,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5741.35 | bwd_inner: 5698.34 | bwd_allreduce: 42.97 | step: 18.90

  9%|▉         | 3618/41250 [8:43:32<90:29:16,  8.66s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 1.205641508102417, 'learning_rate': 3.9651815865883794e-05, 'epoch': 0.88}

  9%|▉         | 3618/41250 [8:43:32<90:29:16,  8.66s/it][2025-04-25 16:41:16,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:41:16,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.71 | bwd_microstep: 5777.79 | bwd_inner_microstep: 5660.10 | bwd_allreduce_microstep: 117.65 | step_microstep: 18.70
[2025-04-25 16:41:16,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.71 | bwd: 5777.81 | bwd_inner: 5660.10 | bwd_allreduce: 117.67 | step: 18.70

  9%|▉         | 3619/41250 [8:43:41<90:35:17,  8.67s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.7546906471252441, 'learning_rate': 3.965152406553564e-05, 'epoch': 0.88}

  9%|▉         | 3619/41250 [8:43:41<90:35:17,  8.67s/it][2025-04-25 16:41:24,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-25 16:41:24,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.37 | bwd_microstep: 5699.17 | bwd_inner_microstep: 5649.76 | bwd_allreduce_microstep: 49.36 | step_microstep: 19.50
[2025-04-25 16:41:24,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.37 | bwd: 5699.19 | bwd_inner: 5649.76 | bwd_allreduce: 49.38 | step: 19.50

  9%|▉         | 3620/41250 [8:43:50<90:23:23,  8.65s/it]
                                                         
{'loss': 0.0786, 'grad_norm': 0.7494755387306213, 'learning_rate': 3.96512321440397e-05, 'epoch': 0.88}

  9%|▉         | 3620/41250 [8:43:50<90:23:23,  8.65s/it][2025-04-25 16:41:33,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:41:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.74 | bwd_microstep: 5699.81 | bwd_inner_microstep: 5687.03 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.59
[2025-04-25 16:41:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.74 | bwd: 5699.83 | bwd_inner: 5687.03 | bwd_allreduce: 12.76 | step: 18.60

  9%|▉         | 3621/41250 [8:43:58<90:19:18,  8.64s/it]
                                                         
{'loss': 0.1254, 'grad_norm': 0.9129690527915955, 'learning_rate': 3.9650940101397785e-05, 'epoch': 0.88}

  9%|▉         | 3621/41250 [8:43:58<90:19:18,  8.64s/it][2025-04-25 16:41:41,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:41:41,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.27 | bwd_microstep: 5699.78 | bwd_inner_microstep: 5643.33 | bwd_allreduce_microstep: 56.40 | step_microstep: 18.58
[2025-04-25 16:41:41,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.27 | bwd: 5699.79 | bwd_inner: 5643.33 | bwd_allreduce: 56.42 | step: 18.58

  9%|▉         | 3622/41250 [8:44:07<90:11:52,  8.63s/it]
                                                         
{'loss': 0.0935, 'grad_norm': 0.7458563446998596, 'learning_rate': 3.965064793761169e-05, 'epoch': 0.88}

  9%|▉         | 3622/41250 [8:44:07<90:11:52,  8.63s/it][2025-04-25 16:41:50,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:41:50,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5673.44 | bwd_inner_microstep: 5634.53 | bwd_allreduce_microstep: 38.87 | step_microstep: 19.01
[2025-04-25 16:41:50,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5673.46 | bwd_inner: 5634.53 | bwd_allreduce: 38.89 | step: 19.01

  9%|▉         | 3623/41250 [8:44:15<90:02:17,  8.61s/it]
                                                         
{'loss': 0.161, 'grad_norm': 2.2006759643554688, 'learning_rate': 3.965035565268321e-05, 'epoch': 0.88}

  9%|▉         | 3623/41250 [8:44:15<90:02:17,  8.61s/it][2025-04-25 16:41:59,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:41:59,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.24 | bwd_microstep: 5779.29 | bwd_inner_microstep: 5766.44 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.01
[2025-04-25 16:41:59,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.24 | bwd: 5779.30 | bwd_inner: 5766.44 | bwd_allreduce: 12.82 | step: 19.02

  9%|▉         | 3624/41250 [8:44:24<90:26:40,  8.65s/it]
                                                         
{'loss': 0.0996, 'grad_norm': 0.8995067477226257, 'learning_rate': 3.965006324661415e-05, 'epoch': 0.88}

  9%|▉         | 3624/41250 [8:44:24<90:26:40,  8.65s/it][2025-04-25 16:42:07,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 16:42:07,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.89 | bwd_microstep: 5697.19 | bwd_inner_microstep: 5683.86 | bwd_allreduce_microstep: 13.28 | step_microstep: 18.93
[2025-04-25 16:42:07,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.89 | bwd: 5697.21 | bwd_inner: 5683.86 | bwd_allreduce: 13.30 | step: 18.93

  9%|▉         | 3625/41250 [8:44:33<90:21:50,  8.65s/it]
                                                         
{'loss': 0.3005, 'grad_norm': 1.0676568746566772, 'learning_rate': 3.9649770719406325e-05, 'epoch': 0.88}

  9%|▉         | 3625/41250 [8:44:33<90:21:50,  8.65s/it][2025-04-25 16:42:16,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 16:42:16,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.87 | bwd_microstep: 5725.80 | bwd_inner_microstep: 5706.46 | bwd_allreduce_microstep: 19.29 | step_microstep: 19.06
[2025-04-25 16:42:16,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.87 | bwd: 5725.81 | bwd_inner: 5706.46 | bwd_allreduce: 19.30 | step: 19.07

  9%|▉         | 3626/41250 [8:44:41<90:26:05,  8.65s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 2.585141181945801, 'learning_rate': 3.964947807106152e-05, 'epoch': 0.88}

  9%|▉         | 3626/41250 [8:44:41<90:26:05,  8.65s/it][2025-04-25 16:42:25,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:42:25,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 6021.89 | bwd_inner_microstep: 5644.14 | bwd_allreduce_microstep: 377.71 | step_microstep: 18.33
[2025-04-25 16:42:25,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 6021.91 | bwd_inner: 5644.14 | bwd_allreduce: 377.72 | step: 18.33

  9%|▉         | 3627/41250 [8:44:50<91:19:03,  8.74s/it]
                                                         
{'loss': 0.0439, 'grad_norm': 1.713860273361206, 'learning_rate': 3.964918530158155e-05, 'epoch': 0.88}

  9%|▉         | 3627/41250 [8:44:50<91:19:03,  8.74s/it][2025-04-25 16:42:34,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:42:34,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5670.75 | bwd_inner_microstep: 5635.59 | bwd_allreduce_microstep: 35.11 | step_microstep: 18.65
[2025-04-25 16:42:34,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5670.76 | bwd_inner: 5635.59 | bwd_allreduce: 35.13 | step: 18.66

  9%|▉         | 3628/41250 [8:44:59<90:50:03,  8.69s/it]
                                                         
{'loss': 0.1929, 'grad_norm': 1.490341067314148, 'learning_rate': 3.964889241096821e-05, 'epoch': 0.88}

  9%|▉         | 3628/41250 [8:44:59<90:50:03,  8.69s/it][2025-04-25 16:42:42,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 16:42:42,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.23 | bwd_microstep: 5740.40 | bwd_inner_microstep: 5644.45 | bwd_allreduce_microstep: 95.90 | step_microstep: 18.57
[2025-04-25 16:42:42,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.23 | bwd: 5740.41 | bwd_inner: 5644.45 | bwd_allreduce: 95.92 | step: 18.57

  9%|▉         | 3629/41250 [8:45:08<90:40:30,  8.68s/it]
                                                         
{'loss': 0.1864, 'grad_norm': 1.5890564918518066, 'learning_rate': 3.964859939922333e-05, 'epoch': 0.88}

  9%|▉         | 3629/41250 [8:45:08<90:40:30,  8.68s/it][2025-04-25 16:42:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:42:51,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5722.47 | bwd_inner_microstep: 5698.18 | bwd_allreduce_microstep: 24.25 | step_microstep: 18.45
[2025-04-25 16:42:51,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5722.49 | bwd_inner: 5698.18 | bwd_allreduce: 24.26 | step: 18.45

  9%|▉         | 3630/41250 [8:45:16<90:36:26,  8.67s/it]
                                                         
{'loss': 0.1195, 'grad_norm': 1.1443747282028198, 'learning_rate': 3.964830626634869e-05, 'epoch': 0.88}

  9%|▉         | 3630/41250 [8:45:16<90:36:26,  8.67s/it][2025-04-25 16:43:00,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 16:43:00,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.60 | bwd_microstep: 5895.78 | bwd_inner_microstep: 5649.62 | bwd_allreduce_microstep: 246.11 | step_microstep: 19.17
[2025-04-25 16:43:00,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.60 | bwd: 5895.79 | bwd_inner: 5649.62 | bwd_allreduce: 246.13 | step: 19.17

  9%|▉         | 3631/41250 [8:45:25<91:01:27,  8.71s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 1.0471911430358887, 'learning_rate': 3.964801301234611e-05, 'epoch': 0.88}

  9%|▉         | 3631/41250 [8:45:25<91:01:27,  8.71s/it][2025-04-25 16:43:08,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 16:43:08,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.24 | bwd_microstep: 5706.43 | bwd_inner_microstep: 5647.02 | bwd_allreduce_microstep: 59.36 | step_microstep: 18.64
[2025-04-25 16:43:08,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.24 | bwd: 5706.44 | bwd_inner: 5647.02 | bwd_allreduce: 59.38 | step: 18.64

  9%|▉         | 3632/41250 [8:45:34<90:43:29,  8.68s/it]
                                                         
{'loss': 0.207, 'grad_norm': 2.1062185764312744, 'learning_rate': 3.9647719637217386e-05, 'epoch': 0.88}

  9%|▉         | 3632/41250 [8:45:34<90:43:29,  8.68s/it][2025-04-25 16:43:17,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:43:17,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2940.40 | bwd_microstep: 5863.00 | bwd_inner_microstep: 5850.19 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.46
[2025-04-25 16:43:17,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2940.40 | bwd: 5863.01 | bwd_inner: 5850.19 | bwd_allreduce: 12.79 | step: 18.46

  9%|▉         | 3633/41250 [8:45:43<91:22:18,  8.74s/it]
                                                         
{'loss': 0.2054, 'grad_norm': 1.3672858476638794, 'learning_rate': 3.9647426140964335e-05, 'epoch': 0.88}

  9%|▉         | 3633/41250 [8:45:43<91:22:18,  8.74s/it][2025-04-25 16:43:26,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:43:26,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.01 | bwd_microstep: 5753.52 | bwd_inner_microstep: 5697.84 | bwd_allreduce_microstep: 55.64 | step_microstep: 18.39
[2025-04-25 16:43:26,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.01 | bwd: 5753.53 | bwd_inner: 5697.84 | bwd_allreduce: 55.65 | step: 18.39

  9%|▉         | 3634/41250 [8:45:51<91:12:36,  8.73s/it]
                                                         
{'loss': 0.1223, 'grad_norm': 1.209355115890503, 'learning_rate': 3.964713252358877e-05, 'epoch': 0.88}

  9%|▉         | 3634/41250 [8:45:51<91:12:36,  8.73s/it][2025-04-25 16:43:35,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:43:35,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5703.74 | bwd_inner_microstep: 5660.79 | bwd_allreduce_microstep: 42.90 | step_microstep: 18.91
[2025-04-25 16:43:35,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5703.76 | bwd_inner: 5660.79 | bwd_allreduce: 42.92 | step: 18.91

  9%|▉         | 3635/41250 [8:46:00<90:51:55,  8.70s/it]
                                                         
{'loss': 0.1387, 'grad_norm': 3.3649349212646484, 'learning_rate': 3.96468387850925e-05, 'epoch': 0.88}

  9%|▉         | 3635/41250 [8:46:00<90:51:55,  8.70s/it][2025-04-25 16:43:43,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 16:43:43,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.43 | bwd_microstep: 5795.00 | bwd_inner_microstep: 5782.08 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.23
[2025-04-25 16:43:43,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.44 | bwd: 5795.01 | bwd_inner: 5782.08 | bwd_allreduce: 12.90 | step: 19.23

  9%|▉         | 3636/41250 [8:46:09<91:05:30,  8.72s/it]
                                                         
{'loss': 0.1389, 'grad_norm': 1.9098244905471802, 'learning_rate': 3.9646544925477334e-05, 'epoch': 0.88}

  9%|▉         | 3636/41250 [8:46:09<91:05:30,  8.72s/it][2025-04-25 16:43:52,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:43:52,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.63 | bwd_microstep: 5699.84 | bwd_inner_microstep: 5663.60 | bwd_allreduce_microstep: 36.19 | step_microstep: 18.38
[2025-04-25 16:43:52,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.63 | bwd: 5699.85 | bwd_inner: 5663.60 | bwd_allreduce: 36.21 | step: 18.38

  9%|▉         | 3637/41250 [8:46:17<90:48:10,  8.69s/it]
                                                         
{'loss': 0.0795, 'grad_norm': 0.9894154667854309, 'learning_rate': 3.9646250944745086e-05, 'epoch': 0.88}

  9%|▉         | 3637/41250 [8:46:17<90:48:10,  8.69s/it][2025-04-25 16:44:01,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:44:01,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.41 | bwd_microstep: 5754.11 | bwd_inner_microstep: 5657.80 | bwd_allreduce_microstep: 96.27 | step_microstep: 18.40
[2025-04-25 16:44:01,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.41 | bwd: 5754.13 | bwd_inner: 5657.80 | bwd_allreduce: 96.29 | step: 18.40

  9%|▉         | 3638/41250 [8:46:26<90:42:56,  8.68s/it]
                                                         
{'loss': 0.1773, 'grad_norm': 1.281056523323059, 'learning_rate': 3.9645956842897555e-05, 'epoch': 0.88}

  9%|▉         | 3638/41250 [8:46:26<90:42:56,  8.68s/it][2025-04-25 16:44:09,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 16:44:09,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.89 | bwd_microstep: 5784.64 | bwd_inner_microstep: 5771.87 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.40
[2025-04-25 16:44:09,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.89 | bwd: 5784.66 | bwd_inner: 5771.87 | bwd_allreduce: 12.74 | step: 19.40

  9%|▉         | 3639/41250 [8:46:35<90:56:14,  8.70s/it]
                                                         
{'loss': 0.1318, 'grad_norm': 0.7776308655738831, 'learning_rate': 3.964566261993657e-05, 'epoch': 0.88}

  9%|▉         | 3639/41250 [8:46:35<90:56:14,  8.70s/it][2025-04-25 16:44:18,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 16:44:18,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5761.23 | bwd_inner_microstep: 5694.27 | bwd_allreduce_microstep: 66.92 | step_microstep: 18.95
[2025-04-25 16:44:18,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5761.25 | bwd_inner: 5694.27 | bwd_allreduce: 66.94 | step: 18.96

  9%|▉         | 3640/41250 [8:46:43<90:54:42,  8.70s/it]
                                                         
{'loss': 0.2346, 'grad_norm': 2.1307458877563477, 'learning_rate': 3.964536827586393e-05, 'epoch': 0.88}

  9%|▉         | 3640/41250 [8:46:43<90:54:42,  8.70s/it][2025-04-25 16:44:27,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:44:27,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5785.27 | bwd_inner_microstep: 5694.32 | bwd_allreduce_microstep: 90.90 | step_microstep: 18.74
[2025-04-25 16:44:27,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.07 | bwd: 5785.28 | bwd_inner: 5694.32 | bwd_allreduce: 90.92 | step: 18.74

  9%|▉         | 3641/41250 [8:46:52<90:56:40,  8.71s/it]
                                                         
{'loss': 0.1363, 'grad_norm': 0.8395376801490784, 'learning_rate': 3.9645073810681475e-05, 'epoch': 0.88}

  9%|▉         | 3641/41250 [8:46:52<90:56:40,  8.71s/it][2025-04-25 16:44:36,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:44:36,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.09 | bwd_microstep: 5797.88 | bwd_inner_microstep: 5785.10 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.82
[2025-04-25 16:44:36,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.09 | bwd: 5797.90 | bwd_inner: 5785.10 | bwd_allreduce: 12.75 | step: 18.82

  9%|▉         | 3642/41250 [8:47:01<91:09:12,  8.73s/it]
                                                         
{'loss': 0.1782, 'grad_norm': 1.7028374671936035, 'learning_rate': 3.9644779224390994e-05, 'epoch': 0.88}

  9%|▉         | 3642/41250 [8:47:01<91:09:12,  8.73s/it][2025-04-25 16:44:44,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:44:44,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.83 | bwd_microstep: 5708.72 | bwd_inner_microstep: 5695.83 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.10
[2025-04-25 16:44:44,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.83 | bwd: 5708.73 | bwd_inner: 5695.83 | bwd_allreduce: 12.86 | step: 19.10

  9%|▉         | 3643/41250 [8:47:09<90:53:37,  8.70s/it]
                                                         
{'loss': 0.397, 'grad_norm': 2.3305466175079346, 'learning_rate': 3.964448451699432e-05, 'epoch': 0.88}

  9%|▉         | 3643/41250 [8:47:09<90:53:37,  8.70s/it][2025-04-25 16:44:53,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 16:44:53,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.21 | bwd_microstep: 5772.88 | bwd_inner_microstep: 5651.62 | bwd_allreduce_microstep: 121.21 | step_microstep: 18.91
[2025-04-25 16:44:53,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.21 | bwd: 5772.89 | bwd_inner: 5651.62 | bwd_allreduce: 121.23 | step: 18.91

  9%|▉         | 3644/41250 [8:47:18<90:51:06,  8.70s/it]
                                                         
{'loss': 0.1439, 'grad_norm': 1.5894447565078735, 'learning_rate': 3.964418968849325e-05, 'epoch': 0.88}

  9%|▉         | 3644/41250 [8:47:18<90:51:06,  8.70s/it][2025-04-25 16:45:02,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:45:02,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5760.59 | bwd_inner_microstep: 5693.76 | bwd_allreduce_microstep: 66.79 | step_microstep: 18.52
[2025-04-25 16:45:02,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5760.60 | bwd_inner: 5693.76 | bwd_allreduce: 66.80 | step: 18.52

  9%|▉         | 3645/41250 [8:47:27<90:49:52,  8.70s/it]
                                                         
{'loss': 0.0814, 'grad_norm': 1.4036723375320435, 'learning_rate': 3.964389473888962e-05, 'epoch': 0.88}

  9%|▉         | 3645/41250 [8:47:27<90:49:52,  8.70s/it][2025-04-25 16:45:10,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 16:45:10,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.97 | bwd_microstep: 5724.99 | bwd_inner_microstep: 5652.18 | bwd_allreduce_microstep: 72.75 | step_microstep: 19.05
[2025-04-25 16:45:10,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.97 | bwd: 5725.00 | bwd_inner: 5652.18 | bwd_allreduce: 72.78 | step: 19.05

  9%|▉         | 3646/41250 [8:47:36<90:38:41,  8.68s/it]
                                                         
{'loss': 0.3087, 'grad_norm': 2.143726110458374, 'learning_rate': 3.9643599668185246e-05, 'epoch': 0.88}

  9%|▉         | 3646/41250 [8:47:36<90:38:41,  8.68s/it][2025-04-25 16:45:19,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 16:45:19,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.89 | bwd_microstep: 5748.28 | bwd_inner_microstep: 5700.28 | bwd_allreduce_microstep: 47.95 | step_microstep: 19.11
[2025-04-25 16:45:19,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.89 | bwd: 5748.29 | bwd_inner: 5700.28 | bwd_allreduce: 47.97 | step: 19.12

  9%|▉         | 3647/41250 [8:47:44<90:40:07,  8.68s/it]
                                                         
{'loss': 0.1069, 'grad_norm': 1.156640648841858, 'learning_rate': 3.9643304476381946e-05, 'epoch': 0.88}

  9%|▉         | 3647/41250 [8:47:44<90:40:07,  8.68s/it][2025-04-25 16:45:28,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:45:28,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.18 | bwd_microstep: 5767.50 | bwd_inner_microstep: 5694.43 | bwd_allreduce_microstep: 73.02 | step_microstep: 18.89
[2025-04-25 16:45:28,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.18 | bwd: 5767.52 | bwd_inner: 5694.43 | bwd_allreduce: 73.03 | step: 18.89

  9%|▉         | 3648/41250 [8:47:53<90:44:42,  8.69s/it]
                                                         
{'loss': 0.2504, 'grad_norm': 1.5541150569915771, 'learning_rate': 3.964300916348153e-05, 'epoch': 0.88}

  9%|▉         | 3648/41250 [8:47:53<90:44:42,  8.69s/it][2025-04-25 16:45:36,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 16:45:36,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.12 | bwd_microstep: 5712.10 | bwd_inner_microstep: 5657.78 | bwd_allreduce_microstep: 54.27 | step_microstep: 18.92
[2025-04-25 16:45:36,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.12 | bwd: 5712.11 | bwd_inner: 5657.78 | bwd_allreduce: 54.29 | step: 18.92

  9%|▉         | 3649/41250 [8:48:02<90:33:35,  8.67s/it]
                                                         
{'loss': 0.2798, 'grad_norm': 4.088624000549316, 'learning_rate': 3.964271372948583e-05, 'epoch': 0.88}

  9%|▉         | 3649/41250 [8:48:02<90:33:35,  8.67s/it][2025-04-25 16:45:45,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 16:45:45,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.16 | bwd_microstep: 5722.25 | bwd_inner_microstep: 5703.82 | bwd_allreduce_microstep: 18.36 | step_microstep: 18.92
[2025-04-25 16:45:45,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.16 | bwd: 5722.26 | bwd_inner: 5703.82 | bwd_allreduce: 18.39 | step: 18.93

  9%|▉         | 3650/41250 [8:48:10<90:33:53,  8.67s/it]
                                                         
{'loss': 0.4032, 'grad_norm': 3.6866538524627686, 'learning_rate': 3.9642418174396665e-05, 'epoch': 0.88}

  9%|▉         | 3650/41250 [8:48:10<90:33:53,  8.67s/it][2025-04-25 16:45:54,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 16:45:54,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.89 | bwd_microstep: 5753.48 | bwd_inner_microstep: 5651.14 | bwd_allreduce_microstep: 102.29 | step_microstep: 18.89
[2025-04-25 16:45:54,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.89 | bwd: 5753.50 | bwd_inner: 5651.14 | bwd_allreduce: 102.31 | step: 18.89

  9%|▉         | 3651/41250 [8:48:19<90:33:51,  8.67s/it]
                                                         
{'loss': 0.1824, 'grad_norm': 1.223846673965454, 'learning_rate': 3.9642122498215846e-05, 'epoch': 0.89}

  9%|▉         | 3651/41250 [8:48:19<90:33:51,  8.67s/it][2025-04-25 16:46:02,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 16:46:02,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5759.17 | bwd_inner_microstep: 5659.92 | bwd_allreduce_microstep: 99.20 | step_microstep: 19.29
[2025-04-25 16:46:02,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5759.19 | bwd_inner: 5659.92 | bwd_allreduce: 99.22 | step: 19.30

  9%|▉         | 3652/41250 [8:48:28<90:34:59,  8.67s/it]
                                                         
{'loss': 0.2287, 'grad_norm': 1.750367522239685, 'learning_rate': 3.964182670094522e-05, 'epoch': 0.89}

  9%|▉         | 3652/41250 [8:48:28<90:34:59,  8.67s/it][2025-04-25 16:46:11,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-25 16:46:11,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.88 | bwd_microstep: 5701.62 | bwd_inner_microstep: 5687.70 | bwd_allreduce_microstep: 13.86 | step_microstep: 19.61
[2025-04-25 16:46:11,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.88 | bwd: 5701.63 | bwd_inner: 5687.70 | bwd_allreduce: 13.89 | step: 19.61

  9%|▉         | 3653/41250 [8:48:36<90:28:52,  8.66s/it]
                                                         
{'loss': 0.2342, 'grad_norm': 1.6540534496307373, 'learning_rate': 3.9641530782586586e-05, 'epoch': 0.89}

  9%|▉         | 3653/41250 [8:48:36<90:28:52,  8.66s/it][2025-04-25 16:46:19,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 16:46:19,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.03 | bwd_microstep: 5705.04 | bwd_inner_microstep: 5663.53 | bwd_allreduce_microstep: 41.46 | step_microstep: 18.96
[2025-04-25 16:46:19,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.03 | bwd: 5705.05 | bwd_inner: 5663.53 | bwd_allreduce: 41.48 | step: 18.97

  9%|▉         | 3654/41250 [8:48:45<90:21:51,  8.65s/it]
                                                         
{'loss': 0.2411, 'grad_norm': 2.0015416145324707, 'learning_rate': 3.9641234743141784e-05, 'epoch': 0.89}

  9%|▉         | 3654/41250 [8:48:45<90:21:51,  8.65s/it][2025-04-25 16:46:28,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 16:46:28,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.26 | bwd_microstep: 5736.20 | bwd_inner_microstep: 5714.54 | bwd_allreduce_microstep: 21.60 | step_microstep: 18.81
[2025-04-25 16:46:28,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.26 | bwd: 5736.21 | bwd_inner: 5714.54 | bwd_allreduce: 21.62 | step: 18.81

  9%|▉         | 3655/41250 [8:48:54<90:27:52,  8.66s/it]
                                                         
{'loss': 0.153, 'grad_norm': 2.0356626510620117, 'learning_rate': 3.9640938582612625e-05, 'epoch': 0.89}

  9%|▉         | 3655/41250 [8:48:54<90:27:52,  8.66s/it][2025-04-25 16:46:37,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.24 | optimizer_step: 0.99
[2025-04-25 16:46:37,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.15 | bwd_microstep: 5756.49 | bwd_inner_microstep: 5663.00 | bwd_allreduce_microstep: 93.43 | step_microstep: 20.06
[2025-04-25 16:46:37,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.15 | bwd: 5756.50 | bwd_inner: 5663.00 | bwd_allreduce: 93.46 | step: 20.07

  9%|▉         | 3656/41250 [8:49:02<90:29:39,  8.67s/it]
                                                         
{'loss': 0.2174, 'grad_norm': 1.5514094829559326, 'learning_rate': 3.9640642301000947e-05, 'epoch': 0.89}

  9%|▉         | 3656/41250 [8:49:02<90:29:39,  8.67s/it][2025-04-25 16:46:45,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.26 | optimizer_step: 0.95
[2025-04-25 16:46:45,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.22 | bwd_microstep: 5720.97 | bwd_inner_microstep: 5651.18 | bwd_allreduce_microstep: 69.74 | step_microstep: 20.16
[2025-04-25 16:46:45,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.22 | bwd: 5720.98 | bwd_inner: 5651.18 | bwd_allreduce: 69.76 | step: 20.17

  9%|▉         | 3657/41250 [8:49:11<90:25:27,  8.66s/it]
                                                         
{'loss': 0.1927, 'grad_norm': 1.705797553062439, 'learning_rate': 3.964034589830857e-05, 'epoch': 0.89}

  9%|▉         | 3657/41250 [8:49:11<90:25:27,  8.66s/it][2025-04-25 16:46:54,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 16:46:54,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5726.97 | bwd_inner_microstep: 5694.64 | bwd_allreduce_microstep: 32.27 | step_microstep: 19.23
[2025-04-25 16:46:54,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5726.98 | bwd_inner: 5694.64 | bwd_allreduce: 32.30 | step: 19.24

  9%|▉         | 3658/41250 [8:49:19<90:26:58,  8.66s/it]
                                                         
{'loss': 0.2894, 'grad_norm': 1.8190598487854004, 'learning_rate': 3.964004937453733e-05, 'epoch': 0.89}

  9%|▉         | 3658/41250 [8:49:19<90:26:58,  8.66s/it][2025-04-25 16:47:03,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:47:03,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.34 | bwd_microstep: 5743.89 | bwd_inner_microstep: 5701.72 | bwd_allreduce_microstep: 42.12 | step_microstep: 18.81
[2025-04-25 16:47:03,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5743.90 | bwd_inner: 5701.72 | bwd_allreduce: 42.14 | step: 18.81

  9%|▉         | 3659/41250 [8:49:28<90:30:15,  8.67s/it]
                                                         
{'loss': 0.2841, 'grad_norm': 2.813265085220337, 'learning_rate': 3.963975272968904e-05, 'epoch': 0.89}

  9%|▉         | 3659/41250 [8:49:28<90:30:15,  8.67s/it][2025-04-25 16:47:11,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.12 | optimizer_step: 0.95
[2025-04-25 16:47:12,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.51 | bwd_microstep: 5713.90 | bwd_inner_microstep: 5700.81 | bwd_allreduce_microstep: 13.04 | step_microstep: 19.09
[2025-04-25 16:47:12,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.51 | bwd: 5713.92 | bwd_inner: 5700.81 | bwd_allreduce: 13.06 | step: 19.09

  9%|▉         | 3660/41250 [8:49:37<90:27:59,  8.66s/it]
                                                         
{'loss': 0.1762, 'grad_norm': 1.6492265462875366, 'learning_rate': 3.963945596376555e-05, 'epoch': 0.89}

  9%|▉         | 3660/41250 [8:49:37<90:27:59,  8.66s/it][2025-04-25 16:47:20,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 16:47:20,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5778.00 | bwd_inner_microstep: 5685.64 | bwd_allreduce_microstep: 92.32 | step_microstep: 18.86
[2025-04-25 16:47:20,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5778.01 | bwd_inner: 5685.64 | bwd_allreduce: 92.33 | step: 18.86

  9%|▉         | 3661/41250 [8:49:46<90:36:54,  8.68s/it]
                                                         
{'loss': 0.1753, 'grad_norm': 1.6037039756774902, 'learning_rate': 3.963915907676866e-05, 'epoch': 0.89}

  9%|▉         | 3661/41250 [8:49:46<90:36:54,  8.68s/it][2025-04-25 16:47:29,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.20 | optimizer_step: 0.94
[2025-04-25 16:47:29,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.28 | bwd_microstep: 5713.57 | bwd_inner_microstep: 5645.89 | bwd_allreduce_microstep: 67.62 | step_microstep: 19.40
[2025-04-25 16:47:29,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.28 | bwd: 5713.59 | bwd_inner: 5645.89 | bwd_allreduce: 67.65 | step: 19.40

  9%|▉         | 3662/41250 [8:49:54<90:28:20,  8.67s/it]
                                                         
{'loss': 0.1049, 'grad_norm': 0.9698144197463989, 'learning_rate': 3.9638862068700234e-05, 'epoch': 0.89}

  9%|▉         | 3662/41250 [8:49:54<90:28:20,  8.67s/it][2025-04-25 16:47:38,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.25 | optimizer_step: 1.02
[2025-04-25 16:47:38,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5747.74 | bwd_inner_microstep: 5653.10 | bwd_allreduce_microstep: 94.58 | step_microstep: 19.72
[2025-04-25 16:47:38,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5747.75 | bwd_inner: 5653.10 | bwd_allreduce: 94.60 | step: 19.72

  9%|▉         | 3663/41250 [8:50:03<90:27:51,  8.66s/it]
                                                         
{'loss': 0.1067, 'grad_norm': 0.7291426658630371, 'learning_rate': 3.963856493956208e-05, 'epoch': 0.89}

  9%|▉         | 3663/41250 [8:50:03<90:27:51,  8.66s/it][2025-04-25 16:47:46,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 16:47:46,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.71 | bwd_microstep: 5734.28 | bwd_inner_microstep: 5688.32 | bwd_allreduce_microstep: 45.91 | step_microstep: 18.82
[2025-04-25 16:47:46,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.71 | bwd: 5734.29 | bwd_inner: 5688.32 | bwd_allreduce: 45.93 | step: 18.82

  9%|▉         | 3664/41250 [8:50:12<90:29:14,  8.67s/it]
                                                         
{'loss': 0.0946, 'grad_norm': 1.0150035619735718, 'learning_rate': 3.963826768935603e-05, 'epoch': 0.89}

  9%|▉         | 3664/41250 [8:50:12<90:29:14,  8.67s/it][2025-04-25 16:47:55,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:47:55,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.86 | bwd_microstep: 5687.76 | bwd_inner_microstep: 5674.93 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.81
[2025-04-25 16:47:55,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.86 | bwd: 5687.78 | bwd_inner: 5674.92 | bwd_allreduce: 12.81 | step: 18.81

  9%|▉         | 3665/41250 [8:50:20<90:22:18,  8.66s/it]
                                                         
{'loss': 0.148, 'grad_norm': 1.7198266983032227, 'learning_rate': 3.9637970318083925e-05, 'epoch': 0.89}

  9%|▉         | 3665/41250 [8:50:20<90:22:18,  8.66s/it][2025-04-25 16:48:03,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:48:03,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.48 | bwd_microstep: 5722.27 | bwd_inner_microstep: 5700.51 | bwd_allreduce_microstep: 21.72 | step_microstep: 18.47
[2025-04-25 16:48:03,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.48 | bwd: 5722.28 | bwd_inner: 5700.51 | bwd_allreduce: 21.73 | step: 18.47

  9%|▉         | 3666/41250 [8:50:29<90:22:31,  8.66s/it]
                                                         
{'loss': 0.1535, 'grad_norm': 1.144394874572754, 'learning_rate': 3.96376728257476e-05, 'epoch': 0.89}

  9%|▉         | 3666/41250 [8:50:29<90:22:31,  8.66s/it][2025-04-25 16:48:12,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:48:12,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.93 | bwd_microstep: 5718.63 | bwd_inner_microstep: 5702.31 | bwd_allreduce_microstep: 16.27 | step_microstep: 18.80
[2025-04-25 16:48:12,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.93 | bwd: 5718.65 | bwd_inner: 5702.31 | bwd_allreduce: 16.29 | step: 18.80

  9%|▉         | 3667/41250 [8:50:37<90:22:05,  8.66s/it]
                                                         
{'loss': 0.1843, 'grad_norm': 1.6346968412399292, 'learning_rate': 3.9637375212348876e-05, 'epoch': 0.89}

  9%|▉         | 3667/41250 [8:50:37<90:22:05,  8.66s/it][2025-04-25 16:48:21,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 16:48:21,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.75 | bwd_microstep: 5768.79 | bwd_inner_microstep: 5658.46 | bwd_allreduce_microstep: 110.28 | step_microstep: 19.12
[2025-04-25 16:48:21,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.75 | bwd: 5768.81 | bwd_inner: 5658.46 | bwd_allreduce: 110.31 | step: 19.12

  9%|▉         | 3668/41250 [8:50:46<90:26:29,  8.66s/it]
                                                         
{'loss': 0.1885, 'grad_norm': 1.116265058517456, 'learning_rate': 3.96370774778896e-05, 'epoch': 0.89}

  9%|▉         | 3668/41250 [8:50:46<90:26:29,  8.66s/it][2025-04-25 16:48:30,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:48:30,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.70 | bwd_microstep: 5786.73 | bwd_inner_microstep: 5773.89 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.46
[2025-04-25 16:48:30,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.70 | bwd: 5786.74 | bwd_inner: 5773.89 | bwd_allreduce: 12.81 | step: 18.47

  9%|▉         | 3669/41250 [8:50:55<90:44:25,  8.69s/it]
                                                         
{'loss': 0.128, 'grad_norm': 1.1234047412872314, 'learning_rate': 3.963677962237161e-05, 'epoch': 0.89}

  9%|▉         | 3669/41250 [8:50:55<90:44:25,  8.69s/it][2025-04-25 16:48:38,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:48:38,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5713.04 | bwd_inner_microstep: 5700.27 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.23
[2025-04-25 16:48:38,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5713.05 | bwd_inner: 5700.27 | bwd_allreduce: 12.74 | step: 18.24

  9%|▉         | 3670/41250 [8:51:04<90:35:29,  8.68s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 0.9077857732772827, 'learning_rate': 3.963648164579673e-05, 'epoch': 0.89}

  9%|▉         | 3670/41250 [8:51:04<90:35:29,  8.68s/it][2025-04-25 16:48:47,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 16:48:47,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.97 | bwd_microstep: 5736.21 | bwd_inner_microstep: 5703.32 | bwd_allreduce_microstep: 32.85 | step_microstep: 19.15
[2025-04-25 16:48:47,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.97 | bwd: 5736.23 | bwd_inner: 5703.32 | bwd_allreduce: 32.87 | step: 19.15

  9%|▉         | 3671/41250 [8:51:12<90:34:45,  8.68s/it]
                                                         
{'loss': 0.2103, 'grad_norm': 1.062593936920166, 'learning_rate': 3.963618354816681e-05, 'epoch': 0.89}

  9%|▉         | 3671/41250 [8:51:12<90:34:45,  8.68s/it][2025-04-25 16:48:56,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 16:48:56,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.34 | bwd_microstep: 5717.56 | bwd_inner_microstep: 5693.61 | bwd_allreduce_microstep: 23.91 | step_microstep: 18.45
[2025-04-25 16:48:56,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.34 | bwd: 5717.57 | bwd_inner: 5693.61 | bwd_allreduce: 23.92 | step: 18.45

  9%|▉         | 3672/41250 [8:51:21<90:28:31,  8.67s/it]
                                                         
{'loss': 0.2192, 'grad_norm': 1.6942094564437866, 'learning_rate': 3.963588532948367e-05, 'epoch': 0.89}

  9%|▉         | 3672/41250 [8:51:21<90:28:31,  8.67s/it][2025-04-25 16:49:04,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 16:49:04,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.43 | bwd_microstep: 5744.33 | bwd_inner_microstep: 5643.66 | bwd_allreduce_microstep: 100.62 | step_microstep: 18.48
[2025-04-25 16:49:04,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.43 | bwd: 5744.34 | bwd_inner: 5643.66 | bwd_allreduce: 100.64 | step: 18.49

  9%|▉         | 3673/41250 [8:51:30<90:27:22,  8.67s/it]
                                                         
{'loss': 0.1282, 'grad_norm': 1.5124555826187134, 'learning_rate': 3.963558698974916e-05, 'epoch': 0.89}

  9%|▉         | 3673/41250 [8:51:30<90:27:22,  8.67s/it][2025-04-25 16:49:13,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 16:49:13,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.36 | bwd_microstep: 5715.22 | bwd_inner_microstep: 5702.71 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.55
[2025-04-25 16:49:13,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.36 | bwd: 5715.23 | bwd_inner: 5702.71 | bwd_allreduce: 12.49 | step: 18.55

  9%|▉         | 3674/41250 [8:51:38<90:23:12,  8.66s/it]
                                                         
{'loss': 0.2534, 'grad_norm': 2.221567392349243, 'learning_rate': 3.963528852896513e-05, 'epoch': 0.89}

  9%|▉         | 3674/41250 [8:51:38<90:23:12,  8.66s/it][2025-04-25 16:49:22,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:49:22,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.94 | bwd_microstep: 5777.85 | bwd_inner_microstep: 5765.09 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.22
[2025-04-25 16:49:22,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.94 | bwd: 5777.87 | bwd_inner: 5765.09 | bwd_allreduce: 12.73 | step: 18.23

  9%|▉         | 3675/41250 [8:51:47<90:42:15,  8.69s/it]
                                                         
{'loss': 0.3088, 'grad_norm': 2.4145705699920654, 'learning_rate': 3.96349899471334e-05, 'epoch': 0.89}

  9%|▉         | 3675/41250 [8:51:47<90:42:15,  8.69s/it][2025-04-25 16:49:30,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 16:49:30,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.47 | bwd_microstep: 5780.96 | bwd_inner_microstep: 5768.09 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.25
[2025-04-25 16:49:30,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.47 | bwd: 5780.97 | bwd_inner: 5768.09 | bwd_allreduce: 12.83 | step: 19.25

  9%|▉         | 3676/41250 [8:51:56<90:52:19,  8.71s/it]
                                                         
{'loss': 0.0615, 'grad_norm': 1.0469313859939575, 'learning_rate': 3.963469124425582e-05, 'epoch': 0.89}

  9%|▉         | 3676/41250 [8:51:56<90:52:19,  8.71s/it][2025-04-25 16:49:39,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.22 | optimizer_step: 0.92
[2025-04-25 16:49:39,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.77 | bwd_microstep: 5709.36 | bwd_inner_microstep: 5696.06 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.61
[2025-04-25 16:49:39,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.77 | bwd: 5709.38 | bwd_inner: 5696.06 | bwd_allreduce: 13.28 | step: 19.62

  9%|▉         | 3677/41250 [8:52:04<90:39:08,  8.69s/it]
                                                         
{'loss': 0.2928, 'grad_norm': 4.999480724334717, 'learning_rate': 3.9634392420334224e-05, 'epoch': 0.89}

  9%|▉         | 3677/41250 [8:52:04<90:39:08,  8.69s/it][2025-04-25 16:49:48,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-25 16:49:48,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.56 | bwd_microstep: 5745.62 | bwd_inner_microstep: 5711.63 | bwd_allreduce_microstep: 33.95 | step_microstep: 19.15
[2025-04-25 16:49:48,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.56 | bwd: 5745.64 | bwd_inner: 5711.63 | bwd_allreduce: 33.96 | step: 19.15

  9%|▉         | 3678/41250 [8:52:13<90:40:40,  8.69s/it]
                                                         
{'loss': 0.1605, 'grad_norm': 1.4619545936584473, 'learning_rate': 3.963409347537047e-05, 'epoch': 0.89}

  9%|▉         | 3678/41250 [8:52:13<90:40:40,  8.69s/it][2025-04-25 16:49:57,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.19 | optimizer_step: 1.01
[2025-04-25 16:49:57,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.61 | bwd_microstep: 5980.10 | bwd_inner_microstep: 5686.21 | bwd_allreduce_microstep: 293.84 | step_microstep: 19.73
[2025-04-25 16:49:57,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.61 | bwd: 5980.11 | bwd_inner: 5686.20 | bwd_allreduce: 293.87 | step: 19.73

  9%|▉         | 3679/41250 [8:52:22<91:25:34,  8.76s/it]
                                                         
{'loss': 0.0343, 'grad_norm': 0.3635123074054718, 'learning_rate': 3.9633794409366385e-05, 'epoch': 0.89}

  9%|▉         | 3679/41250 [8:52:22<91:25:34,  8.76s/it][2025-04-25 16:50:05,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 16:50:05,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.21 | bwd_microstep: 5694.19 | bwd_inner_microstep: 5652.02 | bwd_allreduce_microstep: 42.12 | step_microstep: 19.43
[2025-04-25 16:50:05,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.21 | bwd: 5694.21 | bwd_inner: 5652.02 | bwd_allreduce: 42.14 | step: 19.44

  9%|▉         | 3680/41250 [8:52:31<90:56:23,  8.71s/it]
                                                         
{'loss': 0.078, 'grad_norm': 0.74578857421875, 'learning_rate': 3.9633495222323827e-05, 'epoch': 0.89}

  9%|▉         | 3680/41250 [8:52:31<90:56:23,  8.71s/it][2025-04-25 16:50:14,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:50:14,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.71 | bwd_microstep: 5702.68 | bwd_inner_microstep: 5653.54 | bwd_allreduce_microstep: 49.09 | step_microstep: 19.01
[2025-04-25 16:50:14,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.71 | bwd: 5702.69 | bwd_inner: 5653.54 | bwd_allreduce: 49.11 | step: 19.01

  9%|▉         | 3681/41250 [8:52:39<90:37:50,  8.68s/it]
                                                         
{'loss': 0.1256, 'grad_norm': 1.9610798358917236, 'learning_rate': 3.963319591424463e-05, 'epoch': 0.89}

  9%|▉         | 3681/41250 [8:52:39<90:37:50,  8.68s/it][2025-04-25 16:50:22,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 16:50:22,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.75 | bwd_microstep: 5716.79 | bwd_inner_microstep: 5703.32 | bwd_allreduce_microstep: 13.41 | step_microstep: 19.03
[2025-04-25 16:50:22,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.75 | bwd: 5716.80 | bwd_inner: 5703.32 | bwd_allreduce: 13.43 | step: 19.03

  9%|▉         | 3682/41250 [8:52:48<90:32:16,  8.68s/it]
                                                         
{'loss': 0.0631, 'grad_norm': 1.1468833684921265, 'learning_rate': 3.963289648513063e-05, 'epoch': 0.89}

  9%|▉         | 3682/41250 [8:52:48<90:32:16,  8.68s/it][2025-04-25 16:50:31,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 16:50:31,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.38 | bwd_microstep: 5899.44 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 244.15 | step_microstep: 18.82
[2025-04-25 16:50:31,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.38 | bwd: 5899.45 | bwd_inner: 5655.24 | bwd_allreduce: 244.17 | step: 18.83

  9%|▉         | 3683/41250 [8:52:57<90:58:01,  8.72s/it]
                                                         
{'loss': 0.2672, 'grad_norm': 2.0374245643615723, 'learning_rate': 3.96325969349837e-05, 'epoch': 0.89}

  9%|▉         | 3683/41250 [8:52:57<90:58:01,  8.72s/it][2025-04-25 16:50:40,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 1.19
[2025-04-25 16:50:40,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5778.66 | bwd_inner_microstep: 5702.81 | bwd_allreduce_microstep: 75.81 | step_microstep: 19.53
[2025-04-25 16:50:40,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.03 | bwd: 5778.68 | bwd_inner: 5702.81 | bwd_allreduce: 75.82 | step: 19.53

  9%|▉         | 3684/41250 [8:53:05<90:57:20,  8.72s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.584931492805481, 'learning_rate': 3.963229726380567e-05, 'epoch': 0.89}

  9%|▉         | 3684/41250 [8:53:05<90:57:20,  8.72s/it][2025-04-25 16:50:49,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 16:50:49,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.89 | bwd_microstep: 5763.51 | bwd_inner_microstep: 5692.31 | bwd_allreduce_microstep: 71.16 | step_microstep: 18.73
[2025-04-25 16:50:49,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.89 | bwd: 5763.53 | bwd_inner: 5692.31 | bwd_allreduce: 71.18 | step: 18.73

  9%|▉         | 3685/41250 [8:53:14<90:53:26,  8.71s/it]
                                                         
{'loss': 0.2523, 'grad_norm': 2.046079635620117, 'learning_rate': 3.963199747159838e-05, 'epoch': 0.89}

  9%|▉         | 3685/41250 [8:53:14<90:53:26,  8.71s/it][2025-04-25 16:50:57,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 16:50:57,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.96 | bwd_microstep: 5799.88 | bwd_inner_microstep: 5787.27 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.69
[2025-04-25 16:50:57,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.96 | bwd: 5799.89 | bwd_inner: 5787.27 | bwd_allreduce: 12.59 | step: 18.69

  9%|▉         | 3686/41250 [8:53:23<91:05:47,  8.73s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 1.0601615905761719, 'learning_rate': 3.963169755836369e-05, 'epoch': 0.89}

  9%|▉         | 3686/41250 [8:53:23<91:05:47,  8.73s/it][2025-04-25 16:51:06,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:51:06,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5775.88 | bwd_inner_microstep: 5663.57 | bwd_allreduce_microstep: 112.27 | step_microstep: 18.65
[2025-04-25 16:51:06,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5775.89 | bwd_inner: 5663.57 | bwd_allreduce: 112.28 | step: 18.65

  9%|▉         | 3687/41250 [8:53:31<90:57:09,  8.72s/it]
                                                         
{'loss': 0.1204, 'grad_norm': 1.659409999847412, 'learning_rate': 3.963139752410345e-05, 'epoch': 0.89}

  9%|▉         | 3687/41250 [8:53:31<90:57:09,  8.72s/it][2025-04-25 16:51:15,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:51:15,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5786.71 | bwd_inner_microstep: 5669.65 | bwd_allreduce_microstep: 117.02 | step_microstep: 18.82
[2025-04-25 16:51:15,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5786.73 | bwd_inner: 5669.65 | bwd_allreduce: 117.04 | step: 18.82

  9%|▉         | 3688/41250 [8:53:40<90:54:11,  8.71s/it]
                                                         
{'loss': 0.1289, 'grad_norm': 0.8672336935997009, 'learning_rate': 3.963109736881951e-05, 'epoch': 0.89}

  9%|▉         | 3688/41250 [8:53:40<90:54:11,  8.71s/it][2025-04-25 16:51:24,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:51:24,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.23 | bwd_microstep: 5789.09 | bwd_inner_microstep: 5776.42 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.73
[2025-04-25 16:51:24,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.23 | bwd: 5789.10 | bwd_inner: 5776.42 | bwd_allreduce: 12.64 | step: 18.73

  9%|▉         | 3689/41250 [8:53:49<91:03:15,  8.73s/it]
                                                         
{'loss': 0.3563, 'grad_norm': 4.817575454711914, 'learning_rate': 3.96307970925137e-05, 'epoch': 0.89}

  9%|▉         | 3689/41250 [8:53:49<91:03:15,  8.73s/it][2025-04-25 16:51:32,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:51:32,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5803.80 | bwd_inner_microstep: 5655.20 | bwd_allreduce_microstep: 148.56 | step_microstep: 18.70
[2025-04-25 16:51:32,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5803.82 | bwd_inner: 5655.20 | bwd_allreduce: 148.58 | step: 18.70

  9%|▉         | 3690/41250 [8:53:58<91:01:04,  8.72s/it]
                                                         
{'loss': 0.3167, 'grad_norm': 3.706394672393799, 'learning_rate': 3.96304966951879e-05, 'epoch': 0.89}

  9%|▉         | 3690/41250 [8:53:58<91:01:04,  8.72s/it][2025-04-25 16:51:41,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:51:41,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.72 | bwd_microstep: 5709.32 | bwd_inner_microstep: 5679.71 | bwd_allreduce_microstep: 29.57 | step_microstep: 18.69
[2025-04-25 16:51:41,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.72 | bwd: 5709.33 | bwd_inner: 5679.71 | bwd_allreduce: 29.58 | step: 18.70

  9%|▉         | 3691/41250 [8:54:06<90:42:05,  8.69s/it]
                                                         
{'loss': 0.0369, 'grad_norm': 0.8852997422218323, 'learning_rate': 3.9630196176843947e-05, 'epoch': 0.89}

  9%|▉         | 3691/41250 [8:54:06<90:42:05,  8.69s/it][2025-04-25 16:51:50,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:51:50,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.80 | bwd_microstep: 5719.14 | bwd_inner_microstep: 5706.36 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.90
[2025-04-25 16:51:50,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.80 | bwd: 5719.15 | bwd_inner: 5706.36 | bwd_allreduce: 12.75 | step: 18.91

  9%|▉         | 3692/41250 [8:54:15<90:34:37,  8.68s/it]
                                                         
{'loss': 0.2238, 'grad_norm': 2.7375261783599854, 'learning_rate': 3.962989553748369e-05, 'epoch': 0.9}

  9%|▉         | 3692/41250 [8:54:15<90:34:37,  8.68s/it][2025-04-25 16:51:58,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 16:51:58,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.06 | bwd_microstep: 5716.21 | bwd_inner_microstep: 5703.06 | bwd_allreduce_microstep: 13.10 | step_microstep: 18.89
[2025-04-25 16:51:58,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.05 | bwd: 5716.22 | bwd_inner: 5703.06 | bwd_allreduce: 13.12 | step: 18.89

  9%|▉         | 3693/41250 [8:54:24<90:28:05,  8.67s/it]
                                                         
{'loss': 0.0468, 'grad_norm': 0.46728795766830444, 'learning_rate': 3.9629594777108996e-05, 'epoch': 0.9}

  9%|▉         | 3693/41250 [8:54:24<90:28:05,  8.67s/it][2025-04-25 16:52:07,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:52:07,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5714.27 | bwd_inner_microstep: 5701.52 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.53
[2025-04-25 16:52:07,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5714.28 | bwd_inner: 5701.52 | bwd_allreduce: 12.72 | step: 18.53

  9%|▉         | 3694/41250 [8:54:32<90:22:35,  8.66s/it]
                                                         
{'loss': 0.0774, 'grad_norm': 1.007031798362732, 'learning_rate': 3.9629293895721707e-05, 'epoch': 0.9}

  9%|▉         | 3694/41250 [8:54:32<90:22:35,  8.66s/it][2025-04-25 16:52:16,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:52:16,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.19 | bwd_microstep: 5707.27 | bwd_inner_microstep: 5668.70 | bwd_allreduce_microstep: 38.53 | step_microstep: 18.33
[2025-04-25 16:52:16,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.19 | bwd: 5707.28 | bwd_inner: 5668.70 | bwd_allreduce: 38.55 | step: 18.33

  9%|▉         | 3695/41250 [8:54:41<90:14:06,  8.65s/it]
                                                         
{'loss': 0.1874, 'grad_norm': 2.1291871070861816, 'learning_rate': 3.9628992893323685e-05, 'epoch': 0.9}

  9%|▉         | 3695/41250 [8:54:41<90:14:06,  8.65s/it][2025-04-25 16:52:24,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 16:52:24,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.01 | bwd_microstep: 5911.68 | bwd_inner_microstep: 5666.11 | bwd_allreduce_microstep: 245.52 | step_microstep: 18.93
[2025-04-25 16:52:24,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.01 | bwd: 5911.69 | bwd_inner: 5666.11 | bwd_allreduce: 245.54 | step: 18.93

  9%|▉         | 3696/41250 [8:54:50<90:47:06,  8.70s/it]
                                                         
{'loss': 0.214, 'grad_norm': 3.3760437965393066, 'learning_rate': 3.9628691769916775e-05, 'epoch': 0.9}

  9%|▉         | 3696/41250 [8:54:50<90:47:06,  8.70s/it][2025-04-25 16:52:33,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 16:52:33,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.84 | bwd_microstep: 5757.75 | bwd_inner_microstep: 5696.47 | bwd_allreduce_microstep: 61.24 | step_microstep: 18.43
[2025-04-25 16:52:33,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5757.76 | bwd_inner: 5696.46 | bwd_allreduce: 61.26 | step: 18.43

  9%|▉         | 3697/41250 [8:54:58<90:44:28,  8.70s/it]
                                                         
{'loss': 0.178, 'grad_norm': 1.976136565208435, 'learning_rate': 3.9628390525502846e-05, 'epoch': 0.9}

  9%|▉         | 3697/41250 [8:54:58<90:44:28,  8.70s/it][2025-04-25 16:52:42,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:52:42,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.26 | bwd_microstep: 5773.85 | bwd_inner_microstep: 5761.11 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.78
[2025-04-25 16:52:42,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.26 | bwd: 5773.87 | bwd_inner: 5761.11 | bwd_allreduce: 12.72 | step: 18.78

  9%|▉         | 3698/41250 [8:55:07<90:54:10,  8.71s/it]
                                                         
{'loss': 0.185, 'grad_norm': 2.9536995887756348, 'learning_rate': 3.962808916008375e-05, 'epoch': 0.9}

  9%|▉         | 3698/41250 [8:55:07<90:54:10,  8.71s/it][2025-04-25 16:52:50,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-25 16:52:50,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.11 | bwd_microstep: 5694.34 | bwd_inner_microstep: 5659.98 | bwd_allreduce_microstep: 34.32 | step_microstep: 19.64
[2025-04-25 16:52:50,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.11 | bwd: 5694.36 | bwd_inner: 5659.98 | bwd_allreduce: 34.34 | step: 19.64

  9%|▉         | 3699/41250 [8:55:16<90:36:08,  8.69s/it]
                                                         
{'loss': 0.1055, 'grad_norm': 1.277174711227417, 'learning_rate': 3.962778767366135e-05, 'epoch': 0.9}

  9%|▉         | 3699/41250 [8:55:16<90:36:08,  8.69s/it][2025-04-25 16:52:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 16:52:59,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.20 | bwd_microstep: 5710.70 | bwd_inner_microstep: 5662.73 | bwd_allreduce_microstep: 47.91 | step_microstep: 18.85
[2025-04-25 16:52:59,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.20 | bwd: 5710.71 | bwd_inner: 5662.73 | bwd_allreduce: 47.94 | step: 18.85

  9%|▉         | 3700/41250 [8:55:24<90:24:34,  8.67s/it]
                                                         
{'loss': 0.1915, 'grad_norm': 2.0520288944244385, 'learning_rate': 3.962748606623749e-05, 'epoch': 0.9}

  9%|▉         | 3700/41250 [8:55:24<90:24:34,  8.67s/it][2025-04-25 16:53:08,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:53:08,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5712.46 | bwd_inner_microstep: 5699.79 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.90
[2025-04-25 16:53:08,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.29 | bwd: 5712.47 | bwd_inner: 5699.79 | bwd_allreduce: 12.64 | step: 18.90

  9%|▉         | 3701/41250 [8:55:33<90:20:38,  8.66s/it]
                                                         
{'loss': 0.259, 'grad_norm': 1.7408194541931152, 'learning_rate': 3.962718433781404e-05, 'epoch': 0.9}

  9%|▉         | 3701/41250 [8:55:33<90:20:38,  8.66s/it][2025-04-25 16:53:16,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:53:16,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.94 | bwd_microstep: 5709.96 | bwd_inner_microstep: 5658.62 | bwd_allreduce_microstep: 51.30 | step_microstep: 18.86
[2025-04-25 16:53:16,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.94 | bwd: 5709.97 | bwd_inner: 5658.61 | bwd_allreduce: 51.32 | step: 18.86

  9%|▉         | 3702/41250 [8:55:42<90:13:43,  8.65s/it]
                                                         
{'loss': 0.1696, 'grad_norm': 2.4449539184570312, 'learning_rate': 3.9626882488392864e-05, 'epoch': 0.9}

  9%|▉         | 3702/41250 [8:55:42<90:13:43,  8.65s/it][2025-04-25 16:53:25,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:53:25,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.39 | bwd_microstep: 5776.90 | bwd_inner_microstep: 5656.76 | bwd_allreduce_microstep: 120.09 | step_microstep: 18.60
[2025-04-25 16:53:25,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.39 | bwd: 5776.91 | bwd_inner: 5656.76 | bwd_allreduce: 120.11 | step: 18.61

  9%|▉         | 3703/41250 [8:55:50<90:20:46,  8.66s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 2.432003974914551, 'learning_rate': 3.9626580517975816e-05, 'epoch': 0.9}

  9%|▉         | 3703/41250 [8:55:50<90:20:46,  8.66s/it][2025-04-25 16:53:34,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:53:34,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5742.59 | bwd_inner_microstep: 5693.30 | bwd_allreduce_microstep: 49.24 | step_microstep: 18.88
[2025-04-25 16:53:34,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5742.60 | bwd_inner: 5693.30 | bwd_allreduce: 49.26 | step: 18.89

  9%|▉         | 3704/41250 [8:55:59<90:22:33,  8.67s/it]
                                                         
{'loss': 0.2327, 'grad_norm': 1.4176437854766846, 'learning_rate': 3.962627842656476e-05, 'epoch': 0.9}

  9%|▉         | 3704/41250 [8:55:59<90:22:33,  8.67s/it][2025-04-25 16:53:43,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.29 | optimizer_step: 1.02
[2025-04-25 16:53:43,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5984.83 | bwd_inner_microstep: 5699.84 | bwd_allreduce_microstep: 284.93 | step_microstep: 19.94
[2025-04-25 16:53:43,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5984.85 | bwd_inner: 5699.84 | bwd_allreduce: 284.96 | step: 19.94

  9%|▉         | 3705/41250 [8:56:08<91:09:48,  8.74s/it]
                                                         
{'loss': 0.3381, 'grad_norm': 2.871657609939575, 'learning_rate': 3.962597621416156e-05, 'epoch': 0.9}

  9%|▉         | 3705/41250 [8:56:08<91:09:48,  8.74s/it][2025-04-25 16:53:51,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 16:53:51,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.62 | bwd_microstep: 5699.23 | bwd_inner_microstep: 5686.36 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.60
[2025-04-25 16:53:51,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.62 | bwd: 5699.24 | bwd_inner: 5686.36 | bwd_allreduce: 12.84 | step: 18.60

  9%|▉         | 3706/41250 [8:56:17<90:49:02,  8.71s/it]
                                                         
{'loss': 0.2018, 'grad_norm': 2.479600667953491, 'learning_rate': 3.9625673880768076e-05, 'epoch': 0.9}

  9%|▉         | 3706/41250 [8:56:17<90:49:02,  8.71s/it][2025-04-25 16:54:00,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 16:54:00,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.93 | bwd_microstep: 5705.37 | bwd_inner_microstep: 5692.79 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.47
[2025-04-25 16:54:00,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5705.39 | bwd_inner: 5692.79 | bwd_allreduce: 12.55 | step: 18.47

  9%|▉         | 3707/41250 [8:56:25<90:35:03,  8.69s/it]
                                                         
{'loss': 0.0399, 'grad_norm': 0.8745871186256409, 'learning_rate': 3.962537142638616e-05, 'epoch': 0.9}

  9%|▉         | 3707/41250 [8:56:25<90:35:03,  8.69s/it][2025-04-25 16:54:09,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.99
[2025-04-25 16:54:09,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.94 | bwd_microstep: 5884.13 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 225.12 | step_microstep: 18.17
[2025-04-25 16:54:09,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.95 | bwd: 5884.14 | bwd_inner: 5658.97 | bwd_allreduce: 225.13 | step: 18.17

  9%|▉         | 3708/41250 [8:56:34<90:59:17,  8.73s/it]
                                                         
{'loss': 0.2217, 'grad_norm': 2.6725471019744873, 'learning_rate': 3.962506885101771e-05, 'epoch': 0.9}

  9%|▉         | 3708/41250 [8:56:34<90:59:17,  8.73s/it][2025-04-25 16:54:17,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 16:54:17,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5667.21 | bwd_inner_microstep: 5633.15 | bwd_allreduce_microstep: 34.02 | step_microstep: 18.47
[2025-04-25 16:54:17,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5667.22 | bwd_inner: 5633.15 | bwd_allreduce: 34.04 | step: 18.47

  9%|▉         | 3709/41250 [8:56:43<90:32:23,  8.68s/it]
                                                         
{'loss': 0.0831, 'grad_norm': 1.3199684619903564, 'learning_rate': 3.962476615466456e-05, 'epoch': 0.9}

  9%|▉         | 3709/41250 [8:56:43<90:32:23,  8.68s/it][2025-04-25 16:54:26,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 7.20 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 16:54:26,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.32 | bwd_microstep: 5701.72 | bwd_inner_microstep: 5688.94 | bwd_allreduce_microstep: 12.74 | step_microstep: 20.20
[2025-04-25 16:54:26,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.32 | bwd: 5701.73 | bwd_inner: 5688.94 | bwd_allreduce: 12.75 | step: 20.21

  9%|▉         | 3710/41250 [8:56:51<90:23:57,  8.67s/it]
                                                         
{'loss': 0.242, 'grad_norm': 1.6862149238586426, 'learning_rate': 3.9624463337328586e-05, 'epoch': 0.9}

  9%|▉         | 3710/41250 [8:56:51<90:23:57,  8.67s/it][2025-04-25 16:54:35,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 16:54:35,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5689.04 | bwd_inner_microstep: 5651.08 | bwd_allreduce_microstep: 37.92 | step_microstep: 18.96
[2025-04-25 16:54:35,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5689.06 | bwd_inner: 5651.08 | bwd_allreduce: 37.94 | step: 18.97

  9%|▉         | 3711/41250 [8:57:00<90:12:30,  8.65s/it]
                                                         
{'loss': 0.3477, 'grad_norm': 2.4379847049713135, 'learning_rate': 3.962416039901166e-05, 'epoch': 0.9}

  9%|▉         | 3711/41250 [8:57:00<90:12:30,  8.65s/it][2025-04-25 16:54:43,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 16:54:43,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.05 | bwd_microstep: 5759.86 | bwd_inner_microstep: 5659.25 | bwd_allreduce_microstep: 100.56 | step_microstep: 18.75
[2025-04-25 16:54:43,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.05 | bwd: 5759.88 | bwd_inner: 5659.25 | bwd_allreduce: 100.58 | step: 18.75

  9%|▉         | 3712/41250 [8:57:09<90:15:40,  8.66s/it]
                                                         
{'loss': 0.0329, 'grad_norm': 0.5314418077468872, 'learning_rate': 3.9623857339715636e-05, 'epoch': 0.9}

  9%|▉         | 3712/41250 [8:57:09<90:15:40,  8.66s/it][2025-04-25 16:54:52,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:54:52,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5733.76 | bwd_inner_microstep: 5689.42 | bwd_allreduce_microstep: 44.29 | step_microstep: 18.75
[2025-04-25 16:54:52,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5733.77 | bwd_inner: 5689.42 | bwd_allreduce: 44.30 | step: 18.76

  9%|▉         | 3713/41250 [8:57:17<90:16:25,  8.66s/it]
                                                         
{'loss': 0.462, 'grad_norm': 2.0705349445343018, 'learning_rate': 3.96235541594424e-05, 'epoch': 0.9}

  9%|▉         | 3713/41250 [8:57:17<90:16:25,  8.66s/it][2025-04-25 16:55:01,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.25 | optimizer_step: 0.98
[2025-04-25 16:55:01,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5768.45 | bwd_inner_microstep: 5651.51 | bwd_allreduce_microstep: 116.89 | step_microstep: 19.58
[2025-04-25 16:55:01,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5768.47 | bwd_inner: 5651.51 | bwd_allreduce: 116.91 | step: 19.58

  9%|▉         | 3714/41250 [8:57:26<90:20:43,  8.66s/it]
                                                         
{'loss': 0.0866, 'grad_norm': 0.813468337059021, 'learning_rate': 3.962325085819381e-05, 'epoch': 0.9}

  9%|▉         | 3714/41250 [8:57:26<90:20:43,  8.66s/it][2025-04-25 16:55:09,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 16:55:09,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.52 | bwd_microstep: 5744.17 | bwd_inner_microstep: 5655.52 | bwd_allreduce_microstep: 88.60 | step_microstep: 19.00
[2025-04-25 16:55:09,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.52 | bwd: 5744.18 | bwd_inner: 5655.52 | bwd_allreduce: 88.61 | step: 19.00

  9%|▉         | 3715/41250 [8:57:35<90:19:02,  8.66s/it]
                                                         
{'loss': 0.11, 'grad_norm': 1.7179073095321655, 'learning_rate': 3.962294743597173e-05, 'epoch': 0.9}

  9%|▉         | 3715/41250 [8:57:35<90:19:02,  8.66s/it][2025-04-25 16:55:18,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:55:18,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.98 | bwd_microstep: 5711.27 | bwd_inner_microstep: 5698.34 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.79
[2025-04-25 16:55:18,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.98 | bwd: 5711.28 | bwd_inner: 5698.34 | bwd_allreduce: 12.90 | step: 18.79

  9%|▉         | 3716/41250 [8:57:43<90:15:49,  8.66s/it]
                                                         
{'loss': 0.0679, 'grad_norm': 1.072646975517273, 'learning_rate': 3.962264389277805e-05, 'epoch': 0.9}

  9%|▉         | 3716/41250 [8:57:43<90:15:49,  8.66s/it][2025-04-25 16:55:26,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:55:26,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.08 | bwd_microstep: 5735.04 | bwd_inner_microstep: 5650.67 | bwd_allreduce_microstep: 84.32 | step_microstep: 18.53
[2025-04-25 16:55:26,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.08 | bwd: 5735.06 | bwd_inner: 5650.67 | bwd_allreduce: 84.34 | step: 18.53

  9%|▉         | 3717/41250 [8:57:52<90:13:25,  8.65s/it]
                                                         
{'loss': 0.0666, 'grad_norm': 0.6910247206687927, 'learning_rate': 3.962234022861463e-05, 'epoch': 0.9}

  9%|▉         | 3717/41250 [8:57:52<90:13:25,  8.65s/it][2025-04-25 16:55:35,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:55:35,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.29 | bwd_microstep: 5679.81 | bwd_inner_microstep: 5645.09 | bwd_allreduce_microstep: 34.67 | step_microstep: 18.41
[2025-04-25 16:55:35,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.29 | bwd: 5679.82 | bwd_inner: 5645.09 | bwd_allreduce: 34.69 | step: 18.41

  9%|▉         | 3718/41250 [8:58:00<90:00:34,  8.63s/it]
                                                         
{'loss': 0.0542, 'grad_norm': 1.049999713897705, 'learning_rate': 3.962203644348334e-05, 'epoch': 0.9}

  9%|▉         | 3718/41250 [8:58:00<90:00:34,  8.63s/it][2025-04-25 16:55:44,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 16:55:44,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.33 | bwd_microstep: 5773.11 | bwd_inner_microstep: 5666.69 | bwd_allreduce_microstep: 106.38 | step_microstep: 18.76
[2025-04-25 16:55:44,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.33 | bwd: 5773.13 | bwd_inner: 5666.69 | bwd_allreduce: 106.39 | step: 18.77

  9%|▉         | 3719/41250 [8:58:09<90:08:56,  8.65s/it]
                                                         
{'loss': 0.1396, 'grad_norm': 1.9376009702682495, 'learning_rate': 3.962173253738605e-05, 'epoch': 0.9}

  9%|▉         | 3719/41250 [8:58:09<90:08:56,  8.65s/it][2025-04-25 16:55:52,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:55:52,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.36 | bwd_microstep: 5719.69 | bwd_inner_microstep: 5703.63 | bwd_allreduce_microstep: 16.01 | step_microstep: 18.76
[2025-04-25 16:55:52,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.36 | bwd: 5719.71 | bwd_inner: 5703.63 | bwd_allreduce: 16.03 | step: 18.76

  9%|▉         | 3720/41250 [8:58:18<90:11:06,  8.65s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 2.1413586139678955, 'learning_rate': 3.962142851032464e-05, 'epoch': 0.9}

  9%|▉         | 3720/41250 [8:58:18<90:11:06,  8.65s/it][2025-04-25 16:56:01,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.34 | optimizer_step: 1.03
[2025-04-25 16:56:01,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.90 | bwd_microstep: 5712.71 | bwd_inner_microstep: 5698.83 | bwd_allreduce_microstep: 13.82 | step_microstep: 20.03
[2025-04-25 16:56:01,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.90 | bwd: 5712.73 | bwd_inner: 5698.83 | bwd_allreduce: 13.85 | step: 20.03

  9%|▉         | 3721/41250 [8:58:26<90:09:34,  8.65s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 2.4402694702148438, 'learning_rate': 3.962112436230099e-05, 'epoch': 0.9}

  9%|▉         | 3721/41250 [8:58:26<90:09:34,  8.65s/it][2025-04-25 16:56:10,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:56:10,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.27 | bwd_microstep: 5720.20 | bwd_inner_microstep: 5697.93 | bwd_allreduce_microstep: 22.22 | step_microstep: 18.59
[2025-04-25 16:56:10,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.27 | bwd: 5720.21 | bwd_inner: 5697.93 | bwd_allreduce: 22.24 | step: 18.59

  9%|▉         | 3722/41250 [8:58:35<90:09:45,  8.65s/it]
                                                         
{'loss': 0.2366, 'grad_norm': 1.8866543769836426, 'learning_rate': 3.9620820093316956e-05, 'epoch': 0.9}

  9%|▉         | 3722/41250 [8:58:35<90:09:45,  8.65s/it][2025-04-25 16:56:18,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 16:56:18,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.97 | bwd_microstep: 5708.47 | bwd_inner_microstep: 5693.88 | bwd_allreduce_microstep: 14.55 | step_microstep: 18.88
[2025-04-25 16:56:18,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.97 | bwd: 5708.48 | bwd_inner: 5693.88 | bwd_allreduce: 14.57 | step: 18.88

  9%|▉         | 3723/41250 [8:58:44<90:07:38,  8.65s/it]
                                                         
{'loss': 0.0751, 'grad_norm': 1.3417274951934814, 'learning_rate': 3.962051570337443e-05, 'epoch': 0.9}

  9%|▉         | 3723/41250 [8:58:44<90:07:38,  8.65s/it][2025-04-25 16:56:27,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 16:56:27,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5707.26 | bwd_inner_microstep: 5694.40 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.95
[2025-04-25 16:56:27,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5707.27 | bwd_inner: 5694.40 | bwd_allreduce: 12.83 | step: 18.96

  9%|▉         | 3724/41250 [8:58:52<90:06:27,  8.64s/it]
                                                         
{'loss': 0.1842, 'grad_norm': 3.007021903991699, 'learning_rate': 3.962021119247529e-05, 'epoch': 0.9}

  9%|▉         | 3724/41250 [8:58:52<90:06:27,  8.64s/it][2025-04-25 16:56:36,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:56:36,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.95 | bwd_microstep: 5738.84 | bwd_inner_microstep: 5704.58 | bwd_allreduce_microstep: 34.22 | step_microstep: 18.79
[2025-04-25 16:56:36,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.95 | bwd: 5738.86 | bwd_inner: 5704.58 | bwd_allreduce: 34.24 | step: 18.80

  9%|▉         | 3725/41250 [8:59:01<90:11:12,  8.65s/it]
                                                         
{'loss': 0.0382, 'grad_norm': 0.5541731715202332, 'learning_rate': 3.96199065606214e-05, 'epoch': 0.9}

  9%|▉         | 3725/41250 [8:59:01<90:11:12,  8.65s/it][2025-04-25 16:56:44,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 16:56:44,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.30 | bwd_microstep: 5706.57 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.86
[2025-04-25 16:56:44,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.30 | bwd: 5706.59 | bwd_inner: 5693.53 | bwd_allreduce: 13.01 | step: 18.86

  9%|▉         | 3726/41250 [8:59:10<90:10:51,  8.65s/it]
                                                         
{'loss': 0.4093, 'grad_norm': 2.316000461578369, 'learning_rate': 3.961960180781464e-05, 'epoch': 0.9}

  9%|▉         | 3726/41250 [8:59:10<90:10:51,  8.65s/it][2025-04-25 16:56:53,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:56:53,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.57 | bwd_microstep: 5719.07 | bwd_inner_microstep: 5695.13 | bwd_allreduce_microstep: 23.89 | step_microstep: 18.84
[2025-04-25 16:56:53,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.57 | bwd: 5719.08 | bwd_inner: 5695.13 | bwd_allreduce: 23.90 | step: 18.84

  9%|▉         | 3727/41250 [8:59:18<90:11:30,  8.65s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 5.59266996383667, 'learning_rate': 3.9619296934056907e-05, 'epoch': 0.9}

  9%|▉         | 3727/41250 [8:59:18<90:11:30,  8.65s/it][2025-04-25 16:57:02,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.30 | optimizer_step: 1.04
[2025-04-25 16:57:02,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.37 | bwd_microstep: 5715.43 | bwd_inner_microstep: 5644.76 | bwd_allreduce_microstep: 70.61 | step_microstep: 19.94
[2025-04-25 16:57:02,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.37 | bwd: 5715.45 | bwd_inner: 5644.76 | bwd_allreduce: 70.64 | step: 19.95

  9%|▉         | 3728/41250 [8:59:27<90:06:05,  8.64s/it]
                                                         
{'loss': 0.1182, 'grad_norm': 1.2084780931472778, 'learning_rate': 3.961899193935006e-05, 'epoch': 0.9}

  9%|▉         | 3728/41250 [8:59:27<90:06:05,  8.64s/it][2025-04-25 16:57:10,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.20 | optimizer_step: 0.93
[2025-04-25 16:57:10,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5789.07 | bwd_inner_microstep: 5642.63 | bwd_allreduce_microstep: 146.39 | step_microstep: 19.48
[2025-04-25 16:57:10,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.93 | bwd: 5789.08 | bwd_inner: 5642.63 | bwd_allreduce: 146.41 | step: 19.48

  9%|▉         | 3729/41250 [8:59:36<90:16:14,  8.66s/it]
                                                         
{'loss': 0.1763, 'grad_norm': 2.119236946105957, 'learning_rate': 3.9618686823695986e-05, 'epoch': 0.9}

  9%|▉         | 3729/41250 [8:59:36<90:16:14,  8.66s/it][2025-04-25 16:57:19,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:57:19,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.52 | bwd_microstep: 5699.62 | bwd_inner_microstep: 5686.83 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.76
[2025-04-25 16:57:19,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5699.63 | bwd_inner: 5686.83 | bwd_allreduce: 12.76 | step: 18.77

  9%|▉         | 3730/41250 [8:59:44<90:10:14,  8.65s/it]
                                                         
{'loss': 0.0583, 'grad_norm': 1.641797423362732, 'learning_rate': 3.961838158709657e-05, 'epoch': 0.9}

  9%|▉         | 3730/41250 [8:59:44<90:10:14,  8.65s/it][2025-04-25 16:57:28,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:57:28,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.95 | bwd_microstep: 5741.28 | bwd_inner_microstep: 5705.73 | bwd_allreduce_microstep: 35.51 | step_microstep: 18.94
[2025-04-25 16:57:28,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.95 | bwd: 5741.30 | bwd_inner: 5705.73 | bwd_allreduce: 35.53 | step: 18.94

  9%|▉         | 3731/41250 [8:59:53<90:13:54,  8.66s/it]
                                                         
{'loss': 0.2326, 'grad_norm': 2.876168727874756, 'learning_rate': 3.961807622955369e-05, 'epoch': 0.9}

  9%|▉         | 3731/41250 [8:59:53<90:13:54,  8.66s/it][2025-04-25 16:57:36,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 16:57:36,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.92 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5705.51 | bwd_allreduce_microstep: 12.27 | step_microstep: 18.98
[2025-04-25 16:57:36,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.92 | bwd: 5717.84 | bwd_inner: 5705.52 | bwd_allreduce: 12.29 | step: 18.98

  9%|▉         | 3732/41250 [9:00:02<90:12:53,  8.66s/it]
                                                         
{'loss': 0.2516, 'grad_norm': 2.354490041732788, 'learning_rate': 3.961777075106923e-05, 'epoch': 0.9}

  9%|▉         | 3732/41250 [9:00:02<90:12:53,  8.66s/it][2025-04-25 16:57:45,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 16:57:45,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5733.45 | bwd_inner_microstep: 5715.07 | bwd_allreduce_microstep: 18.34 | step_microstep: 18.86
[2025-04-25 16:57:45,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5733.47 | bwd_inner: 5715.07 | bwd_allreduce: 18.36 | step: 18.86

  9%|▉         | 3733/41250 [9:00:10<90:16:05,  8.66s/it]
                                                         
{'loss': 0.1808, 'grad_norm': 2.5396370887756348, 'learning_rate': 3.961746515164506e-05, 'epoch': 0.9}

  9%|▉         | 3733/41250 [9:00:10<90:16:05,  8.66s/it][2025-04-25 16:57:54,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.05 | optimizer_step: 0.95
[2025-04-25 16:57:54,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.90 | bwd_microstep: 5716.09 | bwd_inner_microstep: 5703.15 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.95
[2025-04-25 16:57:54,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.90 | bwd: 5716.11 | bwd_inner: 5703.15 | bwd_allreduce: 12.91 | step: 18.95

  9%|▉         | 3734/41250 [9:00:19<90:13:25,  8.66s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.4621580839157104, 'learning_rate': 3.961715943128309e-05, 'epoch': 0.91}

  9%|▉         | 3734/41250 [9:00:19<90:13:25,  8.66s/it][2025-04-25 16:58:02,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 16:58:02,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.34 | bwd_microstep: 5877.53 | bwd_inner_microstep: 5864.60 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.82
[2025-04-25 16:58:02,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.34 | bwd: 5877.54 | bwd_inner: 5864.60 | bwd_allreduce: 12.90 | step: 18.83

  9%|▉         | 3735/41250 [9:00:28<90:57:45,  8.73s/it]
                                                         
{'loss': 0.4129, 'grad_norm': 3.3239166736602783, 'learning_rate': 3.961685358998518e-05, 'epoch': 0.91}

  9%|▉         | 3735/41250 [9:00:28<90:57:45,  8.73s/it][2025-04-25 16:58:11,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 16:58:11,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.80 | bwd_microstep: 5699.86 | bwd_inner_microstep: 5667.39 | bwd_allreduce_microstep: 32.43 | step_microstep: 18.56
[2025-04-25 16:58:11,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.80 | bwd: 5699.88 | bwd_inner: 5667.39 | bwd_allreduce: 32.44 | step: 18.56

  9%|▉         | 3736/41250 [9:00:36<90:36:24,  8.70s/it]
                                                         
{'loss': 0.2852, 'grad_norm': 1.6225217580795288, 'learning_rate': 3.961654762775323e-05, 'epoch': 0.91}

  9%|▉         | 3736/41250 [9:00:36<90:36:24,  8.70s/it][2025-04-25 16:58:20,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 16:58:20,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.99 | bwd_microstep: 5724.40 | bwd_inner_microstep: 5711.92 | bwd_allreduce_microstep: 12.44 | step_microstep: 18.85
[2025-04-25 16:58:20,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.99 | bwd: 5724.42 | bwd_inner: 5711.92 | bwd_allreduce: 12.46 | step: 18.85

  9%|▉         | 3737/41250 [9:00:45<90:29:54,  8.68s/it]
                                                         
{'loss': 0.0559, 'grad_norm': 0.5388271808624268, 'learning_rate': 3.961624154458913e-05, 'epoch': 0.91}

  9%|▉         | 3737/41250 [9:00:45<90:29:54,  8.68s/it][2025-04-25 16:58:28,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 16:58:28,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.01 | bwd_microstep: 5785.72 | bwd_inner_microstep: 5669.61 | bwd_allreduce_microstep: 116.06 | step_microstep: 19.11
[2025-04-25 16:58:28,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.01 | bwd: 5785.73 | bwd_inner: 5669.61 | bwd_allreduce: 116.08 | step: 19.12

  9%|▉         | 3738/41250 [9:00:54<90:32:30,  8.69s/it]
                                                         
{'loss': 0.0896, 'grad_norm': 1.3025829792022705, 'learning_rate': 3.9615935340494746e-05, 'epoch': 0.91}

  9%|▉         | 3738/41250 [9:00:54<90:32:30,  8.69s/it][2025-04-25 16:58:37,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:58:37,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.45 | bwd_microstep: 5746.61 | bwd_inner_microstep: 5719.89 | bwd_allreduce_microstep: 26.68 | step_microstep: 18.67
[2025-04-25 16:58:37,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.45 | bwd: 5746.62 | bwd_inner: 5719.89 | bwd_allreduce: 26.70 | step: 18.67

  9%|▉         | 3739/41250 [9:01:02<90:31:54,  8.69s/it]
                                                         
{'loss': 0.3505, 'grad_norm': 2.7064952850341797, 'learning_rate': 3.9615629015471985e-05, 'epoch': 0.91}

  9%|▉         | 3739/41250 [9:01:02<90:31:54,  8.69s/it][2025-04-25 16:58:46,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 16:58:46,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.76 | bwd_microstep: 5835.72 | bwd_inner_microstep: 5803.92 | bwd_allreduce_microstep: 31.75 | step_microstep: 18.89
[2025-04-25 16:58:46,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.76 | bwd: 5835.73 | bwd_inner: 5803.92 | bwd_allreduce: 31.77 | step: 18.89

  9%|▉         | 3740/41250 [9:01:11<90:55:21,  8.73s/it]
                                                         
{'loss': 0.1899, 'grad_norm': 1.9249809980392456, 'learning_rate': 3.9615322569522735e-05, 'epoch': 0.91}

  9%|▉         | 3740/41250 [9:01:11<90:55:21,  8.73s/it][2025-04-25 16:58:55,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 1.13
[2025-04-25 16:58:55,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.56 | bwd_microstep: 5784.12 | bwd_inner_microstep: 5656.77 | bwd_allreduce_microstep: 127.30 | step_microstep: 18.83
[2025-04-25 16:58:55,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.56 | bwd: 5784.13 | bwd_inner: 5656.77 | bwd_allreduce: 127.32 | step: 18.83

  9%|▉         | 3741/41250 [9:01:20<90:50:25,  8.72s/it]
                                                         
{'loss': 0.1639, 'grad_norm': 1.655286431312561, 'learning_rate': 3.961501600264887e-05, 'epoch': 0.91}

  9%|▉         | 3741/41250 [9:01:20<90:50:25,  8.72s/it][2025-04-25 16:59:03,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 16:59:03,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.94 | bwd_microstep: 5837.69 | bwd_inner_microstep: 5722.83 | bwd_allreduce_microstep: 114.81 | step_microstep: 18.83
[2025-04-25 16:59:03,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.94 | bwd: 5837.70 | bwd_inner: 5722.83 | bwd_allreduce: 114.83 | step: 18.83

  9%|▉         | 3742/41250 [9:01:29<91:01:27,  8.74s/it]
                                                         
{'loss': 0.2821, 'grad_norm': 1.5240472555160522, 'learning_rate': 3.96147093148523e-05, 'epoch': 0.91}

  9%|▉         | 3742/41250 [9:01:29<91:01:27,  8.74s/it][2025-04-25 16:59:12,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 16:59:12,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5716.48 | bwd_inner_microstep: 5703.70 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.68
[2025-04-25 16:59:12,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5716.49 | bwd_inner: 5703.70 | bwd_allreduce: 12.76 | step: 18.68

  9%|▉         | 3743/41250 [9:01:37<90:44:10,  8.71s/it]
                                                         
{'loss': 0.2222, 'grad_norm': 1.4812183380126953, 'learning_rate': 3.96144025061349e-05, 'epoch': 0.91}

  9%|▉         | 3743/41250 [9:01:37<90:44:10,  8.71s/it][2025-04-25 16:59:21,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-25 16:59:21,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.06 | bwd_microstep: 5757.97 | bwd_inner_microstep: 5704.90 | bwd_allreduce_microstep: 53.03 | step_microstep: 18.58
[2025-04-25 16:59:21,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.06 | bwd: 5757.98 | bwd_inner: 5704.90 | bwd_allreduce: 53.05 | step: 18.58

  9%|▉         | 3744/41250 [9:01:46<90:41:34,  8.71s/it]
                                                         
{'loss': 0.0611, 'grad_norm': 1.0649211406707764, 'learning_rate': 3.961409557649856e-05, 'epoch': 0.91}

  9%|▉         | 3744/41250 [9:01:46<90:41:34,  8.71s/it][2025-04-25 16:59:29,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 16:59:29,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.96 | bwd_microstep: 5776.41 | bwd_inner_microstep: 5668.29 | bwd_allreduce_microstep: 108.08 | step_microstep: 18.91
[2025-04-25 16:59:29,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.96 | bwd: 5776.42 | bwd_inner: 5668.29 | bwd_allreduce: 108.09 | step: 18.92

  9%|▉         | 3745/41250 [9:01:55<90:38:08,  8.70s/it]
                                                         
{'loss': 0.312, 'grad_norm': 2.482408046722412, 'learning_rate': 3.961378852594519e-05, 'epoch': 0.91}

  9%|▉         | 3745/41250 [9:01:55<90:38:08,  8.70s/it][2025-04-25 16:59:38,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 16:59:38,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5711.61 | bwd_inner_microstep: 5698.88 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.75
[2025-04-25 16:59:38,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5711.62 | bwd_inner: 5698.88 | bwd_allreduce: 12.70 | step: 18.76

  9%|▉         | 3746/41250 [9:02:03<90:27:14,  8.68s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 1.196201205253601, 'learning_rate': 3.961348135447666e-05, 'epoch': 0.91}

  9%|▉         | 3746/41250 [9:02:03<90:27:14,  8.68s/it][2025-04-25 16:59:47,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 16:59:47,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.68 | bwd_microstep: 5716.26 | bwd_inner_microstep: 5703.54 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.98
[2025-04-25 16:59:47,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.68 | bwd: 5716.27 | bwd_inner: 5703.54 | bwd_allreduce: 12.69 | step: 18.98

  9%|▉         | 3747/41250 [9:02:12<90:20:32,  8.67s/it]
                                                         
{'loss': 0.1512, 'grad_norm': 1.2816102504730225, 'learning_rate': 3.9613174062094885e-05, 'epoch': 0.91}

  9%|▉         | 3747/41250 [9:02:12<90:20:32,  8.67s/it][2025-04-25 16:59:55,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 16:59:55,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.84 | bwd_microstep: 5700.77 | bwd_inner_microstep: 5666.77 | bwd_allreduce_microstep: 33.96 | step_microstep: 18.77
[2025-04-25 16:59:55,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.84 | bwd: 5700.79 | bwd_inner: 5666.77 | bwd_allreduce: 33.98 | step: 18.77

  9%|▉         | 3748/41250 [9:02:21<90:09:53,  8.66s/it]
                                                         
{'loss': 0.1139, 'grad_norm': 0.9153991937637329, 'learning_rate': 3.961286664880174e-05, 'epoch': 0.91}

  9%|▉         | 3748/41250 [9:02:21<90:09:53,  8.66s/it][2025-04-25 17:00:04,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 17:00:04,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.45 | bwd_microstep: 5692.15 | bwd_inner_microstep: 5654.26 | bwd_allreduce_microstep: 37.84 | step_microstep: 18.32
[2025-04-25 17:00:04,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.45 | bwd: 5692.16 | bwd_inner: 5654.26 | bwd_allreduce: 37.86 | step: 18.32

  9%|▉         | 3749/41250 [9:02:29<90:03:11,  8.64s/it]
                                                         
{'loss': 0.2508, 'grad_norm': 1.4272767305374146, 'learning_rate': 3.9612559114599135e-05, 'epoch': 0.91}

  9%|▉         | 3749/41250 [9:02:29<90:03:11,  8.64s/it][2025-04-25 17:00:13,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 17:00:13,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.94 | bwd_microstep: 5762.73 | bwd_inner_microstep: 5692.16 | bwd_allreduce_microstep: 70.53 | step_microstep: 18.38
[2025-04-25 17:00:13,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.94 | bwd: 5762.75 | bwd_inner: 5692.16 | bwd_allreduce: 70.55 | step: 18.39

  9%|▉         | 3750/41250 [9:02:38<90:12:22,  8.66s/it]
                                                         
{'loss': 0.2095, 'grad_norm': 1.0177063941955566, 'learning_rate': 3.9612251459488966e-05, 'epoch': 0.91}

  9%|▉         | 3750/41250 [9:02:38<90:12:22,  8.66s/it][2025-04-25 17:00:21,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:00:21,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.31 | bwd_microstep: 5753.99 | bwd_inner_microstep: 5702.24 | bwd_allreduce_microstep: 51.70 | step_microstep: 19.06
[2025-04-25 17:00:21,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.31 | bwd: 5754.00 | bwd_inner: 5702.24 | bwd_allreduce: 51.72 | step: 19.06

  9%|▉         | 3751/41250 [9:02:47<90:17:25,  8.67s/it]
                                                         
{'loss': 0.1038, 'grad_norm': 0.9602949619293213, 'learning_rate': 3.961194368347312e-05, 'epoch': 0.91}

  9%|▉         | 3751/41250 [9:02:47<90:17:25,  8.67s/it][2025-04-25 17:00:30,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:00:30,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.47 | bwd_microstep: 5741.49 | bwd_inner_microstep: 5690.60 | bwd_allreduce_microstep: 50.85 | step_microstep: 18.25
[2025-04-25 17:00:30,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.47 | bwd: 5741.50 | bwd_inner: 5690.60 | bwd_allreduce: 50.87 | step: 18.26

  9%|▉         | 3752/41250 [9:02:55<90:17:04,  8.67s/it]
                                                         
{'loss': 0.1145, 'grad_norm': 1.4483187198638916, 'learning_rate': 3.9611635786553495e-05, 'epoch': 0.91}

  9%|▉         | 3752/41250 [9:02:55<90:17:04,  8.67s/it][2025-04-25 17:00:39,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:00:39,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.74 | bwd_microstep: 5712.57 | bwd_inner_microstep: 5699.71 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.36
[2025-04-25 17:00:39,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.74 | bwd: 5712.58 | bwd_inner: 5699.71 | bwd_allreduce: 12.84 | step: 18.36

  9%|▉         | 3753/41250 [9:03:04<90:16:38,  8.67s/it]
                                                         
{'loss': 0.0269, 'grad_norm': 0.2515845000743866, 'learning_rate': 3.9611327768731994e-05, 'epoch': 0.91}

  9%|▉         | 3753/41250 [9:03:04<90:16:38,  8.67s/it][2025-04-25 17:00:47,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.69 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:00:47,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.00 | bwd_microstep: 5749.24 | bwd_inner_microstep: 5706.09 | bwd_allreduce_microstep: 43.10 | step_microstep: 18.82
[2025-04-25 17:00:47,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.00 | bwd: 5749.25 | bwd_inner: 5706.09 | bwd_allreduce: 43.12 | step: 18.82

  9%|▉         | 3754/41250 [9:03:13<90:19:54,  8.67s/it]
                                                         
{'loss': 0.1373, 'grad_norm': 0.9031159281730652, 'learning_rate': 3.961101963001052e-05, 'epoch': 0.91}

  9%|▉         | 3754/41250 [9:03:13<90:19:54,  8.67s/it][2025-04-25 17:00:56,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:00:56,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.80 | bwd_microstep: 5736.83 | bwd_inner_microstep: 5704.23 | bwd_allreduce_microstep: 32.55 | step_microstep: 18.62
[2025-04-25 17:00:56,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.80 | bwd: 5736.84 | bwd_inner: 5704.23 | bwd_allreduce: 32.57 | step: 18.62

  9%|▉         | 3755/41250 [9:03:21<90:20:00,  8.67s/it]
                                                         
{'loss': 0.134, 'grad_norm': 1.4665031433105469, 'learning_rate': 3.961071137039097e-05, 'epoch': 0.91}

  9%|▉         | 3755/41250 [9:03:21<90:20:00,  8.67s/it][2025-04-25 17:01:05,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:01:05,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5730.10 | bwd_inner_microstep: 5697.33 | bwd_allreduce_microstep: 32.72 | step_microstep: 18.33
[2025-04-25 17:01:05,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5730.11 | bwd_inner: 5697.33 | bwd_allreduce: 32.74 | step: 18.34

  9%|▉         | 3756/41250 [9:03:30<90:18:01,  8.67s/it]
                                                         
{'loss': 0.3913, 'grad_norm': 2.790292739868164, 'learning_rate': 3.961040298987523e-05, 'epoch': 0.91}

  9%|▉         | 3756/41250 [9:03:30<90:18:01,  8.67s/it][2025-04-25 17:01:13,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:01:13,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.88 | bwd_microstep: 5731.18 | bwd_inner_microstep: 5696.85 | bwd_allreduce_microstep: 34.29 | step_microstep: 18.51
[2025-04-25 17:01:13,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.88 | bwd: 5731.20 | bwd_inner: 5696.85 | bwd_allreduce: 34.31 | step: 18.51

  9%|▉         | 3757/41250 [9:03:39<90:18:46,  8.67s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.2735320329666138, 'learning_rate': 3.961009448846521e-05, 'epoch': 0.91}

  9%|▉         | 3757/41250 [9:03:39<90:18:46,  8.67s/it][2025-04-25 17:01:22,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:01:22,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.82 | bwd_microstep: 5669.49 | bwd_inner_microstep: 5654.29 | bwd_allreduce_microstep: 15.16 | step_microstep: 18.49
[2025-04-25 17:01:22,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.82 | bwd: 5669.50 | bwd_inner: 5654.29 | bwd_allreduce: 15.17 | step: 18.49

  9%|▉         | 3758/41250 [9:03:47<90:02:38,  8.65s/it]
                                                         
{'loss': 0.2227, 'grad_norm': 1.5447580814361572, 'learning_rate': 3.960978586616283e-05, 'epoch': 0.91}

  9%|▉         | 3758/41250 [9:03:47<90:02:38,  8.65s/it][2025-04-25 17:01:31,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:01:31,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.16 | bwd_microstep: 5710.34 | bwd_inner_microstep: 5695.69 | bwd_allreduce_microstep: 14.61 | step_microstep: 18.36
[2025-04-25 17:01:31,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.16 | bwd: 5710.35 | bwd_inner: 5695.69 | bwd_allreduce: 14.62 | step: 18.36

  9%|▉         | 3759/41250 [9:03:56<90:03:37,  8.65s/it]
                                                         
{'loss': 0.1174, 'grad_norm': 0.9007444381713867, 'learning_rate': 3.960947712296997e-05, 'epoch': 0.91}

  9%|▉         | 3759/41250 [9:03:56<90:03:37,  8.65s/it][2025-04-25 17:01:39,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 17:01:39,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.59 | bwd_microstep: 5750.38 | bwd_inner_microstep: 5642.85 | bwd_allreduce_microstep: 107.49 | step_microstep: 18.59
[2025-04-25 17:01:39,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.59 | bwd: 5750.39 | bwd_inner: 5642.85 | bwd_allreduce: 107.51 | step: 18.59

  9%|▉         | 3760/41250 [9:04:05<90:05:12,  8.65s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 4.244321823120117, 'learning_rate': 3.960916825888854e-05, 'epoch': 0.91}

  9%|▉         | 3760/41250 [9:04:05<90:05:12,  8.65s/it][2025-04-25 17:01:48,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 17:01:48,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.10 | bwd_microstep: 5706.70 | bwd_inner_microstep: 5693.84 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.02
[2025-04-25 17:01:48,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.10 | bwd: 5706.71 | bwd_inner: 5693.84 | bwd_allreduce: 12.84 | step: 19.02

  9%|▉         | 3761/41250 [9:04:13<90:04:29,  8.65s/it]
                                                         
{'loss': 0.1744, 'grad_norm': 1.4726518392562866, 'learning_rate': 3.960885927392044e-05, 'epoch': 0.91}

  9%|▉         | 3761/41250 [9:04:13<90:04:29,  8.65s/it][2025-04-25 17:01:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:01:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.55 | bwd_microstep: 5735.58 | bwd_inner_microstep: 5686.68 | bwd_allreduce_microstep: 48.85 | step_microstep: 17.94
[2025-04-25 17:01:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.55 | bwd: 5735.59 | bwd_inner: 5686.68 | bwd_allreduce: 48.87 | step: 17.95

  9%|▉         | 3762/41250 [9:04:22<90:07:43,  8.66s/it]
                                                         
{'loss': 0.1473, 'grad_norm': 1.9080960750579834, 'learning_rate': 3.960855016806759e-05, 'epoch': 0.91}

  9%|▉         | 3762/41250 [9:04:22<90:07:43,  8.66s/it][2025-04-25 17:02:05,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 17:02:05,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.80 | bwd_microstep: 5666.40 | bwd_inner_microstep: 5646.58 | bwd_allreduce_microstep: 19.77 | step_microstep: 18.81
[2025-04-25 17:02:05,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.80 | bwd: 5666.41 | bwd_inner: 5646.58 | bwd_allreduce: 19.79 | step: 18.82

  9%|▉         | 3763/41250 [9:04:30<89:55:18,  8.64s/it]
                                                         
{'loss': 0.0692, 'grad_norm': 1.560904860496521, 'learning_rate': 3.960824094133187e-05, 'epoch': 0.91}

  9%|▉         | 3763/41250 [9:04:30<89:55:18,  8.64s/it][2025-04-25 17:02:14,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:02:14,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.09 | bwd_microstep: 5769.85 | bwd_inner_microstep: 5757.19 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.18
[2025-04-25 17:02:14,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.09 | bwd: 5769.87 | bwd_inner: 5757.19 | bwd_allreduce: 12.63 | step: 18.18

  9%|▉         | 3764/41250 [9:04:39<90:14:14,  8.67s/it]
                                                         
{'loss': 0.4263, 'grad_norm': 4.5732879638671875, 'learning_rate': 3.960793159371522e-05, 'epoch': 0.91}

  9%|▉         | 3764/41250 [9:04:39<90:14:14,  8.67s/it][2025-04-25 17:02:23,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 1.04
[2025-04-25 17:02:23,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.62 | bwd_microstep: 5663.61 | bwd_inner_microstep: 5651.07 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.29
[2025-04-25 17:02:23,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.62 | bwd: 5663.62 | bwd_inner: 5651.07 | bwd_allreduce: 12.52 | step: 18.29

  9%|▉         | 3765/41250 [9:04:48<90:00:53,  8.64s/it]
                                                         
{'loss': 0.1067, 'grad_norm': 1.1630226373672485, 'learning_rate': 3.9607622125219513e-05, 'epoch': 0.91}

  9%|▉         | 3765/41250 [9:04:48<90:00:53,  8.64s/it][2025-04-25 17:02:31,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 17:02:31,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5734.89 | bwd_inner_microstep: 5691.33 | bwd_allreduce_microstep: 43.52 | step_microstep: 17.98
[2025-04-25 17:02:31,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5734.90 | bwd_inner: 5691.33 | bwd_allreduce: 43.54 | step: 17.99

  9%|▉         | 3766/41250 [9:04:57<90:06:37,  8.65s/it]
                                                         
{'loss': 0.0988, 'grad_norm': 1.012160062789917, 'learning_rate': 3.960731253584668e-05, 'epoch': 0.91}

  9%|▉         | 3766/41250 [9:04:57<90:06:37,  8.65s/it][2025-04-25 17:02:40,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:02:40,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5671.78 | bwd_inner_microstep: 5652.58 | bwd_allreduce_microstep: 19.15 | step_microstep: 18.57
[2025-04-25 17:02:40,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5671.79 | bwd_inner: 5652.58 | bwd_allreduce: 19.17 | step: 18.58

  9%|▉         | 3767/41250 [9:05:05<89:53:06,  8.63s/it]
                                                         
{'loss': 0.277, 'grad_norm': 1.7866110801696777, 'learning_rate': 3.960700282559862e-05, 'epoch': 0.91}

  9%|▉         | 3767/41250 [9:05:05<89:53:06,  8.63s/it][2025-04-25 17:02:48,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 17:02:48,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.12 | bwd_microstep: 5685.21 | bwd_inner_microstep: 5656.80 | bwd_allreduce_microstep: 28.36 | step_microstep: 18.62
[2025-04-25 17:02:48,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.12 | bwd: 5685.22 | bwd_inner: 5656.80 | bwd_allreduce: 28.38 | step: 18.62

  9%|▉         | 3768/41250 [9:05:14<89:47:05,  8.62s/it]
                                                         
{'loss': 0.3658, 'grad_norm': 2.0757741928100586, 'learning_rate': 3.960669299447724e-05, 'epoch': 0.91}

  9%|▉         | 3768/41250 [9:05:14<89:47:05,  8.62s/it][2025-04-25 17:02:57,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:02:57,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.82 | bwd_microstep: 5744.98 | bwd_inner_microstep: 5689.30 | bwd_allreduce_microstep: 55.64 | step_microstep: 18.72
[2025-04-25 17:02:57,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.82 | bwd: 5744.99 | bwd_inner: 5689.30 | bwd_allreduce: 55.66 | step: 18.72

  9%|▉         | 3769/41250 [9:05:22<89:57:02,  8.64s/it]
                                                         
{'loss': 0.2827, 'grad_norm': 2.535247564315796, 'learning_rate': 3.9606383042484456e-05, 'epoch': 0.91}

  9%|▉         | 3769/41250 [9:05:22<89:57:02,  8.64s/it][2025-04-25 17:03:06,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 17:03:06,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.96 | bwd_microstep: 5722.56 | bwd_inner_microstep: 5700.39 | bwd_allreduce_microstep: 22.13 | step_microstep: 18.50
[2025-04-25 17:03:06,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.96 | bwd: 5722.58 | bwd_inner: 5700.39 | bwd_allreduce: 22.15 | step: 18.50

  9%|▉         | 3770/41250 [9:05:31<90:00:04,  8.64s/it]
                                                         
{'loss': 0.181, 'grad_norm': 1.5973478555679321, 'learning_rate': 3.960607296962218e-05, 'epoch': 0.91}

  9%|▉         | 3770/41250 [9:05:31<90:00:04,  8.64s/it][2025-04-25 17:03:14,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 17:03:14,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.56 | bwd_microstep: 5694.35 | bwd_inner_microstep: 5654.73 | bwd_allreduce_microstep: 39.57 | step_microstep: 18.49
[2025-04-25 17:03:14,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.56 | bwd: 5694.36 | bwd_inner: 5654.73 | bwd_allreduce: 39.59 | step: 18.49

  9%|▉         | 3771/41250 [9:05:40<89:52:55,  8.63s/it]
                                                         
{'loss': 0.1428, 'grad_norm': 1.2236913442611694, 'learning_rate': 3.9605762775892326e-05, 'epoch': 0.91}

  9%|▉         | 3771/41250 [9:05:40<89:52:55,  8.63s/it][2025-04-25 17:03:23,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-25 17:03:23,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.46 | bwd_microstep: 5726.32 | bwd_inner_microstep: 5713.05 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.09
[2025-04-25 17:03:23,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.46 | bwd: 5726.34 | bwd_inner: 5713.05 | bwd_allreduce: 13.25 | step: 19.10

  9%|▉         | 3772/41250 [9:05:48<89:59:03,  8.64s/it]
                                                         
{'loss': 0.594, 'grad_norm': 2.378988027572632, 'learning_rate': 3.960545246129679e-05, 'epoch': 0.91}

  9%|▉         | 3772/41250 [9:05:48<89:59:03,  8.64s/it][2025-04-25 17:03:32,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:03:32,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.01 | bwd_microstep: 5677.12 | bwd_inner_microstep: 5650.32 | bwd_allreduce_microstep: 26.75 | step_microstep: 18.30
[2025-04-25 17:03:32,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.01 | bwd: 5677.13 | bwd_inner: 5650.32 | bwd_allreduce: 26.77 | step: 18.30

  9%|▉         | 3773/41250 [9:05:57<89:49:43,  8.63s/it]
                                                         
{'loss': 0.2077, 'grad_norm': 1.4708396196365356, 'learning_rate': 3.96051420258375e-05, 'epoch': 0.91}

  9%|▉         | 3773/41250 [9:05:57<89:49:43,  8.63s/it][2025-04-25 17:03:40,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:03:40,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.44 | bwd_microstep: 5926.80 | bwd_inner_microstep: 5643.86 | bwd_allreduce_microstep: 282.90 | step_microstep: 18.47
[2025-04-25 17:03:40,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.44 | bwd: 5926.81 | bwd_inner: 5643.86 | bwd_allreduce: 282.91 | step: 18.47

  9%|▉         | 3774/41250 [9:06:06<90:27:26,  8.69s/it]
                                                         
{'loss': 0.1971, 'grad_norm': 2.2188193798065186, 'learning_rate': 3.960483146951636e-05, 'epoch': 0.91}

  9%|▉         | 3774/41250 [9:06:06<90:27:26,  8.69s/it][2025-04-25 17:03:49,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:03:49,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.10 | bwd_microstep: 5700.44 | bwd_inner_microstep: 5650.74 | bwd_allreduce_microstep: 49.66 | step_microstep: 18.48
[2025-04-25 17:03:49,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.10 | bwd: 5700.46 | bwd_inner: 5650.74 | bwd_allreduce: 49.68 | step: 18.49

  9%|▉         | 3775/41250 [9:06:14<90:14:17,  8.67s/it]
                                                         
{'loss': 0.135, 'grad_norm': 1.585123062133789, 'learning_rate': 3.9604520792335305e-05, 'epoch': 0.92}

  9%|▉         | 3775/41250 [9:06:14<90:14:17,  8.67s/it][2025-04-25 17:03:58,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:03:58,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5765.27 | bwd_inner_microstep: 5703.46 | bwd_allreduce_microstep: 61.77 | step_microstep: 18.53
[2025-04-25 17:03:58,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5765.28 | bwd_inner: 5703.46 | bwd_allreduce: 61.79 | step: 18.53

  9%|▉         | 3776/41250 [9:06:23<90:19:47,  8.68s/it]
                                                         
{'loss': 0.2016, 'grad_norm': 2.4871199131011963, 'learning_rate': 3.960420999429622e-05, 'epoch': 0.92}

  9%|▉         | 3776/41250 [9:06:23<90:19:47,  8.68s/it][2025-04-25 17:04:06,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.13 | optimizer_step: 0.98
[2025-04-25 17:04:06,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5749.94 | bwd_inner_microstep: 5660.85 | bwd_allreduce_microstep: 89.02 | step_microstep: 18.79
[2025-04-25 17:04:06,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5749.96 | bwd_inner: 5660.85 | bwd_allreduce: 89.05 | step: 18.79

  9%|▉         | 3777/41250 [9:06:32<90:18:19,  8.68s/it]
                                                         
{'loss': 0.1042, 'grad_norm': 0.7807578444480896, 'learning_rate': 3.960389907540104e-05, 'epoch': 0.92}

  9%|▉         | 3777/41250 [9:06:32<90:18:19,  8.68s/it][2025-04-25 17:04:15,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:04:15,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.26 | bwd_microstep: 5755.05 | bwd_inner_microstep: 5707.17 | bwd_allreduce_microstep: 47.84 | step_microstep: 18.53
[2025-04-25 17:04:15,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.26 | bwd: 5755.06 | bwd_inner: 5707.17 | bwd_allreduce: 47.85 | step: 18.53

  9%|▉         | 3778/41250 [9:06:40<90:21:48,  8.68s/it]
                                                         
{'loss': 0.0663, 'grad_norm': 1.2169991731643677, 'learning_rate': 3.9603588035651686e-05, 'epoch': 0.92}

  9%|▉         | 3778/41250 [9:06:40<90:21:48,  8.68s/it][2025-04-25 17:04:24,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.48 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 17:04:24,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.68 | bwd_microstep: 5739.37 | bwd_inner_microstep: 5719.35 | bwd_allreduce_microstep: 19.98 | step_microstep: 21.15
[2025-04-25 17:04:24,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.68 | bwd: 5739.38 | bwd_inner: 5719.35 | bwd_allreduce: 20.00 | step: 21.16

  9%|▉         | 3779/41250 [9:06:49<90:23:55,  8.68s/it]
                                                         
{'loss': 0.0659, 'grad_norm': 0.6426492929458618, 'learning_rate': 3.9603276875050055e-05, 'epoch': 0.92}

  9%|▉         | 3779/41250 [9:06:49<90:23:55,  8.68s/it][2025-04-25 17:04:32,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:04:32,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.50 | bwd_microstep: 5714.87 | bwd_inner_microstep: 5663.52 | bwd_allreduce_microstep: 51.30 | step_microstep: 18.72
[2025-04-25 17:04:32,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.50 | bwd: 5714.88 | bwd_inner: 5663.52 | bwd_allreduce: 51.32 | step: 18.72

  9%|▉         | 3780/41250 [9:06:58<90:14:01,  8.67s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 1.114706039428711, 'learning_rate': 3.960296559359809e-05, 'epoch': 0.92}

  9%|▉         | 3780/41250 [9:06:58<90:14:01,  8.67s/it][2025-04-25 17:04:41,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 17:04:41,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.74 | bwd_microstep: 5745.81 | bwd_inner_microstep: 5711.93 | bwd_allreduce_microstep: 33.83 | step_microstep: 18.89
[2025-04-25 17:04:41,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.74 | bwd: 5745.82 | bwd_inner: 5711.93 | bwd_allreduce: 33.84 | step: 18.90

  9%|▉         | 3781/41250 [9:07:06<90:17:46,  8.68s/it]
                                                         
{'loss': 0.1173, 'grad_norm': 1.1479285955429077, 'learning_rate': 3.960265419129769e-05, 'epoch': 0.92}

  9%|▉         | 3781/41250 [9:07:06<90:17:46,  8.68s/it][2025-04-25 17:04:50,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:04:50,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.04 | bwd_microstep: 5780.68 | bwd_inner_microstep: 5711.22 | bwd_allreduce_microstep: 69.41 | step_microstep: 18.79
[2025-04-25 17:04:50,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5780.69 | bwd_inner: 5711.22 | bwd_allreduce: 69.43 | step: 18.80

  9%|▉         | 3782/41250 [9:07:15<90:25:01,  8.69s/it]
                                                         
{'loss': 0.2057, 'grad_norm': 2.5570380687713623, 'learning_rate': 3.960234266815078e-05, 'epoch': 0.92}

  9%|▉         | 3782/41250 [9:07:15<90:25:01,  8.69s/it][2025-04-25 17:04:59,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:04:59,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.02 | bwd_microstep: 5800.39 | bwd_inner_microstep: 5787.58 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.66
[2025-04-25 17:04:59,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.02 | bwd: 5800.40 | bwd_inner: 5787.57 | bwd_allreduce: 12.79 | step: 18.66

  9%|▉         | 3783/41250 [9:07:24<90:41:02,  8.71s/it]
                                                         
{'loss': 0.2718, 'grad_norm': 1.6037395000457764, 'learning_rate': 3.960203102415929e-05, 'epoch': 0.92}

  9%|▉         | 3783/41250 [9:07:24<90:41:02,  8.71s/it][2025-04-25 17:05:07,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 17:05:07,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.12 | bwd_microstep: 5750.59 | bwd_inner_microstep: 5672.74 | bwd_allreduce_microstep: 77.81 | step_microstep: 19.14
[2025-04-25 17:05:07,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.12 | bwd: 5750.60 | bwd_inner: 5672.74 | bwd_allreduce: 77.82 | step: 19.14

  9%|▉         | 3784/41250 [9:07:33<90:33:31,  8.70s/it]
                                                         
{'loss': 0.1352, 'grad_norm': 1.076451063156128, 'learning_rate': 3.9601719259325136e-05, 'epoch': 0.92}

  9%|▉         | 3784/41250 [9:07:33<90:33:31,  8.70s/it][2025-04-25 17:05:16,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 17:05:16,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5736.88 | bwd_inner_microstep: 5701.80 | bwd_allreduce_microstep: 35.03 | step_microstep: 19.51
[2025-04-25 17:05:16,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5736.89 | bwd_inner: 5701.80 | bwd_allreduce: 35.05 | step: 19.52

  9%|▉         | 3785/41250 [9:07:41<90:27:16,  8.69s/it]
                                                         
{'loss': 0.1746, 'grad_norm': 1.2598567008972168, 'learning_rate': 3.960140737365024e-05, 'epoch': 0.92}

  9%|▉         | 3785/41250 [9:07:41<90:27:16,  8.69s/it][2025-04-25 17:05:25,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 17:05:25,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.55 | bwd_microstep: 5735.62 | bwd_inner_microstep: 5722.45 | bwd_allreduce_microstep: 13.12 | step_microstep: 19.18
[2025-04-25 17:05:25,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.55 | bwd: 5735.63 | bwd_inner: 5722.45 | bwd_allreduce: 13.14 | step: 19.18

  9%|▉         | 3786/41250 [9:07:50<90:24:14,  8.69s/it]
                                                         
{'loss': 0.0958, 'grad_norm': 1.1853573322296143, 'learning_rate': 3.9601095367136506e-05, 'epoch': 0.92}

  9%|▉         | 3786/41250 [9:07:50<90:24:14,  8.69s/it][2025-04-25 17:05:33,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:05:33,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5778.69 | bwd_inner_microstep: 5664.78 | bwd_allreduce_microstep: 113.86 | step_microstep: 18.49
[2025-04-25 17:05:33,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5778.70 | bwd_inner: 5664.78 | bwd_allreduce: 113.88 | step: 18.49

  9%|▉         | 3787/41250 [9:07:59<90:24:52,  8.69s/it]
                                                         
{'loss': 0.1641, 'grad_norm': 0.9956166744232178, 'learning_rate': 3.9600783239785895e-05, 'epoch': 0.92}

  9%|▉         | 3787/41250 [9:07:59<90:24:52,  8.69s/it][2025-04-25 17:05:42,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 17:05:42,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.96 | bwd_microstep: 5780.50 | bwd_inner_microstep: 5668.07 | bwd_allreduce_microstep: 112.39 | step_microstep: 18.93
[2025-04-25 17:05:42,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5780.51 | bwd_inner: 5668.07 | bwd_allreduce: 112.40 | step: 18.93

  9%|▉         | 3788/41250 [9:08:07<90:26:10,  8.69s/it]
                                                         
{'loss': 0.2106, 'grad_norm': 1.1957334280014038, 'learning_rate': 3.9600470991600295e-05, 'epoch': 0.92}

  9%|▉         | 3788/41250 [9:08:07<90:26:10,  8.69s/it][2025-04-25 17:05:51,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 17:05:51,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.86 | bwd_microstep: 5737.51 | bwd_inner_microstep: 5724.80 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.62
[2025-04-25 17:05:51,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.86 | bwd: 5737.53 | bwd_inner: 5724.80 | bwd_allreduce: 12.69 | step: 18.62

  9%|▉         | 3789/41250 [9:08:16<90:22:34,  8.69s/it]
                                                         
{'loss': 0.2752, 'grad_norm': 1.6293566226959229, 'learning_rate': 3.960015862258166e-05, 'epoch': 0.92}

  9%|▉         | 3789/41250 [9:08:16<90:22:34,  8.69s/it][2025-04-25 17:05:59,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 17:05:59,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.68 | bwd_microstep: 5742.32 | bwd_inner_microstep: 5716.58 | bwd_allreduce_microstep: 25.70 | step_microstep: 19.40
[2025-04-25 17:05:59,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.68 | bwd: 5742.33 | bwd_inner: 5716.58 | bwd_allreduce: 25.72 | step: 19.40

  9%|▉         | 3790/41250 [9:08:25<90:22:38,  8.69s/it]
                                                         
{'loss': 0.1752, 'grad_norm': 1.8041110038757324, 'learning_rate': 3.959984613273189e-05, 'epoch': 0.92}

  9%|▉         | 3790/41250 [9:08:25<90:22:38,  8.69s/it][2025-04-25 17:06:08,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:06:08,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.12 | bwd_microstep: 5745.40 | bwd_inner_microstep: 5706.69 | bwd_allreduce_microstep: 38.67 | step_microstep: 18.72
[2025-04-25 17:06:08,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.12 | bwd: 5745.42 | bwd_inner: 5706.69 | bwd_allreduce: 38.69 | step: 18.72

  9%|▉         | 3791/41250 [9:08:33<90:22:53,  8.69s/it]
                                                         
{'loss': 0.259, 'grad_norm': 1.770593285560608, 'learning_rate': 3.959953352205293e-05, 'epoch': 0.92}

  9%|▉         | 3791/41250 [9:08:33<90:22:53,  8.69s/it][2025-04-25 17:06:17,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.16 | optimizer_step: 1.06
[2025-04-25 17:06:17,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5784.07 | bwd_inner_microstep: 5659.23 | bwd_allreduce_microstep: 124.78 | step_microstep: 19.43
[2025-04-25 17:06:17,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5784.09 | bwd_inner: 5659.23 | bwd_allreduce: 124.81 | step: 19.43

  9%|▉         | 3792/41250 [9:08:42<90:25:49,  8.69s/it]
                                                         
{'loss': 0.1519, 'grad_norm': 1.2055542469024658, 'learning_rate': 3.95992207905467e-05, 'epoch': 0.92}

  9%|▉         | 3792/41250 [9:08:42<90:25:49,  8.69s/it][2025-04-25 17:06:25,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 17:06:25,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.10 | bwd_microstep: 5686.32 | bwd_inner_microstep: 5659.96 | bwd_allreduce_microstep: 26.31 | step_microstep: 18.82
[2025-04-25 17:06:25,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.10 | bwd: 5686.34 | bwd_inner: 5659.96 | bwd_allreduce: 26.33 | step: 18.83

  9%|▉         | 3793/41250 [9:08:51<90:10:17,  8.67s/it]
                                                         
{'loss': 0.0922, 'grad_norm': 1.5625890493392944, 'learning_rate': 3.9598907938215126e-05, 'epoch': 0.92}

  9%|▉         | 3793/41250 [9:08:51<90:10:17,  8.67s/it][2025-04-25 17:06:34,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:06:34,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.54 | bwd_microstep: 5726.86 | bwd_inner_microstep: 5700.64 | bwd_allreduce_microstep: 26.17 | step_microstep: 18.50
[2025-04-25 17:06:34,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.54 | bwd: 5726.87 | bwd_inner: 5700.64 | bwd_allreduce: 26.19 | step: 18.51

  9%|▉         | 3794/41250 [9:08:59<90:11:12,  8.67s/it]
                                                         
{'loss': 0.2234, 'grad_norm': 3.3364510536193848, 'learning_rate': 3.9598594965060145e-05, 'epoch': 0.92}

  9%|▉         | 3794/41250 [9:08:59<90:11:12,  8.67s/it][2025-04-25 17:06:43,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 17:06:43,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.22 | bwd_microstep: 5764.39 | bwd_inner_microstep: 5675.46 | bwd_allreduce_microstep: 88.87 | step_microstep: 18.70
[2025-04-25 17:06:43,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.22 | bwd: 5764.40 | bwd_inner: 5675.46 | bwd_allreduce: 88.89 | step: 18.70

  9%|▉         | 3795/41250 [9:09:08<90:14:03,  8.67s/it]
                                                         
{'loss': 0.1206, 'grad_norm': 1.4645206928253174, 'learning_rate': 3.959828187108367e-05, 'epoch': 0.92}

  9%|▉         | 3795/41250 [9:09:08<90:14:03,  8.67s/it][2025-04-25 17:06:51,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.09 | optimizer_step: 1.02
[2025-04-25 17:06:51,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.04 | bwd_microstep: 5705.67 | bwd_inner_microstep: 5652.17 | bwd_allreduce_microstep: 53.45 | step_microstep: 19.83
[2025-04-25 17:06:51,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.04 | bwd: 5705.68 | bwd_inner: 5652.17 | bwd_allreduce: 53.47 | step: 19.82

  9%|▉         | 3796/41250 [9:09:17<90:05:52,  8.66s/it]
                                                         
{'loss': 0.1373, 'grad_norm': 0.8329775333404541, 'learning_rate': 3.9597968656287655e-05, 'epoch': 0.92}

  9%|▉         | 3796/41250 [9:09:17<90:05:52,  8.66s/it][2025-04-25 17:07:00,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:07:00,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5736.62 | bwd_inner_microstep: 5695.76 | bwd_allreduce_microstep: 40.81 | step_microstep: 19.02
[2025-04-25 17:07:00,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5736.63 | bwd_inner: 5695.76 | bwd_allreduce: 40.83 | step: 19.02

  9%|▉         | 3797/41250 [9:09:25<90:06:34,  8.66s/it]
                                                         
{'loss': 0.0883, 'grad_norm': 1.0702886581420898, 'learning_rate': 3.959765532067401e-05, 'epoch': 0.92}

  9%|▉         | 3797/41250 [9:09:25<90:06:34,  8.66s/it][2025-04-25 17:07:09,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:07:09,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.44 | bwd_microstep: 5775.09 | bwd_inner_microstep: 5665.73 | bwd_allreduce_microstep: 109.32 | step_microstep: 18.37
[2025-04-25 17:07:09,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.44 | bwd: 5775.11 | bwd_inner: 5665.73 | bwd_allreduce: 109.34 | step: 18.37

  9%|▉         | 3798/41250 [9:09:34<90:11:52,  8.67s/it]
                                                         
{'loss': 0.1083, 'grad_norm': 0.7018879652023315, 'learning_rate': 3.9597341864244684e-05, 'epoch': 0.92}

  9%|▉         | 3798/41250 [9:09:34<90:11:52,  8.67s/it][2025-04-25 17:07:17,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:07:17,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5784.76 | bwd_inner_microstep: 5661.06 | bwd_allreduce_microstep: 123.65 | step_microstep: 18.90
[2025-04-25 17:07:17,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5784.77 | bwd_inner: 5661.06 | bwd_allreduce: 123.67 | step: 18.90

  9%|▉         | 3799/41250 [9:09:43<90:18:03,  8.68s/it]
                                                         
{'loss': 0.1213, 'grad_norm': 1.0200512409210205, 'learning_rate': 3.959702828700159e-05, 'epoch': 0.92}

  9%|▉         | 3799/41250 [9:09:43<90:18:03,  8.68s/it][2025-04-25 17:07:26,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 17:07:26,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5726.11 | bwd_inner_microstep: 5685.21 | bwd_allreduce_microstep: 40.85 | step_microstep: 18.75
[2025-04-25 17:07:26,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5726.12 | bwd_inner: 5685.21 | bwd_allreduce: 40.87 | step: 18.76

  9%|▉         | 3800/41250 [9:09:51<90:13:59,  8.67s/it]
                                                         
{'loss': 0.2211, 'grad_norm': 1.1581733226776123, 'learning_rate': 3.9596714588946675e-05, 'epoch': 0.92}

  9%|▉         | 3800/41250 [9:09:51<90:13:59,  8.67s/it][2025-04-25 17:07:35,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 17:07:35,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.53 | bwd_microstep: 5699.84 | bwd_inner_microstep: 5640.96 | bwd_allreduce_microstep: 58.79 | step_microstep: 19.11
[2025-04-25 17:07:35,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.53 | bwd: 5699.85 | bwd_inner: 5640.99 | bwd_allreduce: 58.80 | step: 19.11

  9%|▉         | 3801/41250 [9:10:00<90:03:31,  8.66s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.149603009223938, 'learning_rate': 3.959640077008187e-05, 'epoch': 0.92}

  9%|▉         | 3801/41250 [9:10:00<90:03:31,  8.66s/it][2025-04-25 17:07:43,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:07:43,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5770.59 | bwd_inner_microstep: 5657.71 | bwd_allreduce_microstep: 112.83 | step_microstep: 18.22
[2025-04-25 17:07:43,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5770.60 | bwd_inner: 5657.71 | bwd_allreduce: 112.84 | step: 18.22

  9%|▉         | 3802/41250 [9:10:09<90:08:35,  8.67s/it]
                                                         
{'loss': 0.1386, 'grad_norm': 0.6517847180366516, 'learning_rate': 3.9596086830409106e-05, 'epoch': 0.92}

  9%|▉         | 3802/41250 [9:10:09<90:08:35,  8.67s/it][2025-04-25 17:07:52,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.83 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:07:52,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.39 | bwd_microstep: 5782.50 | bwd_inner_microstep: 5658.38 | bwd_allreduce_microstep: 124.08 | step_microstep: 18.16
[2025-04-25 17:07:52,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.39 | bwd: 5782.52 | bwd_inner: 5658.38 | bwd_allreduce: 124.10 | step: 18.16

  9%|▉         | 3803/41250 [9:10:17<90:13:33,  8.67s/it]
                                                         
{'loss': 0.0851, 'grad_norm': 0.679947555065155, 'learning_rate': 3.959577276993032e-05, 'epoch': 0.92}

  9%|▉         | 3803/41250 [9:10:17<90:13:33,  8.67s/it][2025-04-25 17:08:01,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:08:01,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.33 | bwd_microstep: 5899.61 | bwd_inner_microstep: 5659.89 | bwd_allreduce_microstep: 239.68 | step_microstep: 18.17
[2025-04-25 17:08:01,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.33 | bwd: 5899.62 | bwd_inner: 5659.89 | bwd_allreduce: 239.69 | step: 18.17

  9%|▉         | 3804/41250 [9:10:26<90:40:28,  8.72s/it]
                                                         
{'loss': 0.1281, 'grad_norm': 1.5673801898956299, 'learning_rate': 3.959545858864745e-05, 'epoch': 0.92}

  9%|▉         | 3804/41250 [9:10:26<90:40:28,  8.72s/it][2025-04-25 17:08:10,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 17:08:10,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.20 | bwd_microstep: 5796.34 | bwd_inner_microstep: 5783.94 | bwd_allreduce_microstep: 12.35 | step_microstep: 18.89
[2025-04-25 17:08:10,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.21 | bwd: 5796.35 | bwd_inner: 5783.94 | bwd_allreduce: 12.37 | step: 18.90

  9%|▉         | 3805/41250 [9:10:35<90:53:23,  8.74s/it]
                                                         
{'loss': 0.1309, 'grad_norm': 1.3444421291351318, 'learning_rate': 3.959514428656244e-05, 'epoch': 0.92}

  9%|▉         | 3805/41250 [9:10:35<90:53:23,  8.74s/it][2025-04-25 17:08:18,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 17:08:18,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.13 | bwd_microstep: 5758.04 | bwd_inner_microstep: 5650.91 | bwd_allreduce_microstep: 107.09 | step_microstep: 18.95
[2025-04-25 17:08:18,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.13 | bwd: 5758.05 | bwd_inner: 5650.91 | bwd_allreduce: 107.11 | step: 18.95

  9%|▉         | 3806/41250 [9:10:44<90:39:34,  8.72s/it]
                                                         
{'loss': 0.2117, 'grad_norm': 1.3886948823928833, 'learning_rate': 3.959482986367721e-05, 'epoch': 0.92}

  9%|▉         | 3806/41250 [9:10:44<90:39:34,  8.72s/it][2025-04-25 17:08:27,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:08:27,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.69 | bwd_microstep: 5748.88 | bwd_inner_microstep: 5655.27 | bwd_allreduce_microstep: 93.58 | step_microstep: 18.18
[2025-04-25 17:08:27,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.69 | bwd: 5748.90 | bwd_inner: 5655.27 | bwd_allreduce: 93.59 | step: 18.19

  9%|▉         | 3807/41250 [9:10:52<90:29:13,  8.70s/it]
                                                         
{'loss': 0.2538, 'grad_norm': 1.1194108724594116, 'learning_rate': 3.959451531999371e-05, 'epoch': 0.92}

  9%|▉         | 3807/41250 [9:10:52<90:29:13,  8.70s/it][2025-04-25 17:08:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:08:36,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.93 | bwd_microstep: 5723.70 | bwd_inner_microstep: 5690.64 | bwd_allreduce_microstep: 33.02 | step_microstep: 18.64
[2025-04-25 17:08:36,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.93 | bwd: 5723.72 | bwd_inner: 5690.64 | bwd_allreduce: 33.04 | step: 18.64

  9%|▉         | 3808/41250 [9:11:01<90:23:35,  8.69s/it]
                                                         
{'loss': 0.0903, 'grad_norm': 1.2833112478256226, 'learning_rate': 3.959420065551387e-05, 'epoch': 0.92}

  9%|▉         | 3808/41250 [9:11:01<90:23:35,  8.69s/it][2025-04-25 17:08:44,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:08:44,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.72 | bwd_microstep: 5757.31 | bwd_inner_microstep: 5653.18 | bwd_allreduce_microstep: 104.09 | step_microstep: 18.55
[2025-04-25 17:08:44,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.72 | bwd: 5757.33 | bwd_inner: 5653.18 | bwd_allreduce: 104.10 | step: 18.55

  9%|▉         | 3809/41250 [9:11:10<90:20:57,  8.69s/it]
                                                         
{'loss': 0.1797, 'grad_norm': 1.5741716623306274, 'learning_rate': 3.959388587023965e-05, 'epoch': 0.92}

  9%|▉         | 3809/41250 [9:11:10<90:20:57,  8.69s/it][2025-04-25 17:08:53,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:08:53,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.79 | bwd_microstep: 5709.00 | bwd_inner_microstep: 5695.77 | bwd_allreduce_microstep: 13.17 | step_microstep: 18.81
[2025-04-25 17:08:53,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.79 | bwd: 5709.01 | bwd_inner: 5695.77 | bwd_allreduce: 13.20 | step: 18.82

  9%|▉         | 3810/41250 [9:11:18<90:12:36,  8.67s/it]
                                                         
{'loss': 0.2486, 'grad_norm': 2.5355920791625977, 'learning_rate': 3.9593570964172973e-05, 'epoch': 0.92}

  9%|▉         | 3810/41250 [9:11:18<90:12:36,  8.67s/it][2025-04-25 17:09:02,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 17:09:02,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.67 | bwd_microstep: 5693.24 | bwd_inner_microstep: 5680.47 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.85
[2025-04-25 17:09:02,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5693.25 | bwd_inner: 5680.47 | bwd_allreduce: 12.74 | step: 18.85

  9%|▉         | 3811/41250 [9:11:27<90:03:31,  8.66s/it]
                                                         
{'loss': 0.1722, 'grad_norm': 1.2221050262451172, 'learning_rate': 3.9593255937315786e-05, 'epoch': 0.92}

  9%|▉         | 3811/41250 [9:11:27<90:03:31,  8.66s/it][2025-04-25 17:09:10,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:09:10,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.86 | bwd_microstep: 5729.11 | bwd_inner_microstep: 5701.78 | bwd_allreduce_microstep: 27.29 | step_microstep: 18.55
[2025-04-25 17:09:10,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.86 | bwd: 5729.12 | bwd_inner: 5701.78 | bwd_allreduce: 27.31 | step: 18.56

  9%|▉         | 3812/41250 [9:11:36<90:04:35,  8.66s/it]
                                                         
{'loss': 0.0707, 'grad_norm': 1.2124512195587158, 'learning_rate': 3.959294078967003e-05, 'epoch': 0.92}

  9%|▉         | 3812/41250 [9:11:36<90:04:35,  8.66s/it][2025-04-25 17:09:19,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.20 | optimizer_step: 0.93
[2025-04-25 17:09:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.05 | bwd_microstep: 5729.30 | bwd_inner_microstep: 5688.42 | bwd_allreduce_microstep: 40.82 | step_microstep: 19.32
[2025-04-25 17:09:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.05 | bwd: 5729.31 | bwd_inner: 5688.42 | bwd_allreduce: 40.84 | step: 19.33

  9%|▉         | 3813/41250 [9:11:44<90:03:15,  8.66s/it]
                                                         
{'loss': 0.1255, 'grad_norm': 1.007340908050537, 'learning_rate': 3.959262552123764e-05, 'epoch': 0.92}

  9%|▉         | 3813/41250 [9:11:44<90:03:15,  8.66s/it][2025-04-25 17:09:28,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 17:09:28,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.75 | bwd_microstep: 6003.11 | bwd_inner_microstep: 5692.24 | bwd_allreduce_microstep: 310.83 | step_microstep: 18.97
[2025-04-25 17:09:28,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.75 | bwd: 6003.13 | bwd_inner: 5692.24 | bwd_allreduce: 310.85 | step: 18.97

  9%|▉         | 3814/41250 [9:11:53<90:56:13,  8.74s/it]
                                                         
{'loss': 0.405, 'grad_norm': 2.3421342372894287, 'learning_rate': 3.959231013202057e-05, 'epoch': 0.92}

  9%|▉         | 3814/41250 [9:11:53<90:56:13,  8.74s/it][2025-04-25 17:09:36,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:09:36,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.76 | bwd_microstep: 5671.92 | bwd_inner_microstep: 5644.12 | bwd_allreduce_microstep: 27.76 | step_microstep: 18.65
[2025-04-25 17:09:36,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.76 | bwd: 5671.94 | bwd_inner: 5644.12 | bwd_allreduce: 27.77 | step: 18.65

  9%|▉         | 3815/41250 [9:12:02<90:26:30,  8.70s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 0.8905684351921082, 'learning_rate': 3.9591994622020766e-05, 'epoch': 0.92}

  9%|▉         | 3815/41250 [9:12:02<90:26:30,  8.70s/it][2025-04-25 17:09:45,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.81 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 17:09:45,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.98 | bwd_microstep: 5779.38 | bwd_inner_microstep: 5651.08 | bwd_allreduce_microstep: 128.25 | step_microstep: 19.40
[2025-04-25 17:09:45,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.98 | bwd: 5779.39 | bwd_inner: 5651.08 | bwd_allreduce: 128.27 | step: 19.40

  9%|▉         | 3816/41250 [9:12:10<90:24:19,  8.69s/it]
                                                         
{'loss': 0.1294, 'grad_norm': 1.3749134540557861, 'learning_rate': 3.959167899124017e-05, 'epoch': 0.93}

  9%|▉         | 3816/41250 [9:12:10<90:24:19,  8.69s/it][2025-04-25 17:09:54,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 17:09:54,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5705.79 | bwd_inner_microstep: 5693.01 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.83
[2025-04-25 17:09:54,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5705.80 | bwd_inner: 5693.01 | bwd_allreduce: 12.75 | step: 18.83

  9%|▉         | 3817/41250 [9:12:19<90:14:06,  8.68s/it]
                                                         
{'loss': 0.1502, 'grad_norm': 1.9978411197662354, 'learning_rate': 3.9591363239680726e-05, 'epoch': 0.93}

  9%|▉         | 3817/41250 [9:12:19<90:14:06,  8.68s/it][2025-04-25 17:10:03,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 17:10:03,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2944.04 | bwd_microstep: 5894.28 | bwd_inner_microstep: 5881.44 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.36
[2025-04-25 17:10:03,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2944.04 | bwd: 5894.29 | bwd_inner: 5881.44 | bwd_allreduce: 12.82 | step: 19.36

  9%|▉         | 3818/41250 [9:12:28<90:59:40,  8.75s/it]
                                                         
{'loss': 0.1069, 'grad_norm': 1.1009764671325684, 'learning_rate': 3.959104736734438e-05, 'epoch': 0.93}

  9%|▉         | 3818/41250 [9:12:28<90:59:40,  8.75s/it][2025-04-25 17:10:11,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 17:10:11,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.60 | bwd_microstep: 5691.71 | bwd_inner_microstep: 5655.58 | bwd_allreduce_microstep: 36.08 | step_microstep: 18.77
[2025-04-25 17:10:11,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.60 | bwd: 5691.72 | bwd_inner: 5655.58 | bwd_allreduce: 36.10 | step: 18.78

  9%|▉         | 3819/41250 [9:12:37<90:31:53,  8.71s/it]
                                                         
{'loss': 0.2339, 'grad_norm': 2.8020660877227783, 'learning_rate': 3.959073137423308e-05, 'epoch': 0.93}

  9%|▉         | 3819/41250 [9:12:37<90:31:53,  8.71s/it][2025-04-25 17:10:20,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:10:20,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.83 | bwd_microstep: 5741.58 | bwd_inner_microstep: 5647.46 | bwd_allreduce_microstep: 94.08 | step_microstep: 18.78
[2025-04-25 17:10:20,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.83 | bwd: 5741.60 | bwd_inner: 5647.46 | bwd_allreduce: 94.10 | step: 18.78

  9%|▉         | 3820/41250 [9:12:45<90:23:56,  8.69s/it]
                                                         
{'loss': 0.1263, 'grad_norm': 1.0672672986984253, 'learning_rate': 3.959041526034877e-05, 'epoch': 0.93}

  9%|▉         | 3820/41250 [9:12:45<90:23:56,  8.69s/it][2025-04-25 17:10:29,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 17:10:29,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.60 | bwd_microstep: 5677.26 | bwd_inner_microstep: 5656.56 | bwd_allreduce_microstep: 20.65 | step_microstep: 18.88
[2025-04-25 17:10:29,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.60 | bwd: 5677.28 | bwd_inner: 5656.56 | bwd_allreduce: 20.67 | step: 18.88

  9%|▉         | 3821/41250 [9:12:54<90:06:51,  8.67s/it]
                                                         
{'loss': 0.0667, 'grad_norm': 0.8128005266189575, 'learning_rate': 3.959009902569341e-05, 'epoch': 0.93}

  9%|▉         | 3821/41250 [9:12:54<90:06:51,  8.67s/it][2025-04-25 17:10:37,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 17:10:37,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.24 | bwd_microstep: 5728.33 | bwd_inner_microstep: 5691.11 | bwd_allreduce_microstep: 37.16 | step_microstep: 20.16
[2025-04-25 17:10:37,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.24 | bwd: 5728.35 | bwd_inner: 5691.11 | bwd_allreduce: 37.19 | step: 20.16

  9%|▉         | 3822/41250 [9:13:03<90:06:31,  8.67s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.601666808128357, 'learning_rate': 3.958978267026894e-05, 'epoch': 0.93}

  9%|▉         | 3822/41250 [9:13:03<90:06:31,  8.67s/it][2025-04-25 17:10:46,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 17:10:46,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5758.51 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 60.81 | step_microstep: 18.88
[2025-04-25 17:10:46,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5758.52 | bwd_inner: 5697.65 | bwd_allreduce: 60.83 | step: 18.88

  9%|▉         | 3823/41250 [9:13:11<90:10:32,  8.67s/it]
                                                         
{'loss': 0.2321, 'grad_norm': 2.041666269302368, 'learning_rate': 3.958946619407731e-05, 'epoch': 0.93}

  9%|▉         | 3823/41250 [9:13:11<90:10:32,  8.67s/it][2025-04-25 17:10:55,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:10:55,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.17 | bwd_microstep: 5689.58 | bwd_inner_microstep: 5661.36 | bwd_allreduce_microstep: 28.18 | step_microstep: 19.11
[2025-04-25 17:10:55,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.17 | bwd: 5689.60 | bwd_inner: 5661.36 | bwd_allreduce: 28.20 | step: 19.11

  9%|▉         | 3824/41250 [9:13:20<89:59:47,  8.66s/it]
                                                         
{'loss': 0.2095, 'grad_norm': 1.7245094776153564, 'learning_rate': 3.958914959712048e-05, 'epoch': 0.93}

  9%|▉         | 3824/41250 [9:13:20<89:59:47,  8.66s/it][2025-04-25 17:11:03,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 17:11:03,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5768.98 | bwd_inner_microstep: 5689.07 | bwd_allreduce_microstep: 79.86 | step_microstep: 18.74
[2025-04-25 17:11:03,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5768.99 | bwd_inner: 5689.07 | bwd_allreduce: 79.88 | step: 18.75

  9%|▉         | 3825/41250 [9:13:29<90:07:15,  8.67s/it]
                                                         
{'loss': 0.1425, 'grad_norm': 1.2750014066696167, 'learning_rate': 3.958883287940039e-05, 'epoch': 0.93}

  9%|▉         | 3825/41250 [9:13:29<90:07:15,  8.67s/it][2025-04-25 17:11:12,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:11:12,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.73 | bwd_microstep: 5768.63 | bwd_inner_microstep: 5664.80 | bwd_allreduce_microstep: 103.78 | step_microstep: 18.47
[2025-04-25 17:11:12,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.73 | bwd: 5768.64 | bwd_inner: 5664.80 | bwd_allreduce: 103.80 | step: 18.47

  9%|▉         | 3826/41250 [9:13:37<90:09:47,  8.67s/it]
                                                         
{'loss': 0.3544, 'grad_norm': 2.6588525772094727, 'learning_rate': 3.9588516040919e-05, 'epoch': 0.93}

  9%|▉         | 3826/41250 [9:13:37<90:09:47,  8.67s/it][2025-04-25 17:11:21,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-25 17:11:21,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.83 | bwd_microstep: 5699.44 | bwd_inner_microstep: 5686.03 | bwd_allreduce_microstep: 13.37 | step_microstep: 19.48
[2025-04-25 17:11:21,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.83 | bwd: 5699.46 | bwd_inner: 5686.03 | bwd_allreduce: 13.39 | step: 19.48

  9%|▉         | 3827/41250 [9:13:46<90:01:14,  8.66s/it]
                                                         
{'loss': 0.2423, 'grad_norm': 2.334362745285034, 'learning_rate': 3.958819908167826e-05, 'epoch': 0.93}

  9%|▉         | 3827/41250 [9:13:46<90:01:14,  8.66s/it][2025-04-25 17:11:29,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 17:11:29,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.49 | bwd_microstep: 5711.56 | bwd_inner_microstep: 5698.50 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.81
[2025-04-25 17:11:29,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.49 | bwd: 5711.57 | bwd_inner: 5698.50 | bwd_allreduce: 13.03 | step: 18.81

  9%|▉         | 3828/41250 [9:13:55<89:58:46,  8.66s/it]
                                                         
{'loss': 0.1535, 'grad_norm': 2.082373857498169, 'learning_rate': 3.958788200168013e-05, 'epoch': 0.93}

  9%|▉         | 3828/41250 [9:13:55<89:58:46,  8.66s/it][2025-04-25 17:11:38,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:11:38,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.54 | bwd_microstep: 5788.27 | bwd_inner_microstep: 5665.26 | bwd_allreduce_microstep: 122.94 | step_microstep: 18.79
[2025-04-25 17:11:38,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.56 | bwd: 5788.27 | bwd_inner: 5665.25 | bwd_allreduce: 122.96 | step: 18.79

  9%|▉         | 3829/41250 [9:14:03<90:08:26,  8.67s/it]
                                                         
{'loss': 0.2306, 'grad_norm': 1.508876919746399, 'learning_rate': 3.958756480092656e-05, 'epoch': 0.93}

  9%|▉         | 3829/41250 [9:14:03<90:08:26,  8.67s/it][2025-04-25 17:11:47,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 17:11:47,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.75 | bwd_microstep: 5749.73 | bwd_inner_microstep: 5718.89 | bwd_allreduce_microstep: 30.79 | step_microstep: 19.08
[2025-04-25 17:11:47,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.75 | bwd: 5749.74 | bwd_inner: 5718.89 | bwd_allreduce: 30.81 | step: 19.09

  9%|▉         | 3830/41250 [9:14:12<90:10:38,  8.68s/it]
                                                         
{'loss': 0.1309, 'grad_norm': 1.1952638626098633, 'learning_rate': 3.9587247479419504e-05, 'epoch': 0.93}

  9%|▉         | 3830/41250 [9:14:12<90:10:38,  8.68s/it][2025-04-25 17:11:55,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 17:11:55,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.79 | bwd_microstep: 5758.16 | bwd_inner_microstep: 5705.23 | bwd_allreduce_microstep: 52.88 | step_microstep: 19.00
[2025-04-25 17:11:55,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5758.18 | bwd_inner: 5705.23 | bwd_allreduce: 52.90 | step: 19.00

  9%|▉         | 3831/41250 [9:14:21<90:14:11,  8.68s/it]
                                                         
{'loss': 0.3437, 'grad_norm': 2.908994674682617, 'learning_rate': 3.9586930037160915e-05, 'epoch': 0.93}

  9%|▉         | 3831/41250 [9:14:21<90:14:11,  8.68s/it][2025-04-25 17:12:04,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.20 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:12:04,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.52 | bwd_microstep: 5779.77 | bwd_inner_microstep: 5703.89 | bwd_allreduce_microstep: 75.83 | step_microstep: 20.08
[2025-04-25 17:12:04,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.52 | bwd: 5779.79 | bwd_inner: 5703.89 | bwd_allreduce: 75.85 | step: 20.09

  9%|▉         | 3832/41250 [9:14:29<90:21:09,  8.69s/it]
                                                         
{'loss': 0.1109, 'grad_norm': 1.1242363452911377, 'learning_rate': 3.958661247415276e-05, 'epoch': 0.93}

  9%|▉         | 3832/41250 [9:14:29<90:21:09,  8.69s/it][2025-04-25 17:12:13,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:12:13,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5757.66 | bwd_inner_microstep: 5694.18 | bwd_allreduce_microstep: 63.44 | step_microstep: 18.72
[2025-04-25 17:12:13,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.49 | bwd: 5757.68 | bwd_inner: 5694.18 | bwd_allreduce: 63.46 | step: 18.72

  9%|▉         | 3833/41250 [9:14:38<90:20:48,  8.69s/it]
                                                         
{'loss': 0.1105, 'grad_norm': 1.2442444562911987, 'learning_rate': 3.958629479039699e-05, 'epoch': 0.93}

  9%|▉         | 3833/41250 [9:14:38<90:20:48,  8.69s/it][2025-04-25 17:12:21,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.16 | optimizer_step: 1.01
[2025-04-25 17:12:21,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.79 | bwd_microstep: 5755.55 | bwd_inner_microstep: 5728.86 | bwd_allreduce_microstep: 26.65 | step_microstep: 19.28
[2025-04-25 17:12:21,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.79 | bwd: 5755.56 | bwd_inner: 5728.86 | bwd_allreduce: 26.66 | step: 19.28

  9%|▉         | 3834/41250 [9:14:47<90:21:51,  8.69s/it]
                                                         
{'loss': 0.3012, 'grad_norm': 1.6908690929412842, 'learning_rate': 3.958597698589557e-05, 'epoch': 0.93}

  9%|▉         | 3834/41250 [9:14:47<90:21:51,  8.69s/it][2025-04-25 17:12:30,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 17:12:30,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.78 | bwd_microstep: 5773.19 | bwd_inner_microstep: 5656.96 | bwd_allreduce_microstep: 116.18 | step_microstep: 18.60
[2025-04-25 17:12:30,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.78 | bwd: 5773.20 | bwd_inner: 5656.96 | bwd_allreduce: 116.20 | step: 18.60

  9%|▉         | 3835/41250 [9:14:55<90:20:03,  8.69s/it]
                                                         
{'loss': 0.1481, 'grad_norm': 4.29367208480835, 'learning_rate': 3.958565906065045e-05, 'epoch': 0.93}

  9%|▉         | 3835/41250 [9:14:55<90:20:03,  8.69s/it][2025-04-25 17:12:39,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:12:39,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.74 | bwd_microstep: 5771.42 | bwd_inner_microstep: 5671.74 | bwd_allreduce_microstep: 99.64 | step_microstep: 18.77
[2025-04-25 17:12:39,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.74 | bwd: 5771.44 | bwd_inner: 5671.74 | bwd_allreduce: 99.66 | step: 18.77

  9%|▉         | 3836/41250 [9:15:04<90:18:34,  8.69s/it]
                                                         
{'loss': 0.2272, 'grad_norm': 1.772733211517334, 'learning_rate': 3.958534101466359e-05, 'epoch': 0.93}

  9%|▉         | 3836/41250 [9:15:04<90:18:34,  8.69s/it][2025-04-25 17:12:47,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:12:47,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.79 | bwd_microstep: 5784.37 | bwd_inner_microstep: 5659.15 | bwd_allreduce_microstep: 125.17 | step_microstep: 18.28
[2025-04-25 17:12:47,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.79 | bwd: 5784.38 | bwd_inner: 5659.15 | bwd_allreduce: 125.19 | step: 18.28

  9%|▉         | 3837/41250 [9:15:13<90:19:50,  8.69s/it]
                                                         
{'loss': 0.2534, 'grad_norm': 1.8911691904067993, 'learning_rate': 3.9585022847936957e-05, 'epoch': 0.93}

  9%|▉         | 3837/41250 [9:15:13<90:19:50,  8.69s/it][2025-04-25 17:12:56,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:12:56,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.79 | bwd_microstep: 5803.94 | bwd_inner_microstep: 5668.42 | bwd_allreduce_microstep: 135.48 | step_microstep: 18.63
[2025-04-25 17:12:56,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.79 | bwd: 5803.96 | bwd_inner: 5668.42 | bwd_allreduce: 135.50 | step: 18.63

  9%|▉         | 3838/41250 [9:15:22<90:24:23,  8.70s/it]
                                                         
{'loss': 0.212, 'grad_norm': 1.627981424331665, 'learning_rate': 3.958470456047251e-05, 'epoch': 0.93}

  9%|▉         | 3838/41250 [9:15:22<90:24:23,  8.70s/it][2025-04-25 17:13:05,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.65 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:13:05,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.98 | bwd_microstep: 5767.48 | bwd_inner_microstep: 5693.16 | bwd_allreduce_microstep: 74.27 | step_microstep: 19.06
[2025-04-25 17:13:05,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.98 | bwd: 5767.49 | bwd_inner: 5693.16 | bwd_allreduce: 74.29 | step: 19.07

  9%|▉         | 3839/41250 [9:15:30<90:24:16,  8.70s/it]
                                                         
{'loss': 0.2338, 'grad_norm': 1.967821717262268, 'learning_rate': 3.958438615227222e-05, 'epoch': 0.93}

  9%|▉         | 3839/41250 [9:15:30<90:24:16,  8.70s/it][2025-04-25 17:13:14,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 17:13:14,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5774.99 | bwd_inner_microstep: 5707.37 | bwd_allreduce_microstep: 67.58 | step_microstep: 18.45
[2025-04-25 17:13:14,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5775.00 | bwd_inner: 5707.37 | bwd_allreduce: 67.59 | step: 18.46

  9%|▉         | 3840/41250 [9:15:39<90:25:21,  8.70s/it]
                                                         
{'loss': 0.132, 'grad_norm': 1.7314342260360718, 'learning_rate': 3.958406762333803e-05, 'epoch': 0.93}

  9%|▉         | 3840/41250 [9:15:39<90:25:21,  8.70s/it][2025-04-25 17:13:22,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:13:22,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5779.56 | bwd_inner_microstep: 5703.77 | bwd_allreduce_microstep: 75.75 | step_microstep: 18.23
[2025-04-25 17:13:22,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5779.57 | bwd_inner: 5703.77 | bwd_allreduce: 75.77 | step: 18.23

  9%|▉         | 3841/41250 [9:15:48<90:28:31,  8.71s/it]
                                                         
{'loss': 0.25, 'grad_norm': 1.4142708778381348, 'learning_rate': 3.9583748973671925e-05, 'epoch': 0.93}

  9%|▉         | 3841/41250 [9:15:48<90:28:31,  8.71s/it][2025-04-25 17:13:31,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 17:13:31,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.61 | bwd_microstep: 5751.27 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 57.20 | step_microstep: 18.48
[2025-04-25 17:13:31,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.61 | bwd: 5751.28 | bwd_inner: 5694.02 | bwd_allreduce: 57.22 | step: 18.48

  9%|▉         | 3842/41250 [9:15:56<90:23:21,  8.70s/it]
                                                         
{'loss': 0.0275, 'grad_norm': 0.2557748556137085, 'learning_rate': 3.958343020327585e-05, 'epoch': 0.93}

  9%|▉         | 3842/41250 [9:15:56<90:23:21,  8.70s/it][2025-04-25 17:13:40,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:13:40,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.85 | bwd_microstep: 5728.51 | bwd_inner_microstep: 5715.71 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.46
[2025-04-25 17:13:40,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.85 | bwd: 5728.52 | bwd_inner: 5715.71 | bwd_allreduce: 12.77 | step: 18.46

  9%|▉         | 3843/41250 [9:16:05<90:18:48,  8.69s/it]
                                                         
{'loss': 0.0315, 'grad_norm': 0.4543265700340271, 'learning_rate': 3.958311131215179e-05, 'epoch': 0.93}

  9%|▉         | 3843/41250 [9:16:05<90:18:48,  8.69s/it][2025-04-25 17:13:48,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 17:13:48,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.73 | bwd_microstep: 5735.82 | bwd_inner_microstep: 5722.85 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.43
[2025-04-25 17:13:48,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.73 | bwd: 5735.83 | bwd_inner: 5722.85 | bwd_allreduce: 12.93 | step: 19.44

  9%|▉         | 3844/41250 [9:16:14<90:15:36,  8.69s/it]
                                                         
{'loss': 0.2868, 'grad_norm': 5.123600006103516, 'learning_rate': 3.958279230030169e-05, 'epoch': 0.93}

  9%|▉         | 3844/41250 [9:16:14<90:15:36,  8.69s/it][2025-04-25 17:13:57,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:13:57,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5771.38 | bwd_inner_microstep: 5647.76 | bwd_allreduce_microstep: 123.58 | step_microstep: 18.72
[2025-04-25 17:13:57,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5771.40 | bwd_inner: 5647.76 | bwd_allreduce: 123.60 | step: 18.72

  9%|▉         | 3845/41250 [9:16:22<90:14:45,  8.69s/it]
                                                         
{'loss': 0.2663, 'grad_norm': 1.2372658252716064, 'learning_rate': 3.9582473167727535e-05, 'epoch': 0.93}

  9%|▉         | 3845/41250 [9:16:22<90:14:45,  8.69s/it][2025-04-25 17:14:06,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:14:06,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.42 | bwd_microstep: 5719.32 | bwd_inner_microstep: 5706.22 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.87
[2025-04-25 17:14:06,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.42 | bwd: 5719.33 | bwd_inner: 5706.22 | bwd_allreduce: 13.08 | step: 18.88

  9%|▉         | 3846/41250 [9:16:31<90:08:40,  8.68s/it]
                                                         
{'loss': 0.0559, 'grad_norm': 0.6587693691253662, 'learning_rate': 3.958215391443128e-05, 'epoch': 0.93}

  9%|▉         | 3846/41250 [9:16:31<90:08:40,  8.68s/it][2025-04-25 17:14:14,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 17:14:14,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5710.50 | bwd_inner_microstep: 5660.12 | bwd_allreduce_microstep: 50.33 | step_microstep: 18.90
[2025-04-25 17:14:14,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5710.51 | bwd_inner: 5660.12 | bwd_allreduce: 50.35 | step: 18.91

  9%|▉         | 3847/41250 [9:16:40<89:59:34,  8.66s/it]
                                                         
{'loss': 0.1609, 'grad_norm': 1.118959665298462, 'learning_rate': 3.95818345404149e-05, 'epoch': 0.93}

  9%|▉         | 3847/41250 [9:16:40<89:59:34,  8.66s/it][2025-04-25 17:14:23,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 17:14:23,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.89 | bwd_microstep: 5727.35 | bwd_inner_microstep: 5713.82 | bwd_allreduce_microstep: 13.47 | step_microstep: 19.22
[2025-04-25 17:14:23,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.89 | bwd: 5727.37 | bwd_inner: 5713.82 | bwd_allreduce: 13.50 | step: 19.22

  9%|▉         | 3848/41250 [9:16:48<90:01:19,  8.66s/it]
                                                         
{'loss': 0.2256, 'grad_norm': 2.1784467697143555, 'learning_rate': 3.958151504568036e-05, 'epoch': 0.93}

  9%|▉         | 3848/41250 [9:16:48<90:01:19,  8.66s/it][2025-04-25 17:14:32,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-25 17:14:32,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.36 | bwd_microstep: 5741.33 | bwd_inner_microstep: 5697.33 | bwd_allreduce_microstep: 43.95 | step_microstep: 19.64
[2025-04-25 17:14:32,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.36 | bwd: 5741.35 | bwd_inner: 5697.33 | bwd_allreduce: 43.97 | step: 19.64

  9%|▉         | 3849/41250 [9:16:57<90:04:41,  8.67s/it]
                                                         
{'loss': 0.1476, 'grad_norm': 1.4068992137908936, 'learning_rate': 3.9581195430229636e-05, 'epoch': 0.93}

  9%|▉         | 3849/41250 [9:16:57<90:04:41,  8.67s/it][2025-04-25 17:14:40,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 17:14:40,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.43 | bwd_microstep: 5720.74 | bwd_inner_microstep: 5707.83 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.47
[2025-04-25 17:14:40,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.43 | bwd: 5720.76 | bwd_inner: 5707.83 | bwd_allreduce: 12.88 | step: 19.47

  9%|▉         | 3850/41250 [9:17:06<90:02:02,  8.67s/it]
                                                         
{'loss': 0.3467, 'grad_norm': 4.152499198913574, 'learning_rate': 3.958087569406469e-05, 'epoch': 0.93}

  9%|▉         | 3850/41250 [9:17:06<90:02:02,  8.67s/it][2025-04-25 17:14:49,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 17:14:49,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5700.46 | bwd_inner_microstep: 5654.57 | bwd_allreduce_microstep: 45.84 | step_microstep: 18.98
[2025-04-25 17:14:49,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5700.47 | bwd_inner: 5654.57 | bwd_allreduce: 45.86 | step: 18.99

  9%|▉         | 3851/41250 [9:17:14<89:51:30,  8.65s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 2.5522966384887695, 'learning_rate': 3.958055583718749e-05, 'epoch': 0.93}

  9%|▉         | 3851/41250 [9:17:14<89:51:30,  8.65s/it][2025-04-25 17:14:58,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.38 | optimizer_step: 0.91
[2025-04-25 17:14:58,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.36 | bwd_microstep: 5695.62 | bwd_inner_microstep: 5662.24 | bwd_allreduce_microstep: 33.33 | step_microstep: 19.77
[2025-04-25 17:14:58,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.36 | bwd: 5695.64 | bwd_inner: 5662.24 | bwd_allreduce: 33.35 | step: 19.78

  9%|▉         | 3852/41250 [9:17:23<89:43:45,  8.64s/it]
                                                         
{'loss': 0.2214, 'grad_norm': 1.608640432357788, 'learning_rate': 3.958023585960002e-05, 'epoch': 0.93}

  9%|▉         | 3852/41250 [9:17:23<89:43:45,  8.64s/it][2025-04-25 17:15:06,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:15:06,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5764.47 | bwd_inner_microstep: 5654.94 | bwd_allreduce_microstep: 109.49 | step_microstep: 18.62
[2025-04-25 17:15:06,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5764.49 | bwd_inner: 5654.94 | bwd_allreduce: 109.51 | step: 18.62

  9%|▉         | 3853/41250 [9:17:32<89:50:13,  8.65s/it]
                                                         
{'loss': 0.3709, 'grad_norm': 2.501481294631958, 'learning_rate': 3.957991576130425e-05, 'epoch': 0.93}

  9%|▉         | 3853/41250 [9:17:32<89:50:13,  8.65s/it][2025-04-25 17:15:15,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 17:15:15,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.93 | bwd_microstep: 5765.67 | bwd_inner_microstep: 5753.10 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.68
[2025-04-25 17:15:15,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.94 | bwd: 5765.69 | bwd_inner: 5753.10 | bwd_allreduce: 12.54 | step: 18.69

  9%|▉         | 3854/41250 [9:17:40<90:05:30,  8.67s/it]
                                                         
{'loss': 0.3027, 'grad_norm': 1.4514777660369873, 'learning_rate': 3.957959554230215e-05, 'epoch': 0.93}

  9%|▉         | 3854/41250 [9:17:40<90:05:30,  8.67s/it][2025-04-25 17:15:24,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:15:24,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5714.08 | bwd_inner_microstep: 5701.31 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.47
[2025-04-25 17:15:24,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5714.10 | bwd_inner: 5701.31 | bwd_allreduce: 12.74 | step: 18.47

  9%|▉         | 3855/41250 [9:17:49<90:00:57,  8.67s/it]
                                                         
{'loss': 0.1688, 'grad_norm': 2.661764621734619, 'learning_rate': 3.957927520259569e-05, 'epoch': 0.93}

  9%|▉         | 3855/41250 [9:17:49<90:00:57,  8.67s/it][2025-04-25 17:15:32,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 17:15:32,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5725.82 | bwd_inner_microstep: 5684.23 | bwd_allreduce_microstep: 41.55 | step_microstep: 19.20
[2025-04-25 17:15:32,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5725.84 | bwd_inner: 5684.23 | bwd_allreduce: 41.57 | step: 19.20

  9%|▉         | 3856/41250 [9:17:58<89:58:06,  8.66s/it]
                                                         
{'loss': 0.1802, 'grad_norm': 1.2202719449996948, 'learning_rate': 3.957895474218686e-05, 'epoch': 0.93}

  9%|▉         | 3856/41250 [9:17:58<89:58:06,  8.66s/it][2025-04-25 17:15:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 17:15:41,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.56 | bwd_microstep: 5754.50 | bwd_inner_microstep: 5652.10 | bwd_allreduce_microstep: 102.35 | step_microstep: 18.92
[2025-04-25 17:15:41,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.56 | bwd: 5754.51 | bwd_inner: 5652.10 | bwd_allreduce: 102.38 | step: 18.92

  9%|▉         | 3857/41250 [9:18:06<89:58:03,  8.66s/it]
                                                         
{'loss': 0.2439, 'grad_norm': 2.1770591735839844, 'learning_rate': 3.957863416107761e-05, 'epoch': 0.94}

  9%|▉         | 3857/41250 [9:18:06<89:58:03,  8.66s/it][2025-04-25 17:15:50,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 17:15:50,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.79 | bwd_microstep: 5753.73 | bwd_inner_microstep: 5656.62 | bwd_allreduce_microstep: 97.06 | step_microstep: 18.86
[2025-04-25 17:15:50,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.79 | bwd: 5753.75 | bwd_inner: 5656.62 | bwd_allreduce: 97.08 | step: 18.86

  9%|▉         | 3858/41250 [9:18:15<89:59:20,  8.66s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 2.236886978149414, 'learning_rate': 3.957831345926994e-05, 'epoch': 0.94}

  9%|▉         | 3858/41250 [9:18:15<89:59:20,  8.66s/it][2025-04-25 17:15:58,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 17:15:58,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5709.11 | bwd_inner_microstep: 5696.04 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.93
[2025-04-25 17:15:58,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.76 | bwd: 5709.13 | bwd_inner: 5696.04 | bwd_allreduce: 13.04 | step: 18.93

  9%|▉         | 3859/41250 [9:18:24<89:55:57,  8.66s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 1.209367275238037, 'learning_rate': 3.957799263676582e-05, 'epoch': 0.94}

  9%|▉         | 3859/41250 [9:18:24<89:55:57,  8.66s/it][2025-04-25 17:16:07,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:16:07,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5735.90 | bwd_inner_microstep: 5700.74 | bwd_allreduce_microstep: 35.12 | step_microstep: 18.72
[2025-04-25 17:16:07,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5735.91 | bwd_inner: 5700.74 | bwd_allreduce: 35.13 | step: 18.73

  9%|▉         | 3860/41250 [9:18:32<89:56:01,  8.66s/it]
                                                         
{'loss': 0.1209, 'grad_norm': 1.1272836923599243, 'learning_rate': 3.9577671693567226e-05, 'epoch': 0.94}

  9%|▉         | 3860/41250 [9:18:32<89:56:01,  8.66s/it][2025-04-25 17:16:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 17:16:16,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.39 | bwd_microstep: 5794.04 | bwd_inner_microstep: 5781.00 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.98
[2025-04-25 17:16:16,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.39 | bwd: 5794.06 | bwd_inner: 5781.00 | bwd_allreduce: 13.02 | step: 18.99

  9%|▉         | 3861/41250 [9:18:41<90:15:17,  8.69s/it]
                                                         
{'loss': 0.1752, 'grad_norm': 1.281298279762268, 'learning_rate': 3.957735062967613e-05, 'epoch': 0.94}

  9%|▉         | 3861/41250 [9:18:41<90:15:17,  8.69s/it][2025-04-25 17:16:24,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 17:16:24,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.29 | bwd_microstep: 5838.43 | bwd_inner_microstep: 5696.80 | bwd_allreduce_microstep: 141.58 | step_microstep: 18.81
[2025-04-25 17:16:24,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.29 | bwd: 5838.44 | bwd_inner: 5696.80 | bwd_allreduce: 141.59 | step: 18.82

  9%|▉         | 3862/41250 [9:18:50<90:29:47,  8.71s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 1.262754201889038, 'learning_rate': 3.9577029445094526e-05, 'epoch': 0.94}

  9%|▉         | 3862/41250 [9:18:50<90:29:47,  8.71s/it][2025-04-25 17:16:33,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:16:33,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5682.11 | bwd_inner_microstep: 5638.96 | bwd_allreduce_microstep: 43.10 | step_microstep: 19.02
[2025-04-25 17:16:33,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5682.12 | bwd_inner: 5638.96 | bwd_allreduce: 43.11 | step: 19.02

  9%|▉         | 3863/41250 [9:18:58<90:08:57,  8.68s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.7904208898544312, 'learning_rate': 3.957670813982438e-05, 'epoch': 0.94}

  9%|▉         | 3863/41250 [9:18:58<90:08:57,  8.68s/it][2025-04-25 17:16:42,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 17:16:42,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5727.35 | bwd_inner_microstep: 5687.17 | bwd_allreduce_microstep: 40.12 | step_microstep: 19.02
[2025-04-25 17:16:42,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5727.36 | bwd_inner: 5687.17 | bwd_allreduce: 40.14 | step: 19.04

  9%|▉         | 3864/41250 [9:19:07<90:04:45,  8.67s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 0.8355427384376526, 'learning_rate': 3.9576386713867683e-05, 'epoch': 0.94}

  9%|▉         | 3864/41250 [9:19:07<90:04:45,  8.67s/it][2025-04-25 17:16:50,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:16:50,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.63 | bwd_microstep: 5684.46 | bwd_inner_microstep: 5645.79 | bwd_allreduce_microstep: 38.63 | step_microstep: 18.35
[2025-04-25 17:16:50,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.63 | bwd: 5684.48 | bwd_inner: 5645.79 | bwd_allreduce: 38.64 | step: 18.36

  9%|▉         | 3865/41250 [9:19:16<89:50:08,  8.65s/it]
                                                         
{'loss': 0.1773, 'grad_norm': 1.8186333179473877, 'learning_rate': 3.957606516722641e-05, 'epoch': 0.94}

  9%|▉         | 3865/41250 [9:19:16<89:50:08,  8.65s/it][2025-04-25 17:16:59,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 17:16:59,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.12 | bwd_microstep: 5892.95 | bwd_inner_microstep: 5695.29 | bwd_allreduce_microstep: 197.62 | step_microstep: 18.57
[2025-04-25 17:16:59,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.12 | bwd: 5892.96 | bwd_inner: 5695.29 | bwd_allreduce: 197.64 | step: 18.58

  9%|▉         | 3866/41250 [9:19:24<90:23:07,  8.70s/it]
                                                         
{'loss': 0.2067, 'grad_norm': 1.0083647966384888, 'learning_rate': 3.957574349990254e-05, 'epoch': 0.94}

  9%|▉         | 3866/41250 [9:19:24<90:23:07,  8.70s/it][2025-04-25 17:17:08,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 17:17:08,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.36 | bwd_microstep: 5789.03 | bwd_inner_microstep: 5776.17 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.99
[2025-04-25 17:17:08,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.36 | bwd: 5789.04 | bwd_inner: 5776.17 | bwd_allreduce: 12.83 | step: 18.99

  9%|▉         | 3867/41250 [9:19:33<90:35:31,  8.72s/it]
                                                         
{'loss': 0.1816, 'grad_norm': 0.9428156614303589, 'learning_rate': 3.9575421711898075e-05, 'epoch': 0.94}

  9%|▉         | 3867/41250 [9:19:33<90:35:31,  8.72s/it][2025-04-25 17:17:16,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:17:16,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5688.68 | bwd_inner_microstep: 5644.99 | bwd_allreduce_microstep: 43.64 | step_microstep: 18.59
[2025-04-25 17:17:16,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5688.69 | bwd_inner: 5644.99 | bwd_allreduce: 43.66 | step: 18.60

  9%|▉         | 3868/41250 [9:19:42<90:11:17,  8.69s/it]
                                                         
{'loss': 0.1818, 'grad_norm': 1.4494539499282837, 'learning_rate': 3.9575099803214975e-05, 'epoch': 0.94}

  9%|▉         | 3868/41250 [9:19:42<90:11:17,  8.69s/it][2025-04-25 17:17:25,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.33 | optimizer_step: 0.90
[2025-04-25 17:17:25,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.44 | bwd_microstep: 5768.80 | bwd_inner_microstep: 5699.50 | bwd_allreduce_microstep: 69.25 | step_microstep: 19.43
[2025-04-25 17:17:25,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.44 | bwd: 5768.81 | bwd_inner: 5699.50 | bwd_allreduce: 69.27 | step: 19.43

  9%|▉         | 3869/41250 [9:19:51<90:16:59,  8.69s/it]
                                                         
{'loss': 0.1998, 'grad_norm': 1.5459108352661133, 'learning_rate': 3.957477777385524e-05, 'epoch': 0.94}

  9%|▉         | 3869/41250 [9:19:51<90:16:59,  8.69s/it][2025-04-25 17:17:34,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:17:34,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.20 | bwd_microstep: 5713.83 | bwd_inner_microstep: 5701.01 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.73
[2025-04-25 17:17:34,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.20 | bwd: 5713.85 | bwd_inner: 5701.01 | bwd_allreduce: 12.80 | step: 18.72

  9%|▉         | 3870/41250 [9:19:59<90:09:04,  8.68s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.2835661172866821, 'learning_rate': 3.957445562382085e-05, 'epoch': 0.94}

  9%|▉         | 3870/41250 [9:19:59<90:09:04,  8.68s/it][2025-04-25 17:17:43,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:17:43,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.78 | bwd_microstep: 5739.64 | bwd_inner_microstep: 5654.44 | bwd_allreduce_microstep: 85.16 | step_microstep: 18.63
[2025-04-25 17:17:43,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.78 | bwd: 5739.65 | bwd_inner: 5654.44 | bwd_allreduce: 85.18 | step: 18.63

  9%|▉         | 3871/41250 [9:20:08<90:05:55,  8.68s/it]
                                                         
{'loss': 0.2388, 'grad_norm': 2.0905444622039795, 'learning_rate': 3.957413335311379e-05, 'epoch': 0.94}

  9%|▉         | 3871/41250 [9:20:08<90:05:55,  8.68s/it][2025-04-25 17:17:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:17:51,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5756.73 | bwd_inner_microstep: 5697.70 | bwd_allreduce_microstep: 58.99 | step_microstep: 18.78
[2025-04-25 17:17:51,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5756.75 | bwd_inner: 5697.70 | bwd_allreduce: 59.00 | step: 18.78

  9%|▉         | 3872/41250 [9:20:17<90:07:59,  8.68s/it]
                                                         
{'loss': 0.174, 'grad_norm': 1.3689486980438232, 'learning_rate': 3.9573810961736046e-05, 'epoch': 0.94}

  9%|▉         | 3872/41250 [9:20:17<90:07:59,  8.68s/it][2025-04-25 17:18:00,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:18:00,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5773.67 | bwd_inner_microstep: 5670.50 | bwd_allreduce_microstep: 103.12 | step_microstep: 18.63
[2025-04-25 17:18:00,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.48 | bwd: 5773.68 | bwd_inner: 5670.50 | bwd_allreduce: 103.14 | step: 18.63

  9%|▉         | 3873/41250 [9:20:25<90:08:03,  8.68s/it]
                                                         
{'loss': 0.0995, 'grad_norm': 0.8520533442497253, 'learning_rate': 3.9573488449689606e-05, 'epoch': 0.94}

  9%|▉         | 3873/41250 [9:20:25<90:08:03,  8.68s/it][2025-04-25 17:18:09,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 17:18:09,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.72 | bwd_microstep: 5749.90 | bwd_inner_microstep: 5659.62 | bwd_allreduce_microstep: 90.23 | step_microstep: 19.20
[2025-04-25 17:18:09,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.72 | bwd: 5749.91 | bwd_inner: 5659.62 | bwd_allreduce: 90.25 | step: 19.20

  9%|▉         | 3874/41250 [9:20:34<90:06:11,  8.68s/it]
                                                         
{'loss': 0.0775, 'grad_norm': 0.9053645133972168, 'learning_rate': 3.9573165816976464e-05, 'epoch': 0.94}

  9%|▉         | 3874/41250 [9:20:34<90:06:11,  8.68s/it][2025-04-25 17:18:17,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:18:17,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.91 | bwd_microstep: 5704.44 | bwd_inner_microstep: 5690.19 | bwd_allreduce_microstep: 14.21 | step_microstep: 18.84
[2025-04-25 17:18:17,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.91 | bwd: 5704.45 | bwd_inner: 5690.19 | bwd_allreduce: 14.23 | step: 18.84

  9%|▉         | 3875/41250 [9:20:43<89:58:18,  8.67s/it]
                                                         
{'loss': 0.1249, 'grad_norm': 1.1276153326034546, 'learning_rate': 3.95728430635986e-05, 'epoch': 0.94}

  9%|▉         | 3875/41250 [9:20:43<89:58:18,  8.67s/it][2025-04-25 17:18:26,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:18:26,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.63 | bwd_microstep: 5867.39 | bwd_inner_microstep: 5854.70 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.85
[2025-04-25 17:18:26,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.63 | bwd: 5867.41 | bwd_inner: 5854.70 | bwd_allreduce: 12.66 | step: 18.85

  9%|▉         | 3876/41250 [9:20:51<90:39:41,  8.73s/it]
                                                         
{'loss': 0.0811, 'grad_norm': 0.7474228143692017, 'learning_rate': 3.9572520189558014e-05, 'epoch': 0.94}

  9%|▉         | 3876/41250 [9:20:51<90:39:41,  8.73s/it][2025-04-25 17:18:35,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 17:18:35,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.09 | bwd_microstep: 5768.18 | bwd_inner_microstep: 5696.59 | bwd_allreduce_microstep: 71.54 | step_microstep: 18.93
[2025-04-25 17:18:35,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.09 | bwd: 5768.20 | bwd_inner: 5696.59 | bwd_allreduce: 71.56 | step: 18.93

  9%|▉         | 3877/41250 [9:21:00<90:33:25,  8.72s/it]
                                                         
{'loss': 0.2155, 'grad_norm': 1.823549747467041, 'learning_rate': 3.957219719485669e-05, 'epoch': 0.94}

  9%|▉         | 3877/41250 [9:21:00<90:33:25,  8.72s/it][2025-04-25 17:18:43,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:18:43,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.27 | bwd_microstep: 5755.65 | bwd_inner_microstep: 5692.55 | bwd_allreduce_microstep: 63.05 | step_microstep: 19.08
[2025-04-25 17:18:43,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.27 | bwd: 5755.66 | bwd_inner: 5692.55 | bwd_allreduce: 63.07 | step: 19.09

  9%|▉         | 3878/41250 [9:21:09<90:26:54,  8.71s/it]
                                                         
{'loss': 0.1585, 'grad_norm': 1.3591490983963013, 'learning_rate': 3.957187407949661e-05, 'epoch': 0.94}

  9%|▉         | 3878/41250 [9:21:09<90:26:54,  8.71s/it][2025-04-25 17:18:52,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 17:18:52,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5709.66 | bwd_inner_microstep: 5696.86 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.94
[2025-04-25 17:18:52,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5709.68 | bwd_inner: 5696.86 | bwd_allreduce: 12.77 | step: 18.95

  9%|▉         | 3879/41250 [9:21:17<90:15:10,  8.69s/it]
                                                         
{'loss': 0.1741, 'grad_norm': 1.2925662994384766, 'learning_rate': 3.957155084347979e-05, 'epoch': 0.94}

  9%|▉         | 3879/41250 [9:21:17<90:15:10,  8.69s/it][2025-04-25 17:19:01,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 1.72 | optimizer_step: 1.13
[2025-04-25 17:19:01,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.52 | bwd_microstep: 5710.08 | bwd_inner_microstep: 5665.03 | bwd_allreduce_microstep: 45.00 | step_microstep: 22.29
[2025-04-25 17:19:01,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.52 | bwd: 5710.10 | bwd_inner: 5665.03 | bwd_allreduce: 45.03 | step: 22.29

  9%|▉         | 3880/41250 [9:21:26<90:04:25,  8.68s/it]
                                                         
{'loss': 0.132, 'grad_norm': 1.1473206281661987, 'learning_rate': 3.95712274868082e-05, 'epoch': 0.94}

  9%|▉         | 3880/41250 [9:21:26<90:04:25,  8.68s/it][2025-04-25 17:19:09,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:19:09,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5723.96 | bwd_inner_microstep: 5711.06 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.91
[2025-04-25 17:19:09,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5723.97 | bwd_inner: 5711.06 | bwd_allreduce: 12.87 | step: 18.91

  9%|▉         | 3881/41250 [9:21:35<90:02:58,  8.68s/it]
                                                         
{'loss': 0.1717, 'grad_norm': 1.7078704833984375, 'learning_rate': 3.9570904009483844e-05, 'epoch': 0.94}

  9%|▉         | 3881/41250 [9:21:35<90:02:58,  8.68s/it][2025-04-25 17:19:18,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 17:19:18,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.52 | bwd_microstep: 5707.80 | bwd_inner_microstep: 5694.98 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.67
[2025-04-25 17:19:18,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.52 | bwd: 5707.81 | bwd_inner: 5694.98 | bwd_allreduce: 12.79 | step: 18.68

  9%|▉         | 3882/41250 [9:21:43<89:55:58,  8.66s/it]
                                                         
{'loss': 0.2589, 'grad_norm': 1.8324763774871826, 'learning_rate': 3.957058041150871e-05, 'epoch': 0.94}

  9%|▉         | 3882/41250 [9:21:43<89:55:58,  8.66s/it][2025-04-25 17:19:27,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 17:19:27,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5717.90 | bwd_inner_microstep: 5705.13 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.65
[2025-04-25 17:19:27,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5717.92 | bwd_inner: 5705.13 | bwd_allreduce: 12.75 | step: 18.65

  9%|▉         | 3883/41250 [9:21:52<89:53:56,  8.66s/it]
                                                         
{'loss': 0.0608, 'grad_norm': 0.6596835255622864, 'learning_rate': 3.9570256692884805e-05, 'epoch': 0.94}

  9%|▉         | 3883/41250 [9:21:52<89:53:56,  8.66s/it][2025-04-25 17:19:35,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 17:19:35,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.55 | bwd_microstep: 5760.90 | bwd_inner_microstep: 5718.64 | bwd_allreduce_microstep: 42.22 | step_microstep: 18.79
[2025-04-25 17:19:35,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.55 | bwd: 5760.92 | bwd_inner: 5718.64 | bwd_allreduce: 42.23 | step: 18.79

  9%|▉         | 3884/41250 [9:22:01<90:01:40,  8.67s/it]
                                                         
{'loss': 0.0438, 'grad_norm': 0.6735982894897461, 'learning_rate': 3.956993285361411e-05, 'epoch': 0.94}

  9%|▉         | 3884/41250 [9:22:01<90:01:40,  8.67s/it][2025-04-25 17:19:44,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.16
[2025-04-25 17:19:44,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.10 | bwd_microstep: 5723.04 | bwd_inner_microstep: 5710.34 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.74
[2025-04-25 17:19:44,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.10 | bwd: 5723.06 | bwd_inner: 5710.34 | bwd_allreduce: 12.67 | step: 19.74

  9%|▉         | 3885/41250 [9:22:09<90:01:44,  8.67s/it]
                                                         
{'loss': 0.0868, 'grad_norm': 1.144249439239502, 'learning_rate': 3.956960889369864e-05, 'epoch': 0.94}

  9%|▉         | 3885/41250 [9:22:09<90:01:44,  8.67s/it][2025-04-25 17:19:53,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 17:19:53,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.87 | bwd_microstep: 5739.15 | bwd_inner_microstep: 5726.26 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.08
[2025-04-25 17:19:53,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.87 | bwd: 5739.16 | bwd_inner: 5726.26 | bwd_allreduce: 12.86 | step: 19.08

  9%|▉         | 3886/41250 [9:22:18<90:03:37,  8.68s/it]
                                                         
{'loss': 0.3845, 'grad_norm': 2.7231388092041016, 'learning_rate': 3.9569284813140364e-05, 'epoch': 0.94}

  9%|▉         | 3886/41250 [9:22:18<90:03:37,  8.68s/it][2025-04-25 17:20:01,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:20:01,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.80 | bwd_microstep: 5746.28 | bwd_inner_microstep: 5707.19 | bwd_allreduce_microstep: 39.05 | step_microstep: 18.75
[2025-04-25 17:20:01,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.80 | bwd: 5746.30 | bwd_inner: 5707.19 | bwd_allreduce: 39.07 | step: 18.75

  9%|▉         | 3887/41250 [9:22:27<90:06:32,  8.68s/it]
                                                         
{'loss': 0.0878, 'grad_norm': 1.5512245893478394, 'learning_rate': 3.956896061194131e-05, 'epoch': 0.94}

  9%|▉         | 3887/41250 [9:22:27<90:06:32,  8.68s/it][2025-04-25 17:20:10,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-25 17:20:10,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.54 | bwd_microstep: 5703.57 | bwd_inner_microstep: 5690.20 | bwd_allreduce_microstep: 13.32 | step_microstep: 19.22
[2025-04-25 17:20:10,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.54 | bwd: 5703.59 | bwd_inner: 5690.20 | bwd_allreduce: 13.34 | step: 19.23

  9%|▉         | 3888/41250 [9:22:35<89:59:10,  8.67s/it]
                                                         
{'loss': 0.0448, 'grad_norm': 0.5784025192260742, 'learning_rate': 3.9568636290103456e-05, 'epoch': 0.94}

  9%|▉         | 3888/41250 [9:22:35<89:59:10,  8.67s/it][2025-04-25 17:20:19,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.09 | optimizer_step: 0.90
[2025-04-25 17:20:19,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.28 | bwd_microstep: 6001.31 | bwd_inner_microstep: 5715.35 | bwd_allreduce_microstep: 285.86 | step_microstep: 19.14
[2025-04-25 17:20:19,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.28 | bwd: 6001.33 | bwd_inner: 5715.35 | bwd_allreduce: 285.92 | step: 19.13

  9%|▉         | 3889/41250 [9:22:44<90:52:52,  8.76s/it]
                                                         
{'loss': 0.1797, 'grad_norm': 2.5356221199035645, 'learning_rate': 3.956831184762881e-05, 'epoch': 0.94}

  9%|▉         | 3889/41250 [9:22:44<90:52:52,  8.76s/it][2025-04-25 17:20:28,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.06 | optimizer_step: 0.97
[2025-04-25 17:20:28,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.58 | bwd_microstep: 5763.02 | bwd_inner_microstep: 5701.89 | bwd_allreduce_microstep: 61.08 | step_microstep: 19.13
[2025-04-25 17:20:28,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.58 | bwd: 5763.03 | bwd_inner: 5701.89 | bwd_allreduce: 61.10 | step: 19.13

  9%|▉         | 3890/41250 [9:22:53<90:42:38,  8.74s/it]
                                                         
{'loss': 0.1514, 'grad_norm': 1.1170552968978882, 'learning_rate': 3.9567987284519366e-05, 'epoch': 0.94}

  9%|▉         | 3890/41250 [9:22:53<90:42:38,  8.74s/it][2025-04-25 17:20:36,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.15 | optimizer_step: 0.91
[2025-04-25 17:20:36,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.52 | bwd_microstep: 5744.83 | bwd_inner_microstep: 5708.78 | bwd_allreduce_microstep: 36.00 | step_microstep: 18.94
[2025-04-25 17:20:36,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.52 | bwd: 5744.84 | bwd_inner: 5708.78 | bwd_allreduce: 36.02 | step: 18.95

  9%|▉         | 3891/41250 [9:23:02<90:32:47,  8.73s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 2.1245181560516357, 'learning_rate': 3.956766260077713e-05, 'epoch': 0.94}

  9%|▉         | 3891/41250 [9:23:02<90:32:47,  8.73s/it][2025-04-25 17:20:45,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.44 | optimizer_step: 0.94
[2025-04-25 17:20:45,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.18 | bwd_microstep: 5731.36 | bwd_inner_microstep: 5717.64 | bwd_allreduce_microstep: 13.66 | step_microstep: 19.77
[2025-04-25 17:20:45,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.18 | bwd: 5731.37 | bwd_inner: 5717.64 | bwd_allreduce: 13.69 | step: 19.78

  9%|▉         | 3892/41250 [9:23:10<90:22:29,  8.71s/it]
                                                         
{'loss': 0.1185, 'grad_norm': 0.8448246121406555, 'learning_rate': 3.956733779640411e-05, 'epoch': 0.94}

  9%|▉         | 3892/41250 [9:23:10<90:22:29,  8.71s/it][2025-04-25 17:20:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:20:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.13 | bwd_microstep: 5729.16 | bwd_inner_microstep: 5705.91 | bwd_allreduce_microstep: 23.21 | step_microstep: 18.77
[2025-04-25 17:20:54,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.13 | bwd: 5729.17 | bwd_inner: 5705.91 | bwd_allreduce: 23.22 | step: 18.77

  9%|▉         | 3893/41250 [9:23:19<90:16:04,  8.70s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.763019323348999, 'learning_rate': 3.95670128714023e-05, 'epoch': 0.94}

  9%|▉         | 3893/41250 [9:23:19<90:16:04,  8.70s/it][2025-04-25 17:21:02,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:21:02,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.05 | bwd_microstep: 5694.74 | bwd_inner_microstep: 5662.51 | bwd_allreduce_microstep: 32.18 | step_microstep: 18.87
[2025-04-25 17:21:02,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.05 | bwd: 5694.75 | bwd_inner: 5662.51 | bwd_allreduce: 32.20 | step: 18.88

  9%|▉         | 3894/41250 [9:23:28<89:58:37,  8.67s/it]
                                                         
{'loss': 0.1333, 'grad_norm': 1.073056936264038, 'learning_rate': 3.95666878257737e-05, 'epoch': 0.94}

  9%|▉         | 3894/41250 [9:23:28<89:58:37,  8.67s/it][2025-04-25 17:21:11,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 17:21:11,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.44 | bwd_microstep: 5842.68 | bwd_inner_microstep: 5667.12 | bwd_allreduce_microstep: 175.52 | step_microstep: 18.35
[2025-04-25 17:21:11,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.44 | bwd: 5842.70 | bwd_inner: 5667.12 | bwd_allreduce: 175.54 | step: 18.35

  9%|▉         | 3895/41250 [9:23:36<90:15:01,  8.70s/it]
                                                         
{'loss': 0.2436, 'grad_norm': 1.2789138555526733, 'learning_rate': 3.9566362659520314e-05, 'epoch': 0.94}

  9%|▉         | 3895/41250 [9:23:37<90:15:01,  8.70s/it][2025-04-25 17:21:20,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:21:20,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.25 | bwd_microstep: 5723.04 | bwd_inner_microstep: 5699.46 | bwd_allreduce_microstep: 23.54 | step_microstep: 18.89
[2025-04-25 17:21:20,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.25 | bwd: 5723.05 | bwd_inner: 5699.46 | bwd_allreduce: 23.55 | step: 18.89

  9%|▉         | 3896/41250 [9:23:45<90:07:12,  8.69s/it]
                                                         
{'loss': 0.1683, 'grad_norm': 1.8649922609329224, 'learning_rate': 3.956603737264416e-05, 'epoch': 0.94}

  9%|▉         | 3896/41250 [9:23:45<90:07:12,  8.69s/it][2025-04-25 17:21:29,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.48 | optimizer_step: 0.94
[2025-04-25 17:21:29,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5772.08 | bwd_inner_microstep: 5698.50 | bwd_allreduce_microstep: 73.54 | step_microstep: 19.92
[2025-04-25 17:21:29,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5772.10 | bwd_inner: 5698.50 | bwd_allreduce: 73.56 | step: 19.92

  9%|▉         | 3897/41250 [9:23:54<90:11:32,  8.69s/it]
                                                         
{'loss': 0.1493, 'grad_norm': 1.1868648529052734, 'learning_rate': 3.956571196514722e-05, 'epoch': 0.94}

  9%|▉         | 3897/41250 [9:23:54<90:11:32,  8.69s/it][2025-04-25 17:21:37,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 17:21:37,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.25 | bwd_microstep: 5737.04 | bwd_inner_microstep: 5702.01 | bwd_allreduce_microstep: 34.98 | step_microstep: 19.10
[2025-04-25 17:21:37,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.25 | bwd: 5737.05 | bwd_inner: 5702.01 | bwd_allreduce: 35.00 | step: 19.10

  9%|▉         | 3898/41250 [9:24:03<90:07:28,  8.69s/it]
                                                         
{'loss': 0.0771, 'grad_norm': 1.0943653583526611, 'learning_rate': 3.956538643703153e-05, 'epoch': 0.94}

  9%|▉         | 3898/41250 [9:24:03<90:07:28,  8.69s/it][2025-04-25 17:21:46,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:21:46,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.64 | bwd_microstep: 5732.44 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 26.19 | step_microstep: 19.06
[2025-04-25 17:21:46,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.64 | bwd: 5732.45 | bwd_inner: 5706.20 | bwd_allreduce: 26.21 | step: 19.07

  9%|▉         | 3899/41250 [9:24:11<90:03:22,  8.68s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 2.363495111465454, 'learning_rate': 3.9565060788299075e-05, 'epoch': 0.95}

  9%|▉         | 3899/41250 [9:24:11<90:03:22,  8.68s/it][2025-04-25 17:21:55,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:21:55,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5704.47 | bwd_inner_microstep: 5691.03 | bwd_allreduce_microstep: 13.39 | step_microstep: 18.58
[2025-04-25 17:21:55,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5704.48 | bwd_inner: 5691.03 | bwd_allreduce: 13.41 | step: 18.58

  9%|▉         | 3900/41250 [9:24:20<89:55:02,  8.67s/it]
                                                         
{'loss': 0.1608, 'grad_norm': 0.8384041786193848, 'learning_rate': 3.956473501895187e-05, 'epoch': 0.95}

  9%|▉         | 3900/41250 [9:24:20<89:55:02,  8.67s/it][2025-04-25 17:22:03,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:22:03,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.26 | bwd_microstep: 5683.56 | bwd_inner_microstep: 5670.97 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.87
[2025-04-25 17:22:03,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.26 | bwd: 5683.57 | bwd_inner: 5670.97 | bwd_allreduce: 12.57 | step: 18.87

  9%|▉         | 3901/41250 [9:24:28<89:45:52,  8.65s/it]
                                                         
{'loss': 0.2174, 'grad_norm': 1.0942749977111816, 'learning_rate': 3.956440912899192e-05, 'epoch': 0.95}

  9%|▉         | 3901/41250 [9:24:28<89:45:52,  8.65s/it][2025-04-25 17:22:12,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:22:12,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.77 | bwd_microstep: 5710.12 | bwd_inner_microstep: 5696.29 | bwd_allreduce_microstep: 13.78 | step_microstep: 18.93
[2025-04-25 17:22:12,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.77 | bwd: 5710.13 | bwd_inner: 5696.29 | bwd_allreduce: 13.80 | step: 18.93

  9%|▉         | 3902/41250 [9:24:37<89:44:57,  8.65s/it]
                                                         
{'loss': 0.0286, 'grad_norm': 0.297259658575058, 'learning_rate': 3.956408311842124e-05, 'epoch': 0.95}

  9%|▉         | 3902/41250 [9:24:37<89:44:57,  8.65s/it][2025-04-25 17:22:20,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:22:20,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.71 | bwd_microstep: 5698.49 | bwd_inner_microstep: 5685.67 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.68
[2025-04-25 17:22:20,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.71 | bwd: 5698.51 | bwd_inner: 5685.67 | bwd_allreduce: 12.79 | step: 18.68

  9%|▉         | 3903/41250 [9:24:46<89:40:20,  8.64s/it]
                                                         
{'loss': 0.1082, 'grad_norm': 2.0103700160980225, 'learning_rate': 3.9563756987241825e-05, 'epoch': 0.95}

  9%|▉         | 3903/41250 [9:24:46<89:40:20,  8.64s/it][2025-04-25 17:22:29,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 17:22:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.53 | bwd_microstep: 5814.95 | bwd_inner_microstep: 5772.63 | bwd_allreduce_microstep: 42.27 | step_microstep: 19.22
[2025-04-25 17:22:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.53 | bwd: 5814.97 | bwd_inner: 5772.63 | bwd_allreduce: 42.29 | step: 19.23

  9%|▉         | 3904/41250 [9:24:55<90:06:46,  8.69s/it]
                                                         
{'loss': 0.0386, 'grad_norm': 0.438026487827301, 'learning_rate': 3.9563430735455706e-05, 'epoch': 0.95}

  9%|▉         | 3904/41250 [9:24:55<90:06:46,  8.69s/it][2025-04-25 17:22:38,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.17 | optimizer_step: 1.00
[2025-04-25 17:22:38,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.67 | bwd_microstep: 5737.82 | bwd_inner_microstep: 5677.49 | bwd_allreduce_microstep: 60.28 | step_microstep: 19.58
[2025-04-25 17:22:38,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.67 | bwd: 5737.83 | bwd_inner: 5677.49 | bwd_allreduce: 60.30 | step: 19.59

  9%|▉         | 3905/41250 [9:25:03<90:03:14,  8.68s/it]
                                                         
{'loss': 0.0575, 'grad_norm': 0.6270372867584229, 'learning_rate': 3.9563104363064876e-05, 'epoch': 0.95}

  9%|▉         | 3905/41250 [9:25:03<90:03:14,  8.68s/it][2025-04-25 17:22:47,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 17:22:47,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.27 | bwd_microstep: 5777.61 | bwd_inner_microstep: 5663.44 | bwd_allreduce_microstep: 114.12 | step_microstep: 18.81
[2025-04-25 17:22:47,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.27 | bwd: 5777.62 | bwd_inner: 5663.44 | bwd_allreduce: 114.14 | step: 18.81

  9%|▉         | 3906/41250 [9:25:12<90:04:05,  8.68s/it]
                                                         
{'loss': 0.0181, 'grad_norm': 0.15588946640491486, 'learning_rate': 3.956277787007136e-05, 'epoch': 0.95}

  9%|▉         | 3906/41250 [9:25:12<90:04:05,  8.68s/it][2025-04-25 17:22:55,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:22:55,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.88 | bwd_microstep: 5889.18 | bwd_inner_microstep: 5654.38 | bwd_allreduce_microstep: 234.75 | step_microstep: 18.48
[2025-04-25 17:22:55,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.88 | bwd: 5889.20 | bwd_inner: 5654.38 | bwd_allreduce: 234.77 | step: 18.48

  9%|▉         | 3907/41250 [9:25:21<90:26:47,  8.72s/it]
                                                         
{'loss': 0.1543, 'grad_norm': 1.0188437700271606, 'learning_rate': 3.9562451256477164e-05, 'epoch': 0.95}

  9%|▉         | 3907/41250 [9:25:21<90:26:47,  8.72s/it][2025-04-25 17:23:04,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 17:23:04,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5692.84 | bwd_inner_microstep: 5676.10 | bwd_allreduce_microstep: 16.69 | step_microstep: 18.79
[2025-04-25 17:23:04,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5692.85 | bwd_inner: 5676.10 | bwd_allreduce: 16.71 | step: 18.79

  9%|▉         | 3908/41250 [9:25:29<90:08:26,  8.69s/it]
                                                         
{'loss': 0.1957, 'grad_norm': 2.4294016361236572, 'learning_rate': 3.956212452228431e-05, 'epoch': 0.95}

  9%|▉         | 3908/41250 [9:25:29<90:08:26,  8.69s/it][2025-04-25 17:23:13,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:23:13,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.30 | bwd_microstep: 5779.11 | bwd_inner_microstep: 5652.01 | bwd_allreduce_microstep: 127.05 | step_microstep: 18.98
[2025-04-25 17:23:13,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.30 | bwd: 5779.13 | bwd_inner: 5652.01 | bwd_allreduce: 127.07 | step: 18.98

  9%|▉         | 3909/41250 [9:25:38<90:07:58,  8.69s/it]
                                                         
{'loss': 0.0937, 'grad_norm': 1.6659462451934814, 'learning_rate': 3.95617976674948e-05, 'epoch': 0.95}

  9%|▉         | 3909/41250 [9:25:38<90:07:58,  8.69s/it][2025-04-25 17:23:21,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:23:21,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.32 | bwd_microstep: 5693.41 | bwd_inner_microstep: 5680.54 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.00
[2025-04-25 17:23:21,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.32 | bwd: 5693.42 | bwd_inner: 5680.54 | bwd_allreduce: 12.84 | step: 19.00

  9%|▉         | 3910/41250 [9:25:47<89:54:05,  8.67s/it]
                                                         
{'loss': 0.2243, 'grad_norm': 1.5669728517532349, 'learning_rate': 3.9561470692110664e-05, 'epoch': 0.95}

  9%|▉         | 3910/41250 [9:25:47<89:54:05,  8.67s/it][2025-04-25 17:23:30,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:23:30,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.55 | bwd_microstep: 5710.79 | bwd_inner_microstep: 5697.87 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.89
[2025-04-25 17:23:30,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.55 | bwd: 5710.81 | bwd_inner: 5697.87 | bwd_allreduce: 12.90 | step: 18.89

  9%|▉         | 3911/41250 [9:25:55<89:50:57,  8.66s/it]
                                                         
{'loss': 0.0607, 'grad_norm': 0.8097484111785889, 'learning_rate': 3.95611435961339e-05, 'epoch': 0.95}

  9%|▉         | 3911/41250 [9:25:55<89:50:57,  8.66s/it][2025-04-25 17:23:39,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 17:23:39,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.63 | bwd_microstep: 5685.05 | bwd_inner_microstep: 5652.36 | bwd_allreduce_microstep: 32.64 | step_microstep: 19.51
[2025-04-25 17:23:39,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.63 | bwd: 5685.07 | bwd_inner: 5652.36 | bwd_allreduce: 32.66 | step: 19.51

  9%|▉         | 3912/41250 [9:26:04<89:39:35,  8.64s/it]
                                                         
{'loss': 0.0472, 'grad_norm': 0.6783496141433716, 'learning_rate': 3.9560816379566536e-05, 'epoch': 0.95}

  9%|▉         | 3912/41250 [9:26:04<89:39:35,  8.64s/it][2025-04-25 17:23:47,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 17:23:47,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.06 | bwd_microstep: 5795.85 | bwd_inner_microstep: 5783.00 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.16
[2025-04-25 17:23:47,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.06 | bwd: 5795.87 | bwd_inner: 5783.00 | bwd_allreduce: 12.83 | step: 19.16

  9%|▉         | 3913/41250 [9:26:13<90:02:21,  8.68s/it]
                                                         
{'loss': 0.2337, 'grad_norm': 1.96892511844635, 'learning_rate': 3.9560489042410586e-05, 'epoch': 0.95}

  9%|▉         | 3913/41250 [9:26:13<90:02:21,  8.68s/it][2025-04-25 17:23:56,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 17:23:56,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.44 | bwd_microstep: 5729.44 | bwd_inner_microstep: 5690.56 | bwd_allreduce_microstep: 38.83 | step_microstep: 18.88
[2025-04-25 17:23:56,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.44 | bwd: 5729.45 | bwd_inner: 5690.56 | bwd_allreduce: 38.85 | step: 18.88

  9%|▉         | 3914/41250 [9:26:21<89:57:56,  8.67s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 2.0707287788391113, 'learning_rate': 3.9560161584668064e-05, 'epoch': 0.95}

  9%|▉         | 3914/41250 [9:26:21<89:57:56,  8.67s/it][2025-04-25 17:24:05,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.14 | optimizer_step: 0.90
[2025-04-25 17:24:05,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.18 | bwd_microstep: 5716.45 | bwd_inner_microstep: 5703.74 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.37
[2025-04-25 17:24:05,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.18 | bwd: 5716.47 | bwd_inner: 5703.74 | bwd_allreduce: 12.69 | step: 19.38

  9%|▉         | 3915/41250 [9:26:30<89:53:11,  8.67s/it]
                                                         
{'loss': 0.1808, 'grad_norm': 1.5407218933105469, 'learning_rate': 3.9559834006341e-05, 'epoch': 0.95}

  9%|▉         | 3915/41250 [9:26:30<89:53:11,  8.67s/it][2025-04-25 17:24:13,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:24:13,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.45 | bwd_microstep: 5743.49 | bwd_inner_microstep: 5701.87 | bwd_allreduce_microstep: 41.57 | step_microstep: 18.71
[2025-04-25 17:24:13,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.45 | bwd: 5743.50 | bwd_inner: 5701.87 | bwd_allreduce: 41.59 | step: 18.71

  9%|▉         | 3916/41250 [9:26:39<89:53:49,  8.67s/it]
                                                         
{'loss': 0.1114, 'grad_norm': 1.3097920417785645, 'learning_rate': 3.9559506307431404e-05, 'epoch': 0.95}

  9%|▉         | 3916/41250 [9:26:39<89:53:49,  8.67s/it][2025-04-25 17:24:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:24:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5753.30 | bwd_inner_microstep: 5707.20 | bwd_allreduce_microstep: 46.05 | step_microstep: 18.60
[2025-04-25 17:24:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5753.31 | bwd_inner: 5707.20 | bwd_allreduce: 46.07 | step: 18.60

  9%|▉         | 3917/41250 [9:26:47<89:56:44,  8.67s/it]
                                                         
{'loss': 0.2313, 'grad_norm': 2.972632884979248, 'learning_rate': 3.95591784879413e-05, 'epoch': 0.95}

  9%|▉         | 3917/41250 [9:26:47<89:56:44,  8.67s/it][2025-04-25 17:24:31,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 17:24:31,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5720.78 | bwd_inner_microstep: 5707.77 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.59
[2025-04-25 17:24:31,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5720.80 | bwd_inner: 5707.77 | bwd_allreduce: 12.99 | step: 19.59

  9%|▉         | 3918/41250 [9:26:56<89:54:01,  8.67s/it]
                                                         
{'loss': 0.4827, 'grad_norm': 2.5422322750091553, 'learning_rate': 3.95588505478727e-05, 'epoch': 0.95}

  9%|▉         | 3918/41250 [9:26:56<89:54:01,  8.67s/it][2025-04-25 17:24:39,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 17:24:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.89 | bwd_microstep: 5783.88 | bwd_inner_microstep: 5661.76 | bwd_allreduce_microstep: 122.07 | step_microstep: 18.95
[2025-04-25 17:24:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.89 | bwd: 5783.89 | bwd_inner: 5661.76 | bwd_allreduce: 122.09 | step: 18.95

 10%|▉         | 3919/41250 [9:27:05<89:59:29,  8.68s/it]
                                                         
{'loss': 0.2821, 'grad_norm': 1.700358510017395, 'learning_rate': 3.955852248722764e-05, 'epoch': 0.95}

 10%|▉         | 3919/41250 [9:27:05<89:59:29,  8.68s/it][2025-04-25 17:24:48,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:24:48,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5743.86 | bwd_inner_microstep: 5692.89 | bwd_allreduce_microstep: 50.93 | step_microstep: 18.95
[2025-04-25 17:24:48,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5743.88 | bwd_inner: 5692.89 | bwd_allreduce: 50.95 | step: 18.95

 10%|▉         | 3920/41250 [9:27:13<89:58:25,  8.68s/it]
                                                         
{'loss': 0.0915, 'grad_norm': 1.7935945987701416, 'learning_rate': 3.955819430600814e-05, 'epoch': 0.95}

 10%|▉         | 3920/41250 [9:27:13<89:58:25,  8.68s/it][2025-04-25 17:24:57,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 17:24:57,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2903.58 | bwd_microstep: 5785.82 | bwd_inner_microstep: 5772.71 | bwd_allreduce_microstep: 13.06 | step_microstep: 19.19
[2025-04-25 17:24:57,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2903.58 | bwd: 5785.83 | bwd_inner: 5772.71 | bwd_allreduce: 13.08 | step: 19.19

 10%|▉         | 3921/41250 [9:27:22<90:16:27,  8.71s/it]
                                                         
{'loss': 0.0989, 'grad_norm': 1.2093960046768188, 'learning_rate': 3.955786600421621e-05, 'epoch': 0.95}

 10%|▉         | 3921/41250 [9:27:22<90:16:27,  8.71s/it][2025-04-25 17:25:05,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:25:05,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.96 | bwd_microstep: 5739.80 | bwd_inner_microstep: 5703.09 | bwd_allreduce_microstep: 36.67 | step_microstep: 18.78
[2025-04-25 17:25:05,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.96 | bwd: 5739.82 | bwd_inner: 5703.09 | bwd_allreduce: 36.68 | step: 18.78

 10%|▉         | 3922/41250 [9:27:31<90:11:11,  8.70s/it]
                                                         
{'loss': 0.1417, 'grad_norm': 1.8892381191253662, 'learning_rate': 3.955753758185389e-05, 'epoch': 0.95}

 10%|▉         | 3922/41250 [9:27:31<90:11:11,  8.70s/it][2025-04-25 17:25:14,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.03 | optimizer_step: 1.16
[2025-04-25 17:25:14,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.33 | bwd_microstep: 5744.70 | bwd_inner_microstep: 5709.31 | bwd_allreduce_microstep: 35.34 | step_microstep: 19.26
[2025-04-25 17:25:14,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.33 | bwd: 5744.72 | bwd_inner: 5709.31 | bwd_allreduce: 35.36 | step: 19.26

 10%|▉         | 3923/41250 [9:27:39<90:09:16,  8.69s/it]
                                                         
{'loss': 0.1746, 'grad_norm': 1.4778883457183838, 'learning_rate': 3.9557209038923196e-05, 'epoch': 0.95}

 10%|▉         | 3923/41250 [9:27:39<90:09:16,  8.69s/it][2025-04-25 17:25:23,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 17:25:23,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5744.61 | bwd_inner_microstep: 5709.55 | bwd_allreduce_microstep: 35.01 | step_microstep: 18.98
[2025-04-25 17:25:23,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5744.63 | bwd_inner: 5709.55 | bwd_allreduce: 35.04 | step: 18.98

 10%|▉         | 3924/41250 [9:27:48<90:06:11,  8.69s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.722179889678955, 'learning_rate': 3.955688037542615e-05, 'epoch': 0.95}

 10%|▉         | 3924/41250 [9:27:48<90:06:11,  8.69s/it][2025-04-25 17:25:31,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:25:31,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.10 | bwd_microstep: 5727.10 | bwd_inner_microstep: 5714.33 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.62
[2025-04-25 17:25:31,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.10 | bwd: 5727.11 | bwd_inner: 5714.33 | bwd_allreduce: 12.74 | step: 18.62

 10%|▉         | 3925/41250 [9:27:57<90:02:25,  8.68s/it]
                                                         
{'loss': 0.1321, 'grad_norm': 1.5305780172348022, 'learning_rate': 3.9556551591364786e-05, 'epoch': 0.95}

 10%|▉         | 3925/41250 [9:27:57<90:02:25,  8.68s/it][2025-04-25 17:25:40,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.41 | optimizer_step: 1.00
[2025-04-25 17:25:40,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.86 | bwd_microstep: 5702.64 | bwd_inner_microstep: 5661.87 | bwd_allreduce_microstep: 40.72 | step_microstep: 21.11
[2025-04-25 17:25:40,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.86 | bwd: 5702.66 | bwd_inner: 5661.86 | bwd_allreduce: 40.75 | step: 21.11

 10%|▉         | 3926/41250 [9:28:05<89:52:06,  8.67s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 2.48421573638916, 'learning_rate': 3.9556222686741136e-05, 'epoch': 0.95}

 10%|▉         | 3926/41250 [9:28:05<89:52:06,  8.67s/it][2025-04-25 17:25:49,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 17:25:49,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.02 | bwd_microstep: 5732.25 | bwd_inner_microstep: 5709.63 | bwd_allreduce_microstep: 22.57 | step_microstep: 18.62
[2025-04-25 17:25:49,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.02 | bwd: 5732.26 | bwd_inner: 5709.63 | bwd_allreduce: 22.59 | step: 18.63

 10%|▉         | 3927/41250 [9:28:14<89:55:47,  8.67s/it]
                                                         
{'loss': 0.1659, 'grad_norm': 1.9491134881973267, 'learning_rate': 3.955589366155721e-05, 'epoch': 0.95}

 10%|▉         | 3927/41250 [9:28:14<89:55:47,  8.67s/it][2025-04-25 17:25:58,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.26 | optimizer_step: 0.89
[2025-04-25 17:25:58,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.66 | bwd_microstep: 5818.03 | bwd_inner_microstep: 5653.45 | bwd_allreduce_microstep: 164.52 | step_microstep: 19.71
[2025-04-25 17:25:58,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.66 | bwd: 5818.05 | bwd_inner: 5653.45 | bwd_allreduce: 164.55 | step: 19.72

 10%|▉         | 3928/41250 [9:28:23<90:06:15,  8.69s/it]
                                                         
{'loss': 0.0849, 'grad_norm': 0.8665704727172852, 'learning_rate': 3.9555564515815044e-05, 'epoch': 0.95}

 10%|▉         | 3928/41250 [9:28:23<90:06:15,  8.69s/it][2025-04-25 17:26:06,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 17:26:06,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.25 | bwd_microstep: 5704.67 | bwd_inner_microstep: 5691.77 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.90
[2025-04-25 17:26:06,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.25 | bwd: 5704.69 | bwd_inner: 5691.77 | bwd_allreduce: 12.87 | step: 18.90

 10%|▉         | 3929/41250 [9:28:32<89:57:38,  8.68s/it]
                                                         
{'loss': 0.1541, 'grad_norm': 1.4518120288848877, 'learning_rate': 3.9555235249516675e-05, 'epoch': 0.95}

 10%|▉         | 3929/41250 [9:28:32<89:57:38,  8.68s/it][2025-04-25 17:26:15,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.17 | optimizer_step: 1.06
[2025-04-25 17:26:15,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.95 | bwd_microstep: 5689.24 | bwd_inner_microstep: 5675.43 | bwd_allreduce_microstep: 13.75 | step_microstep: 20.48
[2025-04-25 17:26:15,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.95 | bwd: 5689.25 | bwd_inner: 5675.43 | bwd_allreduce: 13.78 | step: 20.48

 10%|▉         | 3930/41250 [9:28:40<89:47:16,  8.66s/it]
                                                         
{'loss': 0.0821, 'grad_norm': 0.9459685683250427, 'learning_rate': 3.955490586266413e-05, 'epoch': 0.95}

 10%|▉         | 3930/41250 [9:28:40<89:47:16,  8.66s/it][2025-04-25 17:26:23,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 17:26:23,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.70 | bwd_microstep: 5705.02 | bwd_inner_microstep: 5655.25 | bwd_allreduce_microstep: 49.72 | step_microstep: 19.05
[2025-04-25 17:26:23,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.70 | bwd: 5705.03 | bwd_inner: 5655.25 | bwd_allreduce: 49.75 | step: 19.05

 10%|▉         | 3931/41250 [9:28:49<89:38:44,  8.65s/it]
                                                         
{'loss': 0.2106, 'grad_norm': 4.827730178833008, 'learning_rate': 3.955457635525943e-05, 'epoch': 0.95}

 10%|▉         | 3931/41250 [9:28:49<89:38:44,  8.65s/it][2025-04-25 17:26:32,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 17:26:32,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.34 | bwd_microstep: 5843.40 | bwd_inner_microstep: 5708.83 | bwd_allreduce_microstep: 134.52 | step_microstep: 19.11
[2025-04-25 17:26:32,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.34 | bwd: 5843.42 | bwd_inner: 5708.83 | bwd_allreduce: 134.54 | step: 19.12

 10%|▉         | 3932/41250 [9:28:58<90:06:02,  8.69s/it]
                                                         
{'loss': 0.0682, 'grad_norm': 0.7925592660903931, 'learning_rate': 3.955424672730462e-05, 'epoch': 0.95}

 10%|▉         | 3932/41250 [9:28:58<90:06:02,  8.69s/it][2025-04-25 17:26:41,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:26:41,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.39 | bwd_microstep: 5691.07 | bwd_inner_microstep: 5671.31 | bwd_allreduce_microstep: 19.72 | step_microstep: 19.22
[2025-04-25 17:26:41,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.39 | bwd: 5691.09 | bwd_inner: 5671.31 | bwd_allreduce: 19.74 | step: 19.22

 10%|▉         | 3933/41250 [9:29:06<89:51:42,  8.67s/it]
                                                         
{'loss': 0.3962, 'grad_norm': 3.0368545055389404, 'learning_rate': 3.9553916978801714e-05, 'epoch': 0.95}

 10%|▉         | 3933/41250 [9:29:06<89:51:42,  8.67s/it][2025-04-25 17:26:50,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:26:50,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.16 | bwd_microstep: 5780.93 | bwd_inner_microstep: 5704.86 | bwd_allreduce_microstep: 76.03 | step_microstep: 18.59
[2025-04-25 17:26:50,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.16 | bwd: 5780.95 | bwd_inner: 5704.86 | bwd_allreduce: 76.04 | step: 18.59

 10%|▉         | 3934/41250 [9:29:15<90:00:37,  8.68s/it]
                                                         
{'loss': 0.2198, 'grad_norm': 5.1835103034973145, 'learning_rate': 3.955358710975277e-05, 'epoch': 0.95}

 10%|▉         | 3934/41250 [9:29:15<90:00:37,  8.68s/it][2025-04-25 17:26:58,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:26:58,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.04 | bwd_microstep: 5766.78 | bwd_inner_microstep: 5710.07 | bwd_allreduce_microstep: 56.66 | step_microstep: 18.86
[2025-04-25 17:26:58,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.04 | bwd: 5766.79 | bwd_inner: 5710.07 | bwd_allreduce: 56.68 | step: 18.86

 10%|▉         | 3935/41250 [9:29:24<90:05:39,  8.69s/it]
                                                         
{'loss': 0.1254, 'grad_norm': 1.5132652521133423, 'learning_rate': 3.955325712015979e-05, 'epoch': 0.95}

 10%|▉         | 3935/41250 [9:29:24<90:05:39,  8.69s/it][2025-04-25 17:27:07,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 17:27:07,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5710.16 | bwd_inner_microstep: 5697.18 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.36
[2025-04-25 17:27:07,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5710.17 | bwd_inner: 5697.18 | bwd_allreduce: 12.95 | step: 19.37

 10%|▉         | 3936/41250 [9:29:32<89:56:40,  8.68s/it]
                                                         
{'loss': 0.1474, 'grad_norm': 1.351916790008545, 'learning_rate': 3.955292701002484e-05, 'epoch': 0.95}

 10%|▉         | 3936/41250 [9:29:32<89:56:40,  8.68s/it][2025-04-25 17:27:16,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 17:27:16,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.32 | bwd_microstep: 5763.79 | bwd_inner_microstep: 5663.24 | bwd_allreduce_microstep: 100.50 | step_microstep: 18.73
[2025-04-25 17:27:16,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.32 | bwd: 5763.81 | bwd_inner: 5663.24 | bwd_allreduce: 100.52 | step: 18.73

 10%|▉         | 3937/41250 [9:29:41<89:57:20,  8.68s/it]
                                                         
{'loss': 0.0989, 'grad_norm': 1.3257403373718262, 'learning_rate': 3.955259677934994e-05, 'epoch': 0.95}

 10%|▉         | 3937/41250 [9:29:41<89:57:20,  8.68s/it][2025-04-25 17:27:24,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:27:24,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.98 | bwd_microstep: 5800.13 | bwd_inner_microstep: 5719.99 | bwd_allreduce_microstep: 80.10 | step_microstep: 18.60
[2025-04-25 17:27:24,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.98 | bwd: 5800.14 | bwd_inner: 5719.98 | bwd_allreduce: 80.11 | step: 18.60

 10%|▉         | 3938/41250 [9:29:50<90:07:59,  8.70s/it]
                                                         
{'loss': 0.2, 'grad_norm': 1.499864101409912, 'learning_rate': 3.955226642813712e-05, 'epoch': 0.95}

 10%|▉         | 3938/41250 [9:29:50<90:07:59,  8.70s/it][2025-04-25 17:27:33,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:27:33,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.34 | bwd_microstep: 5801.26 | bwd_inner_microstep: 5788.47 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.48
[2025-04-25 17:27:33,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.34 | bwd: 5801.28 | bwd_inner: 5788.47 | bwd_allreduce: 12.76 | step: 18.49

 10%|▉         | 3939/41250 [9:29:58<90:23:34,  8.72s/it]
                                                         
{'loss': 0.4567, 'grad_norm': 2.6003472805023193, 'learning_rate': 3.955193595638842e-05, 'epoch': 0.95}

 10%|▉         | 3939/41250 [9:29:58<90:23:34,  8.72s/it][2025-04-25 17:27:42,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:27:42,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.60 | bwd_microstep: 5768.78 | bwd_inner_microstep: 5723.68 | bwd_allreduce_microstep: 45.05 | step_microstep: 18.87
[2025-04-25 17:27:42,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.60 | bwd: 5768.79 | bwd_inner: 5723.68 | bwd_allreduce: 45.07 | step: 18.87

 10%|▉         | 3940/41250 [9:30:07<90:22:01,  8.72s/it]
                                                         
{'loss': 0.2102, 'grad_norm': 1.123397946357727, 'learning_rate': 3.955160536410589e-05, 'epoch': 0.96}

 10%|▉         | 3940/41250 [9:30:07<90:22:01,  8.72s/it][2025-04-25 17:27:50,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:27:50,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.21 | bwd_microstep: 5726.39 | bwd_inner_microstep: 5706.81 | bwd_allreduce_microstep: 19.53 | step_microstep: 18.42
[2025-04-25 17:27:50,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.21 | bwd: 5726.40 | bwd_inner: 5706.81 | bwd_allreduce: 19.55 | step: 18.42

 10%|▉         | 3941/41250 [9:30:16<90:11:04,  8.70s/it]
                                                         
{'loss': 0.0305, 'grad_norm': 0.3659295439720154, 'learning_rate': 3.955127465129156e-05, 'epoch': 0.96}

 10%|▉         | 3941/41250 [9:30:16<90:11:04,  8.70s/it][2025-04-25 17:27:59,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 17:27:59,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.50 | bwd_microstep: 5758.62 | bwd_inner_microstep: 5697.89 | bwd_allreduce_microstep: 60.69 | step_microstep: 19.26
[2025-04-25 17:27:59,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.50 | bwd: 5758.63 | bwd_inner: 5697.89 | bwd_allreduce: 60.71 | step: 19.26

 10%|▉         | 3942/41250 [9:30:24<90:08:47,  8.70s/it]
                                                         
{'loss': 0.1798, 'grad_norm': 2.128032922744751, 'learning_rate': 3.955094381794746e-05, 'epoch': 0.96}

 10%|▉         | 3942/41250 [9:30:25<90:08:47,  8.70s/it][2025-04-25 17:28:08,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 17:28:08,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.31 | bwd_microstep: 5783.04 | bwd_inner_microstep: 5653.00 | bwd_allreduce_microstep: 129.99 | step_microstep: 19.02
[2025-04-25 17:28:08,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.31 | bwd: 5783.06 | bwd_inner: 5653.00 | bwd_allreduce: 130.01 | step: 19.02

 10%|▉         | 3943/41250 [9:30:33<90:09:04,  8.70s/it]
                                                         
{'loss': 0.056, 'grad_norm': 0.5947864055633545, 'learning_rate': 3.9550612864075636e-05, 'epoch': 0.96}

 10%|▉         | 3943/41250 [9:30:33<90:09:04,  8.70s/it][2025-04-25 17:28:17,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:28:17,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5740.23 | bwd_inner_microstep: 5713.37 | bwd_allreduce_microstep: 26.81 | step_microstep: 18.95
[2025-04-25 17:28:17,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5740.25 | bwd_inner: 5713.37 | bwd_allreduce: 26.83 | step: 18.95

 10%|▉         | 3944/41250 [9:30:42<90:05:11,  8.69s/it]
                                                         
{'loss': 0.0214, 'grad_norm': 0.3934248387813568, 'learning_rate': 3.955028178967813e-05, 'epoch': 0.96}

 10%|▉         | 3944/41250 [9:30:42<90:05:11,  8.69s/it][2025-04-25 17:28:25,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 17:28:25,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.69 | bwd_microstep: 5748.75 | bwd_inner_microstep: 5652.34 | bwd_allreduce_microstep: 96.36 | step_microstep: 18.96
[2025-04-25 17:28:25,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5748.77 | bwd_inner: 5652.34 | bwd_allreduce: 96.39 | step: 18.96

 10%|▉         | 3945/41250 [9:30:51<89:57:53,  8.68s/it]
                                                         
{'loss': 0.1699, 'grad_norm': 1.2898077964782715, 'learning_rate': 3.954995059475699e-05, 'epoch': 0.96}

 10%|▉         | 3945/41250 [9:30:51<89:57:53,  8.68s/it][2025-04-25 17:28:34,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 17:28:34,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.77 | bwd_microstep: 5710.09 | bwd_inner_microstep: 5697.37 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.74
[2025-04-25 17:28:34,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.77 | bwd: 5710.10 | bwd_inner: 5697.37 | bwd_allreduce: 12.69 | step: 18.75

 10%|▉         | 3946/41250 [9:30:59<89:50:44,  8.67s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.2016029357910156, 'learning_rate': 3.954961927931424e-05, 'epoch': 0.96}

 10%|▉         | 3946/41250 [9:30:59<89:50:44,  8.67s/it][2025-04-25 17:28:43,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:28:43,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.66 | bwd_microstep: 5766.55 | bwd_inner_microstep: 5661.85 | bwd_allreduce_microstep: 104.66 | step_microstep: 18.51
[2025-04-25 17:28:43,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.66 | bwd: 5766.57 | bwd_inner: 5661.85 | bwd_allreduce: 104.68 | step: 18.51

 10%|▉         | 3947/41250 [9:31:08<89:51:47,  8.67s/it]
                                                         
{'loss': 0.1985, 'grad_norm': 1.1207644939422607, 'learning_rate': 3.954928784335194e-05, 'epoch': 0.96}

 10%|▉         | 3947/41250 [9:31:08<89:51:47,  8.67s/it][2025-04-25 17:28:51,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:28:51,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.26 | bwd_microstep: 5764.46 | bwd_inner_microstep: 5685.09 | bwd_allreduce_microstep: 79.33 | step_microstep: 19.01
[2025-04-25 17:28:51,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.26 | bwd: 5764.47 | bwd_inner: 5685.09 | bwd_allreduce: 79.34 | step: 19.01

 10%|▉         | 3948/41250 [9:31:17<89:55:08,  8.68s/it]
                                                         
{'loss': 0.0982, 'grad_norm': 2.49066162109375, 'learning_rate': 3.954895628687212e-05, 'epoch': 0.96}

 10%|▉         | 3948/41250 [9:31:17<89:55:08,  8.68s/it][2025-04-25 17:29:00,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 17:29:00,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.99 | bwd_microstep: 5814.17 | bwd_inner_microstep: 5649.00 | bwd_allreduce_microstep: 165.11 | step_microstep: 19.01
[2025-04-25 17:29:00,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.99 | bwd: 5814.19 | bwd_inner: 5649.00 | bwd_allreduce: 165.14 | step: 19.01

 10%|▉         | 3949/41250 [9:31:25<90:06:56,  8.70s/it]
                                                         
{'loss': 0.1776, 'grad_norm': 1.8085544109344482, 'learning_rate': 3.9548624609876836e-05, 'epoch': 0.96}

 10%|▉         | 3949/41250 [9:31:25<90:06:56,  8.70s/it][2025-04-25 17:29:09,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 17:29:09,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5732.31 | bwd_inner_microstep: 5683.19 | bwd_allreduce_microstep: 49.08 | step_microstep: 19.14
[2025-04-25 17:29:09,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5732.33 | bwd_inner: 5683.19 | bwd_allreduce: 49.09 | step: 19.14

 10%|▉         | 3950/41250 [9:31:34<90:00:16,  8.69s/it]
                                                         
{'loss': 0.5043, 'grad_norm': 2.665605306625366, 'learning_rate': 3.9548292812368124e-05, 'epoch': 0.96}

 10%|▉         | 3950/41250 [9:31:34<90:00:16,  8.69s/it][2025-04-25 17:29:17,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 17:29:17,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.51 | bwd_microstep: 5741.48 | bwd_inner_microstep: 5681.98 | bwd_allreduce_microstep: 59.46 | step_microstep: 19.42
[2025-04-25 17:29:17,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.51 | bwd: 5741.49 | bwd_inner: 5681.98 | bwd_allreduce: 59.48 | step: 19.42

 10%|▉         | 3951/41250 [9:31:43<89:59:08,  8.69s/it]
                                                         
{'loss': 0.6761, 'grad_norm': 2.7391562461853027, 'learning_rate': 3.9547960894348023e-05, 'epoch': 0.96}

 10%|▉         | 3951/41250 [9:31:43<89:59:08,  8.69s/it][2025-04-25 17:29:26,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 17:29:26,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.61 | bwd_microstep: 5742.94 | bwd_inner_microstep: 5658.51 | bwd_allreduce_microstep: 84.38 | step_microstep: 19.01
[2025-04-25 17:29:26,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.61 | bwd: 5742.95 | bwd_inner: 5658.51 | bwd_allreduce: 84.40 | step: 19.03

 10%|▉         | 3952/41250 [9:31:51<89:55:41,  8.68s/it]
                                                         
{'loss': 0.2662, 'grad_norm': 2.6754379272460938, 'learning_rate': 3.9547628855818594e-05, 'epoch': 0.96}

 10%|▉         | 3952/41250 [9:31:51<89:55:41,  8.68s/it][2025-04-25 17:29:35,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.19 | optimizer_step: 1.09
[2025-04-25 17:29:35,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5679.11 | bwd_inner_microstep: 5653.24 | bwd_allreduce_microstep: 25.80 | step_microstep: 20.61
[2025-04-25 17:29:35,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5679.13 | bwd_inner: 5653.24 | bwd_allreduce: 25.83 | step: 20.61

 10%|▉         | 3953/41250 [9:32:00<89:40:25,  8.66s/it]
                                                         
{'loss': 0.1167, 'grad_norm': 1.195387601852417, 'learning_rate': 3.954729669678188e-05, 'epoch': 0.96}

 10%|▉         | 3953/41250 [9:32:00<89:40:25,  8.66s/it][2025-04-25 17:29:43,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.35 | optimizer_step: 0.95
[2025-04-25 17:29:43,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.55 | bwd_microstep: 5710.43 | bwd_inner_microstep: 5648.01 | bwd_allreduce_microstep: 62.36 | step_microstep: 19.78
[2025-04-25 17:29:43,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5710.44 | bwd_inner: 5648.01 | bwd_allreduce: 62.38 | step: 19.79

 10%|▉         | 3954/41250 [9:32:09<89:34:44,  8.65s/it]
                                                         
{'loss': 0.2926, 'grad_norm': 2.0523488521575928, 'learning_rate': 3.9546964417239926e-05, 'epoch': 0.96}

 10%|▉         | 3954/41250 [9:32:09<89:34:44,  8.65s/it][2025-04-25 17:29:52,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:29:52,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.79 | bwd_microstep: 5775.92 | bwd_inner_microstep: 5661.15 | bwd_allreduce_microstep: 114.72 | step_microstep: 19.13
[2025-04-25 17:29:52,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.79 | bwd: 5775.93 | bwd_inner: 5661.15 | bwd_allreduce: 114.74 | step: 19.13

 10%|▉         | 3955/41250 [9:32:17<89:43:03,  8.66s/it]
                                                         
{'loss': 0.2841, 'grad_norm': 1.6910523176193237, 'learning_rate': 3.954663201719478e-05, 'epoch': 0.96}

 10%|▉         | 3955/41250 [9:32:17<89:43:03,  8.66s/it][2025-04-25 17:30:01,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:30:01,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.35 | bwd_microstep: 5725.83 | bwd_inner_microstep: 5703.68 | bwd_allreduce_microstep: 22.11 | step_microstep: 18.95
[2025-04-25 17:30:01,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.35 | bwd: 5725.85 | bwd_inner: 5703.68 | bwd_allreduce: 22.12 | step: 18.95

 10%|▉         | 3956/41250 [9:32:26<89:44:47,  8.66s/it]
                                                         
{'loss': 0.2342, 'grad_norm': 2.9865357875823975, 'learning_rate': 3.9546299496648494e-05, 'epoch': 0.96}

 10%|▉         | 3956/41250 [9:32:26<89:44:47,  8.66s/it][2025-04-25 17:30:09,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-25 17:30:09,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.92 | bwd_microstep: 5756.19 | bwd_inner_microstep: 5651.00 | bwd_allreduce_microstep: 105.14 | step_microstep: 19.34
[2025-04-25 17:30:09,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.92 | bwd: 5756.21 | bwd_inner: 5651.00 | bwd_allreduce: 105.17 | step: 19.34

 10%|▉         | 3957/41250 [9:32:35<89:45:47,  8.67s/it]
                                                         
{'loss': 0.2151, 'grad_norm': 2.939371109008789, 'learning_rate': 3.9545966855603116e-05, 'epoch': 0.96}

 10%|▉         | 3957/41250 [9:32:35<89:45:47,  8.67s/it][2025-04-25 17:30:18,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 17:30:18,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.96 | bwd_microstep: 5815.59 | bwd_inner_microstep: 5701.42 | bwd_allreduce_microstep: 114.13 | step_microstep: 19.32
[2025-04-25 17:30:18,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.96 | bwd: 5815.60 | bwd_inner: 5701.42 | bwd_allreduce: 114.14 | step: 19.32

 10%|▉         | 3958/41250 [9:32:43<90:01:33,  8.69s/it]
                                                         
{'loss': 0.1402, 'grad_norm': 0.8078702688217163, 'learning_rate': 3.9545634094060694e-05, 'epoch': 0.96}

 10%|▉         | 3958/41250 [9:32:43<90:01:33,  8.69s/it][2025-04-25 17:30:27,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:30:27,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5710.70 | bwd_inner_microstep: 5695.14 | bwd_allreduce_microstep: 15.50 | step_microstep: 19.62
[2025-04-25 17:30:27,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5710.72 | bwd_inner: 5695.14 | bwd_allreduce: 15.52 | step: 19.62

 10%|▉         | 3959/41250 [9:32:52<89:53:38,  8.68s/it]
                                                         
{'loss': 0.3566, 'grad_norm': 2.2397048473358154, 'learning_rate': 3.954530121202328e-05, 'epoch': 0.96}

 10%|▉         | 3959/41250 [9:32:52<89:53:38,  8.68s/it][2025-04-25 17:30:35,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 7.05 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:30:35,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.56 | bwd_microstep: 5689.54 | bwd_inner_microstep: 5653.52 | bwd_allreduce_microstep: 35.98 | step_microstep: 20.49
[2025-04-25 17:30:35,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.56 | bwd: 5689.56 | bwd_inner: 5653.52 | bwd_allreduce: 35.99 | step: 20.49

 10%|▉         | 3960/41250 [9:33:01<89:39:00,  8.65s/it]
                                                         
{'loss': 0.1388, 'grad_norm': 0.7622479796409607, 'learning_rate': 3.954496820949293e-05, 'epoch': 0.96}

 10%|▉         | 3960/41250 [9:33:01<89:39:00,  8.65s/it][2025-04-25 17:30:44,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:30:44,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.33 | bwd_microstep: 5844.32 | bwd_inner_microstep: 5694.41 | bwd_allreduce_microstep: 149.87 | step_microstep: 18.71
[2025-04-25 17:30:44,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.33 | bwd: 5844.34 | bwd_inner: 5694.41 | bwd_allreduce: 149.88 | step: 18.72

 10%|▉         | 3961/41250 [9:33:09<90:01:00,  8.69s/it]
                                                         
{'loss': 0.212, 'grad_norm': 1.6200299263000488, 'learning_rate': 3.95446350864717e-05, 'epoch': 0.96}

 10%|▉         | 3961/41250 [9:33:09<90:01:00,  8.69s/it][2025-04-25 17:30:53,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.06
[2025-04-25 17:30:53,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5740.17 | bwd_inner_microstep: 5700.52 | bwd_allreduce_microstep: 39.58 | step_microstep: 20.25
[2025-04-25 17:30:53,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.69 | bwd: 5740.19 | bwd_inner: 5700.52 | bwd_allreduce: 39.62 | step: 20.25

 10%|▉         | 3962/41250 [9:33:18<89:58:42,  8.69s/it]
                                                         
{'loss': 0.0605, 'grad_norm': 0.5405651330947876, 'learning_rate': 3.9544301842961636e-05, 'epoch': 0.96}

 10%|▉         | 3962/41250 [9:33:18<89:58:42,  8.69s/it][2025-04-25 17:31:01,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 17:31:01,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.05 | bwd_microstep: 5712.65 | bwd_inner_microstep: 5700.03 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.98
[2025-04-25 17:31:01,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.05 | bwd: 5712.67 | bwd_inner: 5700.03 | bwd_allreduce: 12.59 | step: 18.99

 10%|▉         | 3963/41250 [9:33:27<89:53:04,  8.68s/it]
                                                         
{'loss': 0.2065, 'grad_norm': 1.2501965761184692, 'learning_rate': 3.9543968478964794e-05, 'epoch': 0.96}

 10%|▉         | 3963/41250 [9:33:27<89:53:04,  8.68s/it][2025-04-25 17:31:10,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:31:10,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5678.78 | bwd_inner_microstep: 5652.05 | bwd_allreduce_microstep: 26.69 | step_microstep: 18.91
[2025-04-25 17:31:10,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5678.80 | bwd_inner: 5652.05 | bwd_allreduce: 26.70 | step: 18.91

 10%|▉         | 3964/41250 [9:33:35<89:36:46,  8.65s/it]
                                                         
{'loss': 0.1005, 'grad_norm': 0.8166825175285339, 'learning_rate': 3.9543634994483235e-05, 'epoch': 0.96}

 10%|▉         | 3964/41250 [9:33:35<89:36:46,  8.65s/it][2025-04-25 17:31:19,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:31:19,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.57 | bwd_microstep: 5761.79 | bwd_inner_microstep: 5658.24 | bwd_allreduce_microstep: 103.51 | step_microstep: 18.91
[2025-04-25 17:31:19,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.57 | bwd: 5761.81 | bwd_inner: 5658.24 | bwd_allreduce: 103.53 | step: 18.91

 10%|▉         | 3965/41250 [9:33:44<89:39:36,  8.66s/it]
                                                         
{'loss': 0.1743, 'grad_norm': 1.7808395624160767, 'learning_rate': 3.9543301389519005e-05, 'epoch': 0.96}

 10%|▉         | 3965/41250 [9:33:44<89:39:36,  8.66s/it][2025-04-25 17:31:27,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 17:31:27,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.31 | bwd_microstep: 5716.30 | bwd_inner_microstep: 5703.77 | bwd_allreduce_microstep: 12.49 | step_microstep: 17.83
[2025-04-25 17:31:27,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.31 | bwd: 5716.31 | bwd_inner: 5703.77 | bwd_allreduce: 12.50 | step: 17.83

 10%|▉         | 3966/41250 [9:33:53<89:37:41,  8.65s/it]
                                                         
{'loss': 0.0549, 'grad_norm': 0.42843830585479736, 'learning_rate': 3.954296766407417e-05, 'epoch': 0.96}

 10%|▉         | 3966/41250 [9:33:53<89:37:41,  8.65s/it][2025-04-25 17:31:36,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:31:36,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3097.40 | bwd_microstep: 5665.75 | bwd_inner_microstep: 5652.91 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.54
[2025-04-25 17:31:36,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3097.40 | bwd: 5665.77 | bwd_inner: 5652.91 | bwd_allreduce: 12.82 | step: 18.54

 10%|▉         | 3967/41250 [9:34:01<90:13:07,  8.71s/it]
                                                         
{'loss': 0.225, 'grad_norm': 2.194779634475708, 'learning_rate': 3.9542633818150784e-05, 'epoch': 0.96}

 10%|▉         | 3967/41250 [9:34:01<90:13:07,  8.71s/it][2025-04-25 17:31:45,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 17:31:45,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.59 | bwd_microstep: 5759.93 | bwd_inner_microstep: 5649.79 | bwd_allreduce_microstep: 110.10 | step_microstep: 18.87
[2025-04-25 17:31:45,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.59 | bwd: 5759.95 | bwd_inner: 5649.79 | bwd_allreduce: 110.11 | step: 18.87

 10%|▉         | 3968/41250 [9:34:10<90:05:06,  8.70s/it]
                                                         
{'loss': 0.2036, 'grad_norm': 1.9217954874038696, 'learning_rate': 3.95422998517509e-05, 'epoch': 0.96}

 10%|▉         | 3968/41250 [9:34:10<90:05:06,  8.70s/it][2025-04-25 17:31:53,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:31:53,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.93 | bwd_microstep: 5682.00 | bwd_inner_microstep: 5669.13 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.65
[2025-04-25 17:31:53,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.93 | bwd: 5682.01 | bwd_inner: 5669.13 | bwd_allreduce: 12.84 | step: 18.66

 10%|▉         | 3969/41250 [9:34:19<89:48:05,  8.67s/it]
                                                         
{'loss': 0.1374, 'grad_norm': 1.067766547203064, 'learning_rate': 3.9541965764876585e-05, 'epoch': 0.96}

 10%|▉         | 3969/41250 [9:34:19<89:48:05,  8.67s/it][2025-04-25 17:32:02,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:32:02,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.89 | bwd_microstep: 5841.34 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 147.42 | step_microstep: 18.68
[2025-04-25 17:32:02,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.89 | bwd: 5841.35 | bwd_inner: 5693.87 | bwd_allreduce: 147.44 | step: 18.69

 10%|▉         | 3970/41250 [9:34:27<90:07:56,  8.70s/it]
                                                         
{'loss': 0.2319, 'grad_norm': 1.452559232711792, 'learning_rate': 3.954163155752989e-05, 'epoch': 0.96}

 10%|▉         | 3970/41250 [9:34:27<90:07:56,  8.70s/it][2025-04-25 17:32:11,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 17:32:11,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.78 | bwd_microstep: 5766.33 | bwd_inner_microstep: 5668.64 | bwd_allreduce_microstep: 97.64 | step_microstep: 19.58
[2025-04-25 17:32:11,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.78 | bwd: 5766.34 | bwd_inner: 5668.64 | bwd_allreduce: 97.66 | step: 19.58

 10%|▉         | 3971/41250 [9:34:36<90:05:16,  8.70s/it]
                                                         
{'loss': 0.1754, 'grad_norm': 1.328850269317627, 'learning_rate': 3.954129722971289e-05, 'epoch': 0.96}

 10%|▉         | 3971/41250 [9:34:36<90:05:16,  8.70s/it][2025-04-25 17:32:20,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.17 | optimizer_step: 0.93
[2025-04-25 17:32:20,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.72 | bwd_microstep: 5718.57 | bwd_inner_microstep: 5705.41 | bwd_allreduce_microstep: 13.11 | step_microstep: 20.24
[2025-04-25 17:32:20,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.72 | bwd: 5718.58 | bwd_inner: 5705.41 | bwd_allreduce: 13.13 | step: 20.24

 10%|▉         | 3972/41250 [9:34:45<89:59:00,  8.69s/it]
                                                         
{'loss': 0.2098, 'grad_norm': 1.7936797142028809, 'learning_rate': 3.9540962781427626e-05, 'epoch': 0.96}

 10%|▉         | 3972/41250 [9:34:45<89:59:00,  8.69s/it][2025-04-25 17:32:28,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:32:28,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.43 | bwd_microstep: 5745.04 | bwd_inner_microstep: 5669.32 | bwd_allreduce_microstep: 75.67 | step_microstep: 18.34
[2025-04-25 17:32:28,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.43 | bwd: 5745.05 | bwd_inner: 5669.32 | bwd_allreduce: 75.69 | step: 18.35

 10%|▉         | 3973/41250 [9:34:54<89:57:38,  8.69s/it]
                                                         
{'loss': 0.1777, 'grad_norm': 1.637317180633545, 'learning_rate': 3.954062821267617e-05, 'epoch': 0.96}

 10%|▉         | 3973/41250 [9:34:54<89:57:38,  8.69s/it][2025-04-25 17:32:37,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:32:37,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.75 | bwd_microstep: 5802.50 | bwd_inner_microstep: 5789.64 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.29
[2025-04-25 17:32:37,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.75 | bwd: 5802.51 | bwd_inner: 5789.64 | bwd_allreduce: 12.83 | step: 18.30

 10%|▉         | 3974/41250 [9:35:02<90:14:38,  8.72s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 1.0140362977981567, 'learning_rate': 3.954029352346059e-05, 'epoch': 0.96}

 10%|▉         | 3974/41250 [9:35:02<90:14:38,  8.72s/it][2025-04-25 17:32:46,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:32:46,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.50 | bwd_microstep: 5712.07 | bwd_inner_microstep: 5697.28 | bwd_allreduce_microstep: 14.74 | step_microstep: 18.18
[2025-04-25 17:32:46,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.50 | bwd: 5712.08 | bwd_inner: 5697.28 | bwd_allreduce: 14.76 | step: 18.18

 10%|▉         | 3975/41250 [9:35:11<90:01:55,  8.70s/it]
                                                         
{'loss': 0.1336, 'grad_norm': 1.11337149143219, 'learning_rate': 3.953995871378294e-05, 'epoch': 0.96}

 10%|▉         | 3975/41250 [9:35:11<90:01:55,  8.70s/it][2025-04-25 17:32:54,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 17:32:54,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.40 | bwd_microstep: 5681.56 | bwd_inner_microstep: 5668.82 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.49
[2025-04-25 17:32:54,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.40 | bwd: 5681.57 | bwd_inner: 5668.82 | bwd_allreduce: 12.71 | step: 18.49

 10%|▉         | 3976/41250 [9:35:20<89:47:54,  8.67s/it]
                                                         
{'loss': 0.2179, 'grad_norm': 1.836059331893921, 'learning_rate': 3.9539623783645284e-05, 'epoch': 0.96}

 10%|▉         | 3976/41250 [9:35:20<89:47:54,  8.67s/it][2025-04-25 17:33:03,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:33:03,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.14 | bwd_microstep: 5738.46 | bwd_inner_microstep: 5725.84 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.65
[2025-04-25 17:33:03,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.14 | bwd: 5738.48 | bwd_inner: 5725.84 | bwd_allreduce: 12.60 | step: 18.65

 10%|▉         | 3977/41250 [9:35:28<89:49:35,  8.68s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 0.6781360507011414, 'learning_rate': 3.95392887330497e-05, 'epoch': 0.96}

 10%|▉         | 3977/41250 [9:35:28<89:49:35,  8.68s/it][2025-04-25 17:33:12,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 17:33:12,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.79 | bwd_microstep: 5741.36 | bwd_inner_microstep: 5716.36 | bwd_allreduce_microstep: 24.96 | step_microstep: 18.25
[2025-04-25 17:33:12,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.79 | bwd: 5741.37 | bwd_inner: 5716.36 | bwd_allreduce: 24.97 | step: 18.25

 10%|▉         | 3978/41250 [9:35:37<89:53:59,  8.68s/it]
                                                         
{'loss': 0.2172, 'grad_norm': 1.829516053199768, 'learning_rate': 3.9538953561998236e-05, 'epoch': 0.96}

 10%|▉         | 3978/41250 [9:35:37<89:53:59,  8.68s/it][2025-04-25 17:33:20,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-25 17:33:20,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5688.12 | bwd_inner_microstep: 5663.79 | bwd_allreduce_microstep: 24.29 | step_microstep: 18.91
[2025-04-25 17:33:20,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5688.14 | bwd_inner: 5663.79 | bwd_allreduce: 24.31 | step: 18.91

 10%|▉         | 3979/41250 [9:35:46<89:39:15,  8.66s/it]
                                                         
{'loss': 0.0998, 'grad_norm': 0.6515937447547913, 'learning_rate': 3.953861827049297e-05, 'epoch': 0.96}

 10%|▉         | 3979/41250 [9:35:46<89:39:15,  8.66s/it][2025-04-25 17:33:29,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 17:33:29,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.72 | bwd_microstep: 5727.41 | bwd_inner_microstep: 5713.86 | bwd_allreduce_microstep: 13.50 | step_microstep: 19.33
[2025-04-25 17:33:29,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.72 | bwd: 5727.43 | bwd_inner: 5713.86 | bwd_allreduce: 13.52 | step: 19.33

 10%|▉         | 3980/41250 [9:35:54<89:42:51,  8.67s/it]
                                                         
{'loss': 0.235, 'grad_norm': 1.1000021696090698, 'learning_rate': 3.953828285853597e-05, 'epoch': 0.96}

 10%|▉         | 3980/41250 [9:35:54<89:42:51,  8.67s/it][2025-04-25 17:33:38,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 17:33:38,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.38 | bwd_microstep: 5706.41 | bwd_inner_microstep: 5658.41 | bwd_allreduce_microstep: 47.95 | step_microstep: 18.96
[2025-04-25 17:33:38,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.38 | bwd: 5706.42 | bwd_inner: 5658.41 | bwd_allreduce: 47.97 | step: 18.97

 10%|▉         | 3981/41250 [9:36:03<89:34:48,  8.65s/it]
                                                         
{'loss': 0.0895, 'grad_norm': 0.5765814185142517, 'learning_rate': 3.953794732612929e-05, 'epoch': 0.97}

 10%|▉         | 3981/41250 [9:36:03<89:34:48,  8.65s/it][2025-04-25 17:33:46,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 17:33:46,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.76 | bwd_microstep: 5731.87 | bwd_inner_microstep: 5718.60 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.24
[2025-04-25 17:33:46,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.76 | bwd: 5731.88 | bwd_inner: 5718.60 | bwd_allreduce: 13.24 | step: 19.24

 10%|▉         | 3982/41250 [9:36:12<89:37:48,  8.66s/it]
                                                         
{'loss': 0.0933, 'grad_norm': 0.7125764489173889, 'learning_rate': 3.9537611673275017e-05, 'epoch': 0.97}

 10%|▉         | 3982/41250 [9:36:12<89:37:48,  8.66s/it][2025-04-25 17:33:55,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-25 17:33:55,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.79 | bwd_microstep: 5885.67 | bwd_inner_microstep: 5794.83 | bwd_allreduce_microstep: 90.80 | step_microstep: 18.92
[2025-04-25 17:33:55,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.79 | bwd: 5885.69 | bwd_inner: 5794.83 | bwd_allreduce: 90.82 | step: 18.92

 10%|▉         | 3983/41250 [9:36:20<90:15:59,  8.72s/it]
                                                         
{'loss': 0.1872, 'grad_norm': 1.4475879669189453, 'learning_rate': 3.95372758999752e-05, 'epoch': 0.97}

 10%|▉         | 3983/41250 [9:36:20<90:15:59,  8.72s/it][2025-04-25 17:34:04,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 17:34:04,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.61 | bwd_microstep: 5732.49 | bwd_inner_microstep: 5719.61 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.11
[2025-04-25 17:34:04,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.62 | bwd: 5732.51 | bwd_inner: 5719.61 | bwd_allreduce: 12.85 | step: 19.11

 10%|▉         | 3984/41250 [9:36:29<90:08:50,  8.71s/it]
                                                         
{'loss': 0.0698, 'grad_norm': 0.6297333240509033, 'learning_rate': 3.953694000623193e-05, 'epoch': 0.97}

 10%|▉         | 3984/41250 [9:36:29<90:08:50,  8.71s/it][2025-04-25 17:34:12,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:34:12,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.01 | bwd_microstep: 5743.41 | bwd_inner_microstep: 5703.59 | bwd_allreduce_microstep: 39.78 | step_microstep: 19.08
[2025-04-25 17:34:12,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.01 | bwd: 5743.42 | bwd_inner: 5703.59 | bwd_allreduce: 39.80 | step: 19.09

 10%|▉         | 3985/41250 [9:36:38<90:03:21,  8.70s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 0.7236608862876892, 'learning_rate': 3.953660399204726e-05, 'epoch': 0.97}

 10%|▉         | 3985/41250 [9:36:38<90:03:21,  8.70s/it][2025-04-25 17:34:21,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-25 17:34:21,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.53 | bwd_microstep: 5805.30 | bwd_inner_microstep: 5790.65 | bwd_allreduce_microstep: 14.59 | step_microstep: 19.18
[2025-04-25 17:34:21,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.53 | bwd: 5805.31 | bwd_inner: 5790.65 | bwd_allreduce: 14.62 | step: 19.18

 10%|▉         | 3986/41250 [9:36:47<90:19:00,  8.73s/it]
                                                         
{'loss': 0.1332, 'grad_norm': 0.855831503868103, 'learning_rate': 3.9536267857423276e-05, 'epoch': 0.97}

 10%|▉         | 3986/41250 [9:36:47<90:19:00,  8.73s/it][2025-04-25 17:34:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 17:34:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.86 | bwd_microstep: 5853.92 | bwd_inner_microstep: 5705.56 | bwd_allreduce_microstep: 148.32 | step_microstep: 18.64
[2025-04-25 17:34:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.86 | bwd: 5853.94 | bwd_inner: 5705.56 | bwd_allreduce: 148.33 | step: 18.65

 10%|▉         | 3987/41250 [9:36:55<90:30:10,  8.74s/it]
                                                         
{'loss': 0.2398, 'grad_norm': 1.842632532119751, 'learning_rate': 3.953593160236204e-05, 'epoch': 0.97}

 10%|▉         | 3987/41250 [9:36:55<90:30:10,  8.74s/it][2025-04-25 17:34:39,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:34:39,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.94 | bwd_microstep: 5706.20 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.71
[2025-04-25 17:34:39,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.94 | bwd: 5706.21 | bwd_inner: 5693.53 | bwd_allreduce: 12.64 | step: 18.71

 10%|▉         | 3988/41250 [9:37:04<90:13:26,  8.72s/it]
                                                         
{'loss': 0.0671, 'grad_norm': 0.8199146389961243, 'learning_rate': 3.953559522686563e-05, 'epoch': 0.97}

 10%|▉         | 3988/41250 [9:37:04<90:13:26,  8.72s/it][2025-04-25 17:34:47,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:34:47,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.76 | bwd_microstep: 5735.52 | bwd_inner_microstep: 5661.37 | bwd_allreduce_microstep: 74.11 | step_microstep: 18.27
[2025-04-25 17:34:47,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.76 | bwd: 5735.53 | bwd_inner: 5661.37 | bwd_allreduce: 74.12 | step: 18.28

 10%|▉         | 3989/41250 [9:37:13<90:03:27,  8.70s/it]
                                                         
{'loss': 0.0604, 'grad_norm': 0.8177845478057861, 'learning_rate': 3.953525873093611e-05, 'epoch': 0.97}

 10%|▉         | 3989/41250 [9:37:13<90:03:27,  8.70s/it][2025-04-25 17:34:56,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:34:56,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5755.74 | bwd_inner_microstep: 5701.48 | bwd_allreduce_microstep: 54.21 | step_microstep: 18.69
[2025-04-25 17:34:56,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5755.75 | bwd_inner: 5701.48 | bwd_allreduce: 54.22 | step: 18.69

 10%|▉         | 3990/41250 [9:37:21<90:02:20,  8.70s/it]
                                                         
{'loss': 0.058, 'grad_norm': 1.109427809715271, 'learning_rate': 3.953492211457557e-05, 'epoch': 0.97}

 10%|▉         | 3990/41250 [9:37:21<90:02:20,  8.70s/it][2025-04-25 17:35:05,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 17:35:05,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.54 | bwd_microstep: 5725.93 | bwd_inner_microstep: 5713.09 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.95
[2025-04-25 17:35:05,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.54 | bwd: 5725.95 | bwd_inner: 5713.09 | bwd_allreduce: 12.82 | step: 18.95

 10%|▉         | 3991/41250 [9:37:30<89:54:33,  8.69s/it]
                                                         
{'loss': 0.1068, 'grad_norm': 1.2027921676635742, 'learning_rate': 3.953458537778608e-05, 'epoch': 0.97}

 10%|▉         | 3991/41250 [9:37:30<89:54:33,  8.69s/it][2025-04-25 17:35:13,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.19 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 17:35:13,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.93 | bwd_microstep: 5730.57 | bwd_inner_microstep: 5717.72 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.27
[2025-04-25 17:35:13,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.93 | bwd: 5730.58 | bwd_inner: 5717.72 | bwd_allreduce: 12.82 | step: 19.27

 10%|▉         | 3992/41250 [9:37:39<89:51:33,  8.68s/it]
                                                         
{'loss': 0.188, 'grad_norm': 2.6217634677886963, 'learning_rate': 3.953424852056971e-05, 'epoch': 0.97}

 10%|▉         | 3992/41250 [9:37:39<89:51:33,  8.68s/it][2025-04-25 17:35:22,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:35:22,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.02 | bwd_microstep: 5772.86 | bwd_inner_microstep: 5666.61 | bwd_allreduce_microstep: 106.21 | step_microstep: 18.86
[2025-04-25 17:35:22,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.02 | bwd: 5772.88 | bwd_inner: 5666.61 | bwd_allreduce: 106.23 | step: 18.86

 10%|▉         | 3993/41250 [9:37:47<89:53:58,  8.69s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 1.0954469442367554, 'learning_rate': 3.953391154292854e-05, 'epoch': 0.97}

 10%|▉         | 3993/41250 [9:37:47<89:53:58,  8.69s/it][2025-04-25 17:35:31,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:35:31,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.85 | bwd_microstep: 5777.49 | bwd_inner_microstep: 5656.02 | bwd_allreduce_microstep: 121.42 | step_microstep: 18.23
[2025-04-25 17:35:31,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.85 | bwd: 5777.50 | bwd_inner: 5656.02 | bwd_allreduce: 121.44 | step: 18.24

 10%|▉         | 3994/41250 [9:37:56<89:53:47,  8.69s/it]
                                                         
{'loss': 0.0393, 'grad_norm': 0.32642659544944763, 'learning_rate': 3.9533574444864644e-05, 'epoch': 0.97}

 10%|▉         | 3994/41250 [9:37:56<89:53:47,  8.69s/it][2025-04-25 17:35:39,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:35:39,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.74 | bwd_microstep: 5682.47 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 24.84 | step_microstep: 18.51
[2025-04-25 17:35:39,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.74 | bwd: 5682.48 | bwd_inner: 5657.59 | bwd_allreduce: 24.85 | step: 18.52

 10%|▉         | 3995/41250 [9:38:05<89:36:57,  8.66s/it]
                                                         
{'loss': 0.4067, 'grad_norm': 1.7817305326461792, 'learning_rate': 3.95332372263801e-05, 'epoch': 0.97}

 10%|▉         | 3995/41250 [9:38:05<89:36:57,  8.66s/it][2025-04-25 17:35:48,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.24 | optimizer_step: 0.99
[2025-04-25 17:35:48,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 6230.08 | bwd_inner_microstep: 5701.90 | bwd_allreduce_microstep: 528.13 | step_microstep: 19.06
[2025-04-25 17:35:48,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 6230.09 | bwd_inner: 5701.90 | bwd_allreduce: 528.15 | step: 19.06

 10%|▉         | 3996/41250 [9:38:14<91:11:01,  8.81s/it]
                                                         
{'loss': 0.0368, 'grad_norm': 0.4545108675956726, 'learning_rate': 3.9532899887477e-05, 'epoch': 0.97}

 10%|▉         | 3996/41250 [9:38:14<91:11:01,  8.81s/it][2025-04-25 17:35:57,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:35:57,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5698.78 | bwd_inner_microstep: 5684.57 | bwd_allreduce_microstep: 14.17 | step_microstep: 18.19
[2025-04-25 17:35:57,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5698.79 | bwd_inner: 5684.57 | bwd_allreduce: 14.19 | step: 18.19

 10%|▉         | 3997/41250 [9:38:22<90:37:38,  8.76s/it]
                                                         
{'loss': 0.3208, 'grad_norm': 1.8943287134170532, 'learning_rate': 3.9532562428157405e-05, 'epoch': 0.97}

 10%|▉         | 3997/41250 [9:38:22<90:37:38,  8.76s/it][2025-04-25 17:36:06,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:36:06,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.89 | bwd_microstep: 5699.75 | bwd_inner_microstep: 5686.67 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.70
[2025-04-25 17:36:06,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.89 | bwd: 5699.76 | bwd_inner: 5686.67 | bwd_allreduce: 13.05 | step: 18.70

 10%|▉         | 3998/41250 [9:38:31<90:13:09,  8.72s/it]
                                                         
{'loss': 0.0208, 'grad_norm': 0.1913617104291916, 'learning_rate': 3.953222484842342e-05, 'epoch': 0.97}

 10%|▉         | 3998/41250 [9:38:31<90:13:09,  8.72s/it][2025-04-25 17:36:14,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:36:14,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5707.02 | bwd_inner_microstep: 5693.93 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.65
[2025-04-25 17:36:14,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5707.03 | bwd_inner: 5693.93 | bwd_allreduce: 13.06 | step: 18.65

 10%|▉         | 3999/41250 [9:38:40<89:57:47,  8.69s/it]
                                                         
{'loss': 0.1424, 'grad_norm': 1.051087498664856, 'learning_rate': 3.95318871482771e-05, 'epoch': 0.97}

 10%|▉         | 3999/41250 [9:38:40<89:57:47,  8.69s/it][2025-04-25 17:36:23,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:36:23,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.54 | bwd_microstep: 5770.73 | bwd_inner_microstep: 5757.85 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.51
[2025-04-25 17:36:23,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.54 | bwd: 5770.74 | bwd_inner: 5757.85 | bwd_allreduce: 12.86 | step: 18.51

 10%|▉         | 4000/41250 [9:38:48<90:06:07,  8.71s/it]
                                                         
{'loss': 0.0748, 'grad_norm': 1.1845279932022095, 'learning_rate': 3.953154932772053e-05, 'epoch': 0.97}

 10%|▉         | 4000/41250 [9:38:48<90:06:07,  8.71s/it][2025-04-25 17:36:32,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:36:32,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.13 | bwd_microstep: 5669.95 | bwd_inner_microstep: 5655.95 | bwd_allreduce_microstep: 13.96 | step_microstep: 18.68
[2025-04-25 17:36:32,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.13 | bwd: 5669.96 | bwd_inner: 5655.95 | bwd_allreduce: 13.97 | step: 18.68

 10%|▉         | 4001/41250 [9:38:57<89:44:24,  8.67s/it]
                                                         
{'loss': 0.2012, 'grad_norm': 2.0158305168151855, 'learning_rate': 3.9531211386755815e-05, 'epoch': 0.97}

 10%|▉         | 4001/41250 [9:38:57<89:44:24,  8.67s/it][2025-04-25 17:36:40,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:36:40,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.96 | bwd_microstep: 5678.09 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 32.71 | step_microstep: 18.40
[2025-04-25 17:36:40,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.96 | bwd: 5678.10 | bwd_inner: 5645.34 | bwd_allreduce: 32.72 | step: 18.40

 10%|▉         | 4002/41250 [9:39:06<89:28:42,  8.65s/it]
                                                         
{'loss': 0.2404, 'grad_norm': 0.9654673933982849, 'learning_rate': 3.9530873325385014e-05, 'epoch': 0.97}

 10%|▉         | 4002/41250 [9:39:06<89:28:42,  8.65s/it][2025-04-25 17:36:49,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 17:36:49,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5716.17 | bwd_inner_microstep: 5692.18 | bwd_allreduce_microstep: 23.94 | step_microstep: 18.99
[2025-04-25 17:36:49,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5716.18 | bwd_inner: 5692.18 | bwd_allreduce: 23.96 | step: 18.99

 10%|▉         | 4003/41250 [9:39:14<89:29:06,  8.65s/it]
                                                         
{'loss': 0.0664, 'grad_norm': 0.9256415367126465, 'learning_rate': 3.953053514361022e-05, 'epoch': 0.97}

 10%|▉         | 4003/41250 [9:39:14<89:29:06,  8.65s/it][2025-04-25 17:36:58,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:36:58,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.59 | bwd_microstep: 5685.61 | bwd_inner_microstep: 5651.04 | bwd_allreduce_microstep: 34.53 | step_microstep: 18.38
[2025-04-25 17:36:58,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.59 | bwd: 5685.62 | bwd_inner: 5651.04 | bwd_allreduce: 34.54 | step: 18.38

 10%|▉         | 4004/41250 [9:39:23<89:18:17,  8.63s/it]
                                                         
{'loss': 0.0715, 'grad_norm': 0.7022204399108887, 'learning_rate': 3.953019684143352e-05, 'epoch': 0.97}

 10%|▉         | 4004/41250 [9:39:23<89:18:17,  8.63s/it][2025-04-25 17:37:06,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 17:37:06,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.50 | bwd_microstep: 5752.46 | bwd_inner_microstep: 5645.16 | bwd_allreduce_microstep: 107.25 | step_microstep: 19.04
[2025-04-25 17:37:06,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.50 | bwd: 5752.47 | bwd_inner: 5645.16 | bwd_allreduce: 107.28 | step: 19.04

 10%|▉         | 4005/41250 [9:39:32<89:24:41,  8.64s/it]
                                                         
{'loss': 0.2908, 'grad_norm': 2.011852979660034, 'learning_rate': 3.952985841885701e-05, 'epoch': 0.97}

 10%|▉         | 4005/41250 [9:39:32<89:24:41,  8.64s/it][2025-04-25 17:37:15,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 17:37:15,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5681.75 | bwd_inner_microstep: 5658.16 | bwd_allreduce_microstep: 23.54 | step_microstep: 18.87
[2025-04-25 17:37:15,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5681.76 | bwd_inner: 5658.16 | bwd_allreduce: 23.55 | step: 18.87

 10%|▉         | 4006/41250 [9:39:40<89:15:26,  8.63s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 1.4211164712905884, 'learning_rate': 3.9529519875882754e-05, 'epoch': 0.97}

 10%|▉         | 4006/41250 [9:39:40<89:15:26,  8.63s/it][2025-04-25 17:37:23,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-25 17:37:23,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.74 | bwd_microstep: 5673.62 | bwd_inner_microstep: 5656.50 | bwd_allreduce_microstep: 17.06 | step_microstep: 19.88
[2025-04-25 17:37:23,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.74 | bwd: 5673.63 | bwd_inner: 5656.50 | bwd_allreduce: 17.09 | step: 19.88

 10%|▉         | 4007/41250 [9:39:49<89:08:29,  8.62s/it]
                                                         
{'loss': 0.1128, 'grad_norm': 1.208055853843689, 'learning_rate': 3.9529181212512844e-05, 'epoch': 0.97}

 10%|▉         | 4007/41250 [9:39:49<89:08:29,  8.62s/it][2025-04-25 17:37:32,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 17:37:32,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.96 | bwd_microstep: 5676.97 | bwd_inner_microstep: 5657.71 | bwd_allreduce_microstep: 19.21 | step_microstep: 19.00
[2025-04-25 17:37:32,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.97 | bwd: 5676.99 | bwd_inner: 5657.71 | bwd_allreduce: 19.23 | step: 19.00

 10%|▉         | 4008/41250 [9:39:57<89:04:26,  8.61s/it]
                                                         
{'loss': 0.0778, 'grad_norm': 1.1379674673080444, 'learning_rate': 3.952884242874938e-05, 'epoch': 0.97}

 10%|▉         | 4008/41250 [9:39:57<89:04:26,  8.61s/it][2025-04-25 17:37:41,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 17:37:41,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.11 | bwd_microstep: 5731.99 | bwd_inner_microstep: 5697.12 | bwd_allreduce_microstep: 34.82 | step_microstep: 18.82
[2025-04-25 17:37:41,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.11 | bwd: 5732.00 | bwd_inner: 5697.12 | bwd_allreduce: 34.84 | step: 18.82

 10%|▉         | 4009/41250 [9:40:06<89:14:11,  8.63s/it]
                                                         
{'loss': 0.2064, 'grad_norm': 1.2492045164108276, 'learning_rate': 3.952850352459444e-05, 'epoch': 0.97}

 10%|▉         | 4009/41250 [9:40:06<89:14:11,  8.63s/it][2025-04-25 17:37:49,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 17:37:49,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.17 | bwd_microstep: 5724.07 | bwd_inner_microstep: 5705.08 | bwd_allreduce_microstep: 18.94 | step_microstep: 19.13
[2025-04-25 17:37:49,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.17 | bwd: 5724.08 | bwd_inner: 5705.08 | bwd_allreduce: 18.96 | step: 19.13

 10%|▉         | 4010/41250 [9:40:15<89:19:51,  8.64s/it]
                                                         
{'loss': 0.1778, 'grad_norm': 1.4326796531677246, 'learning_rate': 3.9528164500050116e-05, 'epoch': 0.97}

 10%|▉         | 4010/41250 [9:40:15<89:19:51,  8.64s/it][2025-04-25 17:37:58,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:37:58,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.35 | bwd_microstep: 5715.42 | bwd_inner_microstep: 5702.71 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.57
[2025-04-25 17:37:58,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.35 | bwd: 5715.43 | bwd_inner: 5702.71 | bwd_allreduce: 12.68 | step: 18.57

 10%|▉         | 4011/41250 [9:40:23<89:23:02,  8.64s/it]
                                                         
{'loss': 0.0823, 'grad_norm': 1.1464917659759521, 'learning_rate': 3.95278253551185e-05, 'epoch': 0.97}

 10%|▉         | 4011/41250 [9:40:23<89:23:02,  8.64s/it][2025-04-25 17:38:07,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 17:38:07,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.83 | bwd_microstep: 5772.88 | bwd_inner_microstep: 5648.15 | bwd_allreduce_microstep: 124.68 | step_microstep: 18.87
[2025-04-25 17:38:07,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.83 | bwd: 5772.90 | bwd_inner: 5648.15 | bwd_allreduce: 124.70 | step: 18.87

 10%|▉         | 4012/41250 [9:40:32<89:30:33,  8.65s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 1.0595184564590454, 'learning_rate': 3.952748608980169e-05, 'epoch': 0.97}

 10%|▉         | 4012/41250 [9:40:32<89:30:33,  8.65s/it][2025-04-25 17:38:15,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-25 17:38:15,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.00 | bwd_microstep: 5747.04 | bwd_inner_microstep: 5689.66 | bwd_allreduce_microstep: 57.32 | step_microstep: 19.69
[2025-04-25 17:38:15,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.00 | bwd: 5747.05 | bwd_inner: 5689.66 | bwd_allreduce: 57.35 | step: 19.69

 10%|▉         | 4013/41250 [9:40:41<89:36:12,  8.66s/it]
                                                         
{'loss': 0.0408, 'grad_norm': 0.6203743815422058, 'learning_rate': 3.952714670410176e-05, 'epoch': 0.97}

 10%|▉         | 4013/41250 [9:40:41<89:36:12,  8.66s/it][2025-04-25 17:38:24,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:38:24,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5786.44 | bwd_inner_microstep: 5655.30 | bwd_allreduce_microstep: 131.10 | step_microstep: 18.74
[2025-04-25 17:38:24,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5786.45 | bwd_inner: 5655.30 | bwd_allreduce: 131.11 | step: 18.75

 10%|▉         | 4014/41250 [9:40:49<89:43:46,  8.68s/it]
                                                         
{'loss': 0.4161, 'grad_norm': 3.567807197570801, 'learning_rate': 3.952680719802081e-05, 'epoch': 0.97}

 10%|▉         | 4014/41250 [9:40:49<89:43:46,  8.68s/it][2025-04-25 17:38:33,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:38:33,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5712.70 | bwd_inner_microstep: 5699.86 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.75
[2025-04-25 17:38:33,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5712.72 | bwd_inner: 5699.86 | bwd_allreduce: 12.81 | step: 18.75

 10%|▉         | 4015/41250 [9:40:58<89:38:33,  8.67s/it]
                                                         
{'loss': 0.3996, 'grad_norm': 4.150702953338623, 'learning_rate': 3.952646757156093e-05, 'epoch': 0.97}

 10%|▉         | 4015/41250 [9:40:58<89:38:33,  8.67s/it][2025-04-25 17:38:41,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 17:38:41,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.78 | bwd_microstep: 5717.85 | bwd_inner_microstep: 5705.05 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.81
[2025-04-25 17:38:41,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.78 | bwd: 5717.86 | bwd_inner: 5705.05 | bwd_allreduce: 12.77 | step: 18.81

 10%|▉         | 4016/41250 [9:41:07<89:38:34,  8.67s/it]
                                                         
{'loss': 0.2272, 'grad_norm': 2.148263931274414, 'learning_rate': 3.952612782472423e-05, 'epoch': 0.97}

 10%|▉         | 4016/41250 [9:41:07<89:38:34,  8.67s/it][2025-04-25 17:38:50,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:38:50,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.13 | bwd_microstep: 5707.75 | bwd_inner_microstep: 5652.50 | bwd_allreduce_microstep: 55.21 | step_microstep: 18.57
[2025-04-25 17:38:50,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.13 | bwd: 5707.76 | bwd_inner: 5652.50 | bwd_allreduce: 55.23 | step: 18.58

 10%|▉         | 4017/41250 [9:41:15<89:33:28,  8.66s/it]
                                                         
{'loss': 0.1525, 'grad_norm': 0.806987464427948, 'learning_rate': 3.952578795751279e-05, 'epoch': 0.97}

 10%|▉         | 4017/41250 [9:41:15<89:33:28,  8.66s/it][2025-04-25 17:38:59,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-25 17:38:59,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.05 | bwd_microstep: 5791.23 | bwd_inner_microstep: 5661.58 | bwd_allreduce_microstep: 129.60 | step_microstep: 19.04
[2025-04-25 17:38:59,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.05 | bwd: 5791.24 | bwd_inner: 5661.58 | bwd_allreduce: 129.62 | step: 19.05

 10%|▉         | 4018/41250 [9:41:24<89:44:06,  8.68s/it]
                                                         
{'loss': 0.2739, 'grad_norm': 1.612557291984558, 'learning_rate': 3.95254479699287e-05, 'epoch': 0.97}

 10%|▉         | 4018/41250 [9:41:24<89:44:06,  8.68s/it][2025-04-25 17:39:07,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 17:39:07,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.93 | bwd_microstep: 5749.09 | bwd_inner_microstep: 5705.35 | bwd_allreduce_microstep: 43.70 | step_microstep: 18.71
[2025-04-25 17:39:07,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5749.10 | bwd_inner: 5705.35 | bwd_allreduce: 43.71 | step: 18.71

 10%|▉         | 4019/41250 [9:41:33<89:44:14,  8.68s/it]
                                                         
{'loss': 0.4378, 'grad_norm': 2.4431357383728027, 'learning_rate': 3.952510786197407e-05, 'epoch': 0.97}

 10%|▉         | 4019/41250 [9:41:33<89:44:14,  8.68s/it][2025-04-25 17:39:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:39:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.67 | bwd_microstep: 5696.54 | bwd_inner_microstep: 5661.51 | bwd_allreduce_microstep: 34.98 | step_microstep: 18.43
[2025-04-25 17:39:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.67 | bwd: 5696.55 | bwd_inner: 5661.51 | bwd_allreduce: 35.00 | step: 18.44

 10%|▉         | 4020/41250 [9:41:41<89:33:18,  8.66s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.3836116790771484, 'learning_rate': 3.952476763365099e-05, 'epoch': 0.97}

 10%|▉         | 4020/41250 [9:41:41<89:33:18,  8.66s/it][2025-04-25 17:39:25,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:39:25,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.11 | bwd_microstep: 5771.93 | bwd_inner_microstep: 5695.16 | bwd_allreduce_microstep: 76.73 | step_microstep: 18.57
[2025-04-25 17:39:25,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.11 | bwd: 5771.95 | bwd_inner: 5695.16 | bwd_allreduce: 76.75 | step: 18.57

 10%|▉         | 4021/41250 [9:41:50<89:44:15,  8.68s/it]
                                                         
{'loss': 0.1858, 'grad_norm': 1.7554303407669067, 'learning_rate': 3.952442728496156e-05, 'epoch': 0.97}

 10%|▉         | 4021/41250 [9:41:50<89:44:15,  8.68s/it][2025-04-25 17:39:33,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 17:39:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5765.44 | bwd_inner_microstep: 5698.28 | bwd_allreduce_microstep: 67.11 | step_microstep: 18.34
[2025-04-25 17:39:33,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5765.46 | bwd_inner: 5698.28 | bwd_allreduce: 67.14 | step: 18.34

 10%|▉         | 4022/41250 [9:41:59<89:49:02,  8.69s/it]
                                                         
{'loss': 0.2325, 'grad_norm': 2.142301321029663, 'learning_rate': 3.952408681590787e-05, 'epoch': 0.98}

 10%|▉         | 4022/41250 [9:41:59<89:49:02,  8.69s/it][2025-04-25 17:39:42,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:39:42,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.92 | bwd_microstep: 5740.56 | bwd_inner_microstep: 5715.81 | bwd_allreduce_microstep: 24.70 | step_microstep: 18.61
[2025-04-25 17:39:42,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.92 | bwd: 5740.57 | bwd_inner: 5715.81 | bwd_allreduce: 24.72 | step: 18.61

 10%|▉         | 4023/41250 [9:42:07<89:48:22,  8.68s/it]
                                                         
{'loss': 0.2686, 'grad_norm': 1.6197928190231323, 'learning_rate': 3.952374622649204e-05, 'epoch': 0.98}

 10%|▉         | 4023/41250 [9:42:07<89:48:22,  8.68s/it][2025-04-25 17:39:51,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:39:51,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.86 | bwd_microstep: 5799.61 | bwd_inner_microstep: 5664.42 | bwd_allreduce_microstep: 135.15 | step_microstep: 18.39
[2025-04-25 17:39:51,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.86 | bwd: 5799.63 | bwd_inner: 5664.42 | bwd_allreduce: 135.16 | step: 18.39

 10%|▉         | 4024/41250 [9:42:16<89:55:25,  8.70s/it]
                                                         
{'loss': 0.0915, 'grad_norm': 0.9263080358505249, 'learning_rate': 3.9523405516716134e-05, 'epoch': 0.98}

 10%|▉         | 4024/41250 [9:42:16<89:55:25,  8.70s/it][2025-04-25 17:40:00,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:40:00,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.60 | bwd_microstep: 5800.89 | bwd_inner_microstep: 5662.54 | bwd_allreduce_microstep: 138.30 | step_microstep: 18.37
[2025-04-25 17:40:00,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.60 | bwd: 5800.90 | bwd_inner: 5662.54 | bwd_allreduce: 138.32 | step: 18.37

 10%|▉         | 4025/41250 [9:42:25<90:01:19,  8.71s/it]
                                                         
{'loss': 0.2015, 'grad_norm': 1.5517821311950684, 'learning_rate': 3.9523064686582284e-05, 'epoch': 0.98}

 10%|▉         | 4025/41250 [9:42:25<90:01:19,  8.71s/it][2025-04-25 17:40:08,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 17:40:08,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.20 | bwd_microstep: 5769.46 | bwd_inner_microstep: 5707.25 | bwd_allreduce_microstep: 62.17 | step_microstep: 18.91
[2025-04-25 17:40:08,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.20 | bwd: 5769.48 | bwd_inner: 5707.25 | bwd_allreduce: 62.19 | step: 18.92

 10%|▉         | 4026/41250 [9:42:34<90:02:03,  8.71s/it]
                                                         
{'loss': 0.1905, 'grad_norm': 2.170687437057495, 'learning_rate': 3.952272373609258e-05, 'epoch': 0.98}

 10%|▉         | 4026/41250 [9:42:34<90:02:03,  8.71s/it][2025-04-25 17:40:17,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:40:17,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.51 | bwd_microstep: 5799.16 | bwd_inner_microstep: 5654.58 | bwd_allreduce_microstep: 144.53 | step_microstep: 18.35
[2025-04-25 17:40:17,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.51 | bwd: 5799.17 | bwd_inner: 5654.58 | bwd_allreduce: 144.55 | step: 18.36

 10%|▉         | 4027/41250 [9:42:42<90:03:12,  8.71s/it]
                                                         
{'loss': 0.0445, 'grad_norm': 0.7923468947410583, 'learning_rate': 3.952238266524912e-05, 'epoch': 0.98}

 10%|▉         | 4027/41250 [9:42:42<90:03:12,  8.71s/it][2025-04-25 17:40:26,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.11 | optimizer_step: 1.11
[2025-04-25 17:40:26,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.45 | bwd_microstep: 5918.87 | bwd_inner_microstep: 5664.11 | bwd_allreduce_microstep: 254.70 | step_microstep: 20.12
[2025-04-25 17:40:26,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.45 | bwd: 5918.89 | bwd_inner: 5664.11 | bwd_allreduce: 254.73 | step: 20.13

 10%|▉         | 4028/41250 [9:42:51<90:27:06,  8.75s/it]
                                                         
{'loss': 0.2354, 'grad_norm': 1.2257226705551147, 'learning_rate': 3.9522041474054014e-05, 'epoch': 0.98}

 10%|▉         | 4028/41250 [9:42:51<90:27:06,  8.75s/it][2025-04-25 17:40:34,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:40:34,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.84 | bwd_microstep: 5711.36 | bwd_inner_microstep: 5666.10 | bwd_allreduce_microstep: 45.21 | step_microstep: 18.74
[2025-04-25 17:40:34,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.84 | bwd: 5711.37 | bwd_inner: 5666.10 | bwd_allreduce: 45.23 | step: 18.75

 10%|▉         | 4029/41250 [9:43:00<90:06:36,  8.72s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 2.2092137336730957, 'learning_rate': 3.952170016250936e-05, 'epoch': 0.98}

 10%|▉         | 4029/41250 [9:43:00<90:06:36,  8.72s/it][2025-04-25 17:40:43,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 17:40:43,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.79 | bwd_microstep: 5763.87 | bwd_inner_microstep: 5665.89 | bwd_allreduce_microstep: 97.94 | step_microstep: 18.56
[2025-04-25 17:40:43,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.79 | bwd: 5763.89 | bwd_inner: 5665.89 | bwd_allreduce: 97.95 | step: 18.56

 10%|▉         | 4030/41250 [9:43:08<90:02:15,  8.71s/it]
                                                         
{'loss': 0.1183, 'grad_norm': 1.1747803688049316, 'learning_rate': 3.952135873061726e-05, 'epoch': 0.98}

 10%|▉         | 4030/41250 [9:43:08<90:02:15,  8.71s/it][2025-04-25 17:40:52,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:40:52,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5786.26 | bwd_inner_microstep: 5655.94 | bwd_allreduce_microstep: 130.27 | step_microstep: 18.37
[2025-04-25 17:40:52,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5786.27 | bwd_inner: 5655.94 | bwd_allreduce: 130.29 | step: 18.37

 10%|▉         | 4031/41250 [9:43:17<89:59:56,  8.71s/it]
                                                         
{'loss': 0.1697, 'grad_norm': 1.5854339599609375, 'learning_rate': 3.9521017178379836e-05, 'epoch': 0.98}

 10%|▉         | 4031/41250 [9:43:17<89:59:56,  8.71s/it][2025-04-25 17:41:01,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:41:01,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5789.49 | bwd_inner_microstep: 5710.24 | bwd_allreduce_microstep: 79.21 | step_microstep: 18.26
[2025-04-25 17:41:01,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5789.51 | bwd_inner: 5710.24 | bwd_allreduce: 79.22 | step: 18.26

 10%|▉         | 4032/41250 [9:43:26<90:04:35,  8.71s/it]
                                                         
{'loss': 0.2041, 'grad_norm': 1.5868945121765137, 'learning_rate': 3.952067550579917e-05, 'epoch': 0.98}

 10%|▉         | 4032/41250 [9:43:26<90:04:35,  8.71s/it][2025-04-25 17:41:09,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 17:41:09,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.24 | bwd_microstep: 5720.04 | bwd_inner_microstep: 5707.30 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.55
[2025-04-25 17:41:09,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.24 | bwd: 5720.06 | bwd_inner: 5707.30 | bwd_allreduce: 12.71 | step: 18.55

 10%|▉         | 4033/41250 [9:43:35<89:58:14,  8.70s/it]
                                                         
{'loss': 0.1335, 'grad_norm': 1.4804232120513916, 'learning_rate': 3.952033371287739e-05, 'epoch': 0.98}

 10%|▉         | 4033/41250 [9:43:35<89:58:14,  8.70s/it][2025-04-25 17:41:18,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.16 | optimizer_step: 0.89
[2025-04-25 17:41:18,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.42 | bwd_microstep: 5775.69 | bwd_inner_microstep: 5645.23 | bwd_allreduce_microstep: 130.42 | step_microstep: 19.65
[2025-04-25 17:41:18,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.42 | bwd: 5775.70 | bwd_inner: 5645.23 | bwd_allreduce: 130.43 | step: 19.66

 10%|▉         | 4034/41250 [9:43:43<89:58:06,  8.70s/it]
                                                         
{'loss': 0.0762, 'grad_norm': 1.1412956714630127, 'learning_rate': 3.951999179961658e-05, 'epoch': 0.98}

 10%|▉         | 4034/41250 [9:43:43<89:58:06,  8.70s/it][2025-04-25 17:41:27,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 17:41:27,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.44 | bwd_microstep: 5745.32 | bwd_inner_microstep: 5721.44 | bwd_allreduce_microstep: 23.84 | step_microstep: 18.81
[2025-04-25 17:41:27,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.44 | bwd: 5745.33 | bwd_inner: 5721.44 | bwd_allreduce: 23.85 | step: 18.81

 10%|▉         | 4035/41250 [9:43:52<89:55:32,  8.70s/it]
                                                         
{'loss': 0.0756, 'grad_norm': 1.0607004165649414, 'learning_rate': 3.951964976601886e-05, 'epoch': 0.98}

 10%|▉         | 4035/41250 [9:43:52<89:55:32,  8.70s/it][2025-04-25 17:41:36,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.25 | optimizer_step: 0.94
[2025-04-25 17:41:36,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.18 | bwd_microstep: 5952.02 | bwd_inner_microstep: 5666.41 | bwd_allreduce_microstep: 285.55 | step_microstep: 19.40
[2025-04-25 17:41:36,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.18 | bwd: 5952.03 | bwd_inner: 5666.41 | bwd_allreduce: 285.58 | step: 19.41

 10%|▉         | 4036/41250 [9:44:01<90:26:21,  8.75s/it]
                                                         
{'loss': 0.3671, 'grad_norm': 2.1515989303588867, 'learning_rate': 3.951930761208635e-05, 'epoch': 0.98}

 10%|▉         | 4036/41250 [9:44:01<90:26:21,  8.75s/it][2025-04-25 17:41:44,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:41:44,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5758.32 | bwd_inner_microstep: 5691.03 | bwd_allreduce_microstep: 67.25 | step_microstep: 18.67
[2025-04-25 17:41:44,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5758.33 | bwd_inner: 5691.03 | bwd_allreduce: 67.27 | step: 18.67

 10%|▉         | 4037/41250 [9:44:10<90:16:03,  8.73s/it]
                                                         
{'loss': 0.0714, 'grad_norm': 0.6519840955734253, 'learning_rate': 3.951896533782114e-05, 'epoch': 0.98}

 10%|▉         | 4037/41250 [9:44:10<90:16:03,  8.73s/it][2025-04-25 17:41:53,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 17:41:53,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.42 | bwd_microstep: 5771.07 | bwd_inner_microstep: 5650.39 | bwd_allreduce_microstep: 120.63 | step_microstep: 19.12
[2025-04-25 17:41:53,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.42 | bwd: 5771.08 | bwd_inner: 5650.39 | bwd_allreduce: 120.65 | step: 19.12

 10%|▉         | 4038/41250 [9:44:18<90:05:44,  8.72s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 0.8818817138671875, 'learning_rate': 3.951862294322535e-05, 'epoch': 0.98}

 10%|▉         | 4038/41250 [9:44:18<90:05:44,  8.72s/it][2025-04-25 17:42:02,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 17:42:02,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.44 | bwd_microstep: 5709.41 | bwd_inner_microstep: 5696.20 | bwd_allreduce_microstep: 13.16 | step_microstep: 18.74
[2025-04-25 17:42:02,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.44 | bwd: 5709.42 | bwd_inner: 5696.20 | bwd_allreduce: 13.18 | step: 18.74

 10%|▉         | 4039/41250 [9:44:27<89:53:12,  8.70s/it]
                                                         
{'loss': 0.0911, 'grad_norm': 1.2320104837417603, 'learning_rate': 3.951828042830109e-05, 'epoch': 0.98}

 10%|▉         | 4039/41250 [9:44:27<89:53:12,  8.70s/it][2025-04-25 17:42:10,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.08 | optimizer_step: 1.08
[2025-04-25 17:42:10,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.41 | bwd_microstep: 5941.76 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 284.16 | step_microstep: 19.64
[2025-04-25 17:42:10,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.41 | bwd: 5941.78 | bwd_inner: 5657.55 | bwd_allreduce: 284.18 | step: 19.64

 10%|▉         | 4040/41250 [9:44:36<90:23:02,  8.74s/it]
                                                         
{'loss': 0.0568, 'grad_norm': 0.5087231397628784, 'learning_rate': 3.951793779305046e-05, 'epoch': 0.98}

 10%|▉         | 4040/41250 [9:44:36<90:23:02,  8.74s/it][2025-04-25 17:42:19,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-25 17:42:19,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.24 | bwd_microstep: 5725.79 | bwd_inner_microstep: 5697.94 | bwd_allreduce_microstep: 27.80 | step_microstep: 18.93
[2025-04-25 17:42:19,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.24 | bwd: 5725.81 | bwd_inner: 5697.94 | bwd_allreduce: 27.82 | step: 18.95

 10%|▉         | 4041/41250 [9:44:44<90:10:02,  8.72s/it]
                                                         
{'loss': 0.0929, 'grad_norm': 1.3637416362762451, 'learning_rate': 3.9517595037475586e-05, 'epoch': 0.98}

 10%|▉         | 4041/41250 [9:44:44<90:10:02,  8.72s/it][2025-04-25 17:42:28,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:42:28,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.50 | bwd_microstep: 5848.30 | bwd_inner_microstep: 5699.98 | bwd_allreduce_microstep: 148.27 | step_microstep: 18.92
[2025-04-25 17:42:28,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.50 | bwd: 5848.31 | bwd_inner: 5699.98 | bwd_allreduce: 148.29 | step: 18.93

 10%|▉         | 4042/41250 [9:44:53<90:21:11,  8.74s/it]
                                                         
{'loss': 0.1434, 'grad_norm': 0.9836834073066711, 'learning_rate': 3.9517252161578586e-05, 'epoch': 0.98}

 10%|▉         | 4042/41250 [9:44:53<90:21:11,  8.74s/it][2025-04-25 17:42:37,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 17:42:37,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.17 | bwd_microstep: 5774.81 | bwd_inner_microstep: 5645.68 | bwd_allreduce_microstep: 129.07 | step_microstep: 19.15
[2025-04-25 17:42:37,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.17 | bwd: 5774.82 | bwd_inner: 5645.68 | bwd_allreduce: 129.09 | step: 19.15

 10%|▉         | 4043/41250 [9:45:02<90:10:09,  8.72s/it]
                                                         
{'loss': 0.2642, 'grad_norm': 3.5559301376342773, 'learning_rate': 3.951690916536156e-05, 'epoch': 0.98}

 10%|▉         | 4043/41250 [9:45:02<90:10:09,  8.72s/it][2025-04-25 17:42:45,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.91 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 17:42:45,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.01 | bwd_microstep: 5735.67 | bwd_inner_microstep: 5696.69 | bwd_allreduce_microstep: 38.94 | step_microstep: 19.59
[2025-04-25 17:42:45,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.01 | bwd: 5735.69 | bwd_inner: 5696.69 | bwd_allreduce: 38.95 | step: 19.59

 10%|▉         | 4044/41250 [9:45:11<89:58:12,  8.71s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.5883569717407227, 'learning_rate': 3.9516566048826625e-05, 'epoch': 0.98}

 10%|▉         | 4044/41250 [9:45:11<89:58:12,  8.71s/it][2025-04-25 17:42:54,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:42:54,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5676.42 | bwd_inner_microstep: 5644.65 | bwd_allreduce_microstep: 31.72 | step_microstep: 18.62
[2025-04-25 17:42:54,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5676.43 | bwd_inner: 5644.65 | bwd_allreduce: 31.74 | step: 18.62

 10%|▉         | 4045/41250 [9:45:19<89:36:38,  8.67s/it]
                                                         
{'loss': 0.2112, 'grad_norm': 1.807154655456543, 'learning_rate': 3.951622281197591e-05, 'epoch': 0.98}

 10%|▉         | 4045/41250 [9:45:19<89:36:38,  8.67s/it][2025-04-25 17:43:03,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 17:43:03,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.98 | bwd_microstep: 5794.13 | bwd_inner_microstep: 5781.75 | bwd_allreduce_microstep: 12.34 | step_microstep: 18.85
[2025-04-25 17:43:03,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.98 | bwd: 5794.14 | bwd_inner: 5781.75 | bwd_allreduce: 12.35 | step: 18.85

 10%|▉         | 4046/41250 [9:45:28<89:54:26,  8.70s/it]
                                                         
{'loss': 0.1895, 'grad_norm': 2.9618523120880127, 'learning_rate': 3.95158794548115e-05, 'epoch': 0.98}

 10%|▉         | 4046/41250 [9:45:28<89:54:26,  8.70s/it][2025-04-25 17:43:11,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 17:43:11,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5763.63 | bwd_inner_microstep: 5655.57 | bwd_allreduce_microstep: 108.01 | step_microstep: 18.86
[2025-04-25 17:43:11,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5763.64 | bwd_inner: 5655.57 | bwd_allreduce: 108.03 | step: 18.86

 10%|▉         | 4047/41250 [9:45:37<89:50:37,  8.69s/it]
                                                         
{'loss': 0.1328, 'grad_norm': 1.344544529914856, 'learning_rate': 3.951553597733555e-05, 'epoch': 0.98}

 10%|▉         | 4047/41250 [9:45:37<89:50:37,  8.69s/it][2025-04-25 17:43:20,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.75 | optimizer_gradients: 1.31 | optimizer_step: 0.93
[2025-04-25 17:43:20,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.40 | bwd_microstep: 5712.34 | bwd_inner_microstep: 5699.61 | bwd_allreduce_microstep: 12.69 | step_microstep: 21.26
[2025-04-25 17:43:20,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.40 | bwd: 5712.36 | bwd_inner: 5699.61 | bwd_allreduce: 12.71 | step: 21.26

 10%|▉         | 4048/41250 [9:45:45<89:43:14,  8.68s/it]
                                                         
{'loss': 0.2953, 'grad_norm': 2.4205100536346436, 'learning_rate': 3.951519237955015e-05, 'epoch': 0.98}

 10%|▉         | 4048/41250 [9:45:45<89:43:14,  8.68s/it][2025-04-25 17:43:29,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 17:43:29,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5736.03 | bwd_inner_microstep: 5648.93 | bwd_allreduce_microstep: 87.05 | step_microstep: 19.36
[2025-04-25 17:43:29,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5736.05 | bwd_inner: 5648.93 | bwd_allreduce: 87.07 | step: 19.36

 10%|▉         | 4049/41250 [9:45:54<89:37:22,  8.67s/it]
                                                         
{'loss': 0.1022, 'grad_norm': 2.0084705352783203, 'learning_rate': 3.9514848661457435e-05, 'epoch': 0.98}

 10%|▉         | 4049/41250 [9:45:54<89:37:22,  8.67s/it][2025-04-25 17:43:37,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:43:37,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.64 | bwd_microstep: 5720.67 | bwd_inner_microstep: 5707.37 | bwd_allreduce_microstep: 13.26 | step_microstep: 18.65
[2025-04-25 17:43:37,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.64 | bwd: 5720.68 | bwd_inner: 5707.37 | bwd_allreduce: 13.28 | step: 18.66

 10%|▉         | 4050/41250 [9:46:03<89:33:43,  8.67s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.3371943235397339, 'learning_rate': 3.9514504823059516e-05, 'epoch': 0.98}

 10%|▉         | 4050/41250 [9:46:03<89:33:43,  8.67s/it][2025-04-25 17:43:46,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:43:46,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.76 | bwd_microstep: 5759.47 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 96.21 | step_microstep: 18.59
[2025-04-25 17:43:46,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.76 | bwd: 5759.48 | bwd_inner: 5663.21 | bwd_allreduce: 96.23 | step: 18.59

 10%|▉         | 4051/41250 [9:46:11<89:34:43,  8.67s/it]
                                                         
{'loss': 0.2794, 'grad_norm': 2.0146234035491943, 'learning_rate': 3.951416086435851e-05, 'epoch': 0.98}

 10%|▉         | 4051/41250 [9:46:11<89:34:43,  8.67s/it][2025-04-25 17:43:54,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 17:43:54,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.74 | bwd_microstep: 5681.59 | bwd_inner_microstep: 5664.94 | bwd_allreduce_microstep: 16.61 | step_microstep: 19.36
[2025-04-25 17:43:54,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.74 | bwd: 5681.61 | bwd_inner: 5664.94 | bwd_allreduce: 16.62 | step: 19.36

 10%|▉         | 4052/41250 [9:46:20<89:21:25,  8.65s/it]
                                                         
{'loss': 0.2114, 'grad_norm': 0.9907561540603638, 'learning_rate': 3.951381678535653e-05, 'epoch': 0.98}

 10%|▉         | 4052/41250 [9:46:20<89:21:25,  8.65s/it][2025-04-25 17:44:03,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 17:44:03,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.56 | bwd_microstep: 5689.78 | bwd_inner_microstep: 5653.38 | bwd_allreduce_microstep: 36.36 | step_microstep: 18.65
[2025-04-25 17:44:03,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.56 | bwd: 5689.80 | bwd_inner: 5653.38 | bwd_allreduce: 36.37 | step: 18.65

 10%|▉         | 4053/41250 [9:46:28<89:12:38,  8.63s/it]
                                                         
{'loss': 0.194, 'grad_norm': 1.8810539245605469, 'learning_rate': 3.9513472586055724e-05, 'epoch': 0.98}

 10%|▉         | 4053/41250 [9:46:28<89:12:38,  8.63s/it][2025-04-25 17:44:12,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:44:12,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5689.83 | bwd_inner_microstep: 5659.42 | bwd_allreduce_microstep: 30.36 | step_microstep: 18.62
[2025-04-25 17:44:12,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5689.84 | bwd_inner: 5659.42 | bwd_allreduce: 30.38 | step: 18.62

 10%|▉         | 4054/41250 [9:46:37<89:06:10,  8.62s/it]
                                                         
{'loss': 0.1957, 'grad_norm': 1.8561668395996094, 'learning_rate': 3.951312826645819e-05, 'epoch': 0.98}

 10%|▉         | 4054/41250 [9:46:37<89:06:10,  8.62s/it][2025-04-25 17:44:20,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:44:20,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5760.52 | bwd_inner_microstep: 5658.23 | bwd_allreduce_microstep: 102.25 | step_microstep: 18.55
[2025-04-25 17:44:20,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5760.53 | bwd_inner: 5658.23 | bwd_allreduce: 102.26 | step: 18.55

 10%|▉         | 4055/41250 [9:46:46<89:14:29,  8.64s/it]
                                                         
{'loss': 0.1111, 'grad_norm': 1.091292142868042, 'learning_rate': 3.9512783826566055e-05, 'epoch': 0.98}

 10%|▉         | 4055/41250 [9:46:46<89:14:29,  8.64s/it][2025-04-25 17:44:29,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:44:29,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.61 | bwd_microstep: 5712.31 | bwd_inner_microstep: 5699.67 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.68
[2025-04-25 17:44:29,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.61 | bwd: 5712.32 | bwd_inner: 5699.67 | bwd_allreduce: 12.61 | step: 18.68

 10%|▉         | 4056/41250 [9:46:54<89:14:43,  8.64s/it]
                                                         
{'loss': 0.2042, 'grad_norm': 1.7363026142120361, 'learning_rate': 3.951243926638145e-05, 'epoch': 0.98}

 10%|▉         | 4056/41250 [9:46:54<89:14:43,  8.64s/it][2025-04-25 17:44:38,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 17:44:38,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5706.12 | bwd_inner_microstep: 5693.32 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.18
[2025-04-25 17:44:38,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5706.13 | bwd_inner: 5693.32 | bwd_allreduce: 12.77 | step: 19.18

 10%|▉         | 4057/41250 [9:47:03<89:14:46,  8.64s/it]
                                                         
{'loss': 0.3022, 'grad_norm': 2.280815362930298, 'learning_rate': 3.95120945859065e-05, 'epoch': 0.98}

 10%|▉         | 4057/41250 [9:47:03<89:14:46,  8.64s/it][2025-04-25 17:44:47,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.24 | optimizer_step: 0.93
[2025-04-25 17:44:47,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.78 | bwd_microstep: 6022.82 | bwd_inner_microstep: 5645.38 | bwd_allreduce_microstep: 377.39 | step_microstep: 18.94
[2025-04-25 17:44:47,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 6022.84 | bwd_inner: 5645.38 | bwd_allreduce: 377.41 | step: 18.94

 10%|▉         | 4058/41250 [9:47:12<90:09:45,  8.73s/it]
                                                         
{'loss': 0.3192, 'grad_norm': 3.9002304077148438, 'learning_rate': 3.9511749785143317e-05, 'epoch': 0.98}

 10%|▉         | 4058/41250 [9:47:12<90:09:45,  8.73s/it][2025-04-25 17:44:55,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.97
[2025-04-25 17:44:55,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.01 | bwd_microstep: 5756.02 | bwd_inner_microstep: 5653.58 | bwd_allreduce_microstep: 102.39 | step_microstep: 18.19
[2025-04-25 17:44:55,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.01 | bwd: 5756.04 | bwd_inner: 5653.58 | bwd_allreduce: 102.41 | step: 18.20

 10%|▉         | 4059/41250 [9:47:21<89:58:20,  8.71s/it]
                                                         
{'loss': 0.2239, 'grad_norm': 1.8718340396881104, 'learning_rate': 3.9511404864094034e-05, 'epoch': 0.98}

 10%|▉         | 4059/41250 [9:47:21<89:58:20,  8.71s/it][2025-04-25 17:45:04,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:45:04,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5870.15 | bwd_inner_microstep: 5654.48 | bwd_allreduce_microstep: 215.63 | step_microstep: 18.50
[2025-04-25 17:45:04,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5870.17 | bwd_inner: 5654.48 | bwd_allreduce: 215.64 | step: 18.50

 10%|▉         | 4060/41250 [9:47:29<90:11:13,  8.73s/it]
                                                         
{'loss': 0.1924, 'grad_norm': 1.8256254196166992, 'learning_rate': 3.951105982276078e-05, 'epoch': 0.98}

 10%|▉         | 4060/41250 [9:47:29<90:11:13,  8.73s/it][2025-04-25 17:45:13,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:45:13,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.15 | bwd_microstep: 5744.94 | bwd_inner_microstep: 5706.96 | bwd_allreduce_microstep: 37.93 | step_microstep: 18.60
[2025-04-25 17:45:13,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.15 | bwd: 5744.95 | bwd_inner: 5706.96 | bwd_allreduce: 37.95 | step: 18.60

 10%|▉         | 4061/41250 [9:47:38<90:00:57,  8.71s/it]
                                                         
{'loss': 0.0749, 'grad_norm': 1.278847336769104, 'learning_rate': 3.9510714661145684e-05, 'epoch': 0.98}

 10%|▉         | 4061/41250 [9:47:38<90:00:57,  8.71s/it][2025-04-25 17:45:21,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:45:21,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5692.44 | bwd_inner_microstep: 5650.08 | bwd_allreduce_microstep: 42.32 | step_microstep: 18.62
[2025-04-25 17:45:21,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 5692.46 | bwd_inner: 5650.08 | bwd_allreduce: 42.34 | step: 18.61

 10%|▉         | 4062/41250 [9:47:47<89:39:22,  8.68s/it]
                                                         
{'loss': 0.1091, 'grad_norm': 0.9908390641212463, 'learning_rate': 3.9510369379250866e-05, 'epoch': 0.98}

 10%|▉         | 4062/41250 [9:47:47<89:39:22,  8.68s/it][2025-04-25 17:45:30,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:45:30,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.48 | bwd_microstep: 5749.90 | bwd_inner_microstep: 5660.92 | bwd_allreduce_microstep: 88.93 | step_microstep: 18.53
[2025-04-25 17:45:30,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.48 | bwd: 5749.92 | bwd_inner: 5660.92 | bwd_allreduce: 88.95 | step: 18.53

 10%|▉         | 4063/41250 [9:47:55<89:37:31,  8.68s/it]
                                                         
{'loss': 0.1525, 'grad_norm': 1.5874171257019043, 'learning_rate': 3.9510023977078453e-05, 'epoch': 0.98}

 10%|▉         | 4063/41250 [9:47:55<89:37:31,  8.68s/it][2025-04-25 17:45:39,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.17 | optimizer_step: 0.91
[2025-04-25 17:45:39,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.75 | bwd_microstep: 5767.78 | bwd_inner_microstep: 5652.00 | bwd_allreduce_microstep: 115.72 | step_microstep: 19.01
[2025-04-25 17:45:39,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.75 | bwd: 5767.80 | bwd_inner: 5652.00 | bwd_allreduce: 115.74 | step: 19.01

 10%|▉         | 4064/41250 [9:48:04<89:37:03,  8.68s/it]
                                                         
{'loss': 0.0585, 'grad_norm': 1.2028920650482178, 'learning_rate': 3.950967845463059e-05, 'epoch': 0.99}

 10%|▉         | 4064/41250 [9:48:04<89:37:03,  8.68s/it][2025-04-25 17:45:47,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 17:45:47,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.21 | bwd_microstep: 5760.62 | bwd_inner_microstep: 5710.80 | bwd_allreduce_microstep: 49.76 | step_microstep: 19.98
[2025-04-25 17:45:47,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.21 | bwd: 5760.64 | bwd_inner: 5710.80 | bwd_allreduce: 49.78 | step: 19.98

 10%|▉         | 4065/41250 [9:48:13<89:41:37,  8.68s/it]
                                                         
{'loss': 0.3138, 'grad_norm': 1.9356013536453247, 'learning_rate': 3.950933281190939e-05, 'epoch': 0.99}

 10%|▉         | 4065/41250 [9:48:13<89:41:37,  8.68s/it][2025-04-25 17:45:56,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 17:45:56,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5702.42 | bwd_inner_microstep: 5650.78 | bwd_allreduce_microstep: 51.59 | step_microstep: 18.77
[2025-04-25 17:45:56,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.82 | bwd: 5702.43 | bwd_inner: 5650.78 | bwd_allreduce: 51.61 | step: 18.78

 10%|▉         | 4066/41250 [9:48:21<89:28:56,  8.66s/it]
                                                         
{'loss': 0.1678, 'grad_norm': 1.7326555252075195, 'learning_rate': 3.950898704891699e-05, 'epoch': 0.99}

 10%|▉         | 4066/41250 [9:48:21<89:28:56,  8.66s/it][2025-04-25 17:46:05,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 17:46:05,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.84 | bwd_microstep: 5739.02 | bwd_inner_microstep: 5709.15 | bwd_allreduce_microstep: 29.83 | step_microstep: 18.33
[2025-04-25 17:46:05,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.84 | bwd: 5739.04 | bwd_inner: 5709.15 | bwd_allreduce: 29.84 | step: 18.33

 10%|▉         | 4067/41250 [9:48:30<89:31:44,  8.67s/it]
                                                         
{'loss': 0.1607, 'grad_norm': 0.7857925295829773, 'learning_rate': 3.950864116565552e-05, 'epoch': 0.99}

 10%|▉         | 4067/41250 [9:48:30<89:31:44,  8.67s/it][2025-04-25 17:46:13,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 17:46:13,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.84 | bwd_microstep: 5739.89 | bwd_inner_microstep: 5696.85 | bwd_allreduce_microstep: 42.99 | step_microstep: 18.87
[2025-04-25 17:46:13,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.84 | bwd: 5739.91 | bwd_inner: 5696.85 | bwd_allreduce: 43.01 | step: 18.87

 10%|▉         | 4068/41250 [9:48:39<89:34:08,  8.67s/it]
                                                         
{'loss': 0.0561, 'grad_norm': 0.6525914072990417, 'learning_rate': 3.950829516212712e-05, 'epoch': 0.99}

 10%|▉         | 4068/41250 [9:48:39<89:34:08,  8.67s/it][2025-04-25 17:46:22,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 17:46:22,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.75 | bwd_microstep: 5790.75 | bwd_inner_microstep: 5662.09 | bwd_allreduce_microstep: 128.61 | step_microstep: 18.58
[2025-04-25 17:46:22,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.75 | bwd: 5790.76 | bwd_inner: 5662.09 | bwd_allreduce: 128.63 | step: 18.58

 10%|▉         | 4069/41250 [9:48:47<89:40:10,  8.68s/it]
                                                         
{'loss': 0.042, 'grad_norm': 0.8401353359222412, 'learning_rate': 3.9507949038333915e-05, 'epoch': 0.99}

 10%|▉         | 4069/41250 [9:48:47<89:40:10,  8.68s/it][2025-04-25 17:46:31,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 17:46:31,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.64 | bwd_microstep: 5726.79 | bwd_inner_microstep: 5664.16 | bwd_allreduce_microstep: 62.58 | step_microstep: 18.79
[2025-04-25 17:46:31,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.64 | bwd: 5726.81 | bwd_inner: 5664.16 | bwd_allreduce: 62.60 | step: 18.79

 10%|▉         | 4070/41250 [9:48:56<89:32:56,  8.67s/it]
                                                         
{'loss': 0.0596, 'grad_norm': 0.6222354769706726, 'learning_rate': 3.950760279427805e-05, 'epoch': 0.99}

 10%|▉         | 4070/41250 [9:48:56<89:32:56,  8.67s/it][2025-04-25 17:46:39,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 17:46:39,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.02 | bwd_microstep: 5775.87 | bwd_inner_microstep: 5700.34 | bwd_allreduce_microstep: 75.49 | step_microstep: 19.71
[2025-04-25 17:46:39,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.02 | bwd: 5775.89 | bwd_inner: 5700.34 | bwd_allreduce: 75.51 | step: 19.71

 10%|▉         | 4071/41250 [9:49:05<89:40:53,  8.68s/it]
                                                         
{'loss': 0.2665, 'grad_norm': 3.291006088256836, 'learning_rate': 3.9507256429961635e-05, 'epoch': 0.99}

 10%|▉         | 4071/41250 [9:49:05<89:40:53,  8.68s/it][2025-04-25 17:46:48,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 17:46:48,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5760.12 | bwd_inner_microstep: 5696.97 | bwd_allreduce_microstep: 63.10 | step_microstep: 18.91
[2025-04-25 17:46:48,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5760.13 | bwd_inner: 5696.97 | bwd_allreduce: 63.12 | step: 18.91

 10%|▉         | 4072/41250 [9:49:13<89:42:07,  8.69s/it]
                                                         
{'loss': 0.0216, 'grad_norm': 0.29674041271209717, 'learning_rate': 3.950690994538683e-05, 'epoch': 0.99}

 10%|▉         | 4072/41250 [9:49:13<89:42:07,  8.69s/it][2025-04-25 17:46:57,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:46:57,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.77 | bwd_microstep: 5876.52 | bwd_inner_microstep: 5721.06 | bwd_allreduce_microstep: 155.41 | step_microstep: 18.85
[2025-04-25 17:46:57,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.77 | bwd: 5876.53 | bwd_inner: 5721.06 | bwd_allreduce: 155.43 | step: 18.85

 10%|▉         | 4073/41250 [9:49:22<90:07:28,  8.73s/it]
                                                         
{'loss': 0.2235, 'grad_norm': 1.9734833240509033, 'learning_rate': 3.950656334055576e-05, 'epoch': 0.99}

 10%|▉         | 4073/41250 [9:49:22<90:07:28,  8.73s/it][2025-04-25 17:47:06,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:47:06,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.24 | bwd_microstep: 5725.63 | bwd_inner_microstep: 5712.80 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.46
[2025-04-25 17:47:06,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.24 | bwd: 5725.64 | bwd_inner: 5712.80 | bwd_allreduce: 12.80 | step: 18.47

 10%|▉         | 4074/41250 [9:49:31<89:55:11,  8.71s/it]
                                                         
{'loss': 0.3686, 'grad_norm': 1.770984172821045, 'learning_rate': 3.950621661547057e-05, 'epoch': 0.99}

 10%|▉         | 4074/41250 [9:49:31<89:55:11,  8.71s/it][2025-04-25 17:47:14,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:47:14,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.78 | bwd_microstep: 5730.64 | bwd_inner_microstep: 5717.85 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.89
[2025-04-25 17:47:14,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.78 | bwd: 5730.65 | bwd_inner: 5717.85 | bwd_allreduce: 12.77 | step: 18.89

 10%|▉         | 4075/41250 [9:49:40<89:50:33,  8.70s/it]
                                                         
{'loss': 0.0356, 'grad_norm': 0.6463630795478821, 'learning_rate': 3.9505869770133387e-05, 'epoch': 0.99}

 10%|▉         | 4075/41250 [9:49:40<89:50:33,  8.70s/it][2025-04-25 17:47:23,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.75 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:47:23,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5724.61 | bwd_inner_microstep: 5711.48 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.55
[2025-04-25 17:47:23,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5724.62 | bwd_inner: 5711.48 | bwd_allreduce: 13.10 | step: 19.55

 10%|▉         | 4076/41250 [9:49:48<89:43:46,  8.69s/it]
                                                         
{'loss': 0.0437, 'grad_norm': 0.5127654671669006, 'learning_rate': 3.950552280454635e-05, 'epoch': 0.99}

 10%|▉         | 4076/41250 [9:49:48<89:43:46,  8.69s/it][2025-04-25 17:47:32,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:47:32,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.48 | bwd_microstep: 5774.44 | bwd_inner_microstep: 5722.90 | bwd_allreduce_microstep: 51.50 | step_microstep: 18.83
[2025-04-25 17:47:32,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.48 | bwd: 5774.45 | bwd_inner: 5722.90 | bwd_allreduce: 51.52 | step: 18.84

 10%|▉         | 4077/41250 [9:49:57<89:48:13,  8.70s/it]
                                                         
{'loss': 0.2949, 'grad_norm': 3.093329429626465, 'learning_rate': 3.9505175718711604e-05, 'epoch': 0.99}

 10%|▉         | 4077/41250 [9:49:57<89:48:13,  8.70s/it][2025-04-25 17:47:40,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 17:47:40,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.05 | bwd_microstep: 5801.96 | bwd_inner_microstep: 5667.07 | bwd_allreduce_microstep: 134.83 | step_microstep: 18.63
[2025-04-25 17:47:40,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.05 | bwd: 5801.98 | bwd_inner: 5667.07 | bwd_allreduce: 134.85 | step: 18.63

 10%|▉         | 4078/41250 [9:50:06<89:53:15,  8.71s/it]
                                                         
{'loss': 0.2476, 'grad_norm': 1.4240602254867554, 'learning_rate': 3.950482851263128e-05, 'epoch': 0.99}

 10%|▉         | 4078/41250 [9:50:06<89:53:15,  8.71s/it][2025-04-25 17:47:49,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:47:49,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.59 | bwd_microstep: 5771.51 | bwd_inner_microstep: 5695.82 | bwd_allreduce_microstep: 75.65 | step_microstep: 18.49
[2025-04-25 17:47:49,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.59 | bwd: 5771.53 | bwd_inner: 5695.82 | bwd_allreduce: 75.66 | step: 18.49

 10%|▉         | 4079/41250 [9:50:14<89:53:27,  8.71s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.5857733488082886, 'learning_rate': 3.9504481186307535e-05, 'epoch': 0.99}

 10%|▉         | 4079/41250 [9:50:14<89:53:27,  8.71s/it][2025-04-25 17:47:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 17:47:58,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.43 | bwd_microstep: 5713.59 | bwd_inner_microstep: 5700.82 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.38
[2025-04-25 17:47:58,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.43 | bwd: 5713.60 | bwd_inner: 5700.82 | bwd_allreduce: 12.74 | step: 19.38

 10%|▉         | 4080/41250 [9:50:23<89:43:15,  8.69s/it]
                                                         
{'loss': 0.1785, 'grad_norm': 1.3140794038772583, 'learning_rate': 3.950413373974249e-05, 'epoch': 0.99}

 10%|▉         | 4080/41250 [9:50:23<89:43:15,  8.69s/it][2025-04-25 17:48:06,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.82 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:48:06,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.74 | bwd_microstep: 5793.81 | bwd_inner_microstep: 5713.11 | bwd_allreduce_microstep: 80.66 | step_microstep: 20.11
[2025-04-25 17:48:06,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.74 | bwd: 5793.83 | bwd_inner: 5713.11 | bwd_allreduce: 80.68 | step: 20.12

 10%|▉         | 4081/41250 [9:50:32<89:51:54,  8.70s/it]
                                                         
{'loss': 0.0471, 'grad_norm': 0.5731232166290283, 'learning_rate': 3.9503786172938305e-05, 'epoch': 0.99}

 10%|▉         | 4081/41250 [9:50:32<89:51:54,  8.70s/it][2025-04-25 17:48:15,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 17:48:15,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5717.38 | bwd_inner_microstep: 5662.41 | bwd_allreduce_microstep: 54.92 | step_microstep: 19.04
[2025-04-25 17:48:15,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5717.39 | bwd_inner: 5662.41 | bwd_allreduce: 54.94 | step: 19.04

 10%|▉         | 4082/41250 [9:50:40<89:38:44,  8.68s/it]
                                                         
{'loss': 0.2105, 'grad_norm': 3.5962116718292236, 'learning_rate': 3.950343848589711e-05, 'epoch': 0.99}

 10%|▉         | 4082/41250 [9:50:40<89:38:44,  8.68s/it][2025-04-25 17:48:24,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:48:24,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.42 | bwd_microstep: 5737.85 | bwd_inner_microstep: 5725.41 | bwd_allreduce_microstep: 12.39 | step_microstep: 18.60
[2025-04-25 17:48:24,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.42 | bwd: 5737.86 | bwd_inner: 5725.41 | bwd_allreduce: 12.41 | step: 18.61

 10%|▉         | 4083/41250 [9:50:49<89:39:47,  8.68s/it]
                                                         
{'loss': 0.5303, 'grad_norm': 2.8773303031921387, 'learning_rate': 3.9503090678621056e-05, 'epoch': 0.99}

 10%|▉         | 4083/41250 [9:50:49<89:39:47,  8.68s/it][2025-04-25 17:48:32,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:48:32,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5700.25 | bwd_inner_microstep: 5657.41 | bwd_allreduce_microstep: 42.80 | step_microstep: 18.94
[2025-04-25 17:48:32,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5700.26 | bwd_inner: 5657.41 | bwd_allreduce: 42.81 | step: 18.94

 10%|▉         | 4084/41250 [9:50:58<89:26:35,  8.66s/it]
                                                         
{'loss': 0.0921, 'grad_norm': 1.5815436840057373, 'learning_rate': 3.950274275111228e-05, 'epoch': 0.99}

 10%|▉         | 4084/41250 [9:50:58<89:26:35,  8.66s/it][2025-04-25 17:48:41,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:48:41,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.29 | bwd_microstep: 5710.87 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.82
[2025-04-25 17:48:41,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.29 | bwd: 5710.89 | bwd_inner: 5697.92 | bwd_allreduce: 12.93 | step: 18.82

 10%|▉         | 4085/41250 [9:51:06<89:24:04,  8.66s/it]
                                                         
{'loss': 0.1518, 'grad_norm': 1.154945731163025, 'learning_rate': 3.9502394703372934e-05, 'epoch': 0.99}

 10%|▉         | 4085/41250 [9:51:06<89:24:04,  8.66s/it][2025-04-25 17:48:50,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:48:50,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.23 | bwd_microstep: 5772.06 | bwd_inner_microstep: 5664.02 | bwd_allreduce_microstep: 108.00 | step_microstep: 18.54
[2025-04-25 17:48:50,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.23 | bwd: 5772.08 | bwd_inner: 5664.02 | bwd_allreduce: 108.02 | step: 18.54

 10%|▉         | 4086/41250 [9:51:15<89:29:22,  8.67s/it]
                                                         
{'loss': 0.0867, 'grad_norm': 1.1018753051757812, 'learning_rate': 3.950204653540516e-05, 'epoch': 0.99}

 10%|▉         | 4086/41250 [9:51:15<89:29:22,  8.67s/it][2025-04-25 17:48:58,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:48:58,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.84 | bwd_microstep: 5790.94 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 141.61 | step_microstep: 18.47
[2025-04-25 17:48:58,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.84 | bwd: 5790.95 | bwd_inner: 5649.29 | bwd_allreduce: 141.62 | step: 18.47

 10%|▉         | 4087/41250 [9:51:24<89:33:51,  8.68s/it]
                                                         
{'loss': 0.1255, 'grad_norm': 1.4394080638885498, 'learning_rate': 3.95016982472111e-05, 'epoch': 0.99}

 10%|▉         | 4087/41250 [9:51:24<89:33:51,  8.68s/it][2025-04-25 17:49:07,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 17:49:07,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.54 | bwd_microstep: 5711.84 | bwd_inner_microstep: 5659.73 | bwd_allreduce_microstep: 52.07 | step_microstep: 18.94
[2025-04-25 17:49:07,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.55 | bwd: 5711.86 | bwd_inner: 5659.73 | bwd_allreduce: 52.08 | step: 18.94

 10%|▉         | 4088/41250 [9:51:32<89:23:58,  8.66s/it]
                                                         
{'loss': 0.2473, 'grad_norm': 1.6099426746368408, 'learning_rate': 3.950134983879292e-05, 'epoch': 0.99}

 10%|▉         | 4088/41250 [9:51:32<89:23:58,  8.66s/it][2025-04-25 17:49:16,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 1.23 | optimizer_step: 0.91
[2025-04-25 17:49:16,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.11 | bwd_microstep: 5772.01 | bwd_inner_microstep: 5691.80 | bwd_allreduce_microstep: 80.14 | step_microstep: 19.51
[2025-04-25 17:49:16,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.11 | bwd: 5772.03 | bwd_inner: 5691.80 | bwd_allreduce: 80.17 | step: 19.51

 10%|▉         | 4089/41250 [9:51:41<89:31:52,  8.67s/it]
                                                         
{'loss': 0.2631, 'grad_norm': 2.8276777267456055, 'learning_rate': 3.950100131015274e-05, 'epoch': 0.99}

 10%|▉         | 4089/41250 [9:51:41<89:31:52,  8.67s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0xc9dc100] moov atom not found
[17:49:17] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4...
[2025-04-25 17:49:25,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 17:49:25,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.83 | bwd_microstep: 5827.93 | bwd_inner_microstep: 5760.90 | bwd_allreduce_microstep: 66.99 | step_microstep: 18.50
[2025-04-25 17:49:25,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.83 | bwd: 5827.94 | bwd_inner: 5760.90 | bwd_allreduce: 67.00 | step: 18.50

 10%|▉         | 4090/41250 [9:51:50<89:52:38,  8.71s/it]
                                                         
{'loss': 0.2172, 'grad_norm': 1.767427682876587, 'learning_rate': 3.9500652661292725e-05, 'epoch': 0.99}

 10%|▉         | 4090/41250 [9:51:50<89:52:38,  8.71s/it][2025-04-25 17:49:33,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 17:49:33,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.29 | bwd_microstep: 5714.44 | bwd_inner_microstep: 5701.50 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.43
[2025-04-25 17:49:33,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.29 | bwd: 5714.45 | bwd_inner: 5701.50 | bwd_allreduce: 12.91 | step: 19.43

 10%|▉         | 4091/41250 [9:51:58<89:41:57,  8.69s/it]
                                                         
{'loss': 0.3289, 'grad_norm': 2.1806817054748535, 'learning_rate': 3.9500303892215026e-05, 'epoch': 0.99}

 10%|▉         | 4091/41250 [9:51:58<89:41:57,  8.69s/it][2025-04-25 17:49:42,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 17:49:42,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.81 | bwd_microstep: 5779.98 | bwd_inner_microstep: 5654.64 | bwd_allreduce_microstep: 125.30 | step_microstep: 18.48
[2025-04-25 17:49:42,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.81 | bwd: 5780.00 | bwd_inner: 5654.64 | bwd_allreduce: 125.31 | step: 18.49

 10%|▉         | 4092/41250 [9:52:07<89:41:31,  8.69s/it]
                                                         
{'loss': 0.2624, 'grad_norm': 1.485580563545227, 'learning_rate': 3.9499955002921784e-05, 'epoch': 0.99}

 10%|▉         | 4092/41250 [9:52:07<89:41:31,  8.69s/it][2025-04-25 17:49:51,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:49:51,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.25 | bwd_microstep: 5868.73 | bwd_inner_microstep: 5855.93 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.55
[2025-04-25 17:49:51,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.25 | bwd: 5868.74 | bwd_inner: 5855.93 | bwd_allreduce: 12.77 | step: 18.56

 10%|▉         | 4093/41250 [9:52:16<90:18:17,  8.75s/it]
                                                         
{'loss': 0.222, 'grad_norm': 1.654794692993164, 'learning_rate': 3.949960599341516e-05, 'epoch': 0.99}

 10%|▉         | 4093/41250 [9:52:16<90:18:17,  8.75s/it][2025-04-25 17:49:59,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:49:59,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5740.31 | bwd_inner_microstep: 5698.26 | bwd_allreduce_microstep: 42.00 | step_microstep: 18.45
[2025-04-25 17:49:59,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5740.32 | bwd_inner: 5698.26 | bwd_allreduce: 42.02 | step: 18.46

 10%|▉         | 4094/41250 [9:52:25<90:04:36,  8.73s/it]
                                                         
{'loss': 0.1794, 'grad_norm': 1.4607092142105103, 'learning_rate': 3.94992568636973e-05, 'epoch': 0.99}

 10%|▉         | 4094/41250 [9:52:25<90:04:36,  8.73s/it][2025-04-25 17:50:08,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 17:50:08,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.49 | bwd_microstep: 5880.79 | bwd_inner_microstep: 5625.85 | bwd_allreduce_microstep: 254.89 | step_microstep: 18.19
[2025-04-25 17:50:08,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.49 | bwd: 5880.80 | bwd_inner: 5625.85 | bwd_allreduce: 254.91 | step: 18.19

 10%|▉         | 4095/41250 [9:52:34<90:14:33,  8.74s/it]
                                                         
{'loss': 0.1267, 'grad_norm': 1.1727806329727173, 'learning_rate': 3.949890761377035e-05, 'epoch': 0.99}

 10%|▉         | 4095/41250 [9:52:34<90:14:33,  8.74s/it][2025-04-25 17:50:17,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 17:50:17,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5852.12 | bwd_inner_microstep: 5681.77 | bwd_allreduce_microstep: 170.30 | step_microstep: 18.41
[2025-04-25 17:50:17,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5852.13 | bwd_inner: 5681.77 | bwd_allreduce: 170.32 | step: 18.41

 10%|▉         | 4096/41250 [9:52:42<90:21:23,  8.75s/it]
                                                         
{'loss': 0.0713, 'grad_norm': 1.0915939807891846, 'learning_rate': 3.949855824363647e-05, 'epoch': 0.99}

 10%|▉         | 4096/41250 [9:52:42<90:21:23,  8.75s/it][2025-04-25 17:50:26,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:50:26,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.33 | bwd_microstep: 5727.60 | bwd_inner_microstep: 5679.07 | bwd_allreduce_microstep: 48.49 | step_microstep: 18.78
[2025-04-25 17:50:26,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.33 | bwd: 5727.62 | bwd_inner: 5679.07 | bwd_allreduce: 48.50 | step: 18.78

 10%|▉         | 4097/41250 [9:52:51<90:02:50,  8.73s/it]
                                                         
{'loss': 0.1242, 'grad_norm': 1.0683280229568481, 'learning_rate': 3.9498208753297824e-05, 'epoch': 0.99}

 10%|▉         | 4097/41250 [9:52:51<90:02:50,  8.73s/it][2025-04-25 17:50:34,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:50:34,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.38 | bwd_microstep: 5864.65 | bwd_inner_microstep: 5709.67 | bwd_allreduce_microstep: 154.93 | step_microstep: 17.96
[2025-04-25 17:50:34,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.38 | bwd: 5864.67 | bwd_inner: 5709.67 | bwd_allreduce: 154.95 | step: 17.96

 10%|▉         | 4098/41250 [9:53:00<90:16:05,  8.75s/it]
                                                         
{'loss': 0.0509, 'grad_norm': 0.5164527893066406, 'learning_rate': 3.949785914275655e-05, 'epoch': 0.99}

 10%|▉         | 4098/41250 [9:53:00<90:16:05,  8.75s/it][2025-04-25 17:50:43,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 17:50:43,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5701.68 | bwd_inner_microstep: 5689.04 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.66
[2025-04-25 17:50:43,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5701.69 | bwd_inner: 5689.04 | bwd_allreduce: 12.61 | step: 18.67

 10%|▉         | 4099/41250 [9:53:08<89:54:36,  8.71s/it]
                                                         
{'loss': 0.1644, 'grad_norm': 1.3896143436431885, 'learning_rate': 3.949750941201481e-05, 'epoch': 0.99}

 10%|▉         | 4099/41250 [9:53:08<89:54:36,  8.71s/it][2025-04-25 17:50:52,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 17:50:52,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.94 | bwd_microstep: 5697.26 | bwd_inner_microstep: 5639.35 | bwd_allreduce_microstep: 57.85 | step_microstep: 18.48
[2025-04-25 17:50:52,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.94 | bwd: 5697.27 | bwd_inner: 5639.35 | bwd_allreduce: 57.87 | step: 18.48

 10%|▉         | 4100/41250 [9:53:17<89:33:32,  8.68s/it]
                                                         
{'loss': 0.3338, 'grad_norm': 1.9028708934783936, 'learning_rate': 3.949715956107475e-05, 'epoch': 0.99}

 10%|▉         | 4100/41250 [9:53:17<89:33:32,  8.68s/it][2025-04-25 17:51:00,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-25 17:51:00,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.92 | bwd_microstep: 5746.73 | bwd_inner_microstep: 5647.11 | bwd_allreduce_microstep: 99.57 | step_microstep: 19.20
[2025-04-25 17:51:00,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.92 | bwd: 5746.74 | bwd_inner: 5647.11 | bwd_allreduce: 99.59 | step: 19.20

 10%|▉         | 4101/41250 [9:53:26<89:30:27,  8.67s/it]
                                                         
{'loss': 0.1559, 'grad_norm': 1.5157172679901123, 'learning_rate': 3.9496809589938555e-05, 'epoch': 0.99}

 10%|▉         | 4101/41250 [9:53:26<89:30:27,  8.67s/it][2025-04-25 17:51:09,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 17:51:09,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2927.57 | bwd_microstep: 5862.55 | bwd_inner_microstep: 5849.51 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.66
[2025-04-25 17:51:09,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2927.57 | bwd: 5862.56 | bwd_inner: 5849.51 | bwd_allreduce: 13.01 | step: 18.66

 10%|▉         | 4102/41250 [9:53:35<90:07:11,  8.73s/it]
                                                         
{'loss': 0.1005, 'grad_norm': 1.1858302354812622, 'learning_rate': 3.949645949860835e-05, 'epoch': 0.99}

 10%|▉         | 4102/41250 [9:53:35<90:07:11,  8.73s/it][2025-04-25 17:51:18,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.85 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:51:18,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.15 | bwd_microstep: 5755.92 | bwd_inner_microstep: 5640.78 | bwd_allreduce_microstep: 115.10 | step_microstep: 18.21
[2025-04-25 17:51:18,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.15 | bwd: 5755.93 | bwd_inner: 5640.78 | bwd_allreduce: 115.12 | step: 18.22

 10%|▉         | 4103/41250 [9:53:43<89:54:41,  8.71s/it]
                                                         
{'loss': 0.2305, 'grad_norm': 2.5489890575408936, 'learning_rate': 3.9496109287086314e-05, 'epoch': 0.99}

 10%|▉         | 4103/41250 [9:53:43<89:54:41,  8.71s/it][2025-04-25 17:51:26,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:51:26,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5695.54 | bwd_inner_microstep: 5646.38 | bwd_allreduce_microstep: 49.12 | step_microstep: 18.84
[2025-04-25 17:51:26,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.79 | bwd: 5695.55 | bwd_inner: 5646.38 | bwd_allreduce: 49.13 | step: 18.84

 10%|▉         | 4104/41250 [9:53:52<89:34:29,  8.68s/it]
                                                         
{'loss': 0.2889, 'grad_norm': 2.2328271865844727, 'learning_rate': 3.94957589553746e-05, 'epoch': 0.99}

 10%|▉         | 4104/41250 [9:53:52<89:34:29,  8.68s/it][2025-04-25 17:51:35,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 17:51:35,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.88 | bwd_microstep: 5750.52 | bwd_inner_microstep: 5642.64 | bwd_allreduce_microstep: 107.83 | step_microstep: 18.94
[2025-04-25 17:51:35,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.88 | bwd: 5750.54 | bwd_inner: 5642.64 | bwd_allreduce: 107.85 | step: 18.95

 10%|▉         | 4105/41250 [9:54:00<89:29:44,  8.67s/it]
                                                         
{'loss': 0.1998, 'grad_norm': 2.087681770324707, 'learning_rate': 3.949540850347536e-05, 'epoch': 1.0}

 10%|▉         | 4105/41250 [9:54:00<89:29:44,  8.67s/it][2025-04-25 17:51:44,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 17:51:44,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.50 | bwd_microstep: 5702.53 | bwd_inner_microstep: 5689.46 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.83
[2025-04-25 17:51:44,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.51 | bwd: 5702.55 | bwd_inner: 5689.46 | bwd_allreduce: 13.05 | step: 18.84

 10%|▉         | 4106/41250 [9:54:09<89:22:58,  8.66s/it]
                                                         
{'loss': 0.3165, 'grad_norm': 2.126156806945801, 'learning_rate': 3.9495057931390764e-05, 'epoch': 1.0}

 10%|▉         | 4106/41250 [9:54:09<89:22:58,  8.66s/it][2025-04-25 17:51:52,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 17:51:52,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5721.07 | bwd_inner_microstep: 5708.29 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.71
[2025-04-25 17:51:52,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5721.08 | bwd_inner: 5708.29 | bwd_allreduce: 12.75 | step: 18.71

 10%|▉         | 4107/41250 [9:54:18<89:24:09,  8.67s/it]
                                                         
{'loss': 0.1002, 'grad_norm': 1.394290804862976, 'learning_rate': 3.949470723912297e-05, 'epoch': 1.0}

 10%|▉         | 4107/41250 [9:54:18<89:24:09,  8.67s/it][2025-04-25 17:52:01,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-25 17:52:01,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.44 | bwd_microstep: 5720.05 | bwd_inner_microstep: 5706.88 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.17
[2025-04-25 17:52:01,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.44 | bwd: 5720.07 | bwd_inner: 5706.88 | bwd_allreduce: 13.15 | step: 19.17

 10%|▉         | 4108/41250 [9:54:26<89:24:56,  8.67s/it]
                                                         
{'loss': 0.1565, 'grad_norm': 1.9094713926315308, 'learning_rate': 3.9494356426674143e-05, 'epoch': 1.0}

 10%|▉         | 4108/41250 [9:54:26<89:24:56,  8.67s/it][2025-04-25 17:52:10,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.17 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 17:52:10,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.89 | bwd_microstep: 5814.47 | bwd_inner_microstep: 5712.22 | bwd_allreduce_microstep: 102.20 | step_microstep: 19.82
[2025-04-25 17:52:10,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.89 | bwd: 5814.48 | bwd_inner: 5712.22 | bwd_allreduce: 102.22 | step: 19.82

 10%|▉         | 4109/41250 [9:54:35<89:42:55,  8.70s/it]
                                                         
{'loss': 0.1043, 'grad_norm': 1.1032931804656982, 'learning_rate': 3.949400549404644e-05, 'epoch': 1.0}

 10%|▉         | 4109/41250 [9:54:35<89:42:55,  8.70s/it][2025-04-25 17:52:18,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 17:52:18,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.64 | bwd_microstep: 5665.25 | bwd_inner_microstep: 5645.11 | bwd_allreduce_microstep: 20.10 | step_microstep: 18.35
[2025-04-25 17:52:18,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.64 | bwd: 5665.27 | bwd_inner: 5645.11 | bwd_allreduce: 20.11 | step: 18.35

 10%|▉         | 4110/41250 [9:54:44<89:25:38,  8.67s/it]
                                                         
{'loss': 0.1087, 'grad_norm': 1.09033203125, 'learning_rate': 3.9493654441242026e-05, 'epoch': 1.0}

 10%|▉         | 4110/41250 [9:54:44<89:25:38,  8.67s/it][2025-04-25 17:52:27,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 17:52:27,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.03 | bwd_microstep: 5756.53 | bwd_inner_microstep: 5658.89 | bwd_allreduce_microstep: 97.59 | step_microstep: 18.95
[2025-04-25 17:52:27,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.03 | bwd: 5756.54 | bwd_inner: 5658.89 | bwd_allreduce: 97.60 | step: 18.95

 10%|▉         | 4111/41250 [9:54:52<89:27:11,  8.67s/it]
                                                         
{'loss': 0.2342, 'grad_norm': 1.867676854133606, 'learning_rate': 3.9493303268263064e-05, 'epoch': 1.0}

 10%|▉         | 4111/41250 [9:54:52<89:27:11,  8.67s/it][2025-04-25 17:52:36,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:52:36,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5708.30 | bwd_inner_microstep: 5695.39 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.66
[2025-04-25 17:52:36,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5708.32 | bwd_inner: 5695.39 | bwd_allreduce: 12.89 | step: 18.66

 10%|▉         | 4112/41250 [9:55:01<89:23:04,  8.66s/it]
                                                         
{'loss': 0.2588, 'grad_norm': 2.1001296043395996, 'learning_rate': 3.949295197511173e-05, 'epoch': 1.0}

 10%|▉         | 4112/41250 [9:55:01<89:23:04,  8.66s/it][2025-04-25 17:52:45,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:52:45,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5772.99 | bwd_inner_microstep: 5670.85 | bwd_allreduce_microstep: 102.09 | step_microstep: 18.62
[2025-04-25 17:52:45,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5773.00 | bwd_inner: 5670.85 | bwd_allreduce: 102.11 | step: 18.62

 10%|▉         | 4113/41250 [9:55:10<89:29:29,  8.68s/it]
                                                         
{'loss': 0.0795, 'grad_norm': 0.7764765024185181, 'learning_rate': 3.949260056179017e-05, 'epoch': 1.0}

 10%|▉         | 4113/41250 [9:55:10<89:29:29,  8.68s/it][2025-04-25 17:52:53,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 17:52:53,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.08 | bwd_microstep: 5710.73 | bwd_inner_microstep: 5698.17 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.65
[2025-04-25 17:52:53,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.08 | bwd: 5710.74 | bwd_inner: 5698.17 | bwd_allreduce: 12.53 | step: 18.65

 10%|▉         | 4114/41250 [9:55:18<89:26:05,  8.67s/it]
                                                         
{'loss': 0.1728, 'grad_norm': 1.1350712776184082, 'learning_rate': 3.949224902830057e-05, 'epoch': 1.0}

 10%|▉         | 4114/41250 [9:55:18<89:26:05,  8.67s/it][2025-04-25 17:53:02,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 17:53:02,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.79 | bwd_microstep: 5732.53 | bwd_inner_microstep: 5713.07 | bwd_allreduce_microstep: 19.42 | step_microstep: 18.68
[2025-04-25 17:53:02,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.79 | bwd: 5732.54 | bwd_inner: 5713.07 | bwd_allreduce: 19.43 | step: 18.68

 10%|▉         | 4115/41250 [9:55:27<89:28:01,  8.67s/it]
                                                         
{'loss': 0.2035, 'grad_norm': 1.657604455947876, 'learning_rate': 3.949189737464509e-05, 'epoch': 1.0}

 10%|▉         | 4115/41250 [9:55:27<89:28:01,  8.67s/it][2025-04-25 17:53:11,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 17:53:11,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.01 | bwd_microstep: 5734.81 | bwd_inner_microstep: 5695.78 | bwd_allreduce_microstep: 38.98 | step_microstep: 18.72
[2025-04-25 17:53:11,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.01 | bwd: 5734.82 | bwd_inner: 5695.78 | bwd_allreduce: 39.00 | step: 18.72

 10%|▉         | 4116/41250 [9:55:36<89:29:31,  8.68s/it]
                                                         
{'loss': 0.1519, 'grad_norm': 1.5503575801849365, 'learning_rate': 3.94915456008259e-05, 'epoch': 1.0}

 10%|▉         | 4116/41250 [9:55:36<89:29:31,  8.68s/it][2025-04-25 17:53:19,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 17:53:19,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5727.00 | bwd_inner_microstep: 5714.17 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.20
[2025-04-25 17:53:19,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5727.01 | bwd_inner: 5714.17 | bwd_allreduce: 12.80 | step: 19.20

 10%|▉         | 4117/41250 [9:55:45<89:27:08,  8.67s/it]
                                                         
{'loss': 0.2687, 'grad_norm': 3.8589577674865723, 'learning_rate': 3.949119370684515e-05, 'epoch': 1.0}

 10%|▉         | 4117/41250 [9:55:45<89:27:08,  8.67s/it][2025-04-25 17:53:28,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 17:53:28,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.59 | bwd_microstep: 5813.77 | bwd_inner_microstep: 5658.13 | bwd_allreduce_microstep: 155.59 | step_microstep: 18.57
[2025-04-25 17:53:28,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.59 | bwd: 5813.79 | bwd_inner: 5658.13 | bwd_allreduce: 155.61 | step: 18.57

 10%|▉         | 4118/41250 [9:55:53<89:35:51,  8.69s/it]
                                                         
{'loss': 0.3643, 'grad_norm': 2.9205973148345947, 'learning_rate': 3.949084169270504e-05, 'epoch': 1.0}

 10%|▉         | 4118/41250 [9:55:53<89:35:51,  8.69s/it][2025-04-25 17:53:37,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:53:37,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.33 | bwd_microstep: 5792.65 | bwd_inner_microstep: 5709.81 | bwd_allreduce_microstep: 82.80 | step_microstep: 18.51
[2025-04-25 17:53:37,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.33 | bwd: 5792.67 | bwd_inner: 5709.81 | bwd_allreduce: 82.82 | step: 18.52

 10%|▉         | 4119/41250 [9:56:02<89:42:14,  8.70s/it]
                                                         
{'loss': 0.1218, 'grad_norm': 1.1026244163513184, 'learning_rate': 3.949048955840772e-05, 'epoch': 1.0}

 10%|▉         | 4119/41250 [9:56:02<89:42:14,  8.70s/it][2025-04-25 17:53:45,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:53:45,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5732.40 | bwd_inner_microstep: 5719.65 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.18
[2025-04-25 17:53:45,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5732.41 | bwd_inner: 5719.65 | bwd_allreduce: 12.72 | step: 18.18

 10%|▉         | 4120/41250 [9:56:11<89:36:20,  8.69s/it]
                                                         
{'loss': 0.2788, 'grad_norm': 1.6800094842910767, 'learning_rate': 3.949013730395535e-05, 'epoch': 1.0}

 10%|▉         | 4120/41250 [9:56:11<89:36:20,  8.69s/it][2025-04-25 17:53:54,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.26 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 17:53:54,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.91 | bwd_microstep: 5859.16 | bwd_inner_microstep: 5711.40 | bwd_allreduce_microstep: 147.71 | step_microstep: 19.40
[2025-04-25 17:53:54,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.91 | bwd: 5859.17 | bwd_inner: 5711.40 | bwd_allreduce: 147.73 | step: 19.41

 10%|▉         | 4121/41250 [9:56:19<89:55:43,  8.72s/it]
                                                         
{'loss': 0.4607, 'grad_norm': 2.119431972503662, 'learning_rate': 3.9489784929350135e-05, 'epoch': 1.0}

 10%|▉         | 4121/41250 [9:56:19<89:55:43,  8.72s/it][2025-04-25 17:54:03,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 17:54:03,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.54 | bwd_microstep: 5714.56 | bwd_inner_microstep: 5668.60 | bwd_allreduce_microstep: 45.92 | step_microstep: 18.39
[2025-04-25 17:54:03,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.54 | bwd: 5714.57 | bwd_inner: 5668.60 | bwd_allreduce: 45.94 | step: 18.39

 10%|▉         | 4122/41250 [9:56:28<89:37:53,  8.69s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 0.9611542224884033, 'learning_rate': 3.9489432434594224e-05, 'epoch': 1.0}

 10%|▉         | 4122/41250 [9:56:28<89:37:53,  8.69s/it][2025-04-25 17:54:11,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 17:54:11,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.86 | bwd_microstep: 5797.59 | bwd_inner_microstep: 5667.36 | bwd_allreduce_microstep: 130.18 | step_microstep: 18.65
[2025-04-25 17:54:11,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.86 | bwd: 5797.60 | bwd_inner: 5667.36 | bwd_allreduce: 130.20 | step: 18.65

 10%|▉         | 4123/41250 [9:56:37<89:42:14,  8.70s/it]
                                                         
{'loss': 0.1421, 'grad_norm': 1.7527751922607422, 'learning_rate': 3.94890798196898e-05, 'epoch': 1.0}

 10%|▉         | 4123/41250 [9:56:37<89:42:14,  8.70s/it][2025-04-25 17:54:20,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 17:54:20,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.39 | bwd_microstep: 5758.72 | bwd_inner_microstep: 5713.68 | bwd_allreduce_microstep: 45.00 | step_microstep: 18.89
[2025-04-25 17:54:20,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.39 | bwd: 5758.74 | bwd_inner: 5713.68 | bwd_allreduce: 45.02 | step: 18.89

 10%|▉         | 4124/41250 [9:56:45<89:40:55,  8.70s/it]
                                                         
{'loss': 0.2211, 'grad_norm': 1.2536022663116455, 'learning_rate': 3.948872708463903e-05, 'epoch': 1.0}

 10%|▉         | 4124/41250 [9:56:45<89:40:55,  8.70s/it]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-25 17:54:31,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 17:54:31,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.56 | bwd_microstep: 5710.88 | bwd_inner_microstep: 5698.12 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.62
[2025-04-25 17:54:31,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.56 | bwd: 5710.89 | bwd_inner: 5698.12 | bwd_allreduce: 12.73 | step: 18.62

 10%|█         | 4125/41250 [9:56:56<96:16:23,  9.34s/it]evaluate!
evaluate!

                                                         
{'loss': 0.2629, 'grad_norm': 2.808717727661133, 'learning_rate': 3.948837422944409e-05, 'epoch': 1.0}

 10%|█         | 4125/41250 [9:56:56<96:16:23,  9.34s/it]evaluate!
1
1
1
[WARNING|trainer.py:803] 2025-04-25 17:54:34,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-25 17:54:34,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-25 17:54:34,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
2
2
[WARNING|trainer.py:803] 2025-04-25 17:54:36,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-25 17:54:36,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-25 17:54:36,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
3
3
[WARNING|trainer.py:803] 2025-04-25 17:54:39,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
New best accuracy: 0.3333333333333333. Saving model...
[WARNING|trainer.py:803] 2025-04-25 17:54:39,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
New best accuracy: 0.3333333333333333. Saving model...
[WARNING|trainer.py:803] 2025-04-25 17:54:39,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
New best accuracy: 0.3333333333333333. Saving model...
[INFO|trainer.py:3910] 2025-04-25 17:54:51,084 >> Saving model checkpoint to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast
[INFO|configuration_utils.py:420] 2025-04-25 17:54:51,089 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/config.json
[INFO|configuration_utils.py:909] 2025-04-25 17:54:51,090 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/generation_config.json
evaluate!
evaluate!
1
1
[WARNING|trainer.py:803] 2025-04-25 17:54:56,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-25 17:54:56,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
2
[WARNING|trainer.py:803] 2025-04-25 17:54:59,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-25 17:54:59,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
3
[WARNING|trainer.py:803] 2025-04-25 17:55:01,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
[WARNING|trainer.py:803] 2025-04-25 17:55:01,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
[2025-04-25 17:55:03,957] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 17:55:03,970] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:55:09,491] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 17:55:09,494] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:55:15,047] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 17:55:15,288] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:55:20,839] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 17:55:21,081] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[INFO|modeling_utils.py:2996] 2025-04-25 17:55:46,649 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2491] 2025-04-25 17:55:46,652 >> tokenizer config file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/tokenizer_config.json
[INFO|tokenization_utils_base.py:2500] 2025-04-25 17:55:46,652 >> Special tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/special_tokens_map.json
[INFO|tokenization_utils_base.py:2553] 2025-04-25 17:55:46,653 >> added tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/added_tokens.json
04/25/2025 17:55:48 - INFO - __main__ - Saved LoRA weights to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/lora_weights.pth
evaluate!
1
[WARNING|trainer.py:803] 2025-04-25 17:55:51,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
[WARNING|trainer.py:803] 2025-04-25 17:55:54,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
[WARNING|trainer.py:803] 2025-04-25 17:55:57,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
[2025-04-25 17:55:59,769] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:56:05,873] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:56:11,469] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:56:17,177] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-25 17:56:33,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.09 | optimizer_step: 1.12
[2025-04-25 17:56:33,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.16 | bwd_microstep: 5665.04 | bwd_inner_microstep: 5652.11 | bwd_allreduce_microstep: 12.88 | step_microstep: 20.39
[2025-04-25 17:56:33,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.13 | bwd: 5665.05 | bwd_inner: 5652.11 | bwd_allreduce: 12.90 | step: 20.39

 10%|█         | 4126/41250 [9:58:58<444:27:29, 43.10s/it]
                                                          
{'loss': 0.0967, 'grad_norm': 1.5433621406555176, 'learning_rate': 3.948802125410715e-05, 'epoch': 1.0}

 10%|█         | 4126/41250 [9:58:58<444:27:29, 43.10s/it][2025-04-25 17:56:41,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.11 | optimizer_step: 1.10
[2025-04-25 17:56:41,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2802.50 | bwd_microstep: 5644.42 | bwd_inner_microstep: 5609.47 | bwd_allreduce_microstep: 34.89 | step_microstep: 20.05
[2025-04-25 17:56:41,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2802.50 | bwd: 5644.44 | bwd_inner: 5609.47 | bwd_allreduce: 34.92 | step: 20.05

 10%|█         | 4127/41250 [9:59:07<337:30:54, 32.73s/it]
                                                          
{'loss': 0.1664, 'grad_norm': 1.3101110458374023, 'learning_rate': 3.9487668158630396e-05, 'epoch': 1.0}

 10%|█         | 4127/41250 [9:59:07<337:30:54, 32.73s/it][2025-04-25 17:56:50,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 1.29
[2025-04-25 17:56:50,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.50 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5672.05 | bwd_allreduce_microstep: 43.37 | step_microstep: 20.24
[2025-04-25 17:56:50,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.50 | bwd: 5715.49 | bwd_inner: 5672.05 | bwd_allreduce: 43.39 | step: 20.24

 10%|█         | 4128/41250 [9:59:15<262:59:08, 25.50s/it]
                                                          
{'loss': 0.1747, 'grad_norm': 1.3050565719604492, 'learning_rate': 3.9487314943016006e-05, 'epoch': 1.0}

 10%|█         | 4128/41250 [9:59:15<262:59:08, 25.50s/it][2025-04-25 17:56:59,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.10
[2025-04-25 17:56:59,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.50 | bwd_microstep: 5694.06 | bwd_inner_microstep: 5651.10 | bwd_allreduce_microstep: 42.91 | step_microstep: 19.92
[2025-04-25 17:56:59,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.50 | bwd: 5694.08 | bwd_inner: 5651.10 | bwd_allreduce: 42.94 | step: 19.92

 10%|█         | 4129/41250 [9:59:24<210:45:20, 20.44s/it]
                                                          
{'loss': 0.1765, 'grad_norm': 1.246640682220459, 'learning_rate': 3.9486961607266146e-05, 'epoch': 1.0}

 10%|█         | 4129/41250 [9:59:24<210:45:20, 20.44s/it][2025-04-25 17:57:07,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-25 17:57:07,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.24 | bwd_microstep: 5656.06 | bwd_inner_microstep: 5634.02 | bwd_allreduce_microstep: 21.98 | step_microstep: 19.71
[2025-04-25 17:57:07,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.24 | bwd: 5656.07 | bwd_inner: 5634.02 | bwd_allreduce: 22.01 | step: 19.71

 10%|█         | 4130/41250 [9:59:33<174:00:18, 16.88s/it]
                                                          
{'loss': 0.1559, 'grad_norm': 1.3961122035980225, 'learning_rate': 3.9486608151383007e-05, 'epoch': 1.0}

 10%|█         | 4130/41250 [9:59:33<174:00:18, 16.88s/it][2025-04-25 17:57:16,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 17:57:16,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.87 | bwd_microstep: 5729.87 | bwd_inner_microstep: 5630.09 | bwd_allreduce_microstep: 99.73 | step_microstep: 18.97
[2025-04-25 17:57:16,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.87 | bwd: 5729.88 | bwd_inner: 5630.09 | bwd_allreduce: 99.75 | step: 18.97

 10%|█         | 4131/41250 [9:59:41<148:29:59, 14.40s/it]
                                                          
{'loss': 0.1372, 'grad_norm': 1.2956308126449585, 'learning_rate': 3.9486254575368755e-05, 'epoch': 1.0}

 10%|█         | 4131/41250 [9:59:41<148:29:59, 14.40s/it][2025-04-25 17:57:24,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 17:57:24,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.61 | bwd_microstep: 5692.23 | bwd_inner_microstep: 5636.88 | bwd_allreduce_microstep: 55.31 | step_microstep: 18.94
[2025-04-25 17:57:24,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.61 | bwd: 5692.25 | bwd_inner: 5636.88 | bwd_allreduce: 55.33 | step: 18.94

 10%|█         | 4132/41250 [9:59:50<130:32:00, 12.66s/it]
                                                          
{'loss': 0.2273, 'grad_norm': 7.36805534362793, 'learning_rate': 3.948590087922558e-05, 'epoch': 1.0}

 10%|█         | 4132/41250 [9:59:50<130:32:00, 12.66s/it][2025-04-25 17:57:33,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 17:57:33,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.21 | bwd_microstep: 5735.57 | bwd_inner_microstep: 5670.08 | bwd_allreduce_microstep: 65.45 | step_microstep: 18.74
[2025-04-25 17:57:33,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.21 | bwd: 5735.58 | bwd_inner: 5670.08 | bwd_allreduce: 65.46 | step: 18.75

 10%|█         | 4133/41250 [9:59:58<118:10:19, 11.46s/it]
                                                          
{'loss': 0.114, 'grad_norm': 0.7711699604988098, 'learning_rate': 3.948554706295565e-05, 'epoch': 1.0}

 10%|█         | 4133/41250 [9:59:58<118:10:19, 11.46s/it][2025-04-25 17:57:42,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 17:57:42,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.70 | bwd_microstep: 5882.06 | bwd_inner_microstep: 5669.07 | bwd_allreduce_microstep: 212.94 | step_microstep: 18.56
[2025-04-25 17:57:42,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.70 | bwd: 5882.07 | bwd_inner: 5669.07 | bwd_allreduce: 212.95 | step: 18.57

 10%|█         | 4134/41250 [10:00:07<109:56:41, 10.66s/it]
                                                           
{'loss': 0.048, 'grad_norm': 0.8537027835845947, 'learning_rate': 3.948519312656116e-05, 'epoch': 1.0}

 10%|█         | 4134/41250 [10:00:07<109:56:41, 10.66s/it][2025-04-25 17:57:51,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 17:57:51,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.33 | bwd_microstep: 5688.03 | bwd_inner_microstep: 5671.93 | bwd_allreduce_microstep: 16.05 | step_microstep: 19.42
[2025-04-25 17:57:51,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.33 | bwd: 5688.04 | bwd_inner: 5671.93 | bwd_allreduce: 16.07 | step: 19.42

 10%|█         | 4135/41250 [10:00:16<103:36:24, 10.05s/it]
                                                           
{'loss': 0.108, 'grad_norm': 2.1665878295898438, 'learning_rate': 3.94848390700443e-05, 'epoch': 1.0}

 10%|█         | 4135/41250 [10:00:16<103:36:24, 10.05s/it][2025-04-25 17:57:59,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 17:57:59,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.00 | bwd_microstep: 5745.48 | bwd_inner_microstep: 5637.94 | bwd_allreduce_microstep: 107.49 | step_microstep: 19.38
[2025-04-25 17:57:59,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.00 | bwd: 5745.50 | bwd_inner: 5637.94 | bwd_allreduce: 107.51 | step: 19.38

 10%|█         | 4136/41250 [10:00:24<99:18:35,  9.63s/it] 
                                                          
{'loss': 0.2206, 'grad_norm': 1.2619880437850952, 'learning_rate': 3.948448489340722e-05, 'epoch': 1.0}

 10%|█         | 4136/41250 [10:00:24<99:18:35,  9.63s/it][2025-04-25 17:58:08,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 17:58:08,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.52 | bwd_microstep: 5757.14 | bwd_inner_microstep: 5670.63 | bwd_allreduce_microstep: 86.46 | step_microstep: 18.84
[2025-04-25 17:58:08,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.52 | bwd: 5757.16 | bwd_inner: 5670.63 | bwd_allreduce: 86.48 | step: 18.85

 10%|█         | 4137/41250 [10:00:33<96:21:40,  9.35s/it]
                                                          
{'loss': 0.232, 'grad_norm': 0.8654947280883789, 'learning_rate': 3.948413059665214e-05, 'epoch': 1.0}

 10%|█         | 4137/41250 [10:00:33<96:21:40,  9.35s/it][2025-04-25 17:58:16,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 17:58:16,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5692.04 | bwd_inner_microstep: 5633.00 | bwd_allreduce_microstep: 58.99 | step_microstep: 19.02
[2025-04-25 17:58:16,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5692.05 | bwd_inner: 5633.00 | bwd_allreduce: 59.01 | step: 19.02

 10%|█         | 4138/41250 [10:00:42<94:02:53,  9.12s/it]
                                                          
{'loss': 0.0877, 'grad_norm': 0.8022608160972595, 'learning_rate': 3.9483776179781216e-05, 'epoch': 1.0}

 10%|█         | 4138/41250 [10:00:42<94:02:53,  9.12s/it][2025-04-25 17:58:25,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 17:58:25,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.12 | bwd_microstep: 5743.95 | bwd_inner_microstep: 5692.64 | bwd_allreduce_microstep: 51.26 | step_microstep: 19.34
[2025-04-25 17:58:25,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.12 | bwd: 5743.96 | bwd_inner: 5692.64 | bwd_allreduce: 51.28 | step: 19.34

 10%|█         | 4139/41250 [10:00:50<92:39:58,  8.99s/it]
                                                          
{'loss': 0.1978, 'grad_norm': 1.5828359127044678, 'learning_rate': 3.948342164279665e-05, 'epoch': 1.0}

 10%|█         | 4139/41250 [10:00:50<92:39:58,  8.99s/it][2025-04-25 17:58:34,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 17:58:34,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.34 | bwd_microstep: 5709.21 | bwd_inner_microstep: 5689.77 | bwd_allreduce_microstep: 19.39 | step_microstep: 19.05
[2025-04-25 17:58:34,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.34 | bwd: 5709.23 | bwd_inner: 5689.77 | bwd_allreduce: 19.41 | step: 19.05

 10%|█         | 4140/41250 [10:00:59<91:35:41,  8.89s/it]
                                                          
{'loss': 0.1855, 'grad_norm': 1.5996220111846924, 'learning_rate': 3.948306698570061e-05, 'epoch': 1.0}

 10%|█         | 4140/41250 [10:00:59<91:35:41,  8.89s/it][2025-04-25 17:58:42,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 17:58:42,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.45 | bwd_microstep: 5697.32 | bwd_inner_microstep: 5684.31 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.17
[2025-04-25 17:58:42,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.45 | bwd: 5697.33 | bwd_inner: 5684.31 | bwd_allreduce: 12.98 | step: 19.18

 10%|█         | 4141/41250 [10:01:08<90:46:11,  8.81s/it]
                                                          
{'loss': 0.1522, 'grad_norm': 1.473616600036621, 'learning_rate': 3.948271220849531e-05, 'epoch': 1.0}

 10%|█         | 4141/41250 [10:01:08<90:46:11,  8.81s/it][2025-04-25 17:58:51,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 17:58:51,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.60 | bwd_microstep: 5678.18 | bwd_inner_microstep: 5655.33 | bwd_allreduce_microstep: 22.79 | step_microstep: 19.18
[2025-04-25 17:58:51,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.60 | bwd: 5678.19 | bwd_inner: 5655.33 | bwd_allreduce: 22.82 | step: 19.18

 10%|█         | 4142/41250 [10:01:16<90:06:24,  8.74s/it]
                                                          
{'loss': 0.1321, 'grad_norm': 1.1322617530822754, 'learning_rate': 3.948235731118291e-05, 'epoch': 1.0}

 10%|█         | 4142/41250 [10:01:16<90:06:24,  8.74s/it][2025-04-25 17:59:00,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 17:59:00,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.99 | bwd_microstep: 5874.40 | bwd_inner_microstep: 5861.78 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.79
[2025-04-25 17:59:00,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.99 | bwd: 5874.41 | bwd_inner: 5861.78 | bwd_allreduce: 12.60 | step: 18.79

 10%|█         | 4143/41250 [10:01:25<90:33:36,  8.79s/it]
                                                          
{'loss': 0.1587, 'grad_norm': 1.1864161491394043, 'learning_rate': 3.948200229376561e-05, 'epoch': 1.0}

 10%|█         | 4143/41250 [10:01:25<90:33:36,  8.79s/it][2025-04-25 17:59:09,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 0.93
[2025-04-25 17:59:09,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.95 | bwd_microstep: 5747.84 | bwd_inner_microstep: 5686.30 | bwd_allreduce_microstep: 61.49 | step_microstep: 19.25
[2025-04-25 17:59:09,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.95 | bwd: 5747.85 | bwd_inner: 5686.30 | bwd_allreduce: 61.51 | step: 19.25

 10%|█         | 4144/41250 [10:01:34<90:14:15,  8.75s/it]
                                                          
{'loss': 0.1562, 'grad_norm': 2.6250088214874268, 'learning_rate': 3.948164715624561e-05, 'epoch': 1.0}

 10%|█         | 4144/41250 [10:01:34<90:14:15,  8.75s/it][2025-04-25 17:59:17,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 17:59:17,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.77 | bwd_microstep: 5736.41 | bwd_inner_microstep: 5659.23 | bwd_allreduce_microstep: 77.14 | step_microstep: 18.82
[2025-04-25 17:59:17,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.77 | bwd: 5736.43 | bwd_inner: 5659.23 | bwd_allreduce: 77.15 | step: 18.83

 10%|█         | 4145/41250 [10:01:43<89:56:00,  8.73s/it]
                                                          
{'loss': 0.1122, 'grad_norm': 0.9504488110542297, 'learning_rate': 3.948129189862507e-05, 'epoch': 1.0}

 10%|█         | 4145/41250 [10:01:43<89:56:00,  8.73s/it][2025-04-25 17:59:26,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 17:59:26,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.50 | bwd_microstep: 5884.71 | bwd_inner_microstep: 5653.80 | bwd_allreduce_microstep: 230.87 | step_microstep: 18.67
[2025-04-25 17:59:26,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.50 | bwd: 5884.73 | bwd_inner: 5653.80 | bwd_allreduce: 230.89 | step: 18.68

 10%|█         | 4146/41250 [10:01:51<90:08:30,  8.75s/it]
                                                          
{'loss': 0.1389, 'grad_norm': 1.6825302839279175, 'learning_rate': 3.948093652090621e-05, 'epoch': 1.01}

 10%|█         | 4146/41250 [10:01:51<90:08:30,  8.75s/it][2025-04-25 17:59:35,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 17:59:35,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 5768.28 | bwd_inner_microstep: 5658.77 | bwd_allreduce_microstep: 109.46 | step_microstep: 19.27
[2025-04-25 17:59:35,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 5768.29 | bwd_inner: 5658.77 | bwd_allreduce: 109.48 | step: 19.27

 10%|█         | 4147/41250 [10:02:00<89:57:45,  8.73s/it]
                                                          
{'loss': 0.2164, 'grad_norm': 1.5684460401535034, 'learning_rate': 3.94805810230912e-05, 'epoch': 1.01}

 10%|█         | 4147/41250 [10:02:00<89:57:45,  8.73s/it][2025-04-25 17:59:43,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.47 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 17:59:43,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.07 | bwd_microstep: 5796.78 | bwd_inner_microstep: 5658.14 | bwd_allreduce_microstep: 138.60 | step_microstep: 20.32
[2025-04-25 17:59:43,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.07 | bwd: 5796.80 | bwd_inner: 5658.14 | bwd_allreduce: 138.62 | step: 20.32

 10%|█         | 4148/41250 [10:02:09<89:53:32,  8.72s/it]
                                                          
{'loss': 0.0538, 'grad_norm': 0.6008443832397461, 'learning_rate': 3.9480225405182246e-05, 'epoch': 1.01}

 10%|█         | 4148/41250 [10:02:09<89:53:32,  8.72s/it][2025-04-25 17:59:52,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 17:59:52,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.79 | bwd_microstep: 5789.71 | bwd_inner_microstep: 5670.08 | bwd_allreduce_microstep: 119.58 | step_microstep: 18.53
[2025-04-25 17:59:52,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.79 | bwd: 5789.72 | bwd_inner: 5670.08 | bwd_allreduce: 119.60 | step: 18.53

 10%|█         | 4149/41250 [10:02:17<89:50:02,  8.72s/it]
                                                          
{'loss': 0.0956, 'grad_norm': 0.7498725652694702, 'learning_rate': 3.947986966718153e-05, 'epoch': 1.01}

 10%|█         | 4149/41250 [10:02:17<89:50:02,  8.72s/it][2025-04-25 18:00:01,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:00:01,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.28 | bwd_microstep: 5711.34 | bwd_inner_microstep: 5639.66 | bwd_allreduce_microstep: 71.63 | step_microstep: 19.23
[2025-04-25 18:00:01,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.28 | bwd: 5711.35 | bwd_inner: 5639.66 | bwd_allreduce: 71.65 | step: 19.23

 10%|█         | 4150/41250 [10:02:26<89:32:09,  8.69s/it]
                                                          
{'loss': 0.0863, 'grad_norm': 0.7300331592559814, 'learning_rate': 3.9479513809091254e-05, 'epoch': 1.01}

 10%|█         | 4150/41250 [10:02:26<89:32:09,  8.69s/it][2025-04-25 18:00:09,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 1.13
[2025-04-25 18:00:09,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5769.63 | bwd_inner_microstep: 5729.90 | bwd_allreduce_microstep: 39.67 | step_microstep: 19.31
[2025-04-25 18:00:09,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5769.64 | bwd_inner: 5729.90 | bwd_allreduce: 39.70 | step: 19.31

 10%|█         | 4151/41250 [10:02:35<89:36:21,  8.70s/it]
                                                          
{'loss': 0.1633, 'grad_norm': 1.2381879091262817, 'learning_rate': 3.94791578309136e-05, 'epoch': 1.01}

 10%|█         | 4151/41250 [10:02:35<89:36:21,  8.70s/it][2025-04-25 18:00:18,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 18:00:18,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.41 | bwd_microstep: 5705.72 | bwd_inner_microstep: 5653.16 | bwd_allreduce_microstep: 52.51 | step_microstep: 19.03
[2025-04-25 18:00:18,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.41 | bwd: 5705.73 | bwd_inner: 5653.16 | bwd_allreduce: 52.52 | step: 19.03

 10%|█         | 4152/41250 [10:02:43<89:22:13,  8.67s/it]
                                                          
{'loss': 0.1048, 'grad_norm': 1.0615270137786865, 'learning_rate': 3.947880173265077e-05, 'epoch': 1.01}

 10%|█         | 4152/41250 [10:02:43<89:22:13,  8.67s/it][2025-04-25 18:00:27,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:00:27,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.08 | bwd_microstep: 5773.40 | bwd_inner_microstep: 5662.34 | bwd_allreduce_microstep: 111.01 | step_microstep: 18.73
[2025-04-25 18:00:27,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.08 | bwd: 5773.41 | bwd_inner: 5662.34 | bwd_allreduce: 111.03 | step: 18.73

 10%|█         | 4153/41250 [10:02:52<89:23:53,  8.68s/it]
                                                          
{'loss': 0.0781, 'grad_norm': 0.7601968050003052, 'learning_rate': 3.947844551430496e-05, 'epoch': 1.01}

 10%|█         | 4153/41250 [10:02:52<89:23:53,  8.68s/it][2025-04-25 18:00:35,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:00:35,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.14 | bwd_microstep: 5751.74 | bwd_inner_microstep: 5665.90 | bwd_allreduce_microstep: 85.79 | step_microstep: 18.61
[2025-04-25 18:00:35,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.14 | bwd: 5751.76 | bwd_inner: 5665.90 | bwd_allreduce: 85.81 | step: 18.62

 10%|█         | 4154/41250 [10:03:01<89:23:35,  8.68s/it]
                                                          
{'loss': 0.2023, 'grad_norm': 1.0136252641677856, 'learning_rate': 3.947808917587837e-05, 'epoch': 1.01}

 10%|█         | 4154/41250 [10:03:01<89:23:35,  8.68s/it][2025-04-25 18:00:44,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.24
[2025-04-25 18:00:44,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.61 | bwd_microstep: 5755.21 | bwd_inner_microstep: 5711.59 | bwd_allreduce_microstep: 43.57 | step_microstep: 19.74
[2025-04-25 18:00:44,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.61 | bwd: 5755.22 | bwd_inner: 5711.59 | bwd_allreduce: 43.59 | step: 19.74

 10%|█         | 4155/41250 [10:03:09<89:27:34,  8.68s/it]
                                                          
{'loss': 0.1403, 'grad_norm': 1.389998435974121, 'learning_rate': 3.9477732717373186e-05, 'epoch': 1.01}

 10%|█         | 4155/41250 [10:03:09<89:27:34,  8.68s/it][2025-04-25 18:00:53,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.76 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:00:53,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.90 | bwd_microstep: 5733.90 | bwd_inner_microstep: 5708.63 | bwd_allreduce_microstep: 25.23 | step_microstep: 19.88
[2025-04-25 18:00:53,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.90 | bwd: 5733.92 | bwd_inner: 5708.63 | bwd_allreduce: 25.25 | step: 19.89

 10%|█         | 4156/41250 [10:03:18<89:25:32,  8.68s/it]
                                                          
{'loss': 0.1432, 'grad_norm': 1.5966302156448364, 'learning_rate': 3.947737613879161e-05, 'epoch': 1.01}

 10%|█         | 4156/41250 [10:03:18<89:25:32,  8.68s/it][2025-04-25 18:01:01,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-25 18:01:01,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.90 | bwd_microstep: 5784.57 | bwd_inner_microstep: 5673.09 | bwd_allreduce_microstep: 111.43 | step_microstep: 19.34
[2025-04-25 18:01:01,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.90 | bwd: 5784.59 | bwd_inner: 5673.09 | bwd_allreduce: 111.45 | step: 19.35

 10%|█         | 4157/41250 [10:03:27<89:31:21,  8.69s/it]
                                                          
{'loss': 0.1382, 'grad_norm': 1.4558185338974, 'learning_rate': 3.947701944013584e-05, 'epoch': 1.01}

 10%|█         | 4157/41250 [10:03:27<89:31:21,  8.69s/it][2025-04-25 18:01:10,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.26 | optimizer_step: 0.98
[2025-04-25 18:01:10,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2941.93 | bwd_microstep: 5903.17 | bwd_inner_microstep: 5890.14 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.51
[2025-04-25 18:01:10,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2941.93 | bwd: 5903.19 | bwd_inner: 5890.14 | bwd_allreduce: 13.00 | step: 19.51

 10%|█         | 4158/41250 [10:03:36<90:16:37,  8.76s/it]
                                                          
{'loss': 0.2006, 'grad_norm': 2.0930919647216797, 'learning_rate': 3.947666262140809e-05, 'epoch': 1.01}

 10%|█         | 4158/41250 [10:03:36<90:16:37,  8.76s/it][2025-04-25 18:01:19,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 18:01:19,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.17 | bwd_microstep: 5764.09 | bwd_inner_microstep: 5667.65 | bwd_allreduce_microstep: 96.39 | step_microstep: 18.87
[2025-04-25 18:01:19,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.17 | bwd: 5764.10 | bwd_inner: 5667.65 | bwd_allreduce: 96.41 | step: 18.87

 10%|█         | 4159/41250 [10:03:44<90:01:16,  8.74s/it]
                                                          
{'loss': 0.2323, 'grad_norm': 1.9333583116531372, 'learning_rate': 3.947630568261053e-05, 'epoch': 1.01}

 10%|█         | 4159/41250 [10:03:44<90:01:16,  8.74s/it][2025-04-25 18:01:28,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-25 18:01:28,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.10 | bwd_microstep: 5754.26 | bwd_inner_microstep: 5670.74 | bwd_allreduce_microstep: 83.47 | step_microstep: 19.28
[2025-04-25 18:01:28,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.10 | bwd: 5754.27 | bwd_inner: 5670.74 | bwd_allreduce: 83.49 | step: 19.28

 10%|█         | 4160/41250 [10:03:53<89:48:55,  8.72s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.4183326959609985, 'learning_rate': 3.9475948623745376e-05, 'epoch': 1.01}

 10%|█         | 4160/41250 [10:03:53<89:48:55,  8.72s/it][2025-04-25 18:01:37,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-25 18:01:37,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.29 | bwd_microstep: 5802.46 | bwd_inner_microstep: 5669.47 | bwd_allreduce_microstep: 132.95 | step_microstep: 18.58
[2025-04-25 18:01:37,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.29 | bwd: 5802.47 | bwd_inner: 5669.47 | bwd_allreduce: 132.96 | step: 18.58

 10%|█         | 4161/41250 [10:04:02<89:50:33,  8.72s/it]
                                                          
{'loss': 0.0824, 'grad_norm': 0.9515659809112549, 'learning_rate': 3.947559144481484e-05, 'epoch': 1.01}

 10%|█         | 4161/41250 [10:04:02<89:50:33,  8.72s/it][2025-04-25 18:01:45,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:01:45,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.33 | bwd_microstep: 5770.79 | bwd_inner_microstep: 5657.02 | bwd_allreduce_microstep: 113.73 | step_microstep: 18.80
[2025-04-25 18:01:45,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.33 | bwd: 5770.80 | bwd_inner: 5657.02 | bwd_allreduce: 113.74 | step: 18.80

 10%|█         | 4162/41250 [10:04:11<89:43:22,  8.71s/it]
                                                          
{'loss': 0.0128, 'grad_norm': 0.12015310674905777, 'learning_rate': 3.9475234145821104e-05, 'epoch': 1.01}

 10%|█         | 4162/41250 [10:04:11<89:43:22,  8.71s/it][2025-04-25 18:01:54,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 18:01:54,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.72 | bwd_microstep: 5732.12 | bwd_inner_microstep: 5712.94 | bwd_allreduce_microstep: 19.13 | step_microstep: 18.97
[2025-04-25 18:01:54,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.73 | bwd: 5732.13 | bwd_inner: 5712.94 | bwd_allreduce: 19.15 | step: 18.97

 10%|█         | 4163/41250 [10:04:19<89:36:31,  8.70s/it]
                                                          
{'loss': 0.2137, 'grad_norm': 2.2988533973693848, 'learning_rate': 3.947487672676638e-05, 'epoch': 1.01}

 10%|█         | 4163/41250 [10:04:19<89:36:31,  8.70s/it][2025-04-25 18:02:02,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 18:02:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.91 | bwd_microstep: 5703.46 | bwd_inner_microstep: 5690.76 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.04
[2025-04-25 18:02:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.91 | bwd: 5703.47 | bwd_inner: 5690.76 | bwd_allreduce: 12.67 | step: 19.04

 10%|█         | 4164/41250 [10:04:28<89:26:04,  8.68s/it]
                                                          
{'loss': 0.1216, 'grad_norm': 1.7952409982681274, 'learning_rate': 3.947451918765288e-05, 'epoch': 1.01}

 10%|█         | 4164/41250 [10:04:28<89:26:04,  8.68s/it][2025-04-25 18:02:11,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:02:11,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.02 | bwd_microstep: 5802.98 | bwd_inner_microstep: 5790.30 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.11
[2025-04-25 18:02:11,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.02 | bwd: 5802.99 | bwd_inner: 5790.30 | bwd_allreduce: 12.65 | step: 18.11

 10%|█         | 4165/41250 [10:04:37<89:44:17,  8.71s/it]
                                                          
{'loss': 0.0652, 'grad_norm': 0.9131054282188416, 'learning_rate': 3.94741615284828e-05, 'epoch': 1.01}

 10%|█         | 4165/41250 [10:04:37<89:44:17,  8.71s/it][2025-04-25 18:02:20,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 18:02:20,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.74 | bwd_microstep: 5703.44 | bwd_inner_microstep: 5663.99 | bwd_allreduce_microstep: 39.40 | step_microstep: 19.26
[2025-04-25 18:02:20,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.74 | bwd: 5703.46 | bwd_inner: 5663.99 | bwd_allreduce: 39.43 | step: 19.27

 10%|█         | 4166/41250 [10:04:45<89:28:38,  8.69s/it]
                                                          
{'loss': 0.0228, 'grad_norm': 0.2111085206270218, 'learning_rate': 3.947380374925834e-05, 'epoch': 1.01}

 10%|█         | 4166/41250 [10:04:45<89:28:38,  8.69s/it][2025-04-25 18:02:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 18:02:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.43 | bwd_microstep: 5706.11 | bwd_inner_microstep: 5693.39 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.72
[2025-04-25 18:02:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.43 | bwd: 5706.13 | bwd_inner: 5693.39 | bwd_allreduce: 12.69 | step: 18.72

 10%|█         | 4167/41250 [10:04:54<89:19:47,  8.67s/it]
                                                          
{'loss': 0.0831, 'grad_norm': 1.265779972076416, 'learning_rate': 3.94734458499817e-05, 'epoch': 1.01}

 10%|█         | 4167/41250 [10:04:54<89:19:47,  8.67s/it][2025-04-25 18:02:37,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 1.17
[2025-04-25 18:02:37,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5698.71 | bwd_inner_microstep: 5660.32 | bwd_allreduce_microstep: 38.35 | step_microstep: 19.26
[2025-04-25 18:02:37,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5698.73 | bwd_inner: 5660.32 | bwd_allreduce: 38.37 | step: 19.26

 10%|█         | 4168/41250 [10:05:02<89:08:26,  8.65s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 1.9594755172729492, 'learning_rate': 3.947308783065511e-05, 'epoch': 1.01}

 10%|█         | 4168/41250 [10:05:02<89:08:26,  8.65s/it][2025-04-25 18:02:46,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.23 | optimizer_step: 0.89
[2025-04-25 18:02:46,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.47 | bwd_microstep: 5701.33 | bwd_inner_microstep: 5666.73 | bwd_allreduce_microstep: 34.55 | step_microstep: 19.22
[2025-04-25 18:02:46,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.47 | bwd: 5701.34 | bwd_inner: 5666.73 | bwd_allreduce: 34.57 | step: 19.22

 10%|█         | 4169/41250 [10:05:11<89:02:18,  8.64s/it]
                                                          
{'loss': 0.2158, 'grad_norm': 1.2009261846542358, 'learning_rate': 3.947272969128076e-05, 'epoch': 1.01}

 10%|█         | 4169/41250 [10:05:11<89:02:18,  8.64s/it][2025-04-25 18:02:54,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:02:54,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.24 | bwd_microstep: 5755.00 | bwd_inner_microstep: 5698.52 | bwd_allreduce_microstep: 56.44 | step_microstep: 18.51
[2025-04-25 18:02:54,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.24 | bwd: 5755.02 | bwd_inner: 5698.52 | bwd_allreduce: 56.46 | step: 18.51

 10%|█         | 4170/41250 [10:05:20<89:09:51,  8.66s/it]
                                                          
{'loss': 0.0888, 'grad_norm': 1.1178159713745117, 'learning_rate': 3.947237143186086e-05, 'epoch': 1.01}

 10%|█         | 4170/41250 [10:05:20<89:09:51,  8.66s/it][2025-04-25 18:03:03,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 1.00
[2025-04-25 18:03:03,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5689.27 | bwd_inner_microstep: 5676.67 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.41
[2025-04-25 18:03:03,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5689.28 | bwd_inner: 5676.67 | bwd_allreduce: 12.57 | step: 18.42

 10%|█         | 4171/41250 [10:05:28<89:02:26,  8.64s/it]
                                                          
{'loss': 0.218, 'grad_norm': 1.8544337749481201, 'learning_rate': 3.947201305239763e-05, 'epoch': 1.01}

 10%|█         | 4171/41250 [10:05:28<89:02:26,  8.64s/it][2025-04-25 18:03:12,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-25 18:03:12,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.55 | bwd_microstep: 5723.41 | bwd_inner_microstep: 5700.43 | bwd_allreduce_microstep: 22.92 | step_microstep: 19.94
[2025-04-25 18:03:12,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.55 | bwd: 5723.43 | bwd_inner: 5700.43 | bwd_allreduce: 22.95 | step: 19.94

 10%|█         | 4172/41250 [10:05:37<89:05:40,  8.65s/it]
                                                          
{'loss': 0.5387, 'grad_norm': 2.278240442276001, 'learning_rate': 3.947165455289325e-05, 'epoch': 1.01}

 10%|█         | 4172/41250 [10:05:37<89:05:40,  8.65s/it][2025-04-25 18:03:20,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:03:20,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.04 | bwd_microstep: 5755.79 | bwd_inner_microstep: 5663.28 | bwd_allreduce_microstep: 92.46 | step_microstep: 18.79
[2025-04-25 18:03:20,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.04 | bwd: 5755.81 | bwd_inner: 5663.28 | bwd_allreduce: 92.48 | step: 18.79

 10%|█         | 4173/41250 [10:05:46<89:09:36,  8.66s/it]
                                                          
{'loss': 0.1244, 'grad_norm': 1.2545166015625, 'learning_rate': 3.947129593334996e-05, 'epoch': 1.01}

 10%|█         | 4173/41250 [10:05:46<89:09:36,  8.66s/it][2025-04-25 18:03:29,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 18:03:29,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.44 | bwd_microstep: 5762.19 | bwd_inner_microstep: 5646.43 | bwd_allreduce_microstep: 115.71 | step_microstep: 18.86
[2025-04-25 18:03:29,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.44 | bwd: 5762.20 | bwd_inner: 5646.43 | bwd_allreduce: 115.72 | step: 18.87

 10%|█         | 4174/41250 [10:05:54<89:12:46,  8.66s/it]
                                                          
{'loss': 0.1622, 'grad_norm': 0.6883653402328491, 'learning_rate': 3.9470937193769964e-05, 'epoch': 1.01}

 10%|█         | 4174/41250 [10:05:54<89:12:46,  8.66s/it][2025-04-25 18:03:38,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-25 18:03:38,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.56 | bwd_microstep: 5714.96 | bwd_inner_microstep: 5701.91 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.82
[2025-04-25 18:03:38,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5714.97 | bwd_inner: 5701.91 | bwd_allreduce: 13.02 | step: 18.82

 10%|█         | 4175/41250 [10:06:03<89:09:52,  8.66s/it]
                                                          
{'loss': 0.0602, 'grad_norm': 0.5664167404174805, 'learning_rate': 3.947057833415547e-05, 'epoch': 1.01}

 10%|█         | 4175/41250 [10:06:03<89:09:52,  8.66s/it][2025-04-25 18:03:46,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.35 | optimizer_step: 1.06
[2025-04-25 18:03:46,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.90 | bwd_microstep: 5704.94 | bwd_inner_microstep: 5690.84 | bwd_allreduce_microstep: 14.03 | step_microstep: 20.40
[2025-04-25 18:03:46,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.91 | bwd: 5704.95 | bwd_inner: 5690.84 | bwd_allreduce: 14.07 | step: 20.40

 10%|█         | 4176/41250 [10:06:12<89:06:00,  8.65s/it]
                                                          
{'loss': 0.0768, 'grad_norm': 1.050527811050415, 'learning_rate': 3.9470219354508685e-05, 'epoch': 1.01}

 10%|█         | 4176/41250 [10:06:12<89:06:00,  8.65s/it][2025-04-25 18:03:55,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 18:03:55,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.93 | bwd_microstep: 5739.69 | bwd_inner_microstep: 5697.43 | bwd_allreduce_microstep: 42.20 | step_microstep: 19.28
[2025-04-25 18:03:55,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.94 | bwd: 5739.70 | bwd_inner: 5697.43 | bwd_allreduce: 42.23 | step: 19.28

 10%|█         | 4177/41250 [10:06:20<89:08:51,  8.66s/it]
                                                          
{'loss': 0.0303, 'grad_norm': 0.3459623456001282, 'learning_rate': 3.946986025483183e-05, 'epoch': 1.01}

 10%|█         | 4177/41250 [10:06:20<89:08:51,  8.66s/it][2025-04-25 18:04:04,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:04:04,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.30 | bwd_microstep: 5691.18 | bwd_inner_microstep: 5678.32 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.68
[2025-04-25 18:04:04,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.30 | bwd: 5691.20 | bwd_inner: 5678.32 | bwd_allreduce: 12.83 | step: 18.68

 10%|█         | 4178/41250 [10:06:29<89:02:43,  8.65s/it]
                                                          
{'loss': 0.2619, 'grad_norm': 2.2571332454681396, 'learning_rate': 3.9469501035127115e-05, 'epoch': 1.01}

 10%|█         | 4178/41250 [10:06:29<89:02:43,  8.65s/it][2025-04-25 18:04:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 0.95
[2025-04-25 18:04:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.25 | bwd_microstep: 5868.58 | bwd_inner_microstep: 5652.55 | bwd_allreduce_microstep: 215.99 | step_microstep: 19.00
[2025-04-25 18:04:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.25 | bwd: 5868.60 | bwd_inner: 5652.55 | bwd_allreduce: 216.00 | step: 19.01

 10%|█         | 4179/41250 [10:06:38<89:27:16,  8.69s/it]
                                                          
{'loss': 0.1018, 'grad_norm': 1.5447839498519897, 'learning_rate': 3.946914169539675e-05, 'epoch': 1.01}

 10%|█         | 4179/41250 [10:06:38<89:27:16,  8.69s/it][2025-04-25 18:04:21,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:04:21,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.85 | bwd_microstep: 5721.56 | bwd_inner_microstep: 5680.97 | bwd_allreduce_microstep: 40.54 | step_microstep: 18.68
[2025-04-25 18:04:21,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.85 | bwd: 5721.57 | bwd_inner: 5680.97 | bwd_allreduce: 40.56 | step: 18.68

 10%|█         | 4180/41250 [10:06:46<89:18:43,  8.67s/it]
                                                          
{'loss': 0.0372, 'grad_norm': 0.8868253827095032, 'learning_rate': 3.946878223564297e-05, 'epoch': 1.01}

 10%|█         | 4180/41250 [10:06:46<89:18:43,  8.67s/it][2025-04-25 18:04:30,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 1.12
[2025-04-25 18:04:30,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.31 | bwd_microstep: 5696.13 | bwd_inner_microstep: 5683.47 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.74
[2025-04-25 18:04:30,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.31 | bwd: 5696.14 | bwd_inner: 5683.47 | bwd_allreduce: 12.63 | step: 18.74

 10%|█         | 4181/41250 [10:06:55<89:09:39,  8.66s/it]
                                                          
{'loss': 0.111, 'grad_norm': 2.586524486541748, 'learning_rate': 3.946842265586797e-05, 'epoch': 1.01}

 10%|█         | 4181/41250 [10:06:55<89:09:39,  8.66s/it][2025-04-25 18:04:39,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:04:39,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.73 | bwd_microstep: 5887.51 | bwd_inner_microstep: 5644.71 | bwd_allreduce_microstep: 242.75 | step_microstep: 18.87
[2025-04-25 18:04:39,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.73 | bwd: 5887.52 | bwd_inner: 5644.71 | bwd_allreduce: 242.77 | step: 18.87

 10%|█         | 4182/41250 [10:07:04<89:33:54,  8.70s/it]
                                                          
{'loss': 0.2207, 'grad_norm': 3.264554023742676, 'learning_rate': 3.946806295607397e-05, 'epoch': 1.01}

 10%|█         | 4182/41250 [10:07:04<89:33:54,  8.70s/it][2025-04-25 18:04:47,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 18:04:47,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.84 | bwd_microstep: 5774.35 | bwd_inner_microstep: 5761.29 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.57
[2025-04-25 18:04:47,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.84 | bwd: 5774.37 | bwd_inner: 5761.29 | bwd_allreduce: 13.03 | step: 18.57

 10%|█         | 4183/41250 [10:07:13<89:42:15,  8.71s/it]
                                                          
{'loss': 0.2048, 'grad_norm': 1.818853497505188, 'learning_rate': 3.946770313626319e-05, 'epoch': 1.01}

 10%|█         | 4183/41250 [10:07:13<89:42:15,  8.71s/it][2025-04-25 18:04:56,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:04:56,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.97 | bwd_microstep: 5778.90 | bwd_inner_microstep: 5765.90 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.95
[2025-04-25 18:04:56,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.97 | bwd: 5778.91 | bwd_inner: 5765.90 | bwd_allreduce: 12.97 | step: 18.95

 10%|█         | 4184/41250 [10:07:21<89:48:38,  8.72s/it]
                                                          
{'loss': 0.0375, 'grad_norm': 0.754041850566864, 'learning_rate': 3.946734319643785e-05, 'epoch': 1.01}

 10%|█         | 4184/41250 [10:07:21<89:48:38,  8.72s/it][2025-04-25 18:05:05,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:05:05,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.85 | bwd_microstep: 5784.06 | bwd_inner_microstep: 5771.13 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.68
[2025-04-25 18:05:05,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.85 | bwd: 5784.08 | bwd_inner: 5771.13 | bwd_allreduce: 12.91 | step: 18.68

 10%|█         | 4185/41250 [10:07:30<89:54:12,  8.73s/it]
                                                          
{'loss': 0.0606, 'grad_norm': 0.6877186894416809, 'learning_rate': 3.946698313660017e-05, 'epoch': 1.01}

 10%|█         | 4185/41250 [10:07:30<89:54:12,  8.73s/it][2025-04-25 18:05:13,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:05:13,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5741.02 | bwd_inner_microstep: 5645.12 | bwd_allreduce_microstep: 95.85 | step_microstep: 19.00
[2025-04-25 18:05:13,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5741.03 | bwd_inner: 5645.12 | bwd_allreduce: 95.87 | step: 19.00

 10%|█         | 4186/41250 [10:07:39<89:38:25,  8.71s/it]
                                                          
{'loss': 0.241, 'grad_norm': 2.102073907852173, 'learning_rate': 3.946662295675237e-05, 'epoch': 1.01}

 10%|█         | 4186/41250 [10:07:39<89:38:25,  8.71s/it][2025-04-25 18:05:22,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 18:05:22,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.62 | bwd_microstep: 5683.49 | bwd_inner_microstep: 5634.05 | bwd_allreduce_microstep: 49.39 | step_microstep: 18.96
[2025-04-25 18:05:22,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.62 | bwd: 5683.51 | bwd_inner: 5634.05 | bwd_allreduce: 49.42 | step: 18.97

 10%|█         | 4187/41250 [10:07:47<89:17:49,  8.67s/it]
                                                          
{'loss': 0.1497, 'grad_norm': 1.5247257947921753, 'learning_rate': 3.946626265689667e-05, 'epoch': 1.02}

 10%|█         | 4187/41250 [10:07:47<89:17:49,  8.67s/it][2025-04-25 18:05:31,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 18:05:31,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5695.41 | bwd_inner_microstep: 5682.53 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.88
[2025-04-25 18:05:31,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.42 | bwd: 5695.43 | bwd_inner: 5682.53 | bwd_allreduce: 12.86 | step: 18.89

 10%|█         | 4188/41250 [10:07:56<89:07:47,  8.66s/it]
                                                          
{'loss': 0.0561, 'grad_norm': 0.6838136315345764, 'learning_rate': 3.946590223703528e-05, 'epoch': 1.02}

 10%|█         | 4188/41250 [10:07:56<89:07:47,  8.66s/it][2025-04-25 18:05:39,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 18:05:39,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.40 | bwd_microstep: 5678.03 | bwd_inner_microstep: 5646.05 | bwd_allreduce_microstep: 31.93 | step_microstep: 18.44
[2025-04-25 18:05:39,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.40 | bwd: 5678.04 | bwd_inner: 5646.05 | bwd_allreduce: 31.95 | step: 18.44

 10%|█         | 4189/41250 [10:08:05<88:56:01,  8.64s/it]
                                                          
{'loss': 0.2374, 'grad_norm': 3.00146222114563, 'learning_rate': 3.946554169717044e-05, 'epoch': 1.02}

 10%|█         | 4189/41250 [10:08:05<88:56:01,  8.64s/it][2025-04-25 18:05:48,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:05:48,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.41 | bwd_microstep: 5728.99 | bwd_inner_microstep: 5700.08 | bwd_allreduce_microstep: 28.87 | step_microstep: 18.51
[2025-04-25 18:05:48,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.41 | bwd: 5729.01 | bwd_inner: 5700.08 | bwd_allreduce: 28.89 | step: 18.51

 10%|█         | 4190/41250 [10:08:13<89:00:48,  8.65s/it]
                                                          
{'loss': 0.0283, 'grad_norm': 0.8433277010917664, 'learning_rate': 3.9465181037304365e-05, 'epoch': 1.02}

 10%|█         | 4190/41250 [10:08:13<89:00:48,  8.65s/it][2025-04-25 18:05:56,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 18:05:56,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.84 | bwd_microstep: 5683.37 | bwd_inner_microstep: 5662.98 | bwd_allreduce_microstep: 20.35 | step_microstep: 19.23
[2025-04-25 18:05:56,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.84 | bwd: 5683.39 | bwd_inner: 5662.98 | bwd_allreduce: 20.37 | step: 19.23

 10%|█         | 4191/41250 [10:08:22<88:53:39,  8.64s/it]
                                                          
{'loss': 0.0372, 'grad_norm': 1.0760380029678345, 'learning_rate': 3.946482025743927e-05, 'epoch': 1.02}

 10%|█         | 4191/41250 [10:08:22<88:53:39,  8.64s/it][2025-04-25 18:06:05,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 18:06:05,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3084.09 | bwd_microstep: 5652.60 | bwd_inner_microstep: 5639.79 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.87
[2025-04-25 18:06:05,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3084.09 | bwd: 5652.61 | bwd_inner: 5639.79 | bwd_allreduce: 12.78 | step: 18.87

 10%|█         | 4192/41250 [10:08:31<89:27:54,  8.69s/it]
                                                          
{'loss': 0.0519, 'grad_norm': 0.9165551066398621, 'learning_rate': 3.9464459357577394e-05, 'epoch': 1.02}

 10%|█         | 4192/41250 [10:08:31<89:27:54,  8.69s/it][2025-04-25 18:06:14,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 18:06:14,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.99 | bwd_microstep: 5737.83 | bwd_inner_microstep: 5710.48 | bwd_allreduce_microstep: 27.30 | step_microstep: 18.93
[2025-04-25 18:06:14,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.99 | bwd: 5737.84 | bwd_inner: 5710.48 | bwd_allreduce: 27.32 | step: 18.93

 10%|█         | 4193/41250 [10:08:39<89:24:23,  8.69s/it]
                                                          
{'loss': 0.3023, 'grad_norm': 2.790635108947754, 'learning_rate': 3.9464098337720955e-05, 'epoch': 1.02}

 10%|█         | 4193/41250 [10:08:39<89:24:23,  8.69s/it][2025-04-25 18:06:23,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:06:23,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.75 | bwd_microstep: 5753.12 | bwd_inner_microstep: 5664.17 | bwd_allreduce_microstep: 88.90 | step_microstep: 18.73
[2025-04-25 18:06:23,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.75 | bwd: 5753.13 | bwd_inner: 5664.17 | bwd_allreduce: 88.92 | step: 18.74

 10%|█         | 4194/41250 [10:08:48<89:19:58,  8.68s/it]
                                                          
{'loss': 0.1288, 'grad_norm': 1.1642423868179321, 'learning_rate': 3.946373719787217e-05, 'epoch': 1.02}

 10%|█         | 4194/41250 [10:08:48<89:19:58,  8.68s/it][2025-04-25 18:06:31,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 18:06:31,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.27 | bwd_microstep: 5767.66 | bwd_inner_microstep: 5685.11 | bwd_allreduce_microstep: 82.51 | step_microstep: 18.46
[2025-04-25 18:06:31,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.27 | bwd: 5767.67 | bwd_inner: 5685.10 | bwd_allreduce: 82.53 | step: 18.47

 10%|█         | 4195/41250 [10:08:57<89:26:15,  8.69s/it]
                                                          
{'loss': 0.0491, 'grad_norm': 0.8368769288063049, 'learning_rate': 3.946337593803328e-05, 'epoch': 1.02}

 10%|█         | 4195/41250 [10:08:57<89:26:15,  8.69s/it][2025-04-25 18:06:40,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 18:06:40,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.85 | bwd_microstep: 5699.21 | bwd_inner_microstep: 5686.29 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.04
[2025-04-25 18:06:40,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.85 | bwd: 5699.22 | bwd_inner: 5686.28 | bwd_allreduce: 12.89 | step: 19.04

 10%|█         | 4196/41250 [10:09:05<89:17:24,  8.68s/it]
                                                          
{'loss': 0.0779, 'grad_norm': 0.9053632020950317, 'learning_rate': 3.94630145582065e-05, 'epoch': 1.02}

 10%|█         | 4196/41250 [10:09:05<89:17:24,  8.68s/it][2025-04-25 18:06:49,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.13
[2025-04-25 18:06:49,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.74 | bwd_microstep: 5763.10 | bwd_inner_microstep: 5659.86 | bwd_allreduce_microstep: 103.19 | step_microstep: 19.36
[2025-04-25 18:06:49,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.74 | bwd: 5763.11 | bwd_inner: 5659.86 | bwd_allreduce: 103.21 | step: 19.36

 10%|█         | 4197/41250 [10:09:14<89:19:09,  8.68s/it]
                                                          
{'loss': 0.1546, 'grad_norm': 2.071751356124878, 'learning_rate': 3.946265305839407e-05, 'epoch': 1.02}

 10%|█         | 4197/41250 [10:09:14<89:19:09,  8.68s/it][2025-04-25 18:06:57,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.08
[2025-04-25 18:06:57,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.43 | bwd_microstep: 5783.24 | bwd_inner_microstep: 5659.36 | bwd_allreduce_microstep: 123.82 | step_microstep: 20.01
[2025-04-25 18:06:57,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.43 | bwd: 5783.25 | bwd_inner: 5659.36 | bwd_allreduce: 123.84 | step: 20.01

 10%|█         | 4198/41250 [10:09:23<89:23:01,  8.68s/it]
                                                          
{'loss': 0.1573, 'grad_norm': 2.034393072128296, 'learning_rate': 3.946229143859821e-05, 'epoch': 1.02}

 10%|█         | 4198/41250 [10:09:23<89:23:01,  8.68s/it][2025-04-25 18:07:06,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-25 18:07:06,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.26 | bwd_microstep: 5747.60 | bwd_inner_microstep: 5643.06 | bwd_allreduce_microstep: 104.48 | step_microstep: 19.04
[2025-04-25 18:07:06,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.26 | bwd: 5747.61 | bwd_inner: 5643.06 | bwd_allreduce: 104.51 | step: 19.04

 10%|█         | 4199/41250 [10:09:31<89:21:28,  8.68s/it]
                                                          
{'loss': 0.116, 'grad_norm': 1.955825924873352, 'learning_rate': 3.946192969882115e-05, 'epoch': 1.02}

 10%|█         | 4199/41250 [10:09:31<89:21:28,  8.68s/it][2025-04-25 18:07:15,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.10 | optimizer_step: 0.97
[2025-04-25 18:07:15,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.89 | bwd_microstep: 5834.92 | bwd_inner_microstep: 5788.37 | bwd_allreduce_microstep: 46.50 | step_microstep: 19.84
[2025-04-25 18:07:15,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.89 | bwd: 5834.94 | bwd_inner: 5788.37 | bwd_allreduce: 46.52 | step: 19.85

 10%|█         | 4200/41250 [10:09:40<89:45:31,  8.72s/it]
                                                          
{'loss': 0.1984, 'grad_norm': 2.021057605743408, 'learning_rate': 3.9461567839065117e-05, 'epoch': 1.02}

 10%|█         | 4200/41250 [10:09:40<89:45:31,  8.72s/it][2025-04-25 18:07:24,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.07 | optimizer_step: 1.11
[2025-04-25 18:07:24,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.84 | bwd_microstep: 5768.34 | bwd_inner_microstep: 5664.03 | bwd_allreduce_microstep: 104.26 | step_microstep: 19.26
[2025-04-25 18:07:24,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.84 | bwd: 5768.36 | bwd_inner: 5664.02 | bwd_allreduce: 104.28 | step: 19.26

 10%|█         | 4201/41250 [10:09:49<89:40:11,  8.71s/it]
                                                          
{'loss': 0.1897, 'grad_norm': 1.534706473350525, 'learning_rate': 3.9461205859332355e-05, 'epoch': 1.02}

 10%|█         | 4201/41250 [10:09:49<89:40:11,  8.71s/it][2025-04-25 18:07:32,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.07 | optimizer_step: 1.24
[2025-04-25 18:07:32,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.30 | bwd_microstep: 5719.29 | bwd_inner_microstep: 5706.58 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.70
[2025-04-25 18:07:32,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.30 | bwd: 5719.31 | bwd_inner: 5706.58 | bwd_allreduce: 12.68 | step: 19.70

 10%|█         | 4202/41250 [10:09:58<89:31:37,  8.70s/it]
                                                          
{'loss': 0.044, 'grad_norm': 1.0958623886108398, 'learning_rate': 3.946084375962509e-05, 'epoch': 1.02}

 10%|█         | 4202/41250 [10:09:58<89:31:37,  8.70s/it][2025-04-25 18:07:41,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:07:41,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.45 | bwd_microstep: 5747.08 | bwd_inner_microstep: 5724.05 | bwd_allreduce_microstep: 22.98 | step_microstep: 18.83
[2025-04-25 18:07:41,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.45 | bwd: 5747.09 | bwd_inner: 5724.05 | bwd_allreduce: 23.00 | step: 18.84

 10%|█         | 4203/41250 [10:10:06<89:31:07,  8.70s/it]
                                                          
{'loss': 0.131, 'grad_norm': 1.4508492946624756, 'learning_rate': 3.946048153994554e-05, 'epoch': 1.02}

 10%|█         | 4203/41250 [10:10:06<89:31:07,  8.70s/it][2025-04-25 18:07:50,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:07:50,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.02 | bwd_microstep: 5738.58 | bwd_inner_microstep: 5720.89 | bwd_allreduce_microstep: 17.64 | step_microstep: 19.15
[2025-04-25 18:07:50,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.02 | bwd: 5738.59 | bwd_inner: 5720.89 | bwd_allreduce: 17.66 | step: 19.16

 10%|█         | 4204/41250 [10:10:15<89:29:24,  8.70s/it]
                                                          
{'loss': 0.0446, 'grad_norm': 0.5830694437026978, 'learning_rate': 3.946011920029595e-05, 'epoch': 1.02}

 10%|█         | 4204/41250 [10:10:15<89:29:24,  8.70s/it][2025-04-25 18:07:59,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:07:59,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.73 | bwd_microstep: 6037.47 | bwd_inner_microstep: 5652.23 | bwd_allreduce_microstep: 385.20 | step_microstep: 18.54
[2025-04-25 18:07:59,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.73 | bwd: 6037.49 | bwd_inner: 5652.23 | bwd_allreduce: 385.22 | step: 18.55

 10%|█         | 4205/41250 [10:10:24<90:17:49,  8.77s/it]
                                                          
{'loss': 0.2696, 'grad_norm': 2.912604331970215, 'learning_rate': 3.945975674067855e-05, 'epoch': 1.02}

 10%|█         | 4205/41250 [10:10:24<90:17:49,  8.77s/it][2025-04-25 18:08:07,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:08:07,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.74 | bwd_microstep: 5853.06 | bwd_inner_microstep: 5717.46 | bwd_allreduce_microstep: 135.54 | step_microstep: 18.77
[2025-04-25 18:08:07,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.74 | bwd: 5853.07 | bwd_inner: 5717.46 | bwd_allreduce: 135.56 | step: 18.77

 10%|█         | 4206/41250 [10:10:33<90:21:34,  8.78s/it]
                                                          
{'loss': 0.315, 'grad_norm': 1.664613962173462, 'learning_rate': 3.945939416109559e-05, 'epoch': 1.02}

 10%|█         | 4206/41250 [10:10:33<90:21:34,  8.78s/it][2025-04-25 18:08:16,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 18:08:16,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.55 | bwd_microstep: 5736.51 | bwd_inner_microstep: 5723.70 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.07
[2025-04-25 18:08:16,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.55 | bwd: 5736.53 | bwd_inner: 5723.70 | bwd_allreduce: 12.79 | step: 19.07

 10%|█         | 4207/41250 [10:10:41<90:03:55,  8.75s/it]
                                                          
{'loss': 0.2255, 'grad_norm': 2.1335668563842773, 'learning_rate': 3.945903146154928e-05, 'epoch': 1.02}

 10%|█         | 4207/41250 [10:10:41<90:03:55,  8.75s/it][2025-04-25 18:08:25,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 18:08:25,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.54 | bwd_microstep: 5722.99 | bwd_inner_microstep: 5710.51 | bwd_allreduce_microstep: 12.44 | step_microstep: 19.00
[2025-04-25 18:08:25,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.54 | bwd: 5723.01 | bwd_inner: 5710.51 | bwd_allreduce: 12.46 | step: 19.00

 10%|█         | 4208/41250 [10:10:50<89:49:22,  8.73s/it]
                                                          
{'loss': 0.1128, 'grad_norm': 1.2596015930175781, 'learning_rate': 3.9458668642041875e-05, 'epoch': 1.02}

 10%|█         | 4208/41250 [10:10:50<89:49:22,  8.73s/it][2025-04-25 18:08:33,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:08:33,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.39 | bwd_microstep: 5717.87 | bwd_inner_microstep: 5704.92 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.62
[2025-04-25 18:08:33,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.39 | bwd: 5717.88 | bwd_inner: 5704.92 | bwd_allreduce: 12.92 | step: 18.63

 10%|█         | 4209/41250 [10:10:59<89:39:11,  8.71s/it]
                                                          
{'loss': 0.1353, 'grad_norm': 3.8359878063201904, 'learning_rate': 3.9458305702575607e-05, 'epoch': 1.02}

 10%|█         | 4209/41250 [10:10:59<89:39:11,  8.71s/it][2025-04-25 18:08:42,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:08:42,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.93 | bwd_microstep: 5717.90 | bwd_inner_microstep: 5654.78 | bwd_allreduce_microstep: 63.07 | step_microstep: 18.87
[2025-04-25 18:08:42,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.93 | bwd: 5717.91 | bwd_inner: 5654.78 | bwd_allreduce: 63.09 | step: 18.87

 10%|█         | 4210/41250 [10:11:07<89:23:39,  8.69s/it]
                                                          
{'loss': 0.2175, 'grad_norm': 1.7950379848480225, 'learning_rate': 3.94579426431527e-05, 'epoch': 1.02}

 10%|█         | 4210/41250 [10:11:07<89:23:39,  8.69s/it][2025-04-25 18:08:51,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:08:51,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.39 | bwd_microstep: 5778.98 | bwd_inner_microstep: 5650.69 | bwd_allreduce_microstep: 128.25 | step_microstep: 18.62
[2025-04-25 18:08:51,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.39 | bwd: 5778.99 | bwd_inner: 5650.69 | bwd_allreduce: 128.26 | step: 18.62

 10%|█         | 4211/41250 [10:11:16<89:23:55,  8.69s/it]
                                                          
{'loss': 0.2346, 'grad_norm': 3.5496091842651367, 'learning_rate': 3.945757946377542e-05, 'epoch': 1.02}

 10%|█         | 4211/41250 [10:11:16<89:23:55,  8.69s/it][2025-04-25 18:08:59,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:08:59,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.94 | bwd_microstep: 5771.51 | bwd_inner_microstep: 5665.01 | bwd_allreduce_microstep: 106.44 | step_microstep: 18.57
[2025-04-25 18:08:59,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.94 | bwd: 5771.52 | bwd_inner: 5665.01 | bwd_allreduce: 106.46 | step: 18.57

 10%|█         | 4212/41250 [10:11:25<89:25:32,  8.69s/it]
                                                          
{'loss': 0.0522, 'grad_norm': 0.5724160671234131, 'learning_rate': 3.9457216164445974e-05, 'epoch': 1.02}

 10%|█         | 4212/41250 [10:11:25<89:25:32,  8.69s/it][2025-04-25 18:09:08,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 18:09:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.96 | bwd_microstep: 5876.31 | bwd_inner_microstep: 5650.41 | bwd_allreduce_microstep: 225.85 | step_microstep: 19.17
[2025-04-25 18:09:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.96 | bwd: 5876.32 | bwd_inner: 5650.41 | bwd_allreduce: 225.87 | step: 19.17

 10%|█         | 4213/41250 [10:11:34<89:46:37,  8.73s/it]
                                                          
{'loss': 0.1156, 'grad_norm': 2.5822794437408447, 'learning_rate': 3.9456852745166624e-05, 'epoch': 1.02}

 10%|█         | 4213/41250 [10:11:34<89:46:37,  8.73s/it][2025-04-25 18:09:17,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 18:09:17,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.56 | bwd_microstep: 5733.66 | bwd_inner_microstep: 5697.19 | bwd_allreduce_microstep: 36.43 | step_microstep: 18.59
[2025-04-25 18:09:17,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.56 | bwd: 5733.68 | bwd_inner: 5697.19 | bwd_allreduce: 36.45 | step: 18.59

 10%|█         | 4214/41250 [10:11:42<89:37:43,  8.71s/it]
                                                          
{'loss': 0.1478, 'grad_norm': 2.150280714035034, 'learning_rate': 3.945648920593961e-05, 'epoch': 1.02}

 10%|█         | 4214/41250 [10:11:42<89:37:43,  8.71s/it][2025-04-25 18:09:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 18:09:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.20 | bwd_microstep: 5731.10 | bwd_inner_microstep: 5697.52 | bwd_allreduce_microstep: 33.53 | step_microstep: 18.81
[2025-04-25 18:09:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.20 | bwd: 5731.11 | bwd_inner: 5697.52 | bwd_allreduce: 33.55 | step: 18.81

 10%|█         | 4215/41250 [10:11:51<89:31:01,  8.70s/it]
                                                          
{'loss': 0.1028, 'grad_norm': 2.2400944232940674, 'learning_rate': 3.945612554676716e-05, 'epoch': 1.02}

 10%|█         | 4215/41250 [10:11:51<89:31:01,  8.70s/it][2025-04-25 18:09:34,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 18:09:34,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.81 | bwd_microstep: 5793.65 | bwd_inner_microstep: 5780.85 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.70
[2025-04-25 18:09:34,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.81 | bwd: 5793.66 | bwd_inner: 5780.85 | bwd_allreduce: 12.77 | step: 18.70

 10%|█         | 4216/41250 [10:12:00<89:43:36,  8.72s/it]
                                                          
{'loss': 0.2295, 'grad_norm': 1.7833269834518433, 'learning_rate': 3.945576176765152e-05, 'epoch': 1.02}

 10%|█         | 4216/41250 [10:12:00<89:43:36,  8.72s/it][2025-04-25 18:09:43,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:09:43,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.25 | bwd_microstep: 5709.47 | bwd_inner_microstep: 5696.48 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.94
[2025-04-25 18:09:43,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.25 | bwd: 5709.49 | bwd_inner: 5696.48 | bwd_allreduce: 12.97 | step: 18.95

 10%|█         | 4217/41250 [10:12:08<89:28:56,  8.70s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 1.1248224973678589, 'learning_rate': 3.9455397868594944e-05, 'epoch': 1.02}

 10%|█         | 4217/41250 [10:12:08<89:28:56,  8.70s/it][2025-04-25 18:09:52,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 18:09:52,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5700.80 | bwd_inner_microstep: 5668.57 | bwd_allreduce_microstep: 32.19 | step_microstep: 18.58
[2025-04-25 18:09:52,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.99 | bwd: 5700.81 | bwd_inner: 5668.57 | bwd_allreduce: 32.20 | step: 18.59

 10%|█         | 4218/41250 [10:12:17<89:13:06,  8.67s/it]
                                                          
{'loss': 0.3929, 'grad_norm': 2.5217535495758057, 'learning_rate': 3.945503384959966e-05, 'epoch': 1.02}

 10%|█         | 4218/41250 [10:12:17<89:13:06,  8.67s/it][2025-04-25 18:10:00,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:10:00,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.26 | bwd_microstep: 5700.45 | bwd_inner_microstep: 5648.89 | bwd_allreduce_microstep: 51.52 | step_microstep: 18.82
[2025-04-25 18:10:00,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.26 | bwd: 5700.47 | bwd_inner: 5648.89 | bwd_allreduce: 51.54 | step: 18.82

 10%|█         | 4219/41250 [10:12:26<89:02:38,  8.66s/it]
                                                          
{'loss': 0.047, 'grad_norm': 1.6845982074737549, 'learning_rate': 3.945466971066792e-05, 'epoch': 1.02}

 10%|█         | 4219/41250 [10:12:26<89:02:38,  8.66s/it][2025-04-25 18:10:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:10:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.02 | bwd_microstep: 5766.49 | bwd_inner_microstep: 5647.19 | bwd_allreduce_microstep: 119.25 | step_microstep: 18.81
[2025-04-25 18:10:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.02 | bwd: 5766.50 | bwd_inner: 5647.19 | bwd_allreduce: 119.27 | step: 18.81

 10%|█         | 4220/41250 [10:12:34<89:07:32,  8.66s/it]
                                                          
{'loss': 0.2931, 'grad_norm': 2.276106595993042, 'learning_rate': 3.945430545180197e-05, 'epoch': 1.02}

 10%|█         | 4220/41250 [10:12:34<89:07:32,  8.66s/it][2025-04-25 18:10:18,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:10:18,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.32 | bwd_microstep: 5738.72 | bwd_inner_microstep: 5673.80 | bwd_allreduce_microstep: 64.87 | step_microstep: 18.85
[2025-04-25 18:10:18,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.32 | bwd: 5738.74 | bwd_inner: 5673.80 | bwd_allreduce: 64.89 | step: 18.85

 10%|█         | 4221/41250 [10:12:43<89:08:24,  8.67s/it]
                                                          
{'loss': 0.0809, 'grad_norm': 1.3934506177902222, 'learning_rate': 3.945394107300406e-05, 'epoch': 1.02}

 10%|█         | 4221/41250 [10:12:43<89:08:24,  8.67s/it][2025-04-25 18:10:26,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.06 | optimizer_step: 1.16
[2025-04-25 18:10:26,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.83 | bwd_microstep: 5691.72 | bwd_inner_microstep: 5654.11 | bwd_allreduce_microstep: 37.54 | step_microstep: 19.73
[2025-04-25 18:10:26,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.83 | bwd: 5691.73 | bwd_inner: 5654.11 | bwd_allreduce: 37.57 | step: 19.73

 10%|█         | 4222/41250 [10:12:52<88:59:15,  8.65s/it]
                                                          
{'loss': 0.1893, 'grad_norm': 2.4775314331054688, 'learning_rate': 3.945357657427642e-05, 'epoch': 1.02}

 10%|█         | 4222/41250 [10:12:52<88:59:15,  8.65s/it][2025-04-25 18:10:35,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 18:10:35,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.05 | bwd_microstep: 5675.06 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 17.42 | step_microstep: 19.30
[2025-04-25 18:10:35,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.05 | bwd: 5675.08 | bwd_inner: 5657.59 | bwd_allreduce: 17.44 | step: 19.30

 10%|█         | 4223/41250 [10:13:00<88:47:32,  8.63s/it]
                                                          
{'loss': 0.1513, 'grad_norm': 1.8368407487869263, 'learning_rate': 3.945321195562131e-05, 'epoch': 1.02}

 10%|█         | 4223/41250 [10:13:00<88:47:32,  8.63s/it][2025-04-25 18:10:43,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:10:43,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.31 | bwd_microstep: 5738.74 | bwd_inner_microstep: 5683.10 | bwd_allreduce_microstep: 55.59 | step_microstep: 18.44
[2025-04-25 18:10:43,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5738.75 | bwd_inner: 5683.10 | bwd_allreduce: 55.61 | step: 18.44

 10%|█         | 4224/41250 [10:13:09<88:54:32,  8.64s/it]
                                                          
{'loss': 0.117, 'grad_norm': 2.159970998764038, 'learning_rate': 3.9452847217040976e-05, 'epoch': 1.02}

 10%|█         | 4224/41250 [10:13:09<88:54:32,  8.64s/it][2025-04-25 18:10:52,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 18:10:52,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.91 | bwd_microstep: 5717.85 | bwd_inner_microstep: 5705.08 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.71
[2025-04-25 18:10:52,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.91 | bwd: 5717.86 | bwd_inner: 5705.08 | bwd_allreduce: 12.74 | step: 18.71

 10%|█         | 4225/41250 [10:13:17<88:55:35,  8.65s/it]
                                                          
{'loss': 0.1362, 'grad_norm': 1.8845840692520142, 'learning_rate': 3.9452482358537655e-05, 'epoch': 1.02}

 10%|█         | 4225/41250 [10:13:17<88:55:35,  8.65s/it][2025-04-25 18:11:01,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:11:01,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5728.44 | bwd_inner_microstep: 5679.40 | bwd_allreduce_microstep: 48.99 | step_microstep: 18.57
[2025-04-25 18:11:01,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5728.45 | bwd_inner: 5679.40 | bwd_allreduce: 49.01 | step: 18.58

 10%|█         | 4226/41250 [10:13:26<88:58:52,  8.65s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 2.1824045181274414, 'learning_rate': 3.945211738011362e-05, 'epoch': 1.02}

 10%|█         | 4226/41250 [10:13:26<88:58:52,  8.65s/it][2025-04-25 18:11:09,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:11:09,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.69 | bwd_microstep: 5684.12 | bwd_inner_microstep: 5641.79 | bwd_allreduce_microstep: 42.29 | step_microstep: 18.64
[2025-04-25 18:11:09,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.69 | bwd: 5684.13 | bwd_inner: 5641.79 | bwd_allreduce: 42.30 | step: 18.64

 10%|█         | 4227/41250 [10:13:35<88:51:17,  8.64s/it]
                                                          
{'loss': 0.2321, 'grad_norm': 1.972327709197998, 'learning_rate': 3.9451752281771096e-05, 'epoch': 1.02}

 10%|█         | 4227/41250 [10:13:35<88:51:17,  8.64s/it][2025-04-25 18:11:18,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:11:18,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.03 | bwd_microstep: 5722.73 | bwd_inner_microstep: 5709.90 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.43
[2025-04-25 18:11:18,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.03 | bwd: 5722.74 | bwd_inner: 5709.91 | bwd_allreduce: 12.80 | step: 18.43

 10%|█         | 4228/41250 [10:13:43<88:55:21,  8.65s/it]
                                                          
{'loss': 0.1322, 'grad_norm': 0.9054320454597473, 'learning_rate': 3.945138706351236e-05, 'epoch': 1.02}

 10%|█         | 4228/41250 [10:13:43<88:55:21,  8.65s/it][2025-04-25 18:11:27,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 18:11:27,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 6063.37 | bwd_inner_microstep: 5687.07 | bwd_allreduce_microstep: 376.25 | step_microstep: 18.64
[2025-04-25 18:11:27,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.17 | bwd: 6063.38 | bwd_inner: 5687.07 | bwd_allreduce: 376.27 | step: 18.65

 10%|█         | 4229/41250 [10:13:52<89:59:09,  8.75s/it]
                                                          
{'loss': 0.0639, 'grad_norm': 0.4967726469039917, 'learning_rate': 3.945102172533963e-05, 'epoch': 1.03}

 10%|█         | 4229/41250 [10:13:52<89:59:09,  8.75s/it][2025-04-25 18:11:36,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 18:11:36,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5720.67 | bwd_inner_microstep: 5689.66 | bwd_allreduce_microstep: 30.95 | step_microstep: 18.66
[2025-04-25 18:11:36,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5720.68 | bwd_inner: 5689.66 | bwd_allreduce: 30.98 | step: 18.66

 10%|█         | 4230/41250 [10:14:01<89:41:25,  8.72s/it]
                                                          
{'loss': 0.1186, 'grad_norm': 1.6046632528305054, 'learning_rate': 3.945065626725519e-05, 'epoch': 1.03}

 10%|█         | 4230/41250 [10:14:01<89:41:25,  8.72s/it][2025-04-25 18:11:44,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 18:11:44,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.97 | bwd_microstep: 5687.12 | bwd_inner_microstep: 5674.38 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.77
[2025-04-25 18:11:44,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.97 | bwd: 5687.14 | bwd_inner: 5674.38 | bwd_allreduce: 12.71 | step: 18.77

 10%|█         | 4231/41250 [10:14:10<89:24:29,  8.69s/it]
                                                          
{'loss': 0.0621, 'grad_norm': 0.9168894290924072, 'learning_rate': 3.9450290689261274e-05, 'epoch': 1.03}

 10%|█         | 4231/41250 [10:14:10<89:24:29,  8.69s/it][2025-04-25 18:11:53,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 18:11:53,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5722.13 | bwd_inner_microstep: 5680.21 | bwd_allreduce_microstep: 41.88 | step_microstep: 18.79
[2025-04-25 18:11:53,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5722.14 | bwd_inner: 5680.21 | bwd_allreduce: 41.90 | step: 18.79

 10%|█         | 4232/41250 [10:14:18<89:18:12,  8.68s/it]
                                                          
{'loss': 0.3139, 'grad_norm': 2.5868072509765625, 'learning_rate': 3.944992499136015e-05, 'epoch': 1.03}

 10%|█         | 4232/41250 [10:14:18<89:18:12,  8.68s/it][2025-04-25 18:12:02,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-25 18:12:02,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5731.30 | bwd_inner_microstep: 5689.21 | bwd_allreduce_microstep: 42.05 | step_microstep: 18.84
[2025-04-25 18:12:02,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5731.31 | bwd_inner: 5689.21 | bwd_allreduce: 42.07 | step: 18.84

 10%|█         | 4233/41250 [10:14:27<89:15:18,  8.68s/it]
                                                          
{'loss': 0.2273, 'grad_norm': 3.0464329719543457, 'learning_rate': 3.944955917355405e-05, 'epoch': 1.03}

 10%|█         | 4233/41250 [10:14:27<89:15:18,  8.68s/it][2025-04-25 18:12:10,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:12:10,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5690.04 | bwd_inner_microstep: 5649.87 | bwd_allreduce_microstep: 40.12 | step_microstep: 18.68
[2025-04-25 18:12:10,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5690.05 | bwd_inner: 5649.87 | bwd_allreduce: 40.14 | step: 18.69

 10%|█         | 4234/41250 [10:14:36<89:00:13,  8.66s/it]
                                                          
{'loss': 0.1933, 'grad_norm': 1.6493109464645386, 'learning_rate': 3.9449193235845254e-05, 'epoch': 1.03}

 10%|█         | 4234/41250 [10:14:36<89:00:13,  8.66s/it][2025-04-25 18:12:19,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:12:19,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.43 | bwd_microstep: 5715.10 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.56
[2025-04-25 18:12:19,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.43 | bwd: 5715.11 | bwd_inner: 5702.41 | bwd_allreduce: 12.66 | step: 18.56

 10%|█         | 4235/41250 [10:14:44<88:59:36,  8.66s/it]
                                                          
{'loss': 0.0285, 'grad_norm': 0.5292505025863647, 'learning_rate': 3.9448827178236e-05, 'epoch': 1.03}

 10%|█         | 4235/41250 [10:14:44<88:59:36,  8.66s/it][2025-04-25 18:12:28,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:12:28,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.10 | bwd_microstep: 5720.34 | bwd_inner_microstep: 5685.29 | bwd_allreduce_microstep: 35.00 | step_microstep: 18.65
[2025-04-25 18:12:28,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.11 | bwd: 5720.35 | bwd_inner: 5685.29 | bwd_allreduce: 35.02 | step: 18.66

 10%|█         | 4236/41250 [10:14:53<89:00:16,  8.66s/it]
                                                          
{'loss': 0.1028, 'grad_norm': 0.9120008945465088, 'learning_rate': 3.944846100072856e-05, 'epoch': 1.03}

 10%|█         | 4236/41250 [10:14:53<89:00:16,  8.66s/it][2025-04-25 18:12:36,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:12:36,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.28 | bwd_microstep: 5750.45 | bwd_inner_microstep: 5692.27 | bwd_allreduce_microstep: 58.14 | step_microstep: 18.44
[2025-04-25 18:12:36,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.28 | bwd: 5750.46 | bwd_inner: 5692.27 | bwd_allreduce: 58.15 | step: 18.44

 10%|█         | 4237/41250 [10:15:02<89:06:59,  8.67s/it]
                                                          
{'loss': 0.0605, 'grad_norm': 0.9694133400917053, 'learning_rate': 3.944809470332518e-05, 'epoch': 1.03}

 10%|█         | 4237/41250 [10:15:02<89:06:59,  8.67s/it][2025-04-25 18:12:45,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 18:12:45,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.86 | bwd_microstep: 5746.19 | bwd_inner_microstep: 5653.28 | bwd_allreduce_microstep: 92.87 | step_microstep: 18.49
[2025-04-25 18:12:45,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.86 | bwd: 5746.21 | bwd_inner: 5653.28 | bwd_allreduce: 92.88 | step: 18.50

 10%|█         | 4238/41250 [10:15:10<89:06:25,  8.67s/it]
                                                          
{'loss': 0.1495, 'grad_norm': 1.792251706123352, 'learning_rate': 3.944772828602812e-05, 'epoch': 1.03}

 10%|█         | 4238/41250 [10:15:10<89:06:25,  8.67s/it][2025-04-25 18:12:54,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:12:54,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.03 | bwd_microstep: 5807.43 | bwd_inner_microstep: 5767.83 | bwd_allreduce_microstep: 39.54 | step_microstep: 18.45
[2025-04-25 18:12:54,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.03 | bwd: 5807.44 | bwd_inner: 5767.83 | bwd_allreduce: 39.57 | step: 18.45

 10%|█         | 4239/41250 [10:15:19<89:28:10,  8.70s/it]
                                                          
{'loss': 0.2474, 'grad_norm': 1.8370968103408813, 'learning_rate': 3.944736174883964e-05, 'epoch': 1.03}

 10%|█         | 4239/41250 [10:15:19<89:28:10,  8.70s/it][2025-04-25 18:13:02,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-25 18:13:02,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5733.44 | bwd_inner_microstep: 5650.90 | bwd_allreduce_microstep: 82.50 | step_microstep: 19.12
[2025-04-25 18:13:02,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5733.46 | bwd_inner: 5650.90 | bwd_allreduce: 82.51 | step: 19.12

 10%|█         | 4240/41250 [10:15:28<89:20:18,  8.69s/it]
                                                          
{'loss': 0.2255, 'grad_norm': 4.632996082305908, 'learning_rate': 3.9446995091762e-05, 'epoch': 1.03}

 10%|█         | 4240/41250 [10:15:28<89:20:18,  8.69s/it][2025-04-25 18:13:11,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:13:11,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.66 | bwd_microstep: 5753.16 | bwd_inner_microstep: 5693.19 | bwd_allreduce_microstep: 59.93 | step_microstep: 18.47
[2025-04-25 18:13:11,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.66 | bwd: 5753.17 | bwd_inner: 5693.19 | bwd_allreduce: 59.95 | step: 18.48

 10%|█         | 4241/41250 [10:15:36<89:20:50,  8.69s/it]
                                                          
{'loss': 0.0609, 'grad_norm': 0.6934396028518677, 'learning_rate': 3.9446628314797456e-05, 'epoch': 1.03}

 10%|█         | 4241/41250 [10:15:36<89:20:50,  8.69s/it][2025-04-25 18:13:20,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 18:13:20,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.38 | bwd_microstep: 5773.34 | bwd_inner_microstep: 5656.52 | bwd_allreduce_microstep: 116.77 | step_microstep: 18.65
[2025-04-25 18:13:20,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.38 | bwd: 5773.35 | bwd_inner: 5656.52 | bwd_allreduce: 116.79 | step: 18.65

 10%|█         | 4242/41250 [10:15:45<89:22:32,  8.69s/it]
                                                          
{'loss': 0.2469, 'grad_norm': 1.8963345289230347, 'learning_rate': 3.9446261417948284e-05, 'epoch': 1.03}

 10%|█         | 4242/41250 [10:15:45<89:22:32,  8.69s/it][2025-04-25 18:13:28,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:13:28,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.49 | bwd_microstep: 5763.91 | bwd_inner_microstep: 5669.30 | bwd_allreduce_microstep: 94.56 | step_microstep: 18.38
[2025-04-25 18:13:28,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.49 | bwd: 5763.92 | bwd_inner: 5669.30 | bwd_allreduce: 94.58 | step: 18.38

 10%|█         | 4243/41250 [10:15:54<89:21:06,  8.69s/it]
                                                          
{'loss': 0.0451, 'grad_norm': 0.7321364879608154, 'learning_rate': 3.9445894401216723e-05, 'epoch': 1.03}

 10%|█         | 4243/41250 [10:15:54<89:21:06,  8.69s/it][2025-04-25 18:13:37,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:13:37,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.11 | bwd_microstep: 5730.57 | bwd_inner_microstep: 5717.53 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.52
[2025-04-25 18:13:37,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.11 | bwd: 5730.59 | bwd_inner: 5717.53 | bwd_allreduce: 13.02 | step: 18.53

 10%|█         | 4244/41250 [10:16:02<89:17:52,  8.69s/it]
                                                          
{'loss': 0.1571, 'grad_norm': 2.0496609210968018, 'learning_rate': 3.944552726460506e-05, 'epoch': 1.03}

 10%|█         | 4244/41250 [10:16:02<89:17:52,  8.69s/it][2025-04-25 18:13:46,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:13:46,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.73 | bwd_microstep: 5751.42 | bwd_inner_microstep: 5647.47 | bwd_allreduce_microstep: 103.91 | step_microstep: 18.52
[2025-04-25 18:13:46,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.73 | bwd: 5751.44 | bwd_inner: 5647.47 | bwd_allreduce: 103.93 | step: 18.52

 10%|█         | 4245/41250 [10:16:11<89:15:26,  8.68s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 0.9276850819587708, 'learning_rate': 3.944516000811554e-05, 'epoch': 1.03}

 10%|█         | 4245/41250 [10:16:11<89:15:26,  8.68s/it][2025-04-25 18:13:54,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:13:54,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.28 | bwd_microstep: 5683.63 | bwd_inner_microstep: 5670.81 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.72
[2025-04-25 18:13:54,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.28 | bwd: 5683.64 | bwd_inner: 5670.81 | bwd_allreduce: 12.78 | step: 18.73

 10%|█         | 4246/41250 [10:16:20<89:02:24,  8.66s/it]
                                                          
{'loss': 0.2254, 'grad_norm': 1.5918402671813965, 'learning_rate': 3.9444792631750434e-05, 'epoch': 1.03}

 10%|█         | 4246/41250 [10:16:20<89:02:24,  8.66s/it][2025-04-25 18:14:03,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:14:03,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.11 | bwd_microstep: 5686.45 | bwd_inner_microstep: 5670.17 | bwd_allreduce_microstep: 16.24 | step_microstep: 18.52
[2025-04-25 18:14:03,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.11 | bwd: 5686.47 | bwd_inner: 5670.17 | bwd_allreduce: 16.26 | step: 18.52

 10%|█         | 4247/41250 [10:16:28<88:54:14,  8.65s/it]
                                                          
{'loss': 0.2249, 'grad_norm': 1.73623788356781, 'learning_rate': 3.9444425135512e-05, 'epoch': 1.03}

 10%|█         | 4247/41250 [10:16:28<88:54:14,  8.65s/it][2025-04-25 18:14:12,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 18:14:12,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.01 | bwd_microstep: 5689.46 | bwd_inner_microstep: 5666.41 | bwd_allreduce_microstep: 23.00 | step_microstep: 18.83
[2025-04-25 18:14:12,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.01 | bwd: 5689.48 | bwd_inner: 5666.41 | bwd_allreduce: 23.02 | step: 18.83

 10%|█         | 4248/41250 [10:16:37<88:48:34,  8.64s/it]
                                                          
{'loss': 0.083, 'grad_norm': 1.151031732559204, 'learning_rate': 3.944405751940252e-05, 'epoch': 1.03}

 10%|█         | 4248/41250 [10:16:37<88:48:34,  8.64s/it][2025-04-25 18:14:20,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 18:14:20,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.82 | bwd_microstep: 5735.96 | bwd_inner_microstep: 5723.07 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.72
[2025-04-25 18:14:20,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.82 | bwd: 5735.97 | bwd_inner: 5723.07 | bwd_allreduce: 12.86 | step: 18.73

 10%|█         | 4249/41250 [10:16:46<88:55:01,  8.65s/it]
                                                          
{'loss': 0.0537, 'grad_norm': 0.8216007351875305, 'learning_rate': 3.944368978342424e-05, 'epoch': 1.03}

 10%|█         | 4249/41250 [10:16:46<88:55:01,  8.65s/it][2025-04-25 18:14:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 18:14:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.02 | bwd_microstep: 5691.65 | bwd_inner_microstep: 5671.39 | bwd_allreduce_microstep: 20.21 | step_microstep: 20.14
[2025-04-25 18:14:29,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.02 | bwd: 5691.67 | bwd_inner: 5671.39 | bwd_allreduce: 20.24 | step: 20.14

 10%|█         | 4250/41250 [10:16:54<88:48:25,  8.64s/it]
                                                          
{'loss': 0.0128, 'grad_norm': 0.18519435822963715, 'learning_rate': 3.9443321927579444e-05, 'epoch': 1.03}

 10%|█         | 4250/41250 [10:16:54<88:48:25,  8.64s/it][2025-04-25 18:14:38,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 18:14:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.40 | bwd_microstep: 5743.73 | bwd_inner_microstep: 5663.10 | bwd_allreduce_microstep: 80.58 | step_microstep: 18.68
[2025-04-25 18:14:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.40 | bwd: 5743.74 | bwd_inner: 5663.10 | bwd_allreduce: 80.60 | step: 18.69

 10%|█         | 4251/41250 [10:17:03<88:52:13,  8.65s/it]
                                                          
{'loss': 0.324, 'grad_norm': 1.9927241802215576, 'learning_rate': 3.944295395187039e-05, 'epoch': 1.03}

 10%|█         | 4251/41250 [10:17:03<88:52:13,  8.65s/it][2025-04-25 18:14:46,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 18:14:46,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.98 | bwd_microstep: 5757.74 | bwd_inner_microstep: 5715.84 | bwd_allreduce_microstep: 41.84 | step_microstep: 19.18
[2025-04-25 18:14:46,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.98 | bwd: 5757.76 | bwd_inner: 5715.84 | bwd_allreduce: 41.87 | step: 19.18

 10%|█         | 4252/41250 [10:17:12<89:00:44,  8.66s/it]
                                                          
{'loss': 0.2774, 'grad_norm': 2.106180429458618, 'learning_rate': 3.944258585629935e-05, 'epoch': 1.03}

 10%|█         | 4252/41250 [10:17:12<89:00:44,  8.66s/it][2025-04-25 18:14:55,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:14:55,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.90 | bwd_microstep: 5801.28 | bwd_inner_microstep: 5657.74 | bwd_allreduce_microstep: 143.50 | step_microstep: 18.81
[2025-04-25 18:14:55,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.90 | bwd: 5801.30 | bwd_inner: 5657.74 | bwd_allreduce: 143.52 | step: 18.81

 10%|█         | 4253/41250 [10:17:20<89:11:18,  8.68s/it]
                                                          
{'loss': 0.0655, 'grad_norm': 0.8271169662475586, 'learning_rate': 3.944221764086859e-05, 'epoch': 1.03}

 10%|█         | 4253/41250 [10:17:20<89:11:18,  8.68s/it][2025-04-25 18:15:04,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 18:15:04,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.50 | bwd_microstep: 5769.84 | bwd_inner_microstep: 5673.08 | bwd_allreduce_microstep: 96.69 | step_microstep: 19.44
[2025-04-25 18:15:04,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.50 | bwd: 5769.85 | bwd_inner: 5673.08 | bwd_allreduce: 96.72 | step: 19.44

 10%|█         | 4254/41250 [10:17:29<89:13:35,  8.68s/it]
                                                          
{'loss': 0.196, 'grad_norm': 1.4587161540985107, 'learning_rate': 3.944184930558039e-05, 'epoch': 1.03}

 10%|█         | 4254/41250 [10:17:29<89:13:35,  8.68s/it][2025-04-25 18:15:12,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 18:15:12,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.35 | bwd_microstep: 5721.73 | bwd_inner_microstep: 5708.86 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.99
[2025-04-25 18:15:12,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.35 | bwd: 5721.74 | bwd_inner: 5708.86 | bwd_allreduce: 12.85 | step: 19.00

 10%|█         | 4255/41250 [10:17:38<89:09:07,  8.68s/it]
                                                          
{'loss': 0.0371, 'grad_norm': 1.0150933265686035, 'learning_rate': 3.944148085043701e-05, 'epoch': 1.03}

 10%|█         | 4255/41250 [10:17:38<89:09:07,  8.68s/it][2025-04-25 18:15:21,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 18:15:21,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.53 | bwd_microstep: 5728.19 | bwd_inner_microstep: 5714.72 | bwd_allreduce_microstep: 13.40 | step_microstep: 19.12
[2025-04-25 18:15:21,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.53 | bwd: 5728.20 | bwd_inner: 5714.72 | bwd_allreduce: 13.42 | step: 19.12

 10%|█         | 4256/41250 [10:17:46<89:08:59,  8.68s/it]
                                                          
{'loss': 0.0778, 'grad_norm': 0.9586832523345947, 'learning_rate': 3.944111227544071e-05, 'epoch': 1.03}

 10%|█         | 4256/41250 [10:17:46<89:08:59,  8.68s/it][2025-04-25 18:15:30,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 18:15:30,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5753.64 | bwd_inner_microstep: 5679.16 | bwd_allreduce_microstep: 74.42 | step_microstep: 18.79
[2025-04-25 18:15:30,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5753.65 | bwd_inner: 5679.16 | bwd_allreduce: 74.45 | step: 18.79

 10%|█         | 4257/41250 [10:17:55<89:11:25,  8.68s/it]
                                                          
{'loss': 0.0494, 'grad_norm': 0.8940937519073486, 'learning_rate': 3.9440743580593795e-05, 'epoch': 1.03}

 10%|█         | 4257/41250 [10:17:55<89:11:25,  8.68s/it][2025-04-25 18:15:38,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:15:38,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.86 | bwd_microstep: 5731.32 | bwd_inner_microstep: 5716.75 | bwd_allreduce_microstep: 14.53 | step_microstep: 18.32
[2025-04-25 18:15:38,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.86 | bwd: 5731.33 | bwd_inner: 5716.75 | bwd_allreduce: 14.54 | step: 18.33

 10%|█         | 4258/41250 [10:18:04<89:13:16,  8.68s/it]
                                                          
{'loss': 0.0259, 'grad_norm': 0.29654863476753235, 'learning_rate': 3.944037476589851e-05, 'epoch': 1.03}

 10%|█         | 4258/41250 [10:18:04<89:13:16,  8.68s/it][2025-04-25 18:15:47,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 18:15:47,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.16 | bwd_microstep: 5760.24 | bwd_inner_microstep: 5702.20 | bwd_allreduce_microstep: 58.00 | step_microstep: 18.38
[2025-04-25 18:15:47,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.16 | bwd: 5760.25 | bwd_inner: 5702.20 | bwd_allreduce: 58.02 | step: 18.38

 10%|█         | 4259/41250 [10:18:12<89:16:33,  8.69s/it]
                                                          
{'loss': 0.108, 'grad_norm': 1.480064034461975, 'learning_rate': 3.944000583135714e-05, 'epoch': 1.03}

 10%|█         | 4259/41250 [10:18:12<89:16:33,  8.69s/it][2025-04-25 18:15:56,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 18:15:56,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.67 | bwd_microstep: 5695.17 | bwd_inner_microstep: 5668.09 | bwd_allreduce_microstep: 27.03 | step_microstep: 18.73
[2025-04-25 18:15:56,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.67 | bwd: 5695.18 | bwd_inner: 5668.09 | bwd_allreduce: 27.05 | step: 18.73

 10%|█         | 4260/41250 [10:18:21<89:03:58,  8.67s/it]
                                                          
{'loss': 0.0937, 'grad_norm': 4.116370677947998, 'learning_rate': 3.9439636776971965e-05, 'epoch': 1.03}

 10%|█         | 4260/41250 [10:18:21<89:03:58,  8.67s/it][2025-04-25 18:16:04,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:16:04,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.18 | bwd_microstep: 5741.60 | bwd_inner_microstep: 5723.00 | bwd_allreduce_microstep: 18.56 | step_microstep: 18.62
[2025-04-25 18:16:04,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.19 | bwd: 5741.62 | bwd_inner: 5723.00 | bwd_allreduce: 18.57 | step: 18.62

 10%|█         | 4261/41250 [10:18:30<89:08:15,  8.68s/it]
                                                          
{'loss': 0.062, 'grad_norm': 0.885590672492981, 'learning_rate': 3.943926760274525e-05, 'epoch': 1.03}

 10%|█         | 4261/41250 [10:18:30<89:08:15,  8.68s/it][2025-04-25 18:16:13,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.95
[2025-04-25 18:16:13,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.05 | bwd_microstep: 5733.48 | bwd_inner_microstep: 5720.67 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.24
[2025-04-25 18:16:13,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.05 | bwd: 5733.49 | bwd_inner: 5720.67 | bwd_allreduce: 12.78 | step: 18.24

 10%|█         | 4262/41250 [10:18:38<89:09:19,  8.68s/it]
                                                          
{'loss': 0.0845, 'grad_norm': 0.7767930030822754, 'learning_rate': 3.9438898308679264e-05, 'epoch': 1.03}

 10%|█         | 4262/41250 [10:18:38<89:09:19,  8.68s/it][2025-04-25 18:16:22,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 18:16:22,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.12 | bwd_microstep: 5749.68 | bwd_inner_microstep: 5707.52 | bwd_allreduce_microstep: 42.11 | step_microstep: 18.70
[2025-04-25 18:16:22,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.13 | bwd: 5749.69 | bwd_inner: 5707.52 | bwd_allreduce: 42.13 | step: 18.70

 10%|█         | 4263/41250 [10:18:47<89:12:55,  8.68s/it]
                                                          
{'loss': 0.0673, 'grad_norm': 1.0258992910385132, 'learning_rate': 3.94385288947763e-05, 'epoch': 1.03}

 10%|█         | 4263/41250 [10:18:47<89:12:55,  8.68s/it][2025-04-25 18:16:30,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 18:16:30,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.34 | bwd_microstep: 5699.09 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 36.96 | step_microstep: 19.60
[2025-04-25 18:16:30,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.34 | bwd: 5699.10 | bwd_inner: 5662.08 | bwd_allreduce: 36.98 | step: 19.61

 10%|█         | 4264/41250 [10:18:56<89:01:32,  8.67s/it]
                                                          
{'loss': 0.0856, 'grad_norm': 1.3644062280654907, 'learning_rate': 3.943815936103863e-05, 'epoch': 1.03}

 10%|█         | 4264/41250 [10:18:56<89:01:32,  8.67s/it][2025-04-25 18:16:39,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 18:16:39,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.59 | bwd_microstep: 5727.23 | bwd_inner_microstep: 5714.38 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.88
[2025-04-25 18:16:39,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.59 | bwd: 5727.24 | bwd_inner: 5714.38 | bwd_allreduce: 12.82 | step: 18.88

 10%|█         | 4265/41250 [10:19:04<89:02:49,  8.67s/it]
                                                          
{'loss': 0.2842, 'grad_norm': 2.544776678085327, 'learning_rate': 3.943778970746853e-05, 'epoch': 1.03}

 10%|█         | 4265/41250 [10:19:04<89:02:49,  8.67s/it][2025-04-25 18:16:48,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.31 | optimizer_step: 0.96
[2025-04-25 18:16:48,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.83 | bwd_microstep: 5741.71 | bwd_inner_microstep: 5728.33 | bwd_allreduce_microstep: 13.33 | step_microstep: 19.81
[2025-04-25 18:16:48,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.83 | bwd: 5741.72 | bwd_inner: 5728.33 | bwd_allreduce: 13.35 | step: 19.81

 10%|█         | 4266/41250 [10:19:13<89:07:14,  8.67s/it]
                                                          
{'loss': 0.0987, 'grad_norm': 1.7793251276016235, 'learning_rate': 3.9437419934068274e-05, 'epoch': 1.03}

 10%|█         | 4266/41250 [10:19:13<89:07:14,  8.67s/it][2025-04-25 18:16:56,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 18:16:56,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.96 | bwd_microstep: 5708.51 | bwd_inner_microstep: 5695.63 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.60
[2025-04-25 18:16:56,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.96 | bwd: 5708.53 | bwd_inner: 5695.63 | bwd_allreduce: 12.85 | step: 18.60

 10%|█         | 4267/41250 [10:19:22<88:59:40,  8.66s/it]
                                                          
{'loss': 0.0744, 'grad_norm': 1.1843347549438477, 'learning_rate': 3.943705004084016e-05, 'epoch': 1.03}

 10%|█         | 4267/41250 [10:19:22<88:59:40,  8.66s/it][2025-04-25 18:17:05,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 18:17:05,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.99 | bwd_microstep: 5707.14 | bwd_inner_microstep: 5694.42 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.02
[2025-04-25 18:17:05,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.99 | bwd: 5707.16 | bwd_inner: 5694.42 | bwd_allreduce: 12.69 | step: 19.02

 10%|█         | 4268/41250 [10:19:30<88:55:00,  8.66s/it]
                                                          
{'loss': 0.0993, 'grad_norm': 0.7667320966720581, 'learning_rate': 3.943668002778644e-05, 'epoch': 1.03}

 10%|█         | 4268/41250 [10:19:30<88:55:00,  8.66s/it][2025-04-25 18:17:14,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:17:14,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.97 | bwd_microstep: 5745.28 | bwd_inner_microstep: 5689.23 | bwd_allreduce_microstep: 56.00 | step_microstep: 19.14
[2025-04-25 18:17:14,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.97 | bwd: 5745.30 | bwd_inner: 5689.23 | bwd_allreduce: 56.02 | step: 19.14

 10%|█         | 4269/41250 [10:19:39<88:57:56,  8.66s/it]
                                                          
{'loss': 0.1238, 'grad_norm': 1.6069034337997437, 'learning_rate': 3.943630989490942e-05, 'epoch': 1.03}

 10%|█         | 4269/41250 [10:19:39<88:57:56,  8.66s/it][2025-04-25 18:17:22,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 18:17:22,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.23 | bwd_microstep: 5756.84 | bwd_inner_microstep: 5646.66 | bwd_allreduce_microstep: 110.13 | step_microstep: 19.29
[2025-04-25 18:17:22,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.23 | bwd: 5756.85 | bwd_inner: 5646.66 | bwd_allreduce: 110.15 | step: 19.29

 10%|█         | 4270/41250 [10:19:48<88:57:48,  8.66s/it]
                                                          
{'loss': 0.1232, 'grad_norm': 1.6003575325012207, 'learning_rate': 3.9435939642211375e-05, 'epoch': 1.04}

 10%|█         | 4270/41250 [10:19:48<88:57:48,  8.66s/it][2025-04-25 18:17:31,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.28 | optimizer_step: 0.90
[2025-04-25 18:17:31,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.93 | bwd_microstep: 5861.68 | bwd_inner_microstep: 5654.19 | bwd_allreduce_microstep: 207.43 | step_microstep: 19.78
[2025-04-25 18:17:31,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.93 | bwd: 5861.69 | bwd_inner: 5654.19 | bwd_allreduce: 207.46 | step: 19.78

 10%|█         | 4271/41250 [10:19:57<89:18:23,  8.69s/it]
                                                          
{'loss': 0.151, 'grad_norm': 1.754258394241333, 'learning_rate': 3.943556926969458e-05, 'epoch': 1.04}

 10%|█         | 4271/41250 [10:19:57<89:18:23,  8.69s/it][2025-04-25 18:17:40,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-25 18:17:40,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5682.34 | bwd_inner_microstep: 5655.50 | bwd_allreduce_microstep: 26.80 | step_microstep: 19.13
[2025-04-25 18:17:40,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5682.35 | bwd_inner: 5655.50 | bwd_allreduce: 26.81 | step: 19.13

 10%|█         | 4272/41250 [10:20:05<89:00:33,  8.67s/it]
                                                          
{'loss': 0.1932, 'grad_norm': 2.4293034076690674, 'learning_rate': 3.943519877736132e-05, 'epoch': 1.04}

 10%|█         | 4272/41250 [10:20:05<89:00:33,  8.67s/it][2025-04-25 18:17:48,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 18:17:48,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.49 | bwd_microstep: 5758.28 | bwd_inner_microstep: 5643.08 | bwd_allreduce_microstep: 115.16 | step_microstep: 18.66
[2025-04-25 18:17:48,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.49 | bwd: 5758.30 | bwd_inner: 5643.08 | bwd_allreduce: 115.18 | step: 18.67

 10%|█         | 4273/41250 [10:20:14<88:59:55,  8.66s/it]
                                                          
{'loss': 0.0499, 'grad_norm': 0.6260738372802734, 'learning_rate': 3.943482816521389e-05, 'epoch': 1.04}

 10%|█         | 4273/41250 [10:20:14<88:59:55,  8.66s/it][2025-04-25 18:17:57,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 18:17:57,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.13 | bwd_microstep: 5697.10 | bwd_inner_microstep: 5684.27 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.76
[2025-04-25 18:17:57,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.13 | bwd: 5697.11 | bwd_inner: 5684.27 | bwd_allreduce: 12.79 | step: 18.76

 10%|█         | 4274/41250 [10:20:22<88:51:39,  8.65s/it]
                                                          
{'loss': 0.1496, 'grad_norm': 1.40872061252594, 'learning_rate': 3.943445743325456e-05, 'epoch': 1.04}

 10%|█         | 4274/41250 [10:20:22<88:51:39,  8.65s/it][2025-04-25 18:18:06,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 18:18:06,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.34 | bwd_microstep: 5691.46 | bwd_inner_microstep: 5653.08 | bwd_allreduce_microstep: 38.34 | step_microstep: 18.72
[2025-04-25 18:18:06,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.34 | bwd: 5691.47 | bwd_inner: 5653.08 | bwd_allreduce: 38.36 | step: 18.72

 10%|█         | 4275/41250 [10:20:31<88:41:52,  8.64s/it]
                                                          
{'loss': 0.0636, 'grad_norm': 0.5519015192985535, 'learning_rate': 3.943408658148563e-05, 'epoch': 1.04}

 10%|█         | 4275/41250 [10:20:31<88:41:52,  8.64s/it][2025-04-25 18:18:14,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:18:14,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.70 | bwd_microstep: 5750.69 | bwd_inner_microstep: 5688.25 | bwd_allreduce_microstep: 62.39 | step_microstep: 19.06
[2025-04-25 18:18:14,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.70 | bwd: 5750.71 | bwd_inner: 5688.25 | bwd_allreduce: 62.41 | step: 19.06

 10%|█         | 4276/41250 [10:20:40<88:49:09,  8.65s/it]
                                                          
{'loss': 0.036, 'grad_norm': 0.6887385845184326, 'learning_rate': 3.9433715609909376e-05, 'epoch': 1.04}

 10%|█         | 4276/41250 [10:20:40<88:49:09,  8.65s/it][2025-04-25 18:18:23,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:18:23,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.83 | bwd_microstep: 5751.76 | bwd_inner_microstep: 5690.95 | bwd_allreduce_microstep: 60.76 | step_microstep: 18.65
[2025-04-25 18:18:23,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.83 | bwd: 5751.78 | bwd_inner: 5690.95 | bwd_allreduce: 60.78 | step: 18.65

 10%|█         | 4277/41250 [10:20:48<88:54:46,  8.66s/it]
                                                          
{'loss': 0.1042, 'grad_norm': 2.644340753555298, 'learning_rate': 3.943334451852808e-05, 'epoch': 1.04}

 10%|█         | 4277/41250 [10:20:48<88:54:46,  8.66s/it][2025-04-25 18:18:32,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 18:18:32,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.71 | bwd_microstep: 5739.49 | bwd_inner_microstep: 5688.84 | bwd_allreduce_microstep: 50.60 | step_microstep: 19.48
[2025-04-25 18:18:32,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.71 | bwd: 5739.51 | bwd_inner: 5688.84 | bwd_allreduce: 50.62 | step: 19.48

 10%|█         | 4278/41250 [10:20:57<88:56:50,  8.66s/it]
                                                          
{'loss': 0.0894, 'grad_norm': 1.0842552185058594, 'learning_rate': 3.943297330734405e-05, 'epoch': 1.04}

 10%|█         | 4278/41250 [10:20:57<88:56:50,  8.66s/it][2025-04-25 18:18:40,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:18:40,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5737.17 | bwd_inner_microstep: 5701.74 | bwd_allreduce_microstep: 35.39 | step_microstep: 18.43
[2025-04-25 18:18:40,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5737.18 | bwd_inner: 5701.74 | bwd_allreduce: 35.40 | step: 18.44

 10%|█         | 4279/41250 [10:21:06<88:59:33,  8.67s/it]
                                                          
{'loss': 0.1123, 'grad_norm': 2.578372001647949, 'learning_rate': 3.943260197635955e-05, 'epoch': 1.04}

 10%|█         | 4279/41250 [10:21:06<88:59:33,  8.67s/it][2025-04-25 18:18:49,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:18:49,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.09 | bwd_microstep: 5692.32 | bwd_inner_microstep: 5635.46 | bwd_allreduce_microstep: 56.81 | step_microstep: 18.59
[2025-04-25 18:18:49,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.09 | bwd: 5692.33 | bwd_inner: 5635.46 | bwd_allreduce: 56.83 | step: 18.60

 10%|█         | 4280/41250 [10:21:14<88:48:04,  8.65s/it]
                                                          
{'loss': 0.1077, 'grad_norm': 1.6248053312301636, 'learning_rate': 3.943223052557689e-05, 'epoch': 1.04}

 10%|█         | 4280/41250 [10:21:14<88:48:04,  8.65s/it][2025-04-25 18:18:58,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 18:18:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5710.85 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.12
[2025-04-25 18:18:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5710.87 | bwd_inner: 5697.92 | bwd_allreduce: 12.91 | step: 19.13

 10%|█         | 4281/41250 [10:21:23<88:47:56,  8.65s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 2.5546717643737793, 'learning_rate': 3.9431858954998345e-05, 'epoch': 1.04}

 10%|█         | 4281/41250 [10:21:23<88:47:56,  8.65s/it][2025-04-25 18:19:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:19:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.56 | bwd_microstep: 5723.07 | bwd_inner_microstep: 5649.99 | bwd_allreduce_microstep: 73.03 | step_microstep: 18.57
[2025-04-25 18:19:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.56 | bwd: 5723.08 | bwd_inner: 5649.99 | bwd_allreduce: 73.05 | step: 18.58

 10%|█         | 4282/41250 [10:21:32<88:46:58,  8.65s/it]
                                                          
{'loss': 0.0914, 'grad_norm': 2.1073076725006104, 'learning_rate': 3.943148726462621e-05, 'epoch': 1.04}

 10%|█         | 4282/41250 [10:21:32<88:46:58,  8.65s/it][2025-04-25 18:19:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:19:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.27 | bwd_microstep: 5681.90 | bwd_inner_microstep: 5644.72 | bwd_allreduce_microstep: 37.14 | step_microstep: 18.28
[2025-04-25 18:19:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.27 | bwd: 5681.91 | bwd_inner: 5644.72 | bwd_allreduce: 37.15 | step: 18.28

 10%|█         | 4283/41250 [10:21:40<88:36:53,  8.63s/it]
                                                          
{'loss': 0.2201, 'grad_norm': 2.288560390472412, 'learning_rate': 3.9431115454462785e-05, 'epoch': 1.04}

 10%|█         | 4283/41250 [10:21:40<88:36:53,  8.63s/it][2025-04-25 18:19:24,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:19:24,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.59 | bwd_microstep: 5730.39 | bwd_inner_microstep: 5704.55 | bwd_allreduce_microstep: 25.80 | step_microstep: 18.70
[2025-04-25 18:19:24,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.59 | bwd: 5730.40 | bwd_inner: 5704.55 | bwd_allreduce: 25.82 | step: 18.71

 10%|█         | 4284/41250 [10:21:49<88:45:14,  8.64s/it]
                                                          
{'loss': 0.1239, 'grad_norm': 1.5183223485946655, 'learning_rate': 3.943074352451035e-05, 'epoch': 1.04}

 10%|█         | 4284/41250 [10:21:49<88:45:14,  8.64s/it][2025-04-25 18:19:32,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:19:32,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5676.27 | bwd_inner_microstep: 5647.03 | bwd_allreduce_microstep: 29.20 | step_microstep: 18.10
[2025-04-25 18:19:32,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5676.29 | bwd_inner: 5647.03 | bwd_allreduce: 29.21 | step: 18.10

 10%|█         | 4285/41250 [10:21:57<88:34:44,  8.63s/it]
                                                          
{'loss': 0.1194, 'grad_norm': 1.3224231004714966, 'learning_rate': 3.943037147477121e-05, 'epoch': 1.04}

 10%|█         | 4285/41250 [10:21:57<88:34:44,  8.63s/it][2025-04-25 18:19:41,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 18:19:41,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.51 | bwd_microstep: 5695.93 | bwd_inner_microstep: 5682.90 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.08
[2025-04-25 18:19:41,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.51 | bwd: 5695.94 | bwd_inner: 5682.90 | bwd_allreduce: 13.00 | step: 19.08

 10%|█         | 4286/41250 [10:22:06<88:36:20,  8.63s/it]
                                                          
{'loss': 0.0763, 'grad_norm': 0.8323181867599487, 'learning_rate': 3.9429999305247646e-05, 'epoch': 1.04}

 10%|█         | 4286/41250 [10:22:06<88:36:20,  8.63s/it][2025-04-25 18:19:49,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:19:49,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.64 | bwd_microstep: 5672.56 | bwd_inner_microstep: 5659.87 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.54
[2025-04-25 18:19:49,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.64 | bwd: 5672.58 | bwd_inner: 5659.86 | bwd_allreduce: 12.67 | step: 18.54

 10%|█         | 4287/41250 [10:22:15<88:27:29,  8.62s/it]
                                                          
{'loss': 0.1341, 'grad_norm': 1.9119104146957397, 'learning_rate': 3.942962701594196e-05, 'epoch': 1.04}

 10%|█         | 4287/41250 [10:22:15<88:27:29,  8.62s/it][2025-04-25 18:19:58,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 18:19:58,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.89 | bwd_microstep: 5706.30 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.90
[2025-04-25 18:19:58,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.89 | bwd: 5706.32 | bwd_inner: 5693.53 | bwd_allreduce: 12.75 | step: 18.90

 10%|█         | 4288/41250 [10:22:23<88:33:26,  8.63s/it]
                                                          
{'loss': 0.0691, 'grad_norm': 1.1976312398910522, 'learning_rate': 3.942925460685644e-05, 'epoch': 1.04}

 10%|█         | 4288/41250 [10:22:23<88:33:26,  8.63s/it][2025-04-25 18:20:07,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:20:07,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.97 | bwd_microstep: 5686.35 | bwd_inner_microstep: 5655.78 | bwd_allreduce_microstep: 30.53 | step_microstep: 18.25
[2025-04-25 18:20:07,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.97 | bwd: 5686.37 | bwd_inner: 5655.78 | bwd_allreduce: 30.55 | step: 18.25

 10%|█         | 4289/41250 [10:22:32<88:29:37,  8.62s/it]
                                                          
{'loss': 0.27, 'grad_norm': 2.857119083404541, 'learning_rate': 3.9428882077993386e-05, 'epoch': 1.04}

 10%|█         | 4289/41250 [10:22:32<88:29:37,  8.62s/it][2025-04-25 18:20:15,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:20:15,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.47 | bwd_microstep: 5674.56 | bwd_inner_microstep: 5661.93 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.91
[2025-04-25 18:20:15,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.47 | bwd: 5674.58 | bwd_inner: 5661.93 | bwd_allreduce: 12.60 | step: 18.92

 10%|█         | 4290/41250 [10:22:41<88:27:48,  8.62s/it]
                                                          
{'loss': 0.0666, 'grad_norm': 1.0534813404083252, 'learning_rate': 3.942850942935511e-05, 'epoch': 1.04}

 10%|█         | 4290/41250 [10:22:41<88:27:48,  8.62s/it][2025-04-25 18:20:24,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 1.02 | optimizer_step: 1.17
[2025-04-25 18:20:24,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.02 | bwd_microstep: 5859.69 | bwd_inner_microstep: 5660.17 | bwd_allreduce_microstep: 199.47 | step_microstep: 18.87
[2025-04-25 18:20:24,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.02 | bwd: 5859.70 | bwd_inner: 5660.17 | bwd_allreduce: 199.49 | step: 18.87

 10%|█         | 4291/41250 [10:22:49<88:56:26,  8.66s/it]
                                                          
{'loss': 0.0947, 'grad_norm': 2.228644609451294, 'learning_rate': 3.942813666094388e-05, 'epoch': 1.04}

 10%|█         | 4291/41250 [10:22:49<88:56:26,  8.66s/it][2025-04-25 18:20:33,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:20:33,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.71 | bwd_microstep: 5709.61 | bwd_inner_microstep: 5696.65 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.82
[2025-04-25 18:20:33,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5709.62 | bwd_inner: 5696.65 | bwd_allreduce: 12.93 | step: 18.82

 10%|█         | 4292/41250 [10:22:58<88:53:35,  8.66s/it]
                                                          
{'loss': 0.0841, 'grad_norm': 1.236938714981079, 'learning_rate': 3.942776377276201e-05, 'epoch': 1.04}

 10%|█         | 4292/41250 [10:22:58<88:53:35,  8.66s/it][2025-04-25 18:20:41,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 18:20:41,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.64 | bwd_microstep: 5758.32 | bwd_inner_microstep: 5659.13 | bwd_allreduce_microstep: 99.15 | step_microstep: 18.84
[2025-04-25 18:20:41,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.64 | bwd: 5758.33 | bwd_inner: 5659.13 | bwd_allreduce: 99.16 | step: 18.84

 10%|█         | 4293/41250 [10:23:07<88:56:12,  8.66s/it]
                                                          
{'loss': 0.4083, 'grad_norm': 3.261368989944458, 'learning_rate': 3.9427390764811805e-05, 'epoch': 1.04}

 10%|█         | 4293/41250 [10:23:07<88:56:12,  8.66s/it][2025-04-25 18:20:50,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-25 18:20:50,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5766.24 | bwd_inner_microstep: 5656.88 | bwd_allreduce_microstep: 109.31 | step_microstep: 19.07
[2025-04-25 18:20:50,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5766.26 | bwd_inner: 5656.88 | bwd_allreduce: 109.33 | step: 19.07

 10%|█         | 4294/41250 [10:23:15<88:59:47,  8.67s/it]
                                                          
{'loss': 0.0648, 'grad_norm': 0.7541224360466003, 'learning_rate': 3.9427017637095554e-05, 'epoch': 1.04}

 10%|█         | 4294/41250 [10:23:15<88:59:47,  8.67s/it][2025-04-25 18:20:59,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 18:20:59,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.65 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5704.68 | bwd_allreduce_microstep: 64.38 | step_microstep: 18.75
[2025-04-25 18:20:59,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.65 | bwd: 5769.12 | bwd_inner: 5704.68 | bwd_allreduce: 64.39 | step: 18.75

 10%|█         | 4295/41250 [10:23:24<89:07:43,  8.68s/it]
                                                          
{'loss': 0.2856, 'grad_norm': 2.0439558029174805, 'learning_rate': 3.942664438961556e-05, 'epoch': 1.04}

 10%|█         | 4295/41250 [10:23:24<89:07:43,  8.68s/it][2025-04-25 18:21:07,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 18:21:07,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5702.79 | bwd_inner_microstep: 5690.01 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.80
[2025-04-25 18:21:07,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5702.80 | bwd_inner: 5690.01 | bwd_allreduce: 12.76 | step: 18.80

 10%|█         | 4296/41250 [10:23:33<89:00:47,  8.67s/it]
                                                          
{'loss': 0.0198, 'grad_norm': 0.29425621032714844, 'learning_rate': 3.942627102237412e-05, 'epoch': 1.04}

 10%|█         | 4296/41250 [10:23:33<89:00:47,  8.67s/it][2025-04-25 18:21:16,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 18:21:16,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.59 | bwd_microstep: 5786.56 | bwd_inner_microstep: 5660.52 | bwd_allreduce_microstep: 125.98 | step_microstep: 18.95
[2025-04-25 18:21:16,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.59 | bwd: 5786.57 | bwd_inner: 5660.52 | bwd_allreduce: 126.01 | step: 18.95

 10%|█         | 4297/41250 [10:23:41<89:06:20,  8.68s/it]
                                                          
{'loss': 0.0252, 'grad_norm': 0.4940983057022095, 'learning_rate': 3.942589753537355e-05, 'epoch': 1.04}

 10%|█         | 4297/41250 [10:23:41<89:06:20,  8.68s/it][2025-04-25 18:21:25,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 18:21:25,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.24 | bwd_microstep: 5882.80 | bwd_inner_microstep: 5712.22 | bwd_allreduce_microstep: 170.53 | step_microstep: 19.30
[2025-04-25 18:21:25,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.24 | bwd: 5882.82 | bwd_inner: 5712.22 | bwd_allreduce: 170.55 | step: 19.31

 10%|█         | 4298/41250 [10:23:50<89:33:45,  8.73s/it]
                                                          
{'loss': 0.2121, 'grad_norm': 2.6713039875030518, 'learning_rate': 3.942552392861613e-05, 'epoch': 1.04}

 10%|█         | 4298/41250 [10:23:50<89:33:45,  8.73s/it][2025-04-25 18:21:34,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 18:21:34,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.82 | bwd_microstep: 5768.51 | bwd_inner_microstep: 5702.90 | bwd_allreduce_microstep: 65.56 | step_microstep: 19.01
[2025-04-25 18:21:34,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.81 | bwd: 5768.53 | bwd_inner: 5702.90 | bwd_allreduce: 65.58 | step: 19.01

 10%|█         | 4299/41250 [10:23:59<89:29:58,  8.72s/it]
                                                          
{'loss': 0.1348, 'grad_norm': 1.8745875358581543, 'learning_rate': 3.942515020210418e-05, 'epoch': 1.04}

 10%|█         | 4299/41250 [10:23:59<89:29:58,  8.72s/it][2025-04-25 18:21:42,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 18:21:42,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.98 | bwd_microstep: 5776.01 | bwd_inner_microstep: 5667.91 | bwd_allreduce_microstep: 108.04 | step_microstep: 19.36
[2025-04-25 18:21:42,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.98 | bwd: 5776.02 | bwd_inner: 5667.91 | bwd_allreduce: 108.06 | step: 19.36

 10%|█         | 4300/41250 [10:24:08<89:24:36,  8.71s/it]
                                                          
{'loss': 0.0319, 'grad_norm': 0.7025327086448669, 'learning_rate': 3.942477635584e-05, 'epoch': 1.04}

 10%|█         | 4300/41250 [10:24:08<89:24:36,  8.71s/it][2025-04-25 18:21:51,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:21:51,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.47 | bwd_microstep: 5748.61 | bwd_inner_microstep: 5686.79 | bwd_allreduce_microstep: 61.77 | step_microstep: 18.85
[2025-04-25 18:21:51,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.47 | bwd: 5748.62 | bwd_inner: 5686.79 | bwd_allreduce: 61.79 | step: 18.85

 10%|█         | 4301/41250 [10:24:16<89:19:05,  8.70s/it]
                                                          
{'loss': 0.2064, 'grad_norm': 1.1024681329727173, 'learning_rate': 3.94244023898259e-05, 'epoch': 1.04}

 10%|█         | 4301/41250 [10:24:16<89:19:05,  8.70s/it][2025-04-25 18:22:00,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:22:00,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.07 | bwd_microstep: 5701.31 | bwd_inner_microstep: 5651.96 | bwd_allreduce_microstep: 49.30 | step_microstep: 18.38
[2025-04-25 18:22:00,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.07 | bwd: 5701.33 | bwd_inner: 5651.96 | bwd_allreduce: 49.32 | step: 18.38

 10%|█         | 4302/41250 [10:24:25<89:02:03,  8.67s/it]
                                                          
{'loss': 0.1254, 'grad_norm': 2.6810591220855713, 'learning_rate': 3.942402830406418e-05, 'epoch': 1.04}

 10%|█         | 4302/41250 [10:24:25<89:02:03,  8.67s/it][2025-04-25 18:22:08,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 18:22:08,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.09 | bwd_microstep: 5812.36 | bwd_inner_microstep: 5674.18 | bwd_allreduce_microstep: 138.13 | step_microstep: 18.49
[2025-04-25 18:22:08,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.09 | bwd: 5812.37 | bwd_inner: 5674.18 | bwd_allreduce: 138.15 | step: 18.50

 10%|█         | 4303/41250 [10:24:34<89:12:45,  8.69s/it]
                                                          
{'loss': 0.0804, 'grad_norm': 1.3666743040084839, 'learning_rate': 3.9423654098557136e-05, 'epoch': 1.04}

 10%|█         | 4303/41250 [10:24:34<89:12:45,  8.69s/it][2025-04-25 18:22:17,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 18:22:17,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.19 | bwd_microstep: 5726.75 | bwd_inner_microstep: 5713.75 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.35
[2025-04-25 18:22:17,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.19 | bwd: 5726.76 | bwd_inner: 5713.75 | bwd_allreduce: 12.97 | step: 19.36

 10%|█         | 4304/41250 [10:24:42<89:07:51,  8.68s/it]
                                                          
{'loss': 0.0574, 'grad_norm': 0.8311811089515686, 'learning_rate': 3.94232797733071e-05, 'epoch': 1.04}

 10%|█         | 4304/41250 [10:24:42<89:07:51,  8.68s/it][2025-04-25 18:22:26,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 18:22:26,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.39 | bwd_microstep: 5784.82 | bwd_inner_microstep: 5685.95 | bwd_allreduce_microstep: 98.82 | step_microstep: 18.72
[2025-04-25 18:22:26,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.39 | bwd: 5784.83 | bwd_inner: 5685.95 | bwd_allreduce: 98.84 | step: 18.73

 10%|█         | 4305/41250 [10:24:51<89:13:28,  8.69s/it]
                                                          
{'loss': 0.0762, 'grad_norm': 1.1117693185806274, 'learning_rate': 3.942290532831635e-05, 'epoch': 1.04}

 10%|█         | 4305/41250 [10:24:51<89:13:28,  8.69s/it][2025-04-25 18:22:34,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 18:22:34,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.27 | bwd_microstep: 5759.23 | bwd_inner_microstep: 5693.90 | bwd_allreduce_microstep: 65.28 | step_microstep: 18.95
[2025-04-25 18:22:34,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.27 | bwd: 5759.24 | bwd_inner: 5693.90 | bwd_allreduce: 65.30 | step: 18.95

 10%|█         | 4306/41250 [10:25:00<89:13:02,  8.69s/it]
                                                          
{'loss': 0.0643, 'grad_norm': 1.2738224267959595, 'learning_rate': 3.942253076358721e-05, 'epoch': 1.04}

 10%|█         | 4306/41250 [10:25:00<89:13:02,  8.69s/it][2025-04-25 18:22:43,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:22:43,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.56 | bwd_microstep: 5787.73 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 130.10 | step_microstep: 18.72
[2025-04-25 18:22:43,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.56 | bwd: 5787.74 | bwd_inner: 5657.59 | bwd_allreduce: 130.11 | step: 18.72

 10%|█         | 4307/41250 [10:25:08<89:15:54,  8.70s/it]
                                                          
{'loss': 0.1853, 'grad_norm': 2.93125581741333, 'learning_rate': 3.9422156079122e-05, 'epoch': 1.04}

 10%|█         | 4307/41250 [10:25:08<89:15:54,  8.70s/it][2025-04-25 18:22:52,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 18:22:52,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.62 | bwd_microstep: 5766.06 | bwd_inner_microstep: 5671.22 | bwd_allreduce_microstep: 94.79 | step_microstep: 19.17
[2025-04-25 18:22:52,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.62 | bwd: 5766.07 | bwd_inner: 5671.22 | bwd_allreduce: 94.81 | step: 19.17

 10%|█         | 4308/41250 [10:25:17<89:14:11,  8.70s/it]
                                                          
{'loss': 0.0437, 'grad_norm': 1.386686086654663, 'learning_rate': 3.9421781274923006e-05, 'epoch': 1.04}

 10%|█         | 4308/41250 [10:25:17<89:14:11,  8.70s/it][2025-04-25 18:23:01,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 18:23:01,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.16 | bwd_microstep: 5770.97 | bwd_inner_microstep: 5708.27 | bwd_allreduce_microstep: 62.66 | step_microstep: 18.79
[2025-04-25 18:23:01,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.16 | bwd: 5770.98 | bwd_inner: 5708.27 | bwd_allreduce: 62.68 | step: 18.80

 10%|█         | 4309/41250 [10:25:26<89:16:00,  8.70s/it]
                                                          
{'loss': 0.0632, 'grad_norm': 0.7490788102149963, 'learning_rate': 3.9421406350992555e-05, 'epoch': 1.04}

 10%|█         | 4309/41250 [10:25:26<89:16:00,  8.70s/it][2025-04-25 18:23:09,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:23:09,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5717.98 | bwd_inner_microstep: 5705.11 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.07
[2025-04-25 18:23:09,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5718.00 | bwd_inner: 5705.11 | bwd_allreduce: 12.84 | step: 19.07

 10%|█         | 4310/41250 [10:25:34<89:07:34,  8.69s/it]
                                                          
{'loss': 0.3915, 'grad_norm': 2.596740245819092, 'learning_rate': 3.9421031307332954e-05, 'epoch': 1.04}

 10%|█         | 4310/41250 [10:25:34<89:07:34,  8.69s/it][2025-04-25 18:23:18,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:23:18,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5782.61 | bwd_inner_microstep: 5661.75 | bwd_allreduce_microstep: 120.82 | step_microstep: 18.46
[2025-04-25 18:23:18,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5782.63 | bwd_inner: 5661.74 | bwd_allreduce: 120.84 | step: 18.46

 10%|█         | 4311/41250 [10:25:43<89:08:50,  8.69s/it]
                                                          
{'loss': 0.1742, 'grad_norm': 3.0394287109375, 'learning_rate': 3.9420656143946514e-05, 'epoch': 1.05}

 10%|█         | 4311/41250 [10:25:43<89:08:50,  8.69s/it][2025-04-25 18:23:27,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 1.14 | optimizer_step: 0.90
[2025-04-25 18:23:27,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.79 | bwd_microstep: 5735.10 | bwd_inner_microstep: 5716.71 | bwd_allreduce_microstep: 18.34 | step_microstep: 20.92
[2025-04-25 18:23:27,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.79 | bwd: 5735.11 | bwd_inner: 5716.71 | bwd_allreduce: 18.36 | step: 20.92

 10%|█         | 4312/41250 [10:25:52<89:06:54,  8.69s/it]
                                                          
{'loss': 0.2819, 'grad_norm': 2.173081398010254, 'learning_rate': 3.942028086083555e-05, 'epoch': 1.05}

 10%|█         | 4312/41250 [10:25:52<89:06:54,  8.69s/it][2025-04-25 18:23:35,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:23:35,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.27 | bwd_microstep: 5768.96 | bwd_inner_microstep: 5699.06 | bwd_allreduce_microstep: 69.85 | step_microstep: 18.83
[2025-04-25 18:23:35,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.27 | bwd: 5768.98 | bwd_inner: 5699.06 | bwd_allreduce: 69.87 | step: 18.83

 10%|█         | 4313/41250 [10:26:01<89:11:01,  8.69s/it]
                                                          
{'loss': 0.2025, 'grad_norm': 3.664738178253174, 'learning_rate': 3.941990545800238e-05, 'epoch': 1.05}

 10%|█         | 4313/41250 [10:26:01<89:11:01,  8.69s/it][2025-04-25 18:23:44,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:23:44,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.89 | bwd_microstep: 5776.91 | bwd_inner_microstep: 5645.75 | bwd_allreduce_microstep: 131.11 | step_microstep: 18.77
[2025-04-25 18:23:44,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.89 | bwd: 5776.92 | bwd_inner: 5645.75 | bwd_allreduce: 131.13 | step: 18.77

 10%|█         | 4314/41250 [10:26:09<89:08:47,  8.69s/it]
                                                          
{'loss': 0.1121, 'grad_norm': 1.8317548036575317, 'learning_rate': 3.94195299354493e-05, 'epoch': 1.05}

 10%|█         | 4314/41250 [10:26:09<89:08:47,  8.69s/it][2025-04-25 18:23:53,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 18:23:53,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.68 | bwd_microstep: 5700.62 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 42.92 | step_microstep: 19.16
[2025-04-25 18:23:53,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.68 | bwd: 5700.64 | bwd_inner: 5657.65 | bwd_allreduce: 42.94 | step: 19.17

 10%|█         | 4315/41250 [10:26:18<88:55:20,  8.67s/it]
                                                          
{'loss': 0.0822, 'grad_norm': 1.649374008178711, 'learning_rate': 3.9419154293178646e-05, 'epoch': 1.05}

 10%|█         | 4315/41250 [10:26:18<88:55:20,  8.67s/it][2025-04-25 18:24:01,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.24 | optimizer_step: 0.91
[2025-04-25 18:24:01,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.88 | bwd_microstep: 5764.13 | bwd_inner_microstep: 5654.46 | bwd_allreduce_microstep: 109.62 | step_microstep: 19.26
[2025-04-25 18:24:01,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.88 | bwd: 5764.15 | bwd_inner: 5654.46 | bwd_allreduce: 109.64 | step: 19.27

 10%|█         | 4316/41250 [10:26:27<89:01:06,  8.68s/it]
                                                          
{'loss': 0.0518, 'grad_norm': 0.9528141617774963, 'learning_rate': 3.941877853119272e-05, 'epoch': 1.05}

 10%|█         | 4316/41250 [10:26:27<89:01:06,  8.68s/it][2025-04-25 18:24:10,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.07 | optimizer_step: 0.98
[2025-04-25 18:24:10,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5759.00 | bwd_inner_microstep: 5661.18 | bwd_allreduce_microstep: 97.78 | step_microstep: 18.62
[2025-04-25 18:24:10,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.09 | bwd: 5759.02 | bwd_inner: 5661.18 | bwd_allreduce: 97.80 | step: 18.63

 10%|█         | 4317/41250 [10:26:35<88:59:46,  8.67s/it]
                                                          
{'loss': 0.0378, 'grad_norm': 0.7443650364875793, 'learning_rate': 3.941840264949385e-05, 'epoch': 1.05}

 10%|█         | 4317/41250 [10:26:35<88:59:46,  8.67s/it][2025-04-25 18:24:19,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 18:24:19,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.58 | bwd_microstep: 5681.94 | bwd_inner_microstep: 5656.25 | bwd_allreduce_microstep: 25.64 | step_microstep: 18.58
[2025-04-25 18:24:19,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.58 | bwd: 5681.95 | bwd_inner: 5656.25 | bwd_allreduce: 25.66 | step: 18.58

 10%|█         | 4318/41250 [10:26:44<88:45:12,  8.65s/it]
                                                          
{'loss': 0.1728, 'grad_norm': 2.928795337677002, 'learning_rate': 3.941802664808434e-05, 'epoch': 1.05}

 10%|█         | 4318/41250 [10:26:44<88:45:12,  8.65s/it][2025-04-25 18:24:27,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 18:24:27,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.85 | bwd_microstep: 5760.97 | bwd_inner_microstep: 5716.52 | bwd_allreduce_microstep: 44.40 | step_microstep: 18.85
[2025-04-25 18:24:27,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.85 | bwd: 5760.98 | bwd_inner: 5716.52 | bwd_allreduce: 44.42 | step: 18.85

 10%|█         | 4319/41250 [10:26:53<88:53:25,  8.66s/it]
                                                          
{'loss': 0.2351, 'grad_norm': 1.6294091939926147, 'learning_rate': 3.941765052696652e-05, 'epoch': 1.05}

 10%|█         | 4319/41250 [10:26:53<88:53:25,  8.66s/it][2025-04-25 18:24:36,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.07 | optimizer_step: 0.91
[2025-04-25 18:24:36,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.52 | bwd_microstep: 5799.32 | bwd_inner_microstep: 5638.91 | bwd_allreduce_microstep: 160.36 | step_microstep: 19.00
[2025-04-25 18:24:36,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.52 | bwd: 5799.33 | bwd_inner: 5638.91 | bwd_allreduce: 160.38 | step: 19.00

 10%|█         | 4320/41250 [10:27:01<89:00:53,  8.68s/it]
                                                          
{'loss': 0.3101, 'grad_norm': 2.0307178497314453, 'learning_rate': 3.941727428614271e-05, 'epoch': 1.05}

 10%|█         | 4320/41250 [10:27:01<89:00:53,  8.68s/it][2025-04-25 18:24:45,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:24:45,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.12 | bwd_microstep: 5741.88 | bwd_inner_microstep: 5701.97 | bwd_allreduce_microstep: 39.87 | step_microstep: 18.60
[2025-04-25 18:24:45,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.12 | bwd: 5741.90 | bwd_inner: 5701.96 | bwd_allreduce: 39.89 | step: 18.61

 10%|█         | 4321/41250 [10:27:10<89:00:11,  8.68s/it]
                                                          
{'loss': 0.2983, 'grad_norm': 1.9110794067382812, 'learning_rate': 3.9416897925615206e-05, 'epoch': 1.05}

 10%|█         | 4321/41250 [10:27:10<89:00:11,  8.68s/it][2025-04-25 18:24:53,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 18:24:53,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.73 | bwd_microstep: 5724.27 | bwd_inner_microstep: 5688.72 | bwd_allreduce_microstep: 35.51 | step_microstep: 18.71
[2025-04-25 18:24:53,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.73 | bwd: 5724.29 | bwd_inner: 5688.72 | bwd_allreduce: 35.53 | step: 18.71

 10%|█         | 4322/41250 [10:27:19<88:57:29,  8.67s/it]
                                                          
{'loss': 0.2062, 'grad_norm': 1.1079087257385254, 'learning_rate': 3.941652144538636e-05, 'epoch': 1.05}

 10%|█         | 4322/41250 [10:27:19<88:57:29,  8.67s/it][2025-04-25 18:25:02,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:25:02,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5730.18 | bwd_inner_microstep: 5680.00 | bwd_allreduce_microstep: 50.13 | step_microstep: 18.87
[2025-04-25 18:25:02,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.42 | bwd: 5730.19 | bwd_inner: 5680.00 | bwd_allreduce: 50.15 | step: 18.87

 10%|█         | 4323/41250 [10:27:27<88:54:04,  8.67s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 2.8997669219970703, 'learning_rate': 3.941614484545848e-05, 'epoch': 1.05}

 10%|█         | 4323/41250 [10:27:27<88:54:04,  8.67s/it][2025-04-25 18:25:11,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:25:11,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.67 | bwd_microstep: 5745.47 | bwd_inner_microstep: 5705.18 | bwd_allreduce_microstep: 40.24 | step_microstep: 18.61
[2025-04-25 18:25:11,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.67 | bwd: 5745.48 | bwd_inner: 5705.18 | bwd_allreduce: 40.26 | step: 18.61

 10%|█         | 4324/41250 [10:27:36<88:55:42,  8.67s/it]
                                                          
{'loss': 0.1747, 'grad_norm': 1.6146695613861084, 'learning_rate': 3.941576812583387e-05, 'epoch': 1.05}

 10%|█         | 4324/41250 [10:27:36<88:55:42,  8.67s/it][2025-04-25 18:25:19,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:25:19,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5768.32 | bwd_inner_microstep: 5646.52 | bwd_allreduce_microstep: 121.75 | step_microstep: 19.47
[2025-04-25 18:25:19,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.09 | bwd: 5768.33 | bwd_inner: 5646.52 | bwd_allreduce: 121.77 | step: 19.48

 10%|█         | 4325/41250 [10:27:45<88:57:19,  8.67s/it]
                                                          
{'loss': 0.1352, 'grad_norm': 0.5939320921897888, 'learning_rate': 3.941539128651488e-05, 'epoch': 1.05}

 10%|█         | 4325/41250 [10:27:45<88:57:19,  8.67s/it][2025-04-25 18:25:28,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 18:25:28,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.43 | bwd_microstep: 5731.78 | bwd_inner_microstep: 5678.58 | bwd_allreduce_microstep: 53.16 | step_microstep: 19.26
[2025-04-25 18:25:28,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.43 | bwd: 5731.79 | bwd_inner: 5678.58 | bwd_allreduce: 53.18 | step: 19.26

 10%|█         | 4326/41250 [10:27:53<88:56:56,  8.67s/it]
                                                          
{'loss': 0.2079, 'grad_norm': 2.6648716926574707, 'learning_rate': 3.9415014327503825e-05, 'epoch': 1.05}

 10%|█         | 4326/41250 [10:27:53<88:56:56,  8.67s/it][2025-04-25 18:25:37,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:25:37,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.79 | bwd_microstep: 5742.36 | bwd_inner_microstep: 5641.19 | bwd_allreduce_microstep: 101.12 | step_microstep: 18.65
[2025-04-25 18:25:37,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.79 | bwd: 5742.37 | bwd_inner: 5641.19 | bwd_allreduce: 101.14 | step: 18.66

 10%|█         | 4327/41250 [10:28:02<88:52:29,  8.67s/it]
                                                          
{'loss': 0.1469, 'grad_norm': 1.7123693227767944, 'learning_rate': 3.9414637248803014e-05, 'epoch': 1.05}

 10%|█         | 4327/41250 [10:28:02<88:52:29,  8.67s/it][2025-04-25 18:25:45,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 18:25:45,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.78 | bwd_microstep: 5693.90 | bwd_inner_microstep: 5641.47 | bwd_allreduce_microstep: 52.39 | step_microstep: 18.90
[2025-04-25 18:25:45,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.78 | bwd: 5693.92 | bwd_inner: 5641.47 | bwd_allreduce: 52.41 | step: 18.90

 10%|█         | 4328/41250 [10:28:10<88:39:18,  8.64s/it]
                                                          
{'loss': 0.1442, 'grad_norm': 1.333683729171753, 'learning_rate': 3.9414260050414795e-05, 'epoch': 1.05}

 10%|█         | 4328/41250 [10:28:10<88:39:18,  8.64s/it][2025-04-25 18:25:54,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:25:54,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.94 | bwd_microstep: 5766.25 | bwd_inner_microstep: 5645.31 | bwd_allreduce_microstep: 120.89 | step_microstep: 18.59
[2025-04-25 18:25:54,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.94 | bwd: 5766.27 | bwd_inner: 5645.31 | bwd_allreduce: 120.91 | step: 18.59

 10%|█         | 4329/41250 [10:28:19<88:44:09,  8.65s/it]
                                                          
{'loss': 0.1586, 'grad_norm': 1.489928126335144, 'learning_rate': 3.941388273234147e-05, 'epoch': 1.05}

 10%|█         | 4329/41250 [10:28:19<88:44:09,  8.65s/it][2025-04-25 18:26:03,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 18:26:03,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.36 | bwd_microstep: 5744.34 | bwd_inner_microstep: 5685.55 | bwd_allreduce_microstep: 58.74 | step_microstep: 18.73
[2025-04-25 18:26:03,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.36 | bwd: 5744.36 | bwd_inner: 5685.55 | bwd_allreduce: 58.76 | step: 18.73

 10%|█         | 4330/41250 [10:28:28<88:47:13,  8.66s/it]
                                                          
{'loss': 0.1794, 'grad_norm': 3.48454213142395, 'learning_rate': 3.9413505294585386e-05, 'epoch': 1.05}

 10%|█         | 4330/41250 [10:28:28<88:47:13,  8.66s/it][2025-04-25 18:26:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 18:26:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.47 | bwd_microstep: 5736.42 | bwd_inner_microstep: 5647.72 | bwd_allreduce_microstep: 88.65 | step_microstep: 18.57
[2025-04-25 18:26:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.47 | bwd: 5736.43 | bwd_inner: 5647.72 | bwd_allreduce: 88.67 | step: 18.57

 10%|█         | 4331/41250 [10:28:36<88:43:42,  8.65s/it]
                                                          
{'loss': 0.1551, 'grad_norm': 6.716338157653809, 'learning_rate': 3.941312773714885e-05, 'epoch': 1.05}

 10%|█         | 4331/41250 [10:28:36<88:43:42,  8.65s/it][2025-04-25 18:26:20,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 18:26:20,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.91 | bwd_microstep: 5696.31 | bwd_inner_microstep: 5683.36 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.06
[2025-04-25 18:26:20,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.91 | bwd: 5696.33 | bwd_inner: 5683.36 | bwd_allreduce: 12.92 | step: 19.06

 11%|█         | 4332/41250 [10:28:45<88:39:49,  8.65s/it]
                                                          
{'loss': 0.1955, 'grad_norm': 2.0575735569000244, 'learning_rate': 3.9412750060034205e-05, 'epoch': 1.05}

 11%|█         | 4332/41250 [10:28:45<88:39:49,  8.65s/it][2025-04-25 18:26:28,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:26:28,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5712.72 | bwd_inner_microstep: 5699.89 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.78
[2025-04-25 18:26:28,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5712.73 | bwd_inner: 5699.89 | bwd_allreduce: 12.80 | step: 18.78

 11%|█         | 4333/41250 [10:28:54<88:39:27,  8.65s/it]
                                                          
{'loss': 0.1713, 'grad_norm': 1.5923789739608765, 'learning_rate': 3.941237226324378e-05, 'epoch': 1.05}

 11%|█         | 4333/41250 [10:28:54<88:39:27,  8.65s/it][2025-04-25 18:26:37,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:26:37,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.40 | bwd_microstep: 5696.88 | bwd_inner_microstep: 5684.05 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.82
[2025-04-25 18:26:37,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.40 | bwd: 5696.89 | bwd_inner: 5684.05 | bwd_allreduce: 12.80 | step: 18.83

 11%|█         | 4334/41250 [10:29:02<88:36:45,  8.64s/it]
                                                          
{'loss': 0.1475, 'grad_norm': 2.819801092147827, 'learning_rate': 3.9411994346779895e-05, 'epoch': 1.05}

 11%|█         | 4334/41250 [10:29:02<88:36:45,  8.64s/it][2025-04-25 18:26:46,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 0.99
[2025-04-25 18:26:46,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.15 | bwd_microstep: 5777.50 | bwd_inner_microstep: 5764.16 | bwd_allreduce_microstep: 13.28 | step_microstep: 19.39
[2025-04-25 18:26:46,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.15 | bwd: 5777.51 | bwd_inner: 5764.16 | bwd_allreduce: 13.31 | step: 19.39

 11%|█         | 4335/41250 [10:29:11<88:55:28,  8.67s/it]
                                                          
{'loss': 0.1029, 'grad_norm': 1.757896900177002, 'learning_rate': 3.941161631064488e-05, 'epoch': 1.05}

 11%|█         | 4335/41250 [10:29:11<88:55:28,  8.67s/it][2025-04-25 18:26:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.15 | optimizer_step: 0.91
[2025-04-25 18:26:55,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.47 | bwd_microstep: 5779.93 | bwd_inner_microstep: 5767.28 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.80
[2025-04-25 18:26:55,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.47 | bwd: 5779.94 | bwd_inner: 5767.28 | bwd_allreduce: 12.62 | step: 19.81

 11%|█         | 4336/41250 [10:29:20<89:08:06,  8.69s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 0.9106292128562927, 'learning_rate': 3.941123815484107e-05, 'epoch': 1.05}

 11%|█         | 4336/41250 [10:29:20<89:08:06,  8.69s/it][2025-04-25 18:27:03,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:27:03,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.28 | bwd_microstep: 5738.38 | bwd_inner_microstep: 5691.27 | bwd_allreduce_microstep: 47.06 | step_microstep: 18.33
[2025-04-25 18:27:03,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.28 | bwd: 5738.39 | bwd_inner: 5691.27 | bwd_allreduce: 47.08 | step: 18.33

 11%|█         | 4337/41250 [10:29:29<89:02:50,  8.68s/it]
                                                          
{'loss': 0.3589, 'grad_norm': 4.45988655090332, 'learning_rate': 3.9410859879370795e-05, 'epoch': 1.05}

 11%|█         | 4337/41250 [10:29:29<89:02:50,  8.68s/it][2025-04-25 18:27:12,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:27:12,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.78 | bwd_microstep: 5747.92 | bwd_inner_microstep: 5657.56 | bwd_allreduce_microstep: 90.31 | step_microstep: 18.27
[2025-04-25 18:27:12,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.78 | bwd: 5747.94 | bwd_inner: 5657.56 | bwd_allreduce: 90.33 | step: 18.27

 11%|█         | 4338/41250 [10:29:37<88:58:51,  8.68s/it]
                                                          
{'loss': 0.0354, 'grad_norm': 0.37355369329452515, 'learning_rate': 3.941048148423638e-05, 'epoch': 1.05}

 11%|█         | 4338/41250 [10:29:37<88:58:51,  8.68s/it][2025-04-25 18:27:21,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:27:21,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.27 | bwd_microstep: 5722.96 | bwd_inner_microstep: 5709.97 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.52
[2025-04-25 18:27:21,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.27 | bwd: 5722.97 | bwd_inner: 5709.97 | bwd_allreduce: 12.96 | step: 18.52

 11%|█         | 4339/41250 [10:29:46<88:54:46,  8.67s/it]
                                                          
{'loss': 0.1773, 'grad_norm': 1.9248709678649902, 'learning_rate': 3.9410102969440175e-05, 'epoch': 1.05}

 11%|█         | 4339/41250 [10:29:46<88:54:46,  8.67s/it][2025-04-25 18:27:29,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-25 18:27:29,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.63 | bwd_microstep: 5685.02 | bwd_inner_microstep: 5668.05 | bwd_allreduce_microstep: 16.86 | step_microstep: 20.26
[2025-04-25 18:27:29,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.63 | bwd: 5685.04 | bwd_inner: 5668.05 | bwd_allreduce: 16.91 | step: 20.25

 11%|█         | 4340/41250 [10:29:54<88:42:16,  8.65s/it]
                                                          
{'loss': 0.1613, 'grad_norm': 1.8060245513916016, 'learning_rate': 3.94097243349845e-05, 'epoch': 1.05}

 11%|█         | 4340/41250 [10:29:54<88:42:16,  8.65s/it][2025-04-25 18:27:38,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:27:38,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.01 | bwd_microstep: 5881.43 | bwd_inner_microstep: 5868.95 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.31
[2025-04-25 18:27:38,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.01 | bwd: 5881.45 | bwd_inner: 5868.95 | bwd_allreduce: 12.45 | step: 18.31

 11%|█         | 4341/41250 [10:30:03<89:28:04,  8.73s/it]
                                                          
{'loss': 0.0538, 'grad_norm': 1.7891043424606323, 'learning_rate': 3.9409345580871694e-05, 'epoch': 1.05}

 11%|█         | 4341/41250 [10:30:03<89:28:04,  8.73s/it][2025-04-25 18:27:47,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 18:27:47,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5762.35 | bwd_inner_microstep: 5700.51 | bwd_allreduce_microstep: 61.80 | step_microstep: 18.34
[2025-04-25 18:27:47,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5762.36 | bwd_inner: 5700.51 | bwd_allreduce: 61.81 | step: 18.35

 11%|█         | 4342/41250 [10:30:12<89:20:33,  8.71s/it]
                                                          
{'loss': 0.0509, 'grad_norm': 1.172315239906311, 'learning_rate': 3.940896670710408e-05, 'epoch': 1.05}

 11%|█         | 4342/41250 [10:30:12<89:20:33,  8.71s/it][2025-04-25 18:27:55,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:27:55,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5778.89 | bwd_inner_microstep: 5650.90 | bwd_allreduce_microstep: 127.95 | step_microstep: 18.60
[2025-04-25 18:27:55,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.48 | bwd: 5778.91 | bwd_inner: 5650.90 | bwd_allreduce: 127.97 | step: 18.60

 11%|█         | 4343/41250 [10:30:21<89:15:01,  8.71s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 0.6610034108161926, 'learning_rate': 3.940858771368401e-05, 'epoch': 1.05}

 11%|█         | 4343/41250 [10:30:21<89:15:01,  8.71s/it][2025-04-25 18:28:04,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:28:04,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.29 | bwd_microstep: 5738.50 | bwd_inner_microstep: 5697.20 | bwd_allreduce_microstep: 41.25 | step_microstep: 18.52
[2025-04-25 18:28:04,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.29 | bwd: 5738.51 | bwd_inner: 5697.20 | bwd_allreduce: 41.27 | step: 18.52

 11%|█         | 4344/41250 [10:30:29<89:10:06,  8.70s/it]
                                                          
{'loss': 0.1083, 'grad_norm': 2.063920736312866, 'learning_rate': 3.940820860061382e-05, 'epoch': 1.05}

 11%|█         | 4344/41250 [10:30:29<89:10:06,  8.70s/it][2025-04-25 18:28:13,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 18:28:13,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.88 | bwd_microstep: 5707.48 | bwd_inner_microstep: 5694.61 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.88
[2025-04-25 18:28:13,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.88 | bwd: 5707.49 | bwd_inner: 5694.61 | bwd_allreduce: 12.84 | step: 18.88

 11%|█         | 4345/41250 [10:30:38<89:01:24,  8.68s/it]
                                                          
{'loss': 0.1689, 'grad_norm': 1.796738624572754, 'learning_rate': 3.940782936789584e-05, 'epoch': 1.05}

 11%|█         | 4345/41250 [10:30:38<89:01:24,  8.68s/it][2025-04-25 18:28:21,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-25 18:28:21,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.29 | bwd_microstep: 5737.65 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 35.66 | step_microstep: 19.66
[2025-04-25 18:28:21,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.29 | bwd: 5737.67 | bwd_inner: 5701.94 | bwd_allreduce: 35.68 | step: 19.66

 11%|█         | 4346/41250 [10:30:47<89:01:04,  8.68s/it]
                                                          
{'loss': 0.0712, 'grad_norm': 0.8431259393692017, 'learning_rate': 3.9407450015532404e-05, 'epoch': 1.05}

 11%|█         | 4346/41250 [10:30:47<89:01:04,  8.68s/it][2025-04-25 18:28:30,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.18 | optimizer_step: 1.13
[2025-04-25 18:28:30,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5739.44 | bwd_inner_microstep: 5669.24 | bwd_allreduce_microstep: 70.13 | step_microstep: 20.13
[2025-04-25 18:28:30,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5739.46 | bwd_inner: 5669.24 | bwd_allreduce: 70.17 | step: 20.13

 11%|█         | 4347/41250 [10:30:55<88:59:39,  8.68s/it]
                                                          
{'loss': 0.123, 'grad_norm': 0.7493043541908264, 'learning_rate': 3.940707054352586e-05, 'epoch': 1.05}

 11%|█         | 4347/41250 [10:30:55<88:59:39,  8.68s/it][2025-04-25 18:28:39,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 18:28:39,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.15 | bwd_microstep: 5760.31 | bwd_inner_microstep: 5704.43 | bwd_allreduce_microstep: 55.83 | step_microstep: 19.15
[2025-04-25 18:28:39,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.15 | bwd: 5760.33 | bwd_inner: 5704.43 | bwd_allreduce: 55.86 | step: 19.15

 11%|█         | 4348/41250 [10:31:04<89:05:08,  8.69s/it]
                                                          
{'loss': 0.0886, 'grad_norm': 1.3719218969345093, 'learning_rate': 3.940669095187853e-05, 'epoch': 1.05}

 11%|█         | 4348/41250 [10:31:04<89:05:08,  8.69s/it][2025-04-25 18:28:47,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:28:47,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.95 | bwd_microstep: 5747.40 | bwd_inner_microstep: 5657.27 | bwd_allreduce_microstep: 90.08 | step_microstep: 18.63
[2025-04-25 18:28:47,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.95 | bwd: 5747.41 | bwd_inner: 5657.27 | bwd_allreduce: 90.10 | step: 18.63

 11%|█         | 4349/41250 [10:31:13<89:02:31,  8.69s/it]
                                                          
{'loss': 0.1333, 'grad_norm': 0.8937363624572754, 'learning_rate': 3.940631124059278e-05, 'epoch': 1.05}

 11%|█         | 4349/41250 [10:31:13<89:02:31,  8.69s/it][2025-04-25 18:28:56,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:28:56,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.78 | bwd_microstep: 5700.86 | bwd_inner_microstep: 5675.35 | bwd_allreduce_microstep: 25.46 | step_microstep: 18.62
[2025-04-25 18:28:56,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.78 | bwd: 5700.87 | bwd_inner: 5675.35 | bwd_allreduce: 25.48 | step: 18.62

 11%|█         | 4350/41250 [10:31:21<88:51:32,  8.67s/it]
                                                          
{'loss': 0.0367, 'grad_norm': 0.5375616550445557, 'learning_rate': 3.940593140967094e-05, 'epoch': 1.05}

 11%|█         | 4350/41250 [10:31:21<88:51:32,  8.67s/it][2025-04-25 18:29:05,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 18:29:05,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5792.45 | bwd_inner_microstep: 5665.28 | bwd_allreduce_microstep: 127.12 | step_microstep: 18.84
[2025-04-25 18:29:05,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5792.46 | bwd_inner: 5665.28 | bwd_allreduce: 127.14 | step: 18.84

 11%|█         | 4351/41250 [10:31:30<89:03:08,  8.69s/it]
                                                          
{'loss': 0.1449, 'grad_norm': 1.3790946006774902, 'learning_rate': 3.9405551459115344e-05, 'epoch': 1.05}

 11%|█         | 4351/41250 [10:31:30<89:03:08,  8.69s/it][2025-04-25 18:29:14,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 18:29:14,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.08 | bwd_microstep: 5747.26 | bwd_inner_microstep: 5709.18 | bwd_allreduce_microstep: 38.03 | step_microstep: 18.87
[2025-04-25 18:29:14,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5747.27 | bwd_inner: 5709.18 | bwd_allreduce: 38.05 | step: 18.88

 11%|█         | 4352/41250 [10:31:39<89:03:22,  8.69s/it]
                                                          
{'loss': 0.1458, 'grad_norm': 1.9944881200790405, 'learning_rate': 3.9405171388928345e-05, 'epoch': 1.06}

 11%|█         | 4352/41250 [10:31:39<89:03:22,  8.69s/it][2025-04-25 18:29:22,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 18:29:22,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.25 | bwd_microstep: 5693.97 | bwd_inner_microstep: 5672.12 | bwd_allreduce_microstep: 21.80 | step_microstep: 18.85
[2025-04-25 18:29:22,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.25 | bwd: 5693.98 | bwd_inner: 5672.12 | bwd_allreduce: 21.82 | step: 18.85

 11%|█         | 4353/41250 [10:31:47<88:53:14,  8.67s/it]
                                                          
{'loss': 0.1745, 'grad_norm': 1.9208825826644897, 'learning_rate': 3.940479119911228e-05, 'epoch': 1.06}

 11%|█         | 4353/41250 [10:31:47<88:53:14,  8.67s/it][2025-04-25 18:29:31,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:29:31,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.66 | bwd_microstep: 5773.66 | bwd_inner_microstep: 5665.47 | bwd_allreduce_microstep: 108.15 | step_microstep: 18.89
[2025-04-25 18:29:31,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.66 | bwd: 5773.68 | bwd_inner: 5665.47 | bwd_allreduce: 108.17 | step: 18.90

 11%|█         | 4354/41250 [10:31:56<88:57:31,  8.68s/it]
                                                          
{'loss': 0.2992, 'grad_norm': 4.0982255935668945, 'learning_rate': 3.940441088966949e-05, 'epoch': 1.06}

 11%|█         | 4354/41250 [10:31:56<88:57:31,  8.68s/it][2025-04-25 18:29:40,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 18:29:40,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.90 | bwd_microstep: 5744.23 | bwd_inner_microstep: 5660.06 | bwd_allreduce_microstep: 84.12 | step_microstep: 18.67
[2025-04-25 18:29:40,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.90 | bwd: 5744.24 | bwd_inner: 5660.06 | bwd_allreduce: 84.14 | step: 18.68

 11%|█         | 4355/41250 [10:32:05<88:55:59,  8.68s/it]
                                                          
{'loss': 0.0164, 'grad_norm': 0.25418123602867126, 'learning_rate': 3.940403046060232e-05, 'epoch': 1.06}

 11%|█         | 4355/41250 [10:32:05<88:55:59,  8.68s/it][2025-04-25 18:29:48,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 18:29:48,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.91 | bwd_microstep: 5718.44 | bwd_inner_microstep: 5705.68 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.89
[2025-04-25 18:29:48,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.91 | bwd: 5718.45 | bwd_inner: 5705.68 | bwd_allreduce: 12.73 | step: 18.89

 11%|█         | 4356/41250 [10:32:14<88:53:01,  8.67s/it]
                                                          
{'loss': 0.0577, 'grad_norm': 0.9849978685379028, 'learning_rate': 3.9403649911913134e-05, 'epoch': 1.06}

 11%|█         | 4356/41250 [10:32:14<88:53:01,  8.67s/it][2025-04-25 18:29:57,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 18:29:57,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.16 | bwd_microstep: 5817.31 | bwd_inner_microstep: 5648.49 | bwd_allreduce_microstep: 168.76 | step_microstep: 18.91
[2025-04-25 18:29:57,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.16 | bwd: 5817.32 | bwd_inner: 5648.49 | bwd_allreduce: 168.78 | step: 18.92

 11%|█         | 4357/41250 [10:32:22<89:04:32,  8.69s/it]
                                                          
{'loss': 0.1355, 'grad_norm': 1.2141590118408203, 'learning_rate': 3.940326924360425e-05, 'epoch': 1.06}

 11%|█         | 4357/41250 [10:32:22<89:04:32,  8.69s/it][2025-04-25 18:30:06,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:30:06,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.13 | bwd_microstep: 5728.91 | bwd_inner_microstep: 5700.79 | bwd_allreduce_microstep: 28.06 | step_microstep: 19.00
[2025-04-25 18:30:06,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.13 | bwd: 5728.93 | bwd_inner: 5700.79 | bwd_allreduce: 28.09 | step: 19.01

 11%|█         | 4358/41250 [10:32:31<89:01:17,  8.69s/it]
                                                          
{'loss': 0.1808, 'grad_norm': 1.2154712677001953, 'learning_rate': 3.940288845567803e-05, 'epoch': 1.06}

 11%|█         | 4358/41250 [10:32:31<89:01:17,  8.69s/it][2025-04-25 18:30:14,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:30:14,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.41 | bwd_microstep: 5729.34 | bwd_inner_microstep: 5716.62 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.88
[2025-04-25 18:30:14,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.41 | bwd: 5729.35 | bwd_inner: 5716.62 | bwd_allreduce: 12.69 | step: 18.88

 11%|█         | 4359/41250 [10:32:40<88:57:29,  8.68s/it]
                                                          
{'loss': 0.1459, 'grad_norm': 1.4988512992858887, 'learning_rate': 3.940250754813682e-05, 'epoch': 1.06}

 11%|█         | 4359/41250 [10:32:40<88:57:29,  8.68s/it][2025-04-25 18:30:23,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.97
[2025-04-25 18:30:23,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.57 | bwd_microstep: 5699.98 | bwd_inner_microstep: 5673.21 | bwd_allreduce_microstep: 26.72 | step_microstep: 19.20
[2025-04-25 18:30:23,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.57 | bwd: 5699.99 | bwd_inner: 5673.21 | bwd_allreduce: 26.74 | step: 19.20

 11%|█         | 4360/41250 [10:32:48<88:45:16,  8.66s/it]
                                                          
{'loss': 0.018, 'grad_norm': 0.18255771696567535, 'learning_rate': 3.940212652098297e-05, 'epoch': 1.06}

 11%|█         | 4360/41250 [10:32:48<88:45:16,  8.66s/it][2025-04-25 18:30:32,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 18:30:32,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.24 | bwd_microstep: 5727.11 | bwd_inner_microstep: 5713.88 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.32
[2025-04-25 18:30:32,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.24 | bwd: 5727.12 | bwd_inner: 5713.88 | bwd_allreduce: 13.20 | step: 19.32

 11%|█         | 4361/41250 [10:32:57<88:46:12,  8.66s/it]
                                                          
{'loss': 0.1089, 'grad_norm': 1.5533779859542847, 'learning_rate': 3.940174537421882e-05, 'epoch': 1.06}

 11%|█         | 4361/41250 [10:32:57<88:46:12,  8.66s/it][2025-04-25 18:30:40,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:30:40,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.67 | bwd_microstep: 5752.55 | bwd_inner_microstep: 5714.97 | bwd_allreduce_microstep: 37.54 | step_microstep: 19.03
[2025-04-25 18:30:40,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.67 | bwd: 5752.57 | bwd_inner: 5714.97 | bwd_allreduce: 37.56 | step: 19.03

 11%|█         | 4362/41250 [10:33:06<88:51:41,  8.67s/it]
                                                          
{'loss': 0.1644, 'grad_norm': 1.7783571481704712, 'learning_rate': 3.940136410784673e-05, 'epoch': 1.06}

 11%|█         | 4362/41250 [10:33:06<88:51:41,  8.67s/it][2025-04-25 18:30:49,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.16 | optimizer_step: 1.03
[2025-04-25 18:30:49,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.39 | bwd_microstep: 5712.44 | bwd_inner_microstep: 5699.55 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.40
[2025-04-25 18:30:49,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.39 | bwd: 5712.46 | bwd_inner: 5699.55 | bwd_allreduce: 12.87 | step: 19.41

 11%|█         | 4363/41250 [10:33:14<88:46:49,  8.66s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 2.5252654552459717, 'learning_rate': 3.9400982721869045e-05, 'epoch': 1.06}

 11%|█         | 4363/41250 [10:33:14<88:46:49,  8.66s/it][2025-04-25 18:30:58,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 18:30:58,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.09 | bwd_microstep: 5733.56 | bwd_inner_microstep: 5709.60 | bwd_allreduce_microstep: 23.90 | step_microstep: 19.08
[2025-04-25 18:30:58,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.09 | bwd: 5733.57 | bwd_inner: 5709.60 | bwd_allreduce: 23.92 | step: 19.08

 11%|█         | 4364/41250 [10:33:23<88:48:34,  8.67s/it]
                                                          
{'loss': 0.183, 'grad_norm': 3.0939369201660156, 'learning_rate': 3.940060121628812e-05, 'epoch': 1.06}

 11%|█         | 4364/41250 [10:33:23<88:48:34,  8.67s/it][2025-04-25 18:31:06,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.11 | optimizer_step: 1.03
[2025-04-25 18:31:06,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.65 | bwd_microstep: 5689.57 | bwd_inner_microstep: 5642.86 | bwd_allreduce_microstep: 46.65 | step_microstep: 19.29
[2025-04-25 18:31:06,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.65 | bwd: 5689.58 | bwd_inner: 5642.86 | bwd_allreduce: 46.67 | step: 19.29

 11%|█         | 4365/41250 [10:33:32<88:36:07,  8.65s/it]
                                                          
{'loss': 0.1147, 'grad_norm': 1.7657740116119385, 'learning_rate': 3.94002195911063e-05, 'epoch': 1.06}

 11%|█         | 4365/41250 [10:33:32<88:36:07,  8.65s/it][2025-04-25 18:31:15,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 18:31:15,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.17 | bwd_microstep: 5725.42 | bwd_inner_microstep: 5709.34 | bwd_allreduce_microstep: 16.03 | step_microstep: 18.85
[2025-04-25 18:31:15,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.17 | bwd: 5725.44 | bwd_inner: 5709.34 | bwd_allreduce: 16.05 | step: 18.85

 11%|█         | 4366/41250 [10:33:40<88:39:22,  8.65s/it]
                                                          
{'loss': 0.0868, 'grad_norm': 1.0539568662643433, 'learning_rate': 3.939983784632595e-05, 'epoch': 1.06}

 11%|█         | 4366/41250 [10:33:40<88:39:22,  8.65s/it][2025-04-25 18:31:23,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 18:31:23,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.04 | bwd_microstep: 5701.07 | bwd_inner_microstep: 5688.32 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.80
[2025-04-25 18:31:23,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.05 | bwd: 5701.08 | bwd_inner: 5688.32 | bwd_allreduce: 12.72 | step: 18.80

 11%|█         | 4367/41250 [10:33:49<88:35:17,  8.65s/it]
                                                          
{'loss': 0.1783, 'grad_norm': 1.577417016029358, 'learning_rate': 3.939945598194941e-05, 'epoch': 1.06}

 11%|█         | 4367/41250 [10:33:49<88:35:17,  8.65s/it][2025-04-25 18:31:32,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 18:31:32,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.96 | bwd_microstep: 5711.85 | bwd_inner_microstep: 5699.09 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.88
[2025-04-25 18:31:32,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.96 | bwd: 5711.87 | bwd_inner: 5699.09 | bwd_allreduce: 12.74 | step: 18.89

 11%|█         | 4368/41250 [10:33:57<88:37:00,  8.65s/it]
                                                          
{'loss': 0.0508, 'grad_norm': 0.614050030708313, 'learning_rate': 3.9399073997979036e-05, 'epoch': 1.06}

 11%|█         | 4368/41250 [10:33:57<88:37:00,  8.65s/it][2025-04-25 18:31:41,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:31:41,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.28 | bwd_microstep: 5674.00 | bwd_inner_microstep: 5661.16 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.80
[2025-04-25 18:31:41,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.28 | bwd: 5674.02 | bwd_inner: 5661.16 | bwd_allreduce: 12.82 | step: 18.80

 11%|█         | 4369/41250 [10:34:06<88:26:17,  8.63s/it]
                                                          
{'loss': 0.0943, 'grad_norm': 1.0155844688415527, 'learning_rate': 3.939869189441719e-05, 'epoch': 1.06}

 11%|█         | 4369/41250 [10:34:06<88:26:17,  8.63s/it][2025-04-25 18:31:49,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 18:31:49,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.05 | bwd_microstep: 5725.26 | bwd_inner_microstep: 5651.62 | bwd_allreduce_microstep: 73.59 | step_microstep: 18.69
[2025-04-25 18:31:49,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.05 | bwd: 5725.27 | bwd_inner: 5651.62 | bwd_allreduce: 73.61 | step: 18.71

 11%|█         | 4370/41250 [10:34:15<88:28:48,  8.64s/it]
                                                          
{'loss': 0.1436, 'grad_norm': 1.3770383596420288, 'learning_rate': 3.939830967126622e-05, 'epoch': 1.06}

 11%|█         | 4370/41250 [10:34:15<88:28:48,  8.64s/it][2025-04-25 18:31:58,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:31:58,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.62 | bwd_microstep: 5731.62 | bwd_inner_microstep: 5642.94 | bwd_allreduce_microstep: 88.64 | step_microstep: 18.83
[2025-04-25 18:31:58,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.62 | bwd: 5731.63 | bwd_inner: 5642.94 | bwd_allreduce: 88.65 | step: 18.84

 11%|█         | 4371/41250 [10:34:23<88:32:44,  8.64s/it]
                                                          
{'loss': 0.1558, 'grad_norm': 2.2909882068634033, 'learning_rate': 3.9397927328528496e-05, 'epoch': 1.06}

 11%|█         | 4371/41250 [10:34:23<88:32:44,  8.64s/it][2025-04-25 18:32:07,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:32:07,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.07 | bwd_microstep: 5753.41 | bwd_inner_microstep: 5655.53 | bwd_allreduce_microstep: 97.83 | step_microstep: 18.74
[2025-04-25 18:32:07,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.07 | bwd: 5753.42 | bwd_inner: 5655.53 | bwd_allreduce: 97.85 | step: 18.74

 11%|█         | 4372/41250 [10:34:32<88:38:53,  8.65s/it]
                                                          
{'loss': 0.1227, 'grad_norm': 1.7595747709274292, 'learning_rate': 3.9397544866206355e-05, 'epoch': 1.06}

 11%|█         | 4372/41250 [10:34:32<88:38:53,  8.65s/it][2025-04-25 18:32:15,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:32:15,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5697.02 | bwd_inner_microstep: 5639.95 | bwd_allreduce_microstep: 57.01 | step_microstep: 18.82
[2025-04-25 18:32:15,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5697.04 | bwd_inner: 5639.95 | bwd_allreduce: 57.04 | step: 18.82

 11%|█         | 4373/41250 [10:34:41<88:31:23,  8.64s/it]
                                                          
{'loss': 0.1915, 'grad_norm': 2.7969486713409424, 'learning_rate': 3.939716228430217e-05, 'epoch': 1.06}

 11%|█         | 4373/41250 [10:34:41<88:31:23,  8.64s/it][2025-04-25 18:32:24,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 18:32:24,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.37 | bwd_microstep: 5710.59 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.34
[2025-04-25 18:32:24,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.37 | bwd: 5710.60 | bwd_inner: 5697.92 | bwd_allreduce: 12.65 | step: 19.34

 11%|█         | 4374/41250 [10:34:49<88:32:11,  8.64s/it]
                                                          
{'loss': 0.0887, 'grad_norm': 0.7422292232513428, 'learning_rate': 3.93967795828183e-05, 'epoch': 1.06}

 11%|█         | 4374/41250 [10:34:49<88:32:11,  8.64s/it][2025-04-25 18:32:33,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 18:32:33,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.91 | bwd_microstep: 5676.56 | bwd_inner_microstep: 5663.78 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.46
[2025-04-25 18:32:33,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.91 | bwd: 5676.57 | bwd_inner: 5663.78 | bwd_allreduce: 12.75 | step: 18.46

 11%|█         | 4375/41250 [10:34:58<88:23:17,  8.63s/it]
                                                          
{'loss': 0.2203, 'grad_norm': 1.7668137550354004, 'learning_rate': 3.939639676175709e-05, 'epoch': 1.06}

 11%|█         | 4375/41250 [10:34:58<88:23:17,  8.63s/it][2025-04-25 18:32:41,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:32:41,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.76 | bwd_microstep: 5723.07 | bwd_inner_microstep: 5647.12 | bwd_allreduce_microstep: 75.91 | step_microstep: 18.69
[2025-04-25 18:32:41,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.76 | bwd: 5723.08 | bwd_inner: 5647.12 | bwd_allreduce: 75.92 | step: 18.69

 11%|█         | 4376/41250 [10:35:07<88:26:01,  8.63s/it]
                                                          
{'loss': 0.2211, 'grad_norm': 1.5097174644470215, 'learning_rate': 3.939601382112091e-05, 'epoch': 1.06}

 11%|█         | 4376/41250 [10:35:07<88:26:01,  8.63s/it][2025-04-25 18:32:50,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 18:32:50,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.99 | bwd_microstep: 5754.16 | bwd_inner_microstep: 5655.49 | bwd_allreduce_microstep: 98.61 | step_microstep: 18.54
[2025-04-25 18:32:50,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.99 | bwd: 5754.17 | bwd_inner: 5655.49 | bwd_allreduce: 98.63 | step: 18.54

 11%|█         | 4377/41250 [10:35:15<88:31:22,  8.64s/it]
                                                          
{'loss': 0.1688, 'grad_norm': 2.0830674171447754, 'learning_rate': 3.939563076091213e-05, 'epoch': 1.06}

 11%|█         | 4377/41250 [10:35:15<88:31:22,  8.64s/it][2025-04-25 18:32:59,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:32:59,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.78 | bwd_microstep: 5756.03 | bwd_inner_microstep: 5638.52 | bwd_allreduce_microstep: 117.47 | step_microstep: 18.60
[2025-04-25 18:32:59,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.78 | bwd: 5756.05 | bwd_inner: 5638.52 | bwd_allreduce: 117.49 | step: 18.60

 11%|█         | 4378/41250 [10:35:24<88:36:43,  8.65s/it]
                                                          
{'loss': 0.3217, 'grad_norm': 2.794948101043701, 'learning_rate': 3.939524758113309e-05, 'epoch': 1.06}

 11%|█         | 4378/41250 [10:35:24<88:36:43,  8.65s/it][2025-04-25 18:33:07,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:33:07,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5820.64 | bwd_inner_microstep: 5652.76 | bwd_allreduce_microstep: 167.83 | step_microstep: 18.89
[2025-04-25 18:33:07,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.77 | bwd: 5820.65 | bwd_inner: 5652.76 | bwd_allreduce: 167.85 | step: 18.90

 11%|█         | 4379/41250 [10:35:33<88:54:30,  8.68s/it]
                                                          
{'loss': 0.3309, 'grad_norm': 2.016970634460449, 'learning_rate': 3.939486428178616e-05, 'epoch': 1.06}

 11%|█         | 4379/41250 [10:35:33<88:54:30,  8.68s/it][2025-04-25 18:33:16,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:33:16,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.04 | bwd_microstep: 5690.29 | bwd_inner_microstep: 5650.56 | bwd_allreduce_microstep: 39.69 | step_microstep: 18.79
[2025-04-25 18:33:16,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.04 | bwd: 5690.30 | bwd_inner: 5650.56 | bwd_allreduce: 39.70 | step: 18.79

 11%|█         | 4380/41250 [10:35:41<88:38:24,  8.65s/it]
                                                          
{'loss': 0.1316, 'grad_norm': 4.002390384674072, 'learning_rate': 3.939448086287372e-05, 'epoch': 1.06}

 11%|█         | 4380/41250 [10:35:41<88:38:24,  8.65s/it][2025-04-25 18:33:25,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:33:25,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.36 | bwd_microstep: 5771.24 | bwd_inner_microstep: 5655.70 | bwd_allreduce_microstep: 115.49 | step_microstep: 18.79
[2025-04-25 18:33:25,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.36 | bwd: 5771.25 | bwd_inner: 5655.70 | bwd_allreduce: 115.51 | step: 18.79

 11%|█         | 4381/41250 [10:35:50<88:42:50,  8.66s/it]
                                                          
{'loss': 0.1699, 'grad_norm': 1.4788113832473755, 'learning_rate': 3.939409732439811e-05, 'epoch': 1.06}

 11%|█         | 4381/41250 [10:35:50<88:42:50,  8.66s/it][2025-04-25 18:33:33,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:33:33,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.08 | bwd_microstep: 5691.68 | bwd_inner_microstep: 5639.46 | bwd_allreduce_microstep: 52.17 | step_microstep: 18.75
[2025-04-25 18:33:33,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.08 | bwd: 5691.69 | bwd_inner: 5639.46 | bwd_allreduce: 52.18 | step: 18.76

 11%|█         | 4382/41250 [10:35:58<88:31:09,  8.64s/it]
                                                          
{'loss': 0.4661, 'grad_norm': 4.3773980140686035, 'learning_rate': 3.939371366636171e-05, 'epoch': 1.06}

 11%|█         | 4382/41250 [10:35:58<88:31:09,  8.64s/it][2025-04-25 18:33:42,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-25 18:33:42,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.19 | bwd_microstep: 5711.50 | bwd_inner_microstep: 5698.27 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.35
[2025-04-25 18:33:42,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.19 | bwd: 5711.52 | bwd_inner: 5698.27 | bwd_allreduce: 13.20 | step: 19.35

 11%|█         | 4383/41250 [10:36:07<88:30:01,  8.64s/it]
                                                          
{'loss': 0.1928, 'grad_norm': 1.420462965965271, 'learning_rate': 3.939332988876688e-05, 'epoch': 1.06}

 11%|█         | 4383/41250 [10:36:07<88:30:01,  8.64s/it][2025-04-25 18:33:51,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-25 18:33:51,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.91 | bwd_microstep: 5985.14 | bwd_inner_microstep: 5700.71 | bwd_allreduce_microstep: 284.38 | step_microstep: 19.44
[2025-04-25 18:33:51,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.91 | bwd: 5985.15 | bwd_inner: 5700.71 | bwd_allreduce: 284.40 | step: 19.44

 11%|█         | 4384/41250 [10:36:16<89:22:06,  8.73s/it]
                                                          
{'loss': 0.203, 'grad_norm': 1.6352598667144775, 'learning_rate': 3.9392945991615986e-05, 'epoch': 1.06}

 11%|█         | 4384/41250 [10:36:16<89:22:06,  8.73s/it][2025-04-25 18:33:59,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:33:59,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.30 | bwd_microstep: 5728.66 | bwd_inner_microstep: 5703.27 | bwd_allreduce_microstep: 25.34 | step_microstep: 18.89
[2025-04-25 18:33:59,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.30 | bwd: 5728.68 | bwd_inner: 5703.27 | bwd_allreduce: 25.36 | step: 18.89

 11%|█         | 4385/41250 [10:36:25<89:11:56,  8.71s/it]
                                                          
{'loss': 0.2102, 'grad_norm': 2.190444231033325, 'learning_rate': 3.939256197491139e-05, 'epoch': 1.06}

 11%|█         | 4385/41250 [10:36:25<89:11:56,  8.71s/it][2025-04-25 18:34:08,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.12 | optimizer_step: 1.04
[2025-04-25 18:34:08,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.24 | bwd_microstep: 5743.27 | bwd_inner_microstep: 5657.28 | bwd_allreduce_microstep: 85.93 | step_microstep: 19.47
[2025-04-25 18:34:08,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.24 | bwd: 5743.29 | bwd_inner: 5657.28 | bwd_allreduce: 85.96 | step: 19.47

 11%|█         | 4386/41250 [10:36:33<89:03:07,  8.70s/it]
                                                          
{'loss': 0.0791, 'grad_norm': 0.6603943705558777, 'learning_rate': 3.939217783865546e-05, 'epoch': 1.06}

 11%|█         | 4386/41250 [10:36:33<89:03:07,  8.70s/it][2025-04-25 18:34:17,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 18:34:17,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.37 | bwd_microstep: 5790.57 | bwd_inner_microstep: 5777.71 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.25
[2025-04-25 18:34:17,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.37 | bwd: 5790.59 | bwd_inner: 5777.71 | bwd_allreduce: 12.83 | step: 19.25

 11%|█         | 4387/41250 [10:36:42<89:15:17,  8.72s/it]
                                                          
{'loss': 0.277, 'grad_norm': 3.764476776123047, 'learning_rate': 3.939179358285058e-05, 'epoch': 1.06}

 11%|█         | 4387/41250 [10:36:42<89:15:17,  8.72s/it][2025-04-25 18:34:25,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 18:34:25,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5710.55 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.80
[2025-04-25 18:34:25,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5710.57 | bwd_inner: 5697.91 | bwd_allreduce: 12.62 | step: 18.80

 11%|█         | 4388/41250 [10:36:51<89:01:40,  8.69s/it]
                                                          
{'loss': 0.1864, 'grad_norm': 2.2933590412139893, 'learning_rate': 3.93914092074991e-05, 'epoch': 1.06}

 11%|█         | 4388/41250 [10:36:51<89:01:40,  8.69s/it][2025-04-25 18:34:34,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:34:34,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.73 | bwd_microstep: 5687.46 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 24.20 | step_microstep: 18.47
[2025-04-25 18:34:34,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.73 | bwd: 5687.47 | bwd_inner: 5663.21 | bwd_allreduce: 24.22 | step: 18.47

 11%|█         | 4389/41250 [10:36:59<88:44:24,  8.67s/it]
                                                          
{'loss': 0.1718, 'grad_norm': 1.1613348722457886, 'learning_rate': 3.93910247126034e-05, 'epoch': 1.06}

 11%|█         | 4389/41250 [10:36:59<88:44:24,  8.67s/it][2025-04-25 18:34:43,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 18:34:43,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.84 | bwd_microstep: 5710.94 | bwd_inner_microstep: 5658.33 | bwd_allreduce_microstep: 52.55 | step_microstep: 20.26
[2025-04-25 18:34:43,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.84 | bwd: 5710.96 | bwd_inner: 5658.33 | bwd_allreduce: 52.57 | step: 20.26

 11%|█         | 4390/41250 [10:37:08<88:36:07,  8.65s/it]
                                                          
{'loss': 0.1018, 'grad_norm': 2.734412670135498, 'learning_rate': 3.939064009816584e-05, 'epoch': 1.06}

 11%|█         | 4390/41250 [10:37:08<88:36:07,  8.65s/it][2025-04-25 18:34:51,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:34:51,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.82 | bwd_microstep: 5732.64 | bwd_inner_microstep: 5719.92 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.71
[2025-04-25 18:34:51,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.82 | bwd: 5732.65 | bwd_inner: 5719.92 | bwd_allreduce: 12.69 | step: 18.71

 11%|█         | 4391/41250 [10:37:17<88:39:22,  8.66s/it]
                                                          
{'loss': 0.0552, 'grad_norm': 1.0721778869628906, 'learning_rate': 3.93902553641888e-05, 'epoch': 1.06}

 11%|█         | 4391/41250 [10:37:17<88:39:22,  8.66s/it][2025-04-25 18:35:00,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:35:00,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.84 | bwd_microstep: 5771.75 | bwd_inner_microstep: 5689.19 | bwd_allreduce_microstep: 82.51 | step_microstep: 18.82
[2025-04-25 18:35:00,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.84 | bwd: 5771.76 | bwd_inner: 5689.19 | bwd_allreduce: 82.53 | step: 18.82

 11%|█         | 4392/41250 [10:37:25<88:47:47,  8.67s/it]
                                                          
{'loss': 0.272, 'grad_norm': 2.5128087997436523, 'learning_rate': 3.938987051067465e-05, 'epoch': 1.06}

 11%|█         | 4392/41250 [10:37:25<88:47:47,  8.67s/it][2025-04-25 18:35:09,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 18:35:09,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.81 | bwd_microstep: 5896.24 | bwd_inner_microstep: 5700.53 | bwd_allreduce_microstep: 195.66 | step_microstep: 18.44
[2025-04-25 18:35:09,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.81 | bwd: 5896.25 | bwd_inner: 5700.53 | bwd_allreduce: 195.68 | step: 18.44

 11%|█         | 4393/41250 [10:37:34<89:15:50,  8.72s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.4485118091106415, 'learning_rate': 3.938948553762576e-05, 'epoch': 1.06}

 11%|█         | 4393/41250 [10:37:34<89:15:50,  8.72s/it][2025-04-25 18:35:18,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-25 18:35:18,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.96 | bwd_microstep: 5790.84 | bwd_inner_microstep: 5703.61 | bwd_allreduce_microstep: 87.18 | step_microstep: 18.82
[2025-04-25 18:35:18,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.96 | bwd: 5790.85 | bwd_inner: 5703.61 | bwd_allreduce: 87.20 | step: 18.82

 11%|█         | 4394/41250 [10:37:43<89:16:28,  8.72s/it]
                                                          
{'loss': 0.3051, 'grad_norm': 1.8765010833740234, 'learning_rate': 3.938910044504451e-05, 'epoch': 1.07}

 11%|█         | 4394/41250 [10:37:43<89:16:28,  8.72s/it][2025-04-25 18:35:26,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 18:35:26,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5770.41 | bwd_inner_microstep: 5662.93 | bwd_allreduce_microstep: 107.43 | step_microstep: 19.09
[2025-04-25 18:35:26,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5770.43 | bwd_inner: 5662.93 | bwd_allreduce: 107.45 | step: 19.10

 11%|█         | 4395/41250 [10:37:52<89:13:12,  8.72s/it]
                                                          
{'loss': 0.1475, 'grad_norm': 1.8870352506637573, 'learning_rate': 3.938871523293327e-05, 'epoch': 1.07}

 11%|█         | 4395/41250 [10:37:52<89:13:12,  8.72s/it][2025-04-25 18:35:35,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 18:35:35,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5780.01 | bwd_inner_microstep: 5655.17 | bwd_allreduce_microstep: 124.79 | step_microstep: 18.93
[2025-04-25 18:35:35,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5780.02 | bwd_inner: 5655.17 | bwd_allreduce: 124.81 | step: 18.93

 11%|█         | 4396/41250 [10:38:00<89:09:34,  8.71s/it]
                                                          
{'loss': 0.1032, 'grad_norm': 1.218840479850769, 'learning_rate': 3.938832990129441e-05, 'epoch': 1.07}

 11%|█         | 4396/41250 [10:38:00<89:09:34,  8.71s/it][2025-04-25 18:35:44,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 18:35:44,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.20 | bwd_microstep: 5739.42 | bwd_inner_microstep: 5664.86 | bwd_allreduce_microstep: 74.51 | step_microstep: 19.16
[2025-04-25 18:35:44,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.20 | bwd: 5739.44 | bwd_inner: 5664.86 | bwd_allreduce: 74.53 | step: 19.16

 11%|█         | 4397/41250 [10:38:09<88:59:01,  8.69s/it]
                                                          
{'loss': 0.1292, 'grad_norm': 1.3917323350906372, 'learning_rate': 3.9387944450130306e-05, 'epoch': 1.07}

 11%|█         | 4397/41250 [10:38:09<88:59:01,  8.69s/it][2025-04-25 18:35:52,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.20 | optimizer_step: 0.94
[2025-04-25 18:35:52,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.06 | bwd_microstep: 5763.58 | bwd_inner_microstep: 5693.94 | bwd_allreduce_microstep: 69.59 | step_microstep: 19.43
[2025-04-25 18:35:52,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.06 | bwd: 5763.59 | bwd_inner: 5693.94 | bwd_allreduce: 69.61 | step: 19.43

 11%|█         | 4398/41250 [10:38:18<88:59:53,  8.69s/it]
                                                          
{'loss': 0.0682, 'grad_norm': 0.7918466329574585, 'learning_rate': 3.938755887944335e-05, 'epoch': 1.07}

 11%|█         | 4398/41250 [10:38:18<88:59:53,  8.69s/it][2025-04-25 18:36:01,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 18:36:01,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.46 | bwd_microstep: 5832.25 | bwd_inner_microstep: 5659.06 | bwd_allreduce_microstep: 173.14 | step_microstep: 18.70
[2025-04-25 18:36:01,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.46 | bwd: 5832.27 | bwd_inner: 5659.06 | bwd_allreduce: 173.16 | step: 18.70

 11%|█         | 4399/41250 [10:38:26<89:10:31,  8.71s/it]
                                                          
{'loss': 0.191, 'grad_norm': 1.6946619749069214, 'learning_rate': 3.9387173189235896e-05, 'epoch': 1.07}

 11%|█         | 4399/41250 [10:38:26<89:10:31,  8.71s/it][2025-04-25 18:36:10,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 18:36:10,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.48 | bwd_microstep: 5776.73 | bwd_inner_microstep: 5707.07 | bwd_allreduce_microstep: 69.61 | step_microstep: 19.27
[2025-04-25 18:36:10,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.48 | bwd: 5776.74 | bwd_inner: 5707.07 | bwd_allreduce: 69.63 | step: 19.27

 11%|█         | 4400/41250 [10:38:35<89:12:10,  8.71s/it]
                                                          
{'loss': 0.2191, 'grad_norm': 1.5005919933319092, 'learning_rate': 3.938678737951034e-05, 'epoch': 1.07}

 11%|█         | 4400/41250 [10:38:35<89:12:10,  8.71s/it][2025-04-25 18:36:19,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:36:19,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.18 | bwd_microstep: 5773.94 | bwd_inner_microstep: 5654.60 | bwd_allreduce_microstep: 119.29 | step_microstep: 18.62
[2025-04-25 18:36:19,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.18 | bwd: 5773.96 | bwd_inner: 5654.60 | bwd_allreduce: 119.31 | step: 18.63

 11%|█         | 4401/41250 [10:38:44<89:08:21,  8.71s/it]
                                                          
{'loss': 0.0291, 'grad_norm': 0.30797114968299866, 'learning_rate': 3.938640145026905e-05, 'epoch': 1.07}

 11%|█         | 4401/41250 [10:38:44<89:08:21,  8.71s/it][2025-04-25 18:36:27,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:36:27,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.58 | bwd_microstep: 5728.39 | bwd_inner_microstep: 5715.52 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.66
[2025-04-25 18:36:27,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.58 | bwd: 5728.40 | bwd_inner: 5715.52 | bwd_allreduce: 12.84 | step: 18.66

 11%|█         | 4402/41250 [10:38:53<89:01:40,  8.70s/it]
                                                          
{'loss': 0.0985, 'grad_norm': 0.942489504814148, 'learning_rate': 3.9386015401514406e-05, 'epoch': 1.07}

 11%|█         | 4402/41250 [10:38:53<89:01:40,  8.70s/it][2025-04-25 18:36:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 18:36:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.33 | bwd_microstep: 5768.48 | bwd_inner_microstep: 5671.46 | bwd_allreduce_microstep: 96.97 | step_microstep: 18.80
[2025-04-25 18:36:36,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.33 | bwd: 5768.49 | bwd_inner: 5671.46 | bwd_allreduce: 96.99 | step: 18.80

 11%|█         | 4403/41250 [10:39:01<89:03:20,  8.70s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.446420431137085, 'learning_rate': 3.9385629233248795e-05, 'epoch': 1.07}

 11%|█         | 4403/41250 [10:39:01<89:03:20,  8.70s/it][2025-04-25 18:36:45,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 18:36:45,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.61 | bwd_microstep: 5739.95 | bwd_inner_microstep: 5664.57 | bwd_allreduce_microstep: 75.33 | step_microstep: 18.75
[2025-04-25 18:36:45,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.61 | bwd: 5739.96 | bwd_inner: 5664.57 | bwd_allreduce: 75.35 | step: 18.75

 11%|█         | 4404/41250 [10:39:10<88:56:39,  8.69s/it]
                                                          
{'loss': 0.0737, 'grad_norm': 0.6900420188903809, 'learning_rate': 3.9385242945474586e-05, 'epoch': 1.07}

 11%|█         | 4404/41250 [10:39:10<88:56:39,  8.69s/it][2025-04-25 18:36:53,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:36:53,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.58 | bwd_microstep: 5731.26 | bwd_inner_microstep: 5718.56 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.64
[2025-04-25 18:36:53,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.58 | bwd: 5731.27 | bwd_inner: 5718.56 | bwd_allreduce: 12.67 | step: 18.64

 11%|█         | 4405/41250 [10:39:19<88:54:01,  8.69s/it]
                                                          
{'loss': 0.1729, 'grad_norm': 1.9499508142471313, 'learning_rate': 3.938485653819417e-05, 'epoch': 1.07}

 11%|█         | 4405/41250 [10:39:19<88:54:01,  8.69s/it][2025-04-25 18:37:02,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:37:02,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.34 | bwd_microstep: 5717.18 | bwd_inner_microstep: 5704.10 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.90
[2025-04-25 18:37:02,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.34 | bwd: 5717.19 | bwd_inner: 5704.10 | bwd_allreduce: 13.06 | step: 18.91

 11%|█         | 4406/41250 [10:39:27<88:48:00,  8.68s/it]
                                                          
{'loss': 0.0776, 'grad_norm': 0.8000380396842957, 'learning_rate': 3.938447001140993e-05, 'epoch': 1.07}

 11%|█         | 4406/41250 [10:39:27<88:48:00,  8.68s/it][2025-04-25 18:37:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:37:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.14 | bwd_microstep: 5757.69 | bwd_inner_microstep: 5687.64 | bwd_allreduce_microstep: 70.01 | step_microstep: 18.54
[2025-04-25 18:37:11,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.14 | bwd: 5757.70 | bwd_inner: 5687.64 | bwd_allreduce: 70.02 | step: 18.54

 11%|█         | 4407/41250 [10:39:36<88:50:43,  8.68s/it]
                                                          
{'loss': 0.0941, 'grad_norm': 1.1551799774169922, 'learning_rate': 3.938408336512424e-05, 'epoch': 1.07}

 11%|█         | 4407/41250 [10:39:36<88:50:43,  8.68s/it][2025-04-25 18:37:19,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 18:37:19,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5885.92 | bwd_inner_microstep: 5701.14 | bwd_allreduce_microstep: 184.72 | step_microstep: 18.65
[2025-04-25 18:37:19,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5885.93 | bwd_inner: 5701.14 | bwd_allreduce: 184.75 | step: 18.65

 11%|█         | 4408/41250 [10:39:45<89:17:33,  8.73s/it]
                                                          
{'loss': 0.0816, 'grad_norm': 1.5748038291931152, 'learning_rate': 3.938369659933949e-05, 'epoch': 1.07}

 11%|█         | 4408/41250 [10:39:45<89:17:33,  8.73s/it][2025-04-25 18:37:28,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 18:37:28,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.17 | bwd_microstep: 5713.07 | bwd_inner_microstep: 5653.30 | bwd_allreduce_microstep: 59.72 | step_microstep: 18.78
[2025-04-25 18:37:28,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.17 | bwd: 5713.08 | bwd_inner: 5653.30 | bwd_allreduce: 59.74 | step: 18.79

 11%|█         | 4409/41250 [10:39:53<89:00:20,  8.70s/it]
                                                          
{'loss': 0.3335, 'grad_norm': 2.518967628479004, 'learning_rate': 3.938330971405806e-05, 'epoch': 1.07}

 11%|█         | 4409/41250 [10:39:53<89:00:20,  8.70s/it][2025-04-25 18:37:37,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 18:37:37,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5741.48 | bwd_inner_microstep: 5660.97 | bwd_allreduce_microstep: 80.47 | step_microstep: 19.22
[2025-04-25 18:37:37,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.76 | bwd: 5741.50 | bwd_inner: 5660.97 | bwd_allreduce: 80.49 | step: 19.22

 11%|█         | 4410/41250 [10:40:02<88:54:42,  8.69s/it]
                                                          
{'loss': 0.2221, 'grad_norm': 1.7776679992675781, 'learning_rate': 3.938292270928234e-05, 'epoch': 1.07}

 11%|█         | 4410/41250 [10:40:02<88:54:42,  8.69s/it][2025-04-25 18:37:45,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.72 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:37:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5698.90 | bwd_inner_microstep: 5686.40 | bwd_allreduce_microstep: 12.45 | step_microstep: 19.14
[2025-04-25 18:37:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5698.91 | bwd_inner: 5686.40 | bwd_allreduce: 12.47 | step: 19.14

 11%|█         | 4411/41250 [10:40:11<88:43:51,  8.67s/it]
                                                          
{'loss': 0.0244, 'grad_norm': 0.7130568623542786, 'learning_rate': 3.938253558501472e-05, 'epoch': 1.07}

 11%|█         | 4411/41250 [10:40:11<88:43:51,  8.67s/it][2025-04-25 18:37:54,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.27 | optimizer_step: 1.02
[2025-04-25 18:37:54,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.49 | bwd_microstep: 5785.57 | bwd_inner_microstep: 5662.56 | bwd_allreduce_microstep: 122.96 | step_microstep: 19.91
[2025-04-25 18:37:54,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.49 | bwd: 5785.59 | bwd_inner: 5662.56 | bwd_allreduce: 122.98 | step: 19.91

 11%|█         | 4412/41250 [10:40:19<88:49:05,  8.68s/it]
                                                          
{'loss': 0.0699, 'grad_norm': 1.0552712678909302, 'learning_rate': 3.9382148341257575e-05, 'epoch': 1.07}

 11%|█         | 4412/41250 [10:40:19<88:49:05,  8.68s/it][2025-04-25 18:38:03,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 18:38:03,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5807.97 | bwd_inner_microstep: 5651.98 | bwd_allreduce_microstep: 155.94 | step_microstep: 19.02
[2025-04-25 18:38:03,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5807.98 | bwd_inner: 5651.98 | bwd_allreduce: 155.96 | step: 19.02

 11%|█         | 4413/41250 [10:40:28<88:56:45,  8.69s/it]
                                                          
{'loss': 0.2272, 'grad_norm': 1.8068305253982544, 'learning_rate': 3.9381760978013304e-05, 'epoch': 1.07}

 11%|█         | 4413/41250 [10:40:28<88:56:45,  8.69s/it][2025-04-25 18:38:11,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:38:11,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.23 | bwd_microstep: 5706.06 | bwd_inner_microstep: 5693.42 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.75
[2025-04-25 18:38:11,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.23 | bwd: 5706.07 | bwd_inner: 5693.42 | bwd_allreduce: 12.61 | step: 18.75

 11%|█         | 4414/41250 [10:40:37<88:46:09,  8.68s/it]
                                                          
{'loss': 0.2516, 'grad_norm': 2.5417561531066895, 'learning_rate': 3.938137349528428e-05, 'epoch': 1.07}

 11%|█         | 4414/41250 [10:40:37<88:46:09,  8.68s/it][2025-04-25 18:38:20,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:38:20,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5703.05 | bwd_inner_microstep: 5690.17 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.87
[2025-04-25 18:38:20,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.40 | bwd: 5703.07 | bwd_inner: 5690.17 | bwd_allreduce: 12.86 | step: 18.88

 11%|█         | 4415/41250 [10:40:45<88:38:53,  8.66s/it]
                                                          
{'loss': 0.2104, 'grad_norm': 1.9450719356536865, 'learning_rate': 3.938098589307291e-05, 'epoch': 1.07}

 11%|█         | 4415/41250 [10:40:45<88:38:53,  8.66s/it][2025-04-25 18:38:29,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:38:29,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5769.23 | bwd_inner_microstep: 5654.08 | bwd_allreduce_microstep: 115.11 | step_microstep: 18.64
[2025-04-25 18:38:29,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5769.25 | bwd_inner: 5654.08 | bwd_allreduce: 115.13 | step: 18.64

 11%|█         | 4416/41250 [10:40:54<88:41:49,  8.67s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 0.6060265898704529, 'learning_rate': 3.938059817138157e-05, 'epoch': 1.07}

 11%|█         | 4416/41250 [10:40:54<88:41:49,  8.67s/it][2025-04-25 18:38:37,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.12 | optimizer_step: 1.16
[2025-04-25 18:38:37,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.51 | bwd_microstep: 5770.46 | bwd_inner_microstep: 5650.24 | bwd_allreduce_microstep: 120.16 | step_microstep: 19.72
[2025-04-25 18:38:37,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.51 | bwd: 5770.47 | bwd_inner: 5650.24 | bwd_allreduce: 120.19 | step: 19.72

 11%|█         | 4417/41250 [10:41:03<88:44:54,  8.67s/it]
                                                          
{'loss': 0.044, 'grad_norm': 0.7225815057754517, 'learning_rate': 3.9380210330212644e-05, 'epoch': 1.07}

 11%|█         | 4417/41250 [10:41:03<88:44:54,  8.67s/it][2025-04-25 18:38:46,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.30 | optimizer_step: 0.95
[2025-04-25 18:38:46,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.99 | bwd_microstep: 5679.14 | bwd_inner_microstep: 5654.12 | bwd_allreduce_microstep: 24.96 | step_microstep: 19.96
[2025-04-25 18:38:46,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.99 | bwd: 5679.15 | bwd_inner: 5654.12 | bwd_allreduce: 24.98 | step: 19.96

 11%|█         | 4418/41250 [10:41:11<88:32:27,  8.65s/it]
                                                          
{'loss': 0.3643, 'grad_norm': 2.5174543857574463, 'learning_rate': 3.9379822369568544e-05, 'epoch': 1.07}

 11%|█         | 4418/41250 [10:41:11<88:32:27,  8.65s/it][2025-04-25 18:38:55,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:38:55,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5717.67 | bwd_inner_microstep: 5704.85 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.04
[2025-04-25 18:38:55,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5717.69 | bwd_inner: 5704.85 | bwd_allreduce: 12.80 | step: 19.04

 11%|█         | 4419/41250 [10:41:20<88:31:30,  8.65s/it]
                                                          
{'loss': 0.0935, 'grad_norm': 1.332194209098816, 'learning_rate': 3.937943428945165e-05, 'epoch': 1.07}

 11%|█         | 4419/41250 [10:41:20<88:31:30,  8.65s/it][2025-04-25 18:39:03,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.19 | optimizer_step: 0.89
[2025-04-25 18:39:03,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.14 | bwd_microstep: 5747.91 | bwd_inner_microstep: 5645.02 | bwd_allreduce_microstep: 102.84 | step_microstep: 19.16
[2025-04-25 18:39:03,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.14 | bwd: 5747.92 | bwd_inner: 5645.02 | bwd_allreduce: 102.85 | step: 19.16

 11%|█         | 4420/41250 [10:41:29<88:32:28,  8.65s/it]
                                                          
{'loss': 0.2032, 'grad_norm': 1.5955408811569214, 'learning_rate': 3.937904608986435e-05, 'epoch': 1.07}

 11%|█         | 4420/41250 [10:41:29<88:32:28,  8.65s/it][2025-04-25 18:39:12,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.15 | optimizer_step: 0.92
[2025-04-25 18:39:12,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.00 | bwd_microstep: 5873.31 | bwd_inner_microstep: 5662.96 | bwd_allreduce_microstep: 210.30 | step_microstep: 18.97
[2025-04-25 18:39:12,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.00 | bwd: 5873.32 | bwd_inner: 5662.96 | bwd_allreduce: 210.32 | step: 18.97

 11%|█         | 4421/41250 [10:41:37<88:56:25,  8.69s/it]
                                                          
{'loss': 0.1767, 'grad_norm': 1.8288027048110962, 'learning_rate': 3.937865777080905e-05, 'epoch': 1.07}

 11%|█         | 4421/41250 [10:41:37<88:56:25,  8.69s/it][2025-04-25 18:39:21,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.14 | optimizer_step: 0.92
[2025-04-25 18:39:21,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5858.40 | bwd_inner_microstep: 5686.32 | bwd_allreduce_microstep: 172.03 | step_microstep: 18.74
[2025-04-25 18:39:21,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5858.41 | bwd_inner: 5686.32 | bwd_allreduce: 172.05 | step: 18.74

 11%|█         | 4422/41250 [10:41:46<89:13:26,  8.72s/it]
                                                          
{'loss': 0.1271, 'grad_norm': 1.1595779657363892, 'learning_rate': 3.937826933228813e-05, 'epoch': 1.07}

 11%|█         | 4422/41250 [10:41:46<89:13:26,  8.72s/it][2025-04-25 18:39:30,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 18:39:30,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.82 | bwd_microstep: 5711.36 | bwd_inner_microstep: 5698.42 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.03
[2025-04-25 18:39:30,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.82 | bwd: 5711.38 | bwd_inner: 5698.42 | bwd_allreduce: 12.91 | step: 19.03

 11%|█         | 4423/41250 [10:41:55<88:58:09,  8.70s/it]
                                                          
{'loss': 0.1151, 'grad_norm': 1.0234304666519165, 'learning_rate': 3.9377880774303994e-05, 'epoch': 1.07}

 11%|█         | 4423/41250 [10:41:55<88:58:09,  8.70s/it][2025-04-25 18:39:38,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:39:38,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.75 | bwd_microstep: 5692.81 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 40.30 | step_microstep: 19.30
[2025-04-25 18:39:38,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.75 | bwd: 5692.82 | bwd_inner: 5652.46 | bwd_allreduce: 40.32 | step: 19.31

 11%|█         | 4424/41250 [10:42:03<88:40:43,  8.67s/it]
                                                          
{'loss': 0.2381, 'grad_norm': 2.004948139190674, 'learning_rate': 3.9377492096859036e-05, 'epoch': 1.07}

 11%|█         | 4424/41250 [10:42:03<88:40:43,  8.67s/it][2025-04-25 18:39:47,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 18:39:47,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.67 | bwd_microstep: 5671.55 | bwd_inner_microstep: 5635.10 | bwd_allreduce_microstep: 36.40 | step_microstep: 18.97
[2025-04-25 18:39:47,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.67 | bwd: 5671.56 | bwd_inner: 5635.10 | bwd_allreduce: 36.42 | step: 18.97

 11%|█         | 4425/41250 [10:42:12<88:24:06,  8.64s/it]
                                                          
{'loss': 0.2867, 'grad_norm': 1.8418034315109253, 'learning_rate': 3.9377103299955645e-05, 'epoch': 1.07}

 11%|█         | 4425/41250 [10:42:12<88:24:06,  8.64s/it][2025-04-25 18:39:56,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:39:56,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.63 | bwd_microstep: 5820.61 | bwd_inner_microstep: 5755.61 | bwd_allreduce_microstep: 64.96 | step_microstep: 18.63
[2025-04-25 18:39:56,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.63 | bwd: 5820.63 | bwd_inner: 5755.61 | bwd_allreduce: 64.98 | step: 18.63

 11%|█         | 4426/41250 [10:42:21<88:49:23,  8.68s/it]
                                                          
{'loss': 0.1271, 'grad_norm': 0.7210027575492859, 'learning_rate': 3.937671438359623e-05, 'epoch': 1.07}

 11%|█         | 4426/41250 [10:42:21<88:49:23,  8.68s/it][2025-04-25 18:40:04,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:40:04,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.23 | bwd_microstep: 5736.82 | bwd_inner_microstep: 5688.26 | bwd_allreduce_microstep: 48.52 | step_microstep: 18.98
[2025-04-25 18:40:04,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.23 | bwd: 5736.84 | bwd_inner: 5688.26 | bwd_allreduce: 48.53 | step: 18.98

 11%|█         | 4427/41250 [10:42:30<88:44:54,  8.68s/it]
                                                          
{'loss': 0.1041, 'grad_norm': 1.1251496076583862, 'learning_rate': 3.9376325347783176e-05, 'epoch': 1.07}

 11%|█         | 4427/41250 [10:42:30<88:44:54,  8.68s/it][2025-04-25 18:40:13,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 18:40:13,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.74 | bwd_microstep: 5700.59 | bwd_inner_microstep: 5687.72 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.12
[2025-04-25 18:40:13,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.74 | bwd: 5700.61 | bwd_inner: 5687.72 | bwd_allreduce: 12.85 | step: 19.12

 11%|█         | 4428/41250 [10:42:38<88:35:35,  8.66s/it]
                                                          
{'loss': 0.2991, 'grad_norm': 2.2353999614715576, 'learning_rate': 3.9375936192518895e-05, 'epoch': 1.07}

 11%|█         | 4428/41250 [10:42:38<88:35:35,  8.66s/it][2025-04-25 18:40:21,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 18:40:21,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.28 | bwd_microstep: 5734.22 | bwd_inner_microstep: 5652.64 | bwd_allreduce_microstep: 81.53 | step_microstep: 18.95
[2025-04-25 18:40:21,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.28 | bwd: 5734.24 | bwd_inner: 5652.64 | bwd_allreduce: 81.55 | step: 18.95

 11%|█         | 4429/41250 [10:42:47<88:31:58,  8.66s/it]
                                                          
{'loss': 0.0438, 'grad_norm': 0.40286049246788025, 'learning_rate': 3.937554691780577e-05, 'epoch': 1.07}

 11%|█         | 4429/41250 [10:42:47<88:31:58,  8.66s/it][2025-04-25 18:40:30,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:40:30,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5701.39 | bwd_inner_microstep: 5688.39 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.03
[2025-04-25 18:40:30,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5701.40 | bwd_inner: 5688.39 | bwd_allreduce: 12.97 | step: 19.04

 11%|█         | 4430/41250 [10:42:55<88:26:52,  8.65s/it]
                                                          
{'loss': 0.2159, 'grad_norm': 2.207287073135376, 'learning_rate': 3.9375157523646215e-05, 'epoch': 1.07}

 11%|█         | 4430/41250 [10:42:55<88:26:52,  8.65s/it][2025-04-25 18:40:39,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.73 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:40:39,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.11 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5696.56 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.40
[2025-04-25 18:40:39,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.11 | bwd: 5709.57 | bwd_inner: 5696.56 | bwd_allreduce: 12.97 | step: 19.40

 11%|█         | 4431/41250 [10:43:04<88:27:36,  8.65s/it]
                                                          
{'loss': 0.2461, 'grad_norm': 2.833239793777466, 'learning_rate': 3.9374768010042626e-05, 'epoch': 1.07}

 11%|█         | 4431/41250 [10:43:04<88:27:36,  8.65s/it][2025-04-25 18:40:47,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:40:47,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.21 | bwd_microstep: 5674.00 | bwd_inner_microstep: 5647.81 | bwd_allreduce_microstep: 26.14 | step_microstep: 18.61
[2025-04-25 18:40:47,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.21 | bwd: 5674.01 | bwd_inner: 5647.81 | bwd_allreduce: 26.16 | step: 18.62

 11%|█         | 4432/41250 [10:43:13<88:15:00,  8.63s/it]
                                                          
{'loss': 0.2277, 'grad_norm': 1.44098699092865, 'learning_rate': 3.93743783769974e-05, 'epoch': 1.07}

 11%|█         | 4432/41250 [10:43:13<88:15:00,  8.63s/it][2025-04-25 18:40:56,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:40:56,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5692.02 | bwd_inner_microstep: 5679.43 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.31
[2025-04-25 18:40:56,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5692.03 | bwd_inner: 5679.43 | bwd_allreduce: 12.57 | step: 18.31

 11%|█         | 4433/41250 [10:43:21<88:14:17,  8.63s/it]
                                                          
{'loss': 0.1688, 'grad_norm': 2.2626988887786865, 'learning_rate': 3.937398862451294e-05, 'epoch': 1.07}

 11%|█         | 4433/41250 [10:43:21<88:14:17,  8.63s/it][2025-04-25 18:41:05,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 18:41:05,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.50 | bwd_microstep: 5749.11 | bwd_inner_microstep: 5648.51 | bwd_allreduce_microstep: 100.55 | step_microstep: 19.19
[2025-04-25 18:41:05,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.50 | bwd: 5749.12 | bwd_inner: 5648.51 | bwd_allreduce: 100.57 | step: 19.19

 11%|█         | 4434/41250 [10:43:30<88:18:16,  8.63s/it]
                                                          
{'loss': 0.1729, 'grad_norm': 1.4599367380142212, 'learning_rate': 3.9373598752591647e-05, 'epoch': 1.07}

 11%|█         | 4434/41250 [10:43:30<88:18:16,  8.63s/it][2025-04-25 18:41:13,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.58 | optimizer_gradients: 1.34 | optimizer_step: 0.90
[2025-04-25 18:41:13,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.79 | bwd_microstep: 5797.91 | bwd_inner_microstep: 5785.12 | bwd_allreduce_microstep: 12.74 | step_microstep: 21.10
[2025-04-25 18:41:13,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.79 | bwd: 5797.92 | bwd_inner: 5785.12 | bwd_allreduce: 12.76 | step: 21.11

 11%|█         | 4435/41250 [10:43:39<88:42:57,  8.68s/it]
                                                          
{'loss': 0.164, 'grad_norm': 2.6037352085113525, 'learning_rate': 3.937320876123594e-05, 'epoch': 1.08}

 11%|█         | 4435/41250 [10:43:39<88:42:57,  8.68s/it][2025-04-25 18:41:22,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:41:22,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.07 | bwd_microstep: 5717.66 | bwd_inner_microstep: 5689.35 | bwd_allreduce_microstep: 28.26 | step_microstep: 21.21
[2025-04-25 18:41:22,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.07 | bwd: 5717.67 | bwd_inner: 5689.35 | bwd_allreduce: 28.28 | step: 21.21

 11%|█         | 4436/41250 [10:43:47<88:38:13,  8.67s/it]
                                                          
{'loss': 0.2557, 'grad_norm': 2.169285535812378, 'learning_rate': 3.937281865044821e-05, 'epoch': 1.08}

 11%|█         | 4436/41250 [10:43:47<88:38:13,  8.67s/it][2025-04-25 18:41:31,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 18:41:31,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.58 | bwd_microstep: 5748.85 | bwd_inner_microstep: 5693.81 | bwd_allreduce_microstep: 54.99 | step_microstep: 19.04
[2025-04-25 18:41:31,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.58 | bwd: 5748.87 | bwd_inner: 5693.81 | bwd_allreduce: 55.02 | step: 19.04

 11%|█         | 4437/41250 [10:43:56<88:40:10,  8.67s/it]
                                                          
{'loss': 0.081, 'grad_norm': 0.5970475077629089, 'learning_rate': 3.937242842023085e-05, 'epoch': 1.08}

 11%|█         | 4437/41250 [10:43:56<88:40:10,  8.67s/it][2025-04-25 18:41:39,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 18:41:39,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.41 | bwd_microstep: 5689.60 | bwd_inner_microstep: 5640.21 | bwd_allreduce_microstep: 49.34 | step_microstep: 18.67
[2025-04-25 18:41:39,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.41 | bwd: 5689.61 | bwd_inner: 5640.21 | bwd_allreduce: 49.35 | step: 18.67

 11%|█         | 4438/41250 [10:44:05<88:26:12,  8.65s/it]
                                                          
{'loss': 0.316, 'grad_norm': 1.703989863395691, 'learning_rate': 3.937203807058629e-05, 'epoch': 1.08}

 11%|█         | 4438/41250 [10:44:05<88:26:12,  8.65s/it][2025-04-25 18:41:48,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 18:41:48,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5703.21 | bwd_inner_microstep: 5690.24 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.17
[2025-04-25 18:41:48,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5703.22 | bwd_inner: 5690.24 | bwd_allreduce: 12.94 | step: 19.17

 11%|█         | 4439/41250 [10:44:13<88:23:53,  8.65s/it]
                                                          
{'loss': 0.2568, 'grad_norm': 2.739767074584961, 'learning_rate': 3.937164760151692e-05, 'epoch': 1.08}

 11%|█         | 4439/41250 [10:44:13<88:23:53,  8.65s/it][2025-04-25 18:41:57,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:41:57,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5743.57 | bwd_inner_microstep: 5695.00 | bwd_allreduce_microstep: 48.53 | step_microstep: 18.78
[2025-04-25 18:41:57,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5743.59 | bwd_inner: 5695.00 | bwd_allreduce: 48.54 | step: 18.78

 11%|█         | 4440/41250 [10:44:22<88:28:59,  8.65s/it]
                                                          
{'loss': 0.0195, 'grad_norm': 0.3446187674999237, 'learning_rate': 3.937125701302516e-05, 'epoch': 1.08}

 11%|█         | 4440/41250 [10:44:22<88:28:59,  8.65s/it][2025-04-25 18:42:05,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:42:05,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.29 | bwd_microstep: 5762.01 | bwd_inner_microstep: 5656.52 | bwd_allreduce_microstep: 105.45 | step_microstep: 18.85
[2025-04-25 18:42:05,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.29 | bwd: 5762.02 | bwd_inner: 5656.52 | bwd_allreduce: 105.46 | step: 18.85

 11%|█         | 4441/41250 [10:44:31<88:32:37,  8.66s/it]
                                                          
{'loss': 0.0641, 'grad_norm': 0.949478268623352, 'learning_rate': 3.937086630511341e-05, 'epoch': 1.08}

 11%|█         | 4441/41250 [10:44:31<88:32:37,  8.66s/it][2025-04-25 18:42:14,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:42:14,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.68 | bwd_microstep: 5715.58 | bwd_inner_microstep: 5702.77 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.01
[2025-04-25 18:42:14,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.68 | bwd: 5715.59 | bwd_inner: 5702.77 | bwd_allreduce: 12.77 | step: 19.01

 11%|█         | 4442/41250 [10:44:39<88:31:05,  8.66s/it]
                                                          
{'loss': 0.0444, 'grad_norm': 0.6931912302970886, 'learning_rate': 3.9370475477784076e-05, 'epoch': 1.08}

 11%|█         | 4442/41250 [10:44:39<88:31:05,  8.66s/it][2025-04-25 18:42:23,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.26 | optimizer_step: 1.03
[2025-04-25 18:42:23,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.53 | bwd_microstep: 5752.77 | bwd_inner_microstep: 5682.25 | bwd_allreduce_microstep: 70.46 | step_microstep: 19.83
[2025-04-25 18:42:23,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.53 | bwd: 5752.78 | bwd_inner: 5682.25 | bwd_allreduce: 70.49 | step: 19.83

 11%|█         | 4443/41250 [10:44:48<88:36:25,  8.67s/it]
                                                          
{'loss': 0.0191, 'grad_norm': 0.33196255564689636, 'learning_rate': 3.937008453103957e-05, 'epoch': 1.08}

 11%|█         | 4443/41250 [10:44:48<88:36:25,  8.67s/it][2025-04-25 18:42:31,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:42:31,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5748.03 | bwd_inner_microstep: 5694.61 | bwd_allreduce_microstep: 53.38 | step_microstep: 18.52
[2025-04-25 18:42:31,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5748.05 | bwd_inner: 5694.61 | bwd_allreduce: 53.40 | step: 18.52

 11%|█         | 4444/41250 [10:44:57<88:39:24,  8.67s/it]
                                                          
{'loss': 0.1721, 'grad_norm': 1.4878524541854858, 'learning_rate': 3.93696934648823e-05, 'epoch': 1.08}

 11%|█         | 4444/41250 [10:44:57<88:39:24,  8.67s/it][2025-04-25 18:42:40,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 18:42:40,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.55 | bwd_microstep: 5718.77 | bwd_inner_microstep: 5659.62 | bwd_allreduce_microstep: 59.11 | step_microstep: 18.58
[2025-04-25 18:42:40,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.55 | bwd: 5718.79 | bwd_inner: 5659.62 | bwd_allreduce: 59.13 | step: 18.58

 11%|█         | 4445/41250 [10:45:05<88:31:56,  8.66s/it]
                                                          
{'loss': 0.115, 'grad_norm': 1.5637929439544678, 'learning_rate': 3.936930227931469e-05, 'epoch': 1.08}

 11%|█         | 4445/41250 [10:45:05<88:31:56,  8.66s/it][2025-04-25 18:42:49,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:42:49,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.28 | bwd_microstep: 5728.51 | bwd_inner_microstep: 5715.83 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.80
[2025-04-25 18:42:49,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.28 | bwd: 5728.53 | bwd_inner: 5715.83 | bwd_allreduce: 12.66 | step: 18.80

 11%|█         | 4446/41250 [10:45:14<88:32:51,  8.66s/it]
                                                          
{'loss': 0.0759, 'grad_norm': 0.9053823947906494, 'learning_rate': 3.936891097433913e-05, 'epoch': 1.08}

 11%|█         | 4446/41250 [10:45:14<88:32:51,  8.66s/it][2025-04-25 18:42:57,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:42:57,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.52 | bwd_microstep: 5748.00 | bwd_inner_microstep: 5703.77 | bwd_allreduce_microstep: 44.19 | step_microstep: 18.91
[2025-04-25 18:42:57,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.52 | bwd: 5748.02 | bwd_inner: 5703.77 | bwd_allreduce: 44.21 | step: 18.92

 11%|█         | 4447/41250 [10:45:23<88:36:55,  8.67s/it]
                                                          
{'loss': 0.0395, 'grad_norm': 0.5109105706214905, 'learning_rate': 3.936851954995805e-05, 'epoch': 1.08}

 11%|█         | 4447/41250 [10:45:23<88:36:55,  8.67s/it][2025-04-25 18:43:06,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.25 | optimizer_step: 1.01
[2025-04-25 18:43:06,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.71 | bwd_microstep: 5740.61 | bwd_inner_microstep: 5653.27 | bwd_allreduce_microstep: 87.28 | step_microstep: 19.50
[2025-04-25 18:43:06,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.72 | bwd: 5740.62 | bwd_inner: 5653.27 | bwd_allreduce: 87.31 | step: 19.51

 11%|█         | 4448/41250 [10:45:31<88:34:10,  8.66s/it]
                                                          
{'loss': 0.132, 'grad_norm': 2.068089723587036, 'learning_rate': 3.936812800617385e-05, 'epoch': 1.08}

 11%|█         | 4448/41250 [10:45:31<88:34:10,  8.66s/it][2025-04-25 18:43:15,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-25 18:43:15,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.50 | bwd_microstep: 5718.23 | bwd_inner_microstep: 5703.83 | bwd_allreduce_microstep: 14.33 | step_microstep: 19.30
[2025-04-25 18:43:15,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.50 | bwd: 5718.24 | bwd_inner: 5703.83 | bwd_allreduce: 14.36 | step: 19.30

 11%|█         | 4449/41250 [10:45:40<88:33:13,  8.66s/it]
                                                          
{'loss': 0.171, 'grad_norm': 1.8694559335708618, 'learning_rate': 3.936773634298896e-05, 'epoch': 1.08}

 11%|█         | 4449/41250 [10:45:40<88:33:13,  8.66s/it][2025-04-25 18:43:23,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 18:43:23,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.48 | bwd_microstep: 5801.44 | bwd_inner_microstep: 5666.21 | bwd_allreduce_microstep: 135.18 | step_microstep: 18.59
[2025-04-25 18:43:23,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.48 | bwd: 5801.45 | bwd_inner: 5666.21 | bwd_allreduce: 135.20 | step: 18.59

 11%|█         | 4450/41250 [10:45:49<88:42:00,  8.68s/it]
                                                          
{'loss': 0.0372, 'grad_norm': 0.7233862280845642, 'learning_rate': 3.936734456040578e-05, 'epoch': 1.08}

 11%|█         | 4450/41250 [10:45:49<88:42:00,  8.68s/it][2025-04-25 18:43:32,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.82 | optimizer_gradients: 1.25 | optimizer_step: 1.03
[2025-04-25 18:43:32,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.62 | bwd_microstep: 5707.46 | bwd_inner_microstep: 5694.12 | bwd_allreduce_microstep: 13.27 | step_microstep: 20.04
[2025-04-25 18:43:32,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.62 | bwd: 5707.48 | bwd_inner: 5694.12 | bwd_allreduce: 13.30 | step: 20.04

 11%|█         | 4451/41250 [10:45:57<88:37:41,  8.67s/it]
                                                          
{'loss': 0.0897, 'grad_norm': 2.108818292617798, 'learning_rate': 3.936695265842673e-05, 'epoch': 1.08}

 11%|█         | 4451/41250 [10:45:57<88:37:41,  8.67s/it][2025-04-25 18:43:41,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 1.09
[2025-04-25 18:43:41,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.36 | bwd_microstep: 5710.09 | bwd_inner_microstep: 5696.84 | bwd_allreduce_microstep: 13.20 | step_microstep: 18.91
[2025-04-25 18:43:41,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.36 | bwd: 5710.10 | bwd_inner: 5696.84 | bwd_allreduce: 13.22 | step: 18.92

 11%|█         | 4452/41250 [10:46:06<88:32:47,  8.66s/it]
                                                          
{'loss': 0.3424, 'grad_norm': 5.507380485534668, 'learning_rate': 3.9366560637054226e-05, 'epoch': 1.08}

 11%|█         | 4452/41250 [10:46:06<88:32:47,  8.66s/it][2025-04-25 18:43:49,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 18:43:49,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.63 | bwd_microstep: 5756.50 | bwd_inner_microstep: 5717.76 | bwd_allreduce_microstep: 38.70 | step_microstep: 18.65
[2025-04-25 18:43:49,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.63 | bwd: 5756.52 | bwd_inner: 5717.76 | bwd_allreduce: 38.72 | step: 18.66

 11%|█         | 4453/41250 [10:46:15<88:38:59,  8.67s/it]
                                                          
{'loss': 0.1579, 'grad_norm': 1.580683708190918, 'learning_rate': 3.936616849629069e-05, 'epoch': 1.08}

 11%|█         | 4453/41250 [10:46:15<88:38:59,  8.67s/it][2025-04-25 18:43:58,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:43:58,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.74 | bwd_microstep: 5805.99 | bwd_inner_microstep: 5657.93 | bwd_allreduce_microstep: 148.01 | step_microstep: 18.83
[2025-04-25 18:43:58,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.74 | bwd: 5806.00 | bwd_inner: 5657.93 | bwd_allreduce: 148.02 | step: 18.84

 11%|█         | 4454/41250 [10:46:23<88:50:04,  8.69s/it]
                                                          
{'loss': 0.0792, 'grad_norm': 1.2625483274459839, 'learning_rate': 3.9365776236138524e-05, 'epoch': 1.08}

 11%|█         | 4454/41250 [10:46:23<88:50:04,  8.69s/it][2025-04-25 18:44:07,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 18:44:07,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.67 | bwd_microstep: 5788.52 | bwd_inner_microstep: 5661.93 | bwd_allreduce_microstep: 126.53 | step_microstep: 18.84
[2025-04-25 18:44:07,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.67 | bwd: 5788.53 | bwd_inner: 5661.93 | bwd_allreduce: 126.55 | step: 18.84

 11%|█         | 4455/41250 [10:46:32<88:52:17,  8.70s/it]
                                                          
{'loss': 0.0386, 'grad_norm': 0.5451412200927734, 'learning_rate': 3.936538385660016e-05, 'epoch': 1.08}

 11%|█         | 4455/41250 [10:46:32<88:52:17,  8.70s/it][2025-04-25 18:44:16,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.19 | optimizer_step: 1.02
[2025-04-25 18:44:16,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.38 | bwd_microstep: 5899.00 | bwd_inner_microstep: 5715.71 | bwd_allreduce_microstep: 183.24 | step_microstep: 19.48
[2025-04-25 18:44:16,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.38 | bwd: 5899.01 | bwd_inner: 5715.71 | bwd_allreduce: 183.26 | step: 19.48

 11%|█         | 4456/41250 [10:46:41<89:18:54,  8.74s/it]
                                                          
{'loss': 0.0532, 'grad_norm': 0.4197375476360321, 'learning_rate': 3.9364991357678014e-05, 'epoch': 1.08}

 11%|█         | 4456/41250 [10:46:41<89:18:54,  8.74s/it][2025-04-25 18:44:24,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 18:44:24,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.58 | bwd_microstep: 5715.71 | bwd_inner_microstep: 5670.93 | bwd_allreduce_microstep: 44.72 | step_microstep: 19.00
[2025-04-25 18:44:24,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.58 | bwd: 5715.72 | bwd_inner: 5670.93 | bwd_allreduce: 44.74 | step: 19.00

 11%|█         | 4457/41250 [10:46:50<89:00:34,  8.71s/it]
                                                          
{'loss': 0.0285, 'grad_norm': 0.5160013437271118, 'learning_rate': 3.936459873937451e-05, 'epoch': 1.08}

 11%|█         | 4457/41250 [10:46:50<89:00:34,  8.71s/it][2025-04-25 18:44:33,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.12 | optimizer_step: 0.97
[2025-04-25 18:44:33,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.74 | bwd_microstep: 5770.21 | bwd_inner_microstep: 5666.51 | bwd_allreduce_microstep: 103.65 | step_microstep: 19.11
[2025-04-25 18:44:33,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.74 | bwd: 5770.23 | bwd_inner: 5666.51 | bwd_allreduce: 103.67 | step: 19.11

 11%|█         | 4458/41250 [10:46:58<88:57:52,  8.70s/it]
                                                          
{'loss': 0.0929, 'grad_norm': 1.7485343217849731, 'learning_rate': 3.9364206001692055e-05, 'epoch': 1.08}

 11%|█         | 4458/41250 [10:46:58<88:57:52,  8.70s/it][2025-04-25 18:44:42,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 18:44:42,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.76 | bwd_microstep: 5783.23 | bwd_inner_microstep: 5661.81 | bwd_allreduce_microstep: 121.36 | step_microstep: 18.88
[2025-04-25 18:44:42,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.76 | bwd: 5783.24 | bwd_inner: 5661.81 | bwd_allreduce: 121.38 | step: 18.88

 11%|█         | 4459/41250 [10:47:07<88:57:01,  8.70s/it]
                                                          
{'loss': 0.1865, 'grad_norm': 1.6928972005844116, 'learning_rate': 3.936381314463309e-05, 'epoch': 1.08}

 11%|█         | 4459/41250 [10:47:07<88:57:01,  8.70s/it][2025-04-25 18:44:50,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:44:50,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.47 | bwd_microstep: 5772.10 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 69.64 | step_microstep: 19.00
[2025-04-25 18:44:50,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.47 | bwd: 5772.11 | bwd_inner: 5702.41 | bwd_allreduce: 69.66 | step: 19.00

 11%|█         | 4460/41250 [10:47:16<88:59:55,  8.71s/it]
                                                          
{'loss': 0.3383, 'grad_norm': 1.6606882810592651, 'learning_rate': 3.936342016820001e-05, 'epoch': 1.08}

 11%|█         | 4460/41250 [10:47:16<88:59:55,  8.71s/it][2025-04-25 18:44:59,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 18:44:59,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.89 | bwd_microstep: 5703.83 | bwd_inner_microstep: 5663.06 | bwd_allreduce_microstep: 40.73 | step_microstep: 18.61
[2025-04-25 18:44:59,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.89 | bwd: 5703.84 | bwd_inner: 5663.06 | bwd_allreduce: 40.74 | step: 18.62

 11%|█         | 4461/41250 [10:47:24<88:42:42,  8.68s/it]
                                                          
{'loss': 0.083, 'grad_norm': 1.0882588624954224, 'learning_rate': 3.936302707239527e-05, 'epoch': 1.08}

 11%|█         | 4461/41250 [10:47:24<88:42:42,  8.68s/it][2025-04-25 18:45:08,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 18:45:08,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5740.58 | bwd_inner_microstep: 5704.75 | bwd_allreduce_microstep: 35.79 | step_microstep: 18.86
[2025-04-25 18:45:08,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5740.60 | bwd_inner: 5704.75 | bwd_allreduce: 35.81 | step: 18.86

 11%|█         | 4462/41250 [10:47:33<88:42:41,  8.68s/it]
                                                          
{'loss': 0.0763, 'grad_norm': 1.3595836162567139, 'learning_rate': 3.936263385722127e-05, 'epoch': 1.08}

 11%|█         | 4462/41250 [10:47:33<88:42:41,  8.68s/it][2025-04-25 18:45:16,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-25 18:45:16,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.87 | bwd_microstep: 5798.88 | bwd_inner_microstep: 5652.17 | bwd_allreduce_microstep: 146.65 | step_microstep: 19.48
[2025-04-25 18:45:16,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.87 | bwd: 5798.90 | bwd_inner: 5652.17 | bwd_allreduce: 146.68 | step: 19.49

 11%|█         | 4463/41250 [10:47:42<88:48:09,  8.69s/it]
                                                          
{'loss': 0.1718, 'grad_norm': 1.495214819908142, 'learning_rate': 3.936224052268044e-05, 'epoch': 1.08}

 11%|█         | 4463/41250 [10:47:42<88:48:09,  8.69s/it][2025-04-25 18:45:25,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 18:45:25,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.83 | bwd_microstep: 5764.10 | bwd_inner_microstep: 5690.99 | bwd_allreduce_microstep: 73.07 | step_microstep: 19.16
[2025-04-25 18:45:25,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.83 | bwd: 5764.12 | bwd_inner: 5690.99 | bwd_allreduce: 73.09 | step: 19.17

 11%|█         | 4464/41250 [10:47:50<88:51:03,  8.70s/it]
                                                          
{'loss': 0.1638, 'grad_norm': 2.1290366649627686, 'learning_rate': 3.93618470687752e-05, 'epoch': 1.08}

 11%|█         | 4464/41250 [10:47:50<88:51:03,  8.70s/it][2025-04-25 18:45:34,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:45:34,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.53 | bwd_microstep: 5704.99 | bwd_inner_microstep: 5692.04 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.79
[2025-04-25 18:45:34,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.53 | bwd: 5705.01 | bwd_inner: 5692.04 | bwd_allreduce: 12.92 | step: 18.79

 11%|█         | 4465/41250 [10:47:59<88:42:18,  8.68s/it]
                                                          
{'loss': 0.0521, 'grad_norm': 0.7306700348854065, 'learning_rate': 3.936145349550799e-05, 'epoch': 1.08}

 11%|█         | 4465/41250 [10:47:59<88:42:18,  8.68s/it][2025-04-25 18:45:42,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.17 | optimizer_step: 0.96
[2025-04-25 18:45:42,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.47 | bwd_microstep: 5756.77 | bwd_inner_microstep: 5658.85 | bwd_allreduce_microstep: 97.87 | step_microstep: 19.05
[2025-04-25 18:45:42,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.47 | bwd: 5756.78 | bwd_inner: 5658.85 | bwd_allreduce: 97.89 | step: 19.06

 11%|█         | 4466/41250 [10:48:08<88:41:47,  8.68s/it]
                                                          
{'loss': 0.0279, 'grad_norm': 0.8303221464157104, 'learning_rate': 3.9361059802881226e-05, 'epoch': 1.08}

 11%|█         | 4466/41250 [10:48:08<88:41:47,  8.68s/it][2025-04-25 18:45:51,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:45:51,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.13 | bwd_microstep: 5725.13 | bwd_inner_microstep: 5695.68 | bwd_allreduce_microstep: 29.40 | step_microstep: 18.46
[2025-04-25 18:45:51,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.13 | bwd: 5725.14 | bwd_inner: 5695.68 | bwd_allreduce: 29.42 | step: 18.46

 11%|█         | 4467/41250 [10:48:16<88:39:50,  8.68s/it]
                                                          
{'loss': 0.247, 'grad_norm': 2.485633611679077, 'learning_rate': 3.936066599089733e-05, 'epoch': 1.08}

 11%|█         | 4467/41250 [10:48:16<88:39:50,  8.68s/it][2025-04-25 18:46:00,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 18:46:00,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.84 | bwd_microstep: 5788.85 | bwd_inner_microstep: 5647.61 | bwd_allreduce_microstep: 141.19 | step_microstep: 19.18
[2025-04-25 18:46:00,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.84 | bwd: 5788.86 | bwd_inner: 5647.61 | bwd_allreduce: 141.21 | step: 19.18

 11%|█         | 4468/41250 [10:48:25<88:44:02,  8.68s/it]
                                                          
{'loss': 0.1751, 'grad_norm': 1.1373461484909058, 'learning_rate': 3.936027205955874e-05, 'epoch': 1.08}

 11%|█         | 4468/41250 [10:48:25<88:44:02,  8.68s/it][2025-04-25 18:46:08,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:46:08,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.70 | bwd_microstep: 5710.03 | bwd_inner_microstep: 5639.20 | bwd_allreduce_microstep: 70.77 | step_microstep: 18.43
[2025-04-25 18:46:08,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.70 | bwd: 5710.05 | bwd_inner: 5639.20 | bwd_allreduce: 70.80 | step: 18.44

 11%|█         | 4469/41250 [10:48:34<88:32:53,  8.67s/it]
                                                          
{'loss': 0.2761, 'grad_norm': 1.48638117313385, 'learning_rate': 3.935987800886788e-05, 'epoch': 1.08}

 11%|█         | 4469/41250 [10:48:34<88:32:53,  8.67s/it][2025-04-25 18:46:17,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-25 18:46:17,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.45 | bwd_microstep: 5755.28 | bwd_inner_microstep: 5651.84 | bwd_allreduce_microstep: 103.40 | step_microstep: 18.81
[2025-04-25 18:46:17,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.46 | bwd: 5755.29 | bwd_inner: 5651.83 | bwd_allreduce: 103.42 | step: 18.82

 11%|█         | 4470/41250 [10:48:42<88:33:42,  8.67s/it]
                                                          
{'loss': 0.1612, 'grad_norm': 1.872536063194275, 'learning_rate': 3.9359483838827185e-05, 'epoch': 1.08}

 11%|█         | 4470/41250 [10:48:42<88:33:42,  8.67s/it][2025-04-25 18:46:26,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 18:46:26,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.18 | bwd_microstep: 5710.10 | bwd_inner_microstep: 5697.35 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.67
[2025-04-25 18:46:26,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.18 | bwd: 5710.11 | bwd_inner: 5697.35 | bwd_allreduce: 12.72 | step: 18.68

 11%|█         | 4471/41250 [10:48:51<88:30:48,  8.66s/it]
                                                          
{'loss': 0.0624, 'grad_norm': 1.7096320390701294, 'learning_rate': 3.935908954943908e-05, 'epoch': 1.08}

 11%|█         | 4471/41250 [10:48:51<88:30:48,  8.66s/it][2025-04-25 18:46:34,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:46:34,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.19 | bwd_microstep: 5712.24 | bwd_inner_microstep: 5698.11 | bwd_allreduce_microstep: 14.09 | step_microstep: 18.68
[2025-04-25 18:46:34,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5712.25 | bwd_inner: 5698.11 | bwd_allreduce: 14.10 | step: 18.69

 11%|█         | 4472/41250 [10:49:00<88:28:29,  8.66s/it]
                                                          
{'loss': 0.2341, 'grad_norm': 2.134669542312622, 'learning_rate': 3.935869514070599e-05, 'epoch': 1.08}

 11%|█         | 4472/41250 [10:49:00<88:28:29,  8.66s/it][2025-04-25 18:46:43,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:46:43,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.56 | bwd_microstep: 5743.90 | bwd_inner_microstep: 5639.97 | bwd_allreduce_microstep: 103.88 | step_microstep: 18.40
[2025-04-25 18:46:43,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.56 | bwd: 5743.91 | bwd_inner: 5639.97 | bwd_allreduce: 103.90 | step: 18.40

 11%|█         | 4473/41250 [10:49:08<88:27:18,  8.66s/it]
                                                          
{'loss': 0.2038, 'grad_norm': 2.6601359844207764, 'learning_rate': 3.9358300612630354e-05, 'epoch': 1.08}

 11%|█         | 4473/41250 [10:49:08<88:27:18,  8.66s/it][2025-04-25 18:46:52,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.09 | optimizer_step: 1.04
[2025-04-25 18:46:52,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.53 | bwd_microstep: 5743.03 | bwd_inner_microstep: 5679.99 | bwd_allreduce_microstep: 62.98 | step_microstep: 19.78
[2025-04-25 18:46:52,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.53 | bwd: 5743.05 | bwd_inner: 5679.99 | bwd_allreduce: 63.01 | step: 19.78

 11%|█         | 4474/41250 [10:49:17<88:31:44,  8.67s/it]
                                                          
{'loss': 0.1967, 'grad_norm': 2.7388408184051514, 'learning_rate': 3.93579059652146e-05, 'epoch': 1.08}

 11%|█         | 4474/41250 [10:49:17<88:31:44,  8.67s/it][2025-04-25 18:47:00,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.23 | optimizer_step: 0.93
[2025-04-25 18:47:00,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.59 | bwd_microstep: 5732.55 | bwd_inner_microstep: 5697.02 | bwd_allreduce_microstep: 35.48 | step_microstep: 19.39
[2025-04-25 18:47:00,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.59 | bwd: 5732.57 | bwd_inner: 5697.02 | bwd_allreduce: 35.51 | step: 19.40

 11%|█         | 4475/41250 [10:49:26<88:32:28,  8.67s/it]
                                                          
{'loss': 0.0922, 'grad_norm': 1.4723496437072754, 'learning_rate': 3.935751119846117e-05, 'epoch': 1.08}

 11%|█         | 4475/41250 [10:49:26<88:32:28,  8.67s/it][2025-04-25 18:47:09,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 18:47:09,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.99 | bwd_microstep: 5743.52 | bwd_inner_microstep: 5683.85 | bwd_allreduce_microstep: 59.63 | step_microstep: 19.12
[2025-04-25 18:47:09,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.00 | bwd: 5743.54 | bwd_inner: 5683.85 | bwd_allreduce: 59.65 | step: 19.13

 11%|█         | 4476/41250 [10:49:34<88:34:10,  8.67s/it]
                                                          
{'loss': 0.1156, 'grad_norm': 1.481529712677002, 'learning_rate': 3.935711631237248e-05, 'epoch': 1.09}

 11%|█         | 4476/41250 [10:49:34<88:34:10,  8.67s/it][2025-04-25 18:47:18,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 18:47:18,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.36 | bwd_microstep: 5732.00 | bwd_inner_microstep: 5683.47 | bwd_allreduce_microstep: 48.48 | step_microstep: 19.09
[2025-04-25 18:47:18,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.36 | bwd: 5732.02 | bwd_inner: 5683.47 | bwd_allreduce: 48.50 | step: 19.09

 11%|█         | 4477/41250 [10:49:43<88:31:03,  8.67s/it]
                                                          
{'loss': 0.4719, 'grad_norm': 4.076874256134033, 'learning_rate': 3.9356721306950984e-05, 'epoch': 1.09}

 11%|█         | 4477/41250 [10:49:43<88:31:03,  8.67s/it][2025-04-25 18:47:26,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:47:26,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5682.84 | bwd_inner_microstep: 5645.49 | bwd_allreduce_microstep: 37.30 | step_microstep: 18.81
[2025-04-25 18:47:26,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5682.86 | bwd_inner: 5645.49 | bwd_allreduce: 37.32 | step: 18.80

 11%|█         | 4478/41250 [10:49:52<88:18:29,  8.65s/it]
                                                          
{'loss': 0.1309, 'grad_norm': 2.8889074325561523, 'learning_rate': 3.935632618219911e-05, 'epoch': 1.09}

 11%|█         | 4478/41250 [10:49:52<88:18:29,  8.65s/it][2025-04-25 18:47:35,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 18:47:35,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5701.79 | bwd_inner_microstep: 5652.76 | bwd_allreduce_microstep: 48.98 | step_microstep: 18.61
[2025-04-25 18:47:35,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5701.81 | bwd_inner: 5652.76 | bwd_allreduce: 49.00 | step: 18.61

 11%|█         | 4479/41250 [10:50:00<88:11:28,  8.63s/it]
                                                          
{'loss': 0.2098, 'grad_norm': 2.43686580657959, 'learning_rate': 3.935593093811928e-05, 'epoch': 1.09}

 11%|█         | 4479/41250 [10:50:00<88:11:28,  8.63s/it][2025-04-25 18:47:44,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 18:47:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.64 | bwd_microstep: 5745.45 | bwd_inner_microstep: 5644.68 | bwd_allreduce_microstep: 100.73 | step_microstep: 19.36
[2025-04-25 18:47:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.64 | bwd: 5745.47 | bwd_inner: 5644.68 | bwd_allreduce: 100.75 | step: 19.36

 11%|█         | 4480/41250 [10:50:09<88:15:17,  8.64s/it]
                                                          
{'loss': 0.2445, 'grad_norm': 2.05794095993042, 'learning_rate': 3.935553557471396e-05, 'epoch': 1.09}

 11%|█         | 4480/41250 [10:50:09<88:15:17,  8.64s/it][2025-04-25 18:47:52,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:47:52,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.47 | bwd_microstep: 5705.06 | bwd_inner_microstep: 5692.18 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.61
[2025-04-25 18:47:52,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.47 | bwd: 5705.07 | bwd_inner: 5692.18 | bwd_allreduce: 12.85 | step: 18.61

 11%|█         | 4481/41250 [10:50:18<88:13:45,  8.64s/it]
                                                          
{'loss': 0.204, 'grad_norm': 2.763901472091675, 'learning_rate': 3.935514009198556e-05, 'epoch': 1.09}

 11%|█         | 4481/41250 [10:50:18<88:13:45,  8.64s/it][2025-04-25 18:48:01,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:48:01,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3113.80 | bwd_microstep: 5697.63 | bwd_inner_microstep: 5684.86 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.87
[2025-04-25 18:48:01,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3113.80 | bwd: 5697.65 | bwd_inner: 5684.85 | bwd_allreduce: 12.75 | step: 18.87

 11%|█         | 4482/41250 [10:50:26<89:00:32,  8.71s/it]
                                                          
{'loss': 0.1184, 'grad_norm': 1.1089304685592651, 'learning_rate': 3.9354744489936526e-05, 'epoch': 1.09}

 11%|█         | 4482/41250 [10:50:26<89:00:32,  8.71s/it][2025-04-25 18:48:10,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:48:10,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.38 | bwd_microstep: 5680.12 | bwd_inner_microstep: 5632.91 | bwd_allreduce_microstep: 47.17 | step_microstep: 18.65
[2025-04-25 18:48:10,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.38 | bwd: 5680.14 | bwd_inner: 5632.91 | bwd_allreduce: 47.18 | step: 18.66

 11%|█         | 4483/41250 [10:50:35<88:35:22,  8.67s/it]
                                                          
{'loss': 0.0556, 'grad_norm': 0.6284872889518738, 'learning_rate': 3.9354348768569305e-05, 'epoch': 1.09}

 11%|█         | 4483/41250 [10:50:35<88:35:22,  8.67s/it][2025-04-25 18:48:18,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:48:18,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.99 | bwd_microstep: 5742.28 | bwd_inner_microstep: 5649.28 | bwd_allreduce_microstep: 92.95 | step_microstep: 18.74
[2025-04-25 18:48:18,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.99 | bwd: 5742.29 | bwd_inner: 5649.28 | bwd_allreduce: 92.97 | step: 18.74

 11%|█         | 4484/41250 [10:50:44<88:29:10,  8.66s/it]
                                                          
{'loss': 0.0851, 'grad_norm': 1.2230061292648315, 'learning_rate': 3.9353952927886326e-05, 'epoch': 1.09}

 11%|█         | 4484/41250 [10:50:44<88:29:10,  8.66s/it][2025-04-25 18:48:27,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:48:27,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.59 | bwd_microstep: 5749.53 | bwd_inner_microstep: 5647.89 | bwd_allreduce_microstep: 101.60 | step_microstep: 18.49
[2025-04-25 18:48:27,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.59 | bwd: 5749.54 | bwd_inner: 5647.89 | bwd_allreduce: 101.62 | step: 18.49

 11%|█         | 4485/41250 [10:50:52<88:27:41,  8.66s/it]
                                                          
{'loss': 0.0899, 'grad_norm': 2.1784865856170654, 'learning_rate': 3.9353556967890036e-05, 'epoch': 1.09}

 11%|█         | 4485/41250 [10:50:52<88:27:41,  8.66s/it][2025-04-25 18:48:36,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-25 18:48:36,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.89 | bwd_microstep: 5698.29 | bwd_inner_microstep: 5646.19 | bwd_allreduce_microstep: 52.05 | step_microstep: 19.48
[2025-04-25 18:48:36,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.89 | bwd: 5698.31 | bwd_inner: 5646.19 | bwd_allreduce: 52.08 | step: 19.48

 11%|█         | 4486/41250 [10:51:01<88:16:08,  8.64s/it]
                                                          
{'loss': 0.1985, 'grad_norm': 2.308450698852539, 'learning_rate': 3.935316088858287e-05, 'epoch': 1.09}

 11%|█         | 4486/41250 [10:51:01<88:16:08,  8.64s/it][2025-04-25 18:48:44,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 18:48:44,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5732.78 | bwd_inner_microstep: 5692.91 | bwd_allreduce_microstep: 39.83 | step_microstep: 18.90
[2025-04-25 18:48:44,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5732.79 | bwd_inner: 5692.90 | bwd_allreduce: 39.84 | step: 18.90

 11%|█         | 4487/41250 [10:51:10<88:20:48,  8.65s/it]
                                                          
{'loss': 0.2893, 'grad_norm': 2.4250080585479736, 'learning_rate': 3.935276468996728e-05, 'epoch': 1.09}

 11%|█         | 4487/41250 [10:51:10<88:20:48,  8.65s/it][2025-04-25 18:48:53,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 18:48:53,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.47 | bwd_microstep: 5684.53 | bwd_inner_microstep: 5651.91 | bwd_allreduce_microstep: 32.58 | step_microstep: 18.79
[2025-04-25 18:48:53,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.47 | bwd: 5684.55 | bwd_inner: 5651.91 | bwd_allreduce: 32.60 | step: 18.79

 11%|█         | 4488/41250 [10:51:18<88:10:51,  8.64s/it]
                                                          
{'loss': 0.4054, 'grad_norm': 2.6328678131103516, 'learning_rate': 3.935236837204571e-05, 'epoch': 1.09}

 11%|█         | 4488/41250 [10:51:18<88:10:51,  8.64s/it][2025-04-25 18:49:01,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:49:01,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5717.11 | bwd_inner_microstep: 5704.39 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.92
[2025-04-25 18:49:01,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5717.13 | bwd_inner: 5704.39 | bwd_allreduce: 12.69 | step: 18.92

 11%|█         | 4489/41250 [10:51:27<88:13:48,  8.64s/it]
                                                          
{'loss': 0.2347, 'grad_norm': 3.02933406829834, 'learning_rate': 3.935197193482058e-05, 'epoch': 1.09}

 11%|█         | 4489/41250 [10:51:27<88:13:48,  8.64s/it][2025-04-25 18:49:10,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 18:49:10,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.41 | bwd_microstep: 5716.48 | bwd_inner_microstep: 5703.43 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.20
[2025-04-25 18:49:10,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.41 | bwd: 5716.49 | bwd_inner: 5703.43 | bwd_allreduce: 13.02 | step: 19.20

 11%|█         | 4490/41250 [10:51:35<88:16:03,  8.64s/it]
                                                          
{'loss': 0.1738, 'grad_norm': 1.503122329711914, 'learning_rate': 3.9351575378294356e-05, 'epoch': 1.09}

 11%|█         | 4490/41250 [10:51:35<88:16:03,  8.64s/it][2025-04-25 18:49:19,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 18:49:19,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5773.24 | bwd_inner_microstep: 5655.35 | bwd_allreduce_microstep: 117.84 | step_microstep: 19.08
[2025-04-25 18:49:19,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5773.25 | bwd_inner: 5655.35 | bwd_allreduce: 117.86 | step: 19.08

 11%|█         | 4491/41250 [10:51:44<88:23:20,  8.66s/it]
                                                          
{'loss': 0.2459, 'grad_norm': 1.5094022750854492, 'learning_rate': 3.935117870246947e-05, 'epoch': 1.09}

 11%|█         | 4491/41250 [10:51:44<88:23:20,  8.66s/it][2025-04-25 18:49:27,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 18:49:27,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.96 | bwd_microstep: 5710.35 | bwd_inner_microstep: 5697.78 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.62
[2025-04-25 18:49:27,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.96 | bwd: 5710.37 | bwd_inner: 5697.78 | bwd_allreduce: 12.55 | step: 18.63

 11%|█         | 4492/41250 [10:51:53<88:21:35,  8.65s/it]
                                                          
{'loss': 0.0761, 'grad_norm': 1.1473582983016968, 'learning_rate': 3.935078190734838e-05, 'epoch': 1.09}

 11%|█         | 4492/41250 [10:51:53<88:21:35,  8.65s/it][2025-04-25 18:49:36,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.02 | optimizer_step: 1.24
[2025-04-25 18:49:36,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.70 | bwd_microstep: 5741.56 | bwd_inner_microstep: 5702.15 | bwd_allreduce_microstep: 39.35 | step_microstep: 19.27
[2025-04-25 18:49:36,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.70 | bwd: 5741.57 | bwd_inner: 5702.15 | bwd_allreduce: 39.38 | step: 19.28

 11%|█         | 4493/41250 [10:52:01<88:25:47,  8.66s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 0.8658097386360168, 'learning_rate': 3.9350384992933523e-05, 'epoch': 1.09}

 11%|█         | 4493/41250 [10:52:01<88:25:47,  8.66s/it][2025-04-25 18:49:45,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 18:49:45,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.97 | bwd_microstep: 5705.23 | bwd_inner_microstep: 5692.42 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.63
[2025-04-25 18:49:45,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.97 | bwd: 5705.24 | bwd_inner: 5692.42 | bwd_allreduce: 12.78 | step: 18.63

 11%|█         | 4494/41250 [10:52:10<88:20:20,  8.65s/it]
                                                          
{'loss': 0.1634, 'grad_norm': 3.0577893257141113, 'learning_rate': 3.934998795922735e-05, 'epoch': 1.09}

 11%|█         | 4494/41250 [10:52:10<88:20:20,  8.65s/it][2025-04-25 18:49:53,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:49:53,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.05 | bwd_microstep: 5694.63 | bwd_inner_microstep: 5654.02 | bwd_allreduce_microstep: 40.56 | step_microstep: 18.58
[2025-04-25 18:49:53,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.05 | bwd: 5694.64 | bwd_inner: 5654.02 | bwd_allreduce: 40.58 | step: 18.59

 11%|█         | 4495/41250 [10:52:19<88:12:35,  8.64s/it]
                                                          
{'loss': 0.0591, 'grad_norm': 0.6008265614509583, 'learning_rate': 3.9349590806232305e-05, 'epoch': 1.09}

 11%|█         | 4495/41250 [10:52:19<88:12:35,  8.64s/it][2025-04-25 18:50:02,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 18:50:02,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.91 | bwd_microstep: 5717.15 | bwd_inner_microstep: 5661.16 | bwd_allreduce_microstep: 55.95 | step_microstep: 19.12
[2025-04-25 18:50:02,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.91 | bwd: 5717.17 | bwd_inner: 5661.16 | bwd_allreduce: 55.96 | step: 19.12

 11%|█         | 4496/41250 [10:52:27<88:10:46,  8.64s/it]
                                                          
{'loss': 0.2158, 'grad_norm': 3.804691791534424, 'learning_rate': 3.934919353395084e-05, 'epoch': 1.09}

 11%|█         | 4496/41250 [10:52:27<88:10:46,  8.64s/it][2025-04-25 18:50:11,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 18:50:11,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.25 | bwd_microstep: 5765.60 | bwd_inner_microstep: 5697.94 | bwd_allreduce_microstep: 67.61 | step_microstep: 19.07
[2025-04-25 18:50:11,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.25 | bwd: 5765.62 | bwd_inner: 5697.94 | bwd_allreduce: 67.63 | step: 19.07

 11%|█         | 4497/41250 [10:52:36<88:23:10,  8.66s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 1.613997459411621, 'learning_rate': 3.9348796142385406e-05, 'epoch': 1.09}

 11%|█         | 4497/41250 [10:52:36<88:23:10,  8.66s/it][2025-04-25 18:50:19,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:50:19,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.03 | bwd_microstep: 5788.79 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 135.25 | step_microstep: 18.60
[2025-04-25 18:50:19,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.03 | bwd: 5788.80 | bwd_inner: 5653.50 | bwd_allreduce: 135.26 | step: 18.60

 11%|█         | 4498/41250 [10:52:45<88:31:11,  8.67s/it]
                                                          
{'loss': 0.1865, 'grad_norm': 1.805816650390625, 'learning_rate': 3.934839863153845e-05, 'epoch': 1.09}

 11%|█         | 4498/41250 [10:52:45<88:31:11,  8.67s/it][2025-04-25 18:50:28,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 18:50:28,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5708.04 | bwd_inner_microstep: 5695.34 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.77
[2025-04-25 18:50:28,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5708.05 | bwd_inner: 5695.34 | bwd_allreduce: 12.67 | step: 18.77

 11%|█         | 4499/41250 [10:52:54<89:01:49,  8.72s/it]
                                                          
{'loss': 0.1771, 'grad_norm': 1.8707342147827148, 'learning_rate': 3.934800100141242e-05, 'epoch': 1.09}

 11%|█         | 4499/41250 [10:52:54<89:01:49,  8.72s/it][2025-04-25 18:50:37,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:50:37,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.16 | bwd_microstep: 5725.76 | bwd_inner_microstep: 5672.40 | bwd_allreduce_microstep: 53.31 | step_microstep: 18.79
[2025-04-25 18:50:37,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.16 | bwd: 5725.77 | bwd_inner: 5672.40 | bwd_allreduce: 53.33 | step: 18.79

 11%|█         | 4500/41250 [10:53:02<88:48:27,  8.70s/it]
                                                          
{'loss': 0.1694, 'grad_norm': 1.9052107334136963, 'learning_rate': 3.934760325200977e-05, 'epoch': 1.09}

 11%|█         | 4500/41250 [10:53:02<88:48:27,  8.70s/it][2025-04-25 18:50:46,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 18:50:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.48 | bwd_microstep: 5717.78 | bwd_inner_microstep: 5704.98 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.96
[2025-04-25 18:50:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.48 | bwd: 5717.79 | bwd_inner: 5704.98 | bwd_allreduce: 12.77 | step: 18.97

 11%|█         | 4501/41250 [10:53:11<88:40:12,  8.69s/it]
                                                          
{'loss': 0.2118, 'grad_norm': 1.5735963582992554, 'learning_rate': 3.934720538333294e-05, 'epoch': 1.09}

 11%|█         | 4501/41250 [10:53:11<88:40:12,  8.69s/it][2025-04-25 18:50:54,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 18:50:54,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.57 | bwd_microstep: 5811.51 | bwd_inner_microstep: 5665.51 | bwd_allreduce_microstep: 145.95 | step_microstep: 18.79
[2025-04-25 18:50:54,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.57 | bwd: 5811.52 | bwd_inner: 5665.51 | bwd_allreduce: 145.97 | step: 18.79

 11%|█         | 4502/41250 [10:53:20<88:48:19,  8.70s/it]
                                                          
{'loss': 0.2224, 'grad_norm': 1.467522144317627, 'learning_rate': 3.934680739538441e-05, 'epoch': 1.09}

 11%|█         | 4502/41250 [10:53:20<88:48:19,  8.70s/it][2025-04-25 18:51:03,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 18:51:03,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.92 | bwd_microstep: 5744.68 | bwd_inner_microstep: 5661.33 | bwd_allreduce_microstep: 83.30 | step_microstep: 18.73
[2025-04-25 18:51:03,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.92 | bwd: 5744.69 | bwd_inner: 5661.33 | bwd_allreduce: 83.32 | step: 18.73

 11%|█         | 4503/41250 [10:53:28<88:40:42,  8.69s/it]
                                                          
{'loss': 0.2918, 'grad_norm': 1.7469674348831177, 'learning_rate': 3.9346409288166603e-05, 'epoch': 1.09}

 11%|█         | 4503/41250 [10:53:28<88:40:42,  8.69s/it][2025-04-25 18:51:12,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:51:12,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.10 | bwd_microstep: 5785.54 | bwd_inner_microstep: 5693.21 | bwd_allreduce_microstep: 92.28 | step_microstep: 18.53
[2025-04-25 18:51:12,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.10 | bwd: 5785.55 | bwd_inner: 5693.21 | bwd_allreduce: 92.30 | step: 18.53

 11%|█         | 4504/41250 [10:53:37<88:45:29,  8.70s/it]
                                                          
{'loss': 0.1021, 'grad_norm': 1.6817435026168823, 'learning_rate': 3.9346011061682e-05, 'epoch': 1.09}

 11%|█         | 4504/41250 [10:53:37<88:45:29,  8.70s/it][2025-04-25 18:51:20,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.86 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:51:20,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.78 | bwd_microstep: 5792.68 | bwd_inner_microstep: 5645.98 | bwd_allreduce_microstep: 146.65 | step_microstep: 19.40
[2025-04-25 18:51:20,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.78 | bwd: 5792.69 | bwd_inner: 5645.98 | bwd_allreduce: 146.67 | step: 19.41

 11%|█         | 4505/41250 [10:53:46<88:47:13,  8.70s/it]
                                                          
{'loss': 0.2349, 'grad_norm': 1.7538763284683228, 'learning_rate': 3.934561271593304e-05, 'epoch': 1.09}

 11%|█         | 4505/41250 [10:53:46<88:47:13,  8.70s/it][2025-04-25 18:51:29,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 18:51:29,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5727.23 | bwd_inner_microstep: 5653.32 | bwd_allreduce_microstep: 73.86 | step_microstep: 19.17
[2025-04-25 18:51:29,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5727.25 | bwd_inner: 5653.32 | bwd_allreduce: 73.89 | step: 19.17

 11%|█         | 4506/41250 [10:53:54<88:36:09,  8.68s/it]
                                                          
{'loss': 0.114, 'grad_norm': 1.1388229131698608, 'learning_rate': 3.9345214250922176e-05, 'epoch': 1.09}

 11%|█         | 4506/41250 [10:53:54<88:36:09,  8.68s/it][2025-04-25 18:51:38,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:51:38,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.68 | bwd_microstep: 5783.56 | bwd_inner_microstep: 5719.62 | bwd_allreduce_microstep: 63.90 | step_microstep: 18.61
[2025-04-25 18:51:38,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.68 | bwd: 5783.58 | bwd_inner: 5719.62 | bwd_allreduce: 63.92 | step: 18.61

 11%|█         | 4507/41250 [10:54:03<88:43:23,  8.69s/it]
                                                          
{'loss': 0.1658, 'grad_norm': 1.545922875404358, 'learning_rate': 3.9344815666651885e-05, 'epoch': 1.09}

 11%|█         | 4507/41250 [10:54:03<88:43:23,  8.69s/it][2025-04-25 18:51:46,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:51:46,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.90 | bwd_microstep: 5717.67 | bwd_inner_microstep: 5680.66 | bwd_allreduce_microstep: 36.96 | step_microstep: 18.74
[2025-04-25 18:51:46,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.90 | bwd: 5717.68 | bwd_inner: 5680.66 | bwd_allreduce: 36.98 | step: 18.75

 11%|█         | 4508/41250 [10:54:12<88:34:52,  8.68s/it]
                                                          
{'loss': 0.0643, 'grad_norm': 0.8357555270195007, 'learning_rate': 3.93444169631246e-05, 'epoch': 1.09}

 11%|█         | 4508/41250 [10:54:12<88:34:52,  8.68s/it][2025-04-25 18:51:55,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:51:55,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.74 | bwd_microstep: 5796.00 | bwd_inner_microstep: 5659.96 | bwd_allreduce_microstep: 135.99 | step_microstep: 18.73
[2025-04-25 18:51:55,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.74 | bwd: 5796.01 | bwd_inner: 5659.96 | bwd_allreduce: 136.01 | step: 18.73

 11%|█         | 4509/41250 [10:54:20<88:39:54,  8.69s/it]
                                                          
{'loss': 0.1487, 'grad_norm': 1.9823205471038818, 'learning_rate': 3.93440181403428e-05, 'epoch': 1.09}

 11%|█         | 4509/41250 [10:54:20<88:39:54,  8.69s/it][2025-04-25 18:52:04,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:52:04,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.69 | bwd_microstep: 5707.46 | bwd_inner_microstep: 5694.67 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.16
[2025-04-25 18:52:04,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.69 | bwd: 5707.47 | bwd_inner: 5694.67 | bwd_allreduce: 12.77 | step: 18.17

 11%|█         | 4510/41250 [10:54:29<88:31:36,  8.67s/it]
                                                          
{'loss': 0.1022, 'grad_norm': 1.128544569015503, 'learning_rate': 3.934361919830892e-05, 'epoch': 1.09}

 11%|█         | 4510/41250 [10:54:29<88:31:36,  8.67s/it][2025-04-25 18:52:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 18:52:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5805.60 | bwd_inner_microstep: 5666.83 | bwd_allreduce_microstep: 138.72 | step_microstep: 18.60
[2025-04-25 18:52:12,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5805.61 | bwd_inner: 5666.83 | bwd_allreduce: 138.74 | step: 18.60

 11%|█         | 4511/41250 [10:54:38<88:39:08,  8.69s/it]
                                                          
{'loss': 0.0558, 'grad_norm': 0.6568917632102966, 'learning_rate': 3.9343220137025436e-05, 'epoch': 1.09}

 11%|█         | 4511/41250 [10:54:38<88:39:08,  8.69s/it][2025-04-25 18:52:21,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.62 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:52:21,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5708.79 | bwd_inner_microstep: 5694.84 | bwd_allreduce_microstep: 13.91 | step_microstep: 18.68
[2025-04-25 18:52:21,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5708.81 | bwd_inner: 5694.84 | bwd_allreduce: 13.92 | step: 18.69

 11%|█         | 4512/41250 [10:54:46<88:29:57,  8.67s/it]
                                                          
{'loss': 0.2255, 'grad_norm': 1.5887351036071777, 'learning_rate': 3.9342820956494805e-05, 'epoch': 1.09}

 11%|█         | 4512/41250 [10:54:46<88:29:57,  8.67s/it][2025-04-25 18:52:30,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 1.09
[2025-04-25 18:52:30,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.30 | bwd_microstep: 5786.09 | bwd_inner_microstep: 5704.56 | bwd_allreduce_microstep: 81.49 | step_microstep: 18.68
[2025-04-25 18:52:30,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.30 | bwd: 5786.10 | bwd_inner: 5704.56 | bwd_allreduce: 81.51 | step: 18.68

 11%|█         | 4513/41250 [10:54:55<88:38:09,  8.69s/it]
                                                          
{'loss': 0.1601, 'grad_norm': 3.4159319400787354, 'learning_rate': 3.9342421656719486e-05, 'epoch': 1.09}

 11%|█         | 4513/41250 [10:54:55<88:38:09,  8.69s/it][2025-04-25 18:52:39,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-25 18:52:39,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5803.62 | bwd_inner_microstep: 5659.74 | bwd_allreduce_microstep: 143.82 | step_microstep: 18.58
[2025-04-25 18:52:39,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5803.63 | bwd_inner: 5659.74 | bwd_allreduce: 143.84 | step: 18.58

 11%|█         | 4514/41250 [10:55:04<88:43:05,  8.69s/it]
                                                          
{'loss': 0.2819, 'grad_norm': 2.117114543914795, 'learning_rate': 3.9342022237701945e-05, 'epoch': 1.09}

 11%|█         | 4514/41250 [10:55:04<88:43:05,  8.69s/it][2025-04-25 18:52:47,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.04 | optimizer_step: 1.15
[2025-04-25 18:52:47,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.35 | bwd_microstep: 5794.01 | bwd_inner_microstep: 5660.28 | bwd_allreduce_microstep: 133.68 | step_microstep: 19.15
[2025-04-25 18:52:47,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.35 | bwd: 5794.02 | bwd_inner: 5660.28 | bwd_allreduce: 133.70 | step: 19.15

 11%|█         | 4515/41250 [10:55:13<88:44:17,  8.70s/it]
                                                          
{'loss': 0.1234, 'grad_norm': 1.4453462362289429, 'learning_rate': 3.934162269944464e-05, 'epoch': 1.09}

 11%|█         | 4515/41250 [10:55:13<88:44:17,  8.70s/it][2025-04-25 18:52:56,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 18:52:56,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.72 | bwd_microstep: 5809.43 | bwd_inner_microstep: 5644.27 | bwd_allreduce_microstep: 165.12 | step_microstep: 18.69
[2025-04-25 18:52:56,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.72 | bwd: 5809.44 | bwd_inner: 5644.27 | bwd_allreduce: 165.13 | step: 18.69

 11%|█         | 4516/41250 [10:55:21<88:47:23,  8.70s/it]
                                                          
{'loss': 0.2478, 'grad_norm': 1.698708176612854, 'learning_rate': 3.9341223041950044e-05, 'epoch': 1.09}

 11%|█         | 4516/41250 [10:55:21<88:47:23,  8.70s/it][2025-04-25 18:53:05,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 18:53:05,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.29 | bwd_microstep: 5702.72 | bwd_inner_microstep: 5641.12 | bwd_allreduce_microstep: 61.54 | step_microstep: 18.86
[2025-04-25 18:53:05,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.29 | bwd: 5702.73 | bwd_inner: 5641.12 | bwd_allreduce: 61.56 | step: 18.86

 11%|█         | 4517/41250 [10:55:30<88:29:07,  8.67s/it]
                                                          
{'loss': 0.0554, 'grad_norm': 0.6598738431930542, 'learning_rate': 3.93408232652206e-05, 'epoch': 1.1}

 11%|█         | 4517/41250 [10:55:30<88:29:07,  8.67s/it][2025-04-25 18:53:13,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:53:13,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5770.25 | bwd_inner_microstep: 5676.33 | bwd_allreduce_microstep: 93.87 | step_microstep: 18.73
[2025-04-25 18:53:13,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5770.26 | bwd_inner: 5676.33 | bwd_allreduce: 93.89 | step: 18.74

 11%|█         | 4518/41250 [10:55:39<88:34:07,  8.68s/it]
                                                          
{'loss': 0.3153, 'grad_norm': 2.1002492904663086, 'learning_rate': 3.934042336925879e-05, 'epoch': 1.1}

 11%|█         | 4518/41250 [10:55:39<88:34:07,  8.68s/it][2025-04-25 18:53:22,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:53:22,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5732.97 | bwd_inner_microstep: 5707.64 | bwd_allreduce_microstep: 25.28 | step_microstep: 19.35
[2025-04-25 18:53:22,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5732.98 | bwd_inner: 5707.64 | bwd_allreduce: 25.30 | step: 19.35

 11%|█         | 4519/41250 [10:55:47<88:31:09,  8.68s/it]
                                                          
{'loss': 0.0862, 'grad_norm': 1.5788171291351318, 'learning_rate': 3.934002335406707e-05, 'epoch': 1.1}

 11%|█         | 4519/41250 [10:55:47<88:31:09,  8.68s/it][2025-04-25 18:53:31,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 18:53:31,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.45 | bwd_microstep: 5756.19 | bwd_inner_microstep: 5654.21 | bwd_allreduce_microstep: 101.93 | step_microstep: 18.75
[2025-04-25 18:53:31,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.45 | bwd: 5756.21 | bwd_inner: 5654.21 | bwd_allreduce: 101.95 | step: 18.75

 11%|█         | 4520/41250 [10:55:56<88:29:09,  8.67s/it]
                                                          
{'loss': 0.0859, 'grad_norm': 1.1612995862960815, 'learning_rate': 3.9339623219647917e-05, 'epoch': 1.1}

 11%|█         | 4520/41250 [10:55:56<88:29:09,  8.67s/it][2025-04-25 18:53:39,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 18:53:39,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.15 | bwd_microstep: 5779.55 | bwd_inner_microstep: 5654.90 | bwd_allreduce_microstep: 124.61 | step_microstep: 18.63
[2025-04-25 18:53:39,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.15 | bwd: 5779.57 | bwd_inner: 5654.90 | bwd_allreduce: 124.63 | step: 18.63

 11%|█         | 4521/41250 [10:56:05<88:32:48,  8.68s/it]
                                                          
{'loss': 0.2228, 'grad_norm': 1.6770527362823486, 'learning_rate': 3.9339222966003784e-05, 'epoch': 1.1}

 11%|█         | 4521/41250 [10:56:05<88:32:48,  8.68s/it][2025-04-25 18:53:48,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:53:48,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.80 | bwd_microstep: 5721.26 | bwd_inner_microstep: 5682.38 | bwd_allreduce_microstep: 38.82 | step_microstep: 18.80
[2025-04-25 18:53:48,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.80 | bwd: 5721.27 | bwd_inner: 5682.38 | bwd_allreduce: 38.85 | step: 18.80

 11%|█         | 4522/41250 [10:56:13<88:27:39,  8.67s/it]
                                                          
{'loss': 0.2356, 'grad_norm': 1.0870553255081177, 'learning_rate': 3.933882259313714e-05, 'epoch': 1.1}

 11%|█         | 4522/41250 [10:56:13<88:27:39,  8.67s/it][2025-04-25 18:53:57,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 18:53:57,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.62 | bwd_microstep: 5704.87 | bwd_inner_microstep: 5692.18 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.92
[2025-04-25 18:53:57,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.62 | bwd: 5704.89 | bwd_inner: 5692.18 | bwd_allreduce: 12.66 | step: 18.92

 11%|█         | 4523/41250 [10:56:22<88:22:32,  8.66s/it]
                                                          
{'loss': 0.1273, 'grad_norm': 1.0783185958862305, 'learning_rate': 3.933842210105048e-05, 'epoch': 1.1}

 11%|█         | 4523/41250 [10:56:22<88:22:32,  8.66s/it][2025-04-25 18:54:05,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-25 18:54:05,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.78 | bwd_microstep: 5678.04 | bwd_inner_microstep: 5658.83 | bwd_allreduce_microstep: 19.17 | step_microstep: 19.02
[2025-04-25 18:54:05,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.78 | bwd: 5678.06 | bwd_inner: 5658.83 | bwd_allreduce: 19.19 | step: 19.02

 11%|█         | 4524/41250 [10:56:30<88:09:56,  8.64s/it]
                                                          
{'loss': 0.0592, 'grad_norm': 1.3513944149017334, 'learning_rate': 3.933802148974623e-05, 'epoch': 1.1}

 11%|█         | 4524/41250 [10:56:30<88:09:56,  8.64s/it][2025-04-25 18:54:14,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 18:54:14,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.37 | bwd_microstep: 5696.35 | bwd_inner_microstep: 5637.39 | bwd_allreduce_microstep: 58.92 | step_microstep: 18.51
[2025-04-25 18:54:14,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.37 | bwd: 5696.36 | bwd_inner: 5637.38 | bwd_allreduce: 58.94 | step: 18.51

 11%|█         | 4525/41250 [10:56:39<88:01:17,  8.63s/it]
                                                          
{'loss': 0.2173, 'grad_norm': 2.3078246116638184, 'learning_rate': 3.933762075922689e-05, 'epoch': 1.1}

 11%|█         | 4525/41250 [10:56:39<88:01:17,  8.63s/it][2025-04-25 18:54:22,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.96 | optimizer_step: 1.02
[2025-04-25 18:54:22,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.50 | bwd_microstep: 5756.66 | bwd_inner_microstep: 5692.22 | bwd_allreduce_microstep: 64.40 | step_microstep: 18.29
[2025-04-25 18:54:22,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.50 | bwd: 5756.67 | bwd_inner: 5692.22 | bwd_allreduce: 64.42 | step: 18.29

 11%|█         | 4526/41250 [10:56:48<88:11:23,  8.65s/it]
                                                          
{'loss': 0.0387, 'grad_norm': 0.3723589777946472, 'learning_rate': 3.933721990949492e-05, 'epoch': 1.1}

 11%|█         | 4526/41250 [10:56:48<88:11:23,  8.65s/it][2025-04-25 18:54:31,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 18:54:31,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.14 | bwd_microstep: 5868.20 | bwd_inner_microstep: 5643.90 | bwd_allreduce_microstep: 224.26 | step_microstep: 18.55
[2025-04-25 18:54:31,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.14 | bwd: 5868.22 | bwd_inner: 5643.90 | bwd_allreduce: 224.28 | step: 18.56

 11%|█         | 4527/41250 [10:56:57<88:34:21,  8.68s/it]
                                                          
{'loss': 0.1727, 'grad_norm': 1.9564958810806274, 'learning_rate': 3.933681894055279e-05, 'epoch': 1.1}

 11%|█         | 4527/41250 [10:56:57<88:34:21,  8.68s/it][2025-04-25 18:54:40,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 18:54:40,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.73 | bwd_microstep: 5888.24 | bwd_inner_microstep: 5875.41 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.78
[2025-04-25 18:54:40,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.73 | bwd: 5888.25 | bwd_inner: 5875.41 | bwd_allreduce: 12.80 | step: 18.78

 11%|█         | 4528/41250 [10:57:05<89:15:49,  8.75s/it]
                                                          
{'loss': 0.0875, 'grad_norm': 1.4494255781173706, 'learning_rate': 3.933641785240298e-05, 'epoch': 1.1}

 11%|█         | 4528/41250 [10:57:05<89:15:49,  8.75s/it][2025-04-25 18:54:49,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 18:54:49,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.39 | bwd_microstep: 5701.78 | bwd_inner_microstep: 5688.96 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.75
[2025-04-25 18:54:49,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.39 | bwd: 5701.79 | bwd_inner: 5688.96 | bwd_allreduce: 12.79 | step: 18.75

 11%|█         | 4529/41250 [10:57:14<88:53:16,  8.71s/it]
                                                          
{'loss': 0.1698, 'grad_norm': 1.0681403875350952, 'learning_rate': 3.933601664504796e-05, 'epoch': 1.1}

 11%|█         | 4529/41250 [10:57:14<88:53:16,  8.71s/it][2025-04-25 18:54:57,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 18:54:57,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 5758.73 | bwd_inner_microstep: 5645.72 | bwd_allreduce_microstep: 112.96 | step_microstep: 19.04
[2025-04-25 18:54:57,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 5758.74 | bwd_inner: 5645.72 | bwd_allreduce: 112.97 | step: 19.05

 11%|█         | 4530/41250 [10:57:23<88:44:54,  8.70s/it]
                                                          
{'loss': 0.1359, 'grad_norm': 2.0846006870269775, 'learning_rate': 3.933561531849019e-05, 'epoch': 1.1}

 11%|█         | 4530/41250 [10:57:23<88:44:54,  8.70s/it][2025-04-25 18:55:06,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 18:55:06,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.30 | bwd_microstep: 5866.84 | bwd_inner_microstep: 5641.43 | bwd_allreduce_microstep: 225.37 | step_microstep: 19.23
[2025-04-25 18:55:06,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.30 | bwd: 5866.86 | bwd_inner: 5641.43 | bwd_allreduce: 225.38 | step: 19.24

 11%|█         | 4531/41250 [10:57:32<88:58:30,  8.72s/it]
                                                          
{'loss': 0.1426, 'grad_norm': 1.4769189357757568, 'learning_rate': 3.933521387273216e-05, 'epoch': 1.1}

 11%|█         | 4531/41250 [10:57:32<88:58:30,  8.72s/it][2025-04-25 18:55:15,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:55:15,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5752.47 | bwd_inner_microstep: 5698.58 | bwd_allreduce_microstep: 53.84 | step_microstep: 18.92
[2025-04-25 18:55:15,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5752.48 | bwd_inner: 5698.58 | bwd_allreduce: 53.86 | step: 18.93

 11%|█         | 4532/41250 [10:57:40<88:51:27,  8.71s/it]
                                                          
{'loss': 0.1074, 'grad_norm': 1.531858205795288, 'learning_rate': 3.933481230777634e-05, 'epoch': 1.1}

 11%|█         | 4532/41250 [10:57:40<88:51:27,  8.71s/it][2025-04-25 18:55:23,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 18:55:23,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5700.79 | bwd_inner_microstep: 5687.88 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.38
[2025-04-25 18:55:23,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5700.80 | bwd_inner: 5687.88 | bwd_allreduce: 12.88 | step: 18.38

 11%|█         | 4533/41250 [10:57:49<88:35:45,  8.69s/it]
                                                          
{'loss': 0.3277, 'grad_norm': 2.9874019622802734, 'learning_rate': 3.9334410623625203e-05, 'epoch': 1.1}

 11%|█         | 4533/41250 [10:57:49<88:35:45,  8.69s/it][2025-04-25 18:55:32,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 18:55:32,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.23 | bwd_microstep: 5691.25 | bwd_inner_microstep: 5655.16 | bwd_allreduce_microstep: 36.04 | step_microstep: 18.76
[2025-04-25 18:55:32,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.23 | bwd: 5691.27 | bwd_inner: 5655.16 | bwd_allreduce: 36.06 | step: 18.77

 11%|█         | 4534/41250 [10:57:57<88:21:45,  8.66s/it]
                                                          
{'loss': 0.2192, 'grad_norm': 2.2776594161987305, 'learning_rate': 3.9334008820281236e-05, 'epoch': 1.1}

 11%|█         | 4534/41250 [10:57:57<88:21:45,  8.66s/it][2025-04-25 18:55:41,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:55:41,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.61 | bwd_microstep: 5685.29 | bwd_inner_microstep: 5642.17 | bwd_allreduce_microstep: 43.07 | step_microstep: 18.75
[2025-04-25 18:55:41,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.62 | bwd: 5685.30 | bwd_inner: 5642.17 | bwd_allreduce: 43.09 | step: 18.75

 11%|█         | 4535/41250 [10:58:06<88:07:25,  8.64s/it]
                                                          
{'loss': 0.1959, 'grad_norm': 2.2234511375427246, 'learning_rate': 3.93336068977469e-05, 'epoch': 1.1}

 11%|█         | 4535/41250 [10:58:06<88:07:25,  8.64s/it][2025-04-25 18:55:49,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 18:55:49,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.16 | bwd_microstep: 5720.14 | bwd_inner_microstep: 5707.42 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.94
[2025-04-25 18:55:49,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.17 | bwd: 5720.15 | bwd_inner: 5707.42 | bwd_allreduce: 12.69 | step: 18.94

 11%|█         | 4536/41250 [10:58:15<88:10:53,  8.65s/it]
                                                          
{'loss': 0.1388, 'grad_norm': 1.2779003381729126, 'learning_rate': 3.933320485602468e-05, 'epoch': 1.1}

 11%|█         | 4536/41250 [10:58:15<88:10:53,  8.65s/it][2025-04-25 18:55:58,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 18:55:58,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.17 | bwd_microstep: 5720.42 | bwd_inner_microstep: 5690.87 | bwd_allreduce_microstep: 29.51 | step_microstep: 19.15
[2025-04-25 18:55:58,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.17 | bwd: 5720.44 | bwd_inner: 5690.87 | bwd_allreduce: 29.53 | step: 19.15

 11%|█         | 4537/41250 [10:58:23<88:13:17,  8.65s/it]
                                                          
{'loss': 0.1598, 'grad_norm': 1.5102986097335815, 'learning_rate': 3.933280269511705e-05, 'epoch': 1.1}

 11%|█         | 4537/41250 [10:58:23<88:13:17,  8.65s/it][2025-04-25 18:56:07,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 18:56:07,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5745.22 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 87.72 | step_microstep: 18.81
[2025-04-25 18:56:07,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5745.23 | bwd_inner: 5657.45 | bwd_allreduce: 87.74 | step: 18.81

 11%|█         | 4538/41250 [10:58:32<88:14:31,  8.65s/it]
                                                          
{'loss': 0.0922, 'grad_norm': 1.1246477365493774, 'learning_rate': 3.93324004150265e-05, 'epoch': 1.1}

 11%|█         | 4538/41250 [10:58:32<88:14:31,  8.65s/it][2025-04-25 18:56:15,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 18:56:15,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.31 | bwd_microstep: 5688.52 | bwd_inner_microstep: 5645.39 | bwd_allreduce_microstep: 43.09 | step_microstep: 18.41
[2025-04-25 18:56:15,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.31 | bwd: 5688.53 | bwd_inner: 5645.39 | bwd_allreduce: 43.11 | step: 18.42

 11%|█         | 4539/41250 [10:58:41<88:05:44,  8.64s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 1.1636536121368408, 'learning_rate': 3.93319980157555e-05, 'epoch': 1.1}

 11%|█         | 4539/41250 [10:58:41<88:05:44,  8.64s/it][2025-04-25 18:56:24,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:56:24,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.63 | bwd_microstep: 5697.21 | bwd_inner_microstep: 5650.50 | bwd_allreduce_microstep: 46.66 | step_microstep: 18.93
[2025-04-25 18:56:24,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.63 | bwd: 5697.22 | bwd_inner: 5650.50 | bwd_allreduce: 46.68 | step: 18.93

 11%|█         | 4540/41250 [10:58:49<88:01:00,  8.63s/it]
                                                          
{'loss': 0.1543, 'grad_norm': 2.8506689071655273, 'learning_rate': 3.933159549730654e-05, 'epoch': 1.1}

 11%|█         | 4540/41250 [10:58:49<88:01:00,  8.63s/it][2025-04-25 18:56:33,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.19 | optimizer_step: 0.91
[2025-04-25 18:56:33,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5712.51 | bwd_inner_microstep: 5698.87 | bwd_allreduce_microstep: 13.59 | step_microstep: 19.33
[2025-04-25 18:56:33,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5712.52 | bwd_inner: 5698.87 | bwd_allreduce: 13.61 | step: 19.33

 11%|█         | 4541/41250 [10:58:58<88:05:24,  8.64s/it]
                                                          
{'loss': 0.1242, 'grad_norm': 1.7450355291366577, 'learning_rate': 3.93311928596821e-05, 'epoch': 1.1}

 11%|█         | 4541/41250 [10:58:58<88:05:24,  8.64s/it][2025-04-25 18:56:41,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 18:56:41,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.15 | bwd_microstep: 5761.86 | bwd_inner_microstep: 5654.33 | bwd_allreduce_microstep: 107.48 | step_microstep: 18.61
[2025-04-25 18:56:41,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.15 | bwd: 5761.88 | bwd_inner: 5654.33 | bwd_allreduce: 107.50 | step: 18.61

 11%|█         | 4542/41250 [10:59:07<88:12:37,  8.65s/it]
                                                          
{'loss': 0.1236, 'grad_norm': 2.2751147747039795, 'learning_rate': 3.9330790102884646e-05, 'epoch': 1.1}

 11%|█         | 4542/41250 [10:59:07<88:12:37,  8.65s/it][2025-04-25 18:56:50,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 18:56:50,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5752.33 | bwd_inner_microstep: 5699.93 | bwd_allreduce_microstep: 52.34 | step_microstep: 18.63
[2025-04-25 18:56:50,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5752.35 | bwd_inner: 5699.93 | bwd_allreduce: 52.37 | step: 18.63

 11%|█         | 4543/41250 [10:59:15<88:21:06,  8.66s/it]
                                                          
{'loss': 0.1268, 'grad_norm': 1.0480449199676514, 'learning_rate': 3.933038722691669e-05, 'epoch': 1.1}

 11%|█         | 4543/41250 [10:59:15<88:21:06,  8.66s/it][2025-04-25 18:56:59,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 18:56:59,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.03 | bwd_microstep: 5787.55 | bwd_inner_microstep: 5670.04 | bwd_allreduce_microstep: 117.45 | step_microstep: 19.36
[2025-04-25 18:56:59,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.03 | bwd: 5787.56 | bwd_inner: 5670.04 | bwd_allreduce: 117.48 | step: 19.36

 11%|█         | 4544/41250 [10:59:24<88:28:56,  8.68s/it]
                                                          
{'loss': 0.0462, 'grad_norm': 0.7029061913490295, 'learning_rate': 3.932998423178068e-05, 'epoch': 1.1}

 11%|█         | 4544/41250 [10:59:24<88:28:56,  8.68s/it][2025-04-25 18:57:07,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 18:57:07,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.36 | bwd_microstep: 5724.17 | bwd_inner_microstep: 5711.27 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.48
[2025-04-25 18:57:07,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.36 | bwd: 5724.19 | bwd_inner: 5711.27 | bwd_allreduce: 12.88 | step: 18.49

 11%|█         | 4545/41250 [10:59:33<88:26:38,  8.67s/it]
                                                          
{'loss': 0.1589, 'grad_norm': 2.0694775581359863, 'learning_rate': 3.9329581117479125e-05, 'epoch': 1.1}

 11%|█         | 4545/41250 [10:59:33<88:26:38,  8.67s/it][2025-04-25 18:57:16,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 18:57:16,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.01 | bwd_microstep: 5787.26 | bwd_inner_microstep: 5664.79 | bwd_allreduce_microstep: 122.42 | step_microstep: 19.07
[2025-04-25 18:57:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.01 | bwd: 5787.28 | bwd_inner: 5664.79 | bwd_allreduce: 122.44 | step: 19.07

 11%|█         | 4546/41250 [10:59:41<88:32:03,  8.68s/it]
                                                          
{'loss': 0.3852, 'grad_norm': 2.812570810317993, 'learning_rate': 3.932917788401451e-05, 'epoch': 1.1}

 11%|█         | 4546/41250 [10:59:41<88:32:03,  8.68s/it][2025-04-25 18:57:25,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:57:25,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5772.90 | bwd_inner_microstep: 5715.12 | bwd_allreduce_microstep: 57.73 | step_microstep: 18.36
[2025-04-25 18:57:25,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5772.91 | bwd_inner: 5715.12 | bwd_allreduce: 57.75 | step: 18.36

 11%|█         | 4547/41250 [10:59:50<88:36:34,  8.69s/it]
                                                          
{'loss': 0.1538, 'grad_norm': 1.2244116067886353, 'learning_rate': 3.932877453138931e-05, 'epoch': 1.1}

 11%|█         | 4547/41250 [10:59:50<88:36:34,  8.69s/it][2025-04-25 18:57:33,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:57:33,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.13 | bwd_microstep: 5806.85 | bwd_inner_microstep: 5671.17 | bwd_allreduce_microstep: 135.63 | step_microstep: 18.67
[2025-04-25 18:57:33,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.13 | bwd: 5806.87 | bwd_inner: 5671.17 | bwd_allreduce: 135.66 | step: 18.67

 11%|█         | 4548/41250 [10:59:59<88:42:44,  8.70s/it]
                                                          
{'loss': 0.109, 'grad_norm': 1.8208914995193481, 'learning_rate': 3.9328371059606015e-05, 'epoch': 1.1}

 11%|█         | 4548/41250 [10:59:59<88:42:44,  8.70s/it][2025-04-25 18:57:42,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 18:57:42,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.92 | bwd_microstep: 5734.26 | bwd_inner_microstep: 5659.11 | bwd_allreduce_microstep: 75.11 | step_microstep: 18.78
[2025-04-25 18:57:42,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.92 | bwd: 5734.27 | bwd_inner: 5659.11 | bwd_allreduce: 75.12 | step: 18.78

 11%|█         | 4549/41250 [11:00:07<88:32:20,  8.68s/it]
                                                          
{'loss': 0.2296, 'grad_norm': 1.3054227828979492, 'learning_rate': 3.932796746866712e-05, 'epoch': 1.1}

 11%|█         | 4549/41250 [11:00:07<88:32:20,  8.68s/it][2025-04-25 18:57:51,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 18:57:51,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.92 | bwd_microstep: 5813.36 | bwd_inner_microstep: 5649.88 | bwd_allreduce_microstep: 163.44 | step_microstep: 19.84
[2025-04-25 18:57:51,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.92 | bwd: 5813.38 | bwd_inner: 5649.88 | bwd_allreduce: 163.45 | step: 19.84

 11%|█         | 4550/41250 [11:00:16<88:39:06,  8.70s/it]
                                                          
{'loss': 0.177, 'grad_norm': 1.2824547290802002, 'learning_rate': 3.9327563758575106e-05, 'epoch': 1.1}

 11%|█         | 4550/41250 [11:00:16<88:39:06,  8.70s/it][2025-04-25 18:58:00,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 18:58:00,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.29 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5717.05 | bwd_allreduce_microstep: 46.31 | step_microstep: 18.69
[2025-04-25 18:58:00,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.29 | bwd: 5763.41 | bwd_inner: 5717.04 | bwd_allreduce: 46.32 | step: 18.69

 11%|█         | 4551/41250 [11:00:25<88:40:33,  8.70s/it]
                                                          
{'loss': 0.189, 'grad_norm': 1.4757014513015747, 'learning_rate': 3.9327159929332464e-05, 'epoch': 1.1}

 11%|█         | 4551/41250 [11:00:25<88:40:33,  8.70s/it][2025-04-25 18:58:08,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 18:58:08,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.37 | bwd_microstep: 5835.11 | bwd_inner_microstep: 5665.33 | bwd_allreduce_microstep: 169.74 | step_microstep: 18.34
[2025-04-25 18:58:08,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.37 | bwd: 5835.13 | bwd_inner: 5665.33 | bwd_allreduce: 169.75 | step: 18.34

 11%|█         | 4552/41250 [11:00:34<88:48:46,  8.71s/it]
                                                          
{'loss': 0.2469, 'grad_norm': 2.194135904312134, 'learning_rate': 3.932675598094168e-05, 'epoch': 1.1}

 11%|█         | 4552/41250 [11:00:34<88:48:46,  8.71s/it][2025-04-25 18:58:17,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 18:58:17,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.70 | bwd_microstep: 5744.70 | bwd_inner_microstep: 5653.09 | bwd_allreduce_microstep: 91.57 | step_microstep: 18.63
[2025-04-25 18:58:17,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.70 | bwd: 5744.71 | bwd_inner: 5653.09 | bwd_allreduce: 91.58 | step: 18.63

 11%|█         | 4553/41250 [11:00:42<88:37:38,  8.69s/it]
                                                          
{'loss': 0.0401, 'grad_norm': 0.41865476965904236, 'learning_rate': 3.9326351913405246e-05, 'epoch': 1.1}

 11%|█         | 4553/41250 [11:00:42<88:37:38,  8.69s/it][2025-04-25 18:58:26,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 18:58:26,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5723.09 | bwd_inner_microstep: 5699.39 | bwd_allreduce_microstep: 23.66 | step_microstep: 18.57
[2025-04-25 18:58:26,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5723.11 | bwd_inner: 5699.39 | bwd_allreduce: 23.68 | step: 18.58

 11%|█         | 4554/41250 [11:00:51<88:29:19,  8.68s/it]
                                                          
{'loss': 0.0834, 'grad_norm': 2.745962619781494, 'learning_rate': 3.932594772672565e-05, 'epoch': 1.1}

 11%|█         | 4554/41250 [11:00:51<88:29:19,  8.68s/it][2025-04-25 18:58:34,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 18:58:34,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.49 | bwd_microstep: 5827.98 | bwd_inner_microstep: 5663.10 | bwd_allreduce_microstep: 164.83 | step_microstep: 18.68
[2025-04-25 18:58:34,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.49 | bwd: 5827.99 | bwd_inner: 5663.10 | bwd_allreduce: 164.85 | step: 18.68

 11%|█         | 4555/41250 [11:01:00<88:39:18,  8.70s/it]
                                                          
{'loss': 0.0622, 'grad_norm': 0.9960505962371826, 'learning_rate': 3.9325543420905395e-05, 'epoch': 1.1}

 11%|█         | 4555/41250 [11:01:00<88:39:18,  8.70s/it][2025-04-25 18:58:43,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.09 | optimizer_step: 0.98
[2025-04-25 18:58:43,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.06 | bwd_microstep: 5781.82 | bwd_inner_microstep: 5768.46 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.49
[2025-04-25 18:58:43,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.06 | bwd: 5781.83 | bwd_inner: 5768.46 | bwd_allreduce: 13.32 | step: 19.50

 11%|█         | 4556/41250 [11:01:08<88:48:39,  8.71s/it]
                                                          
{'loss': 0.2152, 'grad_norm': 1.3789695501327515, 'learning_rate': 3.932513899594697e-05, 'epoch': 1.1}

 11%|█         | 4556/41250 [11:01:08<88:48:39,  8.71s/it][2025-04-25 18:58:52,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 18:58:52,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5780.42 | bwd_inner_microstep: 5667.80 | bwd_allreduce_microstep: 112.58 | step_microstep: 18.40
[2025-04-25 18:58:52,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.49 | bwd: 5780.44 | bwd_inner: 5667.80 | bwd_allreduce: 112.60 | step: 18.40

 11%|█         | 4557/41250 [11:01:17<88:45:01,  8.71s/it]
                                                          
{'loss': 0.0465, 'grad_norm': 0.6915189623832703, 'learning_rate': 3.9324734451852856e-05, 'epoch': 1.1}

 11%|█         | 4557/41250 [11:01:17<88:45:01,  8.71s/it][2025-04-25 18:59:00,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 18:59:00,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.98 | bwd_microstep: 5723.56 | bwd_inner_microstep: 5711.00 | bwd_allreduce_microstep: 12.52 | step_microstep: 19.31
[2025-04-25 18:59:00,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.98 | bwd: 5723.58 | bwd_inner: 5711.00 | bwd_allreduce: 12.53 | step: 19.32

 11%|█         | 4558/41250 [11:01:26<88:34:36,  8.69s/it]
                                                          
{'loss': 0.0555, 'grad_norm': 0.7151979207992554, 'learning_rate': 3.932432978862555e-05, 'epoch': 1.1}

 11%|█         | 4558/41250 [11:01:26<88:34:36,  8.69s/it][2025-04-25 18:59:09,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 18:59:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.08 | bwd_microstep: 5737.71 | bwd_inner_microstep: 5668.87 | bwd_allreduce_microstep: 68.79 | step_microstep: 18.70
[2025-04-25 18:59:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.08 | bwd: 5737.72 | bwd_inner: 5668.87 | bwd_allreduce: 68.81 | step: 18.70

 11%|█         | 4559/41250 [11:01:34<88:27:10,  8.68s/it]
                                                          
{'loss': 0.0819, 'grad_norm': 1.004449486732483, 'learning_rate': 3.932392500626756e-05, 'epoch': 1.11}

 11%|█         | 4559/41250 [11:01:34<88:27:10,  8.68s/it][2025-04-25 18:59:18,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 18:59:18,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5802.33 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 144.69 | step_microstep: 18.42
[2025-04-25 18:59:18,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5802.35 | bwd_inner: 5657.59 | bwd_allreduce: 144.71 | step: 18.43

 11%|█         | 4560/41250 [11:01:43<88:33:06,  8.69s/it]
                                                          
{'loss': 0.0978, 'grad_norm': 1.0364960432052612, 'learning_rate': 3.932352010478137e-05, 'epoch': 1.11}

 11%|█         | 4560/41250 [11:01:43<88:33:06,  8.69s/it][2025-04-25 18:59:26,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 18:59:26,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.25 | bwd_microstep: 5810.93 | bwd_inner_microstep: 5663.69 | bwd_allreduce_microstep: 147.20 | step_microstep: 19.08
[2025-04-25 18:59:26,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.26 | bwd: 5810.95 | bwd_inner: 5663.69 | bwd_allreduce: 147.21 | step: 19.08

 11%|█         | 4561/41250 [11:01:52<88:39:41,  8.70s/it]
                                                          
{'loss': 0.2334, 'grad_norm': 1.7201008796691895, 'learning_rate': 3.9323115084169485e-05, 'epoch': 1.11}

 11%|█         | 4561/41250 [11:01:52<88:39:41,  8.70s/it][2025-04-25 18:59:35,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 18:59:35,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.57 | bwd_microstep: 5707.77 | bwd_inner_microstep: 5695.05 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.77
[2025-04-25 18:59:35,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.57 | bwd: 5707.78 | bwd_inner: 5695.05 | bwd_allreduce: 12.69 | step: 18.78

 11%|█         | 4562/41250 [11:02:00<88:28:56,  8.68s/it]
                                                          
{'loss': 0.1034, 'grad_norm': 1.5235018730163574, 'learning_rate': 3.932270994443439e-05, 'epoch': 1.11}

 11%|█         | 4562/41250 [11:02:00<88:28:56,  8.68s/it][2025-04-25 18:59:44,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 18:59:44,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.38 | bwd_microstep: 5796.53 | bwd_inner_microstep: 5783.86 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.54
[2025-04-25 18:59:44,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.38 | bwd: 5796.54 | bwd_inner: 5783.86 | bwd_allreduce: 12.64 | step: 18.54

 11%|█         | 4563/41250 [11:02:09<88:44:24,  8.71s/it]
                                                          
{'loss': 0.2135, 'grad_norm': 2.1221041679382324, 'learning_rate': 3.932230468557859e-05, 'epoch': 1.11}

 11%|█         | 4563/41250 [11:02:09<88:44:24,  8.71s/it][2025-04-25 18:59:53,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 18:59:53,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5747.15 | bwd_inner_microstep: 5709.60 | bwd_allreduce_microstep: 37.50 | step_microstep: 19.02
[2025-04-25 18:59:53,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5747.16 | bwd_inner: 5709.60 | bwd_allreduce: 37.52 | step: 19.02

 11%|█         | 4564/41250 [11:02:18<88:40:00,  8.70s/it]
                                                          
{'loss': 0.2895, 'grad_norm': 1.8766146898269653, 'learning_rate': 3.9321899307604586e-05, 'epoch': 1.11}

 11%|█         | 4564/41250 [11:02:18<88:40:00,  8.70s/it][2025-04-25 19:00:01,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 19:00:01,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5751.62 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 58.71 | step_microstep: 18.79
[2025-04-25 19:00:01,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5751.63 | bwd_inner: 5692.86 | bwd_allreduce: 58.73 | step: 18.79

 11%|█         | 4565/41250 [11:02:27<88:36:32,  8.70s/it]
                                                          
{'loss': 0.0804, 'grad_norm': 1.1136682033538818, 'learning_rate': 3.9321493810514864e-05, 'epoch': 1.11}

 11%|█         | 4565/41250 [11:02:27<88:36:32,  8.70s/it][2025-04-25 19:00:10,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-25 19:00:10,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.00 | bwd_microstep: 5757.96 | bwd_inner_microstep: 5661.44 | bwd_allreduce_microstep: 96.47 | step_microstep: 18.84
[2025-04-25 19:00:10,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.00 | bwd: 5757.97 | bwd_inner: 5661.44 | bwd_allreduce: 96.49 | step: 18.85

 11%|█         | 4566/41250 [11:02:35<88:34:02,  8.69s/it]
                                                          
{'loss': 0.2405, 'grad_norm': 2.052882432937622, 'learning_rate': 3.932108819431194e-05, 'epoch': 1.11}

 11%|█         | 4566/41250 [11:02:35<88:34:02,  8.69s/it][2025-04-25 19:00:19,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:00:19,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.30 | bwd_microstep: 5776.64 | bwd_inner_microstep: 5652.13 | bwd_allreduce_microstep: 124.46 | step_microstep: 18.68
[2025-04-25 19:00:19,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.30 | bwd: 5776.65 | bwd_inner: 5652.13 | bwd_allreduce: 124.48 | step: 18.68

 11%|█         | 4567/41250 [11:02:44<88:34:27,  8.69s/it]
                                                          
{'loss': 0.0923, 'grad_norm': 0.8716554045677185, 'learning_rate': 3.9320682458998306e-05, 'epoch': 1.11}

 11%|█         | 4567/41250 [11:02:44<88:34:27,  8.69s/it][2025-04-25 19:00:27,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.36 | optimizer_step: 1.06
[2025-04-25 19:00:27,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5710.86 | bwd_inner_microstep: 5696.65 | bwd_allreduce_microstep: 14.15 | step_microstep: 20.35
[2025-04-25 19:00:27,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5710.88 | bwd_inner: 5696.65 | bwd_allreduce: 14.18 | step: 20.35

 11%|█         | 4568/41250 [11:02:53<88:25:59,  8.68s/it]
                                                          
{'loss': 0.028, 'grad_norm': 0.6333507299423218, 'learning_rate': 3.9320276604576466e-05, 'epoch': 1.11}

 11%|█         | 4568/41250 [11:02:53<88:25:59,  8.68s/it][2025-04-25 19:00:36,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 19:00:36,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2940.22 | bwd_microstep: 5878.07 | bwd_inner_microstep: 5865.00 | bwd_allreduce_microstep: 13.02 | step_microstep: 19.11
[2025-04-25 19:00:36,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2940.22 | bwd: 5878.08 | bwd_inner: 5865.00 | bwd_allreduce: 13.04 | step: 19.11

 11%|█         | 4569/41250 [11:03:02<89:07:24,  8.75s/it]
                                                          
{'loss': 0.2313, 'grad_norm': 1.7262234687805176, 'learning_rate': 3.931987063104892e-05, 'epoch': 1.11}

 11%|█         | 4569/41250 [11:03:02<89:07:24,  8.75s/it][2025-04-25 19:00:45,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:00:45,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.18 | bwd_microstep: 5745.92 | bwd_inner_microstep: 5646.98 | bwd_allreduce_microstep: 98.89 | step_microstep: 18.50
[2025-04-25 19:00:45,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.18 | bwd: 5745.94 | bwd_inner: 5646.98 | bwd_allreduce: 98.91 | step: 18.51

 11%|█         | 4570/41250 [11:03:10<88:50:50,  8.72s/it]
                                                          
{'loss': 0.1781, 'grad_norm': 2.5242016315460205, 'learning_rate': 3.931946453841817e-05, 'epoch': 1.11}

 11%|█         | 4570/41250 [11:03:10<88:50:50,  8.72s/it][2025-04-25 19:00:53,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.24 | optimizer_step: 1.03
[2025-04-25 19:00:53,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.65 | bwd_microstep: 5714.11 | bwd_inner_microstep: 5673.84 | bwd_allreduce_microstep: 40.20 | step_microstep: 19.61
[2025-04-25 19:00:53,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.67 | bwd: 5714.11 | bwd_inner: 5673.82 | bwd_allreduce: 40.22 | step: 19.61

 11%|█         | 4571/41250 [11:03:19<88:38:10,  8.70s/it]
                                                          
{'loss': 0.2504, 'grad_norm': 1.8800952434539795, 'learning_rate': 3.931905832668672e-05, 'epoch': 1.11}

 11%|█         | 4571/41250 [11:03:19<88:38:10,  8.70s/it][2025-04-25 19:01:02,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:01:02,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.99 | bwd_microstep: 5872.67 | bwd_inner_microstep: 5859.82 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.89
[2025-04-25 19:01:02,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.99 | bwd: 5872.68 | bwd_inner: 5859.82 | bwd_allreduce: 12.82 | step: 18.89

 11%|█         | 4572/41250 [11:03:28<89:13:32,  8.76s/it]
                                                          
{'loss': 0.2138, 'grad_norm': 2.530850410461426, 'learning_rate': 3.931865199585708e-05, 'epoch': 1.11}

 11%|█         | 4572/41250 [11:03:28<89:13:32,  8.76s/it][2025-04-25 19:01:11,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.73 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:01:11,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.44 | bwd_microstep: 5717.71 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 16.83 | step_microstep: 20.67
[2025-04-25 19:01:11,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.44 | bwd: 5717.72 | bwd_inner: 5700.83 | bwd_allreduce: 16.85 | step: 20.68

 11%|█         | 4573/41250 [11:03:36<88:56:40,  8.73s/it]
                                                          
{'loss': 0.0599, 'grad_norm': 1.7945847511291504, 'learning_rate': 3.931824554593175e-05, 'epoch': 1.11}

 11%|█         | 4573/41250 [11:03:36<88:56:40,  8.73s/it][2025-04-25 19:01:20,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 19:01:20,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.60 | bwd_microstep: 5737.45 | bwd_inner_microstep: 5685.95 | bwd_allreduce_microstep: 51.44 | step_microstep: 19.23
[2025-04-25 19:01:20,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.60 | bwd: 5737.46 | bwd_inner: 5685.95 | bwd_allreduce: 51.47 | step: 19.23

 11%|█         | 4574/41250 [11:03:45<88:45:49,  8.71s/it]
                                                          
{'loss': 0.1837, 'grad_norm': 6.341294765472412, 'learning_rate': 3.9317838976913234e-05, 'epoch': 1.11}

 11%|█         | 4574/41250 [11:03:45<88:45:49,  8.71s/it][2025-04-25 19:01:28,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.06 | optimizer_step: 0.93
[2025-04-25 19:01:28,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.07 | bwd_microstep: 5710.13 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.94
[2025-04-25 19:01:28,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.07 | bwd: 5710.14 | bwd_inner: 5696.99 | bwd_allreduce: 13.11 | step: 18.94

 11%|█         | 4575/41250 [11:03:54<88:33:20,  8.69s/it]
                                                          
{'loss': 0.0943, 'grad_norm': 0.9967871904373169, 'learning_rate': 3.931743228880404e-05, 'epoch': 1.11}

 11%|█         | 4575/41250 [11:03:54<88:33:20,  8.69s/it][2025-04-25 19:01:37,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:01:37,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.55 | bwd_microstep: 5754.86 | bwd_inner_microstep: 5689.92 | bwd_allreduce_microstep: 64.90 | step_microstep: 18.64
[2025-04-25 19:01:37,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.55 | bwd: 5754.87 | bwd_inner: 5689.92 | bwd_allreduce: 64.91 | step: 18.64

 11%|█         | 4576/41250 [11:04:02<88:31:03,  8.69s/it]
                                                          
{'loss': 0.035, 'grad_norm': 0.4574890732765198, 'learning_rate': 3.9317025481606676e-05, 'epoch': 1.11}

 11%|█         | 4576/41250 [11:04:02<88:31:03,  8.69s/it][2025-04-25 19:01:46,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:01:46,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.43 | bwd_microstep: 5696.70 | bwd_inner_microstep: 5637.18 | bwd_allreduce_microstep: 59.46 | step_microstep: 18.72
[2025-04-25 19:01:46,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.43 | bwd: 5696.71 | bwd_inner: 5637.18 | bwd_allreduce: 59.49 | step: 18.72

 11%|█         | 4577/41250 [11:04:11<88:15:57,  8.66s/it]
                                                          
{'loss': 0.1866, 'grad_norm': 1.736091136932373, 'learning_rate': 3.931661855532365e-05, 'epoch': 1.11}

 11%|█         | 4577/41250 [11:04:11<88:15:57,  8.66s/it][2025-04-25 19:01:54,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.15
[2025-04-25 19:01:54,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.60 | bwd_microstep: 5748.97 | bwd_inner_microstep: 5638.14 | bwd_allreduce_microstep: 110.78 | step_microstep: 19.71
[2025-04-25 19:01:54,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.60 | bwd: 5748.98 | bwd_inner: 5638.14 | bwd_allreduce: 110.80 | step: 19.71

 11%|█         | 4578/41250 [11:04:20<88:14:44,  8.66s/it]
                                                          
{'loss': 0.2633, 'grad_norm': 1.8835053443908691, 'learning_rate': 3.9316211509957465e-05, 'epoch': 1.11}

 11%|█         | 4578/41250 [11:04:20<88:14:44,  8.66s/it][2025-04-25 19:02:03,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:02:03,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.76 | bwd_microstep: 5737.32 | bwd_inner_microstep: 5656.52 | bwd_allreduce_microstep: 80.76 | step_microstep: 18.51
[2025-04-25 19:02:03,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.76 | bwd: 5737.33 | bwd_inner: 5656.52 | bwd_allreduce: 80.77 | step: 18.51

 11%|█         | 4579/41250 [11:04:28<88:11:44,  8.66s/it]
                                                          
{'loss': 0.1029, 'grad_norm': 1.436594009399414, 'learning_rate': 3.931580434551064e-05, 'epoch': 1.11}

 11%|█         | 4579/41250 [11:04:28<88:11:44,  8.66s/it][2025-04-25 19:02:12,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:02:12,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.15 | bwd_microstep: 5766.98 | bwd_inner_microstep: 5635.77 | bwd_allreduce_microstep: 131.15 | step_microstep: 18.94
[2025-04-25 19:02:12,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.15 | bwd: 5766.99 | bwd_inner: 5635.77 | bwd_allreduce: 131.18 | step: 18.94

 11%|█         | 4580/41250 [11:04:37<88:13:54,  8.66s/it]
                                                          
{'loss': 0.3264, 'grad_norm': 3.4999420642852783, 'learning_rate': 3.931539706198568e-05, 'epoch': 1.11}

 11%|█         | 4580/41250 [11:04:37<88:13:54,  8.66s/it][2025-04-25 19:02:20,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-25 19:02:20,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.34 | bwd_microstep: 5752.00 | bwd_inner_microstep: 5642.99 | bwd_allreduce_microstep: 108.96 | step_microstep: 19.25
[2025-04-25 19:02:20,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.34 | bwd: 5752.01 | bwd_inner: 5642.99 | bwd_allreduce: 108.98 | step: 19.25

 11%|█         | 4581/41250 [11:04:46<88:13:48,  8.66s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.4001758098602295, 'learning_rate': 3.93149896593851e-05, 'epoch': 1.11}

 11%|█         | 4581/41250 [11:04:46<88:13:48,  8.66s/it][2025-04-25 19:02:29,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 19:02:29,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.00 | bwd_microstep: 5702.62 | bwd_inner_microstep: 5689.27 | bwd_allreduce_microstep: 13.30 | step_microstep: 18.89
[2025-04-25 19:02:29,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.00 | bwd: 5702.63 | bwd_inner: 5689.27 | bwd_allreduce: 13.32 | step: 18.89

 11%|█         | 4582/41250 [11:04:54<88:07:22,  8.65s/it]
                                                          
{'loss': 0.4059, 'grad_norm': 2.4316213130950928, 'learning_rate': 3.9314582137711405e-05, 'epoch': 1.11}

 11%|█         | 4582/41250 [11:04:54<88:07:22,  8.65s/it][2025-04-25 19:02:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 19:02:38,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.28 | bwd_microstep: 5742.70 | bwd_inner_microstep: 5714.40 | bwd_allreduce_microstep: 28.25 | step_microstep: 19.32
[2025-04-25 19:02:38,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.28 | bwd: 5742.72 | bwd_inner: 5714.40 | bwd_allreduce: 28.27 | step: 19.32

 11%|█         | 4583/41250 [11:05:03<88:13:45,  8.66s/it]
                                                          
{'loss': 0.1653, 'grad_norm': 1.2079308032989502, 'learning_rate': 3.9314174496967104e-05, 'epoch': 1.11}

 11%|█         | 4583/41250 [11:05:03<88:13:45,  8.66s/it][2025-04-25 19:02:46,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.07 | optimizer_step: 0.93
[2025-04-25 19:02:46,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.23 | bwd_microstep: 5754.72 | bwd_inner_microstep: 5708.28 | bwd_allreduce_microstep: 46.39 | step_microstep: 19.47
[2025-04-25 19:02:46,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.23 | bwd: 5754.74 | bwd_inner: 5708.28 | bwd_allreduce: 46.41 | step: 19.47

 11%|█         | 4584/41250 [11:05:12<88:19:20,  8.67s/it]
                                                          
{'loss': 0.1602, 'grad_norm': 1.3148317337036133, 'learning_rate': 3.9313766737154725e-05, 'epoch': 1.11}

 11%|█         | 4584/41250 [11:05:12<88:19:20,  8.67s/it][2025-04-25 19:02:55,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:02:55,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.01 | bwd_microstep: 5700.05 | bwd_inner_microstep: 5687.10 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.91
[2025-04-25 19:02:55,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.01 | bwd: 5700.07 | bwd_inner: 5687.10 | bwd_allreduce: 12.93 | step: 18.91

 11%|█         | 4585/41250 [11:05:20<88:15:01,  8.66s/it]
                                                          
{'loss': 0.1225, 'grad_norm': 1.532497525215149, 'learning_rate': 3.931335885827677e-05, 'epoch': 1.11}

 11%|█         | 4585/41250 [11:05:20<88:15:01,  8.66s/it][2025-04-25 19:03:04,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.14 | optimizer_step: 0.97
[2025-04-25 19:03:04,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.17 | bwd_microstep: 5749.80 | bwd_inner_microstep: 5653.09 | bwd_allreduce_microstep: 96.66 | step_microstep: 19.17
[2025-04-25 19:03:04,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5749.82 | bwd_inner: 5653.09 | bwd_allreduce: 96.68 | step: 19.17

 11%|█         | 4586/41250 [11:05:29<88:13:58,  8.66s/it]
                                                          
{'loss': 0.066, 'grad_norm': 0.6060198545455933, 'learning_rate': 3.931295086033575e-05, 'epoch': 1.11}

 11%|█         | 4586/41250 [11:05:29<88:13:58,  8.66s/it][2025-04-25 19:03:12,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.09 | optimizer_step: 1.25
[2025-04-25 19:03:12,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.15 | bwd_microstep: 5763.65 | bwd_inner_microstep: 5751.00 | bwd_allreduce_microstep: 12.60 | step_microstep: 20.02
[2025-04-25 19:03:12,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.15 | bwd: 5763.66 | bwd_inner: 5751.00 | bwd_allreduce: 12.62 | step: 20.02

 11%|█         | 4587/41250 [11:05:38<88:27:01,  8.69s/it]
                                                          
{'loss': 0.1341, 'grad_norm': 1.756473422050476, 'learning_rate': 3.9312542743334195e-05, 'epoch': 1.11}

 11%|█         | 4587/41250 [11:05:38<88:27:01,  8.69s/it][2025-04-25 19:03:21,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.33 | optimizer_step: 0.90
[2025-04-25 19:03:21,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.14 | bwd_microstep: 5747.52 | bwd_inner_microstep: 5697.89 | bwd_allreduce_microstep: 49.57 | step_microstep: 19.73
[2025-04-25 19:03:21,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.14 | bwd: 5747.53 | bwd_inner: 5697.89 | bwd_allreduce: 49.60 | step: 19.73

 11%|█         | 4588/41250 [11:05:46<88:26:17,  8.68s/it]
                                                          
{'loss': 0.1727, 'grad_norm': 1.1536343097686768, 'learning_rate': 3.931213450727461e-05, 'epoch': 1.11}

 11%|█         | 4588/41250 [11:05:46<88:26:17,  8.68s/it][2025-04-25 19:03:30,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 19:03:30,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5699.52 | bwd_inner_microstep: 5686.19 | bwd_allreduce_microstep: 13.28 | step_microstep: 19.68
[2025-04-25 19:03:30,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5699.53 | bwd_inner: 5686.19 | bwd_allreduce: 13.30 | step: 19.68

 11%|█         | 4589/41250 [11:05:55<88:17:05,  8.67s/it]
                                                          
{'loss': 0.1086, 'grad_norm': 1.668796181678772, 'learning_rate': 3.931172615215952e-05, 'epoch': 1.11}

 11%|█         | 4589/41250 [11:05:55<88:17:05,  8.67s/it][2025-04-25 19:03:38,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 19:03:38,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.44 | bwd_microstep: 5718.99 | bwd_inner_microstep: 5706.39 | bwd_allreduce_microstep: 12.55 | step_microstep: 19.28
[2025-04-25 19:03:38,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.44 | bwd: 5719.00 | bwd_inner: 5706.39 | bwd_allreduce: 12.57 | step: 19.28

 11%|█         | 4590/41250 [11:06:04<88:15:38,  8.67s/it]
                                                          
{'loss': 0.0721, 'grad_norm': 1.0471773147583008, 'learning_rate': 3.9311317677991426e-05, 'epoch': 1.11}

 11%|█         | 4590/41250 [11:06:04<88:15:38,  8.67s/it][2025-04-25 19:03:47,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 19:03:47,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.02 | bwd_microstep: 5714.73 | bwd_inner_microstep: 5649.48 | bwd_allreduce_microstep: 65.20 | step_microstep: 19.07
[2025-04-25 19:03:47,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.02 | bwd: 5714.75 | bwd_inner: 5649.48 | bwd_allreduce: 65.22 | step: 19.07

 11%|█         | 4591/41250 [11:06:12<88:08:13,  8.66s/it]
                                                          
{'loss': 0.1529, 'grad_norm': 1.2147377729415894, 'learning_rate': 3.931090908477286e-05, 'epoch': 1.11}

 11%|█         | 4591/41250 [11:06:12<88:08:13,  8.66s/it][2025-04-25 19:03:56,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 19:03:56,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.14 | bwd_microstep: 5774.64 | bwd_inner_microstep: 5698.14 | bwd_allreduce_microstep: 76.44 | step_microstep: 20.08
[2025-04-25 19:03:56,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.14 | bwd: 5774.65 | bwd_inner: 5698.14 | bwd_allreduce: 76.46 | step: 20.08

 11%|█         | 4592/41250 [11:06:21<88:19:42,  8.67s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 2.055354356765747, 'learning_rate': 3.931050037250634e-05, 'epoch': 1.11}

 11%|█         | 4592/41250 [11:06:21<88:19:42,  8.67s/it][2025-04-25 19:04:04,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.11 | optimizer_step: 0.96
[2025-04-25 19:04:04,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.51 | bwd_microstep: 5711.26 | bwd_inner_microstep: 5664.33 | bwd_allreduce_microstep: 46.88 | step_microstep: 19.31
[2025-04-25 19:04:04,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.51 | bwd: 5711.27 | bwd_inner: 5664.33 | bwd_allreduce: 46.89 | step: 19.31

 11%|█         | 4593/41250 [11:06:30<88:11:58,  8.66s/it]
                                                          
{'loss': 0.0886, 'grad_norm': 1.7564442157745361, 'learning_rate': 3.931009154119438e-05, 'epoch': 1.11}

 11%|█         | 4593/41250 [11:06:30<88:11:58,  8.66s/it][2025-04-25 19:04:13,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 19:04:13,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.65 | bwd_microstep: 5758.53 | bwd_inner_microstep: 5699.92 | bwd_allreduce_microstep: 58.52 | step_microstep: 19.75
[2025-04-25 19:04:13,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.66 | bwd: 5758.55 | bwd_inner: 5699.92 | bwd_allreduce: 58.56 | step: 19.74

 11%|█         | 4594/41250 [11:06:38<88:17:32,  8.67s/it]
                                                          
{'loss': 0.2523, 'grad_norm': 1.2622593641281128, 'learning_rate': 3.930968259083951e-05, 'epoch': 1.11}

 11%|█         | 4594/41250 [11:06:38<88:17:32,  8.67s/it][2025-04-25 19:04:22,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:04:22,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.47 | bwd_microstep: 5787.69 | bwd_inner_microstep: 5656.55 | bwd_allreduce_microstep: 131.08 | step_microstep: 18.67
[2025-04-25 19:04:22,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.47 | bwd: 5787.70 | bwd_inner: 5656.55 | bwd_allreduce: 131.11 | step: 18.67

 11%|█         | 4595/41250 [11:06:47<88:23:21,  8.68s/it]
                                                          
{'loss': 0.1424, 'grad_norm': 2.740917205810547, 'learning_rate': 3.9309273521444236e-05, 'epoch': 1.11}

 11%|█         | 4595/41250 [11:06:47<88:23:21,  8.68s/it][2025-04-25 19:04:30,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 19:04:30,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.49 | bwd_microstep: 5759.07 | bwd_inner_microstep: 5713.78 | bwd_allreduce_microstep: 45.25 | step_microstep: 18.93
[2025-04-25 19:04:30,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.49 | bwd: 5759.08 | bwd_inner: 5713.78 | bwd_allreduce: 45.26 | step: 18.93

 11%|█         | 4596/41250 [11:06:56<88:26:17,  8.69s/it]
                                                          
{'loss': 0.3702, 'grad_norm': 3.1296234130859375, 'learning_rate': 3.930886433301109e-05, 'epoch': 1.11}

 11%|█         | 4596/41250 [11:06:56<88:26:17,  8.69s/it][2025-04-25 19:04:39,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 19:04:39,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5774.18 | bwd_inner_microstep: 5700.66 | bwd_allreduce_microstep: 73.47 | step_microstep: 19.04
[2025-04-25 19:04:39,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5774.19 | bwd_inner: 5700.66 | bwd_allreduce: 73.49 | step: 19.05

 11%|█         | 4597/41250 [11:07:04<88:30:19,  8.69s/it]
                                                          
{'loss': 0.1269, 'grad_norm': 1.6055759191513062, 'learning_rate': 3.9308455025542606e-05, 'epoch': 1.11}

 11%|█         | 4597/41250 [11:07:04<88:30:19,  8.69s/it][2025-04-25 19:04:48,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 19:04:48,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.76 | bwd_microstep: 5705.34 | bwd_inner_microstep: 5665.35 | bwd_allreduce_microstep: 39.94 | step_microstep: 18.72
[2025-04-25 19:04:48,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.76 | bwd: 5705.36 | bwd_inner: 5665.35 | bwd_allreduce: 39.96 | step: 18.73

 11%|█         | 4598/41250 [11:07:13<88:18:48,  8.67s/it]
                                                          
{'loss': 0.088, 'grad_norm': 1.1385233402252197, 'learning_rate': 3.930804559904128e-05, 'epoch': 1.11}

 11%|█         | 4598/41250 [11:07:13<88:18:48,  8.67s/it][2025-04-25 19:04:56,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.33 | optimizer_step: 0.94
[2025-04-25 19:04:56,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.46 | bwd_microstep: 5724.67 | bwd_inner_microstep: 5667.18 | bwd_allreduce_microstep: 57.44 | step_microstep: 21.01
[2025-04-25 19:04:56,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.46 | bwd: 5724.68 | bwd_inner: 5667.18 | bwd_allreduce: 57.46 | step: 21.02

 11%|█         | 4599/41250 [11:07:22<88:13:54,  8.67s/it]
                                                          
{'loss': 0.3029, 'grad_norm': 1.9439752101898193, 'learning_rate': 3.930763605350966e-05, 'epoch': 1.11}

 11%|█         | 4599/41250 [11:07:22<88:13:54,  8.67s/it][2025-04-25 19:05:05,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 19:05:05,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.61 | bwd_microstep: 5714.40 | bwd_inner_microstep: 5666.76 | bwd_allreduce_microstep: 47.60 | step_microstep: 18.31
[2025-04-25 19:05:05,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.61 | bwd: 5714.41 | bwd_inner: 5666.76 | bwd_allreduce: 47.61 | step: 18.31

 11%|█         | 4600/41250 [11:07:30<88:07:11,  8.66s/it]
                                                          
{'loss': 0.1051, 'grad_norm': 1.7912379503250122, 'learning_rate': 3.930722638895025e-05, 'epoch': 1.12}

 11%|█         | 4600/41250 [11:07:30<88:07:11,  8.66s/it][2025-04-25 19:05:14,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:05:14,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.20 | bwd_microstep: 5688.22 | bwd_inner_microstep: 5671.68 | bwd_allreduce_microstep: 16.50 | step_microstep: 17.97
[2025-04-25 19:05:14,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.20 | bwd: 5688.23 | bwd_inner: 5671.68 | bwd_allreduce: 16.52 | step: 17.97

 11%|█         | 4601/41250 [11:07:39<87:59:57,  8.64s/it]
                                                          
{'loss': 0.2708, 'grad_norm': 2.329646587371826, 'learning_rate': 3.9306816605365596e-05, 'epoch': 1.12}

 11%|█         | 4601/41250 [11:07:39<87:59:57,  8.64s/it][2025-04-25 19:05:22,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:05:22,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.86 | bwd_microstep: 5772.55 | bwd_inner_microstep: 5713.27 | bwd_allreduce_microstep: 59.24 | step_microstep: 18.56
[2025-04-25 19:05:22,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.86 | bwd: 5772.56 | bwd_inner: 5713.27 | bwd_allreduce: 59.26 | step: 18.57

 11%|█         | 4602/41250 [11:07:48<88:13:26,  8.67s/it]
                                                          
{'loss': 0.1339, 'grad_norm': 1.0368999242782593, 'learning_rate': 3.9306406702758215e-05, 'epoch': 1.12}

 11%|█         | 4602/41250 [11:07:48<88:13:26,  8.67s/it][2025-04-25 19:05:31,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:05:31,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2905.80 | bwd_microstep: 5804.95 | bwd_inner_microstep: 5792.31 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.05
[2025-04-25 19:05:31,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2905.80 | bwd: 5804.96 | bwd_inner: 5792.31 | bwd_allreduce: 12.61 | step: 18.06

 11%|█         | 4603/41250 [11:07:56<88:37:11,  8.71s/it]
                                                          
{'loss': 0.0559, 'grad_norm': 0.9343948364257812, 'learning_rate': 3.9305996681130637e-05, 'epoch': 1.12}

 11%|█         | 4603/41250 [11:07:56<88:37:11,  8.71s/it][2025-04-25 19:05:40,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 19:05:40,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.51 | bwd_microstep: 5795.55 | bwd_inner_microstep: 5670.55 | bwd_allreduce_microstep: 124.95 | step_microstep: 19.02
[2025-04-25 19:05:40,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.51 | bwd: 5795.56 | bwd_inner: 5670.55 | bwd_allreduce: 124.97 | step: 19.02

 11%|█         | 4604/41250 [11:08:05<88:38:44,  8.71s/it]
                                                          
{'loss': 0.1589, 'grad_norm': 1.4977457523345947, 'learning_rate': 3.930558654048538e-05, 'epoch': 1.12}

 11%|█         | 4604/41250 [11:08:05<88:38:44,  8.71s/it][2025-04-25 19:05:49,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:05:49,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.28 | bwd_microstep: 5802.01 | bwd_inner_microstep: 5709.81 | bwd_allreduce_microstep: 92.15 | step_microstep: 18.38
[2025-04-25 19:05:49,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.28 | bwd: 5802.03 | bwd_inner: 5709.81 | bwd_allreduce: 92.17 | step: 18.39

 11%|█         | 4605/41250 [11:08:14<88:44:34,  8.72s/it]
                                                          
{'loss': 0.0441, 'grad_norm': 1.8959487676620483, 'learning_rate': 3.930517628082498e-05, 'epoch': 1.12}

 11%|█         | 4605/41250 [11:08:14<88:44:34,  8.72s/it][2025-04-25 19:05:57,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-25 19:05:57,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.13 | bwd_microstep: 5730.48 | bwd_inner_microstep: 5672.77 | bwd_allreduce_microstep: 57.66 | step_microstep: 19.02
[2025-04-25 19:05:57,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.13 | bwd: 5730.49 | bwd_inner: 5672.77 | bwd_allreduce: 57.68 | step: 19.02

 11%|█         | 4606/41250 [11:08:23<88:30:34,  8.70s/it]
                                                          
{'loss': 0.2896, 'grad_norm': 1.7620083093643188, 'learning_rate': 3.9304765902151964e-05, 'epoch': 1.12}

 11%|█         | 4606/41250 [11:08:23<88:30:34,  8.70s/it][2025-04-25 19:06:06,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 19:06:06,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.79 | bwd_microstep: 5925.45 | bwd_inner_microstep: 5654.86 | bwd_allreduce_microstep: 270.54 | step_microstep: 18.70
[2025-04-25 19:06:06,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.80 | bwd: 5925.47 | bwd_inner: 5654.86 | bwd_allreduce: 270.56 | step: 18.70

 11%|█         | 4607/41250 [11:08:31<88:57:00,  8.74s/it]
                                                          
{'loss': 0.1025, 'grad_norm': 0.8973283171653748, 'learning_rate': 3.930435540446887e-05, 'epoch': 1.12}

 11%|█         | 4607/41250 [11:08:31<88:57:00,  8.74s/it][2025-04-25 19:06:15,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 19:06:15,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.23 | bwd_microstep: 5796.76 | bwd_inner_microstep: 5783.00 | bwd_allreduce_microstep: 13.69 | step_microstep: 19.42
[2025-04-25 19:06:15,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.23 | bwd: 5796.77 | bwd_inner: 5783.00 | bwd_allreduce: 13.72 | step: 19.44

 11%|█         | 4608/41250 [11:08:40<89:03:18,  8.75s/it]
                                                          
{'loss': 0.0736, 'grad_norm': 1.2439167499542236, 'learning_rate': 3.930394478777822e-05, 'epoch': 1.12}

 11%|█         | 4608/41250 [11:08:40<89:03:18,  8.75s/it][2025-04-25 19:06:24,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 19:06:24,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.09 | bwd_microstep: 5722.33 | bwd_inner_microstep: 5709.66 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.02
[2025-04-25 19:06:24,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.09 | bwd: 5722.35 | bwd_inner: 5709.66 | bwd_allreduce: 12.64 | step: 19.02

 11%|█         | 4609/41250 [11:08:49<88:47:17,  8.72s/it]
                                                          
{'loss': 0.2007, 'grad_norm': 1.7923405170440674, 'learning_rate': 3.930353405208254e-05, 'epoch': 1.12}

 11%|█         | 4609/41250 [11:08:49<88:47:17,  8.72s/it][2025-04-25 19:06:32,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.33 | optimizer_step: 1.03
[2025-04-25 19:06:32,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.05 | bwd_microstep: 5751.39 | bwd_inner_microstep: 5711.59 | bwd_allreduce_microstep: 39.74 | step_microstep: 19.86
[2025-04-25 19:06:32,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.05 | bwd: 5751.40 | bwd_inner: 5711.58 | bwd_allreduce: 39.77 | step: 19.86

 11%|█         | 4610/41250 [11:08:58<88:43:50,  8.72s/it]
                                                          
{'loss': 0.2066, 'grad_norm': 1.4302186965942383, 'learning_rate': 3.9303123197384375e-05, 'epoch': 1.12}

 11%|█         | 4610/41250 [11:08:58<88:43:50,  8.72s/it][2025-04-25 19:06:41,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 19:06:41,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5717.49 | bwd_inner_microstep: 5704.52 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.11
[2025-04-25 19:06:41,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5717.50 | bwd_inner: 5704.52 | bwd_allreduce: 12.94 | step: 19.11

 11%|█         | 4611/41250 [11:09:06<88:31:55,  8.70s/it]
                                                          
{'loss': 0.2966, 'grad_norm': 2.8984038829803467, 'learning_rate': 3.930271222368625e-05, 'epoch': 1.12}

 11%|█         | 4611/41250 [11:09:06<88:31:55,  8.70s/it][2025-04-25 19:06:50,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:06:50,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.42 | bwd_microstep: 5758.19 | bwd_inner_microstep: 5678.84 | bwd_allreduce_microstep: 79.30 | step_microstep: 19.10
[2025-04-25 19:06:50,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.42 | bwd: 5758.20 | bwd_inner: 5678.84 | bwd_allreduce: 79.32 | step: 19.10

 11%|█         | 4612/41250 [11:09:15<88:29:10,  8.69s/it]
                                                          
{'loss': 0.1879, 'grad_norm': 2.2638251781463623, 'learning_rate': 3.9302301130990705e-05, 'epoch': 1.12}

 11%|█         | 4612/41250 [11:09:15<88:29:10,  8.69s/it][2025-04-25 19:06:58,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:06:58,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.90 | bwd_microstep: 5761.66 | bwd_inner_microstep: 5659.24 | bwd_allreduce_microstep: 102.38 | step_microstep: 18.84
[2025-04-25 19:06:58,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.90 | bwd: 5761.68 | bwd_inner: 5659.24 | bwd_allreduce: 102.40 | step: 18.85

 11%|█         | 4613/41250 [11:09:24<88:25:55,  8.69s/it]
                                                          
{'loss': 0.273, 'grad_norm': 1.7546948194503784, 'learning_rate': 3.930188991930027e-05, 'epoch': 1.12}

 11%|█         | 4613/41250 [11:09:24<88:25:55,  8.69s/it][2025-04-25 19:07:07,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 19:07:07,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.50 | bwd_microstep: 5793.31 | bwd_inner_microstep: 5658.84 | bwd_allreduce_microstep: 134.42 | step_microstep: 19.40
[2025-04-25 19:07:07,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.50 | bwd: 5793.32 | bwd_inner: 5658.84 | bwd_allreduce: 134.44 | step: 19.40

 11%|█         | 4614/41250 [11:09:32<88:28:22,  8.69s/it]
                                                          
{'loss': 0.0459, 'grad_norm': 0.6387472152709961, 'learning_rate': 3.930147858861747e-05, 'epoch': 1.12}

 11%|█         | 4614/41250 [11:09:32<88:28:22,  8.69s/it][2025-04-25 19:07:16,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:07:16,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.27 | bwd_microstep: 5706.09 | bwd_inner_microstep: 5643.75 | bwd_allreduce_microstep: 62.29 | step_microstep: 18.85
[2025-04-25 19:07:16,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.27 | bwd: 5706.10 | bwd_inner: 5643.75 | bwd_allreduce: 62.31 | step: 18.85

 11%|█         | 4615/41250 [11:09:41<88:14:33,  8.67s/it]
                                                          
{'loss': 0.1399, 'grad_norm': 1.0723421573638916, 'learning_rate': 3.9301067138944864e-05, 'epoch': 1.12}

 11%|█         | 4615/41250 [11:09:41<88:14:33,  8.67s/it][2025-04-25 19:07:24,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.11 | optimizer_step: 1.03
[2025-04-25 19:07:24,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.98 | bwd_microstep: 5784.01 | bwd_inner_microstep: 5647.78 | bwd_allreduce_microstep: 136.16 | step_microstep: 19.67
[2025-04-25 19:07:24,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.98 | bwd: 5784.03 | bwd_inner: 5647.78 | bwd_allreduce: 136.20 | step: 19.68

 11%|█         | 4616/41250 [11:09:50<88:18:51,  8.68s/it]
                                                          
{'loss': 0.178, 'grad_norm': 1.1782130002975464, 'learning_rate': 3.930065557028497e-05, 'epoch': 1.12}

 11%|█         | 4616/41250 [11:09:50<88:18:51,  8.68s/it][2025-04-25 19:07:33,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:07:33,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.89 | bwd_microstep: 5751.16 | bwd_inner_microstep: 5702.65 | bwd_allreduce_microstep: 48.47 | step_microstep: 18.20
[2025-04-25 19:07:33,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.89 | bwd: 5751.17 | bwd_inner: 5702.65 | bwd_allreduce: 48.48 | step: 18.20

 11%|█         | 4617/41250 [11:09:58<88:20:44,  8.68s/it]
                                                          
{'loss': 0.0744, 'grad_norm': 1.0950591564178467, 'learning_rate': 3.9300243882640326e-05, 'epoch': 1.12}

 11%|█         | 4617/41250 [11:09:58<88:20:44,  8.68s/it][2025-04-25 19:07:42,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:07:42,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.53 | bwd_microstep: 5762.65 | bwd_inner_microstep: 5647.84 | bwd_allreduce_microstep: 114.77 | step_microstep: 17.92
[2025-04-25 19:07:42,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.53 | bwd: 5762.66 | bwd_inner: 5647.84 | bwd_allreduce: 114.78 | step: 17.93

 11%|█         | 4618/41250 [11:10:07<88:21:57,  8.68s/it]
                                                          
{'loss': 0.1051, 'grad_norm': 0.9404296278953552, 'learning_rate': 3.929983207601348e-05, 'epoch': 1.12}

 11%|█         | 4618/41250 [11:10:07<88:21:57,  8.68s/it][2025-04-25 19:07:50,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 19:07:50,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5706.75 | bwd_inner_microstep: 5643.62 | bwd_allreduce_microstep: 63.09 | step_microstep: 18.70
[2025-04-25 19:07:50,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5706.76 | bwd_inner: 5643.62 | bwd_allreduce: 63.11 | step: 18.71

 11%|█         | 4619/41250 [11:10:16<88:10:23,  8.67s/it]
                                                          
{'loss': 0.1121, 'grad_norm': 0.7978342771530151, 'learning_rate': 3.929942015040697e-05, 'epoch': 1.12}

 11%|█         | 4619/41250 [11:10:16<88:10:23,  8.67s/it][2025-04-25 19:07:59,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:07:59,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.50 | bwd_microstep: 5776.47 | bwd_inner_microstep: 5649.43 | bwd_allreduce_microstep: 127.00 | step_microstep: 18.87
[2025-04-25 19:07:59,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.50 | bwd: 5776.49 | bwd_inner: 5649.43 | bwd_allreduce: 127.01 | step: 18.87

 11%|█         | 4620/41250 [11:10:24<88:16:02,  8.67s/it]
                                                          
{'loss': 0.2065, 'grad_norm': 2.305922031402588, 'learning_rate': 3.929900810582332e-05, 'epoch': 1.12}

 11%|█         | 4620/41250 [11:10:24<88:16:02,  8.67s/it][2025-04-25 19:08:08,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 19:08:08,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.48 | bwd_microstep: 5689.70 | bwd_inner_microstep: 5661.14 | bwd_allreduce_microstep: 28.52 | step_microstep: 18.33
[2025-04-25 19:08:08,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.48 | bwd: 5689.72 | bwd_inner: 5661.14 | bwd_allreduce: 28.54 | step: 18.33

 11%|█         | 4621/41250 [11:10:33<88:03:24,  8.65s/it]
                                                          
{'loss': 0.1135, 'grad_norm': 2.8591206073760986, 'learning_rate': 3.929859594226509e-05, 'epoch': 1.12}

 11%|█         | 4621/41250 [11:10:33<88:03:24,  8.65s/it][2025-04-25 19:08:16,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 19:08:16,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.19 | bwd_microstep: 5680.32 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 30.14 | step_microstep: 19.61
[2025-04-25 19:08:16,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.19 | bwd: 5680.33 | bwd_inner: 5650.14 | bwd_allreduce: 30.16 | step: 19.62

 11%|█         | 4622/41250 [11:10:42<87:55:20,  8.64s/it]
                                                          
{'loss': 0.1458, 'grad_norm': 2.3223838806152344, 'learning_rate': 3.92981836597348e-05, 'epoch': 1.12}

 11%|█         | 4622/41250 [11:10:42<87:55:20,  8.64s/it][2025-04-25 19:08:25,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:08:25,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.46 | bwd_microstep: 5716.34 | bwd_inner_microstep: 5700.47 | bwd_allreduce_microstep: 15.83 | step_microstep: 18.58
[2025-04-25 19:08:25,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.46 | bwd: 5716.36 | bwd_inner: 5700.47 | bwd_allreduce: 15.85 | step: 18.58

 11%|█         | 4623/41250 [11:10:50<87:58:43,  8.65s/it]
                                                          
{'loss': 0.1919, 'grad_norm': 2.426478862762451, 'learning_rate': 3.929777125823502e-05, 'epoch': 1.12}

 11%|█         | 4623/41250 [11:10:50<87:58:43,  8.65s/it][2025-04-25 19:08:34,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.16 | optimizer_step: 1.05
[2025-04-25 19:08:34,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.30 | bwd_microstep: 5746.79 | bwd_inner_microstep: 5650.64 | bwd_allreduce_microstep: 96.09 | step_microstep: 19.49
[2025-04-25 19:08:34,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.30 | bwd: 5746.81 | bwd_inner: 5650.64 | bwd_allreduce: 96.12 | step: 19.50

 11%|█         | 4624/41250 [11:10:59<88:01:04,  8.65s/it]
                                                          
{'loss': 0.3352, 'grad_norm': 2.637467622756958, 'learning_rate': 3.9297358737768265e-05, 'epoch': 1.12}

 11%|█         | 4624/41250 [11:10:59<88:01:04,  8.65s/it][2025-04-25 19:08:42,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:08:42,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5739.74 | bwd_inner_microstep: 5683.85 | bwd_allreduce_microstep: 55.85 | step_microstep: 18.31
[2025-04-25 19:08:42,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5739.76 | bwd_inner: 5683.85 | bwd_allreduce: 55.87 | step: 18.31

 11%|█         | 4625/41250 [11:11:08<88:04:34,  8.66s/it]
                                                          
{'loss': 0.0641, 'grad_norm': 1.4774078130722046, 'learning_rate': 3.929694609833709e-05, 'epoch': 1.12}

 11%|█         | 4625/41250 [11:11:08<88:04:34,  8.66s/it][2025-04-25 19:08:51,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 19:08:51,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5709.20 | bwd_inner_microstep: 5696.59 | bwd_allreduce_microstep: 12.56 | step_microstep: 17.76
[2025-04-25 19:08:51,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5709.21 | bwd_inner: 5696.59 | bwd_allreduce: 12.58 | step: 17.76

 11%|█         | 4626/41250 [11:11:16<88:02:51,  8.65s/it]
                                                          
{'loss': 0.1073, 'grad_norm': 1.5386073589324951, 'learning_rate': 3.929653333994404e-05, 'epoch': 1.12}

 11%|█         | 4626/41250 [11:11:16<88:02:51,  8.65s/it][2025-04-25 19:09:00,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:09:00,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.86 | bwd_microstep: 5770.32 | bwd_inner_microstep: 5656.89 | bwd_allreduce_microstep: 113.38 | step_microstep: 18.55
[2025-04-25 19:09:00,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.86 | bwd: 5770.33 | bwd_inner: 5656.89 | bwd_allreduce: 113.40 | step: 18.56

 11%|█         | 4627/41250 [11:11:25<88:07:58,  8.66s/it]
                                                          
{'loss': 0.2121, 'grad_norm': 1.5341253280639648, 'learning_rate': 3.9296120462591655e-05, 'epoch': 1.12}

 11%|█         | 4627/41250 [11:11:25<88:07:58,  8.66s/it][2025-04-25 19:09:08,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:09:08,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.20 | bwd_microstep: 5995.86 | bwd_inner_microstep: 5681.38 | bwd_allreduce_microstep: 314.43 | step_microstep: 18.55
[2025-04-25 19:09:08,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.20 | bwd: 5995.88 | bwd_inner: 5681.38 | bwd_allreduce: 314.45 | step: 18.55

 11%|█         | 4628/41250 [11:11:34<88:57:53,  8.75s/it]
                                                          
{'loss': 0.1814, 'grad_norm': 1.2128883600234985, 'learning_rate': 3.9295707466282486e-05, 'epoch': 1.12}

 11%|█         | 4628/41250 [11:11:34<88:57:53,  8.75s/it][2025-04-25 19:09:17,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:09:17,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.11 | bwd_microstep: 5688.41 | bwd_inner_microstep: 5650.06 | bwd_allreduce_microstep: 38.29 | step_microstep: 18.47
[2025-04-25 19:09:17,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.11 | bwd: 5688.42 | bwd_inner: 5650.06 | bwd_allreduce: 38.31 | step: 18.48

 11%|█         | 4629/41250 [11:11:42<88:32:39,  8.70s/it]
                                                          
{'loss': 0.2956, 'grad_norm': 1.8438291549682617, 'learning_rate': 3.929529435101907e-05, 'epoch': 1.12}

 11%|█         | 4629/41250 [11:11:42<88:32:39,  8.70s/it][2025-04-25 19:09:26,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:09:26,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5697.15 | bwd_inner_microstep: 5684.44 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.17
[2025-04-25 19:09:26,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5697.16 | bwd_inner: 5684.44 | bwd_allreduce: 12.68 | step: 18.17

 11%|█         | 4630/41250 [11:11:51<88:19:23,  8.68s/it]
                                                          
{'loss': 0.1402, 'grad_norm': 1.2220367193222046, 'learning_rate': 3.929488111680396e-05, 'epoch': 1.12}

 11%|█         | 4630/41250 [11:11:51<88:19:23,  8.68s/it][2025-04-25 19:09:34,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:09:34,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5695.78 | bwd_inner_microstep: 5647.26 | bwd_allreduce_microstep: 48.48 | step_microstep: 18.41
[2025-04-25 19:09:34,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5695.79 | bwd_inner: 5647.26 | bwd_allreduce: 48.50 | step: 18.41

 11%|█         | 4631/41250 [11:12:00<88:06:01,  8.66s/it]
                                                          
{'loss': 0.091, 'grad_norm': 0.6347318887710571, 'learning_rate': 3.929446776363971e-05, 'epoch': 1.12}

 11%|█         | 4631/41250 [11:12:00<88:06:01,  8.66s/it][2025-04-25 19:09:43,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:09:43,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.24 | bwd_microstep: 5742.97 | bwd_inner_microstep: 5658.19 | bwd_allreduce_microstep: 84.73 | step_microstep: 18.58
[2025-04-25 19:09:43,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.24 | bwd: 5742.98 | bwd_inner: 5658.19 | bwd_allreduce: 84.75 | step: 18.58

 11%|█         | 4632/41250 [11:12:08<88:05:32,  8.66s/it]
                                                          
{'loss': 0.2822, 'grad_norm': 2.350827217102051, 'learning_rate': 3.929405429152886e-05, 'epoch': 1.12}

 11%|█         | 4632/41250 [11:12:08<88:05:32,  8.66s/it][2025-04-25 19:09:52,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:09:52,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.41 | bwd_microstep: 5865.87 | bwd_inner_microstep: 5658.63 | bwd_allreduce_microstep: 207.20 | step_microstep: 18.27
[2025-04-25 19:09:52,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.41 | bwd: 5865.88 | bwd_inner: 5658.62 | bwd_allreduce: 207.22 | step: 18.28

 11%|█         | 4633/41250 [11:12:17<88:27:54,  8.70s/it]
                                                          
{'loss': 0.0711, 'grad_norm': 0.7345250248908997, 'learning_rate': 3.9293640700473956e-05, 'epoch': 1.12}

 11%|█         | 4633/41250 [11:12:17<88:27:54,  8.70s/it][2025-04-25 19:10:00,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-25 19:10:00,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.45 | bwd_microstep: 5791.39 | bwd_inner_microstep: 5652.84 | bwd_allreduce_microstep: 138.50 | step_microstep: 19.44
[2025-04-25 19:10:00,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.45 | bwd: 5791.41 | bwd_inner: 5652.84 | bwd_allreduce: 138.53 | step: 19.44

 11%|█         | 4634/41250 [11:12:26<88:28:09,  8.70s/it]
                                                          
{'loss': 0.1859, 'grad_norm': 2.134308338165283, 'learning_rate': 3.929322699047755e-05, 'epoch': 1.12}

 11%|█         | 4634/41250 [11:12:26<88:28:09,  8.70s/it][2025-04-25 19:10:09,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:10:09,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.06 | bwd_microstep: 5702.57 | bwd_inner_microstep: 5659.37 | bwd_allreduce_microstep: 43.16 | step_microstep: 18.55
[2025-04-25 19:10:09,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.06 | bwd: 5702.58 | bwd_inner: 5659.37 | bwd_allreduce: 43.17 | step: 18.56

 11%|█         | 4635/41250 [11:12:34<88:14:01,  8.68s/it]
                                                          
{'loss': 0.0527, 'grad_norm': 0.35782182216644287, 'learning_rate': 3.92928131615422e-05, 'epoch': 1.12}

 11%|█         | 4635/41250 [11:12:34<88:14:01,  8.68s/it][2025-04-25 19:10:18,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.24 | optimizer_step: 0.94
[2025-04-25 19:10:18,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.49 | bwd_microstep: 5707.52 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 59.01 | step_microstep: 19.31
[2025-04-25 19:10:18,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.50 | bwd: 5707.53 | bwd_inner: 5648.45 | bwd_allreduce: 59.03 | step: 19.32

 11%|█         | 4636/41250 [11:12:43<88:04:22,  8.66s/it]
                                                          
{'loss': 0.2619, 'grad_norm': 2.636866331100464, 'learning_rate': 3.929239921367045e-05, 'epoch': 1.12}

 11%|█         | 4636/41250 [11:12:43<88:04:22,  8.66s/it][2025-04-25 19:10:26,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 19:10:26,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5776.95 | bwd_inner_microstep: 5655.19 | bwd_allreduce_microstep: 121.72 | step_microstep: 18.87
[2025-04-25 19:10:26,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5776.96 | bwd_inner: 5655.19 | bwd_allreduce: 121.74 | step: 18.88

 11%|█         | 4637/41250 [11:12:52<88:10:12,  8.67s/it]
                                                          
{'loss': 0.0876, 'grad_norm': 0.9262834787368774, 'learning_rate': 3.9291985146864856e-05, 'epoch': 1.12}

 11%|█         | 4637/41250 [11:12:52<88:10:12,  8.67s/it][2025-04-25 19:10:35,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 19:10:35,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5721.42 | bwd_inner_microstep: 5707.96 | bwd_allreduce_microstep: 13.41 | step_microstep: 18.86
[2025-04-25 19:10:35,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5721.43 | bwd_inner: 5707.96 | bwd_allreduce: 13.43 | step: 18.87

 11%|█         | 4638/41250 [11:13:00<88:08:42,  8.67s/it]
                                                          
{'loss': 0.1617, 'grad_norm': 1.1464555263519287, 'learning_rate': 3.929157096112796e-05, 'epoch': 1.12}

 11%|█         | 4638/41250 [11:13:00<88:08:42,  8.67s/it][2025-04-25 19:10:44,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 19:10:44,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.37 | bwd_microstep: 5919.91 | bwd_inner_microstep: 5711.12 | bwd_allreduce_microstep: 208.62 | step_microstep: 20.21
[2025-04-25 19:10:44,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.37 | bwd: 5919.97 | bwd_inner: 5711.12 | bwd_allreduce: 208.71 | step: 20.20

 11%|█         | 4639/41250 [11:13:09<88:43:09,  8.72s/it]
                                                          
{'loss': 0.1419, 'grad_norm': 1.9256068468093872, 'learning_rate': 3.929115665646233e-05, 'epoch': 1.12}

 11%|█         | 4639/41250 [11:13:09<88:43:09,  8.72s/it][2025-04-25 19:10:53,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:10:53,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.12 | bwd_microstep: 5706.70 | bwd_inner_microstep: 5671.90 | bwd_allreduce_microstep: 34.76 | step_microstep: 18.30
[2025-04-25 19:10:53,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.12 | bwd: 5706.72 | bwd_inner: 5671.90 | bwd_allreduce: 34.78 | step: 18.30

 11%|█         | 4640/41250 [11:13:18<88:23:54,  8.69s/it]
                                                          
{'loss': 0.1358, 'grad_norm': 2.017378807067871, 'learning_rate': 3.9290742232870514e-05, 'epoch': 1.12}

 11%|█         | 4640/41250 [11:13:18<88:23:54,  8.69s/it][2025-04-25 19:11:01,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-25 19:11:01,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.61 | bwd_microstep: 5719.80 | bwd_inner_microstep: 5664.53 | bwd_allreduce_microstep: 55.21 | step_microstep: 18.22
[2025-04-25 19:11:01,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.61 | bwd: 5719.81 | bwd_inner: 5664.53 | bwd_allreduce: 55.24 | step: 18.22

 11%|█▏        | 4641/41250 [11:13:26<88:12:47,  8.67s/it]
                                                          
{'loss': 0.2068, 'grad_norm': 1.1889952421188354, 'learning_rate': 3.929032769035507e-05, 'epoch': 1.13}

 11%|█▏        | 4641/41250 [11:13:26<88:12:47,  8.67s/it][2025-04-25 19:11:10,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 19:11:10,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.51 | bwd_microstep: 5747.50 | bwd_inner_microstep: 5709.14 | bwd_allreduce_microstep: 38.32 | step_microstep: 18.79
[2025-04-25 19:11:10,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.51 | bwd: 5747.52 | bwd_inner: 5709.14 | bwd_allreduce: 38.34 | step: 18.79

 11%|█▏        | 4642/41250 [11:13:35<88:18:17,  8.68s/it]
                                                          
{'loss': 0.082, 'grad_norm': 1.1911170482635498, 'learning_rate': 3.928991302891854e-05, 'epoch': 1.13}

 11%|█▏        | 4642/41250 [11:13:35<88:18:17,  8.68s/it][2025-04-25 19:11:19,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.22 | optimizer_step: 1.02
[2025-04-25 19:11:19,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.91 | bwd_microstep: 5783.29 | bwd_inner_microstep: 5669.82 | bwd_allreduce_microstep: 113.41 | step_microstep: 19.62
[2025-04-25 19:11:19,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.91 | bwd: 5783.31 | bwd_inner: 5669.82 | bwd_allreduce: 113.44 | step: 19.62

 11%|█▏        | 4643/41250 [11:13:44<88:21:27,  8.69s/it]
                                                          
{'loss': 0.0595, 'grad_norm': 0.6864256262779236, 'learning_rate': 3.92894982485635e-05, 'epoch': 1.13}

 11%|█▏        | 4643/41250 [11:13:44<88:21:27,  8.69s/it][2025-04-25 19:11:27,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 19:11:27,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.35 | bwd_microstep: 5708.76 | bwd_inner_microstep: 5672.79 | bwd_allreduce_microstep: 35.92 | step_microstep: 18.64
[2025-04-25 19:11:27,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.35 | bwd: 5708.77 | bwd_inner: 5672.79 | bwd_allreduce: 35.94 | step: 18.64

 11%|█▏        | 4644/41250 [11:13:53<88:13:20,  8.68s/it]
                                                          
{'loss': 0.1135, 'grad_norm': 0.820340096950531, 'learning_rate': 3.928908334929249e-05, 'epoch': 1.13}

 11%|█▏        | 4644/41250 [11:13:53<88:13:20,  8.68s/it][2025-04-25 19:11:36,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:11:36,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.69 | bwd_microstep: 5719.30 | bwd_inner_microstep: 5663.95 | bwd_allreduce_microstep: 55.31 | step_microstep: 19.15
[2025-04-25 19:11:36,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.69 | bwd: 5719.32 | bwd_inner: 5663.95 | bwd_allreduce: 55.32 | step: 19.15

 11%|█▏        | 4645/41250 [11:14:01<88:06:36,  8.67s/it]
                                                          
{'loss': 0.1886, 'grad_norm': 1.5512456893920898, 'learning_rate': 3.9288668331108074e-05, 'epoch': 1.13}

 11%|█▏        | 4645/41250 [11:14:01<88:06:36,  8.67s/it][2025-04-25 19:11:45,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 19:11:45,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.38 | bwd_microstep: 5755.23 | bwd_inner_microstep: 5692.33 | bwd_allreduce_microstep: 62.86 | step_microstep: 18.64
[2025-04-25 19:11:45,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.38 | bwd: 5755.25 | bwd_inner: 5692.33 | bwd_allreduce: 62.88 | step: 18.64

 11%|█▏        | 4646/41250 [11:14:10<88:12:00,  8.67s/it]
                                                          
{'loss': 0.148, 'grad_norm': 1.3895223140716553, 'learning_rate': 3.9288253194012815e-05, 'epoch': 1.13}

 11%|█▏        | 4646/41250 [11:14:10<88:12:00,  8.67s/it][2025-04-25 19:11:53,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:11:53,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.41 | bwd_microstep: 5792.40 | bwd_inner_microstep: 5676.98 | bwd_allreduce_microstep: 115.38 | step_microstep: 18.75
[2025-04-25 19:11:53,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.41 | bwd: 5792.42 | bwd_inner: 5676.98 | bwd_allreduce: 115.40 | step: 18.75

 11%|█▏        | 4647/41250 [11:14:19<88:18:03,  8.68s/it]
                                                          
{'loss': 0.0967, 'grad_norm': 0.7746520638465881, 'learning_rate': 3.928783793800927e-05, 'epoch': 1.13}

 11%|█▏        | 4647/41250 [11:14:19<88:18:03,  8.68s/it][2025-04-25 19:12:02,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:12:02,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5898.77 | bwd_inner_microstep: 5712.50 | bwd_allreduce_microstep: 186.23 | step_microstep: 18.74
[2025-04-25 19:12:02,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5898.79 | bwd_inner: 5712.50 | bwd_allreduce: 186.25 | step: 18.74

 11%|█▏        | 4648/41250 [11:14:27<88:45:04,  8.73s/it]
                                                          
{'loss': 0.0379, 'grad_norm': 0.3687887489795685, 'learning_rate': 3.92874225631e-05, 'epoch': 1.13}

 11%|█▏        | 4648/41250 [11:14:27<88:45:04,  8.73s/it][2025-04-25 19:12:11,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:12:11,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5770.27 | bwd_inner_microstep: 5696.54 | bwd_allreduce_microstep: 73.68 | step_microstep: 18.68
[2025-04-25 19:12:11,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5770.29 | bwd_inner: 5696.54 | bwd_allreduce: 73.70 | step: 18.68

 11%|█▏        | 4649/41250 [11:14:36<88:40:42,  8.72s/it]
                                                          
{'loss': 0.281, 'grad_norm': 2.3898613452911377, 'learning_rate': 3.928700706928756e-05, 'epoch': 1.13}

 11%|█▏        | 4649/41250 [11:14:36<88:40:42,  8.72s/it][2025-04-25 19:12:20,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.24 | optimizer_step: 0.98
[2025-04-25 19:12:20,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5808.42 | bwd_inner_microstep: 5659.77 | bwd_allreduce_microstep: 148.59 | step_microstep: 19.51
[2025-04-25 19:12:20,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5808.43 | bwd_inner: 5659.77 | bwd_allreduce: 148.61 | step: 19.51

 11%|█▏        | 4650/41250 [11:14:45<88:41:13,  8.72s/it]
                                                          
{'loss': 0.3318, 'grad_norm': 2.670694589614868, 'learning_rate': 3.9286591456574516e-05, 'epoch': 1.13}

 11%|█▏        | 4650/41250 [11:14:45<88:41:13,  8.72s/it][2025-04-25 19:12:28,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 19:12:28,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.68 | bwd_microstep: 5718.90 | bwd_inner_microstep: 5702.92 | bwd_allreduce_microstep: 15.94 | step_microstep: 18.97
[2025-04-25 19:12:28,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.68 | bwd: 5718.92 | bwd_inner: 5702.92 | bwd_allreduce: 15.96 | step: 18.97

 11%|█▏        | 4651/41250 [11:14:53<88:28:39,  8.70s/it]
                                                          
{'loss': 0.051, 'grad_norm': 0.660612165927887, 'learning_rate': 3.928617572496342e-05, 'epoch': 1.13}

 11%|█▏        | 4651/41250 [11:14:53<88:28:39,  8.70s/it][2025-04-25 19:12:37,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 19:12:37,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.95 | bwd_microstep: 5790.45 | bwd_inner_microstep: 5667.67 | bwd_allreduce_microstep: 122.73 | step_microstep: 19.16
[2025-04-25 19:12:37,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.95 | bwd: 5790.47 | bwd_inner: 5667.67 | bwd_allreduce: 122.76 | step: 19.16

 11%|█▏        | 4652/41250 [11:15:02<88:28:53,  8.70s/it]
                                                          
{'loss': 0.0964, 'grad_norm': 1.759534239768982, 'learning_rate': 3.928575987445686e-05, 'epoch': 1.13}

 11%|█▏        | 4652/41250 [11:15:02<88:28:53,  8.70s/it][2025-04-25 19:12:46,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 19:12:46,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.16 | bwd_microstep: 5727.95 | bwd_inner_microstep: 5714.81 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.85
[2025-04-25 19:12:46,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.16 | bwd: 5727.96 | bwd_inner: 5714.81 | bwd_allreduce: 13.10 | step: 18.85

 11%|█▏        | 4653/41250 [11:15:11<88:21:43,  8.69s/it]
                                                          
{'loss': 0.0418, 'grad_norm': 0.42324793338775635, 'learning_rate': 3.928534390505737e-05, 'epoch': 1.13}

 11%|█▏        | 4653/41250 [11:15:11<88:21:43,  8.69s/it][2025-04-25 19:12:54,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:12:54,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.90 | bwd_microstep: 5790.38 | bwd_inner_microstep: 5700.33 | bwd_allreduce_microstep: 89.99 | step_microstep: 18.76
[2025-04-25 19:12:54,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.90 | bwd: 5790.39 | bwd_inner: 5700.33 | bwd_allreduce: 90.01 | step: 18.76

 11%|█▏        | 4654/41250 [11:15:20<88:28:10,  8.70s/it]
                                                          
{'loss': 0.2243, 'grad_norm': 1.7683931589126587, 'learning_rate': 3.928492781676754e-05, 'epoch': 1.13}

 11%|█▏        | 4654/41250 [11:15:20<88:28:10,  8.70s/it][2025-04-25 19:13:03,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 19:13:03,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.27 | bwd_microstep: 5778.38 | bwd_inner_microstep: 5690.99 | bwd_allreduce_microstep: 87.35 | step_microstep: 18.78
[2025-04-25 19:13:03,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.27 | bwd: 5778.40 | bwd_inner: 5690.99 | bwd_allreduce: 87.37 | step: 18.78

 11%|█▏        | 4655/41250 [11:15:28<88:29:17,  8.70s/it]
                                                          
{'loss': 0.4991, 'grad_norm': 2.7528560161590576, 'learning_rate': 3.928451160958991e-05, 'epoch': 1.13}

 11%|█▏        | 4655/41250 [11:15:28<88:29:17,  8.70s/it][2025-04-25 19:13:12,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:13:12,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5783.36 | bwd_inner_microstep: 5694.76 | bwd_allreduce_microstep: 88.54 | step_microstep: 18.65
[2025-04-25 19:13:12,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5783.37 | bwd_inner: 5694.76 | bwd_allreduce: 88.56 | step: 18.65

 11%|█▏        | 4656/41250 [11:15:37<88:32:17,  8.71s/it]
                                                          
{'loss': 0.1195, 'grad_norm': 1.9477767944335938, 'learning_rate': 3.928409528352707e-05, 'epoch': 1.13}

 11%|█▏        | 4656/41250 [11:15:37<88:32:17,  8.71s/it][2025-04-25 19:13:20,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 19:13:20,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.37 | bwd_microstep: 5787.75 | bwd_inner_microstep: 5667.08 | bwd_allreduce_microstep: 120.62 | step_microstep: 19.09
[2025-04-25 19:13:20,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.37 | bwd: 5787.76 | bwd_inner: 5667.08 | bwd_allreduce: 120.64 | step: 19.10

 11%|█▏        | 4657/41250 [11:15:46<88:32:24,  8.71s/it]
                                                          
{'loss': 0.1211, 'grad_norm': 3.1814639568328857, 'learning_rate': 3.9283678838581576e-05, 'epoch': 1.13}

 11%|█▏        | 4657/41250 [11:15:46<88:32:24,  8.71s/it][2025-04-25 19:13:29,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 19:13:29,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.20 | bwd_microstep: 5800.87 | bwd_inner_microstep: 5658.19 | bwd_allreduce_microstep: 142.63 | step_microstep: 19.06
[2025-04-25 19:13:29,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.20 | bwd: 5800.88 | bwd_inner: 5658.19 | bwd_allreduce: 142.65 | step: 19.06

 11%|█▏        | 4658/41250 [11:15:54<88:33:43,  8.71s/it]
                                                          
{'loss': 0.1909, 'grad_norm': 2.323002338409424, 'learning_rate': 3.928326227475599e-05, 'epoch': 1.13}

 11%|█▏        | 4658/41250 [11:15:54<88:33:43,  8.71s/it][2025-04-25 19:13:38,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:13:38,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5708.93 | bwd_inner_microstep: 5695.69 | bwd_allreduce_microstep: 13.19 | step_microstep: 18.91
[2025-04-25 19:13:38,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5708.94 | bwd_inner: 5695.69 | bwd_allreduce: 13.21 | step: 18.91

 11%|█▏        | 4659/41250 [11:16:03<88:20:17,  8.69s/it]
                                                          
{'loss': 0.1074, 'grad_norm': 1.444057822227478, 'learning_rate': 3.928284559205288e-05, 'epoch': 1.13}

 11%|█▏        | 4659/41250 [11:16:03<88:20:17,  8.69s/it][2025-04-25 19:13:46,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 19:13:46,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.08 | bwd_microstep: 5711.74 | bwd_inner_microstep: 5698.95 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.07
[2025-04-25 19:13:46,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.08 | bwd: 5711.75 | bwd_inner: 5698.95 | bwd_allreduce: 12.76 | step: 19.07

 11%|█▏        | 4660/41250 [11:16:12<88:10:56,  8.68s/it]
                                                          
{'loss': 0.063, 'grad_norm': 1.1147665977478027, 'learning_rate': 3.9282428790474835e-05, 'epoch': 1.13}

 11%|█▏        | 4660/41250 [11:16:12<88:10:56,  8.68s/it][2025-04-25 19:13:55,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:13:55,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.36 | bwd_microstep: 5714.06 | bwd_inner_microstep: 5656.46 | bwd_allreduce_microstep: 57.57 | step_microstep: 17.97
[2025-04-25 19:13:55,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.36 | bwd: 5714.07 | bwd_inner: 5656.46 | bwd_allreduce: 57.58 | step: 17.98

 11%|█▏        | 4661/41250 [11:16:20<88:02:01,  8.66s/it]
                                                          
{'loss': 0.1138, 'grad_norm': 1.0744119882583618, 'learning_rate': 3.9282011870024394e-05, 'epoch': 1.13}

 11%|█▏        | 4661/41250 [11:16:20<88:02:01,  8.66s/it][2025-04-25 19:14:04,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.05 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:14:04,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5770.93 | bwd_inner_microstep: 5693.08 | bwd_allreduce_microstep: 77.81 | step_microstep: 19.63
[2025-04-25 19:14:04,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5770.94 | bwd_inner: 5693.08 | bwd_allreduce: 77.82 | step: 19.63

 11%|█▏        | 4662/41250 [11:16:29<88:09:41,  8.67s/it]
                                                          
{'loss': 0.1716, 'grad_norm': 1.2245436906814575, 'learning_rate': 3.928159483070415e-05, 'epoch': 1.13}

 11%|█▏        | 4662/41250 [11:16:29<88:09:41,  8.67s/it][2025-04-25 19:14:13,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 1.01
[2025-04-25 19:14:13,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.45 | bwd_microstep: 5909.79 | bwd_inner_microstep: 5654.62 | bwd_allreduce_microstep: 255.14 | step_microstep: 18.56
[2025-04-25 19:14:13,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.45 | bwd: 5909.81 | bwd_inner: 5654.62 | bwd_allreduce: 255.15 | step: 18.56

 11%|█▏        | 4663/41250 [11:16:38<88:37:30,  8.72s/it]
                                                          
{'loss': 0.1044, 'grad_norm': 1.2515987157821655, 'learning_rate': 3.9281177672516665e-05, 'epoch': 1.13}

 11%|█▏        | 4663/41250 [11:16:38<88:37:30,  8.72s/it][2025-04-25 19:14:21,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:14:21,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5766.92 | bwd_inner_microstep: 5688.61 | bwd_allreduce_microstep: 78.26 | step_microstep: 18.44
[2025-04-25 19:14:21,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.76 | bwd: 5766.93 | bwd_inner: 5688.60 | bwd_allreduce: 78.28 | step: 18.44

 11%|█▏        | 4664/41250 [11:16:47<88:36:10,  8.72s/it]
                                                          
{'loss': 0.1079, 'grad_norm': 1.0714143514633179, 'learning_rate': 3.9280760395464507e-05, 'epoch': 1.13}

 11%|█▏        | 4664/41250 [11:16:47<88:36:10,  8.72s/it][2025-04-25 19:14:30,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 19:14:30,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.16 | bwd_microstep: 5782.52 | bwd_inner_microstep: 5652.07 | bwd_allreduce_microstep: 130.41 | step_microstep: 18.91
[2025-04-25 19:14:30,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.16 | bwd: 5782.53 | bwd_inner: 5652.07 | bwd_allreduce: 130.42 | step: 18.91

 11%|█▏        | 4665/41250 [11:16:55<88:32:29,  8.71s/it]
                                                          
{'loss': 0.1349, 'grad_norm': 1.32918381690979, 'learning_rate': 3.928034299955026e-05, 'epoch': 1.13}

 11%|█▏        | 4665/41250 [11:16:55<88:32:29,  8.71s/it][2025-04-25 19:14:39,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:14:39,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5776.64 | bwd_inner_microstep: 5653.96 | bwd_allreduce_microstep: 122.64 | step_microstep: 18.60
[2025-04-25 19:14:39,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5776.65 | bwd_inner: 5653.95 | bwd_allreduce: 122.66 | step: 18.60

 11%|█▏        | 4666/41250 [11:17:04<88:27:00,  8.70s/it]
                                                          
{'loss': 0.1226, 'grad_norm': 1.2820580005645752, 'learning_rate': 3.927992548477649e-05, 'epoch': 1.13}

 11%|█▏        | 4666/41250 [11:17:04<88:27:00,  8.70s/it][2025-04-25 19:14:47,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 19:14:47,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.57 | bwd_microstep: 5790.63 | bwd_inner_microstep: 5649.85 | bwd_allreduce_microstep: 140.74 | step_microstep: 18.62
[2025-04-25 19:14:47,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.57 | bwd: 5790.64 | bwd_inner: 5649.85 | bwd_allreduce: 140.76 | step: 18.63

 11%|█▏        | 4667/41250 [11:17:13<88:26:32,  8.70s/it]
                                                          
{'loss': 0.1994, 'grad_norm': 2.7955243587493896, 'learning_rate': 3.927950785114576e-05, 'epoch': 1.13}

 11%|█▏        | 4667/41250 [11:17:13<88:26:32,  8.70s/it][2025-04-25 19:14:56,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:14:56,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.81 | bwd_microstep: 5899.05 | bwd_inner_microstep: 5655.61 | bwd_allreduce_microstep: 243.39 | step_microstep: 18.45
[2025-04-25 19:14:56,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.81 | bwd: 5899.06 | bwd_inner: 5655.61 | bwd_allreduce: 243.41 | step: 18.45

 11%|█▏        | 4668/41250 [11:17:22<88:47:50,  8.74s/it]
                                                          
{'loss': 0.2469, 'grad_norm': 3.356201171875, 'learning_rate': 3.927909009866067e-05, 'epoch': 1.13}

 11%|█▏        | 4668/41250 [11:17:22<88:47:50,  8.74s/it][2025-04-25 19:15:05,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:15:05,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5686.89 | bwd_inner_microstep: 5660.49 | bwd_allreduce_microstep: 26.36 | step_microstep: 18.37
[2025-04-25 19:15:05,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5686.91 | bwd_inner: 5660.49 | bwd_allreduce: 26.38 | step: 18.37

 11%|█▏        | 4669/41250 [11:17:30<88:23:20,  8.70s/it]
                                                          
{'loss': 0.4425, 'grad_norm': 3.2496368885040283, 'learning_rate': 3.927867222732378e-05, 'epoch': 1.13}

 11%|█▏        | 4669/41250 [11:17:30<88:23:20,  8.70s/it][2025-04-25 19:15:13,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.69 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:15:13,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.09 | bwd_microstep: 5687.30 | bwd_inner_microstep: 5672.58 | bwd_allreduce_microstep: 14.67 | step_microstep: 18.63
[2025-04-25 19:15:13,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.09 | bwd: 5687.31 | bwd_inner: 5672.58 | bwd_allreduce: 14.69 | step: 18.64

 11%|█▏        | 4670/41250 [11:17:39<88:07:46,  8.67s/it]
                                                          
{'loss': 0.084, 'grad_norm': 1.0883643627166748, 'learning_rate': 3.927825423713766e-05, 'epoch': 1.13}

 11%|█▏        | 4670/41250 [11:17:39<88:07:46,  8.67s/it][2025-04-25 19:15:22,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:15:22,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.26 | bwd_microstep: 5724.65 | bwd_inner_microstep: 5680.32 | bwd_allreduce_microstep: 44.28 | step_microstep: 18.14
[2025-04-25 19:15:22,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.26 | bwd: 5724.67 | bwd_inner: 5680.32 | bwd_allreduce: 44.30 | step: 18.15

 11%|█▏        | 4671/41250 [11:17:47<88:04:18,  8.67s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 1.0362863540649414, 'learning_rate': 3.927783612810491e-05, 'epoch': 1.13}

 11%|█▏        | 4671/41250 [11:17:47<88:04:18,  8.67s/it][2025-04-25 19:15:31,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 19:15:31,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.93 | bwd_microstep: 5726.61 | bwd_inner_microstep: 5705.51 | bwd_allreduce_microstep: 21.05 | step_microstep: 18.60
[2025-04-25 19:15:31,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.93 | bwd: 5726.62 | bwd_inner: 5705.51 | bwd_allreduce: 21.07 | step: 18.60

 11%|█▏        | 4672/41250 [11:17:56<88:03:35,  8.67s/it]
                                                          
{'loss': 0.0788, 'grad_norm': 1.1241873502731323, 'learning_rate': 3.927741790022808e-05, 'epoch': 1.13}

 11%|█▏        | 4672/41250 [11:17:56<88:03:35,  8.67s/it][2025-04-25 19:15:39,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:15:39,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5741.92 | bwd_inner_microstep: 5691.45 | bwd_allreduce_microstep: 50.43 | step_microstep: 18.05
[2025-04-25 19:15:39,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5741.93 | bwd_inner: 5691.45 | bwd_allreduce: 50.44 | step: 18.05

 11%|█▏        | 4673/41250 [11:18:05<88:05:22,  8.67s/it]
                                                          
{'loss': 0.0646, 'grad_norm': 0.9979038834571838, 'learning_rate': 3.927699955350976e-05, 'epoch': 1.13}

 11%|█▏        | 4673/41250 [11:18:05<88:05:22,  8.67s/it][2025-04-25 19:15:48,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:15:48,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.07 | bwd_microstep: 5694.46 | bwd_inner_microstep: 5681.88 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.29
[2025-04-25 19:15:48,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.07 | bwd: 5694.48 | bwd_inner: 5681.88 | bwd_allreduce: 12.56 | step: 18.29

 11%|█▏        | 4674/41250 [11:18:13<87:59:07,  8.66s/it]
                                                          
{'loss': 0.0243, 'grad_norm': 0.3116523325443268, 'learning_rate': 3.927658108795253e-05, 'epoch': 1.13}

 11%|█▏        | 4674/41250 [11:18:13<87:59:07,  8.66s/it][2025-04-25 19:15:57,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:15:57,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5718.67 | bwd_inner_microstep: 5690.01 | bwd_allreduce_microstep: 28.61 | step_microstep: 18.44
[2025-04-25 19:15:57,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5718.68 | bwd_inner: 5690.01 | bwd_allreduce: 28.63 | step: 18.45

 11%|█▏        | 4675/41250 [11:18:22<87:57:14,  8.66s/it]
                                                          
{'loss': 0.1892, 'grad_norm': 2.537076711654663, 'learning_rate': 3.927616250355897e-05, 'epoch': 1.13}

 11%|█▏        | 4675/41250 [11:18:22<87:57:14,  8.66s/it][2025-04-25 19:16:05,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:16:05,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.58 | bwd_microstep: 5754.16 | bwd_inner_microstep: 5650.12 | bwd_allreduce_microstep: 103.99 | step_microstep: 18.37
[2025-04-25 19:16:05,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.58 | bwd: 5754.17 | bwd_inner: 5650.12 | bwd_allreduce: 104.01 | step: 18.38

 11%|█▏        | 4676/41250 [11:18:31<88:01:00,  8.66s/it]
                                                          
{'loss': 0.2068, 'grad_norm': 2.352227210998535, 'learning_rate': 3.927574380033166e-05, 'epoch': 1.13}

 11%|█▏        | 4676/41250 [11:18:31<88:01:00,  8.66s/it][2025-04-25 19:16:14,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-25 19:16:14,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.49 | bwd_microstep: 5704.69 | bwd_inner_microstep: 5690.03 | bwd_allreduce_microstep: 14.62 | step_microstep: 18.39
[2025-04-25 19:16:14,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.49 | bwd: 5704.70 | bwd_inner: 5690.03 | bwd_allreduce: 14.63 | step: 18.39

 11%|█▏        | 4677/41250 [11:18:39<87:58:29,  8.66s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 2.859191656112671, 'learning_rate': 3.927532497827318e-05, 'epoch': 1.13}

 11%|█▏        | 4677/41250 [11:18:39<87:58:29,  8.66s/it][2025-04-25 19:16:23,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:16:23,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.24 | bwd_microstep: 5726.27 | bwd_inner_microstep: 5694.45 | bwd_allreduce_microstep: 31.78 | step_microstep: 18.38
[2025-04-25 19:16:23,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.24 | bwd: 5726.28 | bwd_inner: 5694.45 | bwd_allreduce: 31.80 | step: 18.38

 11%|█▏        | 4678/41250 [11:18:48<88:00:02,  8.66s/it]
                                                          
{'loss': 0.1833, 'grad_norm': 5.555526256561279, 'learning_rate': 3.927490603738611e-05, 'epoch': 1.13}

 11%|█▏        | 4678/41250 [11:18:48<88:00:02,  8.66s/it][2025-04-25 19:16:31,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:16:31,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5680.37 | bwd_inner_microstep: 5646.61 | bwd_allreduce_microstep: 33.72 | step_microstep: 18.41
[2025-04-25 19:16:31,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5680.38 | bwd_inner: 5646.61 | bwd_allreduce: 33.73 | step: 18.42

 11%|█▏        | 4679/41250 [11:18:57<87:47:07,  8.64s/it]
                                                          
{'loss': 0.0926, 'grad_norm': 1.6370929479599, 'learning_rate': 3.927448697767304e-05, 'epoch': 1.13}

 11%|█▏        | 4679/41250 [11:18:57<87:47:07,  8.64s/it][2025-04-25 19:16:40,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:16:40,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.78 | bwd_microstep: 5754.29 | bwd_inner_microstep: 5649.63 | bwd_allreduce_microstep: 104.61 | step_microstep: 18.78
[2025-04-25 19:16:40,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.78 | bwd: 5754.30 | bwd_inner: 5649.63 | bwd_allreduce: 104.63 | step: 18.79

 11%|█▏        | 4680/41250 [11:19:05<87:53:01,  8.65s/it]
                                                          
{'loss': 0.0992, 'grad_norm': 1.5625085830688477, 'learning_rate': 3.9274067799136555e-05, 'epoch': 1.13}

 11%|█▏        | 4680/41250 [11:19:05<87:53:01,  8.65s/it][2025-04-25 19:16:49,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:16:49,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.64 | bwd_microstep: 5762.59 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 110.26 | step_microstep: 18.43
[2025-04-25 19:16:49,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.64 | bwd: 5762.60 | bwd_inner: 5652.29 | bwd_allreduce: 110.27 | step: 18.43

 11%|█▏        | 4681/41250 [11:19:14<87:59:09,  8.66s/it]
                                                          
{'loss': 0.0865, 'grad_norm': 1.8521767854690552, 'learning_rate': 3.927364850177922e-05, 'epoch': 1.13}

 11%|█▏        | 4681/41250 [11:19:14<87:59:09,  8.66s/it][2025-04-25 19:16:57,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:16:57,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.71 | bwd_microstep: 5711.39 | bwd_inner_microstep: 5698.60 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.61
[2025-04-25 19:16:57,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.71 | bwd: 5711.40 | bwd_inner: 5698.60 | bwd_allreduce: 12.76 | step: 18.62

 11%|█▏        | 4682/41250 [11:19:23<87:58:34,  8.66s/it]
                                                          
{'loss': 0.0671, 'grad_norm': 0.8976309299468994, 'learning_rate': 3.927322908560364e-05, 'epoch': 1.14}

 11%|█▏        | 4682/41250 [11:19:23<87:58:34,  8.66s/it][2025-04-25 19:17:06,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 19:17:06,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.32 | bwd_microstep: 5692.48 | bwd_inner_microstep: 5656.19 | bwd_allreduce_microstep: 36.24 | step_microstep: 18.75
[2025-04-25 19:17:06,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.32 | bwd: 5692.49 | bwd_inner: 5656.19 | bwd_allreduce: 36.26 | step: 18.76

 11%|█▏        | 4683/41250 [11:19:31<87:49:46,  8.65s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 1.5411763191223145, 'learning_rate': 3.927280955061239e-05, 'epoch': 1.14}

 11%|█▏        | 4683/41250 [11:19:31<87:49:46,  8.65s/it][2025-04-25 19:17:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:17:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.49 | bwd_microstep: 5736.14 | bwd_inner_microstep: 5698.15 | bwd_allreduce_microstep: 37.94 | step_microstep: 18.59
[2025-04-25 19:17:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.49 | bwd: 5736.16 | bwd_inner: 5698.15 | bwd_allreduce: 37.96 | step: 18.60

 11%|█▏        | 4684/41250 [11:19:40<87:53:54,  8.65s/it]
                                                          
{'loss': 0.1053, 'grad_norm': 1.2314990758895874, 'learning_rate': 3.927238989680806e-05, 'epoch': 1.14}

 11%|█▏        | 4684/41250 [11:19:40<87:53:54,  8.65s/it][2025-04-25 19:17:23,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:17:23,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.11 | bwd_microstep: 5740.41 | bwd_inner_microstep: 5661.02 | bwd_allreduce_microstep: 79.34 | step_microstep: 18.60
[2025-04-25 19:17:23,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.11 | bwd: 5740.42 | bwd_inner: 5661.02 | bwd_allreduce: 79.36 | step: 18.60

 11%|█▏        | 4685/41250 [11:19:49<87:56:05,  8.66s/it]
                                                          
{'loss': 0.0143, 'grad_norm': 0.6147171258926392, 'learning_rate': 3.927197012419324e-05, 'epoch': 1.14}

 11%|█▏        | 4685/41250 [11:19:49<87:56:05,  8.66s/it][2025-04-25 19:17:32,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-25 19:17:32,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.65 | bwd_microstep: 5772.74 | bwd_inner_microstep: 5656.63 | bwd_allreduce_microstep: 116.06 | step_microstep: 19.02
[2025-04-25 19:17:32,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.65 | bwd: 5772.75 | bwd_inner: 5656.63 | bwd_allreduce: 116.08 | step: 19.02

 11%|█▏        | 4686/41250 [11:19:57<88:02:33,  8.67s/it]
                                                          
{'loss': 0.041, 'grad_norm': 0.8177539110183716, 'learning_rate': 3.927155023277052e-05, 'epoch': 1.14}

 11%|█▏        | 4686/41250 [11:19:57<88:02:33,  8.67s/it][2025-04-25 19:17:41,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:17:41,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.31 | bwd_microstep: 5770.02 | bwd_inner_microstep: 5666.35 | bwd_allreduce_microstep: 103.62 | step_microstep: 18.53
[2025-04-25 19:17:41,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.31 | bwd: 5770.04 | bwd_inner: 5666.35 | bwd_allreduce: 103.64 | step: 18.54

 11%|█▏        | 4687/41250 [11:20:06<88:06:34,  8.68s/it]
                                                          
{'loss': 0.0322, 'grad_norm': 0.7651330232620239, 'learning_rate': 3.927113022254247e-05, 'epoch': 1.14}

 11%|█▏        | 4687/41250 [11:20:06<88:06:34,  8.68s/it][2025-04-25 19:17:49,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 19:17:49,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.82 | bwd_microstep: 5769.25 | bwd_inner_microstep: 5661.31 | bwd_allreduce_microstep: 107.88 | step_microstep: 18.82
[2025-04-25 19:17:49,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.82 | bwd: 5769.26 | bwd_inner: 5661.31 | bwd_allreduce: 107.91 | step: 18.83

 11%|█▏        | 4688/41250 [11:20:15<88:10:14,  8.68s/it]
                                                          
{'loss': 0.1791, 'grad_norm': 1.0371373891830444, 'learning_rate': 3.92707100935117e-05, 'epoch': 1.14}

 11%|█▏        | 4688/41250 [11:20:15<88:10:14,  8.68s/it][2025-04-25 19:17:58,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:17:58,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.11 | bwd_microstep: 5709.24 | bwd_inner_microstep: 5675.54 | bwd_allreduce_microstep: 33.65 | step_microstep: 18.69
[2025-04-25 19:17:58,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.11 | bwd: 5709.25 | bwd_inner: 5675.54 | bwd_allreduce: 33.67 | step: 18.69

 11%|█▏        | 4689/41250 [11:20:23<88:01:47,  8.67s/it]
                                                          
{'loss': 0.0507, 'grad_norm': 1.0139471292495728, 'learning_rate': 3.92702898456808e-05, 'epoch': 1.14}

 11%|█▏        | 4689/41250 [11:20:23<88:01:47,  8.67s/it][2025-04-25 19:18:07,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 19:18:07,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.76 | bwd_microstep: 5730.22 | bwd_inner_microstep: 5717.51 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.91
[2025-04-25 19:18:07,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.76 | bwd: 5730.24 | bwd_inner: 5717.51 | bwd_allreduce: 12.68 | step: 18.91

 11%|█▏        | 4690/41250 [11:20:32<88:01:46,  8.67s/it]
                                                          
{'loss': 0.1213, 'grad_norm': 1.8272289037704468, 'learning_rate': 3.926986947905234e-05, 'epoch': 1.14}

 11%|█▏        | 4690/41250 [11:20:32<88:01:46,  8.67s/it][2025-04-25 19:18:15,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-25 19:18:15,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.67 | bwd_microstep: 5716.76 | bwd_inner_microstep: 5703.85 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.99
[2025-04-25 19:18:15,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.67 | bwd: 5716.78 | bwd_inner: 5703.85 | bwd_allreduce: 12.88 | step: 18.99

 11%|█▏        | 4691/41250 [11:20:41<87:59:18,  8.66s/it]
                                                          
{'loss': 0.1249, 'grad_norm': 3.2163541316986084, 'learning_rate': 3.926944899362893e-05, 'epoch': 1.14}

 11%|█▏        | 4691/41250 [11:20:41<87:59:18,  8.66s/it][2025-04-25 19:18:24,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.18 | optimizer_step: 0.92
[2025-04-25 19:18:24,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.43 | bwd_microstep: 5749.89 | bwd_inner_microstep: 5702.40 | bwd_allreduce_microstep: 47.42 | step_microstep: 18.82
[2025-04-25 19:18:24,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.43 | bwd: 5749.90 | bwd_inner: 5702.40 | bwd_allreduce: 47.45 | step: 18.82

 11%|█▏        | 4692/41250 [11:20:49<88:04:47,  8.67s/it]
                                                          
{'loss': 0.1412, 'grad_norm': 3.090867519378662, 'learning_rate': 3.926902838941316e-05, 'epoch': 1.14}

 11%|█▏        | 4692/41250 [11:20:49<88:04:47,  8.67s/it][2025-04-25 19:18:33,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 19:18:33,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.46 | bwd_microstep: 5724.88 | bwd_inner_microstep: 5712.03 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.02
[2025-04-25 19:18:33,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.46 | bwd: 5724.90 | bwd_inner: 5712.03 | bwd_allreduce: 12.83 | step: 19.02

 11%|█▏        | 4693/41250 [11:20:58<88:03:38,  8.67s/it]
                                                          
{'loss': 0.3142, 'grad_norm': 1.9771058559417725, 'learning_rate': 3.9268607666407614e-05, 'epoch': 1.14}

 11%|█▏        | 4693/41250 [11:20:58<88:03:38,  8.67s/it][2025-04-25 19:18:42,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:18:42,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2942.50 | bwd_microstep: 5877.82 | bwd_inner_microstep: 5864.97 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.79
[2025-04-25 19:18:42,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2942.50 | bwd: 5877.83 | bwd_inner: 5864.97 | bwd_allreduce: 12.82 | step: 18.79

 11%|█▏        | 4694/41250 [11:21:07<88:46:46,  8.74s/it]
                                                          
{'loss': 0.1584, 'grad_norm': 2.678492784500122, 'learning_rate': 3.92681868246149e-05, 'epoch': 1.14}

 11%|█▏        | 4694/41250 [11:21:07<88:46:46,  8.74s/it][2025-04-25 19:18:50,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-25 19:18:50,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.86 | bwd_microstep: 5765.49 | bwd_inner_microstep: 5706.94 | bwd_allreduce_microstep: 58.51 | step_microstep: 19.20
[2025-04-25 19:18:50,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.86 | bwd: 5765.50 | bwd_inner: 5706.94 | bwd_allreduce: 58.53 | step: 19.20

 11%|█▏        | 4695/41250 [11:21:16<88:40:28,  8.73s/it]
                                                          
{'loss': 0.1598, 'grad_norm': 1.9384561777114868, 'learning_rate': 3.92677658640376e-05, 'epoch': 1.14}

 11%|█▏        | 4695/41250 [11:21:16<88:40:28,  8.73s/it][2025-04-25 19:18:59,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 19:18:59,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.17 | bwd_microstep: 5890.32 | bwd_inner_microstep: 5712.63 | bwd_allreduce_microstep: 177.63 | step_microstep: 18.96
[2025-04-25 19:18:59,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.17 | bwd: 5890.33 | bwd_inner: 5712.63 | bwd_allreduce: 177.65 | step: 18.96

 11%|█▏        | 4696/41250 [11:21:24<88:58:47,  8.76s/it]
                                                          
{'loss': 0.062, 'grad_norm': 1.7465440034866333, 'learning_rate': 3.926734478467831e-05, 'epoch': 1.14}

 11%|█▏        | 4696/41250 [11:21:24<88:58:47,  8.76s/it][2025-04-25 19:19:08,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 19:19:08,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.17 | bwd_microstep: 5717.48 | bwd_inner_microstep: 5665.23 | bwd_allreduce_microstep: 52.20 | step_microstep: 19.02
[2025-04-25 19:19:08,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.17 | bwd: 5717.49 | bwd_inner: 5665.23 | bwd_allreduce: 52.22 | step: 19.02

 11%|█▏        | 4697/41250 [11:21:33<88:34:41,  8.72s/it]
                                                          
{'loss': 0.1531, 'grad_norm': 1.3437628746032715, 'learning_rate': 3.926692358653963e-05, 'epoch': 1.14}

 11%|█▏        | 4697/41250 [11:21:33<88:34:41,  8.72s/it][2025-04-25 19:19:16,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:19:16,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.26 | bwd_microstep: 5799.95 | bwd_inner_microstep: 5651.35 | bwd_allreduce_microstep: 148.55 | step_microstep: 20.00
[2025-04-25 19:19:16,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.26 | bwd: 5799.96 | bwd_inner: 5651.35 | bwd_allreduce: 148.57 | step: 20.01

 11%|█▏        | 4698/41250 [11:21:42<88:33:14,  8.72s/it]
                                                          
{'loss': 0.0835, 'grad_norm': 2.0113463401794434, 'learning_rate': 3.926650226962416e-05, 'epoch': 1.14}

 11%|█▏        | 4698/41250 [11:21:42<88:33:14,  8.72s/it][2025-04-25 19:19:25,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:19:25,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5801.61 | bwd_inner_microstep: 5663.95 | bwd_allreduce_microstep: 137.61 | step_microstep: 18.56
[2025-04-25 19:19:25,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5801.63 | bwd_inner: 5663.95 | bwd_allreduce: 137.63 | step: 18.57

 11%|█▏        | 4699/41250 [11:21:50<88:31:00,  8.72s/it]
                                                          
{'loss': 0.3577, 'grad_norm': 4.263206481933594, 'learning_rate': 3.926608083393449e-05, 'epoch': 1.14}

 11%|█▏        | 4699/41250 [11:21:50<88:31:00,  8.72s/it][2025-04-25 19:19:34,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:19:34,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.41 | bwd_microstep: 5775.27 | bwd_inner_microstep: 5762.48 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.53
[2025-04-25 19:19:34,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.41 | bwd: 5775.28 | bwd_inner: 5762.48 | bwd_allreduce: 12.77 | step: 18.53

 11%|█▏        | 4700/41250 [11:21:59<88:34:19,  8.72s/it]
                                                          
{'loss': 0.2392, 'grad_norm': 2.5362701416015625, 'learning_rate': 3.926565927947322e-05, 'epoch': 1.14}

 11%|█▏        | 4700/41250 [11:21:59<88:34:19,  8.72s/it][2025-04-25 19:19:43,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 19:19:43,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.92 | bwd_microstep: 5733.91 | bwd_inner_microstep: 5720.63 | bwd_allreduce_microstep: 13.23 | step_microstep: 19.30
[2025-04-25 19:19:43,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.92 | bwd: 5733.92 | bwd_inner: 5720.63 | bwd_allreduce: 13.25 | step: 19.30

 11%|█▏        | 4701/41250 [11:22:08<88:24:51,  8.71s/it]
                                                          
{'loss': 0.0993, 'grad_norm': 2.744521141052246, 'learning_rate': 3.926523760624295e-05, 'epoch': 1.14}

 11%|█▏        | 4701/41250 [11:22:08<88:24:51,  8.71s/it][2025-04-25 19:19:51,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.95 | optimizer_step: 1.01
[2025-04-25 19:19:51,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.98 | bwd_microstep: 5720.20 | bwd_inner_microstep: 5662.57 | bwd_allreduce_microstep: 57.59 | step_microstep: 18.31
[2025-04-25 19:19:51,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.98 | bwd: 5720.21 | bwd_inner: 5662.56 | bwd_allreduce: 57.61 | step: 18.31

 11%|█▏        | 4702/41250 [11:22:17<88:10:42,  8.69s/it]
                                                          
{'loss': 0.0466, 'grad_norm': 0.8221468329429626, 'learning_rate': 3.9264815814246286e-05, 'epoch': 1.14}

 11%|█▏        | 4702/41250 [11:22:17<88:10:42,  8.69s/it][2025-04-25 19:20:00,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 19:20:00,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.97 | bwd_microstep: 5808.65 | bwd_inner_microstep: 5668.51 | bwd_allreduce_microstep: 140.10 | step_microstep: 19.45
[2025-04-25 19:20:00,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.97 | bwd: 5808.66 | bwd_inner: 5668.50 | bwd_allreduce: 140.11 | step: 19.45

 11%|█▏        | 4703/41250 [11:22:25<88:17:50,  8.70s/it]
                                                          
{'loss': 0.0461, 'grad_norm': 0.9115947484970093, 'learning_rate': 3.926439390348581e-05, 'epoch': 1.14}

 11%|█▏        | 4703/41250 [11:22:25<88:17:50,  8.70s/it][2025-04-25 19:20:09,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:20:09,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.17 | bwd_microstep: 5721.33 | bwd_inner_microstep: 5708.26 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.88
[2025-04-25 19:20:09,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.17 | bwd: 5721.34 | bwd_inner: 5708.26 | bwd_allreduce: 13.04 | step: 18.88

 11%|█▏        | 4704/41250 [11:22:34<88:10:00,  8.68s/it]
                                                          
{'loss': 0.0637, 'grad_norm': 1.208533763885498, 'learning_rate': 3.9263971873964146e-05, 'epoch': 1.14}

 11%|█▏        | 4704/41250 [11:22:34<88:10:00,  8.68s/it][2025-04-25 19:20:17,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 19:20:17,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.34 | bwd_microstep: 5767.33 | bwd_inner_microstep: 5693.41 | bwd_allreduce_microstep: 73.88 | step_microstep: 18.77
[2025-04-25 19:20:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.34 | bwd: 5767.35 | bwd_inner: 5693.41 | bwd_allreduce: 73.90 | step: 18.78

 11%|█▏        | 4705/41250 [11:22:43<88:12:56,  8.69s/it]
                                                          
{'loss': 0.211, 'grad_norm': 2.211055278778076, 'learning_rate': 3.926354972568388e-05, 'epoch': 1.14}

 11%|█▏        | 4705/41250 [11:22:43<88:12:56,  8.69s/it][2025-04-25 19:20:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:20:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.31 | bwd_microstep: 5757.37 | bwd_inner_microstep: 5696.52 | bwd_allreduce_microstep: 60.81 | step_microstep: 18.67
[2025-04-25 19:20:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.31 | bwd: 5757.39 | bwd_inner: 5696.52 | bwd_allreduce: 60.82 | step: 18.67

 11%|█▏        | 4706/41250 [11:22:51<88:11:49,  8.69s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 1.2444218397140503, 'learning_rate': 3.926312745864761e-05, 'epoch': 1.14}

 11%|█▏        | 4706/41250 [11:22:51<88:11:49,  8.69s/it][2025-04-25 19:20:35,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 19:20:35,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5784.06 | bwd_inner_microstep: 5725.55 | bwd_allreduce_microstep: 58.46 | step_microstep: 19.29
[2025-04-25 19:20:35,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5784.08 | bwd_inner: 5725.55 | bwd_allreduce: 58.49 | step: 19.28

 11%|█▏        | 4707/41250 [11:23:00<88:19:02,  8.70s/it]
                                                          
{'loss': 0.0346, 'grad_norm': 0.5523371696472168, 'learning_rate': 3.9262705072857966e-05, 'epoch': 1.14}

 11%|█▏        | 4707/41250 [11:23:00<88:19:02,  8.70s/it][2025-04-25 19:20:43,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.07 | optimizer_step: 1.27
[2025-04-25 19:20:43,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.94 | bwd_microstep: 5742.84 | bwd_inner_microstep: 5663.86 | bwd_allreduce_microstep: 78.94 | step_microstep: 20.06
[2025-04-25 19:20:43,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.94 | bwd: 5742.86 | bwd_inner: 5663.86 | bwd_allreduce: 78.96 | step: 20.06

 11%|█▏        | 4708/41250 [11:23:09<88:15:45,  8.70s/it]
                                                          
{'loss': 0.2243, 'grad_norm': 2.9358279705047607, 'learning_rate': 3.9262282568317524e-05, 'epoch': 1.14}

 11%|█▏        | 4708/41250 [11:23:09<88:15:45,  8.70s/it][2025-04-25 19:20:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.23 | optimizer_step: 0.89
[2025-04-25 19:20:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.39 | bwd_microstep: 5720.00 | bwd_inner_microstep: 5648.81 | bwd_allreduce_microstep: 71.14 | step_microstep: 20.62
[2025-04-25 19:20:52,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.39 | bwd: 5720.02 | bwd_inner: 5648.81 | bwd_allreduce: 71.16 | step: 20.61

 11%|█▏        | 4709/41250 [11:23:17<88:05:45,  8.68s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.8836102485656738, 'learning_rate': 3.92618599450289e-05, 'epoch': 1.14}

 11%|█▏        | 4709/41250 [11:23:17<88:05:45,  8.68s/it][2025-04-25 19:21:01,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:21:01,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.33 | bwd_microstep: 5792.72 | bwd_inner_microstep: 5656.88 | bwd_allreduce_microstep: 135.79 | step_microstep: 18.76
[2025-04-25 19:21:01,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.33 | bwd: 5792.73 | bwd_inner: 5656.88 | bwd_allreduce: 135.81 | step: 18.77

 11%|█▏        | 4710/41250 [11:23:26<88:11:42,  8.69s/it]
                                                          
{'loss': 0.0523, 'grad_norm': 1.4962635040283203, 'learning_rate': 3.9261437202994696e-05, 'epoch': 1.14}

 11%|█▏        | 4710/41250 [11:23:26<88:11:42,  8.69s/it][2025-04-25 19:21:09,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:21:09,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.05 | bwd_microstep: 5813.29 | bwd_inner_microstep: 5651.33 | bwd_allreduce_microstep: 161.90 | step_microstep: 18.76
[2025-04-25 19:21:09,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.05 | bwd: 5813.30 | bwd_inner: 5651.33 | bwd_allreduce: 161.93 | step: 18.77

 11%|█▏        | 4711/41250 [11:23:35<88:18:50,  8.70s/it]
                                                          
{'loss': 0.1227, 'grad_norm': 2.575023651123047, 'learning_rate': 3.926101434221752e-05, 'epoch': 1.14}

 11%|█▏        | 4711/41250 [11:23:35<88:18:50,  8.70s/it][2025-04-25 19:21:18,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 19:21:18,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5708.24 | bwd_inner_microstep: 5687.77 | bwd_allreduce_microstep: 20.42 | step_microstep: 19.18
[2025-04-25 19:21:18,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5708.26 | bwd_inner: 5687.77 | bwd_allreduce: 20.45 | step: 19.18

 11%|█▏        | 4712/41250 [11:23:43<88:06:39,  8.68s/it]
                                                          
{'loss': 0.1377, 'grad_norm': 3.126865863800049, 'learning_rate': 3.926059136269998e-05, 'epoch': 1.14}

 11%|█▏        | 4712/41250 [11:23:43<88:06:39,  8.68s/it][2025-04-25 19:21:27,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 19:21:27,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.65 | bwd_microstep: 5695.24 | bwd_inner_microstep: 5682.47 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.75
[2025-04-25 19:21:27,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.65 | bwd: 5695.25 | bwd_inner: 5682.47 | bwd_allreduce: 12.74 | step: 18.75

 11%|█▏        | 4713/41250 [11:23:52<87:56:07,  8.66s/it]
                                                          
{'loss': 0.1651, 'grad_norm': 2.919682025909424, 'learning_rate': 3.926016826444468e-05, 'epoch': 1.14}

 11%|█▏        | 4713/41250 [11:23:52<87:56:07,  8.66s/it][2025-04-25 19:21:35,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:21:35,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.33 | bwd_microstep: 5777.94 | bwd_inner_microstep: 5653.02 | bwd_allreduce_microstep: 124.87 | step_microstep: 18.85
[2025-04-25 19:21:35,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.33 | bwd: 5777.95 | bwd_inner: 5653.01 | bwd_allreduce: 124.89 | step: 18.85

 11%|█▏        | 4714/41250 [11:24:01<88:00:48,  8.67s/it]
                                                          
{'loss': 0.2658, 'grad_norm': 1.7092006206512451, 'learning_rate': 3.925974504745424e-05, 'epoch': 1.14}

 11%|█▏        | 4714/41250 [11:24:01<88:00:48,  8.67s/it][2025-04-25 19:21:44,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 19:21:44,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.82 | bwd_microstep: 5706.75 | bwd_inner_microstep: 5694.18 | bwd_allreduce_microstep: 12.52 | step_microstep: 19.52
[2025-04-25 19:21:44,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.82 | bwd: 5706.77 | bwd_inner: 5694.18 | bwd_allreduce: 12.55 | step: 19.52

 11%|█▏        | 4715/41250 [11:24:09<87:56:41,  8.67s/it]
                                                          
{'loss': 0.0184, 'grad_norm': 0.33914363384246826, 'learning_rate': 3.9259321711731255e-05, 'epoch': 1.14}

 11%|█▏        | 4715/41250 [11:24:09<87:56:41,  8.67s/it][2025-04-25 19:21:53,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:21:53,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.87 | bwd_microstep: 5753.13 | bwd_inner_microstep: 5652.21 | bwd_allreduce_microstep: 100.88 | step_microstep: 18.83
[2025-04-25 19:21:53,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.87 | bwd: 5753.15 | bwd_inner: 5652.21 | bwd_allreduce: 100.89 | step: 18.83

 11%|█▏        | 4716/41250 [11:24:18<87:55:23,  8.66s/it]
                                                          
{'loss': 0.0658, 'grad_norm': 1.05362069606781, 'learning_rate': 3.9258898257278335e-05, 'epoch': 1.14}

 11%|█▏        | 4716/41250 [11:24:18<87:55:23,  8.66s/it][2025-04-25 19:22:01,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 19:22:01,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5755.92 | bwd_inner_microstep: 5654.97 | bwd_allreduce_microstep: 100.91 | step_microstep: 18.62
[2025-04-25 19:22:01,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5755.94 | bwd_inner: 5654.97 | bwd_allreduce: 100.93 | step: 18.62

 11%|█▏        | 4717/41250 [11:24:27<87:55:56,  8.66s/it]
                                                          
{'loss': 0.067, 'grad_norm': 1.4624416828155518, 'learning_rate': 3.92584746840981e-05, 'epoch': 1.14}

 11%|█▏        | 4717/41250 [11:24:27<87:55:56,  8.66s/it][2025-04-25 19:22:10,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 1.00
[2025-04-25 19:22:10,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.73 | bwd_microstep: 5765.51 | bwd_inner_microstep: 5652.93 | bwd_allreduce_microstep: 112.51 | step_microstep: 19.56
[2025-04-25 19:22:10,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.73 | bwd: 5765.52 | bwd_inner: 5652.93 | bwd_allreduce: 112.54 | step: 19.56

 11%|█▏        | 4718/41250 [11:24:35<87:58:26,  8.67s/it]
                                                          
{'loss': 0.15, 'grad_norm': 2.750675678253174, 'learning_rate': 3.925805099219315e-05, 'epoch': 1.14}

 11%|█▏        | 4718/41250 [11:24:35<87:58:26,  8.67s/it][2025-04-25 19:22:19,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 19:22:19,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.07 | bwd_microstep: 5684.51 | bwd_inner_microstep: 5671.75 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.07
[2025-04-25 19:22:19,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.07 | bwd: 5684.52 | bwd_inner: 5671.75 | bwd_allreduce: 12.73 | step: 19.07

 11%|█▏        | 4719/41250 [11:24:44<87:50:10,  8.66s/it]
                                                          
{'loss': 0.0251, 'grad_norm': 1.008902668952942, 'learning_rate': 3.9257627181566116e-05, 'epoch': 1.14}

 11%|█▏        | 4719/41250 [11:24:44<87:50:10,  8.66s/it][2025-04-25 19:22:27,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:22:27,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.76 | bwd_microstep: 5703.66 | bwd_inner_microstep: 5690.92 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.75
[2025-04-25 19:22:27,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.76 | bwd: 5703.67 | bwd_inner: 5690.92 | bwd_allreduce: 12.71 | step: 18.76

 11%|█▏        | 4720/41250 [11:24:53<87:47:57,  8.65s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 2.081171751022339, 'learning_rate': 3.9257203252219594e-05, 'epoch': 1.14}

 11%|█▏        | 4720/41250 [11:24:53<87:47:57,  8.65s/it][2025-04-25 19:22:36,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:22:36,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.78 | bwd_microstep: 5675.65 | bwd_inner_microstep: 5641.92 | bwd_allreduce_microstep: 33.69 | step_microstep: 18.37
[2025-04-25 19:22:36,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.78 | bwd: 5675.67 | bwd_inner: 5641.92 | bwd_allreduce: 33.71 | step: 18.37

 11%|█▏        | 4721/41250 [11:25:01<87:37:30,  8.64s/it]
                                                          
{'loss': 0.0957, 'grad_norm': 1.8928916454315186, 'learning_rate': 3.925677920415619e-05, 'epoch': 1.14}

 11%|█▏        | 4721/41250 [11:25:01<87:37:30,  8.64s/it][2025-04-25 19:22:45,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-25 19:22:45,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.98 | bwd_microstep: 5744.41 | bwd_inner_microstep: 5650.84 | bwd_allreduce_microstep: 93.53 | step_microstep: 18.42
[2025-04-25 19:22:45,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.98 | bwd: 5744.43 | bwd_inner: 5650.85 | bwd_allreduce: 93.54 | step: 18.42

 11%|█▏        | 4722/41250 [11:25:10<87:40:04,  8.64s/it]
                                                          
{'loss': 0.1285, 'grad_norm': 2.4342761039733887, 'learning_rate': 3.9256355037378543e-05, 'epoch': 1.14}

 11%|█▏        | 4722/41250 [11:25:10<87:40:04,  8.64s/it][2025-04-25 19:22:53,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 19:22:53,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.73 | bwd_microstep: 5715.15 | bwd_inner_microstep: 5702.07 | bwd_allreduce_microstep: 13.03 | step_microstep: 19.21
[2025-04-25 19:22:53,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.73 | bwd: 5715.16 | bwd_inner: 5702.07 | bwd_allreduce: 13.05 | step: 19.21

 11%|█▏        | 4723/41250 [11:25:19<87:43:37,  8.65s/it]
                                                          
{'loss': 0.0614, 'grad_norm': 1.5568972826004028, 'learning_rate': 3.9255930751889246e-05, 'epoch': 1.14}

 11%|█▏        | 4723/41250 [11:25:19<87:43:37,  8.65s/it][2025-04-25 19:23:02,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:23:02,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.72 | bwd_microstep: 5749.61 | bwd_inner_microstep: 5644.30 | bwd_allreduce_microstep: 105.27 | step_microstep: 18.39
[2025-04-25 19:23:02,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.72 | bwd: 5749.62 | bwd_inner: 5644.30 | bwd_allreduce: 105.29 | step: 18.39

 11%|█▏        | 4724/41250 [11:25:27<87:45:03,  8.65s/it]
                                                          
{'loss': 0.1055, 'grad_norm': 2.111485004425049, 'learning_rate': 3.925550634769093e-05, 'epoch': 1.15}

 11%|█▏        | 4724/41250 [11:25:27<87:45:03,  8.65s/it][2025-04-25 19:23:11,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:23:11,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.70 | bwd_microstep: 5709.74 | bwd_inner_microstep: 5695.05 | bwd_allreduce_microstep: 14.65 | step_microstep: 19.08
[2025-04-25 19:23:11,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.70 | bwd: 5709.76 | bwd_inner: 5695.05 | bwd_allreduce: 14.67 | step: 19.08

 11%|█▏        | 4725/41250 [11:25:36<87:47:26,  8.65s/it]
                                                          
{'loss': 0.0407, 'grad_norm': 1.5377516746520996, 'learning_rate': 3.92550818247862e-05, 'epoch': 1.15}

 11%|█▏        | 4725/41250 [11:25:36<87:47:26,  8.65s/it][2025-04-25 19:23:19,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 19:23:19,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.53 | bwd_microstep: 5710.76 | bwd_inner_microstep: 5697.61 | bwd_allreduce_microstep: 13.10 | step_microstep: 19.19
[2025-04-25 19:23:19,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.53 | bwd: 5710.77 | bwd_inner: 5697.61 | bwd_allreduce: 13.12 | step: 19.19

 11%|█▏        | 4726/41250 [11:25:45<87:46:28,  8.65s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.423675060272217, 'learning_rate': 3.925465718317768e-05, 'epoch': 1.15}

 11%|█▏        | 4726/41250 [11:25:45<87:46:28,  8.65s/it][2025-04-25 19:23:28,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 19:23:28,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.25 | bwd_microstep: 5736.32 | bwd_inner_microstep: 5702.08 | bwd_allreduce_microstep: 34.19 | step_microstep: 18.97
[2025-04-25 19:23:28,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.25 | bwd: 5736.33 | bwd_inner: 5702.08 | bwd_allreduce: 34.21 | step: 18.97

 11%|█▏        | 4727/41250 [11:25:53<87:51:02,  8.66s/it]
                                                          
{'loss': 0.0094, 'grad_norm': 0.13889364898204803, 'learning_rate': 3.925423242286799e-05, 'epoch': 1.15}

 11%|█▏        | 4727/41250 [11:25:53<87:51:02,  8.66s/it][2025-04-25 19:23:37,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 19:23:37,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5719.11 | bwd_inner_microstep: 5706.32 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.21
[2025-04-25 19:23:37,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5719.12 | bwd_inner: 5706.32 | bwd_allreduce: 12.76 | step: 19.21

 11%|█▏        | 4728/41250 [11:26:02<87:49:37,  8.66s/it]
                                                          
{'loss': 0.0137, 'grad_norm': 0.38143959641456604, 'learning_rate': 3.925380754385974e-05, 'epoch': 1.15}

 11%|█▏        | 4728/41250 [11:26:02<87:49:37,  8.66s/it][2025-04-25 19:23:45,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 19:23:45,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.14 | bwd_microstep: 5741.50 | bwd_inner_microstep: 5694.90 | bwd_allreduce_microstep: 46.55 | step_microstep: 18.91
[2025-04-25 19:23:45,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.14 | bwd: 5741.51 | bwd_inner: 5694.90 | bwd_allreduce: 46.57 | step: 18.91

 11%|█▏        | 4729/41250 [11:26:11<87:51:45,  8.66s/it]
                                                          
{'loss': 0.0998, 'grad_norm': 1.8601948022842407, 'learning_rate': 3.925338254615555e-05, 'epoch': 1.15}

 11%|█▏        | 4729/41250 [11:26:11<87:51:45,  8.66s/it][2025-04-25 19:23:54,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:23:54,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.03 | bwd_microstep: 5700.02 | bwd_inner_microstep: 5687.14 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.55
[2025-04-25 19:23:54,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.03 | bwd: 5700.04 | bwd_inner: 5687.14 | bwd_allreduce: 12.86 | step: 18.55

 11%|█▏        | 4730/41250 [11:26:19<87:45:35,  8.65s/it]
                                                          
{'loss': 0.2843, 'grad_norm': 2.7128095626831055, 'learning_rate': 3.925295742975805e-05, 'epoch': 1.15}

 11%|█▏        | 4730/41250 [11:26:19<87:45:35,  8.65s/it][2025-04-25 19:24:03,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:24:03,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.37 | bwd_microstep: 5764.59 | bwd_inner_microstep: 5661.20 | bwd_allreduce_microstep: 103.35 | step_microstep: 18.91
[2025-04-25 19:24:03,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.37 | bwd: 5764.61 | bwd_inner: 5661.20 | bwd_allreduce: 103.36 | step: 18.91

 11%|█▏        | 4731/41250 [11:26:28<87:49:43,  8.66s/it]
                                                          
{'loss': 0.1673, 'grad_norm': 4.155867576599121, 'learning_rate': 3.925253219466985e-05, 'epoch': 1.15}

 11%|█▏        | 4731/41250 [11:26:28<87:49:43,  8.66s/it][2025-04-25 19:24:11,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 19:24:11,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.77 | bwd_microstep: 5783.79 | bwd_inner_microstep: 5660.77 | bwd_allreduce_microstep: 122.98 | step_microstep: 18.70
[2025-04-25 19:24:11,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.77 | bwd: 5783.80 | bwd_inner: 5660.77 | bwd_allreduce: 122.99 | step: 18.70

 11%|█▏        | 4732/41250 [11:26:37<87:55:40,  8.67s/it]
                                                          
{'loss': 0.2152, 'grad_norm': 2.3533174991607666, 'learning_rate': 3.925210684089358e-05, 'epoch': 1.15}

 11%|█▏        | 4732/41250 [11:26:37<87:55:40,  8.67s/it][2025-04-25 19:24:20,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-25 19:24:20,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5758.95 | bwd_inner_microstep: 5694.53 | bwd_allreduce_microstep: 64.38 | step_microstep: 18.52
[2025-04-25 19:24:20,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5758.96 | bwd_inner: 5694.53 | bwd_allreduce: 64.39 | step: 18.53

 11%|█▏        | 4733/41250 [11:26:45<87:59:36,  8.67s/it]
                                                          
{'loss': 0.0259, 'grad_norm': 1.0426514148712158, 'learning_rate': 3.925168136843185e-05, 'epoch': 1.15}

 11%|█▏        | 4733/41250 [11:26:45<87:59:36,  8.67s/it][2025-04-25 19:24:29,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:24:29,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5678.27 | bwd_inner_microstep: 5656.00 | bwd_allreduce_microstep: 22.22 | step_microstep: 18.40
[2025-04-25 19:24:29,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5678.28 | bwd_inner: 5656.00 | bwd_allreduce: 22.24 | step: 18.40

 11%|█▏        | 4734/41250 [11:26:54<87:47:13,  8.65s/it]
                                                          
{'loss': 0.0591, 'grad_norm': 1.4352678060531616, 'learning_rate': 3.92512557772873e-05, 'epoch': 1.15}

 11%|█▏        | 4734/41250 [11:26:54<87:47:13,  8.65s/it][2025-04-25 19:24:37,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 19:24:37,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.68 | bwd_microstep: 5740.94 | bwd_inner_microstep: 5704.86 | bwd_allreduce_microstep: 36.04 | step_microstep: 18.70
[2025-04-25 19:24:37,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.68 | bwd: 5740.96 | bwd_inner: 5704.85 | bwd_allreduce: 36.06 | step: 18.71

 11%|█▏        | 4735/41250 [11:27:03<87:50:36,  8.66s/it]
                                                          
{'loss': 0.0529, 'grad_norm': 1.2228344678878784, 'learning_rate': 3.925083006746254e-05, 'epoch': 1.15}

 11%|█▏        | 4735/41250 [11:27:03<87:50:36,  8.66s/it][2025-04-25 19:24:46,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 19:24:46,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.25 | bwd_microstep: 5748.02 | bwd_inner_microstep: 5708.98 | bwd_allreduce_microstep: 38.99 | step_microstep: 19.40
[2025-04-25 19:24:46,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.25 | bwd: 5748.04 | bwd_inner: 5708.98 | bwd_allreduce: 39.01 | step: 19.40

 11%|█▏        | 4736/41250 [11:27:11<87:55:29,  8.67s/it]
                                                          
{'loss': 0.2383, 'grad_norm': 3.5267081260681152, 'learning_rate': 3.92504042389602e-05, 'epoch': 1.15}

 11%|█▏        | 4736/41250 [11:27:11<87:55:29,  8.67s/it][2025-04-25 19:24:55,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 19:24:55,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.73 | bwd_microstep: 5712.59 | bwd_inner_microstep: 5663.22 | bwd_allreduce_microstep: 49.33 | step_microstep: 18.95
[2025-04-25 19:24:55,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.73 | bwd: 5712.61 | bwd_inner: 5663.22 | bwd_allreduce: 49.35 | step: 18.95

 11%|█▏        | 4737/41250 [11:27:20<87:48:38,  8.66s/it]
                                                          
{'loss': 0.3074, 'grad_norm': 3.05086088180542, 'learning_rate': 3.92499782917829e-05, 'epoch': 1.15}

 11%|█▏        | 4737/41250 [11:27:20<87:48:38,  8.66s/it][2025-04-25 19:25:03,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:25:03,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.80 | bwd_microstep: 5719.65 | bwd_inner_microstep: 5706.84 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.84
[2025-04-25 19:25:03,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.80 | bwd: 5719.66 | bwd_inner: 5706.84 | bwd_allreduce: 12.78 | step: 18.84

 11%|█▏        | 4738/41250 [11:27:29<87:50:39,  8.66s/it]
                                                          
{'loss': 0.0562, 'grad_norm': 0.821743905544281, 'learning_rate': 3.924955222593328e-05, 'epoch': 1.15}

 11%|█▏        | 4738/41250 [11:27:29<87:50:39,  8.66s/it][2025-04-25 19:25:12,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 19:25:12,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.41 | bwd_microstep: 5761.24 | bwd_inner_microstep: 5720.39 | bwd_allreduce_microstep: 40.81 | step_microstep: 18.65
[2025-04-25 19:25:12,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.41 | bwd: 5761.26 | bwd_inner: 5720.39 | bwd_allreduce: 40.83 | step: 18.65

 11%|█▏        | 4739/41250 [11:27:37<87:59:42,  8.68s/it]
                                                          
{'loss': 0.1054, 'grad_norm': 2.19539737701416, 'learning_rate': 3.924912604141395e-05, 'epoch': 1.15}

 11%|█▏        | 4739/41250 [11:27:37<87:59:42,  8.68s/it][2025-04-25 19:25:21,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:25:21,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.03 | bwd_microstep: 5791.34 | bwd_inner_microstep: 5667.75 | bwd_allreduce_microstep: 123.55 | step_microstep: 18.44
[2025-04-25 19:25:21,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.03 | bwd: 5791.36 | bwd_inner: 5667.75 | bwd_allreduce: 123.56 | step: 18.44

 11%|█▏        | 4740/41250 [11:27:46<88:06:23,  8.69s/it]
                                                          
{'loss': 0.1446, 'grad_norm': 1.9763798713684082, 'learning_rate': 3.9248699738227547e-05, 'epoch': 1.15}

 11%|█▏        | 4740/41250 [11:27:46<88:06:23,  8.69s/it][2025-04-25 19:25:29,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:25:29,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.26 | bwd_microstep: 5714.26 | bwd_inner_microstep: 5656.53 | bwd_allreduce_microstep: 57.68 | step_microstep: 18.16
[2025-04-25 19:25:29,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.26 | bwd: 5714.27 | bwd_inner: 5656.53 | bwd_allreduce: 57.70 | step: 18.17

 11%|█▏        | 4741/41250 [11:27:55<87:56:08,  8.67s/it]
                                                          
{'loss': 0.0764, 'grad_norm': 1.2706516981124878, 'learning_rate': 3.9248273316376693e-05, 'epoch': 1.15}

 11%|█▏        | 4741/41250 [11:27:55<87:56:08,  8.67s/it][2025-04-25 19:25:38,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:25:38,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.18 | bwd_microstep: 5781.12 | bwd_inner_microstep: 5712.96 | bwd_allreduce_microstep: 68.11 | step_microstep: 18.32
[2025-04-25 19:25:38,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.18 | bwd: 5781.14 | bwd_inner: 5712.96 | bwd_allreduce: 68.13 | step: 18.32

 11%|█▏        | 4742/41250 [11:28:03<88:04:56,  8.69s/it]
                                                          
{'loss': 0.0319, 'grad_norm': 0.6237672567367554, 'learning_rate': 3.924784677586402e-05, 'epoch': 1.15}

 11%|█▏        | 4742/41250 [11:28:03<88:04:56,  8.69s/it][2025-04-25 19:25:47,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:25:47,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.04 | bwd_microstep: 5758.11 | bwd_inner_microstep: 5721.16 | bwd_allreduce_microstep: 36.91 | step_microstep: 18.31
[2025-04-25 19:25:47,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.04 | bwd: 5758.12 | bwd_inner: 5721.16 | bwd_allreduce: 36.92 | step: 18.32

 11%|█▏        | 4743/41250 [11:28:12<88:08:12,  8.69s/it]
                                                          
{'loss': 0.3157, 'grad_norm': 1.6603177785873413, 'learning_rate': 3.924742011669215e-05, 'epoch': 1.15}

 11%|█▏        | 4743/41250 [11:28:12<88:08:12,  8.69s/it][2025-04-25 19:25:55,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-25 19:25:55,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.99 | bwd_microstep: 5896.42 | bwd_inner_microstep: 5712.80 | bwd_allreduce_microstep: 183.57 | step_microstep: 18.41
[2025-04-25 19:25:55,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.99 | bwd: 5896.44 | bwd_inner: 5712.80 | bwd_allreduce: 183.59 | step: 18.41

 12%|█▏        | 4744/41250 [11:28:21<88:34:28,  8.73s/it]
                                                          
{'loss': 0.1862, 'grad_norm': 1.2356312274932861, 'learning_rate': 3.9246993338863736e-05, 'epoch': 1.15}

 12%|█▏        | 4744/41250 [11:28:21<88:34:28,  8.73s/it][2025-04-25 19:26:04,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.22 | optimizer_step: 0.96
[2025-04-25 19:26:04,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.78 | bwd_microstep: 5784.62 | bwd_inner_microstep: 5651.77 | bwd_allreduce_microstep: 132.80 | step_microstep: 18.88
[2025-04-25 19:26:04,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.78 | bwd: 5784.63 | bwd_inner: 5651.77 | bwd_allreduce: 132.82 | step: 18.88

 12%|█▏        | 4745/41250 [11:28:30<88:29:53,  8.73s/it]
                                                          
{'loss': 0.2066, 'grad_norm': 3.0010263919830322, 'learning_rate': 3.9246566442381386e-05, 'epoch': 1.15}

 12%|█▏        | 4745/41250 [11:28:30<88:29:53,  8.73s/it][2025-04-25 19:26:13,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:26:13,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.82 | bwd_microstep: 5712.66 | bwd_inner_microstep: 5666.92 | bwd_allreduce_microstep: 45.70 | step_microstep: 18.20
[2025-04-25 19:26:13,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.82 | bwd: 5712.67 | bwd_inner: 5666.92 | bwd_allreduce: 45.71 | step: 18.20

 12%|█▏        | 4746/41250 [11:28:38<88:12:43,  8.70s/it]
                                                          
{'loss': 0.0712, 'grad_norm': 1.340577244758606, 'learning_rate': 3.924613942724774e-05, 'epoch': 1.15}

 12%|█▏        | 4746/41250 [11:28:38<88:12:43,  8.70s/it][2025-04-25 19:26:22,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:26:22,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2906.02 | bwd_microstep: 5805.18 | bwd_inner_microstep: 5792.59 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.65
[2025-04-25 19:26:22,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2906.02 | bwd: 5805.19 | bwd_inner: 5792.59 | bwd_allreduce: 12.56 | step: 18.65

 12%|█▏        | 4747/41250 [11:28:47<88:29:59,  8.73s/it]
                                                          
{'loss': 0.1567, 'grad_norm': 1.9100004434585571, 'learning_rate': 3.924571229346543e-05, 'epoch': 1.15}

 12%|█▏        | 4747/41250 [11:28:47<88:29:59,  8.73s/it][2025-04-25 19:26:30,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 1.04
[2025-04-25 19:26:30,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.63 | bwd_microstep: 5776.17 | bwd_inner_microstep: 5677.31 | bwd_allreduce_microstep: 98.81 | step_microstep: 18.35
[2025-04-25 19:26:30,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.63 | bwd: 5776.18 | bwd_inner: 5677.30 | bwd_allreduce: 98.83 | step: 18.35

 12%|█▏        | 4748/41250 [11:28:56<88:24:43,  8.72s/it]
                                                          
{'loss': 0.1329, 'grad_norm': 3.5723228454589844, 'learning_rate': 3.924528504103709e-05, 'epoch': 1.15}

 12%|█▏        | 4748/41250 [11:28:56<88:24:43,  8.72s/it][2025-04-25 19:26:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:26:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.33 | bwd_microstep: 6063.64 | bwd_inner_microstep: 5667.23 | bwd_allreduce_microstep: 396.37 | step_microstep: 18.38
[2025-04-25 19:26:39,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.33 | bwd: 6063.66 | bwd_inner: 5667.23 | bwd_allreduce: 396.39 | step: 18.38

 12%|█▏        | 4749/41250 [11:29:05<89:13:39,  8.80s/it]
                                                          
{'loss': 0.0249, 'grad_norm': 0.5316081047058105, 'learning_rate': 3.9244857669965355e-05, 'epoch': 1.15}

 12%|█▏        | 4749/41250 [11:29:05<89:13:39,  8.80s/it][2025-04-25 19:26:48,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 19:26:48,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.01 | bwd_microstep: 5758.10 | bwd_inner_microstep: 5706.31 | bwd_allreduce_microstep: 51.75 | step_microstep: 18.29
[2025-04-25 19:26:48,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.01 | bwd: 5758.11 | bwd_inner: 5706.31 | bwd_allreduce: 51.77 | step: 18.29

 12%|█▏        | 4750/41250 [11:29:13<88:53:35,  8.77s/it]
                                                          
{'loss': 0.0714, 'grad_norm': 1.1207773685455322, 'learning_rate': 3.9244430180252855e-05, 'epoch': 1.15}

 12%|█▏        | 4750/41250 [11:29:13<88:53:35,  8.77s/it][2025-04-25 19:26:57,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:26:57,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5774.48 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 116.88 | step_microstep: 18.03
[2025-04-25 19:26:57,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5774.49 | bwd_inner: 5657.55 | bwd_allreduce: 116.90 | step: 18.03

 12%|█▏        | 4751/41250 [11:29:22<88:42:39,  8.75s/it]
                                                          
{'loss': 0.1861, 'grad_norm': 1.5320042371749878, 'learning_rate': 3.9244002571902224e-05, 'epoch': 1.15}

 12%|█▏        | 4751/41250 [11:29:22<88:42:39,  8.75s/it][2025-04-25 19:27:05,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 19:27:05,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.28 | bwd_microstep: 5770.22 | bwd_inner_microstep: 5668.98 | bwd_allreduce_microstep: 101.18 | step_microstep: 19.32
[2025-04-25 19:27:05,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.28 | bwd: 5770.23 | bwd_inner: 5668.98 | bwd_allreduce: 101.21 | step: 19.32

 12%|█▏        | 4752/41250 [11:29:31<88:34:02,  8.74s/it]
                                                          
{'loss': 0.2852, 'grad_norm': 1.9850375652313232, 'learning_rate': 3.924357484491611e-05, 'epoch': 1.15}

 12%|█▏        | 4752/41250 [11:29:31<88:34:02,  8.74s/it][2025-04-25 19:27:14,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 19:27:14,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.09 | bwd_microstep: 5714.10 | bwd_inner_microstep: 5701.36 | bwd_allreduce_microstep: 12.70 | step_microstep: 17.90
[2025-04-25 19:27:14,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.09 | bwd: 5714.11 | bwd_inner: 5701.36 | bwd_allreduce: 12.71 | step: 17.90

 12%|█▏        | 4753/41250 [11:29:39<88:19:26,  8.71s/it]
                                                          
{'loss': 0.0784, 'grad_norm': 1.2855380773544312, 'learning_rate': 3.924314699929714e-05, 'epoch': 1.15}

 12%|█▏        | 4753/41250 [11:29:39<88:19:26,  8.71s/it][2025-04-25 19:27:23,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:27:23,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.50 | bwd_microstep: 5784.61 | bwd_inner_microstep: 5684.60 | bwd_allreduce_microstep: 99.97 | step_microstep: 18.10
[2025-04-25 19:27:23,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.50 | bwd: 5784.62 | bwd_inner: 5684.60 | bwd_allreduce: 99.98 | step: 18.11

 12%|█▏        | 4754/41250 [11:29:48<88:19:36,  8.71s/it]
                                                          
{'loss': 0.101, 'grad_norm': 1.9069010019302368, 'learning_rate': 3.924271903504795e-05, 'epoch': 1.15}

 12%|█▏        | 4754/41250 [11:29:48<88:19:36,  8.71s/it][2025-04-25 19:27:31,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:27:31,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.16 | bwd_microstep: 5709.08 | bwd_inner_microstep: 5696.48 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.22
[2025-04-25 19:27:31,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.16 | bwd: 5709.10 | bwd_inner: 5696.48 | bwd_allreduce: 12.58 | step: 18.22

 12%|█▏        | 4755/41250 [11:29:57<88:07:44,  8.69s/it]
                                                          
{'loss': 0.3026, 'grad_norm': 3.2761895656585693, 'learning_rate': 3.9242290952171184e-05, 'epoch': 1.15}

 12%|█▏        | 4755/41250 [11:29:57<88:07:44,  8.69s/it][2025-04-25 19:27:40,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:27:40,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.12 | bwd_microstep: 5768.68 | bwd_inner_microstep: 5671.77 | bwd_allreduce_microstep: 96.87 | step_microstep: 18.54
[2025-04-25 19:27:40,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.12 | bwd: 5768.69 | bwd_inner: 5671.77 | bwd_allreduce: 96.88 | step: 18.54

 12%|█▏        | 4756/41250 [11:30:05<88:05:11,  8.69s/it]
                                                          
{'loss': 0.0705, 'grad_norm': 0.6696234345436096, 'learning_rate': 3.924186275066948e-05, 'epoch': 1.15}

 12%|█▏        | 4756/41250 [11:30:05<88:05:11,  8.69s/it][2025-04-25 19:27:49,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:27:49,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.00 | bwd_microstep: 5778.19 | bwd_inner_microstep: 5662.61 | bwd_allreduce_microstep: 115.54 | step_microstep: 18.31
[2025-04-25 19:27:49,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.00 | bwd: 5778.21 | bwd_inner: 5662.61 | bwd_allreduce: 115.56 | step: 18.32

 12%|█▏        | 4757/41250 [11:30:14<88:05:14,  8.69s/it]
                                                          
{'loss': 0.3643, 'grad_norm': 2.593266248703003, 'learning_rate': 3.924143443054548e-05, 'epoch': 1.15}

 12%|█▏        | 4757/41250 [11:30:14<88:05:14,  8.69s/it][2025-04-25 19:27:58,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:27:58,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5786.94 | bwd_inner_microstep: 5648.22 | bwd_allreduce_microstep: 138.67 | step_microstep: 18.95
[2025-04-25 19:27:58,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5786.95 | bwd_inner: 5648.22 | bwd_allreduce: 138.69 | step: 18.95

 12%|█▏        | 4758/41250 [11:30:23<88:07:33,  8.69s/it]
                                                          
{'loss': 0.0113, 'grad_norm': 0.1715356558561325, 'learning_rate': 3.9241005991801814e-05, 'epoch': 1.15}

 12%|█▏        | 4758/41250 [11:30:23<88:07:33,  8.69s/it][2025-04-25 19:28:06,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.23 | optimizer_step: 0.99
[2025-04-25 19:28:06,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5738.65 | bwd_inner_microstep: 5679.72 | bwd_allreduce_microstep: 58.87 | step_microstep: 19.39
[2025-04-25 19:28:06,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5738.66 | bwd_inner: 5679.72 | bwd_allreduce: 58.90 | step: 19.39

 12%|█▏        | 4759/41250 [11:30:32<88:03:31,  8.69s/it]
                                                          
{'loss': 0.0945, 'grad_norm': 0.9567952752113342, 'learning_rate': 3.9240577434441134e-05, 'epoch': 1.15}

 12%|█▏        | 4759/41250 [11:30:32<88:03:31,  8.69s/it][2025-04-25 19:28:15,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.08 | optimizer_step: 0.95
[2025-04-25 19:28:15,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.52 | bwd_microstep: 5759.65 | bwd_inner_microstep: 5695.62 | bwd_allreduce_microstep: 63.98 | step_microstep: 19.93
[2025-04-25 19:28:15,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5759.67 | bwd_inner: 5695.62 | bwd_allreduce: 64.01 | step: 19.93

 12%|█▏        | 4760/41250 [11:30:40<88:04:02,  8.69s/it]
                                                          
{'loss': 0.0971, 'grad_norm': 1.3594484329223633, 'learning_rate': 3.924014875846608e-05, 'epoch': 1.15}

 12%|█▏        | 4760/41250 [11:30:40<88:04:02,  8.69s/it][2025-04-25 19:28:24,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 19:28:24,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 5773.20 | bwd_inner_microstep: 5648.61 | bwd_allreduce_microstep: 124.53 | step_microstep: 19.45
[2025-04-25 19:28:24,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 5773.22 | bwd_inner: 5648.61 | bwd_allreduce: 124.55 | step: 19.45

 12%|█▏        | 4761/41250 [11:30:49<88:03:47,  8.69s/it]
                                                          
{'loss': 0.0922, 'grad_norm': 1.2038213014602661, 'learning_rate': 3.923971996387929e-05, 'epoch': 1.15}

 12%|█▏        | 4761/41250 [11:30:49<88:03:47,  8.69s/it][2025-04-25 19:28:32,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 19:28:32,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.11 | bwd_microstep: 5777.74 | bwd_inner_microstep: 5653.70 | bwd_allreduce_microstep: 124.00 | step_microstep: 19.24
[2025-04-25 19:28:32,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.11 | bwd: 5777.76 | bwd_inner: 5653.70 | bwd_allreduce: 124.01 | step: 19.24

 12%|█▏        | 4762/41250 [11:30:58<88:03:42,  8.69s/it]
                                                          
{'loss': 0.052, 'grad_norm': 1.1598602533340454, 'learning_rate': 3.923929105068341e-05, 'epoch': 1.15}

 12%|█▏        | 4762/41250 [11:30:58<88:03:42,  8.69s/it][2025-04-25 19:28:41,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 19:28:41,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.72 | bwd_microstep: 5771.24 | bwd_inner_microstep: 5651.03 | bwd_allreduce_microstep: 120.16 | step_microstep: 18.67
[2025-04-25 19:28:41,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.72 | bwd: 5771.25 | bwd_inner: 5651.03 | bwd_allreduce: 120.18 | step: 18.67

 12%|█▏        | 4763/41250 [11:31:06<88:02:50,  8.69s/it]
                                                          
{'loss': 0.0938, 'grad_norm': 2.278381109237671, 'learning_rate': 3.923886201888109e-05, 'epoch': 1.15}

 12%|█▏        | 4763/41250 [11:31:06<88:02:50,  8.69s/it][2025-04-25 19:28:50,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 19:28:50,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.55 | bwd_microstep: 5714.00 | bwd_inner_microstep: 5701.05 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.00
[2025-04-25 19:28:50,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.55 | bwd: 5714.02 | bwd_inner: 5701.05 | bwd_allreduce: 12.92 | step: 19.01

 12%|█▏        | 4764/41250 [11:31:15<87:55:53,  8.68s/it]
                                                          
{'loss': 0.2118, 'grad_norm': 2.749673843383789, 'learning_rate': 3.923843286847497e-05, 'epoch': 1.15}

 12%|█▏        | 4764/41250 [11:31:15<87:55:53,  8.68s/it][2025-04-25 19:28:58,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 19:28:58,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.85 | bwd_microstep: 5672.17 | bwd_inner_microstep: 5658.83 | bwd_allreduce_microstep: 13.28 | step_microstep: 19.26
[2025-04-25 19:28:58,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.85 | bwd: 5672.19 | bwd_inner: 5658.83 | bwd_allreduce: 13.31 | step: 19.26

 12%|█▏        | 4765/41250 [11:31:24<87:41:06,  8.65s/it]
                                                          
{'loss': 0.0794, 'grad_norm': 1.0415256023406982, 'learning_rate': 3.923800359946769e-05, 'epoch': 1.16}

 12%|█▏        | 4765/41250 [11:31:24<87:41:06,  8.65s/it][2025-04-25 19:29:07,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 19:29:07,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2930.91 | bwd_microstep: 5874.25 | bwd_inner_microstep: 5861.58 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.85
[2025-04-25 19:29:07,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2930.91 | bwd: 5874.27 | bwd_inner: 5861.58 | bwd_allreduce: 12.64 | step: 18.86

 12%|█▏        | 4766/41250 [11:31:32<88:23:53,  8.72s/it]
                                                          
{'loss': 0.2483, 'grad_norm': 1.9342526197433472, 'learning_rate': 3.92375742118619e-05, 'epoch': 1.16}

 12%|█▏        | 4766/41250 [11:31:32<88:23:53,  8.72s/it][2025-04-25 19:29:16,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-25 19:29:16,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.91 | bwd_microstep: 5769.59 | bwd_inner_microstep: 5756.86 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.41
[2025-04-25 19:29:16,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.91 | bwd: 5769.61 | bwd_inner: 5756.86 | bwd_allreduce: 12.70 | step: 19.41

 12%|█▏        | 4767/41250 [11:31:41<88:26:44,  8.73s/it]
                                                          
{'loss': 0.0547, 'grad_norm': 0.6261408925056458, 'learning_rate': 3.923714470566026e-05, 'epoch': 1.16}

 12%|█▏        | 4767/41250 [11:31:41<88:26:44,  8.73s/it][2025-04-25 19:29:25,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:29:25,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.10 | bwd_microstep: 5760.82 | bwd_inner_microstep: 5654.29 | bwd_allreduce_microstep: 106.48 | step_microstep: 18.51
[2025-04-25 19:29:25,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.10 | bwd: 5760.83 | bwd_inner: 5654.29 | bwd_allreduce: 106.50 | step: 18.51

 12%|█▏        | 4768/41250 [11:31:50<88:18:53,  8.71s/it]
                                                          
{'loss': 0.0724, 'grad_norm': 1.1256295442581177, 'learning_rate': 3.923671508086539e-05, 'epoch': 1.16}

 12%|█▏        | 4768/41250 [11:31:50<88:18:53,  8.71s/it][2025-04-25 19:29:33,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:29:33,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5699.25 | bwd_inner_microstep: 5651.46 | bwd_allreduce_microstep: 47.74 | step_microstep: 18.56
[2025-04-25 19:29:33,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5699.26 | bwd_inner: 5651.46 | bwd_allreduce: 47.76 | step: 18.57

 12%|█▏        | 4769/41250 [11:31:58<88:01:41,  8.69s/it]
                                                          
{'loss': 0.3306, 'grad_norm': 2.228577136993408, 'learning_rate': 3.923628533747997e-05, 'epoch': 1.16}

 12%|█▏        | 4769/41250 [11:31:58<88:01:41,  8.69s/it][2025-04-25 19:29:42,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 19:29:42,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.64 | bwd_microstep: 5779.40 | bwd_inner_microstep: 5766.53 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.69
[2025-04-25 19:29:42,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.64 | bwd: 5779.42 | bwd_inner: 5766.53 | bwd_allreduce: 12.84 | step: 18.69

 12%|█▏        | 4770/41250 [11:32:07<88:13:41,  8.71s/it]
                                                          
{'loss': 0.1992, 'grad_norm': 1.445123314857483, 'learning_rate': 3.923585547550663e-05, 'epoch': 1.16}

 12%|█▏        | 4770/41250 [11:32:07<88:13:41,  8.71s/it][2025-04-25 19:29:50,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.19 | optimizer_step: 0.98
[2025-04-25 19:29:50,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.04 | bwd_microstep: 5666.62 | bwd_inner_microstep: 5653.27 | bwd_allreduce_microstep: 13.29 | step_microstep: 19.19
[2025-04-25 19:29:50,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.04 | bwd: 5666.63 | bwd_inner: 5653.27 | bwd_allreduce: 13.31 | step: 19.19

 12%|█▏        | 4771/41250 [11:32:16<87:52:56,  8.67s/it]
                                                          
{'loss': 0.0365, 'grad_norm': 0.7106852531433105, 'learning_rate': 3.9235425494948025e-05, 'epoch': 1.16}

 12%|█▏        | 4771/41250 [11:32:16<87:52:56,  8.67s/it][2025-04-25 19:29:59,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 19:29:59,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.51 | bwd_microstep: 5700.60 | bwd_inner_microstep: 5639.52 | bwd_allreduce_microstep: 61.03 | step_microstep: 19.35
[2025-04-25 19:29:59,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.51 | bwd: 5700.61 | bwd_inner: 5639.52 | bwd_allreduce: 61.05 | step: 19.35

 12%|█▏        | 4772/41250 [11:32:24<87:41:22,  8.65s/it]
                                                          
{'loss': 0.1574, 'grad_norm': 2.663306713104248, 'learning_rate': 3.9234995395806806e-05, 'epoch': 1.16}

 12%|█▏        | 4772/41250 [11:32:24<87:41:22,  8.65s/it][2025-04-25 19:30:08,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:30:08,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.64 | bwd_microstep: 5740.13 | bwd_inner_microstep: 5648.99 | bwd_allreduce_microstep: 91.09 | step_microstep: 18.48
[2025-04-25 19:30:08,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.64 | bwd: 5740.15 | bwd_inner: 5648.99 | bwd_allreduce: 91.11 | step: 18.49

 12%|█▏        | 4773/41250 [11:32:33<87:41:51,  8.66s/it]
                                                          
{'loss': 0.1464, 'grad_norm': 2.180725336074829, 'learning_rate': 3.923456517808562e-05, 'epoch': 1.16}

 12%|█▏        | 4773/41250 [11:32:33<87:41:51,  8.66s/it][2025-04-25 19:30:16,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 19:30:16,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.95 | bwd_microstep: 5685.39 | bwd_inner_microstep: 5644.36 | bwd_allreduce_microstep: 40.97 | step_microstep: 19.20
[2025-04-25 19:30:16,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.95 | bwd: 5685.40 | bwd_inner: 5644.36 | bwd_allreduce: 40.99 | step: 19.20

 12%|█▏        | 4774/41250 [11:32:42<87:30:54,  8.64s/it]
                                                          
{'loss': 0.1231, 'grad_norm': 2.4635796546936035, 'learning_rate': 3.923413484178713e-05, 'epoch': 1.16}

 12%|█▏        | 4774/41250 [11:32:42<87:30:54,  8.64s/it][2025-04-25 19:30:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 19:30:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5897.30 | bwd_inner_microstep: 5884.72 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.72
[2025-04-25 19:30:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5897.31 | bwd_inner: 5884.72 | bwd_allreduce: 12.55 | step: 18.72

 12%|█▏        | 4775/41250 [11:32:50<88:02:03,  8.69s/it]
                                                          
{'loss': 0.1626, 'grad_norm': 3.3246028423309326, 'learning_rate': 3.9233704386913965e-05, 'epoch': 1.16}

 12%|█▏        | 4775/41250 [11:32:50<88:02:03,  8.69s/it][2025-04-25 19:30:34,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.16
[2025-04-25 19:30:34,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.69 | bwd_microstep: 5735.39 | bwd_inner_microstep: 5688.01 | bwd_allreduce_microstep: 47.32 | step_microstep: 19.53
[2025-04-25 19:30:34,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.69 | bwd: 5735.41 | bwd_inner: 5688.01 | bwd_allreduce: 47.35 | step: 19.53

 12%|█▏        | 4776/41250 [11:32:59<87:57:09,  8.68s/it]
                                                          
{'loss': 0.102, 'grad_norm': 3.5901575088500977, 'learning_rate': 3.923327381346881e-05, 'epoch': 1.16}

 12%|█▏        | 4776/41250 [11:32:59<87:57:09,  8.68s/it][2025-04-25 19:30:42,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:30:42,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5687.26 | bwd_inner_microstep: 5674.45 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.79
[2025-04-25 19:30:42,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5687.27 | bwd_inner: 5674.45 | bwd_allreduce: 12.78 | step: 18.79

 12%|█▏        | 4777/41250 [11:33:08<87:44:49,  8.66s/it]
                                                          
{'loss': 0.0133, 'grad_norm': 0.27499011158943176, 'learning_rate': 3.9232843121454305e-05, 'epoch': 1.16}

 12%|█▏        | 4777/41250 [11:33:08<87:44:49,  8.66s/it][2025-04-25 19:30:51,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:30:51,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.63 | bwd_microstep: 5726.87 | bwd_inner_microstep: 5647.19 | bwd_allreduce_microstep: 79.63 | step_microstep: 18.47
[2025-04-25 19:30:51,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.63 | bwd: 5726.88 | bwd_inner: 5647.19 | bwd_allreduce: 79.64 | step: 18.47

 12%|█▏        | 4778/41250 [11:33:16<87:41:14,  8.66s/it]
                                                          
{'loss': 0.1359, 'grad_norm': 1.062434196472168, 'learning_rate': 3.92324123108731e-05, 'epoch': 1.16}

 12%|█▏        | 4778/41250 [11:33:16<87:41:14,  8.66s/it][2025-04-25 19:31:00,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:31:00,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.63 | bwd_microstep: 5745.50 | bwd_inner_microstep: 5691.73 | bwd_allreduce_microstep: 53.73 | step_microstep: 18.47
[2025-04-25 19:31:00,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.63 | bwd: 5745.51 | bwd_inner: 5691.73 | bwd_allreduce: 53.75 | step: 18.47

 12%|█▏        | 4779/41250 [11:33:25<87:45:48,  8.66s/it]
                                                          
{'loss': 0.2815, 'grad_norm': 3.4740922451019287, 'learning_rate': 3.923198138172786e-05, 'epoch': 1.16}

 12%|█▏        | 4779/41250 [11:33:25<87:45:48,  8.66s/it][2025-04-25 19:31:08,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:31:08,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.81 | bwd_microstep: 5744.75 | bwd_inner_microstep: 5686.32 | bwd_allreduce_microstep: 58.39 | step_microstep: 18.30
[2025-04-25 19:31:08,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.81 | bwd: 5744.76 | bwd_inner: 5686.32 | bwd_allreduce: 58.40 | step: 18.30

 12%|█▏        | 4780/41250 [11:33:34<87:49:00,  8.67s/it]
                                                          
{'loss': 0.2158, 'grad_norm': 2.32328200340271, 'learning_rate': 3.923155033402124e-05, 'epoch': 1.16}

 12%|█▏        | 4780/41250 [11:33:34<87:49:00,  8.67s/it][2025-04-25 19:31:17,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 19:31:17,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5748.38 | bwd_inner_microstep: 5696.98 | bwd_allreduce_microstep: 51.35 | step_microstep: 18.50
[2025-04-25 19:31:17,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5748.39 | bwd_inner: 5696.98 | bwd_allreduce: 51.37 | step: 18.50

 12%|█▏        | 4781/41250 [11:33:42<87:51:12,  8.67s/it]
                                                          
{'loss': 0.1004, 'grad_norm': 2.3112552165985107, 'learning_rate': 3.923111916775588e-05, 'epoch': 1.16}

 12%|█▏        | 4781/41250 [11:33:42<87:51:12,  8.67s/it][2025-04-25 19:31:26,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:31:26,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.92 | bwd_microstep: 5696.87 | bwd_inner_microstep: 5657.35 | bwd_allreduce_microstep: 39.47 | step_microstep: 18.79
[2025-04-25 19:31:26,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.92 | bwd: 5696.88 | bwd_inner: 5657.35 | bwd_allreduce: 39.49 | step: 18.79

 12%|█▏        | 4782/41250 [11:33:51<87:40:59,  8.66s/it]
                                                          
{'loss': 0.1679, 'grad_norm': 3.6094658374786377, 'learning_rate': 3.923068788293447e-05, 'epoch': 1.16}

 12%|█▏        | 4782/41250 [11:33:51<87:40:59,  8.66s/it][2025-04-25 19:31:34,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 19:31:34,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.07 | bwd_microstep: 5719.49 | bwd_inner_microstep: 5706.82 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.51
[2025-04-25 19:31:34,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.07 | bwd: 5719.51 | bwd_inner: 5706.82 | bwd_allreduce: 12.65 | step: 18.51

 12%|█▏        | 4783/41250 [11:34:00<87:38:47,  8.65s/it]
                                                          
{'loss': 0.0194, 'grad_norm': 0.4176330268383026, 'learning_rate': 3.923025647955964e-05, 'epoch': 1.16}

 12%|█▏        | 4783/41250 [11:34:00<87:38:47,  8.65s/it][2025-04-25 19:31:43,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:31:43,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.08 | bwd_microstep: 5713.75 | bwd_inner_microstep: 5672.48 | bwd_allreduce_microstep: 41.22 | step_microstep: 18.59
[2025-04-25 19:31:43,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.09 | bwd: 5713.77 | bwd_inner: 5672.48 | bwd_allreduce: 41.24 | step: 18.60

 12%|█▏        | 4784/41250 [11:34:08<87:33:57,  8.64s/it]
                                                          
{'loss': 0.2624, 'grad_norm': 1.9751349687576294, 'learning_rate': 3.922982495763407e-05, 'epoch': 1.16}

 12%|█▏        | 4784/41250 [11:34:08<87:33:57,  8.64s/it][2025-04-25 19:31:52,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:31:52,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5779.79 | bwd_inner_microstep: 5665.14 | bwd_allreduce_microstep: 114.60 | step_microstep: 18.63
[2025-04-25 19:31:52,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5779.80 | bwd_inner: 5665.14 | bwd_allreduce: 114.62 | step: 18.63

 12%|█▏        | 4785/41250 [11:34:17<87:42:22,  8.66s/it]
                                                          
{'loss': 0.1135, 'grad_norm': 1.2706512212753296, 'learning_rate': 3.92293933171604e-05, 'epoch': 1.16}

 12%|█▏        | 4785/41250 [11:34:17<87:42:22,  8.66s/it][2025-04-25 19:32:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.26 | optimizer_step: 1.03
[2025-04-25 19:32:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.45 | bwd_microstep: 5765.76 | bwd_inner_microstep: 5700.75 | bwd_allreduce_microstep: 64.95 | step_microstep: 19.87
[2025-04-25 19:32:00,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.45 | bwd: 5765.78 | bwd_inner: 5700.75 | bwd_allreduce: 64.98 | step: 19.87

 12%|█▏        | 4786/41250 [11:34:26<87:49:19,  8.67s/it]
                                                          
{'loss': 0.3513, 'grad_norm': 2.152869939804077, 'learning_rate': 3.9228961558141315e-05, 'epoch': 1.16}

 12%|█▏        | 4786/41250 [11:34:26<87:49:19,  8.67s/it][2025-04-25 19:32:09,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:32:09,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.06 | bwd_microstep: 5695.19 | bwd_inner_microstep: 5682.32 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.43
[2025-04-25 19:32:09,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.06 | bwd: 5695.20 | bwd_inner: 5682.32 | bwd_allreduce: 12.84 | step: 18.43

 12%|█▏        | 4787/41250 [11:34:34<87:44:22,  8.66s/it]
                                                          
{'loss': 0.1473, 'grad_norm': 2.2794017791748047, 'learning_rate': 3.922852968057946e-05, 'epoch': 1.16}

 12%|█▏        | 4787/41250 [11:34:34<87:44:22,  8.66s/it][2025-04-25 19:32:18,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:32:18,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5704.53 | bwd_inner_microstep: 5650.55 | bwd_allreduce_microstep: 53.93 | step_microstep: 18.20
[2025-04-25 19:32:18,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5704.54 | bwd_inner: 5650.55 | bwd_allreduce: 53.95 | step: 18.21

 12%|█▏        | 4788/41250 [11:34:43<87:35:50,  8.65s/it]
                                                          
{'loss': 0.2584, 'grad_norm': 2.2258410453796387, 'learning_rate': 3.9228097684477496e-05, 'epoch': 1.16}

 12%|█▏        | 4788/41250 [11:34:43<87:35:50,  8.65s/it][2025-04-25 19:32:26,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:32:26,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.32 | bwd_microstep: 5755.90 | bwd_inner_microstep: 5700.37 | bwd_allreduce_microstep: 55.49 | step_microstep: 18.81
[2025-04-25 19:32:26,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.32 | bwd: 5755.91 | bwd_inner: 5700.37 | bwd_allreduce: 55.51 | step: 18.81

 12%|█▏        | 4789/41250 [11:34:52<87:43:48,  8.66s/it]
                                                          
{'loss': 0.1411, 'grad_norm': 2.5132060050964355, 'learning_rate': 3.9227665569838096e-05, 'epoch': 1.16}

 12%|█▏        | 4789/41250 [11:34:52<87:43:48,  8.66s/it][2025-04-25 19:32:35,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:32:35,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.56 | bwd_microstep: 5731.63 | bwd_inner_microstep: 5718.95 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.66
[2025-04-25 19:32:35,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.56 | bwd: 5731.64 | bwd_inner: 5718.94 | bwd_allreduce: 12.66 | step: 18.66

 12%|█▏        | 4790/41250 [11:35:00<87:45:25,  8.66s/it]
                                                          
{'loss': 0.0674, 'grad_norm': 3.9907150268554688, 'learning_rate': 3.9227233336663924e-05, 'epoch': 1.16}

 12%|█▏        | 4790/41250 [11:35:00<87:45:25,  8.66s/it][2025-04-25 19:32:44,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:32:44,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.22 | bwd_microstep: 5720.19 | bwd_inner_microstep: 5665.75 | bwd_allreduce_microstep: 54.39 | step_microstep: 18.36
[2025-04-25 19:32:44,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.23 | bwd: 5720.20 | bwd_inner: 5665.75 | bwd_allreduce: 54.41 | step: 18.36

 12%|█▏        | 4791/41250 [11:35:09<87:41:12,  8.66s/it]
                                                          
{'loss': 0.1275, 'grad_norm': 1.7730860710144043, 'learning_rate': 3.9226800984957635e-05, 'epoch': 1.16}

 12%|█▏        | 4791/41250 [11:35:09<87:41:12,  8.66s/it][2025-04-25 19:32:52,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:32:52,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5752.48 | bwd_inner_microstep: 5700.59 | bwd_allreduce_microstep: 51.84 | step_microstep: 18.53
[2025-04-25 19:32:52,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5752.49 | bwd_inner: 5700.59 | bwd_allreduce: 51.86 | step: 18.54

 12%|█▏        | 4792/41250 [11:35:18<87:45:53,  8.67s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 1.1670159101486206, 'learning_rate': 3.92263685147219e-05, 'epoch': 1.16}

 12%|█▏        | 4792/41250 [11:35:18<87:45:53,  8.67s/it][2025-04-25 19:33:01,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:33:01,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.02 | bwd_microstep: 5730.80 | bwd_inner_microstep: 5717.98 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.26
[2025-04-25 19:33:01,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.02 | bwd: 5730.81 | bwd_inner: 5717.98 | bwd_allreduce: 12.79 | step: 18.26

 12%|█▏        | 4793/41250 [11:35:26<87:47:16,  8.67s/it]
                                                          
{'loss': 0.1332, 'grad_norm': 1.3106619119644165, 'learning_rate': 3.9225935925959385e-05, 'epoch': 1.16}

 12%|█▏        | 4793/41250 [11:35:26<87:47:16,  8.67s/it][2025-04-25 19:33:10,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:33:10,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.60 | bwd_microstep: 5953.28 | bwd_inner_microstep: 5662.67 | bwd_allreduce_microstep: 290.56 | step_microstep: 18.24
[2025-04-25 19:33:10,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 5953.29 | bwd_inner: 5662.67 | bwd_allreduce: 290.58 | step: 18.24

 12%|█▏        | 4794/41250 [11:35:35<88:23:58,  8.73s/it]
                                                          
{'loss': 0.2348, 'grad_norm': 1.7338823080062866, 'learning_rate': 3.922550321867276e-05, 'epoch': 1.16}

 12%|█▏        | 4794/41250 [11:35:35<88:23:58,  8.73s/it][2025-04-25 19:33:19,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 19:33:19,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.27 | bwd_microstep: 5785.48 | bwd_inner_microstep: 5695.12 | bwd_allreduce_microstep: 90.32 | step_microstep: 18.41
[2025-04-25 19:33:19,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.27 | bwd: 5785.49 | bwd_inner: 5695.12 | bwd_allreduce: 90.33 | step: 18.41

 12%|█▏        | 4795/41250 [11:35:44<88:23:37,  8.73s/it]
                                                          
{'loss': 0.1991, 'grad_norm': 2.3131675720214844, 'learning_rate': 3.922507039286468e-05, 'epoch': 1.16}

 12%|█▏        | 4795/41250 [11:35:44<88:23:37,  8.73s/it][2025-04-25 19:33:28,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:33:28,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.30 | bwd_microstep: 6039.86 | bwd_inner_microstep: 5711.07 | bwd_allreduce_microstep: 328.74 | step_microstep: 18.40
[2025-04-25 19:33:28,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.31 | bwd: 6039.87 | bwd_inner: 5711.07 | bwd_allreduce: 328.76 | step: 18.40

 12%|█▏        | 4796/41250 [11:35:53<89:10:25,  8.81s/it]
                                                          
{'loss': 0.1804, 'grad_norm': 2.366947889328003, 'learning_rate': 3.922463744853783e-05, 'epoch': 1.16}

 12%|█▏        | 4796/41250 [11:35:53<89:10:25,  8.81s/it][2025-04-25 19:33:36,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:33:36,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.73 | bwd_microstep: 5760.08 | bwd_inner_microstep: 5701.36 | bwd_allreduce_microstep: 58.68 | step_microstep: 17.98
[2025-04-25 19:33:36,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.73 | bwd: 5760.09 | bwd_inner: 5701.36 | bwd_allreduce: 58.70 | step: 17.99

 12%|█▏        | 4797/41250 [11:36:02<88:51:34,  8.78s/it]
                                                          
{'loss': 0.1266, 'grad_norm': 1.287366509437561, 'learning_rate': 3.922420438569487e-05, 'epoch': 1.16}

 12%|█▏        | 4797/41250 [11:36:02<88:51:34,  8.78s/it][2025-04-25 19:33:45,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:33:45,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5722.45 | bwd_inner_microstep: 5702.33 | bwd_allreduce_microstep: 20.08 | step_microstep: 18.17
[2025-04-25 19:33:45,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5722.46 | bwd_inner: 5702.33 | bwd_allreduce: 20.09 | step: 18.18

 12%|█▏        | 4798/41250 [11:36:10<88:28:31,  8.74s/it]
                                                          
{'loss': 0.2097, 'grad_norm': 2.1113126277923584, 'learning_rate': 3.9223771204338475e-05, 'epoch': 1.16}

 12%|█▏        | 4798/41250 [11:36:10<88:28:31,  8.74s/it][2025-04-25 19:33:54,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 19:33:54,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.16 | bwd_microstep: 5771.90 | bwd_inner_microstep: 5672.30 | bwd_allreduce_microstep: 99.56 | step_microstep: 18.89
[2025-04-25 19:33:54,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.16 | bwd: 5771.92 | bwd_inner: 5672.30 | bwd_allreduce: 99.58 | step: 18.90

 12%|█▏        | 4799/41250 [11:36:19<88:21:04,  8.73s/it]
                                                          
{'loss': 0.1576, 'grad_norm': 2.3674511909484863, 'learning_rate': 3.9223337904471304e-05, 'epoch': 1.16}

 12%|█▏        | 4799/41250 [11:36:19<88:21:04,  8.73s/it][2025-04-25 19:34:02,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:34:02,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.37 | bwd_microstep: 5782.23 | bwd_inner_microstep: 5670.61 | bwd_allreduce_microstep: 111.58 | step_microstep: 18.78
[2025-04-25 19:34:02,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.37 | bwd: 5782.24 | bwd_inner: 5670.61 | bwd_allreduce: 111.59 | step: 18.79

 12%|█▏        | 4800/41250 [11:36:28<88:16:27,  8.72s/it]
                                                          
{'loss': 0.2322, 'grad_norm': 3.1812644004821777, 'learning_rate': 3.9222904486096045e-05, 'epoch': 1.16}

 12%|█▏        | 4800/41250 [11:36:28<88:16:27,  8.72s/it][2025-04-25 19:34:11,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 19:34:11,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.22 | bwd_microstep: 5794.23 | bwd_inner_microstep: 5659.96 | bwd_allreduce_microstep: 134.23 | step_microstep: 18.32
[2025-04-25 19:34:11,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.22 | bwd: 5794.24 | bwd_inner: 5659.96 | bwd_allreduce: 134.24 | step: 18.32

 12%|█▏        | 4801/41250 [11:36:36<88:14:29,  8.72s/it]
                                                          
{'loss': 0.1573, 'grad_norm': 1.598960041999817, 'learning_rate': 3.922247094921535e-05, 'epoch': 1.16}

 12%|█▏        | 4801/41250 [11:36:36<88:14:29,  8.72s/it][2025-04-25 19:34:20,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:34:20,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.14 | bwd_microstep: 5776.64 | bwd_inner_microstep: 5666.40 | bwd_allreduce_microstep: 110.20 | step_microstep: 18.51
[2025-04-25 19:34:20,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.14 | bwd: 5776.65 | bwd_inner: 5666.40 | bwd_allreduce: 110.22 | step: 18.51

 12%|█▏        | 4802/41250 [11:36:45<88:13:22,  8.71s/it]
                                                          
{'loss': 0.0427, 'grad_norm': 0.7067662477493286, 'learning_rate': 3.9222037293831914e-05, 'epoch': 1.16}

 12%|█▏        | 4802/41250 [11:36:45<88:13:22,  8.71s/it][2025-04-25 19:34:29,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 19:34:29,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.33 | bwd_microstep: 5771.20 | bwd_inner_microstep: 5758.59 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.20
[2025-04-25 19:34:29,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.33 | bwd: 5771.21 | bwd_inner: 5758.58 | bwd_allreduce: 12.58 | step: 18.20

 12%|█▏        | 4803/41250 [11:36:54<88:19:36,  8.72s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 1.2229222059249878, 'learning_rate': 3.922160351994839e-05, 'epoch': 1.16}

 12%|█▏        | 4803/41250 [11:36:54<88:19:36,  8.72s/it][2025-04-25 19:34:37,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 19:34:37,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.79 | bwd_microstep: 5715.63 | bwd_inner_microstep: 5646.41 | bwd_allreduce_microstep: 69.18 | step_microstep: 18.35
[2025-04-25 19:34:37,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.79 | bwd: 5715.65 | bwd_inner: 5646.41 | bwd_allreduce: 69.19 | step: 18.36

 12%|█▏        | 4804/41250 [11:37:02<88:02:24,  8.70s/it]
                                                          
{'loss': 0.0917, 'grad_norm': 0.8582670092582703, 'learning_rate': 3.922116962756746e-05, 'epoch': 1.16}

 12%|█▏        | 4804/41250 [11:37:02<88:02:24,  8.70s/it][2025-04-25 19:34:46,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:34:46,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.96 | bwd_microstep: 5766.13 | bwd_inner_microstep: 5651.49 | bwd_allreduce_microstep: 114.60 | step_microstep: 18.30
[2025-04-25 19:34:46,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.96 | bwd: 5766.14 | bwd_inner: 5651.49 | bwd_allreduce: 114.61 | step: 18.31

 12%|█▏        | 4805/41250 [11:37:11<87:59:17,  8.69s/it]
                                                          
{'loss': 0.1215, 'grad_norm': 2.3662259578704834, 'learning_rate': 3.922073561669181e-05, 'epoch': 1.16}

 12%|█▏        | 4805/41250 [11:37:11<87:59:17,  8.69s/it][2025-04-25 19:34:54,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:34:54,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.88 | bwd_microstep: 5686.03 | bwd_inner_microstep: 5661.01 | bwd_allreduce_microstep: 24.97 | step_microstep: 18.55
[2025-04-25 19:34:54,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.88 | bwd: 5686.04 | bwd_inner: 5661.01 | bwd_allreduce: 24.99 | step: 18.55

 12%|█▏        | 4806/41250 [11:37:20<87:45:49,  8.67s/it]
                                                          
{'loss': 0.0417, 'grad_norm': 0.6139357686042786, 'learning_rate': 3.922030148732409e-05, 'epoch': 1.17}

 12%|█▏        | 4806/41250 [11:37:20<87:45:49,  8.67s/it][2025-04-25 19:35:03,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:35:03,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.70 | bwd_microstep: 5700.31 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.49
[2025-04-25 19:35:03,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.70 | bwd: 5700.33 | bwd_inner: 5687.54 | bwd_allreduce: 12.75 | step: 18.50

 12%|█▏        | 4807/41250 [11:37:28<87:41:19,  8.66s/it]
                                                          
{'loss': 0.0823, 'grad_norm': 0.7478948831558228, 'learning_rate': 3.9219867239467005e-05, 'epoch': 1.17}

 12%|█▏        | 4807/41250 [11:37:28<87:41:19,  8.66s/it][2025-04-25 19:35:12,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:35:12,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.60 | bwd_microstep: 5860.90 | bwd_inner_microstep: 5698.32 | bwd_allreduce_microstep: 162.55 | step_microstep: 18.42
[2025-04-25 19:35:12,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.60 | bwd: 5860.92 | bwd_inner: 5698.32 | bwd_allreduce: 162.56 | step: 18.42

 12%|█▏        | 4808/41250 [11:37:37<88:07:57,  8.71s/it]
                                                          
{'loss': 0.2969, 'grad_norm': 1.8237119913101196, 'learning_rate': 3.9219432873123216e-05, 'epoch': 1.17}

 12%|█▏        | 4808/41250 [11:37:37<88:07:57,  8.71s/it][2025-04-25 19:35:21,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:35:21,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.16 | bwd_microstep: 5718.95 | bwd_inner_microstep: 5704.16 | bwd_allreduce_microstep: 14.74 | step_microstep: 18.56
[2025-04-25 19:35:21,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.16 | bwd: 5718.96 | bwd_inner: 5704.16 | bwd_allreduce: 14.76 | step: 18.57

 12%|█▏        | 4809/41250 [11:37:46<88:00:42,  8.69s/it]
                                                          
{'loss': 0.1527, 'grad_norm': 1.5175973176956177, 'learning_rate': 3.92189983882954e-05, 'epoch': 1.17}

 12%|█▏        | 4809/41250 [11:37:46<88:00:42,  8.69s/it][2025-04-25 19:35:29,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:35:29,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.11 | bwd_microstep: 5720.47 | bwd_inner_microstep: 5707.75 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.46
[2025-04-25 19:35:29,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.11 | bwd: 5720.49 | bwd_inner: 5707.75 | bwd_allreduce: 12.70 | step: 18.47

 12%|█▏        | 4810/41250 [11:37:55<87:55:34,  8.69s/it]
                                                          
{'loss': 0.0665, 'grad_norm': 0.8502180576324463, 'learning_rate': 3.921856378498624e-05, 'epoch': 1.17}

 12%|█▏        | 4810/41250 [11:37:55<87:55:34,  8.69s/it][2025-04-25 19:35:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:35:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5738.48 | bwd_inner_microstep: 5649.93 | bwd_allreduce_microstep: 88.51 | step_microstep: 17.65
[2025-04-25 19:35:38,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5738.50 | bwd_inner: 5649.93 | bwd_allreduce: 88.53 | step: 17.65

 12%|█▏        | 4811/41250 [11:38:03<87:48:18,  8.67s/it]
                                                          
{'loss': 0.16, 'grad_norm': 2.1533284187316895, 'learning_rate': 3.921812906319842e-05, 'epoch': 1.17}

 12%|█▏        | 4811/41250 [11:38:03<87:48:18,  8.67s/it][2025-04-25 19:35:47,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:35:47,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5715.85 | bwd_inner_microstep: 5703.20 | bwd_allreduce_microstep: 12.61 | step_microstep: 17.91
[2025-04-25 19:35:47,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5715.87 | bwd_inner: 5703.20 | bwd_allreduce: 12.62 | step: 17.92

 12%|█▏        | 4812/41250 [11:38:12<87:44:25,  8.67s/it]
                                                          
{'loss': 0.1914, 'grad_norm': 2.0669641494750977, 'learning_rate': 3.9217694222934605e-05, 'epoch': 1.17}

 12%|█▏        | 4812/41250 [11:38:12<87:44:25,  8.67s/it][2025-04-25 19:35:55,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:35:55,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.50 | bwd_microstep: 5715.40 | bwd_inner_microstep: 5702.74 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.40
[2025-04-25 19:35:55,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.50 | bwd: 5715.42 | bwd_inner: 5702.74 | bwd_allreduce: 12.63 | step: 18.40

 12%|█▏        | 4813/41250 [11:38:21<87:41:48,  8.66s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 1.2013497352600098, 'learning_rate': 3.9217259264197494e-05, 'epoch': 1.17}

 12%|█▏        | 4813/41250 [11:38:21<87:41:48,  8.66s/it][2025-04-25 19:36:04,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 19:36:04,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.84 | bwd_microstep: 5767.81 | bwd_inner_microstep: 5656.37 | bwd_allreduce_microstep: 111.39 | step_microstep: 18.80
[2025-04-25 19:36:04,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.84 | bwd: 5767.83 | bwd_inner: 5656.37 | bwd_allreduce: 111.42 | step: 18.80

 12%|█▏        | 4814/41250 [11:38:29<87:43:48,  8.67s/it]
                                                          
{'loss': 0.0578, 'grad_norm': 1.226884365081787, 'learning_rate': 3.921682418698975e-05, 'epoch': 1.17}

 12%|█▏        | 4814/41250 [11:38:29<87:43:48,  8.67s/it][2025-04-25 19:36:13,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:36:13,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.10 | bwd_microstep: 5746.52 | bwd_inner_microstep: 5662.63 | bwd_allreduce_microstep: 83.84 | step_microstep: 18.20
[2025-04-25 19:36:13,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.10 | bwd: 5746.53 | bwd_inner: 5662.63 | bwd_allreduce: 83.86 | step: 18.21

 12%|█▏        | 4815/41250 [11:38:38<87:44:17,  8.67s/it]
                                                          
{'loss': 0.0907, 'grad_norm': 0.9660002589225769, 'learning_rate': 3.921638899131407e-05, 'epoch': 1.17}

 12%|█▏        | 4815/41250 [11:38:38<87:44:17,  8.67s/it][2025-04-25 19:36:21,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:36:21,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.83 | bwd_microstep: 5733.58 | bwd_inner_microstep: 5704.79 | bwd_allreduce_microstep: 28.74 | step_microstep: 18.32
[2025-04-25 19:36:21,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.83 | bwd: 5733.59 | bwd_inner: 5704.79 | bwd_allreduce: 28.76 | step: 18.32

 12%|█▏        | 4816/41250 [11:38:47<87:44:58,  8.67s/it]
                                                          
{'loss': 0.1045, 'grad_norm': 2.243445634841919, 'learning_rate': 3.921595367717313e-05, 'epoch': 1.17}

 12%|█▏        | 4816/41250 [11:38:47<87:44:58,  8.67s/it][2025-04-25 19:36:30,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:36:30,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.97 | bwd_microstep: 5716.10 | bwd_inner_microstep: 5703.20 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.64
[2025-04-25 19:36:30,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.97 | bwd: 5716.11 | bwd_inner: 5703.20 | bwd_allreduce: 12.87 | step: 18.65

 12%|█▏        | 4817/41250 [11:38:55<87:44:11,  8.67s/it]
                                                          
{'loss': 0.1509, 'grad_norm': 1.8264247179031372, 'learning_rate': 3.9215518244569623e-05, 'epoch': 1.17}

 12%|█▏        | 4817/41250 [11:38:55<87:44:11,  8.67s/it][2025-04-25 19:36:39,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 19:36:39,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.91 | bwd_microstep: 5750.29 | bwd_inner_microstep: 5649.98 | bwd_allreduce_microstep: 100.27 | step_microstep: 18.46
[2025-04-25 19:36:39,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.91 | bwd: 5750.30 | bwd_inner: 5649.98 | bwd_allreduce: 100.29 | step: 18.47

 12%|█▏        | 4818/41250 [11:39:04<87:44:01,  8.67s/it]
                                                          
{'loss': 0.0819, 'grad_norm': 1.0606857538223267, 'learning_rate': 3.9215082693506214e-05, 'epoch': 1.17}

 12%|█▏        | 4818/41250 [11:39:04<87:44:01,  8.67s/it][2025-04-25 19:36:47,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:36:47,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.88 | bwd_microstep: 5736.99 | bwd_inner_microstep: 5693.62 | bwd_allreduce_microstep: 43.33 | step_microstep: 18.47
[2025-04-25 19:36:47,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.88 | bwd: 5737.01 | bwd_inner: 5693.62 | bwd_allreduce: 43.35 | step: 18.48

 12%|█▏        | 4819/41250 [11:39:13<87:45:05,  8.67s/it]
                                                          
{'loss': 0.2346, 'grad_norm': 1.4530978202819824, 'learning_rate': 3.9214647023985606e-05, 'epoch': 1.17}

 12%|█▏        | 4819/41250 [11:39:13<87:45:05,  8.67s/it][2025-04-25 19:36:56,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.93
[2025-04-25 19:36:56,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.40 | bwd_microstep: 5727.85 | bwd_inner_microstep: 5696.60 | bwd_allreduce_microstep: 31.21 | step_microstep: 18.36
[2025-04-25 19:36:56,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.40 | bwd: 5727.86 | bwd_inner: 5696.60 | bwd_allreduce: 31.23 | step: 18.36

 12%|█▏        | 4820/41250 [11:39:21<87:43:27,  8.67s/it]
                                                          
{'loss': 0.0998, 'grad_norm': 0.998527467250824, 'learning_rate': 3.9214211236010475e-05, 'epoch': 1.17}

 12%|█▏        | 4820/41250 [11:39:21<87:43:27,  8.67s/it][2025-04-25 19:37:05,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:37:05,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.23 | bwd_microstep: 5730.79 | bwd_inner_microstep: 5681.63 | bwd_allreduce_microstep: 49.12 | step_microstep: 18.61
[2025-04-25 19:37:05,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.23 | bwd: 5730.81 | bwd_inner: 5681.63 | bwd_allreduce: 49.13 | step: 18.61

 12%|█▏        | 4821/41250 [11:39:30<87:43:18,  8.67s/it]
                                                          
{'loss': 0.0722, 'grad_norm': 1.5825917720794678, 'learning_rate': 3.9213775329583514e-05, 'epoch': 1.17}

 12%|█▏        | 4821/41250 [11:39:30<87:43:18,  8.67s/it][2025-04-25 19:37:13,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:37:13,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.60 | bwd_microstep: 5682.23 | bwd_inner_microstep: 5642.31 | bwd_allreduce_microstep: 39.87 | step_microstep: 18.50
[2025-04-25 19:37:13,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 5682.24 | bwd_inner: 5642.31 | bwd_allreduce: 39.89 | step: 18.50

 12%|█▏        | 4822/41250 [11:39:38<87:31:01,  8.65s/it]
                                                          
{'loss': 0.2085, 'grad_norm': 3.083127737045288, 'learning_rate': 3.921333930470741e-05, 'epoch': 1.17}

 12%|█▏        | 4822/41250 [11:39:38<87:31:01,  8.65s/it][2025-04-25 19:37:22,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 19:37:22,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.78 | bwd_microstep: 5727.09 | bwd_inner_microstep: 5695.92 | bwd_allreduce_microstep: 31.13 | step_microstep: 18.72
[2025-04-25 19:37:22,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.78 | bwd: 5727.10 | bwd_inner: 5695.92 | bwd_allreduce: 31.15 | step: 18.73

 12%|█▏        | 4823/41250 [11:39:47<87:34:04,  8.65s/it]
                                                          
{'loss': 0.0607, 'grad_norm': 0.8865305781364441, 'learning_rate': 3.921290316138485e-05, 'epoch': 1.17}

 12%|█▏        | 4823/41250 [11:39:47<87:34:04,  8.65s/it][2025-04-25 19:37:30,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 19:37:30,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.20 | bwd_microstep: 5679.18 | bwd_inner_microstep: 5652.20 | bwd_allreduce_microstep: 26.94 | step_microstep: 18.55
[2025-04-25 19:37:30,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.20 | bwd: 5679.19 | bwd_inner: 5652.20 | bwd_allreduce: 26.95 | step: 18.56

 12%|█▏        | 4824/41250 [11:39:56<87:23:23,  8.64s/it]
                                                          
{'loss': 0.1122, 'grad_norm': 1.6111036539077759, 'learning_rate': 3.9212466899618506e-05, 'epoch': 1.17}

 12%|█▏        | 4824/41250 [11:39:56<87:23:23,  8.64s/it][2025-04-25 19:37:39,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 19:37:39,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.46 | bwd_microstep: 5672.13 | bwd_inner_microstep: 5659.46 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.76
[2025-04-25 19:37:39,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.46 | bwd: 5672.15 | bwd_inner: 5659.46 | bwd_allreduce: 12.65 | step: 18.76

 12%|█▏        | 4825/41250 [11:40:04<87:16:09,  8.63s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 1.485959768295288, 'learning_rate': 3.92120305194111e-05, 'epoch': 1.17}

 12%|█▏        | 4825/41250 [11:40:04<87:16:09,  8.63s/it][2025-04-25 19:37:48,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-25 19:37:48,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5759.78 | bwd_inner_microstep: 5652.96 | bwd_allreduce_microstep: 106.75 | step_microstep: 20.20
[2025-04-25 19:37:48,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5759.79 | bwd_inner: 5652.96 | bwd_allreduce: 106.78 | step: 20.20

 12%|█▏        | 4826/41250 [11:40:13<87:25:04,  8.64s/it]
                                                          
{'loss': 0.1013, 'grad_norm': 1.5348538160324097, 'learning_rate': 3.921159402076529e-05, 'epoch': 1.17}

 12%|█▏        | 4826/41250 [11:40:13<87:25:04,  8.64s/it][2025-04-25 19:37:56,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 19:37:56,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.66 | bwd_microstep: 5728.55 | bwd_inner_microstep: 5681.74 | bwd_allreduce_microstep: 46.75 | step_microstep: 19.16
[2025-04-25 19:37:56,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.66 | bwd: 5728.56 | bwd_inner: 5681.74 | bwd_allreduce: 46.78 | step: 19.16

 12%|█▏        | 4827/41250 [11:40:22<87:29:06,  8.65s/it]
                                                          
{'loss': 0.1106, 'grad_norm': 1.5222660303115845, 'learning_rate': 3.921115740368379e-05, 'epoch': 1.17}

 12%|█▏        | 4827/41250 [11:40:22<87:29:06,  8.65s/it][2025-04-25 19:38:05,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-25 19:38:05,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.64 | bwd_microstep: 5714.50 | bwd_inner_microstep: 5688.96 | bwd_allreduce_microstep: 25.50 | step_microstep: 18.43
[2025-04-25 19:38:05,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.64 | bwd: 5714.51 | bwd_inner: 5688.96 | bwd_allreduce: 25.51 | step: 18.44

 12%|█▏        | 4828/41250 [11:40:30<87:29:32,  8.65s/it]
                                                          
{'loss': 0.2548, 'grad_norm': 3.8109006881713867, 'learning_rate': 3.921072066816928e-05, 'epoch': 1.17}

 12%|█▏        | 4828/41250 [11:40:30<87:29:32,  8.65s/it][2025-04-25 19:38:14,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:38:14,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.36 | bwd_microstep: 5787.41 | bwd_inner_microstep: 5774.60 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.71
[2025-04-25 19:38:14,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.36 | bwd: 5787.42 | bwd_inner: 5774.60 | bwd_allreduce: 12.79 | step: 18.71

 12%|█▏        | 4829/41250 [11:40:39<87:49:43,  8.68s/it]
                                                          
{'loss': 0.1384, 'grad_norm': 3.3414554595947266, 'learning_rate': 3.9210283814224454e-05, 'epoch': 1.17}

 12%|█▏        | 4829/41250 [11:40:39<87:49:43,  8.68s/it][2025-04-25 19:38:22,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 19:38:22,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.35 | bwd_microstep: 5697.84 | bwd_inner_microstep: 5685.10 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.73
[2025-04-25 19:38:22,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.35 | bwd: 5697.86 | bwd_inner: 5685.10 | bwd_allreduce: 12.72 | step: 18.73

 12%|█▏        | 4830/41250 [11:40:48<87:38:54,  8.66s/it]
                                                          
{'loss': 0.1172, 'grad_norm': 4.202263832092285, 'learning_rate': 3.920984684185202e-05, 'epoch': 1.17}

 12%|█▏        | 4830/41250 [11:40:48<87:38:54,  8.66s/it][2025-04-25 19:38:31,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 19:38:31,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.87 | bwd_microstep: 5685.93 | bwd_inner_microstep: 5638.66 | bwd_allreduce_microstep: 47.22 | step_microstep: 18.74
[2025-04-25 19:38:31,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.87 | bwd: 5685.94 | bwd_inner: 5638.66 | bwd_allreduce: 47.24 | step: 18.74

 12%|█▏        | 4831/41250 [11:40:56<87:28:09,  8.65s/it]
                                                          
{'loss': 0.1225, 'grad_norm': 2.9106147289276123, 'learning_rate': 3.9209409751054643e-05, 'epoch': 1.17}

 12%|█▏        | 4831/41250 [11:40:56<87:28:09,  8.65s/it][2025-04-25 19:38:40,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:38:40,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.52 | bwd_microstep: 5681.41 | bwd_inner_microstep: 5637.52 | bwd_allreduce_microstep: 43.85 | step_microstep: 18.68
[2025-04-25 19:38:40,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.52 | bwd: 5681.43 | bwd_inner: 5637.52 | bwd_allreduce: 43.87 | step: 18.68

 12%|█▏        | 4832/41250 [11:41:05<87:17:21,  8.63s/it]
                                                          
{'loss': 0.182, 'grad_norm': 1.7207306623458862, 'learning_rate': 3.920897254183504e-05, 'epoch': 1.17}

 12%|█▏        | 4832/41250 [11:41:05<87:17:21,  8.63s/it][2025-04-25 19:38:48,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 19:38:48,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.37 | bwd_microstep: 5880.28 | bwd_inner_microstep: 5650.43 | bwd_allreduce_microstep: 229.81 | step_microstep: 18.86
[2025-04-25 19:38:48,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.37 | bwd: 5880.29 | bwd_inner: 5650.43 | bwd_allreduce: 229.82 | step: 18.86

 12%|█▏        | 4833/41250 [11:41:14<87:46:16,  8.68s/it]
                                                          
{'loss': 0.0788, 'grad_norm': 1.4711214303970337, 'learning_rate': 3.9208535214195894e-05, 'epoch': 1.17}

 12%|█▏        | 4833/41250 [11:41:14<87:46:16,  8.68s/it][2025-04-25 19:38:57,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:38:57,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.35 | bwd_microstep: 5691.23 | bwd_inner_microstep: 5678.33 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.58
[2025-04-25 19:38:57,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.35 | bwd: 5691.25 | bwd_inner: 5678.33 | bwd_allreduce: 12.87 | step: 18.59

 12%|█▏        | 4834/41250 [11:41:22<87:36:54,  8.66s/it]
                                                          
{'loss': 0.1298, 'grad_norm': 1.7749028205871582, 'learning_rate': 3.920809776813991e-05, 'epoch': 1.17}

 12%|█▏        | 4834/41250 [11:41:22<87:36:54,  8.66s/it][2025-04-25 19:39:06,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 19:39:06,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.51 | bwd_microstep: 5736.14 | bwd_inner_microstep: 5688.47 | bwd_allreduce_microstep: 47.62 | step_microstep: 19.07
[2025-04-25 19:39:06,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.51 | bwd: 5736.15 | bwd_inner: 5688.47 | bwd_allreduce: 47.64 | step: 19.08

 12%|█▏        | 4835/41250 [11:41:31<87:38:34,  8.66s/it]
                                                          
{'loss': 0.1857, 'grad_norm': 1.4377646446228027, 'learning_rate': 3.920766020366979e-05, 'epoch': 1.17}

 12%|█▏        | 4835/41250 [11:41:31<87:38:34,  8.66s/it][2025-04-25 19:39:14,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-25 19:39:14,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.41 | bwd_microstep: 5673.35 | bwd_inner_microstep: 5655.52 | bwd_allreduce_microstep: 17.78 | step_microstep: 18.52
[2025-04-25 19:39:14,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.41 | bwd: 5673.36 | bwd_inner: 5655.52 | bwd_allreduce: 17.80 | step: 18.52

 12%|█▏        | 4836/41250 [11:41:40<87:27:14,  8.65s/it]
                                                          
{'loss': 0.1312, 'grad_norm': 1.7172436714172363, 'learning_rate': 3.9207222520788206e-05, 'epoch': 1.17}

 12%|█▏        | 4836/41250 [11:41:40<87:27:14,  8.65s/it][2025-04-25 19:39:23,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:39:23,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.14 | bwd_microstep: 5743.80 | bwd_inner_microstep: 5689.88 | bwd_allreduce_microstep: 53.88 | step_microstep: 18.35
[2025-04-25 19:39:23,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.14 | bwd: 5743.82 | bwd_inner: 5689.88 | bwd_allreduce: 53.90 | step: 18.35

 12%|█▏        | 4837/41250 [11:41:48<87:32:00,  8.65s/it]
                                                          
{'loss': 0.3176, 'grad_norm': 4.809261322021484, 'learning_rate': 3.920678471949789e-05, 'epoch': 1.17}

 12%|█▏        | 4837/41250 [11:41:48<87:32:00,  8.65s/it][2025-04-25 19:39:32,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:39:32,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.19 | bwd_microstep: 5747.70 | bwd_inner_microstep: 5684.71 | bwd_allreduce_microstep: 62.94 | step_microstep: 18.48
[2025-04-25 19:39:32,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.19 | bwd: 5747.71 | bwd_inner: 5684.71 | bwd_allreduce: 62.95 | step: 18.48

 12%|█▏        | 4838/41250 [11:41:57<87:39:13,  8.67s/it]
                                                          
{'loss': 0.1239, 'grad_norm': 2.1053736209869385, 'learning_rate': 3.920634679980151e-05, 'epoch': 1.17}

 12%|█▏        | 4838/41250 [11:41:57<87:39:13,  8.67s/it][2025-04-25 19:39:41,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:39:41,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2942.59 | bwd_microstep: 5870.59 | bwd_inner_microstep: 5857.74 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.56
[2025-04-25 19:39:41,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2942.59 | bwd: 5870.60 | bwd_inner: 5857.73 | bwd_allreduce: 12.82 | step: 18.56

 12%|█▏        | 4839/41250 [11:42:06<88:21:40,  8.74s/it]
                                                          
{'loss': 0.1901, 'grad_norm': 1.5719406604766846, 'learning_rate': 3.920590876170179e-05, 'epoch': 1.17}

 12%|█▏        | 4839/41250 [11:42:06<88:21:40,  8.74s/it][2025-04-25 19:39:49,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 19:39:49,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5724.96 | bwd_inner_microstep: 5712.26 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.62
[2025-04-25 19:39:49,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5724.97 | bwd_inner: 5712.26 | bwd_allreduce: 12.67 | step: 18.63

 12%|█▏        | 4840/41250 [11:42:15<88:08:55,  8.72s/it]
                                                          
{'loss': 0.2201, 'grad_norm': 3.573864221572876, 'learning_rate': 3.920547060520141e-05, 'epoch': 1.17}

 12%|█▏        | 4840/41250 [11:42:15<88:08:55,  8.72s/it][2025-04-25 19:39:58,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 19:39:58,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.95 | bwd_microstep: 5731.68 | bwd_inner_microstep: 5711.16 | bwd_allreduce_microstep: 20.48 | step_microstep: 18.56
[2025-04-25 19:39:58,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.95 | bwd: 5731.70 | bwd_inner: 5711.16 | bwd_allreduce: 20.50 | step: 18.56

 12%|█▏        | 4841/41250 [11:42:23<88:01:55,  8.70s/it]
                                                          
{'loss': 0.1699, 'grad_norm': 1.4992388486862183, 'learning_rate': 3.920503233030309e-05, 'epoch': 1.17}

 12%|█▏        | 4841/41250 [11:42:23<88:01:55,  8.70s/it][2025-04-25 19:40:07,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:40:07,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2940.57 | bwd_microstep: 5899.36 | bwd_inner_microstep: 5886.45 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.55
[2025-04-25 19:40:07,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2940.57 | bwd: 5899.37 | bwd_inner: 5886.45 | bwd_allreduce: 12.88 | step: 18.55

 12%|█▏        | 4842/41250 [11:42:32<88:42:19,  8.77s/it]
                                                          
{'loss': 0.1389, 'grad_norm': 2.3708906173706055, 'learning_rate': 3.920459393700952e-05, 'epoch': 1.17}

 12%|█▏        | 4842/41250 [11:42:32<88:42:19,  8.77s/it][2025-04-25 19:40:15,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:40:15,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.36 | bwd_microstep: 5730.86 | bwd_inner_microstep: 5718.03 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.31
[2025-04-25 19:40:15,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.36 | bwd: 5730.88 | bwd_inner: 5718.03 | bwd_allreduce: 12.81 | step: 18.31

 12%|█▏        | 4843/41250 [11:42:41<88:23:52,  8.74s/it]
                                                          
{'loss': 0.3231, 'grad_norm': 3.4561588764190674, 'learning_rate': 3.92041554253234e-05, 'epoch': 1.17}

 12%|█▏        | 4843/41250 [11:42:41<88:23:52,  8.74s/it][2025-04-25 19:40:24,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:40:24,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.49 | bwd_microstep: 5730.32 | bwd_inner_microstep: 5717.51 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.28
[2025-04-25 19:40:24,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.49 | bwd: 5730.34 | bwd_inner: 5717.51 | bwd_allreduce: 12.78 | step: 18.28

 12%|█▏        | 4844/41250 [11:42:49<88:11:53,  8.72s/it]
                                                          
{'loss': 0.1212, 'grad_norm': 1.8072365522384644, 'learning_rate': 3.920371679524745e-05, 'epoch': 1.17}

 12%|█▏        | 4844/41250 [11:42:49<88:11:53,  8.72s/it][2025-04-25 19:40:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:40:33,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5765.49 | bwd_inner_microstep: 5664.58 | bwd_allreduce_microstep: 100.86 | step_microstep: 18.47
[2025-04-25 19:40:33,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.37 | bwd: 5765.50 | bwd_inner: 5664.58 | bwd_allreduce: 100.88 | step: 18.48

 12%|█▏        | 4845/41250 [11:42:58<88:07:30,  8.71s/it]
                                                          
{'loss': 0.0644, 'grad_norm': 1.0230131149291992, 'learning_rate': 3.920327804678436e-05, 'epoch': 1.17}

 12%|█▏        | 4845/41250 [11:42:58<88:07:30,  8.71s/it][2025-04-25 19:40:42,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:40:42,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.03 | bwd_microstep: 5751.64 | bwd_inner_microstep: 5696.45 | bwd_allreduce_microstep: 55.15 | step_microstep: 18.24
[2025-04-25 19:40:42,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.03 | bwd: 5751.65 | bwd_inner: 5696.45 | bwd_allreduce: 55.16 | step: 18.24

 12%|█▏        | 4846/41250 [11:43:07<88:03:15,  8.71s/it]
                                                          
{'loss': 0.1704, 'grad_norm': 2.146838426589966, 'learning_rate': 3.920283917993683e-05, 'epoch': 1.17}

 12%|█▏        | 4846/41250 [11:43:07<88:03:15,  8.71s/it][2025-04-25 19:40:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 19:40:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.73 | bwd_microstep: 5880.93 | bwd_inner_microstep: 5712.15 | bwd_allreduce_microstep: 168.72 | step_microstep: 19.07
[2025-04-25 19:40:50,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.73 | bwd: 5880.94 | bwd_inner: 5712.15 | bwd_allreduce: 168.74 | step: 19.08

 12%|█▏        | 4847/41250 [11:43:16<88:25:54,  8.75s/it]
                                                          
{'loss': 0.2343, 'grad_norm': 2.5090417861938477, 'learning_rate': 3.920240019470758e-05, 'epoch': 1.18}

 12%|█▏        | 4847/41250 [11:43:16<88:25:54,  8.75s/it][2025-04-25 19:40:59,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:40:59,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.80 | bwd_microstep: 5725.55 | bwd_inner_microstep: 5668.28 | bwd_allreduce_microstep: 57.23 | step_microstep: 18.26
[2025-04-25 19:40:59,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.80 | bwd: 5725.56 | bwd_inner: 5668.28 | bwd_allreduce: 57.25 | step: 18.27

 12%|█▏        | 4848/41250 [11:43:24<88:08:10,  8.72s/it]
                                                          
{'loss': 0.184, 'grad_norm': 2.1636621952056885, 'learning_rate': 3.920196109109931e-05, 'epoch': 1.18}

 12%|█▏        | 4848/41250 [11:43:24<88:08:10,  8.72s/it][2025-04-25 19:41:08,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:41:08,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.74 | bwd_microstep: 5751.83 | bwd_inner_microstep: 5692.99 | bwd_allreduce_microstep: 58.79 | step_microstep: 18.15
[2025-04-25 19:41:08,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.74 | bwd: 5751.84 | bwd_inner: 5692.99 | bwd_allreduce: 58.81 | step: 18.15

 12%|█▏        | 4849/41250 [11:43:33<88:03:09,  8.71s/it]
                                                          
{'loss': 0.0439, 'grad_norm': 1.6585344076156616, 'learning_rate': 3.920152186911472e-05, 'epoch': 1.18}

 12%|█▏        | 4849/41250 [11:43:33<88:03:09,  8.71s/it][2025-04-25 19:41:16,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-25 19:41:16,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.78 | bwd_microstep: 5722.75 | bwd_inner_microstep: 5709.15 | bwd_allreduce_microstep: 13.55 | step_microstep: 19.31
[2025-04-25 19:41:16,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.78 | bwd: 5722.77 | bwd_inner: 5709.15 | bwd_allreduce: 13.57 | step: 19.31

 12%|█▏        | 4850/41250 [11:43:42<87:55:59,  8.70s/it]
                                                          
{'loss': 0.0435, 'grad_norm': 1.4097739458084106, 'learning_rate': 3.920108252875653e-05, 'epoch': 1.18}

 12%|█▏        | 4850/41250 [11:43:42<87:55:59,  8.70s/it][2025-04-25 19:41:25,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:41:25,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.07 | bwd_microstep: 5763.84 | bwd_inner_microstep: 5713.34 | bwd_allreduce_microstep: 50.46 | step_microstep: 18.76
[2025-04-25 19:41:25,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.07 | bwd: 5763.86 | bwd_inner: 5713.34 | bwd_allreduce: 50.48 | step: 18.76

 12%|█▏        | 4851/41250 [11:43:50<87:57:41,  8.70s/it]
                                                          
{'loss': 0.0588, 'grad_norm': 0.9105182886123657, 'learning_rate': 3.920064307002744e-05, 'epoch': 1.18}

 12%|█▏        | 4851/41250 [11:43:50<87:57:41,  8.70s/it][2025-04-25 19:41:34,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:41:34,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.53 | bwd_microstep: 5724.10 | bwd_inner_microstep: 5668.29 | bwd_allreduce_microstep: 55.76 | step_microstep: 18.66
[2025-04-25 19:41:34,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.53 | bwd: 5724.11 | bwd_inner: 5668.29 | bwd_allreduce: 55.78 | step: 18.66

 12%|█▏        | 4852/41250 [11:43:59<87:48:37,  8.69s/it]
                                                          
{'loss': 0.1146, 'grad_norm': 1.9405802488327026, 'learning_rate': 3.920020349293017e-05, 'epoch': 1.18}

 12%|█▏        | 4852/41250 [11:43:59<87:48:37,  8.69s/it][2025-04-25 19:41:42,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 19:41:42,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.71 | bwd_microstep: 5774.02 | bwd_inner_microstep: 5666.09 | bwd_allreduce_microstep: 107.89 | step_microstep: 18.53
[2025-04-25 19:41:42,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.71 | bwd: 5774.03 | bwd_inner: 5666.09 | bwd_allreduce: 107.91 | step: 18.53

 12%|█▏        | 4853/41250 [11:44:08<87:50:41,  8.69s/it]
                                                          
{'loss': 0.1169, 'grad_norm': 0.988203227519989, 'learning_rate': 3.919976379746742e-05, 'epoch': 1.18}

 12%|█▏        | 4853/41250 [11:44:08<87:50:41,  8.69s/it][2025-04-25 19:41:51,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-25 19:41:51,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.70 | bwd_microstep: 5763.98 | bwd_inner_microstep: 5711.24 | bwd_allreduce_microstep: 52.69 | step_microstep: 18.58
[2025-04-25 19:41:51,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.70 | bwd: 5763.99 | bwd_inner: 5711.24 | bwd_allreduce: 52.71 | step: 18.58

 12%|█▏        | 4854/41250 [11:44:16<87:52:26,  8.69s/it]
                                                          
{'loss': 0.1423, 'grad_norm': 2.924574613571167, 'learning_rate': 3.91993239836419e-05, 'epoch': 1.18}

 12%|█▏        | 4854/41250 [11:44:16<87:52:26,  8.69s/it][2025-04-25 19:42:00,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:42:00,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.44 | bwd_microstep: 6021.68 | bwd_inner_microstep: 5705.87 | bwd_allreduce_microstep: 315.77 | step_microstep: 18.24
[2025-04-25 19:42:00,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.44 | bwd: 6021.69 | bwd_inner: 5705.87 | bwd_allreduce: 315.79 | step: 18.24

 12%|█▏        | 4855/41250 [11:44:25<88:42:20,  8.77s/it]
                                                          
{'loss': 0.0649, 'grad_norm': 1.0636889934539795, 'learning_rate': 3.919888405145632e-05, 'epoch': 1.18}

 12%|█▏        | 4855/41250 [11:44:25<88:42:20,  8.77s/it][2025-04-25 19:42:09,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 19:42:09,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.61 | bwd_microstep: 5769.92 | bwd_inner_microstep: 5715.07 | bwd_allreduce_microstep: 54.80 | step_microstep: 18.54
[2025-04-25 19:42:09,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.61 | bwd: 5769.94 | bwd_inner: 5715.07 | bwd_allreduce: 54.82 | step: 18.54

 12%|█▏        | 4856/41250 [11:44:34<88:31:08,  8.76s/it]
                                                          
{'loss': 0.119, 'grad_norm': 1.9209041595458984, 'learning_rate': 3.91984440009134e-05, 'epoch': 1.18}

 12%|█▏        | 4856/41250 [11:44:34<88:31:08,  8.76s/it][2025-04-25 19:42:18,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 19:42:18,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.36 | bwd_microstep: 5791.44 | bwd_inner_microstep: 5660.92 | bwd_allreduce_microstep: 130.47 | step_microstep: 18.90
[2025-04-25 19:42:18,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.36 | bwd: 5791.46 | bwd_inner: 5660.92 | bwd_allreduce: 130.49 | step: 18.90

 12%|█▏        | 4857/41250 [11:44:43<88:23:30,  8.74s/it]
                                                          
{'loss': 0.2512, 'grad_norm': 2.472579002380371, 'learning_rate': 3.919800383201585e-05, 'epoch': 1.18}

 12%|█▏        | 4857/41250 [11:44:43<88:23:30,  8.74s/it][2025-04-25 19:42:26,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 19:42:26,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.09 | bwd_microstep: 5904.08 | bwd_inner_microstep: 5639.81 | bwd_allreduce_microstep: 264.22 | step_microstep: 19.57
[2025-04-25 19:42:26,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.09 | bwd: 5904.10 | bwd_inner: 5639.81 | bwd_allreduce: 264.24 | step: 19.58

 12%|█▏        | 4858/41250 [11:44:52<88:35:01,  8.76s/it]
                                                          
{'loss': 0.2018, 'grad_norm': 2.426089286804199, 'learning_rate': 3.9197563544766385e-05, 'epoch': 1.18}

 12%|█▏        | 4858/41250 [11:44:52<88:35:01,  8.76s/it][2025-04-25 19:42:35,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 19:42:35,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5763.60 | bwd_inner_microstep: 5699.74 | bwd_allreduce_microstep: 63.83 | step_microstep: 18.60
[2025-04-25 19:42:35,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5763.62 | bwd_inner: 5699.73 | bwd_allreduce: 63.84 | step: 18.61

 12%|█▏        | 4859/41250 [11:45:00<88:22:44,  8.74s/it]
                                                          
{'loss': 0.1143, 'grad_norm': 1.447676420211792, 'learning_rate': 3.919712313916771e-05, 'epoch': 1.18}

 12%|█▏        | 4859/41250 [11:45:00<88:22:44,  8.74s/it][2025-04-25 19:42:44,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:42:44,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2779.41 | bwd_microstep: 5827.43 | bwd_inner_microstep: 5568.73 | bwd_allreduce_microstep: 258.66 | step_microstep: 18.48
[2025-04-25 19:42:44,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2779.41 | bwd: 5827.45 | bwd_inner: 5568.73 | bwd_allreduce: 258.68 | step: 18.48

 12%|█▏        | 4860/41250 [11:45:09<88:12:35,  8.73s/it]
                                                          
{'loss': 0.1222, 'grad_norm': 2.007612466812134, 'learning_rate': 3.919668261522255e-05, 'epoch': 1.18}

 12%|█▏        | 4860/41250 [11:45:09<88:12:35,  8.73s/it][2025-04-25 19:42:53,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:42:53,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.39 | bwd_microstep: 5875.68 | bwd_inner_microstep: 5863.01 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.43
[2025-04-25 19:42:53,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.39 | bwd: 5875.70 | bwd_inner: 5863.01 | bwd_allreduce: 12.65 | step: 18.43

 12%|█▏        | 4861/41250 [11:45:18<88:43:23,  8.78s/it]
                                                          
{'loss': 0.0912, 'grad_norm': 2.141772508621216, 'learning_rate': 3.9196241972933615e-05, 'epoch': 1.18}

 12%|█▏        | 4861/41250 [11:45:18<88:43:23,  8.78s/it][2025-04-25 19:43:01,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 19:43:01,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.25 | bwd_microstep: 5769.50 | bwd_inner_microstep: 5694.45 | bwd_allreduce_microstep: 75.00 | step_microstep: 19.31
[2025-04-25 19:43:01,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.25 | bwd: 5769.51 | bwd_inner: 5694.45 | bwd_allreduce: 75.02 | step: 19.31

 12%|█▏        | 4862/41250 [11:45:27<88:29:11,  8.75s/it]
                                                          
{'loss': 0.3301, 'grad_norm': 3.4105751514434814, 'learning_rate': 3.919580121230362e-05, 'epoch': 1.18}

 12%|█▏        | 4862/41250 [11:45:27<88:29:11,  8.75s/it][2025-04-25 19:43:10,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 19:43:10,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.95 | bwd_microstep: 5699.19 | bwd_inner_microstep: 5678.47 | bwd_allreduce_microstep: 20.68 | step_microstep: 18.49
[2025-04-25 19:43:10,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.95 | bwd: 5699.20 | bwd_inner: 5678.47 | bwd_allreduce: 20.70 | step: 18.49

 12%|█▏        | 4863/41250 [11:45:35<88:05:06,  8.71s/it]
                                                          
{'loss': 0.3549, 'grad_norm': 4.196229934692383, 'learning_rate': 3.919536033333529e-05, 'epoch': 1.18}

 12%|█▏        | 4863/41250 [11:45:35<88:05:06,  8.71s/it][2025-04-25 19:43:19,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:43:19,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5753.24 | bwd_inner_microstep: 5661.19 | bwd_allreduce_microstep: 92.01 | step_microstep: 18.55
[2025-04-25 19:43:19,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5753.26 | bwd_inner: 5661.19 | bwd_allreduce: 92.03 | step: 18.55

 12%|█▏        | 4864/41250 [11:45:44<87:55:14,  8.70s/it]
                                                          
{'loss': 0.2785, 'grad_norm': 1.8126400709152222, 'learning_rate': 3.9194919336031345e-05, 'epoch': 1.18}

 12%|█▏        | 4864/41250 [11:45:44<87:55:14,  8.70s/it][2025-04-25 19:43:27,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:43:27,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.41 | bwd_microstep: 5794.60 | bwd_inner_microstep: 5630.39 | bwd_allreduce_microstep: 164.17 | step_microstep: 18.51
[2025-04-25 19:43:27,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.41 | bwd: 5794.61 | bwd_inner: 5630.39 | bwd_allreduce: 164.19 | step: 18.52

 12%|█▏        | 4865/41250 [11:45:53<87:54:54,  8.70s/it]
                                                          
{'loss': 0.1567, 'grad_norm': 1.360508680343628, 'learning_rate': 3.919447822039449e-05, 'epoch': 1.18}

 12%|█▏        | 4865/41250 [11:45:53<87:54:54,  8.70s/it][2025-04-25 19:43:36,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:43:36,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.09 | bwd_microstep: 5704.15 | bwd_inner_microstep: 5691.48 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.67
[2025-04-25 19:43:36,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.09 | bwd: 5704.16 | bwd_inner: 5691.48 | bwd_allreduce: 12.64 | step: 18.67

 12%|█▏        | 4866/41250 [11:46:01<87:42:58,  8.68s/it]
                                                          
{'loss': 0.0555, 'grad_norm': 1.3309224843978882, 'learning_rate': 3.9194036986427455e-05, 'epoch': 1.18}

 12%|█▏        | 4866/41250 [11:46:01<87:42:58,  8.68s/it][2025-04-25 19:43:45,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:43:45,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5703.57 | bwd_inner_microstep: 5690.70 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.59
[2025-04-25 19:43:45,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5703.59 | bwd_inner: 5690.70 | bwd_allreduce: 12.85 | step: 18.59

 12%|█▏        | 4867/41250 [11:46:10<87:35:16,  8.67s/it]
                                                          
{'loss': 0.4712, 'grad_norm': 4.319969177246094, 'learning_rate': 3.919359563413296e-05, 'epoch': 1.18}

 12%|█▏        | 4867/41250 [11:46:10<87:35:16,  8.67s/it][2025-04-25 19:43:53,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:43:53,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.92 | bwd_microstep: 5680.60 | bwd_inner_microstep: 5656.29 | bwd_allreduce_microstep: 24.27 | step_microstep: 18.56
[2025-04-25 19:43:53,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.92 | bwd: 5680.61 | bwd_inner: 5656.29 | bwd_allreduce: 24.28 | step: 18.56

 12%|█▏        | 4868/41250 [11:46:18<87:21:37,  8.64s/it]
                                                          
{'loss': 0.1471, 'grad_norm': 1.6069906949996948, 'learning_rate': 3.919315416351372e-05, 'epoch': 1.18}

 12%|█▏        | 4868/41250 [11:46:19<87:21:37,  8.64s/it][2025-04-25 19:44:02,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 19:44:02,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.99 | bwd_microstep: 5711.45 | bwd_inner_microstep: 5698.62 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.05
[2025-04-25 19:44:02,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.99 | bwd: 5711.46 | bwd_inner: 5698.62 | bwd_allreduce: 12.80 | step: 19.06

 12%|█▏        | 4869/41250 [11:46:27<87:22:18,  8.65s/it]
                                                          
{'loss': 0.1987, 'grad_norm': 1.5827598571777344, 'learning_rate': 3.919271257457246e-05, 'epoch': 1.18}

 12%|█▏        | 4869/41250 [11:46:27<87:22:18,  8.65s/it][2025-04-25 19:44:10,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 19:44:10,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.01 | bwd_microstep: 5711.57 | bwd_inner_microstep: 5698.83 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.96
[2025-04-25 19:44:10,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.01 | bwd: 5711.59 | bwd_inner: 5698.83 | bwd_allreduce: 12.72 | step: 18.96

 12%|█▏        | 4870/41250 [11:46:36<87:23:10,  8.65s/it]
                                                          
{'loss': 0.2467, 'grad_norm': 3.1520044803619385, 'learning_rate': 3.9192270867311904e-05, 'epoch': 1.18}

 12%|█▏        | 4870/41250 [11:46:36<87:23:10,  8.65s/it][2025-04-25 19:44:19,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:44:19,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.02 | bwd_microstep: 5691.15 | bwd_inner_microstep: 5678.51 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.92
[2025-04-25 19:44:19,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.02 | bwd: 5691.16 | bwd_inner: 5678.51 | bwd_allreduce: 12.61 | step: 18.93

 12%|█▏        | 4871/41250 [11:46:44<87:18:23,  8.64s/it]
                                                          
{'loss': 0.1575, 'grad_norm': 1.5772602558135986, 'learning_rate': 3.919182904173477e-05, 'epoch': 1.18}

 12%|█▏        | 4871/41250 [11:46:44<87:18:23,  8.64s/it][2025-04-25 19:44:28,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.12 | optimizer_step: 0.94
[2025-04-25 19:44:28,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.31 | bwd_microstep: 5707.51 | bwd_inner_microstep: 5694.38 | bwd_allreduce_microstep: 13.09 | step_microstep: 19.02
[2025-04-25 19:44:28,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.31 | bwd: 5707.53 | bwd_inner: 5694.38 | bwd_allreduce: 13.11 | step: 19.03

 12%|█▏        | 4872/41250 [11:46:53<87:17:59,  8.64s/it]
                                                          
{'loss': 0.1882, 'grad_norm': 3.8629677295684814, 'learning_rate': 3.919138709784379e-05, 'epoch': 1.18}

 12%|█▏        | 4872/41250 [11:46:53<87:17:59,  8.64s/it][2025-04-25 19:44:36,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 19:44:36,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.22 | bwd_microstep: 5750.03 | bwd_inner_microstep: 5639.53 | bwd_allreduce_microstep: 110.45 | step_microstep: 18.74
[2025-04-25 19:44:36,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.22 | bwd: 5750.04 | bwd_inner: 5639.53 | bwd_allreduce: 110.47 | step: 18.74

 12%|█▏        | 4873/41250 [11:47:02<87:21:12,  8.64s/it]
                                                          
{'loss': 0.1097, 'grad_norm': 1.1184332370758057, 'learning_rate': 3.919094503564168e-05, 'epoch': 1.18}

 12%|█▏        | 4873/41250 [11:47:02<87:21:12,  8.64s/it][2025-04-25 19:44:45,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-25 19:44:45,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5736.96 | bwd_inner_microstep: 5651.85 | bwd_allreduce_microstep: 85.07 | step_microstep: 19.19
[2025-04-25 19:44:45,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5736.98 | bwd_inner: 5651.85 | bwd_allreduce: 85.09 | step: 19.20

 12%|█▏        | 4874/41250 [11:47:10<87:22:02,  8.65s/it]
                                                          
{'loss': 0.1172, 'grad_norm': 0.8755242228507996, 'learning_rate': 3.919050285513117e-05, 'epoch': 1.18}

 12%|█▏        | 4874/41250 [11:47:10<87:22:02,  8.65s/it][2025-04-25 19:44:54,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:44:54,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.87 | bwd_microstep: 5677.91 | bwd_inner_microstep: 5638.09 | bwd_allreduce_microstep: 39.78 | step_microstep: 18.50
[2025-04-25 19:44:54,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.87 | bwd: 5677.92 | bwd_inner: 5638.09 | bwd_allreduce: 39.79 | step: 18.50

 12%|█▏        | 4875/41250 [11:47:19<87:10:50,  8.63s/it]
                                                          
{'loss': 0.0412, 'grad_norm': 0.736634373664856, 'learning_rate': 3.919006055631498e-05, 'epoch': 1.18}

 12%|█▏        | 4875/41250 [11:47:19<87:10:50,  8.63s/it][2025-04-25 19:45:02,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:45:02,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.40 | bwd_microstep: 5717.19 | bwd_inner_microstep: 5704.51 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.67
[2025-04-25 19:45:02,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.40 | bwd: 5717.21 | bwd_inner: 5704.51 | bwd_allreduce: 12.65 | step: 18.67

 12%|█▏        | 4876/41250 [11:47:28<87:15:36,  8.64s/it]
                                                          
{'loss': 0.146, 'grad_norm': 1.6597696542739868, 'learning_rate': 3.918961813919584e-05, 'epoch': 1.18}

 12%|█▏        | 4876/41250 [11:47:28<87:15:36,  8.64s/it][2025-04-25 19:45:11,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 19:45:11,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5665.25 | bwd_inner_microstep: 5652.34 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.81
[2025-04-25 19:45:11,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5665.27 | bwd_inner: 5652.34 | bwd_allreduce: 12.88 | step: 18.81

 12%|█▏        | 4877/41250 [11:47:36<87:04:40,  8.62s/it]
                                                          
{'loss': 0.2474, 'grad_norm': 2.1404333114624023, 'learning_rate': 3.918917560377648e-05, 'epoch': 1.18}

 12%|█▏        | 4877/41250 [11:47:36<87:04:40,  8.62s/it][2025-04-25 19:45:19,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 19:45:19,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.72 | bwd_microstep: 5701.28 | bwd_inner_microstep: 5688.43 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.87
[2025-04-25 19:45:19,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.72 | bwd: 5701.29 | bwd_inner: 5688.43 | bwd_allreduce: 12.82 | step: 18.87

 12%|█▏        | 4878/41250 [11:47:45<87:05:54,  8.62s/it]
                                                          
{'loss': 0.0482, 'grad_norm': 1.0074851512908936, 'learning_rate': 3.918873295005963e-05, 'epoch': 1.18}

 12%|█▏        | 4878/41250 [11:47:45<87:05:54,  8.62s/it][2025-04-25 19:45:28,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 19:45:28,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.71 | bwd_microstep: 5739.30 | bwd_inner_microstep: 5641.95 | bwd_allreduce_microstep: 97.30 | step_microstep: 19.45
[2025-04-25 19:45:28,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.71 | bwd: 5739.32 | bwd_inner: 5641.95 | bwd_allreduce: 97.32 | step: 19.45

 12%|█▏        | 4879/41250 [11:47:53<87:10:24,  8.63s/it]
                                                          
{'loss': 0.0431, 'grad_norm': 0.6852734088897705, 'learning_rate': 3.918829017804802e-05, 'epoch': 1.18}

 12%|█▏        | 4879/41250 [11:47:53<87:10:24,  8.63s/it][2025-04-25 19:45:37,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:45:37,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.35 | bwd_microstep: 5739.05 | bwd_inner_microstep: 5644.03 | bwd_allreduce_microstep: 94.98 | step_microstep: 18.97
[2025-04-25 19:45:37,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.35 | bwd: 5739.07 | bwd_inner: 5644.03 | bwd_allreduce: 94.99 | step: 18.97

 12%|█▏        | 4880/41250 [11:48:02<87:13:52,  8.63s/it]
                                                          
{'loss': 0.1842, 'grad_norm': 1.0001435279846191, 'learning_rate': 3.918784728774436e-05, 'epoch': 1.18}

 12%|█▏        | 4880/41250 [11:48:02<87:13:52,  8.63s/it][2025-04-25 19:45:45,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 19:45:45,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.70 | bwd_microstep: 5670.33 | bwd_inner_microstep: 5656.19 | bwd_allreduce_microstep: 14.10 | step_microstep: 18.71
[2025-04-25 19:45:45,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.70 | bwd: 5670.34 | bwd_inner: 5656.19 | bwd_allreduce: 14.11 | step: 18.71

 12%|█▏        | 4881/41250 [11:48:11<87:04:03,  8.62s/it]
                                                          
{'loss': 0.0695, 'grad_norm': 1.0219918489456177, 'learning_rate': 3.918740427915141e-05, 'epoch': 1.18}

 12%|█▏        | 4881/41250 [11:48:11<87:04:03,  8.62s/it][2025-04-25 19:45:54,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:45:54,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.91 | bwd_microstep: 5740.45 | bwd_inner_microstep: 5636.77 | bwd_allreduce_microstep: 103.64 | step_microstep: 18.64
[2025-04-25 19:45:54,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.91 | bwd: 5740.47 | bwd_inner: 5636.77 | bwd_allreduce: 103.66 | step: 18.64

 12%|█▏        | 4882/41250 [11:48:19<87:08:21,  8.63s/it]
                                                          
{'loss': 0.2087, 'grad_norm': 2.4843990802764893, 'learning_rate': 3.918696115227188e-05, 'epoch': 1.18}

 12%|█▏        | 4882/41250 [11:48:19<87:08:21,  8.63s/it][2025-04-25 19:46:03,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 19:46:03,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.77 | bwd_microstep: 5744.44 | bwd_inner_microstep: 5648.22 | bwd_allreduce_microstep: 96.18 | step_microstep: 18.61
[2025-04-25 19:46:03,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.77 | bwd: 5744.46 | bwd_inner: 5648.22 | bwd_allreduce: 96.20 | step: 18.61

 12%|█▏        | 4883/41250 [11:48:28<87:13:42,  8.63s/it]
                                                          
{'loss': 0.1691, 'grad_norm': 2.2570009231567383, 'learning_rate': 3.9186517907108517e-05, 'epoch': 1.18}

 12%|█▏        | 4883/41250 [11:48:28<87:13:42,  8.63s/it][2025-04-25 19:46:11,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:46:11,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.95 | bwd_microstep: 5710.73 | bwd_inner_microstep: 5697.82 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.79
[2025-04-25 19:46:11,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.95 | bwd: 5710.74 | bwd_inner: 5697.82 | bwd_allreduce: 12.87 | step: 18.80

 12%|█▏        | 4884/41250 [11:48:37<87:17:26,  8.64s/it]
                                                          
{'loss': 0.2307, 'grad_norm': 2.4534101486206055, 'learning_rate': 3.9186074543664034e-05, 'epoch': 1.18}

 12%|█▏        | 4884/41250 [11:48:37<87:17:26,  8.64s/it][2025-04-25 19:46:20,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 19:46:20,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.15 | bwd_microstep: 5733.35 | bwd_inner_microstep: 5644.96 | bwd_allreduce_microstep: 88.35 | step_microstep: 18.70
[2025-04-25 19:46:20,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.15 | bwd: 5733.36 | bwd_inner: 5644.96 | bwd_allreduce: 88.36 | step: 18.70

 12%|█▏        | 4885/41250 [11:48:45<87:19:48,  8.65s/it]
                                                          
{'loss': 0.1183, 'grad_norm': 1.3105171918869019, 'learning_rate': 3.918563106194118e-05, 'epoch': 1.18}

 12%|█▏        | 4885/41250 [11:48:45<87:19:48,  8.65s/it][2025-04-25 19:46:29,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:46:29,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.30 | bwd_microstep: 5678.91 | bwd_inner_microstep: 5656.78 | bwd_allreduce_microstep: 22.09 | step_microstep: 18.53
[2025-04-25 19:46:29,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.30 | bwd: 5678.93 | bwd_inner: 5656.78 | bwd_allreduce: 22.11 | step: 18.54

 12%|█▏        | 4886/41250 [11:48:54<87:09:52,  8.63s/it]
                                                          
{'loss': 0.2375, 'grad_norm': 3.244279623031616, 'learning_rate': 3.918518746194268e-05, 'epoch': 1.18}

 12%|█▏        | 4886/41250 [11:48:54<87:09:52,  8.63s/it][2025-04-25 19:46:37,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:46:37,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.12 | bwd_microstep: 5733.31 | bwd_inner_microstep: 5691.24 | bwd_allreduce_microstep: 42.04 | step_microstep: 18.51
[2025-04-25 19:46:37,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.12 | bwd: 5733.33 | bwd_inner: 5691.24 | bwd_allreduce: 42.05 | step: 18.52

 12%|█▏        | 4887/41250 [11:49:03<87:17:44,  8.64s/it]
                                                          
{'loss': 0.0878, 'grad_norm': 1.354318618774414, 'learning_rate': 3.9184743743671274e-05, 'epoch': 1.18}

 12%|█▏        | 4887/41250 [11:49:03<87:17:44,  8.64s/it][2025-04-25 19:46:46,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 19:46:46,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.77 | bwd_microstep: 5708.49 | bwd_inner_microstep: 5695.91 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.50
[2025-04-25 19:46:46,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.77 | bwd: 5708.50 | bwd_inner: 5695.91 | bwd_allreduce: 12.55 | step: 18.51

 12%|█▏        | 4888/41250 [11:49:11<87:20:57,  8.65s/it]
                                                          
{'loss': 0.0717, 'grad_norm': 1.2301052808761597, 'learning_rate': 3.9184299907129705e-05, 'epoch': 1.18}

 12%|█▏        | 4888/41250 [11:49:11<87:20:57,  8.65s/it][2025-04-25 19:46:55,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 19:46:55,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.03 | bwd_microstep: 5765.22 | bwd_inner_microstep: 5650.54 | bwd_allreduce_microstep: 114.63 | step_microstep: 18.73
[2025-04-25 19:46:55,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.03 | bwd: 5765.23 | bwd_inner: 5650.54 | bwd_allreduce: 114.65 | step: 18.73

 12%|█▏        | 4889/41250 [11:49:20<87:25:25,  8.66s/it]
                                                          
{'loss': 0.2746, 'grad_norm': 3.211171865463257, 'learning_rate': 3.9183855952320694e-05, 'epoch': 1.19}

 12%|█▏        | 4889/41250 [11:49:20<87:25:25,  8.66s/it][2025-04-25 19:47:03,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 19:47:03,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5772.65 | bwd_inner_microstep: 5691.28 | bwd_allreduce_microstep: 81.33 | step_microstep: 18.83
[2025-04-25 19:47:03,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5772.66 | bwd_inner: 5691.28 | bwd_allreduce: 81.35 | step: 18.84

 12%|█▏        | 4890/41250 [11:49:29<87:34:19,  8.67s/it]
                                                          
{'loss': 0.0428, 'grad_norm': 1.417708396911621, 'learning_rate': 3.9183411879246984e-05, 'epoch': 1.19}

 12%|█▏        | 4890/41250 [11:49:29<87:34:19,  8.67s/it][2025-04-25 19:47:12,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:47:12,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.99 | bwd_microstep: 5860.27 | bwd_inner_microstep: 5700.78 | bwd_allreduce_microstep: 159.44 | step_microstep: 18.53
[2025-04-25 19:47:12,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.99 | bwd: 5860.28 | bwd_inner: 5700.78 | bwd_allreduce: 159.46 | step: 18.53

 12%|█▏        | 4891/41250 [11:49:37<87:57:38,  8.71s/it]
                                                          
{'loss': 0.081, 'grad_norm': 1.002457857131958, 'learning_rate': 3.918296768791131e-05, 'epoch': 1.19}

 12%|█▏        | 4891/41250 [11:49:37<87:57:38,  8.71s/it][2025-04-25 19:47:21,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 19:47:21,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.91 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5665.98 | bwd_allreduce_microstep: 97.37 | step_microstep: 18.73
[2025-04-25 19:47:21,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.92 | bwd: 5763.41 | bwd_inner: 5665.98 | bwd_allreduce: 97.39 | step: 18.74

 12%|█▏        | 4892/41250 [11:49:46<87:53:49,  8.70s/it]
                                                          
{'loss': 0.1733, 'grad_norm': 2.23846435546875, 'learning_rate': 3.918252337831641e-05, 'epoch': 1.19}

 12%|█▏        | 4892/41250 [11:49:46<87:53:49,  8.70s/it][2025-04-25 19:47:29,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 19:47:29,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.30 | bwd_microstep: 5733.90 | bwd_inner_microstep: 5720.90 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.11
[2025-04-25 19:47:29,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.30 | bwd: 5733.91 | bwd_inner: 5720.90 | bwd_allreduce: 12.97 | step: 19.11

 12%|█▏        | 4893/41250 [11:49:55<87:48:06,  8.69s/it]
                                                          
{'loss': 0.2027, 'grad_norm': 2.3698770999908447, 'learning_rate': 3.9182078950465034e-05, 'epoch': 1.19}

 12%|█▏        | 4893/41250 [11:49:55<87:48:06,  8.69s/it][2025-04-25 19:47:38,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 19:47:38,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5709.62 | bwd_inner_microstep: 5696.81 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.01
[2025-04-25 19:47:38,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.70 | bwd: 5709.64 | bwd_inner: 5696.81 | bwd_allreduce: 12.78 | step: 19.02

 12%|█▏        | 4894/41250 [11:50:03<87:38:09,  8.68s/it]
                                                          
{'loss': 0.1133, 'grad_norm': 1.3070080280303955, 'learning_rate': 3.918163440435992e-05, 'epoch': 1.19}

 12%|█▏        | 4894/41250 [11:50:03<87:38:09,  8.68s/it][2025-04-25 19:47:47,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 19:47:47,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.83 | bwd_microstep: 5716.55 | bwd_inner_microstep: 5659.11 | bwd_allreduce_microstep: 57.39 | step_microstep: 19.10
[2025-04-25 19:47:47,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.83 | bwd: 5716.56 | bwd_inner: 5659.11 | bwd_allreduce: 57.41 | step: 19.10

 12%|█▏        | 4895/41250 [11:50:12<87:29:26,  8.66s/it]
                                                          
{'loss': 0.2731, 'grad_norm': 3.3088297843933105, 'learning_rate': 3.918118974000379e-05, 'epoch': 1.19}

 12%|█▏        | 4895/41250 [11:50:12<87:29:26,  8.66s/it][2025-04-25 19:47:55,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:47:55,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.61 | bwd_microstep: 5724.09 | bwd_inner_microstep: 5711.23 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.84
[2025-04-25 19:47:55,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.61 | bwd: 5724.11 | bwd_inner: 5711.23 | bwd_allreduce: 12.83 | step: 18.84

 12%|█▏        | 4896/41250 [11:50:21<87:28:45,  8.66s/it]
                                                          
{'loss': 0.1675, 'grad_norm': 0.9338627457618713, 'learning_rate': 3.9180744957399405e-05, 'epoch': 1.19}

 12%|█▏        | 4896/41250 [11:50:21<87:28:45,  8.66s/it][2025-04-25 19:48:04,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:48:04,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5757.06 | bwd_inner_microstep: 5705.94 | bwd_allreduce_microstep: 51.07 | step_microstep: 18.56
[2025-04-25 19:48:04,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5757.07 | bwd_inner: 5705.94 | bwd_allreduce: 51.09 | step: 18.57

 12%|█▏        | 4897/41250 [11:50:29<87:33:44,  8.67s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 2.3074493408203125, 'learning_rate': 3.918030005654949e-05, 'epoch': 1.19}

 12%|█▏        | 4897/41250 [11:50:29<87:33:44,  8.67s/it][2025-04-25 19:48:13,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 19:48:13,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.01 | bwd_microstep: 5719.22 | bwd_inner_microstep: 5669.19 | bwd_allreduce_microstep: 49.99 | step_microstep: 19.06
[2025-04-25 19:48:13,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.01 | bwd: 5719.23 | bwd_inner: 5669.19 | bwd_allreduce: 50.00 | step: 19.07

 12%|█▏        | 4898/41250 [11:50:38<87:27:40,  8.66s/it]
                                                          
{'loss': 0.0113, 'grad_norm': 0.308078795671463, 'learning_rate': 3.917985503745682e-05, 'epoch': 1.19}

 12%|█▏        | 4898/41250 [11:50:38<87:27:40,  8.66s/it][2025-04-25 19:48:21,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.22 | optimizer_step: 0.92
[2025-04-25 19:48:21,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.74 | bwd_microstep: 5753.35 | bwd_inner_microstep: 5717.36 | bwd_allreduce_microstep: 35.93 | step_microstep: 19.39
[2025-04-25 19:48:21,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.74 | bwd: 5753.36 | bwd_inner: 5717.36 | bwd_allreduce: 35.95 | step: 19.40

 12%|█▏        | 4899/41250 [11:50:47<87:33:11,  8.67s/it]
                                                          
{'loss': 0.1515, 'grad_norm': 1.2012324333190918, 'learning_rate': 3.9179409900124096e-05, 'epoch': 1.19}

 12%|█▏        | 4899/41250 [11:50:47<87:33:11,  8.67s/it][2025-04-25 19:48:30,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 19:48:30,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5795.73 | bwd_inner_microstep: 5668.36 | bwd_allreduce_microstep: 127.32 | step_microstep: 19.06
[2025-04-25 19:48:30,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5795.74 | bwd_inner: 5668.36 | bwd_allreduce: 127.34 | step: 19.06

 12%|█▏        | 4900/41250 [11:50:55<87:39:54,  8.68s/it]
                                                          
{'loss': 0.126, 'grad_norm': 2.362168073654175, 'learning_rate': 3.917896464455409e-05, 'epoch': 1.19}

 12%|█▏        | 4900/41250 [11:50:55<87:39:54,  8.68s/it][2025-04-25 19:48:39,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:48:39,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.88 | bwd_microstep: 5723.16 | bwd_inner_microstep: 5670.29 | bwd_allreduce_microstep: 52.82 | step_microstep: 18.79
[2025-04-25 19:48:39,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.88 | bwd: 5723.17 | bwd_inner: 5670.29 | bwd_allreduce: 52.84 | step: 18.79

 12%|█▏        | 4901/41250 [11:51:04<87:32:44,  8.67s/it]
                                                          
{'loss': 0.1805, 'grad_norm': 2.7314343452453613, 'learning_rate': 3.917851927074954e-05, 'epoch': 1.19}

 12%|█▏        | 4901/41250 [11:51:04<87:32:44,  8.67s/it][2025-04-25 19:48:47,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:48:47,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.54 | bwd_microstep: 5802.10 | bwd_inner_microstep: 5663.96 | bwd_allreduce_microstep: 138.10 | step_microstep: 18.84
[2025-04-25 19:48:47,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.54 | bwd: 5802.12 | bwd_inner: 5663.96 | bwd_allreduce: 138.12 | step: 18.84

 12%|█▏        | 4902/41250 [11:51:13<87:41:09,  8.68s/it]
                                                          
{'loss': 0.0999, 'grad_norm': 2.5192489624023438, 'learning_rate': 3.917807377871319e-05, 'epoch': 1.19}

 12%|█▏        | 4902/41250 [11:51:13<87:41:09,  8.68s/it][2025-04-25 19:48:56,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 19:48:56,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.41 | bwd_microstep: 5787.71 | bwd_inner_microstep: 5656.44 | bwd_allreduce_microstep: 131.22 | step_microstep: 19.05
[2025-04-25 19:48:56,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.41 | bwd: 5787.72 | bwd_inner: 5656.44 | bwd_allreduce: 131.24 | step: 19.06

 12%|█▏        | 4903/41250 [11:51:21<87:44:24,  8.69s/it]
                                                          
{'loss': 0.0951, 'grad_norm': 1.5211549997329712, 'learning_rate': 3.917762816844779e-05, 'epoch': 1.19}

 12%|█▏        | 4903/41250 [11:51:21<87:44:24,  8.69s/it][2025-04-25 19:49:05,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:49:05,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.52 | bwd_microstep: 5729.27 | bwd_inner_microstep: 5661.24 | bwd_allreduce_microstep: 67.98 | step_microstep: 19.06
[2025-04-25 19:49:05,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.52 | bwd: 5729.28 | bwd_inner: 5661.24 | bwd_allreduce: 68.00 | step: 19.06

 12%|█▏        | 4904/41250 [11:51:30<87:37:37,  8.68s/it]
                                                          
{'loss': 0.0529, 'grad_norm': 0.7636629939079285, 'learning_rate': 3.9177182439956074e-05, 'epoch': 1.19}

 12%|█▏        | 4904/41250 [11:51:30<87:37:37,  8.68s/it][2025-04-25 19:49:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:49:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.87 | bwd_microstep: 5872.92 | bwd_inner_microstep: 5715.32 | bwd_allreduce_microstep: 157.55 | step_microstep: 19.05
[2025-04-25 19:49:14,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.87 | bwd: 5872.93 | bwd_inner: 5715.32 | bwd_allreduce: 157.57 | step: 19.05

 12%|█▏        | 4905/41250 [11:51:39<88:01:39,  8.72s/it]
                                                          
{'loss': 0.2894, 'grad_norm': 7.451047897338867, 'learning_rate': 3.917673659324081e-05, 'epoch': 1.19}

 12%|█▏        | 4905/41250 [11:51:39<88:01:39,  8.72s/it][2025-04-25 19:49:22,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 19:49:22,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.37 | bwd_microstep: 5778.42 | bwd_inner_microstep: 5692.84 | bwd_allreduce_microstep: 85.54 | step_microstep: 18.52
[2025-04-25 19:49:22,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.37 | bwd: 5778.44 | bwd_inner: 5692.84 | bwd_allreduce: 85.56 | step: 18.53

 12%|█▏        | 4906/41250 [11:51:48<87:59:39,  8.72s/it]
                                                          
{'loss': 0.0939, 'grad_norm': 0.8279299139976501, 'learning_rate': 3.917629062830473e-05, 'epoch': 1.19}

 12%|█▏        | 4906/41250 [11:51:48<87:59:39,  8.72s/it][2025-04-25 19:49:31,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 19:49:31,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5776.23 | bwd_inner_microstep: 5647.08 | bwd_allreduce_microstep: 129.11 | step_microstep: 19.00
[2025-04-25 19:49:31,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5776.25 | bwd_inner: 5647.08 | bwd_allreduce: 129.13 | step: 19.00

 12%|█▏        | 4907/41250 [11:51:56<87:54:49,  8.71s/it]
                                                          
{'loss': 0.3709, 'grad_norm': 6.58607292175293, 'learning_rate': 3.9175844545150593e-05, 'epoch': 1.19}

 12%|█▏        | 4907/41250 [11:51:56<87:54:49,  8.71s/it][2025-04-25 19:49:40,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:49:40,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.90 | bwd_microstep: 5720.83 | bwd_inner_microstep: 5707.83 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.67
[2025-04-25 19:49:40,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.90 | bwd: 5720.84 | bwd_inner: 5707.83 | bwd_allreduce: 12.97 | step: 18.67

 12%|█▏        | 4908/41250 [11:52:05<87:46:11,  8.69s/it]
                                                          
{'loss': 0.0356, 'grad_norm': 0.37198176980018616, 'learning_rate': 3.917539834378115e-05, 'epoch': 1.19}

 12%|█▏        | 4908/41250 [11:52:05<87:46:11,  8.69s/it][2025-04-25 19:49:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 19:49:48,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.27 | bwd_microstep: 5808.64 | bwd_inner_microstep: 5654.10 | bwd_allreduce_microstep: 154.49 | step_microstep: 18.71
[2025-04-25 19:49:48,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.27 | bwd: 5808.65 | bwd_inner: 5654.10 | bwd_allreduce: 154.50 | step: 18.71

 12%|█▏        | 4909/41250 [11:52:14<87:49:56,  8.70s/it]
                                                          
{'loss': 0.1397, 'grad_norm': 2.2528231143951416, 'learning_rate': 3.917495202419914e-05, 'epoch': 1.19}

 12%|█▏        | 4909/41250 [11:52:14<87:49:56,  8.70s/it][2025-04-25 19:49:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:49:57,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.77 | bwd_microstep: 5727.05 | bwd_inner_microstep: 5664.86 | bwd_allreduce_microstep: 62.14 | step_microstep: 18.74
[2025-04-25 19:49:57,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.77 | bwd: 5727.06 | bwd_inner: 5664.86 | bwd_allreduce: 62.16 | step: 18.74

 12%|█▏        | 4910/41250 [11:52:22<87:38:19,  8.68s/it]
                                                          
{'loss': 0.2077, 'grad_norm': 1.0547384023666382, 'learning_rate': 3.917450558640733e-05, 'epoch': 1.19}

 12%|█▏        | 4910/41250 [11:52:22<87:38:19,  8.68s/it][2025-04-25 19:50:06,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:50:06,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.92 | bwd_microstep: 5765.83 | bwd_inner_microstep: 5665.06 | bwd_allreduce_microstep: 100.72 | step_microstep: 18.45
[2025-04-25 19:50:06,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.92 | bwd: 5765.84 | bwd_inner: 5665.06 | bwd_allreduce: 100.74 | step: 18.45

 12%|█▏        | 4911/41250 [11:52:31<87:38:03,  8.68s/it]
                                                          
{'loss': 0.0883, 'grad_norm': 1.2049721479415894, 'learning_rate': 3.917405903040846e-05, 'epoch': 1.19}

 12%|█▏        | 4911/41250 [11:52:31<87:38:03,  8.68s/it][2025-04-25 19:50:14,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 19:50:14,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.65 | bwd_microstep: 5768.13 | bwd_inner_microstep: 5689.63 | bwd_allreduce_microstep: 78.45 | step_microstep: 19.19
[2025-04-25 19:50:14,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.65 | bwd: 5768.15 | bwd_inner: 5689.63 | bwd_allreduce: 78.47 | step: 19.19

 12%|█▏        | 4912/41250 [11:52:40<87:41:17,  8.69s/it]
                                                          
{'loss': 0.3934, 'grad_norm': 4.98158597946167, 'learning_rate': 3.9173612356205286e-05, 'epoch': 1.19}

 12%|█▏        | 4912/41250 [11:52:40<87:41:17,  8.69s/it][2025-04-25 19:50:23,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:50:23,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.66 | bwd_microstep: 5711.66 | bwd_inner_microstep: 5658.21 | bwd_allreduce_microstep: 53.40 | step_microstep: 18.58
[2025-04-25 19:50:23,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.66 | bwd: 5711.67 | bwd_inner: 5658.21 | bwd_allreduce: 53.42 | step: 18.58

 12%|█▏        | 4913/41250 [11:52:48<87:29:39,  8.67s/it]
                                                          
{'loss': 0.3064, 'grad_norm': 3.220306873321533, 'learning_rate': 3.917316556380056e-05, 'epoch': 1.19}

 12%|█▏        | 4913/41250 [11:52:48<87:29:39,  8.67s/it][2025-04-25 19:50:32,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:50:32,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.81 | bwd_microstep: 5788.37 | bwd_inner_microstep: 5775.59 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.57
[2025-04-25 19:50:32,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.81 | bwd: 5788.38 | bwd_inner: 5775.59 | bwd_allreduce: 12.75 | step: 18.57

 12%|█▏        | 4914/41250 [11:52:57<87:45:51,  8.70s/it]
                                                          
{'loss': 0.1036, 'grad_norm': 1.3132442235946655, 'learning_rate': 3.9172718653197055e-05, 'epoch': 1.19}

 12%|█▏        | 4914/41250 [11:52:57<87:45:51,  8.70s/it][2025-04-25 19:50:40,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:50:40,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5698.18 | bwd_inner_microstep: 5654.26 | bwd_allreduce_microstep: 43.88 | step_microstep: 18.77
[2025-04-25 19:50:40,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5698.20 | bwd_inner: 5654.26 | bwd_allreduce: 43.90 | step: 18.77

 12%|█▏        | 4915/41250 [11:53:06<87:30:33,  8.67s/it]
                                                          
{'loss': 0.0947, 'grad_norm': 1.7092684507369995, 'learning_rate': 3.91722716243975e-05, 'epoch': 1.19}

 12%|█▏        | 4915/41250 [11:53:06<87:30:33,  8.67s/it][2025-04-25 19:50:49,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 19:50:49,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5739.07 | bwd_inner_microstep: 5702.39 | bwd_allreduce_microstep: 36.63 | step_microstep: 19.16
[2025-04-25 19:50:49,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5739.08 | bwd_inner: 5702.39 | bwd_allreduce: 36.65 | step: 19.16

 12%|█▏        | 4916/41250 [11:53:14<87:31:19,  8.67s/it]
                                                          
{'loss': 0.122, 'grad_norm': 2.2786474227905273, 'learning_rate': 3.917182447740466e-05, 'epoch': 1.19}

 12%|█▏        | 4916/41250 [11:53:14<87:31:19,  8.67s/it][2025-04-25 19:50:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 19:50:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.32 | bwd_microstep: 5691.58 | bwd_inner_microstep: 5657.71 | bwd_allreduce_microstep: 33.81 | step_microstep: 19.12
[2025-04-25 19:50:58,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.32 | bwd: 5691.59 | bwd_inner: 5657.71 | bwd_allreduce: 33.84 | step: 19.12

 12%|█▏        | 4917/41250 [11:53:23<87:18:10,  8.65s/it]
                                                          
{'loss': 0.2152, 'grad_norm': 2.255427598953247, 'learning_rate': 3.91713772122213e-05, 'epoch': 1.19}

 12%|█▏        | 4917/41250 [11:53:23<87:18:10,  8.65s/it][2025-04-25 19:51:06,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:51:06,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.83 | bwd_microstep: 5879.07 | bwd_inner_microstep: 5688.41 | bwd_allreduce_microstep: 190.62 | step_microstep: 18.39
[2025-04-25 19:51:06,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5879.09 | bwd_inner: 5688.41 | bwd_allreduce: 190.63 | step: 18.39

 12%|█▏        | 4918/41250 [11:53:32<87:45:02,  8.69s/it]
                                                          
{'loss': 0.1935, 'grad_norm': 1.3475937843322754, 'learning_rate': 3.917092982885016e-05, 'epoch': 1.19}

 12%|█▏        | 4918/41250 [11:53:32<87:45:02,  8.69s/it][2025-04-25 19:51:15,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:51:15,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5694.09 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.86
[2025-04-25 19:51:15,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.17 | bwd: 5706.79 | bwd_inner: 5694.09 | bwd_allreduce: 12.66 | step: 18.86

 12%|█▏        | 4919/41250 [11:53:40<87:34:02,  8.68s/it]
                                                          
{'loss': 0.0497, 'grad_norm': 1.1688164472579956, 'learning_rate': 3.9170482327294017e-05, 'epoch': 1.19}

 12%|█▏        | 4919/41250 [11:53:40<87:34:02,  8.68s/it][2025-04-25 19:51:24,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 19:51:24,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5760.00 | bwd_inner_microstep: 5688.50 | bwd_allreduce_microstep: 71.46 | step_microstep: 18.62
[2025-04-25 19:51:24,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5760.01 | bwd_inner: 5688.50 | bwd_allreduce: 71.48 | step: 18.62

 12%|█▏        | 4920/41250 [11:53:49<87:36:58,  8.68s/it]
                                                          
{'loss': 0.0609, 'grad_norm': 0.6372210383415222, 'learning_rate': 3.9170034707555616e-05, 'epoch': 1.19}

 12%|█▏        | 4920/41250 [11:53:49<87:36:58,  8.68s/it][2025-04-25 19:51:32,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.35 | optimizer_step: 1.03
[2025-04-25 19:51:32,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.65 | bwd_microstep: 5712.98 | bwd_inner_microstep: 5700.25 | bwd_allreduce_microstep: 12.68 | step_microstep: 20.69
[2025-04-25 19:51:32,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.65 | bwd: 5712.99 | bwd_inner: 5700.25 | bwd_allreduce: 12.70 | step: 20.70

 12%|█▏        | 4921/41250 [11:53:58<87:31:18,  8.67s/it]
                                                          
{'loss': 0.1985, 'grad_norm': 1.504049301147461, 'learning_rate': 3.916958696963773e-05, 'epoch': 1.19}

 12%|█▏        | 4921/41250 [11:53:58<87:31:18,  8.67s/it][2025-04-25 19:51:41,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:51:41,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5760.29 | bwd_inner_microstep: 5702.77 | bwd_allreduce_microstep: 57.48 | step_microstep: 19.14
[2025-04-25 19:51:41,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5760.31 | bwd_inner: 5702.77 | bwd_allreduce: 57.50 | step: 19.14

 12%|█▏        | 4922/41250 [11:54:06<87:34:28,  8.68s/it]
                                                          
{'loss': 0.1123, 'grad_norm': 2.293336868286133, 'learning_rate': 3.91691391135431e-05, 'epoch': 1.19}

 12%|█▏        | 4922/41250 [11:54:06<87:34:28,  8.68s/it][2025-04-25 19:51:50,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:51:50,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5765.42 | bwd_inner_microstep: 5655.87 | bwd_allreduce_microstep: 109.50 | step_microstep: 18.89
[2025-04-25 19:51:50,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.75 | bwd: 5765.44 | bwd_inner: 5655.87 | bwd_allreduce: 109.52 | step: 18.89

 12%|█▏        | 4923/41250 [11:54:15<87:34:16,  8.68s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 1.3378359079360962, 'learning_rate': 3.9168691139274495e-05, 'epoch': 1.19}

 12%|█▏        | 4923/41250 [11:54:15<87:34:16,  8.68s/it][2025-04-25 19:51:58,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:51:58,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.73 | bwd_microstep: 5750.05 | bwd_inner_microstep: 5635.33 | bwd_allreduce_microstep: 114.67 | step_microstep: 18.60
[2025-04-25 19:51:58,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.73 | bwd: 5750.06 | bwd_inner: 5635.33 | bwd_allreduce: 114.69 | step: 18.61

 12%|█▏        | 4924/41250 [11:54:24<87:29:32,  8.67s/it]
                                                          
{'loss': 0.0467, 'grad_norm': 0.7870063781738281, 'learning_rate': 3.916824304683469e-05, 'epoch': 1.19}

 12%|█▏        | 4924/41250 [11:54:24<87:29:32,  8.67s/it][2025-04-25 19:52:07,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:52:07,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.72 | bwd_microstep: 5739.28 | bwd_inner_microstep: 5698.71 | bwd_allreduce_microstep: 40.52 | step_microstep: 18.62
[2025-04-25 19:52:07,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.72 | bwd: 5739.29 | bwd_inner: 5698.71 | bwd_allreduce: 40.54 | step: 18.62

 12%|█▏        | 4925/41250 [11:54:32<87:29:50,  8.67s/it]
                                                          
{'loss': 0.2576, 'grad_norm': 2.001943826675415, 'learning_rate': 3.916779483622644e-05, 'epoch': 1.19}

 12%|█▏        | 4925/41250 [11:54:32<87:29:50,  8.67s/it][2025-04-25 19:52:16,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 1.11
[2025-04-25 19:52:16,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5863.08 | bwd_inner_microstep: 5645.62 | bwd_allreduce_microstep: 217.40 | step_microstep: 20.16
[2025-04-25 19:52:16,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.92 | bwd: 5863.10 | bwd_inner: 5645.62 | bwd_allreduce: 217.43 | step: 20.16

 12%|█▏        | 4926/41250 [11:54:41<87:47:51,  8.70s/it]
                                                          
{'loss': 0.1806, 'grad_norm': 2.6412672996520996, 'learning_rate': 3.916734650745249e-05, 'epoch': 1.19}

 12%|█▏        | 4926/41250 [11:54:41<87:47:51,  8.70s/it][2025-04-25 19:52:25,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:52:25,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.37 | bwd_microstep: 5726.92 | bwd_inner_microstep: 5685.06 | bwd_allreduce_microstep: 41.82 | step_microstep: 18.02
[2025-04-25 19:52:25,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.37 | bwd: 5726.93 | bwd_inner: 5685.06 | bwd_allreduce: 41.84 | step: 18.02

 12%|█▏        | 4927/41250 [11:54:50<87:39:12,  8.69s/it]
                                                          
{'loss': 0.1825, 'grad_norm': 2.1910688877105713, 'learning_rate': 3.9166898060515626e-05, 'epoch': 1.19}

 12%|█▏        | 4927/41250 [11:54:50<87:39:12,  8.69s/it][2025-04-25 19:52:33,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:52:33,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5721.57 | bwd_inner_microstep: 5691.29 | bwd_allreduce_microstep: 30.24 | step_microstep: 18.81
[2025-04-25 19:52:33,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5721.59 | bwd_inner: 5691.29 | bwd_allreduce: 30.26 | step: 18.81

 12%|█▏        | 4928/41250 [11:54:59<87:32:53,  8.68s/it]
                                                          
{'loss': 0.0725, 'grad_norm': 1.0185788869857788, 'learning_rate': 3.9166449495418606e-05, 'epoch': 1.19}

 12%|█▏        | 4928/41250 [11:54:59<87:32:53,  8.68s/it][2025-04-25 19:52:42,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.91
[2025-04-25 19:52:42,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.13 | bwd_microstep: 5671.86 | bwd_inner_microstep: 5642.00 | bwd_allreduce_microstep: 29.83 | step_microstep: 18.38
[2025-04-25 19:52:42,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.13 | bwd: 5671.88 | bwd_inner: 5642.00 | bwd_allreduce: 29.84 | step: 18.39

 12%|█▏        | 4929/41250 [11:55:07<87:15:38,  8.65s/it]
                                                          
{'loss': 0.3219, 'grad_norm': 3.0213778018951416, 'learning_rate': 3.9166000812164196e-05, 'epoch': 1.19}

 12%|█▏        | 4929/41250 [11:55:07<87:15:38,  8.65s/it][2025-04-25 19:52:50,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:52:50,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.29 | bwd_microstep: 5686.67 | bwd_inner_microstep: 5643.32 | bwd_allreduce_microstep: 43.30 | step_microstep: 18.60
[2025-04-25 19:52:50,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.30 | bwd: 5686.68 | bwd_inner: 5643.32 | bwd_allreduce: 43.32 | step: 18.60

 12%|█▏        | 4930/41250 [11:55:16<87:05:22,  8.63s/it]
                                                          
{'loss': 0.0907, 'grad_norm': 1.049283742904663, 'learning_rate': 3.916555201075516e-05, 'epoch': 1.2}

 12%|█▏        | 4930/41250 [11:55:16<87:05:22,  8.63s/it][2025-04-25 19:52:59,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:52:59,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.66 | bwd_microstep: 5712.79 | bwd_inner_microstep: 5700.08 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.34
[2025-04-25 19:52:59,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.66 | bwd: 5712.80 | bwd_inner: 5700.08 | bwd_allreduce: 12.69 | step: 18.34

 12%|█▏        | 4931/41250 [11:55:24<87:07:29,  8.64s/it]
                                                          
{'loss': 0.0839, 'grad_norm': 1.096662998199463, 'learning_rate': 3.916510309119427e-05, 'epoch': 1.2}

 12%|█▏        | 4931/41250 [11:55:24<87:07:29,  8.64s/it][2025-04-25 19:53:08,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 19:53:08,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 5722.52 | bwd_inner_microstep: 5697.11 | bwd_allreduce_microstep: 25.36 | step_microstep: 18.69
[2025-04-25 19:53:08,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 5722.53 | bwd_inner: 5697.11 | bwd_allreduce: 25.37 | step: 18.69

 12%|█▏        | 4932/41250 [11:55:33<87:10:17,  8.64s/it]
                                                          
{'loss': 0.2855, 'grad_norm': 2.4025378227233887, 'learning_rate': 3.916465405348428e-05, 'epoch': 1.2}

 12%|█▏        | 4932/41250 [11:55:33<87:10:17,  8.64s/it][2025-04-25 19:53:16,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-25 19:53:16,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.23 | bwd_microstep: 5760.14 | bwd_inner_microstep: 5655.44 | bwd_allreduce_microstep: 104.65 | step_microstep: 19.36
[2025-04-25 19:53:16,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.23 | bwd: 5760.15 | bwd_inner: 5655.44 | bwd_allreduce: 104.67 | step: 19.36

 12%|█▏        | 4933/41250 [11:55:42<87:16:14,  8.65s/it]
                                                          
{'loss': 0.0647, 'grad_norm': 1.373764157295227, 'learning_rate': 3.916420489762797e-05, 'epoch': 1.2}

 12%|█▏        | 4933/41250 [11:55:42<87:16:14,  8.65s/it][2025-04-25 19:53:25,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:53:25,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.39 | bwd_microstep: 5742.04 | bwd_inner_microstep: 5689.95 | bwd_allreduce_microstep: 52.05 | step_microstep: 18.77
[2025-04-25 19:53:25,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.39 | bwd: 5742.05 | bwd_inner: 5689.95 | bwd_allreduce: 52.07 | step: 18.77

 12%|█▏        | 4934/41250 [11:55:50<87:20:41,  8.66s/it]
                                                          
{'loss': 0.1918, 'grad_norm': 1.9404672384262085, 'learning_rate': 3.9163755623628105e-05, 'epoch': 1.2}

 12%|█▏        | 4934/41250 [11:55:50<87:20:41,  8.66s/it][2025-04-25 19:53:34,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:53:34,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5740.42 | bwd_inner_microstep: 5654.75 | bwd_allreduce_microstep: 85.63 | step_microstep: 18.85
[2025-04-25 19:53:34,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5740.44 | bwd_inner: 5654.75 | bwd_allreduce: 85.65 | step: 18.85

 12%|█▏        | 4935/41250 [11:55:59<87:19:52,  8.66s/it]
                                                          
{'loss': 0.0986, 'grad_norm': 1.6045273542404175, 'learning_rate': 3.916330623148746e-05, 'epoch': 1.2}

 12%|█▏        | 4935/41250 [11:55:59<87:19:52,  8.66s/it][2025-04-25 19:53:42,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 19:53:42,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.75 | bwd_microstep: 5704.98 | bwd_inner_microstep: 5656.64 | bwd_allreduce_microstep: 48.30 | step_microstep: 18.65
[2025-04-25 19:53:42,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.75 | bwd: 5704.99 | bwd_inner: 5656.64 | bwd_allreduce: 48.31 | step: 18.66

 12%|█▏        | 4936/41250 [11:56:08<87:11:53,  8.64s/it]
                                                          
{'loss': 0.1489, 'grad_norm': 1.9522737264633179, 'learning_rate': 3.91628567212088e-05, 'epoch': 1.2}

 12%|█▏        | 4936/41250 [11:56:08<87:11:53,  8.64s/it][2025-04-25 19:53:51,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:53:51,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5753.81 | bwd_inner_microstep: 5708.33 | bwd_allreduce_microstep: 45.43 | step_microstep: 18.56
[2025-04-25 19:53:51,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5753.82 | bwd_inner: 5708.33 | bwd_allreduce: 45.45 | step: 18.56

 12%|█▏        | 4937/41250 [11:56:16<87:19:19,  8.66s/it]
                                                          
{'loss': 0.323, 'grad_norm': 2.4790031909942627, 'learning_rate': 3.91624070927949e-05, 'epoch': 1.2}

 12%|█▏        | 4937/41250 [11:56:16<87:19:19,  8.66s/it][2025-04-25 19:54:00,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 19:54:00,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.99 | bwd_microstep: 5756.11 | bwd_inner_microstep: 5707.82 | bwd_allreduce_microstep: 48.24 | step_microstep: 18.87
[2025-04-25 19:54:00,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.99 | bwd: 5756.13 | bwd_inner: 5707.82 | bwd_allreduce: 48.26 | step: 18.88

 12%|█▏        | 4938/41250 [11:56:25<87:25:48,  8.67s/it]
                                                          
{'loss': 0.1678, 'grad_norm': 1.9026670455932617, 'learning_rate': 3.916195734624853e-05, 'epoch': 1.2}

 12%|█▏        | 4938/41250 [11:56:25<87:25:48,  8.67s/it][2025-04-25 19:54:08,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 19:54:08,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.51 | bwd_microstep: 5790.53 | bwd_inner_microstep: 5646.92 | bwd_allreduce_microstep: 143.56 | step_microstep: 19.05
[2025-04-25 19:54:08,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.51 | bwd: 5790.54 | bwd_inner: 5646.92 | bwd_allreduce: 143.58 | step: 19.05

 12%|█▏        | 4939/41250 [11:56:34<87:32:19,  8.68s/it]
                                                          
{'loss': 0.1773, 'grad_norm': 2.023768663406372, 'learning_rate': 3.916150748157245e-05, 'epoch': 1.2}

 12%|█▏        | 4939/41250 [11:56:34<87:32:19,  8.68s/it][2025-04-25 19:54:17,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:54:17,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.44 | bwd_microstep: 5776.20 | bwd_inner_microstep: 5699.36 | bwd_allreduce_microstep: 76.79 | step_microstep: 18.42
[2025-04-25 19:54:17,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.44 | bwd: 5776.21 | bwd_inner: 5699.36 | bwd_allreduce: 76.81 | step: 18.42

 12%|█▏        | 4940/41250 [11:56:42<87:38:45,  8.69s/it]
                                                          
{'loss': 0.1089, 'grad_norm': 1.6138650178909302, 'learning_rate': 3.916105749876946e-05, 'epoch': 1.2}

 12%|█▏        | 4940/41250 [11:56:42<87:38:45,  8.69s/it][2025-04-25 19:54:26,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 19:54:26,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.78 | bwd_microstep: 5725.15 | bwd_inner_microstep: 5712.58 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.24
[2025-04-25 19:54:26,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.78 | bwd: 5725.16 | bwd_inner: 5712.58 | bwd_allreduce: 12.54 | step: 18.24

 12%|█▏        | 4941/41250 [11:56:51<87:32:54,  8.68s/it]
                                                          
{'loss': 0.2104, 'grad_norm': 2.7858588695526123, 'learning_rate': 3.916060739784231e-05, 'epoch': 1.2}

 12%|█▏        | 4941/41250 [11:56:51<87:32:54,  8.68s/it][2025-04-25 19:54:34,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 19:54:34,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.03 | bwd_microstep: 5748.06 | bwd_inner_microstep: 5697.46 | bwd_allreduce_microstep: 50.55 | step_microstep: 18.52
[2025-04-25 19:54:34,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.03 | bwd: 5748.07 | bwd_inner: 5697.46 | bwd_allreduce: 50.57 | step: 18.52

 12%|█▏        | 4942/41250 [11:57:00<87:34:21,  8.68s/it]
                                                          
{'loss': 0.042, 'grad_norm': 0.705148458480835, 'learning_rate': 3.916015717879379e-05, 'epoch': 1.2}

 12%|█▏        | 4942/41250 [11:57:00<87:34:21,  8.68s/it][2025-04-25 19:54:43,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 19:54:43,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.94 | bwd_microstep: 5719.65 | bwd_inner_microstep: 5706.85 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.80
[2025-04-25 19:54:43,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.94 | bwd: 5719.66 | bwd_inner: 5706.85 | bwd_allreduce: 12.77 | step: 18.80

 12%|█▏        | 4943/41250 [11:57:08<87:31:08,  8.68s/it]
                                                          
{'loss': 0.1445, 'grad_norm': 1.4438657760620117, 'learning_rate': 3.915970684162666e-05, 'epoch': 1.2}

 12%|█▏        | 4943/41250 [11:57:08<87:31:08,  8.68s/it][2025-04-25 19:54:52,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.97
[2025-04-25 19:54:52,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.43 | bwd_microstep: 5772.69 | bwd_inner_microstep: 5661.82 | bwd_allreduce_microstep: 110.82 | step_microstep: 18.96
[2025-04-25 19:54:52,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.43 | bwd: 5772.71 | bwd_inner: 5661.82 | bwd_allreduce: 110.84 | step: 18.96

 12%|█▏        | 4944/41250 [11:57:17<87:33:58,  8.68s/it]
                                                          
{'loss': 0.0735, 'grad_norm': 2.4272286891937256, 'learning_rate': 3.915925638634372e-05, 'epoch': 1.2}

 12%|█▏        | 4944/41250 [11:57:17<87:33:58,  8.68s/it][2025-04-25 19:55:01,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:55:01,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.84 | bwd_microstep: 5725.57 | bwd_inner_microstep: 5712.66 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.57
[2025-04-25 19:55:01,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.84 | bwd: 5725.58 | bwd_inner: 5712.65 | bwd_allreduce: 12.88 | step: 18.58

 12%|█▏        | 4945/41250 [11:57:26<87:33:18,  8.68s/it]
                                                          
{'loss': 0.4038, 'grad_norm': 3.6571130752563477, 'learning_rate': 3.915880581294772e-05, 'epoch': 1.2}

 12%|█▏        | 4945/41250 [11:57:26<87:33:18,  8.68s/it][2025-04-25 19:55:09,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.15
[2025-04-25 19:55:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5761.53 | bwd_inner_microstep: 5718.85 | bwd_allreduce_microstep: 42.63 | step_microstep: 19.44
[2025-04-25 19:55:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5761.54 | bwd_inner: 5718.85 | bwd_allreduce: 42.65 | step: 19.45

 12%|█▏        | 4946/41250 [11:57:35<87:36:34,  8.69s/it]
                                                          
{'loss': 0.1219, 'grad_norm': 1.824683666229248, 'learning_rate': 3.9158355121441456e-05, 'epoch': 1.2}

 12%|█▏        | 4946/41250 [11:57:35<87:36:34,  8.69s/it][2025-04-25 19:55:18,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 19:55:18,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.39 | bwd_microstep: 5696.46 | bwd_inner_microstep: 5665.53 | bwd_allreduce_microstep: 30.89 | step_microstep: 18.63
[2025-04-25 19:55:18,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.39 | bwd: 5696.48 | bwd_inner: 5665.53 | bwd_allreduce: 30.90 | step: 18.63

 12%|█▏        | 4947/41250 [11:57:43<87:26:18,  8.67s/it]
                                                          
{'loss': 0.1348, 'grad_norm': 2.3260419368743896, 'learning_rate': 3.91579043118277e-05, 'epoch': 1.2}

 12%|█▏        | 4947/41250 [11:57:43<87:26:18,  8.67s/it][2025-04-25 19:55:26,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:55:26,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.58 | bwd_microstep: 5705.86 | bwd_inner_microstep: 5668.57 | bwd_allreduce_microstep: 37.25 | step_microstep: 18.61
[2025-04-25 19:55:26,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.58 | bwd: 5705.88 | bwd_inner: 5668.57 | bwd_allreduce: 37.27 | step: 18.62

 12%|█▏        | 4948/41250 [11:57:52<87:19:51,  8.66s/it]
                                                          
{'loss': 0.0833, 'grad_norm': 1.9073394536972046, 'learning_rate': 3.915745338410923e-05, 'epoch': 1.2}

 12%|█▏        | 4948/41250 [11:57:52<87:19:51,  8.66s/it][2025-04-25 19:55:35,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 19:55:35,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.83 | bwd_microstep: 5745.38 | bwd_inner_microstep: 5709.61 | bwd_allreduce_microstep: 35.73 | step_microstep: 18.46
[2025-04-25 19:55:35,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.83 | bwd: 5745.40 | bwd_inner: 5709.61 | bwd_allreduce: 35.75 | step: 18.46

 12%|█▏        | 4949/41250 [11:58:01<87:26:51,  8.67s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 3.4950976371765137, 'learning_rate': 3.9157002338288835e-05, 'epoch': 1.2}

 12%|█▏        | 4949/41250 [11:58:01<87:26:51,  8.67s/it][2025-04-25 19:55:44,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:55:44,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.30 | bwd_microstep: 5705.35 | bwd_inner_microstep: 5663.22 | bwd_allreduce_microstep: 42.09 | step_microstep: 18.26
[2025-04-25 19:55:44,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.30 | bwd: 5705.36 | bwd_inner: 5663.22 | bwd_allreduce: 42.11 | step: 18.26

 12%|█▏        | 4950/41250 [11:58:09<87:20:32,  8.66s/it]
                                                          
{'loss': 0.2942, 'grad_norm': 4.529632091522217, 'learning_rate': 3.9156551174369284e-05, 'epoch': 1.2}

 12%|█▏        | 4950/41250 [11:58:09<87:20:32,  8.66s/it][2025-04-25 19:55:53,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:55:53,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.51 | bwd_microstep: 5790.13 | bwd_inner_microstep: 5673.42 | bwd_allreduce_microstep: 116.67 | step_microstep: 18.62
[2025-04-25 19:55:53,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.51 | bwd: 5790.14 | bwd_inner: 5673.42 | bwd_allreduce: 116.68 | step: 18.62

 12%|█▏        | 4951/41250 [11:58:18<87:30:37,  8.68s/it]
                                                          
{'loss': 0.141, 'grad_norm': 2.012799024581909, 'learning_rate': 3.9156099892353364e-05, 'epoch': 1.2}

 12%|█▏        | 4951/41250 [11:58:18<87:30:37,  8.68s/it][2025-04-25 19:56:01,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 19:56:01,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5802.24 | bwd_inner_microstep: 5661.47 | bwd_allreduce_microstep: 140.73 | step_microstep: 18.65
[2025-04-25 19:56:01,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5802.26 | bwd_inner: 5661.47 | bwd_allreduce: 140.75 | step: 18.65

 12%|█▏        | 4952/41250 [11:58:27<87:36:27,  8.69s/it]
                                                          
{'loss': 0.0398, 'grad_norm': 0.586517870426178, 'learning_rate': 3.915564849224385e-05, 'epoch': 1.2}

 12%|█▏        | 4952/41250 [11:58:27<87:36:27,  8.69s/it][2025-04-25 19:56:10,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.11 | optimizer_step: 1.05
[2025-04-25 19:56:10,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.79 | bwd_microstep: 5738.13 | bwd_inner_microstep: 5724.37 | bwd_allreduce_microstep: 13.71 | step_microstep: 19.97
[2025-04-25 19:56:10,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.79 | bwd: 5738.15 | bwd_inner: 5724.37 | bwd_allreduce: 13.73 | step: 19.97

 12%|█▏        | 4953/41250 [11:58:35<87:34:29,  8.69s/it]
                                                          
{'loss': 0.0417, 'grad_norm': 0.5316907167434692, 'learning_rate': 3.915519697404354e-05, 'epoch': 1.2}

 12%|█▏        | 4953/41250 [11:58:35<87:34:29,  8.69s/it][2025-04-25 19:56:19,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 19:56:19,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.61 | bwd_microstep: 5754.56 | bwd_inner_microstep: 5716.79 | bwd_allreduce_microstep: 37.72 | step_microstep: 18.53
[2025-04-25 19:56:19,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.61 | bwd: 5754.57 | bwd_inner: 5716.79 | bwd_allreduce: 37.74 | step: 18.53

 12%|█▏        | 4954/41250 [11:58:44<87:35:22,  8.69s/it]
                                                          
{'loss': 0.1121, 'grad_norm': 0.9589769244194031, 'learning_rate': 3.9154745337755204e-05, 'epoch': 1.2}

 12%|█▏        | 4954/41250 [11:58:44<87:35:22,  8.69s/it][2025-04-25 19:56:27,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 19:56:27,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.60 | bwd_microstep: 5774.15 | bwd_inner_microstep: 5704.82 | bwd_allreduce_microstep: 69.29 | step_microstep: 18.66
[2025-04-25 19:56:27,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.60 | bwd: 5774.17 | bwd_inner: 5704.82 | bwd_allreduce: 69.31 | step: 18.66

 12%|█▏        | 4955/41250 [11:58:53<87:39:40,  8.69s/it]
                                                          
{'loss': 0.118, 'grad_norm': 1.6285144090652466, 'learning_rate': 3.915429358338163e-05, 'epoch': 1.2}

 12%|█▏        | 4955/41250 [11:58:53<87:39:40,  8.69s/it][2025-04-25 19:56:36,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 19:56:36,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.31 | bwd_microstep: 5780.61 | bwd_inner_microstep: 5709.09 | bwd_allreduce_microstep: 71.48 | step_microstep: 19.00
[2025-04-25 19:56:36,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.31 | bwd: 5780.63 | bwd_inner: 5709.09 | bwd_allreduce: 71.49 | step: 19.00

 12%|█▏        | 4956/41250 [11:59:01<87:44:49,  8.70s/it]
                                                          
{'loss': 0.1578, 'grad_norm': 1.5781985521316528, 'learning_rate': 3.9153841710925605e-05, 'epoch': 1.2}

 12%|█▏        | 4956/41250 [11:59:01<87:44:49,  8.70s/it][2025-04-25 19:56:45,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:56:45,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.04 | bwd_microstep: 5771.63 | bwd_inner_microstep: 5715.88 | bwd_allreduce_microstep: 55.70 | step_microstep: 18.80
[2025-04-25 19:56:45,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.04 | bwd: 5771.64 | bwd_inner: 5715.88 | bwd_allreduce: 55.72 | step: 18.80

 12%|█▏        | 4957/41250 [11:59:10<87:45:33,  8.71s/it]
                                                          
{'loss': 0.094, 'grad_norm': 1.3534042835235596, 'learning_rate': 3.9153389720389914e-05, 'epoch': 1.2}

 12%|█▏        | 4957/41250 [11:59:10<87:45:33,  8.71s/it][2025-04-25 19:56:53,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:56:53,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.43 | bwd_microstep: 5733.46 | bwd_inner_microstep: 5703.07 | bwd_allreduce_microstep: 30.34 | step_microstep: 19.04
[2025-04-25 19:56:53,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.43 | bwd: 5733.48 | bwd_inner: 5703.07 | bwd_allreduce: 30.36 | step: 19.04

 12%|█▏        | 4958/41250 [11:59:19<87:41:13,  8.70s/it]
                                                          
{'loss': 0.0997, 'grad_norm': 2.0155587196350098, 'learning_rate': 3.9152937611777336e-05, 'epoch': 1.2}

 12%|█▏        | 4958/41250 [11:59:19<87:41:13,  8.70s/it][2025-04-25 19:57:02,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 19:57:02,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.38 | bwd_microstep: 5784.31 | bwd_inner_microstep: 5666.40 | bwd_allreduce_microstep: 117.87 | step_microstep: 18.69
[2025-04-25 19:57:02,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.38 | bwd: 5784.33 | bwd_inner: 5666.40 | bwd_allreduce: 117.89 | step: 18.69

 12%|█▏        | 4959/41250 [11:59:27<87:42:01,  8.70s/it]
                                                          
{'loss': 0.1237, 'grad_norm': 2.0113131999969482, 'learning_rate': 3.9152485385090675e-05, 'epoch': 1.2}

 12%|█▏        | 4959/41250 [11:59:27<87:42:01,  8.70s/it][2025-04-25 19:57:11,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 19:57:11,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.41 | bwd_microstep: 5739.34 | bwd_inner_microstep: 5705.69 | bwd_allreduce_microstep: 33.60 | step_microstep: 19.49
[2025-04-25 19:57:11,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.41 | bwd: 5739.35 | bwd_inner: 5705.69 | bwd_allreduce: 33.61 | step: 19.49

 12%|█▏        | 4960/41250 [11:59:36<87:37:54,  8.69s/it]
                                                          
{'loss': 0.085, 'grad_norm': 1.701134204864502, 'learning_rate': 3.91520330403327e-05, 'epoch': 1.2}

 12%|█▏        | 4960/41250 [11:59:36<87:37:54,  8.69s/it][2025-04-25 19:57:19,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 19:57:19,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5715.79 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 54.81 | step_microstep: 19.53
[2025-04-25 19:57:19,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5715.80 | bwd_inner: 5660.93 | bwd_allreduce: 54.83 | step: 19.53

 12%|█▏        | 4961/41250 [11:59:45<87:26:54,  8.68s/it]
                                                          
{'loss': 0.1712, 'grad_norm': 3.7515008449554443, 'learning_rate': 3.9151580577506215e-05, 'epoch': 1.2}

 12%|█▏        | 4961/41250 [11:59:45<87:26:54,  8.68s/it][2025-04-25 19:57:28,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 19:57:28,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.58 | bwd_microstep: 5706.61 | bwd_inner_microstep: 5639.40 | bwd_allreduce_microstep: 67.17 | step_microstep: 19.38
[2025-04-25 19:57:28,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.58 | bwd: 5706.62 | bwd_inner: 5639.40 | bwd_allreduce: 67.18 | step: 19.38

 12%|█▏        | 4962/41250 [11:59:53<87:15:49,  8.66s/it]
                                                          
{'loss': 0.2344, 'grad_norm': 2.0149145126342773, 'learning_rate': 3.9151127996614e-05, 'epoch': 1.2}

 12%|█▏        | 4962/41250 [11:59:53<87:15:49,  8.66s/it][2025-04-25 19:57:37,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:57:37,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.83 | bwd_microstep: 5757.27 | bwd_inner_microstep: 5653.40 | bwd_allreduce_microstep: 103.83 | step_microstep: 18.86
[2025-04-25 19:57:37,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.83 | bwd: 5757.29 | bwd_inner: 5653.40 | bwd_allreduce: 103.85 | step: 18.86

 12%|█▏        | 4963/41250 [12:00:02<87:17:44,  8.66s/it]
                                                          
{'loss': 0.0366, 'grad_norm': 0.455233633518219, 'learning_rate': 3.9150675297658846e-05, 'epoch': 1.2}

 12%|█▏        | 4963/41250 [12:00:02<87:17:44,  8.66s/it][2025-04-25 19:57:46,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 19:57:46,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5864.73 | bwd_inner_microstep: 5716.95 | bwd_allreduce_microstep: 147.73 | step_microstep: 19.02
[2025-04-25 19:57:46,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.42 | bwd: 5864.75 | bwd_inner: 5716.95 | bwd_allreduce: 147.75 | step: 19.02

 12%|█▏        | 4964/41250 [12:00:11<87:43:36,  8.70s/it]
                                                          
{'loss': 0.1173, 'grad_norm': 1.497464895248413, 'learning_rate': 3.915022248064355e-05, 'epoch': 1.2}

 12%|█▏        | 4964/41250 [12:00:11<87:43:36,  8.70s/it][2025-04-25 19:57:54,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 19:57:54,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.68 | bwd_microstep: 5703.42 | bwd_inner_microstep: 5690.54 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.84
[2025-04-25 19:57:54,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.68 | bwd: 5703.43 | bwd_inner: 5690.54 | bwd_allreduce: 12.85 | step: 18.84

 12%|█▏        | 4965/41250 [12:00:20<87:30:17,  8.68s/it]
                                                          
{'loss': 0.0818, 'grad_norm': 1.3105581998825073, 'learning_rate': 3.914976954557089e-05, 'epoch': 1.2}

 12%|█▏        | 4965/41250 [12:00:20<87:30:17,  8.68s/it][2025-04-25 19:58:03,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:58:03,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.57 | bwd_microstep: 5791.94 | bwd_inner_microstep: 5644.42 | bwd_allreduce_microstep: 147.48 | step_microstep: 18.85
[2025-04-25 19:58:03,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.57 | bwd: 5791.96 | bwd_inner: 5644.42 | bwd_allreduce: 147.50 | step: 18.85

 12%|█▏        | 4966/41250 [12:00:28<87:32:57,  8.69s/it]
                                                          
{'loss': 0.1748, 'grad_norm': 2.1041455268859863, 'learning_rate': 3.9149316492443677e-05, 'epoch': 1.2}

 12%|█▏        | 4966/41250 [12:00:28<87:32:57,  8.69s/it][2025-04-25 19:58:12,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:58:12,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.08 | bwd_microstep: 5783.18 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 132.99 | step_microstep: 19.01
[2025-04-25 19:58:12,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.08 | bwd: 5783.19 | bwd_inner: 5650.14 | bwd_allreduce: 133.01 | step: 19.02

 12%|█▏        | 4967/41250 [12:00:37<87:33:55,  8.69s/it]
                                                          
{'loss': 0.2085, 'grad_norm': 2.15913462638855, 'learning_rate': 3.914886332126469e-05, 'epoch': 1.2}

 12%|█▏        | 4967/41250 [12:00:37<87:33:55,  8.69s/it][2025-04-25 19:58:20,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 19:58:20,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.79 | bwd_microstep: 5728.72 | bwd_inner_microstep: 5699.62 | bwd_allreduce_microstep: 29.05 | step_microstep: 18.88
[2025-04-25 19:58:20,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.79 | bwd: 5728.73 | bwd_inner: 5699.62 | bwd_allreduce: 29.07 | step: 18.88

 12%|█▏        | 4968/41250 [12:00:46<87:29:18,  8.68s/it]
                                                          
{'loss': 0.0871, 'grad_norm': 1.7109946012496948, 'learning_rate': 3.9148410032036724e-05, 'epoch': 1.2}

 12%|█▏        | 4968/41250 [12:00:46<87:29:18,  8.68s/it][2025-04-25 19:58:29,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 19:58:29,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.77 | bwd_microstep: 5739.65 | bwd_inner_microstep: 5686.44 | bwd_allreduce_microstep: 53.17 | step_microstep: 19.04
[2025-04-25 19:58:29,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.77 | bwd: 5739.67 | bwd_inner: 5686.44 | bwd_allreduce: 53.18 | step: 19.04

 12%|█▏        | 4969/41250 [12:00:54<87:27:27,  8.68s/it]
                                                          
{'loss': 0.1493, 'grad_norm': 2.4113035202026367, 'learning_rate': 3.914795662476258e-05, 'epoch': 1.2}

 12%|█▏        | 4969/41250 [12:00:54<87:27:27,  8.68s/it][2025-04-25 19:58:38,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 19:58:38,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 6010.47 | bwd_inner_microstep: 5633.80 | bwd_allreduce_microstep: 376.62 | step_microstep: 19.21
[2025-04-25 19:58:38,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 6010.48 | bwd_inner: 5633.80 | bwd_allreduce: 376.64 | step: 19.21

 12%|█▏        | 4970/41250 [12:01:03<88:11:48,  8.75s/it]
                                                          
{'loss': 0.065, 'grad_norm': 1.0255277156829834, 'learning_rate': 3.9147503099445055e-05, 'epoch': 1.2}

 12%|█▏        | 4970/41250 [12:01:03<88:11:48,  8.75s/it][2025-04-25 19:58:46,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.07 | optimizer_step: 0.97
[2025-04-25 19:58:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.73 | bwd_microstep: 5687.59 | bwd_inner_microstep: 5637.91 | bwd_allreduce_microstep: 49.62 | step_microstep: 19.30
[2025-04-25 19:58:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.73 | bwd: 5687.61 | bwd_inner: 5637.91 | bwd_allreduce: 49.65 | step: 19.30

 12%|█▏        | 4971/41250 [12:01:12<87:43:07,  8.70s/it]
                                                          
{'loss': 0.0825, 'grad_norm': 3.303624153137207, 'learning_rate': 3.914704945608693e-05, 'epoch': 1.21}

 12%|█▏        | 4971/41250 [12:01:12<87:43:07,  8.70s/it][2025-04-25 19:58:55,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 19:58:55,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.96 | bwd_microstep: 5693.55 | bwd_inner_microstep: 5680.75 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.24
[2025-04-25 19:58:55,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.96 | bwd: 5693.56 | bwd_inner: 5680.75 | bwd_allreduce: 12.77 | step: 19.24

 12%|█▏        | 4972/41250 [12:01:20<87:27:21,  8.68s/it]
                                                          
{'loss': 0.0361, 'grad_norm': 0.6159043312072754, 'learning_rate': 3.914659569469102e-05, 'epoch': 1.21}

 12%|█▏        | 4972/41250 [12:01:20<87:27:21,  8.68s/it][2025-04-25 19:59:04,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:59:04,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.59 | bwd_microstep: 5734.50 | bwd_inner_microstep: 5671.05 | bwd_allreduce_microstep: 63.40 | step_microstep: 18.57
[2025-04-25 19:59:04,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.59 | bwd: 5734.51 | bwd_inner: 5671.05 | bwd_allreduce: 63.42 | step: 18.58

 12%|█▏        | 4973/41250 [12:01:29<87:23:10,  8.67s/it]
                                                          
{'loss': 0.1209, 'grad_norm': 2.3566412925720215, 'learning_rate': 3.9146141815260104e-05, 'epoch': 1.21}

 12%|█▏        | 4973/41250 [12:01:29<87:23:10,  8.67s/it][2025-04-25 19:59:12,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 19:59:12,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.60 | bwd_microstep: 5693.76 | bwd_inner_microstep: 5628.00 | bwd_allreduce_microstep: 65.71 | step_microstep: 18.42
[2025-04-25 19:59:12,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.60 | bwd: 5693.77 | bwd_inner: 5628.00 | bwd_allreduce: 65.73 | step: 18.43

 12%|█▏        | 4974/41250 [12:01:38<87:09:44,  8.65s/it]
                                                          
{'loss': 0.2752, 'grad_norm': 3.125633716583252, 'learning_rate': 3.914568781779699e-05, 'epoch': 1.21}

 12%|█▏        | 4974/41250 [12:01:38<87:09:44,  8.65s/it][2025-04-25 19:59:21,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 19:59:21,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5698.00 | bwd_inner_microstep: 5629.31 | bwd_allreduce_microstep: 68.64 | step_microstep: 18.63
[2025-04-25 19:59:21,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5698.01 | bwd_inner: 5629.31 | bwd_allreduce: 68.66 | step: 18.63

 12%|█▏        | 4975/41250 [12:01:46<87:00:57,  8.64s/it]
                                                          
{'loss': 0.1183, 'grad_norm': 1.8595918416976929, 'learning_rate': 3.914523370230448e-05, 'epoch': 1.21}

 12%|█▏        | 4975/41250 [12:01:46<87:00:57,  8.64s/it][2025-04-25 19:59:29,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-25 19:59:29,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5659.99 | bwd_inner_microstep: 5645.24 | bwd_allreduce_microstep: 14.70 | step_microstep: 18.71
[2025-04-25 19:59:29,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5660.01 | bwd_inner: 5645.24 | bwd_allreduce: 14.72 | step: 18.72

 12%|█▏        | 4976/41250 [12:01:55<86:50:51,  8.62s/it]
                                                          
{'loss': 0.1315, 'grad_norm': 2.796468734741211, 'learning_rate': 3.914477946878537e-05, 'epoch': 1.21}

 12%|█▏        | 4976/41250 [12:01:55<86:50:51,  8.62s/it][2025-04-25 19:59:38,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 19:59:38,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.47 | bwd_microstep: 5726.15 | bwd_inner_microstep: 5673.35 | bwd_allreduce_microstep: 52.75 | step_microstep: 18.37
[2025-04-25 19:59:38,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.47 | bwd: 5726.16 | bwd_inner: 5673.35 | bwd_allreduce: 52.77 | step: 18.37

 12%|█▏        | 4977/41250 [12:02:03<86:55:50,  8.63s/it]
                                                          
{'loss': 0.1904, 'grad_norm': 3.0751256942749023, 'learning_rate': 3.9144325117242464e-05, 'epoch': 1.21}

 12%|█▏        | 4977/41250 [12:02:03<86:55:50,  8.63s/it][2025-04-25 19:59:47,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 19:59:47,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.78 | bwd_microstep: 5750.26 | bwd_inner_microstep: 5633.75 | bwd_allreduce_microstep: 116.46 | step_microstep: 18.50
[2025-04-25 19:59:47,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.78 | bwd: 5750.27 | bwd_inner: 5633.75 | bwd_allreduce: 116.47 | step: 18.50

 12%|█▏        | 4978/41250 [12:02:12<87:00:27,  8.64s/it]
                                                          
{'loss': 0.1271, 'grad_norm': 1.2477110624313354, 'learning_rate': 3.914387064767856e-05, 'epoch': 1.21}

 12%|█▏        | 4978/41250 [12:02:12<87:00:27,  8.64s/it][2025-04-25 19:59:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 19:59:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.98 | bwd_microstep: 5695.81 | bwd_inner_microstep: 5683.03 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.61
[2025-04-25 19:59:55,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.98 | bwd: 5695.83 | bwd_inner: 5683.03 | bwd_allreduce: 12.75 | step: 18.61

 12%|█▏        | 4979/41250 [12:02:21<86:58:12,  8.63s/it]
                                                          
{'loss': 0.0895, 'grad_norm': 1.662016749382019, 'learning_rate': 3.914341606009645e-05, 'epoch': 1.21}

 12%|█▏        | 4979/41250 [12:02:21<86:58:12,  8.63s/it][2025-04-25 20:00:04,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:00:04,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.15 | bwd_microstep: 5736.08 | bwd_inner_microstep: 5678.55 | bwd_allreduce_microstep: 57.48 | step_microstep: 18.50
[2025-04-25 20:00:04,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.15 | bwd: 5736.09 | bwd_inner: 5678.55 | bwd_allreduce: 57.50 | step: 18.50

 12%|█▏        | 4980/41250 [12:02:29<87:03:24,  8.64s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 0.7744361758232117, 'learning_rate': 3.9142961354498956e-05, 'epoch': 1.21}

 12%|█▏        | 4980/41250 [12:02:29<87:03:24,  8.64s/it][2025-04-25 20:00:13,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 20:00:13,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.31 | bwd_microstep: 5718.39 | bwd_inner_microstep: 5688.34 | bwd_allreduce_microstep: 30.01 | step_microstep: 18.97
[2025-04-25 20:00:13,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.32 | bwd: 5718.40 | bwd_inner: 5688.34 | bwd_allreduce: 30.03 | step: 18.98

 12%|█▏        | 4981/41250 [12:02:38<87:04:54,  8.64s/it]
                                                          
{'loss': 0.1609, 'grad_norm': 1.3515477180480957, 'learning_rate': 3.914250653088886e-05, 'epoch': 1.21}

 12%|█▏        | 4981/41250 [12:02:38<87:04:54,  8.64s/it][2025-04-25 20:00:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:00:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5694.25 | bwd_inner_microstep: 5681.44 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.60
[2025-04-25 20:00:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.71 | bwd: 5694.26 | bwd_inner: 5681.44 | bwd_allreduce: 12.78 | step: 18.61

 12%|█▏        | 4982/41250 [12:02:47<87:01:09,  8.64s/it]
                                                          
{'loss': 0.1592, 'grad_norm': 3.1764118671417236, 'learning_rate': 3.9142051589268975e-05, 'epoch': 1.21}

 12%|█▏        | 4982/41250 [12:02:47<87:01:09,  8.64s/it][2025-04-25 20:00:30,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:00:30,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.41 | bwd_microstep: 5741.96 | bwd_inner_microstep: 5683.91 | bwd_allreduce_microstep: 58.00 | step_microstep: 18.61
[2025-04-25 20:00:30,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.41 | bwd: 5741.97 | bwd_inner: 5683.91 | bwd_allreduce: 58.02 | step: 18.61

 12%|█▏        | 4983/41250 [12:02:55<87:09:55,  8.65s/it]
                                                          
{'loss': 0.1383, 'grad_norm': 1.1897492408752441, 'learning_rate': 3.914159652964211e-05, 'epoch': 1.21}

 12%|█▏        | 4983/41250 [12:02:55<87:09:55,  8.65s/it][2025-04-25 20:00:39,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 20:00:39,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.87 | bwd_microstep: 5702.69 | bwd_inner_microstep: 5662.48 | bwd_allreduce_microstep: 40.16 | step_microstep: 19.27
[2025-04-25 20:00:39,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.87 | bwd: 5702.71 | bwd_inner: 5662.48 | bwd_allreduce: 40.18 | step: 19.28

 12%|█▏        | 4984/41250 [12:03:04<87:03:35,  8.64s/it]
                                                          
{'loss': 0.1093, 'grad_norm': 1.878277063369751, 'learning_rate': 3.914114135201108e-05, 'epoch': 1.21}

 12%|█▏        | 4984/41250 [12:03:04<87:03:35,  8.64s/it][2025-04-25 20:00:47,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 20:00:47,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 5775.57 | bwd_inner_microstep: 5659.03 | bwd_allreduce_microstep: 116.50 | step_microstep: 18.81
[2025-04-25 20:00:47,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 5775.59 | bwd_inner: 5659.03 | bwd_allreduce: 116.52 | step: 18.81

 12%|█▏        | 4985/41250 [12:03:13<87:11:17,  8.66s/it]
                                                          
{'loss': 0.1417, 'grad_norm': 1.152449369430542, 'learning_rate': 3.914068605637867e-05, 'epoch': 1.21}

 12%|█▏        | 4985/41250 [12:03:13<87:11:17,  8.66s/it][2025-04-25 20:00:56,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:00:56,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5888.26 | bwd_inner_microstep: 5660.86 | bwd_allreduce_microstep: 227.35 | step_microstep: 18.39
[2025-04-25 20:00:56,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.16 | bwd: 5888.28 | bwd_inner: 5660.86 | bwd_allreduce: 227.37 | step: 18.39

 12%|█▏        | 4986/41250 [12:03:21<87:37:13,  8.70s/it]
                                                          
{'loss': 0.0798, 'grad_norm': 1.2755616903305054, 'learning_rate': 3.914023064274769e-05, 'epoch': 1.21}

 12%|█▏        | 4986/41250 [12:03:21<87:37:13,  8.70s/it][2025-04-25 20:01:05,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:01:05,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5715.10 | bwd_inner_microstep: 5702.36 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.80
[2025-04-25 20:01:05,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5715.11 | bwd_inner: 5702.36 | bwd_allreduce: 12.71 | step: 18.80

 12%|█▏        | 4987/41250 [12:03:30<87:28:00,  8.68s/it]
                                                          
{'loss': 0.0179, 'grad_norm': 0.3066762089729309, 'learning_rate': 3.913977511112095e-05, 'epoch': 1.21}

 12%|█▏        | 4987/41250 [12:03:30<87:28:00,  8.68s/it][2025-04-25 20:01:14,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 20:01:14,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.89 | bwd_microstep: 5787.81 | bwd_inner_microstep: 5652.15 | bwd_allreduce_microstep: 135.62 | step_microstep: 19.16
[2025-04-25 20:01:14,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.89 | bwd: 5787.82 | bwd_inner: 5652.15 | bwd_allreduce: 135.64 | step: 19.16

 12%|█▏        | 4988/41250 [12:03:39<87:32:37,  8.69s/it]
                                                          
{'loss': 0.099, 'grad_norm': 1.5418002605438232, 'learning_rate': 3.913931946150127e-05, 'epoch': 1.21}

 12%|█▏        | 4988/41250 [12:03:39<87:32:37,  8.69s/it][2025-04-25 20:01:22,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 20:01:22,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.39 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5704.49 | bwd_allreduce_microstep: 13.29 | step_microstep: 19.26
[2025-04-25 20:01:22,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.39 | bwd: 5717.84 | bwd_inner: 5704.49 | bwd_allreduce: 13.31 | step: 19.27

 12%|█▏        | 4989/41250 [12:03:47<87:25:12,  8.68s/it]
                                                          
{'loss': 0.1826, 'grad_norm': 2.231804132461548, 'learning_rate': 3.913886369389145e-05, 'epoch': 1.21}

 12%|█▏        | 4989/41250 [12:03:47<87:25:12,  8.68s/it][2025-04-25 20:01:31,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:01:31,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.10 | bwd_microstep: 5776.20 | bwd_inner_microstep: 5763.39 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.33
[2025-04-25 20:01:31,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.10 | bwd: 5776.22 | bwd_inner: 5763.39 | bwd_allreduce: 12.79 | step: 19.34

 12%|█▏        | 4990/41250 [12:03:56<87:37:04,  8.70s/it]
                                                          
{'loss': 0.4217, 'grad_norm': 2.3078975677490234, 'learning_rate': 3.913840780829429e-05, 'epoch': 1.21}

 12%|█▏        | 4990/41250 [12:03:56<87:37:04,  8.70s/it][2025-04-25 20:01:40,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 20:01:40,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.32 | bwd_microstep: 5776.70 | bwd_inner_microstep: 5657.88 | bwd_allreduce_microstep: 118.77 | step_microstep: 19.68
[2025-04-25 20:01:40,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.32 | bwd: 5776.71 | bwd_inner: 5657.88 | bwd_allreduce: 118.79 | step: 19.68

 12%|█▏        | 4991/41250 [12:04:05<87:34:51,  8.70s/it]
                                                          
{'loss': 0.1023, 'grad_norm': 1.7703533172607422, 'learning_rate': 3.913795180471262e-05, 'epoch': 1.21}

 12%|█▏        | 4991/41250 [12:04:05<87:34:51,  8.70s/it][2025-04-25 20:01:48,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:01:48,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.45 | bwd_microstep: 5723.64 | bwd_inner_microstep: 5649.19 | bwd_allreduce_microstep: 74.40 | step_microstep: 18.99
[2025-04-25 20:01:48,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.45 | bwd: 5723.65 | bwd_inner: 5649.19 | bwd_allreduce: 74.42 | step: 18.99

 12%|█▏        | 4992/41250 [12:04:14<87:23:11,  8.68s/it]
                                                          
{'loss': 0.0859, 'grad_norm': 1.9050469398498535, 'learning_rate': 3.9137495683149245e-05, 'epoch': 1.21}

 12%|█▏        | 4992/41250 [12:04:14<87:23:11,  8.68s/it][2025-04-25 20:01:57,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.19 | optimizer_step: 0.94
[2025-04-25 20:01:57,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.84 | bwd_microstep: 5710.28 | bwd_inner_microstep: 5697.13 | bwd_allreduce_microstep: 13.09 | step_microstep: 19.21
[2025-04-25 20:01:57,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.84 | bwd: 5710.30 | bwd_inner: 5697.13 | bwd_allreduce: 13.12 | step: 19.21

 12%|█▏        | 4993/41250 [12:04:22<87:19:04,  8.67s/it]
                                                          
{'loss': 0.1149, 'grad_norm': 1.2638007402420044, 'learning_rate': 3.9137039443606965e-05, 'epoch': 1.21}

 12%|█▏        | 4993/41250 [12:04:22<87:19:04,  8.67s/it][2025-04-25 20:02:06,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 20:02:06,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.20 | bwd_microstep: 5725.77 | bwd_inner_microstep: 5712.93 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.31
[2025-04-25 20:02:06,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.21 | bwd: 5725.78 | bwd_inner: 5712.93 | bwd_allreduce: 12.81 | step: 19.31

 12%|█▏        | 4994/41250 [12:04:31<87:18:04,  8.67s/it]
                                                          
{'loss': 0.1194, 'grad_norm': 1.9572725296020508, 'learning_rate': 3.91365830860886e-05, 'epoch': 1.21}

 12%|█▏        | 4994/41250 [12:04:31<87:18:04,  8.67s/it][2025-04-25 20:02:14,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 20:02:14,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.67 | bwd_microstep: 5784.70 | bwd_inner_microstep: 5708.35 | bwd_allreduce_microstep: 76.30 | step_microstep: 18.94
[2025-04-25 20:02:14,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.67 | bwd: 5784.71 | bwd_inner: 5708.35 | bwd_allreduce: 76.32 | step: 18.94

 12%|█▏        | 4995/41250 [12:04:40<87:26:55,  8.68s/it]
                                                          
{'loss': 0.0806, 'grad_norm': 1.2304291725158691, 'learning_rate': 3.9136126610596965e-05, 'epoch': 1.21}

 12%|█▏        | 4995/41250 [12:04:40<87:26:55,  8.68s/it][2025-04-25 20:02:23,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 20:02:23,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5783.00 | bwd_inner_microstep: 5666.45 | bwd_allreduce_microstep: 116.51 | step_microstep: 19.04
[2025-04-25 20:02:23,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5783.02 | bwd_inner: 5666.45 | bwd_allreduce: 116.53 | step: 19.04

 12%|█▏        | 4996/41250 [12:04:48<87:29:02,  8.69s/it]
                                                          
{'loss': 0.1497, 'grad_norm': 2.648470640182495, 'learning_rate': 3.913567001713488e-05, 'epoch': 1.21}

 12%|█▏        | 4996/41250 [12:04:48<87:29:02,  8.69s/it][2025-04-25 20:02:32,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 20:02:32,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.72 | bwd_microstep: 5724.55 | bwd_inner_microstep: 5657.91 | bwd_allreduce_microstep: 66.59 | step_microstep: 18.76
[2025-04-25 20:02:32,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.72 | bwd: 5724.57 | bwd_inner: 5657.91 | bwd_allreduce: 66.61 | step: 18.76

 12%|█▏        | 4997/41250 [12:04:57<87:20:25,  8.67s/it]
                                                          
{'loss': 0.0517, 'grad_norm': 1.6863369941711426, 'learning_rate': 3.913521330570515e-05, 'epoch': 1.21}

 12%|█▏        | 4997/41250 [12:04:57<87:20:25,  8.67s/it][2025-04-25 20:02:40,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.12 | optimizer_step: 0.91
[2025-04-25 20:02:40,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.73 | bwd_microstep: 5772.65 | bwd_inner_microstep: 5721.28 | bwd_allreduce_microstep: 51.33 | step_microstep: 18.90
[2025-04-25 20:02:40,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.73 | bwd: 5772.67 | bwd_inner: 5721.28 | bwd_allreduce: 51.35 | step: 18.90

 12%|█▏        | 4998/41250 [12:05:06<87:27:15,  8.68s/it]
                                                          
{'loss': 0.1823, 'grad_norm': 3.343055248260498, 'learning_rate': 3.913475647631059e-05, 'epoch': 1.21}

 12%|█▏        | 4998/41250 [12:05:06<87:27:15,  8.68s/it][2025-04-25 20:02:49,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:02:49,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5773.08 | bwd_inner_microstep: 5720.31 | bwd_allreduce_microstep: 52.72 | step_microstep: 18.33
[2025-04-25 20:02:49,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5773.09 | bwd_inner: 5720.31 | bwd_allreduce: 52.74 | step: 18.34

 12%|█▏        | 4999/41250 [12:05:14<87:31:34,  8.69s/it]
                                                          
{'loss': 0.5073, 'grad_norm': 4.116705894470215, 'learning_rate': 3.913429952895402e-05, 'epoch': 1.21}

 12%|█▏        | 4999/41250 [12:05:14<87:31:34,  8.69s/it][2025-04-25 20:02:58,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.14
[2025-04-25 20:02:58,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.34 | bwd_microstep: 5784.49 | bwd_inner_microstep: 5660.70 | bwd_allreduce_microstep: 123.74 | step_microstep: 19.62
[2025-04-25 20:02:58,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.34 | bwd: 5784.51 | bwd_inner: 5660.70 | bwd_allreduce: 123.76 | step: 19.62

 12%|█▏        | 5000/41250 [12:05:23<87:32:27,  8.69s/it]
                                                          
{'loss': 0.0483, 'grad_norm': 1.5798704624176025, 'learning_rate': 3.9133842463638266e-05, 'epoch': 1.21}

 12%|█▏        | 5000/41250 [12:05:23<87:32:27,  8.69s/it][2025-04-25 20:03:06,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:03:06,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5712.37 | bwd_inner_microstep: 5699.65 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.71
[2025-04-25 20:03:06,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.29 | bwd: 5712.39 | bwd_inner: 5699.65 | bwd_allreduce: 12.70 | step: 18.71

 12%|█▏        | 5001/41250 [12:05:32<87:23:49,  8.68s/it]
                                                          
{'loss': 0.049, 'grad_norm': 1.3240474462509155, 'learning_rate': 3.913338528036612e-05, 'epoch': 1.21}

 12%|█▏        | 5001/41250 [12:05:32<87:23:49,  8.68s/it][2025-04-25 20:03:15,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:03:15,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.35 | bwd_microstep: 5733.40 | bwd_inner_microstep: 5667.25 | bwd_allreduce_microstep: 66.10 | step_microstep: 18.54
[2025-04-25 20:03:15,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.35 | bwd: 5733.41 | bwd_inner: 5667.25 | bwd_allreduce: 66.12 | step: 18.54

 12%|█▏        | 5002/41250 [12:05:40<87:18:18,  8.67s/it]
                                                          
{'loss': 0.0841, 'grad_norm': 2.360732078552246, 'learning_rate': 3.913292797914043e-05, 'epoch': 1.21}

 12%|█▏        | 5002/41250 [12:05:40<87:18:18,  8.67s/it][2025-04-25 20:03:24,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:03:24,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.74 | bwd_microstep: 5736.35 | bwd_inner_microstep: 5710.85 | bwd_allreduce_microstep: 25.46 | step_microstep: 18.89
[2025-04-25 20:03:24,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.74 | bwd: 5736.37 | bwd_inner: 5710.85 | bwd_allreduce: 25.47 | step: 18.89

 12%|█▏        | 5003/41250 [12:05:49<87:20:01,  8.67s/it]
                                                          
{'loss': 0.0258, 'grad_norm': 0.7353870868682861, 'learning_rate': 3.9132470559964e-05, 'epoch': 1.21}

 12%|█▏        | 5003/41250 [12:05:49<87:20:01,  8.67s/it][2025-04-25 20:03:32,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 20:03:32,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5798.24 | bwd_inner_microstep: 5658.36 | bwd_allreduce_microstep: 139.82 | step_microstep: 18.87
[2025-04-25 20:03:32,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.04 | bwd: 5798.25 | bwd_inner: 5658.36 | bwd_allreduce: 139.84 | step: 18.88

 12%|█▏        | 5004/41250 [12:05:58<87:27:22,  8.69s/it]
                                                          
{'loss': 0.1154, 'grad_norm': 3.8901925086975098, 'learning_rate': 3.9132013022839654e-05, 'epoch': 1.21}

 12%|█▏        | 5004/41250 [12:05:58<87:27:22,  8.69s/it][2025-04-25 20:03:41,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-25 20:03:41,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.69 | bwd_microstep: 5781.74 | bwd_inner_microstep: 5650.00 | bwd_allreduce_microstep: 131.69 | step_microstep: 19.49
[2025-04-25 20:03:41,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5781.75 | bwd_inner: 5650.00 | bwd_allreduce: 131.71 | step: 19.50

 12%|█▏        | 5005/41250 [12:06:06<87:27:38,  8.69s/it]
                                                          
{'loss': 0.2919, 'grad_norm': 4.008839130401611, 'learning_rate': 3.9131555367770206e-05, 'epoch': 1.21}

 12%|█▏        | 5005/41250 [12:06:06<87:27:38,  8.69s/it][2025-04-25 20:03:50,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:03:50,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5762.06 | bwd_inner_microstep: 5701.18 | bwd_allreduce_microstep: 60.84 | step_microstep: 18.77
[2025-04-25 20:03:50,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5762.08 | bwd_inner: 5701.18 | bwd_allreduce: 60.85 | step: 18.77

 12%|█▏        | 5006/41250 [12:06:15<87:29:55,  8.69s/it]
                                                          
{'loss': 0.1804, 'grad_norm': 4.946050643920898, 'learning_rate': 3.913109759475848e-05, 'epoch': 1.21}

 12%|█▏        | 5006/41250 [12:06:15<87:29:55,  8.69s/it][2025-04-25 20:03:59,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.07 | optimizer_step: 1.09
[2025-04-25 20:03:59,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.72 | bwd_microstep: 5793.41 | bwd_inner_microstep: 5660.49 | bwd_allreduce_microstep: 132.87 | step_microstep: 19.24
[2025-04-25 20:03:59,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.72 | bwd: 5793.42 | bwd_inner: 5660.49 | bwd_allreduce: 132.89 | step: 19.25

 12%|█▏        | 5007/41250 [12:06:24<87:32:57,  8.70s/it]
                                                          
{'loss': 0.0791, 'grad_norm': 1.5028494596481323, 'learning_rate': 3.913063970380731e-05, 'epoch': 1.21}

 12%|█▏        | 5007/41250 [12:06:24<87:32:57,  8.70s/it][2025-04-25 20:04:07,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:04:07,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5746.05 | bwd_inner_microstep: 5697.73 | bwd_allreduce_microstep: 48.28 | step_microstep: 18.93
[2025-04-25 20:04:07,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5746.06 | bwd_inner: 5697.73 | bwd_allreduce: 48.29 | step: 18.94

 12%|█▏        | 5008/41250 [12:06:33<87:29:51,  8.69s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 2.1430089473724365, 'learning_rate': 3.91301816949195e-05, 'epoch': 1.21}

 12%|█▏        | 5008/41250 [12:06:33<87:29:51,  8.69s/it][2025-04-25 20:04:16,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 20:04:16,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5849.93 | bwd_inner_microstep: 5700.53 | bwd_allreduce_microstep: 149.35 | step_microstep: 18.97
[2025-04-25 20:04:16,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5849.95 | bwd_inner: 5700.53 | bwd_allreduce: 149.37 | step: 18.97

 12%|█▏        | 5009/41250 [12:06:41<87:45:47,  8.72s/it]
                                                          
{'loss': 0.058, 'grad_norm': 0.7106497883796692, 'learning_rate': 3.912972356809789e-05, 'epoch': 1.21}

 12%|█▏        | 5009/41250 [12:06:41<87:45:47,  8.72s/it][2025-04-25 20:04:25,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:04:25,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5761.80 | bwd_inner_microstep: 5703.05 | bwd_allreduce_microstep: 58.70 | step_microstep: 18.94
[2025-04-25 20:04:25,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5761.81 | bwd_inner: 5703.05 | bwd_allreduce: 58.72 | step: 18.94

 12%|█▏        | 5010/41250 [12:06:50<87:41:12,  8.71s/it]
                                                          
{'loss': 0.1867, 'grad_norm': 1.5591517686843872, 'learning_rate': 3.912926532334529e-05, 'epoch': 1.21}

 12%|█▏        | 5010/41250 [12:06:50<87:41:12,  8.71s/it][2025-04-25 20:04:33,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:04:33,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5737.35 | bwd_inner_microstep: 5677.38 | bwd_allreduce_microstep: 59.93 | step_microstep: 18.70
[2025-04-25 20:04:33,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5737.37 | bwd_inner: 5677.38 | bwd_allreduce: 59.95 | step: 18.70

 12%|█▏        | 5011/41250 [12:06:59<87:33:32,  8.70s/it]
                                                          
{'loss': 0.1329, 'grad_norm': 1.522217869758606, 'learning_rate': 3.912880696066453e-05, 'epoch': 1.21}

 12%|█▏        | 5011/41250 [12:06:59<87:33:32,  8.70s/it][2025-04-25 20:04:42,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 20:04:42,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5705.51 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.23
[2025-04-25 20:04:42,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.94 | bwd: 5705.52 | bwd_inner: 5692.86 | bwd_allreduce: 12.62 | step: 19.23

 12%|█▏        | 5012/41250 [12:07:07<87:22:36,  8.68s/it]
                                                          
{'loss': 0.1252, 'grad_norm': 1.374525785446167, 'learning_rate': 3.9128348480058444e-05, 'epoch': 1.22}

 12%|█▏        | 5012/41250 [12:07:07<87:22:36,  8.68s/it][2025-04-25 20:04:51,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 20:04:51,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.70 | bwd_microstep: 5738.35 | bwd_inner_microstep: 5687.69 | bwd_allreduce_microstep: 50.62 | step_microstep: 18.55
[2025-04-25 20:04:51,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.70 | bwd: 5738.37 | bwd_inner: 5687.69 | bwd_allreduce: 50.64 | step: 18.56

 12%|█▏        | 5013/41250 [12:07:16<87:21:45,  8.68s/it]
                                                          
{'loss': 0.3143, 'grad_norm': 2.638021230697632, 'learning_rate': 3.912788988152985e-05, 'epoch': 1.22}

 12%|█▏        | 5013/41250 [12:07:16<87:21:45,  8.68s/it][2025-04-25 20:04:59,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 20:04:59,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.02 | bwd_microstep: 5774.64 | bwd_inner_microstep: 5657.53 | bwd_allreduce_microstep: 117.07 | step_microstep: 18.83
[2025-04-25 20:04:59,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.02 | bwd: 5774.65 | bwd_inner: 5657.53 | bwd_allreduce: 117.09 | step: 18.83

 12%|█▏        | 5014/41250 [12:07:25<87:26:06,  8.69s/it]
                                                          
{'loss': 0.4135, 'grad_norm': 6.624649524688721, 'learning_rate': 3.912743116508158e-05, 'epoch': 1.22}

 12%|█▏        | 5014/41250 [12:07:25<87:26:06,  8.69s/it][2025-04-25 20:05:08,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:05:08,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.90 | bwd_microstep: 5686.61 | bwd_inner_microstep: 5643.14 | bwd_allreduce_microstep: 43.43 | step_microstep: 18.62
[2025-04-25 20:05:08,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.90 | bwd: 5686.62 | bwd_inner: 5643.14 | bwd_allreduce: 43.44 | step: 18.63

 12%|█▏        | 5015/41250 [12:07:33<87:10:42,  8.66s/it]
                                                          
{'loss': 0.1049, 'grad_norm': 1.9475866556167603, 'learning_rate': 3.9126972330716455e-05, 'epoch': 1.22}

 12%|█▏        | 5015/41250 [12:07:33<87:10:42,  8.66s/it][2025-04-25 20:05:17,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:05:17,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.49 | bwd_microstep: 5691.07 | bwd_inner_microstep: 5657.91 | bwd_allreduce_microstep: 33.11 | step_microstep: 19.13
[2025-04-25 20:05:17,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.49 | bwd: 5691.08 | bwd_inner: 5657.91 | bwd_allreduce: 33.13 | step: 19.13

 12%|█▏        | 5016/41250 [12:07:42<87:02:45,  8.65s/it]
                                                          
{'loss': 0.1095, 'grad_norm': 1.510125994682312, 'learning_rate': 3.912651337843731e-05, 'epoch': 1.22}

 12%|█▏        | 5016/41250 [12:07:42<87:02:45,  8.65s/it][2025-04-25 20:05:25,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 20:05:25,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.88 | bwd_microstep: 5780.36 | bwd_inner_microstep: 5649.58 | bwd_allreduce_microstep: 130.73 | step_microstep: 18.50
[2025-04-25 20:05:25,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.88 | bwd: 5780.37 | bwd_inner: 5649.58 | bwd_allreduce: 130.74 | step: 18.50

 12%|█▏        | 5017/41250 [12:07:51<87:12:33,  8.66s/it]
                                                          
{'loss': 0.1416, 'grad_norm': 1.1436816453933716, 'learning_rate': 3.912605430824697e-05, 'epoch': 1.22}

 12%|█▏        | 5017/41250 [12:07:51<87:12:33,  8.66s/it][2025-04-25 20:05:34,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 20:05:34,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.29 | bwd_microstep: 5709.31 | bwd_inner_microstep: 5696.59 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.18
[2025-04-25 20:05:34,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.29 | bwd: 5709.32 | bwd_inner: 5696.59 | bwd_allreduce: 12.70 | step: 19.18

 12%|█▏        | 5018/41250 [12:07:59<87:09:44,  8.66s/it]
                                                          
{'loss': 0.2378, 'grad_norm': 2.4052534103393555, 'learning_rate': 3.9125595120148266e-05, 'epoch': 1.22}

 12%|█▏        | 5018/41250 [12:07:59<87:09:44,  8.66s/it][2025-04-25 20:05:43,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 20:05:43,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5703.80 | bwd_inner_microstep: 5690.87 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.89
[2025-04-25 20:05:43,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5703.82 | bwd_inner: 5690.87 | bwd_allreduce: 12.91 | step: 18.89

 12%|█▏        | 5019/41250 [12:08:08<87:05:08,  8.65s/it]
                                                          
{'loss': 0.1368, 'grad_norm': 1.9468337297439575, 'learning_rate': 3.912513581414403e-05, 'epoch': 1.22}

 12%|█▏        | 5019/41250 [12:08:08<87:05:08,  8.65s/it][2025-04-25 20:05:51,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 20:05:51,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.37 | bwd_microstep: 5715.49 | bwd_inner_microstep: 5702.45 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.68
[2025-04-25 20:05:51,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.37 | bwd: 5715.51 | bwd_inner: 5702.45 | bwd_allreduce: 13.01 | step: 18.68

 12%|█▏        | 5020/41250 [12:08:17<87:05:47,  8.65s/it]
                                                          
{'loss': 0.1255, 'grad_norm': 1.4539841413497925, 'learning_rate': 3.91246763902371e-05, 'epoch': 1.22}

 12%|█▏        | 5020/41250 [12:08:17<87:05:47,  8.65s/it][2025-04-25 20:06:00,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:06:00,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.52 | bwd_microstep: 5728.00 | bwd_inner_microstep: 5686.58 | bwd_allreduce_microstep: 41.37 | step_microstep: 18.58
[2025-04-25 20:06:00,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.52 | bwd: 5728.01 | bwd_inner: 5686.58 | bwd_allreduce: 41.39 | step: 18.59

 12%|█▏        | 5021/41250 [12:08:25<87:08:04,  8.66s/it]
                                                          
{'loss': 0.0984, 'grad_norm': 2.749279737472534, 'learning_rate': 3.912421684843029e-05, 'epoch': 1.22}

 12%|█▏        | 5021/41250 [12:08:25<87:08:04,  8.66s/it][2025-04-25 20:06:09,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 20:06:09,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.82 | bwd_microstep: 5718.02 | bwd_inner_microstep: 5705.21 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.67
[2025-04-25 20:06:09,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.82 | bwd: 5718.04 | bwd_inner: 5705.21 | bwd_allreduce: 12.79 | step: 18.67

 12%|█▏        | 5022/41250 [12:08:34<87:08:51,  8.66s/it]
                                                          
{'loss': 0.1086, 'grad_norm': 1.2631206512451172, 'learning_rate': 3.912375718872646e-05, 'epoch': 1.22}

 12%|█▏        | 5022/41250 [12:08:34<87:08:51,  8.66s/it][2025-04-25 20:06:17,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:06:17,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5680.03 | bwd_inner_microstep: 5640.27 | bwd_allreduce_microstep: 39.71 | step_microstep: 18.61
[2025-04-25 20:06:17,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5680.04 | bwd_inner: 5640.27 | bwd_allreduce: 39.73 | step: 18.61

 12%|█▏        | 5023/41250 [12:08:42<86:56:44,  8.64s/it]
                                                          
{'loss': 0.1003, 'grad_norm': 1.817197322845459, 'learning_rate': 3.912329741112841e-05, 'epoch': 1.22}

 12%|█▏        | 5023/41250 [12:08:42<86:56:44,  8.64s/it][2025-04-25 20:06:26,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:06:26,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.54 | bwd_microstep: 5667.42 | bwd_inner_microstep: 5642.49 | bwd_allreduce_microstep: 24.89 | step_microstep: 18.29
[2025-04-25 20:06:26,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.54 | bwd: 5667.43 | bwd_inner: 5642.49 | bwd_allreduce: 24.90 | step: 18.29

 12%|█▏        | 5024/41250 [12:08:51<86:47:35,  8.63s/it]
                                                          
{'loss': 0.1737, 'grad_norm': 4.090746879577637, 'learning_rate': 3.912283751563901e-05, 'epoch': 1.22}

 12%|█▏        | 5024/41250 [12:08:51<86:47:35,  8.63s/it][2025-04-25 20:06:34,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:06:34,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.01 | bwd_microstep: 5715.08 | bwd_inner_microstep: 5695.36 | bwd_allreduce_microstep: 19.68 | step_microstep: 18.62
[2025-04-25 20:06:34,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.01 | bwd: 5715.09 | bwd_inner: 5695.36 | bwd_allreduce: 19.69 | step: 18.63

 12%|█▏        | 5025/41250 [12:09:00<86:51:36,  8.63s/it]
                                                          
{'loss': 0.1028, 'grad_norm': 1.5080643892288208, 'learning_rate': 3.912237750226107e-05, 'epoch': 1.22}

 12%|█▏        | 5025/41250 [12:09:00<86:51:36,  8.63s/it][2025-04-25 20:06:43,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:06:43,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.11 | bwd_microstep: 5700.89 | bwd_inner_microstep: 5688.19 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.50
[2025-04-25 20:06:43,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.11 | bwd: 5700.91 | bwd_inner: 5688.19 | bwd_allreduce: 12.68 | step: 18.50

 12%|█▏        | 5026/41250 [12:09:08<86:53:07,  8.63s/it]
                                                          
{'loss': 0.0487, 'grad_norm': 0.6209538578987122, 'learning_rate': 3.912191737099743e-05, 'epoch': 1.22}

 12%|█▏        | 5026/41250 [12:09:08<86:53:07,  8.63s/it][2025-04-25 20:06:52,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:06:52,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.98 | bwd_microstep: 5715.80 | bwd_inner_microstep: 5702.95 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.70
[2025-04-25 20:06:52,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.98 | bwd: 5715.82 | bwd_inner: 5702.95 | bwd_allreduce: 12.83 | step: 18.70

 12%|█▏        | 5027/41250 [12:09:17<86:59:58,  8.65s/it]
                                                          
{'loss': 0.0862, 'grad_norm': 0.8333643078804016, 'learning_rate': 3.912145712185094e-05, 'epoch': 1.22}

 12%|█▏        | 5027/41250 [12:09:17<86:59:58,  8.65s/it][2025-04-25 20:07:00,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 20:07:00,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.99 | bwd_microstep: 5747.48 | bwd_inner_microstep: 5644.78 | bwd_allreduce_microstep: 102.65 | step_microstep: 18.36
[2025-04-25 20:07:00,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.99 | bwd: 5747.49 | bwd_inner: 5644.78 | bwd_allreduce: 102.67 | step: 18.36

 12%|█▏        | 5028/41250 [12:09:26<87:03:27,  8.65s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 1.395484209060669, 'learning_rate': 3.912099675482442e-05, 'epoch': 1.22}

 12%|█▏        | 5028/41250 [12:09:26<87:03:27,  8.65s/it][2025-04-25 20:07:09,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:07:09,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.44 | bwd_microstep: 5673.11 | bwd_inner_microstep: 5654.19 | bwd_allreduce_microstep: 18.88 | step_microstep: 18.26
[2025-04-25 20:07:09,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.44 | bwd: 5673.12 | bwd_inner: 5654.19 | bwd_allreduce: 18.89 | step: 18.26

 12%|█▏        | 5029/41250 [12:09:34<86:52:36,  8.63s/it]
                                                          
{'loss': 0.1443, 'grad_norm': 1.3053386211395264, 'learning_rate': 3.912053626992072e-05, 'epoch': 1.22}

 12%|█▏        | 5029/41250 [12:09:34<86:52:36,  8.63s/it][2025-04-25 20:07:18,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:07:18,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.13 | bwd_microstep: 5754.42 | bwd_inner_microstep: 5696.53 | bwd_allreduce_microstep: 57.84 | step_microstep: 18.51
[2025-04-25 20:07:18,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.13 | bwd: 5754.43 | bwd_inner: 5696.53 | bwd_allreduce: 57.86 | step: 18.51

 12%|█▏        | 5030/41250 [12:09:43<87:04:10,  8.65s/it]
                                                          
{'loss': 0.1345, 'grad_norm': 1.6842046976089478, 'learning_rate': 3.912007566714267e-05, 'epoch': 1.22}

 12%|█▏        | 5030/41250 [12:09:43<87:04:10,  8.65s/it][2025-04-25 20:07:26,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 20:07:26,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.06 | bwd_microstep: 5857.69 | bwd_inner_microstep: 5652.70 | bwd_allreduce_microstep: 204.95 | step_microstep: 19.04
[2025-04-25 20:07:26,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.06 | bwd: 5857.70 | bwd_inner: 5652.70 | bwd_allreduce: 204.96 | step: 19.04

 12%|█▏        | 5031/41250 [12:09:52<87:25:49,  8.69s/it]
                                                          
{'loss': 0.0773, 'grad_norm': 1.2687104940414429, 'learning_rate': 3.911961494649312e-05, 'epoch': 1.22}

 12%|█▏        | 5031/41250 [12:09:52<87:25:49,  8.69s/it][2025-04-25 20:07:35,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:07:35,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5692.62 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 33.72 | step_microstep: 18.50
[2025-04-25 20:07:35,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5692.63 | bwd_inner: 5658.86 | bwd_allreduce: 33.74 | step: 18.50

 12%|█▏        | 5032/41250 [12:10:00<87:09:32,  8.66s/it]
                                                          
{'loss': 0.0645, 'grad_norm': 0.9208755493164062, 'learning_rate': 3.91191541079749e-05, 'epoch': 1.22}

 12%|█▏        | 5032/41250 [12:10:00<87:09:32,  8.66s/it][2025-04-25 20:07:44,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:07:44,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.54 | bwd_microstep: 5698.92 | bwd_inner_microstep: 5686.35 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.65
[2025-04-25 20:07:44,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.54 | bwd: 5698.93 | bwd_inner: 5686.35 | bwd_allreduce: 12.54 | step: 18.65

 12%|█▏        | 5033/41250 [12:10:09<87:05:42,  8.66s/it]
                                                          
{'loss': 0.1784, 'grad_norm': 2.3678932189941406, 'learning_rate': 3.911869315159086e-05, 'epoch': 1.22}

 12%|█▏        | 5033/41250 [12:10:09<87:05:42,  8.66s/it][2025-04-25 20:07:52,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 20:07:52,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.11 | bwd_microstep: 5763.74 | bwd_inner_microstep: 5651.67 | bwd_allreduce_microstep: 112.02 | step_microstep: 18.42
[2025-04-25 20:07:52,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.11 | bwd: 5763.75 | bwd_inner: 5651.67 | bwd_allreduce: 112.04 | step: 18.43

 12%|█▏        | 5034/41250 [12:10:18<87:10:57,  8.67s/it]
                                                          
{'loss': 0.1842, 'grad_norm': 1.895382046699524, 'learning_rate': 3.9118232077343834e-05, 'epoch': 1.22}

 12%|█▏        | 5034/41250 [12:10:18<87:10:57,  8.67s/it][2025-04-25 20:08:01,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:08:01,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.95 | bwd_microstep: 5702.67 | bwd_inner_microstep: 5690.15 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.64
[2025-04-25 20:08:01,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.95 | bwd: 5702.68 | bwd_inner: 5690.15 | bwd_allreduce: 12.49 | step: 18.64

 12%|█▏        | 5035/41250 [12:10:26<87:06:13,  8.66s/it]
                                                          
{'loss': 0.0506, 'grad_norm': 0.5842990279197693, 'learning_rate': 3.9117770885236665e-05, 'epoch': 1.22}

 12%|█▏        | 5035/41250 [12:10:26<87:06:13,  8.66s/it][2025-04-25 20:08:10,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 20:08:10,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.34 | bwd_microstep: 5755.73 | bwd_inner_microstep: 5641.43 | bwd_allreduce_microstep: 114.26 | step_microstep: 18.53
[2025-04-25 20:08:10,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.34 | bwd: 5755.74 | bwd_inner: 5641.43 | bwd_allreduce: 114.27 | step: 18.54

 12%|█▏        | 5036/41250 [12:10:35<87:07:01,  8.66s/it]
                                                          
{'loss': 0.079, 'grad_norm': 1.7772780656814575, 'learning_rate': 3.911730957527221e-05, 'epoch': 1.22}

 12%|█▏        | 5036/41250 [12:10:35<87:07:01,  8.66s/it][2025-04-25 20:08:18,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 20:08:18,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.27 | bwd_microstep: 5789.60 | bwd_inner_microstep: 5776.85 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.25
[2025-04-25 20:08:18,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.27 | bwd: 5789.62 | bwd_inner: 5776.85 | bwd_allreduce: 12.72 | step: 18.25

 12%|█▏        | 5037/41250 [12:10:44<87:24:57,  8.69s/it]
                                                          
{'loss': 0.0952, 'grad_norm': 2.0197103023529053, 'learning_rate': 3.911684814745329e-05, 'epoch': 1.22}

 12%|█▏        | 5037/41250 [12:10:44<87:24:57,  8.69s/it][2025-04-25 20:08:27,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:08:27,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.68 | bwd_microstep: 5746.62 | bwd_inner_microstep: 5657.93 | bwd_allreduce_microstep: 88.64 | step_microstep: 18.59
[2025-04-25 20:08:27,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.68 | bwd: 5746.63 | bwd_inner: 5657.93 | bwd_allreduce: 88.66 | step: 18.59

 12%|█▏        | 5038/41250 [12:10:52<87:20:32,  8.68s/it]
                                                          
{'loss': 0.1246, 'grad_norm': 2.08027982711792, 'learning_rate': 3.911638660178276e-05, 'epoch': 1.22}

 12%|█▏        | 5038/41250 [12:10:52<87:20:32,  8.68s/it][2025-04-25 20:08:36,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:08:36,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.38 | bwd_microstep: 5826.95 | bwd_inner_microstep: 5705.79 | bwd_allreduce_microstep: 121.11 | step_microstep: 18.69
[2025-04-25 20:08:36,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.38 | bwd: 5826.96 | bwd_inner: 5705.79 | bwd_allreduce: 121.13 | step: 18.69

 12%|█▏        | 5039/41250 [12:11:01<87:38:05,  8.71s/it]
                                                          
{'loss': 0.221, 'grad_norm': 4.040012359619141, 'learning_rate': 3.911592493826348e-05, 'epoch': 1.22}

 12%|█▏        | 5039/41250 [12:11:01<87:38:05,  8.71s/it][2025-04-25 20:08:44,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:08:44,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.34 | bwd_microstep: 5714.95 | bwd_inner_microstep: 5651.90 | bwd_allreduce_microstep: 62.99 | step_microstep: 19.02
[2025-04-25 20:08:44,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.34 | bwd: 5714.96 | bwd_inner: 5651.90 | bwd_allreduce: 63.01 | step: 19.02

 12%|█▏        | 5040/41250 [12:11:10<87:22:39,  8.69s/it]
                                                          
{'loss': 0.0541, 'grad_norm': 0.9043728113174438, 'learning_rate': 3.911546315689827e-05, 'epoch': 1.22}

 12%|█▏        | 5040/41250 [12:11:10<87:22:39,  8.69s/it][2025-04-25 20:08:53,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 20:08:53,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.16 | bwd_microstep: 5769.68 | bwd_inner_microstep: 5721.74 | bwd_allreduce_microstep: 47.89 | step_microstep: 19.32
[2025-04-25 20:08:53,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.16 | bwd: 5769.69 | bwd_inner: 5721.74 | bwd_allreduce: 47.91 | step: 19.32

 12%|█▏        | 5041/41250 [12:11:19<87:26:27,  8.69s/it]
                                                          
{'loss': 0.0546, 'grad_norm': 1.4566926956176758, 'learning_rate': 3.911500125768999e-05, 'epoch': 1.22}

 12%|█▏        | 5041/41250 [12:11:19<87:26:27,  8.69s/it][2025-04-25 20:09:02,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:09:02,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5772.15 | bwd_inner_microstep: 5717.88 | bwd_allreduce_microstep: 54.22 | step_microstep: 18.55
[2025-04-25 20:09:02,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5772.16 | bwd_inner: 5717.88 | bwd_allreduce: 54.24 | step: 18.55

 12%|█▏        | 5042/41250 [12:11:27<87:29:03,  8.70s/it]
                                                          
{'loss': 0.1122, 'grad_norm': 1.566741943359375, 'learning_rate': 3.911453924064149e-05, 'epoch': 1.22}

 12%|█▏        | 5042/41250 [12:11:27<87:29:03,  8.70s/it][2025-04-25 20:09:11,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 20:09:11,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.96 | bwd_microstep: 5735.69 | bwd_inner_microstep: 5722.78 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.04
[2025-04-25 20:09:11,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.96 | bwd: 5735.71 | bwd_inner: 5722.78 | bwd_allreduce: 12.88 | step: 19.04

 12%|█▏        | 5043/41250 [12:11:36<87:25:43,  8.69s/it]
                                                          
{'loss': 0.3892, 'grad_norm': 4.925854206085205, 'learning_rate': 3.911407710575562e-05, 'epoch': 1.22}

 12%|█▏        | 5043/41250 [12:11:36<87:25:43,  8.69s/it][2025-04-25 20:09:19,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-25 20:09:19,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.15 | bwd_microstep: 5768.42 | bwd_inner_microstep: 5669.35 | bwd_allreduce_microstep: 99.02 | step_microstep: 18.75
[2025-04-25 20:09:19,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.15 | bwd: 5768.43 | bwd_inner: 5669.35 | bwd_allreduce: 99.04 | step: 18.75

 12%|█▏        | 5044/41250 [12:11:45<87:24:36,  8.69s/it]
                                                          
{'loss': 0.4229, 'grad_norm': 2.192835569381714, 'learning_rate': 3.911361485303522e-05, 'epoch': 1.22}

 12%|█▏        | 5044/41250 [12:11:45<87:24:36,  8.69s/it][2025-04-25 20:09:28,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:09:28,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.59 | bwd_microstep: 5716.75 | bwd_inner_microstep: 5703.79 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.77
[2025-04-25 20:09:28,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.59 | bwd: 5716.76 | bwd_inner: 5703.79 | bwd_allreduce: 12.93 | step: 18.77

 12%|█▏        | 5045/41250 [12:11:53<87:17:43,  8.68s/it]
                                                          
{'loss': 0.0715, 'grad_norm': 1.2645682096481323, 'learning_rate': 3.9113152482483145e-05, 'epoch': 1.22}

 12%|█▏        | 5045/41250 [12:11:53<87:17:43,  8.68s/it][2025-04-25 20:09:37,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:09:37,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.54 | bwd_microstep: 5718.11 | bwd_inner_microstep: 5705.44 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.65
[2025-04-25 20:09:37,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.54 | bwd: 5718.13 | bwd_inner: 5705.44 | bwd_allreduce: 12.65 | step: 18.65

 12%|█▏        | 5046/41250 [12:12:02<87:13:28,  8.67s/it]
                                                          
{'loss': 0.0781, 'grad_norm': 1.0807932615280151, 'learning_rate': 3.911268999410224e-05, 'epoch': 1.22}

 12%|█▏        | 5046/41250 [12:12:02<87:13:28,  8.67s/it][2025-04-25 20:09:45,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 20:09:45,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.16 | bwd_microstep: 5767.39 | bwd_inner_microstep: 5703.48 | bwd_allreduce_microstep: 63.86 | step_microstep: 18.83
[2025-04-25 20:09:45,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.16 | bwd: 5767.41 | bwd_inner: 5703.48 | bwd_allreduce: 63.88 | step: 18.83

 12%|█▏        | 5047/41250 [12:12:11<87:17:51,  8.68s/it]
                                                          
{'loss': 0.3425, 'grad_norm': 5.316107749938965, 'learning_rate': 3.9112227387895366e-05, 'epoch': 1.22}

 12%|█▏        | 5047/41250 [12:12:11<87:17:51,  8.68s/it][2025-04-25 20:09:54,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:09:54,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5780.40 | bwd_inner_microstep: 5656.04 | bwd_allreduce_microstep: 124.31 | step_microstep: 18.74
[2025-04-25 20:09:54,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5780.41 | bwd_inner: 5656.04 | bwd_allreduce: 124.33 | step: 18.74

 12%|█▏        | 5048/41250 [12:12:19<87:19:45,  8.68s/it]
                                                          
{'loss': 0.1066, 'grad_norm': 1.2257190942764282, 'learning_rate': 3.911176466386537e-05, 'epoch': 1.22}

 12%|█▏        | 5048/41250 [12:12:19<87:19:45,  8.68s/it][2025-04-25 20:10:03,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 1.03
[2025-04-25 20:10:03,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.28 | bwd_microstep: 5781.73 | bwd_inner_microstep: 5719.23 | bwd_allreduce_microstep: 62.44 | step_microstep: 19.97
[2025-04-25 20:10:03,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.28 | bwd: 5781.75 | bwd_inner: 5719.23 | bwd_allreduce: 62.47 | step: 19.98

 12%|█▏        | 5049/41250 [12:12:28<87:27:07,  8.70s/it]
                                                          
{'loss': 0.0202, 'grad_norm': 0.5248527526855469, 'learning_rate': 3.91113018220151e-05, 'epoch': 1.22}

 12%|█▏        | 5049/41250 [12:12:28<87:27:07,  8.70s/it][2025-04-25 20:10:12,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:10:12,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2903.23 | bwd_microstep: 5851.62 | bwd_inner_microstep: 5791.87 | bwd_allreduce_microstep: 59.70 | step_microstep: 18.81
[2025-04-25 20:10:12,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2903.23 | bwd: 5851.63 | bwd_inner: 5791.87 | bwd_allreduce: 59.72 | step: 18.81

 12%|█▏        | 5050/41250 [12:12:37<87:52:27,  8.74s/it]
                                                          
{'loss': 0.2238, 'grad_norm': 1.9877865314483643, 'learning_rate': 3.9110838862347414e-05, 'epoch': 1.22}

 12%|█▏        | 5050/41250 [12:12:37<87:52:27,  8.74s/it][2025-04-25 20:10:20,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 20:10:20,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.40 | bwd_microstep: 5754.95 | bwd_inner_microstep: 5715.22 | bwd_allreduce_microstep: 39.69 | step_microstep: 18.98
[2025-04-25 20:10:20,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.40 | bwd: 5754.96 | bwd_inner: 5715.21 | bwd_allreduce: 39.71 | step: 18.98

 12%|█▏        | 5051/41250 [12:12:46<87:44:47,  8.73s/it]
                                                          
{'loss': 0.0549, 'grad_norm': 0.9897903203964233, 'learning_rate': 3.911037578486516e-05, 'epoch': 1.22}

 12%|█▏        | 5051/41250 [12:12:46<87:44:47,  8.73s/it][2025-04-25 20:10:29,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:10:29,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.82 | bwd_microstep: 5774.23 | bwd_inner_microstep: 5669.81 | bwd_allreduce_microstep: 104.37 | step_microstep: 19.32
[2025-04-25 20:10:29,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.82 | bwd: 5774.24 | bwd_inner: 5669.81 | bwd_allreduce: 104.39 | step: 19.33

 12%|█▏        | 5052/41250 [12:12:54<87:37:56,  8.72s/it]
                                                          
{'loss': 0.1498, 'grad_norm': 2.4899444580078125, 'learning_rate': 3.910991258957121e-05, 'epoch': 1.22}

 12%|█▏        | 5052/41250 [12:12:54<87:37:56,  8.72s/it][2025-04-25 20:10:38,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:10:38,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.36 | bwd_microstep: 5732.87 | bwd_inner_microstep: 5719.95 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.91
[2025-04-25 20:10:38,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.36 | bwd: 5732.88 | bwd_inner: 5719.95 | bwd_allreduce: 12.88 | step: 18.91

 12%|█▏        | 5053/41250 [12:13:03<87:30:01,  8.70s/it]
                                                          
{'loss': 0.1505, 'grad_norm': 1.8523670434951782, 'learning_rate': 3.910944927646839e-05, 'epoch': 1.22}

 12%|█▏        | 5053/41250 [12:13:03<87:30:01,  8.70s/it][2025-04-25 20:10:46,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 20:10:46,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.02 | bwd_microstep: 5709.27 | bwd_inner_microstep: 5658.43 | bwd_allreduce_microstep: 50.80 | step_microstep: 19.04
[2025-04-25 20:10:46,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.02 | bwd: 5709.29 | bwd_inner: 5658.43 | bwd_allreduce: 50.82 | step: 19.04

 12%|█▏        | 5054/41250 [12:13:12<87:17:35,  8.68s/it]
                                                          
{'loss': 0.1403, 'grad_norm': 2.547337293624878, 'learning_rate': 3.910898584555959e-05, 'epoch': 1.23}

 12%|█▏        | 5054/41250 [12:13:12<87:17:35,  8.68s/it][2025-04-25 20:10:55,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:10:55,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.04 | bwd_microstep: 5755.51 | bwd_inner_microstep: 5658.68 | bwd_allreduce_microstep: 96.78 | step_microstep: 18.58
[2025-04-25 20:10:55,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.04 | bwd: 5755.52 | bwd_inner: 5658.68 | bwd_allreduce: 96.80 | step: 18.59

 12%|█▏        | 5055/41250 [12:13:20<87:16:04,  8.68s/it]
                                                          
{'loss': 0.2557, 'grad_norm': 3.742323160171509, 'learning_rate': 3.910852229684764e-05, 'epoch': 1.23}

 12%|█▏        | 5055/41250 [12:13:20<87:16:04,  8.68s/it][2025-04-25 20:11:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:11:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5695.55 | bwd_inner_microstep: 5665.42 | bwd_allreduce_microstep: 30.09 | step_microstep: 18.75
[2025-04-25 20:11:04,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5695.57 | bwd_inner: 5665.42 | bwd_allreduce: 30.11 | step: 18.75

 12%|█▏        | 5056/41250 [12:13:29<87:05:43,  8.66s/it]
                                                          
{'loss': 0.032, 'grad_norm': 0.4658140242099762, 'learning_rate': 3.910805863033542e-05, 'epoch': 1.23}

 12%|█▏        | 5056/41250 [12:13:29<87:05:43,  8.66s/it][2025-04-25 20:11:12,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:11:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.29 | bwd_microstep: 5701.40 | bwd_inner_microstep: 5661.72 | bwd_allreduce_microstep: 39.63 | step_microstep: 18.48
[2025-04-25 20:11:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.29 | bwd: 5701.41 | bwd_inner: 5661.72 | bwd_allreduce: 39.65 | step: 18.48

 12%|█▏        | 5057/41250 [12:13:37<86:58:31,  8.65s/it]
                                                          
{'loss': 0.0369, 'grad_norm': 0.5797690749168396, 'learning_rate': 3.910759484602577e-05, 'epoch': 1.23}

 12%|█▏        | 5057/41250 [12:13:37<86:58:31,  8.65s/it][2025-04-25 20:11:21,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:11:21,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.90 | bwd_microstep: 5708.32 | bwd_inner_microstep: 5695.55 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.61
[2025-04-25 20:11:21,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.90 | bwd: 5708.33 | bwd_inner: 5695.55 | bwd_allreduce: 12.75 | step: 18.61

 12%|█▏        | 5058/41250 [12:13:46<86:57:06,  8.65s/it]
                                                          
{'loss': 0.0862, 'grad_norm': 0.8855921030044556, 'learning_rate': 3.910713094392156e-05, 'epoch': 1.23}

 12%|█▏        | 5058/41250 [12:13:46<86:57:06,  8.65s/it][2025-04-25 20:11:30,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 20:11:30,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.86 | bwd_microstep: 5782.57 | bwd_inner_microstep: 5769.70 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.11
[2025-04-25 20:11:30,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.86 | bwd: 5782.58 | bwd_inner: 5769.70 | bwd_allreduce: 12.84 | step: 19.11

 12%|█▏        | 5059/41250 [12:13:55<87:15:23,  8.68s/it]
                                                          
{'loss': 0.3588, 'grad_norm': 2.986276626586914, 'learning_rate': 3.910666692402564e-05, 'epoch': 1.23}

 12%|█▏        | 5059/41250 [12:13:55<87:15:23,  8.68s/it][2025-04-25 20:11:38,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 20:11:38,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.80 | bwd_microstep: 5695.59 | bwd_inner_microstep: 5662.43 | bwd_allreduce_microstep: 33.11 | step_microstep: 18.95
[2025-04-25 20:11:38,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.80 | bwd: 5695.61 | bwd_inner: 5662.43 | bwd_allreduce: 33.13 | step: 18.95

 12%|█▏        | 5060/41250 [12:14:04<87:05:49,  8.66s/it]
                                                          
{'loss': 0.1269, 'grad_norm': 1.4574915170669556, 'learning_rate': 3.910620278634088e-05, 'epoch': 1.23}

 12%|█▏        | 5060/41250 [12:14:04<87:05:49,  8.66s/it][2025-04-25 20:11:47,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:11:47,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.07 | bwd_microstep: 5746.58 | bwd_inner_microstep: 5710.98 | bwd_allreduce_microstep: 35.55 | step_microstep: 18.62
[2025-04-25 20:11:47,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.07 | bwd: 5746.59 | bwd_inner: 5710.98 | bwd_allreduce: 35.57 | step: 18.62

 12%|█▏        | 5061/41250 [12:14:12<87:09:07,  8.67s/it]
                                                          
{'loss': 0.0477, 'grad_norm': 1.215200424194336, 'learning_rate': 3.910573853087014e-05, 'epoch': 1.23}

 12%|█▏        | 5061/41250 [12:14:12<87:09:07,  8.67s/it][2025-04-25 20:11:56,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 20:11:56,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5726.43 | bwd_inner_microstep: 5678.62 | bwd_allreduce_microstep: 47.77 | step_microstep: 18.55
[2025-04-25 20:11:56,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5726.45 | bwd_inner: 5678.62 | bwd_allreduce: 47.79 | step: 18.55

 12%|█▏        | 5062/41250 [12:14:21<87:07:48,  8.67s/it]
                                                          
{'loss': 0.0256, 'grad_norm': 0.6524646282196045, 'learning_rate': 3.910527415761627e-05, 'epoch': 1.23}

 12%|█▏        | 5062/41250 [12:14:21<87:07:48,  8.67s/it][2025-04-25 20:12:04,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:12:04,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.90 | bwd_microstep: 5687.03 | bwd_inner_microstep: 5649.73 | bwd_allreduce_microstep: 37.26 | step_microstep: 18.64
[2025-04-25 20:12:04,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5687.04 | bwd_inner: 5649.73 | bwd_allreduce: 37.27 | step: 18.64

 12%|█▏        | 5063/41250 [12:14:29<86:54:49,  8.65s/it]
                                                          
{'loss': 0.0624, 'grad_norm': 1.566728115081787, 'learning_rate': 3.9104809666582156e-05, 'epoch': 1.23}

 12%|█▏        | 5063/41250 [12:14:29<86:54:49,  8.65s/it][2025-04-25 20:12:13,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:12:13,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.14 | bwd_microstep: 5686.70 | bwd_inner_microstep: 5660.04 | bwd_allreduce_microstep: 26.61 | step_microstep: 18.46
[2025-04-25 20:12:13,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.14 | bwd: 5686.71 | bwd_inner: 5660.04 | bwd_allreduce: 26.63 | step: 18.46

 12%|█▏        | 5064/41250 [12:14:38<86:46:45,  8.63s/it]
                                                          
{'loss': 0.0184, 'grad_norm': 0.3900707960128784, 'learning_rate': 3.910434505777064e-05, 'epoch': 1.23}

 12%|█▏        | 5064/41250 [12:14:38<86:46:45,  8.63s/it][2025-04-25 20:12:21,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 20:12:21,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5742.36 | bwd_inner_microstep: 5684.24 | bwd_allreduce_microstep: 58.08 | step_microstep: 19.02
[2025-04-25 20:12:21,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5742.37 | bwd_inner: 5684.24 | bwd_allreduce: 58.09 | step: 19.02

 12%|█▏        | 5065/41250 [12:14:47<86:54:01,  8.65s/it]
                                                          
{'loss': 0.0311, 'grad_norm': 0.7081826329231262, 'learning_rate': 3.91038803311846e-05, 'epoch': 1.23}

 12%|█▏        | 5065/41250 [12:14:47<86:54:01,  8.65s/it][2025-04-25 20:12:30,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:12:30,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.50 | bwd_microstep: 5753.66 | bwd_inner_microstep: 5640.64 | bwd_allreduce_microstep: 112.97 | step_microstep: 18.90
[2025-04-25 20:12:30,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.50 | bwd: 5753.67 | bwd_inner: 5640.64 | bwd_allreduce: 112.99 | step: 18.90

 12%|█▏        | 5066/41250 [12:14:55<86:57:07,  8.65s/it]
                                                          
{'loss': 0.1676, 'grad_norm': 3.1262905597686768, 'learning_rate': 3.910341548682689e-05, 'epoch': 1.23}

 12%|█▏        | 5066/41250 [12:14:55<86:57:07,  8.65s/it][2025-04-25 20:12:39,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 20:12:39,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.07 | bwd_microstep: 5879.16 | bwd_inner_microstep: 5638.96 | bwd_allreduce_microstep: 240.15 | step_microstep: 18.37
[2025-04-25 20:12:39,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.07 | bwd: 5879.17 | bwd_inner: 5638.96 | bwd_allreduce: 240.17 | step: 18.37

 12%|█▏        | 5067/41250 [12:15:04<87:23:59,  8.70s/it]
                                                          
{'loss': 0.1829, 'grad_norm': 2.9240429401397705, 'learning_rate': 3.910295052470038e-05, 'epoch': 1.23}

 12%|█▏        | 5067/41250 [12:15:04<87:23:59,  8.70s/it][2025-04-25 20:12:48,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:12:48,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.61 | bwd_microstep: 5748.94 | bwd_inner_microstep: 5645.30 | bwd_allreduce_microstep: 103.60 | step_microstep: 18.47
[2025-04-25 20:12:48,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.61 | bwd: 5748.96 | bwd_inner: 5645.29 | bwd_allreduce: 103.62 | step: 18.48

 12%|█▏        | 5068/41250 [12:15:13<87:17:42,  8.69s/it]
                                                          
{'loss': 0.2031, 'grad_norm': 1.4885188341140747, 'learning_rate': 3.910248544480794e-05, 'epoch': 1.23}

 12%|█▏        | 5068/41250 [12:15:13<87:17:42,  8.69s/it][2025-04-25 20:12:56,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 20:12:56,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.83 | bwd_microstep: 5714.71 | bwd_inner_microstep: 5701.85 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.96
[2025-04-25 20:12:56,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.83 | bwd: 5714.72 | bwd_inner: 5701.85 | bwd_allreduce: 12.83 | step: 18.96

 12%|█▏        | 5069/41250 [12:15:22<87:11:52,  8.68s/it]
                                                          
{'loss': 0.0427, 'grad_norm': 1.332484245300293, 'learning_rate': 3.910202024715244e-05, 'epoch': 1.23}

 12%|█▏        | 5069/41250 [12:15:22<87:11:52,  8.68s/it][2025-04-25 20:13:05,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 20:13:05,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.74 | bwd_microstep: 5753.44 | bwd_inner_microstep: 5635.68 | bwd_allreduce_microstep: 117.71 | step_microstep: 18.75
[2025-04-25 20:13:05,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.74 | bwd: 5753.45 | bwd_inner: 5635.68 | bwd_allreduce: 117.73 | step: 18.75

 12%|█▏        | 5070/41250 [12:15:30<87:10:43,  8.67s/it]
                                                          
{'loss': 0.1721, 'grad_norm': 1.5071417093276978, 'learning_rate': 3.9101554931736736e-05, 'epoch': 1.23}

 12%|█▏        | 5070/41250 [12:15:30<87:10:43,  8.67s/it][2025-04-25 20:13:14,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 20:13:14,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5749.86 | bwd_inner_microstep: 5643.28 | bwd_allreduce_microstep: 106.52 | step_microstep: 18.92
[2025-04-25 20:13:14,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5749.87 | bwd_inner: 5643.28 | bwd_allreduce: 106.55 | step: 18.93

 12%|█▏        | 5071/41250 [12:15:39<87:08:50,  8.67s/it]
                                                          
{'loss': 0.128, 'grad_norm': 1.9287441968917847, 'learning_rate': 3.91010894985637e-05, 'epoch': 1.23}

 12%|█▏        | 5071/41250 [12:15:39<87:08:50,  8.67s/it][2025-04-25 20:13:22,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 20:13:22,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.61 | bwd_microstep: 5704.03 | bwd_inner_microstep: 5691.21 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.48
[2025-04-25 20:13:22,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.61 | bwd: 5704.04 | bwd_inner: 5691.21 | bwd_allreduce: 12.79 | step: 19.48

 12%|█▏        | 5072/41250 [12:15:47<87:05:30,  8.67s/it]
                                                          
{'loss': 0.2056, 'grad_norm': 1.568760633468628, 'learning_rate': 3.910062394763621e-05, 'epoch': 1.23}

 12%|█▏        | 5072/41250 [12:15:47<87:05:30,  8.67s/it][2025-04-25 20:13:31,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:13:31,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.93 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5644.39 | bwd_allreduce_microstep: 71.03 | step_microstep: 18.48
[2025-04-25 20:13:31,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.93 | bwd: 5715.49 | bwd_inner: 5644.39 | bwd_allreduce: 71.05 | step: 18.48

 12%|█▏        | 5073/41250 [12:15:56<87:01:08,  8.66s/it]
                                                          
{'loss': 0.3266, 'grad_norm': 4.317389965057373, 'learning_rate': 3.910015827895713e-05, 'epoch': 1.23}

 12%|█▏        | 5073/41250 [12:15:56<87:01:08,  8.66s/it][2025-04-25 20:13:39,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 20:13:39,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.47 | bwd_microstep: 5717.00 | bwd_inner_microstep: 5650.26 | bwd_allreduce_microstep: 66.69 | step_microstep: 19.47
[2025-04-25 20:13:39,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.47 | bwd: 5717.02 | bwd_inner: 5650.26 | bwd_allreduce: 66.71 | step: 19.47

 12%|█▏        | 5074/41250 [12:16:05<86:57:43,  8.65s/it]
                                                          
{'loss': 0.007, 'grad_norm': 0.10062289237976074, 'learning_rate': 3.909969249252933e-05, 'epoch': 1.23}

 12%|█▏        | 5074/41250 [12:16:05<86:57:43,  8.65s/it][2025-04-25 20:13:48,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 20:13:48,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.19 | bwd_microstep: 5700.42 | bwd_inner_microstep: 5682.22 | bwd_allreduce_microstep: 18.16 | step_microstep: 19.18
[2025-04-25 20:13:48,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.19 | bwd: 5700.44 | bwd_inner: 5682.22 | bwd_allreduce: 18.18 | step: 19.18

 12%|█▏        | 5075/41250 [12:16:13<86:55:47,  8.65s/it]
                                                          
{'loss': 0.0641, 'grad_norm': 1.4348504543304443, 'learning_rate': 3.9099226588355676e-05, 'epoch': 1.23}

 12%|█▏        | 5075/41250 [12:16:13<86:55:47,  8.65s/it][2025-04-25 20:13:57,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 1.15
[2025-04-25 20:13:57,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.56 | bwd_microstep: 5698.30 | bwd_inner_microstep: 5685.72 | bwd_allreduce_microstep: 12.53 | step_microstep: 19.34
[2025-04-25 20:13:57,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.56 | bwd: 5698.31 | bwd_inner: 5685.72 | bwd_allreduce: 12.55 | step: 19.34

 12%|█▏        | 5076/41250 [12:16:22<86:52:41,  8.65s/it]
                                                          
{'loss': 0.1838, 'grad_norm': 3.48087739944458, 'learning_rate': 3.9098760566439056e-05, 'epoch': 1.23}

 12%|█▏        | 5076/41250 [12:16:22<86:52:41,  8.65s/it][2025-04-25 20:14:05,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 20:14:05,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.02 | bwd_microstep: 5673.69 | bwd_inner_microstep: 5654.89 | bwd_allreduce_microstep: 18.75 | step_microstep: 18.92
[2025-04-25 20:14:05,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.02 | bwd: 5673.70 | bwd_inner: 5654.89 | bwd_allreduce: 18.77 | step: 18.92

 12%|█▏        | 5077/41250 [12:16:31<86:43:34,  8.63s/it]
                                                          
{'loss': 0.0892, 'grad_norm': 1.8518404960632324, 'learning_rate': 3.909829442678232e-05, 'epoch': 1.23}

 12%|█▏        | 5077/41250 [12:16:31<86:43:34,  8.63s/it][2025-04-25 20:14:14,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:14:14,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.24 | bwd_microstep: 5699.40 | bwd_inner_microstep: 5686.55 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.52
[2025-04-25 20:14:14,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.24 | bwd: 5699.41 | bwd_inner: 5686.55 | bwd_allreduce: 12.81 | step: 18.52

 12%|█▏        | 5078/41250 [12:16:39<86:46:03,  8.64s/it]
                                                          
{'loss': 0.189, 'grad_norm': 2.3064677715301514, 'learning_rate': 3.9097828169388375e-05, 'epoch': 1.23}

 12%|█▏        | 5078/41250 [12:16:39<86:46:03,  8.64s/it][2025-04-25 20:14:23,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 20:14:23,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.05 | bwd_microstep: 5703.86 | bwd_inner_microstep: 5691.11 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.02
[2025-04-25 20:14:23,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.05 | bwd: 5703.87 | bwd_inner: 5691.11 | bwd_allreduce: 12.73 | step: 19.02

 12%|█▏        | 5079/41250 [12:16:48<86:45:24,  8.63s/it]
                                                          
{'loss': 0.0588, 'grad_norm': 1.8209543228149414, 'learning_rate': 3.9097361794260067e-05, 'epoch': 1.23}

 12%|█▏        | 5079/41250 [12:16:48<86:45:24,  8.63s/it][2025-04-25 20:14:31,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 20:14:31,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.97 | bwd_microstep: 5670.50 | bwd_inner_microstep: 5653.38 | bwd_allreduce_microstep: 17.07 | step_microstep: 18.75
[2025-04-25 20:14:31,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.97 | bwd: 5670.51 | bwd_inner: 5653.38 | bwd_allreduce: 17.09 | step: 18.76

 12%|█▏        | 5080/41250 [12:16:57<86:38:48,  8.62s/it]
                                                          
{'loss': 0.2444, 'grad_norm': 3.145698308944702, 'learning_rate': 3.909689530140028e-05, 'epoch': 1.23}

 12%|█▏        | 5080/41250 [12:16:57<86:38:48,  8.62s/it][2025-04-25 20:14:40,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:14:40,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5745.15 | bwd_inner_microstep: 5704.24 | bwd_allreduce_microstep: 40.87 | step_microstep: 18.70
[2025-04-25 20:14:40,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5745.17 | bwd_inner: 5704.24 | bwd_allreduce: 40.88 | step: 18.70

 12%|█▏        | 5081/41250 [12:17:05<86:49:40,  8.64s/it]
                                                          
{'loss': 0.0343, 'grad_norm': 0.49042725563049316, 'learning_rate': 3.9096428690811886e-05, 'epoch': 1.23}

 12%|█▏        | 5081/41250 [12:17:05<86:49:40,  8.64s/it][2025-04-25 20:14:49,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:14:49,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.25 | bwd_microstep: 5788.74 | bwd_inner_microstep: 5775.84 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.36
[2025-04-25 20:14:49,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.25 | bwd: 5788.75 | bwd_inner: 5775.84 | bwd_allreduce: 12.87 | step: 18.36

 12%|█▏        | 5082/41250 [12:17:14<87:11:31,  8.68s/it]
                                                          
{'loss': 0.1004, 'grad_norm': 1.0388877391815186, 'learning_rate': 3.909596196249776e-05, 'epoch': 1.23}

 12%|█▏        | 5082/41250 [12:17:14<87:11:31,  8.68s/it][2025-04-25 20:14:57,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.13
[2025-04-25 20:14:57,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.37 | bwd_microstep: 5706.95 | bwd_inner_microstep: 5694.05 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.13
[2025-04-25 20:14:57,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.37 | bwd: 5706.97 | bwd_inner: 5694.05 | bwd_allreduce: 12.87 | step: 19.13

 12%|█▏        | 5083/41250 [12:17:23<87:06:07,  8.67s/it]
                                                          
{'loss': 0.1147, 'grad_norm': 2.022648334503174, 'learning_rate': 3.9095495116460794e-05, 'epoch': 1.23}

 12%|█▏        | 5083/41250 [12:17:23<87:06:07,  8.67s/it][2025-04-25 20:15:06,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 20:15:06,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.14 | bwd_microstep: 5774.76 | bwd_inner_microstep: 5640.09 | bwd_allreduce_microstep: 134.62 | step_microstep: 18.64
[2025-04-25 20:15:06,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.14 | bwd: 5774.78 | bwd_inner: 5640.09 | bwd_allreduce: 134.64 | step: 18.65

 12%|█▏        | 5084/41250 [12:17:31<87:07:41,  8.67s/it]
                                                          
{'loss': 0.2314, 'grad_norm': 1.267189383506775, 'learning_rate': 3.909502815270385e-05, 'epoch': 1.23}

 12%|█▏        | 5084/41250 [12:17:31<87:07:41,  8.67s/it][2025-04-25 20:15:15,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:15:15,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5701.61 | bwd_inner_microstep: 5651.38 | bwd_allreduce_microstep: 50.18 | step_microstep: 18.68
[2025-04-25 20:15:15,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5701.62 | bwd_inner: 5651.38 | bwd_allreduce: 50.20 | step: 18.68

 12%|█▏        | 5085/41250 [12:17:40<86:56:29,  8.65s/it]
                                                          
{'loss': 0.0941, 'grad_norm': 6.133732318878174, 'learning_rate': 3.909456107122982e-05, 'epoch': 1.23}

 12%|█▏        | 5085/41250 [12:17:40<86:56:29,  8.65s/it][2025-04-25 20:15:23,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:15:23,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5736.84 | bwd_inner_microstep: 5695.03 | bwd_allreduce_microstep: 41.77 | step_microstep: 18.59
[2025-04-25 20:15:23,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5736.85 | bwd_inner: 5695.03 | bwd_allreduce: 41.78 | step: 18.59

 12%|█▏        | 5086/41250 [12:17:49<86:58:44,  8.66s/it]
                                                          
{'loss': 0.113, 'grad_norm': 1.5671069622039795, 'learning_rate': 3.909409387204158e-05, 'epoch': 1.23}

 12%|█▏        | 5086/41250 [12:17:49<86:58:44,  8.66s/it][2025-04-25 20:15:32,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:15:32,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.77 | bwd_microstep: 5718.65 | bwd_inner_microstep: 5705.87 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.44
[2025-04-25 20:15:32,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.77 | bwd: 5718.66 | bwd_inner: 5705.87 | bwd_allreduce: 12.75 | step: 18.45

 12%|█▏        | 5087/41250 [12:17:57<86:59:22,  8.66s/it]
                                                          
{'loss': 0.0259, 'grad_norm': 0.4037093222141266, 'learning_rate': 3.9093626555142e-05, 'epoch': 1.23}

 12%|█▏        | 5087/41250 [12:17:57<86:59:22,  8.66s/it][2025-04-25 20:15:41,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:15:41,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.73 | bwd_microstep: 5768.88 | bwd_inner_microstep: 5702.71 | bwd_allreduce_microstep: 66.13 | step_microstep: 18.89
[2025-04-25 20:15:41,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.73 | bwd: 5768.90 | bwd_inner: 5702.71 | bwd_allreduce: 66.15 | step: 18.90

 12%|█▏        | 5088/41250 [12:18:06<87:08:02,  8.67s/it]
                                                          
{'loss': 0.0772, 'grad_norm': 1.7295770645141602, 'learning_rate': 3.909315912053397e-05, 'epoch': 1.23}

 12%|█▏        | 5088/41250 [12:18:06<87:08:02,  8.67s/it][2025-04-25 20:15:49,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:15:49,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.33 | bwd_microstep: 5737.13 | bwd_inner_microstep: 5718.22 | bwd_allreduce_microstep: 18.86 | step_microstep: 18.58
[2025-04-25 20:15:49,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.33 | bwd: 5737.14 | bwd_inner: 5718.22 | bwd_allreduce: 18.88 | step: 18.59

 12%|█▏        | 5089/41250 [12:18:15<87:10:08,  8.68s/it]
                                                          
{'loss': 0.0548, 'grad_norm': 1.396491289138794, 'learning_rate': 3.9092691568220365e-05, 'epoch': 1.23}

 12%|█▏        | 5089/41250 [12:18:15<87:10:08,  8.68s/it][2025-04-25 20:15:58,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 20:15:58,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5694.50 | bwd_inner_microstep: 5658.37 | bwd_allreduce_microstep: 36.08 | step_microstep: 18.93
[2025-04-25 20:15:58,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5694.52 | bwd_inner: 5658.37 | bwd_allreduce: 36.10 | step: 18.94

 12%|█▏        | 5090/41250 [12:18:23<86:59:40,  8.66s/it]
                                                          
{'loss': 0.321, 'grad_norm': 4.7440571784973145, 'learning_rate': 3.909222389820408e-05, 'epoch': 1.23}

 12%|█▏        | 5090/41250 [12:18:23<86:59:40,  8.66s/it][2025-04-25 20:16:07,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 20:16:07,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.19 | bwd_microstep: 5890.92 | bwd_inner_microstep: 5692.18 | bwd_allreduce_microstep: 198.69 | step_microstep: 18.55
[2025-04-25 20:16:07,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.19 | bwd: 5890.93 | bwd_inner: 5692.18 | bwd_allreduce: 198.71 | step: 18.56

 12%|█▏        | 5091/41250 [12:18:32<87:29:35,  8.71s/it]
                                                          
{'loss': 0.2472, 'grad_norm': 3.061587333679199, 'learning_rate': 3.909175611048798e-05, 'epoch': 1.23}

 12%|█▏        | 5091/41250 [12:18:32<87:29:35,  8.71s/it][2025-04-25 20:16:15,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-25 20:16:15,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.68 | bwd_microstep: 5702.26 | bwd_inner_microstep: 5666.37 | bwd_allreduce_microstep: 35.84 | step_microstep: 18.61
[2025-04-25 20:16:15,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.69 | bwd: 5702.27 | bwd_inner: 5666.37 | bwd_allreduce: 35.86 | step: 18.61

 12%|█▏        | 5092/41250 [12:18:41<87:13:25,  8.68s/it]
                                                          
{'loss': 0.0473, 'grad_norm': 1.5806008577346802, 'learning_rate': 3.909128820507497e-05, 'epoch': 1.23}

 12%|█▏        | 5092/41250 [12:18:41<87:13:25,  8.68s/it][2025-04-25 20:16:24,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:16:24,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.08 | bwd_microstep: 5874.80 | bwd_inner_microstep: 5708.37 | bwd_allreduce_microstep: 166.37 | step_microstep: 18.57
[2025-04-25 20:16:24,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.08 | bwd: 5874.81 | bwd_inner: 5708.37 | bwd_allreduce: 166.39 | step: 18.57

 12%|█▏        | 5093/41250 [12:18:50<87:38:39,  8.73s/it]
                                                          
{'loss': 0.0711, 'grad_norm': 1.129448652267456, 'learning_rate': 3.9090820181967915e-05, 'epoch': 1.23}

 12%|█▏        | 5093/41250 [12:18:50<87:38:39,  8.73s/it][2025-04-25 20:16:33,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:16:33,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.68 | bwd_microstep: 5746.21 | bwd_inner_microstep: 5713.99 | bwd_allreduce_microstep: 32.18 | step_microstep: 18.76
[2025-04-25 20:16:33,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.68 | bwd: 5746.22 | bwd_inner: 5713.99 | bwd_allreduce: 32.19 | step: 18.76

 12%|█▏        | 5094/41250 [12:18:58<87:33:10,  8.72s/it]
                                                          
{'loss': 0.1173, 'grad_norm': 2.590569019317627, 'learning_rate': 3.909035204116971e-05, 'epoch': 1.23}

 12%|█▏        | 5094/41250 [12:18:58<87:33:10,  8.72s/it][2025-04-25 20:16:42,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:16:42,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.00 | bwd_microstep: 5786.63 | bwd_inner_microstep: 5678.67 | bwd_allreduce_microstep: 107.92 | step_microstep: 18.46
[2025-04-25 20:16:42,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.00 | bwd: 5786.65 | bwd_inner: 5678.67 | bwd_allreduce: 107.94 | step: 18.46

 12%|█▏        | 5095/41250 [12:19:07<87:32:03,  8.72s/it]
                                                          
{'loss': 0.2681, 'grad_norm': 3.9895949363708496, 'learning_rate': 3.908988378268324e-05, 'epoch': 1.24}

 12%|█▏        | 5095/41250 [12:19:07<87:32:03,  8.72s/it][2025-04-25 20:16:50,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:16:50,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.77 | bwd_microstep: 5742.95 | bwd_inner_microstep: 5717.32 | bwd_allreduce_microstep: 25.59 | step_microstep: 18.98
[2025-04-25 20:16:50,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.77 | bwd: 5742.96 | bwd_inner: 5717.32 | bwd_allreduce: 25.60 | step: 18.98

 12%|█▏        | 5096/41250 [12:19:16<87:27:10,  8.71s/it]
                                                          
{'loss': 0.2662, 'grad_norm': 2.6057276725769043, 'learning_rate': 3.9089415406511386e-05, 'epoch': 1.24}

 12%|█▏        | 5096/41250 [12:19:16<87:27:10,  8.71s/it][2025-04-25 20:16:59,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:16:59,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.42 | bwd_microstep: 5714.16 | bwd_inner_microstep: 5701.31 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.64
[2025-04-25 20:16:59,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.42 | bwd: 5714.17 | bwd_inner: 5701.31 | bwd_allreduce: 12.82 | step: 18.64

 12%|█▏        | 5097/41250 [12:19:24<87:16:54,  8.69s/it]
                                                          
{'loss': 0.1282, 'grad_norm': 1.8050233125686646, 'learning_rate': 3.908894691265705e-05, 'epoch': 1.24}

 12%|█▏        | 5097/41250 [12:19:24<87:16:54,  8.69s/it][2025-04-25 20:17:08,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 20:17:08,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.88 | bwd_microstep: 5722.93 | bwd_inner_microstep: 5708.46 | bwd_allreduce_microstep: 14.41 | step_microstep: 19.90
[2025-04-25 20:17:08,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5722.95 | bwd_inner: 5708.46 | bwd_allreduce: 14.44 | step: 19.90

 12%|█▏        | 5098/41250 [12:19:33<87:10:01,  8.68s/it]
                                                          
{'loss': 0.0152, 'grad_norm': 0.23927578330039978, 'learning_rate': 3.90884783011231e-05, 'epoch': 1.24}

 12%|█▏        | 5098/41250 [12:19:33<87:10:01,  8.68s/it][2025-04-25 20:17:16,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:17:16,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.09 | bwd_microstep: 5721.17 | bwd_inner_microstep: 5708.43 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.54
[2025-04-25 20:17:16,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.09 | bwd: 5721.19 | bwd_inner: 5708.43 | bwd_allreduce: 12.72 | step: 18.54

 12%|█▏        | 5099/41250 [12:19:42<87:06:32,  8.67s/it]
                                                          
{'loss': 0.1491, 'grad_norm': 1.816884994506836, 'learning_rate': 3.908800957191245e-05, 'epoch': 1.24}

 12%|█▏        | 5099/41250 [12:19:42<87:06:32,  8.67s/it][2025-04-25 20:17:25,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:17:25,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.08 | bwd_microstep: 5709.58 | bwd_inner_microstep: 5666.36 | bwd_allreduce_microstep: 43.18 | step_microstep: 18.82
[2025-04-25 20:17:25,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.08 | bwd: 5709.59 | bwd_inner: 5666.36 | bwd_allreduce: 43.19 | step: 18.83

 12%|█▏        | 5100/41250 [12:19:50<86:57:21,  8.66s/it]
                                                          
{'loss': 0.126, 'grad_norm': 2.330949068069458, 'learning_rate': 3.908754072502797e-05, 'epoch': 1.24}

 12%|█▏        | 5100/41250 [12:19:50<86:57:21,  8.66s/it][2025-04-25 20:17:34,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:17:34,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5923.10 | bwd_inner_microstep: 5698.02 | bwd_allreduce_microstep: 225.03 | step_microstep: 18.60
[2025-04-25 20:17:34,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5923.11 | bwd_inner: 5698.02 | bwd_allreduce: 225.05 | step: 18.61

 12%|█▏        | 5101/41250 [12:19:59<87:33:02,  8.72s/it]
                                                          
{'loss': 0.0881, 'grad_norm': 1.3635625839233398, 'learning_rate': 3.908707176047255e-05, 'epoch': 1.24}

 12%|█▏        | 5101/41250 [12:19:59<87:33:02,  8.72s/it][2025-04-25 20:17:43,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.11 | optimizer_step: 0.93
[2025-04-25 20:17:43,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.18 | bwd_microstep: 5936.55 | bwd_inner_microstep: 5873.01 | bwd_allreduce_microstep: 63.49 | step_microstep: 18.89
[2025-04-25 20:17:43,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.18 | bwd: 5936.56 | bwd_inner: 5873.01 | bwd_allreduce: 63.50 | step: 18.89

 12%|█▏        | 5102/41250 [12:20:08<88:16:27,  8.79s/it]
                                                          
{'loss': 0.219, 'grad_norm': 4.669386386871338, 'learning_rate': 3.9086602678249095e-05, 'epoch': 1.24}

 12%|█▏        | 5102/41250 [12:20:08<88:16:27,  8.79s/it][2025-04-25 20:17:51,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 20:17:51,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.87 | bwd_microstep: 5729.07 | bwd_inner_microstep: 5669.38 | bwd_allreduce_microstep: 59.65 | step_microstep: 18.91
[2025-04-25 20:17:51,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.87 | bwd: 5729.09 | bwd_inner: 5669.37 | bwd_allreduce: 59.67 | step: 18.91

 12%|█▏        | 5103/41250 [12:20:17<87:51:32,  8.75s/it]
                                                          
{'loss': 0.067, 'grad_norm': 1.1736118793487549, 'learning_rate': 3.9086133478360483e-05, 'epoch': 1.24}

 12%|█▏        | 5103/41250 [12:20:17<87:51:32,  8.75s/it][2025-04-25 20:18:00,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:18:00,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.47 | bwd_microstep: 5719.05 | bwd_inner_microstep: 5660.62 | bwd_allreduce_microstep: 58.39 | step_microstep: 18.58
[2025-04-25 20:18:00,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.47 | bwd: 5719.06 | bwd_inner: 5660.62 | bwd_allreduce: 58.41 | step: 18.58

 12%|█▏        | 5104/41250 [12:20:25<87:29:47,  8.71s/it]
                                                          
{'loss': 0.187, 'grad_norm': 1.2914249897003174, 'learning_rate': 3.908566416080962e-05, 'epoch': 1.24}

 12%|█▏        | 5104/41250 [12:20:25<87:29:47,  8.71s/it][2025-04-25 20:18:09,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:18:09,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.85 | bwd_microstep: 5717.57 | bwd_inner_microstep: 5704.86 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.54
[2025-04-25 20:18:09,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.85 | bwd: 5717.58 | bwd_inner: 5704.86 | bwd_allreduce: 12.67 | step: 18.55

 12%|█▏        | 5105/41250 [12:20:34<87:18:20,  8.70s/it]
                                                          
{'loss': 0.0144, 'grad_norm': 0.3197064697742462, 'learning_rate': 3.908519472559938e-05, 'epoch': 1.24}

 12%|█▏        | 5105/41250 [12:20:34<87:18:20,  8.70s/it][2025-04-25 20:18:17,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:18:17,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.79 | bwd_microstep: 5776.00 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 126.97 | step_microstep: 18.74
[2025-04-25 20:18:17,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.79 | bwd: 5776.02 | bwd_inner: 5648.98 | bwd_allreduce: 127.00 | step: 18.74

 12%|█▏        | 5106/41250 [12:20:43<87:16:26,  8.69s/it]
                                                          
{'loss': 0.5619, 'grad_norm': 2.822614908218384, 'learning_rate': 3.908472517273267e-05, 'epoch': 1.24}

 12%|█▏        | 5106/41250 [12:20:43<87:16:26,  8.69s/it][2025-04-25 20:18:26,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:18:26,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.91 | bwd_microstep: 5727.57 | bwd_inner_microstep: 5713.69 | bwd_allreduce_microstep: 13.84 | step_microstep: 18.79
[2025-04-25 20:18:26,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.91 | bwd: 5727.58 | bwd_inner: 5713.69 | bwd_allreduce: 13.86 | step: 18.79

 12%|█▏        | 5107/41250 [12:20:51<87:14:00,  8.69s/it]
                                                          
{'loss': 0.2245, 'grad_norm': 2.068852186203003, 'learning_rate': 3.908425550221239e-05, 'epoch': 1.24}

 12%|█▏        | 5107/41250 [12:20:51<87:14:00,  8.69s/it][2025-04-25 20:18:35,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 20:18:35,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.52 | bwd_microstep: 5862.17 | bwd_inner_microstep: 5659.37 | bwd_allreduce_microstep: 202.75 | step_microstep: 19.06
[2025-04-25 20:18:35,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.52 | bwd: 5862.18 | bwd_inner: 5659.37 | bwd_allreduce: 202.77 | step: 19.07

 12%|█▏        | 5108/41250 [12:21:00<87:31:16,  8.72s/it]
                                                          
{'loss': 0.2123, 'grad_norm': 1.4290481805801392, 'learning_rate': 3.908378571404142e-05, 'epoch': 1.24}

 12%|█▏        | 5108/41250 [12:21:00<87:31:16,  8.72s/it][2025-04-25 20:18:44,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 20:18:44,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.69 | bwd_microstep: 5888.86 | bwd_inner_microstep: 5647.86 | bwd_allreduce_microstep: 240.94 | step_microstep: 18.94
[2025-04-25 20:18:44,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5888.87 | bwd_inner: 5647.86 | bwd_allreduce: 240.96 | step: 18.94

 12%|█▏        | 5109/41250 [12:21:09<87:48:18,  8.75s/it]
                                                          
{'loss': 0.2894, 'grad_norm': 3.7952444553375244, 'learning_rate': 3.908331580822268e-05, 'epoch': 1.24}

 12%|█▏        | 5109/41250 [12:21:09<87:48:18,  8.75s/it][2025-04-25 20:18:52,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:18:52,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.62 | bwd_microstep: 5728.95 | bwd_inner_microstep: 5716.16 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.72
[2025-04-25 20:18:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.62 | bwd: 5728.96 | bwd_inner: 5716.16 | bwd_allreduce: 12.77 | step: 18.73

 12%|█▏        | 5110/41250 [12:21:18<87:36:02,  8.73s/it]
                                                          
{'loss': 0.0263, 'grad_norm': 0.49451249837875366, 'learning_rate': 3.908284578475903e-05, 'epoch': 1.24}

 12%|█▏        | 5110/41250 [12:21:18<87:36:02,  8.73s/it][2025-04-25 20:19:01,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:19:01,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.70 | bwd_microstep: 5754.47 | bwd_inner_microstep: 5644.59 | bwd_allreduce_microstep: 109.83 | step_microstep: 18.86
[2025-04-25 20:19:01,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.70 | bwd: 5754.48 | bwd_inner: 5644.59 | bwd_allreduce: 109.85 | step: 18.87

 12%|█▏        | 5111/41250 [12:21:26<87:27:55,  8.71s/it]
                                                          
{'loss': 0.4182, 'grad_norm': 3.0642518997192383, 'learning_rate': 3.90823756436534e-05, 'epoch': 1.24}

 12%|█▏        | 5111/41250 [12:21:26<87:27:55,  8.71s/it][2025-04-25 20:19:10,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:19:10,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.07 | bwd_microstep: 5709.07 | bwd_inner_microstep: 5696.28 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.76
[2025-04-25 20:19:10,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.07 | bwd: 5709.08 | bwd_inner: 5696.28 | bwd_allreduce: 12.75 | step: 18.77

 12%|█▏        | 5112/41250 [12:21:35<87:17:33,  8.70s/it]
                                                          
{'loss': 0.1697, 'grad_norm': 2.1752779483795166, 'learning_rate': 3.908190538490869e-05, 'epoch': 1.24}

 12%|█▏        | 5112/41250 [12:21:35<87:17:33,  8.70s/it][2025-04-25 20:19:18,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:19:18,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.59 | bwd_microstep: 5732.61 | bwd_inner_microstep: 5688.58 | bwd_allreduce_microstep: 43.99 | step_microstep: 18.20
[2025-04-25 20:19:18,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.59 | bwd: 5732.62 | bwd_inner: 5688.58 | bwd_allreduce: 44.01 | step: 18.21

 12%|█▏        | 5113/41250 [12:21:44<87:14:31,  8.69s/it]
                                                          
{'loss': 0.0634, 'grad_norm': 1.166725516319275, 'learning_rate': 3.908143500852776e-05, 'epoch': 1.24}

 12%|█▏        | 5113/41250 [12:21:44<87:14:31,  8.69s/it][2025-04-25 20:19:27,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 20:19:27,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5763.10 | bwd_inner_microstep: 5693.88 | bwd_allreduce_microstep: 69.17 | step_microstep: 18.44
[2025-04-25 20:19:27,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5763.12 | bwd_inner: 5693.88 | bwd_allreduce: 69.19 | step: 18.44

 12%|█▏        | 5114/41250 [12:21:52<87:15:51,  8.69s/it]
                                                          
{'loss': 0.1411, 'grad_norm': 1.2683814764022827, 'learning_rate': 3.908096451451356e-05, 'epoch': 1.24}

 12%|█▏        | 5114/41250 [12:21:52<87:15:51,  8.69s/it][2025-04-25 20:19:36,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 20:19:36,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.34 | bwd_microstep: 5735.42 | bwd_inner_microstep: 5689.16 | bwd_allreduce_microstep: 46.22 | step_microstep: 18.68
[2025-04-25 20:19:36,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.34 | bwd: 5735.44 | bwd_inner: 5689.16 | bwd_allreduce: 46.24 | step: 18.68

 12%|█▏        | 5115/41250 [12:22:01<87:12:28,  8.69s/it]
                                                          
{'loss': 0.0524, 'grad_norm': 0.840408444404602, 'learning_rate': 3.908049390286896e-05, 'epoch': 1.24}

 12%|█▏        | 5115/41250 [12:22:01<87:12:28,  8.69s/it][2025-04-25 20:19:44,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:19:44,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.07 | bwd_microstep: 5700.68 | bwd_inner_microstep: 5649.10 | bwd_allreduce_microstep: 51.53 | step_microstep: 18.70
[2025-04-25 20:19:44,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.08 | bwd: 5700.69 | bwd_inner: 5649.10 | bwd_allreduce: 51.55 | step: 18.70

 12%|█▏        | 5116/41250 [12:22:10<86:59:12,  8.67s/it]
                                                          
{'loss': 0.2198, 'grad_norm': 2.6080141067504883, 'learning_rate': 3.908002317359686e-05, 'epoch': 1.24}

 12%|█▏        | 5116/41250 [12:22:10<86:59:12,  8.67s/it][2025-04-25 20:19:53,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:19:53,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.89 | bwd_microstep: 5683.75 | bwd_inner_microstep: 5647.94 | bwd_allreduce_microstep: 35.77 | step_microstep: 18.66
[2025-04-25 20:19:53,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.90 | bwd: 5683.76 | bwd_inner: 5647.94 | bwd_allreduce: 35.78 | step: 18.67

 12%|█▏        | 5117/41250 [12:22:18<86:47:54,  8.65s/it]
                                                          
{'loss': 0.1167, 'grad_norm': 1.8331241607666016, 'learning_rate': 3.9079552326700174e-05, 'epoch': 1.24}

 12%|█▏        | 5117/41250 [12:22:18<86:47:54,  8.65s/it][2025-04-25 20:20:02,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.20 | optimizer_step: 1.02
[2025-04-25 20:20:02,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.93 | bwd_microstep: 5757.93 | bwd_inner_microstep: 5654.01 | bwd_allreduce_microstep: 103.88 | step_microstep: 19.64
[2025-04-25 20:20:02,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.93 | bwd: 5757.94 | bwd_inner: 5654.01 | bwd_allreduce: 103.90 | step: 19.64

 12%|█▏        | 5118/41250 [12:22:27<86:52:12,  8.66s/it]
                                                          
{'loss': 0.1716, 'grad_norm': 4.635616302490234, 'learning_rate': 3.90790813621818e-05, 'epoch': 1.24}

 12%|█▏        | 5118/41250 [12:22:27<86:52:12,  8.66s/it][2025-04-25 20:20:10,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 20:20:10,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.97 | bwd_microstep: 5713.61 | bwd_inner_microstep: 5701.10 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.81
[2025-04-25 20:20:10,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.97 | bwd: 5713.62 | bwd_inner: 5701.10 | bwd_allreduce: 12.49 | step: 18.81

 12%|█▏        | 5119/41250 [12:22:36<86:52:31,  8.66s/it]
                                                          
{'loss': 0.2781, 'grad_norm': 2.351172685623169, 'learning_rate': 3.907861028004465e-05, 'epoch': 1.24}

 12%|█▏        | 5119/41250 [12:22:36<86:52:31,  8.66s/it][2025-04-25 20:20:19,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 20:20:19,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.64 | bwd_microstep: 5738.82 | bwd_inner_microstep: 5684.31 | bwd_allreduce_microstep: 54.47 | step_microstep: 18.55
[2025-04-25 20:20:19,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.64 | bwd: 5738.84 | bwd_inner: 5684.31 | bwd_allreduce: 54.49 | step: 18.55

 12%|█▏        | 5120/41250 [12:22:44<86:54:28,  8.66s/it]
                                                          
{'loss': 0.1146, 'grad_norm': 1.5039328336715698, 'learning_rate': 3.907813908029161e-05, 'epoch': 1.24}

 12%|█▏        | 5120/41250 [12:22:44<86:54:28,  8.66s/it][2025-04-25 20:20:28,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 20:20:28,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.77 | bwd_microstep: 5762.71 | bwd_inner_microstep: 5699.02 | bwd_allreduce_microstep: 63.65 | step_microstep: 19.08
[2025-04-25 20:20:28,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.77 | bwd: 5762.73 | bwd_inner: 5699.02 | bwd_allreduce: 63.67 | step: 19.08

 12%|█▏        | 5121/41250 [12:22:53<87:02:18,  8.67s/it]
                                                          
{'loss': 0.1867, 'grad_norm': 1.7193186283111572, 'learning_rate': 3.90776677629256e-05, 'epoch': 1.24}

 12%|█▏        | 5121/41250 [12:22:53<87:02:18,  8.67s/it][2025-04-25 20:20:36,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 20:20:36,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.99 | bwd_microstep: 5710.28 | bwd_inner_microstep: 5697.90 | bwd_allreduce_microstep: 12.33 | step_microstep: 18.55
[2025-04-25 20:20:36,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.99 | bwd: 5710.29 | bwd_inner: 5697.90 | bwd_allreduce: 12.35 | step: 18.55

 12%|█▏        | 5122/41250 [12:23:02<86:56:18,  8.66s/it]
                                                          
{'loss': 0.1994, 'grad_norm': 2.149970769882202, 'learning_rate': 3.907719632794952e-05, 'epoch': 1.24}

 12%|█▏        | 5122/41250 [12:23:02<86:56:18,  8.66s/it][2025-04-25 20:20:45,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:20:45,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.47 | bwd_microstep: 5682.97 | bwd_inner_microstep: 5645.39 | bwd_allreduce_microstep: 37.53 | step_microstep: 18.72
[2025-04-25 20:20:45,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.47 | bwd: 5682.98 | bwd_inner: 5645.39 | bwd_allreduce: 37.55 | step: 18.72

 12%|█▏        | 5123/41250 [12:23:10<86:43:20,  8.64s/it]
                                                          
{'loss': 0.1035, 'grad_norm': 1.6354224681854248, 'learning_rate': 3.907672477536627e-05, 'epoch': 1.24}

 12%|█▏        | 5123/41250 [12:23:10<86:43:20,  8.64s/it][2025-04-25 20:20:54,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:20:54,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.34 | bwd_microstep: 5781.16 | bwd_inner_microstep: 5768.60 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.54
[2025-04-25 20:20:54,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.34 | bwd: 5781.18 | bwd_inner: 5768.60 | bwd_allreduce: 12.54 | step: 18.54

 12%|█▏        | 5124/41250 [12:23:19<87:02:05,  8.67s/it]
                                                          
{'loss': 0.1726, 'grad_norm': 3.536432981491089, 'learning_rate': 3.907625310517878e-05, 'epoch': 1.24}

 12%|█▏        | 5124/41250 [12:23:19<87:02:05,  8.67s/it][2025-04-25 20:21:02,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:21:02,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.47 | bwd_microstep: 5681.99 | bwd_inner_microstep: 5646.61 | bwd_allreduce_microstep: 35.34 | step_microstep: 18.39
[2025-04-25 20:21:02,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.47 | bwd: 5682.00 | bwd_inner: 5646.61 | bwd_allreduce: 35.35 | step: 18.39

 12%|█▏        | 5125/41250 [12:23:28<86:46:06,  8.65s/it]
                                                          
{'loss': 0.3302, 'grad_norm': 2.0984888076782227, 'learning_rate': 3.907578131738993e-05, 'epoch': 1.24}

 12%|█▏        | 5125/41250 [12:23:28<86:46:06,  8.65s/it][2025-04-25 20:21:11,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:21:11,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5720.94 | bwd_inner_microstep: 5707.93 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.99
[2025-04-25 20:21:11,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5720.96 | bwd_inner: 5707.93 | bwd_allreduce: 12.99 | step: 18.99

 12%|█▏        | 5126/41250 [12:23:36<86:48:17,  8.65s/it]
                                                          
{'loss': 0.2026, 'grad_norm': 1.7363612651824951, 'learning_rate': 3.907530941200264e-05, 'epoch': 1.24}

 12%|█▏        | 5126/41250 [12:23:36<86:48:17,  8.65s/it][2025-04-25 20:21:19,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 20:21:19,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5659.54 | bwd_inner_microstep: 5646.75 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.03
[2025-04-25 20:21:19,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5659.55 | bwd_inner: 5646.75 | bwd_allreduce: 12.76 | step: 19.04

 12%|█▏        | 5127/41250 [12:23:45<86:34:25,  8.63s/it]
                                                          
{'loss': 0.0358, 'grad_norm': 3.837252378463745, 'learning_rate': 3.907483738901982e-05, 'epoch': 1.24}

 12%|█▏        | 5127/41250 [12:23:45<86:34:25,  8.63s/it][2025-04-25 20:21:28,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 20:21:28,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5817.29 | bwd_inner_microstep: 5690.36 | bwd_allreduce_microstep: 126.88 | step_microstep: 19.21
[2025-04-25 20:21:28,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5817.30 | bwd_inner: 5690.36 | bwd_allreduce: 126.90 | step: 19.21

 12%|█▏        | 5128/41250 [12:23:54<86:58:16,  8.67s/it]
                                                          
{'loss': 0.1428, 'grad_norm': 1.5710395574569702, 'learning_rate': 3.907436524844438e-05, 'epoch': 1.24}

 12%|█▏        | 5128/41250 [12:23:54<86:58:16,  8.67s/it][2025-04-25 20:21:37,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:21:37,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.23 | bwd_microstep: 5817.42 | bwd_inner_microstep: 5699.91 | bwd_allreduce_microstep: 117.47 | step_microstep: 18.88
[2025-04-25 20:21:37,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.23 | bwd: 5817.44 | bwd_inner: 5699.91 | bwd_allreduce: 117.48 | step: 18.88

 12%|█▏        | 5129/41250 [12:24:02<87:16:28,  8.70s/it]
                                                          
{'loss': 0.1586, 'grad_norm': 1.6313564777374268, 'learning_rate': 3.907389299027923e-05, 'epoch': 1.24}

 12%|█▏        | 5129/41250 [12:24:02<87:16:28,  8.70s/it][2025-04-25 20:21:46,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 20:21:46,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.19 | bwd_microstep: 5738.65 | bwd_inner_microstep: 5653.40 | bwd_allreduce_microstep: 85.19 | step_microstep: 18.70
[2025-04-25 20:21:46,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.19 | bwd: 5738.66 | bwd_inner: 5653.40 | bwd_allreduce: 85.21 | step: 18.71

 12%|█▏        | 5130/41250 [12:24:11<87:07:15,  8.68s/it]
                                                          
{'loss': 0.1257, 'grad_norm': 1.4172769784927368, 'learning_rate': 3.907342061452729e-05, 'epoch': 1.24}

 12%|█▏        | 5130/41250 [12:24:11<87:07:15,  8.68s/it][2025-04-25 20:21:54,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 20:21:54,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.08 | bwd_microstep: 5672.92 | bwd_inner_microstep: 5652.66 | bwd_allreduce_microstep: 20.21 | step_microstep: 19.23
[2025-04-25 20:21:54,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.08 | bwd: 5672.93 | bwd_inner: 5652.66 | bwd_allreduce: 20.23 | step: 19.23

 12%|█▏        | 5131/41250 [12:24:20<86:50:45,  8.66s/it]
                                                          
{'loss': 0.1122, 'grad_norm': 1.4044033288955688, 'learning_rate': 3.9072948121191456e-05, 'epoch': 1.24}

 12%|█▏        | 5131/41250 [12:24:20<86:50:45,  8.66s/it][2025-04-25 20:22:03,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:22:03,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.79 | bwd_microstep: 5865.90 | bwd_inner_microstep: 5852.97 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.78
[2025-04-25 20:22:03,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2929.79 | bwd: 5865.91 | bwd_inner: 5852.97 | bwd_allreduce: 12.90 | step: 18.79

 12%|█▏        | 5132/41250 [12:24:28<87:30:37,  8.72s/it]
                                                          
{'loss': 0.2958, 'grad_norm': 2.7824742794036865, 'learning_rate': 3.907247551027465e-05, 'epoch': 1.24}

 12%|█▏        | 5132/41250 [12:24:28<87:30:37,  8.72s/it][2025-04-25 20:22:12,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 20:22:12,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.11 | bwd_microstep: 5682.06 | bwd_inner_microstep: 5669.53 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.61
[2025-04-25 20:22:12,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.11 | bwd: 5682.07 | bwd_inner: 5669.53 | bwd_allreduce: 12.50 | step: 18.61

 12%|█▏        | 5133/41250 [12:24:37<87:10:19,  8.69s/it]
                                                          
{'loss': 0.1446, 'grad_norm': 2.38338565826416, 'learning_rate': 3.907200278177978e-05, 'epoch': 1.24}

 12%|█▏        | 5133/41250 [12:24:37<87:10:19,  8.69s/it][2025-04-25 20:22:20,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:22:20,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.00 | bwd_microstep: 5717.46 | bwd_inner_microstep: 5702.10 | bwd_allreduce_microstep: 15.31 | step_microstep: 18.66
[2025-04-25 20:22:20,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.00 | bwd: 5717.47 | bwd_inner: 5702.10 | bwd_allreduce: 15.33 | step: 18.67

 12%|█▏        | 5134/41250 [12:24:46<87:01:26,  8.67s/it]
                                                          
{'loss': 0.0646, 'grad_norm': 0.7593385577201843, 'learning_rate': 3.907152993570976e-05, 'epoch': 1.24}

 12%|█▏        | 5134/41250 [12:24:46<87:01:26,  8.67s/it][2025-04-25 20:22:29,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:22:29,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.82 | bwd_microstep: 5717.96 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 18.72 | step_microstep: 19.00
[2025-04-25 20:22:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.83 | bwd: 5717.97 | bwd_inner: 5699.18 | bwd_allreduce: 18.74 | step: 19.00

 12%|█▏        | 5135/41250 [12:24:54<86:57:08,  8.67s/it]
                                                          
{'loss': 0.1776, 'grad_norm': 1.68631112575531, 'learning_rate': 3.907105697206752e-05, 'epoch': 1.24}

 12%|█▏        | 5135/41250 [12:24:54<86:57:08,  8.67s/it][2025-04-25 20:22:38,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-25 20:22:38,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5700.13 | bwd_inner_microstep: 5687.41 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.11
[2025-04-25 20:22:38,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5700.14 | bwd_inner: 5687.41 | bwd_allreduce: 12.69 | step: 19.11

 12%|█▏        | 5136/41250 [12:25:03<86:50:57,  8.66s/it]
                                                          
{'loss': 0.1131, 'grad_norm': 2.1146373748779297, 'learning_rate': 3.9070583890855964e-05, 'epoch': 1.25}

 12%|█▏        | 5136/41250 [12:25:03<86:50:57,  8.66s/it][2025-04-25 20:22:46,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 20:22:46,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.87 | bwd_microstep: 5699.46 | bwd_inner_microstep: 5665.03 | bwd_allreduce_microstep: 34.39 | step_microstep: 18.92
[2025-04-25 20:22:46,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.87 | bwd: 5699.47 | bwd_inner: 5665.03 | bwd_allreduce: 34.41 | step: 18.92

 12%|█▏        | 5137/41250 [12:25:12<86:42:47,  8.64s/it]
                                                          
{'loss': 0.0915, 'grad_norm': 0.976996123790741, 'learning_rate': 3.9070110692078e-05, 'epoch': 1.25}

 12%|█▏        | 5137/41250 [12:25:12<86:42:47,  8.64s/it][2025-04-25 20:22:55,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:22:55,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5763.66 | bwd_inner_microstep: 5649.06 | bwd_allreduce_microstep: 114.55 | step_microstep: 18.98
[2025-04-25 20:22:55,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5763.68 | bwd_inner: 5649.06 | bwd_allreduce: 114.58 | step: 18.99

 12%|█▏        | 5138/41250 [12:25:20<86:48:51,  8.65s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.3338494300842285, 'learning_rate': 3.906963737573656e-05, 'epoch': 1.25}

 12%|█▏        | 5138/41250 [12:25:20<86:48:51,  8.65s/it][2025-04-25 20:23:04,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:23:04,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5742.39 | bwd_inner_microstep: 5701.73 | bwd_allreduce_microstep: 40.61 | step_microstep: 18.55
[2025-04-25 20:23:04,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5742.41 | bwd_inner: 5701.73 | bwd_allreduce: 40.63 | step: 18.55

 12%|█▏        | 5139/41250 [12:25:29<86:53:32,  8.66s/it]
                                                          
{'loss': 0.1431, 'grad_norm': 1.5251471996307373, 'learning_rate': 3.9069163941834556e-05, 'epoch': 1.25}

 12%|█▏        | 5139/41250 [12:25:29<86:53:32,  8.66s/it][2025-04-25 20:23:12,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 20:23:12,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.43 | bwd_microstep: 5715.75 | bwd_inner_microstep: 5703.07 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.21
[2025-04-25 20:23:12,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.43 | bwd: 5715.77 | bwd_inner: 5703.07 | bwd_allreduce: 12.66 | step: 19.21

 12%|█▏        | 5140/41250 [12:25:38<86:52:03,  8.66s/it]
                                                          
{'loss': 0.1738, 'grad_norm': 1.2734259366989136, 'learning_rate': 3.906869039037491e-05, 'epoch': 1.25}

 12%|█▏        | 5140/41250 [12:25:38<86:52:03,  8.66s/it][2025-04-25 20:23:21,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:23:21,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.23 | bwd_microstep: 5788.05 | bwd_inner_microstep: 5661.96 | bwd_allreduce_microstep: 126.04 | step_microstep: 18.77
[2025-04-25 20:23:21,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.23 | bwd: 5788.06 | bwd_inner: 5661.96 | bwd_allreduce: 126.06 | step: 18.78

 12%|█▏        | 5141/41250 [12:25:46<87:01:28,  8.68s/it]
                                                          
{'loss': 0.0951, 'grad_norm': 1.9146206378936768, 'learning_rate': 3.9068216721360536e-05, 'epoch': 1.25}

 12%|█▏        | 5141/41250 [12:25:46<87:01:28,  8.68s/it][2025-04-25 20:23:30,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:23:30,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.93 | bwd_microstep: 5715.93 | bwd_inner_microstep: 5665.81 | bwd_allreduce_microstep: 50.07 | step_microstep: 18.69
[2025-04-25 20:23:30,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.93 | bwd: 5715.94 | bwd_inner: 5665.81 | bwd_allreduce: 50.09 | step: 18.69

 12%|█▏        | 5142/41250 [12:25:55<86:52:48,  8.66s/it]
                                                          
{'loss': 0.0466, 'grad_norm': 0.6211170554161072, 'learning_rate': 3.906774293479435e-05, 'epoch': 1.25}

 12%|█▏        | 5142/41250 [12:25:55<86:52:48,  8.66s/it][2025-04-25 20:23:38,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:23:38,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.91 | bwd_microstep: 5781.14 | bwd_inner_microstep: 5659.65 | bwd_allreduce_microstep: 121.44 | step_microstep: 18.79
[2025-04-25 20:23:38,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.91 | bwd: 5781.16 | bwd_inner: 5659.65 | bwd_allreduce: 121.46 | step: 18.79

 12%|█▏        | 5143/41250 [12:26:04<86:57:38,  8.67s/it]
                                                          
{'loss': 0.088, 'grad_norm': 1.0007144212722778, 'learning_rate': 3.906726903067929e-05, 'epoch': 1.25}

 12%|█▏        | 5143/41250 [12:26:04<86:57:38,  8.67s/it][2025-04-25 20:23:47,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 20:23:47,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.90 | bwd_microstep: 5739.64 | bwd_inner_microstep: 5704.44 | bwd_allreduce_microstep: 35.15 | step_microstep: 18.87
[2025-04-25 20:23:47,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.90 | bwd: 5739.66 | bwd_inner: 5704.44 | bwd_allreduce: 35.17 | step: 18.87

 12%|█▏        | 5144/41250 [12:26:12<86:59:31,  8.67s/it]
                                                          
{'loss': 0.0586, 'grad_norm': 0.705538272857666, 'learning_rate': 3.906679500901826e-05, 'epoch': 1.25}

 12%|█▏        | 5144/41250 [12:26:12<86:59:31,  8.67s/it][2025-04-25 20:23:56,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 1.13
[2025-04-25 20:23:56,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.83 | bwd_microstep: 5784.90 | bwd_inner_microstep: 5661.79 | bwd_allreduce_microstep: 123.06 | step_microstep: 19.48
[2025-04-25 20:23:56,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.84 | bwd: 5784.92 | bwd_inner: 5661.79 | bwd_allreduce: 123.08 | step: 19.48

 12%|█▏        | 5145/41250 [12:26:21<87:04:47,  8.68s/it]
                                                          
{'loss': 0.0786, 'grad_norm': 0.97065269947052, 'learning_rate': 3.906632086981419e-05, 'epoch': 1.25}

 12%|█▏        | 5145/41250 [12:26:21<87:04:47,  8.68s/it][2025-04-25 20:24:04,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:24:04,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5809.11 | bwd_inner_microstep: 5672.81 | bwd_allreduce_microstep: 136.26 | step_microstep: 18.98
[2025-04-25 20:24:04,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5809.12 | bwd_inner: 5672.81 | bwd_allreduce: 136.28 | step: 18.98

 12%|█▏        | 5146/41250 [12:26:30<87:11:27,  8.69s/it]
                                                          
{'loss': 0.1352, 'grad_norm': 1.9754246473312378, 'learning_rate': 3.906584661307e-05, 'epoch': 1.25}

 12%|█▏        | 5146/41250 [12:26:30<87:11:27,  8.69s/it][2025-04-25 20:24:13,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:24:13,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5803.12 | bwd_inner_microstep: 5662.71 | bwd_allreduce_microstep: 140.37 | step_microstep: 18.85
[2025-04-25 20:24:13,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5803.13 | bwd_inner: 5662.71 | bwd_allreduce: 140.38 | step: 18.85

 12%|█▏        | 5147/41250 [12:26:38<87:15:18,  8.70s/it]
                                                          
{'loss': 0.1775, 'grad_norm': 1.4949736595153809, 'learning_rate': 3.906537223878863e-05, 'epoch': 1.25}

 12%|█▏        | 5147/41250 [12:26:38<87:15:18,  8.70s/it][2025-04-25 20:24:22,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 20:24:22,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.87 | bwd_microstep: 5776.35 | bwd_inner_microstep: 5664.81 | bwd_allreduce_microstep: 111.50 | step_microstep: 18.92
[2025-04-25 20:24:22,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.87 | bwd: 5776.36 | bwd_inner: 5664.80 | bwd_allreduce: 111.52 | step: 18.92

 12%|█▏        | 5148/41250 [12:26:47<87:15:15,  8.70s/it]
                                                          
{'loss': 0.1774, 'grad_norm': 1.8159130811691284, 'learning_rate': 3.906489774697298e-05, 'epoch': 1.25}

 12%|█▏        | 5148/41250 [12:26:47<87:15:15,  8.70s/it][2025-04-25 20:24:30,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:24:30,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.19 | bwd_microstep: 5768.61 | bwd_inner_microstep: 5714.74 | bwd_allreduce_microstep: 53.82 | step_microstep: 18.74
[2025-04-25 20:24:30,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.19 | bwd: 5768.62 | bwd_inner: 5714.74 | bwd_allreduce: 53.84 | step: 18.74

 12%|█▏        | 5149/41250 [12:26:56<87:16:34,  8.70s/it]
                                                          
{'loss': 0.1902, 'grad_norm': 1.4925401210784912, 'learning_rate': 3.9064423137625975e-05, 'epoch': 1.25}

 12%|█▏        | 5149/41250 [12:26:56<87:16:34,  8.70s/it][2025-04-25 20:24:39,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:24:39,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5795.28 | bwd_inner_microstep: 5690.01 | bwd_allreduce_microstep: 105.23 | step_microstep: 18.97
[2025-04-25 20:24:39,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5795.30 | bwd_inner: 5690.01 | bwd_allreduce: 105.25 | step: 18.97

 12%|█▏        | 5150/41250 [12:27:05<87:21:25,  8.71s/it]
                                                          
{'loss': 0.1051, 'grad_norm': 1.2674483060836792, 'learning_rate': 3.906394841075056e-05, 'epoch': 1.25}

 12%|█▏        | 5150/41250 [12:27:05<87:21:25,  8.71s/it][2025-04-25 20:24:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 20:24:48,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5721.92 | bwd_inner_microstep: 5648.60 | bwd_allreduce_microstep: 73.27 | step_microstep: 18.91
[2025-04-25 20:24:48,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5721.94 | bwd_inner: 5648.60 | bwd_allreduce: 73.29 | step: 18.91

 12%|█▏        | 5151/41250 [12:27:13<87:06:25,  8.69s/it]
                                                          
{'loss': 0.1267, 'grad_norm': 1.634439468383789, 'learning_rate': 3.9063473566349657e-05, 'epoch': 1.25}

 12%|█▏        | 5151/41250 [12:27:13<87:06:25,  8.69s/it][2025-04-25 20:24:57,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:24:57,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.99 | bwd_microstep: 5785.51 | bwd_inner_microstep: 5772.82 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.70
[2025-04-25 20:24:57,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.99 | bwd: 5785.52 | bwd_inner: 5772.82 | bwd_allreduce: 12.66 | step: 18.70

 12%|█▏        | 5152/41250 [12:27:22<87:18:56,  8.71s/it]
                                                          
{'loss': 0.1445, 'grad_norm': 1.6852047443389893, 'learning_rate': 3.906299860442619e-05, 'epoch': 1.25}

 12%|█▏        | 5152/41250 [12:27:22<87:18:56,  8.71s/it][2025-04-25 20:25:05,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:25:05,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.21 | bwd_microstep: 5787.80 | bwd_inner_microstep: 5661.46 | bwd_allreduce_microstep: 126.29 | step_microstep: 18.59
[2025-04-25 20:25:05,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.21 | bwd: 5787.81 | bwd_inner: 5661.46 | bwd_allreduce: 126.31 | step: 18.59

 12%|█▏        | 5153/41250 [12:27:31<87:18:59,  8.71s/it]
                                                          
{'loss': 0.1909, 'grad_norm': 1.799399733543396, 'learning_rate': 3.9062523524983076e-05, 'epoch': 1.25}

 12%|█▏        | 5153/41250 [12:27:31<87:18:59,  8.71s/it][2025-04-25 20:25:14,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:25:14,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5760.18 | bwd_inner_microstep: 5700.35 | bwd_allreduce_microstep: 59.78 | step_microstep: 18.63
[2025-04-25 20:25:14,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5760.19 | bwd_inner: 5700.35 | bwd_allreduce: 59.80 | step: 18.63

 12%|█▏        | 5154/41250 [12:27:39<87:17:04,  8.71s/it]
                                                          
{'loss': 0.1917, 'grad_norm': 2.320650577545166, 'learning_rate': 3.906204832802326e-05, 'epoch': 1.25}

 12%|█▏        | 5154/41250 [12:27:39<87:17:04,  8.71s/it][2025-04-25 20:25:23,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 20:25:23,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.83 | bwd_microstep: 5753.28 | bwd_inner_microstep: 5696.68 | bwd_allreduce_microstep: 56.54 | step_microstep: 19.23
[2025-04-25 20:25:23,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.83 | bwd: 5753.30 | bwd_inner: 5696.68 | bwd_allreduce: 56.56 | step: 19.23

 12%|█▏        | 5155/41250 [12:27:48<87:14:20,  8.70s/it]
                                                          
{'loss': 0.0364, 'grad_norm': 0.6395145654678345, 'learning_rate': 3.906157301354966e-05, 'epoch': 1.25}

 12%|█▏        | 5155/41250 [12:27:48<87:14:20,  8.70s/it][2025-04-25 20:25:31,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:25:31,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.06 | bwd_microstep: 5732.37 | bwd_inner_microstep: 5719.45 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.95
[2025-04-25 20:25:31,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.06 | bwd: 5732.38 | bwd_inner: 5719.45 | bwd_allreduce: 12.90 | step: 18.96

 12%|█▏        | 5156/41250 [12:27:57<87:09:27,  8.69s/it]
                                                          
{'loss': 0.1876, 'grad_norm': 1.9100866317749023, 'learning_rate': 3.9061097581565215e-05, 'epoch': 1.25}

 12%|█▏        | 5156/41250 [12:27:57<87:09:27,  8.69s/it][2025-04-25 20:25:40,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 20:25:40,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5759.20 | bwd_inner_microstep: 5696.40 | bwd_allreduce_microstep: 62.75 | step_microstep: 19.21
[2025-04-25 20:25:40,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5759.22 | bwd_inner: 5696.40 | bwd_allreduce: 62.77 | step: 19.21

 13%|█▎        | 5157/41250 [12:28:05<87:09:56,  8.69s/it]
                                                          
{'loss': 0.0335, 'grad_norm': 0.8751997947692871, 'learning_rate': 3.906062203207285e-05, 'epoch': 1.25}

 13%|█▎        | 5157/41250 [12:28:05<87:09:56,  8.69s/it][2025-04-25 20:25:49,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:25:49,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.04 | bwd_microstep: 5741.94 | bwd_inner_microstep: 5697.60 | bwd_allreduce_microstep: 44.29 | step_microstep: 19.28
[2025-04-25 20:25:49,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.04 | bwd: 5741.95 | bwd_inner: 5697.60 | bwd_allreduce: 44.30 | step: 19.27

 13%|█▎        | 5158/41250 [12:28:14<87:07:37,  8.69s/it]
                                                          
{'loss': 0.143, 'grad_norm': 1.7243084907531738, 'learning_rate': 3.906014636507551e-05, 'epoch': 1.25}

 13%|█▎        | 5158/41250 [12:28:14<87:07:37,  8.69s/it][2025-04-25 20:25:58,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 20:25:58,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.01 | bwd_microstep: 5876.28 | bwd_inner_microstep: 5658.14 | bwd_allreduce_microstep: 218.10 | step_microstep: 18.73
[2025-04-25 20:25:58,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.01 | bwd: 5876.30 | bwd_inner: 5658.14 | bwd_allreduce: 218.11 | step: 18.73

 13%|█▎        | 5159/41250 [12:28:23<87:24:29,  8.72s/it]
                                                          
{'loss': 0.0296, 'grad_norm': 0.5002350807189941, 'learning_rate': 3.905967058057611e-05, 'epoch': 1.25}

 13%|█▎        | 5159/41250 [12:28:23<87:24:29,  8.72s/it][2025-04-25 20:26:06,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:26:06,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.44 | bwd_microstep: 5693.78 | bwd_inner_microstep: 5662.20 | bwd_allreduce_microstep: 31.54 | step_microstep: 18.91
[2025-04-25 20:26:06,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.45 | bwd: 5693.79 | bwd_inner: 5662.20 | bwd_allreduce: 31.55 | step: 18.91

 13%|█▎        | 5160/41250 [12:28:31<87:04:48,  8.69s/it]
                                                          
{'loss': 0.0175, 'grad_norm': 0.47299832105636597, 'learning_rate': 3.9059194678577587e-05, 'epoch': 1.25}

 13%|█▎        | 5160/41250 [12:28:31<87:04:48,  8.69s/it][2025-04-25 20:26:15,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.25 | optimizer_step: 1.03
[2025-04-25 20:26:15,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5764.05 | bwd_inner_microstep: 5692.40 | bwd_allreduce_microstep: 71.60 | step_microstep: 19.92
[2025-04-25 20:26:15,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.70 | bwd: 5764.07 | bwd_inner: 5692.40 | bwd_allreduce: 71.63 | step: 19.92

 13%|█▎        | 5161/41250 [12:28:40<87:05:47,  8.69s/it]
                                                          
{'loss': 0.2249, 'grad_norm': 1.955040693283081, 'learning_rate': 3.905871865908288e-05, 'epoch': 1.25}

 13%|█▎        | 5161/41250 [12:28:40<87:05:47,  8.69s/it][2025-04-25 20:26:24,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 20:26:24,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.65 | bwd_microstep: 5718.21 | bwd_inner_microstep: 5705.38 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.93
[2025-04-25 20:26:24,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.65 | bwd: 5718.22 | bwd_inner: 5705.38 | bwd_allreduce: 12.80 | step: 18.93

 13%|█▎        | 5162/41250 [12:28:49<87:01:23,  8.68s/it]
                                                          
{'loss': 0.0437, 'grad_norm': 1.9246015548706055, 'learning_rate': 3.905824252209492e-05, 'epoch': 1.25}

 13%|█▎        | 5162/41250 [12:28:49<87:01:23,  8.68s/it][2025-04-25 20:26:32,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:26:32,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.47 | bwd_microstep: 5762.61 | bwd_inner_microstep: 5653.40 | bwd_allreduce_microstep: 109.17 | step_microstep: 18.56
[2025-04-25 20:26:32,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.47 | bwd: 5762.63 | bwd_inner: 5653.40 | bwd_allreduce: 109.18 | step: 18.56

 13%|█▎        | 5163/41250 [12:28:58<87:02:10,  8.68s/it]
                                                          
{'loss': 0.3352, 'grad_norm': 1.823768138885498, 'learning_rate': 3.905776626761664e-05, 'epoch': 1.25}

 13%|█▎        | 5163/41250 [12:28:58<87:02:10,  8.68s/it][2025-04-25 20:26:41,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-25 20:26:41,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.39 | bwd_microstep: 5753.76 | bwd_inner_microstep: 5643.03 | bwd_allreduce_microstep: 110.68 | step_microstep: 19.01
[2025-04-25 20:26:41,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.39 | bwd: 5753.78 | bwd_inner: 5643.03 | bwd_allreduce: 110.70 | step: 19.01

 13%|█▎        | 5164/41250 [12:29:06<87:00:42,  8.68s/it]
                                                          
{'loss': 0.2047, 'grad_norm': 1.6656275987625122, 'learning_rate': 3.9057289895650985e-05, 'epoch': 1.25}

 13%|█▎        | 5164/41250 [12:29:06<87:00:42,  8.68s/it][2025-04-25 20:26:50,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:26:50,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.13 | bwd_microstep: 5769.77 | bwd_inner_microstep: 5647.73 | bwd_allreduce_microstep: 122.00 | step_microstep: 18.58
[2025-04-25 20:26:50,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.13 | bwd: 5769.79 | bwd_inner: 5647.73 | bwd_allreduce: 122.01 | step: 18.58

 13%|█▎        | 5165/41250 [12:29:15<87:00:16,  8.68s/it]
                                                          
{'loss': 0.0548, 'grad_norm': 0.9107530117034912, 'learning_rate': 3.905681340620088e-05, 'epoch': 1.25}

 13%|█▎        | 5165/41250 [12:29:15<87:00:16,  8.68s/it][2025-04-25 20:26:58,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.11 | optimizer_step: 1.21
[2025-04-25 20:26:58,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5791.34 | bwd_inner_microstep: 5652.38 | bwd_allreduce_microstep: 138.90 | step_microstep: 19.74
[2025-04-25 20:26:58,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5791.36 | bwd_inner: 5652.38 | bwd_allreduce: 138.93 | step: 19.74

 13%|█▎        | 5166/41250 [12:29:24<87:03:21,  8.69s/it]
                                                          
{'loss': 0.078, 'grad_norm': 1.2179808616638184, 'learning_rate': 3.905633679926928e-05, 'epoch': 1.25}

 13%|█▎        | 5166/41250 [12:29:24<87:03:21,  8.69s/it][2025-04-25 20:27:07,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 20:27:07,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.92 | bwd_microstep: 5698.34 | bwd_inner_microstep: 5685.47 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.95
[2025-04-25 20:27:07,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.92 | bwd: 5698.35 | bwd_inner: 5685.47 | bwd_allreduce: 12.84 | step: 18.95

 13%|█▎        | 5167/41250 [12:29:32<86:52:29,  8.67s/it]
                                                          
{'loss': 0.0869, 'grad_norm': 2.3306570053100586, 'learning_rate': 3.90558600748591e-05, 'epoch': 1.25}

 13%|█▎        | 5167/41250 [12:29:32<86:52:29,  8.67s/it][2025-04-25 20:27:16,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 20:27:16,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5727.39 | bwd_inner_microstep: 5694.90 | bwd_allreduce_microstep: 32.44 | step_microstep: 18.83
[2025-04-25 20:27:16,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5727.41 | bwd_inner: 5694.90 | bwd_allreduce: 32.46 | step: 18.83

 13%|█▎        | 5168/41250 [12:29:41<86:51:29,  8.67s/it]
                                                          
{'loss': 0.1157, 'grad_norm': 1.8850886821746826, 'learning_rate': 3.9055383232973294e-05, 'epoch': 1.25}

 13%|█▎        | 5168/41250 [12:29:41<86:51:29,  8.67s/it][2025-04-25 20:27:24,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:27:24,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.17 | bwd_microstep: 5733.19 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 42.29 | step_microstep: 18.56
[2025-04-25 20:27:24,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.17 | bwd: 5733.20 | bwd_inner: 5690.85 | bwd_allreduce: 42.31 | step: 18.57

 13%|█▎        | 5169/41250 [12:29:50<86:51:33,  8.67s/it]
                                                          
{'loss': 0.1314, 'grad_norm': 1.5204381942749023, 'learning_rate': 3.905490627361481e-05, 'epoch': 1.25}

 13%|█▎        | 5169/41250 [12:29:50<86:51:33,  8.67s/it][2025-04-25 20:27:33,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:27:33,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.19 | bwd_microstep: 5769.34 | bwd_inner_microstep: 5756.27 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.89
[2025-04-25 20:27:33,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.19 | bwd: 5769.35 | bwd_inner: 5756.27 | bwd_allreduce: 13.03 | step: 18.89

 13%|█▎        | 5170/41250 [12:29:58<87:02:30,  8.68s/it]
                                                          
{'loss': 0.2834, 'grad_norm': 2.1598241329193115, 'learning_rate': 3.905442919678656e-05, 'epoch': 1.25}

 13%|█▎        | 5170/41250 [12:29:58<87:02:30,  8.68s/it][2025-04-25 20:27:42,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-25 20:27:42,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5728.28 | bwd_inner_microstep: 5685.69 | bwd_allreduce_microstep: 42.54 | step_microstep: 18.70
[2025-04-25 20:27:42,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5728.29 | bwd_inner: 5685.69 | bwd_allreduce: 42.56 | step: 18.70

 13%|█▎        | 5171/41250 [12:30:07<86:57:25,  8.68s/it]
                                                          
{'loss': 0.0776, 'grad_norm': 1.346614122390747, 'learning_rate': 3.9053952002491515e-05, 'epoch': 1.25}

 13%|█▎        | 5171/41250 [12:30:07<86:57:25,  8.68s/it][2025-04-25 20:27:50,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:27:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.14 | bwd_microstep: 5872.18 | bwd_inner_microstep: 5647.44 | bwd_allreduce_microstep: 224.70 | step_microstep: 18.67
[2025-04-25 20:27:50,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.14 | bwd: 5872.20 | bwd_inner: 5647.44 | bwd_allreduce: 224.72 | step: 18.67

 13%|█▎        | 5172/41250 [12:30:16<87:16:42,  8.71s/it]
                                                          
{'loss': 0.073, 'grad_norm': 1.1311737298965454, 'learning_rate': 3.905347469073259e-05, 'epoch': 1.25}

 13%|█▎        | 5172/41250 [12:30:16<87:16:42,  8.71s/it][2025-04-25 20:27:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 20:27:59,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.03 | bwd_microstep: 5744.92 | bwd_inner_microstep: 5656.67 | bwd_allreduce_microstep: 88.21 | step_microstep: 18.95
[2025-04-25 20:27:59,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.03 | bwd: 5744.93 | bwd_inner: 5656.67 | bwd_allreduce: 88.22 | step: 18.95

 13%|█▎        | 5173/41250 [12:30:24<87:08:38,  8.70s/it]
                                                          
{'loss': 0.2036, 'grad_norm': 3.1469860076904297, 'learning_rate': 3.905299726151275e-05, 'epoch': 1.25}

 13%|█▎        | 5173/41250 [12:30:24<87:08:38,  8.70s/it][2025-04-25 20:28:08,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 20:28:08,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.98 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5688.52 | bwd_allreduce_microstep: 29.27 | step_microstep: 18.38
[2025-04-25 20:28:08,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.98 | bwd: 5717.84 | bwd_inner: 5688.52 | bwd_allreduce: 29.28 | step: 18.38

 13%|█▎        | 5174/41250 [12:30:33<87:01:34,  8.68s/it]
                                                          
{'loss': 0.2401, 'grad_norm': 3.698014259338379, 'learning_rate': 3.905251971483493e-05, 'epoch': 1.25}

 13%|█▎        | 5174/41250 [12:30:33<87:01:34,  8.68s/it][2025-04-25 20:28:16,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 20:28:16,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.95 | bwd_microstep: 5691.56 | bwd_inner_microstep: 5678.75 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.89
[2025-04-25 20:28:16,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.95 | bwd: 5691.58 | bwd_inner: 5678.75 | bwd_allreduce: 12.78 | step: 18.90

 13%|█▎        | 5175/41250 [12:30:42<86:52:21,  8.67s/it]
                                                          
{'loss': 0.146, 'grad_norm': 1.6693072319030762, 'learning_rate': 3.905204205070207e-05, 'epoch': 1.25}

 13%|█▎        | 5175/41250 [12:30:42<86:52:21,  8.67s/it][2025-04-25 20:28:25,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 20:28:25,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.35 | bwd_microstep: 5731.05 | bwd_inner_microstep: 5648.16 | bwd_allreduce_microstep: 82.83 | step_microstep: 18.96
[2025-04-25 20:28:25,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.35 | bwd: 5731.06 | bwd_inner: 5648.16 | bwd_allreduce: 82.86 | step: 18.97

 13%|█▎        | 5176/41250 [12:30:50<86:48:41,  8.66s/it]
                                                          
{'loss': 0.0482, 'grad_norm': 0.6208038330078125, 'learning_rate': 3.905156426911712e-05, 'epoch': 1.25}

 13%|█▎        | 5176/41250 [12:30:50<86:48:41,  8.66s/it][2025-04-25 20:28:34,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:28:34,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.01 | bwd_microstep: 5725.88 | bwd_inner_microstep: 5691.79 | bwd_allreduce_microstep: 34.04 | step_microstep: 18.84
[2025-04-25 20:28:34,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.01 | bwd: 5725.90 | bwd_inner: 5691.79 | bwd_allreduce: 34.06 | step: 18.84

 13%|█▎        | 5177/41250 [12:30:59<86:47:06,  8.66s/it]
                                                          
{'loss': 0.0551, 'grad_norm': 0.625353991985321, 'learning_rate': 3.905108637008302e-05, 'epoch': 1.26}

 13%|█▎        | 5177/41250 [12:30:59<86:47:06,  8.66s/it][2025-04-25 20:28:42,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 20:28:42,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.05 | bwd_microstep: 5733.77 | bwd_inner_microstep: 5691.56 | bwd_allreduce_microstep: 42.16 | step_microstep: 18.53
[2025-04-25 20:28:42,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.05 | bwd: 5733.78 | bwd_inner: 5691.56 | bwd_allreduce: 42.18 | step: 18.53

 13%|█▎        | 5178/41250 [12:31:08<86:47:52,  8.66s/it]
                                                          
{'loss': 0.3999, 'grad_norm': 4.0473737716674805, 'learning_rate': 3.9050608353602723e-05, 'epoch': 1.26}

 13%|█▎        | 5178/41250 [12:31:08<86:47:52,  8.66s/it][2025-04-25 20:28:51,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 20:28:51,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.52 | bwd_microstep: 5726.23 | bwd_inner_microstep: 5675.30 | bwd_allreduce_microstep: 50.89 | step_microstep: 19.24
[2025-04-25 20:28:51,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.52 | bwd: 5726.25 | bwd_inner: 5675.30 | bwd_allreduce: 50.90 | step: 19.24

 13%|█▎        | 5179/41250 [12:31:16<86:45:33,  8.66s/it]
                                                          
{'loss': 0.0665, 'grad_norm': 1.2620210647583008, 'learning_rate': 3.905013021967917e-05, 'epoch': 1.26}

 13%|█▎        | 5179/41250 [12:31:16<86:45:33,  8.66s/it][2025-04-25 20:29:00,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:29:00,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.03 | bwd_microstep: 5846.77 | bwd_inner_microstep: 5690.56 | bwd_allreduce_microstep: 156.16 | step_microstep: 18.79
[2025-04-25 20:29:00,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.03 | bwd: 5846.78 | bwd_inner: 5690.57 | bwd_allreduce: 156.18 | step: 18.79

 13%|█▎        | 5180/41250 [12:31:25<87:06:55,  8.69s/it]
                                                          
{'loss': 0.1691, 'grad_norm': 1.723955750465393, 'learning_rate': 3.904965196831532e-05, 'epoch': 1.26}

 13%|█▎        | 5180/41250 [12:31:25<87:06:55,  8.69s/it][2025-04-25 20:29:08,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:29:08,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5719.53 | bwd_inner_microstep: 5707.17 | bwd_allreduce_microstep: 12.32 | step_microstep: 18.67
[2025-04-25 20:29:08,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5719.55 | bwd_inner: 5707.17 | bwd_allreduce: 12.33 | step: 18.67

 13%|█▎        | 5181/41250 [12:31:34<86:59:38,  8.68s/it]
                                                          
{'loss': 0.3451, 'grad_norm': 5.188592433929443, 'learning_rate': 3.90491735995141e-05, 'epoch': 1.26}

 13%|█▎        | 5181/41250 [12:31:34<86:59:38,  8.68s/it][2025-04-25 20:29:17,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 20:29:17,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.04 | bwd_microstep: 5734.64 | bwd_inner_microstep: 5654.46 | bwd_allreduce_microstep: 80.13 | step_microstep: 18.59
[2025-04-25 20:29:17,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.04 | bwd: 5734.65 | bwd_inner: 5654.46 | bwd_allreduce: 80.15 | step: 18.59

 13%|█▎        | 5182/41250 [12:31:42<86:52:41,  8.67s/it]
                                                          
{'loss': 0.0872, 'grad_norm': 2.5260002613067627, 'learning_rate': 3.904869511327848e-05, 'epoch': 1.26}

 13%|█▎        | 5182/41250 [12:31:42<86:52:41,  8.67s/it][2025-04-25 20:29:26,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 20:29:26,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.19 | bwd_microstep: 5699.95 | bwd_inner_microstep: 5687.07 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.92
[2025-04-25 20:29:26,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5699.96 | bwd_inner: 5687.07 | bwd_allreduce: 12.85 | step: 18.93

 13%|█▎        | 5183/41250 [12:31:51<86:46:59,  8.66s/it]
                                                          
{'loss': 0.2711, 'grad_norm': 3.271097421646118, 'learning_rate': 3.9048216509611396e-05, 'epoch': 1.26}

 13%|█▎        | 5183/41250 [12:31:51<86:46:59,  8.66s/it][2025-04-25 20:29:34,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 20:29:34,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.97 | bwd_microstep: 5757.27 | bwd_inner_microstep: 5642.04 | bwd_allreduce_microstep: 115.18 | step_microstep: 18.92
[2025-04-25 20:29:34,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.97 | bwd: 5757.28 | bwd_inner: 5642.04 | bwd_allreduce: 115.20 | step: 18.92

 13%|█▎        | 5184/41250 [12:32:00<86:48:21,  8.66s/it]
                                                          
{'loss': 0.0741, 'grad_norm': 0.8324564695358276, 'learning_rate': 3.90477377885158e-05, 'epoch': 1.26}

 13%|█▎        | 5184/41250 [12:32:00<86:48:21,  8.66s/it][2025-04-25 20:29:43,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:29:43,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.97 | bwd_microstep: 5693.57 | bwd_inner_microstep: 5640.85 | bwd_allreduce_microstep: 52.67 | step_microstep: 18.27
[2025-04-25 20:29:43,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.97 | bwd: 5693.58 | bwd_inner: 5640.85 | bwd_allreduce: 52.69 | step: 18.27

 13%|█▎        | 5185/41250 [12:32:08<86:36:50,  8.65s/it]
                                                          
{'loss': 0.168, 'grad_norm': 2.392679214477539, 'learning_rate': 3.904725894999466e-05, 'epoch': 1.26}

 13%|█▎        | 5185/41250 [12:32:08<86:36:50,  8.65s/it][2025-04-25 20:29:52,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 20:29:52,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5685.72 | bwd_inner_microstep: 5654.50 | bwd_allreduce_microstep: 31.17 | step_microstep: 18.72
[2025-04-25 20:29:52,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5685.73 | bwd_inner: 5654.50 | bwd_allreduce: 31.19 | step: 18.72

 13%|█▎        | 5186/41250 [12:32:17<86:31:12,  8.64s/it]
                                                          
{'loss': 0.2355, 'grad_norm': 2.7832961082458496, 'learning_rate': 3.904677999405091e-05, 'epoch': 1.26}

 13%|█▎        | 5186/41250 [12:32:17<86:31:12,  8.64s/it][2025-04-25 20:30:00,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 20:30:00,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2943.95 | bwd_microstep: 5896.88 | bwd_inner_microstep: 5884.13 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.39
[2025-04-25 20:30:00,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2943.95 | bwd: 5896.90 | bwd_inner: 5884.13 | bwd_allreduce: 12.72 | step: 18.39

 13%|█▎        | 5187/41250 [12:32:26<87:23:04,  8.72s/it]
                                                          
{'loss': 0.3155, 'grad_norm': 2.2418243885040283, 'learning_rate': 3.90463009206875e-05, 'epoch': 1.26}

 13%|█▎        | 5187/41250 [12:32:26<87:23:04,  8.72s/it][2025-04-25 20:30:09,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:30:09,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.97 | bwd_microstep: 5689.85 | bwd_inner_microstep: 5656.96 | bwd_allreduce_microstep: 32.84 | step_microstep: 18.66
[2025-04-25 20:30:09,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.97 | bwd: 5689.86 | bwd_inner: 5656.96 | bwd_allreduce: 32.86 | step: 18.66

 13%|█▎        | 5188/41250 [12:32:34<87:03:05,  8.69s/it]
                                                          
{'loss': 0.061, 'grad_norm': 1.5095945596694946, 'learning_rate': 3.9045821729907395e-05, 'epoch': 1.26}

 13%|█▎        | 5188/41250 [12:32:34<87:03:05,  8.69s/it][2025-04-25 20:30:18,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:30:18,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.60 | bwd_microstep: 5764.70 | bwd_inner_microstep: 5646.49 | bwd_allreduce_microstep: 118.17 | step_microstep: 18.50
[2025-04-25 20:30:18,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.60 | bwd: 5764.72 | bwd_inner: 5646.49 | bwd_allreduce: 118.19 | step: 18.50

 13%|█▎        | 5189/41250 [12:32:43<87:00:47,  8.69s/it]
                                                          
{'loss': 0.2213, 'grad_norm': 2.3301243782043457, 'learning_rate': 3.904534242171355e-05, 'epoch': 1.26}

 13%|█▎        | 5189/41250 [12:32:43<87:00:47,  8.69s/it][2025-04-25 20:30:26,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:30:26,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.17 | bwd_microstep: 5734.73 | bwd_inner_microstep: 5703.16 | bwd_allreduce_microstep: 31.52 | step_microstep: 18.53
[2025-04-25 20:30:26,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.17 | bwd: 5734.74 | bwd_inner: 5703.16 | bwd_allreduce: 31.54 | step: 18.53

 13%|█▎        | 5190/41250 [12:32:52<86:59:12,  8.68s/it]
                                                          
{'loss': 0.1765, 'grad_norm': 2.400830030441284, 'learning_rate': 3.9044862996108915e-05, 'epoch': 1.26}

 13%|█▎        | 5190/41250 [12:32:52<86:59:12,  8.68s/it][2025-04-25 20:30:35,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 20:30:35,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.53 | bwd_microstep: 5763.91 | bwd_inner_microstep: 5678.40 | bwd_allreduce_microstep: 85.47 | step_microstep: 18.96
[2025-04-25 20:30:35,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.53 | bwd: 5763.92 | bwd_inner: 5678.40 | bwd_allreduce: 85.48 | step: 18.97

 13%|█▎        | 5191/41250 [12:33:00<86:59:29,  8.68s/it]
                                                          
{'loss': 0.1056, 'grad_norm': 3.665532350540161, 'learning_rate': 3.9044383453096445e-05, 'epoch': 1.26}

 13%|█▎        | 5191/41250 [12:33:00<86:59:29,  8.68s/it][2025-04-25 20:30:44,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 20:30:44,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.04 | bwd_microstep: 5690.07 | bwd_inner_microstep: 5663.89 | bwd_allreduce_microstep: 26.13 | step_microstep: 18.69
[2025-04-25 20:30:44,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.04 | bwd: 5690.08 | bwd_inner: 5663.89 | bwd_allreduce: 26.15 | step: 18.69

 13%|█▎        | 5192/41250 [12:33:09<86:47:15,  8.66s/it]
                                                          
{'loss': 0.0401, 'grad_norm': 1.02812659740448, 'learning_rate': 3.904390379267909e-05, 'epoch': 1.26}

 13%|█▎        | 5192/41250 [12:33:09<86:47:15,  8.66s/it][2025-04-25 20:30:52,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:30:52,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.21 | bwd_microstep: 5750.73 | bwd_inner_microstep: 5689.29 | bwd_allreduce_microstep: 61.39 | step_microstep: 18.30
[2025-04-25 20:30:52,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.21 | bwd: 5750.74 | bwd_inner: 5689.29 | bwd_allreduce: 61.41 | step: 18.30

 13%|█▎        | 5193/41250 [12:33:18<86:54:14,  8.68s/it]
                                                          
{'loss': 0.3683, 'grad_norm': 3.2963669300079346, 'learning_rate': 3.9043424014859816e-05, 'epoch': 1.26}

 13%|█▎        | 5193/41250 [12:33:18<86:54:14,  8.68s/it][2025-04-25 20:31:01,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 20:31:01,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5706.47 | bwd_inner_microstep: 5693.74 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.35
[2025-04-25 20:31:01,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5706.48 | bwd_inner: 5693.74 | bwd_allreduce: 12.70 | step: 18.35

 13%|█▎        | 5194/41250 [12:33:26<86:50:19,  8.67s/it]
                                                          
{'loss': 0.2396, 'grad_norm': 1.939832091331482, 'learning_rate': 3.904294411964159e-05, 'epoch': 1.26}

 13%|█▎        | 5194/41250 [12:33:26<86:50:19,  8.67s/it][2025-04-25 20:31:10,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 20:31:10,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2903.65 | bwd_microstep: 5797.92 | bwd_inner_microstep: 5785.19 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.49
[2025-04-25 20:31:10,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2903.65 | bwd: 5797.94 | bwd_inner: 5785.19 | bwd_allreduce: 12.71 | step: 18.50

 13%|█▎        | 5195/41250 [12:33:35<87:11:01,  8.71s/it]
                                                          
{'loss': 0.2349, 'grad_norm': 2.04771089553833, 'learning_rate': 3.904246410702734e-05, 'epoch': 1.26}

 13%|█▎        | 5195/41250 [12:33:35<87:11:01,  8.71s/it][2025-04-25 20:31:19,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 20:31:19,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.23 | bwd_microstep: 5786.73 | bwd_inner_microstep: 5709.01 | bwd_allreduce_microstep: 77.68 | step_microstep: 18.60
[2025-04-25 20:31:19,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.23 | bwd: 5786.75 | bwd_inner: 5709.01 | bwd_allreduce: 77.69 | step: 18.60

 13%|█▎        | 5196/41250 [12:33:44<87:15:45,  8.71s/it]
                                                          
{'loss': 0.1449, 'grad_norm': 2.763888120651245, 'learning_rate': 3.9041983977020057e-05, 'epoch': 1.26}

 13%|█▎        | 5196/41250 [12:33:44<87:15:45,  8.71s/it][2025-04-25 20:31:27,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:31:27,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2869.79 | bwd_microstep: 5739.93 | bwd_inner_microstep: 5709.15 | bwd_allreduce_microstep: 30.73 | step_microstep: 18.53
[2025-04-25 20:31:27,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2869.79 | bwd: 5739.94 | bwd_inner: 5709.15 | bwd_allreduce: 30.75 | step: 18.53

 13%|█▎        | 5197/41250 [12:33:53<87:11:55,  8.71s/it]
                                                          
{'loss': 0.047, 'grad_norm': 0.6097029447555542, 'learning_rate': 3.9041503729622685e-05, 'epoch': 1.26}

 13%|█▎        | 5197/41250 [12:33:53<87:11:55,  8.71s/it][2025-04-25 20:31:36,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 20:31:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.88 | bwd_microstep: 5779.85 | bwd_inner_microstep: 5672.48 | bwd_allreduce_microstep: 107.32 | step_microstep: 19.44
[2025-04-25 20:31:36,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.88 | bwd: 5779.86 | bwd_inner: 5672.48 | bwd_allreduce: 107.34 | step: 19.45

 13%|█▎        | 5198/41250 [12:34:01<87:11:28,  8.71s/it]
                                                          
{'loss': 0.3464, 'grad_norm': 2.8145716190338135, 'learning_rate': 3.904102336483819e-05, 'epoch': 1.26}

 13%|█▎        | 5198/41250 [12:34:01<87:11:28,  8.71s/it][2025-04-25 20:31:45,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 20:31:45,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.49 | bwd_microstep: 5789.75 | bwd_inner_microstep: 5669.32 | bwd_allreduce_microstep: 120.39 | step_microstep: 18.72
[2025-04-25 20:31:45,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.49 | bwd: 5789.76 | bwd_inner: 5669.32 | bwd_allreduce: 120.40 | step: 18.73

 13%|█▎        | 5199/41250 [12:34:10<87:12:45,  8.71s/it]
                                                          
{'loss': 0.0713, 'grad_norm': 1.0165331363677979, 'learning_rate': 3.904054288266953e-05, 'epoch': 1.26}

 13%|█▎        | 5199/41250 [12:34:10<87:12:45,  8.71s/it][2025-04-25 20:31:53,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 20:31:53,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.22 | bwd_microstep: 5811.32 | bwd_inner_microstep: 5675.75 | bwd_allreduce_microstep: 135.52 | step_microstep: 19.37
[2025-04-25 20:31:53,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.22 | bwd: 5811.33 | bwd_inner: 5675.75 | bwd_allreduce: 135.54 | step: 19.37

 13%|█▎        | 5200/41250 [12:34:19<87:17:08,  8.72s/it]
                                                          
{'loss': 0.0126, 'grad_norm': 0.17515237629413605, 'learning_rate': 3.904006228311967e-05, 'epoch': 1.26}

 13%|█▎        | 5200/41250 [12:34:19<87:17:08,  8.72s/it][2025-04-25 20:32:02,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 20:32:02,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5779.12 | bwd_inner_microstep: 5660.19 | bwd_allreduce_microstep: 118.89 | step_microstep: 18.80
[2025-04-25 20:32:02,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5779.14 | bwd_inner: 5660.19 | bwd_allreduce: 118.91 | step: 18.80

 13%|█▎        | 5201/41250 [12:34:28<87:16:44,  8.72s/it]
                                                          
{'loss': 0.2155, 'grad_norm': 2.3512213230133057, 'learning_rate': 3.903958156619156e-05, 'epoch': 1.26}

 13%|█▎        | 5201/41250 [12:34:28<87:16:44,  8.72s/it][2025-04-25 20:32:11,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:32:11,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.80 | bwd_microstep: 5780.32 | bwd_inner_microstep: 5669.87 | bwd_allreduce_microstep: 110.40 | step_microstep: 18.63
[2025-04-25 20:32:11,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.80 | bwd: 5780.33 | bwd_inner: 5669.87 | bwd_allreduce: 110.42 | step: 18.64

 13%|█▎        | 5202/41250 [12:34:36<87:14:05,  8.71s/it]
                                                          
{'loss': 0.101, 'grad_norm': 2.0349414348602295, 'learning_rate': 3.9039100731888187e-05, 'epoch': 1.26}

 13%|█▎        | 5202/41250 [12:34:36<87:14:05,  8.71s/it][2025-04-25 20:32:20,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:32:20,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.48 | bwd_microstep: 5770.40 | bwd_inner_microstep: 5703.92 | bwd_allreduce_microstep: 66.42 | step_microstep: 18.76
[2025-04-25 20:32:20,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.48 | bwd: 5770.41 | bwd_inner: 5703.92 | bwd_allreduce: 66.44 | step: 18.76

 13%|█▎        | 5203/41250 [12:34:45<87:15:12,  8.71s/it]
                                                          
{'loss': 0.0602, 'grad_norm': 0.8866743445396423, 'learning_rate': 3.90386197802125e-05, 'epoch': 1.26}

 13%|█▎        | 5203/41250 [12:34:45<87:15:12,  8.71s/it][2025-04-25 20:32:28,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 20:32:28,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.32 | bwd_microstep: 5707.88 | bwd_inner_microstep: 5667.84 | bwd_allreduce_microstep: 40.00 | step_microstep: 19.20
[2025-04-25 20:32:28,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.32 | bwd: 5707.89 | bwd_inner: 5667.84 | bwd_allreduce: 40.01 | step: 19.21

 13%|█▎        | 5204/41250 [12:34:54<87:00:43,  8.69s/it]
                                                          
{'loss': 0.0507, 'grad_norm': 0.7431170344352722, 'learning_rate': 3.903813871116746e-05, 'epoch': 1.26}

 13%|█▎        | 5204/41250 [12:34:54<87:00:43,  8.69s/it][2025-04-25 20:32:37,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:32:37,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.18 | bwd_microstep: 5726.45 | bwd_inner_microstep: 5713.35 | bwd_allreduce_microstep: 13.05 | step_microstep: 18.31
[2025-04-25 20:32:37,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.18 | bwd: 5726.46 | bwd_inner: 5713.35 | bwd_allreduce: 13.07 | step: 18.32

 13%|█▎        | 5205/41250 [12:35:02<86:57:13,  8.68s/it]
                                                          
{'loss': 0.1761, 'grad_norm': 1.4141696691513062, 'learning_rate': 3.903765752475605e-05, 'epoch': 1.26}

 13%|█▎        | 5205/41250 [12:35:02<86:57:13,  8.68s/it][2025-04-25 20:32:46,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-25 20:32:46,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.41 | bwd_microstep: 5766.68 | bwd_inner_microstep: 5715.24 | bwd_allreduce_microstep: 51.40 | step_microstep: 18.59
[2025-04-25 20:32:46,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.41 | bwd: 5766.70 | bwd_inner: 5715.24 | bwd_allreduce: 51.42 | step: 18.59

 13%|█▎        | 5206/41250 [12:35:11<87:01:28,  8.69s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 1.5096745491027832, 'learning_rate': 3.903717622098121e-05, 'epoch': 1.26}

 13%|█▎        | 5206/41250 [12:35:11<87:01:28,  8.69s/it][2025-04-25 20:32:54,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:32:54,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.13 | bwd_microstep: 5730.94 | bwd_inner_microstep: 5715.37 | bwd_allreduce_microstep: 15.53 | step_microstep: 18.66
[2025-04-25 20:32:54,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.13 | bwd: 5730.96 | bwd_inner: 5715.37 | bwd_allreduce: 15.55 | step: 18.67

 13%|█▎        | 5207/41250 [12:35:20<86:59:05,  8.69s/it]
                                                          
{'loss': 0.1957, 'grad_norm': 1.5843240022659302, 'learning_rate': 3.903669479984594e-05, 'epoch': 1.26}

 13%|█▎        | 5207/41250 [12:35:20<86:59:05,  8.69s/it][2025-04-25 20:33:03,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-25 20:33:03,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.96 | bwd_microstep: 5952.04 | bwd_inner_microstep: 5674.35 | bwd_allreduce_microstep: 277.64 | step_microstep: 18.69
[2025-04-25 20:33:03,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.96 | bwd: 5952.05 | bwd_inner: 5674.35 | bwd_allreduce: 277.66 | step: 18.69

 13%|█▎        | 5208/41250 [12:35:29<87:32:28,  8.74s/it]
                                                          
{'loss': 0.0704, 'grad_norm': 1.0838452577590942, 'learning_rate': 3.903621326135318e-05, 'epoch': 1.26}

 13%|█▎        | 5208/41250 [12:35:29<87:32:28,  8.74s/it][2025-04-25 20:33:12,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.24 | optimizer_step: 1.04
[2025-04-25 20:33:12,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.52 | bwd_microstep: 5773.11 | bwd_inner_microstep: 5649.26 | bwd_allreduce_microstep: 123.79 | step_microstep: 20.08
[2025-04-25 20:33:12,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.52 | bwd: 5773.13 | bwd_inner: 5649.26 | bwd_allreduce: 123.82 | step: 20.09

 13%|█▎        | 5209/41250 [12:35:37<87:21:15,  8.73s/it]
                                                          
{'loss': 0.3394, 'grad_norm': 3.6605679988861084, 'learning_rate': 3.903573160550592e-05, 'epoch': 1.26}

 13%|█▎        | 5209/41250 [12:35:37<87:21:15,  8.73s/it][2025-04-25 20:33:21,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-25 20:33:21,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.14 | bwd_microstep: 5715.29 | bwd_inner_microstep: 5702.36 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.24
[2025-04-25 20:33:21,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.14 | bwd: 5715.30 | bwd_inner: 5702.36 | bwd_allreduce: 12.90 | step: 19.25

 13%|█▎        | 5210/41250 [12:35:46<87:09:51,  8.71s/it]
                                                          
{'loss': 0.0715, 'grad_norm': 1.7776235342025757, 'learning_rate': 3.903524983230711e-05, 'epoch': 1.26}

 13%|█▎        | 5210/41250 [12:35:46<87:09:51,  8.71s/it][2025-04-25 20:33:29,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:33:29,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.38 | bwd_microstep: 5758.67 | bwd_inner_microstep: 5684.56 | bwd_allreduce_microstep: 74.07 | step_microstep: 18.84
[2025-04-25 20:33:29,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.38 | bwd: 5758.68 | bwd_inner: 5684.56 | bwd_allreduce: 74.09 | step: 18.84

 13%|█▎        | 5211/41250 [12:35:55<87:05:29,  8.70s/it]
                                                          
{'loss': 0.0209, 'grad_norm': 0.24360746145248413, 'learning_rate': 3.9034767941759736e-05, 'epoch': 1.26}

 13%|█▎        | 5211/41250 [12:35:55<87:05:29,  8.70s/it][2025-04-25 20:33:38,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:33:38,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5772.48 | bwd_inner_microstep: 5652.55 | bwd_allreduce_microstep: 119.88 | step_microstep: 18.95
[2025-04-25 20:33:38,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5772.49 | bwd_inner: 5652.55 | bwd_allreduce: 119.90 | step: 18.95

 13%|█▎        | 5212/41250 [12:36:03<87:02:06,  8.69s/it]
                                                          
{'loss': 0.3268, 'grad_norm': 2.059321880340576, 'learning_rate': 3.903428593386675e-05, 'epoch': 1.26}

 13%|█▎        | 5212/41250 [12:36:03<87:02:06,  8.69s/it][2025-04-25 20:33:47,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 20:33:47,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.33 | bwd_microstep: 5779.46 | bwd_inner_microstep: 5652.20 | bwd_allreduce_microstep: 127.21 | step_microstep: 19.30
[2025-04-25 20:33:47,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.33 | bwd: 5779.47 | bwd_inner: 5652.20 | bwd_allreduce: 127.23 | step: 19.30

 13%|█▎        | 5213/41250 [12:36:12<87:01:38,  8.69s/it]
                                                          
{'loss': 0.1681, 'grad_norm': 1.590093731880188, 'learning_rate': 3.903380380863115e-05, 'epoch': 1.26}

 13%|█▎        | 5213/41250 [12:36:12<87:01:38,  8.69s/it][2025-04-25 20:33:55,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 20:33:55,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.14 | bwd_microstep: 5732.38 | bwd_inner_microstep: 5697.49 | bwd_allreduce_microstep: 34.85 | step_microstep: 19.27
[2025-04-25 20:33:55,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.14 | bwd: 5732.39 | bwd_inner: 5697.49 | bwd_allreduce: 34.86 | step: 19.27

 13%|█▎        | 5214/41250 [12:36:21<86:58:09,  8.69s/it]
                                                          
{'loss': 0.2126, 'grad_norm': 5.252957344055176, 'learning_rate': 3.9033321566055885e-05, 'epoch': 1.26}

 13%|█▎        | 5214/41250 [12:36:21<86:58:09,  8.69s/it][2025-04-25 20:34:04,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.33 | optimizer_step: 1.06
[2025-04-25 20:34:04,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.57 | bwd_microstep: 5767.19 | bwd_inner_microstep: 5656.19 | bwd_allreduce_microstep: 110.94 | step_microstep: 20.03
[2025-04-25 20:34:04,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.57 | bwd: 5767.21 | bwd_inner: 5656.19 | bwd_allreduce: 110.96 | step: 20.03

 13%|█▎        | 5215/41250 [12:36:29<86:56:04,  8.69s/it]
                                                          
{'loss': 0.0711, 'grad_norm': 1.1080176830291748, 'learning_rate': 3.903283920614393e-05, 'epoch': 1.26}

 13%|█▎        | 5215/41250 [12:36:29<86:56:04,  8.69s/it][2025-04-25 20:34:13,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 20:34:13,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5705.98 | bwd_inner_microstep: 5693.06 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.25
[2025-04-25 20:34:13,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5705.99 | bwd_inner: 5693.06 | bwd_allreduce: 12.89 | step: 19.25

 13%|█▎        | 5216/41250 [12:36:38<86:48:24,  8.67s/it]
                                                          
{'loss': 0.216, 'grad_norm': 2.104565382003784, 'learning_rate': 3.9032356728898274e-05, 'epoch': 1.26}

 13%|█▎        | 5216/41250 [12:36:38<86:48:24,  8.67s/it][2025-04-25 20:34:21,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:34:21,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.04 | bwd_microstep: 5745.96 | bwd_inner_microstep: 5694.48 | bwd_allreduce_microstep: 51.43 | step_microstep: 18.33
[2025-04-25 20:34:21,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.04 | bwd: 5745.98 | bwd_inner: 5694.48 | bwd_allreduce: 51.45 | step: 18.33

 13%|█▎        | 5217/41250 [12:36:47<86:50:01,  8.68s/it]
                                                          
{'loss': 0.2479, 'grad_norm': 2.428995132446289, 'learning_rate': 3.9031874134321885e-05, 'epoch': 1.26}

 13%|█▎        | 5217/41250 [12:36:47<86:50:01,  8.68s/it][2025-04-25 20:34:30,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 20:34:30,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.04 | bwd_microstep: 5684.96 | bwd_inner_microstep: 5671.94 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.19
[2025-04-25 20:34:30,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.05 | bwd: 5684.98 | bwd_inner: 5671.94 | bwd_allreduce: 13.00 | step: 19.19

 13%|█▎        | 5218/41250 [12:36:55<86:38:41,  8.66s/it]
                                                          
{'loss': 0.2144, 'grad_norm': 2.1224312782287598, 'learning_rate': 3.9031391422417725e-05, 'epoch': 1.26}

 13%|█▎        | 5218/41250 [12:36:55<86:38:41,  8.66s/it][2025-04-25 20:34:39,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:34:39,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5745.00 | bwd_inner_microstep: 5702.60 | bwd_allreduce_microstep: 42.35 | step_microstep: 18.75
[2025-04-25 20:34:39,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5745.01 | bwd_inner: 5702.60 | bwd_allreduce: 42.37 | step: 18.76

 13%|█▎        | 5219/41250 [12:37:04<86:43:04,  8.66s/it]
                                                          
{'loss': 0.2486, 'grad_norm': 2.053959369659424, 'learning_rate': 3.903090859318879e-05, 'epoch': 1.27}

 13%|█▎        | 5219/41250 [12:37:04<86:43:04,  8.66s/it][2025-04-25 20:34:47,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:34:47,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.60 | bwd_microstep: 5690.79 | bwd_inner_microstep: 5678.19 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.61
[2025-04-25 20:34:47,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.60 | bwd: 5690.80 | bwd_inner: 5678.19 | bwd_allreduce: 12.56 | step: 18.61

 13%|█▎        | 5220/41250 [12:37:13<86:35:01,  8.65s/it]
                                                          
{'loss': 0.0842, 'grad_norm': 1.325960397720337, 'learning_rate': 3.9030425646638044e-05, 'epoch': 1.27}

 13%|█▎        | 5220/41250 [12:37:13<86:35:01,  8.65s/it][2025-04-25 20:34:56,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.03 | optimizer_step: 1.13
[2025-04-25 20:34:56,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.36 | bwd_microstep: 5683.21 | bwd_inner_microstep: 5655.89 | bwd_allreduce_microstep: 27.28 | step_microstep: 18.96
[2025-04-25 20:34:56,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.36 | bwd: 5683.23 | bwd_inner: 5655.89 | bwd_allreduce: 27.29 | step: 18.96

 13%|█▎        | 5221/41250 [12:37:21<86:23:45,  8.63s/it]
                                                          
{'loss': 0.5577, 'grad_norm': 3.4328904151916504, 'learning_rate': 3.902994258276846e-05, 'epoch': 1.27}

 13%|█▎        | 5221/41250 [12:37:21<86:23:45,  8.63s/it][2025-04-25 20:35:04,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:35:04,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5751.59 | bwd_inner_microstep: 5652.22 | bwd_allreduce_microstep: 99.32 | step_microstep: 18.53
[2025-04-25 20:35:04,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.99 | bwd: 5751.60 | bwd_inner: 5652.22 | bwd_allreduce: 99.34 | step: 18.54

 13%|█▎        | 5222/41250 [12:37:30<86:29:32,  8.64s/it]
                                                          
{'loss': 0.1346, 'grad_norm': 1.3831435441970825, 'learning_rate': 3.902945940158303e-05, 'epoch': 1.27}

 13%|█▎        | 5222/41250 [12:37:30<86:29:32,  8.64s/it][2025-04-25 20:35:13,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 20:35:13,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5687.43 | bwd_inner_microstep: 5640.88 | bwd_allreduce_microstep: 46.50 | step_microstep: 19.09
[2025-04-25 20:35:13,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5687.44 | bwd_inner: 5640.88 | bwd_allreduce: 46.52 | step: 19.10

 13%|█▎        | 5223/41250 [12:37:38<86:21:43,  8.63s/it]
                                                          
{'loss': 0.0339, 'grad_norm': 0.6695288419723511, 'learning_rate': 3.902897610308472e-05, 'epoch': 1.27}

 13%|█▎        | 5223/41250 [12:37:38<86:21:43,  8.63s/it][2025-04-25 20:35:22,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:35:22,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.19 | bwd_microstep: 5715.98 | bwd_inner_microstep: 5703.17 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.77
[2025-04-25 20:35:22,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.19 | bwd: 5715.99 | bwd_inner: 5703.17 | bwd_allreduce: 12.78 | step: 18.78

 13%|█▎        | 5224/41250 [12:37:47<86:25:30,  8.64s/it]
                                                          
{'loss': 0.0688, 'grad_norm': 0.8529980182647705, 'learning_rate': 3.9028492687276515e-05, 'epoch': 1.27}

 13%|█▎        | 5224/41250 [12:37:47<86:25:30,  8.64s/it][2025-04-25 20:35:30,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:35:30,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.56 | bwd_microstep: 5782.85 | bwd_inner_microstep: 5770.32 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.17
[2025-04-25 20:35:30,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.56 | bwd: 5782.86 | bwd_inner: 5770.32 | bwd_allreduce: 12.50 | step: 18.17

 13%|█▎        | 5225/41250 [12:37:56<86:46:26,  8.67s/it]
                                                          
{'loss': 0.125, 'grad_norm': 1.3741517066955566, 'learning_rate': 3.9028009154161403e-05, 'epoch': 1.27}

 13%|█▎        | 5225/41250 [12:37:56<86:46:26,  8.67s/it][2025-04-25 20:35:39,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 20:35:39,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.21 | bwd_microstep: 5748.60 | bwd_inner_microstep: 5680.97 | bwd_allreduce_microstep: 67.58 | step_microstep: 19.28
[2025-04-25 20:35:39,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.21 | bwd: 5748.62 | bwd_inner: 5680.97 | bwd_allreduce: 67.60 | step: 19.28

 13%|█▎        | 5226/41250 [12:38:04<86:47:22,  8.67s/it]
                                                          
{'loss': 0.0568, 'grad_norm': 0.7324853539466858, 'learning_rate': 3.902752550374235e-05, 'epoch': 1.27}

 13%|█▎        | 5226/41250 [12:38:04<86:47:22,  8.67s/it][2025-04-25 20:35:48,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 20:35:48,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5700.50 | bwd_inner_microstep: 5687.62 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.08
[2025-04-25 20:35:48,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5700.51 | bwd_inner: 5687.62 | bwd_allreduce: 12.85 | step: 19.09

 13%|█▎        | 5227/41250 [12:38:13<86:39:18,  8.66s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.225992202758789, 'learning_rate': 3.902704173602235e-05, 'epoch': 1.27}

 13%|█▎        | 5227/41250 [12:38:13<86:39:18,  8.66s/it][2025-04-25 20:35:56,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:35:56,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5714.00 | bwd_inner_microstep: 5679.77 | bwd_allreduce_microstep: 34.18 | step_microstep: 18.82
[2025-04-25 20:35:56,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5714.01 | bwd_inner: 5679.77 | bwd_allreduce: 34.20 | step: 18.83

 13%|█▎        | 5228/41250 [12:38:22<86:36:50,  8.66s/it]
                                                          
{'loss': 0.0538, 'grad_norm': 0.8265960812568665, 'learning_rate': 3.902655785100438e-05, 'epoch': 1.27}

 13%|█▎        | 5228/41250 [12:38:22<86:36:50,  8.66s/it][2025-04-25 20:36:05,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:36:05,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.02 | bwd_microstep: 5679.17 | bwd_inner_microstep: 5649.97 | bwd_allreduce_microstep: 29.15 | step_microstep: 18.47
[2025-04-25 20:36:05,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.02 | bwd: 5679.18 | bwd_inner: 5649.97 | bwd_allreduce: 29.17 | step: 18.47

 13%|█▎        | 5229/41250 [12:38:30<86:24:49,  8.64s/it]
                                                          
{'loss': 0.0856, 'grad_norm': 1.0257365703582764, 'learning_rate': 3.902607384869142e-05, 'epoch': 1.27}

 13%|█▎        | 5229/41250 [12:38:30<86:24:49,  8.64s/it][2025-04-25 20:36:14,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 20:36:14,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.59 | bwd_microstep: 5705.03 | bwd_inner_microstep: 5692.19 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.84
[2025-04-25 20:36:14,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.59 | bwd: 5705.04 | bwd_inner: 5692.19 | bwd_allreduce: 12.81 | step: 18.84

 13%|█▎        | 5230/41250 [12:38:39<86:25:22,  8.64s/it]
                                                          
{'loss': 0.2575, 'grad_norm': 1.9906117916107178, 'learning_rate': 3.902558972908646e-05, 'epoch': 1.27}

 13%|█▎        | 5230/41250 [12:38:39<86:25:22,  8.64s/it][2025-04-25 20:36:22,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 20:36:22,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.07 | bwd_microstep: 5671.55 | bwd_inner_microstep: 5658.14 | bwd_allreduce_microstep: 13.36 | step_microstep: 18.78
[2025-04-25 20:36:22,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.07 | bwd: 5671.56 | bwd_inner: 5658.14 | bwd_allreduce: 13.38 | step: 18.78

 13%|█▎        | 5231/41250 [12:38:48<86:16:34,  8.62s/it]
                                                          
{'loss': 0.1149, 'grad_norm': 3.7135748863220215, 'learning_rate': 3.9025105492192476e-05, 'epoch': 1.27}

 13%|█▎        | 5231/41250 [12:38:48<86:16:34,  8.62s/it][2025-04-25 20:36:31,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 20:36:31,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.25 | bwd_microstep: 5702.09 | bwd_inner_microstep: 5685.83 | bwd_allreduce_microstep: 16.21 | step_microstep: 19.17
[2025-04-25 20:36:31,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5702.10 | bwd_inner: 5685.83 | bwd_allreduce: 16.23 | step: 19.18

 13%|█▎        | 5232/41250 [12:38:56<86:18:55,  8.63s/it]
                                                          
{'loss': 0.0607, 'grad_norm': 0.7787355184555054, 'learning_rate': 3.902462113801246e-05, 'epoch': 1.27}

 13%|█▎        | 5232/41250 [12:38:56<86:18:55,  8.63s/it][2025-04-25 20:36:40,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 20:36:40,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.93 | bwd_microstep: 5760.64 | bwd_inner_microstep: 5636.67 | bwd_allreduce_microstep: 123.92 | step_microstep: 18.85
[2025-04-25 20:36:40,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.93 | bwd: 5760.65 | bwd_inner: 5636.67 | bwd_allreduce: 123.94 | step: 18.85

 13%|█▎        | 5233/41250 [12:39:05<86:26:45,  8.64s/it]
                                                          
{'loss': 0.3226, 'grad_norm': 1.3911728858947754, 'learning_rate': 3.9024136666549406e-05, 'epoch': 1.27}

 13%|█▎        | 5233/41250 [12:39:05<86:26:45,  8.64s/it][2025-04-25 20:36:48,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:36:48,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.11 | bwd_microstep: 5737.04 | bwd_inner_microstep: 5644.76 | bwd_allreduce_microstep: 92.24 | step_microstep: 18.27
[2025-04-25 20:36:48,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.11 | bwd: 5737.05 | bwd_inner: 5644.76 | bwd_allreduce: 92.25 | step: 18.27

 13%|█▎        | 5234/41250 [12:39:14<86:29:46,  8.65s/it]
                                                          
{'loss': 0.0518, 'grad_norm': 0.6512015461921692, 'learning_rate': 3.9023652077806284e-05, 'epoch': 1.27}

 13%|█▎        | 5234/41250 [12:39:14<86:29:46,  8.65s/it][2025-04-25 20:36:57,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:36:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.43 | bwd_microstep: 5729.23 | bwd_inner_microstep: 5640.24 | bwd_allreduce_microstep: 88.94 | step_microstep: 18.76
[2025-04-25 20:36:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.43 | bwd: 5729.24 | bwd_inner: 5640.24 | bwd_allreduce: 88.96 | step: 18.77

 13%|█▎        | 5235/41250 [12:39:22<86:32:20,  8.65s/it]
                                                          
{'loss': 0.1963, 'grad_norm': 1.6571452617645264, 'learning_rate': 3.9023167371786084e-05, 'epoch': 1.27}

 13%|█▎        | 5235/41250 [12:39:22<86:32:20,  8.65s/it][2025-04-25 20:37:05,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:37:05,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.53 | bwd_microstep: 5696.33 | bwd_inner_microstep: 5653.49 | bwd_allreduce_microstep: 42.79 | step_microstep: 18.83
[2025-04-25 20:37:05,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.53 | bwd: 5696.34 | bwd_inner: 5653.49 | bwd_allreduce: 42.81 | step: 18.84

 13%|█▎        | 5236/41250 [12:39:31<86:25:15,  8.64s/it]
                                                          
{'loss': 0.1833, 'grad_norm': 1.9767824411392212, 'learning_rate': 3.9022682548491807e-05, 'epoch': 1.27}

 13%|█▎        | 5236/41250 [12:39:31<86:25:15,  8.64s/it][2025-04-25 20:37:14,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:37:14,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5739.82 | bwd_inner_microstep: 5652.10 | bwd_allreduce_microstep: 87.68 | step_microstep: 18.78
[2025-04-25 20:37:14,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5739.83 | bwd_inner: 5652.10 | bwd_allreduce: 87.69 | step: 18.78

 13%|█▎        | 5237/41250 [12:39:39<86:28:06,  8.64s/it]
                                                          
{'loss': 0.0488, 'grad_norm': 0.972159743309021, 'learning_rate': 3.902219760792643e-05, 'epoch': 1.27}

 13%|█▎        | 5237/41250 [12:39:39<86:28:06,  8.64s/it][2025-04-25 20:37:23,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 20:37:23,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.17 | bwd_microstep: 5735.96 | bwd_inner_microstep: 5690.73 | bwd_allreduce_microstep: 45.18 | step_microstep: 19.22
[2025-04-25 20:37:23,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5735.97 | bwd_inner: 5690.74 | bwd_allreduce: 45.20 | step: 19.23

 13%|█▎        | 5238/41250 [12:39:48<86:32:10,  8.65s/it]
                                                          
{'loss': 0.1001, 'grad_norm': 1.2373214960098267, 'learning_rate': 3.902171255009294e-05, 'epoch': 1.27}

 13%|█▎        | 5238/41250 [12:39:48<86:32:10,  8.65s/it][2025-04-25 20:37:31,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 20:37:31,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.22 | bwd_microstep: 5765.50 | bwd_inner_microstep: 5639.47 | bwd_allreduce_microstep: 125.98 | step_microstep: 18.85
[2025-04-25 20:37:31,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.22 | bwd: 5765.51 | bwd_inner: 5639.47 | bwd_allreduce: 126.00 | step: 18.85

 13%|█▎        | 5239/41250 [12:39:57<86:35:18,  8.66s/it]
                                                          
{'loss': 0.2456, 'grad_norm': 1.967659592628479, 'learning_rate': 3.9021227374994346e-05, 'epoch': 1.27}

 13%|█▎        | 5239/41250 [12:39:57<86:35:18,  8.66s/it][2025-04-25 20:37:40,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 20:37:40,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.32 | bwd_microstep: 5701.98 | bwd_inner_microstep: 5688.46 | bwd_allreduce_microstep: 13.47 | step_microstep: 18.87
[2025-04-25 20:37:40,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.32 | bwd: 5701.99 | bwd_inner: 5688.46 | bwd_allreduce: 13.49 | step: 18.88

 13%|█▎        | 5240/41250 [12:40:05<86:32:05,  8.65s/it]
                                                          
{'loss': 0.3665, 'grad_norm': 2.196247100830078, 'learning_rate': 3.902074208263362e-05, 'epoch': 1.27}

 13%|█▎        | 5240/41250 [12:40:05<86:32:05,  8.65s/it][2025-04-25 20:37:49,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:37:49,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.36 | bwd_microstep: 5747.08 | bwd_inner_microstep: 5693.52 | bwd_allreduce_microstep: 53.51 | step_microstep: 18.75
[2025-04-25 20:37:49,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.36 | bwd: 5747.10 | bwd_inner: 5693.52 | bwd_allreduce: 53.53 | step: 18.75

 13%|█▎        | 5241/41250 [12:40:14<86:36:39,  8.66s/it]
                                                          
{'loss': 0.2497, 'grad_norm': 2.431851863861084, 'learning_rate': 3.9020256673013765e-05, 'epoch': 1.27}

 13%|█▎        | 5241/41250 [12:40:14<86:36:39,  8.66s/it][2025-04-25 20:37:57,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:37:57,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5696.04 | bwd_inner_microstep: 5683.22 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.85
[2025-04-25 20:37:57,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5696.05 | bwd_inner: 5683.22 | bwd_allreduce: 12.79 | step: 18.85

 13%|█▎        | 5242/41250 [12:40:23<86:30:54,  8.65s/it]
                                                          
{'loss': 0.2712, 'grad_norm': 2.8871781826019287, 'learning_rate': 3.9019771146137764e-05, 'epoch': 1.27}

 13%|█▎        | 5242/41250 [12:40:23<86:30:54,  8.65s/it][2025-04-25 20:38:06,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:38:06,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.15 | bwd_microstep: 5740.26 | bwd_inner_microstep: 5704.39 | bwd_allreduce_microstep: 35.83 | step_microstep: 18.51
[2025-04-25 20:38:06,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.15 | bwd: 5740.27 | bwd_inner: 5704.39 | bwd_allreduce: 35.85 | step: 18.51

 13%|█▎        | 5243/41250 [12:40:31<86:37:05,  8.66s/it]
                                                          
{'loss': 0.2867, 'grad_norm': 3.747800588607788, 'learning_rate': 3.901928550200861e-05, 'epoch': 1.27}

 13%|█▎        | 5243/41250 [12:40:31<86:37:05,  8.66s/it][2025-04-25 20:38:15,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 20:38:15,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.99 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5658.70 | bwd_allreduce_microstep: 50.82 | step_microstep: 19.07
[2025-04-25 20:38:15,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.99 | bwd: 5709.58 | bwd_inner: 5658.70 | bwd_allreduce: 50.84 | step: 19.07

 13%|█▎        | 5244/41250 [12:40:40<86:30:09,  8.65s/it]
                                                          
{'loss': 0.1033, 'grad_norm': 1.5048028230667114, 'learning_rate': 3.901879974062931e-05, 'epoch': 1.27}

 13%|█▎        | 5244/41250 [12:40:40<86:30:09,  8.65s/it][2025-04-25 20:38:23,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 20:38:23,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2904.49 | bwd_microstep: 5794.59 | bwd_inner_microstep: 5781.81 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.60
[2025-04-25 20:38:23,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2904.49 | bwd: 5794.60 | bwd_inner: 5781.81 | bwd_allreduce: 12.74 | step: 18.60

 13%|█▎        | 5245/41250 [12:40:49<86:54:48,  8.69s/it]
                                                          
{'loss': 0.073, 'grad_norm': 1.009722352027893, 'learning_rate': 3.9018313862002846e-05, 'epoch': 1.27}

 13%|█▎        | 5245/41250 [12:40:49<86:54:48,  8.69s/it][2025-04-25 20:38:32,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:38:32,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.66 | bwd_microstep: 5777.73 | bwd_inner_microstep: 5668.58 | bwd_allreduce_microstep: 109.10 | step_microstep: 18.53
[2025-04-25 20:38:32,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.66 | bwd: 5777.74 | bwd_inner: 5668.58 | bwd_allreduce: 109.12 | step: 18.53

 13%|█▎        | 5246/41250 [12:40:58<86:55:08,  8.69s/it]
                                                          
{'loss': 0.0652, 'grad_norm': 0.696983814239502, 'learning_rate': 3.901782786613222e-05, 'epoch': 1.27}

 13%|█▎        | 5246/41250 [12:40:58<86:55:08,  8.69s/it][2025-04-25 20:38:41,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:38:41,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5761.51 | bwd_inner_microstep: 5702.96 | bwd_allreduce_microstep: 58.50 | step_microstep: 18.54
[2025-04-25 20:38:41,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5761.52 | bwd_inner: 5702.96 | bwd_allreduce: 58.52 | step: 18.55

 13%|█▎        | 5247/41250 [12:41:06<86:55:13,  8.69s/it]
                                                          
{'loss': 0.1771, 'grad_norm': 1.554031491279602, 'learning_rate': 3.9017341753020424e-05, 'epoch': 1.27}

 13%|█▎        | 5247/41250 [12:41:06<86:55:13,  8.69s/it][2025-04-25 20:38:50,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.46 | optimizer_step: 1.08
[2025-04-25 20:38:50,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.31 | bwd_microstep: 5725.64 | bwd_inner_microstep: 5711.25 | bwd_allreduce_microstep: 14.31 | step_microstep: 20.62
[2025-04-25 20:38:50,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.31 | bwd: 5725.66 | bwd_inner: 5711.25 | bwd_allreduce: 14.35 | step: 20.62

 13%|█▎        | 5248/41250 [12:41:15<86:51:56,  8.69s/it]
                                                          
{'loss': 0.0703, 'grad_norm': 0.7013179659843445, 'learning_rate': 3.901685552267046e-05, 'epoch': 1.27}

 13%|█▎        | 5248/41250 [12:41:15<86:51:56,  8.69s/it][2025-04-25 20:38:58,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:38:58,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.27 | bwd_microstep: 5776.90 | bwd_inner_microstep: 5716.64 | bwd_allreduce_microstep: 60.22 | step_microstep: 18.62
[2025-04-25 20:38:58,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.27 | bwd: 5776.92 | bwd_inner: 5716.64 | bwd_allreduce: 60.23 | step: 18.62

 13%|█▎        | 5249/41250 [12:41:24<86:57:32,  8.70s/it]
                                                          
{'loss': 0.0844, 'grad_norm': 1.4649126529693604, 'learning_rate': 3.901636917508532e-05, 'epoch': 1.27}

 13%|█▎        | 5249/41250 [12:41:24<86:57:32,  8.70s/it][2025-04-25 20:39:07,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 1.25 | optimizer_step: 0.95
[2025-04-25 20:39:07,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.81 | bwd_microstep: 5745.19 | bwd_inner_microstep: 5704.83 | bwd_allreduce_microstep: 40.31 | step_microstep: 19.48
[2025-04-25 20:39:07,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.81 | bwd: 5745.21 | bwd_inner: 5704.82 | bwd_allreduce: 40.33 | step: 19.48

 13%|█▎        | 5250/41250 [12:41:32<86:56:59,  8.69s/it]
                                                          
{'loss': 0.1519, 'grad_norm': 3.9304394721984863, 'learning_rate': 3.9015882710268e-05, 'epoch': 1.27}

 13%|█▎        | 5250/41250 [12:41:32<86:56:59,  8.69s/it][2025-04-25 20:39:16,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 20:39:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5749.80 | bwd_inner_microstep: 5702.34 | bwd_allreduce_microstep: 47.41 | step_microstep: 18.59
[2025-04-25 20:39:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5749.81 | bwd_inner: 5702.34 | bwd_allreduce: 47.43 | step: 18.59

 13%|█▎        | 5251/41250 [12:41:41<86:53:59,  8.69s/it]
                                                          
{'loss': 0.0466, 'grad_norm': 1.1253445148468018, 'learning_rate': 3.901539612822151e-05, 'epoch': 1.27}

 13%|█▎        | 5251/41250 [12:41:41<86:53:59,  8.69s/it][2025-04-25 20:39:24,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 20:39:24,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.63 | bwd_microstep: 5705.46 | bwd_inner_microstep: 5665.89 | bwd_allreduce_microstep: 39.53 | step_microstep: 19.09
[2025-04-25 20:39:24,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.63 | bwd: 5705.48 | bwd_inner: 5665.89 | bwd_allreduce: 39.55 | step: 19.09

 13%|█▎        | 5252/41250 [12:41:50<86:42:50,  8.67s/it]
                                                          
{'loss': 0.1064, 'grad_norm': 1.393225073814392, 'learning_rate': 3.9014909428948844e-05, 'epoch': 1.27}

 13%|█▎        | 5252/41250 [12:41:50<86:42:50,  8.67s/it][2025-04-25 20:39:33,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 20:39:33,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2942.33 | bwd_microstep: 5887.63 | bwd_inner_microstep: 5874.83 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.90
[2025-04-25 20:39:33,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2942.33 | bwd: 5887.65 | bwd_inner: 5874.83 | bwd_allreduce: 12.78 | step: 18.91

 13%|█▎        | 5253/41250 [12:41:59<87:26:31,  8.74s/it]
                                                          
{'loss': 0.1137, 'grad_norm': 1.557653546333313, 'learning_rate': 3.901442261245299e-05, 'epoch': 1.27}

 13%|█▎        | 5253/41250 [12:41:59<87:26:31,  8.74s/it][2025-04-25 20:39:42,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:39:42,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.89 | bwd_microstep: 5901.32 | bwd_inner_microstep: 5707.34 | bwd_allreduce_microstep: 193.94 | step_microstep: 18.83
[2025-04-25 20:39:42,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.89 | bwd: 5901.33 | bwd_inner: 5707.34 | bwd_allreduce: 193.96 | step: 18.83

 13%|█▎        | 5254/41250 [12:42:07<87:43:15,  8.77s/it]
                                                          
{'loss': 0.2955, 'grad_norm': 2.172912359237671, 'learning_rate': 3.901393567873697e-05, 'epoch': 1.27}

 13%|█▎        | 5254/41250 [12:42:07<87:43:15,  8.77s/it][2025-04-25 20:39:51,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-25 20:39:51,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.32 | bwd_microstep: 5757.47 | bwd_inner_microstep: 5715.21 | bwd_allreduce_microstep: 42.22 | step_microstep: 19.37
[2025-04-25 20:39:51,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.32 | bwd: 5757.49 | bwd_inner: 5715.21 | bwd_allreduce: 42.24 | step: 19.37

 13%|█▎        | 5255/41250 [12:42:16<87:30:39,  8.75s/it]
                                                          
{'loss': 0.0592, 'grad_norm': 0.9939250349998474, 'learning_rate': 3.901344862780378e-05, 'epoch': 1.27}

 13%|█▎        | 5255/41250 [12:42:16<87:30:39,  8.75s/it][2025-04-25 20:39:59,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 20:39:59,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.81 | bwd_microstep: 5808.54 | bwd_inner_microstep: 5666.94 | bwd_allreduce_microstep: 141.55 | step_microstep: 19.16
[2025-04-25 20:39:59,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.81 | bwd: 5808.55 | bwd_inner: 5666.94 | bwd_allreduce: 141.57 | step: 19.16

 13%|█▎        | 5256/41250 [12:42:25<87:25:39,  8.74s/it]
                                                          
{'loss': 0.1937, 'grad_norm': 2.009052276611328, 'learning_rate': 3.901296145965641e-05, 'epoch': 1.27}

 13%|█▎        | 5256/41250 [12:42:25<87:25:39,  8.74s/it][2025-04-25 20:40:08,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:40:08,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.25 | bwd_microstep: 5705.16 | bwd_inner_microstep: 5692.39 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.71
[2025-04-25 20:40:08,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.25 | bwd: 5705.17 | bwd_inner: 5692.39 | bwd_allreduce: 12.74 | step: 18.71

 13%|█▎        | 5257/41250 [12:42:33<87:08:40,  8.72s/it]
                                                          
{'loss': 0.1493, 'grad_norm': 1.3962600231170654, 'learning_rate': 3.9012474174297875e-05, 'epoch': 1.27}

 13%|█▎        | 5257/41250 [12:42:33<87:08:40,  8.72s/it][2025-04-25 20:40:17,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 20:40:17,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5779.43 | bwd_inner_microstep: 5648.69 | bwd_allreduce_microstep: 130.69 | step_microstep: 18.81
[2025-04-25 20:40:17,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5779.44 | bwd_inner: 5648.69 | bwd_allreduce: 130.71 | step: 18.81

 13%|█▎        | 5258/41250 [12:42:42<87:04:53,  8.71s/it]
                                                          
{'loss': 0.0621, 'grad_norm': 0.9779583215713501, 'learning_rate': 3.9011986771731176e-05, 'epoch': 1.27}

 13%|█▎        | 5258/41250 [12:42:42<87:04:53,  8.71s/it][2025-04-25 20:40:26,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:40:26,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.72 | bwd_microstep: 5773.35 | bwd_inner_microstep: 5670.03 | bwd_allreduce_microstep: 103.28 | step_microstep: 18.81
[2025-04-25 20:40:26,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.72 | bwd: 5773.37 | bwd_inner: 5670.03 | bwd_allreduce: 103.29 | step: 18.81

 13%|█▎        | 5259/41250 [12:42:51<87:02:50,  8.71s/it]
                                                          
{'loss': 0.1334, 'grad_norm': 1.3732142448425293, 'learning_rate': 3.9011499251959316e-05, 'epoch': 1.27}

 13%|█▎        | 5259/41250 [12:42:51<87:02:50,  8.71s/it][2025-04-25 20:40:34,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 20:40:34,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5733.80 | bwd_inner_microstep: 5640.85 | bwd_allreduce_microstep: 92.91 | step_microstep: 18.92
[2025-04-25 20:40:34,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5733.82 | bwd_inner: 5640.85 | bwd_allreduce: 92.93 | step: 18.94

 13%|█▎        | 5260/41250 [12:42:59<86:51:34,  8.69s/it]
                                                          
{'loss': 0.0937, 'grad_norm': 1.0533536672592163, 'learning_rate': 3.901101161498531e-05, 'epoch': 1.28}

 13%|█▎        | 5260/41250 [12:42:59<86:51:34,  8.69s/it][2025-04-25 20:40:43,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-25 20:40:43,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.49 | bwd_microstep: 5785.91 | bwd_inner_microstep: 5656.71 | bwd_allreduce_microstep: 129.16 | step_microstep: 18.88
[2025-04-25 20:40:43,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.49 | bwd: 5785.92 | bwd_inner: 5656.71 | bwd_allreduce: 129.17 | step: 18.89

 13%|█▎        | 5261/41250 [12:43:08<86:53:59,  8.69s/it]
                                                          
{'loss': 0.2944, 'grad_norm': 3.5812265872955322, 'learning_rate': 3.901052386081215e-05, 'epoch': 1.28}

 13%|█▎        | 5261/41250 [12:43:08<86:53:59,  8.69s/it][2025-04-25 20:40:52,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:40:52,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.94 | bwd_microstep: 5713.98 | bwd_inner_microstep: 5701.19 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.88
[2025-04-25 20:40:52,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.94 | bwd: 5713.99 | bwd_inner: 5701.19 | bwd_allreduce: 12.76 | step: 18.88

 13%|█▎        | 5262/41250 [12:43:17<86:47:05,  8.68s/it]
                                                          
{'loss': 0.1228, 'grad_norm': 1.163267970085144, 'learning_rate': 3.901003598944285e-05, 'epoch': 1.28}

 13%|█▎        | 5262/41250 [12:43:17<86:47:05,  8.68s/it][2025-04-25 20:41:00,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 20:41:00,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.67 | bwd_microstep: 5735.14 | bwd_inner_microstep: 5722.52 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.74
[2025-04-25 20:41:00,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.67 | bwd: 5735.15 | bwd_inner: 5722.52 | bwd_allreduce: 12.59 | step: 18.74

 13%|█▎        | 5263/41250 [12:43:26<86:45:28,  8.68s/it]
                                                          
{'loss': 0.2593, 'grad_norm': 2.3963623046875, 'learning_rate': 3.9009548000880424e-05, 'epoch': 1.28}

 13%|█▎        | 5263/41250 [12:43:26<86:45:28,  8.68s/it][2025-04-25 20:41:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 20:41:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.18 | bwd_microstep: 5780.04 | bwd_inner_microstep: 5767.43 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.53
[2025-04-25 20:41:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.18 | bwd: 5780.06 | bwd_inner: 5767.43 | bwd_allreduce: 12.59 | step: 18.54

 13%|█▎        | 5264/41250 [12:43:34<86:57:33,  8.70s/it]
                                                          
{'loss': 0.1576, 'grad_norm': 2.840195417404175, 'learning_rate': 3.9009059895127864e-05, 'epoch': 1.28}

 13%|█▎        | 5264/41250 [12:43:34<86:57:33,  8.70s/it][2025-04-25 20:41:18,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-25 20:41:18,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.50 | bwd_microstep: 5685.09 | bwd_inner_microstep: 5657.27 | bwd_allreduce_microstep: 27.78 | step_microstep: 18.32
[2025-04-25 20:41:18,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.50 | bwd: 5685.10 | bwd_inner: 5657.27 | bwd_allreduce: 27.79 | step: 18.32

 13%|█▎        | 5265/41250 [12:43:43<86:38:44,  8.67s/it]
                                                          
{'loss': 0.1488, 'grad_norm': 2.8034257888793945, 'learning_rate': 3.9008571672188195e-05, 'epoch': 1.28}

 13%|█▎        | 5265/41250 [12:43:43<86:38:44,  8.67s/it][2025-04-25 20:41:26,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:41:26,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.76 | bwd_microstep: 5760.80 | bwd_inner_microstep: 5702.24 | bwd_allreduce_microstep: 58.52 | step_microstep: 18.65
[2025-04-25 20:41:26,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.76 | bwd: 5760.82 | bwd_inner: 5702.24 | bwd_allreduce: 58.54 | step: 18.65

 13%|█▎        | 5266/41250 [12:43:52<86:42:53,  8.68s/it]
                                                          
{'loss': 0.2026, 'grad_norm': 2.3955180644989014, 'learning_rate': 3.900808333206442e-05, 'epoch': 1.28}

 13%|█▎        | 5266/41250 [12:43:52<86:42:53,  8.68s/it][2025-04-25 20:41:35,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:41:35,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.33 | bwd_microstep: 5684.11 | bwd_inner_microstep: 5662.82 | bwd_allreduce_microstep: 21.25 | step_microstep: 18.82
[2025-04-25 20:41:35,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.33 | bwd: 5684.13 | bwd_inner: 5662.82 | bwd_allreduce: 21.27 | step: 18.82

 13%|█▎        | 5267/41250 [12:44:00<86:29:23,  8.65s/it]
                                                          
{'loss': 0.1328, 'grad_norm': 2.4166641235351562, 'learning_rate': 3.900759487475955e-05, 'epoch': 1.28}

 13%|█▎        | 5267/41250 [12:44:00<86:29:23,  8.65s/it][2025-04-25 20:41:43,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 20:41:43,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5695.09 | bwd_inner_microstep: 5664.31 | bwd_allreduce_microstep: 30.73 | step_microstep: 19.37
[2025-04-25 20:41:43,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5695.10 | bwd_inner: 5664.31 | bwd_allreduce: 30.75 | step: 19.37

 13%|█▎        | 5268/41250 [12:44:09<86:21:24,  8.64s/it]
                                                          
{'loss': 0.1289, 'grad_norm': 1.1627761125564575, 'learning_rate': 3.900710630027659e-05, 'epoch': 1.28}

 13%|█▎        | 5268/41250 [12:44:09<86:21:24,  8.64s/it][2025-04-25 20:41:52,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 20:41:52,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.08 | bwd_microstep: 5772.63 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 124.14 | step_microstep: 18.99
[2025-04-25 20:41:52,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.09 | bwd: 5772.64 | bwd_inner: 5648.44 | bwd_allreduce: 124.16 | step: 18.99

 13%|█▎        | 5269/41250 [12:44:17<86:28:15,  8.65s/it]
                                                          
{'loss': 0.187, 'grad_norm': 2.2360453605651855, 'learning_rate': 3.900661760861857e-05, 'epoch': 1.28}

 13%|█▎        | 5269/41250 [12:44:17<86:28:15,  8.65s/it][2025-04-25 20:42:01,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:42:01,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.02 | bwd_microstep: 5754.39 | bwd_inner_microstep: 5645.38 | bwd_allreduce_microstep: 108.96 | step_microstep: 18.72
[2025-04-25 20:42:01,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.02 | bwd: 5754.40 | bwd_inner: 5645.38 | bwd_allreduce: 108.97 | step: 18.72

 13%|█▎        | 5270/41250 [12:44:26<86:30:16,  8.66s/it]
                                                          
{'loss': 0.0239, 'grad_norm': 0.6363493204116821, 'learning_rate': 3.900612879978848e-05, 'epoch': 1.28}

 13%|█▎        | 5270/41250 [12:44:26<86:30:16,  8.66s/it][2025-04-25 20:42:09,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:42:09,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.41 | bwd_microstep: 5694.38 | bwd_inner_microstep: 5636.98 | bwd_allreduce_microstep: 57.35 | step_microstep: 18.88
[2025-04-25 20:42:09,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.41 | bwd: 5694.39 | bwd_inner: 5636.98 | bwd_allreduce: 57.37 | step: 18.88

 13%|█▎        | 5271/41250 [12:44:35<86:20:37,  8.64s/it]
                                                          
{'loss': 0.2919, 'grad_norm': 2.094874620437622, 'learning_rate': 3.900563987378935e-05, 'epoch': 1.28}

 13%|█▎        | 5271/41250 [12:44:35<86:20:37,  8.64s/it][2025-04-25 20:42:18,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:42:18,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.12 | bwd_microstep: 5771.97 | bwd_inner_microstep: 5759.06 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.86
[2025-04-25 20:42:18,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.12 | bwd: 5771.98 | bwd_inner: 5759.06 | bwd_allreduce: 12.88 | step: 18.86

 13%|█▎        | 5272/41250 [12:44:43<86:38:22,  8.67s/it]
                                                          
{'loss': 0.1489, 'grad_norm': 3.87359881401062, 'learning_rate': 3.9005150830624185e-05, 'epoch': 1.28}

 13%|█▎        | 5272/41250 [12:44:43<86:38:22,  8.67s/it][2025-04-25 20:42:27,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-25 20:42:27,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.19 | bwd_microstep: 5710.95 | bwd_inner_microstep: 5642.93 | bwd_allreduce_microstep: 67.98 | step_microstep: 19.61
[2025-04-25 20:42:27,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.19 | bwd: 5710.97 | bwd_inner: 5642.93 | bwd_allreduce: 67.99 | step: 19.62

 13%|█▎        | 5273/41250 [12:44:52<86:28:34,  8.65s/it]
                                                          
{'loss': 0.1616, 'grad_norm': 2.7788050174713135, 'learning_rate': 3.9004661670296e-05, 'epoch': 1.28}

 13%|█▎        | 5273/41250 [12:44:52<86:28:34,  8.65s/it][2025-04-25 20:42:35,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 20:42:35,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.03 | bwd_microstep: 5772.09 | bwd_inner_microstep: 5759.06 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.96
[2025-04-25 20:42:35,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.03 | bwd: 5772.10 | bwd_inner: 5759.06 | bwd_allreduce: 13.00 | step: 18.96

 13%|█▎        | 5274/41250 [12:45:01<86:45:08,  8.68s/it]
                                                          
{'loss': 0.0852, 'grad_norm': 0.916693389415741, 'learning_rate': 3.900417239280782e-05, 'epoch': 1.28}

 13%|█▎        | 5274/41250 [12:45:01<86:45:08,  8.68s/it][2025-04-25 20:42:44,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.95
[2025-04-25 20:42:44,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.77 | bwd_microstep: 5724.93 | bwd_inner_microstep: 5696.83 | bwd_allreduce_microstep: 28.04 | step_microstep: 18.30
[2025-04-25 20:42:44,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.77 | bwd: 5724.94 | bwd_inner: 5696.83 | bwd_allreduce: 28.06 | step: 18.31

 13%|█▎        | 5275/41250 [12:45:09<86:41:26,  8.68s/it]
                                                          
{'loss': 0.103, 'grad_norm': 1.159017562866211, 'learning_rate': 3.900368299816266e-05, 'epoch': 1.28}

 13%|█▎        | 5275/41250 [12:45:09<86:41:26,  8.68s/it][2025-04-25 20:42:53,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 20:42:53,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.05 | bwd_microstep: 5748.17 | bwd_inner_microstep: 5653.36 | bwd_allreduce_microstep: 94.77 | step_microstep: 19.38
[2025-04-25 20:42:53,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.05 | bwd: 5748.19 | bwd_inner: 5653.36 | bwd_allreduce: 94.79 | step: 19.38

 13%|█▎        | 5276/41250 [12:45:18<86:38:47,  8.67s/it]
                                                          
{'loss': 0.2825, 'grad_norm': 2.1857359409332275, 'learning_rate': 3.900319348636352e-05, 'epoch': 1.28}

 13%|█▎        | 5276/41250 [12:45:18<86:38:47,  8.67s/it][2025-04-25 20:43:01,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 20:43:01,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.31 | bwd_microstep: 5696.59 | bwd_inner_microstep: 5683.96 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.51
[2025-04-25 20:43:01,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.31 | bwd: 5696.60 | bwd_inner: 5683.96 | bwd_allreduce: 12.60 | step: 18.52

 13%|█▎        | 5277/41250 [12:45:27<86:31:14,  8.66s/it]
                                                          
{'loss': 0.0779, 'grad_norm': 1.9813487529754639, 'learning_rate': 3.900270385741344e-05, 'epoch': 1.28}

 13%|█▎        | 5277/41250 [12:45:27<86:31:14,  8.66s/it][2025-04-25 20:43:10,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:43:10,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.06 | bwd_microstep: 5870.98 | bwd_inner_microstep: 5858.14 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.92
[2025-04-25 20:43:10,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.06 | bwd: 5870.99 | bwd_inner: 5858.14 | bwd_allreduce: 12.81 | step: 18.92

 13%|█▎        | 5278/41250 [12:45:36<87:10:39,  8.72s/it]
                                                          
{'loss': 0.0898, 'grad_norm': 2.324204444885254, 'learning_rate': 3.900221411131542e-05, 'epoch': 1.28}

 13%|█▎        | 5278/41250 [12:45:36<87:10:39,  8.72s/it][2025-04-25 20:43:19,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:43:19,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.45 | bwd_microstep: 5688.28 | bwd_inner_microstep: 5653.91 | bwd_allreduce_microstep: 34.32 | step_microstep: 18.72
[2025-04-25 20:43:19,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.45 | bwd: 5688.30 | bwd_inner: 5653.91 | bwd_allreduce: 34.34 | step: 18.73

 13%|█▎        | 5279/41250 [12:45:44<86:47:33,  8.69s/it]
                                                          
{'loss': 0.1231, 'grad_norm': 2.2180676460266113, 'learning_rate': 3.90017242480725e-05, 'epoch': 1.28}

 13%|█▎        | 5279/41250 [12:45:44<86:47:33,  8.69s/it][2025-04-25 20:43:27,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.21
[2025-04-25 20:43:27,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.69 | bwd_microstep: 5672.84 | bwd_inner_microstep: 5656.18 | bwd_allreduce_microstep: 16.62 | step_microstep: 19.29
[2025-04-25 20:43:27,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.69 | bwd: 5672.85 | bwd_inner: 5656.18 | bwd_allreduce: 16.63 | step: 19.29

 13%|█▎        | 5280/41250 [12:45:53<86:30:33,  8.66s/it]
                                                          
{'loss': 0.1447, 'grad_norm': 1.8251460790634155, 'learning_rate': 3.900123426768767e-05, 'epoch': 1.28}

 13%|█▎        | 5280/41250 [12:45:53<86:30:33,  8.66s/it][2025-04-25 20:43:36,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 20:43:36,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5760.45 | bwd_inner_microstep: 5644.59 | bwd_allreduce_microstep: 115.82 | step_microstep: 18.98
[2025-04-25 20:43:36,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.79 | bwd: 5760.47 | bwd_inner: 5644.59 | bwd_allreduce: 115.84 | step: 18.97

 13%|█▎        | 5281/41250 [12:46:01<86:32:40,  8.66s/it]
                                                          
{'loss': 0.1121, 'grad_norm': 1.335759162902832, 'learning_rate': 3.900074417016398e-05, 'epoch': 1.28}

 13%|█▎        | 5281/41250 [12:46:01<86:32:40,  8.66s/it][2025-04-25 20:43:45,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:43:45,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.66 | bwd_microstep: 5699.07 | bwd_inner_microstep: 5686.30 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.66
[2025-04-25 20:43:45,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.66 | bwd: 5699.08 | bwd_inner: 5686.30 | bwd_allreduce: 12.75 | step: 18.67

 13%|█▎        | 5282/41250 [12:46:10<86:25:45,  8.65s/it]
                                                          
{'loss': 0.3314, 'grad_norm': 3.1146724224090576, 'learning_rate': 3.900025395550444e-05, 'epoch': 1.28}

 13%|█▎        | 5282/41250 [12:46:10<86:25:45,  8.65s/it][2025-04-25 20:43:53,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:43:53,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.55 | bwd_microstep: 5709.27 | bwd_inner_microstep: 5696.72 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.50
[2025-04-25 20:43:53,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5709.28 | bwd_inner: 5696.72 | bwd_allreduce: 12.52 | step: 18.51

 13%|█▎        | 5283/41250 [12:46:19<86:23:47,  8.65s/it]
                                                          
{'loss': 0.3809, 'grad_norm': 2.8897244930267334, 'learning_rate': 3.8999763623712065e-05, 'epoch': 1.28}

 13%|█▎        | 5283/41250 [12:46:19<86:23:47,  8.65s/it][2025-04-25 20:44:02,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:44:02,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.27 | bwd_microstep: 5734.32 | bwd_inner_microstep: 5645.61 | bwd_allreduce_microstep: 88.67 | step_microstep: 18.27
[2025-04-25 20:44:02,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.27 | bwd: 5734.33 | bwd_inner: 5645.61 | bwd_allreduce: 88.68 | step: 18.27

 13%|█▎        | 5284/41250 [12:46:27<86:23:22,  8.65s/it]
                                                          
{'loss': 0.0388, 'grad_norm': 0.653759777545929, 'learning_rate': 3.899927317478989e-05, 'epoch': 1.28}

 13%|█▎        | 5284/41250 [12:46:27<86:23:22,  8.65s/it][2025-04-25 20:44:11,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 20:44:11,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.29 | bwd_microstep: 5848.08 | bwd_inner_microstep: 5692.36 | bwd_allreduce_microstep: 155.66 | step_microstep: 19.95
[2025-04-25 20:44:11,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.29 | bwd: 5848.10 | bwd_inner: 5692.36 | bwd_allreduce: 155.69 | step: 19.95

 13%|█▎        | 5285/41250 [12:46:36<86:46:58,  8.69s/it]
                                                          
{'loss': 0.1058, 'grad_norm': 1.5419501066207886, 'learning_rate': 3.899878260874092e-05, 'epoch': 1.28}

 13%|█▎        | 5285/41250 [12:46:36<86:46:58,  8.69s/it][2025-04-25 20:44:19,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.19 | optimizer_step: 0.89
[2025-04-25 20:44:19,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.57 | bwd_microstep: 5697.17 | bwd_inner_microstep: 5643.93 | bwd_allreduce_microstep: 53.19 | step_microstep: 18.86
[2025-04-25 20:44:19,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.57 | bwd: 5697.18 | bwd_inner: 5643.93 | bwd_allreduce: 53.21 | step: 18.86

 13%|█▎        | 5286/41250 [12:46:45<86:32:23,  8.66s/it]
                                                          
{'loss': 0.1653, 'grad_norm': 2.969127655029297, 'learning_rate': 3.8998291925568213e-05, 'epoch': 1.28}

 13%|█▎        | 5286/41250 [12:46:45<86:32:23,  8.66s/it][2025-04-25 20:44:28,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 20:44:28,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5726.97 | bwd_inner_microstep: 5692.88 | bwd_allreduce_microstep: 34.04 | step_microstep: 18.55
[2025-04-25 20:44:28,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5726.99 | bwd_inner: 5692.88 | bwd_allreduce: 34.06 | step: 18.56

 13%|█▎        | 5287/41250 [12:46:53<86:31:10,  8.66s/it]
                                                          
{'loss': 0.257, 'grad_norm': 3.8821935653686523, 'learning_rate': 3.8997801125274754e-05, 'epoch': 1.28}

 13%|█▎        | 5287/41250 [12:46:53<86:31:10,  8.66s/it][2025-04-25 20:44:37,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 20:44:37,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5769.08 | bwd_inner_microstep: 5665.37 | bwd_allreduce_microstep: 103.67 | step_microstep: 18.58
[2025-04-25 20:44:37,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5769.10 | bwd_inner: 5665.37 | bwd_allreduce: 103.69 | step: 18.59

 13%|█▎        | 5288/41250 [12:47:02<86:33:59,  8.67s/it]
                                                          
{'loss': 0.0378, 'grad_norm': 1.343819499015808, 'learning_rate': 3.8997310207863594e-05, 'epoch': 1.28}

 13%|█▎        | 5288/41250 [12:47:02<86:33:59,  8.67s/it][2025-04-25 20:44:45,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 20:44:45,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.59 | bwd_microstep: 5691.82 | bwd_inner_microstep: 5678.91 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.74
[2025-04-25 20:44:45,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.59 | bwd: 5691.83 | bwd_inner: 5678.91 | bwd_allreduce: 12.88 | step: 18.74

 13%|█▎        | 5289/41250 [12:47:11<86:26:24,  8.65s/it]
                                                          
{'loss': 0.1978, 'grad_norm': 1.6305135488510132, 'learning_rate': 3.899681917333776e-05, 'epoch': 1.28}

 13%|█▎        | 5289/41250 [12:47:11<86:26:24,  8.65s/it][2025-04-25 20:44:54,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 20:44:54,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.04 | bwd_microstep: 5794.88 | bwd_inner_microstep: 5782.15 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.60
[2025-04-25 20:44:54,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.04 | bwd: 5794.89 | bwd_inner: 5782.15 | bwd_allreduce: 12.70 | step: 18.60

 13%|█▎        | 5290/41250 [12:47:20<86:46:42,  8.69s/it]
                                                          
{'loss': 0.0753, 'grad_norm': 0.7305170297622681, 'learning_rate': 3.899632802170027e-05, 'epoch': 1.28}

 13%|█▎        | 5290/41250 [12:47:20<86:46:42,  8.69s/it][2025-04-25 20:45:03,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-25 20:45:03,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.57 | bwd_microstep: 5759.53 | bwd_inner_microstep: 5688.31 | bwd_allreduce_microstep: 71.16 | step_microstep: 20.09
[2025-04-25 20:45:03,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.57 | bwd: 5759.54 | bwd_inner: 5688.31 | bwd_allreduce: 71.19 | step: 20.09

 13%|█▎        | 5291/41250 [12:47:28<86:48:04,  8.69s/it]
                                                          
{'loss': 0.2865, 'grad_norm': 3.5986685752868652, 'learning_rate': 3.899583675295415e-05, 'epoch': 1.28}

 13%|█▎        | 5291/41250 [12:47:28<86:48:04,  8.69s/it][2025-04-25 20:45:12,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-25 20:45:12,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.68 | bwd_microstep: 5766.93 | bwd_inner_microstep: 5659.85 | bwd_allreduce_microstep: 107.03 | step_microstep: 19.63
[2025-04-25 20:45:12,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.68 | bwd: 5766.94 | bwd_inner: 5659.85 | bwd_allreduce: 107.06 | step: 19.63

 13%|█▎        | 5292/41250 [12:47:37<86:48:42,  8.69s/it]
                                                          
{'loss': 0.2434, 'grad_norm': 3.064574956893921, 'learning_rate': 3.899534536710243e-05, 'epoch': 1.28}

 13%|█▎        | 5292/41250 [12:47:37<86:48:42,  8.69s/it][2025-04-25 20:45:20,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 20:45:20,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.67 | bwd_microstep: 5771.96 | bwd_inner_microstep: 5656.73 | bwd_allreduce_microstep: 115.19 | step_microstep: 18.66
[2025-04-25 20:45:20,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.67 | bwd: 5771.97 | bwd_inner: 5656.73 | bwd_allreduce: 115.20 | step: 18.66

 13%|█▎        | 5293/41250 [12:47:46<86:46:36,  8.69s/it]
                                                          
{'loss': 0.0744, 'grad_norm': 0.7591884136199951, 'learning_rate': 3.899485386414815e-05, 'epoch': 1.28}

 13%|█▎        | 5293/41250 [12:47:46<86:46:36,  8.69s/it][2025-04-25 20:45:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 20:45:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5805.01 | bwd_inner_microstep: 5660.76 | bwd_allreduce_microstep: 144.20 | step_microstep: 18.45
[2025-04-25 20:45:29,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5805.02 | bwd_inner: 5660.76 | bwd_allreduce: 144.22 | step: 18.45

 13%|█▎        | 5294/41250 [12:47:54<86:51:50,  8.70s/it]
                                                          
{'loss': 0.1101, 'grad_norm': 2.268676519393921, 'learning_rate': 3.8994362244094326e-05, 'epoch': 1.28}

 13%|█▎        | 5294/41250 [12:47:54<86:51:50,  8.70s/it][2025-04-25 20:45:38,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.23 | optimizer_step: 1.11
[2025-04-25 20:45:38,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.19 | bwd_microstep: 5754.59 | bwd_inner_microstep: 5718.40 | bwd_allreduce_microstep: 36.13 | step_microstep: 19.80
[2025-04-25 20:45:38,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.19 | bwd: 5754.61 | bwd_inner: 5718.40 | bwd_allreduce: 36.16 | step: 19.80

 13%|█▎        | 5295/41250 [12:48:03<86:51:47,  8.70s/it]
                                                          
{'loss': 0.0992, 'grad_norm': 1.4767423868179321, 'learning_rate': 3.8993870506944e-05, 'epoch': 1.28}

 13%|█▎        | 5295/41250 [12:48:03<86:51:47,  8.70s/it][2025-04-25 20:45:46,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 20:45:46,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5709.70 | bwd_inner_microstep: 5696.48 | bwd_allreduce_microstep: 13.17 | step_microstep: 19.30
[2025-04-25 20:45:46,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5709.71 | bwd_inner: 5696.48 | bwd_allreduce: 13.19 | step: 19.30

 13%|█▎        | 5296/41250 [12:48:12<86:41:18,  8.68s/it]
                                                          
{'loss': 0.0464, 'grad_norm': 1.1050931215286255, 'learning_rate': 3.8993378652700196e-05, 'epoch': 1.28}

 13%|█▎        | 5296/41250 [12:48:12<86:41:18,  8.68s/it][2025-04-25 20:45:55,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:45:55,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.13 | bwd_microstep: 5776.65 | bwd_inner_microstep: 5707.89 | bwd_allreduce_microstep: 68.72 | step_microstep: 18.82
[2025-04-25 20:45:55,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.13 | bwd: 5776.66 | bwd_inner: 5707.89 | bwd_allreduce: 68.73 | step: 18.82

 13%|█▎        | 5297/41250 [12:48:20<86:47:56,  8.69s/it]
                                                          
{'loss': 0.1397, 'grad_norm': 2.3522632122039795, 'learning_rate': 3.8992886681365946e-05, 'epoch': 1.28}

 13%|█▎        | 5297/41250 [12:48:20<86:47:56,  8.69s/it][2025-04-25 20:46:04,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 20:46:04,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.15 | bwd_microstep: 5891.04 | bwd_inner_microstep: 5877.76 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.01
[2025-04-25 20:46:04,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.15 | bwd: 5891.05 | bwd_inner: 5877.76 | bwd_allreduce: 13.25 | step: 19.02

 13%|█▎        | 5298/41250 [12:48:29<87:27:23,  8.76s/it]
                                                          
{'loss': 0.1554, 'grad_norm': 1.4029279947280884, 'learning_rate': 3.8992394592944286e-05, 'epoch': 1.28}

 13%|█▎        | 5298/41250 [12:48:29<87:27:23,  8.76s/it][2025-04-25 20:46:13,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 20:46:13,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.42 | bwd_microstep: 5769.35 | bwd_inner_microstep: 5662.67 | bwd_allreduce_microstep: 106.63 | step_microstep: 18.69
[2025-04-25 20:46:13,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.42 | bwd: 5769.36 | bwd_inner: 5662.67 | bwd_allreduce: 106.65 | step: 18.69

 13%|█▎        | 5299/41250 [12:48:38<87:14:40,  8.74s/it]
                                                          
{'loss': 0.0444, 'grad_norm': 0.5884720087051392, 'learning_rate': 3.899190238743825e-05, 'epoch': 1.28}

 13%|█▎        | 5299/41250 [12:48:38<87:14:40,  8.74s/it][2025-04-25 20:46:21,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:46:21,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.50 | bwd_microstep: 5706.19 | bwd_inner_microstep: 5693.33 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.64
[2025-04-25 20:46:21,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.50 | bwd: 5706.21 | bwd_inner: 5693.33 | bwd_allreduce: 12.83 | step: 18.65

 13%|█▎        | 5300/41250 [12:48:47<86:57:36,  8.71s/it]
                                                          
{'loss': 0.2198, 'grad_norm': 1.9400051832199097, 'learning_rate': 3.899141006485087e-05, 'epoch': 1.28}

 13%|█▎        | 5300/41250 [12:48:47<86:57:36,  8.71s/it][2025-04-25 20:46:30,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.23 | optimizer_step: 1.00
[2025-04-25 20:46:30,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5799.64 | bwd_inner_microstep: 5656.99 | bwd_allreduce_microstep: 142.60 | step_microstep: 19.39
[2025-04-25 20:46:30,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5799.66 | bwd_inner: 5656.99 | bwd_allreduce: 142.62 | step: 19.39

 13%|█▎        | 5301/41250 [12:48:55<86:58:46,  8.71s/it]
                                                          
{'loss': 0.2973, 'grad_norm': 2.47452712059021, 'learning_rate': 3.8990917625185186e-05, 'epoch': 1.29}

 13%|█▎        | 5301/41250 [12:48:55<86:58:46,  8.71s/it][2025-04-25 20:46:39,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 20:46:39,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.26 | bwd_microstep: 5728.86 | bwd_inner_microstep: 5716.07 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.80
[2025-04-25 20:46:39,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.26 | bwd: 5728.87 | bwd_inner: 5716.07 | bwd_allreduce: 12.76 | step: 18.80

 13%|█▎        | 5302/41250 [12:49:04<86:50:49,  8.70s/it]
                                                          
{'loss': 0.0297, 'grad_norm': 0.7823610901832581, 'learning_rate': 3.8990425068444227e-05, 'epoch': 1.29}

 13%|█▎        | 5302/41250 [12:49:04<86:50:49,  8.70s/it][2025-04-25 20:46:47,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 20:46:47,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.27 | bwd_microstep: 5779.79 | bwd_inner_microstep: 5715.31 | bwd_allreduce_microstep: 64.42 | step_microstep: 18.81
[2025-04-25 20:46:47,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.27 | bwd: 5779.80 | bwd_inner: 5715.31 | bwd_allreduce: 64.45 | step: 18.82

 13%|█▎        | 5303/41250 [12:49:13<86:54:47,  8.70s/it]
                                                          
{'loss': 0.1164, 'grad_norm': 2.0880062580108643, 'learning_rate': 3.898993239463103e-05, 'epoch': 1.29}

 13%|█▎        | 5303/41250 [12:49:13<86:54:47,  8.70s/it][2025-04-25 20:46:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:46:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.63 | bwd_microstep: 5804.69 | bwd_inner_microstep: 5792.21 | bwd_allreduce_microstep: 12.44 | step_microstep: 18.51
[2025-04-25 20:46:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.63 | bwd: 5804.71 | bwd_inner: 5792.21 | bwd_allreduce: 12.46 | step: 18.51

 13%|█▎        | 5304/41250 [12:49:21<87:07:50,  8.73s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 1.278975009918213, 'learning_rate': 3.898943960374864e-05, 'epoch': 1.29}

 13%|█▎        | 5304/41250 [12:49:21<87:07:50,  8.73s/it][2025-04-25 20:47:05,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:47:05,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.45 | bwd_microstep: 5734.30 | bwd_inner_microstep: 5654.23 | bwd_allreduce_microstep: 80.03 | step_microstep: 18.77
[2025-04-25 20:47:05,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.45 | bwd: 5734.31 | bwd_inner: 5654.23 | bwd_allreduce: 80.05 | step: 18.77

 13%|█▎        | 5305/41250 [12:49:30<86:52:52,  8.70s/it]
                                                          
{'loss': 0.0825, 'grad_norm': 2.7062830924987793, 'learning_rate': 3.898894669580009e-05, 'epoch': 1.29}

 13%|█▎        | 5305/41250 [12:49:30<86:52:52,  8.70s/it][2025-04-25 20:47:14,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:47:14,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.06 | bwd_microstep: 5879.50 | bwd_inner_microstep: 5704.00 | bwd_allreduce_microstep: 175.46 | step_microstep: 18.38
[2025-04-25 20:47:14,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.07 | bwd: 5879.51 | bwd_inner: 5704.00 | bwd_allreduce: 175.47 | step: 18.38

 13%|█▎        | 5306/41250 [12:49:39<87:14:44,  8.74s/it]
                                                          
{'loss': 0.0688, 'grad_norm': 1.2679163217544556, 'learning_rate': 3.8988453670788416e-05, 'epoch': 1.29}

 13%|█▎        | 5306/41250 [12:49:39<87:14:44,  8.74s/it][2025-04-25 20:47:22,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:47:22,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.09 | bwd_microstep: 5711.60 | bwd_inner_microstep: 5663.68 | bwd_allreduce_microstep: 47.88 | step_microstep: 18.43
[2025-04-25 20:47:22,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.09 | bwd: 5711.61 | bwd_inner: 5663.68 | bwd_allreduce: 47.89 | step: 18.43

 13%|█▎        | 5307/41250 [12:49:48<86:53:44,  8.70s/it]
                                                          
{'loss': 0.2662, 'grad_norm': 3.1570379734039307, 'learning_rate': 3.898796052871666e-05, 'epoch': 1.29}

 13%|█▎        | 5307/41250 [12:49:48<86:53:44,  8.70s/it][2025-04-25 20:47:31,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 20:47:31,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5708.58 | bwd_inner_microstep: 5668.78 | bwd_allreduce_microstep: 39.75 | step_microstep: 18.84
[2025-04-25 20:47:31,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5708.59 | bwd_inner: 5668.78 | bwd_allreduce: 39.77 | step: 18.84

 13%|█▎        | 5308/41250 [12:49:56<86:42:52,  8.69s/it]
                                                          
{'loss': 0.1416, 'grad_norm': 3.7084598541259766, 'learning_rate': 3.898746726958786e-05, 'epoch': 1.29}

 13%|█▎        | 5308/41250 [12:49:56<86:42:52,  8.69s/it][2025-04-25 20:47:40,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:47:40,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.68 | bwd_microstep: 5731.55 | bwd_inner_microstep: 5718.52 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.89
[2025-04-25 20:47:40,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.68 | bwd: 5731.57 | bwd_inner: 5718.52 | bwd_allreduce: 13.01 | step: 18.90

 13%|█▎        | 5309/41250 [12:50:05<86:40:42,  8.68s/it]
                                                          
{'loss': 0.5431, 'grad_norm': 6.383234024047852, 'learning_rate': 3.8986973893405064e-05, 'epoch': 1.29}

 13%|█▎        | 5309/41250 [12:50:05<86:40:42,  8.68s/it][2025-04-25 20:47:48,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:47:48,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.10 | bwd_microstep: 5754.93 | bwd_inner_microstep: 5661.83 | bwd_allreduce_microstep: 93.06 | step_microstep: 18.33
[2025-04-25 20:47:48,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.10 | bwd: 5754.95 | bwd_inner: 5661.83 | bwd_allreduce: 93.08 | step: 18.33

 13%|█▎        | 5310/41250 [12:50:14<86:37:47,  8.68s/it]
                                                          
{'loss': 0.0755, 'grad_norm': 1.9767638444900513, 'learning_rate': 3.8986480400171306e-05, 'epoch': 1.29}

 13%|█▎        | 5310/41250 [12:50:14<86:37:47,  8.68s/it][2025-04-25 20:47:57,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:47:57,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.77 | bwd_microstep: 5722.02 | bwd_inner_microstep: 5709.24 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.68
[2025-04-25 20:47:57,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.77 | bwd: 5722.03 | bwd_inner: 5709.24 | bwd_allreduce: 12.75 | step: 18.68

 13%|█▎        | 5311/41250 [12:50:22<86:36:48,  8.68s/it]
                                                          
{'loss': 0.0748, 'grad_norm': 1.579447627067566, 'learning_rate': 3.898598678988963e-05, 'epoch': 1.29}

 13%|█▎        | 5311/41250 [12:50:22<86:36:48,  8.68s/it][2025-04-25 20:48:06,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:48:06,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5695.43 | bwd_inner_microstep: 5660.35 | bwd_allreduce_microstep: 35.03 | step_microstep: 18.79
[2025-04-25 20:48:06,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5695.45 | bwd_inner: 5660.35 | bwd_allreduce: 35.05 | step: 18.79

 13%|█▎        | 5312/41250 [12:50:31<86:24:10,  8.66s/it]
                                                          
{'loss': 0.2175, 'grad_norm': 3.4025449752807617, 'learning_rate': 3.8985493062563075e-05, 'epoch': 1.29}

 13%|█▎        | 5312/41250 [12:50:31<86:24:10,  8.66s/it][2025-04-25 20:48:14,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.17 | optimizer_step: 1.12
[2025-04-25 20:48:14,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5755.81 | bwd_inner_microstep: 5684.37 | bwd_allreduce_microstep: 71.38 | step_microstep: 19.45
[2025-04-25 20:48:14,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5755.83 | bwd_inner: 5684.37 | bwd_allreduce: 71.41 | step: 19.45

 13%|█▎        | 5313/41250 [12:50:40<86:31:31,  8.67s/it]
                                                          
{'loss': 0.1003, 'grad_norm': 1.4936007261276245, 'learning_rate': 3.89849992181947e-05, 'epoch': 1.29}

 13%|█▎        | 5313/41250 [12:50:40<86:31:31,  8.67s/it][2025-04-25 20:48:23,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:48:23,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.65 | bwd_microstep: 5763.82 | bwd_inner_microstep: 5692.72 | bwd_allreduce_microstep: 71.06 | step_microstep: 18.84
[2025-04-25 20:48:23,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.65 | bwd: 5763.83 | bwd_inner: 5692.72 | bwd_allreduce: 71.07 | step: 18.85

 13%|█▎        | 5314/41250 [12:50:48<86:35:00,  8.67s/it]
                                                          
{'loss': 0.2692, 'grad_norm': 2.176755428314209, 'learning_rate': 3.898450525678754e-05, 'epoch': 1.29}

 13%|█▎        | 5314/41250 [12:50:48<86:35:00,  8.67s/it][2025-04-25 20:48:32,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-25 20:48:32,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5712.67 | bwd_inner_microstep: 5658.09 | bwd_allreduce_microstep: 54.54 | step_microstep: 19.01
[2025-04-25 20:48:32,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5712.69 | bwd_inner: 5658.09 | bwd_allreduce: 54.55 | step: 19.01

 13%|█▎        | 5315/41250 [12:50:57<86:25:53,  8.66s/it]
                                                          
{'loss': 0.0582, 'grad_norm': 1.550394892692566, 'learning_rate': 3.8984011178344625e-05, 'epoch': 1.29}

 13%|█▎        | 5315/41250 [12:50:57<86:25:53,  8.66s/it][2025-04-25 20:48:40,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:48:40,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.80 | bwd_microstep: 5741.26 | bwd_inner_microstep: 5681.19 | bwd_allreduce_microstep: 60.03 | step_microstep: 18.78
[2025-04-25 20:48:40,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.80 | bwd: 5741.28 | bwd_inner: 5681.19 | bwd_allreduce: 60.05 | step: 18.78

 13%|█▎        | 5316/41250 [12:51:05<86:26:49,  8.66s/it]
                                                          
{'loss': 0.1575, 'grad_norm': 1.5664092302322388, 'learning_rate': 3.898351698286902e-05, 'epoch': 1.29}

 13%|█▎        | 5316/41250 [12:51:06<86:26:49,  8.66s/it][2025-04-25 20:48:49,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:48:49,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.50 | bwd_microstep: 5712.08 | bwd_inner_microstep: 5699.10 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.89
[2025-04-25 20:48:49,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.50 | bwd: 5712.10 | bwd_inner: 5699.10 | bwd_allreduce: 12.95 | step: 18.89

 13%|█▎        | 5317/41250 [12:51:14<86:23:35,  8.66s/it]
                                                          
{'loss': 0.0541, 'grad_norm': 1.3806331157684326, 'learning_rate': 3.898302267036378e-05, 'epoch': 1.29}

 13%|█▎        | 5317/41250 [12:51:14<86:23:35,  8.66s/it][2025-04-25 20:48:57,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:48:57,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.56 | bwd_microstep: 5708.35 | bwd_inner_microstep: 5695.41 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.84
[2025-04-25 20:48:57,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.56 | bwd: 5708.36 | bwd_inner: 5695.41 | bwd_allreduce: 12.91 | step: 18.84

 13%|█▎        | 5318/41250 [12:51:23<86:20:13,  8.65s/it]
                                                          
{'loss': 0.1044, 'grad_norm': 2.6360912322998047, 'learning_rate': 3.898252824083192e-05, 'epoch': 1.29}

 13%|█▎        | 5318/41250 [12:51:23<86:20:13,  8.65s/it][2025-04-25 20:49:06,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:49:06,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.45 | bwd_microstep: 5767.45 | bwd_inner_microstep: 5660.65 | bwd_allreduce_microstep: 106.75 | step_microstep: 18.46
[2025-04-25 20:49:06,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.45 | bwd: 5767.47 | bwd_inner: 5660.65 | bwd_allreduce: 106.77 | step: 18.46

 13%|█▎        | 5319/41250 [12:51:31<86:25:42,  8.66s/it]
                                                          
{'loss': 0.0727, 'grad_norm': 1.7556785345077515, 'learning_rate': 3.898203369427652e-05, 'epoch': 1.29}

 13%|█▎        | 5319/41250 [12:51:31<86:25:42,  8.66s/it][2025-04-25 20:49:15,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:49:15,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.06 | bwd_microstep: 5723.39 | bwd_inner_microstep: 5710.60 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.80
[2025-04-25 20:49:15,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.06 | bwd: 5723.40 | bwd_inner: 5710.60 | bwd_allreduce: 12.76 | step: 18.80

 13%|█▎        | 5320/41250 [12:51:40<86:25:10,  8.66s/it]
                                                          
{'loss': 0.3169, 'grad_norm': 3.668302059173584, 'learning_rate': 3.8981539030700614e-05, 'epoch': 1.29}

 13%|█▎        | 5320/41250 [12:51:40<86:25:10,  8.66s/it][2025-04-25 20:49:23,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 20:49:23,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5763.72 | bwd_inner_microstep: 5642.06 | bwd_allreduce_microstep: 121.60 | step_microstep: 18.83
[2025-04-25 20:49:23,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5763.74 | bwd_inner: 5642.06 | bwd_allreduce: 121.62 | step: 18.83

 13%|█▎        | 5321/41250 [12:51:49<86:28:21,  8.66s/it]
                                                          
{'loss': 0.081, 'grad_norm': 1.7508270740509033, 'learning_rate': 3.8981044250107246e-05, 'epoch': 1.29}

 13%|█▎        | 5321/41250 [12:51:49<86:28:21,  8.66s/it][2025-04-25 20:49:32,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:49:32,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5720.21 | bwd_inner_microstep: 5693.47 | bwd_allreduce_microstep: 26.70 | step_microstep: 18.84
[2025-04-25 20:49:32,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5720.22 | bwd_inner: 5693.47 | bwd_allreduce: 26.71 | step: 18.84

 13%|█▎        | 5322/41250 [12:51:57<86:25:36,  8.66s/it]
                                                          
{'loss': 0.1557, 'grad_norm': 1.391118049621582, 'learning_rate': 3.898054935249948e-05, 'epoch': 1.29}

 13%|█▎        | 5322/41250 [12:51:57<86:25:36,  8.66s/it][2025-04-25 20:49:41,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 20:49:41,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.01 | bwd_microstep: 5719.11 | bwd_inner_microstep: 5635.90 | bwd_allreduce_microstep: 83.17 | step_microstep: 18.84
[2025-04-25 20:49:41,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.01 | bwd: 5719.13 | bwd_inner: 5635.90 | bwd_allreduce: 83.19 | step: 18.84

 13%|█▎        | 5323/41250 [12:52:06<86:18:42,  8.65s/it]
                                                          
{'loss': 0.263, 'grad_norm': 3.46669602394104, 'learning_rate': 3.898005433788036e-05, 'epoch': 1.29}

 13%|█▎        | 5323/41250 [12:52:06<86:18:42,  8.65s/it][2025-04-25 20:49:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:49:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.65 | bwd_microstep: 5685.50 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 35.31 | step_microstep: 18.58
[2025-04-25 20:49:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.65 | bwd: 5685.52 | bwd_inner: 5650.14 | bwd_allreduce: 35.33 | step: 18.58

 13%|█▎        | 5324/41250 [12:52:15<86:09:12,  8.63s/it]
                                                          
{'loss': 0.1042, 'grad_norm': 1.6797568798065186, 'learning_rate': 3.897955920625294e-05, 'epoch': 1.29}

 13%|█▎        | 5324/41250 [12:52:15<86:09:12,  8.63s/it][2025-04-25 20:49:58,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 20:49:58,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.14 | bwd_microstep: 5724.31 | bwd_inner_microstep: 5711.72 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.55
[2025-04-25 20:49:58,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.14 | bwd: 5724.33 | bwd_inner: 5711.72 | bwd_allreduce: 12.57 | step: 18.55

 13%|█▎        | 5325/41250 [12:52:23<86:14:13,  8.64s/it]
                                                          
{'loss': 0.0417, 'grad_norm': 0.763512372970581, 'learning_rate': 3.897906395762027e-05, 'epoch': 1.29}

 13%|█▎        | 5325/41250 [12:52:23<86:14:13,  8.64s/it][2025-04-25 20:50:07,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 20:50:07,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.64 | bwd_microstep: 5784.54 | bwd_inner_microstep: 5771.83 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.01
[2025-04-25 20:50:07,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.64 | bwd: 5784.56 | bwd_inner: 5771.83 | bwd_allreduce: 12.69 | step: 19.02

 13%|█▎        | 5326/41250 [12:52:32<86:34:48,  8.68s/it]
                                                          
{'loss': 0.1669, 'grad_norm': 1.1182764768600464, 'learning_rate': 3.8978568591985397e-05, 'epoch': 1.29}

 13%|█▎        | 5326/41250 [12:52:32<86:34:48,  8.68s/it][2025-04-25 20:50:15,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:50:15,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.26 | bwd_microstep: 5717.37 | bwd_inner_microstep: 5704.50 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.88
[2025-04-25 20:50:15,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.26 | bwd: 5717.39 | bwd_inner: 5704.49 | bwd_allreduce: 12.85 | step: 18.88

 13%|█▎        | 5327/41250 [12:52:41<86:30:01,  8.67s/it]
                                                          
{'loss': 0.102, 'grad_norm': 1.007108211517334, 'learning_rate': 3.897807310935139e-05, 'epoch': 1.29}

 13%|█▎        | 5327/41250 [12:52:41<86:30:01,  8.67s/it][2025-04-25 20:50:24,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-25 20:50:24,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.20 | bwd_microstep: 5685.10 | bwd_inner_microstep: 5671.18 | bwd_allreduce_microstep: 13.86 | step_microstep: 20.20
[2025-04-25 20:50:24,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.20 | bwd: 5685.11 | bwd_inner: 5671.18 | bwd_allreduce: 13.89 | step: 20.20

 13%|█▎        | 5328/41250 [12:52:49<86:19:45,  8.65s/it]
                                                          
{'loss': 0.1037, 'grad_norm': 2.1144182682037354, 'learning_rate': 3.897757750972129e-05, 'epoch': 1.29}

 13%|█▎        | 5328/41250 [12:52:49<86:19:45,  8.65s/it][2025-04-25 20:50:33,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 20:50:33,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.30 | bwd_microstep: 5694.36 | bwd_inner_microstep: 5681.58 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.83
[2025-04-25 20:50:33,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.30 | bwd: 5694.37 | bwd_inner: 5681.58 | bwd_allreduce: 12.75 | step: 18.83

 13%|█▎        | 5329/41250 [12:52:58<86:14:29,  8.64s/it]
                                                          
{'loss': 0.1879, 'grad_norm': 1.248883843421936, 'learning_rate': 3.8977081793098156e-05, 'epoch': 1.29}

 13%|█▎        | 5329/41250 [12:52:58<86:14:29,  8.64s/it][2025-04-25 20:50:41,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:50:41,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.52 | bwd_microstep: 5711.22 | bwd_inner_microstep: 5698.33 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.89
[2025-04-25 20:50:41,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.53 | bwd: 5711.23 | bwd_inner: 5698.33 | bwd_allreduce: 12.86 | step: 18.89

 13%|█▎        | 5330/41250 [12:53:07<86:14:09,  8.64s/it]
                                                          
{'loss': 0.0687, 'grad_norm': 2.392622709274292, 'learning_rate': 3.897658595948505e-05, 'epoch': 1.29}

 13%|█▎        | 5330/41250 [12:53:07<86:14:09,  8.64s/it][2025-04-25 20:50:50,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 20:50:50,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5713.53 | bwd_inner_microstep: 5700.87 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.70
[2025-04-25 20:50:50,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5713.54 | bwd_inner: 5700.87 | bwd_allreduce: 12.63 | step: 18.70

 13%|█▎        | 5331/41250 [12:53:15<86:14:33,  8.64s/it]
                                                          
{'loss': 0.2547, 'grad_norm': 2.2274696826934814, 'learning_rate': 3.8976090008885024e-05, 'epoch': 1.29}

 13%|█▎        | 5331/41250 [12:53:15<86:14:33,  8.64s/it][2025-04-25 20:50:59,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 20:50:59,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5845.22 | bwd_inner_microstep: 5635.06 | bwd_allreduce_microstep: 210.11 | step_microstep: 19.30
[2025-04-25 20:50:59,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5845.24 | bwd_inner: 5635.06 | bwd_allreduce: 210.13 | step: 19.30

 13%|█▎        | 5332/41250 [12:53:24<86:34:31,  8.68s/it]
                                                          
{'loss': 0.1289, 'grad_norm': 2.136383056640625, 'learning_rate': 3.897559394130113e-05, 'epoch': 1.29}

 13%|█▎        | 5332/41250 [12:53:24<86:34:31,  8.68s/it][2025-04-25 20:51:07,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:51:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.17 | bwd_microstep: 5744.81 | bwd_inner_microstep: 5662.83 | bwd_allreduce_microstep: 81.93 | step_microstep: 18.84
[2025-04-25 20:51:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5744.83 | bwd_inner: 5662.83 | bwd_allreduce: 81.95 | step: 18.85

 13%|█▎        | 5333/41250 [12:53:33<86:30:29,  8.67s/it]
                                                          
{'loss': 0.2246, 'grad_norm': 3.028548240661621, 'learning_rate': 3.8975097756736434e-05, 'epoch': 1.29}

 13%|█▎        | 5333/41250 [12:53:33<86:30:29,  8.67s/it][2025-04-25 20:51:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-25 20:51:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.75 | bwd_microstep: 5708.59 | bwd_inner_microstep: 5687.68 | bwd_allreduce_microstep: 20.87 | step_microstep: 18.54
[2025-04-25 20:51:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.75 | bwd: 5708.60 | bwd_inner: 5687.68 | bwd_allreduce: 20.89 | step: 18.55

 13%|█▎        | 5334/41250 [12:53:41<86:25:15,  8.66s/it]
                                                          
{'loss': 0.0834, 'grad_norm': 1.4576940536499023, 'learning_rate': 3.8974601455193995e-05, 'epoch': 1.29}

 13%|█▎        | 5334/41250 [12:53:41<86:25:15,  8.66s/it][2025-04-25 20:51:25,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-25 20:51:25,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.11 | bwd_microstep: 5730.30 | bwd_inner_microstep: 5708.47 | bwd_allreduce_microstep: 21.77 | step_microstep: 20.19
[2025-04-25 20:51:25,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.11 | bwd: 5730.31 | bwd_inner: 5708.47 | bwd_allreduce: 21.80 | step: 20.19

 13%|█▎        | 5335/41250 [12:53:50<86:27:16,  8.67s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 1.5402885675430298, 'learning_rate': 3.897410503667687e-05, 'epoch': 1.29}

 13%|█▎        | 5335/41250 [12:53:50<86:27:16,  8.67s/it][2025-04-25 20:51:33,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 20:51:33,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.62 | bwd_microstep: 5761.52 | bwd_inner_microstep: 5644.23 | bwd_allreduce_microstep: 117.24 | step_microstep: 18.86
[2025-04-25 20:51:33,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.62 | bwd: 5761.54 | bwd_inner: 5644.23 | bwd_allreduce: 117.26 | step: 18.86

 13%|█▎        | 5336/41250 [12:53:59<86:27:52,  8.67s/it]
                                                          
{'loss': 0.0177, 'grad_norm': 0.5361781716346741, 'learning_rate': 3.897360850118812e-05, 'epoch': 1.29}

 13%|█▎        | 5336/41250 [12:53:59<86:27:52,  8.67s/it][2025-04-25 20:51:42,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 20:51:42,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.33 | bwd_microstep: 5762.11 | bwd_inner_microstep: 5656.00 | bwd_allreduce_microstep: 106.07 | step_microstep: 18.68
[2025-04-25 20:51:42,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.33 | bwd: 5762.13 | bwd_inner: 5656.00 | bwd_allreduce: 106.09 | step: 18.68

 13%|█▎        | 5337/41250 [12:54:07<86:28:02,  8.67s/it]
                                                          
{'loss': 0.0248, 'grad_norm': 0.3793656527996063, 'learning_rate': 3.89731118487308e-05, 'epoch': 1.29}

 13%|█▎        | 5337/41250 [12:54:07<86:28:02,  8.67s/it][2025-04-25 20:51:51,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:51:51,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.30 | bwd_microstep: 5724.93 | bwd_inner_microstep: 5701.43 | bwd_allreduce_microstep: 23.46 | step_microstep: 19.00
[2025-04-25 20:51:51,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.30 | bwd: 5724.94 | bwd_inner: 5701.43 | bwd_allreduce: 23.47 | step: 19.00

 13%|█▎        | 5338/41250 [12:54:16<86:25:47,  8.66s/it]
                                                          
{'loss': 0.1266, 'grad_norm': 1.6418863534927368, 'learning_rate': 3.897261507930798e-05, 'epoch': 1.29}

 13%|█▎        | 5338/41250 [12:54:16<86:25:47,  8.66s/it][2025-04-25 20:51:59,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:51:59,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.36 | bwd_microstep: 5772.08 | bwd_inner_microstep: 5682.82 | bwd_allreduce_microstep: 89.21 | step_microstep: 18.58
[2025-04-25 20:51:59,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.36 | bwd: 5772.09 | bwd_inner: 5682.82 | bwd_allreduce: 89.23 | step: 18.58

 13%|█▎        | 5339/41250 [12:54:25<86:32:52,  8.68s/it]
                                                          
{'loss': 0.133, 'grad_norm': 1.769993782043457, 'learning_rate': 3.897211819292272e-05, 'epoch': 1.29}

 13%|█▎        | 5339/41250 [12:54:25<86:32:52,  8.68s/it][2025-04-25 20:52:08,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 20:52:08,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.67 | bwd_microstep: 5773.53 | bwd_inner_microstep: 5651.40 | bwd_allreduce_microstep: 122.09 | step_microstep: 18.61
[2025-04-25 20:52:08,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.68 | bwd: 5773.54 | bwd_inner: 5651.40 | bwd_allreduce: 122.10 | step: 18.61

 13%|█▎        | 5340/41250 [12:54:33<86:32:58,  8.68s/it]
                                                          
{'loss': 0.209, 'grad_norm': 2.334758996963501, 'learning_rate': 3.8971621189578085e-05, 'epoch': 1.29}

 13%|█▎        | 5340/41250 [12:54:33<86:32:58,  8.68s/it][2025-04-25 20:52:17,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.10 | optimizer_step: 0.99
[2025-04-25 20:52:17,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.76 | bwd_microstep: 5708.04 | bwd_inner_microstep: 5695.10 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.54
[2025-04-25 20:52:17,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.76 | bwd: 5708.06 | bwd_inner: 5695.10 | bwd_allreduce: 12.91 | step: 19.54

 13%|█▎        | 5341/41250 [12:54:42<86:25:52,  8.67s/it]
                                                          
{'loss': 0.0705, 'grad_norm': 1.5546172857284546, 'learning_rate': 3.897112406927713e-05, 'epoch': 1.29}

 13%|█▎        | 5341/41250 [12:54:42<86:25:52,  8.67s/it][2025-04-25 20:52:26,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-25 20:52:26,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5920.43 | bwd_inner_microstep: 5691.69 | bwd_allreduce_microstep: 228.68 | step_microstep: 19.31
[2025-04-25 20:52:26,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5920.45 | bwd_inner: 5691.69 | bwd_allreduce: 228.71 | step: 19.31

 13%|█▎        | 5342/41250 [12:54:51<87:00:14,  8.72s/it]
                                                          
{'loss': 0.1173, 'grad_norm': 1.6627469062805176, 'learning_rate': 3.8970626832022934e-05, 'epoch': 1.3}

 13%|█▎        | 5342/41250 [12:54:51<87:00:14,  8.72s/it][2025-04-25 20:52:34,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:52:34,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.63 | bwd_microstep: 5944.38 | bwd_inner_microstep: 5651.56 | bwd_allreduce_microstep: 292.77 | step_microstep: 19.00
[2025-04-25 20:52:34,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.63 | bwd: 5944.40 | bwd_inner: 5651.56 | bwd_allreduce: 292.79 | step: 19.00

 13%|█▎        | 5343/41250 [12:55:00<87:23:58,  8.76s/it]
                                                          
{'loss': 0.0449, 'grad_norm': 0.8988699316978455, 'learning_rate': 3.897012947781855e-05, 'epoch': 1.3}

 13%|█▎        | 5343/41250 [12:55:00<87:23:58,  8.76s/it][2025-04-25 20:52:43,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-25 20:52:43,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.81 | bwd_microstep: 5766.62 | bwd_inner_microstep: 5688.01 | bwd_allreduce_microstep: 78.55 | step_microstep: 19.08
[2025-04-25 20:52:43,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.81 | bwd: 5766.63 | bwd_inner: 5688.01 | bwd_allreduce: 78.57 | step: 19.08

 13%|█▎        | 5344/41250 [12:55:08<87:12:21,  8.74s/it]
                                                          
{'loss': 0.1197, 'grad_norm': 2.405693531036377, 'learning_rate': 3.896963200666706e-05, 'epoch': 1.3}

 13%|█▎        | 5344/41250 [12:55:08<87:12:21,  8.74s/it][2025-04-25 20:52:52,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.21 | optimizer_step: 1.06
[2025-04-25 20:52:52,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.84 | bwd_microstep: 5743.43 | bwd_inner_microstep: 5700.10 | bwd_allreduce_microstep: 43.27 | step_microstep: 20.55
[2025-04-25 20:52:52,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.84 | bwd: 5743.45 | bwd_inner: 5700.10 | bwd_allreduce: 43.30 | step: 20.55

 13%|█▎        | 5345/41250 [12:55:17<87:00:43,  8.72s/it]
                                                          
{'loss': 0.3924, 'grad_norm': 3.4353883266448975, 'learning_rate': 3.896913441857151e-05, 'epoch': 1.3}

 13%|█▎        | 5345/41250 [12:55:17<87:00:43,  8.72s/it][2025-04-25 20:53:00,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 20:53:00,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.60 | bwd_microstep: 5763.20 | bwd_inner_microstep: 5703.98 | bwd_allreduce_microstep: 59.17 | step_microstep: 19.05
[2025-04-25 20:53:00,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.60 | bwd: 5763.21 | bwd_inner: 5703.98 | bwd_allreduce: 59.19 | step: 19.05

 13%|█▎        | 5346/41250 [12:55:26<86:57:05,  8.72s/it]
                                                          
{'loss': 0.0191, 'grad_norm': 0.2267979085445404, 'learning_rate': 3.8968636713534975e-05, 'epoch': 1.3}

 13%|█▎        | 5346/41250 [12:55:26<86:57:05,  8.72s/it][2025-04-25 20:53:09,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:53:09,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.85 | bwd_microstep: 5714.38 | bwd_inner_microstep: 5667.19 | bwd_allreduce_microstep: 47.14 | step_microstep: 18.75
[2025-04-25 20:53:09,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.85 | bwd: 5714.39 | bwd_inner: 5667.19 | bwd_allreduce: 47.16 | step: 18.75

 13%|█▎        | 5347/41250 [12:55:34<86:41:19,  8.69s/it]
                                                          
{'loss': 0.1644, 'grad_norm': 2.3520567417144775, 'learning_rate': 3.896813889156053e-05, 'epoch': 1.3}

 13%|█▎        | 5347/41250 [12:55:34<86:41:19,  8.69s/it][2025-04-25 20:53:18,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:53:18,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.16 | bwd_microstep: 5801.69 | bwd_inner_microstep: 5788.98 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.77
[2025-04-25 20:53:18,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.16 | bwd: 5801.70 | bwd_inner: 5788.98 | bwd_allreduce: 12.69 | step: 18.78

 13%|█▎        | 5348/41250 [12:55:43<86:56:14,  8.72s/it]
                                                          
{'loss': 0.214, 'grad_norm': 3.2804782390594482, 'learning_rate': 3.896764095265124e-05, 'epoch': 1.3}

 13%|█▎        | 5348/41250 [12:55:43<86:56:14,  8.72s/it][2025-04-25 20:53:27,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:53:27,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5720.14 | bwd_inner_microstep: 5707.44 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.82
[2025-04-25 20:53:27,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5720.15 | bwd_inner: 5707.44 | bwd_allreduce: 12.67 | step: 18.82

 13%|█▎        | 5349/41250 [12:55:52<86:45:15,  8.70s/it]
                                                          
{'loss': 0.295, 'grad_norm': 2.3660857677459717, 'learning_rate': 3.8967142896810177e-05, 'epoch': 1.3}

 13%|█▎        | 5349/41250 [12:55:52<86:45:15,  8.70s/it][2025-04-25 20:53:35,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:53:35,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.28 | bwd_microstep: 5727.18 | bwd_inner_microstep: 5714.46 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.70
[2025-04-25 20:53:35,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.28 | bwd: 5727.19 | bwd_inner: 5714.45 | bwd_allreduce: 12.70 | step: 18.70

 13%|█▎        | 5350/41250 [12:56:01<86:38:23,  8.69s/it]
                                                          
{'loss': 0.1228, 'grad_norm': 2.381660223007202, 'learning_rate': 3.89666447240404e-05, 'epoch': 1.3}

 13%|█▎        | 5350/41250 [12:56:01<86:38:23,  8.69s/it][2025-04-25 20:53:44,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:53:44,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5724.93 | bwd_inner_microstep: 5712.17 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.86
[2025-04-25 20:53:44,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5724.94 | bwd_inner: 5712.17 | bwd_allreduce: 12.73 | step: 18.86

 13%|█▎        | 5351/41250 [12:56:09<86:34:52,  8.68s/it]
                                                          
{'loss': 0.2828, 'grad_norm': 1.6854013204574585, 'learning_rate': 3.8966146434344996e-05, 'epoch': 1.3}

 13%|█▎        | 5351/41250 [12:56:09<86:34:52,  8.68s/it][2025-04-25 20:53:53,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:53:53,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.43 | bwd_microstep: 5735.54 | bwd_inner_microstep: 5719.36 | bwd_allreduce_microstep: 16.13 | step_microstep: 18.99
[2025-04-25 20:53:53,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.43 | bwd: 5735.55 | bwd_inner: 5719.36 | bwd_allreduce: 16.15 | step: 18.99

 13%|█▎        | 5352/41250 [12:56:18<86:35:28,  8.68s/it]
                                                          
{'loss': 0.1967, 'grad_norm': 2.0797078609466553, 'learning_rate': 3.8965648027727024e-05, 'epoch': 1.3}

 13%|█▎        | 5352/41250 [12:56:18<86:35:28,  8.68s/it][2025-04-25 20:54:01,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:54:01,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5717.25 | bwd_inner_microstep: 5661.60 | bwd_allreduce_microstep: 55.60 | step_microstep: 18.55
[2025-04-25 20:54:01,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5717.26 | bwd_inner: 5661.60 | bwd_allreduce: 55.62 | step: 18.55

 13%|█▎        | 5353/41250 [12:56:27<86:25:00,  8.67s/it]
                                                          
{'loss': 0.2529, 'grad_norm': 2.134019613265991, 'learning_rate': 3.896514950418957e-05, 'epoch': 1.3}

 13%|█▎        | 5353/41250 [12:56:27<86:25:00,  8.67s/it][2025-04-25 20:54:10,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 20:54:10,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.98 | bwd_microstep: 5815.40 | bwd_inner_microstep: 5661.47 | bwd_allreduce_microstep: 153.88 | step_microstep: 19.27
[2025-04-25 20:54:10,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.98 | bwd: 5815.41 | bwd_inner: 5661.47 | bwd_allreduce: 153.90 | step: 19.27

 13%|█▎        | 5354/41250 [12:56:35<86:36:44,  8.69s/it]
                                                          
{'loss': 0.0211, 'grad_norm': 0.23205003142356873, 'learning_rate': 3.89646508637357e-05, 'epoch': 1.3}

 13%|█▎        | 5354/41250 [12:56:35<86:36:44,  8.69s/it][2025-04-25 20:54:19,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 0.96
[2025-04-25 20:54:19,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.87 | bwd_microstep: 5796.39 | bwd_inner_microstep: 5661.55 | bwd_allreduce_microstep: 134.79 | step_microstep: 19.05
[2025-04-25 20:54:19,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.88 | bwd: 5796.40 | bwd_inner: 5661.56 | bwd_allreduce: 134.81 | step: 19.06

 13%|█▎        | 5355/41250 [12:56:44<86:41:09,  8.69s/it]
                                                          
{'loss': 0.112, 'grad_norm': 2.975705862045288, 'learning_rate': 3.896415210636848e-05, 'epoch': 1.3}

 13%|█▎        | 5355/41250 [12:56:44<86:41:09,  8.69s/it][2025-04-25 20:54:27,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:54:27,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.34 | bwd_microstep: 5808.95 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 151.06 | step_microstep: 18.59
[2025-04-25 20:54:27,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.34 | bwd: 5808.96 | bwd_inner: 5657.84 | bwd_allreduce: 151.08 | step: 18.60

 13%|█▎        | 5356/41250 [12:56:53<86:46:03,  8.70s/it]
                                                          
{'loss': 0.2154, 'grad_norm': 1.9624645709991455, 'learning_rate': 3.896365323209099e-05, 'epoch': 1.3}

 13%|█▎        | 5356/41250 [12:56:53<86:46:03,  8.70s/it][2025-04-25 20:54:36,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 20:54:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.75 | bwd_microstep: 5772.08 | bwd_inner_microstep: 5705.19 | bwd_allreduce_microstep: 66.85 | step_microstep: 18.26
[2025-04-25 20:54:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.75 | bwd: 5772.09 | bwd_inner: 5705.19 | bwd_allreduce: 66.86 | step: 18.26

 13%|█▎        | 5357/41250 [12:57:01<86:46:23,  8.70s/it]
                                                          
{'loss': 0.1432, 'grad_norm': 3.642180919647217, 'learning_rate': 3.8963154240906315e-05, 'epoch': 1.3}

 13%|█▎        | 5357/41250 [12:57:01<86:46:23,  8.70s/it][2025-04-25 20:54:45,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 20:54:45,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.09 | bwd_microstep: 5713.52 | bwd_inner_microstep: 5700.56 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.63
[2025-04-25 20:54:45,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.09 | bwd: 5713.54 | bwd_inner: 5700.56 | bwd_allreduce: 12.93 | step: 18.63

 13%|█▎        | 5358/41250 [12:57:10<86:35:26,  8.69s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 1.4851106405258179, 'learning_rate': 3.896265513281752e-05, 'epoch': 1.3}

 13%|█▎        | 5358/41250 [12:57:10<86:35:26,  8.69s/it][2025-04-25 20:54:53,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.10 | optimizer_step: 1.01
[2025-04-25 20:54:53,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5777.76 | bwd_inner_microstep: 5708.22 | bwd_allreduce_microstep: 69.49 | step_microstep: 18.74
[2025-04-25 20:54:53,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5777.77 | bwd_inner: 5708.22 | bwd_allreduce: 69.51 | step: 18.74

 13%|█▎        | 5359/41250 [12:57:19<86:40:19,  8.69s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 3.4346909523010254, 'learning_rate': 3.8962155907827695e-05, 'epoch': 1.3}

 13%|█▎        | 5359/41250 [12:57:19<86:40:19,  8.69s/it][2025-04-25 20:55:02,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 20:55:02,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.28 | bwd_microstep: 5743.66 | bwd_inner_microstep: 5691.92 | bwd_allreduce_microstep: 51.69 | step_microstep: 18.71
[2025-04-25 20:55:02,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.28 | bwd: 5743.67 | bwd_inner: 5691.92 | bwd_allreduce: 51.71 | step: 18.71

 13%|█▎        | 5360/41250 [12:57:27<86:42:13,  8.70s/it]
                                                          
{'loss': 0.0426, 'grad_norm': 0.8912521600723267, 'learning_rate': 3.89616565659399e-05, 'epoch': 1.3}

 13%|█▎        | 5360/41250 [12:57:27<86:42:13,  8.70s/it][2025-04-25 20:55:11,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 20:55:11,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.73 | bwd_microstep: 5795.96 | bwd_inner_microstep: 5782.98 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.39
[2025-04-25 20:55:11,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.73 | bwd: 5795.98 | bwd_inner: 5782.98 | bwd_allreduce: 12.95 | step: 19.39

 13%|█▎        | 5361/41250 [12:57:36<86:57:35,  8.72s/it]
                                                          
{'loss': 0.1213, 'grad_norm': 0.9914132952690125, 'learning_rate': 3.896115710715722e-05, 'epoch': 1.3}

 13%|█▎        | 5361/41250 [12:57:36<86:57:35,  8.72s/it][2025-04-25 20:55:20,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 20:55:20,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.59 | bwd_microstep: 5907.32 | bwd_inner_microstep: 5688.38 | bwd_allreduce_microstep: 218.90 | step_microstep: 18.93
[2025-04-25 20:55:20,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.59 | bwd: 5907.34 | bwd_inner: 5688.38 | bwd_allreduce: 218.92 | step: 18.93

 13%|█▎        | 5362/41250 [12:57:45<87:19:02,  8.76s/it]
                                                          
{'loss': 0.2273, 'grad_norm': 3.973499298095703, 'learning_rate': 3.896065753148274e-05, 'epoch': 1.3}

 13%|█▎        | 5362/41250 [12:57:45<87:19:02,  8.76s/it][2025-04-25 20:55:28,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 20:55:28,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.34 | bwd_microstep: 5737.33 | bwd_inner_microstep: 5711.18 | bwd_allreduce_microstep: 26.10 | step_microstep: 19.11
[2025-04-25 20:55:28,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5737.34 | bwd_inner: 5711.18 | bwd_allreduce: 26.12 | step: 19.11

 13%|█▎        | 5363/41250 [12:57:54<87:03:30,  8.73s/it]
                                                          
{'loss': 0.1304, 'grad_norm': 2.1885979175567627, 'learning_rate': 3.896015783891954e-05, 'epoch': 1.3}

 13%|█▎        | 5363/41250 [12:57:54<87:03:30,  8.73s/it][2025-04-25 20:55:37,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:55:37,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5706.54 | bwd_inner_microstep: 5659.69 | bwd_allreduce_microstep: 46.80 | step_microstep: 18.91
[2025-04-25 20:55:37,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5706.55 | bwd_inner: 5659.69 | bwd_allreduce: 46.82 | step: 18.92

 13%|█▎        | 5364/41250 [12:58:02<86:44:11,  8.70s/it]
                                                          
{'loss': 0.1981, 'grad_norm': 3.0357449054718018, 'learning_rate': 3.8959658029470686e-05, 'epoch': 1.3}

 13%|█▎        | 5364/41250 [12:58:02<86:44:11,  8.70s/it][2025-04-25 20:55:46,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.34 | optimizer_step: 1.06
[2025-04-25 20:55:46,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.03 | bwd_microstep: 5684.54 | bwd_inner_microstep: 5667.47 | bwd_allreduce_microstep: 17.01 | step_microstep: 19.96
[2025-04-25 20:55:46,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.03 | bwd: 5684.56 | bwd_inner: 5667.47 | bwd_allreduce: 17.04 | step: 19.96

 13%|█▎        | 5365/41250 [12:58:11<86:25:49,  8.67s/it]
                                                          
{'loss': 0.1636, 'grad_norm': 2.1661264896392822, 'learning_rate': 3.8959158103139275e-05, 'epoch': 1.3}

 13%|█▎        | 5365/41250 [12:58:11<86:25:49,  8.67s/it][2025-04-25 20:55:54,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-25 20:55:54,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.76 | bwd_microstep: 5734.14 | bwd_inner_microstep: 5685.43 | bwd_allreduce_microstep: 48.67 | step_microstep: 19.11
[2025-04-25 20:55:54,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.76 | bwd: 5734.16 | bwd_inner: 5685.43 | bwd_allreduce: 48.69 | step: 19.11

 13%|█▎        | 5366/41250 [12:58:20<86:24:51,  8.67s/it]
                                                          
{'loss': 0.128, 'grad_norm': 2.3526453971862793, 'learning_rate': 3.895865805992838e-05, 'epoch': 1.3}

 13%|█▎        | 5366/41250 [12:58:20<86:24:51,  8.67s/it][2025-04-25 20:56:03,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 20:56:03,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.23 | bwd_microstep: 5746.71 | bwd_inner_microstep: 5694.90 | bwd_allreduce_microstep: 51.77 | step_microstep: 18.60
[2025-04-25 20:56:03,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.23 | bwd: 5746.72 | bwd_inner: 5694.90 | bwd_allreduce: 51.78 | step: 18.60

 13%|█▎        | 5367/41250 [12:58:28<86:25:34,  8.67s/it]
                                                          
{'loss': 0.104, 'grad_norm': 1.4592487812042236, 'learning_rate': 3.895815789984109e-05, 'epoch': 1.3}

 13%|█▎        | 5367/41250 [12:58:28<86:25:34,  8.67s/it][2025-04-25 20:56:12,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 20:56:12,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5773.21 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 109.95 | step_microstep: 18.72
[2025-04-25 20:56:12,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5773.22 | bwd_inner: 5663.21 | bwd_allreduce: 109.96 | step: 18.73

 13%|█▎        | 5368/41250 [12:58:37<86:28:39,  8.68s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 0.9066182374954224, 'learning_rate': 3.895765762288049e-05, 'epoch': 1.3}

 13%|█▎        | 5368/41250 [12:58:37<86:28:39,  8.68s/it][2025-04-25 20:56:20,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.26 | optimizer_step: 0.97
[2025-04-25 20:56:20,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5711.07 | bwd_inner_microstep: 5651.15 | bwd_allreduce_microstep: 59.86 | step_microstep: 19.51
[2025-04-25 20:56:20,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5711.08 | bwd_inner: 5651.15 | bwd_allreduce: 59.89 | step: 19.52

 13%|█▎        | 5369/41250 [12:58:46<86:19:23,  8.66s/it]
                                                          
{'loss': 0.1105, 'grad_norm': 1.4671945571899414, 'learning_rate': 3.895715722904965e-05, 'epoch': 1.3}

 13%|█▎        | 5369/41250 [12:58:46<86:19:23,  8.66s/it][2025-04-25 20:56:29,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 20:56:29,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5750.63 | bwd_inner_microstep: 5678.77 | bwd_allreduce_microstep: 71.82 | step_microstep: 19.05
[2025-04-25 20:56:29,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5750.64 | bwd_inner: 5678.77 | bwd_allreduce: 71.83 | step: 19.05

 13%|█▎        | 5370/41250 [12:58:54<86:22:04,  8.67s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 0.8624488711357117, 'learning_rate': 3.8956656718351674e-05, 'epoch': 1.3}

 13%|█▎        | 5370/41250 [12:58:54<86:22:04,  8.67s/it][2025-04-25 20:56:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 20:56:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.62 | bwd_microstep: 5733.93 | bwd_inner_microstep: 5654.20 | bwd_allreduce_microstep: 79.69 | step_microstep: 18.44
[2025-04-25 20:56:38,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.62 | bwd: 5733.94 | bwd_inner: 5654.20 | bwd_allreduce: 79.70 | step: 18.44

 13%|█▎        | 5371/41250 [12:59:03<86:18:07,  8.66s/it]
                                                          
{'loss': 0.2834, 'grad_norm': 1.972223162651062, 'learning_rate': 3.895615609078963e-05, 'epoch': 1.3}

 13%|█▎        | 5371/41250 [12:59:03<86:18:07,  8.66s/it][2025-04-25 20:56:46,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 20:56:46,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.06 | bwd_microstep: 5744.18 | bwd_inner_microstep: 5684.01 | bwd_allreduce_microstep: 60.12 | step_microstep: 18.77
[2025-04-25 20:56:46,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.06 | bwd: 5744.19 | bwd_inner: 5684.01 | bwd_allreduce: 60.14 | step: 18.77

 13%|█▎        | 5372/41250 [12:59:12<86:20:05,  8.66s/it]
                                                          
{'loss': 0.139, 'grad_norm': 1.191310167312622, 'learning_rate': 3.895565534636661e-05, 'epoch': 1.3}

 13%|█▎        | 5372/41250 [12:59:12<86:20:05,  8.66s/it][2025-04-25 20:56:55,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 20:56:55,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.43 | bwd_microstep: 5759.22 | bwd_inner_microstep: 5691.67 | bwd_allreduce_microstep: 67.51 | step_microstep: 18.80
[2025-04-25 20:56:55,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.43 | bwd: 5759.23 | bwd_inner: 5691.67 | bwd_allreduce: 67.52 | step: 18.80

 13%|█▎        | 5373/41250 [12:59:20<86:24:38,  8.67s/it]
                                                          
{'loss': 0.0645, 'grad_norm': 1.540278434753418, 'learning_rate': 3.8955154485085716e-05, 'epoch': 1.3}

 13%|█▎        | 5373/41250 [12:59:20<86:24:38,  8.67s/it][2025-04-25 20:57:04,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 20:57:04,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.76 | bwd_microstep: 5742.18 | bwd_inner_microstep: 5652.34 | bwd_allreduce_microstep: 89.79 | step_microstep: 18.71
[2025-04-25 20:57:04,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.76 | bwd: 5742.19 | bwd_inner: 5652.34 | bwd_allreduce: 89.81 | step: 18.72

 13%|█▎        | 5374/41250 [12:59:29<86:23:09,  8.67s/it]
                                                          
{'loss': 0.1971, 'grad_norm': 2.541630744934082, 'learning_rate': 3.895465350695001e-05, 'epoch': 1.3}

 13%|█▎        | 5374/41250 [12:59:29<86:23:09,  8.67s/it][2025-04-25 20:57:12,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 1.01
[2025-04-25 20:57:12,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5746.00 | bwd_inner_microstep: 5689.61 | bwd_allreduce_microstep: 56.33 | step_microstep: 19.55
[2025-04-25 20:57:12,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5746.02 | bwd_inner: 5689.61 | bwd_allreduce: 56.35 | step: 19.56

 13%|█▎        | 5375/41250 [12:59:38<86:26:21,  8.67s/it]
                                                          
{'loss': 0.0428, 'grad_norm': 0.5607209205627441, 'learning_rate': 3.89541524119626e-05, 'epoch': 1.3}

 13%|█▎        | 5375/41250 [12:59:38<86:26:21,  8.67s/it][2025-04-25 20:57:21,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:57:21,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.87 | bwd_microstep: 5723.34 | bwd_inner_microstep: 5696.10 | bwd_allreduce_microstep: 27.19 | step_microstep: 18.48
[2025-04-25 20:57:21,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.87 | bwd: 5723.35 | bwd_inner: 5696.10 | bwd_allreduce: 27.20 | step: 18.48

 13%|█▎        | 5376/41250 [12:59:46<86:21:59,  8.67s/it]
                                                          
{'loss': 0.0822, 'grad_norm': 0.9110553860664368, 'learning_rate': 3.895365120012657e-05, 'epoch': 1.3}

 13%|█▎        | 5376/41250 [12:59:46<86:21:59,  8.67s/it][2025-04-25 20:57:30,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:57:30,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.47 | bwd_microstep: 5743.96 | bwd_inner_microstep: 5703.94 | bwd_allreduce_microstep: 39.98 | step_microstep: 18.36
[2025-04-25 20:57:30,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.47 | bwd: 5743.97 | bwd_inner: 5703.94 | bwd_allreduce: 40.00 | step: 18.36

 13%|█▎        | 5377/41250 [12:59:55<86:23:32,  8.67s/it]
                                                          
{'loss': 0.2316, 'grad_norm': 2.963923931121826, 'learning_rate': 3.8953149871445005e-05, 'epoch': 1.3}

 13%|█▎        | 5377/41250 [12:59:55<86:23:32,  8.67s/it][2025-04-25 20:57:38,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.94
[2025-04-25 20:57:38,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5706.63 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.25
[2025-04-25 20:57:38,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5706.64 | bwd_inner: 5693.87 | bwd_allreduce: 12.73 | step: 19.25

 13%|█▎        | 5378/41250 [13:00:04<86:20:02,  8.66s/it]
                                                          
{'loss': 0.126, 'grad_norm': 2.3201181888580322, 'learning_rate': 3.8952648425921e-05, 'epoch': 1.3}

 13%|█▎        | 5378/41250 [13:00:04<86:20:02,  8.66s/it][2025-04-25 20:57:47,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 20:57:47,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.03 | bwd_microstep: 5692.90 | bwd_inner_microstep: 5680.17 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.74
[2025-04-25 20:57:47,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.03 | bwd: 5692.92 | bwd_inner: 5680.17 | bwd_allreduce: 12.70 | step: 18.75

 13%|█▎        | 5379/41250 [13:00:12<86:15:16,  8.66s/it]
                                                          
{'loss': 0.229, 'grad_norm': 3.53867506980896, 'learning_rate': 3.895214686355764e-05, 'epoch': 1.3}

 13%|█▎        | 5379/41250 [13:00:12<86:15:16,  8.66s/it][2025-04-25 20:57:56,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 20:57:56,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.03 | bwd_microstep: 5756.11 | bwd_inner_microstep: 5652.98 | bwd_allreduce_microstep: 103.08 | step_microstep: 19.17
[2025-04-25 20:57:56,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.03 | bwd: 5756.13 | bwd_inner: 5652.98 | bwd_allreduce: 103.11 | step: 19.17

 13%|█▎        | 5380/41250 [13:00:21<86:20:19,  8.67s/it]
                                                          
{'loss': 0.0486, 'grad_norm': 0.9903451204299927, 'learning_rate': 3.895164518435803e-05, 'epoch': 1.3}

 13%|█▎        | 5380/41250 [13:00:21<86:20:19,  8.67s/it][2025-04-25 20:58:04,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:58:04,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.88 | bwd_microstep: 5708.82 | bwd_inner_microstep: 5696.01 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.56
[2025-04-25 20:58:04,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.88 | bwd: 5708.83 | bwd_inner: 5696.01 | bwd_allreduce: 12.78 | step: 18.57

 13%|█▎        | 5381/41250 [13:00:30<86:17:14,  8.66s/it]
                                                          
{'loss': 0.0734, 'grad_norm': 1.1505106687545776, 'learning_rate': 3.895114338832525e-05, 'epoch': 1.3}

 13%|█▎        | 5381/41250 [13:00:30<86:17:14,  8.66s/it][2025-04-25 20:58:13,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 20:58:13,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5716.16 | bwd_inner_microstep: 5703.25 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.76
[2025-04-25 20:58:13,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5716.18 | bwd_inner: 5703.25 | bwd_allreduce: 12.87 | step: 18.76

 13%|█▎        | 5382/41250 [13:00:38<86:16:18,  8.66s/it]
                                                          
{'loss': 0.1197, 'grad_norm': 1.0969042778015137, 'learning_rate': 3.89506414754624e-05, 'epoch': 1.3}

 13%|█▎        | 5382/41250 [13:00:38<86:16:18,  8.66s/it][2025-04-25 20:58:22,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 20:58:22,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5712.19 | bwd_inner_microstep: 5699.31 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.07
[2025-04-25 20:58:22,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5712.21 | bwd_inner: 5699.31 | bwd_allreduce: 12.86 | step: 19.07

 13%|█▎        | 5383/41250 [13:00:47<86:14:00,  8.66s/it]
                                                          
{'loss': 0.1696, 'grad_norm': 1.6733300685882568, 'learning_rate': 3.8950139445772574e-05, 'epoch': 1.3}

 13%|█▎        | 5383/41250 [13:00:47<86:14:00,  8.66s/it][2025-04-25 20:58:30,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 20:58:30,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.33 | bwd_microstep: 5709.85 | bwd_inner_microstep: 5697.04 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.66
[2025-04-25 20:58:30,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.33 | bwd: 5709.86 | bwd_inner: 5697.04 | bwd_allreduce: 12.78 | step: 18.66

 13%|█▎        | 5384/41250 [13:00:56<86:13:52,  8.66s/it]
                                                          
{'loss': 0.1339, 'grad_norm': 1.2872401475906372, 'learning_rate': 3.894963729925886e-05, 'epoch': 1.31}

 13%|█▎        | 5384/41250 [13:00:56<86:13:52,  8.66s/it][2025-04-25 20:58:39,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-25 20:58:39,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.74 | bwd_microstep: 5687.08 | bwd_inner_microstep: 5647.24 | bwd_allreduce_microstep: 39.79 | step_microstep: 18.88
[2025-04-25 20:58:39,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.74 | bwd: 5687.09 | bwd_inner: 5647.24 | bwd_allreduce: 39.81 | step: 18.88

 13%|█▎        | 5385/41250 [13:01:04<86:04:43,  8.64s/it]
                                                          
{'loss': 0.1595, 'grad_norm': 1.3653745651245117, 'learning_rate': 3.894913503592437e-05, 'epoch': 1.31}

 13%|█▎        | 5385/41250 [13:01:04<86:04:43,  8.64s/it][2025-04-25 20:58:47,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.08 | optimizer_step: 1.07
[2025-04-25 20:58:47,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.99 | bwd_microstep: 5700.74 | bwd_inner_microstep: 5659.27 | bwd_allreduce_microstep: 41.42 | step_microstep: 19.45
[2025-04-25 20:58:47,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.99 | bwd: 5700.76 | bwd_inner: 5659.27 | bwd_allreduce: 41.44 | step: 19.45

 13%|█▎        | 5386/41250 [13:01:13<86:00:55,  8.63s/it]
                                                          
{'loss': 0.4091, 'grad_norm': 2.8910772800445557, 'learning_rate': 3.894863265577218e-05, 'epoch': 1.31}

 13%|█▎        | 5386/41250 [13:01:13<86:00:55,  8.63s/it][2025-04-25 20:58:56,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 20:58:56,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.40 | bwd_microstep: 5769.28 | bwd_inner_microstep: 5657.30 | bwd_allreduce_microstep: 111.92 | step_microstep: 19.03
[2025-04-25 20:58:56,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.40 | bwd: 5769.29 | bwd_inner: 5657.30 | bwd_allreduce: 111.94 | step: 19.03

 13%|█▎        | 5387/41250 [13:01:21<86:09:53,  8.65s/it]
                                                          
{'loss': 0.0717, 'grad_norm': 1.632311224937439, 'learning_rate': 3.89481301588054e-05, 'epoch': 1.31}

 13%|█▎        | 5387/41250 [13:01:21<86:09:53,  8.65s/it][2025-04-25 20:59:05,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.05
[2025-04-25 20:59:05,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.50 | bwd_microstep: 5697.34 | bwd_inner_microstep: 5678.21 | bwd_allreduce_microstep: 19.08 | step_microstep: 19.73
[2025-04-25 20:59:05,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.49 | bwd: 5697.35 | bwd_inner: 5678.21 | bwd_allreduce: 19.10 | step: 19.73

 13%|█▎        | 5388/41250 [13:01:30<86:05:34,  8.64s/it]
                                                          
{'loss': 0.0778, 'grad_norm': 2.2529516220092773, 'learning_rate': 3.8947627545027124e-05, 'epoch': 1.31}

 13%|█▎        | 5388/41250 [13:01:30<86:05:34,  8.64s/it][2025-04-25 20:59:13,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 20:59:13,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.15 | bwd_microstep: 5752.96 | bwd_inner_microstep: 5701.09 | bwd_allreduce_microstep: 51.82 | step_microstep: 18.94
[2025-04-25 20:59:13,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.15 | bwd: 5752.98 | bwd_inner: 5701.09 | bwd_allreduce: 51.84 | step: 18.94

 13%|█▎        | 5389/41250 [13:01:39<86:12:28,  8.65s/it]
                                                          
{'loss': 0.122, 'grad_norm': 2.237243890762329, 'learning_rate': 3.894712481444045e-05, 'epoch': 1.31}

 13%|█▎        | 5389/41250 [13:01:39<86:12:28,  8.65s/it][2025-04-25 20:59:22,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 20:59:22,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.51 | bwd_microstep: 5715.25 | bwd_inner_microstep: 5660.77 | bwd_allreduce_microstep: 54.43 | step_microstep: 18.80
[2025-04-25 20:59:22,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.51 | bwd: 5715.26 | bwd_inner: 5660.77 | bwd_allreduce: 54.45 | step: 18.80

 13%|█▎        | 5390/41250 [13:01:47<86:08:19,  8.65s/it]
                                                          
{'loss': 0.0719, 'grad_norm': 1.352735161781311, 'learning_rate': 3.894662196704848e-05, 'epoch': 1.31}

 13%|█▎        | 5390/41250 [13:01:47<86:08:19,  8.65s/it][2025-04-25 20:59:31,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 20:59:31,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5713.69 | bwd_inner_microstep: 5701.00 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.66
[2025-04-25 20:59:31,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5713.70 | bwd_inner: 5701.00 | bwd_allreduce: 12.66 | step: 18.66

 13%|█▎        | 5391/41250 [13:01:56<86:07:40,  8.65s/it]
                                                          
{'loss': 0.248, 'grad_norm': 2.4674599170684814, 'learning_rate': 3.894611900285431e-05, 'epoch': 1.31}

 13%|█▎        | 5391/41250 [13:01:56<86:07:40,  8.65s/it][2025-04-25 20:59:39,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 20:59:39,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.79 | bwd_microstep: 5717.00 | bwd_inner_microstep: 5664.33 | bwd_allreduce_microstep: 52.62 | step_microstep: 18.60
[2025-04-25 20:59:39,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.79 | bwd: 5717.01 | bwd_inner: 5664.33 | bwd_allreduce: 52.64 | step: 18.61

 13%|█▎        | 5392/41250 [13:02:05<86:05:02,  8.64s/it]
                                                          
{'loss': 0.108, 'grad_norm': 2.345466136932373, 'learning_rate': 3.894561592186105e-05, 'epoch': 1.31}

 13%|█▎        | 5392/41250 [13:02:05<86:05:02,  8.64s/it][2025-04-25 20:59:48,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 20:59:48,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.28 | bwd_microstep: 5763.08 | bwd_inner_microstep: 5652.58 | bwd_allreduce_microstep: 110.46 | step_microstep: 18.68
[2025-04-25 20:59:48,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.28 | bwd: 5763.09 | bwd_inner: 5652.58 | bwd_allreduce: 110.47 | step: 18.69

 13%|█▎        | 5393/41250 [13:02:13<86:11:11,  8.65s/it]
                                                          
{'loss': 0.1283, 'grad_norm': 1.6437636613845825, 'learning_rate': 3.8945112724071784e-05, 'epoch': 1.31}

 13%|█▎        | 5393/41250 [13:02:13<86:11:11,  8.65s/it][2025-04-25 20:59:57,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.18
[2025-04-25 20:59:57,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.66 | bwd_microstep: 5717.86 | bwd_inner_microstep: 5705.23 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.65
[2025-04-25 20:59:57,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.66 | bwd: 5717.88 | bwd_inner: 5705.23 | bwd_allreduce: 12.61 | step: 19.65

 13%|█▎        | 5394/41250 [13:02:22<86:12:07,  8.65s/it]
                                                          
{'loss': 0.0715, 'grad_norm': 1.340706467628479, 'learning_rate': 3.894460940948963e-05, 'epoch': 1.31}

 13%|█▎        | 5394/41250 [13:02:22<86:12:07,  8.65s/it][2025-04-25 21:00:05,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 21:00:05,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.11 | bwd_microstep: 5777.81 | bwd_inner_microstep: 5699.36 | bwd_allreduce_microstep: 78.40 | step_microstep: 18.76
[2025-04-25 21:00:05,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.10 | bwd: 5777.82 | bwd_inner: 5699.36 | bwd_allreduce: 78.42 | step: 18.77

 13%|█▎        | 5395/41250 [13:02:31<86:22:12,  8.67s/it]
                                                          
{'loss': 0.0379, 'grad_norm': 0.39968129992485046, 'learning_rate': 3.8944105978117685e-05, 'epoch': 1.31}

 13%|█▎        | 5395/41250 [13:02:31<86:22:12,  8.67s/it][2025-04-25 21:00:14,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:00:14,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.88 | bwd_microstep: 5739.60 | bwd_inner_microstep: 5705.73 | bwd_allreduce_microstep: 33.83 | step_microstep: 18.77
[2025-04-25 21:00:14,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.88 | bwd: 5739.62 | bwd_inner: 5705.73 | bwd_allreduce: 33.85 | step: 18.77

 13%|█▎        | 5396/41250 [13:02:39<86:23:29,  8.67s/it]
                                                          
{'loss': 0.157, 'grad_norm': 2.3863630294799805, 'learning_rate': 3.894360242995905e-05, 'epoch': 1.31}

 13%|█▎        | 5396/41250 [13:02:39<86:23:29,  8.67s/it][2025-04-25 21:00:23,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:00:23,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.60 | bwd_microstep: 5715.90 | bwd_inner_microstep: 5656.18 | bwd_allreduce_microstep: 59.66 | step_microstep: 18.40
[2025-04-25 21:00:23,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.60 | bwd: 5715.91 | bwd_inner: 5656.18 | bwd_allreduce: 59.68 | step: 18.41

 13%|█▎        | 5397/41250 [13:02:48<86:15:19,  8.66s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.8622846603393555, 'learning_rate': 3.894309876501684e-05, 'epoch': 1.31}

 13%|█▎        | 5397/41250 [13:02:48<86:15:19,  8.66s/it][2025-04-25 21:00:31,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 21:00:31,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.66 | bwd_microstep: 5758.85 | bwd_inner_microstep: 5702.12 | bwd_allreduce_microstep: 56.67 | step_microstep: 19.01
[2025-04-25 21:00:31,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.66 | bwd: 5758.86 | bwd_inner: 5702.12 | bwd_allreduce: 56.69 | step: 19.01

 13%|█▎        | 5398/41250 [13:02:57<86:21:58,  8.67s/it]
                                                          
{'loss': 0.2071, 'grad_norm': 3.9497554302215576, 'learning_rate': 3.894259498329415e-05, 'epoch': 1.31}

 13%|█▎        | 5398/41250 [13:02:57<86:21:58,  8.67s/it][2025-04-25 21:00:40,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 21:00:40,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.27 | bwd_microstep: 5755.96 | bwd_inner_microstep: 5708.15 | bwd_allreduce_microstep: 47.75 | step_microstep: 19.09
[2025-04-25 21:00:40,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.27 | bwd: 5755.97 | bwd_inner: 5708.15 | bwd_allreduce: 47.77 | step: 19.09

 13%|█▎        | 5399/41250 [13:03:05<86:25:49,  8.68s/it]
                                                          
{'loss': 0.1596, 'grad_norm': 2.5982367992401123, 'learning_rate': 3.894209108479408e-05, 'epoch': 1.31}

 13%|█▎        | 5399/41250 [13:03:05<86:25:49,  8.68s/it][2025-04-25 21:00:49,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 21:00:49,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5726.54 | bwd_inner_microstep: 5713.68 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.48
[2025-04-25 21:00:49,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5726.55 | bwd_inner: 5713.68 | bwd_allreduce: 12.83 | step: 19.48

 13%|█▎        | 5400/41250 [13:03:14<86:23:58,  8.68s/it]
                                                          
{'loss': 0.046, 'grad_norm': 0.7645642161369324, 'learning_rate': 3.8941587069519746e-05, 'epoch': 1.31}

 13%|█▎        | 5400/41250 [13:03:14<86:23:58,  8.68s/it][2025-04-25 21:00:57,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.06 | optimizer_step: 0.94
[2025-04-25 21:00:57,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.75 | bwd_microstep: 5759.09 | bwd_inner_microstep: 5718.15 | bwd_allreduce_microstep: 40.90 | step_microstep: 18.72
[2025-04-25 21:00:57,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.75 | bwd: 5759.11 | bwd_inner: 5718.15 | bwd_allreduce: 40.92 | step: 18.72

 13%|█▎        | 5401/41250 [13:03:23<86:27:39,  8.68s/it]
                                                          
{'loss': 0.1107, 'grad_norm': 5.893352508544922, 'learning_rate': 3.8941082937474256e-05, 'epoch': 1.31}

 13%|█▎        | 5401/41250 [13:03:23<86:27:39,  8.68s/it][2025-04-25 21:01:06,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:01:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5773.61 | bwd_inner_microstep: 5695.00 | bwd_allreduce_microstep: 78.57 | step_microstep: 18.76
[2025-04-25 21:01:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5773.63 | bwd_inner: 5695.00 | bwd_allreduce: 78.59 | step: 18.77

 13%|█▎        | 5402/41250 [13:03:32<86:31:54,  8.69s/it]
                                                          
{'loss': 0.0988, 'grad_norm': 1.104246973991394, 'learning_rate': 3.894057868866071e-05, 'epoch': 1.31}

 13%|█▎        | 5402/41250 [13:03:32<86:31:54,  8.69s/it][2025-04-25 21:01:15,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:01:15,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5736.29 | bwd_inner_microstep: 5723.87 | bwd_allreduce_microstep: 12.37 | step_microstep: 18.91
[2025-04-25 21:01:15,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5736.30 | bwd_inner: 5723.87 | bwd_allreduce: 12.38 | step: 18.91

 13%|█▎        | 5403/41250 [13:03:40<86:28:53,  8.69s/it]
                                                          
{'loss': 0.1091, 'grad_norm': 4.1121625900268555, 'learning_rate': 3.8940074323082224e-05, 'epoch': 1.31}

 13%|█▎        | 5403/41250 [13:03:40<86:28:53,  8.69s/it][2025-04-25 21:01:24,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.02
[2025-04-25 21:01:24,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.32 | bwd_microstep: 5779.00 | bwd_inner_microstep: 5726.32 | bwd_allreduce_microstep: 52.63 | step_microstep: 19.15
[2025-04-25 21:01:24,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.32 | bwd: 5779.02 | bwd_inner: 5726.32 | bwd_allreduce: 52.66 | step: 19.15

 13%|█▎        | 5404/41250 [13:03:49<86:34:58,  8.70s/it]
                                                          
{'loss': 0.268, 'grad_norm': 1.5505865812301636, 'learning_rate': 3.8939569840741916e-05, 'epoch': 1.31}

 13%|█▎        | 5404/41250 [13:03:49<86:34:58,  8.70s/it][2025-04-25 21:01:32,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 1.02
[2025-04-25 21:01:32,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.09 | bwd_microstep: 5714.11 | bwd_inner_microstep: 5701.68 | bwd_allreduce_microstep: 12.38 | step_microstep: 19.66
[2025-04-25 21:01:32,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.09 | bwd: 5714.12 | bwd_inner: 5701.68 | bwd_allreduce: 12.40 | step: 19.66

 13%|█▎        | 5405/41250 [13:03:58<86:28:47,  8.69s/it]
                                                          
{'loss': 0.2592, 'grad_norm': 4.284611225128174, 'learning_rate': 3.893906524164288e-05, 'epoch': 1.31}

 13%|█▎        | 5405/41250 [13:03:58<86:28:47,  8.69s/it][2025-04-25 21:01:41,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 21:01:41,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.38 | bwd_microstep: 5706.51 | bwd_inner_microstep: 5693.49 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.21
[2025-04-25 21:01:41,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.38 | bwd: 5706.53 | bwd_inner: 5693.49 | bwd_allreduce: 12.99 | step: 19.22

 13%|█▎        | 5406/41250 [13:04:06<86:21:33,  8.67s/it]
                                                          
{'loss': 0.1396, 'grad_norm': 1.4438618421554565, 'learning_rate': 3.893856052578822e-05, 'epoch': 1.31}

 13%|█▎        | 5406/41250 [13:04:06<86:21:33,  8.67s/it][2025-04-25 21:01:50,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 21:01:50,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.91 | bwd_microstep: 5729.91 | bwd_inner_microstep: 5663.47 | bwd_allreduce_microstep: 66.40 | step_microstep: 18.92
[2025-04-25 21:01:50,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.91 | bwd: 5729.93 | bwd_inner: 5663.47 | bwd_allreduce: 66.42 | step: 18.93

 13%|█▎        | 5407/41250 [13:04:15<86:17:01,  8.67s/it]
                                                          
{'loss': 0.0766, 'grad_norm': 0.9799110293388367, 'learning_rate': 3.893805569318107e-05, 'epoch': 1.31}

 13%|█▎        | 5407/41250 [13:04:15<86:17:01,  8.67s/it][2025-04-25 21:01:58,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:01:58,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.32 | bwd_microstep: 5772.15 | bwd_inner_microstep: 5657.07 | bwd_allreduce_microstep: 115.03 | step_microstep: 18.82
[2025-04-25 21:01:58,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.32 | bwd: 5772.16 | bwd_inner: 5657.07 | bwd_allreduce: 115.05 | step: 18.82

 13%|█▎        | 5408/41250 [13:04:24<86:20:21,  8.67s/it]
                                                          
{'loss': 0.2607, 'grad_norm': 2.6129093170166016, 'learning_rate': 3.893755074382453e-05, 'epoch': 1.31}

 13%|█▎        | 5408/41250 [13:04:24<86:20:21,  8.67s/it][2025-04-25 21:02:07,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:02:07,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.94 | bwd_microstep: 5768.68 | bwd_inner_microstep: 5709.87 | bwd_allreduce_microstep: 58.75 | step_microstep: 18.84
[2025-04-25 21:02:07,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.94 | bwd: 5768.69 | bwd_inner: 5709.87 | bwd_allreduce: 58.78 | step: 18.84

 13%|█▎        | 5409/41250 [13:04:32<86:27:28,  8.68s/it]
                                                          
{'loss': 0.2248, 'grad_norm': 1.9308488368988037, 'learning_rate': 3.893704567772171e-05, 'epoch': 1.31}

 13%|█▎        | 5409/41250 [13:04:32<86:27:28,  8.68s/it][2025-04-25 21:02:16,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:02:16,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.98 | bwd_microstep: 5725.19 | bwd_inner_microstep: 5658.50 | bwd_allreduce_microstep: 66.64 | step_microstep: 18.90
[2025-04-25 21:02:16,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.98 | bwd: 5725.21 | bwd_inner: 5658.50 | bwd_allreduce: 66.66 | step: 18.90

 13%|█▎        | 5410/41250 [13:04:41<86:18:41,  8.67s/it]
                                                          
{'loss': 0.0797, 'grad_norm': 1.3268511295318604, 'learning_rate': 3.893654049487574e-05, 'epoch': 1.31}

 13%|█▎        | 5410/41250 [13:04:41<86:18:41,  8.67s/it][2025-04-25 21:02:24,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.24 | optimizer_step: 1.08
[2025-04-25 21:02:24,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.74 | bwd_microstep: 5905.29 | bwd_inner_microstep: 5695.86 | bwd_allreduce_microstep: 209.38 | step_microstep: 19.90
[2025-04-25 21:02:24,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.74 | bwd: 5905.32 | bwd_inner: 5695.86 | bwd_allreduce: 209.40 | step: 19.90

 13%|█▎        | 5411/41250 [13:04:50<86:49:35,  8.72s/it]
                                                          
{'loss': 0.2004, 'grad_norm': 1.7883166074752808, 'learning_rate': 3.893603519528971e-05, 'epoch': 1.31}

 13%|█▎        | 5411/41250 [13:04:50<86:49:35,  8.72s/it][2025-04-25 21:02:33,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 21:02:33,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.89 | bwd_microstep: 5795.18 | bwd_inner_microstep: 5648.38 | bwd_allreduce_microstep: 146.76 | step_microstep: 19.13
[2025-04-25 21:02:33,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.89 | bwd: 5795.20 | bwd_inner: 5648.38 | bwd_allreduce: 146.78 | step: 19.13

 13%|█▎        | 5412/41250 [13:04:58<86:46:13,  8.72s/it]
                                                          
{'loss': 0.046, 'grad_norm': 1.0099287033081055, 'learning_rate': 3.8935529778966755e-05, 'epoch': 1.31}

 13%|█▎        | 5412/41250 [13:04:58<86:46:13,  8.72s/it][2025-04-25 21:02:42,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 21:02:42,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.72 | bwd_microstep: 5703.49 | bwd_inner_microstep: 5689.73 | bwd_allreduce_microstep: 13.71 | step_microstep: 18.78
[2025-04-25 21:02:42,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.72 | bwd: 5703.50 | bwd_inner: 5689.73 | bwd_allreduce: 13.72 | step: 18.79

 13%|█▎        | 5413/41250 [13:05:07<86:30:46,  8.69s/it]
                                                          
{'loss': 0.0797, 'grad_norm': 1.1743543148040771, 'learning_rate': 3.893502424590998e-05, 'epoch': 1.31}

 13%|█▎        | 5413/41250 [13:05:07<86:30:46,  8.69s/it][2025-04-25 21:02:50,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:02:50,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.27 | bwd_microstep: 5784.66 | bwd_inner_microstep: 5653.67 | bwd_allreduce_microstep: 130.95 | step_microstep: 18.85
[2025-04-25 21:02:50,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.27 | bwd: 5784.68 | bwd_inner: 5653.67 | bwd_allreduce: 130.97 | step: 18.85

 13%|█▎        | 5414/41250 [13:05:16<86:32:43,  8.69s/it]
                                                          
{'loss': 0.1451, 'grad_norm': 3.203889846801758, 'learning_rate': 3.8934518596122504e-05, 'epoch': 1.31}

 13%|█▎        | 5414/41250 [13:05:16<86:32:43,  8.69s/it][2025-04-25 21:02:59,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.95 | optimizer_step: 1.06
[2025-04-25 21:02:59,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.55 | bwd_microstep: 5796.61 | bwd_inner_microstep: 5645.29 | bwd_allreduce_microstep: 151.27 | step_microstep: 18.41
[2025-04-25 21:02:59,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.55 | bwd: 5796.62 | bwd_inner: 5645.29 | bwd_allreduce: 151.29 | step: 18.41

 13%|█▎        | 5415/41250 [13:05:24<86:33:59,  8.70s/it]
                                                          
{'loss': 0.0809, 'grad_norm': 1.5897918939590454, 'learning_rate': 3.893401282960745e-05, 'epoch': 1.31}

 13%|█▎        | 5415/41250 [13:05:24<86:33:59,  8.70s/it][2025-04-25 21:03:08,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 21:03:08,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.76 | bwd_microstep: 5702.56 | bwd_inner_microstep: 5689.94 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.61
[2025-04-25 21:03:08,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.76 | bwd: 5702.57 | bwd_inner: 5689.94 | bwd_allreduce: 12.59 | step: 18.62

 13%|█▎        | 5416/41250 [13:05:33<86:21:22,  8.68s/it]
                                                          
{'loss': 0.0206, 'grad_norm': 0.2628147602081299, 'learning_rate': 3.893350694636792e-05, 'epoch': 1.31}

 13%|█▎        | 5416/41250 [13:05:33<86:21:22,  8.68s/it][2025-04-25 21:03:16,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 21:03:16,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.81 | bwd_microstep: 5729.70 | bwd_inner_microstep: 5702.48 | bwd_allreduce_microstep: 27.16 | step_microstep: 18.90
[2025-04-25 21:03:16,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.81 | bwd: 5729.71 | bwd_inner: 5702.48 | bwd_allreduce: 27.18 | step: 18.91

 13%|█▎        | 5417/41250 [13:05:42<86:18:52,  8.67s/it]
                                                          
{'loss': 0.0899, 'grad_norm': 3.6997830867767334, 'learning_rate': 3.893300094640705e-05, 'epoch': 1.31}

 13%|█▎        | 5417/41250 [13:05:42<86:18:52,  8.67s/it][2025-04-25 21:03:25,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 21:03:25,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.58 | bwd_microstep: 5711.25 | bwd_inner_microstep: 5698.07 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.08
[2025-04-25 21:03:25,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.58 | bwd: 5711.27 | bwd_inner: 5698.07 | bwd_allreduce: 13.16 | step: 19.08

 13%|█▎        | 5418/41250 [13:05:50<86:14:01,  8.66s/it]
                                                          
{'loss': 0.2456, 'grad_norm': 3.1421515941619873, 'learning_rate': 3.893249482972796e-05, 'epoch': 1.31}

 13%|█▎        | 5418/41250 [13:05:50<86:14:01,  8.66s/it][2025-04-25 21:03:34,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 21:03:34,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.73 | bwd_microstep: 5727.48 | bwd_inner_microstep: 5705.54 | bwd_allreduce_microstep: 21.89 | step_microstep: 18.64
[2025-04-25 21:03:34,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.73 | bwd: 5727.49 | bwd_inner: 5705.54 | bwd_allreduce: 21.91 | step: 18.64

 13%|█▎        | 5419/41250 [13:05:59<86:13:33,  8.66s/it]
                                                          
{'loss': 0.0283, 'grad_norm': 0.6052314043045044, 'learning_rate': 3.893198859633375e-05, 'epoch': 1.31}

 13%|█▎        | 5419/41250 [13:05:59<86:13:33,  8.66s/it][2025-04-25 21:03:42,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 21:03:42,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.37 | bwd_microstep: 5767.96 | bwd_inner_microstep: 5755.23 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.76
[2025-04-25 21:03:42,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.37 | bwd: 5767.98 | bwd_inner: 5755.23 | bwd_allreduce: 12.70 | step: 18.77

 13%|█▎        | 5420/41250 [13:06:08<86:25:28,  8.68s/it]
                                                          
{'loss': 0.0813, 'grad_norm': 2.1148884296417236, 'learning_rate': 3.893148224622757e-05, 'epoch': 1.31}

 13%|█▎        | 5420/41250 [13:06:08<86:25:28,  8.68s/it][2025-04-25 21:03:51,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 21:03:51,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.04 | bwd_microstep: 5706.03 | bwd_inner_microstep: 5642.42 | bwd_allreduce_microstep: 63.56 | step_microstep: 18.81
[2025-04-25 21:03:51,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.04 | bwd: 5706.05 | bwd_inner: 5642.42 | bwd_allreduce: 63.58 | step: 18.81

 13%|█▎        | 5421/41250 [13:06:16<86:11:44,  8.66s/it]
                                                          
{'loss': 0.1193, 'grad_norm': 4.322851181030273, 'learning_rate': 3.8930975779412513e-05, 'epoch': 1.31}

 13%|█▎        | 5421/41250 [13:06:16<86:11:44,  8.66s/it][2025-04-25 21:04:00,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:04:00,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.08 | bwd_microstep: 5708.00 | bwd_inner_microstep: 5695.23 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.83
[2025-04-25 21:04:00,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.08 | bwd: 5708.01 | bwd_inner: 5695.23 | bwd_allreduce: 12.73 | step: 18.83

 13%|█▎        | 5422/41250 [13:06:25<86:10:04,  8.66s/it]
                                                          
{'loss': 0.2472, 'grad_norm': 3.3339078426361084, 'learning_rate': 3.8930469195891715e-05, 'epoch': 1.31}

 13%|█▎        | 5422/41250 [13:06:25<86:10:04,  8.66s/it][2025-04-25 21:04:08,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 21:04:08,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.90 | bwd_microstep: 5717.80 | bwd_inner_microstep: 5648.63 | bwd_allreduce_microstep: 69.12 | step_microstep: 19.28
[2025-04-25 21:04:08,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.90 | bwd: 5717.81 | bwd_inner: 5648.63 | bwd_allreduce: 69.14 | step: 19.28

 13%|█▎        | 5423/41250 [13:06:34<86:03:10,  8.65s/it]
                                                          
{'loss': 0.03, 'grad_norm': 0.7398542761802673, 'learning_rate': 3.892996249566831e-05, 'epoch': 1.31}

 13%|█▎        | 5423/41250 [13:06:34<86:03:10,  8.65s/it][2025-04-25 21:04:17,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:04:17,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 5729.73 | bwd_inner_microstep: 5685.15 | bwd_allreduce_microstep: 44.53 | step_microstep: 18.68
[2025-04-25 21:04:17,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.18 | bwd: 5729.74 | bwd_inner: 5685.15 | bwd_allreduce: 44.55 | step: 18.67

 13%|█▎        | 5424/41250 [13:06:42<86:04:58,  8.65s/it]
                                                          
{'loss': 0.1605, 'grad_norm': 1.0949814319610596, 'learning_rate': 3.8929455678745394e-05, 'epoch': 1.31}

 13%|█▎        | 5424/41250 [13:06:42<86:04:58,  8.65s/it][2025-04-25 21:04:26,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:04:26,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.08 | bwd_microstep: 5842.04 | bwd_inner_microstep: 5700.39 | bwd_allreduce_microstep: 141.61 | step_microstep: 18.51
[2025-04-25 21:04:26,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.08 | bwd: 5842.06 | bwd_inner: 5700.39 | bwd_allreduce: 141.63 | step: 18.51

 13%|█▎        | 5425/41250 [13:06:51<86:26:57,  8.69s/it]
                                                          
{'loss': 0.0875, 'grad_norm': 1.1240589618682861, 'learning_rate': 3.892894874512611e-05, 'epoch': 1.32}

 13%|█▎        | 5425/41250 [13:06:51<86:26:57,  8.69s/it][2025-04-25 21:04:34,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 0.97
[2025-04-25 21:04:34,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.99 | bwd_microstep: 5774.68 | bwd_inner_microstep: 5633.38 | bwd_allreduce_microstep: 141.25 | step_microstep: 18.98
[2025-04-25 21:04:34,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.99 | bwd: 5774.69 | bwd_inner: 5633.38 | bwd_allreduce: 141.27 | step: 18.98

 13%|█▎        | 5426/41250 [13:07:00<86:25:05,  8.68s/it]
                                                          
{'loss': 0.2893, 'grad_norm': 4.271388053894043, 'learning_rate': 3.892844169481359e-05, 'epoch': 1.32}

 13%|█▎        | 5426/41250 [13:07:00<86:25:05,  8.68s/it][2025-04-25 21:04:43,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:04:43,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.09 | bwd_microstep: 5889.24 | bwd_inner_microstep: 5641.32 | bwd_allreduce_microstep: 247.87 | step_microstep: 18.81
[2025-04-25 21:04:43,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.09 | bwd: 5889.25 | bwd_inner: 5641.32 | bwd_allreduce: 247.89 | step: 18.81

 13%|█▎        | 5427/41250 [13:07:09<86:45:24,  8.72s/it]
                                                          
{'loss': 0.0395, 'grad_norm': 1.9438347816467285, 'learning_rate': 3.892793452781094e-05, 'epoch': 1.32}

 13%|█▎        | 5427/41250 [13:07:09<86:45:24,  8.72s/it][2025-04-25 21:04:52,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:04:52,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.18 | bwd_microstep: 5738.82 | bwd_inner_microstep: 5682.34 | bwd_allreduce_microstep: 56.44 | step_microstep: 18.45
[2025-04-25 21:04:52,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.18 | bwd: 5738.84 | bwd_inner: 5682.34 | bwd_allreduce: 56.46 | step: 18.45

 13%|█▎        | 5428/41250 [13:07:17<86:36:11,  8.70s/it]
                                                          
{'loss': 0.0405, 'grad_norm': 0.7490013241767883, 'learning_rate': 3.8927427244121295e-05, 'epoch': 1.32}

 13%|█▎        | 5428/41250 [13:07:17<86:36:11,  8.70s/it][2025-04-25 21:05:01,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:05:01,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.68 | bwd_microstep: 5703.04 | bwd_inner_microstep: 5690.14 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.49
[2025-04-25 21:05:01,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.68 | bwd: 5703.05 | bwd_inner: 5690.14 | bwd_allreduce: 12.87 | step: 18.49

 13%|█▎        | 5429/41250 [13:07:26<86:23:32,  8.68s/it]
                                                          
{'loss': 0.0548, 'grad_norm': 1.8502411842346191, 'learning_rate': 3.8926919843747795e-05, 'epoch': 1.32}

 13%|█▎        | 5429/41250 [13:07:26<86:23:32,  8.68s/it][2025-04-25 21:05:09,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:05:09,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.36 | bwd_microstep: 5704.76 | bwd_inner_microstep: 5692.05 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.83
[2025-04-25 21:05:09,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.36 | bwd: 5704.77 | bwd_inner: 5692.05 | bwd_allreduce: 12.68 | step: 18.83

 13%|█▎        | 5430/41250 [13:07:35<86:14:06,  8.67s/it]
                                                          
{'loss': 0.1481, 'grad_norm': 2.1636273860931396, 'learning_rate': 3.892641232669355e-05, 'epoch': 1.32}

 13%|█▎        | 5430/41250 [13:07:35<86:14:06,  8.67s/it][2025-04-25 21:05:18,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 21:05:18,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5780.68 | bwd_inner_microstep: 5646.65 | bwd_allreduce_microstep: 133.98 | step_microstep: 19.10
[2025-04-25 21:05:18,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5780.70 | bwd_inner: 5646.65 | bwd_allreduce: 134.00 | step: 19.11

 13%|█▎        | 5431/41250 [13:07:43<86:17:46,  8.67s/it]
                                                          
{'loss': 0.0907, 'grad_norm': 2.026061773300171, 'learning_rate': 3.8925904692961697e-05, 'epoch': 1.32}

 13%|█▎        | 5431/41250 [13:07:43<86:17:46,  8.67s/it][2025-04-25 21:05:27,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 21:05:27,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.13 | bwd_microstep: 5850.50 | bwd_inner_microstep: 5650.85 | bwd_allreduce_microstep: 199.60 | step_microstep: 19.08
[2025-04-25 21:05:27,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.13 | bwd: 5850.51 | bwd_inner: 5650.85 | bwd_allreduce: 199.62 | step: 19.08

 13%|█▎        | 5432/41250 [13:07:52<86:35:13,  8.70s/it]
                                                          
{'loss': 0.0375, 'grad_norm': 0.48664146661758423, 'learning_rate': 3.892539694255536e-05, 'epoch': 1.32}

 13%|█▎        | 5432/41250 [13:07:52<86:35:13,  8.70s/it][2025-04-25 21:05:35,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 21:05:35,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5709.23 | bwd_inner_microstep: 5684.54 | bwd_allreduce_microstep: 24.64 | step_microstep: 18.87
[2025-04-25 21:05:35,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5709.24 | bwd_inner: 5684.54 | bwd_allreduce: 24.66 | step: 18.87

 13%|█▎        | 5433/41250 [13:08:01<86:25:17,  8.69s/it]
                                                          
{'loss': 0.0341, 'grad_norm': 0.8262473940849304, 'learning_rate': 3.892488907547768e-05, 'epoch': 1.32}

 13%|█▎        | 5433/41250 [13:08:01<86:25:17,  8.69s/it][2025-04-25 21:05:44,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:05:44,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.14 | bwd_microstep: 5763.22 | bwd_inner_microstep: 5643.76 | bwd_allreduce_microstep: 119.42 | step_microstep: 18.60
[2025-04-25 21:05:44,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.14 | bwd: 5763.23 | bwd_inner: 5643.76 | bwd_allreduce: 119.43 | step: 18.60

 13%|█▎        | 5434/41250 [13:08:09<86:22:33,  8.68s/it]
                                                          
{'loss': 0.2434, 'grad_norm': 3.668747901916504, 'learning_rate': 3.8924381091731775e-05, 'epoch': 1.32}

 13%|█▎        | 5434/41250 [13:08:09<86:22:33,  8.68s/it][2025-04-25 21:05:53,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:05:53,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.91 | bwd_microstep: 5679.54 | bwd_inner_microstep: 5643.87 | bwd_allreduce_microstep: 35.63 | step_microstep: 18.74
[2025-04-25 21:05:53,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.91 | bwd: 5679.56 | bwd_inner: 5643.87 | bwd_allreduce: 35.65 | step: 18.74

 13%|█▎        | 5435/41250 [13:08:18<86:07:28,  8.66s/it]
                                                          
{'loss': 0.3185, 'grad_norm': 5.491805553436279, 'learning_rate': 3.892387299132079e-05, 'epoch': 1.32}

 13%|█▎        | 5435/41250 [13:08:18<86:07:28,  8.66s/it][2025-04-25 21:06:01,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:06:01,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.23 | bwd_microstep: 5751.01 | bwd_inner_microstep: 5682.65 | bwd_allreduce_microstep: 68.31 | step_microstep: 18.53
[2025-04-25 21:06:01,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.23 | bwd: 5751.03 | bwd_inner: 5682.65 | bwd_allreduce: 68.33 | step: 18.53

 13%|█▎        | 5436/41250 [13:08:27<86:14:19,  8.67s/it]
                                                          
{'loss': 0.3172, 'grad_norm': 2.7136361598968506, 'learning_rate': 3.892336477424784e-05, 'epoch': 1.32}

 13%|█▎        | 5436/41250 [13:08:27<86:14:19,  8.67s/it][2025-04-25 21:06:10,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 21:06:10,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5727.73 | bwd_inner_microstep: 5686.72 | bwd_allreduce_microstep: 40.96 | step_microstep: 18.96
[2025-04-25 21:06:10,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5727.74 | bwd_inner: 5686.72 | bwd_allreduce: 40.98 | step: 18.96

 13%|█▎        | 5437/41250 [13:08:35<86:13:18,  8.67s/it]
                                                          
{'loss': 0.0737, 'grad_norm': 1.2436856031417847, 'learning_rate': 3.892285644051607e-05, 'epoch': 1.32}

 13%|█▎        | 5437/41250 [13:08:35<86:13:18,  8.67s/it][2025-04-25 21:06:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 21:06:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.73 | bwd_microstep: 5694.78 | bwd_inner_microstep: 5647.73 | bwd_allreduce_microstep: 47.01 | step_microstep: 18.68
[2025-04-25 21:06:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.73 | bwd: 5694.79 | bwd_inner: 5647.73 | bwd_allreduce: 47.02 | step: 18.68

 13%|█▎        | 5438/41250 [13:08:44<86:00:56,  8.65s/it]
                                                          
{'loss': 0.1401, 'grad_norm': 3.042322874069214, 'learning_rate': 3.892234799012862e-05, 'epoch': 1.32}

 13%|█▎        | 5438/41250 [13:08:44<86:00:56,  8.65s/it][2025-04-25 21:06:27,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:06:27,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.48 | bwd_microstep: 5771.10 | bwd_inner_microstep: 5758.27 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.62
[2025-04-25 21:06:27,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.48 | bwd: 5771.11 | bwd_inner: 5758.27 | bwd_allreduce: 12.79 | step: 18.62

 13%|█▎        | 5439/41250 [13:08:53<86:16:53,  8.67s/it]
                                                          
{'loss': 0.1019, 'grad_norm': 2.8826181888580322, 'learning_rate': 3.892183942308862e-05, 'epoch': 1.32}

 13%|█▎        | 5439/41250 [13:08:53<86:16:53,  8.67s/it][2025-04-25 21:06:36,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:06:36,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.12 | bwd_microstep: 5724.41 | bwd_inner_microstep: 5711.62 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.85
[2025-04-25 21:06:36,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.12 | bwd: 5724.43 | bwd_inner: 5711.62 | bwd_allreduce: 12.76 | step: 18.86

 13%|█▎        | 5440/41250 [13:09:01<86:16:26,  8.67s/it]
                                                          
{'loss': 0.0624, 'grad_norm': 1.2844380140304565, 'learning_rate': 3.892133073939919e-05, 'epoch': 1.32}

 13%|█▎        | 5440/41250 [13:09:01<86:16:26,  8.67s/it][2025-04-25 21:06:45,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:06:45,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.10 | bwd_microstep: 5780.51 | bwd_inner_microstep: 5665.29 | bwd_allreduce_microstep: 115.17 | step_microstep: 18.67
[2025-04-25 21:06:45,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.10 | bwd: 5780.52 | bwd_inner: 5665.29 | bwd_allreduce: 115.18 | step: 18.67

 13%|█▎        | 5441/41250 [13:09:10<86:24:06,  8.69s/it]
                                                          
{'loss': 0.0557, 'grad_norm': 0.9871096014976501, 'learning_rate': 3.8920821939063476e-05, 'epoch': 1.32}

 13%|█▎        | 5441/41250 [13:09:10<86:24:06,  8.69s/it][2025-04-25 21:06:53,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 21:06:53,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.21 | bwd_microstep: 5702.17 | bwd_inner_microstep: 5650.24 | bwd_allreduce_microstep: 51.89 | step_microstep: 19.16
[2025-04-25 21:06:53,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.21 | bwd: 5702.19 | bwd_inner: 5650.24 | bwd_allreduce: 51.91 | step: 19.16

 13%|█▎        | 5442/41250 [13:09:19<86:10:13,  8.66s/it]
                                                          
{'loss': 0.1111, 'grad_norm': 3.059678554534912, 'learning_rate': 3.8920313022084625e-05, 'epoch': 1.32}

 13%|█▎        | 5442/41250 [13:09:19<86:10:13,  8.66s/it][2025-04-25 21:07:02,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:07:02,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.42 | bwd_microstep: 6024.77 | bwd_inner_microstep: 5661.50 | bwd_allreduce_microstep: 363.22 | step_microstep: 18.47
[2025-04-25 21:07:02,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.42 | bwd: 6024.78 | bwd_inner: 5661.50 | bwd_allreduce: 363.24 | step: 18.48

 13%|█▎        | 5443/41250 [13:09:28<86:59:19,  8.75s/it]
                                                          
{'loss': 0.0967, 'grad_norm': 1.5255939960479736, 'learning_rate': 3.891980398846576e-05, 'epoch': 1.32}

 13%|█▎        | 5443/41250 [13:09:28<86:59:19,  8.75s/it][2025-04-25 21:07:11,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:07:11,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.85 | bwd_microstep: 5763.02 | bwd_inner_microstep: 5660.96 | bwd_allreduce_microstep: 102.01 | step_microstep: 18.83
[2025-04-25 21:07:11,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.85 | bwd: 5763.04 | bwd_inner: 5660.96 | bwd_allreduce: 102.03 | step: 18.83

 13%|█▎        | 5444/41250 [13:09:36<86:48:40,  8.73s/it]
                                                          
{'loss': 0.2959, 'grad_norm': 2.8076705932617188, 'learning_rate': 3.891929483821003e-05, 'epoch': 1.32}

 13%|█▎        | 5444/41250 [13:09:36<86:48:40,  8.73s/it][2025-04-25 21:07:20,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:07:20,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.69 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5705.14 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.46
[2025-04-25 21:07:20,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.69 | bwd: 5717.95 | bwd_inner: 5705.14 | bwd_allreduce: 12.78 | step: 18.46

 13%|█▎        | 5445/41250 [13:09:45<86:36:30,  8.71s/it]
                                                          
{'loss': 0.1582, 'grad_norm': 2.3650522232055664, 'learning_rate': 3.891878557132057e-05, 'epoch': 1.32}

 13%|█▎        | 5445/41250 [13:09:45<86:36:30,  8.71s/it][2025-04-25 21:07:28,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 21:07:28,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5775.77 | bwd_inner_microstep: 5670.67 | bwd_allreduce_microstep: 105.05 | step_microstep: 18.52
[2025-04-25 21:07:28,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5775.78 | bwd_inner: 5670.67 | bwd_allreduce: 105.07 | step: 18.52

 13%|█▎        | 5446/41250 [13:09:54<86:33:16,  8.70s/it]
                                                          
{'loss': 0.1108, 'grad_norm': 7.728677272796631, 'learning_rate': 3.891827618780051e-05, 'epoch': 1.32}

 13%|█▎        | 5446/41250 [13:09:54<86:33:16,  8.70s/it][2025-04-25 21:07:37,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-25 21:07:37,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5748.35 | bwd_inner_microstep: 5703.14 | bwd_allreduce_microstep: 45.17 | step_microstep: 19.31
[2025-04-25 21:07:37,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5748.37 | bwd_inner: 5703.14 | bwd_allreduce: 45.19 | step: 19.32

 13%|█▎        | 5447/41250 [13:10:02<86:30:52,  8.70s/it]
                                                          
{'loss': 0.0156, 'grad_norm': 0.22974760830402374, 'learning_rate': 3.8917766687653e-05, 'epoch': 1.32}

 13%|█▎        | 5447/41250 [13:10:02<86:30:52,  8.70s/it][2025-04-25 21:07:46,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:07:46,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5762.59 | bwd_inner_microstep: 5673.32 | bwd_allreduce_microstep: 89.23 | step_microstep: 18.36
[2025-04-25 21:07:46,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5762.60 | bwd_inner: 5673.32 | bwd_allreduce: 89.24 | step: 18.36

 13%|█▎        | 5448/41250 [13:10:11<86:28:17,  8.69s/it]
                                                          
{'loss': 0.0331, 'grad_norm': 0.7451435923576355, 'learning_rate': 3.8917257070881184e-05, 'epoch': 1.32}

 13%|█▎        | 5448/41250 [13:10:11<86:28:17,  8.69s/it][2025-04-25 21:07:54,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 21:07:54,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.76 | bwd_microstep: 5795.73 | bwd_inner_microstep: 5783.04 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.53
[2025-04-25 21:07:54,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.76 | bwd: 5795.74 | bwd_inner: 5783.04 | bwd_allreduce: 12.66 | step: 18.53

 13%|█▎        | 5449/41250 [13:10:20<86:42:49,  8.72s/it]
                                                          
{'loss': 0.3177, 'grad_norm': 3.7769882678985596, 'learning_rate': 3.89167473374882e-05, 'epoch': 1.32}

 13%|█▎        | 5449/41250 [13:10:20<86:42:49,  8.72s/it][2025-04-25 21:08:03,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:08:03,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.50 | bwd_microstep: 5714.24 | bwd_inner_microstep: 5701.72 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.72
[2025-04-25 21:08:03,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.50 | bwd: 5714.26 | bwd_inner: 5701.72 | bwd_allreduce: 12.50 | step: 18.72

 13%|█▎        | 5450/41250 [13:10:28<86:31:36,  8.70s/it]
                                                          
{'loss': 0.1553, 'grad_norm': 2.403780698776245, 'learning_rate': 3.891623748747718e-05, 'epoch': 1.32}

 13%|█▎        | 5450/41250 [13:10:28<86:31:36,  8.70s/it][2025-04-25 21:08:12,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-25 21:08:12,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.77 | bwd_microstep: 5870.91 | bwd_inner_microstep: 5672.49 | bwd_allreduce_microstep: 198.36 | step_microstep: 19.39
[2025-04-25 21:08:12,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.77 | bwd: 5870.92 | bwd_inner: 5672.49 | bwd_allreduce: 198.38 | step: 19.39

 13%|█▎        | 5451/41250 [13:10:37<86:50:00,  8.73s/it]
                                                          
{'loss': 0.0495, 'grad_norm': 1.1586625576019287, 'learning_rate': 3.891572752085128e-05, 'epoch': 1.32}

 13%|█▎        | 5451/41250 [13:10:37<86:50:00,  8.73s/it][2025-04-25 21:08:21,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:08:21,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.49 | bwd_microstep: 5754.79 | bwd_inner_microstep: 5699.36 | bwd_allreduce_microstep: 55.38 | step_microstep: 18.71
[2025-04-25 21:08:21,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.49 | bwd: 5754.81 | bwd_inner: 5699.36 | bwd_allreduce: 55.40 | step: 18.71

 13%|█▎        | 5452/41250 [13:10:46<86:44:02,  8.72s/it]
                                                          
{'loss': 0.064, 'grad_norm': 1.7462748289108276, 'learning_rate': 3.8915217437613646e-05, 'epoch': 1.32}

 13%|█▎        | 5452/41250 [13:10:46<86:44:02,  8.72s/it][2025-04-25 21:08:29,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 21:08:29,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5796.57 | bwd_inner_microstep: 5705.76 | bwd_allreduce_microstep: 90.75 | step_microstep: 18.98
[2025-04-25 21:08:29,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5796.58 | bwd_inner: 5705.76 | bwd_allreduce: 90.77 | step: 18.98

 13%|█▎        | 5453/41250 [13:10:55<86:46:46,  8.73s/it]
                                                          
{'loss': 0.0203, 'grad_norm': 0.2604392170906067, 'learning_rate': 3.891470723776741e-05, 'epoch': 1.32}

 13%|█▎        | 5453/41250 [13:10:55<86:46:46,  8.73s/it][2025-04-25 21:08:38,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.09 | optimizer_step: 1.03
[2025-04-25 21:08:38,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.30 | bwd_microstep: 5804.25 | bwd_inner_microstep: 5663.68 | bwd_allreduce_microstep: 140.52 | step_microstep: 19.66
[2025-04-25 21:08:38,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.30 | bwd: 5804.26 | bwd_inner: 5663.68 | bwd_allreduce: 140.54 | step: 19.67

 13%|█▎        | 5454/41250 [13:11:03<86:45:01,  8.72s/it]
                                                          
{'loss': 0.0509, 'grad_norm': 0.7223865389823914, 'learning_rate': 3.891419692131573e-05, 'epoch': 1.32}

 13%|█▎        | 5454/41250 [13:11:03<86:45:01,  8.72s/it][2025-04-25 21:08:47,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 21:08:47,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.56 | bwd_microstep: 5805.93 | bwd_inner_microstep: 5792.88 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.99
[2025-04-25 21:08:47,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.56 | bwd: 5805.95 | bwd_inner: 5792.88 | bwd_allreduce: 13.03 | step: 18.99

 13%|█▎        | 5455/41250 [13:11:12<86:55:20,  8.74s/it]
                                                          
{'loss': 0.0262, 'grad_norm': 0.47586020827293396, 'learning_rate': 3.891368648826174e-05, 'epoch': 1.32}

 13%|█▎        | 5455/41250 [13:11:12<86:55:20,  8.74s/it][2025-04-25 21:08:55,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 21:08:55,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.38 | bwd_microstep: 5752.98 | bwd_inner_microstep: 5698.28 | bwd_allreduce_microstep: 54.64 | step_microstep: 19.02
[2025-04-25 21:08:55,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.38 | bwd: 5753.00 | bwd_inner: 5698.28 | bwd_allreduce: 54.66 | step: 19.02

 13%|█▎        | 5456/41250 [13:11:21<86:46:17,  8.73s/it]
                                                          
{'loss': 0.1823, 'grad_norm': 2.898010730743408, 'learning_rate': 3.8913175938608603e-05, 'epoch': 1.32}

 13%|█▎        | 5456/41250 [13:11:21<86:46:17,  8.73s/it][2025-04-25 21:09:04,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 21:09:04,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.84 | bwd_microstep: 5782.42 | bwd_inner_microstep: 5769.59 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.45
[2025-04-25 21:09:04,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.84 | bwd: 5782.44 | bwd_inner: 5769.59 | bwd_allreduce: 12.80 | step: 19.45

 13%|█▎        | 5457/41250 [13:11:30<86:51:08,  8.74s/it]
                                                          
{'loss': 0.1782, 'grad_norm': 1.8374974727630615, 'learning_rate': 3.891266527235945e-05, 'epoch': 1.32}

 13%|█▎        | 5457/41250 [13:11:30<86:51:08,  8.74s/it][2025-04-25 21:09:13,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-25 21:09:13,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.88 | bwd_microstep: 5755.48 | bwd_inner_microstep: 5717.07 | bwd_allreduce_microstep: 38.35 | step_microstep: 19.42
[2025-04-25 21:09:13,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.89 | bwd: 5755.49 | bwd_inner: 5717.07 | bwd_allreduce: 38.38 | step: 19.42

 13%|█▎        | 5458/41250 [13:11:38<86:43:45,  8.72s/it]
                                                          
{'loss': 0.3303, 'grad_norm': 2.472928047180176, 'learning_rate': 3.8912154489517434e-05, 'epoch': 1.32}

 13%|█▎        | 5458/41250 [13:11:38<86:43:45,  8.72s/it][2025-04-25 21:09:22,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:09:22,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.95 | bwd_microstep: 5778.76 | bwd_inner_microstep: 5669.16 | bwd_allreduce_microstep: 109.55 | step_microstep: 18.68
[2025-04-25 21:09:22,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.95 | bwd: 5778.78 | bwd_inner: 5669.16 | bwd_allreduce: 109.57 | step: 18.68

 13%|█▎        | 5459/41250 [13:11:47<86:39:16,  8.72s/it]
                                                          
{'loss': 0.0975, 'grad_norm': 3.11958909034729, 'learning_rate': 3.8911643590085706e-05, 'epoch': 1.32}

 13%|█▎        | 5459/41250 [13:11:47<86:39:16,  8.72s/it][2025-04-25 21:09:30,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.20 | optimizer_step: 0.97
[2025-04-25 21:09:30,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.75 | bwd_microstep: 5793.32 | bwd_inner_microstep: 5655.34 | bwd_allreduce_microstep: 137.92 | step_microstep: 19.45
[2025-04-25 21:09:30,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.75 | bwd: 5793.33 | bwd_inner: 5655.34 | bwd_allreduce: 137.94 | step: 19.45

 13%|█▎        | 5460/41250 [13:11:56<86:37:02,  8.71s/it]
                                                          
{'loss': 0.0243, 'grad_norm': 0.4908677637577057, 'learning_rate': 3.8911132574067413e-05, 'epoch': 1.32}

 13%|█▎        | 5460/41250 [13:11:56<86:37:02,  8.71s/it][2025-04-25 21:09:39,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 21:09:39,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.63 | bwd_microstep: 5728.07 | bwd_inner_microstep: 5662.85 | bwd_allreduce_microstep: 65.17 | step_microstep: 18.82
[2025-04-25 21:09:39,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.63 | bwd: 5728.08 | bwd_inner: 5662.85 | bwd_allreduce: 65.19 | step: 18.82

 13%|█▎        | 5461/41250 [13:12:04<86:25:49,  8.69s/it]
                                                          
{'loss': 0.0135, 'grad_norm': 0.402901291847229, 'learning_rate': 3.8910621441465716e-05, 'epoch': 1.32}

 13%|█▎        | 5461/41250 [13:12:04<86:25:49,  8.69s/it][2025-04-25 21:09:48,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-25 21:09:48,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.49 | bwd_microstep: 5793.65 | bwd_inner_microstep: 5667.98 | bwd_allreduce_microstep: 125.62 | step_microstep: 19.59
[2025-04-25 21:09:48,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.49 | bwd: 5793.66 | bwd_inner: 5667.98 | bwd_allreduce: 125.64 | step: 19.59

 13%|█▎        | 5462/41250 [13:12:13<86:29:33,  8.70s/it]
                                                          
{'loss': 0.2015, 'grad_norm': 2.8371548652648926, 'learning_rate': 3.891011019228375e-05, 'epoch': 1.32}

 13%|█▎        | 5462/41250 [13:12:13<86:29:33,  8.70s/it][2025-04-25 21:09:56,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 21:09:56,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.20 | bwd_microstep: 5773.20 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 124.70 | step_microstep: 19.17
[2025-04-25 21:09:56,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.20 | bwd: 5773.22 | bwd_inner: 5648.44 | bwd_allreduce: 124.72 | step: 19.17

 13%|█▎        | 5463/41250 [13:12:22<86:27:19,  8.70s/it]
                                                          
{'loss': 0.2296, 'grad_norm': 1.677554726600647, 'learning_rate': 3.890959882652467e-05, 'epoch': 1.32}

 13%|█▎        | 5463/41250 [13:12:22<86:27:19,  8.70s/it][2025-04-25 21:10:05,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.15 | optimizer_step: 1.04
[2025-04-25 21:10:05,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5847.27 | bwd_inner_microstep: 5694.56 | bwd_allreduce_microstep: 152.65 | step_microstep: 19.96
[2025-04-25 21:10:05,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5847.29 | bwd_inner: 5694.56 | bwd_allreduce: 152.68 | step: 19.96

 13%|█▎        | 5464/41250 [13:12:31<86:43:46,  8.72s/it]
                                                          
{'loss': 0.0533, 'grad_norm': 2.072084426879883, 'learning_rate': 3.890908734419164e-05, 'epoch': 1.32}

 13%|█▎        | 5464/41250 [13:12:31<86:43:46,  8.72s/it][2025-04-25 21:10:14,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.30 | optimizer_step: 1.09
[2025-04-25 21:10:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.02 | bwd_microstep: 5770.36 | bwd_inner_microstep: 5706.66 | bwd_allreduce_microstep: 63.64 | step_microstep: 20.43
[2025-04-25 21:10:14,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.02 | bwd: 5770.38 | bwd_inner: 5706.66 | bwd_allreduce: 63.66 | step: 20.43

 13%|█▎        | 5465/41250 [13:12:39<86:41:32,  8.72s/it]
                                                          
{'loss': 0.3754, 'grad_norm': 2.4627952575683594, 'learning_rate': 3.890857574528781e-05, 'epoch': 1.32}

 13%|█▎        | 5465/41250 [13:12:39<86:41:32,  8.72s/it][2025-04-25 21:10:23,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 21:10:23,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.11 | bwd_microstep: 5995.88 | bwd_inner_microstep: 5696.87 | bwd_allreduce_microstep: 298.97 | step_microstep: 18.68
[2025-04-25 21:10:23,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.11 | bwd: 5995.90 | bwd_inner: 5696.87 | bwd_allreduce: 298.99 | step: 18.68

 13%|█▎        | 5466/41250 [13:12:48<87:17:54,  8.78s/it]
                                                          
{'loss': 0.2051, 'grad_norm': 2.9105887413024902, 'learning_rate': 3.890806402981632e-05, 'epoch': 1.33}

 13%|█▎        | 5466/41250 [13:12:48<87:17:54,  8.78s/it][2025-04-25 21:10:32,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:10:32,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5784.78 | bwd_inner_microstep: 5653.22 | bwd_allreduce_microstep: 131.51 | step_microstep: 18.50
[2025-04-25 21:10:32,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5784.80 | bwd_inner: 5653.22 | bwd_allreduce: 131.53 | step: 18.50

 13%|█▎        | 5467/41250 [13:12:57<87:01:17,  8.75s/it]
                                                          
{'loss': 0.3624, 'grad_norm': 3.3244874477386475, 'learning_rate': 3.890755219778034e-05, 'epoch': 1.33}

 13%|█▎        | 5467/41250 [13:12:57<87:01:17,  8.75s/it][2025-04-25 21:10:40,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 21:10:40,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.29 | bwd_microstep: 5766.05 | bwd_inner_microstep: 5655.81 | bwd_allreduce_microstep: 110.19 | step_microstep: 18.52
[2025-04-25 21:10:40,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.29 | bwd: 5766.06 | bwd_inner: 5655.81 | bwd_allreduce: 110.21 | step: 18.52

 13%|█▎        | 5468/41250 [13:13:06<86:46:20,  8.73s/it]
                                                          
{'loss': 0.0856, 'grad_norm': 6.781400680541992, 'learning_rate': 3.890704024918302e-05, 'epoch': 1.33}

 13%|█▎        | 5468/41250 [13:13:06<86:46:20,  8.73s/it][2025-04-25 21:10:49,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:10:49,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.52 | bwd_microstep: 5701.84 | bwd_inner_microstep: 5647.62 | bwd_allreduce_microstep: 54.17 | step_microstep: 18.42
[2025-04-25 21:10:49,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.52 | bwd: 5701.85 | bwd_inner: 5647.62 | bwd_allreduce: 54.19 | step: 18.42

 13%|█▎        | 5469/41250 [13:13:14<86:24:24,  8.69s/it]
                                                          
{'loss': 0.0522, 'grad_norm': 0.9585870504379272, 'learning_rate': 3.890652818402752e-05, 'epoch': 1.33}

 13%|█▎        | 5469/41250 [13:13:14<86:24:24,  8.69s/it][2025-04-25 21:10:58,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 21:10:58,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.64 | bwd_microstep: 5764.54 | bwd_inner_microstep: 5693.59 | bwd_allreduce_microstep: 70.89 | step_microstep: 19.33
[2025-04-25 21:10:58,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.64 | bwd: 5764.55 | bwd_inner: 5693.59 | bwd_allreduce: 70.91 | step: 19.33

 13%|█▎        | 5470/41250 [13:13:23<86:25:11,  8.70s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 1.8189021348953247, 'learning_rate': 3.8906016002316974e-05, 'epoch': 1.33}

 13%|█▎        | 5470/41250 [13:13:23<86:25:11,  8.70s/it][2025-04-25 21:11:06,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.16 | optimizer_step: 0.97
[2025-04-25 21:11:06,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.24 | bwd_microstep: 5688.00 | bwd_inner_microstep: 5646.21 | bwd_allreduce_microstep: 41.75 | step_microstep: 19.38
[2025-04-25 21:11:06,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.24 | bwd: 5688.01 | bwd_inner: 5646.21 | bwd_allreduce: 41.77 | step: 19.39

 13%|█▎        | 5471/41250 [13:13:31<86:07:20,  8.67s/it]
                                                          
{'loss': 0.1907, 'grad_norm': 1.5859200954437256, 'learning_rate': 3.890550370405457e-05, 'epoch': 1.33}

 13%|█▎        | 5471/41250 [13:13:31<86:07:20,  8.67s/it][2025-04-25 21:11:15,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[2025-04-25 21:11:15,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.34 | bwd_microstep: 5725.82 | bwd_inner_microstep: 5697.44 | bwd_allreduce_microstep: 28.33 | step_microstep: 18.88
[2025-04-25 21:11:15,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.34 | bwd: 5725.84 | bwd_inner: 5697.44 | bwd_allreduce: 28.35 | step: 18.89

 13%|█▎        | 5472/41250 [13:13:40<86:06:12,  8.66s/it]
                                                          
{'loss': 0.127, 'grad_norm': 1.532275915145874, 'learning_rate': 3.890499128924345e-05, 'epoch': 1.33}

 13%|█▎        | 5472/41250 [13:13:40<86:06:12,  8.66s/it][2025-04-25 21:11:23,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.10 | optimizer_step: 1.02
[2025-04-25 21:11:23,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.92 | bwd_microstep: 5739.60 | bwd_inner_microstep: 5648.39 | bwd_allreduce_microstep: 91.15 | step_microstep: 19.02
[2025-04-25 21:11:23,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.92 | bwd: 5739.62 | bwd_inner: 5648.39 | bwd_allreduce: 91.17 | step: 19.02

 13%|█▎        | 5473/41250 [13:13:49<86:05:45,  8.66s/it]
                                                          
{'loss': 0.0664, 'grad_norm': 1.3962030410766602, 'learning_rate': 3.890447875788679e-05, 'epoch': 1.33}

 13%|█▎        | 5473/41250 [13:13:49<86:05:45,  8.66s/it][2025-04-25 21:11:32,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 21:11:32,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.08 | bwd_microstep: 5730.88 | bwd_inner_microstep: 5698.65 | bwd_allreduce_microstep: 32.18 | step_microstep: 19.21
[2025-04-25 21:11:32,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.08 | bwd: 5730.89 | bwd_inner: 5698.65 | bwd_allreduce: 32.20 | step: 19.21

 13%|█▎        | 5474/41250 [13:13:57<86:06:13,  8.66s/it]
                                                          
{'loss': 0.0719, 'grad_norm': 1.631029725074768, 'learning_rate': 3.890396610998773e-05, 'epoch': 1.33}

 13%|█▎        | 5474/41250 [13:13:57<86:06:13,  8.66s/it][2025-04-25 21:11:41,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 21:11:41,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.72 | bwd_microstep: 6018.66 | bwd_inner_microstep: 5658.19 | bwd_allreduce_microstep: 360.43 | step_microstep: 18.93
[2025-04-25 21:11:41,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.72 | bwd: 6018.68 | bwd_inner: 5658.19 | bwd_allreduce: 360.45 | step: 18.93

 13%|█▎        | 5475/41250 [13:14:06<86:53:18,  8.74s/it]
                                                          
{'loss': 0.5073, 'grad_norm': 5.107706546783447, 'learning_rate': 3.890345334554943e-05, 'epoch': 1.33}

 13%|█▎        | 5475/41250 [13:14:06<86:53:18,  8.74s/it][2025-04-25 21:11:50,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:11:50,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.98 | bwd_microstep: 5764.81 | bwd_inner_microstep: 5682.00 | bwd_allreduce_microstep: 82.76 | step_microstep: 18.61
[2025-04-25 21:11:50,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.98 | bwd: 5764.82 | bwd_inner: 5682.00 | bwd_allreduce: 82.78 | step: 18.62

 13%|█▎        | 5476/41250 [13:14:15<86:42:55,  8.73s/it]
                                                          
{'loss': 0.1439, 'grad_norm': 1.3937944173812866, 'learning_rate': 3.890294046457507e-05, 'epoch': 1.33}

 13%|█▎        | 5476/41250 [13:14:15<86:42:55,  8.73s/it][2025-04-25 21:11:58,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:11:58,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.21 | bwd_microstep: 5770.10 | bwd_inner_microstep: 5642.59 | bwd_allreduce_microstep: 127.47 | step_microstep: 18.72
[2025-04-25 21:11:58,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.22 | bwd: 5770.11 | bwd_inner: 5642.59 | bwd_allreduce: 127.48 | step: 18.72

 13%|█▎        | 5477/41250 [13:14:24<86:34:07,  8.71s/it]
                                                          
{'loss': 0.102, 'grad_norm': 0.9179266095161438, 'learning_rate': 3.8902427467067787e-05, 'epoch': 1.33}

 13%|█▎        | 5477/41250 [13:14:24<86:34:07,  8.71s/it][2025-04-25 21:12:07,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:12:07,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.36 | bwd_microstep: 5768.51 | bwd_inner_microstep: 5648.07 | bwd_allreduce_microstep: 120.39 | step_microstep: 19.03
[2025-04-25 21:12:07,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.36 | bwd: 5768.52 | bwd_inner: 5648.07 | bwd_allreduce: 120.41 | step: 19.03

 13%|█▎        | 5478/41250 [13:14:32<86:27:48,  8.70s/it]
                                                          
{'loss': 0.0195, 'grad_norm': 0.47357940673828125, 'learning_rate': 3.890191435303076e-05, 'epoch': 1.33}

 13%|█▎        | 5478/41250 [13:14:32<86:27:48,  8.70s/it][2025-04-25 21:12:16,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.12 | optimizer_step: 1.05
[2025-04-25 21:12:16,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.19 | bwd_microstep: 5765.47 | bwd_inner_microstep: 5631.41 | bwd_allreduce_microstep: 134.00 | step_microstep: 19.74
[2025-04-25 21:12:16,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.20 | bwd: 5765.49 | bwd_inner: 5631.41 | bwd_allreduce: 134.03 | step: 19.74

 13%|█▎        | 5479/41250 [13:14:41<86:21:53,  8.69s/it]
                                                          
{'loss': 0.2338, 'grad_norm': 2.8662660121917725, 'learning_rate': 3.890140112246715e-05, 'epoch': 1.33}

 13%|█▎        | 5479/41250 [13:14:41<86:21:53,  8.69s/it][2025-04-25 21:12:25,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 21:12:25,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5891.15 | bwd_inner_microstep: 5633.81 | bwd_allreduce_microstep: 257.29 | step_microstep: 19.01
[2025-04-25 21:12:25,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5891.16 | bwd_inner: 5633.81 | bwd_allreduce: 257.31 | step: 19.02

 13%|█▎        | 5480/41250 [13:14:50<86:41:05,  8.72s/it]
                                                          
{'loss': 0.0731, 'grad_norm': 0.7891958355903625, 'learning_rate': 3.890088777538012e-05, 'epoch': 1.33}

 13%|█▎        | 5480/41250 [13:14:50<86:41:05,  8.72s/it][2025-04-25 21:12:33,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:12:33,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.33 | bwd_microstep: 5777.69 | bwd_inner_microstep: 5633.42 | bwd_allreduce_microstep: 144.22 | step_microstep: 18.87
[2025-04-25 21:12:33,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.33 | bwd: 5777.70 | bwd_inner: 5633.42 | bwd_allreduce: 144.24 | step: 18.87

 13%|█▎        | 5481/41250 [13:14:59<86:32:26,  8.71s/it]
                                                          
{'loss': 0.0525, 'grad_norm': 1.8168134689331055, 'learning_rate': 3.890037431177284e-05, 'epoch': 1.33}

 13%|█▎        | 5481/41250 [13:14:59<86:32:26,  8.71s/it][2025-04-25 21:12:42,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 0.98
[2025-04-25 21:12:42,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5694.71 | bwd_inner_microstep: 5643.40 | bwd_allreduce_microstep: 51.27 | step_microstep: 18.86
[2025-04-25 21:12:42,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5694.73 | bwd_inner: 5643.40 | bwd_allreduce: 51.28 | step: 18.86

 13%|█▎        | 5482/41250 [13:15:07<86:14:57,  8.68s/it]
                                                          
{'loss': 0.0249, 'grad_norm': 0.39514419436454773, 'learning_rate': 3.8899860731648466e-05, 'epoch': 1.33}

 13%|█▎        | 5482/41250 [13:15:07<86:14:57,  8.68s/it][2025-04-25 21:12:50,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 21:12:50,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.45 | bwd_microstep: 5687.94 | bwd_inner_microstep: 5644.02 | bwd_allreduce_microstep: 43.87 | step_microstep: 18.97
[2025-04-25 21:12:50,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.45 | bwd: 5687.96 | bwd_inner: 5644.02 | bwd_allreduce: 43.89 | step: 18.98

 13%|█▎        | 5483/41250 [13:15:16<86:02:01,  8.66s/it]
                                                          
{'loss': 0.3749, 'grad_norm': 2.1315784454345703, 'learning_rate': 3.889934703501017e-05, 'epoch': 1.33}

 13%|█▎        | 5483/41250 [13:15:16<86:02:01,  8.66s/it][2025-04-25 21:12:59,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.23 | optimizer_step: 1.05
[2025-04-25 21:12:59,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.25 | bwd_microstep: 5740.63 | bwd_inner_microstep: 5641.67 | bwd_allreduce_microstep: 98.91 | step_microstep: 19.57
[2025-04-25 21:12:59,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.25 | bwd: 5740.65 | bwd_inner: 5641.67 | bwd_allreduce: 98.93 | step: 19.57

 13%|█▎        | 5484/41250 [13:15:24<86:01:17,  8.66s/it]
                                                          
{'loss': 0.1904, 'grad_norm': 2.0090699195861816, 'learning_rate': 3.8898833221861115e-05, 'epoch': 1.33}

 13%|█▎        | 5484/41250 [13:15:24<86:01:17,  8.66s/it][2025-04-25 21:13:08,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:13:08,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.67 | bwd_microstep: 5734.83 | bwd_inner_microstep: 5691.68 | bwd_allreduce_microstep: 43.10 | step_microstep: 18.97
[2025-04-25 21:13:08,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.67 | bwd: 5734.84 | bwd_inner: 5691.68 | bwd_allreduce: 43.12 | step: 18.98

 13%|█▎        | 5485/41250 [13:15:33<86:02:13,  8.66s/it]
                                                          
{'loss': 0.1404, 'grad_norm': 2.889772891998291, 'learning_rate': 3.8898319292204464e-05, 'epoch': 1.33}

 13%|█▎        | 5485/41250 [13:15:33<86:02:13,  8.66s/it][2025-04-25 21:13:16,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 21:13:16,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.01 | bwd_microstep: 5698.74 | bwd_inner_microstep: 5644.74 | bwd_allreduce_microstep: 53.95 | step_microstep: 19.01
[2025-04-25 21:13:16,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.01 | bwd: 5698.75 | bwd_inner: 5644.74 | bwd_allreduce: 53.96 | step: 19.01

 13%|█▎        | 5486/41250 [13:15:42<85:52:36,  8.64s/it]
                                                          
{'loss': 0.4454, 'grad_norm': 3.715909242630005, 'learning_rate': 3.88978052460434e-05, 'epoch': 1.33}

 13%|█▎        | 5486/41250 [13:15:42<85:52:36,  8.64s/it][2025-04-25 21:13:25,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 21:13:25,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.83 | bwd_microstep: 5704.30 | bwd_inner_microstep: 5691.31 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.20
[2025-04-25 21:13:25,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.83 | bwd: 5704.32 | bwd_inner: 5691.31 | bwd_allreduce: 12.95 | step: 19.20

 13%|█▎        | 5487/41250 [13:15:50<85:50:47,  8.64s/it]
                                                          
{'loss': 0.0498, 'grad_norm': 1.4337108135223389, 'learning_rate': 3.889729108338108e-05, 'epoch': 1.33}

 13%|█▎        | 5487/41250 [13:15:50<85:50:47,  8.64s/it][2025-04-25 21:13:34,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 21:13:34,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.53 | bwd_microstep: 5729.43 | bwd_inner_microstep: 5716.65 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.22
[2025-04-25 21:13:34,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.53 | bwd: 5729.44 | bwd_inner: 5716.65 | bwd_allreduce: 12.76 | step: 19.22

 13%|█▎        | 5488/41250 [13:15:59<85:55:56,  8.65s/it]
                                                          
{'loss': 0.1787, 'grad_norm': 2.047485589981079, 'learning_rate': 3.889677680422068e-05, 'epoch': 1.33}

 13%|█▎        | 5488/41250 [13:15:59<85:55:56,  8.65s/it][2025-04-25 21:13:42,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:13:42,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.99 | bwd_microstep: 5729.18 | bwd_inner_microstep: 5709.61 | bwd_allreduce_microstep: 19.52 | step_microstep: 18.78
[2025-04-25 21:13:42,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.99 | bwd: 5729.20 | bwd_inner: 5709.61 | bwd_allreduce: 19.55 | step: 18.79

 13%|█▎        | 5489/41250 [13:16:08<85:59:02,  8.66s/it]
                                                          
{'loss': 0.1453, 'grad_norm': 4.281248092651367, 'learning_rate': 3.8896262408565365e-05, 'epoch': 1.33}

 13%|█▎        | 5489/41250 [13:16:08<85:59:02,  8.66s/it][2025-04-25 21:13:51,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 21:13:51,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.61 | bwd_microstep: 5686.33 | bwd_inner_microstep: 5662.79 | bwd_allreduce_microstep: 23.48 | step_microstep: 19.23
[2025-04-25 21:13:51,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.62 | bwd: 5686.34 | bwd_inner: 5662.79 | bwd_allreduce: 23.50 | step: 19.23

 13%|█▎        | 5490/41250 [13:16:16<85:54:03,  8.65s/it]
                                                          
{'loss': 0.1843, 'grad_norm': 2.1088523864746094, 'learning_rate': 3.889574789641831e-05, 'epoch': 1.33}

 13%|█▎        | 5490/41250 [13:16:16<85:54:03,  8.65s/it][2025-04-25 21:14:00,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 21:14:00,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.28 | bwd_microstep: 5737.01 | bwd_inner_microstep: 5707.38 | bwd_allreduce_microstep: 29.59 | step_microstep: 18.91
[2025-04-25 21:14:00,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.28 | bwd: 5737.03 | bwd_inner: 5707.38 | bwd_allreduce: 29.61 | step: 18.91

 13%|█▎        | 5491/41250 [13:16:25<85:58:40,  8.66s/it]
                                                          
{'loss': 0.1696, 'grad_norm': 2.3685309886932373, 'learning_rate': 3.889523326778269e-05, 'epoch': 1.33}

 13%|█▎        | 5491/41250 [13:16:25<85:58:40,  8.66s/it][2025-04-25 21:14:08,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.18 | optimizer_step: 1.04
[2025-04-25 21:14:08,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.12 | bwd_microstep: 5721.00 | bwd_inner_microstep: 5708.08 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.90
[2025-04-25 21:14:08,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.12 | bwd: 5721.02 | bwd_inner: 5708.08 | bwd_allreduce: 12.90 | step: 19.90

 13%|█▎        | 5492/41250 [13:16:34<85:59:20,  8.66s/it]
                                                          
{'loss': 0.2582, 'grad_norm': 2.6051878929138184, 'learning_rate': 3.889471852266167e-05, 'epoch': 1.33}

 13%|█▎        | 5492/41250 [13:16:34<85:59:20,  8.66s/it][2025-04-25 21:14:17,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 21:14:17,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.86 | bwd_microstep: 5795.74 | bwd_inner_microstep: 5782.83 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.99
[2025-04-25 21:14:17,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.86 | bwd: 5795.76 | bwd_inner: 5782.83 | bwd_allreduce: 12.88 | step: 18.99

 13%|█▎        | 5493/41250 [13:16:42<86:20:17,  8.69s/it]
                                                          
{'loss': 0.1994, 'grad_norm': 2.134068489074707, 'learning_rate': 3.889420366105842e-05, 'epoch': 1.33}

 13%|█▎        | 5493/41250 [13:16:42<86:20:17,  8.69s/it][2025-04-25 21:14:26,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 21:14:26,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5773.73 | bwd_inner_microstep: 5664.64 | bwd_allreduce_microstep: 109.04 | step_microstep: 19.15
[2025-04-25 21:14:26,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5773.74 | bwd_inner: 5664.64 | bwd_allreduce: 109.06 | step: 19.15

 13%|█▎        | 5494/41250 [13:16:51<86:20:00,  8.69s/it]
                                                          
{'loss': 0.0688, 'grad_norm': 1.203930139541626, 'learning_rate': 3.889368868297613e-05, 'epoch': 1.33}

 13%|█▎        | 5494/41250 [13:16:51<86:20:00,  8.69s/it][2025-04-25 21:14:34,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.21
[2025-04-25 21:14:34,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5748.42 | bwd_inner_microstep: 5690.03 | bwd_allreduce_microstep: 58.35 | step_microstep: 19.36
[2025-04-25 21:14:34,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5748.44 | bwd_inner: 5690.03 | bwd_allreduce: 58.37 | step: 19.36

 13%|█▎        | 5495/41250 [13:17:00<86:18:18,  8.69s/it]
                                                          
{'loss': 0.1758, 'grad_norm': 2.577298879623413, 'learning_rate': 3.889317358841797e-05, 'epoch': 1.33}

 13%|█▎        | 5495/41250 [13:17:00<86:18:18,  8.69s/it][2025-04-25 21:14:43,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 21:14:43,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.76 | bwd_microstep: 5805.33 | bwd_inner_microstep: 5662.85 | bwd_allreduce_microstep: 142.43 | step_microstep: 19.04
[2025-04-25 21:14:43,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.76 | bwd: 5805.35 | bwd_inner: 5662.85 | bwd_allreduce: 142.45 | step: 19.05

 13%|█▎        | 5496/41250 [13:17:08<86:23:38,  8.70s/it]
                                                          
{'loss': 0.1694, 'grad_norm': 2.8119096755981445, 'learning_rate': 3.88926583773871e-05, 'epoch': 1.33}

 13%|█▎        | 5496/41250 [13:17:08<86:23:38,  8.70s/it][2025-04-25 21:14:52,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:14:52,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.07 | bwd_microstep: 5773.83 | bwd_inner_microstep: 5703.23 | bwd_allreduce_microstep: 70.55 | step_microstep: 18.84
[2025-04-25 21:14:52,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.07 | bwd: 5773.84 | bwd_inner: 5703.23 | bwd_allreduce: 70.57 | step: 18.84

 13%|█▎        | 5497/41250 [13:17:17<86:24:50,  8.70s/it]
                                                          
{'loss': 0.0228, 'grad_norm': 0.5616419315338135, 'learning_rate': 3.889214304988671e-05, 'epoch': 1.33}

 13%|█▎        | 5497/41250 [13:17:17<86:24:50,  8.70s/it][2025-04-25 21:15:01,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:15:01,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.80 | bwd_microstep: 5805.59 | bwd_inner_microstep: 5792.75 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.69
[2025-04-25 21:15:01,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.80 | bwd: 5805.61 | bwd_inner: 5792.75 | bwd_allreduce: 12.82 | step: 18.70

 13%|█▎        | 5498/41250 [13:17:26<86:39:17,  8.73s/it]
                                                          
{'loss': 0.1287, 'grad_norm': 1.7678215503692627, 'learning_rate': 3.8891627605919977e-05, 'epoch': 1.33}

 13%|█▎        | 5498/41250 [13:17:26<86:39:17,  8.73s/it][2025-04-25 21:15:09,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-25 21:15:09,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.47 | bwd_microstep: 5715.88 | bwd_inner_microstep: 5669.23 | bwd_allreduce_microstep: 46.59 | step_microstep: 19.28
[2025-04-25 21:15:09,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.47 | bwd: 5715.89 | bwd_inner: 5669.23 | bwd_allreduce: 46.61 | step: 19.28

 13%|█▎        | 5499/41250 [13:17:35<86:23:37,  8.70s/it]
                                                          
{'loss': 0.0618, 'grad_norm': 1.4847462177276611, 'learning_rate': 3.889111204549007e-05, 'epoch': 1.33}

 13%|█▎        | 5499/41250 [13:17:35<86:23:37,  8.70s/it][2025-04-25 21:15:18,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 21:15:18,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.48 | bwd_microstep: 5721.05 | bwd_inner_microstep: 5708.12 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.97
[2025-04-25 21:15:18,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.48 | bwd: 5721.07 | bwd_inner: 5708.12 | bwd_allreduce: 12.90 | step: 18.97

 13%|█▎        | 5500/41250 [13:17:43<86:18:10,  8.69s/it]
                                                          
{'loss': 0.1607, 'grad_norm': 0.8976242542266846, 'learning_rate': 3.889059636860017e-05, 'epoch': 1.33}

 13%|█▎        | 5500/41250 [13:17:43<86:18:10,  8.69s/it][2025-04-25 21:15:27,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:15:27,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.24 | bwd_microstep: 5764.86 | bwd_inner_microstep: 5652.64 | bwd_allreduce_microstep: 112.18 | step_microstep: 18.72
[2025-04-25 21:15:27,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.24 | bwd: 5764.87 | bwd_inner: 5652.64 | bwd_allreduce: 112.19 | step: 18.73

 13%|█▎        | 5501/41250 [13:17:52<86:15:56,  8.69s/it]
                                                          
{'loss': 0.0649, 'grad_norm': 1.8299729824066162, 'learning_rate': 3.889008057525347e-05, 'epoch': 1.33}

 13%|█▎        | 5501/41250 [13:17:52<86:15:56,  8.69s/it][2025-04-25 21:15:35,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.02 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:15:35,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5727.31 | bwd_inner_microstep: 5714.42 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.27
[2025-04-25 21:15:35,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5727.32 | bwd_inner: 5714.42 | bwd_allreduce: 12.86 | step: 19.27

 13%|█▎        | 5502/41250 [13:18:01<86:11:36,  8.68s/it]
                                                          
{'loss': 0.1546, 'grad_norm': 1.486798644065857, 'learning_rate': 3.8889564665453135e-05, 'epoch': 1.33}

 13%|█▎        | 5502/41250 [13:18:01<86:11:36,  8.68s/it][2025-04-25 21:15:44,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:15:44,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5781.51 | bwd_inner_microstep: 5669.72 | bwd_allreduce_microstep: 111.74 | step_microstep: 19.06
[2025-04-25 21:15:44,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.29 | bwd: 5781.52 | bwd_inner: 5669.73 | bwd_allreduce: 111.76 | step: 19.06

 13%|█▎        | 5503/41250 [13:18:09<86:17:17,  8.69s/it]
                                                          
{'loss': 0.1301, 'grad_norm': 1.4280034303665161, 'learning_rate': 3.8889048639202346e-05, 'epoch': 1.33}

 13%|█▎        | 5503/41250 [13:18:09<86:17:17,  8.69s/it][2025-04-25 21:15:53,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:15:53,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.68 | bwd_microstep: 5719.22 | bwd_inner_microstep: 5706.31 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.88
[2025-04-25 21:15:53,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.68 | bwd: 5719.24 | bwd_inner: 5706.31 | bwd_allreduce: 12.88 | step: 18.88

 13%|█▎        | 5504/41250 [13:18:18<86:14:00,  8.68s/it]
                                                          
{'loss': 0.2158, 'grad_norm': 2.5425209999084473, 'learning_rate': 3.8888532496504286e-05, 'epoch': 1.33}

 13%|█▎        | 5504/41250 [13:18:18<86:14:00,  8.68s/it][2025-04-25 21:16:01,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 21:16:01,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.29 | bwd_microstep: 5802.77 | bwd_inner_microstep: 5789.85 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.08
[2025-04-25 21:16:01,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.29 | bwd: 5802.78 | bwd_inner: 5789.85 | bwd_allreduce: 12.89 | step: 19.08

 13%|█▎        | 5505/41250 [13:18:27<86:31:51,  8.71s/it]
                                                          
{'loss': 0.2176, 'grad_norm': 4.34747838973999, 'learning_rate': 3.888801623736215e-05, 'epoch': 1.33}

 13%|█▎        | 5505/41250 [13:18:27<86:31:51,  8.71s/it][2025-04-25 21:16:10,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:16:10,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.18 | bwd_microstep: 5800.42 | bwd_inner_microstep: 5708.76 | bwd_allreduce_microstep: 91.61 | step_microstep: 18.62
[2025-04-25 21:16:10,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.18 | bwd: 5800.43 | bwd_inner: 5708.76 | bwd_allreduce: 91.63 | step: 18.63

 13%|█▎        | 5506/41250 [13:18:36<86:35:22,  8.72s/it]
                                                          
{'loss': 0.1358, 'grad_norm': 1.7987653017044067, 'learning_rate': 3.8887499861779093e-05, 'epoch': 1.33}

 13%|█▎        | 5506/41250 [13:18:36<86:35:22,  8.72s/it][2025-04-25 21:16:19,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 21:16:19,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.44 | bwd_microstep: 5903.16 | bwd_inner_microstep: 5677.19 | bwd_allreduce_microstep: 225.93 | step_microstep: 19.09
[2025-04-25 21:16:19,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.44 | bwd: 5903.18 | bwd_inner: 5677.19 | bwd_allreduce: 225.95 | step: 19.10

 13%|█▎        | 5507/41250 [13:18:44<86:52:39,  8.75s/it]
                                                          
{'loss': 0.1887, 'grad_norm': 2.370481014251709, 'learning_rate': 3.8886983369758324e-05, 'epoch': 1.34}

 13%|█▎        | 5507/41250 [13:18:44<86:52:39,  8.75s/it][2025-04-25 21:16:28,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 21:16:28,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.67 | bwd_microstep: 5750.69 | bwd_inner_microstep: 5705.46 | bwd_allreduce_microstep: 45.19 | step_microstep: 18.74
[2025-04-25 21:16:28,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.67 | bwd: 5750.70 | bwd_inner: 5705.45 | bwd_allreduce: 45.20 | step: 18.74

 13%|█▎        | 5508/41250 [13:18:53<86:41:55,  8.73s/it]
                                                          
{'loss': 0.1088, 'grad_norm': 2.67803955078125, 'learning_rate': 3.8886466761303016e-05, 'epoch': 1.34}

 13%|█▎        | 5508/41250 [13:18:53<86:41:55,  8.73s/it][2025-04-25 21:16:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:16:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.22 | bwd_microstep: 5708.22 | bwd_inner_microstep: 5667.23 | bwd_allreduce_microstep: 40.94 | step_microstep: 18.51
[2025-04-25 21:16:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.22 | bwd: 5708.24 | bwd_inner: 5667.23 | bwd_allreduce: 40.96 | step: 18.52

 13%|█▎        | 5509/41250 [13:19:02<86:22:46,  8.70s/it]
                                                          
{'loss': 0.0636, 'grad_norm': 1.9809812307357788, 'learning_rate': 3.8885950036416355e-05, 'epoch': 1.34}

 13%|█▎        | 5509/41250 [13:19:02<86:22:46,  8.70s/it][2025-04-25 21:16:45,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:16:45,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.02 | bwd_microstep: 5767.42 | bwd_inner_microstep: 5702.16 | bwd_allreduce_microstep: 65.21 | step_microstep: 18.50
[2025-04-25 21:16:45,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.02 | bwd: 5767.43 | bwd_inner: 5702.16 | bwd_allreduce: 65.23 | step: 18.50

 13%|█▎        | 5510/41250 [13:19:10<86:23:23,  8.70s/it]
                                                          
{'loss': 0.1855, 'grad_norm': 3.606659173965454, 'learning_rate': 3.8885433195101526e-05, 'epoch': 1.34}

 13%|█▎        | 5510/41250 [13:19:10<86:23:23,  8.70s/it][2025-04-25 21:16:54,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 21:16:54,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.54 | bwd_microstep: 5755.17 | bwd_inner_microstep: 5705.65 | bwd_allreduce_microstep: 49.47 | step_microstep: 19.43
[2025-04-25 21:16:54,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.54 | bwd: 5755.18 | bwd_inner: 5705.65 | bwd_allreduce: 49.49 | step: 19.43

 13%|█▎        | 5511/41250 [13:19:19<86:22:05,  8.70s/it]
                                                          
{'loss': 0.0717, 'grad_norm': 1.6154595613479614, 'learning_rate': 3.888491623736171e-05, 'epoch': 1.34}

 13%|█▎        | 5511/41250 [13:19:19<86:22:05,  8.70s/it][2025-04-25 21:17:03,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:17:03,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.05 | bwd_microstep: 5797.14 | bwd_inner_microstep: 5784.27 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.74
[2025-04-25 21:17:03,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.05 | bwd: 5797.16 | bwd_inner: 5784.27 | bwd_allreduce: 12.84 | step: 18.75

 13%|█▎        | 5512/41250 [13:19:28<86:34:47,  8.72s/it]
                                                          
{'loss': 0.1833, 'grad_norm': 2.5004568099975586, 'learning_rate': 3.8884399163200115e-05, 'epoch': 1.34}

 13%|█▎        | 5512/41250 [13:19:28<86:34:47,  8.72s/it][2025-04-25 21:17:11,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 21:17:11,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.02 | bwd_microstep: 5918.27 | bwd_inner_microstep: 5646.17 | bwd_allreduce_microstep: 272.05 | step_microstep: 18.95
[2025-04-25 21:17:11,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.02 | bwd: 5918.28 | bwd_inner: 5646.17 | bwd_allreduce: 272.07 | step: 18.95

 13%|█▎        | 5513/41250 [13:19:37<86:53:46,  8.75s/it]
                                                          
{'loss': 0.0447, 'grad_norm': 1.013670563697815, 'learning_rate': 3.8883881972619904e-05, 'epoch': 1.34}

 13%|█▎        | 5513/41250 [13:19:37<86:53:46,  8.75s/it][2025-04-25 21:17:20,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:17:20,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5704.39 | bwd_inner_microstep: 5691.52 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.14
[2025-04-25 21:17:20,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5704.41 | bwd_inner: 5691.52 | bwd_allreduce: 12.85 | step: 19.14

 13%|█▎        | 5514/41250 [13:19:45<86:32:22,  8.72s/it]
                                                          
{'loss': 0.0844, 'grad_norm': 1.0146520137786865, 'learning_rate': 3.888336466562427e-05, 'epoch': 1.34}

 13%|█▎        | 5514/41250 [13:19:45<86:32:22,  8.72s/it][2025-04-25 21:17:29,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:17:29,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5753.87 | bwd_inner_microstep: 5707.76 | bwd_allreduce_microstep: 46.06 | step_microstep: 18.66
[2025-04-25 21:17:29,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5753.88 | bwd_inner: 5707.76 | bwd_allreduce: 46.08 | step: 18.66

 13%|█▎        | 5515/41250 [13:19:54<86:27:06,  8.71s/it]
                                                          
{'loss': 0.3565, 'grad_norm': 2.572575092315674, 'learning_rate': 3.888284724221641e-05, 'epoch': 1.34}

 13%|█▎        | 5515/41250 [13:19:54<86:27:06,  8.71s/it][2025-04-25 21:17:37,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:17:37,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.12 | bwd_microstep: 5680.75 | bwd_inner_microstep: 5655.45 | bwd_allreduce_microstep: 25.25 | step_microstep: 18.76
[2025-04-25 21:17:37,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.12 | bwd: 5680.76 | bwd_inner: 5655.45 | bwd_allreduce: 25.26 | step: 18.76

 13%|█▎        | 5516/41250 [13:20:03<86:07:50,  8.68s/it]
                                                          
{'loss': 0.1748, 'grad_norm': 4.495077133178711, 'learning_rate': 3.8882329702399515e-05, 'epoch': 1.34}

 13%|█▎        | 5516/41250 [13:20:03<86:07:50,  8.68s/it][2025-04-25 21:17:46,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:17:46,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.06 | bwd_microstep: 5743.56 | bwd_inner_microstep: 5684.40 | bwd_allreduce_microstep: 59.12 | step_microstep: 18.60
[2025-04-25 21:17:46,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.06 | bwd: 5743.58 | bwd_inner: 5684.40 | bwd_allreduce: 59.13 | step: 18.60

 13%|█▎        | 5517/41250 [13:20:11<86:07:10,  8.68s/it]
                                                          
{'loss': 0.0926, 'grad_norm': 1.09340238571167, 'learning_rate': 3.888181204617677e-05, 'epoch': 1.34}

 13%|█▎        | 5517/41250 [13:20:11<86:07:10,  8.68s/it][2025-04-25 21:17:55,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.09 | optimizer_step: 1.02
[2025-04-25 21:17:55,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.82 | bwd_microstep: 5757.18 | bwd_inner_microstep: 5681.19 | bwd_allreduce_microstep: 75.94 | step_microstep: 18.95
[2025-04-25 21:17:55,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.82 | bwd: 5757.20 | bwd_inner: 5681.19 | bwd_allreduce: 75.96 | step: 18.96

 13%|█▎        | 5518/41250 [13:20:20<86:08:41,  8.68s/it]
                                                          
{'loss': 0.2007, 'grad_norm': 2.256058931350708, 'learning_rate': 3.8881294273551364e-05, 'epoch': 1.34}

 13%|█▎        | 5518/41250 [13:20:20<86:08:41,  8.68s/it][2025-04-25 21:18:03,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:18:03,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.91 | bwd_microstep: 5753.09 | bwd_inner_microstep: 5657.56 | bwd_allreduce_microstep: 95.48 | step_microstep: 18.45
[2025-04-25 21:18:03,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.91 | bwd: 5753.10 | bwd_inner: 5657.56 | bwd_allreduce: 95.50 | step: 18.45

 13%|█▎        | 5519/41250 [13:20:29<86:05:55,  8.67s/it]
                                                          
{'loss': 0.1747, 'grad_norm': 2.9966695308685303, 'learning_rate': 3.88807763845265e-05, 'epoch': 1.34}

 13%|█▎        | 5519/41250 [13:20:29<86:05:55,  8.67s/it][2025-04-25 21:18:12,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.13
[2025-04-25 21:18:12,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.45 | bwd_microstep: 5694.43 | bwd_inner_microstep: 5641.46 | bwd_allreduce_microstep: 52.93 | step_microstep: 19.20
[2025-04-25 21:18:12,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.45 | bwd: 5694.45 | bwd_inner: 5641.46 | bwd_allreduce: 52.95 | step: 19.20

 13%|█▎        | 5520/41250 [13:20:37<85:53:01,  8.65s/it]
                                                          
{'loss': 0.0564, 'grad_norm': 1.0483108758926392, 'learning_rate': 3.888025837910536e-05, 'epoch': 1.34}

 13%|█▎        | 5520/41250 [13:20:37<85:53:01,  8.65s/it][2025-04-25 21:18:21,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 21:18:21,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.03 | bwd_microstep: 5762.23 | bwd_inner_microstep: 5680.85 | bwd_allreduce_microstep: 81.33 | step_microstep: 18.71
[2025-04-25 21:18:21,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.03 | bwd: 5762.24 | bwd_inner: 5680.85 | bwd_allreduce: 81.35 | step: 18.71

 13%|█▎        | 5521/41250 [13:20:46<85:58:54,  8.66s/it]
                                                          
{'loss': 0.1182, 'grad_norm': 1.8858964443206787, 'learning_rate': 3.887974025729114e-05, 'epoch': 1.34}

 13%|█▎        | 5521/41250 [13:20:46<85:58:54,  8.66s/it][2025-04-25 21:18:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:18:29,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.21 | bwd_microstep: 5709.28 | bwd_inner_microstep: 5643.15 | bwd_allreduce_microstep: 66.08 | step_microstep: 19.00
[2025-04-25 21:18:29,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.21 | bwd: 5709.30 | bwd_inner: 5643.15 | bwd_allreduce: 66.10 | step: 19.00

 13%|█▎        | 5522/41250 [13:20:55<85:49:28,  8.65s/it]
                                                          
{'loss': 0.0219, 'grad_norm': 0.5126562118530273, 'learning_rate': 3.887922201908703e-05, 'epoch': 1.34}

 13%|█▎        | 5522/41250 [13:20:55<85:49:28,  8.65s/it][2025-04-25 21:18:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 21:18:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.41 | bwd_microstep: 5713.59 | bwd_inner_microstep: 5700.80 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.55
[2025-04-25 21:18:38,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.41 | bwd: 5713.60 | bwd_inner: 5700.80 | bwd_allreduce: 12.76 | step: 18.55

 13%|█▎        | 5523/41250 [13:21:03<85:50:53,  8.65s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 1.146093726158142, 'learning_rate': 3.8878703664496235e-05, 'epoch': 1.34}

 13%|█▎        | 5523/41250 [13:21:03<85:50:53,  8.65s/it][2025-04-25 21:18:47,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 21:18:47,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5778.28 | bwd_inner_microstep: 5643.27 | bwd_allreduce_microstep: 134.96 | step_microstep: 18.87
[2025-04-25 21:18:47,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5778.29 | bwd_inner: 5643.27 | bwd_allreduce: 134.98 | step: 18.88

 13%|█▎        | 5524/41250 [13:21:12<85:57:07,  8.66s/it]
                                                          
{'loss': 0.0497, 'grad_norm': 0.916450560092926, 'learning_rate': 3.8878185193521945e-05, 'epoch': 1.34}

 13%|█▎        | 5524/41250 [13:21:12<85:57:07,  8.66s/it][2025-04-25 21:18:55,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:18:55,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5740.94 | bwd_inner_microstep: 5699.35 | bwd_allreduce_microstep: 41.55 | step_microstep: 18.71
[2025-04-25 21:18:55,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5740.95 | bwd_inner: 5699.35 | bwd_allreduce: 41.56 | step: 18.71

 13%|█▎        | 5525/41250 [13:21:21<85:59:42,  8.67s/it]
                                                          
{'loss': 0.0758, 'grad_norm': 1.8716577291488647, 'learning_rate': 3.8877666606167354e-05, 'epoch': 1.34}

 13%|█▎        | 5525/41250 [13:21:21<85:59:42,  8.67s/it][2025-04-25 21:19:04,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:19:04,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.92 | bwd_microstep: 5761.76 | bwd_inner_microstep: 5666.14 | bwd_allreduce_microstep: 95.57 | step_microstep: 18.84
[2025-04-25 21:19:04,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.92 | bwd: 5761.77 | bwd_inner: 5666.14 | bwd_allreduce: 95.59 | step: 18.84

 13%|█▎        | 5526/41250 [13:21:29<86:01:02,  8.67s/it]
                                                          
{'loss': 0.0528, 'grad_norm': 1.3208311796188354, 'learning_rate': 3.887714790243566e-05, 'epoch': 1.34}

 13%|█▎        | 5526/41250 [13:21:29<86:01:02,  8.67s/it][2025-04-25 21:19:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:19:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.04 | bwd_microstep: 5697.67 | bwd_inner_microstep: 5646.18 | bwd_allreduce_microstep: 51.44 | step_microstep: 18.94
[2025-04-25 21:19:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.04 | bwd: 5697.69 | bwd_inner: 5646.18 | bwd_allreduce: 51.46 | step: 18.95

 13%|█▎        | 5527/41250 [13:21:38<85:49:29,  8.65s/it]
                                                          
{'loss': 0.2225, 'grad_norm': 2.131061553955078, 'learning_rate': 3.887662908233007e-05, 'epoch': 1.34}

 13%|█▎        | 5527/41250 [13:21:38<85:49:29,  8.65s/it][2025-04-25 21:19:21,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 21:19:21,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.06 | bwd_microstep: 5697.71 | bwd_inner_microstep: 5684.94 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.64
[2025-04-25 21:19:21,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.06 | bwd: 5697.73 | bwd_inner: 5684.94 | bwd_allreduce: 12.75 | step: 18.64

 13%|█▎        | 5528/41250 [13:21:46<85:45:39,  8.64s/it]
                                                          
{'loss': 0.205, 'grad_norm': 2.88101863861084, 'learning_rate': 3.887611014585377e-05, 'epoch': 1.34}

 13%|█▎        | 5528/41250 [13:21:46<85:45:39,  8.64s/it][2025-04-25 21:19:30,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 1.04
[2025-04-25 21:19:30,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.30 | bwd_microstep: 5743.53 | bwd_inner_microstep: 5707.92 | bwd_allreduce_microstep: 35.55 | step_microstep: 19.82
[2025-04-25 21:19:30,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.30 | bwd: 5743.55 | bwd_inner: 5707.92 | bwd_allreduce: 35.58 | step: 19.82

 13%|█▎        | 5529/41250 [13:21:55<85:51:58,  8.65s/it]
                                                          
{'loss': 0.1095, 'grad_norm': 4.274804592132568, 'learning_rate': 3.8875591093009964e-05, 'epoch': 1.34}

 13%|█▎        | 5529/41250 [13:21:55<85:51:58,  8.65s/it][2025-04-25 21:19:38,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:19:38,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5731.51 | bwd_inner_microstep: 5688.97 | bwd_allreduce_microstep: 42.49 | step_microstep: 18.79
[2025-04-25 21:19:38,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.30 | bwd: 5731.52 | bwd_inner: 5688.97 | bwd_allreduce: 42.50 | step: 18.79

 13%|█▎        | 5530/41250 [13:22:04<85:53:42,  8.66s/it]
                                                          
{'loss': 0.2012, 'grad_norm': 3.0580852031707764, 'learning_rate': 3.887507192380185e-05, 'epoch': 1.34}

 13%|█▎        | 5530/41250 [13:22:04<85:53:42,  8.66s/it][2025-04-25 21:19:47,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 21:19:47,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.23 | bwd_microstep: 5788.61 | bwd_inner_microstep: 5775.82 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.99
[2025-04-25 21:19:47,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.23 | bwd: 5788.63 | bwd_inner: 5775.82 | bwd_allreduce: 12.76 | step: 19.00

 13%|█▎        | 5531/41250 [13:22:13<86:10:58,  8.69s/it]
                                                          
{'loss': 0.1165, 'grad_norm': 2.864020347595215, 'learning_rate': 3.887455263823263e-05, 'epoch': 1.34}

 13%|█▎        | 5531/41250 [13:22:13<86:10:58,  8.69s/it][2025-04-25 21:19:56,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:19:56,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.04 | bwd_microstep: 5713.93 | bwd_inner_microstep: 5691.68 | bwd_allreduce_microstep: 22.21 | step_microstep: 18.35
[2025-04-25 21:19:56,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.04 | bwd: 5713.94 | bwd_inner: 5691.68 | bwd_allreduce: 22.22 | step: 18.35

 13%|█▎        | 5532/41250 [13:22:21<86:02:04,  8.67s/it]
                                                          
{'loss': 0.2305, 'grad_norm': 1.948059320449829, 'learning_rate': 3.88740332363055e-05, 'epoch': 1.34}

 13%|█▎        | 5532/41250 [13:22:21<86:02:04,  8.67s/it][2025-04-25 21:20:05,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:20:05,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.57 | bwd_microstep: 5736.55 | bwd_inner_microstep: 5677.01 | bwd_allreduce_microstep: 59.49 | step_microstep: 18.83
[2025-04-25 21:20:05,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.57 | bwd: 5736.56 | bwd_inner: 5677.01 | bwd_allreduce: 59.51 | step: 18.84

 13%|█▎        | 5533/41250 [13:22:30<86:02:04,  8.67s/it]
                                                          
{'loss': 0.2975, 'grad_norm': 3.5976412296295166, 'learning_rate': 3.887351371802368e-05, 'epoch': 1.34}

 13%|█▎        | 5533/41250 [13:22:30<86:02:04,  8.67s/it][2025-04-25 21:20:13,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 21:20:13,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.32 | bwd_microstep: 5690.16 | bwd_inner_microstep: 5655.69 | bwd_allreduce_microstep: 34.42 | step_microstep: 19.27
[2025-04-25 21:20:13,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.32 | bwd: 5690.17 | bwd_inner: 5655.69 | bwd_allreduce: 34.44 | step: 19.27

 13%|█▎        | 5534/41250 [13:22:38<85:49:39,  8.65s/it]
                                                          
{'loss': 0.2349, 'grad_norm': 2.965714454650879, 'learning_rate': 3.8872994083390355e-05, 'epoch': 1.34}

 13%|█▎        | 5534/41250 [13:22:38<85:49:39,  8.65s/it][2025-04-25 21:20:22,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 21:20:22,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.72 | bwd_microstep: 5731.18 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 30.96 | step_microstep: 19.10
[2025-04-25 21:20:22,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.72 | bwd: 5731.20 | bwd_inner: 5700.17 | bwd_allreduce: 30.98 | step: 19.10

 13%|█▎        | 5535/41250 [13:22:47<85:53:12,  8.66s/it]
                                                          
{'loss': 0.0106, 'grad_norm': 0.1783975064754486, 'learning_rate': 3.887247433240873e-05, 'epoch': 1.34}

 13%|█▎        | 5535/41250 [13:22:47<85:53:12,  8.66s/it][2025-04-25 21:20:31,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 21:20:31,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.24 | bwd_microstep: 5788.30 | bwd_inner_microstep: 5652.53 | bwd_allreduce_microstep: 135.71 | step_microstep: 18.97
[2025-04-25 21:20:31,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.24 | bwd: 5788.31 | bwd_inner: 5652.53 | bwd_allreduce: 135.74 | step: 18.97

 13%|█▎        | 5536/41250 [13:22:56<86:00:35,  8.67s/it]
                                                          
{'loss': 0.1923, 'grad_norm': 3.8296260833740234, 'learning_rate': 3.887195446508202e-05, 'epoch': 1.34}

 13%|█▎        | 5536/41250 [13:22:56<86:00:35,  8.67s/it][2025-04-25 21:20:39,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:20:39,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5753.09 | bwd_inner_microstep: 5712.74 | bwd_allreduce_microstep: 40.31 | step_microstep: 18.52
[2025-04-25 21:20:39,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.14 | bwd: 5753.10 | bwd_inner: 5712.74 | bwd_allreduce: 40.32 | step: 18.53

 13%|█▎        | 5537/41250 [13:23:05<86:04:11,  8.68s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 3.2234268188476562, 'learning_rate': 3.887143448141341e-05, 'epoch': 1.34}

 13%|█▎        | 5537/41250 [13:23:05<86:04:11,  8.68s/it][2025-04-25 21:20:48,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:20:48,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.18 | bwd_microstep: 5769.09 | bwd_inner_microstep: 5654.47 | bwd_allreduce_microstep: 114.57 | step_microstep: 18.64
[2025-04-25 21:20:48,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.18 | bwd: 5769.10 | bwd_inner: 5654.47 | bwd_allreduce: 114.59 | step: 18.64

 13%|█▎        | 5538/41250 [13:23:13<86:03:54,  8.68s/it]
                                                          
{'loss': 0.2457, 'grad_norm': 2.760890483856201, 'learning_rate': 3.887091438140613e-05, 'epoch': 1.34}

 13%|█▎        | 5538/41250 [13:23:13<86:03:54,  8.68s/it][2025-04-25 21:20:57,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 21:20:57,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5768.57 | bwd_inner_microstep: 5667.18 | bwd_allreduce_microstep: 101.34 | step_microstep: 18.83
[2025-04-25 21:20:57,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5768.58 | bwd_inner: 5667.18 | bwd_allreduce: 101.36 | step: 18.84

 13%|█▎        | 5539/41250 [13:23:22<86:05:23,  8.68s/it]
                                                          
{'loss': 0.2305, 'grad_norm': 3.2994790077209473, 'learning_rate': 3.887039416506337e-05, 'epoch': 1.34}

 13%|█▎        | 5539/41250 [13:23:22<86:05:23,  8.68s/it][2025-04-25 21:21:05,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 21:21:05,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.45 | bwd_microstep: 5738.47 | bwd_inner_microstep: 5710.92 | bwd_allreduce_microstep: 27.51 | step_microstep: 18.86
[2025-04-25 21:21:05,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.45 | bwd: 5738.49 | bwd_inner: 5710.92 | bwd_allreduce: 27.53 | step: 18.85

 13%|█▎        | 5540/41250 [13:23:31<86:05:15,  8.68s/it]
                                                          
{'loss': 0.2486, 'grad_norm': 3.8228609561920166, 'learning_rate': 3.886987383238834e-05, 'epoch': 1.34}

 13%|█▎        | 5540/41250 [13:23:31<86:05:15,  8.68s/it][2025-04-25 21:21:14,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 21:21:14,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.51 | bwd_microstep: 5760.33 | bwd_inner_microstep: 5665.91 | bwd_allreduce_microstep: 94.37 | step_microstep: 18.81
[2025-04-25 21:21:14,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.51 | bwd: 5760.34 | bwd_inner: 5665.91 | bwd_allreduce: 94.39 | step: 18.81

 13%|█▎        | 5541/41250 [13:23:39<86:04:43,  8.68s/it]
                                                          
{'loss': 0.0825, 'grad_norm': 2.0746686458587646, 'learning_rate': 3.886935338338425e-05, 'epoch': 1.34}

 13%|█▎        | 5541/41250 [13:23:39<86:04:43,  8.68s/it][2025-04-25 21:21:23,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:21:23,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.29 | bwd_microstep: 5773.14 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 109.88 | step_microstep: 18.66
[2025-04-25 21:21:23,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5773.15 | bwd_inner: 5663.21 | bwd_allreduce: 109.90 | step: 18.66

 13%|█▎        | 5542/41250 [13:23:48<86:05:56,  8.68s/it]
                                                          
{'loss': 0.0437, 'grad_norm': 1.3790768384933472, 'learning_rate': 3.886883281805431e-05, 'epoch': 1.34}

 13%|█▎        | 5542/41250 [13:23:48<86:05:56,  8.68s/it][2025-04-25 21:21:31,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:21:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.21 | bwd_microstep: 5782.08 | bwd_inner_microstep: 5665.73 | bwd_allreduce_microstep: 116.30 | step_microstep: 18.76
[2025-04-25 21:21:31,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.21 | bwd: 5782.09 | bwd_inner: 5665.73 | bwd_allreduce: 116.32 | step: 18.76

 13%|█▎        | 5543/41250 [13:23:57<86:08:37,  8.69s/it]
                                                          
{'loss': 0.0787, 'grad_norm': 1.313536524772644, 'learning_rate': 3.886831213640172e-05, 'epoch': 1.34}

 13%|█▎        | 5543/41250 [13:23:57<86:08:37,  8.69s/it][2025-04-25 21:21:40,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:21:40,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.78 | bwd_microstep: 5733.02 | bwd_inner_microstep: 5720.25 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.84
[2025-04-25 21:21:40,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.78 | bwd: 5733.03 | bwd_inner: 5720.25 | bwd_allreduce: 12.75 | step: 18.84

 13%|█▎        | 5544/41250 [13:24:05<86:06:02,  8.68s/it]
                                                          
{'loss': 0.1961, 'grad_norm': 2.2888612747192383, 'learning_rate': 3.886779133842971e-05, 'epoch': 1.34}

 13%|█▎        | 5544/41250 [13:24:05<86:06:02,  8.68s/it][2025-04-25 21:21:49,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:21:49,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.55 | bwd_microstep: 5764.83 | bwd_inner_microstep: 5660.43 | bwd_allreduce_microstep: 104.36 | step_microstep: 18.75
[2025-04-25 21:21:49,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.55 | bwd: 5764.84 | bwd_inner: 5660.43 | bwd_allreduce: 104.38 | step: 18.75

 13%|█▎        | 5545/41250 [13:24:14<86:05:47,  8.68s/it]
                                                          
{'loss': 0.1472, 'grad_norm': 2.335383415222168, 'learning_rate': 3.8867270424141466e-05, 'epoch': 1.34}

 13%|█▎        | 5545/41250 [13:24:14<86:05:47,  8.68s/it][2025-04-25 21:21:57,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 21:21:57,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.33 | bwd_microstep: 5771.82 | bwd_inner_microstep: 5658.47 | bwd_allreduce_microstep: 113.29 | step_microstep: 19.06
[2025-04-25 21:21:57,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.33 | bwd: 5771.84 | bwd_inner: 5658.47 | bwd_allreduce: 113.32 | step: 19.06

 13%|█▎        | 5546/41250 [13:24:23<86:06:56,  8.68s/it]
                                                          
{'loss': 0.0791, 'grad_norm': 1.3003511428833008, 'learning_rate': 3.886674939354022e-05, 'epoch': 1.34}

 13%|█▎        | 5546/41250 [13:24:23<86:06:56,  8.68s/it][2025-04-25 21:22:06,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.29 | optimizer_step: 1.05
[2025-04-25 21:22:06,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.63 | bwd_microstep: 5747.29 | bwd_inner_microstep: 5725.60 | bwd_allreduce_microstep: 21.63 | step_microstep: 19.89
[2025-04-25 21:22:06,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.63 | bwd: 5747.31 | bwd_inner: 5725.60 | bwd_allreduce: 21.66 | step: 19.89

 13%|█▎        | 5547/41250 [13:24:31<86:07:32,  8.68s/it]
                                                          
{'loss': 0.1619, 'grad_norm': 2.5763514041900635, 'learning_rate': 3.886622824662917e-05, 'epoch': 1.34}

 13%|█▎        | 5547/41250 [13:24:31<86:07:32,  8.68s/it][2025-04-25 21:22:15,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:22:15,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.02 | bwd_microstep: 5717.66 | bwd_inner_microstep: 5704.90 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.73
[2025-04-25 21:22:15,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.02 | bwd: 5717.68 | bwd_inner: 5704.90 | bwd_allreduce: 12.73 | step: 18.74

 13%|█▎        | 5548/41250 [13:24:40<86:02:29,  8.68s/it]
                                                          
{'loss': 0.5752, 'grad_norm': 3.633366584777832, 'learning_rate': 3.886570698341153e-05, 'epoch': 1.34}

 13%|█▎        | 5548/41250 [13:24:40<86:02:29,  8.68s/it][2025-04-25 21:22:23,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 21:22:23,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.39 | bwd_microstep: 5798.58 | bwd_inner_microstep: 5658.31 | bwd_allreduce_microstep: 140.21 | step_microstep: 18.98
[2025-04-25 21:22:23,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.39 | bwd: 5798.59 | bwd_inner: 5658.31 | bwd_allreduce: 140.24 | step: 18.98

 13%|█▎        | 5549/41250 [13:24:49<86:09:47,  8.69s/it]
                                                          
{'loss': 0.0297, 'grad_norm': 0.6086517572402954, 'learning_rate': 3.886518560389053e-05, 'epoch': 1.35}

 13%|█▎        | 5549/41250 [13:24:49<86:09:47,  8.69s/it][2025-04-25 21:22:32,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:22:32,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5714.12 | bwd_inner_microstep: 5667.74 | bwd_allreduce_microstep: 46.34 | step_microstep: 18.85
[2025-04-25 21:22:32,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5714.14 | bwd_inner: 5667.74 | bwd_allreduce: 46.36 | step: 18.85

 13%|█▎        | 5550/41250 [13:24:57<86:01:42,  8.68s/it]
                                                          
{'loss': 0.2043, 'grad_norm': 2.834888219833374, 'learning_rate': 3.886466410806936e-05, 'epoch': 1.35}

 13%|█▎        | 5550/41250 [13:24:57<86:01:42,  8.68s/it][2025-04-25 21:22:41,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:22:41,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.49 | bwd_microstep: 5755.28 | bwd_inner_microstep: 5712.50 | bwd_allreduce_microstep: 42.73 | step_microstep: 18.77
[2025-04-25 21:22:41,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.49 | bwd: 5755.29 | bwd_inner: 5712.50 | bwd_allreduce: 42.75 | step: 18.77

 13%|█▎        | 5551/41250 [13:25:06<86:07:04,  8.68s/it]
                                                          
{'loss': 0.2453, 'grad_norm': 3.4232537746429443, 'learning_rate': 3.886414249595125e-05, 'epoch': 1.35}

 13%|█▎        | 5551/41250 [13:25:06<86:07:04,  8.68s/it][2025-04-25 21:22:49,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:22:49,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.20 | bwd_microstep: 5774.40 | bwd_inner_microstep: 5675.42 | bwd_allreduce_microstep: 98.93 | step_microstep: 18.79
[2025-04-25 21:22:49,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.20 | bwd: 5774.41 | bwd_inner: 5675.42 | bwd_allreduce: 98.95 | step: 18.80

 13%|█▎        | 5552/41250 [13:25:15<86:08:08,  8.69s/it]
                                                          
{'loss': 0.2157, 'grad_norm': 3.613065242767334, 'learning_rate': 3.8863620767539406e-05, 'epoch': 1.35}

 13%|█▎        | 5552/41250 [13:25:15<86:08:08,  8.69s/it][2025-04-25 21:22:58,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:22:58,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.10 | bwd_microstep: 5731.36 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 73.76 | step_microstep: 18.49
[2025-04-25 21:22:58,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.10 | bwd: 5731.37 | bwd_inner: 5657.55 | bwd_allreduce: 73.78 | step: 18.49

 13%|█▎        | 5553/41250 [13:25:23<86:01:25,  8.68s/it]
                                                          
{'loss': 0.1398, 'grad_norm': 2.1698226928710938, 'learning_rate': 3.8863098922837055e-05, 'epoch': 1.35}

 13%|█▎        | 5553/41250 [13:25:23<86:01:25,  8.68s/it][2025-04-25 21:23:07,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 21:23:07,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.11 | bwd_microstep: 5781.75 | bwd_inner_microstep: 5703.87 | bwd_allreduce_microstep: 77.83 | step_microstep: 18.94
[2025-04-25 21:23:07,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.11 | bwd: 5781.76 | bwd_inner: 5703.87 | bwd_allreduce: 77.85 | step: 18.95

 13%|█▎        | 5554/41250 [13:25:32<86:08:40,  8.69s/it]
                                                          
{'loss': 0.1704, 'grad_norm': 1.6090606451034546, 'learning_rate': 3.886257696184741e-05, 'epoch': 1.35}

 13%|█▎        | 5554/41250 [13:25:32<86:08:40,  8.69s/it][2025-04-25 21:23:15,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:23:15,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5714.36 | bwd_inner_microstep: 5675.69 | bwd_allreduce_microstep: 38.62 | step_microstep: 18.61
[2025-04-25 21:23:15,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5714.37 | bwd_inner: 5675.69 | bwd_allreduce: 38.64 | step: 18.62

 13%|█▎        | 5555/41250 [13:25:41<85:58:30,  8.67s/it]
                                                          
{'loss': 0.2387, 'grad_norm': 2.0865395069122314, 'learning_rate': 3.886205488457369e-05, 'epoch': 1.35}

 13%|█▎        | 5555/41250 [13:25:41<85:58:30,  8.67s/it][2025-04-25 21:23:24,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:23:24,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.58 | bwd_microstep: 5796.94 | bwd_inner_microstep: 5659.29 | bwd_allreduce_microstep: 137.60 | step_microstep: 18.75
[2025-04-25 21:23:24,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.58 | bwd: 5796.96 | bwd_inner: 5659.29 | bwd_allreduce: 137.62 | step: 18.75

 13%|█▎        | 5556/41250 [13:25:49<86:04:35,  8.68s/it]
                                                          
{'loss': 0.1746, 'grad_norm': 3.1179513931274414, 'learning_rate': 3.88615326910191e-05, 'epoch': 1.35}

 13%|█▎        | 5556/41250 [13:25:49<86:04:35,  8.68s/it][2025-04-25 21:23:33,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:23:33,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5790.15 | bwd_inner_microstep: 5710.25 | bwd_allreduce_microstep: 79.85 | step_microstep: 18.63
[2025-04-25 21:23:33,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5790.17 | bwd_inner: 5710.25 | bwd_allreduce: 79.87 | step: 18.63

 13%|█▎        | 5557/41250 [13:25:58<86:12:18,  8.69s/it]
                                                          
{'loss': 0.2462, 'grad_norm': 3.1114351749420166, 'learning_rate': 3.886101038118688e-05, 'epoch': 1.35}

 13%|█▎        | 5557/41250 [13:25:58<86:12:18,  8.69s/it][2025-04-25 21:23:42,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:23:42,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5764.99 | bwd_inner_microstep: 5705.83 | bwd_allreduce_microstep: 59.11 | step_microstep: 18.90
[2025-04-25 21:23:42,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5765.01 | bwd_inner: 5705.83 | bwd_allreduce: 59.13 | step: 18.91

 13%|█▎        | 5558/41250 [13:26:07<86:12:41,  8.70s/it]
                                                          
{'loss': 0.0692, 'grad_norm': 1.680681824684143, 'learning_rate': 3.886048795508024e-05, 'epoch': 1.35}

 13%|█▎        | 5558/41250 [13:26:07<86:12:41,  8.70s/it][2025-04-25 21:23:50,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:23:50,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5730.05 | bwd_inner_microstep: 5655.09 | bwd_allreduce_microstep: 74.91 | step_microstep: 18.74
[2025-04-25 21:23:50,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5730.06 | bwd_inner: 5655.09 | bwd_allreduce: 74.93 | step: 18.74

 13%|█▎        | 5559/41250 [13:26:16<86:02:18,  8.68s/it]
                                                          
{'loss': 0.0768, 'grad_norm': 1.6565593481063843, 'learning_rate': 3.8859965412702406e-05, 'epoch': 1.35}

 13%|█▎        | 5559/41250 [13:26:16<86:02:18,  8.68s/it][2025-04-25 21:23:59,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 21:23:59,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.38 | bwd_microstep: 5780.87 | bwd_inner_microstep: 5656.68 | bwd_allreduce_microstep: 124.14 | step_microstep: 19.17
[2025-04-25 21:23:59,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.38 | bwd: 5780.88 | bwd_inner: 5656.68 | bwd_allreduce: 124.16 | step: 19.17

 13%|█▎        | 5560/41250 [13:26:24<86:05:43,  8.68s/it]
                                                          
{'loss': 0.0212, 'grad_norm': 0.5249252915382385, 'learning_rate': 3.885944275405659e-05, 'epoch': 1.35}

 13%|█▎        | 5560/41250 [13:26:24<86:05:43,  8.68s/it][2025-04-25 21:24:08,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 21:24:08,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5719.28 | bwd_inner_microstep: 5651.07 | bwd_allreduce_microstep: 68.16 | step_microstep: 18.69
[2025-04-25 21:24:08,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5719.29 | bwd_inner: 5651.07 | bwd_allreduce: 68.18 | step: 18.70

 13%|█▎        | 5561/41250 [13:26:33<85:55:14,  8.67s/it]
                                                          
{'loss': 0.1919, 'grad_norm': 1.2650532722473145, 'learning_rate': 3.885891997914602e-05, 'epoch': 1.35}

 13%|█▎        | 5561/41250 [13:26:33<85:55:14,  8.67s/it][2025-04-25 21:24:16,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 21:24:16,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5785.76 | bwd_inner_microstep: 5644.03 | bwd_allreduce_microstep: 141.69 | step_microstep: 18.94
[2025-04-25 21:24:16,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5785.78 | bwd_inner: 5644.03 | bwd_allreduce: 141.71 | step: 18.95

 13%|█▎        | 5562/41250 [13:26:42<86:00:05,  8.68s/it]
                                                          
{'loss': 0.0541, 'grad_norm': 0.892282247543335, 'learning_rate': 3.885839708797392e-05, 'epoch': 1.35}

 13%|█▎        | 5562/41250 [13:26:42<86:00:05,  8.68s/it][2025-04-25 21:24:25,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 21:24:25,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.74 | bwd_microstep: 5749.65 | bwd_inner_microstep: 5707.20 | bwd_allreduce_microstep: 42.39 | step_microstep: 18.86
[2025-04-25 21:24:25,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.74 | bwd: 5749.66 | bwd_inner: 5707.20 | bwd_allreduce: 42.42 | step: 18.86

 13%|█▎        | 5563/41250 [13:26:50<86:01:09,  8.68s/it]
                                                          
{'loss': 0.1442, 'grad_norm': 3.5981218814849854, 'learning_rate': 3.8857874080543504e-05, 'epoch': 1.35}

 13%|█▎        | 5563/41250 [13:26:50<86:01:09,  8.68s/it][2025-04-25 21:24:34,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:24:34,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.31 | bwd_microstep: 5768.31 | bwd_inner_microstep: 5656.47 | bwd_allreduce_microstep: 111.81 | step_microstep: 18.52
[2025-04-25 21:24:34,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.31 | bwd: 5768.33 | bwd_inner: 5656.47 | bwd_allreduce: 111.82 | step: 18.53

 13%|█▎        | 5564/41250 [13:26:59<86:01:25,  8.68s/it]
                                                          
{'loss': 0.107, 'grad_norm': 1.8996095657348633, 'learning_rate': 3.885735095685801e-05, 'epoch': 1.35}

 13%|█▎        | 5564/41250 [13:26:59<86:01:25,  8.68s/it][2025-04-25 21:24:42,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 21:24:42,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.90 | bwd_microstep: 5715.99 | bwd_inner_microstep: 5659.39 | bwd_allreduce_microstep: 56.56 | step_microstep: 18.71
[2025-04-25 21:24:42,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.90 | bwd: 5716.01 | bwd_inner: 5659.39 | bwd_allreduce: 56.58 | step: 18.71

 13%|█▎        | 5565/41250 [13:27:08<85:52:55,  8.66s/it]
                                                          
{'loss': 0.2075, 'grad_norm': 1.5018328428268433, 'learning_rate': 3.8856827716920654e-05, 'epoch': 1.35}

 13%|█▎        | 5565/41250 [13:27:08<85:52:55,  8.66s/it][2025-04-25 21:24:51,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:24:51,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.11 | bwd_microstep: 5737.06 | bwd_inner_microstep: 5698.78 | bwd_allreduce_microstep: 38.23 | step_microstep: 18.88
[2025-04-25 21:24:51,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.11 | bwd: 5737.07 | bwd_inner: 5698.78 | bwd_allreduce: 38.24 | step: 18.89

 13%|█▎        | 5566/41250 [13:27:16<85:54:42,  8.67s/it]
                                                          
{'loss': 0.0599, 'grad_norm': 2.3577446937561035, 'learning_rate': 3.885630436073466e-05, 'epoch': 1.35}

 13%|█▎        | 5566/41250 [13:27:16<85:54:42,  8.67s/it][2025-04-25 21:25:00,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:25:00,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.39 | bwd_microstep: 5790.00 | bwd_inner_microstep: 5777.18 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.74
[2025-04-25 21:25:00,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.39 | bwd: 5790.02 | bwd_inner: 5777.18 | bwd_allreduce: 12.79 | step: 18.74

 13%|█▎        | 5567/41250 [13:27:25<86:12:04,  8.70s/it]
                                                          
{'loss': 0.1543, 'grad_norm': 2.3263967037200928, 'learning_rate': 3.8855780888303266e-05, 'epoch': 1.35}

 13%|█▎        | 5567/41250 [13:27:25<86:12:04,  8.70s/it][2025-04-25 21:25:08,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:25:08,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.50 | bwd_microstep: 5708.60 | bwd_inner_microstep: 5651.38 | bwd_allreduce_microstep: 57.18 | step_microstep: 18.84
[2025-04-25 21:25:08,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.50 | bwd: 5708.62 | bwd_inner: 5651.38 | bwd_allreduce: 57.20 | step: 18.84

 13%|█▎        | 5568/41250 [13:27:34<85:57:01,  8.67s/it]
                                                          
{'loss': 0.1221, 'grad_norm': 3.1319596767425537, 'learning_rate': 3.885525729962969e-05, 'epoch': 1.35}

 13%|█▎        | 5568/41250 [13:27:34<85:57:01,  8.67s/it][2025-04-25 21:25:17,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:25:17,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5748.50 | bwd_inner_microstep: 5656.13 | bwd_allreduce_microstep: 92.33 | step_microstep: 18.77
[2025-04-25 21:25:17,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5748.51 | bwd_inner: 5656.13 | bwd_allreduce: 92.34 | step: 18.78

 14%|█▎        | 5569/41250 [13:27:42<85:55:31,  8.67s/it]
                                                          
{'loss': 0.0627, 'grad_norm': 1.277498722076416, 'learning_rate': 3.885473359471716e-05, 'epoch': 1.35}

 14%|█▎        | 5569/41250 [13:27:42<85:55:31,  8.67s/it][2025-04-25 21:25:26,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:25:26,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.11 | bwd_microstep: 5700.88 | bwd_inner_microstep: 5688.06 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.77
[2025-04-25 21:25:26,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.11 | bwd: 5700.89 | bwd_inner: 5688.06 | bwd_allreduce: 12.79 | step: 18.77

 14%|█▎        | 5570/41250 [13:27:51<85:48:07,  8.66s/it]
                                                          
{'loss': 0.0432, 'grad_norm': 0.6865904331207275, 'learning_rate': 3.88542097735689e-05, 'epoch': 1.35}

 14%|█▎        | 5570/41250 [13:27:51<85:48:07,  8.66s/it][2025-04-25 21:25:34,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 21:25:34,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.43 | bwd_microstep: 5712.42 | bwd_inner_microstep: 5699.57 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.62
[2025-04-25 21:25:34,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.43 | bwd: 5712.44 | bwd_inner: 5699.57 | bwd_allreduce: 12.82 | step: 18.63

 14%|█▎        | 5571/41250 [13:28:00<85:46:37,  8.65s/it]
                                                          
{'loss': 0.1115, 'grad_norm': 1.7200937271118164, 'learning_rate': 3.885368583618816e-05, 'epoch': 1.35}

 14%|█▎        | 5571/41250 [13:28:00<85:46:37,  8.65s/it][2025-04-25 21:25:43,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 21:25:43,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.00 | bwd_microstep: 6013.48 | bwd_inner_microstep: 5700.62 | bwd_allreduce_microstep: 312.82 | step_microstep: 19.18
[2025-04-25 21:25:43,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.00 | bwd: 6013.50 | bwd_inner: 5700.62 | bwd_allreduce: 312.84 | step: 19.18

 14%|█▎        | 5572/41250 [13:28:08<86:39:06,  8.74s/it]
                                                          
{'loss': 0.1158, 'grad_norm': 1.838423728942871, 'learning_rate': 3.885316178257814e-05, 'epoch': 1.35}

 14%|█▎        | 5572/41250 [13:28:08<86:39:06,  8.74s/it][2025-04-25 21:25:52,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:25:52,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.86 | bwd_microstep: 5786.86 | bwd_inner_microstep: 5645.06 | bwd_allreduce_microstep: 141.76 | step_microstep: 18.57
[2025-04-25 21:25:52,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.86 | bwd: 5786.88 | bwd_inner: 5645.06 | bwd_allreduce: 141.78 | step: 18.57

 14%|█▎        | 5573/41250 [13:28:17<86:30:25,  8.73s/it]
                                                          
{'loss': 0.1738, 'grad_norm': 2.2174878120422363, 'learning_rate': 3.885263761274209e-05, 'epoch': 1.35}

 14%|█▎        | 5573/41250 [13:28:17<86:30:25,  8.73s/it][2025-04-25 21:26:01,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:26:01,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.18 | bwd_microstep: 5756.65 | bwd_inner_microstep: 5654.07 | bwd_allreduce_microstep: 102.53 | step_microstep: 18.55
[2025-04-25 21:26:01,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.18 | bwd: 5756.66 | bwd_inner: 5654.07 | bwd_allreduce: 102.55 | step: 18.55

 14%|█▎        | 5574/41250 [13:28:26<86:18:40,  8.71s/it]
                                                          
{'loss': 0.0585, 'grad_norm': 1.440537929534912, 'learning_rate': 3.8852113326683236e-05, 'epoch': 1.35}

 14%|█▎        | 5574/41250 [13:28:26<86:18:40,  8.71s/it][2025-04-25 21:26:09,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 21:26:09,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.70 | bwd_microstep: 5860.01 | bwd_inner_microstep: 5686.57 | bwd_allreduce_microstep: 173.39 | step_microstep: 18.73
[2025-04-25 21:26:09,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.70 | bwd: 5860.02 | bwd_inner: 5686.57 | bwd_allreduce: 173.41 | step: 18.73

 14%|█▎        | 5575/41250 [13:28:35<86:32:08,  8.73s/it]
                                                          
{'loss': 0.1101, 'grad_norm': 2.3798370361328125, 'learning_rate': 3.885158892440481e-05, 'epoch': 1.35}

 14%|█▎        | 5575/41250 [13:28:35<86:32:08,  8.73s/it][2025-04-25 21:26:18,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:26:18,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5695.20 | bwd_inner_microstep: 5682.44 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.83
[2025-04-25 21:26:18,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5695.21 | bwd_inner: 5682.44 | bwd_allreduce: 12.74 | step: 18.83

 14%|█▎        | 5576/41250 [13:28:43<86:14:05,  8.70s/it]
                                                          
{'loss': 0.133, 'grad_norm': 2.703328847885132, 'learning_rate': 3.885106440591006e-05, 'epoch': 1.35}

 14%|█▎        | 5576/41250 [13:28:43<86:14:05,  8.70s/it][2025-04-25 21:26:27,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 21:26:27,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.99 | bwd_microstep: 5784.56 | bwd_inner_microstep: 5771.87 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.75
[2025-04-25 21:26:27,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.99 | bwd: 5784.58 | bwd_inner: 5771.87 | bwd_allreduce: 12.67 | step: 18.75

 14%|█▎        | 5577/41250 [13:28:52<86:22:19,  8.72s/it]
                                                          
{'loss': 0.1849, 'grad_norm': 1.9333152770996094, 'learning_rate': 3.885053977120219e-05, 'epoch': 1.35}

 14%|█▎        | 5577/41250 [13:28:52<86:22:19,  8.72s/it][2025-04-25 21:26:35,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:26:35,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.34 | bwd_microstep: 5767.19 | bwd_inner_microstep: 5649.89 | bwd_allreduce_microstep: 117.25 | step_microstep: 18.40
[2025-04-25 21:26:35,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.34 | bwd: 5767.20 | bwd_inner: 5649.89 | bwd_allreduce: 117.26 | step: 18.40

 14%|█▎        | 5578/41250 [13:29:01<86:14:02,  8.70s/it]
                                                          
{'loss': 0.1415, 'grad_norm': 3.4903173446655273, 'learning_rate': 3.885001502028446e-05, 'epoch': 1.35}

 14%|█▎        | 5578/41250 [13:29:01<86:14:02,  8.70s/it][2025-04-25 21:26:44,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.24 | optimizer_step: 1.04
[2025-04-25 21:26:44,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.42 | bwd_microstep: 5758.70 | bwd_inner_microstep: 5658.04 | bwd_allreduce_microstep: 100.59 | step_microstep: 19.68
[2025-04-25 21:26:44,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.42 | bwd: 5758.71 | bwd_inner: 5658.04 | bwd_allreduce: 100.62 | step: 19.68

 14%|█▎        | 5579/41250 [13:29:09<86:08:41,  8.69s/it]
                                                          
{'loss': 0.1173, 'grad_norm': 1.816239833831787, 'learning_rate': 3.884949015316009e-05, 'epoch': 1.35}

 14%|█▎        | 5579/41250 [13:29:09<86:08:41,  8.69s/it][2025-04-25 21:26:53,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 21:26:53,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.85 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5646.05 | bwd_allreduce_microstep: 120.95 | step_microstep: 18.94
[2025-04-25 21:26:53,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.85 | bwd: 5767.05 | bwd_inner: 5646.05 | bwd_allreduce: 120.96 | step: 18.95

 14%|█▎        | 5580/41250 [13:29:18<86:05:29,  8.69s/it]
                                                          
{'loss': 0.0802, 'grad_norm': 1.1241098642349243, 'learning_rate': 3.884896516983232e-05, 'epoch': 1.35}

 14%|█▎        | 5580/41250 [13:29:18<86:05:29,  8.69s/it][2025-04-25 21:27:01,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:27:01,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5698.11 | bwd_inner_microstep: 5685.29 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.61
[2025-04-25 21:27:01,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.84 | bwd: 5698.12 | bwd_inner: 5685.29 | bwd_allreduce: 12.78 | step: 18.61

 14%|█▎        | 5581/41250 [13:29:27<85:55:03,  8.67s/it]
                                                          
{'loss': 0.1845, 'grad_norm': 4.35075044631958, 'learning_rate': 3.884844007030439e-05, 'epoch': 1.35}

 14%|█▎        | 5581/41250 [13:29:27<85:55:03,  8.67s/it][2025-04-25 21:27:10,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:27:10,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.60 | bwd_microstep: 5758.44 | bwd_inner_microstep: 5641.62 | bwd_allreduce_microstep: 116.77 | step_microstep: 18.94
[2025-04-25 21:27:10,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.60 | bwd: 5758.45 | bwd_inner: 5641.62 | bwd_allreduce: 116.79 | step: 18.95

 14%|█▎        | 5582/41250 [13:29:35<85:53:48,  8.67s/it]
                                                          
{'loss': 0.1141, 'grad_norm': 2.520479917526245, 'learning_rate': 3.884791485457953e-05, 'epoch': 1.35}

 14%|█▎        | 5582/41250 [13:29:35<85:53:48,  8.67s/it][2025-04-25 21:27:19,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:27:19,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.13 | bwd_microstep: 5731.41 | bwd_inner_microstep: 5685.07 | bwd_allreduce_microstep: 46.29 | step_microstep: 18.38
[2025-04-25 21:27:19,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.13 | bwd: 5731.42 | bwd_inner: 5685.07 | bwd_allreduce: 46.31 | step: 18.39

 14%|█▎        | 5583/41250 [13:29:44<85:51:42,  8.67s/it]
                                                          
{'loss': 0.1906, 'grad_norm': 1.4924067258834839, 'learning_rate': 3.884738952266099e-05, 'epoch': 1.35}

 14%|█▎        | 5583/41250 [13:29:44<85:51:42,  8.67s/it][2025-04-25 21:27:27,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 21:27:27,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.75 | bwd_microstep: 5722.92 | bwd_inner_microstep: 5709.86 | bwd_allreduce_microstep: 13.02 | step_microstep: 19.46
[2025-04-25 21:27:27,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.75 | bwd: 5722.94 | bwd_inner: 5709.86 | bwd_allreduce: 13.04 | step: 19.46

 14%|█▎        | 5584/41250 [13:29:53<85:50:01,  8.66s/it]
                                                          
{'loss': 0.1322, 'grad_norm': 6.728465557098389, 'learning_rate': 3.8846864074551995e-05, 'epoch': 1.35}

 14%|█▎        | 5584/41250 [13:29:53<85:50:01,  8.66s/it][2025-04-25 21:27:36,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.08 | optimizer_step: 0.94
[2025-04-25 21:27:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.96 | bwd_microstep: 5853.83 | bwd_inner_microstep: 5672.65 | bwd_allreduce_microstep: 181.12 | step_microstep: 19.27
[2025-04-25 21:27:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.96 | bwd: 5853.85 | bwd_inner: 5672.65 | bwd_allreduce: 181.15 | step: 19.27

 14%|█▎        | 5585/41250 [13:30:01<86:11:06,  8.70s/it]
                                                          
{'loss': 0.0825, 'grad_norm': 1.4726125001907349, 'learning_rate': 3.884633851025579e-05, 'epoch': 1.35}

 14%|█▎        | 5585/41250 [13:30:01<86:11:06,  8.70s/it][2025-04-25 21:27:45,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.25 | optimizer_step: 0.91
[2025-04-25 21:27:45,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.93 | bwd_microstep: 5698.67 | bwd_inner_microstep: 5685.20 | bwd_allreduce_microstep: 13.40 | step_microstep: 19.48
[2025-04-25 21:27:45,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.93 | bwd: 5698.68 | bwd_inner: 5685.20 | bwd_allreduce: 13.43 | step: 19.48

 14%|█▎        | 5586/41250 [13:30:10<85:58:02,  8.68s/it]
                                                          
{'loss': 0.0845, 'grad_norm': 1.978248119354248, 'learning_rate': 3.884581282977561e-05, 'epoch': 1.35}

 14%|█▎        | 5586/41250 [13:30:10<85:58:02,  8.68s/it][2025-04-25 21:27:53,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.24 | optimizer_step: 1.03
[2025-04-25 21:27:53,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.17 | bwd_microstep: 5748.19 | bwd_inner_microstep: 5686.73 | bwd_allreduce_microstep: 61.41 | step_microstep: 19.43
[2025-04-25 21:27:53,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.17 | bwd: 5748.21 | bwd_inner: 5686.73 | bwd_allreduce: 61.43 | step: 19.43

 14%|█▎        | 5587/41250 [13:30:19<85:58:14,  8.68s/it]
                                                          
{'loss': 0.2495, 'grad_norm': 3.417552947998047, 'learning_rate': 3.884528703311471e-05, 'epoch': 1.35}

 14%|█▎        | 5587/41250 [13:30:19<85:58:14,  8.68s/it][2025-04-25 21:28:02,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:28:02,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5758.54 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 65.64 | step_microstep: 19.02
[2025-04-25 21:28:02,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5758.55 | bwd_inner: 5692.86 | bwd_allreduce: 65.65 | step: 19.03

 14%|█▎        | 5588/41250 [13:30:27<86:01:57,  8.68s/it]
                                                          
{'loss': 0.1455, 'grad_norm': 3.415832757949829, 'learning_rate': 3.8844761120276325e-05, 'epoch': 1.35}

 14%|█▎        | 5588/41250 [13:30:27<86:01:57,  8.68s/it][2025-04-25 21:28:11,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:28:11,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5726.74 | bwd_inner_microstep: 5700.05 | bwd_allreduce_microstep: 26.63 | step_microstep: 18.76
[2025-04-25 21:28:11,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5726.76 | bwd_inner: 5700.05 | bwd_allreduce: 26.65 | step: 18.76

 14%|█▎        | 5589/41250 [13:30:36<85:58:16,  8.68s/it]
                                                          
{'loss': 0.1193, 'grad_norm': 1.8688690662384033, 'learning_rate': 3.8844235091263686e-05, 'epoch': 1.35}

 14%|█▎        | 5589/41250 [13:30:36<85:58:16,  8.68s/it][2025-04-25 21:28:19,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 21:28:19,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5713.23 | bwd_inner_microstep: 5700.48 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.09
[2025-04-25 21:28:19,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5713.24 | bwd_inner: 5700.48 | bwd_allreduce: 12.72 | step: 19.09

 14%|█▎        | 5590/41250 [13:30:45<85:52:38,  8.67s/it]
                                                          
{'loss': 0.0401, 'grad_norm': 1.4990702867507935, 'learning_rate': 3.8843708946080044e-05, 'epoch': 1.36}

 14%|█▎        | 5590/41250 [13:30:45<85:52:38,  8.67s/it][2025-04-25 21:28:28,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 21:28:28,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.88 | bwd_microstep: 5690.70 | bwd_inner_microstep: 5656.50 | bwd_allreduce_microstep: 34.15 | step_microstep: 18.86
[2025-04-25 21:28:28,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.88 | bwd: 5690.72 | bwd_inner: 5656.50 | bwd_allreduce: 34.17 | step: 18.86

 14%|█▎        | 5591/41250 [13:30:53<85:40:32,  8.65s/it]
                                                          
{'loss': 0.0842, 'grad_norm': 2.3593077659606934, 'learning_rate': 3.8843182684728646e-05, 'epoch': 1.36}

 14%|█▎        | 5591/41250 [13:30:53<85:40:32,  8.65s/it][2025-04-25 21:28:37,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:28:37,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.50 | bwd_microstep: 5752.78 | bwd_inner_microstep: 5705.48 | bwd_allreduce_microstep: 47.26 | step_microstep: 18.75
[2025-04-25 21:28:37,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.50 | bwd: 5752.80 | bwd_inner: 5705.48 | bwd_allreduce: 47.28 | step: 18.75

 14%|█▎        | 5592/41250 [13:31:02<85:47:27,  8.66s/it]
                                                          
{'loss': 0.0656, 'grad_norm': 1.8343713283538818, 'learning_rate': 3.884265630721273e-05, 'epoch': 1.36}

 14%|█▎        | 5592/41250 [13:31:02<85:47:27,  8.66s/it][2025-04-25 21:28:45,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:28:45,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.09 | bwd_microstep: 5783.03 | bwd_inner_microstep: 5654.45 | bwd_allreduce_microstep: 128.54 | step_microstep: 18.70
[2025-04-25 21:28:45,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.09 | bwd: 5783.05 | bwd_inner: 5654.45 | bwd_allreduce: 128.56 | step: 18.70

 14%|█▎        | 5593/41250 [13:31:11<85:53:05,  8.67s/it]
                                                          
{'loss': 0.291, 'grad_norm': 2.758741617202759, 'learning_rate': 3.884212981353555e-05, 'epoch': 1.36}

 14%|█▎        | 5593/41250 [13:31:11<85:53:05,  8.67s/it][2025-04-25 21:28:54,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 21:28:54,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.75 | bwd_microstep: 5740.14 | bwd_inner_microstep: 5711.94 | bwd_allreduce_microstep: 28.15 | step_microstep: 19.12
[2025-04-25 21:28:54,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.75 | bwd: 5740.16 | bwd_inner: 5711.95 | bwd_allreduce: 28.17 | step: 19.12

 14%|█▎        | 5594/41250 [13:31:19<85:54:12,  8.67s/it]
                                                          
{'loss': 0.1629, 'grad_norm': 3.3252358436584473, 'learning_rate': 3.884160320370034e-05, 'epoch': 1.36}

 14%|█▎        | 5594/41250 [13:31:19<85:54:12,  8.67s/it][2025-04-25 21:29:03,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:29:03,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.71 | bwd_microstep: 5704.50 | bwd_inner_microstep: 5691.72 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.54
[2025-04-25 21:29:03,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.71 | bwd: 5704.51 | bwd_inner: 5691.72 | bwd_allreduce: 12.75 | step: 18.54

 14%|█▎        | 5595/41250 [13:31:28<85:49:26,  8.67s/it]
                                                          
{'loss': 0.6303, 'grad_norm': 3.311692714691162, 'learning_rate': 3.884107647771036e-05, 'epoch': 1.36}

 14%|█▎        | 5595/41250 [13:31:28<85:49:26,  8.67s/it][2025-04-25 21:29:11,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:29:11,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.64 | bwd_microstep: 5740.80 | bwd_inner_microstep: 5701.62 | bwd_allreduce_microstep: 39.13 | step_microstep: 18.54
[2025-04-25 21:29:11,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.64 | bwd: 5740.81 | bwd_inner: 5701.62 | bwd_allreduce: 39.15 | step: 18.54

 14%|█▎        | 5596/41250 [13:31:37<85:54:55,  8.67s/it]
                                                          
{'loss': 0.0599, 'grad_norm': 1.2621138095855713, 'learning_rate': 3.884054963556884e-05, 'epoch': 1.36}

 14%|█▎        | 5596/41250 [13:31:37<85:54:55,  8.67s/it][2025-04-25 21:29:20,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:29:20,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.33 | bwd_microstep: 5777.62 | bwd_inner_microstep: 5718.03 | bwd_allreduce_microstep: 59.55 | step_microstep: 17.93
[2025-04-25 21:29:20,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.33 | bwd: 5777.63 | bwd_inner: 5718.03 | bwd_allreduce: 59.56 | step: 17.94

 14%|█▎        | 5597/41250 [13:31:45<86:02:54,  8.69s/it]
                                                          
{'loss': 0.0523, 'grad_norm': 0.9731487035751343, 'learning_rate': 3.8840022677279035e-05, 'epoch': 1.36}

 14%|█▎        | 5597/41250 [13:31:45<86:02:54,  8.69s/it][2025-04-25 21:29:29,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:29:29,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.07 | bwd_microstep: 5714.37 | bwd_inner_microstep: 5663.01 | bwd_allreduce_microstep: 51.31 | step_microstep: 18.59
[2025-04-25 21:29:29,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.07 | bwd: 5714.38 | bwd_inner: 5663.01 | bwd_allreduce: 51.33 | step: 18.59

 14%|█▎        | 5598/41250 [13:31:54<85:53:44,  8.67s/it]
                                                          
{'loss': 0.0523, 'grad_norm': 1.1931345462799072, 'learning_rate': 3.8839495602844204e-05, 'epoch': 1.36}

 14%|█▎        | 5598/41250 [13:31:54<85:53:44,  8.67s/it][2025-04-25 21:29:37,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 21:29:37,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.61 | bwd_microstep: 5767.31 | bwd_inner_microstep: 5657.52 | bwd_allreduce_microstep: 109.75 | step_microstep: 18.90
[2025-04-25 21:29:37,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.61 | bwd: 5767.33 | bwd_inner: 5657.52 | bwd_allreduce: 109.77 | step: 18.90

 14%|█▎        | 5599/41250 [13:32:03<85:55:06,  8.68s/it]
                                                          
{'loss': 0.3435, 'grad_norm': 2.04844069480896, 'learning_rate': 3.883896841226758e-05, 'epoch': 1.36}

 14%|█▎        | 5599/41250 [13:32:03<85:55:06,  8.68s/it][2025-04-25 21:29:46,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 21:29:46,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.53 | bwd_microstep: 5775.77 | bwd_inner_microstep: 5666.03 | bwd_allreduce_microstep: 109.69 | step_microstep: 18.80
[2025-04-25 21:29:46,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.53 | bwd: 5775.78 | bwd_inner: 5666.03 | bwd_allreduce: 109.71 | step: 18.80

 14%|█▎        | 5600/41250 [13:32:11<85:57:44,  8.68s/it]
                                                          
{'loss': 0.0447, 'grad_norm': 1.2559137344360352, 'learning_rate': 3.883844110555243e-05, 'epoch': 1.36}

 14%|█▎        | 5600/41250 [13:32:11<85:57:44,  8.68s/it][2025-04-25 21:29:55,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:29:55,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.34 | bwd_microstep: 5796.10 | bwd_inner_microstep: 5667.55 | bwd_allreduce_microstep: 128.51 | step_microstep: 18.34
[2025-04-25 21:29:55,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.34 | bwd: 5796.11 | bwd_inner: 5667.55 | bwd_allreduce: 128.52 | step: 18.34

 14%|█▎        | 5601/41250 [13:32:20<86:03:59,  8.69s/it]
                                                          
{'loss': 0.1002, 'grad_norm': 2.6007039546966553, 'learning_rate': 3.883791368270199e-05, 'epoch': 1.36}

 14%|█▎        | 5601/41250 [13:32:20<86:03:59,  8.69s/it][2025-04-25 21:30:04,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:30:04,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.63 | bwd_microstep: 5719.85 | bwd_inner_microstep: 5707.26 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.52
[2025-04-25 21:30:04,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.63 | bwd: 5719.87 | bwd_inner: 5707.26 | bwd_allreduce: 12.56 | step: 18.52

 14%|█▎        | 5602/41250 [13:32:29<85:59:06,  8.68s/it]
                                                          
{'loss': 0.2588, 'grad_norm': 3.5359280109405518, 'learning_rate': 3.883738614371952e-05, 'epoch': 1.36}

 14%|█▎        | 5602/41250 [13:32:29<85:59:06,  8.68s/it][2025-04-25 21:30:12,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-25 21:30:12,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5747.63 | bwd_inner_microstep: 5653.73 | bwd_allreduce_microstep: 93.85 | step_microstep: 19.66
[2025-04-25 21:30:12,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5747.64 | bwd_inner: 5653.73 | bwd_allreduce: 93.86 | step: 19.67

 14%|█▎        | 5603/41250 [13:32:38<85:55:29,  8.68s/it]
                                                          
{'loss': 0.192, 'grad_norm': 2.546175479888916, 'learning_rate': 3.883685848860827e-05, 'epoch': 1.36}

 14%|█▎        | 5603/41250 [13:32:38<85:55:29,  8.68s/it][2025-04-25 21:30:21,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:30:21,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.96 | bwd_microstep: 5928.61 | bwd_inner_microstep: 5670.26 | bwd_allreduce_microstep: 258.29 | step_microstep: 19.03
[2025-04-25 21:30:21,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.96 | bwd: 5928.62 | bwd_inner: 5670.26 | bwd_allreduce: 258.31 | step: 19.04

 14%|█▎        | 5604/41250 [13:32:46<86:25:30,  8.73s/it]
                                                          
{'loss': 0.0367, 'grad_norm': 0.7083260416984558, 'learning_rate': 3.883633071737149e-05, 'epoch': 1.36}

 14%|█▎        | 5604/41250 [13:32:46<86:25:30,  8.73s/it][2025-04-25 21:30:30,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:30:30,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.72 | bwd_microstep: 5737.96 | bwd_inner_microstep: 5651.31 | bwd_allreduce_microstep: 86.60 | step_microstep: 19.01
[2025-04-25 21:30:30,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.72 | bwd: 5737.97 | bwd_inner: 5651.31 | bwd_allreduce: 86.62 | step: 19.01

 14%|█▎        | 5605/41250 [13:32:55<86:10:45,  8.70s/it]
                                                          
{'loss': 0.0228, 'grad_norm': 0.9621867537498474, 'learning_rate': 3.883580283001244e-05, 'epoch': 1.36}

 14%|█▎        | 5605/41250 [13:32:55<86:10:45,  8.70s/it][2025-04-25 21:30:38,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-25 21:30:38,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.12 | bwd_microstep: 5726.72 | bwd_inner_microstep: 5713.84 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.42
[2025-04-25 21:30:38,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.12 | bwd: 5726.73 | bwd_inner: 5713.84 | bwd_allreduce: 12.85 | step: 19.42

 14%|█▎        | 5606/41250 [13:33:04<86:04:58,  8.69s/it]
                                                          
{'loss': 0.0923, 'grad_norm': 0.9281152486801147, 'learning_rate': 3.883527482653436e-05, 'epoch': 1.36}

 14%|█▎        | 5606/41250 [13:33:04<86:04:58,  8.69s/it][2025-04-25 21:30:47,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-25 21:30:47,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5922.53 | bwd_inner_microstep: 5713.13 | bwd_allreduce_microstep: 209.34 | step_microstep: 19.08
[2025-04-25 21:30:47,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5922.54 | bwd_inner: 5713.13 | bwd_allreduce: 209.36 | step: 19.08

 14%|█▎        | 5607/41250 [13:33:13<86:34:44,  8.74s/it]
                                                          
{'loss': 0.1534, 'grad_norm': 2.35833477973938, 'learning_rate': 3.883474670694053e-05, 'epoch': 1.36}

 14%|█▎        | 5607/41250 [13:33:13<86:34:44,  8.74s/it][2025-04-25 21:30:56,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:30:56,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5712.23 | bwd_inner_microstep: 5699.37 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.10
[2025-04-25 21:30:56,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5712.24 | bwd_inner: 5699.37 | bwd_allreduce: 12.83 | step: 19.10

 14%|█▎        | 5608/41250 [13:33:21<86:17:04,  8.72s/it]
                                                          
{'loss': 0.1491, 'grad_norm': 1.6757112741470337, 'learning_rate': 3.883421847123419e-05, 'epoch': 1.36}

 14%|█▎        | 5608/41250 [13:33:21<86:17:04,  8.72s/it][2025-04-25 21:31:05,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:31:05,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.91 | bwd_microstep: 5788.13 | bwd_inner_microstep: 5720.02 | bwd_allreduce_microstep: 68.07 | step_microstep: 18.36
[2025-04-25 21:31:05,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.91 | bwd: 5788.15 | bwd_inner: 5720.02 | bwd_allreduce: 68.09 | step: 18.36

 14%|█▎        | 5609/41250 [13:33:30<86:20:00,  8.72s/it]
                                                          
{'loss': 0.0352, 'grad_norm': 0.9720394611358643, 'learning_rate': 3.883369011941859e-05, 'epoch': 1.36}

 14%|█▎        | 5609/41250 [13:33:30<86:20:00,  8.72s/it][2025-04-25 21:31:13,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:31:13,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.08 | bwd_microstep: 5892.73 | bwd_inner_microstep: 5700.25 | bwd_allreduce_microstep: 192.44 | step_microstep: 18.41
[2025-04-25 21:31:13,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.08 | bwd: 5892.74 | bwd_inner: 5700.25 | bwd_allreduce: 192.45 | step: 18.41

 14%|█▎        | 5610/41250 [13:33:39<86:37:39,  8.75s/it]
                                                          
{'loss': 0.0305, 'grad_norm': 1.1314795017242432, 'learning_rate': 3.883316165149699e-05, 'epoch': 1.36}

 14%|█▎        | 5610/41250 [13:33:39<86:37:39,  8.75s/it][2025-04-25 21:31:22,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.26 | optimizer_step: 1.00
[2025-04-25 21:31:22,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.58 | bwd_microstep: 5727.06 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 68.16 | step_microstep: 19.53
[2025-04-25 21:31:22,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.58 | bwd: 5727.08 | bwd_inner: 5658.86 | bwd_allreduce: 68.18 | step: 19.53

 14%|█▎        | 5611/41250 [13:33:47<86:17:56,  8.72s/it]
                                                          
{'loss': 0.2084, 'grad_norm': 3.372615337371826, 'learning_rate': 3.883263306747267e-05, 'epoch': 1.36}

 14%|█▎        | 5611/41250 [13:33:47<86:17:56,  8.72s/it][2025-04-25 21:31:31,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 21:31:31,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.98 | bwd_microstep: 5776.23 | bwd_inner_microstep: 5763.61 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.59
[2025-04-25 21:31:31,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.98 | bwd: 5776.24 | bwd_inner: 5763.61 | bwd_allreduce: 12.59 | step: 18.59

 14%|█▎        | 5612/41250 [13:33:56<86:22:04,  8.72s/it]
                                                          
{'loss': 0.0922, 'grad_norm': 1.7531616687774658, 'learning_rate': 3.883210436734885e-05, 'epoch': 1.36}

 14%|█▎        | 5612/41250 [13:33:56<86:22:04,  8.72s/it][2025-04-25 21:31:40,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:31:40,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.93 | bwd_microstep: 5774.87 | bwd_inner_microstep: 5699.14 | bwd_allreduce_microstep: 75.69 | step_microstep: 18.49
[2025-04-25 21:31:40,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.93 | bwd: 5774.89 | bwd_inner: 5699.14 | bwd_allreduce: 75.71 | step: 18.49

 14%|█▎        | 5613/41250 [13:34:05<86:18:35,  8.72s/it]
                                                          
{'loss': 0.0322, 'grad_norm': 1.0576574802398682, 'learning_rate': 3.8831575551128825e-05, 'epoch': 1.36}

 14%|█▎        | 5613/41250 [13:34:05<86:18:35,  8.72s/it][2025-04-25 21:31:48,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 21:31:48,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.70 | bwd_microstep: 5776.02 | bwd_inner_microstep: 5653.38 | bwd_allreduce_microstep: 122.60 | step_microstep: 18.55
[2025-04-25 21:31:48,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.70 | bwd: 5776.04 | bwd_inner: 5653.38 | bwd_allreduce: 122.62 | step: 18.55

 14%|█▎        | 5614/41250 [13:34:14<86:12:36,  8.71s/it]
                                                          
{'loss': 0.0144, 'grad_norm': 0.4161290228366852, 'learning_rate': 3.883104661881584e-05, 'epoch': 1.36}

 14%|█▎        | 5614/41250 [13:34:14<86:12:36,  8.71s/it][2025-04-25 21:31:57,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:31:57,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.61 | bwd_microstep: 5685.06 | bwd_inner_microstep: 5667.34 | bwd_allreduce_microstep: 17.68 | step_microstep: 18.43
[2025-04-25 21:31:57,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.61 | bwd: 5685.08 | bwd_inner: 5667.34 | bwd_allreduce: 17.69 | step: 18.43

 14%|█▎        | 5615/41250 [13:34:22<85:52:24,  8.68s/it]
                                                          
{'loss': 0.0374, 'grad_norm': 2.3548338413238525, 'learning_rate': 3.8830517570413156e-05, 'epoch': 1.36}

 14%|█▎        | 5615/41250 [13:34:22<85:52:24,  8.68s/it][2025-04-25 21:32:06,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 21:32:06,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.52 | bwd_microstep: 5874.18 | bwd_inner_microstep: 5688.04 | bwd_allreduce_microstep: 186.09 | step_microstep: 18.38
[2025-04-25 21:32:06,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.52 | bwd: 5874.19 | bwd_inner: 5688.04 | bwd_allreduce: 186.11 | step: 18.38

 14%|█▎        | 5616/41250 [13:34:31<86:14:25,  8.71s/it]
                                                          
{'loss': 0.0979, 'grad_norm': 4.407697677612305, 'learning_rate': 3.8829988405924026e-05, 'epoch': 1.36}

 14%|█▎        | 5616/41250 [13:34:31<86:14:25,  8.71s/it][2025-04-25 21:32:14,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.13
[2025-04-25 21:32:14,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5718.07 | bwd_inner_microstep: 5705.15 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.20
[2025-04-25 21:32:14,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5718.08 | bwd_inner: 5705.15 | bwd_allreduce: 12.88 | step: 19.20

 14%|█▎        | 5617/41250 [13:34:40<86:02:28,  8.69s/it]
                                                          
{'loss': 0.0205, 'grad_norm': 0.4558219015598297, 'learning_rate': 3.8829459125351734e-05, 'epoch': 1.36}

 14%|█▎        | 5617/41250 [13:34:40<86:02:28,  8.69s/it][2025-04-25 21:32:23,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:32:23,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.53 | bwd_microstep: 5790.04 | bwd_inner_microstep: 5777.09 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.71
[2025-04-25 21:32:23,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.53 | bwd: 5790.05 | bwd_inner: 5777.09 | bwd_allreduce: 12.92 | step: 18.71

 14%|█▎        | 5618/41250 [13:34:48<86:14:10,  8.71s/it]
                                                          
{'loss': 0.0637, 'grad_norm': 1.9162484407424927, 'learning_rate': 3.882892972869952e-05, 'epoch': 1.36}

 14%|█▎        | 5618/41250 [13:34:48<86:14:10,  8.71s/it][2025-04-25 21:32:32,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:32:32,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.80 | bwd_microstep: 5714.59 | bwd_inner_microstep: 5651.60 | bwd_allreduce_microstep: 62.95 | step_microstep: 18.41
[2025-04-25 21:32:32,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.80 | bwd: 5714.61 | bwd_inner: 5651.60 | bwd_allreduce: 62.97 | step: 18.41

 14%|█▎        | 5619/41250 [13:34:57<85:57:27,  8.68s/it]
                                                          
{'loss': 0.3918, 'grad_norm': 2.5789434909820557, 'learning_rate': 3.882840021597066e-05, 'epoch': 1.36}

 14%|█▎        | 5619/41250 [13:34:57<85:57:27,  8.68s/it][2025-04-25 21:32:40,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 21:32:40,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5730.48 | bwd_inner_microstep: 5683.84 | bwd_allreduce_microstep: 46.60 | step_microstep: 18.89
[2025-04-25 21:32:40,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5730.49 | bwd_inner: 5683.84 | bwd_allreduce: 46.62 | step: 18.90

 14%|█▎        | 5620/41250 [13:35:06<85:51:47,  8.68s/it]
                                                          
{'loss': 0.1586, 'grad_norm': 4.1691789627075195, 'learning_rate': 3.882787058716842e-05, 'epoch': 1.36}

 14%|█▎        | 5620/41250 [13:35:06<85:51:47,  8.68s/it][2025-04-25 21:32:49,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 21:32:49,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5772.14 | bwd_inner_microstep: 5653.09 | bwd_allreduce_microstep: 119.00 | step_microstep: 18.87
[2025-04-25 21:32:49,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5772.15 | bwd_inner: 5653.09 | bwd_allreduce: 119.02 | step: 18.87

 14%|█▎        | 5621/41250 [13:35:14<85:52:57,  8.68s/it]
                                                          
{'loss': 0.0928, 'grad_norm': 3.250216484069824, 'learning_rate': 3.882734084229606e-05, 'epoch': 1.36}

 14%|█▎        | 5621/41250 [13:35:14<85:52:57,  8.68s/it][2025-04-25 21:32:58,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 21:32:58,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5729.53 | bwd_inner_microstep: 5710.33 | bwd_allreduce_microstep: 19.14 | step_microstep: 19.59
[2025-04-25 21:32:58,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5729.54 | bwd_inner: 5710.33 | bwd_allreduce: 19.16 | step: 19.60

 14%|█▎        | 5622/41250 [13:35:23<85:50:41,  8.67s/it]
                                                          
{'loss': 0.0594, 'grad_norm': 1.2396906614303589, 'learning_rate': 3.882681098135685e-05, 'epoch': 1.36}

 14%|█▎        | 5622/41250 [13:35:23<85:50:41,  8.67s/it][2025-04-25 21:33:06,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 21:33:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5722.68 | bwd_inner_microstep: 5703.30 | bwd_allreduce_microstep: 19.32 | step_microstep: 19.11
[2025-04-25 21:33:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5722.69 | bwd_inner: 5703.30 | bwd_allreduce: 19.35 | step: 19.11

 14%|█▎        | 5623/41250 [13:35:32<85:48:09,  8.67s/it]
                                                          
{'loss': 0.1831, 'grad_norm': 2.304884195327759, 'learning_rate': 3.882628100435404e-05, 'epoch': 1.36}

 14%|█▎        | 5623/41250 [13:35:32<85:48:09,  8.67s/it][2025-04-25 21:33:15,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 21:33:15,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5729.90 | bwd_inner_microstep: 5695.26 | bwd_allreduce_microstep: 34.59 | step_microstep: 18.92
[2025-04-25 21:33:15,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.83 | bwd: 5729.91 | bwd_inner: 5695.26 | bwd_allreduce: 34.61 | step: 18.92

 14%|█▎        | 5624/41250 [13:35:40<85:46:36,  8.67s/it]
                                                          
{'loss': 0.4025, 'grad_norm': 8.447113990783691, 'learning_rate': 3.882575091129092e-05, 'epoch': 1.36}

 14%|█▎        | 5624/41250 [13:35:40<85:46:36,  8.67s/it][2025-04-25 21:33:24,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:33:24,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.43 | bwd_microstep: 5790.69 | bwd_inner_microstep: 5777.96 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.78
[2025-04-25 21:33:24,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.43 | bwd: 5790.70 | bwd_inner: 5777.96 | bwd_allreduce: 12.70 | step: 18.79

 14%|█▎        | 5625/41250 [13:35:49<86:02:56,  8.70s/it]
                                                          
{'loss': 0.0373, 'grad_norm': 1.1791044473648071, 'learning_rate': 3.882522070217075e-05, 'epoch': 1.36}

 14%|█▎        | 5625/41250 [13:35:49<86:02:56,  8.70s/it][2025-04-25 21:33:32,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:33:32,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5720.14 | bwd_inner_microstep: 5697.93 | bwd_allreduce_microstep: 22.17 | step_microstep: 18.84
[2025-04-25 21:33:32,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5720.15 | bwd_inner: 5697.93 | bwd_allreduce: 22.18 | step: 18.85

 14%|█▎        | 5626/41250 [13:35:58<85:54:48,  8.68s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 1.727655291557312, 'learning_rate': 3.8824690376996794e-05, 'epoch': 1.36}

 14%|█▎        | 5626/41250 [13:35:58<85:54:48,  8.68s/it][2025-04-25 21:33:41,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 21:33:41,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.43 | bwd_microstep: 5853.96 | bwd_inner_microstep: 5647.29 | bwd_allreduce_microstep: 206.61 | step_microstep: 19.42
[2025-04-25 21:33:41,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.43 | bwd: 5853.97 | bwd_inner: 5647.29 | bwd_allreduce: 206.64 | step: 19.42

 14%|█▎        | 5627/41250 [13:36:06<86:09:00,  8.71s/it]
                                                          
{'loss': 0.087, 'grad_norm': 2.512007474899292, 'learning_rate': 3.882415993577232e-05, 'epoch': 1.36}

 14%|█▎        | 5627/41250 [13:36:06<86:09:00,  8.71s/it][2025-04-25 21:33:50,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:33:50,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.38 | bwd_microstep: 5733.90 | bwd_inner_microstep: 5694.34 | bwd_allreduce_microstep: 39.51 | step_microstep: 18.56
[2025-04-25 21:33:50,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.38 | bwd: 5733.92 | bwd_inner: 5694.34 | bwd_allreduce: 39.53 | step: 18.56

 14%|█▎        | 5628/41250 [13:36:15<86:02:09,  8.69s/it]
                                                          
{'loss': 0.0793, 'grad_norm': 1.9861080646514893, 'learning_rate': 3.8823629378500614e-05, 'epoch': 1.36}

 14%|█▎        | 5628/41250 [13:36:15<86:02:09,  8.69s/it][2025-04-25 21:33:58,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 21:33:58,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5690.10 | bwd_inner_microstep: 5648.62 | bwd_allreduce_microstep: 41.44 | step_microstep: 19.12
[2025-04-25 21:33:58,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5690.12 | bwd_inner: 5648.62 | bwd_allreduce: 41.46 | step: 19.12

 14%|█▎        | 5629/41250 [13:36:24<85:44:37,  8.67s/it]
                                                          
{'loss': 0.4645, 'grad_norm': 2.7274010181427, 'learning_rate': 3.8823098705184926e-05, 'epoch': 1.36}

 14%|█▎        | 5629/41250 [13:36:24<85:44:37,  8.67s/it][2025-04-25 21:34:07,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-25 21:34:07,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5759.89 | bwd_inner_microstep: 5648.14 | bwd_allreduce_microstep: 111.70 | step_microstep: 19.25
[2025-04-25 21:34:07,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5759.91 | bwd_inner: 5648.14 | bwd_allreduce: 111.73 | step: 19.25

 14%|█▎        | 5630/41250 [13:36:32<85:44:58,  8.67s/it]
                                                          
{'loss': 0.0441, 'grad_norm': 0.8993282318115234, 'learning_rate': 3.882256791582854e-05, 'epoch': 1.36}

 14%|█▎        | 5630/41250 [13:36:32<85:44:58,  8.67s/it][2025-04-25 21:34:16,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:34:16,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5707.60 | bwd_inner_microstep: 5650.59 | bwd_allreduce_microstep: 56.96 | step_microstep: 18.75
[2025-04-25 21:34:16,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5707.61 | bwd_inner: 5650.59 | bwd_allreduce: 56.98 | step: 18.75

 14%|█▎        | 5631/41250 [13:36:41<85:36:01,  8.65s/it]
                                                          
{'loss': 0.115, 'grad_norm': 2.43778395652771, 'learning_rate': 3.882203701043473e-05, 'epoch': 1.37}

 14%|█▎        | 5631/41250 [13:36:41<85:36:01,  8.65s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0xc90db80] moov atom not found
[21:34:16] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4...
[2025-04-25 21:34:24,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:34:24,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.06 | bwd_microstep: 5705.26 | bwd_inner_microstep: 5692.26 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.79
[2025-04-25 21:34:24,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.06 | bwd: 5705.27 | bwd_inner: 5692.26 | bwd_allreduce: 12.97 | step: 18.79

 14%|█▎        | 5632/41250 [13:36:50<85:32:10,  8.65s/it]
                                                          
{'loss': 0.2289, 'grad_norm': 1.743869423866272, 'learning_rate': 3.8821505989006756e-05, 'epoch': 1.37}

 14%|█▎        | 5632/41250 [13:36:50<85:32:10,  8.65s/it][2025-04-25 21:34:33,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 21:34:33,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5716.69 | bwd_inner_microstep: 5703.99 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.63
[2025-04-25 21:34:33,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5716.70 | bwd_inner: 5703.99 | bwd_allreduce: 12.67 | step: 18.63

 14%|█▎        | 5633/41250 [13:36:58<85:33:16,  8.65s/it]
                                                          
{'loss': 0.4112, 'grad_norm': 4.213136672973633, 'learning_rate': 3.8820974851547906e-05, 'epoch': 1.37}

 14%|█▎        | 5633/41250 [13:36:58<85:33:16,  8.65s/it][2025-04-25 21:34:42,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 21:34:42,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.95 | bwd_microstep: 5755.96 | bwd_inner_microstep: 5641.66 | bwd_allreduce_microstep: 114.25 | step_microstep: 18.96
[2025-04-25 21:34:42,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.95 | bwd: 5755.97 | bwd_inner: 5641.66 | bwd_allreduce: 114.27 | step: 18.97

 14%|█▎        | 5634/41250 [13:37:07<85:36:04,  8.65s/it]
                                                          
{'loss': 0.1015, 'grad_norm': 2.4715452194213867, 'learning_rate': 3.882044359806144e-05, 'epoch': 1.37}

 14%|█▎        | 5634/41250 [13:37:07<85:36:04,  8.65s/it][2025-04-25 21:34:50,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:34:50,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.66 | bwd_microstep: 5693.51 | bwd_inner_microstep: 5680.45 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.63
[2025-04-25 21:34:50,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.66 | bwd: 5693.52 | bwd_inner: 5680.45 | bwd_allreduce: 13.03 | step: 18.63

 14%|█▎        | 5635/41250 [13:37:16<85:29:52,  8.64s/it]
                                                          
{'loss': 0.0965, 'grad_norm': 1.0410650968551636, 'learning_rate': 3.881991222855065e-05, 'epoch': 1.37}

 14%|█▎        | 5635/41250 [13:37:16<85:29:52,  8.64s/it][2025-04-25 21:34:59,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 21:34:59,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.30 | bwd_microstep: 5679.73 | bwd_inner_microstep: 5642.74 | bwd_allreduce_microstep: 36.94 | step_microstep: 18.62
[2025-04-25 21:34:59,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.30 | bwd: 5679.74 | bwd_inner: 5642.74 | bwd_allreduce: 36.95 | step: 18.63

 14%|█▎        | 5636/41250 [13:37:24<85:19:31,  8.63s/it]
                                                          
{'loss': 0.2283, 'grad_norm': 2.855778217315674, 'learning_rate': 3.8819380743018794e-05, 'epoch': 1.37}

 14%|█▎        | 5636/41250 [13:37:24<85:19:31,  8.63s/it][2025-04-25 21:35:07,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 21:35:07,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5757.89 | bwd_inner_microstep: 5655.91 | bwd_allreduce_microstep: 101.93 | step_microstep: 18.35
[2025-04-25 21:35:07,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5757.90 | bwd_inner: 5655.91 | bwd_allreduce: 101.95 | step: 18.35

 14%|█▎        | 5637/41250 [13:37:33<85:27:31,  8.64s/it]
                                                          
{'loss': 0.3016, 'grad_norm': 2.748706579208374, 'learning_rate': 3.881884914146916e-05, 'epoch': 1.37}

 14%|█▎        | 5637/41250 [13:37:33<85:27:31,  8.64s/it][2025-04-25 21:35:16,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 21:35:16,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.84 | bwd_microstep: 5755.97 | bwd_inner_microstep: 5648.88 | bwd_allreduce_microstep: 107.04 | step_microstep: 18.95
[2025-04-25 21:35:16,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.84 | bwd: 5755.99 | bwd_inner: 5648.88 | bwd_allreduce: 107.06 | step: 18.96

 14%|█▎        | 5638/41250 [13:37:41<85:31:50,  8.65s/it]
                                                          
{'loss': 0.0908, 'grad_norm': 1.8227765560150146, 'learning_rate': 3.8818317423905025e-05, 'epoch': 1.37}

 14%|█▎        | 5638/41250 [13:37:41<85:31:50,  8.65s/it][2025-04-25 21:35:25,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:35:25,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.82 | bwd_microstep: 5743.02 | bwd_inner_microstep: 5639.58 | bwd_allreduce_microstep: 103.38 | step_microstep: 18.63
[2025-04-25 21:35:25,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.82 | bwd: 5743.03 | bwd_inner: 5639.58 | bwd_allreduce: 103.40 | step: 18.63

 14%|█▎        | 5639/41250 [13:37:50<85:31:32,  8.65s/it]
                                                          
{'loss': 0.0968, 'grad_norm': 1.524330735206604, 'learning_rate': 3.8817785590329664e-05, 'epoch': 1.37}

 14%|█▎        | 5639/41250 [13:37:50<85:31:32,  8.65s/it][2025-04-25 21:35:33,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:35:33,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.48 | bwd_microstep: 5758.09 | bwd_inner_microstep: 5679.35 | bwd_allreduce_microstep: 78.69 | step_microstep: 18.75
[2025-04-25 21:35:33,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.48 | bwd: 5758.10 | bwd_inner: 5679.35 | bwd_allreduce: 78.71 | step: 18.75

 14%|█▎        | 5640/41250 [13:37:59<85:38:39,  8.66s/it]
                                                          
{'loss': 0.2818, 'grad_norm': 3.4086015224456787, 'learning_rate': 3.881725364074636e-05, 'epoch': 1.37}

 14%|█▎        | 5640/41250 [13:37:59<85:38:39,  8.66s/it][2025-04-25 21:35:42,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 21:35:42,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5706.12 | bwd_inner_microstep: 5659.84 | bwd_allreduce_microstep: 46.23 | step_microstep: 18.82
[2025-04-25 21:35:42,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5706.14 | bwd_inner: 5659.84 | bwd_allreduce: 46.25 | step: 18.82

 14%|█▎        | 5641/41250 [13:38:07<85:32:19,  8.65s/it]
                                                          
{'loss': 0.1329, 'grad_norm': 3.8678267002105713, 'learning_rate': 3.881672157515838e-05, 'epoch': 1.37}

 14%|█▎        | 5641/41250 [13:38:07<85:32:19,  8.65s/it][2025-04-25 21:35:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 21:35:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.16 | bwd_microstep: 5704.94 | bwd_inner_microstep: 5649.66 | bwd_allreduce_microstep: 55.23 | step_microstep: 18.96
[2025-04-25 21:35:51,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.16 | bwd: 5704.95 | bwd_inner: 5649.66 | bwd_allreduce: 55.25 | step: 18.96

 14%|█▎        | 5642/41250 [13:38:16<85:26:11,  8.64s/it]
                                                          
{'loss': 0.3987, 'grad_norm': 3.1546993255615234, 'learning_rate': 3.881618939356901e-05, 'epoch': 1.37}

 14%|█▎        | 5642/41250 [13:38:16<85:26:11,  8.64s/it][2025-04-25 21:35:59,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:35:59,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.13 | bwd_microstep: 5781.27 | bwd_inner_microstep: 5639.21 | bwd_allreduce_microstep: 142.02 | step_microstep: 18.72
[2025-04-25 21:35:59,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.13 | bwd: 5781.29 | bwd_inner: 5639.21 | bwd_allreduce: 142.04 | step: 18.72

 14%|█▎        | 5643/41250 [13:38:25<85:34:41,  8.65s/it]
                                                          
{'loss': 0.2714, 'grad_norm': 4.207535266876221, 'learning_rate': 3.881565709598154e-05, 'epoch': 1.37}

 14%|█▎        | 5643/41250 [13:38:25<85:34:41,  8.65s/it][2025-04-25 21:36:08,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:36:08,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.90 | bwd_microstep: 5761.39 | bwd_inner_microstep: 5705.23 | bwd_allreduce_microstep: 56.12 | step_microstep: 18.66
[2025-04-25 21:36:08,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.90 | bwd: 5761.41 | bwd_inner: 5705.23 | bwd_allreduce: 56.13 | step: 18.66

 14%|█▎        | 5644/41250 [13:38:33<85:42:38,  8.67s/it]
                                                          
{'loss': 0.2097, 'grad_norm': 2.3665547370910645, 'learning_rate': 3.881512468239925e-05, 'epoch': 1.37}

 14%|█▎        | 5644/41250 [13:38:33<85:42:38,  8.67s/it][2025-04-25 21:36:17,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:36:17,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5717.77 | bwd_inner_microstep: 5705.11 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.46
[2025-04-25 21:36:17,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5717.79 | bwd_inner: 5705.11 | bwd_allreduce: 12.63 | step: 18.46

 14%|█▎        | 5645/41250 [13:38:42<85:39:47,  8.66s/it]
                                                          
{'loss': 0.0361, 'grad_norm': 1.2173537015914917, 'learning_rate': 3.881459215282541e-05, 'epoch': 1.37}

 14%|█▎        | 5645/41250 [13:38:42<85:39:47,  8.66s/it][2025-04-25 21:36:25,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 21:36:25,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.70 | bwd_microstep: 5695.73 | bwd_inner_microstep: 5682.93 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.33
[2025-04-25 21:36:25,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.70 | bwd: 5695.74 | bwd_inner: 5682.93 | bwd_allreduce: 12.78 | step: 19.33

 14%|█▎        | 5646/41250 [13:38:51<85:34:57,  8.65s/it]
                                                          
{'loss': 0.1305, 'grad_norm': 1.590264081954956, 'learning_rate': 3.881405950726331e-05, 'epoch': 1.37}

 14%|█▎        | 5646/41250 [13:38:51<85:34:57,  8.65s/it][2025-04-25 21:36:34,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 21:36:34,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.35 | bwd_microstep: 5704.52 | bwd_inner_microstep: 5656.73 | bwd_allreduce_microstep: 47.74 | step_microstep: 19.23
[2025-04-25 21:36:34,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.35 | bwd: 5704.54 | bwd_inner: 5656.73 | bwd_allreduce: 47.76 | step: 19.23

 14%|█▎        | 5647/41250 [13:38:59<85:28:13,  8.64s/it]
                                                          
{'loss': 0.1251, 'grad_norm': 2.260995388031006, 'learning_rate': 3.881352674571624e-05, 'epoch': 1.37}

 14%|█▎        | 5647/41250 [13:38:59<85:28:13,  8.64s/it][2025-04-25 21:36:43,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:36:43,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.49 | bwd_microstep: 5740.42 | bwd_inner_microstep: 5714.46 | bwd_allreduce_microstep: 25.91 | step_microstep: 19.02
[2025-04-25 21:36:43,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.50 | bwd: 5740.43 | bwd_inner: 5714.46 | bwd_allreduce: 25.93 | step: 19.02

 14%|█▎        | 5648/41250 [13:39:08<85:35:10,  8.65s/it]
                                                          
{'loss': 0.0831, 'grad_norm': 2.4447743892669678, 'learning_rate': 3.881299386818748e-05, 'epoch': 1.37}

 14%|█▎        | 5648/41250 [13:39:08<85:35:10,  8.65s/it][2025-04-25 21:36:51,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:36:51,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.68 | bwd_microstep: 5727.17 | bwd_inner_microstep: 5714.38 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.75
[2025-04-25 21:36:51,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.68 | bwd: 5727.19 | bwd_inner: 5714.38 | bwd_allreduce: 12.77 | step: 18.75

 14%|█▎        | 5649/41250 [13:39:17<85:37:29,  8.66s/it]
                                                          
{'loss': 0.1265, 'grad_norm': 1.1228097677230835, 'learning_rate': 3.88124608746803e-05, 'epoch': 1.37}

 14%|█▎        | 5649/41250 [13:39:17<85:37:29,  8.66s/it][2025-04-25 21:37:00,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:37:00,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.99 | bwd_microstep: 5786.92 | bwd_inner_microstep: 5774.18 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.75
[2025-04-25 21:37:00,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.99 | bwd: 5786.93 | bwd_inner: 5774.18 | bwd_allreduce: 12.71 | step: 18.75

 14%|█▎        | 5650/41250 [13:39:25<85:54:46,  8.69s/it]
                                                          
{'loss': 0.0456, 'grad_norm': 0.6913841962814331, 'learning_rate': 3.8811927765198005e-05, 'epoch': 1.37}

 14%|█▎        | 5650/41250 [13:39:25<85:54:46,  8.69s/it][2025-04-25 21:37:09,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:37:09,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5793.58 | bwd_inner_microstep: 5663.79 | bwd_allreduce_microstep: 129.73 | step_microstep: 19.10
[2025-04-25 21:37:09,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5793.59 | bwd_inner: 5663.79 | bwd_allreduce: 129.75 | step: 19.10

 14%|█▎        | 5651/41250 [13:39:34<85:58:49,  8.69s/it]
                                                          
{'loss': 0.0548, 'grad_norm': 0.9587269425392151, 'learning_rate': 3.881139453974387e-05, 'epoch': 1.37}

 14%|█▎        | 5651/41250 [13:39:34<85:58:49,  8.69s/it][2025-04-25 21:37:17,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 21:37:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.96 | bwd_microstep: 5717.22 | bwd_inner_microstep: 5652.02 | bwd_allreduce_microstep: 65.16 | step_microstep: 18.95
[2025-04-25 21:37:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.96 | bwd: 5717.24 | bwd_inner: 5652.02 | bwd_allreduce: 65.18 | step: 18.96

 14%|█▎        | 5652/41250 [13:39:43<85:48:11,  8.68s/it]
                                                          
{'loss': 0.0975, 'grad_norm': 1.3430930376052856, 'learning_rate': 3.8810861198321196e-05, 'epoch': 1.37}

 14%|█▎        | 5652/41250 [13:39:43<85:48:11,  8.68s/it][2025-04-25 21:37:26,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:37:26,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.84 | bwd_microstep: 5850.59 | bwd_inner_microstep: 5711.24 | bwd_allreduce_microstep: 139.30 | step_microstep: 18.40
[2025-04-25 21:37:26,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.84 | bwd: 5850.60 | bwd_inner: 5711.24 | bwd_allreduce: 139.32 | step: 18.41

 14%|█▎        | 5653/41250 [13:39:52<86:08:59,  8.71s/it]
                                                          
{'loss': 0.1305, 'grad_norm': 1.3731762170791626, 'learning_rate': 3.881032774093326e-05, 'epoch': 1.37}

 14%|█▎        | 5653/41250 [13:39:52<86:08:59,  8.71s/it][2025-04-25 21:37:35,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:37:35,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.89 | bwd_microstep: 5751.49 | bwd_inner_microstep: 5692.90 | bwd_allreduce_microstep: 58.54 | step_microstep: 18.53
[2025-04-25 21:37:35,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.89 | bwd: 5751.50 | bwd_inner: 5692.90 | bwd_allreduce: 58.56 | step: 18.54

 14%|█▎        | 5654/41250 [13:40:00<86:08:30,  8.71s/it]
                                                          
{'loss': 0.2211, 'grad_norm': 2.1762962341308594, 'learning_rate': 3.8809794167583354e-05, 'epoch': 1.37}

 14%|█▎        | 5654/41250 [13:40:00<86:08:30,  8.71s/it][2025-04-25 21:37:44,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 21:37:44,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.43 | bwd_microstep: 5781.22 | bwd_inner_microstep: 5716.50 | bwd_allreduce_microstep: 64.68 | step_microstep: 18.43
[2025-04-25 21:37:44,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.43 | bwd: 5781.24 | bwd_inner: 5716.50 | bwd_allreduce: 64.70 | step: 18.43

 14%|█▎        | 5655/41250 [13:40:09<86:09:04,  8.71s/it]
                                                          
{'loss': 0.1266, 'grad_norm': 1.1526734828948975, 'learning_rate': 3.8809260478274774e-05, 'epoch': 1.37}

 14%|█▎        | 5655/41250 [13:40:09<86:09:04,  8.71s/it][2025-04-25 21:37:52,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 21:37:52,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.46 | bwd_microstep: 5719.22 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.01
[2025-04-25 21:37:52,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.46 | bwd: 5719.24 | bwd_inner: 5706.20 | bwd_allreduce: 12.99 | step: 19.02

 14%|█▎        | 5656/41250 [13:40:18<85:58:31,  8.70s/it]
                                                          
{'loss': 0.0177, 'grad_norm': 0.22662527859210968, 'learning_rate': 3.8808726673010785e-05, 'epoch': 1.37}

 14%|█▎        | 5656/41250 [13:40:18<85:58:31,  8.70s/it][2025-04-25 21:38:01,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.12 | optimizer_step: 0.96
[2025-04-25 21:38:01,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5733.87 | bwd_inner_microstep: 5659.08 | bwd_allreduce_microstep: 74.73 | step_microstep: 18.96
[2025-04-25 21:38:01,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5733.88 | bwd_inner: 5659.08 | bwd_allreduce: 74.75 | step: 18.96

 14%|█▎        | 5657/41250 [13:40:26<85:50:33,  8.68s/it]
                                                          
{'loss': 0.1722, 'grad_norm': 1.7321053743362427, 'learning_rate': 3.880819275179471e-05, 'epoch': 1.37}

 14%|█▎        | 5657/41250 [13:40:26<85:50:33,  8.68s/it][2025-04-25 21:38:10,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.22 | optimizer_step: 0.99
[2025-04-25 21:38:10,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.80 | bwd_microstep: 5714.07 | bwd_inner_microstep: 5700.56 | bwd_allreduce_microstep: 13.46 | step_microstep: 19.60
[2025-04-25 21:38:10,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.80 | bwd: 5714.09 | bwd_inner: 5700.56 | bwd_allreduce: 13.48 | step: 19.60

 14%|█▎        | 5658/41250 [13:40:35<85:44:13,  8.67s/it]
                                                          
{'loss': 0.1208, 'grad_norm': 1.132050633430481, 'learning_rate': 3.880765871462982e-05, 'epoch': 1.37}

 14%|█▎        | 5658/41250 [13:40:35<85:44:13,  8.67s/it][2025-04-25 21:38:18,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 21:38:18,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.35 | bwd_microstep: 5717.57 | bwd_inner_microstep: 5667.46 | bwd_allreduce_microstep: 50.06 | step_microstep: 19.25
[2025-04-25 21:38:18,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.35 | bwd: 5717.58 | bwd_inner: 5667.46 | bwd_allreduce: 50.09 | step: 19.25

 14%|█▎        | 5659/41250 [13:40:44<85:37:27,  8.66s/it]
                                                          
{'loss': 0.1909, 'grad_norm': 1.4809356927871704, 'learning_rate': 3.8807124561519414e-05, 'epoch': 1.37}

 14%|█▎        | 5659/41250 [13:40:44<85:37:27,  8.66s/it][2025-04-25 21:38:27,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-25 21:38:27,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.44 | bwd_microstep: 5721.56 | bwd_inner_microstep: 5708.74 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.63
[2025-04-25 21:38:27,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.44 | bwd: 5721.57 | bwd_inner: 5708.74 | bwd_allreduce: 12.79 | step: 19.64

 14%|█▎        | 5660/41250 [13:40:52<85:37:35,  8.66s/it]
                                                          
{'loss': 0.0932, 'grad_norm': 1.1971241235733032, 'learning_rate': 3.880659029246679e-05, 'epoch': 1.37}

 14%|█▎        | 5660/41250 [13:40:52<85:37:35,  8.66s/it][2025-04-25 21:38:36,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-25 21:38:36,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.72 | bwd_microstep: 5727.55 | bwd_inner_microstep: 5668.23 | bwd_allreduce_microstep: 59.27 | step_microstep: 19.25
[2025-04-25 21:38:36,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.72 | bwd: 5727.56 | bwd_inner: 5668.23 | bwd_allreduce: 59.29 | step: 19.25

 14%|█▎        | 5661/41250 [13:41:01<85:35:09,  8.66s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 1.7765965461730957, 'learning_rate': 3.8806055907475224e-05, 'epoch': 1.37}

 14%|█▎        | 5661/41250 [13:41:01<85:35:09,  8.66s/it][2025-04-25 21:38:44,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 21:38:44,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.71 | bwd_microstep: 5722.59 | bwd_inner_microstep: 5660.48 | bwd_allreduce_microstep: 62.07 | step_microstep: 19.00
[2025-04-25 21:38:44,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.71 | bwd: 5722.61 | bwd_inner: 5660.48 | bwd_allreduce: 62.09 | step: 19.00

 14%|█▎        | 5662/41250 [13:41:10<85:31:52,  8.65s/it]
                                                          
{'loss': 0.0533, 'grad_norm': 0.766474723815918, 'learning_rate': 3.880552140654803e-05, 'epoch': 1.37}

 14%|█▎        | 5662/41250 [13:41:10<85:31:52,  8.65s/it][2025-04-25 21:38:53,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:38:53,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.41 | bwd_microstep: 5758.93 | bwd_inner_microstep: 5718.18 | bwd_allreduce_microstep: 40.70 | step_microstep: 18.91
[2025-04-25 21:38:53,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.41 | bwd: 5758.94 | bwd_inner: 5718.18 | bwd_allreduce: 40.72 | step: 18.91

 14%|█▎        | 5663/41250 [13:41:18<85:40:31,  8.67s/it]
                                                          
{'loss': 0.2171, 'grad_norm': 3.9582526683807373, 'learning_rate': 3.88049867896885e-05, 'epoch': 1.37}

 14%|█▎        | 5663/41250 [13:41:18<85:40:31,  8.67s/it][2025-04-25 21:39:02,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-25 21:39:02,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5794.52 | bwd_inner_microstep: 5691.64 | bwd_allreduce_microstep: 102.83 | step_microstep: 19.33
[2025-04-25 21:39:02,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5794.54 | bwd_inner: 5691.64 | bwd_allreduce: 102.85 | step: 19.33

 14%|█▎        | 5664/41250 [13:41:27<85:51:35,  8.69s/it]
                                                          
{'loss': 0.1024, 'grad_norm': 1.4898571968078613, 'learning_rate': 3.880445205689991e-05, 'epoch': 1.37}

 14%|█▎        | 5664/41250 [13:41:27<85:51:35,  8.69s/it][2025-04-25 21:39:10,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:39:10,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.45 | bwd_microstep: 5758.55 | bwd_inner_microstep: 5666.53 | bwd_allreduce_microstep: 91.97 | step_microstep: 18.66
[2025-04-25 21:39:10,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.45 | bwd: 5758.56 | bwd_inner: 5666.53 | bwd_allreduce: 91.99 | step: 18.67

 14%|█▎        | 5665/41250 [13:41:36<85:49:13,  8.68s/it]
                                                          
{'loss': 0.085, 'grad_norm': 1.9482736587524414, 'learning_rate': 3.8803917208185583e-05, 'epoch': 1.37}

 14%|█▎        | 5665/41250 [13:41:36<85:49:13,  8.68s/it][2025-04-25 21:39:19,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 21:39:19,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.07 | bwd_microstep: 5784.60 | bwd_inner_microstep: 5709.84 | bwd_allreduce_microstep: 74.72 | step_microstep: 18.93
[2025-04-25 21:39:19,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.07 | bwd: 5784.62 | bwd_inner: 5709.84 | bwd_allreduce: 74.74 | step: 18.94

 14%|█▎        | 5666/41250 [13:41:44<85:55:43,  8.69s/it]
                                                          
{'loss': 0.0937, 'grad_norm': 2.194953203201294, 'learning_rate': 3.8803382243548806e-05, 'epoch': 1.37}

 14%|█▎        | 5666/41250 [13:41:44<85:55:43,  8.69s/it][2025-04-25 21:39:28,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.15 | optimizer_step: 0.93
[2025-04-25 21:39:28,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.71 | bwd_microstep: 5764.43 | bwd_inner_microstep: 5688.09 | bwd_allreduce_microstep: 76.29 | step_microstep: 18.96
[2025-04-25 21:39:28,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.71 | bwd: 5764.44 | bwd_inner: 5688.09 | bwd_allreduce: 76.31 | step: 18.96

 14%|█▎        | 5667/41250 [13:41:53<85:55:21,  8.69s/it]
                                                          
{'loss': 0.0958, 'grad_norm': 1.327024221420288, 'learning_rate': 3.880284716299287e-05, 'epoch': 1.37}

 14%|█▎        | 5667/41250 [13:41:53<85:55:21,  8.69s/it][2025-04-25 21:39:36,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:39:36,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.23 | bwd_microstep: 5732.07 | bwd_inner_microstep: 5710.42 | bwd_allreduce_microstep: 21.60 | step_microstep: 18.90
[2025-04-25 21:39:36,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.23 | bwd: 5732.08 | bwd_inner: 5710.42 | bwd_allreduce: 21.62 | step: 18.91

 14%|█▎        | 5668/41250 [13:42:02<85:52:29,  8.69s/it]
                                                          
{'loss': 0.1478, 'grad_norm': 4.88890266418457, 'learning_rate': 3.880231196652108e-05, 'epoch': 1.37}

 14%|█▎        | 5668/41250 [13:42:02<85:52:29,  8.69s/it][2025-04-25 21:39:45,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 21:39:45,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.68 | bwd_microstep: 5705.57 | bwd_inner_microstep: 5692.76 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.60
[2025-04-25 21:39:45,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.68 | bwd: 5705.58 | bwd_inner: 5692.76 | bwd_allreduce: 12.78 | step: 18.61

 14%|█▎        | 5669/41250 [13:42:10<85:43:28,  8.67s/it]
                                                          
{'loss': 0.1502, 'grad_norm': 4.2795090675354, 'learning_rate': 3.8801776654136744e-05, 'epoch': 1.37}

 14%|█▎        | 5669/41250 [13:42:10<85:43:28,  8.67s/it][2025-04-25 21:39:54,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 21:39:54,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.88 | bwd_microstep: 5759.08 | bwd_inner_microstep: 5700.40 | bwd_allreduce_microstep: 58.63 | step_microstep: 19.27
[2025-04-25 21:39:54,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.88 | bwd: 5759.09 | bwd_inner: 5700.40 | bwd_allreduce: 58.65 | step: 19.27

 14%|█▎        | 5670/41250 [13:42:19<85:46:48,  8.68s/it]
                                                          
{'loss': 0.0408, 'grad_norm': 0.6673532128334045, 'learning_rate': 3.880124122584315e-05, 'epoch': 1.37}

 14%|█▎        | 5670/41250 [13:42:19<85:46:48,  8.68s/it][2025-04-25 21:40:02,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:40:02,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.13 | bwd_microstep: 5762.66 | bwd_inner_microstep: 5688.77 | bwd_allreduce_microstep: 73.84 | step_microstep: 18.44
[2025-04-25 21:40:02,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.13 | bwd: 5762.67 | bwd_inner: 5688.77 | bwd_allreduce: 73.86 | step: 18.45

 14%|█▎        | 5671/41250 [13:42:28<85:49:06,  8.68s/it]
                                                          
{'loss': 0.2906, 'grad_norm': 2.000570297241211, 'learning_rate': 3.8800705681643594e-05, 'epoch': 1.37}

 14%|█▎        | 5671/41250 [13:42:28<85:49:06,  8.68s/it][2025-04-25 21:40:11,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:40:11,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.12 | bwd_microstep: 5718.46 | bwd_inner_microstep: 5663.78 | bwd_allreduce_microstep: 54.64 | step_microstep: 18.55
[2025-04-25 21:40:11,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.12 | bwd: 5718.47 | bwd_inner: 5663.78 | bwd_allreduce: 54.65 | step: 18.55

 14%|█▍        | 5672/41250 [13:42:36<85:39:16,  8.67s/it]
                                                          
{'loss': 0.0453, 'grad_norm': 1.0938867330551147, 'learning_rate': 3.880017002154139e-05, 'epoch': 1.38}

 14%|█▍        | 5672/41250 [13:42:36<85:39:16,  8.67s/it][2025-04-25 21:40:20,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:40:20,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.98 | bwd_microstep: 5752.32 | bwd_inner_microstep: 5697.30 | bwd_allreduce_microstep: 54.97 | step_microstep: 18.52
[2025-04-25 21:40:20,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.98 | bwd: 5752.33 | bwd_inner: 5697.30 | bwd_allreduce: 54.99 | step: 18.52

 14%|█▍        | 5673/41250 [13:42:45<85:42:17,  8.67s/it]
                                                          
{'loss': 0.1525, 'grad_norm': 3.697646379470825, 'learning_rate': 3.8799634245539835e-05, 'epoch': 1.38}

 14%|█▍        | 5673/41250 [13:42:45<85:42:17,  8.67s/it][2025-04-25 21:40:28,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 21:40:28,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5733.77 | bwd_inner_microstep: 5695.68 | bwd_allreduce_microstep: 38.05 | step_microstep: 18.71
[2025-04-25 21:40:28,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5733.78 | bwd_inner: 5695.68 | bwd_allreduce: 38.06 | step: 18.72

 14%|█▍        | 5674/41250 [13:42:54<85:41:24,  8.67s/it]
                                                          
{'loss': 0.2965, 'grad_norm': 1.5924859046936035, 'learning_rate': 3.879909835364223e-05, 'epoch': 1.38}

 14%|█▍        | 5674/41250 [13:42:54<85:41:24,  8.67s/it][2025-04-25 21:40:37,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:40:37,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.55 | bwd_microstep: 5852.96 | bwd_inner_microstep: 5709.89 | bwd_allreduce_microstep: 143.02 | step_microstep: 18.62
[2025-04-25 21:40:37,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.55 | bwd: 5852.98 | bwd_inner: 5709.89 | bwd_allreduce: 143.05 | step: 18.62

 14%|█▍        | 5675/41250 [13:43:03<86:02:49,  8.71s/it]
                                                          
{'loss': 0.2503, 'grad_norm': 3.517625570297241, 'learning_rate': 3.879856234585189e-05, 'epoch': 1.38}

 14%|█▍        | 5675/41250 [13:43:03<86:02:49,  8.71s/it][2025-04-25 21:40:46,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.12
[2025-04-25 21:40:46,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.85 | bwd_microstep: 5788.83 | bwd_inner_microstep: 5776.07 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.28
[2025-04-25 21:40:46,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.85 | bwd: 5788.84 | bwd_inner: 5776.07 | bwd_allreduce: 12.73 | step: 19.28

 14%|█▍        | 5676/41250 [13:43:11<86:11:32,  8.72s/it]
                                                          
{'loss': 0.0364, 'grad_norm': 0.6321446299552917, 'learning_rate': 3.87980262221721e-05, 'epoch': 1.38}

 14%|█▍        | 5676/41250 [13:43:11<86:11:32,  8.72s/it][2025-04-25 21:40:55,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:40:55,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.60 | bwd_microstep: 5726.02 | bwd_inner_microstep: 5713.20 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.79
[2025-04-25 21:40:55,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.60 | bwd: 5726.04 | bwd_inner: 5713.20 | bwd_allreduce: 12.80 | step: 18.80

 14%|█▍        | 5677/41250 [13:43:20<86:01:28,  8.71s/it]
                                                          
{'loss': 0.0785, 'grad_norm': 1.5374460220336914, 'learning_rate': 3.879748998260619e-05, 'epoch': 1.38}

 14%|█▍        | 5677/41250 [13:43:20<86:01:28,  8.71s/it][2025-04-25 21:41:03,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:41:03,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.51 | bwd_microstep: 5753.77 | bwd_inner_microstep: 5696.42 | bwd_allreduce_microstep: 57.30 | step_microstep: 18.68
[2025-04-25 21:41:03,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.51 | bwd: 5753.78 | bwd_inner: 5696.42 | bwd_allreduce: 57.32 | step: 18.68

 14%|█▍        | 5678/41250 [13:43:29<85:58:29,  8.70s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 4.93423318862915, 'learning_rate': 3.879695362715744e-05, 'epoch': 1.38}

 14%|█▍        | 5678/41250 [13:43:29<85:58:29,  8.70s/it][2025-04-25 21:41:12,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:41:12,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.10 | bwd_microstep: 5757.08 | bwd_inner_microstep: 5641.69 | bwd_allreduce_microstep: 115.34 | step_microstep: 18.62
[2025-04-25 21:41:12,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.10 | bwd: 5757.09 | bwd_inner: 5641.69 | bwd_allreduce: 115.35 | step: 18.62

 14%|█▍        | 5679/41250 [13:43:37<85:51:15,  8.69s/it]
                                                          
{'loss': 0.2899, 'grad_norm': 1.6766951084136963, 'learning_rate': 3.8796417155829166e-05, 'epoch': 1.38}

 14%|█▍        | 5679/41250 [13:43:37<85:51:15,  8.69s/it][2025-04-25 21:41:21,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 21:41:21,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.77 | bwd_microstep: 5679.96 | bwd_inner_microstep: 5639.00 | bwd_allreduce_microstep: 40.91 | step_microstep: 18.73
[2025-04-25 21:41:21,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.77 | bwd: 5679.98 | bwd_inner: 5639.00 | bwd_allreduce: 40.93 | step: 18.73

 14%|█▍        | 5680/41250 [13:43:46<85:32:39,  8.66s/it]
                                                          
{'loss': 0.2479, 'grad_norm': 2.0503129959106445, 'learning_rate': 3.879588056862468e-05, 'epoch': 1.38}

 14%|█▍        | 5680/41250 [13:43:46<85:32:39,  8.66s/it][2025-04-25 21:41:30,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:41:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.64 | bwd_microstep: 6048.28 | bwd_inner_microstep: 5635.02 | bwd_allreduce_microstep: 413.21 | step_microstep: 18.66
[2025-04-25 21:41:30,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.64 | bwd: 6048.29 | bwd_inner: 5635.02 | bwd_allreduce: 413.23 | step: 18.66

 14%|█▍        | 5681/41250 [13:43:55<86:24:49,  8.75s/it]
                                                          
{'loss': 0.2306, 'grad_norm': 2.9101030826568604, 'learning_rate': 3.879534386554729e-05, 'epoch': 1.38}

 14%|█▍        | 5681/41250 [13:43:55<86:24:49,  8.75s/it][2025-04-25 21:41:38,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:41:38,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5679.22 | bwd_inner_microstep: 5645.46 | bwd_allreduce_microstep: 33.71 | step_microstep: 18.53
[2025-04-25 21:41:38,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5679.23 | bwd_inner: 5645.46 | bwd_allreduce: 33.73 | step: 18.53

 14%|█▍        | 5682/41250 [13:44:03<85:56:55,  8.70s/it]
                                                          
{'loss': 0.1706, 'grad_norm': 2.070218801498413, 'learning_rate': 3.87948070466003e-05, 'epoch': 1.38}

 14%|█▍        | 5682/41250 [13:44:03<85:56:55,  8.70s/it][2025-04-25 21:41:47,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 21:41:47,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5736.00 | bwd_inner_microstep: 5673.15 | bwd_allreduce_microstep: 62.82 | step_microstep: 18.76
[2025-04-25 21:41:47,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5736.02 | bwd_inner: 5673.14 | bwd_allreduce: 62.83 | step: 18.76

 14%|█▍        | 5683/41250 [13:44:12<85:49:04,  8.69s/it]
                                                          
{'loss': 0.0595, 'grad_norm': 1.7908625602722168, 'learning_rate': 3.8794270111787024e-05, 'epoch': 1.38}

 14%|█▍        | 5683/41250 [13:44:12<85:49:04,  8.69s/it][2025-04-25 21:41:55,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:41:55,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.06 | bwd_microstep: 5693.45 | bwd_inner_microstep: 5680.67 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.88
[2025-04-25 21:41:55,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.06 | bwd: 5693.46 | bwd_inner: 5680.67 | bwd_allreduce: 12.75 | step: 18.88

 14%|█▍        | 5684/41250 [13:44:21<85:36:12,  8.66s/it]
                                                          
{'loss': 0.162, 'grad_norm': 2.7150332927703857, 'learning_rate': 3.879373306111077e-05, 'epoch': 1.38}

 14%|█▍        | 5684/41250 [13:44:21<85:36:12,  8.66s/it][2025-04-25 21:42:04,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.17 | optimizer_step: 0.94
[2025-04-25 21:42:04,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.47 | bwd_microstep: 5705.45 | bwd_inner_microstep: 5692.15 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.18
[2025-04-25 21:42:04,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.47 | bwd: 5705.47 | bwd_inner: 5692.15 | bwd_allreduce: 13.27 | step: 19.18

 14%|█▍        | 5685/41250 [13:44:29<85:30:09,  8.65s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 1.6177912950515747, 'learning_rate': 3.879319589457484e-05, 'epoch': 1.38}

 14%|█▍        | 5685/41250 [13:44:29<85:30:09,  8.65s/it][2025-04-25 21:42:13,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:42:13,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.69 | bwd_microstep: 5689.66 | bwd_inner_microstep: 5676.67 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.74
[2025-04-25 21:42:13,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.69 | bwd: 5689.67 | bwd_inner: 5676.67 | bwd_allreduce: 12.96 | step: 18.74

 14%|█▍        | 5686/41250 [13:44:38<85:25:25,  8.65s/it]
                                                          
{'loss': 0.3128, 'grad_norm': 1.8212904930114746, 'learning_rate': 3.879265861218256e-05, 'epoch': 1.38}

 14%|█▍        | 5686/41250 [13:44:38<85:25:25,  8.65s/it][2025-04-25 21:42:21,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:42:21,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.46 | bwd_microstep: 5778.01 | bwd_inner_microstep: 5765.16 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.57
[2025-04-25 21:42:21,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.46 | bwd: 5778.03 | bwd_inner: 5765.16 | bwd_allreduce: 12.83 | step: 18.57

 14%|█▍        | 5687/41250 [13:44:47<85:42:40,  8.68s/it]
                                                          
{'loss': 0.2135, 'grad_norm': 2.758702039718628, 'learning_rate': 3.8792121213937234e-05, 'epoch': 1.38}

 14%|█▍        | 5687/41250 [13:44:47<85:42:40,  8.68s/it][2025-04-25 21:42:30,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:42:30,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 6043.67 | bwd_inner_microstep: 5647.51 | bwd_allreduce_microstep: 396.11 | step_microstep: 18.92
[2025-04-25 21:42:30,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 6043.68 | bwd_inner: 5647.51 | bwd_allreduce: 396.13 | step: 18.92

 14%|█▍        | 5688/41250 [13:44:56<86:31:27,  8.76s/it]
                                                          
{'loss': 0.1004, 'grad_norm': 2.7078707218170166, 'learning_rate': 3.879158369984218e-05, 'epoch': 1.38}

 14%|█▍        | 5688/41250 [13:44:56<86:31:27,  8.76s/it][2025-04-25 21:42:39,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 21:42:39,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5777.50 | bwd_inner_microstep: 5655.87 | bwd_allreduce_microstep: 121.58 | step_microstep: 18.96
[2025-04-25 21:42:39,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5777.52 | bwd_inner: 5655.87 | bwd_allreduce: 121.61 | step: 18.96

 14%|█▍        | 5689/41250 [13:45:04<86:19:25,  8.74s/it]
                                                          
{'loss': 0.1639, 'grad_norm': 3.0762693881988525, 'learning_rate': 3.879104606990071e-05, 'epoch': 1.38}

 14%|█▍        | 5689/41250 [13:45:04<86:19:25,  8.74s/it][2025-04-25 21:42:48,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:42:48,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.53 | bwd_microstep: 5759.47 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 110.14 | step_microstep: 18.11
[2025-04-25 21:42:48,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.53 | bwd: 5759.49 | bwd_inner: 5649.29 | bwd_allreduce: 110.16 | step: 18.11

 14%|█▍        | 5690/41250 [13:45:13<86:08:40,  8.72s/it]
                                                          
{'loss': 0.2262, 'grad_norm': 3.3197925090789795, 'learning_rate': 3.879050832411613e-05, 'epoch': 1.38}

 14%|█▍        | 5690/41250 [13:45:13<86:08:40,  8.72s/it][2025-04-25 21:42:56,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 1.09
[2025-04-25 21:42:56,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.63 | bwd_microstep: 5763.20 | bwd_inner_microstep: 5647.82 | bwd_allreduce_microstep: 115.34 | step_microstep: 18.33
[2025-04-25 21:42:56,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5763.21 | bwd_inner: 5647.82 | bwd_allreduce: 115.35 | step: 18.34

 14%|█▍        | 5691/41250 [13:45:22<85:59:58,  8.71s/it]
                                                          
{'loss': 0.0699, 'grad_norm': 2.224595785140991, 'learning_rate': 3.8789970462491764e-05, 'epoch': 1.38}

 14%|█▍        | 5691/41250 [13:45:22<85:59:58,  8.71s/it][2025-04-25 21:43:05,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 21:43:05,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.12 | bwd_microstep: 5710.63 | bwd_inner_microstep: 5697.79 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.23
[2025-04-25 21:43:05,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.12 | bwd: 5710.64 | bwd_inner: 5697.79 | bwd_allreduce: 12.81 | step: 18.24

 14%|█▍        | 5692/41250 [13:45:30<85:51:14,  8.69s/it]
                                                          
{'loss': 0.1388, 'grad_norm': 1.692805290222168, 'learning_rate': 3.878943248503093e-05, 'epoch': 1.38}

 14%|█▍        | 5692/41250 [13:45:30<85:51:14,  8.69s/it][2025-04-25 21:43:14,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 21:43:14,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.85 | bwd_microstep: 5787.73 | bwd_inner_microstep: 5775.18 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.63
[2025-04-25 21:43:14,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.85 | bwd: 5787.75 | bwd_inner: 5775.18 | bwd_allreduce: 12.52 | step: 18.63

 14%|█▍        | 5693/41250 [13:45:39<86:02:42,  8.71s/it]
                                                          
{'loss': 0.1884, 'grad_norm': 1.659735918045044, 'learning_rate': 3.878889439173694e-05, 'epoch': 1.38}

 14%|█▍        | 5693/41250 [13:45:39<86:02:42,  8.71s/it][2025-04-25 21:43:22,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:43:22,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5714.66 | bwd_inner_microstep: 5701.92 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.49
[2025-04-25 21:43:22,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5714.67 | bwd_inner: 5701.92 | bwd_allreduce: 12.71 | step: 18.50

 14%|█▍        | 5694/41250 [13:45:48<85:51:39,  8.69s/it]
                                                          
{'loss': 0.1433, 'grad_norm': 2.13973069190979, 'learning_rate': 3.878835618261311e-05, 'epoch': 1.38}

 14%|█▍        | 5694/41250 [13:45:48<85:51:39,  8.69s/it][2025-04-25 21:43:31,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:43:31,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5709.34 | bwd_inner_microstep: 5650.25 | bwd_allreduce_microstep: 59.05 | step_microstep: 18.18
[2025-04-25 21:43:31,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5709.36 | bwd_inner: 5650.25 | bwd_allreduce: 59.07 | step: 18.18

 14%|█▍        | 5695/41250 [13:45:56<85:39:21,  8.67s/it]
                                                          
{'loss': 0.2706, 'grad_norm': 3.4817028045654297, 'learning_rate': 3.878781785766276e-05, 'epoch': 1.38}

 14%|█▍        | 5695/41250 [13:45:56<85:39:21,  8.67s/it][2025-04-25 21:43:40,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 21:43:40,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5766.57 | bwd_inner_microstep: 5652.01 | bwd_allreduce_microstep: 114.51 | step_microstep: 19.25
[2025-04-25 21:43:40,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5766.58 | bwd_inner: 5652.01 | bwd_allreduce: 114.53 | step: 19.25

 14%|█▍        | 5696/41250 [13:46:05<85:41:05,  8.68s/it]
                                                          
{'loss': 0.074, 'grad_norm': 1.452658772468567, 'learning_rate': 3.87872794168892e-05, 'epoch': 1.38}

 14%|█▍        | 5696/41250 [13:46:05<85:41:05,  8.68s/it][2025-04-25 21:43:48,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:43:48,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.38 | bwd_microstep: 5726.08 | bwd_inner_microstep: 5663.39 | bwd_allreduce_microstep: 62.64 | step_microstep: 18.53
[2025-04-25 21:43:48,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.38 | bwd: 5726.09 | bwd_inner: 5663.39 | bwd_allreduce: 62.66 | step: 18.53

 14%|█▍        | 5697/41250 [13:46:14<85:36:34,  8.67s/it]
                                                          
{'loss': 0.0797, 'grad_norm': 1.38713538646698, 'learning_rate': 3.878674086029577e-05, 'epoch': 1.38}

 14%|█▍        | 5697/41250 [13:46:14<85:36:34,  8.67s/it][2025-04-25 21:43:57,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-25 21:43:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.40 | bwd_microstep: 5687.39 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 24.12 | step_microstep: 18.88
[2025-04-25 21:43:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.40 | bwd: 5687.40 | bwd_inner: 5663.21 | bwd_allreduce: 24.14 | step: 18.88

 14%|█▍        | 5698/41250 [13:46:22<85:27:27,  8.65s/it]
                                                          
{'loss': 0.1831, 'grad_norm': 2.263737916946411, 'learning_rate': 3.8786202187885776e-05, 'epoch': 1.38}

 14%|█▍        | 5698/41250 [13:46:22<85:27:27,  8.65s/it][2025-04-25 21:44:06,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:44:06,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.98 | bwd_microstep: 5778.62 | bwd_inner_microstep: 5722.88 | bwd_allreduce_microstep: 55.70 | step_microstep: 18.89
[2025-04-25 21:44:06,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.98 | bwd: 5778.63 | bwd_inner: 5722.88 | bwd_allreduce: 55.71 | step: 18.90

 14%|█▍        | 5699/41250 [13:46:31<85:39:01,  8.67s/it]
                                                          
{'loss': 0.169, 'grad_norm': 1.8436477184295654, 'learning_rate': 3.878566339966254e-05, 'epoch': 1.38}

 14%|█▍        | 5699/41250 [13:46:31<85:39:01,  8.67s/it][2025-04-25 21:44:14,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 21:44:14,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.77 | bwd_microstep: 5772.90 | bwd_inner_microstep: 5715.81 | bwd_allreduce_microstep: 57.05 | step_microstep: 19.14
[2025-04-25 21:44:14,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.77 | bwd: 5772.91 | bwd_inner: 5715.81 | bwd_allreduce: 57.06 | step: 19.14

 14%|█▍        | 5700/41250 [13:46:40<85:45:20,  8.68s/it]
                                                          
{'loss': 0.0331, 'grad_norm': 0.571779727935791, 'learning_rate': 3.878512449562938e-05, 'epoch': 1.38}

 14%|█▍        | 5700/41250 [13:46:40<85:45:20,  8.68s/it][2025-04-25 21:44:23,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:44:23,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.45 | bwd_microstep: 5710.07 | bwd_inner_microstep: 5697.09 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.67
[2025-04-25 21:44:23,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.45 | bwd: 5710.08 | bwd_inner: 5697.09 | bwd_allreduce: 12.95 | step: 18.67

 14%|█▍        | 5701/41250 [13:46:48<85:37:18,  8.67s/it]
                                                          
{'loss': 0.1822, 'grad_norm': 1.72425377368927, 'learning_rate': 3.878458547578962e-05, 'epoch': 1.38}

 14%|█▍        | 5701/41250 [13:46:48<85:37:18,  8.67s/it][2025-04-25 21:44:32,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 21:44:32,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.56 | bwd_microstep: 5718.51 | bwd_inner_microstep: 5706.16 | bwd_allreduce_microstep: 12.30 | step_microstep: 18.32
[2025-04-25 21:44:32,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.56 | bwd: 5718.52 | bwd_inner: 5706.16 | bwd_allreduce: 12.32 | step: 18.32

 14%|█▍        | 5702/41250 [13:46:57<85:33:52,  8.67s/it]
                                                          
{'loss': 0.0599, 'grad_norm': 1.4293957948684692, 'learning_rate': 3.878404634014659e-05, 'epoch': 1.38}

 14%|█▍        | 5702/41250 [13:46:57<85:33:52,  8.67s/it][2025-04-25 21:44:40,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-25 21:44:40,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.61 | bwd_microstep: 5721.50 | bwd_inner_microstep: 5676.91 | bwd_allreduce_microstep: 44.55 | step_microstep: 19.23
[2025-04-25 21:44:40,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.61 | bwd: 5721.52 | bwd_inner: 5676.91 | bwd_allreduce: 44.57 | step: 19.23

 14%|█▍        | 5703/41250 [13:47:06<85:28:52,  8.66s/it]
                                                          
{'loss': 0.0855, 'grad_norm': 1.9913661479949951, 'learning_rate': 3.878350708870361e-05, 'epoch': 1.38}

 14%|█▍        | 5703/41250 [13:47:06<85:28:52,  8.66s/it][2025-04-25 21:44:49,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:44:49,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.58 | bwd_microstep: 5730.93 | bwd_inner_microstep: 5718.23 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.92
[2025-04-25 21:44:49,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.58 | bwd: 5730.94 | bwd_inner: 5718.23 | bwd_allreduce: 12.67 | step: 18.93

 14%|█▍        | 5704/41250 [13:47:14<85:31:23,  8.66s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.6355468034744263, 'learning_rate': 3.8782967721464e-05, 'epoch': 1.38}

 14%|█▍        | 5704/41250 [13:47:14<85:31:23,  8.66s/it][2025-04-25 21:44:58,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:44:58,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.81 | bwd_microstep: 5786.34 | bwd_inner_microstep: 5773.58 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.70
[2025-04-25 21:44:58,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.81 | bwd: 5786.36 | bwd_inner: 5773.58 | bwd_allreduce: 12.74 | step: 18.70

 14%|█▍        | 5705/41250 [13:47:23<85:48:12,  8.69s/it]
                                                          
{'loss': 0.1853, 'grad_norm': 1.3517807722091675, 'learning_rate': 3.878242823843109e-05, 'epoch': 1.38}

 14%|█▍        | 5705/41250 [13:47:23<85:48:12,  8.69s/it][2025-04-25 21:45:07,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:45:07,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.73 | bwd_microstep: 5797.74 | bwd_inner_microstep: 5671.83 | bwd_allreduce_microstep: 125.86 | step_microstep: 18.93
[2025-04-25 21:45:07,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.73 | bwd: 5797.75 | bwd_inner: 5671.83 | bwd_allreduce: 125.88 | step: 18.93

 14%|█▍        | 5706/41250 [13:47:32<85:52:35,  8.70s/it]
                                                          
{'loss': 0.2022, 'grad_norm': 2.8608899116516113, 'learning_rate': 3.87818886396082e-05, 'epoch': 1.38}

 14%|█▍        | 5706/41250 [13:47:32<85:52:35,  8.70s/it][2025-04-25 21:45:15,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 1.23 | optimizer_step: 0.93
[2025-04-25 21:45:15,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5720.41 | bwd_inner_microstep: 5676.64 | bwd_allreduce_microstep: 43.73 | step_microstep: 19.08
[2025-04-25 21:45:15,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5720.42 | bwd_inner: 5676.64 | bwd_allreduce: 43.75 | step: 19.08

 14%|█▍        | 5707/41250 [13:47:41<85:41:39,  8.68s/it]
                                                          
{'loss': 0.0279, 'grad_norm': 0.5598965287208557, 'learning_rate': 3.878134892499866e-05, 'epoch': 1.38}

 14%|█▍        | 5707/41250 [13:47:41<85:41:39,  8.68s/it][2025-04-25 21:45:24,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-25 21:45:24,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.94 | bwd_microstep: 5746.56 | bwd_inner_microstep: 5708.17 | bwd_allreduce_microstep: 38.35 | step_microstep: 18.72
[2025-04-25 21:45:24,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.94 | bwd: 5746.57 | bwd_inner: 5708.17 | bwd_allreduce: 38.37 | step: 18.72

 14%|█▍        | 5708/41250 [13:47:49<85:42:02,  8.68s/it]
                                                          
{'loss': 0.2436, 'grad_norm': 1.7796581983566284, 'learning_rate': 3.878080909460581e-05, 'epoch': 1.38}

 14%|█▍        | 5708/41250 [13:47:49<85:42:02,  8.68s/it][2025-04-25 21:45:33,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:45:33,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.70 | bwd_microstep: 5790.40 | bwd_inner_microstep: 5661.18 | bwd_allreduce_microstep: 129.17 | step_microstep: 18.96
[2025-04-25 21:45:33,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.70 | bwd: 5790.41 | bwd_inner: 5661.18 | bwd_allreduce: 129.19 | step: 18.96

 14%|█▍        | 5709/41250 [13:47:58<85:48:58,  8.69s/it]
                                                          
{'loss': 0.0784, 'grad_norm': 0.7603529691696167, 'learning_rate': 3.8780269148432956e-05, 'epoch': 1.38}

 14%|█▍        | 5709/41250 [13:47:58<85:48:58,  8.69s/it][2025-04-25 21:45:41,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 21:45:41,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.58 | bwd_microstep: 5873.33 | bwd_inner_microstep: 5716.93 | bwd_allreduce_microstep: 156.36 | step_microstep: 19.34
[2025-04-25 21:45:41,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.58 | bwd: 5873.35 | bwd_inner: 5716.93 | bwd_allreduce: 156.38 | step: 19.34

 14%|█▍        | 5710/41250 [13:48:07<86:09:44,  8.73s/it]
                                                          
{'loss': 0.0848, 'grad_norm': 1.5750917196273804, 'learning_rate': 3.877972908648344e-05, 'epoch': 1.38}

 14%|█▍        | 5710/41250 [13:48:07<86:09:44,  8.73s/it][2025-04-25 21:45:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 21:45:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.36 | bwd_microstep: 5730.16 | bwd_inner_microstep: 5717.35 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.68
[2025-04-25 21:45:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.36 | bwd: 5730.17 | bwd_inner: 5717.35 | bwd_allreduce: 12.78 | step: 18.68

 14%|█▍        | 5711/41250 [13:48:15<86:00:38,  8.71s/it]
                                                          
{'loss': 0.0549, 'grad_norm': 0.9476281404495239, 'learning_rate': 3.8779188908760585e-05, 'epoch': 1.38}

 14%|█▍        | 5711/41250 [13:48:15<86:00:38,  8.71s/it][2025-04-25 21:45:59,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:45:59,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5762.76 | bwd_inner_microstep: 5710.60 | bwd_allreduce_microstep: 52.12 | step_microstep: 18.79
[2025-04-25 21:45:59,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5762.77 | bwd_inner: 5710.60 | bwd_allreduce: 52.13 | step: 18.80

 14%|█▍        | 5712/41250 [13:48:24<85:57:32,  8.71s/it]
                                                          
{'loss': 0.1242, 'grad_norm': 1.7636487483978271, 'learning_rate': 3.877864861526773e-05, 'epoch': 1.38}

 14%|█▍        | 5712/41250 [13:48:24<85:57:32,  8.71s/it][2025-04-25 21:46:07,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:46:07,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.44 | bwd_microstep: 5693.97 | bwd_inner_microstep: 5668.84 | bwd_allreduce_microstep: 25.08 | step_microstep: 18.65
[2025-04-25 21:46:07,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.44 | bwd: 5693.98 | bwd_inner: 5668.84 | bwd_allreduce: 25.10 | step: 18.65

 14%|█▍        | 5713/41250 [13:48:33<85:39:57,  8.68s/it]
                                                          
{'loss': 0.1109, 'grad_norm': 1.8546826839447021, 'learning_rate': 3.87781082060082e-05, 'epoch': 1.38}

 14%|█▍        | 5713/41250 [13:48:33<85:39:57,  8.68s/it][2025-04-25 21:46:16,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 21:46:16,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.74 | bwd_microstep: 5716.64 | bwd_inner_microstep: 5703.94 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.67
[2025-04-25 21:46:16,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.74 | bwd: 5716.65 | bwd_inner: 5703.94 | bwd_allreduce: 12.66 | step: 18.68

 14%|█▍        | 5714/41250 [13:48:41<85:36:53,  8.67s/it]
                                                          
{'loss': 0.1722, 'grad_norm': 5.103842735290527, 'learning_rate': 3.877756768098532e-05, 'epoch': 1.39}

 14%|█▍        | 5714/41250 [13:48:41<85:36:53,  8.67s/it][2025-04-25 21:46:25,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:46:25,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.22 | bwd_microstep: 5753.00 | bwd_inner_microstep: 5711.76 | bwd_allreduce_microstep: 41.19 | step_microstep: 18.83
[2025-04-25 21:46:25,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.22 | bwd: 5753.02 | bwd_inner: 5711.77 | bwd_allreduce: 41.21 | step: 18.83

 14%|█▍        | 5715/41250 [13:48:50<85:41:02,  8.68s/it]
                                                          
{'loss': 0.0553, 'grad_norm': 1.152808666229248, 'learning_rate': 3.877702704020243e-05, 'epoch': 1.39}

 14%|█▍        | 5715/41250 [13:48:50<85:41:02,  8.68s/it][2025-04-25 21:46:33,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 21:46:33,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.81 | bwd_microstep: 5764.27 | bwd_inner_microstep: 5656.71 | bwd_allreduce_microstep: 107.52 | step_microstep: 18.55
[2025-04-25 21:46:33,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.81 | bwd: 5764.29 | bwd_inner: 5656.71 | bwd_allreduce: 107.54 | step: 18.55

 14%|█▍        | 5716/41250 [13:48:59<85:40:22,  8.68s/it]
                                                          
{'loss': 0.1067, 'grad_norm': 1.8836474418640137, 'learning_rate': 3.877648628366286e-05, 'epoch': 1.39}

 14%|█▍        | 5716/41250 [13:48:59<85:40:22,  8.68s/it][2025-04-25 21:46:42,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.95
[2025-04-25 21:46:42,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.77 | bwd_microstep: 5715.80 | bwd_inner_microstep: 5702.89 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.25
[2025-04-25 21:46:42,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.77 | bwd: 5715.81 | bwd_inner: 5702.89 | bwd_allreduce: 12.88 | step: 19.25

 14%|█▍        | 5717/41250 [13:49:07<85:34:20,  8.67s/it]
                                                          
{'loss': 0.0911, 'grad_norm': 1.4370940923690796, 'learning_rate': 3.877594541136995e-05, 'epoch': 1.39}

 14%|█▍        | 5717/41250 [13:49:07<85:34:20,  8.67s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x18ff2480] moov atom not found
[21:46:44] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4...
[2025-04-25 21:46:51,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 21:46:51,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5696.29 | bwd_inner_microstep: 5644.68 | bwd_allreduce_microstep: 51.56 | step_microstep: 18.71
[2025-04-25 21:46:51,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5696.30 | bwd_inner: 5644.68 | bwd_allreduce: 51.58 | step: 18.71

 14%|█▍        | 5718/41250 [13:49:16<85:23:26,  8.65s/it]
                                                          
{'loss': 0.036, 'grad_norm': 0.5774238705635071, 'learning_rate': 3.877540442332703e-05, 'epoch': 1.39}

 14%|█▍        | 5718/41250 [13:49:16<85:23:26,  8.65s/it][2025-04-25 21:46:59,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 21:46:59,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.53 | bwd_microstep: 5736.50 | bwd_inner_microstep: 5685.37 | bwd_allreduce_microstep: 51.09 | step_microstep: 18.64
[2025-04-25 21:46:59,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.53 | bwd: 5736.52 | bwd_inner: 5685.37 | bwd_allreduce: 51.10 | step: 18.64

 14%|█▍        | 5719/41250 [13:49:25<85:25:03,  8.65s/it]
                                                          
{'loss': 0.2643, 'grad_norm': 1.9334242343902588, 'learning_rate': 3.877486331953743e-05, 'epoch': 1.39}

 14%|█▍        | 5719/41250 [13:49:25<85:25:03,  8.65s/it][2025-04-25 21:47:08,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:47:08,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.99 | bwd_microstep: 5734.76 | bwd_inner_microstep: 5710.43 | bwd_allreduce_microstep: 24.28 | step_microstep: 19.08
[2025-04-25 21:47:08,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.99 | bwd: 5734.78 | bwd_inner: 5710.43 | bwd_allreduce: 24.30 | step: 19.08

 14%|█▍        | 5720/41250 [13:49:33<85:27:02,  8.66s/it]
                                                          
{'loss': 0.0731, 'grad_norm': 2.510830879211426, 'learning_rate': 3.877432210000449e-05, 'epoch': 1.39}

 14%|█▍        | 5720/41250 [13:49:33<85:27:02,  8.66s/it][2025-04-25 21:47:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:47:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.63 | bwd_microstep: 5770.26 | bwd_inner_microstep: 5657.00 | bwd_allreduce_microstep: 113.22 | step_microstep: 18.65
[2025-04-25 21:47:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.63 | bwd: 5770.27 | bwd_inner: 5657.00 | bwd_allreduce: 113.23 | step: 18.66

 14%|█▍        | 5721/41250 [13:49:42<85:32:14,  8.67s/it]
                                                          
{'loss': 0.2478, 'grad_norm': 3.187695026397705, 'learning_rate': 3.877378076473156e-05, 'epoch': 1.39}

 14%|█▍        | 5721/41250 [13:49:42<85:32:14,  8.67s/it][2025-04-25 21:47:25,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:47:25,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.97 | bwd_microstep: 5751.67 | bwd_inner_microstep: 5688.45 | bwd_allreduce_microstep: 63.17 | step_microstep: 18.91
[2025-04-25 21:47:25,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.97 | bwd: 5751.68 | bwd_inner: 5688.45 | bwd_allreduce: 63.19 | step: 18.91

 14%|█▍        | 5722/41250 [13:49:51<85:34:45,  8.67s/it]
                                                          
{'loss': 0.2536, 'grad_norm': 2.5074808597564697, 'learning_rate': 3.8773239313721945e-05, 'epoch': 1.39}

 14%|█▍        | 5722/41250 [13:49:51<85:34:45,  8.67s/it][2025-04-25 21:47:34,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 21:47:34,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5722.01 | bwd_inner_microstep: 5693.78 | bwd_allreduce_microstep: 28.18 | step_microstep: 19.21
[2025-04-25 21:47:34,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5722.02 | bwd_inner: 5693.78 | bwd_allreduce: 28.20 | step: 19.21

 14%|█▍        | 5723/41250 [13:49:59<85:32:33,  8.67s/it]
                                                          
{'loss': 0.2854, 'grad_norm': 4.297073841094971, 'learning_rate': 3.877269774697901e-05, 'epoch': 1.39}

 14%|█▍        | 5723/41250 [13:49:59<85:32:33,  8.67s/it][2025-04-25 21:47:43,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:47:43,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.50 | bwd_microstep: 5703.92 | bwd_inner_microstep: 5650.13 | bwd_allreduce_microstep: 53.75 | step_microstep: 18.83
[2025-04-25 21:47:43,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.50 | bwd: 5703.94 | bwd_inner: 5650.13 | bwd_allreduce: 53.76 | step: 18.83

 14%|█▍        | 5724/41250 [13:50:08<85:22:13,  8.65s/it]
                                                          
{'loss': 0.0795, 'grad_norm': 1.662947416305542, 'learning_rate': 3.8772156064506086e-05, 'epoch': 1.39}

 14%|█▍        | 5724/41250 [13:50:08<85:22:13,  8.65s/it][2025-04-25 21:47:52,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 21:47:52,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5997.45 | bwd_inner_microstep: 5703.28 | bwd_allreduce_microstep: 294.12 | step_microstep: 19.29
[2025-04-25 21:47:52,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5997.46 | bwd_inner: 5703.28 | bwd_allreduce: 294.14 | step: 19.29

 14%|█▍        | 5725/41250 [13:50:17<86:11:23,  8.73s/it]
                                                          
{'loss': 0.0127, 'grad_norm': 0.17695143818855286, 'learning_rate': 3.877161426630652e-05, 'epoch': 1.39}

 14%|█▍        | 5725/41250 [13:50:17<86:11:23,  8.73s/it][2025-04-25 21:48:00,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.10 | optimizer_step: 0.95
[2025-04-25 21:48:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.53 | bwd_microstep: 5747.95 | bwd_inner_microstep: 5704.44 | bwd_allreduce_microstep: 43.47 | step_microstep: 19.07
[2025-04-25 21:48:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5747.96 | bwd_inner: 5704.44 | bwd_allreduce: 43.48 | step: 19.08

 14%|█▍        | 5726/41250 [13:50:26<86:02:20,  8.72s/it]
                                                          
{'loss': 0.03, 'grad_norm': 0.795855700969696, 'learning_rate': 3.877107235238363e-05, 'epoch': 1.39}

 14%|█▍        | 5726/41250 [13:50:26<86:02:20,  8.72s/it][2025-04-25 21:48:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 21:48:09,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.06 | bwd_microstep: 5746.23 | bwd_inner_microstep: 5688.77 | bwd_allreduce_microstep: 57.41 | step_microstep: 18.63
[2025-04-25 21:48:09,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.07 | bwd: 5746.24 | bwd_inner: 5688.77 | bwd_allreduce: 57.43 | step: 18.63

 14%|█▍        | 5727/41250 [13:50:34<85:53:56,  8.71s/it]
                                                          
{'loss': 0.0582, 'grad_norm': 1.7485995292663574, 'learning_rate': 3.877053032274078e-05, 'epoch': 1.39}

 14%|█▍        | 5727/41250 [13:50:34<85:53:56,  8.71s/it][2025-04-25 21:48:18,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 21:48:18,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.32 | bwd_microstep: 5691.68 | bwd_inner_microstep: 5678.89 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.79
[2025-04-25 21:48:18,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.33 | bwd: 5691.70 | bwd_inner: 5678.89 | bwd_allreduce: 12.77 | step: 18.79

 14%|█▍        | 5728/41250 [13:50:43<85:37:55,  8.68s/it]
                                                          
{'loss': 0.0589, 'grad_norm': 1.2237637042999268, 'learning_rate': 3.876998817738131e-05, 'epoch': 1.39}

 14%|█▍        | 5728/41250 [13:50:43<85:37:55,  8.68s/it][2025-04-25 21:48:26,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 21:48:26,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5718.13 | bwd_inner_microstep: 5705.41 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.67
[2025-04-25 21:48:26,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5718.14 | bwd_inner: 5705.41 | bwd_allreduce: 12.69 | step: 18.68

 14%|█▍        | 5729/41250 [13:50:52<85:32:14,  8.67s/it]
                                                          
{'loss': 0.1944, 'grad_norm': 1.9234812259674072, 'learning_rate': 3.876944591630854e-05, 'epoch': 1.39}

 14%|█▍        | 5729/41250 [13:50:52<85:32:14,  8.67s/it][2025-04-25 21:48:35,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:48:35,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.17 | bwd_microstep: 5709.76 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.77
[2025-04-25 21:48:35,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.17 | bwd: 5709.77 | bwd_inner: 5696.99 | bwd_allreduce: 12.74 | step: 18.77

 14%|█▍        | 5730/41250 [13:51:00<85:26:27,  8.66s/it]
                                                          
{'loss': 0.0695, 'grad_norm': 2.868527412414551, 'learning_rate': 3.8768903539525824e-05, 'epoch': 1.39}

 14%|█▍        | 5730/41250 [13:51:00<85:26:27,  8.66s/it][2025-04-25 21:48:44,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 1.08
[2025-04-25 21:48:44,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5983.28 | bwd_inner_microstep: 5705.75 | bwd_allreduce_microstep: 277.48 | step_microstep: 19.46
[2025-04-25 21:48:44,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.73 | bwd: 5983.30 | bwd_inner: 5705.75 | bwd_allreduce: 277.50 | step: 19.47

 14%|█▍        | 5731/41250 [13:51:09<86:12:05,  8.74s/it]
                                                          
{'loss': 0.08, 'grad_norm': 2.5555636882781982, 'learning_rate': 3.8768361047036525e-05, 'epoch': 1.39}

 14%|█▍        | 5731/41250 [13:51:09<86:12:05,  8.74s/it][2025-04-25 21:48:52,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.12 | optimizer_step: 1.00
[2025-04-25 21:48:52,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.97 | bwd_microstep: 5729.61 | bwd_inner_microstep: 5685.65 | bwd_allreduce_microstep: 43.91 | step_microstep: 19.07
[2025-04-25 21:48:52,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.97 | bwd: 5729.62 | bwd_inner: 5685.65 | bwd_allreduce: 43.93 | step: 19.08

 14%|█▍        | 5732/41250 [13:51:18<85:59:36,  8.72s/it]
                                                          
{'loss': 0.051, 'grad_norm': 0.6453459858894348, 'learning_rate': 3.876781843884396e-05, 'epoch': 1.39}

 14%|█▍        | 5732/41250 [13:51:18<85:59:36,  8.72s/it][2025-04-25 21:49:01,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:49:01,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.68 | bwd_microstep: 5698.40 | bwd_inner_microstep: 5685.55 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.40
[2025-04-25 21:49:01,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.68 | bwd: 5698.41 | bwd_inner: 5685.55 | bwd_allreduce: 12.82 | step: 18.40

 14%|█▍        | 5733/41250 [13:51:26<85:42:51,  8.69s/it]
                                                          
{'loss': 0.0564, 'grad_norm': 1.4197590351104736, 'learning_rate': 3.876727571495149e-05, 'epoch': 1.39}

 14%|█▍        | 5733/41250 [13:51:26<85:42:51,  8.69s/it][2025-04-25 21:49:10,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:49:10,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.70 | bwd_microstep: 5722.28 | bwd_inner_microstep: 5674.83 | bwd_allreduce_microstep: 47.41 | step_microstep: 18.77
[2025-04-25 21:49:10,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.70 | bwd: 5722.30 | bwd_inner: 5674.83 | bwd_allreduce: 47.42 | step: 18.77

 14%|█▍        | 5734/41250 [13:51:35<85:38:49,  8.68s/it]
                                                          
{'loss': 0.111, 'grad_norm': 1.3781685829162598, 'learning_rate': 3.876673287536246e-05, 'epoch': 1.39}

 14%|█▍        | 5734/41250 [13:51:35<85:38:49,  8.68s/it][2025-04-25 21:49:18,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 21:49:18,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.65 | bwd_microstep: 5744.77 | bwd_inner_microstep: 5650.88 | bwd_allreduce_microstep: 93.84 | step_microstep: 19.05
[2025-04-25 21:49:18,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.65 | bwd: 5744.79 | bwd_inner: 5650.88 | bwd_allreduce: 93.86 | step: 19.06

 14%|█▍        | 5735/41250 [13:51:44<85:34:04,  8.67s/it]
                                                          
{'loss': 0.1382, 'grad_norm': 1.264591932296753, 'learning_rate': 3.876618992008021e-05, 'epoch': 1.39}

 14%|█▍        | 5735/41250 [13:51:44<85:34:04,  8.67s/it][2025-04-25 21:49:27,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:49:27,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.68 | bwd_microstep: 5764.84 | bwd_inner_microstep: 5645.43 | bwd_allreduce_microstep: 119.37 | step_microstep: 18.51
[2025-04-25 21:49:27,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.68 | bwd: 5764.85 | bwd_inner: 5645.43 | bwd_allreduce: 119.38 | step: 18.52

 14%|█▍        | 5736/41250 [13:51:52<85:34:28,  8.67s/it]
                                                          
{'loss': 0.1935, 'grad_norm': 2.544330358505249, 'learning_rate': 3.8765646849108096e-05, 'epoch': 1.39}

 14%|█▍        | 5736/41250 [13:51:52<85:34:28,  8.67s/it][2025-04-25 21:49:36,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 21:49:36,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5696.60 | bwd_inner_microstep: 5683.90 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.56
[2025-04-25 21:49:36,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5696.61 | bwd_inner: 5683.90 | bwd_allreduce: 12.67 | step: 18.57

 14%|█▍        | 5737/41250 [13:52:01<85:26:54,  8.66s/it]
                                                          
{'loss': 0.2428, 'grad_norm': 2.290504217147827, 'learning_rate': 3.876510366244945e-05, 'epoch': 1.39}

 14%|█▍        | 5737/41250 [13:52:01<85:26:54,  8.66s/it][2025-04-25 21:49:44,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:49:44,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.27 | bwd_microstep: 5767.28 | bwd_inner_microstep: 5637.03 | bwd_allreduce_microstep: 130.21 | step_microstep: 18.35
[2025-04-25 21:49:44,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.27 | bwd: 5767.30 | bwd_inner: 5637.03 | bwd_allreduce: 130.23 | step: 18.35

 14%|█▍        | 5738/41250 [13:52:10<85:29:13,  8.67s/it]
                                                          
{'loss': 0.1397, 'grad_norm': 3.4501497745513916, 'learning_rate': 3.876456036010764e-05, 'epoch': 1.39}

 14%|█▍        | 5738/41250 [13:52:10<85:29:13,  8.67s/it][2025-04-25 21:49:53,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:49:53,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.70 | bwd_microstep: 5713.11 | bwd_inner_microstep: 5685.48 | bwd_allreduce_microstep: 27.59 | step_microstep: 18.20
[2025-04-25 21:49:53,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.70 | bwd: 5713.13 | bwd_inner: 5685.48 | bwd_allreduce: 27.61 | step: 18.20

 14%|█▍        | 5739/41250 [13:52:18<85:26:55,  8.66s/it]
                                                          
{'loss': 0.084, 'grad_norm': 1.0185654163360596, 'learning_rate': 3.8764016942085996e-05, 'epoch': 1.39}

 14%|█▍        | 5739/41250 [13:52:18<85:26:55,  8.66s/it][2025-04-25 21:50:02,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:50:02,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.29 | bwd_microstep: 5709.54 | bwd_inner_microstep: 5696.46 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.63
[2025-04-25 21:50:02,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.29 | bwd: 5709.55 | bwd_inner: 5696.46 | bwd_allreduce: 13.05 | step: 18.63

 14%|█▍        | 5740/41250 [13:52:27<85:25:44,  8.66s/it]
                                                          
{'loss': 0.3283, 'grad_norm': 4.299954414367676, 'learning_rate': 3.8763473408387884e-05, 'epoch': 1.39}

 14%|█▍        | 5740/41250 [13:52:27<85:25:44,  8.66s/it][2025-04-25 21:50:10,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 21:50:10,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.48 | bwd_microstep: 5735.84 | bwd_inner_microstep: 5691.70 | bwd_allreduce_microstep: 44.10 | step_microstep: 18.98
[2025-04-25 21:50:10,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.48 | bwd: 5735.85 | bwd_inner: 5691.70 | bwd_allreduce: 44.12 | step: 18.99

 14%|█▍        | 5741/41250 [13:52:36<85:27:52,  8.66s/it]
                                                          
{'loss': 0.0626, 'grad_norm': 1.2254819869995117, 'learning_rate': 3.876292975901665e-05, 'epoch': 1.39}

 14%|█▍        | 5741/41250 [13:52:36<85:27:52,  8.66s/it][2025-04-25 21:50:19,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 21:50:19,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2930.15 | bwd_microstep: 5888.54 | bwd_inner_microstep: 5875.69 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.15
[2025-04-25 21:50:19,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2930.15 | bwd: 5888.56 | bwd_inner: 5875.69 | bwd_allreduce: 12.82 | step: 19.15

 14%|█▍        | 5742/41250 [13:52:45<86:11:23,  8.74s/it]
                                                          
{'loss': 0.0956, 'grad_norm': 1.398473858833313, 'learning_rate': 3.8762385993975646e-05, 'epoch': 1.39}

 14%|█▍        | 5742/41250 [13:52:45<86:11:23,  8.74s/it][2025-04-25 21:50:28,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:50:28,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.22 | bwd_microstep: 5879.03 | bwd_inner_microstep: 5650.07 | bwd_allreduce_microstep: 228.92 | step_microstep: 18.69
[2025-04-25 21:50:28,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.22 | bwd: 5879.04 | bwd_inner: 5650.07 | bwd_allreduce: 228.93 | step: 18.70

 14%|█▍        | 5743/41250 [13:52:53<86:20:05,  8.75s/it]
                                                          
{'loss': 0.2321, 'grad_norm': 1.5424368381500244, 'learning_rate': 3.8761842113268214e-05, 'epoch': 1.39}

 14%|█▍        | 5743/41250 [13:52:53<86:20:05,  8.75s/it][2025-04-25 21:50:37,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:50:37,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.52 | bwd_microstep: 5686.94 | bwd_inner_microstep: 5658.38 | bwd_allreduce_microstep: 28.52 | step_microstep: 18.46
[2025-04-25 21:50:37,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.52 | bwd: 5686.95 | bwd_inner: 5658.38 | bwd_allreduce: 28.53 | step: 18.46

 14%|█▍        | 5744/41250 [13:53:02<85:54:22,  8.71s/it]
                                                          
{'loss': 0.0877, 'grad_norm': 1.6143277883529663, 'learning_rate': 3.876129811689772e-05, 'epoch': 1.39}

 14%|█▍        | 5744/41250 [13:53:02<85:54:22,  8.71s/it][2025-04-25 21:50:45,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:50:45,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.16 | bwd_microstep: 5787.58 | bwd_inner_microstep: 5661.36 | bwd_allreduce_microstep: 126.17 | step_microstep: 18.79
[2025-04-25 21:50:45,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.16 | bwd: 5787.60 | bwd_inner: 5661.36 | bwd_allreduce: 126.19 | step: 18.79

 14%|█▍        | 5745/41250 [13:53:11<85:53:19,  8.71s/it]
                                                          
{'loss': 0.0882, 'grad_norm': 2.8928706645965576, 'learning_rate': 3.8760754004867516e-05, 'epoch': 1.39}

 14%|█▍        | 5745/41250 [13:53:11<85:53:19,  8.71s/it][2025-04-25 21:50:54,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:50:54,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5707.90 | bwd_inner_microstep: 5646.92 | bwd_allreduce_microstep: 60.93 | step_microstep: 19.02
[2025-04-25 21:50:54,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5707.91 | bwd_inner: 5646.92 | bwd_allreduce: 60.95 | step: 19.03

 14%|█▍        | 5746/41250 [13:53:19<85:37:21,  8.68s/it]
                                                          
{'loss': 0.18, 'grad_norm': 4.241868495941162, 'learning_rate': 3.876020977718096e-05, 'epoch': 1.39}

 14%|█▍        | 5746/41250 [13:53:19<85:37:21,  8.68s/it][2025-04-25 21:51:03,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-25 21:51:03,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.03 | bwd_microstep: 5718.59 | bwd_inner_microstep: 5650.51 | bwd_allreduce_microstep: 68.03 | step_microstep: 19.18
[2025-04-25 21:51:03,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.03 | bwd: 5718.61 | bwd_inner: 5650.51 | bwd_allreduce: 68.05 | step: 19.19

 14%|█▍        | 5747/41250 [13:53:28<85:28:50,  8.67s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 1.0823923349380493, 'learning_rate': 3.875966543384139e-05, 'epoch': 1.39}

 14%|█▍        | 5747/41250 [13:53:28<85:28:50,  8.67s/it][2025-04-25 21:51:11,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 21:51:11,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.54 | bwd_microstep: 5744.20 | bwd_inner_microstep: 5699.80 | bwd_allreduce_microstep: 44.36 | step_microstep: 19.03
[2025-04-25 21:51:11,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.54 | bwd: 5744.22 | bwd_inner: 5699.80 | bwd_allreduce: 44.38 | step: 19.03

 14%|█▍        | 5748/41250 [13:53:37<85:31:03,  8.67s/it]
                                                          
{'loss': 0.0396, 'grad_norm': 1.451864242553711, 'learning_rate': 3.875912097485218e-05, 'epoch': 1.39}

 14%|█▍        | 5748/41250 [13:53:37<85:31:03,  8.67s/it][2025-04-25 21:51:20,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-25 21:51:20,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5708.05 | bwd_inner_microstep: 5694.76 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.28
[2025-04-25 21:51:20,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5708.06 | bwd_inner: 5694.76 | bwd_allreduce: 13.26 | step: 19.28

 14%|█▍        | 5749/41250 [13:53:45<85:25:23,  8.66s/it]
                                                          
{'loss': 0.0301, 'grad_norm': 0.800682783126831, 'learning_rate': 3.875857640021667e-05, 'epoch': 1.39}

 14%|█▍        | 5749/41250 [13:53:45<85:25:23,  8.66s/it][2025-04-25 21:51:29,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 21:51:29,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.26 | bwd_microstep: 5753.46 | bwd_inner_microstep: 5656.96 | bwd_allreduce_microstep: 96.45 | step_microstep: 18.72
[2025-04-25 21:51:29,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.26 | bwd: 5753.47 | bwd_inner: 5656.96 | bwd_allreduce: 96.47 | step: 18.72

 14%|█▍        | 5750/41250 [13:53:54<85:26:36,  8.66s/it]
                                                          
{'loss': 0.0717, 'grad_norm': 1.20710289478302, 'learning_rate': 3.8758031709938234e-05, 'epoch': 1.39}

 14%|█▍        | 5750/41250 [13:53:54<85:26:36,  8.66s/it][2025-04-25 21:51:37,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:51:37,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5730.47 | bwd_inner_microstep: 5717.52 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.06
[2025-04-25 21:51:37,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5730.48 | bwd_inner: 5717.52 | bwd_allreduce: 12.92 | step: 19.06

 14%|█▍        | 5751/41250 [13:54:03<85:27:27,  8.67s/it]
                                                          
{'loss': 0.2405, 'grad_norm': 3.2214772701263428, 'learning_rate': 3.875748690402022e-05, 'epoch': 1.39}

 14%|█▍        | 5751/41250 [13:54:03<85:27:27,  8.67s/it][2025-04-25 21:51:46,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:51:46,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.68 | bwd_microstep: 5732.70 | bwd_inner_microstep: 5720.02 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.59
[2025-04-25 21:51:46,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.68 | bwd: 5732.72 | bwd_inner: 5720.02 | bwd_allreduce: 12.66 | step: 18.60

 14%|█▍        | 5752/41250 [13:54:11<85:30:15,  8.67s/it]
                                                          
{'loss': 0.2253, 'grad_norm': 2.9906158447265625, 'learning_rate': 3.875694198246598e-05, 'epoch': 1.39}

 14%|█▍        | 5752/41250 [13:54:11<85:30:15,  8.67s/it][2025-04-25 21:51:55,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:51:55,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.76 | bwd_microstep: 5775.93 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 124.07 | step_microstep: 18.98
[2025-04-25 21:51:55,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.76 | bwd: 5775.94 | bwd_inner: 5651.81 | bwd_allreduce: 124.09 | step: 18.98

 14%|█▍        | 5753/41250 [13:54:20<85:34:22,  8.68s/it]
                                                          
{'loss': 0.1889, 'grad_norm': 1.5983004570007324, 'learning_rate': 3.875639694527889e-05, 'epoch': 1.39}

 14%|█▍        | 5753/41250 [13:54:20<85:34:22,  8.68s/it][2025-04-25 21:52:03,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 21:52:03,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.82 | bwd_microstep: 5786.86 | bwd_inner_microstep: 5661.79 | bwd_allreduce_microstep: 125.03 | step_microstep: 18.90
[2025-04-25 21:52:03,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.82 | bwd: 5786.88 | bwd_inner: 5661.79 | bwd_allreduce: 125.04 | step: 18.90

 14%|█▍        | 5754/41250 [13:54:29<85:40:38,  8.69s/it]
                                                          
{'loss': 0.1078, 'grad_norm': 1.0637422800064087, 'learning_rate': 3.8755851792462306e-05, 'epoch': 1.39}

 14%|█▍        | 5754/41250 [13:54:29<85:40:38,  8.69s/it][2025-04-25 21:52:12,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:52:12,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.17 | bwd_microstep: 5753.53 | bwd_inner_microstep: 5700.34 | bwd_allreduce_microstep: 53.15 | step_microstep: 18.34
[2025-04-25 21:52:12,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.17 | bwd: 5753.54 | bwd_inner: 5700.34 | bwd_allreduce: 53.16 | step: 18.35

 14%|█▍        | 5755/41250 [13:54:37<85:42:18,  8.69s/it]
                                                          
{'loss': 0.3498, 'grad_norm': 2.4493143558502197, 'learning_rate': 3.8755306524019584e-05, 'epoch': 1.4}

 14%|█▍        | 5755/41250 [13:54:37<85:42:18,  8.69s/it][2025-04-25 21:52:21,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 21:52:21,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.24 | bwd_microstep: 5908.15 | bwd_inner_microstep: 5658.30 | bwd_allreduce_microstep: 249.80 | step_microstep: 18.99
[2025-04-25 21:52:21,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.24 | bwd: 5908.17 | bwd_inner: 5658.30 | bwd_allreduce: 249.82 | step: 19.00

 14%|█▍        | 5756/41250 [13:54:46<86:07:09,  8.73s/it]
                                                          
{'loss': 0.0504, 'grad_norm': 0.8119387626647949, 'learning_rate': 3.875476113995408e-05, 'epoch': 1.4}

 14%|█▍        | 5756/41250 [13:54:46<86:07:09,  8.73s/it][2025-04-25 21:52:30,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 21:52:30,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.45 | bwd_microstep: 5758.26 | bwd_inner_microstep: 5692.54 | bwd_allreduce_microstep: 65.67 | step_microstep: 19.02
[2025-04-25 21:52:30,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.45 | bwd: 5758.28 | bwd_inner: 5692.54 | bwd_allreduce: 65.69 | step: 19.02

 14%|█▍        | 5757/41250 [13:54:55<85:59:06,  8.72s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 1.8181382417678833, 'learning_rate': 3.8754215640269166e-05, 'epoch': 1.4}

 14%|█▍        | 5757/41250 [13:54:55<85:59:06,  8.72s/it][2025-04-25 21:52:38,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:52:38,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.66 | bwd_microstep: 5713.83 | bwd_inner_microstep: 5668.95 | bwd_allreduce_microstep: 44.83 | step_microstep: 18.84
[2025-04-25 21:52:38,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.66 | bwd: 5713.85 | bwd_inner: 5668.95 | bwd_allreduce: 44.85 | step: 18.84

 14%|█▍        | 5758/41250 [13:55:04<85:43:13,  8.69s/it]
                                                          
{'loss': 0.0515, 'grad_norm': 0.9847307801246643, 'learning_rate': 3.87536700249682e-05, 'epoch': 1.4}

 14%|█▍        | 5758/41250 [13:55:04<85:43:13,  8.69s/it][2025-04-25 21:52:47,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 21:52:47,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.98 | bwd_microstep: 5738.80 | bwd_inner_microstep: 5725.80 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.99
[2025-04-25 21:52:47,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.98 | bwd: 5738.81 | bwd_inner: 5725.80 | bwd_allreduce: 12.97 | step: 18.99

 14%|█▍        | 5759/41250 [13:55:12<85:41:28,  8.69s/it]
                                                          
{'loss': 0.1692, 'grad_norm': 3.4113454818725586, 'learning_rate': 3.875312429405454e-05, 'epoch': 1.4}

 14%|█▍        | 5759/41250 [13:55:12<85:41:28,  8.69s/it][2025-04-25 21:52:56,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.13
[2025-04-25 21:52:56,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.36 | bwd_microstep: 5784.08 | bwd_inner_microstep: 5710.54 | bwd_allreduce_microstep: 73.49 | step_microstep: 19.47
[2025-04-25 21:52:56,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.36 | bwd: 5784.10 | bwd_inner: 5710.54 | bwd_allreduce: 73.51 | step: 19.47

 14%|█▍        | 5760/41250 [13:55:21<85:46:53,  8.70s/it]
                                                          
{'loss': 0.0676, 'grad_norm': 2.1067605018615723, 'learning_rate': 3.8752578447531564e-05, 'epoch': 1.4}

 14%|█▍        | 5760/41250 [13:55:21<85:46:53,  8.70s/it][2025-04-25 21:53:04,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 21:53:04,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.08 | bwd_microstep: 5729.25 | bwd_inner_microstep: 5716.45 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.99
[2025-04-25 21:53:04,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.08 | bwd: 5729.27 | bwd_inner: 5716.45 | bwd_allreduce: 12.78 | step: 18.99

 14%|█▍        | 5761/41250 [13:55:30<85:40:59,  8.69s/it]
                                                          
{'loss': 0.0359, 'grad_norm': 1.0150946378707886, 'learning_rate': 3.875203248540263e-05, 'epoch': 1.4}

 14%|█▍        | 5761/41250 [13:55:30<85:40:59,  8.69s/it][2025-04-25 21:53:13,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.16 | optimizer_step: 0.92
[2025-04-25 21:53:13,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.60 | bwd_microstep: 5860.90 | bwd_inner_microstep: 5712.39 | bwd_allreduce_microstep: 148.46 | step_microstep: 19.02
[2025-04-25 21:53:13,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.61 | bwd: 5860.92 | bwd_inner: 5712.39 | bwd_allreduce: 148.48 | step: 19.02

 14%|█▍        | 5762/41250 [13:55:38<85:59:53,  8.72s/it]
                                                          
{'loss': 0.1584, 'grad_norm': 1.7788664102554321, 'learning_rate': 3.875148640767111e-05, 'epoch': 1.4}

 14%|█▍        | 5762/41250 [13:55:38<85:59:53,  8.72s/it][2025-04-25 21:53:22,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 21:53:22,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5705.77 | bwd_inner_microstep: 5693.00 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.03
[2025-04-25 21:53:22,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5705.79 | bwd_inner: 5693.00 | bwd_allreduce: 12.73 | step: 19.03

 14%|█▍        | 5763/41250 [13:55:47<85:45:00,  8.70s/it]
                                                          
{'loss': 0.0879, 'grad_norm': 0.9363600015640259, 'learning_rate': 3.875094021434036e-05, 'epoch': 1.4}

 14%|█▍        | 5763/41250 [13:55:47<85:45:00,  8.70s/it][2025-04-25 21:53:30,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:53:30,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.99 | bwd_microstep: 5748.50 | bwd_inner_microstep: 5703.54 | bwd_allreduce_microstep: 44.91 | step_microstep: 18.52
[2025-04-25 21:53:30,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.99 | bwd: 5748.51 | bwd_inner: 5703.54 | bwd_allreduce: 44.93 | step: 18.52

 14%|█▍        | 5764/41250 [13:55:56<85:41:42,  8.69s/it]
                                                          
{'loss': 0.0964, 'grad_norm': 2.0029497146606445, 'learning_rate': 3.8750393905413746e-05, 'epoch': 1.4}

 14%|█▍        | 5764/41250 [13:55:56<85:41:42,  8.69s/it][2025-04-25 21:53:39,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 21:53:39,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5783.14 | bwd_inner_microstep: 5651.08 | bwd_allreduce_microstep: 132.01 | step_microstep: 19.32
[2025-04-25 21:53:39,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5783.15 | bwd_inner: 5651.08 | bwd_allreduce: 132.03 | step: 19.32

 14%|█▍        | 5765/41250 [13:56:04<85:41:08,  8.69s/it]
                                                          
{'loss': 0.0474, 'grad_norm': 1.3364062309265137, 'learning_rate': 3.874984748089465e-05, 'epoch': 1.4}

 14%|█▍        | 5765/41250 [13:56:04<85:41:08,  8.69s/it][2025-04-25 21:53:48,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:53:48,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5715.18 | bwd_inner_microstep: 5663.30 | bwd_allreduce_microstep: 51.84 | step_microstep: 18.60
[2025-04-25 21:53:48,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.45 | bwd: 5715.20 | bwd_inner: 5663.30 | bwd_allreduce: 51.86 | step: 18.60

 14%|█▍        | 5766/41250 [13:56:13<85:29:47,  8.67s/it]
                                                          
{'loss': 0.229, 'grad_norm': 1.9999562501907349, 'learning_rate': 3.874930094078643e-05, 'epoch': 1.4}

 14%|█▍        | 5766/41250 [13:56:13<85:29:47,  8.67s/it][2025-04-25 21:53:56,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 21:53:56,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.86 | bwd_microstep: 5773.76 | bwd_inner_microstep: 5649.41 | bwd_allreduce_microstep: 124.30 | step_microstep: 19.12
[2025-04-25 21:53:56,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.86 | bwd: 5773.77 | bwd_inner: 5649.41 | bwd_allreduce: 124.32 | step: 19.13

 14%|█▍        | 5767/41250 [13:56:22<85:32:33,  8.68s/it]
                                                          
{'loss': 0.2812, 'grad_norm': 2.878133773803711, 'learning_rate': 3.874875428509246e-05, 'epoch': 1.4}

 14%|█▍        | 5767/41250 [13:56:22<85:32:33,  8.68s/it][2025-04-25 21:54:05,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:54:05,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.08 | bwd_microstep: 5697.84 | bwd_inner_microstep: 5661.31 | bwd_allreduce_microstep: 36.49 | step_microstep: 19.04
[2025-04-25 21:54:05,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.08 | bwd: 5697.86 | bwd_inner: 5661.31 | bwd_allreduce: 36.51 | step: 19.05

 14%|█▍        | 5768/41250 [13:56:30<85:20:19,  8.66s/it]
                                                          
{'loss': 0.1274, 'grad_norm': 1.839848279953003, 'learning_rate': 3.87482075138161e-05, 'epoch': 1.4}

 14%|█▍        | 5768/41250 [13:56:30<85:20:19,  8.66s/it][2025-04-25 21:54:14,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 21:54:14,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.20 | bwd_microstep: 5717.75 | bwd_inner_microstep: 5704.84 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.68
[2025-04-25 21:54:14,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.20 | bwd: 5717.77 | bwd_inner: 5704.84 | bwd_allreduce: 12.88 | step: 18.68

 14%|█▍        | 5769/41250 [13:56:39<85:20:05,  8.66s/it]
                                                          
{'loss': 0.1088, 'grad_norm': 0.9682438969612122, 'learning_rate': 3.874766062696073e-05, 'epoch': 1.4}

 14%|█▍        | 5769/41250 [13:56:39<85:20:05,  8.66s/it][2025-04-25 21:54:22,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-25 21:54:22,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.72 | bwd_microstep: 5763.50 | bwd_inner_microstep: 5681.93 | bwd_allreduce_microstep: 81.52 | step_microstep: 19.17
[2025-04-25 21:54:22,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.72 | bwd: 5763.51 | bwd_inner: 5681.93 | bwd_allreduce: 81.54 | step: 19.17

 14%|█▍        | 5770/41250 [13:56:48<85:27:45,  8.67s/it]
                                                          
{'loss': 0.3608, 'grad_norm': 2.564032793045044, 'learning_rate': 3.874711362452972e-05, 'epoch': 1.4}

 14%|█▍        | 5770/41250 [13:56:48<85:27:45,  8.67s/it][2025-04-25 21:54:31,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 21:54:31,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.21 | bwd_microstep: 5749.01 | bwd_inner_microstep: 5706.32 | bwd_allreduce_microstep: 42.64 | step_microstep: 18.84
[2025-04-25 21:54:31,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.21 | bwd: 5749.02 | bwd_inner: 5706.32 | bwd_allreduce: 42.66 | step: 18.84

 14%|█▍        | 5771/41250 [13:56:56<85:30:18,  8.68s/it]
                                                          
{'loss': 0.0236, 'grad_norm': 0.5178430676460266, 'learning_rate': 3.874656650652644e-05, 'epoch': 1.4}

 14%|█▍        | 5771/41250 [13:56:56<85:30:18,  8.68s/it][2025-04-25 21:54:40,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-25 21:54:40,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.28 | bwd_microstep: 5773.28 | bwd_inner_microstep: 5655.88 | bwd_allreduce_microstep: 117.35 | step_microstep: 19.30
[2025-04-25 21:54:40,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.28 | bwd: 5773.30 | bwd_inner: 5655.88 | bwd_allreduce: 117.37 | step: 19.30

 14%|█▍        | 5772/41250 [13:57:05<85:33:45,  8.68s/it]
                                                          
{'loss': 0.1811, 'grad_norm': 1.289467692375183, 'learning_rate': 3.874601927295427e-05, 'epoch': 1.4}

 14%|█▍        | 5772/41250 [13:57:05<85:33:45,  8.68s/it][2025-04-25 21:54:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 21:54:48,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.70 | bwd_microstep: 5720.92 | bwd_inner_microstep: 5646.57 | bwd_allreduce_microstep: 74.31 | step_microstep: 19.29
[2025-04-25 21:54:48,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.70 | bwd: 5720.94 | bwd_inner: 5646.57 | bwd_allreduce: 74.32 | step: 19.29

 14%|█▍        | 5773/41250 [13:57:14<85:25:47,  8.67s/it]
                                                          
{'loss': 0.2867, 'grad_norm': 1.769667625427246, 'learning_rate': 3.8745471923816576e-05, 'epoch': 1.4}

 14%|█▍        | 5773/41250 [13:57:14<85:25:47,  8.67s/it][2025-04-25 21:54:57,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-25 21:54:57,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.97 | bwd_microstep: 5698.52 | bwd_inner_microstep: 5685.67 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.97
[2025-04-25 21:54:57,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.97 | bwd: 5698.53 | bwd_inner: 5685.67 | bwd_allreduce: 12.82 | step: 18.98

 14%|█▍        | 5774/41250 [13:57:22<85:17:29,  8.66s/it]
                                                          
{'loss': 0.1252, 'grad_norm': 1.562077283859253, 'learning_rate': 3.8744924459116734e-05, 'epoch': 1.4}

 14%|█▍        | 5774/41250 [13:57:22<85:17:29,  8.66s/it][2025-04-25 21:55:06,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 21:55:06,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.40 | bwd_microstep: 5749.40 | bwd_inner_microstep: 5688.34 | bwd_allreduce_microstep: 61.01 | step_microstep: 18.60
[2025-04-25 21:55:06,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.40 | bwd: 5749.41 | bwd_inner: 5688.34 | bwd_allreduce: 61.03 | step: 18.61

 14%|█▍        | 5775/41250 [13:57:31<85:22:03,  8.66s/it]
                                                          
{'loss': 0.1899, 'grad_norm': 1.6367706060409546, 'learning_rate': 3.874437687885812e-05, 'epoch': 1.4}

 14%|█▍        | 5775/41250 [13:57:31<85:22:03,  8.66s/it][2025-04-25 21:55:14,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 21:55:14,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.33 | bwd_microstep: 5733.77 | bwd_inner_microstep: 5698.09 | bwd_allreduce_microstep: 35.63 | step_microstep: 19.10
[2025-04-25 21:55:14,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.33 | bwd: 5733.78 | bwd_inner: 5698.09 | bwd_allreduce: 35.65 | step: 19.10

 14%|█▍        | 5776/41250 [13:57:40<85:21:55,  8.66s/it]
                                                          
{'loss': 0.0277, 'grad_norm': 0.8317041397094727, 'learning_rate': 3.874382918304411e-05, 'epoch': 1.4}

 14%|█▍        | 5776/41250 [13:57:40<85:21:55,  8.66s/it][2025-04-25 21:55:23,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 21:55:23,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5737.21 | bwd_inner_microstep: 5709.80 | bwd_allreduce_microstep: 27.36 | step_microstep: 19.34
[2025-04-25 21:55:23,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5737.23 | bwd_inner: 5709.80 | bwd_allreduce: 27.38 | step: 19.34

 14%|█▍        | 5777/41250 [13:57:48<85:23:24,  8.67s/it]
                                                          
{'loss': 0.0394, 'grad_norm': 0.4655708372592926, 'learning_rate': 3.8743281371678085e-05, 'epoch': 1.4}

 14%|█▍        | 5777/41250 [13:57:48<85:23:24,  8.67s/it][2025-04-25 21:55:32,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 21:55:32,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.85 | bwd_microstep: 5736.03 | bwd_inner_microstep: 5645.96 | bwd_allreduce_microstep: 90.02 | step_microstep: 18.93
[2025-04-25 21:55:32,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.85 | bwd: 5736.05 | bwd_inner: 5645.96 | bwd_allreduce: 90.04 | step: 18.93

 14%|█▍        | 5778/41250 [13:57:57<85:21:06,  8.66s/it]
                                                          
{'loss': 0.2015, 'grad_norm': 2.456530809402466, 'learning_rate': 3.874273344476341e-05, 'epoch': 1.4}

 14%|█▍        | 5778/41250 [13:57:57<85:21:06,  8.66s/it][2025-04-25 21:55:40,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 21:55:40,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.63 | bwd_microstep: 5677.37 | bwd_inner_microstep: 5651.32 | bwd_allreduce_microstep: 26.01 | step_microstep: 18.72
[2025-04-25 21:55:40,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.63 | bwd: 5677.38 | bwd_inner: 5651.32 | bwd_allreduce: 26.02 | step: 18.73

 14%|█▍        | 5779/41250 [13:58:06<85:10:56,  8.65s/it]
                                                          
{'loss': 0.1342, 'grad_norm': 1.7136582136154175, 'learning_rate': 3.874218540230347e-05, 'epoch': 1.4}

 14%|█▍        | 5779/41250 [13:58:06<85:10:56,  8.65s/it][2025-04-25 21:55:49,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 21:55:49,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.49 | bwd_microstep: 5740.87 | bwd_inner_microstep: 5644.43 | bwd_allreduce_microstep: 96.40 | step_microstep: 18.96
[2025-04-25 21:55:49,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.49 | bwd: 5740.89 | bwd_inner: 5644.43 | bwd_allreduce: 96.42 | step: 18.97

 14%|█▍        | 5780/41250 [13:58:14<85:12:20,  8.65s/it]
                                                          
{'loss': 0.2544, 'grad_norm': 2.474437952041626, 'learning_rate': 3.8741637244301646e-05, 'epoch': 1.4}

 14%|█▍        | 5780/41250 [13:58:14<85:12:20,  8.65s/it][2025-04-25 21:55:58,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 21:55:58,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.50 | bwd_microstep: 5740.86 | bwd_inner_microstep: 5700.82 | bwd_allreduce_microstep: 39.99 | step_microstep: 19.00
[2025-04-25 21:55:58,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.50 | bwd: 5740.87 | bwd_inner: 5700.82 | bwd_allreduce: 40.01 | step: 19.00

 14%|█▍        | 5781/41250 [13:58:23<85:17:19,  8.66s/it]
                                                          
{'loss': 0.0387, 'grad_norm': 0.7303967475891113, 'learning_rate': 3.8741088970761314e-05, 'epoch': 1.4}

 14%|█▍        | 5781/41250 [13:58:23<85:17:19,  8.66s/it][2025-04-25 21:56:06,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 21:56:06,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.14 | bwd_microstep: 5738.99 | bwd_inner_microstep: 5689.55 | bwd_allreduce_microstep: 49.39 | step_microstep: 18.83
[2025-04-25 21:56:06,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.14 | bwd: 5739.00 | bwd_inner: 5689.55 | bwd_allreduce: 49.41 | step: 18.83

 14%|█▍        | 5782/41250 [13:58:32<85:19:21,  8.66s/it]
                                                          
{'loss': 0.0477, 'grad_norm': 0.9105989933013916, 'learning_rate': 3.874054058168585e-05, 'epoch': 1.4}

 14%|█▍        | 5782/41250 [13:58:32<85:19:21,  8.66s/it][2025-04-25 21:56:15,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-25 21:56:15,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.00 | bwd_microstep: 5699.39 | bwd_inner_microstep: 5686.80 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.58
[2025-04-25 21:56:15,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.00 | bwd: 5699.40 | bwd_inner: 5686.80 | bwd_allreduce: 12.56 | step: 18.58

 14%|█▍        | 5783/41250 [13:58:40<85:13:41,  8.65s/it]
                                                          
{'loss': 0.0724, 'grad_norm': 0.7996006011962891, 'learning_rate': 3.873999207707865e-05, 'epoch': 1.4}

 14%|█▍        | 5783/41250 [13:58:40<85:13:41,  8.65s/it][2025-04-25 21:56:24,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 21:56:24,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.27 | bwd_microstep: 5715.12 | bwd_inner_microstep: 5702.53 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.48
[2025-04-25 21:56:24,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.27 | bwd: 5715.13 | bwd_inner: 5702.53 | bwd_allreduce: 12.56 | step: 18.49

 14%|█▍        | 5784/41250 [13:58:49<85:14:09,  8.65s/it]
                                                          
{'loss': 0.2017, 'grad_norm': 3.4448628425598145, 'learning_rate': 3.873944345694308e-05, 'epoch': 1.4}

 14%|█▍        | 5784/41250 [13:58:49<85:14:09,  8.65s/it][2025-04-25 21:56:32,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 21:56:32,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.20 | bwd_microstep: 5712.04 | bwd_inner_microstep: 5699.07 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.60
[2025-04-25 21:56:32,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.20 | bwd: 5712.05 | bwd_inner: 5699.07 | bwd_allreduce: 12.94 | step: 18.60

 14%|█▍        | 5785/41250 [13:58:58<85:12:10,  8.65s/it]
                                                          
{'loss': 0.1224, 'grad_norm': 2.808861255645752, 'learning_rate': 3.8738894721282524e-05, 'epoch': 1.4}

 14%|█▍        | 5785/41250 [13:58:58<85:12:10,  8.65s/it][2025-04-25 21:56:41,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:56:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.10 | bwd_microstep: 5771.09 | bwd_inner_microstep: 5631.34 | bwd_allreduce_microstep: 139.70 | step_microstep: 18.62
[2025-04-25 21:56:41,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.10 | bwd: 5771.11 | bwd_inner: 5631.34 | bwd_allreduce: 139.72 | step: 18.62

 14%|█▍        | 5786/41250 [13:59:06<85:17:21,  8.66s/it]
                                                          
{'loss': 0.208, 'grad_norm': 2.737283945083618, 'learning_rate': 3.873834587010037e-05, 'epoch': 1.4}

 14%|█▍        | 5786/41250 [13:59:06<85:17:21,  8.66s/it][2025-04-25 21:56:50,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-25 21:56:50,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.74 | bwd_microstep: 5756.00 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 97.09 | step_microstep: 19.13
[2025-04-25 21:56:50,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.74 | bwd: 5756.02 | bwd_inner: 5658.86 | bwd_allreduce: 97.11 | step: 19.13

 14%|█▍        | 5787/41250 [13:59:15<85:18:26,  8.66s/it]
                                                          
{'loss': 0.0705, 'grad_norm': 1.140403151512146, 'learning_rate': 3.87377969034e-05, 'epoch': 1.4}

 14%|█▍        | 5787/41250 [13:59:15<85:18:26,  8.66s/it][2025-04-25 21:56:58,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:56:58,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.02 | bwd_microstep: 5716.72 | bwd_inner_microstep: 5704.04 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.78
[2025-04-25 21:56:58,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.02 | bwd: 5716.73 | bwd_inner: 5704.04 | bwd_allreduce: 12.65 | step: 18.78

 14%|█▍        | 5788/41250 [13:59:24<85:16:40,  8.66s/it]
                                                          
{'loss': 0.1487, 'grad_norm': 1.8643600940704346, 'learning_rate': 3.873724782118479e-05, 'epoch': 1.4}

 14%|█▍        | 5788/41250 [13:59:24<85:16:40,  8.66s/it][2025-04-25 21:57:07,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.05 | optimizer_step: 1.09
[2025-04-25 21:57:07,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.74 | bwd_microstep: 5730.90 | bwd_inner_microstep: 5683.37 | bwd_allreduce_microstep: 47.48 | step_microstep: 19.02
[2025-04-25 21:57:07,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.74 | bwd: 5730.92 | bwd_inner: 5683.37 | bwd_allreduce: 47.50 | step: 19.02

 14%|█▍        | 5789/41250 [13:59:32<85:15:50,  8.66s/it]
                                                          
{'loss': 0.1807, 'grad_norm': 2.8208658695220947, 'learning_rate': 3.873669862345814e-05, 'epoch': 1.4}

 14%|█▍        | 5789/41250 [13:59:32<85:15:50,  8.66s/it][2025-04-25 21:57:15,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:57:15,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.23 | bwd_microstep: 5682.11 | bwd_inner_microstep: 5651.02 | bwd_allreduce_microstep: 31.04 | step_microstep: 19.15
[2025-04-25 21:57:15,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.23 | bwd: 5682.12 | bwd_inner: 5651.02 | bwd_allreduce: 31.05 | step: 19.15

 14%|█▍        | 5790/41250 [13:59:41<85:04:34,  8.64s/it]
                                                          
{'loss': 0.1729, 'grad_norm': 5.7255988121032715, 'learning_rate': 3.873614931022343e-05, 'epoch': 1.4}

 14%|█▍        | 5790/41250 [13:59:41<85:04:34,  8.64s/it][2025-04-25 21:57:24,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 21:57:24,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.83 | bwd_microstep: 5756.62 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 101.33 | step_microstep: 18.57
[2025-04-25 21:57:24,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.83 | bwd: 5756.64 | bwd_inner: 5655.24 | bwd_allreduce: 101.35 | step: 18.57

 14%|█▍        | 5791/41250 [13:59:49<85:11:53,  8.65s/it]
                                                          
{'loss': 0.2557, 'grad_norm': 1.2293674945831299, 'learning_rate': 3.873559988148404e-05, 'epoch': 1.4}

 14%|█▍        | 5791/41250 [13:59:49<85:11:53,  8.65s/it][2025-04-25 21:57:33,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.08 | optimizer_step: 1.04
[2025-04-25 21:57:33,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5759.88 | bwd_inner_microstep: 5658.70 | bwd_allreduce_microstep: 101.13 | step_microstep: 19.85
[2025-04-25 21:57:33,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5759.90 | bwd_inner: 5658.70 | bwd_allreduce: 101.15 | step: 19.85

 14%|█▍        | 5792/41250 [13:59:58<85:16:26,  8.66s/it]
                                                          
{'loss': 0.1377, 'grad_norm': 2.3128035068511963, 'learning_rate': 3.873505033724336e-05, 'epoch': 1.4}

 14%|█▍        | 5792/41250 [13:59:58<85:16:26,  8.66s/it][2025-04-25 21:57:41,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.11 | optimizer_step: 1.05
[2025-04-25 21:57:41,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5701.61 | bwd_inner_microstep: 5687.93 | bwd_allreduce_microstep: 13.62 | step_microstep: 19.43
[2025-04-25 21:57:41,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5701.63 | bwd_inner: 5687.93 | bwd_allreduce: 13.65 | step: 19.43

 14%|█▍        | 5793/41250 [14:00:07<85:13:28,  8.65s/it]
                                                          
{'loss': 0.0897, 'grad_norm': 4.790627479553223, 'learning_rate': 3.8734500677504785e-05, 'epoch': 1.4}

 14%|█▍        | 5793/41250 [14:00:07<85:13:28,  8.65s/it][2025-04-25 21:57:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 21:57:50,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.85 | bwd_microstep: 5670.67 | bwd_inner_microstep: 5657.99 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.25
[2025-04-25 21:57:50,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5670.68 | bwd_inner: 5657.99 | bwd_allreduce: 12.65 | step: 19.25

 14%|█▍        | 5794/41250 [14:00:15<85:05:37,  8.64s/it]
                                                          
{'loss': 0.1319, 'grad_norm': 2.966675043106079, 'learning_rate': 3.8733950902271695e-05, 'epoch': 1.4}

 14%|█▍        | 5794/41250 [14:00:15<85:05:37,  8.64s/it][2025-04-25 21:57:59,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 21:57:59,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5708.27 | bwd_inner_microstep: 5648.59 | bwd_allreduce_microstep: 59.62 | step_microstep: 19.62
[2025-04-25 21:57:59,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5708.29 | bwd_inner: 5648.59 | bwd_allreduce: 59.65 | step: 19.62

 14%|█▍        | 5795/41250 [14:00:24<85:01:15,  8.63s/it]
                                                          
{'loss': 0.263, 'grad_norm': 3.1326234340667725, 'learning_rate': 3.8733401011547485e-05, 'epoch': 1.4}

 14%|█▍        | 5795/41250 [14:00:24<85:01:15,  8.63s/it][2025-04-25 21:58:07,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.22 | optimizer_step: 1.11
[2025-04-25 21:58:07,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5754.47 | bwd_inner_microstep: 5701.20 | bwd_allreduce_microstep: 53.19 | step_microstep: 20.95
[2025-04-25 21:58:07,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5754.50 | bwd_inner: 5701.20 | bwd_allreduce: 53.23 | step: 20.94

 14%|█▍        | 5796/41250 [14:00:33<85:11:56,  8.65s/it]
                                                          
{'loss': 0.0841, 'grad_norm': 2.0961413383483887, 'learning_rate': 3.8732851005335544e-05, 'epoch': 1.41}

 14%|█▍        | 5796/41250 [14:00:33<85:11:56,  8.65s/it][2025-04-25 21:58:16,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 21:58:16,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.62 | bwd_microstep: 5718.91 | bwd_inner_microstep: 5706.17 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.92
[2025-04-25 21:58:16,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.62 | bwd: 5718.92 | bwd_inner: 5706.17 | bwd_allreduce: 12.71 | step: 18.93

 14%|█▍        | 5797/41250 [14:00:41<85:13:45,  8.65s/it]
                                                          
{'loss': 0.1503, 'grad_norm': 2.870396375656128, 'learning_rate': 3.873230088363926e-05, 'epoch': 1.41}

 14%|█▍        | 5797/41250 [14:00:41<85:13:45,  8.65s/it][2025-04-25 21:58:25,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 21:58:25,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.59 | bwd_microstep: 5692.96 | bwd_inner_microstep: 5661.31 | bwd_allreduce_microstep: 31.60 | step_microstep: 18.79
[2025-04-25 21:58:25,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.59 | bwd: 5692.98 | bwd_inner: 5661.31 | bwd_allreduce: 31.62 | step: 18.79

 14%|█▍        | 5798/41250 [14:00:50<85:04:43,  8.64s/it]
                                                          
{'loss': 0.0365, 'grad_norm': 0.35562247037887573, 'learning_rate': 3.873175064646202e-05, 'epoch': 1.41}

 14%|█▍        | 5798/41250 [14:00:50<85:04:43,  8.64s/it][2025-04-25 21:58:33,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-25 21:58:33,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.61 | bwd_microstep: 5778.16 | bwd_inner_microstep: 5665.95 | bwd_allreduce_microstep: 112.17 | step_microstep: 19.00
[2025-04-25 21:58:33,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.61 | bwd: 5778.18 | bwd_inner: 5665.95 | bwd_allreduce: 112.19 | step: 19.00

 14%|█▍        | 5799/41250 [14:00:59<85:15:51,  8.66s/it]
                                                          
{'loss': 0.0656, 'grad_norm': 1.1915571689605713, 'learning_rate': 3.873120029380723e-05, 'epoch': 1.41}

 14%|█▍        | 5799/41250 [14:00:59<85:15:51,  8.66s/it][2025-04-25 21:58:42,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 21:58:42,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.96 | bwd_microstep: 5749.83 | bwd_inner_microstep: 5707.25 | bwd_allreduce_microstep: 42.52 | step_microstep: 19.02
[2025-04-25 21:58:42,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.96 | bwd: 5749.84 | bwd_inner: 5707.25 | bwd_allreduce: 42.54 | step: 19.02

 14%|█▍        | 5800/41250 [14:01:07<85:21:23,  8.67s/it]
                                                          
{'loss': 0.2048, 'grad_norm': 2.5068509578704834, 'learning_rate': 3.873064982567827e-05, 'epoch': 1.41}

 14%|█▍        | 5800/41250 [14:01:07<85:21:23,  8.67s/it][2025-04-25 21:58:51,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-25 21:58:51,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.19 | bwd_microstep: 5700.13 | bwd_inner_microstep: 5672.43 | bwd_allreduce_microstep: 27.66 | step_microstep: 19.53
[2025-04-25 21:58:51,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.19 | bwd: 5700.15 | bwd_inner: 5672.43 | bwd_allreduce: 27.68 | step: 19.53

 14%|█▍        | 5801/41250 [14:01:16<85:13:46,  8.66s/it]
                                                          
{'loss': 0.1041, 'grad_norm': 1.9750382900238037, 'learning_rate': 3.8730099242078534e-05, 'epoch': 1.41}

 14%|█▍        | 5801/41250 [14:01:16<85:13:46,  8.66s/it][2025-04-25 21:58:59,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 21:58:59,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.44 | bwd_microstep: 5773.08 | bwd_inner_microstep: 5697.28 | bwd_allreduce_microstep: 75.75 | step_microstep: 19.33
[2025-04-25 21:58:59,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.44 | bwd: 5773.09 | bwd_inner: 5697.28 | bwd_allreduce: 75.77 | step: 19.34

 14%|█▍        | 5802/41250 [14:01:25<85:23:35,  8.67s/it]
                                                          
{'loss': 0.381, 'grad_norm': 2.4362690448760986, 'learning_rate': 3.8729548543011423e-05, 'epoch': 1.41}

 14%|█▍        | 5802/41250 [14:01:25<85:23:35,  8.67s/it][2025-04-25 21:59:08,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-25 21:59:08,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.65 | bwd_microstep: 5809.42 | bwd_inner_microstep: 5796.44 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.41
[2025-04-25 21:59:08,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.65 | bwd: 5809.44 | bwd_inner: 5796.44 | bwd_allreduce: 12.96 | step: 19.41

 14%|█▍        | 5803/41250 [14:01:33<85:44:15,  8.71s/it]
                                                          
{'loss': 0.1431, 'grad_norm': 2.0251810550689697, 'learning_rate': 3.872899772848033e-05, 'epoch': 1.41}

 14%|█▍        | 5803/41250 [14:01:34<85:44:15,  8.71s/it][2025-04-25 21:59:17,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-25 21:59:17,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.07 | bwd_microstep: 5717.71 | bwd_inner_microstep: 5660.49 | bwd_allreduce_microstep: 57.16 | step_microstep: 19.42
[2025-04-25 21:59:17,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.07 | bwd: 5717.73 | bwd_inner: 5660.49 | bwd_allreduce: 57.19 | step: 19.44

 14%|█▍        | 5804/41250 [14:01:42<85:30:36,  8.68s/it]
                                                          
{'loss': 0.1567, 'grad_norm': 1.5209641456604004, 'learning_rate': 3.8728446798488655e-05, 'epoch': 1.41}

 14%|█▍        | 5804/41250 [14:01:42<85:30:36,  8.68s/it][2025-04-25 21:59:25,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 21:59:25,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.02 | bwd_microstep: 5729.68 | bwd_inner_microstep: 5716.77 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.71
[2025-04-25 21:59:25,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.02 | bwd: 5729.69 | bwd_inner: 5716.77 | bwd_allreduce: 12.88 | step: 18.72

 14%|█▍        | 5805/41250 [14:01:51<85:27:38,  8.68s/it]
                                                          
{'loss': 0.4979, 'grad_norm': 3.0175983905792236, 'learning_rate': 3.872789575303978e-05, 'epoch': 1.41}

 14%|█▍        | 5805/41250 [14:01:51<85:27:38,  8.68s/it][2025-04-25 21:59:34,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.08
[2025-04-25 21:59:34,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.73 | bwd_microstep: 5719.52 | bwd_inner_microstep: 5694.17 | bwd_allreduce_microstep: 25.30 | step_microstep: 19.49
[2025-04-25 21:59:34,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.73 | bwd: 5719.54 | bwd_inner: 5694.16 | bwd_allreduce: 25.33 | step: 19.49

 14%|█▍        | 5806/41250 [14:01:59<85:24:37,  8.68s/it]
                                                          
{'loss': 0.0555, 'grad_norm': 0.8129034638404846, 'learning_rate': 3.8727344592137116e-05, 'epoch': 1.41}

 14%|█▍        | 5806/41250 [14:01:59<85:24:37,  8.68s/it][2025-04-25 21:59:43,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 21:59:43,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.21 | bwd_microstep: 6009.68 | bwd_inner_microstep: 5724.81 | bwd_allreduce_microstep: 284.82 | step_microstep: 18.85
[2025-04-25 21:59:43,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.21 | bwd: 6009.69 | bwd_inner: 5724.81 | bwd_allreduce: 284.84 | step: 18.85

 14%|█▍        | 5807/41250 [14:02:08<86:13:45,  8.76s/it]
                                                          
{'loss': 0.4203, 'grad_norm': 3.721673011779785, 'learning_rate': 3.872679331578406e-05, 'epoch': 1.41}

 14%|█▍        | 5807/41250 [14:02:08<86:13:45,  8.76s/it][2025-04-25 21:59:52,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 21:59:52,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5796.91 | bwd_inner_microstep: 5648.59 | bwd_allreduce_microstep: 148.27 | step_microstep: 18.88
[2025-04-25 21:59:52,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5796.93 | bwd_inner: 5648.59 | bwd_allreduce: 148.29 | step: 18.88

 14%|█▍        | 5808/41250 [14:02:17<86:04:13,  8.74s/it]
                                                          
{'loss': 0.0282, 'grad_norm': 0.31457871198654175, 'learning_rate': 3.8726241923984e-05, 'epoch': 1.41}

 14%|█▍        | 5808/41250 [14:02:17<86:04:13,  8.74s/it][2025-04-25 22:00:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:00:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5708.92 | bwd_inner_microstep: 5658.77 | bwd_allreduce_microstep: 50.10 | step_microstep: 18.72
[2025-04-25 22:00:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5708.93 | bwd_inner: 5658.77 | bwd_allreduce: 50.12 | step: 18.72

 14%|█▍        | 5809/41250 [14:02:26<85:41:59,  8.71s/it]
                                                          
{'loss': 0.2274, 'grad_norm': 2.065526008605957, 'learning_rate': 3.8725690416740346e-05, 'epoch': 1.41}

 14%|█▍        | 5809/41250 [14:02:26<85:41:59,  8.71s/it][2025-04-25 22:00:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:00:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.12 | bwd_microstep: 5714.13 | bwd_inner_microstep: 5677.88 | bwd_allreduce_microstep: 36.21 | step_microstep: 18.84
[2025-04-25 22:00:09,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.12 | bwd: 5714.15 | bwd_inner: 5677.88 | bwd_allreduce: 36.22 | step: 18.84

 14%|█▍        | 5810/41250 [14:02:34<85:28:26,  8.68s/it]
                                                          
{'loss': 0.1031, 'grad_norm': 1.5965031385421753, 'learning_rate': 3.872513879405649e-05, 'epoch': 1.41}

 14%|█▍        | 5810/41250 [14:02:34<85:28:26,  8.68s/it][2025-04-25 22:00:18,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.32 | optimizer_step: 1.05
[2025-04-25 22:00:18,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.15 | bwd_microstep: 5772.41 | bwd_inner_microstep: 5667.78 | bwd_allreduce_microstep: 104.57 | step_microstep: 19.98
[2025-04-25 22:00:18,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.15 | bwd: 5772.43 | bwd_inner: 5667.78 | bwd_allreduce: 104.60 | step: 19.98

 14%|█▍        | 5811/41250 [14:02:43<85:30:50,  8.69s/it]
                                                          
{'loss': 0.1219, 'grad_norm': 3.568040609359741, 'learning_rate': 3.8724587055935836e-05, 'epoch': 1.41}

 14%|█▍        | 5811/41250 [14:02:43<85:30:50,  8.69s/it][2025-04-25 22:00:27,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-25 22:00:27,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.92 | bwd_microstep: 5859.38 | bwd_inner_microstep: 5713.43 | bwd_allreduce_microstep: 145.89 | step_microstep: 19.11
[2025-04-25 22:00:27,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.92 | bwd: 5859.39 | bwd_inner: 5713.43 | bwd_allreduce: 145.91 | step: 19.11

 14%|█▍        | 5812/41250 [14:02:52<85:51:18,  8.72s/it]
                                                          
{'loss': 0.089, 'grad_norm': 1.3775241374969482, 'learning_rate': 3.872403520238179e-05, 'epoch': 1.41}

 14%|█▍        | 5812/41250 [14:02:52<85:51:18,  8.72s/it][2025-04-25 22:00:35,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 22:00:35,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.32 | bwd_microstep: 5787.89 | bwd_inner_microstep: 5704.02 | bwd_allreduce_microstep: 83.82 | step_microstep: 19.25
[2025-04-25 22:00:35,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.32 | bwd: 5787.90 | bwd_inner: 5704.02 | bwd_allreduce: 83.84 | step: 19.25

 14%|█▍        | 5813/41250 [14:03:01<85:51:31,  8.72s/it]
                                                          
{'loss': 0.1359, 'grad_norm': 1.4103896617889404, 'learning_rate': 3.872348323339774e-05, 'epoch': 1.41}

 14%|█▍        | 5813/41250 [14:03:01<85:51:31,  8.72s/it][2025-04-25 22:00:44,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 22:00:44,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.71 | bwd_microstep: 5714.80 | bwd_inner_microstep: 5702.11 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.43
[2025-04-25 22:00:44,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.71 | bwd: 5714.81 | bwd_inner: 5702.11 | bwd_allreduce: 12.66 | step: 19.44

 14%|█▍        | 5814/41250 [14:03:09<85:38:27,  8.70s/it]
                                                          
{'loss': 0.023, 'grad_norm': 0.3041434586048126, 'learning_rate': 3.872293114898711e-05, 'epoch': 1.41}

 14%|█▍        | 5814/41250 [14:03:09<85:38:27,  8.70s/it][2025-04-25 22:00:53,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 22:00:53,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.58 | bwd_microstep: 5731.51 | bwd_inner_microstep: 5718.16 | bwd_allreduce_microstep: 13.30 | step_microstep: 18.91
[2025-04-25 22:00:53,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.58 | bwd: 5731.52 | bwd_inner: 5718.16 | bwd_allreduce: 13.32 | step: 18.91

 14%|█▍        | 5815/41250 [14:03:18<85:34:33,  8.69s/it]
                                                          
{'loss': 0.1467, 'grad_norm': 2.096825361251831, 'learning_rate': 3.872237894915329e-05, 'epoch': 1.41}

 14%|█▍        | 5815/41250 [14:03:18<85:34:33,  8.69s/it][2025-04-25 22:01:01,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:01:01,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5713.76 | bwd_inner_microstep: 5700.98 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.89
[2025-04-25 22:01:01,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5713.77 | bwd_inner: 5700.98 | bwd_allreduce: 12.75 | step: 18.89

 14%|█▍        | 5816/41250 [14:03:27<85:25:56,  8.68s/it]
                                                          
{'loss': 0.0616, 'grad_norm': 0.8939443230628967, 'learning_rate': 3.872182663389969e-05, 'epoch': 1.41}

 14%|█▍        | 5816/41250 [14:03:27<85:25:56,  8.68s/it][2025-04-25 22:01:10,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:01:10,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.13 | bwd_microstep: 5737.53 | bwd_inner_microstep: 5709.79 | bwd_allreduce_microstep: 27.69 | step_microstep: 18.53
[2025-04-25 22:01:10,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.13 | bwd: 5737.54 | bwd_inner: 5709.79 | bwd_allreduce: 27.71 | step: 18.53

 14%|█▍        | 5817/41250 [14:03:35<85:27:50,  8.68s/it]
                                                          
{'loss': 0.1168, 'grad_norm': 2.2168638706207275, 'learning_rate': 3.87212742032297e-05, 'epoch': 1.41}

 14%|█▍        | 5817/41250 [14:03:35<85:27:50,  8.68s/it][2025-04-25 22:01:19,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-25 22:01:19,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.45 | bwd_microstep: 5696.07 | bwd_inner_microstep: 5663.11 | bwd_allreduce_microstep: 32.91 | step_microstep: 19.19
[2025-04-25 22:01:19,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.45 | bwd: 5696.08 | bwd_inner: 5663.11 | bwd_allreduce: 32.93 | step: 19.19

 14%|█▍        | 5818/41250 [14:03:44<85:16:56,  8.66s/it]
                                                          
{'loss': 0.2891, 'grad_norm': 2.666503667831421, 'learning_rate': 3.872072165714674e-05, 'epoch': 1.41}

 14%|█▍        | 5818/41250 [14:03:44<85:16:56,  8.66s/it][2025-04-25 22:01:27,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 22:01:27,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.40 | bwd_microstep: 5726.64 | bwd_inner_microstep: 5713.90 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.70
[2025-04-25 22:01:27,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.40 | bwd: 5726.65 | bwd_inner: 5713.90 | bwd_allreduce: 12.71 | step: 18.71

 14%|█▍        | 5819/41250 [14:03:53<85:18:29,  8.67s/it]
                                                          
{'loss': 0.143, 'grad_norm': 2.4554970264434814, 'learning_rate': 3.872016899565422e-05, 'epoch': 1.41}

 14%|█▍        | 5819/41250 [14:03:53<85:18:29,  8.67s/it][2025-04-25 22:01:36,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:01:36,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.13 | bwd_microstep: 5737.80 | bwd_inner_microstep: 5706.16 | bwd_allreduce_microstep: 31.59 | step_microstep: 18.69
[2025-04-25 22:01:36,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.13 | bwd: 5737.81 | bwd_inner: 5706.16 | bwd_allreduce: 31.61 | step: 18.69

 14%|█▍        | 5820/41250 [14:04:01<85:18:10,  8.67s/it]
                                                          
{'loss': 0.6133, 'grad_norm': 2.0198183059692383, 'learning_rate': 3.8719616218755533e-05, 'epoch': 1.41}

 14%|█▍        | 5820/41250 [14:04:01<85:18:10,  8.67s/it][2025-04-25 22:01:45,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:01:45,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5726.77 | bwd_inner_microstep: 5697.33 | bwd_allreduce_microstep: 29.40 | step_microstep: 18.40
[2025-04-25 22:01:45,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5726.78 | bwd_inner: 5697.33 | bwd_allreduce: 29.42 | step: 18.40

 14%|█▍        | 5821/41250 [14:04:10<85:16:57,  8.67s/it]
                                                          
{'loss': 0.105, 'grad_norm': 1.589539647102356, 'learning_rate': 3.87190633264541e-05, 'epoch': 1.41}

 14%|█▍        | 5821/41250 [14:04:10<85:16:57,  8.67s/it][2025-04-25 22:01:53,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:01:53,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.18 | bwd_microstep: 5852.85 | bwd_inner_microstep: 5678.73 | bwd_allreduce_microstep: 174.06 | step_microstep: 18.47
[2025-04-25 22:01:53,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.18 | bwd: 5852.87 | bwd_inner: 5678.73 | bwd_allreduce: 174.08 | step: 18.47

 14%|█▍        | 5822/41250 [14:04:19<85:38:20,  8.70s/it]
                                                          
{'loss': 0.2914, 'grad_norm': 3.5780904293060303, 'learning_rate': 3.871851031875332e-05, 'epoch': 1.41}

 14%|█▍        | 5822/41250 [14:04:19<85:38:20,  8.70s/it][2025-04-25 22:02:02,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 22:02:02,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5703.45 | bwd_inner_microstep: 5650.35 | bwd_allreduce_microstep: 53.05 | step_microstep: 19.21
[2025-04-25 22:02:02,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5703.46 | bwd_inner: 5650.35 | bwd_allreduce: 53.07 | step: 19.21

 14%|█▍        | 5823/41250 [14:04:27<85:22:39,  8.68s/it]
                                                          
{'loss': 0.1485, 'grad_norm': 1.3786659240722656, 'learning_rate': 3.87179571956566e-05, 'epoch': 1.41}

 14%|█▍        | 5823/41250 [14:04:27<85:22:39,  8.68s/it][2025-04-25 22:02:11,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-25 22:02:11,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.23 | bwd_microstep: 5724.39 | bwd_inner_microstep: 5711.48 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.95
[2025-04-25 22:02:11,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.23 | bwd: 5724.40 | bwd_inner: 5711.48 | bwd_allreduce: 12.88 | step: 18.95

 14%|█▍        | 5824/41250 [14:04:36<85:20:08,  8.67s/it]
                                                          
{'loss': 0.1346, 'grad_norm': 1.1440273523330688, 'learning_rate': 3.871740395716737e-05, 'epoch': 1.41}

 14%|█▍        | 5824/41250 [14:04:36<85:20:08,  8.67s/it][2025-04-25 22:02:19,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-25 22:02:19,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5748.54 | bwd_inner_microstep: 5696.84 | bwd_allreduce_microstep: 51.65 | step_microstep: 18.93
[2025-04-25 22:02:19,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5748.56 | bwd_inner: 5696.84 | bwd_allreduce: 51.67 | step: 18.93

 14%|█▍        | 5825/41250 [14:04:45<85:22:19,  8.68s/it]
                                                          
{'loss': 0.1584, 'grad_norm': 2.0580742359161377, 'learning_rate': 3.871685060328901e-05, 'epoch': 1.41}

 14%|█▍        | 5825/41250 [14:04:45<85:22:19,  8.68s/it][2025-04-25 22:02:28,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:02:28,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.40 | bwd_microstep: 5694.05 | bwd_inner_microstep: 5648.77 | bwd_allreduce_microstep: 45.23 | step_microstep: 18.77
[2025-04-25 22:02:28,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.40 | bwd: 5694.06 | bwd_inner: 5648.76 | bwd_allreduce: 45.25 | step: 18.77

 14%|█▍        | 5826/41250 [14:04:53<85:09:22,  8.65s/it]
                                                          
{'loss': 0.0996, 'grad_norm': 1.2529923915863037, 'learning_rate': 3.871629713402496e-05, 'epoch': 1.41}

 14%|█▍        | 5826/41250 [14:04:53<85:09:22,  8.65s/it][2025-04-25 22:02:37,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 22:02:37,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.05 | bwd_microstep: 5736.94 | bwd_inner_microstep: 5709.37 | bwd_allreduce_microstep: 27.51 | step_microstep: 18.92
[2025-04-25 22:02:37,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.05 | bwd: 5736.96 | bwd_inner: 5709.37 | bwd_allreduce: 27.54 | step: 18.92

 14%|█▍        | 5827/41250 [14:05:02<85:13:06,  8.66s/it]
                                                          
{'loss': 0.0453, 'grad_norm': 0.562140166759491, 'learning_rate': 3.871574354937861e-05, 'epoch': 1.41}

 14%|█▍        | 5827/41250 [14:05:02<85:13:06,  8.66s/it][2025-04-25 22:02:45,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-25 22:02:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.12 | bwd_microstep: 5685.21 | bwd_inner_microstep: 5655.63 | bwd_allreduce_microstep: 29.53 | step_microstep: 19.16
[2025-04-25 22:02:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.12 | bwd: 5685.22 | bwd_inner: 5655.63 | bwd_allreduce: 29.55 | step: 19.16

 14%|█▍        | 5828/41250 [14:05:11<85:01:27,  8.64s/it]
                                                          
{'loss': 0.0434, 'grad_norm': 0.6718298196792603, 'learning_rate': 3.871518984935339e-05, 'epoch': 1.41}

 14%|█▍        | 5828/41250 [14:05:11<85:01:27,  8.64s/it][2025-04-25 22:02:54,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:02:54,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.23 | bwd_microstep: 5725.70 | bwd_inner_microstep: 5680.36 | bwd_allreduce_microstep: 45.29 | step_microstep: 18.38
[2025-04-25 22:02:54,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.23 | bwd: 5725.71 | bwd_inner: 5680.36 | bwd_allreduce: 45.31 | step: 18.39

 14%|█▍        | 5829/41250 [14:05:19<85:03:03,  8.64s/it]
                                                          
{'loss': 0.1285, 'grad_norm': 0.8264704942703247, 'learning_rate': 3.87146360339527e-05, 'epoch': 1.41}

 14%|█▍        | 5829/41250 [14:05:19<85:03:03,  8.64s/it][2025-04-25 22:03:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:03:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.19 | bwd_microstep: 5738.35 | bwd_inner_microstep: 5658.85 | bwd_allreduce_microstep: 79.45 | step_microstep: 18.34
[2025-04-25 22:03:02,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.19 | bwd: 5738.36 | bwd_inner: 5658.85 | bwd_allreduce: 79.47 | step: 18.34

 14%|█▍        | 5830/41250 [14:05:28<85:03:36,  8.65s/it]
                                                          
{'loss': 0.016, 'grad_norm': 0.2515619397163391, 'learning_rate': 3.8714082103179956e-05, 'epoch': 1.41}

 14%|█▍        | 5830/41250 [14:05:28<85:03:36,  8.65s/it][2025-04-25 22:03:11,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:03:11,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.53 | bwd_microstep: 5693.55 | bwd_inner_microstep: 5644.68 | bwd_allreduce_microstep: 48.83 | step_microstep: 18.80
[2025-04-25 22:03:11,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.53 | bwd: 5693.57 | bwd_inner: 5644.68 | bwd_allreduce: 48.85 | step: 18.80

 14%|█▍        | 5831/41250 [14:05:36<84:55:17,  8.63s/it]
                                                          
{'loss': 0.0951, 'grad_norm': 1.327409267425537, 'learning_rate': 3.871352805703859e-05, 'epoch': 1.41}

 14%|█▍        | 5831/41250 [14:05:36<84:55:17,  8.63s/it][2025-04-25 22:03:20,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 22:03:20,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.60 | bwd_microstep: 5728.83 | bwd_inner_microstep: 5686.99 | bwd_allreduce_microstep: 41.80 | step_microstep: 18.78
[2025-04-25 22:03:20,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.60 | bwd: 5728.85 | bwd_inner: 5686.99 | bwd_allreduce: 41.82 | step: 18.78

 14%|█▍        | 5832/41250 [14:05:45<84:59:52,  8.64s/it]
                                                          
{'loss': 0.064, 'grad_norm': 1.007927417755127, 'learning_rate': 3.8712973895531993e-05, 'epoch': 1.41}

 14%|█▍        | 5832/41250 [14:05:45<84:59:52,  8.64s/it][2025-04-25 22:03:28,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:03:28,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.41 | bwd_microstep: 5704.13 | bwd_inner_microstep: 5654.61 | bwd_allreduce_microstep: 49.46 | step_microstep: 18.60
[2025-04-25 22:03:28,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.41 | bwd: 5704.14 | bwd_inner: 5654.61 | bwd_allreduce: 49.49 | step: 18.60

 14%|█▍        | 5833/41250 [14:05:54<84:55:05,  8.63s/it]
                                                          
{'loss': 0.2102, 'grad_norm': 3.49479079246521, 'learning_rate': 3.87124196186636e-05, 'epoch': 1.41}

 14%|█▍        | 5833/41250 [14:05:54<84:55:05,  8.63s/it][2025-04-25 22:03:37,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:03:37,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5706.92 | bwd_inner_microstep: 5694.16 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.81
[2025-04-25 22:03:37,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5706.93 | bwd_inner: 5694.16 | bwd_allreduce: 12.73 | step: 18.81

 14%|█▍        | 5834/41250 [14:06:02<84:55:57,  8.63s/it]
                                                          
{'loss': 0.1024, 'grad_norm': 1.0134410858154297, 'learning_rate': 3.871186522643681e-05, 'epoch': 1.41}

 14%|█▍        | 5834/41250 [14:06:02<84:55:57,  8.63s/it][2025-04-25 22:03:46,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:03:46,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.36 | bwd_microstep: 5751.65 | bwd_inner_microstep: 5697.20 | bwd_allreduce_microstep: 54.40 | step_microstep: 18.57
[2025-04-25 22:03:46,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.36 | bwd: 5751.66 | bwd_inner: 5697.20 | bwd_allreduce: 54.42 | step: 18.58

 14%|█▍        | 5835/41250 [14:06:11<85:06:27,  8.65s/it]
                                                          
{'loss': 0.0532, 'grad_norm': 1.0244821310043335, 'learning_rate': 3.871131071885507e-05, 'epoch': 1.41}

 14%|█▍        | 5835/41250 [14:06:11<85:06:27,  8.65s/it][2025-04-25 22:03:54,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.06 | optimizer_step: 1.20
[2025-04-25 22:03:54,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5712.16 | bwd_inner_microstep: 5698.86 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.96
[2025-04-25 22:03:54,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5712.18 | bwd_inner: 5698.86 | bwd_allreduce: 13.27 | step: 19.97

 14%|█▍        | 5836/41250 [14:06:20<85:05:01,  8.65s/it]
                                                          
{'loss': 0.019, 'grad_norm': 0.19857332110404968, 'learning_rate': 3.871075609592177e-05, 'epoch': 1.41}

 14%|█▍        | 5836/41250 [14:06:20<85:05:01,  8.65s/it][2025-04-25 22:04:03,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:04:03,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5707.17 | bwd_inner_microstep: 5694.62 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.57
[2025-04-25 22:04:03,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5707.18 | bwd_inner: 5694.62 | bwd_allreduce: 12.53 | step: 18.57

 14%|█▍        | 5837/41250 [14:06:28<85:03:07,  8.65s/it]
                                                          
{'loss': 0.1136, 'grad_norm': 1.5644125938415527, 'learning_rate': 3.8710201357640326e-05, 'epoch': 1.42}

 14%|█▍        | 5837/41250 [14:06:28<85:03:07,  8.65s/it][2025-04-25 22:04:12,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:04:12,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.61 | bwd_microstep: 5759.62 | bwd_inner_microstep: 5693.56 | bwd_allreduce_microstep: 66.02 | step_microstep: 18.89
[2025-04-25 22:04:12,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.61 | bwd: 5759.64 | bwd_inner: 5693.56 | bwd_allreduce: 66.04 | step: 18.89

 14%|█▍        | 5838/41250 [14:06:37<85:09:48,  8.66s/it]
                                                          
{'loss': 0.0678, 'grad_norm': 1.9971327781677246, 'learning_rate': 3.870964650401418e-05, 'epoch': 1.42}

 14%|█▍        | 5838/41250 [14:06:37<85:09:48,  8.66s/it][2025-04-25 22:04:21,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:04:21,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.94 | bwd_microstep: 5882.63 | bwd_inner_microstep: 5869.82 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.70
[2025-04-25 22:04:21,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.94 | bwd: 5882.64 | bwd_inner: 5869.82 | bwd_allreduce: 12.78 | step: 18.71

 14%|█▍        | 5839/41250 [14:06:46<85:52:47,  8.73s/it]
                                                          
{'loss': 0.1048, 'grad_norm': 2.275702953338623, 'learning_rate': 3.870909153504675e-05, 'epoch': 1.42}

 14%|█▍        | 5839/41250 [14:06:46<85:52:47,  8.73s/it][2025-04-25 22:04:29,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.10 | optimizer_step: 0.95
[2025-04-25 22:04:29,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5730.38 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 39.44 | step_microstep: 18.91
[2025-04-25 22:04:29,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5730.39 | bwd_inner: 5690.89 | bwd_allreduce: 39.46 | step: 18.91

 14%|█▍        | 5840/41250 [14:06:55<85:41:23,  8.71s/it]
                                                          
{'loss': 0.1769, 'grad_norm': 2.6905784606933594, 'learning_rate': 3.870853645074144e-05, 'epoch': 1.42}

 14%|█▍        | 5840/41250 [14:06:55<85:41:23,  8.71s/it][2025-04-25 22:04:38,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:04:38,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5693.78 | bwd_inner_microstep: 5660.49 | bwd_allreduce_microstep: 33.25 | step_microstep: 18.70
[2025-04-25 22:04:38,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5693.80 | bwd_inner: 5660.49 | bwd_allreduce: 33.26 | step: 18.71

 14%|█▍        | 5841/41250 [14:07:03<85:22:12,  8.68s/it]
                                                          
{'loss': 0.0705, 'grad_norm': 1.7422692775726318, 'learning_rate': 3.8707981251101695e-05, 'epoch': 1.42}

 14%|█▍        | 5841/41250 [14:07:03<85:22:12,  8.68s/it][2025-04-25 22:04:46,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.08 | optimizer_step: 1.22
[2025-04-25 22:04:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5697.22 | bwd_inner_microstep: 5658.31 | bwd_allreduce_microstep: 38.86 | step_microstep: 19.59
[2025-04-25 22:04:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5697.23 | bwd_inner: 5658.31 | bwd_allreduce: 38.89 | step: 19.59

 14%|█▍        | 5842/41250 [14:07:12<85:10:10,  8.66s/it]
                                                          
{'loss': 0.3238, 'grad_norm': 3.001801013946533, 'learning_rate': 3.8707425936130916e-05, 'epoch': 1.42}

 14%|█▍        | 5842/41250 [14:07:12<85:10:10,  8.66s/it][2025-04-25 22:04:55,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-25 22:04:55,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.32 | bwd_microstep: 5754.35 | bwd_inner_microstep: 5695.87 | bwd_allreduce_microstep: 58.43 | step_microstep: 18.93
[2025-04-25 22:04:55,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.32 | bwd: 5754.36 | bwd_inner: 5695.87 | bwd_allreduce: 58.45 | step: 18.93

 14%|█▍        | 5843/41250 [14:07:20<85:15:54,  8.67s/it]
                                                          
{'loss': 0.1998, 'grad_norm': 2.065520763397217, 'learning_rate': 3.8706870505832536e-05, 'epoch': 1.42}

 14%|█▍        | 5843/41250 [14:07:20<85:15:54,  8.67s/it][2025-04-25 22:05:04,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 22:05:04,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5756.01 | bwd_inner_microstep: 5700.97 | bwd_allreduce_microstep: 54.99 | step_microstep: 19.15
[2025-04-25 22:05:04,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5756.03 | bwd_inner: 5700.97 | bwd_allreduce: 55.01 | step: 19.15

 14%|█▍        | 5844/41250 [14:07:29<85:19:44,  8.68s/it]
                                                          
{'loss': 0.1756, 'grad_norm': 1.3237673044204712, 'learning_rate': 3.8706314960209984e-05, 'epoch': 1.42}

 14%|█▍        | 5844/41250 [14:07:29<85:19:44,  8.68s/it][2025-04-25 22:05:13,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:05:13,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.93 | bwd_microstep: 5773.90 | bwd_inner_microstep: 5657.10 | bwd_allreduce_microstep: 116.76 | step_microstep: 18.62
[2025-04-25 22:05:13,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.93 | bwd: 5773.91 | bwd_inner: 5657.10 | bwd_allreduce: 116.78 | step: 18.63

 14%|█▍        | 5845/41250 [14:07:38<85:21:00,  8.68s/it]
                                                          
{'loss': 0.131, 'grad_norm': 3.102499008178711, 'learning_rate': 3.8705759299266673e-05, 'epoch': 1.42}

 14%|█▍        | 5845/41250 [14:07:38<85:21:00,  8.68s/it][2025-04-25 22:05:21,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:05:21,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.57 | bwd_microstep: 5756.75 | bwd_inner_microstep: 5663.91 | bwd_allreduce_microstep: 92.80 | step_microstep: 18.62
[2025-04-25 22:05:21,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.57 | bwd: 5756.77 | bwd_inner: 5663.91 | bwd_allreduce: 92.82 | step: 18.62

 14%|█▍        | 5846/41250 [14:07:47<85:20:31,  8.68s/it]
                                                          
{'loss': 0.1817, 'grad_norm': 1.7901272773742676, 'learning_rate': 3.870520352300604e-05, 'epoch': 1.42}

 14%|█▍        | 5846/41250 [14:07:47<85:20:31,  8.68s/it][2025-04-25 22:05:30,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-25 22:05:30,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.22 | bwd_microstep: 5783.67 | bwd_inner_microstep: 5668.51 | bwd_allreduce_microstep: 115.11 | step_microstep: 18.90
[2025-04-25 22:05:30,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.22 | bwd: 5783.68 | bwd_inner: 5668.51 | bwd_allreduce: 115.13 | step: 18.91

 14%|█▍        | 5847/41250 [14:07:55<85:24:49,  8.69s/it]
                                                          
{'loss': 0.0375, 'grad_norm': 0.5672743320465088, 'learning_rate': 3.870464763143151e-05, 'epoch': 1.42}

 14%|█▍        | 5847/41250 [14:07:55<85:24:49,  8.69s/it][2025-04-25 22:05:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.62 | optimizer_step: 1.01
[2025-04-25 22:05:39,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.44 | bwd_microstep: 5893.41 | bwd_inner_microstep: 5664.21 | bwd_allreduce_microstep: 229.14 | step_microstep: 20.84
[2025-04-25 22:05:39,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.44 | bwd: 5893.43 | bwd_inner: 5664.21 | bwd_allreduce: 229.17 | step: 20.84

 14%|█▍        | 5848/41250 [14:08:04<85:48:20,  8.73s/it]
                                                          
{'loss': 0.1334, 'grad_norm': 1.2383677959442139, 'learning_rate': 3.87040916245465e-05, 'epoch': 1.42}

 14%|█▍        | 5848/41250 [14:08:04<85:48:20,  8.73s/it][2025-04-25 22:05:48,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 22:05:48,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 6069.24 | bwd_inner_microstep: 5655.86 | bwd_allreduce_microstep: 413.33 | step_microstep: 19.09
[2025-04-25 22:05:48,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 6069.25 | bwd_inner: 5655.86 | bwd_allreduce: 413.35 | step: 19.09

 14%|█▍        | 5849/41250 [14:08:13<86:33:37,  8.80s/it]
                                                          
{'loss': 0.109, 'grad_norm': 0.9964556694030762, 'learning_rate': 3.870353550235445e-05, 'epoch': 1.42}

 14%|█▍        | 5849/41250 [14:08:13<86:33:37,  8.80s/it][2025-04-25 22:05:56,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:05:56,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.02 | bwd_microstep: 5785.62 | bwd_inner_microstep: 5664.72 | bwd_allreduce_microstep: 120.86 | step_microstep: 19.25
[2025-04-25 22:05:56,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.02 | bwd: 5785.64 | bwd_inner: 5664.72 | bwd_allreduce: 120.88 | step: 19.25

 14%|█▍        | 5850/41250 [14:08:22<86:15:19,  8.77s/it]
                                                          
{'loss': 0.3356, 'grad_norm': 2.9847264289855957, 'learning_rate': 3.8702979264858774e-05, 'epoch': 1.42}

 14%|█▍        | 5850/41250 [14:08:22<86:15:19,  8.77s/it][2025-04-25 22:06:05,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.30 | optimizer_step: 1.06
[2025-04-25 22:06:05,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.08 | bwd_microstep: 5767.91 | bwd_inner_microstep: 5716.84 | bwd_allreduce_microstep: 51.01 | step_microstep: 19.88
[2025-04-25 22:06:05,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.08 | bwd: 5767.92 | bwd_inner: 5716.84 | bwd_allreduce: 51.03 | step: 19.89

 14%|█▍        | 5851/41250 [14:08:30<86:04:11,  8.75s/it]
                                                          
{'loss': 0.0658, 'grad_norm': 1.6175215244293213, 'learning_rate': 3.870242291206291e-05, 'epoch': 1.42}

 14%|█▍        | 5851/41250 [14:08:30<86:04:11,  8.75s/it][2025-04-25 22:06:14,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:06:14,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5776.29 | bwd_inner_microstep: 5667.79 | bwd_allreduce_microstep: 108.46 | step_microstep: 18.63
[2025-04-25 22:06:14,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5776.31 | bwd_inner: 5667.79 | bwd_allreduce: 108.47 | step: 18.64

 14%|█▍        | 5852/41250 [14:08:39<85:53:17,  8.73s/it]
                                                          
{'loss': 0.083, 'grad_norm': 1.007958173751831, 'learning_rate': 3.870186644397029e-05, 'epoch': 1.42}

 14%|█▍        | 5852/41250 [14:08:39<85:53:17,  8.73s/it][2025-04-25 22:06:22,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:06:22,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.02 | bwd_microstep: 5702.54 | bwd_inner_microstep: 5669.45 | bwd_allreduce_microstep: 33.04 | step_microstep: 18.88
[2025-04-25 22:06:22,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.02 | bwd: 5702.55 | bwd_inner: 5669.45 | bwd_allreduce: 33.06 | step: 18.89

 14%|█▍        | 5853/41250 [14:08:48<85:33:39,  8.70s/it]
                                                          
{'loss': 0.0571, 'grad_norm': 1.5164377689361572, 'learning_rate': 3.870130986058434e-05, 'epoch': 1.42}

 14%|█▍        | 5853/41250 [14:08:48<85:33:39,  8.70s/it][2025-04-25 22:06:31,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:06:31,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.04 | bwd_microstep: 5706.90 | bwd_inner_microstep: 5673.94 | bwd_allreduce_microstep: 32.92 | step_microstep: 18.48
[2025-04-25 22:06:31,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.04 | bwd: 5706.92 | bwd_inner: 5673.94 | bwd_allreduce: 32.94 | step: 18.49

 14%|█▍        | 5854/41250 [14:08:56<85:19:22,  8.68s/it]
                                                          
{'loss': 0.095, 'grad_norm': 2.077878475189209, 'learning_rate': 3.8700753161908497e-05, 'epoch': 1.42}

 14%|█▍        | 5854/41250 [14:08:56<85:19:22,  8.68s/it][2025-04-25 22:06:40,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:06:40,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.51 | bwd_microstep: 5771.99 | bwd_inner_microstep: 5702.80 | bwd_allreduce_microstep: 69.13 | step_microstep: 18.99
[2025-04-25 22:06:40,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.51 | bwd: 5772.00 | bwd_inner: 5702.80 | bwd_allreduce: 69.15 | step: 18.99

 14%|█▍        | 5855/41250 [14:09:05<85:26:33,  8.69s/it]
                                                          
{'loss': 0.0831, 'grad_norm': 1.235107660293579, 'learning_rate': 3.870019634794619e-05, 'epoch': 1.42}

 14%|█▍        | 5855/41250 [14:09:05<85:26:33,  8.69s/it][2025-04-25 22:06:48,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-25 22:06:48,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.17 | bwd_microstep: 5731.37 | bwd_inner_microstep: 5718.61 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.17
[2025-04-25 22:06:48,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.17 | bwd: 5731.38 | bwd_inner: 5718.61 | bwd_allreduce: 12.73 | step: 19.17

 14%|█▍        | 5856/41250 [14:09:14<85:22:31,  8.68s/it]
                                                          
{'loss': 0.3052, 'grad_norm': 2.3226277828216553, 'learning_rate': 3.8699639418700845e-05, 'epoch': 1.42}

 14%|█▍        | 5856/41250 [14:09:14<85:22:31,  8.68s/it][2025-04-25 22:06:57,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:06:57,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.09 | bwd_microstep: 5732.11 | bwd_inner_microstep: 5719.45 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.84
[2025-04-25 22:06:57,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.09 | bwd: 5732.12 | bwd_inner: 5719.45 | bwd_allreduce: 12.62 | step: 18.84

 14%|█▍        | 5857/41250 [14:09:22<85:21:33,  8.68s/it]
                                                          
{'loss': 0.2001, 'grad_norm': 2.172253370285034, 'learning_rate': 3.8699082374175904e-05, 'epoch': 1.42}

 14%|█▍        | 5857/41250 [14:09:22<85:21:33,  8.68s/it][2025-04-25 22:07:06,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:07:06,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.76 | bwd_microstep: 5704.58 | bwd_inner_microstep: 5660.84 | bwd_allreduce_microstep: 43.69 | step_microstep: 18.49
[2025-04-25 22:07:06,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.76 | bwd: 5704.59 | bwd_inner: 5660.84 | bwd_allreduce: 43.71 | step: 18.49

 14%|█▍        | 5858/41250 [14:09:31<85:11:10,  8.66s/it]
                                                          
{'loss': 0.1731, 'grad_norm': 1.618879795074463, 'learning_rate': 3.86985252143748e-05, 'epoch': 1.42}

 14%|█▍        | 5858/41250 [14:09:31<85:11:10,  8.66s/it][2025-04-25 22:07:14,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 22:07:14,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.25 | bwd_microstep: 5772.64 | bwd_inner_microstep: 5675.61 | bwd_allreduce_microstep: 96.99 | step_microstep: 18.98
[2025-04-25 22:07:14,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.25 | bwd: 5772.66 | bwd_inner: 5675.61 | bwd_allreduce: 97.00 | step: 18.98

 14%|█▍        | 5859/41250 [14:09:40<85:15:15,  8.67s/it]
                                                          
{'loss': 0.0044, 'grad_norm': 0.053636614233255386, 'learning_rate': 3.869796793930096e-05, 'epoch': 1.42}

 14%|█▍        | 5859/41250 [14:09:40<85:15:15,  8.67s/it][2025-04-25 22:07:23,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 22:07:23,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.68 | bwd_microstep: 5716.45 | bwd_inner_microstep: 5665.21 | bwd_allreduce_microstep: 51.19 | step_microstep: 18.91
[2025-04-25 22:07:23,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.68 | bwd: 5716.47 | bwd_inner: 5665.21 | bwd_allreduce: 51.21 | step: 18.92

 14%|█▍        | 5860/41250 [14:09:48<85:08:14,  8.66s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 1.1256051063537598, 'learning_rate': 3.869741054895784e-05, 'epoch': 1.42}

 14%|█▍        | 5860/41250 [14:09:48<85:08:14,  8.66s/it][2025-04-25 22:07:32,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:07:32,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.33 | bwd_microstep: 5729.90 | bwd_inner_microstep: 5716.97 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.05
[2025-04-25 22:07:32,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.33 | bwd: 5729.91 | bwd_inner: 5716.97 | bwd_allreduce: 12.90 | step: 19.06

 14%|█▍        | 5861/41250 [14:09:57<85:13:07,  8.67s/it]
                                                          
{'loss': 0.1415, 'grad_norm': 3.0619754791259766, 'learning_rate': 3.8696853043348846e-05, 'epoch': 1.42}

 14%|█▍        | 5861/41250 [14:09:57<85:13:07,  8.67s/it][2025-04-25 22:07:40,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 22:07:40,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5710.62 | bwd_inner_microstep: 5697.69 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.39
[2025-04-25 22:07:40,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5710.64 | bwd_inner: 5697.69 | bwd_allreduce: 12.90 | step: 19.40

 14%|█▍        | 5862/41250 [14:10:06<85:08:47,  8.66s/it]
                                                          
{'loss': 0.1131, 'grad_norm': 2.320702075958252, 'learning_rate': 3.869629542247744e-05, 'epoch': 1.42}

 14%|█▍        | 5862/41250 [14:10:06<85:08:47,  8.66s/it][2025-04-25 22:07:49,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 22:07:49,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.04 | bwd_microstep: 5715.86 | bwd_inner_microstep: 5703.27 | bwd_allreduce_microstep: 12.54 | step_microstep: 19.00
[2025-04-25 22:07:49,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.04 | bwd: 5715.87 | bwd_inner: 5703.27 | bwd_allreduce: 12.56 | step: 19.00

 14%|█▍        | 5863/41250 [14:10:14<85:05:59,  8.66s/it]
                                                          
{'loss': 0.276, 'grad_norm': 2.7636325359344482, 'learning_rate': 3.8695737686347034e-05, 'epoch': 1.42}

 14%|█▍        | 5863/41250 [14:10:14<85:05:59,  8.66s/it][2025-04-25 22:07:58,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:07:58,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.51 | bwd_microstep: 5821.24 | bwd_inner_microstep: 5649.68 | bwd_allreduce_microstep: 171.52 | step_microstep: 18.68
[2025-04-25 22:07:58,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.51 | bwd: 5821.26 | bwd_inner: 5649.68 | bwd_allreduce: 171.54 | step: 18.69

 14%|█▍        | 5864/41250 [14:10:23<85:19:17,  8.68s/it]
                                                          
{'loss': 0.0798, 'grad_norm': 1.2966474294662476, 'learning_rate': 3.86951798349611e-05, 'epoch': 1.42}

 14%|█▍        | 5864/41250 [14:10:23<85:19:17,  8.68s/it][2025-04-25 22:08:06,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.15 | optimizer_step: 0.90
[2025-04-25 22:08:06,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5711.39 | bwd_inner_microstep: 5663.51 | bwd_allreduce_microstep: 47.83 | step_microstep: 18.98
[2025-04-25 22:08:06,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5711.40 | bwd_inner: 5663.51 | bwd_allreduce: 47.85 | step: 18.98

 14%|█▍        | 5865/41250 [14:10:32<85:09:01,  8.66s/it]
                                                          
{'loss': 0.0398, 'grad_norm': 0.5650381445884705, 'learning_rate': 3.869462186832305e-05, 'epoch': 1.42}

 14%|█▍        | 5865/41250 [14:10:32<85:09:01,  8.66s/it][2025-04-25 22:08:15,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:08:15,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5724.33 | bwd_inner_microstep: 5711.54 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.83
[2025-04-25 22:08:15,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5724.35 | bwd_inner: 5711.54 | bwd_allreduce: 12.77 | step: 18.83

 14%|█▍        | 5866/41250 [14:10:40<85:08:19,  8.66s/it]
                                                          
{'loss': 0.2435, 'grad_norm': 1.778333067893982, 'learning_rate': 3.869406378643634e-05, 'epoch': 1.42}

 14%|█▍        | 5866/41250 [14:10:40<85:08:19,  8.66s/it][2025-04-25 22:08:24,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:08:24,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.77 | bwd_microstep: 5779.89 | bwd_inner_microstep: 5650.69 | bwd_allreduce_microstep: 129.16 | step_microstep: 18.76
[2025-04-25 22:08:24,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.77 | bwd: 5779.91 | bwd_inner: 5650.69 | bwd_allreduce: 129.18 | step: 18.76

 14%|█▍        | 5867/41250 [14:10:49<85:13:08,  8.67s/it]
                                                          
{'loss': 0.0373, 'grad_norm': 0.8243726491928101, 'learning_rate': 3.86935055893044e-05, 'epoch': 1.42}

 14%|█▍        | 5867/41250 [14:10:49<85:13:08,  8.67s/it][2025-04-25 22:08:33,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-25 22:08:33,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.40 | bwd_microstep: 6025.79 | bwd_inner_microstep: 5698.72 | bwd_allreduce_microstep: 327.02 | step_microstep: 19.11
[2025-04-25 22:08:33,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.40 | bwd: 6025.80 | bwd_inner: 5698.72 | bwd_allreduce: 327.04 | step: 19.11

 14%|█▍        | 5868/41250 [14:10:58<86:03:30,  8.76s/it]
                                                          
{'loss': 0.0899, 'grad_norm': 1.4782840013504028, 'learning_rate': 3.869294727693067e-05, 'epoch': 1.42}

 14%|█▍        | 5868/41250 [14:10:58<86:03:30,  8.76s/it][2025-04-25 22:08:41,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 22:08:41,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.04 | bwd_microstep: 5692.31 | bwd_inner_microstep: 5653.29 | bwd_allreduce_microstep: 38.97 | step_microstep: 18.83
[2025-04-25 22:08:41,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.04 | bwd: 5692.33 | bwd_inner: 5653.29 | bwd_allreduce: 38.99 | step: 18.83

 14%|█▍        | 5869/41250 [14:11:07<85:37:00,  8.71s/it]
                                                          
{'loss': 0.0486, 'grad_norm': 2.0385007858276367, 'learning_rate': 3.8692388849318604e-05, 'epoch': 1.42}

 14%|█▍        | 5869/41250 [14:11:07<85:37:00,  8.71s/it][2025-04-25 22:08:50,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 22:08:50,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.80 | bwd_microstep: 5783.85 | bwd_inner_microstep: 5652.90 | bwd_allreduce_microstep: 130.90 | step_microstep: 19.10
[2025-04-25 22:08:50,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.80 | bwd: 5783.87 | bwd_inner: 5652.90 | bwd_allreduce: 130.92 | step: 19.11

 14%|█▍        | 5870/41250 [14:11:15<85:33:46,  8.71s/it]
                                                          
{'loss': 0.0883, 'grad_norm': 3.974820613861084, 'learning_rate': 3.869183030647164e-05, 'epoch': 1.42}

 14%|█▍        | 5870/41250 [14:11:15<85:33:46,  8.71s/it][2025-04-25 22:08:59,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 22:08:59,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.66 | bwd_microstep: 5688.14 | bwd_inner_microstep: 5650.88 | bwd_allreduce_microstep: 37.21 | step_microstep: 18.86
[2025-04-25 22:08:59,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.66 | bwd: 5688.15 | bwd_inner: 5650.88 | bwd_allreduce: 37.23 | step: 18.87

 14%|█▍        | 5871/41250 [14:11:24<85:16:07,  8.68s/it]
                                                          
{'loss': 0.3802, 'grad_norm': 4.343644142150879, 'learning_rate': 3.869127164839321e-05, 'epoch': 1.42}

 14%|█▍        | 5871/41250 [14:11:24<85:16:07,  8.68s/it][2025-04-25 22:09:07,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 22:09:07,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.76 | bwd_microstep: 5899.31 | bwd_inner_microstep: 5646.80 | bwd_allreduce_microstep: 252.46 | step_microstep: 19.00
[2025-04-25 22:09:07,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.76 | bwd: 5899.33 | bwd_inner: 5646.80 | bwd_allreduce: 252.48 | step: 19.00

 14%|█▍        | 5872/41250 [14:11:33<85:38:18,  8.71s/it]
                                                          
{'loss': 0.1942, 'grad_norm': 2.052727222442627, 'learning_rate': 3.869071287508678e-05, 'epoch': 1.42}

 14%|█▍        | 5872/41250 [14:11:33<85:38:18,  8.71s/it][2025-04-25 22:09:16,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 22:09:16,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5704.12 | bwd_inner_microstep: 5690.97 | bwd_allreduce_microstep: 13.10 | step_microstep: 19.45
[2025-04-25 22:09:16,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5704.14 | bwd_inner: 5690.97 | bwd_allreduce: 13.12 | step: 19.46

 14%|█▍        | 5873/41250 [14:11:41<85:25:15,  8.69s/it]
                                                          
{'loss': 0.1636, 'grad_norm': 1.5939736366271973, 'learning_rate': 3.869015398655576e-05, 'epoch': 1.42}

 14%|█▍        | 5873/41250 [14:11:41<85:25:15,  8.69s/it][2025-04-25 22:09:25,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 22:09:25,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.96 | bwd_microstep: 5744.09 | bwd_inner_microstep: 5647.75 | bwd_allreduce_microstep: 96.28 | step_microstep: 19.12
[2025-04-25 22:09:25,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.96 | bwd: 5744.10 | bwd_inner: 5647.75 | bwd_allreduce: 96.30 | step: 19.13

 14%|█▍        | 5874/41250 [14:11:50<85:18:18,  8.68s/it]
                                                          
{'loss': 0.2495, 'grad_norm': 2.8331356048583984, 'learning_rate': 3.8689594982803635e-05, 'epoch': 1.42}

 14%|█▍        | 5874/41250 [14:11:50<85:18:18,  8.68s/it][2025-04-25 22:09:33,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 22:09:33,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5748.82 | bwd_inner_microstep: 5651.86 | bwd_allreduce_microstep: 96.91 | step_microstep: 18.71
[2025-04-25 22:09:33,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5748.83 | bwd_inner: 5651.86 | bwd_allreduce: 96.93 | step: 18.72

 14%|█▍        | 5875/41250 [14:11:59<85:14:23,  8.67s/it]
                                                          
{'loss': 0.1116, 'grad_norm': 5.107389450073242, 'learning_rate': 3.868903586383382e-05, 'epoch': 1.42}

 14%|█▍        | 5875/41250 [14:11:59<85:14:23,  8.67s/it][2025-04-25 22:09:42,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:09:42,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5721.96 | bwd_inner_microstep: 5709.21 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.85
[2025-04-25 22:09:42,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5721.98 | bwd_inner: 5709.21 | bwd_allreduce: 12.72 | step: 18.85

 14%|█▍        | 5876/41250 [14:12:07<85:11:37,  8.67s/it]
                                                          
{'loss': 0.0433, 'grad_norm': 0.6804834604263306, 'learning_rate': 3.868847662964978e-05, 'epoch': 1.42}

 14%|█▍        | 5876/41250 [14:12:07<85:11:37,  8.67s/it][2025-04-25 22:09:51,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 22:09:51,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.72 | bwd_microstep: 5736.57 | bwd_inner_microstep: 5675.31 | bwd_allreduce_microstep: 61.21 | step_microstep: 19.44
[2025-04-25 22:09:51,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.72 | bwd: 5736.58 | bwd_inner: 5675.31 | bwd_allreduce: 61.23 | step: 19.44

 14%|█▍        | 5877/41250 [14:12:16<85:10:55,  8.67s/it]
                                                          
{'loss': 0.1588, 'grad_norm': 2.6494460105895996, 'learning_rate': 3.8687917280254965e-05, 'epoch': 1.42}

 14%|█▍        | 5877/41250 [14:12:16<85:10:55,  8.67s/it][2025-04-25 22:09:59,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.99
[2025-04-25 22:09:59,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.79 | bwd_microstep: 5764.22 | bwd_inner_microstep: 5751.65 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.66
[2025-04-25 22:09:59,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.79 | bwd: 5764.24 | bwd_inner: 5751.65 | bwd_allreduce: 12.54 | step: 18.66

 14%|█▍        | 5878/41250 [14:12:25<85:22:47,  8.69s/it]
                                                          
{'loss': 0.171, 'grad_norm': 2.324094533920288, 'learning_rate': 3.868735781565281e-05, 'epoch': 1.42}

 14%|█▍        | 5878/41250 [14:12:25<85:22:47,  8.69s/it][2025-04-25 22:10:08,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:10:08,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.98 | bwd_microstep: 5714.29 | bwd_inner_microstep: 5692.34 | bwd_allreduce_microstep: 21.90 | step_microstep: 19.04
[2025-04-25 22:10:08,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.98 | bwd: 5714.30 | bwd_inner: 5692.34 | bwd_allreduce: 21.91 | step: 19.04

 14%|█▍        | 5879/41250 [14:12:33<85:14:54,  8.68s/it]
                                                          
{'loss': 0.1494, 'grad_norm': 1.583552598953247, 'learning_rate': 3.868679823584677e-05, 'epoch': 1.43}

 14%|█▍        | 5879/41250 [14:12:33<85:14:54,  8.68s/it][2025-04-25 22:10:17,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 22:10:17,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.94 | bwd_microstep: 5705.69 | bwd_inner_microstep: 5693.23 | bwd_allreduce_microstep: 12.42 | step_microstep: 18.34
[2025-04-25 22:10:17,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.94 | bwd: 5705.71 | bwd_inner: 5693.23 | bwd_allreduce: 12.44 | step: 18.34

 14%|█▍        | 5880/41250 [14:12:42<85:08:01,  8.67s/it]
                                                          
{'loss': 0.0275, 'grad_norm': 0.6912130117416382, 'learning_rate': 3.868623854084029e-05, 'epoch': 1.43}

 14%|█▍        | 5880/41250 [14:12:42<85:08:01,  8.67s/it][2025-04-25 22:10:25,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 22:10:25,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.11 | bwd_microstep: 5766.05 | bwd_inner_microstep: 5696.47 | bwd_allreduce_microstep: 69.53 | step_microstep: 18.69
[2025-04-25 22:10:25,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.11 | bwd: 5766.07 | bwd_inner: 5696.47 | bwd_allreduce: 69.55 | step: 18.69

 14%|█▍        | 5881/41250 [14:12:51<85:13:04,  8.67s/it]
                                                          
{'loss': 0.0141, 'grad_norm': 0.25424790382385254, 'learning_rate': 3.868567873063683e-05, 'epoch': 1.43}

 14%|█▍        | 5881/41250 [14:12:51<85:13:04,  8.67s/it][2025-04-25 22:10:34,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:10:34,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5714.25 | bwd_inner_microstep: 5701.36 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.10
[2025-04-25 22:10:34,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5714.26 | bwd_inner: 5701.36 | bwd_allreduce: 12.86 | step: 19.10

 14%|█▍        | 5882/41250 [14:12:59<85:08:17,  8.67s/it]
                                                          
{'loss': 0.1724, 'grad_norm': 2.563837766647339, 'learning_rate': 3.8685118805239836e-05, 'epoch': 1.43}

 14%|█▍        | 5882/41250 [14:12:59<85:08:17,  8.67s/it][2025-04-25 22:10:43,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-25 22:10:43,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5694.00 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 42.13 | step_microstep: 19.68
[2025-04-25 22:10:43,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5694.01 | bwd_inner: 5651.81 | bwd_allreduce: 42.16 | step: 19.68

 14%|█▍        | 5883/41250 [14:13:08<84:57:14,  8.65s/it]
                                                          
{'loss': 0.0701, 'grad_norm': 1.7607955932617188, 'learning_rate': 3.8684558764652756e-05, 'epoch': 1.43}

 14%|█▍        | 5883/41250 [14:13:08<84:57:14,  8.65s/it][2025-04-25 22:10:51,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 1.02
[2025-04-25 22:10:51,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.04 | bwd_microstep: 5788.62 | bwd_inner_microstep: 5640.07 | bwd_allreduce_microstep: 148.50 | step_microstep: 19.31
[2025-04-25 22:10:51,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.04 | bwd: 5788.64 | bwd_inner: 5640.07 | bwd_allreduce: 148.52 | step: 19.31

 14%|█▍        | 5884/41250 [14:13:17<85:05:16,  8.66s/it]
                                                          
{'loss': 0.2652, 'grad_norm': 2.559706211090088, 'learning_rate': 3.868399860887905e-05, 'epoch': 1.43}

 14%|█▍        | 5884/41250 [14:13:17<85:05:16,  8.66s/it][2025-04-25 22:11:00,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-25 22:11:00,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5697.35 | bwd_inner_microstep: 5654.34 | bwd_allreduce_microstep: 42.95 | step_microstep: 19.07
[2025-04-25 22:11:00,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5697.37 | bwd_inner: 5654.34 | bwd_allreduce: 42.98 | step: 19.06

 14%|█▍        | 5885/41250 [14:13:25<84:55:14,  8.64s/it]
                                                          
{'loss': 0.249, 'grad_norm': 2.973750591278076, 'learning_rate': 3.868343833792216e-05, 'epoch': 1.43}

 14%|█▍        | 5885/41250 [14:13:25<84:55:14,  8.64s/it][2025-04-25 22:11:09,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:11:09,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.57 | bwd_microstep: 5694.18 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 42.31 | step_microstep: 18.85
[2025-04-25 22:11:09,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.57 | bwd: 5694.19 | bwd_inner: 5651.81 | bwd_allreduce: 42.33 | step: 18.85

 14%|█▍        | 5886/41250 [14:13:34<84:47:38,  8.63s/it]
                                                          
{'loss': 0.1041, 'grad_norm': 1.9636603593826294, 'learning_rate': 3.868287795178556e-05, 'epoch': 1.43}

 14%|█▍        | 5886/41250 [14:13:34<84:47:38,  8.63s/it][2025-04-25 22:11:17,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 22:11:17,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.28 | bwd_microstep: 5702.17 | bwd_inner_microstep: 5689.16 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.68
[2025-04-25 22:11:17,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.28 | bwd: 5702.18 | bwd_inner: 5689.16 | bwd_allreduce: 12.98 | step: 18.69

 14%|█▍        | 5887/41250 [14:13:43<84:47:53,  8.63s/it]
                                                          
{'loss': 0.0369, 'grad_norm': 0.760072648525238, 'learning_rate': 3.868231745047268e-05, 'epoch': 1.43}

 14%|█▍        | 5887/41250 [14:13:43<84:47:53,  8.63s/it][2025-04-25 22:11:26,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 22:11:26,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.79 | bwd_microstep: 5786.29 | bwd_inner_microstep: 5773.32 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.23
[2025-04-25 22:11:26,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.79 | bwd: 5786.31 | bwd_inner: 5773.32 | bwd_allreduce: 12.95 | step: 19.23

 14%|█▍        | 5888/41250 [14:13:51<85:09:08,  8.67s/it]
                                                          
{'loss': 0.2406, 'grad_norm': 1.9756300449371338, 'learning_rate': 3.8681756833987e-05, 'epoch': 1.43}

 14%|█▍        | 5888/41250 [14:13:51<85:09:08,  8.67s/it][2025-04-25 22:11:35,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.18 | optimizer_step: 1.03
[2025-04-25 22:11:35,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5739.83 | bwd_inner_microstep: 5684.14 | bwd_allreduce_microstep: 55.64 | step_microstep: 19.61
[2025-04-25 22:11:35,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5739.84 | bwd_inner: 5684.14 | bwd_allreduce: 55.66 | step: 19.61

 14%|█▍        | 5889/41250 [14:14:00<85:09:30,  8.67s/it]
                                                          
{'loss': 0.1033, 'grad_norm': 1.810861587524414, 'learning_rate': 3.868119610233195e-05, 'epoch': 1.43}

 14%|█▍        | 5889/41250 [14:14:00<85:09:30,  8.67s/it][2025-04-25 22:11:43,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-25 22:11:43,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.40 | bwd_microstep: 5737.11 | bwd_inner_microstep: 5690.19 | bwd_allreduce_microstep: 46.87 | step_microstep: 18.68
[2025-04-25 22:11:43,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.40 | bwd: 5737.12 | bwd_inner: 5690.19 | bwd_allreduce: 46.89 | step: 18.68

 14%|█▍        | 5890/41250 [14:14:09<85:08:59,  8.67s/it]
                                                          
{'loss': 0.073, 'grad_norm': 3.261234760284424, 'learning_rate': 3.868063525551101e-05, 'epoch': 1.43}

 14%|█▍        | 5890/41250 [14:14:09<85:08:59,  8.67s/it][2025-04-25 22:11:52,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-25 22:11:52,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.55 | bwd_microstep: 5753.31 | bwd_inner_microstep: 5684.07 | bwd_allreduce_microstep: 69.18 | step_microstep: 18.98
[2025-04-25 22:11:52,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.55 | bwd: 5753.32 | bwd_inner: 5684.07 | bwd_allreduce: 69.20 | step: 18.99

 14%|█▍        | 5891/41250 [14:14:17<85:10:28,  8.67s/it]
                                                          
{'loss': 0.2021, 'grad_norm': 2.3671507835388184, 'learning_rate': 3.8680074293527624e-05, 'epoch': 1.43}

 14%|█▍        | 5891/41250 [14:14:17<85:10:28,  8.67s/it][2025-04-25 22:12:01,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:12:01,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.29 | bwd_microstep: 5787.60 | bwd_inner_microstep: 5774.48 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.75
[2025-04-25 22:12:01,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.29 | bwd: 5787.62 | bwd_inner: 5774.48 | bwd_allreduce: 13.09 | step: 18.76

 14%|█▍        | 5892/41250 [14:14:26<85:24:43,  8.70s/it]
                                                          
{'loss': 0.0534, 'grad_norm': 1.0790445804595947, 'learning_rate': 3.8679513216385256e-05, 'epoch': 1.43}

 14%|█▍        | 5892/41250 [14:14:26<85:24:43,  8.70s/it][2025-04-25 22:12:09,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-25 22:12:09,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5760.36 | bwd_inner_microstep: 5645.62 | bwd_allreduce_microstep: 114.68 | step_microstep: 19.30
[2025-04-25 22:12:09,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5760.37 | bwd_inner: 5645.62 | bwd_allreduce: 114.71 | step: 19.31

 14%|█▍        | 5893/41250 [14:14:35<85:20:17,  8.69s/it]
                                                          
{'loss': 0.1718, 'grad_norm': 3.6917927265167236, 'learning_rate': 3.867895202408736e-05, 'epoch': 1.43}

 14%|█▍        | 5893/41250 [14:14:35<85:20:17,  8.69s/it][2025-04-25 22:12:18,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:12:18,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.20 | bwd_microstep: 5749.98 | bwd_inner_microstep: 5657.77 | bwd_allreduce_microstep: 92.17 | step_microstep: 18.77
[2025-04-25 22:12:18,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.20 | bwd: 5750.00 | bwd_inner: 5657.77 | bwd_allreduce: 92.19 | step: 18.77

 14%|█▍        | 5894/41250 [14:14:43<85:15:15,  8.68s/it]
                                                          
{'loss': 0.102, 'grad_norm': 0.9546758532524109, 'learning_rate': 3.86783907166374e-05, 'epoch': 1.43}

 14%|█▍        | 5894/41250 [14:14:43<85:15:15,  8.68s/it][2025-04-25 22:12:27,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:12:27,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5702.18 | bwd_inner_microstep: 5653.57 | bwd_allreduce_microstep: 48.57 | step_microstep: 18.86
[2025-04-25 22:12:27,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5702.20 | bwd_inner: 5653.57 | bwd_allreduce: 48.59 | step: 18.86

 14%|█▍        | 5895/41250 [14:14:52<85:02:42,  8.66s/it]
                                                          
{'loss': 0.1558, 'grad_norm': 2.3560545444488525, 'learning_rate': 3.867782929403884e-05, 'epoch': 1.43}

 14%|█▍        | 5895/41250 [14:14:52<85:02:42,  8.66s/it][2025-04-25 22:12:35,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:12:35,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.00 | bwd_microstep: 5757.25 | bwd_inner_microstep: 5651.70 | bwd_allreduce_microstep: 105.52 | step_microstep: 18.81
[2025-04-25 22:12:35,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.00 | bwd: 5757.27 | bwd_inner: 5651.70 | bwd_allreduce: 105.53 | step: 18.82

 14%|█▍        | 5896/41250 [14:15:01<85:03:30,  8.66s/it]
                                                          
{'loss': 0.0996, 'grad_norm': 4.357958793640137, 'learning_rate': 3.867726775629513e-05, 'epoch': 1.43}

 14%|█▍        | 5896/41250 [14:15:01<85:03:30,  8.66s/it][2025-04-25 22:12:44,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:12:44,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.95 | bwd_microstep: 5792.94 | bwd_inner_microstep: 5644.69 | bwd_allreduce_microstep: 148.20 | step_microstep: 18.83
[2025-04-25 22:12:44,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.95 | bwd: 5792.95 | bwd_inner: 5644.69 | bwd_allreduce: 148.22 | step: 18.83

 14%|█▍        | 5897/41250 [14:15:09<85:10:10,  8.67s/it]
                                                          
{'loss': 0.252, 'grad_norm': 2.911107063293457, 'learning_rate': 3.867670610340974e-05, 'epoch': 1.43}

 14%|█▍        | 5897/41250 [14:15:09<85:10:10,  8.67s/it][2025-04-25 22:12:53,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 22:12:53,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.02 | bwd_microstep: 5751.96 | bwd_inner_microstep: 5709.23 | bwd_allreduce_microstep: 42.68 | step_microstep: 18.89
[2025-04-25 22:12:53,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.02 | bwd: 5751.97 | bwd_inner: 5709.23 | bwd_allreduce: 42.70 | step: 18.89

 14%|█▍        | 5898/41250 [14:15:18<85:12:08,  8.68s/it]
                                                          
{'loss': 0.1467, 'grad_norm': 1.9213134050369263, 'learning_rate': 3.867614433538613e-05, 'epoch': 1.43}

 14%|█▍        | 5898/41250 [14:15:18<85:12:08,  8.68s/it][2025-04-25 22:13:01,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:13:01,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5780.87 | bwd_inner_microstep: 5698.39 | bwd_allreduce_microstep: 82.42 | step_microstep: 18.46
[2025-04-25 22:13:01,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5780.88 | bwd_inner: 5698.39 | bwd_allreduce: 82.44 | step: 18.46

 14%|█▍        | 5899/41250 [14:15:27<85:19:07,  8.69s/it]
                                                          
{'loss': 0.1016, 'grad_norm': 1.6278600692749023, 'learning_rate': 3.8675582452227764e-05, 'epoch': 1.43}

 14%|█▍        | 5899/41250 [14:15:27<85:19:07,  8.69s/it][2025-04-25 22:13:10,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 1.14
[2025-04-25 22:13:10,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2930.41 | bwd_microstep: 5877.89 | bwd_inner_microstep: 5865.08 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.42
[2025-04-25 22:13:10,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2930.41 | bwd: 5877.91 | bwd_inner: 5865.08 | bwd_allreduce: 12.79 | step: 19.42

 14%|█▍        | 5900/41250 [14:15:36<85:55:10,  8.75s/it]
                                                          
{'loss': 0.0621, 'grad_norm': 3.8436145782470703, 'learning_rate': 3.867502045393811e-05, 'epoch': 1.43}

 14%|█▍        | 5900/41250 [14:15:36<85:55:10,  8.75s/it][2025-04-25 22:13:19,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:13:19,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.52 | bwd_microstep: 5789.56 | bwd_inner_microstep: 5776.73 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.60
[2025-04-25 22:13:19,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.52 | bwd: 5789.57 | bwd_inner: 5776.73 | bwd_allreduce: 12.80 | step: 18.60

 14%|█▍        | 5901/41250 [14:15:44<85:56:02,  8.75s/it]
                                                          
{'loss': 0.4661, 'grad_norm': 3.337528705596924, 'learning_rate': 3.867445834052062e-05, 'epoch': 1.43}

 14%|█▍        | 5901/41250 [14:15:44<85:56:02,  8.75s/it][2025-04-25 22:13:28,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:13:28,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.17 | bwd_microstep: 5895.77 | bwd_inner_microstep: 5669.47 | bwd_allreduce_microstep: 226.26 | step_microstep: 18.67
[2025-04-25 22:13:28,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.17 | bwd: 5895.78 | bwd_inner: 5669.47 | bwd_allreduce: 226.28 | step: 18.67

 14%|█▍        | 5902/41250 [14:15:53<86:05:46,  8.77s/it]
                                                          
{'loss': 0.1562, 'grad_norm': 2.041858673095703, 'learning_rate': 3.867389611197877e-05, 'epoch': 1.43}

 14%|█▍        | 5902/41250 [14:15:53<86:05:46,  8.77s/it][2025-04-25 22:13:37,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-25 22:13:37,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.95 | bwd_microstep: 5721.47 | bwd_inner_microstep: 5708.20 | bwd_allreduce_microstep: 13.22 | step_microstep: 20.27
[2025-04-25 22:13:37,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.95 | bwd: 5721.48 | bwd_inner: 5708.20 | bwd_allreduce: 13.24 | step: 20.27

 14%|█▍        | 5903/41250 [14:16:02<85:46:32,  8.74s/it]
                                                          
{'loss': 0.5524, 'grad_norm': 5.1037797927856445, 'learning_rate': 3.8673333768316025e-05, 'epoch': 1.43}

 14%|█▍        | 5903/41250 [14:16:02<85:46:32,  8.74s/it][2025-04-25 22:13:45,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-25 22:13:45,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.82 | bwd_microstep: 5772.68 | bwd_inner_microstep: 5720.24 | bwd_allreduce_microstep: 52.39 | step_microstep: 18.55
[2025-04-25 22:13:45,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.82 | bwd: 5772.69 | bwd_inner: 5720.24 | bwd_allreduce: 52.41 | step: 18.55

 14%|█▍        | 5904/41250 [14:16:11<85:42:23,  8.73s/it]
                                                          
{'loss': 0.1854, 'grad_norm': 1.8685697317123413, 'learning_rate': 3.8672771309535845e-05, 'epoch': 1.43}

 14%|█▍        | 5904/41250 [14:16:11<85:42:23,  8.73s/it][2025-04-25 22:13:54,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-25 22:13:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.38 | bwd_microstep: 5835.44 | bwd_inner_microstep: 5782.51 | bwd_allreduce_microstep: 52.88 | step_microstep: 18.74
[2025-04-25 22:13:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.38 | bwd: 5835.45 | bwd_inner: 5782.51 | bwd_allreduce: 52.90 | step: 18.75

 14%|█▍        | 5905/41250 [14:16:19<85:56:57,  8.75s/it]
                                                          
{'loss': 0.3468, 'grad_norm': 3.194444179534912, 'learning_rate': 3.867220873564171e-05, 'epoch': 1.43}

 14%|█▍        | 5905/41250 [14:16:19<85:56:57,  8.75s/it][2025-04-25 22:14:03,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.09
[2025-04-25 22:14:03,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5714.83 | bwd_inner_microstep: 5664.20 | bwd_allreduce_microstep: 50.58 | step_microstep: 19.07
[2025-04-25 22:14:03,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5714.84 | bwd_inner: 5664.19 | bwd_allreduce: 50.60 | step: 19.07

 14%|█▍        | 5906/41250 [14:16:28<85:35:08,  8.72s/it]
                                                          
{'loss': 0.0439, 'grad_norm': 0.8200770616531372, 'learning_rate': 3.8671646046637075e-05, 'epoch': 1.43}

 14%|█▍        | 5906/41250 [14:16:28<85:35:08,  8.72s/it][2025-04-25 22:14:11,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:14:11,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.93 | bwd_microstep: 5737.59 | bwd_inner_microstep: 5672.23 | bwd_allreduce_microstep: 65.32 | step_microstep: 18.40
[2025-04-25 22:14:11,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.93 | bwd: 5737.60 | bwd_inner: 5672.23 | bwd_allreduce: 65.33 | step: 18.41

 14%|█▍        | 5907/41250 [14:16:37<85:23:12,  8.70s/it]
                                                          
{'loss': 0.1478, 'grad_norm': 1.877320408821106, 'learning_rate': 3.8671083242525424e-05, 'epoch': 1.43}

 14%|█▍        | 5907/41250 [14:16:37<85:23:12,  8.70s/it][2025-04-25 22:14:20,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:14:20,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.26 | bwd_microstep: 5707.57 | bwd_inner_microstep: 5694.83 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.84
[2025-04-25 22:14:20,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.26 | bwd: 5707.59 | bwd_inner: 5694.83 | bwd_allreduce: 12.72 | step: 18.85

 14%|█▍        | 5908/41250 [14:16:45<85:13:56,  8.68s/it]
                                                          
{'loss': 0.1685, 'grad_norm': 3.549952983856201, 'learning_rate': 3.867052032331021e-05, 'epoch': 1.43}

 14%|█▍        | 5908/41250 [14:16:45<85:13:56,  8.68s/it][2025-04-25 22:14:29,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-25 22:14:29,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.44 | bwd_microstep: 5718.26 | bwd_inner_microstep: 5705.04 | bwd_allreduce_microstep: 13.17 | step_microstep: 19.49
[2025-04-25 22:14:29,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.44 | bwd: 5718.27 | bwd_inner: 5705.04 | bwd_allreduce: 13.19 | step: 19.50

 14%|█▍        | 5909/41250 [14:16:54<85:08:44,  8.67s/it]
                                                          
{'loss': 0.027, 'grad_norm': 0.34629666805267334, 'learning_rate': 3.866995728899491e-05, 'epoch': 1.43}

 14%|█▍        | 5909/41250 [14:16:54<85:08:44,  8.67s/it][2025-04-25 22:14:37,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 22:14:37,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5722.07 | bwd_inner_microstep: 5709.26 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.74
[2025-04-25 22:14:37,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5722.09 | bwd_inner: 5709.26 | bwd_allreduce: 12.78 | step: 18.74

 14%|█▍        | 5910/41250 [14:17:03<85:05:40,  8.67s/it]
                                                          
{'loss': 0.0801, 'grad_norm': 1.1805609464645386, 'learning_rate': 3.8669394139582996e-05, 'epoch': 1.43}

 14%|█▍        | 5910/41250 [14:17:03<85:05:40,  8.67s/it][2025-04-25 22:14:46,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 22:14:46,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.00 | bwd_microstep: 5768.91 | bwd_inner_microstep: 5712.88 | bwd_allreduce_microstep: 55.98 | step_microstep: 19.20
[2025-04-25 22:14:46,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.00 | bwd: 5768.93 | bwd_inner: 5712.88 | bwd_allreduce: 56.00 | step: 19.20

 14%|█▍        | 5911/41250 [14:17:11<85:13:28,  8.68s/it]
                                                          
{'loss': 0.28, 'grad_norm': 2.4993250370025635, 'learning_rate': 3.866883087507794e-05, 'epoch': 1.43}

 14%|█▍        | 5911/41250 [14:17:11<85:13:28,  8.68s/it][2025-04-25 22:14:55,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:14:55,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.39 | bwd_microstep: 5794.32 | bwd_inner_microstep: 5781.47 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.89
[2025-04-25 22:14:55,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.39 | bwd: 5794.33 | bwd_inner: 5781.47 | bwd_allreduce: 12.82 | step: 18.89

 14%|█▍        | 5912/41250 [14:17:20<85:29:34,  8.71s/it]
                                                          
{'loss': 0.2481, 'grad_norm': 3.1441564559936523, 'learning_rate': 3.866826749548321e-05, 'epoch': 1.43}

 14%|█▍        | 5912/41250 [14:17:20<85:29:34,  8.71s/it][2025-04-25 22:15:03,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.24 | optimizer_step: 0.96
[2025-04-25 22:15:03,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5756.28 | bwd_inner_microstep: 5696.46 | bwd_allreduce_microstep: 59.77 | step_microstep: 19.45
[2025-04-25 22:15:03,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5756.29 | bwd_inner: 5696.46 | bwd_allreduce: 59.79 | step: 19.45

 14%|█▍        | 5913/41250 [14:17:29<85:26:33,  8.70s/it]
                                                          
{'loss': 0.1069, 'grad_norm': 1.441693663597107, 'learning_rate': 3.866770400080229e-05, 'epoch': 1.43}

 14%|█▍        | 5913/41250 [14:17:29<85:26:33,  8.70s/it][2025-04-25 22:15:12,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 22:15:12,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.64 | bwd_microstep: 5813.30 | bwd_inner_microstep: 5656.78 | bwd_allreduce_microstep: 156.47 | step_microstep: 18.78
[2025-04-25 22:15:12,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5813.31 | bwd_inner: 5656.78 | bwd_allreduce: 156.48 | step: 18.78

 14%|█▍        | 5914/41250 [14:17:38<85:29:57,  8.71s/it]
                                                          
{'loss': 0.0912, 'grad_norm': 1.7000855207443237, 'learning_rate': 3.8667140391038646e-05, 'epoch': 1.43}

 14%|█▍        | 5914/41250 [14:17:38<85:29:57,  8.71s/it][2025-04-25 22:15:21,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:15:21,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.90 | bwd_microstep: 5715.92 | bwd_inner_microstep: 5702.95 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.89
[2025-04-25 22:15:21,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.90 | bwd: 5715.94 | bwd_inner: 5702.95 | bwd_allreduce: 12.94 | step: 18.89

 14%|█▍        | 5915/41250 [14:17:46<85:18:40,  8.69s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 3.4430594444274902, 'learning_rate': 3.866657666619575e-05, 'epoch': 1.43}

 14%|█▍        | 5915/41250 [14:17:46<85:18:40,  8.69s/it][2025-04-25 22:15:30,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.11 | optimizer_step: 1.03
[2025-04-25 22:15:30,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.60 | bwd_microstep: 5762.74 | bwd_inner_microstep: 5698.73 | bwd_allreduce_microstep: 63.96 | step_microstep: 19.98
[2025-04-25 22:15:30,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.60 | bwd: 5762.76 | bwd_inner: 5698.73 | bwd_allreduce: 63.98 | step: 19.98

 14%|█▍        | 5916/41250 [14:17:55<85:20:08,  8.69s/it]
                                                          
{'loss': 0.0914, 'grad_norm': 0.9425786137580872, 'learning_rate': 3.8666012826277074e-05, 'epoch': 1.43}

 14%|█▍        | 5916/41250 [14:17:55<85:20:08,  8.69s/it][2025-04-25 22:15:38,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:15:38,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.79 | bwd_microstep: 5715.32 | bwd_inner_microstep: 5702.49 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.01
[2025-04-25 22:15:38,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5715.33 | bwd_inner: 5702.49 | bwd_allreduce: 12.80 | step: 19.01

 14%|█▍        | 5917/41250 [14:18:04<85:12:20,  8.68s/it]
                                                          
{'loss': 0.211, 'grad_norm': 2.4640305042266846, 'learning_rate': 3.866544887128611e-05, 'epoch': 1.43}

 14%|█▍        | 5917/41250 [14:18:04<85:12:20,  8.68s/it][2025-04-25 22:15:47,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 22:15:47,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5762.88 | bwd_inner_microstep: 5689.41 | bwd_allreduce_microstep: 73.43 | step_microstep: 18.66
[2025-04-25 22:15:47,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5762.89 | bwd_inner: 5689.41 | bwd_allreduce: 73.44 | step: 18.66

 14%|█▍        | 5918/41250 [14:18:12<85:13:29,  8.68s/it]
                                                          
{'loss': 0.0716, 'grad_norm': 1.3453577756881714, 'learning_rate': 3.866488480122632e-05, 'epoch': 1.43}

 14%|█▍        | 5918/41250 [14:18:12<85:13:29,  8.68s/it][2025-04-25 22:15:56,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 22:15:56,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.58 | bwd_microstep: 5760.75 | bwd_inner_microstep: 5698.77 | bwd_allreduce_microstep: 61.93 | step_microstep: 18.89
[2025-04-25 22:15:56,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.58 | bwd: 5760.76 | bwd_inner: 5698.77 | bwd_allreduce: 61.94 | step: 18.90

 14%|█▍        | 5919/41250 [14:18:21<85:14:23,  8.69s/it]
                                                          
{'loss': 0.182, 'grad_norm': 3.023306131362915, 'learning_rate': 3.866432061610119e-05, 'epoch': 1.43}

 14%|█▍        | 5919/41250 [14:18:21<85:14:23,  8.69s/it][2025-04-25 22:16:04,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 22:16:04,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.93 | bwd_microstep: 5736.57 | bwd_inner_microstep: 5705.07 | bwd_allreduce_microstep: 31.45 | step_microstep: 18.88
[2025-04-25 22:16:04,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.93 | bwd: 5736.58 | bwd_inner: 5705.07 | bwd_allreduce: 31.47 | step: 18.88

 14%|█▍        | 5920/41250 [14:18:30<85:11:59,  8.68s/it]
                                                          
{'loss': 0.0952, 'grad_norm': 2.215611457824707, 'learning_rate': 3.866375631591419e-05, 'epoch': 1.44}

 14%|█▍        | 5920/41250 [14:18:30<85:11:59,  8.68s/it][2025-04-25 22:16:13,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-25 22:16:13,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.84 | bwd_microstep: 5775.88 | bwd_inner_microstep: 5662.15 | bwd_allreduce_microstep: 113.69 | step_microstep: 19.22
[2025-04-25 22:16:13,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.84 | bwd: 5775.90 | bwd_inner: 5662.15 | bwd_allreduce: 113.71 | step: 19.23

 14%|█▍        | 5921/41250 [14:18:38<85:12:35,  8.68s/it]
                                                          
{'loss': 0.0641, 'grad_norm': 1.576524257659912, 'learning_rate': 3.86631919006688e-05, 'epoch': 1.44}

 14%|█▍        | 5921/41250 [14:18:38<85:12:35,  8.68s/it][2025-04-25 22:16:22,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:16:22,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5694.55 | bwd_inner_microstep: 5648.42 | bwd_allreduce_microstep: 46.08 | step_microstep: 18.84
[2025-04-25 22:16:22,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5694.57 | bwd_inner: 5648.42 | bwd_allreduce: 46.11 | step: 18.84

 14%|█▍        | 5922/41250 [14:18:47<84:59:33,  8.66s/it]
                                                          
{'loss': 0.0936, 'grad_norm': 1.9162911176681519, 'learning_rate': 3.8662627370368516e-05, 'epoch': 1.44}

 14%|█▍        | 5922/41250 [14:18:47<84:59:33,  8.66s/it][2025-04-25 22:16:30,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.25 | optimizer_step: 0.97
[2025-04-25 22:16:30,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.45 | bwd_microstep: 5793.32 | bwd_inner_microstep: 5658.18 | bwd_allreduce_microstep: 135.08 | step_microstep: 19.62
[2025-04-25 22:16:30,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.45 | bwd: 5793.33 | bwd_inner: 5658.18 | bwd_allreduce: 135.11 | step: 19.62

 14%|█▍        | 5923/41250 [14:18:56<85:07:03,  8.67s/it]
                                                          
{'loss': 0.1433, 'grad_norm': 3.1753737926483154, 'learning_rate': 3.86620627250168e-05, 'epoch': 1.44}

 14%|█▍        | 5923/41250 [14:18:56<85:07:03,  8.67s/it][2025-04-25 22:16:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-25 22:16:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.02 | bwd_microstep: 5736.48 | bwd_inner_microstep: 5704.43 | bwd_allreduce_microstep: 32.01 | step_microstep: 19.00
[2025-04-25 22:16:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.02 | bwd: 5736.50 | bwd_inner: 5704.43 | bwd_allreduce: 32.02 | step: 19.00

 14%|█▍        | 5924/41250 [14:19:04<85:06:39,  8.67s/it]
                                                          
{'loss': 0.286, 'grad_norm': 2.4158575534820557, 'learning_rate': 3.8661497964617134e-05, 'epoch': 1.44}

 14%|█▍        | 5924/41250 [14:19:04<85:06:39,  8.67s/it][2025-04-25 22:16:48,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-25 22:16:48,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.31 | bwd_microstep: 5817.21 | bwd_inner_microstep: 5639.80 | bwd_allreduce_microstep: 177.36 | step_microstep: 18.87
[2025-04-25 22:16:48,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.31 | bwd: 5817.22 | bwd_inner: 5639.80 | bwd_allreduce: 177.38 | step: 18.87

 14%|█▍        | 5925/41250 [14:19:13<85:15:21,  8.69s/it]
                                                          
{'loss': 0.2645, 'grad_norm': 2.4800374507904053, 'learning_rate': 3.8660933089173004e-05, 'epoch': 1.44}

 14%|█▍        | 5925/41250 [14:19:13<85:15:21,  8.69s/it][2025-04-25 22:16:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-25 22:16:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.24 | bwd_microstep: 5697.86 | bwd_inner_microstep: 5684.81 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.82
[2025-04-25 22:16:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.24 | bwd: 5697.87 | bwd_inner: 5684.81 | bwd_allreduce: 13.02 | step: 18.82

 14%|█▍        | 5926/41250 [14:19:22<85:04:04,  8.67s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.828735113143921, 'learning_rate': 3.8660368098687896e-05, 'epoch': 1.44}

 14%|█▍        | 5926/41250 [14:19:22<85:04:04,  8.67s/it][2025-04-25 22:17:05,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:17:05,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.63 | bwd_microstep: 5766.90 | bwd_inner_microstep: 5754.04 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.03
[2025-04-25 22:17:05,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.63 | bwd: 5766.92 | bwd_inner: 5754.04 | bwd_allreduce: 12.84 | step: 19.04

 14%|█▍        | 5927/41250 [14:19:30<85:14:34,  8.69s/it]
                                                          
{'loss': 0.0594, 'grad_norm': 1.8917381763458252, 'learning_rate': 3.8659802993165285e-05, 'epoch': 1.44}

 14%|█▍        | 5927/41250 [14:19:30<85:14:34,  8.69s/it][2025-04-25 22:17:14,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-25 22:17:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.45 | bwd_microstep: 5709.10 | bwd_inner_microstep: 5643.42 | bwd_allreduce_microstep: 65.62 | step_microstep: 18.97
[2025-04-25 22:17:14,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.45 | bwd: 5709.12 | bwd_inner: 5643.42 | bwd_allreduce: 65.65 | step: 18.97

 14%|█▍        | 5928/41250 [14:19:39<85:02:17,  8.67s/it]
                                                          
{'loss': 0.0853, 'grad_norm': 1.8385581970214844, 'learning_rate': 3.865923777260866e-05, 'epoch': 1.44}

 14%|█▍        | 5928/41250 [14:19:39<85:02:17,  8.67s/it][2025-04-25 22:17:22,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-25 22:17:22,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.64 | bwd_microstep: 5699.48 | bwd_inner_microstep: 5641.40 | bwd_allreduce_microstep: 58.04 | step_microstep: 19.25
[2025-04-25 22:17:22,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.64 | bwd: 5699.50 | bwd_inner: 5641.40 | bwd_allreduce: 58.06 | step: 19.25

 14%|█▍        | 5929/41250 [14:19:48<84:51:13,  8.65s/it]
                                                          
{'loss': 0.1192, 'grad_norm': 0.979945182800293, 'learning_rate': 3.86586724370215e-05, 'epoch': 1.44}

 14%|█▍        | 5929/41250 [14:19:48<84:51:13,  8.65s/it][2025-04-25 22:17:31,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:17:31,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.97 | bwd_microstep: 5725.67 | bwd_inner_microstep: 5712.94 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.84
[2025-04-25 22:17:31,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.97 | bwd: 5725.68 | bwd_inner: 5712.94 | bwd_allreduce: 12.70 | step: 18.85

 14%|█▍        | 5930/41250 [14:19:56<84:54:17,  8.65s/it]
                                                          
{'loss': 0.1176, 'grad_norm': 1.4870407581329346, 'learning_rate': 3.8658106986407304e-05, 'epoch': 1.44}

 14%|█▍        | 5930/41250 [14:19:56<84:54:17,  8.65s/it][2025-04-25 22:17:40,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.22 | optimizer_step: 0.93
[2025-04-25 22:17:40,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.90 | bwd_microstep: 5897.97 | bwd_inner_microstep: 5654.53 | bwd_allreduce_microstep: 243.39 | step_microstep: 19.25
[2025-04-25 22:17:40,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.90 | bwd: 5897.99 | bwd_inner: 5654.53 | bwd_allreduce: 243.42 | step: 19.26

 14%|█▍        | 5931/41250 [14:20:05<85:21:45,  8.70s/it]
                                                          
{'loss': 0.1146, 'grad_norm': 2.5337350368499756, 'learning_rate': 3.865754142076954e-05, 'epoch': 1.44}

 14%|█▍        | 5931/41250 [14:20:05<85:21:45,  8.70s/it][2025-04-25 22:17:48,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:17:48,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.82 | bwd_microstep: 5706.68 | bwd_inner_microstep: 5694.01 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.76
[2025-04-25 22:17:48,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5706.70 | bwd_inner: 5694.01 | bwd_allreduce: 12.65 | step: 18.76

 14%|█▍        | 5932/41250 [14:20:14<85:11:14,  8.68s/it]
                                                          
{'loss': 0.1371, 'grad_norm': 1.6217522621154785, 'learning_rate': 3.8656975740111705e-05, 'epoch': 1.44}

 14%|█▍        | 5932/41250 [14:20:14<85:11:14,  8.68s/it][2025-04-25 22:17:57,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:17:57,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5692.93 | bwd_inner_microstep: 5680.26 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.32
[2025-04-25 22:17:57,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5692.95 | bwd_inner: 5680.26 | bwd_allreduce: 12.64 | step: 18.32

 14%|█▍        | 5933/41250 [14:20:22<85:00:36,  8.67s/it]
                                                          
{'loss': 0.0414, 'grad_norm': 1.659089207649231, 'learning_rate': 3.865640994443729e-05, 'epoch': 1.44}

 14%|█▍        | 5933/41250 [14:20:22<85:00:36,  8.67s/it][2025-04-25 22:18:06,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 22:18:06,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.69 | bwd_microstep: 5788.07 | bwd_inner_microstep: 5774.78 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.05
[2025-04-25 22:18:06,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.69 | bwd: 5788.08 | bwd_inner: 5774.78 | bwd_allreduce: 13.26 | step: 19.06

 14%|█▍        | 5934/41250 [14:20:31<85:17:24,  8.69s/it]
                                                          
{'loss': 0.0759, 'grad_norm': 1.2036981582641602, 'learning_rate': 3.865584403374977e-05, 'epoch': 1.44}

 14%|█▍        | 5934/41250 [14:20:31<85:17:24,  8.69s/it][2025-04-25 22:18:14,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-25 22:18:14,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.78 | bwd_microstep: 5750.15 | bwd_inner_microstep: 5686.12 | bwd_allreduce_microstep: 63.98 | step_microstep: 19.01
[2025-04-25 22:18:14,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.78 | bwd: 5750.16 | bwd_inner: 5686.12 | bwd_allreduce: 63.99 | step: 19.01

 14%|█▍        | 5935/41250 [14:20:40<85:14:33,  8.69s/it]
                                                          
{'loss': 0.0821, 'grad_norm': 1.5328139066696167, 'learning_rate': 3.8655278008052636e-05, 'epoch': 1.44}

 14%|█▍        | 5935/41250 [14:20:40<85:14:33,  8.69s/it][2025-04-25 22:18:23,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:18:23,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.12 | bwd_microstep: 5697.69 | bwd_inner_microstep: 5684.83 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.02
[2025-04-25 22:18:23,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.12 | bwd: 5697.71 | bwd_inner: 5684.83 | bwd_allreduce: 12.84 | step: 19.03

 14%|█▍        | 5936/41250 [14:20:48<85:02:41,  8.67s/it]
                                                          
{'loss': 0.0757, 'grad_norm': 1.5966695547103882, 'learning_rate': 3.865471186734939e-05, 'epoch': 1.44}

 14%|█▍        | 5936/41250 [14:20:48<85:02:41,  8.67s/it][2025-04-25 22:18:32,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 22:18:32,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.65 | bwd_microstep: 5693.48 | bwd_inner_microstep: 5680.71 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.73
[2025-04-25 22:18:32,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.65 | bwd: 5693.49 | bwd_inner: 5680.71 | bwd_allreduce: 12.74 | step: 18.73

 14%|█▍        | 5937/41250 [14:20:57<84:55:41,  8.66s/it]
                                                          
{'loss': 0.1249, 'grad_norm': 1.4220380783081055, 'learning_rate': 3.865414561164351e-05, 'epoch': 1.44}

 14%|█▍        | 5937/41250 [14:20:57<84:55:41,  8.66s/it][2025-04-25 22:18:40,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:18:40,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.65 | bwd_microstep: 5754.42 | bwd_inner_microstep: 5650.23 | bwd_allreduce_microstep: 104.14 | step_microstep: 18.94
[2025-04-25 22:18:40,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.65 | bwd: 5754.43 | bwd_inner: 5650.23 | bwd_allreduce: 104.16 | step: 18.94

 14%|█▍        | 5938/41250 [14:21:06<84:56:24,  8.66s/it]
                                                          
{'loss': 0.1465, 'grad_norm': 1.4781441688537598, 'learning_rate': 3.86535792409385e-05, 'epoch': 1.44}

 14%|█▍        | 5938/41250 [14:21:06<84:56:24,  8.66s/it][2025-04-25 22:18:49,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-25 22:18:49,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.36 | bwd_microstep: 5741.68 | bwd_inner_microstep: 5693.11 | bwd_allreduce_microstep: 48.52 | step_microstep: 19.40
[2025-04-25 22:18:49,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.36 | bwd: 5741.70 | bwd_inner: 5693.11 | bwd_allreduce: 48.54 | step: 19.41

 14%|█▍        | 5939/41250 [14:21:14<84:57:35,  8.66s/it]
                                                          
{'loss': 0.0234, 'grad_norm': 0.35413485765457153, 'learning_rate': 3.8653012755237836e-05, 'epoch': 1.44}

 14%|█▍        | 5939/41250 [14:21:14<84:57:35,  8.66s/it][2025-04-25 22:18:58,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:18:58,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5720.32 | bwd_inner_microstep: 5652.72 | bwd_allreduce_microstep: 67.55 | step_microstep: 18.75
[2025-04-25 22:18:58,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5720.33 | bwd_inner: 5652.72 | bwd_allreduce: 67.57 | step: 18.75

 14%|█▍        | 5940/41250 [14:21:23<84:51:48,  8.65s/it]
                                                          
{'loss': 0.2304, 'grad_norm': 6.402402877807617, 'learning_rate': 3.865244615454502e-05, 'epoch': 1.44}

 14%|█▍        | 5940/41250 [14:21:23<84:51:48,  8.65s/it][2025-04-25 22:19:06,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 22:19:06,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.93 | bwd_microstep: 5789.92 | bwd_inner_microstep: 5777.14 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.96
[2025-04-25 22:19:06,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.93 | bwd: 5789.93 | bwd_inner: 5777.14 | bwd_allreduce: 12.75 | step: 18.96

 14%|█▍        | 5941/41250 [14:21:32<85:11:39,  8.69s/it]
                                                          
{'loss': 0.3099, 'grad_norm': 2.423943281173706, 'learning_rate': 3.865187943886354e-05, 'epoch': 1.44}

 14%|█▍        | 5941/41250 [14:21:32<85:11:39,  8.69s/it][2025-04-25 22:19:15,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:19:15,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5691.57 | bwd_inner_microstep: 5678.79 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.82
[2025-04-25 22:19:15,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5691.59 | bwd_inner: 5678.79 | bwd_allreduce: 12.75 | step: 18.83

 14%|█▍        | 5942/41250 [14:21:40<85:01:31,  8.67s/it]
                                                          
{'loss': 0.1458, 'grad_norm': 2.434711217880249, 'learning_rate': 3.8651312608196897e-05, 'epoch': 1.44}

 14%|█▍        | 5942/41250 [14:21:40<85:01:31,  8.67s/it][2025-04-25 22:19:24,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:19:24,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5756.52 | bwd_inner_microstep: 5661.30 | bwd_allreduce_microstep: 95.17 | step_microstep: 18.74
[2025-04-25 22:19:24,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5756.53 | bwd_inner: 5661.30 | bwd_allreduce: 95.19 | step: 18.74

 14%|█▍        | 5943/41250 [14:21:49<85:01:23,  8.67s/it]
                                                          
{'loss': 0.1229, 'grad_norm': 3.243161201477051, 'learning_rate': 3.865074566254858e-05, 'epoch': 1.44}

 14%|█▍        | 5943/41250 [14:21:49<85:01:23,  8.67s/it][2025-04-25 22:19:33,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:19:33,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5872.47 | bwd_inner_microstep: 5690.70 | bwd_allreduce_microstep: 181.73 | step_microstep: 18.47
[2025-04-25 22:19:33,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5872.49 | bwd_inner: 5690.70 | bwd_allreduce: 181.75 | step: 18.47

 14%|█▍        | 5944/41250 [14:21:58<85:25:34,  8.71s/it]
                                                          
{'loss': 0.1333, 'grad_norm': 1.3955222368240356, 'learning_rate': 3.8650178601922074e-05, 'epoch': 1.44}

 14%|█▍        | 5944/41250 [14:21:58<85:25:34,  8.71s/it][2025-04-25 22:19:41,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:19:41,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.78 | bwd_microstep: 5714.23 | bwd_inner_microstep: 5701.43 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.75
[2025-04-25 22:19:41,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.78 | bwd: 5714.25 | bwd_inner: 5701.43 | bwd_allreduce: 12.77 | step: 18.75

 14%|█▍        | 5945/41250 [14:22:06<85:16:44,  8.70s/it]
                                                          
{'loss': 0.435, 'grad_norm': 2.375783681869507, 'learning_rate': 3.86496114263209e-05, 'epoch': 1.44}

 14%|█▍        | 5945/41250 [14:22:06<85:16:44,  8.70s/it][2025-04-25 22:19:50,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.13 | optimizer_step: 1.06
[2025-04-25 22:19:50,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.45 | bwd_microstep: 5870.50 | bwd_inner_microstep: 5698.95 | bwd_allreduce_microstep: 171.50 | step_microstep: 19.29
[2025-04-25 22:19:50,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.45 | bwd: 5870.51 | bwd_inner: 5698.95 | bwd_allreduce: 171.52 | step: 19.30

 14%|█▍        | 5946/41250 [14:22:15<85:37:23,  8.73s/it]
                                                          
{'loss': 0.2476, 'grad_norm': 2.703193187713623, 'learning_rate': 3.8649044135748525e-05, 'epoch': 1.44}

 14%|█▍        | 5946/41250 [14:22:15<85:37:23,  8.73s/it][2025-04-25 22:19:59,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.06 | optimizer_step: 1.08
[2025-04-25 22:19:59,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.66 | bwd_microstep: 5701.00 | bwd_inner_microstep: 5687.75 | bwd_allreduce_microstep: 13.20 | step_microstep: 19.32
[2025-04-25 22:19:59,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.66 | bwd: 5701.01 | bwd_inner: 5687.75 | bwd_allreduce: 13.22 | step: 19.33

 14%|█▍        | 5947/41250 [14:22:24<85:20:21,  8.70s/it]
                                                          
{'loss': 0.142, 'grad_norm': 3.522315502166748, 'learning_rate': 3.8648476730208464e-05, 'epoch': 1.44}

 14%|█▍        | 5947/41250 [14:22:24<85:20:21,  8.70s/it][2025-04-25 22:20:07,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:20:07,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.29 | bwd_microstep: 5803.79 | bwd_inner_microstep: 5661.28 | bwd_allreduce_microstep: 142.47 | step_microstep: 18.88
[2025-04-25 22:20:07,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.29 | bwd: 5803.81 | bwd_inner: 5661.28 | bwd_allreduce: 142.49 | step: 18.88

 14%|█▍        | 5948/41250 [14:22:33<85:23:27,  8.71s/it]
                                                          
{'loss': 0.1381, 'grad_norm': 2.9432857036590576, 'learning_rate': 3.864790920970422e-05, 'epoch': 1.44}

 14%|█▍        | 5948/41250 [14:22:33<85:23:27,  8.71s/it][2025-04-25 22:20:16,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:20:16,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3084.37 | bwd_microstep: 5699.57 | bwd_inner_microstep: 5686.95 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.45
[2025-04-25 22:20:16,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3084.37 | bwd: 5699.58 | bwd_inner: 5686.95 | bwd_allreduce: 12.60 | step: 18.46

 14%|█▍        | 5949/41250 [14:22:42<85:50:59,  8.75s/it]
                                                          
{'loss': 0.1469, 'grad_norm': 1.0792123079299927, 'learning_rate': 3.864734157423928e-05, 'epoch': 1.44}

 14%|█▍        | 5949/41250 [14:22:42<85:50:59,  8.75s/it][2025-04-25 22:20:25,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:20:25,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5770.70 | bwd_inner_microstep: 5660.03 | bwd_allreduce_microstep: 110.62 | step_microstep: 18.45
[2025-04-25 22:20:25,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5770.71 | bwd_inner: 5660.03 | bwd_allreduce: 110.64 | step: 18.45

 14%|█▍        | 5950/41250 [14:22:50<85:38:42,  8.73s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 3.167292594909668, 'learning_rate': 3.8646773823817147e-05, 'epoch': 1.44}

 14%|█▍        | 5950/41250 [14:22:50<85:38:42,  8.73s/it][2025-04-25 22:20:34,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 22:20:34,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.13 | bwd_microstep: 5725.97 | bwd_inner_microstep: 5665.40 | bwd_allreduce_microstep: 60.52 | step_microstep: 18.48
[2025-04-25 22:20:34,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.13 | bwd: 5725.98 | bwd_inner: 5665.40 | bwd_allreduce: 60.54 | step: 18.49

 14%|█▍        | 5951/41250 [14:22:59<85:21:15,  8.70s/it]
                                                          
{'loss': 0.0657, 'grad_norm': 1.445813536643982, 'learning_rate': 3.8646205958441324e-05, 'epoch': 1.44}

 14%|█▍        | 5951/41250 [14:22:59<85:21:15,  8.70s/it][2025-04-25 22:20:42,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 1.02
[2025-04-25 22:20:42,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.03 | bwd_microstep: 5759.84 | bwd_inner_microstep: 5708.33 | bwd_allreduce_microstep: 51.45 | step_microstep: 19.50
[2025-04-25 22:20:42,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.03 | bwd: 5759.85 | bwd_inner: 5708.33 | bwd_allreduce: 51.48 | step: 19.50

 14%|█▍        | 5952/41250 [14:23:08<85:19:24,  8.70s/it]
                                                          
{'loss': 0.1367, 'grad_norm': 3.08185076713562, 'learning_rate': 3.86456379781153e-05, 'epoch': 1.44}

 14%|█▍        | 5952/41250 [14:23:08<85:19:24,  8.70s/it][2025-04-25 22:20:51,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.24 | optimizer_step: 0.95
[2025-04-25 22:20:51,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.82 | bwd_microstep: 5755.44 | bwd_inner_microstep: 5699.15 | bwd_allreduce_microstep: 56.24 | step_microstep: 19.57
[2025-04-25 22:20:51,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.82 | bwd: 5755.45 | bwd_inner: 5699.15 | bwd_allreduce: 56.26 | step: 19.57

 14%|█▍        | 5953/41250 [14:23:16<85:17:36,  8.70s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 2.8464760780334473, 'learning_rate': 3.864506988284259e-05, 'epoch': 1.44}

 14%|█▍        | 5953/41250 [14:23:16<85:17:36,  8.70s/it][2025-04-25 22:21:00,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:21:00,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.73 | bwd_microstep: 5745.31 | bwd_inner_microstep: 5715.25 | bwd_allreduce_microstep: 30.01 | step_microstep: 18.88
[2025-04-25 22:21:00,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.73 | bwd: 5745.32 | bwd_inner: 5715.25 | bwd_allreduce: 30.03 | step: 18.89

 14%|█▍        | 5954/41250 [14:23:25<85:14:56,  8.69s/it]
                                                          
{'loss': 0.3636, 'grad_norm': 2.4434759616851807, 'learning_rate': 3.86445016726267e-05, 'epoch': 1.44}

 14%|█▍        | 5954/41250 [14:23:25<85:14:56,  8.69s/it][2025-04-25 22:21:08,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 22:21:08,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.35 | bwd_microstep: 5779.04 | bwd_inner_microstep: 5701.07 | bwd_allreduce_microstep: 77.92 | step_microstep: 18.78
[2025-04-25 22:21:08,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.35 | bwd: 5779.05 | bwd_inner: 5701.07 | bwd_allreduce: 77.94 | step: 18.78

 14%|█▍        | 5955/41250 [14:23:34<85:18:20,  8.70s/it]
                                                          
{'loss': 0.1265, 'grad_norm': 2.035027265548706, 'learning_rate': 3.864393334747111e-05, 'epoch': 1.44}

 14%|█▍        | 5955/41250 [14:23:34<85:18:20,  8.70s/it][2025-04-25 22:21:17,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:21:17,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.60 | bwd_microstep: 5717.97 | bwd_inner_microstep: 5705.19 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.33
[2025-04-25 22:21:17,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.60 | bwd: 5717.98 | bwd_inner: 5705.19 | bwd_allreduce: 12.75 | step: 18.34

 14%|█▍        | 5956/41250 [14:23:42<85:13:23,  8.69s/it]
                                                          
{'loss': 0.0921, 'grad_norm': 1.7291606664657593, 'learning_rate': 3.864336490737935e-05, 'epoch': 1.44}

 14%|█▍        | 5956/41250 [14:23:42<85:13:23,  8.69s/it][2025-04-25 22:21:26,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 22:21:26,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.78 | bwd_microstep: 5785.89 | bwd_inner_microstep: 5671.28 | bwd_allreduce_microstep: 114.57 | step_microstep: 18.35
[2025-04-25 22:21:26,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.78 | bwd: 5785.91 | bwd_inner: 5671.28 | bwd_allreduce: 114.59 | step: 18.35

 14%|█▍        | 5957/41250 [14:23:51<85:14:07,  8.69s/it]
                                                          
{'loss': 0.188, 'grad_norm': 2.4686014652252197, 'learning_rate': 3.86427963523549e-05, 'epoch': 1.44}

 14%|█▍        | 5957/41250 [14:23:51<85:14:07,  8.69s/it][2025-04-25 22:21:34,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:21:34,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.56 | bwd_microstep: 5768.80 | bwd_inner_microstep: 5708.76 | bwd_allreduce_microstep: 60.00 | step_microstep: 18.15
[2025-04-25 22:21:34,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.56 | bwd: 5768.82 | bwd_inner: 5708.76 | bwd_allreduce: 60.02 | step: 18.15

 14%|█▍        | 5958/41250 [14:24:00<85:16:34,  8.70s/it]
                                                          
{'loss': 0.0257, 'grad_norm': 0.2589578628540039, 'learning_rate': 3.864222768240129e-05, 'epoch': 1.44}

 14%|█▍        | 5958/41250 [14:24:00<85:16:34,  8.70s/it][2025-04-25 22:21:43,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.21
[2025-04-25 22:21:43,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.59 | bwd_microstep: 5721.71 | bwd_inner_microstep: 5709.21 | bwd_allreduce_microstep: 12.46 | step_microstep: 19.80
[2025-04-25 22:21:43,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.59 | bwd: 5721.73 | bwd_inner: 5709.21 | bwd_allreduce: 12.48 | step: 19.80

 14%|█▍        | 5959/41250 [14:24:08<85:12:06,  8.69s/it]
                                                          
{'loss': 0.1147, 'grad_norm': 1.7898619174957275, 'learning_rate': 3.8641658897522e-05, 'epoch': 1.44}

 14%|█▍        | 5959/41250 [14:24:08<85:12:06,  8.69s/it][2025-04-25 22:21:52,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.06
[2025-04-25 22:21:52,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.91 | bwd_microstep: 5772.66 | bwd_inner_microstep: 5661.35 | bwd_allreduce_microstep: 111.26 | step_microstep: 19.79
[2025-04-25 22:21:52,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.91 | bwd: 5772.67 | bwd_inner: 5661.35 | bwd_allreduce: 111.28 | step: 19.79

 14%|█▍        | 5960/41250 [14:24:17<85:12:52,  8.69s/it]
                                                          
{'loss': 0.2601, 'grad_norm': 4.500431537628174, 'learning_rate': 3.864108999772056e-05, 'epoch': 1.44}

 14%|█▍        | 5960/41250 [14:24:17<85:12:52,  8.69s/it][2025-04-25 22:22:00,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 22:22:00,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.07 | bwd_microstep: 5753.05 | bwd_inner_microstep: 5688.84 | bwd_allreduce_microstep: 64.16 | step_microstep: 19.06
[2025-04-25 22:22:00,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.07 | bwd: 5753.06 | bwd_inner: 5688.84 | bwd_allreduce: 64.18 | step: 19.06

 14%|█▍        | 5961/41250 [14:24:26<85:12:47,  8.69s/it]
                                                          
{'loss': 0.0892, 'grad_norm': 1.3783509731292725, 'learning_rate': 3.8640520983000457e-05, 'epoch': 1.45}

 14%|█▍        | 5961/41250 [14:24:26<85:12:47,  8.69s/it][2025-04-25 22:22:09,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 0.91
[2025-04-25 22:22:09,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.92 | bwd_microstep: 5733.16 | bwd_inner_microstep: 5654.82 | bwd_allreduce_microstep: 78.28 | step_microstep: 19.19
[2025-04-25 22:22:09,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5733.18 | bwd_inner: 5654.82 | bwd_allreduce: 78.31 | step: 19.19

 14%|█▍        | 5962/41250 [14:24:34<85:04:28,  8.68s/it]
                                                          
{'loss': 0.0597, 'grad_norm': 0.5523503422737122, 'learning_rate': 3.863995185336522e-05, 'epoch': 1.45}

 14%|█▍        | 5962/41250 [14:24:34<85:04:28,  8.68s/it][2025-04-25 22:22:18,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.06 | optimizer_step: 0.97
[2025-04-25 22:22:18,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.00 | bwd_microstep: 5779.92 | bwd_inner_microstep: 5699.38 | bwd_allreduce_microstep: 80.49 | step_microstep: 19.06
[2025-04-25 22:22:18,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.00 | bwd: 5779.93 | bwd_inner: 5699.38 | bwd_allreduce: 80.51 | step: 19.07

 14%|█▍        | 5963/41250 [14:24:43<85:10:15,  8.69s/it]
                                                          
{'loss': 0.0634, 'grad_norm': 0.8338819742202759, 'learning_rate': 3.863938260881834e-05, 'epoch': 1.45}

 14%|█▍        | 5963/41250 [14:24:43<85:10:15,  8.69s/it][2025-04-25 22:22:26,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 22:22:26,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.45 | bwd_microstep: 5770.01 | bwd_inner_microstep: 5662.39 | bwd_allreduce_microstep: 107.57 | step_microstep: 18.91
[2025-04-25 22:22:26,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.45 | bwd: 5770.02 | bwd_inner: 5662.39 | bwd_allreduce: 107.59 | step: 18.91

 14%|█▍        | 5964/41250 [14:24:52<85:09:10,  8.69s/it]
                                                          
{'loss': 0.2165, 'grad_norm': 2.267787218093872, 'learning_rate': 3.863881324936333e-05, 'epoch': 1.45}

 14%|█▍        | 5964/41250 [14:24:52<85:09:10,  8.69s/it][2025-04-25 22:22:35,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:22:35,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.57 | bwd_microstep: 5789.45 | bwd_inner_microstep: 5658.89 | bwd_allreduce_microstep: 130.51 | step_microstep: 18.91
[2025-04-25 22:22:35,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.57 | bwd: 5789.46 | bwd_inner: 5658.89 | bwd_allreduce: 130.53 | step: 18.91

 14%|█▍        | 5965/41250 [14:25:01<85:12:00,  8.69s/it]
                                                          
{'loss': 0.2801, 'grad_norm': 3.9203174114227295, 'learning_rate': 3.863824377500371e-05, 'epoch': 1.45}

 14%|█▍        | 5965/41250 [14:25:01<85:12:00,  8.69s/it][2025-04-25 22:22:44,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:22:44,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.93 | bwd_microstep: 5776.54 | bwd_inner_microstep: 5660.47 | bwd_allreduce_microstep: 116.03 | step_microstep: 18.79
[2025-04-25 22:22:44,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.93 | bwd: 5776.56 | bwd_inner: 5660.47 | bwd_allreduce: 116.04 | step: 18.79

 14%|█▍        | 5966/41250 [14:25:09<85:11:21,  8.69s/it]
                                                          
{'loss': 0.029, 'grad_norm': 0.3702692687511444, 'learning_rate': 3.8637674185742975e-05, 'epoch': 1.45}

 14%|█▍        | 5966/41250 [14:25:09<85:11:21,  8.69s/it][2025-04-25 22:22:53,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.34 | optimizer_step: 1.04
[2025-04-25 22:22:53,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.35 | bwd_microstep: 5934.88 | bwd_inner_microstep: 5684.91 | bwd_allreduce_microstep: 249.91 | step_microstep: 19.94
[2025-04-25 22:22:53,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.35 | bwd: 5934.89 | bwd_inner: 5684.91 | bwd_allreduce: 249.93 | step: 19.95

 14%|█▍        | 5967/41250 [14:25:18<85:41:52,  8.74s/it]
                                                          
{'loss': 0.2104, 'grad_norm': 4.268947124481201, 'learning_rate': 3.863710448158465e-05, 'epoch': 1.45}

 14%|█▍        | 5967/41250 [14:25:18<85:41:52,  8.74s/it][2025-04-25 22:23:02,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 22:23:02,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 6025.76 | bwd_inner_microstep: 5657.95 | bwd_allreduce_microstep: 367.77 | step_microstep: 18.94
[2025-04-25 22:23:02,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 6025.78 | bwd_inner: 5657.95 | bwd_allreduce: 367.79 | step: 18.95

 14%|█▍        | 5968/41250 [14:25:27<86:15:42,  8.80s/it]
                                                          
{'loss': 0.1063, 'grad_norm': 1.1939221620559692, 'learning_rate': 3.863653466253223e-05, 'epoch': 1.45}

 14%|█▍        | 5968/41250 [14:25:27<86:15:42,  8.80s/it][2025-04-25 22:23:10,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:23:10,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.60 | bwd_microstep: 5791.16 | bwd_inner_microstep: 5778.40 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.74
[2025-04-25 22:23:10,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.60 | bwd: 5791.18 | bwd_inner: 5778.40 | bwd_allreduce: 12.73 | step: 18.74

 14%|█▍        | 5969/41250 [14:25:36<86:08:42,  8.79s/it]
                                                          
{'loss': 0.0637, 'grad_norm': 1.0410579442977905, 'learning_rate': 3.8635964728589256e-05, 'epoch': 1.45}

 14%|█▍        | 5969/41250 [14:25:36<86:08:42,  8.79s/it][2025-04-25 22:23:19,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:23:19,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5750.54 | bwd_inner_microstep: 5686.12 | bwd_allreduce_microstep: 64.37 | step_microstep: 18.44
[2025-04-25 22:23:19,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5750.55 | bwd_inner: 5686.12 | bwd_allreduce: 64.39 | step: 18.44

 14%|█▍        | 5970/41250 [14:25:44<85:49:52,  8.76s/it]
                                                          
{'loss': 0.0637, 'grad_norm': 1.261069655418396, 'learning_rate': 3.863539467975922e-05, 'epoch': 1.45}

 14%|█▍        | 5970/41250 [14:25:44<85:49:52,  8.76s/it][2025-04-25 22:23:28,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:23:28,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.51 | bwd_microstep: 5792.52 | bwd_inner_microstep: 5660.68 | bwd_allreduce_microstep: 131.79 | step_microstep: 18.77
[2025-04-25 22:23:28,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.51 | bwd: 5792.53 | bwd_inner: 5660.68 | bwd_allreduce: 131.80 | step: 18.77

 14%|█▍        | 5971/41250 [14:25:53<85:40:19,  8.74s/it]
                                                          
{'loss': 0.1405, 'grad_norm': 2.610260486602783, 'learning_rate': 3.863482451604563e-05, 'epoch': 1.45}

 14%|█▍        | 5971/41250 [14:25:53<85:40:19,  8.74s/it][2025-04-25 22:23:37,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 22:23:37,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.86 | bwd_microstep: 5840.40 | bwd_inner_microstep: 5690.47 | bwd_allreduce_microstep: 149.88 | step_microstep: 18.72
[2025-04-25 22:23:37,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.86 | bwd: 5840.42 | bwd_inner: 5690.47 | bwd_allreduce: 149.90 | step: 18.72

 14%|█▍        | 5972/41250 [14:26:02<85:45:40,  8.75s/it]
                                                          
{'loss': 0.2438, 'grad_norm': 2.577087163925171, 'learning_rate': 3.8634254237452024e-05, 'epoch': 1.45}

 14%|█▍        | 5972/41250 [14:26:02<85:45:40,  8.75s/it][2025-04-25 22:23:45,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-25 22:23:45,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.96 | bwd_microstep: 5697.56 | bwd_inner_microstep: 5684.44 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.67
[2025-04-25 22:23:45,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.96 | bwd: 5697.58 | bwd_inner: 5684.44 | bwd_allreduce: 13.10 | step: 18.67

 14%|█▍        | 5973/41250 [14:26:11<85:22:36,  8.71s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.05701744556427, 'learning_rate': 3.86336838439819e-05, 'epoch': 1.45}

 14%|█▍        | 5973/41250 [14:26:11<85:22:36,  8.71s/it][2025-04-25 22:23:54,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 22:23:54,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5701.66 | bwd_inner_microstep: 5656.23 | bwd_allreduce_microstep: 45.37 | step_microstep: 18.88
[2025-04-25 22:23:54,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.89 | bwd: 5701.67 | bwd_inner: 5656.23 | bwd_allreduce: 45.40 | step: 18.88

 14%|█▍        | 5974/41250 [14:26:19<85:04:35,  8.68s/it]
                                                          
{'loss': 0.0934, 'grad_norm': 1.773788571357727, 'learning_rate': 3.863311333563879e-05, 'epoch': 1.45}

 14%|█▍        | 5974/41250 [14:26:19<85:04:35,  8.68s/it][2025-04-25 22:24:03,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 22:24:03,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.29 | bwd_microstep: 5863.49 | bwd_inner_microstep: 5850.53 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.31
[2025-04-25 22:24:03,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.29 | bwd: 5863.51 | bwd_inner: 5850.53 | bwd_allreduce: 12.94 | step: 19.31

 14%|█▍        | 5975/41250 [14:26:28<85:39:34,  8.74s/it]
                                                          
{'loss': 0.2093, 'grad_norm': 2.0757343769073486, 'learning_rate': 3.863254271242619e-05, 'epoch': 1.45}

 14%|█▍        | 5975/41250 [14:26:28<85:39:34,  8.74s/it][2025-04-25 22:24:11,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:24:11,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5749.86 | bwd_inner_microstep: 5677.73 | bwd_allreduce_microstep: 72.09 | step_microstep: 18.68
[2025-04-25 22:24:11,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5749.88 | bwd_inner: 5677.73 | bwd_allreduce: 72.11 | step: 18.68

 14%|█▍        | 5976/41250 [14:26:37<85:28:02,  8.72s/it]
                                                          
{'loss': 0.25, 'grad_norm': 2.1790034770965576, 'learning_rate': 3.8631971974347637e-05, 'epoch': 1.45}

 14%|█▍        | 5976/41250 [14:26:37<85:28:02,  8.72s/it][2025-04-25 22:24:20,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 22:24:20,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.00 | bwd_microstep: 5775.03 | bwd_inner_microstep: 5761.66 | bwd_allreduce_microstep: 13.33 | step_microstep: 19.16
[2025-04-25 22:24:20,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.00 | bwd: 5775.05 | bwd_inner: 5761.66 | bwd_allreduce: 13.35 | step: 19.17

 14%|█▍        | 5977/41250 [14:26:45<85:31:00,  8.73s/it]
                                                          
{'loss': 0.1381, 'grad_norm': 1.943813443183899, 'learning_rate': 3.8631401121406635e-05, 'epoch': 1.45}

 14%|█▍        | 5977/41250 [14:26:45<85:31:00,  8.73s/it][2025-04-25 22:24:29,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:24:29,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 5769.96 | bwd_inner_microstep: 5649.09 | bwd_allreduce_microstep: 120.82 | step_microstep: 18.55
[2025-04-25 22:24:29,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.28 | bwd: 5769.97 | bwd_inner: 5649.09 | bwd_allreduce: 120.84 | step: 18.55

 14%|█▍        | 5978/41250 [14:26:54<85:22:45,  8.71s/it]
                                                          
{'loss': 0.1852, 'grad_norm': 1.6303398609161377, 'learning_rate': 3.863083015360672e-05, 'epoch': 1.45}

 14%|█▍        | 5978/41250 [14:26:54<85:22:45,  8.71s/it][2025-04-25 22:24:37,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 22:24:37,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.91 | bwd_microstep: 5717.55 | bwd_inner_microstep: 5704.84 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.57
[2025-04-25 22:24:37,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.91 | bwd: 5717.56 | bwd_inner: 5704.84 | bwd_allreduce: 12.68 | step: 18.57

 14%|█▍        | 5979/41250 [14:27:03<85:11:10,  8.69s/it]
                                                          
{'loss': 0.393, 'grad_norm': 5.184871673583984, 'learning_rate': 3.8630259070951386e-05, 'epoch': 1.45}

 14%|█▍        | 5979/41250 [14:27:03<85:11:10,  8.69s/it][2025-04-25 22:24:46,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:24:46,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.25 | bwd_microstep: 5704.45 | bwd_inner_microstep: 5691.85 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.78
[2025-04-25 22:24:46,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.25 | bwd: 5704.46 | bwd_inner: 5691.85 | bwd_allreduce: 12.57 | step: 18.78

 14%|█▍        | 5980/41250 [14:27:11<85:01:21,  8.68s/it]
                                                          
{'loss': 0.0341, 'grad_norm': 1.246018409729004, 'learning_rate': 3.862968787344418e-05, 'epoch': 1.45}

 14%|█▍        | 5980/41250 [14:27:11<85:01:21,  8.68s/it][2025-04-25 22:24:55,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:24:55,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.67 | bwd_microstep: 5709.39 | bwd_inner_microstep: 5696.51 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.75
[2025-04-25 22:24:55,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.67 | bwd: 5709.40 | bwd_inner: 5696.51 | bwd_allreduce: 12.85 | step: 18.76

 14%|█▍        | 5981/41250 [14:27:20<84:54:20,  8.67s/it]
                                                          
{'loss': 0.0859, 'grad_norm': 2.345658540725708, 'learning_rate': 3.862911656108861e-05, 'epoch': 1.45}

 14%|█▍        | 5981/41250 [14:27:20<84:54:20,  8.67s/it][2025-04-25 22:25:03,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:25:03,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5751.93 | bwd_inner_microstep: 5653.06 | bwd_allreduce_microstep: 98.83 | step_microstep: 18.69
[2025-04-25 22:25:03,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5751.95 | bwd_inner: 5653.06 | bwd_allreduce: 98.85 | step: 18.70

 15%|█▍        | 5982/41250 [14:27:29<84:53:37,  8.67s/it]
                                                          
{'loss': 0.1086, 'grad_norm': 5.272680282592773, 'learning_rate': 3.86285451338882e-05, 'epoch': 1.45}

 15%|█▍        | 5982/41250 [14:27:29<84:53:37,  8.67s/it][2025-04-25 22:25:12,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-25 22:25:12,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.46 | bwd_microstep: 5748.72 | bwd_inner_microstep: 5699.01 | bwd_allreduce_microstep: 49.66 | step_microstep: 18.45
[2025-04-25 22:25:12,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.46 | bwd: 5748.74 | bwd_inner: 5699.01 | bwd_allreduce: 49.68 | step: 18.45

 15%|█▍        | 5983/41250 [14:27:37<84:55:26,  8.67s/it]
                                                          
{'loss': 0.0437, 'grad_norm': 0.667789876461029, 'learning_rate': 3.8627973591846474e-05, 'epoch': 1.45}

 15%|█▍        | 5983/41250 [14:27:37<84:55:26,  8.67s/it][2025-04-25 22:25:21,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-25 22:25:21,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.68 | bwd_microstep: 5700.28 | bwd_inner_microstep: 5687.51 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.09
[2025-04-25 22:25:21,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.68 | bwd: 5700.30 | bwd_inner: 5687.51 | bwd_allreduce: 12.74 | step: 19.09

 15%|█▍        | 5984/41250 [14:27:46<84:48:29,  8.66s/it]
                                                          
{'loss': 0.0895, 'grad_norm': 2.0766124725341797, 'learning_rate': 3.862740193496695e-05, 'epoch': 1.45}

 15%|█▍        | 5984/41250 [14:27:46<84:48:29,  8.66s/it][2025-04-25 22:25:29,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-25 22:25:29,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.43 | bwd_microstep: 5741.90 | bwd_inner_microstep: 5689.38 | bwd_allreduce_microstep: 52.47 | step_microstep: 19.22
[2025-04-25 22:25:29,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.43 | bwd: 5741.91 | bwd_inner: 5689.38 | bwd_allreduce: 52.49 | step: 19.22

 15%|█▍        | 5985/41250 [14:27:55<84:51:07,  8.66s/it]
                                                          
{'loss': 0.2955, 'grad_norm': 3.746577024459839, 'learning_rate': 3.862683016325316e-05, 'epoch': 1.45}

 15%|█▍        | 5985/41250 [14:27:55<84:51:07,  8.66s/it][2025-04-25 22:25:38,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:25:38,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.58 | bwd_microstep: 5732.00 | bwd_inner_microstep: 5691.88 | bwd_allreduce_microstep: 40.07 | step_microstep: 18.78
[2025-04-25 22:25:38,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.58 | bwd: 5732.01 | bwd_inner: 5691.88 | bwd_allreduce: 40.09 | step: 18.78

 15%|█▍        | 5986/41250 [14:28:03<84:51:17,  8.66s/it]
                                                          
{'loss': 0.066, 'grad_norm': 1.641679286956787, 'learning_rate': 3.862625827670863e-05, 'epoch': 1.45}

 15%|█▍        | 5986/41250 [14:28:03<84:51:17,  8.66s/it][2025-04-25 22:25:47,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:25:47,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.57 | bwd_microstep: 5742.71 | bwd_inner_microstep: 5689.17 | bwd_allreduce_microstep: 53.49 | step_microstep: 18.80
[2025-04-25 22:25:47,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.57 | bwd: 5742.72 | bwd_inner: 5689.16 | bwd_allreduce: 53.52 | step: 18.80

 15%|█▍        | 5987/41250 [14:28:12<84:52:39,  8.67s/it]
                                                          
{'loss': 0.1472, 'grad_norm': 1.0787123441696167, 'learning_rate': 3.862568627533688e-05, 'epoch': 1.45}

 15%|█▍        | 5987/41250 [14:28:12<84:52:39,  8.67s/it][2025-04-25 22:25:55,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:25:55,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.29 | bwd_microstep: 5732.88 | bwd_inner_microstep: 5691.50 | bwd_allreduce_microstep: 41.33 | step_microstep: 18.86
[2025-04-25 22:25:55,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.29 | bwd: 5732.89 | bwd_inner: 5691.50 | bwd_allreduce: 41.35 | step: 18.86

 15%|█▍        | 5988/41250 [14:28:21<84:53:13,  8.67s/it]
                                                          
{'loss': 0.1178, 'grad_norm': 3.8970842361450195, 'learning_rate': 3.862511415914144e-05, 'epoch': 1.45}

 15%|█▍        | 5988/41250 [14:28:21<84:53:13,  8.67s/it][2025-04-25 22:26:04,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.08 | optimizer_step: 0.96
[2025-04-25 22:26:04,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.31 | bwd_microstep: 5765.29 | bwd_inner_microstep: 5650.33 | bwd_allreduce_microstep: 114.91 | step_microstep: 19.43
[2025-04-25 22:26:04,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.31 | bwd: 5765.30 | bwd_inner: 5650.33 | bwd_allreduce: 114.93 | step: 19.43

 15%|█▍        | 5989/41250 [14:28:29<84:54:13,  8.67s/it]
                                                          
{'loss': 0.3216, 'grad_norm': 4.374232292175293, 'learning_rate': 3.862454192812583e-05, 'epoch': 1.45}

 15%|█▍        | 5989/41250 [14:28:29<84:54:13,  8.67s/it][2025-04-25 22:26:13,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 22:26:13,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.41 | bwd_microstep: 5761.72 | bwd_inner_microstep: 5655.51 | bwd_allreduce_microstep: 106.16 | step_microstep: 18.83
[2025-04-25 22:26:13,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.41 | bwd: 5761.73 | bwd_inner: 5655.51 | bwd_allreduce: 106.18 | step: 18.83

 15%|█▍        | 5990/41250 [14:28:38<84:55:01,  8.67s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.5697267055511475, 'learning_rate': 3.862396958229358e-05, 'epoch': 1.45}

 15%|█▍        | 5990/41250 [14:28:38<84:55:01,  8.67s/it][2025-04-25 22:26:22,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-25 22:26:22,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.04 | bwd_microstep: 5887.06 | bwd_inner_microstep: 5874.31 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.83
[2025-04-25 22:26:22,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.04 | bwd: 5887.08 | bwd_inner: 5874.31 | bwd_allreduce: 12.73 | step: 18.83

 15%|█▍        | 5991/41250 [14:28:47<85:36:19,  8.74s/it]
                                                          
{'loss': 0.2402, 'grad_norm': 4.361567497253418, 'learning_rate': 3.862339712164822e-05, 'epoch': 1.45}

 15%|█▍        | 5991/41250 [14:28:47<85:36:19,  8.74s/it][2025-04-25 22:26:31,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 22:26:31,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.19 | bwd_microstep: 5891.53 | bwd_inner_microstep: 5878.67 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.19
[2025-04-25 22:26:31,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2929.19 | bwd: 5891.54 | bwd_inner: 5878.67 | bwd_allreduce: 12.83 | step: 19.19

 15%|█▍        | 5992/41250 [14:28:56<86:05:11,  8.79s/it]
                                                          
{'loss': 0.4158, 'grad_norm': 4.031471252441406, 'learning_rate': 3.862282454619329e-05, 'epoch': 1.45}

 15%|█▍        | 5992/41250 [14:28:56<86:05:11,  8.79s/it][2025-04-25 22:26:39,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 22:26:39,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.08 | bwd_microstep: 5712.94 | bwd_inner_microstep: 5700.14 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.69
[2025-04-25 22:26:39,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.08 | bwd: 5712.95 | bwd_inner: 5700.14 | bwd_allreduce: 12.77 | step: 18.70

 15%|█▍        | 5993/41250 [14:29:05<85:40:18,  8.75s/it]
                                                          
{'loss': 0.3106, 'grad_norm': 5.127185344696045, 'learning_rate': 3.862225185593229e-05, 'epoch': 1.45}

 15%|█▍        | 5993/41250 [14:29:05<85:40:18,  8.75s/it][2025-04-25 22:26:48,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 22:26:48,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5762.41 | bwd_inner_microstep: 5646.14 | bwd_allreduce_microstep: 116.22 | step_microstep: 19.08
[2025-04-25 22:26:48,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5762.42 | bwd_inner: 5646.14 | bwd_allreduce: 116.24 | step: 19.08

 15%|█▍        | 5994/41250 [14:29:13<85:27:04,  8.73s/it]
                                                          
{'loss': 0.0551, 'grad_norm': 0.5558867454528809, 'learning_rate': 3.862167905086879e-05, 'epoch': 1.45}

 15%|█▍        | 5994/41250 [14:29:13<85:27:04,  8.73s/it][2025-04-25 22:26:57,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 22:26:57,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.44 | bwd_microstep: 5846.59 | bwd_inner_microstep: 5693.32 | bwd_allreduce_microstep: 153.22 | step_microstep: 19.56
[2025-04-25 22:26:57,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.44 | bwd: 5846.61 | bwd_inner: 5693.32 | bwd_allreduce: 153.24 | step: 19.56

 15%|█▍        | 5995/41250 [14:29:22<85:35:45,  8.74s/it]
                                                          
{'loss': 0.2408, 'grad_norm': 1.8026617765426636, 'learning_rate': 3.8621106131006295e-05, 'epoch': 1.45}

 15%|█▍        | 5995/41250 [14:29:22<85:35:45,  8.74s/it][2025-04-25 22:27:05,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 22:27:05,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.94 | bwd_microstep: 5708.81 | bwd_inner_microstep: 5692.16 | bwd_allreduce_microstep: 16.61 | step_microstep: 18.64
[2025-04-25 22:27:05,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.94 | bwd: 5708.82 | bwd_inner: 5692.16 | bwd_allreduce: 16.62 | step: 18.64

 15%|█▍        | 5996/41250 [14:29:31<85:17:53,  8.71s/it]
                                                          
{'loss': 0.1117, 'grad_norm': 1.3722176551818848, 'learning_rate': 3.862053309634834e-05, 'epoch': 1.45}

 15%|█▍        | 5996/41250 [14:29:31<85:17:53,  8.71s/it][2025-04-25 22:27:14,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:27:14,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.08 | bwd_microstep: 5763.88 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 68.03 | step_microstep: 18.71
[2025-04-25 22:27:14,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.08 | bwd: 5763.89 | bwd_inner: 5695.81 | bwd_allreduce: 68.04 | step: 18.72

 15%|█▍        | 5997/41250 [14:29:39<85:15:10,  8.71s/it]
                                                          
{'loss': 0.1395, 'grad_norm': 2.2690796852111816, 'learning_rate': 3.8619959946898465e-05, 'epoch': 1.45}

 15%|█▍        | 5997/41250 [14:29:39<85:15:10,  8.71s/it][2025-04-25 22:27:23,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:27:23,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.82 | bwd_microstep: 5695.62 | bwd_inner_microstep: 5658.74 | bwd_allreduce_microstep: 36.84 | step_microstep: 18.69
[2025-04-25 22:27:23,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.82 | bwd: 5695.64 | bwd_inner: 5658.74 | bwd_allreduce: 36.86 | step: 18.69

 15%|█▍        | 5998/41250 [14:29:48<84:57:22,  8.68s/it]
                                                          
{'loss': 0.1772, 'grad_norm': 7.9394378662109375, 'learning_rate': 3.86193866826602e-05, 'epoch': 1.45}

 15%|█▍        | 5998/41250 [14:29:48<84:57:22,  8.68s/it][2025-04-25 22:27:32,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:27:32,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.66 | bwd_microstep: 6025.85 | bwd_inner_microstep: 5666.17 | bwd_allreduce_microstep: 359.64 | step_microstep: 18.67
[2025-04-25 22:27:32,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.66 | bwd: 6025.87 | bwd_inner: 5666.17 | bwd_allreduce: 359.65 | step: 18.67

 15%|█▍        | 5999/41250 [14:29:57<85:44:08,  8.76s/it]
                                                          
{'loss': 0.2833, 'grad_norm': 3.2787697315216064, 'learning_rate': 3.861881330363708e-05, 'epoch': 1.45}

 15%|█▍        | 5999/41250 [14:29:57<85:44:08,  8.76s/it][2025-04-25 22:27:40,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 22:27:40,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.63 | bwd_microstep: 5765.30 | bwd_inner_microstep: 5703.66 | bwd_allreduce_microstep: 61.59 | step_microstep: 18.77
[2025-04-25 22:27:40,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.63 | bwd: 5765.31 | bwd_inner: 5703.66 | bwd_allreduce: 61.61 | step: 18.78

 15%|█▍        | 6000/41250 [14:30:06<85:34:09,  8.74s/it]
                                                          
{'loss': 0.0896, 'grad_norm': 1.4204987287521362, 'learning_rate': 3.861823980983263e-05, 'epoch': 1.45}

 15%|█▍        | 6000/41250 [14:30:06<85:34:09,  8.74s/it][2025-04-25 22:27:49,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:27:49,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.19 | bwd_microstep: 5706.12 | bwd_inner_microstep: 5663.04 | bwd_allreduce_microstep: 43.04 | step_microstep: 18.28
[2025-04-25 22:27:49,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.19 | bwd: 5706.13 | bwd_inner: 5663.04 | bwd_allreduce: 43.05 | step: 18.28

 15%|█▍        | 6001/41250 [14:30:14<85:13:30,  8.70s/it]
                                                          
{'loss': 0.5359, 'grad_norm': 3.469266176223755, 'learning_rate': 3.86176662012504e-05, 'epoch': 1.45}

 15%|█▍        | 6001/41250 [14:30:14<85:13:30,  8.70s/it][2025-04-25 22:27:58,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 22:27:58,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.53 | bwd_microstep: 5770.51 | bwd_inner_microstep: 5706.76 | bwd_allreduce_microstep: 63.70 | step_microstep: 18.54
[2025-04-25 22:27:58,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.53 | bwd: 5770.52 | bwd_inner: 5706.76 | bwd_allreduce: 63.72 | step: 18.54

 15%|█▍        | 6002/41250 [14:30:23<85:13:07,  8.70s/it]
                                                          
{'loss': 0.2288, 'grad_norm': 2.8164730072021484, 'learning_rate': 3.861709247789391e-05, 'epoch': 1.46}

 15%|█▍        | 6002/41250 [14:30:23<85:13:07,  8.70s/it][2025-04-25 22:28:06,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:28:06,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.89 | bwd_microstep: 5710.93 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.71
[2025-04-25 22:28:06,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.89 | bwd: 5710.94 | bwd_inner: 5697.92 | bwd_allreduce: 12.98 | step: 18.71

 15%|█▍        | 6003/41250 [14:30:32<85:01:32,  8.68s/it]
                                                          
{'loss': 0.2511, 'grad_norm': 1.9134937524795532, 'learning_rate': 3.861651863976672e-05, 'epoch': 1.46}

 15%|█▍        | 6003/41250 [14:30:32<85:01:32,  8.68s/it][2025-04-25 22:28:15,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 22:28:15,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.51 | bwd_microstep: 5785.98 | bwd_inner_microstep: 5675.66 | bwd_allreduce_microstep: 110.28 | step_microstep: 19.17
[2025-04-25 22:28:15,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.51 | bwd: 5785.99 | bwd_inner: 5675.66 | bwd_allreduce: 110.30 | step: 19.17

 15%|█▍        | 6004/41250 [14:30:40<85:04:19,  8.69s/it]
                                                          
{'loss': 0.0346, 'grad_norm': 3.3747308254241943, 'learning_rate': 3.8615944686872344e-05, 'epoch': 1.46}

 15%|█▍        | 6004/41250 [14:30:40<85:04:19,  8.69s/it][2025-04-25 22:28:24,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.07 | optimizer_step: 0.95
[2025-04-25 22:28:24,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 5728.49 | bwd_inner_microstep: 5659.84 | bwd_allreduce_microstep: 68.59 | step_microstep: 19.85
[2025-04-25 22:28:24,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.28 | bwd: 5728.50 | bwd_inner: 5659.84 | bwd_allreduce: 68.61 | step: 19.85

 15%|█▍        | 6005/41250 [14:30:49<84:55:38,  8.67s/it]
                                                          
{'loss': 0.0664, 'grad_norm': 1.5825631618499756, 'learning_rate': 3.8615370619214336e-05, 'epoch': 1.46}

 15%|█▍        | 6005/41250 [14:30:49<84:55:38,  8.67s/it][2025-04-25 22:28:32,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-25 22:28:32,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.98 | bwd_microstep: 5731.13 | bwd_inner_microstep: 5664.43 | bwd_allreduce_microstep: 66.65 | step_microstep: 19.10
[2025-04-25 22:28:32,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.98 | bwd: 5731.15 | bwd_inner: 5664.43 | bwd_allreduce: 66.67 | step: 19.10

 15%|█▍        | 6006/41250 [14:30:58<84:50:43,  8.67s/it]
                                                          
{'loss': 0.2139, 'grad_norm': 3.776607036590576, 'learning_rate': 3.8614796436796226e-05, 'epoch': 1.46}

 15%|█▍        | 6006/41250 [14:30:58<84:50:43,  8.67s/it][2025-04-25 22:28:41,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-25 22:28:41,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.52 | bwd_microstep: 5728.28 | bwd_inner_microstep: 5653.25 | bwd_allreduce_microstep: 74.99 | step_microstep: 18.45
[2025-04-25 22:28:41,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.52 | bwd: 5728.29 | bwd_inner: 5653.25 | bwd_allreduce: 75.00 | step: 18.45

 15%|█▍        | 6007/41250 [14:31:06<84:46:54,  8.66s/it]
                                                          
{'loss': 0.5148, 'grad_norm': 3.078779697418213, 'learning_rate': 3.861422213962156e-05, 'epoch': 1.46}

 15%|█▍        | 6007/41250 [14:31:06<84:46:54,  8.66s/it][2025-04-25 22:28:50,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:28:50,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.25 | bwd_microstep: 5744.52 | bwd_inner_microstep: 5664.01 | bwd_allreduce_microstep: 80.46 | step_microstep: 18.47
[2025-04-25 22:28:50,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.25 | bwd: 5744.53 | bwd_inner: 5664.01 | bwd_allreduce: 80.48 | step: 18.48

 15%|█▍        | 6008/41250 [14:31:15<84:47:31,  8.66s/it]
                                                          
{'loss': 0.0372, 'grad_norm': 0.718842625617981, 'learning_rate': 3.861364772769388e-05, 'epoch': 1.46}

 15%|█▍        | 6008/41250 [14:31:15<84:47:31,  8.66s/it][2025-04-25 22:28:58,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:28:58,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.60 | bwd_microstep: 5739.21 | bwd_inner_microstep: 5663.33 | bwd_allreduce_microstep: 75.84 | step_microstep: 18.56
[2025-04-25 22:28:58,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.60 | bwd: 5739.22 | bwd_inner: 5663.33 | bwd_allreduce: 75.86 | step: 18.57

 15%|█▍        | 6009/41250 [14:31:23<84:45:54,  8.66s/it]
                                                          
{'loss': 0.0494, 'grad_norm': 1.043354868888855, 'learning_rate': 3.861307320101672e-05, 'epoch': 1.46}

 15%|█▍        | 6009/41250 [14:31:23<84:45:54,  8.66s/it][2025-04-25 22:29:07,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:29:07,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.88 | bwd_microstep: 5719.59 | bwd_inner_microstep: 5661.41 | bwd_allreduce_microstep: 58.13 | step_microstep: 18.55
[2025-04-25 22:29:07,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.88 | bwd: 5719.60 | bwd_inner: 5661.41 | bwd_allreduce: 58.15 | step: 18.55

 15%|█▍        | 6010/41250 [14:31:32<84:41:13,  8.65s/it]
                                                          
{'loss': 0.1703, 'grad_norm': 2.671112060546875, 'learning_rate': 3.861249855959362e-05, 'epoch': 1.46}

 15%|█▍        | 6010/41250 [14:31:32<84:41:13,  8.65s/it][2025-04-25 22:29:16,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 1.13
[2025-04-25 22:29:16,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.17 | bwd_microstep: 5777.33 | bwd_inner_microstep: 5708.24 | bwd_allreduce_microstep: 69.05 | step_microstep: 18.94
[2025-04-25 22:29:16,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.17 | bwd: 5777.35 | bwd_inner: 5708.24 | bwd_allreduce: 69.07 | step: 18.94

 15%|█▍        | 6011/41250 [14:31:41<84:51:58,  8.67s/it]
                                                          
{'loss': 0.0468, 'grad_norm': 0.6078846454620361, 'learning_rate': 3.861192380342813e-05, 'epoch': 1.46}

 15%|█▍        | 6011/41250 [14:31:41<84:51:58,  8.67s/it][2025-04-25 22:29:24,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 22:29:24,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.98 | bwd_microstep: 5775.31 | bwd_inner_microstep: 5657.54 | bwd_allreduce_microstep: 117.71 | step_microstep: 19.10
[2025-04-25 22:29:24,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.98 | bwd: 5775.32 | bwd_inner: 5657.54 | bwd_allreduce: 117.73 | step: 19.11

 15%|█▍        | 6012/41250 [14:31:50<84:55:12,  8.68s/it]
                                                          
{'loss': 0.1012, 'grad_norm': 1.638817548751831, 'learning_rate': 3.861134893252379e-05, 'epoch': 1.46}

 15%|█▍        | 6012/41250 [14:31:50<84:55:12,  8.68s/it][2025-04-25 22:29:33,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:29:33,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.95 | bwd_microstep: 5802.32 | bwd_inner_microstep: 5650.64 | bwd_allreduce_microstep: 151.64 | step_microstep: 18.43
[2025-04-25 22:29:33,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.95 | bwd: 5802.34 | bwd_inner: 5650.64 | bwd_allreduce: 151.66 | step: 18.44

 15%|█▍        | 6013/41250 [14:31:58<85:01:57,  8.69s/it]
                                                          
{'loss': 0.0414, 'grad_norm': 0.3184720277786255, 'learning_rate': 3.8610773946884145e-05, 'epoch': 1.46}

 15%|█▍        | 6013/41250 [14:31:58<85:01:57,  8.69s/it][2025-04-25 22:29:42,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:29:42,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.74 | bwd_microstep: 5766.33 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 108.62 | step_microstep: 18.75
[2025-04-25 22:29:42,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.74 | bwd: 5766.34 | bwd_inner: 5657.65 | bwd_allreduce: 108.65 | step: 18.75

 15%|█▍        | 6014/41250 [14:32:07<84:59:15,  8.68s/it]
                                                          
{'loss': 0.1876, 'grad_norm': 4.004354953765869, 'learning_rate': 3.861019884651274e-05, 'epoch': 1.46}

 15%|█▍        | 6014/41250 [14:32:07<84:59:15,  8.68s/it][2025-04-25 22:29:50,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:29:50,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.97 | bwd_microstep: 5769.03 | bwd_inner_microstep: 5703.17 | bwd_allreduce_microstep: 65.82 | step_microstep: 18.67
[2025-04-25 22:29:50,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.97 | bwd: 5769.05 | bwd_inner: 5703.17 | bwd_allreduce: 65.84 | step: 18.67

 15%|█▍        | 6015/41250 [14:32:16<85:02:21,  8.69s/it]
                                                          
{'loss': 0.1337, 'grad_norm': 3.2183620929718018, 'learning_rate': 3.860962363141312e-05, 'epoch': 1.46}

 15%|█▍        | 6015/41250 [14:32:16<85:02:21,  8.69s/it][2025-04-25 22:29:59,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:29:59,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5726.39 | bwd_inner_microstep: 5713.76 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.70
[2025-04-25 22:29:59,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5726.41 | bwd_inner: 5713.76 | bwd_allreduce: 12.60 | step: 18.71

 15%|█▍        | 6016/41250 [14:32:24<84:57:30,  8.68s/it]
                                                          
{'loss': 0.1351, 'grad_norm': 1.0827490091323853, 'learning_rate': 3.860904830158883e-05, 'epoch': 1.46}

 15%|█▍        | 6016/41250 [14:32:24<84:57:30,  8.68s/it][2025-04-25 22:30:08,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:30:08,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.56 | bwd_microstep: 5721.98 | bwd_inner_microstep: 5709.48 | bwd_allreduce_microstep: 12.45 | step_microstep: 18.63
[2025-04-25 22:30:08,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.56 | bwd: 5722.00 | bwd_inner: 5709.48 | bwd_allreduce: 12.47 | step: 18.64

 15%|█▍        | 6017/41250 [14:32:33<84:53:32,  8.67s/it]
                                                          
{'loss': 0.0677, 'grad_norm': 2.362746477127075, 'learning_rate': 3.860847285704342e-05, 'epoch': 1.46}

 15%|█▍        | 6017/41250 [14:32:33<84:53:32,  8.67s/it][2025-04-25 22:30:16,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 22:30:16,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.71 | bwd_microstep: 5698.73 | bwd_inner_microstep: 5685.87 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.53
[2025-04-25 22:30:16,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.71 | bwd: 5698.75 | bwd_inner: 5685.87 | bwd_allreduce: 12.83 | step: 19.53

 15%|█▍        | 6018/41250 [14:32:42<84:45:46,  8.66s/it]
                                                          
{'loss': 0.0881, 'grad_norm': 1.0314360857009888, 'learning_rate': 3.860789729778043e-05, 'epoch': 1.46}

 15%|█▍        | 6018/41250 [14:32:42<84:45:46,  8.66s/it][2025-04-25 22:30:25,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.25 | optimizer_step: 0.96
[2025-04-25 22:30:25,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.58 | bwd_microstep: 5757.33 | bwd_inner_microstep: 5698.79 | bwd_allreduce_microstep: 58.49 | step_microstep: 19.78
[2025-04-25 22:30:25,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.58 | bwd: 5757.34 | bwd_inner: 5698.79 | bwd_allreduce: 58.51 | step: 19.79

 15%|█▍        | 6019/41250 [14:32:50<84:50:47,  8.67s/it]
                                                          
{'loss': 0.0673, 'grad_norm': 1.0969582796096802, 'learning_rate': 3.8607321623803414e-05, 'epoch': 1.46}

 15%|█▍        | 6019/41250 [14:32:50<84:50:47,  8.67s/it][2025-04-25 22:30:34,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:30:34,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.67 | bwd_microstep: 5756.52 | bwd_inner_microstep: 5649.61 | bwd_allreduce_microstep: 106.86 | step_microstep: 18.81
[2025-04-25 22:30:34,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.67 | bwd: 5756.53 | bwd_inner: 5649.61 | bwd_allreduce: 106.88 | step: 18.81

 15%|█▍        | 6020/41250 [14:32:59<84:50:30,  8.67s/it]
                                                          
{'loss': 0.3519, 'grad_norm': 3.006700277328491, 'learning_rate': 3.860674583511592e-05, 'epoch': 1.46}

 15%|█▍        | 6020/41250 [14:32:59<84:50:30,  8.67s/it][2025-04-25 22:30:42,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-25 22:30:42,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.35 | bwd_microstep: 5783.64 | bwd_inner_microstep: 5645.91 | bwd_allreduce_microstep: 137.69 | step_microstep: 19.07
[2025-04-25 22:30:42,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.35 | bwd: 5783.65 | bwd_inner: 5645.91 | bwd_allreduce: 137.70 | step: 19.07

 15%|█▍        | 6021/41250 [14:33:08<84:54:23,  8.68s/it]
                                                          
{'loss': 0.0874, 'grad_norm': 2.1098389625549316, 'learning_rate': 3.86061699317215e-05, 'epoch': 1.46}

 15%|█▍        | 6021/41250 [14:33:08<84:54:23,  8.68s/it][2025-04-25 22:30:51,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 22:30:51,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.37 | bwd_microstep: 5782.65 | bwd_inner_microstep: 5769.68 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.93
[2025-04-25 22:30:51,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.37 | bwd: 5782.66 | bwd_inner: 5769.68 | bwd_allreduce: 12.93 | step: 18.93

 15%|█▍        | 6022/41250 [14:33:16<85:08:40,  8.70s/it]
                                                          
{'loss': 0.211, 'grad_norm': 2.4093122482299805, 'learning_rate': 3.86055939136237e-05, 'epoch': 1.46}

 15%|█▍        | 6022/41250 [14:33:16<85:08:40,  8.70s/it][2025-04-25 22:31:00,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-25 22:31:00,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 5688.59 | bwd_inner_microstep: 5662.89 | bwd_allreduce_microstep: 25.66 | step_microstep: 19.28
[2025-04-25 22:31:00,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.28 | bwd: 5688.60 | bwd_inner: 5662.89 | bwd_allreduce: 25.67 | step: 19.28

 15%|█▍        | 6023/41250 [14:33:25<84:51:15,  8.67s/it]
                                                          
{'loss': 0.1648, 'grad_norm': 0.9961243271827698, 'learning_rate': 3.8605017780826074e-05, 'epoch': 1.46}

 15%|█▍        | 6023/41250 [14:33:25<84:51:15,  8.67s/it][2025-04-25 22:31:08,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:31:08,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.26 | bwd_microstep: 5700.98 | bwd_inner_microstep: 5688.07 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.98
[2025-04-25 22:31:08,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.26 | bwd: 5701.00 | bwd_inner: 5688.07 | bwd_allreduce: 12.88 | step: 18.99

 15%|█▍        | 6024/41250 [14:33:34<84:43:51,  8.66s/it]
                                                          
{'loss': 0.1807, 'grad_norm': 2.960679769515991, 'learning_rate': 3.860444153333217e-05, 'epoch': 1.46}

 15%|█▍        | 6024/41250 [14:33:34<84:43:51,  8.66s/it][2025-04-25 22:31:17,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.08 | optimizer_step: 1.06
[2025-04-25 22:31:17,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5749.12 | bwd_inner_microstep: 5702.97 | bwd_allreduce_microstep: 46.09 | step_microstep: 19.44
[2025-04-25 22:31:17,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5749.13 | bwd_inner: 5702.97 | bwd_allreduce: 46.11 | step: 19.45

 15%|█▍        | 6025/41250 [14:33:42<84:47:47,  8.67s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 2.1977336406707764, 'learning_rate': 3.860386517114555e-05, 'epoch': 1.46}

 15%|█▍        | 6025/41250 [14:33:42<84:47:47,  8.67s/it][2025-04-25 22:31:26,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.19 | optimizer_step: 1.01
[2025-04-25 22:31:26,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.63 | bwd_microstep: 5747.31 | bwd_inner_microstep: 5684.21 | bwd_allreduce_microstep: 63.05 | step_microstep: 19.23
[2025-04-25 22:31:26,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.63 | bwd: 5747.32 | bwd_inner: 5684.21 | bwd_allreduce: 63.07 | step: 19.23

 15%|█▍        | 6026/41250 [14:33:51<84:48:49,  8.67s/it]
                                                          
{'loss': 0.0669, 'grad_norm': 2.1447784900665283, 'learning_rate': 3.860328869426975e-05, 'epoch': 1.46}

 15%|█▍        | 6026/41250 [14:33:51<84:48:49,  8.67s/it][2025-04-25 22:31:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:31:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.71 | bwd_microstep: 5721.76 | bwd_inner_microstep: 5709.07 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.65
[2025-04-25 22:31:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.71 | bwd: 5721.77 | bwd_inner: 5709.07 | bwd_allreduce: 12.66 | step: 18.65

 15%|█▍        | 6027/41250 [14:34:00<84:47:02,  8.67s/it]
                                                          
{'loss': 0.379, 'grad_norm': 7.965217590332031, 'learning_rate': 3.860271210270835e-05, 'epoch': 1.46}

 15%|█▍        | 6027/41250 [14:34:00<84:47:02,  8.67s/it][2025-04-25 22:31:43,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 22:31:43,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.01 | bwd_microstep: 5711.46 | bwd_inner_microstep: 5698.86 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.71
[2025-04-25 22:31:43,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.01 | bwd: 5711.47 | bwd_inner: 5698.86 | bwd_allreduce: 12.57 | step: 18.71

 15%|█▍        | 6028/41250 [14:34:08<84:42:14,  8.66s/it]
                                                          
{'loss': 0.0819, 'grad_norm': 1.471078634262085, 'learning_rate': 3.8602135396464874e-05, 'epoch': 1.46}

 15%|█▍        | 6028/41250 [14:34:08<84:42:14,  8.66s/it][2025-04-25 22:31:52,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 22:31:52,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.03 | bwd_microstep: 5753.20 | bwd_inner_microstep: 5652.85 | bwd_allreduce_microstep: 100.31 | step_microstep: 18.57
[2025-04-25 22:31:52,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.03 | bwd: 5753.21 | bwd_inner: 5652.85 | bwd_allreduce: 100.33 | step: 18.57

 15%|█▍        | 6029/41250 [14:34:17<84:43:32,  8.66s/it]
                                                          
{'loss': 0.1149, 'grad_norm': 1.5126733779907227, 'learning_rate': 3.86015585755429e-05, 'epoch': 1.46}

 15%|█▍        | 6029/41250 [14:34:17<84:43:32,  8.66s/it][2025-04-25 22:32:00,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:32:00,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.25 | bwd_microstep: 5683.63 | bwd_inner_microstep: 5671.02 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.63
[2025-04-25 22:32:00,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.26 | bwd: 5683.65 | bwd_inner: 5671.02 | bwd_allreduce: 12.58 | step: 18.63

 15%|█▍        | 6030/41250 [14:34:26<84:33:31,  8.64s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 1.4321759939193726, 'learning_rate': 3.860098163994597e-05, 'epoch': 1.46}

 15%|█▍        | 6030/41250 [14:34:26<84:33:31,  8.64s/it][2025-04-25 22:32:09,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:32:09,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.88 | bwd_microstep: 5739.55 | bwd_inner_microstep: 5655.79 | bwd_allreduce_microstep: 83.72 | step_microstep: 18.44
[2025-04-25 22:32:09,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.88 | bwd: 5739.56 | bwd_inner: 5655.79 | bwd_allreduce: 83.74 | step: 18.44

 15%|█▍        | 6031/41250 [14:34:34<84:34:05,  8.64s/it]
                                                          
{'loss': 0.4045, 'grad_norm': 3.781477689743042, 'learning_rate': 3.8600404589677655e-05, 'epoch': 1.46}

 15%|█▍        | 6031/41250 [14:34:34<84:34:05,  8.64s/it][2025-04-25 22:32:18,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:32:18,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5721.78 | bwd_inner_microstep: 5709.23 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.66
[2025-04-25 22:32:18,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5721.80 | bwd_inner: 5709.23 | bwd_allreduce: 12.52 | step: 18.66

 15%|█▍        | 6032/41250 [14:34:43<84:35:36,  8.65s/it]
                                                          
{'loss': 0.1428, 'grad_norm': 1.0697413682937622, 'learning_rate': 3.8599827424741497e-05, 'epoch': 1.46}

 15%|█▍        | 6032/41250 [14:34:43<84:35:36,  8.65s/it][2025-04-25 22:32:26,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:32:26,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5710.43 | bwd_inner_microstep: 5697.77 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.70
[2025-04-25 22:32:26,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5710.44 | bwd_inner: 5697.77 | bwd_allreduce: 12.64 | step: 18.70

 15%|█▍        | 6033/41250 [14:34:51<84:34:14,  8.65s/it]
                                                          
{'loss': 0.342, 'grad_norm': 1.3906996250152588, 'learning_rate': 3.8599250145141065e-05, 'epoch': 1.46}

 15%|█▍        | 6033/41250 [14:34:51<84:34:14,  8.65s/it][2025-04-25 22:32:35,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:32:35,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.97 | bwd_microstep: 5724.01 | bwd_inner_microstep: 5646.30 | bwd_allreduce_microstep: 77.67 | step_microstep: 18.32
[2025-04-25 22:32:35,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.97 | bwd: 5724.03 | bwd_inner: 5646.30 | bwd_allreduce: 77.68 | step: 18.32

 15%|█▍        | 6034/41250 [14:35:00<84:33:34,  8.64s/it]
                                                          
{'loss': 0.1929, 'grad_norm': 2.4597911834716797, 'learning_rate': 3.859867275087991e-05, 'epoch': 1.46}

 15%|█▍        | 6034/41250 [14:35:00<84:33:34,  8.64s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x111d3b80] moov atom not found
[22:32:36] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4...
[2025-04-25 22:32:43,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:32:43,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.30 | bwd_microstep: 5690.19 | bwd_inner_microstep: 5677.50 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.31
[2025-04-25 22:32:43,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.30 | bwd: 5690.20 | bwd_inner: 5677.50 | bwd_allreduce: 12.66 | step: 18.31

 15%|█▍        | 6035/41250 [14:35:09<84:27:22,  8.63s/it]
                                                          
{'loss': 0.1523, 'grad_norm': 1.336107850074768, 'learning_rate': 3.85980952419616e-05, 'epoch': 1.46}

 15%|█▍        | 6035/41250 [14:35:09<84:27:22,  8.63s/it][2025-04-25 22:32:52,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:32:52,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.42 | bwd_microstep: 5758.52 | bwd_inner_microstep: 5633.42 | bwd_allreduce_microstep: 125.06 | step_microstep: 18.38
[2025-04-25 22:32:52,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.42 | bwd: 5758.53 | bwd_inner: 5633.42 | bwd_allreduce: 125.07 | step: 18.39

 15%|█▍        | 6036/41250 [14:35:17<84:32:47,  8.64s/it]
                                                          
{'loss': 0.0864, 'grad_norm': 1.113574504852295, 'learning_rate': 3.8597517618389696e-05, 'epoch': 1.46}

 15%|█▍        | 6036/41250 [14:35:17<84:32:47,  8.64s/it][2025-04-25 22:33:01,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.95 | optimizer_step: 1.02
[2025-04-25 22:33:01,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.69 | bwd_microstep: 6017.83 | bwd_inner_microstep: 5648.42 | bwd_allreduce_microstep: 369.36 | step_microstep: 18.20
[2025-04-25 22:33:01,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.69 | bwd: 6017.84 | bwd_inner: 5648.42 | bwd_allreduce: 369.38 | step: 18.21

 15%|█▍        | 6037/41250 [14:35:26<85:22:03,  8.73s/it]
                                                          
{'loss': 0.1654, 'grad_norm': 1.218349814414978, 'learning_rate': 3.859693988016775e-05, 'epoch': 1.46}

 15%|█▍        | 6037/41250 [14:35:26<85:22:03,  8.73s/it][2025-04-25 22:33:10,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.94 | optimizer_step: 0.93
[2025-04-25 22:33:10,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.29 | bwd_microstep: 5750.74 | bwd_inner_microstep: 5678.89 | bwd_allreduce_microstep: 71.80 | step_microstep: 18.40
[2025-04-25 22:33:10,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.29 | bwd: 5750.75 | bwd_inner: 5678.89 | bwd_allreduce: 71.82 | step: 18.40

 15%|█▍        | 6038/41250 [14:35:35<85:12:47,  8.71s/it]
                                                          
{'loss': 0.0382, 'grad_norm': 0.44539186358451843, 'learning_rate': 3.859636202729932e-05, 'epoch': 1.46}

 15%|█▍        | 6038/41250 [14:35:35<85:12:47,  8.71s/it][2025-04-25 22:33:18,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:33:18,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5746.02 | bwd_inner_microstep: 5642.96 | bwd_allreduce_microstep: 103.01 | step_microstep: 18.29
[2025-04-25 22:33:18,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5746.03 | bwd_inner: 5642.96 | bwd_allreduce: 103.02 | step: 18.29

 15%|█▍        | 6039/41250 [14:35:44<85:02:33,  8.69s/it]
                                                          
{'loss': 0.1243, 'grad_norm': 2.9664084911346436, 'learning_rate': 3.859578405978797e-05, 'epoch': 1.46}

 15%|█▍        | 6039/41250 [14:35:44<85:02:33,  8.69s/it][2025-04-25 22:33:27,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.24 | optimizer_step: 0.99
[2025-04-25 22:33:27,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.36 | bwd_microstep: 5761.81 | bwd_inner_microstep: 5706.53 | bwd_allreduce_microstep: 55.22 | step_microstep: 19.33
[2025-04-25 22:33:27,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.36 | bwd: 5761.82 | bwd_inner: 5706.53 | bwd_allreduce: 55.24 | step: 19.33

 15%|█▍        | 6040/41250 [14:35:52<85:02:43,  8.70s/it]
                                                          
{'loss': 0.2261, 'grad_norm': 3.1006715297698975, 'learning_rate': 3.859520597763728e-05, 'epoch': 1.46}

 15%|█▍        | 6040/41250 [14:35:52<85:02:43,  8.70s/it][2025-04-25 22:33:36,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 22:33:36,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.31 | bwd_microstep: 5784.50 | bwd_inner_microstep: 5637.79 | bwd_allreduce_microstep: 146.66 | step_microstep: 18.77
[2025-04-25 22:33:36,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.31 | bwd: 5784.51 | bwd_inner: 5637.79 | bwd_allreduce: 146.68 | step: 18.78

 15%|█▍        | 6041/41250 [14:36:01<85:01:38,  8.69s/it]
                                                          
{'loss': 0.1902, 'grad_norm': 2.7475080490112305, 'learning_rate': 3.8594627780850806e-05, 'epoch': 1.46}

 15%|█▍        | 6041/41250 [14:36:01<85:01:38,  8.69s/it][2025-04-25 22:33:44,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:33:44,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.21 | bwd_microstep: 5737.45 | bwd_inner_microstep: 5706.24 | bwd_allreduce_microstep: 31.17 | step_microstep: 18.80
[2025-04-25 22:33:44,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.21 | bwd: 5737.46 | bwd_inner: 5706.24 | bwd_allreduce: 31.19 | step: 18.80

 15%|█▍        | 6042/41250 [14:36:10<84:56:40,  8.69s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 1.3141539096832275, 'learning_rate': 3.859404946943211e-05, 'epoch': 1.46}

 15%|█▍        | 6042/41250 [14:36:10<84:56:40,  8.69s/it][2025-04-25 22:33:53,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:33:53,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.40 | bwd_microstep: 5693.59 | bwd_inner_microstep: 5656.03 | bwd_allreduce_microstep: 37.52 | step_microstep: 18.44
[2025-04-25 22:33:53,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.40 | bwd: 5693.61 | bwd_inner: 5656.03 | bwd_allreduce: 37.54 | step: 18.44

 15%|█▍        | 6043/41250 [14:36:18<84:41:23,  8.66s/it]
                                                          
{'loss': 0.1987, 'grad_norm': 2.712811231613159, 'learning_rate': 3.859347104338475e-05, 'epoch': 1.46}

 15%|█▍        | 6043/41250 [14:36:18<84:41:23,  8.66s/it][2025-04-25 22:34:02,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:34:02,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5769.42 | bwd_inner_microstep: 5652.62 | bwd_allreduce_microstep: 116.76 | step_microstep: 18.51
[2025-04-25 22:34:02,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5769.43 | bwd_inner: 5652.62 | bwd_allreduce: 116.77 | step: 18.51

 15%|█▍        | 6044/41250 [14:36:27<84:44:33,  8.67s/it]
                                                          
{'loss': 0.2535, 'grad_norm': 1.9107818603515625, 'learning_rate': 3.8592892502712305e-05, 'epoch': 1.47}

 15%|█▍        | 6044/41250 [14:36:27<84:44:33,  8.67s/it][2025-04-25 22:34:10,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:34:10,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.16 | bwd_microstep: 5728.44 | bwd_inner_microstep: 5707.54 | bwd_allreduce_microstep: 20.86 | step_microstep: 18.50
[2025-04-25 22:34:10,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.16 | bwd: 5728.45 | bwd_inner: 5707.54 | bwd_allreduce: 20.87 | step: 18.50

 15%|█▍        | 6045/41250 [14:36:36<84:44:08,  8.66s/it]
                                                          
{'loss': 0.0547, 'grad_norm': 0.8077385425567627, 'learning_rate': 3.859231384741833e-05, 'epoch': 1.47}

 15%|█▍        | 6045/41250 [14:36:36<84:44:08,  8.66s/it][2025-04-25 22:34:19,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:34:19,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.28 | bwd_microstep: 5757.34 | bwd_inner_microstep: 5654.13 | bwd_allreduce_microstep: 103.16 | step_microstep: 18.49
[2025-04-25 22:34:19,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.28 | bwd: 5757.35 | bwd_inner: 5654.13 | bwd_allreduce: 103.18 | step: 18.49

 15%|█▍        | 6046/41250 [14:36:44<84:45:46,  8.67s/it]
                                                          
{'loss': 0.2543, 'grad_norm': 3.6806631088256836, 'learning_rate': 3.8591735077506396e-05, 'epoch': 1.47}

 15%|█▍        | 6046/41250 [14:36:44<84:45:46,  8.67s/it][2025-04-25 22:34:28,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:34:28,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.52 | bwd_microstep: 5753.65 | bwd_inner_microstep: 5700.42 | bwd_allreduce_microstep: 53.19 | step_microstep: 18.37
[2025-04-25 22:34:28,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.52 | bwd: 5753.66 | bwd_inner: 5700.42 | bwd_allreduce: 53.20 | step: 18.38

 15%|█▍        | 6047/41250 [14:36:53<84:48:16,  8.67s/it]
                                                          
{'loss': 0.2885, 'grad_norm': 2.596372127532959, 'learning_rate': 3.859115619298007e-05, 'epoch': 1.47}

 15%|█▍        | 6047/41250 [14:36:53<84:48:16,  8.67s/it][2025-04-25 22:34:36,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:34:36,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5777.37 | bwd_inner_microstep: 5655.39 | bwd_allreduce_microstep: 121.95 | step_microstep: 18.30
[2025-04-25 22:34:36,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5777.39 | bwd_inner: 5655.39 | bwd_allreduce: 121.96 | step: 18.30

 15%|█▍        | 6048/41250 [14:37:02<84:49:49,  8.68s/it]
                                                          
{'loss': 0.1754, 'grad_norm': 1.7823270559310913, 'learning_rate': 3.859057719384292e-05, 'epoch': 1.47}

 15%|█▍        | 6048/41250 [14:37:02<84:49:49,  8.68s/it][2025-04-25 22:34:45,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:34:45,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5712.54 | bwd_inner_microstep: 5699.96 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.42
[2025-04-25 22:34:45,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5712.56 | bwd_inner: 5699.96 | bwd_allreduce: 12.55 | step: 18.42

 15%|█▍        | 6049/41250 [14:37:10<84:44:08,  8.67s/it]
                                                          
{'loss': 0.1769, 'grad_norm': 1.8995798826217651, 'learning_rate': 3.858999808009853e-05, 'epoch': 1.47}

 15%|█▍        | 6049/41250 [14:37:10<84:44:08,  8.67s/it][2025-04-25 22:34:54,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:34:54,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.43 | bwd_microstep: 5718.82 | bwd_inner_microstep: 5705.94 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.53
[2025-04-25 22:34:54,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.43 | bwd: 5718.83 | bwd_inner: 5705.94 | bwd_allreduce: 12.86 | step: 18.53

 15%|█▍        | 6050/41250 [14:37:19<84:41:14,  8.66s/it]
                                                          
{'loss': 0.0852, 'grad_norm': 1.1659653186798096, 'learning_rate': 3.858941885175045e-05, 'epoch': 1.47}

 15%|█▍        | 6050/41250 [14:37:19<84:41:14,  8.66s/it][2025-04-25 22:35:02,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.97
[2025-04-25 22:35:02,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5784.78 | bwd_inner_microstep: 5644.57 | bwd_allreduce_microstep: 140.17 | step_microstep: 18.26
[2025-04-25 22:35:02,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5784.79 | bwd_inner: 5644.57 | bwd_allreduce: 140.18 | step: 18.27

 15%|█▍        | 6051/41250 [14:37:28<84:46:51,  8.67s/it]
                                                          
{'loss': 0.1277, 'grad_norm': 1.1411675214767456, 'learning_rate': 3.858883950880226e-05, 'epoch': 1.47}

 15%|█▍        | 6051/41250 [14:37:28<84:46:51,  8.67s/it][2025-04-25 22:35:11,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.07 | optimizer_step: 1.00
[2025-04-25 22:35:11,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.69 | bwd_microstep: 5772.25 | bwd_inner_microstep: 5648.49 | bwd_allreduce_microstep: 123.71 | step_microstep: 19.35
[2025-04-25 22:35:11,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.69 | bwd: 5772.27 | bwd_inner: 5648.49 | bwd_allreduce: 123.73 | step: 19.35

 15%|█▍        | 6052/41250 [14:37:36<84:49:55,  8.68s/it]
                                                          
{'loss': 0.2397, 'grad_norm': 2.151775598526001, 'learning_rate': 3.858826005125753e-05, 'epoch': 1.47}

 15%|█▍        | 6052/41250 [14:37:36<84:49:55,  8.68s/it][2025-04-25 22:35:20,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:35:20,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2905.68 | bwd_microstep: 5798.22 | bwd_inner_microstep: 5785.49 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.45
[2025-04-25 22:35:20,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2905.68 | bwd: 5798.23 | bwd_inner: 5785.49 | bwd_allreduce: 12.71 | step: 18.45

 15%|█▍        | 6053/41250 [14:37:45<85:10:16,  8.71s/it]
                                                          
{'loss': 0.1041, 'grad_norm': 1.1700177192687988, 'learning_rate': 3.858768047911984e-05, 'epoch': 1.47}

 15%|█▍        | 6053/41250 [14:37:45<85:10:16,  8.71s/it][2025-04-25 22:35:28,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 22:35:28,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.12 | bwd_microstep: 5725.10 | bwd_inner_microstep: 5712.39 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.49
[2025-04-25 22:35:28,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.12 | bwd: 5725.11 | bwd_inner: 5712.39 | bwd_allreduce: 12.68 | step: 18.50

 15%|█▍        | 6054/41250 [14:37:54<85:02:06,  8.70s/it]
                                                          
{'loss': 0.1303, 'grad_norm': 1.2000117301940918, 'learning_rate': 3.8587100792392744e-05, 'epoch': 1.47}

 15%|█▍        | 6054/41250 [14:37:54<85:02:06,  8.70s/it][2025-04-25 22:35:37,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:35:37,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.86 | bwd_microstep: 5774.54 | bwd_inner_microstep: 5712.08 | bwd_allreduce_microstep: 62.41 | step_microstep: 18.95
[2025-04-25 22:35:37,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.86 | bwd: 5774.55 | bwd_inner: 5712.08 | bwd_allreduce: 62.43 | step: 18.95

 15%|█▍        | 6055/41250 [14:38:03<85:05:09,  8.70s/it]
                                                          
{'loss': 0.2119, 'grad_norm': 1.4694048166275024, 'learning_rate': 3.858652099107984e-05, 'epoch': 1.47}

 15%|█▍        | 6055/41250 [14:38:03<85:05:09,  8.70s/it][2025-04-25 22:35:46,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 22:35:46,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.24 | bwd_microstep: 5760.24 | bwd_inner_microstep: 5694.14 | bwd_allreduce_microstep: 66.05 | step_microstep: 19.28
[2025-04-25 22:35:46,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.24 | bwd: 5760.26 | bwd_inner: 5694.14 | bwd_allreduce: 66.07 | step: 19.28

 15%|█▍        | 6056/41250 [14:38:11<85:03:46,  8.70s/it]
                                                          
{'loss': 0.2578, 'grad_norm': 2.9190850257873535, 'learning_rate': 3.8585941075184675e-05, 'epoch': 1.47}

 15%|█▍        | 6056/41250 [14:38:11<85:03:46,  8.70s/it][2025-04-25 22:35:55,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:35:55,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.08 | bwd_microstep: 5718.51 | bwd_inner_microstep: 5690.02 | bwd_allreduce_microstep: 28.44 | step_microstep: 19.02
[2025-04-25 22:35:55,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.08 | bwd: 5718.52 | bwd_inner: 5690.02 | bwd_allreduce: 28.46 | step: 19.02

 15%|█▍        | 6057/41250 [14:38:20<84:55:14,  8.69s/it]
                                                          
{'loss': 0.0708, 'grad_norm': 0.7106594443321228, 'learning_rate': 3.8585361044710856e-05, 'epoch': 1.47}

 15%|█▍        | 6057/41250 [14:38:20<84:55:14,  8.69s/it][2025-04-25 22:36:03,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:36:03,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.65 | bwd_microstep: 5720.39 | bwd_inner_microstep: 5671.41 | bwd_allreduce_microstep: 48.93 | step_microstep: 19.23
[2025-04-25 22:36:03,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.65 | bwd: 5720.40 | bwd_inner: 5671.41 | bwd_allreduce: 48.95 | step: 19.23

 15%|█▍        | 6058/41250 [14:38:29<84:46:41,  8.67s/it]
                                                          
{'loss': 0.1476, 'grad_norm': 0.9464519619941711, 'learning_rate': 3.858478089966193e-05, 'epoch': 1.47}

 15%|█▍        | 6058/41250 [14:38:29<84:46:41,  8.67s/it][2025-04-25 22:36:12,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:36:12,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.12 | bwd_microstep: 5780.46 | bwd_inner_microstep: 5652.72 | bwd_allreduce_microstep: 127.69 | step_microstep: 18.55
[2025-04-25 22:36:12,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.12 | bwd: 5780.48 | bwd_inner: 5652.72 | bwd_allreduce: 127.71 | step: 18.56

 15%|█▍        | 6059/41250 [14:38:37<84:50:40,  8.68s/it]
                                                          
{'loss': 0.2547, 'grad_norm': 3.0924010276794434, 'learning_rate': 3.858420064004149e-05, 'epoch': 1.47}

 15%|█▍        | 6059/41250 [14:38:37<84:50:40,  8.68s/it][2025-04-25 22:36:21,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:36:21,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.23 | bwd_microstep: 5772.10 | bwd_inner_microstep: 5699.12 | bwd_allreduce_microstep: 72.93 | step_microstep: 18.70
[2025-04-25 22:36:21,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.23 | bwd: 5772.11 | bwd_inner: 5699.12 | bwd_allreduce: 72.95 | step: 18.71

 15%|█▍        | 6060/41250 [14:38:46<84:54:09,  8.69s/it]
                                                          
{'loss': 0.2941, 'grad_norm': 2.818307876586914, 'learning_rate': 3.8583620265853104e-05, 'epoch': 1.47}

 15%|█▍        | 6060/41250 [14:38:46<84:54:09,  8.69s/it][2025-04-25 22:36:29,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:36:29,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5784.01 | bwd_inner_microstep: 5659.09 | bwd_allreduce_microstep: 124.88 | step_microstep: 18.44
[2025-04-25 22:36:29,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5784.03 | bwd_inner: 5659.09 | bwd_allreduce: 124.90 | step: 18.44

 15%|█▍        | 6061/41250 [14:38:55<84:55:13,  8.69s/it]
                                                          
{'loss': 0.1607, 'grad_norm': 3.5408012866973877, 'learning_rate': 3.8583039777100355e-05, 'epoch': 1.47}

 15%|█▍        | 6061/41250 [14:38:55<84:55:13,  8.69s/it][2025-04-25 22:36:38,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 22:36:38,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5721.79 | bwd_inner_microstep: 5660.24 | bwd_allreduce_microstep: 61.51 | step_microstep: 18.53
[2025-04-25 22:36:38,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.45 | bwd: 5721.81 | bwd_inner: 5660.24 | bwd_allreduce: 61.53 | step: 18.53

 15%|█▍        | 6062/41250 [14:39:03<84:45:55,  8.67s/it]
                                                          
{'loss': 0.3557, 'grad_norm': 3.9034135341644287, 'learning_rate': 3.858245917378683e-05, 'epoch': 1.47}

 15%|█▍        | 6062/41250 [14:39:03<84:45:55,  8.67s/it][2025-04-25 22:36:47,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.34 | optimizer_step: 1.05
[2025-04-25 22:36:47,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.41 | bwd_microstep: 5755.87 | bwd_inner_microstep: 5712.07 | bwd_allreduce_microstep: 43.73 | step_microstep: 20.32
[2025-04-25 22:36:47,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.41 | bwd: 5755.88 | bwd_inner: 5712.07 | bwd_allreduce: 43.76 | step: 20.32

 15%|█▍        | 6063/41250 [14:39:12<84:50:58,  8.68s/it]
                                                          
{'loss': 0.0898, 'grad_norm': 1.1424237489700317, 'learning_rate': 3.8581878455916084e-05, 'epoch': 1.47}

 15%|█▍        | 6063/41250 [14:39:12<84:50:58,  8.68s/it][2025-04-25 22:36:55,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 22:36:55,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5721.64 | bwd_inner_microstep: 5657.15 | bwd_allreduce_microstep: 64.44 | step_microstep: 18.93
[2025-04-25 22:36:55,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.48 | bwd: 5721.66 | bwd_inner: 5657.15 | bwd_allreduce: 64.46 | step: 18.94

 15%|█▍        | 6064/41250 [14:39:21<84:42:00,  8.67s/it]
                                                          
{'loss': 0.128, 'grad_norm': 1.6786420345306396, 'learning_rate': 3.8581297623491724e-05, 'epoch': 1.47}

 15%|█▍        | 6064/41250 [14:39:21<84:42:00,  8.67s/it][2025-04-25 22:37:04,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-25 22:37:04,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.63 | bwd_microstep: 5723.34 | bwd_inner_microstep: 5710.11 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.37
[2025-04-25 22:37:04,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.63 | bwd: 5723.35 | bwd_inner: 5710.11 | bwd_allreduce: 13.20 | step: 19.37

 15%|█▍        | 6065/41250 [14:39:29<84:40:32,  8.66s/it]
                                                          
{'loss': 0.0617, 'grad_norm': 1.8239099979400635, 'learning_rate': 3.858071667651731e-05, 'epoch': 1.47}

 15%|█▍        | 6065/41250 [14:39:29<84:40:32,  8.66s/it][2025-04-25 22:37:13,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.04 | optimizer_step: 1.08
[2025-04-25 22:37:13,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.51 | bwd_microstep: 5747.13 | bwd_inner_microstep: 5686.79 | bwd_allreduce_microstep: 60.30 | step_microstep: 18.70
[2025-04-25 22:37:13,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.51 | bwd: 5747.15 | bwd_inner: 5686.79 | bwd_allreduce: 60.31 | step: 18.70

 15%|█▍        | 6066/41250 [14:39:38<84:42:16,  8.67s/it]
                                                          
{'loss': 0.1621, 'grad_norm': 1.8253917694091797, 'learning_rate': 3.858013561499644e-05, 'epoch': 1.47}

 15%|█▍        | 6066/41250 [14:39:38<84:42:16,  8.67s/it][2025-04-25 22:37:21,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:37:21,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.87 | bwd_microstep: 5770.37 | bwd_inner_microstep: 5658.90 | bwd_allreduce_microstep: 111.43 | step_microstep: 18.62
[2025-04-25 22:37:21,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.87 | bwd: 5770.39 | bwd_inner: 5658.90 | bwd_allreduce: 111.44 | step: 18.62

 15%|█▍        | 6067/41250 [14:39:47<84:45:16,  8.67s/it]
                                                          
{'loss': 0.0482, 'grad_norm': 0.9281389713287354, 'learning_rate': 3.857955443893269e-05, 'epoch': 1.47}

 15%|█▍        | 6067/41250 [14:39:47<84:45:16,  8.67s/it][2025-04-25 22:37:30,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 22:37:30,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.99 | bwd_microstep: 5720.71 | bwd_inner_microstep: 5658.36 | bwd_allreduce_microstep: 62.30 | step_microstep: 18.74
[2025-04-25 22:37:30,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.99 | bwd: 5720.73 | bwd_inner: 5658.36 | bwd_allreduce: 62.32 | step: 18.74

 15%|█▍        | 6068/41250 [14:39:55<84:38:00,  8.66s/it]
                                                          
{'loss': 0.0365, 'grad_norm': 0.5670573711395264, 'learning_rate': 3.8578973148329635e-05, 'epoch': 1.47}

 15%|█▍        | 6068/41250 [14:39:55<84:38:00,  8.66s/it][2025-04-25 22:37:39,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:37:39,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5793.54 | bwd_inner_microstep: 5660.38 | bwd_allreduce_microstep: 133.11 | step_microstep: 18.68
[2025-04-25 22:37:39,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5793.56 | bwd_inner: 5660.38 | bwd_allreduce: 133.13 | step: 18.68

 15%|█▍        | 6069/41250 [14:40:04<84:45:50,  8.67s/it]
                                                          
{'loss': 0.1937, 'grad_norm': 1.8588240146636963, 'learning_rate': 3.857839174319087e-05, 'epoch': 1.47}

 15%|█▍        | 6069/41250 [14:40:04<84:45:50,  8.67s/it][2025-04-25 22:37:47,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.05 | optimizer_step: 1.11
[2025-04-25 22:37:47,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.31 | bwd_microstep: 5776.95 | bwd_inner_microstep: 5661.55 | bwd_allreduce_microstep: 115.35 | step_microstep: 19.29
[2025-04-25 22:37:47,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.31 | bwd: 5776.96 | bwd_inner: 5661.55 | bwd_allreduce: 115.37 | step: 19.29

 15%|█▍        | 6070/41250 [14:40:13<84:48:27,  8.68s/it]
                                                          
{'loss': 0.1793, 'grad_norm': 1.4817287921905518, 'learning_rate': 3.857781022351997e-05, 'epoch': 1.47}

 15%|█▍        | 6070/41250 [14:40:13<84:48:27,  8.68s/it][2025-04-25 22:37:56,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.28 | optimizer_step: 0.94
[2025-04-25 22:37:56,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.88 | bwd_microstep: 5758.06 | bwd_inner_microstep: 5658.44 | bwd_allreduce_microstep: 99.57 | step_microstep: 19.45
[2025-04-25 22:37:56,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.88 | bwd: 5758.07 | bwd_inner: 5658.43 | bwd_allreduce: 99.59 | step: 19.45

 15%|█▍        | 6071/41250 [14:40:21<84:47:20,  8.68s/it]
                                                          
{'loss': 0.1491, 'grad_norm': 1.4356129169464111, 'learning_rate': 3.857722858932052e-05, 'epoch': 1.47}

 15%|█▍        | 6071/41250 [14:40:21<84:47:20,  8.68s/it][2025-04-25 22:38:05,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.18 | optimizer_step: 0.91
[2025-04-25 22:38:05,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.43 | bwd_microstep: 5778.81 | bwd_inner_microstep: 5645.84 | bwd_allreduce_microstep: 132.93 | step_microstep: 18.76
[2025-04-25 22:38:05,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.43 | bwd: 5778.83 | bwd_inner: 5645.84 | bwd_allreduce: 132.95 | step: 18.77

 15%|█▍        | 6072/41250 [14:40:30<84:48:43,  8.68s/it]
                                                          
{'loss': 0.1335, 'grad_norm': 2.444269895553589, 'learning_rate': 3.857664684059612e-05, 'epoch': 1.47}

 15%|█▍        | 6072/41250 [14:40:30<84:48:43,  8.68s/it][2025-04-25 22:38:13,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.12 | optimizer_step: 0.91
[2025-04-25 22:38:13,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.79 | bwd_microstep: 5729.94 | bwd_inner_microstep: 5699.29 | bwd_allreduce_microstep: 30.61 | step_microstep: 18.80
[2025-04-25 22:38:13,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.79 | bwd: 5729.96 | bwd_inner: 5699.29 | bwd_allreduce: 30.63 | step: 18.80

 15%|█▍        | 6073/41250 [14:40:39<84:46:20,  8.68s/it]
                                                          
{'loss': 0.1738, 'grad_norm': 3.2738027572631836, 'learning_rate': 3.8576064977350335e-05, 'epoch': 1.47}

 15%|█▍        | 6073/41250 [14:40:39<84:46:20,  8.68s/it][2025-04-25 22:38:22,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:38:22,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.99 | bwd_microstep: 5715.50 | bwd_inner_microstep: 5703.15 | bwd_allreduce_microstep: 12.31 | step_microstep: 18.77
[2025-04-25 22:38:22,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.99 | bwd: 5715.52 | bwd_inner: 5703.15 | bwd_allreduce: 12.33 | step: 18.78

 15%|█▍        | 6074/41250 [14:40:47<84:43:24,  8.67s/it]
                                                          
{'loss': 0.193, 'grad_norm': 2.668656349182129, 'learning_rate': 3.8575482999586774e-05, 'epoch': 1.47}

 15%|█▍        | 6074/41250 [14:40:47<84:43:24,  8.67s/it][2025-04-25 22:38:31,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:38:31,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.54 | bwd_microstep: 5695.01 | bwd_inner_microstep: 5637.06 | bwd_allreduce_microstep: 57.90 | step_microstep: 18.76
[2025-04-25 22:38:31,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.54 | bwd: 5695.02 | bwd_inner: 5637.06 | bwd_allreduce: 57.92 | step: 18.76

 15%|█▍        | 6075/41250 [14:40:56<84:31:37,  8.65s/it]
                                                          
{'loss': 0.0554, 'grad_norm': 0.7869809865951538, 'learning_rate': 3.857490090730901e-05, 'epoch': 1.47}

 15%|█▍        | 6075/41250 [14:40:56<84:31:37,  8.65s/it][2025-04-25 22:38:39,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 22:38:39,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.17 | bwd_microstep: 5751.72 | bwd_inner_microstep: 5650.42 | bwd_allreduce_microstep: 101.25 | step_microstep: 19.19
[2025-04-25 22:38:39,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.17 | bwd: 5751.74 | bwd_inner: 5650.42 | bwd_allreduce: 101.27 | step: 19.19

 15%|█▍        | 6076/41250 [14:41:05<84:32:20,  8.65s/it]
                                                          
{'loss': 0.0821, 'grad_norm': 0.7256941199302673, 'learning_rate': 3.857431870052063e-05, 'epoch': 1.47}

 15%|█▍        | 6076/41250 [14:41:05<84:32:20,  8.65s/it][2025-04-25 22:38:48,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-25 22:38:48,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.23 | bwd_microstep: 5778.90 | bwd_inner_microstep: 5765.61 | bwd_allreduce_microstep: 13.25 | step_microstep: 18.97
[2025-04-25 22:38:48,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.23 | bwd: 5778.92 | bwd_inner: 5765.60 | bwd_allreduce: 13.27 | step: 18.97

 15%|█▍        | 6077/41250 [14:41:13<84:51:10,  8.68s/it]
                                                          
{'loss': 0.1527, 'grad_norm': 1.6713612079620361, 'learning_rate': 3.8573736379225234e-05, 'epoch': 1.47}

 15%|█▍        | 6077/41250 [14:41:13<84:51:10,  8.68s/it][2025-04-25 22:38:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:38:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.52 | bwd_microstep: 5694.98 | bwd_inner_microstep: 5649.30 | bwd_allreduce_microstep: 45.64 | step_microstep: 18.51
[2025-04-25 22:38:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.52 | bwd: 5695.00 | bwd_inner: 5649.30 | bwd_allreduce: 45.65 | step: 18.51

 15%|█▍        | 6078/41250 [14:41:22<84:37:36,  8.66s/it]
                                                          
{'loss': 0.2342, 'grad_norm': 1.2728734016418457, 'learning_rate': 3.8573153943426405e-05, 'epoch': 1.47}

 15%|█▍        | 6078/41250 [14:41:22<84:37:36,  8.66s/it][2025-04-25 22:39:05,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:39:05,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.57 | bwd_microstep: 5732.86 | bwd_inner_microstep: 5702.53 | bwd_allreduce_microstep: 30.29 | step_microstep: 18.43
[2025-04-25 22:39:05,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.57 | bwd: 5732.88 | bwd_inner: 5702.53 | bwd_allreduce: 30.31 | step: 18.43

 15%|█▍        | 6079/41250 [14:41:31<84:39:07,  8.66s/it]
                                                          
{'loss': 0.16, 'grad_norm': 2.0672128200531006, 'learning_rate': 3.8572571393127735e-05, 'epoch': 1.47}

 15%|█▍        | 6079/41250 [14:41:31<84:39:07,  8.66s/it][2025-04-25 22:39:14,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:39:14,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.03 | bwd_microstep: 5675.21 | bwd_inner_microstep: 5643.76 | bwd_allreduce_microstep: 31.41 | step_microstep: 18.59
[2025-04-25 22:39:14,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.03 | bwd: 5675.22 | bwd_inner: 5643.76 | bwd_allreduce: 31.42 | step: 18.59

 15%|█▍        | 6080/41250 [14:41:39<84:27:01,  8.64s/it]
                                                          
{'loss': 0.2636, 'grad_norm': 2.1917002201080322, 'learning_rate': 3.8571988728332816e-05, 'epoch': 1.47}

 15%|█▍        | 6080/41250 [14:41:39<84:27:01,  8.64s/it][2025-04-25 22:39:23,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:39:23,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5696.07 | bwd_inner_microstep: 5683.28 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.68
[2025-04-25 22:39:23,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5696.08 | bwd_inner: 5683.28 | bwd_allreduce: 12.76 | step: 18.68

 15%|█▍        | 6081/41250 [14:41:48<84:24:07,  8.64s/it]
                                                          
{'loss': 0.201, 'grad_norm': 3.958906650543213, 'learning_rate': 3.857140594904525e-05, 'epoch': 1.47}

 15%|█▍        | 6081/41250 [14:41:48<84:24:07,  8.64s/it][2025-04-25 22:39:31,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 22:39:31,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5689.98 | bwd_inner_microstep: 5643.94 | bwd_allreduce_microstep: 46.00 | step_microstep: 19.02
[2025-04-25 22:39:31,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5690.00 | bwd_inner: 5643.94 | bwd_allreduce: 46.01 | step: 19.02

 15%|█▍        | 6082/41250 [14:41:56<84:16:36,  8.63s/it]
                                                          
{'loss': 0.1709, 'grad_norm': 1.649725079536438, 'learning_rate': 3.8570823055268605e-05, 'epoch': 1.47}

 15%|█▍        | 6082/41250 [14:41:56<84:16:36,  8.63s/it][2025-04-25 22:39:40,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 22:39:40,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5688.23 | bwd_inner_microstep: 5648.63 | bwd_allreduce_microstep: 39.56 | step_microstep: 18.62
[2025-04-25 22:39:40,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5688.24 | bwd_inner: 5648.63 | bwd_allreduce: 39.57 | step: 18.62

 15%|█▍        | 6083/41250 [14:42:05<84:12:22,  8.62s/it]
                                                          
{'loss': 0.1508, 'grad_norm': 2.7458348274230957, 'learning_rate': 3.857024004700649e-05, 'epoch': 1.47}

 15%|█▍        | 6083/41250 [14:42:05<84:12:22,  8.62s/it][2025-04-25 22:39:48,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:39:48,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.01 | bwd_microstep: 5767.70 | bwd_inner_microstep: 5633.02 | bwd_allreduce_microstep: 134.64 | step_microstep: 18.43
[2025-04-25 22:39:48,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.01 | bwd: 5767.71 | bwd_inner: 5633.02 | bwd_allreduce: 134.65 | step: 18.43

 15%|█▍        | 6084/41250 [14:42:14<84:23:07,  8.64s/it]
                                                          
{'loss': 0.0374, 'grad_norm': 1.0035048723220825, 'learning_rate': 3.8569656924262495e-05, 'epoch': 1.47}

 15%|█▍        | 6084/41250 [14:42:14<84:23:07,  8.64s/it][2025-04-25 22:39:57,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 22:39:57,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.62 | bwd_microstep: 5727.29 | bwd_inner_microstep: 5683.32 | bwd_allreduce_microstep: 43.93 | step_microstep: 18.50
[2025-04-25 22:39:57,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.62 | bwd: 5727.31 | bwd_inner: 5683.32 | bwd_allreduce: 43.95 | step: 18.50

 15%|█▍        | 6085/41250 [14:42:22<84:27:47,  8.65s/it]
                                                          
{'loss': 0.1282, 'grad_norm': 3.2309632301330566, 'learning_rate': 3.8569073687040224e-05, 'epoch': 1.48}

 15%|█▍        | 6085/41250 [14:42:22<84:27:47,  8.65s/it][2025-04-25 22:40:06,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:40:06,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5721.28 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 21.54 | step_microstep: 18.45
[2025-04-25 22:40:06,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5721.29 | bwd_inner: 5699.69 | bwd_allreduce: 21.56 | step: 18.45

 15%|█▍        | 6086/41250 [14:42:31<84:28:30,  8.65s/it]
                                                          
{'loss': 0.1492, 'grad_norm': 2.697903871536255, 'learning_rate': 3.8568490335343266e-05, 'epoch': 1.48}

 15%|█▍        | 6086/41250 [14:42:31<84:28:30,  8.65s/it][2025-04-25 22:40:14,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:40:14,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.66 | bwd_microstep: 5844.05 | bwd_inner_microstep: 5644.69 | bwd_allreduce_microstep: 199.32 | step_microstep: 18.47
[2025-04-25 22:40:14,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.66 | bwd: 5844.07 | bwd_inner: 5644.69 | bwd_allreduce: 199.34 | step: 18.48

 15%|█▍        | 6087/41250 [14:42:40<84:47:48,  8.68s/it]
                                                          
{'loss': 0.0581, 'grad_norm': 1.035782814025879, 'learning_rate': 3.856790686917522e-05, 'epoch': 1.48}

 15%|█▍        | 6087/41250 [14:42:40<84:47:48,  8.68s/it][2025-04-25 22:40:23,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:40:23,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5718.73 | bwd_inner_microstep: 5683.32 | bwd_allreduce_microstep: 35.36 | step_microstep: 18.43
[2025-04-25 22:40:23,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5718.74 | bwd_inner: 5683.32 | bwd_allreduce: 35.37 | step: 18.43

 15%|█▍        | 6088/41250 [14:42:48<84:42:37,  8.67s/it]
                                                          
{'loss': 0.1835, 'grad_norm': 2.374643564224243, 'learning_rate': 3.856732328853967e-05, 'epoch': 1.48}

 15%|█▍        | 6088/41250 [14:42:48<84:42:37,  8.67s/it][2025-04-25 22:40:32,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 22:40:32,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.32 | bwd_microstep: 5684.77 | bwd_inner_microstep: 5657.82 | bwd_allreduce_microstep: 26.90 | step_microstep: 19.61
[2025-04-25 22:40:32,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.32 | bwd: 5684.78 | bwd_inner: 5657.82 | bwd_allreduce: 26.92 | step: 19.62

 15%|█▍        | 6089/41250 [14:42:57<84:32:16,  8.66s/it]
                                                          
{'loss': 0.1522, 'grad_norm': 3.0391266345977783, 'learning_rate': 3.856673959344023e-05, 'epoch': 1.48}

 15%|█▍        | 6089/41250 [14:42:57<84:32:16,  8.66s/it][2025-04-25 22:40:40,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 22:40:40,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.79 | bwd_microstep: 5767.73 | bwd_inner_microstep: 5650.25 | bwd_allreduce_microstep: 117.42 | step_microstep: 18.99
[2025-04-25 22:40:40,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.79 | bwd: 5767.74 | bwd_inner: 5650.25 | bwd_allreduce: 117.44 | step: 18.99

 15%|█▍        | 6090/41250 [14:43:06<84:36:22,  8.66s/it]
                                                          
{'loss': 0.1691, 'grad_norm': 3.3485734462738037, 'learning_rate': 3.856615578388049e-05, 'epoch': 1.48}

 15%|█▍        | 6090/41250 [14:43:06<84:36:22,  8.66s/it][2025-04-25 22:40:49,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:40:49,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.14 | bwd_microstep: 5767.48 | bwd_inner_microstep: 5662.74 | bwd_allreduce_microstep: 104.69 | step_microstep: 18.81
[2025-04-25 22:40:49,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.14 | bwd: 5767.49 | bwd_inner: 5662.74 | bwd_allreduce: 104.71 | step: 18.81

 15%|█▍        | 6091/41250 [14:43:14<84:38:32,  8.67s/it]
                                                          
{'loss': 0.145, 'grad_norm': 1.572174310684204, 'learning_rate': 3.8565571859864056e-05, 'epoch': 1.48}

 15%|█▍        | 6091/41250 [14:43:14<84:38:32,  8.67s/it][2025-04-25 22:40:58,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:40:58,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 6046.41 | bwd_inner_microstep: 5649.51 | bwd_allreduce_microstep: 396.85 | step_microstep: 18.53
[2025-04-25 22:40:58,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 6046.42 | bwd_inner: 5649.51 | bwd_allreduce: 396.86 | step: 18.53

 15%|█▍        | 6092/41250 [14:43:23<85:29:33,  8.75s/it]
                                                          
{'loss': 0.1116, 'grad_norm': 2.338496208190918, 'learning_rate': 3.856498782139451e-05, 'epoch': 1.48}

 15%|█▍        | 6092/41250 [14:43:23<85:29:33,  8.75s/it][2025-04-25 22:41:07,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:41:07,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.95 | bwd_microstep: 5705.03 | bwd_inner_microstep: 5692.35 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.78
[2025-04-25 22:41:07,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.95 | bwd: 5705.05 | bwd_inner: 5692.35 | bwd_allreduce: 12.65 | step: 18.79

 15%|█▍        | 6093/41250 [14:43:32<85:09:28,  8.72s/it]
                                                          
{'loss': 0.0723, 'grad_norm': 1.5163829326629639, 'learning_rate': 3.856440366847548e-05, 'epoch': 1.48}

 15%|█▍        | 6093/41250 [14:43:32<85:09:28,  8.72s/it][2025-04-25 22:41:15,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-25 22:41:15,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.05 | bwd_microstep: 5759.75 | bwd_inner_microstep: 5701.10 | bwd_allreduce_microstep: 58.60 | step_microstep: 18.22
[2025-04-25 22:41:15,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.05 | bwd: 5759.76 | bwd_inner: 5701.10 | bwd_allreduce: 58.62 | step: 18.22

 15%|█▍        | 6094/41250 [14:43:41<85:04:34,  8.71s/it]
                                                          
{'loss': 0.1879, 'grad_norm': 3.0695433616638184, 'learning_rate': 3.856381940111055e-05, 'epoch': 1.48}

 15%|█▍        | 6094/41250 [14:43:41<85:04:34,  8.71s/it][2025-04-25 22:41:24,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:41:24,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5883.84 | bwd_inner_microstep: 5715.89 | bwd_allreduce_microstep: 167.91 | step_microstep: 18.44
[2025-04-25 22:41:24,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5883.85 | bwd_inner: 5715.89 | bwd_allreduce: 167.92 | step: 18.44

 15%|█▍        | 6095/41250 [14:43:50<85:24:09,  8.75s/it]
                                                          
{'loss': 0.0202, 'grad_norm': 0.48207953572273254, 'learning_rate': 3.856323501930333e-05, 'epoch': 1.48}

 15%|█▍        | 6095/41250 [14:43:50<85:24:09,  8.75s/it][2025-04-25 22:41:33,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:41:33,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.86 | bwd_microstep: 5770.02 | bwd_inner_microstep: 5699.12 | bwd_allreduce_microstep: 70.86 | step_microstep: 18.38
[2025-04-25 22:41:33,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.86 | bwd: 5770.04 | bwd_inner: 5699.12 | bwd_allreduce: 70.87 | step: 18.38

 15%|█▍        | 6096/41250 [14:43:58<85:16:12,  8.73s/it]
                                                          
{'loss': 0.1408, 'grad_norm': 1.3682284355163574, 'learning_rate': 3.856265052305741e-05, 'epoch': 1.48}

 15%|█▍        | 6096/41250 [14:43:58<85:16:12,  8.73s/it][2025-04-25 22:41:42,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:41:42,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5791.61 | bwd_inner_microstep: 5665.66 | bwd_allreduce_microstep: 125.90 | step_microstep: 18.45
[2025-04-25 22:41:42,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5791.62 | bwd_inner: 5665.66 | bwd_allreduce: 125.92 | step: 18.45

 15%|█▍        | 6097/41250 [14:44:07<85:11:23,  8.72s/it]
                                                          
{'loss': 0.038, 'grad_norm': 0.5344656109809875, 'learning_rate': 3.85620659123764e-05, 'epoch': 1.48}

 15%|█▍        | 6097/41250 [14:44:07<85:11:23,  8.72s/it][2025-04-25 22:41:50,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 22:41:50,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.92 | bwd_microstep: 5702.04 | bwd_inner_microstep: 5659.54 | bwd_allreduce_microstep: 42.47 | step_microstep: 18.79
[2025-04-25 22:41:50,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.92 | bwd: 5702.06 | bwd_inner: 5659.54 | bwd_allreduce: 42.48 | step: 18.78

 15%|█▍        | 6098/41250 [14:44:16<84:52:41,  8.69s/it]
                                                          
{'loss': 0.3558, 'grad_norm': 1.6231334209442139, 'learning_rate': 3.85614811872639e-05, 'epoch': 1.48}

 15%|█▍        | 6098/41250 [14:44:16<84:52:41,  8.69s/it][2025-04-25 22:41:59,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.23 | optimizer_step: 0.92
[2025-04-25 22:41:59,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.57 | bwd_microstep: 5754.65 | bwd_inner_microstep: 5694.78 | bwd_allreduce_microstep: 59.83 | step_microstep: 19.45
[2025-04-25 22:41:59,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.57 | bwd: 5754.67 | bwd_inner: 5694.78 | bwd_allreduce: 59.85 | step: 19.45

 15%|█▍        | 6099/41250 [14:44:24<84:53:11,  8.69s/it]
                                                          
{'loss': 0.384, 'grad_norm': 3.0569567680358887, 'learning_rate': 3.856089634772353e-05, 'epoch': 1.48}

 15%|█▍        | 6099/41250 [14:44:24<84:53:11,  8.69s/it][2025-04-25 22:42:08,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:42:08,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.13 | bwd_microstep: 5763.85 | bwd_inner_microstep: 5691.10 | bwd_allreduce_microstep: 72.71 | step_microstep: 18.29
[2025-04-25 22:42:08,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.13 | bwd: 5763.86 | bwd_inner: 5691.10 | bwd_allreduce: 72.72 | step: 18.29

 15%|█▍        | 6100/41250 [14:44:33<84:52:44,  8.69s/it]
                                                          
{'loss': 0.2871, 'grad_norm': 2.9677653312683105, 'learning_rate': 3.856031139375888e-05, 'epoch': 1.48}

 15%|█▍        | 6100/41250 [14:44:33<84:52:44,  8.69s/it][2025-04-25 22:42:16,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.21 | optimizer_step: 0.94
[2025-04-25 22:42:16,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.35 | bwd_microstep: 5718.22 | bwd_inner_microstep: 5705.59 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.76
[2025-04-25 22:42:16,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.35 | bwd: 5718.24 | bwd_inner: 5705.59 | bwd_allreduce: 12.61 | step: 18.77

 15%|█▍        | 6101/41250 [14:44:42<84:46:54,  8.68s/it]
                                                          
{'loss': 0.1104, 'grad_norm': 1.8857102394104004, 'learning_rate': 3.855972632537356e-05, 'epoch': 1.48}

 15%|█▍        | 6101/41250 [14:44:42<84:46:54,  8.68s/it][2025-04-25 22:42:25,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.15 | optimizer_step: 1.07
[2025-04-25 22:42:25,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2944.25 | bwd_microstep: 5890.13 | bwd_inner_microstep: 5876.16 | bwd_allreduce_microstep: 13.91 | step_microstep: 20.43
[2025-04-25 22:42:25,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2944.25 | bwd: 5890.14 | bwd_inner: 5876.16 | bwd_allreduce: 13.94 | step: 20.43

 15%|█▍        | 6102/41250 [14:44:51<85:28:21,  8.75s/it]
                                                          
{'loss': 0.2015, 'grad_norm': 5.8595075607299805, 'learning_rate': 3.855914114257118e-05, 'epoch': 1.48}

 15%|█▍        | 6102/41250 [14:44:51<85:28:21,  8.75s/it][2025-04-25 22:42:34,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:42:34,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.19 | bwd_microstep: 5756.95 | bwd_inner_microstep: 5722.10 | bwd_allreduce_microstep: 34.80 | step_microstep: 18.92
[2025-04-25 22:42:34,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.19 | bwd: 5756.97 | bwd_inner: 5722.10 | bwd_allreduce: 34.82 | step: 18.92

 15%|█▍        | 6103/41250 [14:44:59<85:18:03,  8.74s/it]
                                                          
{'loss': 0.0304, 'grad_norm': 0.7226480841636658, 'learning_rate': 3.855855584535534e-05, 'epoch': 1.48}

 15%|█▍        | 6103/41250 [14:44:59<85:18:03,  8.74s/it][2025-04-25 22:42:43,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:42:43,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.12 | bwd_microstep: 5781.94 | bwd_inner_microstep: 5665.79 | bwd_allreduce_microstep: 116.10 | step_microstep: 18.98
[2025-04-25 22:42:43,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.12 | bwd: 5781.95 | bwd_inner: 5665.79 | bwd_allreduce: 116.12 | step: 18.99

 15%|█▍        | 6104/41250 [14:45:08<85:14:13,  8.73s/it]
                                                          
{'loss': 0.465, 'grad_norm': 6.394726753234863, 'learning_rate': 3.855797043372966e-05, 'epoch': 1.48}

 15%|█▍        | 6104/41250 [14:45:08<85:14:13,  8.73s/it][2025-04-25 22:42:51,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 22:42:51,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.29 | bwd_microstep: 5767.22 | bwd_inner_microstep: 5690.96 | bwd_allreduce_microstep: 76.21 | step_microstep: 18.72
[2025-04-25 22:42:51,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.29 | bwd: 5767.23 | bwd_inner: 5690.96 | bwd_allreduce: 76.23 | step: 18.73

 15%|█▍        | 6105/41250 [14:45:17<85:09:41,  8.72s/it]
                                                          
{'loss': 0.195, 'grad_norm': 1.3097726106643677, 'learning_rate': 3.855738490769774e-05, 'epoch': 1.48}

 15%|█▍        | 6105/41250 [14:45:17<85:09:41,  8.72s/it][2025-04-25 22:43:00,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 1.00
[2025-04-25 22:43:00,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.70 | bwd_microstep: 5765.32 | bwd_inner_microstep: 5700.05 | bwd_allreduce_microstep: 65.23 | step_microstep: 18.52
[2025-04-25 22:43:00,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.70 | bwd: 5765.33 | bwd_inner: 5700.05 | bwd_allreduce: 65.25 | step: 18.52

 15%|█▍        | 6106/41250 [14:45:25<85:04:36,  8.71s/it]
                                                          
{'loss': 0.1152, 'grad_norm': 4.100393295288086, 'learning_rate': 3.8556799267263195e-05, 'epoch': 1.48}

 15%|█▍        | 6106/41250 [14:45:25<85:04:36,  8.71s/it][2025-04-25 22:43:09,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:43:09,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.66 | bwd_microstep: 5823.64 | bwd_inner_microstep: 5661.29 | bwd_allreduce_microstep: 162.31 | step_microstep: 18.47
[2025-04-25 22:43:09,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.66 | bwd: 5823.65 | bwd_inner: 5661.29 | bwd_allreduce: 162.32 | step: 18.47

 15%|█▍        | 6107/41250 [14:45:34<85:09:07,  8.72s/it]
                                                          
{'loss': 0.1244, 'grad_norm': 2.3923091888427734, 'learning_rate': 3.855621351242963e-05, 'epoch': 1.48}

 15%|█▍        | 6107/41250 [14:45:34<85:09:07,  8.72s/it][2025-04-25 22:43:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-25 22:43:17,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.97 | bwd_microstep: 5768.31 | bwd_inner_microstep: 5707.89 | bwd_allreduce_microstep: 60.38 | step_microstep: 19.10
[2025-04-25 22:43:17,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.97 | bwd: 5768.32 | bwd_inner: 5707.89 | bwd_allreduce: 60.39 | step: 19.10

 15%|█▍        | 6108/41250 [14:45:43<85:07:38,  8.72s/it]
                                                          
{'loss': 0.0737, 'grad_norm': 1.604665994644165, 'learning_rate': 3.855562764320065e-05, 'epoch': 1.48}

 15%|█▍        | 6108/41250 [14:45:43<85:07:38,  8.72s/it][2025-04-25 22:43:26,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 22:43:26,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.87 | bwd_microstep: 5795.05 | bwd_inner_microstep: 5782.17 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.18
[2025-04-25 22:43:26,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.87 | bwd: 5795.07 | bwd_inner: 5782.17 | bwd_allreduce: 12.86 | step: 19.19

 15%|█▍        | 6109/41250 [14:45:52<85:16:58,  8.74s/it]
                                                          
{'loss': 0.1094, 'grad_norm': 1.5783054828643799, 'learning_rate': 3.8555041659579885e-05, 'epoch': 1.48}

 15%|█▍        | 6109/41250 [14:45:52<85:16:58,  8.74s/it][2025-04-25 22:43:35,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:43:35,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5710.22 | bwd_inner_microstep: 5693.35 | bwd_allreduce_microstep: 16.83 | step_microstep: 18.84
[2025-04-25 22:43:35,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.48 | bwd: 5710.23 | bwd_inner: 5693.35 | bwd_allreduce: 16.84 | step: 18.84

 15%|█▍        | 6110/41250 [14:46:00<84:59:52,  8.71s/it]
                                                          
{'loss': 0.0493, 'grad_norm': 0.6803320646286011, 'learning_rate': 3.855445556157093e-05, 'epoch': 1.48}

 15%|█▍        | 6110/41250 [14:46:00<84:59:52,  8.71s/it][2025-04-25 22:43:44,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.17 | optimizer_step: 0.93
[2025-04-25 22:43:44,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.82 | bwd_microstep: 5778.77 | bwd_inner_microstep: 5695.62 | bwd_allreduce_microstep: 83.10 | step_microstep: 19.01
[2025-04-25 22:43:44,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.82 | bwd: 5778.78 | bwd_inner: 5695.62 | bwd_allreduce: 83.12 | step: 19.01

 15%|█▍        | 6111/41250 [14:46:09<85:00:14,  8.71s/it]
                                                          
{'loss': 0.2111, 'grad_norm': 2.2007675170898438, 'learning_rate': 3.855386934917742e-05, 'epoch': 1.48}

 15%|█▍        | 6111/41250 [14:46:09<85:00:14,  8.71s/it][2025-04-25 22:43:52,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:43:52,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.00 | bwd_microstep: 5734.79 | bwd_inner_microstep: 5708.25 | bwd_allreduce_microstep: 26.50 | step_microstep: 18.81
[2025-04-25 22:43:52,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.00 | bwd: 5734.80 | bwd_inner: 5708.25 | bwd_allreduce: 26.51 | step: 18.81

 15%|█▍        | 6112/41250 [14:46:18<84:54:11,  8.70s/it]
                                                          
{'loss': 0.2271, 'grad_norm': 2.8475306034088135, 'learning_rate': 3.8553283022402947e-05, 'epoch': 1.48}

 15%|█▍        | 6112/41250 [14:46:18<84:54:11,  8.70s/it][2025-04-25 22:44:01,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-25 22:44:01,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.31 | bwd_microstep: 5997.78 | bwd_inner_microstep: 5768.26 | bwd_allreduce_microstep: 229.47 | step_microstep: 19.45
[2025-04-25 22:44:01,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.31 | bwd: 5997.79 | bwd_inner: 5768.26 | bwd_allreduce: 229.49 | step: 19.45

 15%|█▍        | 6113/41250 [14:46:27<85:41:53,  8.78s/it]
                                                          
{'loss': 0.1954, 'grad_norm': 2.0391738414764404, 'learning_rate': 3.8552696581251136e-05, 'epoch': 1.48}

 15%|█▍        | 6113/41250 [14:46:27<85:41:53,  8.78s/it][2025-04-25 22:44:10,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 22:44:10,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.07 | bwd_microstep: 5762.78 | bwd_inner_microstep: 5711.34 | bwd_allreduce_microstep: 51.40 | step_microstep: 18.82
[2025-04-25 22:44:10,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.07 | bwd: 5762.80 | bwd_inner: 5711.34 | bwd_allreduce: 51.42 | step: 18.82

 15%|█▍        | 6114/41250 [14:46:35<85:28:00,  8.76s/it]
                                                          
{'loss': 0.0779, 'grad_norm': 1.6984562873840332, 'learning_rate': 3.855211002572559e-05, 'epoch': 1.48}

 15%|█▍        | 6114/41250 [14:46:35<85:28:00,  8.76s/it][2025-04-25 22:44:19,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 22:44:19,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.41 | bwd_microstep: 5792.90 | bwd_inner_microstep: 5655.53 | bwd_allreduce_microstep: 137.32 | step_microstep: 19.30
[2025-04-25 22:44:19,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.41 | bwd: 5792.91 | bwd_inner: 5655.53 | bwd_allreduce: 137.34 | step: 19.30

 15%|█▍        | 6115/41250 [14:46:44<85:18:55,  8.74s/it]
                                                          
{'loss': 0.1118, 'grad_norm': 1.7360550165176392, 'learning_rate': 3.8551523355829943e-05, 'epoch': 1.48}

 15%|█▍        | 6115/41250 [14:46:44<85:18:55,  8.74s/it][2025-04-25 22:44:27,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:44:27,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.02 | bwd_microstep: 5698.59 | bwd_inner_microstep: 5654.95 | bwd_allreduce_microstep: 43.59 | step_microstep: 18.59
[2025-04-25 22:44:27,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.02 | bwd: 5698.60 | bwd_inner: 5654.95 | bwd_allreduce: 43.60 | step: 18.60

 15%|█▍        | 6116/41250 [14:46:53<84:55:43,  8.70s/it]
                                                          
{'loss': 0.116, 'grad_norm': 3.732700824737549, 'learning_rate': 3.8550936571567806e-05, 'epoch': 1.48}

 15%|█▍        | 6116/41250 [14:46:53<84:55:43,  8.70s/it][2025-04-25 22:44:36,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 22:44:36,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.92 | bwd_microstep: 5764.93 | bwd_inner_microstep: 5655.96 | bwd_allreduce_microstep: 108.93 | step_microstep: 18.74
[2025-04-25 22:44:36,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.92 | bwd: 5764.94 | bwd_inner: 5655.96 | bwd_allreduce: 108.94 | step: 18.74

 15%|█▍        | 6117/41250 [14:47:01<84:51:42,  8.70s/it]
                                                          
{'loss': 0.037, 'grad_norm': 1.5926142930984497, 'learning_rate': 3.8550349672942784e-05, 'epoch': 1.48}

 15%|█▍        | 6117/41250 [14:47:01<84:51:42,  8.70s/it][2025-04-25 22:44:45,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-25 22:44:45,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.72 | bwd_microstep: 5713.61 | bwd_inner_microstep: 5700.97 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.77
[2025-04-25 22:44:45,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.72 | bwd: 5713.62 | bwd_inner: 5700.97 | bwd_allreduce: 12.61 | step: 18.77

 15%|█▍        | 6118/41250 [14:47:10<84:43:34,  8.68s/it]
                                                          
{'loss': 0.1315, 'grad_norm': 2.694427728652954, 'learning_rate': 3.854976265995851e-05, 'epoch': 1.48}

 15%|█▍        | 6118/41250 [14:47:10<84:43:34,  8.68s/it][2025-04-25 22:44:53,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-25 22:44:53,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.09 | bwd_microstep: 5686.10 | bwd_inner_microstep: 5653.60 | bwd_allreduce_microstep: 32.46 | step_microstep: 19.17
[2025-04-25 22:44:53,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.09 | bwd: 5686.11 | bwd_inner: 5653.60 | bwd_allreduce: 32.47 | step: 19.17

 15%|█▍        | 6119/41250 [14:47:19<84:29:00,  8.66s/it]
                                                          
{'loss': 0.2185, 'grad_norm': 2.397686719894409, 'learning_rate': 3.85491755326186e-05, 'epoch': 1.48}

 15%|█▍        | 6119/41250 [14:47:19<84:29:00,  8.66s/it][2025-04-25 22:45:02,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:45:02,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5718.11 | bwd_inner_microstep: 5696.65 | bwd_allreduce_microstep: 21.42 | step_microstep: 18.79
[2025-04-25 22:45:02,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5718.12 | bwd_inner: 5696.65 | bwd_allreduce: 21.43 | step: 18.80

 15%|█▍        | 6120/41250 [14:47:27<84:27:34,  8.66s/it]
                                                          
{'loss': 0.2087, 'grad_norm': 2.3097383975982666, 'learning_rate': 3.854858829092667e-05, 'epoch': 1.48}

 15%|█▍        | 6120/41250 [14:47:27<84:27:34,  8.66s/it][2025-04-25 22:45:11,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:45:11,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.60 | bwd_microstep: 5745.22 | bwd_inner_microstep: 5687.13 | bwd_allreduce_microstep: 58.05 | step_microstep: 18.86
[2025-04-25 22:45:11,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.60 | bwd: 5745.24 | bwd_inner: 5687.12 | bwd_allreduce: 58.07 | step: 18.86

 15%|█▍        | 6121/41250 [14:47:36<84:30:26,  8.66s/it]
                                                          
{'loss': 0.0744, 'grad_norm': 1.47463858127594, 'learning_rate': 3.854800093488634e-05, 'epoch': 1.48}

 15%|█▍        | 6121/41250 [14:47:36<84:30:26,  8.66s/it][2025-04-25 22:45:19,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:45:19,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.10 | bwd_microstep: 5747.75 | bwd_inner_microstep: 5697.34 | bwd_allreduce_microstep: 50.36 | step_microstep: 18.88
[2025-04-25 22:45:19,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.10 | bwd: 5747.77 | bwd_inner: 5697.34 | bwd_allreduce: 50.38 | step: 18.88

 15%|█▍        | 6122/41250 [14:47:45<84:34:02,  8.67s/it]
                                                          
{'loss': 0.1605, 'grad_norm': 4.118828773498535, 'learning_rate': 3.854741346450124e-05, 'epoch': 1.48}

 15%|█▍        | 6122/41250 [14:47:45<84:34:02,  8.67s/it][2025-04-25 22:45:28,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 22:45:28,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.91 | bwd_microstep: 5780.32 | bwd_inner_microstep: 5651.66 | bwd_allreduce_microstep: 128.62 | step_microstep: 18.60
[2025-04-25 22:45:28,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.91 | bwd: 5780.34 | bwd_inner: 5651.66 | bwd_allreduce: 128.64 | step: 18.60

 15%|█▍        | 6123/41250 [14:47:53<84:37:16,  8.67s/it]
                                                          
{'loss': 0.1076, 'grad_norm': 2.025045394897461, 'learning_rate': 3.8546825879774975e-05, 'epoch': 1.48}

 15%|█▍        | 6123/41250 [14:47:53<84:37:16,  8.67s/it][2025-04-25 22:45:37,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 22:45:37,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.45 | bwd_microstep: 5697.41 | bwd_inner_microstep: 5684.64 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.83
[2025-04-25 22:45:37,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.45 | bwd: 5697.43 | bwd_inner: 5684.64 | bwd_allreduce: 12.74 | step: 18.83

 15%|█▍        | 6124/41250 [14:48:02<84:28:08,  8.66s/it]
                                                          
{'loss': 0.1178, 'grad_norm': 1.3667101860046387, 'learning_rate': 3.854623818071118e-05, 'epoch': 1.48}

 15%|█▍        | 6124/41250 [14:48:02<84:28:08,  8.66s/it][2025-04-25 22:45:45,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:45:45,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5703.50 | bwd_inner_microstep: 5655.88 | bwd_allreduce_microstep: 47.58 | step_microstep: 18.40
[2025-04-25 22:45:45,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5703.51 | bwd_inner: 5655.88 | bwd_allreduce: 47.59 | step: 18.40

 15%|█▍        | 6125/41250 [14:48:10<84:20:45,  8.64s/it]
                                                          
{'loss': 0.17, 'grad_norm': 1.346262812614441, 'learning_rate': 3.8545650367313475e-05, 'epoch': 1.48}

 15%|█▍        | 6125/41250 [14:48:10<84:20:45,  8.64s/it][2025-04-25 22:45:54,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:45:54,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.07 | bwd_microstep: 5767.83 | bwd_inner_microstep: 5646.24 | bwd_allreduce_microstep: 121.54 | step_microstep: 18.53
[2025-04-25 22:45:54,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.07 | bwd: 5767.84 | bwd_inner: 5646.24 | bwd_allreduce: 121.56 | step: 18.53

 15%|█▍        | 6126/41250 [14:48:19<84:27:18,  8.66s/it]
                                                          
{'loss': 0.0358, 'grad_norm': 1.34906005859375, 'learning_rate': 3.8545062439585475e-05, 'epoch': 1.49}

 15%|█▍        | 6126/41250 [14:48:19<84:27:18,  8.66s/it][2025-04-25 22:46:02,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 22:46:02,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5759.85 | bwd_inner_microstep: 5646.98 | bwd_allreduce_microstep: 112.83 | step_microstep: 18.50
[2025-04-25 22:46:02,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5759.86 | bwd_inner: 5646.98 | bwd_allreduce: 112.84 | step: 18.50

 15%|█▍        | 6127/41250 [14:48:28<84:30:23,  8.66s/it]
                                                          
{'loss': 0.1306, 'grad_norm': 1.5474921464920044, 'learning_rate': 3.854447439753082e-05, 'epoch': 1.49}

 15%|█▍        | 6127/41250 [14:48:28<84:30:23,  8.66s/it][2025-04-25 22:46:11,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 22:46:11,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.23 | bwd_microstep: 5682.38 | bwd_inner_microstep: 5653.92 | bwd_allreduce_microstep: 28.41 | step_microstep: 18.59
[2025-04-25 22:46:11,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.23 | bwd: 5682.40 | bwd_inner: 5653.92 | bwd_allreduce: 28.43 | step: 18.60

 15%|█▍        | 6128/41250 [14:48:36<84:18:01,  8.64s/it]
                                                          
{'loss': 0.2267, 'grad_norm': 3.750621795654297, 'learning_rate': 3.854388624115313e-05, 'epoch': 1.49}

 15%|█▍        | 6128/41250 [14:48:36<84:18:01,  8.64s/it][2025-04-25 22:46:20,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 22:46:20,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.42 | bwd_microstep: 5715.04 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.34
[2025-04-25 22:46:20,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.42 | bwd: 5715.05 | bwd_inner: 5702.41 | bwd_allreduce: 12.60 | step: 18.34

 15%|█▍        | 6129/41250 [14:48:45<84:20:12,  8.64s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 1.6302876472473145, 'learning_rate': 3.854329797045603e-05, 'epoch': 1.49}

 15%|█▍        | 6129/41250 [14:48:45<84:20:12,  8.64s/it][2025-04-25 22:46:28,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:46:28,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5747.99 | bwd_inner_microstep: 5643.57 | bwd_allreduce_microstep: 104.37 | step_microstep: 18.43
[2025-04-25 22:46:28,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5748.00 | bwd_inner: 5643.57 | bwd_allreduce: 104.39 | step: 18.43

 15%|█▍        | 6130/41250 [14:48:54<84:24:32,  8.65s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.2418947219848633, 'learning_rate': 3.8542709585443136e-05, 'epoch': 1.49}

 15%|█▍        | 6130/41250 [14:48:54<84:24:32,  8.65s/it][2025-04-25 22:46:37,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-25 22:46:37,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.97 | bwd_microstep: 5746.72 | bwd_inner_microstep: 5674.27 | bwd_allreduce_microstep: 72.39 | step_microstep: 20.25
[2025-04-25 22:46:37,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.97 | bwd: 5746.74 | bwd_inner: 5674.27 | bwd_allreduce: 72.42 | step: 20.25

 15%|█▍        | 6131/41250 [14:49:02<84:29:06,  8.66s/it]
                                                          
{'loss': 0.2228, 'grad_norm': 3.5927791595458984, 'learning_rate': 3.854212108611809e-05, 'epoch': 1.49}

 15%|█▍        | 6131/41250 [14:49:02<84:29:06,  8.66s/it][2025-04-25 22:46:46,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.08 | optimizer_step: 1.12
[2025-04-25 22:46:46,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.57 | bwd_microstep: 5740.31 | bwd_inner_microstep: 5686.93 | bwd_allreduce_microstep: 53.32 | step_microstep: 19.22
[2025-04-25 22:46:46,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.57 | bwd: 5740.32 | bwd_inner: 5686.93 | bwd_allreduce: 53.34 | step: 19.22

 15%|█▍        | 6132/41250 [14:49:11<84:32:34,  8.67s/it]
                                                          
{'loss': 0.3336, 'grad_norm': 2.748608112335205, 'learning_rate': 3.854153247248452e-05, 'epoch': 1.49}

 15%|█▍        | 6132/41250 [14:49:11<84:32:34,  8.67s/it][2025-04-25 22:46:54,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.10 | optimizer_step: 0.98
[2025-04-25 22:46:54,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.05 | bwd_microstep: 5772.83 | bwd_inner_microstep: 5759.67 | bwd_allreduce_microstep: 13.11 | step_microstep: 19.29
[2025-04-25 22:46:54,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.05 | bwd: 5772.85 | bwd_inner: 5759.67 | bwd_allreduce: 13.13 | step: 19.29

 15%|█▍        | 6133/41250 [14:49:20<84:44:19,  8.69s/it]
                                                          
{'loss': 0.1199, 'grad_norm': 2.1759111881256104, 'learning_rate': 3.854094374454603e-05, 'epoch': 1.49}

 15%|█▍        | 6133/41250 [14:49:20<84:44:19,  8.69s/it][2025-04-25 22:47:03,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.33 | optimizer_step: 1.06
[2025-04-25 22:47:03,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.22 | bwd_microstep: 5767.54 | bwd_inner_microstep: 5657.94 | bwd_allreduce_microstep: 109.53 | step_microstep: 20.16
[2025-04-25 22:47:03,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.22 | bwd: 5767.56 | bwd_inner: 5657.94 | bwd_allreduce: 109.56 | step: 20.16

 15%|█▍        | 6134/41250 [14:49:28<84:42:54,  8.68s/it]
                                                          
{'loss': 0.0682, 'grad_norm': 1.709874153137207, 'learning_rate': 3.854035490230629e-05, 'epoch': 1.49}

 15%|█▍        | 6134/41250 [14:49:28<84:42:54,  8.68s/it][2025-04-25 22:47:12,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:47:12,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.69 | bwd_microstep: 5758.52 | bwd_inner_microstep: 5702.25 | bwd_allreduce_microstep: 56.23 | step_microstep: 18.77
[2025-04-25 22:47:12,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.69 | bwd: 5758.54 | bwd_inner: 5702.25 | bwd_allreduce: 56.24 | step: 18.77

 15%|█▍        | 6135/41250 [14:49:37<84:44:30,  8.69s/it]
                                                          
{'loss': 0.2387, 'grad_norm': 4.299732208251953, 'learning_rate': 3.85397659457689e-05, 'epoch': 1.49}

 15%|█▍        | 6135/41250 [14:49:37<84:44:30,  8.69s/it][2025-04-25 22:47:21,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 22:47:21,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.54 | bwd_microstep: 5861.50 | bwd_inner_microstep: 5688.17 | bwd_allreduce_microstep: 173.28 | step_microstep: 19.24
[2025-04-25 22:47:21,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.54 | bwd: 5861.51 | bwd_inner: 5688.17 | bwd_allreduce: 173.30 | step: 19.24

 15%|█▍        | 6136/41250 [14:49:46<85:02:28,  8.72s/it]
                                                          
{'loss': 0.2292, 'grad_norm': 3.217113494873047, 'learning_rate': 3.8539176874937505e-05, 'epoch': 1.49}

 15%|█▍        | 6136/41250 [14:49:46<85:02:28,  8.72s/it][2025-04-25 22:47:29,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 22:47:29,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.23 | bwd_microstep: 5787.99 | bwd_inner_microstep: 5775.11 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.80
[2025-04-25 22:47:29,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.23 | bwd: 5788.01 | bwd_inner: 5775.11 | bwd_allreduce: 12.86 | step: 18.80

 15%|█▍        | 6137/41250 [14:49:55<85:08:58,  8.73s/it]
                                                          
{'loss': 0.4097, 'grad_norm': 2.867867946624756, 'learning_rate': 3.8538587689815726e-05, 'epoch': 1.49}

 15%|█▍        | 6137/41250 [14:49:55<85:08:58,  8.73s/it][2025-04-25 22:47:38,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-25 22:47:38,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5704.57 | bwd_inner_microstep: 5691.86 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.67
[2025-04-25 22:47:38,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5704.58 | bwd_inner: 5691.86 | bwd_allreduce: 12.68 | step: 19.67

 15%|█▍        | 6138/41250 [14:50:03<84:52:40,  8.70s/it]
                                                          
{'loss': 0.0532, 'grad_norm': 1.513671875, 'learning_rate': 3.853799839040719e-05, 'epoch': 1.49}

 15%|█▍        | 6138/41250 [14:50:03<84:52:40,  8.70s/it][2025-04-25 22:47:47,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:47:47,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5871.78 | bwd_inner_microstep: 5679.66 | bwd_allreduce_microstep: 192.07 | step_microstep: 18.75
[2025-04-25 22:47:47,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.18 | bwd: 5871.79 | bwd_inner: 5679.66 | bwd_allreduce: 192.09 | step: 18.75

 15%|█▍        | 6139/41250 [14:50:12<85:11:08,  8.73s/it]
                                                          
{'loss': 0.204, 'grad_norm': 2.087547540664673, 'learning_rate': 3.853740897671556e-05, 'epoch': 1.49}

 15%|█▍        | 6139/41250 [14:50:12<85:11:08,  8.73s/it][2025-04-25 22:47:56,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:47:56,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.38 | bwd_microstep: 5770.32 | bwd_inner_microstep: 5656.67 | bwd_allreduce_microstep: 113.60 | step_microstep: 18.89
[2025-04-25 22:47:56,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.38 | bwd: 5770.33 | bwd_inner: 5656.67 | bwd_allreduce: 113.62 | step: 18.89

 15%|█▍        | 6140/41250 [14:50:21<85:03:21,  8.72s/it]
                                                          
{'loss': 0.382, 'grad_norm': 4.015090465545654, 'learning_rate': 3.853681944874444e-05, 'epoch': 1.49}

 15%|█▍        | 6140/41250 [14:50:21<85:03:21,  8.72s/it][2025-04-25 22:48:04,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:48:04,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.34 | bwd_microstep: 5849.96 | bwd_inner_microstep: 5704.96 | bwd_allreduce_microstep: 144.96 | step_microstep: 18.24
[2025-04-25 22:48:04,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.34 | bwd: 5849.97 | bwd_inner: 5704.95 | bwd_allreduce: 144.97 | step: 18.24

 15%|█▍        | 6141/41250 [14:50:30<85:17:22,  8.75s/it]
                                                          
{'loss': 0.1667, 'grad_norm': 2.4314091205596924, 'learning_rate': 3.853622980649747e-05, 'epoch': 1.49}

 15%|█▍        | 6141/41250 [14:50:30<85:17:22,  8.75s/it][2025-04-25 22:48:13,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-25 22:48:13,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.85 | bwd_microstep: 5770.60 | bwd_inner_microstep: 5653.22 | bwd_allreduce_microstep: 117.33 | step_microstep: 19.06
[2025-04-25 22:48:13,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.04 | bwd: 5770.61 | bwd_inner: 5653.22 | bwd_allreduce: 117.35 | step: 19.07

 15%|█▍        | 6142/41250 [14:50:38<85:08:33,  8.73s/it]
                                                          
{'loss': 0.0624, 'grad_norm': 0.9849414825439453, 'learning_rate': 3.85356400499783e-05, 'epoch': 1.49}

 15%|█▍        | 6142/41250 [14:50:38<85:08:33,  8.73s/it][2025-04-25 22:48:22,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:48:22,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.59 | bwd_microstep: 5775.52 | bwd_inner_microstep: 5662.70 | bwd_allreduce_microstep: 112.78 | step_microstep: 18.27
[2025-04-25 22:48:22,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.59 | bwd: 5775.53 | bwd_inner: 5662.70 | bwd_allreduce: 112.80 | step: 18.27

 15%|█▍        | 6143/41250 [14:50:47<85:02:31,  8.72s/it]
                                                          
{'loss': 0.0735, 'grad_norm': 2.251028537750244, 'learning_rate': 3.8535050179190545e-05, 'epoch': 1.49}

 15%|█▍        | 6143/41250 [14:50:47<85:02:31,  8.72s/it][2025-04-25 22:48:30,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.18 | optimizer_step: 0.92
[2025-04-25 22:48:30,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.41 | bwd_microstep: 5759.81 | bwd_inner_microstep: 5707.81 | bwd_allreduce_microstep: 51.95 | step_microstep: 18.81
[2025-04-25 22:48:30,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.41 | bwd: 5759.82 | bwd_inner: 5707.81 | bwd_allreduce: 51.97 | step: 18.81

 15%|█▍        | 6144/41250 [14:50:56<84:58:24,  8.71s/it]
                                                          
{'loss': 0.2281, 'grad_norm': 2.9236836433410645, 'learning_rate': 3.853446019413786e-05, 'epoch': 1.49}

 15%|█▍        | 6144/41250 [14:50:56<84:58:24,  8.71s/it][2025-04-25 22:48:39,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 22:48:39,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.04 | bwd_microstep: 5749.38 | bwd_inner_microstep: 5719.37 | bwd_allreduce_microstep: 29.97 | step_microstep: 18.71
[2025-04-25 22:48:39,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.04 | bwd: 5749.40 | bwd_inner: 5719.37 | bwd_allreduce: 29.99 | step: 18.71

 15%|█▍        | 6145/41250 [14:51:04<84:55:32,  8.71s/it]
                                                          
{'loss': 0.2978, 'grad_norm': 4.084099292755127, 'learning_rate': 3.8533870094823866e-05, 'epoch': 1.49}

 15%|█▍        | 6145/41250 [14:51:04<84:55:32,  8.71s/it][2025-04-25 22:48:48,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 22:48:48,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.32 | bwd_microstep: 5723.16 | bwd_inner_microstep: 5671.13 | bwd_allreduce_microstep: 51.98 | step_microstep: 18.44
[2025-04-25 22:48:48,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.32 | bwd: 5723.17 | bwd_inner: 5671.13 | bwd_allreduce: 52.00 | step: 18.44

 15%|█▍        | 6146/41250 [14:51:13<84:44:13,  8.69s/it]
                                                          
{'loss': 0.3981, 'grad_norm': 2.6619210243225098, 'learning_rate': 3.8533279881252206e-05, 'epoch': 1.49}

 15%|█▍        | 6146/41250 [14:51:13<84:44:13,  8.69s/it][2025-04-25 22:48:56,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-25 22:48:56,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.98 | bwd_microstep: 5750.68 | bwd_inner_microstep: 5708.76 | bwd_allreduce_microstep: 41.88 | step_microstep: 18.98
[2025-04-25 22:48:56,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.98 | bwd: 5750.69 | bwd_inner: 5708.76 | bwd_allreduce: 41.89 | step: 18.98

 15%|█▍        | 6147/41250 [14:51:22<84:43:57,  8.69s/it]
                                                          
{'loss': 0.1607, 'grad_norm': 3.4278616905212402, 'learning_rate': 3.853268955342653e-05, 'epoch': 1.49}

 15%|█▍        | 6147/41250 [14:51:22<84:43:57,  8.69s/it][2025-04-25 22:49:05,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:49:05,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.33 | bwd_microstep: 5791.85 | bwd_inner_microstep: 5660.63 | bwd_allreduce_microstep: 131.17 | step_microstep: 18.80
[2025-04-25 22:49:05,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.33 | bwd: 5791.86 | bwd_inner: 5660.63 | bwd_allreduce: 131.18 | step: 18.80

 15%|█▍        | 6148/41250 [14:51:31<84:47:38,  8.70s/it]
                                                          
{'loss': 0.2998, 'grad_norm': 4.13748836517334, 'learning_rate': 3.8532099111350464e-05, 'epoch': 1.49}

 15%|█▍        | 6148/41250 [14:51:31<84:47:38,  8.70s/it][2025-04-25 22:49:14,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:49:14,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.31 | bwd_microstep: 5756.27 | bwd_inner_microstep: 5712.08 | bwd_allreduce_microstep: 44.15 | step_microstep: 18.72
[2025-04-25 22:49:14,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.31 | bwd: 5756.29 | bwd_inner: 5712.08 | bwd_allreduce: 44.17 | step: 18.72

 15%|█▍        | 6149/41250 [14:51:39<84:49:16,  8.70s/it]
                                                          
{'loss': 0.1861, 'grad_norm': 5.627776145935059, 'learning_rate': 3.8531508555027646e-05, 'epoch': 1.49}

 15%|█▍        | 6149/41250 [14:51:39<84:49:16,  8.70s/it][2025-04-25 22:49:23,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:49:23,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.33 | bwd_microstep: 5752.84 | bwd_inner_microstep: 5663.08 | bwd_allreduce_microstep: 89.72 | step_microstep: 18.54
[2025-04-25 22:49:23,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.33 | bwd: 5752.86 | bwd_inner: 5663.08 | bwd_allreduce: 89.73 | step: 18.54

 15%|█▍        | 6150/41250 [14:51:48<84:47:02,  8.70s/it]
                                                          
{'loss': 0.3029, 'grad_norm': 1.8546802997589111, 'learning_rate': 3.8530917884461735e-05, 'epoch': 1.49}

 15%|█▍        | 6150/41250 [14:51:48<84:47:02,  8.70s/it][2025-04-25 22:49:31,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-25 22:49:31,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.42 | bwd_microstep: 5765.21 | bwd_inner_microstep: 5650.58 | bwd_allreduce_microstep: 114.58 | step_microstep: 19.13
[2025-04-25 22:49:31,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.42 | bwd: 5765.23 | bwd_inner: 5650.58 | bwd_allreduce: 114.60 | step: 19.14

 15%|█▍        | 6151/41250 [14:51:57<84:44:19,  8.69s/it]
                                                          
{'loss': 0.1358, 'grad_norm': 1.6705126762390137, 'learning_rate': 3.853032709965635e-05, 'epoch': 1.49}

 15%|█▍        | 6151/41250 [14:51:57<84:44:19,  8.69s/it][2025-04-25 22:49:40,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 22:49:40,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.18 | bwd_microstep: 5700.15 | bwd_inner_microstep: 5664.36 | bwd_allreduce_microstep: 35.74 | step_microstep: 18.61
[2025-04-25 22:49:40,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.18 | bwd: 5700.16 | bwd_inner: 5664.36 | bwd_allreduce: 35.76 | step: 18.61

 15%|█▍        | 6152/41250 [14:52:05<84:32:40,  8.67s/it]
                                                          
{'loss': 0.0958, 'grad_norm': 8.85157585144043, 'learning_rate': 3.852973620061515e-05, 'epoch': 1.49}

 15%|█▍        | 6152/41250 [14:52:05<84:32:40,  8.67s/it][2025-04-25 22:49:49,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:49:49,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.15 | bwd_microstep: 5712.37 | bwd_inner_microstep: 5699.59 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.62
[2025-04-25 22:49:49,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.15 | bwd: 5712.39 | bwd_inner: 5699.59 | bwd_allreduce: 12.75 | step: 18.63

 15%|█▍        | 6153/41250 [14:52:14<84:30:03,  8.67s/it]
                                                          
{'loss': 0.0364, 'grad_norm': 0.6150863766670227, 'learning_rate': 3.852914518734176e-05, 'epoch': 1.49}

 15%|█▍        | 6153/41250 [14:52:14<84:30:03,  8.67s/it][2025-04-25 22:49:57,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:49:57,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2908.51 | bwd_microstep: 5805.56 | bwd_inner_microstep: 5792.73 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.59
[2025-04-25 22:49:57,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2908.51 | bwd: 5805.57 | bwd_inner: 5792.73 | bwd_allreduce: 12.81 | step: 18.59

 15%|█▍        | 6154/41250 [14:52:23<84:52:16,  8.71s/it]
                                                          
{'loss': 0.2872, 'grad_norm': 2.720568895339966, 'learning_rate': 3.8528554059839845e-05, 'epoch': 1.49}

 15%|█▍        | 6154/41250 [14:52:23<84:52:16,  8.71s/it][2025-04-25 22:50:06,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:50:06,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.25 | bwd_microstep: 5784.68 | bwd_inner_microstep: 5664.11 | bwd_allreduce_microstep: 120.52 | step_microstep: 18.18
[2025-04-25 22:50:06,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.25 | bwd: 5784.69 | bwd_inner: 5664.11 | bwd_allreduce: 120.54 | step: 18.18

 15%|█▍        | 6155/41250 [14:52:31<84:51:11,  8.70s/it]
                                                          
{'loss': 0.0746, 'grad_norm': 2.5417208671569824, 'learning_rate': 3.852796281811304e-05, 'epoch': 1.49}

 15%|█▍        | 6155/41250 [14:52:31<84:51:11,  8.70s/it][2025-04-25 22:50:15,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:50:15,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.78 | bwd_microstep: 5767.07 | bwd_inner_microstep: 5703.19 | bwd_allreduce_microstep: 63.84 | step_microstep: 18.58
[2025-04-25 22:50:15,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.78 | bwd: 5767.08 | bwd_inner: 5703.19 | bwd_allreduce: 63.85 | step: 18.59

 15%|█▍        | 6156/41250 [14:52:40<84:52:04,  8.71s/it]
                                                          
{'loss': 0.0117, 'grad_norm': 0.13042360544204712, 'learning_rate': 3.852737146216498e-05, 'epoch': 1.49}

 15%|█▍        | 6156/41250 [14:52:40<84:52:04,  8.71s/it][2025-04-25 22:50:23,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:50:23,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.28 | bwd_microstep: 5737.54 | bwd_inner_microstep: 5688.87 | bwd_allreduce_microstep: 48.62 | step_microstep: 18.01
[2025-04-25 22:50:23,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.28 | bwd: 5737.55 | bwd_inner: 5688.87 | bwd_allreduce: 48.64 | step: 18.01

 15%|█▍        | 6157/41250 [14:52:49<84:48:08,  8.70s/it]
                                                          
{'loss': 0.1204, 'grad_norm': 1.9024100303649902, 'learning_rate': 3.852677999199932e-05, 'epoch': 1.49}

 15%|█▍        | 6157/41250 [14:52:49<84:48:08,  8.70s/it][2025-04-25 22:50:32,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 22:50:32,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.24 | bwd_microstep: 5690.51 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 30.54 | step_microstep: 18.68
[2025-04-25 22:50:32,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.24 | bwd: 5690.52 | bwd_inner: 5659.93 | bwd_allreduce: 30.56 | step: 18.68

 15%|█▍        | 6158/41250 [14:52:57<84:34:18,  8.68s/it]
                                                          
{'loss': 0.0416, 'grad_norm': 1.0778157711029053, 'learning_rate': 3.852618840761971e-05, 'epoch': 1.49}

 15%|█▍        | 6158/41250 [14:52:57<84:34:18,  8.68s/it][2025-04-25 22:50:41,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:50:41,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.10 | bwd_microstep: 5790.36 | bwd_inner_microstep: 5777.61 | bwd_allreduce_microstep: 12.71 | step_microstep: 17.67
[2025-04-25 22:50:41,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.10 | bwd: 5790.37 | bwd_inner: 5777.60 | bwd_allreduce: 12.73 | step: 17.68

 15%|█▍        | 6159/41250 [14:53:06<84:47:57,  8.70s/it]
                                                          
{'loss': 0.0882, 'grad_norm': 0.8754249215126038, 'learning_rate': 3.852559670902979e-05, 'epoch': 1.49}

 15%|█▍        | 6159/41250 [14:53:06<84:47:57,  8.70s/it][2025-04-25 22:50:49,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:50:49,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.78 | bwd_microstep: 5711.23 | bwd_inner_microstep: 5695.29 | bwd_allreduce_microstep: 15.90 | step_microstep: 18.65
[2025-04-25 22:50:49,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.78 | bwd: 5711.25 | bwd_inner: 5695.29 | bwd_allreduce: 15.92 | step: 18.66

 15%|█▍        | 6160/41250 [14:53:15<84:40:00,  8.69s/it]
                                                          
{'loss': 0.1788, 'grad_norm': 1.6947002410888672, 'learning_rate': 3.852500489623321e-05, 'epoch': 1.49}

 15%|█▍        | 6160/41250 [14:53:15<84:40:00,  8.69s/it][2025-04-25 22:50:58,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:50:58,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.61 | bwd_microstep: 5728.27 | bwd_inner_microstep: 5691.94 | bwd_allreduce_microstep: 36.28 | step_microstep: 18.64
[2025-04-25 22:50:58,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.61 | bwd: 5728.28 | bwd_inner: 5691.94 | bwd_allreduce: 36.30 | step: 18.64

 15%|█▍        | 6161/41250 [14:53:23<84:36:30,  8.68s/it]
                                                          
{'loss': 0.0349, 'grad_norm': 0.42656180262565613, 'learning_rate': 3.852441296923362e-05, 'epoch': 1.49}

 15%|█▍        | 6161/41250 [14:53:23<84:36:30,  8.68s/it][2025-04-25 22:51:07,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:51:07,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.90 | bwd_microstep: 5701.48 | bwd_inner_microstep: 5648.38 | bwd_allreduce_microstep: 53.06 | step_microstep: 18.56
[2025-04-25 22:51:07,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.90 | bwd: 5701.49 | bwd_inner: 5648.38 | bwd_allreduce: 53.08 | step: 18.56

 15%|█▍        | 6162/41250 [14:53:32<84:25:15,  8.66s/it]
                                                          
{'loss': 0.0871, 'grad_norm': 0.9051409959793091, 'learning_rate': 3.852382092803467e-05, 'epoch': 1.49}

 15%|█▍        | 6162/41250 [14:53:32<84:25:15,  8.66s/it][2025-04-25 22:51:15,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:51:15,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5746.70 | bwd_inner_microstep: 5694.08 | bwd_allreduce_microstep: 52.57 | step_microstep: 18.24
[2025-04-25 22:51:15,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5746.71 | bwd_inner: 5694.08 | bwd_allreduce: 52.58 | step: 18.24

 15%|█▍        | 6163/41250 [14:53:41<84:28:01,  8.67s/it]
                                                          
{'loss': 0.1088, 'grad_norm': 4.249880313873291, 'learning_rate': 3.852322877264e-05, 'epoch': 1.49}

 15%|█▍        | 6163/41250 [14:53:41<84:28:01,  8.67s/it][2025-04-25 22:51:24,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:51:24,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.49 | bwd_microstep: 5723.99 | bwd_inner_microstep: 5685.60 | bwd_allreduce_microstep: 38.35 | step_microstep: 18.43
[2025-04-25 22:51:24,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5724.01 | bwd_inner: 5685.60 | bwd_allreduce: 38.37 | step: 18.44

 15%|█▍        | 6164/41250 [14:53:49<84:27:32,  8.67s/it]
                                                          
{'loss': 0.0789, 'grad_norm': 1.435632348060608, 'learning_rate': 3.8522636503053275e-05, 'epoch': 1.49}

 15%|█▍        | 6164/41250 [14:53:49<84:27:32,  8.67s/it][2025-04-25 22:51:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:51:33,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.74 | bwd_microstep: 5793.30 | bwd_inner_microstep: 5780.61 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.22
[2025-04-25 22:51:33,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.74 | bwd: 5793.31 | bwd_inner: 5780.61 | bwd_allreduce: 12.66 | step: 18.22

 15%|█▍        | 6165/41250 [14:53:58<84:47:13,  8.70s/it]
                                                          
{'loss': 0.0473, 'grad_norm': 2.2895290851593018, 'learning_rate': 3.852204411927814e-05, 'epoch': 1.49}

 15%|█▍        | 6165/41250 [14:53:58<84:47:13,  8.70s/it][2025-04-25 22:51:42,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:51:42,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.85 | bwd_microstep: 5719.65 | bwd_inner_microstep: 5699.75 | bwd_allreduce_microstep: 19.87 | step_microstep: 18.34
[2025-04-25 22:51:42,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.86 | bwd: 5719.67 | bwd_inner: 5699.74 | bwd_allreduce: 19.88 | step: 18.34

 15%|█▍        | 6166/41250 [14:54:07<84:39:13,  8.69s/it]
                                                          
{'loss': 0.0877, 'grad_norm': 2.008397340774536, 'learning_rate': 3.852145162131824e-05, 'epoch': 1.49}

 15%|█▍        | 6166/41250 [14:54:07<84:39:13,  8.69s/it][2025-04-25 22:51:50,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 22:51:50,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.96 | bwd_microstep: 5677.54 | bwd_inner_microstep: 5631.88 | bwd_allreduce_microstep: 45.61 | step_microstep: 18.60
[2025-04-25 22:51:50,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.96 | bwd: 5677.55 | bwd_inner: 5631.88 | bwd_allreduce: 45.63 | step: 18.61

 15%|█▍        | 6167/41250 [14:54:15<84:21:20,  8.66s/it]
                                                          
{'loss': 0.1657, 'grad_norm': 1.2612799406051636, 'learning_rate': 3.8520859009177236e-05, 'epoch': 1.5}

 15%|█▍        | 6167/41250 [14:54:15<84:21:20,  8.66s/it][2025-04-25 22:51:59,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:51:59,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.80 | bwd_microstep: 5734.35 | bwd_inner_microstep: 5680.75 | bwd_allreduce_microstep: 53.56 | step_microstep: 18.19
[2025-04-25 22:51:59,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.80 | bwd: 5734.37 | bwd_inner: 5680.75 | bwd_allreduce: 53.58 | step: 18.19

 15%|█▍        | 6168/41250 [14:54:24<84:24:55,  8.66s/it]
                                                          
{'loss': 0.1137, 'grad_norm': 1.6131759881973267, 'learning_rate': 3.852026628285878e-05, 'epoch': 1.5}

 15%|█▍        | 6168/41250 [14:54:24<84:24:55,  8.66s/it][2025-04-25 22:52:07,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:52:07,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.49 | bwd_microstep: 5723.45 | bwd_inner_microstep: 5686.71 | bwd_allreduce_microstep: 36.70 | step_microstep: 18.26
[2025-04-25 22:52:07,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.49 | bwd: 5723.47 | bwd_inner: 5686.71 | bwd_allreduce: 36.71 | step: 18.26

 15%|█▍        | 6169/41250 [14:54:33<84:24:50,  8.66s/it]
                                                          
{'loss': 0.292, 'grad_norm': 3.054283618927002, 'learning_rate': 3.8519673442366524e-05, 'epoch': 1.5}

 15%|█▍        | 6169/41250 [14:54:33<84:24:50,  8.66s/it][2025-04-25 22:52:16,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:52:16,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.81 | bwd_microstep: 5753.96 | bwd_inner_microstep: 5647.83 | bwd_allreduce_microstep: 106.09 | step_microstep: 18.13
[2025-04-25 22:52:16,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.82 | bwd: 5753.97 | bwd_inner: 5647.83 | bwd_allreduce: 106.11 | step: 18.13

 15%|█▍        | 6170/41250 [14:54:41<84:26:03,  8.66s/it]
                                                          
{'loss': 0.0375, 'grad_norm': 0.6268666982650757, 'learning_rate': 3.8519080487704116e-05, 'epoch': 1.5}

 15%|█▍        | 6170/41250 [14:54:41<84:26:03,  8.66s/it][2025-04-25 22:52:25,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:52:25,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.27 | bwd_microstep: 5713.26 | bwd_inner_microstep: 5700.48 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.45
[2025-04-25 22:52:25,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.27 | bwd: 5713.27 | bwd_inner: 5700.48 | bwd_allreduce: 12.75 | step: 18.45

 15%|█▍        | 6171/41250 [14:54:50<84:23:58,  8.66s/it]
                                                          
{'loss': 0.1004, 'grad_norm': 1.8482292890548706, 'learning_rate': 3.851848741887524e-05, 'epoch': 1.5}

 15%|█▍        | 6171/41250 [14:54:50<84:23:58,  8.66s/it][2025-04-25 22:52:33,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:52:33,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.34 | bwd_microstep: 5704.74 | bwd_inner_microstep: 5643.92 | bwd_allreduce_microstep: 60.78 | step_microstep: 18.36
[2025-04-25 22:52:33,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.34 | bwd: 5704.76 | bwd_inner: 5643.92 | bwd_allreduce: 60.80 | step: 18.37

 15%|█▍        | 6172/41250 [14:54:59<84:17:14,  8.65s/it]
                                                          
{'loss': 0.2443, 'grad_norm': 2.441150665283203, 'learning_rate': 3.8517894235883513e-05, 'epoch': 1.5}

 15%|█▍        | 6172/41250 [14:54:59<84:17:14,  8.65s/it][2025-04-25 22:52:42,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:52:42,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.17 | bwd_microstep: 5757.52 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 112.48 | step_microstep: 18.38
[2025-04-25 22:52:42,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.17 | bwd: 5757.54 | bwd_inner: 5645.00 | bwd_allreduce: 112.50 | step: 18.39

 15%|█▍        | 6173/41250 [14:55:07<84:20:51,  8.66s/it]
                                                          
{'loss': 0.1564, 'grad_norm': 2.3411781787872314, 'learning_rate': 3.851730093873262e-05, 'epoch': 1.5}

 15%|█▍        | 6173/41250 [14:55:07<84:20:51,  8.66s/it][2025-04-25 22:52:51,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 22:52:51,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5697.70 | bwd_inner_microstep: 5684.83 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.38
[2025-04-25 22:52:51,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5697.71 | bwd_inner: 5684.83 | bwd_allreduce: 12.84 | step: 18.38

 15%|█▍        | 6174/41250 [14:55:16<84:14:48,  8.65s/it]
                                                          
{'loss': 0.1519, 'grad_norm': 1.1046115159988403, 'learning_rate': 3.85167075274262e-05, 'epoch': 1.5}

 15%|█▍        | 6174/41250 [14:55:16<84:14:48,  8.65s/it][2025-04-25 22:52:59,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:52:59,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.78 | bwd_microstep: 5699.22 | bwd_inner_microstep: 5656.85 | bwd_allreduce_microstep: 42.33 | step_microstep: 18.57
[2025-04-25 22:52:59,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.78 | bwd: 5699.24 | bwd_inner: 5656.85 | bwd_allreduce: 42.35 | step: 18.57

 15%|█▍        | 6175/41250 [14:55:25<84:10:02,  8.64s/it]
                                                          
{'loss': 0.0642, 'grad_norm': 1.2515075206756592, 'learning_rate': 3.851611400196793e-05, 'epoch': 1.5}

 15%|█▍        | 6175/41250 [14:55:25<84:10:02,  8.64s/it][2025-04-25 22:53:08,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:53:08,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.17 | bwd_microstep: 5723.79 | bwd_inner_microstep: 5692.65 | bwd_allreduce_microstep: 31.10 | step_microstep: 18.38
[2025-04-25 22:53:08,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.17 | bwd: 5723.80 | bwd_inner: 5692.65 | bwd_allreduce: 31.12 | step: 18.39

 15%|█▍        | 6176/41250 [14:55:33<84:14:45,  8.65s/it]
                                                          
{'loss': 0.0932, 'grad_norm': 1.2805044651031494, 'learning_rate': 3.851552036236145e-05, 'epoch': 1.5}

 15%|█▍        | 6176/41250 [14:55:33<84:14:45,  8.65s/it][2025-04-25 22:53:17,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:53:17,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.05 | bwd_microstep: 5711.87 | bwd_inner_microstep: 5661.75 | bwd_allreduce_microstep: 50.08 | step_microstep: 18.17
[2025-04-25 22:53:17,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.05 | bwd: 5711.88 | bwd_inner: 5661.74 | bwd_allreduce: 50.10 | step: 18.17

 15%|█▍        | 6177/41250 [14:55:42<84:10:58,  8.64s/it]
                                                          
{'loss': 0.2782, 'grad_norm': 3.670531749725342, 'learning_rate': 3.851492660861043e-05, 'epoch': 1.5}

 15%|█▍        | 6177/41250 [14:55:42<84:10:58,  8.64s/it][2025-04-25 22:53:25,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 22:53:25,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.10 | bwd_microstep: 5767.07 | bwd_inner_microstep: 5662.38 | bwd_allreduce_microstep: 104.65 | step_microstep: 18.46
[2025-04-25 22:53:25,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.10 | bwd: 5767.08 | bwd_inner: 5662.38 | bwd_allreduce: 104.66 | step: 18.46

 15%|█▍        | 6178/41250 [14:55:51<84:19:57,  8.66s/it]
                                                          
{'loss': 0.1357, 'grad_norm': 4.0106892585754395, 'learning_rate': 3.8514332740718534e-05, 'epoch': 1.5}

 15%|█▍        | 6178/41250 [14:55:51<84:19:57,  8.66s/it][2025-04-25 22:53:34,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:53:34,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.92 | bwd_microstep: 5768.76 | bwd_inner_microstep: 5666.39 | bwd_allreduce_microstep: 102.32 | step_microstep: 18.44
[2025-04-25 22:53:34,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5768.77 | bwd_inner: 5666.39 | bwd_allreduce: 102.34 | step: 18.44

 15%|█▍        | 6179/41250 [14:55:59<84:23:52,  8.66s/it]
                                                          
{'loss': 0.1105, 'grad_norm': 1.5755935907363892, 'learning_rate': 3.851373875868942e-05, 'epoch': 1.5}

 15%|█▍        | 6179/41250 [14:55:59<84:23:52,  8.66s/it][2025-04-25 22:53:43,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 22:53:43,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.11 | bwd_microstep: 5768.47 | bwd_inner_microstep: 5657.11 | bwd_allreduce_microstep: 111.31 | step_microstep: 18.82
[2025-04-25 22:53:43,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.11 | bwd: 5768.48 | bwd_inner: 5657.11 | bwd_allreduce: 111.33 | step: 18.82

 15%|█▍        | 6180/41250 [14:56:08<84:27:52,  8.67s/it]
                                                          
{'loss': 0.1273, 'grad_norm': 2.3600151538848877, 'learning_rate': 3.8513144662526745e-05, 'epoch': 1.5}

 15%|█▍        | 6180/41250 [14:56:08<84:27:52,  8.67s/it][2025-04-25 22:53:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:53:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.72 | bwd_microstep: 5769.56 | bwd_inner_microstep: 5661.46 | bwd_allreduce_microstep: 108.05 | step_microstep: 19.06
[2025-04-25 22:53:51,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.72 | bwd: 5769.58 | bwd_inner: 5661.46 | bwd_allreduce: 108.07 | step: 19.06

 15%|█▍        | 6181/41250 [14:56:17<84:30:25,  8.68s/it]
                                                          
{'loss': 0.3245, 'grad_norm': 3.9293692111968994, 'learning_rate': 3.8512550452234175e-05, 'epoch': 1.5}

 15%|█▍        | 6181/41250 [14:56:17<84:30:25,  8.68s/it][2025-04-25 22:54:00,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:54:00,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.07 | bwd_microstep: 5715.67 | bwd_inner_microstep: 5702.97 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.52
[2025-04-25 22:54:00,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.07 | bwd: 5715.68 | bwd_inner: 5702.97 | bwd_allreduce: 12.67 | step: 18.53

 15%|█▍        | 6182/41250 [14:56:25<84:27:37,  8.67s/it]
                                                          
{'loss': 0.2518, 'grad_norm': 5.110779762268066, 'learning_rate': 3.851195612781537e-05, 'epoch': 1.5}

 15%|█▍        | 6182/41250 [14:56:25<84:27:37,  8.67s/it][2025-04-25 22:54:09,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 22:54:09,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5713.62 | bwd_inner_microstep: 5663.31 | bwd_allreduce_microstep: 50.26 | step_microstep: 18.89
[2025-04-25 22:54:09,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5713.63 | bwd_inner: 5663.31 | bwd_allreduce: 50.28 | step: 18.89

 15%|█▍        | 6183/41250 [14:56:34<84:20:50,  8.66s/it]
                                                          
{'loss': 0.2755, 'grad_norm': 2.525625228881836, 'learning_rate': 3.8511361689274e-05, 'epoch': 1.5}

 15%|█▍        | 6183/41250 [14:56:34<84:20:50,  8.66s/it][2025-04-25 22:54:17,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-25 22:54:17,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.25 | bwd_microstep: 5752.52 | bwd_inner_microstep: 5689.01 | bwd_allreduce_microstep: 63.47 | step_microstep: 18.74
[2025-04-25 22:54:17,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5752.54 | bwd_inner: 5689.01 | bwd_allreduce: 63.49 | step: 18.74

 15%|█▍        | 6184/41250 [14:56:43<84:25:45,  8.67s/it]
                                                          
{'loss': 0.1363, 'grad_norm': 2.4097681045532227, 'learning_rate': 3.851076713661372e-05, 'epoch': 1.5}

 15%|█▍        | 6184/41250 [14:56:43<84:25:45,  8.67s/it][2025-04-25 22:54:26,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:54:26,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.35 | bwd_microstep: 5790.25 | bwd_inner_microstep: 5644.08 | bwd_allreduce_microstep: 146.13 | step_microstep: 18.05
[2025-04-25 22:54:26,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.35 | bwd: 5790.27 | bwd_inner: 5644.08 | bwd_allreduce: 146.15 | step: 18.05

 15%|█▍        | 6185/41250 [14:56:51<84:32:06,  8.68s/it]
                                                          
{'loss': 0.1014, 'grad_norm': 2.7060701847076416, 'learning_rate': 3.851017246983821e-05, 'epoch': 1.5}

 15%|█▍        | 6185/41250 [14:56:51<84:32:06,  8.68s/it][2025-04-25 22:54:35,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 22:54:35,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.62 | bwd_microstep: 5898.22 | bwd_inner_microstep: 5659.24 | bwd_allreduce_microstep: 238.94 | step_microstep: 18.64
[2025-04-25 22:54:35,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.62 | bwd: 5898.24 | bwd_inner: 5659.24 | bwd_allreduce: 238.96 | step: 18.64

 15%|█▍        | 6186/41250 [14:57:00<84:58:02,  8.72s/it]
                                                          
{'loss': 0.0554, 'grad_norm': 1.121355652809143, 'learning_rate': 3.850957768895112e-05, 'epoch': 1.5}

 15%|█▍        | 6186/41250 [14:57:00<84:58:02,  8.72s/it][2025-04-25 22:54:44,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-25 22:54:44,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.30 | bwd_microstep: 5790.61 | bwd_inner_microstep: 5666.06 | bwd_allreduce_microstep: 124.50 | step_microstep: 18.71
[2025-04-25 22:54:44,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.30 | bwd: 5790.63 | bwd_inner: 5666.06 | bwd_allreduce: 124.52 | step: 18.72

 15%|█▍        | 6187/41250 [14:57:09<84:55:18,  8.72s/it]
                                                          
{'loss': 0.2221, 'grad_norm': 1.7660428285598755, 'learning_rate': 3.8508982793956126e-05, 'epoch': 1.5}

 15%|█▍        | 6187/41250 [14:57:09<84:55:18,  8.72s/it][2025-04-25 22:54:52,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.22 | optimizer_step: 0.99
[2025-04-25 22:54:52,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.79 | bwd_microstep: 5729.95 | bwd_inner_microstep: 5716.68 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.63
[2025-04-25 22:54:52,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.79 | bwd: 5729.96 | bwd_inner: 5716.68 | bwd_allreduce: 13.24 | step: 19.63

 15%|█▌        | 6188/41250 [14:57:18<84:46:23,  8.70s/it]
                                                          
{'loss': 0.1031, 'grad_norm': 1.3913403749465942, 'learning_rate': 3.85083877848569e-05, 'epoch': 1.5}

 15%|█▌        | 6188/41250 [14:57:18<84:46:23,  8.70s/it][2025-04-25 22:55:01,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-25 22:55:01,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.49 | bwd_microstep: 5713.85 | bwd_inner_microstep: 5701.03 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.02
[2025-04-25 22:55:01,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.49 | bwd: 5713.87 | bwd_inner: 5701.02 | bwd_allreduce: 12.80 | step: 19.02

 15%|█▌        | 6189/41250 [14:57:26<84:35:55,  8.69s/it]
                                                          
{'loss': 0.1866, 'grad_norm': 2.446626663208008, 'learning_rate': 3.850779266165709e-05, 'epoch': 1.5}

 15%|█▌        | 6189/41250 [14:57:26<84:35:55,  8.69s/it][2025-04-25 22:55:10,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.29 | optimizer_step: 0.95
[2025-04-25 22:55:10,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.11 | bwd_microstep: 5785.40 | bwd_inner_microstep: 5706.18 | bwd_allreduce_microstep: 79.17 | step_microstep: 19.48
[2025-04-25 22:55:10,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.11 | bwd: 5785.42 | bwd_inner: 5706.18 | bwd_allreduce: 79.20 | step: 19.48

 15%|█▌        | 6190/41250 [14:57:35<84:42:32,  8.70s/it]
                                                          
{'loss': 0.1667, 'grad_norm': 1.8829782009124756, 'learning_rate': 3.850719742436039e-05, 'epoch': 1.5}

 15%|█▌        | 6190/41250 [14:57:35<84:42:32,  8.70s/it][2025-04-25 22:55:18,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.17 | optimizer_step: 1.11
[2025-04-25 22:55:18,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.73 | bwd_microstep: 5761.81 | bwd_inner_microstep: 5713.20 | bwd_allreduce_microstep: 48.55 | step_microstep: 19.84
[2025-04-25 22:55:18,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.73 | bwd: 5761.82 | bwd_inner: 5713.20 | bwd_allreduce: 48.57 | step: 19.84

 15%|█▌        | 6191/41250 [14:57:44<84:43:23,  8.70s/it]
                                                          
{'loss': 0.3206, 'grad_norm': 2.4337284564971924, 'learning_rate': 3.850660207297046e-05, 'epoch': 1.5}

 15%|█▌        | 6191/41250 [14:57:44<84:43:23,  8.70s/it][2025-04-25 22:55:27,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 22:55:27,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.62 | bwd_microstep: 5764.58 | bwd_inner_microstep: 5687.50 | bwd_allreduce_microstep: 77.03 | step_microstep: 19.04
[2025-04-25 22:55:27,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.62 | bwd: 5764.59 | bwd_inner: 5687.50 | bwd_allreduce: 77.05 | step: 19.04

 15%|█▌        | 6192/41250 [14:57:52<84:43:27,  8.70s/it]
                                                          
{'loss': 0.0805, 'grad_norm': 3.823103904724121, 'learning_rate': 3.850600660749096e-05, 'epoch': 1.5}

 15%|█▌        | 6192/41250 [14:57:52<84:43:27,  8.70s/it][2025-04-25 22:55:36,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.36 | optimizer_step: 1.07
[2025-04-25 22:55:36,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.61 | bwd_microstep: 5729.05 | bwd_inner_microstep: 5697.38 | bwd_allreduce_microstep: 31.60 | step_microstep: 20.46
[2025-04-25 22:55:36,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.61 | bwd: 5729.06 | bwd_inner: 5697.38 | bwd_allreduce: 31.63 | step: 20.46

 15%|█▌        | 6193/41250 [14:58:01<84:37:44,  8.69s/it]
                                                          
{'loss': 0.0674, 'grad_norm': 1.1589604616165161, 'learning_rate': 3.8505411027925574e-05, 'epoch': 1.5}

 15%|█▌        | 6193/41250 [14:58:01<84:37:44,  8.69s/it][2025-04-25 22:55:44,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:55:44,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.71 | bwd_microstep: 5722.19 | bwd_inner_microstep: 5709.46 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.10
[2025-04-25 22:55:44,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.71 | bwd: 5722.20 | bwd_inner: 5709.46 | bwd_allreduce: 12.70 | step: 19.10

 15%|█▌        | 6194/41250 [14:58:10<84:31:22,  8.68s/it]
                                                          
{'loss': 0.2994, 'grad_norm': 1.6273412704467773, 'learning_rate': 3.850481533427797e-05, 'epoch': 1.5}

 15%|█▌        | 6194/41250 [14:58:10<84:31:22,  8.68s/it][2025-04-25 22:55:53,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-25 22:55:53,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.47 | bwd_microstep: 5778.57 | bwd_inner_microstep: 5672.10 | bwd_allreduce_microstep: 106.42 | step_microstep: 19.27
[2025-04-25 22:55:53,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.47 | bwd: 5778.58 | bwd_inner: 5672.10 | bwd_allreduce: 106.44 | step: 19.27

 15%|█▌        | 6195/41250 [14:58:18<84:35:29,  8.69s/it]
                                                          
{'loss': 0.0507, 'grad_norm': 1.2732257843017578, 'learning_rate': 3.850421952655181e-05, 'epoch': 1.5}

 15%|█▌        | 6195/41250 [14:58:18<84:35:29,  8.69s/it][2025-04-25 22:56:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:56:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.62 | bwd_microstep: 5780.91 | bwd_inner_microstep: 5657.28 | bwd_allreduce_microstep: 123.59 | step_microstep: 18.66
[2025-04-25 22:56:02,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.62 | bwd: 5780.93 | bwd_inner: 5657.28 | bwd_allreduce: 123.61 | step: 18.67

 15%|█▌        | 6196/41250 [14:58:27<84:35:23,  8.69s/it]
                                                          
{'loss': 0.1656, 'grad_norm': 2.6563336849212646, 'learning_rate': 3.850362360475079e-05, 'epoch': 1.5}

 15%|█▌        | 6196/41250 [14:58:27<84:35:23,  8.69s/it][2025-04-25 22:56:11,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 22:56:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.34 | bwd_microstep: 5863.73 | bwd_inner_microstep: 5850.67 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.69
[2025-04-25 22:56:11,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.34 | bwd: 5863.74 | bwd_inner: 5850.67 | bwd_allreduce: 13.03 | step: 19.69

 15%|█▌        | 6197/41250 [14:58:36<85:09:01,  8.75s/it]
                                                          
{'loss': 0.1175, 'grad_norm': 2.104738473892212, 'learning_rate': 3.850302756887856e-05, 'epoch': 1.5}

 15%|█▌        | 6197/41250 [14:58:36<85:09:01,  8.75s/it][2025-04-25 22:56:19,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:56:19,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.89 | bwd_microstep: 5725.03 | bwd_inner_microstep: 5712.31 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.77
[2025-04-25 22:56:19,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.89 | bwd: 5725.05 | bwd_inner: 5712.31 | bwd_allreduce: 12.70 | step: 18.78

 15%|█▌        | 6198/41250 [14:58:45<84:54:38,  8.72s/it]
                                                          
{'loss': 0.1689, 'grad_norm': 2.7549808025360107, 'learning_rate': 3.850243141893881e-05, 'epoch': 1.5}

 15%|█▌        | 6198/41250 [14:58:45<84:54:38,  8.72s/it][2025-04-25 22:56:28,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:56:28,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.47 | bwd_microstep: 5879.23 | bwd_inner_microstep: 5708.61 | bwd_allreduce_microstep: 170.58 | step_microstep: 18.82
[2025-04-25 22:56:28,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.47 | bwd: 5879.25 | bwd_inner: 5708.61 | bwd_allreduce: 170.60 | step: 18.82

 15%|█▌        | 6199/41250 [14:58:53<85:10:50,  8.75s/it]
                                                          
{'loss': 0.1928, 'grad_norm': 1.6946865320205688, 'learning_rate': 3.850183515493521e-05, 'epoch': 1.5}

 15%|█▌        | 6199/41250 [14:58:53<85:10:50,  8.75s/it][2025-04-25 22:56:37,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:56:37,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.41 | bwd_microstep: 5721.65 | bwd_inner_microstep: 5648.63 | bwd_allreduce_microstep: 72.98 | step_microstep: 18.87
[2025-04-25 22:56:37,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.41 | bwd: 5721.66 | bwd_inner: 5648.63 | bwd_allreduce: 72.99 | step: 18.88

 15%|█▌        | 6200/41250 [14:59:02<84:49:51,  8.71s/it]
                                                          
{'loss': 0.0957, 'grad_norm': 1.8498295545578003, 'learning_rate': 3.850123877687143e-05, 'epoch': 1.5}

 15%|█▌        | 6200/41250 [14:59:02<84:49:51,  8.71s/it][2025-04-25 22:56:45,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-25 22:56:45,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.69 | bwd_microstep: 5717.00 | bwd_inner_microstep: 5704.12 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.38
[2025-04-25 22:56:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.69 | bwd: 5717.02 | bwd_inner: 5704.12 | bwd_allreduce: 12.86 | step: 19.38

 15%|█▌        | 6201/41250 [14:59:11<84:38:48,  8.69s/it]
                                                          
{'loss': 0.1657, 'grad_norm': 2.120957612991333, 'learning_rate': 3.8500642284751156e-05, 'epoch': 1.5}

 15%|█▌        | 6201/41250 [14:59:11<84:38:48,  8.69s/it][2025-04-25 22:56:54,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 22:56:54,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.69 | bwd_microstep: 5761.82 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 59.83 | step_microstep: 18.82
[2025-04-25 22:56:54,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.69 | bwd: 5761.83 | bwd_inner: 5701.94 | bwd_allreduce: 59.85 | step: 18.82

 15%|█▌        | 6202/41250 [14:59:19<84:40:46,  8.70s/it]
                                                          
{'loss': 0.0903, 'grad_norm': 1.5932804346084595, 'learning_rate': 3.850004567857806e-05, 'epoch': 1.5}

 15%|█▌        | 6202/41250 [14:59:19<84:40:46,  8.70s/it][2025-04-25 22:57:03,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 22:57:03,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.98 | bwd_microstep: 5785.30 | bwd_inner_microstep: 5642.03 | bwd_allreduce_microstep: 143.22 | step_microstep: 18.69
[2025-04-25 22:57:03,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.98 | bwd: 5785.31 | bwd_inner: 5642.03 | bwd_allreduce: 143.24 | step: 18.70

 15%|█▌        | 6203/41250 [14:59:28<84:40:30,  8.70s/it]
                                                          
{'loss': 0.1424, 'grad_norm': 1.3928691148757935, 'learning_rate': 3.849944895835582e-05, 'epoch': 1.5}

 15%|█▌        | 6203/41250 [14:59:28<84:40:30,  8.70s/it][2025-04-25 22:57:12,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:57:12,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.19 | bwd_microstep: 5769.52 | bwd_inner_microstep: 5757.01 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.66
[2025-04-25 22:57:12,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.19 | bwd: 5769.53 | bwd_inner: 5757.01 | bwd_allreduce: 12.48 | step: 18.66

 15%|█▌        | 6204/41250 [14:59:37<84:45:31,  8.71s/it]
                                                          
{'loss': 0.2484, 'grad_norm': 6.391931056976318, 'learning_rate': 3.849885212408811e-05, 'epoch': 1.5}

 15%|█▌        | 6204/41250 [14:59:37<84:45:31,  8.71s/it][2025-04-25 22:57:20,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-25 22:57:20,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5716.48 | bwd_inner_microstep: 5703.69 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.70
[2025-04-25 22:57:20,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5716.49 | bwd_inner: 5703.69 | bwd_allreduce: 12.76 | step: 18.70

 15%|█▌        | 6205/41250 [14:59:45<84:35:45,  8.69s/it]
                                                          
{'loss': 0.1074, 'grad_norm': 1.4411994218826294, 'learning_rate': 3.849825517577863e-05, 'epoch': 1.5}

 15%|█▌        | 6205/41250 [14:59:45<84:35:45,  8.69s/it][2025-04-25 22:57:29,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 22:57:29,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.29 | bwd_microstep: 5809.55 | bwd_inner_microstep: 5645.22 | bwd_allreduce_microstep: 164.27 | step_microstep: 18.98
[2025-04-25 22:57:29,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.29 | bwd: 5809.57 | bwd_inner: 5645.22 | bwd_allreduce: 164.29 | step: 18.98

 15%|█▌        | 6206/41250 [14:59:54<84:40:33,  8.70s/it]
                                                          
{'loss': 0.1777, 'grad_norm': 2.9328393936157227, 'learning_rate': 3.849765811343103e-05, 'epoch': 1.5}

 15%|█▌        | 6206/41250 [14:59:54<84:40:33,  8.70s/it][2025-04-25 22:57:38,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:57:38,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5766.96 | bwd_inner_microstep: 5652.30 | bwd_allreduce_microstep: 114.61 | step_microstep: 18.79
[2025-04-25 22:57:38,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5766.98 | bwd_inner: 5652.30 | bwd_allreduce: 114.63 | step: 18.79

 15%|█▌        | 6207/41250 [15:00:03<84:36:24,  8.69s/it]
                                                          
{'loss': 0.0903, 'grad_norm': 1.5230047702789307, 'learning_rate': 3.8497060937049014e-05, 'epoch': 1.5}

 15%|█▌        | 6207/41250 [15:00:03<84:36:24,  8.69s/it][2025-04-25 22:57:46,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 22:57:46,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.04 | bwd_microstep: 5767.73 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 119.23 | step_microstep: 18.69
[2025-04-25 22:57:46,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.04 | bwd: 5767.74 | bwd_inner: 5648.45 | bwd_allreduce: 119.25 | step: 18.69

 15%|█▌        | 6208/41250 [15:00:12<84:35:02,  8.69s/it]
                                                          
{'loss': 0.0878, 'grad_norm': 1.7612193822860718, 'learning_rate': 3.849646364663626e-05, 'epoch': 1.5}

 15%|█▌        | 6208/41250 [15:00:12<84:35:02,  8.69s/it][2025-04-25 22:57:55,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 0.91
[2025-04-25 22:57:55,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5738.79 | bwd_inner_microstep: 5687.37 | bwd_allreduce_microstep: 51.37 | step_microstep: 18.40
[2025-04-25 22:57:55,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5738.80 | bwd_inner: 5687.37 | bwd_allreduce: 51.39 | step: 18.40

 15%|█▌        | 6209/41250 [15:00:20<84:31:01,  8.68s/it]
                                                          
{'loss': 0.1453, 'grad_norm': 1.7372223138809204, 'learning_rate': 3.849586624219644e-05, 'epoch': 1.51}

 15%|█▌        | 6209/41250 [15:00:20<84:31:01,  8.68s/it][2025-04-25 22:58:04,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 22:58:04,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.27 | bwd_microstep: 5894.88 | bwd_inner_microstep: 5649.45 | bwd_allreduce_microstep: 245.38 | step_microstep: 18.59
[2025-04-25 22:58:04,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.27 | bwd: 5894.89 | bwd_inner: 5649.45 | bwd_allreduce: 245.40 | step: 18.59

 15%|█▌        | 6210/41250 [15:00:29<84:51:53,  8.72s/it]
                                                          
{'loss': 0.1915, 'grad_norm': 2.5882668495178223, 'learning_rate': 3.849526872373324e-05, 'epoch': 1.51}

 15%|█▌        | 6210/41250 [15:00:29<84:51:53,  8.72s/it][2025-04-25 22:58:12,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-25 22:58:12,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.42 | bwd_microstep: 5747.32 | bwd_inner_microstep: 5691.38 | bwd_allreduce_microstep: 55.89 | step_microstep: 18.86
[2025-04-25 22:58:12,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.42 | bwd: 5747.33 | bwd_inner: 5691.38 | bwd_allreduce: 55.91 | step: 18.86

 15%|█▌        | 6211/41250 [15:00:38<84:43:38,  8.71s/it]
                                                          
{'loss': 0.0652, 'grad_norm': 1.0084086656570435, 'learning_rate': 3.849467109125034e-05, 'epoch': 1.51}

 15%|█▌        | 6211/41250 [15:00:38<84:43:38,  8.71s/it][2025-04-25 22:58:21,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 22:58:21,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.72 | bwd_microstep: 5709.47 | bwd_inner_microstep: 5636.97 | bwd_allreduce_microstep: 72.45 | step_microstep: 18.70
[2025-04-25 22:58:21,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.72 | bwd: 5709.48 | bwd_inner: 5636.97 | bwd_allreduce: 72.47 | step: 18.71

 15%|█▌        | 6212/41250 [15:00:46<84:27:11,  8.68s/it]
                                                          
{'loss': 0.143, 'grad_norm': 4.994961261749268, 'learning_rate': 3.849407334475144e-05, 'epoch': 1.51}

 15%|█▌        | 6212/41250 [15:00:46<84:27:11,  8.68s/it][2025-04-25 22:58:30,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 22:58:30,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.09 | bwd_microstep: 5710.26 | bwd_inner_microstep: 5697.43 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.69
[2025-04-25 22:58:30,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.09 | bwd: 5710.28 | bwd_inner: 5697.43 | bwd_allreduce: 12.80 | step: 18.69

 15%|█▌        | 6213/41250 [15:00:55<84:19:58,  8.67s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 1.656506896018982, 'learning_rate': 3.8493475484240216e-05, 'epoch': 1.51}

 15%|█▌        | 6213/41250 [15:00:55<84:19:58,  8.67s/it][2025-04-25 22:58:38,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:58:38,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.15 | bwd_microstep: 5697.87 | bwd_inner_microstep: 5685.19 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.67
[2025-04-25 22:58:38,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.15 | bwd: 5697.89 | bwd_inner: 5685.19 | bwd_allreduce: 12.66 | step: 18.68

 15%|█▌        | 6214/41250 [15:01:04<84:12:37,  8.65s/it]
                                                          
{'loss': 0.2776, 'grad_norm': 2.6605417728424072, 'learning_rate': 3.8492877509720354e-05, 'epoch': 1.51}

 15%|█▌        | 6214/41250 [15:01:04<84:12:37,  8.65s/it][2025-04-25 22:58:47,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:58:47,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.57 | bwd_microstep: 5754.30 | bwd_inner_microstep: 5680.99 | bwd_allreduce_microstep: 73.26 | step_microstep: 18.58
[2025-04-25 22:58:47,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.57 | bwd: 5754.32 | bwd_inner: 5680.99 | bwd_allreduce: 73.28 | step: 18.59

 15%|█▌        | 6215/41250 [15:01:12<84:16:49,  8.66s/it]
                                                          
{'loss': 0.0833, 'grad_norm': 0.9651370644569397, 'learning_rate': 3.849227942119554e-05, 'epoch': 1.51}

 15%|█▌        | 6215/41250 [15:01:12<84:16:49,  8.66s/it][2025-04-25 22:58:56,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:58:56,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.65 | bwd_microstep: 5786.31 | bwd_inner_microstep: 5649.31 | bwd_allreduce_microstep: 136.96 | step_microstep: 18.78
[2025-04-25 22:58:56,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.65 | bwd: 5786.33 | bwd_inner: 5649.31 | bwd_allreduce: 136.97 | step: 18.78

 15%|█▌        | 6216/41250 [15:01:21<84:22:35,  8.67s/it]
                                                          
{'loss': 0.054, 'grad_norm': 1.4919657707214355, 'learning_rate': 3.849168121866946e-05, 'epoch': 1.51}

 15%|█▌        | 6216/41250 [15:01:21<84:22:35,  8.67s/it][2025-04-25 22:59:04,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:59:04,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.39 | bwd_microstep: 5697.51 | bwd_inner_microstep: 5684.73 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.72
[2025-04-25 22:59:04,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.39 | bwd: 5697.52 | bwd_inner: 5684.73 | bwd_allreduce: 12.75 | step: 18.72

 15%|█▌        | 6217/41250 [15:01:30<84:13:41,  8.66s/it]
                                                          
{'loss': 0.038, 'grad_norm': 0.8252747654914856, 'learning_rate': 3.84910829021458e-05, 'epoch': 1.51}

 15%|█▌        | 6217/41250 [15:01:30<84:13:41,  8.66s/it][2025-04-25 22:59:13,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 22:59:13,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.50 | bwd_microstep: 5737.08 | bwd_inner_microstep: 5700.95 | bwd_allreduce_microstep: 36.09 | step_microstep: 18.88
[2025-04-25 22:59:13,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.50 | bwd: 5737.09 | bwd_inner: 5700.95 | bwd_allreduce: 36.11 | step: 18.88

 15%|█▌        | 6218/41250 [15:01:38<84:16:10,  8.66s/it]
                                                          
{'loss': 0.2056, 'grad_norm': 3.8313026428222656, 'learning_rate': 3.849048447162825e-05, 'epoch': 1.51}

 15%|█▌        | 6218/41250 [15:01:38<84:16:10,  8.66s/it][2025-04-25 22:59:22,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 22:59:22,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.63 | bwd_microstep: 5757.20 | bwd_inner_microstep: 5691.55 | bwd_allreduce_microstep: 65.60 | step_microstep: 18.62
[2025-04-25 22:59:22,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.63 | bwd: 5757.21 | bwd_inner: 5691.55 | bwd_allreduce: 65.62 | step: 18.62

 15%|█▌        | 6219/41250 [15:01:47<84:20:20,  8.67s/it]
                                                          
{'loss': 0.0492, 'grad_norm': 0.670257568359375, 'learning_rate': 3.84898859271205e-05, 'epoch': 1.51}

 15%|█▌        | 6219/41250 [15:01:47<84:20:20,  8.67s/it][2025-04-25 22:59:30,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 22:59:30,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.08 | bwd_microstep: 5685.85 | bwd_inner_microstep: 5650.72 | bwd_allreduce_microstep: 35.09 | step_microstep: 19.19
[2025-04-25 22:59:30,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.08 | bwd: 5685.87 | bwd_inner: 5650.72 | bwd_allreduce: 35.11 | step: 19.19

 15%|█▌        | 6220/41250 [15:01:56<84:07:44,  8.65s/it]
                                                          
{'loss': 0.1384, 'grad_norm': 2.2895395755767822, 'learning_rate': 3.848928726862624e-05, 'epoch': 1.51}

 15%|█▌        | 6220/41250 [15:01:56<84:07:44,  8.65s/it][2025-04-25 22:59:39,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 22:59:39,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.27 | bwd_microstep: 5698.90 | bwd_inner_microstep: 5651.05 | bwd_allreduce_microstep: 47.80 | step_microstep: 18.54
[2025-04-25 22:59:39,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.27 | bwd: 5698.91 | bwd_inner: 5651.05 | bwd_allreduce: 47.82 | step: 18.54

 15%|█▌        | 6221/41250 [15:02:04<84:00:47,  8.63s/it]
                                                          
{'loss': 0.0506, 'grad_norm': 1.690800428390503, 'learning_rate': 3.848868849614916e-05, 'epoch': 1.51}

 15%|█▌        | 6221/41250 [15:02:04<84:00:47,  8.63s/it][2025-04-25 22:59:47,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 22:59:47,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.06 | bwd_microstep: 5730.94 | bwd_inner_microstep: 5689.88 | bwd_allreduce_microstep: 41.02 | step_microstep: 18.32
[2025-04-25 22:59:47,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.06 | bwd: 5730.96 | bwd_inner: 5689.88 | bwd_allreduce: 41.03 | step: 18.33

 15%|█▌        | 6222/41250 [15:02:13<84:06:55,  8.64s/it]
                                                          
{'loss': 0.0728, 'grad_norm': 1.4212366342544556, 'learning_rate': 3.848808960969296e-05, 'epoch': 1.51}

 15%|█▌        | 6222/41250 [15:02:13<84:06:55,  8.64s/it][2025-04-25 22:59:56,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 22:59:56,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.93 | bwd_microstep: 5765.17 | bwd_inner_microstep: 5665.15 | bwd_allreduce_microstep: 99.98 | step_microstep: 18.34
[2025-04-25 22:59:56,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.93 | bwd: 5765.18 | bwd_inner: 5665.15 | bwd_allreduce: 99.99 | step: 18.35

 15%|█▌        | 6223/41250 [15:02:21<84:12:52,  8.66s/it]
                                                          
{'loss': 0.135, 'grad_norm': 4.277607440948486, 'learning_rate': 3.848749060926131e-05, 'epoch': 1.51}

 15%|█▌        | 6223/41250 [15:02:21<84:12:52,  8.66s/it][2025-04-25 23:00:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-25 23:00:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.76 | bwd_microstep: 5760.56 | bwd_inner_microstep: 5657.28 | bwd_allreduce_microstep: 103.23 | step_microstep: 18.71
[2025-04-25 23:00:05,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.76 | bwd: 5760.57 | bwd_inner: 5657.28 | bwd_allreduce: 103.24 | step: 18.71

 15%|█▌        | 6224/41250 [15:02:30<84:15:31,  8.66s/it]
                                                          
{'loss': 0.2751, 'grad_norm': 3.292712688446045, 'learning_rate': 3.8486891494857926e-05, 'epoch': 1.51}

 15%|█▌        | 6224/41250 [15:02:30<84:15:31,  8.66s/it][2025-04-25 23:00:14,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:00:14,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5881.22 | bwd_inner_microstep: 5706.19 | bwd_allreduce_microstep: 174.98 | step_microstep: 18.63
[2025-04-25 23:00:14,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.67 | bwd: 5881.24 | bwd_inner: 5706.19 | bwd_allreduce: 175.00 | step: 18.64

 15%|█▌        | 6225/41250 [15:02:39<84:42:53,  8.71s/it]
                                                          
{'loss': 0.1274, 'grad_norm': 2.5201358795166016, 'learning_rate': 3.8486292266486495e-05, 'epoch': 1.51}

 15%|█▌        | 6225/41250 [15:02:39<84:42:53,  8.71s/it][2025-04-25 23:00:22,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 23:00:22,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.80 | bwd_microstep: 5720.36 | bwd_inner_microstep: 5707.65 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.03
[2025-04-25 23:00:22,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.80 | bwd: 5720.37 | bwd_inner: 5707.65 | bwd_allreduce: 12.68 | step: 19.03

 15%|█▌        | 6226/41250 [15:02:48<84:33:34,  8.69s/it]
                                                          
{'loss': 0.0643, 'grad_norm': 1.8000659942626953, 'learning_rate': 3.8485692924150704e-05, 'epoch': 1.51}

 15%|█▌        | 6226/41250 [15:02:48<84:33:34,  8.69s/it][2025-04-25 23:00:31,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 23:00:31,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.33 | bwd_microstep: 5708.65 | bwd_inner_microstep: 5695.54 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.93
[2025-04-25 23:00:31,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.33 | bwd: 5708.66 | bwd_inner: 5695.54 | bwd_allreduce: 13.08 | step: 18.93

 15%|█▌        | 6227/41250 [15:02:56<84:23:31,  8.67s/it]
                                                          
{'loss': 0.0698, 'grad_norm': 1.0044256448745728, 'learning_rate': 3.848509346785425e-05, 'epoch': 1.51}

 15%|█▌        | 6227/41250 [15:02:56<84:23:31,  8.67s/it][2025-04-25 23:00:40,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:00:40,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.33 | bwd_microstep: 5728.56 | bwd_inner_microstep: 5657.89 | bwd_allreduce_microstep: 70.63 | step_microstep: 18.66
[2025-04-25 23:00:40,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.33 | bwd: 5728.57 | bwd_inner: 5657.89 | bwd_allreduce: 70.64 | step: 18.67

 15%|█▌        | 6228/41250 [15:03:05<84:17:34,  8.66s/it]
                                                          
{'loss': 0.023, 'grad_norm': 0.3697053790092468, 'learning_rate': 3.848449389760083e-05, 'epoch': 1.51}

 15%|█▌        | 6228/41250 [15:03:05<84:17:34,  8.66s/it][2025-04-25 23:00:48,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:00:48,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.23 | bwd_microstep: 5740.21 | bwd_inner_microstep: 5667.74 | bwd_allreduce_microstep: 72.43 | step_microstep: 18.39
[2025-04-25 23:00:48,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.23 | bwd: 5740.23 | bwd_inner: 5667.74 | bwd_allreduce: 72.44 | step: 18.40

 15%|█▌        | 6229/41250 [15:03:14<84:21:16,  8.67s/it]
                                                          
{'loss': 0.2478, 'grad_norm': 2.7276980876922607, 'learning_rate': 3.8483894213394145e-05, 'epoch': 1.51}

 15%|█▌        | 6229/41250 [15:03:14<84:21:16,  8.67s/it][2025-04-25 23:00:57,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:00:57,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.34 | bwd_microstep: 5717.77 | bwd_inner_microstep: 5704.93 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.78
[2025-04-25 23:00:57,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.34 | bwd: 5717.78 | bwd_inner: 5704.94 | bwd_allreduce: 12.81 | step: 18.78

 15%|█▌        | 6230/41250 [15:03:22<84:18:18,  8.67s/it]
                                                          
{'loss': 0.135, 'grad_norm': 1.7075779438018799, 'learning_rate': 3.848329441523789e-05, 'epoch': 1.51}

 15%|█▌        | 6230/41250 [15:03:22<84:18:18,  8.67s/it][2025-04-25 23:01:06,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-25 23:01:06,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.03 | bwd_microstep: 5716.24 | bwd_inner_microstep: 5702.90 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.24
[2025-04-25 23:01:06,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.03 | bwd: 5716.26 | bwd_inner: 5702.90 | bwd_allreduce: 13.32 | step: 19.24

 15%|█▌        | 6231/41250 [15:03:31<84:17:20,  8.67s/it]
                                                          
{'loss': 0.2411, 'grad_norm': 2.5513272285461426, 'learning_rate': 3.8482694503135755e-05, 'epoch': 1.51}

 15%|█▌        | 6231/41250 [15:03:31<84:17:20,  8.67s/it][2025-04-25 23:01:14,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.11 | optimizer_step: 0.98
[2025-04-25 23:01:14,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.65 | bwd_microstep: 5765.22 | bwd_inner_microstep: 5669.51 | bwd_allreduce_microstep: 95.66 | step_microstep: 19.04
[2025-04-25 23:01:14,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.65 | bwd: 5765.24 | bwd_inner: 5669.51 | bwd_allreduce: 95.68 | step: 19.04

 15%|█▌        | 6232/41250 [15:03:40<84:22:47,  8.67s/it]
                                                          
{'loss': 0.1934, 'grad_norm': 1.8881112337112427, 'learning_rate': 3.848209447709144e-05, 'epoch': 1.51}

 15%|█▌        | 6232/41250 [15:03:40<84:22:47,  8.67s/it][2025-04-25 23:01:23,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:01:23,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.44 | bwd_microstep: 5765.45 | bwd_inner_microstep: 5664.18 | bwd_allreduce_microstep: 101.23 | step_microstep: 18.78
[2025-04-25 23:01:23,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.44 | bwd: 5765.47 | bwd_inner: 5664.18 | bwd_allreduce: 101.24 | step: 18.78

 15%|█▌        | 6233/41250 [15:03:48<84:25:25,  8.68s/it]
                                                          
{'loss': 0.0858, 'grad_norm': 2.4616172313690186, 'learning_rate': 3.8481494337108655e-05, 'epoch': 1.51}

 15%|█▌        | 6233/41250 [15:03:48<84:25:25,  8.68s/it][2025-04-25 23:01:32,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:01:32,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.69 | bwd_microstep: 5745.52 | bwd_inner_microstep: 5704.00 | bwd_allreduce_microstep: 41.48 | step_microstep: 18.57
[2025-04-25 23:01:32,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.69 | bwd: 5745.54 | bwd_inner: 5704.00 | bwd_allreduce: 41.50 | step: 18.58

 15%|█▌        | 6234/41250 [15:03:57<84:26:41,  8.68s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 1.4057248830795288, 'learning_rate': 3.848089408319109e-05, 'epoch': 1.51}

 15%|█▌        | 6234/41250 [15:03:57<84:26:41,  8.68s/it][2025-04-25 23:01:40,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-25 23:01:40,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.23 | bwd_microstep: 5708.89 | bwd_inner_microstep: 5696.25 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.34
[2025-04-25 23:01:40,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.23 | bwd: 5708.91 | bwd_inner: 5696.25 | bwd_allreduce: 12.62 | step: 18.34

 15%|█▌        | 6235/41250 [15:04:06<84:20:57,  8.67s/it]
                                                          
{'loss': 0.0873, 'grad_norm': 1.6314393281936646, 'learning_rate': 3.848029371534245e-05, 'epoch': 1.51}

 15%|█▌        | 6235/41250 [15:04:06<84:20:57,  8.67s/it][2025-04-25 23:01:49,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:01:49,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.98 | bwd_microstep: 5702.58 | bwd_inner_microstep: 5690.22 | bwd_allreduce_microstep: 12.32 | step_microstep: 18.56
[2025-04-25 23:01:49,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.98 | bwd: 5702.59 | bwd_inner: 5690.22 | bwd_allreduce: 12.34 | step: 18.56

 15%|█▌        | 6236/41250 [15:04:14<84:29:08,  8.69s/it]
                                                          
{'loss': 0.0632, 'grad_norm': 1.1282817125320435, 'learning_rate': 3.847969323356643e-05, 'epoch': 1.51}

 15%|█▌        | 6236/41250 [15:04:14<84:29:08,  8.69s/it][2025-04-25 23:01:58,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-25 23:01:58,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.74 | bwd_microstep: 5721.01 | bwd_inner_microstep: 5657.64 | bwd_allreduce_microstep: 63.32 | step_microstep: 18.42
[2025-04-25 23:01:58,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.74 | bwd: 5721.02 | bwd_inner: 5657.64 | bwd_allreduce: 63.34 | step: 18.42

 15%|█▌        | 6237/41250 [15:04:23<84:19:51,  8.67s/it]
                                                          
{'loss': 0.1619, 'grad_norm': 2.893524169921875, 'learning_rate': 3.847909263786674e-05, 'epoch': 1.51}

 15%|█▌        | 6237/41250 [15:04:23<84:19:51,  8.67s/it][2025-04-25 23:02:06,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-25 23:02:06,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.47 | bwd_microstep: 5774.43 | bwd_inner_microstep: 5672.92 | bwd_allreduce_microstep: 101.45 | step_microstep: 18.63
[2025-04-25 23:02:06,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.47 | bwd: 5774.45 | bwd_inner: 5672.92 | bwd_allreduce: 101.47 | step: 18.63

 15%|█▌        | 6238/41250 [15:04:32<84:25:33,  8.68s/it]
                                                          
{'loss': 0.0903, 'grad_norm': 0.982167661190033, 'learning_rate': 3.847849192824708e-05, 'epoch': 1.51}

 15%|█▌        | 6238/41250 [15:04:32<84:25:33,  8.68s/it][2025-04-25 23:02:15,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.21 | optimizer_step: 1.04
[2025-04-25 23:02:15,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.00 | bwd_microstep: 5738.08 | bwd_inner_microstep: 5716.75 | bwd_allreduce_microstep: 21.28 | step_microstep: 19.80
[2025-04-25 23:02:15,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.00 | bwd: 5738.09 | bwd_inner: 5716.75 | bwd_allreduce: 21.30 | step: 19.80

 15%|█▌        | 6239/41250 [15:04:40<84:25:22,  8.68s/it]
                                                          
{'loss': 0.0732, 'grad_norm': 1.1033720970153809, 'learning_rate': 3.847789110471115e-05, 'epoch': 1.51}

 15%|█▌        | 6239/41250 [15:04:40<84:25:22,  8.68s/it][2025-04-25 23:02:24,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 23:02:24,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5794.67 | bwd_inner_microstep: 5650.66 | bwd_allreduce_microstep: 143.96 | step_microstep: 18.69
[2025-04-25 23:02:24,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5794.69 | bwd_inner: 5650.66 | bwd_allreduce: 143.98 | step: 18.69

 15%|█▌        | 6240/41250 [15:04:49<84:29:44,  8.69s/it]
                                                          
{'loss': 0.3038, 'grad_norm': 2.864821434020996, 'learning_rate': 3.847729016726265e-05, 'epoch': 1.51}

 15%|█▌        | 6240/41250 [15:04:49<84:29:44,  8.69s/it][2025-04-25 23:02:32,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:02:32,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.81 | bwd_microstep: 5725.25 | bwd_inner_microstep: 5664.82 | bwd_allreduce_microstep: 60.38 | step_microstep: 18.62
[2025-04-25 23:02:32,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.81 | bwd: 5725.26 | bwd_inner: 5664.82 | bwd_allreduce: 60.40 | step: 18.62

 15%|█▌        | 6241/41250 [15:04:58<84:20:05,  8.67s/it]
                                                          
{'loss': 0.2516, 'grad_norm': 2.4901020526885986, 'learning_rate': 3.84766891159053e-05, 'epoch': 1.51}

 15%|█▌        | 6241/41250 [15:04:58<84:20:05,  8.67s/it][2025-04-25 23:02:41,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:02:41,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.20 | bwd_microstep: 5763.71 | bwd_inner_microstep: 5694.77 | bwd_allreduce_microstep: 68.90 | step_microstep: 18.49
[2025-04-25 23:02:41,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.20 | bwd: 5763.73 | bwd_inner: 5694.77 | bwd_allreduce: 68.92 | step: 18.50

 15%|█▌        | 6242/41250 [15:05:06<84:23:33,  8.68s/it]
                                                          
{'loss': 0.2199, 'grad_norm': 4.853008270263672, 'learning_rate': 3.847608795064279e-05, 'epoch': 1.51}

 15%|█▌        | 6242/41250 [15:05:06<84:23:33,  8.68s/it][2025-04-25 23:02:50,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 23:02:50,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.30 | bwd_microstep: 5694.09 | bwd_inner_microstep: 5681.33 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.39
[2025-04-25 23:02:50,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.30 | bwd: 5694.10 | bwd_inner: 5681.33 | bwd_allreduce: 12.73 | step: 19.39

 15%|█▌        | 6243/41250 [15:05:15<84:13:21,  8.66s/it]
                                                          
{'loss': 0.3164, 'grad_norm': 3.4501376152038574, 'learning_rate': 3.8475486671478826e-05, 'epoch': 1.51}

 15%|█▌        | 6243/41250 [15:05:15<84:13:21,  8.66s/it][2025-04-25 23:02:58,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 23:02:58,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.39 | bwd_microstep: 5749.95 | bwd_inner_microstep: 5713.54 | bwd_allreduce_microstep: 36.37 | step_microstep: 18.83
[2025-04-25 23:02:58,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.39 | bwd: 5749.96 | bwd_inner: 5713.54 | bwd_allreduce: 36.38 | step: 18.84

 15%|█▌        | 6244/41250 [15:05:24<84:18:12,  8.67s/it]
                                                          
{'loss': 0.0168, 'grad_norm': 0.4772365391254425, 'learning_rate': 3.847488527841713e-05, 'epoch': 1.51}

 15%|█▌        | 6244/41250 [15:05:24<84:18:12,  8.67s/it][2025-04-25 23:03:07,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 23:03:07,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.04 | bwd_microstep: 5750.75 | bwd_inner_microstep: 5698.88 | bwd_allreduce_microstep: 51.82 | step_microstep: 18.44
[2025-04-25 23:03:07,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.04 | bwd: 5750.76 | bwd_inner: 5698.88 | bwd_allreduce: 51.84 | step: 18.45

 15%|█▌        | 6245/41250 [15:05:32<84:19:51,  8.67s/it]
                                                          
{'loss': 0.1212, 'grad_norm': 2.0631139278411865, 'learning_rate': 3.84742837714614e-05, 'epoch': 1.51}

 15%|█▌        | 6245/41250 [15:05:32<84:19:51,  8.67s/it][2025-04-25 23:03:16,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.14 | optimizer_step: 1.10
[2025-04-25 23:03:16,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.06 | bwd_microstep: 5786.96 | bwd_inner_microstep: 5653.91 | bwd_allreduce_microstep: 132.99 | step_microstep: 19.23
[2025-04-25 23:03:16,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.06 | bwd: 5786.97 | bwd_inner: 5653.91 | bwd_allreduce: 133.01 | step: 19.23

 15%|█▌        | 6246/41250 [15:05:41<84:23:59,  8.68s/it]
                                                          
{'loss': 0.0325, 'grad_norm': 0.4841156005859375, 'learning_rate': 3.847368215061534e-05, 'epoch': 1.51}

 15%|█▌        | 6246/41250 [15:05:41<84:23:59,  8.68s/it][2025-04-25 23:03:24,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 0.95
[2025-04-25 23:03:24,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5759.05 | bwd_inner_microstep: 5677.79 | bwd_allreduce_microstep: 81.22 | step_microstep: 19.51
[2025-04-25 23:03:24,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5759.07 | bwd_inner: 5677.78 | bwd_allreduce: 81.24 | step: 19.51

 15%|█▌        | 6247/41250 [15:05:50<84:26:48,  8.69s/it]
                                                          
{'loss': 0.1117, 'grad_norm': 1.7511976957321167, 'learning_rate': 3.847308041588266e-05, 'epoch': 1.51}

 15%|█▌        | 6247/41250 [15:05:50<84:26:48,  8.69s/it][2025-04-25 23:03:33,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-25 23:03:33,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.26 | bwd_microstep: 5712.30 | bwd_inner_microstep: 5699.32 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.38
[2025-04-25 23:03:33,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.26 | bwd: 5712.31 | bwd_inner: 5699.32 | bwd_allreduce: 12.95 | step: 19.38

 15%|█▌        | 6248/41250 [15:05:58<84:19:54,  8.67s/it]
                                                          
{'loss': 0.0878, 'grad_norm': 3.8195557594299316, 'learning_rate': 3.847247856726708e-05, 'epoch': 1.51}

 15%|█▌        | 6248/41250 [15:05:58<84:19:54,  8.67s/it][2025-04-25 23:03:42,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-25 23:03:42,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.85 | bwd_microstep: 5879.57 | bwd_inner_microstep: 5654.84 | bwd_allreduce_microstep: 224.68 | step_microstep: 18.47
[2025-04-25 23:03:42,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.85 | bwd: 5879.58 | bwd_inner: 5654.84 | bwd_allreduce: 224.70 | step: 18.47

 15%|█▌        | 6249/41250 [15:06:07<84:39:53,  8.71s/it]
                                                          
{'loss': 0.2679, 'grad_norm': 1.8577169179916382, 'learning_rate': 3.84718766047723e-05, 'epoch': 1.51}

 15%|█▌        | 6249/41250 [15:06:07<84:39:53,  8.71s/it][2025-04-25 23:03:51,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 0.93
[2025-04-25 23:03:51,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.97 | bwd_microstep: 5746.03 | bwd_inner_microstep: 5707.50 | bwd_allreduce_microstep: 38.48 | step_microstep: 19.43
[2025-04-25 23:03:51,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.97 | bwd: 5746.04 | bwd_inner: 5707.50 | bwd_allreduce: 38.50 | step: 19.43

 15%|█▌        | 6250/41250 [15:06:16<84:34:52,  8.70s/it]
                                                          
{'loss': 0.1257, 'grad_norm': 1.8648438453674316, 'learning_rate': 3.847127452840204e-05, 'epoch': 1.52}

 15%|█▌        | 6250/41250 [15:06:16<84:34:52,  8.70s/it][2025-04-25 23:03:59,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:03:59,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.76 | bwd_microstep: 5758.48 | bwd_inner_microstep: 5649.54 | bwd_allreduce_microstep: 108.89 | step_microstep: 18.64
[2025-04-25 23:03:59,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.76 | bwd: 5758.49 | bwd_inner: 5649.54 | bwd_allreduce: 108.91 | step: 18.64

 15%|█▌        | 6251/41250 [15:06:25<84:30:49,  8.69s/it]
                                                          
{'loss': 0.1157, 'grad_norm': 1.9997502565383911, 'learning_rate': 3.8470672338159995e-05, 'epoch': 1.52}

 15%|█▌        | 6251/41250 [15:06:25<84:30:49,  8.69s/it][2025-04-25 23:04:08,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-25 23:04:08,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.36 | bwd_microstep: 5762.97 | bwd_inner_microstep: 5686.97 | bwd_allreduce_microstep: 75.96 | step_microstep: 19.28
[2025-04-25 23:04:08,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.36 | bwd: 5762.98 | bwd_inner: 5686.96 | bwd_allreduce: 75.98 | step: 19.28

 15%|█▌        | 6252/41250 [15:06:33<84:30:56,  8.69s/it]
                                                          
{'loss': 0.1452, 'grad_norm': 4.1308794021606445, 'learning_rate': 3.847007003404989e-05, 'epoch': 1.52}

 15%|█▌        | 6252/41250 [15:06:33<84:30:56,  8.69s/it][2025-04-25 23:04:17,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:04:17,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.41 | bwd_microstep: 5733.14 | bwd_inner_microstep: 5695.19 | bwd_allreduce_microstep: 37.91 | step_microstep: 18.81
[2025-04-25 23:04:17,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.41 | bwd: 5733.15 | bwd_inner: 5695.19 | bwd_allreduce: 37.92 | step: 18.81

 15%|█▌        | 6253/41250 [15:06:42<84:28:27,  8.69s/it]
                                                          
{'loss': 0.1595, 'grad_norm': 1.733755350112915, 'learning_rate': 3.8469467616075446e-05, 'epoch': 1.52}

 15%|█▌        | 6253/41250 [15:06:42<84:28:27,  8.69s/it][2025-04-25 23:04:26,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:04:26,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2947.34 | bwd_microstep: 5878.21 | bwd_inner_microstep: 5865.29 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.66
[2025-04-25 23:04:26,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2947.34 | bwd: 5878.22 | bwd_inner: 5865.29 | bwd_allreduce: 12.89 | step: 18.66

 15%|█▌        | 6254/41250 [15:06:51<85:07:05,  8.76s/it]
                                                          
{'loss': 0.0648, 'grad_norm': 1.254811406135559, 'learning_rate': 3.846886508424036e-05, 'epoch': 1.52}

 15%|█▌        | 6254/41250 [15:06:51<85:07:05,  8.76s/it][2025-04-25 23:04:34,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 23:04:34,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.42 | bwd_microstep: 5695.31 | bwd_inner_microstep: 5653.32 | bwd_allreduce_microstep: 41.95 | step_microstep: 18.75
[2025-04-25 23:04:34,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.42 | bwd: 5695.33 | bwd_inner: 5653.31 | bwd_allreduce: 41.97 | step: 18.75

 15%|█▌        | 6255/41250 [15:06:59<84:40:18,  8.71s/it]
                                                          
{'loss': 0.0175, 'grad_norm': 0.27086466550827026, 'learning_rate': 3.846826243854835e-05, 'epoch': 1.52}

 15%|█▌        | 6255/41250 [15:06:59<84:40:18,  8.71s/it][2025-04-25 23:04:43,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 23:04:43,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.93 | bwd_microstep: 5674.04 | bwd_inner_microstep: 5654.57 | bwd_allreduce_microstep: 19.43 | step_microstep: 18.45
[2025-04-25 23:04:43,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.93 | bwd: 5674.05 | bwd_inner: 5654.57 | bwd_allreduce: 19.44 | step: 18.46

 15%|█▌        | 6256/41250 [15:07:08<84:19:48,  8.68s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.4538718461990356, 'learning_rate': 3.846765967900314e-05, 'epoch': 1.52}

 15%|█▌        | 6256/41250 [15:07:08<84:19:48,  8.68s/it][2025-04-25 23:04:51,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-25 23:04:51,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.93 | bwd_microstep: 5670.94 | bwd_inner_microstep: 5640.80 | bwd_allreduce_microstep: 30.09 | step_microstep: 18.88
[2025-04-25 23:04:51,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.93 | bwd: 5670.95 | bwd_inner: 5640.80 | bwd_allreduce: 30.11 | step: 18.88

 15%|█▌        | 6257/41250 [15:07:17<84:04:40,  8.65s/it]
                                                          
{'loss': 0.1805, 'grad_norm': 1.9608901739120483, 'learning_rate': 3.846705680560844e-05, 'epoch': 1.52}

 15%|█▌        | 6257/41250 [15:07:17<84:04:40,  8.65s/it][2025-04-25 23:05:00,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:05:00,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.91 | bwd_microstep: 5694.42 | bwd_inner_microstep: 5640.26 | bwd_allreduce_microstep: 54.11 | step_microstep: 18.47
[2025-04-25 23:05:00,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.91 | bwd: 5694.43 | bwd_inner: 5640.26 | bwd_allreduce: 54.13 | step: 18.47

 15%|█▌        | 6258/41250 [15:07:25<83:57:03,  8.64s/it]
                                                          
{'loss': 0.0291, 'grad_norm': 0.5779341459274292, 'learning_rate': 3.8466453818367956e-05, 'epoch': 1.52}

 15%|█▌        | 6258/41250 [15:07:25<83:57:03,  8.64s/it][2025-04-25 23:05:09,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:05:09,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.75 | bwd_microstep: 5692.12 | bwd_inner_microstep: 5679.34 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.57
[2025-04-25 23:05:09,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.75 | bwd: 5692.13 | bwd_inner: 5679.34 | bwd_allreduce: 12.75 | step: 18.58

 15%|█▌        | 6259/41250 [15:07:34<83:54:33,  8.63s/it]
                                                          
{'loss': 0.3896, 'grad_norm': 5.2414045333862305, 'learning_rate': 3.8465850717285433e-05, 'epoch': 1.52}

 15%|█▌        | 6259/41250 [15:07:34<83:54:33,  8.63s/it][2025-04-25 23:05:17,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.31 | optimizer_step: 1.05
[2025-04-25 23:05:17,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.95 | bwd_microstep: 5756.39 | bwd_inner_microstep: 5644.76 | bwd_allreduce_microstep: 111.58 | step_microstep: 19.98
[2025-04-25 23:05:17,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.95 | bwd: 5756.41 | bwd_inner: 5644.76 | bwd_allreduce: 111.61 | step: 19.98

 15%|█▌        | 6260/41250 [15:07:43<84:01:37,  8.65s/it]
                                                          
{'loss': 0.0304, 'grad_norm': 0.6117140650749207, 'learning_rate': 3.846524750236456e-05, 'epoch': 1.52}

 15%|█▌        | 6260/41250 [15:07:43<84:01:37,  8.65s/it][2025-04-25 23:05:26,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 23:05:26,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.14 | bwd_microstep: 5710.10 | bwd_inner_microstep: 5697.37 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.13
[2025-04-25 23:05:26,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.14 | bwd: 5710.11 | bwd_inner: 5697.37 | bwd_allreduce: 12.70 | step: 19.13

 15%|█▌        | 6261/41250 [15:07:51<84:01:55,  8.65s/it]
                                                          
{'loss': 0.2084, 'grad_norm': 1.5681614875793457, 'learning_rate': 3.846464417360908e-05, 'epoch': 1.52}

 15%|█▌        | 6261/41250 [15:07:51<84:01:55,  8.65s/it][2025-04-25 23:05:35,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-25 23:05:35,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.94 | bwd_microstep: 5700.61 | bwd_inner_microstep: 5687.77 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.14
[2025-04-25 23:05:35,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.94 | bwd: 5700.62 | bwd_inner: 5687.77 | bwd_allreduce: 12.81 | step: 19.14

 15%|█▌        | 6262/41250 [15:08:00<84:00:01,  8.64s/it]
                                                          
{'loss': 0.0819, 'grad_norm': 1.613232970237732, 'learning_rate': 3.846404073102269e-05, 'epoch': 1.52}

 15%|█▌        | 6262/41250 [15:08:00<84:00:01,  8.64s/it][2025-04-25 23:05:43,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:05:43,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.24 | bwd_microstep: 5694.75 | bwd_inner_microstep: 5653.33 | bwd_allreduce_microstep: 41.37 | step_microstep: 18.88
[2025-04-25 23:05:43,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.24 | bwd: 5694.77 | bwd_inner: 5653.33 | bwd_allreduce: 41.39 | step: 18.88

 15%|█▌        | 6263/41250 [15:08:08<83:54:32,  8.63s/it]
                                                          
{'loss': 0.1228, 'grad_norm': 2.965669870376587, 'learning_rate': 3.846343717460912e-05, 'epoch': 1.52}

 15%|█▌        | 6263/41250 [15:08:08<83:54:32,  8.63s/it][2025-04-25 23:05:52,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:05:52,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5799.48 | bwd_inner_microstep: 5657.07 | bwd_allreduce_microstep: 142.36 | step_microstep: 18.67
[2025-04-25 23:05:52,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5799.49 | bwd_inner: 5657.07 | bwd_allreduce: 142.38 | step: 18.67

 15%|█▌        | 6264/41250 [15:08:17<84:07:55,  8.66s/it]
                                                          
{'loss': 0.2052, 'grad_norm': 5.254317760467529, 'learning_rate': 3.84628335043721e-05, 'epoch': 1.52}

 15%|█▌        | 6264/41250 [15:08:17<84:07:55,  8.66s/it][2025-04-25 23:06:01,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:06:01,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.94 | bwd_microstep: 5725.86 | bwd_inner_microstep: 5696.17 | bwd_allreduce_microstep: 29.64 | step_microstep: 18.71
[2025-04-25 23:06:01,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.94 | bwd: 5725.88 | bwd_inner: 5696.17 | bwd_allreduce: 29.66 | step: 18.71

 15%|█▌        | 6265/41250 [15:08:26<84:09:51,  8.66s/it]
                                                          
{'loss': 0.1322, 'grad_norm': 1.3543000221252441, 'learning_rate': 3.846222972031534e-05, 'epoch': 1.52}

 15%|█▌        | 6265/41250 [15:08:26<84:09:51,  8.66s/it][2025-04-25 23:06:09,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 23:06:09,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5718.12 | bwd_inner_microstep: 5705.21 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.83
[2025-04-25 23:06:09,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5718.13 | bwd_inner: 5705.21 | bwd_allreduce: 12.88 | step: 18.84

 15%|█▌        | 6266/41250 [15:08:34<84:09:53,  8.66s/it]
                                                          
{'loss': 0.0262, 'grad_norm': 0.6086506843566895, 'learning_rate': 3.8461625822442564e-05, 'epoch': 1.52}

 15%|█▌        | 6266/41250 [15:08:34<84:09:53,  8.66s/it][2025-04-25 23:06:18,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-25 23:06:18,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5871.24 | bwd_inner_microstep: 5720.74 | bwd_allreduce_microstep: 150.46 | step_microstep: 18.95
[2025-04-25 23:06:18,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5871.26 | bwd_inner: 5720.73 | bwd_allreduce: 150.48 | step: 18.96

 15%|█▌        | 6267/41250 [15:08:43<84:35:04,  8.70s/it]
                                                          
{'loss': 0.1659, 'grad_norm': 1.939268708229065, 'learning_rate': 3.8461021810757495e-05, 'epoch': 1.52}

 15%|█▌        | 6267/41250 [15:08:43<84:35:04,  8.70s/it][2025-04-25 23:06:27,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:06:27,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5709.04 | bwd_inner_microstep: 5695.99 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.70
[2025-04-25 23:06:27,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5709.06 | bwd_inner: 5695.99 | bwd_allreduce: 13.02 | step: 18.70

 15%|█▌        | 6268/41250 [15:08:52<84:24:25,  8.69s/it]
                                                          
{'loss': 0.1431, 'grad_norm': 3.334550619125366, 'learning_rate': 3.846041768526385e-05, 'epoch': 1.52}

 15%|█▌        | 6268/41250 [15:08:52<84:24:25,  8.69s/it][2025-04-25 23:06:35,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-25 23:06:35,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.54 | bwd_microstep: 5793.48 | bwd_inner_microstep: 5651.98 | bwd_allreduce_microstep: 141.46 | step_microstep: 18.49
[2025-04-25 23:06:35,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.54 | bwd: 5793.49 | bwd_inner: 5651.98 | bwd_allreduce: 141.48 | step: 18.49

 15%|█▌        | 6269/41250 [15:09:01<84:27:55,  8.69s/it]
                                                          
{'loss': 0.0542, 'grad_norm': 1.8547084331512451, 'learning_rate': 3.845981344596538e-05, 'epoch': 1.52}

 15%|█▌        | 6269/41250 [15:09:01<84:27:55,  8.69s/it][2025-04-25 23:06:44,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:06:44,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.24 | bwd_microstep: 5795.59 | bwd_inner_microstep: 5658.67 | bwd_allreduce_microstep: 136.88 | step_microstep: 18.67
[2025-04-25 23:06:44,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.24 | bwd: 5795.60 | bwd_inner: 5658.67 | bwd_allreduce: 136.90 | step: 18.67

 15%|█▌        | 6270/41250 [15:09:09<84:32:58,  8.70s/it]
                                                          
{'loss': 0.4741, 'grad_norm': 5.472161293029785, 'learning_rate': 3.8459209092865776e-05, 'epoch': 1.52}

 15%|█▌        | 6270/41250 [15:09:09<84:32:58,  8.70s/it][2025-04-25 23:06:53,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:06:53,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.88 | bwd_microstep: 5724.61 | bwd_inner_microstep: 5655.71 | bwd_allreduce_microstep: 68.86 | step_microstep: 18.45
[2025-04-25 23:06:53,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.88 | bwd: 5724.62 | bwd_inner: 5655.71 | bwd_allreduce: 68.88 | step: 18.45

 15%|█▌        | 6271/41250 [15:09:18<84:21:09,  8.68s/it]
                                                          
{'loss': 0.0644, 'grad_norm': 1.1365259885787964, 'learning_rate': 3.845860462596878e-05, 'epoch': 1.52}

 15%|█▌        | 6271/41250 [15:09:18<84:21:09,  8.68s/it][2025-04-25 23:07:01,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 23:07:01,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5787.39 | bwd_inner_microstep: 5669.75 | bwd_allreduce_microstep: 117.60 | step_microstep: 18.83
[2025-04-25 23:07:01,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.45 | bwd: 5787.40 | bwd_inner: 5669.75 | bwd_allreduce: 117.61 | step: 18.83

 15%|█▌        | 6272/41250 [15:09:27<84:24:40,  8.69s/it]
                                                          
{'loss': 0.0427, 'grad_norm': 0.9046356678009033, 'learning_rate': 3.845800004527812e-05, 'epoch': 1.52}

 15%|█▌        | 6272/41250 [15:09:27<84:24:40,  8.69s/it][2025-04-25 23:07:10,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 23:07:10,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.53 | bwd_microstep: 5788.84 | bwd_inner_microstep: 5710.67 | bwd_allreduce_microstep: 78.12 | step_microstep: 18.15
[2025-04-25 23:07:10,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.53 | bwd: 5788.85 | bwd_inner: 5710.67 | bwd_allreduce: 78.14 | step: 18.16

 15%|█▌        | 6273/41250 [15:09:35<84:31:29,  8.70s/it]
                                                          
{'loss': 0.3342, 'grad_norm': 6.2184624671936035, 'learning_rate': 3.8457395350797525e-05, 'epoch': 1.52}

 15%|█▌        | 6273/41250 [15:09:35<84:31:29,  8.70s/it][2025-04-25 23:07:19,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-25 23:07:19,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.61 | bwd_microstep: 5782.16 | bwd_inner_microstep: 5659.26 | bwd_allreduce_microstep: 122.85 | step_microstep: 19.73
[2025-04-25 23:07:19,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.61 | bwd: 5782.18 | bwd_inner: 5659.26 | bwd_allreduce: 122.87 | step: 19.74

 15%|█▌        | 6274/41250 [15:09:44<84:31:27,  8.70s/it]
                                                          
{'loss': 0.241, 'grad_norm': 2.485456705093384, 'learning_rate': 3.8456790542530715e-05, 'epoch': 1.52}

 15%|█▌        | 6274/41250 [15:09:44<84:31:27,  8.70s/it][2025-04-25 23:07:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:07:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.14 | bwd_microstep: 5711.59 | bwd_inner_microstep: 5667.85 | bwd_allreduce_microstep: 43.70 | step_microstep: 19.07
[2025-04-25 23:07:27,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.14 | bwd: 5711.61 | bwd_inner: 5667.85 | bwd_allreduce: 43.72 | step: 19.07

 15%|█▌        | 6275/41250 [15:09:53<84:20:21,  8.68s/it]
                                                          
{'loss': 0.4911, 'grad_norm': 3.73717999458313, 'learning_rate': 3.8456185620481414e-05, 'epoch': 1.52}

 15%|█▌        | 6275/41250 [15:09:53<84:20:21,  8.68s/it][2025-04-25 23:07:36,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-25 23:07:36,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.80 | bwd_microstep: 5712.23 | bwd_inner_microstep: 5698.88 | bwd_allreduce_microstep: 13.29 | step_microstep: 19.29
[2025-04-25 23:07:36,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.80 | bwd: 5712.24 | bwd_inner: 5698.88 | bwd_allreduce: 13.32 | step: 19.29

 15%|█▌        | 6276/41250 [15:10:01<84:14:53,  8.67s/it]
                                                          
{'loss': 0.0292, 'grad_norm': 0.5185178518295288, 'learning_rate': 3.845558058465336e-05, 'epoch': 1.52}

 15%|█▌        | 6276/41250 [15:10:01<84:14:53,  8.67s/it][2025-04-25 23:07:45,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 1.13
[2025-04-25 23:07:45,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.59 | bwd_microstep: 5720.73 | bwd_inner_microstep: 5699.66 | bwd_allreduce_microstep: 21.02 | step_microstep: 19.31
[2025-04-25 23:07:45,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.59 | bwd: 5720.75 | bwd_inner: 5699.66 | bwd_allreduce: 21.04 | step: 19.31

 15%|█▌        | 6277/41250 [15:10:10<84:12:08,  8.67s/it]
                                                          
{'loss': 0.2878, 'grad_norm': 2.238090753555298, 'learning_rate': 3.84549754350503e-05, 'epoch': 1.52}

 15%|█▌        | 6277/41250 [15:10:10<84:12:08,  8.67s/it][2025-04-25 23:07:53,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 23:07:53,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.02 | bwd_microstep: 5724.44 | bwd_inner_microstep: 5711.66 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.81
[2025-04-25 23:07:53,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.02 | bwd: 5724.45 | bwd_inner: 5711.66 | bwd_allreduce: 12.75 | step: 18.81

 15%|█▌        | 6278/41250 [15:10:19<84:11:47,  8.67s/it]
                                                          
{'loss': 0.1883, 'grad_norm': 1.5364543199539185, 'learning_rate': 3.8454370171675926e-05, 'epoch': 1.52}

 15%|█▌        | 6278/41250 [15:10:19<84:11:47,  8.67s/it][2025-04-25 23:08:02,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-25 23:08:02,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.35 | bwd_microstep: 5782.80 | bwd_inner_microstep: 5661.29 | bwd_allreduce_microstep: 121.46 | step_microstep: 18.60
[2025-04-25 23:08:02,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.35 | bwd: 5782.81 | bwd_inner: 5661.29 | bwd_allreduce: 121.48 | step: 18.60

 15%|█▌        | 6279/41250 [15:10:27<84:19:42,  8.68s/it]
                                                          
{'loss': 0.1821, 'grad_norm': 1.4536638259887695, 'learning_rate': 3.8453764794534e-05, 'epoch': 1.52}

 15%|█▌        | 6279/41250 [15:10:27<84:19:42,  8.68s/it][2025-04-25 23:08:11,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:08:11,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.40 | bwd_microstep: 5783.44 | bwd_inner_microstep: 5706.83 | bwd_allreduce_microstep: 76.56 | step_microstep: 18.77
[2025-04-25 23:08:11,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.40 | bwd: 5783.45 | bwd_inner: 5706.83 | bwd_allreduce: 76.58 | step: 18.78

 15%|█▌        | 6280/41250 [15:10:36<84:27:44,  8.70s/it]
                                                          
{'loss': 0.1019, 'grad_norm': 1.0843712091445923, 'learning_rate': 3.845315930362824e-05, 'epoch': 1.52}

 15%|█▌        | 6280/41250 [15:10:36<84:27:44,  8.70s/it][2025-04-25 23:08:20,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:08:20,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.89 | bwd_microstep: 5762.90 | bwd_inner_microstep: 5706.17 | bwd_allreduce_microstep: 56.67 | step_microstep: 18.72
[2025-04-25 23:08:20,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.89 | bwd: 5762.92 | bwd_inner: 5706.17 | bwd_allreduce: 56.69 | step: 18.72

 15%|█▌        | 6281/41250 [15:10:45<84:31:26,  8.70s/it]
                                                          
{'loss': 0.2218, 'grad_norm': 2.4362258911132812, 'learning_rate': 3.8452553698962376e-05, 'epoch': 1.52}

 15%|█▌        | 6281/41250 [15:10:45<84:31:26,  8.70s/it][2025-04-25 23:08:28,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.24 | optimizer_step: 1.01
[2025-04-25 23:08:28,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.69 | bwd_microstep: 5757.33 | bwd_inner_microstep: 5696.67 | bwd_allreduce_microstep: 60.59 | step_microstep: 19.60
[2025-04-25 23:08:28,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.69 | bwd: 5757.34 | bwd_inner: 5696.67 | bwd_allreduce: 60.62 | step: 19.60

 15%|█▌        | 6282/41250 [15:10:54<84:29:59,  8.70s/it]
                                                          
{'loss': 0.1528, 'grad_norm': 3.07206654548645, 'learning_rate': 3.845194798054015e-05, 'epoch': 1.52}

 15%|█▌        | 6282/41250 [15:10:54<84:29:59,  8.70s/it][2025-04-25 23:08:37,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-25 23:08:37,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2945.54 | bwd_microstep: 5893.13 | bwd_inner_microstep: 5880.39 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.56
[2025-04-25 23:08:37,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2945.54 | bwd: 5893.14 | bwd_inner: 5880.39 | bwd_allreduce: 12.71 | step: 18.56

 15%|█▌        | 6283/41250 [15:11:03<85:09:44,  8.77s/it]
                                                          
{'loss': 0.1108, 'grad_norm': 1.2784457206726074, 'learning_rate': 3.8451342148365306e-05, 'epoch': 1.52}

 15%|█▌        | 6283/41250 [15:11:03<85:09:44,  8.77s/it][2025-04-25 23:08:46,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:08:46,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2901.26 | bwd_microstep: 5794.48 | bwd_inner_microstep: 5781.61 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.74
[2025-04-25 23:08:46,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2901.26 | bwd: 5794.49 | bwd_inner: 5781.61 | bwd_allreduce: 12.85 | step: 18.74

 15%|█▌        | 6284/41250 [15:11:11<85:11:28,  8.77s/it]
                                                          
{'loss': 0.1261, 'grad_norm': 2.3021726608276367, 'learning_rate': 3.845073620244156e-05, 'epoch': 1.52}

 15%|█▌        | 6284/41250 [15:11:11<85:11:28,  8.77s/it][2025-04-25 23:08:55,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:08:55,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.27 | bwd_microstep: 5799.45 | bwd_inner_microstep: 5786.69 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.72
[2025-04-25 23:08:55,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.27 | bwd: 5799.47 | bwd_inner: 5786.69 | bwd_allreduce: 12.74 | step: 18.72

 15%|█▌        | 6285/41250 [15:11:20<85:11:42,  8.77s/it]
                                                          
{'loss': 0.1171, 'grad_norm': 0.7819494009017944, 'learning_rate': 3.845013014277265e-05, 'epoch': 1.52}

 15%|█▌        | 6285/41250 [15:11:20<85:11:42,  8.77s/it][2025-04-25 23:09:03,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:09:03,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.35 | bwd_microstep: 5760.92 | bwd_inner_microstep: 5703.89 | bwd_allreduce_microstep: 56.99 | step_microstep: 18.84
[2025-04-25 23:09:03,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.35 | bwd: 5760.93 | bwd_inner: 5703.89 | bwd_allreduce: 57.00 | step: 18.84

 15%|█▌        | 6286/41250 [15:11:29<84:59:37,  8.75s/it]
                                                          
{'loss': 0.3017, 'grad_norm': 1.8850603103637695, 'learning_rate': 3.844952396936232e-05, 'epoch': 1.52}

 15%|█▌        | 6286/41250 [15:11:29<84:59:37,  8.75s/it][2025-04-25 23:09:12,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-25 23:09:12,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5783.36 | bwd_inner_microstep: 5650.46 | bwd_allreduce_microstep: 132.84 | step_microstep: 18.45
[2025-04-25 23:09:12,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5783.38 | bwd_inner: 5650.46 | bwd_allreduce: 132.87 | step: 18.46

 15%|█▌        | 6287/41250 [15:11:37<84:51:48,  8.74s/it]
                                                          
{'loss': 0.1462, 'grad_norm': 2.1206183433532715, 'learning_rate': 3.8448917682214305e-05, 'epoch': 1.52}

 15%|█▌        | 6287/41250 [15:11:37<84:51:48,  8.74s/it][2025-04-25 23:09:21,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.09 | optimizer_step: 1.00
[2025-04-25 23:09:21,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.19 | bwd_microstep: 5710.32 | bwd_inner_microstep: 5646.34 | bwd_allreduce_microstep: 63.92 | step_microstep: 19.74
[2025-04-25 23:09:21,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.19 | bwd: 5710.34 | bwd_inner: 5646.34 | bwd_allreduce: 63.95 | step: 19.75

 15%|█▌        | 6288/41250 [15:11:46<84:32:30,  8.71s/it]
                                                          
{'loss': 0.0446, 'grad_norm': 1.1971698999404907, 'learning_rate': 3.844831128133234e-05, 'epoch': 1.52}

 15%|█▌        | 6288/41250 [15:11:46<84:32:30,  8.71s/it][2025-04-25 23:09:30,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:09:30,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.44 | bwd_microstep: 5841.59 | bwd_inner_microstep: 5689.36 | bwd_allreduce_microstep: 152.18 | step_microstep: 18.82
[2025-04-25 23:09:30,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.44 | bwd: 5841.60 | bwd_inner: 5689.36 | bwd_allreduce: 152.20 | step: 18.82

 15%|█▌        | 6289/41250 [15:11:55<84:46:13,  8.73s/it]
                                                          
{'loss': 0.132, 'grad_norm': 1.562506079673767, 'learning_rate': 3.844770476672016e-05, 'epoch': 1.52}

 15%|█▌        | 6289/41250 [15:11:55<84:46:13,  8.73s/it][2025-04-25 23:09:38,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 23:09:38,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5761.40 | bwd_inner_microstep: 5704.78 | bwd_allreduce_microstep: 56.58 | step_microstep: 18.75
[2025-04-25 23:09:38,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5761.42 | bwd_inner: 5704.78 | bwd_allreduce: 56.60 | step: 18.76

 15%|█▌        | 6290/41250 [15:12:04<84:40:15,  8.72s/it]
                                                          
{'loss': 0.0371, 'grad_norm': 0.5937759280204773, 'learning_rate': 3.8447098138381515e-05, 'epoch': 1.52}

 15%|█▌        | 6290/41250 [15:12:04<84:40:15,  8.72s/it][2025-04-25 23:09:47,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:09:47,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.34 | bwd_microstep: 5739.51 | bwd_inner_microstep: 5649.70 | bwd_allreduce_microstep: 89.77 | step_microstep: 18.39
[2025-04-25 23:09:47,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.35 | bwd: 5739.52 | bwd_inner: 5649.70 | bwd_allreduce: 89.78 | step: 18.39

 15%|█▌        | 6291/41250 [15:12:12<84:31:11,  8.70s/it]
                                                          
{'loss': 0.1634, 'grad_norm': 3.1056480407714844, 'learning_rate': 3.844649139632014e-05, 'epoch': 1.53}

 15%|█▌        | 6291/41250 [15:12:12<84:31:11,  8.70s/it][2025-04-25 23:09:56,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-25 23:09:56,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.24 | bwd_microstep: 5696.33 | bwd_inner_microstep: 5683.52 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.58
[2025-04-25 23:09:56,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.24 | bwd: 5696.34 | bwd_inner: 5683.52 | bwd_allreduce: 12.78 | step: 18.58

 15%|█▌        | 6292/41250 [15:12:21<84:18:49,  8.68s/it]
                                                          
{'loss': 0.2237, 'grad_norm': 2.809368848800659, 'learning_rate': 3.844588454053977e-05, 'epoch': 1.53}

 15%|█▌        | 6292/41250 [15:12:21<84:18:49,  8.68s/it][2025-04-25 23:10:04,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:10:04,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.07 | bwd_microstep: 5747.83 | bwd_inner_microstep: 5684.43 | bwd_allreduce_microstep: 63.36 | step_microstep: 18.62
[2025-04-25 23:10:04,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.08 | bwd: 5747.84 | bwd_inner: 5684.43 | bwd_allreduce: 63.38 | step: 18.63

 15%|█▌        | 6293/41250 [15:12:30<84:19:53,  8.68s/it]
                                                          
{'loss': 0.0795, 'grad_norm': 3.086463689804077, 'learning_rate': 3.844527757104415e-05, 'epoch': 1.53}

 15%|█▌        | 6293/41250 [15:12:30<84:19:53,  8.68s/it][2025-04-25 23:10:13,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 23:10:13,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.33 | bwd_microstep: 5772.80 | bwd_inner_microstep: 5759.99 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.71
[2025-04-25 23:10:13,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.33 | bwd: 5772.82 | bwd_inner: 5759.99 | bwd_allreduce: 12.79 | step: 18.72

 15%|█▌        | 6294/41250 [15:12:38<84:31:58,  8.71s/it]
                                                          
{'loss': 0.1138, 'grad_norm': 1.8313766717910767, 'learning_rate': 3.844467048783702e-05, 'epoch': 1.53}

 15%|█▌        | 6294/41250 [15:12:38<84:31:58,  8.71s/it][2025-04-25 23:10:22,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-25 23:10:22,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.27 | bwd_microstep: 5855.65 | bwd_inner_microstep: 5686.66 | bwd_allreduce_microstep: 168.95 | step_microstep: 18.91
[2025-04-25 23:10:22,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.27 | bwd: 5855.66 | bwd_inner: 5686.66 | bwd_allreduce: 168.96 | step: 18.92

 15%|█▌        | 6295/41250 [15:12:47<84:46:45,  8.73s/it]
                                                          
{'loss': 0.5085, 'grad_norm': 3.3431484699249268, 'learning_rate': 3.8444063290922125e-05, 'epoch': 1.53}

 15%|█▌        | 6295/41250 [15:12:47<84:46:45,  8.73s/it][2025-04-25 23:10:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 23:10:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.48 | bwd_microstep: 5787.64 | bwd_inner_microstep: 5643.12 | bwd_allreduce_microstep: 144.47 | step_microstep: 19.18
[2025-04-25 23:10:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.48 | bwd: 5787.66 | bwd_inner: 5643.12 | bwd_allreduce: 144.49 | step: 19.19

 15%|█▌        | 6296/41250 [15:12:56<84:40:46,  8.72s/it]
                                                          
{'loss': 0.1139, 'grad_norm': 1.3788905143737793, 'learning_rate': 3.844345598030321e-05, 'epoch': 1.53}

 15%|█▌        | 6296/41250 [15:12:56<84:40:46,  8.72s/it][2025-04-25 23:10:39,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 23:10:39,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5774.52 | bwd_inner_microstep: 5649.96 | bwd_allreduce_microstep: 124.51 | step_microstep: 19.02
[2025-04-25 23:10:39,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5774.53 | bwd_inner: 5649.96 | bwd_allreduce: 124.53 | step: 19.02

 15%|█▌        | 6297/41250 [15:13:05<84:35:14,  8.71s/it]
                                                          
{'loss': 0.3057, 'grad_norm': 3.749654531478882, 'learning_rate': 3.844284855598401e-05, 'epoch': 1.53}

 15%|█▌        | 6297/41250 [15:13:05<84:35:14,  8.71s/it][2025-04-25 23:10:48,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:10:48,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.72 | bwd_microstep: 5706.40 | bwd_inner_microstep: 5693.58 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.70
[2025-04-25 23:10:48,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.72 | bwd: 5706.41 | bwd_inner: 5693.58 | bwd_allreduce: 12.79 | step: 18.71

 15%|█▌        | 6298/41250 [15:13:13<84:23:58,  8.69s/it]
                                                          
{'loss': 0.092, 'grad_norm': 2.037689447402954, 'learning_rate': 3.844224101796829e-05, 'epoch': 1.53}

 15%|█▌        | 6298/41250 [15:13:13<84:23:58,  8.69s/it][2025-04-25 23:10:56,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 23:10:56,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.98 | bwd_microstep: 5710.97 | bwd_inner_microstep: 5698.21 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.63
[2025-04-25 23:10:56,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.98 | bwd: 5710.99 | bwd_inner: 5698.21 | bwd_allreduce: 12.74 | step: 18.63

 15%|█▌        | 6299/41250 [15:13:22<84:15:51,  8.68s/it]
                                                          
{'loss': 0.1082, 'grad_norm': 1.2878994941711426, 'learning_rate': 3.844163336625977e-05, 'epoch': 1.53}

 15%|█▌        | 6299/41250 [15:13:22<84:15:51,  8.68s/it][2025-04-25 23:11:05,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.10
[2025-04-25 23:11:05,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.08 | bwd_microstep: 5705.51 | bwd_inner_microstep: 5641.67 | bwd_allreduce_microstep: 63.80 | step_microstep: 19.12
[2025-04-25 23:11:05,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.08 | bwd: 5705.52 | bwd_inner: 5641.67 | bwd_allreduce: 63.81 | step: 19.12

 15%|█▌        | 6300/41250 [15:13:30<84:04:25,  8.66s/it]
                                                          
{'loss': 0.0969, 'grad_norm': 0.801978349685669, 'learning_rate': 3.844102560086221e-05, 'epoch': 1.53}

 15%|█▌        | 6300/41250 [15:13:30<84:04:25,  8.66s/it][2025-04-25 23:11:14,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-25 23:11:14,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.83 | bwd_microstep: 5687.78 | bwd_inner_microstep: 5647.16 | bwd_allreduce_microstep: 40.57 | step_microstep: 18.66
[2025-04-25 23:11:14,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.83 | bwd: 5687.79 | bwd_inner: 5647.16 | bwd_allreduce: 40.59 | step: 18.67

 15%|█▌        | 6301/41250 [15:13:39<83:54:22,  8.64s/it]
                                                          
{'loss': 0.1081, 'grad_norm': 2.524066686630249, 'learning_rate': 3.844041772177935e-05, 'epoch': 1.53}

 15%|█▌        | 6301/41250 [15:13:39<83:54:22,  8.64s/it][2025-04-25 23:11:22,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:11:22,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.86 | bwd_microstep: 5781.06 | bwd_inner_microstep: 5642.16 | bwd_allreduce_microstep: 138.87 | step_microstep: 18.27
[2025-04-25 23:11:22,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.86 | bwd: 5781.08 | bwd_inner: 5642.15 | bwd_allreduce: 138.88 | step: 18.27

 15%|█▌        | 6302/41250 [15:13:48<84:01:15,  8.66s/it]
                                                          
{'loss': 0.2251, 'grad_norm': 1.921859622001648, 'learning_rate': 3.843980972901495e-05, 'epoch': 1.53}

 15%|█▌        | 6302/41250 [15:13:48<84:01:15,  8.66s/it][2025-04-25 23:11:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-25 23:11:31,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.72 | bwd_microstep: 5699.59 | bwd_inner_microstep: 5686.94 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.32
[2025-04-25 23:11:31,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.72 | bwd: 5699.60 | bwd_inner: 5686.94 | bwd_allreduce: 12.62 | step: 18.33

 15%|█▌        | 6303/41250 [15:13:56<83:56:26,  8.65s/it]
                                                          
{'loss': 0.1492, 'grad_norm': 2.8894906044006348, 'learning_rate': 3.8439201622572746e-05, 'epoch': 1.53}

 15%|█▌        | 6303/41250 [15:13:56<83:56:26,  8.65s/it][2025-04-25 23:11:40,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-25 23:11:40,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.57 | bwd_microstep: 5699.80 | bwd_inner_microstep: 5687.19 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.65
[2025-04-25 23:11:40,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.57 | bwd: 5699.81 | bwd_inner: 5687.19 | bwd_allreduce: 12.59 | step: 18.65

 15%|█▌        | 6304/41250 [15:14:05<83:52:43,  8.64s/it]
                                                          
{'loss': 0.1553, 'grad_norm': 2.8738698959350586, 'learning_rate': 3.843859340245649e-05, 'epoch': 1.53}

 15%|█▌        | 6304/41250 [15:14:05<83:52:43,  8.64s/it][2025-04-25 23:11:48,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 23:11:48,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.31 | bwd_microstep: 5757.54 | bwd_inner_microstep: 5662.59 | bwd_allreduce_microstep: 94.91 | step_microstep: 18.84
[2025-04-25 23:11:48,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.31 | bwd: 5757.55 | bwd_inner: 5662.59 | bwd_allreduce: 94.92 | step: 18.85

 15%|█▌        | 6305/41250 [15:14:14<83:59:20,  8.65s/it]
                                                          
{'loss': 0.1713, 'grad_norm': 2.73095703125, 'learning_rate': 3.8437985068669936e-05, 'epoch': 1.53}

 15%|█▌        | 6305/41250 [15:14:14<83:59:20,  8.65s/it][2025-04-25 23:11:57,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:11:57,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.71 | bwd_microstep: 5707.30 | bwd_inner_microstep: 5694.74 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.46
[2025-04-25 23:11:57,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.71 | bwd: 5707.31 | bwd_inner: 5694.74 | bwd_allreduce: 12.53 | step: 18.46

 15%|█▌        | 6306/41250 [15:14:22<83:56:43,  8.65s/it]
                                                          
{'loss': 0.1615, 'grad_norm': 2.635965585708618, 'learning_rate': 3.843737662121682e-05, 'epoch': 1.53}

 15%|█▌        | 6306/41250 [15:14:22<83:56:43,  8.65s/it][2025-04-25 23:12:06,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 23:12:06,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.18 | bwd_microstep: 5698.28 | bwd_inner_microstep: 5685.60 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.92
[2025-04-25 23:12:06,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.18 | bwd: 5698.29 | bwd_inner: 5685.60 | bwd_allreduce: 12.66 | step: 17.92

 15%|█▌        | 6307/41250 [15:14:31<83:52:13,  8.64s/it]
                                                          
{'loss': 0.1288, 'grad_norm': 1.0378483533859253, 'learning_rate': 3.8436768060100915e-05, 'epoch': 1.53}

 15%|█▌        | 6307/41250 [15:14:31<83:52:13,  8.64s/it][2025-04-25 23:12:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:12:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.87 | bwd_microstep: 5708.06 | bwd_inner_microstep: 5695.59 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.58
[2025-04-25 23:12:14,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.87 | bwd: 5708.07 | bwd_inner: 5695.59 | bwd_allreduce: 12.43 | step: 18.58

 15%|█▌        | 6308/41250 [15:14:40<83:52:23,  8.64s/it]
                                                          
{'loss': 0.1725, 'grad_norm': 2.521916627883911, 'learning_rate': 3.8436159385325965e-05, 'epoch': 1.53}

 15%|█▌        | 6308/41250 [15:14:40<83:52:23,  8.64s/it][2025-04-25 23:12:23,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.93
[2025-04-25 23:12:23,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.42 | bwd_microstep: 5711.23 | bwd_inner_microstep: 5698.58 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.79
[2025-04-25 23:12:23,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.42 | bwd: 5711.24 | bwd_inner: 5698.58 | bwd_allreduce: 12.62 | step: 18.79

 15%|█▌        | 6309/41250 [15:14:48<83:54:17,  8.64s/it]
                                                          
{'loss': 0.0832, 'grad_norm': 2.386345386505127, 'learning_rate': 3.843555059689571e-05, 'epoch': 1.53}

 15%|█▌        | 6309/41250 [15:14:48<83:54:17,  8.64s/it][2025-04-25 23:12:32,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 23:12:32,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.13 | bwd_microstep: 5773.88 | bwd_inner_microstep: 5662.25 | bwd_allreduce_microstep: 111.59 | step_microstep: 17.95
[2025-04-25 23:12:32,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.13 | bwd: 5773.90 | bwd_inner: 5662.25 | bwd_allreduce: 111.61 | step: 17.95

 15%|█▌        | 6310/41250 [15:14:57<84:01:25,  8.66s/it]
                                                          
{'loss': 0.3166, 'grad_norm': 2.2623403072357178, 'learning_rate': 3.843494169481391e-05, 'epoch': 1.53}

 15%|█▌        | 6310/41250 [15:14:57<84:01:25,  8.66s/it][2025-04-25 23:12:40,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 23:12:40,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.99 | bwd_microstep: 5700.01 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 40.99 | step_microstep: 19.61
[2025-04-25 23:12:40,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.99 | bwd: 5700.03 | bwd_inner: 5658.97 | bwd_allreduce: 41.02 | step: 19.61

 15%|█▌        | 6311/41250 [15:15:06<83:55:09,  8.65s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 0.6734538674354553, 'learning_rate': 3.8434332679084325e-05, 'epoch': 1.53}

 15%|█▌        | 6311/41250 [15:15:06<83:55:09,  8.65s/it][2025-04-25 23:12:49,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:12:49,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.29 | bwd_microstep: 5712.02 | bwd_inner_microstep: 5656.32 | bwd_allreduce_microstep: 55.66 | step_microstep: 18.92
[2025-04-25 23:12:49,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.29 | bwd: 5712.03 | bwd_inner: 5656.31 | bwd_allreduce: 55.68 | step: 18.92

 15%|█▌        | 6312/41250 [15:15:14<83:50:50,  8.64s/it]
                                                          
{'loss': 0.103, 'grad_norm': 1.6983932256698608, 'learning_rate': 3.84337235497107e-05, 'epoch': 1.53}

 15%|█▌        | 6312/41250 [15:15:14<83:50:50,  8.64s/it][2025-04-25 23:12:58,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:12:58,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5753.14 | bwd_inner_microstep: 5715.48 | bwd_allreduce_microstep: 37.61 | step_microstep: 18.99
[2025-04-25 23:12:58,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5753.15 | bwd_inner: 5715.48 | bwd_allreduce: 37.63 | step: 18.99

 15%|█▌        | 6313/41250 [15:15:23<83:59:19,  8.65s/it]
                                                          
{'loss': 0.2319, 'grad_norm': 1.082553505897522, 'learning_rate': 3.84331143066968e-05, 'epoch': 1.53}

 15%|█▌        | 6313/41250 [15:15:23<83:59:19,  8.65s/it][2025-04-25 23:13:06,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.12 | optimizer_step: 1.03
[2025-04-25 23:13:06,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.16 | bwd_microstep: 5788.88 | bwd_inner_microstep: 5655.40 | bwd_allreduce_microstep: 133.42 | step_microstep: 19.63
[2025-04-25 23:13:06,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.16 | bwd: 5788.89 | bwd_inner: 5655.40 | bwd_allreduce: 133.45 | step: 19.63

 15%|█▌        | 6314/41250 [15:15:32<84:07:55,  8.67s/it]
                                                          
{'loss': 0.0882, 'grad_norm': 2.3435633182525635, 'learning_rate': 3.8432504950046376e-05, 'epoch': 1.53}

 15%|█▌        | 6314/41250 [15:15:32<84:07:55,  8.67s/it][2025-04-25 23:13:15,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 23:13:15,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.84 | bwd_microstep: 5796.04 | bwd_inner_microstep: 5666.66 | bwd_allreduce_microstep: 129.34 | step_microstep: 18.68
[2025-04-25 23:13:15,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.84 | bwd: 5796.06 | bwd_inner: 5666.66 | bwd_allreduce: 129.35 | step: 18.69

 15%|█▌        | 6315/41250 [15:15:40<84:14:45,  8.68s/it]
                                                          
{'loss': 0.1337, 'grad_norm': 3.473900556564331, 'learning_rate': 3.843189547976318e-05, 'epoch': 1.53}

 15%|█▌        | 6315/41250 [15:15:40<84:14:45,  8.68s/it][2025-04-25 23:13:24,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:13:24,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.99 | bwd_microstep: 5727.56 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 74.69 | step_microstep: 18.08
[2025-04-25 23:13:24,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.99 | bwd: 5727.57 | bwd_inner: 5652.83 | bwd_allreduce: 74.70 | step: 18.09

 15%|█▌        | 6316/41250 [15:15:49<84:09:47,  8.67s/it]
                                                          
{'loss': 0.0465, 'grad_norm': 0.4608171284198761, 'learning_rate': 3.843128589585098e-05, 'epoch': 1.53}

 15%|█▌        | 6316/41250 [15:15:49<84:09:47,  8.67s/it][2025-04-25 23:13:32,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:13:32,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2906.47 | bwd_microstep: 5799.09 | bwd_inner_microstep: 5786.40 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.25
[2025-04-25 23:13:32,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2906.47 | bwd: 5799.10 | bwd_inner: 5786.40 | bwd_allreduce: 12.66 | step: 18.26

 15%|█▌        | 6317/41250 [15:15:58<84:29:30,  8.71s/it]
                                                          
{'loss': 0.0361, 'grad_norm': 0.30753186345100403, 'learning_rate': 3.8430676198313526e-05, 'epoch': 1.53}

 15%|█▌        | 6317/41250 [15:15:58<84:29:30,  8.71s/it][2025-04-25 23:13:41,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:13:41,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.25 | bwd_microstep: 5721.93 | bwd_inner_microstep: 5661.08 | bwd_allreduce_microstep: 60.81 | step_microstep: 18.25
[2025-04-25 23:13:41,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.25 | bwd: 5721.94 | bwd_inner: 5661.08 | bwd_allreduce: 60.82 | step: 18.25

 15%|█▌        | 6318/41250 [15:16:06<84:18:13,  8.69s/it]
                                                          
{'loss': 0.2706, 'grad_norm': 2.6805758476257324, 'learning_rate': 3.843006638715457e-05, 'epoch': 1.53}

 15%|█▌        | 6318/41250 [15:16:06<84:18:13,  8.69s/it][2025-04-25 23:13:50,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-25 23:13:50,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.95 | bwd_microstep: 5800.01 | bwd_inner_microstep: 5662.47 | bwd_allreduce_microstep: 137.49 | step_microstep: 18.43
[2025-04-25 23:13:50,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.95 | bwd: 5800.02 | bwd_inner: 5662.47 | bwd_allreduce: 137.51 | step: 18.43

 15%|█▌        | 6319/41250 [15:16:15<84:23:36,  8.70s/it]
                                                          
{'loss': 0.2575, 'grad_norm': 2.4632229804992676, 'learning_rate': 3.842945646237789e-05, 'epoch': 1.53}

 15%|█▌        | 6319/41250 [15:16:15<84:23:36,  8.70s/it][2025-04-25 23:13:58,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:13:58,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5749.40 | bwd_inner_microstep: 5712.18 | bwd_allreduce_microstep: 37.17 | step_microstep: 18.77
[2025-04-25 23:13:58,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5749.41 | bwd_inner: 5712.18 | bwd_allreduce: 37.19 | step: 18.78

 15%|█▌        | 6320/41250 [15:16:24<84:21:32,  8.69s/it]
                                                          
{'loss': 0.1364, 'grad_norm': 1.5986143350601196, 'learning_rate': 3.8428846423987234e-05, 'epoch': 1.53}

 15%|█▌        | 6320/41250 [15:16:24<84:21:32,  8.69s/it][2025-04-25 23:14:07,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:14:07,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.07 | bwd_microstep: 5705.43 | bwd_inner_microstep: 5692.72 | bwd_allreduce_microstep: 12.67 | step_microstep: 17.99
[2025-04-25 23:14:07,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.08 | bwd: 5705.44 | bwd_inner: 5692.72 | bwd_allreduce: 12.68 | step: 17.99

 15%|█▌        | 6321/41250 [15:16:32<84:11:37,  8.68s/it]
                                                          
{'loss': 0.1623, 'grad_norm': 1.2871153354644775, 'learning_rate': 3.842823627198636e-05, 'epoch': 1.53}

 15%|█▌        | 6321/41250 [15:16:32<84:11:37,  8.68s/it][2025-04-25 23:14:16,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:14:16,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.01 | bwd_microstep: 5783.14 | bwd_inner_microstep: 5652.10 | bwd_allreduce_microstep: 130.98 | step_microstep: 18.67
[2025-04-25 23:14:16,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.01 | bwd: 5783.15 | bwd_inner: 5652.10 | bwd_allreduce: 131.00 | step: 18.67

 15%|█▌        | 6322/41250 [15:16:41<84:13:28,  8.68s/it]
                                                          
{'loss': 0.0614, 'grad_norm': 0.9615607261657715, 'learning_rate': 3.8427626006379035e-05, 'epoch': 1.53}

 15%|█▌        | 6322/41250 [15:16:41<84:13:28,  8.68s/it][2025-04-25 23:14:25,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:14:25,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.58 | bwd_microstep: 5872.52 | bwd_inner_microstep: 5859.71 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.34
[2025-04-25 23:14:25,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.59 | bwd: 5872.53 | bwd_inner: 5859.71 | bwd_allreduce: 12.77 | step: 18.35

 15%|█▌        | 6323/41250 [15:16:50<84:49:54,  8.74s/it]
                                                          
{'loss': 0.201, 'grad_norm': 1.7695400714874268, 'learning_rate': 3.842701562716903e-05, 'epoch': 1.53}

 15%|█▌        | 6323/41250 [15:16:50<84:49:54,  8.74s/it][2025-04-25 23:14:33,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:14:33,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.83 | bwd_microstep: 5773.81 | bwd_inner_microstep: 5691.69 | bwd_allreduce_microstep: 82.07 | step_microstep: 18.44
[2025-04-25 23:14:33,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.83 | bwd: 5773.82 | bwd_inner: 5691.69 | bwd_allreduce: 82.09 | step: 18.44

 15%|█▌        | 6324/41250 [15:16:59<84:42:14,  8.73s/it]
                                                          
{'loss': 0.0291, 'grad_norm': 0.25101396441459656, 'learning_rate': 3.842640513436008e-05, 'epoch': 1.53}

 15%|█▌        | 6324/41250 [15:16:59<84:42:14,  8.73s/it][2025-04-25 23:14:42,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 1.12
[2025-04-25 23:14:42,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.52 | bwd_microstep: 5759.21 | bwd_inner_microstep: 5703.85 | bwd_allreduce_microstep: 55.31 | step_microstep: 18.56
[2025-04-25 23:14:42,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.52 | bwd: 5759.23 | bwd_inner: 5703.86 | bwd_allreduce: 55.33 | step: 18.57

 15%|█▌        | 6325/41250 [15:17:07<84:36:47,  8.72s/it]
                                                          
{'loss': 0.2915, 'grad_norm': 5.663130760192871, 'learning_rate': 3.842579452795598e-05, 'epoch': 1.53}

 15%|█▌        | 6325/41250 [15:17:07<84:36:47,  8.72s/it][2025-04-25 23:14:51,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.13 | optimizer_step: 0.98
[2025-04-25 23:14:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.28 | bwd_microstep: 5765.89 | bwd_inner_microstep: 5688.90 | bwd_allreduce_microstep: 76.94 | step_microstep: 19.30
[2025-04-25 23:14:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.28 | bwd: 5765.90 | bwd_inner: 5688.90 | bwd_allreduce: 76.96 | step: 19.30

 15%|█▌        | 6326/41250 [15:17:16<84:32:58,  8.72s/it]
                                                          
{'loss': 0.0792, 'grad_norm': 1.271106243133545, 'learning_rate': 3.8425183807960476e-05, 'epoch': 1.53}

 15%|█▌        | 6326/41250 [15:17:16<84:32:58,  8.72s/it][2025-04-25 23:14:59,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:14:59,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.38 | bwd_microstep: 5708.99 | bwd_inner_microstep: 5654.68 | bwd_allreduce_microstep: 54.27 | step_microstep: 18.37
[2025-04-25 23:14:59,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.38 | bwd: 5709.00 | bwd_inner: 5654.68 | bwd_allreduce: 54.28 | step: 18.37

 15%|█▌        | 6327/41250 [15:17:25<84:18:02,  8.69s/it]
                                                          
{'loss': 0.1701, 'grad_norm': 1.4566214084625244, 'learning_rate': 3.842457297437734e-05, 'epoch': 1.53}

 15%|█▌        | 6327/41250 [15:17:25<84:18:02,  8.69s/it][2025-04-25 23:15:08,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:15:08,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.66 | bwd_microstep: 5756.12 | bwd_inner_microstep: 5650.90 | bwd_allreduce_microstep: 105.17 | step_microstep: 18.95
[2025-04-25 23:15:08,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.66 | bwd: 5756.13 | bwd_inner: 5650.90 | bwd_allreduce: 105.19 | step: 18.94

 15%|█▌        | 6328/41250 [15:17:33<84:14:00,  8.68s/it]
                                                          
{'loss': 0.1314, 'grad_norm': 3.1762537956237793, 'learning_rate': 3.8423962027210337e-05, 'epoch': 1.53}

 15%|█▌        | 6328/41250 [15:17:33<84:14:00,  8.68s/it][2025-04-25 23:15:17,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:15:17,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5737.97 | bwd_inner_microstep: 5696.85 | bwd_allreduce_microstep: 41.07 | step_microstep: 18.60
[2025-04-25 23:15:17,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5737.98 | bwd_inner: 5696.85 | bwd_allreduce: 41.09 | step: 18.60

 15%|█▌        | 6329/41250 [15:17:42<84:11:15,  8.68s/it]
                                                          
{'loss': 0.2322, 'grad_norm': 2.1339259147644043, 'learning_rate': 3.842335096646323e-05, 'epoch': 1.53}

 15%|█▌        | 6329/41250 [15:17:42<84:11:15,  8.68s/it][2025-04-25 23:15:25,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:15:25,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.86 | bwd_microstep: 5689.08 | bwd_inner_microstep: 5660.13 | bwd_allreduce_microstep: 28.91 | step_microstep: 18.75
[2025-04-25 23:15:25,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.86 | bwd: 5689.09 | bwd_inner: 5660.13 | bwd_allreduce: 28.92 | step: 18.75

 15%|█▌        | 6330/41250 [15:17:51<83:58:25,  8.66s/it]
                                                          
{'loss': 0.0783, 'grad_norm': 3.986788034439087, 'learning_rate': 3.8422739792139785e-05, 'epoch': 1.53}

 15%|█▌        | 6330/41250 [15:17:51<83:58:25,  8.66s/it][2025-04-25 23:15:34,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 23:15:34,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5794.57 | bwd_inner_microstep: 5643.07 | bwd_allreduce_microstep: 151.45 | step_microstep: 19.40
[2025-04-25 23:15:34,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5794.58 | bwd_inner: 5643.07 | bwd_allreduce: 151.47 | step: 19.40

 15%|█▌        | 6331/41250 [15:17:59<84:06:05,  8.67s/it]
                                                          
{'loss': 0.0875, 'grad_norm': 1.0577313899993896, 'learning_rate': 3.842212850424377e-05, 'epoch': 1.53}

 15%|█▌        | 6331/41250 [15:17:59<84:06:05,  8.67s/it][2025-04-25 23:15:43,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:15:43,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.76 | bwd_microstep: 5876.07 | bwd_inner_microstep: 5863.42 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.35
[2025-04-25 23:15:43,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.76 | bwd: 5876.08 | bwd_inner: 5863.42 | bwd_allreduce: 12.62 | step: 18.36

 15%|█▌        | 6332/41250 [15:18:08<84:44:22,  8.74s/it]
                                                          
{'loss': 0.2582, 'grad_norm': 3.1905019283294678, 'learning_rate': 3.842151710277897e-05, 'epoch': 1.54}

 15%|█▌        | 6332/41250 [15:18:08<84:44:22,  8.74s/it][2025-04-25 23:15:52,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 23:15:52,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.31 | bwd_microstep: 5765.23 | bwd_inner_microstep: 5652.09 | bwd_allreduce_microstep: 113.09 | step_microstep: 18.55
[2025-04-25 23:15:52,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.31 | bwd: 5765.24 | bwd_inner: 5652.09 | bwd_allreduce: 113.11 | step: 18.55

 15%|█▌        | 6333/41250 [15:18:17<84:32:56,  8.72s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 0.8991254568099976, 'learning_rate': 3.8420905587749125e-05, 'epoch': 1.54}

 15%|█▌        | 6333/41250 [15:18:17<84:32:56,  8.72s/it][2025-04-25 23:16:00,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:16:00,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.81 | bwd_microstep: 5776.25 | bwd_inner_microstep: 5763.38 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.61
[2025-04-25 23:16:00,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.81 | bwd: 5776.27 | bwd_inner: 5763.38 | bwd_allreduce: 12.85 | step: 18.62

 15%|█▌        | 6334/41250 [15:18:26<84:36:41,  8.72s/it]
                                                          
{'loss': 0.0647, 'grad_norm': 1.2684556245803833, 'learning_rate': 3.842029395915803e-05, 'epoch': 1.54}

 15%|█▌        | 6334/41250 [15:18:26<84:36:41,  8.72s/it][2025-04-25 23:16:09,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:16:09,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.45 | bwd_microstep: 5687.51 | bwd_inner_microstep: 5658.03 | bwd_allreduce_microstep: 29.44 | step_microstep: 18.57
[2025-04-25 23:16:09,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.45 | bwd: 5687.52 | bwd_inner: 5658.03 | bwd_allreduce: 29.46 | step: 18.57

 15%|█▌        | 6335/41250 [15:18:34<84:14:30,  8.69s/it]
                                                          
{'loss': 0.0908, 'grad_norm': 1.1029443740844727, 'learning_rate': 3.8419682217009446e-05, 'epoch': 1.54}

 15%|█▌        | 6335/41250 [15:18:34<84:14:30,  8.69s/it][2025-04-25 23:16:18,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.10 | optimizer_step: 1.14
[2025-04-25 23:16:18,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.59 | bwd_microstep: 5711.88 | bwd_inner_microstep: 5628.47 | bwd_allreduce_microstep: 83.36 | step_microstep: 19.57
[2025-04-25 23:16:18,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.59 | bwd: 5711.90 | bwd_inner: 5628.47 | bwd_allreduce: 83.38 | step: 19.57

 15%|█▌        | 6336/41250 [15:18:43<84:02:06,  8.66s/it]
                                                          
{'loss': 0.2148, 'grad_norm': 1.357204794883728, 'learning_rate': 3.841907036130714e-05, 'epoch': 1.54}

 15%|█▌        | 6336/41250 [15:18:43<84:02:06,  8.66s/it][2025-04-25 23:16:26,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-25 23:16:26,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.52 | bwd_microstep: 5759.27 | bwd_inner_microstep: 5634.53 | bwd_allreduce_microstep: 124.70 | step_microstep: 18.60
[2025-04-25 23:16:26,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.52 | bwd: 5759.28 | bwd_inner: 5634.53 | bwd_allreduce: 124.72 | step: 18.60

 15%|█▌        | 6337/41250 [15:18:52<84:02:10,  8.67s/it]
                                                          
{'loss': 0.1328, 'grad_norm': 1.3781911134719849, 'learning_rate': 3.841845839205489e-05, 'epoch': 1.54}

 15%|█▌        | 6337/41250 [15:18:52<84:02:10,  8.67s/it][2025-04-25 23:16:35,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:16:35,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.19 | bwd_microstep: 5694.72 | bwd_inner_microstep: 5681.67 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.59
[2025-04-25 23:16:35,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.19 | bwd: 5694.73 | bwd_inner: 5681.67 | bwd_allreduce: 13.02 | step: 18.59

 15%|█▌        | 6338/41250 [15:19:00<83:54:20,  8.65s/it]
                                                          
{'loss': 0.0837, 'grad_norm': 1.908326506614685, 'learning_rate': 3.841784630925647e-05, 'epoch': 1.54}

 15%|█▌        | 6338/41250 [15:19:00<83:54:20,  8.65s/it][2025-04-25 23:16:43,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:16:43,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.38 | bwd_microstep: 5761.72 | bwd_inner_microstep: 5675.46 | bwd_allreduce_microstep: 86.22 | step_microstep: 18.31
[2025-04-25 23:16:43,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.38 | bwd: 5761.74 | bwd_inner: 5675.46 | bwd_allreduce: 86.24 | step: 18.32

 15%|█▌        | 6339/41250 [15:19:09<83:59:33,  8.66s/it]
                                                          
{'loss': 0.0444, 'grad_norm': 0.5092731714248657, 'learning_rate': 3.841723411291564e-05, 'epoch': 1.54}

 15%|█▌        | 6339/41250 [15:19:09<83:59:33,  8.66s/it][2025-04-25 23:16:52,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 23:16:52,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.24 | bwd_microstep: 5743.35 | bwd_inner_microstep: 5688.46 | bwd_allreduce_microstep: 54.85 | step_microstep: 18.42
[2025-04-25 23:16:52,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.24 | bwd: 5743.37 | bwd_inner: 5688.46 | bwd_allreduce: 54.87 | step: 18.42

 15%|█▌        | 6340/41250 [15:19:17<84:00:13,  8.66s/it]
                                                          
{'loss': 0.1315, 'grad_norm': 2.471027135848999, 'learning_rate': 3.841662180303619e-05, 'epoch': 1.54}

 15%|█▌        | 6340/41250 [15:19:17<84:00:13,  8.66s/it][2025-04-25 23:17:01,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:17:01,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.21 | bwd_microstep: 5771.30 | bwd_inner_microstep: 5758.65 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.16
[2025-04-25 23:17:01,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.21 | bwd: 5771.31 | bwd_inner: 5758.65 | bwd_allreduce: 12.62 | step: 18.16

 15%|█▌        | 6341/41250 [15:19:26<84:11:25,  8.68s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 1.2944872379302979, 'learning_rate': 3.841600937962189e-05, 'epoch': 1.54}

 15%|█▌        | 6341/41250 [15:19:26<84:11:25,  8.68s/it][2025-04-25 23:17:10,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:17:10,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.66 | bwd_microstep: 5750.12 | bwd_inner_microstep: 5703.29 | bwd_allreduce_microstep: 46.79 | step_microstep: 18.16
[2025-04-25 23:17:10,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.66 | bwd: 5750.13 | bwd_inner: 5703.29 | bwd_allreduce: 46.80 | step: 18.16

 15%|█▌        | 6342/41250 [15:19:35<84:11:34,  8.68s/it]
                                                          
{'loss': 0.1626, 'grad_norm': 2.2632155418395996, 'learning_rate': 3.8415396842676515e-05, 'epoch': 1.54}

 15%|█▌        | 6342/41250 [15:19:35<84:11:34,  8.68s/it][2025-04-25 23:17:18,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-25 23:17:18,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.66 | bwd_microstep: 5782.21 | bwd_inner_microstep: 5654.48 | bwd_allreduce_microstep: 127.67 | step_microstep: 18.95
[2025-04-25 23:17:18,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.66 | bwd: 5782.23 | bwd_inner: 5654.48 | bwd_allreduce: 127.70 | step: 18.95

 15%|█▌        | 6343/41250 [15:19:44<84:13:53,  8.69s/it]
                                                          
{'loss': 0.254, 'grad_norm': 2.0606119632720947, 'learning_rate': 3.841478419220384e-05, 'epoch': 1.54}

 15%|█▌        | 6343/41250 [15:19:44<84:13:53,  8.69s/it][2025-04-25 23:17:27,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-25 23:17:27,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5743.23 | bwd_inner_microstep: 5691.47 | bwd_allreduce_microstep: 51.70 | step_microstep: 19.28
[2025-04-25 23:17:27,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5743.24 | bwd_inner: 5691.47 | bwd_allreduce: 51.73 | step: 19.29

 15%|█▌        | 6344/41250 [15:19:52<84:12:25,  8.68s/it]
                                                          
{'loss': 0.1123, 'grad_norm': 0.8823277950286865, 'learning_rate': 3.8414171428207636e-05, 'epoch': 1.54}

 15%|█▌        | 6344/41250 [15:19:52<84:12:25,  8.68s/it][2025-04-25 23:17:36,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:17:36,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5706.23 | bwd_inner_microstep: 5694.03 | bwd_allreduce_microstep: 12.16 | step_microstep: 17.87
[2025-04-25 23:17:36,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5706.24 | bwd_inner: 5694.03 | bwd_allreduce: 12.17 | step: 17.87

 15%|█▌        | 6345/41250 [15:20:01<84:05:15,  8.67s/it]
                                                          
{'loss': 0.1126, 'grad_norm': 1.5557836294174194, 'learning_rate': 3.8413558550691694e-05, 'epoch': 1.54}

 15%|█▌        | 6345/41250 [15:20:01<84:05:15,  8.67s/it][2025-04-25 23:17:44,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-25 23:17:44,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.48 | bwd_microstep: 5716.99 | bwd_inner_microstep: 5704.15 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.11
[2025-04-25 23:17:44,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.48 | bwd: 5717.00 | bwd_inner: 5704.15 | bwd_allreduce: 12.81 | step: 19.12

 15%|█▌        | 6346/41250 [15:20:10<84:01:50,  8.67s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 1.717598557472229, 'learning_rate': 3.8412945559659784e-05, 'epoch': 1.54}

 15%|█▌        | 6346/41250 [15:20:10<84:01:50,  8.67s/it][2025-04-25 23:17:53,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:17:53,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.81 | bwd_microstep: 5775.67 | bwd_inner_microstep: 5654.91 | bwd_allreduce_microstep: 120.72 | step_microstep: 18.79
[2025-04-25 23:17:53,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.81 | bwd: 5775.69 | bwd_inner: 5654.91 | bwd_allreduce: 120.74 | step: 18.80

 15%|█▌        | 6347/41250 [15:20:18<84:06:20,  8.67s/it]
                                                          
{'loss': 0.0876, 'grad_norm': 2.974748134613037, 'learning_rate': 3.841233245511569e-05, 'epoch': 1.54}

 15%|█▌        | 6347/41250 [15:20:18<84:06:20,  8.67s/it][2025-04-25 23:18:02,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 23:18:02,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.08 | bwd_microstep: 5717.45 | bwd_inner_microstep: 5704.68 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.77
[2025-04-25 23:18:02,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.08 | bwd: 5717.46 | bwd_inner: 5704.68 | bwd_allreduce: 12.73 | step: 18.78

 15%|█▌        | 6348/41250 [15:20:27<84:04:09,  8.67s/it]
                                                          
{'loss': 0.1476, 'grad_norm': 1.43306303024292, 'learning_rate': 3.841171923706318e-05, 'epoch': 1.54}

 15%|█▌        | 6348/41250 [15:20:27<84:04:09,  8.67s/it][2025-04-25 23:18:10,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-25 23:18:10,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.60 | bwd_microstep: 5767.99 | bwd_inner_microstep: 5689.92 | bwd_allreduce_microstep: 78.02 | step_microstep: 19.21
[2025-04-25 23:18:10,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.60 | bwd: 5768.01 | bwd_inner: 5689.92 | bwd_allreduce: 78.04 | step: 19.22

 15%|█▌        | 6349/41250 [15:20:36<84:08:41,  8.68s/it]
                                                          
{'loss': 0.1022, 'grad_norm': 1.2463676929473877, 'learning_rate': 3.8411105905506047e-05, 'epoch': 1.54}

 15%|█▌        | 6349/41250 [15:20:36<84:08:41,  8.68s/it][2025-04-25 23:18:19,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-25 23:18:19,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.64 | bwd_microstep: 5770.30 | bwd_inner_microstep: 5692.81 | bwd_allreduce_microstep: 77.45 | step_microstep: 19.10
[2025-04-25 23:18:19,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.64 | bwd: 5770.32 | bwd_inner: 5692.81 | bwd_allreduce: 77.47 | step: 19.10

 15%|█▌        | 6350/41250 [15:20:44<84:14:05,  8.69s/it]
                                                          
{'loss': 0.0798, 'grad_norm': 1.6691595315933228, 'learning_rate': 3.841049246044807e-05, 'epoch': 1.54}

 15%|█▌        | 6350/41250 [15:20:44<84:14:05,  8.69s/it][2025-04-25 23:18:28,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:18:28,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.42 | bwd_microstep: 5782.10 | bwd_inner_microstep: 5769.03 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.76
[2025-04-25 23:18:28,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.42 | bwd: 5782.11 | bwd_inner: 5769.03 | bwd_allreduce: 13.05 | step: 18.76

 15%|█▌        | 6351/41250 [15:20:53<84:25:04,  8.71s/it]
                                                          
{'loss': 0.2382, 'grad_norm': 4.337391376495361, 'learning_rate': 3.840987890189302e-05, 'epoch': 1.54}

 15%|█▌        | 6351/41250 [15:20:53<84:25:04,  8.71s/it][2025-04-25 23:18:36,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-25 23:18:36,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.69 | bwd_microstep: 5713.82 | bwd_inner_microstep: 5657.60 | bwd_allreduce_microstep: 56.18 | step_microstep: 19.38
[2025-04-25 23:18:36,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5713.84 | bwd_inner: 5657.59 | bwd_allreduce: 56.20 | step: 19.38

 15%|█▌        | 6352/41250 [15:21:02<84:10:39,  8.68s/it]
                                                          
{'loss': 0.1941, 'grad_norm': 1.3374817371368408, 'learning_rate': 3.84092652298447e-05, 'epoch': 1.54}

 15%|█▌        | 6352/41250 [15:21:02<84:10:39,  8.68s/it][2025-04-25 23:18:45,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 23:18:45,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5784.94 | bwd_inner_microstep: 5654.59 | bwd_allreduce_microstep: 130.30 | step_microstep: 18.67
[2025-04-25 23:18:45,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5784.96 | bwd_inner: 5654.59 | bwd_allreduce: 130.32 | step: 18.67

 15%|█▌        | 6353/41250 [15:21:10<84:13:00,  8.69s/it]
                                                          
{'loss': 0.3126, 'grad_norm': 3.1010241508483887, 'learning_rate': 3.8408651444306867e-05, 'epoch': 1.54}

 15%|█▌        | 6353/41250 [15:21:10<84:13:00,  8.69s/it][2025-04-25 23:18:54,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 23:18:54,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5723.67 | bwd_inner_microstep: 5665.67 | bwd_allreduce_microstep: 57.95 | step_microstep: 18.93
[2025-04-25 23:18:54,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5723.69 | bwd_inner: 5665.67 | bwd_allreduce: 57.97 | step: 18.94

 15%|█▌        | 6354/41250 [15:21:19<84:03:54,  8.67s/it]
                                                          
{'loss': 0.2766, 'grad_norm': 5.51011848449707, 'learning_rate': 3.8408037545283325e-05, 'epoch': 1.54}

 15%|█▌        | 6354/41250 [15:21:19<84:03:54,  8.67s/it][2025-04-25 23:19:02,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 23:19:02,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.83 | bwd_microstep: 5717.71 | bwd_inner_microstep: 5662.41 | bwd_allreduce_microstep: 55.24 | step_microstep: 18.80
[2025-04-25 23:19:02,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.83 | bwd: 5717.73 | bwd_inner: 5662.41 | bwd_allreduce: 55.26 | step: 18.80

 15%|█▌        | 6355/41250 [15:21:28<83:58:13,  8.66s/it]
                                                          
{'loss': 0.2275, 'grad_norm': 2.243175745010376, 'learning_rate': 3.840742353277785e-05, 'epoch': 1.54}

 15%|█▌        | 6355/41250 [15:21:28<83:58:13,  8.66s/it][2025-04-25 23:19:11,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-25 23:19:11,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.57 | bwd_microstep: 5786.17 | bwd_inner_microstep: 5697.98 | bwd_allreduce_microstep: 88.15 | step_microstep: 19.08
[2025-04-25 23:19:11,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.57 | bwd: 5786.19 | bwd_inner: 5697.98 | bwd_allreduce: 88.17 | step: 19.08

 15%|█▌        | 6356/41250 [15:21:36<84:07:58,  8.68s/it]
                                                          
{'loss': 0.2598, 'grad_norm': 1.9220561981201172, 'learning_rate': 3.840680940679423e-05, 'epoch': 1.54}

 15%|█▌        | 6356/41250 [15:21:36<84:07:58,  8.68s/it][2025-04-25 23:19:20,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:19:20,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5778.75 | bwd_inner_microstep: 5688.29 | bwd_allreduce_microstep: 90.41 | step_microstep: 17.86
[2025-04-25 23:19:20,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5778.76 | bwd_inner: 5688.29 | bwd_allreduce: 90.43 | step: 17.86

 15%|█▌        | 6357/41250 [15:21:45<84:13:15,  8.69s/it]
                                                          
{'loss': 0.1712, 'grad_norm': 1.4553481340408325, 'learning_rate': 3.8406195167336256e-05, 'epoch': 1.54}

 15%|█▌        | 6357/41250 [15:21:45<84:13:15,  8.69s/it][2025-04-25 23:19:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:19:29,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.48 | bwd_microstep: 5778.08 | bwd_inner_microstep: 5765.25 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.21
[2025-04-25 23:19:29,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.48 | bwd: 5778.09 | bwd_inner: 5765.25 | bwd_allreduce: 12.80 | step: 18.21

 15%|█▌        | 6358/41250 [15:21:54<84:23:42,  8.71s/it]
                                                          
{'loss': 0.2759, 'grad_norm': 2.408078908920288, 'learning_rate': 3.840558081440771e-05, 'epoch': 1.54}

 15%|█▌        | 6358/41250 [15:21:54<84:23:42,  8.71s/it][2025-04-25 23:19:37,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:19:37,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.37 | bwd_microstep: 5723.69 | bwd_inner_microstep: 5711.06 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.61
[2025-04-25 23:19:37,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.37 | bwd: 5723.71 | bwd_inner: 5711.06 | bwd_allreduce: 12.60 | step: 18.61

 15%|█▌        | 6359/41250 [15:22:03<84:15:37,  8.69s/it]
                                                          
{'loss': 0.0579, 'grad_norm': 0.73814857006073, 'learning_rate': 3.8404966348012366e-05, 'epoch': 1.54}

 15%|█▌        | 6359/41250 [15:22:03<84:15:37,  8.69s/it][2025-04-25 23:19:46,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 23:19:46,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.52 | bwd_microstep: 5758.01 | bwd_inner_microstep: 5706.72 | bwd_allreduce_microstep: 51.25 | step_microstep: 18.87
[2025-04-25 23:19:46,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.52 | bwd: 5758.02 | bwd_inner: 5706.72 | bwd_allreduce: 51.27 | step: 18.88

 15%|█▌        | 6360/41250 [15:22:11<84:17:59,  8.70s/it]
                                                          
{'loss': 0.0492, 'grad_norm': 0.7702252268791199, 'learning_rate': 3.8404351768154034e-05, 'epoch': 1.54}

 15%|█▌        | 6360/41250 [15:22:11<84:17:59,  8.70s/it][2025-04-25 23:19:55,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:19:55,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.36 | bwd_microstep: 5796.31 | bwd_inner_microstep: 5783.29 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.64
[2025-04-25 23:19:55,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.36 | bwd: 5796.33 | bwd_inner: 5783.29 | bwd_allreduce: 13.00 | step: 18.64

 15%|█▌        | 6361/41250 [15:22:20<84:30:38,  8.72s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 1.7041698694229126, 'learning_rate': 3.840373707483649e-05, 'epoch': 1.54}

 15%|█▌        | 6361/41250 [15:22:20<84:30:38,  8.72s/it][2025-04-25 23:20:03,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-25 23:20:03,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.06 | bwd_microstep: 5700.47 | bwd_inner_microstep: 5659.46 | bwd_allreduce_microstep: 40.96 | step_microstep: 19.17
[2025-04-25 23:20:03,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.06 | bwd: 5700.48 | bwd_inner: 5659.46 | bwd_allreduce: 40.98 | step: 19.17

 15%|█▌        | 6362/41250 [15:22:29<84:13:36,  8.69s/it]
                                                          
{'loss': 0.3049, 'grad_norm': 2.535534143447876, 'learning_rate': 3.8403122268063524e-05, 'epoch': 1.54}

 15%|█▌        | 6362/41250 [15:22:29<84:13:36,  8.69s/it][2025-04-25 23:20:12,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:20:12,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.14 | bwd_microstep: 5894.64 | bwd_inner_microstep: 5664.33 | bwd_allreduce_microstep: 230.27 | step_microstep: 18.31
[2025-04-25 23:20:12,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.14 | bwd: 5894.65 | bwd_inner: 5664.33 | bwd_allreduce: 230.28 | step: 18.31

 15%|█▌        | 6363/41250 [15:22:37<84:36:21,  8.73s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 1.7101491689682007, 'learning_rate': 3.8402507347838934e-05, 'epoch': 1.54}

 15%|█▌        | 6363/41250 [15:22:37<84:36:21,  8.73s/it][2025-04-25 23:20:21,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:20:21,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5816.21 | bwd_inner_microstep: 5666.75 | bwd_allreduce_microstep: 149.41 | step_microstep: 18.63
[2025-04-25 23:20:21,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5816.22 | bwd_inner: 5666.75 | bwd_allreduce: 149.43 | step: 18.64

 15%|█▌        | 6364/41250 [15:22:46<84:36:05,  8.73s/it]
                                                          
{'loss': 0.1606, 'grad_norm': 1.178436040878296, 'learning_rate': 3.840189231416651e-05, 'epoch': 1.54}

 15%|█▌        | 6364/41250 [15:22:46<84:36:05,  8.73s/it][2025-04-25 23:20:30,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-25 23:20:30,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.23 | bwd_microstep: 5875.29 | bwd_inner_microstep: 5650.10 | bwd_allreduce_microstep: 225.15 | step_microstep: 17.74
[2025-04-25 23:20:30,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.23 | bwd: 5875.31 | bwd_inner: 5650.10 | bwd_allreduce: 225.17 | step: 17.74

 15%|█▌        | 6365/41250 [15:22:55<84:45:38,  8.75s/it]
                                                          
{'loss': 0.0824, 'grad_norm': 0.861057460308075, 'learning_rate': 3.8401277167050034e-05, 'epoch': 1.54}

 15%|█▌        | 6365/41250 [15:22:55<84:45:38,  8.75s/it][2025-04-25 23:20:38,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:20:38,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.21 | bwd_microstep: 5693.66 | bwd_inner_microstep: 5680.96 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.18
[2025-04-25 23:20:38,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.21 | bwd: 5693.67 | bwd_inner: 5680.95 | bwd_allreduce: 12.67 | step: 18.18

 15%|█▌        | 6366/41250 [15:23:04<84:23:56,  8.71s/it]
                                                          
{'loss': 0.1645, 'grad_norm': 1.7715907096862793, 'learning_rate': 3.8400661906493305e-05, 'epoch': 1.54}

 15%|█▌        | 6366/41250 [15:23:04<84:23:56,  8.71s/it][2025-04-25 23:20:47,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.14
[2025-04-25 23:20:47,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.74 | bwd_microstep: 5731.28 | bwd_inner_microstep: 5718.20 | bwd_allreduce_microstep: 13.03 | step_microstep: 19.12
[2025-04-25 23:20:47,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.74 | bwd: 5731.30 | bwd_inner: 5718.20 | bwd_allreduce: 13.05 | step: 19.13

 15%|█▌        | 6367/41250 [15:23:12<84:17:38,  8.70s/it]
                                                          
{'loss': 0.1697, 'grad_norm': 1.6265007257461548, 'learning_rate': 3.8400046532500114e-05, 'epoch': 1.54}

 15%|█▌        | 6367/41250 [15:23:12<84:17:38,  8.70s/it][2025-04-25 23:20:56,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-25 23:20:56,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5706.01 | bwd_inner_microstep: 5692.93 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.88
[2025-04-25 23:20:56,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5706.02 | bwd_inner: 5692.93 | bwd_allreduce: 13.05 | step: 18.88

 15%|█▌        | 6368/41250 [15:23:21<84:06:22,  8.68s/it]
                                                          
{'loss': 0.0884, 'grad_norm': 1.305525302886963, 'learning_rate': 3.839943104507425e-05, 'epoch': 1.54}

 15%|█▌        | 6368/41250 [15:23:21<84:06:22,  8.68s/it][2025-04-25 23:21:04,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-25 23:21:04,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.91 | bwd_microstep: 5705.48 | bwd_inner_microstep: 5692.68 | bwd_allreduce_microstep: 12.75 | step_microstep: 17.69
[2025-04-25 23:21:04,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.91 | bwd: 5705.49 | bwd_inner: 5692.68 | bwd_allreduce: 12.77 | step: 17.69

 15%|█▌        | 6369/41250 [15:23:30<84:00:00,  8.67s/it]
                                                          
{'loss': 0.0706, 'grad_norm': 1.3762401342391968, 'learning_rate': 3.8398815444219514e-05, 'epoch': 1.54}

 15%|█▌        | 6369/41250 [15:23:30<84:00:00,  8.67s/it][2025-04-25 23:21:13,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:21:13,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.56 | bwd_microstep: 5755.75 | bwd_inner_microstep: 5674.62 | bwd_allreduce_microstep: 81.08 | step_microstep: 18.72
[2025-04-25 23:21:13,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.56 | bwd: 5755.76 | bwd_inner: 5674.62 | bwd_allreduce: 81.10 | step: 18.72

 15%|█▌        | 6370/41250 [15:23:38<84:03:29,  8.68s/it]
                                                          
{'loss': 0.1052, 'grad_norm': 2.3331708908081055, 'learning_rate': 3.839819972993971e-05, 'epoch': 1.54}

 15%|█▌        | 6370/41250 [15:23:38<84:03:29,  8.68s/it][2025-04-25 23:21:22,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:21:22,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.08 | bwd_microstep: 5705.40 | bwd_inner_microstep: 5692.59 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.63
[2025-04-25 23:21:22,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.08 | bwd: 5705.42 | bwd_inner: 5692.59 | bwd_allreduce: 12.78 | step: 18.63

 15%|█▌        | 6371/41250 [15:23:47<83:57:34,  8.67s/it]
                                                          
{'loss': 0.0572, 'grad_norm': 0.6941977739334106, 'learning_rate': 3.839758390223861e-05, 'epoch': 1.54}

 15%|█▌        | 6371/41250 [15:23:47<83:57:34,  8.67s/it][2025-04-25 23:21:30,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 23:21:30,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.07 | bwd_microstep: 5689.53 | bwd_inner_microstep: 5677.05 | bwd_allreduce_microstep: 12.44 | step_microstep: 18.94
[2025-04-25 23:21:30,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.07 | bwd: 5689.55 | bwd_inner: 5677.05 | bwd_allreduce: 12.46 | step: 18.95

 15%|█▌        | 6372/41250 [15:23:55<83:48:30,  8.65s/it]
                                                          
{'loss': 0.3634, 'grad_norm': 2.819687604904175, 'learning_rate': 3.839696796112003e-05, 'epoch': 1.54}

 15%|█▌        | 6372/41250 [15:23:55<83:48:30,  8.65s/it][2025-04-25 23:21:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:21:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.26 | bwd_microstep: 5704.36 | bwd_inner_microstep: 5691.67 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.63
[2025-04-25 23:21:39,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.26 | bwd: 5704.37 | bwd_inner: 5691.67 | bwd_allreduce: 12.66 | step: 18.64

 15%|█▌        | 6373/41250 [15:24:04<83:48:37,  8.65s/it]
                                                          
{'loss': 0.0596, 'grad_norm': 0.6737018823623657, 'learning_rate': 3.839635190658776e-05, 'epoch': 1.54}

 15%|█▌        | 6373/41250 [15:24:04<83:48:37,  8.65s/it][2025-04-25 23:21:47,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:21:47,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5709.91 | bwd_inner_microstep: 5638.14 | bwd_allreduce_microstep: 71.73 | step_microstep: 18.20
[2025-04-25 23:21:47,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5709.92 | bwd_inner: 5638.14 | bwd_allreduce: 71.74 | step: 18.20

 15%|█▌        | 6374/41250 [15:24:13<83:44:00,  8.64s/it]
                                                          
{'loss': 0.1767, 'grad_norm': 1.642554521560669, 'learning_rate': 3.83957357386456e-05, 'epoch': 1.55}

 15%|█▌        | 6374/41250 [15:24:13<83:44:00,  8.64s/it][2025-04-25 23:21:56,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:21:56,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.08 | bwd_microstep: 5793.26 | bwd_inner_microstep: 5780.67 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.09
[2025-04-25 23:21:56,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.08 | bwd: 5793.27 | bwd_inner: 5780.67 | bwd_allreduce: 12.56 | step: 18.09

 15%|█▌        | 6375/41250 [15:24:22<84:05:04,  8.68s/it]
                                                          
{'loss': 0.1929, 'grad_norm': 1.835411787033081, 'learning_rate': 3.839511945729735e-05, 'epoch': 1.55}

 15%|█▌        | 6375/41250 [15:24:22<84:05:04,  8.68s/it][2025-04-25 23:22:05,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.08 | optimizer_step: 0.95
[2025-04-25 23:22:05,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.62 | bwd_microstep: 5727.60 | bwd_inner_microstep: 5688.94 | bwd_allreduce_microstep: 38.61 | step_microstep: 19.31
[2025-04-25 23:22:05,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.62 | bwd: 5727.62 | bwd_inner: 5688.94 | bwd_allreduce: 38.63 | step: 19.31

 15%|█▌        | 6376/41250 [15:24:30<84:01:47,  8.67s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.7381097674369812, 'learning_rate': 3.8394503062546805e-05, 'epoch': 1.55}

 15%|█▌        | 6376/41250 [15:24:30<84:01:47,  8.67s/it][2025-04-25 23:22:13,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.10 | optimizer_step: 0.90
[2025-04-25 23:22:13,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.12 | bwd_microstep: 5668.39 | bwd_inner_microstep: 5643.00 | bwd_allreduce_microstep: 25.35 | step_microstep: 19.38
[2025-04-25 23:22:13,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.12 | bwd: 5668.41 | bwd_inner: 5643.00 | bwd_allreduce: 25.36 | step: 19.38

 15%|█▌        | 6377/41250 [15:24:39<83:44:40,  8.65s/it]
                                                          
{'loss': 0.0175, 'grad_norm': 0.26265788078308105, 'learning_rate': 3.839388655439776e-05, 'epoch': 1.55}

 15%|█▌        | 6377/41250 [15:24:39<83:44:40,  8.65s/it][2025-04-25 23:22:22,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:22:22,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.62 | bwd_microstep: 5736.45 | bwd_inner_microstep: 5650.21 | bwd_allreduce_microstep: 86.19 | step_microstep: 18.85
[2025-04-25 23:22:22,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.62 | bwd: 5736.46 | bwd_inner: 5650.21 | bwd_allreduce: 86.21 | step: 18.85

 15%|█▌        | 6378/41250 [15:24:47<83:44:19,  8.64s/it]
                                                          
{'loss': 0.0327, 'grad_norm': 0.5730663537979126, 'learning_rate': 3.839326993285403e-05, 'epoch': 1.55}

 15%|█▌        | 6378/41250 [15:24:47<83:44:19,  8.64s/it][2025-04-25 23:22:31,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:22:31,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.38 | bwd_microstep: 5689.82 | bwd_inner_microstep: 5644.32 | bwd_allreduce_microstep: 45.46 | step_microstep: 18.64
[2025-04-25 23:22:31,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.38 | bwd: 5689.84 | bwd_inner: 5644.32 | bwd_allreduce: 45.48 | step: 18.64

 15%|█▌        | 6379/41250 [15:24:56<83:35:59,  8.63s/it]
                                                          
{'loss': 0.0192, 'grad_norm': 0.424753874540329, 'learning_rate': 3.8392653197919406e-05, 'epoch': 1.55}

 15%|█▌        | 6379/41250 [15:24:56<83:35:59,  8.63s/it][2025-04-25 23:22:39,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:22:39,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.60 | bwd_microstep: 5693.46 | bwd_inner_microstep: 5680.56 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.99
[2025-04-25 23:22:39,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.60 | bwd: 5693.48 | bwd_inner: 5680.56 | bwd_allreduce: 12.87 | step: 18.99

 15%|█▌        | 6380/41250 [15:25:05<83:33:16,  8.63s/it]
                                                          
{'loss': 0.1309, 'grad_norm': 1.7281415462493896, 'learning_rate': 3.83920363495977e-05, 'epoch': 1.55}

 15%|█▌        | 6380/41250 [15:25:05<83:33:16,  8.63s/it][2025-04-25 23:22:48,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-25 23:22:48,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.44 | bwd_microstep: 5760.06 | bwd_inner_microstep: 5646.36 | bwd_allreduce_microstep: 113.65 | step_microstep: 19.34
[2025-04-25 23:22:48,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.44 | bwd: 5760.07 | bwd_inner: 5646.35 | bwd_allreduce: 113.67 | step: 19.34

 15%|█▌        | 6381/41250 [15:25:13<83:41:24,  8.64s/it]
                                                          
{'loss': 0.2319, 'grad_norm': 1.3648968935012817, 'learning_rate': 3.839141938789269e-05, 'epoch': 1.55}

 15%|█▌        | 6381/41250 [15:25:13<83:41:24,  8.64s/it][2025-04-25 23:22:57,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-25 23:22:57,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5718.41 | bwd_inner_microstep: 5701.95 | bwd_allreduce_microstep: 16.42 | step_microstep: 18.41
[2025-04-25 23:22:57,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5718.42 | bwd_inner: 5701.95 | bwd_allreduce: 16.44 | step: 18.41

 15%|█▌        | 6382/41250 [15:25:22<83:44:09,  8.65s/it]
                                                          
{'loss': 0.0652, 'grad_norm': 3.1402828693389893, 'learning_rate': 3.8390802312808214e-05, 'epoch': 1.55}

 15%|█▌        | 6382/41250 [15:25:22<83:44:09,  8.65s/it][2025-04-25 23:23:05,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-25 23:23:05,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.86 | bwd_microstep: 5872.18 | bwd_inner_microstep: 5652.74 | bwd_allreduce_microstep: 219.39 | step_microstep: 18.38
[2025-04-25 23:23:05,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.86 | bwd: 5872.19 | bwd_inner: 5652.74 | bwd_allreduce: 219.41 | step: 18.38

 15%|█▌        | 6383/41250 [15:25:31<84:08:08,  8.69s/it]
                                                          
{'loss': 0.11, 'grad_norm': 1.6980472803115845, 'learning_rate': 3.839018512434805e-05, 'epoch': 1.55}

 15%|█▌        | 6383/41250 [15:25:31<84:08:08,  8.69s/it][2025-04-25 23:23:14,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:23:14,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.13 | bwd_microstep: 5672.43 | bwd_inner_microstep: 5642.88 | bwd_allreduce_microstep: 29.51 | step_microstep: 18.26
[2025-04-25 23:23:14,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.13 | bwd: 5672.44 | bwd_inner: 5642.88 | bwd_allreduce: 29.52 | step: 18.26

 15%|█▌        | 6384/41250 [15:25:39<83:48:53,  8.65s/it]
                                                          
{'loss': 0.4074, 'grad_norm': 2.9902312755584717, 'learning_rate': 3.838956782251601e-05, 'epoch': 1.55}

 15%|█▌        | 6384/41250 [15:25:39<83:48:53,  8.65s/it][2025-04-25 23:23:23,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 23:23:23,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.38 | bwd_microstep: 5717.78 | bwd_inner_microstep: 5705.26 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.17
[2025-04-25 23:23:23,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.38 | bwd: 5717.79 | bwd_inner: 5705.26 | bwd_allreduce: 12.49 | step: 18.17

 15%|█▌        | 6385/41250 [15:25:48<83:50:08,  8.66s/it]
                                                          
{'loss': 0.1655, 'grad_norm': 2.1123604774475098, 'learning_rate': 3.838895040731591e-05, 'epoch': 1.55}

 15%|█▌        | 6385/41250 [15:25:48<83:50:08,  8.66s/it][2025-04-25 23:23:31,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-25 23:23:31,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.72 | bwd_microstep: 5782.55 | bwd_inner_microstep: 5769.85 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.82
[2025-04-25 23:23:31,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.72 | bwd: 5782.56 | bwd_inner: 5769.86 | bwd_allreduce: 12.66 | step: 18.82

 15%|█▌        | 6386/41250 [15:25:57<84:08:29,  8.69s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 1.067183256149292, 'learning_rate': 3.838833287875155e-05, 'epoch': 1.55}

 15%|█▌        | 6386/41250 [15:25:57<84:08:29,  8.69s/it][2025-04-25 23:23:40,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:23:40,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.78 | bwd_microstep: 5746.72 | bwd_inner_microstep: 5688.17 | bwd_allreduce_microstep: 58.51 | step_microstep: 18.71
[2025-04-25 23:23:40,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.78 | bwd: 5746.74 | bwd_inner: 5688.17 | bwd_allreduce: 58.53 | step: 18.71

 15%|█▌        | 6387/41250 [15:26:05<84:05:54,  8.68s/it]
                                                          
{'loss': 0.0713, 'grad_norm': 0.9125928282737732, 'learning_rate': 3.838771523682672e-05, 'epoch': 1.55}

 15%|█▌        | 6387/41250 [15:26:05<84:05:54,  8.68s/it][2025-04-25 23:23:49,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:23:49,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.45 | bwd_microstep: 5753.00 | bwd_inner_microstep: 5657.39 | bwd_allreduce_microstep: 95.57 | step_microstep: 18.11
[2025-04-25 23:23:49,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.45 | bwd: 5753.01 | bwd_inner: 5657.39 | bwd_allreduce: 95.59 | step: 18.11

 15%|█▌        | 6388/41250 [15:26:14<84:04:34,  8.68s/it]
                                                          
{'loss': 0.4079, 'grad_norm': 3.556107759475708, 'learning_rate': 3.838709748154525e-05, 'epoch': 1.55}

 15%|█▌        | 6388/41250 [15:26:14<84:04:34,  8.68s/it][2025-04-25 23:23:57,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 23:23:57,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.54 | bwd_microstep: 5695.32 | bwd_inner_microstep: 5652.87 | bwd_allreduce_microstep: 42.40 | step_microstep: 18.33
[2025-04-25 23:23:57,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.54 | bwd: 5695.33 | bwd_inner: 5652.87 | bwd_allreduce: 42.42 | step: 18.33

 15%|█▌        | 6389/41250 [15:26:23<83:53:30,  8.66s/it]
                                                          
{'loss': 0.2551, 'grad_norm': 2.5806314945220947, 'learning_rate': 3.838647961291094e-05, 'epoch': 1.55}

 15%|█▌        | 6389/41250 [15:26:23<83:53:30,  8.66s/it][2025-04-25 23:24:06,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 23:24:06,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.23 | bwd_microstep: 5748.05 | bwd_inner_microstep: 5679.93 | bwd_allreduce_microstep: 68.08 | step_microstep: 18.91
[2025-04-25 23:24:06,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.23 | bwd: 5748.06 | bwd_inner: 5679.93 | bwd_allreduce: 68.09 | step: 18.91

 15%|█▌        | 6390/41250 [15:26:31<83:55:24,  8.67s/it]
                                                          
{'loss': 0.2002, 'grad_norm': 3.760608434677124, 'learning_rate': 3.83858616309276e-05, 'epoch': 1.55}

 15%|█▌        | 6390/41250 [15:26:31<83:55:24,  8.67s/it][2025-04-25 23:24:15,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:24:15,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.29 | bwd_microstep: 5771.72 | bwd_inner_microstep: 5654.99 | bwd_allreduce_microstep: 116.69 | step_microstep: 18.00
[2025-04-25 23:24:15,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.29 | bwd: 5771.73 | bwd_inner: 5654.99 | bwd_allreduce: 116.70 | step: 18.00

 15%|█▌        | 6391/41250 [15:26:40<83:59:02,  8.67s/it]
                                                          
{'loss': 0.065, 'grad_norm': 1.1326466798782349, 'learning_rate': 3.838524353559904e-05, 'epoch': 1.55}

 15%|█▌        | 6391/41250 [15:26:40<83:59:02,  8.67s/it][2025-04-25 23:24:23,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:24:23,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.64 | bwd_microstep: 5755.84 | bwd_inner_microstep: 5687.74 | bwd_allreduce_microstep: 68.06 | step_microstep: 18.24
[2025-04-25 23:24:23,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.64 | bwd: 5755.85 | bwd_inner: 5687.74 | bwd_allreduce: 68.07 | step: 18.24

 15%|█▌        | 6392/41250 [15:26:49<84:00:52,  8.68s/it]
                                                          
{'loss': 0.1636, 'grad_norm': 2.2691152095794678, 'learning_rate': 3.838462532692907e-05, 'epoch': 1.55}

 15%|█▌        | 6392/41250 [15:26:49<84:00:52,  8.68s/it][2025-04-25 23:24:32,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-25 23:24:32,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5719.56 | bwd_inner_microstep: 5652.44 | bwd_allreduce_microstep: 67.07 | step_microstep: 18.75
[2025-04-25 23:24:32,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5719.58 | bwd_inner: 5652.44 | bwd_allreduce: 67.09 | step: 18.75

 15%|█▌        | 6393/41250 [15:26:57<83:53:19,  8.66s/it]
                                                          
{'loss': 0.1342, 'grad_norm': 2.1804003715515137, 'learning_rate': 3.83840070049215e-05, 'epoch': 1.55}

 15%|█▌        | 6393/41250 [15:26:57<83:53:19,  8.66s/it][2025-04-25 23:24:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:24:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.42 | bwd_microstep: 5728.26 | bwd_inner_microstep: 5715.50 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.57
[2025-04-25 23:24:41,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.42 | bwd: 5728.27 | bwd_inner: 5715.50 | bwd_allreduce: 12.73 | step: 18.57

 16%|█▌        | 6394/41250 [15:27:06<83:54:37,  8.67s/it]
                                                          
{'loss': 0.3134, 'grad_norm': 4.783049583435059, 'learning_rate': 3.8383388569580146e-05, 'epoch': 1.55}

 16%|█▌        | 6394/41250 [15:27:06<83:54:37,  8.67s/it][2025-04-25 23:24:49,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:24:49,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5755.91 | bwd_inner_microstep: 5703.41 | bwd_allreduce_microstep: 52.45 | step_microstep: 18.76
[2025-04-25 23:24:49,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5755.92 | bwd_inner: 5703.41 | bwd_allreduce: 52.47 | step: 18.76

 16%|█▌        | 6395/41250 [15:27:15<83:58:06,  8.67s/it]
                                                          
{'loss': 0.1556, 'grad_norm': 1.6841742992401123, 'learning_rate': 3.838277002090881e-05, 'epoch': 1.55}

 16%|█▌        | 6395/41250 [15:27:15<83:58:06,  8.67s/it][2025-04-25 23:24:58,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:24:58,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.92 | bwd_microstep: 5779.28 | bwd_inner_microstep: 5705.72 | bwd_allreduce_microstep: 73.52 | step_microstep: 18.72
[2025-04-25 23:24:58,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.92 | bwd: 5779.30 | bwd_inner: 5705.72 | bwd_allreduce: 73.53 | step: 18.72

 16%|█▌        | 6396/41250 [15:27:23<84:05:28,  8.69s/it]
                                                          
{'loss': 0.3169, 'grad_norm': 2.5497405529022217, 'learning_rate': 3.838215135891132e-05, 'epoch': 1.55}

 16%|█▌        | 6396/41250 [15:27:23<84:05:28,  8.69s/it][2025-04-25 23:25:07,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-25 23:25:07,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.90 | bwd_microstep: 5795.02 | bwd_inner_microstep: 5662.00 | bwd_allreduce_microstep: 132.97 | step_microstep: 18.99
[2025-04-25 23:25:07,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.90 | bwd: 5795.04 | bwd_inner: 5662.00 | bwd_allreduce: 132.99 | step: 19.00

 16%|█▌        | 6397/41250 [15:27:32<84:09:21,  8.69s/it]
                                                          
{'loss': 0.2818, 'grad_norm': 2.053745985031128, 'learning_rate': 3.8381532583591477e-05, 'epoch': 1.55}

 16%|█▌        | 6397/41250 [15:27:32<84:09:21,  8.69s/it][2025-04-25 23:25:16,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 23:25:16,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.46 | bwd_microstep: 5798.03 | bwd_inner_microstep: 5785.25 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.02
[2025-04-25 23:25:16,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.46 | bwd: 5798.05 | bwd_inner: 5785.25 | bwd_allreduce: 12.75 | step: 19.02

 16%|█▌        | 6398/41250 [15:27:41<84:25:25,  8.72s/it]
                                                          
{'loss': 0.1884, 'grad_norm': 3.421018600463867, 'learning_rate': 3.838091369495311e-05, 'epoch': 1.55}

 16%|█▌        | 6398/41250 [15:27:41<84:25:25,  8.72s/it][2025-04-25 23:25:24,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-25 23:25:24,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.36 | bwd_microstep: 5806.56 | bwd_inner_microstep: 5665.60 | bwd_allreduce_microstep: 140.91 | step_microstep: 19.13
[2025-04-25 23:25:24,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.36 | bwd: 5806.57 | bwd_inner: 5665.60 | bwd_allreduce: 140.93 | step: 19.13

 16%|█▌        | 6399/41250 [15:27:50<84:26:40,  8.72s/it]
                                                          
{'loss': 0.0457, 'grad_norm': 1.0786139965057373, 'learning_rate': 3.838029469300001e-05, 'epoch': 1.55}

 16%|█▌        | 6399/41250 [15:27:50<84:26:40,  8.72s/it][2025-04-25 23:25:33,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:25:33,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.22 | bwd_microstep: 5802.34 | bwd_inner_microstep: 5699.37 | bwd_allreduce_microstep: 102.92 | step_microstep: 18.81
[2025-04-25 23:25:33,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.22 | bwd: 5802.35 | bwd_inner: 5699.37 | bwd_allreduce: 102.94 | step: 18.82

 16%|█▌        | 6400/41250 [15:27:58<84:28:40,  8.73s/it]
                                                          
{'loss': 0.1274, 'grad_norm': 1.5270748138427734, 'learning_rate': 3.837967557773602e-05, 'epoch': 1.55}

 16%|█▌        | 6400/41250 [15:27:58<84:28:40,  8.73s/it][2025-04-25 23:25:42,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 23:25:42,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.66 | bwd_microstep: 5756.24 | bwd_inner_microstep: 5719.67 | bwd_allreduce_microstep: 36.53 | step_microstep: 18.82
[2025-04-25 23:25:42,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.66 | bwd: 5756.26 | bwd_inner: 5719.67 | bwd_allreduce: 36.55 | step: 18.83

 16%|█▌        | 6401/41250 [15:28:07<84:23:47,  8.72s/it]
                                                          
{'loss': 0.0159, 'grad_norm': 0.19412730634212494, 'learning_rate': 3.837905634916494e-05, 'epoch': 1.55}

 16%|█▌        | 6401/41250 [15:28:07<84:23:47,  8.72s/it][2025-04-25 23:25:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:25:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.34 | bwd_microstep: 5782.74 | bwd_inner_microstep: 5661.89 | bwd_allreduce_microstep: 120.80 | step_microstep: 18.46
[2025-04-25 23:25:51,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.34 | bwd: 5782.75 | bwd_inner: 5661.89 | bwd_allreduce: 120.81 | step: 18.47

 16%|█▌        | 6402/41250 [15:28:16<84:19:20,  8.71s/it]
                                                          
{'loss': 0.1682, 'grad_norm': 2.125520706176758, 'learning_rate': 3.83784370072906e-05, 'epoch': 1.55}

 16%|█▌        | 6402/41250 [15:28:16<84:19:20,  8.71s/it][2025-04-25 23:25:59,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 23:25:59,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.61 | bwd_microstep: 5715.46 | bwd_inner_microstep: 5663.75 | bwd_allreduce_microstep: 51.66 | step_microstep: 18.47
[2025-04-25 23:25:59,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.61 | bwd: 5715.47 | bwd_inner: 5663.75 | bwd_allreduce: 51.68 | step: 18.47

 16%|█▌        | 6403/41250 [15:28:24<84:06:48,  8.69s/it]
                                                          
{'loss': 0.0487, 'grad_norm': 0.8214364647865295, 'learning_rate': 3.83778175521168e-05, 'epoch': 1.55}

 16%|█▌        | 6403/41250 [15:28:24<84:06:48,  8.69s/it][2025-04-25 23:26:08,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:26:08,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.76 | bwd_microstep: 5770.15 | bwd_inner_microstep: 5693.03 | bwd_allreduce_microstep: 77.08 | step_microstep: 18.80
[2025-04-25 23:26:08,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.76 | bwd: 5770.16 | bwd_inner: 5693.03 | bwd_allreduce: 77.09 | step: 18.81

 16%|█▌        | 6404/41250 [15:28:33<84:08:39,  8.69s/it]
                                                          
{'loss': 0.315, 'grad_norm': 3.033923625946045, 'learning_rate': 3.837719798364738e-05, 'epoch': 1.55}

 16%|█▌        | 6404/41250 [15:28:33<84:08:39,  8.69s/it][2025-04-25 23:26:17,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 23:26:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.89 | bwd_microstep: 5741.38 | bwd_inner_microstep: 5714.57 | bwd_allreduce_microstep: 26.77 | step_microstep: 18.78
[2025-04-25 23:26:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.89 | bwd: 5741.40 | bwd_inner: 5714.57 | bwd_allreduce: 26.79 | step: 18.79

 16%|█▌        | 6405/41250 [15:28:42<84:06:23,  8.69s/it]
                                                          
{'loss': 0.1567, 'grad_norm': 2.174812078475952, 'learning_rate': 3.837657830188614e-05, 'epoch': 1.55}

 16%|█▌        | 6405/41250 [15:28:42<84:06:23,  8.69s/it][2025-04-25 23:26:25,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:26:25,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.96 | bwd_microstep: 5781.74 | bwd_inner_microstep: 5663.08 | bwd_allreduce_microstep: 118.61 | step_microstep: 18.30
[2025-04-25 23:26:25,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.96 | bwd: 5781.75 | bwd_inner: 5663.08 | bwd_allreduce: 118.63 | step: 18.30

 16%|█▌        | 6406/41250 [15:28:51<84:08:14,  8.69s/it]
                                                          
{'loss': 0.0588, 'grad_norm': 1.5345295667648315, 'learning_rate': 3.8375958506836914e-05, 'epoch': 1.55}

 16%|█▌        | 6406/41250 [15:28:51<84:08:14,  8.69s/it][2025-04-25 23:26:34,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:26:34,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.08 | bwd_microstep: 5794.13 | bwd_inner_microstep: 5781.39 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.77
[2025-04-25 23:26:34,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.08 | bwd: 5794.14 | bwd_inner: 5781.39 | bwd_allreduce: 12.71 | step: 18.77

 16%|█▌        | 6407/41250 [15:28:59<84:20:50,  8.71s/it]
                                                          
{'loss': 0.1584, 'grad_norm': 2.236931324005127, 'learning_rate': 3.8375338598503515e-05, 'epoch': 1.55}

 16%|█▌        | 6407/41250 [15:28:59<84:20:50,  8.71s/it][2025-04-25 23:26:43,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-25 23:26:43,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.46 | bwd_microstep: 5704.35 | bwd_inner_microstep: 5651.49 | bwd_allreduce_microstep: 52.81 | step_microstep: 19.10
[2025-04-25 23:26:43,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.46 | bwd: 5704.36 | bwd_inner: 5651.49 | bwd_allreduce: 52.83 | step: 19.10

 16%|█▌        | 6408/41250 [15:29:08<84:03:11,  8.68s/it]
                                                          
{'loss': 0.2041, 'grad_norm': 3.319100856781006, 'learning_rate': 3.837471857688978e-05, 'epoch': 1.55}

 16%|█▌        | 6408/41250 [15:29:08<84:03:11,  8.68s/it][2025-04-25 23:26:51,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:26:51,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5849.85 | bwd_inner_microstep: 5685.90 | bwd_allreduce_microstep: 163.91 | step_microstep: 18.44
[2025-04-25 23:26:51,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5849.86 | bwd_inner: 5685.90 | bwd_allreduce: 163.92 | step: 18.44

 16%|█▌        | 6409/41250 [15:29:17<84:20:16,  8.71s/it]
                                                          
{'loss': 0.233, 'grad_norm': 6.563120365142822, 'learning_rate': 3.83740984419995e-05, 'epoch': 1.55}

 16%|█▌        | 6409/41250 [15:29:17<84:20:16,  8.71s/it][2025-04-25 23:27:00,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:27:00,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5697.32 | bwd_inner_microstep: 5657.23 | bwd_allreduce_microstep: 40.05 | step_microstep: 18.55
[2025-04-25 23:27:00,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5697.33 | bwd_inner: 5657.23 | bwd_allreduce: 40.06 | step: 18.56

 16%|█▌        | 6410/41250 [15:29:25<84:01:13,  8.68s/it]
                                                          
{'loss': 0.203, 'grad_norm': 1.9401817321777344, 'learning_rate': 3.837347819383653e-05, 'epoch': 1.55}

 16%|█▌        | 6410/41250 [15:29:25<84:01:13,  8.68s/it][2025-04-25 23:27:09,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:27:09,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5702.13 | bwd_inner_microstep: 5645.22 | bwd_allreduce_microstep: 56.86 | step_microstep: 18.70
[2025-04-25 23:27:09,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5702.14 | bwd_inner: 5645.22 | bwd_allreduce: 56.88 | step: 18.71

 16%|█▌        | 6411/41250 [15:29:34<83:48:54,  8.66s/it]
                                                          
{'loss': 0.2174, 'grad_norm': 2.8672077655792236, 'learning_rate': 3.837285783240468e-05, 'epoch': 1.55}

 16%|█▌        | 6411/41250 [15:29:34<83:48:54,  8.66s/it][2025-04-25 23:27:17,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 23:27:17,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.92 | bwd_microstep: 5779.21 | bwd_inner_microstep: 5702.67 | bwd_allreduce_microstep: 76.48 | step_microstep: 19.13
[2025-04-25 23:27:17,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.92 | bwd: 5779.22 | bwd_inner: 5702.67 | bwd_allreduce: 76.51 | step: 19.13

 16%|█▌        | 6412/41250 [15:29:43<83:57:07,  8.68s/it]
                                                          
{'loss': 0.1219, 'grad_norm': 1.4335182905197144, 'learning_rate': 3.8372237357707766e-05, 'epoch': 1.55}

 16%|█▌        | 6412/41250 [15:29:43<83:57:07,  8.68s/it][2025-04-25 23:27:26,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-25 23:27:26,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5716.72 | bwd_inner_microstep: 5703.95 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.62
[2025-04-25 23:27:26,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5716.74 | bwd_inner: 5703.95 | bwd_allreduce: 12.74 | step: 18.62

 16%|█▌        | 6413/41250 [15:29:51<83:53:10,  8.67s/it]
                                                          
{'loss': 0.1453, 'grad_norm': 5.373349189758301, 'learning_rate': 3.8371616769749635e-05, 'epoch': 1.55}

 16%|█▌        | 6413/41250 [15:29:51<83:53:10,  8.67s/it][2025-04-25 23:27:35,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 23:27:35,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.62 | bwd_microstep: 5692.00 | bwd_inner_microstep: 5642.57 | bwd_allreduce_microstep: 49.38 | step_microstep: 19.10
[2025-04-25 23:27:35,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.62 | bwd: 5692.01 | bwd_inner: 5642.57 | bwd_allreduce: 49.40 | step: 19.10

 16%|█▌        | 6414/41250 [15:30:00<83:43:19,  8.65s/it]
                                                          
{'loss': 0.2311, 'grad_norm': 1.9397656917572021, 'learning_rate': 3.837099606853409e-05, 'epoch': 1.55}

 16%|█▌        | 6414/41250 [15:30:00<83:43:19,  8.65s/it][2025-04-25 23:27:43,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:27:43,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.68 | bwd_microstep: 5766.17 | bwd_inner_microstep: 5633.11 | bwd_allreduce_microstep: 133.02 | step_microstep: 18.39
[2025-04-25 23:27:43,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.68 | bwd: 5766.19 | bwd_inner: 5633.11 | bwd_allreduce: 133.04 | step: 18.39

 16%|█▌        | 6415/41250 [15:30:09<83:46:20,  8.66s/it]
                                                          
{'loss': 0.1318, 'grad_norm': 1.7532000541687012, 'learning_rate': 3.837037525406497e-05, 'epoch': 1.56}

 16%|█▌        | 6415/41250 [15:30:09<83:46:20,  8.66s/it][2025-04-25 23:27:52,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.08 | optimizer_step: 1.04
[2025-04-25 23:27:52,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.07 | bwd_microstep: 5780.38 | bwd_inner_microstep: 5648.56 | bwd_allreduce_microstep: 131.77 | step_microstep: 19.61
[2025-04-25 23:27:52,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.07 | bwd: 5780.40 | bwd_inner: 5648.56 | bwd_allreduce: 131.79 | step: 19.61

 16%|█▌        | 6416/41250 [15:30:17<83:51:55,  8.67s/it]
                                                          
{'loss': 0.1273, 'grad_norm': 1.3529399633407593, 'learning_rate': 3.83697543263461e-05, 'epoch': 1.56}

 16%|█▌        | 6416/41250 [15:30:17<83:51:55,  8.67s/it][2025-04-25 23:28:01,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:28:01,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.36 | bwd_microstep: 5848.20 | bwd_inner_microstep: 5649.55 | bwd_allreduce_microstep: 198.61 | step_microstep: 18.43
[2025-04-25 23:28:01,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.36 | bwd: 5848.22 | bwd_inner: 5649.55 | bwd_allreduce: 198.62 | step: 18.43

 16%|█▌        | 6417/41250 [15:30:26<84:09:49,  8.70s/it]
                                                          
{'loss': 0.1191, 'grad_norm': 1.3118690252304077, 'learning_rate': 3.836913328538131e-05, 'epoch': 1.56}

 16%|█▌        | 6417/41250 [15:30:26<84:09:49,  8.70s/it][2025-04-25 23:28:10,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 23:28:10,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2930.45 | bwd_microstep: 5876.01 | bwd_inner_microstep: 5863.00 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.00
[2025-04-25 23:28:10,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2930.45 | bwd: 5876.03 | bwd_inner: 5863.00 | bwd_allreduce: 12.99 | step: 19.00

 16%|█▌        | 6418/41250 [15:30:35<84:43:12,  8.76s/it]
                                                          
{'loss': 0.2123, 'grad_norm': 2.53794002532959, 'learning_rate': 3.836851213117443e-05, 'epoch': 1.56}

 16%|█▌        | 6418/41250 [15:30:35<84:43:12,  8.76s/it][2025-04-25 23:28:18,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:28:18,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5749.58 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 105.84 | step_microstep: 18.43
[2025-04-25 23:28:18,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5749.59 | bwd_inner: 5643.69 | bwd_allreduce: 105.86 | step: 18.44

 16%|█▌        | 6419/41250 [15:30:44<84:26:18,  8.73s/it]
                                                          
{'loss': 0.1109, 'grad_norm': 1.634657859802246, 'learning_rate': 3.836789086372928e-05, 'epoch': 1.56}

 16%|█▌        | 6419/41250 [15:30:44<84:26:18,  8.73s/it][2025-04-25 23:28:27,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-25 23:28:27,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.87 | bwd_microstep: 5704.54 | bwd_inner_microstep: 5691.94 | bwd_allreduce_microstep: 12.54 | step_microstep: 19.39
[2025-04-25 23:28:27,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.87 | bwd: 5704.55 | bwd_inner: 5691.94 | bwd_allreduce: 12.56 | step: 19.39

 16%|█▌        | 6420/41250 [15:30:52<84:09:56,  8.70s/it]
                                                          
{'loss': 0.1211, 'grad_norm': 1.7086992263793945, 'learning_rate': 3.83672694830497e-05, 'epoch': 1.56}

 16%|█▌        | 6420/41250 [15:30:52<84:09:56,  8.70s/it][2025-04-25 23:28:36,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.19 | optimizer_step: 0.96
[2025-04-25 23:28:36,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.51 | bwd_microstep: 5690.70 | bwd_inner_microstep: 5677.35 | bwd_allreduce_microstep: 13.29 | step_microstep: 19.57
[2025-04-25 23:28:36,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.51 | bwd: 5690.71 | bwd_inner: 5677.35 | bwd_allreduce: 13.32 | step: 19.57

 16%|█▌        | 6421/41250 [15:31:01<83:55:00,  8.67s/it]
                                                          
{'loss': 0.1649, 'grad_norm': 2.66927170753479, 'learning_rate': 3.836664798913951e-05, 'epoch': 1.56}

 16%|█▌        | 6421/41250 [15:31:01<83:55:00,  8.67s/it][2025-04-25 23:28:44,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:28:44,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2812.14 | bwd_microstep: 5767.76 | bwd_inner_microstep: 5623.46 | bwd_allreduce_microstep: 144.26 | step_microstep: 18.58
[2025-04-25 23:28:44,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2812.14 | bwd: 5767.78 | bwd_inner: 5623.46 | bwd_allreduce: 144.28 | step: 18.59

 16%|█▌        | 6422/41250 [15:31:10<83:53:11,  8.67s/it]
                                                          
{'loss': 0.1314, 'grad_norm': 1.4072176218032837, 'learning_rate': 3.836602638200255e-05, 'epoch': 1.56}

 16%|█▌        | 6422/41250 [15:31:10<83:53:11,  8.67s/it][2025-04-25 23:28:53,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:28:53,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.59 | bwd_microstep: 5661.53 | bwd_inner_microstep: 5641.37 | bwd_allreduce_microstep: 20.11 | step_microstep: 18.51
[2025-04-25 23:28:53,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.59 | bwd: 5661.54 | bwd_inner: 5641.37 | bwd_allreduce: 20.13 | step: 18.51

 16%|█▌        | 6423/41250 [15:31:18<83:35:28,  8.64s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 0.6427146792411804, 'learning_rate': 3.836540466164265e-05, 'epoch': 1.56}

 16%|█▌        | 6423/41250 [15:31:18<83:35:28,  8.64s/it][2025-04-25 23:29:01,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:29:01,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.62 | bwd_microstep: 5744.76 | bwd_inner_microstep: 5631.26 | bwd_allreduce_microstep: 113.46 | step_microstep: 18.60
[2025-04-25 23:29:01,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.62 | bwd: 5744.78 | bwd_inner: 5631.26 | bwd_allreduce: 113.47 | step: 18.60

 16%|█▌        | 6424/41250 [15:31:27<83:35:29,  8.64s/it]
                                                          
{'loss': 0.1406, 'grad_norm': 1.5662806034088135, 'learning_rate': 3.836478282806364e-05, 'epoch': 1.56}

 16%|█▌        | 6424/41250 [15:31:27<83:35:29,  8.64s/it][2025-04-25 23:29:10,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:29:10,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.62 | bwd_microstep: 5681.47 | bwd_inner_microstep: 5666.66 | bwd_allreduce_microstep: 14.76 | step_microstep: 18.16
[2025-04-25 23:29:10,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.62 | bwd: 5681.49 | bwd_inner: 5666.66 | bwd_allreduce: 14.78 | step: 18.16

 16%|█▌        | 6425/41250 [15:31:35<83:29:07,  8.63s/it]
                                                          
{'loss': 0.11, 'grad_norm': 5.398685455322266, 'learning_rate': 3.836416088126936e-05, 'epoch': 1.56}

 16%|█▌        | 6425/41250 [15:31:35<83:29:07,  8.63s/it][2025-04-25 23:29:19,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-25 23:29:19,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.29 | bwd_microstep: 5758.65 | bwd_inner_microstep: 5644.10 | bwd_allreduce_microstep: 114.50 | step_microstep: 19.42
[2025-04-25 23:29:19,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.29 | bwd: 5758.67 | bwd_inner: 5644.10 | bwd_allreduce: 114.52 | step: 19.43

 16%|█▌        | 6426/41250 [15:31:44<83:34:45,  8.64s/it]
                                                          
{'loss': 0.2367, 'grad_norm': 1.8415865898132324, 'learning_rate': 3.836353882126364e-05, 'epoch': 1.56}

 16%|█▌        | 6426/41250 [15:31:44<83:34:45,  8.64s/it][2025-04-25 23:29:27,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 23:29:27,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5767.44 | bwd_inner_microstep: 5679.52 | bwd_allreduce_microstep: 87.87 | step_microstep: 19.06
[2025-04-25 23:29:27,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5767.45 | bwd_inner: 5679.52 | bwd_allreduce: 87.88 | step: 19.06

 16%|█▌        | 6427/41250 [15:31:53<83:44:29,  8.66s/it]
                                                          
{'loss': 0.261, 'grad_norm': 1.4756821393966675, 'learning_rate': 3.8362916648050314e-05, 'epoch': 1.56}

 16%|█▌        | 6427/41250 [15:31:53<83:44:29,  8.66s/it][2025-04-25 23:29:36,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:29:36,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.09 | bwd_microstep: 5746.80 | bwd_inner_microstep: 5687.01 | bwd_allreduce_microstep: 59.74 | step_microstep: 18.69
[2025-04-25 23:29:36,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.09 | bwd: 5746.82 | bwd_inner: 5687.01 | bwd_allreduce: 59.76 | step: 18.69

 16%|█▌        | 6428/41250 [15:32:01<83:47:54,  8.66s/it]
                                                          
{'loss': 0.0899, 'grad_norm': 1.1788970232009888, 'learning_rate': 3.836229436163322e-05, 'epoch': 1.56}

 16%|█▌        | 6428/41250 [15:32:01<83:47:54,  8.66s/it][2025-04-25 23:29:45,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:29:45,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.10 | bwd_microstep: 5773.40 | bwd_inner_microstep: 5630.85 | bwd_allreduce_microstep: 142.50 | step_microstep: 18.75
[2025-04-25 23:29:45,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.10 | bwd: 5773.41 | bwd_inner: 5630.85 | bwd_allreduce: 142.52 | step: 18.75

 16%|█▌        | 6429/41250 [15:32:10<83:49:16,  8.67s/it]
                                                          
{'loss': 0.2916, 'grad_norm': 3.1990723609924316, 'learning_rate': 3.8361671962016196e-05, 'epoch': 1.56}

 16%|█▌        | 6429/41250 [15:32:10<83:49:16,  8.67s/it][2025-04-25 23:29:53,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 23:29:53,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.20 | bwd_microstep: 5713.60 | bwd_inner_microstep: 5642.43 | bwd_allreduce_microstep: 71.13 | step_microstep: 18.59
[2025-04-25 23:29:53,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.20 | bwd: 5713.61 | bwd_inner: 5642.43 | bwd_allreduce: 71.14 | step: 18.59

 16%|█▌        | 6430/41250 [15:32:19<83:40:45,  8.65s/it]
                                                          
{'loss': 0.0947, 'grad_norm': 1.03264319896698, 'learning_rate': 3.836104944920307e-05, 'epoch': 1.56}

 16%|█▌        | 6430/41250 [15:32:19<83:40:45,  8.65s/it][2025-04-25 23:30:02,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 23:30:02,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.66 | bwd_microstep: 5689.65 | bwd_inner_microstep: 5654.18 | bwd_allreduce_microstep: 35.43 | step_microstep: 18.79
[2025-04-25 23:30:02,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.66 | bwd: 5689.66 | bwd_inner: 5654.18 | bwd_allreduce: 35.44 | step: 18.79

 16%|█▌        | 6431/41250 [15:32:27<83:30:46,  8.63s/it]
                                                          
{'loss': 0.0405, 'grad_norm': 0.9460753202438354, 'learning_rate': 3.836042682319769e-05, 'epoch': 1.56}

 16%|█▌        | 6431/41250 [15:32:27<83:30:46,  8.63s/it][2025-04-25 23:30:11,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-25 23:30:11,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5735.84 | bwd_inner_microstep: 5697.55 | bwd_allreduce_microstep: 38.25 | step_microstep: 18.75
[2025-04-25 23:30:11,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5735.85 | bwd_inner: 5697.55 | bwd_allreduce: 38.26 | step: 18.75

 16%|█▌        | 6432/41250 [15:32:36<83:36:32,  8.64s/it]
                                                          
{'loss': 0.0543, 'grad_norm': 0.9420209527015686, 'learning_rate': 3.8359804084003885e-05, 'epoch': 1.56}

 16%|█▌        | 6432/41250 [15:32:36<83:36:32,  8.64s/it][2025-04-25 23:30:19,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-25 23:30:19,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.24 | bwd_microstep: 5704.40 | bwd_inner_microstep: 5659.74 | bwd_allreduce_microstep: 44.61 | step_microstep: 18.68
[2025-04-25 23:30:19,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.24 | bwd: 5704.42 | bwd_inner: 5659.74 | bwd_allreduce: 44.63 | step: 18.68

 16%|█▌        | 6433/41250 [15:32:45<83:32:47,  8.64s/it]
                                                          
{'loss': 0.1441, 'grad_norm': 3.5703020095825195, 'learning_rate': 3.8359181231625495e-05, 'epoch': 1.56}

 16%|█▌        | 6433/41250 [15:32:45<83:32:47,  8.64s/it][2025-04-25 23:30:28,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 23:30:28,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.02 | bwd_microstep: 5759.50 | bwd_inner_microstep: 5653.43 | bwd_allreduce_microstep: 106.02 | step_microstep: 18.80
[2025-04-25 23:30:28,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.02 | bwd: 5759.52 | bwd_inner: 5653.43 | bwd_allreduce: 106.04 | step: 18.81

 16%|█▌        | 6434/41250 [15:32:53<83:37:58,  8.65s/it]
                                                          
{'loss': 0.1265, 'grad_norm': 1.7166829109191895, 'learning_rate': 3.835855826606637e-05, 'epoch': 1.56}

 16%|█▌        | 6434/41250 [15:32:53<83:37:58,  8.65s/it][2025-04-25 23:30:37,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 23:30:37,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5776.58 | bwd_inner_microstep: 5651.97 | bwd_allreduce_microstep: 124.56 | step_microstep: 18.73
[2025-04-25 23:30:37,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 5776.59 | bwd_inner: 5651.97 | bwd_allreduce: 124.58 | step: 18.74

 16%|█▌        | 6435/41250 [15:33:02<83:44:09,  8.66s/it]
                                                          
{'loss': 0.2233, 'grad_norm': 3.1787025928497314, 'learning_rate': 3.8357935187330345e-05, 'epoch': 1.56}

 16%|█▌        | 6435/41250 [15:33:02<83:44:09,  8.66s/it][2025-04-25 23:30:45,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:30:45,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5729.70 | bwd_inner_microstep: 5652.98 | bwd_allreduce_microstep: 76.67 | step_microstep: 19.03
[2025-04-25 23:30:45,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5729.71 | bwd_inner: 5652.98 | bwd_allreduce: 76.69 | step: 19.03

 16%|█▌        | 6436/41250 [15:33:11<83:40:32,  8.65s/it]
                                                          
{'loss': 0.1675, 'grad_norm': 4.309459209442139, 'learning_rate': 3.8357311995421264e-05, 'epoch': 1.56}

 16%|█▌        | 6436/41250 [15:33:11<83:40:32,  8.65s/it][2025-04-25 23:30:54,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-25 23:30:54,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5747.20 | bwd_inner_microstep: 5693.06 | bwd_allreduce_microstep: 54.09 | step_microstep: 19.52
[2025-04-25 23:30:54,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5747.21 | bwd_inner: 5693.06 | bwd_allreduce: 54.11 | step: 19.52

 16%|█▌        | 6437/41250 [15:33:19<83:46:03,  8.66s/it]
                                                          
{'loss': 0.0444, 'grad_norm': 0.6888256669044495, 'learning_rate': 3.835668869034296e-05, 'epoch': 1.56}

 16%|█▌        | 6437/41250 [15:33:19<83:46:03,  8.66s/it][2025-04-25 23:31:03,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-25 23:31:03,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5745.70 | bwd_inner_microstep: 5701.38 | bwd_allreduce_microstep: 44.27 | step_microstep: 19.43
[2025-04-25 23:31:03,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5745.72 | bwd_inner: 5701.38 | bwd_allreduce: 44.29 | step: 19.43

 16%|█▌        | 6438/41250 [15:33:28<83:48:54,  8.67s/it]
                                                          
{'loss': 0.178, 'grad_norm': 2.5696096420288086, 'learning_rate': 3.835606527209928e-05, 'epoch': 1.56}

 16%|█▌        | 6438/41250 [15:33:28<83:48:54,  8.67s/it][2025-04-25 23:31:11,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:31:11,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.42 | bwd_microstep: 5771.21 | bwd_inner_microstep: 5693.33 | bwd_allreduce_microstep: 77.83 | step_microstep: 18.82
[2025-04-25 23:31:11,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.42 | bwd: 5771.23 | bwd_inner: 5693.33 | bwd_allreduce: 77.85 | step: 18.83

 16%|█▌        | 6439/41250 [15:33:37<83:55:04,  8.68s/it]
                                                          
{'loss': 0.2511, 'grad_norm': 2.5942022800445557, 'learning_rate': 3.835544174069407e-05, 'epoch': 1.56}

 16%|█▌        | 6439/41250 [15:33:37<83:55:04,  8.68s/it][2025-04-25 23:31:20,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 23:31:20,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5766.99 | bwd_inner_microstep: 5651.95 | bwd_allreduce_microstep: 115.00 | step_microstep: 18.88
[2025-04-25 23:31:20,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5767.00 | bwd_inner: 5651.95 | bwd_allreduce: 115.01 | step: 18.88

 16%|█▌        | 6440/41250 [15:33:45<83:54:44,  8.68s/it]
                                                          
{'loss': 0.1952, 'grad_norm': 1.965164065361023, 'learning_rate': 3.835481809613117e-05, 'epoch': 1.56}

 16%|█▌        | 6440/41250 [15:33:45<83:54:44,  8.68s/it][2025-04-25 23:31:29,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:31:29,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.62 | bwd_microstep: 5722.90 | bwd_inner_microstep: 5657.49 | bwd_allreduce_microstep: 65.36 | step_microstep: 18.59
[2025-04-25 23:31:29,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.62 | bwd: 5722.91 | bwd_inner: 5657.49 | bwd_allreduce: 65.38 | step: 18.59

 16%|█▌        | 6441/41250 [15:33:54<83:47:05,  8.67s/it]
                                                          
{'loss': 0.2532, 'grad_norm': 4.52247953414917, 'learning_rate': 3.835419433841443e-05, 'epoch': 1.56}

 16%|█▌        | 6441/41250 [15:33:54<83:47:05,  8.67s/it][2025-04-25 23:31:37,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 23:31:37,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.71 | bwd_microstep: 5715.19 | bwd_inner_microstep: 5702.31 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.36
[2025-04-25 23:31:37,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.71 | bwd: 5715.20 | bwd_inner: 5702.31 | bwd_allreduce: 12.85 | step: 19.36

 16%|█▌        | 6442/41250 [15:34:03<83:45:44,  8.66s/it]
                                                          
{'loss': 0.0461, 'grad_norm': 1.430730938911438, 'learning_rate': 3.835357046754768e-05, 'epoch': 1.56}

 16%|█▌        | 6442/41250 [15:34:03<83:45:44,  8.66s/it][2025-04-25 23:31:46,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.12 | optimizer_step: 1.01
[2025-04-25 23:31:46,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.90 | bwd_microstep: 5713.66 | bwd_inner_microstep: 5700.24 | bwd_allreduce_microstep: 13.38 | step_microstep: 19.33
[2025-04-25 23:31:46,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.90 | bwd: 5713.67 | bwd_inner: 5700.23 | bwd_allreduce: 13.40 | step: 19.34

 16%|█▌        | 6443/41250 [15:34:11<83:43:25,  8.66s/it]
                                                          
{'loss': 0.1694, 'grad_norm': 1.183205008506775, 'learning_rate': 3.83529464835348e-05, 'epoch': 1.56}

 16%|█▌        | 6443/41250 [15:34:11<83:43:25,  8.66s/it][2025-04-25 23:31:55,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-25 23:31:55,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.22 | bwd_microstep: 5785.62 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 137.12 | step_microstep: 18.91
[2025-04-25 23:31:55,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.22 | bwd: 5785.63 | bwd_inner: 5648.45 | bwd_allreduce: 137.14 | step: 18.91

 16%|█▌        | 6444/41250 [15:34:20<83:49:07,  8.67s/it]
                                                          
{'loss': 0.0645, 'grad_norm': 1.8186874389648438, 'learning_rate': 3.83523223863796e-05, 'epoch': 1.56}

 16%|█▌        | 6444/41250 [15:34:20<83:49:07,  8.67s/it][2025-04-25 23:32:03,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:32:03,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.82 | bwd_microstep: 5717.07 | bwd_inner_microstep: 5652.53 | bwd_allreduce_microstep: 64.49 | step_microstep: 18.82
[2025-04-25 23:32:03,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.82 | bwd: 5717.08 | bwd_inner: 5652.53 | bwd_allreduce: 64.51 | step: 18.82

 16%|█▌        | 6445/41250 [15:34:29<83:42:11,  8.66s/it]
                                                          
{'loss': 0.4479, 'grad_norm': 9.998440742492676, 'learning_rate': 3.835169817608594e-05, 'epoch': 1.56}

 16%|█▌        | 6445/41250 [15:34:29<83:42:11,  8.66s/it][2025-04-25 23:32:12,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 23:32:12,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.47 | bwd_microstep: 5705.22 | bwd_inner_microstep: 5637.20 | bwd_allreduce_microstep: 67.97 | step_microstep: 19.04
[2025-04-25 23:32:12,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.47 | bwd: 5705.23 | bwd_inner: 5637.20 | bwd_allreduce: 67.98 | step: 19.04

 16%|█▌        | 6446/41250 [15:34:37<83:36:02,  8.65s/it]
                                                          
{'loss': 0.0455, 'grad_norm': 0.6092248558998108, 'learning_rate': 3.835107385265768e-05, 'epoch': 1.56}

 16%|█▌        | 6446/41250 [15:34:37<83:36:02,  8.65s/it][2025-04-25 23:32:21,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:32:21,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.35 | bwd_microstep: 5785.98 | bwd_inner_microstep: 5648.71 | bwd_allreduce_microstep: 137.22 | step_microstep: 18.60
[2025-04-25 23:32:21,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.35 | bwd: 5786.00 | bwd_inner: 5648.71 | bwd_allreduce: 137.24 | step: 18.60

 16%|█▌        | 6447/41250 [15:34:46<83:45:00,  8.66s/it]
                                                          
{'loss': 0.1225, 'grad_norm': 4.48621129989624, 'learning_rate': 3.835044941609865e-05, 'epoch': 1.56}

 16%|█▌        | 6447/41250 [15:34:46<83:45:00,  8.66s/it][2025-04-25 23:32:29,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 23:32:29,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.60 | bwd_microstep: 5783.24 | bwd_inner_microstep: 5656.99 | bwd_allreduce_microstep: 126.20 | step_microstep: 18.81
[2025-04-25 23:32:29,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.60 | bwd: 5783.25 | bwd_inner: 5656.99 | bwd_allreduce: 126.22 | step: 18.82

 16%|█▌        | 6448/41250 [15:34:55<83:50:26,  8.67s/it]
                                                          
{'loss': 0.2236, 'grad_norm': 1.4648945331573486, 'learning_rate': 3.834982486641272e-05, 'epoch': 1.56}

 16%|█▌        | 6448/41250 [15:34:55<83:50:26,  8.67s/it][2025-04-25 23:32:38,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-25 23:32:38,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.78 | bwd_microstep: 5776.37 | bwd_inner_microstep: 5654.36 | bwd_allreduce_microstep: 121.96 | step_microstep: 18.81
[2025-04-25 23:32:38,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.78 | bwd: 5776.39 | bwd_inner: 5654.36 | bwd_allreduce: 121.99 | step: 18.81

 16%|█▌        | 6449/41250 [15:35:03<83:52:53,  8.68s/it]
                                                          
{'loss': 0.0827, 'grad_norm': 1.7769064903259277, 'learning_rate': 3.834920020360372e-05, 'epoch': 1.56}

 16%|█▌        | 6449/41250 [15:35:03<83:52:53,  8.68s/it][2025-04-25 23:32:47,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:32:47,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5722.32 | bwd_inner_microstep: 5636.82 | bwd_allreduce_microstep: 85.45 | step_microstep: 18.55
[2025-04-25 23:32:47,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5722.34 | bwd_inner: 5636.82 | bwd_allreduce: 85.47 | step: 18.56

 16%|█▌        | 6450/41250 [15:35:12<83:45:57,  8.67s/it]
                                                          
{'loss': 0.0232, 'grad_norm': 0.284763365983963, 'learning_rate': 3.834857542767551e-05, 'epoch': 1.56}

 16%|█▌        | 6450/41250 [15:35:12<83:45:57,  8.67s/it][2025-04-25 23:32:56,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:32:56,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2950.58 | bwd_microstep: 5894.92 | bwd_inner_microstep: 5882.15 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.75
[2025-04-25 23:32:56,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2950.58 | bwd: 5894.94 | bwd_inner: 5882.15 | bwd_allreduce: 12.74 | step: 18.75

 16%|█▌        | 6451/41250 [15:35:21<84:33:29,  8.75s/it]
                                                          
{'loss': 0.0583, 'grad_norm': 0.7552205920219421, 'learning_rate': 3.834795053863195e-05, 'epoch': 1.56}

 16%|█▌        | 6451/41250 [15:35:21<84:33:29,  8.75s/it][2025-04-25 23:33:04,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 23:33:04,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.26 | bwd_microstep: 5704.33 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 13.39 | step_microstep: 19.40
[2025-04-25 23:33:04,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.26 | bwd: 5704.34 | bwd_inner: 5690.89 | bwd_allreduce: 13.41 | step: 19.40

 16%|█▌        | 6452/41250 [15:35:29<84:14:31,  8.72s/it]
                                                          
{'loss': 0.1264, 'grad_norm': 1.262980341911316, 'learning_rate': 3.834732553647687e-05, 'epoch': 1.56}

 16%|█▌        | 6452/41250 [15:35:29<84:14:31,  8.72s/it][2025-04-25 23:33:13,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 23:33:13,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5776.61 | bwd_inner_microstep: 5650.55 | bwd_allreduce_microstep: 126.01 | step_microstep: 18.63
[2025-04-25 23:33:13,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5776.62 | bwd_inner: 5650.55 | bwd_allreduce: 126.03 | step: 18.64

 16%|█▌        | 6453/41250 [15:35:38<84:09:32,  8.71s/it]
                                                          
{'loss': 0.0655, 'grad_norm': 1.6514068841934204, 'learning_rate': 3.8346700421214147e-05, 'epoch': 1.56}

 16%|█▌        | 6453/41250 [15:35:38<84:09:32,  8.71s/it][2025-04-25 23:33:22,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:33:22,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.35 | bwd_microstep: 5778.54 | bwd_inner_microstep: 5649.09 | bwd_allreduce_microstep: 129.40 | step_microstep: 18.51
[2025-04-25 23:33:22,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.36 | bwd: 5778.55 | bwd_inner: 5649.09 | bwd_allreduce: 129.42 | step: 18.52

 16%|█▌        | 6454/41250 [15:35:47<84:05:59,  8.70s/it]
                                                          
{'loss': 0.0932, 'grad_norm': 1.6076817512512207, 'learning_rate': 3.8346075192847623e-05, 'epoch': 1.56}

 16%|█▌        | 6454/41250 [15:35:47<84:05:59,  8.70s/it][2025-04-25 23:33:30,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-25 23:33:30,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.20 | bwd_microstep: 5762.92 | bwd_inner_microstep: 5706.69 | bwd_allreduce_microstep: 56.18 | step_microstep: 18.39
[2025-04-25 23:33:30,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.20 | bwd: 5762.93 | bwd_inner: 5706.69 | bwd_allreduce: 56.20 | step: 18.39

 16%|█▌        | 6455/41250 [15:35:56<84:04:45,  8.70s/it]
                                                          
{'loss': 0.0602, 'grad_norm': 1.73088538646698, 'learning_rate': 3.834544985138115e-05, 'epoch': 1.56}

 16%|█▌        | 6455/41250 [15:35:56<84:04:45,  8.70s/it][2025-04-25 23:33:39,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:33:39,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.40 | bwd_microstep: 5764.01 | bwd_inner_microstep: 5638.21 | bwd_allreduce_microstep: 125.75 | step_microstep: 18.43
[2025-04-25 23:33:39,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.40 | bwd: 5764.02 | bwd_inner: 5638.21 | bwd_allreduce: 125.77 | step: 18.44

 16%|█▌        | 6456/41250 [15:36:04<83:59:42,  8.69s/it]
                                                          
{'loss': 0.2443, 'grad_norm': 1.6093748807907104, 'learning_rate': 3.834482439681859e-05, 'epoch': 1.57}

 16%|█▌        | 6456/41250 [15:36:04<83:59:42,  8.69s/it][2025-04-25 23:33:48,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:33:48,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.79 | bwd_microstep: 5754.16 | bwd_inner_microstep: 5652.23 | bwd_allreduce_microstep: 101.88 | step_microstep: 18.42
[2025-04-25 23:33:48,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.79 | bwd: 5754.17 | bwd_inner: 5652.23 | bwd_allreduce: 101.90 | step: 18.42

 16%|█▌        | 6457/41250 [15:36:13<83:54:52,  8.68s/it]
                                                          
{'loss': 0.1026, 'grad_norm': 2.2397894859313965, 'learning_rate': 3.83441988291638e-05, 'epoch': 1.57}

 16%|█▌        | 6457/41250 [15:36:13<83:54:52,  8.68s/it][2025-04-25 23:33:56,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:33:56,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.07 | bwd_microstep: 5688.06 | bwd_inner_microstep: 5647.60 | bwd_allreduce_microstep: 40.42 | step_microstep: 18.70
[2025-04-25 23:33:56,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.07 | bwd: 5688.07 | bwd_inner: 5647.60 | bwd_allreduce: 40.43 | step: 18.70

 16%|█▌        | 6458/41250 [15:36:21<83:39:59,  8.66s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 2.352433204650879, 'learning_rate': 3.8343573148420626e-05, 'epoch': 1.57}

 16%|█▌        | 6458/41250 [15:36:21<83:39:59,  8.66s/it][2025-04-25 23:34:05,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:34:05,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.73 | bwd_microstep: 5683.51 | bwd_inner_microstep: 5659.88 | bwd_allreduce_microstep: 23.58 | step_microstep: 18.87
[2025-04-25 23:34:05,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.73 | bwd: 5683.52 | bwd_inner: 5659.88 | bwd_allreduce: 23.60 | step: 18.87

 16%|█▌        | 6459/41250 [15:36:30<83:29:04,  8.64s/it]
                                                          
{'loss': 0.2929, 'grad_norm': 3.325901508331299, 'learning_rate': 3.8342947354592925e-05, 'epoch': 1.57}

 16%|█▌        | 6459/41250 [15:36:30<83:29:04,  8.64s/it][2025-04-25 23:34:13,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 23:34:13,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5691.87 | bwd_inner_microstep: 5637.72 | bwd_allreduce_microstep: 54.11 | step_microstep: 19.39
[2025-04-25 23:34:13,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5691.89 | bwd_inner: 5637.72 | bwd_allreduce: 54.12 | step: 19.40

 16%|█▌        | 6460/41250 [15:36:39<83:23:10,  8.63s/it]
                                                          
{'loss': 0.0246, 'grad_norm': 0.45593973994255066, 'learning_rate': 3.834232144768457e-05, 'epoch': 1.57}

 16%|█▌        | 6460/41250 [15:36:39<83:23:10,  8.63s/it][2025-04-25 23:34:22,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:34:22,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.10 | bwd_microstep: 5893.65 | bwd_inner_microstep: 5644.40 | bwd_allreduce_microstep: 249.20 | step_microstep: 18.80
[2025-04-25 23:34:22,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.10 | bwd: 5893.66 | bwd_inner: 5644.40 | bwd_allreduce: 249.22 | step: 18.81

 16%|█▌        | 6461/41250 [15:36:47<83:52:52,  8.68s/it]
                                                          
{'loss': 0.1552, 'grad_norm': 2.607750654220581, 'learning_rate': 3.834169542769941e-05, 'epoch': 1.57}

 16%|█▌        | 6461/41250 [15:36:47<83:52:52,  8.68s/it][2025-04-25 23:34:31,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-25 23:34:31,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.27 | bwd_microstep: 5874.11 | bwd_inner_microstep: 5665.60 | bwd_allreduce_microstep: 208.46 | step_microstep: 19.21
[2025-04-25 23:34:31,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.27 | bwd: 5874.12 | bwd_inner: 5665.60 | bwd_allreduce: 208.48 | step: 19.21

 16%|█▌        | 6462/41250 [15:36:56<84:14:06,  8.72s/it]
                                                          
{'loss': 0.0658, 'grad_norm': 3.7407913208007812, 'learning_rate': 3.834106929464131e-05, 'epoch': 1.57}

 16%|█▌        | 6462/41250 [15:36:56<84:14:06,  8.72s/it][2025-04-25 23:34:40,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-25 23:34:40,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.78 | bwd_microstep: 5698.91 | bwd_inner_microstep: 5636.33 | bwd_allreduce_microstep: 62.54 | step_microstep: 19.27
[2025-04-25 23:34:40,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.78 | bwd: 5698.93 | bwd_inner: 5636.33 | bwd_allreduce: 62.55 | step: 19.27

 16%|█▌        | 6463/41250 [15:37:05<83:54:15,  8.68s/it]
                                                          
{'loss': 0.2012, 'grad_norm': 1.6177374124526978, 'learning_rate': 3.8340443048514124e-05, 'epoch': 1.57}

 16%|█▌        | 6463/41250 [15:37:05<83:54:15,  8.68s/it][2025-04-25 23:34:48,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-25 23:34:48,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.09 | bwd_microstep: 5755.62 | bwd_inner_microstep: 5742.81 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.93
[2025-04-25 23:34:48,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.09 | bwd: 5755.63 | bwd_inner: 5742.81 | bwd_allreduce: 12.78 | step: 18.93

 16%|█▌        | 6464/41250 [15:37:14<83:59:23,  8.69s/it]
                                                          
{'loss': 0.1112, 'grad_norm': 1.4033679962158203, 'learning_rate': 3.8339816689321706e-05, 'epoch': 1.57}

 16%|█▌        | 6464/41250 [15:37:14<83:59:23,  8.69s/it][2025-04-25 23:34:57,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.32 | optimizer_step: 1.08
[2025-04-25 23:34:57,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.74 | bwd_microstep: 5771.31 | bwd_inner_microstep: 5758.00 | bwd_allreduce_microstep: 13.24 | step_microstep: 20.38
[2025-04-25 23:34:57,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.74 | bwd: 5771.32 | bwd_inner: 5758.00 | bwd_allreduce: 13.27 | step: 20.39

 16%|█▌        | 6465/41250 [15:37:22<84:07:52,  8.71s/it]
                                                          
{'loss': 0.122, 'grad_norm': 1.4551782608032227, 'learning_rate': 3.833919021706793e-05, 'epoch': 1.57}

 16%|█▌        | 6465/41250 [15:37:22<84:07:52,  8.71s/it][2025-04-25 23:35:06,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-25 23:35:06,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.67 | bwd_microstep: 5741.00 | bwd_inner_microstep: 5642.15 | bwd_allreduce_microstep: 98.79 | step_microstep: 18.67
[2025-04-25 23:35:06,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.67 | bwd: 5741.01 | bwd_inner: 5642.15 | bwd_allreduce: 98.81 | step: 18.67

 16%|█▌        | 6466/41250 [15:37:31<83:57:54,  8.69s/it]
                                                          
{'loss': 0.0835, 'grad_norm': 2.646437644958496, 'learning_rate': 3.8338563631756655e-05, 'epoch': 1.57}

 16%|█▌        | 6466/41250 [15:37:31<83:57:54,  8.69s/it][2025-04-25 23:35:14,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:35:14,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.19 | bwd_microstep: 5702.31 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 58.58 | step_microstep: 18.59
[2025-04-25 23:35:14,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.19 | bwd: 5702.33 | bwd_inner: 5643.69 | bwd_allreduce: 58.59 | step: 18.59

 16%|█▌        | 6467/41250 [15:37:40<83:43:03,  8.66s/it]
                                                          
{'loss': 0.2738, 'grad_norm': 12.091011047363281, 'learning_rate': 3.833793693339175e-05, 'epoch': 1.57}

 16%|█▌        | 6467/41250 [15:37:40<83:43:03,  8.66s/it][2025-04-25 23:35:23,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.05 | optimizer_step: 1.23
[2025-04-25 23:35:23,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.70 | bwd_microstep: 5681.78 | bwd_inner_microstep: 5669.00 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.57
[2025-04-25 23:35:23,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.70 | bwd: 5681.79 | bwd_inner: 5669.00 | bwd_allreduce: 12.75 | step: 19.57

 16%|█▌        | 6468/41250 [15:37:48<83:34:44,  8.65s/it]
                                                          
{'loss': 0.205, 'grad_norm': 1.6103428602218628, 'learning_rate': 3.833731012197706e-05, 'epoch': 1.57}

 16%|█▌        | 6468/41250 [15:37:48<83:34:44,  8.65s/it][2025-04-25 23:35:32,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 23:35:32,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.20 | bwd_microstep: 5895.27 | bwd_inner_microstep: 5669.31 | bwd_allreduce_microstep: 225.92 | step_microstep: 18.46
[2025-04-25 23:35:32,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.20 | bwd: 5895.28 | bwd_inner: 5669.31 | bwd_allreduce: 225.94 | step: 18.46

 16%|█▌        | 6469/41250 [15:37:57<84:05:01,  8.70s/it]
                                                          
{'loss': 0.2557, 'grad_norm': 2.159867286682129, 'learning_rate': 3.833668319751646e-05, 'epoch': 1.57}

 16%|█▌        | 6469/41250 [15:37:57<84:05:01,  8.70s/it][2025-04-25 23:35:40,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-25 23:35:40,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.53 | bwd_microstep: 5725.42 | bwd_inner_microstep: 5692.39 | bwd_allreduce_microstep: 32.97 | step_microstep: 19.06
[2025-04-25 23:35:40,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.53 | bwd: 5725.43 | bwd_inner: 5692.39 | bwd_allreduce: 32.99 | step: 19.06

 16%|█▌        | 6470/41250 [15:38:06<83:57:27,  8.69s/it]
                                                          
{'loss': 0.1963, 'grad_norm': 1.671687126159668, 'learning_rate': 3.8336056160013814e-05, 'epoch': 1.57}

 16%|█▌        | 6470/41250 [15:38:06<83:57:27,  8.69s/it][2025-04-25 23:35:49,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-25 23:35:49,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.46 | bwd_microstep: 5707.37 | bwd_inner_microstep: 5694.41 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.53
[2025-04-25 23:35:49,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.46 | bwd: 5707.38 | bwd_inner: 5694.41 | bwd_allreduce: 12.93 | step: 19.53

 16%|█▌        | 6471/41250 [15:38:14<83:48:58,  8.68s/it]
                                                          
{'loss': 0.0773, 'grad_norm': 2.1984591484069824, 'learning_rate': 3.833542900947299e-05, 'epoch': 1.57}

 16%|█▌        | 6471/41250 [15:38:14<83:48:58,  8.68s/it][2025-04-25 23:35:58,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-25 23:35:58,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.64 | bwd_microstep: 5718.72 | bwd_inner_microstep: 5706.07 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.18
[2025-04-25 23:35:58,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.64 | bwd: 5718.73 | bwd_inner: 5706.07 | bwd_allreduce: 12.62 | step: 19.19

 16%|█▌        | 6472/41250 [15:38:23<83:45:28,  8.67s/it]
                                                          
{'loss': 0.1777, 'grad_norm': 1.7932586669921875, 'learning_rate': 3.833480174589785e-05, 'epoch': 1.57}

 16%|█▌        | 6472/41250 [15:38:23<83:45:28,  8.67s/it][2025-04-25 23:36:06,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:36:06,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5694.28 | bwd_inner_microstep: 5681.43 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.23
[2025-04-25 23:36:06,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5694.29 | bwd_inner: 5681.43 | bwd_allreduce: 12.82 | step: 19.23

 16%|█▌        | 6473/41250 [15:38:32<83:37:05,  8.66s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 1.3387833833694458, 'learning_rate': 3.833417436929227e-05, 'epoch': 1.57}

 16%|█▌        | 6473/41250 [15:38:32<83:37:05,  8.66s/it][2025-04-25 23:36:15,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-25 23:36:15,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5723.49 | bwd_inner_microstep: 5710.52 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.72
[2025-04-25 23:36:15,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5723.50 | bwd_inner: 5710.52 | bwd_allreduce: 12.94 | step: 18.72

 16%|█▌        | 6474/41250 [15:38:40<83:37:12,  8.66s/it]
                                                          
{'loss': 0.0275, 'grad_norm': 0.5260657668113708, 'learning_rate': 3.833354687966011e-05, 'epoch': 1.57}

 16%|█▌        | 6474/41250 [15:38:40<83:37:12,  8.66s/it][2025-04-25 23:36:24,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:36:24,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.40 | bwd_microstep: 5768.94 | bwd_inner_microstep: 5686.32 | bwd_allreduce_microstep: 82.57 | step_microstep: 18.34
[2025-04-25 23:36:24,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.40 | bwd: 5768.96 | bwd_inner: 5686.32 | bwd_allreduce: 82.58 | step: 18.34

 16%|█▌        | 6475/41250 [15:38:49<83:44:03,  8.67s/it]
                                                          
{'loss': 0.1014, 'grad_norm': 1.5252244472503662, 'learning_rate': 3.833291927700523e-05, 'epoch': 1.57}

 16%|█▌        | 6475/41250 [15:38:49<83:44:03,  8.67s/it][2025-04-25 23:36:32,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.25 | optimizer_step: 1.00
[2025-04-25 23:36:32,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.08 | bwd_microstep: 5758.71 | bwd_inner_microstep: 5696.27 | bwd_allreduce_microstep: 62.39 | step_microstep: 19.57
[2025-04-25 23:36:32,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.08 | bwd: 5758.72 | bwd_inner: 5696.27 | bwd_allreduce: 62.41 | step: 19.57

 16%|█▌        | 6476/41250 [15:38:58<83:47:49,  8.68s/it]
                                                          
{'loss': 0.2198, 'grad_norm': 2.4450085163116455, 'learning_rate': 3.833229156133152e-05, 'epoch': 1.57}

 16%|█▌        | 6476/41250 [15:38:58<83:47:49,  8.68s/it][2025-04-25 23:36:41,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-25 23:36:41,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5776.19 | bwd_inner_microstep: 5690.29 | bwd_allreduce_microstep: 85.85 | step_microstep: 19.10
[2025-04-25 23:36:41,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5776.21 | bwd_inner: 5690.30 | bwd_allreduce: 85.87 | step: 19.10

 16%|█▌        | 6477/41250 [15:39:06<83:54:12,  8.69s/it]
                                                          
{'loss': 0.1481, 'grad_norm': 2.0350778102874756, 'learning_rate': 3.833166373264283e-05, 'epoch': 1.57}

 16%|█▌        | 6477/41250 [15:39:06<83:54:12,  8.69s/it][2025-04-25 23:36:50,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-25 23:36:50,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.54 | bwd_microstep: 5779.47 | bwd_inner_microstep: 5654.03 | bwd_allreduce_microstep: 125.40 | step_microstep: 18.33
[2025-04-25 23:36:50,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.54 | bwd: 5779.49 | bwd_inner: 5654.03 | bwd_allreduce: 125.42 | step: 18.33

 16%|█▌        | 6478/41250 [15:39:15<83:55:21,  8.69s/it]
                                                          
{'loss': 0.1684, 'grad_norm': 1.4832831621170044, 'learning_rate': 3.833103579094305e-05, 'epoch': 1.57}

 16%|█▌        | 6478/41250 [15:39:15<83:55:21,  8.69s/it][2025-04-25 23:36:58,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:36:58,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.73 | bwd_microstep: 5762.12 | bwd_inner_microstep: 5663.67 | bwd_allreduce_microstep: 98.40 | step_microstep: 18.44
[2025-04-25 23:36:58,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.73 | bwd: 5762.13 | bwd_inner: 5663.67 | bwd_allreduce: 98.42 | step: 18.44

 16%|█▌        | 6479/41250 [15:39:24<83:54:35,  8.69s/it]
                                                          
{'loss': 0.1062, 'grad_norm': 1.6123765707015991, 'learning_rate': 3.833040773623603e-05, 'epoch': 1.57}

 16%|█▌        | 6479/41250 [15:39:24<83:54:35,  8.69s/it][2025-04-25 23:37:07,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-25 23:37:07,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5716.14 | bwd_inner_microstep: 5652.24 | bwd_allreduce_microstep: 63.85 | step_microstep: 18.54
[2025-04-25 23:37:07,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5716.15 | bwd_inner: 5652.24 | bwd_allreduce: 63.87 | step: 18.54

 16%|█▌        | 6480/41250 [15:39:32<83:43:48,  8.67s/it]
                                                          
{'loss': 0.1381, 'grad_norm': 2.6612236499786377, 'learning_rate': 3.832977956852566e-05, 'epoch': 1.57}

 16%|█▌        | 6480/41250 [15:39:32<83:43:48,  8.67s/it][2025-04-25 23:37:16,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-25 23:37:16,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.83 | bwd_microstep: 5749.82 | bwd_inner_microstep: 5654.08 | bwd_allreduce_microstep: 95.70 | step_microstep: 19.03
[2025-04-25 23:37:16,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.83 | bwd: 5749.84 | bwd_inner: 5654.08 | bwd_allreduce: 95.72 | step: 19.03

 16%|█▌        | 6481/41250 [15:39:41<83:44:53,  8.67s/it]
                                                          
{'loss': 0.1399, 'grad_norm': 3.7715234756469727, 'learning_rate': 3.83291512878158e-05, 'epoch': 1.57}

 16%|█▌        | 6481/41250 [15:39:41<83:44:53,  8.67s/it][2025-04-25 23:37:24,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:37:24,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.52 | bwd_microstep: 5768.73 | bwd_inner_microstep: 5701.06 | bwd_allreduce_microstep: 67.63 | step_microstep: 18.15
[2025-04-25 23:37:24,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.52 | bwd: 5768.75 | bwd_inner: 5701.06 | bwd_allreduce: 67.65 | step: 18.15

 16%|█▌        | 6482/41250 [15:39:50<83:49:40,  8.68s/it]
                                                          
{'loss': 0.086, 'grad_norm': 1.1478928327560425, 'learning_rate': 3.8328522894110324e-05, 'epoch': 1.57}

 16%|█▌        | 6482/41250 [15:39:50<83:49:40,  8.68s/it][2025-04-25 23:37:33,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:37:33,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5705.39 | bwd_inner_microstep: 5690.07 | bwd_allreduce_microstep: 15.27 | step_microstep: 18.57
[2025-04-25 23:37:33,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5705.40 | bwd_inner: 5690.07 | bwd_allreduce: 15.29 | step: 18.57

 16%|█▌        | 6483/41250 [15:39:58<83:41:48,  8.67s/it]
                                                          
{'loss': 0.0887, 'grad_norm': 1.0749666690826416, 'learning_rate': 3.832789438741311e-05, 'epoch': 1.57}

 16%|█▌        | 6483/41250 [15:39:58<83:41:48,  8.67s/it][2025-04-25 23:37:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:37:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.45 | bwd_microstep: 5782.07 | bwd_inner_microstep: 5658.53 | bwd_allreduce_microstep: 123.50 | step_microstep: 18.86
[2025-04-25 23:37:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.45 | bwd: 5782.08 | bwd_inner: 5658.53 | bwd_allreduce: 123.51 | step: 18.86

 16%|█▌        | 6484/41250 [15:40:07<83:46:31,  8.67s/it]
                                                          
{'loss': 0.1848, 'grad_norm': 1.7725355625152588, 'learning_rate': 3.832726576772803e-05, 'epoch': 1.57}

 16%|█▌        | 6484/41250 [15:40:07<83:46:31,  8.67s/it][2025-04-25 23:37:50,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 23:37:50,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.49 | bwd_microstep: 5783.10 | bwd_inner_microstep: 5654.13 | bwd_allreduce_microstep: 128.92 | step_microstep: 18.87
[2025-04-25 23:37:50,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.49 | bwd: 5783.11 | bwd_inner: 5654.13 | bwd_allreduce: 128.94 | step: 18.87

 16%|█▌        | 6485/41250 [15:40:16<83:50:29,  8.68s/it]
                                                          
{'loss': 0.1435, 'grad_norm': 2.2905261516571045, 'learning_rate': 3.8326637035058966e-05, 'epoch': 1.57}

 16%|█▌        | 6485/41250 [15:40:16<83:50:29,  8.68s/it][2025-04-25 23:37:59,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:37:59,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.21 | bwd_microstep: 5773.72 | bwd_inner_microstep: 5653.70 | bwd_allreduce_microstep: 119.98 | step_microstep: 18.37
[2025-04-25 23:37:59,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.21 | bwd: 5773.73 | bwd_inner: 5653.70 | bwd_allreduce: 120.00 | step: 18.38

 16%|█▌        | 6486/41250 [15:40:24<83:51:17,  8.68s/it]
                                                          
{'loss': 0.0302, 'grad_norm': 0.3211205303668976, 'learning_rate': 3.832600818940979e-05, 'epoch': 1.57}

 16%|█▌        | 6486/41250 [15:40:24<83:51:17,  8.68s/it][2025-04-25 23:38:08,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:38:08,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5714.16 | bwd_inner_microstep: 5663.80 | bwd_allreduce_microstep: 50.31 | step_microstep: 18.49
[2025-04-25 23:38:08,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.96 | bwd: 5714.17 | bwd_inner: 5663.80 | bwd_allreduce: 50.33 | step: 18.49

 16%|█▌        | 6487/41250 [15:40:33<83:42:18,  8.67s/it]
                                                          
{'loss': 0.2254, 'grad_norm': 3.394179105758667, 'learning_rate': 3.832537923078437e-05, 'epoch': 1.57}

 16%|█▌        | 6487/41250 [15:40:33<83:42:18,  8.67s/it][2025-04-25 23:38:16,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-25 23:38:16,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.58 | bwd_microstep: 5751.90 | bwd_inner_microstep: 5687.73 | bwd_allreduce_microstep: 64.12 | step_microstep: 19.01
[2025-04-25 23:38:16,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.58 | bwd: 5751.91 | bwd_inner: 5687.73 | bwd_allreduce: 64.14 | step: 19.02

 16%|█▌        | 6488/41250 [15:40:42<83:46:09,  8.68s/it]
                                                          
{'loss': 0.1076, 'grad_norm': 1.2411681413650513, 'learning_rate': 3.83247501591866e-05, 'epoch': 1.57}

 16%|█▌        | 6488/41250 [15:40:42<83:46:09,  8.68s/it][2025-04-25 23:38:25,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 23:38:25,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.94 | bwd_microstep: 5701.74 | bwd_inner_microstep: 5688.87 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.02
[2025-04-25 23:38:25,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.94 | bwd: 5701.75 | bwd_inner: 5688.87 | bwd_allreduce: 12.85 | step: 19.02

 16%|█▌        | 6489/41250 [15:40:50<83:40:32,  8.67s/it]
                                                          
{'loss': 0.4069, 'grad_norm': 2.3714382648468018, 'learning_rate': 3.832412097462035e-05, 'epoch': 1.57}

 16%|█▌        | 6489/41250 [15:40:50<83:40:32,  8.67s/it][2025-04-25 23:38:34,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:38:34,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.26 | bwd_microstep: 5773.48 | bwd_inner_microstep: 5658.02 | bwd_allreduce_microstep: 115.41 | step_microstep: 18.62
[2025-04-25 23:38:34,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.26 | bwd: 5773.50 | bwd_inner: 5658.02 | bwd_allreduce: 115.43 | step: 18.62

 16%|█▌        | 6490/41250 [15:40:59<83:45:15,  8.67s/it]
                                                          
{'loss': 0.0814, 'grad_norm': 0.654781699180603, 'learning_rate': 3.83234916770895e-05, 'epoch': 1.57}

 16%|█▌        | 6490/41250 [15:40:59<83:45:15,  8.67s/it][2025-04-25 23:38:42,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-25 23:38:42,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.50 | bwd_microstep: 5709.82 | bwd_inner_microstep: 5696.96 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.09
[2025-04-25 23:38:42,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.50 | bwd: 5709.83 | bwd_inner: 5696.96 | bwd_allreduce: 12.83 | step: 19.09

 16%|█▌        | 6491/41250 [15:41:08<83:39:11,  8.66s/it]
                                                          
{'loss': 0.1783, 'grad_norm': 2.9367690086364746, 'learning_rate': 3.8322862266597916e-05, 'epoch': 1.57}

 16%|█▌        | 6491/41250 [15:41:08<83:39:11,  8.66s/it][2025-04-25 23:38:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:38:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.64 | bwd_microstep: 5707.32 | bwd_inner_microstep: 5645.89 | bwd_allreduce_microstep: 61.38 | step_microstep: 18.42
[2025-04-25 23:38:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5707.33 | bwd_inner: 5645.89 | bwd_allreduce: 61.40 | step: 18.42

 16%|█▌        | 6492/41250 [15:41:16<83:31:10,  8.65s/it]
                                                          
{'loss': 0.0761, 'grad_norm': 1.3143948316574097, 'learning_rate': 3.83222327431495e-05, 'epoch': 1.57}

 16%|█▌        | 6492/41250 [15:41:16<83:31:10,  8.65s/it][2025-04-25 23:39:00,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 23:39:00,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.89 | bwd_microstep: 5718.57 | bwd_inner_microstep: 5704.81 | bwd_allreduce_microstep: 13.71 | step_microstep: 19.06
[2025-04-25 23:39:00,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.89 | bwd: 5718.58 | bwd_inner: 5704.81 | bwd_allreduce: 13.73 | step: 19.06

 16%|█▌        | 6493/41250 [15:41:25<83:33:21,  8.65s/it]
                                                          
{'loss': 0.0698, 'grad_norm': 0.9367244243621826, 'learning_rate': 3.832160310674812e-05, 'epoch': 1.57}

 16%|█▌        | 6493/41250 [15:41:25<83:33:21,  8.65s/it][2025-04-25 23:39:08,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:39:08,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.99 | bwd_microstep: 5723.39 | bwd_inner_microstep: 5640.31 | bwd_allreduce_microstep: 83.03 | step_microstep: 18.40
[2025-04-25 23:39:08,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.99 | bwd: 5723.40 | bwd_inner: 5640.31 | bwd_allreduce: 83.05 | step: 18.41

 16%|█▌        | 6494/41250 [15:41:34<83:28:30,  8.65s/it]
                                                          
{'loss': 0.0219, 'grad_norm': 0.7791122794151306, 'learning_rate': 3.832097335739766e-05, 'epoch': 1.57}

 16%|█▌        | 6494/41250 [15:41:34<83:28:30,  8.65s/it][2025-04-25 23:39:17,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:39:17,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.25 | bwd_microstep: 5687.97 | bwd_inner_microstep: 5645.94 | bwd_allreduce_microstep: 41.99 | step_microstep: 18.60
[2025-04-25 23:39:17,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.25 | bwd: 5687.99 | bwd_inner: 5645.94 | bwd_allreduce: 42.00 | step: 18.60

 16%|█▌        | 6495/41250 [15:41:42<83:19:42,  8.63s/it]
                                                          
{'loss': 0.085, 'grad_norm': 1.9460126161575317, 'learning_rate': 3.8320343495102004e-05, 'epoch': 1.57}

 16%|█▌        | 6495/41250 [15:41:42<83:19:42,  8.63s/it][2025-04-25 23:39:26,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:39:26,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5726.50 | bwd_inner_microstep: 5707.46 | bwd_allreduce_microstep: 19.00 | step_microstep: 18.21
[2025-04-25 23:39:26,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5726.52 | bwd_inner: 5707.46 | bwd_allreduce: 19.02 | step: 18.21

 16%|█▌        | 6496/41250 [15:41:51<83:25:17,  8.64s/it]
                                                          
{'loss': 0.2199, 'grad_norm': 1.9210036993026733, 'learning_rate': 3.8319713519865035e-05, 'epoch': 1.57}

 16%|█▌        | 6496/41250 [15:41:51<83:25:17,  8.64s/it][2025-04-25 23:39:34,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-25 23:39:34,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.35 | bwd_microstep: 5699.48 | bwd_inner_microstep: 5686.69 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.17
[2025-04-25 23:39:34,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.35 | bwd: 5699.50 | bwd_inner: 5686.69 | bwd_allreduce: 12.76 | step: 19.17

 16%|█▌        | 6497/41250 [15:42:00<83:22:28,  8.64s/it]
                                                          
{'loss': 0.0556, 'grad_norm': 0.9591057896614075, 'learning_rate': 3.831908343169064e-05, 'epoch': 1.58}

 16%|█▌        | 6497/41250 [15:42:00<83:22:28,  8.64s/it][2025-04-25 23:39:43,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:39:43,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5728.51 | bwd_inner_microstep: 5701.22 | bwd_allreduce_microstep: 27.25 | step_microstep: 18.29
[2025-04-25 23:39:43,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5728.52 | bwd_inner: 5701.22 | bwd_allreduce: 27.27 | step: 18.29

 16%|█▌        | 6498/41250 [15:42:08<83:26:42,  8.64s/it]
                                                          
{'loss': 0.3252, 'grad_norm': 3.0559399127960205, 'learning_rate': 3.831845323058269e-05, 'epoch': 1.58}

 16%|█▌        | 6498/41250 [15:42:08<83:26:42,  8.64s/it][2025-04-25 23:39:52,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-25 23:39:52,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.93 | bwd_microstep: 5718.92 | bwd_inner_microstep: 5644.94 | bwd_allreduce_microstep: 73.93 | step_microstep: 18.59
[2025-04-25 23:39:52,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.93 | bwd: 5718.93 | bwd_inner: 5644.94 | bwd_allreduce: 73.95 | step: 18.59

 16%|█▌        | 6499/41250 [15:42:17<83:23:20,  8.64s/it]
                                                          
{'loss': 0.1894, 'grad_norm': 3.4009459018707275, 'learning_rate': 3.831782291654507e-05, 'epoch': 1.58}

 16%|█▌        | 6499/41250 [15:42:17<83:23:20,  8.64s/it][2025-04-25 23:40:00,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 23:40:00,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.07 | bwd_microstep: 5746.86 | bwd_inner_microstep: 5647.74 | bwd_allreduce_microstep: 99.08 | step_microstep: 18.51
[2025-04-25 23:40:00,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.08 | bwd: 5746.88 | bwd_inner: 5647.74 | bwd_allreduce: 99.10 | step: 18.51

 16%|█▌        | 6500/41250 [15:42:26<83:25:34,  8.64s/it]
                                                          
{'loss': 0.0542, 'grad_norm': 0.9626426696777344, 'learning_rate': 3.8317192489581694e-05, 'epoch': 1.58}

 16%|█▌        | 6500/41250 [15:42:26<83:25:34,  8.64s/it][2025-04-25 23:40:09,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:40:09,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.04 | bwd_microstep: 5783.63 | bwd_inner_microstep: 5770.83 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.66
[2025-04-25 23:40:09,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.04 | bwd: 5783.65 | bwd_inner: 5770.83 | bwd_allreduce: 12.78 | step: 18.67

 16%|█▌        | 6501/41250 [15:42:34<83:44:07,  8.67s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.349880576133728, 'learning_rate': 3.831656194969642e-05, 'epoch': 1.58}

 16%|█▌        | 6501/41250 [15:42:34<83:44:07,  8.67s/it][2025-04-25 23:40:18,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:40:18,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5714.01 | bwd_inner_microstep: 5695.76 | bwd_allreduce_microstep: 18.21 | step_microstep: 18.70
[2025-04-25 23:40:18,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5714.02 | bwd_inner: 5695.75 | bwd_allreduce: 18.23 | step: 18.70

 16%|█▌        | 6502/41250 [15:42:43<83:38:45,  8.67s/it]
                                                          
{'loss': 0.1613, 'grad_norm': 2.8333473205566406, 'learning_rate': 3.831593129689315e-05, 'epoch': 1.58}

 16%|█▌        | 6502/41250 [15:42:43<83:38:45,  8.67s/it][2025-04-25 23:40:26,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:40:26,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.06 | bwd_microstep: 5756.28 | bwd_inner_microstep: 5648.53 | bwd_allreduce_microstep: 107.71 | step_microstep: 18.41
[2025-04-25 23:40:26,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.06 | bwd: 5756.29 | bwd_inner: 5648.53 | bwd_allreduce: 107.72 | step: 18.41

 16%|█▌        | 6503/41250 [15:42:52<83:39:20,  8.67s/it]
                                                          
{'loss': 0.167, 'grad_norm': 3.102912425994873, 'learning_rate': 3.831530053117576e-05, 'epoch': 1.58}

 16%|█▌        | 6503/41250 [15:42:52<83:39:20,  8.67s/it][2025-04-25 23:40:35,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:40:35,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.89 | bwd_microstep: 5697.24 | bwd_inner_microstep: 5684.66 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.68
[2025-04-25 23:40:35,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.89 | bwd: 5697.25 | bwd_inner: 5684.66 | bwd_allreduce: 12.54 | step: 18.68

 16%|█▌        | 6504/41250 [15:43:00<83:31:51,  8.65s/it]
                                                          
{'loss': 0.2158, 'grad_norm': 4.602999210357666, 'learning_rate': 3.831466965254814e-05, 'epoch': 1.58}

 16%|█▌        | 6504/41250 [15:43:00<83:31:51,  8.65s/it][2025-04-25 23:40:43,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-25 23:40:43,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.53 | bwd_microstep: 5699.35 | bwd_inner_microstep: 5643.42 | bwd_allreduce_microstep: 55.88 | step_microstep: 18.74
[2025-04-25 23:40:43,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.53 | bwd: 5699.36 | bwd_inner: 5643.42 | bwd_allreduce: 55.90 | step: 18.74

 16%|█▌        | 6505/41250 [15:43:09<83:23:20,  8.64s/it]
                                                          
{'loss': 0.1612, 'grad_norm': 4.4208574295043945, 'learning_rate': 3.8314038661014195e-05, 'epoch': 1.58}

 16%|█▌        | 6505/41250 [15:43:09<83:23:20,  8.64s/it][2025-04-25 23:40:52,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-25 23:40:52,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.75 | bwd_microstep: 5714.42 | bwd_inner_microstep: 5648.59 | bwd_allreduce_microstep: 65.78 | step_microstep: 18.79
[2025-04-25 23:40:52,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.75 | bwd: 5714.43 | bwd_inner: 5648.59 | bwd_allreduce: 65.79 | step: 18.79

 16%|█▌        | 6506/41250 [15:43:17<83:20:44,  8.64s/it]
                                                          
{'loss': 0.0589, 'grad_norm': 1.122819423675537, 'learning_rate': 3.8313407556577805e-05, 'epoch': 1.58}

 16%|█▌        | 6506/41250 [15:43:17<83:20:44,  8.64s/it][2025-04-25 23:41:01,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-25 23:41:01,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.01 | bwd_microstep: 5710.21 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.67
[2025-04-25 23:41:01,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.01 | bwd: 5710.23 | bwd_inner: 5697.65 | bwd_allreduce: 12.53 | step: 18.68

 16%|█▌        | 6507/41250 [15:43:26<83:23:54,  8.64s/it]
                                                          
{'loss': 0.1608, 'grad_norm': 2.291126251220703, 'learning_rate': 3.831277633924285e-05, 'epoch': 1.58}

 16%|█▌        | 6507/41250 [15:43:26<83:23:54,  8.64s/it][2025-04-25 23:41:09,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-25 23:41:09,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5681.72 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 24.08 | step_microstep: 18.96
[2025-04-25 23:41:09,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5681.73 | bwd_inner: 5657.59 | bwd_allreduce: 24.10 | step: 18.96

 16%|█▌        | 6508/41250 [15:43:35<83:19:59,  8.64s/it]
                                                          
{'loss': 0.2586, 'grad_norm': 2.165029764175415, 'learning_rate': 3.8312145009013235e-05, 'epoch': 1.58}

 16%|█▌        | 6508/41250 [15:43:35<83:19:59,  8.64s/it][2025-04-25 23:41:18,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-25 23:41:18,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.38 | bwd_microstep: 5768.30 | bwd_inner_microstep: 5642.04 | bwd_allreduce_microstep: 126.22 | step_microstep: 18.73
[2025-04-25 23:41:18,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.38 | bwd: 5768.32 | bwd_inner: 5642.04 | bwd_allreduce: 126.24 | step: 18.73

 16%|█▌        | 6509/41250 [15:43:43<83:27:44,  8.65s/it]
                                                          
{'loss': 0.0179, 'grad_norm': 0.3823154866695404, 'learning_rate': 3.831151356589284e-05, 'epoch': 1.58}

 16%|█▌        | 6509/41250 [15:43:43<83:27:44,  8.65s/it][2025-04-25 23:41:27,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-25 23:41:27,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.48 | bwd_microstep: 5711.51 | bwd_inner_microstep: 5698.87 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.13
[2025-04-25 23:41:27,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.48 | bwd: 5711.53 | bwd_inner: 5698.87 | bwd_allreduce: 12.61 | step: 19.13

 16%|█▌        | 6510/41250 [15:43:52<83:28:39,  8.65s/it]
                                                          
{'loss': 0.0894, 'grad_norm': 1.1152276992797852, 'learning_rate': 3.8310882009885576e-05, 'epoch': 1.58}

 16%|█▌        | 6510/41250 [15:43:52<83:28:39,  8.65s/it][2025-04-25 23:41:35,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 23:41:35,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.26 | bwd_microstep: 5753.34 | bwd_inner_microstep: 5686.67 | bwd_allreduce_microstep: 66.63 | step_microstep: 18.73
[2025-04-25 23:41:35,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.26 | bwd: 5753.35 | bwd_inner: 5686.67 | bwd_allreduce: 66.65 | step: 18.73

 16%|█▌        | 6511/41250 [15:44:01<83:36:07,  8.66s/it]
                                                          
{'loss': 0.0643, 'grad_norm': 1.0973607301712036, 'learning_rate': 3.831025034099532e-05, 'epoch': 1.58}

 16%|█▌        | 6511/41250 [15:44:01<83:36:07,  8.66s/it][2025-04-25 23:41:44,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-25 23:41:44,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.77 | bwd_microstep: 5768.01 | bwd_inner_microstep: 5675.29 | bwd_allreduce_microstep: 92.67 | step_microstep: 18.52
[2025-04-25 23:41:44,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.77 | bwd: 5768.02 | bwd_inner: 5675.29 | bwd_allreduce: 92.69 | step: 18.52

 16%|█▌        | 6512/41250 [15:44:09<83:41:55,  8.67s/it]
                                                          
{'loss': 0.2379, 'grad_norm': 3.8625757694244385, 'learning_rate': 3.830961855922598e-05, 'epoch': 1.58}

 16%|█▌        | 6512/41250 [15:44:09<83:41:55,  8.67s/it][2025-04-25 23:41:53,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:41:53,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.16 | bwd_microstep: 5876.91 | bwd_inner_microstep: 5692.95 | bwd_allreduce_microstep: 183.92 | step_microstep: 18.90
[2025-04-25 23:41:53,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.16 | bwd: 5876.93 | bwd_inner: 5692.95 | bwd_allreduce: 183.94 | step: 18.90

 16%|█▌        | 6513/41250 [15:44:18<84:04:24,  8.71s/it]
                                                          
{'loss': 0.0645, 'grad_norm': 0.6381708979606628, 'learning_rate': 3.830898666458144e-05, 'epoch': 1.58}

 16%|█▌        | 6513/41250 [15:44:18<84:04:24,  8.71s/it][2025-04-25 23:42:02,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:42:02,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.36 | bwd_microstep: 5837.39 | bwd_inner_microstep: 5694.24 | bwd_allreduce_microstep: 143.09 | step_microstep: 18.59
[2025-04-25 23:42:02,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.37 | bwd: 5837.41 | bwd_inner: 5694.24 | bwd_allreduce: 143.11 | step: 18.59

 16%|█▌        | 6514/41250 [15:44:27<84:14:33,  8.73s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.73996639251709, 'learning_rate': 3.83083546570656e-05, 'epoch': 1.58}

 16%|█▌        | 6514/41250 [15:44:27<84:14:33,  8.73s/it][2025-04-25 23:42:10,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 1.01
[2025-04-25 23:42:10,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.61 | bwd_microstep: 5724.38 | bwd_inner_microstep: 5641.31 | bwd_allreduce_microstep: 83.02 | step_microstep: 19.51
[2025-04-25 23:42:10,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.61 | bwd: 5724.39 | bwd_inner: 5641.31 | bwd_allreduce: 83.04 | step: 19.52

 16%|█▌        | 6515/41250 [15:44:36<83:56:04,  8.70s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.723901391029358, 'learning_rate': 3.8307722536682345e-05, 'epoch': 1.58}

 16%|█▌        | 6515/41250 [15:44:36<83:56:04,  8.70s/it][2025-04-25 23:42:19,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 23:42:19,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.86 | bwd_microstep: 5700.49 | bwd_inner_microstep: 5687.73 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.14
[2025-04-25 23:42:19,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.86 | bwd: 5700.51 | bwd_inner: 5687.73 | bwd_allreduce: 12.74 | step: 19.15

 16%|█▌        | 6516/41250 [15:44:44<83:44:23,  8.68s/it]
                                                          
{'loss': 0.0461, 'grad_norm': 1.9139963388442993, 'learning_rate': 3.8307090303435585e-05, 'epoch': 1.58}

 16%|█▌        | 6516/41250 [15:44:44<83:44:23,  8.68s/it][2025-04-25 23:42:28,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:42:28,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.93 | bwd_microstep: 5762.07 | bwd_inner_microstep: 5679.62 | bwd_allreduce_microstep: 82.40 | step_microstep: 18.76
[2025-04-25 23:42:28,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.93 | bwd: 5762.08 | bwd_inner: 5679.62 | bwd_allreduce: 82.42 | step: 18.77

 16%|█▌        | 6517/41250 [15:44:53<83:48:03,  8.69s/it]
                                                          
{'loss': 0.0885, 'grad_norm': 1.8288445472717285, 'learning_rate': 3.830645795732921e-05, 'epoch': 1.58}

 16%|█▌        | 6517/41250 [15:44:53<83:48:03,  8.69s/it][2025-04-25 23:42:36,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:42:36,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5714.87 | bwd_inner_microstep: 5660.66 | bwd_allreduce_microstep: 54.16 | step_microstep: 18.54
[2025-04-25 23:42:36,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5714.88 | bwd_inner: 5660.66 | bwd_allreduce: 54.18 | step: 18.54

 16%|█▌        | 6518/41250 [15:45:02<83:37:28,  8.67s/it]
                                                          
{'loss': 0.2626, 'grad_norm': 1.6486464738845825, 'learning_rate': 3.830582549836713e-05, 'epoch': 1.58}

 16%|█▌        | 6518/41250 [15:45:02<83:37:28,  8.67s/it][2025-04-25 23:42:45,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:42:45,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.39 | bwd_microstep: 5705.96 | bwd_inner_microstep: 5657.49 | bwd_allreduce_microstep: 48.44 | step_microstep: 18.68
[2025-04-25 23:42:45,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.39 | bwd: 5705.98 | bwd_inner: 5657.49 | bwd_allreduce: 48.45 | step: 18.68

 16%|█▌        | 6519/41250 [15:45:10<83:29:21,  8.65s/it]
                                                          
{'loss': 0.1256, 'grad_norm': 2.7215867042541504, 'learning_rate': 3.830519292655324e-05, 'epoch': 1.58}

 16%|█▌        | 6519/41250 [15:45:10<83:29:21,  8.65s/it][2025-04-25 23:42:54,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:42:54,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.56 | bwd_microstep: 5737.30 | bwd_inner_microstep: 5686.22 | bwd_allreduce_microstep: 51.04 | step_microstep: 18.59
[2025-04-25 23:42:54,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5737.32 | bwd_inner: 5686.22 | bwd_allreduce: 51.05 | step: 18.59

 16%|█▌        | 6520/41250 [15:45:19<83:32:21,  8.66s/it]
                                                          
{'loss': 0.0225, 'grad_norm': 0.46173083782196045, 'learning_rate': 3.830456024189143e-05, 'epoch': 1.58}

 16%|█▌        | 6520/41250 [15:45:19<83:32:21,  8.66s/it][2025-04-25 23:43:02,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-25 23:43:02,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5704.05 | bwd_inner_microstep: 5662.31 | bwd_allreduce_microstep: 41.70 | step_microstep: 18.84
[2025-04-25 23:43:02,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5704.06 | bwd_inner: 5662.31 | bwd_allreduce: 41.71 | step: 18.84

 16%|█▌        | 6521/41250 [15:45:28<83:25:01,  8.65s/it]
                                                          
{'loss': 0.2116, 'grad_norm': 3.196582078933716, 'learning_rate': 3.830392744438561e-05, 'epoch': 1.58}

 16%|█▌        | 6521/41250 [15:45:28<83:25:01,  8.65s/it][2025-04-25 23:43:11,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-25 23:43:11,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.39 | bwd_microstep: 5881.58 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 233.08 | step_microstep: 18.35
[2025-04-25 23:43:11,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.39 | bwd: 5881.59 | bwd_inner: 5648.45 | bwd_allreduce: 233.10 | step: 18.35

 16%|█▌        | 6522/41250 [15:45:36<83:53:37,  8.70s/it]
                                                          
{'loss': 0.1185, 'grad_norm': 1.7298883199691772, 'learning_rate': 3.830329453403967e-05, 'epoch': 1.58}

 16%|█▌        | 6522/41250 [15:45:36<83:53:37,  8.70s/it][2025-04-25 23:43:20,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:43:20,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.75 | bwd_microstep: 5720.46 | bwd_inner_microstep: 5708.06 | bwd_allreduce_microstep: 12.36 | step_microstep: 18.52
[2025-04-25 23:43:20,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.75 | bwd: 5720.47 | bwd_inner: 5708.06 | bwd_allreduce: 12.37 | step: 18.52

 16%|█▌        | 6523/41250 [15:45:45<83:46:50,  8.69s/it]
                                                          
{'loss': 0.1046, 'grad_norm': 2.0734565258026123, 'learning_rate': 3.8302661510857526e-05, 'epoch': 1.58}

 16%|█▌        | 6523/41250 [15:45:45<83:46:50,  8.69s/it][2025-04-25 23:43:28,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-25 23:43:28,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.76 | bwd_microstep: 5710.25 | bwd_inner_microstep: 5652.23 | bwd_allreduce_microstep: 57.98 | step_microstep: 18.48
[2025-04-25 23:43:28,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.76 | bwd: 5710.27 | bwd_inner: 5652.23 | bwd_allreduce: 58.00 | step: 18.48

 16%|█▌        | 6524/41250 [15:45:54<83:37:56,  8.67s/it]
                                                          
{'loss': 0.064, 'grad_norm': 0.587409257888794, 'learning_rate': 3.830202837484307e-05, 'epoch': 1.58}

 16%|█▌        | 6524/41250 [15:45:54<83:37:56,  8.67s/it][2025-04-25 23:43:37,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:43:37,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.73 | bwd_microstep: 5856.83 | bwd_inner_microstep: 5691.57 | bwd_allreduce_microstep: 165.22 | step_microstep: 18.49
[2025-04-25 23:43:37,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.73 | bwd: 5856.84 | bwd_inner: 5691.57 | bwd_allreduce: 165.23 | step: 18.49

 16%|█▌        | 6525/41250 [15:46:02<84:00:24,  8.71s/it]
                                                          
{'loss': 0.0351, 'grad_norm': 0.4824138581752777, 'learning_rate': 3.8301395126000215e-05, 'epoch': 1.58}

 16%|█▌        | 6525/41250 [15:46:02<84:00:24,  8.71s/it][2025-04-25 23:43:46,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-25 23:43:46,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.16 | bwd_microstep: 5703.23 | bwd_inner_microstep: 5664.12 | bwd_allreduce_microstep: 39.07 | step_microstep: 18.93
[2025-04-25 23:43:46,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.16 | bwd: 5703.25 | bwd_inner: 5664.11 | bwd_allreduce: 39.09 | step: 18.93

 16%|█▌        | 6526/41250 [15:46:11<83:46:51,  8.69s/it]
                                                          
{'loss': 0.097, 'grad_norm': 1.4090828895568848, 'learning_rate': 3.830076176433285e-05, 'epoch': 1.58}

 16%|█▌        | 6526/41250 [15:46:11<83:46:51,  8.69s/it][2025-04-25 23:43:54,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:43:54,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.38 | bwd_microstep: 5786.67 | bwd_inner_microstep: 5657.71 | bwd_allreduce_microstep: 128.91 | step_microstep: 18.54
[2025-04-25 23:43:54,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.38 | bwd: 5786.68 | bwd_inner: 5657.71 | bwd_allreduce: 128.93 | step: 18.54

 16%|█▌        | 6527/41250 [15:46:20<83:50:33,  8.69s/it]
                                                          
{'loss': 0.2529, 'grad_norm': 1.7448354959487915, 'learning_rate': 3.8300128289844896e-05, 'epoch': 1.58}

 16%|█▌        | 6527/41250 [15:46:20<83:50:33,  8.69s/it][2025-04-25 23:44:03,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-25 23:44:03,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.15 | bwd_microstep: 5807.11 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 149.47 | step_microstep: 19.05
[2025-04-25 23:44:03,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.15 | bwd: 5807.12 | bwd_inner: 5657.59 | bwd_allreduce: 149.49 | step: 19.05

 16%|█▌        | 6528/41250 [15:46:28<83:55:59,  8.70s/it]
                                                          
{'loss': 0.3842, 'grad_norm': 4.340869426727295, 'learning_rate': 3.8299494702540245e-05, 'epoch': 1.58}

 16%|█▌        | 6528/41250 [15:46:28<83:55:59,  8.70s/it][2025-04-25 23:44:12,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:44:12,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.26 | bwd_microstep: 5763.45 | bwd_inner_microstep: 5668.77 | bwd_allreduce_microstep: 94.63 | step_microstep: 18.38
[2025-04-25 23:44:12,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.26 | bwd: 5763.46 | bwd_inner: 5668.77 | bwd_allreduce: 94.65 | step: 18.38

 16%|█▌        | 6529/41250 [15:46:37<83:52:25,  8.70s/it]
                                                          
{'loss': 0.2721, 'grad_norm': 5.689908504486084, 'learning_rate': 3.829886100242281e-05, 'epoch': 1.58}

 16%|█▌        | 6529/41250 [15:46:37<83:52:25,  8.70s/it][2025-04-25 23:44:21,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 23:44:21,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.14 | bwd_microstep: 5780.17 | bwd_inner_microstep: 5646.68 | bwd_allreduce_microstep: 133.45 | step_microstep: 19.35
[2025-04-25 23:44:21,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.14 | bwd: 5780.19 | bwd_inner: 5646.68 | bwd_allreduce: 133.47 | step: 19.35

 16%|█▌        | 6530/41250 [15:46:46<83:53:22,  8.70s/it]
                                                          
{'loss': 0.1831, 'grad_norm': 1.9161490201950073, 'learning_rate': 3.82982271894965e-05, 'epoch': 1.58}

 16%|█▌        | 6530/41250 [15:46:46<83:53:22,  8.70s/it][2025-04-25 23:44:29,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:44:29,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5749.22 | bwd_inner_microstep: 5714.73 | bwd_allreduce_microstep: 34.45 | step_microstep: 18.96
[2025-04-25 23:44:29,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5749.23 | bwd_inner: 5714.73 | bwd_allreduce: 34.46 | step: 18.96

 16%|█▌        | 6531/41250 [15:46:55<83:51:51,  8.70s/it]
                                                          
{'loss': 0.0857, 'grad_norm': 0.8916834592819214, 'learning_rate': 3.829759326376522e-05, 'epoch': 1.58}

 16%|█▌        | 6531/41250 [15:46:55<83:51:51,  8.70s/it][2025-04-25 23:44:38,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 23:44:38,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3071.81 | bwd_microstep: 5662.93 | bwd_inner_microstep: 5650.16 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.92
[2025-04-25 23:44:38,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3071.81 | bwd: 5662.94 | bwd_inner: 5650.16 | bwd_allreduce: 12.74 | step: 18.93

 16%|█▌        | 6532/41250 [15:47:03<84:12:52,  8.73s/it]
                                                          
{'loss': 0.1262, 'grad_norm': 1.078928828239441, 'learning_rate': 3.829695922523287e-05, 'epoch': 1.58}

 16%|█▌        | 6532/41250 [15:47:03<84:12:52,  8.73s/it][2025-04-25 23:44:47,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-25 23:44:47,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.50 | bwd_microstep: 5784.78 | bwd_inner_microstep: 5650.31 | bwd_allreduce_microstep: 134.42 | step_microstep: 18.79
[2025-04-25 23:44:47,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.50 | bwd: 5784.79 | bwd_inner: 5650.31 | bwd_allreduce: 134.44 | step: 18.79

 16%|█▌        | 6533/41250 [15:47:12<84:06:45,  8.72s/it]
                                                          
{'loss': 0.1797, 'grad_norm': 7.468048095703125, 'learning_rate': 3.8296325073903363e-05, 'epoch': 1.58}

 16%|█▌        | 6533/41250 [15:47:12<84:06:45,  8.72s/it][2025-04-25 23:44:55,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-25 23:44:55,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.70 | bwd_microstep: 5698.72 | bwd_inner_microstep: 5653.95 | bwd_allreduce_microstep: 44.73 | step_microstep: 19.32
[2025-04-25 23:44:55,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.70 | bwd: 5698.73 | bwd_inner: 5653.95 | bwd_allreduce: 44.74 | step: 19.33

 16%|█▌        | 6534/41250 [15:47:21<83:47:24,  8.69s/it]
                                                          
{'loss': 0.2227, 'grad_norm': 4.034696578979492, 'learning_rate': 3.8295690809780614e-05, 'epoch': 1.58}

 16%|█▌        | 6534/41250 [15:47:21<83:47:24,  8.69s/it][2025-04-25 23:45:04,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 23:45:04,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.38 | bwd_microstep: 5790.44 | bwd_inner_microstep: 5638.16 | bwd_allreduce_microstep: 152.23 | step_microstep: 18.90
[2025-04-25 23:45:04,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.38 | bwd: 5790.45 | bwd_inner: 5638.16 | bwd_allreduce: 152.26 | step: 18.91

 16%|█▌        | 6535/41250 [15:47:29<83:47:58,  8.69s/it]
                                                          
{'loss': 0.1507, 'grad_norm': 4.256382942199707, 'learning_rate': 3.829505643286853e-05, 'epoch': 1.58}

 16%|█▌        | 6535/41250 [15:47:29<83:47:58,  8.69s/it][2025-04-25 23:45:13,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-25 23:45:13,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.13 | bwd_microstep: 5850.51 | bwd_inner_microstep: 5698.56 | bwd_allreduce_microstep: 151.91 | step_microstep: 18.44
[2025-04-25 23:45:13,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.13 | bwd: 5850.52 | bwd_inner: 5698.56 | bwd_allreduce: 151.92 | step: 18.44

 16%|█▌        | 6536/41250 [15:47:38<84:06:03,  8.72s/it]
                                                          
{'loss': 0.0792, 'grad_norm': 1.281614899635315, 'learning_rate': 3.8294421943171025e-05, 'epoch': 1.58}

 16%|█▌        | 6536/41250 [15:47:38<84:06:03,  8.72s/it][2025-04-25 23:45:22,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-25 23:45:22,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.17 | bwd_microstep: 5791.92 | bwd_inner_microstep: 5779.21 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.59
[2025-04-25 23:45:22,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.17 | bwd: 5791.93 | bwd_inner: 5779.21 | bwd_allreduce: 12.68 | step: 18.59

 16%|█▌        | 6537/41250 [15:47:47<84:13:53,  8.74s/it]
                                                          
{'loss': 0.1283, 'grad_norm': 2.138418674468994, 'learning_rate': 3.829378734069201e-05, 'epoch': 1.58}

 16%|█▌        | 6537/41250 [15:47:47<84:13:53,  8.74s/it][2025-04-25 23:45:30,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 1.09
[2025-04-25 23:45:30,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5729.47 | bwd_inner_microstep: 5688.49 | bwd_allreduce_microstep: 40.93 | step_microstep: 18.65
[2025-04-25 23:45:30,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5729.48 | bwd_inner: 5688.49 | bwd_allreduce: 40.95 | step: 18.65

 16%|█▌        | 6538/41250 [15:47:56<84:00:03,  8.71s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.6459226608276367, 'learning_rate': 3.829315262543538e-05, 'epoch': 1.58}

 16%|█▌        | 6538/41250 [15:47:56<84:00:03,  8.71s/it][2025-04-25 23:45:39,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-25 23:45:39,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5696.72 | bwd_inner_microstep: 5683.84 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.91
[2025-04-25 23:45:39,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5696.74 | bwd_inner: 5683.84 | bwd_allreduce: 12.85 | step: 18.91

 16%|█▌        | 6539/41250 [15:48:04<83:46:41,  8.69s/it]
                                                          
{'loss': 0.049, 'grad_norm': 1.0101432800292969, 'learning_rate': 3.8292517797405076e-05, 'epoch': 1.59}

 16%|█▌        | 6539/41250 [15:48:04<83:46:41,  8.69s/it][2025-04-25 23:45:48,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:45:48,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.23 | bwd_microstep: 5735.60 | bwd_inner_microstep: 5675.08 | bwd_allreduce_microstep: 60.46 | step_microstep: 18.97
[2025-04-25 23:45:48,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.23 | bwd: 5735.61 | bwd_inner: 5675.08 | bwd_allreduce: 60.48 | step: 18.98

 16%|█▌        | 6540/41250 [15:48:13<83:41:47,  8.68s/it]
                                                          
{'loss': 0.0742, 'grad_norm': 1.6190894842147827, 'learning_rate': 3.8291882856604985e-05, 'epoch': 1.59}

 16%|█▌        | 6540/41250 [15:48:13<83:41:47,  8.68s/it][2025-04-25 23:45:56,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-25 23:45:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.78 | bwd_microstep: 5669.60 | bwd_inner_microstep: 5656.25 | bwd_allreduce_microstep: 13.31 | step_microstep: 19.12
[2025-04-25 23:45:56,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.78 | bwd: 5669.62 | bwd_inner: 5656.25 | bwd_allreduce: 13.33 | step: 19.13

 16%|█▌        | 6541/41250 [15:48:21<83:27:45,  8.66s/it]
                                                          
{'loss': 0.0871, 'grad_norm': 1.7056450843811035, 'learning_rate': 3.8291247803039044e-05, 'epoch': 1.59}

 16%|█▌        | 6541/41250 [15:48:21<83:27:45,  8.66s/it][2025-04-25 23:46:05,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:46:05,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.60 | bwd_microstep: 5697.19 | bwd_inner_microstep: 5648.58 | bwd_allreduce_microstep: 48.57 | step_microstep: 18.89
[2025-04-25 23:46:05,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.60 | bwd: 5697.21 | bwd_inner: 5648.58 | bwd_allreduce: 48.58 | step: 18.89

 16%|█▌        | 6542/41250 [15:48:30<83:19:39,  8.64s/it]
                                                          
{'loss': 0.2091, 'grad_norm': 4.754953861236572, 'learning_rate': 3.829061263671115e-05, 'epoch': 1.59}

 16%|█▌        | 6542/41250 [15:48:30<83:19:39,  8.64s/it][2025-04-25 23:46:13,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:46:13,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.31 | bwd_microstep: 5781.16 | bwd_inner_microstep: 5640.35 | bwd_allreduce_microstep: 140.76 | step_microstep: 18.64
[2025-04-25 23:46:13,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.31 | bwd: 5781.17 | bwd_inner: 5640.35 | bwd_allreduce: 140.78 | step: 18.64

 16%|█▌        | 6543/41250 [15:48:39<83:27:08,  8.66s/it]
                                                          
{'loss': 0.0452, 'grad_norm': 1.1318522691726685, 'learning_rate': 3.8289977357625237e-05, 'epoch': 1.59}

 16%|█▌        | 6543/41250 [15:48:39<83:27:08,  8.66s/it][2025-04-25 23:46:22,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 23:46:22,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.06 | bwd_microstep: 5694.75 | bwd_inner_microstep: 5681.74 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.75
[2025-04-25 23:46:22,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.06 | bwd: 5694.76 | bwd_inner: 5681.74 | bwd_allreduce: 12.98 | step: 18.75

 16%|█▌        | 6544/41250 [15:48:47<83:21:42,  8.65s/it]
                                                          
{'loss': 0.0163, 'grad_norm': 0.42357197403907776, 'learning_rate': 3.82893419657852e-05, 'epoch': 1.59}

 16%|█▌        | 6544/41250 [15:48:47<83:21:42,  8.65s/it][2025-04-25 23:46:31,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-25 23:46:31,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5750.90 | bwd_inner_microstep: 5683.60 | bwd_allreduce_microstep: 67.25 | step_microstep: 18.80
[2025-04-25 23:46:31,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5750.91 | bwd_inner: 5683.60 | bwd_allreduce: 67.27 | step: 18.80

 16%|█▌        | 6545/41250 [15:48:56<83:27:43,  8.66s/it]
                                                          
{'loss': 0.0201, 'grad_norm': 1.034962773323059, 'learning_rate': 3.828870646119498e-05, 'epoch': 1.59}

 16%|█▌        | 6545/41250 [15:48:56<83:27:43,  8.66s/it][2025-04-25 23:46:39,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-25 23:46:39,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.38 | bwd_microstep: 5789.08 | bwd_inner_microstep: 5652.01 | bwd_allreduce_microstep: 137.03 | step_microstep: 18.81
[2025-04-25 23:46:39,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.38 | bwd: 5789.10 | bwd_inner: 5652.01 | bwd_allreduce: 137.04 | step: 18.81

 16%|█▌        | 6546/41250 [15:49:05<83:34:26,  8.67s/it]
                                                          
{'loss': 0.0705, 'grad_norm': 2.1382832527160645, 'learning_rate': 3.828807084385847e-05, 'epoch': 1.59}

 16%|█▌        | 6546/41250 [15:49:05<83:34:26,  8.67s/it][2025-04-25 23:46:48,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:46:48,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.31 | bwd_microstep: 5707.21 | bwd_inner_microstep: 5636.34 | bwd_allreduce_microstep: 70.83 | step_microstep: 18.61
[2025-04-25 23:46:48,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.31 | bwd: 5707.23 | bwd_inner: 5636.34 | bwd_allreduce: 70.84 | step: 18.61

 16%|█▌        | 6547/41250 [15:49:13<83:23:51,  8.65s/it]
                                                          
{'loss': 0.1133, 'grad_norm': 1.585102915763855, 'learning_rate': 3.828743511377961e-05, 'epoch': 1.59}

 16%|█▌        | 6547/41250 [15:49:13<83:23:51,  8.65s/it][2025-04-25 23:46:57,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:46:57,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.59 | bwd_microstep: 5774.42 | bwd_inner_microstep: 5761.64 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.24
[2025-04-25 23:46:57,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.59 | bwd: 5774.43 | bwd_inner: 5761.64 | bwd_allreduce: 12.75 | step: 18.25

 16%|█▌        | 6548/41250 [15:49:22<83:40:14,  8.68s/it]
                                                          
{'loss': 0.2863, 'grad_norm': 5.650994777679443, 'learning_rate': 3.82867992709623e-05, 'epoch': 1.59}

 16%|█▌        | 6548/41250 [15:49:22<83:40:14,  8.68s/it][2025-04-25 23:47:05,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:47:05,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.84 | bwd_microstep: 5766.00 | bwd_inner_microstep: 5635.21 | bwd_allreduce_microstep: 130.73 | step_microstep: 18.28
[2025-04-25 23:47:05,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.84 | bwd: 5766.01 | bwd_inner: 5635.21 | bwd_allreduce: 130.75 | step: 18.29

 16%|█▌        | 6549/41250 [15:49:31<83:38:51,  8.68s/it]
                                                          
{'loss': 0.2535, 'grad_norm': 2.5328171253204346, 'learning_rate': 3.828616331541048e-05, 'epoch': 1.59}

 16%|█▌        | 6549/41250 [15:49:31<83:38:51,  8.68s/it][2025-04-25 23:47:14,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-25 23:47:14,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.28 | bwd_microstep: 5735.48 | bwd_inner_microstep: 5687.58 | bwd_allreduce_microstep: 47.84 | step_microstep: 18.94
[2025-04-25 23:47:14,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.28 | bwd: 5735.49 | bwd_inner: 5687.58 | bwd_allreduce: 47.86 | step: 18.94

 16%|█▌        | 6550/41250 [15:49:39<83:36:11,  8.67s/it]
                                                          
{'loss': 0.0789, 'grad_norm': 1.0851484537124634, 'learning_rate': 3.828552724712805e-05, 'epoch': 1.59}

 16%|█▌        | 6550/41250 [15:49:39<83:36:11,  8.67s/it][2025-04-25 23:47:23,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:47:23,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.59 | bwd_microstep: 5752.24 | bwd_inner_microstep: 5676.59 | bwd_allreduce_microstep: 75.61 | step_microstep: 18.51
[2025-04-25 23:47:23,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.59 | bwd: 5752.26 | bwd_inner: 5676.59 | bwd_allreduce: 75.63 | step: 18.51

 16%|█▌        | 6551/41250 [15:49:48<83:36:33,  8.67s/it]
                                                          
{'loss': 0.2741, 'grad_norm': 2.1918087005615234, 'learning_rate': 3.828489106611894e-05, 'epoch': 1.59}

 16%|█▌        | 6551/41250 [15:49:48<83:36:33,  8.67s/it][2025-04-25 23:47:31,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-25 23:47:31,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.56 | bwd_microstep: 5694.62 | bwd_inner_microstep: 5639.82 | bwd_allreduce_microstep: 54.75 | step_microstep: 19.02
[2025-04-25 23:47:31,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.56 | bwd: 5694.64 | bwd_inner: 5639.82 | bwd_allreduce: 54.78 | step: 19.02

 16%|█▌        | 6552/41250 [15:49:57<83:24:28,  8.65s/it]
                                                          
{'loss': 0.227, 'grad_norm': 3.108659267425537, 'learning_rate': 3.828425477238708e-05, 'epoch': 1.59}

 16%|█▌        | 6552/41250 [15:49:57<83:24:28,  8.65s/it][2025-04-25 23:47:40,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:47:40,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.14 | bwd_microstep: 5803.19 | bwd_inner_microstep: 5632.35 | bwd_allreduce_microstep: 170.79 | step_microstep: 18.88
[2025-04-25 23:47:40,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.14 | bwd: 5803.20 | bwd_inner: 5632.35 | bwd_allreduce: 170.81 | step: 18.89

 16%|█▌        | 6553/41250 [15:50:05<83:35:28,  8.67s/it]
                                                          
{'loss': 0.2323, 'grad_norm': 1.826289176940918, 'learning_rate': 3.828361836593638e-05, 'epoch': 1.59}

 16%|█▌        | 6553/41250 [15:50:05<83:35:28,  8.67s/it][2025-04-25 23:47:49,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:47:49,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5983.89 | bwd_inner_microstep: 5697.85 | bwd_allreduce_microstep: 285.99 | step_microstep: 18.51
[2025-04-25 23:47:49,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5983.90 | bwd_inner: 5697.85 | bwd_allreduce: 286.01 | step: 18.51

 16%|█▌        | 6554/41250 [15:50:14<84:18:14,  8.75s/it]
                                                          
{'loss': 0.0564, 'grad_norm': 1.1337298154830933, 'learning_rate': 3.8282981846770774e-05, 'epoch': 1.59}

 16%|█▌        | 6554/41250 [15:50:14<84:18:14,  8.75s/it][2025-04-25 23:47:58,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-25 23:47:58,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.70 | bwd_microstep: 5711.93 | bwd_inner_microstep: 5698.94 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.46
[2025-04-25 23:47:58,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.70 | bwd: 5711.95 | bwd_inner: 5698.94 | bwd_allreduce: 12.97 | step: 18.47

 16%|█▌        | 6555/41250 [15:50:23<83:58:58,  8.71s/it]
                                                          
{'loss': 0.1982, 'grad_norm': 2.0654265880584717, 'learning_rate': 3.8282345214894184e-05, 'epoch': 1.59}

 16%|█▌        | 6555/41250 [15:50:23<83:58:58,  8.71s/it][2025-04-25 23:48:06,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-25 23:48:06,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.51 | bwd_microstep: 5754.60 | bwd_inner_microstep: 5649.58 | bwd_allreduce_microstep: 104.98 | step_microstep: 18.62
[2025-04-25 23:48:06,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.51 | bwd: 5754.61 | bwd_inner: 5649.58 | bwd_allreduce: 104.99 | step: 18.62

 16%|█▌        | 6556/41250 [15:50:32<83:50:27,  8.70s/it]
                                                          
{'loss': 0.1654, 'grad_norm': 2.716905117034912, 'learning_rate': 3.828170847031053e-05, 'epoch': 1.59}

 16%|█▌        | 6556/41250 [15:50:32<83:50:27,  8.70s/it][2025-04-25 23:48:15,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:48:15,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.23 | bwd_microstep: 5788.03 | bwd_inner_microstep: 5645.85 | bwd_allreduce_microstep: 142.14 | step_microstep: 18.51
[2025-04-25 23:48:15,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.23 | bwd: 5788.05 | bwd_inner: 5645.85 | bwd_allreduce: 142.16 | step: 18.51

 16%|█▌        | 6557/41250 [15:50:40<83:48:51,  8.70s/it]
                                                          
{'loss': 0.0681, 'grad_norm': 1.3288376331329346, 'learning_rate': 3.828107161302373e-05, 'epoch': 1.59}

 16%|█▌        | 6557/41250 [15:50:40<83:48:51,  8.70s/it][2025-04-25 23:48:24,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 23:48:24,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.86 | bwd_microstep: 5905.83 | bwd_inner_microstep: 5644.32 | bwd_allreduce_microstep: 261.45 | step_microstep: 18.87
[2025-04-25 23:48:24,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.86 | bwd: 5905.84 | bwd_inner: 5644.32 | bwd_allreduce: 261.47 | step: 18.87

 16%|█▌        | 6558/41250 [15:50:49<84:09:22,  8.73s/it]
                                                          
{'loss': 0.2793, 'grad_norm': 5.197048187255859, 'learning_rate': 3.828043464303773e-05, 'epoch': 1.59}

 16%|█▌        | 6558/41250 [15:50:49<84:09:22,  8.73s/it][2025-04-25 23:48:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:48:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.22 | bwd_microstep: 5799.75 | bwd_inner_microstep: 5786.86 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.76
[2025-04-25 23:48:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.22 | bwd: 5799.76 | bwd_inner: 5786.86 | bwd_allreduce: 12.86 | step: 18.76

 16%|█▌        | 6559/41250 [15:50:58<84:17:04,  8.75s/it]
                                                          
{'loss': 0.2098, 'grad_norm': 4.600319862365723, 'learning_rate': 3.827979756035644e-05, 'epoch': 1.59}

 16%|█▌        | 6559/41250 [15:50:58<84:17:04,  8.75s/it][2025-04-25 23:48:41,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-25 23:48:41,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5787.70 | bwd_inner_microstep: 5647.40 | bwd_allreduce_microstep: 140.26 | step_microstep: 19.21
[2025-04-25 23:48:41,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5787.72 | bwd_inner: 5647.40 | bwd_allreduce: 140.28 | step: 19.22

 16%|█▌        | 6560/41250 [15:51:07<84:10:00,  8.73s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 1.8443490266799927, 'learning_rate': 3.827916036498379e-05, 'epoch': 1.59}

 16%|█▌        | 6560/41250 [15:51:07<84:10:00,  8.73s/it][2025-04-25 23:48:50,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:48:50,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5770.82 | bwd_inner_microstep: 5704.54 | bwd_allreduce_microstep: 66.24 | step_microstep: 18.61
[2025-04-25 23:48:50,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5770.83 | bwd_inner: 5704.54 | bwd_allreduce: 66.26 | step: 18.61

 16%|█▌        | 6561/41250 [15:51:15<84:03:58,  8.72s/it]
                                                          
{'loss': 0.0836, 'grad_norm': 1.4390652179718018, 'learning_rate': 3.827852305692372e-05, 'epoch': 1.59}

 16%|█▌        | 6561/41250 [15:51:15<84:03:58,  8.72s/it][2025-04-25 23:48:59,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:48:59,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.23 | bwd_microstep: 5764.72 | bwd_inner_microstep: 5702.24 | bwd_allreduce_microstep: 62.44 | step_microstep: 18.63
[2025-04-25 23:48:59,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.23 | bwd: 5764.73 | bwd_inner: 5702.24 | bwd_allreduce: 62.45 | step: 18.63

 16%|█▌        | 6562/41250 [15:51:24<83:58:49,  8.72s/it]
                                                          
{'loss': 0.0844, 'grad_norm': 2.518033027648926, 'learning_rate': 3.827788563618015e-05, 'epoch': 1.59}

 16%|█▌        | 6562/41250 [15:51:24<83:58:49,  8.72s/it][2025-04-25 23:49:07,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:49:07,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.57 | bwd_microstep: 5773.43 | bwd_inner_microstep: 5664.11 | bwd_allreduce_microstep: 109.28 | step_microstep: 18.61
[2025-04-25 23:49:07,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.57 | bwd: 5773.44 | bwd_inner: 5664.11 | bwd_allreduce: 109.29 | step: 18.61

 16%|█▌        | 6563/41250 [15:51:33<83:58:05,  8.71s/it]
                                                          
{'loss': 0.2604, 'grad_norm': 2.2788193225860596, 'learning_rate': 3.8277248102757e-05, 'epoch': 1.59}

 16%|█▌        | 6563/41250 [15:51:33<83:58:05,  8.71s/it][2025-04-25 23:49:16,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:49:16,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.22 | bwd_microstep: 5705.49 | bwd_inner_microstep: 5692.77 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.65
[2025-04-25 23:49:16,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.22 | bwd: 5705.51 | bwd_inner: 5692.77 | bwd_allreduce: 12.69 | step: 18.65

 16%|█▌        | 6564/41250 [15:51:41<83:46:00,  8.69s/it]
                                                          
{'loss': 0.2708, 'grad_norm': 3.13101863861084, 'learning_rate': 3.827661045665822e-05, 'epoch': 1.59}

 16%|█▌        | 6564/41250 [15:51:41<83:46:00,  8.69s/it][2025-04-25 23:49:25,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:49:25,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.21 | bwd_microstep: 5702.22 | bwd_inner_microstep: 5687.57 | bwd_allreduce_microstep: 14.60 | step_microstep: 17.96
[2025-04-25 23:49:25,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.21 | bwd: 5702.23 | bwd_inner: 5687.57 | bwd_allreduce: 14.62 | step: 17.96

 16%|█▌        | 6565/41250 [15:51:50<83:36:15,  8.68s/it]
                                                          
{'loss': 0.0625, 'grad_norm': 0.9626396298408508, 'learning_rate': 3.8275972697887726e-05, 'epoch': 1.59}

 16%|█▌        | 6565/41250 [15:51:50<83:36:15,  8.68s/it][2025-04-25 23:49:33,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-25 23:49:33,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.38 | bwd_microstep: 5727.65 | bwd_inner_microstep: 5715.01 | bwd_allreduce_microstep: 12.58 | step_microstep: 19.07
[2025-04-25 23:49:33,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.38 | bwd: 5727.66 | bwd_inner: 5715.01 | bwd_allreduce: 12.60 | step: 19.07

 16%|█▌        | 6566/41250 [15:51:59<83:37:02,  8.68s/it]
                                                          
{'loss': 0.1463, 'grad_norm': 2.2108211517333984, 'learning_rate': 3.8275334826449466e-05, 'epoch': 1.59}

 16%|█▌        | 6566/41250 [15:51:59<83:37:02,  8.68s/it][2025-04-25 23:49:42,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.18 | optimizer_step: 0.91
[2025-04-25 23:49:42,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.27 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5666.13 | bwd_allreduce_microstep: 100.87 | step_microstep: 18.70
[2025-04-25 23:49:42,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.28 | bwd: 5767.06 | bwd_inner: 5666.13 | bwd_allreduce: 100.89 | step: 18.70

 16%|█▌        | 6567/41250 [15:52:07<83:40:34,  8.69s/it]
                                                          
{'loss': 0.1738, 'grad_norm': 3.0890302658081055, 'learning_rate': 3.827469684234735e-05, 'epoch': 1.59}

 16%|█▌        | 6567/41250 [15:52:07<83:40:34,  8.69s/it][2025-04-25 23:49:51,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-25 23:49:51,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5766.64 | bwd_inner_microstep: 5692.78 | bwd_allreduce_microstep: 73.81 | step_microstep: 18.82
[2025-04-25 23:49:51,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5766.65 | bwd_inner: 5692.78 | bwd_allreduce: 73.83 | step: 18.83

 16%|█▌        | 6568/41250 [15:52:16<83:42:59,  8.69s/it]
                                                          
{'loss': 0.1414, 'grad_norm': 2.6571404933929443, 'learning_rate': 3.827405874558532e-05, 'epoch': 1.59}

 16%|█▌        | 6568/41250 [15:52:16<83:42:59,  8.69s/it][2025-04-25 23:50:00,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-25 23:50:00,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5756.02 | bwd_inner_microstep: 5703.90 | bwd_allreduce_microstep: 52.07 | step_microstep: 19.20
[2025-04-25 23:50:00,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5756.03 | bwd_inner: 5703.90 | bwd_allreduce: 52.09 | step: 19.20

 16%|█▌        | 6569/41250 [15:52:25<83:43:40,  8.69s/it]
                                                          
{'loss': 0.1292, 'grad_norm': 1.5773566961288452, 'learning_rate': 3.827342053616732e-05, 'epoch': 1.59}

 16%|█▌        | 6569/41250 [15:52:25<83:43:40,  8.69s/it][2025-04-25 23:50:08,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:50:08,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5761.98 | bwd_inner_microstep: 5701.09 | bwd_allreduce_microstep: 60.84 | step_microstep: 18.62
[2025-04-25 23:50:08,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5761.99 | bwd_inner: 5701.09 | bwd_allreduce: 60.86 | step: 18.63

 16%|█▌        | 6570/41250 [15:52:34<83:44:45,  8.69s/it]
                                                          
{'loss': 0.1368, 'grad_norm': 2.438987970352173, 'learning_rate': 3.8272782214097276e-05, 'epoch': 1.59}

 16%|█▌        | 6570/41250 [15:52:34<83:44:45,  8.69s/it][2025-04-25 23:50:17,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:50:17,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.89 | bwd_microstep: 5718.47 | bwd_inner_microstep: 5705.88 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.51
[2025-04-25 23:50:17,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.89 | bwd: 5718.49 | bwd_inner: 5705.88 | bwd_allreduce: 12.57 | step: 18.51

 16%|█▌        | 6571/41250 [15:52:42<83:39:15,  8.68s/it]
                                                          
{'loss': 0.256, 'grad_norm': 3.4783871173858643, 'learning_rate': 3.827214377937912e-05, 'epoch': 1.59}

 16%|█▌        | 6571/41250 [15:52:42<83:39:15,  8.68s/it][2025-04-25 23:50:26,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.28 | optimizer_step: 1.04
[2025-04-25 23:50:26,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5727.44 | bwd_inner_microstep: 5689.74 | bwd_allreduce_microstep: 37.65 | step_microstep: 20.04
[2025-04-25 23:50:26,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.49 | bwd: 5727.45 | bwd_inner: 5689.74 | bwd_allreduce: 37.67 | step: 20.05

 16%|█▌        | 6572/41250 [15:52:51<83:34:46,  8.68s/it]
                                                          
{'loss': 0.0624, 'grad_norm': 1.5454639196395874, 'learning_rate': 3.827150523201679e-05, 'epoch': 1.59}

 16%|█▌        | 6572/41250 [15:52:51<83:34:46,  8.68s/it][2025-04-25 23:50:34,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-25 23:50:34,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.47 | bwd_microstep: 5774.35 | bwd_inner_microstep: 5649.37 | bwd_allreduce_microstep: 124.93 | step_microstep: 18.45
[2025-04-25 23:50:34,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.47 | bwd: 5774.36 | bwd_inner: 5649.37 | bwd_allreduce: 124.95 | step: 18.46

 16%|█▌        | 6573/41250 [15:53:00<83:35:25,  8.68s/it]
                                                          
{'loss': 0.2406, 'grad_norm': 1.9462337493896484, 'learning_rate': 3.827086657201423e-05, 'epoch': 1.59}

 16%|█▌        | 6573/41250 [15:53:00<83:35:25,  8.68s/it][2025-04-25 23:50:43,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:50:43,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5702.98 | bwd_inner_microstep: 5654.50 | bwd_allreduce_microstep: 48.44 | step_microstep: 18.66
[2025-04-25 23:50:43,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.92 | bwd: 5703.00 | bwd_inner: 5654.50 | bwd_allreduce: 48.45 | step: 18.66

 16%|█▌        | 6574/41250 [15:53:08<83:23:39,  8.66s/it]
                                                          
{'loss': 0.0206, 'grad_norm': 0.19028429687023163, 'learning_rate': 3.8270227799375376e-05, 'epoch': 1.59}

 16%|█▌        | 6574/41250 [15:53:08<83:23:39,  8.66s/it][2025-04-25 23:50:52,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:50:52,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.15 | bwd_microstep: 5756.25 | bwd_inner_microstep: 5696.80 | bwd_allreduce_microstep: 59.41 | step_microstep: 18.86
[2025-04-25 23:50:52,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.16 | bwd: 5756.26 | bwd_inner: 5696.79 | bwd_allreduce: 59.42 | step: 18.87

 16%|█▌        | 6575/41250 [15:53:17<83:28:24,  8.67s/it]
                                                          
{'loss': 0.1963, 'grad_norm': 1.8097047805786133, 'learning_rate': 3.826958891410415e-05, 'epoch': 1.59}

 16%|█▌        | 6575/41250 [15:53:17<83:28:24,  8.67s/it][2025-04-25 23:51:00,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:51:00,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5795.17 | bwd_inner_microstep: 5659.23 | bwd_allreduce_microstep: 135.90 | step_microstep: 19.24
[2025-04-25 23:51:00,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5795.19 | bwd_inner: 5659.23 | bwd_allreduce: 135.92 | step: 19.24

 16%|█▌        | 6576/41250 [15:53:26<83:35:17,  8.68s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 0.6161969304084778, 'learning_rate': 3.82689499162045e-05, 'epoch': 1.59}

 16%|█▌        | 6576/41250 [15:53:26<83:35:17,  8.68s/it][2025-04-25 23:51:09,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:51:09,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.35 | bwd_microstep: 5764.19 | bwd_inner_microstep: 5676.84 | bwd_allreduce_microstep: 87.31 | step_microstep: 18.83
[2025-04-25 23:51:09,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.35 | bwd: 5764.20 | bwd_inner: 5676.84 | bwd_allreduce: 87.32 | step: 18.84

 16%|█▌        | 6577/41250 [15:53:34<83:37:32,  8.68s/it]
                                                          
{'loss': 0.1671, 'grad_norm': 2.5845906734466553, 'learning_rate': 3.826831080568038e-05, 'epoch': 1.59}

 16%|█▌        | 6577/41250 [15:53:34<83:37:32,  8.68s/it][2025-04-25 23:51:18,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-25 23:51:18,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.25 | bwd_microstep: 5704.10 | bwd_inner_microstep: 5691.03 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.05
[2025-04-25 23:51:18,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.25 | bwd: 5704.12 | bwd_inner: 5691.03 | bwd_allreduce: 13.04 | step: 19.05

 16%|█▌        | 6578/41250 [15:53:43<83:28:36,  8.67s/it]
                                                          
{'loss': 0.3543, 'grad_norm': 2.6254212856292725, 'learning_rate': 3.826767158253571e-05, 'epoch': 1.59}

 16%|█▌        | 6578/41250 [15:53:43<83:28:36,  8.67s/it][2025-04-25 23:51:26,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:51:26,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.60 | bwd_microstep: 5739.42 | bwd_inner_microstep: 5679.37 | bwd_allreduce_microstep: 60.01 | step_microstep: 18.58
[2025-04-25 23:51:26,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.60 | bwd: 5739.44 | bwd_inner: 5679.37 | bwd_allreduce: 60.03 | step: 18.58

 16%|█▌        | 6579/41250 [15:53:52<83:27:48,  8.67s/it]
                                                          
{'loss': 0.145, 'grad_norm': 2.0294926166534424, 'learning_rate': 3.8267032246774435e-05, 'epoch': 1.59}

 16%|█▌        | 6579/41250 [15:53:52<83:27:48,  8.67s/it][2025-04-25 23:51:35,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-25 23:51:35,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.05 | bwd_microstep: 5701.54 | bwd_inner_microstep: 5688.61 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.63
[2025-04-25 23:51:35,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.05 | bwd: 5701.56 | bwd_inner: 5688.61 | bwd_allreduce: 12.90 | step: 18.63

 16%|█▌        | 6580/41250 [15:54:00<83:20:22,  8.65s/it]
                                                          
{'loss': 0.0351, 'grad_norm': 0.4046112596988678, 'learning_rate': 3.8266392798400505e-05, 'epoch': 1.6}

 16%|█▌        | 6580/41250 [15:54:00<83:20:22,  8.65s/it][2025-04-25 23:51:44,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.13 | optimizer_step: 1.03
[2025-04-25 23:51:44,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.85 | bwd_microstep: 5762.46 | bwd_inner_microstep: 5707.29 | bwd_allreduce_microstep: 55.11 | step_microstep: 19.68
[2025-04-25 23:51:44,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.85 | bwd: 5762.47 | bwd_inner: 5707.29 | bwd_allreduce: 55.14 | step: 19.68

 16%|█▌        | 6581/41250 [15:54:09<83:28:32,  8.67s/it]
                                                          
{'loss': 0.147, 'grad_norm': 3.097174882888794, 'learning_rate': 3.8265753237417847e-05, 'epoch': 1.6}

 16%|█▌        | 6581/41250 [15:54:09<83:28:32,  8.67s/it][2025-04-25 23:51:52,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:51:52,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.71 | bwd_microstep: 5704.65 | bwd_inner_microstep: 5691.82 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.63
[2025-04-25 23:51:52,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.71 | bwd: 5704.66 | bwd_inner: 5691.82 | bwd_allreduce: 12.80 | step: 18.63

 16%|█▌        | 6582/41250 [15:54:18<83:22:31,  8.66s/it]
                                                          
{'loss': 0.2185, 'grad_norm': 4.507137775421143, 'learning_rate': 3.8265113563830425e-05, 'epoch': 1.6}

 16%|█▌        | 6582/41250 [15:54:18<83:22:31,  8.66s/it][2025-04-25 23:52:01,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-25 23:52:01,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.20 | bwd_microstep: 5706.40 | bwd_inner_microstep: 5693.66 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.66
[2025-04-25 23:52:01,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.20 | bwd: 5706.42 | bwd_inner: 5693.66 | bwd_allreduce: 12.72 | step: 18.66

 16%|█▌        | 6583/41250 [15:54:26<83:20:47,  8.66s/it]
                                                          
{'loss': 0.1875, 'grad_norm': 1.3512026071548462, 'learning_rate': 3.8264473777642164e-05, 'epoch': 1.6}

 16%|█▌        | 6583/41250 [15:54:26<83:20:47,  8.66s/it][2025-04-25 23:52:09,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:52:09,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.33 | bwd_microstep: 5693.17 | bwd_inner_microstep: 5680.27 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.73
[2025-04-25 23:52:09,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.33 | bwd: 5693.18 | bwd_inner: 5680.27 | bwd_allreduce: 12.88 | step: 18.73

 16%|█▌        | 6584/41250 [15:54:35<83:16:47,  8.65s/it]
                                                          
{'loss': 0.281, 'grad_norm': 1.4678126573562622, 'learning_rate': 3.826383387885701e-05, 'epoch': 1.6}

 16%|█▌        | 6584/41250 [15:54:35<83:16:47,  8.65s/it][2025-04-25 23:52:18,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:52:18,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.54 | bwd_microstep: 5762.17 | bwd_inner_microstep: 5749.30 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.68
[2025-04-25 23:52:18,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.54 | bwd: 5762.18 | bwd_inner: 5749.30 | bwd_allreduce: 12.84 | step: 18.68

 16%|█▌        | 6585/41250 [15:54:44<83:29:44,  8.67s/it]
                                                          
{'loss': 0.1111, 'grad_norm': 0.9641860723495483, 'learning_rate': 3.826319386747892e-05, 'epoch': 1.6}

 16%|█▌        | 6585/41250 [15:54:44<83:29:44,  8.67s/it][2025-04-25 23:52:27,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:52:27,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.57 | bwd_microstep: 5816.93 | bwd_inner_microstep: 5766.33 | bwd_allreduce_microstep: 50.56 | step_microstep: 18.57
[2025-04-25 23:52:27,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.57 | bwd: 5816.94 | bwd_inner: 5766.33 | bwd_allreduce: 50.57 | step: 18.58

 16%|█▌        | 6586/41250 [15:54:52<83:49:14,  8.71s/it]
                                                          
{'loss': 0.2458, 'grad_norm': 2.581698417663574, 'learning_rate': 3.826255374351183e-05, 'epoch': 1.6}

 16%|█▌        | 6586/41250 [15:54:52<83:49:14,  8.71s/it][2025-04-25 23:52:36,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:52:36,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.15 | bwd_microstep: 5758.16 | bwd_inner_microstep: 5634.39 | bwd_allreduce_microstep: 123.72 | step_microstep: 18.32
[2025-04-25 23:52:36,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.15 | bwd: 5758.17 | bwd_inner: 5634.39 | bwd_allreduce: 123.74 | step: 18.32

 16%|█▌        | 6587/41250 [15:55:01<83:44:26,  8.70s/it]
                                                          
{'loss': 0.1675, 'grad_norm': 1.7877860069274902, 'learning_rate': 3.826191350695969e-05, 'epoch': 1.6}

 16%|█▌        | 6587/41250 [15:55:01<83:44:26,  8.70s/it][2025-04-25 23:52:44,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-25 23:52:44,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.11 | bwd_microstep: 5702.81 | bwd_inner_microstep: 5640.63 | bwd_allreduce_microstep: 62.13 | step_microstep: 18.33
[2025-04-25 23:52:44,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.11 | bwd: 5702.82 | bwd_inner: 5640.63 | bwd_allreduce: 62.15 | step: 18.34

 16%|█▌        | 6588/41250 [15:55:10<83:31:02,  8.67s/it]
                                                          
{'loss': 0.1753, 'grad_norm': 3.073418378829956, 'learning_rate': 3.826127315782644e-05, 'epoch': 1.6}

 16%|█▌        | 6588/41250 [15:55:10<83:31:02,  8.67s/it][2025-04-25 23:52:53,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-25 23:52:53,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.84 | bwd_microstep: 5692.37 | bwd_inner_microstep: 5679.16 | bwd_allreduce_microstep: 13.16 | step_microstep: 19.19
[2025-04-25 23:52:53,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.84 | bwd: 5692.39 | bwd_inner: 5679.16 | bwd_allreduce: 13.18 | step: 19.20

 16%|█▌        | 6589/41250 [15:55:18<83:21:38,  8.66s/it]
                                                          
{'loss': 0.2381, 'grad_norm': 1.9331878423690796, 'learning_rate': 3.8260632696116035e-05, 'epoch': 1.6}

 16%|█▌        | 6589/41250 [15:55:18<83:21:38,  8.66s/it][2025-04-25 23:53:02,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 23:53:02,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.36 | bwd_microstep: 5693.63 | bwd_inner_microstep: 5643.85 | bwd_allreduce_microstep: 49.74 | step_microstep: 18.57
[2025-04-25 23:53:02,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.36 | bwd: 5693.65 | bwd_inner: 5643.85 | bwd_allreduce: 49.76 | step: 18.58

 16%|█▌        | 6590/41250 [15:55:27<83:10:50,  8.64s/it]
                                                          
{'loss': 0.3688, 'grad_norm': 1.923591136932373, 'learning_rate': 3.825999212183242e-05, 'epoch': 1.6}

 16%|█▌        | 6590/41250 [15:55:27<83:10:50,  8.64s/it][2025-04-25 23:53:10,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-25 23:53:10,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.41 | bwd_microstep: 5834.66 | bwd_inner_microstep: 5674.90 | bwd_allreduce_microstep: 159.72 | step_microstep: 18.52
[2025-04-25 23:53:10,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.41 | bwd: 5834.67 | bwd_inner: 5674.90 | bwd_allreduce: 159.74 | step: 18.53

 16%|█▌        | 6591/41250 [15:55:36<83:32:10,  8.68s/it]
                                                          
{'loss': 0.1473, 'grad_norm': 2.0853333473205566, 'learning_rate': 3.825935143497955e-05, 'epoch': 1.6}

 16%|█▌        | 6591/41250 [15:55:36<83:32:10,  8.68s/it][2025-04-25 23:53:19,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:53:19,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.02 | bwd_microstep: 5692.64 | bwd_inner_microstep: 5679.79 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.52
[2025-04-25 23:53:19,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.02 | bwd: 5692.65 | bwd_inner: 5679.79 | bwd_allreduce: 12.82 | step: 18.53

 16%|█▌        | 6592/41250 [15:55:44<83:22:14,  8.66s/it]
                                                          
{'loss': 0.1262, 'grad_norm': 5.295734882354736, 'learning_rate': 3.825871063556137e-05, 'epoch': 1.6}

 16%|█▌        | 6592/41250 [15:55:44<83:22:14,  8.66s/it][2025-04-25 23:53:28,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:53:28,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.26 | bwd_microstep: 5743.86 | bwd_inner_microstep: 5682.91 | bwd_allreduce_microstep: 60.90 | step_microstep: 18.19
[2025-04-25 23:53:28,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.26 | bwd: 5743.87 | bwd_inner: 5682.91 | bwd_allreduce: 60.92 | step: 18.20

 16%|█▌        | 6593/41250 [15:55:53<83:23:36,  8.66s/it]
                                                          
{'loss': 0.1636, 'grad_norm': 4.0525102615356445, 'learning_rate': 3.8258069723581826e-05, 'epoch': 1.6}

 16%|█▌        | 6593/41250 [15:55:53<83:23:36,  8.66s/it][2025-04-25 23:53:36,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:53:36,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.95 | bwd_microstep: 5856.79 | bwd_inner_microstep: 5691.23 | bwd_allreduce_microstep: 165.52 | step_microstep: 18.52
[2025-04-25 23:53:36,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.95 | bwd: 5856.80 | bwd_inner: 5691.23 | bwd_allreduce: 165.53 | step: 18.53

 16%|█▌        | 6594/41250 [15:56:02<83:45:53,  8.70s/it]
                                                          
{'loss': 0.1695, 'grad_norm': 2.162536859512329, 'learning_rate': 3.8257428699044874e-05, 'epoch': 1.6}

 16%|█▌        | 6594/41250 [15:56:02<83:45:53,  8.70s/it][2025-04-25 23:53:45,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:53:45,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.23 | bwd_microstep: 6027.77 | bwd_inner_microstep: 5653.52 | bwd_allreduce_microstep: 374.21 | step_microstep: 18.36
[2025-04-25 23:53:45,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.23 | bwd: 6027.79 | bwd_inner: 5653.52 | bwd_allreduce: 374.23 | step: 18.37

 16%|█▌        | 6595/41250 [15:56:11<84:28:21,  8.78s/it]
                                                          
{'loss': 0.0871, 'grad_norm': 1.2454789876937866, 'learning_rate': 3.825678756195446e-05, 'epoch': 1.6}

 16%|█▌        | 6595/41250 [15:56:11<84:28:21,  8.78s/it][2025-04-25 23:53:54,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:53:54,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.63 | bwd_microstep: 5770.10 | bwd_inner_microstep: 5694.42 | bwd_allreduce_microstep: 75.64 | step_microstep: 18.50
[2025-04-25 23:53:54,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.63 | bwd: 5770.12 | bwd_inner: 5694.42 | bwd_allreduce: 75.66 | step: 18.50

 16%|█▌        | 6596/41250 [15:56:19<84:16:17,  8.75s/it]
                                                          
{'loss': 0.0625, 'grad_norm': 1.8815714120864868, 'learning_rate': 3.8256146312314546e-05, 'epoch': 1.6}

 16%|█▌        | 6596/41250 [15:56:19<84:16:17,  8.75s/it][2025-04-25 23:54:03,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:54:03,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.94 | bwd_microstep: 5756.85 | bwd_inner_microstep: 5650.04 | bwd_allreduce_microstep: 106.76 | step_microstep: 18.59
[2025-04-25 23:54:03,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.94 | bwd: 5756.86 | bwd_inner: 5650.04 | bwd_allreduce: 106.78 | step: 18.59

 16%|█▌        | 6597/41250 [15:56:28<84:01:41,  8.73s/it]
                                                          
{'loss': 0.099, 'grad_norm': 0.8210248947143555, 'learning_rate': 3.8255504950129086e-05, 'epoch': 1.6}

 16%|█▌        | 6597/41250 [15:56:28<84:01:41,  8.73s/it][2025-04-25 23:54:11,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-25 23:54:11,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.66 | bwd_microstep: 5701.31 | bwd_inner_microstep: 5642.20 | bwd_allreduce_microstep: 59.06 | step_microstep: 19.13
[2025-04-25 23:54:11,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.66 | bwd: 5701.32 | bwd_inner: 5642.20 | bwd_allreduce: 59.08 | step: 19.14

 16%|█▌        | 6598/41250 [15:56:37<83:42:39,  8.70s/it]
                                                          
{'loss': 0.0666, 'grad_norm': 0.892745852470398, 'learning_rate': 3.825486347540202e-05, 'epoch': 1.6}

 16%|█▌        | 6598/41250 [15:56:37<83:42:39,  8.70s/it][2025-04-25 23:54:20,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-25 23:54:20,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.99 | bwd_microstep: 5736.15 | bwd_inner_microstep: 5691.26 | bwd_allreduce_microstep: 44.83 | step_microstep: 18.88
[2025-04-25 23:54:20,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.99 | bwd: 5736.16 | bwd_inner: 5691.26 | bwd_allreduce: 44.86 | step: 18.89

 16%|█▌        | 6599/41250 [15:56:45<83:39:17,  8.69s/it]
                                                          
{'loss': 0.1165, 'grad_norm': 0.9396278858184814, 'learning_rate': 3.8254221888137316e-05, 'epoch': 1.6}

 16%|█▌        | 6599/41250 [15:56:45<83:39:17,  8.69s/it][2025-04-25 23:54:29,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-25 23:54:29,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.18 | bwd_microstep: 5757.62 | bwd_inner_microstep: 5659.01 | bwd_allreduce_microstep: 98.57 | step_microstep: 18.24
[2025-04-25 23:54:29,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.18 | bwd: 5757.64 | bwd_inner: 5659.01 | bwd_allreduce: 98.58 | step: 18.24

 16%|█▌        | 6600/41250 [15:56:54<83:36:32,  8.69s/it]
                                                          
{'loss': 0.1736, 'grad_norm': 1.546128273010254, 'learning_rate': 3.825358018833893e-05, 'epoch': 1.6}

 16%|█▌        | 6600/41250 [15:56:54<83:36:32,  8.69s/it][2025-04-25 23:54:37,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:54:37,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.13 | bwd_microstep: 5798.72 | bwd_inner_microstep: 5647.06 | bwd_allreduce_microstep: 151.61 | step_microstep: 18.25
[2025-04-25 23:54:37,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.13 | bwd: 5798.73 | bwd_inner: 5647.06 | bwd_allreduce: 151.63 | step: 18.25

 16%|█▌        | 6601/41250 [15:57:03<83:39:00,  8.69s/it]
                                                          
{'loss': 0.2296, 'grad_norm': 1.0812448263168335, 'learning_rate': 3.8252938376010805e-05, 'epoch': 1.6}

 16%|█▌        | 6601/41250 [15:57:03<83:39:00,  8.69s/it][2025-04-25 23:54:46,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-25 23:54:46,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.91 | bwd_microstep: 5718.93 | bwd_inner_microstep: 5644.88 | bwd_allreduce_microstep: 74.00 | step_microstep: 18.77
[2025-04-25 23:54:46,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.91 | bwd: 5718.94 | bwd_inner: 5644.88 | bwd_allreduce: 74.01 | step: 18.77

 16%|█▌        | 6602/41250 [15:57:11<83:27:32,  8.67s/it]
                                                          
{'loss': 0.082, 'grad_norm': 1.0212043523788452, 'learning_rate': 3.82522964511569e-05, 'epoch': 1.6}

 16%|█▌        | 6602/41250 [15:57:11<83:27:32,  8.67s/it][2025-04-25 23:54:55,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:54:55,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.05 | bwd_microstep: 5754.92 | bwd_inner_microstep: 5688.96 | bwd_allreduce_microstep: 65.91 | step_microstep: 18.56
[2025-04-25 23:54:55,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.05 | bwd: 5754.93 | bwd_inner: 5688.96 | bwd_allreduce: 65.93 | step: 18.56

 16%|█▌        | 6603/41250 [15:57:20<83:30:06,  8.68s/it]
                                                          
{'loss': 0.1154, 'grad_norm': 1.3526275157928467, 'learning_rate': 3.825165441378119e-05, 'epoch': 1.6}

 16%|█▌        | 6603/41250 [15:57:20<83:30:06,  8.68s/it][2025-04-25 23:55:03,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:55:03,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5765.66 | bwd_inner_microstep: 5704.14 | bwd_allreduce_microstep: 61.47 | step_microstep: 18.64
[2025-04-25 23:55:03,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5765.67 | bwd_inner: 5704.14 | bwd_allreduce: 61.49 | step: 18.64

 16%|█▌        | 6604/41250 [15:57:29<83:33:38,  8.68s/it]
                                                          
{'loss': 0.2558, 'grad_norm': 2.007232666015625, 'learning_rate': 3.8251012263887615e-05, 'epoch': 1.6}

 16%|█▌        | 6604/41250 [15:57:29<83:33:38,  8.68s/it][2025-04-25 23:55:12,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-25 23:55:12,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.70 | bwd_microstep: 5770.92 | bwd_inner_microstep: 5758.24 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.32
[2025-04-25 23:55:12,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.70 | bwd: 5770.93 | bwd_inner: 5758.24 | bwd_allreduce: 12.65 | step: 19.32

 16%|█▌        | 6605/41250 [15:57:37<83:43:33,  8.70s/it]
                                                          
{'loss': 0.1697, 'grad_norm': 2.060821294784546, 'learning_rate': 3.825037000148013e-05, 'epoch': 1.6}

 16%|█▌        | 6605/41250 [15:57:37<83:43:33,  8.70s/it][2025-04-25 23:55:21,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.94 | optimizer_step: 1.08
[2025-04-25 23:55:21,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.53 | bwd_microstep: 5788.45 | bwd_inner_microstep: 5657.48 | bwd_allreduce_microstep: 130.93 | step_microstep: 18.53
[2025-04-25 23:55:21,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.53 | bwd: 5788.47 | bwd_inner: 5657.47 | bwd_allreduce: 130.95 | step: 18.54

 16%|█▌        | 6606/41250 [15:57:46<83:43:36,  8.70s/it]
                                                          
{'loss': 0.272, 'grad_norm': 3.228623628616333, 'learning_rate': 3.824972762656272e-05, 'epoch': 1.6}

 16%|█▌        | 6606/41250 [15:57:46<83:43:36,  8.70s/it][2025-04-25 23:55:29,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-25 23:55:29,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.78 | bwd_microstep: 5717.85 | bwd_inner_microstep: 5658.79 | bwd_allreduce_microstep: 59.01 | step_microstep: 18.71
[2025-04-25 23:55:29,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.78 | bwd: 5717.87 | bwd_inner: 5658.79 | bwd_allreduce: 59.03 | step: 18.72

 16%|█▌        | 6607/41250 [15:57:55<83:33:05,  8.68s/it]
                                                          
{'loss': 0.2256, 'grad_norm': 3.5151307582855225, 'learning_rate': 3.824908513913932e-05, 'epoch': 1.6}

 16%|█▌        | 6607/41250 [15:57:55<83:33:05,  8.68s/it][2025-04-25 23:55:38,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:55:38,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5706.02 | bwd_inner_microstep: 5658.90 | bwd_allreduce_microstep: 47.08 | step_microstep: 18.72
[2025-04-25 23:55:38,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5706.03 | bwd_inner: 5658.90 | bwd_allreduce: 47.09 | step: 18.72

 16%|█▌        | 6608/41250 [15:58:03<83:21:46,  8.66s/it]
                                                          
{'loss': 0.0883, 'grad_norm': 1.3068981170654297, 'learning_rate': 3.82484425392139e-05, 'epoch': 1.6}

 16%|█▌        | 6608/41250 [15:58:03<83:21:46,  8.66s/it][2025-04-25 23:55:47,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:55:47,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.98 | bwd_microstep: 5726.02 | bwd_inner_microstep: 5713.19 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.53
[2025-04-25 23:55:47,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.98 | bwd: 5726.03 | bwd_inner: 5713.19 | bwd_allreduce: 12.80 | step: 18.53

 16%|█▌        | 6609/41250 [15:58:12<83:23:28,  8.67s/it]
                                                          
{'loss': 0.0501, 'grad_norm': 0.8206552863121033, 'learning_rate': 3.824779982679042e-05, 'epoch': 1.6}

 16%|█▌        | 6609/41250 [15:58:12<83:23:28,  8.67s/it][2025-04-25 23:55:55,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-25 23:55:55,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.53 | bwd_microstep: 5766.93 | bwd_inner_microstep: 5697.28 | bwd_allreduce_microstep: 69.61 | step_microstep: 18.59
[2025-04-25 23:55:55,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.53 | bwd: 5766.95 | bwd_inner: 5697.28 | bwd_allreduce: 69.62 | step: 18.59

 16%|█▌        | 6610/41250 [15:58:21<83:30:08,  8.68s/it]
                                                          
{'loss': 0.0943, 'grad_norm': 1.5548899173736572, 'learning_rate': 3.824715700187285e-05, 'epoch': 1.6}

 16%|█▌        | 6610/41250 [15:58:21<83:30:08,  8.68s/it][2025-04-25 23:56:04,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:56:04,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5778.00 | bwd_inner_microstep: 5655.35 | bwd_allreduce_microstep: 122.61 | step_microstep: 18.40
[2025-04-25 23:56:04,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5778.01 | bwd_inner: 5655.35 | bwd_allreduce: 122.63 | step: 18.41

 16%|█▌        | 6611/41250 [15:58:29<83:33:04,  8.68s/it]
                                                          
{'loss': 0.0386, 'grad_norm': 0.6692743301391602, 'learning_rate': 3.8246514064465135e-05, 'epoch': 1.6}

 16%|█▌        | 6611/41250 [15:58:29<83:33:04,  8.68s/it][2025-04-25 23:56:13,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.24
[2025-04-25 23:56:13,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5751.88 | bwd_inner_microstep: 5666.68 | bwd_allreduce_microstep: 85.15 | step_microstep: 19.68
[2025-04-25 23:56:13,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5751.89 | bwd_inner: 5666.68 | bwd_allreduce: 85.17 | step: 19.69

 16%|█▌        | 6612/41250 [15:58:38<83:32:45,  8.68s/it]
                                                          
{'loss': 0.1906, 'grad_norm': 1.8633825778961182, 'learning_rate': 3.824587101457126e-05, 'epoch': 1.6}

 16%|█▌        | 6612/41250 [15:58:38<83:32:45,  8.68s/it][2025-04-25 23:56:22,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-25 23:56:22,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.04 | bwd_microstep: 5774.00 | bwd_inner_microstep: 5640.58 | bwd_allreduce_microstep: 133.38 | step_microstep: 18.52
[2025-04-25 23:56:22,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.04 | bwd: 5774.01 | bwd_inner: 5640.58 | bwd_allreduce: 133.39 | step: 18.52

 16%|█▌        | 6613/41250 [15:58:47<83:33:08,  8.68s/it]
                                                          
{'loss': 0.1665, 'grad_norm': 2.949537754058838, 'learning_rate': 3.824522785219517e-05, 'epoch': 1.6}

 16%|█▌        | 6613/41250 [15:58:47<83:33:08,  8.68s/it][2025-04-25 23:56:30,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-25 23:56:30,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.26 | bwd_microstep: 5679.27 | bwd_inner_microstep: 5654.70 | bwd_allreduce_microstep: 24.53 | step_microstep: 18.66
[2025-04-25 23:56:30,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.26 | bwd: 5679.29 | bwd_inner: 5654.70 | bwd_allreduce: 24.55 | step: 18.66

 16%|█▌        | 6614/41250 [15:58:55<83:20:25,  8.66s/it]
                                                          
{'loss': 0.1091, 'grad_norm': 2.105473756790161, 'learning_rate': 3.824458457734085e-05, 'epoch': 1.6}

 16%|█▌        | 6614/41250 [15:58:55<83:20:25,  8.66s/it][2025-04-25 23:56:39,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-25 23:56:39,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.37 | bwd_microstep: 5709.77 | bwd_inner_microstep: 5697.03 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.90
[2025-04-25 23:56:39,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.37 | bwd: 5709.79 | bwd_inner: 5697.03 | bwd_allreduce: 12.71 | step: 18.90

 16%|█▌        | 6615/41250 [15:59:04<83:19:04,  8.66s/it]
                                                          
{'loss': 0.0461, 'grad_norm': 0.5629950165748596, 'learning_rate': 3.824394119001224e-05, 'epoch': 1.6}

 16%|█▌        | 6615/41250 [15:59:04<83:19:04,  8.66s/it][2025-04-25 23:56:47,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.14
[2025-04-25 23:56:47,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5704.85 | bwd_inner_microstep: 5691.93 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.34
[2025-04-25 23:56:47,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5704.87 | bwd_inner: 5691.93 | bwd_allreduce: 12.89 | step: 19.34

 16%|█▌        | 6616/41250 [15:59:13<83:15:32,  8.65s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.5498137474060059, 'learning_rate': 3.8243297690213334e-05, 'epoch': 1.6}

 16%|█▌        | 6616/41250 [15:59:13<83:15:32,  8.65s/it][2025-04-25 23:56:56,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-25 23:56:56,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.46 | bwd_microstep: 5737.15 | bwd_inner_microstep: 5686.42 | bwd_allreduce_microstep: 50.69 | step_microstep: 18.51
[2025-04-25 23:56:56,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.46 | bwd: 5737.17 | bwd_inner: 5686.42 | bwd_allreduce: 50.71 | step: 18.51

 16%|█▌        | 6617/41250 [15:59:21<83:21:05,  8.66s/it]
                                                          
{'loss': 0.151, 'grad_norm': 2.2439751625061035, 'learning_rate': 3.824265407794808e-05, 'epoch': 1.6}

 16%|█▌        | 6617/41250 [15:59:21<83:21:05,  8.66s/it][2025-04-25 23:57:05,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:57:05,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.60 | bwd_microstep: 5707.81 | bwd_inner_microstep: 5694.99 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.73
[2025-04-25 23:57:05,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.60 | bwd: 5707.83 | bwd_inner: 5694.99 | bwd_allreduce: 12.79 | step: 18.74

 16%|█▌        | 6618/41250 [15:59:30<83:18:23,  8.66s/it]
                                                          
{'loss': 0.169, 'grad_norm': 1.5543197393417358, 'learning_rate': 3.824201035322045e-05, 'epoch': 1.6}

 16%|█▌        | 6618/41250 [15:59:30<83:18:23,  8.66s/it][2025-04-25 23:57:13,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:57:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.94 | bwd_microstep: 5726.88 | bwd_inner_microstep: 5709.67 | bwd_allreduce_microstep: 17.17 | step_microstep: 18.44
[2025-04-25 23:57:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.94 | bwd: 5726.89 | bwd_inner: 5709.67 | bwd_allreduce: 17.19 | step: 18.45

 16%|█▌        | 6619/41250 [15:59:39<83:19:31,  8.66s/it]
                                                          
{'loss': 0.0888, 'grad_norm': 1.7850067615509033, 'learning_rate': 3.8241366516034416e-05, 'epoch': 1.6}

 16%|█▌        | 6619/41250 [15:59:39<83:19:31,  8.66s/it][2025-04-25 23:57:22,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-25 23:57:22,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5749.49 | bwd_inner_microstep: 5700.06 | bwd_allreduce_microstep: 49.37 | step_microstep: 18.52
[2025-04-25 23:57:22,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5749.50 | bwd_inner: 5700.06 | bwd_allreduce: 49.39 | step: 18.52

 16%|█▌        | 6620/41250 [15:59:47<83:23:58,  8.67s/it]
                                                          
{'loss': 0.1896, 'grad_norm': 2.591186285018921, 'learning_rate': 3.8240722566393945e-05, 'epoch': 1.6}

 16%|█▌        | 6620/41250 [15:59:47<83:23:58,  8.67s/it][2025-04-25 23:57:31,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-25 23:57:31,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.11 | bwd_microstep: 5718.51 | bwd_inner_microstep: 5692.68 | bwd_allreduce_microstep: 25.79 | step_microstep: 18.90
[2025-04-25 23:57:31,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.11 | bwd: 5718.53 | bwd_inner: 5692.68 | bwd_allreduce: 25.81 | step: 18.91

 16%|█▌        | 6621/41250 [15:59:56<83:21:55,  8.67s/it]
                                                          
{'loss': 0.0413, 'grad_norm': 1.162328839302063, 'learning_rate': 3.824007850430301e-05, 'epoch': 1.61}

 16%|█▌        | 6621/41250 [15:59:56<83:21:55,  8.67s/it][2025-04-25 23:57:39,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 0.94
[2025-04-25 23:57:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.70 | bwd_microstep: 5756.22 | bwd_inner_microstep: 5682.73 | bwd_allreduce_microstep: 73.44 | step_microstep: 19.28
[2025-04-25 23:57:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.70 | bwd: 5756.23 | bwd_inner: 5682.73 | bwd_allreduce: 73.46 | step: 19.28

 16%|█▌        | 6622/41250 [16:00:05<83:25:36,  8.67s/it]
                                                          
{'loss': 0.0296, 'grad_norm': 0.37405115365982056, 'learning_rate': 3.823943432976558e-05, 'epoch': 1.61}

 16%|█▌        | 6622/41250 [16:00:05<83:25:36,  8.67s/it][2025-04-25 23:57:48,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-25 23:57:48,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.30 | bwd_microstep: 5785.63 | bwd_inner_microstep: 5642.32 | bwd_allreduce_microstep: 143.26 | step_microstep: 18.69
[2025-04-25 23:57:48,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.30 | bwd: 5785.65 | bwd_inner: 5642.32 | bwd_allreduce: 143.28 | step: 18.69

 16%|█▌        | 6623/41250 [16:00:13<83:28:23,  8.68s/it]
                                                          
{'loss': 0.1484, 'grad_norm': 1.347522258758545, 'learning_rate': 3.823879004278562e-05, 'epoch': 1.61}

 16%|█▌        | 6623/41250 [16:00:13<83:28:23,  8.68s/it][2025-04-25 23:57:57,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-25 23:57:57,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.94 | bwd_microstep: 5781.06 | bwd_inner_microstep: 5768.13 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.89
[2025-04-25 23:57:57,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.94 | bwd: 5781.08 | bwd_inner: 5768.13 | bwd_allreduce: 12.90 | step: 18.89

 16%|█▌        | 6624/41250 [16:00:22<83:41:12,  8.70s/it]
                                                          
{'loss': 0.0666, 'grad_norm': 0.7825517058372498, 'learning_rate': 3.8238145643367106e-05, 'epoch': 1.61}

 16%|█▌        | 6624/41250 [16:00:22<83:41:12,  8.70s/it][2025-04-25 23:58:05,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.13 | optimizer_step: 1.03
[2025-04-25 23:58:05,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.43 | bwd_microstep: 5675.09 | bwd_inner_microstep: 5639.77 | bwd_allreduce_microstep: 35.26 | step_microstep: 20.00
[2025-04-25 23:58:05,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.43 | bwd: 5675.11 | bwd_inner: 5639.77 | bwd_allreduce: 35.29 | step: 20.00

 16%|█▌        | 6625/41250 [16:00:31<83:21:28,  8.67s/it]
                                                          
{'loss': 0.114, 'grad_norm': 1.417270302772522, 'learning_rate': 3.823750113151402e-05, 'epoch': 1.61}

 16%|█▌        | 6625/41250 [16:00:31<83:21:28,  8.67s/it][2025-04-25 23:58:14,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-25 23:58:14,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5692.75 | bwd_inner_microstep: 5679.76 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.79
[2025-04-25 23:58:14,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5692.76 | bwd_inner: 5679.76 | bwd_allreduce: 12.96 | step: 18.80

 16%|█▌        | 6626/41250 [16:00:39<83:13:59,  8.65s/it]
                                                          
{'loss': 0.113, 'grad_norm': 1.920913815498352, 'learning_rate': 3.823685650723032e-05, 'epoch': 1.61}

 16%|█▌        | 6626/41250 [16:00:39<83:13:59,  8.65s/it][2025-04-25 23:58:23,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:58:23,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.14 | bwd_microstep: 5749.32 | bwd_inner_microstep: 5642.38 | bwd_allreduce_microstep: 106.89 | step_microstep: 18.88
[2025-04-25 23:58:23,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.14 | bwd: 5749.33 | bwd_inner: 5642.38 | bwd_allreduce: 106.91 | step: 18.88

 16%|█▌        | 6627/41250 [16:00:48<83:13:45,  8.65s/it]
                                                          
{'loss': 0.2446, 'grad_norm': 2.614367961883545, 'learning_rate': 3.823621177051998e-05, 'epoch': 1.61}

 16%|█▌        | 6627/41250 [16:00:48<83:13:45,  8.65s/it][2025-04-25 23:58:31,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:58:31,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5709.34 | bwd_inner_microstep: 5696.63 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.73
[2025-04-25 23:58:31,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5709.36 | bwd_inner: 5696.63 | bwd_allreduce: 12.68 | step: 18.73

 16%|█▌        | 6628/41250 [16:00:57<83:11:29,  8.65s/it]
                                                          
{'loss': 0.2962, 'grad_norm': 3.5909764766693115, 'learning_rate': 3.8235566921386994e-05, 'epoch': 1.61}

 16%|█▌        | 6628/41250 [16:00:57<83:11:29,  8.65s/it][2025-04-25 23:58:40,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-25 23:58:40,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.43 | bwd_microstep: 5745.47 | bwd_inner_microstep: 5673.27 | bwd_allreduce_microstep: 72.16 | step_microstep: 18.72
[2025-04-25 23:58:40,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.43 | bwd: 5745.49 | bwd_inner: 5673.27 | bwd_allreduce: 72.18 | step: 18.72

 16%|█▌        | 6629/41250 [16:01:05<83:14:06,  8.66s/it]
                                                          
{'loss': 0.2906, 'grad_norm': 2.3339309692382812, 'learning_rate': 3.823492195983532e-05, 'epoch': 1.61}

 16%|█▌        | 6629/41250 [16:01:05<83:14:06,  8.66s/it][2025-04-25 23:58:49,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-25 23:58:49,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.17 | bwd_microstep: 5754.51 | bwd_inner_microstep: 5679.58 | bwd_allreduce_microstep: 74.87 | step_microstep: 18.83
[2025-04-25 23:58:49,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.17 | bwd: 5754.52 | bwd_inner: 5679.58 | bwd_allreduce: 74.89 | step: 18.83

 16%|█▌        | 6630/41250 [16:01:14<83:18:49,  8.66s/it]
                                                          
{'loss': 0.1354, 'grad_norm': 1.310814380645752, 'learning_rate': 3.8234276885868935e-05, 'epoch': 1.61}

 16%|█▌        | 6630/41250 [16:01:14<83:18:49,  8.66s/it][2025-04-25 23:58:57,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-25 23:58:57,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.96 | bwd_microstep: 5687.07 | bwd_inner_microstep: 5639.20 | bwd_allreduce_microstep: 47.82 | step_microstep: 19.00
[2025-04-25 23:58:57,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.96 | bwd: 5687.08 | bwd_inner: 5639.20 | bwd_allreduce: 47.84 | step: 19.00

 16%|█▌        | 6631/41250 [16:01:23<83:07:12,  8.64s/it]
                                                          
{'loss': 0.2843, 'grad_norm': 2.245800733566284, 'learning_rate': 3.823363169949182e-05, 'epoch': 1.61}

 16%|█▌        | 6631/41250 [16:01:23<83:07:12,  8.64s/it][2025-04-25 23:59:06,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.09 | optimizer_step: 1.23
[2025-04-25 23:59:06,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5766.48 | bwd_inner_microstep: 5674.93 | bwd_allreduce_microstep: 91.50 | step_microstep: 20.04
[2025-04-25 23:59:06,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5766.50 | bwd_inner: 5674.93 | bwd_allreduce: 91.53 | step: 20.04

 16%|█▌        | 6632/41250 [16:01:31<83:16:01,  8.66s/it]
                                                          
{'loss': 0.4524, 'grad_norm': 4.557259559631348, 'learning_rate': 3.8232986400707956e-05, 'epoch': 1.61}

 16%|█▌        | 6632/41250 [16:01:31<83:16:01,  8.66s/it][2025-04-25 23:59:15,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-25 23:59:15,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.54 | bwd_microstep: 5763.04 | bwd_inner_microstep: 5648.48 | bwd_allreduce_microstep: 114.51 | step_microstep: 19.17
[2025-04-25 23:59:15,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.54 | bwd: 5763.06 | bwd_inner: 5648.48 | bwd_allreduce: 114.53 | step: 19.17

 16%|█▌        | 6633/41250 [16:01:40<83:18:30,  8.66s/it]
                                                          
{'loss': 0.2114, 'grad_norm': 3.5634493827819824, 'learning_rate': 3.823234098952132e-05, 'epoch': 1.61}

 16%|█▌        | 6633/41250 [16:01:40<83:18:30,  8.66s/it][2025-04-25 23:59:23,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-25 23:59:23,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.95 | bwd_microstep: 5745.46 | bwd_inner_microstep: 5688.92 | bwd_allreduce_microstep: 56.50 | step_microstep: 18.97
[2025-04-25 23:59:23,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.95 | bwd: 5745.48 | bwd_inner: 5688.92 | bwd_allreduce: 56.51 | step: 18.98

 16%|█▌        | 6634/41250 [16:01:49<83:20:39,  8.67s/it]
                                                          
{'loss': 0.0479, 'grad_norm': 1.5787333250045776, 'learning_rate': 3.823169546593588e-05, 'epoch': 1.61}

 16%|█▌        | 6634/41250 [16:01:49<83:20:39,  8.67s/it][2025-04-25 23:59:32,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-25 23:59:32,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.91 | bwd_microstep: 5729.81 | bwd_inner_microstep: 5680.01 | bwd_allreduce_microstep: 49.76 | step_microstep: 19.02
[2025-04-25 23:59:32,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.91 | bwd: 5729.83 | bwd_inner: 5680.01 | bwd_allreduce: 49.77 | step: 19.03

 16%|█▌        | 6635/41250 [16:01:57<83:18:35,  8.66s/it]
                                                          
{'loss': 0.0936, 'grad_norm': 1.040139079093933, 'learning_rate': 3.8231049829955626e-05, 'epoch': 1.61}

 16%|█▌        | 6635/41250 [16:01:57<83:18:35,  8.66s/it][2025-04-25 23:59:41,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-25 23:59:41,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.08 | bwd_microstep: 5735.09 | bwd_inner_microstep: 5708.73 | bwd_allreduce_microstep: 26.32 | step_microstep: 18.92
[2025-04-25 23:59:41,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.08 | bwd: 5735.11 | bwd_inner: 5708.73 | bwd_allreduce: 26.34 | step: 18.92

 16%|█▌        | 6636/41250 [16:02:06<83:18:57,  8.67s/it]
                                                          
{'loss': 0.0611, 'grad_norm': 1.1488533020019531, 'learning_rate': 3.8230404081584536e-05, 'epoch': 1.61}

 16%|█▌        | 6636/41250 [16:02:06<83:18:57,  8.67s/it][2025-04-25 23:59:49,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.23 | optimizer_step: 0.93
[2025-04-25 23:59:49,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.63 | bwd_microstep: 5746.17 | bwd_inner_microstep: 5647.26 | bwd_allreduce_microstep: 98.87 | step_microstep: 19.17
[2025-04-25 23:59:49,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.63 | bwd: 5746.19 | bwd_inner: 5647.26 | bwd_allreduce: 98.88 | step: 19.18

 16%|█▌        | 6637/41250 [16:02:15<83:16:34,  8.66s/it]
                                                          
{'loss': 0.0626, 'grad_norm': 1.0964891910552979, 'learning_rate': 3.8229758220826595e-05, 'epoch': 1.61}

 16%|█▌        | 6637/41250 [16:02:15<83:16:34,  8.66s/it][2025-04-25 23:59:58,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-25 23:59:58,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5711.94 | bwd_inner_microstep: 5651.80 | bwd_allreduce_microstep: 60.09 | step_microstep: 19.16
[2025-04-25 23:59:58,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.10 | bwd: 5711.96 | bwd_inner: 5651.80 | bwd_allreduce: 60.11 | step: 19.17

 16%|█▌        | 6638/41250 [16:02:23<83:09:26,  8.65s/it]
                                                          
{'loss': 0.0906, 'grad_norm': 1.7225006818771362, 'learning_rate': 3.822911224768577e-05, 'epoch': 1.61}

 16%|█▌        | 6638/41250 [16:02:23<83:09:26,  8.65s/it][2025-04-26 00:00:07,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:00:07,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.87 | bwd_microstep: 5698.33 | bwd_inner_microstep: 5642.84 | bwd_allreduce_microstep: 55.44 | step_microstep: 18.78
[2025-04-26 00:00:07,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.87 | bwd: 5698.35 | bwd_inner: 5642.84 | bwd_allreduce: 55.46 | step: 18.78

 16%|█▌        | 6639/41250 [16:02:32<83:01:34,  8.64s/it]
                                                          
{'loss': 0.1057, 'grad_norm': 3.07773494720459, 'learning_rate': 3.8228466162166066e-05, 'epoch': 1.61}

 16%|█▌        | 6639/41250 [16:02:32<83:01:34,  8.64s/it][2025-04-26 00:00:15,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 00:00:15,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.70 | bwd_microstep: 5775.32 | bwd_inner_microstep: 5651.61 | bwd_allreduce_microstep: 123.67 | step_microstep: 19.52
[2025-04-26 00:00:15,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.70 | bwd: 5775.34 | bwd_inner: 5651.61 | bwd_allreduce: 123.69 | step: 19.52

 16%|█▌        | 6640/41250 [16:02:41<83:09:36,  8.65s/it]
                                                          
{'loss': 0.0728, 'grad_norm': 1.3548004627227783, 'learning_rate': 3.822781996427144e-05, 'epoch': 1.61}

 16%|█▌        | 6640/41250 [16:02:41<83:09:36,  8.65s/it][2025-04-26 00:00:24,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:00:24,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.63 | bwd_microstep: 5705.02 | bwd_inner_microstep: 5652.65 | bwd_allreduce_microstep: 52.32 | step_microstep: 18.57
[2025-04-26 00:00:24,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.63 | bwd: 5705.03 | bwd_inner: 5652.65 | bwd_allreduce: 52.34 | step: 18.57

 16%|█▌        | 6641/41250 [16:02:49<83:03:43,  8.64s/it]
                                                          
{'loss': 0.1057, 'grad_norm': 1.8960678577423096, 'learning_rate': 3.82271736540059e-05, 'epoch': 1.61}

 16%|█▌        | 6641/41250 [16:02:49<83:03:43,  8.64s/it][2025-04-26 00:00:33,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 00:00:33,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.69 | bwd_microstep: 6058.05 | bwd_inner_microstep: 5650.10 | bwd_allreduce_microstep: 407.90 | step_microstep: 19.05
[2025-04-26 00:00:33,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.69 | bwd: 6058.06 | bwd_inner: 5650.10 | bwd_allreduce: 407.92 | step: 19.05

 16%|█▌        | 6642/41250 [16:02:58<83:59:58,  8.74s/it]
                                                          
{'loss': 0.2397, 'grad_norm': 5.403155326843262, 'learning_rate': 3.822652723137341e-05, 'epoch': 1.61}

 16%|█▌        | 6642/41250 [16:02:58<83:59:58,  8.74s/it][2025-04-26 00:00:41,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 00:00:41,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.40 | bwd_microstep: 5701.73 | bwd_inner_microstep: 5688.88 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.90
[2025-04-26 00:00:41,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.40 | bwd: 5701.74 | bwd_inner: 5688.88 | bwd_allreduce: 12.82 | step: 18.90

 16%|█▌        | 6643/41250 [16:03:07<83:40:39,  8.70s/it]
                                                          
{'loss': 0.2323, 'grad_norm': 2.1131227016448975, 'learning_rate': 3.8225880696377975e-05, 'epoch': 1.61}

 16%|█▌        | 6643/41250 [16:03:07<83:40:39,  8.70s/it][2025-04-26 00:00:50,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 00:00:50,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5715.59 | bwd_inner_microstep: 5702.82 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.96
[2025-04-26 00:00:50,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5715.60 | bwd_inner: 5702.82 | bwd_allreduce: 12.74 | step: 18.97

 16%|█▌        | 6644/41250 [16:03:15<83:30:53,  8.69s/it]
                                                          
{'loss': 0.1371, 'grad_norm': 1.9538136720657349, 'learning_rate': 3.8225234049023564e-05, 'epoch': 1.61}

 16%|█▌        | 6644/41250 [16:03:15<83:30:53,  8.69s/it][2025-04-26 00:00:59,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:00:59,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.53 | bwd_microstep: 5765.42 | bwd_inner_microstep: 5696.30 | bwd_allreduce_microstep: 69.07 | step_microstep: 18.95
[2025-04-26 00:00:59,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.53 | bwd: 5765.43 | bwd_inner: 5696.30 | bwd_allreduce: 69.09 | step: 18.95

 16%|█▌        | 6645/41250 [16:03:24<83:31:46,  8.69s/it]
                                                          
{'loss': 0.0621, 'grad_norm': 2.696687936782837, 'learning_rate': 3.822458728931417e-05, 'epoch': 1.61}

 16%|█▌        | 6645/41250 [16:03:24<83:31:46,  8.69s/it][2025-04-26 00:01:08,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 00:01:08,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.92 | bwd_microstep: 5779.10 | bwd_inner_microstep: 5657.86 | bwd_allreduce_microstep: 121.19 | step_microstep: 19.30
[2025-04-26 00:01:08,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.92 | bwd: 5779.11 | bwd_inner: 5657.86 | bwd_allreduce: 121.21 | step: 19.31

 16%|█▌        | 6646/41250 [16:03:33<83:32:24,  8.69s/it]
                                                          
{'loss': 0.1482, 'grad_norm': 2.1756088733673096, 'learning_rate': 3.8223940417253784e-05, 'epoch': 1.61}

 16%|█▌        | 6646/41250 [16:03:33<83:32:24,  8.69s/it][2025-04-26 00:01:16,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:01:16,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5763.73 | bwd_inner_microstep: 5711.65 | bwd_allreduce_microstep: 52.03 | step_microstep: 18.88
[2025-04-26 00:01:16,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5763.74 | bwd_inner: 5711.65 | bwd_allreduce: 52.05 | step: 18.89

 16%|█▌        | 6647/41250 [16:03:42<83:33:13,  8.69s/it]
                                                          
{'loss': 0.047, 'grad_norm': 0.9030144810676575, 'learning_rate': 3.8223293432846384e-05, 'epoch': 1.61}

 16%|█▌        | 6647/41250 [16:03:42<83:33:13,  8.69s/it][2025-04-26 00:01:25,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:01:25,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5871.55 | bwd_inner_microstep: 5653.17 | bwd_allreduce_microstep: 218.33 | step_microstep: 18.43
[2025-04-26 00:01:25,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5871.57 | bwd_inner: 5653.17 | bwd_allreduce: 218.35 | step: 18.43

 16%|█▌        | 6648/41250 [16:03:50<83:48:47,  8.72s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.4922209978103638, 'learning_rate': 3.8222646336095966e-05, 'epoch': 1.61}

 16%|█▌        | 6648/41250 [16:03:50<83:48:47,  8.72s/it][2025-04-26 00:01:34,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 00:01:34,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.44 | bwd_microstep: 5796.01 | bwd_inner_microstep: 5783.32 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.57
[2025-04-26 00:01:34,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.44 | bwd: 5796.02 | bwd_inner: 5783.32 | bwd_allreduce: 12.66 | step: 18.58

 16%|█▌        | 6649/41250 [16:03:59<83:57:52,  8.74s/it]
                                                          
{'loss': 0.211, 'grad_norm': 2.120043992996216, 'learning_rate': 3.8221999127006524e-05, 'epoch': 1.61}

 16%|█▌        | 6649/41250 [16:03:59<83:57:52,  8.74s/it][2025-04-26 00:01:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 00:01:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.43 | bwd_microstep: 5714.51 | bwd_inner_microstep: 5701.80 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.06
[2025-04-26 00:01:42,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.43 | bwd: 5714.52 | bwd_inner: 5701.80 | bwd_allreduce: 12.69 | step: 19.06

 16%|█▌        | 6650/41250 [16:04:08<83:43:38,  8.71s/it]
                                                          
{'loss': 0.183, 'grad_norm': 5.59145450592041, 'learning_rate': 3.8221351805582034e-05, 'epoch': 1.61}

 16%|█▌        | 6650/41250 [16:04:08<83:43:38,  8.71s/it][2025-04-26 00:01:51,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 00:01:51,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.04 | bwd_microstep: 5717.78 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 60.18 | step_microstep: 18.43
[2025-04-26 00:01:51,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.04 | bwd: 5717.79 | bwd_inner: 5657.55 | bwd_allreduce: 60.20 | step: 18.43

 16%|█▌        | 6651/41250 [16:04:16<83:29:39,  8.69s/it]
                                                          
{'loss': 0.2461, 'grad_norm': 3.6803245544433594, 'learning_rate': 3.8220704371826494e-05, 'epoch': 1.61}

 16%|█▌        | 6651/41250 [16:04:16<83:29:39,  8.69s/it][2025-04-26 00:02:00,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.07 | optimizer_step: 1.02
[2025-04-26 00:02:00,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.32 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5666.00 | bwd_allreduce_microstep: 46.86 | step_microstep: 19.54
[2025-04-26 00:02:00,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.32 | bwd: 5712.93 | bwd_inner: 5666.00 | bwd_allreduce: 46.88 | step: 19.54

 16%|█▌        | 6652/41250 [16:04:25<83:19:56,  8.67s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 1.0214706659317017, 'learning_rate': 3.82200568257439e-05, 'epoch': 1.61}

 16%|█▌        | 6652/41250 [16:04:25<83:19:56,  8.67s/it][2025-04-26 00:02:08,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 00:02:08,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.84 | bwd_microstep: 5788.10 | bwd_inner_microstep: 5775.36 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.76
[2025-04-26 00:02:08,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.84 | bwd: 5788.12 | bwd_inner: 5775.36 | bwd_allreduce: 12.72 | step: 18.76

 16%|█▌        | 6653/41250 [16:04:34<83:35:37,  8.70s/it]
                                                          
{'loss': 0.0549, 'grad_norm': 1.0663636922836304, 'learning_rate': 3.821940916733824e-05, 'epoch': 1.61}

 16%|█▌        | 6653/41250 [16:04:34<83:35:37,  8.70s/it][2025-04-26 00:02:17,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 00:02:17,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.31 | bwd_microstep: 5722.62 | bwd_inner_microstep: 5709.50 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.04
[2025-04-26 00:02:17,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.31 | bwd: 5722.63 | bwd_inner: 5709.50 | bwd_allreduce: 13.09 | step: 19.04

 16%|█▌        | 6654/41250 [16:04:42<83:31:03,  8.69s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 0.9680225253105164, 'learning_rate': 3.82187613966135e-05, 'epoch': 1.61}

 16%|█▌        | 6654/41250 [16:04:42<83:31:03,  8.69s/it][2025-04-26 00:02:26,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:02:26,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.58 | bwd_microstep: 5780.79 | bwd_inner_microstep: 5693.44 | bwd_allreduce_microstep: 87.30 | step_microstep: 18.43
[2025-04-26 00:02:26,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.58 | bwd: 5780.80 | bwd_inner: 5693.44 | bwd_allreduce: 87.31 | step: 18.43

 16%|█▌        | 6655/41250 [16:04:51<83:35:54,  8.70s/it]
                                                          
{'loss': 0.1359, 'grad_norm': 3.1550045013427734, 'learning_rate': 3.821811351357368e-05, 'epoch': 1.61}

 16%|█▌        | 6655/41250 [16:04:51<83:35:54,  8.70s/it][2025-04-26 00:02:34,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.28 | optimizer_step: 1.05
[2025-04-26 00:02:34,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.64 | bwd_microstep: 5717.14 | bwd_inner_microstep: 5649.09 | bwd_allreduce_microstep: 67.99 | step_microstep: 20.08
[2025-04-26 00:02:34,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.64 | bwd: 5717.15 | bwd_inner: 5649.09 | bwd_allreduce: 68.02 | step: 20.08

 16%|█▌        | 6656/41250 [16:05:00<83:24:40,  8.68s/it]
                                                          
{'loss': 0.2111, 'grad_norm': 1.4830327033996582, 'learning_rate': 3.821746551822278e-05, 'epoch': 1.61}

 16%|█▌        | 6656/41250 [16:05:00<83:24:40,  8.68s/it][2025-04-26 00:02:43,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 00:02:43,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.49 | bwd_microstep: 5711.83 | bwd_inner_microstep: 5690.96 | bwd_allreduce_microstep: 20.82 | step_microstep: 19.06
[2025-04-26 00:02:43,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.49 | bwd: 5711.84 | bwd_inner: 5690.96 | bwd_allreduce: 20.84 | step: 19.07

 16%|█▌        | 6657/41250 [16:05:08<83:17:29,  8.67s/it]
                                                          
{'loss': 0.0254, 'grad_norm': 0.436768114566803, 'learning_rate': 3.821681741056479e-05, 'epoch': 1.61}

 16%|█▌        | 6657/41250 [16:05:08<83:17:29,  8.67s/it][2025-04-26 00:02:52,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 00:02:52,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5690.08 | bwd_inner_microstep: 5663.23 | bwd_allreduce_microstep: 26.78 | step_microstep: 19.08
[2025-04-26 00:02:52,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5690.10 | bwd_inner: 5663.23 | bwd_allreduce: 26.81 | step: 19.09

 16%|█▌        | 6658/41250 [16:05:17<83:06:12,  8.65s/it]
                                                          
{'loss': 0.3518, 'grad_norm': 2.0826621055603027, 'learning_rate': 3.8216169190603694e-05, 'epoch': 1.61}

 16%|█▌        | 6658/41250 [16:05:17<83:06:12,  8.65s/it][2025-04-26 00:03:00,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:03:00,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.34 | bwd_microstep: 5693.79 | bwd_inner_microstep: 5681.01 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.60
[2025-04-26 00:03:00,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.35 | bwd: 5693.80 | bwd_inner: 5681.01 | bwd_allreduce: 12.75 | step: 18.61

 16%|█▌        | 6659/41250 [16:05:26<83:01:07,  8.64s/it]
                                                          
{'loss': 0.0643, 'grad_norm': 1.272721529006958, 'learning_rate': 3.82155208583435e-05, 'epoch': 1.61}

 16%|█▌        | 6659/41250 [16:05:26<83:01:07,  8.64s/it][2025-04-26 00:03:09,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 00:03:09,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.40 | bwd_microstep: 5763.97 | bwd_inner_microstep: 5685.93 | bwd_allreduce_microstep: 77.99 | step_microstep: 18.72
[2025-04-26 00:03:09,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.40 | bwd: 5763.99 | bwd_inner: 5685.93 | bwd_allreduce: 78.01 | step: 18.72

 16%|█▌        | 6660/41250 [16:05:34<83:09:38,  8.66s/it]
                                                          
{'loss': 0.1464, 'grad_norm': 2.5112664699554443, 'learning_rate': 3.821487241378821e-05, 'epoch': 1.61}

 16%|█▌        | 6660/41250 [16:05:34<83:09:38,  8.66s/it][2025-04-26 00:03:18,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 00:03:18,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.02 | bwd_microstep: 5712.93 | bwd_inner_microstep: 5653.27 | bwd_allreduce_microstep: 59.58 | step_microstep: 18.70
[2025-04-26 00:03:18,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.02 | bwd: 5712.95 | bwd_inner: 5653.27 | bwd_allreduce: 59.63 | step: 18.71

 16%|█▌        | 6661/41250 [16:05:43<83:04:14,  8.65s/it]
                                                          
{'loss': 0.2279, 'grad_norm': 2.236743688583374, 'learning_rate': 3.82142238569418e-05, 'epoch': 1.61}

 16%|█▌        | 6661/41250 [16:05:43<83:04:14,  8.65s/it][2025-04-26 00:03:26,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.29 | optimizer_step: 1.06
[2025-04-26 00:03:26,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.86 | bwd_microstep: 5765.44 | bwd_inner_microstep: 5684.43 | bwd_allreduce_microstep: 80.95 | step_microstep: 20.08
[2025-04-26 00:03:26,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.86 | bwd: 5765.46 | bwd_inner: 5684.43 | bwd_allreduce: 80.98 | step: 20.08

 16%|█▌        | 6662/41250 [16:05:52<83:12:10,  8.66s/it]
                                                          
{'loss': 0.1332, 'grad_norm': 1.0822944641113281, 'learning_rate': 3.821357518780829e-05, 'epoch': 1.62}

 16%|█▌        | 6662/41250 [16:05:52<83:12:10,  8.66s/it][2025-04-26 00:03:35,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 00:03:35,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5710.67 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.89
[2025-04-26 00:03:35,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5710.68 | bwd_inner: 5697.91 | bwd_allreduce: 12.73 | step: 18.89

 16%|█▌        | 6663/41250 [16:06:00<83:09:06,  8.65s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 2.2827467918395996, 'learning_rate': 3.8212926406391676e-05, 'epoch': 1.62}

 16%|█▌        | 6663/41250 [16:06:00<83:09:06,  8.65s/it][2025-04-26 00:03:44,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:03:44,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.32 | bwd_microstep: 5694.18 | bwd_inner_microstep: 5681.52 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.71
[2025-04-26 00:03:44,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.32 | bwd: 5694.20 | bwd_inner: 5681.52 | bwd_allreduce: 12.64 | step: 18.71

 16%|█▌        | 6664/41250 [16:06:09<83:02:06,  8.64s/it]
                                                          
{'loss': 0.1088, 'grad_norm': 1.8587123155593872, 'learning_rate': 3.8212277512695955e-05, 'epoch': 1.62}

 16%|█▌        | 6664/41250 [16:06:09<83:02:06,  8.64s/it][2025-04-26 00:03:52,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 00:03:52,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5748.24 | bwd_inner_microstep: 5697.71 | bwd_allreduce_microstep: 50.49 | step_microstep: 18.48
[2025-04-26 00:03:52,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5748.26 | bwd_inner: 5697.71 | bwd_allreduce: 50.50 | step: 18.48

 16%|█▌        | 6665/41250 [16:06:18<83:08:39,  8.65s/it]
                                                          
{'loss': 0.0974, 'grad_norm': 1.7141846418380737, 'learning_rate': 3.8211628506725116e-05, 'epoch': 1.62}

 16%|█▌        | 6665/41250 [16:06:18<83:08:39,  8.65s/it][2025-04-26 00:04:01,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:04:01,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.00 | bwd_microstep: 5863.10 | bwd_inner_microstep: 5692.31 | bwd_allreduce_microstep: 170.75 | step_microstep: 18.83
[2025-04-26 00:04:01,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.00 | bwd: 5863.12 | bwd_inner: 5692.31 | bwd_allreduce: 170.76 | step: 18.83

 16%|█▌        | 6666/41250 [16:06:26<83:32:24,  8.70s/it]
                                                          
{'loss': 0.0784, 'grad_norm': 0.9671879410743713, 'learning_rate': 3.821097938848318e-05, 'epoch': 1.62}

 16%|█▌        | 6666/41250 [16:06:26<83:32:24,  8.70s/it][2025-04-26 00:04:10,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 00:04:10,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.70 | bwd_microstep: 5690.50 | bwd_inner_microstep: 5644.62 | bwd_allreduce_microstep: 45.83 | step_microstep: 19.35
[2025-04-26 00:04:10,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.70 | bwd: 5690.51 | bwd_inner: 5644.62 | bwd_allreduce: 45.85 | step: 19.35

 16%|█▌        | 6667/41250 [16:06:35<83:16:33,  8.67s/it]
                                                          
{'loss': 0.2221, 'grad_norm': 3.1682846546173096, 'learning_rate': 3.821033015797412e-05, 'epoch': 1.62}

 16%|█▌        | 6667/41250 [16:06:35<83:16:33,  8.67s/it][2025-04-26 00:04:18,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 00:04:18,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.00 | bwd_microstep: 5762.82 | bwd_inner_microstep: 5749.99 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.97
[2025-04-26 00:04:18,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.00 | bwd: 5762.83 | bwd_inner: 5749.99 | bwd_allreduce: 12.80 | step: 18.97

 16%|█▌        | 6668/41250 [16:06:44<83:25:36,  8.68s/it]
                                                          
{'loss': 0.0717, 'grad_norm': 0.8185484409332275, 'learning_rate': 3.820968081520197e-05, 'epoch': 1.62}

 16%|█▌        | 6668/41250 [16:06:44<83:25:36,  8.68s/it][2025-04-26 00:04:27,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 00:04:27,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5699.42 | bwd_inner_microstep: 5686.39 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.28
[2025-04-26 00:04:27,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5699.43 | bwd_inner: 5686.39 | bwd_allreduce: 13.00 | step: 19.28

 16%|█▌        | 6669/41250 [16:06:52<83:16:16,  8.67s/it]
                                                          
{'loss': 0.039, 'grad_norm': 0.6870862245559692, 'learning_rate': 3.820903136017072e-05, 'epoch': 1.62}

 16%|█▌        | 6669/41250 [16:06:52<83:16:16,  8.67s/it][2025-04-26 00:04:36,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 00:04:36,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5726.35 | bwd_inner_microstep: 5699.33 | bwd_allreduce_microstep: 26.97 | step_microstep: 19.49
[2025-04-26 00:04:36,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5726.37 | bwd_inner: 5699.33 | bwd_allreduce: 26.99 | step: 19.49

 16%|█▌        | 6670/41250 [16:07:01<83:14:27,  8.67s/it]
                                                          
{'loss': 0.1448, 'grad_norm': 2.4065937995910645, 'learning_rate': 3.8208381792884374e-05, 'epoch': 1.62}

 16%|█▌        | 6670/41250 [16:07:01<83:14:27,  8.67s/it][2025-04-26 00:04:44,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 00:04:44,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5718.96 | bwd_inner_microstep: 5698.69 | bwd_allreduce_microstep: 20.23 | step_microstep: 18.57
[2025-04-26 00:04:44,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5718.97 | bwd_inner: 5698.68 | bwd_allreduce: 20.25 | step: 18.57

 16%|█▌        | 6671/41250 [16:07:10<83:12:41,  8.66s/it]
                                                          
{'loss': 0.1321, 'grad_norm': 2.709787368774414, 'learning_rate': 3.8207732113346924e-05, 'epoch': 1.62}

 16%|█▌        | 6671/41250 [16:07:10<83:12:41,  8.66s/it][2025-04-26 00:04:53,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:04:53,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.36 | bwd_microstep: 5785.90 | bwd_inner_microstep: 5773.12 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.90
[2025-04-26 00:04:53,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.36 | bwd: 5785.92 | bwd_inner: 5773.12 | bwd_allreduce: 12.75 | step: 18.91

 16%|█▌        | 6672/41250 [16:07:18<83:29:18,  8.69s/it]
                                                          
{'loss': 0.1476, 'grad_norm': 2.473151683807373, 'learning_rate': 3.8207082321562404e-05, 'epoch': 1.62}

 16%|█▌        | 6672/41250 [16:07:18<83:29:18,  8.69s/it][2025-04-26 00:05:02,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:05:02,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.40 | bwd_microstep: 5694.74 | bwd_inner_microstep: 5642.30 | bwd_allreduce_microstep: 52.39 | step_microstep: 18.83
[2025-04-26 00:05:02,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.40 | bwd: 5694.75 | bwd_inner: 5642.30 | bwd_allreduce: 52.41 | step: 18.83

 16%|█▌        | 6673/41250 [16:07:27<83:13:05,  8.66s/it]
                                                          
{'loss': 0.2794, 'grad_norm': 2.278064250946045, 'learning_rate': 3.820643241753478e-05, 'epoch': 1.62}

 16%|█▌        | 6673/41250 [16:07:27<83:13:05,  8.66s/it][2025-04-26 00:05:10,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 00:05:10,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.34 | bwd_microstep: 5723.47 | bwd_inner_microstep: 5679.05 | bwd_allreduce_microstep: 44.36 | step_microstep: 18.47
[2025-04-26 00:05:10,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.34 | bwd: 5723.48 | bwd_inner: 5679.05 | bwd_allreduce: 44.38 | step: 18.47

 16%|█▌        | 6674/41250 [16:07:36<83:11:48,  8.66s/it]
                                                          
{'loss': 0.3527, 'grad_norm': 2.875328302383423, 'learning_rate': 3.82057824012681e-05, 'epoch': 1.62}

 16%|█▌        | 6674/41250 [16:07:36<83:11:48,  8.66s/it][2025-04-26 00:05:19,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 00:05:19,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.69 | bwd_microstep: 5749.48 | bwd_inner_microstep: 5690.32 | bwd_allreduce_microstep: 59.11 | step_microstep: 18.86
[2025-04-26 00:05:19,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.69 | bwd: 5749.49 | bwd_inner: 5690.32 | bwd_allreduce: 59.13 | step: 18.86

 16%|█▌        | 6675/41250 [16:07:44<83:14:06,  8.67s/it]
                                                          
{'loss': 0.2012, 'grad_norm': 2.0642318725585938, 'learning_rate': 3.8205132272766346e-05, 'epoch': 1.62}

 16%|█▌        | 6675/41250 [16:07:44<83:14:06,  8.67s/it][2025-04-26 00:05:28,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.29 | optimizer_step: 1.06
[2025-04-26 00:05:28,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.73 | bwd_microstep: 5748.60 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 57.69 | step_microstep: 19.93
[2025-04-26 00:05:28,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.73 | bwd: 5748.62 | bwd_inner: 5690.85 | bwd_allreduce: 57.71 | step: 19.93

 16%|█▌        | 6676/41250 [16:07:53<83:17:30,  8.67s/it]
                                                          
{'loss': 0.0301, 'grad_norm': 1.0690184831619263, 'learning_rate': 3.820448203203354e-05, 'epoch': 1.62}

 16%|█▌        | 6676/41250 [16:07:53<83:17:30,  8.67s/it][2025-04-26 00:05:36,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 00:05:36,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.57 | bwd_microstep: 5753.46 | bwd_inner_microstep: 5645.63 | bwd_allreduce_microstep: 107.78 | step_microstep: 18.71
[2025-04-26 00:05:36,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.57 | bwd: 5753.47 | bwd_inner: 5645.63 | bwd_allreduce: 107.80 | step: 18.71

 16%|█▌        | 6677/41250 [16:08:02<83:14:41,  8.67s/it]
                                                          
{'loss': 0.1418, 'grad_norm': 1.0023994445800781, 'learning_rate': 3.820383167907367e-05, 'epoch': 1.62}

 16%|█▌        | 6677/41250 [16:08:02<83:14:41,  8.67s/it][2025-04-26 00:05:45,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-26 00:05:45,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.14 | bwd_microstep: 5754.81 | bwd_inner_microstep: 5651.80 | bwd_allreduce_microstep: 102.96 | step_microstep: 19.21
[2025-04-26 00:05:45,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.14 | bwd: 5754.82 | bwd_inner: 5651.79 | bwd_allreduce: 102.98 | step: 19.21

 16%|█▌        | 6678/41250 [16:08:10<83:13:48,  8.67s/it]
                                                          
{'loss': 0.2184, 'grad_norm': 1.9411851167678833, 'learning_rate': 3.820318121389076e-05, 'epoch': 1.62}

 16%|█▌        | 6678/41250 [16:08:10<83:13:48,  8.67s/it][2025-04-26 00:05:54,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 00:05:54,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.23 | bwd_microstep: 5695.51 | bwd_inner_microstep: 5650.23 | bwd_allreduce_microstep: 45.23 | step_microstep: 19.24
[2025-04-26 00:05:54,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.22 | bwd: 5695.52 | bwd_inner: 5650.23 | bwd_allreduce: 45.24 | step: 19.24

 16%|█▌        | 6679/41250 [16:08:19<83:03:23,  8.65s/it]
                                                          
{'loss': 0.1, 'grad_norm': 2.002990484237671, 'learning_rate': 3.820253063648883e-05, 'epoch': 1.62}

 16%|█▌        | 6679/41250 [16:08:19<83:03:23,  8.65s/it][2025-04-26 00:06:02,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:06:02,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.54 | bwd_microstep: 5691.09 | bwd_inner_microstep: 5651.88 | bwd_allreduce_microstep: 39.16 | step_microstep: 18.24
[2025-04-26 00:06:02,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.54 | bwd: 5691.11 | bwd_inner: 5651.88 | bwd_allreduce: 39.18 | step: 18.24

 16%|█▌        | 6680/41250 [16:08:28<82:54:45,  8.63s/it]
                                                          
{'loss': 0.0791, 'grad_norm': 2.7767210006713867, 'learning_rate': 3.820187994687187e-05, 'epoch': 1.62}

 16%|█▌        | 6680/41250 [16:08:28<82:54:45,  8.63s/it][2025-04-26 00:06:11,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:06:11,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.02 | bwd_microstep: 5749.94 | bwd_inner_microstep: 5653.19 | bwd_allreduce_microstep: 96.70 | step_microstep: 18.66
[2025-04-26 00:06:11,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.02 | bwd: 5749.95 | bwd_inner: 5653.19 | bwd_allreduce: 96.72 | step: 18.66

 16%|█▌        | 6681/41250 [16:08:36<82:58:12,  8.64s/it]
                                                          
{'loss': 0.1806, 'grad_norm': 1.783096194267273, 'learning_rate': 3.8201229145043905e-05, 'epoch': 1.62}

 16%|█▌        | 6681/41250 [16:08:36<82:58:12,  8.64s/it][2025-04-26 00:06:20,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 00:06:20,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.08 | bwd_microstep: 5697.04 | bwd_inner_microstep: 5640.82 | bwd_allreduce_microstep: 56.18 | step_microstep: 18.87
[2025-04-26 00:06:20,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.08 | bwd: 5697.06 | bwd_inner: 5640.82 | bwd_allreduce: 56.20 | step: 18.87

 16%|█▌        | 6682/41250 [16:08:45<82:52:27,  8.63s/it]
                                                          
{'loss': 0.0895, 'grad_norm': 1.1127570867538452, 'learning_rate': 3.820057823100894e-05, 'epoch': 1.62}

 16%|█▌        | 6682/41250 [16:08:45<82:52:27,  8.63s/it][2025-04-26 00:06:28,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:06:28,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.79 | bwd_microstep: 5706.97 | bwd_inner_microstep: 5693.69 | bwd_allreduce_microstep: 13.23 | step_microstep: 18.40
[2025-04-26 00:06:28,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.79 | bwd: 5706.99 | bwd_inner: 5693.69 | bwd_allreduce: 13.25 | step: 18.40

 16%|█▌        | 6683/41250 [16:08:54<82:54:30,  8.63s/it]
                                                          
{'loss': 0.2184, 'grad_norm': 3.4070687294006348, 'learning_rate': 3.819992720477098e-05, 'epoch': 1.62}

 16%|█▌        | 6683/41250 [16:08:54<82:54:30,  8.63s/it][2025-04-26 00:06:37,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 1.02
[2025-04-26 00:06:37,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.43 | bwd_microstep: 5791.34 | bwd_inner_microstep: 5651.86 | bwd_allreduce_microstep: 139.44 | step_microstep: 19.64
[2025-04-26 00:06:37,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.43 | bwd: 5791.35 | bwd_inner: 5651.86 | bwd_allreduce: 139.46 | step: 19.64

 16%|█▌        | 6684/41250 [16:09:02<83:06:40,  8.66s/it]
                                                          
{'loss': 0.1062, 'grad_norm': 2.2886712551116943, 'learning_rate': 3.819927606633406e-05, 'epoch': 1.62}

 16%|█▌        | 6684/41250 [16:09:02<83:06:40,  8.66s/it][2025-04-26 00:06:46,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:06:46,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.65 | bwd_microstep: 5714.88 | bwd_inner_microstep: 5678.55 | bwd_allreduce_microstep: 36.28 | step_microstep: 18.50
[2025-04-26 00:06:46,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.65 | bwd: 5714.89 | bwd_inner: 5678.55 | bwd_allreduce: 36.30 | step: 18.50

 16%|█▌        | 6685/41250 [16:09:11<83:03:43,  8.65s/it]
                                                          
{'loss': 0.1198, 'grad_norm': 2.6281850337982178, 'learning_rate': 3.8198624815702184e-05, 'epoch': 1.62}

 16%|█▌        | 6685/41250 [16:09:11<83:03:43,  8.65s/it][2025-04-26 00:06:54,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:06:54,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5775.80 | bwd_inner_microstep: 5653.70 | bwd_allreduce_microstep: 122.06 | step_microstep: 18.58
[2025-04-26 00:06:54,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5775.82 | bwd_inner: 5653.70 | bwd_allreduce: 122.07 | step: 18.58

 16%|█▌        | 6686/41250 [16:09:20<83:09:37,  8.66s/it]
                                                          
{'loss': 0.1559, 'grad_norm': 1.4559088945388794, 'learning_rate': 3.819797345287936e-05, 'epoch': 1.62}

 16%|█▌        | 6686/41250 [16:09:20<83:09:37,  8.66s/it][2025-04-26 00:07:03,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:07:03,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.17 | bwd_microstep: 5770.62 | bwd_inner_microstep: 5649.32 | bwd_allreduce_microstep: 121.25 | step_microstep: 18.45
[2025-04-26 00:07:03,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.17 | bwd: 5770.63 | bwd_inner: 5649.32 | bwd_allreduce: 121.27 | step: 18.47

 16%|█▌        | 6687/41250 [16:09:28<83:12:54,  8.67s/it]
                                                          
{'loss': 0.123, 'grad_norm': 1.9347221851348877, 'learning_rate': 3.8197321977869605e-05, 'epoch': 1.62}

 16%|█▌        | 6687/41250 [16:09:28<83:12:54,  8.67s/it][2025-04-26 00:07:12,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:07:12,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5784.86 | bwd_inner_microstep: 5650.39 | bwd_allreduce_microstep: 134.42 | step_microstep: 18.66
[2025-04-26 00:07:12,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5784.87 | bwd_inner: 5650.39 | bwd_allreduce: 134.44 | step: 18.67

 16%|█▌        | 6688/41250 [16:09:37<83:17:46,  8.68s/it]
                                                          
{'loss': 0.0919, 'grad_norm': 1.4027963876724243, 'learning_rate': 3.819667039067695e-05, 'epoch': 1.62}

 16%|█▌        | 6688/41250 [16:09:37<83:17:46,  8.68s/it][2025-04-26 00:07:20,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:07:20,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.99 | bwd_microstep: 5722.18 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 64.29 | step_microstep: 18.48
[2025-04-26 00:07:20,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.99 | bwd: 5722.19 | bwd_inner: 5657.84 | bwd_allreduce: 64.31 | step: 18.48

 16%|█▌        | 6689/41250 [16:09:46<83:11:35,  8.67s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 1.2794920206069946, 'learning_rate': 3.819601869130539e-05, 'epoch': 1.62}

 16%|█▌        | 6689/41250 [16:09:46<83:11:35,  8.67s/it][2025-04-26 00:07:29,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 00:07:29,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.50 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5693.88 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.81
[2025-04-26 00:07:29,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.50 | bwd: 5706.79 | bwd_inner: 5693.87 | bwd_allreduce: 12.87 | step: 18.81

 16%|█▌        | 6690/41250 [16:09:54<83:08:21,  8.66s/it]
                                                          
{'loss': 0.113, 'grad_norm': 1.7271478176116943, 'learning_rate': 3.8195366879758964e-05, 'epoch': 1.62}

 16%|█▌        | 6690/41250 [16:09:54<83:08:21,  8.66s/it][2025-04-26 00:07:38,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 00:07:38,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.82 | bwd_microstep: 5794.57 | bwd_inner_microstep: 5656.99 | bwd_allreduce_microstep: 137.53 | step_microstep: 19.26
[2025-04-26 00:07:38,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.82 | bwd: 5794.58 | bwd_inner: 5656.99 | bwd_allreduce: 137.55 | step: 19.26

 16%|█▌        | 6691/41250 [16:10:03<83:17:06,  8.68s/it]
                                                          
{'loss': 0.1879, 'grad_norm': 2.428492546081543, 'learning_rate': 3.8194714956041677e-05, 'epoch': 1.62}

 16%|█▌        | 6691/41250 [16:10:03<83:17:06,  8.68s/it][2025-04-26 00:07:46,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.13
[2025-04-26 00:07:46,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5766.45 | bwd_inner_microstep: 5710.14 | bwd_allreduce_microstep: 56.25 | step_microstep: 19.63
[2025-04-26 00:07:46,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5766.46 | bwd_inner: 5710.14 | bwd_allreduce: 56.27 | step: 19.63

 16%|█▌        | 6692/41250 [16:10:12<83:23:01,  8.69s/it]
                                                          
{'loss': 0.0944, 'grad_norm': 0.9694430828094482, 'learning_rate': 3.819406292015755e-05, 'epoch': 1.62}

 16%|█▌        | 6692/41250 [16:10:12<83:23:01,  8.69s/it][2025-04-26 00:07:55,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 00:07:55,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.91 | bwd_microstep: 5727.83 | bwd_inner_microstep: 5642.56 | bwd_allreduce_microstep: 85.22 | step_microstep: 19.10
[2025-04-26 00:07:55,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.91 | bwd: 5727.84 | bwd_inner: 5642.56 | bwd_allreduce: 85.24 | step: 19.11

 16%|█▌        | 6693/41250 [16:10:20<83:14:04,  8.67s/it]
                                                          
{'loss': 0.0585, 'grad_norm': 1.0424073934555054, 'learning_rate': 3.81934107721106e-05, 'epoch': 1.62}

 16%|█▌        | 6693/41250 [16:10:20<83:14:04,  8.67s/it][2025-04-26 00:08:04,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-26 00:08:04,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.54 | bwd_microstep: 5791.41 | bwd_inner_microstep: 5680.82 | bwd_allreduce_microstep: 110.54 | step_microstep: 19.47
[2025-04-26 00:08:04,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.54 | bwd: 5791.43 | bwd_inner: 5680.82 | bwd_allreduce: 110.56 | step: 19.47

 16%|█▌        | 6694/41250 [16:10:29<83:22:49,  8.69s/it]
                                                          
{'loss': 0.2232, 'grad_norm': 1.920749306678772, 'learning_rate': 3.819275851190486e-05, 'epoch': 1.62}

 16%|█▌        | 6694/41250 [16:10:29<83:22:49,  8.69s/it][2025-04-26 00:08:12,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:08:12,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.99 | bwd_microstep: 5776.77 | bwd_inner_microstep: 5764.45 | bwd_allreduce_microstep: 12.27 | step_microstep: 18.97
[2025-04-26 00:08:12,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.99 | bwd: 5776.78 | bwd_inner: 5764.45 | bwd_allreduce: 12.29 | step: 18.97

 16%|█▌        | 6695/41250 [16:10:38<83:32:29,  8.70s/it]
                                                          
{'loss': 0.1603, 'grad_norm': 4.035044193267822, 'learning_rate': 3.8192106139544334e-05, 'epoch': 1.62}

 16%|█▌        | 6695/41250 [16:10:38<83:32:29,  8.70s/it][2025-04-26 00:08:21,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:08:21,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.15 | bwd_microstep: 5884.75 | bwd_inner_microstep: 5712.36 | bwd_allreduce_microstep: 172.34 | step_microstep: 18.72
[2025-04-26 00:08:21,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.15 | bwd: 5884.77 | bwd_inner: 5712.36 | bwd_allreduce: 172.36 | step: 18.73

 16%|█▌        | 6696/41250 [16:10:47<83:53:29,  8.74s/it]
                                                          
{'loss': 0.1031, 'grad_norm': 2.414093017578125, 'learning_rate': 3.819145365503305e-05, 'epoch': 1.62}

 16%|█▌        | 6696/41250 [16:10:47<83:53:29,  8.74s/it][2025-04-26 00:08:30,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:08:30,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.67 | bwd_microstep: 5770.37 | bwd_inner_microstep: 5713.15 | bwd_allreduce_microstep: 57.17 | step_microstep: 18.43
[2025-04-26 00:08:30,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.67 | bwd: 5770.38 | bwd_inner: 5713.15 | bwd_allreduce: 57.19 | step: 18.43

 16%|█▌        | 6697/41250 [16:10:55<83:46:37,  8.73s/it]
                                                          
{'loss': 0.137, 'grad_norm': 2.451051950454712, 'learning_rate': 3.819080105837504e-05, 'epoch': 1.62}

 16%|█▌        | 6697/41250 [16:10:55<83:46:37,  8.73s/it][2025-04-26 00:08:39,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:08:39,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.29 | bwd_microstep: 5779.94 | bwd_inner_microstep: 5692.52 | bwd_allreduce_microstep: 87.36 | step_microstep: 18.76
[2025-04-26 00:08:39,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.29 | bwd: 5779.95 | bwd_inner: 5692.52 | bwd_allreduce: 87.38 | step: 18.77

 16%|█▌        | 6698/41250 [16:11:04<83:43:14,  8.72s/it]
                                                          
{'loss': 0.0619, 'grad_norm': 0.9682704210281372, 'learning_rate': 3.8190148349574316e-05, 'epoch': 1.62}

 16%|█▌        | 6698/41250 [16:11:04<83:43:14,  8.72s/it][2025-04-26 00:08:47,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 00:08:47,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.05 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5700.00 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.28
[2025-04-26 00:08:47,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.05 | bwd: 5712.93 | bwd_inner: 5700.00 | bwd_allreduce: 12.89 | step: 19.28

 16%|█▌        | 6699/41250 [16:11:13<83:29:32,  8.70s/it]
                                                          
{'loss': 0.3197, 'grad_norm': 3.4583005905151367, 'learning_rate': 3.8189495528634906e-05, 'epoch': 1.62}

 16%|█▌        | 6699/41250 [16:11:13<83:29:32,  8.70s/it][2025-04-26 00:08:56,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 00:08:56,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.98 | bwd_microstep: 5707.80 | bwd_inner_microstep: 5695.13 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.09
[2025-04-26 00:08:56,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.98 | bwd: 5707.81 | bwd_inner: 5695.13 | bwd_allreduce: 12.64 | step: 19.09

 16%|█▌        | 6700/41250 [16:11:21<83:18:40,  8.68s/it]
                                                          
{'loss': 0.2007, 'grad_norm': 2.6519787311553955, 'learning_rate': 3.8188842595560836e-05, 'epoch': 1.62}

 16%|█▌        | 6700/41250 [16:11:21<83:18:40,  8.68s/it][2025-04-26 00:09:05,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:09:05,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5820.48 | bwd_inner_microstep: 5645.71 | bwd_allreduce_microstep: 174.73 | step_microstep: 19.09
[2025-04-26 00:09:05,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5820.49 | bwd_inner: 5645.71 | bwd_allreduce: 174.74 | step: 19.10

 16%|█▌        | 6701/41250 [16:11:30<83:26:15,  8.69s/it]
                                                          
{'loss': 0.1298, 'grad_norm': 1.53311026096344, 'learning_rate': 3.818818955035612e-05, 'epoch': 1.62}

 16%|█▌        | 6701/41250 [16:11:30<83:26:15,  8.69s/it][2025-04-26 00:09:14,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 00:09:14,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5939.92 | bwd_inner_microstep: 5646.50 | bwd_allreduce_microstep: 293.38 | step_microstep: 18.86
[2025-04-26 00:09:14,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5939.94 | bwd_inner: 5646.50 | bwd_allreduce: 293.40 | step: 18.86

 16%|█▌        | 6702/41250 [16:11:39<83:53:34,  8.74s/it]
                                                          
{'loss': 0.4589, 'grad_norm': 4.553058624267578, 'learning_rate': 3.81875363930248e-05, 'epoch': 1.62}

 16%|█▌        | 6702/41250 [16:11:39<83:53:34,  8.74s/it][2025-04-26 00:09:22,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:09:22,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.06 | bwd_microstep: 5881.95 | bwd_inner_microstep: 5677.22 | bwd_allreduce_microstep: 204.69 | step_microstep: 19.03
[2025-04-26 00:09:22,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.06 | bwd: 5881.97 | bwd_inner: 5677.22 | bwd_allreduce: 204.71 | step: 19.04

 16%|█▌        | 6703/41250 [16:11:48<84:06:20,  8.76s/it]
                                                          
{'loss': 0.1448, 'grad_norm': 2.1102609634399414, 'learning_rate': 3.81868831235709e-05, 'epoch': 1.62}

 16%|█▌        | 6703/41250 [16:11:48<84:06:20,  8.76s/it][2025-04-26 00:09:31,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.27 | optimizer_step: 1.06
[2025-04-26 00:09:31,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.46 | bwd_microstep: 5699.50 | bwd_inner_microstep: 5685.56 | bwd_allreduce_microstep: 13.87 | step_microstep: 20.21
[2025-04-26 00:09:31,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.46 | bwd: 5699.51 | bwd_inner: 5685.56 | bwd_allreduce: 13.90 | step: 20.21

 16%|█▋        | 6704/41250 [16:11:56<83:45:15,  8.73s/it]
                                                          
{'loss': 0.1677, 'grad_norm': 1.7323369979858398, 'learning_rate': 3.818622974199843e-05, 'epoch': 1.63}

 16%|█▋        | 6704/41250 [16:11:56<83:45:15,  8.73s/it][2025-04-26 00:09:40,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-26 00:09:40,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.78 | bwd_microstep: 5714.11 | bwd_inner_microstep: 5701.06 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.02
[2025-04-26 00:09:40,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.78 | bwd: 5714.12 | bwd_inner: 5701.06 | bwd_allreduce: 13.02 | step: 19.02

 16%|█▋        | 6705/41250 [16:12:05<83:31:35,  8.70s/it]
                                                          
{'loss': 0.1884, 'grad_norm': 1.6175658702850342, 'learning_rate': 3.818557624831144e-05, 'epoch': 1.63}

 16%|█▋        | 6705/41250 [16:12:05<83:31:35,  8.70s/it][2025-04-26 00:09:48,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 00:09:48,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.93 | bwd_microstep: 5740.39 | bwd_inner_microstep: 5692.64 | bwd_allreduce_microstep: 47.71 | step_microstep: 19.22
[2025-04-26 00:09:48,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.93 | bwd: 5740.41 | bwd_inner: 5692.64 | bwd_allreduce: 47.73 | step: 19.23

 16%|█▋        | 6706/41250 [16:12:14<83:26:13,  8.70s/it]
                                                          
{'loss': 0.2182, 'grad_norm': 5.697456359863281, 'learning_rate': 3.818492264251394e-05, 'epoch': 1.63}

 16%|█▋        | 6706/41250 [16:12:14<83:26:13,  8.70s/it][2025-04-26 00:09:57,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 00:09:57,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5780.26 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 130.93 | step_microstep: 18.75
[2025-04-26 00:09:57,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5780.28 | bwd_inner: 5649.29 | bwd_allreduce: 130.95 | step: 18.75

 16%|█▋        | 6707/41250 [16:12:22<83:25:10,  8.69s/it]
                                                          
{'loss': 0.2779, 'grad_norm': 2.599959135055542, 'learning_rate': 3.818426892460998e-05, 'epoch': 1.63}

 16%|█▋        | 6707/41250 [16:12:22<83:25:10,  8.69s/it][2025-04-26 00:10:06,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 00:10:06,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.63 | bwd_microstep: 5708.81 | bwd_inner_microstep: 5695.68 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.90
[2025-04-26 00:10:06,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.63 | bwd: 5708.83 | bwd_inner: 5695.68 | bwd_allreduce: 13.11 | step: 18.91

 16%|█▋        | 6708/41250 [16:12:31<83:16:00,  8.68s/it]
                                                          
{'loss': 0.0257, 'grad_norm': 0.4857819378376007, 'learning_rate': 3.818361509460358e-05, 'epoch': 1.63}

 16%|█▋        | 6708/41250 [16:12:31<83:16:00,  8.68s/it][2025-04-26 00:10:14,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 00:10:14,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.31 | bwd_microstep: 5686.02 | bwd_inner_microstep: 5673.55 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.74
[2025-04-26 00:10:14,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.31 | bwd: 5686.03 | bwd_inner: 5673.55 | bwd_allreduce: 12.44 | step: 18.74

 16%|█▋        | 6709/41250 [16:12:40<83:04:28,  8.66s/it]
                                                          
{'loss': 0.1989, 'grad_norm': 1.7417594194412231, 'learning_rate': 3.818296115249876e-05, 'epoch': 1.63}

 16%|█▋        | 6709/41250 [16:12:40<83:04:28,  8.66s/it][2025-04-26 00:10:23,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 00:10:23,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.42 | bwd_microstep: 5785.14 | bwd_inner_microstep: 5772.47 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.19
[2025-04-26 00:10:23,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.42 | bwd: 5785.15 | bwd_inner: 5772.47 | bwd_allreduce: 12.64 | step: 19.19

 16%|█▋        | 6710/41250 [16:12:48<83:21:00,  8.69s/it]
                                                          
{'loss': 0.2491, 'grad_norm': 2.4291179180145264, 'learning_rate': 3.8182307098299564e-05, 'epoch': 1.63}

 16%|█▋        | 6710/41250 [16:12:48<83:21:00,  8.69s/it][2025-04-26 00:10:32,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 00:10:32,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.95 | bwd_microstep: 5761.13 | bwd_inner_microstep: 5748.24 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.93
[2025-04-26 00:10:32,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.95 | bwd: 5761.15 | bwd_inner: 5748.24 | bwd_allreduce: 12.87 | step: 18.94

 16%|█▋        | 6711/41250 [16:12:57<83:27:55,  8.70s/it]
                                                          
{'loss': 0.1349, 'grad_norm': 1.7617063522338867, 'learning_rate': 3.818165293201002e-05, 'epoch': 1.63}

 16%|█▋        | 6711/41250 [16:12:57<83:27:55,  8.70s/it][2025-04-26 00:10:40,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 00:10:40,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.32 | bwd_microstep: 5776.49 | bwd_inner_microstep: 5639.46 | bwd_allreduce_microstep: 136.99 | step_microstep: 19.11
[2025-04-26 00:10:40,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.32 | bwd: 5776.50 | bwd_inner: 5639.46 | bwd_allreduce: 137.01 | step: 19.11

 16%|█▋        | 6712/41250 [16:13:06<83:24:18,  8.69s/it]
                                                          
{'loss': 0.2194, 'grad_norm': 2.362750291824341, 'learning_rate': 3.818099865363417e-05, 'epoch': 1.63}

 16%|█▋        | 6712/41250 [16:13:06<83:24:18,  8.69s/it][2025-04-26 00:10:49,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:10:49,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.58 | bwd_microstep: 5786.45 | bwd_inner_microstep: 5654.43 | bwd_allreduce_microstep: 131.97 | step_microstep: 18.64
[2025-04-26 00:10:49,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.58 | bwd: 5786.47 | bwd_inner: 5654.43 | bwd_allreduce: 131.99 | step: 18.64

 16%|█▋        | 6713/41250 [16:13:14<83:23:55,  8.69s/it]
                                                          
{'loss': 0.3346, 'grad_norm': 1.7827624082565308, 'learning_rate': 3.818034426317603e-05, 'epoch': 1.63}

 16%|█▋        | 6713/41250 [16:13:14<83:23:55,  8.69s/it][2025-04-26 00:10:58,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:10:58,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.53 | bwd_microstep: 5738.63 | bwd_inner_microstep: 5677.69 | bwd_allreduce_microstep: 60.90 | step_microstep: 18.83
[2025-04-26 00:10:58,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.53 | bwd: 5738.64 | bwd_inner: 5677.69 | bwd_allreduce: 60.91 | step: 18.83

 16%|█▋        | 6714/41250 [16:13:23<83:18:54,  8.68s/it]
                                                          
{'loss': 0.0987, 'grad_norm': 1.2950730323791504, 'learning_rate': 3.8179689760639645e-05, 'epoch': 1.63}

 16%|█▋        | 6714/41250 [16:13:23<83:18:54,  8.68s/it][2025-04-26 00:11:06,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:11:06,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.62 | bwd_microstep: 5678.53 | bwd_inner_microstep: 5635.48 | bwd_allreduce_microstep: 43.00 | step_microstep: 18.61
[2025-04-26 00:11:06,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.62 | bwd: 5678.54 | bwd_inner: 5635.48 | bwd_allreduce: 43.01 | step: 18.62

 16%|█▋        | 6715/41250 [16:13:32<83:01:14,  8.65s/it]
                                                          
{'loss': 0.0202, 'grad_norm': 0.23108172416687012, 'learning_rate': 3.8179035146029047e-05, 'epoch': 1.63}

 16%|█▋        | 6715/41250 [16:13:32<83:01:14,  8.65s/it][2025-04-26 00:11:15,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.09 | optimizer_step: 1.08
[2025-04-26 00:11:15,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.96 | bwd_microstep: 5741.28 | bwd_inner_microstep: 5690.91 | bwd_allreduce_microstep: 50.32 | step_microstep: 19.34
[2025-04-26 00:11:15,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.96 | bwd: 5741.30 | bwd_inner: 5690.91 | bwd_allreduce: 50.35 | step: 19.34

 16%|█▋        | 6716/41250 [16:13:40<83:04:37,  8.66s/it]
                                                          
{'loss': 0.0493, 'grad_norm': 0.9438426494598389, 'learning_rate': 3.817838041934828e-05, 'epoch': 1.63}

 16%|█▋        | 6716/41250 [16:13:40<83:04:37,  8.66s/it][2025-04-26 00:11:24,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:11:24,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.79 | bwd_microstep: 5750.49 | bwd_inner_microstep: 5697.47 | bwd_allreduce_microstep: 52.98 | step_microstep: 18.73
[2025-04-26 00:11:24,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.79 | bwd: 5750.51 | bwd_inner: 5697.47 | bwd_allreduce: 53.00 | step: 18.74

 16%|█▋        | 6717/41250 [16:13:49<83:07:43,  8.67s/it]
                                                          
{'loss': 0.1681, 'grad_norm': 1.939092755317688, 'learning_rate': 3.817772558060137e-05, 'epoch': 1.63}

 16%|█▋        | 6717/41250 [16:13:49<83:07:43,  8.67s/it][2025-04-26 00:11:32,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:11:32,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.06 | bwd_microstep: 5753.46 | bwd_inner_microstep: 5634.13 | bwd_allreduce_microstep: 119.29 | step_microstep: 18.75
[2025-04-26 00:11:32,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.06 | bwd: 5753.47 | bwd_inner: 5634.13 | bwd_allreduce: 119.31 | step: 18.75

 16%|█▋        | 6718/41250 [16:13:58<83:06:45,  8.66s/it]
                                                          
{'loss': 0.2061, 'grad_norm': 1.9491477012634277, 'learning_rate': 3.8177070629792356e-05, 'epoch': 1.63}

 16%|█▋        | 6718/41250 [16:13:58<83:06:45,  8.66s/it][2025-04-26 00:11:41,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 00:11:41,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.47 | bwd_microstep: 5875.37 | bwd_inner_microstep: 5647.76 | bwd_allreduce_microstep: 227.55 | step_microstep: 18.79
[2025-04-26 00:11:41,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.46 | bwd: 5875.38 | bwd_inner: 5647.76 | bwd_allreduce: 227.57 | step: 18.79

 16%|█▋        | 6719/41250 [16:14:06<83:28:57,  8.70s/it]
                                                          
{'loss': 0.1581, 'grad_norm': 2.0971312522888184, 'learning_rate': 3.8176415566925274e-05, 'epoch': 1.63}

 16%|█▋        | 6719/41250 [16:14:06<83:28:57,  8.70s/it][2025-04-26 00:11:50,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-26 00:11:50,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.89 | bwd_microstep: 5685.63 | bwd_inner_microstep: 5637.21 | bwd_allreduce_microstep: 48.38 | step_microstep: 18.92
[2025-04-26 00:11:50,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.89 | bwd: 5685.64 | bwd_inner: 5637.21 | bwd_allreduce: 48.39 | step: 18.92

 16%|█▋        | 6720/41250 [16:14:15<83:10:05,  8.67s/it]
                                                          
{'loss': 0.0933, 'grad_norm': 1.576606273651123, 'learning_rate': 3.817576039200417e-05, 'epoch': 1.63}

 16%|█▋        | 6720/41250 [16:14:15<83:10:05,  8.67s/it][2025-04-26 00:11:58,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:11:58,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.82 | bwd_microstep: 5742.20 | bwd_inner_microstep: 5689.33 | bwd_allreduce_microstep: 52.83 | step_microstep: 18.94
[2025-04-26 00:11:58,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.82 | bwd: 5742.22 | bwd_inner: 5689.33 | bwd_allreduce: 52.85 | step: 18.95

 16%|█▋        | 6721/41250 [16:14:24<83:09:12,  8.67s/it]
                                                          
{'loss': 0.361, 'grad_norm': 2.079922914505005, 'learning_rate': 3.8175105105033075e-05, 'epoch': 1.63}

 16%|█▋        | 6721/41250 [16:14:24<83:09:12,  8.67s/it][2025-04-26 00:12:07,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 00:12:07,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5772.09 | bwd_inner_microstep: 5677.43 | bwd_allreduce_microstep: 94.62 | step_microstep: 19.14
[2025-04-26 00:12:07,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.37 | bwd: 5772.11 | bwd_inner: 5677.43 | bwd_allreduce: 94.63 | step: 19.14

 16%|█▋        | 6722/41250 [16:14:32<83:13:57,  8.68s/it]
                                                          
{'loss': 0.1767, 'grad_norm': 3.078165054321289, 'learning_rate': 3.817444970601603e-05, 'epoch': 1.63}

 16%|█▋        | 6722/41250 [16:14:32<83:13:57,  8.68s/it][2025-04-26 00:12:16,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 00:12:16,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.20 | bwd_microstep: 5692.57 | bwd_inner_microstep: 5679.85 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.78
[2025-04-26 00:12:16,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.20 | bwd: 5692.58 | bwd_inner: 5679.85 | bwd_allreduce: 12.69 | step: 18.78

 16%|█▋        | 6723/41250 [16:14:41<83:05:28,  8.66s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 1.0252424478530884, 'learning_rate': 3.817379419495708e-05, 'epoch': 1.63}

 16%|█▋        | 6723/41250 [16:14:41<83:05:28,  8.66s/it][2025-04-26 00:12:24,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 00:12:24,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.22 | bwd_microstep: 5779.30 | bwd_inner_microstep: 5657.42 | bwd_allreduce_microstep: 121.84 | step_microstep: 18.69
[2025-04-26 00:12:24,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.22 | bwd: 5779.31 | bwd_inner: 5657.42 | bwd_allreduce: 121.85 | step: 18.69

 16%|█▋        | 6724/41250 [16:14:50<83:09:34,  8.67s/it]
                                                          
{'loss': 0.0691, 'grad_norm': 1.6152678728103638, 'learning_rate': 3.817313857186027e-05, 'epoch': 1.63}

 16%|█▋        | 6724/41250 [16:14:50<83:09:34,  8.67s/it][2025-04-26 00:12:33,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 00:12:33,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.61 | bwd_microstep: 5731.17 | bwd_inner_microstep: 5700.60 | bwd_allreduce_microstep: 30.53 | step_microstep: 18.75
[2025-04-26 00:12:33,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.61 | bwd: 5731.19 | bwd_inner: 5700.60 | bwd_allreduce: 30.55 | step: 18.75

 16%|█▋        | 6725/41250 [16:14:58<83:08:11,  8.67s/it]
                                                          
{'loss': 0.0685, 'grad_norm': 1.1357942819595337, 'learning_rate': 3.817248283672962e-05, 'epoch': 1.63}

 16%|█▋        | 6725/41250 [16:14:58<83:08:11,  8.67s/it][2025-04-26 00:12:42,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:12:42,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.84 | bwd_microstep: 5715.71 | bwd_inner_microstep: 5702.91 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.58
[2025-04-26 00:12:42,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.84 | bwd: 5715.72 | bwd_inner: 5702.91 | bwd_allreduce: 12.77 | step: 18.58

 16%|█▋        | 6726/41250 [16:15:07<83:05:40,  8.66s/it]
                                                          
{'loss': 0.0787, 'grad_norm': 0.9357116222381592, 'learning_rate': 3.81718269895692e-05, 'epoch': 1.63}

 16%|█▋        | 6726/41250 [16:15:07<83:05:40,  8.66s/it][2025-04-26 00:12:50,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:12:50,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.80 | bwd_microstep: 5778.38 | bwd_inner_microstep: 5651.21 | bwd_allreduce_microstep: 127.13 | step_microstep: 18.53
[2025-04-26 00:12:50,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.80 | bwd: 5778.39 | bwd_inner: 5651.21 | bwd_allreduce: 127.15 | step: 18.53

 16%|█▋        | 6727/41250 [16:15:16<83:09:45,  8.67s/it]
                                                          
{'loss': 0.0248, 'grad_norm': 0.34154626727104187, 'learning_rate': 3.8171171030383043e-05, 'epoch': 1.63}

 16%|█▋        | 6727/41250 [16:15:16<83:09:45,  8.67s/it][2025-04-26 00:12:59,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-26 00:12:59,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.01 | bwd_microstep: 5707.09 | bwd_inner_microstep: 5656.73 | bwd_allreduce_microstep: 50.31 | step_microstep: 18.88
[2025-04-26 00:12:59,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.01 | bwd: 5707.11 | bwd_inner: 5656.73 | bwd_allreduce: 50.33 | step: 18.88

 16%|█▋        | 6728/41250 [16:15:24<83:00:43,  8.66s/it]
                                                          
{'loss': 0.0857, 'grad_norm': 0.8892666697502136, 'learning_rate': 3.8170514959175185e-05, 'epoch': 1.63}

 16%|█▋        | 6728/41250 [16:15:24<83:00:43,  8.66s/it][2025-04-26 00:13:08,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.00
[2025-04-26 00:13:08,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.38 | bwd_microstep: 5807.06 | bwd_inner_microstep: 5663.65 | bwd_allreduce_microstep: 143.37 | step_microstep: 18.55
[2025-04-26 00:13:08,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.38 | bwd: 5807.08 | bwd_inner: 5663.65 | bwd_allreduce: 143.39 | step: 18.55

 16%|█▋        | 6729/41250 [16:15:33<83:11:22,  8.68s/it]
                                                          
{'loss': 0.1456, 'grad_norm': 1.9486117362976074, 'learning_rate': 3.816985877594967e-05, 'epoch': 1.63}

 16%|█▋        | 6729/41250 [16:15:33<83:11:22,  8.68s/it][2025-04-26 00:13:16,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:13:16,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5805.10 | bwd_inner_microstep: 5654.97 | bwd_allreduce_microstep: 150.08 | step_microstep: 18.62
[2025-04-26 00:13:16,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5805.11 | bwd_inner: 5654.97 | bwd_allreduce: 150.09 | step: 18.62

 16%|█▋        | 6730/41250 [16:15:42<83:18:43,  8.69s/it]
                                                          
{'loss': 0.2656, 'grad_norm': 1.5827038288116455, 'learning_rate': 3.8169202480710557e-05, 'epoch': 1.63}

 16%|█▋        | 6730/41250 [16:15:42<83:18:43,  8.69s/it][2025-04-26 00:13:25,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:13:25,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5778.05 | bwd_inner_microstep: 5710.54 | bwd_allreduce_microstep: 67.46 | step_microstep: 18.58
[2025-04-26 00:13:25,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5778.06 | bwd_inner: 5710.54 | bwd_allreduce: 67.48 | step: 18.59

 16%|█▋        | 6731/41250 [16:15:51<83:22:05,  8.69s/it]
                                                          
{'loss': 0.1129, 'grad_norm': 1.519233226776123, 'learning_rate': 3.816854607346188e-05, 'epoch': 1.63}

 16%|█▋        | 6731/41250 [16:15:51<83:22:05,  8.69s/it][2025-04-26 00:13:34,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 00:13:34,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.00 | bwd_microstep: 5748.85 | bwd_inner_microstep: 5678.98 | bwd_allreduce_microstep: 69.83 | step_microstep: 19.47
[2025-04-26 00:13:34,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.00 | bwd: 5748.87 | bwd_inner: 5678.98 | bwd_allreduce: 69.85 | step: 19.47

 16%|█▋        | 6732/41250 [16:15:59<83:22:26,  8.70s/it]
                                                          
{'loss': 0.2028, 'grad_norm': 1.4752343893051147, 'learning_rate': 3.8167889554207695e-05, 'epoch': 1.63}

 16%|█▋        | 6732/41250 [16:15:59<83:22:26,  8.70s/it][2025-04-26 00:13:43,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:13:43,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.32 | bwd_microstep: 5789.52 | bwd_inner_microstep: 5646.26 | bwd_allreduce_microstep: 143.21 | step_microstep: 18.61
[2025-04-26 00:13:43,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.32 | bwd: 5789.53 | bwd_inner: 5646.26 | bwd_allreduce: 143.23 | step: 18.61

 16%|█▋        | 6733/41250 [16:16:08<83:22:10,  8.70s/it]
                                                          
{'loss': 0.0627, 'grad_norm': 2.102294683456421, 'learning_rate': 3.8167232922952046e-05, 'epoch': 1.63}

 16%|█▋        | 6733/41250 [16:16:08<83:22:10,  8.70s/it][2025-04-26 00:13:51,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:13:51,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.97 | bwd_microstep: 5714.98 | bwd_inner_microstep: 5647.96 | bwd_allreduce_microstep: 66.97 | step_microstep: 18.89
[2025-04-26 00:13:51,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.97 | bwd: 5714.99 | bwd_inner: 5647.96 | bwd_allreduce: 66.99 | step: 18.89

 16%|█▋        | 6734/41250 [16:16:17<83:09:16,  8.67s/it]
                                                          
{'loss': 0.0669, 'grad_norm': 1.0532704591751099, 'learning_rate': 3.816657617969897e-05, 'epoch': 1.63}

 16%|█▋        | 6734/41250 [16:16:17<83:09:16,  8.67s/it][2025-04-26 00:14:00,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:14:00,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.82 | bwd_microstep: 5718.22 | bwd_inner_microstep: 5651.82 | bwd_allreduce_microstep: 66.35 | step_microstep: 18.63
[2025-04-26 00:14:00,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.82 | bwd: 5718.23 | bwd_inner: 5651.82 | bwd_allreduce: 66.37 | step: 18.64

 16%|█▋        | 6735/41250 [16:16:25<83:00:50,  8.66s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 0.9009657502174377, 'learning_rate': 3.8165919324452533e-05, 'epoch': 1.63}

 16%|█▋        | 6735/41250 [16:16:25<83:00:50,  8.66s/it][2025-04-26 00:14:09,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:14:09,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5767.15 | bwd_inner_microstep: 5665.92 | bwd_allreduce_microstep: 101.18 | step_microstep: 18.59
[2025-04-26 00:14:09,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5767.16 | bwd_inner: 5665.92 | bwd_allreduce: 101.19 | step: 18.59

 16%|█▋        | 6736/41250 [16:16:34<83:03:46,  8.66s/it]
                                                          
{'loss': 0.1749, 'grad_norm': 1.6849385499954224, 'learning_rate': 3.8165262357216774e-05, 'epoch': 1.63}

 16%|█▋        | 6736/41250 [16:16:34<83:03:46,  8.66s/it][2025-04-26 00:14:17,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-26 00:14:17,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.96 | bwd_microstep: 5771.33 | bwd_inner_microstep: 5661.70 | bwd_allreduce_microstep: 109.58 | step_microstep: 19.35
[2025-04-26 00:14:17,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.96 | bwd: 5771.34 | bwd_inner: 5661.69 | bwd_allreduce: 109.60 | step: 19.36

 16%|█▋        | 6737/41250 [16:16:43<83:07:12,  8.67s/it]
                                                          
{'loss': 0.1778, 'grad_norm': 2.6982340812683105, 'learning_rate': 3.816460527799575e-05, 'epoch': 1.63}

 16%|█▋        | 6737/41250 [16:16:43<83:07:12,  8.67s/it][2025-04-26 00:14:26,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:14:26,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.79 | bwd_microstep: 5711.94 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.65
[2025-04-26 00:14:26,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.79 | bwd: 5711.95 | bwd_inner: 5699.18 | bwd_allreduce: 12.73 | step: 18.65

 16%|█▋        | 6738/41250 [16:16:51<83:02:30,  8.66s/it]
                                                          
{'loss': 0.0697, 'grad_norm': 2.405750036239624, 'learning_rate': 3.81639480867935e-05, 'epoch': 1.63}

 16%|█▋        | 6738/41250 [16:16:51<83:02:30,  8.66s/it][2025-04-26 00:14:35,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:14:35,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5790.57 | bwd_inner_microstep: 5660.50 | bwd_allreduce_microstep: 130.02 | step_microstep: 18.65
[2025-04-26 00:14:35,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5790.58 | bwd_inner: 5660.50 | bwd_allreduce: 130.04 | step: 18.65

 16%|█▋        | 6739/41250 [16:17:00<83:09:40,  8.67s/it]
                                                          
{'loss': 0.1808, 'grad_norm': 1.643941879272461, 'learning_rate': 3.8163290783614086e-05, 'epoch': 1.63}

 16%|█▋        | 6739/41250 [16:17:00<83:09:40,  8.67s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x2021cb00] moov atom not found
[00:14:36] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4...
[2025-04-26 00:14:43,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 00:14:43,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.02 | bwd_microstep: 5866.57 | bwd_inner_microstep: 5696.72 | bwd_allreduce_microstep: 169.80 | step_microstep: 18.54
[2025-04-26 00:14:43,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.02 | bwd: 5866.58 | bwd_inner: 5696.72 | bwd_allreduce: 169.82 | step: 18.54

 16%|█▋        | 6740/41250 [16:17:09<83:29:54,  8.71s/it]
                                                          
{'loss': 0.4008, 'grad_norm': 2.503296375274658, 'learning_rate': 3.816263336846156e-05, 'epoch': 1.63}

 16%|█▋        | 6740/41250 [16:17:09<83:29:54,  8.71s/it][2025-04-26 00:14:52,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:14:52,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.96 | bwd_microstep: 5786.76 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 131.47 | step_microstep: 18.93
[2025-04-26 00:14:52,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.96 | bwd: 5786.78 | bwd_inner: 5655.24 | bwd_allreduce: 131.49 | step: 18.93

 16%|█▋        | 6741/41250 [16:17:17<83:27:10,  8.71s/it]
                                                          
{'loss': 0.0342, 'grad_norm': 0.459414541721344, 'learning_rate': 3.816197584133997e-05, 'epoch': 1.63}

 16%|█▋        | 6741/41250 [16:17:17<83:27:10,  8.71s/it][2025-04-26 00:15:01,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:15:01,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.95 | bwd_microstep: 5701.60 | bwd_inner_microstep: 5657.86 | bwd_allreduce_microstep: 43.69 | step_microstep: 18.86
[2025-04-26 00:15:01,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.95 | bwd: 5701.61 | bwd_inner: 5657.86 | bwd_allreduce: 43.71 | step: 18.86

 16%|█▋        | 6742/41250 [16:17:26<83:11:02,  8.68s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 1.9312071800231934, 'learning_rate': 3.816131820225336e-05, 'epoch': 1.63}

 16%|█▋        | 6742/41250 [16:17:26<83:11:02,  8.68s/it][2025-04-26 00:15:09,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:15:09,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5715.64 | bwd_inner_microstep: 5702.85 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.63
[2025-04-26 00:15:09,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5715.66 | bwd_inner: 5702.85 | bwd_allreduce: 12.77 | step: 18.63

 16%|█▋        | 6743/41250 [16:17:35<83:05:49,  8.67s/it]
                                                          
{'loss': 0.1186, 'grad_norm': 1.3811516761779785, 'learning_rate': 3.8160660451205804e-05, 'epoch': 1.63}

 16%|█▋        | 6743/41250 [16:17:35<83:05:49,  8.67s/it][2025-04-26 00:15:18,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 00:15:18,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5701.36 | bwd_inner_microstep: 5688.39 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.77
[2025-04-26 00:15:18,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5701.37 | bwd_inner: 5688.39 | bwd_allreduce: 12.93 | step: 18.77

 16%|█▋        | 6744/41250 [16:17:43<82:59:17,  8.66s/it]
                                                          
{'loss': 0.0784, 'grad_norm': 1.3723690509796143, 'learning_rate': 3.8160002588201355e-05, 'epoch': 1.63}

 16%|█▋        | 6744/41250 [16:17:43<82:59:17,  8.66s/it][2025-04-26 00:15:27,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.88
[2025-04-26 00:15:27,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5762.66 | bwd_inner_microstep: 5680.45 | bwd_allreduce_microstep: 82.16 | step_microstep: 18.43
[2025-04-26 00:15:27,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5762.67 | bwd_inner: 5680.45 | bwd_allreduce: 82.18 | step: 18.43

 16%|█▋        | 6745/41250 [16:17:52<83:06:15,  8.67s/it]
                                                          
{'loss': 0.3516, 'grad_norm': 3.046987771987915, 'learning_rate': 3.8159344613244054e-05, 'epoch': 1.64}

 16%|█▋        | 6745/41250 [16:17:52<83:06:15,  8.67s/it][2025-04-26 00:15:35,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-26 00:15:35,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5707.44 | bwd_inner_microstep: 5694.81 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.68
[2025-04-26 00:15:35,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5707.45 | bwd_inner: 5694.80 | bwd_allreduce: 12.61 | step: 18.69

 16%|█▋        | 6746/41250 [16:18:01<83:01:19,  8.66s/it]
                                                          
{'loss': 0.1888, 'grad_norm': 2.641197443008423, 'learning_rate': 3.8158686526337965e-05, 'epoch': 1.64}

 16%|█▋        | 6746/41250 [16:18:01<83:01:19,  8.66s/it][2025-04-26 00:15:44,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 00:15:44,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.44 | bwd_microstep: 5719.80 | bwd_inner_microstep: 5698.14 | bwd_allreduce_microstep: 21.62 | step_microstep: 18.92
[2025-04-26 00:15:44,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.44 | bwd: 5719.82 | bwd_inner: 5698.14 | bwd_allreduce: 21.64 | step: 18.92

 16%|█▋        | 6747/41250 [16:18:09<83:01:04,  8.66s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.5482485294342041, 'learning_rate': 3.815802832748715e-05, 'epoch': 1.64}

 16%|█▋        | 6747/41250 [16:18:09<83:01:04,  8.66s/it][2025-04-26 00:15:53,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:15:53,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.54 | bwd_microstep: 5720.81 | bwd_inner_microstep: 5674.80 | bwd_allreduce_microstep: 45.97 | step_microstep: 18.61
[2025-04-26 00:15:53,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5720.82 | bwd_inner: 5674.80 | bwd_allreduce: 45.98 | step: 18.62

 16%|█▋        | 6748/41250 [16:18:18<83:00:52,  8.66s/it]
                                                          
{'loss': 0.1707, 'grad_norm': 2.1724281311035156, 'learning_rate': 3.8157370016695656e-05, 'epoch': 1.64}

 16%|█▋        | 6748/41250 [16:18:18<83:00:52,  8.66s/it][2025-04-26 00:16:01,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:16:01,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.12 | bwd_microstep: 5718.67 | bwd_inner_microstep: 5704.66 | bwd_allreduce_microstep: 13.97 | step_microstep: 18.40
[2025-04-26 00:16:01,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.12 | bwd: 5718.68 | bwd_inner: 5704.66 | bwd_allreduce: 13.98 | step: 18.40

 16%|█▋        | 6749/41250 [16:18:27<83:02:40,  8.67s/it]
                                                          
{'loss': 0.0495, 'grad_norm': 0.6873926520347595, 'learning_rate': 3.815671159396755e-05, 'epoch': 1.64}

 16%|█▋        | 6749/41250 [16:18:27<83:02:40,  8.67s/it][2025-04-26 00:16:10,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:16:10,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5694.29 | bwd_inner_microstep: 5646.96 | bwd_allreduce_microstep: 47.28 | step_microstep: 19.06
[2025-04-26 00:16:10,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5694.30 | bwd_inner: 5646.96 | bwd_allreduce: 47.30 | step: 19.06

 16%|█▋        | 6750/41250 [16:18:35<82:52:17,  8.65s/it]
                                                          
{'loss': 0.3612, 'grad_norm': 3.7542500495910645, 'learning_rate': 3.815605305930689e-05, 'epoch': 1.64}

 16%|█▋        | 6750/41250 [16:18:35<82:52:17,  8.65s/it][2025-04-26 00:16:19,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 00:16:19,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5751.37 | bwd_inner_microstep: 5695.60 | bwd_allreduce_microstep: 55.71 | step_microstep: 19.13
[2025-04-26 00:16:19,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5751.39 | bwd_inner: 5695.60 | bwd_allreduce: 55.73 | step: 19.13

 16%|█▋        | 6751/41250 [16:18:44<82:59:21,  8.66s/it]
                                                          
{'loss': 0.1354, 'grad_norm': 1.2723621129989624, 'learning_rate': 3.815539441271773e-05, 'epoch': 1.64}

 16%|█▋        | 6751/41250 [16:18:44<82:59:21,  8.66s/it][2025-04-26 00:16:27,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 00:16:27,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5695.97 | bwd_inner_microstep: 5682.84 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.74
[2025-04-26 00:16:27,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5695.99 | bwd_inner: 5682.84 | bwd_allreduce: 13.09 | step: 18.74

 16%|█▋        | 6752/41250 [16:18:53<82:53:19,  8.65s/it]
                                                          
{'loss': 0.0386, 'grad_norm': 0.5527887344360352, 'learning_rate': 3.815473565420414e-05, 'epoch': 1.64}

 16%|█▋        | 6752/41250 [16:18:53<82:53:19,  8.65s/it][2025-04-26 00:16:36,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.14 | optimizer_step: 1.03
[2025-04-26 00:16:36,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5762.16 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 113.68 | step_microstep: 19.13
[2025-04-26 00:16:36,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5762.17 | bwd_inner: 5648.44 | bwd_allreduce: 113.69 | step: 19.14

 16%|█▋        | 6753/41250 [16:19:01<82:57:01,  8.66s/it]
                                                          
{'loss': 0.0866, 'grad_norm': 1.118107795715332, 'learning_rate': 3.815407678377017e-05, 'epoch': 1.64}

 16%|█▋        | 6753/41250 [16:19:01<82:57:01,  8.66s/it][2025-04-26 00:16:44,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:16:44,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 5683.58 | bwd_inner_microstep: 5644.05 | bwd_allreduce_microstep: 39.49 | step_microstep: 18.70
[2025-04-26 00:16:44,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 5683.59 | bwd_inner: 5644.05 | bwd_allreduce: 39.50 | step: 18.71

 16%|█▋        | 6754/41250 [16:19:10<82:45:58,  8.64s/it]
                                                          
{'loss': 0.1739, 'grad_norm': 3.5410923957824707, 'learning_rate': 3.81534178014199e-05, 'epoch': 1.64}

 16%|█▋        | 6754/41250 [16:19:10<82:45:58,  8.64s/it][2025-04-26 00:16:53,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 00:16:53,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.59 | bwd_microstep: 5742.79 | bwd_inner_microstep: 5646.00 | bwd_allreduce_microstep: 96.74 | step_microstep: 18.97
[2025-04-26 00:16:53,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.59 | bwd: 5742.81 | bwd_inner: 5646.00 | bwd_allreduce: 96.77 | step: 18.98

 16%|█▋        | 6755/41250 [16:19:18<82:50:16,  8.65s/it]
                                                          
{'loss': 0.0905, 'grad_norm': 1.533473014831543, 'learning_rate': 3.815275870715736e-05, 'epoch': 1.64}

 16%|█▋        | 6755/41250 [16:19:18<82:50:16,  8.65s/it][2025-04-26 00:17:02,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-26 00:17:02,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.45 | bwd_microstep: 5744.44 | bwd_inner_microstep: 5677.08 | bwd_allreduce_microstep: 67.30 | step_microstep: 19.05
[2025-04-26 00:17:02,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.45 | bwd: 5744.46 | bwd_inner: 5677.08 | bwd_allreduce: 67.33 | step: 19.05

 16%|█▋        | 6756/41250 [16:19:27<82:55:43,  8.65s/it]
                                                          
{'loss': 0.0726, 'grad_norm': 1.0478328466415405, 'learning_rate': 3.815209950098664e-05, 'epoch': 1.64}

 16%|█▋        | 6756/41250 [16:19:27<82:55:43,  8.65s/it][2025-04-26 00:17:10,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 00:17:10,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5718.89 | bwd_inner_microstep: 5706.11 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.84
[2025-04-26 00:17:10,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5718.90 | bwd_inner: 5706.11 | bwd_allreduce: 12.75 | step: 18.85

 16%|█▋        | 6757/41250 [16:19:36<82:55:08,  8.65s/it]
                                                          
{'loss': 0.0697, 'grad_norm': 1.9335765838623047, 'learning_rate': 3.81514401829118e-05, 'epoch': 1.64}

 16%|█▋        | 6757/41250 [16:19:36<82:55:08,  8.65s/it][2025-04-26 00:17:19,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:17:19,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.07 | bwd_microstep: 5763.60 | bwd_inner_microstep: 5751.01 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.82
[2025-04-26 00:17:19,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.07 | bwd: 5763.61 | bwd_inner: 5751.01 | bwd_allreduce: 12.56 | step: 18.82

 16%|█▋        | 6758/41250 [16:19:44<83:07:02,  8.68s/it]
                                                          
{'loss': 0.1012, 'grad_norm': 1.150116205215454, 'learning_rate': 3.8150780752936903e-05, 'epoch': 1.64}

 16%|█▋        | 6758/41250 [16:19:44<83:07:02,  8.68s/it][2025-04-26 00:17:28,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 00:17:28,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.31 | bwd_microstep: 5734.63 | bwd_inner_microstep: 5712.86 | bwd_allreduce_microstep: 21.72 | step_microstep: 19.12
[2025-04-26 00:17:28,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.31 | bwd: 5734.64 | bwd_inner: 5712.86 | bwd_allreduce: 21.74 | step: 19.12

 16%|█▋        | 6759/41250 [16:19:53<83:04:44,  8.67s/it]
                                                          
{'loss': 0.2151, 'grad_norm': 3.717503786087036, 'learning_rate': 3.815012121106602e-05, 'epoch': 1.64}

 16%|█▋        | 6759/41250 [16:19:53<83:04:44,  8.67s/it][2025-04-26 00:17:36,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:17:36,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.21 | bwd_microstep: 5730.84 | bwd_inner_microstep: 5646.15 | bwd_allreduce_microstep: 84.65 | step_microstep: 18.43
[2025-04-26 00:17:36,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.21 | bwd: 5730.85 | bwd_inner: 5646.15 | bwd_allreduce: 84.66 | step: 18.43

 16%|█▋        | 6760/41250 [16:20:02<83:02:22,  8.67s/it]
                                                          
{'loss': 0.0525, 'grad_norm': 0.779930591583252, 'learning_rate': 3.81494615573032e-05, 'epoch': 1.64}

 16%|█▋        | 6760/41250 [16:20:02<83:02:22,  8.67s/it][2025-04-26 00:17:45,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:17:45,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.07 | bwd_microstep: 5704.50 | bwd_inner_microstep: 5652.12 | bwd_allreduce_microstep: 52.33 | step_microstep: 18.58
[2025-04-26 00:17:45,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.07 | bwd: 5704.52 | bwd_inner: 5652.12 | bwd_allreduce: 52.36 | step: 18.58

 16%|█▋        | 6761/41250 [16:20:10<82:51:34,  8.65s/it]
                                                          
{'loss': 0.224, 'grad_norm': 2.4295647144317627, 'learning_rate': 3.8148801791652516e-05, 'epoch': 1.64}

 16%|█▋        | 6761/41250 [16:20:10<82:51:34,  8.65s/it][2025-04-26 00:17:54,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 00:17:54,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.83 | bwd_microstep: 5771.51 | bwd_inner_microstep: 5683.26 | bwd_allreduce_microstep: 88.21 | step_microstep: 19.06
[2025-04-26 00:17:54,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.83 | bwd: 5771.53 | bwd_inner: 5683.26 | bwd_allreduce: 88.23 | step: 19.06

 16%|█▋        | 6762/41250 [16:20:19<82:59:17,  8.66s/it]
                                                          
{'loss': 0.0574, 'grad_norm': 1.3573923110961914, 'learning_rate': 3.8148141914118045e-05, 'epoch': 1.64}

 16%|█▋        | 6762/41250 [16:20:19<82:59:17,  8.66s/it][2025-04-26 00:18:02,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 00:18:02,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.15 | bwd_microstep: 5742.54 | bwd_inner_microstep: 5695.95 | bwd_allreduce_microstep: 46.54 | step_microstep: 18.47
[2025-04-26 00:18:02,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.15 | bwd: 5742.55 | bwd_inner: 5695.95 | bwd_allreduce: 46.56 | step: 18.47

 16%|█▋        | 6763/41250 [16:20:28<83:03:37,  8.67s/it]
                                                          
{'loss': 0.1255, 'grad_norm': 1.2305967807769775, 'learning_rate': 3.8147481924703844e-05, 'epoch': 1.64}

 16%|█▋        | 6763/41250 [16:20:28<83:03:37,  8.67s/it][2025-04-26 00:18:11,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 00:18:11,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.15 | bwd_microstep: 5879.32 | bwd_inner_microstep: 5681.10 | bwd_allreduce_microstep: 198.17 | step_microstep: 18.76
[2025-04-26 00:18:11,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.15 | bwd: 5879.34 | bwd_inner: 5681.10 | bwd_allreduce: 198.19 | step: 18.76

 16%|█▋        | 6764/41250 [16:20:37<83:26:40,  8.71s/it]
                                                          
{'loss': 0.0555, 'grad_norm': 0.7385594844818115, 'learning_rate': 3.8146821823413985e-05, 'epoch': 1.64}

 16%|█▋        | 6764/41250 [16:20:37<83:26:40,  8.71s/it][2025-04-26 00:18:20,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:18:20,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.50 | bwd_microstep: 5744.98 | bwd_inner_microstep: 5692.45 | bwd_allreduce_microstep: 52.49 | step_microstep: 18.65
[2025-04-26 00:18:20,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5745.00 | bwd_inner: 5692.45 | bwd_allreduce: 52.51 | step: 18.65

 16%|█▋        | 6765/41250 [16:20:45<83:21:50,  8.70s/it]
                                                          
{'loss': 0.0933, 'grad_norm': 2.2985126972198486, 'learning_rate': 3.814616161025255e-05, 'epoch': 1.64}

 16%|█▋        | 6765/41250 [16:20:45<83:21:50,  8.70s/it][2025-04-26 00:18:29,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:18:29,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.90 | bwd_microstep: 5792.75 | bwd_inner_microstep: 5638.80 | bwd_allreduce_microstep: 153.90 | step_microstep: 18.35
[2025-04-26 00:18:29,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.90 | bwd: 5792.76 | bwd_inner: 5638.80 | bwd_allreduce: 153.92 | step: 18.35

 16%|█▋        | 6766/41250 [16:20:54<83:21:06,  8.70s/it]
                                                          
{'loss': 0.1498, 'grad_norm': 2.7651426792144775, 'learning_rate': 3.8145501285223585e-05, 'epoch': 1.64}

 16%|█▋        | 6766/41250 [16:20:54<83:21:06,  8.70s/it][2025-04-26 00:18:37,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 00:18:37,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5742.31 | bwd_inner_microstep: 5682.33 | bwd_allreduce_microstep: 59.93 | step_microstep: 18.88
[2025-04-26 00:18:37,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5742.32 | bwd_inner: 5682.33 | bwd_allreduce: 59.95 | step: 18.89

 16%|█▋        | 6767/41250 [16:21:03<83:15:20,  8.69s/it]
                                                          
{'loss': 0.1874, 'grad_norm': 2.5653574466705322, 'learning_rate': 3.814484084833118e-05, 'epoch': 1.64}

 16%|█▋        | 6767/41250 [16:21:03<83:15:20,  8.69s/it][2025-04-26 00:18:46,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 00:18:46,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.23 | bwd_microstep: 5741.12 | bwd_inner_microstep: 5655.18 | bwd_allreduce_microstep: 85.89 | step_microstep: 18.36
[2025-04-26 00:18:46,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.23 | bwd: 5741.13 | bwd_inner: 5655.18 | bwd_allreduce: 85.91 | step: 18.36

 16%|█▋        | 6768/41250 [16:21:11<83:09:57,  8.68s/it]
                                                          
{'loss': 0.0447, 'grad_norm': 2.8459560871124268, 'learning_rate': 3.81441802995794e-05, 'epoch': 1.64}

 16%|█▋        | 6768/41250 [16:21:11<83:09:57,  8.68s/it][2025-04-26 00:18:55,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 00:18:55,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.67 | bwd_microstep: 5687.38 | bwd_inner_microstep: 5649.31 | bwd_allreduce_microstep: 38.02 | step_microstep: 18.75
[2025-04-26 00:18:55,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.67 | bwd: 5687.39 | bwd_inner: 5649.31 | bwd_allreduce: 38.03 | step: 18.75

 16%|█▋        | 6769/41250 [16:21:20<82:57:15,  8.66s/it]
                                                          
{'loss': 0.2045, 'grad_norm': 3.293626308441162, 'learning_rate': 3.814351963897231e-05, 'epoch': 1.64}

 16%|█▋        | 6769/41250 [16:21:20<82:57:15,  8.66s/it][2025-04-26 00:19:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:19:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.70 | bwd_microstep: 5772.81 | bwd_inner_microstep: 5641.10 | bwd_allreduce_microstep: 131.66 | step_microstep: 18.74
[2025-04-26 00:19:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.70 | bwd: 5772.82 | bwd_inner: 5641.10 | bwd_allreduce: 131.68 | step: 18.74

 16%|█▋        | 6770/41250 [16:21:29<83:01:39,  8.67s/it]
                                                          
{'loss': 0.375, 'grad_norm': 3.5655930042266846, 'learning_rate': 3.8142858866513996e-05, 'epoch': 1.64}

 16%|█▋        | 6770/41250 [16:21:29<83:01:39,  8.67s/it][2025-04-26 00:19:12,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:19:12,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.28 | bwd_microstep: 5710.55 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.50
[2025-04-26 00:19:12,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.28 | bwd: 5710.56 | bwd_inner: 5697.65 | bwd_allreduce: 12.87 | step: 18.50

 16%|█▋        | 6771/41250 [16:21:37<82:58:32,  8.66s/it]
                                                          
{'loss': 0.1035, 'grad_norm': 0.9993962645530701, 'learning_rate': 3.814219798220853e-05, 'epoch': 1.64}

 16%|█▋        | 6771/41250 [16:21:37<82:58:32,  8.66s/it][2025-04-26 00:19:21,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:19:21,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.45 | bwd_microstep: 5768.54 | bwd_inner_microstep: 5630.58 | bwd_allreduce_microstep: 137.92 | step_microstep: 18.44
[2025-04-26 00:19:21,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.45 | bwd: 5768.56 | bwd_inner: 5630.58 | bwd_allreduce: 137.94 | step: 18.44

 16%|█▋        | 6772/41250 [16:21:46<83:01:45,  8.67s/it]
                                                          
{'loss': 0.2285, 'grad_norm': 2.0479395389556885, 'learning_rate': 3.8141536986059974e-05, 'epoch': 1.64}

 16%|█▋        | 6772/41250 [16:21:46<83:01:45,  8.67s/it][2025-04-26 00:19:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:19:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.98 | bwd_microstep: 5750.81 | bwd_inner_microstep: 5690.97 | bwd_allreduce_microstep: 59.79 | step_microstep: 18.66
[2025-04-26 00:19:29,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.98 | bwd: 5750.82 | bwd_inner: 5690.97 | bwd_allreduce: 59.81 | step: 18.66

 16%|█▋        | 6773/41250 [16:21:55<83:07:15,  8.68s/it]
                                                          
{'loss': 0.0225, 'grad_norm': 0.4953915476799011, 'learning_rate': 3.814087587807242e-05, 'epoch': 1.64}

 16%|█▋        | 6773/41250 [16:21:55<83:07:15,  8.68s/it][2025-04-26 00:19:38,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 00:19:38,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.66 | bwd_microstep: 5750.46 | bwd_inner_microstep: 5682.97 | bwd_allreduce_microstep: 67.44 | step_microstep: 19.29
[2025-04-26 00:19:38,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.66 | bwd: 5750.47 | bwd_inner: 5682.97 | bwd_allreduce: 67.46 | step: 19.30

 16%|█▋        | 6774/41250 [16:22:03<83:06:25,  8.68s/it]
                                                          
{'loss': 0.1855, 'grad_norm': 1.544228434562683, 'learning_rate': 3.8140214658249926e-05, 'epoch': 1.64}

 16%|█▋        | 6774/41250 [16:22:03<83:06:25,  8.68s/it][2025-04-26 00:19:47,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 00:19:47,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.30 | bwd_microstep: 5779.08 | bwd_inner_microstep: 5657.25 | bwd_allreduce_microstep: 121.79 | step_microstep: 18.51
[2025-04-26 00:19:47,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.30 | bwd: 5779.09 | bwd_inner: 5657.25 | bwd_allreduce: 121.81 | step: 18.51

 16%|█▋        | 6775/41250 [16:22:12<83:09:13,  8.68s/it]
                                                          
{'loss': 0.1051, 'grad_norm': 1.5925095081329346, 'learning_rate': 3.813955332659658e-05, 'epoch': 1.64}

 16%|█▋        | 6775/41250 [16:22:12<83:09:13,  8.68s/it][2025-04-26 00:19:55,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 00:19:55,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.43 | bwd_microstep: 5719.34 | bwd_inner_microstep: 5706.61 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.54
[2025-04-26 00:19:55,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.43 | bwd: 5719.35 | bwd_inner: 5706.61 | bwd_allreduce: 12.70 | step: 18.54

 16%|█▋        | 6776/41250 [16:22:21<83:04:13,  8.67s/it]
                                                          
{'loss': 0.1911, 'grad_norm': 2.946005344390869, 'learning_rate': 3.813889188311645e-05, 'epoch': 1.64}

 16%|█▋        | 6776/41250 [16:22:21<83:04:13,  8.67s/it][2025-04-26 00:20:04,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:20:04,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.40 | bwd_microstep: 5719.42 | bwd_inner_microstep: 5685.44 | bwd_allreduce_microstep: 33.94 | step_microstep: 18.22
[2025-04-26 00:20:04,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.40 | bwd: 5719.44 | bwd_inner: 5685.44 | bwd_allreduce: 33.95 | step: 18.22

 16%|█▋        | 6777/41250 [16:22:29<82:59:11,  8.67s/it]
                                                          
{'loss': 0.2299, 'grad_norm': 4.1225996017456055, 'learning_rate': 3.8138230327813625e-05, 'epoch': 1.64}

 16%|█▋        | 6777/41250 [16:22:29<82:59:11,  8.67s/it][2025-04-26 00:20:13,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:20:13,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.03 | bwd_microstep: 5751.50 | bwd_inner_microstep: 5709.41 | bwd_allreduce_microstep: 42.05 | step_microstep: 18.34
[2025-04-26 00:20:13,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.03 | bwd: 5751.51 | bwd_inner: 5709.41 | bwd_allreduce: 42.06 | step: 18.34

 16%|█▋        | 6778/41250 [16:22:38<83:03:50,  8.67s/it]
                                                          
{'loss': 0.0533, 'grad_norm': 0.9580344557762146, 'learning_rate': 3.813756866069218e-05, 'epoch': 1.64}

 16%|█▋        | 6778/41250 [16:22:38<83:03:50,  8.67s/it][2025-04-26 00:20:21,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:20:21,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.68 | bwd_microstep: 5692.86 | bwd_inner_microstep: 5667.90 | bwd_allreduce_microstep: 24.92 | step_microstep: 18.91
[2025-04-26 00:20:21,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.68 | bwd: 5692.87 | bwd_inner: 5667.90 | bwd_allreduce: 24.93 | step: 18.92

 16%|█▋        | 6779/41250 [16:22:47<82:53:36,  8.66s/it]
                                                          
{'loss': 0.2075, 'grad_norm': 1.6876612901687622, 'learning_rate': 3.813690688175618e-05, 'epoch': 1.64}

 16%|█▋        | 6779/41250 [16:22:47<82:53:36,  8.66s/it][2025-04-26 00:20:30,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:20:30,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.86 | bwd_microstep: 5800.68 | bwd_inner_microstep: 5716.91 | bwd_allreduce_microstep: 83.72 | step_microstep: 18.69
[2025-04-26 00:20:30,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.86 | bwd: 5800.69 | bwd_inner: 5716.91 | bwd_allreduce: 83.74 | step: 18.69

 16%|█▋        | 6780/41250 [16:22:55<83:07:48,  8.68s/it]
                                                          
{'loss': 0.0865, 'grad_norm': 1.0834929943084717, 'learning_rate': 3.8136244991009734e-05, 'epoch': 1.64}

 16%|█▋        | 6780/41250 [16:22:55<83:07:48,  8.68s/it][2025-04-26 00:20:39,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 00:20:39,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.27 | bwd_microstep: 5770.33 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 108.21 | step_microstep: 18.98
[2025-04-26 00:20:39,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.27 | bwd: 5770.35 | bwd_inner: 5662.08 | bwd_allreduce: 108.23 | step: 18.98

 16%|█▋        | 6781/41250 [16:23:04<83:08:19,  8.68s/it]
                                                          
{'loss': 0.1403, 'grad_norm': 1.2756743431091309, 'learning_rate': 3.8135582988456894e-05, 'epoch': 1.64}

 16%|█▋        | 6781/41250 [16:23:04<83:08:19,  8.68s/it][2025-04-26 00:20:48,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:20:48,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.71 | bwd_microstep: 6021.56 | bwd_inner_microstep: 5712.13 | bwd_allreduce_microstep: 309.38 | step_microstep: 18.64
[2025-04-26 00:20:48,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.71 | bwd: 6021.57 | bwd_inner: 5712.13 | bwd_allreduce: 309.40 | step: 18.64

 16%|█▋        | 6782/41250 [16:23:13<83:56:49,  8.77s/it]
                                                          
{'loss': 0.3656, 'grad_norm': 3.213106393814087, 'learning_rate': 3.8134920874101756e-05, 'epoch': 1.64}

 16%|█▋        | 6782/41250 [16:23:13<83:56:49,  8.77s/it][2025-04-26 00:20:56,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:20:56,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.31 | bwd_microstep: 5761.90 | bwd_inner_microstep: 5662.64 | bwd_allreduce_microstep: 99.22 | step_microstep: 18.55
[2025-04-26 00:20:56,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.31 | bwd: 5761.91 | bwd_inner: 5662.64 | bwd_allreduce: 99.23 | step: 18.55

 16%|█▋        | 6783/41250 [16:23:22<83:42:06,  8.74s/it]
                                                          
{'loss': 0.0517, 'grad_norm': 1.0449382066726685, 'learning_rate': 3.8134258647948394e-05, 'epoch': 1.64}

 16%|█▋        | 6783/41250 [16:23:22<83:42:06,  8.74s/it][2025-04-26 00:21:05,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:21:05,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5715.41 | bwd_inner_microstep: 5702.52 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.78
[2025-04-26 00:21:05,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5715.42 | bwd_inner: 5702.52 | bwd_allreduce: 12.85 | step: 18.78

 16%|█▋        | 6784/41250 [16:23:30<83:25:38,  8.71s/it]
                                                          
{'loss': 0.1475, 'grad_norm': 1.648278832435608, 'learning_rate': 3.813359631000089e-05, 'epoch': 1.64}

 16%|█▋        | 6784/41250 [16:23:30<83:25:38,  8.71s/it][2025-04-26 00:21:14,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 00:21:14,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.87 | bwd_microstep: 5754.43 | bwd_inner_microstep: 5660.88 | bwd_allreduce_microstep: 93.50 | step_microstep: 19.16
[2025-04-26 00:21:14,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.87 | bwd: 5754.45 | bwd_inner: 5660.88 | bwd_allreduce: 93.52 | step: 19.16

 16%|█▋        | 6785/41250 [16:23:39<83:19:45,  8.70s/it]
                                                          
{'loss': 0.0631, 'grad_norm': 1.2542098760604858, 'learning_rate': 3.813293386026334e-05, 'epoch': 1.64}

 16%|█▋        | 6785/41250 [16:23:39<83:19:45,  8.70s/it][2025-04-26 00:21:22,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:21:22,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5755.54 | bwd_inner_microstep: 5681.73 | bwd_allreduce_microstep: 73.77 | step_microstep: 18.55
[2025-04-26 00:21:22,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5755.55 | bwd_inner: 5681.73 | bwd_allreduce: 73.78 | step: 18.55

 16%|█▋        | 6786/41250 [16:23:48<83:17:54,  8.70s/it]
                                                          
{'loss': 0.1716, 'grad_norm': 1.9693660736083984, 'learning_rate': 3.8132271298739814e-05, 'epoch': 1.65}

 16%|█▋        | 6786/41250 [16:23:48<83:17:54,  8.70s/it][2025-04-26 00:21:31,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.30 | optimizer_step: 0.90
[2025-04-26 00:21:31,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.66 | bwd_microstep: 5703.60 | bwd_inner_microstep: 5647.53 | bwd_allreduce_microstep: 56.01 | step_microstep: 19.57
[2025-04-26 00:21:31,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.66 | bwd: 5703.61 | bwd_inner: 5647.53 | bwd_allreduce: 56.03 | step: 19.58

 16%|█▋        | 6787/41250 [16:23:56<83:03:44,  8.68s/it]
                                                          
{'loss': 0.1512, 'grad_norm': 2.757805347442627, 'learning_rate': 3.8131608625434405e-05, 'epoch': 1.65}

 16%|█▋        | 6787/41250 [16:23:56<83:03:44,  8.68s/it][2025-04-26 00:21:40,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.96
[2025-04-26 00:21:40,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.70 | bwd_microstep: 5892.43 | bwd_inner_microstep: 5658.09 | bwd_allreduce_microstep: 234.30 | step_microstep: 18.93
[2025-04-26 00:21:40,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.70 | bwd: 5892.44 | bwd_inner: 5658.09 | bwd_allreduce: 234.31 | step: 18.94

 16%|█▋        | 6788/41250 [16:24:05<83:25:30,  8.71s/it]
                                                          
{'loss': 0.0708, 'grad_norm': 2.520204544067383, 'learning_rate': 3.81309458403512e-05, 'epoch': 1.65}

 16%|█▋        | 6788/41250 [16:24:05<83:25:30,  8.71s/it][2025-04-26 00:21:48,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:21:48,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.80 | bwd_microstep: 5726.55 | bwd_inner_microstep: 5646.01 | bwd_allreduce_microstep: 80.49 | step_microstep: 18.71
[2025-04-26 00:21:48,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.80 | bwd: 5726.56 | bwd_inner: 5646.01 | bwd_allreduce: 80.51 | step: 18.71

 16%|█▋        | 6789/41250 [16:24:14<83:11:50,  8.69s/it]
                                                          
{'loss': 0.0594, 'grad_norm': 1.1258608102798462, 'learning_rate': 3.813028294349427e-05, 'epoch': 1.65}

 16%|█▋        | 6789/41250 [16:24:14<83:11:50,  8.69s/it][2025-04-26 00:21:57,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 00:21:57,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.24 | bwd_microstep: 5755.00 | bwd_inner_microstep: 5662.87 | bwd_allreduce_microstep: 92.08 | step_microstep: 19.11
[2025-04-26 00:21:57,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.24 | bwd: 5755.01 | bwd_inner: 5662.87 | bwd_allreduce: 92.10 | step: 19.11

 16%|█▋        | 6790/41250 [16:24:22<83:06:56,  8.68s/it]
                                                          
{'loss': 0.0263, 'grad_norm': 0.805087149143219, 'learning_rate': 3.8129619934867716e-05, 'epoch': 1.65}

 16%|█▋        | 6790/41250 [16:24:22<83:06:56,  8.68s/it][2025-04-26 00:22:06,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 00:22:06,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.79 | bwd_microstep: 5714.38 | bwd_inner_microstep: 5650.31 | bwd_allreduce_microstep: 64.03 | step_microstep: 18.68
[2025-04-26 00:22:06,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5714.40 | bwd_inner: 5650.31 | bwd_allreduce: 64.04 | step: 18.69

 16%|█▋        | 6791/41250 [16:24:31<82:57:01,  8.67s/it]
                                                          
{'loss': 0.4458, 'grad_norm': 8.807673454284668, 'learning_rate': 3.8128956814475616e-05, 'epoch': 1.65}

 16%|█▋        | 6791/41250 [16:24:31<82:57:01,  8.67s/it][2025-04-26 00:22:14,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:22:14,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.69 | bwd_microstep: 5707.45 | bwd_inner_microstep: 5694.94 | bwd_allreduce_microstep: 12.46 | step_microstep: 18.53
[2025-04-26 00:22:14,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.69 | bwd: 5707.46 | bwd_inner: 5694.94 | bwd_allreduce: 12.48 | step: 18.54

 16%|█▋        | 6792/41250 [16:24:40<82:53:44,  8.66s/it]
                                                          
{'loss': 0.1195, 'grad_norm': 2.455313205718994, 'learning_rate': 3.812829358232207e-05, 'epoch': 1.65}

 16%|█▋        | 6792/41250 [16:24:40<82:53:44,  8.66s/it][2025-04-26 00:22:23,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 00:22:23,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5715.01 | bwd_inner_microstep: 5650.29 | bwd_allreduce_microstep: 64.67 | step_microstep: 18.62
[2025-04-26 00:22:23,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5715.02 | bwd_inner: 5650.29 | bwd_allreduce: 64.68 | step: 18.62

 16%|█▋        | 6793/41250 [16:24:48<82:47:11,  8.65s/it]
                                                          
{'loss': 0.0194, 'grad_norm': 0.40992021560668945, 'learning_rate': 3.812763023841116e-05, 'epoch': 1.65}

 16%|█▋        | 6793/41250 [16:24:48<82:47:11,  8.65s/it][2025-04-26 00:22:32,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:22:32,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.93 | bwd_microstep: 5751.31 | bwd_inner_microstep: 5643.85 | bwd_allreduce_microstep: 107.41 | step_microstep: 18.86
[2025-04-26 00:22:32,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.93 | bwd: 5751.32 | bwd_inner: 5643.85 | bwd_allreduce: 107.43 | step: 18.86

 16%|█▋        | 6794/41250 [16:24:57<82:48:48,  8.65s/it]
                                                          
{'loss': 0.1907, 'grad_norm': 28.006553649902344, 'learning_rate': 3.812696678274697e-05, 'epoch': 1.65}

 16%|█▋        | 6794/41250 [16:24:57<82:48:48,  8.65s/it][2025-04-26 00:22:40,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:22:40,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.48 | bwd_microstep: 5774.13 | bwd_inner_microstep: 5761.48 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.69
[2025-04-26 00:22:40,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.48 | bwd: 5774.14 | bwd_inner: 5761.48 | bwd_allreduce: 12.62 | step: 18.70

 16%|█▋        | 6795/41250 [16:25:06<83:03:34,  8.68s/it]
                                                          
{'loss': 0.1829, 'grad_norm': 1.4405107498168945, 'learning_rate': 3.81263032153336e-05, 'epoch': 1.65}

 16%|█▋        | 6795/41250 [16:25:06<83:03:34,  8.68s/it][2025-04-26 00:22:49,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 00:22:49,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.71 | bwd_microstep: 5721.45 | bwd_inner_microstep: 5708.35 | bwd_allreduce_microstep: 13.05 | step_microstep: 18.53
[2025-04-26 00:22:49,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.71 | bwd: 5721.47 | bwd_inner: 5708.35 | bwd_allreduce: 13.07 | step: 18.53

 16%|█▋        | 6796/41250 [16:25:14<83:00:26,  8.67s/it]
                                                          
{'loss': 0.0733, 'grad_norm': 2.497864007949829, 'learning_rate': 3.8125639536175135e-05, 'epoch': 1.65}

 16%|█▋        | 6796/41250 [16:25:14<83:00:26,  8.67s/it][2025-04-26 00:22:58,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:22:58,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5753.35 | bwd_inner_microstep: 5688.15 | bwd_allreduce_microstep: 65.15 | step_microstep: 17.92
[2025-04-26 00:22:58,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5753.36 | bwd_inner: 5688.15 | bwd_allreduce: 65.17 | step: 17.93

 16%|█▋        | 6797/41250 [16:25:23<83:01:35,  8.68s/it]
                                                          
{'loss': 0.0943, 'grad_norm': 1.5368543863296509, 'learning_rate': 3.812497574527567e-05, 'epoch': 1.65}

 16%|█▋        | 6797/41250 [16:25:23<83:01:35,  8.68s/it][2025-04-26 00:23:06,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.24 | optimizer_step: 1.03
[2025-04-26 00:23:06,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.63 | bwd_microstep: 5787.51 | bwd_inner_microstep: 5637.12 | bwd_allreduce_microstep: 150.35 | step_microstep: 19.72
[2025-04-26 00:23:06,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.63 | bwd: 5787.53 | bwd_inner: 5637.12 | bwd_allreduce: 150.37 | step: 19.72

 16%|█▋        | 6798/41250 [16:25:32<83:04:14,  8.68s/it]
                                                          
{'loss': 0.2718, 'grad_norm': 1.5870431661605835, 'learning_rate': 3.8124311842639286e-05, 'epoch': 1.65}

 16%|█▋        | 6798/41250 [16:25:32<83:04:14,  8.68s/it][2025-04-26 00:23:15,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:23:15,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.89 | bwd_microstep: 5704.69 | bwd_inner_microstep: 5691.80 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.26
[2025-04-26 00:23:15,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.89 | bwd: 5704.70 | bwd_inner: 5691.80 | bwd_allreduce: 12.86 | step: 18.27

 16%|█▋        | 6799/41250 [16:25:40<82:55:41,  8.67s/it]
                                                          
{'loss': 0.1132, 'grad_norm': 2.7401764392852783, 'learning_rate': 3.812364782827009e-05, 'epoch': 1.65}

 16%|█▋        | 6799/41250 [16:25:40<82:55:41,  8.67s/it][2025-04-26 00:23:24,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:23:24,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.78 | bwd_microstep: 5739.22 | bwd_inner_microstep: 5679.87 | bwd_allreduce_microstep: 59.30 | step_microstep: 18.43
[2025-04-26 00:23:24,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.78 | bwd: 5739.23 | bwd_inner: 5679.87 | bwd_allreduce: 59.32 | step: 18.43

 16%|█▋        | 6800/41250 [16:25:49<82:56:09,  8.67s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 1.6352832317352295, 'learning_rate': 3.812298370217217e-05, 'epoch': 1.65}

 16%|█▋        | 6800/41250 [16:25:49<82:56:09,  8.67s/it][2025-04-26 00:23:32,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:23:32,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5780.18 | bwd_inner_microstep: 5648.97 | bwd_allreduce_microstep: 131.16 | step_microstep: 18.66
[2025-04-26 00:23:32,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5780.19 | bwd_inner: 5648.97 | bwd_allreduce: 131.18 | step: 18.67

 16%|█▋        | 6801/41250 [16:25:58<82:58:55,  8.67s/it]
                                                          
{'loss': 0.1891, 'grad_norm': 2.41806960105896, 'learning_rate': 3.812231946434961e-05, 'epoch': 1.65}

 16%|█▋        | 6801/41250 [16:25:58<82:58:55,  8.67s/it][2025-04-26 00:23:41,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:23:41,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5742.82 | bwd_inner_microstep: 5634.62 | bwd_allreduce_microstep: 108.16 | step_microstep: 18.56
[2025-04-26 00:23:41,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5742.83 | bwd_inner: 5634.61 | bwd_allreduce: 108.18 | step: 18.56

 16%|█▋        | 6802/41250 [16:26:06<82:54:58,  8.67s/it]
                                                          
{'loss': 0.095, 'grad_norm': 1.1496381759643555, 'learning_rate': 3.812165511480653e-05, 'epoch': 1.65}

 16%|█▋        | 6802/41250 [16:26:06<82:54:58,  8.67s/it][2025-04-26 00:23:50,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:23:50,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.47 | bwd_microstep: 5755.93 | bwd_inner_microstep: 5677.26 | bwd_allreduce_microstep: 78.63 | step_microstep: 18.48
[2025-04-26 00:23:50,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.47 | bwd: 5755.95 | bwd_inner: 5677.26 | bwd_allreduce: 78.65 | step: 18.48

 16%|█▋        | 6803/41250 [16:26:15<82:58:11,  8.67s/it]
                                                          
{'loss': 0.1978, 'grad_norm': 2.161158323287964, 'learning_rate': 3.8120990653547e-05, 'epoch': 1.65}

 16%|█▋        | 6803/41250 [16:26:15<82:58:11,  8.67s/it][2025-04-26 00:23:59,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 1.10
[2025-04-26 00:23:59,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.27 | bwd_microstep: 5876.07 | bwd_inner_microstep: 5669.36 | bwd_allreduce_microstep: 206.66 | step_microstep: 19.19
[2025-04-26 00:23:59,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.27 | bwd: 5876.09 | bwd_inner: 5669.36 | bwd_allreduce: 206.68 | step: 19.19

 16%|█▋        | 6804/41250 [16:26:24<83:19:33,  8.71s/it]
                                                          
{'loss': 0.0684, 'grad_norm': 1.6350629329681396, 'learning_rate': 3.812032608057513e-05, 'epoch': 1.65}

 16%|█▋        | 6804/41250 [16:26:24<83:19:33,  8.71s/it][2025-04-26 00:24:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-26 00:24:07,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.28 | bwd_microstep: 5872.23 | bwd_inner_microstep: 5631.50 | bwd_allreduce_microstep: 240.69 | step_microstep: 18.33
[2025-04-26 00:24:07,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.28 | bwd: 5872.25 | bwd_inner: 5631.50 | bwd_allreduce: 240.70 | step: 18.34

 16%|█▋        | 6805/41250 [16:26:33<83:29:45,  8.73s/it]
                                                          
{'loss': 0.0644, 'grad_norm': 2.1302621364593506, 'learning_rate': 3.811966139589502e-05, 'epoch': 1.65}

 16%|█▋        | 6805/41250 [16:26:33<83:29:45,  8.73s/it][2025-04-26 00:24:16,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.86 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:24:16,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.65 | bwd_microstep: 5794.08 | bwd_inner_microstep: 5625.53 | bwd_allreduce_microstep: 168.51 | step_microstep: 17.83
[2025-04-26 00:24:16,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.65 | bwd: 5794.10 | bwd_inner: 5625.53 | bwd_allreduce: 168.53 | step: 17.83

 16%|█▋        | 6806/41250 [16:26:41<83:23:32,  8.72s/it]
                                                          
{'loss': 0.1323, 'grad_norm': 4.455916404724121, 'learning_rate': 3.811899659951076e-05, 'epoch': 1.65}

 16%|█▋        | 6806/41250 [16:26:41<83:23:32,  8.72s/it][2025-04-26 00:24:25,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 00:24:25,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.39 | bwd_microstep: 5747.10 | bwd_inner_microstep: 5676.43 | bwd_allreduce_microstep: 70.63 | step_microstep: 18.62
[2025-04-26 00:24:25,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.39 | bwd: 5747.12 | bwd_inner: 5676.43 | bwd_allreduce: 70.65 | step: 18.63

 17%|█▋        | 6807/41250 [16:26:50<83:15:39,  8.70s/it]
                                                          
{'loss': 0.0209, 'grad_norm': 0.5975077152252197, 'learning_rate': 3.8118331691426444e-05, 'epoch': 1.65}

 17%|█▋        | 6807/41250 [16:26:50<83:15:39,  8.70s/it][2025-04-26 00:24:33,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:24:33,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5757.13 | bwd_inner_microstep: 5676.87 | bwd_allreduce_microstep: 80.21 | step_microstep: 18.46
[2025-04-26 00:24:33,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5757.15 | bwd_inner: 5676.87 | bwd_allreduce: 80.23 | step: 18.46

 17%|█▋        | 6808/41250 [16:26:59<83:11:51,  8.70s/it]
                                                          
{'loss': 0.0426, 'grad_norm': 1.2046566009521484, 'learning_rate': 3.811766667164618e-05, 'epoch': 1.65}

 17%|█▋        | 6808/41250 [16:26:59<83:11:51,  8.70s/it][2025-04-26 00:24:42,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 00:24:42,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.44 | bwd_microstep: 5746.63 | bwd_inner_microstep: 5695.97 | bwd_allreduce_microstep: 50.62 | step_microstep: 18.73
[2025-04-26 00:24:42,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.44 | bwd: 5746.65 | bwd_inner: 5695.97 | bwd_allreduce: 50.64 | step: 18.74

 17%|█▋        | 6809/41250 [16:27:07<83:09:42,  8.69s/it]
                                                          
{'loss': 0.0544, 'grad_norm': 1.442014455795288, 'learning_rate': 3.811700154017406e-05, 'epoch': 1.65}

 17%|█▋        | 6809/41250 [16:27:07<83:09:42,  8.69s/it][2025-04-26 00:24:51,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:24:51,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.61 | bwd_microstep: 5716.94 | bwd_inner_microstep: 5692.39 | bwd_allreduce_microstep: 24.50 | step_microstep: 18.86
[2025-04-26 00:24:51,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.61 | bwd: 5716.95 | bwd_inner: 5692.39 | bwd_allreduce: 24.52 | step: 18.86

 17%|█▋        | 6810/41250 [16:27:16<83:03:10,  8.68s/it]
                                                          
{'loss': 0.1038, 'grad_norm': 3.0665338039398193, 'learning_rate': 3.8116336297014195e-05, 'epoch': 1.65}

 17%|█▋        | 6810/41250 [16:27:16<83:03:10,  8.68s/it][2025-04-26 00:24:59,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:24:59,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.90 | bwd_microstep: 5711.64 | bwd_inner_microstep: 5698.89 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.52
[2025-04-26 00:24:59,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.90 | bwd: 5711.65 | bwd_inner: 5698.89 | bwd_allreduce: 12.72 | step: 18.53

 17%|█▋        | 6811/41250 [16:27:25<82:56:34,  8.67s/it]
                                                          
{'loss': 0.1365, 'grad_norm': 1.8964636325836182, 'learning_rate': 3.811567094217068e-05, 'epoch': 1.65}

 17%|█▋        | 6811/41250 [16:27:25<82:56:34,  8.67s/it][2025-04-26 00:25:08,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.09 | optimizer_step: 0.98
[2025-04-26 00:25:08,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.06 | bwd_microstep: 5906.02 | bwd_inner_microstep: 5633.94 | bwd_allreduce_microstep: 272.03 | step_microstep: 18.74
[2025-04-26 00:25:08,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.06 | bwd: 5906.03 | bwd_inner: 5633.94 | bwd_allreduce: 272.05 | step: 18.74

 17%|█▋        | 6812/41250 [16:27:34<83:21:36,  8.71s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 1.6985102891921997, 'learning_rate': 3.8115005475647615e-05, 'epoch': 1.65}

 17%|█▋        | 6812/41250 [16:27:34<83:21:36,  8.71s/it][2025-04-26 00:25:17,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 00:25:17,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.75 | bwd_microstep: 5687.87 | bwd_inner_microstep: 5647.12 | bwd_allreduce_microstep: 40.70 | step_microstep: 18.73
[2025-04-26 00:25:17,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.75 | bwd: 5687.88 | bwd_inner: 5647.12 | bwd_allreduce: 40.72 | step: 18.74

 17%|█▋        | 6813/41250 [16:27:42<83:01:36,  8.68s/it]
                                                          
{'loss': 0.1363, 'grad_norm': 1.9155409336090088, 'learning_rate': 3.81143398974491e-05, 'epoch': 1.65}

 17%|█▋        | 6813/41250 [16:27:42<83:01:36,  8.68s/it][2025-04-26 00:25:25,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 00:25:25,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5719.41 | bwd_inner_microstep: 5706.36 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.79
[2025-04-26 00:25:25,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.78 | bwd: 5719.43 | bwd_inner: 5706.36 | bwd_allreduce: 13.01 | step: 18.79

 17%|█▋        | 6814/41250 [16:27:51<82:58:01,  8.67s/it]
                                                          
{'loss': 0.1328, 'grad_norm': 2.841989517211914, 'learning_rate': 3.8113674207579243e-05, 'epoch': 1.65}

 17%|█▋        | 6814/41250 [16:27:51<82:58:01,  8.67s/it][2025-04-26 00:25:34,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:25:34,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5758.06 | bwd_inner_microstep: 5704.71 | bwd_allreduce_microstep: 53.31 | step_microstep: 18.62
[2025-04-26 00:25:34,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5758.07 | bwd_inner: 5704.71 | bwd_allreduce: 53.33 | step: 18.62

 17%|█▋        | 6815/41250 [16:27:59<83:01:06,  8.68s/it]
                                                          
{'loss': 0.1093, 'grad_norm': 2.2548348903656006, 'learning_rate': 3.811300840604216e-05, 'epoch': 1.65}

 17%|█▋        | 6815/41250 [16:27:59<83:01:06,  8.68s/it][2025-04-26 00:25:43,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:25:43,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.49 | bwd_microstep: 5714.33 | bwd_inner_microstep: 5701.48 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.83
[2025-04-26 00:25:43,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.49 | bwd: 5714.35 | bwd_inner: 5701.48 | bwd_allreduce: 12.82 | step: 18.83

 17%|█▋        | 6816/41250 [16:28:08<82:56:46,  8.67s/it]
                                                          
{'loss': 0.1228, 'grad_norm': 1.9409748315811157, 'learning_rate': 3.811234249284193e-05, 'epoch': 1.65}

 17%|█▋        | 6816/41250 [16:28:08<82:56:46,  8.67s/it][2025-04-26 00:25:51,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 00:25:51,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5784.50 | bwd_inner_microstep: 5659.50 | bwd_allreduce_microstep: 124.95 | step_microstep: 19.38
[2025-04-26 00:25:51,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5784.51 | bwd_inner: 5659.50 | bwd_allreduce: 124.97 | step: 19.38

 17%|█▋        | 6817/41250 [16:28:17<83:01:19,  8.68s/it]
                                                          
{'loss': 0.044, 'grad_norm': 1.197939157485962, 'learning_rate': 3.811167646798267e-05, 'epoch': 1.65}

 17%|█▋        | 6817/41250 [16:28:17<83:01:19,  8.68s/it][2025-04-26 00:26:00,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:26:00,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.80 | bwd_microstep: 5696.87 | bwd_inner_microstep: 5676.29 | bwd_allreduce_microstep: 20.54 | step_microstep: 18.78
[2025-04-26 00:26:00,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.80 | bwd: 5696.88 | bwd_inner: 5676.29 | bwd_allreduce: 20.55 | step: 18.78

 17%|█▋        | 6818/41250 [16:28:25<82:51:28,  8.66s/it]
                                                          
{'loss': 0.3802, 'grad_norm': 1.9465070962905884, 'learning_rate': 3.81110103314685e-05, 'epoch': 1.65}

 17%|█▋        | 6818/41250 [16:28:25<82:51:28,  8.66s/it][2025-04-26 00:26:09,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:26:09,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.22 | bwd_microstep: 5808.33 | bwd_inner_microstep: 5657.05 | bwd_allreduce_microstep: 151.22 | step_microstep: 19.04
[2025-04-26 00:26:09,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.22 | bwd: 5808.35 | bwd_inner: 5657.05 | bwd_allreduce: 151.24 | step: 19.05

 17%|█▋        | 6819/41250 [16:28:34<83:03:07,  8.68s/it]
                                                          
{'loss': 0.2203, 'grad_norm': 2.119314193725586, 'learning_rate': 3.8110344083303495e-05, 'epoch': 1.65}

 17%|█▋        | 6819/41250 [16:28:34<83:03:07,  8.68s/it][2025-04-26 00:26:18,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:26:18,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5767.85 | bwd_inner_microstep: 5644.20 | bwd_allreduce_microstep: 123.62 | step_microstep: 18.04
[2025-04-26 00:26:18,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5767.87 | bwd_inner: 5644.19 | bwd_allreduce: 123.63 | step: 18.04

 17%|█▋        | 6820/41250 [16:28:43<83:01:54,  8.68s/it]
                                                          
{'loss': 0.1831, 'grad_norm': 2.146859884262085, 'learning_rate': 3.810967772349179e-05, 'epoch': 1.65}

 17%|█▋        | 6820/41250 [16:28:43<83:01:54,  8.68s/it][2025-04-26 00:26:26,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:26:26,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2871.34 | bwd_microstep: 5721.30 | bwd_inner_microstep: 5708.64 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.50
[2025-04-26 00:26:26,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2871.34 | bwd: 5721.31 | bwd_inner: 5708.64 | bwd_allreduce: 12.63 | step: 18.50

 17%|█▋        | 6821/41250 [16:28:52<83:00:40,  8.68s/it]
                                                          
{'loss': 0.3641, 'grad_norm': 3.448904514312744, 'learning_rate': 3.8109011252037487e-05, 'epoch': 1.65}

 17%|█▋        | 6821/41250 [16:28:52<83:00:40,  8.68s/it][2025-04-26 00:26:35,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-26 00:26:35,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5789.29 | bwd_inner_microstep: 5697.74 | bwd_allreduce_microstep: 91.50 | step_microstep: 19.14
[2025-04-26 00:26:35,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5789.30 | bwd_inner: 5697.74 | bwd_allreduce: 91.52 | step: 19.15

 17%|█▋        | 6822/41250 [16:29:00<83:07:15,  8.69s/it]
                                                          
{'loss': 0.1253, 'grad_norm': 2.498757839202881, 'learning_rate': 3.810834466894469e-05, 'epoch': 1.65}

 17%|█▋        | 6822/41250 [16:29:00<83:07:15,  8.69s/it][2025-04-26 00:26:44,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:26:44,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.98 | bwd_microstep: 5772.84 | bwd_inner_microstep: 5701.86 | bwd_allreduce_microstep: 70.94 | step_microstep: 18.19
[2025-04-26 00:26:44,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.98 | bwd: 5772.85 | bwd_inner: 5701.86 | bwd_allreduce: 70.96 | step: 18.19

 17%|█▋        | 6823/41250 [16:29:09<83:09:21,  8.70s/it]
                                                          
{'loss': 0.0992, 'grad_norm': 1.4623663425445557, 'learning_rate': 3.810767797421751e-05, 'epoch': 1.65}

 17%|█▋        | 6823/41250 [16:29:09<83:09:21,  8.70s/it][2025-04-26 00:26:52,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 00:26:52,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.65 | bwd_microstep: 5769.47 | bwd_inner_microstep: 5667.12 | bwd_allreduce_microstep: 102.29 | step_microstep: 19.06
[2025-04-26 00:26:52,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.65 | bwd: 5769.48 | bwd_inner: 5667.12 | bwd_allreduce: 102.32 | step: 19.07

 17%|█▋        | 6824/41250 [16:29:18<83:08:09,  8.69s/it]
                                                          
{'loss': 0.1744, 'grad_norm': 1.8846700191497803, 'learning_rate': 3.810701116786005e-05, 'epoch': 1.65}

 17%|█▋        | 6824/41250 [16:29:18<83:08:09,  8.69s/it][2025-04-26 00:27:01,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:27:01,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.32 | bwd_microstep: 5717.32 | bwd_inner_microstep: 5648.95 | bwd_allreduce_microstep: 68.33 | step_microstep: 18.33
[2025-04-26 00:27:01,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.32 | bwd: 5717.33 | bwd_inner: 5648.95 | bwd_allreduce: 68.34 | step: 18.33

 17%|█▋        | 6825/41250 [16:29:26<82:59:13,  8.68s/it]
                                                          
{'loss': 0.0298, 'grad_norm': 0.34940552711486816, 'learning_rate': 3.810634424987644e-05, 'epoch': 1.65}

 17%|█▋        | 6825/41250 [16:29:26<82:59:13,  8.68s/it][2025-04-26 00:27:10,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:27:10,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.34 | bwd_microstep: 5770.12 | bwd_inner_microstep: 5652.19 | bwd_allreduce_microstep: 117.89 | step_microstep: 18.78
[2025-04-26 00:27:10,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.34 | bwd: 5770.13 | bwd_inner: 5652.19 | bwd_allreduce: 117.90 | step: 18.78

 17%|█▋        | 6826/41250 [16:29:35<82:59:23,  8.68s/it]
                                                          
{'loss': 0.1156, 'grad_norm': 1.1530327796936035, 'learning_rate': 3.8105677220270776e-05, 'epoch': 1.65}

 17%|█▋        | 6826/41250 [16:29:35<82:59:23,  8.68s/it][2025-04-26 00:27:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:27:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 5916.63 | bwd_inner_microstep: 5653.71 | bwd_allreduce_microstep: 262.87 | step_microstep: 18.75
[2025-04-26 00:27:18,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 5916.64 | bwd_inner: 5653.71 | bwd_allreduce: 262.89 | step: 18.75

 17%|█▋        | 6827/41250 [16:29:44<83:25:15,  8.72s/it]
                                                          
{'loss': 0.0736, 'grad_norm': 1.4110562801361084, 'learning_rate': 3.8105010079047164e-05, 'epoch': 1.66}

 17%|█▋        | 6827/41250 [16:29:44<83:25:15,  8.72s/it][2025-04-26 00:27:27,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 00:27:27,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5745.36 | bwd_inner_microstep: 5719.32 | bwd_allreduce_microstep: 25.99 | step_microstep: 19.69
[2025-04-26 00:27:27,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5745.37 | bwd_inner: 5719.32 | bwd_allreduce: 26.01 | step: 19.69

 17%|█▋        | 6828/41250 [16:29:52<83:18:55,  8.71s/it]
                                                          
{'loss': 0.027, 'grad_norm': 0.564909040927887, 'learning_rate': 3.810434282620973e-05, 'epoch': 1.66}

 17%|█▋        | 6828/41250 [16:29:52<83:18:55,  8.71s/it][2025-04-26 00:27:36,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 00:27:36,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.27 | bwd_microstep: 5710.58 | bwd_inner_microstep: 5656.38 | bwd_allreduce_microstep: 54.16 | step_microstep: 19.63
[2025-04-26 00:27:36,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.27 | bwd: 5710.60 | bwd_inner: 5656.38 | bwd_allreduce: 54.18 | step: 19.63

 17%|█▋        | 6829/41250 [16:30:01<83:03:55,  8.69s/it]
                                                          
{'loss': 0.1588, 'grad_norm': 2.369352340698242, 'learning_rate': 3.8103675461762584e-05, 'epoch': 1.66}

 17%|█▋        | 6829/41250 [16:30:01<83:03:55,  8.69s/it][2025-04-26 00:27:44,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 00:27:44,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.00 | bwd_microstep: 5778.16 | bwd_inner_microstep: 5653.89 | bwd_allreduce_microstep: 124.22 | step_microstep: 19.14
[2025-04-26 00:27:44,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.00 | bwd: 5778.17 | bwd_inner: 5653.89 | bwd_allreduce: 124.24 | step: 19.15

 17%|█▋        | 6830/41250 [16:30:10<83:05:00,  8.69s/it]
                                                          
{'loss': 0.1972, 'grad_norm': 2.595562219619751, 'learning_rate': 3.810300798570984e-05, 'epoch': 1.66}

 17%|█▋        | 6830/41250 [16:30:10<83:05:00,  8.69s/it][2025-04-26 00:27:53,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:27:53,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5801.25 | bwd_inner_microstep: 5654.41 | bwd_allreduce_microstep: 146.80 | step_microstep: 18.84
[2025-04-26 00:27:53,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5801.26 | bwd_inner: 5654.41 | bwd_allreduce: 146.82 | step: 18.84

 17%|█▋        | 6831/41250 [16:30:19<83:09:00,  8.70s/it]
                                                          
{'loss': 0.1909, 'grad_norm': 1.4146413803100586, 'learning_rate': 3.810234039805561e-05, 'epoch': 1.66}

 17%|█▋        | 6831/41250 [16:30:19<83:09:00,  8.70s/it][2025-04-26 00:28:02,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 00:28:02,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.69 | bwd_microstep: 5721.77 | bwd_inner_microstep: 5709.01 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.45
[2025-04-26 00:28:02,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.69 | bwd: 5721.79 | bwd_inner: 5709.01 | bwd_allreduce: 12.74 | step: 19.45

 17%|█▋        | 6832/41250 [16:30:27<83:02:54,  8.69s/it]
                                                          
{'loss': 0.1769, 'grad_norm': 2.5033743381500244, 'learning_rate': 3.8101672698804016e-05, 'epoch': 1.66}

 17%|█▋        | 6832/41250 [16:30:27<83:02:54,  8.69s/it][2025-04-26 00:28:11,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-26 00:28:11,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.24 | bwd_microstep: 5766.38 | bwd_inner_microstep: 5660.12 | bwd_allreduce_microstep: 106.21 | step_microstep: 19.17
[2025-04-26 00:28:11,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.24 | bwd: 5766.40 | bwd_inner: 5660.12 | bwd_allreduce: 106.23 | step: 19.17

 17%|█▋        | 6833/41250 [16:30:36<83:01:52,  8.69s/it]
                                                          
{'loss': 0.0132, 'grad_norm': 0.3386911153793335, 'learning_rate': 3.810100488795917e-05, 'epoch': 1.66}

 17%|█▋        | 6833/41250 [16:30:36<83:01:52,  8.69s/it][2025-04-26 00:28:19,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:28:19,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5713.21 | bwd_inner_microstep: 5647.93 | bwd_allreduce_microstep: 65.23 | step_microstep: 18.72
[2025-04-26 00:28:19,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5713.22 | bwd_inner: 5647.93 | bwd_allreduce: 65.24 | step: 18.72

 17%|█▋        | 6834/41250 [16:30:44<82:50:54,  8.67s/it]
                                                          
{'loss': 0.0689, 'grad_norm': 1.3846358060836792, 'learning_rate': 3.810033696552519e-05, 'epoch': 1.66}

 17%|█▋        | 6834/41250 [16:30:44<82:50:54,  8.67s/it][2025-04-26 00:28:28,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.14 | optimizer_step: 0.91
[2025-04-26 00:28:28,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5771.77 | bwd_inner_microstep: 5654.83 | bwd_allreduce_microstep: 116.88 | step_microstep: 18.80
[2025-04-26 00:28:28,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5771.78 | bwd_inner: 5654.83 | bwd_allreduce: 116.90 | step: 18.80

 17%|█▋        | 6835/41250 [16:30:53<82:53:12,  8.67s/it]
                                                          
{'loss': 0.1618, 'grad_norm': 1.5245182514190674, 'learning_rate': 3.809966893150619e-05, 'epoch': 1.66}

 17%|█▋        | 6835/41250 [16:30:53<82:53:12,  8.67s/it][2025-04-26 00:28:37,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 00:28:37,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.77 | bwd_microstep: 5764.41 | bwd_inner_microstep: 5653.01 | bwd_allreduce_microstep: 111.35 | step_microstep: 18.88
[2025-04-26 00:28:37,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.77 | bwd: 5764.42 | bwd_inner: 5653.01 | bwd_allreduce: 111.37 | step: 18.88

 17%|█▋        | 6836/41250 [16:31:02<82:55:14,  8.67s/it]
                                                          
{'loss': 0.0599, 'grad_norm': 1.1994946002960205, 'learning_rate': 3.8099000785906295e-05, 'epoch': 1.66}

 17%|█▋        | 6836/41250 [16:31:02<82:55:14,  8.67s/it][2025-04-26 00:28:45,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.11 | optimizer_step: 1.05
[2025-04-26 00:28:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.28 | bwd_microstep: 5732.73 | bwd_inner_microstep: 5692.78 | bwd_allreduce_microstep: 39.89 | step_microstep: 20.00
[2025-04-26 00:28:45,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.28 | bwd: 5732.75 | bwd_inner: 5692.78 | bwd_allreduce: 39.92 | step: 20.00

 17%|█▋        | 6837/41250 [16:31:11<82:53:48,  8.67s/it]
                                                          
{'loss': 0.0474, 'grad_norm': 0.8227823972702026, 'learning_rate': 3.8098332528729616e-05, 'epoch': 1.66}

 17%|█▋        | 6837/41250 [16:31:11<82:53:48,  8.67s/it][2025-04-26 00:28:54,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 00:28:54,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.37 | bwd_microstep: 5755.49 | bwd_inner_microstep: 5654.49 | bwd_allreduce_microstep: 100.95 | step_microstep: 18.91
[2025-04-26 00:28:54,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.37 | bwd: 5755.50 | bwd_inner: 5654.49 | bwd_allreduce: 100.97 | step: 18.92

 17%|█▋        | 6838/41250 [16:31:19<82:52:17,  8.67s/it]
                                                          
{'loss': 0.1651, 'grad_norm': 2.268512487411499, 'learning_rate': 3.809766415998028e-05, 'epoch': 1.66}

 17%|█▋        | 6838/41250 [16:31:19<82:52:17,  8.67s/it][2025-04-26 00:29:03,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:29:03,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.31 | bwd_microstep: 5714.93 | bwd_inner_microstep: 5698.61 | bwd_allreduce_microstep: 16.28 | step_microstep: 18.70
[2025-04-26 00:29:03,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.31 | bwd: 5714.95 | bwd_inner: 5698.61 | bwd_allreduce: 16.30 | step: 18.70

 17%|█▋        | 6839/41250 [16:31:28<82:49:05,  8.66s/it]
                                                          
{'loss': 0.0697, 'grad_norm': 1.0917353630065918, 'learning_rate': 3.80969956796624e-05, 'epoch': 1.66}

 17%|█▋        | 6839/41250 [16:31:28<82:49:05,  8.66s/it][2025-04-26 00:29:11,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 00:29:11,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.33 | bwd_microstep: 5766.65 | bwd_inner_microstep: 5643.50 | bwd_allreduce_microstep: 123.10 | step_microstep: 19.03
[2025-04-26 00:29:11,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.33 | bwd: 5766.66 | bwd_inner: 5643.50 | bwd_allreduce: 123.12 | step: 19.03

 17%|█▋        | 6840/41250 [16:31:37<82:52:06,  8.67s/it]
                                                          
{'loss': 0.0811, 'grad_norm': 1.3762931823730469, 'learning_rate': 3.80963270877801e-05, 'epoch': 1.66}

 17%|█▋        | 6840/41250 [16:31:37<82:52:06,  8.67s/it][2025-04-26 00:29:20,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:29:20,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.73 | bwd_microstep: 5747.44 | bwd_inner_microstep: 5688.63 | bwd_allreduce_microstep: 58.76 | step_microstep: 18.63
[2025-04-26 00:29:20,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.74 | bwd: 5747.45 | bwd_inner: 5688.63 | bwd_allreduce: 58.77 | step: 18.63

 17%|█▋        | 6841/41250 [16:31:45<82:52:40,  8.67s/it]
                                                          
{'loss': 0.1241, 'grad_norm': 1.3006565570831299, 'learning_rate': 3.809565838433751e-05, 'epoch': 1.66}

 17%|█▋        | 6841/41250 [16:31:45<82:52:40,  8.67s/it][2025-04-26 00:29:28,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:29:28,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.74 | bwd_microstep: 5688.72 | bwd_inner_microstep: 5675.96 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.79
[2025-04-26 00:29:28,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.74 | bwd: 5688.73 | bwd_inner: 5675.96 | bwd_allreduce: 12.72 | step: 18.79

 17%|█▋        | 6842/41250 [16:31:54<82:43:05,  8.65s/it]
                                                          
{'loss': 0.2234, 'grad_norm': 3.25276255607605, 'learning_rate': 3.8094989569338735e-05, 'epoch': 1.66}

 17%|█▋        | 6842/41250 [16:31:54<82:43:05,  8.65s/it][2025-04-26 00:29:37,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 0.95
[2025-04-26 00:29:37,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.56 | bwd_microstep: 5748.90 | bwd_inner_microstep: 5643.42 | bwd_allreduce_microstep: 105.43 | step_microstep: 19.37
[2025-04-26 00:29:37,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.56 | bwd: 5748.91 | bwd_inner: 5643.42 | bwd_allreduce: 105.45 | step: 19.37

 17%|█▋        | 6843/41250 [16:32:02<82:45:10,  8.66s/it]
                                                          
{'loss': 0.1351, 'grad_norm': 1.7531460523605347, 'learning_rate': 3.809432064278791e-05, 'epoch': 1.66}

 17%|█▋        | 6843/41250 [16:32:02<82:45:10,  8.66s/it][2025-04-26 00:29:46,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-26 00:29:46,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.05 | bwd_microstep: 5690.71 | bwd_inner_microstep: 5668.00 | bwd_allreduce_microstep: 22.66 | step_microstep: 19.44
[2025-04-26 00:29:46,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.05 | bwd: 5690.72 | bwd_inner: 5668.00 | bwd_allreduce: 22.68 | step: 19.44

 17%|█▋        | 6844/41250 [16:32:11<82:36:19,  8.64s/it]
                                                          
{'loss': 0.2295, 'grad_norm': 3.6771273612976074, 'learning_rate': 3.809365160468917e-05, 'epoch': 1.66}

 17%|█▋        | 6844/41250 [16:32:11<82:36:19,  8.64s/it][2025-04-26 00:29:54,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 00:29:54,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.13 | bwd_microstep: 5695.17 | bwd_inner_microstep: 5682.14 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.14
[2025-04-26 00:29:54,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.13 | bwd: 5695.19 | bwd_inner: 5682.13 | bwd_allreduce: 13.01 | step: 19.14

 17%|█▋        | 6845/41250 [16:32:20<82:32:47,  8.64s/it]
                                                          
{'loss': 0.1212, 'grad_norm': 2.0400965213775635, 'learning_rate': 3.809298245504661e-05, 'epoch': 1.66}

 17%|█▋        | 6845/41250 [16:32:20<82:32:47,  8.64s/it][2025-04-26 00:30:03,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:30:03,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.57 | bwd_microstep: 5692.10 | bwd_inner_microstep: 5663.45 | bwd_allreduce_microstep: 28.60 | step_microstep: 18.98
[2025-04-26 00:30:03,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.57 | bwd: 5692.11 | bwd_inner: 5663.45 | bwd_allreduce: 28.62 | step: 18.98

 17%|█▋        | 6846/41250 [16:32:28<82:28:28,  8.63s/it]
                                                          
{'loss': 0.2621, 'grad_norm': 3.8546695709228516, 'learning_rate': 3.809231319386439e-05, 'epoch': 1.66}

 17%|█▋        | 6846/41250 [16:32:28<82:28:28,  8.63s/it][2025-04-26 00:30:12,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:30:12,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.49 | bwd_microstep: 5937.72 | bwd_inner_microstep: 5746.83 | bwd_allreduce_microstep: 190.84 | step_microstep: 18.68
[2025-04-26 00:30:12,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.49 | bwd: 5937.73 | bwd_inner: 5746.83 | bwd_allreduce: 190.86 | step: 18.68

 17%|█▋        | 6847/41250 [16:32:37<83:13:37,  8.71s/it]
                                                          
{'loss': 0.13, 'grad_norm': 1.3097938299179077, 'learning_rate': 3.80916438211466e-05, 'epoch': 1.66}

 17%|█▋        | 6847/41250 [16:32:37<83:13:37,  8.71s/it][2025-04-26 00:30:21,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:30:21,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.27 | bwd_microstep: 5728.80 | bwd_inner_microstep: 5666.98 | bwd_allreduce_microstep: 61.77 | step_microstep: 18.72
[2025-04-26 00:30:21,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.27 | bwd: 5728.81 | bwd_inner: 5666.98 | bwd_allreduce: 61.79 | step: 18.72

 17%|█▋        | 6848/41250 [16:32:46<83:03:26,  8.69s/it]
                                                          
{'loss': 0.2004, 'grad_norm': 2.5378644466400146, 'learning_rate': 3.80909743368974e-05, 'epoch': 1.66}

 17%|█▋        | 6848/41250 [16:32:46<83:03:26,  8.69s/it][2025-04-26 00:30:29,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 1.12
[2025-04-26 00:30:29,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.00 | bwd_microstep: 5753.62 | bwd_inner_microstep: 5740.86 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.97
[2025-04-26 00:30:29,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.00 | bwd: 5753.63 | bwd_inner: 5740.86 | bwd_allreduce: 12.73 | step: 18.97

 17%|█▋        | 6849/41250 [16:32:55<83:06:42,  8.70s/it]
                                                          
{'loss': 0.246, 'grad_norm': 4.872077941894531, 'learning_rate': 3.809030474112089e-05, 'epoch': 1.66}

 17%|█▋        | 6849/41250 [16:32:55<83:06:42,  8.70s/it][2025-04-26 00:30:38,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:30:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.43 | bwd_microstep: 5684.66 | bwd_inner_microstep: 5669.86 | bwd_allreduce_microstep: 14.75 | step_microstep: 18.52
[2025-04-26 00:30:38,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.43 | bwd: 5684.68 | bwd_inner: 5669.86 | bwd_allreduce: 14.77 | step: 18.53

 17%|█▋        | 6850/41250 [16:33:03<82:51:13,  8.67s/it]
                                                          
{'loss': 0.2321, 'grad_norm': 2.951045513153076, 'learning_rate': 3.808963503382122e-05, 'epoch': 1.66}

 17%|█▋        | 6850/41250 [16:33:03<82:51:13,  8.67s/it][2025-04-26 00:30:47,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:30:47,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.48 | bwd_microstep: 5744.98 | bwd_inner_microstep: 5675.82 | bwd_allreduce_microstep: 69.11 | step_microstep: 18.63
[2025-04-26 00:30:47,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.48 | bwd: 5744.99 | bwd_inner: 5675.82 | bwd_allreduce: 69.13 | step: 18.63

 17%|█▋        | 6851/41250 [16:33:12<82:50:39,  8.67s/it]
                                                          
{'loss': 0.0153, 'grad_norm': 0.2244829684495926, 'learning_rate': 3.808896521500249e-05, 'epoch': 1.66}

 17%|█▋        | 6851/41250 [16:33:12<82:50:39,  8.67s/it][2025-04-26 00:30:55,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 00:30:55,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.30 | bwd_microstep: 5783.52 | bwd_inner_microstep: 5770.79 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.51
[2025-04-26 00:30:55,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.30 | bwd: 5783.53 | bwd_inner: 5770.79 | bwd_allreduce: 12.70 | step: 18.52

 17%|█▋        | 6852/41250 [16:33:21<83:04:41,  8.69s/it]
                                                          
{'loss': 0.0444, 'grad_norm': 1.271704077720642, 'learning_rate': 3.808829528466886e-05, 'epoch': 1.66}

 17%|█▋        | 6852/41250 [16:33:21<83:04:41,  8.69s/it][2025-04-26 00:31:04,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:31:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.37 | bwd_microstep: 5737.50 | bwd_inner_microstep: 5711.80 | bwd_allreduce_microstep: 25.66 | step_microstep: 18.70
[2025-04-26 00:31:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.37 | bwd: 5737.52 | bwd_inner: 5711.80 | bwd_allreduce: 25.68 | step: 18.70

 17%|█▋        | 6853/41250 [16:33:29<83:01:00,  8.69s/it]
                                                          
{'loss': 0.1113, 'grad_norm': 2.1077287197113037, 'learning_rate': 3.808762524282445e-05, 'epoch': 1.66}

 17%|█▋        | 6853/41250 [16:33:29<83:01:00,  8.69s/it][2025-04-26 00:31:13,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 00:31:13,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.92 | bwd_microstep: 6044.83 | bwd_inner_microstep: 5654.88 | bwd_allreduce_microstep: 389.90 | step_microstep: 19.04
[2025-04-26 00:31:13,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 6044.84 | bwd_inner: 5654.88 | bwd_allreduce: 389.91 | step: 19.04

 17%|█▋        | 6854/41250 [16:33:38<83:46:39,  8.77s/it]
                                                          
{'loss': 0.0134, 'grad_norm': 0.3653024435043335, 'learning_rate': 3.808695508947338e-05, 'epoch': 1.66}

 17%|█▋        | 6854/41250 [16:33:38<83:46:39,  8.77s/it][2025-04-26 00:31:22,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:31:22,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.93 | bwd_microstep: 5743.87 | bwd_inner_microstep: 5685.63 | bwd_allreduce_microstep: 58.19 | step_microstep: 18.67
[2025-04-26 00:31:22,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.93 | bwd: 5743.88 | bwd_inner: 5685.63 | bwd_allreduce: 58.21 | step: 18.67

 17%|█▋        | 6855/41250 [16:33:47<83:29:59,  8.74s/it]
                                                          
{'loss': 0.0736, 'grad_norm': 1.229202389717102, 'learning_rate': 3.8086284824619796e-05, 'epoch': 1.66}

 17%|█▋        | 6855/41250 [16:33:47<83:29:59,  8.74s/it][2025-04-26 00:31:30,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:31:30,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.85 | bwd_microstep: 5739.70 | bwd_inner_microstep: 5687.10 | bwd_allreduce_microstep: 52.55 | step_microstep: 18.64
[2025-04-26 00:31:30,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.85 | bwd: 5739.71 | bwd_inner: 5687.10 | bwd_allreduce: 52.57 | step: 18.65

 17%|█▋        | 6856/41250 [16:33:56<83:17:41,  8.72s/it]
                                                          
{'loss': 0.143, 'grad_norm': 3.223684787750244, 'learning_rate': 3.808561444826782e-05, 'epoch': 1.66}

 17%|█▋        | 6856/41250 [16:33:56<83:17:41,  8.72s/it][2025-04-26 00:31:39,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:31:39,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.49 | bwd_microstep: 5742.72 | bwd_inner_microstep: 5694.20 | bwd_allreduce_microstep: 48.47 | step_microstep: 18.38
[2025-04-26 00:31:39,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.49 | bwd: 5742.73 | bwd_inner: 5694.20 | bwd_allreduce: 48.49 | step: 18.38

 17%|█▋        | 6857/41250 [16:34:04<83:10:15,  8.71s/it]
                                                          
{'loss': 0.0779, 'grad_norm': 2.417795419692993, 'learning_rate': 3.808494396042159e-05, 'epoch': 1.66}

 17%|█▋        | 6857/41250 [16:34:04<83:10:15,  8.71s/it][2025-04-26 00:31:48,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 00:31:48,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5763.65 | bwd_inner_microstep: 5697.64 | bwd_allreduce_microstep: 65.97 | step_microstep: 18.94
[2025-04-26 00:31:48,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5763.66 | bwd_inner: 5697.64 | bwd_allreduce: 65.98 | step: 18.95

 17%|█▋        | 6858/41250 [16:34:13<83:09:37,  8.70s/it]
                                                          
{'loss': 0.2746, 'grad_norm': 5.000942230224609, 'learning_rate': 3.8084273361085236e-05, 'epoch': 1.66}

 17%|█▋        | 6858/41250 [16:34:13<83:09:37,  8.70s/it][2025-04-26 00:31:56,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 00:31:56,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5710.35 | bwd_inner_microstep: 5697.66 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.52
[2025-04-26 00:31:56,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5710.36 | bwd_inner: 5697.66 | bwd_allreduce: 12.67 | step: 18.52

 17%|█▋        | 6859/41250 [16:34:22<82:59:05,  8.69s/it]
                                                          
{'loss': 0.1036, 'grad_norm': 2.2365658283233643, 'learning_rate': 3.80836026502629e-05, 'epoch': 1.66}

 17%|█▋        | 6859/41250 [16:34:22<82:59:05,  8.69s/it][2025-04-26 00:32:05,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 00:32:05,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.28 | bwd_microstep: 5765.20 | bwd_inner_microstep: 5675.12 | bwd_allreduce_microstep: 90.03 | step_microstep: 18.67
[2025-04-26 00:32:05,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.28 | bwd: 5765.21 | bwd_inner: 5675.12 | bwd_allreduce: 90.05 | step: 18.67

 17%|█▋        | 6860/41250 [16:34:30<83:00:27,  8.69s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 1.5174026489257812, 'learning_rate': 3.80829318279587e-05, 'epoch': 1.66}

 17%|█▋        | 6860/41250 [16:34:30<83:00:27,  8.69s/it][2025-04-26 00:32:14,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:32:14,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.07 | bwd_microstep: 5722.90 | bwd_inner_microstep: 5653.26 | bwd_allreduce_microstep: 69.60 | step_microstep: 18.27
[2025-04-26 00:32:14,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.07 | bwd: 5722.91 | bwd_inner: 5653.26 | bwd_allreduce: 69.62 | step: 18.27

 17%|█▋        | 6861/41250 [16:34:39<82:50:37,  8.67s/it]
                                                          
{'loss': 0.0507, 'grad_norm': 1.8664488792419434, 'learning_rate': 3.8082260894176785e-05, 'epoch': 1.66}

 17%|█▋        | 6861/41250 [16:34:39<82:50:37,  8.67s/it][2025-04-26 00:32:22,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:32:22,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.41 | bwd_microstep: 5802.41 | bwd_inner_microstep: 5646.22 | bwd_allreduce_microstep: 156.15 | step_microstep: 18.66
[2025-04-26 00:32:22,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.41 | bwd: 5802.42 | bwd_inner: 5646.22 | bwd_allreduce: 156.16 | step: 18.66

 17%|█▋        | 6862/41250 [16:34:48<82:57:18,  8.68s/it]
                                                          
{'loss': 0.1864, 'grad_norm': 1.8861761093139648, 'learning_rate': 3.808158984892129e-05, 'epoch': 1.66}

 17%|█▋        | 6862/41250 [16:34:48<82:57:18,  8.68s/it][2025-04-26 00:32:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:32:31,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2899.34 | bwd_microstep: 5796.42 | bwd_inner_microstep: 5783.68 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.50
[2025-04-26 00:32:31,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2899.34 | bwd: 5796.43 | bwd_inner: 5783.68 | bwd_allreduce: 12.71 | step: 18.50

 17%|█▋        | 6863/41250 [16:34:56<83:13:10,  8.71s/it]
                                                          
{'loss': 0.2427, 'grad_norm': 3.980003833770752, 'learning_rate': 3.808091869219635e-05, 'epoch': 1.66}

 17%|█▋        | 6863/41250 [16:34:56<83:13:10,  8.71s/it][2025-04-26 00:32:40,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:32:40,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.00 | bwd_microstep: 5761.74 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 63.78 | step_microstep: 18.57
[2025-04-26 00:32:40,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.00 | bwd: 5761.76 | bwd_inner: 5697.92 | bwd_allreduce: 63.80 | step: 18.58

 17%|█▋        | 6864/41250 [16:35:05<83:10:36,  8.71s/it]
                                                          
{'loss': 0.0391, 'grad_norm': 2.130321502685547, 'learning_rate': 3.80802474240061e-05, 'epoch': 1.66}

 17%|█▋        | 6864/41250 [16:35:05<83:10:36,  8.71s/it][2025-04-26 00:32:48,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:32:48,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.84 | bwd_microstep: 5764.59 | bwd_inner_microstep: 5670.68 | bwd_allreduce_microstep: 93.87 | step_microstep: 18.46
[2025-04-26 00:32:48,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.84 | bwd: 5764.60 | bwd_inner: 5670.68 | bwd_allreduce: 93.89 | step: 18.47

 17%|█▋        | 6865/41250 [16:35:14<83:05:17,  8.70s/it]
                                                          
{'loss': 0.0331, 'grad_norm': 0.6191770434379578, 'learning_rate': 3.8079576044354685e-05, 'epoch': 1.66}

 17%|█▋        | 6865/41250 [16:35:14<83:05:17,  8.70s/it][2025-04-26 00:32:57,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:32:57,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.55 | bwd_microstep: 5793.27 | bwd_inner_microstep: 5662.00 | bwd_allreduce_microstep: 131.23 | step_microstep: 18.42
[2025-04-26 00:32:57,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.55 | bwd: 5793.29 | bwd_inner: 5662.00 | bwd_allreduce: 131.25 | step: 18.42

 17%|█▋        | 6866/41250 [16:35:22<83:07:14,  8.70s/it]
                                                          
{'loss': 0.0327, 'grad_norm': 0.5166125893592834, 'learning_rate': 3.8078904553246234e-05, 'epoch': 1.66}

 17%|█▋        | 6866/41250 [16:35:22<83:07:14,  8.70s/it][2025-04-26 00:33:06,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 00:33:06,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5783.37 | bwd_inner_microstep: 5699.49 | bwd_allreduce_microstep: 83.83 | step_microstep: 18.54
[2025-04-26 00:33:06,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5783.38 | bwd_inner: 5699.49 | bwd_allreduce: 83.85 | step: 18.54

 17%|█▋        | 6867/41250 [16:35:31<83:09:26,  8.71s/it]
                                                          
{'loss': 0.1649, 'grad_norm': 2.9104087352752686, 'learning_rate': 3.8078232950684896e-05, 'epoch': 1.66}

 17%|█▋        | 6867/41250 [16:35:31<83:09:26,  8.71s/it][2025-04-26 00:33:15,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.21 | optimizer_step: 1.00
[2025-04-26 00:33:15,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.88 | bwd_microstep: 5793.05 | bwd_inner_microstep: 5661.58 | bwd_allreduce_microstep: 131.41 | step_microstep: 19.46
[2025-04-26 00:33:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.88 | bwd: 5793.06 | bwd_inner: 5661.58 | bwd_allreduce: 131.44 | step: 19.46

 17%|█▋        | 6868/41250 [16:35:40<83:08:41,  8.71s/it]
                                                          
{'loss': 0.1105, 'grad_norm': 2.8956706523895264, 'learning_rate': 3.807756123667481e-05, 'epoch': 1.66}

 17%|█▋        | 6868/41250 [16:35:40<83:08:41,  8.71s/it][2025-04-26 00:33:23,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 00:33:23,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.12 | bwd_microstep: 5709.76 | bwd_inner_microstep: 5697.04 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.65
[2025-04-26 00:33:23,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.12 | bwd: 5709.78 | bwd_inner: 5697.04 | bwd_allreduce: 12.70 | step: 18.65

 17%|█▋        | 6869/41250 [16:35:49<82:57:29,  8.69s/it]
                                                          
{'loss': 0.1733, 'grad_norm': 1.8719490766525269, 'learning_rate': 3.8076889411220114e-05, 'epoch': 1.67}

 17%|█▋        | 6869/41250 [16:35:49<82:57:29,  8.69s/it][2025-04-26 00:33:32,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:33:32,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.50 | bwd_microstep: 5724.08 | bwd_inner_microstep: 5638.03 | bwd_allreduce_microstep: 86.01 | step_microstep: 18.63
[2025-04-26 00:33:32,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.50 | bwd: 5724.10 | bwd_inner: 5638.03 | bwd_allreduce: 86.03 | step: 18.63

 17%|█▋        | 6870/41250 [16:35:57<82:48:43,  8.67s/it]
                                                          
{'loss': 0.0304, 'grad_norm': 2.7691211700439453, 'learning_rate': 3.807621747432495e-05, 'epoch': 1.67}

 17%|█▋        | 6870/41250 [16:35:57<82:48:43,  8.67s/it][2025-04-26 00:33:40,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:33:40,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.48 | bwd_microstep: 5687.48 | bwd_inner_microstep: 5648.33 | bwd_allreduce_microstep: 39.10 | step_microstep: 18.31
[2025-04-26 00:33:40,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.48 | bwd: 5687.49 | bwd_inner: 5648.33 | bwd_allreduce: 39.12 | step: 18.31

 17%|█▋        | 6871/41250 [16:36:06<82:37:01,  8.65s/it]
                                                          
{'loss': 0.272, 'grad_norm': 2.8919870853424072, 'learning_rate': 3.807554542599346e-05, 'epoch': 1.67}

 17%|█▋        | 6871/41250 [16:36:06<82:37:01,  8.65s/it][2025-04-26 00:33:49,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:33:49,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.05 | bwd_microstep: 5748.44 | bwd_inner_microstep: 5698.86 | bwd_allreduce_microstep: 49.54 | step_microstep: 18.74
[2025-04-26 00:33:49,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.05 | bwd: 5748.45 | bwd_inner: 5698.86 | bwd_allreduce: 49.55 | step: 18.75

 17%|█▋        | 6872/41250 [16:36:14<82:41:46,  8.66s/it]
                                                          
{'loss': 0.0304, 'grad_norm': 1.3990471363067627, 'learning_rate': 3.807487326622979e-05, 'epoch': 1.67}

 17%|█▋        | 6872/41250 [16:36:14<82:41:46,  8.66s/it][2025-04-26 00:33:58,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 00:33:58,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.82 | bwd_microstep: 5713.94 | bwd_inner_microstep: 5701.58 | bwd_allreduce_microstep: 12.31 | step_microstep: 18.56
[2025-04-26 00:33:58,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.82 | bwd: 5713.95 | bwd_inner: 5701.58 | bwd_allreduce: 12.33 | step: 18.56

 17%|█▋        | 6873/41250 [16:36:23<82:40:06,  8.66s/it]
                                                          
{'loss': 0.0764, 'grad_norm': 0.896733283996582, 'learning_rate': 3.807420099503808e-05, 'epoch': 1.67}

 17%|█▋        | 6873/41250 [16:36:23<82:40:06,  8.66s/it][2025-04-26 00:34:06,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:34:06,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5758.80 | bwd_inner_microstep: 5700.86 | bwd_allreduce_microstep: 57.89 | step_microstep: 18.76
[2025-04-26 00:34:06,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5758.82 | bwd_inner: 5700.86 | bwd_allreduce: 57.91 | step: 18.76

 17%|█▋        | 6874/41250 [16:36:32<82:46:03,  8.67s/it]
                                                          
{'loss': 0.2427, 'grad_norm': 2.6409640312194824, 'learning_rate': 3.807352861242247e-05, 'epoch': 1.67}

 17%|█▋        | 6874/41250 [16:36:32<82:46:03,  8.67s/it][2025-04-26 00:34:15,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 00:34:15,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5710.64 | bwd_inner_microstep: 5697.77 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.72
[2025-04-26 00:34:15,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5710.65 | bwd_inner: 5697.77 | bwd_allreduce: 12.83 | step: 18.72

 17%|█▋        | 6875/41250 [16:36:40<82:42:56,  8.66s/it]
                                                          
{'loss': 0.1624, 'grad_norm': 2.448498487472534, 'learning_rate': 3.807285611838712e-05, 'epoch': 1.67}

 17%|█▋        | 6875/41250 [16:36:40<82:42:56,  8.66s/it][2025-04-26 00:34:24,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:34:24,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.46 | bwd_microstep: 5705.66 | bwd_inner_microstep: 5692.76 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.55
[2025-04-26 00:34:24,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.46 | bwd: 5705.68 | bwd_inner: 5692.76 | bwd_allreduce: 12.88 | step: 18.56

 17%|█▋        | 6876/41250 [16:36:49<82:38:09,  8.65s/it]
                                                          
{'loss': 0.1472, 'grad_norm': 1.9988142251968384, 'learning_rate': 3.8072183512936154e-05, 'epoch': 1.67}

 17%|█▋        | 6876/41250 [16:36:49<82:38:09,  8.65s/it][2025-04-26 00:34:32,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 00:34:32,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.33 | bwd_microstep: 5687.70 | bwd_inner_microstep: 5675.08 | bwd_allreduce_microstep: 12.58 | step_microstep: 19.51
[2025-04-26 00:34:32,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.33 | bwd: 5687.72 | bwd_inner: 5675.08 | bwd_allreduce: 12.60 | step: 19.51

 17%|█▋        | 6877/41250 [16:36:58<82:33:01,  8.65s/it]
                                                          
{'loss': 0.0573, 'grad_norm': 1.2651047706604004, 'learning_rate': 3.807151079607375e-05, 'epoch': 1.67}

 17%|█▋        | 6877/41250 [16:36:58<82:33:01,  8.65s/it][2025-04-26 00:34:41,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:34:41,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.42 | bwd_microstep: 5704.77 | bwd_inner_microstep: 5691.85 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.92
[2025-04-26 00:34:41,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.42 | bwd: 5704.78 | bwd_inner: 5691.85 | bwd_allreduce: 12.89 | step: 18.93

 17%|█▋        | 6878/41250 [16:37:06<82:31:33,  8.64s/it]
                                                          
{'loss': 0.3488, 'grad_norm': 2.880899667739868, 'learning_rate': 3.807083796780402e-05, 'epoch': 1.67}

 17%|█▋        | 6878/41250 [16:37:06<82:31:33,  8.64s/it][2025-04-26 00:34:50,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:34:50,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5730.85 | bwd_inner_microstep: 5694.25 | bwd_allreduce_microstep: 36.56 | step_microstep: 18.64
[2025-04-26 00:34:50,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5730.87 | bwd_inner: 5694.25 | bwd_allreduce: 36.57 | step: 18.65

 17%|█▋        | 6879/41250 [16:37:15<82:34:54,  8.65s/it]
                                                          
{'loss': 0.0538, 'grad_norm': 0.9801366925239563, 'learning_rate': 3.8070165028131134e-05, 'epoch': 1.67}

 17%|█▋        | 6879/41250 [16:37:15<82:34:54,  8.65s/it][2025-04-26 00:34:58,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:34:58,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5768.88 | bwd_inner_microstep: 5645.26 | bwd_allreduce_microstep: 123.57 | step_microstep: 18.61
[2025-04-26 00:34:58,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5768.89 | bwd_inner: 5645.26 | bwd_allreduce: 123.59 | step: 18.62

 17%|█▋        | 6880/41250 [16:37:24<82:39:48,  8.66s/it]
                                                          
{'loss': 0.2095, 'grad_norm': 2.011505126953125, 'learning_rate': 3.8069491977059234e-05, 'epoch': 1.67}

 17%|█▋        | 6880/41250 [16:37:24<82:39:48,  8.66s/it][2025-04-26 00:35:07,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:35:07,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.20 | bwd_microstep: 5760.09 | bwd_inner_microstep: 5644.09 | bwd_allreduce_microstep: 115.95 | step_microstep: 18.53
[2025-04-26 00:35:07,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.20 | bwd: 5760.11 | bwd_inner: 5644.09 | bwd_allreduce: 115.97 | step: 18.54

 17%|█▋        | 6881/41250 [16:37:32<82:41:31,  8.66s/it]
                                                          
{'loss': 0.0482, 'grad_norm': 0.8567681908607483, 'learning_rate': 3.806881881459247e-05, 'epoch': 1.67}

 17%|█▋        | 6881/41250 [16:37:32<82:41:31,  8.66s/it][2025-04-26 00:35:16,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 00:35:16,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.84 | bwd_microstep: 5717.31 | bwd_inner_microstep: 5688.44 | bwd_allreduce_microstep: 28.83 | step_microstep: 18.97
[2025-04-26 00:35:16,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.84 | bwd: 5717.32 | bwd_inner: 5688.44 | bwd_allreduce: 28.84 | step: 18.97

 17%|█▋        | 6882/41250 [16:37:41<82:38:36,  8.66s/it]
                                                          
{'loss': 0.085, 'grad_norm': 1.6263160705566406, 'learning_rate': 3.8068145540735e-05, 'epoch': 1.67}

 17%|█▋        | 6882/41250 [16:37:41<82:38:36,  8.66s/it][2025-04-26 00:35:24,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:35:24,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.54 | bwd_microstep: 5734.78 | bwd_inner_microstep: 5700.21 | bwd_allreduce_microstep: 34.52 | step_microstep: 18.84
[2025-04-26 00:35:24,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.54 | bwd: 5734.79 | bwd_inner: 5700.21 | bwd_allreduce: 34.54 | step: 18.84

 17%|█▋        | 6883/41250 [16:37:50<82:40:03,  8.66s/it]
                                                          
{'loss': 0.2062, 'grad_norm': 2.305143117904663, 'learning_rate': 3.8067472155490954e-05, 'epoch': 1.67}

 17%|█▋        | 6883/41250 [16:37:50<82:40:03,  8.66s/it][2025-04-26 00:35:33,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 00:35:33,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.59 | bwd_microstep: 5733.34 | bwd_inner_microstep: 5685.29 | bwd_allreduce_microstep: 48.01 | step_microstep: 18.77
[2025-04-26 00:35:33,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.59 | bwd: 5733.35 | bwd_inner: 5685.29 | bwd_allreduce: 48.02 | step: 18.78

 17%|█▋        | 6884/41250 [16:37:58<82:39:39,  8.66s/it]
                                                          
{'loss': 0.0716, 'grad_norm': 2.1556780338287354, 'learning_rate': 3.80667986588645e-05, 'epoch': 1.67}

 17%|█▋        | 6884/41250 [16:37:58<82:39:39,  8.66s/it][2025-04-26 00:35:42,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:35:42,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5718.42 | bwd_inner_microstep: 5699.76 | bwd_allreduce_microstep: 18.61 | step_microstep: 18.42
[2025-04-26 00:35:42,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5718.43 | bwd_inner: 5699.76 | bwd_allreduce: 18.63 | step: 18.42

 17%|█▋        | 6885/41250 [16:38:07<82:37:58,  8.66s/it]
                                                          
{'loss': 0.1135, 'grad_norm': 1.521303653717041, 'learning_rate': 3.806612505085979e-05, 'epoch': 1.67}

 17%|█▋        | 6885/41250 [16:38:07<82:37:58,  8.66s/it][2025-04-26 00:35:50,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:35:50,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.05 | bwd_microstep: 5760.88 | bwd_inner_microstep: 5650.66 | bwd_allreduce_microstep: 110.17 | step_microstep: 18.56
[2025-04-26 00:35:50,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.05 | bwd: 5760.89 | bwd_inner: 5650.66 | bwd_allreduce: 110.19 | step: 18.56

 17%|█▋        | 6886/41250 [16:38:16<82:38:45,  8.66s/it]
                                                          
{'loss': 0.0212, 'grad_norm': 0.8572463989257812, 'learning_rate': 3.806545133148097e-05, 'epoch': 1.67}

 17%|█▋        | 6886/41250 [16:38:16<82:38:45,  8.66s/it][2025-04-26 00:35:59,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:35:59,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.53 | bwd_microstep: 5695.68 | bwd_inner_microstep: 5634.71 | bwd_allreduce_microstep: 60.93 | step_microstep: 18.49
[2025-04-26 00:35:59,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.53 | bwd: 5695.69 | bwd_inner: 5634.71 | bwd_allreduce: 60.94 | step: 18.49

 17%|█▋        | 6887/41250 [16:38:24<82:26:58,  8.64s/it]
                                                          
{'loss': 0.2074, 'grad_norm': 1.7776703834533691, 'learning_rate': 3.806477750073219e-05, 'epoch': 1.67}

 17%|█▋        | 6887/41250 [16:38:24<82:26:58,  8.64s/it][2025-04-26 00:36:08,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:36:08,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.01 | bwd_microstep: 5771.52 | bwd_inner_microstep: 5651.10 | bwd_allreduce_microstep: 120.37 | step_microstep: 18.62
[2025-04-26 00:36:08,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.01 | bwd: 5771.53 | bwd_inner: 5651.10 | bwd_allreduce: 120.39 | step: 18.62

 17%|█▋        | 6888/41250 [16:38:33<82:32:43,  8.65s/it]
                                                          
{'loss': 0.1187, 'grad_norm': 3.3492703437805176, 'learning_rate': 3.806410355861762e-05, 'epoch': 1.67}

 17%|█▋        | 6888/41250 [16:38:33<82:32:43,  8.65s/it][2025-04-26 00:36:16,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:36:16,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.72 | bwd_microstep: 5750.33 | bwd_inner_microstep: 5670.82 | bwd_allreduce_microstep: 79.47 | step_microstep: 18.66
[2025-04-26 00:36:16,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.72 | bwd: 5750.34 | bwd_inner: 5670.81 | bwd_allreduce: 79.49 | step: 18.66

 17%|█▋        | 6889/41250 [16:38:42<82:37:18,  8.66s/it]
                                                          
{'loss': 0.4195, 'grad_norm': 2.9247117042541504, 'learning_rate': 3.80634295051414e-05, 'epoch': 1.67}

 17%|█▋        | 6889/41250 [16:38:42<82:37:18,  8.66s/it][2025-04-26 00:36:25,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:36:25,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.37 | bwd_microstep: 5719.51 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 21.82 | step_microstep: 18.34
[2025-04-26 00:36:25,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.37 | bwd: 5719.52 | bwd_inner: 5697.65 | bwd_allreduce: 21.84 | step: 18.34

 17%|█▋        | 6890/41250 [16:38:50<82:36:46,  8.66s/it]
                                                          
{'loss': 0.3397, 'grad_norm': 3.197667121887207, 'learning_rate': 3.806275534030769e-05, 'epoch': 1.67}

 17%|█▋        | 6890/41250 [16:38:50<82:36:46,  8.66s/it][2025-04-26 00:36:34,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 00:36:34,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.82 | bwd_microstep: 5690.42 | bwd_inner_microstep: 5677.63 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.52
[2025-04-26 00:36:34,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 5690.43 | bwd_inner: 5677.63 | bwd_allreduce: 12.76 | step: 18.52

 17%|█▋        | 6891/41250 [16:38:59<82:31:56,  8.65s/it]
                                                          
{'loss': 0.0359, 'grad_norm': 0.6634089946746826, 'learning_rate': 3.806208106412065e-05, 'epoch': 1.67}

 17%|█▋        | 6891/41250 [16:38:59<82:31:56,  8.65s/it][2025-04-26 00:36:42,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:36:42,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.08 | bwd_microstep: 5766.13 | bwd_inner_microstep: 5630.18 | bwd_allreduce_microstep: 135.91 | step_microstep: 18.27
[2025-04-26 00:36:42,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.08 | bwd: 5766.15 | bwd_inner: 5630.18 | bwd_allreduce: 135.92 | step: 18.27

 17%|█▋        | 6892/41250 [16:39:08<82:37:43,  8.66s/it]
                                                          
{'loss': 0.0339, 'grad_norm': 0.38788551092147827, 'learning_rate': 3.806140667658443e-05, 'epoch': 1.67}

 17%|█▋        | 6892/41250 [16:39:08<82:37:43,  8.66s/it][2025-04-26 00:36:51,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 00:36:51,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.35 | bwd_microstep: 5751.34 | bwd_inner_microstep: 5676.00 | bwd_allreduce_microstep: 75.29 | step_microstep: 18.51
[2025-04-26 00:36:51,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.35 | bwd: 5751.35 | bwd_inner: 5676.00 | bwd_allreduce: 75.31 | step: 18.52

 17%|█▋        | 6893/41250 [16:39:16<82:39:56,  8.66s/it]
                                                          
{'loss': 0.1383, 'grad_norm': 2.009521007537842, 'learning_rate': 3.806073217770318e-05, 'epoch': 1.67}

 17%|█▋        | 6893/41250 [16:39:16<82:39:56,  8.66s/it][2025-04-26 00:37:00,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:37:00,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.57 | bwd_microstep: 5761.65 | bwd_inner_microstep: 5646.56 | bwd_allreduce_microstep: 115.04 | step_microstep: 18.73
[2025-04-26 00:37:00,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.57 | bwd: 5761.67 | bwd_inner: 5646.56 | bwd_allreduce: 115.06 | step: 18.73

 17%|█▋        | 6894/41250 [16:39:25<82:42:23,  8.67s/it]
                                                          
{'loss': 0.1172, 'grad_norm': 1.2775171995162964, 'learning_rate': 3.806005756748108e-05, 'epoch': 1.67}

 17%|█▋        | 6894/41250 [16:39:25<82:42:23,  8.67s/it][2025-04-26 00:37:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 00:37:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.47 | bwd_microstep: 5691.67 | bwd_inner_microstep: 5679.01 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.45
[2025-04-26 00:37:08,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.47 | bwd: 5691.69 | bwd_inner: 5679.01 | bwd_allreduce: 12.64 | step: 18.46

 17%|█▋        | 6895/41250 [16:39:34<82:32:40,  8.65s/it]
                                                          
{'loss': 0.0547, 'grad_norm': 0.8279218077659607, 'learning_rate': 3.805938284592228e-05, 'epoch': 1.67}

 17%|█▋        | 6895/41250 [16:39:34<82:32:40,  8.65s/it][2025-04-26 00:37:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-26 00:37:17,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.67 | bwd_microstep: 5739.65 | bwd_inner_microstep: 5693.52 | bwd_allreduce_microstep: 46.09 | step_microstep: 19.26
[2025-04-26 00:37:17,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.67 | bwd: 5739.67 | bwd_inner: 5693.52 | bwd_allreduce: 46.11 | step: 19.26

 17%|█▋        | 6896/41250 [16:39:42<82:35:46,  8.66s/it]
                                                          
{'loss': 0.0686, 'grad_norm': 1.2687994241714478, 'learning_rate': 3.8058708013030935e-05, 'epoch': 1.67}

 17%|█▋        | 6896/41250 [16:39:42<82:35:46,  8.66s/it][2025-04-26 00:37:26,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:37:26,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.28 | bwd_microstep: 5724.72 | bwd_inner_microstep: 5712.05 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.55
[2025-04-26 00:37:26,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.28 | bwd: 5724.74 | bwd_inner: 5712.05 | bwd_allreduce: 12.65 | step: 18.56

 17%|█▋        | 6897/41250 [16:39:51<82:37:46,  8.66s/it]
                                                          
{'loss': 0.1054, 'grad_norm': 2.137465715408325, 'learning_rate': 3.80580330688112e-05, 'epoch': 1.67}

 17%|█▋        | 6897/41250 [16:39:51<82:37:46,  8.66s/it][2025-04-26 00:37:34,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.67 | optimizer_step: 1.13
[2025-04-26 00:37:34,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.21 | bwd_microstep: 5770.00 | bwd_inner_microstep: 5648.62 | bwd_allreduce_microstep: 121.33 | step_microstep: 19.28
[2025-04-26 00:37:34,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.21 | bwd: 5770.01 | bwd_inner: 5648.62 | bwd_allreduce: 121.35 | step: 19.29

 17%|█▋        | 6898/41250 [16:40:00<82:40:32,  8.66s/it]
                                                          
{'loss': 0.1021, 'grad_norm': 2.177665948867798, 'learning_rate': 3.8057358013267244e-05, 'epoch': 1.67}

 17%|█▋        | 6898/41250 [16:40:00<82:40:32,  8.66s/it][2025-04-26 00:37:43,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 00:37:43,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.09 | bwd_microstep: 5707.13 | bwd_inner_microstep: 5693.90 | bwd_allreduce_microstep: 13.17 | step_microstep: 18.86
[2025-04-26 00:37:43,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.09 | bwd: 5707.16 | bwd_inner: 5693.90 | bwd_allreduce: 13.20 | step: 18.86

 17%|█▋        | 6899/41250 [16:40:08<82:36:44,  8.66s/it]
                                                          
{'loss': 0.1303, 'grad_norm': 1.9427961111068726, 'learning_rate': 3.805668284640323e-05, 'epoch': 1.67}

 17%|█▋        | 6899/41250 [16:40:08<82:36:44,  8.66s/it][2025-04-26 00:37:52,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:37:52,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5742.10 | bwd_inner_microstep: 5703.53 | bwd_allreduce_microstep: 38.52 | step_microstep: 18.73
[2025-04-26 00:37:52,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5742.11 | bwd_inner: 5703.53 | bwd_allreduce: 38.54 | step: 18.73

 17%|█▋        | 6900/41250 [16:40:17<82:39:33,  8.66s/it]
                                                          
{'loss': 0.1031, 'grad_norm': 2.1245694160461426, 'learning_rate': 3.805600756822332e-05, 'epoch': 1.67}

 17%|█▋        | 6900/41250 [16:40:17<82:39:33,  8.66s/it][2025-04-26 00:38:00,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:38:00,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5701.49 | bwd_inner_microstep: 5688.76 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.60
[2025-04-26 00:38:00,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5701.51 | bwd_inner: 5688.76 | bwd_allreduce: 12.70 | step: 18.61

 17%|█▋        | 6901/41250 [16:40:25<82:33:53,  8.65s/it]
                                                          
{'loss': 0.0894, 'grad_norm': 1.0539590120315552, 'learning_rate': 3.805533217873167e-05, 'epoch': 1.67}

 17%|█▋        | 6901/41250 [16:40:25<82:33:53,  8.65s/it][2025-04-26 00:38:09,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.14
[2025-04-26 00:38:09,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.25 | bwd_microstep: 5709.43 | bwd_inner_microstep: 5652.10 | bwd_allreduce_microstep: 57.28 | step_microstep: 19.00
[2025-04-26 00:38:09,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.25 | bwd: 5709.44 | bwd_inner: 5652.10 | bwd_allreduce: 57.30 | step: 19.00

 17%|█▋        | 6902/41250 [16:40:34<82:27:39,  8.64s/it]
                                                          
{'loss': 0.2551, 'grad_norm': 6.5947418212890625, 'learning_rate': 3.8054656677932455e-05, 'epoch': 1.67}

 17%|█▋        | 6902/41250 [16:40:34<82:27:39,  8.64s/it][2025-04-26 00:38:17,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:38:17,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5745.86 | bwd_inner_microstep: 5713.89 | bwd_allreduce_microstep: 31.92 | step_microstep: 18.70
[2025-04-26 00:38:17,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5745.87 | bwd_inner: 5713.89 | bwd_allreduce: 31.94 | step: 18.70

 17%|█▋        | 6903/41250 [16:40:43<82:33:39,  8.65s/it]
                                                          
{'loss': 0.0823, 'grad_norm': 1.7723268270492554, 'learning_rate': 3.805398106582983e-05, 'epoch': 1.67}

 17%|█▋        | 6903/41250 [16:40:43<82:33:39,  8.65s/it][2025-04-26 00:38:26,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 00:38:26,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.53 | bwd_microstep: 5784.54 | bwd_inner_microstep: 5659.42 | bwd_allreduce_microstep: 125.07 | step_microstep: 19.23
[2025-04-26 00:38:26,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.53 | bwd: 5784.56 | bwd_inner: 5659.42 | bwd_allreduce: 125.09 | step: 19.23

 17%|█▋        | 6904/41250 [16:40:51<82:41:40,  8.67s/it]
                                                          
{'loss': 0.1107, 'grad_norm': 2.373622417449951, 'learning_rate': 3.805330534242796e-05, 'epoch': 1.67}

 17%|█▋        | 6904/41250 [16:40:51<82:41:40,  8.67s/it][2025-04-26 00:38:35,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:38:35,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5764.30 | bwd_inner_microstep: 5656.37 | bwd_allreduce_microstep: 107.90 | step_microstep: 18.37
[2025-04-26 00:38:35,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5764.32 | bwd_inner: 5656.37 | bwd_allreduce: 107.91 | step: 18.37

 17%|█▋        | 6905/41250 [16:41:00<82:42:12,  8.67s/it]
                                                          
{'loss': 0.2295, 'grad_norm': 1.4764505624771118, 'learning_rate': 3.805262950773101e-05, 'epoch': 1.67}

 17%|█▋        | 6905/41250 [16:41:00<82:42:12,  8.67s/it][2025-04-26 00:38:43,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:38:43,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.00 | bwd_microstep: 5766.87 | bwd_inner_microstep: 5642.74 | bwd_allreduce_microstep: 124.09 | step_microstep: 18.47
[2025-04-26 00:38:43,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.00 | bwd: 5766.89 | bwd_inner: 5642.74 | bwd_allreduce: 124.10 | step: 18.47

 17%|█▋        | 6906/41250 [16:41:09<82:41:53,  8.67s/it]
                                                          
{'loss': 0.1895, 'grad_norm': 1.7805896997451782, 'learning_rate': 3.805195356174316e-05, 'epoch': 1.67}

 17%|█▋        | 6906/41250 [16:41:09<82:41:53,  8.67s/it][2025-04-26 00:38:52,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:38:52,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5737.58 | bwd_inner_microstep: 5660.21 | bwd_allreduce_microstep: 77.32 | step_microstep: 18.56
[2025-04-26 00:38:52,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5737.59 | bwd_inner: 5660.21 | bwd_allreduce: 77.34 | step: 18.56

 17%|█▋        | 6907/41250 [16:41:17<82:37:22,  8.66s/it]
                                                          
{'loss': 0.1758, 'grad_norm': 3.8530213832855225, 'learning_rate': 3.805127750446856e-05, 'epoch': 1.67}

 17%|█▋        | 6907/41250 [16:41:17<82:37:22,  8.66s/it][2025-04-26 00:39:01,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:39:01,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.36 | bwd_microstep: 5753.71 | bwd_inner_microstep: 5707.92 | bwd_allreduce_microstep: 45.75 | step_microstep: 18.38
[2025-04-26 00:39:01,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.36 | bwd: 5753.73 | bwd_inner: 5707.92 | bwd_allreduce: 45.77 | step: 18.38

 17%|█▋        | 6908/41250 [16:41:26<82:41:39,  8.67s/it]
                                                          
{'loss': 0.2212, 'grad_norm': 2.9543075561523438, 'learning_rate': 3.805060133591139e-05, 'epoch': 1.67}

 17%|█▋        | 6908/41250 [16:41:26<82:41:39,  8.67s/it][2025-04-26 00:39:09,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:39:09,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5712.11 | bwd_inner_microstep: 5699.53 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.60
[2025-04-26 00:39:09,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5712.12 | bwd_inner: 5699.53 | bwd_allreduce: 12.55 | step: 18.61

 17%|█▋        | 6909/41250 [16:41:35<82:36:56,  8.66s/it]
                                                          
{'loss': 0.2107, 'grad_norm': 3.012986898422241, 'learning_rate': 3.804992505607581e-05, 'epoch': 1.67}

 17%|█▋        | 6909/41250 [16:41:35<82:36:56,  8.66s/it][2025-04-26 00:39:18,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 00:39:18,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.46 | bwd_microstep: 5725.40 | bwd_inner_microstep: 5657.69 | bwd_allreduce_microstep: 67.66 | step_microstep: 18.21
[2025-04-26 00:39:18,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.46 | bwd: 5725.41 | bwd_inner: 5657.69 | bwd_allreduce: 67.68 | step: 18.21

 17%|█▋        | 6910/41250 [16:41:43<82:32:26,  8.65s/it]
                                                          
{'loss': 0.1493, 'grad_norm': 1.491206407546997, 'learning_rate': 3.804924866496599e-05, 'epoch': 1.68}

 17%|█▋        | 6910/41250 [16:41:43<82:32:26,  8.65s/it][2025-04-26 00:39:27,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 00:39:27,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.25 | bwd_microstep: 5746.15 | bwd_inner_microstep: 5703.31 | bwd_allreduce_microstep: 42.79 | step_microstep: 18.75
[2025-04-26 00:39:27,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5746.16 | bwd_inner: 5703.31 | bwd_allreduce: 42.81 | step: 18.75

 17%|█▋        | 6911/41250 [16:41:52<82:37:51,  8.66s/it]
                                                          
{'loss': 0.3782, 'grad_norm': 3.453822612762451, 'learning_rate': 3.804857216258611e-05, 'epoch': 1.68}

 17%|█▋        | 6911/41250 [16:41:52<82:37:51,  8.66s/it][2025-04-26 00:39:35,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:39:35,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.24 | bwd_microstep: 5796.67 | bwd_inner_microstep: 5654.85 | bwd_allreduce_microstep: 141.77 | step_microstep: 17.92
[2025-04-26 00:39:35,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.24 | bwd: 5796.68 | bwd_inner: 5654.85 | bwd_allreduce: 141.78 | step: 17.92

 17%|█▋        | 6912/41250 [16:42:01<82:44:44,  8.68s/it]
                                                          
{'loss': 0.0845, 'grad_norm': 1.382641077041626, 'learning_rate': 3.804789554894033e-05, 'epoch': 1.68}

 17%|█▋        | 6912/41250 [16:42:01<82:44:44,  8.68s/it][2025-04-26 00:39:44,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:39:44,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5709.72 | bwd_inner_microstep: 5696.92 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.41
[2025-04-26 00:39:44,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5709.73 | bwd_inner: 5696.92 | bwd_allreduce: 12.77 | step: 18.41

 17%|█▋        | 6913/41250 [16:42:09<82:38:15,  8.66s/it]
                                                          
{'loss': 0.1203, 'grad_norm': 1.8298540115356445, 'learning_rate': 3.804721882403281e-05, 'epoch': 1.68}

 17%|█▋        | 6913/41250 [16:42:09<82:38:15,  8.66s/it][2025-04-26 00:39:53,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:39:53,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.52 | bwd_microstep: 5745.80 | bwd_inner_microstep: 5686.52 | bwd_allreduce_microstep: 59.23 | step_microstep: 18.66
[2025-04-26 00:39:53,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.52 | bwd: 5745.81 | bwd_inner: 5686.52 | bwd_allreduce: 59.25 | step: 18.66

 17%|█▋        | 6914/41250 [16:42:18<82:41:04,  8.67s/it]
                                                          
{'loss': 0.1529, 'grad_norm': 1.5493303537368774, 'learning_rate': 3.804654198786775e-05, 'epoch': 1.68}

 17%|█▋        | 6914/41250 [16:42:18<82:41:04,  8.67s/it][2025-04-26 00:40:02,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:40:02,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.87 | bwd_microstep: 5779.79 | bwd_inner_microstep: 5662.33 | bwd_allreduce_microstep: 117.42 | step_microstep: 18.41
[2025-04-26 00:40:02,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.87 | bwd: 5779.80 | bwd_inner: 5662.33 | bwd_allreduce: 117.43 | step: 18.41

 17%|█▋        | 6915/41250 [16:42:27<82:45:02,  8.68s/it]
                                                          
{'loss': 0.4839, 'grad_norm': 1.8985509872436523, 'learning_rate': 3.80458650404493e-05, 'epoch': 1.68}

 17%|█▋        | 6915/41250 [16:42:27<82:45:02,  8.68s/it][2025-04-26 00:40:10,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:40:10,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.77 | bwd_microstep: 5715.49 | bwd_inner_microstep: 5644.05 | bwd_allreduce_microstep: 71.40 | step_microstep: 18.37
[2025-04-26 00:40:10,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.77 | bwd: 5715.51 | bwd_inner: 5644.05 | bwd_allreduce: 71.42 | step: 18.37

 17%|█▋        | 6916/41250 [16:42:35<82:35:53,  8.66s/it]
                                                          
{'loss': 0.2243, 'grad_norm': 3.498418092727661, 'learning_rate': 3.804518798178165e-05, 'epoch': 1.68}

 17%|█▋        | 6916/41250 [16:42:35<82:35:53,  8.66s/it][2025-04-26 00:40:19,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 1.00
[2025-04-26 00:40:19,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.80 | bwd_microstep: 5782.92 | bwd_inner_microstep: 5640.51 | bwd_allreduce_microstep: 142.36 | step_microstep: 18.49
[2025-04-26 00:40:19,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.80 | bwd: 5782.93 | bwd_inner: 5640.51 | bwd_allreduce: 142.38 | step: 18.49

 17%|█▋        | 6917/41250 [16:42:44<82:40:23,  8.67s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 1.6063055992126465, 'learning_rate': 3.804451081186896e-05, 'epoch': 1.68}

 17%|█▋        | 6917/41250 [16:42:44<82:40:23,  8.67s/it][2025-04-26 00:40:27,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:40:27,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.03 | bwd_microstep: 5691.59 | bwd_inner_microstep: 5658.75 | bwd_allreduce_microstep: 32.79 | step_microstep: 18.21
[2025-04-26 00:40:27,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.03 | bwd: 5691.60 | bwd_inner: 5658.75 | bwd_allreduce: 32.81 | step: 18.22

 17%|█▋        | 6918/41250 [16:42:53<82:27:55,  8.65s/it]
                                                          
{'loss': 0.1805, 'grad_norm': 5.531785011291504, 'learning_rate': 3.804383353071541e-05, 'epoch': 1.68}

 17%|█▋        | 6918/41250 [16:42:53<82:27:55,  8.65s/it][2025-04-26 00:40:36,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:40:36,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.81 | bwd_microstep: 5804.68 | bwd_inner_microstep: 5633.56 | bwd_allreduce_microstep: 171.07 | step_microstep: 18.62
[2025-04-26 00:40:36,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.81 | bwd: 5804.69 | bwd_inner: 5633.56 | bwd_allreduce: 171.09 | step: 18.63

 17%|█▋        | 6919/41250 [16:43:01<82:38:01,  8.67s/it]
                                                          
{'loss': 0.255, 'grad_norm': 3.8471667766571045, 'learning_rate': 3.8043156138325185e-05, 'epoch': 1.68}

 17%|█▋        | 6919/41250 [16:43:01<82:38:01,  8.67s/it][2025-04-26 00:40:45,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:40:45,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.80 | bwd_microstep: 5709.32 | bwd_inner_microstep: 5651.99 | bwd_allreduce_microstep: 57.28 | step_microstep: 17.98
[2025-04-26 00:40:45,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.80 | bwd: 5709.33 | bwd_inner: 5651.99 | bwd_allreduce: 57.30 | step: 17.99

 17%|█▋        | 6920/41250 [16:43:10<82:29:27,  8.65s/it]
                                                          
{'loss': 0.095, 'grad_norm': 2.096122980117798, 'learning_rate': 3.8042478634702435e-05, 'epoch': 1.68}

 17%|█▋        | 6920/41250 [16:43:10<82:29:27,  8.65s/it][2025-04-26 00:40:53,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:40:53,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.23 | bwd_microstep: 5699.18 | bwd_inner_microstep: 5686.48 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.32
[2025-04-26 00:40:53,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.23 | bwd: 5699.20 | bwd_inner: 5686.48 | bwd_allreduce: 12.68 | step: 18.33

 17%|█▋        | 6921/41250 [16:43:19<82:26:26,  8.65s/it]
                                                          
{'loss': 0.0747, 'grad_norm': 1.210184097290039, 'learning_rate': 3.8041801019851364e-05, 'epoch': 1.68}

 17%|█▋        | 6921/41250 [16:43:19<82:26:26,  8.65s/it][2025-04-26 00:41:02,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:41:02,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.76 | bwd_microstep: 5756.93 | bwd_inner_microstep: 5640.00 | bwd_allreduce_microstep: 116.89 | step_microstep: 18.56
[2025-04-26 00:41:02,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.76 | bwd: 5756.94 | bwd_inner: 5640.00 | bwd_allreduce: 116.90 | step: 18.57

 17%|█▋        | 6922/41250 [16:43:27<82:29:50,  8.65s/it]
                                                          
{'loss': 0.0888, 'grad_norm': 1.2820383310317993, 'learning_rate': 3.804112329377613e-05, 'epoch': 1.68}

 17%|█▋        | 6922/41250 [16:43:27<82:29:50,  8.65s/it][2025-04-26 00:41:11,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:41:11,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.17 | bwd_microstep: 5703.79 | bwd_inner_microstep: 5691.02 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.77
[2025-04-26 00:41:11,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.17 | bwd: 5703.80 | bwd_inner: 5691.02 | bwd_allreduce: 12.73 | step: 18.77

 17%|█▋        | 6923/41250 [16:43:36<82:25:55,  8.64s/it]
                                                          
{'loss': 0.2755, 'grad_norm': 2.153012990951538, 'learning_rate': 3.804044545648093e-05, 'epoch': 1.68}

 17%|█▋        | 6923/41250 [16:43:36<82:25:55,  8.64s/it][2025-04-26 00:41:19,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.17 | optimizer_step: 0.95
[2025-04-26 00:41:19,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.77 | bwd_microstep: 5797.27 | bwd_inner_microstep: 5631.30 | bwd_allreduce_microstep: 165.89 | step_microstep: 19.87
[2025-04-26 00:41:19,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.77 | bwd: 5797.29 | bwd_inner: 5631.30 | bwd_allreduce: 165.92 | step: 19.87

 17%|█▋        | 6924/41250 [16:43:45<82:36:02,  8.66s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 3.5347955226898193, 'learning_rate': 3.8039767507969925e-05, 'epoch': 1.68}

 17%|█▋        | 6924/41250 [16:43:45<82:36:02,  8.66s/it][2025-04-26 00:41:28,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:41:28,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.61 | bwd_microstep: 5703.30 | bwd_inner_microstep: 5646.26 | bwd_allreduce_microstep: 57.00 | step_microstep: 18.67
[2025-04-26 00:41:28,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.61 | bwd: 5703.31 | bwd_inner: 5646.26 | bwd_allreduce: 57.01 | step: 18.67

 17%|█▋        | 6925/41250 [16:43:53<82:26:34,  8.65s/it]
                                                          
{'loss': 0.0629, 'grad_norm': 0.7358373999595642, 'learning_rate': 3.8039089448247304e-05, 'epoch': 1.68}

 17%|█▋        | 6925/41250 [16:43:53<82:26:34,  8.65s/it][2025-04-26 00:41:37,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.17 | optimizer_step: 1.02
[2025-04-26 00:41:37,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.01 | bwd_microstep: 5698.85 | bwd_inner_microstep: 5644.07 | bwd_allreduce_microstep: 54.73 | step_microstep: 19.63
[2025-04-26 00:41:37,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.01 | bwd: 5698.86 | bwd_inner: 5644.07 | bwd_allreduce: 54.75 | step: 19.63

 17%|█▋        | 6926/41250 [16:44:02<82:19:09,  8.63s/it]
                                                          
{'loss': 0.2019, 'grad_norm': 3.8524467945098877, 'learning_rate': 3.803841127731725e-05, 'epoch': 1.68}

 17%|█▋        | 6926/41250 [16:44:02<82:19:09,  8.63s/it][2025-04-26 00:41:45,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:41:45,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.27 | bwd_microstep: 5705.53 | bwd_inner_microstep: 5692.82 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.44
[2025-04-26 00:41:45,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.27 | bwd: 5705.54 | bwd_inner: 5692.82 | bwd_allreduce: 12.68 | step: 18.44

 17%|█▋        | 6927/41250 [16:44:11<82:18:40,  8.63s/it]
                                                          
{'loss': 0.1297, 'grad_norm': 2.6519665718078613, 'learning_rate': 3.803773299518393e-05, 'epoch': 1.68}

 17%|█▋        | 6927/41250 [16:44:11<82:18:40,  8.63s/it][2025-04-26 00:41:54,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:41:54,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.62 | bwd_microstep: 5700.16 | bwd_inner_microstep: 5687.53 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.46
[2025-04-26 00:41:54,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.62 | bwd: 5700.17 | bwd_inner: 5687.53 | bwd_allreduce: 12.60 | step: 18.47

 17%|█▋        | 6928/41250 [16:44:19<82:17:05,  8.63s/it]
                                                          
{'loss': 0.1383, 'grad_norm': 3.5988476276397705, 'learning_rate': 3.803705460185153e-05, 'epoch': 1.68}

 17%|█▋        | 6928/41250 [16:44:19<82:17:05,  8.63s/it][2025-04-26 00:42:02,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.97
[2025-04-26 00:42:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.93 | bwd_microstep: 5751.70 | bwd_inner_microstep: 5640.81 | bwd_allreduce_microstep: 110.85 | step_microstep: 18.30
[2025-04-26 00:42:02,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.93 | bwd: 5751.72 | bwd_inner: 5640.81 | bwd_allreduce: 110.87 | step: 18.30

 17%|█▋        | 6929/41250 [16:44:28<82:21:13,  8.64s/it]
                                                          
{'loss': 0.2329, 'grad_norm': 2.4032540321350098, 'learning_rate': 3.803637609732424e-05, 'epoch': 1.68}

 17%|█▋        | 6929/41250 [16:44:28<82:21:13,  8.64s/it][2025-04-26 00:42:11,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:42:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2811.71 | bwd_microstep: 5752.18 | bwd_inner_microstep: 5626.46 | bwd_allreduce_microstep: 125.68 | step_microstep: 18.57
[2025-04-26 00:42:11,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2811.71 | bwd: 5752.20 | bwd_inner: 5626.46 | bwd_allreduce: 125.70 | step: 18.58

 17%|█▋        | 6930/41250 [16:44:36<82:22:45,  8.64s/it]
                                                          
{'loss': 0.0219, 'grad_norm': 0.4401237666606903, 'learning_rate': 3.803569748160625e-05, 'epoch': 1.68}

 17%|█▋        | 6930/41250 [16:44:36<82:22:45,  8.64s/it][2025-04-26 00:42:20,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:42:20,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.39 | bwd_microstep: 5732.63 | bwd_inner_microstep: 5674.39 | bwd_allreduce_microstep: 58.20 | step_microstep: 18.41
[2025-04-26 00:42:20,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.39 | bwd: 5732.65 | bwd_inner: 5674.39 | bwd_allreduce: 58.22 | step: 18.41

 17%|█▋        | 6931/41250 [16:44:45<82:24:08,  8.64s/it]
                                                          
{'loss': 0.1624, 'grad_norm': 3.209563732147217, 'learning_rate': 3.803501875470172e-05, 'epoch': 1.68}

 17%|█▋        | 6931/41250 [16:44:45<82:24:08,  8.64s/it][2025-04-26 00:42:28,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:42:28,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.49 | bwd_microstep: 5694.90 | bwd_inner_microstep: 5682.20 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.82
[2025-04-26 00:42:28,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.49 | bwd: 5694.92 | bwd_inner: 5682.20 | bwd_allreduce: 12.68 | step: 18.82

 17%|█▋        | 6932/41250 [16:44:54<82:19:59,  8.64s/it]
                                                          
{'loss': 0.1438, 'grad_norm': 1.9190378189086914, 'learning_rate': 3.803433991661485e-05, 'epoch': 1.68}

 17%|█▋        | 6932/41250 [16:44:54<82:19:59,  8.64s/it][2025-04-26 00:42:37,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:42:37,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.35 | bwd_microstep: 5733.62 | bwd_inner_microstep: 5641.31 | bwd_allreduce_microstep: 92.26 | step_microstep: 18.54
[2025-04-26 00:42:37,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.35 | bwd: 5733.63 | bwd_inner: 5641.31 | bwd_allreduce: 92.28 | step: 18.54

 17%|█▋        | 6933/41250 [16:45:02<82:20:45,  8.64s/it]
                                                          
{'loss': 0.193, 'grad_norm': 1.6762491464614868, 'learning_rate': 3.803366096734983e-05, 'epoch': 1.68}

 17%|█▋        | 6933/41250 [16:45:02<82:20:45,  8.64s/it][2025-04-26 00:42:46,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:42:46,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.80 | bwd_microstep: 5698.27 | bwd_inner_microstep: 5685.53 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.39
[2025-04-26 00:42:46,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.80 | bwd: 5698.29 | bwd_inner: 5685.53 | bwd_allreduce: 12.71 | step: 18.39

 17%|█▋        | 6934/41250 [16:45:11<82:17:41,  8.63s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 1.0400930643081665, 'learning_rate': 3.803298190691082e-05, 'epoch': 1.68}

 17%|█▋        | 6934/41250 [16:45:11<82:17:41,  8.63s/it][2025-04-26 00:42:54,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:42:54,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.64 | bwd_microstep: 5760.94 | bwd_inner_microstep: 5633.55 | bwd_allreduce_microstep: 127.35 | step_microstep: 18.59
[2025-04-26 00:42:54,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.64 | bwd: 5760.95 | bwd_inner: 5633.55 | bwd_allreduce: 127.36 | step: 18.59

 17%|█▋        | 6935/41250 [16:45:20<82:23:03,  8.64s/it]
                                                          
{'loss': 0.1101, 'grad_norm': 1.4478055238723755, 'learning_rate': 3.8032302735302035e-05, 'epoch': 1.68}

 17%|█▋        | 6935/41250 [16:45:20<82:23:03,  8.64s/it][2025-04-26 00:43:03,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 00:43:03,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.19 | bwd_microstep: 5769.96 | bwd_inner_microstep: 5648.89 | bwd_allreduce_microstep: 121.03 | step_microstep: 18.96
[2025-04-26 00:43:03,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.19 | bwd: 5769.97 | bwd_inner: 5648.89 | bwd_allreduce: 121.04 | step: 18.96

 17%|█▋        | 6936/41250 [16:45:28<82:27:53,  8.65s/it]
                                                          
{'loss': 0.326, 'grad_norm': 1.55611252784729, 'learning_rate': 3.803162345252765e-05, 'epoch': 1.68}

 17%|█▋        | 6936/41250 [16:45:28<82:27:53,  8.65s/it][2025-04-26 00:43:12,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 1.05
[2025-04-26 00:43:12,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.75 | bwd_microstep: 5765.70 | bwd_inner_microstep: 5641.29 | bwd_allreduce_microstep: 124.37 | step_microstep: 18.29
[2025-04-26 00:43:12,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.75 | bwd: 5765.72 | bwd_inner: 5641.29 | bwd_allreduce: 124.39 | step: 18.29

 17%|█▋        | 6937/41250 [16:45:37<82:30:36,  8.66s/it]
                                                          
{'loss': 0.411, 'grad_norm': 3.4784209728240967, 'learning_rate': 3.803094405859185e-05, 'epoch': 1.68}

 17%|█▋        | 6937/41250 [16:45:37<82:30:36,  8.66s/it][2025-04-26 00:43:20,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:43:20,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.82 | bwd_microstep: 5709.09 | bwd_inner_microstep: 5652.05 | bwd_allreduce_microstep: 57.00 | step_microstep: 18.45
[2025-04-26 00:43:20,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.82 | bwd: 5709.10 | bwd_inner: 5652.05 | bwd_allreduce: 57.01 | step: 18.46

 17%|█▋        | 6938/41250 [16:45:46<82:22:22,  8.64s/it]
                                                          
{'loss': 0.0999, 'grad_norm': 1.0832020044326782, 'learning_rate': 3.8030264553498825e-05, 'epoch': 1.68}

 17%|█▋        | 6938/41250 [16:45:46<82:22:22,  8.64s/it][2025-04-26 00:43:29,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:43:29,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.57 | bwd_microstep: 5706.91 | bwd_inner_microstep: 5647.24 | bwd_allreduce_microstep: 59.62 | step_microstep: 18.45
[2025-04-26 00:43:29,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.57 | bwd: 5706.92 | bwd_inner: 5647.24 | bwd_allreduce: 59.64 | step: 18.45

 17%|█▋        | 6939/41250 [16:45:54<82:17:11,  8.63s/it]
                                                          
{'loss': 0.1544, 'grad_norm': 1.4553736448287964, 'learning_rate': 3.802958493725277e-05, 'epoch': 1.68}

 17%|█▋        | 6939/41250 [16:45:54<82:17:11,  8.63s/it][2025-04-26 00:43:38,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:43:38,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5751.13 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 50.91 | step_microstep: 18.67
[2025-04-26 00:43:38,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5751.14 | bwd_inner: 5700.17 | bwd_allreduce: 50.93 | step: 18.67

 17%|█▋        | 6940/41250 [16:46:03<82:26:06,  8.65s/it]
                                                          
{'loss': 0.0336, 'grad_norm': 0.42445802688598633, 'learning_rate': 3.802890520985787e-05, 'epoch': 1.68}

 17%|█▋        | 6940/41250 [16:46:03<82:26:06,  8.65s/it][2025-04-26 00:43:46,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.11
[2025-04-26 00:43:46,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2775.92 | bwd_microstep: 5826.09 | bwd_inner_microstep: 5558.89 | bwd_allreduce_microstep: 267.15 | step_microstep: 19.45
[2025-04-26 00:43:46,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2775.92 | bwd: 5826.11 | bwd_inner: 5558.89 | bwd_allreduce: 267.17 | step: 19.45

 17%|█▋        | 6941/41250 [16:46:12<82:32:00,  8.66s/it]
                                                          
{'loss': 0.2035, 'grad_norm': 2.1388940811157227, 'learning_rate': 3.802822537131831e-05, 'epoch': 1.68}

 17%|█▋        | 6941/41250 [16:46:12<82:32:00,  8.66s/it][2025-04-26 00:43:55,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:43:55,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.57 | bwd_microstep: 5713.97 | bwd_inner_microstep: 5701.28 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.91
[2025-04-26 00:43:55,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.57 | bwd: 5713.98 | bwd_inner: 5701.28 | bwd_allreduce: 12.66 | step: 18.91

 17%|█▋        | 6942/41250 [16:46:20<82:29:14,  8.66s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.5967629551887512, 'learning_rate': 3.802754542163829e-05, 'epoch': 1.68}

 17%|█▋        | 6942/41250 [16:46:20<82:29:14,  8.66s/it][2025-04-26 00:44:04,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-26 00:44:04,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.29 | bwd_microstep: 6012.10 | bwd_inner_microstep: 5693.39 | bwd_allreduce_microstep: 318.66 | step_microstep: 18.68
[2025-04-26 00:44:04,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.29 | bwd: 6012.11 | bwd_inner: 5693.39 | bwd_allreduce: 318.68 | step: 18.68

 17%|█▋        | 6943/41250 [16:46:29<83:17:48,  8.74s/it]
                                                          
{'loss': 0.1625, 'grad_norm': 2.167724370956421, 'learning_rate': 3.8026865360822e-05, 'epoch': 1.68}

 17%|█▋        | 6943/41250 [16:46:29<83:17:48,  8.74s/it][2025-04-26 00:44:12,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:44:12,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.98 | bwd_microstep: 5715.80 | bwd_inner_microstep: 5652.59 | bwd_allreduce_microstep: 63.16 | step_microstep: 18.85
[2025-04-26 00:44:12,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.98 | bwd: 5715.81 | bwd_inner: 5652.59 | bwd_allreduce: 63.18 | step: 18.85

 17%|█▋        | 6944/41250 [16:46:38<82:58:40,  8.71s/it]
                                                          
{'loss': 0.1789, 'grad_norm': 1.648089051246643, 'learning_rate': 3.802618518887363e-05, 'epoch': 1.68}

 17%|█▋        | 6944/41250 [16:46:38<82:58:40,  8.71s/it][2025-04-26 00:44:21,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 00:44:21,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5694.23 | bwd_inner_microstep: 5655.11 | bwd_allreduce_microstep: 39.07 | step_microstep: 19.01
[2025-04-26 00:44:21,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5694.25 | bwd_inner: 5655.10 | bwd_allreduce: 39.09 | step: 19.02

 17%|█▋        | 6945/41250 [16:46:46<82:41:30,  8.68s/it]
                                                          
{'loss': 0.0967, 'grad_norm': 0.9955821633338928, 'learning_rate': 3.802550490579737e-05, 'epoch': 1.68}

 17%|█▋        | 6945/41250 [16:46:46<82:41:30,  8.68s/it][2025-04-26 00:44:30,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 00:44:30,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.91 | bwd_microstep: 5755.32 | bwd_inner_microstep: 5657.90 | bwd_allreduce_microstep: 97.37 | step_microstep: 19.21
[2025-04-26 00:44:30,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.91 | bwd: 5755.33 | bwd_inner: 5657.90 | bwd_allreduce: 97.39 | step: 19.22

 17%|█▋        | 6946/41250 [16:46:55<82:39:31,  8.67s/it]
                                                          
{'loss': 0.0684, 'grad_norm': 1.1669479608535767, 'learning_rate': 3.8024824511597425e-05, 'epoch': 1.68}

 17%|█▋        | 6946/41250 [16:46:55<82:39:31,  8.67s/it][2025-04-26 00:44:38,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 00:44:38,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.13 | bwd_microstep: 5710.17 | bwd_inner_microstep: 5697.07 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.04
[2025-04-26 00:44:38,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.13 | bwd: 5710.18 | bwd_inner: 5697.07 | bwd_allreduce: 13.07 | step: 19.04

 17%|█▋        | 6947/41250 [16:47:04<82:34:30,  8.67s/it]
                                                          
{'loss': 0.1443, 'grad_norm': 1.4398727416992188, 'learning_rate': 3.802414400627798e-05, 'epoch': 1.68}

 17%|█▋        | 6947/41250 [16:47:04<82:34:30,  8.67s/it][2025-04-26 00:44:47,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:44:47,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.95 | bwd_microstep: 5769.68 | bwd_inner_microstep: 5703.87 | bwd_allreduce_microstep: 65.77 | step_microstep: 18.86
[2025-04-26 00:44:47,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.95 | bwd: 5769.69 | bwd_inner: 5703.87 | bwd_allreduce: 65.78 | step: 18.86

 17%|█▋        | 6948/41250 [16:47:12<82:40:30,  8.68s/it]
                                                          
{'loss': 0.0531, 'grad_norm': 1.3741321563720703, 'learning_rate': 3.8023463389843226e-05, 'epoch': 1.68}

 17%|█▋        | 6948/41250 [16:47:12<82:40:30,  8.68s/it][2025-04-26 00:44:56,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:44:56,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.08 | bwd_microstep: 5752.87 | bwd_inner_microstep: 5707.76 | bwd_allreduce_microstep: 45.07 | step_microstep: 18.52
[2025-04-26 00:44:56,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.08 | bwd: 5752.89 | bwd_inner: 5707.76 | bwd_allreduce: 45.08 | step: 18.52

 17%|█▋        | 6949/41250 [16:47:21<82:42:31,  8.68s/it]
                                                          
{'loss': 0.1746, 'grad_norm': 3.8203511238098145, 'learning_rate': 3.802278266229737e-05, 'epoch': 1.68}

 17%|█▋        | 6949/41250 [16:47:21<82:42:31,  8.68s/it][2025-04-26 00:45:05,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-26 00:45:05,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.79 | bwd_microstep: 5906.51 | bwd_inner_microstep: 5893.74 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.86
[2025-04-26 00:45:05,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2937.79 | bwd: 5906.53 | bwd_inner: 5893.73 | bwd_allreduce: 12.75 | step: 18.86

 17%|█▋        | 6950/41250 [16:47:30<83:24:36,  8.75s/it]
                                                          
{'loss': 0.1132, 'grad_norm': 1.2503150701522827, 'learning_rate': 3.8022101823644605e-05, 'epoch': 1.68}

 17%|█▋        | 6950/41250 [16:47:30<83:24:36,  8.75s/it][2025-04-26 00:45:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 00:45:13,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.54 | bwd_microstep: 5768.41 | bwd_inner_microstep: 5657.95 | bwd_allreduce_microstep: 110.41 | step_microstep: 18.93
[2025-04-26 00:45:13,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.54 | bwd: 5768.42 | bwd_inner: 5657.95 | bwd_allreduce: 110.43 | step: 18.93

 17%|█▋        | 6951/41250 [16:47:39<83:12:17,  8.73s/it]
                                                          
{'loss': 0.2513, 'grad_norm': 1.6788763999938965, 'learning_rate': 3.8021420873889126e-05, 'epoch': 1.69}

 17%|█▋        | 6951/41250 [16:47:39<83:12:17,  8.73s/it][2025-04-26 00:45:22,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-26 00:45:22,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.25 | bwd_microstep: 5918.09 | bwd_inner_microstep: 5662.31 | bwd_allreduce_microstep: 255.73 | step_microstep: 18.55
[2025-04-26 00:45:22,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.25 | bwd: 5918.10 | bwd_inner: 5662.31 | bwd_allreduce: 255.75 | step: 18.56

 17%|█▋        | 6952/41250 [16:47:48<83:29:10,  8.76s/it]
                                                          
{'loss': 0.0697, 'grad_norm': 1.3730154037475586, 'learning_rate': 3.8020739813035125e-05, 'epoch': 1.69}

 17%|█▋        | 6952/41250 [16:47:48<83:29:10,  8.76s/it][2025-04-26 00:45:31,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 00:45:31,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.47 | bwd_microstep: 5759.66 | bwd_inner_microstep: 5659.15 | bwd_allreduce_microstep: 100.46 | step_microstep: 19.04
[2025-04-26 00:45:31,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.47 | bwd: 5759.67 | bwd_inner: 5659.15 | bwd_allreduce: 100.48 | step: 19.04

 17%|█▋        | 6953/41250 [16:47:56<83:15:35,  8.74s/it]
                                                          
{'loss': 0.0714, 'grad_norm': 1.317674160003662, 'learning_rate': 3.802005864108681e-05, 'epoch': 1.69}

 17%|█▋        | 6953/41250 [16:47:56<83:15:35,  8.74s/it][2025-04-26 00:45:40,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 00:45:40,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5754.64 | bwd_inner_microstep: 5695.83 | bwd_allreduce_microstep: 58.77 | step_microstep: 19.00
[2025-04-26 00:45:40,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5754.66 | bwd_inner: 5695.83 | bwd_allreduce: 58.78 | step: 19.00

 17%|█▋        | 6954/41250 [16:48:05<83:07:22,  8.73s/it]
                                                          
{'loss': 0.204, 'grad_norm': 1.2892504930496216, 'learning_rate': 3.801937735804838e-05, 'epoch': 1.69}

 17%|█▋        | 6954/41250 [16:48:05<83:07:22,  8.73s/it][2025-04-26 00:45:48,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:45:48,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.60 | bwd_microstep: 5711.66 | bwd_inner_microstep: 5657.50 | bwd_allreduce_microstep: 54.12 | step_microstep: 18.43
[2025-04-26 00:45:48,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.60 | bwd: 5711.67 | bwd_inner: 5657.50 | bwd_allreduce: 54.13 | step: 18.43

 17%|█▋        | 6955/41250 [16:48:14<82:49:48,  8.69s/it]
                                                          
{'loss': 0.0928, 'grad_norm': 0.9404202699661255, 'learning_rate': 3.8018695963924045e-05, 'epoch': 1.69}

 17%|█▋        | 6955/41250 [16:48:14<82:49:48,  8.69s/it][2025-04-26 00:45:57,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 00:45:57,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.19 | bwd_microstep: 5698.06 | bwd_inner_microstep: 5685.17 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.77
[2025-04-26 00:45:57,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.19 | bwd: 5698.07 | bwd_inner: 5685.17 | bwd_allreduce: 12.86 | step: 18.77

 17%|█▋        | 6956/41250 [16:48:22<82:38:43,  8.68s/it]
                                                          
{'loss': 0.0529, 'grad_norm': 0.9517133831977844, 'learning_rate': 3.8018014458717976e-05, 'epoch': 1.69}

 17%|█▋        | 6956/41250 [16:48:22<82:38:43,  8.68s/it][2025-04-26 00:46:06,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 00:46:06,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.81 | bwd_microstep: 5692.88 | bwd_inner_microstep: 5660.76 | bwd_allreduce_microstep: 32.07 | step_microstep: 19.42
[2025-04-26 00:46:06,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.81 | bwd: 5692.90 | bwd_inner: 5660.76 | bwd_allreduce: 32.09 | step: 19.42

 17%|█▋        | 6957/41250 [16:48:31<82:30:01,  8.66s/it]
                                                          
{'loss': 0.5726, 'grad_norm': 3.874814033508301, 'learning_rate': 3.8017332842434406e-05, 'epoch': 1.69}

 17%|█▋        | 6957/41250 [16:48:31<82:30:01,  8.66s/it][2025-04-26 00:46:14,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:46:14,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5753.97 | bwd_inner_microstep: 5677.04 | bwd_allreduce_microstep: 76.89 | step_microstep: 18.64
[2025-04-26 00:46:14,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5753.99 | bwd_inner: 5677.04 | bwd_allreduce: 76.91 | step: 18.64

 17%|█▋        | 6958/41250 [16:48:40<82:34:20,  8.67s/it]
                                                          
{'loss': 0.1348, 'grad_norm': 1.9239212274551392, 'learning_rate': 3.801665111507751e-05, 'epoch': 1.69}

 17%|█▋        | 6958/41250 [16:48:40<82:34:20,  8.67s/it][2025-04-26 00:46:23,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:46:23,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.62 | bwd_microstep: 5712.63 | bwd_inner_microstep: 5655.45 | bwd_allreduce_microstep: 57.13 | step_microstep: 18.95
[2025-04-26 00:46:23,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.62 | bwd: 5712.65 | bwd_inner: 5655.45 | bwd_allreduce: 57.15 | step: 18.95

 17%|█▋        | 6959/41250 [16:48:48<82:27:14,  8.66s/it]
                                                          
{'loss': 0.1123, 'grad_norm': 2.4923832416534424, 'learning_rate': 3.801596927665151e-05, 'epoch': 1.69}

 17%|█▋        | 6959/41250 [16:48:48<82:27:14,  8.66s/it][2025-04-26 00:46:32,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 00:46:32,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.05 | bwd_microstep: 5770.03 | bwd_inner_microstep: 5642.69 | bwd_allreduce_microstep: 127.29 | step_microstep: 18.41
[2025-04-26 00:46:32,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.05 | bwd: 5770.05 | bwd_inner: 5642.69 | bwd_allreduce: 127.31 | step: 18.41

 17%|█▋        | 6960/41250 [16:48:57<82:31:20,  8.66s/it]
                                                          
{'loss': 0.1611, 'grad_norm': 1.343345284461975, 'learning_rate': 3.80152873271606e-05, 'epoch': 1.69}

 17%|█▋        | 6960/41250 [16:48:57<82:31:20,  8.66s/it][2025-04-26 00:46:40,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:46:40,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5776.03 | bwd_inner_microstep: 5635.74 | bwd_allreduce_microstep: 140.25 | step_microstep: 18.55
[2025-04-26 00:46:40,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5776.05 | bwd_inner: 5635.73 | bwd_allreduce: 140.27 | step: 18.55

 17%|█▋        | 6961/41250 [16:49:06<82:34:08,  8.67s/it]
                                                          
{'loss': 0.2467, 'grad_norm': 2.616295576095581, 'learning_rate': 3.801460526660899e-05, 'epoch': 1.69}

 17%|█▋        | 6961/41250 [16:49:06<82:34:08,  8.67s/it][2025-04-26 00:46:49,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-26 00:46:49,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.62 | bwd_microstep: 5904.48 | bwd_inner_microstep: 5645.19 | bwd_allreduce_microstep: 259.24 | step_microstep: 19.20
[2025-04-26 00:46:49,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.62 | bwd: 5904.49 | bwd_inner: 5645.19 | bwd_allreduce: 259.26 | step: 19.20

 17%|█▋        | 6962/41250 [16:49:14<83:00:35,  8.72s/it]
                                                          
{'loss': 0.0345, 'grad_norm': 0.5761358141899109, 'learning_rate': 3.801392309500088e-05, 'epoch': 1.69}

 17%|█▋        | 6962/41250 [16:49:14<83:00:35,  8.72s/it][2025-04-26 00:46:58,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 00:46:58,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5699.68 | bwd_inner_microstep: 5655.32 | bwd_allreduce_microstep: 44.31 | step_microstep: 19.04
[2025-04-26 00:46:58,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5699.69 | bwd_inner: 5655.32 | bwd_allreduce: 44.32 | step: 19.04

 17%|█▋        | 6963/41250 [16:49:23<82:42:34,  8.68s/it]
                                                          
{'loss': 0.1444, 'grad_norm': 1.220411777496338, 'learning_rate': 3.801324081234048e-05, 'epoch': 1.69}

 17%|█▋        | 6963/41250 [16:49:23<82:42:34,  8.68s/it][2025-04-26 00:47:06,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.15 | optimizer_step: 0.96
[2025-04-26 00:47:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.71 | bwd_microstep: 5735.77 | bwd_inner_microstep: 5687.84 | bwd_allreduce_microstep: 47.88 | step_microstep: 19.09
[2025-04-26 00:47:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.71 | bwd: 5735.78 | bwd_inner: 5687.84 | bwd_allreduce: 47.90 | step: 19.10

 17%|█▋        | 6964/41250 [16:49:32<82:40:24,  8.68s/it]
                                                          
{'loss': 0.1209, 'grad_norm': 3.6806693077087402, 'learning_rate': 3.801255841863199e-05, 'epoch': 1.69}

 17%|█▋        | 6964/41250 [16:49:32<82:40:24,  8.68s/it][2025-04-26 00:47:15,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:47:15,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.03 | bwd_microstep: 5672.68 | bwd_inner_microstep: 5645.94 | bwd_allreduce_microstep: 26.69 | step_microstep: 18.88
[2025-04-26 00:47:15,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.03 | bwd: 5672.69 | bwd_inner: 5645.94 | bwd_allreduce: 26.71 | step: 18.88

 17%|█▋        | 6965/41250 [16:49:40<82:26:51,  8.66s/it]
                                                          
{'loss': 0.0681, 'grad_norm': 1.5274924039840698, 'learning_rate': 3.801187591387962e-05, 'epoch': 1.69}

 17%|█▋        | 6965/41250 [16:49:40<82:26:51,  8.66s/it][2025-04-26 00:47:24,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:47:24,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.35 | bwd_microstep: 5732.97 | bwd_inner_microstep: 5684.70 | bwd_allreduce_microstep: 48.22 | step_microstep: 18.84
[2025-04-26 00:47:24,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.35 | bwd: 5732.99 | bwd_inner: 5684.70 | bwd_allreduce: 48.24 | step: 18.85

 17%|█▋        | 6966/41250 [16:49:49<82:28:16,  8.66s/it]
                                                          
{'loss': 0.1773, 'grad_norm': 1.9475648403167725, 'learning_rate': 3.801119329808758e-05, 'epoch': 1.69}

 17%|█▋        | 6966/41250 [16:49:49<82:28:16,  8.66s/it][2025-04-26 00:47:32,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:47:32,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.10 | bwd_microstep: 5699.20 | bwd_inner_microstep: 5639.29 | bwd_allreduce_microstep: 59.85 | step_microstep: 19.03
[2025-04-26 00:47:32,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.10 | bwd: 5699.21 | bwd_inner: 5639.29 | bwd_allreduce: 59.87 | step: 19.04

 17%|█▋        | 6967/41250 [16:49:57<82:18:40,  8.64s/it]
                                                          
{'loss': 0.0839, 'grad_norm': 2.1449666023254395, 'learning_rate': 3.801051057126007e-05, 'epoch': 1.69}

 17%|█▋        | 6967/41250 [16:49:57<82:18:40,  8.64s/it][2025-04-26 00:47:41,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 00:47:41,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.92 | bwd_microstep: 5729.48 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 35.57 | step_microstep: 19.09
[2025-04-26 00:47:41,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.92 | bwd: 5729.50 | bwd_inner: 5693.87 | bwd_allreduce: 35.59 | step: 19.09

 17%|█▋        | 6968/41250 [16:50:06<82:23:20,  8.65s/it]
                                                          
{'loss': 0.353, 'grad_norm': 5.508453845977783, 'learning_rate': 3.800982773340131e-05, 'epoch': 1.69}

 17%|█▋        | 6968/41250 [16:50:06<82:23:20,  8.65s/it][2025-04-26 00:47:49,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 00:47:49,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.62 | bwd_microstep: 5702.30 | bwd_inner_microstep: 5689.20 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.25
[2025-04-26 00:47:49,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.62 | bwd: 5702.31 | bwd_inner: 5689.20 | bwd_allreduce: 13.07 | step: 19.25

 17%|█▋        | 6969/41250 [16:50:15<82:21:38,  8.65s/it]
                                                          
{'loss': 0.2618, 'grad_norm': 2.8453335762023926, 'learning_rate': 3.8009144784515514e-05, 'epoch': 1.69}

 17%|█▋        | 6969/41250 [16:50:15<82:21:38,  8.65s/it][2025-04-26 00:47:58,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 00:47:58,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.63 | bwd_microstep: 5748.17 | bwd_inner_microstep: 5681.35 | bwd_allreduce_microstep: 66.77 | step_microstep: 18.98
[2025-04-26 00:47:58,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.63 | bwd: 5748.18 | bwd_inner: 5681.35 | bwd_allreduce: 66.79 | step: 18.98

 17%|█▋        | 6970/41250 [16:50:23<82:26:41,  8.66s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 1.7398149967193604, 'learning_rate': 3.800846172460687e-05, 'epoch': 1.69}

 17%|█▋        | 6970/41250 [16:50:23<82:26:41,  8.66s/it][2025-04-26 00:48:07,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:48:07,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.09 | bwd_microstep: 5689.08 | bwd_inner_microstep: 5645.52 | bwd_allreduce_microstep: 43.52 | step_microstep: 18.60
[2025-04-26 00:48:07,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.09 | bwd: 5689.10 | bwd_inner: 5645.52 | bwd_allreduce: 43.54 | step: 18.60

 17%|█▋        | 6971/41250 [16:50:32<82:17:05,  8.64s/it]
                                                          
{'loss': 0.1705, 'grad_norm': 1.5990275144577026, 'learning_rate': 3.8007778553679605e-05, 'epoch': 1.69}

 17%|█▋        | 6971/41250 [16:50:32<82:17:05,  8.64s/it][2025-04-26 00:48:16,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:48:16,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.34 | bwd_microstep: 5834.37 | bwd_inner_microstep: 5682.63 | bwd_allreduce_microstep: 151.70 | step_microstep: 18.60
[2025-04-26 00:48:16,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.34 | bwd: 5834.38 | bwd_inner: 5682.63 | bwd_allreduce: 151.72 | step: 18.60

 17%|█▋        | 6972/41250 [16:50:41<82:37:19,  8.68s/it]
                                                          
{'loss': 0.125, 'grad_norm': 1.5790280103683472, 'learning_rate': 3.8007095271737924e-05, 'epoch': 1.69}

 17%|█▋        | 6972/41250 [16:50:41<82:37:19,  8.68s/it][2025-04-26 00:48:24,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 00:48:24,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.74 | bwd_microstep: 5686.65 | bwd_inner_microstep: 5639.54 | bwd_allreduce_microstep: 47.07 | step_microstep: 18.44
[2025-04-26 00:48:24,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.74 | bwd: 5686.67 | bwd_inner: 5639.54 | bwd_allreduce: 47.08 | step: 18.44

 17%|█▋        | 6973/41250 [16:50:49<82:22:54,  8.65s/it]
                                                          
{'loss': 0.1035, 'grad_norm': 2.1652705669403076, 'learning_rate': 3.800641187878605e-05, 'epoch': 1.69}

 17%|█▋        | 6973/41250 [16:50:49<82:22:54,  8.65s/it][2025-04-26 00:48:33,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 00:48:33,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.94 | bwd_microstep: 5696.93 | bwd_inner_microstep: 5684.31 | bwd_allreduce_microstep: 12.57 | step_microstep: 19.14
[2025-04-26 00:48:33,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.94 | bwd: 5696.94 | bwd_inner: 5684.31 | bwd_allreduce: 12.59 | step: 19.15

 17%|█▋        | 6974/41250 [16:50:58<82:18:00,  8.64s/it]
                                                          
{'loss': 0.1825, 'grad_norm': 1.7627238035202026, 'learning_rate': 3.800572837482818e-05, 'epoch': 1.69}

 17%|█▋        | 6974/41250 [16:50:58<82:18:00,  8.64s/it][2025-04-26 00:48:41,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:48:41,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.62 | bwd_microstep: 5757.01 | bwd_inner_microstep: 5678.72 | bwd_allreduce_microstep: 78.23 | step_microstep: 18.94
[2025-04-26 00:48:41,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.62 | bwd: 5757.02 | bwd_inner: 5678.72 | bwd_allreduce: 78.25 | step: 18.94

 17%|█▋        | 6975/41250 [16:51:07<82:24:29,  8.66s/it]
                                                          
{'loss': 0.1752, 'grad_norm': 5.374935626983643, 'learning_rate': 3.800504475986854e-05, 'epoch': 1.69}

 17%|█▋        | 6975/41250 [16:51:07<82:24:29,  8.66s/it][2025-04-26 00:48:50,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:48:50,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.42 | bwd_microstep: 5688.85 | bwd_inner_microstep: 5675.88 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.16
[2025-04-26 00:48:50,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.42 | bwd: 5688.87 | bwd_inner: 5675.88 | bwd_allreduce: 12.94 | step: 19.16

 17%|█▋        | 6976/41250 [16:51:15<82:17:45,  8.64s/it]
                                                          
{'loss': 0.1103, 'grad_norm': 1.441941261291504, 'learning_rate': 3.8004361033911335e-05, 'epoch': 1.69}

 17%|█▋        | 6976/41250 [16:51:15<82:17:45,  8.64s/it][2025-04-26 00:48:59,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 00:48:59,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5695.21 | bwd_inner_microstep: 5682.33 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.74
[2025-04-26 00:48:59,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5695.23 | bwd_inner: 5682.33 | bwd_allreduce: 12.85 | step: 19.75

 17%|█▋        | 6977/41250 [16:51:24<82:15:48,  8.64s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 2.6289172172546387, 'learning_rate': 3.8003677196960795e-05, 'epoch': 1.69}

 17%|█▋        | 6977/41250 [16:51:24<82:15:48,  8.64s/it][2025-04-26 00:49:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:49:07,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.26 | bwd_microstep: 5747.96 | bwd_inner_microstep: 5668.50 | bwd_allreduce_microstep: 79.42 | step_microstep: 19.22
[2025-04-26 00:49:07,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.26 | bwd: 5747.98 | bwd_inner: 5668.50 | bwd_allreduce: 79.44 | step: 19.22

 17%|█▋        | 6978/41250 [16:51:33<82:21:00,  8.65s/it]
                                                          
{'loss': 0.2836, 'grad_norm': 4.727651119232178, 'learning_rate': 3.800299324902112e-05, 'epoch': 1.69}

 17%|█▋        | 6978/41250 [16:51:33<82:21:00,  8.65s/it][2025-04-26 00:49:16,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 00:49:16,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.03 | bwd_microstep: 5773.18 | bwd_inner_microstep: 5643.14 | bwd_allreduce_microstep: 129.99 | step_microstep: 19.69
[2025-04-26 00:49:16,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.03 | bwd: 5773.19 | bwd_inner: 5643.14 | bwd_allreduce: 130.02 | step: 19.70

 17%|█▋        | 6979/41250 [16:51:41<82:26:21,  8.66s/it]
                                                          
{'loss': 0.197, 'grad_norm': 2.0654497146606445, 'learning_rate': 3.800230919009653e-05, 'epoch': 1.69}

 17%|█▋        | 6979/41250 [16:51:41<82:26:21,  8.66s/it][2025-04-26 00:49:25,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 00:49:25,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.90 | bwd_microstep: 5705.98 | bwd_inner_microstep: 5692.90 | bwd_allreduce_microstep: 13.03 | step_microstep: 19.17
[2025-04-26 00:49:25,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.90 | bwd: 5706.00 | bwd_inner: 5692.89 | bwd_allreduce: 13.06 | step: 19.17

 17%|█▋        | 6980/41250 [16:51:50<82:22:39,  8.65s/it]
                                                          
{'loss': 0.0305, 'grad_norm': 0.6259673833847046, 'learning_rate': 3.8001625020191253e-05, 'epoch': 1.69}

 17%|█▋        | 6980/41250 [16:51:50<82:22:39,  8.65s/it][2025-04-26 00:49:33,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 00:49:33,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.70 | bwd_microstep: 5686.26 | bwd_inner_microstep: 5647.70 | bwd_allreduce_microstep: 38.52 | step_microstep: 18.76
[2025-04-26 00:49:33,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.70 | bwd: 5686.27 | bwd_inner: 5647.70 | bwd_allreduce: 38.53 | step: 18.77

 17%|█▋        | 6981/41250 [16:51:59<82:12:09,  8.64s/it]
                                                          
{'loss': 0.0782, 'grad_norm': 1.342300534248352, 'learning_rate': 3.80009407393095e-05, 'epoch': 1.69}

 17%|█▋        | 6981/41250 [16:51:59<82:12:09,  8.64s/it][2025-04-26 00:49:42,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:49:42,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.75 | bwd_microstep: 5701.02 | bwd_inner_microstep: 5688.33 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.57
[2025-04-26 00:49:42,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.75 | bwd: 5701.03 | bwd_inner: 5688.33 | bwd_allreduce: 12.66 | step: 18.57

 17%|█▋        | 6982/41250 [16:52:07<82:09:40,  8.63s/it]
                                                          
{'loss': 0.131, 'grad_norm': 1.9550195932388306, 'learning_rate': 3.800025634745548e-05, 'epoch': 1.69}

 17%|█▋        | 6982/41250 [16:52:07<82:09:40,  8.63s/it][2025-04-26 00:49:50,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-26 00:49:50,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.26 | bwd_microstep: 5694.59 | bwd_inner_microstep: 5660.07 | bwd_allreduce_microstep: 34.46 | step_microstep: 19.00
[2025-04-26 00:49:50,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5694.60 | bwd_inner: 5660.07 | bwd_allreduce: 34.48 | step: 18.99

 17%|█▋        | 6983/41250 [16:52:16<82:06:22,  8.63s/it]
                                                          
{'loss': 0.1771, 'grad_norm': 1.517397165298462, 'learning_rate': 3.799957184463342e-05, 'epoch': 1.69}

 17%|█▋        | 6983/41250 [16:52:16<82:06:22,  8.63s/it][2025-04-26 00:49:59,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 00:49:59,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.97 | bwd_microstep: 5736.27 | bwd_inner_microstep: 5686.60 | bwd_allreduce_microstep: 49.63 | step_microstep: 19.29
[2025-04-26 00:49:59,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.97 | bwd: 5736.29 | bwd_inner: 5686.60 | bwd_allreduce: 49.65 | step: 19.30

 17%|█▋        | 6984/41250 [16:52:24<82:15:36,  8.64s/it]
                                                          
{'loss': 0.1156, 'grad_norm': 0.968220591545105, 'learning_rate': 3.7998887230847545e-05, 'epoch': 1.69}

 17%|█▋        | 6984/41250 [16:52:25<82:15:36,  8.64s/it][2025-04-26 00:50:08,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-26 00:50:08,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.59 | bwd_microstep: 5729.88 | bwd_inner_microstep: 5695.89 | bwd_allreduce_microstep: 33.94 | step_microstep: 19.19
[2025-04-26 00:50:08,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.59 | bwd: 5729.89 | bwd_inner: 5695.89 | bwd_allreduce: 33.96 | step: 19.20

 17%|█▋        | 6985/41250 [16:52:33<82:20:03,  8.65s/it]
                                                          
{'loss': 0.0598, 'grad_norm': 0.6925025582313538, 'learning_rate': 3.799820250610207e-05, 'epoch': 1.69}

 17%|█▋        | 6985/41250 [16:52:33<82:20:03,  8.65s/it][2025-04-26 00:50:16,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:50:16,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.52 | bwd_microstep: 5694.65 | bwd_inner_microstep: 5656.13 | bwd_allreduce_microstep: 38.47 | step_microstep: 19.01
[2025-04-26 00:50:16,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.52 | bwd: 5694.66 | bwd_inner: 5656.13 | bwd_allreduce: 38.49 | step: 19.01

 17%|█▋        | 6986/41250 [16:52:42<82:13:00,  8.64s/it]
                                                          
{'loss': 0.1181, 'grad_norm': 2.442380428314209, 'learning_rate': 3.799751767040121e-05, 'epoch': 1.69}

 17%|█▋        | 6986/41250 [16:52:42<82:13:00,  8.64s/it][2025-04-26 00:50:25,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:50:25,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.20 | bwd_microstep: 5899.88 | bwd_inner_microstep: 5644.89 | bwd_allreduce_microstep: 254.94 | step_microstep: 18.51
[2025-04-26 00:50:25,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.20 | bwd: 5899.89 | bwd_inner: 5644.89 | bwd_allreduce: 254.96 | step: 18.51

 17%|█▋        | 6987/41250 [16:52:51<82:41:42,  8.69s/it]
                                                          
{'loss': 0.2268, 'grad_norm': 4.173859119415283, 'learning_rate': 3.79968327237492e-05, 'epoch': 1.69}

 17%|█▋        | 6987/41250 [16:52:51<82:41:42,  8.69s/it][2025-04-26 00:50:34,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.06 | optimizer_step: 1.06
[2025-04-26 00:50:34,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.50 | bwd_microstep: 5758.89 | bwd_inner_microstep: 5702.35 | bwd_allreduce_microstep: 56.49 | step_microstep: 19.66
[2025-04-26 00:50:34,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.50 | bwd: 5758.90 | bwd_inner: 5702.35 | bwd_allreduce: 56.51 | step: 19.66

 17%|█▋        | 6988/41250 [16:52:59<82:42:19,  8.69s/it]
                                                          
{'loss': 0.0896, 'grad_norm': 2.1698408126831055, 'learning_rate': 3.7996147666150256e-05, 'epoch': 1.69}

 17%|█▋        | 6988/41250 [16:52:59<82:42:19,  8.69s/it][2025-04-26 00:50:43,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:50:43,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.14 | bwd_microstep: 5715.52 | bwd_inner_microstep: 5702.90 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.66
[2025-04-26 00:50:43,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.14 | bwd: 5715.54 | bwd_inner: 5702.90 | bwd_allreduce: 12.60 | step: 18.66

 17%|█▋        | 6989/41250 [16:53:08<82:35:12,  8.68s/it]
                                                          
{'loss': 0.1762, 'grad_norm': 4.293949604034424, 'learning_rate': 3.7995462497608595e-05, 'epoch': 1.69}

 17%|█▋        | 6989/41250 [16:53:08<82:35:12,  8.68s/it][2025-04-26 00:50:51,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:50:51,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.55 | bwd_microstep: 5695.96 | bwd_inner_microstep: 5664.16 | bwd_allreduce_microstep: 31.75 | step_microstep: 18.48
[2025-04-26 00:50:51,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.55 | bwd: 5695.97 | bwd_inner: 5664.16 | bwd_allreduce: 31.77 | step: 18.48

 17%|█▋        | 6990/41250 [16:53:17<82:22:26,  8.66s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 2.4090356826782227, 'learning_rate': 3.7994777218128454e-05, 'epoch': 1.69}

 17%|█▋        | 6990/41250 [16:53:17<82:22:26,  8.66s/it][2025-04-26 00:51:00,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:51:00,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.01 | bwd_microstep: 5757.88 | bwd_inner_microstep: 5687.30 | bwd_allreduce_microstep: 70.54 | step_microstep: 18.41
[2025-04-26 00:51:00,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.01 | bwd: 5757.89 | bwd_inner: 5687.30 | bwd_allreduce: 70.55 | step: 18.41

 17%|█▋        | 6991/41250 [16:53:25<82:27:29,  8.66s/it]
                                                          
{'loss': 0.0882, 'grad_norm': 2.013277053833008, 'learning_rate': 3.7994091827714055e-05, 'epoch': 1.69}

 17%|█▋        | 6991/41250 [16:53:25<82:27:29,  8.66s/it][2025-04-26 00:51:09,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:51:09,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.79 | bwd_microstep: 5706.69 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.60
[2025-04-26 00:51:09,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.79 | bwd: 5706.71 | bwd_inner: 5694.02 | bwd_allreduce: 12.64 | step: 18.61

 17%|█▋        | 6992/41250 [16:53:34<82:22:56,  8.66s/it]
                                                          
{'loss': 0.3188, 'grad_norm': 2.645777702331543, 'learning_rate': 3.7993406326369605e-05, 'epoch': 1.7}

 17%|█▋        | 6992/41250 [16:53:34<82:22:56,  8.66s/it][2025-04-26 00:51:17,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:51:17,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.80 | bwd_microstep: 5759.89 | bwd_inner_microstep: 5661.77 | bwd_allreduce_microstep: 98.07 | step_microstep: 18.54
[2025-04-26 00:51:17,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.80 | bwd: 5759.91 | bwd_inner: 5661.77 | bwd_allreduce: 98.09 | step: 18.54

 17%|█▋        | 6993/41250 [16:53:43<82:25:35,  8.66s/it]
                                                          
{'loss': 0.1372, 'grad_norm': 2.235297679901123, 'learning_rate': 3.7992720714099354e-05, 'epoch': 1.7}

 17%|█▋        | 6993/41250 [16:53:43<82:25:35,  8.66s/it][2025-04-26 00:51:26,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:51:26,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5723.83 | bwd_inner_microstep: 5642.35 | bwd_allreduce_microstep: 81.44 | step_microstep: 18.57
[2025-04-26 00:51:26,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5723.84 | bwd_inner: 5642.35 | bwd_allreduce: 81.46 | step: 18.57

 17%|█▋        | 6994/41250 [16:53:51<82:20:21,  8.65s/it]
                                                          
{'loss': 0.0898, 'grad_norm': 1.1491159200668335, 'learning_rate': 3.7992034990907515e-05, 'epoch': 1.7}

 17%|█▋        | 6994/41250 [16:53:51<82:20:21,  8.65s/it][2025-04-26 00:51:34,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-26 00:51:34,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.92 | bwd_microstep: 5710.85 | bwd_inner_microstep: 5698.24 | bwd_allreduce_microstep: 12.56 | step_microstep: 19.06
[2025-04-26 00:51:34,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5710.86 | bwd_inner: 5698.24 | bwd_allreduce: 12.58 | step: 19.07

 17%|█▋        | 6995/41250 [16:54:00<82:17:41,  8.65s/it]
                                                          
{'loss': 0.2908, 'grad_norm': 2.0931479930877686, 'learning_rate': 3.7991349156798325e-05, 'epoch': 1.7}

 17%|█▋        | 6995/41250 [16:54:00<82:17:41,  8.65s/it][2025-04-26 00:51:43,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:51:43,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.05 | bwd_microstep: 5753.97 | bwd_inner_microstep: 5712.09 | bwd_allreduce_microstep: 41.84 | step_microstep: 18.30
[2025-04-26 00:51:43,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.05 | bwd: 5753.98 | bwd_inner: 5712.09 | bwd_allreduce: 41.86 | step: 18.31

 17%|█▋        | 6996/41250 [16:54:09<82:26:54,  8.67s/it]
                                                          
{'loss': 0.0238, 'grad_norm': 0.3598145544528961, 'learning_rate': 3.7990663211776e-05, 'epoch': 1.7}

 17%|█▋        | 6996/41250 [16:54:09<82:26:54,  8.67s/it][2025-04-26 00:51:52,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 00:51:52,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.85 | bwd_microstep: 5774.79 | bwd_inner_microstep: 5762.29 | bwd_allreduce_microstep: 12.46 | step_microstep: 18.27
[2025-04-26 00:51:52,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.86 | bwd: 5774.81 | bwd_inner: 5762.29 | bwd_allreduce: 12.48 | step: 18.27

 17%|█▋        | 6997/41250 [16:54:17<82:41:52,  8.69s/it]
                                                          
{'loss': 0.0793, 'grad_norm': 0.9679085612297058, 'learning_rate': 3.798997715584477e-05, 'epoch': 1.7}

 17%|█▋        | 6997/41250 [16:54:17<82:41:52,  8.69s/it][2025-04-26 00:52:01,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:52:01,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.18 | bwd_microstep: 5784.25 | bwd_inner_microstep: 5665.44 | bwd_allreduce_microstep: 118.76 | step_microstep: 18.81
[2025-04-26 00:52:01,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.18 | bwd: 5784.27 | bwd_inner: 5665.44 | bwd_allreduce: 118.79 | step: 18.81

 17%|█▋        | 6998/41250 [16:54:26<82:44:08,  8.70s/it]
                                                          
{'loss': 0.0476, 'grad_norm': 0.9449998736381531, 'learning_rate': 3.798929098900888e-05, 'epoch': 1.7}

 17%|█▋        | 6998/41250 [16:54:26<82:44:08,  8.70s/it][2025-04-26 00:52:09,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:52:09,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.91 | bwd_microstep: 5745.49 | bwd_inner_microstep: 5709.32 | bwd_allreduce_microstep: 36.13 | step_microstep: 18.67
[2025-04-26 00:52:09,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.91 | bwd: 5745.51 | bwd_inner: 5709.32 | bwd_allreduce: 36.14 | step: 18.68

 17%|█▋        | 6999/41250 [16:54:35<82:42:24,  8.69s/it]
                                                          
{'loss': 0.1713, 'grad_norm': 1.958443284034729, 'learning_rate': 3.798860471127254e-05, 'epoch': 1.7}

 17%|█▋        | 6999/41250 [16:54:35<82:42:24,  8.69s/it][2025-04-26 00:52:18,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:52:18,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.66 | bwd_microstep: 5780.57 | bwd_inner_microstep: 5767.76 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.84
[2025-04-26 00:52:18,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.66 | bwd: 5780.58 | bwd_inner: 5767.76 | bwd_allreduce: 12.77 | step: 18.84

 17%|█▋        | 7000/41250 [16:54:43<82:53:49,  8.71s/it]
                                                          
{'loss': 0.2557, 'grad_norm': 2.4017186164855957, 'learning_rate': 3.798791832264e-05, 'epoch': 1.7}

 17%|█▋        | 7000/41250 [16:54:43<82:53:49,  8.71s/it][2025-04-26 00:52:27,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:52:27,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5708.20 | bwd_inner_microstep: 5695.27 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.97
[2025-04-26 00:52:27,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5708.21 | bwd_inner: 5695.27 | bwd_allreduce: 12.89 | step: 18.98

 17%|█▋        | 7001/41250 [16:54:52<82:40:47,  8.69s/it]
                                                          
{'loss': 0.1828, 'grad_norm': 3.371244430541992, 'learning_rate': 3.798723182311548e-05, 'epoch': 1.7}

 17%|█▋        | 7001/41250 [16:54:52<82:40:47,  8.69s/it][2025-04-26 00:52:35,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 00:52:35,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.06 | bwd_microstep: 5760.50 | bwd_inner_microstep: 5648.79 | bwd_allreduce_microstep: 111.65 | step_microstep: 18.99
[2025-04-26 00:52:35,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.06 | bwd: 5760.51 | bwd_inner: 5648.79 | bwd_allreduce: 111.67 | step: 19.00

 17%|█▋        | 7002/41250 [16:55:01<82:37:10,  8.68s/it]
                                                          
{'loss': 0.0474, 'grad_norm': 2.5778450965881348, 'learning_rate': 3.798654521270321e-05, 'epoch': 1.7}

 17%|█▋        | 7002/41250 [16:55:01<82:37:10,  8.68s/it][2025-04-26 00:52:44,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.21 | optimizer_step: 1.04
[2025-04-26 00:52:44,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5769.00 | bwd_inner_microstep: 5653.28 | bwd_allreduce_microstep: 115.67 | step_microstep: 19.81
[2025-04-26 00:52:44,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5769.02 | bwd_inner: 5653.28 | bwd_allreduce: 115.69 | step: 19.82

 17%|█▋        | 7003/41250 [16:55:09<82:36:23,  8.68s/it]
                                                          
{'loss': 0.0791, 'grad_norm': 1.652101755142212, 'learning_rate': 3.798585849140743e-05, 'epoch': 1.7}

 17%|█▋        | 7003/41250 [16:55:09<82:36:23,  8.68s/it][2025-04-26 00:52:53,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 00:52:53,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.51 | bwd_microstep: 5920.70 | bwd_inner_microstep: 5665.15 | bwd_allreduce_microstep: 255.50 | step_microstep: 18.70
[2025-04-26 00:52:53,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.51 | bwd: 5920.71 | bwd_inner: 5665.15 | bwd_allreduce: 255.52 | step: 18.70

 17%|█▋        | 7004/41250 [16:55:18<83:02:34,  8.73s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 1.4332436323165894, 'learning_rate': 3.7985171659232364e-05, 'epoch': 1.7}

 17%|█▋        | 7004/41250 [16:55:18<83:02:34,  8.73s/it][2025-04-26 00:53:02,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 00:53:02,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.99 | bwd_microstep: 5775.94 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 130.89 | step_microstep: 18.72
[2025-04-26 00:53:02,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.99 | bwd: 5775.96 | bwd_inner: 5645.00 | bwd_allreduce: 130.91 | step: 18.72

 17%|█▋        | 7005/41250 [16:55:27<82:54:15,  8.72s/it]
                                                          
{'loss': 0.2065, 'grad_norm': 3.654470920562744, 'learning_rate': 3.798448471618225e-05, 'epoch': 1.7}

 17%|█▋        | 7005/41250 [16:55:27<82:54:15,  8.72s/it][2025-04-26 00:53:10,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-26 00:53:10,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.41 | bwd_microstep: 5791.16 | bwd_inner_microstep: 5654.50 | bwd_allreduce_microstep: 136.62 | step_microstep: 19.33
[2025-04-26 00:53:10,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.41 | bwd: 5791.18 | bwd_inner: 5654.50 | bwd_allreduce: 136.64 | step: 19.34

 17%|█▋        | 7006/41250 [16:55:36<82:51:36,  8.71s/it]
                                                          
{'loss': 0.3367, 'grad_norm': 3.6412339210510254, 'learning_rate': 3.7983797662261335e-05, 'epoch': 1.7}

 17%|█▋        | 7006/41250 [16:55:36<82:51:36,  8.71s/it][2025-04-26 00:53:19,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 00:53:19,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.22 | bwd_microstep: 5691.72 | bwd_inner_microstep: 5678.90 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.60
[2025-04-26 00:53:19,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.22 | bwd: 5691.73 | bwd_inner: 5678.89 | bwd_allreduce: 12.79 | step: 18.61

 17%|█▋        | 7007/41250 [16:55:44<82:37:05,  8.69s/it]
                                                          
{'loss': 0.1691, 'grad_norm': 2.209867238998413, 'learning_rate': 3.798311049747384e-05, 'epoch': 1.7}

 17%|█▋        | 7007/41250 [16:55:44<82:37:05,  8.69s/it][2025-04-26 00:53:28,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.06 | optimizer_step: 0.99
[2025-04-26 00:53:28,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.32 | bwd_microstep: 5791.42 | bwd_inner_microstep: 5647.25 | bwd_allreduce_microstep: 144.13 | step_microstep: 19.25
[2025-04-26 00:53:28,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.32 | bwd: 5791.44 | bwd_inner: 5647.25 | bwd_allreduce: 144.15 | step: 19.25

 17%|█▋        | 7008/41250 [16:55:53<82:38:16,  8.69s/it]
                                                          
{'loss': 0.0464, 'grad_norm': 1.00788414478302, 'learning_rate': 3.7982423221824004e-05, 'epoch': 1.7}

 17%|█▋        | 7008/41250 [16:55:53<82:38:16,  8.69s/it][2025-04-26 00:53:36,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 00:53:36,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.31 | bwd_microstep: 5754.21 | bwd_inner_microstep: 5689.99 | bwd_allreduce_microstep: 64.18 | step_microstep: 18.63
[2025-04-26 00:53:36,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.31 | bwd: 5754.23 | bwd_inner: 5689.99 | bwd_allreduce: 64.20 | step: 18.64

 17%|█▋        | 7009/41250 [16:56:02<82:37:03,  8.69s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 0.7331605553627014, 'learning_rate': 3.798173583531607e-05, 'epoch': 1.7}

 17%|█▋        | 7009/41250 [16:56:02<82:37:03,  8.69s/it][2025-04-26 00:53:45,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:53:45,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5697.77 | bwd_inner_microstep: 5648.41 | bwd_allreduce_microstep: 49.32 | step_microstep: 18.91
[2025-04-26 00:53:45,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5697.79 | bwd_inner: 5648.41 | bwd_allreduce: 49.34 | step: 18.92

 17%|█▋        | 7010/41250 [16:56:10<82:23:33,  8.66s/it]
                                                          
{'loss': 0.0949, 'grad_norm': 1.7672899961471558, 'learning_rate': 3.798104833795427e-05, 'epoch': 1.7}

 17%|█▋        | 7010/41250 [16:56:10<82:23:33,  8.66s/it][2025-04-26 00:53:53,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:53:53,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.50 | bwd_microstep: 5686.98 | bwd_inner_microstep: 5633.31 | bwd_allreduce_microstep: 53.63 | step_microstep: 18.75
[2025-04-26 00:53:53,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.50 | bwd: 5687.00 | bwd_inner: 5633.31 | bwd_allreduce: 53.64 | step: 18.76

 17%|█▋        | 7011/41250 [16:56:19<82:11:31,  8.64s/it]
                                                          
{'loss': 0.0852, 'grad_norm': 1.9846165180206299, 'learning_rate': 3.798036072974284e-05, 'epoch': 1.7}

 17%|█▋        | 7011/41250 [16:56:19<82:11:31,  8.64s/it][2025-04-26 00:54:02,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 00:54:02,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5710.96 | bwd_inner_microstep: 5698.14 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.92
[2025-04-26 00:54:02,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5710.97 | bwd_inner: 5698.14 | bwd_allreduce: 12.79 | step: 18.93

 17%|█▋        | 7012/41250 [16:56:27<82:11:30,  8.64s/it]
                                                          
{'loss': 0.0941, 'grad_norm': 1.8414827585220337, 'learning_rate': 3.797967301068603e-05, 'epoch': 1.7}

 17%|█▋        | 7012/41250 [16:56:27<82:11:30,  8.64s/it][2025-04-26 00:54:11,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 0.91
[2025-04-26 00:54:11,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5736.41 | bwd_inner_microstep: 5693.54 | bwd_allreduce_microstep: 42.81 | step_microstep: 19.02
[2025-04-26 00:54:11,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.73 | bwd: 5736.42 | bwd_inner: 5693.54 | bwd_allreduce: 42.83 | step: 19.01

 17%|█▋        | 7013/41250 [16:56:36<82:17:24,  8.65s/it]
                                                          
{'loss': 0.1787, 'grad_norm': 2.5969836711883545, 'learning_rate': 3.7978985180788065e-05, 'epoch': 1.7}

 17%|█▋        | 7013/41250 [16:56:36<82:17:24,  8.65s/it][2025-04-26 00:54:19,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 00:54:19,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.76 | bwd_microstep: 5743.56 | bwd_inner_microstep: 5695.06 | bwd_allreduce_microstep: 48.45 | step_microstep: 19.15
[2025-04-26 00:54:19,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.76 | bwd: 5743.57 | bwd_inner: 5695.06 | bwd_allreduce: 48.47 | step: 19.15

 17%|█▋        | 7014/41250 [16:56:45<82:21:36,  8.66s/it]
                                                          
{'loss': 0.0746, 'grad_norm': 1.34222412109375, 'learning_rate': 3.797829724005319e-05, 'epoch': 1.7}

 17%|█▋        | 7014/41250 [16:56:45<82:21:36,  8.66s/it][2025-04-26 00:54:28,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 00:54:28,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.86 | bwd_microstep: 5700.80 | bwd_inner_microstep: 5641.71 | bwd_allreduce_microstep: 59.04 | step_microstep: 18.84
[2025-04-26 00:54:28,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.86 | bwd: 5700.81 | bwd_inner: 5641.71 | bwd_allreduce: 59.06 | step: 18.84

 17%|█▋        | 7015/41250 [16:56:53<82:12:04,  8.64s/it]
                                                          
{'loss': 0.2725, 'grad_norm': 3.7859346866607666, 'learning_rate': 3.797760918848566e-05, 'epoch': 1.7}

 17%|█▋        | 7015/41250 [16:56:53<82:12:04,  8.64s/it][2025-04-26 00:54:37,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 00:54:37,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.44 | bwd_microstep: 5782.12 | bwd_inner_microstep: 5639.27 | bwd_allreduce_microstep: 142.79 | step_microstep: 19.29
[2025-04-26 00:54:37,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.44 | bwd: 5782.13 | bwd_inner: 5639.27 | bwd_allreduce: 142.81 | step: 19.30

 17%|█▋        | 7016/41250 [16:57:02<82:19:33,  8.66s/it]
                                                          
{'loss': 0.2548, 'grad_norm': 3.954528331756592, 'learning_rate': 3.79769210260897e-05, 'epoch': 1.7}

 17%|█▋        | 7016/41250 [16:57:02<82:19:33,  8.66s/it][2025-04-26 00:54:45,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 00:54:45,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.35 | bwd_microstep: 5770.00 | bwd_inner_microstep: 5635.59 | bwd_allreduce_microstep: 134.36 | step_microstep: 18.79
[2025-04-26 00:54:45,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.35 | bwd: 5770.01 | bwd_inner: 5635.59 | bwd_allreduce: 134.38 | step: 18.79

 17%|█▋        | 7017/41250 [16:57:11<82:22:08,  8.66s/it]
                                                          
{'loss': 0.0347, 'grad_norm': 1.4085867404937744, 'learning_rate': 3.797623275286955e-05, 'epoch': 1.7}

 17%|█▋        | 7017/41250 [16:57:11<82:22:08,  8.66s/it][2025-04-26 00:54:54,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 00:54:54,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.94 | bwd_microstep: 5742.76 | bwd_inner_microstep: 5675.91 | bwd_allreduce_microstep: 66.80 | step_microstep: 19.49
[2025-04-26 00:54:54,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.94 | bwd: 5742.78 | bwd_inner: 5675.91 | bwd_allreduce: 66.82 | step: 19.49

 17%|█▋        | 7018/41250 [16:57:19<82:23:17,  8.66s/it]
                                                          
{'loss': 0.2669, 'grad_norm': 2.1174979209899902, 'learning_rate': 3.7975544368829475e-05, 'epoch': 1.7}

 17%|█▋        | 7018/41250 [16:57:19<82:23:17,  8.66s/it][2025-04-26 00:55:03,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 00:55:03,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.05 | bwd_microstep: 5686.04 | bwd_inner_microstep: 5642.02 | bwd_allreduce_microstep: 43.98 | step_microstep: 18.90
[2025-04-26 00:55:03,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.05 | bwd: 5686.05 | bwd_inner: 5642.02 | bwd_allreduce: 44.00 | step: 18.91

 17%|█▋        | 7019/41250 [16:57:28<82:10:02,  8.64s/it]
                                                          
{'loss': 0.0511, 'grad_norm': 1.434583306312561, 'learning_rate': 3.7974855873973696e-05, 'epoch': 1.7}

 17%|█▋        | 7019/41250 [16:57:28<82:10:02,  8.64s/it][2025-04-26 00:55:11,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:55:11,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.05 | bwd_microstep: 5740.31 | bwd_inner_microstep: 5666.66 | bwd_allreduce_microstep: 73.60 | step_microstep: 19.17
[2025-04-26 00:55:11,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.05 | bwd: 5740.33 | bwd_inner: 5666.66 | bwd_allreduce: 73.62 | step: 19.17

 17%|█▋        | 7020/41250 [16:57:37<82:13:43,  8.65s/it]
                                                          
{'loss': 0.1478, 'grad_norm': 2.187274694442749, 'learning_rate': 3.797416726830647e-05, 'epoch': 1.7}

 17%|█▋        | 7020/41250 [16:57:37<82:13:43,  8.65s/it][2025-04-26 00:55:20,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-26 00:55:20,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.02 | bwd_microstep: 5684.82 | bwd_inner_microstep: 5672.16 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.82
[2025-04-26 00:55:20,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.02 | bwd: 5684.84 | bwd_inner: 5672.16 | bwd_allreduce: 12.63 | step: 18.82

 17%|█▋        | 7021/41250 [16:57:45<82:07:20,  8.64s/it]
                                                          
{'loss': 0.0714, 'grad_norm': 1.92880117893219, 'learning_rate': 3.797347855183203e-05, 'epoch': 1.7}

 17%|█▋        | 7021/41250 [16:57:45<82:07:20,  8.64s/it][2025-04-26 00:55:29,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.13 | optimizer_step: 1.07
[2025-04-26 00:55:29,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.56 | bwd_microstep: 5779.30 | bwd_inner_microstep: 5640.02 | bwd_allreduce_microstep: 139.22 | step_microstep: 20.04
[2025-04-26 00:55:29,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.56 | bwd: 5779.32 | bwd_inner: 5640.02 | bwd_allreduce: 139.25 | step: 20.05

 17%|█▋        | 7022/41250 [16:57:54<82:15:05,  8.65s/it]
                                                          
{'loss': 0.3093, 'grad_norm': 4.567498683929443, 'learning_rate': 3.797278972455464e-05, 'epoch': 1.7}

 17%|█▋        | 7022/41250 [16:57:54<82:15:05,  8.65s/it][2025-04-26 00:55:37,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 00:55:37,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.29 | bwd_microstep: 5742.30 | bwd_inner_microstep: 5687.68 | bwd_allreduce_microstep: 54.56 | step_microstep: 18.83
[2025-04-26 00:55:37,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.29 | bwd: 5742.32 | bwd_inner: 5687.68 | bwd_allreduce: 54.59 | step: 18.84

 17%|█▋        | 7023/41250 [16:58:03<82:20:39,  8.66s/it]
                                                          
{'loss': 0.2636, 'grad_norm': 3.793158769607544, 'learning_rate': 3.797210078647853e-05, 'epoch': 1.7}

 17%|█▋        | 7023/41250 [16:58:03<82:20:39,  8.66s/it][2025-04-26 00:55:46,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 00:55:46,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.73 | bwd_microstep: 5735.26 | bwd_inner_microstep: 5693.70 | bwd_allreduce_microstep: 41.51 | step_microstep: 19.32
[2025-04-26 00:55:46,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.73 | bwd: 5735.27 | bwd_inner: 5693.70 | bwd_allreduce: 41.53 | step: 19.32

 17%|█▋        | 7024/41250 [16:58:11<82:19:29,  8.66s/it]
                                                          
{'loss': 0.2823, 'grad_norm': 3.1399192810058594, 'learning_rate': 3.7971411737607956e-05, 'epoch': 1.7}

 17%|█▋        | 7024/41250 [16:58:11<82:19:29,  8.66s/it][2025-04-26 00:55:55,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 00:55:55,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5745.03 | bwd_inner_microstep: 5709.58 | bwd_allreduce_microstep: 35.40 | step_microstep: 19.44
[2025-04-26 00:55:55,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5745.04 | bwd_inner: 5709.58 | bwd_allreduce: 35.42 | step: 19.44

 17%|█▋        | 7025/41250 [16:58:20<82:23:42,  8.67s/it]
                                                          
{'loss': 0.1565, 'grad_norm': 2.6501197814941406, 'learning_rate': 3.797072257794717e-05, 'epoch': 1.7}

 17%|█▋        | 7025/41250 [16:58:20<82:23:42,  8.67s/it][2025-04-26 00:56:03,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:56:03,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.25 | bwd_microstep: 5698.20 | bwd_inner_microstep: 5685.46 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.92
[2025-04-26 00:56:03,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.25 | bwd: 5698.21 | bwd_inner: 5685.46 | bwd_allreduce: 12.71 | step: 18.93

 17%|█▋        | 7026/41250 [16:58:29<82:16:12,  8.65s/it]
                                                          
{'loss': 0.1606, 'grad_norm': 2.4433512687683105, 'learning_rate': 3.7970033307500405e-05, 'epoch': 1.7}

 17%|█▋        | 7026/41250 [16:58:29<82:16:12,  8.65s/it][2025-04-26 00:56:12,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:56:12,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5703.50 | bwd_inner_microstep: 5690.58 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.92
[2025-04-26 00:56:12,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5703.51 | bwd_inner: 5690.58 | bwd_allreduce: 12.89 | step: 18.92

 17%|█▋        | 7027/41250 [16:58:37<82:12:38,  8.65s/it]
                                                          
{'loss': 0.1468, 'grad_norm': 1.1939469575881958, 'learning_rate': 3.796934392627192e-05, 'epoch': 1.7}

 17%|█▋        | 7027/41250 [16:58:37<82:12:38,  8.65s/it][2025-04-26 00:56:21,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 00:56:21,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5739.78 | bwd_inner_microstep: 5710.34 | bwd_allreduce_microstep: 29.39 | step_microstep: 19.37
[2025-04-26 00:56:21,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5739.79 | bwd_inner: 5710.34 | bwd_allreduce: 29.41 | step: 19.37

 17%|█▋        | 7028/41250 [16:58:46<82:17:11,  8.66s/it]
                                                          
{'loss': 0.3679, 'grad_norm': 1.8687437772750854, 'learning_rate': 3.796865443426596e-05, 'epoch': 1.7}

 17%|█▋        | 7028/41250 [16:58:46<82:17:11,  8.66s/it][2025-04-26 00:56:29,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 00:56:29,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.89 | bwd_microstep: 5776.02 | bwd_inner_microstep: 5634.32 | bwd_allreduce_microstep: 141.66 | step_microstep: 18.48
[2025-04-26 00:56:29,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.89 | bwd: 5776.03 | bwd_inner: 5634.32 | bwd_allreduce: 141.68 | step: 18.49

 17%|█▋        | 7029/41250 [16:58:55<82:21:54,  8.66s/it]
                                                          
{'loss': 0.0601, 'grad_norm': 1.3135274648666382, 'learning_rate': 3.7967964831486784e-05, 'epoch': 1.7}

 17%|█▋        | 7029/41250 [16:58:55<82:21:54,  8.66s/it][2025-04-26 00:56:38,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 00:56:38,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.02 | bwd_microstep: 5764.79 | bwd_inner_microstep: 5752.03 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.72
[2025-04-26 00:56:38,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.02 | bwd: 5764.80 | bwd_inner: 5752.03 | bwd_allreduce: 12.73 | step: 18.72

 17%|█▋        | 7030/41250 [16:59:03<82:31:43,  8.68s/it]
                                                          
{'loss': 0.2223, 'grad_norm': 2.076925039291382, 'learning_rate': 3.796727511793864e-05, 'epoch': 1.7}

 17%|█▋        | 7030/41250 [16:59:03<82:31:43,  8.68s/it][2025-04-26 00:56:47,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:56:47,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.64 | bwd_microstep: 5716.45 | bwd_inner_microstep: 5644.98 | bwd_allreduce_microstep: 71.42 | step_microstep: 18.47
[2025-04-26 00:56:47,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.64 | bwd: 5716.46 | bwd_inner: 5644.98 | bwd_allreduce: 71.44 | step: 18.48

 17%|█▋        | 7031/41250 [16:59:12<82:22:15,  8.67s/it]
                                                          
{'loss': 0.3293, 'grad_norm': 2.8108103275299072, 'learning_rate': 3.796658529362578e-05, 'epoch': 1.7}

 17%|█▋        | 7031/41250 [16:59:12<82:22:15,  8.67s/it][2025-04-26 00:56:55,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:56:55,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.10 | bwd_microstep: 5774.55 | bwd_inner_microstep: 5761.98 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.57
[2025-04-26 00:56:55,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.10 | bwd: 5774.56 | bwd_inner: 5761.98 | bwd_allreduce: 12.55 | step: 18.57

 17%|█▋        | 7032/41250 [16:59:21<82:34:10,  8.69s/it]
                                                          
{'loss': 0.1085, 'grad_norm': 1.1664676666259766, 'learning_rate': 3.7965895358552457e-05, 'epoch': 1.7}

 17%|█▋        | 7032/41250 [16:59:21<82:34:10,  8.69s/it][2025-04-26 00:57:04,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 00:57:04,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.03 | bwd_microstep: 5725.46 | bwd_inner_microstep: 5649.13 | bwd_allreduce_microstep: 76.28 | step_microstep: 18.80
[2025-04-26 00:57:04,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.03 | bwd: 5725.48 | bwd_inner: 5649.13 | bwd_allreduce: 76.30 | step: 18.80

 17%|█▋        | 7033/41250 [16:59:29<82:25:04,  8.67s/it]
                                                          
{'loss': 0.0496, 'grad_norm': 0.5667318105697632, 'learning_rate': 3.796520531272292e-05, 'epoch': 1.7}

 17%|█▋        | 7033/41250 [16:59:29<82:25:04,  8.67s/it][2025-04-26 00:57:13,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:57:13,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.41 | bwd_microstep: 5790.75 | bwd_inner_microstep: 5778.26 | bwd_allreduce_microstep: 12.45 | step_microstep: 18.52
[2025-04-26 00:57:13,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.41 | bwd: 5790.77 | bwd_inner: 5778.26 | bwd_allreduce: 12.46 | step: 18.52

 17%|█▋        | 7034/41250 [16:59:38<82:41:02,  8.70s/it]
                                                          
{'loss': 0.1617, 'grad_norm': 1.2958424091339111, 'learning_rate': 3.796451515614142e-05, 'epoch': 1.71}

 17%|█▋        | 7034/41250 [16:59:38<82:41:02,  8.70s/it][2025-04-26 00:57:22,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 00:57:22,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.04 | bwd_microstep: 5779.61 | bwd_inner_microstep: 5657.32 | bwd_allreduce_microstep: 122.25 | step_microstep: 19.00
[2025-04-26 00:57:22,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.04 | bwd: 5779.62 | bwd_inner: 5657.32 | bwd_allreduce: 122.26 | step: 19.01

 17%|█▋        | 7035/41250 [16:59:47<82:40:23,  8.70s/it]
                                                          
{'loss': 0.0655, 'grad_norm': 1.3965504169464111, 'learning_rate': 3.796382488881223e-05, 'epoch': 1.71}

 17%|█▋        | 7035/41250 [16:59:47<82:40:23,  8.70s/it][2025-04-26 00:57:30,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 00:57:30,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.36 | bwd_microstep: 5724.43 | bwd_inner_microstep: 5711.29 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.27
[2025-04-26 00:57:30,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.36 | bwd: 5724.44 | bwd_inner: 5711.29 | bwd_allreduce: 13.11 | step: 19.28

 17%|█▋        | 7036/41250 [16:59:55<82:34:15,  8.69s/it]
                                                          
{'loss': 0.1227, 'grad_norm': 2.257054328918457, 'learning_rate': 3.7963134510739585e-05, 'epoch': 1.71}

 17%|█▋        | 7036/41250 [16:59:55<82:34:15,  8.69s/it][2025-04-26 00:57:39,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 00:57:39,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.04 | bwd_microstep: 5792.17 | bwd_inner_microstep: 5658.46 | bwd_allreduce_microstep: 133.66 | step_microstep: 18.85
[2025-04-26 00:57:39,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.04 | bwd: 5792.19 | bwd_inner: 5658.46 | bwd_allreduce: 133.68 | step: 18.85

 17%|█▋        | 7037/41250 [17:00:04<82:37:23,  8.69s/it]
                                                          
{'loss': 0.2032, 'grad_norm': 1.2513153553009033, 'learning_rate': 3.796244402192775e-05, 'epoch': 1.71}

 17%|█▋        | 7037/41250 [17:00:04<82:37:23,  8.69s/it][2025-04-26 00:57:48,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:57:48,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.57 | bwd_microstep: 5729.49 | bwd_inner_microstep: 5716.75 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.65
[2025-04-26 00:57:48,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.57 | bwd: 5729.50 | bwd_inner: 5716.75 | bwd_allreduce: 12.71 | step: 18.66

 17%|█▋        | 7038/41250 [17:00:13<82:32:23,  8.69s/it]
                                                          
{'loss': 0.2095, 'grad_norm': 1.4077506065368652, 'learning_rate': 3.796175342238097e-05, 'epoch': 1.71}

 17%|█▋        | 7038/41250 [17:00:13<82:32:23,  8.69s/it][2025-04-26 00:57:56,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:57:56,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.88 | bwd_microstep: 5729.06 | bwd_inner_microstep: 5655.06 | bwd_allreduce_microstep: 73.95 | step_microstep: 18.46
[2025-04-26 00:57:56,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.88 | bwd: 5729.07 | bwd_inner: 5655.06 | bwd_allreduce: 73.97 | step: 18.46

 17%|█▋        | 7039/41250 [17:00:22<82:24:14,  8.67s/it]
                                                          
{'loss': 0.1809, 'grad_norm': 2.704439640045166, 'learning_rate': 3.7961062712103534e-05, 'epoch': 1.71}

 17%|█▋        | 7039/41250 [17:00:22<82:24:14,  8.67s/it][2025-04-26 00:58:05,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 00:58:05,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.01 | bwd_microstep: 5719.44 | bwd_inner_microstep: 5686.37 | bwd_allreduce_microstep: 33.03 | step_microstep: 18.31
[2025-04-26 00:58:05,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.01 | bwd: 5719.46 | bwd_inner: 5686.37 | bwd_allreduce: 33.05 | step: 18.31

 17%|█▋        | 7040/41250 [17:00:30<82:20:06,  8.66s/it]
                                                          
{'loss': 0.1029, 'grad_norm': 1.3263033628463745, 'learning_rate': 3.796037189109967e-05, 'epoch': 1.71}

 17%|█▋        | 7040/41250 [17:00:30<82:20:06,  8.66s/it][2025-04-26 00:58:13,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 00:58:13,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.75 | bwd_microstep: 5705.66 | bwd_inner_microstep: 5654.27 | bwd_allreduce_microstep: 51.34 | step_microstep: 18.56
[2025-04-26 00:58:13,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.75 | bwd: 5705.67 | bwd_inner: 5654.27 | bwd_allreduce: 51.36 | step: 18.57

 17%|█▋        | 7041/41250 [17:00:39<82:11:39,  8.65s/it]
                                                          
{'loss': 0.0497, 'grad_norm': 1.1007457971572876, 'learning_rate': 3.795968095937365e-05, 'epoch': 1.71}

 17%|█▋        | 7041/41250 [17:00:39<82:11:39,  8.65s/it][2025-04-26 00:58:22,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 00:58:22,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.79 | bwd_microstep: 5756.37 | bwd_inner_microstep: 5714.16 | bwd_allreduce_microstep: 42.16 | step_microstep: 18.39
[2025-04-26 00:58:22,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.79 | bwd: 5756.39 | bwd_inner: 5714.16 | bwd_allreduce: 42.18 | step: 18.41

 17%|█▋        | 7042/41250 [17:00:47<82:17:58,  8.66s/it]
                                                          
{'loss': 0.1179, 'grad_norm': 1.6440733671188354, 'learning_rate': 3.795898991692972e-05, 'epoch': 1.71}

 17%|█▋        | 7042/41250 [17:00:47<82:17:58,  8.66s/it][2025-04-26 00:58:31,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-26 00:58:31,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.49 | bwd_microstep: 5719.44 | bwd_inner_microstep: 5706.71 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.84
[2025-04-26 00:58:31,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.49 | bwd: 5719.46 | bwd_inner: 5706.71 | bwd_allreduce: 12.71 | step: 18.85

 17%|█▋        | 7043/41250 [17:00:56<82:16:39,  8.66s/it]
                                                          
{'loss': 0.0416, 'grad_norm': 0.5681148171424866, 'learning_rate': 3.795829876377215e-05, 'epoch': 1.71}

 17%|█▋        | 7043/41250 [17:00:56<82:16:39,  8.66s/it][2025-04-26 00:58:39,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 00:58:39,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5751.24 | bwd_inner_microstep: 5699.43 | bwd_allreduce_microstep: 51.77 | step_microstep: 18.59
[2025-04-26 00:58:39,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5751.25 | bwd_inner: 5699.43 | bwd_allreduce: 51.78 | step: 18.59

 17%|█▋        | 7044/41250 [17:01:05<82:20:52,  8.67s/it]
                                                          
{'loss': 0.3254, 'grad_norm': 1.671541452407837, 'learning_rate': 3.79576074999052e-05, 'epoch': 1.71}

 17%|█▋        | 7044/41250 [17:01:05<82:20:52,  8.67s/it][2025-04-26 00:58:48,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 00:58:48,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.58 | bwd_microstep: 5786.27 | bwd_inner_microstep: 5773.63 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.49
[2025-04-26 00:58:48,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.58 | bwd: 5786.28 | bwd_inner: 5773.63 | bwd_allreduce: 12.61 | step: 18.49

 17%|█▋        | 7045/41250 [17:01:14<82:36:55,  8.70s/it]
                                                          
{'loss': 0.0598, 'grad_norm': 0.9211239218711853, 'learning_rate': 3.795691612533314e-05, 'epoch': 1.71}

 17%|█▋        | 7045/41250 [17:01:14<82:36:55,  8.70s/it][2025-04-26 00:58:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:58:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.38 | bwd_microstep: 5694.22 | bwd_inner_microstep: 5681.59 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.74
[2025-04-26 00:58:57,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.38 | bwd: 5694.24 | bwd_inner: 5681.59 | bwd_allreduce: 12.61 | step: 18.74

 17%|█▋        | 7046/41250 [17:01:22<82:23:52,  8.67s/it]
                                                          
{'loss': 0.1605, 'grad_norm': 2.265822410583496, 'learning_rate': 3.7956224640060216e-05, 'epoch': 1.71}

 17%|█▋        | 7046/41250 [17:01:22<82:23:52,  8.67s/it][2025-04-26 00:59:05,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 00:59:05,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.13 | bwd_microstep: 5710.27 | bwd_inner_microstep: 5697.51 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.89
[2025-04-26 00:59:05,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.13 | bwd: 5710.29 | bwd_inner: 5697.51 | bwd_allreduce: 12.74 | step: 18.90

 17%|█▋        | 7047/41250 [17:01:31<82:18:16,  8.66s/it]
                                                          
{'loss': 0.3021, 'grad_norm': 1.9999219179153442, 'learning_rate': 3.7955533044090704e-05, 'epoch': 1.71}

 17%|█▋        | 7047/41250 [17:01:31<82:18:16,  8.66s/it][2025-04-26 00:59:14,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 00:59:14,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5723.79 | bwd_inner_microstep: 5650.84 | bwd_allreduce_microstep: 72.91 | step_microstep: 18.30
[2025-04-26 00:59:14,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5723.80 | bwd_inner: 5650.84 | bwd_allreduce: 72.92 | step: 18.30

 17%|█▋        | 7048/41250 [17:01:39<82:12:22,  8.65s/it]
                                                          
{'loss': 0.2233, 'grad_norm': 2.030385971069336, 'learning_rate': 3.7954841337428866e-05, 'epoch': 1.71}

 17%|█▋        | 7048/41250 [17:01:39<82:12:22,  8.65s/it][2025-04-26 00:59:23,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 00:59:23,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.83 | bwd_microstep: 5728.05 | bwd_inner_microstep: 5698.24 | bwd_allreduce_microstep: 29.76 | step_microstep: 18.59
[2025-04-26 00:59:23,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.83 | bwd: 5728.06 | bwd_inner: 5698.24 | bwd_allreduce: 29.78 | step: 18.59

 17%|█▋        | 7049/41250 [17:01:48<82:13:46,  8.66s/it]
                                                          
{'loss': 0.3644, 'grad_norm': 2.9135279655456543, 'learning_rate': 3.795414952007895e-05, 'epoch': 1.71}

 17%|█▋        | 7049/41250 [17:01:48<82:13:46,  8.66s/it][2025-04-26 00:59:31,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 00:59:31,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.07 | bwd_microstep: 5778.81 | bwd_inner_microstep: 5652.66 | bwd_allreduce_microstep: 126.10 | step_microstep: 19.12
[2025-04-26 00:59:31,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.07 | bwd: 5778.83 | bwd_inner: 5652.66 | bwd_allreduce: 126.12 | step: 19.12

 17%|█▋        | 7050/41250 [17:01:57<82:19:08,  8.67s/it]
                                                          
{'loss': 0.2092, 'grad_norm': 1.9865286350250244, 'learning_rate': 3.795345759204525e-05, 'epoch': 1.71}

 17%|█▋        | 7050/41250 [17:01:57<82:19:08,  8.67s/it][2025-04-26 00:59:40,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-26 00:59:40,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.67 | bwd_microstep: 5688.39 | bwd_inner_microstep: 5647.48 | bwd_allreduce_microstep: 40.86 | step_microstep: 19.98
[2025-04-26 00:59:40,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.67 | bwd: 5688.41 | bwd_inner: 5647.48 | bwd_allreduce: 40.88 | step: 19.98

 17%|█▋        | 7051/41250 [17:02:05<82:06:50,  8.64s/it]
                                                          
{'loss': 0.3999, 'grad_norm': 3.66530442237854, 'learning_rate': 3.7952765553332e-05, 'epoch': 1.71}

 17%|█▋        | 7051/41250 [17:02:05<82:06:50,  8.64s/it][2025-04-26 00:59:49,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-26 00:59:49,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.66 | bwd_microstep: 5845.78 | bwd_inner_microstep: 5691.39 | bwd_allreduce_microstep: 154.33 | step_microstep: 19.04
[2025-04-26 00:59:49,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.66 | bwd: 5845.79 | bwd_inner: 5691.39 | bwd_allreduce: 154.35 | step: 19.04

 17%|█▋        | 7052/41250 [17:02:14<82:29:59,  8.68s/it]
                                                          
{'loss': 0.1811, 'grad_norm': 2.2829110622406006, 'learning_rate': 3.795207340394349e-05, 'epoch': 1.71}

 17%|█▋        | 7052/41250 [17:02:14<82:29:59,  8.68s/it][2025-04-26 00:59:58,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 00:59:58,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.68 | bwd_microstep: 5788.33 | bwd_inner_microstep: 5775.36 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.47
[2025-04-26 00:59:58,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.68 | bwd: 5788.35 | bwd_inner: 5775.36 | bwd_allreduce: 12.95 | step: 19.47

 17%|█▋        | 7053/41250 [17:02:23<82:43:57,  8.71s/it]
                                                          
{'loss': 0.0861, 'grad_norm': 1.3033877611160278, 'learning_rate': 3.7951381143883974e-05, 'epoch': 1.71}

 17%|█▋        | 7053/41250 [17:02:23<82:43:57,  8.71s/it][2025-04-26 01:00:06,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-26 01:00:06,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.93 | bwd_microstep: 5738.61 | bwd_inner_microstep: 5648.59 | bwd_allreduce_microstep: 89.97 | step_microstep: 19.36
[2025-04-26 01:00:06,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.93 | bwd: 5738.63 | bwd_inner: 5648.59 | bwd_allreduce: 89.99 | step: 19.37

 17%|█▋        | 7054/41250 [17:02:32<82:33:40,  8.69s/it]
                                                          
{'loss': 0.3598, 'grad_norm': 3.472356081008911, 'learning_rate': 3.795068877315773e-05, 'epoch': 1.71}

 17%|█▋        | 7054/41250 [17:02:32<82:33:40,  8.69s/it][2025-04-26 01:00:15,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.19 | optimizer_step: 0.91
[2025-04-26 01:00:15,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.06 | bwd_microstep: 5765.73 | bwd_inner_microstep: 5752.79 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.25
[2025-04-26 01:00:15,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.06 | bwd: 5765.75 | bwd_inner: 5752.79 | bwd_allreduce: 12.91 | step: 19.27

 17%|█▋        | 7055/41250 [17:02:40<82:39:22,  8.70s/it]
                                                          
{'loss': 0.0801, 'grad_norm': 0.7055622339248657, 'learning_rate': 3.794999629176902e-05, 'epoch': 1.71}

 17%|█▋        | 7055/41250 [17:02:40<82:39:22,  8.70s/it][2025-04-26 01:00:24,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 01:00:24,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.01 | bwd_microstep: 5871.81 | bwd_inner_microstep: 5858.61 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.33
[2025-04-26 01:00:24,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.01 | bwd: 5871.82 | bwd_inner: 5858.61 | bwd_allreduce: 13.16 | step: 19.34

 17%|█▋        | 7056/41250 [17:02:49<83:10:30,  8.76s/it]
                                                          
{'loss': 0.0905, 'grad_norm': 1.0514405965805054, 'learning_rate': 3.794930369972211e-05, 'epoch': 1.71}

 17%|█▋        | 7056/41250 [17:02:49<83:10:30,  8.76s/it][2025-04-26 01:00:33,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 01:00:33,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.84 | bwd_microstep: 5734.00 | bwd_inner_microstep: 5645.88 | bwd_allreduce_microstep: 88.07 | step_microstep: 18.96
[2025-04-26 01:00:33,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.84 | bwd: 5734.02 | bwd_inner: 5645.88 | bwd_allreduce: 88.09 | step: 18.96

 17%|█▋        | 7057/41250 [17:02:58<82:52:54,  8.73s/it]
                                                          
{'loss': 0.0354, 'grad_norm': 0.6883544325828552, 'learning_rate': 3.794861099702127e-05, 'epoch': 1.71}

 17%|█▋        | 7057/41250 [17:02:58<82:52:54,  8.73s/it][2025-04-26 01:00:41,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.32 | optimizer_step: 0.99
[2025-04-26 01:00:41,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.65 | bwd_microstep: 5743.57 | bwd_inner_microstep: 5646.26 | bwd_allreduce_microstep: 97.25 | step_microstep: 19.96
[2025-04-26 01:00:41,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.65 | bwd: 5743.58 | bwd_inner: 5646.26 | bwd_allreduce: 97.28 | step: 19.96

 17%|█▋        | 7058/41250 [17:03:07<82:40:13,  8.70s/it]
                                                          
{'loss': 0.1974, 'grad_norm': 2.3389463424682617, 'learning_rate': 3.794791818367077e-05, 'epoch': 1.71}

 17%|█▋        | 7058/41250 [17:03:07<82:40:13,  8.70s/it][2025-04-26 01:00:50,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:00:50,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.38 | bwd_microstep: 5673.28 | bwd_inner_microstep: 5640.25 | bwd_allreduce_microstep: 32.98 | step_microstep: 18.48
[2025-04-26 01:00:50,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.38 | bwd: 5673.29 | bwd_inner: 5640.25 | bwd_allreduce: 33.00 | step: 18.49

 17%|█▋        | 7059/41250 [17:03:15<82:17:55,  8.67s/it]
                                                          
{'loss': 0.0899, 'grad_norm': 1.1894468069076538, 'learning_rate': 3.7947225259674894e-05, 'epoch': 1.71}

 17%|█▋        | 7059/41250 [17:03:15<82:17:55,  8.67s/it][2025-04-26 01:00:58,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 01:00:58,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.38 | bwd_microstep: 5768.28 | bwd_inner_microstep: 5755.59 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.04
[2025-04-26 01:00:58,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.38 | bwd: 5768.30 | bwd_inner: 5755.59 | bwd_allreduce: 12.67 | step: 19.05

 17%|█▋        | 7060/41250 [17:03:24<82:29:34,  8.69s/it]
                                                          
{'loss': 0.1762, 'grad_norm': 1.5145111083984375, 'learning_rate': 3.7946532225037905e-05, 'epoch': 1.71}

 17%|█▋        | 7060/41250 [17:03:24<82:29:34,  8.69s/it][2025-04-26 01:01:07,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:01:07,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.66 | bwd_microstep: 5694.96 | bwd_inner_microstep: 5643.79 | bwd_allreduce_microstep: 51.12 | step_microstep: 18.76
[2025-04-26 01:01:07,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.66 | bwd: 5694.97 | bwd_inner: 5643.79 | bwd_allreduce: 51.14 | step: 18.77

 17%|█▋        | 7061/41250 [17:03:32<82:15:02,  8.66s/it]
                                                          
{'loss': 0.2123, 'grad_norm': 2.537604570388794, 'learning_rate': 3.7945839079764075e-05, 'epoch': 1.71}

 17%|█▋        | 7061/41250 [17:03:32<82:15:02,  8.66s/it][2025-04-26 01:01:16,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:01:16,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.71 | bwd_microstep: 5718.67 | bwd_inner_microstep: 5705.87 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.59
[2025-04-26 01:01:16,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.71 | bwd: 5718.68 | bwd_inner: 5705.87 | bwd_allreduce: 12.77 | step: 18.60

 17%|█▋        | 7062/41250 [17:03:41<82:13:04,  8.66s/it]
                                                          
{'loss': 0.1366, 'grad_norm': 1.2553917169570923, 'learning_rate': 3.794514582385767e-05, 'epoch': 1.71}

 17%|█▋        | 7062/41250 [17:03:41<82:13:04,  8.66s/it][2025-04-26 01:01:24,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:01:24,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.38 | bwd_microstep: 5780.71 | bwd_inner_microstep: 5637.01 | bwd_allreduce_microstep: 143.65 | step_microstep: 18.33
[2025-04-26 01:01:24,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.38 | bwd: 5780.72 | bwd_inner: 5637.01 | bwd_allreduce: 143.67 | step: 18.34

 17%|█▋        | 7063/41250 [17:03:50<82:16:38,  8.66s/it]
                                                          
{'loss': 0.1658, 'grad_norm': 1.2543373107910156, 'learning_rate': 3.794445245732297e-05, 'epoch': 1.71}

 17%|█▋        | 7063/41250 [17:03:50<82:16:38,  8.66s/it][2025-04-26 01:01:33,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:01:33,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.49 | bwd_microstep: 5747.61 | bwd_inner_microstep: 5651.18 | bwd_allreduce_microstep: 96.39 | step_microstep: 18.11
[2025-04-26 01:01:33,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.49 | bwd: 5747.62 | bwd_inner: 5651.18 | bwd_allreduce: 96.40 | step: 18.11

 17%|█▋        | 7064/41250 [17:03:58<82:14:34,  8.66s/it]
                                                          
{'loss': 0.2393, 'grad_norm': 5.216795444488525, 'learning_rate': 3.794375898016426e-05, 'epoch': 1.71}

 17%|█▋        | 7064/41250 [17:03:58<82:14:34,  8.66s/it][2025-04-26 01:01:42,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.04
[2025-04-26 01:01:42,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.81 | bwd_microstep: 5700.22 | bwd_inner_microstep: 5663.86 | bwd_allreduce_microstep: 36.32 | step_microstep: 18.54
[2025-04-26 01:01:42,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.81 | bwd: 5700.23 | bwd_inner: 5663.86 | bwd_allreduce: 36.33 | step: 18.54

 17%|█▋        | 7065/41250 [17:04:07<82:05:28,  8.64s/it]
                                                          
{'loss': 0.1243, 'grad_norm': 1.6073555946350098, 'learning_rate': 3.79430653923858e-05, 'epoch': 1.71}

 17%|█▋        | 7065/41250 [17:04:07<82:05:28,  8.64s/it][2025-04-26 01:01:50,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 01:01:50,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.74 | bwd_microstep: 5772.69 | bwd_inner_microstep: 5644.67 | bwd_allreduce_microstep: 127.98 | step_microstep: 18.28
[2025-04-26 01:01:50,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.74 | bwd: 5772.70 | bwd_inner: 5644.67 | bwd_allreduce: 127.99 | step: 18.28

 17%|█▋        | 7066/41250 [17:04:16<82:11:23,  8.66s/it]
                                                          
{'loss': 0.2138, 'grad_norm': 5.101917266845703, 'learning_rate': 3.7942371693991866e-05, 'epoch': 1.71}

 17%|█▋        | 7066/41250 [17:04:16<82:11:23,  8.66s/it][2025-04-26 01:01:59,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-26 01:01:59,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.91 | bwd_microstep: 5719.85 | bwd_inner_microstep: 5707.26 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.23
[2025-04-26 01:01:59,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.91 | bwd: 5719.86 | bwd_inner: 5707.26 | bwd_allreduce: 12.55 | step: 18.23

 17%|█▋        | 7067/41250 [17:04:24<82:14:27,  8.66s/it]
                                                          
{'loss': 0.2515, 'grad_norm': 2.894940137863159, 'learning_rate': 3.7941677884986745e-05, 'epoch': 1.71}

 17%|█▋        | 7067/41250 [17:04:24<82:14:27,  8.66s/it][2025-04-26 01:02:08,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 01:02:08,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.50 | bwd_microstep: 5699.91 | bwd_inner_microstep: 5657.88 | bwd_allreduce_microstep: 41.98 | step_microstep: 18.94
[2025-04-26 01:02:08,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.50 | bwd: 5699.92 | bwd_inner: 5657.88 | bwd_allreduce: 42.00 | step: 18.94

 17%|█▋        | 7068/41250 [17:04:33<82:05:48,  8.65s/it]
                                                          
{'loss': 0.0955, 'grad_norm': 1.8760932683944702, 'learning_rate': 3.7940983965374714e-05, 'epoch': 1.71}

 17%|█▋        | 7068/41250 [17:04:33<82:05:48,  8.65s/it][2025-04-26 01:02:16,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 01:02:16,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.69 | bwd_microstep: 5691.27 | bwd_inner_microstep: 5652.63 | bwd_allreduce_microstep: 38.59 | step_microstep: 18.97
[2025-04-26 01:02:16,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.69 | bwd: 5691.28 | bwd_inner: 5652.63 | bwd_allreduce: 38.60 | step: 18.98

 17%|█▋        | 7069/41250 [17:04:42<81:57:57,  8.63s/it]
                                                          
{'loss': 0.3147, 'grad_norm': 3.034046173095703, 'learning_rate': 3.7940289935160034e-05, 'epoch': 1.71}

 17%|█▋        | 7069/41250 [17:04:42<81:57:57,  8.63s/it][2025-04-26 01:02:25,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 0.96
[2025-04-26 01:02:25,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 6004.19 | bwd_inner_microstep: 5656.91 | bwd_allreduce_microstep: 347.23 | step_microstep: 19.67
[2025-04-26 01:02:25,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.43 | bwd: 6004.21 | bwd_inner: 5656.91 | bwd_allreduce: 347.25 | step: 19.68

 17%|█▋        | 7070/41250 [17:04:51<82:48:36,  8.72s/it]
                                                          
{'loss': 0.0458, 'grad_norm': 0.7000483274459839, 'learning_rate': 3.7939595794347e-05, 'epoch': 1.71}

 17%|█▋        | 7070/41250 [17:04:51<82:48:36,  8.72s/it][2025-04-26 01:02:34,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:02:34,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5694.85 | bwd_inner_microstep: 5653.47 | bwd_allreduce_microstep: 41.33 | step_microstep: 18.80
[2025-04-26 01:02:34,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5694.86 | bwd_inner: 5653.47 | bwd_allreduce: 41.35 | step: 18.80

 17%|█▋        | 7071/41250 [17:04:59<82:28:37,  8.69s/it]
                                                          
{'loss': 0.2367, 'grad_norm': 1.609775185585022, 'learning_rate': 3.7938901542939886e-05, 'epoch': 1.71}

 17%|█▋        | 7071/41250 [17:04:59<82:28:37,  8.69s/it][2025-04-26 01:02:42,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 01:02:42,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.64 | bwd_microstep: 5784.55 | bwd_inner_microstep: 5654.34 | bwd_allreduce_microstep: 130.17 | step_microstep: 18.35
[2025-04-26 01:02:42,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.64 | bwd: 5784.57 | bwd_inner: 5654.34 | bwd_allreduce: 130.19 | step: 18.35

 17%|█▋        | 7072/41250 [17:05:08<82:30:59,  8.69s/it]
                                                          
{'loss': 0.1252, 'grad_norm': 0.8820052742958069, 'learning_rate': 3.7938207180942976e-05, 'epoch': 1.71}

 17%|█▋        | 7072/41250 [17:05:08<82:30:59,  8.69s/it][2025-04-26 01:02:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:02:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.73 | bwd_microstep: 5776.23 | bwd_inner_microstep: 5688.39 | bwd_allreduce_microstep: 87.78 | step_microstep: 18.70
[2025-04-26 01:02:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.73 | bwd: 5776.25 | bwd_inner: 5688.39 | bwd_allreduce: 87.81 | step: 18.70

 17%|█▋        | 7073/41250 [17:05:17<82:33:49,  8.70s/it]
                                                          
{'loss': 0.258, 'grad_norm': 2.389002799987793, 'learning_rate': 3.793751270836055e-05, 'epoch': 1.71}

 17%|█▋        | 7073/41250 [17:05:17<82:33:49,  8.70s/it][2025-04-26 01:03:00,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 01:03:00,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5764.58 | bwd_inner_microstep: 5663.05 | bwd_allreduce_microstep: 101.48 | step_microstep: 18.92
[2025-04-26 01:03:00,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5764.59 | bwd_inner: 5663.05 | bwd_allreduce: 101.49 | step: 18.92

 17%|█▋        | 7074/41250 [17:05:25<82:29:58,  8.69s/it]
                                                          
{'loss': 0.0502, 'grad_norm': 0.7934807538986206, 'learning_rate': 3.793681812519688e-05, 'epoch': 1.71}

 17%|█▋        | 7074/41250 [17:05:25<82:29:58,  8.69s/it][2025-04-26 01:03:09,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-26 01:03:09,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 6024.82 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 322.36 | step_microstep: 18.94
[2025-04-26 01:03:09,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 6024.84 | bwd_inner: 5702.22 | bwd_allreduce: 322.38 | step: 18.94

 17%|█▋        | 7075/41250 [17:05:34<83:16:06,  8.77s/it]
                                                          
{'loss': 0.2137, 'grad_norm': 3.1725993156433105, 'learning_rate': 3.793612343145625e-05, 'epoch': 1.72}

 17%|█▋        | 7075/41250 [17:05:34<83:16:06,  8.77s/it][2025-04-26 01:03:18,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:03:18,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.66 | bwd_microstep: 5760.41 | bwd_inner_microstep: 5689.22 | bwd_allreduce_microstep: 71.15 | step_microstep: 18.78
[2025-04-26 01:03:18,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.66 | bwd: 5760.42 | bwd_inner: 5689.22 | bwd_allreduce: 71.16 | step: 18.78

 17%|█▋        | 7076/41250 [17:05:43<83:02:53,  8.75s/it]
                                                          
{'loss': 0.0818, 'grad_norm': 1.323919653892517, 'learning_rate': 3.7935428627142956e-05, 'epoch': 1.72}

 17%|█▋        | 7076/41250 [17:05:43<83:02:53,  8.75s/it][2025-04-26 01:03:26,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:03:26,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5767.56 | bwd_inner_microstep: 5707.79 | bwd_allreduce_microstep: 59.73 | step_microstep: 18.47
[2025-04-26 01:03:26,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5767.57 | bwd_inner: 5707.79 | bwd_allreduce: 59.74 | step: 18.47

 17%|█▋        | 7077/41250 [17:05:52<82:53:43,  8.73s/it]
                                                          
{'loss': 0.1245, 'grad_norm': 2.420724868774414, 'learning_rate': 3.793473371226126e-05, 'epoch': 1.72}

 17%|█▋        | 7077/41250 [17:05:52<82:53:43,  8.73s/it][2025-04-26 01:03:35,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 01:03:35,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5707.03 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 55.18 | step_microstep: 18.69
[2025-04-26 01:03:35,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5707.04 | bwd_inner: 5651.81 | bwd_allreduce: 55.19 | step: 18.70

 17%|█▋        | 7078/41250 [17:06:00<82:34:02,  8.70s/it]
                                                          
{'loss': 0.1775, 'grad_norm': 1.8409175872802734, 'learning_rate': 3.793403868681547e-05, 'epoch': 1.72}

 17%|█▋        | 7078/41250 [17:06:00<82:34:02,  8.70s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x29f396c0] moov atom not found
[01:03:35] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4...
[2025-04-26 01:03:44,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 01:03:44,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.67 | bwd_microstep: 5795.98 | bwd_inner_microstep: 5665.45 | bwd_allreduce_microstep: 130.48 | step_microstep: 18.83
[2025-04-26 01:03:44,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.67 | bwd: 5795.99 | bwd_inner: 5665.45 | bwd_allreduce: 130.50 | step: 18.83

 17%|█▋        | 7079/41250 [17:06:09<82:36:30,  8.70s/it]
                                                          
{'loss': 0.0755, 'grad_norm': 1.182334303855896, 'learning_rate': 3.7933343550809855e-05, 'epoch': 1.72}

 17%|█▋        | 7079/41250 [17:06:09<82:36:30,  8.70s/it][2025-04-26 01:03:52,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:03:52,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.63 | bwd_microstep: 5725.91 | bwd_inner_microstep: 5713.18 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.79
[2025-04-26 01:03:52,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.63 | bwd: 5725.93 | bwd_inner: 5713.18 | bwd_allreduce: 12.70 | step: 18.79

 17%|█▋        | 7080/41250 [17:06:18<82:29:52,  8.69s/it]
                                                          
{'loss': 0.2987, 'grad_norm': 2.960986375808716, 'learning_rate': 3.7932648304248707e-05, 'epoch': 1.72}

 17%|█▋        | 7080/41250 [17:06:18<82:29:52,  8.69s/it][2025-04-26 01:04:01,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:04:01,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.08 | bwd_microstep: 5800.40 | bwd_inner_microstep: 5787.91 | bwd_allreduce_microstep: 12.45 | step_microstep: 18.90
[2025-04-26 01:04:01,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.08 | bwd: 5800.42 | bwd_inner: 5787.91 | bwd_allreduce: 12.46 | step: 18.90

 17%|█▋        | 7081/41250 [17:06:26<82:44:07,  8.72s/it]
                                                          
{'loss': 0.0888, 'grad_norm': 1.0106022357940674, 'learning_rate': 3.793195294713631e-05, 'epoch': 1.72}

 17%|█▋        | 7081/41250 [17:06:26<82:44:07,  8.72s/it][2025-04-26 01:04:10,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:04:10,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.40 | bwd_microstep: 5787.19 | bwd_inner_microstep: 5653.54 | bwd_allreduce_microstep: 133.60 | step_microstep: 18.63
[2025-04-26 01:04:10,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.40 | bwd: 5787.20 | bwd_inner: 5653.54 | bwd_allreduce: 133.61 | step: 18.63

 17%|█▋        | 7082/41250 [17:06:35<82:39:46,  8.71s/it]
                                                          
{'loss': 0.1771, 'grad_norm': 1.4071922302246094, 'learning_rate': 3.793125747947694e-05, 'epoch': 1.72}

 17%|█▋        | 7082/41250 [17:06:35<82:39:46,  8.71s/it][2025-04-26 01:04:18,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 01:04:18,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.04 | bwd_microstep: 5786.52 | bwd_inner_microstep: 5689.08 | bwd_allreduce_microstep: 97.40 | step_microstep: 19.44
[2025-04-26 01:04:18,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.04 | bwd: 5786.54 | bwd_inner: 5689.08 | bwd_allreduce: 97.41 | step: 19.44

 17%|█▋        | 7083/41250 [17:06:44<82:41:17,  8.71s/it]
                                                          
{'loss': 0.1211, 'grad_norm': 2.4090213775634766, 'learning_rate': 3.79305619012749e-05, 'epoch': 1.72}

 17%|█▋        | 7083/41250 [17:06:44<82:41:17,  8.71s/it][2025-04-26 01:04:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:04:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.32 | bwd_microstep: 5712.52 | bwd_inner_microstep: 5699.76 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.87
[2025-04-26 01:04:27,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.32 | bwd: 5712.54 | bwd_inner: 5699.76 | bwd_allreduce: 12.73 | step: 18.87

 17%|█▋        | 7084/41250 [17:06:52<82:30:34,  8.69s/it]
                                                          
{'loss': 0.2347, 'grad_norm': 1.359980583190918, 'learning_rate': 3.792986621253448e-05, 'epoch': 1.72}

 17%|█▋        | 7084/41250 [17:06:52<82:30:34,  8.69s/it][2025-04-26 01:04:36,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 01:04:36,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.80 | bwd_microstep: 5765.01 | bwd_inner_microstep: 5644.17 | bwd_allreduce_microstep: 120.79 | step_microstep: 19.22
[2025-04-26 01:04:36,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.80 | bwd: 5765.02 | bwd_inner: 5644.17 | bwd_allreduce: 120.81 | step: 19.22

 17%|█▋        | 7085/41250 [17:07:01<82:28:04,  8.69s/it]
                                                          
{'loss': 0.0515, 'grad_norm': 0.49186521768569946, 'learning_rate': 3.7929170413259956e-05, 'epoch': 1.72}

 17%|█▋        | 7085/41250 [17:07:01<82:28:04,  8.69s/it][2025-04-26 01:04:44,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:04:44,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.63 | bwd_microstep: 5782.09 | bwd_inner_microstep: 5645.41 | bwd_allreduce_microstep: 136.63 | step_microstep: 18.86
[2025-04-26 01:04:44,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.63 | bwd: 5782.10 | bwd_inner: 5645.41 | bwd_allreduce: 136.65 | step: 18.86

 17%|█▋        | 7086/41250 [17:07:10<82:28:08,  8.69s/it]
                                                          
{'loss': 0.1296, 'grad_norm': 1.2118732929229736, 'learning_rate': 3.792847450345562e-05, 'epoch': 1.72}

 17%|█▋        | 7086/41250 [17:07:10<82:28:08,  8.69s/it][2025-04-26 01:04:53,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:04:53,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2903.01 | bwd_microstep: 5791.20 | bwd_inner_microstep: 5778.19 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.98
[2025-04-26 01:04:53,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2903.01 | bwd: 5791.21 | bwd_inner: 5778.19 | bwd_allreduce: 12.97 | step: 18.98

 17%|█▋        | 7087/41250 [17:07:19<82:43:01,  8.72s/it]
                                                          
{'loss': 0.1274, 'grad_norm': 1.1806210279464722, 'learning_rate': 3.792777848312577e-05, 'epoch': 1.72}

 17%|█▋        | 7087/41250 [17:07:19<82:43:01,  8.72s/it][2025-04-26 01:05:02,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.18
[2025-04-26 01:05:02,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5700.51 | bwd_inner_microstep: 5658.04 | bwd_allreduce_microstep: 42.42 | step_microstep: 19.37
[2025-04-26 01:05:02,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5700.53 | bwd_inner: 5658.04 | bwd_allreduce: 42.44 | step: 19.37

 17%|█▋        | 7088/41250 [17:07:27<82:25:20,  8.69s/it]
                                                          
{'loss': 0.0683, 'grad_norm': 1.245516300201416, 'learning_rate': 3.792708235227469e-05, 'epoch': 1.72}

 17%|█▋        | 7088/41250 [17:07:27<82:25:20,  8.69s/it][2025-04-26 01:05:10,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.17 | optimizer_step: 0.92
[2025-04-26 01:05:10,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.47 | bwd_microstep: 5704.59 | bwd_inner_microstep: 5662.65 | bwd_allreduce_microstep: 41.89 | step_microstep: 19.12
[2025-04-26 01:05:10,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.47 | bwd: 5704.60 | bwd_inner: 5662.65 | bwd_allreduce: 41.91 | step: 19.13

 17%|█▋        | 7089/41250 [17:07:36<82:13:38,  8.67s/it]
                                                          
{'loss': 0.1245, 'grad_norm': 1.6877096891403198, 'learning_rate': 3.792638611090667e-05, 'epoch': 1.72}

 17%|█▋        | 7089/41250 [17:07:36<82:13:38,  8.67s/it][2025-04-26 01:05:19,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:05:19,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.54 | bwd_microstep: 5710.30 | bwd_inner_microstep: 5697.45 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.86
[2025-04-26 01:05:19,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.54 | bwd: 5710.31 | bwd_inner: 5697.45 | bwd_allreduce: 12.82 | step: 18.86

 17%|█▋        | 7090/41250 [17:07:44<82:09:12,  8.66s/it]
                                                          
{'loss': 0.0933, 'grad_norm': 1.655260443687439, 'learning_rate': 3.792568975902601e-05, 'epoch': 1.72}

 17%|█▋        | 7090/41250 [17:07:44<82:09:12,  8.66s/it][2025-04-26 01:05:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 01:05:28,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.82 | bwd_microstep: 5701.67 | bwd_inner_microstep: 5654.58 | bwd_allreduce_microstep: 47.05 | step_microstep: 18.62
[2025-04-26 01:05:28,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.83 | bwd: 5701.69 | bwd_inner: 5654.58 | bwd_allreduce: 47.07 | step: 18.62

 17%|█▋        | 7091/41250 [17:07:53<82:03:00,  8.65s/it]
                                                          
{'loss': 0.0942, 'grad_norm': 1.0173901319503784, 'learning_rate': 3.7924993296637e-05, 'epoch': 1.72}

 17%|█▋        | 7091/41250 [17:07:53<82:03:00,  8.65s/it][2025-04-26 01:05:36,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 01:05:36,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.31 | bwd_microstep: 5764.60 | bwd_inner_microstep: 5686.56 | bwd_allreduce_microstep: 77.98 | step_microstep: 19.01
[2025-04-26 01:05:36,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.31 | bwd: 5764.62 | bwd_inner: 5686.56 | bwd_allreduce: 78.00 | step: 19.01

 17%|█▋        | 7092/41250 [17:08:02<82:11:02,  8.66s/it]
                                                          
{'loss': 0.0355, 'grad_norm': 0.9444786906242371, 'learning_rate': 3.792429672374392e-05, 'epoch': 1.72}

 17%|█▋        | 7092/41250 [17:08:02<82:11:02,  8.66s/it][2025-04-26 01:05:45,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 01:05:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.56 | bwd_microstep: 5873.60 | bwd_inner_microstep: 5693.68 | bwd_allreduce_microstep: 179.88 | step_microstep: 19.15
[2025-04-26 01:05:45,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5873.61 | bwd_inner: 5693.68 | bwd_allreduce: 179.90 | step: 19.15

 17%|█▋        | 7093/41250 [17:08:11<82:35:24,  8.70s/it]
                                                          
{'loss': 0.1096, 'grad_norm': 1.1785496473312378, 'learning_rate': 3.792360004035109e-05, 'epoch': 1.72}

 17%|█▋        | 7093/41250 [17:08:11<82:35:24,  8.70s/it][2025-04-26 01:05:54,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 0.93
[2025-04-26 01:05:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.17 | bwd_microstep: 5685.99 | bwd_inner_microstep: 5649.01 | bwd_allreduce_microstep: 36.93 | step_microstep: 19.23
[2025-04-26 01:05:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.17 | bwd: 5686.01 | bwd_inner: 5649.01 | bwd_allreduce: 36.95 | step: 19.24

 17%|█▋        | 7094/41250 [17:08:19<82:18:13,  8.67s/it]
                                                          
{'loss': 0.0953, 'grad_norm': 1.8708349466323853, 'learning_rate': 3.792290324646279e-05, 'epoch': 1.72}

 17%|█▋        | 7094/41250 [17:08:19<82:18:13,  8.67s/it][2025-04-26 01:06:02,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:06:02,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.15 | bwd_microstep: 5708.44 | bwd_inner_microstep: 5642.34 | bwd_allreduce_microstep: 66.06 | step_microstep: 18.92
[2025-04-26 01:06:02,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.15 | bwd: 5708.45 | bwd_inner: 5642.34 | bwd_allreduce: 66.07 | step: 18.92

 17%|█▋        | 7095/41250 [17:08:28<82:07:59,  8.66s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 1.45547354221344, 'learning_rate': 3.792220634208331e-05, 'epoch': 1.72}

 17%|█▋        | 7095/41250 [17:08:28<82:07:59,  8.66s/it][2025-04-26 01:06:11,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:06:11,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.14 | bwd_microstep: 5782.80 | bwd_inner_microstep: 5636.18 | bwd_allreduce_microstep: 146.57 | step_microstep: 18.88
[2025-04-26 01:06:11,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.14 | bwd: 5782.82 | bwd_inner: 5636.18 | bwd_allreduce: 146.59 | step: 18.88

 17%|█▋        | 7096/41250 [17:08:36<82:11:47,  8.66s/it]
                                                          
{'loss': 0.1764, 'grad_norm': 3.7820804119110107, 'learning_rate': 3.792150932721695e-05, 'epoch': 1.72}

 17%|█▋        | 7096/41250 [17:08:36<82:11:47,  8.66s/it][2025-04-26 01:06:20,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.30 | optimizer_step: 1.04
[2025-04-26 01:06:20,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.25 | bwd_microstep: 5749.91 | bwd_inner_microstep: 5695.19 | bwd_allreduce_microstep: 54.65 | step_microstep: 20.15
[2025-04-26 01:06:20,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.25 | bwd: 5749.92 | bwd_inner: 5695.19 | bwd_allreduce: 54.68 | step: 20.15

 17%|█▋        | 7097/41250 [17:08:45<82:14:12,  8.67s/it]
                                                          
{'loss': 0.113, 'grad_norm': 2.4533486366271973, 'learning_rate': 3.792081220186802e-05, 'epoch': 1.72}

 17%|█▋        | 7097/41250 [17:08:45<82:14:12,  8.67s/it][2025-04-26 01:06:28,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 01:06:28,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.36 | bwd_microstep: 5705.58 | bwd_inner_microstep: 5692.89 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.01
[2025-04-26 01:06:28,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.36 | bwd: 5705.59 | bwd_inner: 5692.89 | bwd_allreduce: 12.67 | step: 19.01

 17%|█▋        | 7098/41250 [17:08:54<82:08:37,  8.66s/it]
                                                          
{'loss': 0.1752, 'grad_norm': 1.5292415618896484, 'learning_rate': 3.792011496604081e-05, 'epoch': 1.72}

 17%|█▋        | 7098/41250 [17:08:54<82:08:37,  8.66s/it][2025-04-26 01:06:37,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:06:37,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5877.61 | bwd_inner_microstep: 5650.55 | bwd_allreduce_microstep: 227.02 | step_microstep: 18.42
[2025-04-26 01:06:37,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5877.62 | bwd_inner: 5650.55 | bwd_allreduce: 227.03 | step: 18.42

 17%|█▋        | 7099/41250 [17:09:03<82:31:20,  8.70s/it]
                                                          
{'loss': 0.2143, 'grad_norm': 2.2278940677642822, 'learning_rate': 3.7919417619739606e-05, 'epoch': 1.72}

 17%|█▋        | 7099/41250 [17:09:03<82:31:20,  8.70s/it][2025-04-26 01:06:46,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:06:46,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.00 | bwd_microstep: 5726.28 | bwd_inner_microstep: 5693.31 | bwd_allreduce_microstep: 32.92 | step_microstep: 18.48
[2025-04-26 01:06:46,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.00 | bwd: 5726.29 | bwd_inner: 5693.31 | bwd_allreduce: 32.94 | step: 18.48

 17%|█▋        | 7100/41250 [17:09:11<82:26:17,  8.69s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 1.2670996189117432, 'learning_rate': 3.7918720162968714e-05, 'epoch': 1.72}

 17%|█▋        | 7100/41250 [17:09:11<82:26:17,  8.69s/it][2025-04-26 01:06:55,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:06:55,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.05 | bwd_microstep: 5711.11 | bwd_inner_microstep: 5667.68 | bwd_allreduce_microstep: 43.38 | step_microstep: 18.81
[2025-04-26 01:06:55,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.05 | bwd: 5711.13 | bwd_inner: 5667.68 | bwd_allreduce: 43.40 | step: 18.81

 17%|█▋        | 7101/41250 [17:09:20<82:17:51,  8.68s/it]
                                                          
{'loss': 0.182, 'grad_norm': 2.755117893218994, 'learning_rate': 3.7918022595732446e-05, 'epoch': 1.72}

 17%|█▋        | 7101/41250 [17:09:20<82:17:51,  8.68s/it][2025-04-26 01:07:03,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 01:07:03,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5781.34 | bwd_inner_microstep: 5627.83 | bwd_allreduce_microstep: 153.45 | step_microstep: 19.23
[2025-04-26 01:07:03,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5781.35 | bwd_inner: 5627.83 | bwd_allreduce: 153.48 | step: 19.23

 17%|█▋        | 7102/41250 [17:09:29<82:20:01,  8.68s/it]
                                                          
{'loss': 0.0384, 'grad_norm': 0.563278079032898, 'learning_rate': 3.791732491803509e-05, 'epoch': 1.72}

 17%|█▋        | 7102/41250 [17:09:29<82:20:01,  8.68s/it][2025-04-26 01:07:12,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 01:07:12,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.72 | bwd_microstep: 5703.54 | bwd_inner_microstep: 5619.93 | bwd_allreduce_microstep: 83.57 | step_microstep: 18.76
[2025-04-26 01:07:12,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.72 | bwd: 5703.56 | bwd_inner: 5619.93 | bwd_allreduce: 83.58 | step: 18.76

 17%|█▋        | 7103/41250 [17:09:37<82:07:27,  8.66s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 2.118283271789551, 'learning_rate': 3.7916627129880945e-05, 'epoch': 1.72}

 17%|█▋        | 7103/41250 [17:09:37<82:07:27,  8.66s/it][2025-04-26 01:07:20,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 01:07:20,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.73 | bwd_microstep: 5692.47 | bwd_inner_microstep: 5679.54 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.17
[2025-04-26 01:07:20,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.73 | bwd: 5692.48 | bwd_inner: 5679.54 | bwd_allreduce: 12.90 | step: 19.17

 17%|█▋        | 7104/41250 [17:09:46<82:01:08,  8.65s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 1.0221014022827148, 'learning_rate': 3.791592923127433e-05, 'epoch': 1.72}

 17%|█▋        | 7104/41250 [17:09:46<82:01:08,  8.65s/it][2025-04-26 01:07:29,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 01:07:29,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.22 | bwd_microstep: 5719.36 | bwd_inner_microstep: 5706.02 | bwd_allreduce_microstep: 13.29 | step_microstep: 18.68
[2025-04-26 01:07:29,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.22 | bwd: 5719.38 | bwd_inner: 5706.02 | bwd_allreduce: 13.31 | step: 18.68

 17%|█▋        | 7105/41250 [17:09:54<82:00:29,  8.65s/it]
                                                          
{'loss': 0.2144, 'grad_norm': 3.1536166667938232, 'learning_rate': 3.7915231222219525e-05, 'epoch': 1.72}

 17%|█▋        | 7105/41250 [17:09:54<82:00:29,  8.65s/it][2025-04-26 01:07:38,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 01:07:38,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.41 | bwd_microstep: 5754.32 | bwd_inner_microstep: 5642.11 | bwd_allreduce_microstep: 112.16 | step_microstep: 18.68
[2025-04-26 01:07:38,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.41 | bwd: 5754.34 | bwd_inner: 5642.11 | bwd_allreduce: 112.18 | step: 18.69

 17%|█▋        | 7106/41250 [17:10:03<82:03:15,  8.65s/it]
                                                          
{'loss': 0.1195, 'grad_norm': 4.252877235412598, 'learning_rate': 3.7914533102720845e-05, 'epoch': 1.72}

 17%|█▋        | 7106/41250 [17:10:03<82:03:15,  8.65s/it][2025-04-26 01:07:46,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 01:07:46,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.56 | bwd_microstep: 5702.59 | bwd_inner_microstep: 5689.82 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.57
[2025-04-26 01:07:46,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.56 | bwd: 5702.60 | bwd_inner: 5689.82 | bwd_allreduce: 12.74 | step: 18.58

 17%|█▋        | 7107/41250 [17:10:12<81:59:18,  8.64s/it]
                                                          
{'loss': 0.1944, 'grad_norm': 2.699481248855591, 'learning_rate': 3.7913834872782595e-05, 'epoch': 1.72}

 17%|█▋        | 7107/41250 [17:10:12<81:59:18,  8.64s/it][2025-04-26 01:07:55,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:07:55,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.80 | bwd_microstep: 5697.61 | bwd_inner_microstep: 5646.41 | bwd_allreduce_microstep: 51.16 | step_microstep: 18.90
[2025-04-26 01:07:55,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.80 | bwd: 5697.63 | bwd_inner: 5646.41 | bwd_allreduce: 51.17 | step: 18.90

 17%|█▋        | 7108/41250 [17:10:20<81:52:11,  8.63s/it]
                                                          
{'loss': 0.1746, 'grad_norm': 1.848864197731018, 'learning_rate': 3.791313653240907e-05, 'epoch': 1.72}

 17%|█▋        | 7108/41250 [17:10:20<81:52:11,  8.63s/it][2025-04-26 01:08:04,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 01:08:04,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.39 | bwd_microstep: 5694.43 | bwd_inner_microstep: 5649.55 | bwd_allreduce_microstep: 44.83 | step_microstep: 19.35
[2025-04-26 01:08:04,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.39 | bwd: 5694.45 | bwd_inner: 5649.55 | bwd_allreduce: 44.85 | step: 19.36

 17%|█▋        | 7109/41250 [17:10:29<81:47:06,  8.62s/it]
                                                          
{'loss': 0.0699, 'grad_norm': 1.5347546339035034, 'learning_rate': 3.791243808160459e-05, 'epoch': 1.72}

 17%|█▋        | 7109/41250 [17:10:29<81:47:06,  8.62s/it][2025-04-26 01:08:12,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 01:08:12,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.52 | bwd_microstep: 5746.17 | bwd_inner_microstep: 5686.51 | bwd_allreduce_microstep: 59.62 | step_microstep: 18.66
[2025-04-26 01:08:12,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.52 | bwd: 5746.19 | bwd_inner: 5686.51 | bwd_allreduce: 59.64 | step: 18.66

 17%|█▋        | 7110/41250 [17:10:38<81:55:02,  8.64s/it]
                                                          
{'loss': 0.0729, 'grad_norm': 3.826460838317871, 'learning_rate': 3.791173952037345e-05, 'epoch': 1.72}

 17%|█▋        | 7110/41250 [17:10:38<81:55:02,  8.64s/it][2025-04-26 01:08:21,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:08:21,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.60 | bwd_microstep: 5693.42 | bwd_inner_microstep: 5646.68 | bwd_allreduce_microstep: 46.69 | step_microstep: 18.61
[2025-04-26 01:08:21,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.60 | bwd: 5693.44 | bwd_inner: 5646.69 | bwd_allreduce: 46.71 | step: 18.61

 17%|█▋        | 7111/41250 [17:10:46<81:48:50,  8.63s/it]
                                                          
{'loss': 0.0867, 'grad_norm': 0.9792299866676331, 'learning_rate': 3.791104084871995e-05, 'epoch': 1.72}

 17%|█▋        | 7111/41250 [17:10:46<81:48:50,  8.63s/it][2025-04-26 01:08:30,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:08:30,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.87 | bwd_microstep: 5728.11 | bwd_inner_microstep: 5694.19 | bwd_allreduce_microstep: 33.88 | step_microstep: 18.93
[2025-04-26 01:08:30,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.87 | bwd: 5728.12 | bwd_inner: 5694.19 | bwd_allreduce: 33.89 | step: 18.93

 17%|█▋        | 7112/41250 [17:10:55<81:55:40,  8.64s/it]
                                                          
{'loss': 0.1259, 'grad_norm': 2.506441354751587, 'learning_rate': 3.791034206664842e-05, 'epoch': 1.72}

 17%|█▋        | 7112/41250 [17:10:55<81:55:40,  8.64s/it][2025-04-26 01:08:38,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:08:38,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.03 | bwd_microstep: 5772.52 | bwd_inner_microstep: 5689.84 | bwd_allreduce_microstep: 82.63 | step_microstep: 18.81
[2025-04-26 01:08:38,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.03 | bwd: 5772.53 | bwd_inner: 5689.84 | bwd_allreduce: 82.65 | step: 18.81

 17%|█▋        | 7113/41250 [17:11:04<82:06:26,  8.66s/it]
                                                          
{'loss': 0.0373, 'grad_norm': 1.016188144683838, 'learning_rate': 3.7909643174163145e-05, 'epoch': 1.72}

 17%|█▋        | 7113/41250 [17:11:04<82:06:26,  8.66s/it][2025-04-26 01:08:47,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:08:47,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3091.00 | bwd_microstep: 5709.62 | bwd_inner_microstep: 5696.76 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.78
[2025-04-26 01:08:47,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3091.00 | bwd: 5709.64 | bwd_inner: 5696.76 | bwd_allreduce: 12.84 | step: 18.79

 17%|█▋        | 7114/41250 [17:11:12<82:44:12,  8.73s/it]
                                                          
{'loss': 0.0621, 'grad_norm': 1.2671442031860352, 'learning_rate': 3.790894417126844e-05, 'epoch': 1.72}

 17%|█▋        | 7114/41250 [17:11:12<82:44:12,  8.73s/it][2025-04-26 01:08:56,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 01:08:56,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.35 | bwd_microstep: 5697.31 | bwd_inner_microstep: 5684.58 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.64
[2025-04-26 01:08:56,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.35 | bwd: 5697.32 | bwd_inner: 5684.58 | bwd_allreduce: 12.70 | step: 18.65

 17%|█▋        | 7115/41250 [17:11:21<82:26:32,  8.69s/it]
                                                          
{'loss': 0.0702, 'grad_norm': 5.02650785446167, 'learning_rate': 3.7908245057968625e-05, 'epoch': 1.72}

 17%|█▋        | 7115/41250 [17:11:21<82:26:32,  8.69s/it][2025-04-26 01:09:04,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 01:09:04,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5800.78 | bwd_inner_microstep: 5662.66 | bwd_allreduce_microstep: 138.07 | step_microstep: 18.76
[2025-04-26 01:09:04,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5800.79 | bwd_inner: 5662.66 | bwd_allreduce: 138.09 | step: 18.77

 17%|█▋        | 7116/41250 [17:11:30<82:29:20,  8.70s/it]
                                                          
{'loss': 0.023, 'grad_norm': 0.2958524525165558, 'learning_rate': 3.7907545834268004e-05, 'epoch': 1.73}

 17%|█▋        | 7116/41250 [17:11:30<82:29:20,  8.70s/it][2025-04-26 01:09:13,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:09:13,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.50 | bwd_microstep: 5701.69 | bwd_inner_microstep: 5648.97 | bwd_allreduce_microstep: 52.67 | step_microstep: 18.76
[2025-04-26 01:09:13,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.50 | bwd: 5701.70 | bwd_inner: 5648.97 | bwd_allreduce: 52.69 | step: 18.76

 17%|█▋        | 7117/41250 [17:11:38<82:15:00,  8.67s/it]
                                                          
{'loss': 0.0749, 'grad_norm': 1.645468831062317, 'learning_rate': 3.7906846500170875e-05, 'epoch': 1.73}

 17%|█▋        | 7117/41250 [17:11:38<82:15:00,  8.67s/it][2025-04-26 01:09:22,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.06 | optimizer_step: 0.96
[2025-04-26 01:09:22,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.76 | bwd_microstep: 5756.83 | bwd_inner_microstep: 5693.73 | bwd_allreduce_microstep: 63.05 | step_microstep: 19.16
[2025-04-26 01:09:22,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.76 | bwd: 5756.84 | bwd_inner: 5693.73 | bwd_allreduce: 63.07 | step: 19.16

 17%|█▋        | 7118/41250 [17:11:47<82:16:54,  8.68s/it]
                                                          
{'loss': 0.1177, 'grad_norm': 1.068730354309082, 'learning_rate': 3.790614705568156e-05, 'epoch': 1.73}

 17%|█▋        | 7118/41250 [17:11:47<82:16:54,  8.68s/it][2025-04-26 01:09:30,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:09:30,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.57 | bwd_microstep: 5755.40 | bwd_inner_microstep: 5679.30 | bwd_allreduce_microstep: 76.05 | step_microstep: 18.64
[2025-04-26 01:09:30,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.57 | bwd: 5755.41 | bwd_inner: 5679.30 | bwd_allreduce: 76.08 | step: 18.64

 17%|█▋        | 7119/41250 [17:11:56<82:17:58,  8.68s/it]
                                                          
{'loss': 0.2518, 'grad_norm': 2.138831615447998, 'learning_rate': 3.7905447500804374e-05, 'epoch': 1.73}

 17%|█▋        | 7119/41250 [17:11:56<82:17:58,  8.68s/it][2025-04-26 01:09:39,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:09:39,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.81 | bwd_microstep: 5713.40 | bwd_inner_microstep: 5700.52 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.94
[2025-04-26 01:09:39,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.81 | bwd: 5713.42 | bwd_inner: 5700.52 | bwd_allreduce: 12.85 | step: 18.94

 17%|█▋        | 7120/41250 [17:12:04<82:13:52,  8.67s/it]
                                                          
{'loss': 0.0929, 'grad_norm': 1.3612003326416016, 'learning_rate': 3.790474783554363e-05, 'epoch': 1.73}

 17%|█▋        | 7120/41250 [17:12:04<82:13:52,  8.67s/it][2025-04-26 01:09:48,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 1.06
[2025-04-26 01:09:48,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.66 | bwd_microstep: 5886.28 | bwd_inner_microstep: 5698.46 | bwd_allreduce_microstep: 187.78 | step_microstep: 19.56
[2025-04-26 01:09:48,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.66 | bwd: 5886.29 | bwd_inner: 5698.46 | bwd_allreduce: 187.80 | step: 19.56

 17%|█▋        | 7121/41250 [17:12:13<82:39:06,  8.72s/it]
                                                          
{'loss': 0.3394, 'grad_norm': 2.2614827156066895, 'learning_rate': 3.790404805990363e-05, 'epoch': 1.73}

 17%|█▋        | 7121/41250 [17:12:13<82:39:06,  8.72s/it][2025-04-26 01:09:57,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.30 | optimizer_step: 1.03
[2025-04-26 01:09:57,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.70 | bwd_microstep: 5785.88 | bwd_inner_microstep: 5656.66 | bwd_allreduce_microstep: 129.17 | step_microstep: 19.88
[2025-04-26 01:09:57,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.70 | bwd: 5785.90 | bwd_inner: 5656.66 | bwd_allreduce: 129.20 | step: 19.89

 17%|█▋        | 7122/41250 [17:12:22<82:36:19,  8.71s/it]
                                                          
{'loss': 0.2331, 'grad_norm': 1.2955410480499268, 'learning_rate': 3.7903348173888706e-05, 'epoch': 1.73}

 17%|█▋        | 7122/41250 [17:12:22<82:36:19,  8.71s/it][2025-04-26 01:10:05,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 01:10:05,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.52 | bwd_microstep: 5761.26 | bwd_inner_microstep: 5714.23 | bwd_allreduce_microstep: 46.98 | step_microstep: 19.23
[2025-04-26 01:10:05,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.52 | bwd: 5761.27 | bwd_inner: 5714.23 | bwd_allreduce: 47.00 | step: 19.23

 17%|█▋        | 7123/41250 [17:12:31<82:33:39,  8.71s/it]
                                                          
{'loss': 0.4487, 'grad_norm': 3.1624374389648438, 'learning_rate': 3.790264817750315e-05, 'epoch': 1.73}

 17%|█▋        | 7123/41250 [17:12:31<82:33:39,  8.71s/it][2025-04-26 01:10:14,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 01:10:14,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 6037.08 | bwd_inner_microstep: 5644.47 | bwd_allreduce_microstep: 392.56 | step_microstep: 18.79
[2025-04-26 01:10:14,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 6037.09 | bwd_inner: 5644.47 | bwd_allreduce: 392.58 | step: 18.79

 17%|█▋        | 7124/41250 [17:12:40<83:17:17,  8.79s/it]
                                                          
{'loss': 0.0341, 'grad_norm': 0.8708192110061646, 'learning_rate': 3.7901948070751295e-05, 'epoch': 1.73}

 17%|█▋        | 7124/41250 [17:12:40<83:17:17,  8.79s/it][2025-04-26 01:10:23,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 01:10:23,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.46 | bwd_microstep: 5701.70 | bwd_inner_microstep: 5654.03 | bwd_allreduce_microstep: 47.63 | step_microstep: 18.66
[2025-04-26 01:10:23,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.46 | bwd: 5701.72 | bwd_inner: 5654.03 | bwd_allreduce: 47.64 | step: 18.66

 17%|█▋        | 7125/41250 [17:12:48<82:48:41,  8.74s/it]
                                                          
{'loss': 0.2448, 'grad_norm': 3.395456314086914, 'learning_rate': 3.7901247853637444e-05, 'epoch': 1.73}

 17%|█▋        | 7125/41250 [17:12:48<82:48:41,  8.74s/it][2025-04-26 01:10:32,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-26 01:10:32,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.59 | bwd_microstep: 5767.62 | bwd_inner_microstep: 5693.34 | bwd_allreduce_microstep: 74.23 | step_microstep: 19.10
[2025-04-26 01:10:32,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.59 | bwd: 5767.63 | bwd_inner: 5693.34 | bwd_allreduce: 74.25 | step: 19.10

 17%|█▋        | 7126/41250 [17:12:57<82:43:59,  8.73s/it]
                                                          
{'loss': 0.1488, 'grad_norm': 1.7952522039413452, 'learning_rate': 3.790054752616593e-05, 'epoch': 1.73}

 17%|█▋        | 7126/41250 [17:12:57<82:43:59,  8.73s/it][2025-04-26 01:10:40,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 01:10:40,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.93 | bwd_microstep: 5704.62 | bwd_inner_microstep: 5691.86 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.07
[2025-04-26 01:10:40,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5704.63 | bwd_inner: 5691.86 | bwd_allreduce: 12.73 | step: 19.07

 17%|█▋        | 7127/41250 [17:13:06<82:27:41,  8.70s/it]
                                                          
{'loss': 0.1555, 'grad_norm': 2.5031349658966064, 'learning_rate': 3.789984708834106e-05, 'epoch': 1.73}

 17%|█▋        | 7127/41250 [17:13:06<82:27:41,  8.70s/it][2025-04-26 01:10:49,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 01:10:49,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.35 | bwd_microstep: 5703.06 | bwd_inner_microstep: 5689.88 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.98
[2025-04-26 01:10:49,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.35 | bwd: 5703.07 | bwd_inner: 5689.88 | bwd_allreduce: 13.15 | step: 18.98

 17%|█▋        | 7128/41250 [17:13:14<82:16:31,  8.68s/it]
                                                          
{'loss': 0.4257, 'grad_norm': 2.509925127029419, 'learning_rate': 3.789914654016715e-05, 'epoch': 1.73}

 17%|█▋        | 7128/41250 [17:13:14<82:16:31,  8.68s/it][2025-04-26 01:10:57,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 01:10:57,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.92 | bwd_microstep: 5699.84 | bwd_inner_microstep: 5660.98 | bwd_allreduce_microstep: 38.82 | step_microstep: 18.86
[2025-04-26 01:10:57,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.92 | bwd: 5699.86 | bwd_inner: 5660.98 | bwd_allreduce: 38.83 | step: 18.86

 17%|█▋        | 7129/41250 [17:13:23<82:05:27,  8.66s/it]
                                                          
{'loss': 0.1301, 'grad_norm': 3.648496150970459, 'learning_rate': 3.789844588164853e-05, 'epoch': 1.73}

 17%|█▋        | 7129/41250 [17:13:23<82:05:27,  8.66s/it][2025-04-26 01:11:06,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 01:11:06,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.48 | bwd_microstep: 5874.03 | bwd_inner_microstep: 5648.97 | bwd_allreduce_microstep: 225.00 | step_microstep: 18.85
[2025-04-26 01:11:06,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.48 | bwd: 5874.04 | bwd_inner: 5648.97 | bwd_allreduce: 225.03 | step: 18.85

 17%|█▋        | 7130/41250 [17:13:32<82:26:42,  8.70s/it]
                                                          
{'loss': 0.2623, 'grad_norm': 3.8762123584747314, 'learning_rate': 3.78977451127895e-05, 'epoch': 1.73}

 17%|█▋        | 7130/41250 [17:13:32<82:26:42,  8.70s/it][2025-04-26 01:11:15,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 01:11:15,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 5734.79 | bwd_inner_microstep: 5707.68 | bwd_allreduce_microstep: 27.06 | step_microstep: 19.37
[2025-04-26 01:11:15,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5734.80 | bwd_inner: 5707.68 | bwd_allreduce: 27.08 | step: 19.37

 17%|█▋        | 7131/41250 [17:13:40<82:21:40,  8.69s/it]
                                                          
{'loss': 0.1622, 'grad_norm': 2.0942397117614746, 'learning_rate': 3.78970442335944e-05, 'epoch': 1.73}

 17%|█▋        | 7131/41250 [17:13:40<82:21:40,  8.69s/it][2025-04-26 01:11:24,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 01:11:24,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.27 | bwd_microstep: 6040.65 | bwd_inner_microstep: 5653.06 | bwd_allreduce_microstep: 387.54 | step_microstep: 19.23
[2025-04-26 01:11:24,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.27 | bwd: 6040.66 | bwd_inner: 5653.06 | bwd_allreduce: 387.56 | step: 19.23

 17%|█▋        | 7132/41250 [17:13:49<83:05:56,  8.77s/it]
                                                          
{'loss': 0.0334, 'grad_norm': 0.49295270442962646, 'learning_rate': 3.789634324406754e-05, 'epoch': 1.73}

 17%|█▋        | 7132/41250 [17:13:49<83:05:56,  8.77s/it][2025-04-26 01:11:33,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 01:11:33,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.22 | bwd_microstep: 5717.91 | bwd_inner_microstep: 5705.17 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.75
[2025-04-26 01:11:33,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.22 | bwd: 5717.92 | bwd_inner: 5705.17 | bwd_allreduce: 12.71 | step: 18.75

 17%|█▋        | 7133/41250 [17:13:58<82:46:19,  8.73s/it]
                                                          
{'loss': 0.2032, 'grad_norm': 1.869437336921692, 'learning_rate': 3.789564214421324e-05, 'epoch': 1.73}

 17%|█▋        | 7133/41250 [17:13:58<82:46:19,  8.73s/it][2025-04-26 01:11:41,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:11:41,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.10 | bwd_microstep: 5697.40 | bwd_inner_microstep: 5651.13 | bwd_allreduce_microstep: 46.22 | step_microstep: 18.93
[2025-04-26 01:11:41,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.10 | bwd: 5697.42 | bwd_inner: 5651.13 | bwd_allreduce: 46.24 | step: 18.93

 17%|█▋        | 7134/41250 [17:14:06<82:25:40,  8.70s/it]
                                                          
{'loss': 0.1818, 'grad_norm': 4.3543171882629395, 'learning_rate': 3.789494093403583e-05, 'epoch': 1.73}

 17%|█▋        | 7134/41250 [17:14:06<82:25:40,  8.70s/it][2025-04-26 01:11:50,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.14 | optimizer_step: 1.11
[2025-04-26 01:11:50,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.73 | bwd_microstep: 5745.02 | bwd_inner_microstep: 5678.95 | bwd_allreduce_microstep: 66.02 | step_microstep: 19.21
[2025-04-26 01:11:50,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.73 | bwd: 5745.03 | bwd_inner: 5678.95 | bwd_allreduce: 66.04 | step: 19.21

 17%|█▋        | 7135/41250 [17:14:15<82:22:17,  8.69s/it]
                                                          
{'loss': 0.1972, 'grad_norm': 3.2552013397216797, 'learning_rate': 3.789423961353963e-05, 'epoch': 1.73}

 17%|█▋        | 7135/41250 [17:14:15<82:22:17,  8.69s/it][2025-04-26 01:11:58,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:11:58,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.17 | bwd_microstep: 5692.12 | bwd_inner_microstep: 5679.59 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.55
[2025-04-26 01:11:58,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.17 | bwd: 5692.13 | bwd_inner: 5679.59 | bwd_allreduce: 12.49 | step: 18.56

 17%|█▋        | 7136/41250 [17:14:24<82:10:17,  8.67s/it]
                                                          
{'loss': 0.1703, 'grad_norm': 3.0218944549560547, 'learning_rate': 3.7893538182728964e-05, 'epoch': 1.73}

 17%|█▋        | 7136/41250 [17:14:24<82:10:17,  8.67s/it][2025-04-26 01:12:07,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 01:12:07,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5691.55 | bwd_inner_microstep: 5675.91 | bwd_allreduce_microstep: 15.59 | step_microstep: 19.03
[2025-04-26 01:12:07,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5691.57 | bwd_inner: 5675.91 | bwd_allreduce: 15.61 | step: 19.03

 17%|█▋        | 7137/41250 [17:14:32<82:00:28,  8.65s/it]
                                                          
{'loss': 0.058, 'grad_norm': 1.5273315906524658, 'learning_rate': 3.789283664160815e-05, 'epoch': 1.73}

 17%|█▋        | 7137/41250 [17:14:32<82:00:28,  8.65s/it][2025-04-26 01:12:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:12:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5719.31 | bwd_inner_microstep: 5697.35 | bwd_allreduce_microstep: 21.92 | step_microstep: 18.86
[2025-04-26 01:12:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5719.33 | bwd_inner: 5697.35 | bwd_allreduce: 21.93 | step: 18.86

 17%|█▋        | 7138/41250 [17:14:41<81:59:50,  8.65s/it]
                                                          
{'loss': 0.1311, 'grad_norm': 0.5758270025253296, 'learning_rate': 3.789213499018152e-05, 'epoch': 1.73}

 17%|█▋        | 7138/41250 [17:14:41<81:59:50,  8.65s/it][2025-04-26 01:12:24,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:12:24,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.78 | bwd_microstep: 5720.63 | bwd_inner_microstep: 5691.44 | bwd_allreduce_microstep: 29.14 | step_microstep: 18.66
[2025-04-26 01:12:24,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.78 | bwd: 5720.64 | bwd_inner: 5691.44 | bwd_allreduce: 29.16 | step: 18.66

 17%|█▋        | 7139/41250 [17:14:50<81:59:54,  8.65s/it]
                                                          
{'loss': 0.1669, 'grad_norm': 1.9620836973190308, 'learning_rate': 3.7891433228453397e-05, 'epoch': 1.73}

 17%|█▋        | 7139/41250 [17:14:50<81:59:54,  8.65s/it][2025-04-26 01:12:33,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 01:12:33,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.85 | bwd_microstep: 5877.86 | bwd_inner_microstep: 5642.24 | bwd_allreduce_microstep: 235.57 | step_microstep: 19.30
[2025-04-26 01:12:33,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.85 | bwd: 5877.88 | bwd_inner: 5642.24 | bwd_allreduce: 235.60 | step: 19.30

 17%|█▋        | 7140/41250 [17:14:59<82:25:08,  8.70s/it]
                                                          
{'loss': 0.2484, 'grad_norm': 1.235915184020996, 'learning_rate': 3.78907313564281e-05, 'epoch': 1.73}

 17%|█▋        | 7140/41250 [17:14:59<82:25:08,  8.70s/it][2025-04-26 01:12:42,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 01:12:42,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.22 | bwd_microstep: 5772.81 | bwd_inner_microstep: 5641.18 | bwd_allreduce_microstep: 131.58 | step_microstep: 18.87
[2025-04-26 01:12:42,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.22 | bwd: 5772.82 | bwd_inner: 5641.18 | bwd_allreduce: 131.60 | step: 18.87

 17%|█▋        | 7141/41250 [17:15:07<82:22:13,  8.69s/it]
                                                          
{'loss': 0.0352, 'grad_norm': 0.8389608263969421, 'learning_rate': 3.7890029374109974e-05, 'epoch': 1.73}

 17%|█▋        | 7141/41250 [17:15:07<82:22:13,  8.69s/it][2025-04-26 01:12:51,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.45 | optimizer_step: 0.90
[2025-04-26 01:12:51,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.14 | bwd_microstep: 5769.37 | bwd_inner_microstep: 5633.20 | bwd_allreduce_microstep: 136.12 | step_microstep: 19.76
[2025-04-26 01:12:51,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.14 | bwd: 5769.39 | bwd_inner: 5633.20 | bwd_allreduce: 136.14 | step: 19.77

 17%|█▋        | 7142/41250 [17:15:16<82:19:32,  8.69s/it]
                                                          
{'loss': 0.135, 'grad_norm': 1.214301586151123, 'learning_rate': 3.788932728150332e-05, 'epoch': 1.73}

 17%|█▋        | 7142/41250 [17:15:16<82:19:32,  8.69s/it][2025-04-26 01:12:59,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:12:59,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.82 | bwd_microstep: 5726.46 | bwd_inner_microstep: 5681.42 | bwd_allreduce_microstep: 44.98 | step_microstep: 18.60
[2025-04-26 01:12:59,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.82 | bwd: 5726.47 | bwd_inner: 5681.42 | bwd_allreduce: 45.00 | step: 18.60

 17%|█▋        | 7143/41250 [17:15:25<82:14:22,  8.68s/it]
                                                          
{'loss': 0.0403, 'grad_norm': 0.6242594122886658, 'learning_rate': 3.7888625078612494e-05, 'epoch': 1.73}

 17%|█▋        | 7143/41250 [17:15:25<82:14:22,  8.68s/it][2025-04-26 01:13:08,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 01:13:08,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5686.83 | bwd_inner_microstep: 5616.40 | bwd_allreduce_microstep: 70.39 | step_microstep: 18.67
[2025-04-26 01:13:08,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5686.84 | bwd_inner: 5616.40 | bwd_allreduce: 70.40 | step: 18.67

 17%|█▋        | 7144/41250 [17:15:33<82:01:23,  8.66s/it]
                                                          
{'loss': 0.0864, 'grad_norm': 1.4260562658309937, 'learning_rate': 3.7887922765441815e-05, 'epoch': 1.73}

 17%|█▋        | 7144/41250 [17:15:33<82:01:23,  8.66s/it][2025-04-26 01:13:16,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 01:13:16,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.76 | bwd_microstep: 5684.13 | bwd_inner_microstep: 5670.99 | bwd_allreduce_microstep: 13.10 | step_microstep: 18.85
[2025-04-26 01:13:16,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.76 | bwd: 5684.15 | bwd_inner: 5670.99 | bwd_allreduce: 13.11 | step: 18.85

 17%|█▋        | 7145/41250 [17:15:42<81:52:50,  8.64s/it]
                                                          
{'loss': 0.1657, 'grad_norm': 2.2219607830047607, 'learning_rate': 3.788722034199561e-05, 'epoch': 1.73}

 17%|█▋        | 7145/41250 [17:15:42<81:52:50,  8.64s/it][2025-04-26 01:13:25,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.10 | optimizer_step: 1.02
[2025-04-26 01:13:25,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.66 | bwd_microstep: 5706.79 | bwd_inner_microstep: 5693.41 | bwd_allreduce_microstep: 13.32 | step_microstep: 18.98
[2025-04-26 01:13:25,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.66 | bwd: 5706.81 | bwd_inner: 5693.41 | bwd_allreduce: 13.35 | step: 18.99

 17%|█▋        | 7146/41250 [17:15:50<81:54:01,  8.65s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 0.9879442453384399, 'learning_rate': 3.7886517808278205e-05, 'epoch': 1.73}

 17%|█▋        | 7146/41250 [17:15:50<81:54:01,  8.65s/it][2025-04-26 01:13:34,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 01:13:34,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.07 | bwd_microstep: 5722.63 | bwd_inner_microstep: 5676.79 | bwd_allreduce_microstep: 45.79 | step_microstep: 19.04
[2025-04-26 01:13:34,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.07 | bwd: 5722.65 | bwd_inner: 5676.79 | bwd_allreduce: 45.81 | step: 19.04

 17%|█▋        | 7147/41250 [17:15:59<81:55:03,  8.65s/it]
                                                          
{'loss': 0.0948, 'grad_norm': 1.4240745306015015, 'learning_rate': 3.7885815164293936e-05, 'epoch': 1.73}

 17%|█▋        | 7147/41250 [17:15:59<81:55:03,  8.65s/it][2025-04-26 01:13:42,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:13:42,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.68 | bwd_microstep: 5708.69 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.69
[2025-04-26 01:13:42,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.68 | bwd: 5708.70 | bwd_inner: 5695.81 | bwd_allreduce: 12.86 | step: 18.69

 17%|█▋        | 7148/41250 [17:16:08<81:53:58,  8.65s/it]
                                                          
{'loss': 0.0783, 'grad_norm': 1.3048642873764038, 'learning_rate': 3.788511241004714e-05, 'epoch': 1.73}

 17%|█▋        | 7148/41250 [17:16:08<81:53:58,  8.65s/it][2025-04-26 01:13:51,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 01:13:51,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.94 | bwd_microstep: 5717.29 | bwd_inner_microstep: 5649.38 | bwd_allreduce_microstep: 67.87 | step_microstep: 18.23
[2025-04-26 01:13:51,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.94 | bwd: 5717.31 | bwd_inner: 5649.38 | bwd_allreduce: 67.88 | step: 18.23

 17%|█▋        | 7149/41250 [17:16:16<81:49:20,  8.64s/it]
                                                          
{'loss': 0.084, 'grad_norm': 1.0403445959091187, 'learning_rate': 3.788440954554214e-05, 'epoch': 1.73}

 17%|█▋        | 7149/41250 [17:16:16<81:49:20,  8.64s/it][2025-04-26 01:14:00,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:14:00,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.30 | bwd_microstep: 5697.93 | bwd_inner_microstep: 5685.04 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.51
[2025-04-26 01:14:00,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.30 | bwd: 5697.94 | bwd_inner: 5685.04 | bwd_allreduce: 12.86 | step: 18.52

 17%|█▋        | 7150/41250 [17:16:25<81:49:45,  8.64s/it]
                                                          
{'loss': 0.0213, 'grad_norm': 0.2637373208999634, 'learning_rate': 3.788370657078327e-05, 'epoch': 1.73}

 17%|█▋        | 7150/41250 [17:16:25<81:49:45,  8.64s/it][2025-04-26 01:14:08,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 01:14:08,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.65 | bwd_microstep: 5726.25 | bwd_inner_microstep: 5703.95 | bwd_allreduce_microstep: 22.25 | step_microstep: 18.98
[2025-04-26 01:14:08,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.65 | bwd: 5726.26 | bwd_inner: 5703.95 | bwd_allreduce: 22.26 | step: 18.98

 17%|█▋        | 7151/41250 [17:16:34<81:54:55,  8.65s/it]
                                                          
{'loss': 0.1653, 'grad_norm': 0.8042116761207581, 'learning_rate': 3.788300348577487e-05, 'epoch': 1.73}

 17%|█▋        | 7151/41250 [17:16:34<81:54:55,  8.65s/it][2025-04-26 01:14:17,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:14:17,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.91 | bwd_microstep: 5743.83 | bwd_inner_microstep: 5710.69 | bwd_allreduce_microstep: 33.10 | step_microstep: 18.71
[2025-04-26 01:14:17,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.91 | bwd: 5743.85 | bwd_inner: 5710.69 | bwd_allreduce: 33.12 | step: 18.72

 17%|█▋        | 7152/41250 [17:16:42<82:00:09,  8.66s/it]
                                                          
{'loss': 0.1118, 'grad_norm': 3.367612838745117, 'learning_rate': 3.788230029052127e-05, 'epoch': 1.73}

 17%|█▋        | 7152/41250 [17:16:42<82:00:09,  8.66s/it][2025-04-26 01:14:26,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:14:26,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.10 | bwd_microstep: 5736.37 | bwd_inner_microstep: 5643.71 | bwd_allreduce_microstep: 92.62 | step_microstep: 18.45
[2025-04-26 01:14:26,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.10 | bwd: 5736.39 | bwd_inner: 5643.71 | bwd_allreduce: 92.64 | step: 18.45

 17%|█▋        | 7153/41250 [17:16:51<82:00:31,  8.66s/it]
                                                          
{'loss': 0.2113, 'grad_norm': 24.89377212524414, 'learning_rate': 3.788159698502681e-05, 'epoch': 1.73}

 17%|█▋        | 7153/41250 [17:16:51<82:00:31,  8.66s/it][2025-04-26 01:14:34,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:14:34,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.02 | bwd_microstep: 5714.06 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.69
[2025-04-26 01:14:34,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.02 | bwd: 5714.08 | bwd_inner: 5701.15 | bwd_allreduce: 12.89 | step: 18.69

 17%|█▋        | 7154/41250 [17:17:00<81:58:35,  8.66s/it]
                                                          
{'loss': 0.0845, 'grad_norm': 1.157112717628479, 'learning_rate': 3.788089356929582e-05, 'epoch': 1.73}

 17%|█▋        | 7154/41250 [17:17:00<81:58:35,  8.66s/it][2025-04-26 01:14:43,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:14:43,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5734.08 | bwd_inner_microstep: 5700.14 | bwd_allreduce_microstep: 33.90 | step_microstep: 18.37
[2025-04-26 01:14:43,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5734.10 | bwd_inner: 5700.14 | bwd_allreduce: 33.92 | step: 18.38

 17%|█▋        | 7155/41250 [17:17:08<82:01:10,  8.66s/it]
                                                          
{'loss': 0.0179, 'grad_norm': 0.49352723360061646, 'learning_rate': 3.788019004333263e-05, 'epoch': 1.73}

 17%|█▋        | 7155/41250 [17:17:08<82:01:10,  8.66s/it][2025-04-26 01:14:52,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:14:52,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.34 | bwd_microstep: 5711.77 | bwd_inner_microstep: 5698.77 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.29
[2025-04-26 01:14:52,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.34 | bwd: 5711.78 | bwd_inner: 5698.77 | bwd_allreduce: 12.97 | step: 18.29

 17%|█▋        | 7156/41250 [17:17:17<81:57:53,  8.65s/it]
                                                          
{'loss': 0.1114, 'grad_norm': 1.2460300922393799, 'learning_rate': 3.7879486407141596e-05, 'epoch': 1.73}

 17%|█▋        | 7156/41250 [17:17:17<81:57:53,  8.65s/it][2025-04-26 01:15:00,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:15:00,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.30 | bwd_microstep: 5722.74 | bwd_inner_microstep: 5709.70 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.42
[2025-04-26 01:15:00,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.30 | bwd: 5722.76 | bwd_inner: 5709.70 | bwd_allreduce: 13.02 | step: 18.42

 17%|█▋        | 7157/41250 [17:17:26<81:57:13,  8.65s/it]
                                                          
{'loss': 0.1173, 'grad_norm': 2.4878170490264893, 'learning_rate': 3.787878266072704e-05, 'epoch': 1.74}

 17%|█▋        | 7157/41250 [17:17:26<81:57:13,  8.65s/it][2025-04-26 01:15:09,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:15:09,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.24 | bwd_microstep: 5704.99 | bwd_inner_microstep: 5659.40 | bwd_allreduce_microstep: 45.54 | step_microstep: 18.41
[2025-04-26 01:15:09,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.24 | bwd: 5705.01 | bwd_inner: 5659.40 | bwd_allreduce: 45.56 | step: 18.42

 17%|█▋        | 7158/41250 [17:17:34<81:52:33,  8.65s/it]
                                                          
{'loss': 0.3255, 'grad_norm': 4.1654863357543945, 'learning_rate': 3.78780788040933e-05, 'epoch': 1.74}

 17%|█▋        | 7158/41250 [17:17:34<81:52:33,  8.65s/it][2025-04-26 01:15:18,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.30 | optimizer_step: 1.04
[2025-04-26 01:15:18,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.82 | bwd_microstep: 5704.52 | bwd_inner_microstep: 5690.63 | bwd_allreduce_microstep: 13.83 | step_microstep: 20.20
[2025-04-26 01:15:18,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.82 | bwd: 5704.54 | bwd_inner: 5690.63 | bwd_allreduce: 13.86 | step: 20.21

 17%|█▋        | 7159/41250 [17:17:43<81:51:29,  8.64s/it]
                                                          
{'loss': 0.0824, 'grad_norm': 1.4054968357086182, 'learning_rate': 3.787737483724473e-05, 'epoch': 1.74}

 17%|█▋        | 7159/41250 [17:17:43<81:51:29,  8.64s/it][2025-04-26 01:15:26,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 01:15:26,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5787.96 | bwd_inner_microstep: 5656.26 | bwd_allreduce_microstep: 131.65 | step_microstep: 18.67
[2025-04-26 01:15:26,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5787.97 | bwd_inner: 5656.26 | bwd_allreduce: 131.67 | step: 18.68

 17%|█▋        | 7160/41250 [17:17:52<82:00:45,  8.66s/it]
                                                          
{'loss': 0.0343, 'grad_norm': 0.7637341022491455, 'learning_rate': 3.787667076018566e-05, 'epoch': 1.74}

 17%|█▋        | 7160/41250 [17:17:52<82:00:45,  8.66s/it][2025-04-26 01:15:35,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:15:35,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.17 | bwd_microstep: 5764.68 | bwd_inner_microstep: 5654.87 | bwd_allreduce_microstep: 109.76 | step_microstep: 18.26
[2025-04-26 01:15:35,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.17 | bwd: 5764.69 | bwd_inner: 5654.87 | bwd_allreduce: 109.78 | step: 18.27

 17%|█▋        | 7161/41250 [17:18:00<82:02:36,  8.66s/it]
                                                          
{'loss': 0.16, 'grad_norm': 2.292487144470215, 'learning_rate': 3.787596657292042e-05, 'epoch': 1.74}

 17%|█▋        | 7161/41250 [17:18:00<82:02:36,  8.66s/it][2025-04-26 01:15:44,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:15:44,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.30 | bwd_microstep: 5723.70 | bwd_inner_microstep: 5711.08 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.62
[2025-04-26 01:15:44,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.30 | bwd: 5723.71 | bwd_inner: 5711.08 | bwd_allreduce: 12.59 | step: 18.62

 17%|█▋        | 7162/41250 [17:18:09<82:02:08,  8.66s/it]
                                                          
{'loss': 0.2045, 'grad_norm': 2.3152174949645996, 'learning_rate': 3.7875262275453375e-05, 'epoch': 1.74}

 17%|█▋        | 7162/41250 [17:18:09<82:02:08,  8.66s/it][2025-04-26 01:15:52,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:15:52,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.75 | bwd_microstep: 5706.36 | bwd_inner_microstep: 5662.75 | bwd_allreduce_microstep: 43.56 | step_microstep: 18.68
[2025-04-26 01:15:52,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.75 | bwd: 5706.38 | bwd_inner: 5662.75 | bwd_allreduce: 43.58 | step: 18.68

 17%|█▋        | 7163/41250 [17:18:18<81:54:24,  8.65s/it]
                                                          
{'loss': 0.0891, 'grad_norm': 1.4978351593017578, 'learning_rate': 3.7874557867788846e-05, 'epoch': 1.74}

 17%|█▋        | 7163/41250 [17:18:18<81:54:24,  8.65s/it][2025-04-26 01:16:01,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:16:01,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.70 | bwd_microstep: 5790.47 | bwd_inner_microstep: 5688.88 | bwd_allreduce_microstep: 101.54 | step_microstep: 18.49
[2025-04-26 01:16:01,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.70 | bwd: 5790.48 | bwd_inner: 5688.88 | bwd_allreduce: 101.56 | step: 18.49

 17%|█▋        | 7164/41250 [17:18:26<82:05:40,  8.67s/it]
                                                          
{'loss': 0.0236, 'grad_norm': 0.47722944617271423, 'learning_rate': 3.7873853349931186e-05, 'epoch': 1.74}

 17%|█▋        | 7164/41250 [17:18:26<82:05:40,  8.67s/it][2025-04-26 01:16:10,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 01:16:10,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5768.16 | bwd_inner_microstep: 5708.63 | bwd_allreduce_microstep: 59.49 | step_microstep: 19.36
[2025-04-26 01:16:10,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5768.17 | bwd_inner: 5708.63 | bwd_allreduce: 59.51 | step: 19.36

 17%|█▋        | 7165/41250 [17:18:35<82:11:23,  8.68s/it]
                                                          
{'loss': 0.0051, 'grad_norm': 0.12203103303909302, 'learning_rate': 3.787314872188474e-05, 'epoch': 1.74}

 17%|█▋        | 7165/41250 [17:18:35<82:11:23,  8.68s/it][2025-04-26 01:16:18,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:16:18,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.63 | bwd_microstep: 5725.10 | bwd_inner_microstep: 5712.35 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.56
[2025-04-26 01:16:18,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.63 | bwd: 5725.12 | bwd_inner: 5712.35 | bwd_allreduce: 12.72 | step: 18.56

 17%|█▋        | 7166/41250 [17:18:44<82:07:49,  8.67s/it]
                                                          
{'loss': 0.1164, 'grad_norm': 1.7620210647583008, 'learning_rate': 3.787244398365384e-05, 'epoch': 1.74}

 17%|█▋        | 7166/41250 [17:18:44<82:07:49,  8.67s/it][2025-04-26 01:16:27,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 01:16:27,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.82 | bwd_microstep: 5710.96 | bwd_inner_microstep: 5698.42 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.15
[2025-04-26 01:16:27,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.82 | bwd: 5710.98 | bwd_inner: 5698.42 | bwd_allreduce: 12.52 | step: 18.15

 17%|█▋        | 7167/41250 [17:18:52<82:01:42,  8.66s/it]
                                                          
{'loss': 0.1769, 'grad_norm': 3.987546682357788, 'learning_rate': 3.787173913524284e-05, 'epoch': 1.74}

 17%|█▋        | 7167/41250 [17:18:52<82:01:42,  8.66s/it][2025-04-26 01:16:36,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 01:16:36,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.28 | bwd_microstep: 5721.20 | bwd_inner_microstep: 5708.26 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.46
[2025-04-26 01:16:36,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.28 | bwd: 5721.21 | bwd_inner: 5708.26 | bwd_allreduce: 12.91 | step: 18.46

 17%|█▋        | 7168/41250 [17:19:01<81:59:41,  8.66s/it]
                                                          
{'loss': 0.3482, 'grad_norm': 6.865823745727539, 'learning_rate': 3.7871034176656084e-05, 'epoch': 1.74}

 17%|█▋        | 7168/41250 [17:19:01<81:59:41,  8.66s/it][2025-04-26 01:16:44,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:16:44,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.86 | bwd_microstep: 5716.21 | bwd_inner_microstep: 5703.58 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.70
[2025-04-26 01:16:44,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.86 | bwd: 5716.22 | bwd_inner: 5703.58 | bwd_allreduce: 12.60 | step: 18.70

 17%|█▋        | 7169/41250 [17:19:10<81:57:28,  8.66s/it]
                                                          
{'loss': 0.063, 'grad_norm': 2.097896099090576, 'learning_rate': 3.7870329107897925e-05, 'epoch': 1.74}

 17%|█▋        | 7169/41250 [17:19:10<81:57:28,  8.66s/it][2025-04-26 01:16:53,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 01:16:53,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5726.68 | bwd_inner_microstep: 5642.55 | bwd_allreduce_microstep: 84.09 | step_microstep: 18.65
[2025-04-26 01:16:53,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5726.70 | bwd_inner: 5642.55 | bwd_allreduce: 84.11 | step: 18.66

 17%|█▋        | 7170/41250 [17:19:18<81:54:08,  8.65s/it]
                                                          
{'loss': 0.1445, 'grad_norm': 1.697943925857544, 'learning_rate': 3.786962392897269e-05, 'epoch': 1.74}

 17%|█▋        | 7170/41250 [17:19:18<81:54:08,  8.65s/it][2025-04-26 01:17:02,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:17:02,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5773.59 | bwd_inner_microstep: 5693.50 | bwd_allreduce_microstep: 80.04 | step_microstep: 18.51
[2025-04-26 01:17:02,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5773.60 | bwd_inner: 5693.50 | bwd_allreduce: 80.06 | step: 18.51

 17%|█▋        | 7171/41250 [17:19:27<82:03:08,  8.67s/it]
                                                          
{'loss': 0.1447, 'grad_norm': 6.59114933013916, 'learning_rate': 3.786891863988475e-05, 'epoch': 1.74}

 17%|█▋        | 7171/41250 [17:19:27<82:03:08,  8.67s/it][2025-04-26 01:17:10,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-26 01:17:10,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.73 | bwd_microstep: 5768.08 | bwd_inner_microstep: 5656.82 | bwd_allreduce_microstep: 111.22 | step_microstep: 18.50
[2025-04-26 01:17:10,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.73 | bwd: 5768.10 | bwd_inner: 5656.82 | bwd_allreduce: 111.24 | step: 18.50

 17%|█▋        | 7172/41250 [17:19:36<82:04:56,  8.67s/it]
                                                          
{'loss': 0.1036, 'grad_norm': 1.4164670705795288, 'learning_rate': 3.786821324063843e-05, 'epoch': 1.74}

 17%|█▋        | 7172/41250 [17:19:36<82:04:56,  8.67s/it][2025-04-26 01:17:19,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:17:19,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.06 | bwd_microstep: 5705.56 | bwd_inner_microstep: 5645.81 | bwd_allreduce_microstep: 59.71 | step_microstep: 18.38
[2025-04-26 01:17:19,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.06 | bwd: 5705.58 | bwd_inner: 5645.81 | bwd_allreduce: 59.73 | step: 18.39

 17%|█▋        | 7173/41250 [17:19:44<81:54:55,  8.65s/it]
                                                          
{'loss': 0.2264, 'grad_norm': 2.4890010356903076, 'learning_rate': 3.786750773123809e-05, 'epoch': 1.74}

 17%|█▋        | 7173/41250 [17:19:44<81:54:55,  8.65s/it][2025-04-26 01:17:28,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 01:17:28,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.69 | bwd_microstep: 5745.34 | bwd_inner_microstep: 5700.45 | bwd_allreduce_microstep: 44.85 | step_microstep: 18.54
[2025-04-26 01:17:28,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.69 | bwd: 5745.36 | bwd_inner: 5700.45 | bwd_allreduce: 44.87 | step: 18.54

 17%|█▋        | 7174/41250 [17:19:53<82:00:07,  8.66s/it]
                                                          
{'loss': 0.0238, 'grad_norm': 0.5343949198722839, 'learning_rate': 3.7866802111688084e-05, 'epoch': 1.74}

 17%|█▋        | 7174/41250 [17:19:53<82:00:07,  8.66s/it][2025-04-26 01:17:36,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 01:17:36,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.99 | bwd_microstep: 5748.39 | bwd_inner_microstep: 5711.21 | bwd_allreduce_microstep: 37.14 | step_microstep: 18.32
[2025-04-26 01:17:36,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.99 | bwd: 5748.41 | bwd_inner: 5711.21 | bwd_allreduce: 37.15 | step: 18.32

 17%|█▋        | 7175/41250 [17:20:02<82:02:49,  8.67s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 2.2274909019470215, 'learning_rate': 3.7866096381992766e-05, 'epoch': 1.74}

 17%|█▋        | 7175/41250 [17:20:02<82:02:49,  8.67s/it][2025-04-26 01:17:45,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 01:17:45,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5731.06 | bwd_inner_microstep: 5696.52 | bwd_allreduce_microstep: 34.50 | step_microstep: 19.62
[2025-04-26 01:17:45,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5731.08 | bwd_inner: 5696.52 | bwd_allreduce: 34.52 | step: 19.62

 17%|█▋        | 7176/41250 [17:20:10<82:03:15,  8.67s/it]
                                                          
{'loss': 0.4989, 'grad_norm': 4.474792003631592, 'learning_rate': 3.7865390542156466e-05, 'epoch': 1.74}

 17%|█▋        | 7176/41250 [17:20:10<82:03:15,  8.67s/it][2025-04-26 01:17:54,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 01:17:54,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.22 | bwd_microstep: 5891.85 | bwd_inner_microstep: 5654.83 | bwd_allreduce_microstep: 236.98 | step_microstep: 19.13
[2025-04-26 01:17:54,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.22 | bwd: 5891.86 | bwd_inner: 5654.83 | bwd_allreduce: 236.99 | step: 19.13

 17%|█▋        | 7177/41250 [17:20:19<82:25:51,  8.71s/it]
                                                          
{'loss': 0.1829, 'grad_norm': 1.8532142639160156, 'learning_rate': 3.786468459218354e-05, 'epoch': 1.74}

 17%|█▋        | 7177/41250 [17:20:19<82:25:51,  8.71s/it][2025-04-26 01:18:02,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 01:18:02,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.80 | bwd_microstep: 5701.15 | bwd_inner_microstep: 5654.68 | bwd_allreduce_microstep: 46.42 | step_microstep: 18.96
[2025-04-26 01:18:02,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.80 | bwd: 5701.17 | bwd_inner: 5654.68 | bwd_allreduce: 46.44 | step: 18.97

 17%|█▋        | 7178/41250 [17:20:28<82:08:38,  8.68s/it]
                                                          
{'loss': 0.0887, 'grad_norm': 2.9803659915924072, 'learning_rate': 3.7863978532078364e-05, 'epoch': 1.74}

 17%|█▋        | 7178/41250 [17:20:28<82:08:38,  8.68s/it][2025-04-26 01:18:11,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:18:11,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5722.50 | bwd_inner_microstep: 5709.62 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.88
[2025-04-26 01:18:11,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5722.52 | bwd_inner: 5709.62 | bwd_allreduce: 12.85 | step: 18.88

 17%|█▋        | 7179/41250 [17:20:36<82:05:32,  8.67s/it]
                                                          
{'loss': 0.0367, 'grad_norm': 0.5166900753974915, 'learning_rate': 3.786327236184527e-05, 'epoch': 1.74}

 17%|█▋        | 7179/41250 [17:20:36<82:05:32,  8.67s/it][2025-04-26 01:18:20,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 01:18:20,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.71 | bwd_microstep: 5759.12 | bwd_inner_microstep: 5681.51 | bwd_allreduce_microstep: 77.56 | step_microstep: 19.22
[2025-04-26 01:18:20,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.71 | bwd: 5759.13 | bwd_inner: 5681.51 | bwd_allreduce: 77.58 | step: 19.22

 17%|█▋        | 7180/41250 [17:20:45<82:07:42,  8.68s/it]
                                                          
{'loss': 0.0981, 'grad_norm': 2.049466133117676, 'learning_rate': 3.7862566081488614e-05, 'epoch': 1.74}

 17%|█▋        | 7180/41250 [17:20:45<82:07:42,  8.68s/it][2025-04-26 01:18:28,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:18:28,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.51 | bwd_microstep: 5692.40 | bwd_inner_microstep: 5679.29 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.83
[2025-04-26 01:18:28,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.51 | bwd: 5692.42 | bwd_inner: 5679.29 | bwd_allreduce: 13.08 | step: 18.83

 17%|█▋        | 7181/41250 [17:20:54<81:59:20,  8.66s/it]
                                                          
{'loss': 0.4496, 'grad_norm': 4.769265651702881, 'learning_rate': 3.786185969101275e-05, 'epoch': 1.74}

 17%|█▋        | 7181/41250 [17:20:54<81:59:20,  8.66s/it][2025-04-26 01:18:37,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 01:18:37,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.55 | bwd_microstep: 5710.75 | bwd_inner_microstep: 5697.85 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.41
[2025-04-26 01:18:37,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.56 | bwd: 5710.76 | bwd_inner: 5697.85 | bwd_allreduce: 12.86 | step: 19.42

 17%|█▋        | 7182/41250 [17:21:02<81:56:36,  8.66s/it]
                                                          
{'loss': 0.0863, 'grad_norm': 2.0010087490081787, 'learning_rate': 3.786115319042203e-05, 'epoch': 1.74}

 17%|█▋        | 7182/41250 [17:21:02<81:56:36,  8.66s/it][2025-04-26 01:18:46,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:18:46,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.06 | bwd_microstep: 5984.58 | bwd_inner_microstep: 5696.93 | bwd_allreduce_microstep: 287.61 | step_microstep: 18.88
[2025-04-26 01:18:46,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.06 | bwd: 5984.59 | bwd_inner: 5696.93 | bwd_allreduce: 287.63 | step: 18.88

 17%|█▋        | 7183/41250 [17:21:11<82:41:05,  8.74s/it]
                                                          
{'loss': 0.0255, 'grad_norm': 0.7470961213111877, 'learning_rate': 3.786044657972082e-05, 'epoch': 1.74}

 17%|█▋        | 7183/41250 [17:21:11<82:41:05,  8.74s/it][2025-04-26 01:18:55,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:18:55,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.50 | bwd_microstep: 5765.53 | bwd_inner_microstep: 5649.79 | bwd_allreduce_microstep: 115.69 | step_microstep: 18.99
[2025-04-26 01:18:55,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.50 | bwd: 5765.54 | bwd_inner: 5649.79 | bwd_allreduce: 115.71 | step: 18.99

 17%|█▋        | 7184/41250 [17:21:20<82:30:13,  8.72s/it]
                                                          
{'loss': 0.0524, 'grad_norm': 0.9489001631736755, 'learning_rate': 3.785973985891347e-05, 'epoch': 1.74}

 17%|█▋        | 7184/41250 [17:21:20<82:30:13,  8.72s/it][2025-04-26 01:19:03,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:19:03,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.02 | bwd_microstep: 5754.63 | bwd_inner_microstep: 5694.10 | bwd_allreduce_microstep: 60.49 | step_microstep: 19.19
[2025-04-26 01:19:03,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.02 | bwd: 5754.65 | bwd_inner: 5694.10 | bwd_allreduce: 60.51 | step: 19.20

 17%|█▋        | 7185/41250 [17:21:29<82:24:36,  8.71s/it]
                                                          
{'loss': 0.2372, 'grad_norm': 3.233020544052124, 'learning_rate': 3.7859033028004334e-05, 'epoch': 1.74}

 17%|█▋        | 7185/41250 [17:21:29<82:24:36,  8.71s/it][2025-04-26 01:19:12,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 01:19:12,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5705.17 | bwd_inner_microstep: 5692.22 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.15
[2025-04-26 01:19:12,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5705.19 | bwd_inner: 5692.22 | bwd_allreduce: 12.92 | step: 19.15

 17%|█▋        | 7186/41250 [17:21:37<82:11:41,  8.69s/it]
                                                          
{'loss': 0.0522, 'grad_norm': 1.0372843742370605, 'learning_rate': 3.785832608699777e-05, 'epoch': 1.74}

 17%|█▋        | 7186/41250 [17:21:37<82:11:41,  8.69s/it][2025-04-26 01:19:20,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.04 | optimizer_step: 1.19
[2025-04-26 01:19:20,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.35 | bwd_microstep: 5750.22 | bwd_inner_microstep: 5634.60 | bwd_allreduce_microstep: 115.56 | step_microstep: 19.51
[2025-04-26 01:19:20,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.35 | bwd: 5750.23 | bwd_inner: 5634.60 | bwd_allreduce: 115.58 | step: 19.51

 17%|█▋        | 7187/41250 [17:21:46<82:06:37,  8.68s/it]
                                                          
{'loss': 0.0318, 'grad_norm': 1.045538306236267, 'learning_rate': 3.785761903589814e-05, 'epoch': 1.74}

 17%|█▋        | 7187/41250 [17:21:46<82:06:37,  8.68s/it][2025-04-26 01:19:29,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:19:29,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.31 | bwd_microstep: 5750.48 | bwd_inner_microstep: 5646.92 | bwd_allreduce_microstep: 103.51 | step_microstep: 18.81
[2025-04-26 01:19:29,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.31 | bwd: 5750.50 | bwd_inner: 5646.92 | bwd_allreduce: 103.53 | step: 18.81

 17%|█▋        | 7188/41250 [17:21:54<82:03:13,  8.67s/it]
                                                          
{'loss': 0.0955, 'grad_norm': 1.7443342208862305, 'learning_rate': 3.78569118747098e-05, 'epoch': 1.74}

 17%|█▋        | 7188/41250 [17:21:54<82:03:13,  8.67s/it][2025-04-26 01:19:38,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 01:19:38,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.49 | bwd_microstep: 5744.11 | bwd_inner_microstep: 5679.43 | bwd_allreduce_microstep: 64.63 | step_microstep: 19.35
[2025-04-26 01:19:38,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.49 | bwd: 5744.13 | bwd_inner: 5679.43 | bwd_allreduce: 64.65 | step: 19.36

 17%|█▋        | 7189/41250 [17:22:03<82:03:25,  8.67s/it]
                                                          
{'loss': 0.0383, 'grad_norm': 2.1026296615600586, 'learning_rate': 3.7856204603437116e-05, 'epoch': 1.74}

 17%|█▋        | 7189/41250 [17:22:03<82:03:25,  8.67s/it][2025-04-26 01:19:46,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:19:46,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.18 | bwd_microstep: 5758.53 | bwd_inner_microstep: 5646.66 | bwd_allreduce_microstep: 111.83 | step_microstep: 18.70
[2025-04-26 01:19:46,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.18 | bwd: 5758.54 | bwd_inner: 5646.66 | bwd_allreduce: 111.84 | step: 18.70

 17%|█▋        | 7190/41250 [17:22:12<82:02:20,  8.67s/it]
                                                          
{'loss': 0.0523, 'grad_norm': 4.185266971588135, 'learning_rate': 3.785549722208444e-05, 'epoch': 1.74}

 17%|█▋        | 7190/41250 [17:22:12<82:02:20,  8.67s/it][2025-04-26 01:19:55,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:19:55,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.32 | bwd_microstep: 5688.49 | bwd_inner_microstep: 5634.41 | bwd_allreduce_microstep: 54.03 | step_microstep: 18.73
[2025-04-26 01:19:55,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.32 | bwd: 5688.50 | bwd_inner: 5634.41 | bwd_allreduce: 54.05 | step: 18.73

 17%|█▋        | 7191/41250 [17:22:20<81:48:41,  8.65s/it]
                                                          
{'loss': 0.0071, 'grad_norm': 0.16819573938846588, 'learning_rate': 3.7854789730656135e-05, 'epoch': 1.74}

 17%|█▋        | 7191/41250 [17:22:20<81:48:41,  8.65s/it][2025-04-26 01:20:04,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:20:04,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.94 | bwd_microstep: 5775.56 | bwd_inner_microstep: 5641.54 | bwd_allreduce_microstep: 133.97 | step_microstep: 18.76
[2025-04-26 01:20:04,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.94 | bwd: 5775.57 | bwd_inner: 5641.54 | bwd_allreduce: 133.99 | step: 18.76

 17%|█▋        | 7192/41250 [17:22:29<81:54:47,  8.66s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 2.655128240585327, 'learning_rate': 3.7854082129156564e-05, 'epoch': 1.74}

 17%|█▋        | 7192/41250 [17:22:29<81:54:47,  8.66s/it][2025-04-26 01:20:12,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 1.13
[2025-04-26 01:20:12,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.04 | bwd_microstep: 5766.97 | bwd_inner_microstep: 5646.63 | bwd_allreduce_microstep: 120.29 | step_microstep: 18.90
[2025-04-26 01:20:12,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.04 | bwd: 5766.98 | bwd_inner: 5646.63 | bwd_allreduce: 120.31 | step: 18.90

 17%|█▋        | 7193/41250 [17:22:38<81:57:18,  8.66s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 1.6586754322052002, 'learning_rate': 3.7853374417590084e-05, 'epoch': 1.74}

 17%|█▋        | 7193/41250 [17:22:38<81:57:18,  8.66s/it][2025-04-26 01:20:21,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 01:20:21,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.51 | bwd_microstep: 5731.88 | bwd_inner_microstep: 5690.98 | bwd_allreduce_microstep: 40.85 | step_microstep: 18.44
[2025-04-26 01:20:21,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.51 | bwd: 5731.89 | bwd_inner: 5690.98 | bwd_allreduce: 40.87 | step: 18.44

 17%|█▋        | 7194/41250 [17:22:46<81:58:52,  8.67s/it]
                                                          
{'loss': 0.0223, 'grad_norm': 0.6152781844139099, 'learning_rate': 3.785266659596107e-05, 'epoch': 1.74}

 17%|█▋        | 7194/41250 [17:22:46<81:58:52,  8.67s/it][2025-04-26 01:20:30,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 01:20:30,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.12 | bwd_microstep: 5695.93 | bwd_inner_microstep: 5683.23 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.59
[2025-04-26 01:20:30,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.12 | bwd: 5695.94 | bwd_inner: 5683.23 | bwd_allreduce: 12.67 | step: 18.59

 17%|█▋        | 7195/41250 [17:22:55<81:51:30,  8.65s/it]
                                                          
{'loss': 0.3403, 'grad_norm': 1.6522778272628784, 'learning_rate': 3.785195866427387e-05, 'epoch': 1.74}

 17%|█▋        | 7195/41250 [17:22:55<81:51:30,  8.65s/it][2025-04-26 01:20:38,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:20:38,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5766.97 | bwd_inner_microstep: 5655.87 | bwd_allreduce_microstep: 111.05 | step_microstep: 18.98
[2025-04-26 01:20:38,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5766.98 | bwd_inner: 5655.87 | bwd_allreduce: 111.07 | step: 18.98

 17%|█▋        | 7196/41250 [17:23:04<81:55:12,  8.66s/it]
                                                          
{'loss': 0.0354, 'grad_norm': 0.6880890130996704, 'learning_rate': 3.785125062253286e-05, 'epoch': 1.74}

 17%|█▋        | 7196/41250 [17:23:04<81:55:12,  8.66s/it][2025-04-26 01:20:47,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:20:47,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.60 | bwd_microstep: 5766.35 | bwd_inner_microstep: 5710.02 | bwd_allreduce_microstep: 56.28 | step_microstep: 18.26
[2025-04-26 01:20:47,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.61 | bwd: 5766.36 | bwd_inner: 5710.02 | bwd_allreduce: 56.30 | step: 18.27

 17%|█▋        | 7197/41250 [17:23:12<82:01:26,  8.67s/it]
                                                          
{'loss': 0.3677, 'grad_norm': 2.69596529006958, 'learning_rate': 3.7850542470742404e-05, 'epoch': 1.74}

 17%|█▋        | 7197/41250 [17:23:12<82:01:26,  8.67s/it][2025-04-26 01:20:56,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:20:56,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.05 | bwd_microstep: 5689.81 | bwd_inner_microstep: 5653.12 | bwd_allreduce_microstep: 36.63 | step_microstep: 18.73
[2025-04-26 01:20:56,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.05 | bwd: 5689.82 | bwd_inner: 5653.12 | bwd_allreduce: 36.66 | step: 18.74

 17%|█▋        | 7198/41250 [17:23:21<81:48:42,  8.65s/it]
                                                          
{'loss': 0.0106, 'grad_norm': 0.13082551956176758, 'learning_rate': 3.784983420890686e-05, 'epoch': 1.74}

 17%|█▋        | 7198/41250 [17:23:21<81:48:42,  8.65s/it][2025-04-26 01:21:04,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.33 | optimizer_step: 1.05
[2025-04-26 01:21:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5764.57 | bwd_inner_microstep: 5701.68 | bwd_allreduce_microstep: 62.83 | step_microstep: 20.26
[2025-04-26 01:21:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5764.59 | bwd_inner: 5701.68 | bwd_allreduce: 62.86 | step: 20.26

 17%|█▋        | 7199/41250 [17:23:30<81:56:37,  8.66s/it]
                                                          
{'loss': 0.1543, 'grad_norm': 2.916114568710327, 'learning_rate': 3.78491258370306e-05, 'epoch': 1.75}

 17%|█▋        | 7199/41250 [17:23:30<81:56:37,  8.66s/it][2025-04-26 01:21:13,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-26 01:21:13,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.07 | bwd_microstep: 5793.04 | bwd_inner_microstep: 5653.63 | bwd_allreduce_microstep: 139.36 | step_microstep: 19.15
[2025-04-26 01:21:13,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.07 | bwd: 5793.06 | bwd_inner: 5653.63 | bwd_allreduce: 139.38 | step: 19.15

 17%|█▋        | 7200/41250 [17:23:38<82:02:33,  8.67s/it]
                                                          
{'loss': 0.1669, 'grad_norm': 1.2921584844589233, 'learning_rate': 3.784841735511799e-05, 'epoch': 1.75}

 17%|█▋        | 7200/41250 [17:23:38<82:02:33,  8.67s/it][2025-04-26 01:21:22,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:21:22,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 5706.85 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.90
[2025-04-26 01:21:22,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 5706.86 | bwd_inner: 5693.87 | bwd_allreduce: 12.95 | step: 18.90

 17%|█▋        | 7201/41250 [17:23:47<81:56:52,  8.66s/it]
                                                          
{'loss': 0.1027, 'grad_norm': 1.0119730234146118, 'learning_rate': 3.78477087631734e-05, 'epoch': 1.75}

 17%|█▋        | 7201/41250 [17:23:47<81:56:52,  8.66s/it][2025-04-26 01:21:30,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 01:21:30,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.66 | bwd_microstep: 5698.72 | bwd_inner_microstep: 5657.57 | bwd_allreduce_microstep: 41.10 | step_microstep: 18.62
[2025-04-26 01:21:30,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.66 | bwd: 5698.73 | bwd_inner: 5657.57 | bwd_allreduce: 41.12 | step: 18.62

 17%|█▋        | 7202/41250 [17:23:56<81:47:02,  8.65s/it]
                                                          
{'loss': 0.1127, 'grad_norm': 2.1186676025390625, 'learning_rate': 3.784700006120119e-05, 'epoch': 1.75}

 17%|█▋        | 7202/41250 [17:23:56<81:47:02,  8.65s/it][2025-04-26 01:21:39,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:21:39,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.75 | bwd_microstep: 5711.84 | bwd_inner_microstep: 5699.03 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.87
[2025-04-26 01:21:39,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.75 | bwd: 5711.86 | bwd_inner: 5699.03 | bwd_allreduce: 12.78 | step: 18.87

 17%|█▋        | 7203/41250 [17:24:04<81:46:57,  8.65s/it]
                                                          
{'loss': 0.1959, 'grad_norm': 2.2747037410736084, 'learning_rate': 3.784629124920574e-05, 'epoch': 1.75}

 17%|█▋        | 7203/41250 [17:24:04<81:46:57,  8.65s/it][2025-04-26 01:21:48,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:21:48,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.64 | bwd_microstep: 5763.82 | bwd_inner_microstep: 5663.35 | bwd_allreduce_microstep: 100.42 | step_microstep: 18.88
[2025-04-26 01:21:48,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.64 | bwd: 5763.84 | bwd_inner: 5663.35 | bwd_allreduce: 100.44 | step: 18.89

 17%|█▋        | 7204/41250 [17:24:13<81:52:49,  8.66s/it]
                                                          
{'loss': 0.1535, 'grad_norm': 1.6289682388305664, 'learning_rate': 3.784558232719141e-05, 'epoch': 1.75}

 17%|█▋        | 7204/41250 [17:24:13<81:52:49,  8.66s/it][2025-04-26 01:21:56,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 01:21:56,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.87 | bwd_microstep: 5712.55 | bwd_inner_microstep: 5699.82 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.01
[2025-04-26 01:21:56,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.87 | bwd: 5712.57 | bwd_inner: 5699.82 | bwd_allreduce: 12.71 | step: 19.01

 17%|█▋        | 7205/41250 [17:24:22<81:50:14,  8.65s/it]
                                                          
{'loss': 0.0513, 'grad_norm': 1.6720925569534302, 'learning_rate': 3.784487329516258e-05, 'epoch': 1.75}

 17%|█▋        | 7205/41250 [17:24:22<81:50:14,  8.65s/it][2025-04-26 01:22:05,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:22:05,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 6010.71 | bwd_inner_microstep: 5698.51 | bwd_allreduce_microstep: 312.16 | step_microstep: 18.88
[2025-04-26 01:22:05,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 6010.73 | bwd_inner: 5698.51 | bwd_allreduce: 312.18 | step: 18.88

 17%|█▋        | 7206/41250 [17:24:31<82:39:30,  8.74s/it]
                                                          
{'loss': 0.0767, 'grad_norm': 1.4095935821533203, 'learning_rate': 3.7844164153123615e-05, 'epoch': 1.75}

 17%|█▋        | 7206/41250 [17:24:31<82:39:30,  8.74s/it][2025-04-26 01:22:14,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 01:22:14,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.66 | bwd_microstep: 5714.74 | bwd_inner_microstep: 5701.71 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.99
[2025-04-26 01:22:14,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.66 | bwd: 5714.76 | bwd_inner: 5701.71 | bwd_allreduce: 13.00 | step: 18.99

 17%|█▋        | 7207/41250 [17:24:39<82:26:23,  8.72s/it]
                                                          
{'loss': 0.3363, 'grad_norm': 2.2652230262756348, 'learning_rate': 3.7843454901078886e-05, 'epoch': 1.75}

 17%|█▋        | 7207/41250 [17:24:39<82:26:23,  8.72s/it][2025-04-26 01:22:23,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:22:23,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5918.73 | bwd_inner_microstep: 5708.33 | bwd_allreduce_microstep: 210.36 | step_microstep: 18.47
[2025-04-26 01:22:23,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5918.75 | bwd_inner: 5708.33 | bwd_allreduce: 210.37 | step: 18.47

 17%|█▋        | 7208/41250 [17:24:48<82:49:33,  8.76s/it]
                                                          
{'loss': 0.1698, 'grad_norm': 2.7418582439422607, 'learning_rate': 3.784274553903277e-05, 'epoch': 1.75}

 17%|█▋        | 7208/41250 [17:24:48<82:49:33,  8.76s/it][2025-04-26 01:22:31,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-26 01:22:31,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.87 | bwd_microstep: 5719.54 | bwd_inner_microstep: 5658.33 | bwd_allreduce_microstep: 61.16 | step_microstep: 18.67
[2025-04-26 01:22:31,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.87 | bwd: 5719.56 | bwd_inner: 5658.33 | bwd_allreduce: 61.18 | step: 18.67

 17%|█▋        | 7209/41250 [17:24:57<82:28:56,  8.72s/it]
                                                          
{'loss': 0.0901, 'grad_norm': 1.388259768486023, 'learning_rate': 3.784203606698963e-05, 'epoch': 1.75}

 17%|█▋        | 7209/41250 [17:24:57<82:28:56,  8.72s/it][2025-04-26 01:22:40,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 01:22:40,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.47 | bwd_microstep: 5751.53 | bwd_inner_microstep: 5694.33 | bwd_allreduce_microstep: 57.16 | step_microstep: 19.12
[2025-04-26 01:22:40,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.47 | bwd: 5751.54 | bwd_inner: 5694.33 | bwd_allreduce: 57.18 | step: 19.12

 17%|█▋        | 7210/41250 [17:25:05<82:25:19,  8.72s/it]
                                                          
{'loss': 0.1646, 'grad_norm': 1.9459211826324463, 'learning_rate': 3.7841326484953865e-05, 'epoch': 1.75}

 17%|█▋        | 7210/41250 [17:25:05<82:25:19,  8.72s/it][2025-04-26 01:22:49,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:22:49,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.10 | bwd_microstep: 5884.15 | bwd_inner_microstep: 5700.56 | bwd_allreduce_microstep: 183.56 | step_microstep: 18.73
[2025-04-26 01:22:49,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.10 | bwd: 5884.17 | bwd_inner: 5700.56 | bwd_allreduce: 183.57 | step: 18.73

 17%|█▋        | 7211/41250 [17:25:14<82:44:23,  8.75s/it]
                                                          
{'loss': 0.0433, 'grad_norm': 1.0709031820297241, 'learning_rate': 3.784061679292981e-05, 'epoch': 1.75}

 17%|█▋        | 7211/41250 [17:25:14<82:44:23,  8.75s/it][2025-04-26 01:22:58,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:22:58,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5698.89 | bwd_inner_microstep: 5654.88 | bwd_allreduce_microstep: 43.96 | step_microstep: 18.58
[2025-04-26 01:22:58,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5698.90 | bwd_inner: 5654.89 | bwd_allreduce: 43.98 | step: 18.58

 17%|█▋        | 7212/41250 [17:25:23<82:20:57,  8.71s/it]
                                                          
{'loss': 0.0675, 'grad_norm': 1.901116132736206, 'learning_rate': 3.7839906990921876e-05, 'epoch': 1.75}

 17%|█▋        | 7212/41250 [17:25:23<82:20:57,  8.71s/it][2025-04-26 01:23:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:23:06,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.78 | bwd_microstep: 5683.62 | bwd_inner_microstep: 5653.95 | bwd_allreduce_microstep: 29.63 | step_microstep: 18.87
[2025-04-26 01:23:06,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.78 | bwd: 5683.64 | bwd_inner: 5653.95 | bwd_allreduce: 29.65 | step: 18.87

 17%|█▋        | 7213/41250 [17:25:32<82:02:57,  8.68s/it]
                                                          
{'loss': 0.1643, 'grad_norm': 3.0812723636627197, 'learning_rate': 3.783919707893442e-05, 'epoch': 1.75}

 17%|█▋        | 7213/41250 [17:25:32<82:02:57,  8.68s/it][2025-04-26 01:23:15,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 01:23:15,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.94 | bwd_microstep: 5699.17 | bwd_inner_microstep: 5686.56 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.70
[2025-04-26 01:23:15,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.94 | bwd: 5699.19 | bwd_inner: 5686.56 | bwd_allreduce: 12.58 | step: 18.70

 17%|█▋        | 7214/41250 [17:25:40<81:54:06,  8.66s/it]
                                                          
{'loss': 0.1509, 'grad_norm': 1.9147319793701172, 'learning_rate': 3.783848705697182e-05, 'epoch': 1.75}

 17%|█▋        | 7214/41250 [17:25:40<81:54:06,  8.66s/it][2025-04-26 01:23:23,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:23:23,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.63 | bwd_microstep: 5710.94 | bwd_inner_microstep: 5643.23 | bwd_allreduce_microstep: 67.66 | step_microstep: 18.68
[2025-04-26 01:23:23,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.63 | bwd: 5710.95 | bwd_inner: 5643.23 | bwd_allreduce: 67.68 | step: 18.68

 17%|█▋        | 7215/41250 [17:25:49<81:47:33,  8.65s/it]
                                                          
{'loss': 0.2554, 'grad_norm': 4.168917179107666, 'learning_rate': 3.7837776925038454e-05, 'epoch': 1.75}

 17%|█▋        | 7215/41250 [17:25:49<81:47:33,  8.65s/it][2025-04-26 01:23:32,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 01:23:32,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.90 | bwd_microstep: 5686.49 | bwd_inner_microstep: 5646.03 | bwd_allreduce_microstep: 40.41 | step_microstep: 18.85
[2025-04-26 01:23:32,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.90 | bwd: 5686.51 | bwd_inner: 5646.03 | bwd_allreduce: 40.43 | step: 18.85

 17%|█▋        | 7216/41250 [17:25:57<81:38:05,  8.64s/it]
                                                          
{'loss': 0.0326, 'grad_norm': 0.5557559728622437, 'learning_rate': 3.783706668313871e-05, 'epoch': 1.75}

 17%|█▋        | 7216/41250 [17:25:57<81:38:05,  8.64s/it][2025-04-26 01:23:41,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:23:41,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.56 | bwd_microstep: 5704.26 | bwd_inner_microstep: 5649.60 | bwd_allreduce_microstep: 54.62 | step_microstep: 18.62
[2025-04-26 01:23:41,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.56 | bwd: 5704.28 | bwd_inner: 5649.60 | bwd_allreduce: 54.64 | step: 18.63

 17%|█▋        | 7217/41250 [17:26:06<81:36:21,  8.63s/it]
                                                          
{'loss': 0.2933, 'grad_norm': 3.2903192043304443, 'learning_rate': 3.7836356331276946e-05, 'epoch': 1.75}

 17%|█▋        | 7217/41250 [17:26:06<81:36:21,  8.63s/it][2025-04-26 01:23:49,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:23:49,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.19 | bwd_microstep: 5685.78 | bwd_inner_microstep: 5643.05 | bwd_allreduce_microstep: 42.69 | step_microstep: 18.49
[2025-04-26 01:23:49,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.19 | bwd: 5685.80 | bwd_inner: 5643.05 | bwd_allreduce: 42.70 | step: 18.49

 17%|█▋        | 7218/41250 [17:26:15<81:30:47,  8.62s/it]
                                                          
{'loss': 0.044, 'grad_norm': 0.8091095089912415, 'learning_rate': 3.7835645869457566e-05, 'epoch': 1.75}

 17%|█▋        | 7218/41250 [17:26:15<81:30:47,  8.62s/it][2025-04-26 01:23:58,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 01:23:58,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.63 | bwd_microstep: 5815.04 | bwd_inner_microstep: 5777.74 | bwd_allreduce_microstep: 37.25 | step_microstep: 18.56
[2025-04-26 01:23:58,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.63 | bwd: 5815.05 | bwd_inner: 5777.74 | bwd_allreduce: 37.27 | step: 18.56

 18%|█▊        | 7219/41250 [17:26:23<81:58:27,  8.67s/it]
                                                          
{'loss': 0.1313, 'grad_norm': 1.773522973060608, 'learning_rate': 3.783493529768493e-05, 'epoch': 1.75}

 18%|█▊        | 7219/41250 [17:26:23<81:58:27,  8.67s/it][2025-04-26 01:24:07,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.12
[2025-04-26 01:24:07,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5760.81 | bwd_inner_microstep: 5692.63 | bwd_allreduce_microstep: 68.13 | step_microstep: 19.25
[2025-04-26 01:24:07,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5760.82 | bwd_inner: 5692.63 | bwd_allreduce: 68.15 | step: 19.25

 18%|█▊        | 7220/41250 [17:26:32<82:01:58,  8.68s/it]
                                                          
{'loss': 0.2666, 'grad_norm': 2.0109429359436035, 'learning_rate': 3.7834224615963427e-05, 'epoch': 1.75}

 18%|█▊        | 7220/41250 [17:26:32<82:01:58,  8.68s/it][2025-04-26 01:24:15,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:24:15,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.36 | bwd_microstep: 5692.49 | bwd_inner_microstep: 5643.76 | bwd_allreduce_microstep: 48.69 | step_microstep: 18.76
[2025-04-26 01:24:15,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.36 | bwd: 5692.50 | bwd_inner: 5643.76 | bwd_allreduce: 48.70 | step: 18.76

 18%|█▊        | 7221/41250 [17:26:41<81:48:29,  8.65s/it]
                                                          
{'loss': 0.1769, 'grad_norm': 1.8771657943725586, 'learning_rate': 3.7833513824297436e-05, 'epoch': 1.75}

 18%|█▊        | 7221/41250 [17:26:41<81:48:29,  8.65s/it][2025-04-26 01:24:24,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:24:24,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5765.01 | bwd_inner_microstep: 5677.33 | bwd_allreduce_microstep: 87.63 | step_microstep: 18.65
[2025-04-26 01:24:24,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5765.02 | bwd_inner: 5677.33 | bwd_allreduce: 87.65 | step: 18.66

 18%|█▊        | 7222/41250 [17:26:49<81:54:20,  8.67s/it]
                                                          
{'loss': 0.0272, 'grad_norm': 0.4410299062728882, 'learning_rate': 3.7832802922691347e-05, 'epoch': 1.75}

 18%|█▊        | 7222/41250 [17:26:49<81:54:20,  8.67s/it][2025-04-26 01:24:33,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 01:24:33,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.84 | bwd_microstep: 5700.69 | bwd_inner_microstep: 5687.96 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.95
[2025-04-26 01:24:33,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.84 | bwd: 5700.71 | bwd_inner: 5687.96 | bwd_allreduce: 12.71 | step: 18.95

 18%|█▊        | 7223/41250 [17:26:58<81:48:46,  8.66s/it]
                                                          
{'loss': 0.1543, 'grad_norm': 0.6407178044319153, 'learning_rate': 3.783209191114953e-05, 'epoch': 1.75}

 18%|█▊        | 7223/41250 [17:26:58<81:48:46,  8.66s/it][2025-04-26 01:24:41,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:24:41,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.95 | bwd_microstep: 5706.12 | bwd_inner_microstep: 5693.26 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.69
[2025-04-26 01:24:41,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.95 | bwd: 5706.13 | bwd_inner: 5693.26 | bwd_allreduce: 12.83 | step: 18.69

 18%|█▊        | 7224/41250 [17:27:07<81:47:07,  8.65s/it]
                                                          
{'loss': 0.1339, 'grad_norm': 1.3068640232086182, 'learning_rate': 3.783138078967637e-05, 'epoch': 1.75}

 18%|█▊        | 7224/41250 [17:27:07<81:47:07,  8.65s/it][2025-04-26 01:24:50,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 01:24:50,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.34 | bwd_microstep: 5747.18 | bwd_inner_microstep: 5652.38 | bwd_allreduce_microstep: 94.76 | step_microstep: 18.84
[2025-04-26 01:24:50,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.34 | bwd: 5747.19 | bwd_inner: 5652.38 | bwd_allreduce: 94.78 | step: 18.84

 18%|█▊        | 7225/41250 [17:27:15<81:48:41,  8.66s/it]
                                                          
{'loss': 0.2366, 'grad_norm': 6.417294979095459, 'learning_rate': 3.783066955827626e-05, 'epoch': 1.75}

 18%|█▊        | 7225/41250 [17:27:15<81:48:41,  8.66s/it][2025-04-26 01:24:59,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:24:59,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.97 | bwd_microstep: 5742.08 | bwd_inner_microstep: 5640.08 | bwd_allreduce_microstep: 101.95 | step_microstep: 18.53
[2025-04-26 01:24:59,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.97 | bwd: 5742.09 | bwd_inner: 5640.08 | bwd_allreduce: 101.97 | step: 18.53

 18%|█▊        | 7226/41250 [17:27:24<81:47:15,  8.65s/it]
                                                          
{'loss': 0.1425, 'grad_norm': 3.5140020847320557, 'learning_rate': 3.782995821695358e-05, 'epoch': 1.75}

 18%|█▊        | 7226/41250 [17:27:24<81:47:15,  8.65s/it][2025-04-26 01:25:07,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.14 | optimizer_step: 1.05
[2025-04-26 01:25:07,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.08 | bwd_microstep: 5751.60 | bwd_inner_microstep: 5683.59 | bwd_allreduce_microstep: 67.94 | step_microstep: 20.32
[2025-04-26 01:25:07,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.08 | bwd: 5751.61 | bwd_inner: 5683.59 | bwd_allreduce: 67.97 | step: 20.32

 18%|█▊        | 7227/41250 [17:27:33<81:52:27,  8.66s/it]
                                                          
{'loss': 0.1373, 'grad_norm': 2.6805012226104736, 'learning_rate': 3.7829246765712715e-05, 'epoch': 1.75}

 18%|█▊        | 7227/41250 [17:27:33<81:52:27,  8.66s/it][2025-04-26 01:25:16,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 01:25:16,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.05 | bwd_microstep: 5697.85 | bwd_inner_microstep: 5638.96 | bwd_allreduce_microstep: 58.84 | step_microstep: 18.56
[2025-04-26 01:25:16,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.05 | bwd: 5697.86 | bwd_inner: 5638.96 | bwd_allreduce: 58.86 | step: 18.56

 18%|█▊        | 7228/41250 [17:27:41<81:41:05,  8.64s/it]
                                                          
{'loss': 0.1674, 'grad_norm': 1.8439584970474243, 'learning_rate': 3.7828535204558045e-05, 'epoch': 1.75}

 18%|█▊        | 7228/41250 [17:27:41<81:41:05,  8.64s/it][2025-04-26 01:25:25,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:25:25,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.81 | bwd_microstep: 5831.44 | bwd_inner_microstep: 5706.68 | bwd_allreduce_microstep: 124.71 | step_microstep: 18.65
[2025-04-26 01:25:25,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.81 | bwd: 5831.45 | bwd_inner: 5706.68 | bwd_allreduce: 124.73 | step: 18.66

 18%|█▊        | 7229/41250 [17:27:50<82:01:30,  8.68s/it]
                                                          
{'loss': 0.0253, 'grad_norm': 0.32203108072280884, 'learning_rate': 3.782782353349397e-05, 'epoch': 1.75}

 18%|█▊        | 7229/41250 [17:27:50<82:01:30,  8.68s/it][2025-04-26 01:25:33,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 01:25:33,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.78 | bwd_microstep: 5695.96 | bwd_inner_microstep: 5646.78 | bwd_allreduce_microstep: 49.13 | step_microstep: 18.92
[2025-04-26 01:25:33,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.78 | bwd: 5695.98 | bwd_inner: 5646.78 | bwd_allreduce: 49.15 | step: 18.92

 18%|█▊        | 7230/41250 [17:27:59<81:47:40,  8.66s/it]
                                                          
{'loss': 0.2065, 'grad_norm': 2.0850822925567627, 'learning_rate': 3.7827111752524866e-05, 'epoch': 1.75}

 18%|█▊        | 7230/41250 [17:27:59<81:47:40,  8.66s/it][2025-04-26 01:25:42,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 01:25:42,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5696.27 | bwd_inner_microstep: 5639.38 | bwd_allreduce_microstep: 56.84 | step_microstep: 18.71
[2025-04-26 01:25:42,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5696.28 | bwd_inner: 5639.38 | bwd_allreduce: 56.86 | step: 18.72

 18%|█▊        | 7231/41250 [17:28:07<81:37:58,  8.64s/it]
                                                          
{'loss': 0.1069, 'grad_norm': 2.3141162395477295, 'learning_rate': 3.782639986165512e-05, 'epoch': 1.75}

 18%|█▊        | 7231/41250 [17:28:07<81:37:58,  8.64s/it][2025-04-26 01:25:51,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:25:51,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.31 | bwd_microstep: 5853.20 | bwd_inner_microstep: 5699.90 | bwd_allreduce_microstep: 153.26 | step_microstep: 18.90
[2025-04-26 01:25:51,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.31 | bwd: 5853.21 | bwd_inner: 5699.90 | bwd_allreduce: 153.28 | step: 18.91

 18%|█▊        | 7232/41250 [17:28:16<82:02:58,  8.68s/it]
                                                          
{'loss': 0.1514, 'grad_norm': 2.2008824348449707, 'learning_rate': 3.782568786088913e-05, 'epoch': 1.75}

 18%|█▊        | 7232/41250 [17:28:16<82:02:58,  8.68s/it][2025-04-26 01:25:59,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:25:59,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.52 | bwd_microstep: 5764.36 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 115.34 | step_microstep: 18.81
[2025-04-26 01:25:59,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.52 | bwd: 5764.38 | bwd_inner: 5648.98 | bwd_allreduce: 115.36 | step: 18.81

 18%|█▊        | 7233/41250 [17:28:25<82:01:38,  8.68s/it]
                                                          
{'loss': 0.2183, 'grad_norm': 3.8612759113311768, 'learning_rate': 3.782497575023128e-05, 'epoch': 1.75}

 18%|█▊        | 7233/41250 [17:28:25<82:01:38,  8.68s/it][2025-04-26 01:26:08,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.34 | optimizer_step: 1.05
[2025-04-26 01:26:08,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.27 | bwd_microstep: 5758.64 | bwd_inner_microstep: 5628.04 | bwd_allreduce_microstep: 130.53 | step_microstep: 20.30
[2025-04-26 01:26:08,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.27 | bwd: 5758.65 | bwd_inner: 5628.04 | bwd_allreduce: 130.57 | step: 20.30

 18%|█▊        | 7234/41250 [17:28:33<81:58:15,  8.68s/it]
                                                          
{'loss': 0.1779, 'grad_norm': 3.2248504161834717, 'learning_rate': 3.782426352968596e-05, 'epoch': 1.75}

 18%|█▊        | 7234/41250 [17:28:33<81:58:15,  8.68s/it][2025-04-26 01:26:17,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 01:26:17,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.73 | bwd_microstep: 5721.75 | bwd_inner_microstep: 5673.22 | bwd_allreduce_microstep: 48.48 | step_microstep: 18.79
[2025-04-26 01:26:17,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.73 | bwd: 5721.76 | bwd_inner: 5673.22 | bwd_allreduce: 48.50 | step: 18.79

 18%|█▊        | 7235/41250 [17:28:42<81:53:48,  8.67s/it]
                                                          
{'loss': 0.1313, 'grad_norm': 1.7361140251159668, 'learning_rate': 3.7823551199257556e-05, 'epoch': 1.75}

 18%|█▊        | 7235/41250 [17:28:42<81:53:48,  8.67s/it][2025-04-26 01:26:25,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 01:26:25,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5763.44 | bwd_inner_microstep: 5657.89 | bwd_allreduce_microstep: 105.50 | step_microstep: 18.99
[2025-04-26 01:26:25,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5763.45 | bwd_inner: 5657.89 | bwd_allreduce: 105.52 | step: 18.99

 18%|█▊        | 7236/41250 [17:28:51<81:55:03,  8.67s/it]
                                                          
{'loss': 0.239, 'grad_norm': 2.5363869667053223, 'learning_rate': 3.782283875895047e-05, 'epoch': 1.75}

 18%|█▊        | 7236/41250 [17:28:51<81:55:03,  8.67s/it][2025-04-26 01:26:34,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:26:34,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.66 | bwd_microstep: 5689.04 | bwd_inner_microstep: 5651.74 | bwd_allreduce_microstep: 37.25 | step_microstep: 18.53
[2025-04-26 01:26:34,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.66 | bwd: 5689.05 | bwd_inner: 5651.74 | bwd_allreduce: 37.27 | step: 18.54

 18%|█▊        | 7237/41250 [17:28:59<81:42:19,  8.65s/it]
                                                          
{'loss': 0.0765, 'grad_norm': 1.0135115385055542, 'learning_rate': 3.782212620876909e-05, 'epoch': 1.75}

 18%|█▊        | 7237/41250 [17:28:59<81:42:19,  8.65s/it][2025-04-26 01:26:43,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 01:26:43,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.82 | bwd_microstep: 5693.88 | bwd_inner_microstep: 5681.34 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.63
[2025-04-26 01:26:43,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.82 | bwd: 5693.89 | bwd_inner: 5681.34 | bwd_allreduce: 12.51 | step: 18.64

 18%|█▊        | 7238/41250 [17:29:08<81:36:51,  8.64s/it]
                                                          
{'loss': 0.1315, 'grad_norm': 2.0317726135253906, 'learning_rate': 3.78214135487178e-05, 'epoch': 1.75}

 18%|█▊        | 7238/41250 [17:29:08<81:36:51,  8.64s/it][2025-04-26 01:26:51,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:26:51,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5757.98 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 109.48 | step_microstep: 18.51
[2025-04-26 01:26:51,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5757.99 | bwd_inner: 5648.45 | bwd_allreduce: 109.50 | step: 18.51

 18%|█▊        | 7239/41250 [17:29:17<81:41:32,  8.65s/it]
                                                          
{'loss': 0.0919, 'grad_norm': 0.5955415964126587, 'learning_rate': 3.782070077880101e-05, 'epoch': 1.75}

 18%|█▊        | 7239/41250 [17:29:17<81:41:32,  8.65s/it][2025-04-26 01:27:00,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:27:00,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.75 | bwd_microstep: 5712.98 | bwd_inner_microstep: 5700.54 | bwd_allreduce_microstep: 12.40 | step_microstep: 18.64
[2025-04-26 01:27:00,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.75 | bwd: 5713.00 | bwd_inner: 5700.54 | bwd_allreduce: 12.42 | step: 18.64

 18%|█▊        | 7240/41250 [17:29:25<81:39:49,  8.64s/it]
                                                          
{'loss': 0.2338, 'grad_norm': 2.804048776626587, 'learning_rate': 3.7819987899023096e-05, 'epoch': 1.76}

 18%|█▊        | 7240/41250 [17:29:25<81:39:49,  8.64s/it][2025-04-26 01:27:08,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:27:08,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.14 | bwd_microstep: 5754.95 | bwd_inner_microstep: 5646.53 | bwd_allreduce_microstep: 108.38 | step_microstep: 18.42
[2025-04-26 01:27:08,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.14 | bwd: 5754.97 | bwd_inner: 5646.53 | bwd_allreduce: 108.40 | step: 18.42

 18%|█▊        | 7241/41250 [17:29:34<81:42:19,  8.65s/it]
                                                          
{'loss': 0.219, 'grad_norm': 2.261017322540283, 'learning_rate': 3.781927490938847e-05, 'epoch': 1.76}

 18%|█▊        | 7241/41250 [17:29:34<81:42:19,  8.65s/it][2025-04-26 01:27:17,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:27:17,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.69 | bwd_microstep: 5698.10 | bwd_inner_microstep: 5685.49 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.67
[2025-04-26 01:27:17,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.69 | bwd: 5698.11 | bwd_inner: 5685.49 | bwd_allreduce: 12.59 | step: 18.67

 18%|█▊        | 7242/41250 [17:29:42<81:38:12,  8.64s/it]
                                                          
{'loss': 0.2366, 'grad_norm': 3.3679630756378174, 'learning_rate': 3.7818561809901514e-05, 'epoch': 1.76}

 18%|█▊        | 7242/41250 [17:29:42<81:38:12,  8.64s/it][2025-04-26 01:27:26,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 01:27:26,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5902.85 | bwd_inner_microstep: 5666.35 | bwd_allreduce_microstep: 236.46 | step_microstep: 19.11
[2025-04-26 01:27:26,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5902.86 | bwd_inner: 5666.35 | bwd_allreduce: 236.48 | step: 19.11

 18%|█▊        | 7243/41250 [17:29:51<82:07:27,  8.69s/it]
                                                          
{'loss': 0.126, 'grad_norm': 2.1744256019592285, 'learning_rate': 3.7817848600566635e-05, 'epoch': 1.76}

 18%|█▊        | 7243/41250 [17:29:51<82:07:27,  8.69s/it][2025-04-26 01:27:35,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 01:27:35,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.68 | bwd_microstep: 5753.80 | bwd_inner_microstep: 5682.78 | bwd_allreduce_microstep: 70.97 | step_microstep: 18.57
[2025-04-26 01:27:35,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.68 | bwd: 5753.81 | bwd_inner: 5682.78 | bwd_allreduce: 70.98 | step: 18.57

 18%|█▊        | 7244/41250 [17:30:00<82:05:16,  8.69s/it]
                                                          
{'loss': 0.2755, 'grad_norm': 1.421707272529602, 'learning_rate': 3.781713528138822e-05, 'epoch': 1.76}

 18%|█▊        | 7244/41250 [17:30:00<82:05:16,  8.69s/it][2025-04-26 01:27:43,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 01:27:43,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.12 | bwd_microstep: 5737.94 | bwd_inner_microstep: 5718.35 | bwd_allreduce_microstep: 19.54 | step_microstep: 19.01
[2025-04-26 01:27:43,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.12 | bwd: 5737.95 | bwd_inner: 5718.35 | bwd_allreduce: 19.56 | step: 19.01

 18%|█▊        | 7245/41250 [17:30:09<82:04:13,  8.69s/it]
                                                          
{'loss': 0.1794, 'grad_norm': 2.641861915588379, 'learning_rate': 3.781642185237067e-05, 'epoch': 1.76}

 18%|█▊        | 7245/41250 [17:30:09<82:04:13,  8.69s/it][2025-04-26 01:27:52,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:27:52,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.83 | bwd_microstep: 5784.00 | bwd_inner_microstep: 5771.16 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.81
[2025-04-26 01:27:52,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.83 | bwd: 5784.01 | bwd_inner: 5771.16 | bwd_allreduce: 12.82 | step: 18.81

 18%|█▊        | 7246/41250 [17:30:17<82:15:25,  8.71s/it]
                                                          
{'loss': 0.0448, 'grad_norm': 1.0508018732070923, 'learning_rate': 3.7815708313518376e-05, 'epoch': 1.76}

 18%|█▊        | 7246/41250 [17:30:17<82:15:25,  8.71s/it][2025-04-26 01:28:01,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:28:01,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.11 | bwd_microstep: 5700.50 | bwd_inner_microstep: 5663.77 | bwd_allreduce_microstep: 36.68 | step_microstep: 18.94
[2025-04-26 01:28:01,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.11 | bwd: 5700.52 | bwd_inner: 5663.78 | bwd_allreduce: 36.70 | step: 18.94

 18%|█▊        | 7247/41250 [17:30:26<81:58:45,  8.68s/it]
                                                          
{'loss': 0.1394, 'grad_norm': 1.2896174192428589, 'learning_rate': 3.781499466483576e-05, 'epoch': 1.76}

 18%|█▊        | 7247/41250 [17:30:26<81:58:45,  8.68s/it][2025-04-26 01:28:09,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-26 01:28:09,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.26 | bwd_microstep: 5727.19 | bwd_inner_microstep: 5714.18 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.73
[2025-04-26 01:28:09,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.26 | bwd: 5727.20 | bwd_inner: 5714.18 | bwd_allreduce: 12.98 | step: 18.73

 18%|█▊        | 7248/41250 [17:30:35<81:55:19,  8.67s/it]
                                                          
{'loss': 0.1044, 'grad_norm': 1.1864213943481445, 'learning_rate': 3.781428090632719e-05, 'epoch': 1.76}

 18%|█▊        | 7248/41250 [17:30:35<81:55:19,  8.67s/it][2025-04-26 01:28:18,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:28:18,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.83 | bwd_microstep: 5908.80 | bwd_inner_microstep: 5654.85 | bwd_allreduce_microstep: 253.91 | step_microstep: 18.57
[2025-04-26 01:28:18,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.83 | bwd: 5908.81 | bwd_inner: 5654.85 | bwd_allreduce: 253.93 | step: 18.57

 18%|█▊        | 7249/41250 [17:30:43<82:19:37,  8.72s/it]
                                                          
{'loss': 0.2809, 'grad_norm': 4.1288862228393555, 'learning_rate': 3.7813567037997095e-05, 'epoch': 1.76}

 18%|█▊        | 7249/41250 [17:30:43<82:19:37,  8.72s/it][2025-04-26 01:28:27,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:28:27,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.51 | bwd_microstep: 5764.79 | bwd_inner_microstep: 5697.28 | bwd_allreduce_microstep: 67.47 | step_microstep: 18.63
[2025-04-26 01:28:27,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.51 | bwd: 5764.80 | bwd_inner: 5697.28 | bwd_allreduce: 67.49 | step: 18.63

 18%|█▊        | 7250/41250 [17:30:52<82:15:00,  8.71s/it]
                                                          
{'loss': 0.0629, 'grad_norm': 0.8651026487350464, 'learning_rate': 3.7812853059849854e-05, 'epoch': 1.76}

 18%|█▊        | 7250/41250 [17:30:52<82:15:00,  8.71s/it][2025-04-26 01:28:35,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:28:35,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.06 | bwd_microstep: 5709.76 | bwd_inner_microstep: 5697.19 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.60
[2025-04-26 01:28:35,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.06 | bwd: 5709.77 | bwd_inner: 5697.19 | bwd_allreduce: 12.54 | step: 18.60

 18%|█▊        | 7251/41250 [17:31:01<82:04:54,  8.69s/it]
                                                          
{'loss': 0.1122, 'grad_norm': 1.4043116569519043, 'learning_rate': 3.781213897188989e-05, 'epoch': 1.76}

 18%|█▊        | 7251/41250 [17:31:01<82:04:54,  8.69s/it][2025-04-26 01:28:44,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 01:28:44,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.82 | bwd_microstep: 5707.30 | bwd_inner_microstep: 5662.77 | bwd_allreduce_microstep: 44.48 | step_microstep: 19.19
[2025-04-26 01:28:44,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.82 | bwd: 5707.31 | bwd_inner: 5662.77 | bwd_allreduce: 44.49 | step: 19.20

 18%|█▊        | 7252/41250 [17:31:09<81:53:19,  8.67s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 0.8603627681732178, 'learning_rate': 3.781142477412158e-05, 'epoch': 1.76}

 18%|█▊        | 7252/41250 [17:31:09<81:53:19,  8.67s/it][2025-04-26 01:28:53,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:28:53,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5777.83 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 75.86 | step_microstep: 18.39
[2025-04-26 01:28:53,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5777.85 | bwd_inner: 5701.94 | bwd_allreduce: 75.87 | step: 18.40

 18%|█▊        | 7253/41250 [17:31:18<82:00:13,  8.68s/it]
                                                          
{'loss': 0.337, 'grad_norm': 3.7154672145843506, 'learning_rate': 3.781071046654934e-05, 'epoch': 1.76}

 18%|█▊        | 7253/41250 [17:31:18<82:00:13,  8.68s/it][2025-04-26 01:29:02,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 01:29:02,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.69 | bwd_microstep: 5785.74 | bwd_inner_microstep: 5773.01 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.12
[2025-04-26 01:29:02,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.69 | bwd: 5785.76 | bwd_inner: 5773.01 | bwd_allreduce: 12.71 | step: 19.12

 18%|█▊        | 7254/41250 [17:31:27<82:12:47,  8.71s/it]
                                                          
{'loss': 0.0847, 'grad_norm': 0.6741653680801392, 'learning_rate': 3.780999604917758e-05, 'epoch': 1.76}

 18%|█▊        | 7254/41250 [17:31:27<82:12:47,  8.71s/it][2025-04-26 01:29:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 01:29:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.64 | bwd_microstep: 5793.30 | bwd_inner_microstep: 5780.20 | bwd_allreduce_microstep: 13.06 | step_microstep: 19.12
[2025-04-26 01:29:10,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.65 | bwd: 5793.32 | bwd_inner: 5780.20 | bwd_allreduce: 13.08 | step: 19.12

 18%|█▊        | 7255/41250 [17:31:36<82:24:02,  8.73s/it]
                                                          
{'loss': 0.229, 'grad_norm': 4.042088508605957, 'learning_rate': 3.780928152201069e-05, 'epoch': 1.76}

 18%|█▊        | 7255/41250 [17:31:36<82:24:02,  8.73s/it][2025-04-26 01:29:19,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:29:19,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.77 | bwd_microstep: 5772.27 | bwd_inner_microstep: 5654.20 | bwd_allreduce_microstep: 118.03 | step_microstep: 18.80
[2025-04-26 01:29:19,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.77 | bwd: 5772.29 | bwd_inner: 5654.20 | bwd_allreduce: 118.04 | step: 18.80

 18%|█▊        | 7256/41250 [17:31:44<82:18:08,  8.72s/it]
                                                          
{'loss': 0.115, 'grad_norm': 3.5417463779449463, 'learning_rate': 3.780856688505309e-05, 'epoch': 1.76}

 18%|█▊        | 7256/41250 [17:31:44<82:18:08,  8.72s/it][2025-04-26 01:29:28,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.14 | optimizer_step: 1.06
[2025-04-26 01:29:28,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.98 | bwd_microstep: 5694.60 | bwd_inner_microstep: 5656.58 | bwd_allreduce_microstep: 37.96 | step_microstep: 20.20
[2025-04-26 01:29:28,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5694.62 | bwd_inner: 5656.58 | bwd_allreduce: 37.99 | step: 20.21

 18%|█▊        | 7257/41250 [17:31:53<81:59:56,  8.68s/it]
                                                          
{'loss': 0.0751, 'grad_norm': 1.1243584156036377, 'learning_rate': 3.7807852138309174e-05, 'epoch': 1.76}

 18%|█▊        | 7257/41250 [17:31:53<81:59:56,  8.68s/it][2025-04-26 01:29:36,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 01:29:36,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.91 | bwd_microstep: 5797.46 | bwd_inner_microstep: 5650.80 | bwd_allreduce_microstep: 146.61 | step_microstep: 18.94
[2025-04-26 01:29:36,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.91 | bwd: 5797.47 | bwd_inner: 5650.80 | bwd_allreduce: 146.62 | step: 18.94

 18%|█▊        | 7258/41250 [17:32:02<82:03:43,  8.69s/it]
                                                          
{'loss': 0.1328, 'grad_norm': 2.8122713565826416, 'learning_rate': 3.780713728178335e-05, 'epoch': 1.76}

 18%|█▊        | 7258/41250 [17:32:02<82:03:43,  8.69s/it][2025-04-26 01:29:45,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:29:45,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.62 | bwd_microstep: 5741.75 | bwd_inner_microstep: 5701.73 | bwd_allreduce_microstep: 39.97 | step_microstep: 18.69
[2025-04-26 01:29:45,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.62 | bwd: 5741.76 | bwd_inner: 5701.73 | bwd_allreduce: 39.99 | step: 18.69

 18%|█▊        | 7259/41250 [17:32:10<82:00:29,  8.69s/it]
                                                          
{'loss': 0.2198, 'grad_norm': 2.5759801864624023, 'learning_rate': 3.7806422315480035e-05, 'epoch': 1.76}

 18%|█▊        | 7259/41250 [17:32:10<82:00:29,  8.69s/it][2025-04-26 01:29:54,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-26 01:29:54,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.50 | bwd_microstep: 5787.21 | bwd_inner_microstep: 5653.17 | bwd_allreduce_microstep: 134.00 | step_microstep: 18.63
[2025-04-26 01:29:54,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.50 | bwd: 5787.22 | bwd_inner: 5653.16 | bwd_allreduce: 134.01 | step: 18.63

 18%|█▊        | 7260/41250 [17:32:19<82:03:12,  8.69s/it]
                                                          
{'loss': 0.1059, 'grad_norm': 2.3854966163635254, 'learning_rate': 3.780570723940362e-05, 'epoch': 1.76}

 18%|█▊        | 7260/41250 [17:32:19<82:03:12,  8.69s/it][2025-04-26 01:30:02,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 1.00
[2025-04-26 01:30:02,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.82 | bwd_microstep: 5753.13 | bwd_inner_microstep: 5705.84 | bwd_allreduce_microstep: 47.25 | step_microstep: 18.54
[2025-04-26 01:30:02,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.82 | bwd: 5753.15 | bwd_inner: 5705.84 | bwd_allreduce: 47.27 | step: 18.54

 18%|█▊        | 7261/41250 [17:32:28<82:01:20,  8.69s/it]
                                                          
{'loss': 0.1345, 'grad_norm': 2.104391098022461, 'learning_rate': 3.780499205355852e-05, 'epoch': 1.76}

 18%|█▊        | 7261/41250 [17:32:28<82:01:20,  8.69s/it][2025-04-26 01:30:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-26 01:30:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.18 | bwd_microstep: 5705.30 | bwd_inner_microstep: 5653.88 | bwd_allreduce_microstep: 51.38 | step_microstep: 19.09
[2025-04-26 01:30:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.18 | bwd: 5705.31 | bwd_inner: 5653.88 | bwd_allreduce: 51.39 | step: 19.09

 18%|█▊        | 7262/41250 [17:32:36<81:49:51,  8.67s/it]
                                                          
{'loss': 0.1739, 'grad_norm': 1.7092511653900146, 'learning_rate': 3.780427675794916e-05, 'epoch': 1.76}

 18%|█▊        | 7262/41250 [17:32:36<81:49:51,  8.67s/it][2025-04-26 01:30:20,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 01:30:20,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.82 | bwd_microstep: 5736.52 | bwd_inner_microstep: 5688.89 | bwd_allreduce_microstep: 47.59 | step_microstep: 18.72
[2025-04-26 01:30:20,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.83 | bwd: 5736.53 | bwd_inner: 5688.89 | bwd_allreduce: 47.61 | step: 18.72

 18%|█▊        | 7263/41250 [17:32:45<81:49:44,  8.67s/it]
                                                          
{'loss': 0.3596, 'grad_norm': 4.10505485534668, 'learning_rate': 3.780356135257992e-05, 'epoch': 1.76}

 18%|█▊        | 7263/41250 [17:32:45<81:49:44,  8.67s/it][2025-04-26 01:30:28,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:30:28,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.29 | bwd_microstep: 5690.83 | bwd_inner_microstep: 5678.11 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.95
[2025-04-26 01:30:28,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.29 | bwd: 5690.84 | bwd_inner: 5678.11 | bwd_allreduce: 12.68 | step: 18.96

 18%|█▊        | 7264/41250 [17:32:54<81:41:20,  8.65s/it]
                                                          
{'loss': 0.0583, 'grad_norm': 1.0936737060546875, 'learning_rate': 3.780284583745524e-05, 'epoch': 1.76}

 18%|█▊        | 7264/41250 [17:32:54<81:41:20,  8.65s/it][2025-04-26 01:30:37,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-26 01:30:37,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.59 | bwd_microstep: 5729.80 | bwd_inner_microstep: 5682.50 | bwd_allreduce_microstep: 47.26 | step_microstep: 18.72
[2025-04-26 01:30:37,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.59 | bwd: 5729.81 | bwd_inner: 5682.50 | bwd_allreduce: 47.27 | step: 18.72

 18%|█▊        | 7265/41250 [17:33:02<81:41:01,  8.65s/it]
                                                          
{'loss': 0.3102, 'grad_norm': 3.594799518585205, 'learning_rate': 3.780213021257951e-05, 'epoch': 1.76}

 18%|█▊        | 7265/41250 [17:33:02<81:41:01,  8.65s/it][2025-04-26 01:30:46,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:30:46,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.72 | bwd_microstep: 5698.25 | bwd_inner_microstep: 5638.88 | bwd_allreduce_microstep: 59.32 | step_microstep: 18.69
[2025-04-26 01:30:46,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.72 | bwd: 5698.26 | bwd_inner: 5638.88 | bwd_allreduce: 59.34 | step: 18.69

 18%|█▊        | 7266/41250 [17:33:11<81:32:05,  8.64s/it]
                                                          
{'loss': 0.0863, 'grad_norm': 1.1935765743255615, 'learning_rate': 3.780141447795715e-05, 'epoch': 1.76}

 18%|█▊        | 7266/41250 [17:33:11<81:32:05,  8.64s/it][2025-04-26 01:30:54,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 01:30:54,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5772.14 | bwd_inner_microstep: 5633.21 | bwd_allreduce_microstep: 138.89 | step_microstep: 19.01
[2025-04-26 01:30:54,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5772.15 | bwd_inner: 5633.21 | bwd_allreduce: 138.90 | step: 19.01

 18%|█▊        | 7267/41250 [17:33:20<81:39:15,  8.65s/it]
                                                          
{'loss': 0.1852, 'grad_norm': 1.5076017379760742, 'learning_rate': 3.7800698633592564e-05, 'epoch': 1.76}

 18%|█▊        | 7267/41250 [17:33:20<81:39:15,  8.65s/it][2025-04-26 01:31:03,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-26 01:31:03,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.28 | bwd_microstep: 5885.70 | bwd_inner_microstep: 5655.62 | bwd_allreduce_microstep: 230.04 | step_microstep: 18.99
[2025-04-26 01:31:03,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.28 | bwd: 5885.72 | bwd_inner: 5655.62 | bwd_allreduce: 230.06 | step: 18.99

 18%|█▊        | 7268/41250 [17:33:28<82:05:01,  8.70s/it]
                                                          
{'loss': 0.3025, 'grad_norm': 2.2232553958892822, 'learning_rate': 3.779998267949018e-05, 'epoch': 1.76}

 18%|█▊        | 7268/41250 [17:33:28<82:05:01,  8.70s/it][2025-04-26 01:31:12,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:31:12,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.33 | bwd_microstep: 5724.64 | bwd_inner_microstep: 5695.80 | bwd_allreduce_microstep: 28.81 | step_microstep: 18.58
[2025-04-26 01:31:12,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.34 | bwd: 5724.66 | bwd_inner: 5695.80 | bwd_allreduce: 28.82 | step: 18.58

 18%|█▊        | 7269/41250 [17:33:37<81:59:52,  8.69s/it]
                                                          
{'loss': 0.1647, 'grad_norm': 1.7867587804794312, 'learning_rate': 3.779926661565441e-05, 'epoch': 1.76}

 18%|█▊        | 7269/41250 [17:33:37<81:59:52,  8.69s/it][2025-04-26 01:31:20,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 01:31:20,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.51 | bwd_microstep: 5715.72 | bwd_inner_microstep: 5678.85 | bwd_allreduce_microstep: 36.83 | step_microstep: 18.81
[2025-04-26 01:31:20,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.51 | bwd: 5715.73 | bwd_inner: 5678.85 | bwd_allreduce: 36.85 | step: 18.81

 18%|█▊        | 7270/41250 [17:33:46<81:53:50,  8.68s/it]
                                                          
{'loss': 0.0374, 'grad_norm': 0.8910180330276489, 'learning_rate': 3.779855044208965e-05, 'epoch': 1.76}

 18%|█▊        | 7270/41250 [17:33:46<81:53:50,  8.68s/it][2025-04-26 01:31:29,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:31:29,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2920.97 | bwd_microstep: 5856.39 | bwd_inner_microstep: 5843.60 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.59
[2025-04-26 01:31:29,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2920.97 | bwd: 5856.40 | bwd_inner: 5843.60 | bwd_allreduce: 12.76 | step: 18.59

 18%|█▊        | 7271/41250 [17:33:55<82:25:09,  8.73s/it]
                                                          
{'loss': 0.1561, 'grad_norm': 3.9490089416503906, 'learning_rate': 3.779783415880033e-05, 'epoch': 1.76}

 18%|█▊        | 7271/41250 [17:33:55<82:25:09,  8.73s/it][2025-04-26 01:31:38,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:31:38,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5711.67 | bwd_inner_microstep: 5694.28 | bwd_allreduce_microstep: 17.33 | step_microstep: 18.74
[2025-04-26 01:31:38,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5711.69 | bwd_inner: 5694.28 | bwd_allreduce: 17.36 | step: 18.74

 18%|█▊        | 7272/41250 [17:34:03<82:11:02,  8.71s/it]
                                                          
{'loss': 0.135, 'grad_norm': 1.6722702980041504, 'learning_rate': 3.7797117765790864e-05, 'epoch': 1.76}

 18%|█▊        | 7272/41250 [17:34:03<82:11:02,  8.71s/it][2025-04-26 01:31:47,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.07 | optimizer_step: 1.24
[2025-04-26 01:31:47,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.57 | bwd_microstep: 5713.74 | bwd_inner_microstep: 5647.09 | bwd_allreduce_microstep: 66.59 | step_microstep: 19.79
[2025-04-26 01:31:47,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.57 | bwd: 5713.75 | bwd_inner: 5647.09 | bwd_allreduce: 66.62 | step: 19.79

 18%|█▊        | 7273/41250 [17:34:12<81:58:59,  8.69s/it]
                                                          
{'loss': 0.1678, 'grad_norm': 1.601065993309021, 'learning_rate': 3.779640126306567e-05, 'epoch': 1.76}

 18%|█▊        | 7273/41250 [17:34:12<81:58:59,  8.69s/it][2025-04-26 01:31:55,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:31:55,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.73 | bwd_microstep: 5684.89 | bwd_inner_microstep: 5636.65 | bwd_allreduce_microstep: 48.20 | step_microstep: 18.44
[2025-04-26 01:31:55,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.73 | bwd: 5684.90 | bwd_inner: 5636.65 | bwd_allreduce: 48.21 | step: 18.44

 18%|█▊        | 7274/41250 [17:34:20<81:42:28,  8.66s/it]
                                                          
{'loss': 0.1844, 'grad_norm': 2.085780382156372, 'learning_rate': 3.779568465062916e-05, 'epoch': 1.76}

 18%|█▊        | 7274/41250 [17:34:20<81:42:28,  8.66s/it][2025-04-26 01:32:04,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:32:04,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.09 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5637.39 | bwd_allreduce_microstep: 125.97 | step_microstep: 18.57
[2025-04-26 01:32:04,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.09 | bwd: 5763.42 | bwd_inner: 5637.39 | bwd_allreduce: 125.98 | step: 18.58

 18%|█▊        | 7275/41250 [17:34:29<81:44:29,  8.66s/it]
                                                          
{'loss': 0.1443, 'grad_norm': 1.8210422992706299, 'learning_rate': 3.779496792848575e-05, 'epoch': 1.76}

 18%|█▊        | 7275/41250 [17:34:29<81:44:29,  8.66s/it][2025-04-26 01:32:12,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:32:12,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.48 | bwd_microstep: 5765.08 | bwd_inner_microstep: 5649.92 | bwd_allreduce_microstep: 115.12 | step_microstep: 18.85
[2025-04-26 01:32:12,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.48 | bwd: 5765.10 | bwd_inner: 5649.92 | bwd_allreduce: 115.14 | step: 18.86

 18%|█▊        | 7276/41250 [17:34:38<81:45:17,  8.66s/it]
                                                          
{'loss': 0.3032, 'grad_norm': 3.192342519760132, 'learning_rate': 3.779425109663987e-05, 'epoch': 1.76}

 18%|█▊        | 7276/41250 [17:34:38<81:45:17,  8.66s/it][2025-04-26 01:32:21,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:32:21,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.96 | bwd_microstep: 5769.46 | bwd_inner_microstep: 5653.14 | bwd_allreduce_microstep: 116.27 | step_microstep: 18.88
[2025-04-26 01:32:21,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.96 | bwd: 5769.47 | bwd_inner: 5653.14 | bwd_allreduce: 116.29 | step: 18.88

 18%|█▊        | 7277/41250 [17:34:46<81:47:23,  8.67s/it]
                                                          
{'loss': 0.145, 'grad_norm': 3.8613812923431396, 'learning_rate': 3.779353415509593e-05, 'epoch': 1.76}

 18%|█▊        | 7277/41250 [17:34:46<81:47:23,  8.67s/it][2025-04-26 01:32:30,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.28 | optimizer_step: 1.04
[2025-04-26 01:32:30,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.63 | bwd_microstep: 5738.07 | bwd_inner_microstep: 5684.19 | bwd_allreduce_microstep: 53.82 | step_microstep: 20.07
[2025-04-26 01:32:30,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.63 | bwd: 5738.09 | bwd_inner: 5684.19 | bwd_allreduce: 53.85 | step: 20.08

 18%|█▊        | 7278/41250 [17:34:55<81:46:51,  8.67s/it]
                                                          
{'loss': 0.1187, 'grad_norm': 0.8610243797302246, 'learning_rate': 3.779281710385836e-05, 'epoch': 1.76}

 18%|█▊        | 7278/41250 [17:34:55<81:46:51,  8.67s/it][2025-04-26 01:32:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 01:32:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.67 | bwd_microstep: 5768.67 | bwd_inner_microstep: 5755.81 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.08
[2025-04-26 01:32:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.67 | bwd: 5768.69 | bwd_inner: 5755.81 | bwd_allreduce: 12.84 | step: 19.08

 18%|█▊        | 7279/41250 [17:35:04<81:57:10,  8.68s/it]
                                                          
{'loss': 0.1412, 'grad_norm': 1.2485491037368774, 'learning_rate': 3.779209994293156e-05, 'epoch': 1.76}

 18%|█▊        | 7279/41250 [17:35:04<81:57:10,  8.68s/it][2025-04-26 01:32:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:32:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.96 | bwd_microstep: 5753.30 | bwd_inner_microstep: 5696.83 | bwd_allreduce_microstep: 56.42 | step_microstep: 18.63
[2025-04-26 01:32:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.96 | bwd: 5753.31 | bwd_inner: 5696.83 | bwd_allreduce: 56.44 | step: 18.63

 18%|█▊        | 7280/41250 [17:35:13<81:57:36,  8.69s/it]
                                                          
{'loss': 0.1645, 'grad_norm': 1.4136515855789185, 'learning_rate': 3.7791382672319964e-05, 'epoch': 1.76}

 18%|█▊        | 7280/41250 [17:35:13<81:57:36,  8.69s/it][2025-04-26 01:32:56,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:32:56,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.29 | bwd_microstep: 5707.49 | bwd_inner_microstep: 5656.53 | bwd_allreduce_microstep: 50.91 | step_microstep: 18.58
[2025-04-26 01:32:56,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.29 | bwd: 5707.50 | bwd_inner: 5656.53 | bwd_allreduce: 50.93 | step: 18.58

 18%|█▊        | 7281/41250 [17:35:21<81:45:00,  8.66s/it]
                                                          
{'loss': 0.1519, 'grad_norm': 1.254064917564392, 'learning_rate': 3.779066529202799e-05, 'epoch': 1.77}

 18%|█▊        | 7281/41250 [17:35:21<81:45:00,  8.66s/it][2025-04-26 01:33:05,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 01:33:05,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.72 | bwd_microstep: 5773.44 | bwd_inner_microstep: 5658.95 | bwd_allreduce_microstep: 114.43 | step_microstep: 19.00
[2025-04-26 01:33:05,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.72 | bwd: 5773.45 | bwd_inner: 5658.95 | bwd_allreduce: 114.46 | step: 19.00

 18%|█▊        | 7282/41250 [17:35:30<81:50:34,  8.67s/it]
                                                          
{'loss': 0.1287, 'grad_norm': 3.004906177520752, 'learning_rate': 3.7789947802060076e-05, 'epoch': 1.77}

 18%|█▊        | 7282/41250 [17:35:30<81:50:34,  8.67s/it][2025-04-26 01:33:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 01:33:13,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.26 | bwd_microstep: 5744.49 | bwd_inner_microstep: 5701.61 | bwd_allreduce_microstep: 42.84 | step_microstep: 19.03
[2025-04-26 01:33:13,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.26 | bwd: 5744.51 | bwd_inner: 5701.61 | bwd_allreduce: 42.86 | step: 19.03

 18%|█▊        | 7283/41250 [17:35:39<81:51:49,  8.68s/it]
                                                          
{'loss': 0.0671, 'grad_norm': 1.507530927658081, 'learning_rate': 3.778923020242063e-05, 'epoch': 1.77}

 18%|█▊        | 7283/41250 [17:35:39<81:51:49,  8.68s/it][2025-04-26 01:33:22,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:33:22,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.66 | bwd_microstep: 5708.50 | bwd_inner_microstep: 5695.80 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.92
[2025-04-26 01:33:22,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.66 | bwd: 5708.51 | bwd_inner: 5695.80 | bwd_allreduce: 12.67 | step: 18.92

 18%|█▊        | 7284/41250 [17:35:47<81:46:47,  8.67s/it]
                                                          
{'loss': 0.313, 'grad_norm': 3.7436351776123047, 'learning_rate': 3.778851249311408e-05, 'epoch': 1.77}

 18%|█▊        | 7284/41250 [17:35:47<81:46:47,  8.67s/it][2025-04-26 01:33:31,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:33:31,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.39 | bwd_microstep: 5781.33 | bwd_inner_microstep: 5645.76 | bwd_allreduce_microstep: 135.54 | step_microstep: 18.29
[2025-04-26 01:33:31,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.39 | bwd: 5781.35 | bwd_inner: 5645.76 | bwd_allreduce: 135.55 | step: 18.29

 18%|█▊        | 7285/41250 [17:35:56<81:49:42,  8.67s/it]
                                                          
{'loss': 0.1818, 'grad_norm': 3.1978697776794434, 'learning_rate': 3.778779467414484e-05, 'epoch': 1.77}

 18%|█▊        | 7285/41250 [17:35:56<81:49:42,  8.67s/it][2025-04-26 01:33:39,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 01:33:39,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.73 | bwd_microstep: 5715.96 | bwd_inner_microstep: 5652.04 | bwd_allreduce_microstep: 63.88 | step_microstep: 18.77
[2025-04-26 01:33:39,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.73 | bwd: 5715.98 | bwd_inner: 5652.04 | bwd_allreduce: 63.90 | step: 18.77

 18%|█▊        | 7286/41250 [17:36:04<81:40:40,  8.66s/it]
                                                          
{'loss': 0.1507, 'grad_norm': 1.8625836372375488, 'learning_rate': 3.7787076745517353e-05, 'epoch': 1.77}

 18%|█▊        | 7286/41250 [17:36:04<81:40:40,  8.66s/it][2025-04-26 01:33:48,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:33:48,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.81 | bwd_microstep: 5764.13 | bwd_inner_microstep: 5650.87 | bwd_allreduce_microstep: 113.21 | step_microstep: 18.65
[2025-04-26 01:33:48,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.82 | bwd: 5764.15 | bwd_inner: 5650.87 | bwd_allreduce: 113.24 | step: 18.65

 18%|█▊        | 7287/41250 [17:36:13<81:43:15,  8.66s/it]
                                                          
{'loss': 0.2184, 'grad_norm': 2.2269482612609863, 'learning_rate': 3.778635870723603e-05, 'epoch': 1.77}

 18%|█▊        | 7287/41250 [17:36:13<81:43:15,  8.66s/it][2025-04-26 01:33:56,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.15
[2025-04-26 01:33:56,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.53 | bwd_microstep: 5703.73 | bwd_inner_microstep: 5691.11 | bwd_allreduce_microstep: 12.57 | step_microstep: 19.45
[2025-04-26 01:33:56,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.53 | bwd: 5703.74 | bwd_inner: 5691.11 | bwd_allreduce: 12.59 | step: 19.45

 18%|█▊        | 7288/41250 [17:36:22<81:38:48,  8.65s/it]
                                                          
{'loss': 0.1943, 'grad_norm': 1.0199179649353027, 'learning_rate': 3.7785640559305304e-05, 'epoch': 1.77}

 18%|█▊        | 7288/41250 [17:36:22<81:38:48,  8.65s/it][2025-04-26 01:34:05,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:34:05,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5769.72 | bwd_inner_microstep: 5642.04 | bwd_allreduce_microstep: 127.63 | step_microstep: 18.53
[2025-04-26 01:34:05,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5769.73 | bwd_inner: 5642.04 | bwd_allreduce: 127.65 | step: 18.54

 18%|█▊        | 7289/41250 [17:36:30<81:42:57,  8.66s/it]
                                                          
{'loss': 0.0856, 'grad_norm': 2.1149911880493164, 'learning_rate': 3.778492230172961e-05, 'epoch': 1.77}

 18%|█▊        | 7289/41250 [17:36:30<81:42:57,  8.66s/it][2025-04-26 01:34:14,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 01:34:14,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.16 | bwd_microstep: 5774.74 | bwd_inner_microstep: 5694.98 | bwd_allreduce_microstep: 79.70 | step_microstep: 19.06
[2025-04-26 01:34:14,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.16 | bwd: 5774.75 | bwd_inner: 5694.98 | bwd_allreduce: 79.72 | step: 19.07

 18%|█▊        | 7290/41250 [17:36:39<81:51:14,  8.68s/it]
                                                          
{'loss': 0.3199, 'grad_norm': 2.277965545654297, 'learning_rate': 3.778420393451336e-05, 'epoch': 1.77}

 18%|█▊        | 7290/41250 [17:36:39<81:51:14,  8.68s/it][2025-04-26 01:34:23,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:34:23,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.05 | bwd_microstep: 5752.63 | bwd_inner_microstep: 5701.59 | bwd_allreduce_microstep: 51.00 | step_microstep: 18.56
[2025-04-26 01:34:23,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.05 | bwd: 5752.65 | bwd_inner: 5701.59 | bwd_allreduce: 51.01 | step: 18.57

 18%|█▊        | 7291/41250 [17:36:48<81:52:39,  8.68s/it]
                                                          
{'loss': 0.0798, 'grad_norm': 1.5169986486434937, 'learning_rate': 3.7783485457660996e-05, 'epoch': 1.77}

 18%|█▊        | 7291/41250 [17:36:48<81:52:39,  8.68s/it][2025-04-26 01:34:31,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:34:31,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.93 | bwd_microstep: 5754.62 | bwd_inner_microstep: 5703.38 | bwd_allreduce_microstep: 51.20 | step_microstep: 18.82
[2025-04-26 01:34:31,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5754.64 | bwd_inner: 5703.38 | bwd_allreduce: 51.22 | step: 18.82

 18%|█▊        | 7292/41250 [17:36:57<81:54:01,  8.68s/it]
                                                          
{'loss': 0.1118, 'grad_norm': 1.0493124723434448, 'learning_rate': 3.778276687117694e-05, 'epoch': 1.77}

 18%|█▊        | 7292/41250 [17:36:57<81:54:01,  8.68s/it][2025-04-26 01:34:40,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:34:40,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.88 | bwd_microstep: 5781.91 | bwd_inner_microstep: 5769.27 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.65
[2025-04-26 01:34:40,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.88 | bwd: 5781.92 | bwd_inner: 5769.27 | bwd_allreduce: 12.60 | step: 18.66

 18%|█▊        | 7293/41250 [17:37:05<82:04:21,  8.70s/it]
                                                          
{'loss': 0.3961, 'grad_norm': 3.8468971252441406, 'learning_rate': 3.778204817506562e-05, 'epoch': 1.77}

 18%|█▊        | 7293/41250 [17:37:05<82:04:21,  8.70s/it][2025-04-26 01:34:49,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:34:49,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.54 | bwd_microstep: 5771.07 | bwd_inner_microstep: 5710.72 | bwd_allreduce_microstep: 60.29 | step_microstep: 18.63
[2025-04-26 01:34:49,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.54 | bwd: 5771.08 | bwd_inner: 5710.72 | bwd_allreduce: 60.31 | step: 18.63

 18%|█▊        | 7294/41250 [17:37:14<82:03:56,  8.70s/it]
                                                          
{'loss': 0.0763, 'grad_norm': 1.1422648429870605, 'learning_rate': 3.7781329369331474e-05, 'epoch': 1.77}

 18%|█▊        | 7294/41250 [17:37:14<82:03:56,  8.70s/it][2025-04-26 01:34:57,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.09 | optimizer_step: 1.02
[2025-04-26 01:34:57,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.09 | bwd_microstep: 5732.15 | bwd_inner_microstep: 5718.66 | bwd_allreduce_microstep: 13.43 | step_microstep: 19.55
[2025-04-26 01:34:57,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.09 | bwd: 5732.16 | bwd_inner: 5718.66 | bwd_allreduce: 13.46 | step: 19.56

 18%|█▊        | 7295/41250 [17:37:23<81:58:24,  8.69s/it]
                                                          
{'loss': 0.1947, 'grad_norm': 2.4322454929351807, 'learning_rate': 3.778061045397893e-05, 'epoch': 1.77}

 18%|█▊        | 7295/41250 [17:37:23<81:58:24,  8.69s/it][2025-04-26 01:35:06,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:35:06,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.98 | bwd_microstep: 5766.81 | bwd_inner_microstep: 5693.90 | bwd_allreduce_microstep: 72.86 | step_microstep: 18.90
[2025-04-26 01:35:06,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.98 | bwd: 5766.82 | bwd_inner: 5693.90 | bwd_allreduce: 72.88 | step: 18.91

 18%|█▊        | 7296/41250 [17:37:31<81:59:33,  8.69s/it]
                                                          
{'loss': 0.0872, 'grad_norm': 1.1095019578933716, 'learning_rate': 3.777989142901242e-05, 'epoch': 1.77}

 18%|█▊        | 7296/41250 [17:37:31<81:59:33,  8.69s/it][2025-04-26 01:35:15,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 01:35:15,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5787.40 | bwd_inner_microstep: 5650.48 | bwd_allreduce_microstep: 136.87 | step_microstep: 18.86
[2025-04-26 01:35:15,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5787.41 | bwd_inner: 5650.48 | bwd_allreduce: 136.89 | step: 18.86

 18%|█▊        | 7297/41250 [17:37:40<82:00:54,  8.70s/it]
                                                          
{'loss': 0.2587, 'grad_norm': 1.7667063474655151, 'learning_rate': 3.777917229443638e-05, 'epoch': 1.77}

 18%|█▊        | 7297/41250 [17:37:40<82:00:54,  8.70s/it][2025-04-26 01:35:23,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:35:23,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 5789.63 | bwd_inner_microstep: 5706.50 | bwd_allreduce_microstep: 83.08 | step_microstep: 18.91
[2025-04-26 01:35:23,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 5789.65 | bwd_inner: 5706.50 | bwd_allreduce: 83.10 | step: 18.92

 18%|█▊        | 7298/41250 [17:37:49<82:05:20,  8.70s/it]
                                                          
{'loss': 0.1312, 'grad_norm': 1.4023975133895874, 'learning_rate': 3.777845305025523e-05, 'epoch': 1.77}

 18%|█▊        | 7298/41250 [17:37:49<82:05:20,  8.70s/it][2025-04-26 01:35:32,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 01:35:32,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.24 | bwd_microstep: 5801.28 | bwd_inner_microstep: 5660.35 | bwd_allreduce_microstep: 140.88 | step_microstep: 18.92
[2025-04-26 01:35:32,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.24 | bwd: 5801.30 | bwd_inner: 5660.35 | bwd_allreduce: 140.91 | step: 18.92

 18%|█▊        | 7299/41250 [17:37:58<82:06:41,  8.71s/it]
                                                          
{'loss': 0.1804, 'grad_norm': 2.2921979427337646, 'learning_rate': 3.777773369647342e-05, 'epoch': 1.77}

 18%|█▊        | 7299/41250 [17:37:58<82:06:41,  8.71s/it][2025-04-26 01:35:41,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 01:35:41,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.08 | bwd_microstep: 5756.42 | bwd_inner_microstep: 5697.29 | bwd_allreduce_microstep: 59.09 | step_microstep: 19.46
[2025-04-26 01:35:41,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.08 | bwd: 5756.44 | bwd_inner: 5697.29 | bwd_allreduce: 59.11 | step: 19.47

 18%|█▊        | 7300/41250 [17:38:06<82:04:14,  8.70s/it]
                                                          
{'loss': 0.2784, 'grad_norm': 3.6411681175231934, 'learning_rate': 3.777701423309538e-05, 'epoch': 1.77}

 18%|█▊        | 7300/41250 [17:38:06<82:04:14,  8.70s/it][2025-04-26 01:35:50,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:35:50,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.70 | bwd_microstep: 5758.55 | bwd_inner_microstep: 5658.65 | bwd_allreduce_microstep: 99.85 | step_microstep: 18.72
[2025-04-26 01:35:50,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.70 | bwd: 5758.57 | bwd_inner: 5658.65 | bwd_allreduce: 99.87 | step: 18.72

 18%|█▊        | 7301/41250 [17:38:15<82:01:03,  8.70s/it]
                                                          
{'loss': 0.1648, 'grad_norm': 1.8346571922302246, 'learning_rate': 3.777629466012554e-05, 'epoch': 1.77}

 18%|█▊        | 7301/41250 [17:38:15<82:01:03,  8.70s/it][2025-04-26 01:35:58,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 01:35:58,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.00 | bwd_microstep: 5749.05 | bwd_inner_microstep: 5690.27 | bwd_allreduce_microstep: 58.74 | step_microstep: 18.82
[2025-04-26 01:35:58,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.00 | bwd: 5749.07 | bwd_inner: 5690.27 | bwd_allreduce: 58.76 | step: 18.82

 18%|█▊        | 7302/41250 [17:38:24<81:57:44,  8.69s/it]
                                                          
{'loss': 0.0679, 'grad_norm': 2.194138288497925, 'learning_rate': 3.777557497756835e-05, 'epoch': 1.77}

 18%|█▊        | 7302/41250 [17:38:24<81:57:44,  8.69s/it][2025-04-26 01:36:07,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 01:36:07,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.63 | bwd_microstep: 5721.61 | bwd_inner_microstep: 5650.76 | bwd_allreduce_microstep: 70.81 | step_microstep: 18.57
[2025-04-26 01:36:07,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5721.63 | bwd_inner: 5650.76 | bwd_allreduce: 70.83 | step: 18.57

 18%|█▊        | 7303/41250 [17:38:32<81:47:30,  8.67s/it]
                                                          
{'loss': 0.1742, 'grad_norm': 1.5283255577087402, 'learning_rate': 3.7774855185428224e-05, 'epoch': 1.77}

 18%|█▊        | 7303/41250 [17:38:32<81:47:30,  8.67s/it][2025-04-26 01:36:16,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:36:16,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.94 | bwd_microstep: 5702.77 | bwd_inner_microstep: 5689.89 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.92
[2025-04-26 01:36:16,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.94 | bwd: 5702.79 | bwd_inner: 5689.89 | bwd_allreduce: 12.85 | step: 18.93

 18%|█▊        | 7304/41250 [17:38:41<81:40:53,  8.66s/it]
                                                          
{'loss': 0.1374, 'grad_norm': 2.4164388179779053, 'learning_rate': 3.777413528370962e-05, 'epoch': 1.77}

 18%|█▊        | 7304/41250 [17:38:41<81:40:53,  8.66s/it][2025-04-26 01:36:24,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:36:24,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.19 | bwd_microstep: 5767.56 | bwd_inner_microstep: 5657.17 | bwd_allreduce_microstep: 110.34 | step_microstep: 18.82
[2025-04-26 01:36:24,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.19 | bwd: 5767.57 | bwd_inner: 5657.17 | bwd_allreduce: 110.36 | step: 18.82

 18%|█▊        | 7305/41250 [17:38:50<81:43:41,  8.67s/it]
                                                          
{'loss': 0.0983, 'grad_norm': 1.153323769569397, 'learning_rate': 3.7773415272416964e-05, 'epoch': 1.77}

 18%|█▊        | 7305/41250 [17:38:50<81:43:41,  8.67s/it][2025-04-26 01:36:33,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:36:33,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.88 | bwd_microstep: 5722.64 | bwd_inner_microstep: 5626.98 | bwd_allreduce_microstep: 95.61 | step_microstep: 18.46
[2025-04-26 01:36:33,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.88 | bwd: 5722.65 | bwd_inner: 5626.98 | bwd_allreduce: 95.63 | step: 18.46

 18%|█▊        | 7306/41250 [17:38:58<81:36:58,  8.66s/it]
                                                          
{'loss': 0.1063, 'grad_norm': 1.5419564247131348, 'learning_rate': 3.7772695151554704e-05, 'epoch': 1.77}

 18%|█▊        | 7306/41250 [17:38:58<81:36:58,  8.66s/it][2025-04-26 01:36:41,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-26 01:36:41,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.69 | bwd_microstep: 5710.84 | bwd_inner_microstep: 5697.98 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.08
[2025-04-26 01:36:41,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.69 | bwd: 5710.85 | bwd_inner: 5697.98 | bwd_allreduce: 12.83 | step: 19.08

 18%|█▊        | 7307/41250 [17:39:07<81:34:37,  8.65s/it]
                                                          
{'loss': 0.2057, 'grad_norm': 1.3727102279663086, 'learning_rate': 3.777197492112726e-05, 'epoch': 1.77}

 18%|█▊        | 7307/41250 [17:39:07<81:34:37,  8.65s/it][2025-04-26 01:36:50,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 01:36:50,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.90 | bwd_microstep: 5785.48 | bwd_inner_microstep: 5772.44 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.86
[2025-04-26 01:36:50,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.90 | bwd: 5785.50 | bwd_inner: 5772.44 | bwd_allreduce: 13.01 | step: 18.86

 18%|█▊        | 7308/41250 [17:39:16<81:52:36,  8.68s/it]
                                                          
{'loss': 0.0691, 'grad_norm': 1.0433928966522217, 'learning_rate': 3.7771254581139096e-05, 'epoch': 1.77}

 18%|█▊        | 7308/41250 [17:39:16<81:52:36,  8.68s/it][2025-04-26 01:36:59,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 01:36:59,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.57 | bwd_microstep: 5742.50 | bwd_inner_microstep: 5690.43 | bwd_allreduce_microstep: 52.02 | step_microstep: 19.63
[2025-04-26 01:36:59,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.57 | bwd: 5742.52 | bwd_inner: 5690.43 | bwd_allreduce: 52.04 | step: 19.63

 18%|█▊        | 7309/41250 [17:39:24<81:51:30,  8.68s/it]
                                                          
{'loss': 0.0754, 'grad_norm': 0.6334443092346191, 'learning_rate': 3.7770534131594646e-05, 'epoch': 1.77}

 18%|█▊        | 7309/41250 [17:39:24<81:51:30,  8.68s/it][2025-04-26 01:37:08,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:37:08,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.54 | bwd_microstep: 5766.14 | bwd_inner_microstep: 5681.97 | bwd_allreduce_microstep: 84.12 | step_microstep: 18.73
[2025-04-26 01:37:08,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.54 | bwd: 5766.16 | bwd_inner: 5681.97 | bwd_allreduce: 84.14 | step: 18.73

 18%|█▊        | 7310/41250 [17:39:33<81:54:29,  8.69s/it]
                                                          
{'loss': 0.1083, 'grad_norm': 0.8004656434059143, 'learning_rate': 3.776981357249835e-05, 'epoch': 1.77}

 18%|█▊        | 7310/41250 [17:39:33<81:54:29,  8.69s/it][2025-04-26 01:37:16,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 01:37:16,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.22 | bwd_microstep: 5754.81 | bwd_inner_microstep: 5636.33 | bwd_allreduce_microstep: 118.43 | step_microstep: 18.93
[2025-04-26 01:37:16,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.22 | bwd: 5754.82 | bwd_inner: 5636.33 | bwd_allreduce: 118.45 | step: 18.94

 18%|█▊        | 7311/41250 [17:39:42<81:49:52,  8.68s/it]
                                                          
{'loss': 0.1543, 'grad_norm': 1.5271371603012085, 'learning_rate': 3.776909290385464e-05, 'epoch': 1.77}

 18%|█▊        | 7311/41250 [17:39:42<81:49:52,  8.68s/it][2025-04-26 01:37:25,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 01:37:25,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.80 | bwd_microstep: 5700.34 | bwd_inner_microstep: 5683.29 | bwd_allreduce_microstep: 17.01 | step_microstep: 19.01
[2025-04-26 01:37:25,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.80 | bwd: 5700.35 | bwd_inner: 5683.29 | bwd_allreduce: 17.03 | step: 19.01

 18%|█▊        | 7312/41250 [17:39:50<81:40:02,  8.66s/it]
                                                          
{'loss': 0.2067, 'grad_norm': 2.415011405944824, 'learning_rate': 3.776837212566797e-05, 'epoch': 1.77}

 18%|█▊        | 7312/41250 [17:39:50<81:40:02,  8.66s/it][2025-04-26 01:37:34,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:37:34,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.85 | bwd_microstep: 5794.03 | bwd_inner_microstep: 5645.73 | bwd_allreduce_microstep: 148.26 | step_microstep: 18.93
[2025-04-26 01:37:34,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.85 | bwd: 5794.05 | bwd_inner: 5645.73 | bwd_allreduce: 148.28 | step: 18.93

 18%|█▊        | 7313/41250 [17:39:59<81:47:22,  8.68s/it]
                                                          
{'loss': 0.2313, 'grad_norm': 2.8018784523010254, 'learning_rate': 3.7767651237942777e-05, 'epoch': 1.77}

 18%|█▊        | 7313/41250 [17:39:59<81:47:22,  8.68s/it][2025-04-26 01:37:42,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:37:42,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.08 | bwd_microstep: 5698.81 | bwd_inner_microstep: 5686.27 | bwd_allreduce_microstep: 12.49 | step_microstep: 19.03
[2025-04-26 01:37:42,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.08 | bwd: 5698.82 | bwd_inner: 5686.27 | bwd_allreduce: 12.51 | step: 19.03

 18%|█▊        | 7314/41250 [17:40:08<81:38:04,  8.66s/it]
                                                          
{'loss': 0.158, 'grad_norm': 1.7513633966445923, 'learning_rate': 3.776693024068351e-05, 'epoch': 1.77}

 18%|█▊        | 7314/41250 [17:40:08<81:38:04,  8.66s/it][2025-04-26 01:37:51,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:37:51,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5690.48 | bwd_inner_microstep: 5677.64 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.65
[2025-04-26 01:37:51,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5690.49 | bwd_inner: 5677.64 | bwd_allreduce: 12.81 | step: 18.66

 18%|█▊        | 7315/41250 [17:40:16<81:31:02,  8.65s/it]
                                                          
{'loss': 0.0689, 'grad_norm': 1.3205991983413696, 'learning_rate': 3.776620913389462e-05, 'epoch': 1.77}

 18%|█▊        | 7315/41250 [17:40:16<81:31:02,  8.65s/it][2025-04-26 01:38:00,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:38:00,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.85 | bwd_microstep: 5772.68 | bwd_inner_microstep: 5644.49 | bwd_allreduce_microstep: 128.13 | step_microstep: 18.59
[2025-04-26 01:38:00,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.85 | bwd: 5772.69 | bwd_inner: 5644.49 | bwd_allreduce: 128.15 | step: 18.59

 18%|█▊        | 7316/41250 [17:40:25<81:36:01,  8.66s/it]
                                                          
{'loss': 0.0605, 'grad_norm': 0.9576243162155151, 'learning_rate': 3.776548791758054e-05, 'epoch': 1.77}

 18%|█▊        | 7316/41250 [17:40:25<81:36:01,  8.66s/it][2025-04-26 01:38:08,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 01:38:08,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.28 | bwd_microstep: 5763.12 | bwd_inner_microstep: 5647.19 | bwd_allreduce_microstep: 115.88 | step_microstep: 19.31
[2025-04-26 01:38:08,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.28 | bwd: 5763.13 | bwd_inner: 5647.19 | bwd_allreduce: 115.90 | step: 19.32

 18%|█▊        | 7317/41250 [17:40:34<81:37:51,  8.66s/it]
                                                          
{'loss': 0.08, 'grad_norm': 1.4218165874481201, 'learning_rate': 3.776476659174572e-05, 'epoch': 1.77}

 18%|█▊        | 7317/41250 [17:40:34<81:37:51,  8.66s/it][2025-04-26 01:38:17,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 01:38:17,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.35 | bwd_microstep: 5703.04 | bwd_inner_microstep: 5634.01 | bwd_allreduce_microstep: 68.99 | step_microstep: 19.10
[2025-04-26 01:38:17,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.35 | bwd: 5703.06 | bwd_inner: 5634.01 | bwd_allreduce: 69.01 | step: 19.10

 18%|█▊        | 7318/41250 [17:40:42<81:28:06,  8.64s/it]
                                                          
{'loss': 0.0605, 'grad_norm': 1.0827914476394653, 'learning_rate': 3.7764045156394614e-05, 'epoch': 1.77}

 18%|█▊        | 7318/41250 [17:40:42<81:28:06,  8.64s/it][2025-04-26 01:38:25,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 01:38:25,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.81 | bwd_microstep: 5688.61 | bwd_inner_microstep: 5635.29 | bwd_allreduce_microstep: 53.27 | step_microstep: 18.82
[2025-04-26 01:38:25,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.81 | bwd: 5688.62 | bwd_inner: 5635.29 | bwd_allreduce: 53.29 | step: 18.83

 18%|█▊        | 7319/41250 [17:40:51<81:20:04,  8.63s/it]
                                                          
{'loss': 0.2205, 'grad_norm': 2.593327760696411, 'learning_rate': 3.776332361153166e-05, 'epoch': 1.77}

 18%|█▊        | 7319/41250 [17:40:51<81:20:04,  8.63s/it][2025-04-26 01:38:34,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:38:34,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.58 | bwd_microstep: 5877.09 | bwd_inner_microstep: 5639.81 | bwd_allreduce_microstep: 237.24 | step_microstep: 18.64
[2025-04-26 01:38:34,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.58 | bwd: 5877.11 | bwd_inner: 5639.81 | bwd_allreduce: 237.26 | step: 18.64

 18%|█▊        | 7320/41250 [17:40:59<81:45:10,  8.67s/it]
                                                          
{'loss': 0.0868, 'grad_norm': 0.9245887398719788, 'learning_rate': 3.7762601957161304e-05, 'epoch': 1.77}

 18%|█▊        | 7320/41250 [17:40:59<81:45:10,  8.67s/it][2025-04-26 01:38:43,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:38:43,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5719.66 | bwd_inner_microstep: 5672.91 | bwd_allreduce_microstep: 46.70 | step_microstep: 18.55
[2025-04-26 01:38:43,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5719.67 | bwd_inner: 5672.91 | bwd_allreduce: 46.72 | step: 18.55

 18%|█▊        | 7321/41250 [17:41:08<81:40:56,  8.67s/it]
                                                          
{'loss': 0.0935, 'grad_norm': 1.2016100883483887, 'learning_rate': 3.776188019328801e-05, 'epoch': 1.77}

 18%|█▊        | 7321/41250 [17:41:08<81:40:56,  8.67s/it][2025-04-26 01:38:51,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 01:38:51,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.84 | bwd_microstep: 5764.59 | bwd_inner_microstep: 5638.44 | bwd_allreduce_microstep: 126.10 | step_microstep: 18.79
[2025-04-26 01:38:51,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.84 | bwd: 5764.60 | bwd_inner: 5638.44 | bwd_allreduce: 126.12 | step: 18.79

 18%|█▊        | 7322/41250 [17:41:17<81:41:20,  8.67s/it]
                                                          
{'loss': 0.5043, 'grad_norm': 3.09061336517334, 'learning_rate': 3.776115831991622e-05, 'epoch': 1.78}

 18%|█▊        | 7322/41250 [17:41:17<81:41:20,  8.67s/it][2025-04-26 01:39:00,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 01:39:00,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.98 | bwd_microstep: 5840.36 | bwd_inner_microstep: 5673.34 | bwd_allreduce_microstep: 166.97 | step_microstep: 18.77
[2025-04-26 01:39:00,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.98 | bwd: 5840.37 | bwd_inner: 5673.34 | bwd_allreduce: 166.99 | step: 18.77

 18%|█▊        | 7323/41250 [17:41:26<81:57:31,  8.70s/it]
                                                          
{'loss': 0.091, 'grad_norm': 1.1608799695968628, 'learning_rate': 3.7760436337050365e-05, 'epoch': 1.78}

 18%|█▊        | 7323/41250 [17:41:26<81:57:31,  8.70s/it][2025-04-26 01:39:09,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:39:09,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.01 | bwd_microstep: 5771.53 | bwd_inner_microstep: 5678.32 | bwd_allreduce_microstep: 93.16 | step_microstep: 18.56
[2025-04-26 01:39:09,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.01 | bwd: 5771.54 | bwd_inner: 5678.32 | bwd_allreduce: 93.18 | step: 18.56

 18%|█▊        | 7324/41250 [17:41:34<81:56:43,  8.70s/it]
                                                          
{'loss': 0.3087, 'grad_norm': 2.04487943649292, 'learning_rate': 3.775971424469493e-05, 'epoch': 1.78}

 18%|█▊        | 7324/41250 [17:41:34<81:56:43,  8.70s/it][2025-04-26 01:39:18,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:39:18,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.36 | bwd_microstep: 5741.82 | bwd_inner_microstep: 5701.08 | bwd_allreduce_microstep: 40.69 | step_microstep: 18.98
[2025-04-26 01:39:18,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.36 | bwd: 5741.83 | bwd_inner: 5701.08 | bwd_allreduce: 40.71 | step: 18.98

 18%|█▊        | 7325/41250 [17:41:43<81:53:37,  8.69s/it]
                                                          
{'loss': 0.0921, 'grad_norm': 1.4723286628723145, 'learning_rate': 3.7758992042854345e-05, 'epoch': 1.78}

 18%|█▊        | 7325/41250 [17:41:43<81:53:37,  8.69s/it][2025-04-26 01:39:26,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:39:26,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.64 | bwd_microstep: 5744.55 | bwd_inner_microstep: 5694.26 | bwd_allreduce_microstep: 50.25 | step_microstep: 18.54
[2025-04-26 01:39:26,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.64 | bwd: 5744.57 | bwd_inner: 5694.26 | bwd_allreduce: 50.26 | step: 18.54

 18%|█▊        | 7326/41250 [17:41:52<81:51:03,  8.69s/it]
                                                          
{'loss': 0.0964, 'grad_norm': 1.7531070709228516, 'learning_rate': 3.775826973153307e-05, 'epoch': 1.78}

 18%|█▊        | 7326/41250 [17:41:52<81:51:03,  8.69s/it][2025-04-26 01:39:35,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 01:39:35,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.36 | bwd_microstep: 5730.49 | bwd_inner_microstep: 5698.29 | bwd_allreduce_microstep: 32.16 | step_microstep: 19.12
[2025-04-26 01:39:35,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.36 | bwd: 5730.51 | bwd_inner: 5698.29 | bwd_allreduce: 32.18 | step: 19.12

 18%|█▊        | 7327/41250 [17:42:00<81:46:49,  8.68s/it]
                                                          
{'loss': 0.0889, 'grad_norm': 1.5756007432937622, 'learning_rate': 3.775754731073555e-05, 'epoch': 1.78}

 18%|█▊        | 7327/41250 [17:42:00<81:46:49,  8.68s/it][2025-04-26 01:39:44,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 01:39:44,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.12 | bwd_microstep: 5719.57 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 13.31 | step_microstep: 19.43
[2025-04-26 01:39:44,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.12 | bwd: 5719.58 | bwd_inner: 5706.20 | bwd_allreduce: 13.34 | step: 19.43

 18%|█▊        | 7328/41250 [17:42:09<81:42:53,  8.67s/it]
                                                          
{'loss': 0.2061, 'grad_norm': 3.030895471572876, 'learning_rate': 3.7756824780466244e-05, 'epoch': 1.78}

 18%|█▊        | 7328/41250 [17:42:09<81:42:53,  8.67s/it][2025-04-26 01:39:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 01:39:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.56 | bwd_microstep: 5766.88 | bwd_inner_microstep: 5649.94 | bwd_allreduce_microstep: 116.89 | step_microstep: 18.76
[2025-04-26 01:39:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.56 | bwd: 5766.89 | bwd_inner: 5649.94 | bwd_allreduce: 116.91 | step: 18.77

 18%|█▊        | 7329/41250 [17:42:18<81:43:05,  8.67s/it]
                                                          
{'loss': 0.1521, 'grad_norm': 1.7687112092971802, 'learning_rate': 3.7756102140729606e-05, 'epoch': 1.78}

 18%|█▊        | 7329/41250 [17:42:18<81:43:05,  8.67s/it][2025-04-26 01:40:01,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:40:01,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.84 | bwd_microstep: 5747.12 | bwd_inner_microstep: 5689.14 | bwd_allreduce_microstep: 57.94 | step_microstep: 18.71
[2025-04-26 01:40:01,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.84 | bwd: 5747.14 | bwd_inner: 5689.14 | bwd_allreduce: 57.96 | step: 18.72

 18%|█▊        | 7330/41250 [17:42:26<81:43:06,  8.67s/it]
                                                          
{'loss': 0.1472, 'grad_norm': 2.96606707572937, 'learning_rate': 3.7755379391530094e-05, 'epoch': 1.78}

 18%|█▊        | 7330/41250 [17:42:26<81:43:06,  8.67s/it][2025-04-26 01:40:10,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 01:40:10,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.76 | bwd_microstep: 5850.96 | bwd_inner_microstep: 5705.62 | bwd_allreduce_microstep: 145.27 | step_microstep: 18.94
[2025-04-26 01:40:10,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.76 | bwd: 5850.98 | bwd_inner: 5705.62 | bwd_allreduce: 145.30 | step: 18.94

 18%|█▊        | 7331/41250 [17:42:35<82:03:03,  8.71s/it]
                                                          
{'loss': 0.1273, 'grad_norm': 2.179964542388916, 'learning_rate': 3.775465653287216e-05, 'epoch': 1.78}

 18%|█▊        | 7331/41250 [17:42:35<82:03:03,  8.71s/it][2025-04-26 01:40:18,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 01:40:18,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.06 | bwd_microstep: 5779.55 | bwd_inner_microstep: 5643.37 | bwd_allreduce_microstep: 136.12 | step_microstep: 19.44
[2025-04-26 01:40:18,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.06 | bwd: 5779.56 | bwd_inner: 5643.37 | bwd_allreduce: 136.14 | step: 19.44

 18%|█▊        | 7332/41250 [17:42:44<81:59:17,  8.70s/it]
                                                          
{'loss': 0.1845, 'grad_norm': 6.729294776916504, 'learning_rate': 3.7753933564760256e-05, 'epoch': 1.78}

 18%|█▊        | 7332/41250 [17:42:44<81:59:17,  8.70s/it][2025-04-26 01:40:27,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 01:40:27,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.06 | bwd_microstep: 5795.49 | bwd_inner_microstep: 5782.23 | bwd_allreduce_microstep: 13.21 | step_microstep: 19.09
[2025-04-26 01:40:27,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.06 | bwd: 5795.50 | bwd_inner: 5782.23 | bwd_allreduce: 13.23 | step: 19.09

 18%|█▊        | 7333/41250 [17:42:53<82:10:07,  8.72s/it]
                                                          
{'loss': 0.0952, 'grad_norm': 1.022873044013977, 'learning_rate': 3.775321048719885e-05, 'epoch': 1.78}

 18%|█▊        | 7333/41250 [17:42:53<82:10:07,  8.72s/it][2025-04-26 01:40:36,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 01:40:36,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.67 | bwd_microstep: 5810.10 | bwd_inner_microstep: 5655.57 | bwd_allreduce_microstep: 154.48 | step_microstep: 18.73
[2025-04-26 01:40:36,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.67 | bwd: 5810.11 | bwd_inner: 5655.57 | bwd_allreduce: 154.50 | step: 18.73

 18%|█▊        | 7334/41250 [17:43:01<82:10:04,  8.72s/it]
                                                          
{'loss': 0.1773, 'grad_norm': 3.7946696281433105, 'learning_rate': 3.7752487300192386e-05, 'epoch': 1.78}

 18%|█▊        | 7334/41250 [17:43:01<82:10:04,  8.72s/it][2025-04-26 01:40:45,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:40:45,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.43 | bwd_microstep: 5743.66 | bwd_inner_microstep: 5696.31 | bwd_allreduce_microstep: 47.30 | step_microstep: 18.69
[2025-04-26 01:40:45,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.43 | bwd: 5743.67 | bwd_inner: 5696.31 | bwd_allreduce: 47.32 | step: 18.69

 18%|█▊        | 7335/41250 [17:43:10<82:02:43,  8.71s/it]
                                                          
{'loss': 0.3864, 'grad_norm': 2.285442352294922, 'learning_rate': 3.775176400374533e-05, 'epoch': 1.78}

 18%|█▊        | 7335/41250 [17:43:10<82:02:43,  8.71s/it][2025-04-26 01:40:53,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.11
[2025-04-26 01:40:53,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.78 | bwd_microstep: 5750.75 | bwd_inner_microstep: 5708.18 | bwd_allreduce_microstep: 42.51 | step_microstep: 19.65
[2025-04-26 01:40:53,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.78 | bwd: 5750.76 | bwd_inner: 5708.18 | bwd_allreduce: 42.54 | step: 19.65

 18%|█▊        | 7336/41250 [17:43:19<81:59:26,  8.70s/it]
                                                          
{'loss': 0.1649, 'grad_norm': 1.5986312627792358, 'learning_rate': 3.775104059786215e-05, 'epoch': 1.78}

 18%|█▊        | 7336/41250 [17:43:19<81:59:26,  8.70s/it][2025-04-26 01:41:02,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:41:02,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.43 | bwd_microstep: 5716.89 | bwd_inner_microstep: 5704.08 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.63
[2025-04-26 01:41:02,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.43 | bwd: 5716.90 | bwd_inner: 5704.07 | bwd_allreduce: 12.79 | step: 18.63

 18%|█▊        | 7337/41250 [17:43:27<81:50:39,  8.69s/it]
                                                          
{'loss': 0.2217, 'grad_norm': 2.691789388656616, 'learning_rate': 3.775031708254728e-05, 'epoch': 1.78}

 18%|█▊        | 7337/41250 [17:43:27<81:50:39,  8.69s/it][2025-04-26 01:41:11,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.16 | optimizer_step: 1.02
[2025-04-26 01:41:11,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.13 | bwd_microstep: 5793.98 | bwd_inner_microstep: 5780.63 | bwd_allreduce_microstep: 13.29 | step_microstep: 19.11
[2025-04-26 01:41:11,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.13 | bwd: 5793.99 | bwd_inner: 5780.63 | bwd_allreduce: 13.32 | step: 19.12

 18%|█▊        | 7338/41250 [17:43:36<82:04:45,  8.71s/it]
                                                          
{'loss': 0.2367, 'grad_norm': 2.8631021976470947, 'learning_rate': 3.774959345780521e-05, 'epoch': 1.78}

 18%|█▊        | 7338/41250 [17:43:36<82:04:45,  8.71s/it][2025-04-26 01:41:19,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 01:41:19,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5703.35 | bwd_inner_microstep: 5690.30 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.63
[2025-04-26 01:41:19,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5703.36 | bwd_inner: 5690.30 | bwd_allreduce: 13.02 | step: 18.63

 18%|█▊        | 7339/41250 [17:43:45<81:50:08,  8.69s/it]
                                                          
{'loss': 0.1032, 'grad_norm': 1.7478529214859009, 'learning_rate': 3.774886972364039e-05, 'epoch': 1.78}

 18%|█▊        | 7339/41250 [17:43:45<81:50:08,  8.69s/it][2025-04-26 01:41:28,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:41:28,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.10 | bwd_microstep: 5707.91 | bwd_inner_microstep: 5693.00 | bwd_allreduce_microstep: 14.86 | step_microstep: 18.85
[2025-04-26 01:41:28,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.10 | bwd: 5707.92 | bwd_inner: 5693.00 | bwd_allreduce: 14.87 | step: 18.85

 18%|█▊        | 7340/41250 [17:43:53<81:42:02,  8.67s/it]
                                                          
{'loss': 0.256, 'grad_norm': 1.8641027212142944, 'learning_rate': 3.774814588005727e-05, 'epoch': 1.78}

 18%|█▊        | 7340/41250 [17:43:53<81:42:02,  8.67s/it][2025-04-26 01:41:37,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 01:41:37,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5715.13 | bwd_inner_microstep: 5657.08 | bwd_allreduce_microstep: 58.00 | step_microstep: 18.55
[2025-04-26 01:41:37,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5715.14 | bwd_inner: 5657.08 | bwd_allreduce: 58.02 | step: 18.55

 18%|█▊        | 7341/41250 [17:44:02<81:35:15,  8.66s/it]
                                                          
{'loss': 0.1784, 'grad_norm': 1.7995364665985107, 'learning_rate': 3.774742192706032e-05, 'epoch': 1.78}

 18%|█▊        | 7341/41250 [17:44:02<81:35:15,  8.66s/it][2025-04-26 01:41:45,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 01:41:45,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.00 | bwd_microstep: 5701.15 | bwd_inner_microstep: 5688.16 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.41
[2025-04-26 01:41:45,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.00 | bwd: 5701.17 | bwd_inner: 5688.16 | bwd_allreduce: 12.96 | step: 19.42

 18%|█▊        | 7342/41250 [17:44:11<81:30:23,  8.65s/it]
                                                          
{'loss': 0.1308, 'grad_norm': 1.657130479812622, 'learning_rate': 3.774669786465401e-05, 'epoch': 1.78}

 18%|█▊        | 7342/41250 [17:44:11<81:30:23,  8.65s/it][2025-04-26 01:41:54,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:41:54,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.97 | bwd_microstep: 5788.64 | bwd_inner_microstep: 5663.20 | bwd_allreduce_microstep: 125.39 | step_microstep: 18.39
[2025-04-26 01:41:54,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.97 | bwd: 5788.65 | bwd_inner: 5663.20 | bwd_allreduce: 125.41 | step: 18.39

 18%|█▊        | 7343/41250 [17:44:19<81:37:45,  8.67s/it]
                                                          
{'loss': 0.1941, 'grad_norm': 3.9189586639404297, 'learning_rate': 3.77459736928428e-05, 'epoch': 1.78}

 18%|█▊        | 7343/41250 [17:44:19<81:37:45,  8.67s/it][2025-04-26 01:42:03,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:42:03,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5793.53 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 139.99 | step_microstep: 18.61
[2025-04-26 01:42:03,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5793.55 | bwd_inner: 5653.50 | bwd_allreduce: 140.00 | step: 18.61

 18%|█▊        | 7344/41250 [17:44:28<81:43:41,  8.68s/it]
                                                          
{'loss': 0.4, 'grad_norm': 2.202507972717285, 'learning_rate': 3.774524941163115e-05, 'epoch': 1.78}

 18%|█▊        | 7344/41250 [17:44:28<81:43:41,  8.68s/it][2025-04-26 01:42:11,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 01:42:11,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.06 | bwd_microstep: 5796.71 | bwd_inner_microstep: 5658.22 | bwd_allreduce_microstep: 138.45 | step_microstep: 18.55
[2025-04-26 01:42:11,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.06 | bwd: 5796.73 | bwd_inner: 5658.22 | bwd_allreduce: 138.47 | step: 18.55

 18%|█▊        | 7345/41250 [17:44:37<81:47:57,  8.69s/it]
                                                          
{'loss': 0.1259, 'grad_norm': 3.3318843841552734, 'learning_rate': 3.774452502102353e-05, 'epoch': 1.78}

 18%|█▊        | 7345/41250 [17:44:37<81:47:57,  8.69s/it][2025-04-26 01:42:20,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:42:20,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.38 | bwd_microstep: 5773.04 | bwd_inner_microstep: 5654.42 | bwd_allreduce_microstep: 118.57 | step_microstep: 18.70
[2025-04-26 01:42:20,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.38 | bwd: 5773.05 | bwd_inner: 5654.42 | bwd_allreduce: 118.59 | step: 18.70

 18%|█▊        | 7346/41250 [17:44:45<81:47:07,  8.68s/it]
                                                          
{'loss': 0.1034, 'grad_norm': 1.0434978008270264, 'learning_rate': 3.7743800521024395e-05, 'epoch': 1.78}

 18%|█▊        | 7346/41250 [17:44:45<81:47:07,  8.68s/it][2025-04-26 01:42:29,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 01:42:29,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5769.90 | bwd_inner_microstep: 5703.52 | bwd_allreduce_microstep: 66.33 | step_microstep: 19.35
[2025-04-26 01:42:29,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 5769.91 | bwd_inner: 5703.52 | bwd_allreduce: 66.35 | step: 19.35

 18%|█▊        | 7347/41250 [17:44:54<81:50:45,  8.69s/it]
                                                          
{'loss': 0.1765, 'grad_norm': 1.9011317491531372, 'learning_rate': 3.774307591163823e-05, 'epoch': 1.78}

 18%|█▊        | 7347/41250 [17:44:54<81:50:45,  8.69s/it][2025-04-26 01:42:37,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 01:42:37,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5714.25 | bwd_inner_microstep: 5701.37 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.48
[2025-04-26 01:42:37,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5714.27 | bwd_inner: 5701.37 | bwd_allreduce: 12.86 | step: 18.48

 18%|█▊        | 7348/41250 [17:45:03<81:43:09,  8.68s/it]
                                                          
{'loss': 0.0467, 'grad_norm': 1.5716816186904907, 'learning_rate': 3.774235119286949e-05, 'epoch': 1.78}

 18%|█▊        | 7348/41250 [17:45:03<81:43:09,  8.68s/it][2025-04-26 01:42:46,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:42:46,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5644.68 | bwd_allreduce_microstep: 60.16 | step_microstep: 18.51
[2025-04-26 01:42:46,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.16 | bwd: 5704.89 | bwd_inner: 5644.68 | bwd_allreduce: 60.17 | step: 18.51

 18%|█▊        | 7349/41250 [17:45:11<81:32:15,  8.66s/it]
                                                          
{'loss': 0.1703, 'grad_norm': 1.3081060647964478, 'learning_rate': 3.774162636472264e-05, 'epoch': 1.78}

 18%|█▊        | 7349/41250 [17:45:11<81:32:15,  8.66s/it][2025-04-26 01:42:55,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 01:42:55,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5706.60 | bwd_inner_microstep: 5693.91 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.35
[2025-04-26 01:42:55,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5706.61 | bwd_inner: 5693.91 | bwd_allreduce: 12.66 | step: 18.35

 18%|█▊        | 7350/41250 [17:45:20<81:28:02,  8.65s/it]
                                                          
{'loss': 0.1894, 'grad_norm': 1.5274369716644287, 'learning_rate': 3.774090142720216e-05, 'epoch': 1.78}

 18%|█▊        | 7350/41250 [17:45:20<81:28:02,  8.65s/it][2025-04-26 01:43:03,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 01:43:03,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.03 | bwd_microstep: 5770.66 | bwd_inner_microstep: 5687.10 | bwd_allreduce_microstep: 83.52 | step_microstep: 18.53
[2025-04-26 01:43:03,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.03 | bwd: 5770.67 | bwd_inner: 5687.10 | bwd_allreduce: 83.54 | step: 18.54

 18%|█▊        | 7351/41250 [17:45:29<81:35:25,  8.66s/it]
                                                          
{'loss': 0.0495, 'grad_norm': 0.5996806621551514, 'learning_rate': 3.7740176380312504e-05, 'epoch': 1.78}

 18%|█▊        | 7351/41250 [17:45:29<81:35:25,  8.66s/it][2025-04-26 01:43:12,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:43:12,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.92 | bwd_microstep: 5697.76 | bwd_inner_microstep: 5685.14 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.60
[2025-04-26 01:43:12,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.92 | bwd: 5697.77 | bwd_inner: 5685.14 | bwd_allreduce: 12.58 | step: 18.61

 18%|█▊        | 7352/41250 [17:45:37<81:28:43,  8.65s/it]
                                                          
{'loss': 0.3292, 'grad_norm': 1.6043404340744019, 'learning_rate': 3.773945122405815e-05, 'epoch': 1.78}

 18%|█▊        | 7352/41250 [17:45:37<81:28:43,  8.65s/it][2025-04-26 01:43:21,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 1.10
[2025-04-26 01:43:21,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.25 | bwd_microstep: 5786.84 | bwd_inner_microstep: 5773.78 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.38
[2025-04-26 01:43:21,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.25 | bwd: 5786.85 | bwd_inner: 5773.78 | bwd_allreduce: 13.02 | step: 19.39

 18%|█▊        | 7353/41250 [17:45:46<81:46:35,  8.68s/it]
                                                          
{'loss': 0.0612, 'grad_norm': 0.5108978748321533, 'learning_rate': 3.773872595844357e-05, 'epoch': 1.78}

 18%|█▊        | 7353/41250 [17:45:46<81:46:35,  8.68s/it][2025-04-26 01:43:29,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 01:43:29,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.32 | bwd_microstep: 5705.58 | bwd_inner_microstep: 5692.85 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.26
[2025-04-26 01:43:29,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.32 | bwd: 5705.59 | bwd_inner: 5692.85 | bwd_allreduce: 12.70 | step: 19.26

 18%|█▊        | 7354/41250 [17:45:55<81:39:45,  8.67s/it]
                                                          
{'loss': 0.2674, 'grad_norm': 4.1430583000183105, 'learning_rate': 3.7738000583473235e-05, 'epoch': 1.78}

 18%|█▊        | 7354/41250 [17:45:55<81:39:45,  8.67s/it][2025-04-26 01:43:38,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:43:38,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5783.62 | bwd_inner_microstep: 5651.76 | bwd_allreduce_microstep: 131.81 | step_microstep: 18.64
[2025-04-26 01:43:38,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5783.64 | bwd_inner: 5651.76 | bwd_allreduce: 131.83 | step: 18.64

 18%|█▊        | 7355/41250 [17:46:03<81:43:12,  8.68s/it]
                                                          
{'loss': 0.0699, 'grad_norm': 0.8117111325263977, 'learning_rate': 3.773727509915162e-05, 'epoch': 1.78}

 18%|█▊        | 7355/41250 [17:46:03<81:43:12,  8.68s/it][2025-04-26 01:43:47,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 01:43:47,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.57 | bwd_microstep: 5692.37 | bwd_inner_microstep: 5679.53 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.03
[2025-04-26 01:43:47,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.57 | bwd: 5692.38 | bwd_inner: 5679.53 | bwd_allreduce: 12.81 | step: 19.03

 18%|█▊        | 7356/41250 [17:46:12<81:34:42,  8.66s/it]
                                                          
{'loss': 0.0572, 'grad_norm': 2.8607735633850098, 'learning_rate': 3.773654950548319e-05, 'epoch': 1.78}

 18%|█▊        | 7356/41250 [17:46:12<81:34:42,  8.66s/it][2025-04-26 01:43:55,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:43:55,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5709.25 | bwd_inner_microstep: 5696.65 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.76
[2025-04-26 01:43:55,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5709.27 | bwd_inner: 5696.65 | bwd_allreduce: 12.57 | step: 18.76

 18%|█▊        | 7357/41250 [17:46:21<81:32:20,  8.66s/it]
                                                          
{'loss': 0.036, 'grad_norm': 0.3244190812110901, 'learning_rate': 3.7735823802472416e-05, 'epoch': 1.78}

 18%|█▊        | 7357/41250 [17:46:21<81:32:20,  8.66s/it][2025-04-26 01:44:04,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:44:04,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.33 | bwd_microstep: 5763.02 | bwd_inner_microstep: 5641.95 | bwd_allreduce_microstep: 121.03 | step_microstep: 18.94
[2025-04-26 01:44:04,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.34 | bwd: 5763.04 | bwd_inner: 5641.95 | bwd_allreduce: 121.05 | step: 18.94

 18%|█▊        | 7358/41250 [17:46:29<81:34:00,  8.66s/it]
                                                          
{'loss': 0.0307, 'grad_norm': 0.6103740930557251, 'learning_rate': 3.7735097990123784e-05, 'epoch': 1.78}

 18%|█▊        | 7358/41250 [17:46:29<81:34:00,  8.66s/it][2025-04-26 01:44:13,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 01:44:13,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5747.79 | bwd_inner_microstep: 5688.87 | bwd_allreduce_microstep: 58.86 | step_microstep: 19.37
[2025-04-26 01:44:13,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5747.80 | bwd_inner: 5688.87 | bwd_allreduce: 58.89 | step: 19.37

 18%|█▊        | 7359/41250 [17:46:38<81:37:11,  8.67s/it]
                                                          
{'loss': 0.1423, 'grad_norm': 0.9627006649971008, 'learning_rate': 3.773437206844175e-05, 'epoch': 1.78}

 18%|█▊        | 7359/41250 [17:46:38<81:37:11,  8.67s/it][2025-04-26 01:44:21,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:44:21,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.20 | bwd_microstep: 5692.84 | bwd_inner_microstep: 5680.10 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.76
[2025-04-26 01:44:21,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.20 | bwd: 5692.86 | bwd_inner: 5680.10 | bwd_allreduce: 12.72 | step: 18.76

 18%|█▊        | 7360/41250 [17:46:47<81:28:38,  8.66s/it]
                                                          
{'loss': 0.2296, 'grad_norm': 2.1557741165161133, 'learning_rate': 3.773364603743081e-05, 'epoch': 1.78}

 18%|█▊        | 7360/41250 [17:46:47<81:28:38,  8.66s/it][2025-04-26 01:44:30,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:44:30,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.20 | bwd_microstep: 5701.07 | bwd_inner_microstep: 5688.35 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.03
[2025-04-26 01:44:30,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.20 | bwd: 5701.09 | bwd_inner: 5688.35 | bwd_allreduce: 12.70 | step: 19.03

 18%|█▊        | 7361/41250 [17:46:55<81:24:09,  8.65s/it]
                                                          
{'loss': 0.1294, 'grad_norm': 3.329404830932617, 'learning_rate': 3.7732919897095425e-05, 'epoch': 1.78}

 18%|█▊        | 7361/41250 [17:46:55<81:24:09,  8.65s/it][2025-04-26 01:44:39,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:44:39,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.09 | bwd_microstep: 5695.59 | bwd_inner_microstep: 5682.46 | bwd_allreduce_microstep: 13.08 | step_microstep: 19.13
[2025-04-26 01:44:39,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.09 | bwd: 5695.61 | bwd_inner: 5682.46 | bwd_allreduce: 13.10 | step: 19.13

 18%|█▊        | 7362/41250 [17:47:04<81:20:29,  8.64s/it]
                                                          
{'loss': 0.2198, 'grad_norm': 2.3834402561187744, 'learning_rate': 3.773219364744008e-05, 'epoch': 1.78}

 18%|█▊        | 7362/41250 [17:47:04<81:20:29,  8.64s/it][2025-04-26 01:44:47,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:44:47,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.49 | bwd_microstep: 5718.88 | bwd_inner_microstep: 5678.92 | bwd_allreduce_microstep: 39.91 | step_microstep: 18.81
[2025-04-26 01:44:47,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.49 | bwd: 5718.89 | bwd_inner: 5678.92 | bwd_allreduce: 39.93 | step: 18.82

 18%|█▊        | 7363/41250 [17:47:13<81:22:18,  8.64s/it]
                                                          
{'loss': 0.1845, 'grad_norm': 1.297350525856018, 'learning_rate': 3.773146728846925e-05, 'epoch': 1.78}

 18%|█▊        | 7363/41250 [17:47:13<81:22:18,  8.64s/it][2025-04-26 01:44:56,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 01:44:56,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.58 | bwd_microstep: 5714.43 | bwd_inner_microstep: 5701.69 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.87
[2025-04-26 01:44:56,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.58 | bwd: 5714.44 | bwd_inner: 5701.69 | bwd_allreduce: 12.71 | step: 18.87

 18%|█▊        | 7364/41250 [17:47:21<81:23:20,  8.65s/it]
                                                          
{'loss': 0.0957, 'grad_norm': 2.115776538848877, 'learning_rate': 3.773074082018741e-05, 'epoch': 1.79}

 18%|█▊        | 7364/41250 [17:47:21<81:23:20,  8.65s/it][2025-04-26 01:45:04,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:45:04,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.63 | bwd_microstep: 5701.76 | bwd_inner_microstep: 5653.36 | bwd_allreduce_microstep: 48.35 | step_microstep: 18.59
[2025-04-26 01:45:04,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.63 | bwd: 5701.78 | bwd_inner: 5653.36 | bwd_allreduce: 48.37 | step: 18.59

 18%|█▊        | 7365/41250 [17:47:30<81:17:00,  8.64s/it]
                                                          
{'loss': 0.0796, 'grad_norm': 1.1507405042648315, 'learning_rate': 3.7730014242599036e-05, 'epoch': 1.79}

 18%|█▊        | 7365/41250 [17:47:30<81:17:00,  8.64s/it][2025-04-26 01:45:13,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 01:45:13,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.68 | bwd_microstep: 5757.71 | bwd_inner_microstep: 5642.47 | bwd_allreduce_microstep: 115.20 | step_microstep: 18.71
[2025-04-26 01:45:13,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.68 | bwd: 5757.72 | bwd_inner: 5642.47 | bwd_allreduce: 115.21 | step: 18.71

 18%|█▊        | 7366/41250 [17:47:38<81:21:46,  8.64s/it]
                                                          
{'loss': 0.1317, 'grad_norm': 2.4989876747131348, 'learning_rate': 3.772928755570862e-05, 'epoch': 1.79}

 18%|█▊        | 7366/41250 [17:47:38<81:21:46,  8.64s/it][2025-04-26 01:45:22,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:45:22,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.17 | bwd_microstep: 5747.33 | bwd_inner_microstep: 5677.50 | bwd_allreduce_microstep: 69.79 | step_microstep: 18.38
[2025-04-26 01:45:22,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.17 | bwd: 5747.34 | bwd_inner: 5677.50 | bwd_allreduce: 69.81 | step: 18.38

 18%|█▊        | 7367/41250 [17:47:47<81:27:26,  8.65s/it]
                                                          
{'loss': 0.1029, 'grad_norm': 2.081782817840576, 'learning_rate': 3.772856075952063e-05, 'epoch': 1.79}

 18%|█▊        | 7367/41250 [17:47:47<81:27:26,  8.65s/it][2025-04-26 01:45:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:45:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.97 | bwd_microstep: 5758.58 | bwd_inner_microstep: 5651.09 | bwd_allreduce_microstep: 107.45 | step_microstep: 18.49
[2025-04-26 01:45:31,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.97 | bwd: 5758.59 | bwd_inner: 5651.09 | bwd_allreduce: 107.47 | step: 18.49

 18%|█▊        | 7368/41250 [17:47:56<81:30:14,  8.66s/it]
                                                          
{'loss': 0.1357, 'grad_norm': 1.312106966972351, 'learning_rate': 3.772783385403955e-05, 'epoch': 1.79}

 18%|█▊        | 7368/41250 [17:47:56<81:30:14,  8.66s/it][2025-04-26 01:45:39,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:45:39,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.91 | bwd_microstep: 5785.69 | bwd_inner_microstep: 5772.87 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.59
[2025-04-26 01:45:39,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.91 | bwd: 5785.70 | bwd_inner: 5772.87 | bwd_allreduce: 12.79 | step: 18.59

 18%|█▊        | 7369/41250 [17:48:05<81:45:50,  8.69s/it]
                                                          
{'loss': 0.2534, 'grad_norm': 3.2068629264831543, 'learning_rate': 3.772710683926985e-05, 'epoch': 1.79}

 18%|█▊        | 7369/41250 [17:48:05<81:45:50,  8.69s/it][2025-04-26 01:45:48,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:45:48,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.43 | bwd_microstep: 5692.00 | bwd_inner_microstep: 5679.19 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.57
[2025-04-26 01:45:48,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.43 | bwd: 5692.01 | bwd_inner: 5679.19 | bwd_allreduce: 12.78 | step: 18.57

 18%|█▊        | 7370/41250 [17:48:13<81:33:46,  8.67s/it]
                                                          
{'loss': 0.0711, 'grad_norm': 0.9003294110298157, 'learning_rate': 3.772637971521604e-05, 'epoch': 1.79}

 18%|█▊        | 7370/41250 [17:48:13<81:33:46,  8.67s/it][2025-04-26 01:45:57,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:45:57,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.77 | bwd_microstep: 5733.16 | bwd_inner_microstep: 5691.93 | bwd_allreduce_microstep: 41.19 | step_microstep: 18.51
[2025-04-26 01:45:57,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.77 | bwd: 5733.17 | bwd_inner: 5691.93 | bwd_allreduce: 41.20 | step: 18.51

 18%|█▊        | 7371/41250 [17:48:22<81:32:49,  8.67s/it]
                                                          
{'loss': 0.1231, 'grad_norm': 2.0468478202819824, 'learning_rate': 3.772565248188258e-05, 'epoch': 1.79}

 18%|█▊        | 7371/41250 [17:48:22<81:32:49,  8.67s/it][2025-04-26 01:46:05,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.08 | optimizer_step: 1.13
[2025-04-26 01:46:05,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.50 | bwd_microstep: 5710.10 | bwd_inner_microstep: 5697.25 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.79
[2025-04-26 01:46:05,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.50 | bwd: 5710.11 | bwd_inner: 5697.25 | bwd_allreduce: 12.82 | step: 19.79

 18%|█▊        | 7372/41250 [17:48:31<81:29:40,  8.66s/it]
                                                          
{'loss': 0.1718, 'grad_norm': 1.3112399578094482, 'learning_rate': 3.772492513927396e-05, 'epoch': 1.79}

 18%|█▊        | 7372/41250 [17:48:31<81:29:40,  8.66s/it][2025-04-26 01:46:14,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:46:14,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.85 | bwd_microstep: 5767.77 | bwd_inner_microstep: 5689.00 | bwd_allreduce_microstep: 78.72 | step_microstep: 18.35
[2025-04-26 01:46:14,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.85 | bwd: 5767.78 | bwd_inner: 5689.00 | bwd_allreduce: 78.73 | step: 18.35

 18%|█▊        | 7373/41250 [17:48:39<81:35:57,  8.67s/it]
                                                          
{'loss': 0.0504, 'grad_norm': 1.0288970470428467, 'learning_rate': 3.772419768739466e-05, 'epoch': 1.79}

 18%|█▊        | 7373/41250 [17:48:39<81:35:57,  8.67s/it][2025-04-26 01:46:23,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 01:46:23,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.95 | bwd_microstep: 5764.36 | bwd_inner_microstep: 5643.28 | bwd_allreduce_microstep: 121.04 | step_microstep: 18.92
[2025-04-26 01:46:23,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.95 | bwd: 5764.38 | bwd_inner: 5643.28 | bwd_allreduce: 121.06 | step: 18.92

 18%|█▊        | 7374/41250 [17:48:48<81:35:10,  8.67s/it]
                                                          
{'loss': 0.1922, 'grad_norm': 2.811234474182129, 'learning_rate': 3.7723470126249167e-05, 'epoch': 1.79}

 18%|█▊        | 7374/41250 [17:48:48<81:35:10,  8.67s/it][2025-04-26 01:46:31,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:46:31,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.78 | bwd_microstep: 5709.31 | bwd_inner_microstep: 5685.41 | bwd_allreduce_microstep: 23.86 | step_microstep: 18.72
[2025-04-26 01:46:31,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.78 | bwd: 5709.33 | bwd_inner: 5685.40 | bwd_allreduce: 23.88 | step: 18.72

 18%|█▊        | 7375/41250 [17:48:57<81:28:39,  8.66s/it]
                                                          
{'loss': 0.167, 'grad_norm': 1.302159309387207, 'learning_rate': 3.772274245584197e-05, 'epoch': 1.79}

 18%|█▊        | 7375/41250 [17:48:57<81:28:39,  8.66s/it][2025-04-26 01:46:40,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 01:46:40,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.39 | bwd_microstep: 5789.60 | bwd_inner_microstep: 5776.93 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.01
[2025-04-26 01:46:40,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.39 | bwd: 5789.61 | bwd_inner: 5776.93 | bwd_allreduce: 12.64 | step: 19.01

 18%|█▊        | 7376/41250 [17:49:05<81:46:22,  8.69s/it]
                                                          
{'loss': 0.148, 'grad_norm': 2.0100724697113037, 'learning_rate': 3.7722014676177546e-05, 'epoch': 1.79}

 18%|█▊        | 7376/41250 [17:49:05<81:46:22,  8.69s/it][2025-04-26 01:46:49,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 01:46:49,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5773.14 | bwd_inner_microstep: 5647.90 | bwd_allreduce_microstep: 125.18 | step_microstep: 18.75
[2025-04-26 01:46:49,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5773.15 | bwd_inner: 5647.90 | bwd_allreduce: 125.20 | step: 18.75

 18%|█▊        | 7377/41250 [17:49:14<81:45:05,  8.69s/it]
                                                          
{'loss': 0.0312, 'grad_norm': 1.6631090641021729, 'learning_rate': 3.772128678726039e-05, 'epoch': 1.79}

 18%|█▊        | 7377/41250 [17:49:14<81:45:05,  8.69s/it][2025-04-26 01:46:57,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:46:57,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 5789.92 | bwd_inner_microstep: 5661.35 | bwd_allreduce_microstep: 128.53 | step_microstep: 18.70
[2025-04-26 01:46:57,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.28 | bwd: 5789.94 | bwd_inner: 5661.35 | bwd_allreduce: 128.55 | step: 18.70

 18%|█▊        | 7378/41250 [17:49:23<81:47:35,  8.69s/it]
                                                          
{'loss': 0.1148, 'grad_norm': 1.655745506286621, 'learning_rate': 3.772055878909499e-05, 'epoch': 1.79}

 18%|█▊        | 7378/41250 [17:49:23<81:47:35,  8.69s/it][2025-04-26 01:47:06,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:47:06,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.42 | bwd_microstep: 5767.09 | bwd_inner_microstep: 5656.71 | bwd_allreduce_microstep: 110.33 | step_microstep: 18.51
[2025-04-26 01:47:06,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.42 | bwd: 5767.11 | bwd_inner: 5656.71 | bwd_allreduce: 110.35 | step: 18.51

 18%|█▊        | 7379/41250 [17:49:31<81:46:23,  8.69s/it]
                                                          
{'loss': 0.1655, 'grad_norm': 3.676917552947998, 'learning_rate': 3.771983068168583e-05, 'epoch': 1.79}

 18%|█▊        | 7379/41250 [17:49:31<81:46:23,  8.69s/it][2025-04-26 01:47:15,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 01:47:15,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.90 | bwd_microstep: 5776.71 | bwd_inner_microstep: 5655.87 | bwd_allreduce_microstep: 120.79 | step_microstep: 19.16
[2025-04-26 01:47:15,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.90 | bwd: 5776.72 | bwd_inner: 5655.87 | bwd_allreduce: 120.81 | step: 19.16

 18%|█▊        | 7380/41250 [17:49:40<81:45:55,  8.69s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 2.4421563148498535, 'learning_rate': 3.771910246503739e-05, 'epoch': 1.79}

 18%|█▊        | 7380/41250 [17:49:40<81:45:55,  8.69s/it][2025-04-26 01:47:23,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:47:23,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.67 | bwd_microstep: 5770.68 | bwd_inner_microstep: 5663.07 | bwd_allreduce_microstep: 107.57 | step_microstep: 18.41
[2025-04-26 01:47:23,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.67 | bwd: 5770.69 | bwd_inner: 5663.07 | bwd_allreduce: 107.59 | step: 18.41

 18%|█▊        | 7381/41250 [17:49:49<81:44:39,  8.69s/it]
                                                          
{'loss': 0.4239, 'grad_norm': 7.668618679046631, 'learning_rate': 3.771837413915418e-05, 'epoch': 1.79}

 18%|█▊        | 7381/41250 [17:49:49<81:44:39,  8.69s/it][2025-04-26 01:47:32,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 01:47:32,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5716.44 | bwd_inner_microstep: 5664.44 | bwd_allreduce_microstep: 51.96 | step_microstep: 18.57
[2025-04-26 01:47:32,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5716.45 | bwd_inner: 5664.44 | bwd_allreduce: 51.98 | step: 18.57

 18%|█▊        | 7382/41250 [17:49:57<81:35:03,  8.67s/it]
                                                          
{'loss': 0.1183, 'grad_norm': 1.5632778406143188, 'learning_rate': 3.7717645704040676e-05, 'epoch': 1.79}

 18%|█▊        | 7382/41250 [17:49:57<81:35:03,  8.67s/it][2025-04-26 01:47:41,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 01:47:41,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.09 | bwd_microstep: 5759.93 | bwd_inner_microstep: 5706.46 | bwd_allreduce_microstep: 53.42 | step_microstep: 18.70
[2025-04-26 01:47:41,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.09 | bwd: 5759.94 | bwd_inner: 5706.46 | bwd_allreduce: 53.44 | step: 18.70

 18%|█▊        | 7383/41250 [17:50:06<81:42:15,  8.69s/it]
                                                          
{'loss': 0.4174, 'grad_norm': 2.186918020248413, 'learning_rate': 3.7716917159701365e-05, 'epoch': 1.79}

 18%|█▊        | 7383/41250 [17:50:06<81:42:15,  8.69s/it][2025-04-26 01:47:49,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:47:49,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.31 | bwd_microstep: 5716.20 | bwd_inner_microstep: 5703.24 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.41
[2025-04-26 01:47:49,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.31 | bwd: 5716.21 | bwd_inner: 5703.24 | bwd_allreduce: 12.93 | step: 18.41

 18%|█▊        | 7384/41250 [17:50:15<81:38:17,  8.68s/it]
                                                          
{'loss': 0.2937, 'grad_norm': 2.2689244747161865, 'learning_rate': 3.771618850614075e-05, 'epoch': 1.79}

 18%|█▊        | 7384/41250 [17:50:15<81:38:17,  8.68s/it][2025-04-26 01:47:58,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 01:47:58,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.37 | bwd_microstep: 5701.76 | bwd_inner_microstep: 5682.86 | bwd_allreduce_microstep: 18.86 | step_microstep: 19.07
[2025-04-26 01:47:58,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5701.78 | bwd_inner: 5682.86 | bwd_allreduce: 18.87 | step: 19.08

 18%|█▊        | 7385/41250 [17:50:23<81:31:06,  8.67s/it]
                                                          
{'loss': 0.2345, 'grad_norm': 3.1348259449005127, 'learning_rate': 3.7715459743363314e-05, 'epoch': 1.79}

 18%|█▊        | 7385/41250 [17:50:23<81:31:06,  8.67s/it][2025-04-26 01:48:07,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 01:48:07,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.78 | bwd_microstep: 5868.47 | bwd_inner_microstep: 5687.84 | bwd_allreduce_microstep: 180.58 | step_microstep: 19.11
[2025-04-26 01:48:07,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.79 | bwd: 5868.48 | bwd_inner: 5687.84 | bwd_allreduce: 180.60 | step: 19.11

 18%|█▊        | 7386/41250 [17:50:32<81:52:50,  8.70s/it]
                                                          
{'loss': 0.1433, 'grad_norm': 1.9279239177703857, 'learning_rate': 3.771473087137356e-05, 'epoch': 1.79}

 18%|█▊        | 7386/41250 [17:50:32<81:52:50,  8.70s/it][2025-04-26 01:48:15,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 01:48:15,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.96 | bwd_microstep: 5693.87 | bwd_inner_microstep: 5661.27 | bwd_allreduce_microstep: 32.55 | step_microstep: 18.75
[2025-04-26 01:48:15,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.96 | bwd: 5693.88 | bwd_inner: 5661.27 | bwd_allreduce: 32.57 | step: 18.76

 18%|█▊        | 7387/41250 [17:50:41<81:38:31,  8.68s/it]
                                                          
{'loss': 0.0522, 'grad_norm': 1.8846468925476074, 'learning_rate': 3.771400189017596e-05, 'epoch': 1.79}

 18%|█▊        | 7387/41250 [17:50:41<81:38:31,  8.68s/it][2025-04-26 01:48:24,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:48:24,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.28 | bwd_microstep: 5758.38 | bwd_inner_microstep: 5661.56 | bwd_allreduce_microstep: 96.78 | step_microstep: 18.56
[2025-04-26 01:48:24,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.28 | bwd: 5758.40 | bwd_inner: 5661.56 | bwd_allreduce: 96.80 | step: 18.57

 18%|█▊        | 7388/41250 [17:50:49<81:39:09,  8.68s/it]
                                                          
{'loss': 0.0627, 'grad_norm': 0.9422652125358582, 'learning_rate': 3.771327279977504e-05, 'epoch': 1.79}

 18%|█▊        | 7388/41250 [17:50:49<81:39:09,  8.68s/it][2025-04-26 01:48:33,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 01:48:33,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.09 | bwd_microstep: 5780.09 | bwd_inner_microstep: 5767.36 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.41
[2025-04-26 01:48:33,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.09 | bwd: 5780.11 | bwd_inner: 5767.36 | bwd_allreduce: 12.71 | step: 18.42

 18%|█▊        | 7389/41250 [17:50:58<81:50:10,  8.70s/it]
                                                          
{'loss': 0.0773, 'grad_norm': 4.236123085021973, 'learning_rate': 3.7712543600175264e-05, 'epoch': 1.79}

 18%|█▊        | 7389/41250 [17:50:58<81:50:10,  8.70s/it][2025-04-26 01:48:42,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:48:42,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5743.45 | bwd_inner_microstep: 5678.59 | bwd_allreduce_microstep: 64.81 | step_microstep: 18.72
[2025-04-26 01:48:42,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5743.46 | bwd_inner: 5678.59 | bwd_allreduce: 64.83 | step: 18.72

 18%|█▊        | 7390/41250 [17:51:07<81:46:52,  8.69s/it]
                                                          
{'loss': 0.1573, 'grad_norm': 2.5160255432128906, 'learning_rate': 3.771181429138115e-05, 'epoch': 1.79}

 18%|█▊        | 7390/41250 [17:51:07<81:46:52,  8.69s/it][2025-04-26 01:48:50,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:48:50,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.61 | bwd_microstep: 5752.37 | bwd_inner_microstep: 5650.08 | bwd_allreduce_microstep: 102.24 | step_microstep: 18.74
[2025-04-26 01:48:50,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.61 | bwd: 5752.38 | bwd_inner: 5650.08 | bwd_allreduce: 102.26 | step: 18.74

 18%|█▊        | 7391/41250 [17:51:16<81:43:23,  8.69s/it]
                                                          
{'loss': 0.0443, 'grad_norm': 1.2889180183410645, 'learning_rate': 3.771108487339718e-05, 'epoch': 1.79}

 18%|█▊        | 7391/41250 [17:51:16<81:43:23,  8.69s/it][2025-04-26 01:48:59,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 01:48:59,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.78 | bwd_microstep: 5785.73 | bwd_inner_microstep: 5648.33 | bwd_allreduce_microstep: 137.35 | step_microstep: 18.82
[2025-04-26 01:48:59,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.78 | bwd: 5785.74 | bwd_inner: 5648.33 | bwd_allreduce: 137.37 | step: 18.82

 18%|█▊        | 7392/41250 [17:51:24<81:44:22,  8.69s/it]
                                                          
{'loss': 0.2543, 'grad_norm': 1.6859601736068726, 'learning_rate': 3.7710355346227856e-05, 'epoch': 1.79}

 18%|█▊        | 7392/41250 [17:51:24<81:44:22,  8.69s/it][2025-04-26 01:49:08,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 01:49:08,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.42 | bwd_microstep: 5744.43 | bwd_inner_microstep: 5655.00 | bwd_allreduce_microstep: 89.39 | step_microstep: 18.75
[2025-04-26 01:49:08,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.42 | bwd: 5744.44 | bwd_inner: 5655.00 | bwd_allreduce: 89.41 | step: 18.76

 18%|█▊        | 7393/41250 [17:51:33<81:39:21,  8.68s/it]
                                                          
{'loss': 0.1538, 'grad_norm': 1.3703529834747314, 'learning_rate': 3.770962570987767e-05, 'epoch': 1.79}

 18%|█▊        | 7393/41250 [17:51:33<81:39:21,  8.68s/it][2025-04-26 01:49:16,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 01:49:16,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5692.47 | bwd_inner_microstep: 5653.37 | bwd_allreduce_microstep: 39.05 | step_microstep: 19.06
[2025-04-26 01:49:16,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5692.49 | bwd_inner: 5653.37 | bwd_allreduce: 39.07 | step: 19.06

 18%|█▊        | 7394/41250 [17:51:42<81:25:27,  8.66s/it]
                                                          
{'loss': 0.0998, 'grad_norm': 2.7657907009124756, 'learning_rate': 3.770889596435113e-05, 'epoch': 1.79}

 18%|█▊        | 7394/41250 [17:51:42<81:25:27,  8.66s/it][2025-04-26 01:49:25,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:49:25,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5738.37 | bwd_inner_microstep: 5702.03 | bwd_allreduce_microstep: 36.29 | step_microstep: 18.67
[2025-04-26 01:49:25,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5738.38 | bwd_inner: 5702.03 | bwd_allreduce: 36.31 | step: 18.67

 18%|█▊        | 7395/41250 [17:51:50<81:27:59,  8.66s/it]
                                                          
{'loss': 0.1032, 'grad_norm': 1.4548929929733276, 'learning_rate': 3.770816610965273e-05, 'epoch': 1.79}

 18%|█▊        | 7395/41250 [17:51:50<81:27:59,  8.66s/it][2025-04-26 01:49:34,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:49:34,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.20 | bwd_microstep: 5734.53 | bwd_inner_microstep: 5683.61 | bwd_allreduce_microstep: 50.89 | step_microstep: 18.50
[2025-04-26 01:49:34,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.20 | bwd: 5734.55 | bwd_inner: 5683.61 | bwd_allreduce: 50.90 | step: 18.50

 18%|█▊        | 7396/41250 [17:51:59<81:28:13,  8.66s/it]
                                                          
{'loss': 0.1254, 'grad_norm': 2.087845802307129, 'learning_rate': 3.770743614578696e-05, 'epoch': 1.79}

 18%|█▊        | 7396/41250 [17:51:59<81:28:13,  8.66s/it][2025-04-26 01:49:42,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:49:42,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.58 | bwd_microstep: 5786.21 | bwd_inner_microstep: 5773.22 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.05
[2025-04-26 01:49:42,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.58 | bwd: 5786.23 | bwd_inner: 5773.22 | bwd_allreduce: 12.97 | step: 19.05

 18%|█▊        | 7397/41250 [17:52:08<81:44:37,  8.69s/it]
                                                          
{'loss': 0.1191, 'grad_norm': 3.69673490524292, 'learning_rate': 3.770670607275834e-05, 'epoch': 1.79}

 18%|█▊        | 7397/41250 [17:52:08<81:44:37,  8.69s/it][2025-04-26 01:49:51,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:49:51,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.08 | bwd_microstep: 5703.44 | bwd_inner_microstep: 5636.77 | bwd_allreduce_microstep: 66.62 | step_microstep: 18.77
[2025-04-26 01:49:51,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.08 | bwd: 5703.45 | bwd_inner: 5636.77 | bwd_allreduce: 66.64 | step: 18.77

 18%|█▊        | 7398/41250 [17:52:16<81:30:19,  8.67s/it]
                                                          
{'loss': 0.0996, 'grad_norm': 1.4589790105819702, 'learning_rate': 3.770597589057136e-05, 'epoch': 1.79}

 18%|█▊        | 7398/41250 [17:52:16<81:30:19,  8.67s/it][2025-04-26 01:50:00,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 01:50:00,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.25 | bwd_microstep: 5824.12 | bwd_inner_microstep: 5763.10 | bwd_allreduce_microstep: 60.97 | step_microstep: 19.18
[2025-04-26 01:50:00,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.25 | bwd: 5824.13 | bwd_inner: 5763.10 | bwd_allreduce: 60.99 | step: 19.18

 18%|█▊        | 7399/41250 [17:52:25<81:50:29,  8.70s/it]
                                                          
{'loss': 0.1226, 'grad_norm': 1.7006642818450928, 'learning_rate': 3.770524559923052e-05, 'epoch': 1.79}

 18%|█▊        | 7399/41250 [17:52:25<81:50:29,  8.70s/it][2025-04-26 01:50:08,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:50:08,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.88 | bwd_microstep: 5745.42 | bwd_inner_microstep: 5685.19 | bwd_allreduce_microstep: 60.19 | step_microstep: 18.56
[2025-04-26 01:50:08,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.88 | bwd: 5745.44 | bwd_inner: 5685.19 | bwd_allreduce: 60.20 | step: 18.57

 18%|█▊        | 7400/41250 [17:52:34<81:46:14,  8.70s/it]
                                                          
{'loss': 0.079, 'grad_norm': 1.8377211093902588, 'learning_rate': 3.7704515198740315e-05, 'epoch': 1.79}

 18%|█▊        | 7400/41250 [17:52:34<81:46:14,  8.70s/it][2025-04-26 01:50:17,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 01:50:17,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.78 | bwd_microstep: 5773.18 | bwd_inner_microstep: 5631.90 | bwd_allreduce_microstep: 141.22 | step_microstep: 19.10
[2025-04-26 01:50:17,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.78 | bwd: 5773.19 | bwd_inner: 5631.90 | bwd_allreduce: 141.24 | step: 19.11

 18%|█▊        | 7401/41250 [17:52:42<81:43:03,  8.69s/it]
                                                          
{'loss': 0.0674, 'grad_norm': 1.033522367477417, 'learning_rate': 3.7703784689105266e-05, 'epoch': 1.79}

 18%|█▊        | 7401/41250 [17:52:42<81:43:03,  8.69s/it][2025-04-26 01:50:26,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 01:50:26,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5688.48 | bwd_inner_microstep: 5675.92 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.99
[2025-04-26 01:50:26,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5688.50 | bwd_inner: 5675.92 | bwd_allreduce: 12.54 | step: 18.99

 18%|█▊        | 7402/41250 [17:52:51<81:30:59,  8.67s/it]
                                                          
{'loss': 0.0638, 'grad_norm': 1.4359359741210938, 'learning_rate': 3.770305407032986e-05, 'epoch': 1.79}

 18%|█▊        | 7402/41250 [17:52:51<81:30:59,  8.67s/it][2025-04-26 01:50:34,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:50:34,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.75 | bwd_microstep: 5693.44 | bwd_inner_microstep: 5680.54 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.56
[2025-04-26 01:50:34,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.75 | bwd: 5693.46 | bwd_inner: 5680.54 | bwd_allreduce: 12.87 | step: 18.56

 18%|█▊        | 7403/41250 [17:53:00<81:26:02,  8.66s/it]
                                                          
{'loss': 0.1391, 'grad_norm': 2.3913750648498535, 'learning_rate': 3.7702323342418615e-05, 'epoch': 1.79}

 18%|█▊        | 7403/41250 [17:53:00<81:26:02,  8.66s/it][2025-04-26 01:50:43,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.07 | optimizer_step: 1.02
[2025-04-26 01:50:43,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.09 | bwd_microstep: 5741.20 | bwd_inner_microstep: 5688.99 | bwd_allreduce_microstep: 52.16 | step_microstep: 19.63
[2025-04-26 01:50:43,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.09 | bwd: 5741.21 | bwd_inner: 5688.99 | bwd_allreduce: 52.18 | step: 19.64

 18%|█▊        | 7404/41250 [17:53:08<81:27:48,  8.66s/it]
                                                          
{'loss': 0.2496, 'grad_norm': 2.9151949882507324, 'learning_rate': 3.7701592505376024e-05, 'epoch': 1.79}

 18%|█▊        | 7404/41250 [17:53:08<81:27:48,  8.66s/it][2025-04-26 01:50:52,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 01:50:52,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.12 | bwd_microstep: 5714.63 | bwd_inner_microstep: 5701.81 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.61
[2025-04-26 01:50:52,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.12 | bwd: 5714.64 | bwd_inner: 5701.81 | bwd_allreduce: 12.79 | step: 18.61

 18%|█▊        | 7405/41250 [17:53:17<81:26:48,  8.66s/it]
                                                          
{'loss': 0.1861, 'grad_norm': 1.5835340023040771, 'learning_rate': 3.7700861559206595e-05, 'epoch': 1.8}

 18%|█▊        | 7405/41250 [17:53:17<81:26:48,  8.66s/it][2025-04-26 01:51:00,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 1.18 | optimizer_step: 1.10
[2025-04-26 01:51:00,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.76 | bwd_microstep: 5747.24 | bwd_inner_microstep: 5651.24 | bwd_allreduce_microstep: 95.95 | step_microstep: 19.71
[2025-04-26 01:51:00,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.76 | bwd: 5747.25 | bwd_inner: 5651.24 | bwd_allreduce: 95.97 | step: 19.71

 18%|█▊        | 7406/41250 [17:53:26<81:26:03,  8.66s/it]
                                                          
{'loss': 0.1746, 'grad_norm': 2.3170368671417236, 'learning_rate': 3.770013050391484e-05, 'epoch': 1.8}

 18%|█▊        | 7406/41250 [17:53:26<81:26:03,  8.66s/it][2025-04-26 01:51:09,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 01:51:09,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.31 | bwd_microstep: 5695.84 | bwd_inner_microstep: 5641.21 | bwd_allreduce_microstep: 54.58 | step_microstep: 18.75
[2025-04-26 01:51:09,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.31 | bwd: 5695.85 | bwd_inner: 5641.21 | bwd_allreduce: 54.60 | step: 18.76

 18%|█▊        | 7407/41250 [17:53:34<81:15:34,  8.64s/it]
                                                          
{'loss': 0.0581, 'grad_norm': 2.089327573776245, 'learning_rate': 3.7699399339505257e-05, 'epoch': 1.8}

 18%|█▊        | 7407/41250 [17:53:34<81:15:34,  8.64s/it][2025-04-26 01:51:18,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:51:18,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5733.87 | bwd_inner_microstep: 5687.89 | bwd_allreduce_microstep: 45.93 | step_microstep: 18.64
[2025-04-26 01:51:18,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5733.89 | bwd_inner: 5687.89 | bwd_allreduce: 45.95 | step: 18.64

 18%|█▊        | 7408/41250 [17:53:43<81:20:20,  8.65s/it]
                                                          
{'loss': 0.0261, 'grad_norm': 0.3304009437561035, 'learning_rate': 3.769866806598236e-05, 'epoch': 1.8}

 18%|█▊        | 7408/41250 [17:53:43<81:20:20,  8.65s/it][2025-04-26 01:51:26,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 01:51:26,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5693.72 | bwd_inner_microstep: 5642.42 | bwd_allreduce_microstep: 51.26 | step_microstep: 18.82
[2025-04-26 01:51:26,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5693.73 | bwd_inner: 5642.42 | bwd_allreduce: 51.27 | step: 18.82

 18%|█▊        | 7409/41250 [17:53:52<81:11:49,  8.64s/it]
                                                          
{'loss': 0.2178, 'grad_norm': 2.3558743000030518, 'learning_rate': 3.7697936683350656e-05, 'epoch': 1.8}

 18%|█▊        | 7409/41250 [17:53:52<81:11:49,  8.64s/it][2025-04-26 01:51:35,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 01:51:35,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.90 | bwd_microstep: 5697.75 | bwd_inner_microstep: 5644.39 | bwd_allreduce_microstep: 53.31 | step_microstep: 19.24
[2025-04-26 01:51:35,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.90 | bwd: 5697.77 | bwd_inner: 5644.39 | bwd_allreduce: 53.33 | step: 19.24

 18%|█▊        | 7410/41250 [17:54:00<81:06:53,  8.63s/it]
                                                          
{'loss': 0.3323, 'grad_norm': 4.563774585723877, 'learning_rate': 3.769720519161466e-05, 'epoch': 1.8}

 18%|█▊        | 7410/41250 [17:54:00<81:06:53,  8.63s/it][2025-04-26 01:51:43,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 01:51:43,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5735.19 | bwd_inner_microstep: 5693.25 | bwd_allreduce_microstep: 41.88 | step_microstep: 18.65
[2025-04-26 01:51:43,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5735.20 | bwd_inner: 5693.25 | bwd_allreduce: 41.90 | step: 18.65

 18%|█▊        | 7411/41250 [17:54:09<81:13:22,  8.64s/it]
                                                          
{'loss': 0.1739, 'grad_norm': 2.0742664337158203, 'learning_rate': 3.769647359077886e-05, 'epoch': 1.8}

 18%|█▊        | 7411/41250 [17:54:09<81:13:22,  8.64s/it][2025-04-26 01:51:52,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.00 | optimizer_step: 1.16
[2025-04-26 01:51:52,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.75 | bwd_microstep: 5696.66 | bwd_inner_microstep: 5683.88 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.09
[2025-04-26 01:51:52,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.75 | bwd: 5696.67 | bwd_inner: 5683.88 | bwd_allreduce: 12.75 | step: 19.09

 18%|█▊        | 7412/41250 [17:54:17<81:10:47,  8.64s/it]
                                                          
{'loss': 0.0888, 'grad_norm': 1.2489888668060303, 'learning_rate': 3.7695741880847795e-05, 'epoch': 1.8}

 18%|█▊        | 7412/41250 [17:54:17<81:10:47,  8.64s/it][2025-04-26 01:52:01,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 01:52:01,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.80 | bwd_microstep: 5771.21 | bwd_inner_microstep: 5685.88 | bwd_allreduce_microstep: 85.28 | step_microstep: 18.80
[2025-04-26 01:52:01,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.80 | bwd: 5771.22 | bwd_inner: 5685.88 | bwd_allreduce: 85.30 | step: 18.80

 18%|█▊        | 7413/41250 [17:54:26<81:21:02,  8.66s/it]
                                                          
{'loss': 0.1522, 'grad_norm': 4.788893222808838, 'learning_rate': 3.769501006182595e-05, 'epoch': 1.8}

 18%|█▊        | 7413/41250 [17:54:26<81:21:02,  8.66s/it][2025-04-26 01:52:09,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:52:09,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.87 | bwd_microstep: 5704.41 | bwd_inner_microstep: 5691.56 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.93
[2025-04-26 01:52:09,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.87 | bwd: 5704.43 | bwd_inner: 5691.56 | bwd_allreduce: 12.83 | step: 18.94

 18%|█▊        | 7414/41250 [17:54:35<81:16:57,  8.65s/it]
                                                          
{'loss': 0.0891, 'grad_norm': 1.6845567226409912, 'learning_rate': 3.7694278133717855e-05, 'epoch': 1.8}

 18%|█▊        | 7414/41250 [17:54:35<81:16:57,  8.65s/it][2025-04-26 01:52:18,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 01:52:18,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5776.44 | bwd_inner_microstep: 5687.45 | bwd_allreduce_microstep: 88.94 | step_microstep: 18.96
[2025-04-26 01:52:18,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5776.46 | bwd_inner: 5687.45 | bwd_allreduce: 88.96 | step: 18.96

 18%|█▊        | 7415/41250 [17:54:43<81:26:51,  8.67s/it]
                                                          
{'loss': 0.1571, 'grad_norm': 3.158069372177124, 'learning_rate': 3.769354609652802e-05, 'epoch': 1.8}

 18%|█▊        | 7415/41250 [17:54:43<81:26:51,  8.67s/it][2025-04-26 01:52:27,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.21 | optimizer_step: 1.05
[2025-04-26 01:52:27,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.32 | bwd_microstep: 5885.42 | bwd_inner_microstep: 5688.76 | bwd_allreduce_microstep: 196.60 | step_microstep: 19.71
[2025-04-26 01:52:27,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.32 | bwd: 5885.44 | bwd_inner: 5688.76 | bwd_allreduce: 196.63 | step: 19.71

 18%|█▊        | 7416/41250 [17:54:52<81:51:10,  8.71s/it]
                                                          
{'loss': 0.4212, 'grad_norm': 3.3703525066375732, 'learning_rate': 3.769281395026095e-05, 'epoch': 1.8}

 18%|█▊        | 7416/41250 [17:54:52<81:51:10,  8.71s/it][2025-04-26 01:52:36,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:52:36,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5880.05 | bwd_inner_microstep: 5699.56 | bwd_allreduce_microstep: 180.45 | step_microstep: 18.83
[2025-04-26 01:52:36,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5880.06 | bwd_inner: 5699.55 | bwd_allreduce: 180.47 | step: 18.83

 18%|█▊        | 7417/41250 [17:55:01<82:07:39,  8.74s/it]
                                                          
{'loss': 0.1158, 'grad_norm': 3.208064079284668, 'learning_rate': 3.769208169492116e-05, 'epoch': 1.8}

 18%|█▊        | 7417/41250 [17:55:01<82:07:39,  8.74s/it][2025-04-26 01:52:44,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 01:52:44,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.05 | bwd_microstep: 5753.46 | bwd_inner_microstep: 5683.24 | bwd_allreduce_microstep: 70.17 | step_microstep: 18.91
[2025-04-26 01:52:44,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.05 | bwd: 5753.48 | bwd_inner: 5683.24 | bwd_allreduce: 70.19 | step: 18.91

 18%|█▊        | 7418/41250 [17:55:10<81:57:17,  8.72s/it]
                                                          
{'loss': 0.0781, 'grad_norm': 1.9144803285598755, 'learning_rate': 3.769134933051317e-05, 'epoch': 1.8}

 18%|█▊        | 7418/41250 [17:55:10<81:57:17,  8.72s/it][2025-04-26 01:52:53,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-26 01:52:53,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.74 | bwd_microstep: 5771.15 | bwd_inner_microstep: 5697.94 | bwd_allreduce_microstep: 73.16 | step_microstep: 18.63
[2025-04-26 01:52:53,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.74 | bwd: 5771.16 | bwd_inner: 5697.94 | bwd_allreduce: 73.18 | step: 18.63

 18%|█▊        | 7419/41250 [17:55:18<81:53:39,  8.71s/it]
                                                          
{'loss': 0.118, 'grad_norm': 1.6239794492721558, 'learning_rate': 3.769061685704148e-05, 'epoch': 1.8}

 18%|█▊        | 7419/41250 [17:55:18<81:53:39,  8.71s/it][2025-04-26 01:53:02,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 01:53:02,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.85 | bwd_microstep: 5791.39 | bwd_inner_microstep: 5778.63 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.18
[2025-04-26 01:53:02,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.85 | bwd: 5791.40 | bwd_inner: 5778.63 | bwd_allreduce: 12.72 | step: 19.19

 18%|█▊        | 7420/41250 [17:55:27<82:01:37,  8.73s/it]
                                                          
{'loss': 0.1363, 'grad_norm': 1.567510724067688, 'learning_rate': 3.768988427451063e-05, 'epoch': 1.8}

 18%|█▊        | 7420/41250 [17:55:27<82:01:37,  8.73s/it][2025-04-26 01:53:11,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 01:53:11,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.98 | bwd_microstep: 5741.20 | bwd_inner_microstep: 5684.55 | bwd_allreduce_microstep: 56.61 | step_microstep: 18.40
[2025-04-26 01:53:11,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.98 | bwd: 5741.22 | bwd_inner: 5684.55 | bwd_allreduce: 56.63 | step: 18.41

 18%|█▊        | 7421/41250 [17:55:36<81:52:36,  8.71s/it]
                                                          
{'loss': 0.109, 'grad_norm': 2.178358554840088, 'learning_rate': 3.768915158292512e-05, 'epoch': 1.8}

 18%|█▊        | 7421/41250 [17:55:36<81:52:36,  8.71s/it][2025-04-26 01:53:19,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 01:53:19,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.87 | bwd_microstep: 5770.56 | bwd_inner_microstep: 5652.47 | bwd_allreduce_microstep: 118.05 | step_microstep: 18.53
[2025-04-26 01:53:19,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.87 | bwd: 5770.58 | bwd_inner: 5652.47 | bwd_allreduce: 118.07 | step: 18.53

 18%|█▊        | 7422/41250 [17:55:45<81:47:46,  8.70s/it]
                                                          
{'loss': 0.198, 'grad_norm': 5.534649848937988, 'learning_rate': 3.768841878228946e-05, 'epoch': 1.8}

 18%|█▊        | 7422/41250 [17:55:45<81:47:46,  8.70s/it][2025-04-26 01:53:28,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 01:53:28,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.78 | bwd_microstep: 5701.16 | bwd_inner_microstep: 5688.36 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.74
[2025-04-26 01:53:28,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.78 | bwd: 5701.17 | bwd_inner: 5688.36 | bwd_allreduce: 12.77 | step: 18.75

 18%|█▊        | 7423/41250 [17:55:53<81:37:47,  8.69s/it]
                                                          
{'loss': 0.088, 'grad_norm': 3.2948663234710693, 'learning_rate': 3.7687685872608186e-05, 'epoch': 1.8}

 18%|█▊        | 7423/41250 [17:55:53<81:37:47,  8.69s/it][2025-04-26 01:53:37,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 01:53:37,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.05 | bwd_microstep: 5721.94 | bwd_inner_microstep: 5709.04 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.25
[2025-04-26 01:53:37,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.05 | bwd: 5721.96 | bwd_inner: 5709.04 | bwd_allreduce: 12.87 | step: 19.26

 18%|█▊        | 7424/41250 [17:56:02<81:35:10,  8.68s/it]
                                                          
{'loss': 0.0564, 'grad_norm': 0.843117892742157, 'learning_rate': 3.7686952853885805e-05, 'epoch': 1.8}

 18%|█▊        | 7424/41250 [17:56:02<81:35:10,  8.68s/it][2025-04-26 01:53:45,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 0.95
[2025-04-26 01:53:45,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.20 | bwd_microstep: 5867.15 | bwd_inner_microstep: 5721.91 | bwd_allreduce_microstep: 145.20 | step_microstep: 19.40
[2025-04-26 01:53:45,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.20 | bwd: 5867.17 | bwd_inner: 5721.91 | bwd_allreduce: 145.22 | step: 19.41

 18%|█▊        | 7425/41250 [17:56:11<81:57:26,  8.72s/it]
                                                          
{'loss': 0.2325, 'grad_norm': 1.6116573810577393, 'learning_rate': 3.7686219726126845e-05, 'epoch': 1.8}

 18%|█▊        | 7425/41250 [17:56:11<81:57:26,  8.72s/it][2025-04-26 01:53:54,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.10
[2025-04-26 01:53:54,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.55 | bwd_microstep: 5748.23 | bwd_inner_microstep: 5705.28 | bwd_allreduce_microstep: 42.91 | step_microstep: 19.46
[2025-04-26 01:53:54,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.55 | bwd: 5748.25 | bwd_inner: 5705.28 | bwd_allreduce: 42.93 | step: 19.46

 18%|█▊        | 7426/41250 [17:56:19<81:52:31,  8.71s/it]
                                                          
{'loss': 0.2976, 'grad_norm': 2.386112928390503, 'learning_rate': 3.768548648933581e-05, 'epoch': 1.8}

 18%|█▊        | 7426/41250 [17:56:19<81:52:31,  8.71s/it][2025-04-26 01:54:03,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 01:54:03,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.09 | bwd_microstep: 5774.41 | bwd_inner_microstep: 5697.94 | bwd_allreduce_microstep: 76.43 | step_microstep: 18.36
[2025-04-26 01:54:03,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.09 | bwd: 5774.43 | bwd_inner: 5697.94 | bwd_allreduce: 76.45 | step: 18.36

 18%|█▊        | 7427/41250 [17:56:28<81:52:44,  8.71s/it]
                                                          
{'loss': 0.3487, 'grad_norm': 2.1889545917510986, 'learning_rate': 3.768475314351723e-05, 'epoch': 1.8}

 18%|█▊        | 7427/41250 [17:56:28<81:52:44,  8.71s/it][2025-04-26 01:54:11,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:54:11,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5749.35 | bwd_inner_microstep: 5691.45 | bwd_allreduce_microstep: 57.84 | step_microstep: 18.46
[2025-04-26 01:54:11,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5749.36 | bwd_inner: 5691.45 | bwd_allreduce: 57.87 | step: 18.47

 18%|█▊        | 7428/41250 [17:56:37<81:46:58,  8.70s/it]
                                                          
{'loss': 0.1473, 'grad_norm': 1.504954218864441, 'learning_rate': 3.768401968867563e-05, 'epoch': 1.8}

 18%|█▊        | 7428/41250 [17:56:37<81:46:58,  8.70s/it][2025-04-26 01:54:20,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 01:54:20,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.19 | bwd_microstep: 5927.49 | bwd_inner_microstep: 5663.83 | bwd_allreduce_microstep: 263.61 | step_microstep: 18.83
[2025-04-26 01:54:20,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.19 | bwd: 5927.50 | bwd_inner: 5663.83 | bwd_allreduce: 263.63 | step: 18.83

 18%|█▊        | 7429/41250 [17:56:46<82:11:25,  8.75s/it]
                                                          
{'loss': 0.042, 'grad_norm': 0.8735190033912659, 'learning_rate': 3.768328612481553e-05, 'epoch': 1.8}

 18%|█▊        | 7429/41250 [17:56:46<82:11:25,  8.75s/it][2025-04-26 01:54:29,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:54:29,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.38 | bwd_microstep: 5724.61 | bwd_inner_microstep: 5711.74 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.86
[2025-04-26 01:54:29,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.38 | bwd: 5724.62 | bwd_inner: 5711.74 | bwd_allreduce: 12.84 | step: 18.86

 18%|█▊        | 7430/41250 [17:56:54<81:58:06,  8.73s/it]
                                                          
{'loss': 0.2052, 'grad_norm': 2.856370449066162, 'learning_rate': 3.768255245194144e-05, 'epoch': 1.8}

 18%|█▊        | 7430/41250 [17:56:54<81:58:06,  8.73s/it][2025-04-26 01:54:38,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 01:54:38,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.71 | bwd_microstep: 5740.64 | bwd_inner_microstep: 5713.96 | bwd_allreduce_microstep: 26.64 | step_microstep: 19.03
[2025-04-26 01:54:38,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.71 | bwd: 5740.66 | bwd_inner: 5713.96 | bwd_allreduce: 26.66 | step: 19.03

 18%|█▊        | 7431/41250 [17:57:03<81:50:46,  8.71s/it]
                                                          
{'loss': 0.0765, 'grad_norm': 3.1838340759277344, 'learning_rate': 3.76818186700579e-05, 'epoch': 1.8}

 18%|█▊        | 7431/41250 [17:57:03<81:50:46,  8.71s/it][2025-04-26 01:54:46,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 01:54:46,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.44 | bwd_microstep: 5783.13 | bwd_inner_microstep: 5647.00 | bwd_allreduce_microstep: 136.07 | step_microstep: 18.65
[2025-04-26 01:54:46,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.44 | bwd: 5783.14 | bwd_inner: 5647.00 | bwd_allreduce: 136.09 | step: 18.65

 18%|█▊        | 7432/41250 [17:57:12<81:47:46,  8.71s/it]
                                                          
{'loss': 0.2333, 'grad_norm': 1.3065623044967651, 'learning_rate': 3.7681084779169423e-05, 'epoch': 1.8}

 18%|█▊        | 7432/41250 [17:57:12<81:47:46,  8.71s/it][2025-04-26 01:54:55,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.41 | optimizer_gradients: 1.21 | optimizer_step: 0.98
[2025-04-26 01:54:55,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.51 | bwd_microstep: 5756.28 | bwd_inner_microstep: 5654.32 | bwd_allreduce_microstep: 101.91 | step_microstep: 20.44
[2025-04-26 01:54:55,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.51 | bwd: 5756.30 | bwd_inner: 5654.32 | bwd_allreduce: 101.93 | step: 20.44

 18%|█▊        | 7433/41250 [17:57:20<81:41:01,  8.70s/it]
                                                          
{'loss': 0.4076, 'grad_norm': 2.529862403869629, 'learning_rate': 3.768035077928053e-05, 'epoch': 1.8}

 18%|█▊        | 7433/41250 [17:57:20<81:41:01,  8.70s/it][2025-04-26 01:55:04,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 01:55:04,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.14 | bwd_microstep: 5781.35 | bwd_inner_microstep: 5633.85 | bwd_allreduce_microstep: 147.45 | step_microstep: 18.49
[2025-04-26 01:55:04,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.14 | bwd: 5781.36 | bwd_inner: 5633.85 | bwd_allreduce: 147.47 | step: 18.49

 18%|█▊        | 7434/41250 [17:57:29<81:40:18,  8.69s/it]
                                                          
{'loss': 0.1003, 'grad_norm': 1.1132413148880005, 'learning_rate': 3.767961667039576e-05, 'epoch': 1.8}

 18%|█▊        | 7434/41250 [17:57:29<81:40:18,  8.69s/it][2025-04-26 01:55:12,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 01:55:12,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.92 | bwd_microstep: 5686.12 | bwd_inner_microstep: 5651.23 | bwd_allreduce_microstep: 34.85 | step_microstep: 18.60
[2025-04-26 01:55:12,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.92 | bwd: 5686.13 | bwd_inner: 5651.22 | bwd_allreduce: 34.87 | step: 18.60

 18%|█▊        | 7435/41250 [17:57:38<81:26:12,  8.67s/it]
                                                          
{'loss': 0.2952, 'grad_norm': 2.0555734634399414, 'learning_rate': 3.767888245251963e-05, 'epoch': 1.8}

 18%|█▊        | 7435/41250 [17:57:38<81:26:12,  8.67s/it][2025-04-26 01:55:21,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:55:21,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5775.38 | bwd_inner_microstep: 5642.73 | bwd_allreduce_microstep: 132.61 | step_microstep: 18.35
[2025-04-26 01:55:21,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5775.40 | bwd_inner: 5642.73 | bwd_allreduce: 132.62 | step: 18.35

 18%|█▊        | 7436/41250 [17:57:46<81:28:31,  8.67s/it]
                                                          
{'loss': 0.0498, 'grad_norm': 1.3266514539718628, 'learning_rate': 3.767814812565666e-05, 'epoch': 1.8}

 18%|█▊        | 7436/41250 [17:57:46<81:28:31,  8.67s/it][2025-04-26 01:55:30,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:55:30,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.79 | bwd_microstep: 5741.23 | bwd_inner_microstep: 5675.51 | bwd_allreduce_microstep: 65.68 | step_microstep: 18.52
[2025-04-26 01:55:30,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.79 | bwd: 5741.24 | bwd_inner: 5675.51 | bwd_allreduce: 65.70 | step: 18.52

 18%|█▊        | 7437/41250 [17:57:55<81:28:31,  8.67s/it]
                                                          
{'loss': 0.0491, 'grad_norm': 1.0898289680480957, 'learning_rate': 3.7677413689811396e-05, 'epoch': 1.8}

 18%|█▊        | 7437/41250 [17:57:55<81:28:31,  8.67s/it][2025-04-26 01:55:38,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:55:38,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.64 | bwd_microstep: 5696.69 | bwd_inner_microstep: 5683.82 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.71
[2025-04-26 01:55:38,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.64 | bwd: 5696.70 | bwd_inner: 5683.82 | bwd_allreduce: 12.84 | step: 18.71

 18%|█▊        | 7438/41250 [17:58:04<81:21:22,  8.66s/it]
                                                          
{'loss': 0.2041, 'grad_norm': 1.4907597303390503, 'learning_rate': 3.7676679144988344e-05, 'epoch': 1.8}

 18%|█▊        | 7438/41250 [17:58:04<81:21:22,  8.66s/it][2025-04-26 01:55:47,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-26 01:55:47,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.12 | bwd_microstep: 5740.72 | bwd_inner_microstep: 5684.18 | bwd_allreduce_microstep: 56.49 | step_microstep: 19.26
[2025-04-26 01:55:47,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.12 | bwd: 5740.73 | bwd_inner: 5684.18 | bwd_allreduce: 56.50 | step: 19.26

 18%|█▊        | 7439/41250 [17:58:12<81:22:59,  8.67s/it]
                                                          
{'loss': 0.0135, 'grad_norm': 0.22451573610305786, 'learning_rate': 3.7675944491192045e-05, 'epoch': 1.8}

 18%|█▊        | 7439/41250 [17:58:12<81:22:59,  8.67s/it][2025-04-26 01:55:56,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:55:56,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 5679.37 | bwd_inner_microstep: 5645.97 | bwd_allreduce_microstep: 33.35 | step_microstep: 18.59
[2025-04-26 01:55:56,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 5679.39 | bwd_inner: 5645.97 | bwd_allreduce: 33.37 | step: 18.60

 18%|█▊        | 7440/41250 [17:58:21<81:10:40,  8.64s/it]
                                                          
{'loss': 0.0565, 'grad_norm': 1.480241298675537, 'learning_rate': 3.767520972842702e-05, 'epoch': 1.8}

 18%|█▊        | 7440/41250 [17:58:21<81:10:40,  8.64s/it][2025-04-26 01:56:04,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 01:56:04,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.77 | bwd_microstep: 5734.89 | bwd_inner_microstep: 5692.04 | bwd_allreduce_microstep: 42.80 | step_microstep: 19.05
[2025-04-26 01:56:04,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.77 | bwd: 5734.91 | bwd_inner: 5692.04 | bwd_allreduce: 42.82 | step: 19.05

 18%|█▊        | 7441/41250 [17:58:30<81:16:43,  8.65s/it]
                                                          
{'loss': 0.1748, 'grad_norm': 4.267389297485352, 'learning_rate': 3.767447485669781e-05, 'epoch': 1.8}

 18%|█▊        | 7441/41250 [17:58:30<81:16:43,  8.65s/it][2025-04-26 01:56:13,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 01:56:13,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.31 | bwd_microstep: 5685.74 | bwd_inner_microstep: 5654.65 | bwd_allreduce_microstep: 31.04 | step_microstep: 18.80
[2025-04-26 01:56:13,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.31 | bwd: 5685.75 | bwd_inner: 5654.65 | bwd_allreduce: 31.06 | step: 18.80

 18%|█▊        | 7442/41250 [17:58:38<81:07:12,  8.64s/it]
                                                          
{'loss': 0.2145, 'grad_norm': 5.184484481811523, 'learning_rate': 3.767373987600893e-05, 'epoch': 1.8}

 18%|█▊        | 7442/41250 [17:58:38<81:07:12,  8.64s/it][2025-04-26 01:56:21,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 01:56:21,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.49 | bwd_microstep: 5672.84 | bwd_inner_microstep: 5645.85 | bwd_allreduce_microstep: 26.95 | step_microstep: 18.94
[2025-04-26 01:56:21,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.49 | bwd: 5672.85 | bwd_inner: 5645.85 | bwd_allreduce: 26.97 | step: 18.94

 18%|█▊        | 7443/41250 [17:58:47<80:58:57,  8.62s/it]
                                                          
{'loss': 0.1462, 'grad_norm': 1.0679914951324463, 'learning_rate': 3.767300478636493e-05, 'epoch': 1.8}

 18%|█▊        | 7443/41250 [17:58:47<80:58:57,  8.62s/it][2025-04-26 01:56:30,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 01:56:30,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.98 | bwd_microstep: 5689.55 | bwd_inner_microstep: 5639.79 | bwd_allreduce_microstep: 49.71 | step_microstep: 18.67
[2025-04-26 01:56:30,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.98 | bwd: 5689.56 | bwd_inner: 5639.79 | bwd_allreduce: 49.73 | step: 18.67

 18%|█▊        | 7444/41250 [17:58:55<80:54:56,  8.62s/it]
                                                          
{'loss': 0.0989, 'grad_norm': 2.021881103515625, 'learning_rate': 3.767226958777033e-05, 'epoch': 1.8}

 18%|█▊        | 7444/41250 [17:58:55<80:54:56,  8.62s/it][2025-04-26 01:56:39,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 01:56:39,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.04 | bwd_microstep: 5741.22 | bwd_inner_microstep: 5638.66 | bwd_allreduce_microstep: 102.52 | step_microstep: 18.97
[2025-04-26 01:56:39,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.04 | bwd: 5741.23 | bwd_inner: 5638.66 | bwd_allreduce: 102.53 | step: 18.97

 18%|█▊        | 7445/41250 [17:59:04<81:00:31,  8.63s/it]
                                                          
{'loss': 0.0376, 'grad_norm': 0.952304482460022, 'learning_rate': 3.767153428022966e-05, 'epoch': 1.8}

 18%|█▊        | 7445/41250 [17:59:04<81:00:31,  8.63s/it][2025-04-26 01:56:47,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 01:56:47,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.72 | bwd_microstep: 5759.81 | bwd_inner_microstep: 5747.11 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.65
[2025-04-26 01:56:47,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.72 | bwd: 5759.82 | bwd_inner: 5747.11 | bwd_allreduce: 12.68 | step: 18.65

 18%|█▊        | 7446/41250 [17:59:13<81:17:51,  8.66s/it]
                                                          
{'loss': 0.1302, 'grad_norm': 2.1875956058502197, 'learning_rate': 3.767079886374746e-05, 'epoch': 1.81}

 18%|█▊        | 7446/41250 [17:59:13<81:17:51,  8.66s/it][2025-04-26 01:56:56,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 01:56:56,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.21 | bwd_microstep: 5770.04 | bwd_inner_microstep: 5644.55 | bwd_allreduce_microstep: 125.45 | step_microstep: 18.20
[2025-04-26 01:56:56,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.21 | bwd: 5770.05 | bwd_inner: 5644.55 | bwd_allreduce: 125.47 | step: 18.20

 18%|█▊        | 7447/41250 [17:59:21<81:22:46,  8.67s/it]
                                                          
{'loss': 0.0633, 'grad_norm': 1.2054131031036377, 'learning_rate': 3.767006333832826e-05, 'epoch': 1.81}

 18%|█▊        | 7447/41250 [17:59:21<81:22:46,  8.67s/it][2025-04-26 01:57:05,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:57:05,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.42 | bwd_microstep: 5730.41 | bwd_inner_microstep: 5685.50 | bwd_allreduce_microstep: 44.87 | step_microstep: 18.41
[2025-04-26 01:57:05,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.42 | bwd: 5730.42 | bwd_inner: 5685.50 | bwd_allreduce: 44.88 | step: 18.42

 18%|█▊        | 7448/41250 [17:59:30<81:22:11,  8.67s/it]
                                                          
{'loss': 0.1239, 'grad_norm': 1.577268123626709, 'learning_rate': 3.766932770397659e-05, 'epoch': 1.81}

 18%|█▊        | 7448/41250 [17:59:30<81:22:11,  8.67s/it][2025-04-26 01:57:13,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 01:57:13,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5741.94 | bwd_inner_microstep: 5683.15 | bwd_allreduce_microstep: 58.74 | step_microstep: 19.37
[2025-04-26 01:57:13,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5741.95 | bwd_inner: 5683.15 | bwd_allreduce: 58.76 | step: 19.37

 18%|█▊        | 7449/41250 [17:59:39<81:24:53,  8.67s/it]
                                                          
{'loss': 0.0828, 'grad_norm': 1.3735843896865845, 'learning_rate': 3.7668591960697e-05, 'epoch': 1.81}

 18%|█▊        | 7449/41250 [17:59:39<81:24:53,  8.67s/it][2025-04-26 01:57:22,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:57:22,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.65 | bwd_microstep: 5754.40 | bwd_inner_microstep: 5644.53 | bwd_allreduce_microstep: 109.83 | step_microstep: 18.96
[2025-04-26 01:57:22,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.65 | bwd: 5754.41 | bwd_inner: 5644.53 | bwd_allreduce: 109.85 | step: 18.96

 18%|█▊        | 7450/41250 [17:59:47<81:23:23,  8.67s/it]
                                                          
{'loss': 0.2395, 'grad_norm': 2.9982752799987793, 'learning_rate': 3.766785610849401e-05, 'epoch': 1.81}

 18%|█▊        | 7450/41250 [17:59:47<81:23:23,  8.67s/it][2025-04-26 01:57:31,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:57:31,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.32 | bwd_microstep: 5689.47 | bwd_inner_microstep: 5676.55 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.43
[2025-04-26 01:57:31,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.32 | bwd: 5689.48 | bwd_inner: 5676.55 | bwd_allreduce: 12.89 | step: 18.43

 18%|█▊        | 7451/41250 [17:59:56<81:16:34,  8.66s/it]
                                                          
{'loss': 0.0412, 'grad_norm': 1.1723353862762451, 'learning_rate': 3.7667120147372164e-05, 'epoch': 1.81}

 18%|█▊        | 7451/41250 [17:59:56<81:16:34,  8.66s/it][2025-04-26 01:57:39,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:57:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.58 | bwd_microstep: 5731.99 | bwd_inner_microstep: 5676.85 | bwd_allreduce_microstep: 55.10 | step_microstep: 18.55
[2025-04-26 01:57:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.58 | bwd: 5732.00 | bwd_inner: 5676.85 | bwd_allreduce: 55.12 | step: 18.56

 18%|█▊        | 7452/41250 [18:00:05<81:17:03,  8.66s/it]
                                                          
{'loss': 0.2928, 'grad_norm': 2.85420823097229, 'learning_rate': 3.7666384077336e-05, 'epoch': 1.81}

 18%|█▊        | 7452/41250 [18:00:05<81:17:03,  8.66s/it][2025-04-26 01:57:48,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:57:48,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5756.34 | bwd_inner_microstep: 5638.44 | bwd_allreduce_microstep: 117.86 | step_microstep: 18.14
[2025-04-26 01:57:48,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5756.35 | bwd_inner: 5638.44 | bwd_allreduce: 117.87 | step: 18.15

 18%|█▊        | 7453/41250 [18:00:13<81:17:46,  8.66s/it]
                                                          
{'loss': 0.334, 'grad_norm': 4.1131134033203125, 'learning_rate': 3.766564789839004e-05, 'epoch': 1.81}

 18%|█▊        | 7453/41250 [18:00:13<81:17:46,  8.66s/it][2025-04-26 01:57:57,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:57:57,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.30 | bwd_microstep: 5781.80 | bwd_inner_microstep: 5769.08 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.42
[2025-04-26 01:57:57,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.30 | bwd: 5781.82 | bwd_inner: 5769.08 | bwd_allreduce: 12.69 | step: 18.43

 18%|█▊        | 7454/41250 [18:00:22<81:33:38,  8.69s/it]
                                                          
{'loss': 0.2356, 'grad_norm': 4.665336608886719, 'learning_rate': 3.7664911610538844e-05, 'epoch': 1.81}

 18%|█▊        | 7454/41250 [18:00:22<81:33:38,  8.69s/it][2025-04-26 01:58:06,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 01:58:06,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.79 | bwd_microstep: 5724.71 | bwd_inner_microstep: 5654.30 | bwd_allreduce_microstep: 70.36 | step_microstep: 18.83
[2025-04-26 01:58:06,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.79 | bwd: 5724.72 | bwd_inner: 5654.30 | bwd_allreduce: 70.38 | step: 18.84

 18%|█▊        | 7455/41250 [18:00:31<81:28:24,  8.68s/it]
                                                          
{'loss': 0.1563, 'grad_norm': 3.345400333404541, 'learning_rate': 3.766417521378695e-05, 'epoch': 1.81}

 18%|█▊        | 7455/41250 [18:00:31<81:28:24,  8.68s/it][2025-04-26 01:58:14,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:58:14,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5714.54 | bwd_inner_microstep: 5701.82 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.51
[2025-04-26 01:58:14,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5714.56 | bwd_inner: 5701.82 | bwd_allreduce: 12.70 | step: 18.52

 18%|█▊        | 7456/41250 [18:00:40<81:23:36,  8.67s/it]
                                                          
{'loss': 0.2765, 'grad_norm': 5.379055976867676, 'learning_rate': 3.7663438708138876e-05, 'epoch': 1.81}

 18%|█▊        | 7456/41250 [18:00:40<81:23:36,  8.67s/it][2025-04-26 01:58:23,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 01:58:23,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.92 | bwd_microstep: 5757.90 | bwd_inner_microstep: 5656.51 | bwd_allreduce_microstep: 101.35 | step_microstep: 18.70
[2025-04-26 01:58:23,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.92 | bwd: 5757.92 | bwd_inner: 5656.51 | bwd_allreduce: 101.37 | step: 18.70

 18%|█▊        | 7457/41250 [18:00:48<81:24:16,  8.67s/it]
                                                          
{'loss': 0.1627, 'grad_norm': 3.537233591079712, 'learning_rate': 3.766270209359919e-05, 'epoch': 1.81}

 18%|█▊        | 7457/41250 [18:00:48<81:24:16,  8.67s/it][2025-04-26 01:58:32,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 01:58:32,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.55 | bwd_microstep: 5718.90 | bwd_inner_microstep: 5706.18 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.76
[2025-04-26 01:58:32,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.55 | bwd: 5718.91 | bwd_inner: 5706.18 | bwd_allreduce: 12.68 | step: 18.77

 18%|█▊        | 7458/41250 [18:00:57<81:20:28,  8.67s/it]
                                                          
{'loss': 0.1732, 'grad_norm': 1.5534859895706177, 'learning_rate': 3.766196537017241e-05, 'epoch': 1.81}

 18%|█▊        | 7458/41250 [18:00:57<81:20:28,  8.67s/it][2025-04-26 01:58:40,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 01:58:40,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.73 | bwd_microstep: 5694.40 | bwd_inner_microstep: 5650.51 | bwd_allreduce_microstep: 43.84 | step_microstep: 18.61
[2025-04-26 01:58:40,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.73 | bwd: 5694.41 | bwd_inner: 5650.51 | bwd_allreduce: 43.86 | step: 18.61

 18%|█▊        | 7459/41250 [18:01:05<81:10:11,  8.65s/it]
                                                          
{'loss': 0.0286, 'grad_norm': 0.3765430152416229, 'learning_rate': 3.766122853786309e-05, 'epoch': 1.81}

 18%|█▊        | 7459/41250 [18:01:05<81:10:11,  8.65s/it][2025-04-26 01:58:49,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 01:58:49,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.41 | bwd_microstep: 5755.96 | bwd_inner_microstep: 5657.91 | bwd_allreduce_microstep: 98.00 | step_microstep: 19.44
[2025-04-26 01:58:49,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.41 | bwd: 5755.97 | bwd_inner: 5657.91 | bwd_allreduce: 98.03 | step: 19.44

 18%|█▊        | 7460/41250 [18:01:14<81:15:51,  8.66s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 1.0050153732299805, 'learning_rate': 3.766049159667577e-05, 'epoch': 1.81}

 18%|█▊        | 7460/41250 [18:01:14<81:15:51,  8.66s/it][2025-04-26 01:58:57,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 01:58:57,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5772.57 | bwd_inner_microstep: 5699.26 | bwd_allreduce_microstep: 73.26 | step_microstep: 18.69
[2025-04-26 01:58:57,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5772.58 | bwd_inner: 5699.26 | bwd_allreduce: 73.28 | step: 18.70

 18%|█▊        | 7461/41250 [18:01:23<81:24:13,  8.67s/it]
                                                          
{'loss': 0.3186, 'grad_norm': 2.7789130210876465, 'learning_rate': 3.765975454661499e-05, 'epoch': 1.81}

 18%|█▊        | 7461/41250 [18:01:23<81:24:13,  8.67s/it][2025-04-26 01:59:06,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 01:59:06,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.95 | bwd_microstep: 5845.07 | bwd_inner_microstep: 5697.22 | bwd_allreduce_microstep: 147.80 | step_microstep: 18.53
[2025-04-26 01:59:06,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.95 | bwd: 5845.08 | bwd_inner: 5697.22 | bwd_allreduce: 147.82 | step: 18.54

 18%|█▊        | 7462/41250 [18:01:32<81:44:00,  8.71s/it]
                                                          
{'loss': 0.1193, 'grad_norm': 1.936635136604309, 'learning_rate': 3.76590173876853e-05, 'epoch': 1.81}

 18%|█▊        | 7462/41250 [18:01:32<81:44:00,  8.71s/it][2025-04-26 01:59:15,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:59:15,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.88 | bwd_microstep: 5705.13 | bwd_inner_microstep: 5662.54 | bwd_allreduce_microstep: 42.55 | step_microstep: 18.50
[2025-04-26 01:59:15,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.88 | bwd: 5705.14 | bwd_inner: 5662.54 | bwd_allreduce: 42.56 | step: 18.51

 18%|█▊        | 7463/41250 [18:01:40<81:28:02,  8.68s/it]
                                                          
{'loss': 0.0213, 'grad_norm': 0.6008156538009644, 'learning_rate': 3.765828011989124e-05, 'epoch': 1.81}

 18%|█▊        | 7463/41250 [18:01:40<81:28:02,  8.68s/it][2025-04-26 01:59:24,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:59:24,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.97 | bwd_microstep: 5777.53 | bwd_inner_microstep: 5764.62 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.78
[2025-04-26 01:59:24,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.97 | bwd: 5777.54 | bwd_inner: 5764.62 | bwd_allreduce: 12.88 | step: 18.78

 18%|█▊        | 7464/41250 [18:01:49<81:40:47,  8.70s/it]
                                                          
{'loss': 0.2459, 'grad_norm': 2.7734556198120117, 'learning_rate': 3.765754274323736e-05, 'epoch': 1.81}

 18%|█▊        | 7464/41250 [18:01:49<81:40:47,  8.70s/it][2025-04-26 01:59:32,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 01:59:32,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.00 | bwd_microstep: 5740.97 | bwd_inner_microstep: 5697.36 | bwd_allreduce_microstep: 43.56 | step_microstep: 18.95
[2025-04-26 01:59:32,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.01 | bwd: 5740.98 | bwd_inner: 5697.36 | bwd_allreduce: 43.58 | step: 18.95

 18%|█▊        | 7465/41250 [18:01:58<81:37:12,  8.70s/it]
                                                          
{'loss': 0.1708, 'grad_norm': 3.4889042377471924, 'learning_rate': 3.76568052577282e-05, 'epoch': 1.81}

 18%|█▊        | 7465/41250 [18:01:58<81:37:12,  8.70s/it][2025-04-26 01:59:41,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 01:59:41,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.81 | bwd_microstep: 5768.85 | bwd_inner_microstep: 5691.72 | bwd_allreduce_microstep: 77.09 | step_microstep: 18.67
[2025-04-26 01:59:41,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.81 | bwd: 5768.86 | bwd_inner: 5691.72 | bwd_allreduce: 77.10 | step: 18.67

 18%|█▊        | 7466/41250 [18:02:06<81:38:15,  8.70s/it]
                                                          
{'loss': 0.1809, 'grad_norm': 1.9101375341415405, 'learning_rate': 3.765606766336831e-05, 'epoch': 1.81}

 18%|█▊        | 7466/41250 [18:02:06<81:38:15,  8.70s/it][2025-04-26 01:59:50,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 01:59:50,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.76 | bwd_microstep: 5779.02 | bwd_inner_microstep: 5649.88 | bwd_allreduce_microstep: 129.10 | step_microstep: 18.72
[2025-04-26 01:59:50,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.76 | bwd: 5779.04 | bwd_inner: 5649.88 | bwd_allreduce: 129.12 | step: 18.72

 18%|█▊        | 7467/41250 [18:02:15<81:37:35,  8.70s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 0.8923555016517639, 'learning_rate': 3.765532996016223e-05, 'epoch': 1.81}

 18%|█▊        | 7467/41250 [18:02:15<81:37:35,  8.70s/it][2025-04-26 01:59:58,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 01:59:58,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.06 | bwd_microstep: 5755.10 | bwd_inner_microstep: 5708.07 | bwd_allreduce_microstep: 46.99 | step_microstep: 18.42
[2025-04-26 01:59:58,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.06 | bwd: 5755.11 | bwd_inner: 5708.07 | bwd_allreduce: 47.01 | step: 18.42

 18%|█▊        | 7468/41250 [18:02:24<81:36:31,  8.70s/it]
                                                          
{'loss': 0.1249, 'grad_norm': 3.6642723083496094, 'learning_rate': 3.7654592148114514e-05, 'epoch': 1.81}

 18%|█▊        | 7468/41250 [18:02:24<81:36:31,  8.70s/it][2025-04-26 02:00:07,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:00:07,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.31 | bwd_microstep: 5719.28 | bwd_inner_microstep: 5653.09 | bwd_allreduce_microstep: 66.15 | step_microstep: 18.23
[2025-04-26 02:00:07,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.31 | bwd: 5719.29 | bwd_inner: 5653.09 | bwd_allreduce: 66.16 | step: 18.24

 18%|█▊        | 7469/41250 [18:02:32<81:26:23,  8.68s/it]
                                                          
{'loss': 0.2461, 'grad_norm': 5.337395668029785, 'learning_rate': 3.765385422722972e-05, 'epoch': 1.81}

 18%|█▊        | 7469/41250 [18:02:32<81:26:23,  8.68s/it][2025-04-26 02:00:16,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:00:16,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.98 | bwd_microstep: 5784.12 | bwd_inner_microstep: 5646.40 | bwd_allreduce_microstep: 137.68 | step_microstep: 18.46
[2025-04-26 02:00:16,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.98 | bwd: 5784.13 | bwd_inner: 5646.40 | bwd_allreduce: 137.69 | step: 18.46

 18%|█▊        | 7470/41250 [18:02:41<81:29:57,  8.69s/it]
                                                          
{'loss': 0.1361, 'grad_norm': 2.614384889602661, 'learning_rate': 3.765311619751238e-05, 'epoch': 1.81}

 18%|█▊        | 7470/41250 [18:02:41<81:29:57,  8.69s/it][2025-04-26 02:00:24,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:00:24,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.35 | bwd_microstep: 5712.44 | bwd_inner_microstep: 5699.59 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.49
[2025-04-26 02:00:24,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.35 | bwd: 5712.45 | bwd_inner: 5699.59 | bwd_allreduce: 12.81 | step: 18.49

 18%|█▊        | 7471/41250 [18:02:50<81:23:58,  8.68s/it]
                                                          
{'loss': 0.1514, 'grad_norm': 3.079850435256958, 'learning_rate': 3.7652378058967044e-05, 'epoch': 1.81}

 18%|█▊        | 7471/41250 [18:02:50<81:23:58,  8.68s/it][2025-04-26 02:00:33,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:00:33,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.28 | bwd_microstep: 5795.04 | bwd_inner_microstep: 5647.63 | bwd_allreduce_microstep: 147.35 | step_microstep: 18.64
[2025-04-26 02:00:33,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.28 | bwd: 5795.05 | bwd_inner: 5647.63 | bwd_allreduce: 147.37 | step: 18.65

 18%|█▊        | 7472/41250 [18:02:58<81:28:50,  8.68s/it]
                                                          
{'loss': 0.2868, 'grad_norm': 4.501125335693359, 'learning_rate': 3.7651639811598276e-05, 'epoch': 1.81}

 18%|█▊        | 7472/41250 [18:02:58<81:28:50,  8.68s/it][2025-04-26 02:00:42,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-26 02:00:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.07 | bwd_microstep: 5704.49 | bwd_inner_microstep: 5648.00 | bwd_allreduce_microstep: 56.44 | step_microstep: 18.93
[2025-04-26 02:00:42,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.07 | bwd: 5704.50 | bwd_inner: 5648.00 | bwd_allreduce: 56.46 | step: 18.93

 18%|█▊        | 7473/41250 [18:03:07<81:18:15,  8.67s/it]
                                                          
{'loss': 0.1571, 'grad_norm': 1.509244441986084, 'learning_rate': 3.765090145541062e-05, 'epoch': 1.81}

 18%|█▊        | 7473/41250 [18:03:07<81:18:15,  8.67s/it][2025-04-26 02:00:50,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 02:00:50,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.08 | bwd_microstep: 5743.44 | bwd_inner_microstep: 5697.21 | bwd_allreduce_microstep: 46.19 | step_microstep: 18.52
[2025-04-26 02:00:50,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.08 | bwd: 5743.46 | bwd_inner: 5697.21 | bwd_allreduce: 46.21 | step: 18.53

 18%|█▊        | 7474/41250 [18:03:16<81:20:23,  8.67s/it]
                                                          
{'loss': 0.319, 'grad_norm': 2.422044515609741, 'learning_rate': 3.7650162990408625e-05, 'epoch': 1.81}

 18%|█▊        | 7474/41250 [18:03:16<81:20:23,  8.67s/it][2025-04-26 02:00:59,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 02:00:59,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.78 | bwd_microstep: 5790.36 | bwd_inner_microstep: 5777.68 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.86
[2025-04-26 02:00:59,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.78 | bwd: 5790.38 | bwd_inner: 5777.68 | bwd_allreduce: 12.66 | step: 18.86

 18%|█▊        | 7475/41250 [18:03:25<81:37:16,  8.70s/it]
                                                          
{'loss': 0.1583, 'grad_norm': 1.5754706859588623, 'learning_rate': 3.7649424416596856e-05, 'epoch': 1.81}

 18%|█▊        | 7475/41250 [18:03:25<81:37:16,  8.70s/it][2025-04-26 02:01:08,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:01:08,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.72 | bwd_microstep: 5710.33 | bwd_inner_microstep: 5642.64 | bwd_allreduce_microstep: 67.64 | step_microstep: 18.74
[2025-04-26 02:01:08,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.72 | bwd: 5710.34 | bwd_inner: 5642.64 | bwd_allreduce: 67.66 | step: 18.75

 18%|█▊        | 7476/41250 [18:03:33<81:25:28,  8.68s/it]
                                                          
{'loss': 0.1422, 'grad_norm': 2.3914895057678223, 'learning_rate': 3.7648685733979856e-05, 'epoch': 1.81}

 18%|█▊        | 7476/41250 [18:03:33<81:25:28,  8.68s/it][2025-04-26 02:01:17,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:01:17,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.14 | bwd_microstep: 5740.86 | bwd_inner_microstep: 5654.83 | bwd_allreduce_microstep: 85.99 | step_microstep: 18.60
[2025-04-26 02:01:17,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.14 | bwd: 5740.87 | bwd_inner: 5654.83 | bwd_allreduce: 86.00 | step: 18.60

 18%|█▊        | 7477/41250 [18:03:42<81:24:03,  8.68s/it]
                                                          
{'loss': 0.2204, 'grad_norm': 2.188889503479004, 'learning_rate': 3.764794694256217e-05, 'epoch': 1.81}

 18%|█▊        | 7477/41250 [18:03:42<81:24:03,  8.68s/it][2025-04-26 02:01:25,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:01:25,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.40 | bwd_microstep: 5702.31 | bwd_inner_microstep: 5638.66 | bwd_allreduce_microstep: 63.60 | step_microstep: 18.55
[2025-04-26 02:01:25,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.40 | bwd: 5702.33 | bwd_inner: 5638.66 | bwd_allreduce: 63.62 | step: 18.56

 18%|█▊        | 7478/41250 [18:03:50<81:13:11,  8.66s/it]
                                                          
{'loss': 0.0752, 'grad_norm': 1.2003509998321533, 'learning_rate': 3.764720804234837e-05, 'epoch': 1.81}

 18%|█▊        | 7478/41250 [18:03:50<81:13:11,  8.66s/it][2025-04-26 02:01:34,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.14 | optimizer_step: 1.11
[2025-04-26 02:01:34,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.84 | bwd_microstep: 5771.96 | bwd_inner_microstep: 5645.98 | bwd_allreduce_microstep: 125.93 | step_microstep: 19.07
[2025-04-26 02:01:34,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.84 | bwd: 5771.97 | bwd_inner: 5645.98 | bwd_allreduce: 125.95 | step: 19.07

 18%|█▊        | 7479/41250 [18:03:59<81:16:43,  8.66s/it]
                                                          
{'loss': 0.0722, 'grad_norm': 1.2572109699249268, 'learning_rate': 3.7646469033343e-05, 'epoch': 1.81}

 18%|█▊        | 7479/41250 [18:03:59<81:16:43,  8.66s/it][2025-04-26 02:01:42,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 02:01:42,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5691.72 | bwd_inner_microstep: 5678.93 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.80
[2025-04-26 02:01:42,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5691.73 | bwd_inner: 5678.93 | bwd_allreduce: 12.76 | step: 18.81

 18%|█▊        | 7480/41250 [18:04:08<81:09:50,  8.65s/it]
                                                          
{'loss': 0.0241, 'grad_norm': 0.5272358059883118, 'learning_rate': 3.764572991555062e-05, 'epoch': 1.81}

 18%|█▊        | 7480/41250 [18:04:08<81:09:50,  8.65s/it][2025-04-26 02:01:51,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:01:51,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.50 | bwd_microstep: 5744.65 | bwd_inner_microstep: 5674.82 | bwd_allreduce_microstep: 69.79 | step_microstep: 18.50
[2025-04-26 02:01:51,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.50 | bwd: 5744.66 | bwd_inner: 5674.82 | bwd_allreduce: 69.80 | step: 18.51

 18%|█▊        | 7481/41250 [18:04:16<81:14:17,  8.66s/it]
                                                          
{'loss': 0.2061, 'grad_norm': 2.286813259124756, 'learning_rate': 3.764499068897578e-05, 'epoch': 1.81}

 18%|█▊        | 7481/41250 [18:04:16<81:14:17,  8.66s/it][2025-04-26 02:02:00,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:02:00,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5736.45 | bwd_inner_microstep: 5652.14 | bwd_allreduce_microstep: 84.25 | step_microstep: 18.53
[2025-04-26 02:02:00,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5736.46 | bwd_inner: 5652.14 | bwd_allreduce: 84.27 | step: 18.53

 18%|█▊        | 7482/41250 [18:04:25<81:15:58,  8.66s/it]
                                                          
{'loss': 0.0178, 'grad_norm': 0.5058453679084778, 'learning_rate': 3.764425135362305e-05, 'epoch': 1.81}

 18%|█▊        | 7482/41250 [18:04:25<81:15:58,  8.66s/it][2025-04-26 02:02:08,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 02:02:08,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.12 | bwd_microstep: 5734.00 | bwd_inner_microstep: 5680.68 | bwd_allreduce_microstep: 53.28 | step_microstep: 19.22
[2025-04-26 02:02:08,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.12 | bwd: 5734.02 | bwd_inner: 5680.68 | bwd_allreduce: 53.30 | step: 19.22

 18%|█▊        | 7483/41250 [18:04:34<81:17:12,  8.67s/it]
                                                          
{'loss': 0.0817, 'grad_norm': 0.9391086101531982, 'learning_rate': 3.764351190949698e-05, 'epoch': 1.81}

 18%|█▊        | 7483/41250 [18:04:34<81:17:12,  8.67s/it][2025-04-26 02:02:17,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 02:02:17,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.36 | bwd_microstep: 5695.80 | bwd_inner_microstep: 5640.87 | bwd_allreduce_microstep: 54.89 | step_microstep: 18.86
[2025-04-26 02:02:17,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.36 | bwd: 5695.81 | bwd_inner: 5640.87 | bwd_allreduce: 54.91 | step: 18.87

 18%|█▊        | 7484/41250 [18:04:42<81:06:53,  8.65s/it]
                                                          
{'loss': 0.1093, 'grad_norm': 2.344161033630371, 'learning_rate': 3.764277235660213e-05, 'epoch': 1.81}

 18%|█▊        | 7484/41250 [18:04:42<81:06:53,  8.65s/it][2025-04-26 02:02:26,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 02:02:26,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5698.39 | bwd_inner_microstep: 5650.92 | bwd_allreduce_microstep: 47.42 | step_microstep: 19.00
[2025-04-26 02:02:26,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.93 | bwd: 5698.40 | bwd_inner: 5650.92 | bwd_allreduce: 47.44 | step: 19.00

 18%|█▊        | 7485/41250 [18:04:51<81:00:31,  8.64s/it]
                                                          
{'loss': 0.0477, 'grad_norm': 0.6886035203933716, 'learning_rate': 3.7642032694943056e-05, 'epoch': 1.81}

 18%|█▊        | 7485/41250 [18:04:51<81:00:31,  8.64s/it][2025-04-26 02:02:34,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:02:34,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.66 | bwd_microstep: 5706.41 | bwd_inner_microstep: 5693.71 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.65
[2025-04-26 02:02:34,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.66 | bwd: 5706.42 | bwd_inner: 5693.71 | bwd_allreduce: 12.68 | step: 18.65

 18%|█▊        | 7486/41250 [18:05:00<81:00:09,  8.64s/it]
                                                          
{'loss': 0.2092, 'grad_norm': 1.884619116783142, 'learning_rate': 3.7641292924524325e-05, 'epoch': 1.81}

 18%|█▊        | 7486/41250 [18:05:00<81:00:09,  8.64s/it][2025-04-26 02:02:43,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:02:43,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.55 | bwd_microstep: 5694.64 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 49.59 | step_microstep: 18.49
[2025-04-26 02:02:43,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.55 | bwd: 5694.65 | bwd_inner: 5645.00 | bwd_allreduce: 49.61 | step: 18.49

 18%|█▊        | 7487/41250 [18:05:08<80:54:10,  8.63s/it]
                                                          
{'loss': 0.1278, 'grad_norm': 2.3430521488189697, 'learning_rate': 3.764055304535049e-05, 'epoch': 1.82}

 18%|█▊        | 7487/41250 [18:05:08<80:54:10,  8.63s/it][2025-04-26 02:02:52,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 02:02:52,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.93 | bwd_microstep: 5776.06 | bwd_inner_microstep: 5636.12 | bwd_allreduce_microstep: 139.89 | step_microstep: 19.32
[2025-04-26 02:02:52,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.93 | bwd: 5776.07 | bwd_inner: 5636.12 | bwd_allreduce: 139.91 | step: 19.33

 18%|█▊        | 7488/41250 [18:05:17<81:05:18,  8.65s/it]
                                                          
{'loss': 0.1071, 'grad_norm': 2.871640682220459, 'learning_rate': 3.763981305742611e-05, 'epoch': 1.82}

 18%|█▊        | 7488/41250 [18:05:17<81:05:18,  8.65s/it][2025-04-26 02:03:00,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 02:03:00,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.06 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5750.55 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.20
[2025-04-26 02:03:00,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.06 | bwd: 5763.41 | bwd_inner: 5750.55 | bwd_allreduce: 12.82 | step: 19.20

 18%|█▊        | 7489/41250 [18:05:26<81:20:09,  8.67s/it]
                                                          
{'loss': 0.2241, 'grad_norm': 2.106090545654297, 'learning_rate': 3.763907296075576e-05, 'epoch': 1.82}

 18%|█▊        | 7489/41250 [18:05:26<81:20:09,  8.67s/it][2025-04-26 02:03:09,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 02:03:09,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.91 | bwd_microstep: 5758.55 | bwd_inner_microstep: 5635.48 | bwd_allreduce_microstep: 123.03 | step_microstep: 18.78
[2025-04-26 02:03:09,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.91 | bwd: 5758.57 | bwd_inner: 5635.48 | bwd_allreduce: 123.05 | step: 18.79

 18%|█▊        | 7490/41250 [18:05:34<81:19:40,  8.67s/it]
                                                          
{'loss': 0.2293, 'grad_norm': 2.2617416381835938, 'learning_rate': 3.763833275534399e-05, 'epoch': 1.82}

 18%|█▊        | 7490/41250 [18:05:34<81:19:40,  8.67s/it][2025-04-26 02:03:18,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:03:18,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.49 | bwd_microstep: 5695.90 | bwd_inner_microstep: 5683.21 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.39
[2025-04-26 02:03:18,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.49 | bwd: 5695.92 | bwd_inner: 5683.21 | bwd_allreduce: 12.67 | step: 18.40

 18%|█▊        | 7491/41250 [18:05:43<81:11:19,  8.66s/it]
                                                          
{'loss': 0.1269, 'grad_norm': 1.2592639923095703, 'learning_rate': 3.763759244119537e-05, 'epoch': 1.82}

 18%|█▊        | 7491/41250 [18:05:43<81:11:19,  8.66s/it][2025-04-26 02:03:26,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:03:26,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.80 | bwd_microstep: 5734.89 | bwd_inner_microstep: 5697.79 | bwd_allreduce_microstep: 37.06 | step_microstep: 18.80
[2025-04-26 02:03:26,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.80 | bwd: 5734.91 | bwd_inner: 5697.79 | bwd_allreduce: 37.07 | step: 18.80

 18%|█▊        | 7492/41250 [18:05:52<81:12:06,  8.66s/it]
                                                          
{'loss': 0.1429, 'grad_norm': 2.4689817428588867, 'learning_rate': 3.7636852018314466e-05, 'epoch': 1.82}

 18%|█▊        | 7492/41250 [18:05:52<81:12:06,  8.66s/it][2025-04-26 02:03:35,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 02:03:35,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.17 | bwd_microstep: 5776.87 | bwd_inner_microstep: 5650.73 | bwd_allreduce_microstep: 126.08 | step_microstep: 18.70
[2025-04-26 02:03:35,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.17 | bwd: 5776.89 | bwd_inner: 5650.73 | bwd_allreduce: 126.11 | step: 18.71

 18%|█▊        | 7493/41250 [18:06:00<81:17:54,  8.67s/it]
                                                          
{'loss': 0.2665, 'grad_norm': 2.4802443981170654, 'learning_rate': 3.763611148670583e-05, 'epoch': 1.82}

 18%|█▊        | 7493/41250 [18:06:00<81:17:54,  8.67s/it][2025-04-26 02:03:44,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:03:44,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.90 | bwd_microstep: 5709.96 | bwd_inner_microstep: 5697.23 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.51
[2025-04-26 02:03:44,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.91 | bwd: 5709.97 | bwd_inner: 5697.23 | bwd_allreduce: 12.70 | step: 18.51

 18%|█▊        | 7494/41250 [18:06:09<81:12:19,  8.66s/it]
                                                          
{'loss': 0.0613, 'grad_norm': 0.8474976420402527, 'learning_rate': 3.763537084637404e-05, 'epoch': 1.82}

 18%|█▊        | 7494/41250 [18:06:09<81:12:19,  8.66s/it][2025-04-26 02:03:52,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 02:03:52,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.06 | bwd_microstep: 5694.27 | bwd_inner_microstep: 5636.91 | bwd_allreduce_microstep: 57.31 | step_microstep: 18.83
[2025-04-26 02:03:52,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.06 | bwd: 5694.29 | bwd_inner: 5636.91 | bwd_allreduce: 57.33 | step: 18.83

 18%|█▊        | 7495/41250 [18:06:18<81:02:11,  8.64s/it]
                                                          
{'loss': 0.2036, 'grad_norm': 1.6514898538589478, 'learning_rate': 3.7634630097323655e-05, 'epoch': 1.82}

 18%|█▊        | 7495/41250 [18:06:18<81:02:11,  8.64s/it][2025-04-26 02:04:01,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 02:04:01,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.30 | bwd_microstep: 5692.97 | bwd_inner_microstep: 5680.19 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.86
[2025-04-26 02:04:01,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.30 | bwd: 5692.99 | bwd_inner: 5680.19 | bwd_allreduce: 12.76 | step: 18.86

 18%|█▊        | 7496/41250 [18:06:26<81:00:00,  8.64s/it]
                                                          
{'loss': 0.0577, 'grad_norm': 1.2662631273269653, 'learning_rate': 3.763388923955924e-05, 'epoch': 1.82}

 18%|█▊        | 7496/41250 [18:06:26<81:00:00,  8.64s/it][2025-04-26 02:04:10,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:04:10,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.04 | bwd_microstep: 5883.38 | bwd_inner_microstep: 5870.49 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.10
[2025-04-26 02:04:10,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.04 | bwd: 5883.39 | bwd_inner: 5870.49 | bwd_allreduce: 12.86 | step: 19.10

 18%|█▊        | 7497/41250 [18:06:35<81:43:48,  8.72s/it]
                                                          
{'loss': 0.0983, 'grad_norm': 1.668526291847229, 'learning_rate': 3.7633148273085376e-05, 'epoch': 1.82}

 18%|█▊        | 7497/41250 [18:06:35<81:43:48,  8.72s/it][2025-04-26 02:04:18,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 02:04:18,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.71 | bwd_microstep: 5727.34 | bwd_inner_microstep: 5700.48 | bwd_allreduce_microstep: 26.81 | step_microstep: 19.64
[2025-04-26 02:04:18,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.71 | bwd: 5727.36 | bwd_inner: 5700.48 | bwd_allreduce: 26.83 | step: 19.64

 18%|█▊        | 7498/41250 [18:06:44<81:35:07,  8.70s/it]
                                                          
{'loss': 0.3048, 'grad_norm': 6.614038944244385, 'learning_rate': 3.763240719790662e-05, 'epoch': 1.82}

 18%|█▊        | 7498/41250 [18:06:44<81:35:07,  8.70s/it][2025-04-26 02:04:27,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 02:04:27,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.03 | bwd_microstep: 5704.42 | bwd_inner_microstep: 5636.32 | bwd_allreduce_microstep: 68.05 | step_microstep: 18.49
[2025-04-26 02:04:27,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.03 | bwd: 5704.44 | bwd_inner: 5636.32 | bwd_allreduce: 68.07 | step: 18.49

 18%|█▊        | 7499/41250 [18:06:52<81:19:35,  8.67s/it]
                                                          
{'loss': 0.6008, 'grad_norm': 7.514025688171387, 'learning_rate': 3.763166601402754e-05, 'epoch': 1.82}

 18%|█▊        | 7499/41250 [18:06:52<81:19:35,  8.67s/it][2025-04-26 02:04:36,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.11
[2025-04-26 02:04:36,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.73 | bwd_microstep: 5793.89 | bwd_inner_microstep: 5639.69 | bwd_allreduce_microstep: 154.15 | step_microstep: 19.50
[2025-04-26 02:04:36,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.74 | bwd: 5793.90 | bwd_inner: 5639.69 | bwd_allreduce: 154.17 | step: 19.50

 18%|█▊        | 7500/41250 [18:07:01<81:22:28,  8.68s/it]
                                                          
{'loss': 0.2154, 'grad_norm': 1.7492165565490723, 'learning_rate': 3.76309247214527e-05, 'epoch': 1.82}

 18%|█▊        | 7500/41250 [18:07:01<81:22:28,  8.68s/it][2025-04-26 02:04:44,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:04:44,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.04 | bwd_microstep: 5715.54 | bwd_inner_microstep: 5635.89 | bwd_allreduce_microstep: 79.60 | step_microstep: 18.52
[2025-04-26 02:04:44,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.04 | bwd: 5715.56 | bwd_inner: 5635.89 | bwd_allreduce: 79.62 | step: 18.52

 18%|█▊        | 7501/41250 [18:07:10<81:11:17,  8.66s/it]
                                                          
{'loss': 0.2032, 'grad_norm': 2.730041027069092, 'learning_rate': 3.7630183320186685e-05, 'epoch': 1.82}

 18%|█▊        | 7501/41250 [18:07:10<81:11:17,  8.66s/it][2025-04-26 02:04:53,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.15 | optimizer_step: 0.92
[2025-04-26 02:04:53,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.55 | bwd_microstep: 5883.95 | bwd_inner_microstep: 5698.41 | bwd_allreduce_microstep: 185.49 | step_microstep: 19.17
[2025-04-26 02:04:53,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5883.97 | bwd_inner: 5698.41 | bwd_allreduce: 185.51 | step: 19.17

 18%|█▊        | 7502/41250 [18:07:18<81:37:46,  8.71s/it]
                                                          
{'loss': 0.1061, 'grad_norm': 4.395662307739258, 'learning_rate': 3.7629441810234056e-05, 'epoch': 1.82}

 18%|█▊        | 7502/41250 [18:07:18<81:37:46,  8.71s/it][2025-04-26 02:05:02,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 02:05:02,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.42 | bwd_microstep: 5713.56 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.81
[2025-04-26 02:05:02,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.42 | bwd: 5713.57 | bwd_inner: 5700.83 | bwd_allreduce: 12.70 | step: 18.82

 18%|█▊        | 7503/41250 [18:07:27<81:26:30,  8.69s/it]
                                                          
{'loss': 0.0405, 'grad_norm': 0.7355008721351624, 'learning_rate': 3.762870019159938e-05, 'epoch': 1.82}

 18%|█▊        | 7503/41250 [18:07:27<81:26:30,  8.69s/it][2025-04-26 02:05:10,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:05:10,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.22 | bwd_microstep: 5742.14 | bwd_inner_microstep: 5707.68 | bwd_allreduce_microstep: 34.42 | step_microstep: 18.52
[2025-04-26 02:05:10,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.22 | bwd: 5742.16 | bwd_inner: 5707.68 | bwd_allreduce: 34.43 | step: 18.53

 18%|█▊        | 7504/41250 [18:07:36<81:25:01,  8.69s/it]
                                                          
{'loss': 0.1065, 'grad_norm': 1.8430851697921753, 'learning_rate': 3.7627958464287244e-05, 'epoch': 1.82}

 18%|█▊        | 7504/41250 [18:07:36<81:25:01,  8.69s/it][2025-04-26 02:05:19,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.18
[2025-04-26 02:05:19,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.66 | bwd_microstep: 5789.60 | bwd_inner_microstep: 5776.74 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.82
[2025-04-26 02:05:19,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.66 | bwd: 5789.61 | bwd_inner: 5776.74 | bwd_allreduce: 12.82 | step: 19.83

 18%|█▊        | 7505/41250 [18:07:45<81:37:25,  8.71s/it]
                                                          
{'loss': 0.2175, 'grad_norm': 3.384611129760742, 'learning_rate': 3.762721662830221e-05, 'epoch': 1.82}

 18%|█▊        | 7505/41250 [18:07:45<81:37:25,  8.71s/it][2025-04-26 02:05:28,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:05:28,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.15 | bwd_microstep: 5744.27 | bwd_inner_microstep: 5704.71 | bwd_allreduce_microstep: 39.51 | step_microstep: 18.46
[2025-04-26 02:05:28,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.15 | bwd: 5744.28 | bwd_inner: 5704.71 | bwd_allreduce: 39.53 | step: 18.46

 18%|█▊        | 7506/41250 [18:07:53<81:32:04,  8.70s/it]
                                                          
{'loss': 0.0709, 'grad_norm': 0.849391520023346, 'learning_rate': 3.762647468364885e-05, 'epoch': 1.82}

 18%|█▊        | 7506/41250 [18:07:53<81:32:04,  8.70s/it][2025-04-26 02:05:37,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 02:05:37,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.79 | bwd_microstep: 5779.87 | bwd_inner_microstep: 5663.83 | bwd_allreduce_microstep: 116.00 | step_microstep: 19.05
[2025-04-26 02:05:37,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.79 | bwd: 5779.89 | bwd_inner: 5663.83 | bwd_allreduce: 116.02 | step: 19.05

 18%|█▊        | 7507/41250 [18:08:02<81:30:44,  8.70s/it]
                                                          
{'loss': 0.3405, 'grad_norm': 3.033501625061035, 'learning_rate': 3.762573263033174e-05, 'epoch': 1.82}

 18%|█▊        | 7507/41250 [18:08:02<81:30:44,  8.70s/it][2025-04-26 02:05:45,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:05:45,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.16 | bwd_microstep: 5692.92 | bwd_inner_microstep: 5676.90 | bwd_allreduce_microstep: 15.97 | step_microstep: 18.87
[2025-04-26 02:05:45,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.16 | bwd: 5692.93 | bwd_inner: 5676.90 | bwd_allreduce: 15.99 | step: 18.87

 18%|█▊        | 7508/41250 [18:08:11<81:17:19,  8.67s/it]
                                                          
{'loss': 0.1302, 'grad_norm': 1.5089983940124512, 'learning_rate': 3.762499046835545e-05, 'epoch': 1.82}

 18%|█▊        | 7508/41250 [18:08:11<81:17:19,  8.67s/it][2025-04-26 02:05:54,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:05:54,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5698.69 | bwd_inner_microstep: 5669.13 | bwd_allreduce_microstep: 29.51 | step_microstep: 18.51
[2025-04-26 02:05:54,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5698.70 | bwd_inner: 5669.13 | bwd_allreduce: 29.53 | step: 18.52

 18%|█▊        | 7509/41250 [18:08:19<81:07:46,  8.66s/it]
                                                          
{'loss': 0.0654, 'grad_norm': 1.0168042182922363, 'learning_rate': 3.7624248197724564e-05, 'epoch': 1.82}

 18%|█▊        | 7509/41250 [18:08:19<81:07:46,  8.66s/it][2025-04-26 02:06:03,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 02:06:03,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.63 | bwd_microstep: 5794.14 | bwd_inner_microstep: 5658.93 | bwd_allreduce_microstep: 135.16 | step_microstep: 18.89
[2025-04-26 02:06:03,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.63 | bwd: 5794.15 | bwd_inner: 5658.93 | bwd_allreduce: 135.18 | step: 18.90

 18%|█▊        | 7510/41250 [18:08:28<81:16:23,  8.67s/it]
                                                          
{'loss': 0.2024, 'grad_norm': 4.107924938201904, 'learning_rate': 3.762350581844366e-05, 'epoch': 1.82}

 18%|█▊        | 7510/41250 [18:08:28<81:16:23,  8.67s/it][2025-04-26 02:06:11,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 1.12
[2025-04-26 02:06:11,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5698.45 | bwd_inner_microstep: 5685.64 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.57
[2025-04-26 02:06:11,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5698.46 | bwd_inner: 5685.64 | bwd_allreduce: 12.78 | step: 19.58

 18%|█▊        | 7511/41250 [18:08:36<81:08:40,  8.66s/it]
                                                          
{'loss': 0.046, 'grad_norm': 1.3850518465042114, 'learning_rate': 3.7622763330517305e-05, 'epoch': 1.82}

 18%|█▊        | 7511/41250 [18:08:37<81:08:40,  8.66s/it][2025-04-26 02:06:20,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:06:20,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.67 | bwd_microstep: 5716.47 | bwd_inner_microstep: 5703.67 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.15
[2025-04-26 02:06:20,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.67 | bwd: 5716.49 | bwd_inner: 5703.67 | bwd_allreduce: 12.78 | step: 19.16

 18%|█▊        | 7512/41250 [18:08:45<81:07:57,  8.66s/it]
                                                          
{'loss': 0.1051, 'grad_norm': 1.4816441535949707, 'learning_rate': 3.762202073395008e-05, 'epoch': 1.82}

 18%|█▊        | 7512/41250 [18:08:45<81:07:57,  8.66s/it][2025-04-26 02:06:28,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:06:28,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.14 | bwd_microstep: 5702.62 | bwd_inner_microstep: 5661.73 | bwd_allreduce_microstep: 40.85 | step_microstep: 18.84
[2025-04-26 02:06:28,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.14 | bwd: 5702.63 | bwd_inner: 5661.73 | bwd_allreduce: 40.86 | step: 18.85

 18%|█▊        | 7513/41250 [18:08:54<81:00:26,  8.64s/it]
                                                          
{'loss': 0.1256, 'grad_norm': 2.8173983097076416, 'learning_rate': 3.762127802874656e-05, 'epoch': 1.82}

 18%|█▊        | 7513/41250 [18:08:54<81:00:26,  8.64s/it][2025-04-26 02:06:37,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:06:37,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.81 | bwd_microstep: 5717.45 | bwd_inner_microstep: 5704.41 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.65
[2025-04-26 02:06:37,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.81 | bwd: 5717.46 | bwd_inner: 5704.41 | bwd_allreduce: 13.01 | step: 18.66

 18%|█▊        | 7514/41250 [18:09:02<81:01:09,  8.65s/it]
                                                          
{'loss': 0.0747, 'grad_norm': 1.5200659036636353, 'learning_rate': 3.762053521491133e-05, 'epoch': 1.82}

 18%|█▊        | 7514/41250 [18:09:02<81:01:09,  8.65s/it][2025-04-26 02:06:46,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 02:06:46,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.75 | bwd_microstep: 5704.23 | bwd_inner_microstep: 5673.11 | bwd_allreduce_microstep: 31.08 | step_microstep: 18.90
[2025-04-26 02:06:46,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.75 | bwd: 5704.25 | bwd_inner: 5673.11 | bwd_allreduce: 31.10 | step: 18.91

 18%|█▊        | 7515/41250 [18:09:11<80:56:50,  8.64s/it]
                                                          
{'loss': 0.0837, 'grad_norm': 2.528031587600708, 'learning_rate': 3.761979229244897e-05, 'epoch': 1.82}

 18%|█▊        | 7515/41250 [18:09:11<80:56:50,  8.64s/it][2025-04-26 02:06:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 02:06:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.80 | bwd_microstep: 5723.33 | bwd_inner_microstep: 5659.38 | bwd_allreduce_microstep: 63.91 | step_microstep: 18.88
[2025-04-26 02:06:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.80 | bwd: 5723.35 | bwd_inner: 5659.38 | bwd_allreduce: 63.93 | step: 18.88

 18%|█▊        | 7516/41250 [18:09:20<80:55:51,  8.64s/it]
                                                          
{'loss': 0.0317, 'grad_norm': 0.40961843729019165, 'learning_rate': 3.761904926136405e-05, 'epoch': 1.82}

 18%|█▊        | 7516/41250 [18:09:20<80:55:51,  8.64s/it][2025-04-26 02:07:03,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:07:03,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.42 | bwd_microstep: 5693.62 | bwd_inner_microstep: 5654.37 | bwd_allreduce_microstep: 39.20 | step_microstep: 18.50
[2025-04-26 02:07:03,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.42 | bwd: 5693.63 | bwd_inner: 5654.37 | bwd_allreduce: 39.22 | step: 18.50

 18%|█▊        | 7517/41250 [18:09:28<80:51:23,  8.63s/it]
                                                          
{'loss': 0.1587, 'grad_norm': 1.9332960844039917, 'learning_rate': 3.761830612166116e-05, 'epoch': 1.82}

 18%|█▊        | 7517/41250 [18:09:28<80:51:23,  8.63s/it][2025-04-26 02:07:12,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:07:12,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.94 | bwd_microstep: 5726.25 | bwd_inner_microstep: 5652.61 | bwd_allreduce_microstep: 73.59 | step_microstep: 18.75
[2025-04-26 02:07:12,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.94 | bwd: 5726.26 | bwd_inner: 5652.61 | bwd_allreduce: 73.61 | step: 18.75

 18%|█▊        | 7518/41250 [18:09:37<80:52:41,  8.63s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.5472142100334167, 'learning_rate': 3.761756287334488e-05, 'epoch': 1.82}

 18%|█▊        | 7518/41250 [18:09:37<80:52:41,  8.63s/it][2025-04-26 02:07:20,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.34 | optimizer_step: 1.04
[2025-04-26 02:07:20,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.00 | bwd_microstep: 5711.17 | bwd_inner_microstep: 5697.11 | bwd_allreduce_microstep: 14.01 | step_microstep: 20.16
[2025-04-26 02:07:20,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.00 | bwd: 5711.19 | bwd_inner: 5697.10 | bwd_allreduce: 14.03 | step: 20.17

 18%|█▊        | 7519/41250 [18:09:46<80:54:48,  8.64s/it]
                                                          
{'loss': 0.0406, 'grad_norm': 0.6092974543571472, 'learning_rate': 3.761681951641979e-05, 'epoch': 1.82}

 18%|█▊        | 7519/41250 [18:09:46<80:54:48,  8.64s/it][2025-04-26 02:07:29,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 02:07:29,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.56 | bwd_microstep: 5698.79 | bwd_inner_microstep: 5651.64 | bwd_allreduce_microstep: 47.11 | step_microstep: 18.68
[2025-04-26 02:07:29,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.56 | bwd: 5698.80 | bwd_inner: 5651.64 | bwd_allreduce: 47.13 | step: 18.68

 18%|█▊        | 7520/41250 [18:09:54<80:49:58,  8.63s/it]
                                                          
{'loss': 0.2416, 'grad_norm': 2.741051197052002, 'learning_rate': 3.7616076050890466e-05, 'epoch': 1.82}

 18%|█▊        | 7520/41250 [18:09:54<80:49:58,  8.63s/it][2025-04-26 02:07:38,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 02:07:38,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.22 | bwd_microstep: 5907.10 | bwd_inner_microstep: 5661.21 | bwd_allreduce_microstep: 245.84 | step_microstep: 19.09
[2025-04-26 02:07:38,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.22 | bwd: 5907.11 | bwd_inner: 5661.21 | bwd_allreduce: 245.86 | step: 19.09

 18%|█▊        | 7521/41250 [18:10:03<81:21:51,  8.68s/it]
                                                          
{'loss': 0.2391, 'grad_norm': 2.327169418334961, 'learning_rate': 3.76153324767615e-05, 'epoch': 1.82}

 18%|█▊        | 7521/41250 [18:10:03<81:21:51,  8.68s/it][2025-04-26 02:07:46,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:07:46,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5771.91 | bwd_inner_microstep: 5651.38 | bwd_allreduce_microstep: 120.48 | step_microstep: 18.63
[2025-04-26 02:07:46,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5771.92 | bwd_inner: 5651.38 | bwd_allreduce: 120.50 | step: 18.63

 18%|█▊        | 7522/41250 [18:10:12<81:21:04,  8.68s/it]
                                                          
{'loss': 0.2171, 'grad_norm': 4.368993759155273, 'learning_rate': 3.7614588794037475e-05, 'epoch': 1.82}

 18%|█▊        | 7522/41250 [18:10:12<81:21:04,  8.68s/it][2025-04-26 02:07:55,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 02:07:55,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.98 | bwd_microstep: 5715.99 | bwd_inner_microstep: 5659.11 | bwd_allreduce_microstep: 56.83 | step_microstep: 17.99
[2025-04-26 02:07:55,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.98 | bwd: 5716.00 | bwd_inner: 5659.11 | bwd_allreduce: 56.85 | step: 17.99

 18%|█▊        | 7523/41250 [18:10:20<81:11:14,  8.67s/it]
                                                          
{'loss': 0.0293, 'grad_norm': 0.5347867608070374, 'learning_rate': 3.761384500272298e-05, 'epoch': 1.82}

 18%|█▊        | 7523/41250 [18:10:20<81:11:14,  8.67s/it][2025-04-26 02:08:04,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:08:04,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.36 | bwd_microstep: 5748.73 | bwd_inner_microstep: 5651.17 | bwd_allreduce_microstep: 97.51 | step_microstep: 18.57
[2025-04-26 02:08:04,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.36 | bwd: 5748.74 | bwd_inner: 5651.17 | bwd_allreduce: 97.53 | step: 18.57

 18%|█▊        | 7524/41250 [18:10:29<81:10:15,  8.66s/it]
                                                          
{'loss': 0.2122, 'grad_norm': 1.7018033266067505, 'learning_rate': 3.7613101102822593e-05, 'epoch': 1.82}

 18%|█▊        | 7524/41250 [18:10:29<81:10:15,  8.66s/it][2025-04-26 02:08:12,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:08:12,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.88 | bwd_microstep: 5707.36 | bwd_inner_microstep: 5653.05 | bwd_allreduce_microstep: 54.26 | step_microstep: 18.36
[2025-04-26 02:08:12,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.88 | bwd: 5707.37 | bwd_inner: 5653.05 | bwd_allreduce: 54.28 | step: 18.36

 18%|█▊        | 7525/41250 [18:10:38<81:02:06,  8.65s/it]
                                                          
{'loss': 0.3519, 'grad_norm': 2.5958192348480225, 'learning_rate': 3.76123570943409e-05, 'epoch': 1.82}

 18%|█▊        | 7525/41250 [18:10:38<81:02:06,  8.65s/it][2025-04-26 02:08:21,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 02:08:21,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.83 | bwd_microstep: 5687.39 | bwd_inner_microstep: 5674.56 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.48
[2025-04-26 02:08:21,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.83 | bwd: 5687.40 | bwd_inner: 5674.56 | bwd_allreduce: 12.80 | step: 18.48

 18%|█▊        | 7526/41250 [18:10:46<80:55:43,  8.64s/it]
                                                          
{'loss': 0.0563, 'grad_norm': 1.2919237613677979, 'learning_rate': 3.761161297728249e-05, 'epoch': 1.82}

 18%|█▊        | 7526/41250 [18:10:46<80:55:43,  8.64s/it][2025-04-26 02:08:29,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.13 | optimizer_step: 1.03
[2025-04-26 02:08:29,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5705.51 | bwd_inner_microstep: 5647.49 | bwd_allreduce_microstep: 57.96 | step_microstep: 19.81
[2025-04-26 02:08:29,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5705.53 | bwd_inner: 5647.49 | bwd_allreduce: 57.99 | step: 19.81

 18%|█▊        | 7527/41250 [18:10:55<80:51:32,  8.63s/it]
                                                          
{'loss': 0.0449, 'grad_norm': 0.8010697960853577, 'learning_rate': 3.7610868751651943e-05, 'epoch': 1.82}

 18%|█▊        | 7527/41250 [18:10:55<80:51:32,  8.63s/it][2025-04-26 02:08:38,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:08:38,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.28 | bwd_microstep: 5773.93 | bwd_inner_microstep: 5633.97 | bwd_allreduce_microstep: 139.91 | step_microstep: 18.56
[2025-04-26 02:08:38,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.28 | bwd: 5773.94 | bwd_inner: 5633.97 | bwd_allreduce: 139.93 | step: 18.57

 18%|█▊        | 7528/41250 [18:11:03<80:58:29,  8.64s/it]
                                                          
{'loss': 0.1934, 'grad_norm': 2.9174437522888184, 'learning_rate': 3.7610124417453866e-05, 'epoch': 1.82}

 18%|█▊        | 7528/41250 [18:11:03<80:58:29,  8.64s/it][2025-04-26 02:08:47,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:08:47,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.68 | bwd_microstep: 5767.33 | bwd_inner_microstep: 5657.19 | bwd_allreduce_microstep: 110.10 | step_microstep: 18.54
[2025-04-26 02:08:47,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.68 | bwd: 5767.35 | bwd_inner: 5657.19 | bwd_allreduce: 110.12 | step: 18.54

 18%|█▊        | 7529/41250 [18:11:12<81:04:10,  8.65s/it]
                                                          
{'loss': 0.1825, 'grad_norm': 3.4741053581237793, 'learning_rate': 3.760937997469283e-05, 'epoch': 1.83}

 18%|█▊        | 7529/41250 [18:11:12<81:04:10,  8.65s/it][2025-04-26 02:08:56,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 02:08:56,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.62 | bwd_microstep: 5750.06 | bwd_inner_microstep: 5694.18 | bwd_allreduce_microstep: 55.84 | step_microstep: 18.94
[2025-04-26 02:08:56,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.62 | bwd: 5750.07 | bwd_inner: 5694.18 | bwd_allreduce: 55.86 | step: 18.95

 18%|█▊        | 7530/41250 [18:11:21<81:08:15,  8.66s/it]
                                                          
{'loss': 0.3051, 'grad_norm': 20.875995635986328, 'learning_rate': 3.760863542337343e-05, 'epoch': 1.83}

 18%|█▊        | 7530/41250 [18:11:21<81:08:15,  8.66s/it][2025-04-26 02:09:04,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-26 02:09:04,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.48 | bwd_microstep: 5726.30 | bwd_inner_microstep: 5707.64 | bwd_allreduce_microstep: 18.62 | step_microstep: 18.84
[2025-04-26 02:09:04,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.48 | bwd: 5726.32 | bwd_inner: 5707.64 | bwd_allreduce: 18.63 | step: 18.85

 18%|█▊        | 7531/41250 [18:11:29<81:08:09,  8.66s/it]
                                                          
{'loss': 0.1623, 'grad_norm': 1.7427184581756592, 'learning_rate': 3.7607890763500254e-05, 'epoch': 1.83}

 18%|█▊        | 7531/41250 [18:11:30<81:08:09,  8.66s/it][2025-04-26 02:09:13,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:09:13,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5763.41 | bwd_inner_microstep: 5646.58 | bwd_allreduce_microstep: 116.78 | step_microstep: 18.78
[2025-04-26 02:09:13,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5763.42 | bwd_inner: 5646.58 | bwd_allreduce: 116.80 | step: 18.78

 18%|█▊        | 7532/41250 [18:11:38<81:09:44,  8.67s/it]
                                                          
{'loss': 0.0901, 'grad_norm': 1.1502394676208496, 'learning_rate': 3.76071459950779e-05, 'epoch': 1.83}

 18%|█▊        | 7532/41250 [18:11:38<81:09:44,  8.67s/it][2025-04-26 02:09:22,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-26 02:09:22,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.81 | bwd_microstep: 5768.38 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 122.99 | step_microstep: 19.40
[2025-04-26 02:09:22,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.81 | bwd: 5768.40 | bwd_inner: 5645.34 | bwd_allreduce: 123.02 | step: 19.40

 18%|█▊        | 7533/41250 [18:11:47<81:11:09,  8.67s/it]
                                                          
{'loss': 0.1555, 'grad_norm': 1.8360531330108643, 'learning_rate': 3.760640111811095e-05, 'epoch': 1.83}

 18%|█▊        | 7533/41250 [18:11:47<81:11:09,  8.67s/it][2025-04-26 02:09:30,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:09:30,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5741.12 | bwd_inner_microstep: 5704.00 | bwd_allreduce_microstep: 37.08 | step_microstep: 19.00
[2025-04-26 02:09:30,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5741.13 | bwd_inner: 5704.00 | bwd_allreduce: 37.09 | step: 19.00

 18%|█▊        | 7534/41250 [18:11:56<81:11:52,  8.67s/it]
                                                          
{'loss': 0.0808, 'grad_norm': 2.7879014015197754, 'learning_rate': 3.7605656132603995e-05, 'epoch': 1.83}

 18%|█▊        | 7534/41250 [18:11:56<81:11:52,  8.67s/it][2025-04-26 02:09:39,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 02:09:39,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.17 | bwd_microstep: 5747.41 | bwd_inner_microstep: 5647.59 | bwd_allreduce_microstep: 99.78 | step_microstep: 18.85
[2025-04-26 02:09:39,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.17 | bwd: 5747.43 | bwd_inner: 5647.59 | bwd_allreduce: 99.80 | step: 18.85

 18%|█▊        | 7535/41250 [18:12:04<81:08:52,  8.66s/it]
                                                          
{'loss': 0.294, 'grad_norm': 2.2633609771728516, 'learning_rate': 3.760491103856164e-05, 'epoch': 1.83}

 18%|█▊        | 7535/41250 [18:12:04<81:08:52,  8.66s/it][2025-04-26 02:09:48,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 02:09:48,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.12 | bwd_microstep: 5771.37 | bwd_inner_microstep: 5758.48 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.16
[2025-04-26 02:09:48,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.12 | bwd: 5771.38 | bwd_inner: 5758.48 | bwd_allreduce: 12.86 | step: 19.16

 18%|█▊        | 7536/41250 [18:12:13<81:20:45,  8.69s/it]
                                                          
{'loss': 0.2447, 'grad_norm': 3.160491704940796, 'learning_rate': 3.760416583598848e-05, 'epoch': 1.83}

 18%|█▊        | 7536/41250 [18:12:13<81:20:45,  8.69s/it][2025-04-26 02:09:56,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 02:09:56,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.41 | bwd_microstep: 5701.70 | bwd_inner_microstep: 5688.73 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.08
[2025-04-26 02:09:56,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.41 | bwd: 5701.71 | bwd_inner: 5688.73 | bwd_allreduce: 12.94 | step: 19.08

 18%|█▊        | 7537/41250 [18:12:22<81:10:38,  8.67s/it]
                                                          
{'loss': 0.1337, 'grad_norm': 1.925520658493042, 'learning_rate': 3.760342052488908e-05, 'epoch': 1.83}

 18%|█▊        | 7537/41250 [18:12:22<81:10:38,  8.67s/it][2025-04-26 02:10:05,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:10:05,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.99 | bwd_microstep: 5694.83 | bwd_inner_microstep: 5635.53 | bwd_allreduce_microstep: 59.26 | step_microstep: 18.55
[2025-04-26 02:10:05,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.99 | bwd: 5694.85 | bwd_inner: 5635.53 | bwd_allreduce: 59.27 | step: 18.56

 18%|█▊        | 7538/41250 [18:12:30<80:58:18,  8.65s/it]
                                                          
{'loss': 0.102, 'grad_norm': 1.2089189291000366, 'learning_rate': 3.7602675105268065e-05, 'epoch': 1.83}

 18%|█▊        | 7538/41250 [18:12:30<80:58:18,  8.65s/it][2025-04-26 02:10:13,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:10:13,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.24 | bwd_microstep: 5721.95 | bwd_inner_microstep: 5708.98 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.49
[2025-04-26 02:10:13,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.24 | bwd: 5721.96 | bwd_inner: 5708.98 | bwd_allreduce: 12.94 | step: 18.49

 18%|█▊        | 7539/41250 [18:12:39<81:01:52,  8.65s/it]
                                                          
{'loss': 0.0682, 'grad_norm': 1.2000114917755127, 'learning_rate': 3.760192957713003e-05, 'epoch': 1.83}

 18%|█▊        | 7539/41250 [18:12:39<81:01:52,  8.65s/it][2025-04-26 02:10:22,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.29 | optimizer_step: 1.07
[2025-04-26 02:10:22,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.38 | bwd_microstep: 5702.96 | bwd_inner_microstep: 5688.96 | bwd_allreduce_microstep: 13.94 | step_microstep: 20.20
[2025-04-26 02:10:22,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.38 | bwd: 5702.98 | bwd_inner: 5688.96 | bwd_allreduce: 13.97 | step: 20.20

 18%|█▊        | 7540/41250 [18:12:47<80:58:53,  8.65s/it]
                                                          
{'loss': 0.2364, 'grad_norm': 2.463449239730835, 'learning_rate': 3.760118394047955e-05, 'epoch': 1.83}

 18%|█▊        | 7540/41250 [18:12:47<80:58:53,  8.65s/it][2025-04-26 02:10:31,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-26 02:10:31,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.17 | bwd_microstep: 5782.33 | bwd_inner_microstep: 5641.56 | bwd_allreduce_microstep: 140.73 | step_microstep: 18.72
[2025-04-26 02:10:31,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.17 | bwd: 5782.35 | bwd_inner: 5641.56 | bwd_allreduce: 140.74 | step: 18.72

 18%|█▊        | 7541/41250 [18:12:56<81:07:13,  8.66s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 2.370535373687744, 'learning_rate': 3.760043819532123e-05, 'epoch': 1.83}

 18%|█▊        | 7541/41250 [18:12:56<81:07:13,  8.66s/it][2025-04-26 02:10:39,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 02:10:39,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.46 | bwd_microstep: 5781.13 | bwd_inner_microstep: 5641.50 | bwd_allreduce_microstep: 139.58 | step_microstep: 19.03
[2025-04-26 02:10:39,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.46 | bwd: 5781.15 | bwd_inner: 5641.50 | bwd_allreduce: 139.60 | step: 19.03

 18%|█▊        | 7542/41250 [18:13:05<81:10:44,  8.67s/it]
                                                          
{'loss': 0.094, 'grad_norm': 1.5204317569732666, 'learning_rate': 3.7599692341659674e-05, 'epoch': 1.83}

 18%|█▊        | 7542/41250 [18:13:05<81:10:44,  8.67s/it][2025-04-26 02:10:48,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:10:48,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5758.87 | bwd_inner_microstep: 5654.41 | bwd_allreduce_microstep: 104.42 | step_microstep: 18.50
[2025-04-26 02:10:48,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5758.88 | bwd_inner: 5654.41 | bwd_allreduce: 104.44 | step: 18.50

 18%|█▊        | 7543/41250 [18:13:13<81:10:43,  8.67s/it]
                                                          
{'loss': 0.1662, 'grad_norm': 1.3656655550003052, 'learning_rate': 3.759894637949948e-05, 'epoch': 1.83}

 18%|█▊        | 7543/41250 [18:13:13<81:10:43,  8.67s/it][2025-04-26 02:10:57,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:10:57,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.13 | bwd_microstep: 5730.65 | bwd_inner_microstep: 5695.95 | bwd_allreduce_microstep: 34.64 | step_microstep: 19.17
[2025-04-26 02:10:57,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.13 | bwd: 5730.66 | bwd_inner: 5695.95 | bwd_allreduce: 34.66 | step: 19.18

 18%|█▊        | 7544/41250 [18:13:22<81:09:49,  8.67s/it]
                                                          
{'loss': 0.0693, 'grad_norm': 1.5447204113006592, 'learning_rate': 3.759820030884524e-05, 'epoch': 1.83}

 18%|█▊        | 7544/41250 [18:13:22<81:09:49,  8.67s/it][2025-04-26 02:11:05,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 02:11:05,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.06 | bwd_microstep: 5752.37 | bwd_inner_microstep: 5654.01 | bwd_allreduce_microstep: 98.31 | step_microstep: 18.62
[2025-04-26 02:11:05,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.06 | bwd: 5752.39 | bwd_inner: 5654.01 | bwd_allreduce: 98.33 | step: 18.63

 18%|█▊        | 7545/41250 [18:13:31<81:09:10,  8.67s/it]
                                                          
{'loss': 0.0857, 'grad_norm': 2.573286771774292, 'learning_rate': 3.7597454129701555e-05, 'epoch': 1.83}

 18%|█▊        | 7545/41250 [18:13:31<81:09:10,  8.67s/it][2025-04-26 02:11:14,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 02:11:14,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.44 | bwd_microstep: 5680.11 | bwd_inner_microstep: 5644.13 | bwd_allreduce_microstep: 35.94 | step_microstep: 18.50
[2025-04-26 02:11:14,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.44 | bwd: 5680.13 | bwd_inner: 5644.13 | bwd_allreduce: 35.96 | step: 18.50

 18%|█▊        | 7546/41250 [18:13:39<80:57:20,  8.65s/it]
                                                          
{'loss': 0.1877, 'grad_norm': 1.8049148321151733, 'learning_rate': 3.759670784207303e-05, 'epoch': 1.83}

 18%|█▊        | 7546/41250 [18:13:39<80:57:20,  8.65s/it][2025-04-26 02:11:23,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:11:23,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.69 | bwd_microstep: 5696.01 | bwd_inner_microstep: 5647.84 | bwd_allreduce_microstep: 48.13 | step_microstep: 18.56
[2025-04-26 02:11:23,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.69 | bwd: 5696.02 | bwd_inner: 5647.84 | bwd_allreduce: 48.14 | step: 18.57

 18%|█▊        | 7547/41250 [18:13:48<80:53:49,  8.64s/it]
                                                          
{'loss': 0.0927, 'grad_norm': 1.6757495403289795, 'learning_rate': 3.759596144596426e-05, 'epoch': 1.83}

 18%|█▊        | 7547/41250 [18:13:48<80:53:49,  8.64s/it][2025-04-26 02:11:31,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 02:11:31,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.77 | bwd_microstep: 5715.25 | bwd_inner_microstep: 5702.01 | bwd_allreduce_microstep: 13.19 | step_microstep: 18.82
[2025-04-26 02:11:31,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.77 | bwd: 5715.26 | bwd_inner: 5702.01 | bwd_allreduce: 13.21 | step: 18.82

 18%|█▊        | 7548/41250 [18:13:57<80:56:02,  8.65s/it]
                                                          
{'loss': 0.0847, 'grad_norm': 0.9795426726341248, 'learning_rate': 3.7595214941379855e-05, 'epoch': 1.83}

 18%|█▊        | 7548/41250 [18:13:57<80:56:02,  8.65s/it][2025-04-26 02:11:40,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:11:40,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.75 | bwd_microstep: 5742.98 | bwd_inner_microstep: 5709.78 | bwd_allreduce_microstep: 33.16 | step_microstep: 18.43
[2025-04-26 02:11:40,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.75 | bwd: 5742.99 | bwd_inner: 5709.78 | bwd_allreduce: 33.18 | step: 18.44

 18%|█▊        | 7549/41250 [18:14:05<81:03:36,  8.66s/it]
                                                          
{'loss': 0.0676, 'grad_norm': 1.749991774559021, 'learning_rate': 3.7594468328324404e-05, 'epoch': 1.83}

 18%|█▊        | 7549/41250 [18:14:05<81:03:36,  8.66s/it][2025-04-26 02:11:49,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:11:49,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5755.86 | bwd_inner_microstep: 5650.09 | bwd_allreduce_microstep: 105.73 | step_microstep: 18.58
[2025-04-26 02:11:49,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5755.87 | bwd_inner: 5650.09 | bwd_allreduce: 105.75 | step: 18.58

 18%|█▊        | 7550/41250 [18:14:14<81:05:33,  8.66s/it]
                                                          
{'loss': 0.0341, 'grad_norm': 0.6774425506591797, 'learning_rate': 3.7593721606802514e-05, 'epoch': 1.83}

 18%|█▊        | 7550/41250 [18:14:14<81:05:33,  8.66s/it][2025-04-26 02:11:57,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:11:57,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.03 | bwd_microstep: 5717.09 | bwd_inner_microstep: 5704.18 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.73
[2025-04-26 02:11:57,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.03 | bwd: 5717.10 | bwd_inner: 5704.18 | bwd_allreduce: 12.88 | step: 18.73

 18%|█▊        | 7551/41250 [18:14:23<81:04:43,  8.66s/it]
                                                          
{'loss': 0.0834, 'grad_norm': 2.7581377029418945, 'learning_rate': 3.7592974776818794e-05, 'epoch': 1.83}

 18%|█▊        | 7551/41250 [18:14:23<81:04:43,  8.66s/it][2025-04-26 02:12:06,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 02:12:06,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.47 | bwd_microstep: 5745.39 | bwd_inner_microstep: 5690.83 | bwd_allreduce_microstep: 54.51 | step_microstep: 18.26
[2025-04-26 02:12:06,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.47 | bwd: 5745.40 | bwd_inner: 5690.83 | bwd_allreduce: 54.53 | step: 18.26

 18%|█▊        | 7552/41250 [18:14:31<81:07:03,  8.67s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 2.8921658992767334, 'learning_rate': 3.7592227838377846e-05, 'epoch': 1.83}

 18%|█▊        | 7552/41250 [18:14:31<81:07:03,  8.67s/it][2025-04-26 02:12:15,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-26 02:12:15,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5702.30 | bwd_inner_microstep: 5689.48 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.14
[2025-04-26 02:12:15,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5702.32 | bwd_inner: 5689.48 | bwd_allreduce: 12.79 | step: 19.14

 18%|█▊        | 7553/41250 [18:14:40<81:03:16,  8.66s/it]
                                                          
{'loss': 0.114, 'grad_norm': 2.1499183177948, 'learning_rate': 3.7591480791484275e-05, 'epoch': 1.83}

 18%|█▊        | 7553/41250 [18:14:40<81:03:16,  8.66s/it][2025-04-26 02:12:23,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:12:23,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.03 | bwd_microstep: 5736.14 | bwd_inner_microstep: 5692.95 | bwd_allreduce_microstep: 43.14 | step_microstep: 18.32
[2025-04-26 02:12:23,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.03 | bwd: 5736.15 | bwd_inner: 5692.95 | bwd_allreduce: 43.16 | step: 18.32

 18%|█▊        | 7554/41250 [18:14:49<81:06:30,  8.67s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 1.5708030462265015, 'learning_rate': 3.7590733636142685e-05, 'epoch': 1.83}

 18%|█▊        | 7554/41250 [18:14:49<81:06:30,  8.67s/it][2025-04-26 02:12:32,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 02:12:32,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.83 | bwd_microstep: 5748.49 | bwd_inner_microstep: 5698.74 | bwd_allreduce_microstep: 49.71 | step_microstep: 19.04
[2025-04-26 02:12:32,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.83 | bwd: 5748.50 | bwd_inner: 5698.74 | bwd_allreduce: 49.73 | step: 19.04

 18%|█▊        | 7555/41250 [18:14:57<81:08:40,  8.67s/it]
                                                          
{'loss': 0.1457, 'grad_norm': 1.55513596534729, 'learning_rate': 3.7589986372357675e-05, 'epoch': 1.83}

 18%|█▊        | 7555/41250 [18:14:57<81:08:40,  8.67s/it][2025-04-26 02:12:41,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 02:12:41,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5692.50 | bwd_inner_microstep: 5679.78 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.71
[2025-04-26 02:12:41,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 5692.52 | bwd_inner: 5679.77 | bwd_allreduce: 12.70 | step: 18.72

 18%|█▊        | 7556/41250 [18:15:06<81:02:18,  8.66s/it]
                                                          
{'loss': 0.0264, 'grad_norm': 0.9886921644210815, 'learning_rate': 3.758923900013387e-05, 'epoch': 1.83}

 18%|█▊        | 7556/41250 [18:15:06<81:02:18,  8.66s/it][2025-04-26 02:12:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:12:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.37 | bwd_microstep: 5693.91 | bwd_inner_microstep: 5681.03 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.68
[2025-04-26 02:12:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.37 | bwd: 5693.92 | bwd_inner: 5681.03 | bwd_allreduce: 12.85 | step: 18.68

 18%|█▊        | 7557/41250 [18:15:15<80:57:12,  8.65s/it]
                                                          
{'loss': 0.2332, 'grad_norm': 3.1951100826263428, 'learning_rate': 3.758849151947585e-05, 'epoch': 1.83}

 18%|█▊        | 7557/41250 [18:15:15<80:57:12,  8.65s/it][2025-04-26 02:12:58,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:12:58,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5878.38 | bwd_inner_microstep: 5707.24 | bwd_allreduce_microstep: 171.10 | step_microstep: 18.57
[2025-04-26 02:12:58,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5878.40 | bwd_inner: 5707.24 | bwd_allreduce: 171.12 | step: 18.58

 18%|█▊        | 7558/41250 [18:15:23<81:24:09,  8.70s/it]
                                                          
{'loss': 0.2614, 'grad_norm': 4.771475791931152, 'learning_rate': 3.758774393038826e-05, 'epoch': 1.83}

 18%|█▊        | 7558/41250 [18:15:23<81:24:09,  8.70s/it][2025-04-26 02:13:07,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:13:07,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.94 | bwd_microstep: 5703.24 | bwd_inner_microstep: 5690.47 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.02
[2025-04-26 02:13:07,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.94 | bwd: 5703.25 | bwd_inner: 5690.47 | bwd_allreduce: 12.74 | step: 18.03

 18%|█▊        | 7559/41250 [18:15:32<81:13:44,  8.68s/it]
                                                          
{'loss': 0.1365, 'grad_norm': 2.290580987930298, 'learning_rate': 3.758699623287567e-05, 'epoch': 1.83}

 18%|█▊        | 7559/41250 [18:15:32<81:13:44,  8.68s/it][2025-04-26 02:13:15,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 02:13:15,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.53 | bwd_microstep: 5701.51 | bwd_inner_microstep: 5648.57 | bwd_allreduce_microstep: 52.89 | step_microstep: 18.44
[2025-04-26 02:13:15,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.53 | bwd: 5701.52 | bwd_inner: 5648.57 | bwd_allreduce: 52.91 | step: 18.44

 18%|█▊        | 7560/41250 [18:15:41<81:02:29,  8.66s/it]
                                                          
{'loss': 0.0399, 'grad_norm': 0.7614302039146423, 'learning_rate': 3.7586248426942716e-05, 'epoch': 1.83}

 18%|█▊        | 7560/41250 [18:15:41<81:02:29,  8.66s/it][2025-04-26 02:13:24,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:13:24,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.57 | bwd_microstep: 5775.86 | bwd_inner_microstep: 5644.34 | bwd_allreduce_microstep: 131.47 | step_microstep: 18.44
[2025-04-26 02:13:24,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.57 | bwd: 5775.87 | bwd_inner: 5644.34 | bwd_allreduce: 131.49 | step: 18.44

 18%|█▊        | 7561/41250 [18:15:49<81:08:22,  8.67s/it]
                                                          
{'loss': 0.0368, 'grad_norm': 0.5080866813659668, 'learning_rate': 3.758550051259399e-05, 'epoch': 1.83}

 18%|█▊        | 7561/41250 [18:15:49<81:08:22,  8.67s/it][2025-04-26 02:13:33,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 02:13:33,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.65 | bwd_microstep: 5753.32 | bwd_inner_microstep: 5693.50 | bwd_allreduce_microstep: 59.78 | step_microstep: 18.58
[2025-04-26 02:13:33,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.65 | bwd: 5753.33 | bwd_inner: 5693.50 | bwd_allreduce: 59.79 | step: 18.59

 18%|█▊        | 7562/41250 [18:15:58<81:11:26,  8.68s/it]
                                                          
{'loss': 0.1859, 'grad_norm': 3.7991206645965576, 'learning_rate': 3.758475248983413e-05, 'epoch': 1.83}

 18%|█▊        | 7562/41250 [18:15:58<81:11:26,  8.68s/it][2025-04-26 02:13:41,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.88 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:13:41,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.97 | bwd_microstep: 5776.51 | bwd_inner_microstep: 5657.20 | bwd_allreduce_microstep: 119.26 | step_microstep: 18.51
[2025-04-26 02:13:41,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.97 | bwd: 5776.53 | bwd_inner: 5657.20 | bwd_allreduce: 119.28 | step: 18.52

 18%|█▊        | 7563/41250 [18:16:07<81:13:17,  8.68s/it]
                                                          
{'loss': 0.2768, 'grad_norm': 2.052696466445923, 'learning_rate': 3.7584004358667715e-05, 'epoch': 1.83}

 18%|█▊        | 7563/41250 [18:16:07<81:13:17,  8.68s/it][2025-04-26 02:13:50,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:13:50,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.05 | bwd_microstep: 5706.29 | bwd_inner_microstep: 5693.61 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.52
[2025-04-26 02:13:50,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.05 | bwd: 5706.31 | bwd_inner: 5693.61 | bwd_allreduce: 12.65 | step: 18.52

 18%|█▊        | 7564/41250 [18:16:15<81:06:55,  8.67s/it]
                                                          
{'loss': 0.0528, 'grad_norm': 1.0667699575424194, 'learning_rate': 3.7583256119099374e-05, 'epoch': 1.83}

 18%|█▊        | 7564/41250 [18:16:15<81:06:55,  8.67s/it][2025-04-26 02:13:59,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.30 | optimizer_step: 1.04
[2025-04-26 02:13:59,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5868.83 | bwd_inner_microstep: 5641.16 | bwd_allreduce_microstep: 227.61 | step_microstep: 20.00
[2025-04-26 02:13:59,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5868.85 | bwd_inner: 5641.16 | bwd_allreduce: 227.64 | step: 20.00

 18%|█▊        | 7565/41250 [18:16:24<81:25:51,  8.70s/it]
                                                          
{'loss': 0.3019, 'grad_norm': 2.719003677368164, 'learning_rate': 3.758250777113373e-05, 'epoch': 1.83}

 18%|█▊        | 7565/41250 [18:16:24<81:25:51,  8.70s/it][2025-04-26 02:14:08,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:14:08,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.54 | bwd_microstep: 5770.65 | bwd_inner_microstep: 5640.22 | bwd_allreduce_microstep: 130.38 | step_microstep: 18.78
[2025-04-26 02:14:08,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.54 | bwd: 5770.67 | bwd_inner: 5640.22 | bwd_allreduce: 130.40 | step: 18.78

 18%|█▊        | 7566/41250 [18:16:33<81:20:32,  8.69s/it]
                                                          
{'loss': 0.0386, 'grad_norm': 0.7239093780517578, 'learning_rate': 3.758175931477537e-05, 'epoch': 1.83}

 18%|█▊        | 7566/41250 [18:16:33<81:20:32,  8.69s/it][2025-04-26 02:14:16,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 02:14:16,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.24 | bwd_microstep: 5729.51 | bwd_inner_microstep: 5707.95 | bwd_allreduce_microstep: 21.50 | step_microstep: 19.11
[2025-04-26 02:14:16,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.24 | bwd: 5729.52 | bwd_inner: 5707.95 | bwd_allreduce: 21.52 | step: 19.12

 18%|█▊        | 7567/41250 [18:16:42<81:15:09,  8.68s/it]
                                                          
{'loss': 0.2428, 'grad_norm': 4.6000895500183105, 'learning_rate': 3.7581010750028924e-05, 'epoch': 1.83}

 18%|█▊        | 7567/41250 [18:16:42<81:15:09,  8.68s/it][2025-04-26 02:14:25,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 02:14:25,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.04 | bwd_microstep: 5735.05 | bwd_inner_microstep: 5686.46 | bwd_allreduce_microstep: 48.54 | step_microstep: 19.26
[2025-04-26 02:14:25,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5735.07 | bwd_inner: 5686.46 | bwd_allreduce: 48.56 | step: 19.26

 18%|█▊        | 7568/41250 [18:16:50<81:12:51,  8.68s/it]
                                                          
{'loss': 0.0977, 'grad_norm': 3.9959800243377686, 'learning_rate': 3.7580262076899004e-05, 'epoch': 1.83}

 18%|█▊        | 7568/41250 [18:16:50<81:12:51,  8.68s/it][2025-04-26 02:14:34,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:14:34,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.26 | bwd_microstep: 5886.03 | bwd_inner_microstep: 5636.30 | bwd_allreduce_microstep: 249.68 | step_microstep: 18.95
[2025-04-26 02:14:34,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.26 | bwd: 5886.05 | bwd_inner: 5636.30 | bwd_allreduce: 249.70 | step: 18.95

 18%|█▊        | 7569/41250 [18:16:59<81:33:07,  8.72s/it]
                                                          
{'loss': 0.0998, 'grad_norm': 2.1275668144226074, 'learning_rate': 3.7579513295390235e-05, 'epoch': 1.83}

 18%|█▊        | 7569/41250 [18:16:59<81:33:07,  8.72s/it][2025-04-26 02:14:42,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:14:42,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.01 | bwd_microstep: 5688.47 | bwd_inner_microstep: 5675.70 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.03
[2025-04-26 02:14:42,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.01 | bwd: 5688.48 | bwd_inner: 5675.70 | bwd_allreduce: 12.75 | step: 19.03

 18%|█▊        | 7570/41250 [18:17:08<81:16:23,  8.69s/it]
                                                          
{'loss': 0.3138, 'grad_norm': 2.061302661895752, 'learning_rate': 3.757876440550721e-05, 'epoch': 1.84}

 18%|█▊        | 7570/41250 [18:17:08<81:16:23,  8.69s/it][2025-04-26 02:14:51,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:14:51,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.37 | bwd_microstep: 5670.39 | bwd_inner_microstep: 5637.40 | bwd_allreduce_microstep: 32.95 | step_microstep: 18.89
[2025-04-26 02:14:51,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.37 | bwd: 5670.40 | bwd_inner: 5637.40 | bwd_allreduce: 32.97 | step: 18.90

 18%|█▊        | 7571/41250 [18:17:16<80:58:05,  8.65s/it]
                                                          
{'loss': 0.1456, 'grad_norm': 1.5665549039840698, 'learning_rate': 3.757801540725457e-05, 'epoch': 1.84}

 18%|█▊        | 7571/41250 [18:17:16<80:58:05,  8.65s/it][2025-04-26 02:15:00,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 02:15:00,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.16 | bwd_microstep: 5704.57 | bwd_inner_microstep: 5691.54 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.81
[2025-04-26 02:15:00,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.16 | bwd: 5704.58 | bwd_inner: 5691.54 | bwd_allreduce: 13.00 | step: 18.81

 18%|█▊        | 7572/41250 [18:17:25<80:55:10,  8.65s/it]
                                                          
{'loss': 0.1198, 'grad_norm': 1.8899986743927002, 'learning_rate': 3.757726630063692e-05, 'epoch': 1.84}

 18%|█▊        | 7572/41250 [18:17:25<80:55:10,  8.65s/it][2025-04-26 02:15:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.14 | optimizer_step: 1.03
[2025-04-26 02:15:08,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.94 | bwd_microstep: 5746.17 | bwd_inner_microstep: 5652.27 | bwd_allreduce_microstep: 93.79 | step_microstep: 20.58
[2025-04-26 02:15:08,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.94 | bwd: 5746.21 | bwd_inner: 5652.27 | bwd_allreduce: 93.85 | step: 20.57

 18%|█▊        | 7573/41250 [18:17:34<80:56:24,  8.65s/it]
                                                          
{'loss': 0.169, 'grad_norm': 1.8660835027694702, 'learning_rate': 3.757651708565888e-05, 'epoch': 1.84}

 18%|█▊        | 7573/41250 [18:17:34<80:56:24,  8.65s/it][2025-04-26 02:15:17,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 02:15:17,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.28 | bwd_microstep: 5675.13 | bwd_inner_microstep: 5651.19 | bwd_allreduce_microstep: 23.89 | step_microstep: 18.87
[2025-04-26 02:15:17,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.28 | bwd: 5675.14 | bwd_inner: 5651.19 | bwd_allreduce: 23.91 | step: 18.87

 18%|█▊        | 7574/41250 [18:17:42<80:44:44,  8.63s/it]
                                                          
{'loss': 0.0398, 'grad_norm': 0.4117273986339569, 'learning_rate': 3.7575767762325075e-05, 'epoch': 1.84}

 18%|█▊        | 7574/41250 [18:17:42<80:44:44,  8.63s/it][2025-04-26 02:15:25,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 02:15:25,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.05 | bwd_microstep: 5744.44 | bwd_inner_microstep: 5680.41 | bwd_allreduce_microstep: 63.98 | step_microstep: 18.54
[2025-04-26 02:15:25,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.05 | bwd: 5744.45 | bwd_inner: 5680.41 | bwd_allreduce: 64.00 | step: 18.54

 18%|█▊        | 7575/41250 [18:17:51<80:51:52,  8.64s/it]
                                                          
{'loss': 0.0216, 'grad_norm': 0.23894910514354706, 'learning_rate': 3.757501833064011e-05, 'epoch': 1.84}

 18%|█▊        | 7575/41250 [18:17:51<80:51:52,  8.64s/it][2025-04-26 02:15:34,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 02:15:34,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.65 | bwd_microstep: 5742.20 | bwd_inner_microstep: 5643.00 | bwd_allreduce_microstep: 99.15 | step_microstep: 18.70
[2025-04-26 02:15:34,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.65 | bwd: 5742.22 | bwd_inner: 5643.00 | bwd_allreduce: 99.17 | step: 18.70

 18%|█▊        | 7576/41250 [18:17:59<80:50:58,  8.64s/it]
                                                          
{'loss': 0.4465, 'grad_norm': 3.761159658432007, 'learning_rate': 3.7574268790608616e-05, 'epoch': 1.84}

 18%|█▊        | 7576/41250 [18:17:59<80:50:58,  8.64s/it][2025-04-26 02:15:43,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:15:43,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.66 | bwd_microstep: 5774.48 | bwd_inner_microstep: 5761.76 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.77
[2025-04-26 02:15:43,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.66 | bwd: 5774.49 | bwd_inner: 5761.76 | bwd_allreduce: 12.69 | step: 18.77

 18%|█▊        | 7577/41250 [18:18:08<81:06:48,  8.67s/it]
                                                          
{'loss': 0.0689, 'grad_norm': 1.0331701040267944, 'learning_rate': 3.7573519142235216e-05, 'epoch': 1.84}

 18%|█▊        | 7577/41250 [18:18:08<81:06:48,  8.67s/it][2025-04-26 02:15:51,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:15:51,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.57 | bwd_microstep: 5731.56 | bwd_inner_microstep: 5672.76 | bwd_allreduce_microstep: 58.76 | step_microstep: 19.11
[2025-04-26 02:15:51,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.57 | bwd: 5731.58 | bwd_inner: 5672.76 | bwd_allreduce: 58.77 | step: 19.11

 18%|█▊        | 7578/41250 [18:18:17<81:03:39,  8.67s/it]
                                                          
{'loss': 0.3518, 'grad_norm': 3.225346088409424, 'learning_rate': 3.757276938552452e-05, 'epoch': 1.84}

 18%|█▊        | 7578/41250 [18:18:17<81:03:39,  8.67s/it][2025-04-26 02:16:00,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:16:00,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.93 | bwd_microstep: 5699.77 | bwd_inner_microstep: 5686.98 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.56
[2025-04-26 02:16:00,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.93 | bwd: 5699.78 | bwd_inner: 5686.98 | bwd_allreduce: 12.75 | step: 18.57

 18%|█▊        | 7579/41250 [18:18:25<80:56:37,  8.65s/it]
                                                          
{'loss': 0.0522, 'grad_norm': 0.781453013420105, 'learning_rate': 3.7572019520481164e-05, 'epoch': 1.84}

 18%|█▊        | 7579/41250 [18:18:25<80:56:37,  8.65s/it][2025-04-26 02:16:09,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.20
[2025-04-26 02:16:09,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5705.06 | bwd_inner_microstep: 5692.21 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.59
[2025-04-26 02:16:09,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5705.07 | bwd_inner: 5692.21 | bwd_allreduce: 12.82 | step: 19.60

 18%|█▊        | 7580/41250 [18:18:34<80:54:08,  8.65s/it]
                                                          
{'loss': 0.0771, 'grad_norm': 0.8485404253005981, 'learning_rate': 3.757126954710976e-05, 'epoch': 1.84}

 18%|█▊        | 7580/41250 [18:18:34<80:54:08,  8.65s/it][2025-04-26 02:16:17,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.72 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:16:17,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.85 | bwd_microstep: 5763.86 | bwd_inner_microstep: 5648.45 | bwd_allreduce_microstep: 115.35 | step_microstep: 20.22
[2025-04-26 02:16:17,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.85 | bwd: 5763.87 | bwd_inner: 5648.45 | bwd_allreduce: 115.38 | step: 20.22

 18%|█▊        | 7581/41250 [18:18:43<80:57:40,  8.66s/it]
                                                          
{'loss': 0.0461, 'grad_norm': 0.8674254417419434, 'learning_rate': 3.757051946541494e-05, 'epoch': 1.84}

 18%|█▊        | 7581/41250 [18:18:43<80:57:40,  8.66s/it][2025-04-26 02:16:26,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 02:16:26,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.39 | bwd_microstep: 5757.59 | bwd_inner_microstep: 5641.70 | bwd_allreduce_microstep: 115.84 | step_microstep: 18.99
[2025-04-26 02:16:26,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.39 | bwd: 5757.61 | bwd_inner: 5641.70 | bwd_allreduce: 115.87 | step: 18.99

 18%|█▊        | 7582/41250 [18:18:51<80:58:02,  8.66s/it]
                                                          
{'loss': 0.0609, 'grad_norm': 1.9724520444869995, 'learning_rate': 3.756976927540132e-05, 'epoch': 1.84}

 18%|█▊        | 7582/41250 [18:18:51<80:58:02,  8.66s/it][2025-04-26 02:16:35,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:16:35,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.38 | bwd_microstep: 5701.08 | bwd_inner_microstep: 5640.21 | bwd_allreduce_microstep: 60.82 | step_microstep: 18.77
[2025-04-26 02:16:35,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.38 | bwd: 5701.09 | bwd_inner: 5640.21 | bwd_allreduce: 60.84 | step: 18.78

 18%|█▊        | 7583/41250 [18:19:00<80:51:34,  8.65s/it]
                                                          
{'loss': 0.2518, 'grad_norm': 7.268094062805176, 'learning_rate': 3.7569018977073526e-05, 'epoch': 1.84}

 18%|█▊        | 7583/41250 [18:19:00<80:51:34,  8.65s/it][2025-04-26 02:16:43,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 02:16:43,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.40 | bwd_microstep: 5700.89 | bwd_inner_microstep: 5649.17 | bwd_allreduce_microstep: 51.67 | step_microstep: 19.17
[2025-04-26 02:16:43,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.40 | bwd: 5700.91 | bwd_inner: 5649.17 | bwd_allreduce: 51.69 | step: 19.17

 18%|█▊        | 7584/41250 [18:19:09<80:45:21,  8.64s/it]
                                                          
{'loss': 0.1126, 'grad_norm': 1.9192017316818237, 'learning_rate': 3.756826857043619e-05, 'epoch': 1.84}

 18%|█▊        | 7584/41250 [18:19:09<80:45:21,  8.64s/it][2025-04-26 02:16:52,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.16 | optimizer_step: 0.92
[2025-04-26 02:16:52,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.38 | bwd_microstep: 5875.36 | bwd_inner_microstep: 5701.97 | bwd_allreduce_microstep: 173.34 | step_microstep: 18.97
[2025-04-26 02:16:52,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.38 | bwd: 5875.38 | bwd_inner: 5701.97 | bwd_allreduce: 173.36 | step: 18.97

 18%|█▊        | 7585/41250 [18:19:17<81:14:13,  8.69s/it]
                                                          
{'loss': 0.2392, 'grad_norm': 2.6329243183135986, 'learning_rate': 3.756751805549393e-05, 'epoch': 1.84}

 18%|█▊        | 7585/41250 [18:19:17<81:14:13,  8.69s/it][2025-04-26 02:17:01,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:17:01,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.11 | bwd_microstep: 5775.85 | bwd_inner_microstep: 5651.34 | bwd_allreduce_microstep: 124.47 | step_microstep: 18.69
[2025-04-26 02:17:01,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.11 | bwd: 5775.87 | bwd_inner: 5651.34 | bwd_allreduce: 124.49 | step: 18.69

 18%|█▊        | 7586/41250 [18:19:26<81:13:51,  8.69s/it]
                                                          
{'loss': 0.0769, 'grad_norm': 1.593944787979126, 'learning_rate': 3.756676743225137e-05, 'epoch': 1.84}

 18%|█▊        | 7586/41250 [18:19:26<81:13:51,  8.69s/it][2025-04-26 02:17:09,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-26 02:17:09,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5747.92 | bwd_inner_microstep: 5692.26 | bwd_allreduce_microstep: 55.62 | step_microstep: 18.55
[2025-04-26 02:17:09,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5747.93 | bwd_inner: 5692.26 | bwd_allreduce: 55.64 | step: 18.55

 18%|█▊        | 7587/41250 [18:19:35<81:11:25,  8.68s/it]
                                                          
{'loss': 0.1591, 'grad_norm': 2.7274296283721924, 'learning_rate': 3.756601670071316e-05, 'epoch': 1.84}

 18%|█▊        | 7587/41250 [18:19:35<81:11:25,  8.68s/it][2025-04-26 02:17:18,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 02:17:18,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.15 | bwd_microstep: 5759.69 | bwd_inner_microstep: 5644.09 | bwd_allreduce_microstep: 115.55 | step_microstep: 18.73
[2025-04-26 02:17:18,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.15 | bwd: 5759.70 | bwd_inner: 5644.09 | bwd_allreduce: 115.57 | step: 18.74

 18%|█▊        | 7588/41250 [18:19:43<81:08:01,  8.68s/it]
                                                          
{'loss': 0.3605, 'grad_norm': 1.8611782789230347, 'learning_rate': 3.75652658608839e-05, 'epoch': 1.84}

 18%|█▊        | 7588/41250 [18:19:43<81:08:01,  8.68s/it][2025-04-26 02:17:27,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 02:17:27,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5695.92 | bwd_inner_microstep: 5683.14 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.50
[2025-04-26 02:17:27,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5695.93 | bwd_inner: 5683.14 | bwd_allreduce: 12.74 | step: 18.50

 18%|█▊        | 7589/41250 [18:19:52<80:59:00,  8.66s/it]
                                                          
{'loss': 0.0514, 'grad_norm': 2.0539681911468506, 'learning_rate': 3.7564514912768234e-05, 'epoch': 1.84}

 18%|█▊        | 7589/41250 [18:19:52<80:59:00,  8.66s/it][2025-04-26 02:17:35,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:17:35,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.48 | bwd_microstep: 5793.42 | bwd_inner_microstep: 5636.23 | bwd_allreduce_microstep: 157.15 | step_microstep: 18.82
[2025-04-26 02:17:35,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.48 | bwd: 5793.43 | bwd_inner: 5636.23 | bwd_allreduce: 157.16 | step: 18.83

 18%|█▊        | 7590/41250 [18:20:01<81:04:32,  8.67s/it]
                                                          
{'loss': 0.1861, 'grad_norm': 2.259657382965088, 'learning_rate': 3.7563763856370794e-05, 'epoch': 1.84}

 18%|█▊        | 7590/41250 [18:20:01<81:04:32,  8.67s/it][2025-04-26 02:17:44,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:17:44,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 5716.42 | bwd_inner_microstep: 5647.02 | bwd_allreduce_microstep: 69.35 | step_microstep: 18.81
[2025-04-26 02:17:44,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 5716.43 | bwd_inner: 5647.02 | bwd_allreduce: 69.37 | step: 18.81

 18%|█▊        | 7591/41250 [18:20:09<80:56:11,  8.66s/it]
                                                          
{'loss': 0.2403, 'grad_norm': 3.9679040908813477, 'learning_rate': 3.75630126916962e-05, 'epoch': 1.84}

 18%|█▊        | 7591/41250 [18:20:09<80:56:11,  8.66s/it][2025-04-26 02:17:53,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.35 | optimizer_step: 1.04
[2025-04-26 02:17:53,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.39 | bwd_microstep: 5768.00 | bwd_inner_microstep: 5680.48 | bwd_allreduce_microstep: 87.46 | step_microstep: 20.23
[2025-04-26 02:17:53,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.38 | bwd: 5768.02 | bwd_inner: 5680.48 | bwd_allreduce: 87.49 | step: 20.23

 18%|█▊        | 7592/41250 [18:20:18<81:02:39,  8.67s/it]
                                                          
{'loss': 0.2724, 'grad_norm': 2.867647886276245, 'learning_rate': 3.7562261418749085e-05, 'epoch': 1.84}

 18%|█▊        | 7592/41250 [18:20:18<81:02:39,  8.67s/it][2025-04-26 02:18:01,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:18:01,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.44 | bwd_microstep: 5750.42 | bwd_inner_microstep: 5699.56 | bwd_allreduce_microstep: 50.82 | step_microstep: 18.51
[2025-04-26 02:18:01,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.44 | bwd: 5750.44 | bwd_inner: 5699.56 | bwd_allreduce: 50.83 | step: 18.52

 18%|█▊        | 7593/41250 [18:20:27<81:05:43,  8.67s/it]
                                                          
{'loss': 0.191, 'grad_norm': 3.083501100540161, 'learning_rate': 3.756151003753409e-05, 'epoch': 1.84}

 18%|█▊        | 7593/41250 [18:20:27<81:05:43,  8.67s/it][2025-04-26 02:18:10,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:18:10,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5765.53 | bwd_inner_microstep: 5712.92 | bwd_allreduce_microstep: 52.57 | step_microstep: 18.58
[2025-04-26 02:18:10,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5765.55 | bwd_inner: 5712.92 | bwd_allreduce: 52.59 | step: 18.58

 18%|█▊        | 7594/41250 [18:20:35<81:10:00,  8.68s/it]
                                                          
{'loss': 0.1617, 'grad_norm': 2.97817325592041, 'learning_rate': 3.756075854805583e-05, 'epoch': 1.84}

 18%|█▊        | 7594/41250 [18:20:36<81:10:00,  8.68s/it][2025-04-26 02:18:19,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:18:19,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.97 | bwd_microstep: 5709.40 | bwd_inner_microstep: 5655.10 | bwd_allreduce_microstep: 54.26 | step_microstep: 18.45
[2025-04-26 02:18:19,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.97 | bwd: 5709.41 | bwd_inner: 5655.10 | bwd_allreduce: 54.27 | step: 18.45

 18%|█▊        | 7595/41250 [18:20:44<80:58:59,  8.66s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 1.6126221418380737, 'learning_rate': 3.756000695031895e-05, 'epoch': 1.84}

 18%|█▊        | 7595/41250 [18:20:44<80:58:59,  8.66s/it][2025-04-26 02:18:27,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-26 02:18:27,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.94 | bwd_microstep: 5790.06 | bwd_inner_microstep: 5666.27 | bwd_allreduce_microstep: 123.74 | step_microstep: 18.92
[2025-04-26 02:18:28,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.94 | bwd: 5790.08 | bwd_inner: 5666.27 | bwd_allreduce: 123.77 | step: 18.93

 18%|█▊        | 7596/41250 [18:20:53<81:06:13,  8.68s/it]
                                                          
{'loss': 0.2918, 'grad_norm': 2.6556835174560547, 'learning_rate': 3.755925524432808e-05, 'epoch': 1.84}

 18%|█▊        | 7596/41250 [18:20:53<81:06:13,  8.68s/it][2025-04-26 02:18:36,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 02:18:36,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.55 | bwd_microstep: 5791.77 | bwd_inner_microstep: 5661.69 | bwd_allreduce_microstep: 130.03 | step_microstep: 18.57
[2025-04-26 02:18:36,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5791.78 | bwd_inner: 5661.69 | bwd_allreduce: 130.05 | step: 18.57

 18%|█▊        | 7597/41250 [18:21:02<81:10:48,  8.68s/it]
                                                          
{'loss': 0.4433, 'grad_norm': 3.6208910942077637, 'learning_rate': 3.755850343008786e-05, 'epoch': 1.84}

 18%|█▊        | 7597/41250 [18:21:02<81:10:48,  8.68s/it][2025-04-26 02:18:45,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 02:18:45,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.87 | bwd_microstep: 5713.54 | bwd_inner_microstep: 5700.80 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.83
[2025-04-26 02:18:45,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.87 | bwd: 5713.55 | bwd_inner: 5700.80 | bwd_allreduce: 12.71 | step: 18.83

 18%|█▊        | 7598/41250 [18:21:10<81:04:42,  8.67s/it]
                                                          
{'loss': 0.2061, 'grad_norm': 2.2734203338623047, 'learning_rate': 3.755775150760292e-05, 'epoch': 1.84}

 18%|█▊        | 7598/41250 [18:21:10<81:04:42,  8.67s/it][2025-04-26 02:18:54,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 02:18:54,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.23 | bwd_microstep: 5776.85 | bwd_inner_microstep: 5694.51 | bwd_allreduce_microstep: 82.29 | step_microstep: 19.02
[2025-04-26 02:18:54,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.23 | bwd: 5776.86 | bwd_inner: 5694.51 | bwd_allreduce: 82.31 | step: 19.03

 18%|█▊        | 7599/41250 [18:21:19<81:09:44,  8.68s/it]
                                                          
{'loss': 0.2081, 'grad_norm': 2.4323604106903076, 'learning_rate': 3.755699947687789e-05, 'epoch': 1.84}

 18%|█▊        | 7599/41250 [18:21:19<81:09:44,  8.68s/it][2025-04-26 02:19:02,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:19:02,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.58 | bwd_microstep: 5789.33 | bwd_inner_microstep: 5776.44 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.13
[2025-04-26 02:19:02,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.58 | bwd: 5789.35 | bwd_inner: 5776.44 | bwd_allreduce: 12.86 | step: 19.13

 18%|█▊        | 7600/41250 [18:21:28<81:22:31,  8.71s/it]
                                                          
{'loss': 0.2407, 'grad_norm': 1.9750858545303345, 'learning_rate': 3.755624733791742e-05, 'epoch': 1.84}

 18%|█▊        | 7600/41250 [18:21:28<81:22:31,  8.71s/it][2025-04-26 02:19:11,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 02:19:11,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.53 | bwd_microstep: 5727.37 | bwd_inner_microstep: 5697.16 | bwd_allreduce_microstep: 30.16 | step_microstep: 19.25
[2025-04-26 02:19:11,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.53 | bwd: 5727.38 | bwd_inner: 5697.16 | bwd_allreduce: 30.18 | step: 19.25

 18%|█▊        | 7601/41250 [18:21:36<81:16:21,  8.70s/it]
                                                          
{'loss': 0.1515, 'grad_norm': 4.214780807495117, 'learning_rate': 3.755549509072613e-05, 'epoch': 1.84}

 18%|█▊        | 7601/41250 [18:21:36<81:16:21,  8.70s/it][2025-04-26 02:19:20,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:19:20,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.00 | bwd_microstep: 5760.74 | bwd_inner_microstep: 5662.49 | bwd_allreduce_microstep: 98.20 | step_microstep: 19.01
[2025-04-26 02:19:20,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.00 | bwd: 5760.76 | bwd_inner: 5662.49 | bwd_allreduce: 98.22 | step: 19.01

 18%|█▊        | 7602/41250 [18:21:45<81:12:48,  8.69s/it]
                                                          
{'loss': 0.1106, 'grad_norm': 2.0817456245422363, 'learning_rate': 3.7554742735308674e-05, 'epoch': 1.84}

 18%|█▊        | 7602/41250 [18:21:45<81:12:48,  8.69s/it][2025-04-26 02:19:28,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:19:28,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.86 | bwd_microstep: 5718.57 | bwd_inner_microstep: 5705.69 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.54
[2025-04-26 02:19:28,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.86 | bwd: 5718.59 | bwd_inner: 5705.69 | bwd_allreduce: 12.86 | step: 18.54

 18%|█▊        | 7603/41250 [18:21:54<81:06:23,  8.68s/it]
                                                          
{'loss': 0.1154, 'grad_norm': 1.665610671043396, 'learning_rate': 3.7553990271669676e-05, 'epoch': 1.84}

 18%|█▊        | 7603/41250 [18:21:54<81:06:23,  8.68s/it][2025-04-26 02:19:37,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:19:37,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.48 | bwd_microstep: 5763.19 | bwd_inner_microstep: 5691.42 | bwd_allreduce_microstep: 71.73 | step_microstep: 18.69
[2025-04-26 02:19:37,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.48 | bwd: 5763.21 | bwd_inner: 5691.42 | bwd_allreduce: 71.75 | step: 18.69

 18%|█▊        | 7604/41250 [18:22:02<81:09:53,  8.68s/it]
                                                          
{'loss': 0.185, 'grad_norm': 6.374874591827393, 'learning_rate': 3.7553237699813775e-05, 'epoch': 1.84}

 18%|█▊        | 7604/41250 [18:22:02<81:09:53,  8.68s/it][2025-04-26 02:19:46,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:19:46,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.65 | bwd_microstep: 5780.20 | bwd_inner_microstep: 5641.18 | bwd_allreduce_microstep: 138.96 | step_microstep: 18.80
[2025-04-26 02:19:46,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.65 | bwd: 5780.21 | bwd_inner: 5641.18 | bwd_allreduce: 138.98 | step: 18.80

 18%|█▊        | 7605/41250 [18:22:11<81:10:18,  8.69s/it]
                                                          
{'loss': 0.0443, 'grad_norm': 1.8573896884918213, 'learning_rate': 3.755248501974563e-05, 'epoch': 1.84}

 18%|█▊        | 7605/41250 [18:22:11<81:10:18,  8.69s/it][2025-04-26 02:19:54,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 02:19:54,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5697.98 | bwd_inner_microstep: 5651.57 | bwd_allreduce_microstep: 46.36 | step_microstep: 19.57
[2025-04-26 02:19:54,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5697.99 | bwd_inner: 5651.57 | bwd_allreduce: 46.38 | step: 19.58

 18%|█▊        | 7606/41250 [18:22:20<80:57:41,  8.66s/it]
                                                          
{'loss': 0.1435, 'grad_norm': 1.563520073890686, 'learning_rate': 3.7551732231469855e-05, 'epoch': 1.84}

 18%|█▊        | 7606/41250 [18:22:20<80:57:41,  8.66s/it][2025-04-26 02:20:03,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:20:03,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.66 | bwd_microstep: 5779.74 | bwd_inner_microstep: 5652.58 | bwd_allreduce_microstep: 127.12 | step_microstep: 18.24
[2025-04-26 02:20:03,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.66 | bwd: 5779.75 | bwd_inner: 5652.58 | bwd_allreduce: 127.13 | step: 18.25

 18%|█▊        | 7607/41250 [18:22:28<81:01:58,  8.67s/it]
                                                          
{'loss': 0.287, 'grad_norm': 2.5027899742126465, 'learning_rate': 3.755097933499111e-05, 'epoch': 1.84}

 18%|█▊        | 7607/41250 [18:22:28<81:01:58,  8.67s/it][2025-04-26 02:20:12,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 02:20:12,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.34 | bwd_microstep: 5786.09 | bwd_inner_microstep: 5641.49 | bwd_allreduce_microstep: 144.54 | step_microstep: 19.39
[2025-04-26 02:20:12,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.34 | bwd: 5786.10 | bwd_inner: 5641.49 | bwd_allreduce: 144.56 | step: 19.39

 18%|█▊        | 7608/41250 [18:22:37<81:05:05,  8.68s/it]
                                                          
{'loss': 0.0925, 'grad_norm': 1.2791459560394287, 'learning_rate': 3.7550226330314026e-05, 'epoch': 1.84}

 18%|█▊        | 7608/41250 [18:22:37<81:05:05,  8.68s/it][2025-04-26 02:20:20,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:20:20,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.17 | bwd_microstep: 5708.05 | bwd_inner_microstep: 5695.31 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.69
[2025-04-26 02:20:20,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.17 | bwd: 5708.06 | bwd_inner: 5695.31 | bwd_allreduce: 12.71 | step: 18.70

 18%|█▊        | 7609/41250 [18:22:46<81:01:18,  8.67s/it]
                                                          
{'loss': 0.1007, 'grad_norm': 2.0111193656921387, 'learning_rate': 3.754947321744325e-05, 'epoch': 1.84}

 18%|█▊        | 7609/41250 [18:22:46<81:01:18,  8.67s/it][2025-04-26 02:20:29,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:20:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.17 | bwd_microstep: 5736.88 | bwd_inner_microstep: 5649.03 | bwd_allreduce_microstep: 87.80 | step_microstep: 18.77
[2025-04-26 02:20:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.17 | bwd: 5736.89 | bwd_inner: 5649.03 | bwd_allreduce: 87.82 | step: 18.78

 18%|█▊        | 7610/41250 [18:22:54<80:59:18,  8.67s/it]
                                                          
{'loss': 0.122, 'grad_norm': 1.58474600315094, 'learning_rate': 3.754871999638342e-05, 'epoch': 1.84}

 18%|█▊        | 7610/41250 [18:22:54<80:59:18,  8.67s/it][2025-04-26 02:20:38,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 02:20:38,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.91 | bwd_microstep: 5693.06 | bwd_inner_microstep: 5647.37 | bwd_allreduce_microstep: 45.64 | step_microstep: 18.47
[2025-04-26 02:20:38,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5693.07 | bwd_inner: 5647.37 | bwd_allreduce: 45.66 | step: 18.48

 18%|█▊        | 7611/41250 [18:23:03<80:48:39,  8.65s/it]
                                                          
{'loss': 0.0842, 'grad_norm': 1.2612029314041138, 'learning_rate': 3.754796666713919e-05, 'epoch': 1.85}

 18%|█▊        | 7611/41250 [18:23:03<80:48:39,  8.65s/it][2025-04-26 02:20:46,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-26 02:20:46,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.73 | bwd_microstep: 5742.47 | bwd_inner_microstep: 5638.21 | bwd_allreduce_microstep: 104.22 | step_microstep: 18.82
[2025-04-26 02:20:46,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.73 | bwd: 5742.49 | bwd_inner: 5638.21 | bwd_allreduce: 104.24 | step: 18.82

 18%|█▊        | 7612/41250 [18:23:12<80:51:07,  8.65s/it]
                                                          
{'loss': 0.2759, 'grad_norm': 1.4539010524749756, 'learning_rate': 3.7547213229715194e-05, 'epoch': 1.85}

 18%|█▊        | 7612/41250 [18:23:12<80:51:07,  8.65s/it][2025-04-26 02:20:55,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:20:55,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.02 | bwd_microstep: 5762.22 | bwd_inner_microstep: 5641.00 | bwd_allreduce_microstep: 121.18 | step_microstep: 18.62
[2025-04-26 02:20:55,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.02 | bwd: 5762.23 | bwd_inner: 5641.00 | bwd_allreduce: 121.19 | step: 18.62

 18%|█▊        | 7613/41250 [18:23:20<80:55:39,  8.66s/it]
                                                          
{'loss': 0.1239, 'grad_norm': 1.8454614877700806, 'learning_rate': 3.754645968411608e-05, 'epoch': 1.85}

 18%|█▊        | 7613/41250 [18:23:20<80:55:39,  8.66s/it][2025-04-26 02:21:04,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 02:21:04,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.74 | bwd_microstep: 5736.98 | bwd_inner_microstep: 5648.00 | bwd_allreduce_microstep: 88.92 | step_microstep: 18.75
[2025-04-26 02:21:04,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.74 | bwd: 5736.99 | bwd_inner: 5648.00 | bwd_allreduce: 88.94 | step: 18.75

 18%|█▊        | 7614/41250 [18:23:29<80:53:49,  8.66s/it]
                                                          
{'loss': 0.1087, 'grad_norm': 1.0840935707092285, 'learning_rate': 3.754570603034649e-05, 'epoch': 1.85}

 18%|█▊        | 7614/41250 [18:23:29<80:53:49,  8.66s/it][2025-04-26 02:21:12,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:21:12,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.59 | bwd_microstep: 5736.24 | bwd_inner_microstep: 5693.29 | bwd_allreduce_microstep: 42.91 | step_microstep: 18.77
[2025-04-26 02:21:12,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.59 | bwd: 5736.25 | bwd_inner: 5693.29 | bwd_allreduce: 42.92 | step: 18.77

 18%|█▊        | 7615/41250 [18:23:38<80:56:02,  8.66s/it]
                                                          
{'loss': 0.0745, 'grad_norm': 1.1489921808242798, 'learning_rate': 3.7544952268411076e-05, 'epoch': 1.85}

 18%|█▊        | 7615/41250 [18:23:38<80:56:02,  8.66s/it][2025-04-26 02:21:21,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:21:21,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.26 | bwd_microstep: 5780.23 | bwd_inner_microstep: 5650.21 | bwd_allreduce_microstep: 129.97 | step_microstep: 18.51
[2025-04-26 02:21:21,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.26 | bwd: 5780.24 | bwd_inner: 5650.21 | bwd_allreduce: 129.99 | step: 18.51

 18%|█▊        | 7616/41250 [18:23:46<81:01:07,  8.67s/it]
                                                          
{'loss': 0.3268, 'grad_norm': 3.1761162281036377, 'learning_rate': 3.754419839831448e-05, 'epoch': 1.85}

 18%|█▊        | 7616/41250 [18:23:46<81:01:07,  8.67s/it][2025-04-26 02:21:30,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:21:30,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5688.64 | bwd_inner_microstep: 5675.73 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.62
[2025-04-26 02:21:30,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5688.65 | bwd_inner: 5675.73 | bwd_allreduce: 12.88 | step: 18.62

 18%|█▊        | 7617/41250 [18:23:55<80:52:15,  8.66s/it]
                                                          
{'loss': 0.2272, 'grad_norm': 1.3058613538742065, 'learning_rate': 3.7543444420061356e-05, 'epoch': 1.85}

 18%|█▊        | 7617/41250 [18:23:55<80:52:15,  8.66s/it][2025-04-26 02:21:38,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:21:38,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.97 | bwd_microstep: 5784.33 | bwd_inner_microstep: 5771.56 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.74
[2025-04-26 02:21:38,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.97 | bwd: 5784.35 | bwd_inner: 5771.56 | bwd_allreduce: 12.74 | step: 18.74

 18%|█▊        | 7618/41250 [18:24:04<81:08:12,  8.68s/it]
                                                          
{'loss': 0.1255, 'grad_norm': 2.39825439453125, 'learning_rate': 3.754269033365634e-05, 'epoch': 1.85}

 18%|█▊        | 7618/41250 [18:24:04<81:08:12,  8.68s/it][2025-04-26 02:21:47,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:21:47,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.83 | bwd_microstep: 5689.37 | bwd_inner_microstep: 5676.66 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.55
[2025-04-26 02:21:47,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.83 | bwd: 5689.38 | bwd_inner: 5676.66 | bwd_allreduce: 12.68 | step: 18.56

 18%|█▊        | 7619/41250 [18:24:12<80:55:55,  8.66s/it]
                                                          
{'loss': 0.1318, 'grad_norm': 1.5689363479614258, 'learning_rate': 3.75419361391041e-05, 'epoch': 1.85}

 18%|█▊        | 7619/41250 [18:24:12<80:55:55,  8.66s/it][2025-04-26 02:21:56,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 02:21:56,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5734.50 | bwd_inner_microstep: 5694.57 | bwd_allreduce_microstep: 39.89 | step_microstep: 19.00
[2025-04-26 02:21:56,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5734.51 | bwd_inner: 5694.57 | bwd_allreduce: 39.91 | step: 19.00

 18%|█▊        | 7620/41250 [18:24:21<80:55:39,  8.66s/it]
                                                          
{'loss': 0.2556, 'grad_norm': 2.171955108642578, 'learning_rate': 3.7541181836409264e-05, 'epoch': 1.85}

 18%|█▊        | 7620/41250 [18:24:21<80:55:39,  8.66s/it][2025-04-26 02:22:04,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 02:22:04,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.31 | bwd_microstep: 5737.07 | bwd_inner_microstep: 5679.61 | bwd_allreduce_microstep: 57.41 | step_microstep: 19.13
[2025-04-26 02:22:04,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.31 | bwd: 5737.08 | bwd_inner: 5679.61 | bwd_allreduce: 57.43 | step: 19.13

 18%|█▊        | 7621/41250 [18:24:30<80:55:34,  8.66s/it]
                                                          
{'loss': 0.1978, 'grad_norm': 4.497612476348877, 'learning_rate': 3.7540427425576496e-05, 'epoch': 1.85}

 18%|█▊        | 7621/41250 [18:24:30<80:55:34,  8.66s/it][2025-04-26 02:22:13,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 02:22:13,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5679.95 | bwd_inner_microstep: 5647.70 | bwd_allreduce_microstep: 32.20 | step_microstep: 18.95
[2025-04-26 02:22:13,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5679.96 | bwd_inner: 5647.70 | bwd_allreduce: 32.22 | step: 18.95

 18%|█▊        | 7622/41250 [18:24:38<80:44:21,  8.64s/it]
                                                          
{'loss': 0.1214, 'grad_norm': 1.4717488288879395, 'learning_rate': 3.7539672906610445e-05, 'epoch': 1.85}

 18%|█▊        | 7622/41250 [18:24:38<80:44:21,  8.64s/it][2025-04-26 02:22:21,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 02:22:21,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.53 | bwd_microstep: 5671.91 | bwd_inner_microstep: 5659.22 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.86
[2025-04-26 02:22:21,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.53 | bwd: 5671.93 | bwd_inner: 5659.22 | bwd_allreduce: 12.66 | step: 18.87

 18%|█▊        | 7623/41250 [18:24:47<80:36:08,  8.63s/it]
                                                          
{'loss': 0.1352, 'grad_norm': 2.7683913707733154, 'learning_rate': 3.753891827951575e-05, 'epoch': 1.85}

 18%|█▊        | 7623/41250 [18:24:47<80:36:08,  8.63s/it][2025-04-26 02:22:30,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:22:30,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.30 | bwd_microstep: 5708.26 | bwd_inner_microstep: 5641.01 | bwd_allreduce_microstep: 67.20 | step_microstep: 18.98
[2025-04-26 02:22:30,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.30 | bwd: 5708.28 | bwd_inner: 5641.01 | bwd_allreduce: 67.22 | step: 18.98

 18%|█▊        | 7624/41250 [18:24:55<80:33:08,  8.62s/it]
                                                          
{'loss': 0.2531, 'grad_norm': 2.100295305252075, 'learning_rate': 3.753816354429709e-05, 'epoch': 1.85}

 18%|█▊        | 7624/41250 [18:24:55<80:33:08,  8.62s/it][2025-04-26 02:22:39,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:22:39,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.41 | bwd_microstep: 5726.01 | bwd_inner_microstep: 5705.25 | bwd_allreduce_microstep: 20.71 | step_microstep: 18.57
[2025-04-26 02:22:39,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.41 | bwd: 5726.02 | bwd_inner: 5705.25 | bwd_allreduce: 20.73 | step: 18.57

 18%|█▊        | 7625/41250 [18:25:04<80:39:48,  8.64s/it]
                                                          
{'loss': 0.0833, 'grad_norm': 0.753722071647644, 'learning_rate': 3.753740870095909e-05, 'epoch': 1.85}

 18%|█▊        | 7625/41250 [18:25:04<80:39:48,  8.64s/it][2025-04-26 02:22:47,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:22:47,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.24 | bwd_microstep: 5725.85 | bwd_inner_microstep: 5701.14 | bwd_allreduce_microstep: 24.65 | step_microstep: 18.99
[2025-04-26 02:22:47,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.24 | bwd: 5725.86 | bwd_inner: 5701.14 | bwd_allreduce: 24.67 | step: 19.00

 18%|█▊        | 7626/41250 [18:25:13<80:43:33,  8.64s/it]
                                                          
{'loss': 0.1402, 'grad_norm': 0.950484037399292, 'learning_rate': 3.7536653749506415e-05, 'epoch': 1.85}

 18%|█▊        | 7626/41250 [18:25:13<80:43:33,  8.64s/it][2025-04-26 02:22:56,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 02:22:56,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.29 | bwd_microstep: 5781.72 | bwd_inner_microstep: 5636.45 | bwd_allreduce_microstep: 145.23 | step_microstep: 18.93
[2025-04-26 02:22:56,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.29 | bwd: 5781.73 | bwd_inner: 5636.45 | bwd_allreduce: 145.24 | step: 18.93

 18%|█▊        | 7627/41250 [18:25:21<80:50:25,  8.66s/it]
                                                          
{'loss': 0.1869, 'grad_norm': 3.1996192932128906, 'learning_rate': 3.753589868994372e-05, 'epoch': 1.85}

 18%|█▊        | 7627/41250 [18:25:21<80:50:25,  8.66s/it][2025-04-26 02:23:05,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 02:23:05,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.96 | bwd_microstep: 5699.26 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 40.24 | step_microstep: 19.16
[2025-04-26 02:23:05,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.96 | bwd: 5699.27 | bwd_inner: 5658.97 | bwd_allreduce: 40.26 | step: 19.16

 18%|█▊        | 7628/41250 [18:25:30<80:42:34,  8.64s/it]
                                                          
{'loss': 0.058, 'grad_norm': 0.9418116211891174, 'learning_rate': 3.753514352227567e-05, 'epoch': 1.85}

 18%|█▊        | 7628/41250 [18:25:30<80:42:34,  8.64s/it][2025-04-26 02:23:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:23:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.14 | bwd_microstep: 5764.06 | bwd_inner_microstep: 5639.62 | bwd_allreduce_microstep: 124.39 | step_microstep: 18.62
[2025-04-26 02:23:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.14 | bwd: 5764.07 | bwd_inner: 5639.62 | bwd_allreduce: 124.41 | step: 18.62

 18%|█▊        | 7629/41250 [18:25:39<80:47:18,  8.65s/it]
                                                          
{'loss': 0.1719, 'grad_norm': 1.4452967643737793, 'learning_rate': 3.7534388246506894e-05, 'epoch': 1.85}

 18%|█▊        | 7629/41250 [18:25:39<80:47:18,  8.65s/it][2025-04-26 02:23:22,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:23:22,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.63 | bwd_microstep: 5715.91 | bwd_inner_microstep: 5646.50 | bwd_allreduce_microstep: 69.37 | step_microstep: 18.64
[2025-04-26 02:23:22,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.63 | bwd: 5715.92 | bwd_inner: 5646.50 | bwd_allreduce: 69.38 | step: 18.64

 18%|█▊        | 7630/41250 [18:25:47<80:42:32,  8.64s/it]
                                                          
{'loss': 0.0904, 'grad_norm': 1.7073543071746826, 'learning_rate': 3.7533632862642074e-05, 'epoch': 1.85}

 18%|█▊        | 7630/41250 [18:25:47<80:42:32,  8.64s/it][2025-04-26 02:23:31,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:23:31,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5772.12 | bwd_inner_microstep: 5698.56 | bwd_allreduce_microstep: 73.52 | step_microstep: 18.53
[2025-04-26 02:23:31,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5772.14 | bwd_inner: 5698.56 | bwd_allreduce: 73.53 | step: 18.53

 18%|█▊        | 7631/41250 [18:25:56<80:52:34,  8.66s/it]
                                                          
{'loss': 0.0628, 'grad_norm': 0.6892262697219849, 'learning_rate': 3.753287737068585e-05, 'epoch': 1.85}

 18%|█▊        | 7631/41250 [18:25:56<80:52:34,  8.66s/it][2025-04-26 02:23:39,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:23:39,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5702.86 | bwd_inner_microstep: 5690.11 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.86
[2025-04-26 02:23:39,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.07 | bwd: 5702.88 | bwd_inner: 5690.11 | bwd_allreduce: 12.72 | step: 18.87

 19%|█▊        | 7632/41250 [18:26:05<80:47:47,  8.65s/it]
                                                          
{'loss': 0.0259, 'grad_norm': 0.5222805738449097, 'learning_rate': 3.7532121770642885e-05, 'epoch': 1.85}

 19%|█▊        | 7632/41250 [18:26:05<80:47:47,  8.65s/it][2025-04-26 02:23:48,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:23:48,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.34 | bwd_microstep: 5700.71 | bwd_inner_microstep: 5687.85 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.79
[2025-04-26 02:23:48,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.34 | bwd: 5700.72 | bwd_inner: 5687.85 | bwd_allreduce: 12.83 | step: 18.80

 19%|█▊        | 7633/41250 [18:26:13<80:43:49,  8.65s/it]
                                                          
{'loss': 0.1322, 'grad_norm': 1.1384373903274536, 'learning_rate': 3.753136606251784e-05, 'epoch': 1.85}

 19%|█▊        | 7633/41250 [18:26:13<80:43:49,  8.65s/it][2025-04-26 02:23:57,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:23:57,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5891.10 | bwd_inner_microstep: 5653.51 | bwd_allreduce_microstep: 237.55 | step_microstep: 18.41
[2025-04-26 02:23:57,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5891.12 | bwd_inner: 5653.51 | bwd_allreduce: 237.57 | step: 18.41

 19%|█▊        | 7634/41250 [18:26:22<81:09:23,  8.69s/it]
                                                          
{'loss': 0.1727, 'grad_norm': 1.893848180770874, 'learning_rate': 3.753061024631537e-05, 'epoch': 1.85}

 19%|█▊        | 7634/41250 [18:26:22<81:09:23,  8.69s/it][2025-04-26 02:24:05,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:24:05,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5748.00 | bwd_inner_microstep: 5705.98 | bwd_allreduce_microstep: 41.96 | step_microstep: 18.58
[2025-04-26 02:24:05,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5748.01 | bwd_inner: 5705.98 | bwd_allreduce: 41.98 | step: 18.58

 19%|█▊        | 7635/41250 [18:26:31<81:07:24,  8.69s/it]
                                                          
{'loss': 0.1576, 'grad_norm': 1.3118184804916382, 'learning_rate': 3.752985432204013e-05, 'epoch': 1.85}

 19%|█▊        | 7635/41250 [18:26:31<81:07:24,  8.69s/it][2025-04-26 02:24:14,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 02:24:14,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.79 | bwd_microstep: 5741.05 | bwd_inner_microstep: 5699.80 | bwd_allreduce_microstep: 41.21 | step_microstep: 19.16
[2025-04-26 02:24:14,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.79 | bwd: 5741.06 | bwd_inner: 5699.79 | bwd_allreduce: 41.23 | step: 19.16

 19%|█▊        | 7636/41250 [18:26:39<81:05:14,  8.68s/it]
                                                          
{'loss': 0.1211, 'grad_norm': 2.1110076904296875, 'learning_rate': 3.7529098289696795e-05, 'epoch': 1.85}

 19%|█▊        | 7636/41250 [18:26:39<81:05:14,  8.68s/it][2025-04-26 02:24:23,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 02:24:23,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.36 | bwd_microstep: 5706.24 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.05
[2025-04-26 02:24:23,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.36 | bwd: 5706.26 | bwd_inner: 5693.52 | bwd_allreduce: 12.69 | step: 19.05

 19%|█▊        | 7637/41250 [18:26:48<80:56:44,  8.67s/it]
                                                          
{'loss': 0.1776, 'grad_norm': 1.0904046297073364, 'learning_rate': 3.752834214929002e-05, 'epoch': 1.85}

 19%|█▊        | 7637/41250 [18:26:48<80:56:44,  8.67s/it][2025-04-26 02:24:32,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:24:32,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.46 | bwd_microstep: 5887.05 | bwd_inner_microstep: 5700.00 | bwd_allreduce_microstep: 187.01 | step_microstep: 18.83
[2025-04-26 02:24:32,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.46 | bwd: 5887.06 | bwd_inner: 5700.00 | bwd_allreduce: 187.02 | step: 18.83

 19%|█▊        | 7638/41250 [18:26:57<81:22:38,  8.72s/it]
                                                          
{'loss': 0.3663, 'grad_norm': 1.5378687381744385, 'learning_rate': 3.7527585900824454e-05, 'epoch': 1.85}

 19%|█▊        | 7638/41250 [18:26:57<81:22:38,  8.72s/it][2025-04-26 02:24:40,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 02:24:40,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.73 | bwd_microstep: 5782.97 | bwd_inner_microstep: 5770.26 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.56
[2025-04-26 02:24:40,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.73 | bwd: 5782.98 | bwd_inner: 5770.26 | bwd_allreduce: 12.68 | step: 18.56

 19%|█▊        | 7639/41250 [18:27:06<81:28:14,  8.73s/it]
                                                          
{'loss': 0.4649, 'grad_norm': 4.887169361114502, 'learning_rate': 3.7526829544304775e-05, 'epoch': 1.85}

 19%|█▊        | 7639/41250 [18:27:06<81:28:14,  8.73s/it][2025-04-26 02:24:49,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:24:49,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5761.58 | bwd_inner_microstep: 5701.76 | bwd_allreduce_microstep: 59.78 | step_microstep: 18.84
[2025-04-26 02:24:49,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5761.59 | bwd_inner: 5701.76 | bwd_allreduce: 59.79 | step: 18.84

 19%|█▊        | 7640/41250 [18:27:14<81:22:01,  8.72s/it]
                                                          
{'loss': 0.1411, 'grad_norm': 1.0651686191558838, 'learning_rate': 3.7526073079735635e-05, 'epoch': 1.85}

 19%|█▊        | 7640/41250 [18:27:14<81:22:01,  8.72s/it][2025-04-26 02:24:58,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 02:24:58,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5726.29 | bwd_inner_microstep: 5650.48 | bwd_allreduce_microstep: 75.77 | step_microstep: 18.45
[2025-04-26 02:24:58,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5726.30 | bwd_inner: 5650.48 | bwd_allreduce: 75.79 | step: 18.46

 19%|█▊        | 7641/41250 [18:27:23<81:09:07,  8.69s/it]
                                                          
{'loss': 0.0804, 'grad_norm': 1.0299708843231201, 'learning_rate': 3.7525316507121704e-05, 'epoch': 1.85}

 19%|█▊        | 7641/41250 [18:27:23<81:09:07,  8.69s/it][2025-04-26 02:25:06,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.09
[2025-04-26 02:25:06,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.31 | bwd_microstep: 5763.30 | bwd_inner_microstep: 5698.84 | bwd_allreduce_microstep: 64.42 | step_microstep: 18.72
[2025-04-26 02:25:06,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5763.31 | bwd_inner: 5698.84 | bwd_allreduce: 64.43 | step: 18.72

 19%|█▊        | 7642/41250 [18:27:32<81:09:36,  8.69s/it]
                                                          
{'loss': 0.1698, 'grad_norm': 1.9403884410858154, 'learning_rate': 3.7524559826467635e-05, 'epoch': 1.85}

 19%|█▊        | 7642/41250 [18:27:32<81:09:36,  8.69s/it][2025-04-26 02:25:15,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:25:15,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.60 | bwd_microstep: 5708.24 | bwd_inner_microstep: 5694.11 | bwd_allreduce_microstep: 14.09 | step_microstep: 18.81
[2025-04-26 02:25:15,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.60 | bwd: 5708.25 | bwd_inner: 5694.11 | bwd_allreduce: 14.10 | step: 18.81

 19%|█▊        | 7643/41250 [18:27:40<80:59:24,  8.68s/it]
                                                          
{'loss': 0.0829, 'grad_norm': 1.7872580289840698, 'learning_rate': 3.752380303777812e-05, 'epoch': 1.85}

 19%|█▊        | 7643/41250 [18:27:40<80:59:24,  8.68s/it][2025-04-26 02:25:24,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 02:25:24,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.49 | bwd_microstep: 5756.62 | bwd_inner_microstep: 5681.52 | bwd_allreduce_microstep: 75.04 | step_microstep: 18.69
[2025-04-26 02:25:24,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.49 | bwd: 5756.63 | bwd_inner: 5681.52 | bwd_allreduce: 75.06 | step: 18.69

 19%|█▊        | 7644/41250 [18:27:49<81:00:58,  8.68s/it]
                                                          
{'loss': 0.2119, 'grad_norm': 1.5244262218475342, 'learning_rate': 3.7523046141057785e-05, 'epoch': 1.85}

 19%|█▊        | 7644/41250 [18:27:49<81:00:58,  8.68s/it][2025-04-26 02:25:32,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-26 02:25:32,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5808.36 | bwd_inner_microstep: 5653.98 | bwd_allreduce_microstep: 154.33 | step_microstep: 19.31
[2025-04-26 02:25:32,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5808.38 | bwd_inner: 5653.98 | bwd_allreduce: 154.36 | step: 19.31

 19%|█▊        | 7645/41250 [18:27:58<81:07:20,  8.69s/it]
                                                          
{'loss': 0.131, 'grad_norm': 0.8781213760375977, 'learning_rate': 3.752228913631133e-05, 'epoch': 1.85}

 19%|█▊        | 7645/41250 [18:27:58<81:07:20,  8.69s/it][2025-04-26 02:25:41,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-26 02:25:41,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.06 | bwd_microstep: 5700.46 | bwd_inner_microstep: 5686.07 | bwd_allreduce_microstep: 14.35 | step_microstep: 18.95
[2025-04-26 02:25:41,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.06 | bwd: 5700.47 | bwd_inner: 5686.07 | bwd_allreduce: 14.36 | step: 18.96

 19%|█▊        | 7646/41250 [18:28:06<80:57:11,  8.67s/it]
                                                          
{'loss': 0.1452, 'grad_norm': 1.959054708480835, 'learning_rate': 3.752153202354341e-05, 'epoch': 1.85}

 19%|█▊        | 7646/41250 [18:28:06<80:57:11,  8.67s/it][2025-04-26 02:25:50,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 02:25:50,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.10 | bwd_microstep: 5716.11 | bwd_inner_microstep: 5703.19 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.90
[2025-04-26 02:25:50,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.10 | bwd: 5716.12 | bwd_inner: 5703.19 | bwd_allreduce: 12.89 | step: 18.90

 19%|█▊        | 7647/41250 [18:28:15<80:53:32,  8.67s/it]
                                                          
{'loss': 0.1216, 'grad_norm': 1.2207261323928833, 'learning_rate': 3.752077480275869e-05, 'epoch': 1.85}

 19%|█▊        | 7647/41250 [18:28:15<80:53:32,  8.67s/it][2025-04-26 02:25:58,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:25:58,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.65 | bwd_microstep: 5751.71 | bwd_inner_microstep: 5687.02 | bwd_allreduce_microstep: 64.64 | step_microstep: 18.85
[2025-04-26 02:25:58,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.65 | bwd: 5751.72 | bwd_inner: 5687.02 | bwd_allreduce: 64.66 | step: 18.85

 19%|█▊        | 7648/41250 [18:28:24<80:55:44,  8.67s/it]
                                                          
{'loss': 0.0617, 'grad_norm': 0.7868779301643372, 'learning_rate': 3.752001747396183e-05, 'epoch': 1.85}

 19%|█▊        | 7648/41250 [18:28:24<80:55:44,  8.67s/it][2025-04-26 02:26:07,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:26:07,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.75 | bwd_microstep: 5771.38 | bwd_inner_microstep: 5636.75 | bwd_allreduce_microstep: 134.58 | step_microstep: 18.85
[2025-04-26 02:26:07,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.75 | bwd: 5771.39 | bwd_inner: 5636.75 | bwd_allreduce: 134.60 | step: 18.85

 19%|█▊        | 7649/41250 [18:28:32<80:57:04,  8.67s/it]
                                                          
{'loss': 0.1133, 'grad_norm': 3.545379400253296, 'learning_rate': 3.7519260037157515e-05, 'epoch': 1.85}

 19%|█▊        | 7649/41250 [18:28:32<80:57:04,  8.67s/it][2025-04-26 02:26:16,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:26:16,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.79 | bwd_microstep: 5740.33 | bwd_inner_microstep: 5696.51 | bwd_allreduce_microstep: 43.77 | step_microstep: 18.61
[2025-04-26 02:26:16,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.79 | bwd: 5740.34 | bwd_inner: 5696.51 | bwd_allreduce: 43.79 | step: 18.61

 19%|█▊        | 7650/41250 [18:28:41<80:56:19,  8.67s/it]
                                                          
{'loss': 0.122, 'grad_norm': 1.6988922357559204, 'learning_rate': 3.751850249235041e-05, 'epoch': 1.85}

 19%|█▊        | 7650/41250 [18:28:41<80:56:19,  8.67s/it][2025-04-26 02:26:25,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:26:25,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.12 | bwd_microstep: 5869.10 | bwd_inner_microstep: 5703.00 | bwd_allreduce_microstep: 166.06 | step_microstep: 18.93
[2025-04-26 02:26:25,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.12 | bwd: 5869.11 | bwd_inner: 5703.00 | bwd_allreduce: 166.07 | step: 18.93

 19%|█▊        | 7651/41250 [18:28:50<81:18:42,  8.71s/it]
                                                          
{'loss': 0.0897, 'grad_norm': 1.3728922605514526, 'learning_rate': 3.7517744839545185e-05, 'epoch': 1.85}

 19%|█▊        | 7651/41250 [18:28:50<81:18:42,  8.71s/it][2025-04-26 02:26:33,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:26:33,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.73 | bwd_microstep: 5699.61 | bwd_inner_microstep: 5643.53 | bwd_allreduce_microstep: 56.03 | step_microstep: 18.55
[2025-04-26 02:26:33,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.73 | bwd: 5699.62 | bwd_inner: 5643.53 | bwd_allreduce: 56.05 | step: 18.55

 19%|█▊        | 7652/41250 [18:28:58<81:02:06,  8.68s/it]
                                                          
{'loss': 0.206, 'grad_norm': 1.9299432039260864, 'learning_rate': 3.75169870787465e-05, 'epoch': 1.86}

 19%|█▊        | 7652/41250 [18:28:58<81:02:06,  8.68s/it][2025-04-26 02:26:42,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:26:42,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.58 | bwd_microstep: 5713.38 | bwd_inner_microstep: 5648.70 | bwd_allreduce_microstep: 64.63 | step_microstep: 18.35
[2025-04-26 02:26:42,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.58 | bwd: 5713.39 | bwd_inner: 5648.70 | bwd_allreduce: 64.65 | step: 18.36

 19%|█▊        | 7653/41250 [18:29:07<80:50:43,  8.66s/it]
                                                          
{'loss': 0.2037, 'grad_norm': 2.5926334857940674, 'learning_rate': 3.751622920995904e-05, 'epoch': 1.86}

 19%|█▊        | 7653/41250 [18:29:07<80:50:43,  8.66s/it][2025-04-26 02:26:50,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 02:26:50,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.12 | bwd_microstep: 5782.30 | bwd_inner_microstep: 5647.98 | bwd_allreduce_microstep: 134.28 | step_microstep: 18.60
[2025-04-26 02:26:50,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.12 | bwd: 5782.32 | bwd_inner: 5647.98 | bwd_allreduce: 134.29 | step: 18.60

 19%|█▊        | 7654/41250 [18:29:16<80:56:23,  8.67s/it]
                                                          
{'loss': 0.0691, 'grad_norm': 2.1579720973968506, 'learning_rate': 3.751547123318748e-05, 'epoch': 1.86}

 19%|█▊        | 7654/41250 [18:29:16<80:56:23,  8.67s/it][2025-04-26 02:26:59,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:26:59,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.81 | bwd_microstep: 6040.77 | bwd_inner_microstep: 5643.32 | bwd_allreduce_microstep: 397.40 | step_microstep: 18.72
[2025-04-26 02:26:59,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.81 | bwd: 6040.78 | bwd_inner: 5643.32 | bwd_allreduce: 397.42 | step: 18.73

 19%|█▊        | 7655/41250 [18:29:25<81:43:05,  8.76s/it]
                                                          
{'loss': 0.1316, 'grad_norm': 0.8739269971847534, 'learning_rate': 3.751471314843647e-05, 'epoch': 1.86}

 19%|█▊        | 7655/41250 [18:29:25<81:43:05,  8.76s/it][2025-04-26 02:27:08,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 02:27:08,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5723.41 | bwd_inner_microstep: 5686.71 | bwd_allreduce_microstep: 36.65 | step_microstep: 19.55
[2025-04-26 02:27:08,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5723.42 | bwd_inner: 5686.71 | bwd_allreduce: 36.67 | step: 19.55

 19%|█▊        | 7656/41250 [18:29:33<81:25:51,  8.73s/it]
                                                          
{'loss': 0.0738, 'grad_norm': 0.8915503025054932, 'learning_rate': 3.7513954955710704e-05, 'epoch': 1.86}

 19%|█▊        | 7656/41250 [18:29:33<81:25:51,  8.73s/it][2025-04-26 02:27:17,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:27:17,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.62 | bwd_microstep: 5749.96 | bwd_inner_microstep: 5686.31 | bwd_allreduce_microstep: 63.61 | step_microstep: 18.40
[2025-04-26 02:27:17,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.62 | bwd: 5749.98 | bwd_inner: 5686.31 | bwd_allreduce: 63.63 | step: 18.40

 19%|█▊        | 7657/41250 [18:29:42<81:16:50,  8.71s/it]
                                                          
{'loss': 0.1046, 'grad_norm': 1.8434231281280518, 'learning_rate': 3.7513196655014855e-05, 'epoch': 1.86}

 19%|█▊        | 7657/41250 [18:29:42<81:16:50,  8.71s/it][2025-04-26 02:27:25,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:27:25,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.12 | bwd_microstep: 5791.21 | bwd_inner_microstep: 5626.84 | bwd_allreduce_microstep: 164.32 | step_microstep: 18.66
[2025-04-26 02:27:25,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.12 | bwd: 5791.22 | bwd_inner: 5626.84 | bwd_allreduce: 164.34 | step: 18.66

 19%|█▊        | 7658/41250 [18:29:51<81:13:22,  8.70s/it]
                                                          
{'loss': 0.0387, 'grad_norm': 0.42392656207084656, 'learning_rate': 3.751243824635359e-05, 'epoch': 1.86}

 19%|█▊        | 7658/41250 [18:29:51<81:13:22,  8.70s/it][2025-04-26 02:27:34,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.07 | optimizer_step: 0.98
[2025-04-26 02:27:34,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.68 | bwd_microstep: 5753.70 | bwd_inner_microstep: 5623.90 | bwd_allreduce_microstep: 129.75 | step_microstep: 19.38
[2025-04-26 02:27:34,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.68 | bwd: 5753.72 | bwd_inner: 5623.90 | bwd_allreduce: 129.77 | step: 19.38

 19%|█▊        | 7659/41250 [18:29:59<81:05:05,  8.69s/it]
                                                          
{'loss': 0.1022, 'grad_norm': 1.484567642211914, 'learning_rate': 3.751167972973159e-05, 'epoch': 1.86}

 19%|█▊        | 7659/41250 [18:29:59<81:05:05,  8.69s/it][2025-04-26 02:27:43,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:27:43,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.62 | bwd_microstep: 5685.79 | bwd_inner_microstep: 5673.07 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.61
[2025-04-26 02:27:43,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.62 | bwd: 5685.80 | bwd_inner: 5673.07 | bwd_allreduce: 12.69 | step: 18.61

 19%|█▊        | 7660/41250 [18:30:08<80:50:55,  8.66s/it]
                                                          
{'loss': 0.0247, 'grad_norm': 0.455024778842926, 'learning_rate': 3.751092110515353e-05, 'epoch': 1.86}

 19%|█▊        | 7660/41250 [18:30:08<80:50:55,  8.66s/it][2025-04-26 02:27:51,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 02:27:51,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.60 | bwd_microstep: 5762.79 | bwd_inner_microstep: 5642.21 | bwd_allreduce_microstep: 120.54 | step_microstep: 18.72
[2025-04-26 02:27:51,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.60 | bwd: 5762.81 | bwd_inner: 5642.21 | bwd_allreduce: 120.55 | step: 18.73

 19%|█▊        | 7661/41250 [18:30:17<80:51:27,  8.67s/it]
                                                          
{'loss': 0.2828, 'grad_norm': 2.263742685317993, 'learning_rate': 3.751016237262407e-05, 'epoch': 1.86}

 19%|█▊        | 7661/41250 [18:30:17<80:51:27,  8.67s/it][2025-04-26 02:28:00,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:28:00,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.81 | bwd_microstep: 5708.63 | bwd_inner_microstep: 5633.98 | bwd_allreduce_microstep: 74.61 | step_microstep: 18.10
[2025-04-26 02:28:00,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.81 | bwd: 5708.65 | bwd_inner: 5633.98 | bwd_allreduce: 74.63 | step: 18.10

 19%|█▊        | 7662/41250 [18:30:25<80:42:46,  8.65s/it]
                                                          
{'loss': 0.1655, 'grad_norm': 2.7516086101531982, 'learning_rate': 3.750940353214792e-05, 'epoch': 1.86}

 19%|█▊        | 7662/41250 [18:30:25<80:42:46,  8.65s/it][2025-04-26 02:28:09,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:28:09,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.71 | bwd_microstep: 5770.39 | bwd_inner_microstep: 5633.63 | bwd_allreduce_microstep: 136.72 | step_microstep: 18.07
[2025-04-26 02:28:09,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.71 | bwd: 5770.40 | bwd_inner: 5633.63 | bwd_allreduce: 136.73 | step: 18.07

 19%|█▊        | 7663/41250 [18:30:34<80:46:35,  8.66s/it]
                                                          
{'loss': 0.0316, 'grad_norm': 0.6385459899902344, 'learning_rate': 3.7508644583729736e-05, 'epoch': 1.86}

 19%|█▊        | 7663/41250 [18:30:34<80:46:35,  8.66s/it][2025-04-26 02:28:17,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.31 | optimizer_step: 1.06
[2025-04-26 02:28:17,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.94 | bwd_microstep: 5762.81 | bwd_inner_microstep: 5638.11 | bwd_allreduce_microstep: 124.64 | step_microstep: 20.19
[2025-04-26 02:28:17,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.94 | bwd: 5762.83 | bwd_inner: 5638.11 | bwd_allreduce: 124.68 | step: 20.19

 19%|█▊        | 7664/41250 [18:30:43<80:47:43,  8.66s/it]
                                                          
{'loss': 0.11, 'grad_norm': 1.639635682106018, 'learning_rate': 3.7507885527374205e-05, 'epoch': 1.86}

 19%|█▊        | 7664/41250 [18:30:43<80:47:43,  8.66s/it][2025-04-26 02:28:26,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 02:28:26,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5746.33 | bwd_inner_microstep: 5681.41 | bwd_allreduce_microstep: 64.87 | step_microstep: 18.65
[2025-04-26 02:28:26,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5746.34 | bwd_inner: 5681.41 | bwd_allreduce: 64.89 | step: 18.65

 19%|█▊        | 7665/41250 [18:30:51<80:50:18,  8.67s/it]
                                                          
{'loss': 0.2756, 'grad_norm': 4.591912746429443, 'learning_rate': 3.7507126363086e-05, 'epoch': 1.86}

 19%|█▊        | 7665/41250 [18:30:51<80:50:18,  8.67s/it][2025-04-26 02:28:35,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 02:28:35,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.83 | bwd_microstep: 5783.86 | bwd_inner_microstep: 5638.10 | bwd_allreduce_microstep: 145.72 | step_microstep: 18.93
[2025-04-26 02:28:35,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.83 | bwd: 5783.88 | bwd_inner: 5638.10 | bwd_allreduce: 145.73 | step: 18.94

 19%|█▊        | 7666/41250 [18:31:00<80:53:27,  8.67s/it]
                                                          
{'loss': 0.0526, 'grad_norm': 0.7211490273475647, 'learning_rate': 3.75063670908698e-05, 'epoch': 1.86}

 19%|█▊        | 7666/41250 [18:31:00<80:53:27,  8.67s/it][2025-04-26 02:28:43,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 02:28:43,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.40 | bwd_microstep: 5698.69 | bwd_inner_microstep: 5685.71 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.08
[2025-04-26 02:28:43,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.40 | bwd: 5698.70 | bwd_inner: 5685.71 | bwd_allreduce: 12.94 | step: 19.08

 19%|█▊        | 7667/41250 [18:31:09<80:46:26,  8.66s/it]
                                                          
{'loss': 0.0893, 'grad_norm': 1.3749464750289917, 'learning_rate': 3.75056077107303e-05, 'epoch': 1.86}

 19%|█▊        | 7667/41250 [18:31:09<80:46:26,  8.66s/it][2025-04-26 02:28:52,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:28:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.73 | bwd_microstep: 5787.66 | bwd_inner_microstep: 5774.81 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.85
[2025-04-26 02:28:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.73 | bwd: 5787.67 | bwd_inner: 5774.81 | bwd_allreduce: 12.82 | step: 18.86

 19%|█▊        | 7668/41250 [18:31:17<81:01:12,  8.69s/it]
                                                          
{'loss': 0.3992, 'grad_norm': 2.4727728366851807, 'learning_rate': 3.750484822267217e-05, 'epoch': 1.86}

 19%|█▊        | 7668/41250 [18:31:17<81:01:12,  8.69s/it][2025-04-26 02:29:01,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 02:29:01,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5698.41 | bwd_inner_microstep: 5685.75 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.71
[2025-04-26 02:29:01,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5698.42 | bwd_inner: 5685.75 | bwd_allreduce: 12.63 | step: 18.71

 19%|█▊        | 7669/41250 [18:31:26<80:51:41,  8.67s/it]
                                                          
{'loss': 0.1805, 'grad_norm': 3.095423460006714, 'learning_rate': 3.750408862670009e-05, 'epoch': 1.86}

 19%|█▊        | 7669/41250 [18:31:26<80:51:41,  8.67s/it][2025-04-26 02:29:09,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:29:09,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5730.41 | bwd_inner_microstep: 5695.34 | bwd_allreduce_microstep: 35.03 | step_microstep: 18.72
[2025-04-26 02:29:09,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5730.43 | bwd_inner: 5695.34 | bwd_allreduce: 35.04 | step: 18.73

 19%|█▊        | 7670/41250 [18:31:35<80:51:18,  8.67s/it]
                                                          
{'loss': 0.0541, 'grad_norm': 0.7256878614425659, 'learning_rate': 3.750332892281875e-05, 'epoch': 1.86}

 19%|█▊        | 7670/41250 [18:31:35<80:51:18,  8.67s/it][2025-04-26 02:29:18,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:29:18,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5752.40 | bwd_inner_microstep: 5697.56 | bwd_allreduce_microstep: 54.80 | step_microstep: 18.94
[2025-04-26 02:29:18,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5752.42 | bwd_inner: 5697.56 | bwd_allreduce: 54.81 | step: 18.94

 19%|█▊        | 7671/41250 [18:31:43<80:54:07,  8.67s/it]
                                                          
{'loss': 0.2093, 'grad_norm': 2.390334129333496, 'learning_rate': 3.7502569111032824e-05, 'epoch': 1.86}

 19%|█▊        | 7671/41250 [18:31:43<80:54:07,  8.67s/it][2025-04-26 02:29:27,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:29:27,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.19 | bwd_microstep: 5783.20 | bwd_inner_microstep: 5643.44 | bwd_allreduce_microstep: 139.71 | step_microstep: 18.71
[2025-04-26 02:29:27,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.19 | bwd: 5783.22 | bwd_inner: 5643.44 | bwd_allreduce: 139.73 | step: 18.71

 19%|█▊        | 7672/41250 [18:31:52<80:55:51,  8.68s/it]
                                                          
{'loss': 0.0258, 'grad_norm': 0.33774808049201965, 'learning_rate': 3.7501809191347e-05, 'epoch': 1.86}

 19%|█▊        | 7672/41250 [18:31:52<80:55:51,  8.68s/it][2025-04-26 02:29:35,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 02:29:35,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.83 | bwd_microstep: 5710.07 | bwd_inner_microstep: 5697.23 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.74
[2025-04-26 02:29:35,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.83 | bwd: 5710.08 | bwd_inner: 5697.23 | bwd_allreduce: 12.82 | step: 18.75

 19%|█▊        | 7673/41250 [18:32:01<80:48:53,  8.66s/it]
                                                          
{'loss': 0.2221, 'grad_norm': 3.8780195713043213, 'learning_rate': 3.750104916376597e-05, 'epoch': 1.86}

 19%|█▊        | 7673/41250 [18:32:01<80:48:53,  8.66s/it][2025-04-26 02:29:44,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 02:29:44,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5781.75 | bwd_inner_microstep: 5648.52 | bwd_allreduce_microstep: 133.19 | step_microstep: 18.64
[2025-04-26 02:29:44,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.48 | bwd: 5781.76 | bwd_inner: 5648.51 | bwd_allreduce: 133.21 | step: 18.65

 19%|█▊        | 7674/41250 [18:32:09<80:52:36,  8.67s/it]
                                                          
{'loss': 0.3575, 'grad_norm': 2.562004804611206, 'learning_rate': 3.750028902829441e-05, 'epoch': 1.86}

 19%|█▊        | 7674/41250 [18:32:09<80:52:36,  8.67s/it][2025-04-26 02:29:53,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 02:29:53,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.69 | bwd_microstep: 5759.93 | bwd_inner_microstep: 5637.80 | bwd_allreduce_microstep: 122.08 | step_microstep: 18.98
[2025-04-26 02:29:53,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.69 | bwd: 5759.94 | bwd_inner: 5637.80 | bwd_allreduce: 122.10 | step: 18.99

 19%|█▊        | 7675/41250 [18:32:18<80:52:37,  8.67s/it]
                                                          
{'loss': 0.0333, 'grad_norm': 0.7206865549087524, 'learning_rate': 3.7499528784937015e-05, 'epoch': 1.86}

 19%|█▊        | 7675/41250 [18:32:18<80:52:37,  8.67s/it][2025-04-26 02:30:02,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:30:02,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.87 | bwd_microstep: 5912.37 | bwd_inner_microstep: 5655.60 | bwd_allreduce_microstep: 256.72 | step_microstep: 18.62
[2025-04-26 02:30:02,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.87 | bwd: 5912.38 | bwd_inner: 5655.60 | bwd_allreduce: 256.74 | step: 18.63

 19%|█▊        | 7676/41250 [18:32:27<81:19:54,  8.72s/it]
                                                          
{'loss': 0.2032, 'grad_norm': 2.556124687194824, 'learning_rate': 3.7498768433698474e-05, 'epoch': 1.86}

 19%|█▊        | 7676/41250 [18:32:27<81:19:54,  8.72s/it][2025-04-26 02:30:10,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:30:10,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5788.12 | bwd_inner_microstep: 5660.45 | bwd_allreduce_microstep: 127.62 | step_microstep: 18.48
[2025-04-26 02:30:10,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5788.13 | bwd_inner: 5660.45 | bwd_allreduce: 127.64 | step: 18.49

 19%|█▊        | 7677/41250 [18:32:36<81:15:58,  8.71s/it]
                                                          
{'loss': 0.0435, 'grad_norm': 0.7778078317642212, 'learning_rate': 3.7498007974583456e-05, 'epoch': 1.86}

 19%|█▊        | 7677/41250 [18:32:36<81:15:58,  8.71s/it][2025-04-26 02:30:19,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:30:19,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5689.59 | bwd_inner_microstep: 5656.70 | bwd_allreduce_microstep: 32.85 | step_microstep: 18.61
[2025-04-26 02:30:19,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.74 | bwd: 5689.60 | bwd_inner: 5656.70 | bwd_allreduce: 32.86 | step: 18.62

 19%|█▊        | 7678/41250 [18:32:44<80:57:24,  8.68s/it]
                                                          
{'loss': 0.0392, 'grad_norm': 0.602817952632904, 'learning_rate': 3.7497247407596665e-05, 'epoch': 1.86}

 19%|█▊        | 7678/41250 [18:32:44<80:57:24,  8.68s/it][2025-04-26 02:30:27,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 02:30:27,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.52 | bwd_microstep: 5713.53 | bwd_inner_microstep: 5650.92 | bwd_allreduce_microstep: 62.57 | step_microstep: 18.99
[2025-04-26 02:30:27,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.52 | bwd: 5713.54 | bwd_inner: 5650.92 | bwd_allreduce: 62.58 | step: 19.00

 19%|█▊        | 7679/41250 [18:32:53<80:48:21,  8.67s/it]
                                                          
{'loss': 0.421, 'grad_norm': 3.8015167713165283, 'learning_rate': 3.749648673274278e-05, 'epoch': 1.86}

 19%|█▊        | 7679/41250 [18:32:53<80:48:21,  8.67s/it][2025-04-26 02:30:36,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:30:36,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.66 | bwd_microstep: 5720.94 | bwd_inner_microstep: 5657.08 | bwd_allreduce_microstep: 63.82 | step_microstep: 18.78
[2025-04-26 02:30:36,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.66 | bwd: 5720.96 | bwd_inner: 5657.08 | bwd_allreduce: 63.84 | step: 18.78

 19%|█▊        | 7680/41250 [18:33:01<80:43:21,  8.66s/it]
                                                          
{'loss': 0.0656, 'grad_norm': 1.1059954166412354, 'learning_rate': 3.7495725950026505e-05, 'epoch': 1.86}

 19%|█▊        | 7680/41250 [18:33:01<80:43:21,  8.66s/it][2025-04-26 02:30:45,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 02:30:45,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.69 | bwd_microstep: 5728.74 | bwd_inner_microstep: 5650.34 | bwd_allreduce_microstep: 78.35 | step_microstep: 18.26
[2025-04-26 02:30:45,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.69 | bwd: 5728.75 | bwd_inner: 5650.34 | bwd_allreduce: 78.37 | step: 18.26

 19%|█▊        | 7681/41250 [18:33:10<80:39:18,  8.65s/it]
                                                          
{'loss': 0.1871, 'grad_norm': 3.0474889278411865, 'learning_rate': 3.7494965059452514e-05, 'epoch': 1.86}

 19%|█▊        | 7681/41250 [18:33:10<80:39:18,  8.65s/it][2025-04-26 02:30:53,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:30:53,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.44 | bwd_microstep: 5709.85 | bwd_inner_microstep: 5689.94 | bwd_allreduce_microstep: 19.88 | step_microstep: 18.62
[2025-04-26 02:30:53,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.44 | bwd: 5709.87 | bwd_inner: 5689.94 | bwd_allreduce: 19.89 | step: 18.62

 19%|█▊        | 7682/41250 [18:33:19<80:37:43,  8.65s/it]
                                                          
{'loss': 0.0652, 'grad_norm': 1.4344886541366577, 'learning_rate': 3.749420406102551e-05, 'epoch': 1.86}

 19%|█▊        | 7682/41250 [18:33:19<80:37:43,  8.65s/it][2025-04-26 02:31:02,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:31:02,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.68 | bwd_microstep: 5700.95 | bwd_inner_microstep: 5688.03 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.71
[2025-04-26 02:31:02,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.68 | bwd: 5700.96 | bwd_inner: 5688.03 | bwd_allreduce: 12.88 | step: 18.71

 19%|█▊        | 7683/41250 [18:33:27<80:36:47,  8.65s/it]
                                                          
{'loss': 0.0669, 'grad_norm': 3.083219051361084, 'learning_rate': 3.749344295475017e-05, 'epoch': 1.86}

 19%|█▊        | 7683/41250 [18:33:27<80:36:47,  8.65s/it][2025-04-26 02:31:11,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:31:11,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.43 | bwd_microstep: 5715.04 | bwd_inner_microstep: 5644.63 | bwd_allreduce_microstep: 70.37 | step_microstep: 18.84
[2025-04-26 02:31:11,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.43 | bwd: 5715.05 | bwd_inner: 5644.62 | bwd_allreduce: 70.39 | step: 18.84

 19%|█▊        | 7684/41250 [18:33:36<80:34:45,  8.64s/it]
                                                          
{'loss': 0.1314, 'grad_norm': 2.912235736846924, 'learning_rate': 3.7492681740631205e-05, 'epoch': 1.86}

 19%|█▊        | 7684/41250 [18:33:36<80:34:45,  8.64s/it][2025-04-26 02:31:19,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:31:19,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.23 | bwd_microstep: 5721.82 | bwd_inner_microstep: 5650.36 | bwd_allreduce_microstep: 71.42 | step_microstep: 18.30
[2025-04-26 02:31:19,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.23 | bwd: 5721.83 | bwd_inner: 5650.36 | bwd_allreduce: 71.44 | step: 18.30

 19%|█▊        | 7685/41250 [18:33:45<80:34:40,  8.64s/it]
                                                          
{'loss': 0.3555, 'grad_norm': 2.9040586948394775, 'learning_rate': 3.749192041867329e-05, 'epoch': 1.86}

 19%|█▊        | 7685/41250 [18:33:45<80:34:40,  8.64s/it][2025-04-26 02:31:28,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:31:28,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.67 | bwd_microstep: 5779.90 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 119.92 | step_microstep: 18.76
[2025-04-26 02:31:28,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.67 | bwd: 5779.92 | bwd_inner: 5659.93 | bwd_allreduce: 119.94 | step: 18.76

 19%|█▊        | 7686/41250 [18:33:53<80:44:03,  8.66s/it]
                                                          
{'loss': 0.1843, 'grad_norm': 3.2544314861297607, 'learning_rate': 3.7491158988881126e-05, 'epoch': 1.86}

 19%|█▊        | 7686/41250 [18:33:53<80:44:03,  8.66s/it][2025-04-26 02:31:37,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 02:31:37,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.60 | bwd_microstep: 6045.28 | bwd_inner_microstep: 5645.03 | bwd_allreduce_microstep: 400.21 | step_microstep: 18.66
[2025-04-26 02:31:37,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 6045.29 | bwd_inner: 5645.03 | bwd_allreduce: 400.22 | step: 18.67

 19%|█▊        | 7687/41250 [18:34:02<81:35:49,  8.75s/it]
                                                          
{'loss': 0.0712, 'grad_norm': 1.4922674894332886, 'learning_rate': 3.749039745125941e-05, 'epoch': 1.86}

 19%|█▊        | 7687/41250 [18:34:02<81:35:49,  8.75s/it][2025-04-26 02:31:46,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.24 | optimizer_step: 0.94
[2025-04-26 02:31:46,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.38 | bwd_microstep: 5737.98 | bwd_inner_microstep: 5701.29 | bwd_allreduce_microstep: 36.63 | step_microstep: 19.59
[2025-04-26 02:31:46,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.38 | bwd: 5737.99 | bwd_inner: 5701.29 | bwd_allreduce: 36.66 | step: 19.59

 19%|█▊        | 7688/41250 [18:34:11<81:23:34,  8.73s/it]
                                                          
{'loss': 0.0661, 'grad_norm': 0.758581280708313, 'learning_rate': 3.748963580581284e-05, 'epoch': 1.86}

 19%|█▊        | 7688/41250 [18:34:11<81:23:34,  8.73s/it][2025-04-26 02:31:54,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:31:54,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.08 | bwd_microstep: 5749.14 | bwd_inner_microstep: 5706.66 | bwd_allreduce_microstep: 42.43 | step_microstep: 18.60
[2025-04-26 02:31:54,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.08 | bwd: 5749.15 | bwd_inner: 5706.66 | bwd_allreduce: 42.45 | step: 18.60

 19%|█▊        | 7689/41250 [18:34:20<81:17:06,  8.72s/it]
                                                          
{'loss': 0.2012, 'grad_norm': 2.6692521572113037, 'learning_rate': 3.74888740525461e-05, 'epoch': 1.86}

 19%|█▊        | 7689/41250 [18:34:20<81:17:06,  8.72s/it][2025-04-26 02:32:03,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.17 | optimizer_step: 0.92
[2025-04-26 02:32:03,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.93 | bwd_microstep: 5770.36 | bwd_inner_microstep: 5644.72 | bwd_allreduce_microstep: 125.59 | step_microstep: 18.83
[2025-04-26 02:32:03,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.93 | bwd: 5770.38 | bwd_inner: 5644.72 | bwd_allreduce: 125.61 | step: 18.83

 19%|█▊        | 7690/41250 [18:34:28<81:09:42,  8.71s/it]
                                                          
{'loss': 0.1616, 'grad_norm': 3.037764310836792, 'learning_rate': 3.748811219146388e-05, 'epoch': 1.86}

 19%|█▊        | 7690/41250 [18:34:28<81:09:42,  8.71s/it][2025-04-26 02:32:12,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.02 | optimizer_step: 1.18
[2025-04-26 02:32:12,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5758.28 | bwd_inner_microstep: 5685.71 | bwd_allreduce_microstep: 72.51 | step_microstep: 19.27
[2025-04-26 02:32:12,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5758.29 | bwd_inner: 5685.71 | bwd_allreduce: 72.54 | step: 19.27

 19%|█▊        | 7691/41250 [18:34:37<81:08:01,  8.70s/it]
                                                          
{'loss': 0.0552, 'grad_norm': 0.9870890378952026, 'learning_rate': 3.74873502225709e-05, 'epoch': 1.86}

 19%|█▊        | 7691/41250 [18:34:37<81:08:01,  8.70s/it][2025-04-26 02:32:20,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:32:20,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5679.35 | bwd_inner_microstep: 5652.81 | bwd_allreduce_microstep: 26.49 | step_microstep: 18.65
[2025-04-26 02:32:20,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5679.36 | bwd_inner: 5652.81 | bwd_allreduce: 26.51 | step: 18.66

 19%|█▊        | 7692/41250 [18:34:46<80:52:37,  8.68s/it]
                                                          
{'loss': 0.1078, 'grad_norm': 1.3682721853256226, 'learning_rate': 3.748658814587184e-05, 'epoch': 1.86}

 19%|█▊        | 7692/41250 [18:34:46<80:52:37,  8.68s/it][2025-04-26 02:32:29,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 02:32:29,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.11 | bwd_microstep: 5760.62 | bwd_inner_microstep: 5648.58 | bwd_allreduce_microstep: 111.99 | step_microstep: 19.14
[2025-04-26 02:32:29,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.11 | bwd: 5760.63 | bwd_inner: 5648.58 | bwd_allreduce: 112.01 | step: 19.14

 19%|█▊        | 7693/41250 [18:34:54<80:52:14,  8.68s/it]
                                                          
{'loss': 0.0302, 'grad_norm': 0.996963381767273, 'learning_rate': 3.748582596137142e-05, 'epoch': 1.86}

 19%|█▊        | 7693/41250 [18:34:54<80:52:14,  8.68s/it][2025-04-26 02:32:38,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-26 02:32:38,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.70 | bwd_microstep: 5749.61 | bwd_inner_microstep: 5685.30 | bwd_allreduce_microstep: 64.27 | step_microstep: 18.75
[2025-04-26 02:32:38,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.70 | bwd: 5749.63 | bwd_inner: 5685.30 | bwd_allreduce: 64.28 | step: 18.75

 19%|█▊        | 7694/41250 [18:35:03<80:54:41,  8.68s/it]
                                                          
{'loss': 0.2224, 'grad_norm': 1.9239269495010376, 'learning_rate': 3.748506366907431e-05, 'epoch': 1.87}

 19%|█▊        | 7694/41250 [18:35:03<80:54:41,  8.68s/it][2025-04-26 02:32:46,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 02:32:46,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.38 | bwd_microstep: 5706.21 | bwd_inner_microstep: 5654.20 | bwd_allreduce_microstep: 51.96 | step_microstep: 19.13
[2025-04-26 02:32:46,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.38 | bwd: 5706.22 | bwd_inner: 5654.20 | bwd_allreduce: 51.98 | step: 19.13

 19%|█▊        | 7695/41250 [18:35:12<80:44:59,  8.66s/it]
                                                          
{'loss': 0.046, 'grad_norm': 0.8932245969772339, 'learning_rate': 3.748430126898522e-05, 'epoch': 1.87}

 19%|█▊        | 7695/41250 [18:35:12<80:44:59,  8.66s/it][2025-04-26 02:32:55,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 02:32:55,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.42 | bwd_microstep: 5706.42 | bwd_inner_microstep: 5653.27 | bwd_allreduce_microstep: 53.10 | step_microstep: 18.99
[2025-04-26 02:32:55,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.42 | bwd: 5706.44 | bwd_inner: 5653.27 | bwd_allreduce: 53.13 | step: 19.00

 19%|█▊        | 7696/41250 [18:35:20<80:37:32,  8.65s/it]
                                                          
{'loss': 0.1958, 'grad_norm': 2.9292080402374268, 'learning_rate': 3.748353876110885e-05, 'epoch': 1.87}

 19%|█▊        | 7696/41250 [18:35:20<80:37:32,  8.65s/it][2025-04-26 02:33:04,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:33:04,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.13 | bwd_microstep: 5750.20 | bwd_inner_microstep: 5654.38 | bwd_allreduce_microstep: 95.77 | step_microstep: 18.83
[2025-04-26 02:33:04,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.13 | bwd: 5750.21 | bwd_inner: 5654.38 | bwd_allreduce: 95.79 | step: 18.84

 19%|█▊        | 7697/41250 [18:35:29<80:39:37,  8.65s/it]
                                                          
{'loss': 0.1058, 'grad_norm': 2.1087124347686768, 'learning_rate': 3.74827761454499e-05, 'epoch': 1.87}

 19%|█▊        | 7697/41250 [18:35:29<80:39:37,  8.65s/it][2025-04-26 02:33:12,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.19
[2025-04-26 02:33:12,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.60 | bwd_microstep: 5695.41 | bwd_inner_microstep: 5640.94 | bwd_allreduce_microstep: 54.42 | step_microstep: 19.72
[2025-04-26 02:33:12,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.60 | bwd: 5695.43 | bwd_inner: 5640.94 | bwd_allreduce: 54.45 | step: 19.72

 19%|█▊        | 7698/41250 [18:35:38<80:31:16,  8.64s/it]
                                                          
{'loss': 0.1211, 'grad_norm': 1.7929749488830566, 'learning_rate': 3.7482013422013085e-05, 'epoch': 1.87}

 19%|█▊        | 7698/41250 [18:35:38<80:31:16,  8.64s/it][2025-04-26 02:33:21,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:33:21,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.13 | bwd_microstep: 5736.61 | bwd_inner_microstep: 5689.52 | bwd_allreduce_microstep: 47.05 | step_microstep: 18.75
[2025-04-26 02:33:21,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.13 | bwd: 5736.62 | bwd_inner: 5689.52 | bwd_allreduce: 47.07 | step: 18.75

 19%|█▊        | 7699/41250 [18:35:46<80:37:15,  8.65s/it]
                                                          
{'loss': 0.0561, 'grad_norm': 1.2657146453857422, 'learning_rate': 3.748125059080309e-05, 'epoch': 1.87}

 19%|█▊        | 7699/41250 [18:35:46<80:37:15,  8.65s/it][2025-04-26 02:33:30,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 02:33:30,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2926.43 | bwd_microstep: 5875.53 | bwd_inner_microstep: 5862.33 | bwd_allreduce_microstep: 13.16 | step_microstep: 18.75
[2025-04-26 02:33:30,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2926.43 | bwd: 5875.54 | bwd_inner: 5862.32 | bwd_allreduce: 13.17 | step: 18.75

 19%|█▊        | 7700/41250 [18:35:55<81:16:50,  8.72s/it]
                                                          
{'loss': 0.1733, 'grad_norm': 1.549142837524414, 'learning_rate': 3.748048765182462e-05, 'epoch': 1.87}

 19%|█▊        | 7700/41250 [18:35:55<81:16:50,  8.72s/it][2025-04-26 02:33:38,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 02:33:38,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.47 | bwd_microstep: 5761.70 | bwd_inner_microstep: 5647.97 | bwd_allreduce_microstep: 113.68 | step_microstep: 18.64
[2025-04-26 02:33:38,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.47 | bwd: 5761.71 | bwd_inner: 5647.97 | bwd_allreduce: 113.70 | step: 18.64

 19%|█▊        | 7701/41250 [18:36:04<81:08:09,  8.71s/it]
                                                          
{'loss': 0.3092, 'grad_norm': 3.81469464302063, 'learning_rate': 3.747972460508239e-05, 'epoch': 1.87}

 19%|█▊        | 7701/41250 [18:36:04<81:08:09,  8.71s/it][2025-04-26 02:33:47,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:33:47,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.79 | bwd_microstep: 5773.31 | bwd_inner_microstep: 5647.05 | bwd_allreduce_microstep: 126.21 | step_microstep: 19.01
[2025-04-26 02:33:47,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.79 | bwd: 5773.33 | bwd_inner: 5647.05 | bwd_allreduce: 126.23 | step: 19.01

 19%|█▊        | 7702/41250 [18:36:12<81:03:22,  8.70s/it]
                                                          
{'loss': 0.0334, 'grad_norm': 0.7407433390617371, 'learning_rate': 3.7478961450581085e-05, 'epoch': 1.87}

 19%|█▊        | 7702/41250 [18:36:12<81:03:22,  8.70s/it][2025-04-26 02:33:56,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.24 | optimizer_step: 0.99
[2025-04-26 02:33:56,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.50 | bwd_microstep: 5694.37 | bwd_inner_microstep: 5681.47 | bwd_allreduce_microstep: 12.84 | step_microstep: 20.08
[2025-04-26 02:33:56,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.50 | bwd: 5694.38 | bwd_inner: 5681.47 | bwd_allreduce: 12.87 | step: 20.08

 19%|█▊        | 7703/41250 [18:36:21<80:50:22,  8.68s/it]
                                                          
{'loss': 0.0385, 'grad_norm': 1.1069931983947754, 'learning_rate': 3.7478198188325434e-05, 'epoch': 1.87}

 19%|█▊        | 7703/41250 [18:36:21<80:50:22,  8.68s/it][2025-04-26 02:34:05,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 02:34:05,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.27 | bwd_microstep: 5993.44 | bwd_inner_microstep: 5673.87 | bwd_allreduce_microstep: 319.52 | step_microstep: 19.24
[2025-04-26 02:34:05,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.27 | bwd: 5993.45 | bwd_inner: 5673.87 | bwd_allreduce: 319.54 | step: 19.24

 19%|█▊        | 7704/41250 [18:36:30<81:30:57,  8.75s/it]
                                                          
{'loss': 0.1098, 'grad_norm': 1.918540120124817, 'learning_rate': 3.747743481832012e-05, 'epoch': 1.87}

 19%|█▊        | 7704/41250 [18:36:30<81:30:57,  8.75s/it][2025-04-26 02:34:13,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 02:34:13,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.65 | bwd_microstep: 5725.10 | bwd_inner_microstep: 5675.32 | bwd_allreduce_microstep: 49.74 | step_microstep: 18.76
[2025-04-26 02:34:13,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.65 | bwd: 5725.12 | bwd_inner: 5675.32 | bwd_allreduce: 49.76 | step: 18.76

 19%|█▊        | 7705/41250 [18:36:39<81:14:03,  8.72s/it]
                                                          
{'loss': 0.0832, 'grad_norm': 1.9458351135253906, 'learning_rate': 3.747667134056986e-05, 'epoch': 1.87}

 19%|█▊        | 7705/41250 [18:36:39<81:14:03,  8.72s/it][2025-04-26 02:34:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:34:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.03 | bwd_microstep: 5761.99 | bwd_inner_microstep: 5625.36 | bwd_allreduce_microstep: 136.59 | step_microstep: 19.01
[2025-04-26 02:34:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.03 | bwd: 5762.01 | bwd_inner: 5625.36 | bwd_allreduce: 136.61 | step: 19.02

 19%|█▊        | 7706/41250 [18:36:47<81:04:01,  8.70s/it]
                                                          
{'loss': 0.2426, 'grad_norm': 3.396961212158203, 'learning_rate': 3.747590775507936e-05, 'epoch': 1.87}

 19%|█▊        | 7706/41250 [18:36:47<81:04:01,  8.70s/it][2025-04-26 02:34:31,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.07 | optimizer_step: 1.14
[2025-04-26 02:34:31,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.46 | bwd_microstep: 5769.96 | bwd_inner_microstep: 5625.46 | bwd_allreduce_microstep: 144.45 | step_microstep: 19.34
[2025-04-26 02:34:31,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.46 | bwd: 5769.97 | bwd_inner: 5625.46 | bwd_allreduce: 144.47 | step: 19.34

 19%|█▊        | 7707/41250 [18:36:56<80:59:28,  8.69s/it]
                                                          
{'loss': 0.1087, 'grad_norm': 3.031385898590088, 'learning_rate': 3.7475144061853324e-05, 'epoch': 1.87}

 19%|█▊        | 7707/41250 [18:36:56<80:59:28,  8.69s/it][2025-04-26 02:34:39,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:34:39,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5737.84 | bwd_inner_microstep: 5686.34 | bwd_allreduce_microstep: 51.45 | step_microstep: 18.79
[2025-04-26 02:34:39,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5737.85 | bwd_inner: 5686.34 | bwd_allreduce: 51.47 | step: 18.79

 19%|█▊        | 7708/41250 [18:37:05<80:54:25,  8.68s/it]
                                                          
{'loss': 0.0503, 'grad_norm': 1.226146936416626, 'learning_rate': 3.747438026089646e-05, 'epoch': 1.87}

 19%|█▊        | 7708/41250 [18:37:05<80:54:25,  8.68s/it][2025-04-26 02:34:48,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:34:48,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.68 | bwd_microstep: 5690.24 | bwd_inner_microstep: 5677.57 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.64
[2025-04-26 02:34:48,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.68 | bwd: 5690.26 | bwd_inner: 5677.57 | bwd_allreduce: 12.64 | step: 18.64

 19%|█▊        | 7709/41250 [18:37:13<80:41:56,  8.66s/it]
                                                          
{'loss': 0.1261, 'grad_norm': 2.6045901775360107, 'learning_rate': 3.747361635221348e-05, 'epoch': 1.87}

 19%|█▊        | 7709/41250 [18:37:13<80:41:56,  8.66s/it][2025-04-26 02:34:57,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:34:57,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2814.05 | bwd_microstep: 5757.44 | bwd_inner_microstep: 5643.34 | bwd_allreduce_microstep: 114.06 | step_microstep: 18.68
[2025-04-26 02:34:57,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2814.06 | bwd: 5757.45 | bwd_inner: 5643.34 | bwd_allreduce: 114.07 | step: 18.68

 19%|█▊        | 7710/41250 [18:37:22<80:40:43,  8.66s/it]
                                                          
{'loss': 0.0169, 'grad_norm': 0.6618806719779968, 'learning_rate': 3.747285233580909e-05, 'epoch': 1.87}

 19%|█▊        | 7710/41250 [18:37:22<80:40:43,  8.66s/it][2025-04-26 02:35:05,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:35:05,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.28 | bwd_microstep: 5749.11 | bwd_inner_microstep: 5638.22 | bwd_allreduce_microstep: 110.84 | step_microstep: 18.35
[2025-04-26 02:35:05,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.28 | bwd: 5749.13 | bwd_inner: 5638.22 | bwd_allreduce: 110.86 | step: 18.36

 19%|█▊        | 7711/41250 [18:37:31<80:39:57,  8.66s/it]
                                                          
{'loss': 0.0951, 'grad_norm': 1.6398183107376099, 'learning_rate': 3.747208821168801e-05, 'epoch': 1.87}

 19%|█▊        | 7711/41250 [18:37:31<80:39:57,  8.66s/it][2025-04-26 02:35:14,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.14
[2025-04-26 02:35:14,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.24 | bwd_microstep: 5690.26 | bwd_inner_microstep: 5677.36 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.28
[2025-04-26 02:35:14,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.24 | bwd: 5690.27 | bwd_inner: 5677.36 | bwd_allreduce: 12.87 | step: 19.29

 19%|█▊        | 7712/41250 [18:37:39<80:32:43,  8.65s/it]
                                                          
{'loss': 0.0765, 'grad_norm': 1.0906096696853638, 'learning_rate': 3.747132397985494e-05, 'epoch': 1.87}

 19%|█▊        | 7712/41250 [18:37:39<80:32:43,  8.65s/it][2025-04-26 02:35:23,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:35:23,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.77 | bwd_microstep: 5760.96 | bwd_inner_microstep: 5649.57 | bwd_allreduce_microstep: 111.35 | step_microstep: 18.70
[2025-04-26 02:35:23,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.77 | bwd: 5760.97 | bwd_inner: 5649.57 | bwd_allreduce: 111.37 | step: 18.70

 19%|█▊        | 7713/41250 [18:37:48<80:37:14,  8.65s/it]
                                                          
{'loss': 0.2141, 'grad_norm': 1.4056813716888428, 'learning_rate': 3.747055964031459e-05, 'epoch': 1.87}

 19%|█▊        | 7713/41250 [18:37:48<80:37:14,  8.65s/it][2025-04-26 02:35:31,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:35:31,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.47 | bwd_microstep: 5723.32 | bwd_inner_microstep: 5700.54 | bwd_allreduce_microstep: 22.73 | step_microstep: 18.52
[2025-04-26 02:35:31,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.47 | bwd: 5723.33 | bwd_inner: 5700.54 | bwd_allreduce: 22.75 | step: 18.52

 19%|█▊        | 7714/41250 [18:37:56<80:38:26,  8.66s/it]
                                                          
{'loss': 0.2178, 'grad_norm': 2.927652597427368, 'learning_rate': 3.746979519307168e-05, 'epoch': 1.87}

 19%|█▊        | 7714/41250 [18:37:56<80:38:26,  8.66s/it][2025-04-26 02:35:40,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:35:40,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.70 | bwd_microstep: 5711.37 | bwd_inner_microstep: 5698.63 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.71
[2025-04-26 02:35:40,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.70 | bwd: 5711.38 | bwd_inner: 5698.63 | bwd_allreduce: 12.71 | step: 18.71

 19%|█▊        | 7715/41250 [18:38:05<80:35:26,  8.65s/it]
                                                          
{'loss': 0.258, 'grad_norm': 3.302464723587036, 'learning_rate': 3.746903063813092e-05, 'epoch': 1.87}

 19%|█▊        | 7715/41250 [18:38:05<80:35:26,  8.65s/it][2025-04-26 02:35:48,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.14 | optimizer_step: 1.07
[2025-04-26 02:35:48,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.68 | bwd_microstep: 5689.85 | bwd_inner_microstep: 5649.93 | bwd_allreduce_microstep: 39.86 | step_microstep: 20.41
[2025-04-26 02:35:48,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5689.87 | bwd_inner: 5649.93 | bwd_allreduce: 39.89 | step: 20.41

 19%|█▊        | 7716/41250 [18:38:14<80:27:19,  8.64s/it]
                                                          
{'loss': 0.0984, 'grad_norm': 3.551663398742676, 'learning_rate': 3.746826597549701e-05, 'epoch': 1.87}

 19%|█▊        | 7716/41250 [18:38:14<80:27:19,  8.64s/it][2025-04-26 02:35:57,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 0.92
[2025-04-26 02:35:57,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.51 | bwd_microstep: 5772.38 | bwd_inner_microstep: 5647.55 | bwd_allreduce_microstep: 124.78 | step_microstep: 19.84
[2025-04-26 02:35:57,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.51 | bwd: 5772.39 | bwd_inner: 5647.55 | bwd_allreduce: 124.80 | step: 19.84

 19%|█▊        | 7717/41250 [18:38:22<80:35:36,  8.65s/it]
                                                          
{'loss': 0.1338, 'grad_norm': 1.8323569297790527, 'learning_rate': 3.7467501205174686e-05, 'epoch': 1.87}

 19%|█▊        | 7717/41250 [18:38:22<80:35:36,  8.65s/it][2025-04-26 02:36:06,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 02:36:06,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.60 | bwd_microstep: 5991.39 | bwd_inner_microstep: 5700.39 | bwd_allreduce_microstep: 290.95 | step_microstep: 18.79
[2025-04-26 02:36:06,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.60 | bwd: 5991.40 | bwd_inner: 5700.39 | bwd_allreduce: 290.97 | step: 18.80

 19%|█▊        | 7718/41250 [18:38:31<81:21:34,  8.73s/it]
                                                          
{'loss': 0.3587, 'grad_norm': 4.722893238067627, 'learning_rate': 3.7466736327168655e-05, 'epoch': 1.87}

 19%|█▊        | 7718/41250 [18:38:31<81:21:34,  8.73s/it][2025-04-26 02:36:15,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:36:15,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.88 | bwd_microstep: 5694.76 | bwd_inner_microstep: 5651.50 | bwd_allreduce_microstep: 43.20 | step_microstep: 18.79
[2025-04-26 02:36:15,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.88 | bwd: 5694.77 | bwd_inner: 5651.50 | bwd_allreduce: 43.22 | step: 18.79

 19%|█▊        | 7719/41250 [18:38:40<80:59:48,  8.70s/it]
                                                          
{'loss': 0.0825, 'grad_norm': 2.390090227127075, 'learning_rate': 3.746597134148363e-05, 'epoch': 1.87}

 19%|█▊        | 7719/41250 [18:38:40<80:59:48,  8.70s/it][2025-04-26 02:36:23,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 02:36:23,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.55 | bwd_microstep: 5702.30 | bwd_inner_microstep: 5648.51 | bwd_allreduce_microstep: 53.74 | step_microstep: 19.08
[2025-04-26 02:36:23,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.55 | bwd: 5702.31 | bwd_inner: 5648.51 | bwd_allreduce: 53.76 | step: 19.09

 19%|█▊        | 7720/41250 [18:38:49<80:45:03,  8.67s/it]
                                                          
{'loss': 0.1356, 'grad_norm': 2.168428421020508, 'learning_rate': 3.746520624812432e-05, 'epoch': 1.87}

 19%|█▊        | 7720/41250 [18:38:49<80:45:03,  8.67s/it][2025-04-26 02:36:32,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:36:32,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.38 | bwd_microstep: 5710.50 | bwd_inner_microstep: 5697.70 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.30
[2025-04-26 02:36:32,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.38 | bwd: 5710.52 | bwd_inner: 5697.70 | bwd_allreduce: 12.78 | step: 18.30

 19%|█▊        | 7721/41250 [18:38:57<80:42:07,  8.66s/it]
                                                          
{'loss': 0.1054, 'grad_norm': 1.9271867275238037, 'learning_rate': 3.746444104709545e-05, 'epoch': 1.87}

 19%|█▊        | 7721/41250 [18:38:57<80:42:07,  8.66s/it][2025-04-26 02:36:41,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 02:36:41,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.48 | bwd_microstep: 5695.85 | bwd_inner_microstep: 5647.97 | bwd_allreduce_microstep: 47.83 | step_microstep: 18.55
[2025-04-26 02:36:41,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.48 | bwd: 5695.86 | bwd_inner: 5647.97 | bwd_allreduce: 47.85 | step: 18.55

 19%|█▊        | 7722/41250 [18:39:06<80:32:11,  8.65s/it]
                                                          
{'loss': 0.1601, 'grad_norm': 2.4365813732147217, 'learning_rate': 3.746367573840174e-05, 'epoch': 1.87}

 19%|█▊        | 7722/41250 [18:39:06<80:32:11,  8.65s/it][2025-04-26 02:36:50,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 02:36:50,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.43 | bwd_microstep: 6128.81 | bwd_inner_microstep: 5650.61 | bwd_allreduce_microstep: 478.15 | step_microstep: 18.82
[2025-04-26 02:36:50,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.43 | bwd: 6128.82 | bwd_inner: 5650.61 | bwd_allreduce: 478.17 | step: 18.82

 19%|█▊        | 7723/41250 [18:39:15<81:38:48,  8.77s/it]
                                                          
{'loss': 0.1994, 'grad_norm': 1.3863154649734497, 'learning_rate': 3.746291032204791e-05, 'epoch': 1.87}

 19%|█▊        | 7723/41250 [18:39:15<81:38:48,  8.77s/it][2025-04-26 02:36:58,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-26 02:36:58,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.60 | bwd_microstep: 5869.61 | bwd_inner_microstep: 5856.85 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.55
[2025-04-26 02:36:58,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.60 | bwd: 5869.62 | bwd_inner: 5856.85 | bwd_allreduce: 12.73 | step: 19.55

 19%|█▊        | 7724/41250 [18:39:24<81:58:44,  8.80s/it]
                                                          
{'loss': 0.2763, 'grad_norm': 5.018056869506836, 'learning_rate': 3.746214479803865e-05, 'epoch': 1.87}

 19%|█▊        | 7724/41250 [18:39:24<81:58:44,  8.80s/it][2025-04-26 02:37:07,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 02:37:07,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.79 | bwd_microstep: 5700.42 | bwd_inner_microstep: 5687.68 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.98
[2025-04-26 02:37:07,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.79 | bwd: 5700.43 | bwd_inner: 5687.68 | bwd_allreduce: 12.71 | step: 18.98

 19%|█▊        | 7725/41250 [18:39:32<81:29:15,  8.75s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 2.091240167617798, 'learning_rate': 3.746137916637873e-05, 'epoch': 1.87}

 19%|█▊        | 7725/41250 [18:39:32<81:29:15,  8.75s/it][2025-04-26 02:37:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:37:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5742.93 | bwd_inner_microstep: 5712.55 | bwd_allreduce_microstep: 30.33 | step_microstep: 18.68
[2025-04-26 02:37:16,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5742.94 | bwd_inner: 5712.55 | bwd_allreduce: 30.35 | step: 18.68

 19%|█▊        | 7726/41250 [18:39:41<81:17:35,  8.73s/it]
                                                          
{'loss': 0.1574, 'grad_norm': 2.935206413269043, 'learning_rate': 3.746061342707283e-05, 'epoch': 1.87}

 19%|█▊        | 7726/41250 [18:39:41<81:17:35,  8.73s/it][2025-04-26 02:37:24,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:37:24,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5764.71 | bwd_inner_microstep: 5703.95 | bwd_allreduce_microstep: 60.71 | step_microstep: 18.75
[2025-04-26 02:37:24,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5764.72 | bwd_inner: 5703.95 | bwd_allreduce: 60.73 | step: 18.75

 19%|█▊        | 7727/41250 [18:39:50<81:11:48,  8.72s/it]
                                                          
{'loss': 0.0929, 'grad_norm': 1.96268892288208, 'learning_rate': 3.7459847580125675e-05, 'epoch': 1.87}

 19%|█▊        | 7727/41250 [18:39:50<81:11:48,  8.72s/it][2025-04-26 02:37:33,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:37:33,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.46 | bwd_microstep: 5768.00 | bwd_inner_microstep: 5658.94 | bwd_allreduce_microstep: 109.02 | step_microstep: 18.81
[2025-04-26 02:37:33,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.46 | bwd: 5768.02 | bwd_inner: 5658.94 | bwd_allreduce: 109.04 | step: 18.81

 19%|█▊        | 7728/41250 [18:39:58<81:06:14,  8.71s/it]
                                                          
{'loss': 0.0352, 'grad_norm': 0.9511499404907227, 'learning_rate': 3.7459081625542e-05, 'epoch': 1.87}

 19%|█▊        | 7728/41250 [18:39:58<81:06:14,  8.71s/it][2025-04-26 02:37:42,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:37:42,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.02 | bwd_microstep: 5719.02 | bwd_inner_microstep: 5695.29 | bwd_allreduce_microstep: 23.67 | step_microstep: 18.64
[2025-04-26 02:37:42,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.02 | bwd: 5719.04 | bwd_inner: 5695.29 | bwd_allreduce: 23.70 | step: 18.64

 19%|█▊        | 7729/41250 [18:40:07<80:56:51,  8.69s/it]
                                                          
{'loss': 0.3016, 'grad_norm': 5.410097599029541, 'learning_rate': 3.745831556332652e-05, 'epoch': 1.87}

 19%|█▊        | 7729/41250 [18:40:07<80:56:51,  8.69s/it][2025-04-26 02:37:50,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 02:37:50,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.89 | bwd_microstep: 5720.19 | bwd_inner_microstep: 5639.81 | bwd_allreduce_microstep: 80.33 | step_microstep: 19.12
[2025-04-26 02:37:50,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.89 | bwd: 5720.21 | bwd_inner: 5639.81 | bwd_allreduce: 80.35 | step: 19.13

 19%|█▊        | 7730/41250 [18:40:16<80:46:02,  8.67s/it]
                                                          
{'loss': 0.1663, 'grad_norm': 3.6293585300445557, 'learning_rate': 3.7457549393483964e-05, 'epoch': 1.87}

 19%|█▊        | 7730/41250 [18:40:16<80:46:02,  8.67s/it][2025-04-26 02:37:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 02:37:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.25 | bwd_microstep: 5698.73 | bwd_inner_microstep: 5686.15 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.35
[2025-04-26 02:37:59,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.25 | bwd: 5698.74 | bwd_inner: 5686.15 | bwd_allreduce: 12.55 | step: 18.36

 19%|█▊        | 7731/41250 [18:40:24<80:39:41,  8.66s/it]
                                                          
{'loss': 0.1276, 'grad_norm': 3.848191976547241, 'learning_rate': 3.745678311601904e-05, 'epoch': 1.87}

 19%|█▊        | 7731/41250 [18:40:24<80:39:41,  8.66s/it][2025-04-26 02:38:08,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:38:08,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.53 | bwd_microstep: 5693.53 | bwd_inner_microstep: 5655.97 | bwd_allreduce_microstep: 37.52 | step_microstep: 18.50
[2025-04-26 02:38:08,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.53 | bwd: 5693.54 | bwd_inner: 5655.97 | bwd_allreduce: 37.53 | step: 18.50

 19%|█▊        | 7732/41250 [18:40:33<80:32:12,  8.65s/it]
                                                          
{'loss': 0.1364, 'grad_norm': 3.614423990249634, 'learning_rate': 3.7456016730936496e-05, 'epoch': 1.87}

 19%|█▊        | 7732/41250 [18:40:33<80:32:12,  8.65s/it][2025-04-26 02:38:16,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-26 02:38:16,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.70 | bwd_microstep: 5714.95 | bwd_inner_microstep: 5701.99 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.68
[2025-04-26 02:38:16,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.70 | bwd: 5714.96 | bwd_inner: 5701.99 | bwd_allreduce: 12.94 | step: 18.68

 19%|█▊        | 7733/41250 [18:40:42<80:32:06,  8.65s/it]
                                                          
{'loss': 0.1696, 'grad_norm': 1.0674803256988525, 'learning_rate': 3.745525023824103e-05, 'epoch': 1.87}

 19%|█▊        | 7733/41250 [18:40:42<80:32:06,  8.65s/it][2025-04-26 02:38:25,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:38:25,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.48 | bwd_microstep: 5754.70 | bwd_inner_microstep: 5693.60 | bwd_allreduce_microstep: 61.05 | step_microstep: 18.13
[2025-04-26 02:38:25,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.48 | bwd: 5754.71 | bwd_inner: 5693.60 | bwd_allreduce: 61.07 | step: 18.13

 19%|█▊        | 7734/41250 [18:40:50<80:38:55,  8.66s/it]
                                                          
{'loss': 0.487, 'grad_norm': 5.128145217895508, 'learning_rate': 3.745448363793738e-05, 'epoch': 1.87}

 19%|█▊        | 7734/41250 [18:40:50<80:38:55,  8.66s/it][2025-04-26 02:38:34,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:38:34,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.60 | bwd_microstep: 5766.03 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 72.11 | step_microstep: 18.61
[2025-04-26 02:38:34,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.60 | bwd: 5766.04 | bwd_inner: 5693.87 | bwd_allreduce: 72.13 | step: 18.61

 19%|█▉        | 7735/41250 [18:40:59<80:46:13,  8.68s/it]
                                                          
{'loss': 0.2899, 'grad_norm': 3.028493881225586, 'learning_rate': 3.745371693003027e-05, 'epoch': 1.88}

 19%|█▉        | 7735/41250 [18:40:59<80:46:13,  8.68s/it][2025-04-26 02:38:42,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 02:38:42,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.37 | bwd_microstep: 5782.44 | bwd_inner_microstep: 5650.15 | bwd_allreduce_microstep: 132.25 | step_microstep: 18.57
[2025-04-26 02:38:42,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.37 | bwd: 5782.45 | bwd_inner: 5650.15 | bwd_allreduce: 132.27 | step: 18.57

 19%|█▉        | 7736/41250 [18:41:08<80:50:39,  8.68s/it]
                                                          
{'loss': 0.0762, 'grad_norm': 1.360615611076355, 'learning_rate': 3.745295011452444e-05, 'epoch': 1.88}

 19%|█▉        | 7736/41250 [18:41:08<80:50:39,  8.68s/it][2025-04-26 02:38:51,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:38:51,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.15 | bwd_microstep: 5788.43 | bwd_inner_microstep: 5656.26 | bwd_allreduce_microstep: 132.12 | step_microstep: 18.53
[2025-04-26 02:38:51,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.15 | bwd: 5788.44 | bwd_inner: 5656.26 | bwd_allreduce: 132.14 | step: 18.55

 19%|█▉        | 7737/41250 [18:41:16<80:52:49,  8.69s/it]
                                                          
{'loss': 0.1247, 'grad_norm': 2.236340045928955, 'learning_rate': 3.7452183191424595e-05, 'epoch': 1.88}

 19%|█▉        | 7737/41250 [18:41:16<80:52:49,  8.69s/it][2025-04-26 02:39:00,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:39:00,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.96 | bwd_microstep: 5706.21 | bwd_inner_microstep: 5650.56 | bwd_allreduce_microstep: 55.60 | step_microstep: 18.27
[2025-04-26 02:39:00,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.96 | bwd: 5706.22 | bwd_inner: 5650.56 | bwd_allreduce: 55.62 | step: 18.27

 19%|█▉        | 7738/41250 [18:41:25<80:40:00,  8.67s/it]
                                                          
{'loss': 0.1147, 'grad_norm': 1.095980167388916, 'learning_rate': 3.7451416160735475e-05, 'epoch': 1.88}

 19%|█▉        | 7738/41250 [18:41:25<80:40:00,  8.67s/it][2025-04-26 02:39:08,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:39:08,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.30 | bwd_microstep: 5750.83 | bwd_inner_microstep: 5646.11 | bwd_allreduce_microstep: 104.68 | step_microstep: 18.40
[2025-04-26 02:39:08,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.30 | bwd: 5750.85 | bwd_inner: 5646.11 | bwd_allreduce: 104.70 | step: 18.40

 19%|█▉        | 7739/41250 [18:41:34<80:40:38,  8.67s/it]
                                                          
{'loss': 0.0916, 'grad_norm': 1.4935111999511719, 'learning_rate': 3.745064902246181e-05, 'epoch': 1.88}

 19%|█▉        | 7739/41250 [18:41:34<80:40:38,  8.67s/it][2025-04-26 02:39:17,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 02:39:17,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.36 | bwd_microstep: 5754.92 | bwd_inner_microstep: 5680.19 | bwd_allreduce_microstep: 74.68 | step_microstep: 18.73
[2025-04-26 02:39:17,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.36 | bwd: 5754.93 | bwd_inner: 5680.19 | bwd_allreduce: 74.70 | step: 18.73

 19%|█▉        | 7740/41250 [18:41:42<80:43:48,  8.67s/it]
                                                          
{'loss': 0.1043, 'grad_norm': 1.8758310079574585, 'learning_rate': 3.744988177660832e-05, 'epoch': 1.88}

 19%|█▉        | 7740/41250 [18:41:42<80:43:48,  8.67s/it][2025-04-26 02:39:26,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:39:26,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.77 | bwd_microstep: 5767.60 | bwd_inner_microstep: 5687.84 | bwd_allreduce_microstep: 79.72 | step_microstep: 18.75
[2025-04-26 02:39:26,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.77 | bwd: 5767.61 | bwd_inner: 5687.84 | bwd_allreduce: 79.73 | step: 18.75

 19%|█▉        | 7741/41250 [18:41:51<80:48:51,  8.68s/it]
                                                          
{'loss': 0.3375, 'grad_norm': 3.9358344078063965, 'learning_rate': 3.7449114423179746e-05, 'epoch': 1.88}

 19%|█▉        | 7741/41250 [18:41:51<80:48:51,  8.68s/it][2025-04-26 02:39:34,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:39:34,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5694.48 | bwd_inner_microstep: 5634.07 | bwd_allreduce_microstep: 60.36 | step_microstep: 18.83
[2025-04-26 02:39:34,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5694.49 | bwd_inner: 5634.07 | bwd_allreduce: 60.38 | step: 18.83

 19%|█▉        | 7742/41250 [18:42:00<80:36:17,  8.66s/it]
                                                          
{'loss': 0.2485, 'grad_norm': 3.315063953399658, 'learning_rate': 3.744834696218082e-05, 'epoch': 1.88}

 19%|█▉        | 7742/41250 [18:42:00<80:36:17,  8.66s/it][2025-04-26 02:39:43,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:39:43,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.75 | bwd_microstep: 5764.87 | bwd_inner_microstep: 5689.74 | bwd_allreduce_microstep: 75.08 | step_microstep: 18.70
[2025-04-26 02:39:43,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.75 | bwd: 5764.88 | bwd_inner: 5689.74 | bwd_allreduce: 75.10 | step: 18.70

 19%|█▉        | 7743/41250 [18:42:08<80:40:48,  8.67s/it]
                                                          
{'loss': 0.2208, 'grad_norm': 1.9611586332321167, 'learning_rate': 3.744757939361626e-05, 'epoch': 1.88}

 19%|█▉        | 7743/41250 [18:42:08<80:40:48,  8.67s/it][2025-04-26 02:39:52,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:39:52,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5751.21 | bwd_inner_microstep: 5681.56 | bwd_allreduce_microstep: 69.61 | step_microstep: 18.73
[2025-04-26 02:39:52,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.37 | bwd: 5751.22 | bwd_inner: 5681.56 | bwd_allreduce: 69.62 | step: 18.74

 19%|█▉        | 7744/41250 [18:42:17<80:41:45,  8.67s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 1.7522038221359253, 'learning_rate': 3.7446811717490803e-05, 'epoch': 1.88}

 19%|█▉        | 7744/41250 [18:42:17<80:41:45,  8.67s/it][2025-04-26 02:40:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:40:00,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.79 | bwd_microstep: 5701.90 | bwd_inner_microstep: 5688.06 | bwd_allreduce_microstep: 13.79 | step_microstep: 18.18
[2025-04-26 02:40:00,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.79 | bwd: 5701.91 | bwd_inner: 5688.06 | bwd_allreduce: 13.81 | step: 18.18

 19%|█▉        | 7745/41250 [18:42:26<80:33:57,  8.66s/it]
                                                          
{'loss': 0.0677, 'grad_norm': 1.4166680574417114, 'learning_rate': 3.744604393380918e-05, 'epoch': 1.88}

 19%|█▉        | 7745/41250 [18:42:26<80:33:57,  8.66s/it][2025-04-26 02:40:09,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-26 02:40:09,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.72 | bwd_microstep: 5700.87 | bwd_inner_microstep: 5639.78 | bwd_allreduce_microstep: 61.03 | step_microstep: 19.72
[2025-04-26 02:40:09,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.72 | bwd: 5700.89 | bwd_inner: 5639.78 | bwd_allreduce: 61.06 | step: 19.72

 19%|█▉        | 7746/41250 [18:42:34<80:25:00,  8.64s/it]
                                                          
{'loss': 0.0466, 'grad_norm': 0.739491879940033, 'learning_rate': 3.744527604257613e-05, 'epoch': 1.88}

 19%|█▉        | 7746/41250 [18:42:34<80:25:00,  8.64s/it][2025-04-26 02:40:18,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.22 | optimizer_step: 0.92
[2025-04-26 02:40:18,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.92 | bwd_microstep: 5705.42 | bwd_inner_microstep: 5692.23 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.26
[2025-04-26 02:40:18,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.92 | bwd: 5705.44 | bwd_inner: 5692.23 | bwd_allreduce: 13.16 | step: 19.26

 19%|█▉        | 7747/41250 [18:42:43<80:23:19,  8.64s/it]
                                                          
{'loss': 0.1143, 'grad_norm': 1.1823534965515137, 'learning_rate': 3.744450804379639e-05, 'epoch': 1.88}

 19%|█▉        | 7747/41250 [18:42:43<80:23:19,  8.64s/it][2025-04-26 02:40:26,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-26 02:40:26,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.01 | bwd_microstep: 5699.22 | bwd_inner_microstep: 5686.49 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.76
[2025-04-26 02:40:26,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.01 | bwd: 5699.23 | bwd_inner: 5686.49 | bwd_allreduce: 12.70 | step: 18.76

 19%|█▉        | 7748/41250 [18:42:52<80:23:41,  8.64s/it]
                                                          
{'loss': 0.1864, 'grad_norm': 2.524021625518799, 'learning_rate': 3.744373993747469e-05, 'epoch': 1.88}

 19%|█▉        | 7748/41250 [18:42:52<80:23:41,  8.64s/it][2025-04-26 02:40:35,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 02:40:35,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.57 | bwd_microstep: 5710.10 | bwd_inner_microstep: 5640.84 | bwd_allreduce_microstep: 69.22 | step_microstep: 18.36
[2025-04-26 02:40:35,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.57 | bwd: 5710.11 | bwd_inner: 5640.84 | bwd_allreduce: 69.23 | step: 18.37

 19%|█▉        | 7749/41250 [18:43:00<80:19:37,  8.63s/it]
                                                          
{'loss': 0.0758, 'grad_norm': 2.739722728729248, 'learning_rate': 3.7442971723615755e-05, 'epoch': 1.88}

 19%|█▉        | 7749/41250 [18:43:00<80:19:37,  8.63s/it][2025-04-26 02:40:43,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 0.97
[2025-04-26 02:40:43,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.12 | bwd_microstep: 5694.44 | bwd_inner_microstep: 5681.25 | bwd_allreduce_microstep: 13.15 | step_microstep: 18.94
[2025-04-26 02:40:43,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.12 | bwd: 5694.46 | bwd_inner: 5681.25 | bwd_allreduce: 13.17 | step: 18.94

 19%|█▉        | 7750/41250 [18:43:09<80:17:33,  8.63s/it]
                                                          
{'loss': 0.0639, 'grad_norm': 0.6125897169113159, 'learning_rate': 3.744220340222434e-05, 'epoch': 1.88}

 19%|█▉        | 7750/41250 [18:43:09<80:17:33,  8.63s/it][2025-04-26 02:40:52,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:40:52,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.93 | bwd_microstep: 5747.79 | bwd_inner_microstep: 5638.78 | bwd_allreduce_microstep: 108.97 | step_microstep: 18.93
[2025-04-26 02:40:52,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.93 | bwd: 5747.81 | bwd_inner: 5638.78 | bwd_allreduce: 108.99 | step: 18.93

 19%|█▉        | 7751/41250 [18:43:17<80:22:03,  8.64s/it]
                                                          
{'loss': 0.1628, 'grad_norm': 1.77158522605896, 'learning_rate': 3.7441434973305167e-05, 'epoch': 1.88}

 19%|█▉        | 7751/41250 [18:43:17<80:22:03,  8.64s/it][2025-04-26 02:41:01,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-26 02:41:01,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 5731.42 | bwd_inner_microstep: 5690.66 | bwd_allreduce_microstep: 40.72 | step_microstep: 18.51
[2025-04-26 02:41:01,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 5731.44 | bwd_inner: 5690.66 | bwd_allreduce: 40.73 | step: 18.51

 19%|█▉        | 7752/41250 [18:43:26<80:26:05,  8.64s/it]
                                                          
{'loss': 0.3101, 'grad_norm': 2.897172212600708, 'learning_rate': 3.744066643686298e-05, 'epoch': 1.88}

 19%|█▉        | 7752/41250 [18:43:26<80:26:05,  8.64s/it][2025-04-26 02:41:09,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 02:41:09,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.01 | bwd_microstep: 5711.48 | bwd_inner_microstep: 5698.61 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.48
[2025-04-26 02:41:09,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.01 | bwd: 5711.49 | bwd_inner: 5698.61 | bwd_allreduce: 12.84 | step: 18.49

 19%|█▉        | 7753/41250 [18:43:35<80:24:34,  8.64s/it]
                                                          
{'loss': 0.2519, 'grad_norm': 5.349159240722656, 'learning_rate': 3.7439897792902506e-05, 'epoch': 1.88}

 19%|█▉        | 7753/41250 [18:43:35<80:24:34,  8.64s/it][2025-04-26 02:41:18,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:41:18,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.05 | bwd_microstep: 5715.00 | bwd_inner_microstep: 5634.19 | bwd_allreduce_microstep: 80.76 | step_microstep: 18.36
[2025-04-26 02:41:18,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.05 | bwd: 5715.01 | bwd_inner: 5634.19 | bwd_allreduce: 80.78 | step: 18.37

 19%|█▉        | 7754/41250 [18:43:43<80:19:26,  8.63s/it]
                                                          
{'loss': 0.2562, 'grad_norm': 3.2081940174102783, 'learning_rate': 3.74391290414285e-05, 'epoch': 1.88}

 19%|█▉        | 7754/41250 [18:43:43<80:19:26,  8.63s/it][2025-04-26 02:41:27,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-26 02:41:27,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.68 | bwd_microstep: 5874.55 | bwd_inner_microstep: 5861.72 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.18
[2025-04-26 02:41:27,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.68 | bwd: 5874.56 | bwd_inner: 5861.72 | bwd_allreduce: 12.79 | step: 19.18

 19%|█▉        | 7755/41250 [18:43:52<81:02:49,  8.71s/it]
                                                          
{'loss': 0.1139, 'grad_norm': 1.2012370824813843, 'learning_rate': 3.7438360182445694e-05, 'epoch': 1.88}

 19%|█▉        | 7755/41250 [18:43:52<81:02:49,  8.71s/it][2025-04-26 02:41:36,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-26 02:41:36,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5733.65 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 43.39 | step_microstep: 18.85
[2025-04-26 02:41:36,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5733.66 | bwd_inner: 5690.21 | bwd_allreduce: 43.41 | step: 18.85

 19%|█▉        | 7756/41250 [18:44:01<80:55:19,  8.70s/it]
                                                          
{'loss': 0.1681, 'grad_norm': 2.0036449432373047, 'learning_rate': 3.7437591215958827e-05, 'epoch': 1.88}

 19%|█▉        | 7756/41250 [18:44:01<80:55:19,  8.70s/it][2025-04-26 02:41:44,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:41:44,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.12 | bwd_microstep: 5731.60 | bwd_inner_microstep: 5638.90 | bwd_allreduce_microstep: 92.65 | step_microstep: 18.39
[2025-04-26 02:41:44,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.12 | bwd: 5731.61 | bwd_inner: 5638.90 | bwd_allreduce: 92.66 | step: 18.39

 19%|█▉        | 7757/41250 [18:44:10<80:45:56,  8.68s/it]
                                                          
{'loss': 0.1886, 'grad_norm': 2.0013418197631836, 'learning_rate': 3.7436822141972636e-05, 'epoch': 1.88}

 19%|█▉        | 7757/41250 [18:44:10<80:45:56,  8.68s/it][2025-04-26 02:41:53,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:41:53,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.12 | bwd_microstep: 5880.68 | bwd_inner_microstep: 5693.77 | bwd_allreduce_microstep: 186.87 | step_microstep: 18.77
[2025-04-26 02:41:53,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.12 | bwd: 5880.70 | bwd_inner: 5693.77 | bwd_allreduce: 186.89 | step: 18.77

 19%|█▉        | 7758/41250 [18:44:18<81:07:08,  8.72s/it]
                                                          
{'loss': 0.2069, 'grad_norm': 1.6084426641464233, 'learning_rate': 3.7436052960491866e-05, 'epoch': 1.88}

 19%|█▉        | 7758/41250 [18:44:18<81:07:08,  8.72s/it][2025-04-26 02:42:02,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:42:02,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.70 | bwd_microstep: 5758.88 | bwd_inner_microstep: 5646.03 | bwd_allreduce_microstep: 112.81 | step_microstep: 18.84
[2025-04-26 02:42:02,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.70 | bwd: 5758.90 | bwd_inner: 5646.03 | bwd_allreduce: 112.83 | step: 18.84

 19%|█▉        | 7759/41250 [18:44:27<80:59:40,  8.71s/it]
                                                          
{'loss': 0.1465, 'grad_norm': 0.8897202014923096, 'learning_rate': 3.743528367152127e-05, 'epoch': 1.88}

 19%|█▉        | 7759/41250 [18:44:27<80:59:40,  8.71s/it][2025-04-26 02:42:10,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-26 02:42:10,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.12 | bwd_microstep: 5764.44 | bwd_inner_microstep: 5688.98 | bwd_allreduce_microstep: 75.41 | step_microstep: 18.70
[2025-04-26 02:42:10,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.12 | bwd: 5764.45 | bwd_inner: 5688.98 | bwd_allreduce: 75.43 | step: 18.70

 19%|█▉        | 7760/41250 [18:44:36<80:57:17,  8.70s/it]
                                                          
{'loss': 0.0783, 'grad_norm': 3.063530206680298, 'learning_rate': 3.743451427506557e-05, 'epoch': 1.88}

 19%|█▉        | 7760/41250 [18:44:36<80:57:17,  8.70s/it][2025-04-26 02:42:19,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:42:19,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.66 | bwd_microstep: 5688.98 | bwd_inner_microstep: 5676.27 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.65
[2025-04-26 02:42:19,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.66 | bwd: 5689.00 | bwd_inner: 5676.27 | bwd_allreduce: 12.68 | step: 18.65

 19%|█▉        | 7761/41250 [18:44:44<80:43:31,  8.68s/it]
                                                          
{'loss': 0.0376, 'grad_norm': 0.6029969453811646, 'learning_rate': 3.743374477112952e-05, 'epoch': 1.88}

 19%|█▉        | 7761/41250 [18:44:44<80:43:31,  8.68s/it][2025-04-26 02:42:28,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:42:28,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.75 | bwd_microstep: 5688.75 | bwd_inner_microstep: 5671.28 | bwd_allreduce_microstep: 17.42 | step_microstep: 18.46
[2025-04-26 02:42:28,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.75 | bwd: 5688.76 | bwd_inner: 5671.28 | bwd_allreduce: 17.44 | step: 18.46

 19%|█▉        | 7762/41250 [18:44:53<80:33:51,  8.66s/it]
                                                          
{'loss': 0.0412, 'grad_norm': 0.6859224438667297, 'learning_rate': 3.7432975159717865e-05, 'epoch': 1.88}

 19%|█▉        | 7762/41250 [18:44:53<80:33:51,  8.66s/it][2025-04-26 02:42:36,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:42:36,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.29 | bwd_microstep: 5697.93 | bwd_inner_microstep: 5685.11 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.57
[2025-04-26 02:42:36,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.29 | bwd: 5697.95 | bwd_inner: 5685.11 | bwd_allreduce: 12.80 | step: 18.57

 19%|█▉        | 7763/41250 [18:45:02<80:28:12,  8.65s/it]
                                                          
{'loss': 0.2672, 'grad_norm': 1.2640830278396606, 'learning_rate': 3.743220544083534e-05, 'epoch': 1.88}

 19%|█▉        | 7763/41250 [18:45:02<80:28:12,  8.65s/it][2025-04-26 02:42:45,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:42:45,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.95 | bwd_microstep: 5700.77 | bwd_inner_microstep: 5657.79 | bwd_allreduce_microstep: 42.93 | step_microstep: 18.75
[2025-04-26 02:42:45,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.95 | bwd: 5700.79 | bwd_inner: 5657.79 | bwd_allreduce: 42.95 | step: 18.76

 19%|█▉        | 7764/41250 [18:45:10<80:22:02,  8.64s/it]
                                                          
{'loss': 0.1822, 'grad_norm': 1.919074296951294, 'learning_rate': 3.7431435614486704e-05, 'epoch': 1.88}

 19%|█▉        | 7764/41250 [18:45:10<80:22:02,  8.64s/it][2025-04-26 02:42:54,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 02:42:54,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.20 | bwd_microstep: 5764.03 | bwd_inner_microstep: 5686.39 | bwd_allreduce_microstep: 77.60 | step_microstep: 18.76
[2025-04-26 02:42:54,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.20 | bwd: 5764.05 | bwd_inner: 5686.39 | bwd_allreduce: 77.62 | step: 18.76

 19%|█▉        | 7765/41250 [18:45:19<80:31:52,  8.66s/it]
                                                          
{'loss': 0.3198, 'grad_norm': 1.7258522510528564, 'learning_rate': 3.743066568067669e-05, 'epoch': 1.88}

 19%|█▉        | 7765/41250 [18:45:19<80:31:52,  8.66s/it][2025-04-26 02:43:02,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 02:43:02,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.63 | bwd_microstep: 5738.82 | bwd_inner_microstep: 5654.91 | bwd_allreduce_microstep: 83.87 | step_microstep: 18.81
[2025-04-26 02:43:02,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.63 | bwd: 5738.84 | bwd_inner: 5654.91 | bwd_allreduce: 83.89 | step: 18.81

 19%|█▉        | 7766/41250 [18:45:28<80:31:28,  8.66s/it]
                                                          
{'loss': 0.2154, 'grad_norm': 4.403244972229004, 'learning_rate': 3.742989563941006e-05, 'epoch': 1.88}

 19%|█▉        | 7766/41250 [18:45:28<80:31:28,  8.66s/it][2025-04-26 02:43:11,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:43:11,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5716.37 | bwd_inner_microstep: 5659.27 | bwd_allreduce_microstep: 57.05 | step_microstep: 18.74
[2025-04-26 02:43:11,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5716.38 | bwd_inner: 5659.27 | bwd_allreduce: 57.07 | step: 18.74

 19%|█▉        | 7767/41250 [18:45:36<80:27:35,  8.65s/it]
                                                          
{'loss': 0.1059, 'grad_norm': 1.6129070520401, 'learning_rate': 3.7429125490691545e-05, 'epoch': 1.88}

 19%|█▉        | 7767/41250 [18:45:36<80:27:35,  8.65s/it][2025-04-26 02:43:20,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:43:20,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2943.04 | bwd_microstep: 5886.66 | bwd_inner_microstep: 5873.94 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.65
[2025-04-26 02:43:20,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2943.04 | bwd: 5886.67 | bwd_inner: 5873.94 | bwd_allreduce: 12.69 | step: 18.66

 19%|█▉        | 7768/41250 [18:45:45<81:11:43,  8.73s/it]
                                                          
{'loss': 0.1047, 'grad_norm': 2.182828903198242, 'learning_rate': 3.7428355234525906e-05, 'epoch': 1.88}

 19%|█▉        | 7768/41250 [18:45:45<81:11:43,  8.73s/it][2025-04-26 02:43:28,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:43:28,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.05 | bwd_microstep: 5699.41 | bwd_inner_microstep: 5686.59 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.52
[2025-04-26 02:43:28,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.05 | bwd: 5699.42 | bwd_inner: 5686.59 | bwd_allreduce: 12.79 | step: 18.53

 19%|█▉        | 7769/41250 [18:45:54<80:54:52,  8.70s/it]
                                                          
{'loss': 0.0586, 'grad_norm': 1.2691185474395752, 'learning_rate': 3.742758487091788e-05, 'epoch': 1.88}

 19%|█▉        | 7769/41250 [18:45:54<80:54:52,  8.70s/it][2025-04-26 02:43:37,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-26 02:43:37,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.06 | bwd_microstep: 5782.96 | bwd_inner_microstep: 5658.09 | bwd_allreduce_microstep: 124.83 | step_microstep: 18.47
[2025-04-26 02:43:37,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.06 | bwd: 5782.97 | bwd_inner: 5658.09 | bwd_allreduce: 124.84 | step: 18.47

 19%|█▉        | 7770/41250 [18:46:02<80:54:22,  8.70s/it]
                                                          
{'loss': 0.1038, 'grad_norm': 2.182033061981201, 'learning_rate': 3.742681439987221e-05, 'epoch': 1.88}

 19%|█▉        | 7770/41250 [18:46:02<80:54:22,  8.70s/it][2025-04-26 02:43:46,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:43:46,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.97 | bwd_microstep: 5767.70 | bwd_inner_microstep: 5703.41 | bwd_allreduce_microstep: 64.24 | step_microstep: 18.21
[2025-04-26 02:43:46,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.97 | bwd: 5767.71 | bwd_inner: 5703.41 | bwd_allreduce: 64.26 | step: 18.21

 19%|█▉        | 7771/41250 [18:46:11<80:55:30,  8.70s/it]
                                                          
{'loss': 0.2939, 'grad_norm': 2.3294641971588135, 'learning_rate': 3.7426043821393675e-05, 'epoch': 1.88}

 19%|█▉        | 7771/41250 [18:46:11<80:55:30,  8.70s/it][2025-04-26 02:43:55,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 02:43:55,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.33 | bwd_microstep: 5714.04 | bwd_inner_microstep: 5701.27 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.14
[2025-04-26 02:43:55,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.33 | bwd: 5714.05 | bwd_inner: 5701.27 | bwd_allreduce: 12.74 | step: 19.14

 19%|█▉        | 7772/41250 [18:46:20<80:47:06,  8.69s/it]
                                                          
{'loss': 0.0616, 'grad_norm': 0.6529540419578552, 'learning_rate': 3.7425273135486994e-05, 'epoch': 1.88}

 19%|█▉        | 7772/41250 [18:46:20<80:47:06,  8.69s/it][2025-04-26 02:44:03,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:44:03,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5722.65 | bwd_inner_microstep: 5710.20 | bwd_allreduce_microstep: 12.41 | step_microstep: 17.91
[2025-04-26 02:44:03,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5722.66 | bwd_inner: 5710.20 | bwd_allreduce: 12.42 | step: 17.91

 19%|█▉        | 7773/41250 [18:46:28<80:41:16,  8.68s/it]
                                                          
{'loss': 0.0804, 'grad_norm': 2.4052631855010986, 'learning_rate': 3.742450234215694e-05, 'epoch': 1.88}

 19%|█▉        | 7773/41250 [18:46:29<80:41:16,  8.68s/it][2025-04-26 02:44:12,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 02:44:12,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5731.64 | bwd_inner_microstep: 5719.06 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.32
[2025-04-26 02:44:12,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5731.65 | bwd_inner: 5719.06 | bwd_allreduce: 12.55 | step: 18.32

 19%|█▉        | 7774/41250 [18:46:37<80:39:57,  8.67s/it]
                                                          
{'loss': 0.5138, 'grad_norm': 3.0079946517944336, 'learning_rate': 3.742373144140825e-05, 'epoch': 1.88}

 19%|█▉        | 7774/41250 [18:46:37<80:39:57,  8.67s/it][2025-04-26 02:44:21,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:44:21,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.41 | bwd_microstep: 5758.89 | bwd_inner_microstep: 5650.78 | bwd_allreduce_microstep: 108.06 | step_microstep: 18.68
[2025-04-26 02:44:21,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.41 | bwd: 5758.90 | bwd_inner: 5650.78 | bwd_allreduce: 108.08 | step: 18.68

 19%|█▉        | 7775/41250 [18:46:46<80:41:22,  8.68s/it]
                                                          
{'loss': 0.0713, 'grad_norm': 0.8787352442741394, 'learning_rate': 3.742296043324569e-05, 'epoch': 1.88}

 19%|█▉        | 7775/41250 [18:46:46<80:41:22,  8.68s/it][2025-04-26 02:44:29,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:44:29,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.49 | bwd_microstep: 5707.56 | bwd_inner_microstep: 5694.96 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.20
[2025-04-26 02:44:29,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.49 | bwd: 5707.58 | bwd_inner: 5694.96 | bwd_allreduce: 12.57 | step: 18.21

 19%|█▉        | 7776/41250 [18:46:54<80:34:53,  8.67s/it]
                                                          
{'loss': 0.3549, 'grad_norm': 2.1003475189208984, 'learning_rate': 3.742218931767401e-05, 'epoch': 1.89}

 19%|█▉        | 7776/41250 [18:46:54<80:34:53,  8.67s/it][2025-04-26 02:44:38,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:44:38,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5703.98 | bwd_inner_microstep: 5691.06 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.18
[2025-04-26 02:44:38,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5704.00 | bwd_inner: 5691.06 | bwd_allreduce: 12.90 | step: 18.18

 19%|█▉        | 7777/41250 [18:47:03<80:28:54,  8.66s/it]
                                                          
{'loss': 0.0688, 'grad_norm': 2.0911495685577393, 'learning_rate': 3.742141809469795e-05, 'epoch': 1.89}

 19%|█▉        | 7777/41250 [18:47:03<80:28:54,  8.66s/it][2025-04-26 02:44:46,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:44:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.32 | bwd_microstep: 5723.89 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 63.92 | step_microstep: 18.23
[2025-04-26 02:44:46,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.32 | bwd: 5723.91 | bwd_inner: 5659.93 | bwd_allreduce: 63.94 | step: 18.24

 19%|█▉        | 7778/41250 [18:47:12<80:25:19,  8.65s/it]
                                                          
{'loss': 0.0443, 'grad_norm': 0.387297123670578, 'learning_rate': 3.742064676432227e-05, 'epoch': 1.89}

 19%|█▉        | 7778/41250 [18:47:12<80:25:19,  8.65s/it][2025-04-26 02:44:55,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 02:44:55,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.59 | bwd_microstep: 5793.30 | bwd_inner_microstep: 5658.34 | bwd_allreduce_microstep: 134.91 | step_microstep: 18.90
[2025-04-26 02:44:55,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.59 | bwd: 5793.31 | bwd_inner: 5658.34 | bwd_allreduce: 134.93 | step: 18.91

 19%|█▉        | 7779/41250 [18:47:20<80:36:44,  8.67s/it]
                                                          
{'loss': 0.2009, 'grad_norm': 1.6343481540679932, 'learning_rate': 3.7419875326551736e-05, 'epoch': 1.89}

 19%|█▉        | 7779/41250 [18:47:20<80:36:44,  8.67s/it][2025-04-26 02:45:04,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 02:45:04,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.87 | bwd_microstep: 5913.27 | bwd_inner_microstep: 5643.56 | bwd_allreduce_microstep: 269.66 | step_microstep: 18.61
[2025-04-26 02:45:04,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.87 | bwd: 5913.29 | bwd_inner: 5643.56 | bwd_allreduce: 269.68 | step: 18.61

 19%|█▉        | 7780/41250 [18:47:29<81:02:44,  8.72s/it]
                                                          
{'loss': 0.0812, 'grad_norm': 0.655626654624939, 'learning_rate': 3.7419103781391096e-05, 'epoch': 1.89}

 19%|█▉        | 7780/41250 [18:47:29<81:02:44,  8.72s/it][2025-04-26 02:45:13,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 02:45:13,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.46 | bwd_microstep: 5768.05 | bwd_inner_microstep: 5699.37 | bwd_allreduce_microstep: 68.64 | step_microstep: 18.23
[2025-04-26 02:45:13,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.46 | bwd: 5768.06 | bwd_inner: 5699.37 | bwd_allreduce: 68.66 | step: 18.23

 19%|█▉        | 7781/41250 [18:47:38<81:00:18,  8.71s/it]
                                                          
{'loss': 0.4055, 'grad_norm': 2.9023361206054688, 'learning_rate': 3.741833212884511e-05, 'epoch': 1.89}

 19%|█▉        | 7781/41250 [18:47:38<81:00:18,  8.71s/it][2025-04-26 02:45:21,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 02:45:21,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.73 | bwd_microstep: 5696.25 | bwd_inner_microstep: 5654.83 | bwd_allreduce_microstep: 41.38 | step_microstep: 18.60
[2025-04-26 02:45:21,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.73 | bwd: 5696.26 | bwd_inner: 5654.83 | bwd_allreduce: 41.40 | step: 18.60

 19%|█▉        | 7782/41250 [18:47:47<80:43:41,  8.68s/it]
                                                          
{'loss': 0.2632, 'grad_norm': 2.7136268615722656, 'learning_rate': 3.741756036891853e-05, 'epoch': 1.89}

 19%|█▉        | 7782/41250 [18:47:47<80:43:41,  8.68s/it][2025-04-26 02:45:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 02:45:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.41 | bwd_microstep: 5782.06 | bwd_inner_microstep: 5650.55 | bwd_allreduce_microstep: 131.47 | step_microstep: 18.41
[2025-04-26 02:45:30,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.41 | bwd: 5782.08 | bwd_inner: 5650.55 | bwd_allreduce: 131.49 | step: 18.43

 19%|█▉        | 7783/41250 [18:47:55<80:45:48,  8.69s/it]
                                                          
{'loss': 0.1007, 'grad_norm': 2.297952890396118, 'learning_rate': 3.7416788501616114e-05, 'epoch': 1.89}

 19%|█▉        | 7783/41250 [18:47:55<80:45:48,  8.69s/it][2025-04-26 02:45:39,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 02:45:39,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.82 | bwd_microstep: 5776.18 | bwd_inner_microstep: 5646.33 | bwd_allreduce_microstep: 129.80 | step_microstep: 19.36
[2025-04-26 02:45:39,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.82 | bwd: 5776.20 | bwd_inner: 5646.33 | bwd_allreduce: 129.82 | step: 19.37

 19%|█▉        | 7784/41250 [18:48:04<80:46:50,  8.69s/it]
                                                          
{'loss': 0.1819, 'grad_norm': 1.7519114017486572, 'learning_rate': 3.7416016526942625e-05, 'epoch': 1.89}

 19%|█▉        | 7784/41250 [18:48:04<80:46:50,  8.69s/it][2025-04-26 02:45:47,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:45:47,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.73 | bwd_microstep: 5701.10 | bwd_inner_microstep: 5688.30 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.99
[2025-04-26 02:45:47,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.73 | bwd: 5701.12 | bwd_inner: 5688.29 | bwd_allreduce: 12.78 | step: 18.99

 19%|█▉        | 7785/41250 [18:48:13<80:37:32,  8.67s/it]
                                                          
{'loss': 0.0582, 'grad_norm': 1.085708737373352, 'learning_rate': 3.7415244444902816e-05, 'epoch': 1.89}

 19%|█▉        | 7785/41250 [18:48:13<80:37:32,  8.67s/it][2025-04-26 02:45:56,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 02:45:56,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.23 | bwd_microstep: 5775.64 | bwd_inner_microstep: 5636.21 | bwd_allreduce_microstep: 139.38 | step_microstep: 18.83
[2025-04-26 02:45:56,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.23 | bwd: 5775.65 | bwd_inner: 5636.21 | bwd_allreduce: 139.40 | step: 18.83

 19%|█▉        | 7786/41250 [18:48:21<80:39:08,  8.68s/it]
                                                          
{'loss': 0.0403, 'grad_norm': 1.038636565208435, 'learning_rate': 3.741447225550146e-05, 'epoch': 1.89}

 19%|█▉        | 7786/41250 [18:48:21<80:39:08,  8.68s/it][2025-04-26 02:46:05,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.92
[2025-04-26 02:46:05,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.02 | bwd_microstep: 5775.46 | bwd_inner_microstep: 5644.43 | bwd_allreduce_microstep: 130.99 | step_microstep: 18.71
[2025-04-26 02:46:05,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.02 | bwd: 5775.47 | bwd_inner: 5644.43 | bwd_allreduce: 131.00 | step: 18.71

 19%|█▉        | 7787/41250 [18:48:30<80:40:43,  8.68s/it]
                                                          
{'loss': 0.2458, 'grad_norm': 2.2757866382598877, 'learning_rate': 3.74136999587433e-05, 'epoch': 1.89}

 19%|█▉        | 7787/41250 [18:48:30<80:40:43,  8.68s/it][2025-04-26 02:46:13,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:46:13,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.64 | bwd_microstep: 5742.10 | bwd_inner_microstep: 5684.35 | bwd_allreduce_microstep: 57.70 | step_microstep: 18.57
[2025-04-26 02:46:13,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.64 | bwd: 5742.12 | bwd_inner: 5684.35 | bwd_allreduce: 57.72 | step: 18.57

 19%|█▉        | 7788/41250 [18:48:39<80:40:27,  8.68s/it]
                                                          
{'loss': 0.1962, 'grad_norm': 1.4725723266601562, 'learning_rate': 3.7412927554633104e-05, 'epoch': 1.89}

 19%|█▉        | 7788/41250 [18:48:39<80:40:27,  8.68s/it][2025-04-26 02:46:22,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 02:46:22,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.93 | bwd_microstep: 5704.14 | bwd_inner_microstep: 5649.53 | bwd_allreduce_microstep: 54.56 | step_microstep: 17.78
[2025-04-26 02:46:22,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.93 | bwd: 5704.15 | bwd_inner: 5649.53 | bwd_allreduce: 54.58 | step: 17.78

 19%|█▉        | 7789/41250 [18:48:47<80:30:05,  8.66s/it]
                                                          
{'loss': 0.0485, 'grad_norm': 0.7967332601547241, 'learning_rate': 3.741215504317564e-05, 'epoch': 1.89}

 19%|█▉        | 7789/41250 [18:48:47<80:30:05,  8.66s/it][2025-04-26 02:46:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:46:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5685.41 | bwd_inner_microstep: 5672.52 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.74
[2025-04-26 02:46:31,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5685.43 | bwd_inner: 5672.52 | bwd_allreduce: 12.87 | step: 18.74

 19%|█▉        | 7790/41250 [18:48:56<80:23:31,  8.65s/it]
                                                          
{'loss': 0.3384, 'grad_norm': 1.982440710067749, 'learning_rate': 3.741138242437566e-05, 'epoch': 1.89}

 19%|█▉        | 7790/41250 [18:48:56<80:23:31,  8.65s/it][2025-04-26 02:46:39,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:46:39,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.15 | bwd_microstep: 5742.85 | bwd_inner_microstep: 5645.75 | bwd_allreduce_microstep: 97.06 | step_microstep: 18.57
[2025-04-26 02:46:39,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.15 | bwd: 5742.86 | bwd_inner: 5645.75 | bwd_allreduce: 97.07 | step: 18.57

 19%|█▉        | 7791/41250 [18:49:05<80:26:00,  8.65s/it]
                                                          
{'loss': 0.2416, 'grad_norm': 3.9538021087646484, 'learning_rate': 3.741060969823793e-05, 'epoch': 1.89}

 19%|█▉        | 7791/41250 [18:49:05<80:26:00,  8.65s/it][2025-04-26 02:46:48,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-26 02:46:48,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.66 | bwd_microstep: 5766.02 | bwd_inner_microstep: 5753.44 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.93
[2025-04-26 02:46:48,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.66 | bwd: 5766.03 | bwd_inner: 5753.44 | bwd_allreduce: 12.55 | step: 18.93

 19%|█▉        | 7792/41250 [18:49:13<80:40:00,  8.68s/it]
                                                          
{'loss': 0.1008, 'grad_norm': 1.2669600248336792, 'learning_rate': 3.740983686476722e-05, 'epoch': 1.89}

 19%|█▉        | 7792/41250 [18:49:13<80:40:00,  8.68s/it][2025-04-26 02:46:57,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.82 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 02:46:57,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.86 | bwd_microstep: 5717.88 | bwd_inner_microstep: 5681.89 | bwd_allreduce_microstep: 35.95 | step_microstep: 17.63
[2025-04-26 02:46:57,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.86 | bwd: 5717.90 | bwd_inner: 5681.89 | bwd_allreduce: 35.97 | step: 17.63

 19%|█▉        | 7793/41250 [18:49:22<80:35:17,  8.67s/it]
                                                          
{'loss': 0.1365, 'grad_norm': 1.1043176651000977, 'learning_rate': 3.740906392396829e-05, 'epoch': 1.89}

 19%|█▉        | 7793/41250 [18:49:22<80:35:17,  8.67s/it][2025-04-26 02:47:05,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 02:47:05,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.53 | bwd_microstep: 5750.19 | bwd_inner_microstep: 5642.58 | bwd_allreduce_microstep: 107.57 | step_microstep: 18.95
[2025-04-26 02:47:05,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.53 | bwd: 5750.21 | bwd_inner: 5642.58 | bwd_allreduce: 107.58 | step: 18.95

 19%|█▉        | 7794/41250 [18:49:31<80:34:22,  8.67s/it]
                                                          
{'loss': 0.155, 'grad_norm': 3.0944344997406006, 'learning_rate': 3.740829087584591e-05, 'epoch': 1.89}

 19%|█▉        | 7794/41250 [18:49:31<80:34:22,  8.67s/it][2025-04-26 02:47:14,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:47:14,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.65 | bwd_microstep: 5715.21 | bwd_inner_microstep: 5702.43 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.91
[2025-04-26 02:47:14,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.65 | bwd: 5715.22 | bwd_inner: 5702.43 | bwd_allreduce: 12.75 | step: 18.92

 19%|█▉        | 7795/41250 [18:49:39<80:30:17,  8.66s/it]
                                                          
{'loss': 0.1256, 'grad_norm': 2.738616704940796, 'learning_rate': 3.740751772040483e-05, 'epoch': 1.89}

 19%|█▉        | 7795/41250 [18:49:39<80:30:17,  8.66s/it][2025-04-26 02:47:23,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 02:47:23,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.66 | bwd_microstep: 5871.29 | bwd_inner_microstep: 5636.90 | bwd_allreduce_microstep: 234.34 | step_microstep: 18.48
[2025-04-26 02:47:23,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.66 | bwd: 5871.30 | bwd_inner: 5636.90 | bwd_allreduce: 234.36 | step: 18.48

 19%|█▉        | 7796/41250 [18:49:48<80:48:51,  8.70s/it]
                                                          
{'loss': 0.2403, 'grad_norm': 2.2365972995758057, 'learning_rate': 3.740674445764984e-05, 'epoch': 1.89}

 19%|█▉        | 7796/41250 [18:49:48<80:48:51,  8.70s/it][2025-04-26 02:47:32,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 02:47:32,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.07 | bwd_microstep: 5857.95 | bwd_inner_microstep: 5644.23 | bwd_allreduce_microstep: 213.67 | step_microstep: 18.68
[2025-04-26 02:47:32,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.07 | bwd: 5857.96 | bwd_inner: 5644.23 | bwd_allreduce: 213.69 | step: 18.68

 19%|█▉        | 7797/41250 [18:49:57<81:01:08,  8.72s/it]
                                                          
{'loss': 0.154, 'grad_norm': 1.478623390197754, 'learning_rate': 3.740597108758568e-05, 'epoch': 1.89}

 19%|█▉        | 7797/41250 [18:49:57<81:01:08,  8.72s/it][2025-04-26 02:47:40,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:47:40,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.87 | bwd_microstep: 5695.21 | bwd_inner_microstep: 5682.57 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.25
[2025-04-26 02:47:40,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.87 | bwd: 5695.23 | bwd_inner: 5682.57 | bwd_allreduce: 12.62 | step: 18.25

 19%|█▉        | 7798/41250 [18:50:05<80:44:02,  8.69s/it]
                                                          
{'loss': 0.1146, 'grad_norm': 1.634359359741211, 'learning_rate': 3.740519761021714e-05, 'epoch': 1.89}

 19%|█▉        | 7798/41250 [18:50:05<80:44:02,  8.69s/it][2025-04-26 02:47:49,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:47:49,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.31 | bwd_microstep: 5738.68 | bwd_inner_microstep: 5696.57 | bwd_allreduce_microstep: 42.06 | step_microstep: 18.25
[2025-04-26 02:47:49,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.31 | bwd: 5738.69 | bwd_inner: 5696.57 | bwd_allreduce: 42.07 | step: 18.26

 19%|█▉        | 7799/41250 [18:50:14<80:42:43,  8.69s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 0.8654105067253113, 'learning_rate': 3.740442402554898e-05, 'epoch': 1.89}

 19%|█▉        | 7799/41250 [18:50:14<80:42:43,  8.69s/it][2025-04-26 02:47:57,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:47:57,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5700.70 | bwd_inner_microstep: 5687.74 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.58
[2025-04-26 02:47:57,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5700.72 | bwd_inner: 5687.74 | bwd_allreduce: 12.94 | step: 18.58

 19%|█▉        | 7800/41250 [18:50:23<80:32:42,  8.67s/it]
                                                          
{'loss': 0.2194, 'grad_norm': 4.828146934509277, 'learning_rate': 3.740365033358597e-05, 'epoch': 1.89}

 19%|█▉        | 7800/41250 [18:50:23<80:32:42,  8.67s/it][2025-04-26 02:48:06,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:48:06,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.00 | bwd_microstep: 5847.51 | bwd_inner_microstep: 5691.98 | bwd_allreduce_microstep: 155.49 | step_microstep: 17.81
[2025-04-26 02:48:06,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.00 | bwd: 5847.52 | bwd_inner: 5691.98 | bwd_allreduce: 155.50 | step: 17.81

 19%|█▉        | 7801/41250 [18:50:32<80:52:48,  8.70s/it]
                                                          
{'loss': 0.1193, 'grad_norm': 4.992996692657471, 'learning_rate': 3.7402876534332874e-05, 'epoch': 1.89}

 19%|█▉        | 7801/41250 [18:50:32<80:52:48,  8.70s/it][2025-04-26 02:48:15,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:48:15,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.83 | bwd_microstep: 5729.08 | bwd_inner_microstep: 5682.87 | bwd_allreduce_microstep: 46.16 | step_microstep: 17.97
[2025-04-26 02:48:15,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.83 | bwd: 5729.09 | bwd_inner: 5682.87 | bwd_allreduce: 46.17 | step: 17.97

 19%|█▉        | 7802/41250 [18:50:40<80:44:26,  8.69s/it]
                                                          
{'loss': 0.1019, 'grad_norm': 1.7656053304672241, 'learning_rate': 3.7402102627794465e-05, 'epoch': 1.89}

 19%|█▉        | 7802/41250 [18:50:40<80:44:26,  8.69s/it][2025-04-26 02:48:24,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 02:48:24,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5734.19 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 31.75 | step_microstep: 18.62
[2025-04-26 02:48:24,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5734.21 | bwd_inner: 5702.41 | bwd_allreduce: 31.76 | step: 18.63

 19%|█▉        | 7803/41250 [18:50:49<80:40:00,  8.68s/it]
                                                          
{'loss': 0.062, 'grad_norm': 1.057460904121399, 'learning_rate': 3.7401328613975525e-05, 'epoch': 1.89}

 19%|█▉        | 7803/41250 [18:50:49<80:40:00,  8.68s/it][2025-04-26 02:48:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:48:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.28 | bwd_microstep: 5894.24 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 241.74 | step_microstep: 18.31
[2025-04-26 02:48:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.28 | bwd: 5894.26 | bwd_inner: 5652.46 | bwd_allreduce: 241.75 | step: 18.32

 19%|█▉        | 7804/41250 [18:50:58<80:59:35,  8.72s/it]
                                                          
{'loss': 0.1404, 'grad_norm': 2.93847918510437, 'learning_rate': 3.740055449288081e-05, 'epoch': 1.89}

 19%|█▉        | 7804/41250 [18:50:58<80:59:35,  8.72s/it][2025-04-26 02:48:41,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 02:48:41,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.02 | bwd_microstep: 5772.00 | bwd_inner_microstep: 5657.43 | bwd_allreduce_microstep: 114.52 | step_microstep: 18.47
[2025-04-26 02:48:41,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.02 | bwd: 5772.01 | bwd_inner: 5657.43 | bwd_allreduce: 114.54 | step: 18.47

 19%|█▉        | 7805/41250 [18:51:06<80:53:31,  8.71s/it]
                                                          
{'loss': 0.1989, 'grad_norm': 2.0157461166381836, 'learning_rate': 3.739978026451511e-05, 'epoch': 1.89}

 19%|█▉        | 7805/41250 [18:51:06<80:53:31,  8.71s/it][2025-04-26 02:48:50,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 02:48:50,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.45 | bwd_microstep: 5716.80 | bwd_inner_microstep: 5637.29 | bwd_allreduce_microstep: 79.47 | step_microstep: 18.28
[2025-04-26 02:48:50,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.45 | bwd: 5716.81 | bwd_inner: 5637.29 | bwd_allreduce: 79.48 | step: 18.28

 19%|█▉        | 7806/41250 [18:51:15<80:39:08,  8.68s/it]
                                                          
{'loss': 0.3219, 'grad_norm': 2.7822883129119873, 'learning_rate': 3.739900592888317e-05, 'epoch': 1.89}

 19%|█▉        | 7806/41250 [18:51:15<80:39:08,  8.68s/it][2025-04-26 02:48:58,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:48:58,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.09 | bwd_microstep: 5760.29 | bwd_inner_microstep: 5654.36 | bwd_allreduce_microstep: 105.88 | step_microstep: 18.26
[2025-04-26 02:48:58,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.09 | bwd: 5760.30 | bwd_inner: 5654.36 | bwd_allreduce: 105.90 | step: 18.26

 19%|█▉        | 7807/41250 [18:51:24<80:35:49,  8.68s/it]
                                                          
{'loss': 0.1667, 'grad_norm': 1.2466368675231934, 'learning_rate': 3.739823148598979e-05, 'epoch': 1.89}

 19%|█▉        | 7807/41250 [18:51:24<80:35:49,  8.68s/it][2025-04-26 02:49:07,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:49:07,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.88 | bwd_microstep: 5720.24 | bwd_inner_microstep: 5647.94 | bwd_allreduce_microstep: 72.25 | step_microstep: 18.22
[2025-04-26 02:49:07,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.88 | bwd: 5720.25 | bwd_inner: 5647.94 | bwd_allreduce: 72.27 | step: 18.22

 19%|█▉        | 7808/41250 [18:51:32<80:27:05,  8.66s/it]
                                                          
{'loss': 0.0573, 'grad_norm': 0.7113448977470398, 'learning_rate': 3.739745693583974e-05, 'epoch': 1.89}

 19%|█▉        | 7808/41250 [18:51:32<80:27:05,  8.66s/it][2025-04-26 02:49:16,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.79 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:49:16,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.26 | bwd_microstep: 5704.78 | bwd_inner_microstep: 5692.06 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.66
[2025-04-26 02:49:16,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.26 | bwd: 5704.79 | bwd_inner: 5692.06 | bwd_allreduce: 12.70 | step: 17.66

 19%|█▉        | 7809/41250 [18:51:41<80:25:23,  8.66s/it]
                                                          
{'loss': 0.0728, 'grad_norm': 1.8661201000213623, 'learning_rate': 3.739668227843778e-05, 'epoch': 1.89}

 19%|█▉        | 7809/41250 [18:51:41<80:25:23,  8.66s/it][2025-04-26 02:49:24,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:49:24,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.66 | bwd_microstep: 5773.63 | bwd_inner_microstep: 5760.99 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.60
[2025-04-26 02:49:24,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.66 | bwd: 5773.64 | bwd_inner: 5760.99 | bwd_allreduce: 12.60 | step: 18.60

 19%|█▉        | 7810/41250 [18:51:50<80:40:22,  8.68s/it]
                                                          
{'loss': 0.0878, 'grad_norm': 1.2326395511627197, 'learning_rate': 3.7395907513788696e-05, 'epoch': 1.89}

 19%|█▉        | 7810/41250 [18:51:50<80:40:22,  8.68s/it][2025-04-26 02:49:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:49:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.62 | bwd_microstep: 5880.56 | bwd_inner_microstep: 5663.17 | bwd_allreduce_microstep: 217.34 | step_microstep: 18.49
[2025-04-26 02:49:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.62 | bwd: 5880.58 | bwd_inner: 5663.17 | bwd_allreduce: 217.36 | step: 18.49

 19%|█▉        | 7811/41250 [18:51:58<80:57:46,  8.72s/it]
                                                          
{'loss': 0.3369, 'grad_norm': 4.325037956237793, 'learning_rate': 3.7395132641897266e-05, 'epoch': 1.89}

 19%|█▉        | 7811/41250 [18:51:58<80:57:46,  8.72s/it][2025-04-26 02:49:42,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 02:49:42,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.95 | bwd_microstep: 5736.32 | bwd_inner_microstep: 5699.92 | bwd_allreduce_microstep: 36.35 | step_microstep: 18.87
[2025-04-26 02:49:42,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.95 | bwd: 5736.33 | bwd_inner: 5699.92 | bwd_allreduce: 36.37 | step: 18.88

 19%|█▉        | 7812/41250 [18:52:07<80:48:56,  8.70s/it]
                                                          
{'loss': 0.0492, 'grad_norm': 2.26255202293396, 'learning_rate': 3.739435766276827e-05, 'epoch': 1.89}

 19%|█▉        | 7812/41250 [18:52:07<80:48:56,  8.70s/it][2025-04-26 02:49:50,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 02:49:50,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5708.39 | bwd_inner_microstep: 5689.36 | bwd_allreduce_microstep: 18.98 | step_microstep: 18.12
[2025-04-26 02:49:50,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5708.40 | bwd_inner: 5689.36 | bwd_allreduce: 19.00 | step: 18.12

 19%|█▉        | 7813/41250 [18:52:16<80:38:04,  8.68s/it]
                                                          
{'loss': 0.1334, 'grad_norm': 2.203641891479492, 'learning_rate': 3.739358257640648e-05, 'epoch': 1.89}

 19%|█▉        | 7813/41250 [18:52:16<80:38:04,  8.68s/it][2025-04-26 02:49:59,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.99
[2025-04-26 02:49:59,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.57 | bwd_microstep: 5909.13 | bwd_inner_microstep: 5668.23 | bwd_allreduce_microstep: 240.86 | step_microstep: 18.17
[2025-04-26 02:49:59,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.57 | bwd: 5909.14 | bwd_inner: 5668.23 | bwd_allreduce: 240.87 | step: 18.18

 19%|█▉        | 7814/41250 [18:52:25<81:01:44,  8.72s/it]
                                                          
{'loss': 0.0509, 'grad_norm': 0.7801550030708313, 'learning_rate': 3.739280738281667e-05, 'epoch': 1.89}

 19%|█▉        | 7814/41250 [18:52:25<81:01:44,  8.72s/it][2025-04-26 02:50:08,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.99
[2025-04-26 02:50:08,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.50 | bwd_microstep: 5909.28 | bwd_inner_microstep: 5665.96 | bwd_allreduce_microstep: 243.28 | step_microstep: 18.31
[2025-04-26 02:50:08,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.50 | bwd: 5909.29 | bwd_inner: 5665.96 | bwd_allreduce: 243.29 | step: 18.31

 19%|█▉        | 7815/41250 [18:52:33<81:17:42,  8.75s/it]
                                                          
{'loss': 0.2537, 'grad_norm': 2.3311588764190674, 'learning_rate': 3.739203208200362e-05, 'epoch': 1.89}

 19%|█▉        | 7815/41250 [18:52:33<81:17:42,  8.75s/it][2025-04-26 02:50:17,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 02:50:17,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.66 | bwd_microstep: 5787.07 | bwd_inner_microstep: 5665.22 | bwd_allreduce_microstep: 121.80 | step_microstep: 18.41
[2025-04-26 02:50:17,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.66 | bwd: 5787.08 | bwd_inner: 5665.22 | bwd_allreduce: 121.82 | step: 18.41

 19%|█▉        | 7816/41250 [18:52:42<81:07:42,  8.74s/it]
                                                          
{'loss': 0.1072, 'grad_norm': 1.4336906671524048, 'learning_rate': 3.739125667397212e-05, 'epoch': 1.89}

 19%|█▉        | 7816/41250 [18:52:42<81:07:42,  8.74s/it][2025-04-26 02:50:25,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 02:50:25,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.00 | bwd_microstep: 5715.61 | bwd_inner_microstep: 5702.57 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.55
[2025-04-26 02:50:25,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.00 | bwd: 5715.62 | bwd_inner: 5702.57 | bwd_allreduce: 13.01 | step: 18.55

 19%|█▉        | 7817/41250 [18:52:51<80:54:31,  8.71s/it]
                                                          
{'loss': 0.1225, 'grad_norm': 2.370265245437622, 'learning_rate': 3.739048115872694e-05, 'epoch': 1.9}

 19%|█▉        | 7817/41250 [18:52:51<80:54:31,  8.71s/it][2025-04-26 02:50:34,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:50:34,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.44 | bwd_microstep: 5780.09 | bwd_inner_microstep: 5691.82 | bwd_allreduce_microstep: 88.22 | step_microstep: 18.27
[2025-04-26 02:50:34,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.44 | bwd: 5780.10 | bwd_inner: 5691.82 | bwd_allreduce: 88.24 | step: 18.27

 19%|█▉        | 7818/41250 [18:52:59<80:54:29,  8.71s/it]
                                                          
{'loss': 0.0956, 'grad_norm': 1.6837921142578125, 'learning_rate': 3.738970553627287e-05, 'epoch': 1.9}

 19%|█▉        | 7818/41250 [18:52:59<80:54:29,  8.71s/it][2025-04-26 02:50:43,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 02:50:43,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5778.64 | bwd_inner_microstep: 5652.45 | bwd_allreduce_microstep: 126.16 | step_microstep: 18.26
[2025-04-26 02:50:43,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.42 | bwd: 5778.66 | bwd_inner: 5652.45 | bwd_allreduce: 126.17 | step: 18.26

 19%|█▉        | 7819/41250 [18:53:08<80:53:18,  8.71s/it]
                                                          
{'loss': 0.2173, 'grad_norm': 1.857789158821106, 'learning_rate': 3.738892980661468e-05, 'epoch': 1.9}

 19%|█▉        | 7819/41250 [18:53:08<80:53:18,  8.71s/it][2025-04-26 02:50:52,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 02:50:52,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.84 | bwd_microstep: 5742.01 | bwd_inner_microstep: 5690.62 | bwd_allreduce_microstep: 51.34 | step_microstep: 18.65
[2025-04-26 02:50:52,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.84 | bwd: 5742.02 | bwd_inner: 5690.62 | bwd_allreduce: 51.36 | step: 18.65

 19%|█▉        | 7820/41250 [18:53:17<80:48:26,  8.70s/it]
                                                          
{'loss': 0.0788, 'grad_norm': 0.994238018989563, 'learning_rate': 3.738815396975715e-05, 'epoch': 1.9}

 19%|█▉        | 7820/41250 [18:53:17<80:48:26,  8.70s/it][2025-04-26 02:51:00,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.07 | optimizer_step: 1.17
[2025-04-26 02:51:00,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.89 | bwd_microstep: 5715.82 | bwd_inner_microstep: 5668.42 | bwd_allreduce_microstep: 47.36 | step_microstep: 19.66
[2025-04-26 02:51:00,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.89 | bwd: 5715.84 | bwd_inner: 5668.42 | bwd_allreduce: 47.38 | step: 19.67

 19%|█▉        | 7821/41250 [18:53:26<80:36:29,  8.68s/it]
                                                          
{'loss': 0.0314, 'grad_norm': 0.5286582708358765, 'learning_rate': 3.7387378025705085e-05, 'epoch': 1.9}

 19%|█▉        | 7821/41250 [18:53:26<80:36:29,  8.68s/it][2025-04-26 02:51:09,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:51:09,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.62 | bwd_microstep: 5710.88 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 12.92 | step_microstep: 19.05
[2025-04-26 02:51:09,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.62 | bwd: 5710.90 | bwd_inner: 5697.91 | bwd_allreduce: 12.94 | step: 19.05

 19%|█▉        | 7822/41250 [18:53:34<80:30:12,  8.67s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 1.7114256620407104, 'learning_rate': 3.738660197446325e-05, 'epoch': 1.9}

 19%|█▉        | 7822/41250 [18:53:34<80:30:12,  8.67s/it][2025-04-26 02:51:17,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:51:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.41 | bwd_microstep: 5709.45 | bwd_inner_microstep: 5697.17 | bwd_allreduce_microstep: 12.23 | step_microstep: 18.77
[2025-04-26 02:51:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.41 | bwd: 5709.46 | bwd_inner: 5697.17 | bwd_allreduce: 12.25 | step: 18.77

 19%|█▉        | 7823/41250 [18:53:43<80:26:28,  8.66s/it]
                                                          
{'loss': 0.0527, 'grad_norm': 0.9333617091178894, 'learning_rate': 3.738582581603643e-05, 'epoch': 1.9}

 19%|█▉        | 7823/41250 [18:53:43<80:26:28,  8.66s/it][2025-04-26 02:51:26,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:51:26,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.59 | bwd_microstep: 5764.95 | bwd_inner_microstep: 5691.82 | bwd_allreduce_microstep: 73.09 | step_microstep: 18.87
[2025-04-26 02:51:26,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.59 | bwd: 5764.96 | bwd_inner: 5691.82 | bwd_allreduce: 73.10 | step: 18.87

 19%|█▉        | 7824/41250 [18:53:51<80:31:35,  8.67s/it]
                                                          
{'loss': 0.3619, 'grad_norm': 3.1670925617218018, 'learning_rate': 3.7385049550429415e-05, 'epoch': 1.9}

 19%|█▉        | 7824/41250 [18:53:51<80:31:35,  8.67s/it][2025-04-26 02:51:35,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:51:35,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.64 | bwd_microstep: 5740.26 | bwd_inner_microstep: 5692.43 | bwd_allreduce_microstep: 47.79 | step_microstep: 18.58
[2025-04-26 02:51:35,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.64 | bwd: 5740.28 | bwd_inner: 5692.43 | bwd_allreduce: 47.80 | step: 18.58

 19%|█▉        | 7825/41250 [18:54:00<80:31:38,  8.67s/it]
                                                          
{'loss': 0.0934, 'grad_norm': 1.90762197971344, 'learning_rate': 3.7384273177646994e-05, 'epoch': 1.9}

 19%|█▉        | 7825/41250 [18:54:00<80:31:38,  8.67s/it][2025-04-26 02:51:44,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.06 | optimizer_step: 1.03
[2025-04-26 02:51:44,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5744.21 | bwd_inner_microstep: 5688.87 | bwd_allreduce_microstep: 55.29 | step_microstep: 19.27
[2025-04-26 02:51:44,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.75 | bwd: 5744.22 | bwd_inner: 5688.87 | bwd_allreduce: 55.31 | step: 19.27

 19%|█▉        | 7826/41250 [18:54:09<80:32:54,  8.68s/it]
                                                          
{'loss': 0.1957, 'grad_norm': 1.3773881196975708, 'learning_rate': 3.738349669769394e-05, 'epoch': 1.9}

 19%|█▉        | 7826/41250 [18:54:09<80:32:54,  8.68s/it][2025-04-26 02:51:52,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 02:51:52,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.06 | bwd_microstep: 5688.89 | bwd_inner_microstep: 5658.96 | bwd_allreduce_microstep: 29.88 | step_microstep: 18.85
[2025-04-26 02:51:52,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.06 | bwd: 5688.90 | bwd_inner: 5658.96 | bwd_allreduce: 29.89 | step: 18.86

 19%|█▉        | 7827/41250 [18:54:17<80:21:21,  8.66s/it]
                                                          
{'loss': 0.1056, 'grad_norm': 1.582063913345337, 'learning_rate': 3.7382720110575054e-05, 'epoch': 1.9}

 19%|█▉        | 7827/41250 [18:54:17<80:21:21,  8.66s/it][2025-04-26 02:52:01,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 02:52:01,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.64 | bwd_microstep: 5752.61 | bwd_inner_microstep: 5648.83 | bwd_allreduce_microstep: 103.73 | step_microstep: 18.66
[2025-04-26 02:52:01,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.64 | bwd: 5752.62 | bwd_inner: 5648.83 | bwd_allreduce: 103.76 | step: 18.67

 19%|█▉        | 7828/41250 [18:54:26<80:24:01,  8.66s/it]
                                                          
{'loss': 0.0151, 'grad_norm': 0.16523490846157074, 'learning_rate': 3.738194341629511e-05, 'epoch': 1.9}

 19%|█▉        | 7828/41250 [18:54:26<80:24:01,  8.66s/it][2025-04-26 02:52:10,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 02:52:10,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.53 | bwd_microstep: 5868.19 | bwd_inner_microstep: 5713.36 | bwd_allreduce_microstep: 154.78 | step_microstep: 19.49
[2025-04-26 02:52:10,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.53 | bwd: 5868.20 | bwd_inner: 5713.36 | bwd_allreduce: 154.80 | step: 19.49

 19%|█▉        | 7829/41250 [18:54:35<80:48:44,  8.70s/it]
                                                          
{'loss': 0.0686, 'grad_norm': 4.542836666107178, 'learning_rate': 3.738116661485891e-05, 'epoch': 1.9}

 19%|█▉        | 7829/41250 [18:54:35<80:48:44,  8.70s/it][2025-04-26 02:52:18,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:52:18,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5778.55 | bwd_inner_microstep: 5654.11 | bwd_allreduce_microstep: 124.40 | step_microstep: 18.56
[2025-04-26 02:52:18,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5778.57 | bwd_inner: 5654.11 | bwd_allreduce: 124.42 | step: 18.56

 19%|█▉        | 7830/41250 [18:54:44<80:44:54,  8.70s/it]
                                                          
{'loss': 0.0841, 'grad_norm': 2.6795032024383545, 'learning_rate': 3.738038970627124e-05, 'epoch': 1.9}

 19%|█▉        | 7830/41250 [18:54:44<80:44:54,  8.70s/it][2025-04-26 02:52:27,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 02:52:27,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.39 | bwd_microstep: 5707.50 | bwd_inner_microstep: 5650.88 | bwd_allreduce_microstep: 56.58 | step_microstep: 18.47
[2025-04-26 02:52:27,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.39 | bwd: 5707.52 | bwd_inner: 5650.88 | bwd_allreduce: 56.60 | step: 18.47

 19%|█▉        | 7831/41250 [18:54:52<80:30:51,  8.67s/it]
                                                          
{'loss': 0.1405, 'grad_norm': 2.6074435710906982, 'learning_rate': 3.737961269053688e-05, 'epoch': 1.9}

 19%|█▉        | 7831/41250 [18:54:52<80:30:51,  8.67s/it][2025-04-26 02:52:36,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 02:52:36,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.88 | bwd_microstep: 5784.92 | bwd_inner_microstep: 5641.55 | bwd_allreduce_microstep: 143.32 | step_microstep: 18.64
[2025-04-26 02:52:36,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.88 | bwd: 5784.94 | bwd_inner: 5641.56 | bwd_allreduce: 143.34 | step: 18.64

 19%|█▉        | 7832/41250 [18:55:01<80:33:52,  8.68s/it]
                                                          
{'loss': 0.208, 'grad_norm': 2.362736701965332, 'learning_rate': 3.737883556766063e-05, 'epoch': 1.9}

 19%|█▉        | 7832/41250 [18:55:01<80:33:52,  8.68s/it][2025-04-26 02:52:44,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:52:44,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.91 | bwd_microstep: 5760.88 | bwd_inner_microstep: 5674.44 | bwd_allreduce_microstep: 86.40 | step_microstep: 18.92
[2025-04-26 02:52:44,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.91 | bwd: 5760.90 | bwd_inner: 5674.44 | bwd_allreduce: 86.41 | step: 18.93

 19%|█▉        | 7833/41250 [18:55:10<80:35:10,  8.68s/it]
                                                          
{'loss': 0.177, 'grad_norm': 2.253438711166382, 'learning_rate': 3.7378058337647274e-05, 'epoch': 1.9}

 19%|█▉        | 7833/41250 [18:55:10<80:35:10,  8.68s/it][2025-04-26 02:52:53,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:52:53,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.13 | bwd_microstep: 5763.91 | bwd_inner_microstep: 5751.00 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.74
[2025-04-26 02:52:53,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.13 | bwd: 5763.92 | bwd_inner: 5751.00 | bwd_allreduce: 12.88 | step: 18.74

 19%|█▉        | 7834/41250 [18:55:18<80:40:59,  8.69s/it]
                                                          
{'loss': 0.077, 'grad_norm': 0.8875815868377686, 'learning_rate': 3.7377281000501606e-05, 'epoch': 1.9}

 19%|█▉        | 7834/41250 [18:55:18<80:40:59,  8.69s/it][2025-04-26 02:53:02,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.31 | optimizer_step: 1.03
[2025-04-26 02:53:02,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5751.21 | bwd_inner_microstep: 5646.60 | bwd_allreduce_microstep: 104.55 | step_microstep: 20.25
[2025-04-26 02:53:02,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5751.22 | bwd_inner: 5646.60 | bwd_allreduce: 104.57 | step: 20.25

 19%|█▉        | 7835/41250 [18:55:27<80:36:10,  8.68s/it]
                                                          
{'loss': 0.3269, 'grad_norm': 2.958845853805542, 'learning_rate': 3.737650355622842e-05, 'epoch': 1.9}

 19%|█▉        | 7835/41250 [18:55:27<80:36:10,  8.68s/it][2025-04-26 02:53:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:53:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.06 | bwd_microstep: 5774.18 | bwd_inner_microstep: 5645.50 | bwd_allreduce_microstep: 128.64 | step_microstep: 18.90
[2025-04-26 02:53:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.06 | bwd: 5774.19 | bwd_inner: 5645.50 | bwd_allreduce: 128.66 | step: 18.90

 19%|█▉        | 7836/41250 [18:55:36<80:35:29,  8.68s/it]
                                                          
{'loss': 0.0864, 'grad_norm': 1.009624719619751, 'learning_rate': 3.7375726004832506e-05, 'epoch': 1.9}

 19%|█▉        | 7836/41250 [18:55:36<80:35:29,  8.68s/it][2025-04-26 02:53:19,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-26 02:53:19,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5705.91 | bwd_inner_microstep: 5693.17 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.33
[2025-04-26 02:53:19,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5705.92 | bwd_inner: 5693.17 | bwd_allreduce: 12.71 | step: 19.33

 19%|█▉        | 7837/41250 [18:55:44<80:28:37,  8.67s/it]
                                                          
{'loss': 0.2965, 'grad_norm': 2.2630395889282227, 'learning_rate': 3.737494834631866e-05, 'epoch': 1.9}

 19%|█▉        | 7837/41250 [18:55:44<80:28:37,  8.67s/it][2025-04-26 02:53:28,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 02:53:28,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.76 | bwd_microstep: 5670.48 | bwd_inner_microstep: 5635.49 | bwd_allreduce_microstep: 34.95 | step_microstep: 19.05
[2025-04-26 02:53:28,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.76 | bwd: 5670.49 | bwd_inner: 5635.49 | bwd_allreduce: 34.96 | step: 19.05

 19%|█▉        | 7838/41250 [18:55:53<80:14:49,  8.65s/it]
                                                          
{'loss': 0.0739, 'grad_norm': 2.1472933292388916, 'learning_rate': 3.7374170580691674e-05, 'epoch': 1.9}

 19%|█▉        | 7838/41250 [18:55:53<80:14:49,  8.65s/it][2025-04-26 02:53:36,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 02:53:36,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.15 | bwd_microstep: 5685.15 | bwd_inner_microstep: 5629.97 | bwd_allreduce_microstep: 55.13 | step_microstep: 19.09
[2025-04-26 02:53:36,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.15 | bwd: 5685.16 | bwd_inner: 5629.97 | bwd_allreduce: 55.15 | step: 19.09

 19%|█▉        | 7839/41250 [18:56:01<80:04:52,  8.63s/it]
                                                          
{'loss': 0.0508, 'grad_norm': 0.9591458439826965, 'learning_rate': 3.7373392707956346e-05, 'epoch': 1.9}

 19%|█▉        | 7839/41250 [18:56:01<80:04:52,  8.63s/it][2025-04-26 02:53:45,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-26 02:53:45,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.35 | bwd_microstep: 5775.65 | bwd_inner_microstep: 5634.64 | bwd_allreduce_microstep: 140.96 | step_microstep: 19.53
[2025-04-26 02:53:45,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.35 | bwd: 5775.66 | bwd_inner: 5634.64 | bwd_allreduce: 140.98 | step: 19.53

 19%|█▉        | 7840/41250 [18:56:10<80:13:10,  8.64s/it]
                                                          
{'loss': 0.1812, 'grad_norm': 3.9073543548583984, 'learning_rate': 3.7372614728117463e-05, 'epoch': 1.9}

 19%|█▉        | 7840/41250 [18:56:10<80:13:10,  8.64s/it][2025-04-26 02:53:54,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 02:53:54,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.42 | bwd_microstep: 5762.88 | bwd_inner_microstep: 5646.45 | bwd_allreduce_microstep: 116.37 | step_microstep: 19.08
[2025-04-26 02:53:54,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.42 | bwd: 5762.89 | bwd_inner: 5646.45 | bwd_allreduce: 116.39 | step: 19.08

 19%|█▉        | 7841/41250 [18:56:19<80:18:23,  8.65s/it]
                                                          
{'loss': 0.2497, 'grad_norm': 1.7519707679748535, 'learning_rate': 3.7371836641179837e-05, 'epoch': 1.9}

 19%|█▉        | 7841/41250 [18:56:19<80:18:23,  8.65s/it][2025-04-26 02:54:02,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.00
[2025-04-26 02:54:02,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5770.08 | bwd_inner_microstep: 5651.93 | bwd_allreduce_microstep: 118.09 | step_microstep: 19.52
[2025-04-26 02:54:02,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5770.09 | bwd_inner: 5651.93 | bwd_allreduce: 118.12 | step: 19.52

 19%|█▉        | 7842/41250 [18:56:28<80:23:32,  8.66s/it]
                                                          
{'loss': 0.1601, 'grad_norm': 1.5731312036514282, 'learning_rate': 3.737105844714825e-05, 'epoch': 1.9}

 19%|█▉        | 7842/41250 [18:56:28<80:23:32,  8.66s/it][2025-04-26 02:54:11,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:54:11,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.26 | bwd_microstep: 5854.32 | bwd_inner_microstep: 5696.23 | bwd_allreduce_microstep: 158.04 | step_microstep: 18.50
[2025-04-26 02:54:11,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.26 | bwd: 5854.33 | bwd_inner: 5696.23 | bwd_allreduce: 158.06 | step: 18.50

 19%|█▉        | 7843/41250 [18:56:36<80:45:28,  8.70s/it]
                                                          
{'loss': 0.0978, 'grad_norm': 1.5212547779083252, 'learning_rate': 3.73702801460275e-05, 'epoch': 1.9}

 19%|█▉        | 7843/41250 [18:56:36<80:45:28,  8.70s/it][2025-04-26 02:54:20,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 02:54:20,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.44 | bwd_microstep: 5744.05 | bwd_inner_microstep: 5636.32 | bwd_allreduce_microstep: 107.68 | step_microstep: 18.68
[2025-04-26 02:54:20,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.44 | bwd: 5744.06 | bwd_inner: 5636.32 | bwd_allreduce: 107.70 | step: 18.68

 19%|█▉        | 7844/41250 [18:56:45<80:36:50,  8.69s/it]
                                                          
{'loss': 0.2212, 'grad_norm': 2.2539596557617188, 'learning_rate': 3.73695017378224e-05, 'epoch': 1.9}

 19%|█▉        | 7844/41250 [18:56:45<80:36:50,  8.69s/it][2025-04-26 02:54:28,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 02:54:28,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.18 | bwd_microstep: 5757.24 | bwd_inner_microstep: 5685.24 | bwd_allreduce_microstep: 71.94 | step_microstep: 19.20
[2025-04-26 02:54:28,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.18 | bwd: 5757.25 | bwd_inner: 5685.24 | bwd_allreduce: 71.96 | step: 19.20

 19%|█▉        | 7845/41250 [18:56:54<80:36:00,  8.69s/it]
                                                          
{'loss': 0.0602, 'grad_norm': 0.8322860598564148, 'learning_rate': 3.736872322253773e-05, 'epoch': 1.9}

 19%|█▉        | 7845/41250 [18:56:54<80:36:00,  8.69s/it][2025-04-26 02:54:37,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:54:37,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5752.17 | bwd_inner_microstep: 5685.27 | bwd_allreduce_microstep: 66.86 | step_microstep: 18.81
[2025-04-26 02:54:37,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5752.19 | bwd_inner: 5685.27 | bwd_allreduce: 66.88 | step: 18.82

 19%|█▉        | 7846/41250 [18:57:02<80:34:22,  8.68s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 1.7356455326080322, 'learning_rate': 3.73679446001783e-05, 'epoch': 1.9}

 19%|█▉        | 7846/41250 [18:57:02<80:34:22,  8.68s/it][2025-04-26 02:54:46,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:54:46,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.92 | bwd_microstep: 5702.56 | bwd_inner_microstep: 5689.84 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.78
[2025-04-26 02:54:46,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.92 | bwd: 5702.57 | bwd_inner: 5689.84 | bwd_allreduce: 12.69 | step: 18.78

 19%|█▉        | 7847/41250 [18:57:11<80:25:58,  8.67s/it]
                                                          
{'loss': 0.373, 'grad_norm': 5.280000686645508, 'learning_rate': 3.736716587074891e-05, 'epoch': 1.9}

 19%|█▉        | 7847/41250 [18:57:11<80:25:58,  8.67s/it][2025-04-26 02:54:54,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:54:54,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.66 | bwd_microstep: 5701.40 | bwd_inner_microstep: 5644.03 | bwd_allreduce_microstep: 57.33 | step_microstep: 18.77
[2025-04-26 02:54:54,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.66 | bwd: 5701.41 | bwd_inner: 5644.03 | bwd_allreduce: 57.34 | step: 18.77

 19%|█▉        | 7848/41250 [18:57:20<80:16:17,  8.65s/it]
                                                          
{'loss': 0.0533, 'grad_norm': 1.4301989078521729, 'learning_rate': 3.736638703425435e-05, 'epoch': 1.9}

 19%|█▉        | 7848/41250 [18:57:20<80:16:17,  8.65s/it][2025-04-26 02:55:03,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.03 | optimizer_step: 1.23
[2025-04-26 02:55:03,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.45 | bwd_microstep: 6033.56 | bwd_inner_microstep: 5701.49 | bwd_allreduce_microstep: 332.02 | step_microstep: 19.46
[2025-04-26 02:55:03,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.45 | bwd: 6033.57 | bwd_inner: 5701.49 | bwd_allreduce: 332.04 | step: 19.46

 19%|█▉        | 7849/41250 [18:57:29<81:08:44,  8.75s/it]
                                                          
{'loss': 0.2723, 'grad_norm': 2.922126293182373, 'learning_rate': 3.7365608090699436e-05, 'epoch': 1.9}

 19%|█▉        | 7849/41250 [18:57:29<81:08:44,  8.75s/it][2025-04-26 02:55:12,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:55:12,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.45 | bwd_microstep: 5781.20 | bwd_inner_microstep: 5642.53 | bwd_allreduce_microstep: 138.63 | step_microstep: 18.33
[2025-04-26 02:55:12,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.45 | bwd: 5781.22 | bwd_inner: 5642.53 | bwd_allreduce: 138.64 | step: 18.33

 19%|█▉        | 7850/41250 [18:57:37<80:59:38,  8.73s/it]
                                                          
{'loss': 0.0634, 'grad_norm': 2.153057098388672, 'learning_rate': 3.736482904008896e-05, 'epoch': 1.9}

 19%|█▉        | 7850/41250 [18:57:37<80:59:38,  8.73s/it][2025-04-26 02:55:21,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 02:55:21,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.78 | bwd_microstep: 5768.93 | bwd_inner_microstep: 5642.49 | bwd_allreduce_microstep: 126.39 | step_microstep: 18.81
[2025-04-26 02:55:21,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.78 | bwd: 5768.94 | bwd_inner: 5642.49 | bwd_allreduce: 126.41 | step: 18.81

 19%|█▉        | 7851/41250 [18:57:46<80:51:31,  8.72s/it]
                                                          
{'loss': 0.1486, 'grad_norm': 2.012498378753662, 'learning_rate': 3.7364049882427734e-05, 'epoch': 1.9}

 19%|█▉        | 7851/41250 [18:57:46<80:51:31,  8.72s/it][2025-04-26 02:55:29,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.12 | optimizer_step: 0.94
[2025-04-26 02:55:29,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.25 | bwd_microstep: 5888.46 | bwd_inner_microstep: 5651.24 | bwd_allreduce_microstep: 237.17 | step_microstep: 18.67
[2025-04-26 02:55:29,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.25 | bwd: 5888.47 | bwd_inner: 5651.24 | bwd_allreduce: 237.19 | step: 18.68

 19%|█▉        | 7852/41250 [18:57:55<81:06:11,  8.74s/it]
                                                          
{'loss': 0.0284, 'grad_norm': 1.0850852727890015, 'learning_rate': 3.736327061772055e-05, 'epoch': 1.9}

 19%|█▉        | 7852/41250 [18:57:55<81:06:11,  8.74s/it][2025-04-26 02:55:38,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:55:38,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.00 | bwd_microstep: 5783.04 | bwd_inner_microstep: 5663.86 | bwd_allreduce_microstep: 119.14 | step_microstep: 18.54
[2025-04-26 02:55:38,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.00 | bwd: 5783.06 | bwd_inner: 5663.86 | bwd_allreduce: 119.15 | step: 18.55

 19%|█▉        | 7853/41250 [18:58:03<80:58:30,  8.73s/it]
                                                          
{'loss': 0.2381, 'grad_norm': 2.7080068588256836, 'learning_rate': 3.7362491245972224e-05, 'epoch': 1.9}

 19%|█▉        | 7853/41250 [18:58:03<80:58:30,  8.73s/it][2025-04-26 02:55:47,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:55:47,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.94 | bwd_microstep: 5674.57 | bwd_inner_microstep: 5646.50 | bwd_allreduce_microstep: 28.02 | step_microstep: 18.83
[2025-04-26 02:55:47,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.95 | bwd: 5674.58 | bwd_inner: 5646.50 | bwd_allreduce: 28.04 | step: 18.83

 19%|█▉        | 7854/41250 [18:58:12<80:36:38,  8.69s/it]
                                                          
{'loss': 0.2632, 'grad_norm': 4.524087905883789, 'learning_rate': 3.736171176718755e-05, 'epoch': 1.9}

 19%|█▉        | 7854/41250 [18:58:12<80:36:38,  8.69s/it][2025-04-26 02:55:55,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 02:55:55,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5709.17 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 13.30 | step_microstep: 18.76
[2025-04-26 02:55:55,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5709.18 | bwd_inner: 5695.81 | bwd_allreduce: 13.33 | step: 18.76

 19%|█▉        | 7855/41250 [18:58:21<80:28:47,  8.68s/it]
                                                          
{'loss': 0.2729, 'grad_norm': 2.3465194702148438, 'learning_rate': 3.736093218137134e-05, 'epoch': 1.9}

 19%|█▉        | 7855/41250 [18:58:21<80:28:47,  8.68s/it][2025-04-26 02:56:04,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-26 02:56:04,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.44 | bwd_microstep: 5774.66 | bwd_inner_microstep: 5657.80 | bwd_allreduce_microstep: 116.82 | step_microstep: 18.72
[2025-04-26 02:56:04,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.44 | bwd: 5774.67 | bwd_inner: 5657.80 | bwd_allreduce: 116.83 | step: 18.72

 19%|█▉        | 7856/41250 [18:58:29<80:31:53,  8.68s/it]
                                                          
{'loss': 0.0409, 'grad_norm': 0.5965216755867004, 'learning_rate': 3.73601524885284e-05, 'epoch': 1.9}

 19%|█▉        | 7856/41250 [18:58:29<80:31:53,  8.68s/it][2025-04-26 02:56:13,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 02:56:13,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.78 | bwd_microstep: 5733.49 | bwd_inner_microstep: 5720.67 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.16
[2025-04-26 02:56:13,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.78 | bwd: 5733.50 | bwd_inner: 5720.67 | bwd_allreduce: 12.79 | step: 19.16

 19%|█▉        | 7857/41250 [18:58:38<80:30:24,  8.68s/it]
                                                          
{'loss': 0.0488, 'grad_norm': 0.9591202139854431, 'learning_rate': 3.7359372688663526e-05, 'epoch': 1.9}

 19%|█▉        | 7857/41250 [18:58:38<80:30:24,  8.68s/it][2025-04-26 02:56:21,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 02:56:21,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.04 | bwd_microstep: 5690.99 | bwd_inner_microstep: 5663.39 | bwd_allreduce_microstep: 27.54 | step_microstep: 19.01
[2025-04-26 02:56:21,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.04 | bwd: 5691.00 | bwd_inner: 5663.39 | bwd_allreduce: 27.57 | step: 19.02

 19%|█▉        | 7858/41250 [18:58:47<80:19:21,  8.66s/it]
                                                          
{'loss': 0.2223, 'grad_norm': 1.4870494604110718, 'learning_rate': 3.7358592781781544e-05, 'epoch': 1.9}

 19%|█▉        | 7858/41250 [18:58:47<80:19:21,  8.66s/it][2025-04-26 02:56:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:56:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.22 | bwd_microstep: 5779.49 | bwd_inner_microstep: 5638.51 | bwd_allreduce_microstep: 140.93 | step_microstep: 18.68
[2025-04-26 02:56:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.22 | bwd: 5779.50 | bwd_inner: 5638.51 | bwd_allreduce: 140.95 | step: 18.68

 19%|█▉        | 7859/41250 [18:58:55<80:23:51,  8.67s/it]
                                                          
{'loss': 0.096, 'grad_norm': 3.3918004035949707, 'learning_rate': 3.735781276788724e-05, 'epoch': 1.91}

 19%|█▉        | 7859/41250 [18:58:55<80:23:51,  8.67s/it][2025-04-26 02:56:39,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 02:56:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.63 | bwd_microstep: 5740.05 | bwd_inner_microstep: 5723.08 | bwd_allreduce_microstep: 16.93 | step_microstep: 18.76
[2025-04-26 02:56:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.63 | bwd: 5740.07 | bwd_inner: 5723.08 | bwd_allreduce: 16.94 | step: 18.77

 19%|█▉        | 7860/41250 [18:59:04<80:26:46,  8.67s/it]
                                                          
{'loss': 0.1094, 'grad_norm': 1.6689811944961548, 'learning_rate': 3.7357032646985445e-05, 'epoch': 1.91}

 19%|█▉        | 7860/41250 [18:59:04<80:26:46,  8.67s/it][2025-04-26 02:56:47,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 02:56:47,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.08 | bwd_microstep: 5739.40 | bwd_inner_microstep: 5711.18 | bwd_allreduce_microstep: 28.17 | step_microstep: 18.64
[2025-04-26 02:56:47,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5739.41 | bwd_inner: 5711.18 | bwd_allreduce: 28.19 | step: 18.64

 19%|█▉        | 7861/41250 [18:59:13<80:27:39,  8.68s/it]
                                                          
{'loss': 0.0141, 'grad_norm': 0.18269117176532745, 'learning_rate': 3.735625241908096e-05, 'epoch': 1.91}

 19%|█▉        | 7861/41250 [18:59:13<80:27:39,  8.68s/it][2025-04-26 02:56:56,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 02:56:56,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.19 | bwd_microstep: 5701.73 | bwd_inner_microstep: 5688.92 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.99
[2025-04-26 02:56:56,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5701.74 | bwd_inner: 5688.92 | bwd_allreduce: 12.78 | step: 18.99

 19%|█▉        | 7862/41250 [18:59:21<80:21:30,  8.66s/it]
                                                          
{'loss': 0.2421, 'grad_norm': 2.3027544021606445, 'learning_rate': 3.7355472084178577e-05, 'epoch': 1.91}

 19%|█▉        | 7862/41250 [18:59:21<80:21:30,  8.66s/it][2025-04-26 02:57:05,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 02:57:05,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.19 | bwd_microstep: 5763.64 | bwd_inner_microstep: 5660.10 | bwd_allreduce_microstep: 103.49 | step_microstep: 18.72
[2025-04-26 02:57:05,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.19 | bwd: 5763.65 | bwd_inner: 5660.10 | bwd_allreduce: 103.51 | step: 18.72

 19%|█▉        | 7863/41250 [18:59:30<80:25:02,  8.67s/it]
                                                          
{'loss': 0.0324, 'grad_norm': 0.6145033836364746, 'learning_rate': 3.7354691642283134e-05, 'epoch': 1.91}

 19%|█▉        | 7863/41250 [18:59:30<80:25:02,  8.67s/it][2025-04-26 02:57:13,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 02:57:13,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.93 | bwd_microstep: 5728.09 | bwd_inner_microstep: 5707.99 | bwd_allreduce_microstep: 20.05 | step_microstep: 18.97
[2025-04-26 02:57:13,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.93 | bwd: 5728.10 | bwd_inner: 5707.99 | bwd_allreduce: 20.07 | step: 18.97

 19%|█▉        | 7864/41250 [18:59:39<80:25:45,  8.67s/it]
                                                          
{'loss': 0.1977, 'grad_norm': 1.9769057035446167, 'learning_rate': 3.735391109339943e-05, 'epoch': 1.91}

 19%|█▉        | 7864/41250 [18:59:39<80:25:45,  8.67s/it][2025-04-26 02:57:22,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 02:57:22,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.09 | bwd_microstep: 5696.69 | bwd_inner_microstep: 5671.22 | bwd_allreduce_microstep: 25.42 | step_microstep: 18.74
[2025-04-26 02:57:22,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.09 | bwd: 5696.71 | bwd_inner: 5671.22 | bwd_allreduce: 25.44 | step: 18.74

 19%|█▉        | 7865/41250 [18:59:47<80:17:20,  8.66s/it]
                                                          
{'loss': 0.0841, 'grad_norm': 1.62043035030365, 'learning_rate': 3.7353130437532276e-05, 'epoch': 1.91}

 19%|█▉        | 7865/41250 [18:59:47<80:17:20,  8.66s/it][2025-04-26 02:57:31,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 02:57:31,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.69 | bwd_microstep: 5720.25 | bwd_inner_microstep: 5707.24 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.16
[2025-04-26 02:57:31,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.69 | bwd: 5720.26 | bwd_inner: 5707.24 | bwd_allreduce: 12.98 | step: 19.17

 19%|█▉        | 7866/41250 [18:59:56<80:18:12,  8.66s/it]
                                                          
{'loss': 0.0326, 'grad_norm': 1.0948899984359741, 'learning_rate': 3.735234967468649e-05, 'epoch': 1.91}

 19%|█▉        | 7866/41250 [18:59:56<80:18:12,  8.66s/it][2025-04-26 02:57:39,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 02:57:39,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.84 | bwd_microstep: 5739.41 | bwd_inner_microstep: 5700.47 | bwd_allreduce_microstep: 38.90 | step_microstep: 18.81
[2025-04-26 02:57:39,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.84 | bwd: 5739.43 | bwd_inner: 5700.47 | bwd_allreduce: 38.92 | step: 18.82

 19%|█▉        | 7867/41250 [19:00:05<80:20:43,  8.66s/it]
                                                          
{'loss': 0.0949, 'grad_norm': 2.093801736831665, 'learning_rate': 3.735156880486687e-05, 'epoch': 1.91}

 19%|█▉        | 7867/41250 [19:00:05<80:20:43,  8.66s/it][2025-04-26 02:57:48,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 02:57:48,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.48 | bwd_microstep: 5699.96 | bwd_inner_microstep: 5651.73 | bwd_allreduce_microstep: 48.19 | step_microstep: 18.68
[2025-04-26 02:57:48,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.48 | bwd: 5699.98 | bwd_inner: 5651.73 | bwd_allreduce: 48.21 | step: 18.68

 19%|█▉        | 7868/41250 [19:00:13<80:12:43,  8.65s/it]
                                                          
{'loss': 0.087, 'grad_norm': 2.89923357963562, 'learning_rate': 3.7350787828078247e-05, 'epoch': 1.91}

 19%|█▉        | 7868/41250 [19:00:13<80:12:43,  8.65s/it][2025-04-26 02:57:57,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 02:57:57,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5772.16 | bwd_inner_microstep: 5695.10 | bwd_allreduce_microstep: 77.02 | step_microstep: 19.89
[2025-04-26 02:57:57,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5772.18 | bwd_inner: 5695.10 | bwd_allreduce: 77.04 | step: 19.89

 19%|█▉        | 7869/41250 [19:00:22<80:22:01,  8.67s/it]
                                                          
{'loss': 0.1066, 'grad_norm': 2.04044508934021, 'learning_rate': 3.7350006744325435e-05, 'epoch': 1.91}

 19%|█▉        | 7869/41250 [19:00:22<80:22:01,  8.67s/it][2025-04-26 02:58:05,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 02:58:05,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.05 | bwd_microstep: 5774.91 | bwd_inner_microstep: 5639.56 | bwd_allreduce_microstep: 135.30 | step_microstep: 18.81
[2025-04-26 02:58:05,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.05 | bwd: 5774.92 | bwd_inner: 5639.56 | bwd_allreduce: 135.32 | step: 18.81

 19%|█▉        | 7870/41250 [19:00:31<80:23:37,  8.67s/it]
                                                          
{'loss': 0.3735, 'grad_norm': 2.2209160327911377, 'learning_rate': 3.734922555361324e-05, 'epoch': 1.91}

 19%|█▉        | 7870/41250 [19:00:31<80:23:37,  8.67s/it][2025-04-26 02:58:14,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 02:58:14,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.89 | bwd_microstep: 5774.70 | bwd_inner_microstep: 5657.11 | bwd_allreduce_microstep: 117.55 | step_microstep: 18.47
[2025-04-26 02:58:14,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.89 | bwd: 5774.71 | bwd_inner: 5657.10 | bwd_allreduce: 117.57 | step: 18.47

 19%|█▉        | 7871/41250 [19:00:39<80:26:13,  8.68s/it]
                                                          
{'loss': 0.1466, 'grad_norm': 1.7002633810043335, 'learning_rate': 3.734844425594648e-05, 'epoch': 1.91}

 19%|█▉        | 7871/41250 [19:00:39<80:26:13,  8.68s/it][2025-04-26 02:58:23,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-26 02:58:23,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.93 | bwd_microstep: 5753.83 | bwd_inner_microstep: 5652.38 | bwd_allreduce_microstep: 101.39 | step_microstep: 19.06
[2025-04-26 02:58:23,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.93 | bwd: 5753.84 | bwd_inner: 5652.39 | bwd_allreduce: 101.41 | step: 19.07

 19%|█▉        | 7872/41250 [19:00:48<80:25:22,  8.67s/it]
                                                          
{'loss': 0.1421, 'grad_norm': 1.94554603099823, 'learning_rate': 3.734766285132998e-05, 'epoch': 1.91}

 19%|█▉        | 7872/41250 [19:00:48<80:25:22,  8.67s/it][2025-04-26 02:58:31,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 02:58:31,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.83 | bwd_microstep: 5768.04 | bwd_inner_microstep: 5640.54 | bwd_allreduce_microstep: 127.45 | step_microstep: 18.55
[2025-04-26 02:58:31,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.83 | bwd: 5768.05 | bwd_inner: 5640.54 | bwd_allreduce: 127.47 | step: 18.55

 19%|█▉        | 7873/41250 [19:00:57<80:26:43,  8.68s/it]
                                                          
{'loss': 0.0669, 'grad_norm': 1.2356709241867065, 'learning_rate': 3.734688133976854e-05, 'epoch': 1.91}

 19%|█▉        | 7873/41250 [19:00:57<80:26:43,  8.68s/it][2025-04-26 02:58:40,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:58:40,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.63 | bwd_microstep: 5695.41 | bwd_inner_microstep: 5683.01 | bwd_allreduce_microstep: 12.35 | step_microstep: 18.59
[2025-04-26 02:58:40,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.63 | bwd: 5695.42 | bwd_inner: 5683.01 | bwd_allreduce: 12.37 | step: 18.59

 19%|█▉        | 7874/41250 [19:01:05<80:18:34,  8.66s/it]
                                                          
{'loss': 0.099, 'grad_norm': 1.8679242134094238, 'learning_rate': 3.7346099721266995e-05, 'epoch': 1.91}

 19%|█▉        | 7874/41250 [19:01:05<80:18:34,  8.66s/it][2025-04-26 02:58:49,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 02:58:49,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.62 | bwd_microstep: 5706.37 | bwd_inner_microstep: 5693.50 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.80
[2025-04-26 02:58:49,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.63 | bwd: 5706.38 | bwd_inner: 5693.50 | bwd_allreduce: 12.84 | step: 18.80

 19%|█▉        | 7875/41250 [19:01:14<80:13:36,  8.65s/it]
                                                          
{'loss': 0.2004, 'grad_norm': 4.163116931915283, 'learning_rate': 3.734531799583016e-05, 'epoch': 1.91}

 19%|█▉        | 7875/41250 [19:01:14<80:13:36,  8.65s/it][2025-04-26 02:58:57,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 02:58:57,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.82 | bwd_microstep: 5764.22 | bwd_inner_microstep: 5648.74 | bwd_allreduce_microstep: 115.43 | step_microstep: 18.69
[2025-04-26 02:58:57,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.82 | bwd: 5764.24 | bwd_inner: 5648.74 | bwd_allreduce: 115.45 | step: 18.69

 19%|█▉        | 7876/41250 [19:01:23<80:17:34,  8.66s/it]
                                                          
{'loss': 0.1251, 'grad_norm': 4.656129360198975, 'learning_rate': 3.734453616346285e-05, 'epoch': 1.91}

 19%|█▉        | 7876/41250 [19:01:23<80:17:34,  8.66s/it][2025-04-26 02:59:06,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 02:59:06,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.30 | bwd_microstep: 5694.14 | bwd_inner_microstep: 5634.93 | bwd_allreduce_microstep: 59.16 | step_microstep: 19.11
[2025-04-26 02:59:06,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.30 | bwd: 5694.16 | bwd_inner: 5634.93 | bwd_allreduce: 59.19 | step: 19.11

 19%|█▉        | 7877/41250 [19:01:31<80:09:15,  8.65s/it]
                                                          
{'loss': 0.1463, 'grad_norm': 1.0463299751281738, 'learning_rate': 3.734375422416988e-05, 'epoch': 1.91}

 19%|█▉        | 7877/41250 [19:01:31<80:09:15,  8.65s/it][2025-04-26 02:59:15,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 02:59:15,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.61 | bwd_microstep: 5751.37 | bwd_inner_microstep: 5640.50 | bwd_allreduce_microstep: 110.83 | step_microstep: 18.55
[2025-04-26 02:59:15,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.61 | bwd: 5751.39 | bwd_inner: 5640.50 | bwd_allreduce: 110.84 | step: 18.55

 19%|█▉        | 7878/41250 [19:01:40<80:12:12,  8.65s/it]
                                                          
{'loss': 0.179, 'grad_norm': 3.9737510681152344, 'learning_rate': 3.734297217795608e-05, 'epoch': 1.91}

 19%|█▉        | 7878/41250 [19:01:40<80:12:12,  8.65s/it][2025-04-26 02:59:23,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 02:59:23,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.64 | bwd_microstep: 5686.84 | bwd_inner_microstep: 5674.18 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.57
[2025-04-26 02:59:23,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.65 | bwd: 5686.85 | bwd_inner: 5674.18 | bwd_allreduce: 12.63 | step: 18.57

 19%|█▉        | 7879/41250 [19:01:49<80:04:56,  8.64s/it]
                                                          
{'loss': 0.1542, 'grad_norm': 1.6142910718917847, 'learning_rate': 3.734219002482627e-05, 'epoch': 1.91}

 19%|█▉        | 7879/41250 [19:01:49<80:04:56,  8.64s/it][2025-04-26 02:59:32,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 02:59:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.55 | bwd_microstep: 5684.64 | bwd_inner_microstep: 5636.01 | bwd_allreduce_microstep: 48.58 | step_microstep: 18.58
[2025-04-26 02:59:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.55 | bwd: 5684.65 | bwd_inner: 5636.01 | bwd_allreduce: 48.60 | step: 18.58

 19%|█▉        | 7880/41250 [19:01:57<79:56:42,  8.62s/it]
                                                          
{'loss': 0.3117, 'grad_norm': 5.144911766052246, 'learning_rate': 3.734140776478526e-05, 'epoch': 1.91}

 19%|█▉        | 7880/41250 [19:01:57<79:56:42,  8.62s/it][2025-04-26 02:59:41,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 02:59:41,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.40 | bwd_microstep: 5841.56 | bwd_inner_microstep: 5637.06 | bwd_allreduce_microstep: 204.46 | step_microstep: 18.79
[2025-04-26 02:59:41,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.40 | bwd: 5841.57 | bwd_inner: 5637.06 | bwd_allreduce: 204.47 | step: 18.79

 19%|█▉        | 7881/41250 [19:02:06<80:17:42,  8.66s/it]
                                                          
{'loss': 0.1289, 'grad_norm': 1.4631273746490479, 'learning_rate': 3.734062539783789e-05, 'epoch': 1.91}

 19%|█▉        | 7881/41250 [19:02:06<80:17:42,  8.66s/it][2025-04-26 02:59:49,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:59:49,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.60 | bwd_microstep: 5667.92 | bwd_inner_microstep: 5638.46 | bwd_allreduce_microstep: 29.42 | step_microstep: 18.33
[2025-04-26 02:59:49,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.60 | bwd: 5667.94 | bwd_inner: 5638.46 | bwd_allreduce: 29.43 | step: 18.34

 19%|█▉        | 7882/41250 [19:02:14<80:04:53,  8.64s/it]
                                                          
{'loss': 0.2742, 'grad_norm': 2.441119432449341, 'learning_rate': 3.733984292398898e-05, 'epoch': 1.91}

 19%|█▉        | 7882/41250 [19:02:14<80:04:53,  8.64s/it][2025-04-26 02:59:58,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 02:59:58,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5703.92 | bwd_inner_microstep: 5690.99 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.62
[2025-04-26 02:59:58,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5703.93 | bwd_inner: 5690.99 | bwd_allreduce: 12.90 | step: 18.63

 19%|█▉        | 7883/41250 [19:02:23<80:03:59,  8.64s/it]
                                                          
{'loss': 0.1631, 'grad_norm': 2.1868059635162354, 'learning_rate': 3.733906034324334e-05, 'epoch': 1.91}

 19%|█▉        | 7883/41250 [19:02:23<80:03:59,  8.64s/it][2025-04-26 03:00:06,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:00:06,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.42 | bwd_microstep: 5773.31 | bwd_inner_microstep: 5634.67 | bwd_allreduce_microstep: 138.60 | step_microstep: 18.43
[2025-04-26 03:00:06,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.42 | bwd: 5773.32 | bwd_inner: 5634.67 | bwd_allreduce: 138.61 | step: 18.44

 19%|█▉        | 7884/41250 [19:02:32<80:10:30,  8.65s/it]
                                                          
{'loss': 0.2225, 'grad_norm': 1.229912281036377, 'learning_rate': 3.733827765560581e-05, 'epoch': 1.91}

 19%|█▉        | 7884/41250 [19:02:32<80:10:30,  8.65s/it][2025-04-26 03:00:15,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:00:15,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.47 | bwd_microstep: 5772.90 | bwd_inner_microstep: 5643.08 | bwd_allreduce_microstep: 129.78 | step_microstep: 18.42
[2025-04-26 03:00:15,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.47 | bwd: 5772.91 | bwd_inner: 5643.08 | bwd_allreduce: 129.80 | step: 18.42

 19%|█▉        | 7885/41250 [19:02:40<80:14:08,  8.66s/it]
                                                          
{'loss': 0.2659, 'grad_norm': 1.5700414180755615, 'learning_rate': 3.733749486108121e-05, 'epoch': 1.91}

 19%|█▉        | 7885/41250 [19:02:40<80:14:08,  8.66s/it][2025-04-26 03:00:24,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.30 | optimizer_step: 1.05
[2025-04-26 03:00:24,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5738.10 | bwd_inner_microstep: 5697.90 | bwd_allreduce_microstep: 40.14 | step_microstep: 20.06
[2025-04-26 03:00:24,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5738.12 | bwd_inner: 5697.90 | bwd_allreduce: 40.17 | step: 20.06

 19%|█▉        | 7886/41250 [19:02:49<80:17:22,  8.66s/it]
                                                          
{'loss': 0.0201, 'grad_norm': 0.3797388970851898, 'learning_rate': 3.733671195967436e-05, 'epoch': 1.91}

 19%|█▉        | 7886/41250 [19:02:49<80:17:22,  8.66s/it][2025-04-26 03:00:32,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 03:00:32,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.14 | bwd_microstep: 5730.92 | bwd_inner_microstep: 5638.32 | bwd_allreduce_microstep: 92.56 | step_microstep: 19.46
[2025-04-26 03:00:32,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.14 | bwd: 5730.94 | bwd_inner: 5638.32 | bwd_allreduce: 92.58 | step: 19.46

 19%|█▉        | 7887/41250 [19:02:58<80:16:12,  8.66s/it]
                                                          
{'loss': 0.2755, 'grad_norm': 6.529089450836182, 'learning_rate': 3.733592895139008e-05, 'epoch': 1.91}

 19%|█▉        | 7887/41250 [19:02:58<80:16:12,  8.66s/it][2025-04-26 03:00:41,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 03:00:41,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.73 | bwd_microstep: 5749.67 | bwd_inner_microstep: 5695.79 | bwd_allreduce_microstep: 53.82 | step_microstep: 19.03
[2025-04-26 03:00:41,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.73 | bwd: 5749.68 | bwd_inner: 5695.79 | bwd_allreduce: 53.85 | step: 19.04

 19%|█▉        | 7888/41250 [19:03:06<80:22:13,  8.67s/it]
                                                          
{'loss': 0.2174, 'grad_norm': 1.9956766366958618, 'learning_rate': 3.733514583623323e-05, 'epoch': 1.91}

 19%|█▉        | 7888/41250 [19:03:06<80:22:13,  8.67s/it][2025-04-26 03:00:50,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.11 | optimizer_step: 0.90
[2025-04-26 03:00:50,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.80 | bwd_microstep: 5772.92 | bwd_inner_microstep: 5643.62 | bwd_allreduce_microstep: 129.25 | step_microstep: 19.18
[2025-04-26 03:00:50,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.80 | bwd: 5772.94 | bwd_inner: 5643.62 | bwd_allreduce: 129.27 | step: 19.18

 19%|█▉        | 7889/41250 [19:03:15<80:23:24,  8.67s/it]
                                                          
{'loss': 0.0567, 'grad_norm': 2.6411983966827393, 'learning_rate': 3.73343626142086e-05, 'epoch': 1.91}

 19%|█▉        | 7889/41250 [19:03:15<80:23:24,  8.67s/it][2025-04-26 03:00:59,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 03:00:59,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.03 | bwd_microstep: 5735.63 | bwd_inner_microstep: 5683.52 | bwd_allreduce_microstep: 52.06 | step_microstep: 18.95
[2025-04-26 03:00:59,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.03 | bwd: 5735.64 | bwd_inner: 5683.52 | bwd_allreduce: 52.08 | step: 18.95

 19%|█▉        | 7890/41250 [19:03:24<80:23:54,  8.68s/it]
                                                          
{'loss': 0.0561, 'grad_norm': 0.5917049646377563, 'learning_rate': 3.7333579285321036e-05, 'epoch': 1.91}

 19%|█▉        | 7890/41250 [19:03:24<80:23:54,  8.68s/it][2025-04-26 03:01:07,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 03:01:07,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.72 | bwd_microstep: 5692.34 | bwd_inner_microstep: 5652.13 | bwd_allreduce_microstep: 40.17 | step_microstep: 19.05
[2025-04-26 03:01:07,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.72 | bwd: 5692.35 | bwd_inner: 5652.13 | bwd_allreduce: 40.18 | step: 19.05

 19%|█▉        | 7891/41250 [19:03:32<80:11:29,  8.65s/it]
                                                          
{'loss': 0.1774, 'grad_norm': 3.5513319969177246, 'learning_rate': 3.733279584957536e-05, 'epoch': 1.91}

 19%|█▉        | 7891/41250 [19:03:32<80:11:29,  8.65s/it][2025-04-26 03:01:16,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-26 03:01:16,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.03 | bwd_microstep: 5767.78 | bwd_inner_microstep: 5641.68 | bwd_allreduce_microstep: 126.05 | step_microstep: 19.22
[2025-04-26 03:01:16,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.03 | bwd: 5767.80 | bwd_inner: 5641.68 | bwd_allreduce: 126.07 | step: 19.22

 19%|█▉        | 7892/41250 [19:03:41<80:19:18,  8.67s/it]
                                                          
{'loss': 0.0766, 'grad_norm': 0.9400420188903809, 'learning_rate': 3.733201230697642e-05, 'epoch': 1.91}

 19%|█▉        | 7892/41250 [19:03:41<80:19:18,  8.67s/it][2025-04-26 03:01:25,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.28 | optimizer_step: 0.90
[2025-04-26 03:01:25,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.61 | bwd_microstep: 5743.82 | bwd_inner_microstep: 5685.22 | bwd_allreduce_microstep: 58.55 | step_microstep: 19.68
[2025-04-26 03:01:25,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.61 | bwd: 5743.84 | bwd_inner: 5685.22 | bwd_allreduce: 58.57 | step: 19.68

 19%|█▉        | 7893/41250 [19:03:50<80:20:44,  8.67s/it]
                                                          
{'loss': 0.2119, 'grad_norm': 4.222671985626221, 'learning_rate': 3.7331228657529026e-05, 'epoch': 1.91}

 19%|█▉        | 7893/41250 [19:03:50<80:20:44,  8.67s/it][2025-04-26 03:01:33,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 03:01:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5706.44 | bwd_inner_microstep: 5693.81 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.18
[2025-04-26 03:01:33,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.86 | bwd: 5706.45 | bwd_inner: 5693.81 | bwd_allreduce: 12.60 | step: 19.19

 19%|█▉        | 7894/41250 [19:03:58<80:15:57,  8.66s/it]
                                                          
{'loss': 0.2438, 'grad_norm': 2.6121819019317627, 'learning_rate': 3.733044490123802e-05, 'epoch': 1.91}

 19%|█▉        | 7894/41250 [19:03:58<80:15:57,  8.66s/it][2025-04-26 03:01:42,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 03:01:42,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.71 | bwd_microstep: 5706.87 | bwd_inner_microstep: 5644.89 | bwd_allreduce_microstep: 61.93 | step_microstep: 18.81
[2025-04-26 03:01:42,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.71 | bwd: 5706.88 | bwd_inner: 5644.89 | bwd_allreduce: 61.95 | step: 18.81

 19%|█▉        | 7895/41250 [19:04:07<80:07:38,  8.65s/it]
                                                          
{'loss': 0.0567, 'grad_norm': 3.4520974159240723, 'learning_rate': 3.7329661038108225e-05, 'epoch': 1.91}

 19%|█▉        | 7895/41250 [19:04:07<80:07:38,  8.65s/it][2025-04-26 03:01:50,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:01:50,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5801.22 | bwd_inner_microstep: 5691.38 | bwd_allreduce_microstep: 109.80 | step_microstep: 18.77
[2025-04-26 03:01:50,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5801.23 | bwd_inner: 5691.38 | bwd_allreduce: 109.81 | step: 18.77

 19%|█▉        | 7896/41250 [19:04:16<80:20:02,  8.67s/it]
                                                          
{'loss': 0.1284, 'grad_norm': 1.1819169521331787, 'learning_rate': 3.732887706814448e-05, 'epoch': 1.91}

 19%|█▉        | 7896/41250 [19:04:16<80:20:02,  8.67s/it][2025-04-26 03:01:59,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:01:59,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.51 | bwd_microstep: 5794.03 | bwd_inner_microstep: 5642.38 | bwd_allreduce_microstep: 151.61 | step_microstep: 18.52
[2025-04-26 03:01:59,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.51 | bwd: 5794.04 | bwd_inner: 5642.38 | bwd_allreduce: 151.62 | step: 18.53

 19%|█▉        | 7897/41250 [19:04:25<80:25:41,  8.68s/it]
                                                          
{'loss': 0.1777, 'grad_norm': 3.607475757598877, 'learning_rate': 3.732809299135162e-05, 'epoch': 1.91}

 19%|█▉        | 7897/41250 [19:04:25<80:25:41,  8.68s/it][2025-04-26 03:02:08,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.97 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-26 03:02:08,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.59 | bwd_microstep: 5784.13 | bwd_inner_microstep: 5656.44 | bwd_allreduce_microstep: 127.65 | step_microstep: 18.19
[2025-04-26 03:02:08,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.59 | bwd: 5784.14 | bwd_inner: 5656.44 | bwd_allreduce: 127.67 | step: 18.20

 19%|█▉        | 7898/41250 [19:04:33<80:27:57,  8.69s/it]
                                                          
{'loss': 0.1238, 'grad_norm': 1.451407551765442, 'learning_rate': 3.732730880773447e-05, 'epoch': 1.91}

 19%|█▉        | 7898/41250 [19:04:33<80:27:57,  8.69s/it][2025-04-26 03:02:17,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 03:02:17,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5874.32 | bwd_inner_microstep: 5698.28 | bwd_allreduce_microstep: 175.99 | step_microstep: 18.36
[2025-04-26 03:02:17,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5874.34 | bwd_inner: 5698.28 | bwd_allreduce: 176.01 | step: 18.37

 19%|█▉        | 7899/41250 [19:04:42<80:48:43,  8.72s/it]
                                                          
{'loss': 0.1742, 'grad_norm': 0.9812632203102112, 'learning_rate': 3.732652451729787e-05, 'epoch': 1.91}

 19%|█▉        | 7899/41250 [19:04:42<80:48:43,  8.72s/it][2025-04-26 03:02:25,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-26 03:02:25,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5827.36 | bwd_inner_microstep: 5645.14 | bwd_allreduce_microstep: 182.17 | step_microstep: 18.91
[2025-04-26 03:02:25,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5827.37 | bwd_inner: 5645.14 | bwd_allreduce: 182.19 | step: 18.91

 19%|█▉        | 7900/41250 [19:04:51<80:50:33,  8.73s/it]
                                                          
{'loss': 0.3378, 'grad_norm': 2.7800614833831787, 'learning_rate': 3.732574012004665e-05, 'epoch': 1.92}

 19%|█▉        | 7900/41250 [19:04:51<80:50:33,  8.73s/it][2025-04-26 03:02:34,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:02:34,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.17 | bwd_microstep: 5715.79 | bwd_inner_microstep: 5656.63 | bwd_allreduce_microstep: 59.11 | step_microstep: 18.32
[2025-04-26 03:02:34,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.17 | bwd: 5715.80 | bwd_inner: 5656.63 | bwd_allreduce: 59.13 | step: 18.32

 19%|█▉        | 7901/41250 [19:04:59<80:33:00,  8.70s/it]
                                                          
{'loss': 0.1774, 'grad_norm': 1.8070062398910522, 'learning_rate': 3.732495561598565e-05, 'epoch': 1.92}

 19%|█▉        | 7901/41250 [19:04:59<80:33:00,  8.70s/it][2025-04-26 03:02:43,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 03:02:43,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.43 | bwd_microstep: 5726.83 | bwd_inner_microstep: 5651.63 | bwd_allreduce_microstep: 75.15 | step_microstep: 18.95
[2025-04-26 03:02:43,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.43 | bwd: 5726.84 | bwd_inner: 5651.63 | bwd_allreduce: 75.17 | step: 18.95

 19%|█▉        | 7902/41250 [19:05:08<80:22:53,  8.68s/it]
                                                          
{'loss': 0.0716, 'grad_norm': 1.2515931129455566, 'learning_rate': 3.7324171005119716e-05, 'epoch': 1.92}

 19%|█▉        | 7902/41250 [19:05:08<80:22:53,  8.68s/it][2025-04-26 03:02:51,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:02:51,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.45 | bwd_microstep: 5757.69 | bwd_inner_microstep: 5694.09 | bwd_allreduce_microstep: 63.56 | step_microstep: 18.85
[2025-04-26 03:02:51,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.45 | bwd: 5757.71 | bwd_inner: 5694.09 | bwd_allreduce: 63.58 | step: 18.85

 19%|█▉        | 7903/41250 [19:05:17<80:24:42,  8.68s/it]
                                                          
{'loss': 0.215, 'grad_norm': 1.8425160646438599, 'learning_rate': 3.732338628745367e-05, 'epoch': 1.92}

 19%|█▉        | 7903/41250 [19:05:17<80:24:42,  8.68s/it][2025-04-26 03:03:00,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 03:03:00,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.93 | bwd_microstep: 5780.49 | bwd_inner_microstep: 5767.68 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.98
[2025-04-26 03:03:00,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.93 | bwd: 5780.51 | bwd_inner: 5767.68 | bwd_allreduce: 12.78 | step: 18.98

 19%|█▉        | 7904/41250 [19:05:25<80:35:16,  8.70s/it]
                                                          
{'loss': 0.2034, 'grad_norm': 1.8444890975952148, 'learning_rate': 3.732260146299236e-05, 'epoch': 1.92}

 19%|█▉        | 7904/41250 [19:05:25<80:35:16,  8.70s/it][2025-04-26 03:03:09,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:03:09,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.67 | bwd_microstep: 5721.10 | bwd_inner_microstep: 5654.40 | bwd_allreduce_microstep: 66.65 | step_microstep: 18.81
[2025-04-26 03:03:09,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.67 | bwd: 5721.11 | bwd_inner: 5654.40 | bwd_allreduce: 66.67 | step: 18.81

 19%|█▉        | 7905/41250 [19:05:34<80:25:24,  8.68s/it]
                                                          
{'loss': 0.06, 'grad_norm': 0.7454511523246765, 'learning_rate': 3.732181653174061e-05, 'epoch': 1.92}

 19%|█▉        | 7905/41250 [19:05:34<80:25:24,  8.68s/it][2025-04-26 03:03:17,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:03:17,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.08 | bwd_microstep: 5717.09 | bwd_inner_microstep: 5653.99 | bwd_allreduce_microstep: 63.05 | step_microstep: 18.68
[2025-04-26 03:03:17,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.08 | bwd: 5717.10 | bwd_inner: 5653.99 | bwd_allreduce: 63.07 | step: 18.69

 19%|█▉        | 7906/41250 [19:05:43<80:17:24,  8.67s/it]
                                                          
{'loss': 0.1409, 'grad_norm': 1.1755908727645874, 'learning_rate': 3.732103149370327e-05, 'epoch': 1.92}

 19%|█▉        | 7906/41250 [19:05:43<80:17:24,  8.67s/it][2025-04-26 03:03:26,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:03:26,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.96 | bwd_microstep: 5768.23 | bwd_inner_microstep: 5657.35 | bwd_allreduce_microstep: 110.84 | step_microstep: 18.65
[2025-04-26 03:03:26,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5768.25 | bwd_inner: 5657.35 | bwd_allreduce: 110.85 | step: 18.65

 19%|█▉        | 7907/41250 [19:05:51<80:19:31,  8.67s/it]
                                                          
{'loss': 0.0642, 'grad_norm': 0.9501145482063293, 'learning_rate': 3.732024634888518e-05, 'epoch': 1.92}

 19%|█▉        | 7907/41250 [19:05:51<80:19:31,  8.67s/it][2025-04-26 03:03:35,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:03:35,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.30 | bwd_microstep: 5755.97 | bwd_inner_microstep: 5705.89 | bwd_allreduce_microstep: 50.04 | step_microstep: 18.51
[2025-04-26 03:03:35,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.30 | bwd: 5755.98 | bwd_inner: 5705.89 | bwd_allreduce: 50.05 | step: 18.52

 19%|█▉        | 7908/41250 [19:06:00<80:21:43,  8.68s/it]
                                                          
{'loss': 0.0674, 'grad_norm': 1.5506538152694702, 'learning_rate': 3.731946109729118e-05, 'epoch': 1.92}

 19%|█▉        | 7908/41250 [19:06:00<80:21:43,  8.68s/it][2025-04-26 03:03:43,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:03:43,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.19 | bwd_microstep: 5718.52 | bwd_inner_microstep: 5705.73 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.81
[2025-04-26 03:03:43,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5718.54 | bwd_inner: 5705.73 | bwd_allreduce: 12.76 | step: 18.81

 19%|█▉        | 7909/41250 [19:06:09<80:17:12,  8.67s/it]
                                                          
{'loss': 0.0481, 'grad_norm': 0.77467280626297, 'learning_rate': 3.731867573892611e-05, 'epoch': 1.92}

 19%|█▉        | 7909/41250 [19:06:09<80:17:12,  8.67s/it][2025-04-26 03:03:52,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 03:03:52,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.49 | bwd_microstep: 5765.75 | bwd_inner_microstep: 5650.96 | bwd_allreduce_microstep: 114.75 | step_microstep: 18.43
[2025-04-26 03:03:52,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.49 | bwd: 5765.76 | bwd_inner: 5650.96 | bwd_allreduce: 114.76 | step: 18.43

 19%|█▉        | 7910/41250 [19:06:17<80:18:00,  8.67s/it]
                                                          
{'loss': 0.1384, 'grad_norm': 1.3182562589645386, 'learning_rate': 3.7317890273794804e-05, 'epoch': 1.92}

 19%|█▉        | 7910/41250 [19:06:17<80:18:00,  8.67s/it][2025-04-26 03:04:01,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-26 03:04:01,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.80 | bwd_microstep: 5775.19 | bwd_inner_microstep: 5762.27 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.19
[2025-04-26 03:04:01,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.80 | bwd: 5775.20 | bwd_inner: 5762.27 | bwd_allreduce: 12.88 | step: 19.20

 19%|█▉        | 7911/41250 [19:06:26<80:28:55,  8.69s/it]
                                                          
{'loss': 0.157, 'grad_norm': 1.798346757888794, 'learning_rate': 3.731710470190212e-05, 'epoch': 1.92}

 19%|█▉        | 7911/41250 [19:06:26<80:28:55,  8.69s/it][2025-04-26 03:04:10,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.13 | optimizer_step: 1.00
[2025-04-26 03:04:10,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.83 | bwd_microstep: 5810.20 | bwd_inner_microstep: 5652.04 | bwd_allreduce_microstep: 158.11 | step_microstep: 19.43
[2025-04-26 03:04:10,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.83 | bwd: 5810.21 | bwd_inner: 5652.04 | bwd_allreduce: 158.13 | step: 19.43

 19%|█▉        | 7912/41250 [19:06:35<80:33:19,  8.70s/it]
                                                          
{'loss': 0.132, 'grad_norm': 1.1590380668640137, 'learning_rate': 3.731631902325288e-05, 'epoch': 1.92}

 19%|█▉        | 7912/41250 [19:06:35<80:33:19,  8.70s/it][2025-04-26 03:04:18,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:04:18,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.33 | bwd_microstep: 5780.18 | bwd_inner_microstep: 5640.03 | bwd_allreduce_microstep: 140.10 | step_microstep: 18.51
[2025-04-26 03:04:18,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.33 | bwd: 5780.19 | bwd_inner: 5640.03 | bwd_allreduce: 140.12 | step: 18.51

 19%|█▉        | 7913/41250 [19:06:44<80:30:00,  8.69s/it]
                                                          
{'loss': 0.0868, 'grad_norm': 0.9999778866767883, 'learning_rate': 3.731553323785194e-05, 'epoch': 1.92}

 19%|█▉        | 7913/41250 [19:06:44<80:30:00,  8.69s/it][2025-04-26 03:04:27,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:04:27,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.34 | bwd_microstep: 5727.54 | bwd_inner_microstep: 5631.23 | bwd_allreduce_microstep: 96.26 | step_microstep: 18.97
[2025-04-26 03:04:27,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.34 | bwd: 5727.56 | bwd_inner: 5631.23 | bwd_allreduce: 96.28 | step: 18.98

 19%|█▉        | 7914/41250 [19:06:52<80:19:18,  8.67s/it]
                                                          
{'loss': 0.1199, 'grad_norm': 1.7564719915390015, 'learning_rate': 3.731474734570416e-05, 'epoch': 1.92}

 19%|█▉        | 7914/41250 [19:06:52<80:19:18,  8.67s/it][2025-04-26 03:04:35,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 03:04:35,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5707.64 | bwd_inner_microstep: 5694.91 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.91
[2025-04-26 03:04:35,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.86 | bwd: 5707.66 | bwd_inner: 5694.91 | bwd_allreduce: 12.70 | step: 18.91

 19%|█▉        | 7915/41250 [19:07:01<80:13:10,  8.66s/it]
                                                          
{'loss': 0.1059, 'grad_norm': 3.0658795833587646, 'learning_rate': 3.7313961346814356e-05, 'epoch': 1.92}

 19%|█▉        | 7915/41250 [19:07:01<80:13:10,  8.66s/it][2025-04-26 03:04:44,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:04:44,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.22 | bwd_microstep: 5785.62 | bwd_inner_microstep: 5652.41 | bwd_allreduce_microstep: 133.15 | step_microstep: 18.89
[2025-04-26 03:04:44,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.22 | bwd: 5785.63 | bwd_inner: 5652.41 | bwd_allreduce: 133.18 | step: 18.89

 19%|█▉        | 7916/41250 [19:07:10<80:17:45,  8.67s/it]
                                                          
{'loss': 0.0619, 'grad_norm': 0.9082417488098145, 'learning_rate': 3.7313175241187386e-05, 'epoch': 1.92}

 19%|█▉        | 7916/41250 [19:07:10<80:17:45,  8.67s/it][2025-04-26 03:04:53,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 03:04:53,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.66 | bwd_microstep: 5784.28 | bwd_inner_microstep: 5642.00 | bwd_allreduce_microstep: 142.24 | step_microstep: 18.67
[2025-04-26 03:04:53,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.66 | bwd: 5784.30 | bwd_inner: 5642.00 | bwd_allreduce: 142.26 | step: 18.67

 19%|█▉        | 7917/41250 [19:07:18<80:20:40,  8.68s/it]
                                                          
{'loss': 0.0635, 'grad_norm': 1.8001476526260376, 'learning_rate': 3.731238902882809e-05, 'epoch': 1.92}

 19%|█▉        | 7917/41250 [19:07:18<80:20:40,  8.68s/it][2025-04-26 03:05:02,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 03:05:02,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.71 | bwd_microstep: 5710.01 | bwd_inner_microstep: 5696.89 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.93
[2025-04-26 03:05:02,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.71 | bwd: 5710.03 | bwd_inner: 5696.89 | bwd_allreduce: 13.09 | step: 18.93

 19%|█▉        | 7918/41250 [19:07:27<80:15:33,  8.67s/it]
                                                          
{'loss': 0.1539, 'grad_norm': 1.3480944633483887, 'learning_rate': 3.731160270974133e-05, 'epoch': 1.92}

 19%|█▉        | 7918/41250 [19:07:27<80:15:33,  8.67s/it][2025-04-26 03:05:10,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 03:05:10,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.22 | bwd_microstep: 5754.02 | bwd_inner_microstep: 5636.56 | bwd_allreduce_microstep: 117.42 | step_microstep: 18.47
[2025-04-26 03:05:10,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.23 | bwd: 5754.04 | bwd_inner: 5636.56 | bwd_allreduce: 117.44 | step: 18.47

 19%|█▉        | 7919/41250 [19:07:36<80:13:56,  8.67s/it]
                                                          
{'loss': 0.1666, 'grad_norm': 1.2967370748519897, 'learning_rate': 3.731081628393194e-05, 'epoch': 1.92}

 19%|█▉        | 7919/41250 [19:07:36<80:13:56,  8.67s/it][2025-04-26 03:05:19,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.12
[2025-04-26 03:05:19,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.94 | bwd_microstep: 5787.69 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 138.35 | step_microstep: 19.43
[2025-04-26 03:05:19,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.94 | bwd: 5787.70 | bwd_inner: 5649.29 | bwd_allreduce: 138.37 | step: 19.43

 19%|█▉        | 7920/41250 [19:07:44<80:19:00,  8.68s/it]
                                                          
{'loss': 0.2194, 'grad_norm': 0.869810163974762, 'learning_rate': 3.731002975140477e-05, 'epoch': 1.92}

 19%|█▉        | 7920/41250 [19:07:44<80:19:00,  8.68s/it][2025-04-26 03:05:28,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:05:28,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.07 | bwd_microstep: 5770.84 | bwd_inner_microstep: 5758.06 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.87
[2025-04-26 03:05:28,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.07 | bwd: 5770.86 | bwd_inner: 5758.06 | bwd_allreduce: 12.76 | step: 18.88

 19%|█▉        | 7921/41250 [19:07:53<80:28:51,  8.69s/it]
                                                          
{'loss': 0.0846, 'grad_norm': 1.3780314922332764, 'learning_rate': 3.730924311216468e-05, 'epoch': 1.92}

 19%|█▉        | 7921/41250 [19:07:53<80:28:51,  8.69s/it][2025-04-26 03:05:36,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:05:36,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.95 | bwd_microstep: 5704.59 | bwd_inner_microstep: 5636.74 | bwd_allreduce_microstep: 67.79 | step_microstep: 18.83
[2025-04-26 03:05:36,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.95 | bwd: 5704.60 | bwd_inner: 5636.74 | bwd_allreduce: 67.81 | step: 18.83

 19%|█▉        | 7922/41250 [19:08:02<80:14:51,  8.67s/it]
                                                          
{'loss': 0.2874, 'grad_norm': 1.7254716157913208, 'learning_rate': 3.73084563662165e-05, 'epoch': 1.92}

 19%|█▉        | 7922/41250 [19:08:02<80:14:51,  8.67s/it][2025-04-26 03:05:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:05:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.16 | bwd_microstep: 5681.07 | bwd_inner_microstep: 5643.21 | bwd_allreduce_microstep: 37.82 | step_microstep: 18.87
[2025-04-26 03:05:45,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.17 | bwd: 5681.08 | bwd_inner: 5643.21 | bwd_allreduce: 37.83 | step: 18.87

 19%|█▉        | 7923/41250 [19:08:10<80:00:56,  8.64s/it]
                                                          
{'loss': 0.3364, 'grad_norm': 1.4619330167770386, 'learning_rate': 3.7307669513565106e-05, 'epoch': 1.92}

 19%|█▉        | 7923/41250 [19:08:10<80:00:56,  8.64s/it][2025-04-26 03:05:53,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:05:53,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.68 | bwd_microstep: 5743.39 | bwd_inner_microstep: 5681.31 | bwd_allreduce_microstep: 62.03 | step_microstep: 18.79
[2025-04-26 03:05:53,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.68 | bwd: 5743.40 | bwd_inner: 5681.31 | bwd_allreduce: 62.05 | step: 18.80

 19%|█▉        | 7924/41250 [19:08:19<80:05:20,  8.65s/it]
                                                          
{'loss': 0.0769, 'grad_norm': 1.7622699737548828, 'learning_rate': 3.730688255421532e-05, 'epoch': 1.92}

 19%|█▉        | 7924/41250 [19:08:19<80:05:20,  8.65s/it][2025-04-26 03:06:02,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.08 | optimizer_step: 0.97
[2025-04-26 03:06:02,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5693.66 | bwd_inner_microstep: 5644.63 | bwd_allreduce_microstep: 48.98 | step_microstep: 19.28
[2025-04-26 03:06:02,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5693.67 | bwd_inner: 5644.63 | bwd_allreduce: 49.00 | step: 19.28

 19%|█▉        | 7925/41250 [19:08:27<79:56:51,  8.64s/it]
                                                          
{'loss': 0.0468, 'grad_norm': 0.5831716060638428, 'learning_rate': 3.7306095488172014e-05, 'epoch': 1.92}

 19%|█▉        | 7925/41250 [19:08:27<79:56:51,  8.64s/it][2025-04-26 03:06:11,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 03:06:11,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.70 | bwd_microstep: 5706.24 | bwd_inner_microstep: 5693.32 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.71
[2025-04-26 03:06:11,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.70 | bwd: 5706.25 | bwd_inner: 5693.32 | bwd_allreduce: 12.88 | step: 19.71

 19%|█▉        | 7926/41250 [19:08:36<79:58:21,  8.64s/it]
                                                          
{'loss': 0.1827, 'grad_norm': 1.321038842201233, 'learning_rate': 3.730530831544003e-05, 'epoch': 1.92}

 19%|█▉        | 7926/41250 [19:08:36<79:58:21,  8.64s/it][2025-04-26 03:06:19,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.05 | optimizer_step: 0.95
[2025-04-26 03:06:19,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.44 | bwd_microstep: 5677.30 | bwd_inner_microstep: 5633.15 | bwd_allreduce_microstep: 44.09 | step_microstep: 19.33
[2025-04-26 03:06:19,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.44 | bwd: 5677.31 | bwd_inner: 5633.15 | bwd_allreduce: 44.12 | step: 19.33

 19%|█▉        | 7927/41250 [19:08:45<79:48:19,  8.62s/it]
                                                          
{'loss': 0.3219, 'grad_norm': 6.648956298828125, 'learning_rate': 3.730452103602423e-05, 'epoch': 1.92}

 19%|█▉        | 7927/41250 [19:08:45<79:48:19,  8.62s/it][2025-04-26 03:06:28,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 03:06:28,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.70 | bwd_microstep: 5705.16 | bwd_inner_microstep: 5639.31 | bwd_allreduce_microstep: 65.81 | step_microstep: 19.40
[2025-04-26 03:06:28,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.70 | bwd: 5705.18 | bwd_inner: 5639.31 | bwd_allreduce: 65.82 | step: 19.41

 19%|█▉        | 7928/41250 [19:08:53<79:46:21,  8.62s/it]
                                                          
{'loss': 0.36, 'grad_norm': 3.318005084991455, 'learning_rate': 3.730373364992945e-05, 'epoch': 1.92}

 19%|█▉        | 7928/41250 [19:08:53<79:46:21,  8.62s/it][2025-04-26 03:06:37,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 03:06:37,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5764.58 | bwd_inner_microstep: 5645.90 | bwd_allreduce_microstep: 118.63 | step_microstep: 19.34
[2025-04-26 03:06:37,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5764.60 | bwd_inner: 5645.90 | bwd_allreduce: 118.65 | step: 19.34

 19%|█▉        | 7929/41250 [19:09:02<79:55:44,  8.64s/it]
                                                          
{'loss': 0.0881, 'grad_norm': 0.9827333092689514, 'learning_rate': 3.730294615716056e-05, 'epoch': 1.92}

 19%|█▉        | 7929/41250 [19:09:02<79:55:44,  8.64s/it][2025-04-26 03:06:45,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 03:06:45,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.55 | bwd_microstep: 5846.07 | bwd_inner_microstep: 5679.12 | bwd_allreduce_microstep: 166.89 | step_microstep: 19.25
[2025-04-26 03:06:45,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.55 | bwd: 5846.08 | bwd_inner: 5679.12 | bwd_allreduce: 166.92 | step: 19.25

 19%|█▉        | 7930/41250 [19:09:11<80:19:02,  8.68s/it]
                                                          
{'loss': 0.2453, 'grad_norm': 1.443005084991455, 'learning_rate': 3.7302158557722415e-05, 'epoch': 1.92}

 19%|█▉        | 7930/41250 [19:09:11<80:19:02,  8.68s/it][2025-04-26 03:06:54,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-26 03:06:54,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.65 | bwd_microstep: 5702.56 | bwd_inner_microstep: 5689.74 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.19
[2025-04-26 03:06:54,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.65 | bwd: 5702.58 | bwd_inner: 5689.74 | bwd_allreduce: 12.79 | step: 19.19

 19%|█▉        | 7931/41250 [19:09:19<80:11:49,  8.67s/it]
                                                          
{'loss': 0.0793, 'grad_norm': 2.276012420654297, 'learning_rate': 3.730137085161986e-05, 'epoch': 1.92}

 19%|█▉        | 7931/41250 [19:09:19<80:11:49,  8.67s/it][2025-04-26 03:07:03,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:07:03,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.86 | bwd_microstep: 5699.49 | bwd_inner_microstep: 5680.18 | bwd_allreduce_microstep: 19.26 | step_microstep: 19.22
[2025-04-26 03:07:03,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.86 | bwd: 5699.51 | bwd_inner: 5680.18 | bwd_allreduce: 19.28 | step: 19.22

 19%|█▉        | 7932/41250 [19:09:28<80:05:19,  8.65s/it]
                                                          
{'loss': 0.0806, 'grad_norm': 1.9377950429916382, 'learning_rate': 3.730058303885776e-05, 'epoch': 1.92}

 19%|█▉        | 7932/41250 [19:09:28<80:05:19,  8.65s/it][2025-04-26 03:07:11,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-26 03:07:11,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5694.73 | bwd_inner_microstep: 5656.87 | bwd_allreduce_microstep: 37.81 | step_microstep: 19.10
[2025-04-26 03:07:11,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5694.74 | bwd_inner: 5656.87 | bwd_allreduce: 37.83 | step: 19.10

 19%|█▉        | 7933/41250 [19:09:37<79:57:25,  8.64s/it]
                                                          
{'loss': 0.0568, 'grad_norm': 0.9302794337272644, 'learning_rate': 3.729979511944097e-05, 'epoch': 1.92}

 19%|█▉        | 7933/41250 [19:09:37<79:57:25,  8.64s/it][2025-04-26 03:07:20,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 03:07:20,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.06 | bwd_microstep: 5762.19 | bwd_inner_microstep: 5685.25 | bwd_allreduce_microstep: 76.88 | step_microstep: 19.24
[2025-04-26 03:07:20,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.06 | bwd: 5762.21 | bwd_inner: 5685.25 | bwd_allreduce: 76.91 | step: 19.24

 19%|█▉        | 7934/41250 [19:09:45<80:06:23,  8.66s/it]
                                                          
{'loss': 0.0664, 'grad_norm': 1.2978895902633667, 'learning_rate': 3.729900709337434e-05, 'epoch': 1.92}

 19%|█▉        | 7934/41250 [19:09:45<80:06:23,  8.66s/it][2025-04-26 03:07:29,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 03:07:29,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.28 | bwd_microstep: 5758.32 | bwd_inner_microstep: 5690.73 | bwd_allreduce_microstep: 67.53 | step_microstep: 19.21
[2025-04-26 03:07:29,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.28 | bwd: 5758.33 | bwd_inner: 5690.73 | bwd_allreduce: 67.55 | step: 19.22

 19%|█▉        | 7935/41250 [19:09:54<80:11:28,  8.67s/it]
                                                          
{'loss': 0.0931, 'grad_norm': 1.3641525506973267, 'learning_rate': 3.729821896066274e-05, 'epoch': 1.92}

 19%|█▉        | 7935/41250 [19:09:54<80:11:28,  8.67s/it][2025-04-26 03:07:37,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:07:37,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.67 | bwd_microstep: 5768.69 | bwd_inner_microstep: 5654.96 | bwd_allreduce_microstep: 113.69 | step_microstep: 18.70
[2025-04-26 03:07:37,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.67 | bwd: 5768.71 | bwd_inner: 5654.96 | bwd_allreduce: 113.71 | step: 18.70

 19%|█▉        | 7936/41250 [19:10:03<80:13:41,  8.67s/it]
                                                          
{'loss': 0.0904, 'grad_norm': 1.4645546674728394, 'learning_rate': 3.729743072131101e-05, 'epoch': 1.92}

 19%|█▉        | 7936/41250 [19:10:03<80:13:41,  8.67s/it][2025-04-26 03:07:46,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:07:46,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.14 | bwd_microstep: 5770.28 | bwd_inner_microstep: 5685.69 | bwd_allreduce_microstep: 84.55 | step_microstep: 18.70
[2025-04-26 03:07:46,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.14 | bwd: 5770.29 | bwd_inner: 5685.69 | bwd_allreduce: 84.56 | step: 18.70

 19%|█▉        | 7937/41250 [19:10:11<80:20:16,  8.68s/it]
                                                          
{'loss': 0.2087, 'grad_norm': 3.589425802230835, 'learning_rate': 3.7296642375324046e-05, 'epoch': 1.92}

 19%|█▉        | 7937/41250 [19:10:11<80:20:16,  8.68s/it][2025-04-26 03:07:55,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:07:55,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.48 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5655.86 | bwd_allreduce_microstep: 113.20 | step_microstep: 18.77
[2025-04-26 03:07:55,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.48 | bwd: 5769.11 | bwd_inner: 5655.86 | bwd_allreduce: 113.22 | step: 18.78

 19%|█▉        | 7938/41250 [19:10:20<80:19:52,  8.68s/it]
                                                          
{'loss': 0.2117, 'grad_norm': 1.3596364259719849, 'learning_rate': 3.729585392270666e-05, 'epoch': 1.92}

 19%|█▉        | 7938/41250 [19:10:20<80:19:52,  8.68s/it][2025-04-26 03:08:03,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:08:03,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.67 | bwd_microstep: 5713.50 | bwd_inner_microstep: 5701.01 | bwd_allreduce_microstep: 12.44 | step_microstep: 18.48
[2025-04-26 03:08:03,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.67 | bwd: 5713.51 | bwd_inner: 5701.01 | bwd_allreduce: 12.46 | step: 18.49

 19%|█▉        | 7939/41250 [19:10:29<80:12:28,  8.67s/it]
                                                          
{'loss': 0.0765, 'grad_norm': 3.0703814029693604, 'learning_rate': 3.729506536346375e-05, 'epoch': 1.92}

 19%|█▉        | 7939/41250 [19:10:29<80:12:28,  8.67s/it][2025-04-26 03:08:12,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:08:12,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.15 | bwd_microstep: 5754.07 | bwd_inner_microstep: 5692.79 | bwd_allreduce_microstep: 61.24 | step_microstep: 18.57
[2025-04-26 03:08:12,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5754.09 | bwd_inner: 5692.79 | bwd_allreduce: 61.25 | step: 18.57

 19%|█▉        | 7940/41250 [19:10:37<80:14:29,  8.67s/it]
                                                          
{'loss': 0.1183, 'grad_norm': 2.0854787826538086, 'learning_rate': 3.729427669760016e-05, 'epoch': 1.92}

 19%|█▉        | 7940/41250 [19:10:37<80:14:29,  8.67s/it][2025-04-26 03:08:21,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:08:21,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.55 | bwd_microstep: 5780.51 | bwd_inner_microstep: 5682.43 | bwd_allreduce_microstep: 98.04 | step_microstep: 18.34
[2025-04-26 03:08:21,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.55 | bwd: 5780.53 | bwd_inner: 5682.43 | bwd_allreduce: 98.06 | step: 18.34

 19%|█▉        | 7941/41250 [19:10:46<80:19:59,  8.68s/it]
                                                          
{'loss': 0.0815, 'grad_norm': 1.9177812337875366, 'learning_rate': 3.7293487925120754e-05, 'epoch': 1.93}

 19%|█▉        | 7941/41250 [19:10:46<80:19:59,  8.68s/it][2025-04-26 03:08:29,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 0.98
[2025-04-26 03:08:29,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.20 | bwd_microstep: 5728.53 | bwd_inner_microstep: 5658.54 | bwd_allreduce_microstep: 69.95 | step_microstep: 18.43
[2025-04-26 03:08:29,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.20 | bwd: 5728.54 | bwd_inner: 5658.54 | bwd_allreduce: 69.96 | step: 18.43

 19%|█▉        | 7942/41250 [19:10:55<80:11:53,  8.67s/it]
                                                          
{'loss': 0.1894, 'grad_norm': 2.014744997024536, 'learning_rate': 3.72926990460304e-05, 'epoch': 1.93}

 19%|█▉        | 7942/41250 [19:10:55<80:11:53,  8.67s/it][2025-04-26 03:08:38,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:08:38,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5705.43 | bwd_inner_microstep: 5692.80 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.27
[2025-04-26 03:08:38,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5705.44 | bwd_inner: 5692.80 | bwd_allreduce: 12.60 | step: 18.27

 19%|█▉        | 7943/41250 [19:11:03<80:06:01,  8.66s/it]
                                                          
{'loss': 0.1267, 'grad_norm': 2.4106807708740234, 'learning_rate': 3.7291910060333955e-05, 'epoch': 1.93}

 19%|█▉        | 7943/41250 [19:11:03<80:06:01,  8.66s/it][2025-04-26 03:08:47,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:08:47,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.90 | bwd_microstep: 5772.05 | bwd_inner_microstep: 5698.96 | bwd_allreduce_microstep: 73.05 | step_microstep: 18.59
[2025-04-26 03:08:47,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.90 | bwd: 5772.06 | bwd_inner: 5698.96 | bwd_allreduce: 73.07 | step: 18.59

 19%|█▉        | 7944/41250 [19:11:12<80:15:49,  8.68s/it]
                                                          
{'loss': 0.0336, 'grad_norm': 0.6657421588897705, 'learning_rate': 3.729112096803628e-05, 'epoch': 1.93}

 19%|█▉        | 7944/41250 [19:11:12<80:15:49,  8.68s/it][2025-04-26 03:08:55,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:08:55,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5754.04 | bwd_inner_microstep: 5695.41 | bwd_allreduce_microstep: 58.59 | step_microstep: 17.93
[2025-04-26 03:08:55,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5754.05 | bwd_inner: 5695.41 | bwd_allreduce: 58.60 | step: 17.93

 19%|█▉        | 7945/41250 [19:11:21<80:18:06,  8.68s/it]
                                                          
{'loss': 0.0312, 'grad_norm': 0.47408634424209595, 'learning_rate': 3.729033176914225e-05, 'epoch': 1.93}

 19%|█▉        | 7945/41250 [19:11:21<80:18:06,  8.68s/it][2025-04-26 03:09:04,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 03:09:04,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.69 | bwd_microstep: 5799.45 | bwd_inner_microstep: 5786.69 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.22
[2025-04-26 03:09:04,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.69 | bwd: 5799.46 | bwd_inner: 5786.69 | bwd_allreduce: 12.73 | step: 18.23

 19%|█▉        | 7946/41250 [19:11:29<80:32:37,  8.71s/it]
                                                          
{'loss': 0.3581, 'grad_norm': 3.2332303524017334, 'learning_rate': 3.728954246365673e-05, 'epoch': 1.93}

 19%|█▉        | 7946/41250 [19:11:29<80:32:37,  8.71s/it][2025-04-26 03:09:13,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:09:13,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.80 | bwd_microstep: 5715.11 | bwd_inner_microstep: 5702.50 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.30
[2025-04-26 03:09:13,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.80 | bwd: 5715.13 | bwd_inner: 5702.50 | bwd_allreduce: 12.59 | step: 18.30

 19%|█▉        | 7947/41250 [19:11:38<80:22:32,  8.69s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 0.6069901585578918, 'learning_rate': 3.7288753051584584e-05, 'epoch': 1.93}

 19%|█▉        | 7947/41250 [19:11:38<80:22:32,  8.69s/it][2025-04-26 03:09:21,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:09:21,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.12 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5645.21 | bwd_allreduce_microstep: 62.30 | step_microstep: 18.13
[2025-04-26 03:09:21,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.12 | bwd: 5707.56 | bwd_inner: 5645.21 | bwd_allreduce: 62.31 | step: 18.13

 19%|█▉        | 7948/41250 [19:11:47<80:09:45,  8.67s/it]
                                                          
{'loss': 0.1015, 'grad_norm': 1.5355932712554932, 'learning_rate': 3.7287963532930665e-05, 'epoch': 1.93}

 19%|█▉        | 7948/41250 [19:11:47<80:09:45,  8.67s/it][2025-04-26 03:09:30,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:09:30,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.17 | bwd_microstep: 5780.90 | bwd_inner_microstep: 5656.93 | bwd_allreduce_microstep: 123.91 | step_microstep: 18.80
[2025-04-26 03:09:30,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.17 | bwd: 5780.92 | bwd_inner: 5656.93 | bwd_allreduce: 123.93 | step: 18.80

 19%|█▉        | 7949/41250 [19:11:55<80:14:57,  8.68s/it]
                                                          
{'loss': 0.0666, 'grad_norm': 1.1709758043289185, 'learning_rate': 3.728717390769986e-05, 'epoch': 1.93}

 19%|█▉        | 7949/41250 [19:11:55<80:14:57,  8.68s/it][2025-04-26 03:09:39,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:09:39,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.74 | bwd_microstep: 5795.54 | bwd_inner_microstep: 5663.16 | bwd_allreduce_microstep: 132.34 | step_microstep: 18.38
[2025-04-26 03:09:39,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.74 | bwd: 5795.56 | bwd_inner: 5663.16 | bwd_allreduce: 132.36 | step: 18.38

 19%|█▉        | 7950/41250 [19:12:04<80:19:33,  8.68s/it]
                                                          
{'loss': 0.2656, 'grad_norm': 2.4866485595703125, 'learning_rate': 3.728638417589702e-05, 'epoch': 1.93}

 19%|█▉        | 7950/41250 [19:12:04<80:19:33,  8.68s/it][2025-04-26 03:09:47,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 03:09:48,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.21 | bwd_microstep: 5763.97 | bwd_inner_microstep: 5655.94 | bwd_allreduce_microstep: 107.98 | step_microstep: 18.93
[2025-04-26 03:09:48,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.21 | bwd: 5763.98 | bwd_inner: 5655.94 | bwd_allreduce: 108.00 | step: 18.93

 19%|█▉        | 7951/41250 [19:12:13<80:17:41,  8.68s/it]
                                                          
{'loss': 0.0881, 'grad_norm': 1.0738067626953125, 'learning_rate': 3.728559433752703e-05, 'epoch': 1.93}

 19%|█▉        | 7951/41250 [19:12:13<80:17:41,  8.68s/it][2025-04-26 03:09:56,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:09:56,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.07 | bwd_microstep: 5770.74 | bwd_inner_microstep: 5662.43 | bwd_allreduce_microstep: 108.26 | step_microstep: 18.78
[2025-04-26 03:09:56,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.07 | bwd: 5770.75 | bwd_inner: 5662.43 | bwd_allreduce: 108.28 | step: 18.78

 19%|█▉        | 7952/41250 [19:12:22<80:18:05,  8.68s/it]
                                                          
{'loss': 0.0178, 'grad_norm': 0.33761176466941833, 'learning_rate': 3.7284804392594747e-05, 'epoch': 1.93}

 19%|█▉        | 7952/41250 [19:12:22<80:18:05,  8.68s/it][2025-04-26 03:10:05,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.22 | optimizer_step: 0.94
[2025-04-26 03:10:05,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5769.53 | bwd_inner_microstep: 5654.96 | bwd_allreduce_microstep: 114.52 | step_microstep: 19.68
[2025-04-26 03:10:05,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5769.54 | bwd_inner: 5654.96 | bwd_allreduce: 114.54 | step: 19.68

 19%|█▉        | 7953/41250 [19:12:30<80:17:57,  8.68s/it]
                                                          
{'loss': 0.1516, 'grad_norm': 2.1915817260742188, 'learning_rate': 3.728401434110504e-05, 'epoch': 1.93}

 19%|█▉        | 7953/41250 [19:12:30<80:17:57,  8.68s/it][2025-04-26 03:10:14,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:10:14,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.21 | bwd_microstep: 5772.20 | bwd_inner_microstep: 5649.09 | bwd_allreduce_microstep: 123.06 | step_microstep: 18.43
[2025-04-26 03:10:14,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.21 | bwd: 5772.21 | bwd_inner: 5649.09 | bwd_allreduce: 123.08 | step: 18.43

 19%|█▉        | 7954/41250 [19:12:39<80:19:13,  8.68s/it]
                                                          
{'loss': 0.2289, 'grad_norm': 3.0541951656341553, 'learning_rate': 3.7283224183062795e-05, 'epoch': 1.93}

 19%|█▉        | 7954/41250 [19:12:39<80:19:13,  8.68s/it][2025-04-26 03:10:22,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 03:10:22,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.34 | bwd_microstep: 5707.98 | bwd_inner_microstep: 5689.08 | bwd_allreduce_microstep: 18.86 | step_microstep: 18.79
[2025-04-26 03:10:22,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.34 | bwd: 5707.99 | bwd_inner: 5689.08 | bwd_allreduce: 18.88 | step: 18.79

 19%|█▉        | 7955/41250 [19:12:48<80:11:29,  8.67s/it]
                                                          
{'loss': 0.1234, 'grad_norm': 2.425520896911621, 'learning_rate': 3.728243391847286e-05, 'epoch': 1.93}

 19%|█▉        | 7955/41250 [19:12:48<80:11:29,  8.67s/it][2025-04-26 03:10:31,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.32 | optimizer_step: 1.03
[2025-04-26 03:10:31,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.59 | bwd_microstep: 5694.85 | bwd_inner_microstep: 5681.17 | bwd_allreduce_microstep: 13.62 | step_microstep: 19.98
[2025-04-26 03:10:31,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.59 | bwd: 5694.87 | bwd_inner: 5681.17 | bwd_allreduce: 13.65 | step: 19.98

 19%|█▉        | 7956/41250 [19:12:56<80:04:37,  8.66s/it]
                                                          
{'loss': 0.1787, 'grad_norm': 2.1453001499176025, 'learning_rate': 3.728164354734013e-05, 'epoch': 1.93}

 19%|█▉        | 7956/41250 [19:12:56<80:04:37,  8.66s/it][2025-04-26 03:10:39,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 03:10:39,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.15 | bwd_microstep: 5707.43 | bwd_inner_microstep: 5635.36 | bwd_allreduce_microstep: 72.03 | step_microstep: 18.80
[2025-04-26 03:10:39,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.15 | bwd: 5707.44 | bwd_inner: 5635.36 | bwd_allreduce: 72.04 | step: 18.81

 19%|█▉        | 7957/41250 [19:13:05<79:57:24,  8.65s/it]
                                                          
{'loss': 0.1238, 'grad_norm': 2.6602509021759033, 'learning_rate': 3.728085306966946e-05, 'epoch': 1.93}

 19%|█▉        | 7957/41250 [19:13:05<79:57:24,  8.65s/it][2025-04-26 03:10:48,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:10:48,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.25 | bwd_microstep: 5698.76 | bwd_inner_microstep: 5686.21 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.84
[2025-04-26 03:10:48,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.25 | bwd: 5698.77 | bwd_inner: 5686.21 | bwd_allreduce: 12.52 | step: 18.85

 19%|█▉        | 7958/41250 [19:13:13<79:54:15,  8.64s/it]
                                                          
{'loss': 0.1629, 'grad_norm': 2.0360188484191895, 'learning_rate': 3.728006248546573e-05, 'epoch': 1.93}

 19%|█▉        | 7958/41250 [19:13:13<79:54:15,  8.64s/it][2025-04-26 03:10:57,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:10:57,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.34 | bwd_microstep: 5713.37 | bwd_inner_microstep: 5700.47 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.94
[2025-04-26 03:10:57,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.34 | bwd: 5713.39 | bwd_inner: 5700.47 | bwd_allreduce: 12.88 | step: 18.95

 19%|█▉        | 7959/41250 [19:13:22<79:55:31,  8.64s/it]
                                                          
{'loss': 0.0838, 'grad_norm': 1.1416176557540894, 'learning_rate': 3.7279271794733814e-05, 'epoch': 1.93}

 19%|█▉        | 7959/41250 [19:13:22<79:55:31,  8.64s/it][2025-04-26 03:11:05,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:11:05,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.45 | bwd_microstep: 5731.96 | bwd_inner_microstep: 5697.14 | bwd_allreduce_microstep: 34.78 | step_microstep: 18.85
[2025-04-26 03:11:05,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.46 | bwd: 5731.97 | bwd_inner: 5697.14 | bwd_allreduce: 34.80 | step: 18.86

 19%|█▉        | 7960/41250 [19:13:31<79:57:55,  8.65s/it]
                                                          
{'loss': 0.1522, 'grad_norm': 2.971947431564331, 'learning_rate': 3.727848099747859e-05, 'epoch': 1.93}

 19%|█▉        | 7960/41250 [19:13:31<79:57:55,  8.65s/it][2025-04-26 03:11:14,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:11:14,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.13 | bwd_microstep: 5743.69 | bwd_inner_microstep: 5641.06 | bwd_allreduce_microstep: 102.59 | step_microstep: 18.32
[2025-04-26 03:11:14,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.13 | bwd: 5743.70 | bwd_inner: 5641.06 | bwd_allreduce: 102.60 | step: 18.32

 19%|█▉        | 7961/41250 [19:13:39<79:57:56,  8.65s/it]
                                                          
{'loss': 0.4389, 'grad_norm': 2.9058303833007812, 'learning_rate': 3.727769009370493e-05, 'epoch': 1.93}

 19%|█▉        | 7961/41250 [19:13:39<79:57:56,  8.65s/it][2025-04-26 03:11:23,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 03:11:23,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.23 | bwd_microstep: 5741.24 | bwd_inner_microstep: 5714.06 | bwd_allreduce_microstep: 27.14 | step_microstep: 19.09
[2025-04-26 03:11:23,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.23 | bwd: 5741.26 | bwd_inner: 5714.06 | bwd_allreduce: 27.15 | step: 19.09

 19%|█▉        | 7962/41250 [19:13:48<80:04:02,  8.66s/it]
                                                          
{'loss': 0.0986, 'grad_norm': 1.353115200996399, 'learning_rate': 3.72768990834177e-05, 'epoch': 1.93}

 19%|█▉        | 7962/41250 [19:13:48<80:04:02,  8.66s/it][2025-04-26 03:11:31,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:11:31,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.07 | bwd_microstep: 5766.51 | bwd_inner_microstep: 5642.02 | bwd_allreduce_microstep: 124.45 | step_microstep: 18.44
[2025-04-26 03:11:31,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.07 | bwd: 5766.53 | bwd_inner: 5642.02 | bwd_allreduce: 124.47 | step: 18.44

 19%|█▉        | 7963/41250 [19:13:57<80:06:56,  8.66s/it]
                                                          
{'loss': 0.0799, 'grad_norm': 1.5568349361419678, 'learning_rate': 3.727610796662179e-05, 'epoch': 1.93}

 19%|█▉        | 7963/41250 [19:13:57<80:06:56,  8.66s/it][2025-04-26 03:11:40,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:11:40,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.62 | bwd_microstep: 5695.62 | bwd_inner_microstep: 5682.45 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.95
[2025-04-26 03:11:40,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.62 | bwd: 5695.64 | bwd_inner: 5682.45 | bwd_allreduce: 13.15 | step: 18.95

 19%|█▉        | 7964/41250 [19:14:05<79:59:57,  8.65s/it]
                                                          
{'loss': 0.2719, 'grad_norm': 2.615628242492676, 'learning_rate': 3.727531674332207e-05, 'epoch': 1.93}

 19%|█▉        | 7964/41250 [19:14:05<79:59:57,  8.65s/it][2025-04-26 03:11:49,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 03:11:49,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5775.09 | bwd_inner_microstep: 5641.10 | bwd_allreduce_microstep: 133.95 | step_microstep: 18.14
[2025-04-26 03:11:49,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5775.10 | bwd_inner: 5641.10 | bwd_allreduce: 133.96 | step: 18.14

 19%|█▉        | 7965/41250 [19:14:14<80:05:06,  8.66s/it]
                                                          
{'loss': 0.2598, 'grad_norm': 2.9421896934509277, 'learning_rate': 3.727452541352342e-05, 'epoch': 1.93}

 19%|█▉        | 7965/41250 [19:14:14<80:05:06,  8.66s/it][2025-04-26 03:11:57,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:11:57,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.48 | bwd_microstep: 5684.46 | bwd_inner_microstep: 5671.70 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.60
[2025-04-26 03:11:57,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.48 | bwd: 5684.47 | bwd_inner: 5671.70 | bwd_allreduce: 12.73 | step: 18.61

 19%|█▉        | 7966/41250 [19:14:23<79:55:59,  8.65s/it]
                                                          
{'loss': 0.1485, 'grad_norm': 2.126108169555664, 'learning_rate': 3.7273733977230726e-05, 'epoch': 1.93}

 19%|█▉        | 7966/41250 [19:14:23<79:55:59,  8.65s/it][2025-04-26 03:12:06,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:12:06,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.19 | bwd_microstep: 5747.63 | bwd_inner_microstep: 5674.99 | bwd_allreduce_microstep: 72.60 | step_microstep: 18.93
[2025-04-26 03:12:06,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.19 | bwd: 5747.65 | bwd_inner: 5674.99 | bwd_allreduce: 72.62 | step: 18.93

 19%|█▉        | 7967/41250 [19:14:31<80:00:21,  8.65s/it]
                                                          
{'loss': 0.2438, 'grad_norm': 3.4644885063171387, 'learning_rate': 3.727294243444886e-05, 'epoch': 1.93}

 19%|█▉        | 7967/41250 [19:14:31<80:00:21,  8.65s/it][2025-04-26 03:12:15,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.18
[2025-04-26 03:12:15,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.35 | bwd_microstep: 5697.38 | bwd_inner_microstep: 5640.07 | bwd_allreduce_microstep: 57.26 | step_microstep: 19.51
[2025-04-26 03:12:15,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.35 | bwd: 5697.40 | bwd_inner: 5640.07 | bwd_allreduce: 57.28 | step: 19.51

 19%|█▉        | 7968/41250 [19:14:40<79:52:25,  8.64s/it]
                                                          
{'loss': 0.2062, 'grad_norm': 1.6537140607833862, 'learning_rate': 3.727215078518269e-05, 'epoch': 1.93}

 19%|█▉        | 7968/41250 [19:14:40<79:52:25,  8.64s/it][2025-04-26 03:12:23,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 1.02
[2025-04-26 03:12:23,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.04 | bwd_microstep: 5697.27 | bwd_inner_microstep: 5684.18 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.09
[2025-04-26 03:12:23,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.04 | bwd: 5697.29 | bwd_inner: 5684.18 | bwd_allreduce: 13.07 | step: 19.09

 19%|█▉        | 7969/41250 [19:14:49<79:50:49,  8.64s/it]
                                                          
{'loss': 0.0318, 'grad_norm': 0.5292767882347107, 'learning_rate': 3.727135902943712e-05, 'epoch': 1.93}

 19%|█▉        | 7969/41250 [19:14:49<79:50:49,  8.64s/it][2025-04-26 03:12:32,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:12:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5672.46 | bwd_inner_microstep: 5636.97 | bwd_allreduce_microstep: 35.45 | step_microstep: 18.61
[2025-04-26 03:12:32,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5672.47 | bwd_inner: 5636.96 | bwd_allreduce: 35.47 | step: 18.61

 19%|█▉        | 7970/41250 [19:14:57<79:43:17,  8.62s/it]
                                                          
{'loss': 0.2203, 'grad_norm': 2.873655080795288, 'learning_rate': 3.7270567167217014e-05, 'epoch': 1.93}

 19%|█▉        | 7970/41250 [19:14:57<79:43:17,  8.62s/it][2025-04-26 03:12:40,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 03:12:40,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.58 | bwd_microstep: 5702.04 | bwd_inner_microstep: 5689.13 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.13
[2025-04-26 03:12:40,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.58 | bwd: 5702.05 | bwd_inner: 5689.13 | bwd_allreduce: 12.88 | step: 19.14

 19%|█▉        | 7971/41250 [19:15:06<79:45:59,  8.63s/it]
                                                          
{'loss': 0.058, 'grad_norm': 0.9976211190223694, 'learning_rate': 3.7269775198527255e-05, 'epoch': 1.93}

 19%|█▉        | 7971/41250 [19:15:06<79:45:59,  8.63s/it][2025-04-26 03:12:49,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:12:49,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.70 | bwd_microstep: 5946.76 | bwd_inner_microstep: 5645.73 | bwd_allreduce_microstep: 300.98 | step_microstep: 18.71
[2025-04-26 03:12:49,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.70 | bwd: 5946.77 | bwd_inner: 5645.73 | bwd_allreduce: 301.00 | step: 18.71

 19%|█▉        | 7972/41250 [19:15:15<80:22:56,  8.70s/it]
                                                          
{'loss': 0.1117, 'grad_norm': 1.5577534437179565, 'learning_rate': 3.726898312337273e-05, 'epoch': 1.93}

 19%|█▉        | 7972/41250 [19:15:15<80:22:56,  8.70s/it][2025-04-26 03:12:58,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.07 | optimizer_step: 1.20
[2025-04-26 03:12:58,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.22 | bwd_microstep: 5751.17 | bwd_inner_microstep: 5647.52 | bwd_allreduce_microstep: 103.60 | step_microstep: 19.35
[2025-04-26 03:12:58,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.22 | bwd: 5751.18 | bwd_inner: 5647.52 | bwd_allreduce: 103.62 | step: 19.35

 19%|█▉        | 7973/41250 [19:15:23<80:18:43,  8.69s/it]
                                                          
{'loss': 0.0891, 'grad_norm': 1.380790114402771, 'learning_rate': 3.7268190941758325e-05, 'epoch': 1.93}

 19%|█▉        | 7973/41250 [19:15:23<80:18:43,  8.69s/it][2025-04-26 03:13:07,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 03:13:07,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.41 | bwd_microstep: 5768.22 | bwd_inner_microstep: 5639.17 | bwd_allreduce_microstep: 128.99 | step_microstep: 18.80
[2025-04-26 03:13:07,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.41 | bwd: 5768.23 | bwd_inner: 5639.17 | bwd_allreduce: 129.01 | step: 18.80

 19%|█▉        | 7974/41250 [19:15:32<80:16:54,  8.69s/it]
                                                          
{'loss': 0.1135, 'grad_norm': 2.4950740337371826, 'learning_rate': 3.726739865368891e-05, 'epoch': 1.93}

 19%|█▉        | 7974/41250 [19:15:32<80:16:54,  8.69s/it][2025-04-26 03:13:15,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 03:13:15,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.10 | bwd_microstep: 5777.79 | bwd_inner_microstep: 5676.00 | bwd_allreduce_microstep: 101.74 | step_microstep: 19.27
[2025-04-26 03:13:15,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.10 | bwd: 5777.81 | bwd_inner: 5676.00 | bwd_allreduce: 101.76 | step: 19.26

 19%|█▉        | 7975/41250 [19:15:41<80:20:05,  8.69s/it]
                                                          
{'loss': 0.09, 'grad_norm': 2.202418565750122, 'learning_rate': 3.726660625916939e-05, 'epoch': 1.93}

 19%|█▉        | 7975/41250 [19:15:41<80:20:05,  8.69s/it][2025-04-26 03:13:24,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:13:24,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.02 | bwd_microstep: 5712.87 | bwd_inner_microstep: 5700.12 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.92
[2025-04-26 03:13:24,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.02 | bwd: 5712.88 | bwd_inner: 5700.12 | bwd_allreduce: 12.72 | step: 18.93

 19%|█▉        | 7976/41250 [19:15:49<80:12:22,  8.68s/it]
                                                          
{'loss': 0.2236, 'grad_norm': 1.9117599725723267, 'learning_rate': 3.726581375820463e-05, 'epoch': 1.93}

 19%|█▉        | 7976/41250 [19:15:49<80:12:22,  8.68s/it][2025-04-26 03:13:33,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.26 | optimizer_step: 1.00
[2025-04-26 03:13:33,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.66 | bwd_microstep: 5756.89 | bwd_inner_microstep: 5642.13 | bwd_allreduce_microstep: 114.71 | step_microstep: 19.57
[2025-04-26 03:13:33,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.66 | bwd: 5756.91 | bwd_inner: 5642.13 | bwd_allreduce: 114.73 | step: 19.57

 19%|█▉        | 7977/41250 [19:15:58<80:10:53,  8.68s/it]
                                                          
{'loss': 0.2876, 'grad_norm': 3.8188090324401855, 'learning_rate': 3.7265021150799526e-05, 'epoch': 1.93}

 19%|█▉        | 7977/41250 [19:15:58<80:10:53,  8.68s/it][2025-04-26 03:13:41,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:13:41,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.39 | bwd_microstep: 5778.97 | bwd_inner_microstep: 5637.22 | bwd_allreduce_microstep: 141.71 | step_microstep: 18.74
[2025-04-26 03:13:41,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.39 | bwd: 5778.98 | bwd_inner: 5637.22 | bwd_allreduce: 141.72 | step: 18.75

 19%|█▉        | 7978/41250 [19:16:07<80:12:36,  8.68s/it]
                                                          
{'loss': 0.1796, 'grad_norm': 2.0775225162506104, 'learning_rate': 3.7264228436958966e-05, 'epoch': 1.93}

 19%|█▉        | 7978/41250 [19:16:07<80:12:36,  8.68s/it][2025-04-26 03:13:50,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:13:50,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.38 | bwd_microstep: 5692.12 | bwd_inner_microstep: 5679.32 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.72
[2025-04-26 03:13:50,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.38 | bwd: 5692.14 | bwd_inner: 5679.32 | bwd_allreduce: 12.78 | step: 18.72

 19%|█▉        | 7979/41250 [19:16:15<80:03:12,  8.66s/it]
                                                          
{'loss': 0.0254, 'grad_norm': 0.34262606501579285, 'learning_rate': 3.726343561668783e-05, 'epoch': 1.93}

 19%|█▉        | 7979/41250 [19:16:15<80:03:12,  8.66s/it][2025-04-26 03:13:59,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:13:59,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.21 | bwd_microstep: 5733.29 | bwd_inner_microstep: 5698.80 | bwd_allreduce_microstep: 34.45 | step_microstep: 18.44
[2025-04-26 03:13:59,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.21 | bwd: 5733.31 | bwd_inner: 5698.80 | bwd_allreduce: 34.47 | step: 18.45

 19%|█▉        | 7980/41250 [19:16:24<80:03:25,  8.66s/it]
                                                          
{'loss': 0.2225, 'grad_norm': 1.800246000289917, 'learning_rate': 3.726264268999101e-05, 'epoch': 1.93}

 19%|█▉        | 7980/41250 [19:16:24<80:03:25,  8.66s/it][2025-04-26 03:14:07,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:14:07,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.87 | bwd_microstep: 5718.08 | bwd_inner_microstep: 5705.18 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.61
[2025-04-26 03:14:07,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.87 | bwd: 5718.09 | bwd_inner: 5705.18 | bwd_allreduce: 12.87 | step: 18.61

 19%|█▉        | 7981/41250 [19:16:33<80:06:44,  8.67s/it]
                                                          
{'loss': 0.0619, 'grad_norm': 2.8412468433380127, 'learning_rate': 3.726184965687339e-05, 'epoch': 1.93}

 19%|█▉        | 7981/41250 [19:16:33<80:06:44,  8.67s/it][2025-04-26 03:14:16,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:14:16,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.58 | bwd_microstep: 5702.90 | bwd_inner_microstep: 5690.07 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.75
[2025-04-26 03:14:16,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.58 | bwd: 5702.92 | bwd_inner: 5690.07 | bwd_allreduce: 12.81 | step: 18.76

 19%|█▉        | 7982/41250 [19:16:41<80:01:13,  8.66s/it]
                                                          
{'loss': 0.1225, 'grad_norm': 3.8491225242614746, 'learning_rate': 3.726105651733987e-05, 'epoch': 1.94}

 19%|█▉        | 7982/41250 [19:16:41<80:01:13,  8.66s/it][2025-04-26 03:14:25,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 03:14:25,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.10 | bwd_microstep: 5741.85 | bwd_inner_microstep: 5701.83 | bwd_allreduce_microstep: 39.97 | step_microstep: 18.29
[2025-04-26 03:14:25,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.10 | bwd: 5741.86 | bwd_inner: 5701.83 | bwd_allreduce: 39.99 | step: 18.29

 19%|█▉        | 7983/41250 [19:16:50<80:04:35,  8.67s/it]
                                                          
{'loss': 0.304, 'grad_norm': 2.122929811477661, 'learning_rate': 3.7260263271395325e-05, 'epoch': 1.94}

 19%|█▉        | 7983/41250 [19:16:50<80:04:35,  8.67s/it][2025-04-26 03:14:33,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:14:33,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.32 | bwd_microstep: 5736.71 | bwd_inner_microstep: 5692.01 | bwd_allreduce_microstep: 44.66 | step_microstep: 18.58
[2025-04-26 03:14:33,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.32 | bwd: 5736.72 | bwd_inner: 5692.01 | bwd_allreduce: 44.68 | step: 18.59

 19%|█▉        | 7984/41250 [19:16:59<80:05:28,  8.67s/it]
                                                          
{'loss': 0.1011, 'grad_norm': 1.8959314823150635, 'learning_rate': 3.725946991904465e-05, 'epoch': 1.94}

 19%|█▉        | 7984/41250 [19:16:59<80:05:28,  8.67s/it][2025-04-26 03:14:42,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:14:42,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5775.21 | bwd_inner_microstep: 5696.44 | bwd_allreduce_microstep: 78.72 | step_microstep: 18.65
[2025-04-26 03:14:42,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5775.22 | bwd_inner: 5696.44 | bwd_allreduce: 78.74 | step: 18.65

 19%|█▉        | 7985/41250 [19:17:07<80:13:00,  8.68s/it]
                                                          
{'loss': 0.116, 'grad_norm': 1.9440454244613647, 'learning_rate': 3.7258676460292744e-05, 'epoch': 1.94}

 19%|█▉        | 7985/41250 [19:17:07<80:13:00,  8.68s/it][2025-04-26 03:14:51,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 03:14:51,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.43 | bwd_microstep: 5754.29 | bwd_inner_microstep: 5694.46 | bwd_allreduce_microstep: 59.79 | step_microstep: 19.29
[2025-04-26 03:14:51,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.43 | bwd: 5754.30 | bwd_inner: 5694.46 | bwd_allreduce: 59.80 | step: 19.30

 19%|█▉        | 7986/41250 [19:17:16<80:13:55,  8.68s/it]
                                                          
{'loss': 0.1332, 'grad_norm': 1.590816617012024, 'learning_rate': 3.7257882895144485e-05, 'epoch': 1.94}

 19%|█▉        | 7986/41250 [19:17:16<80:13:55,  8.68s/it][2025-04-26 03:14:59,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:14:59,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.33 | bwd_microstep: 5788.60 | bwd_inner_microstep: 5658.93 | bwd_allreduce_microstep: 129.62 | step_microstep: 18.43
[2025-04-26 03:14:59,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.33 | bwd: 5788.61 | bwd_inner: 5658.93 | bwd_allreduce: 129.64 | step: 18.44

 19%|█▉        | 7987/41250 [19:17:25<80:16:03,  8.69s/it]
                                                          
{'loss': 0.0862, 'grad_norm': 3.6912381649017334, 'learning_rate': 3.725708922360478e-05, 'epoch': 1.94}

 19%|█▉        | 7987/41250 [19:17:25<80:16:03,  8.69s/it][2025-04-26 03:15:08,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 1.05
[2025-04-26 03:15:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 5861.09 | bwd_inner_microstep: 5689.66 | bwd_allreduce_microstep: 171.38 | step_microstep: 19.89
[2025-04-26 03:15:08,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 5861.11 | bwd_inner: 5689.66 | bwd_allreduce: 171.41 | step: 19.89

 19%|█▉        | 7988/41250 [19:17:34<80:33:56,  8.72s/it]
                                                          
{'loss': 0.0945, 'grad_norm': 10.21113395690918, 'learning_rate': 3.725629544567851e-05, 'epoch': 1.94}

 19%|█▉        | 7988/41250 [19:17:34<80:33:56,  8.72s/it][2025-04-26 03:15:17,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.03 | optimizer_step: 1.19
[2025-04-26 03:15:17,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.98 | bwd_microstep: 5773.10 | bwd_inner_microstep: 5649.54 | bwd_allreduce_microstep: 123.51 | step_microstep: 19.20
[2025-04-26 03:15:17,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.98 | bwd: 5773.11 | bwd_inner: 5649.54 | bwd_allreduce: 123.53 | step: 19.20

 19%|█▉        | 7989/41250 [19:17:42<80:27:51,  8.71s/it]
                                                          
{'loss': 0.0335, 'grad_norm': 1.1417410373687744, 'learning_rate': 3.7255501561370576e-05, 'epoch': 1.94}

 19%|█▉        | 7989/41250 [19:17:42<80:27:51,  8.71s/it][2025-04-26 03:15:26,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 03:15:26,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.11 | bwd_microstep: 5764.06 | bwd_inner_microstep: 5721.51 | bwd_allreduce_microstep: 42.51 | step_microstep: 18.70
[2025-04-26 03:15:26,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.11 | bwd: 5764.07 | bwd_inner: 5721.51 | bwd_allreduce: 42.52 | step: 18.71

 19%|█▉        | 7990/41250 [19:17:51<80:27:58,  8.71s/it]
                                                          
{'loss': 0.0433, 'grad_norm': 0.858853816986084, 'learning_rate': 3.7254707570685867e-05, 'epoch': 1.94}

 19%|█▉        | 7990/41250 [19:17:51<80:27:58,  8.71s/it][2025-04-26 03:15:34,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 03:15:34,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.45 | bwd_microstep: 5765.00 | bwd_inner_microstep: 5720.62 | bwd_allreduce_microstep: 44.33 | step_microstep: 18.67
[2025-04-26 03:15:34,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.45 | bwd: 5765.01 | bwd_inner: 5720.62 | bwd_allreduce: 44.35 | step: 18.67

 19%|█▉        | 7991/41250 [19:18:00<80:26:37,  8.71s/it]
                                                          
{'loss': 0.1062, 'grad_norm': 2.4419467449188232, 'learning_rate': 3.725391347362928e-05, 'epoch': 1.94}

 19%|█▉        | 7991/41250 [19:18:00<80:26:37,  8.71s/it][2025-04-26 03:15:43,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 03:15:43,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.10 | bwd_microstep: 5781.62 | bwd_inner_microstep: 5659.61 | bwd_allreduce_microstep: 121.97 | step_microstep: 18.48
[2025-04-26 03:15:43,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.10 | bwd: 5781.64 | bwd_inner: 5659.61 | bwd_allreduce: 121.98 | step: 18.48

 19%|█▉        | 7992/41250 [19:18:08<80:24:35,  8.70s/it]
                                                          
{'loss': 0.1424, 'grad_norm': 2.035543203353882, 'learning_rate': 3.725311927020571e-05, 'epoch': 1.94}

 19%|█▉        | 7992/41250 [19:18:08<80:24:35,  8.70s/it][2025-04-26 03:15:52,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:15:52,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.21 | bwd_microstep: 5713.82 | bwd_inner_microstep: 5701.12 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.29
[2025-04-26 03:15:52,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.21 | bwd: 5713.83 | bwd_inner: 5701.12 | bwd_allreduce: 12.67 | step: 18.29

 19%|█▉        | 7993/41250 [19:18:17<80:14:26,  8.69s/it]
                                                          
{'loss': 0.0962, 'grad_norm': 1.4063676595687866, 'learning_rate': 3.725232496042005e-05, 'epoch': 1.94}

 19%|█▉        | 7993/41250 [19:18:17<80:14:26,  8.69s/it][2025-04-26 03:16:00,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:16:00,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.45 | bwd_microstep: 5769.27 | bwd_inner_microstep: 5719.34 | bwd_allreduce_microstep: 49.88 | step_microstep: 18.68
[2025-04-26 03:16:00,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.45 | bwd: 5769.28 | bwd_inner: 5719.34 | bwd_allreduce: 49.90 | step: 18.68

 19%|█▉        | 7994/41250 [19:18:26<80:19:03,  8.69s/it]
                                                          
{'loss': 0.1295, 'grad_norm': 2.041811466217041, 'learning_rate': 3.7251530544277205e-05, 'epoch': 1.94}

 19%|█▉        | 7994/41250 [19:18:26<80:19:03,  8.69s/it][2025-04-26 03:16:09,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 03:16:09,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.94 | bwd_microstep: 5705.02 | bwd_inner_microstep: 5692.09 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.45
[2025-04-26 03:16:09,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.94 | bwd: 5705.03 | bwd_inner: 5692.09 | bwd_allreduce: 12.90 | step: 19.45

 19%|█▉        | 7995/41250 [19:18:34<80:10:30,  8.68s/it]
                                                          
{'loss': 0.1729, 'grad_norm': 4.6222381591796875, 'learning_rate': 3.7250736021782066e-05, 'epoch': 1.94}

 19%|█▉        | 7995/41250 [19:18:34<80:10:30,  8.68s/it][2025-04-26 03:16:18,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:16:18,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.82 | bwd_microstep: 5713.34 | bwd_inner_microstep: 5654.26 | bwd_allreduce_microstep: 59.03 | step_microstep: 19.16
[2025-04-26 03:16:18,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.82 | bwd: 5713.35 | bwd_inner: 5654.26 | bwd_allreduce: 59.05 | step: 19.16

 19%|█▉        | 7996/41250 [19:18:43<80:01:30,  8.66s/it]
                                                          
{'loss': 0.0673, 'grad_norm': 1.7279244661331177, 'learning_rate': 3.7249941392939536e-05, 'epoch': 1.94}

 19%|█▉        | 7996/41250 [19:18:43<80:01:30,  8.66s/it][2025-04-26 03:16:26,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:16:26,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.14 | bwd_microstep: 5765.01 | bwd_inner_microstep: 5698.88 | bwd_allreduce_microstep: 66.09 | step_microstep: 18.65
[2025-04-26 03:16:26,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.14 | bwd: 5765.02 | bwd_inner: 5698.88 | bwd_allreduce: 66.10 | step: 18.65

 19%|█▉        | 7997/41250 [19:18:52<80:06:58,  8.67s/it]
                                                          
{'loss': 0.2204, 'grad_norm': 2.895005702972412, 'learning_rate': 3.7249146657754505e-05, 'epoch': 1.94}

 19%|█▉        | 7997/41250 [19:18:52<80:06:58,  8.67s/it][2025-04-26 03:16:35,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 03:16:35,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.14 | bwd_microstep: 5757.24 | bwd_inner_microstep: 5656.49 | bwd_allreduce_microstep: 100.70 | step_microstep: 19.47
[2025-04-26 03:16:35,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.14 | bwd: 5757.26 | bwd_inner: 5656.49 | bwd_allreduce: 100.72 | step: 19.47

 19%|█▉        | 7998/41250 [19:19:00<80:05:52,  8.67s/it]
                                                          
{'loss': 0.03, 'grad_norm': 1.4967327117919922, 'learning_rate': 3.7248351816231875e-05, 'epoch': 1.94}

 19%|█▉        | 7998/41250 [19:19:00<80:05:52,  8.67s/it][2025-04-26 03:16:44,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:16:44,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.18 | bwd_microstep: 5765.29 | bwd_inner_microstep: 5643.94 | bwd_allreduce_microstep: 121.30 | step_microstep: 18.58
[2025-04-26 03:16:44,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.18 | bwd: 5765.30 | bwd_inner: 5643.94 | bwd_allreduce: 121.32 | step: 18.58

 19%|█▉        | 7999/41250 [19:19:09<80:07:03,  8.67s/it]
                                                          
{'loss': 0.1681, 'grad_norm': 2.1041159629821777, 'learning_rate': 3.724755686837655e-05, 'epoch': 1.94}

 19%|█▉        | 7999/41250 [19:19:09<80:07:03,  8.67s/it][2025-04-26 03:16:53,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 03:16:53,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.15 | bwd_microstep: 5926.67 | bwd_inner_microstep: 5637.03 | bwd_allreduce_microstep: 289.59 | step_microstep: 18.82
[2025-04-26 03:16:53,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.15 | bwd: 5926.68 | bwd_inner: 5637.03 | bwd_allreduce: 289.61 | step: 18.83

 19%|█▉        | 8000/41250 [19:19:18<80:33:04,  8.72s/it]
                                                          
{'loss': 0.4024, 'grad_norm': 3.715576171875, 'learning_rate': 3.724676181419343e-05, 'epoch': 1.94}

 19%|█▉        | 8000/41250 [19:19:18<80:33:04,  8.72s/it][2025-04-26 03:17:01,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:17:01,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.38 | bwd_microstep: 5712.60 | bwd_inner_microstep: 5699.88 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.69
[2025-04-26 03:17:01,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.38 | bwd: 5712.61 | bwd_inner: 5699.88 | bwd_allreduce: 12.69 | step: 18.69

 19%|█▉        | 8001/41250 [19:19:26<80:20:34,  8.70s/it]
                                                          
{'loss': 0.138, 'grad_norm': 2.2388648986816406, 'learning_rate': 3.724596665368742e-05, 'epoch': 1.94}

 19%|█▉        | 8001/41250 [19:19:26<80:20:34,  8.70s/it][2025-04-26 03:17:10,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:17:10,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.60 | bwd_microstep: 5769.99 | bwd_inner_microstep: 5643.71 | bwd_allreduce_microstep: 126.24 | step_microstep: 18.63
[2025-04-26 03:17:10,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.60 | bwd: 5770.00 | bwd_inner: 5643.71 | bwd_allreduce: 126.25 | step: 18.64

 19%|█▉        | 8002/41250 [19:19:35<80:17:59,  8.69s/it]
                                                          
{'loss': 0.2991, 'grad_norm': 2.1534817218780518, 'learning_rate': 3.7245171386863414e-05, 'epoch': 1.94}

 19%|█▉        | 8002/41250 [19:19:35<80:17:59,  8.69s/it][2025-04-26 03:17:18,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 03:17:18,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.56 | bwd_microstep: 5711.24 | bwd_inner_microstep: 5647.32 | bwd_allreduce_microstep: 63.88 | step_microstep: 18.77
[2025-04-26 03:17:18,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.56 | bwd: 5711.25 | bwd_inner: 5647.32 | bwd_allreduce: 63.89 | step: 18.78

 19%|█▉        | 8003/41250 [19:19:44<80:05:08,  8.67s/it]
                                                          
{'loss': 0.0687, 'grad_norm': 1.2536883354187012, 'learning_rate': 3.724437601372632e-05, 'epoch': 1.94}

 19%|█▉        | 8003/41250 [19:19:44<80:05:08,  8.67s/it][2025-04-26 03:17:27,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 03:17:27,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5702.24 | bwd_inner_microstep: 5651.61 | bwd_allreduce_microstep: 50.59 | step_microstep: 18.80
[2025-04-26 03:17:27,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5702.26 | bwd_inner: 5651.61 | bwd_allreduce: 50.61 | step: 18.80

 19%|█▉        | 8004/41250 [19:19:52<79:54:23,  8.65s/it]
                                                          
{'loss': 0.5366, 'grad_norm': 2.429692268371582, 'learning_rate': 3.7243580534281036e-05, 'epoch': 1.94}

 19%|█▉        | 8004/41250 [19:19:52<79:54:23,  8.65s/it][2025-04-26 03:17:36,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:17:36,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.03 | bwd_microstep: 5748.28 | bwd_inner_microstep: 5689.85 | bwd_allreduce_microstep: 58.39 | step_microstep: 18.17
[2025-04-26 03:17:36,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.03 | bwd: 5748.30 | bwd_inner: 5689.85 | bwd_allreduce: 58.41 | step: 18.17

 19%|█▉        | 8005/41250 [19:20:01<79:57:57,  8.66s/it]
                                                          
{'loss': 0.4008, 'grad_norm': 4.968603134155273, 'learning_rate': 3.7242784948532477e-05, 'epoch': 1.94}

 19%|█▉        | 8005/41250 [19:20:01<79:57:57,  8.66s/it][2025-04-26 03:17:44,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 03:17:44,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.10 | bwd_microstep: 5782.20 | bwd_inner_microstep: 5644.51 | bwd_allreduce_microstep: 137.65 | step_microstep: 19.23
[2025-04-26 03:17:44,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.10 | bwd: 5782.22 | bwd_inner: 5644.51 | bwd_allreduce: 137.67 | step: 19.23

 19%|█▉        | 8006/41250 [19:20:10<80:02:18,  8.67s/it]
                                                          
{'loss': 0.1793, 'grad_norm': 4.943327903747559, 'learning_rate': 3.724198925648553e-05, 'epoch': 1.94}

 19%|█▉        | 8006/41250 [19:20:10<80:02:18,  8.67s/it][2025-04-26 03:17:53,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:17:53,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.89 | bwd_microstep: 5773.33 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 123.98 | step_microstep: 19.03
[2025-04-26 03:17:53,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.89 | bwd: 5773.34 | bwd_inner: 5649.29 | bwd_allreduce: 124.00 | step: 19.03

 19%|█▉        | 8007/41250 [19:20:18<80:05:28,  8.67s/it]
                                                          
{'loss': 0.2669, 'grad_norm': 2.868577718734741, 'learning_rate': 3.724119345814512e-05, 'epoch': 1.94}

 19%|█▉        | 8007/41250 [19:20:18<80:05:28,  8.67s/it][2025-04-26 03:18:02,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 03:18:02,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.09 | bwd_microstep: 5703.25 | bwd_inner_microstep: 5690.32 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.34
[2025-04-26 03:18:02,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.09 | bwd: 5703.27 | bwd_inner: 5690.32 | bwd_allreduce: 12.91 | step: 19.34

 19%|█▉        | 8008/41250 [19:20:27<79:59:20,  8.66s/it]
                                                          
{'loss': 0.1935, 'grad_norm': 2.151076316833496, 'learning_rate': 3.724039755351614e-05, 'epoch': 1.94}

 19%|█▉        | 8008/41250 [19:20:27<79:59:20,  8.66s/it][2025-04-26 03:18:10,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:18:10,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.10 | bwd_microstep: 5742.33 | bwd_inner_microstep: 5649.01 | bwd_allreduce_microstep: 93.28 | step_microstep: 18.41
[2025-04-26 03:18:10,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.10 | bwd: 5742.35 | bwd_inner: 5649.01 | bwd_allreduce: 93.29 | step: 18.41

 19%|█▉        | 8009/41250 [19:20:36<79:58:16,  8.66s/it]
                                                          
{'loss': 0.2027, 'grad_norm': 1.5765413045883179, 'learning_rate': 3.7239601542603496e-05, 'epoch': 1.94}

 19%|█▉        | 8009/41250 [19:20:36<79:58:16,  8.66s/it][2025-04-26 03:18:19,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 03:18:19,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5709.77 | bwd_inner_microstep: 5678.33 | bwd_allreduce_microstep: 31.40 | step_microstep: 19.07
[2025-04-26 03:18:19,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5709.79 | bwd_inner: 5678.33 | bwd_allreduce: 31.42 | step: 19.07

 19%|█▉        | 8010/41250 [19:20:44<79:55:04,  8.66s/it]
                                                          
{'loss': 0.0625, 'grad_norm': 1.1015089750289917, 'learning_rate': 3.7238805425412107e-05, 'epoch': 1.94}

 19%|█▉        | 8010/41250 [19:20:44<79:55:04,  8.66s/it][2025-04-26 03:18:28,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:18:28,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.41 | bwd_microstep: 5744.86 | bwd_inner_microstep: 5646.42 | bwd_allreduce_microstep: 98.40 | step_microstep: 18.35
[2025-04-26 03:18:28,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.41 | bwd: 5744.87 | bwd_inner: 5646.42 | bwd_allreduce: 98.41 | step: 18.35

 19%|█▉        | 8011/41250 [19:20:53<79:56:13,  8.66s/it]
                                                          
{'loss': 0.1231, 'grad_norm': 1.8531126976013184, 'learning_rate': 3.723800920194687e-05, 'epoch': 1.94}

 19%|█▉        | 8011/41250 [19:20:53<79:56:13,  8.66s/it][2025-04-26 03:18:36,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 03:18:36,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5835.30 | bwd_inner_microstep: 5665.67 | bwd_allreduce_microstep: 169.57 | step_microstep: 19.03
[2025-04-26 03:18:36,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5835.32 | bwd_inner: 5665.67 | bwd_allreduce: 169.59 | step: 19.04

 19%|█▉        | 8012/41250 [19:21:02<80:16:36,  8.69s/it]
                                                          
{'loss': 0.1424, 'grad_norm': 1.995123028755188, 'learning_rate': 3.723721287221269e-05, 'epoch': 1.94}

 19%|█▉        | 8012/41250 [19:21:02<80:16:36,  8.69s/it][2025-04-26 03:18:45,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:18:45,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.97 | bwd_microstep: 5747.71 | bwd_inner_microstep: 5673.23 | bwd_allreduce_microstep: 74.43 | step_microstep: 18.69
[2025-04-26 03:18:45,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.97 | bwd: 5747.72 | bwd_inner: 5673.23 | bwd_allreduce: 74.45 | step: 18.69

 19%|█▉        | 8013/41250 [19:21:10<80:12:20,  8.69s/it]
                                                          
{'loss': 0.2439, 'grad_norm': 1.8550572395324707, 'learning_rate': 3.723641643621449e-05, 'epoch': 1.94}

 19%|█▉        | 8013/41250 [19:21:10<80:12:20,  8.69s/it][2025-04-26 03:18:54,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.19 | optimizer_step: 1.05
[2025-04-26 03:18:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.87 | bwd_microstep: 6000.08 | bwd_inner_microstep: 5644.75 | bwd_allreduce_microstep: 355.29 | step_microstep: 19.36
[2025-04-26 03:18:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.87 | bwd: 6000.09 | bwd_inner: 5644.75 | bwd_allreduce: 355.31 | step: 19.36

 19%|█▉        | 8014/41250 [19:21:19<80:48:12,  8.75s/it]
                                                          
{'loss': 0.1482, 'grad_norm': 3.0175909996032715, 'learning_rate': 3.7235619893957175e-05, 'epoch': 1.94}

 19%|█▉        | 8014/41250 [19:21:19<80:48:12,  8.75s/it][2025-04-26 03:19:03,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:19:03,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5738.81 | bwd_inner_microstep: 5690.34 | bwd_allreduce_microstep: 48.42 | step_microstep: 18.90
[2025-04-26 03:19:03,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5738.82 | bwd_inner: 5690.34 | bwd_allreduce: 48.44 | step: 18.90

 19%|█▉        | 8015/41250 [19:21:28<80:34:21,  8.73s/it]
                                                          
{'loss': 0.1798, 'grad_norm': 3.0475103855133057, 'learning_rate': 3.7234823245445646e-05, 'epoch': 1.94}

 19%|█▉        | 8015/41250 [19:21:28<80:34:21,  8.73s/it][2025-04-26 03:19:11,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:19:11,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.29 | bwd_microstep: 5684.60 | bwd_inner_microstep: 5640.72 | bwd_allreduce_microstep: 43.83 | step_microstep: 18.92
[2025-04-26 03:19:11,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.29 | bwd: 5684.62 | bwd_inner: 5640.72 | bwd_allreduce: 43.85 | step: 18.92

 19%|█▉        | 8016/41250 [19:21:37<80:11:24,  8.69s/it]
                                                          
{'loss': 0.0523, 'grad_norm': 0.6806823015213013, 'learning_rate': 3.723402649068483e-05, 'epoch': 1.94}

 19%|█▉        | 8016/41250 [19:21:37<80:11:24,  8.69s/it][2025-04-26 03:19:20,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 03:19:20,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5742.11 | bwd_inner_microstep: 5647.50 | bwd_allreduce_microstep: 94.56 | step_microstep: 19.25
[2025-04-26 03:19:20,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5742.12 | bwd_inner: 5647.50 | bwd_allreduce: 94.58 | step: 19.25

 19%|█▉        | 8017/41250 [19:21:45<80:05:24,  8.68s/it]
                                                          
{'loss': 0.0187, 'grad_norm': 0.5980530977249146, 'learning_rate': 3.7233229629679625e-05, 'epoch': 1.94}

 19%|█▉        | 8017/41250 [19:21:45<80:05:24,  8.68s/it][2025-04-26 03:19:29,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:19:29,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.10 | bwd_microstep: 5753.36 | bwd_inner_microstep: 5637.15 | bwd_allreduce_microstep: 116.17 | step_microstep: 18.64
[2025-04-26 03:19:29,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.10 | bwd: 5753.37 | bwd_inner: 5637.15 | bwd_allreduce: 116.18 | step: 18.64

 19%|█▉        | 8018/41250 [19:21:54<80:02:57,  8.67s/it]
                                                          
{'loss': 0.2258, 'grad_norm': 2.5408754348754883, 'learning_rate': 3.723243266243495e-05, 'epoch': 1.94}

 19%|█▉        | 8018/41250 [19:21:54<80:02:57,  8.67s/it][2025-04-26 03:19:37,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-26 03:19:37,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.80 | bwd_microstep: 5777.43 | bwd_inner_microstep: 5682.24 | bwd_allreduce_microstep: 95.13 | step_microstep: 19.62
[2025-04-26 03:19:37,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.80 | bwd: 5777.44 | bwd_inner: 5682.24 | bwd_allreduce: 95.15 | step: 19.62

 19%|█▉        | 8019/41250 [19:22:03<80:09:36,  8.68s/it]
                                                          
{'loss': 0.0518, 'grad_norm': 0.5113413333892822, 'learning_rate': 3.723163558895572e-05, 'epoch': 1.94}

 19%|█▉        | 8019/41250 [19:22:03<80:09:36,  8.68s/it][2025-04-26 03:19:46,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 03:19:46,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5715.77 | bwd_inner_microstep: 5642.06 | bwd_allreduce_microstep: 73.66 | step_microstep: 19.05
[2025-04-26 03:19:46,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5715.78 | bwd_inner: 5642.06 | bwd_allreduce: 73.68 | step: 19.05

 19%|█▉        | 8020/41250 [19:22:11<79:59:24,  8.67s/it]
                                                          
{'loss': 0.1076, 'grad_norm': 0.969128429889679, 'learning_rate': 3.723083840924684e-05, 'epoch': 1.94}

 19%|█▉        | 8020/41250 [19:22:11<79:59:24,  8.67s/it][2025-04-26 03:19:55,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:19:55,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.83 | bwd_microstep: 5752.56 | bwd_inner_microstep: 5692.90 | bwd_allreduce_microstep: 59.61 | step_microstep: 18.62
[2025-04-26 03:19:55,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.83 | bwd: 5752.57 | bwd_inner: 5692.90 | bwd_allreduce: 59.64 | step: 18.63

 19%|█▉        | 8021/41250 [19:22:20<80:06:45,  8.68s/it]
                                                          
{'loss': 0.1457, 'grad_norm': 1.8292908668518066, 'learning_rate': 3.723004112331324e-05, 'epoch': 1.94}

 19%|█▉        | 8021/41250 [19:22:20<80:06:45,  8.68s/it][2025-04-26 03:20:03,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-26 03:20:03,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.93 | bwd_microstep: 5768.98 | bwd_inner_microstep: 5653.48 | bwd_allreduce_microstep: 115.45 | step_microstep: 19.35
[2025-04-26 03:20:03,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.93 | bwd: 5768.99 | bwd_inner: 5653.48 | bwd_allreduce: 115.47 | step: 19.36

 19%|█▉        | 8022/41250 [19:22:29<80:08:26,  8.68s/it]
                                                          
{'loss': 0.0932, 'grad_norm': 1.2958565950393677, 'learning_rate': 3.722924373115981e-05, 'epoch': 1.94}

 19%|█▉        | 8022/41250 [19:22:29<80:08:26,  8.68s/it][2025-04-26 03:20:12,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:20:12,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.62 | bwd_microstep: 5770.05 | bwd_inner_microstep: 5657.11 | bwd_allreduce_microstep: 112.90 | step_microstep: 18.34
[2025-04-26 03:20:12,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.62 | bwd: 5770.06 | bwd_inner: 5657.11 | bwd_allreduce: 112.92 | step: 18.34

 19%|█▉        | 8023/41250 [19:22:37<80:08:06,  8.68s/it]
                                                          
{'loss': 0.146, 'grad_norm': 1.5735522508621216, 'learning_rate': 3.722844623279149e-05, 'epoch': 1.94}

 19%|█▉        | 8023/41250 [19:22:37<80:08:06,  8.68s/it][2025-04-26 03:20:21,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:20:21,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5692.00 | bwd_inner_microstep: 5679.55 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.28
[2025-04-26 03:20:21,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5692.01 | bwd_inner: 5679.55 | bwd_allreduce: 12.42 | step: 18.28

 19%|█▉        | 8024/41250 [19:22:46<79:58:32,  8.67s/it]
                                                          
{'loss': 0.2243, 'grad_norm': 4.328475475311279, 'learning_rate': 3.7227648628213194e-05, 'epoch': 1.95}

 19%|█▉        | 8024/41250 [19:22:46<79:58:32,  8.67s/it][2025-04-26 03:20:29,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:20:29,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.44 | bwd_microstep: 5719.10 | bwd_inner_microstep: 5647.70 | bwd_allreduce_microstep: 71.35 | step_microstep: 18.36
[2025-04-26 03:20:29,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.44 | bwd: 5719.11 | bwd_inner: 5647.70 | bwd_allreduce: 71.37 | step: 18.36

 19%|█▉        | 8025/41250 [19:22:55<79:52:14,  8.65s/it]
                                                          
{'loss': 0.1952, 'grad_norm': 2.072726011276245, 'learning_rate': 3.7226850917429826e-05, 'epoch': 1.95}

 19%|█▉        | 8025/41250 [19:22:55<79:52:14,  8.65s/it][2025-04-26 03:20:38,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 03:20:38,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.38 | bwd_microstep: 5746.49 | bwd_inner_microstep: 5715.08 | bwd_allreduce_microstep: 31.37 | step_microstep: 18.24
[2025-04-26 03:20:38,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.38 | bwd: 5746.50 | bwd_inner: 5715.08 | bwd_allreduce: 31.38 | step: 18.24

 19%|█▉        | 8026/41250 [19:23:03<79:57:15,  8.66s/it]
                                                          
{'loss': 0.1764, 'grad_norm': 2.5055482387542725, 'learning_rate': 3.722605310044631e-05, 'epoch': 1.95}

 19%|█▉        | 8026/41250 [19:23:03<79:57:15,  8.66s/it][2025-04-26 03:20:47,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:20:47,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.03 | bwd_microstep: 5697.65 | bwd_inner_microstep: 5636.58 | bwd_allreduce_microstep: 61.03 | step_microstep: 18.38
[2025-04-26 03:20:47,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.03 | bwd: 5697.66 | bwd_inner: 5636.58 | bwd_allreduce: 61.04 | step: 18.38

 19%|█▉        | 8027/41250 [19:23:12<79:49:45,  8.65s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 1.9101132154464722, 'learning_rate': 3.7225255177267565e-05, 'epoch': 1.95}

 19%|█▉        | 8027/41250 [19:23:12<79:49:45,  8.65s/it][2025-04-26 03:20:55,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 03:20:55,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5778.92 | bwd_inner_microstep: 5656.06 | bwd_allreduce_microstep: 122.81 | step_microstep: 19.25
[2025-04-26 03:20:55,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5778.93 | bwd_inner: 5656.06 | bwd_allreduce: 122.83 | step: 19.25

 19%|█▉        | 8028/41250 [19:23:21<79:56:12,  8.66s/it]
                                                          
{'loss': 0.1699, 'grad_norm': 2.2808969020843506, 'learning_rate': 3.722445714789851e-05, 'epoch': 1.95}

 19%|█▉        | 8028/41250 [19:23:21<79:56:12,  8.66s/it][2025-04-26 03:21:04,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:21:04,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.22 | bwd_microstep: 5708.83 | bwd_inner_microstep: 5696.11 | bwd_allreduce_microstep: 12.67 | step_microstep: 19.06
[2025-04-26 03:21:04,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.22 | bwd: 5708.85 | bwd_inner: 5696.11 | bwd_allreduce: 12.69 | step: 19.06

 19%|█▉        | 8029/41250 [19:23:29<79:53:40,  8.66s/it]
                                                          
{'loss': 0.1856, 'grad_norm': 2.5550527572631836, 'learning_rate': 3.7223659012344064e-05, 'epoch': 1.95}

 19%|█▉        | 8029/41250 [19:23:29<79:53:40,  8.66s/it][2025-04-26 03:21:13,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 03:21:13,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.94 | bwd_microstep: 5748.82 | bwd_inner_microstep: 5693.26 | bwd_allreduce_microstep: 55.51 | step_microstep: 19.19
[2025-04-26 03:21:13,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.94 | bwd: 5748.83 | bwd_inner: 5693.26 | bwd_allreduce: 55.53 | step: 19.19

 19%|█▉        | 8030/41250 [19:23:38<79:57:08,  8.66s/it]
                                                          
{'loss': 0.1718, 'grad_norm': 2.7720580101013184, 'learning_rate': 3.7222860770609145e-05, 'epoch': 1.95}

 19%|█▉        | 8030/41250 [19:23:38<79:57:08,  8.66s/it][2025-04-26 03:21:21,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:21:21,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.80 | bwd_microstep: 5723.08 | bwd_inner_microstep: 5639.14 | bwd_allreduce_microstep: 83.89 | step_microstep: 19.09
[2025-04-26 03:21:21,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.80 | bwd: 5723.10 | bwd_inner: 5639.14 | bwd_allreduce: 83.91 | step: 19.09

 19%|█▉        | 8031/41250 [19:23:47<79:51:08,  8.65s/it]
                                                          
{'loss': 0.2667, 'grad_norm': 3.5089969635009766, 'learning_rate': 3.722206242269869e-05, 'epoch': 1.95}

 19%|█▉        | 8031/41250 [19:23:47<79:51:08,  8.65s/it][2025-04-26 03:21:30,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:21:30,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5774.88 | bwd_inner_microstep: 5650.63 | bwd_allreduce_microstep: 124.20 | step_microstep: 18.73
[2025-04-26 03:21:30,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5774.89 | bwd_inner: 5650.63 | bwd_allreduce: 124.22 | step: 18.73

 19%|█▉        | 8032/41250 [19:23:55<79:56:42,  8.66s/it]
                                                          
{'loss': 0.1017, 'grad_norm': 2.54472017288208, 'learning_rate': 3.722126396861759e-05, 'epoch': 1.95}

 19%|█▉        | 8032/41250 [19:23:55<79:56:42,  8.66s/it][2025-04-26 03:21:39,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 03:21:39,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.14 | bwd_microstep: 5720.84 | bwd_inner_microstep: 5708.12 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.55
[2025-04-26 03:21:39,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.14 | bwd: 5720.86 | bwd_inner: 5708.12 | bwd_allreduce: 12.70 | step: 19.55

 19%|█▉        | 8033/41250 [19:24:04<79:54:38,  8.66s/it]
                                                          
{'loss': 0.2734, 'grad_norm': 1.4905842542648315, 'learning_rate': 3.72204654083708e-05, 'epoch': 1.95}

 19%|█▉        | 8033/41250 [19:24:04<79:54:38,  8.66s/it][2025-04-26 03:21:47,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:21:47,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.33 | bwd_microstep: 5777.22 | bwd_inner_microstep: 5641.98 | bwd_allreduce_microstep: 135.18 | step_microstep: 18.79
[2025-04-26 03:21:47,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.33 | bwd: 5777.24 | bwd_inner: 5641.98 | bwd_allreduce: 135.20 | step: 18.79

 19%|█▉        | 8034/41250 [19:24:13<79:59:51,  8.67s/it]
                                                          
{'loss': 0.0822, 'grad_norm': 1.0143953561782837, 'learning_rate': 3.7219666741963225e-05, 'epoch': 1.95}

 19%|█▉        | 8034/41250 [19:24:13<79:59:51,  8.67s/it][2025-04-26 03:21:56,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.15 | optimizer_step: 1.05
[2025-04-26 03:21:56,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5910.77 | bwd_inner_microstep: 5690.70 | bwd_allreduce_microstep: 220.01 | step_microstep: 20.13
[2025-04-26 03:21:56,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5910.79 | bwd_inner: 5690.70 | bwd_allreduce: 220.04 | step: 20.13

 19%|█▉        | 8035/41250 [19:24:21<80:29:00,  8.72s/it]
                                                          
{'loss': 0.2496, 'grad_norm': 2.00915789604187, 'learning_rate': 3.7218867969399786e-05, 'epoch': 1.95}

 19%|█▉        | 8035/41250 [19:24:21<80:29:00,  8.72s/it][2025-04-26 03:22:05,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 03:22:05,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.03 | bwd_microstep: 5712.77 | bwd_inner_microstep: 5656.64 | bwd_allreduce_microstep: 56.09 | step_microstep: 18.88
[2025-04-26 03:22:05,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.03 | bwd: 5712.78 | bwd_inner: 5656.64 | bwd_allreduce: 56.10 | step: 18.88

 19%|█▉        | 8036/41250 [19:24:30<80:14:15,  8.70s/it]
                                                          
{'loss': 0.1367, 'grad_norm': 2.8705270290374756, 'learning_rate': 3.721806909068541e-05, 'epoch': 1.95}

 19%|█▉        | 8036/41250 [19:24:30<80:14:15,  8.70s/it][2025-04-26 03:22:13,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 03:22:13,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.31 | bwd_microstep: 5721.24 | bwd_inner_microstep: 5654.47 | bwd_allreduce_microstep: 66.72 | step_microstep: 18.47
[2025-04-26 03:22:13,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.31 | bwd: 5721.25 | bwd_inner: 5654.47 | bwd_allreduce: 66.74 | step: 18.48

 19%|█▉        | 8037/41250 [19:24:39<80:03:58,  8.68s/it]
                                                          
{'loss': 0.0378, 'grad_norm': 0.3668220341205597, 'learning_rate': 3.721727010582503e-05, 'epoch': 1.95}

 19%|█▉        | 8037/41250 [19:24:39<80:03:58,  8.68s/it][2025-04-26 03:22:22,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 03:22:22,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5768.01 | bwd_inner_microstep: 5707.88 | bwd_allreduce_microstep: 60.08 | step_microstep: 18.29
[2025-04-26 03:22:22,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5768.02 | bwd_inner: 5707.88 | bwd_allreduce: 60.10 | step: 18.30

 19%|█▉        | 8038/41250 [19:24:47<80:07:25,  8.68s/it]
                                                          
{'loss': 0.0303, 'grad_norm': 0.5118327140808105, 'learning_rate': 3.7216471014823564e-05, 'epoch': 1.95}

 19%|█▉        | 8038/41250 [19:24:47<80:07:25,  8.68s/it][2025-04-26 03:22:31,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:22:31,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.51 | bwd_microstep: 5738.97 | bwd_inner_microstep: 5695.55 | bwd_allreduce_microstep: 43.37 | step_microstep: 18.55
[2025-04-26 03:22:31,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.51 | bwd: 5738.98 | bwd_inner: 5695.55 | bwd_allreduce: 43.39 | step: 18.55

 19%|█▉        | 8039/41250 [19:24:56<80:06:38,  8.68s/it]
                                                          
{'loss': 0.0625, 'grad_norm': 1.5809153318405151, 'learning_rate': 3.721567181768594e-05, 'epoch': 1.95}

 19%|█▉        | 8039/41250 [19:24:56<80:06:38,  8.68s/it][2025-04-26 03:22:39,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 03:22:39,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.45 | bwd_microstep: 5712.17 | bwd_inner_microstep: 5699.32 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.26
[2025-04-26 03:22:39,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.45 | bwd: 5712.18 | bwd_inner: 5699.32 | bwd_allreduce: 12.82 | step: 19.26

 19%|█▉        | 8040/41250 [19:25:05<80:00:53,  8.67s/it]
                                                          
{'loss': 0.1573, 'grad_norm': 2.904890775680542, 'learning_rate': 3.7214872514417094e-05, 'epoch': 1.95}

 19%|█▉        | 8040/41250 [19:25:05<80:00:53,  8.67s/it][2025-04-26 03:22:48,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:22:48,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.73 | bwd_microstep: 5811.52 | bwd_inner_microstep: 5653.62 | bwd_allreduce_microstep: 157.86 | step_microstep: 18.42
[2025-04-26 03:22:48,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.73 | bwd: 5811.53 | bwd_inner: 5653.62 | bwd_allreduce: 157.88 | step: 18.43

 19%|█▉        | 8041/41250 [19:25:13<80:07:45,  8.69s/it]
                                                          
{'loss': 0.0852, 'grad_norm': 0.7487773895263672, 'learning_rate': 3.7214073105021934e-05, 'epoch': 1.95}

 19%|█▉        | 8041/41250 [19:25:13<80:07:45,  8.69s/it][2025-04-26 03:22:57,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:22:57,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5696.62 | bwd_inner_microstep: 5684.16 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.75
[2025-04-26 03:22:57,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5696.63 | bwd_inner: 5684.16 | bwd_allreduce: 12.43 | step: 18.76

 19%|█▉        | 8042/41250 [19:25:22<79:57:24,  8.67s/it]
                                                          
{'loss': 0.0638, 'grad_norm': 1.1290607452392578, 'learning_rate': 3.7213273589505405e-05, 'epoch': 1.95}

 19%|█▉        | 8042/41250 [19:25:22<79:57:24,  8.67s/it][2025-04-26 03:23:05,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:23:05,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.29 | bwd_microstep: 5770.53 | bwd_inner_microstep: 5678.95 | bwd_allreduce_microstep: 91.54 | step_microstep: 18.37
[2025-04-26 03:23:05,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.29 | bwd: 5770.55 | bwd_inner: 5678.95 | bwd_allreduce: 91.55 | step: 18.37

 19%|█▉        | 8043/41250 [19:25:31<80:01:13,  8.68s/it]
                                                          
{'loss': 0.1335, 'grad_norm': 1.3906670808792114, 'learning_rate': 3.7212473967872425e-05, 'epoch': 1.95}

 19%|█▉        | 8043/41250 [19:25:31<80:01:13,  8.68s/it][2025-04-26 03:23:14,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-26 03:23:14,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.03 | bwd_microstep: 5895.09 | bwd_inner_microstep: 5641.29 | bwd_allreduce_microstep: 253.75 | step_microstep: 18.75
[2025-04-26 03:23:14,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.03 | bwd: 5895.11 | bwd_inner: 5641.29 | bwd_allreduce: 253.77 | step: 18.75

 20%|█▉        | 8044/41250 [19:25:40<80:22:04,  8.71s/it]
                                                          
{'loss': 0.0729, 'grad_norm': 2.741832733154297, 'learning_rate': 3.7211674240127935e-05, 'epoch': 1.95}

 20%|█▉        | 8044/41250 [19:25:40<80:22:04,  8.71s/it][2025-04-26 03:23:23,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.13 | optimizer_step: 1.13
[2025-04-26 03:23:23,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.16 | bwd_microstep: 5766.29 | bwd_inner_microstep: 5657.88 | bwd_allreduce_microstep: 108.35 | step_microstep: 19.33
[2025-04-26 03:23:23,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.16 | bwd: 5766.31 | bwd_inner: 5657.88 | bwd_allreduce: 108.38 | step: 19.33

 20%|█▉        | 8045/41250 [19:25:48<80:16:24,  8.70s/it]
                                                          
{'loss': 0.2955, 'grad_norm': 1.4689806699752808, 'learning_rate': 3.7210874406276855e-05, 'epoch': 1.95}

 20%|█▉        | 8045/41250 [19:25:48<80:16:24,  8.70s/it][2025-04-26 03:23:32,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:23:32,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.74 | bwd_microstep: 5747.39 | bwd_inner_microstep: 5658.29 | bwd_allreduce_microstep: 89.06 | step_microstep: 18.69
[2025-04-26 03:23:32,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.74 | bwd: 5747.40 | bwd_inner: 5658.29 | bwd_allreduce: 89.07 | step: 18.70

 20%|█▉        | 8046/41250 [19:25:57<80:10:23,  8.69s/it]
                                                          
{'loss': 0.0753, 'grad_norm': 1.7786375284194946, 'learning_rate': 3.7210074466324116e-05, 'epoch': 1.95}

 20%|█▉        | 8046/41250 [19:25:57<80:10:23,  8.69s/it][2025-04-26 03:23:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-26 03:23:40,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5732.97 | bwd_inner_microstep: 5706.91 | bwd_allreduce_microstep: 26.02 | step_microstep: 18.57
[2025-04-26 03:23:40,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5732.99 | bwd_inner: 5706.91 | bwd_allreduce: 26.04 | step: 18.57

 20%|█▉        | 8047/41250 [19:26:06<80:06:02,  8.68s/it]
                                                          
{'loss': 0.0997, 'grad_norm': 2.149657726287842, 'learning_rate': 3.720927442027466e-05, 'epoch': 1.95}

 20%|█▉        | 8047/41250 [19:26:06<80:06:02,  8.68s/it][2025-04-26 03:23:49,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:23:49,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5772.97 | bwd_inner_microstep: 5642.14 | bwd_allreduce_microstep: 130.78 | step_microstep: 18.48
[2025-04-26 03:23:49,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5772.98 | bwd_inner: 5642.14 | bwd_allreduce: 130.80 | step: 18.48

 20%|█▉        | 8048/41250 [19:26:14<80:04:59,  8.68s/it]
                                                          
{'loss': 0.0481, 'grad_norm': 1.0658848285675049, 'learning_rate': 3.720847426813341e-05, 'epoch': 1.95}

 20%|█▉        | 8048/41250 [19:26:14<80:04:59,  8.68s/it][2025-04-26 03:23:58,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 03:23:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.42 | bwd_microstep: 5737.52 | bwd_inner_microstep: 5684.27 | bwd_allreduce_microstep: 53.19 | step_microstep: 19.33
[2025-04-26 03:23:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.42 | bwd: 5737.53 | bwd_inner: 5684.27 | bwd_allreduce: 53.22 | step: 19.33

 20%|█▉        | 8049/41250 [19:26:23<80:02:57,  8.68s/it]
                                                          
{'loss': 0.1033, 'grad_norm': 3.4217469692230225, 'learning_rate': 3.720767400990529e-05, 'epoch': 1.95}

 20%|█▉        | 8049/41250 [19:26:23<80:02:57,  8.68s/it][2025-04-26 03:24:06,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.10
[2025-04-26 03:24:06,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.35 | bwd_microstep: 5774.99 | bwd_inner_microstep: 5645.35 | bwd_allreduce_microstep: 129.59 | step_microstep: 19.81
[2025-04-26 03:24:06,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.35 | bwd: 5775.01 | bwd_inner: 5645.35 | bwd_allreduce: 129.61 | step: 19.81

 20%|█▉        | 8050/41250 [19:26:32<80:04:15,  8.68s/it]
                                                          
{'loss': 0.0914, 'grad_norm': 1.4040884971618652, 'learning_rate': 3.720687364559526e-05, 'epoch': 1.95}

 20%|█▉        | 8050/41250 [19:26:32<80:04:15,  8.68s/it][2025-04-26 03:24:15,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:24:15,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.52 | bwd_microstep: 5738.35 | bwd_inner_microstep: 5652.84 | bwd_allreduce_microstep: 85.47 | step_microstep: 19.10
[2025-04-26 03:24:15,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.52 | bwd: 5738.36 | bwd_inner: 5652.84 | bwd_allreduce: 85.49 | step: 19.10

 20%|█▉        | 8051/41250 [19:26:40<79:57:57,  8.67s/it]
                                                          
{'loss': 0.2452, 'grad_norm': 3.26017427444458, 'learning_rate': 3.720607317520823e-05, 'epoch': 1.95}

 20%|█▉        | 8051/41250 [19:26:40<79:57:57,  8.67s/it][2025-04-26 03:24:24,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 03:24:24,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.83 | bwd_microstep: 5729.68 | bwd_inner_microstep: 5644.51 | bwd_allreduce_microstep: 85.12 | step_microstep: 18.88
[2025-04-26 03:24:24,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.83 | bwd: 5729.70 | bwd_inner: 5644.51 | bwd_allreduce: 85.14 | step: 18.88

 20%|█▉        | 8052/41250 [19:26:49<79:53:30,  8.66s/it]
                                                          
{'loss': 0.3179, 'grad_norm': 3.0283265113830566, 'learning_rate': 3.720527259874916e-05, 'epoch': 1.95}

 20%|█▉        | 8052/41250 [19:26:49<79:53:30,  8.66s/it][2025-04-26 03:24:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 03:24:32,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.64 | bwd_microstep: 5779.25 | bwd_inner_microstep: 5766.46 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.06
[2025-04-26 03:24:32,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.64 | bwd: 5779.27 | bwd_inner: 5766.46 | bwd_allreduce: 12.77 | step: 19.06

 20%|█▉        | 8053/41250 [19:26:58<80:06:36,  8.69s/it]
                                                          
{'loss': 0.2131, 'grad_norm': 3.1179492473602295, 'learning_rate': 3.720447191622294e-05, 'epoch': 1.95}

 20%|█▉        | 8053/41250 [19:26:58<80:06:36,  8.69s/it][2025-04-26 03:24:41,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 03:24:41,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.08 | bwd_microstep: 5727.35 | bwd_inner_microstep: 5688.41 | bwd_allreduce_microstep: 38.90 | step_microstep: 18.97
[2025-04-26 03:24:41,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.08 | bwd: 5727.36 | bwd_inner: 5688.41 | bwd_allreduce: 38.92 | step: 18.97

 20%|█▉        | 8054/41250 [19:27:06<80:00:50,  8.68s/it]
                                                          
{'loss': 0.0696, 'grad_norm': 0.940660297870636, 'learning_rate': 3.7203671127634555e-05, 'epoch': 1.95}

 20%|█▉        | 8054/41250 [19:27:06<80:00:50,  8.68s/it][2025-04-26 03:24:50,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:24:50,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 5750.41 | bwd_inner_microstep: 5639.62 | bwd_allreduce_microstep: 110.75 | step_microstep: 18.72
[2025-04-26 03:24:50,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 5750.42 | bwd_inner: 5639.62 | bwd_allreduce: 110.76 | step: 18.72

 20%|█▉        | 8055/41250 [19:27:15<79:57:56,  8.67s/it]
                                                          
{'loss': 0.1269, 'grad_norm': 1.7883996963500977, 'learning_rate': 3.7202870232988915e-05, 'epoch': 1.95}

 20%|█▉        | 8055/41250 [19:27:15<79:57:56,  8.67s/it][2025-04-26 03:24:58,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:24:58,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.48 | bwd_microstep: 5675.90 | bwd_inner_microstep: 5639.31 | bwd_allreduce_microstep: 36.55 | step_microstep: 18.78
[2025-04-26 03:24:58,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.49 | bwd: 5675.91 | bwd_inner: 5639.31 | bwd_allreduce: 36.56 | step: 18.79

 20%|█▉        | 8056/41250 [19:27:24<79:45:03,  8.65s/it]
                                                          
{'loss': 0.2067, 'grad_norm': 2.1056790351867676, 'learning_rate': 3.720206923229096e-05, 'epoch': 1.95}

 20%|█▉        | 8056/41250 [19:27:24<79:45:03,  8.65s/it][2025-04-26 03:25:07,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:25:07,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.00 | bwd_microstep: 5737.83 | bwd_inner_microstep: 5695.58 | bwd_allreduce_microstep: 42.21 | step_microstep: 18.65
[2025-04-26 03:25:07,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.00 | bwd: 5737.85 | bwd_inner: 5695.58 | bwd_allreduce: 42.23 | step: 18.65

 20%|█▉        | 8057/41250 [19:27:32<79:50:31,  8.66s/it]
                                                          
{'loss': 0.2265, 'grad_norm': 2.5936906337738037, 'learning_rate': 3.720126812554564e-05, 'epoch': 1.95}

 20%|█▉        | 8057/41250 [19:27:32<79:50:31,  8.66s/it][2025-04-26 03:25:16,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 03:25:16,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.67 | bwd_microstep: 5697.37 | bwd_inner_microstep: 5684.69 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.90
[2025-04-26 03:25:16,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.67 | bwd: 5697.39 | bwd_inner: 5684.69 | bwd_allreduce: 12.65 | step: 18.90

 20%|█▉        | 8058/41250 [19:27:41<79:45:43,  8.65s/it]
                                                          
{'loss': 0.1098, 'grad_norm': 1.814013957977295, 'learning_rate': 3.720046691275788e-05, 'epoch': 1.95}

 20%|█▉        | 8058/41250 [19:27:41<79:45:43,  8.65s/it][2025-04-26 03:25:24,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:25:24,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5732.50 | bwd_inner_microstep: 5686.26 | bwd_allreduce_microstep: 46.20 | step_microstep: 18.43
[2025-04-26 03:25:24,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5732.52 | bwd_inner: 5686.26 | bwd_allreduce: 46.22 | step: 18.44

 20%|█▉        | 8059/41250 [19:27:50<79:48:46,  8.66s/it]
                                                          
{'loss': 0.1098, 'grad_norm': 4.067841053009033, 'learning_rate': 3.7199665593932625e-05, 'epoch': 1.95}

 20%|█▉        | 8059/41250 [19:27:50<79:48:46,  8.66s/it][2025-04-26 03:25:33,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:25:33,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.99 | bwd_microstep: 5767.35 | bwd_inner_microstep: 5754.79 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.73
[2025-04-26 03:25:33,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.99 | bwd: 5767.36 | bwd_inner: 5754.79 | bwd_allreduce: 12.53 | step: 18.73

 20%|█▉        | 8060/41250 [19:27:58<80:01:37,  8.68s/it]
                                                          
{'loss': 0.0989, 'grad_norm': 2.0558574199676514, 'learning_rate': 3.719886416907481e-05, 'epoch': 1.95}

 20%|█▉        | 8060/41250 [19:27:58<80:01:37,  8.68s/it][2025-04-26 03:25:42,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:25:42,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.02 | bwd_microstep: 5736.06 | bwd_inner_microstep: 5693.13 | bwd_allreduce_microstep: 42.88 | step_microstep: 18.26
[2025-04-26 03:25:42,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.02 | bwd: 5736.07 | bwd_inner: 5693.13 | bwd_allreduce: 42.90 | step: 18.26

 20%|█▉        | 8061/41250 [19:28:07<79:59:25,  8.68s/it]
                                                          
{'loss': 0.1619, 'grad_norm': 3.383626937866211, 'learning_rate': 3.7198062638189386e-05, 'epoch': 1.95}

 20%|█▉        | 8061/41250 [19:28:07<79:59:25,  8.68s/it][2025-04-26 03:25:50,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:25:50,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.97 | bwd_microstep: 5766.65 | bwd_inner_microstep: 5645.04 | bwd_allreduce_microstep: 121.58 | step_microstep: 18.22
[2025-04-26 03:25:50,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5766.67 | bwd_inner: 5645.03 | bwd_allreduce: 121.59 | step: 18.22

 20%|█▉        | 8062/41250 [19:28:16<79:59:55,  8.68s/it]
                                                          
{'loss': 0.113, 'grad_norm': 2.1355772018432617, 'learning_rate': 3.719726100128129e-05, 'epoch': 1.95}

 20%|█▉        | 8062/41250 [19:28:16<79:59:55,  8.68s/it][2025-04-26 03:25:59,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 03:25:59,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.69 | bwd_microstep: 5760.89 | bwd_inner_microstep: 5687.82 | bwd_allreduce_microstep: 73.02 | step_microstep: 18.70
[2025-04-26 03:25:59,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.69 | bwd: 5760.91 | bwd_inner: 5687.82 | bwd_allreduce: 73.04 | step: 18.70

 20%|█▉        | 8063/41250 [19:28:24<80:02:32,  8.68s/it]
                                                          
{'loss': 0.269, 'grad_norm': 3.9537465572357178, 'learning_rate': 3.719645925835546e-05, 'epoch': 1.95}

 20%|█▉        | 8063/41250 [19:28:24<80:02:32,  8.68s/it][2025-04-26 03:26:08,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 03:26:08,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.42 | bwd_microstep: 5790.90 | bwd_inner_microstep: 5654.32 | bwd_allreduce_microstep: 136.53 | step_microstep: 19.32
[2025-04-26 03:26:08,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.42 | bwd: 5790.92 | bwd_inner: 5654.32 | bwd_allreduce: 136.55 | step: 19.32

 20%|█▉        | 8064/41250 [19:28:33<80:05:19,  8.69s/it]
                                                          
{'loss': 0.1023, 'grad_norm': 1.46503484249115, 'learning_rate': 3.719565740941684e-05, 'epoch': 1.95}

 20%|█▉        | 8064/41250 [19:28:33<80:05:19,  8.69s/it][2025-04-26 03:26:16,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:26:16,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.29 | bwd_microstep: 5714.42 | bwd_inner_microstep: 5632.32 | bwd_allreduce_microstep: 82.05 | step_microstep: 18.71
[2025-04-26 03:26:16,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.29 | bwd: 5714.44 | bwd_inner: 5632.32 | bwd_allreduce: 82.07 | step: 18.71

 20%|█▉        | 8065/41250 [19:28:42<79:55:29,  8.67s/it]
                                                          
{'loss': 0.0382, 'grad_norm': 1.5900369882583618, 'learning_rate': 3.719485545447038e-05, 'epoch': 1.96}

 20%|█▉        | 8065/41250 [19:28:42<79:55:29,  8.67s/it][2025-04-26 03:26:25,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:26:25,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.86 | bwd_microstep: 5782.89 | bwd_inner_microstep: 5645.20 | bwd_allreduce_microstep: 137.62 | step_microstep: 18.78
[2025-04-26 03:26:25,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.86 | bwd: 5782.91 | bwd_inner: 5645.20 | bwd_allreduce: 137.65 | step: 18.78

 20%|█▉        | 8066/41250 [19:28:50<79:58:55,  8.68s/it]
                                                          
{'loss': 0.0486, 'grad_norm': 0.935218334197998, 'learning_rate': 3.719405339352101e-05, 'epoch': 1.96}

 20%|█▉        | 8066/41250 [19:28:50<79:58:55,  8.68s/it][2025-04-26 03:26:34,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 03:26:34,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.77 | bwd_microstep: 5707.57 | bwd_inner_microstep: 5643.24 | bwd_allreduce_microstep: 64.29 | step_microstep: 18.92
[2025-04-26 03:26:34,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.77 | bwd: 5707.59 | bwd_inner: 5643.24 | bwd_allreduce: 64.30 | step: 18.92

 20%|█▉        | 8067/41250 [19:28:59<79:51:55,  8.66s/it]
                                                          
{'loss': 0.0659, 'grad_norm': 2.061981201171875, 'learning_rate': 3.7193251226573684e-05, 'epoch': 1.96}

 20%|█▉        | 8067/41250 [19:28:59<79:51:55,  8.66s/it][2025-04-26 03:26:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:26:42,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.62 | bwd_microstep: 5762.71 | bwd_inner_microstep: 5657.83 | bwd_allreduce_microstep: 104.84 | step_microstep: 18.37
[2025-04-26 03:26:42,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5762.73 | bwd_inner: 5657.83 | bwd_allreduce: 104.86 | step: 18.37

 20%|█▉        | 8068/41250 [19:29:08<79:52:54,  8.67s/it]
                                                          
{'loss': 0.038, 'grad_norm': 0.8313356637954712, 'learning_rate': 3.719244895363335e-05, 'epoch': 1.96}

 20%|█▉        | 8068/41250 [19:29:08<79:52:54,  8.67s/it][2025-04-26 03:26:51,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:26:51,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.22 | bwd_microstep: 5722.76 | bwd_inner_microstep: 5709.80 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.62
[2025-04-26 03:26:51,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.22 | bwd: 5722.78 | bwd_inner: 5709.80 | bwd_allreduce: 12.94 | step: 18.62

 20%|█▉        | 8069/41250 [19:29:16<79:55:19,  8.67s/it]
                                                          
{'loss': 0.283, 'grad_norm': 2.7161600589752197, 'learning_rate': 3.719164657470495e-05, 'epoch': 1.96}

 20%|█▉        | 8069/41250 [19:29:16<79:55:19,  8.67s/it][2025-04-26 03:27:00,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:27:00,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.42 | bwd_microstep: 5695.88 | bwd_inner_microstep: 5683.36 | bwd_allreduce_microstep: 12.47 | step_microstep: 18.67
[2025-04-26 03:27:00,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.42 | bwd: 5695.89 | bwd_inner: 5683.36 | bwd_allreduce: 12.49 | step: 18.67

 20%|█▉        | 8070/41250 [19:29:25<79:48:04,  8.66s/it]
                                                          
{'loss': 0.0431, 'grad_norm': 0.9957391023635864, 'learning_rate': 3.719084408979343e-05, 'epoch': 1.96}

 20%|█▉        | 8070/41250 [19:29:25<79:48:04,  8.66s/it][2025-04-26 03:27:08,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 03:27:08,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.36 | bwd_microstep: 5746.73 | bwd_inner_microstep: 5705.37 | bwd_allreduce_microstep: 41.32 | step_microstep: 19.15
[2025-04-26 03:27:08,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.36 | bwd: 5746.75 | bwd_inner: 5705.37 | bwd_allreduce: 41.34 | step: 19.15

 20%|█▉        | 8071/41250 [19:29:34<79:51:57,  8.67s/it]
                                                          
{'loss': 0.2833, 'grad_norm': 3.3107197284698486, 'learning_rate': 3.7190041498903734e-05, 'epoch': 1.96}

 20%|█▉        | 8071/41250 [19:29:34<79:51:57,  8.67s/it][2025-04-26 03:27:17,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:27:17,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.41 | bwd_microstep: 5770.92 | bwd_inner_microstep: 5707.43 | bwd_allreduce_microstep: 63.44 | step_microstep: 18.96
[2025-04-26 03:27:17,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.41 | bwd: 5770.93 | bwd_inner: 5707.43 | bwd_allreduce: 63.46 | step: 18.96

 20%|█▉        | 8072/41250 [19:29:42<79:59:25,  8.68s/it]
                                                          
{'loss': 0.2643, 'grad_norm': 1.9520643949508667, 'learning_rate': 3.718923880204081e-05, 'epoch': 1.96}

 20%|█▉        | 8072/41250 [19:29:42<79:59:25,  8.68s/it][2025-04-26 03:27:26,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 03:27:26,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.78 | bwd_microstep: 5712.26 | bwd_inner_microstep: 5699.47 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.01
[2025-04-26 03:27:26,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.78 | bwd: 5712.28 | bwd_inner: 5699.47 | bwd_allreduce: 12.77 | step: 19.02

 20%|█▉        | 8073/41250 [19:29:51<79:53:48,  8.67s/it]
                                                          
{'loss': 0.1703, 'grad_norm': 1.2667462825775146, 'learning_rate': 3.718843599920962e-05, 'epoch': 1.96}

 20%|█▉        | 8073/41250 [19:29:51<79:53:48,  8.67s/it][2025-04-26 03:27:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:27:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.07 | bwd_microstep: 5761.83 | bwd_inner_microstep: 5702.21 | bwd_allreduce_microstep: 59.58 | step_microstep: 18.43
[2025-04-26 03:27:34,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.07 | bwd: 5761.85 | bwd_inner: 5702.21 | bwd_allreduce: 59.60 | step: 18.43

 20%|█▉        | 8074/41250 [19:30:00<79:59:18,  8.68s/it]
                                                          
{'loss': 0.1212, 'grad_norm': 1.6826127767562866, 'learning_rate': 3.718763309041509e-05, 'epoch': 1.96}

 20%|█▉        | 8074/41250 [19:30:00<79:59:18,  8.68s/it][2025-04-26 03:27:43,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:27:43,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.89 | bwd_microstep: 5805.87 | bwd_inner_microstep: 5656.86 | bwd_allreduce_microstep: 148.97 | step_microstep: 19.08
[2025-04-26 03:27:43,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.89 | bwd: 5805.89 | bwd_inner: 5656.86 | bwd_allreduce: 148.99 | step: 19.08

 20%|█▉        | 8075/41250 [19:30:08<80:05:59,  8.69s/it]
                                                          
{'loss': 0.0785, 'grad_norm': 1.4023244380950928, 'learning_rate': 3.71868300756622e-05, 'epoch': 1.96}

 20%|█▉        | 8075/41250 [19:30:08<80:05:59,  8.69s/it][2025-04-26 03:27:52,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 03:27:52,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.15 | bwd_microstep: 5771.48 | bwd_inner_microstep: 5704.90 | bwd_allreduce_microstep: 66.54 | step_microstep: 19.11
[2025-04-26 03:27:52,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.15 | bwd: 5771.49 | bwd_inner: 5704.90 | bwd_allreduce: 66.56 | step: 19.11

 20%|█▉        | 8076/41250 [19:30:17<80:09:54,  8.70s/it]
                                                          
{'loss': 0.0369, 'grad_norm': 1.3606973886489868, 'learning_rate': 3.7186026954955865e-05, 'epoch': 1.96}

 20%|█▉        | 8076/41250 [19:30:17<80:09:54,  8.70s/it][2025-04-26 03:28:01,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:28:01,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.25 | bwd_microstep: 5752.31 | bwd_inner_microstep: 5715.30 | bwd_allreduce_microstep: 36.96 | step_microstep: 18.57
[2025-04-26 03:28:01,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5752.32 | bwd_inner: 5715.30 | bwd_allreduce: 36.98 | step: 18.57

 20%|█▉        | 8077/41250 [19:30:26<80:07:23,  8.70s/it]
                                                          
{'loss': 0.0397, 'grad_norm': 0.6943707466125488, 'learning_rate': 3.718522372830107e-05, 'epoch': 1.96}

 20%|█▉        | 8077/41250 [19:30:26<80:07:23,  8.70s/it][2025-04-26 03:28:09,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:28:09,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5720.29 | bwd_inner_microstep: 5669.13 | bwd_allreduce_microstep: 51.12 | step_microstep: 18.33
[2025-04-26 03:28:09,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5720.31 | bwd_inner: 5669.13 | bwd_allreduce: 51.14 | step: 18.33

 20%|█▉        | 8078/41250 [19:30:35<79:56:35,  8.68s/it]
                                                          
{'loss': 0.1795, 'grad_norm': 1.7341254949569702, 'learning_rate': 3.718442039570274e-05, 'epoch': 1.96}

 20%|█▉        | 8078/41250 [19:30:35<79:56:35,  8.68s/it][2025-04-26 03:28:18,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:28:18,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.28 | bwd_microstep: 5760.90 | bwd_inner_microstep: 5706.71 | bwd_allreduce_microstep: 54.15 | step_microstep: 18.69
[2025-04-26 03:28:18,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.28 | bwd: 5760.91 | bwd_inner: 5706.71 | bwd_allreduce: 54.16 | step: 18.69

 20%|█▉        | 8079/41250 [19:30:43<80:02:30,  8.69s/it]
                                                          
{'loss': 0.0443, 'grad_norm': 1.047080159187317, 'learning_rate': 3.718361695716584e-05, 'epoch': 1.96}

 20%|█▉        | 8079/41250 [19:30:43<80:02:30,  8.69s/it][2025-04-26 03:28:27,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:28:27,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5781.09 | bwd_inner_microstep: 5709.06 | bwd_allreduce_microstep: 71.98 | step_microstep: 18.66
[2025-04-26 03:28:27,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5781.10 | bwd_inner: 5709.06 | bwd_allreduce: 72.00 | step: 18.66

 20%|█▉        | 8080/41250 [19:30:52<80:07:05,  8.70s/it]
                                                          
{'loss': 0.0698, 'grad_norm': 2.0658302307128906, 'learning_rate': 3.7182813412695324e-05, 'epoch': 1.96}

 20%|█▉        | 8080/41250 [19:30:52<80:07:05,  8.70s/it][2025-04-26 03:28:35,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:28:35,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.17 | bwd_microstep: 5794.89 | bwd_inner_microstep: 5650.05 | bwd_allreduce_microstep: 144.79 | step_microstep: 18.67
[2025-04-26 03:28:35,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.17 | bwd: 5794.90 | bwd_inner: 5650.05 | bwd_allreduce: 144.81 | step: 18.67

 20%|█▉        | 8081/41250 [19:31:01<80:09:10,  8.70s/it]
                                                          
{'loss': 0.0659, 'grad_norm': 1.2712328433990479, 'learning_rate': 3.718200976229614e-05, 'epoch': 1.96}

 20%|█▉        | 8081/41250 [19:31:01<80:09:10,  8.70s/it][2025-04-26 03:28:44,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:28:44,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5765.02 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 70.17 | step_microstep: 18.61
[2025-04-26 03:28:44,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5765.03 | bwd_inner: 5694.79 | bwd_allreduce: 70.19 | step: 18.62

 20%|█▉        | 8082/41250 [19:31:09<80:08:56,  8.70s/it]
                                                          
{'loss': 0.1048, 'grad_norm': 2.0721371173858643, 'learning_rate': 3.718120600597325e-05, 'epoch': 1.96}

 20%|█▉        | 8082/41250 [19:31:09<80:08:56,  8.70s/it][2025-04-26 03:28:53,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 03:28:53,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.10 | bwd_microstep: 5793.17 | bwd_inner_microstep: 5690.32 | bwd_allreduce_microstep: 102.81 | step_microstep: 19.17
[2025-04-26 03:28:53,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.10 | bwd: 5793.18 | bwd_inner: 5690.32 | bwd_allreduce: 102.82 | step: 19.18

 20%|█▉        | 8083/41250 [19:31:18<80:12:38,  8.71s/it]
                                                          
{'loss': 0.3915, 'grad_norm': 3.8365774154663086, 'learning_rate': 3.71804021437316e-05, 'epoch': 1.96}

 20%|█▉        | 8083/41250 [19:31:18<80:12:38,  8.71s/it][2025-04-26 03:29:01,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:29:01,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.32 | bwd_microstep: 5787.22 | bwd_inner_microstep: 5651.52 | bwd_allreduce_microstep: 135.65 | step_microstep: 18.35
[2025-04-26 03:29:01,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.32 | bwd: 5787.23 | bwd_inner: 5651.52 | bwd_allreduce: 135.67 | step: 18.35

 20%|█▉        | 8084/41250 [19:31:27<80:10:19,  8.70s/it]
                                                          
{'loss': 0.0846, 'grad_norm': 4.437922954559326, 'learning_rate': 3.717959817557615e-05, 'epoch': 1.96}

 20%|█▉        | 8084/41250 [19:31:27<80:10:19,  8.70s/it][2025-04-26 03:29:10,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:29:10,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.24 | bwd_microstep: 5771.63 | bwd_inner_microstep: 5642.66 | bwd_allreduce_microstep: 128.93 | step_microstep: 18.54
[2025-04-26 03:29:10,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.24 | bwd: 5771.65 | bwd_inner: 5642.66 | bwd_allreduce: 128.95 | step: 18.54

 20%|█▉        | 8085/41250 [19:31:35<80:06:38,  8.70s/it]
                                                          
{'loss': 0.1522, 'grad_norm': 3.157036542892456, 'learning_rate': 3.717879410151186e-05, 'epoch': 1.96}

 20%|█▉        | 8085/41250 [19:31:35<80:06:38,  8.70s/it][2025-04-26 03:29:19,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:29:19,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.30 | bwd_microstep: 5729.54 | bwd_inner_microstep: 5644.22 | bwd_allreduce_microstep: 85.27 | step_microstep: 19.06
[2025-04-26 03:29:19,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.30 | bwd: 5729.56 | bwd_inner: 5644.22 | bwd_allreduce: 85.29 | step: 19.07

 20%|█▉        | 8086/41250 [19:31:44<79:55:54,  8.68s/it]
                                                          
{'loss': 0.2163, 'grad_norm': 2.6744697093963623, 'learning_rate': 3.717798992154368e-05, 'epoch': 1.96}

 20%|█▉        | 8086/41250 [19:31:44<79:55:54,  8.68s/it][2025-04-26 03:29:27,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:29:27,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.73 | bwd_microstep: 5787.05 | bwd_inner_microstep: 5648.39 | bwd_allreduce_microstep: 138.62 | step_microstep: 18.40
[2025-04-26 03:29:27,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.73 | bwd: 5787.07 | bwd_inner: 5648.39 | bwd_allreduce: 138.64 | step: 18.40

 20%|█▉        | 8087/41250 [19:31:53<79:59:01,  8.68s/it]
                                                          
{'loss': 0.0313, 'grad_norm': 0.9520822763442993, 'learning_rate': 3.717718563567657e-05, 'epoch': 1.96}

 20%|█▉        | 8087/41250 [19:31:53<79:59:01,  8.68s/it][2025-04-26 03:29:36,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:29:36,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.47 | bwd_microstep: 5764.05 | bwd_inner_microstep: 5683.47 | bwd_allreduce_microstep: 80.54 | step_microstep: 18.30
[2025-04-26 03:29:36,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.47 | bwd: 5764.07 | bwd_inner: 5683.47 | bwd_allreduce: 80.56 | step: 18.30

 20%|█▉        | 8088/41250 [19:32:01<80:00:20,  8.69s/it]
                                                          
{'loss': 0.0533, 'grad_norm': 1.135956048965454, 'learning_rate': 3.717638124391549e-05, 'epoch': 1.96}

 20%|█▉        | 8088/41250 [19:32:01<80:00:20,  8.69s/it][2025-04-26 03:29:45,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:29:45,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5706.53 | bwd_inner_microstep: 5693.89 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.31
[2025-04-26 03:29:45,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5706.54 | bwd_inner: 5693.89 | bwd_allreduce: 12.62 | step: 18.31

 20%|█▉        | 8089/41250 [19:32:10<79:52:02,  8.67s/it]
                                                          
{'loss': 0.2255, 'grad_norm': 3.5701375007629395, 'learning_rate': 3.71755767462654e-05, 'epoch': 1.96}

 20%|█▉        | 8089/41250 [19:32:10<79:52:02,  8.67s/it][2025-04-26 03:29:53,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:29:53,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5685.74 | bwd_inner_microstep: 5646.88 | bwd_allreduce_microstep: 38.81 | step_microstep: 18.66
[2025-04-26 03:29:53,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5685.75 | bwd_inner: 5646.88 | bwd_allreduce: 38.83 | step: 18.66

 20%|█▉        | 8090/41250 [19:32:19<79:39:39,  8.65s/it]
                                                          
{'loss': 0.0481, 'grad_norm': 2.518338680267334, 'learning_rate': 3.717477214273126e-05, 'epoch': 1.96}

 20%|█▉        | 8090/41250 [19:32:19<79:39:39,  8.65s/it][2025-04-26 03:30:02,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.15 | optimizer_step: 1.05
[2025-04-26 03:30:02,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.61 | bwd_microstep: 5763.73 | bwd_inner_microstep: 5684.69 | bwd_allreduce_microstep: 78.97 | step_microstep: 19.88
[2025-04-26 03:30:02,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.61 | bwd: 5763.75 | bwd_inner: 5684.69 | bwd_allreduce: 79.00 | step: 19.89

 20%|█▉        | 8091/41250 [19:32:27<79:47:58,  8.66s/it]
                                                          
{'loss': 0.1829, 'grad_norm': 1.4360474348068237, 'learning_rate': 3.717396743331802e-05, 'epoch': 1.96}

 20%|█▉        | 8091/41250 [19:32:27<79:47:58,  8.66s/it][2025-04-26 03:30:11,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 03:30:11,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.51 | bwd_microstep: 5699.64 | bwd_inner_microstep: 5649.73 | bwd_allreduce_microstep: 49.87 | step_microstep: 18.92
[2025-04-26 03:30:11,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.51 | bwd: 5699.66 | bwd_inner: 5649.73 | bwd_allreduce: 49.88 | step: 18.92

 20%|█▉        | 8092/41250 [19:32:36<79:39:03,  8.65s/it]
                                                          
{'loss': 0.104, 'grad_norm': 1.4138959646224976, 'learning_rate': 3.717316261803065e-05, 'epoch': 1.96}

 20%|█▉        | 8092/41250 [19:32:36<79:39:03,  8.65s/it][2025-04-26 03:30:19,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.22 | optimizer_step: 0.96
[2025-04-26 03:30:19,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.89 | bwd_microstep: 5786.60 | bwd_inner_microstep: 5680.61 | bwd_allreduce_microstep: 105.94 | step_microstep: 19.33
[2025-04-26 03:30:19,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.89 | bwd: 5786.62 | bwd_inner: 5680.61 | bwd_allreduce: 105.96 | step: 19.34

 20%|█▉        | 8093/41250 [19:32:45<79:49:40,  8.67s/it]
                                                          
{'loss': 0.1964, 'grad_norm': 2.300931453704834, 'learning_rate': 3.717235769687412e-05, 'epoch': 1.96}

 20%|█▉        | 8093/41250 [19:32:45<79:49:40,  8.67s/it][2025-04-26 03:30:28,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.30 | optimizer_step: 1.05
[2025-04-26 03:30:28,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.74 | bwd_microstep: 5709.00 | bwd_inner_microstep: 5694.99 | bwd_allreduce_microstep: 13.95 | step_microstep: 20.01
[2025-04-26 03:30:28,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.74 | bwd: 5709.02 | bwd_inner: 5694.99 | bwd_allreduce: 13.98 | step: 20.01

 20%|█▉        | 8094/41250 [19:32:53<79:45:48,  8.66s/it]
                                                          
{'loss': 0.1533, 'grad_norm': 2.2575340270996094, 'learning_rate': 3.717155266985337e-05, 'epoch': 1.96}

 20%|█▉        | 8094/41250 [19:32:53<79:45:48,  8.66s/it][2025-04-26 03:30:37,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 03:30:37,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.76 | bwd_microstep: 5694.94 | bwd_inner_microstep: 5649.68 | bwd_allreduce_microstep: 45.22 | step_microstep: 18.68
[2025-04-26 03:30:37,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.76 | bwd: 5694.95 | bwd_inner: 5649.68 | bwd_allreduce: 45.24 | step: 18.68

 20%|█▉        | 8095/41250 [19:33:02<79:35:49,  8.64s/it]
                                                          
{'loss': 0.2047, 'grad_norm': 2.2775027751922607, 'learning_rate': 3.717074753697338e-05, 'epoch': 1.96}

 20%|█▉        | 8095/41250 [19:33:02<79:35:49,  8.64s/it][2025-04-26 03:30:45,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:30:45,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.86 | bwd_microstep: 5688.80 | bwd_inner_microstep: 5645.25 | bwd_allreduce_microstep: 43.50 | step_microstep: 18.59
[2025-04-26 03:30:45,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.86 | bwd: 5688.81 | bwd_inner: 5645.25 | bwd_allreduce: 43.52 | step: 18.59

 20%|█▉        | 8096/41250 [19:33:11<79:27:56,  8.63s/it]
                                                          
{'loss': 0.1034, 'grad_norm': 1.9214829206466675, 'learning_rate': 3.716994229823911e-05, 'epoch': 1.96}

 20%|█▉        | 8096/41250 [19:33:11<79:27:56,  8.63s/it][2025-04-26 03:30:54,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:30:54,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.21 | bwd_microstep: 5701.73 | bwd_inner_microstep: 5643.66 | bwd_allreduce_microstep: 58.02 | step_microstep: 18.56
[2025-04-26 03:30:54,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.21 | bwd: 5701.74 | bwd_inner: 5643.66 | bwd_allreduce: 58.04 | step: 18.56

 20%|█▉        | 8097/41250 [19:33:19<79:24:27,  8.62s/it]
                                                          
{'loss': 0.1646, 'grad_norm': 3.425051212310791, 'learning_rate': 3.7169136953655525e-05, 'epoch': 1.96}

 20%|█▉        | 8097/41250 [19:33:19<79:24:27,  8.62s/it][2025-04-26 03:31:03,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.07 | optimizer_step: 1.18
[2025-04-26 03:31:03,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.38 | bwd_microstep: 5874.89 | bwd_inner_microstep: 5675.70 | bwd_allreduce_microstep: 199.14 | step_microstep: 19.59
[2025-04-26 03:31:03,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.38 | bwd: 5874.91 | bwd_inner: 5675.70 | bwd_allreduce: 199.16 | step: 19.59

 20%|█▉        | 8098/41250 [19:33:28<79:53:17,  8.68s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 1.5338010787963867, 'learning_rate': 3.7168331503227586e-05, 'epoch': 1.96}

 20%|█▉        | 8098/41250 [19:33:28<79:53:17,  8.68s/it][2025-04-26 03:31:11,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:31:11,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.64 | bwd_microstep: 5678.00 | bwd_inner_microstep: 5643.73 | bwd_allreduce_microstep: 34.23 | step_microstep: 18.63
[2025-04-26 03:31:11,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.64 | bwd: 5678.01 | bwd_inner: 5643.73 | bwd_allreduce: 34.24 | step: 18.63

 20%|█▉        | 8099/41250 [19:33:37<79:38:41,  8.65s/it]
                                                          
{'loss': 0.1838, 'grad_norm': 3.030965805053711, 'learning_rate': 3.716752594696026e-05, 'epoch': 1.96}

 20%|█▉        | 8099/41250 [19:33:37<79:38:41,  8.65s/it][2025-04-26 03:31:20,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 03:31:20,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5754.34 | bwd_inner_microstep: 5651.73 | bwd_allreduce_microstep: 102.56 | step_microstep: 19.16
[2025-04-26 03:31:20,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5754.36 | bwd_inner: 5651.73 | bwd_allreduce: 102.58 | step: 19.17

 20%|█▉        | 8100/41250 [19:33:45<79:40:38,  8.65s/it]
                                                          
{'loss': 0.0891, 'grad_norm': 1.5660358667373657, 'learning_rate': 3.7166720284858514e-05, 'epoch': 1.96}

 20%|█▉        | 8100/41250 [19:33:45<79:40:38,  8.65s/it][2025-04-26 03:31:28,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:31:28,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.54 | bwd_microstep: 5700.44 | bwd_inner_microstep: 5642.73 | bwd_allreduce_microstep: 57.66 | step_microstep: 18.81
[2025-04-26 03:31:28,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.54 | bwd: 5700.46 | bwd_inner: 5642.73 | bwd_allreduce: 57.68 | step: 18.81

 20%|█▉        | 8101/41250 [19:33:54<79:33:03,  8.64s/it]
                                                          
{'loss': 0.0397, 'grad_norm': 0.9159672856330872, 'learning_rate': 3.716591451692731e-05, 'epoch': 1.96}

 20%|█▉        | 8101/41250 [19:33:54<79:33:03,  8.64s/it][2025-04-26 03:31:37,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 03:31:37,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.43 | bwd_microstep: 5741.13 | bwd_inner_microstep: 5678.87 | bwd_allreduce_microstep: 62.22 | step_microstep: 19.42
[2025-04-26 03:31:37,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.43 | bwd: 5741.15 | bwd_inner: 5678.87 | bwd_allreduce: 62.24 | step: 19.42

 20%|█▉        | 8102/41250 [19:34:02<79:37:51,  8.65s/it]
                                                          
{'loss': 0.1749, 'grad_norm': 2.734135866165161, 'learning_rate': 3.716510864317162e-05, 'epoch': 1.96}

 20%|█▉        | 8102/41250 [19:34:02<79:37:51,  8.65s/it][2025-04-26 03:31:46,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 03:31:46,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.75 | bwd_microstep: 5705.97 | bwd_inner_microstep: 5693.23 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.58
[2025-04-26 03:31:46,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.75 | bwd: 5705.98 | bwd_inner: 5693.23 | bwd_allreduce: 12.71 | step: 18.58

 20%|█▉        | 8103/41250 [19:34:11<79:36:38,  8.65s/it]
                                                          
{'loss': 0.2181, 'grad_norm': 2.2048661708831787, 'learning_rate': 3.7164302663596414e-05, 'epoch': 1.96}

 20%|█▉        | 8103/41250 [19:34:11<79:36:38,  8.65s/it][2025-04-26 03:31:54,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 03:31:54,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.03 | bwd_microstep: 5746.62 | bwd_inner_microstep: 5650.39 | bwd_allreduce_microstep: 96.19 | step_microstep: 18.41
[2025-04-26 03:31:54,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.03 | bwd: 5746.64 | bwd_inner: 5650.39 | bwd_allreduce: 96.20 | step: 18.41

 20%|█▉        | 8104/41250 [19:34:20<79:37:58,  8.65s/it]
                                                          
{'loss': 0.1634, 'grad_norm': 2.424443006515503, 'learning_rate': 3.716349657820666e-05, 'epoch': 1.96}

 20%|█▉        | 8104/41250 [19:34:20<79:37:58,  8.65s/it][2025-04-26 03:32:03,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 03:32:03,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.22 | bwd_microstep: 5697.17 | bwd_inner_microstep: 5643.38 | bwd_allreduce_microstep: 53.73 | step_microstep: 18.52
[2025-04-26 03:32:03,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.22 | bwd: 5697.18 | bwd_inner: 5643.38 | bwd_allreduce: 53.75 | step: 18.52

 20%|█▉        | 8105/41250 [19:34:28<79:31:53,  8.64s/it]
                                                          
{'loss': 0.027, 'grad_norm': 0.6579890847206116, 'learning_rate': 3.7162690387007316e-05, 'epoch': 1.96}

 20%|█▉        | 8105/41250 [19:34:28<79:31:53,  8.64s/it][2025-04-26 03:32:12,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 03:32:12,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.91 | bwd_microstep: 5763.33 | bwd_inner_microstep: 5653.35 | bwd_allreduce_microstep: 109.93 | step_microstep: 19.13
[2025-04-26 03:32:12,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.91 | bwd: 5763.34 | bwd_inner: 5653.35 | bwd_allreduce: 109.95 | step: 19.13

 20%|█▉        | 8106/41250 [19:34:37<79:38:05,  8.65s/it]
                                                          
{'loss': 0.1989, 'grad_norm': 1.873679280281067, 'learning_rate': 3.716188409000337e-05, 'epoch': 1.97}

 20%|█▉        | 8106/41250 [19:34:37<79:38:05,  8.65s/it][2025-04-26 03:32:20,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:32:20,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.50 | bwd_microstep: 5720.41 | bwd_inner_microstep: 5638.52 | bwd_allreduce_microstep: 81.84 | step_microstep: 18.72
[2025-04-26 03:32:20,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.50 | bwd: 5720.42 | bwd_inner: 5638.52 | bwd_allreduce: 81.86 | step: 18.73

 20%|█▉        | 8107/41250 [19:34:46<79:35:57,  8.65s/it]
                                                          
{'loss': 0.1091, 'grad_norm': 1.9352002143859863, 'learning_rate': 3.7161077687199784e-05, 'epoch': 1.97}

 20%|█▉        | 8107/41250 [19:34:46<79:35:57,  8.65s/it][2025-04-26 03:32:29,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 0.98
[2025-04-26 03:32:29,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.99 | bwd_microstep: 5696.73 | bwd_inner_microstep: 5652.95 | bwd_allreduce_microstep: 43.73 | step_microstep: 19.22
[2025-04-26 03:32:29,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.99 | bwd: 5696.74 | bwd_inner: 5652.95 | bwd_allreduce: 43.75 | step: 19.22

 20%|█▉        | 8108/41250 [19:34:54<79:29:36,  8.63s/it]
                                                          
{'loss': 0.3419, 'grad_norm': 2.6685476303100586, 'learning_rate': 3.716027117860152e-05, 'epoch': 1.97}

 20%|█▉        | 8108/41250 [19:34:54<79:29:36,  8.63s/it][2025-04-26 03:32:38,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:32:38,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5704.04 | bwd_inner_microstep: 5650.41 | bwd_allreduce_microstep: 53.58 | step_microstep: 19.34
[2025-04-26 03:32:38,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5704.05 | bwd_inner: 5650.41 | bwd_allreduce: 53.60 | step: 19.34

 20%|█▉        | 8109/41250 [19:35:03<79:25:42,  8.63s/it]
                                                          
{'loss': 0.0525, 'grad_norm': 0.8145923614501953, 'learning_rate': 3.7159464564213576e-05, 'epoch': 1.97}

 20%|█▉        | 8109/41250 [19:35:03<79:25:42,  8.63s/it][2025-04-26 03:32:46,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:32:46,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5747.73 | bwd_inner_microstep: 5708.95 | bwd_allreduce_microstep: 38.73 | step_microstep: 19.27
[2025-04-26 03:32:46,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5747.74 | bwd_inner: 5708.95 | bwd_allreduce: 38.75 | step: 19.27

 20%|█▉        | 8110/41250 [19:35:12<79:34:27,  8.64s/it]
                                                          
{'loss': 0.1435, 'grad_norm': 4.756702899932861, 'learning_rate': 3.71586578440409e-05, 'epoch': 1.97}

 20%|█▉        | 8110/41250 [19:35:12<79:34:27,  8.64s/it][2025-04-26 03:32:55,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:32:55,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.07 | bwd_microstep: 5737.65 | bwd_inner_microstep: 5687.44 | bwd_allreduce_microstep: 50.16 | step_microstep: 18.89
[2025-04-26 03:32:55,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.07 | bwd: 5737.66 | bwd_inner: 5687.44 | bwd_allreduce: 50.18 | step: 18.89

 20%|█▉        | 8111/41250 [19:35:20<79:39:25,  8.65s/it]
                                                          
{'loss': 0.0367, 'grad_norm': 1.098236083984375, 'learning_rate': 3.7157851018088475e-05, 'epoch': 1.97}

 20%|█▉        | 8111/41250 [19:35:20<79:39:25,  8.65s/it][2025-04-26 03:33:04,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.08 | optimizer_step: 1.09
[2025-04-26 03:33:04,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.42 | bwd_microstep: 5720.68 | bwd_inner_microstep: 5662.76 | bwd_allreduce_microstep: 57.86 | step_microstep: 19.62
[2025-04-26 03:33:04,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.42 | bwd: 5720.70 | bwd_inner: 5662.76 | bwd_allreduce: 57.88 | step: 19.62

 20%|█▉        | 8112/41250 [19:35:29<79:35:58,  8.65s/it]
                                                          
{'loss': 0.1648, 'grad_norm': 3.1636953353881836, 'learning_rate': 3.715704408636127e-05, 'epoch': 1.97}

 20%|█▉        | 8112/41250 [19:35:29<79:35:58,  8.65s/it][2025-04-26 03:33:12,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:33:12,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.21 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5692.08 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.93
[2025-04-26 03:33:12,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.21 | bwd: 5704.90 | bwd_inner: 5692.08 | bwd_allreduce: 12.77 | step: 18.93

 20%|█▉        | 8113/41250 [19:35:38<79:34:14,  8.64s/it]
                                                          
{'loss': 0.1872, 'grad_norm': 3.73124623298645, 'learning_rate': 3.7156237048864256e-05, 'epoch': 1.97}

 20%|█▉        | 8113/41250 [19:35:38<79:34:14,  8.64s/it][2025-04-26 03:33:21,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:33:21,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.86 | bwd_microstep: 5793.24 | bwd_inner_microstep: 5653.41 | bwd_allreduce_microstep: 139.78 | step_microstep: 18.90
[2025-04-26 03:33:21,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.86 | bwd: 5793.25 | bwd_inner: 5653.41 | bwd_allreduce: 139.80 | step: 18.90

 20%|█▉        | 8114/41250 [19:35:46<79:43:38,  8.66s/it]
                                                          
{'loss': 0.1707, 'grad_norm': 1.8307074308395386, 'learning_rate': 3.715542990560243e-05, 'epoch': 1.97}

 20%|█▉        | 8114/41250 [19:35:46<79:43:38,  8.66s/it][2025-04-26 03:33:30,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 03:33:30,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.06 | bwd_microstep: 5803.85 | bwd_inner_microstep: 5650.27 | bwd_allreduce_microstep: 153.55 | step_microstep: 18.57
[2025-04-26 03:33:30,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.06 | bwd: 5803.87 | bwd_inner: 5650.27 | bwd_allreduce: 153.56 | step: 18.57

 20%|█▉        | 8115/41250 [19:35:55<79:52:04,  8.68s/it]
                                                          
{'loss': 0.1966, 'grad_norm': 3.965644598007202, 'learning_rate': 3.715462265658074e-05, 'epoch': 1.97}

 20%|█▉        | 8115/41250 [19:35:55<79:52:04,  8.68s/it][2025-04-26 03:33:38,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:33:38,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5763.62 | bwd_inner_microstep: 5707.83 | bwd_allreduce_microstep: 55.75 | step_microstep: 18.62
[2025-04-26 03:33:38,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5763.63 | bwd_inner: 5707.83 | bwd_allreduce: 55.77 | step: 18.62

 20%|█▉        | 8116/41250 [19:36:04<79:54:51,  8.68s/it]
                                                          
{'loss': 0.0708, 'grad_norm': 1.5480328798294067, 'learning_rate': 3.7153815301804186e-05, 'epoch': 1.97}

 20%|█▉        | 8116/41250 [19:36:04<79:54:51,  8.68s/it][2025-04-26 03:33:47,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.34 | optimizer_step: 1.05
[2025-04-26 03:33:47,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.90 | bwd_microstep: 5784.58 | bwd_inner_microstep: 5643.50 | bwd_allreduce_microstep: 141.01 | step_microstep: 20.25
[2025-04-26 03:33:47,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.90 | bwd: 5784.60 | bwd_inner: 5643.50 | bwd_allreduce: 141.04 | step: 20.25

 20%|█▉        | 8117/41250 [19:36:12<79:57:13,  8.69s/it]
                                                          
{'loss': 0.1823, 'grad_norm': 1.9906527996063232, 'learning_rate': 3.715300784127773e-05, 'epoch': 1.97}

 20%|█▉        | 8117/41250 [19:36:12<79:57:13,  8.69s/it][2025-04-26 03:33:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.20 | optimizer_step: 0.94
[2025-04-26 03:33:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.08 | bwd_microstep: 5758.76 | bwd_inner_microstep: 5691.79 | bwd_allreduce_microstep: 66.92 | step_microstep: 19.63
[2025-04-26 03:33:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.08 | bwd: 5758.78 | bwd_inner: 5691.79 | bwd_allreduce: 66.94 | step: 19.63

 20%|█▉        | 8118/41250 [19:36:21<79:58:04,  8.69s/it]
                                                          
{'loss': 0.0362, 'grad_norm': 0.5473706722259521, 'learning_rate': 3.7152200275006356e-05, 'epoch': 1.97}

 20%|█▉        | 8118/41250 [19:36:21<79:58:04,  8.69s/it][2025-04-26 03:34:04,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 03:34:04,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.70 | bwd_microstep: 5734.30 | bwd_inner_microstep: 5721.52 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.92
[2025-04-26 03:34:04,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.70 | bwd: 5734.32 | bwd_inner: 5721.52 | bwd_allreduce: 12.75 | step: 18.92

 20%|█▉        | 8119/41250 [19:36:30<79:55:05,  8.68s/it]
                                                          
{'loss': 0.0695, 'grad_norm': 2.1959173679351807, 'learning_rate': 3.715139260299504e-05, 'epoch': 1.97}

 20%|█▉        | 8119/41250 [19:36:30<79:55:05,  8.68s/it][2025-04-26 03:34:13,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.94 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 03:34:13,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5770.88 | bwd_inner_microstep: 5708.96 | bwd_allreduce_microstep: 61.87 | step_microstep: 18.46
[2025-04-26 03:34:13,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5770.89 | bwd_inner: 5708.96 | bwd_allreduce: 61.89 | step: 18.46

 20%|█▉        | 8120/41250 [19:36:38<79:58:31,  8.69s/it]
                                                          
{'loss': 0.1463, 'grad_norm': 1.6770788431167603, 'learning_rate': 3.715058482524876e-05, 'epoch': 1.97}

 20%|█▉        | 8120/41250 [19:36:38<79:58:31,  8.69s/it][2025-04-26 03:34:22,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 03:34:22,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.53 | bwd_microstep: 5703.46 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 12.57 | step_microstep: 19.07
[2025-04-26 03:34:22,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.53 | bwd: 5703.47 | bwd_inner: 5690.85 | bwd_allreduce: 12.59 | step: 19.07

 20%|█▉        | 8121/41250 [19:36:47<79:48:35,  8.67s/it]
                                                          
{'loss': 0.2027, 'grad_norm': 1.1402812004089355, 'learning_rate': 3.7149776941772494e-05, 'epoch': 1.97}

 20%|█▉        | 8121/41250 [19:36:47<79:48:35,  8.67s/it][2025-04-26 03:34:31,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 03:34:31,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.15 | bwd_microstep: 5878.03 | bwd_inner_microstep: 5692.97 | bwd_allreduce_microstep: 185.01 | step_microstep: 19.29
[2025-04-26 03:34:31,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.15 | bwd: 5878.04 | bwd_inner: 5692.97 | bwd_allreduce: 185.03 | step: 19.29

 20%|█▉        | 8122/41250 [19:36:56<80:11:09,  8.71s/it]
                                                          
{'loss': 0.1382, 'grad_norm': 1.8086050748825073, 'learning_rate': 3.714896895257123e-05, 'epoch': 1.97}

 20%|█▉        | 8122/41250 [19:36:56<80:11:09,  8.71s/it][2025-04-26 03:34:39,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 03:34:39,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.43 | bwd_microstep: 5748.96 | bwd_inner_microstep: 5693.11 | bwd_allreduce_microstep: 55.81 | step_microstep: 18.76
[2025-04-26 03:34:39,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.43 | bwd: 5748.98 | bwd_inner: 5693.11 | bwd_allreduce: 55.83 | step: 18.76

 20%|█▉        | 8123/41250 [19:37:05<80:06:55,  8.71s/it]
                                                          
{'loss': 0.0622, 'grad_norm': 0.9602349400520325, 'learning_rate': 3.714816085764995e-05, 'epoch': 1.97}

 20%|█▉        | 8123/41250 [19:37:05<80:06:55,  8.71s/it][2025-04-26 03:34:48,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:34:48,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.09 | bwd_microstep: 5755.88 | bwd_inner_microstep: 5706.58 | bwd_allreduce_microstep: 49.25 | step_microstep: 19.24
[2025-04-26 03:34:48,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.09 | bwd: 5755.90 | bwd_inner: 5706.58 | bwd_allreduce: 49.27 | step: 19.24

 20%|█▉        | 8124/41250 [19:37:13<80:05:43,  8.70s/it]
                                                          
{'loss': 0.1755, 'grad_norm': 3.504256248474121, 'learning_rate': 3.7147352657013626e-05, 'epoch': 1.97}

 20%|█▉        | 8124/41250 [19:37:13<80:05:43,  8.70s/it][2025-04-26 03:34:57,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:34:57,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.24 | bwd_microstep: 5721.75 | bwd_inner_microstep: 5708.88 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.54
[2025-04-26 03:34:57,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.24 | bwd: 5721.76 | bwd_inner: 5708.88 | bwd_allreduce: 12.85 | step: 18.55

 20%|█▉        | 8125/41250 [19:37:22<79:58:50,  8.69s/it]
                                                          
{'loss': 0.1418, 'grad_norm': 2.9559104442596436, 'learning_rate': 3.714654435066725e-05, 'epoch': 1.97}

 20%|█▉        | 8125/41250 [19:37:22<79:58:50,  8.69s/it][2025-04-26 03:35:05,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 03:35:05,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.54 | bwd_microstep: 5749.22 | bwd_inner_microstep: 5704.78 | bwd_allreduce_microstep: 44.39 | step_microstep: 19.09
[2025-04-26 03:35:05,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5749.24 | bwd_inner: 5704.78 | bwd_allreduce: 44.41 | step: 19.09

 20%|█▉        | 8126/41250 [19:37:31<79:58:08,  8.69s/it]
                                                          
{'loss': 0.1429, 'grad_norm': 3.054588794708252, 'learning_rate': 3.71457359386158e-05, 'epoch': 1.97}

 20%|█▉        | 8126/41250 [19:37:31<79:58:08,  8.69s/it][2025-04-26 03:35:14,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 03:35:14,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.57 | bwd_microstep: 5793.95 | bwd_inner_microstep: 5781.10 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.55
[2025-04-26 03:35:14,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.57 | bwd: 5793.97 | bwd_inner: 5781.10 | bwd_allreduce: 12.82 | step: 19.55

 20%|█▉        | 8127/41250 [19:37:39<80:11:25,  8.72s/it]
                                                          
{'loss': 0.3948, 'grad_norm': 3.2013323307037354, 'learning_rate': 3.7144927420864256e-05, 'epoch': 1.97}

 20%|█▉        | 8127/41250 [19:37:39<80:11:25,  8.72s/it][2025-04-26 03:35:23,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.26 | optimizer_step: 0.97
[2025-04-26 03:35:23,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.08 | bwd_microstep: 5898.11 | bwd_inner_microstep: 5658.65 | bwd_allreduce_microstep: 239.41 | step_microstep: 19.60
[2025-04-26 03:35:23,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.08 | bwd: 5898.12 | bwd_inner: 5658.65 | bwd_allreduce: 239.43 | step: 19.60

 20%|█▉        | 8128/41250 [19:37:48<80:28:08,  8.75s/it]
                                                          
{'loss': 0.0922, 'grad_norm': 1.2439628839492798, 'learning_rate': 3.7144118797417604e-05, 'epoch': 1.97}

 20%|█▉        | 8128/41250 [19:37:48<80:28:08,  8.75s/it][2025-04-26 03:35:32,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 03:35:32,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.34 | bwd_microstep: 5879.79 | bwd_inner_microstep: 5685.71 | bwd_allreduce_microstep: 194.03 | step_microstep: 19.75
[2025-04-26 03:35:32,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.34 | bwd: 5879.80 | bwd_inner: 5685.71 | bwd_allreduce: 194.05 | step: 19.75

 20%|█▉        | 8129/41250 [19:37:57<80:38:58,  8.77s/it]
                                                          
{'loss': 0.15, 'grad_norm': 1.6412625312805176, 'learning_rate': 3.7143310068280845e-05, 'epoch': 1.97}

 20%|█▉        | 8129/41250 [19:37:57<80:38:58,  8.77s/it][2025-04-26 03:35:40,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:35:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.87 | bwd_microstep: 5783.74 | bwd_inner_microstep: 5645.79 | bwd_allreduce_microstep: 137.91 | step_microstep: 19.23
[2025-04-26 03:35:40,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.87 | bwd: 5783.75 | bwd_inner: 5645.79 | bwd_allreduce: 137.93 | step: 19.23

 20%|█▉        | 8130/41250 [19:38:06<80:26:50,  8.74s/it]
                                                          
{'loss': 0.0316, 'grad_norm': 1.1450363397598267, 'learning_rate': 3.7142501233458945e-05, 'epoch': 1.97}

 20%|█▉        | 8130/41250 [19:38:06<80:26:50,  8.74s/it][2025-04-26 03:35:49,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 03:35:49,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.25 | bwd_microstep: 5721.25 | bwd_inner_microstep: 5657.13 | bwd_allreduce_microstep: 64.06 | step_microstep: 19.05
[2025-04-26 03:35:49,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.25 | bwd: 5721.26 | bwd_inner: 5657.13 | bwd_allreduce: 64.08 | step: 19.05

 20%|█▉        | 8131/41250 [19:38:14<80:08:25,  8.71s/it]
                                                          
{'loss': 0.1159, 'grad_norm': 1.3682291507720947, 'learning_rate': 3.71416922929569e-05, 'epoch': 1.97}

 20%|█▉        | 8131/41250 [19:38:14<80:08:25,  8.71s/it][2025-04-26 03:35:58,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-26 03:35:58,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.86 | bwd_microstep: 5713.73 | bwd_inner_microstep: 5700.86 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.30
[2025-04-26 03:35:58,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.86 | bwd: 5713.74 | bwd_inner: 5700.86 | bwd_allreduce: 12.84 | step: 19.30

 20%|█▉        | 8132/41250 [19:38:23<79:57:38,  8.69s/it]
                                                          
{'loss': 0.0556, 'grad_norm': 0.767991304397583, 'learning_rate': 3.7140883246779684e-05, 'epoch': 1.97}

 20%|█▉        | 8132/41250 [19:38:23<79:57:38,  8.69s/it][2025-04-26 03:36:06,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 03:36:06,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.85 | bwd_microstep: 5709.87 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.94
[2025-04-26 03:36:06,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.85 | bwd: 5709.89 | bwd_inner: 5696.95 | bwd_allreduce: 12.89 | step: 18.94

 20%|█▉        | 8133/41250 [19:38:32<79:52:10,  8.68s/it]
                                                          
{'loss': 0.3865, 'grad_norm': 4.146049976348877, 'learning_rate': 3.71400740949323e-05, 'epoch': 1.97}

 20%|█▉        | 8133/41250 [19:38:32<79:52:10,  8.68s/it][2025-04-26 03:36:15,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:36:15,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.61 | bwd_microstep: 5771.99 | bwd_inner_microstep: 5644.02 | bwd_allreduce_microstep: 127.92 | step_microstep: 18.74
[2025-04-26 03:36:15,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.61 | bwd: 5772.00 | bwd_inner: 5644.02 | bwd_allreduce: 127.94 | step: 18.75

 20%|█▉        | 8134/41250 [19:38:40<79:52:45,  8.68s/it]
                                                          
{'loss': 0.2145, 'grad_norm': 1.962787389755249, 'learning_rate': 3.7139264837419727e-05, 'epoch': 1.97}

 20%|█▉        | 8134/41250 [19:38:40<79:52:45,  8.68s/it][2025-04-26 03:36:24,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:36:24,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.58 | bwd_microstep: 5744.82 | bwd_inner_microstep: 5688.61 | bwd_allreduce_microstep: 56.16 | step_microstep: 19.32
[2025-04-26 03:36:24,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.58 | bwd: 5744.83 | bwd_inner: 5688.61 | bwd_allreduce: 56.18 | step: 19.32

 20%|█▉        | 8135/41250 [19:38:49<79:52:11,  8.68s/it]
                                                          
{'loss': 0.0449, 'grad_norm': 0.8213307857513428, 'learning_rate': 3.7138455474246965e-05, 'epoch': 1.97}

 20%|█▉        | 8135/41250 [19:38:49<79:52:11,  8.68s/it][2025-04-26 03:36:32,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.25 | optimizer_step: 1.12
[2025-04-26 03:36:32,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.55 | bwd_microstep: 5749.80 | bwd_inner_microstep: 5696.20 | bwd_allreduce_microstep: 53.53 | step_microstep: 20.01
[2025-04-26 03:36:32,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.55 | bwd: 5749.81 | bwd_inner: 5696.20 | bwd_allreduce: 53.56 | step: 20.01

 20%|█▉        | 8136/41250 [19:38:58<79:52:58,  8.68s/it]
                                                          
{'loss': 0.0824, 'grad_norm': 2.419579267501831, 'learning_rate': 3.7137646005418986e-05, 'epoch': 1.97}

 20%|█▉        | 8136/41250 [19:38:58<79:52:58,  8.68s/it][2025-04-26 03:36:41,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:36:41,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.02 | bwd_microstep: 5701.67 | bwd_inner_microstep: 5688.91 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.76
[2025-04-26 03:36:41,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.02 | bwd: 5701.69 | bwd_inner: 5688.91 | bwd_allreduce: 12.74 | step: 18.77

 20%|█▉        | 8137/41250 [19:39:06<79:43:58,  8.67s/it]
                                                          
{'loss': 0.0761, 'grad_norm': 2.1683919429779053, 'learning_rate': 3.713683643094079e-05, 'epoch': 1.97}

 20%|█▉        | 8137/41250 [19:39:06<79:43:58,  8.67s/it][2025-04-26 03:36:50,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:36:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5704.44 | bwd_inner_microstep: 5691.84 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.60
[2025-04-26 03:36:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5704.45 | bwd_inner: 5691.84 | bwd_allreduce: 12.57 | step: 18.60

 20%|█▉        | 8138/41250 [19:39:15<79:38:14,  8.66s/it]
                                                          
{'loss': 0.1103, 'grad_norm': 1.0689845085144043, 'learning_rate': 3.713602675081737e-05, 'epoch': 1.97}

 20%|█▉        | 8138/41250 [19:39:15<79:38:14,  8.66s/it][2025-04-26 03:36:58,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:36:58,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.01 | bwd_microstep: 5757.52 | bwd_inner_microstep: 5674.00 | bwd_allreduce_microstep: 83.47 | step_microstep: 18.48
[2025-04-26 03:36:58,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.01 | bwd: 5757.53 | bwd_inner: 5674.00 | bwd_allreduce: 83.49 | step: 18.48

 20%|█▉        | 8139/41250 [19:39:24<79:42:08,  8.67s/it]
                                                          
{'loss': 0.3278, 'grad_norm': 2.9847590923309326, 'learning_rate': 3.713521696505373e-05, 'epoch': 1.97}

 20%|█▉        | 8139/41250 [19:39:24<79:42:08,  8.67s/it][2025-04-26 03:37:07,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 03:37:07,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.09 | bwd_microstep: 5761.08 | bwd_inner_microstep: 5637.75 | bwd_allreduce_microstep: 123.29 | step_microstep: 18.66
[2025-04-26 03:37:07,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.09 | bwd: 5761.09 | bwd_inner: 5637.75 | bwd_allreduce: 123.30 | step: 18.66

 20%|█▉        | 8140/41250 [19:39:32<79:42:12,  8.67s/it]
                                                          
{'loss': 0.3283, 'grad_norm': 1.8545008897781372, 'learning_rate': 3.7134407073654835e-05, 'epoch': 1.97}

 20%|█▉        | 8140/41250 [19:39:32<79:42:12,  8.67s/it][2025-04-26 03:37:16,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-26 03:37:16,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.56 | bwd_microstep: 5731.84 | bwd_inner_microstep: 5699.14 | bwd_allreduce_microstep: 32.66 | step_microstep: 18.53
[2025-04-26 03:37:16,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.56 | bwd: 5731.86 | bwd_inner: 5699.13 | bwd_allreduce: 32.68 | step: 18.53

 20%|█▉        | 8141/41250 [19:39:41<79:41:11,  8.66s/it]
                                                          
{'loss': 0.0401, 'grad_norm': 0.9681961536407471, 'learning_rate': 3.713359707662568e-05, 'epoch': 1.97}

 20%|█▉        | 8141/41250 [19:39:41<79:41:11,  8.66s/it][2025-04-26 03:37:24,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:37:24,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.66 | bwd_microstep: 5743.35 | bwd_inner_microstep: 5642.56 | bwd_allreduce_microstep: 100.74 | step_microstep: 18.74
[2025-04-26 03:37:24,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.66 | bwd: 5743.36 | bwd_inner: 5642.56 | bwd_allreduce: 100.76 | step: 18.74

 20%|█▉        | 8142/41250 [19:39:50<79:39:36,  8.66s/it]
                                                          
{'loss': 0.1238, 'grad_norm': 1.681649088859558, 'learning_rate': 3.713278697397129e-05, 'epoch': 1.97}

 20%|█▉        | 8142/41250 [19:39:50<79:39:36,  8.66s/it][2025-04-26 03:37:33,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.07 | optimizer_step: 0.99
[2025-04-26 03:37:33,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.71 | bwd_microstep: 5711.75 | bwd_inner_microstep: 5647.97 | bwd_allreduce_microstep: 63.73 | step_microstep: 19.36
[2025-04-26 03:37:33,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.71 | bwd: 5711.76 | bwd_inner: 5647.97 | bwd_allreduce: 63.75 | step: 19.36

 20%|█▉        | 8143/41250 [19:39:58<79:31:59,  8.65s/it]
                                                          
{'loss': 0.0121, 'grad_norm': 0.26918742060661316, 'learning_rate': 3.7131976765696626e-05, 'epoch': 1.97}

 20%|█▉        | 8143/41250 [19:39:58<79:31:59,  8.65s/it][2025-04-26 03:37:42,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:37:42,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.62 | bwd_microstep: 5769.45 | bwd_inner_microstep: 5646.53 | bwd_allreduce_microstep: 122.87 | step_microstep: 18.75
[2025-04-26 03:37:42,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.62 | bwd: 5769.46 | bwd_inner: 5646.53 | bwd_allreduce: 122.89 | step: 18.75

 20%|█▉        | 8144/41250 [19:40:07<79:35:54,  8.66s/it]
                                                          
{'loss': 0.1161, 'grad_norm': 2.2523627281188965, 'learning_rate': 3.71311664518067e-05, 'epoch': 1.97}

 20%|█▉        | 8144/41250 [19:40:07<79:35:54,  8.66s/it][2025-04-26 03:37:50,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:37:50,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.45 | bwd_microstep: 5754.90 | bwd_inner_microstep: 5647.17 | bwd_allreduce_microstep: 107.69 | step_microstep: 18.41
[2025-04-26 03:37:50,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.45 | bwd: 5754.92 | bwd_inner: 5647.17 | bwd_allreduce: 107.71 | step: 18.42

 20%|█▉        | 8145/41250 [19:40:16<79:36:44,  8.66s/it]
                                                          
{'loss': 0.2411, 'grad_norm': 3.1757421493530273, 'learning_rate': 3.71303560323065e-05, 'epoch': 1.97}

 20%|█▉        | 8145/41250 [19:40:16<79:36:44,  8.66s/it][2025-04-26 03:37:59,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:37:59,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.78 | bwd_microstep: 5732.72 | bwd_inner_microstep: 5639.26 | bwd_allreduce_microstep: 93.42 | step_microstep: 18.39
[2025-04-26 03:37:59,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.78 | bwd: 5732.74 | bwd_inner: 5639.26 | bwd_allreduce: 93.44 | step: 18.40

 20%|█▉        | 8146/41250 [19:40:24<79:34:07,  8.65s/it]
                                                          
{'loss': 0.2858, 'grad_norm': 3.970125436782837, 'learning_rate': 3.712954550720102e-05, 'epoch': 1.97}

 20%|█▉        | 8146/41250 [19:40:24<79:34:07,  8.65s/it][2025-04-26 03:38:08,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:38:08,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5771.47 | bwd_inner_microstep: 5649.35 | bwd_allreduce_microstep: 122.07 | step_microstep: 18.08
[2025-04-26 03:38:08,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5771.49 | bwd_inner: 5649.35 | bwd_allreduce: 122.09 | step: 18.08

 20%|█▉        | 8147/41250 [19:40:33<79:37:46,  8.66s/it]
                                                          
{'loss': 0.2489, 'grad_norm': 2.28961181640625, 'learning_rate': 3.712873487649526e-05, 'epoch': 1.98}

 20%|█▉        | 8147/41250 [19:40:33<79:37:46,  8.66s/it][2025-04-26 03:38:16,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:38:16,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.23 | bwd_microstep: 5774.33 | bwd_inner_microstep: 5640.49 | bwd_allreduce_microstep: 133.80 | step_microstep: 18.13
[2025-04-26 03:38:16,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.23 | bwd: 5774.34 | bwd_inner: 5640.49 | bwd_allreduce: 133.82 | step: 18.14

 20%|█▉        | 8148/41250 [19:40:42<79:39:54,  8.66s/it]
                                                          
{'loss': 0.1393, 'grad_norm': 0.7707547545433044, 'learning_rate': 3.712792414019423e-05, 'epoch': 1.98}

 20%|█▉        | 8148/41250 [19:40:42<79:39:54,  8.66s/it][2025-04-26 03:38:25,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 03:38:25,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.75 | bwd_microstep: 5740.98 | bwd_inner_microstep: 5679.90 | bwd_allreduce_microstep: 61.04 | step_microstep: 18.21
[2025-04-26 03:38:25,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.75 | bwd: 5741.00 | bwd_inner: 5679.90 | bwd_allreduce: 61.06 | step: 18.21

 20%|█▉        | 8149/41250 [19:40:50<79:39:03,  8.66s/it]
                                                          
{'loss': 0.0273, 'grad_norm': 0.4961722493171692, 'learning_rate': 3.712711329830291e-05, 'epoch': 1.98}

 20%|█▉        | 8149/41250 [19:40:50<79:39:03,  8.66s/it][2025-04-26 03:38:34,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:38:34,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.67 | bwd_microstep: 5698.94 | bwd_inner_microstep: 5686.61 | bwd_allreduce_microstep: 12.29 | step_microstep: 18.32
[2025-04-26 03:38:34,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.67 | bwd: 5698.96 | bwd_inner: 5686.61 | bwd_allreduce: 12.31 | step: 18.33

 20%|█▉        | 8150/41250 [19:40:59<79:33:14,  8.65s/it]
                                                          
{'loss': 0.0843, 'grad_norm': 2.7875709533691406, 'learning_rate': 3.712630235082631e-05, 'epoch': 1.98}

 20%|█▉        | 8150/41250 [19:40:59<79:33:14,  8.65s/it][2025-04-26 03:38:42,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:38:42,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.21 | bwd_microstep: 5777.66 | bwd_inner_microstep: 5765.00 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.25
[2025-04-26 03:38:42,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.22 | bwd: 5777.67 | bwd_inner: 5765.00 | bwd_allreduce: 12.63 | step: 18.25

 20%|█▉        | 8151/41250 [19:41:08<79:46:20,  8.68s/it]
                                                          
{'loss': 0.0368, 'grad_norm': 0.7105482816696167, 'learning_rate': 3.7125491297769416e-05, 'epoch': 1.98}

 20%|█▉        | 8151/41250 [19:41:08<79:46:20,  8.68s/it][2025-04-26 03:38:51,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-26 03:38:51,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.90 | bwd_microstep: 5761.41 | bwd_inner_microstep: 5653.72 | bwd_allreduce_microstep: 107.64 | step_microstep: 19.36
[2025-04-26 03:38:51,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.90 | bwd: 5761.43 | bwd_inner: 5653.72 | bwd_allreduce: 107.66 | step: 19.37

 20%|█▉        | 8152/41250 [19:41:16<79:45:34,  8.68s/it]
                                                          
{'loss': 0.0638, 'grad_norm': 1.6021090745925903, 'learning_rate': 3.712468013913724e-05, 'epoch': 1.98}

 20%|█▉        | 8152/41250 [19:41:16<79:45:34,  8.68s/it][2025-04-26 03:39:00,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 03:39:00,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.33 | bwd_microstep: 5746.13 | bwd_inner_microstep: 5650.09 | bwd_allreduce_microstep: 96.00 | step_microstep: 18.55
[2025-04-26 03:39:00,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.33 | bwd: 5746.15 | bwd_inner: 5650.09 | bwd_allreduce: 96.01 | step: 18.55

 20%|█▉        | 8153/41250 [19:41:25<79:41:35,  8.67s/it]
                                                          
{'loss': 0.2531, 'grad_norm': 2.191370964050293, 'learning_rate': 3.7123868874934786e-05, 'epoch': 1.98}

 20%|█▉        | 8153/41250 [19:41:25<79:41:35,  8.67s/it][2025-04-26 03:39:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:39:08,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5703.23 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 12.31 | step_microstep: 18.95
[2025-04-26 03:39:08,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5703.25 | bwd_inner: 5690.89 | bwd_allreduce: 12.32 | step: 18.95

 20%|█▉        | 8154/41250 [19:41:34<79:35:34,  8.66s/it]
                                                          
{'loss': 0.1004, 'grad_norm': 1.6122548580169678, 'learning_rate': 3.712305750516704e-05, 'epoch': 1.98}

 20%|█▉        | 8154/41250 [19:41:34<79:35:34,  8.66s/it][2025-04-26 03:39:17,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:39:17,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.46 | bwd_microstep: 5788.52 | bwd_inner_microstep: 5775.81 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.81
[2025-04-26 03:39:17,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.46 | bwd: 5788.54 | bwd_inner: 5775.82 | bwd_allreduce: 12.68 | step: 18.81

 20%|█▉        | 8155/41250 [19:41:42<79:51:15,  8.69s/it]
                                                          
{'loss': 0.1768, 'grad_norm': 1.761583685874939, 'learning_rate': 3.712224602983901e-05, 'epoch': 1.98}

 20%|█▉        | 8155/41250 [19:41:42<79:51:15,  8.69s/it][2025-04-26 03:39:26,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:39:26,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5772.26 | bwd_inner_microstep: 5653.62 | bwd_allreduce_microstep: 118.59 | step_microstep: 18.65
[2025-04-26 03:39:26,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5772.27 | bwd_inner: 5653.62 | bwd_allreduce: 118.61 | step: 18.66

 20%|█▉        | 8156/41250 [19:41:51<79:51:13,  8.69s/it]
                                                          
{'loss': 0.1802, 'grad_norm': 2.288135290145874, 'learning_rate': 3.712143444895571e-05, 'epoch': 1.98}

 20%|█▉        | 8156/41250 [19:41:51<79:51:13,  8.69s/it][2025-04-26 03:39:34,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-26 03:39:34,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.48 | bwd_microstep: 5710.44 | bwd_inner_microstep: 5697.40 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.27
[2025-04-26 03:39:34,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.48 | bwd: 5710.46 | bwd_inner: 5697.40 | bwd_allreduce: 13.01 | step: 19.27

 20%|█▉        | 8157/41250 [19:42:00<79:44:44,  8.68s/it]
                                                          
{'loss': 0.1377, 'grad_norm': 1.7571979761123657, 'learning_rate': 3.712062276252213e-05, 'epoch': 1.98}

 20%|█▉        | 8157/41250 [19:42:00<79:44:44,  8.68s/it][2025-04-26 03:39:43,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:39:43,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.24 | bwd_microstep: 5703.10 | bwd_inner_microstep: 5690.42 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.07
[2025-04-26 03:39:43,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.25 | bwd: 5703.12 | bwd_inner: 5690.42 | bwd_allreduce: 12.66 | step: 18.08

 20%|█▉        | 8158/41250 [19:42:08<79:37:04,  8.66s/it]
                                                          
{'loss': 0.1, 'grad_norm': 1.8426765203475952, 'learning_rate': 3.7119810970543284e-05, 'epoch': 1.98}

 20%|█▉        | 8158/41250 [19:42:08<79:37:04,  8.66s/it][2025-04-26 03:39:52,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.81 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:39:52,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.86 | bwd_microstep: 5753.12 | bwd_inner_microstep: 5696.61 | bwd_allreduce_microstep: 56.46 | step_microstep: 17.83
[2025-04-26 03:39:52,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.86 | bwd: 5753.13 | bwd_inner: 5696.61 | bwd_allreduce: 56.48 | step: 17.83

 20%|█▉        | 8159/41250 [19:42:17<79:40:54,  8.67s/it]
                                                          
{'loss': 0.2222, 'grad_norm': 3.6188395023345947, 'learning_rate': 3.711899907302416e-05, 'epoch': 1.98}

 20%|█▉        | 8159/41250 [19:42:17<79:40:54,  8.67s/it][2025-04-26 03:40:00,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:40:00,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.81 | bwd_microstep: 5782.92 | bwd_inner_microstep: 5770.15 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.56
[2025-04-26 03:40:00,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.81 | bwd: 5782.93 | bwd_inner: 5770.15 | bwd_allreduce: 12.74 | step: 18.56

 20%|█▉        | 8160/41250 [19:42:26<79:54:43,  8.69s/it]
                                                          
{'loss': 0.1159, 'grad_norm': 3.3805832862854004, 'learning_rate': 3.711818706996978e-05, 'epoch': 1.98}

 20%|█▉        | 8160/41250 [19:42:26<79:54:43,  8.69s/it][2025-04-26 03:40:09,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:40:09,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.05 | bwd_microstep: 5714.84 | bwd_inner_microstep: 5682.39 | bwd_allreduce_microstep: 32.39 | step_microstep: 18.48
[2025-04-26 03:40:09,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.05 | bwd: 5714.85 | bwd_inner: 5682.39 | bwd_allreduce: 32.41 | step: 18.48

 20%|█▉        | 8161/41250 [19:42:34<79:45:54,  8.68s/it]
                                                          
{'loss': 0.0501, 'grad_norm': 0.8286653161048889, 'learning_rate': 3.7117374961385147e-05, 'epoch': 1.98}

 20%|█▉        | 8161/41250 [19:42:34<79:45:54,  8.68s/it][2025-04-26 03:40:18,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.95 | optimizer_step: 0.98
[2025-04-26 03:40:18,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5758.00 | bwd_inner_microstep: 5689.29 | bwd_allreduce_microstep: 68.67 | step_microstep: 18.23
[2025-04-26 03:40:18,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5758.02 | bwd_inner: 5689.29 | bwd_allreduce: 68.69 | step: 18.23

 20%|█▉        | 8162/41250 [19:42:43<79:47:54,  8.68s/it]
                                                          
{'loss': 0.3032, 'grad_norm': 1.6918202638626099, 'learning_rate': 3.7116562747275254e-05, 'epoch': 1.98}

 20%|█▉        | 8162/41250 [19:42:43<79:47:54,  8.68s/it][2025-04-26 03:40:26,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:40:26,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5741.75 | bwd_inner_microstep: 5641.76 | bwd_allreduce_microstep: 99.95 | step_microstep: 18.55
[2025-04-26 03:40:26,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.93 | bwd: 5741.76 | bwd_inner: 5641.76 | bwd_allreduce: 99.96 | step: 18.55

 20%|█▉        | 8163/41250 [19:42:52<79:42:01,  8.67s/it]
                                                          
{'loss': 0.2975, 'grad_norm': 4.528618335723877, 'learning_rate': 3.711575042764513e-05, 'epoch': 1.98}

 20%|█▉        | 8163/41250 [19:42:52<79:42:01,  8.67s/it][2025-04-26 03:40:35,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:40:35,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.04 | bwd_microstep: 5712.96 | bwd_inner_microstep: 5700.04 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.58
[2025-04-26 03:40:35,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.04 | bwd: 5712.98 | bwd_inner: 5700.04 | bwd_allreduce: 12.90 | step: 18.59

 20%|█▉        | 8164/41250 [19:43:00<79:36:51,  8.66s/it]
                                                          
{'loss': 0.0733, 'grad_norm': 1.3022431135177612, 'learning_rate': 3.711493800249976e-05, 'epoch': 1.98}

 20%|█▉        | 8164/41250 [19:43:00<79:36:51,  8.66s/it][2025-04-26 03:40:44,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:40:44,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.61 | bwd_microstep: 5694.81 | bwd_inner_microstep: 5656.73 | bwd_allreduce_microstep: 38.04 | step_microstep: 18.23
[2025-04-26 03:40:44,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.61 | bwd: 5694.82 | bwd_inner: 5656.72 | bwd_allreduce: 38.05 | step: 18.24

 20%|█▉        | 8165/41250 [19:43:09<79:27:41,  8.65s/it]
                                                          
{'loss': 0.1452, 'grad_norm': 1.9144206047058105, 'learning_rate': 3.7114125471844175e-05, 'epoch': 1.98}

 20%|█▉        | 8165/41250 [19:43:09<79:27:41,  8.65s/it][2025-04-26 03:40:52,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.29 | optimizer_step: 1.07
[2025-04-26 03:40:52,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.81 | bwd_microstep: 5724.18 | bwd_inner_microstep: 5710.08 | bwd_allreduce_microstep: 14.04 | step_microstep: 20.33
[2025-04-26 03:40:52,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.81 | bwd: 5724.20 | bwd_inner: 5710.08 | bwd_allreduce: 14.07 | step: 20.33

 20%|█▉        | 8166/41250 [19:43:18<79:30:52,  8.65s/it]
                                                          
{'loss': 0.0548, 'grad_norm': 1.3270039558410645, 'learning_rate': 3.711331283568336e-05, 'epoch': 1.98}

 20%|█▉        | 8166/41250 [19:43:18<79:30:52,  8.65s/it][2025-04-26 03:41:01,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:41:01,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.96 | bwd_microstep: 5758.63 | bwd_inner_microstep: 5686.70 | bwd_allreduce_microstep: 71.88 | step_microstep: 18.44
[2025-04-26 03:41:01,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.96 | bwd: 5758.64 | bwd_inner: 5686.70 | bwd_allreduce: 71.90 | step: 18.44

 20%|█▉        | 8167/41250 [19:43:26<79:37:20,  8.66s/it]
                                                          
{'loss': 0.0988, 'grad_norm': 1.2102493047714233, 'learning_rate': 3.7112500094022345e-05, 'epoch': 1.98}

 20%|█▉        | 8167/41250 [19:43:26<79:37:20,  8.66s/it][2025-04-26 03:41:10,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 03:41:10,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.06 | bwd_microstep: 5723.43 | bwd_inner_microstep: 5651.52 | bwd_allreduce_microstep: 71.87 | step_microstep: 19.48
[2025-04-26 03:41:10,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.06 | bwd: 5723.45 | bwd_inner: 5651.52 | bwd_allreduce: 71.89 | step: 19.50

 20%|█▉        | 8168/41250 [19:43:35<79:32:45,  8.66s/it]
                                                          
{'loss': 0.21, 'grad_norm': 3.1497058868408203, 'learning_rate': 3.711168724686613e-05, 'epoch': 1.98}

 20%|█▉        | 8168/41250 [19:43:35<79:32:45,  8.66s/it][2025-04-26 03:41:18,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:41:18,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.91 | bwd_microstep: 5771.81 | bwd_inner_microstep: 5657.92 | bwd_allreduce_microstep: 113.85 | step_microstep: 18.87
[2025-04-26 03:41:18,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.91 | bwd: 5771.83 | bwd_inner: 5657.92 | bwd_allreduce: 113.87 | step: 18.88

 20%|█▉        | 8169/41250 [19:43:44<79:37:44,  8.67s/it]
                                                          
{'loss': 0.0953, 'grad_norm': 1.1460002660751343, 'learning_rate': 3.711087429421973e-05, 'epoch': 1.98}

 20%|█▉        | 8169/41250 [19:43:44<79:37:44,  8.67s/it][2025-04-26 03:41:27,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.29 | optimizer_step: 1.07
[2025-04-26 03:41:27,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.55 | bwd_microstep: 5701.05 | bwd_inner_microstep: 5681.96 | bwd_allreduce_microstep: 19.02 | step_microstep: 20.43
[2025-04-26 03:41:27,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.55 | bwd: 5701.06 | bwd_inner: 5681.96 | bwd_allreduce: 19.06 | step: 20.43

 20%|█▉        | 8170/41250 [19:43:52<79:31:59,  8.66s/it]
                                                          
{'loss': 0.1092, 'grad_norm': 1.6053060293197632, 'learning_rate': 3.7110061236088156e-05, 'epoch': 1.98}

 20%|█▉        | 8170/41250 [19:43:52<79:31:59,  8.66s/it][2025-04-26 03:41:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-26 03:41:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.96 | bwd_microstep: 5800.72 | bwd_inner_microstep: 5658.42 | bwd_allreduce_microstep: 142.26 | step_microstep: 19.21
[2025-04-26 03:41:36,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.97 | bwd: 5800.74 | bwd_inner: 5658.42 | bwd_allreduce: 142.28 | step: 19.21

 20%|█▉        | 8171/41250 [19:44:01<79:41:16,  8.67s/it]
                                                          
{'loss': 0.0439, 'grad_norm': 0.674374520778656, 'learning_rate': 3.710924807247642e-05, 'epoch': 1.98}

 20%|█▉        | 8171/41250 [19:44:01<79:41:16,  8.67s/it][2025-04-26 03:41:44,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:41:44,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.70 | bwd_microstep: 5793.76 | bwd_inner_microstep: 5781.01 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.79
[2025-04-26 03:41:44,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.70 | bwd: 5793.78 | bwd_inner: 5781.01 | bwd_allreduce: 12.73 | step: 18.79

 20%|█▉        | 8172/41250 [19:44:10<79:56:43,  8.70s/it]
                                                          
{'loss': 0.1745, 'grad_norm': 0.8979306221008301, 'learning_rate': 3.710843480338953e-05, 'epoch': 1.98}

 20%|█▉        | 8172/41250 [19:44:10<79:56:43,  8.70s/it][2025-04-26 03:41:53,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-26 03:41:53,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5748.24 | bwd_inner_microstep: 5693.93 | bwd_allreduce_microstep: 54.26 | step_microstep: 18.56
[2025-04-26 03:41:53,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5748.25 | bwd_inner: 5693.93 | bwd_allreduce: 54.28 | step: 18.57

 20%|█▉        | 8173/41250 [19:44:18<79:52:49,  8.69s/it]
                                                          
{'loss': 0.1637, 'grad_norm': 5.220314025878906, 'learning_rate': 3.7107621428832514e-05, 'epoch': 1.98}

 20%|█▉        | 8173/41250 [19:44:18<79:52:49,  8.69s/it][2025-04-26 03:42:02,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-26 03:42:02,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.89 | bwd_microstep: 5714.19 | bwd_inner_microstep: 5701.28 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.90
[2025-04-26 03:42:02,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.89 | bwd: 5714.21 | bwd_inner: 5701.28 | bwd_allreduce: 12.88 | step: 18.90

 20%|█▉        | 8174/41250 [19:44:27<79:46:10,  8.68s/it]
                                                          
{'loss': 0.0249, 'grad_norm': 0.44565147161483765, 'learning_rate': 3.710680794881037e-05, 'epoch': 1.98}

 20%|█▉        | 8174/41250 [19:44:27<79:46:10,  8.68s/it][2025-04-26 03:42:10,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:42:10,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.08 | bwd_microstep: 5700.60 | bwd_inner_microstep: 5687.73 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.58
[2025-04-26 03:42:10,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.08 | bwd: 5700.61 | bwd_inner: 5687.73 | bwd_allreduce: 12.84 | step: 18.58

 20%|█▉        | 8175/41250 [19:44:36<79:37:12,  8.67s/it]
                                                          
{'loss': 0.1292, 'grad_norm': 2.2255046367645264, 'learning_rate': 3.710599436332812e-05, 'epoch': 1.98}

 20%|█▉        | 8175/41250 [19:44:36<79:37:12,  8.67s/it][2025-04-26 03:42:19,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:42:19,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.50 | bwd_microstep: 5706.08 | bwd_inner_microstep: 5636.95 | bwd_allreduce_microstep: 69.08 | step_microstep: 18.50
[2025-04-26 03:42:19,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.50 | bwd: 5706.09 | bwd_inner: 5636.95 | bwd_allreduce: 69.09 | step: 18.51

 20%|█▉        | 8176/41250 [19:44:44<79:27:06,  8.65s/it]
                                                          
{'loss': 0.1926, 'grad_norm': 1.2540379762649536, 'learning_rate': 3.710518067239078e-05, 'epoch': 1.98}

 20%|█▉        | 8176/41250 [19:44:44<79:27:06,  8.65s/it][2025-04-26 03:42:28,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.31 | optimizer_step: 1.04
[2025-04-26 03:42:28,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.86 | bwd_microstep: 5770.84 | bwd_inner_microstep: 5654.95 | bwd_allreduce_microstep: 115.83 | step_microstep: 19.95
[2025-04-26 03:42:28,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.86 | bwd: 5770.86 | bwd_inner: 5654.95 | bwd_allreduce: 115.86 | step: 19.95

 20%|█▉        | 8177/41250 [19:44:53<79:32:29,  8.66s/it]
                                                          
{'loss': 0.1077, 'grad_norm': 2.837503671646118, 'learning_rate': 3.710436687600337e-05, 'epoch': 1.98}

 20%|█▉        | 8177/41250 [19:44:53<79:32:29,  8.66s/it][2025-04-26 03:42:36,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.94
[2025-04-26 03:42:36,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.20 | bwd_microstep: 5762.43 | bwd_inner_microstep: 5749.18 | bwd_allreduce_microstep: 13.20 | step_microstep: 19.45
[2025-04-26 03:42:36,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.21 | bwd: 5762.45 | bwd_inner: 5749.18 | bwd_allreduce: 13.22 | step: 19.46

 20%|█▉        | 8178/41250 [19:45:02<79:42:56,  8.68s/it]
                                                          
{'loss': 0.149, 'grad_norm': 1.0460646152496338, 'learning_rate': 3.71035529741709e-05, 'epoch': 1.98}

 20%|█▉        | 8178/41250 [19:45:02<79:42:56,  8.68s/it][2025-04-26 03:42:45,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 03:42:45,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.44 | bwd_microstep: 5704.43 | bwd_inner_microstep: 5691.50 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.14
[2025-04-26 03:42:45,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.44 | bwd: 5704.45 | bwd_inner: 5691.50 | bwd_allreduce: 12.91 | step: 19.15

 20%|█▉        | 8179/41250 [19:45:10<79:36:30,  8.67s/it]
                                                          
{'loss': 0.0808, 'grad_norm': 0.9200416803359985, 'learning_rate': 3.710273896689839e-05, 'epoch': 1.98}

 20%|█▉        | 8179/41250 [19:45:10<79:36:30,  8.67s/it][2025-04-26 03:42:54,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 03:42:54,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5675.68 | bwd_inner_microstep: 5647.55 | bwd_allreduce_microstep: 28.08 | step_microstep: 18.83
[2025-04-26 03:42:54,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5675.70 | bwd_inner: 5647.55 | bwd_allreduce: 28.10 | step: 18.83

 20%|█▉        | 8180/41250 [19:45:19<79:22:55,  8.64s/it]
                                                          
{'loss': 0.0733, 'grad_norm': 2.050414562225342, 'learning_rate': 3.710192485419086e-05, 'epoch': 1.98}

 20%|█▉        | 8180/41250 [19:45:19<79:22:55,  8.64s/it][2025-04-26 03:43:02,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 03:43:02,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.34 | bwd_microstep: 5686.71 | bwd_inner_microstep: 5673.66 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.85
[2025-04-26 03:43:02,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.34 | bwd: 5686.72 | bwd_inner: 5673.66 | bwd_allreduce: 13.02 | step: 18.85

 20%|█▉        | 8181/41250 [19:45:28<79:19:19,  8.64s/it]
                                                          
{'loss': 0.0469, 'grad_norm': 0.834411084651947, 'learning_rate': 3.710111063605332e-05, 'epoch': 1.98}

 20%|█▉        | 8181/41250 [19:45:28<79:19:19,  8.64s/it][2025-04-26 03:43:11,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.29 | optimizer_step: 1.05
[2025-04-26 03:43:11,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.01 | bwd_microstep: 6009.64 | bwd_inner_microstep: 5691.31 | bwd_allreduce_microstep: 318.27 | step_microstep: 20.20
[2025-04-26 03:43:11,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.01 | bwd: 6009.66 | bwd_inner: 5691.31 | bwd_allreduce: 318.30 | step: 20.20

 20%|█▉        | 8182/41250 [19:45:37<80:10:55,  8.73s/it]
                                                          
{'loss': 0.0668, 'grad_norm': 1.012890100479126, 'learning_rate': 3.71002963124908e-05, 'epoch': 1.98}

 20%|█▉        | 8182/41250 [19:45:37<80:10:55,  8.73s/it][2025-04-26 03:43:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:43:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.22 | bwd_microstep: 5733.95 | bwd_inner_microstep: 5674.36 | bwd_allreduce_microstep: 59.55 | step_microstep: 18.78
[2025-04-26 03:43:20,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.22 | bwd: 5733.97 | bwd_inner: 5674.36 | bwd_allreduce: 59.57 | step: 18.78

 20%|█▉        | 8183/41250 [19:45:45<79:58:25,  8.71s/it]
                                                          
{'loss': 0.1057, 'grad_norm': 2.2232208251953125, 'learning_rate': 3.709948188350832e-05, 'epoch': 1.98}

 20%|█▉        | 8183/41250 [19:45:45<79:58:25,  8.71s/it][2025-04-26 03:43:28,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-26 03:43:28,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.94 | bwd_microstep: 5710.31 | bwd_inner_microstep: 5633.63 | bwd_allreduce_microstep: 76.62 | step_microstep: 19.49
[2025-04-26 03:43:28,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.94 | bwd: 5710.32 | bwd_inner: 5633.63 | bwd_allreduce: 76.65 | step: 19.49

 20%|█▉        | 8184/41250 [19:45:54<79:42:52,  8.68s/it]
                                                          
{'loss': 0.0546, 'grad_norm': 1.293359637260437, 'learning_rate': 3.709866734911089e-05, 'epoch': 1.98}

 20%|█▉        | 8184/41250 [19:45:54<79:42:52,  8.68s/it][2025-04-26 03:43:37,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 03:43:37,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.30 | bwd_microstep: 5694.65 | bwd_inner_microstep: 5681.75 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.94
[2025-04-26 03:43:37,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.30 | bwd: 5694.66 | bwd_inner: 5681.75 | bwd_allreduce: 12.87 | step: 18.94

 20%|█▉        | 8185/41250 [19:46:02<79:33:06,  8.66s/it]
                                                          
{'loss': 0.1659, 'grad_norm': 3.572227716445923, 'learning_rate': 3.7097852709303545e-05, 'epoch': 1.98}

 20%|█▉        | 8185/41250 [19:46:02<79:33:06,  8.66s/it][2025-04-26 03:43:46,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 03:43:46,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.22 | bwd_microstep: 5741.90 | bwd_inner_microstep: 5679.32 | bwd_allreduce_microstep: 62.54 | step_microstep: 19.04
[2025-04-26 03:43:46,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.22 | bwd: 5741.92 | bwd_inner: 5679.32 | bwd_allreduce: 62.56 | step: 19.05

 20%|█▉        | 8186/41250 [19:46:11<79:33:32,  8.66s/it]
                                                          
{'loss': 0.3289, 'grad_norm': 6.123682498931885, 'learning_rate': 3.70970379640913e-05, 'epoch': 1.98}

 20%|█▉        | 8186/41250 [19:46:11<79:33:32,  8.66s/it][2025-04-26 03:43:54,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.02 | optimizer_step: 1.14
[2025-04-26 03:43:54,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.84 | bwd_microstep: 5721.18 | bwd_inner_microstep: 5703.06 | bwd_allreduce_microstep: 18.08 | step_microstep: 19.56
[2025-04-26 03:43:54,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.84 | bwd: 5721.19 | bwd_inner: 5703.06 | bwd_allreduce: 18.09 | step: 19.56

 20%|█▉        | 8187/41250 [19:46:20<79:32:44,  8.66s/it]
                                                          
{'loss': 0.4208, 'grad_norm': 4.19537353515625, 'learning_rate': 3.709622311347918e-05, 'epoch': 1.98}

 20%|█▉        | 8187/41250 [19:46:20<79:32:44,  8.66s/it][2025-04-26 03:44:03,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 03:44:03,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.08 | bwd_microstep: 5687.74 | bwd_inner_microstep: 5646.63 | bwd_allreduce_microstep: 41.07 | step_microstep: 18.90
[2025-04-26 03:44:03,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.08 | bwd: 5687.76 | bwd_inner: 5646.62 | bwd_allreduce: 41.09 | step: 18.91

 20%|█▉        | 8188/41250 [19:46:28<79:21:04,  8.64s/it]
                                                          
{'loss': 0.0856, 'grad_norm': 2.5497496128082275, 'learning_rate': 3.7095408157472206e-05, 'epoch': 1.98}

 20%|█▉        | 8188/41250 [19:46:28<79:21:04,  8.64s/it][2025-04-26 03:44:12,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:44:12,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.93 | bwd_microstep: 5791.77 | bwd_inner_microstep: 5640.74 | bwd_allreduce_microstep: 150.99 | step_microstep: 18.65
[2025-04-26 03:44:12,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.93 | bwd: 5791.78 | bwd_inner: 5640.74 | bwd_allreduce: 151.00 | step: 18.65

 20%|█▉        | 8189/41250 [19:46:37<79:30:52,  8.66s/it]
                                                          
{'loss': 0.0307, 'grad_norm': 0.8096205592155457, 'learning_rate': 3.70945930960754e-05, 'epoch': 1.99}

 20%|█▉        | 8189/41250 [19:46:37<79:30:52,  8.66s/it][2025-04-26 03:44:20,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 03:44:20,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.56 | bwd_microstep: 5747.77 | bwd_inner_microstep: 5636.68 | bwd_allreduce_microstep: 111.04 | step_microstep: 18.95
[2025-04-26 03:44:20,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.56 | bwd: 5747.78 | bwd_inner: 5636.68 | bwd_allreduce: 111.06 | step: 18.96

 20%|█▉        | 8190/41250 [19:46:46<79:29:53,  8.66s/it]
                                                          
{'loss': 0.2478, 'grad_norm': 4.698051452636719, 'learning_rate': 3.709377792929379e-05, 'epoch': 1.99}

 20%|█▉        | 8190/41250 [19:46:46<79:29:53,  8.66s/it][2025-04-26 03:44:29,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 03:44:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.63 | bwd_microstep: 5726.24 | bwd_inner_microstep: 5687.02 | bwd_allreduce_microstep: 39.18 | step_microstep: 18.89
[2025-04-26 03:44:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.63 | bwd: 5726.25 | bwd_inner: 5687.02 | bwd_allreduce: 39.19 | step: 18.89

 20%|█▉        | 8191/41250 [19:46:54<79:30:38,  8.66s/it]
                                                          
{'loss': 0.149, 'grad_norm': 4.353422164916992, 'learning_rate': 3.709296265713241e-05, 'epoch': 1.99}

 20%|█▉        | 8191/41250 [19:46:54<79:30:38,  8.66s/it][2025-04-26 03:44:38,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:44:38,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.37 | bwd_microstep: 5773.24 | bwd_inner_microstep: 5636.08 | bwd_allreduce_microstep: 137.11 | step_microstep: 18.53
[2025-04-26 03:44:38,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.37 | bwd: 5773.25 | bwd_inner: 5636.08 | bwd_allreduce: 137.13 | step: 18.53

 20%|█▉        | 8192/41250 [19:47:03<79:33:46,  8.66s/it]
                                                          
{'loss': 0.026, 'grad_norm': 0.6496707201004028, 'learning_rate': 3.7092147279596264e-05, 'epoch': 1.99}

 20%|█▉        | 8192/41250 [19:47:03<79:33:46,  8.66s/it][2025-04-26 03:44:46,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:44:46,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.08 | bwd_microstep: 5771.63 | bwd_inner_microstep: 5634.58 | bwd_allreduce_microstep: 137.02 | step_microstep: 18.40
[2025-04-26 03:44:46,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.08 | bwd: 5771.65 | bwd_inner: 5634.58 | bwd_allreduce: 137.03 | step: 18.41

 20%|█▉        | 8193/41250 [19:47:12<79:34:32,  8.67s/it]
                                                          
{'loss': 0.2016, 'grad_norm': 1.6747370958328247, 'learning_rate': 3.7091331796690405e-05, 'epoch': 1.99}

 20%|█▉        | 8193/41250 [19:47:12<79:34:32,  8.67s/it][2025-04-26 03:44:55,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:44:55,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.99 | bwd_microstep: 5706.85 | bwd_inner_microstep: 5693.98 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.69
[2025-04-26 03:44:55,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.99 | bwd: 5706.86 | bwd_inner: 5693.98 | bwd_allreduce: 12.84 | step: 18.69

 20%|█▉        | 8194/41250 [19:47:20<79:31:38,  8.66s/it]
                                                          
{'loss': 0.0784, 'grad_norm': 2.6625406742095947, 'learning_rate': 3.709051620841984e-05, 'epoch': 1.99}

 20%|█▉        | 8194/41250 [19:47:20<79:31:38,  8.66s/it][2025-04-26 03:45:04,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:45:04,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.98 | bwd_microstep: 5757.59 | bwd_inner_microstep: 5689.14 | bwd_allreduce_microstep: 68.41 | step_microstep: 17.92
[2025-04-26 03:45:04,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.97 | bwd: 5757.60 | bwd_inner: 5689.14 | bwd_allreduce: 68.43 | step: 17.93

 20%|█▉        | 8195/41250 [19:47:29<79:35:40,  8.67s/it]
                                                          
{'loss': 0.3874, 'grad_norm': 4.749426364898682, 'learning_rate': 3.7089700514789605e-05, 'epoch': 1.99}

 20%|█▉        | 8195/41250 [19:47:29<79:35:40,  8.67s/it][2025-04-26 03:45:12,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-26 03:45:12,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5691.57 | bwd_inner_microstep: 5638.86 | bwd_allreduce_microstep: 52.65 | step_microstep: 19.07
[2025-04-26 03:45:12,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5691.58 | bwd_inner: 5638.86 | bwd_allreduce: 52.68 | step: 19.07

 20%|█▉        | 8196/41250 [19:47:38<79:26:49,  8.65s/it]
                                                          
{'loss': 0.0917, 'grad_norm': 1.3672428131103516, 'learning_rate': 3.708888471580473e-05, 'epoch': 1.99}

 20%|█▉        | 8196/41250 [19:47:38<79:26:49,  8.65s/it][2025-04-26 03:45:21,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.21 | optimizer_step: 0.89
[2025-04-26 03:45:21,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.65 | bwd_microstep: 5734.70 | bwd_inner_microstep: 5685.89 | bwd_allreduce_microstep: 48.76 | step_microstep: 19.17
[2025-04-26 03:45:21,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.65 | bwd: 5734.72 | bwd_inner: 5685.89 | bwd_allreduce: 48.78 | step: 19.17

 20%|█▉        | 8197/41250 [19:47:46<79:30:20,  8.66s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 1.6747575998306274, 'learning_rate': 3.7088068811470236e-05, 'epoch': 1.99}

 20%|█▉        | 8197/41250 [19:47:46<79:30:20,  8.66s/it][2025-04-26 03:45:30,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 03:45:30,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.18 | bwd_microstep: 5708.00 | bwd_inner_microstep: 5695.00 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.83
[2025-04-26 03:45:30,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.18 | bwd: 5708.02 | bwd_inner: 5695.00 | bwd_allreduce: 12.98 | step: 18.84

 20%|█▉        | 8198/41250 [19:47:55<79:26:49,  8.65s/it]
                                                          
{'loss': 0.2075, 'grad_norm': 2.5873777866363525, 'learning_rate': 3.7087252801791164e-05, 'epoch': 1.99}

 20%|█▉        | 8198/41250 [19:47:55<79:26:49,  8.65s/it][2025-04-26 03:45:38,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 03:45:38,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.84 | bwd_microstep: 5755.12 | bwd_inner_microstep: 5711.05 | bwd_allreduce_microstep: 44.02 | step_microstep: 18.50
[2025-04-26 03:45:38,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.84 | bwd: 5755.13 | bwd_inner: 5711.05 | bwd_allreduce: 44.04 | step: 18.51

 20%|█▉        | 8199/41250 [19:48:04<79:32:39,  8.66s/it]
                                                          
{'loss': 0.071, 'grad_norm': 1.9904769659042358, 'learning_rate': 3.7086436686772535e-05, 'epoch': 1.99}

 20%|█▉        | 8199/41250 [19:48:04<79:32:39,  8.66s/it][2025-04-26 03:45:47,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 03:45:47,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.97 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5645.69 | bwd_allreduce_microstep: 121.30 | step_microstep: 18.71
[2025-04-26 03:45:47,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.97 | bwd: 5767.05 | bwd_inner: 5645.69 | bwd_allreduce: 121.32 | step: 18.72

 20%|█▉        | 8200/41250 [19:48:12<79:35:41,  8.67s/it]
                                                          
{'loss': 0.0571, 'grad_norm': 0.7852814793586731, 'learning_rate': 3.7085620466419395e-05, 'epoch': 1.99}

 20%|█▉        | 8200/41250 [19:48:12<79:35:41,  8.67s/it][2025-04-26 03:45:56,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 1.12 | optimizer_step: 1.05
[2025-04-26 03:45:56,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.42 | bwd_microstep: 5708.99 | bwd_inner_microstep: 5695.18 | bwd_allreduce_microstep: 13.75 | step_microstep: 19.34
[2025-04-26 03:45:56,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.42 | bwd: 5709.01 | bwd_inner: 5695.18 | bwd_allreduce: 13.78 | step: 19.34

 20%|█▉        | 8201/41250 [19:48:21<79:32:07,  8.66s/it]
                                                          
{'loss': 0.037, 'grad_norm': 1.3378111124038696, 'learning_rate': 3.708480414073675e-05, 'epoch': 1.99}

 20%|█▉        | 8201/41250 [19:48:21<79:32:07,  8.66s/it][2025-04-26 03:46:04,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:46:04,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.31 | bwd_microstep: 5803.42 | bwd_inner_microstep: 5653.31 | bwd_allreduce_microstep: 150.07 | step_microstep: 18.20
[2025-04-26 03:46:04,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.31 | bwd: 5803.43 | bwd_inner: 5653.30 | bwd_allreduce: 150.08 | step: 18.21

 20%|█▉        | 8202/41250 [19:48:30<79:40:26,  8.68s/it]
                                                          
{'loss': 0.0718, 'grad_norm': 1.4730015993118286, 'learning_rate': 3.7083987709729656e-05, 'epoch': 1.99}

 20%|█▉        | 8202/41250 [19:48:30<79:40:26,  8.68s/it][2025-04-26 03:46:13,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:46:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5708.44 | bwd_inner_microstep: 5695.48 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.69
[2025-04-26 03:46:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5708.46 | bwd_inner: 5695.48 | bwd_allreduce: 12.93 | step: 18.69

 20%|█▉        | 8203/41250 [19:48:38<79:34:05,  8.67s/it]
                                                          
{'loss': 0.0954, 'grad_norm': 6.324215888977051, 'learning_rate': 3.708317117340314e-05, 'epoch': 1.99}

 20%|█▉        | 8203/41250 [19:48:38<79:34:05,  8.67s/it][2025-04-26 03:46:22,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:46:22,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.05 | bwd_microstep: 5739.04 | bwd_inner_microstep: 5709.12 | bwd_allreduce_microstep: 29.87 | step_microstep: 18.54
[2025-04-26 03:46:22,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.05 | bwd: 5739.05 | bwd_inner: 5709.12 | bwd_allreduce: 29.89 | step: 18.55

 20%|█▉        | 8204/41250 [19:48:47<79:35:54,  8.67s/it]
                                                          
{'loss': 0.111, 'grad_norm': 1.5860542058944702, 'learning_rate': 3.708235453176223e-05, 'epoch': 1.99}

 20%|█▉        | 8204/41250 [19:48:47<79:35:54,  8.67s/it][2025-04-26 03:46:30,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:46:30,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.67 | bwd_microstep: 5706.42 | bwd_inner_microstep: 5644.70 | bwd_allreduce_microstep: 61.68 | step_microstep: 18.29
[2025-04-26 03:46:30,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.67 | bwd: 5706.44 | bwd_inner: 5644.70 | bwd_allreduce: 61.70 | step: 18.29

 20%|█▉        | 8205/41250 [19:48:56<79:27:14,  8.66s/it]
                                                          
{'loss': 0.0803, 'grad_norm': 0.9463546276092529, 'learning_rate': 3.708153778481196e-05, 'epoch': 1.99}

 20%|█▉        | 8205/41250 [19:48:56<79:27:14,  8.66s/it][2025-04-26 03:46:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:46:39,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.53 | bwd_microstep: 5718.96 | bwd_inner_microstep: 5656.70 | bwd_allreduce_microstep: 62.22 | step_microstep: 17.99
[2025-04-26 03:46:39,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.53 | bwd: 5718.98 | bwd_inner: 5656.70 | bwd_allreduce: 62.23 | step: 18.00

 20%|█▉        | 8206/41250 [19:49:04<79:23:47,  8.65s/it]
                                                          
{'loss': 0.4285, 'grad_norm': 3.3844218254089355, 'learning_rate': 3.708072093255738e-05, 'epoch': 1.99}

 20%|█▉        | 8206/41250 [19:49:04<79:23:47,  8.65s/it][2025-04-26 03:46:48,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:46:48,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5724.66 | bwd_inner_microstep: 5645.98 | bwd_allreduce_microstep: 78.64 | step_microstep: 18.47
[2025-04-26 03:46:48,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5724.68 | bwd_inner: 5645.98 | bwd_allreduce: 78.66 | step: 18.47

 20%|█▉        | 8207/41250 [19:49:13<79:21:05,  8.65s/it]
                                                          
{'loss': 0.1854, 'grad_norm': 6.885409832000732, 'learning_rate': 3.707990397500351e-05, 'epoch': 1.99}

 20%|█▉        | 8207/41250 [19:49:13<79:21:05,  8.65s/it][2025-04-26 03:46:56,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 03:46:56,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.37 | bwd_microstep: 5771.71 | bwd_inner_microstep: 5718.16 | bwd_allreduce_microstep: 53.51 | step_microstep: 17.64
[2025-04-26 03:46:56,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.37 | bwd: 5771.72 | bwd_inner: 5718.16 | bwd_allreduce: 53.52 | step: 17.64

 20%|█▉        | 8208/41250 [19:49:22<79:33:16,  8.67s/it]
                                                          
{'loss': 0.1815, 'grad_norm': 3.4806206226348877, 'learning_rate': 3.707908691215539e-05, 'epoch': 1.99}

 20%|█▉        | 8208/41250 [19:49:22<79:33:16,  8.67s/it][2025-04-26 03:47:05,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:47:05,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.93 | bwd_microstep: 5713.55 | bwd_inner_microstep: 5668.76 | bwd_allreduce_microstep: 44.75 | step_microstep: 18.43
[2025-04-26 03:47:05,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.93 | bwd: 5713.57 | bwd_inner: 5668.76 | bwd_allreduce: 44.76 | step: 18.43

 20%|█▉        | 8209/41250 [19:49:30<79:28:35,  8.66s/it]
                                                          
{'loss': 0.0655, 'grad_norm': 0.9509207010269165, 'learning_rate': 3.707826974401806e-05, 'epoch': 1.99}

 20%|█▉        | 8209/41250 [19:49:30<79:28:35,  8.66s/it][2025-04-26 03:47:14,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 03:47:14,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.11 | bwd_microstep: 5731.07 | bwd_inner_microstep: 5718.30 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.06
[2025-04-26 03:47:14,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.11 | bwd: 5731.08 | bwd_inner: 5718.30 | bwd_allreduce: 12.74 | step: 18.06

 20%|█▉        | 8210/41250 [19:49:39<79:30:15,  8.66s/it]
                                                          
{'loss': 0.02, 'grad_norm': 0.8001929521560669, 'learning_rate': 3.707745247059656e-05, 'epoch': 1.99}

 20%|█▉        | 8210/41250 [19:49:39<79:30:15,  8.66s/it][2025-04-26 03:47:22,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.25 | optimizer_step: 0.98
[2025-04-26 03:47:22,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.87 | bwd_microstep: 5792.79 | bwd_inner_microstep: 5635.17 | bwd_allreduce_microstep: 157.57 | step_microstep: 19.59
[2025-04-26 03:47:22,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.87 | bwd: 5792.80 | bwd_inner: 5635.17 | bwd_allreduce: 157.59 | step: 19.60

 20%|█▉        | 8211/41250 [19:49:48<79:36:47,  8.67s/it]
                                                          
{'loss': 0.0754, 'grad_norm': 3.1600027084350586, 'learning_rate': 3.7076635091895925e-05, 'epoch': 1.99}

 20%|█▉        | 8211/41250 [19:49:48<79:36:47,  8.67s/it][2025-04-26 03:47:31,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 03:47:31,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.76 | bwd_microstep: 5720.37 | bwd_inner_microstep: 5688.14 | bwd_allreduce_microstep: 32.19 | step_microstep: 19.21
[2025-04-26 03:47:31,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.76 | bwd: 5720.39 | bwd_inner: 5688.14 | bwd_allreduce: 32.21 | step: 19.21

 20%|█▉        | 8212/41250 [19:49:56<79:36:30,  8.67s/it]
                                                          
{'loss': 0.1226, 'grad_norm': 6.197729110717773, 'learning_rate': 3.7075817607921186e-05, 'epoch': 1.99}

 20%|█▉        | 8212/41250 [19:49:56<79:36:30,  8.67s/it][2025-04-26 03:47:40,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:47:40,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.08 | bwd_microstep: 5770.01 | bwd_inner_microstep: 5649.98 | bwd_allreduce_microstep: 119.98 | step_microstep: 18.62
[2025-04-26 03:47:40,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.08 | bwd: 5770.02 | bwd_inner: 5649.98 | bwd_allreduce: 119.99 | step: 18.62

 20%|█▉        | 8213/41250 [19:50:05<79:37:49,  8.68s/it]
                                                          
{'loss': 0.2174, 'grad_norm': 4.82801628112793, 'learning_rate': 3.7075000018677395e-05, 'epoch': 1.99}

 20%|█▉        | 8213/41250 [19:50:05<79:37:49,  8.68s/it][2025-04-26 03:47:48,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:47:48,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.89 | bwd_microstep: 5715.13 | bwd_inner_microstep: 5702.45 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.85
[2025-04-26 03:47:48,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.89 | bwd: 5715.15 | bwd_inner: 5702.45 | bwd_allreduce: 12.66 | step: 18.86

 20%|█▉        | 8214/41250 [19:50:14<79:34:32,  8.67s/it]
                                                          
{'loss': 0.4054, 'grad_norm': 3.589655637741089, 'learning_rate': 3.707418232416959e-05, 'epoch': 1.99}

 20%|█▉        | 8214/41250 [19:50:14<79:34:32,  8.67s/it][2025-04-26 03:47:57,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 03:47:57,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.31 | bwd_microstep: 5775.89 | bwd_inner_microstep: 5640.05 | bwd_allreduce_microstep: 135.78 | step_microstep: 18.99
[2025-04-26 03:47:57,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.31 | bwd: 5775.90 | bwd_inner: 5640.05 | bwd_allreduce: 135.80 | step: 18.99

 20%|█▉        | 8215/41250 [19:50:22<79:37:42,  8.68s/it]
                                                          
{'loss': 0.0618, 'grad_norm': 1.4045472145080566, 'learning_rate': 3.70733645244028e-05, 'epoch': 1.99}

 20%|█▉        | 8215/41250 [19:50:22<79:37:42,  8.68s/it][2025-04-26 03:48:06,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 03:48:06,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.74 | bwd_microstep: 5877.26 | bwd_inner_microstep: 5864.39 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.08
[2025-04-26 03:48:06,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2929.74 | bwd: 5877.28 | bwd_inner: 5864.39 | bwd_allreduce: 12.85 | step: 19.08

 20%|█▉        | 8216/41250 [19:50:31<80:12:52,  8.74s/it]
                                                          
{'loss': 0.1578, 'grad_norm': 4.764684200286865, 'learning_rate': 3.707254661938208e-05, 'epoch': 1.99}

 20%|█▉        | 8216/41250 [19:50:31<80:12:52,  8.74s/it][2025-04-26 03:48:15,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:48:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5758.97 | bwd_inner_microstep: 5697.59 | bwd_allreduce_microstep: 61.33 | step_microstep: 18.72
[2025-04-26 03:48:15,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5758.98 | bwd_inner: 5697.59 | bwd_allreduce: 61.35 | step: 18.72

 20%|█▉        | 8217/41250 [19:50:40<80:04:36,  8.73s/it]
                                                          
{'loss': 0.1536, 'grad_norm': 2.607351779937744, 'learning_rate': 3.707172860911247e-05, 'epoch': 1.99}

 20%|█▉        | 8217/41250 [19:50:40<80:04:36,  8.73s/it][2025-04-26 03:48:23,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:48:23,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.32 | bwd_microstep: 5761.27 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 65.56 | step_microstep: 18.86
[2025-04-26 03:48:23,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.32 | bwd: 5761.28 | bwd_inner: 5695.67 | bwd_allreduce: 65.57 | step: 18.86

 20%|█▉        | 8218/41250 [19:50:49<79:59:33,  8.72s/it]
                                                          
{'loss': 0.1946, 'grad_norm': 4.11305046081543, 'learning_rate': 3.707091049359901e-05, 'epoch': 1.99}

 20%|█▉        | 8218/41250 [19:50:49<79:59:33,  8.72s/it][2025-04-26 03:48:32,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.12 | optimizer_step: 0.98
[2025-04-26 03:48:32,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.44 | bwd_microstep: 5747.53 | bwd_inner_microstep: 5648.32 | bwd_allreduce_microstep: 99.17 | step_microstep: 19.11
[2025-04-26 03:48:32,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.44 | bwd: 5747.55 | bwd_inner: 5648.32 | bwd_allreduce: 99.19 | step: 19.11

 20%|█▉        | 8219/41250 [19:50:57<79:49:44,  8.70s/it]
                                                          
{'loss': 0.0709, 'grad_norm': 1.3769067525863647, 'learning_rate': 3.707009227284674e-05, 'epoch': 1.99}

 20%|█▉        | 8219/41250 [19:50:57<79:49:44,  8.70s/it][2025-04-26 03:48:41,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:48:41,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.64 | bwd_microstep: 5789.01 | bwd_inner_microstep: 5776.23 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.72
[2025-04-26 03:48:41,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.64 | bwd: 5789.02 | bwd_inner: 5776.23 | bwd_allreduce: 12.75 | step: 18.72

 20%|█▉        | 8220/41250 [19:51:06<79:59:34,  8.72s/it]
                                                          
{'loss': 0.21, 'grad_norm': 4.793348789215088, 'learning_rate': 3.7069273946860714e-05, 'epoch': 1.99}

 20%|█▉        | 8220/41250 [19:51:06<79:59:34,  8.72s/it][2025-04-26 03:48:49,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:48:49,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.20 | bwd_microstep: 5759.91 | bwd_inner_microstep: 5634.82 | bwd_allreduce_microstep: 125.04 | step_microstep: 18.35
[2025-04-26 03:48:49,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.20 | bwd: 5759.92 | bwd_inner: 5634.82 | bwd_allreduce: 125.06 | step: 18.35

 20%|█▉        | 8221/41250 [19:51:15<79:49:37,  8.70s/it]
                                                          
{'loss': 0.3705, 'grad_norm': 3.300631523132324, 'learning_rate': 3.706845551564597e-05, 'epoch': 1.99}

 20%|█▉        | 8221/41250 [19:51:15<79:49:37,  8.70s/it][2025-04-26 03:48:58,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-26 03:48:58,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.67 | bwd_microstep: 5691.92 | bwd_inner_microstep: 5679.19 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.44
[2025-04-26 03:48:58,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.67 | bwd: 5691.93 | bwd_inner: 5679.19 | bwd_allreduce: 12.70 | step: 18.44

 20%|█▉        | 8222/41250 [19:51:23<79:35:50,  8.68s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 0.9716383218765259, 'learning_rate': 3.7067636979207555e-05, 'epoch': 1.99}

 20%|█▉        | 8222/41250 [19:51:23<79:35:50,  8.68s/it][2025-04-26 03:49:07,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.17 | optimizer_step: 1.06
[2025-04-26 03:49:07,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5691.74 | bwd_inner_microstep: 5651.47 | bwd_allreduce_microstep: 40.20 | step_microstep: 20.09
[2025-04-26 03:49:07,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.80 | bwd: 5691.76 | bwd_inner: 5651.47 | bwd_allreduce: 40.23 | step: 20.09

 20%|█▉        | 8223/41250 [19:51:32<79:23:46,  8.65s/it]
                                                          
{'loss': 0.2247, 'grad_norm': 2.7349510192871094, 'learning_rate': 3.7066818337550515e-05, 'epoch': 1.99}

 20%|█▉        | 8223/41250 [19:51:32<79:23:46,  8.65s/it][2025-04-26 03:49:15,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 03:49:15,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.12 | bwd_microstep: 5746.99 | bwd_inner_microstep: 5672.65 | bwd_allreduce_microstep: 74.30 | step_microstep: 19.19
[2025-04-26 03:49:15,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.12 | bwd: 5747.00 | bwd_inner: 5672.65 | bwd_allreduce: 74.31 | step: 19.19

 20%|█▉        | 8224/41250 [19:51:41<79:26:40,  8.66s/it]
                                                          
{'loss': 0.2446, 'grad_norm': 5.319218635559082, 'learning_rate': 3.7065999590679894e-05, 'epoch': 1.99}

 20%|█▉        | 8224/41250 [19:51:41<79:26:40,  8.66s/it][2025-04-26 03:49:24,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 03:49:24,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5701.79 | bwd_inner_microstep: 5688.67 | bwd_allreduce_microstep: 13.07 | step_microstep: 19.12
[2025-04-26 03:49:24,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5701.80 | bwd_inner: 5688.67 | bwd_allreduce: 13.09 | step: 19.12

 20%|█▉        | 8225/41250 [19:51:49<79:21:55,  8.65s/it]
                                                          
{'loss': 0.1084, 'grad_norm': 2.2792985439300537, 'learning_rate': 3.706518073860075e-05, 'epoch': 1.99}

 20%|█▉        | 8225/41250 [19:51:49<79:21:55,  8.65s/it][2025-04-26 03:49:33,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 03:49:33,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5731.17 | bwd_inner_microstep: 5679.73 | bwd_allreduce_microstep: 51.40 | step_microstep: 18.98
[2025-04-26 03:49:33,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5731.18 | bwd_inner: 5679.73 | bwd_allreduce: 51.41 | step: 18.98

 20%|█▉        | 8226/41250 [19:51:58<79:22:59,  8.65s/it]
                                                          
{'loss': 0.0699, 'grad_norm': 1.3460910320281982, 'learning_rate': 3.7064361781318116e-05, 'epoch': 1.99}

 20%|█▉        | 8226/41250 [19:51:58<79:22:59,  8.65s/it][2025-04-26 03:49:41,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 03:49:41,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.83 | bwd_microstep: 5746.96 | bwd_inner_microstep: 5692.93 | bwd_allreduce_microstep: 53.98 | step_microstep: 18.79
[2025-04-26 03:49:41,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.83 | bwd: 5746.97 | bwd_inner: 5692.93 | bwd_allreduce: 54.00 | step: 18.79

 20%|█▉        | 8227/41250 [19:52:07<79:26:28,  8.66s/it]
                                                          
{'loss': 0.0484, 'grad_norm': 1.3344134092330933, 'learning_rate': 3.7063542718837055e-05, 'epoch': 1.99}

 20%|█▉        | 8227/41250 [19:52:07<79:26:28,  8.66s/it][2025-04-26 03:49:50,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-26 03:49:50,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.85 | bwd_microstep: 5741.28 | bwd_inner_microstep: 5647.44 | bwd_allreduce_microstep: 93.78 | step_microstep: 19.43
[2025-04-26 03:49:50,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.85 | bwd: 5741.29 | bwd_inner: 5647.44 | bwd_allreduce: 93.80 | step: 19.43

 20%|█▉        | 8228/41250 [19:52:15<79:24:26,  8.66s/it]
                                                          
{'loss': 0.0352, 'grad_norm': 0.5573418736457825, 'learning_rate': 3.706272355116261e-05, 'epoch': 1.99}

 20%|█▉        | 8228/41250 [19:52:15<79:24:26,  8.66s/it][2025-04-26 03:49:58,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:49:58,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.67 | bwd_microstep: 5688.50 | bwd_inner_microstep: 5647.59 | bwd_allreduce_microstep: 40.86 | step_microstep: 18.68
[2025-04-26 03:49:58,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.67 | bwd: 5688.51 | bwd_inner: 5647.59 | bwd_allreduce: 40.88 | step: 18.68

 20%|█▉        | 8229/41250 [19:52:24<79:13:57,  8.64s/it]
                                                          
{'loss': 0.3757, 'grad_norm': 3.6784462928771973, 'learning_rate': 3.706190427829982e-05, 'epoch': 1.99}

 20%|█▉        | 8229/41250 [19:52:24<79:13:57,  8.64s/it][2025-04-26 03:50:07,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 03:50:07,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.60 | bwd_microstep: 5742.27 | bwd_inner_microstep: 5652.73 | bwd_allreduce_microstep: 89.49 | step_microstep: 19.00
[2025-04-26 03:50:07,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.60 | bwd: 5742.29 | bwd_inner: 5652.73 | bwd_allreduce: 89.51 | step: 19.00

 20%|█▉        | 8230/41250 [19:52:32<79:15:27,  8.64s/it]
                                                          
{'loss': 0.1197, 'grad_norm': 1.3351025581359863, 'learning_rate': 3.706108490025375e-05, 'epoch': 2.0}

 20%|█▉        | 8230/41250 [19:52:32<79:15:27,  8.64s/it][2025-04-26 03:50:16,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:50:16,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.49 | bwd_microstep: 5710.79 | bwd_inner_microstep: 5693.96 | bwd_allreduce_microstep: 16.78 | step_microstep: 18.74
[2025-04-26 03:50:16,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.49 | bwd: 5710.80 | bwd_inner: 5693.96 | bwd_allreduce: 16.80 | step: 18.75

 20%|█▉        | 8231/41250 [19:52:41<79:15:25,  8.64s/it]
                                                          
{'loss': 0.1212, 'grad_norm': 2.002558946609497, 'learning_rate': 3.706026541702945e-05, 'epoch': 2.0}

 20%|█▉        | 8231/41250 [19:52:41<79:15:25,  8.64s/it][2025-04-26 03:50:24,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.12 | optimizer_step: 0.96
[2025-04-26 03:50:24,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.89 | bwd_microstep: 5681.60 | bwd_inner_microstep: 5652.24 | bwd_allreduce_microstep: 29.31 | step_microstep: 18.72
[2025-04-26 03:50:24,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.89 | bwd: 5681.62 | bwd_inner: 5652.24 | bwd_allreduce: 29.33 | step: 18.72

 20%|█▉        | 8232/41250 [19:52:50<79:06:43,  8.63s/it]
                                                          
{'loss': 0.2134, 'grad_norm': 2.6666271686553955, 'learning_rate': 3.7059445828631966e-05, 'epoch': 2.0}

 20%|█▉        | 8232/41250 [19:52:50<79:06:43,  8.63s/it][2025-04-26 03:50:33,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 03:50:33,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.14 | bwd_microstep: 5746.21 | bwd_inner_microstep: 5681.77 | bwd_allreduce_microstep: 64.39 | step_microstep: 19.17
[2025-04-26 03:50:33,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.14 | bwd: 5746.23 | bwd_inner: 5681.77 | bwd_allreduce: 64.41 | step: 19.17

 20%|█▉        | 8233/41250 [19:52:58<79:13:54,  8.64s/it]
                                                          
{'loss': 0.2329, 'grad_norm': 3.985957145690918, 'learning_rate': 3.705862613506636e-05, 'epoch': 2.0}

 20%|█▉        | 8233/41250 [19:52:58<79:13:54,  8.64s/it][2025-04-26 03:50:42,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 03:50:42,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.38 | bwd_microstep: 5747.12 | bwd_inner_microstep: 5648.54 | bwd_allreduce_microstep: 98.53 | step_microstep: 18.43
[2025-04-26 03:50:42,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.38 | bwd: 5747.13 | bwd_inner: 5648.54 | bwd_allreduce: 98.55 | step: 18.43

 20%|█▉        | 8234/41250 [19:53:07<79:16:16,  8.64s/it]
                                                          
{'loss': 0.1034, 'grad_norm': 4.036986827850342, 'learning_rate': 3.705780633633767e-05, 'epoch': 2.0}

 20%|█▉        | 8234/41250 [19:53:07<79:16:16,  8.64s/it][2025-04-26 03:50:50,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.25 | optimizer_step: 0.93
[2025-04-26 03:50:50,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5741.15 | bwd_inner_microstep: 5652.82 | bwd_allreduce_microstep: 88.28 | step_microstep: 19.55
[2025-04-26 03:50:50,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5741.16 | bwd_inner: 5652.82 | bwd_allreduce: 88.30 | step: 19.55

 20%|█▉        | 8235/41250 [19:53:16<79:17:15,  8.65s/it]
                                                          
{'loss': 0.0472, 'grad_norm': 1.2205950021743774, 'learning_rate': 3.7056986432450966e-05, 'epoch': 2.0}

 20%|█▉        | 8235/41250 [19:53:16<79:17:15,  8.65s/it][2025-04-26 03:50:59,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 03:50:59,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.19 | bwd_microstep: 5716.42 | bwd_inner_microstep: 5687.01 | bwd_allreduce_microstep: 29.37 | step_microstep: 19.08
[2025-04-26 03:50:59,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.19 | bwd: 5716.44 | bwd_inner: 5687.01 | bwd_allreduce: 29.39 | step: 19.08

 20%|█▉        | 8236/41250 [19:53:24<79:17:20,  8.65s/it]
                                                          
{'loss': 0.2486, 'grad_norm': 3.3851234912872314, 'learning_rate': 3.705616642341129e-05, 'epoch': 2.0}

 20%|█▉        | 8236/41250 [19:53:24<79:17:20,  8.65s/it][2025-04-26 03:51:08,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-26 03:51:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.62 | bwd_microstep: 5694.02 | bwd_inner_microstep: 5645.29 | bwd_allreduce_microstep: 48.67 | step_microstep: 19.62
[2025-04-26 03:51:08,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.63 | bwd: 5694.03 | bwd_inner: 5645.29 | bwd_allreduce: 48.70 | step: 19.62

 20%|█▉        | 8237/41250 [19:53:33<79:10:44,  8.63s/it]
                                                          
{'loss': 0.092, 'grad_norm': 1.7295074462890625, 'learning_rate': 3.7055346309223706e-05, 'epoch': 2.0}

 20%|█▉        | 8237/41250 [19:53:33<79:10:44,  8.63s/it][2025-04-26 03:51:16,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:51:16,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.85 | bwd_microstep: 5789.91 | bwd_inner_microstep: 5651.41 | bwd_allreduce_microstep: 138.46 | step_microstep: 18.87
[2025-04-26 03:51:16,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.85 | bwd: 5789.92 | bwd_inner: 5651.41 | bwd_allreduce: 138.48 | step: 18.87

 20%|█▉        | 8238/41250 [19:53:42<79:20:43,  8.65s/it]
                                                          
{'loss': 0.0859, 'grad_norm': 2.91619873046875, 'learning_rate': 3.705452608989327e-05, 'epoch': 2.0}

 20%|█▉        | 8238/41250 [19:53:42<79:20:43,  8.65s/it][2025-04-26 03:51:25,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:51:25,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.00 | bwd_microstep: 5704.83 | bwd_inner_microstep: 5646.12 | bwd_allreduce_microstep: 58.67 | step_microstep: 18.63
[2025-04-26 03:51:25,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.00 | bwd: 5704.85 | bwd_inner: 5646.12 | bwd_allreduce: 58.68 | step: 18.64

 20%|█▉        | 8239/41250 [19:53:50<79:14:17,  8.64s/it]
                                                          
{'loss': 0.2063, 'grad_norm': 2.5826733112335205, 'learning_rate': 3.705370576542503e-05, 'epoch': 2.0}

 20%|█▉        | 8239/41250 [19:53:50<79:14:17,  8.64s/it][2025-04-26 03:51:34,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:51:34,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.09 | bwd_microstep: 5774.85 | bwd_inner_microstep: 5648.58 | bwd_allreduce_microstep: 126.23 | step_microstep: 18.82
[2025-04-26 03:51:34,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.09 | bwd: 5774.87 | bwd_inner: 5648.58 | bwd_allreduce: 126.24 | step: 18.82

 20%|█▉        | 8240/41250 [19:53:59<79:20:34,  8.65s/it]
                                                          
{'loss': 0.1668, 'grad_norm': 4.265397071838379, 'learning_rate': 3.705288533582405e-05, 'epoch': 2.0}

 20%|█▉        | 8240/41250 [19:53:59<79:20:34,  8.65s/it][2025-04-26 03:51:42,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:51:42,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.68 | bwd_microstep: 5694.17 | bwd_inner_microstep: 5656.68 | bwd_allreduce_microstep: 37.45 | step_microstep: 18.81
[2025-04-26 03:51:42,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.68 | bwd: 5694.18 | bwd_inner: 5656.68 | bwd_allreduce: 37.46 | step: 18.81

 20%|█▉        | 8241/41250 [19:54:07<79:12:26,  8.64s/it]
                                                          
{'loss': 0.0564, 'grad_norm': 0.8169367909431458, 'learning_rate': 3.705206480109539e-05, 'epoch': 2.0}

 20%|█▉        | 8241/41250 [19:54:08<79:12:26,  8.64s/it][2025-04-26 03:51:51,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.09
[2025-04-26 03:51:51,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.79 | bwd_microstep: 5771.99 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 84.40 | step_microstep: 19.25
[2025-04-26 03:51:51,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.79 | bwd: 5772.01 | bwd_inner: 5687.54 | bwd_allreduce: 84.42 | step: 19.26

 20%|█▉        | 8242/41250 [19:54:16<79:22:03,  8.66s/it]
                                                          
{'loss': 0.1822, 'grad_norm': 3.650533676147461, 'learning_rate': 3.7051244161244094e-05, 'epoch': 2.0}

 20%|█▉        | 8242/41250 [19:54:16<79:22:03,  8.66s/it][2025-04-26 03:52:00,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.12 | optimizer_step: 0.91
[2025-04-26 03:52:00,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.37 | bwd_microstep: 5786.18 | bwd_inner_microstep: 5684.27 | bwd_allreduce_microstep: 101.87 | step_microstep: 18.90
[2025-04-26 03:52:00,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.37 | bwd: 5786.19 | bwd_inner: 5684.27 | bwd_allreduce: 101.88 | step: 18.90

 20%|█▉        | 8243/41250 [19:54:25<79:31:12,  8.67s/it]
                                                          
{'loss': 0.1674, 'grad_norm': 1.8976936340332031, 'learning_rate': 3.7050423416275246e-05, 'epoch': 2.0}

 20%|█▉        | 8243/41250 [19:54:25<79:31:12,  8.67s/it][2025-04-26 03:52:08,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-26 03:52:08,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5754.59 | bwd_inner_microstep: 5656.56 | bwd_allreduce_microstep: 97.98 | step_microstep: 18.72
[2025-04-26 03:52:08,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5754.60 | bwd_inner: 5656.56 | bwd_allreduce: 98.00 | step: 18.72

 20%|█▉        | 8244/41250 [19:54:34<79:29:39,  8.67s/it]
                                                          
{'loss': 0.0861, 'grad_norm': 1.862705945968628, 'learning_rate': 3.704960256619388e-05, 'epoch': 2.0}

 20%|█▉        | 8244/41250 [19:54:34<79:29:39,  8.67s/it][2025-04-26 03:52:17,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 03:52:17,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.91 | bwd_microstep: 5738.02 | bwd_inner_microstep: 5680.23 | bwd_allreduce_microstep: 57.75 | step_microstep: 19.25
[2025-04-26 03:52:17,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.91 | bwd: 5738.03 | bwd_inner: 5680.23 | bwd_allreduce: 57.77 | step: 19.25

 20%|█▉        | 8245/41250 [19:54:42<79:28:06,  8.67s/it]
                                                          
{'loss': 0.0315, 'grad_norm': 0.5722662210464478, 'learning_rate': 3.704878161100507e-05, 'epoch': 2.0}

 20%|█▉        | 8245/41250 [19:54:42<79:28:06,  8.67s/it][2025-04-26 03:52:26,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 03:52:26,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.12 | bwd_microstep: 5790.64 | bwd_inner_microstep: 5777.96 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.89
[2025-04-26 03:52:26,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.13 | bwd: 5790.66 | bwd_inner: 5777.96 | bwd_allreduce: 12.65 | step: 18.89

 20%|█▉        | 8246/41250 [19:54:51<79:42:54,  8.70s/it]
                                                          
{'loss': 0.1537, 'grad_norm': 1.7087929248809814, 'learning_rate': 3.704796055071388e-05, 'epoch': 2.0}

 20%|█▉        | 8246/41250 [19:54:51<79:42:54,  8.70s/it][2025-04-26 03:52:34,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:52:34,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.63 | bwd_microstep: 5760.97 | bwd_inner_microstep: 5699.13 | bwd_allreduce_microstep: 61.80 | step_microstep: 18.77
[2025-04-26 03:52:34,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.63 | bwd: 5760.99 | bwd_inner: 5699.13 | bwd_allreduce: 61.81 | step: 18.78

 20%|█▉        | 8247/41250 [19:55:00<79:42:14,  8.69s/it]
                                                          
{'loss': 0.0604, 'grad_norm': 4.437797546386719, 'learning_rate': 3.704713938532536e-05, 'epoch': 2.0}

 20%|█▉        | 8247/41250 [19:55:00<79:42:14,  8.69s/it][2025-04-26 03:52:43,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 03:52:43,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.60 | bwd_microstep: 5887.28 | bwd_inner_microstep: 5692.20 | bwd_allreduce_microstep: 195.03 | step_microstep: 19.51
[2025-04-26 03:52:43,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.60 | bwd: 5887.30 | bwd_inner: 5692.20 | bwd_allreduce: 195.05 | step: 19.51

 20%|█▉        | 8248/41250 [19:55:09<80:02:01,  8.73s/it]
                                                          
{'loss': 0.0333, 'grad_norm': 0.8060459494590759, 'learning_rate': 3.7046318114844577e-05, 'epoch': 2.0}

 20%|█▉        | 8248/41250 [19:55:09<80:02:01,  8.73s/it][2025-04-26 03:52:52,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:52:52,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.68 | bwd_microstep: 5718.45 | bwd_inner_microstep: 5665.23 | bwd_allreduce_microstep: 53.18 | step_microstep: 18.61
[2025-04-26 03:52:52,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.68 | bwd: 5718.46 | bwd_inner: 5665.23 | bwd_allreduce: 53.19 | step: 18.61

 20%|█▉        | 8249/41250 [19:55:17<79:45:07,  8.70s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 3.424215078353882, 'learning_rate': 3.7045496739276606e-05, 'epoch': 2.0}

 20%|█▉        | 8249/41250 [19:55:17<79:45:07,  8.70s/it]Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-26 03:53:03,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 03:53:03,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.54 | bwd_microstep: 5766.58 | bwd_inner_microstep: 5753.67 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.32
[2025-04-26 03:53:03,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.54 | bwd: 5766.59 | bwd_inner: 5753.67 | bwd_allreduce: 12.88 | step: 19.32

 20%|██        | 8250/41250 [19:55:28<85:47:47,  9.36s/it]evaluate!evaluate!


                                                          
{'loss': 0.1095, 'grad_norm': 1.589577078819275, 'learning_rate': 3.704467525862649e-05, 'epoch': 2.0}

 20%|██        | 8250/41250 [19:55:28<85:47:47,  9.36s/it]evaluate!
1
1
1
[WARNING|trainer.py:803] 2025-04-26 03:53:05,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 03:53:05,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 03:53:06,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
2
2
[WARNING|trainer.py:803] 2025-04-26 03:53:08,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 03:53:08,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 03:53:08,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
3
3
[WARNING|trainer.py:803] 2025-04-26 03:53:10,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
evaluate!
[WARNING|trainer.py:803] 2025-04-26 03:53:11,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
evaluate!
[WARNING|trainer.py:803] 2025-04-26 03:53:11,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
evaluate!
1
[WARNING|trainer.py:803] 2025-04-26 03:53:12,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1
1
[WARNING|trainer.py:803] 2025-04-26 03:53:13,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 03:53:13,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
[WARNING|trainer.py:803] 2025-04-26 03:53:15,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2
2
[WARNING|trainer.py:803] 2025-04-26 03:53:16,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 03:53:16,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
[WARNING|trainer.py:803] 2025-04-26 03:53:17,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
3
3
[WARNING|trainer.py:803] 2025-04-26 03:53:18,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
[WARNING|trainer.py:803] 2025-04-26 03:53:18,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.3333333333333333
[2025-04-26 03:53:19,558] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:20,929] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:21,053] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 03:53:25,140] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:26,637] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:26,660] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 03:53:30,689] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:32,290] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:32,324] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 03:53:36,158] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:37,854] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 03:53:37,967] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 03:53:56,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.15 | optimizer_step: 1.10
[2025-04-26 03:53:56,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.98 | bwd_microstep: 8277.17 | bwd_inner_microstep: 5689.41 | bwd_allreduce_microstep: 2587.68 | step_microstep: 20.92
[2025-04-26 03:53:56,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.98 | bwd: 8277.19 | bwd_inner: 5689.41 | bwd_allreduce: 2587.73 | step: 20.92

 20%|██        | 8251/41250 [19:56:21<205:30:10, 22.42s/it]
                                                           
{'loss': 0.115, 'grad_norm': 12.450986862182617, 'learning_rate': 3.704385367289931e-05, 'epoch': 2.0}

 20%|██        | 8251/41250 [19:56:21<205:30:10, 22.42s/it][2025-04-26 03:54:04,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 03:54:04,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.66 | bwd_microstep: 5730.65 | bwd_inner_microstep: 5685.53 | bwd_allreduce_microstep: 45.07 | step_microstep: 19.72
[2025-04-26 03:54:04,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.66 | bwd: 5730.66 | bwd_inner: 5685.53 | bwd_allreduce: 45.09 | step: 19.72

 20%|██        | 8252/41250 [19:56:30<167:38:59, 18.29s/it]
                                                           
{'loss': 0.0292, 'grad_norm': 0.49935588240623474, 'learning_rate': 3.704303198210012e-05, 'epoch': 2.0}

 20%|██        | 8252/41250 [19:56:30<167:38:59, 18.29s/it][2025-04-26 03:54:13,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:54:13,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.21 | bwd_microstep: 5710.25 | bwd_inner_microstep: 5665.63 | bwd_allreduce_microstep: 44.58 | step_microstep: 18.76
[2025-04-26 03:54:13,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.21 | bwd: 5710.27 | bwd_inner: 5665.63 | bwd_allreduce: 44.59 | step: 18.77

 20%|██        | 8253/41250 [19:56:38<141:06:43, 15.40s/it]
                                                           
{'loss': 0.1062, 'grad_norm': 4.296016216278076, 'learning_rate': 3.704221018623399e-05, 'epoch': 2.0}

 20%|██        | 8253/41250 [19:56:38<141:06:43, 15.40s/it][2025-04-26 03:54:21,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 03:54:21,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.33 | bwd_microstep: 5664.39 | bwd_inner_microstep: 5651.45 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.28
[2025-04-26 03:54:21,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.33 | bwd: 5664.41 | bwd_inner: 5651.45 | bwd_allreduce: 12.91 | step: 19.29

 20%|██        | 8254/41250 [19:56:47<122:21:45, 13.35s/it]
                                                           
{'loss': 0.046, 'grad_norm': 2.329895496368408, 'learning_rate': 3.704138828530599e-05, 'epoch': 2.0}

 20%|██        | 8254/41250 [19:56:47<122:21:45, 13.35s/it][2025-04-26 03:54:30,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 03:54:30,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.39 | bwd_microstep: 5727.00 | bwd_inner_microstep: 5673.31 | bwd_allreduce_microstep: 53.64 | step_microstep: 18.08
[2025-04-26 03:54:30,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.39 | bwd: 5727.01 | bwd_inner: 5673.31 | bwd_allreduce: 53.66 | step: 18.09

 20%|██        | 8255/41250 [19:56:55<109:23:53, 11.94s/it]
                                                           
{'loss': 0.2176, 'grad_norm': 3.610060930252075, 'learning_rate': 3.704056627932118e-05, 'epoch': 2.0}

 20%|██        | 8255/41250 [19:56:55<109:23:53, 11.94s/it][2025-04-26 03:54:39,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 03:54:39,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.37 | bwd_microstep: 5833.11 | bwd_inner_microstep: 5670.26 | bwd_allreduce_microstep: 162.80 | step_microstep: 19.09
[2025-04-26 03:54:39,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.37 | bwd: 5833.12 | bwd_inner: 5670.26 | bwd_allreduce: 162.82 | step: 19.09

 20%|██        | 8256/41250 [19:57:04<100:41:39, 10.99s/it]
                                                           
{'loss': 0.1172, 'grad_norm': 1.948224425315857, 'learning_rate': 3.7039744168284634e-05, 'epoch': 2.0}

 20%|██        | 8256/41250 [19:57:04<100:41:39, 10.99s/it][2025-04-26 03:54:48,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 03:54:48,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.15 | bwd_microstep: 5707.15 | bwd_inner_microstep: 5694.53 | bwd_allreduce_microstep: 12.57 | step_microstep: 19.09
[2025-04-26 03:54:48,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.15 | bwd: 5707.16 | bwd_inner: 5694.53 | bwd_allreduce: 12.59 | step: 19.09

 20%|██        | 8257/41250 [19:57:13<94:13:40, 10.28s/it] 
                                                          
{'loss': 0.1908, 'grad_norm': 3.118178606033325, 'learning_rate': 3.703892195220141e-05, 'epoch': 2.0}

 20%|██        | 8257/41250 [19:57:13<94:13:40, 10.28s/it][2025-04-26 03:54:56,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 03:54:56,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.66 | bwd_microstep: 5731.56 | bwd_inner_microstep: 5635.16 | bwd_allreduce_microstep: 96.35 | step_microstep: 19.52
[2025-04-26 03:54:56,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.66 | bwd: 5731.57 | bwd_inner: 5635.16 | bwd_allreduce: 96.37 | step: 19.53

 20%|██        | 8258/41250 [19:57:21<89:42:04,  9.79s/it]
                                                          
{'loss': 0.0301, 'grad_norm': 0.6445997953414917, 'learning_rate': 3.7038099631076596e-05, 'epoch': 2.0}

 20%|██        | 8258/41250 [19:57:21<89:42:04,  9.79s/it][2025-04-26 03:55:05,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:55:05,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.57 | bwd_microstep: 5735.07 | bwd_inner_microstep: 5684.79 | bwd_allreduce_microstep: 50.24 | step_microstep: 18.84
[2025-04-26 03:55:05,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.57 | bwd: 5735.08 | bwd_inner: 5684.79 | bwd_allreduce: 50.26 | step: 18.84

 20%|██        | 8259/41250 [19:57:30<86:35:41,  9.45s/it]
                                                          
{'loss': 0.1052, 'grad_norm': 5.181938171386719, 'learning_rate': 3.7037277204915236e-05, 'epoch': 2.0}

 20%|██        | 8259/41250 [19:57:30<86:35:41,  9.45s/it][2025-04-26 03:55:13,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-26 03:55:13,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.51 | bwd_microstep: 5741.08 | bwd_inner_microstep: 5667.03 | bwd_allreduce_microstep: 74.01 | step_microstep: 18.64
[2025-04-26 03:55:13,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.51 | bwd: 5741.09 | bwd_inner: 5667.03 | bwd_allreduce: 74.02 | step: 18.64

 20%|██        | 8260/41250 [19:57:39<84:25:52,  9.21s/it]
                                                          
{'loss': 0.0611, 'grad_norm': 2.0673670768737793, 'learning_rate': 3.703645467372242e-05, 'epoch': 2.0}

 20%|██        | 8260/41250 [19:57:39<84:25:52,  9.21s/it][2025-04-26 03:55:22,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.33 | optimizer_step: 1.06
[2025-04-26 03:55:22,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.39 | bwd_microstep: 5797.57 | bwd_inner_microstep: 5673.66 | bwd_allreduce_microstep: 123.85 | step_microstep: 20.15
[2025-04-26 03:55:22,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.39 | bwd: 5797.59 | bwd_inner: 5673.66 | bwd_allreduce: 123.88 | step: 20.15

 20%|██        | 8261/41250 [19:57:48<83:04:45,  9.07s/it]
                                                          
{'loss': 0.1565, 'grad_norm': 2.20037841796875, 'learning_rate': 3.7035632037503206e-05, 'epoch': 2.0}

 20%|██        | 8261/41250 [19:57:48<83:04:45,  9.07s/it][2025-04-26 03:55:31,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 03:55:31,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.15 | bwd_microstep: 5678.07 | bwd_inner_microstep: 5665.42 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.76
[2025-04-26 03:55:31,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.15 | bwd: 5678.08 | bwd_inner: 5665.42 | bwd_allreduce: 12.63 | step: 18.77

 20%|██        | 8262/41250 [19:57:56<81:47:13,  8.93s/it]
                                                          
{'loss': 0.1131, 'grad_norm': 2.456502914428711, 'learning_rate': 3.703480929626268e-05, 'epoch': 2.0}

 20%|██        | 8262/41250 [19:57:56<81:47:13,  8.93s/it][2025-04-26 03:55:39,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 03:55:39,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.34 | bwd_microstep: 5728.54 | bwd_inner_microstep: 5681.17 | bwd_allreduce_microstep: 47.33 | step_microstep: 18.40
[2025-04-26 03:55:39,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.34 | bwd: 5728.56 | bwd_inner: 5681.17 | bwd_allreduce: 47.34 | step: 18.40

 20%|██        | 8263/41250 [19:58:05<81:02:33,  8.84s/it]
                                                          
{'loss': 0.0982, 'grad_norm': 2.8235065937042236, 'learning_rate': 3.7033986450005904e-05, 'epoch': 2.0}

 20%|██        | 8263/41250 [19:58:05<81:02:33,  8.84s/it][2025-04-26 03:55:48,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 03:55:48,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.11 | bwd_microstep: 5718.84 | bwd_inner_microstep: 5669.91 | bwd_allreduce_microstep: 48.89 | step_microstep: 18.66
[2025-04-26 03:55:48,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.11 | bwd: 5718.85 | bwd_inner: 5669.90 | bwd_allreduce: 48.91 | step: 18.67

 20%|██        | 8264/41250 [19:58:13<80:30:57,  8.79s/it]
                                                          
{'loss': 0.1634, 'grad_norm': 1.7482147216796875, 'learning_rate': 3.703316349873794e-05, 'epoch': 2.0}

 20%|██        | 8264/41250 [19:58:13<80:30:57,  8.79s/it][2025-04-26 03:55:57,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 03:55:57,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.17 | bwd_microstep: 5680.15 | bwd_inner_microstep: 5634.48 | bwd_allreduce_microstep: 45.62 | step_microstep: 18.81
[2025-04-26 03:55:57,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.17 | bwd: 5680.16 | bwd_inner: 5634.48 | bwd_allreduce: 45.64 | step: 18.81

 20%|██        | 8265/41250 [19:58:22<79:57:19,  8.73s/it]
                                                          
{'loss': 0.0707, 'grad_norm': 1.8428782224655151, 'learning_rate': 3.7032340442463886e-05, 'epoch': 2.0}

 20%|██        | 8265/41250 [19:58:22<79:57:19,  8.73s/it][2025-04-26 03:56:05,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 03:56:05,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.66 | bwd_microstep: 5674.97 | bwd_inner_microstep: 5662.13 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.48
[2025-04-26 03:56:05,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.66 | bwd: 5674.98 | bwd_inner: 5662.13 | bwd_allreduce: 12.81 | step: 18.49

 20%|██        | 8266/41250 [19:58:31<79:37:11,  8.69s/it]
                                                          
{'loss': 0.1675, 'grad_norm': 3.4046106338500977, 'learning_rate': 3.7031517281188795e-05, 'epoch': 2.0}

 20%|██        | 8266/41250 [19:58:31<79:37:11,  8.69s/it][2025-04-26 03:56:14,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.21 | optimizer_step: 1.03
[2025-04-26 03:56:14,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.59 | bwd_microstep: 5684.31 | bwd_inner_microstep: 5670.94 | bwd_allreduce_microstep: 13.32 | step_microstep: 19.55
[2025-04-26 03:56:14,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.59 | bwd: 5684.32 | bwd_inner: 5670.94 | bwd_allreduce: 13.34 | step: 19.56

 20%|██        | 8267/41250 [19:58:39<79:24:56,  8.67s/it]
                                                          
{'loss': 0.0446, 'grad_norm': 1.0650625228881836, 'learning_rate': 3.7030694014917756e-05, 'epoch': 2.0}

 20%|██        | 8267/41250 [19:58:39<79:24:56,  8.67s/it][2025-04-26 03:56:23,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 03:56:23,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.17 | bwd_microstep: 5705.73 | bwd_inner_microstep: 5693.00 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.52
[2025-04-26 03:56:23,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.17 | bwd: 5705.74 | bwd_inner: 5693.00 | bwd_allreduce: 12.71 | step: 19.52

 20%|██        | 8268/41250 [19:58:48<79:18:35,  8.66s/it]
                                                          
{'loss': 0.0378, 'grad_norm': 0.9459269046783447, 'learning_rate': 3.702987064365584e-05, 'epoch': 2.0}

 20%|██        | 8268/41250 [19:58:48<79:18:35,  8.66s/it][2025-04-26 03:56:31,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:56:31,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.86 | bwd_microstep: 5669.68 | bwd_inner_microstep: 5647.07 | bwd_allreduce_microstep: 22.57 | step_microstep: 18.67
[2025-04-26 03:56:31,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.86 | bwd: 5669.69 | bwd_inner: 5647.07 | bwd_allreduce: 22.59 | step: 18.67

 20%|██        | 8269/41250 [19:58:56<79:05:16,  8.63s/it]
                                                          
{'loss': 0.0848, 'grad_norm': 1.1309731006622314, 'learning_rate': 3.702904716740811e-05, 'epoch': 2.0}

 20%|██        | 8269/41250 [19:58:56<79:05:16,  8.63s/it][2025-04-26 03:56:40,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 03:56:40,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5755.74 | bwd_inner_microstep: 5640.81 | bwd_allreduce_microstep: 114.89 | step_microstep: 18.52
[2025-04-26 03:56:40,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5755.76 | bwd_inner: 5640.81 | bwd_allreduce: 114.91 | step: 18.52

 20%|██        | 8270/41250 [19:59:05<79:10:42,  8.64s/it]
                                                          
{'loss': 0.0547, 'grad_norm': 1.361976146697998, 'learning_rate': 3.702822358617966e-05, 'epoch': 2.0}

 20%|██        | 8270/41250 [19:59:05<79:10:42,  8.64s/it][2025-04-26 03:56:48,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.02 | optimizer_step: 1.22
[2025-04-26 03:56:48,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.55 | bwd_microstep: 5748.11 | bwd_inner_microstep: 5669.16 | bwd_allreduce_microstep: 78.90 | step_microstep: 19.23
[2025-04-26 03:56:48,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.55 | bwd: 5748.12 | bwd_inner: 5669.16 | bwd_allreduce: 78.92 | step: 19.23

 20%|██        | 8271/41250 [19:59:14<79:15:37,  8.65s/it]
                                                          
{'loss': 0.0869, 'grad_norm': 2.0236308574676514, 'learning_rate': 3.7027399899975564e-05, 'epoch': 2.01}

 20%|██        | 8271/41250 [19:59:14<79:15:37,  8.65s/it][2025-04-26 03:56:57,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 03:56:57,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.57 | bwd_microstep: 5681.01 | bwd_inner_microstep: 5633.01 | bwd_allreduce_microstep: 47.95 | step_microstep: 18.84
[2025-04-26 03:56:57,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.57 | bwd: 5681.02 | bwd_inner: 5633.01 | bwd_allreduce: 47.97 | step: 18.84

 20%|██        | 8272/41250 [19:59:22<79:04:15,  8.63s/it]
                                                          
{'loss': 0.1818, 'grad_norm': 4.344736576080322, 'learning_rate': 3.702657610880089e-05, 'epoch': 2.01}

 20%|██        | 8272/41250 [19:59:22<79:04:15,  8.63s/it][2025-04-26 03:57:06,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 03:57:06,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.14 | bwd_microstep: 5691.71 | bwd_inner_microstep: 5648.35 | bwd_allreduce_microstep: 43.32 | step_microstep: 18.54
[2025-04-26 03:57:06,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.14 | bwd: 5691.73 | bwd_inner: 5648.35 | bwd_allreduce: 43.34 | step: 18.54

 20%|██        | 8273/41250 [19:59:31<78:57:35,  8.62s/it]
                                                          
{'loss': 0.0927, 'grad_norm': 1.002933144569397, 'learning_rate': 3.702575221266073e-05, 'epoch': 2.01}

 20%|██        | 8273/41250 [19:59:31<78:57:35,  8.62s/it][2025-04-26 03:57:14,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:57:14,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.31 | bwd_microstep: 5881.99 | bwd_inner_microstep: 5676.17 | bwd_allreduce_microstep: 205.78 | step_microstep: 18.79
[2025-04-26 03:57:14,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.31 | bwd: 5882.01 | bwd_inner: 5676.17 | bwd_allreduce: 205.80 | step: 18.79

 20%|██        | 8274/41250 [19:59:40<79:28:39,  8.68s/it]
                                                          
{'loss': 0.1688, 'grad_norm': 4.633322238922119, 'learning_rate': 3.7024928211560155e-05, 'epoch': 2.01}

 20%|██        | 8274/41250 [19:59:40<79:28:39,  8.68s/it][2025-04-26 03:57:23,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 03:57:23,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.24 | bwd_microstep: 5712.03 | bwd_inner_microstep: 5647.67 | bwd_allreduce_microstep: 64.31 | step_microstep: 19.00
[2025-04-26 03:57:23,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.24 | bwd: 5712.04 | bwd_inner: 5647.67 | bwd_allreduce: 64.33 | step: 19.00

 20%|██        | 8275/41250 [19:59:48<79:19:46,  8.66s/it]
                                                          
{'loss': 0.0777, 'grad_norm': 3.6649184226989746, 'learning_rate': 3.702410410550424e-05, 'epoch': 2.01}

 20%|██        | 8275/41250 [19:59:48<79:19:46,  8.66s/it][2025-04-26 03:57:32,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 03:57:32,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.82 | bwd_microstep: 5721.05 | bwd_inner_microstep: 5699.78 | bwd_allreduce_microstep: 21.22 | step_microstep: 18.84
[2025-04-26 03:57:32,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.82 | bwd: 5721.06 | bwd_inner: 5699.78 | bwd_allreduce: 21.24 | step: 18.84

 20%|██        | 8276/41250 [19:59:57<79:17:45,  8.66s/it]
                                                          
{'loss': 0.0173, 'grad_norm': 0.8510460257530212, 'learning_rate': 3.702327989449808e-05, 'epoch': 2.01}

 20%|██        | 8276/41250 [19:59:57<79:17:45,  8.66s/it][2025-04-26 03:57:40,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 03:57:40,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.78 | bwd_microstep: 5717.02 | bwd_inner_microstep: 5649.11 | bwd_allreduce_microstep: 67.86 | step_microstep: 19.00
[2025-04-26 03:57:40,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.78 | bwd: 5717.04 | bwd_inner: 5649.11 | bwd_allreduce: 67.88 | step: 19.01

 20%|██        | 8277/41250 [20:00:06<79:12:03,  8.65s/it]
                                                          
{'loss': 0.0327, 'grad_norm': 1.2745857238769531, 'learning_rate': 3.702245557854674e-05, 'epoch': 2.01}

 20%|██        | 8277/41250 [20:00:06<79:12:03,  8.65s/it][2025-04-26 03:57:49,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 03:57:49,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.35 | bwd_microstep: 5746.60 | bwd_inner_microstep: 5685.36 | bwd_allreduce_microstep: 61.19 | step_microstep: 18.87
[2025-04-26 03:57:49,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.35 | bwd: 5746.61 | bwd_inner: 5685.36 | bwd_allreduce: 61.21 | step: 18.87

 20%|██        | 8278/41250 [20:00:14<79:17:24,  8.66s/it]
                                                          
{'loss': 0.0365, 'grad_norm': 1.8157235383987427, 'learning_rate': 3.7021631157655325e-05, 'epoch': 2.01}

 20%|██        | 8278/41250 [20:00:14<79:17:24,  8.66s/it][2025-04-26 03:57:58,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.13 | optimizer_step: 1.07
[2025-04-26 03:57:58,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.42 | bwd_microstep: 5720.10 | bwd_inner_microstep: 5643.10 | bwd_allreduce_microstep: 76.94 | step_microstep: 20.23
[2025-04-26 03:57:58,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.42 | bwd: 5720.12 | bwd_inner: 5643.10 | bwd_allreduce: 76.97 | step: 20.23

 20%|██        | 8279/41250 [20:00:23<79:11:55,  8.65s/it]
                                                          
{'loss': 0.1018, 'grad_norm': 1.8680955171585083, 'learning_rate': 3.7020806631828885e-05, 'epoch': 2.01}

 20%|██        | 8279/41250 [20:00:23<79:11:55,  8.65s/it][2025-04-26 03:58:06,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 03:58:06,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.48 | bwd_microstep: 5760.50 | bwd_inner_microstep: 5651.20 | bwd_allreduce_microstep: 109.25 | step_microstep: 18.96
[2025-04-26 03:58:06,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.48 | bwd: 5760.51 | bwd_inner: 5651.20 | bwd_allreduce: 109.27 | step: 18.96

 20%|██        | 8280/41250 [20:00:32<79:16:40,  8.66s/it]
                                                          
{'loss': 0.0927, 'grad_norm': 2.336892604827881, 'learning_rate': 3.7019982001072525e-05, 'epoch': 2.01}

 20%|██        | 8280/41250 [20:00:32<79:16:40,  8.66s/it][2025-04-26 03:58:15,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 03:58:15,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.72 | bwd_microstep: 5799.50 | bwd_inner_microstep: 5786.57 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.52
[2025-04-26 03:58:15,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.72 | bwd: 5799.51 | bwd_inner: 5786.57 | bwd_allreduce: 12.90 | step: 18.52

 20%|██        | 8281/41250 [20:00:40<79:34:58,  8.69s/it]
                                                          
{'loss': 0.2149, 'grad_norm': 2.2950174808502197, 'learning_rate': 3.701915726539133e-05, 'epoch': 2.01}

 20%|██        | 8281/41250 [20:00:40<79:34:58,  8.69s/it][2025-04-26 03:58:24,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 03:58:24,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5788.07 | bwd_inner_microstep: 5695.16 | bwd_allreduce_microstep: 92.85 | step_microstep: 18.76
[2025-04-26 03:58:24,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5788.08 | bwd_inner: 5695.16 | bwd_allreduce: 92.88 | step: 18.76

 20%|██        | 8282/41250 [20:00:49<79:39:31,  8.70s/it]
                                                          
{'loss': 0.0608, 'grad_norm': 1.9427648782730103, 'learning_rate': 3.701833242479037e-05, 'epoch': 2.01}

 20%|██        | 8282/41250 [20:00:49<79:39:31,  8.70s/it][2025-04-26 03:58:32,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 03:58:32,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.97 | bwd_microstep: 5707.95 | bwd_inner_microstep: 5657.14 | bwd_allreduce_microstep: 50.76 | step_microstep: 19.24
[2025-04-26 03:58:32,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.97 | bwd: 5707.96 | bwd_inner: 5657.14 | bwd_allreduce: 50.78 | step: 19.24

 20%|██        | 8283/41250 [20:00:58<79:27:18,  8.68s/it]
                                                          
{'loss': 0.0121, 'grad_norm': 0.5638201236724854, 'learning_rate': 3.7017507479274737e-05, 'epoch': 2.01}

 20%|██        | 8283/41250 [20:00:58<79:27:18,  8.68s/it][2025-04-26 03:58:41,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 03:58:41,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.47 | bwd_microstep: 5756.56 | bwd_inner_microstep: 5712.32 | bwd_allreduce_microstep: 44.20 | step_microstep: 19.30
[2025-04-26 03:58:41,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.47 | bwd: 5756.58 | bwd_inner: 5712.32 | bwd_allreduce: 44.21 | step: 19.30

 20%|██        | 8284/41250 [20:01:06<79:30:37,  8.68s/it]
                                                          
{'loss': 0.2291, 'grad_norm': 1.5324170589447021, 'learning_rate': 3.701668242884952e-05, 'epoch': 2.01}

 20%|██        | 8284/41250 [20:01:06<79:30:37,  8.68s/it][2025-04-26 03:58:50,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 03:58:50,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.50 | bwd_microstep: 5854.79 | bwd_inner_microstep: 5697.00 | bwd_allreduce_microstep: 157.75 | step_microstep: 18.92
[2025-04-26 03:58:50,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.50 | bwd: 5854.81 | bwd_inner: 5697.00 | bwd_allreduce: 157.76 | step: 18.92

 20%|██        | 8285/41250 [20:01:15<79:47:41,  8.71s/it]
                                                          
{'loss': 0.2085, 'grad_norm': 2.370657205581665, 'learning_rate': 3.70158572735198e-05, 'epoch': 2.01}

 20%|██        | 8285/41250 [20:01:15<79:47:41,  8.71s/it][2025-04-26 03:58:59,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-26 03:58:59,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.38 | bwd_microstep: 5708.10 | bwd_inner_microstep: 5695.14 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.73
[2025-04-26 03:58:59,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.38 | bwd: 5708.11 | bwd_inner: 5695.14 | bwd_allreduce: 12.93 | step: 18.72

 20%|██        | 8286/41250 [20:01:24<79:35:37,  8.69s/it]
                                                          
{'loss': 0.0157, 'grad_norm': 0.36310020089149475, 'learning_rate': 3.701503201329067e-05, 'epoch': 2.01}

 20%|██        | 8286/41250 [20:01:24<79:35:37,  8.69s/it][2025-04-26 03:59:07,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.01
[2025-04-26 03:59:07,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.60 | bwd_microstep: 5797.45 | bwd_inner_microstep: 5784.80 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.71
[2025-04-26 03:59:07,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.60 | bwd: 5797.47 | bwd_inner: 5784.80 | bwd_allreduce: 12.63 | step: 18.72

 20%|██        | 8287/41250 [20:01:33<79:48:56,  8.72s/it]
                                                          
{'loss': 0.0341, 'grad_norm': 0.7625123262405396, 'learning_rate': 3.7014206648167214e-05, 'epoch': 2.01}

 20%|██        | 8287/41250 [20:01:33<79:48:56,  8.72s/it][2025-04-26 03:59:16,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.07 | optimizer_step: 1.03
[2025-04-26 03:59:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.35 | bwd_microstep: 5764.18 | bwd_inner_microstep: 5655.47 | bwd_allreduce_microstep: 108.66 | step_microstep: 19.20
[2025-04-26 03:59:16,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.35 | bwd: 5764.19 | bwd_inner: 5655.47 | bwd_allreduce: 108.68 | step: 19.20

 20%|██        | 8288/41250 [20:01:41<79:42:15,  8.71s/it]
                                                          
{'loss': 0.0073, 'grad_norm': 0.11478783190250397, 'learning_rate': 3.7013381178154524e-05, 'epoch': 2.01}

 20%|██        | 8288/41250 [20:01:41<79:42:15,  8.71s/it][2025-04-26 03:59:25,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 03:59:25,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5721.10 | bwd_inner_microstep: 5663.84 | bwd_allreduce_microstep: 57.21 | step_microstep: 19.33
[2025-04-26 03:59:25,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5721.12 | bwd_inner: 5663.84 | bwd_allreduce: 57.23 | step: 19.33

 20%|██        | 8289/41250 [20:01:50<79:30:19,  8.68s/it]
                                                          
{'loss': 0.0228, 'grad_norm': 2.6264734268188477, 'learning_rate': 3.701255560325769e-05, 'epoch': 2.01}

 20%|██        | 8289/41250 [20:01:50<79:30:19,  8.68s/it][2025-04-26 03:59:33,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 03:59:33,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.55 | bwd_microstep: 5725.11 | bwd_inner_microstep: 5712.10 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.10
[2025-04-26 03:59:33,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.55 | bwd: 5725.13 | bwd_inner: 5712.10 | bwd_allreduce: 12.98 | step: 19.11

 20%|██        | 8290/41250 [20:01:59<79:27:59,  8.68s/it]
                                                          
{'loss': 0.0064, 'grad_norm': 0.1403326392173767, 'learning_rate': 3.701172992348179e-05, 'epoch': 2.01}

 20%|██        | 8290/41250 [20:01:59<79:27:59,  8.68s/it][2025-04-26 03:59:42,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.34 | optimizer_step: 0.94
[2025-04-26 03:59:42,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2899.54 | bwd_microstep: 5793.38 | bwd_inner_microstep: 5780.19 | bwd_allreduce_microstep: 13.13 | step_microstep: 20.18
[2025-04-26 03:59:42,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2899.55 | bwd: 5793.39 | bwd_inner: 5780.19 | bwd_allreduce: 13.15 | step: 20.18

 20%|██        | 8291/41250 [20:02:07<79:44:00,  8.71s/it]
                                                          
{'loss': 0.0742, 'grad_norm': 1.7737482786178589, 'learning_rate': 3.701090413883192e-05, 'epoch': 2.01}

 20%|██        | 8291/41250 [20:02:07<79:44:00,  8.71s/it][2025-04-26 03:59:51,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 0.89
[2025-04-26 03:59:51,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.54 | bwd_microstep: 5771.59 | bwd_inner_microstep: 5662.20 | bwd_allreduce_microstep: 109.33 | step_microstep: 19.10
[2025-04-26 03:59:51,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.54 | bwd: 5771.60 | bwd_inner: 5662.20 | bwd_allreduce: 109.36 | step: 19.10

 20%|██        | 8292/41250 [20:02:16<79:42:30,  8.71s/it]
                                                          
{'loss': 0.1531, 'grad_norm': 1.8178794384002686, 'learning_rate': 3.701007824931317e-05, 'epoch': 2.01}

 20%|██        | 8292/41250 [20:02:16<79:42:30,  8.71s/it][2025-04-26 03:59:59,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 03:59:59,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.33 | bwd_microstep: 5714.81 | bwd_inner_microstep: 5659.22 | bwd_allreduce_microstep: 55.54 | step_microstep: 18.73
[2025-04-26 03:59:59,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.33 | bwd: 5714.83 | bwd_inner: 5659.22 | bwd_allreduce: 55.57 | step: 18.73

 20%|██        | 8293/41250 [20:02:25<79:29:15,  8.68s/it]
                                                          
{'loss': 0.1812, 'grad_norm': 3.2375411987304688, 'learning_rate': 3.700925225493063e-05, 'epoch': 2.01}

 20%|██        | 8293/41250 [20:02:25<79:29:15,  8.68s/it][2025-04-26 04:00:08,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 04:00:08,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.51 | bwd_microstep: 5896.91 | bwd_inner_microstep: 5883.68 | bwd_allreduce_microstep: 13.18 | step_microstep: 18.61
[2025-04-26 04:00:08,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.51 | bwd: 5896.92 | bwd_inner: 5883.68 | bwd_allreduce: 13.20 | step: 18.61

 20%|██        | 8294/41250 [20:02:34<80:07:34,  8.75s/it]
                                                          
{'loss': 0.0311, 'grad_norm': 0.8989822268486023, 'learning_rate': 3.70084261556894e-05, 'epoch': 2.01}

 20%|██        | 8294/41250 [20:02:34<80:07:34,  8.75s/it][2025-04-26 04:00:17,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:00:17,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.06 | bwd_microstep: 5774.46 | bwd_inner_microstep: 5664.62 | bwd_allreduce_microstep: 109.79 | step_microstep: 18.11
[2025-04-26 04:00:17,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.06 | bwd: 5774.47 | bwd_inner: 5664.62 | bwd_allreduce: 109.81 | step: 18.11

 20%|██        | 8295/41250 [20:02:42<79:58:22,  8.74s/it]
                                                          
{'loss': 0.1033, 'grad_norm': 3.6340842247009277, 'learning_rate': 3.700759995159457e-05, 'epoch': 2.01}

 20%|██        | 8295/41250 [20:02:42<79:58:22,  8.74s/it][2025-04-26 04:00:26,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 04:00:26,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.46 | bwd_microstep: 5755.26 | bwd_inner_microstep: 5688.72 | bwd_allreduce_microstep: 66.49 | step_microstep: 18.89
[2025-04-26 04:00:26,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.46 | bwd: 5755.27 | bwd_inner: 5688.72 | bwd_allreduce: 66.51 | step: 18.89

 20%|██        | 8296/41250 [20:02:51<79:51:53,  8.72s/it]
                                                          
{'loss': 0.0133, 'grad_norm': 0.32194051146507263, 'learning_rate': 3.7006773642651235e-05, 'epoch': 2.01}

 20%|██        | 8296/41250 [20:02:51<79:51:53,  8.72s/it][2025-04-26 04:00:34,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.23 | optimizer_step: 1.00
[2025-04-26 04:00:34,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.30 | bwd_microstep: 5724.30 | bwd_inner_microstep: 5642.10 | bwd_allreduce_microstep: 82.15 | step_microstep: 19.34
[2025-04-26 04:00:34,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.30 | bwd: 5724.31 | bwd_inner: 5642.10 | bwd_allreduce: 82.17 | step: 19.35

 20%|██        | 8297/41250 [20:03:00<79:37:39,  8.70s/it]
                                                          
{'loss': 0.1313, 'grad_norm': 2.1083168983459473, 'learning_rate': 3.700594722886448e-05, 'epoch': 2.01}

 20%|██        | 8297/41250 [20:03:00<79:37:39,  8.70s/it][2025-04-26 04:00:43,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:00:43,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.28 | bwd_microstep: 5794.88 | bwd_inner_microstep: 5658.15 | bwd_allreduce_microstep: 136.69 | step_microstep: 18.49
[2025-04-26 04:00:43,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.28 | bwd: 5794.90 | bwd_inner: 5658.15 | bwd_allreduce: 136.71 | step: 18.49

 20%|██        | 8298/41250 [20:03:08<79:39:17,  8.70s/it]
                                                          
{'loss': 0.0065, 'grad_norm': 0.0979618951678276, 'learning_rate': 3.700512071023941e-05, 'epoch': 2.01}

 20%|██        | 8298/41250 [20:03:08<79:39:17,  8.70s/it][2025-04-26 04:00:52,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 04:00:52,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.43 | bwd_microstep: 5784.81 | bwd_inner_microstep: 5645.13 | bwd_allreduce_microstep: 139.63 | step_microstep: 18.83
[2025-04-26 04:00:52,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.43 | bwd: 5784.82 | bwd_inner: 5645.13 | bwd_allreduce: 139.64 | step: 18.83

 20%|██        | 8299/41250 [20:03:17<79:37:50,  8.70s/it]
                                                          
{'loss': 0.0443, 'grad_norm': 0.9721679091453552, 'learning_rate': 3.700429408678111e-05, 'epoch': 2.01}

 20%|██        | 8299/41250 [20:03:17<79:37:50,  8.70s/it][2025-04-26 04:01:00,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 04:01:00,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.47 | bwd_microstep: 5713.82 | bwd_inner_microstep: 5659.34 | bwd_allreduce_microstep: 54.43 | step_microstep: 19.14
[2025-04-26 04:01:00,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.47 | bwd: 5713.83 | bwd_inner: 5659.34 | bwd_allreduce: 54.45 | step: 19.13

 20%|██        | 8300/41250 [20:03:26<79:25:30,  8.68s/it]
                                                          
{'loss': 0.0445, 'grad_norm': 1.1234980821609497, 'learning_rate': 3.700346735849468e-05, 'epoch': 2.01}

 20%|██        | 8300/41250 [20:03:26<79:25:30,  8.68s/it][2025-04-26 04:01:09,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-26 04:01:09,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.60 | bwd_microstep: 5734.82 | bwd_inner_microstep: 5704.30 | bwd_allreduce_microstep: 30.47 | step_microstep: 19.02
[2025-04-26 04:01:09,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.60 | bwd: 5734.83 | bwd_inner: 5704.30 | bwd_allreduce: 30.49 | step: 19.03

 20%|██        | 8301/41250 [20:03:34<79:24:11,  8.68s/it]
                                                          
{'loss': 0.2106, 'grad_norm': 3.0970489978790283, 'learning_rate': 3.700264052538522e-05, 'epoch': 2.01}

 20%|██        | 8301/41250 [20:03:34<79:24:11,  8.68s/it][2025-04-26 04:01:18,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.07 | optimizer_step: 0.95
[2025-04-26 04:01:18,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.27 | bwd_microstep: 5696.56 | bwd_inner_microstep: 5683.51 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.58
[2025-04-26 04:01:18,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.27 | bwd: 5696.57 | bwd_inner: 5683.51 | bwd_allreduce: 13.02 | step: 19.59

 20%|██        | 8302/41250 [20:03:43<79:16:32,  8.66s/it]
                                                          
{'loss': 0.2904, 'grad_norm': 3.527890920639038, 'learning_rate': 3.700181358745783e-05, 'epoch': 2.01}

 20%|██        | 8302/41250 [20:03:43<79:16:32,  8.66s/it][2025-04-26 04:01:26,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:01:26,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.78 | bwd_microstep: 5739.62 | bwd_inner_microstep: 5681.08 | bwd_allreduce_microstep: 58.50 | step_microstep: 18.83
[2025-04-26 04:01:26,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.78 | bwd: 5739.63 | bwd_inner: 5681.08 | bwd_allreduce: 58.51 | step: 18.84

 20%|██        | 8303/41250 [20:03:52<79:17:23,  8.66s/it]
                                                          
{'loss': 0.0343, 'grad_norm': 0.7391858100891113, 'learning_rate': 3.70009865447176e-05, 'epoch': 2.01}

 20%|██        | 8303/41250 [20:03:52<79:17:23,  8.66s/it][2025-04-26 04:01:35,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 04:01:35,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.80 | bwd_microstep: 5737.76 | bwd_inner_microstep: 5714.42 | bwd_allreduce_microstep: 23.29 | step_microstep: 19.58
[2025-04-26 04:01:35,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.80 | bwd: 5737.77 | bwd_inner: 5714.42 | bwd_allreduce: 23.31 | step: 19.59

 20%|██        | 8304/41250 [20:04:00<79:19:05,  8.67s/it]
                                                          
{'loss': 0.0826, 'grad_norm': 1.9620847702026367, 'learning_rate': 3.7000159397169636e-05, 'epoch': 2.01}

 20%|██        | 8304/41250 [20:04:00<79:19:05,  8.67s/it][2025-04-26 04:01:44,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:01:44,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.97 | bwd_microstep: 5767.25 | bwd_inner_microstep: 5648.63 | bwd_allreduce_microstep: 118.58 | step_microstep: 18.66
[2025-04-26 04:01:44,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.97 | bwd: 5767.26 | bwd_inner: 5648.63 | bwd_allreduce: 118.59 | step: 18.66

 20%|██        | 8305/41250 [20:04:09<79:22:16,  8.67s/it]
                                                          
{'loss': 0.1885, 'grad_norm': 1.8427672386169434, 'learning_rate': 3.699933214481902e-05, 'epoch': 2.01}

 20%|██        | 8305/41250 [20:04:09<79:22:16,  8.67s/it][2025-04-26 04:01:52,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 04:01:52,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.79 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5694.89 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.05
[2025-04-26 04:01:52,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.79 | bwd: 5707.57 | bwd_inner: 5694.89 | bwd_allreduce: 12.63 | step: 19.05

 20%|██        | 8306/41250 [20:04:18<79:16:03,  8.66s/it]
                                                          
{'loss': 0.0998, 'grad_norm': 4.431005954742432, 'learning_rate': 3.699850478767088e-05, 'epoch': 2.01}

 20%|██        | 8306/41250 [20:04:18<79:16:03,  8.66s/it][2025-04-26 04:02:01,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:02:01,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5732.05 | bwd_inner_microstep: 5700.19 | bwd_allreduce_microstep: 31.82 | step_microstep: 18.65
[2025-04-26 04:02:01,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.83 | bwd: 5732.06 | bwd_inner: 5700.19 | bwd_allreduce: 31.84 | step: 18.65

 20%|██        | 8307/41250 [20:04:26<79:16:46,  8.66s/it]
                                                          
{'loss': 0.1107, 'grad_norm': 1.9958207607269287, 'learning_rate': 3.69976773257303e-05, 'epoch': 2.01}

 20%|██        | 8307/41250 [20:04:26<79:16:46,  8.66s/it][2025-04-26 04:02:10,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 04:02:10,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.35 | bwd_microstep: 5706.70 | bwd_inner_microstep: 5693.82 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.91
[2025-04-26 04:02:10,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.35 | bwd: 5706.71 | bwd_inner: 5693.82 | bwd_allreduce: 12.84 | step: 18.91

 20%|██        | 8308/41250 [20:04:35<79:13:12,  8.66s/it]
                                                          
{'loss': 0.0283, 'grad_norm': 0.6256017088890076, 'learning_rate': 3.699684975900237e-05, 'epoch': 2.01}

 20%|██        | 8308/41250 [20:04:35<79:13:12,  8.66s/it][2025-04-26 04:02:18,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 04:02:18,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.67 | bwd_microstep: 5739.04 | bwd_inner_microstep: 5641.65 | bwd_allreduce_microstep: 97.34 | step_microstep: 18.59
[2025-04-26 04:02:18,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.68 | bwd: 5739.05 | bwd_inner: 5641.65 | bwd_allreduce: 97.36 | step: 18.60

 20%|██        | 8309/41250 [20:04:44<79:11:07,  8.65s/it]
                                                          
{'loss': 0.2795, 'grad_norm': 3.5431010723114014, 'learning_rate': 3.699602208749221e-05, 'epoch': 2.01}

 20%|██        | 8309/41250 [20:04:44<79:11:07,  8.65s/it][2025-04-26 04:02:27,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.20 | optimizer_step: 0.99
[2025-04-26 04:02:27,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.08 | bwd_microstep: 5761.77 | bwd_inner_microstep: 5652.52 | bwd_allreduce_microstep: 109.20 | step_microstep: 19.64
[2025-04-26 04:02:27,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.08 | bwd: 5761.79 | bwd_inner: 5652.52 | bwd_allreduce: 109.23 | step: 19.64

 20%|██        | 8310/41250 [20:04:52<79:14:23,  8.66s/it]
                                                          
{'loss': 0.1287, 'grad_norm': 3.353853225708008, 'learning_rate': 3.6995194311204924e-05, 'epoch': 2.01}

 20%|██        | 8310/41250 [20:04:52<79:14:23,  8.66s/it][2025-04-26 04:02:36,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 04:02:36,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.24 | bwd_microstep: 5776.21 | bwd_inner_microstep: 5657.71 | bwd_allreduce_microstep: 118.45 | step_microstep: 18.75
[2025-04-26 04:02:36,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.24 | bwd: 5776.23 | bwd_inner: 5657.71 | bwd_allreduce: 118.47 | step: 18.75

 20%|██        | 8311/41250 [20:05:01<79:18:28,  8.67s/it]
                                                          
{'loss': 0.0164, 'grad_norm': 0.4206230938434601, 'learning_rate': 3.6994366430145604e-05, 'epoch': 2.01}

 20%|██        | 8311/41250 [20:05:01<79:18:28,  8.67s/it][2025-04-26 04:02:44,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.09 | optimizer_step: 0.98
[2025-04-26 04:02:44,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.78 | bwd_microstep: 5697.23 | bwd_inner_microstep: 5683.84 | bwd_allreduce_microstep: 13.34 | step_microstep: 19.44
[2025-04-26 04:02:44,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.78 | bwd: 5697.25 | bwd_inner: 5683.84 | bwd_allreduce: 13.36 | step: 19.44

 20%|██        | 8312/41250 [20:05:10<79:10:32,  8.65s/it]
                                                          
{'loss': 0.3421, 'grad_norm': 3.744706392288208, 'learning_rate': 3.6993538444319354e-05, 'epoch': 2.02}

 20%|██        | 8312/41250 [20:05:10<79:10:32,  8.65s/it][2025-04-26 04:02:53,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:02:53,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.22 | bwd_microstep: 5771.78 | bwd_inner_microstep: 5653.04 | bwd_allreduce_microstep: 118.70 | step_microstep: 18.60
[2025-04-26 04:02:53,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.22 | bwd: 5771.79 | bwd_inner: 5653.04 | bwd_allreduce: 118.72 | step: 18.61

 20%|██        | 8313/41250 [20:05:18<79:14:24,  8.66s/it]
                                                          
{'loss': 0.0286, 'grad_norm': 0.9030571579933167, 'learning_rate': 3.699271035373129e-05, 'epoch': 2.02}

 20%|██        | 8313/41250 [20:05:18<79:14:24,  8.66s/it][2025-04-26 04:03:02,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 04:03:02,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.38 | bwd_microstep: 5788.09 | bwd_inner_microstep: 5775.34 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.69
[2025-04-26 04:03:02,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.38 | bwd: 5788.10 | bwd_inner: 5775.34 | bwd_allreduce: 12.72 | step: 18.70

 20%|██        | 8314/41250 [20:05:27<79:29:49,  8.69s/it]
                                                          
{'loss': 0.245, 'grad_norm': 1.8476160764694214, 'learning_rate': 3.69918821583865e-05, 'epoch': 2.02}

 20%|██        | 8314/41250 [20:05:27<79:29:49,  8.69s/it][2025-04-26 04:03:10,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:03:10,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.74 | bwd_microstep: 5699.50 | bwd_inner_microstep: 5636.38 | bwd_allreduce_microstep: 63.07 | step_microstep: 18.71
[2025-04-26 04:03:10,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.74 | bwd: 5699.52 | bwd_inner: 5636.38 | bwd_allreduce: 63.09 | step: 18.71

 20%|██        | 8315/41250 [20:05:36<79:15:15,  8.66s/it]
                                                          
{'loss': 0.0693, 'grad_norm': 2.2404050827026367, 'learning_rate': 3.69910538582901e-05, 'epoch': 2.02}

 20%|██        | 8315/41250 [20:05:36<79:15:15,  8.66s/it][2025-04-26 04:03:19,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:03:19,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5757.63 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 57.41 | step_microstep: 18.68
[2025-04-26 04:03:19,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.79 | bwd: 5757.64 | bwd_inner: 5700.17 | bwd_allreduce: 57.43 | step: 18.68

 20%|██        | 8316/41250 [20:05:44<79:19:16,  8.67s/it]
                                                          
{'loss': 0.1501, 'grad_norm': 1.1123743057250977, 'learning_rate': 3.69902254534472e-05, 'epoch': 2.02}

 20%|██        | 8316/41250 [20:05:44<79:19:16,  8.67s/it][2025-04-26 04:03:28,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:03:28,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.28 | bwd_microstep: 5702.58 | bwd_inner_microstep: 5689.85 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.87
[2025-04-26 04:03:28,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.28 | bwd: 5702.60 | bwd_inner: 5689.85 | bwd_allreduce: 12.70 | step: 18.87

 20%|██        | 8317/41250 [20:05:53<79:12:06,  8.66s/it]
                                                          
{'loss': 0.1928, 'grad_norm': 1.413202166557312, 'learning_rate': 3.69893969438629e-05, 'epoch': 2.02}

 20%|██        | 8317/41250 [20:05:53<79:12:06,  8.66s/it][2025-04-26 04:03:36,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 04:03:36,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5719.91 | bwd_inner_microstep: 5687.81 | bwd_allreduce_microstep: 32.05 | step_microstep: 19.32
[2025-04-26 04:03:36,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5719.92 | bwd_inner: 5687.81 | bwd_allreduce: 32.07 | step: 19.32

 20%|██        | 8318/41250 [20:06:02<79:11:07,  8.66s/it]
                                                          
{'loss': 0.0111, 'grad_norm': 0.30280834436416626, 'learning_rate': 3.698856832954231e-05, 'epoch': 2.02}

 20%|██        | 8318/41250 [20:06:02<79:11:07,  8.66s/it][2025-04-26 04:03:45,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:03:45,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5731.23 | bwd_inner_microstep: 5718.23 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.52
[2025-04-26 04:03:45,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.29 | bwd: 5731.24 | bwd_inner: 5718.23 | bwd_allreduce: 12.98 | step: 18.52

 20%|██        | 8319/41250 [20:06:10<79:12:24,  8.66s/it]
                                                          
{'loss': 0.0874, 'grad_norm': 1.850829005241394, 'learning_rate': 3.698773961049053e-05, 'epoch': 2.02}

 20%|██        | 8319/41250 [20:06:10<79:12:24,  8.66s/it][2025-04-26 04:03:54,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 04:03:54,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.45 | bwd_microstep: 5766.14 | bwd_inner_microstep: 5753.45 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.57
[2025-04-26 04:03:54,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.45 | bwd: 5766.16 | bwd_inner: 5753.45 | bwd_allreduce: 12.66 | step: 18.57

 20%|██        | 8320/41250 [20:06:19<79:23:49,  8.68s/it]
                                                          
{'loss': 0.1201, 'grad_norm': 14.866388320922852, 'learning_rate': 3.698691078671268e-05, 'epoch': 2.02}

 20%|██        | 8320/41250 [20:06:19<79:23:49,  8.68s/it][2025-04-26 04:04:02,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:04:02,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5760.78 | bwd_inner_microstep: 5689.70 | bwd_allreduce_microstep: 71.04 | step_microstep: 18.62
[2025-04-26 04:04:02,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5760.80 | bwd_inner: 5689.70 | bwd_allreduce: 71.05 | step: 18.62

 20%|██        | 8321/41250 [20:06:28<79:25:36,  8.68s/it]
                                                          
{'loss': 0.1847, 'grad_norm': 1.0680114030838013, 'learning_rate': 3.698608185821387e-05, 'epoch': 2.02}

 20%|██        | 8321/41250 [20:06:28<79:25:36,  8.68s/it][2025-04-26 04:04:11,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-26 04:04:11,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5691.06 | bwd_inner_microstep: 5678.33 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.83
[2025-04-26 04:04:11,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5691.07 | bwd_inner: 5678.33 | bwd_allreduce: 12.70 | step: 18.83

 20%|██        | 8322/41250 [20:06:36<79:16:22,  8.67s/it]
                                                          
{'loss': 0.0912, 'grad_norm': 3.6967267990112305, 'learning_rate': 3.698525282499921e-05, 'epoch': 2.02}

 20%|██        | 8322/41250 [20:06:36<79:16:22,  8.67s/it][2025-04-26 04:04:20,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:04:20,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5802.31 | bwd_inner_microstep: 5663.36 | bwd_allreduce_microstep: 138.91 | step_microstep: 18.80
[2025-04-26 04:04:20,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5802.33 | bwd_inner: 5663.36 | bwd_allreduce: 138.92 | step: 18.80

 20%|██        | 8323/41250 [20:06:45<79:24:09,  8.68s/it]
                                                          
{'loss': 0.1747, 'grad_norm': 7.851690292358398, 'learning_rate': 3.69844236870738e-05, 'epoch': 2.02}

 20%|██        | 8323/41250 [20:06:45<79:24:09,  8.68s/it][2025-04-26 04:04:28,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.25
[2025-04-26 04:04:28,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.81 | bwd_microstep: 5725.34 | bwd_inner_microstep: 5659.43 | bwd_allreduce_microstep: 65.86 | step_microstep: 19.86
[2025-04-26 04:04:28,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.81 | bwd: 5725.36 | bwd_inner: 5659.43 | bwd_allreduce: 65.88 | step: 19.86

 20%|██        | 8324/41250 [20:06:54<79:17:48,  8.67s/it]
                                                          
{'loss': 0.148, 'grad_norm': 2.651956558227539, 'learning_rate': 3.698359444444276e-05, 'epoch': 2.02}

 20%|██        | 8324/41250 [20:06:54<79:17:48,  8.67s/it][2025-04-26 04:04:37,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 04:04:37,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5714.78 | bwd_inner_microstep: 5701.91 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.01
[2025-04-26 04:04:37,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5714.79 | bwd_inner: 5701.91 | bwd_allreduce: 12.83 | step: 19.01

 20%|██        | 8325/41250 [20:07:02<79:13:15,  8.66s/it]
                                                          
{'loss': 0.2193, 'grad_norm': 1.4294992685317993, 'learning_rate': 3.698276509711121e-05, 'epoch': 2.02}

 20%|██        | 8325/41250 [20:07:02<79:13:15,  8.66s/it][2025-04-26 04:04:46,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:04:46,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.62 | bwd_microstep: 5727.30 | bwd_inner_microstep: 5658.52 | bwd_allreduce_microstep: 68.73 | step_microstep: 19.01
[2025-04-26 04:04:46,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5727.32 | bwd_inner: 5658.52 | bwd_allreduce: 68.75 | step: 19.00

 20%|██        | 8326/41250 [20:07:11<79:10:09,  8.66s/it]
                                                          
{'loss': 0.0741, 'grad_norm': 0.9998537302017212, 'learning_rate': 3.698193564508424e-05, 'epoch': 2.02}

 20%|██        | 8326/41250 [20:07:11<79:10:09,  8.66s/it][2025-04-26 04:04:54,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 04:04:54,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5878.98 | bwd_inner_microstep: 5692.33 | bwd_allreduce_microstep: 186.60 | step_microstep: 19.40
[2025-04-26 04:04:54,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5879.00 | bwd_inner: 5692.33 | bwd_allreduce: 186.63 | step: 19.41

 20%|██        | 8327/41250 [20:07:20<79:35:21,  8.70s/it]
                                                          
{'loss': 0.109, 'grad_norm': 2.0138089656829834, 'learning_rate': 3.6981106088366984e-05, 'epoch': 2.02}

 20%|██        | 8327/41250 [20:07:20<79:35:21,  8.70s/it][2025-04-26 04:05:03,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 04:05:03,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.49 | bwd_microstep: 5770.55 | bwd_inner_microstep: 5659.81 | bwd_allreduce_microstep: 110.69 | step_microstep: 18.98
[2025-04-26 04:05:03,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.49 | bwd: 5770.56 | bwd_inner: 5659.81 | bwd_allreduce: 110.71 | step: 18.98

 20%|██        | 8328/41250 [20:07:28<79:32:49,  8.70s/it]
                                                          
{'loss': 0.1871, 'grad_norm': 4.311672687530518, 'learning_rate': 3.6980276426964545e-05, 'epoch': 2.02}

 20%|██        | 8328/41250 [20:07:28<79:32:49,  8.70s/it][2025-04-26 04:05:12,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-26 04:05:12,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.08 | bwd_microstep: 5779.41 | bwd_inner_microstep: 5698.68 | bwd_allreduce_microstep: 80.67 | step_microstep: 19.08
[2025-04-26 04:05:12,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.08 | bwd: 5779.42 | bwd_inner: 5698.68 | bwd_allreduce: 80.70 | step: 19.08

 20%|██        | 8329/41250 [20:07:37<79:35:08,  8.70s/it]
                                                          
{'loss': 0.0702, 'grad_norm': 2.312218427658081, 'learning_rate': 3.697944666088204e-05, 'epoch': 2.02}

 20%|██        | 8329/41250 [20:07:37<79:35:08,  8.70s/it][2025-04-26 04:05:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 04:05:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.52 | bwd_microstep: 5768.37 | bwd_inner_microstep: 5703.08 | bwd_allreduce_microstep: 65.25 | step_microstep: 18.52
[2025-04-26 04:05:21,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.52 | bwd: 5768.38 | bwd_inner: 5703.08 | bwd_allreduce: 65.26 | step: 18.52

 20%|██        | 8330/41250 [20:07:46<79:35:46,  8.70s/it]
                                                          
{'loss': 0.0496, 'grad_norm': 1.0645071268081665, 'learning_rate': 3.6978616790124594e-05, 'epoch': 2.02}

 20%|██        | 8330/41250 [20:07:46<79:35:46,  8.70s/it][2025-04-26 04:05:29,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.16 | optimizer_step: 0.92
[2025-04-26 04:05:29,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.23 | bwd_microstep: 5802.40 | bwd_inner_microstep: 5670.19 | bwd_allreduce_microstep: 132.16 | step_microstep: 18.88
[2025-04-26 04:05:29,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.23 | bwd: 5802.41 | bwd_inner: 5670.19 | bwd_allreduce: 132.18 | step: 18.88

 20%|██        | 8331/41250 [20:07:55<79:38:01,  8.71s/it]
                                                          
{'loss': 0.3554, 'grad_norm': 4.27353048324585, 'learning_rate': 3.6977786814697314e-05, 'epoch': 2.02}

 20%|██        | 8331/41250 [20:07:55<79:38:01,  8.71s/it][2025-04-26 04:05:38,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 04:05:38,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.73 | bwd_microstep: 5744.99 | bwd_inner_microstep: 5652.73 | bwd_allreduce_microstep: 92.21 | step_microstep: 18.90
[2025-04-26 04:05:38,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.73 | bwd: 5745.01 | bwd_inner: 5652.73 | bwd_allreduce: 92.23 | step: 18.91

 20%|██        | 8332/41250 [20:08:03<79:29:03,  8.69s/it]
                                                          
{'loss': 0.0411, 'grad_norm': 0.8492656350135803, 'learning_rate': 3.697695673460531e-05, 'epoch': 2.02}

 20%|██        | 8332/41250 [20:08:03<79:29:03,  8.69s/it][2025-04-26 04:05:47,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:05:47,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.01 | bwd_microstep: 5731.89 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 74.01 | step_microstep: 19.14
[2025-04-26 04:05:47,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.01 | bwd: 5731.90 | bwd_inner: 5657.84 | bwd_allreduce: 74.03 | step: 19.14

 20%|██        | 8333/41250 [20:08:12<79:21:08,  8.68s/it]
                                                          
{'loss': 0.0424, 'grad_norm': 0.6605932116508484, 'learning_rate': 3.697612654985371e-05, 'epoch': 2.02}

 20%|██        | 8333/41250 [20:08:12<79:21:08,  8.68s/it][2025-04-26 04:05:55,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:05:55,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.97 | bwd_microstep: 5720.66 | bwd_inner_microstep: 5707.96 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.62
[2025-04-26 04:05:55,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.98 | bwd: 5720.68 | bwd_inner: 5707.96 | bwd_allreduce: 12.68 | step: 18.63

 20%|██        | 8334/41250 [20:08:21<79:18:08,  8.67s/it]
                                                          
{'loss': 0.0767, 'grad_norm': 3.2833404541015625, 'learning_rate': 3.6975296260447636e-05, 'epoch': 2.02}

 20%|██        | 8334/41250 [20:08:21<79:18:08,  8.67s/it][2025-04-26 04:06:04,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:06:04,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.34 | bwd_microstep: 5735.49 | bwd_inner_microstep: 5722.77 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.46
[2025-04-26 04:06:04,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.34 | bwd: 5735.51 | bwd_inner: 5722.77 | bwd_allreduce: 12.70 | step: 18.46

 20%|██        | 8335/41250 [20:08:29<79:18:48,  8.67s/it]
                                                          
{'loss': 0.3624, 'grad_norm': 2.280608892440796, 'learning_rate': 3.697446586639219e-05, 'epoch': 2.02}

 20%|██        | 8335/41250 [20:08:29<79:18:48,  8.67s/it][2025-04-26 04:06:13,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:06:13,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.53 | bwd_microstep: 5788.55 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 92.83 | step_microstep: 18.64
[2025-04-26 04:06:13,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.53 | bwd: 5788.56 | bwd_inner: 5695.67 | bwd_allreduce: 92.85 | step: 18.65

 20%|██        | 8336/41250 [20:08:38<79:26:35,  8.69s/it]
                                                          
{'loss': 0.1231, 'grad_norm': 2.110623598098755, 'learning_rate': 3.697363536769251e-05, 'epoch': 2.02}

 20%|██        | 8336/41250 [20:08:38<79:26:35,  8.69s/it][2025-04-26 04:06:21,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 04:06:21,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.97 | bwd_microstep: 5760.24 | bwd_inner_microstep: 5705.06 | bwd_allreduce_microstep: 55.14 | step_microstep: 19.05
[2025-04-26 04:06:21,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.97 | bwd: 5760.25 | bwd_inner: 5705.06 | bwd_allreduce: 55.15 | step: 19.05

 20%|██        | 8337/41250 [20:08:47<79:28:35,  8.69s/it]
                                                          
{'loss': 0.1009, 'grad_norm': 1.7899471521377563, 'learning_rate': 3.697280476435369e-05, 'epoch': 2.02}

 20%|██        | 8337/41250 [20:08:47<79:28:35,  8.69s/it][2025-04-26 04:06:30,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.09 | optimizer_step: 0.97
[2025-04-26 04:06:30,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5772.77 | bwd_inner_microstep: 5667.28 | bwd_allreduce_microstep: 105.44 | step_microstep: 19.09
[2025-04-26 04:06:30,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.99 | bwd: 5772.79 | bwd_inner: 5667.28 | bwd_allreduce: 105.47 | step: 19.09

 20%|██        | 8338/41250 [20:08:55<79:27:13,  8.69s/it]
                                                          
{'loss': 0.0854, 'grad_norm': 0.8198347687721252, 'learning_rate': 3.697197405638088e-05, 'epoch': 2.02}

 20%|██        | 8338/41250 [20:08:55<79:27:13,  8.69s/it][2025-04-26 04:06:39,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 04:06:39,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.14 | bwd_microstep: 5788.24 | bwd_inner_microstep: 5691.94 | bwd_allreduce_microstep: 96.25 | step_microstep: 18.98
[2025-04-26 04:06:39,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.14 | bwd: 5788.26 | bwd_inner: 5691.94 | bwd_allreduce: 96.27 | step: 18.98

 20%|██        | 8339/41250 [20:09:04<79:31:42,  8.70s/it]
                                                          
{'loss': 0.0542, 'grad_norm': 1.593015432357788, 'learning_rate': 3.6971143243779186e-05, 'epoch': 2.02}

 20%|██        | 8339/41250 [20:09:04<79:31:42,  8.70s/it][2025-04-26 04:06:47,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:06:47,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5785.16 | bwd_inner_microstep: 5697.53 | bwd_allreduce_microstep: 87.58 | step_microstep: 18.84
[2025-04-26 04:06:47,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5785.17 | bwd_inner: 5697.53 | bwd_allreduce: 87.60 | step: 18.84

 20%|██        | 8340/41250 [20:09:13<79:33:49,  8.70s/it]
                                                          
{'loss': 0.01, 'grad_norm': 0.16331017017364502, 'learning_rate': 3.697031232655373e-05, 'epoch': 2.02}

 20%|██        | 8340/41250 [20:09:13<79:33:49,  8.70s/it][2025-04-26 04:06:56,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:06:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.21 | bwd_microstep: 5718.72 | bwd_inner_microstep: 5705.96 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.60
[2025-04-26 04:06:56,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.21 | bwd: 5718.73 | bwd_inner: 5705.96 | bwd_allreduce: 12.73 | step: 18.60

 20%|██        | 8341/41250 [20:09:21<79:26:05,  8.69s/it]
                                                          
{'loss': 0.0516, 'grad_norm': 1.0531275272369385, 'learning_rate': 3.696948130470964e-05, 'epoch': 2.02}

 20%|██        | 8341/41250 [20:09:21<79:26:05,  8.69s/it][2025-04-26 04:07:05,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.17 | optimizer_step: 1.05
[2025-04-26 04:07:05,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5713.64 | bwd_inner_microstep: 5661.18 | bwd_allreduce_microstep: 52.41 | step_microstep: 19.61
[2025-04-26 04:07:05,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5713.65 | bwd_inner: 5661.18 | bwd_allreduce: 52.43 | step: 19.61

 20%|██        | 8342/41250 [20:09:30<79:15:36,  8.67s/it]
                                                          
{'loss': 0.0334, 'grad_norm': 0.6125720143318176, 'learning_rate': 3.696865017825204e-05, 'epoch': 2.02}

 20%|██        | 8342/41250 [20:09:30<79:15:36,  8.67s/it][2025-04-26 04:07:13,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:07:13,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.87 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5702.75 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.44
[2025-04-26 04:07:13,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.87 | bwd: 5715.48 | bwd_inner: 5702.75 | bwd_allreduce: 12.69 | step: 18.44

 20%|██        | 8343/41250 [20:09:39<79:11:45,  8.66s/it]
                                                          
{'loss': 0.0978, 'grad_norm': 1.9450950622558594, 'learning_rate': 3.696781894718604e-05, 'epoch': 2.02}

 20%|██        | 8343/41250 [20:09:39<79:11:45,  8.66s/it][2025-04-26 04:07:22,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:07:22,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5695.86 | bwd_inner_microstep: 5683.07 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.89
[2025-04-26 04:07:22,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5695.88 | bwd_inner: 5683.07 | bwd_allreduce: 12.77 | step: 18.89

 20%|██        | 8344/41250 [20:09:47<79:05:19,  8.65s/it]
                                                          
{'loss': 0.0381, 'grad_norm': 0.7217829823493958, 'learning_rate': 3.6966987611516776e-05, 'epoch': 2.02}

 20%|██        | 8344/41250 [20:09:47<79:05:19,  8.65s/it][2025-04-26 04:07:31,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 04:07:31,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.34 | bwd_microstep: 5789.89 | bwd_inner_microstep: 5711.27 | bwd_allreduce_microstep: 78.57 | step_microstep: 18.50
[2025-04-26 04:07:31,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.34 | bwd: 5789.90 | bwd_inner: 5711.27 | bwd_allreduce: 78.59 | step: 18.50

 20%|██        | 8345/41250 [20:09:56<79:17:33,  8.68s/it]
                                                          
{'loss': 0.0689, 'grad_norm': 1.4117976427078247, 'learning_rate': 3.6966156171249365e-05, 'epoch': 2.02}

 20%|██        | 8345/41250 [20:09:56<79:17:33,  8.68s/it][2025-04-26 04:07:39,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:07:39,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.95 | bwd_microstep: 5762.62 | bwd_inner_microstep: 5692.81 | bwd_allreduce_microstep: 69.77 | step_microstep: 19.06
[2025-04-26 04:07:39,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.95 | bwd: 5762.63 | bwd_inner: 5692.81 | bwd_allreduce: 69.78 | step: 19.06

 20%|██        | 8346/41250 [20:10:05<79:21:01,  8.68s/it]
                                                          
{'loss': 0.0305, 'grad_norm': 0.42253735661506653, 'learning_rate': 3.696532462638895e-05, 'epoch': 2.02}

 20%|██        | 8346/41250 [20:10:05<79:21:01,  8.68s/it][2025-04-26 04:07:48,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 04:07:48,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5798.91 | bwd_inner_microstep: 5642.84 | bwd_allreduce_microstep: 156.02 | step_microstep: 19.31
[2025-04-26 04:07:48,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5798.92 | bwd_inner: 5642.84 | bwd_allreduce: 156.04 | step: 19.31

 20%|██        | 8347/41250 [20:10:14<79:26:09,  8.69s/it]
                                                          
{'loss': 0.3426, 'grad_norm': 2.860227108001709, 'learning_rate': 3.6964492976940635e-05, 'epoch': 2.02}

 20%|██        | 8347/41250 [20:10:14<79:26:09,  8.69s/it][2025-04-26 04:07:57,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:07:57,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5754.35 | bwd_inner_microstep: 5689.27 | bwd_allreduce_microstep: 65.04 | step_microstep: 18.80
[2025-04-26 04:07:57,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5754.36 | bwd_inner: 5689.26 | bwd_allreduce: 65.06 | step: 18.81

 20%|██        | 8348/41250 [20:10:22<79:26:35,  8.69s/it]
                                                          
{'loss': 0.1511, 'grad_norm': 1.5930591821670532, 'learning_rate': 3.696366122290957e-05, 'epoch': 2.02}

 20%|██        | 8348/41250 [20:10:22<79:26:35,  8.69s/it]W0426 04:08:06.364300 784474 site-packages/torch/distributed/run.py:793] 
W0426 04:08:06.364300 784474 site-packages/torch/distributed/run.py:793] *****************************************
W0426 04:08:06.364300 784474 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0426 04:08:06.364300 784474 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-26 04:08:08,311] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:08,312] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:08,324] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[2025-04-26 04:08:11,586] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-26 04:08:11,586] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Replace train sampler!!Replace train sampler!!

petrel_client is not installed. Using PIL to load images.petrel_client is not installed. Using PIL to load images.

04/26/2025 04:08:11 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
04/26/2025 04:08:11 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr26_04-08-11_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
04/26/2025 04:08:11 - INFO - __main__ - Loading Tokenizer: /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2032] 2025-04-26 04:08:11,770 >> loading file chat_template.jinja
[2025-04-26 04:08:11,823] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-26 04:08:11,824] [INFO] [comm.py:652:init_distributed] cdb=None
04/26/2025 04:08:11 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
04/26/2025 04:08:11 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
[INFO|tokenization_utils_base.py:2304] 2025-04-26 04:08:12,243 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
04/26/2025 04:08:12 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:694] 2025-04-26 04:08:12,384 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/config.json
[INFO|configuration_utils.py:768] 2025-04-26 04:08:12,386 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-9B-MPO-try0-2",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 4096,
  "image_fold": null,
  "llm_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "/mnt/petrelfs/share_data/wangweiyun/share_ckpt/hf_home/internlm2-chat-7b",
    "add_cross_attention": false,
    "architectures": [
      "InternLM2ForCausalLM"
    ],
    "attn_implementation": "flash_attention_2",
    "auto_map": {
      "AutoConfig": "configuration_internlm2.InternLM2Config",
      "AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
      "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bias": false,
    "bos_token_id": 1,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 2,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 10240,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "min_length": 0,
    "model_type": "internlm2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 32,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 48,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 2,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "factor": 2.0,
      "type": "dynamic"
    },
    "rope_theta": 50000000,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": false,
    "vocab_size": 128142
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "system_message": null,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_img_start_end_token": true,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_attn_implementation_autoset": true,
    "_name_or_path": "pretrained/intern_vit_6b_448px_v1_2/",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.1,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.48.3",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

04/26/2025 04:08:12 - INFO - __main__ - Using flash_attention_2 for InternLM
[INFO|modeling_utils.py:3901] 2025-04-26 04:08:12,387 >> loading weights file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/model.safetensors.index.json
[INFO|modeling_utils.py:1582] 2025-04-26 04:08:12,388 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1140] 2025-04-26 04:08:12,390 >> Generate config GenerationConfig {}

this model
[WARNING|logging.py:328] 2025-04-26 04:08:12,457 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[INFO|configuration_utils.py:1140] 2025-04-26 04:08:12,457 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 2,
  "use_cache": false
}

this model
this model
[WARNING|logging.py:328] 2025-04-26 04:08:12,588 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[WARNING|logging.py:328] 2025-04-26 04:08:12,588 >> InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight1 Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
Parameter containing:
tensor([[-3.9497e-35,  9.9609e-01, -2.1606e-02,  ...,  9.8438e-01,
         -4.3195e-16,  8.5547e-01],
        [-8.8500e+08,  9.9219e-01,  4.3293e+12,  ...,  9.9219e-01,
         -6.0423e-25,  8.7891e-01],
        [-2.0900e+02,  9.8828e-01, -1.3916e-02,  ...,  9.9609e-01,
          9.5695e-34,  9.0234e-01],
        ...,
        [ 6.8247e-06,  3.7575e-04,  8.9720e+14,  ..., -8.5547e-01,
          5.7894e+35, -8.9844e-01],
        [-1.3990e-30, -3.5156e-02,  2.8187e+22,  ..., -8.8672e-01,
         -2.0146e+36, -8.7500e-01],
        [ 4.3335e-03, -7.0801e-02,  1.1744e+10,  ..., -9.1406e-01,
         -1.3209e+37, -8.5156e-01]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 motion_mlp.weight2 Parameter containing:
tensor([[0.0038, 0.0076, 0.0004,  ..., 0.0018, 0.0058, 0.0042],
        [0.0016, 0.0024, 0.0002,  ..., 0.0030, 0.0067, 0.0039],
        [0.0049, 0.0073, 0.0011,  ..., 0.0092, 0.0042, 0.0098],
        ...,
        [0.0022, 0.0044, 0.0077,  ..., 0.0032, 0.0088, 0.0034],
        [0.0026, 0.0081, 0.0054,  ..., 0.0020, 0.0035, 0.0063],
        [0.0018, 0.0065, 0.0002,  ..., 0.0020, 0.0045, 0.0099]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
motion_mlp.weight1 Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
motion_mlp.weight2 motion_mlp.weight2 Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
Parameter containing:
tensor([[0.0052, 0.0048, 0.0044,  ..., 0.0095, 0.0072, 0.0018],
        [0.0027, 0.0013, 0.0090,  ..., 0.0052, 0.0060, 0.0030],
        [0.0056, 0.0089, 0.0038,  ..., 0.0074, 0.0054, 0.0017],
        ...,
        [0.0082, 0.0093, 0.0095,  ..., 0.0005, 0.0048, 0.0056],
        [0.0064, 0.0098, 0.0072,  ..., 0.0095, 0.0014, 0.0092],
        [0.0061, 0.0058, 0.0096,  ..., 0.0016, 0.0072, 0.0038]],
       requires_grad=True)
motion_mlp.bias Parameter containing:
tensor([0., 0., 0.,  ..., 0., 0., 0.], requires_grad=True)
None False
None False
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone
Setting backbone: fragments_backbone

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.82it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.06it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.68it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.12it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.98it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.13it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.13it/s]
[INFO|modeling_utils.py:4888] 2025-04-26 04:08:18,598 >> All model checkpoint weights were used when initializing InternVLChatModel.

[WARNING|modeling_utils.py:4890] 2025-04-26 04:08:18,598 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:1093] 2025-04-26 04:08:18,609 >> loading configuration file /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B/generation_config.json
[INFO|configuration_utils.py:1140] 2025-04-26 04:08:18,609 >> Generate config GenerationConfig {}

04/26/2025 04:08:18 - INFO - __main__ - Finished
04/26/2025 04:08:18 - INFO - __main__ - model.config.force_image_size: 448
04/26/2025 04:08:18 - INFO - __main__ - data_args.force_image_size: 448
04/26/2025 04:08:18 - INFO - __main__ - model.config.vision_config.image_size: 448
04/26/2025 04:08:18 - INFO - __main__ - [Dataset] num_image_token: 256
04/26/2025 04:08:18 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/26/2025 04:08:18 - INFO - __main__ - [Dataset] use_thumbnail: True
04/26/2025 04:08:18 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/26/2025 04:08:18 - INFO - __main__ - Formatting inputs...Skip in lazy mode

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.72it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.43it/s]
[WARNING|modeling_utils.py:4890] 2025-04-26 04:08:18,634 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.66it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.34it/s]
[WARNING|modeling_utils.py:4890] 2025-04-26 04:08:18,648 >> Some weights of InternVLChatModel were not initialized from the model checkpoint at /home/wangjiarui/InternVL/internvl_chat/InternVL3-9B and are newly initialized: ['evaluator.fragments_backbone.layers.0.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.0.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.0.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.0.downsample.norm.bias', 'evaluator.fragments_backbone.layers.0.downsample.norm.weight', 'evaluator.fragments_backbone.layers.0.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.1.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.1.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.1.downsample.norm.bias', 'evaluator.fragments_backbone.layers.1.downsample.norm.weight', 'evaluator.fragments_backbone.layers.1.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.1.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.2.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.2.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.3.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.3.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.4.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.4.norm2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.fragment_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.proj.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.qkv.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.2.blocks.5.attn.relative_position_index', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm1.weight', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.bias', 'evaluator.fragments_backbone.layers.2.blocks.5.norm2.weight', 'evaluator.fragments_backbone.layers.2.downsample.norm.bias', 'evaluator.fragments_backbone.layers.2.downsample.norm.weight', 'evaluator.fragments_backbone.layers.2.downsample.reduction.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.0.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.0.norm2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.proj.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.qkv.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_bias_table', 'evaluator.fragments_backbone.layers.3.blocks.1.attn.relative_position_index', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.mlp.fc2.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm1.weight', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.bias', 'evaluator.fragments_backbone.layers.3.blocks.1.norm2.weight', 'evaluator.fragments_backbone.norm.bias', 'evaluator.fragments_backbone.norm.weight', 'evaluator.fragments_backbone.patch_embed.norm.bias', 'evaluator.fragments_backbone.patch_embed.norm.weight', 'evaluator.fragments_backbone.patch_embed.proj.bias', 'evaluator.fragments_backbone.patch_embed.proj.weight', 'fast_mlp.0.bias', 'fast_mlp.0.weight', 'fast_mlp.1.bias', 'fast_mlp.1.weight', 'fast_mlp.3.bias', 'fast_mlp.3.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
04/26/2025 04:08:19 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 49500
04/26/2025 04:08:19 - INFO - __main__ - [Dataset] num_image_token: 256
04/26/2025 04:08:19 - INFO - __main__ - [Dataset] dynamic_image_size: True
04/26/2025 04:08:19 - INFO - __main__ - [Dataset] use_thumbnail: True
04/26/2025 04:08:19 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6
04/26/2025 04:08:19 - INFO - __main__ - Formatting inputs...Skip in lazy mode
04/26/2025 04:08:19 - INFO - __main__ - Add dataset: sharegpt4v_instruct_gpt4-vision_cap100k with length: 9000
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7f04375aee80>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7efd54152c70>
eval_dataset <torch.utils.data.dataset.ConcatDataset object at 0x7fc2904d3c70>
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 6,291,456 || all params: 310,303,744 || trainable%: 2.0275
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=2,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr26_04-08-11_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr26_04-08-11_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
trainable params: 46,006,272 || all params: 8,847,216,640 || trainable%: 0.5200
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.0.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.1.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.2.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.3.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.4.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.5.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.6.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.7.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.8.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.9.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.10.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.11.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.12.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.13.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.14.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.15.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.16.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.17.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.18.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.19.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.20.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.21.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.22.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.qkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.attn.proj.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - vision_model.base_model.model.encoder.layers.23.mlp.fc2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.0.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.1.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.2.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.3.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.24.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.25.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.26.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.27.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.28.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.29.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.30.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.31.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.32.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.33.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.34.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.35.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.36.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.37.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.38.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.39.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.40.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.41.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.42.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.43.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.44.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.45.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.46.feed_forward.w2.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wo.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w1.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w3.lora_B.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_A.default.weight
04/26/2025 04:08:20 - INFO - __main__ - language_model.base_model.model.model.layers.47.feed_forward.w2.lora_B.default.weight
training_args TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=4125,
eval_strategy=steps,
eval_use_gather_object=False,
evaluation_strategy=steps,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=4e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/runs/Apr26_04-08-11_amax,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=10.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=/home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=4000000,
save_strategy=steps,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
[INFO|trainer.py:741] 2025-04-26 04:08:20,321 >> Using auto half precision backend
[WARNING|trainer.py:803] 2025-04-26 04:08:20,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 04:08:20,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 04:08:20,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 04:08:20,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 04:08:20,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 04:08:20,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[2025-04-26 04:08:20,557] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.3, git-hash=unknown, git-branch=unknown
[2025-04-26 04:08:20,557] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 3
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
[2025-04-26 04:08:26,890] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/wangjiarui/.cache/torch_extensions/py39_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/wangjiarui/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.6663095951080322 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.7029051780700684 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.7036454677581787 seconds
[2025-04-26 04:08:27,950] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2025-04-26 04:08:27,950] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-04-26 04:08:28,037] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-04-26 04:08:28,037] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-04-26 04:08:28,037] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2025-04-26 04:08:28,037] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2025-04-26 04:08:28,037] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2025-04-26 04:08:28,037] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2025-04-26 04:08:28,038] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2025-04-26 04:08:28,448] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-04-26 04:08:28,449] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.08 GB         CA 18.33 GB         Max_CA 18 GB 
[2025-04-26 04:08:28,449] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 25.93 GB, percent = 10.3%
[2025-04-26 04:08:28,622] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-04-26 04:08:28,623] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.11 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-26 04:08:28,623] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 25.94 GB, percent = 10.3%
[2025-04-26 04:08:28,623] [INFO] [stage_1_and_2.py:545:__init__] optimizer state initialized
[2025-04-26 04:08:28,796] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-04-26 04:08:28,797] [INFO] [utils.py:782:see_memory_usage] MA 18.05 GB         Max_MA 18.05 GB         CA 18.39 GB         Max_CA 18 GB 
[2025-04-26 04:08:28,797] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 26.03 GB, percent = 10.3%
[2025-04-26 04:08:28,803] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2025-04-26 04:08:28,803] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2025-04-26 04:08:28,804] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f0417bd0190>
[2025-04-26 04:08:28,804] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-04-26 04:08:28,814] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-26 04:08:28,814] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f0417bc40a0>
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-26 04:08:28,815] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-26 04:08:28,816] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 4e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.01}
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   train_batch_size ............. 12
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  4
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   world_size ................... 3
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-26 04:08:28,817] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-26 04:08:28,818] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-26 04:08:28,818] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2025-04-26 04:08:28,818] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 4e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.01
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 12, 
    "train_micro_batch_size_per_gpu": 4, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:2369] 2025-04-26 04:08:28,819 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-04-26 04:08:28,819 >>   Num examples = 49,500
[INFO|trainer.py:2371] 2025-04-26 04:08:28,819 >>   Num Epochs = 10
[INFO|trainer.py:2372] 2025-04-26 04:08:28,819 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-04-26 04:08:28,819 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:2376] 2025-04-26 04:08:28,819 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2377] 2025-04-26 04:08:28,819 >>   Total optimization steps = 41,250
[INFO|trainer.py:2378] 2025-04-26 04:08:28,826 >>   Number of trainable parameters = 52,297,728

  0%|          | 0/41250 [00:00<?, ?it/s][2025-04-26 04:08:30,393] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:30,436] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:31,000] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 04:08:35,940] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:35,997] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:36,656] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 04:08:41,469] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:41,518] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:42,580] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[2025-04-26 04:08:46,930] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:46,969] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-26 04:08:48,469] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
8 7 7
8 7 7
8 7 7
[2025-04-26 04:09:11,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.83 | optimizer_gradients: 21.84 | optimizer_step: 4.52
[2025-04-26 04:09:11,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 4955.85 | bwd_microstep: 6199.11 | bwd_inner_microstep: 6186.29 | bwd_allreduce_microstep: 12.77 | step_microstep: 85.61
[2025-04-26 04:09:11,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 4955.83 | bwd: 6199.12 | bwd_inner: 6186.30 | bwd_allreduce: 12.79 | step: 85.61

  0%|          | 1/41250 [00:42<489:30:25, 42.72s/it]
                                                     
{'loss': 1.3896, 'grad_norm': 15.712672233581543, 'learning_rate': 3.231017770597739e-08, 'epoch': 0.0}

  0%|          | 1/41250 [00:42<489:30:25, 42.72s/it][2025-04-26 04:09:20,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 04:09:20,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2795.40 | bwd_microstep: 5971.37 | bwd_inner_microstep: 5603.18 | bwd_allreduce_microstep: 368.14 | step_microstep: 19.11
[2025-04-26 04:09:20,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2795.40 | bwd: 5971.38 | bwd_inner: 5603.18 | bwd_allreduce: 368.16 | step: 19.11

  0%|          | 2/41250 [00:51<261:56:20, 22.86s/it]
                                                     
{'loss': 0.9697, 'grad_norm': 11.642318725585938, 'learning_rate': 6.462035541195477e-08, 'epoch': 0.0}

  0%|          | 2/41250 [00:51<261:56:20, 22.86s/it][2025-04-26 04:09:29,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:09:29,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2813.59 | bwd_microstep: 5678.52 | bwd_inner_microstep: 5618.07 | bwd_allreduce_microstep: 60.41 | step_microstep: 18.10
[2025-04-26 04:09:29,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2813.59 | bwd: 5678.53 | bwd_inner: 5618.07 | bwd_allreduce: 60.43 | step: 18.11

  0%|          | 3/41250 [01:00<187:13:57, 16.34s/it]
                                                     
{'loss': 1.1796, 'grad_norm': 17.145618438720703, 'learning_rate': 9.693053311793216e-08, 'epoch': 0.0}

  0%|          | 3/41250 [01:00<187:13:57, 16.34s/it][2025-04-26 04:09:37,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 04:09:37,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2810.38 | bwd_microstep: 5700.09 | bwd_inner_microstep: 5614.61 | bwd_allreduce_microstep: 85.43 | step_microstep: 19.01
[2025-04-26 04:09:37,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2810.38 | bwd: 5700.10 | bwd_inner: 5614.61 | bwd_allreduce: 85.45 | step: 19.01

  0%|          | 4/41250 [01:08<152:12:31, 13.28s/it]
                                                     
{'loss': 0.9791, 'grad_norm': 15.285684585571289, 'learning_rate': 1.2924071082390955e-07, 'epoch': 0.0}

  0%|          | 4/41250 [01:08<152:12:31, 13.28s/it][2025-04-26 04:09:46,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:09:46,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.77 | bwd_microstep: 5634.73 | bwd_inner_microstep: 5610.19 | bwd_allreduce_microstep: 24.50 | step_microstep: 18.50
[2025-04-26 04:09:46,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.77 | bwd: 5634.75 | bwd_inner: 5610.19 | bwd_allreduce: 24.51 | step: 18.50

  0%|          | 5/41250 [01:17<132:34:52, 11.57s/it]
                                                     
{'loss': 1.2348, 'grad_norm': 15.308516502380371, 'learning_rate': 1.6155088852988693e-07, 'epoch': 0.0}

  0%|          | 5/41250 [01:17<132:34:52, 11.57s/it][2025-04-26 04:09:55,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:09:55,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3373.33 | bwd_microstep: 5730.83 | bwd_inner_microstep: 5718.01 | bwd_allreduce_microstep: 12.78 | step_microstep: 17.86
[2025-04-26 04:09:55,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3373.33 | bwd: 5730.84 | bwd_inner: 5718.01 | bwd_allreduce: 12.79 | step: 17.86

  0%|          | 6/41250 [01:26<123:17:16, 10.76s/it]
                                                     
{'loss': 1.0272, 'grad_norm': 12.54334545135498, 'learning_rate': 1.938610662358643e-07, 'epoch': 0.0}

  0%|          | 6/41250 [01:26<123:17:16, 10.76s/it][2025-04-26 04:10:04,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 04:10:04,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.82 | bwd_microstep: 5678.67 | bwd_inner_microstep: 5665.89 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.98
[2025-04-26 04:10:04,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.82 | bwd: 5678.68 | bwd_inner: 5665.89 | bwd_allreduce: 12.75 | step: 18.99

  0%|          | 7/41250 [01:35<115:12:43, 10.06s/it]
                                                     
{'loss': 1.1244, 'grad_norm': 14.790037155151367, 'learning_rate': 2.261712439418417e-07, 'epoch': 0.0}

  0%|          | 7/41250 [01:35<115:12:43, 10.06s/it][2025-04-26 04:10:12,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.10 | optimizer_step: 1.25
[2025-04-26 04:10:12,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2812.80 | bwd_microstep: 5730.15 | bwd_inner_microstep: 5612.32 | bwd_allreduce_microstep: 117.78 | step_microstep: 19.88
[2025-04-26 04:10:12,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2812.80 | bwd: 5730.16 | bwd_inner: 5612.32 | bwd_allreduce: 117.81 | step: 19.88

  0%|          | 8/41250 [01:43<110:00:48,  9.60s/it]
                                                     
{'loss': 1.1281, 'grad_norm': 11.34134292602539, 'learning_rate': 2.584814216478191e-07, 'epoch': 0.0}

  0%|          | 8/41250 [01:43<110:00:48,  9.60s/it][2025-04-26 04:10:21,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:10:21,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.44 | bwd_microstep: 5683.95 | bwd_inner_microstep: 5627.92 | bwd_allreduce_microstep: 55.98 | step_microstep: 19.03
[2025-04-26 04:10:21,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.44 | bwd: 5683.96 | bwd_inner: 5627.92 | bwd_allreduce: 56.00 | step: 19.03

  0%|          | 9/41250 [01:52<106:21:47,  9.28s/it]
                                                     
{'loss': 0.8485, 'grad_norm': 12.623395919799805, 'learning_rate': 2.9079159935379645e-07, 'epoch': 0.0}

  0%|          | 9/41250 [01:52<106:21:47,  9.28s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x2a95b640] moov atom not found
[04:10:21] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4...
[2025-04-26 04:10:29,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 1.17
[2025-04-26 04:10:29,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.08 | bwd_microstep: 5678.78 | bwd_inner_microstep: 5665.02 | bwd_allreduce_microstep: 13.69 | step_microstep: 19.68
[2025-04-26 04:10:29,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.08 | bwd: 5678.79 | bwd_inner: 5665.02 | bwd_allreduce: 13.73 | step: 19.69

  0%|          | 10/41250 [02:01<103:56:09,  9.07s/it]
                                                      
{'loss': 1.1783, 'grad_norm': 15.631743431091309, 'learning_rate': 3.2310177705977386e-07, 'epoch': 0.0}

  0%|          | 10/41250 [02:01<103:56:09,  9.07s/it][2025-04-26 04:10:38,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 04:10:38,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.44 | bwd_microstep: 6077.53 | bwd_inner_microstep: 5686.62 | bwd_allreduce_microstep: 390.87 | step_microstep: 18.89
[2025-04-26 04:10:38,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.44 | bwd: 6077.55 | bwd_inner: 5686.62 | bwd_allreduce: 390.89 | step: 18.90

  0%|          | 11/41250 [02:09<103:38:45,  9.05s/it]
                                                      
{'loss': 1.1247, 'grad_norm': 16.026647567749023, 'learning_rate': 3.5541195476575127e-07, 'epoch': 0.0}

  0%|          | 11/41250 [02:09<103:38:45,  9.05s/it][2025-04-26 04:10:47,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 04:10:47,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.09 | bwd_microstep: 5692.76 | bwd_inner_microstep: 5642.35 | bwd_allreduce_microstep: 50.36 | step_microstep: 18.90
[2025-04-26 04:10:47,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.09 | bwd: 5692.77 | bwd_inner: 5642.35 | bwd_allreduce: 50.38 | step: 18.91

  0%|          | 12/41250 [02:18<102:04:03,  8.91s/it]
                                                      
{'loss': 0.8429, 'grad_norm': 16.592529296875, 'learning_rate': 3.877221324717286e-07, 'epoch': 0.0}

  0%|          | 12/41250 [02:18<102:04:03,  8.91s/it][2025-04-26 04:10:56,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 04:10:56,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.42 | bwd_microstep: 5704.23 | bwd_inner_microstep: 5691.51 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.25
[2025-04-26 04:10:56,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.42 | bwd: 5704.24 | bwd_inner: 5691.51 | bwd_allreduce: 12.69 | step: 19.25

  0%|          | 13/41250 [02:27<101:08:38,  8.83s/it]
                                                      
{'loss': 1.1292, 'grad_norm': 15.492544174194336, 'learning_rate': 4.2003231017770603e-07, 'epoch': 0.0}

  0%|          | 13/41250 [02:27<101:08:38,  8.83s/it][2025-04-26 04:11:04,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 04:11:04,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.16 | bwd_microstep: 5707.87 | bwd_inner_microstep: 5648.42 | bwd_allreduce_microstep: 59.40 | step_microstep: 19.37
[2025-04-26 04:11:04,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.16 | bwd: 5707.88 | bwd_inner: 5648.42 | bwd_allreduce: 59.42 | step: 19.38

  0%|          | 14/41250 [02:35<100:24:11,  8.77s/it]
                                                      
{'loss': 1.1287, 'grad_norm': 14.756266593933105, 'learning_rate': 4.523424878836834e-07, 'epoch': 0.0}

  0%|          | 14/41250 [02:35<100:24:11,  8.77s/it][2025-04-26 04:11:13,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 04:11:13,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.85 | bwd_microstep: 5766.01 | bwd_inner_microstep: 5687.45 | bwd_allreduce_microstep: 78.52 | step_microstep: 19.21
[2025-04-26 04:11:13,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.85 | bwd: 5766.02 | bwd_inner: 5687.45 | bwd_allreduce: 78.53 | step: 19.21

  0%|          | 15/41250 [02:44<100:10:13,  8.75s/it]
                                                      
{'loss': 0.8969, 'grad_norm': 12.205196380615234, 'learning_rate': 4.846526655896607e-07, 'epoch': 0.0}

  0%|          | 15/41250 [02:44<100:10:13,  8.75s/it][2025-04-26 04:11:22,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-26 04:11:22,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.63 | bwd_microstep: 5715.60 | bwd_inner_microstep: 5702.81 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.72
[2025-04-26 04:11:22,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.63 | bwd: 5715.61 | bwd_inner: 5702.81 | bwd_allreduce: 12.75 | step: 18.72

  0%|          | 16/41250 [02:53<99:47:55,  8.71s/it] 
                                                     
{'loss': 0.8467, 'grad_norm': 8.428592681884766, 'learning_rate': 5.169628432956382e-07, 'epoch': 0.0}

  0%|          | 16/41250 [02:53<99:47:55,  8.71s/it][2025-04-26 04:11:30,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-26 04:11:30,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.81 | bwd_microstep: 5726.88 | bwd_inner_microstep: 5652.91 | bwd_allreduce_microstep: 73.92 | step_microstep: 18.55
[2025-04-26 04:11:30,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.81 | bwd: 5726.89 | bwd_inner: 5652.91 | bwd_allreduce: 73.94 | step: 18.55

  0%|          | 17/41250 [03:01<99:31:31,  8.69s/it]
                                                     
{'loss': 0.9563, 'grad_norm': 13.7479829788208, 'learning_rate': 5.492730210016156e-07, 'epoch': 0.0}

  0%|          | 17/41250 [03:01<99:31:31,  8.69s/it][2025-04-26 04:11:39,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 04:11:39,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.29 | bwd_microstep: 5711.19 | bwd_inner_microstep: 5689.92 | bwd_allreduce_microstep: 21.23 | step_microstep: 19.53
[2025-04-26 04:11:39,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.29 | bwd: 5711.21 | bwd_inner: 5689.92 | bwd_allreduce: 21.25 | step: 19.53

  0%|          | 18/41250 [03:10<99:22:00,  8.68s/it]
                                                     
{'loss': 1.1301, 'grad_norm': 14.024998664855957, 'learning_rate': 5.815831987075929e-07, 'epoch': 0.0}

  0%|          | 18/41250 [03:10<99:22:00,  8.68s/it][2025-04-26 04:11:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.24
[2025-04-26 04:11:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.35 | bwd_microstep: 5714.19 | bwd_inner_microstep: 5652.57 | bwd_allreduce_microstep: 61.57 | step_microstep: 19.50
[2025-04-26 04:11:47,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.35 | bwd: 5714.20 | bwd_inner: 5652.57 | bwd_allreduce: 61.59 | step: 19.51

  0%|          | 19/41250 [03:19<99:13:32,  8.66s/it]
                                                     
{'loss': 0.9556, 'grad_norm': 12.504109382629395, 'learning_rate': 6.138933764135703e-07, 'epoch': 0.0}

  0%|          | 19/41250 [03:19<99:13:32,  8.66s/it][2025-04-26 04:11:56,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 04:11:56,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.94 | bwd_microstep: 5724.96 | bwd_inner_microstep: 5647.20 | bwd_allreduce_microstep: 77.71 | step_microstep: 19.08
[2025-04-26 04:11:56,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.94 | bwd: 5724.97 | bwd_inner: 5647.20 | bwd_allreduce: 77.73 | step: 19.09

  0%|          | 20/41250 [03:27<99:06:46,  8.65s/it]
                                                     
{'loss': 0.7227, 'grad_norm': 12.977248191833496, 'learning_rate': 6.462035541195477e-07, 'epoch': 0.0}

  0%|          | 20/41250 [03:27<99:06:46,  8.65s/it][2025-04-26 04:12:05,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 04:12:05,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.53 | bwd_microstep: 5795.88 | bwd_inner_microstep: 5649.24 | bwd_allreduce_microstep: 146.60 | step_microstep: 18.62
[2025-04-26 04:12:05,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.53 | bwd: 5795.90 | bwd_inner: 5649.23 | bwd_allreduce: 146.62 | step: 18.63

  0%|          | 21/41250 [03:36<99:17:26,  8.67s/it]
                                                     
{'loss': 0.9758, 'grad_norm': 12.707098960876465, 'learning_rate': 6.785137318255252e-07, 'epoch': 0.01}

  0%|          | 21/41250 [03:36<99:17:26,  8.67s/it][2025-04-26 04:12:14,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.04
[2025-04-26 04:12:14,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.91 | bwd_microstep: 6120.76 | bwd_inner_microstep: 5657.13 | bwd_allreduce_microstep: 463.59 | step_microstep: 19.08
[2025-04-26 04:12:14,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.91 | bwd: 6120.78 | bwd_inner: 5657.13 | bwd_allreduce: 463.61 | step: 19.08

  0%|          | 22/41250 [03:45<100:32:37,  8.78s/it]
                                                      
{'loss': 1.3535, 'grad_norm': 14.938983917236328, 'learning_rate': 7.108239095315025e-07, 'epoch': 0.01}

  0%|          | 22/41250 [03:45<100:32:37,  8.78s/it][2025-04-26 04:12:22,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:12:22,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.97 | bwd_microstep: 5711.77 | bwd_inner_microstep: 5698.74 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.51
[2025-04-26 04:12:22,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.97 | bwd: 5711.78 | bwd_inner: 5698.74 | bwd_allreduce: 13.00 | step: 18.52

  0%|          | 23/41250 [03:54<100:05:23,  8.74s/it]
                                                      
{'loss': 0.909, 'grad_norm': 12.075336456298828, 'learning_rate': 7.431340872374798e-07, 'epoch': 0.01}

  0%|          | 23/41250 [03:54<100:05:23,  8.74s/it][2025-04-26 04:12:31,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 04:12:31,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5713.72 | bwd_inner_microstep: 5700.87 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.25
[2025-04-26 04:12:31,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5713.73 | bwd_inner: 5700.87 | bwd_allreduce: 12.82 | step: 19.26

  0%|          | 24/41250 [04:02<99:47:20,  8.71s/it] 
                                                     
{'loss': 1.2197, 'grad_norm': 10.735916137695312, 'learning_rate': 7.754442649434572e-07, 'epoch': 0.01}

  0%|          | 24/41250 [04:02<99:47:20,  8.71s/it][2025-04-26 04:12:40,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:12:40,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.77 | bwd_microstep: 5719.19 | bwd_inner_microstep: 5656.66 | bwd_allreduce_microstep: 62.49 | step_microstep: 18.22
[2025-04-26 04:12:40,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.77 | bwd: 5719.20 | bwd_inner: 5656.66 | bwd_allreduce: 62.50 | step: 18.22

  0%|          | 25/41250 [04:11<99:28:57,  8.69s/it]
                                                     
{'loss': 1.0248, 'grad_norm': 17.600980758666992, 'learning_rate': 8.077544426494346e-07, 'epoch': 0.01}

  0%|          | 25/41250 [04:11<99:28:57,  8.69s/it][2025-04-26 04:12:49,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:12:49,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 6099.01 | bwd_inner_microstep: 5700.75 | bwd_allreduce_microstep: 398.22 | step_microstep: 18.54
[2025-04-26 04:12:49,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 6099.02 | bwd_inner: 5700.75 | bwd_allreduce: 398.24 | step: 18.54

  0%|          | 26/41250 [04:20<100:41:07,  8.79s/it]
                                                      
{'loss': 1.0504, 'grad_norm': 13.048023223876953, 'learning_rate': 8.400646203554121e-07, 'epoch': 0.01}

  0%|          | 26/41250 [04:20<100:41:07,  8.79s/it][2025-04-26 04:12:57,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-26 04:12:57,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.66 | bwd_microstep: 5770.10 | bwd_inner_microstep: 5685.75 | bwd_allreduce_microstep: 84.31 | step_microstep: 19.05
[2025-04-26 04:12:57,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.66 | bwd: 5770.12 | bwd_inner: 5685.74 | bwd_allreduce: 84.33 | step: 19.05

  0%|          | 27/41250 [04:29<100:22:09,  8.77s/it]
                                                      
{'loss': 1.0851, 'grad_norm': 12.163162231445312, 'learning_rate': 8.723747980613895e-07, 'epoch': 0.01}

  0%|          | 27/41250 [04:29<100:22:09,  8.77s/it][2025-04-26 04:13:06,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:13:06,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5703.68 | bwd_inner_microstep: 5690.75 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.46
[2025-04-26 04:13:06,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5703.69 | bwd_inner: 5690.75 | bwd_allreduce: 12.90 | step: 18.46

  0%|          | 28/41250 [04:37<99:54:14,  8.72s/it] 
                                                     
{'loss': 1.054, 'grad_norm': 16.290456771850586, 'learning_rate': 9.046849757673668e-07, 'epoch': 0.01}

  0%|          | 28/41250 [04:37<99:54:14,  8.72s/it][2025-04-26 04:13:15,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 04:13:15,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5766.90 | bwd_inner_microstep: 5699.89 | bwd_allreduce_microstep: 66.95 | step_microstep: 19.36
[2025-04-26 04:13:15,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5766.91 | bwd_inner: 5699.89 | bwd_allreduce: 66.98 | step: 19.36

  0%|          | 29/41250 [04:46<99:47:09,  8.71s/it]
                                                     
{'loss': 0.6734, 'grad_norm': 9.28180980682373, 'learning_rate': 9.369951534733441e-07, 'epoch': 0.01}

  0%|          | 29/41250 [04:46<99:47:09,  8.71s/it][2025-04-26 04:13:23,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:13:23,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.11 | bwd_microstep: 5701.66 | bwd_inner_microstep: 5676.44 | bwd_allreduce_microstep: 25.18 | step_microstep: 18.54
[2025-04-26 04:13:23,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.11 | bwd: 5701.67 | bwd_inner: 5676.44 | bwd_allreduce: 25.19 | step: 18.54

  0%|          | 30/41250 [04:55<99:29:55,  8.69s/it]
                                                     
{'loss': 1.2078, 'grad_norm': 15.615530014038086, 'learning_rate': 9.693053311793215e-07, 'epoch': 0.01}

  0%|          | 30/41250 [04:55<99:29:55,  8.69s/it][2025-04-26 04:13:32,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:13:32,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.73 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5685.04 | bwd_allreduce_microstep: 21.69 | step_microstep: 18.41
[2025-04-26 04:13:32,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.73 | bwd: 5706.79 | bwd_inner: 5685.04 | bwd_allreduce: 21.71 | step: 18.42

  0%|          | 31/41250 [05:03<99:18:54,  8.67s/it]
                                                     
{'loss': 0.9832, 'grad_norm': 14.304893493652344, 'learning_rate': 1.001615508885299e-06, 'epoch': 0.01}

  0%|          | 31/41250 [05:03<99:18:54,  8.67s/it][2025-04-26 04:13:41,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 04:13:41,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5711.40 | bwd_inner_microstep: 5643.86 | bwd_allreduce_microstep: 67.49 | step_microstep: 19.02
[2025-04-26 04:13:41,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5711.41 | bwd_inner: 5643.86 | bwd_allreduce: 67.52 | step: 19.03

  0%|          | 32/41250 [05:12<99:09:28,  8.66s/it]
                                                     
{'loss': 1.3729, 'grad_norm': 16.091819763183594, 'learning_rate': 1.0339256865912764e-06, 'epoch': 0.01}

  0%|          | 32/41250 [05:12<99:09:28,  8.66s/it][2025-04-26 04:13:49,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:13:49,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.17 | bwd_microstep: 5716.82 | bwd_inner_microstep: 5704.11 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.44
[2025-04-26 04:13:49,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.18 | bwd: 5716.83 | bwd_inner: 5704.11 | bwd_allreduce: 12.69 | step: 18.44

  0%|          | 33/41250 [05:21<99:08:42,  8.66s/it]
                                                     
{'loss': 0.9312, 'grad_norm': 12.871021270751953, 'learning_rate': 1.0662358642972536e-06, 'epoch': 0.01}

  0%|          | 33/41250 [05:21<99:08:42,  8.66s/it][2025-04-26 04:13:58,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 04:13:58,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.24 | bwd_microstep: 5766.05 | bwd_inner_microstep: 5640.49 | bwd_allreduce_microstep: 125.51 | step_microstep: 18.69
[2025-04-26 04:13:58,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.24 | bwd: 5766.06 | bwd_inner: 5640.49 | bwd_allreduce: 125.53 | step: 18.69

  0%|          | 34/41250 [05:29<99:13:41,  8.67s/it]
                                                     
{'loss': 0.7739, 'grad_norm': 15.670796394348145, 'learning_rate': 1.098546042003231e-06, 'epoch': 0.01}

  0%|          | 34/41250 [05:29<99:13:41,  8.67s/it][2025-04-26 04:14:07,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:14:07,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5710.25 | bwd_inner_microstep: 5697.62 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.34
[2025-04-26 04:14:07,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5710.27 | bwd_inner: 5697.62 | bwd_allreduce: 12.60 | step: 18.34

  0%|          | 35/41250 [05:38<99:07:08,  8.66s/it]
                                                     
{'loss': 0.7089, 'grad_norm': 12.418981552124023, 'learning_rate': 1.1308562197092086e-06, 'epoch': 0.01}

  0%|          | 35/41250 [05:38<99:07:08,  8.66s/it][2025-04-26 04:14:15,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-26 04:14:15,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.37 | bwd_microstep: 5718.70 | bwd_inner_microstep: 5654.77 | bwd_allreduce_microstep: 63.88 | step_microstep: 19.50
[2025-04-26 04:14:15,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.37 | bwd: 5718.71 | bwd_inner: 5654.77 | bwd_allreduce: 63.90 | step: 19.51

  0%|          | 36/41250 [05:46<99:02:50,  8.65s/it]
                                                     
{'loss': 1.1336, 'grad_norm': 15.586511611938477, 'learning_rate': 1.1631663974151858e-06, 'epoch': 0.01}

  0%|          | 36/41250 [05:46<99:02:50,  8.65s/it][2025-04-26 04:14:24,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.12 | optimizer_step: 1.22
[2025-04-26 04:14:24,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.54 | bwd_microstep: 6082.34 | bwd_inner_microstep: 5647.31 | bwd_allreduce_microstep: 434.97 | step_microstep: 19.87
[2025-04-26 04:14:24,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.54 | bwd: 6082.35 | bwd_inner: 5647.30 | bwd_allreduce: 435.00 | step: 19.87

  0%|          | 37/41250 [05:55<100:16:31,  8.76s/it]
                                                      
{'loss': 0.9755, 'grad_norm': 12.607234954833984, 'learning_rate': 1.1954765751211633e-06, 'epoch': 0.01}

  0%|          | 37/41250 [05:55<100:16:31,  8.76s/it][2025-04-26 04:14:33,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-26 04:14:33,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.82 | bwd_microstep: 5759.18 | bwd_inner_microstep: 5641.24 | bwd_allreduce_microstep: 117.89 | step_microstep: 18.89
[2025-04-26 04:14:33,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.82 | bwd: 5759.19 | bwd_inner: 5641.24 | bwd_allreduce: 117.91 | step: 18.89

  0%|          | 38/41250 [06:04<99:58:47,  8.73s/it] 
                                                     
{'loss': 0.8996, 'grad_norm': 14.285001754760742, 'learning_rate': 1.2277867528271405e-06, 'epoch': 0.01}

  0%|          | 38/41250 [06:04<99:58:47,  8.73s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x3f4092c0] moov atom not found
[04:14:33] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4...
[2025-04-26 04:14:42,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 04:14:42,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.58 | bwd_microstep: 5713.58 | bwd_inner_microstep: 5696.16 | bwd_allreduce_microstep: 17.36 | step_microstep: 19.11
[2025-04-26 04:14:42,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.58 | bwd: 5713.59 | bwd_inner: 5696.16 | bwd_allreduce: 17.39 | step: 19.11

  0%|          | 39/41250 [06:13<99:42:54,  8.71s/it]
                                                     
{'loss': 1.1315, 'grad_norm': 12.315929412841797, 'learning_rate': 1.2600969305331182e-06, 'epoch': 0.01}

  0%|          | 39/41250 [06:13<99:42:54,  8.71s/it][2025-04-26 04:14:50,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 04:14:50,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.12 | bwd_microstep: 5695.19 | bwd_inner_microstep: 5682.37 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.76
[2025-04-26 04:14:50,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.12 | bwd: 5695.21 | bwd_inner: 5682.37 | bwd_allreduce: 12.79 | step: 18.77

  0%|          | 40/41250 [06:21<99:25:56,  8.69s/it]
                                                     
{'loss': 1.0521, 'grad_norm': 12.486302375793457, 'learning_rate': 1.2924071082390954e-06, 'epoch': 0.01}

  0%|          | 40/41250 [06:21<99:25:56,  8.69s/it][2025-04-26 04:14:59,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:14:59,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.29 | bwd_microstep: 6191.38 | bwd_inner_microstep: 5682.48 | bwd_allreduce_microstep: 508.85 | step_microstep: 18.49
[2025-04-26 04:14:59,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.29 | bwd: 6191.39 | bwd_inner: 5682.48 | bwd_allreduce: 508.87 | step: 18.50

  0%|          | 41/41250 [06:31<100:56:43,  8.82s/it]
                                                      
{'loss': 1.275, 'grad_norm': 17.14630889892578, 'learning_rate': 1.3247172859450727e-06, 'epoch': 0.01}

  0%|          | 41/41250 [06:31<100:56:43,  8.82s/it][2025-04-26 04:15:08,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 04:15:08,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.52 | bwd_microstep: 5767.82 | bwd_inner_microstep: 5649.50 | bwd_allreduce_microstep: 118.27 | step_microstep: 19.12
[2025-04-26 04:15:08,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.52 | bwd: 5767.83 | bwd_inner: 5649.50 | bwd_allreduce: 118.29 | step: 19.12

  0%|          | 42/41250 [06:39<100:30:03,  8.78s/it]
                                                      
{'loss': 1.2768, 'grad_norm': 17.736404418945312, 'learning_rate': 1.3570274636510504e-06, 'epoch': 0.01}

  0%|          | 42/41250 [06:39<100:30:03,  8.78s/it][2025-04-26 04:15:17,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 04:15:17,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.96 | bwd_microstep: 5776.64 | bwd_inner_microstep: 5633.97 | bwd_allreduce_microstep: 142.61 | step_microstep: 18.67
[2025-04-26 04:15:17,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.96 | bwd: 5776.66 | bwd_inner: 5633.97 | bwd_allreduce: 142.63 | step: 18.67

  0%|          | 43/41250 [06:48<100:10:58,  8.75s/it]
                                                      
{'loss': 0.7882, 'grad_norm': 11.620851516723633, 'learning_rate': 1.3893376413570276e-06, 'epoch': 0.01}

  0%|          | 43/41250 [06:48<100:10:58,  8.75s/it][2025-04-26 04:15:25,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 04:15:25,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.44 | bwd_microstep: 5690.95 | bwd_inner_microstep: 5678.22 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.42
[2025-04-26 04:15:25,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.44 | bwd: 5690.96 | bwd_inner: 5678.22 | bwd_allreduce: 12.70 | step: 19.42

  0%|          | 44/41250 [06:57<99:44:42,  8.71s/it] 
                                                     
{'loss': 1.0664, 'grad_norm': 17.928503036499023, 'learning_rate': 1.421647819063005e-06, 'epoch': 0.01}

  0%|          | 44/41250 [06:57<99:44:42,  8.71s/it][2025-04-26 04:15:34,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 04:15:34,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.92 | bwd_microstep: 5705.86 | bwd_inner_microstep: 5642.56 | bwd_allreduce_microstep: 63.25 | step_microstep: 19.00
[2025-04-26 04:15:34,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.92 | bwd: 5705.87 | bwd_inner: 5642.56 | bwd_allreduce: 63.27 | step: 19.00

  0%|          | 45/41250 [07:05<99:24:06,  8.68s/it]
                                                     
{'loss': 0.9208, 'grad_norm': 16.654226303100586, 'learning_rate': 1.4539579967689823e-06, 'epoch': 0.01}

  0%|          | 45/41250 [07:05<99:24:06,  8.68s/it][2025-04-26 04:15:43,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 04:15:43,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.00 | bwd_microstep: 5696.53 | bwd_inner_microstep: 5637.62 | bwd_allreduce_microstep: 58.87 | step_microstep: 18.46
[2025-04-26 04:15:43,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.00 | bwd: 5696.55 | bwd_inner: 5637.61 | bwd_allreduce: 58.89 | step: 18.47

  0%|          | 46/41250 [07:14<99:06:36,  8.66s/it]
                                                     
{'loss': 1.1417, 'grad_norm': 13.942131996154785, 'learning_rate': 1.4862681744749596e-06, 'epoch': 0.01}

  0%|          | 46/41250 [07:14<99:06:36,  8.66s/it][2025-04-26 04:15:51,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.19
[2025-04-26 04:15:51,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5719.07 | bwd_inner_microstep: 5696.59 | bwd_allreduce_microstep: 22.44 | step_microstep: 19.24
[2025-04-26 04:15:51,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 5719.09 | bwd_inner: 5696.59 | bwd_allreduce: 22.46 | step: 19.24

  0%|          | 47/41250 [07:22<99:04:59,  8.66s/it]
                                                     
{'loss': 0.629, 'grad_norm': 11.237614631652832, 'learning_rate': 1.5185783521809372e-06, 'epoch': 0.01}

  0%|          | 47/41250 [07:22<99:04:59,  8.66s/it][2025-04-26 04:16:00,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.07 | optimizer_step: 1.08
[2025-04-26 04:16:00,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 6083.59 | bwd_inner_microstep: 5645.59 | bwd_allreduce_microstep: 437.95 | step_microstep: 19.67
[2025-04-26 04:16:00,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 6083.60 | bwd_inner: 5645.59 | bwd_allreduce: 437.97 | step: 19.67

  0%|          | 48/41250 [07:31<100:14:58,  8.76s/it]
                                                      
{'loss': 0.5583, 'grad_norm': 8.060515403747559, 'learning_rate': 1.5508885298869145e-06, 'epoch': 0.01}

  0%|          | 48/41250 [07:31<100:14:58,  8.76s/it][2025-04-26 04:16:09,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.25
[2025-04-26 04:16:09,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.77 | bwd_microstep: 5708.46 | bwd_inner_microstep: 5695.57 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.44
[2025-04-26 04:16:09,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.77 | bwd: 5708.48 | bwd_inner: 5695.57 | bwd_allreduce: 12.86 | step: 19.45

  0%|          | 49/41250 [07:40<99:55:15,  8.73s/it] 
                                                     
{'loss': 0.7762, 'grad_norm': 12.325505256652832, 'learning_rate': 1.583198707592892e-06, 'epoch': 0.01}

  0%|          | 49/41250 [07:40<99:55:15,  8.73s/it][2025-04-26 04:16:18,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 04:16:18,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5726.62 | bwd_inner_microstep: 5684.81 | bwd_allreduce_microstep: 41.76 | step_microstep: 18.68
[2025-04-26 04:16:18,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5726.63 | bwd_inner: 5684.81 | bwd_allreduce: 41.78 | step: 18.68

  0%|          | 50/41250 [07:49<99:42:22,  8.71s/it]
                                                     
{'loss': 0.8267, 'grad_norm': 12.989849090576172, 'learning_rate': 1.6155088852988692e-06, 'epoch': 0.01}

  0%|          | 50/41250 [07:49<99:42:22,  8.71s/it][2025-04-26 04:16:26,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 04:16:26,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5685.68 | bwd_inner_microstep: 5649.05 | bwd_allreduce_microstep: 36.58 | step_microstep: 19.32
[2025-04-26 04:16:26,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5685.69 | bwd_inner: 5649.05 | bwd_allreduce: 36.60 | step: 19.32

  0%|          | 51/41250 [07:57<99:19:31,  8.68s/it]
                                                     
{'loss': 0.9371, 'grad_norm': 14.815985679626465, 'learning_rate': 1.6478190630048467e-06, 'epoch': 0.01}

  0%|          | 51/41250 [07:57<99:19:31,  8.68s/it][2025-04-26 04:16:35,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 1.20
[2025-04-26 04:16:35,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5691.51 | bwd_inner_microstep: 5678.28 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.82
[2025-04-26 04:16:35,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5691.52 | bwd_inner: 5678.28 | bwd_allreduce: 13.20 | step: 19.83

  0%|          | 52/41250 [08:06<99:06:41,  8.66s/it]
                                                     
{'loss': 1.1102, 'grad_norm': 17.531326293945312, 'learning_rate': 1.6801292407108241e-06, 'epoch': 0.01}

  0%|          | 52/41250 [08:06<99:06:41,  8.66s/it][2025-04-26 04:16:43,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.08 | optimizer_step: 1.04
[2025-04-26 04:16:43,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.58 | bwd_microstep: 5746.32 | bwd_inner_microstep: 5651.71 | bwd_allreduce_microstep: 94.55 | step_microstep: 19.59
[2025-04-26 04:16:43,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.58 | bwd: 5746.34 | bwd_inner: 5651.71 | bwd_allreduce: 94.58 | step: 19.59

  0%|          | 53/41250 [08:15<99:06:11,  8.66s/it]
                                                     
{'loss': 1.1401, 'grad_norm': 13.939374923706055, 'learning_rate': 1.7124394184168014e-06, 'epoch': 0.01}

  0%|          | 53/41250 [08:15<99:06:11,  8.66s/it][2025-04-26 04:16:52,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 04:16:52,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5765.15 | bwd_inner_microstep: 5642.65 | bwd_allreduce_microstep: 122.44 | step_microstep: 18.75
[2025-04-26 04:16:52,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5765.17 | bwd_inner: 5642.65 | bwd_allreduce: 122.46 | step: 18.75

  0%|          | 54/41250 [08:23<99:09:50,  8.67s/it]
                                                     
{'loss': 1.1396, 'grad_norm': 17.424577713012695, 'learning_rate': 1.744749596122779e-06, 'epoch': 0.01}

  0%|          | 54/41250 [08:23<99:09:50,  8.67s/it][2025-04-26 04:17:01,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 04:17:01,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.53 | bwd_microstep: 6355.42 | bwd_inner_microstep: 5655.71 | bwd_allreduce_microstep: 699.66 | step_microstep: 19.23
[2025-04-26 04:17:01,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.53 | bwd: 6355.43 | bwd_inner: 5655.71 | bwd_allreduce: 699.68 | step: 19.23

  0%|          | 55/41250 [08:33<101:13:12,  8.85s/it]
                                                      
{'loss': 0.7918, 'grad_norm': 12.807659149169922, 'learning_rate': 1.7770597738287563e-06, 'epoch': 0.01}

  0%|          | 55/41250 [08:33<101:13:12,  8.85s/it][2025-04-26 04:17:10,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:17:10,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.33 | bwd_microstep: 5781.68 | bwd_inner_microstep: 5695.92 | bwd_allreduce_microstep: 85.72 | step_microstep: 18.76
[2025-04-26 04:17:10,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.33 | bwd: 5781.69 | bwd_inner: 5695.92 | bwd_allreduce: 85.73 | step: 18.76

  0%|          | 56/41250 [08:41<100:43:16,  8.80s/it]
                                                      
{'loss': 0.6745, 'grad_norm': 9.157393455505371, 'learning_rate': 1.8093699515347335e-06, 'epoch': 0.01}

  0%|          | 56/41250 [08:41<100:43:16,  8.80s/it][2025-04-26 04:17:19,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-26 04:17:19,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.18 | bwd_microstep: 5744.28 | bwd_inner_microstep: 5685.19 | bwd_allreduce_microstep: 59.04 | step_microstep: 18.99
[2025-04-26 04:17:19,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.18 | bwd: 5744.29 | bwd_inner: 5685.19 | bwd_allreduce: 59.06 | step: 18.99

  0%|          | 57/41250 [08:50<100:17:28,  8.76s/it]
                                                      
{'loss': 0.6969, 'grad_norm': 11.133265495300293, 'learning_rate': 1.841680129240711e-06, 'epoch': 0.01}

  0%|          | 57/41250 [08:50<100:17:28,  8.76s/it][2025-04-26 04:17:27,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 04:17:27,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.89 | bwd_microstep: 5776.98 | bwd_inner_microstep: 5645.91 | bwd_allreduce_microstep: 131.03 | step_microstep: 18.26
[2025-04-26 04:17:27,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.89 | bwd: 5776.99 | bwd_inner: 5645.91 | bwd_allreduce: 131.04 | step: 18.26

  0%|          | 58/41250 [08:59<100:00:00,  8.74s/it]
                                                      
{'loss': 0.9009, 'grad_norm': 15.881463050842285, 'learning_rate': 1.8739903069466882e-06, 'epoch': 0.01}

  0%|          | 58/41250 [08:59<100:00:00,  8.74s/it][2025-04-26 04:17:36,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:17:36,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.84 | bwd_microstep: 5706.09 | bwd_inner_microstep: 5693.36 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.48
[2025-04-26 04:17:36,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.84 | bwd: 5706.11 | bwd_inner: 5693.36 | bwd_allreduce: 12.71 | step: 18.49

  0%|          | 59/41250 [09:07<99:41:39,  8.71s/it] 
                                                     
{'loss': 0.9231, 'grad_norm': 12.867300987243652, 'learning_rate': 1.906300484652666e-06, 'epoch': 0.01}

  0%|          | 59/41250 [09:07<99:41:39,  8.71s/it][2025-04-26 04:17:45,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.19
[2025-04-26 04:17:45,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.14 | bwd_microstep: 5712.50 | bwd_inner_microstep: 5652.87 | bwd_allreduce_microstep: 59.57 | step_microstep: 19.21
[2025-04-26 04:17:45,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.14 | bwd: 5712.51 | bwd_inner: 5652.87 | bwd_allreduce: 59.60 | step: 19.22

  0%|          | 60/41250 [09:16<99:22:56,  8.69s/it]
                                                     
{'loss': 0.6072, 'grad_norm': 10.837729454040527, 'learning_rate': 1.938610662358643e-06, 'epoch': 0.01}

  0%|          | 60/41250 [09:16<99:22:56,  8.69s/it][2025-04-26 04:17:53,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 04:17:53,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.37 | bwd_microstep: 5732.54 | bwd_inner_microstep: 5719.62 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.04
[2025-04-26 04:17:53,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.37 | bwd: 5732.55 | bwd_inner: 5719.62 | bwd_allreduce: 12.88 | step: 19.04

  0%|          | 61/41250 [09:25<99:20:12,  8.68s/it]
                                                     
{'loss': 0.9449, 'grad_norm': 17.555065155029297, 'learning_rate': 1.9709208400646204e-06, 'epoch': 0.01}

  0%|          | 61/41250 [09:25<99:20:12,  8.68s/it][2025-04-26 04:18:02,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 04:18:02,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.58 | bwd_microstep: 5694.92 | bwd_inner_microstep: 5681.96 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.21
[2025-04-26 04:18:02,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.58 | bwd: 5694.93 | bwd_inner: 5681.96 | bwd_allreduce: 12.93 | step: 19.22

  0%|          | 62/41250 [09:33<99:08:40,  8.67s/it]
                                                     
{'loss': 0.8376, 'grad_norm': 11.777689933776855, 'learning_rate': 2.003231017770598e-06, 'epoch': 0.02}

  0%|          | 62/41250 [09:33<99:08:40,  8.67s/it][2025-04-26 04:18:11,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.10 | optimizer_step: 1.08
[2025-04-26 04:18:11,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.55 | bwd_microstep: 5855.20 | bwd_inner_microstep: 5722.18 | bwd_allreduce_microstep: 132.97 | step_microstep: 19.60
[2025-04-26 04:18:11,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.55 | bwd: 5855.22 | bwd_inner: 5722.18 | bwd_allreduce: 132.99 | step: 19.60

  0%|          | 63/41250 [09:42<99:34:32,  8.70s/it]
                                                     
{'loss': 0.5816, 'grad_norm': 9.687881469726562, 'learning_rate': 2.0355411954765753e-06, 'epoch': 0.02}

  0%|          | 63/41250 [09:42<99:34:32,  8.70s/it][2025-04-26 04:18:19,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:18:19,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5714.28 | bwd_inner_microstep: 5639.48 | bwd_allreduce_microstep: 74.76 | step_microstep: 18.48
[2025-04-26 04:18:19,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5714.29 | bwd_inner: 5639.48 | bwd_allreduce: 74.78 | step: 18.48

  0%|          | 64/41250 [09:51<99:17:20,  8.68s/it]
                                                     
{'loss': 0.7891, 'grad_norm': 11.632110595703125, 'learning_rate': 2.0678513731825528e-06, 'epoch': 0.02}

  0%|          | 64/41250 [09:51<99:17:20,  8.68s/it][2025-04-26 04:18:28,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:18:28,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5712.82 | bwd_inner_microstep: 5653.89 | bwd_allreduce_microstep: 58.89 | step_microstep: 18.62
[2025-04-26 04:18:28,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5712.83 | bwd_inner: 5653.89 | bwd_allreduce: 58.90 | step: 18.62

  0%|          | 65/41250 [09:59<99:06:33,  8.66s/it]
                                                     
{'loss': 0.7348, 'grad_norm': 13.659263610839844, 'learning_rate': 2.1001615508885302e-06, 'epoch': 0.02}

  0%|          | 65/41250 [09:59<99:06:33,  8.66s/it][2025-04-26 04:18:37,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 04:18:37,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5710.88 | bwd_inner_microstep: 5661.03 | bwd_allreduce_microstep: 49.80 | step_microstep: 19.05
[2025-04-26 04:18:37,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5710.89 | bwd_inner: 5661.03 | bwd_allreduce: 49.82 | step: 19.06

  0%|          | 66/41250 [10:08<98:57:47,  8.65s/it]
                                                     
{'loss': 0.4837, 'grad_norm': 9.677002906799316, 'learning_rate': 2.1324717285945073e-06, 'epoch': 0.02}

  0%|          | 66/41250 [10:08<98:57:47,  8.65s/it][2025-04-26 04:18:45,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 04:18:45,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.43 | bwd_microstep: 5774.55 | bwd_inner_microstep: 5702.22 | bwd_allreduce_microstep: 72.29 | step_microstep: 18.54
[2025-04-26 04:18:45,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.43 | bwd: 5774.56 | bwd_inner: 5702.22 | bwd_allreduce: 72.30 | step: 18.54

  0%|          | 67/41250 [10:17<99:11:04,  8.67s/it]
                                                     
{'loss': 0.4997, 'grad_norm': 9.870966911315918, 'learning_rate': 2.1647819063004847e-06, 'epoch': 0.02}

  0%|          | 67/41250 [10:17<99:11:04,  8.67s/it][2025-04-26 04:18:54,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 04:18:54,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.06 | bwd_microstep: 5794.85 | bwd_inner_microstep: 5655.21 | bwd_allreduce_microstep: 139.59 | step_microstep: 19.17
[2025-04-26 04:18:54,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.06 | bwd: 5794.86 | bwd_inner: 5655.21 | bwd_allreduce: 139.61 | step: 19.17

  0%|          | 68/41250 [10:25<99:18:41,  8.68s/it]
                                                     
{'loss': 0.6328, 'grad_norm': 9.365510940551758, 'learning_rate': 2.197092084006462e-06, 'epoch': 0.02}

  0%|          | 68/41250 [10:25<99:18:41,  8.68s/it][2025-04-26 04:19:03,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 04:19:03,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.82 | bwd_microstep: 5782.39 | bwd_inner_microstep: 5645.40 | bwd_allreduce_microstep: 136.93 | step_microstep: 19.38
[2025-04-26 04:19:03,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.82 | bwd: 5782.40 | bwd_inner: 5645.40 | bwd_allreduce: 136.96 | step: 19.38

  0%|          | 69/41250 [10:34<99:20:39,  8.68s/it]
                                                     
{'loss': 0.7261, 'grad_norm': 9.087240219116211, 'learning_rate': 2.2294022617124397e-06, 'epoch': 0.02}

  0%|          | 69/41250 [10:34<99:20:39,  8.68s/it][2025-04-26 04:19:11,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.29 | optimizer_step: 1.07
[2025-04-26 04:19:11,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.76 | bwd_microstep: 5735.45 | bwd_inner_microstep: 5646.11 | bwd_allreduce_microstep: 89.29 | step_microstep: 19.60
[2025-04-26 04:19:11,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.76 | bwd: 5735.46 | bwd_inner: 5646.11 | bwd_allreduce: 89.31 | step: 19.60

  0%|          | 70/41250 [10:43<99:12:02,  8.67s/it]
                                                     
{'loss': 0.6058, 'grad_norm': 10.456581115722656, 'learning_rate': 2.261712439418417e-06, 'epoch': 0.02}

  0%|          | 70/41250 [10:43<99:12:02,  8.67s/it][2025-04-26 04:19:20,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.14
[2025-04-26 04:19:20,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.33 | bwd_microstep: 5721.94 | bwd_inner_microstep: 5641.31 | bwd_allreduce_microstep: 80.59 | step_microstep: 18.88
[2025-04-26 04:19:20,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.33 | bwd: 5721.96 | bwd_inner: 5641.31 | bwd_allreduce: 80.61 | step: 18.88

  0%|          | 71/41250 [10:51<99:04:35,  8.66s/it]
                                                     
{'loss': 0.7085, 'grad_norm': 9.082622528076172, 'learning_rate': 2.294022617124394e-06, 'epoch': 0.02}

  0%|          | 71/41250 [10:51<99:04:35,  8.66s/it][2025-04-26 04:19:29,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.14
[2025-04-26 04:19:29,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.84 | bwd_microstep: 5764.62 | bwd_inner_microstep: 5680.75 | bwd_allreduce_microstep: 83.81 | step_microstep: 19.78
[2025-04-26 04:19:29,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.84 | bwd: 5764.63 | bwd_inner: 5680.75 | bwd_allreduce: 83.84 | step: 19.78

  0%|          | 72/41250 [11:00<99:12:29,  8.67s/it]
                                                     
{'loss': 0.552, 'grad_norm': 6.397705554962158, 'learning_rate': 2.3263327948303716e-06, 'epoch': 0.02}

  0%|          | 72/41250 [11:00<99:12:29,  8.67s/it][2025-04-26 04:19:37,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 04:19:37,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.43 | bwd_microstep: 5711.26 | bwd_inner_microstep: 5698.38 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.57
[2025-04-26 04:19:37,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.43 | bwd: 5711.27 | bwd_inner: 5698.38 | bwd_allreduce: 12.85 | step: 19.57

  0%|          | 73/41250 [11:09<99:06:51,  8.67s/it]
                                                     
{'loss': 0.4305, 'grad_norm': 5.830964088439941, 'learning_rate': 2.358642972536349e-06, 'epoch': 0.02}

  0%|          | 73/41250 [11:09<99:06:51,  8.67s/it][2025-04-26 04:19:46,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 04:19:46,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.68 | bwd_microstep: 5766.78 | bwd_inner_microstep: 5655.13 | bwd_allreduce_microstep: 111.60 | step_microstep: 19.18
[2025-04-26 04:19:46,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.68 | bwd: 5766.80 | bwd_inner: 5655.13 | bwd_allreduce: 111.62 | step: 19.18

  0%|          | 74/41250 [11:17<99:06:45,  8.67s/it]
                                                     
{'loss': 0.4053, 'grad_norm': 12.515425682067871, 'learning_rate': 2.3909531502423265e-06, 'epoch': 0.02}

  0%|          | 74/41250 [11:17<99:06:45,  8.67s/it][2025-04-26 04:19:55,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 04:19:55,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5752.40 | bwd_inner_microstep: 5680.23 | bwd_allreduce_microstep: 72.12 | step_microstep: 18.81
[2025-04-26 04:19:55,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5752.41 | bwd_inner: 5680.23 | bwd_allreduce: 72.14 | step: 18.81

  0%|          | 75/41250 [11:26<99:11:37,  8.67s/it]
                                                     
{'loss': 0.5973, 'grad_norm': 7.4307966232299805, 'learning_rate': 2.423263327948304e-06, 'epoch': 0.02}

  0%|          | 75/41250 [11:26<99:11:37,  8.67s/it][2025-04-26 04:20:03,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:20:03,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.04 | bwd_microstep: 5715.19 | bwd_inner_microstep: 5653.29 | bwd_allreduce_microstep: 61.86 | step_microstep: 18.57
[2025-04-26 04:20:03,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.04 | bwd: 5715.20 | bwd_inner: 5653.29 | bwd_allreduce: 61.87 | step: 18.57

  0%|          | 76/41250 [11:35<99:01:45,  8.66s/it]
                                                     
{'loss': 0.4943, 'grad_norm': 6.973290920257568, 'learning_rate': 2.455573505654281e-06, 'epoch': 0.02}

  0%|          | 76/41250 [11:35<99:01:45,  8.66s/it][2025-04-26 04:20:12,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 04:20:12,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.84 | bwd_microstep: 5763.59 | bwd_inner_microstep: 5653.06 | bwd_allreduce_microstep: 110.48 | step_microstep: 19.07
[2025-04-26 04:20:12,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.84 | bwd: 5763.60 | bwd_inner: 5653.06 | bwd_allreduce: 110.50 | step: 19.08

  0%|          | 77/41250 [11:43<99:04:41,  8.66s/it]
                                                     
{'loss': 0.4381, 'grad_norm': 5.64487886428833, 'learning_rate': 2.487883683360259e-06, 'epoch': 0.02}

  0%|          | 77/41250 [11:43<99:04:41,  8.66s/it][2025-04-26 04:20:21,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.42 | optimizer_step: 0.90
[2025-04-26 04:20:21,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3272.95 | bwd_microstep: 5880.98 | bwd_inner_microstep: 5867.73 | bwd_allreduce_microstep: 13.20 | step_microstep: 19.68
[2025-04-26 04:20:21,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3272.95 | bwd: 5880.99 | bwd_inner: 5867.73 | bwd_allreduce: 13.22 | step: 19.69

  0%|          | 78/41250 [11:53<101:02:54,  8.84s/it]
                                                      
{'loss': 0.3344, 'grad_norm': 4.419320106506348, 'learning_rate': 2.5201938610662364e-06, 'epoch': 0.02}

  0%|          | 78/41250 [11:53<101:02:54,  8.84s/it][2025-04-26 04:20:30,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:20:30,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.39 | bwd_microstep: 6009.46 | bwd_inner_microstep: 5701.00 | bwd_allreduce_microstep: 308.42 | step_microstep: 18.38
[2025-04-26 04:20:30,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.39 | bwd: 6009.47 | bwd_inner: 5701.00 | bwd_allreduce: 308.43 | step: 18.38

  0%|          | 79/41250 [12:01<101:23:35,  8.87s/it]
                                                      
{'loss': 0.3213, 'grad_norm': 6.099274635314941, 'learning_rate': 2.5525040387722134e-06, 'epoch': 0.02}

  0%|          | 79/41250 [12:01<101:23:35,  8.87s/it][2025-04-26 04:20:39,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 04:20:39,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5733.20 | bwd_inner_microstep: 5697.57 | bwd_allreduce_microstep: 35.59 | step_microstep: 18.78
[2025-04-26 04:20:39,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5733.21 | bwd_inner: 5697.57 | bwd_allreduce: 35.60 | step: 18.79

  0%|          | 80/41250 [12:10<100:43:44,  8.81s/it]
                                                      
{'loss': 0.4903, 'grad_norm': 11.803357124328613, 'learning_rate': 2.584814216478191e-06, 'epoch': 0.02}

  0%|          | 80/41250 [12:10<100:43:44,  8.81s/it][2025-04-26 04:20:48,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 04:20:48,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.53 | bwd_microstep: 5787.11 | bwd_inner_microstep: 5773.99 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.61
[2025-04-26 04:20:48,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.53 | bwd: 5787.12 | bwd_inner: 5773.99 | bwd_allreduce: 13.09 | step: 18.61

  0%|          | 81/41250 [12:19<100:34:30,  8.79s/it]
                                                      
{'loss': 0.6011, 'grad_norm': 3.986501693725586, 'learning_rate': 2.6171243941841683e-06, 'epoch': 0.02}

  0%|          | 81/41250 [12:19<100:34:30,  8.79s/it][2025-04-26 04:20:56,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 04:20:56,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.52 | bwd_microstep: 5704.73 | bwd_inner_microstep: 5691.89 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.09
[2025-04-26 04:20:56,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.52 | bwd: 5704.74 | bwd_inner: 5691.89 | bwd_allreduce: 12.82 | step: 19.09

  0%|          | 82/41250 [12:28<100:03:20,  8.75s/it]
                                                      
{'loss': 0.3908, 'grad_norm': 3.4681246280670166, 'learning_rate': 2.6494345718901454e-06, 'epoch': 0.02}

  0%|          | 82/41250 [12:28<100:03:20,  8.75s/it][2025-04-26 04:21:05,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:21:05,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.31 | bwd_microstep: 5764.03 | bwd_inner_microstep: 5643.64 | bwd_allreduce_microstep: 120.35 | step_microstep: 18.20
[2025-04-26 04:21:05,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.31 | bwd: 5764.05 | bwd_inner: 5643.64 | bwd_allreduce: 120.37 | step: 18.20

  0%|          | 83/41250 [12:36<99:46:09,  8.72s/it] 
                                                     
{'loss': 0.3623, 'grad_norm': 4.050808906555176, 'learning_rate': 2.681744749596123e-06, 'epoch': 0.02}

  0%|          | 83/41250 [12:36<99:46:09,  8.72s/it][2025-04-26 04:21:14,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.15
[2025-04-26 04:21:14,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.30 | bwd_microstep: 5702.46 | bwd_inner_microstep: 5689.23 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.44
[2025-04-26 04:21:14,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.30 | bwd: 5702.47 | bwd_inner: 5689.23 | bwd_allreduce: 13.20 | step: 19.44

  0%|          | 84/41250 [12:45<99:30:07,  8.70s/it]
                                                     
{'loss': 0.2545, 'grad_norm': 4.214967250823975, 'learning_rate': 2.7140549273021007e-06, 'epoch': 0.02}

  0%|          | 84/41250 [12:45<99:30:07,  8.70s/it][2025-04-26 04:21:22,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 04:21:22,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.21 | bwd_microstep: 5755.74 | bwd_inner_microstep: 5638.73 | bwd_allreduce_microstep: 116.97 | step_microstep: 18.92
[2025-04-26 04:21:22,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.21 | bwd: 5755.75 | bwd_inner: 5638.73 | bwd_allreduce: 116.98 | step: 18.92

  0%|          | 85/41250 [12:54<99:22:46,  8.69s/it]
                                                     
{'loss': 0.2993, 'grad_norm': 2.884518623352051, 'learning_rate': 2.7463651050080773e-06, 'epoch': 0.02}

  0%|          | 85/41250 [12:54<99:22:46,  8.69s/it][2025-04-26 04:21:31,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 04:21:31,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5716.83 | bwd_inner_microstep: 5692.23 | bwd_allreduce_microstep: 24.55 | step_microstep: 19.03
[2025-04-26 04:21:31,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5716.84 | bwd_inner: 5692.23 | bwd_allreduce: 24.57 | step: 19.04

  0%|          | 86/41250 [13:02<99:14:42,  8.68s/it]
                                                     
{'loss': 0.5787, 'grad_norm': 3.7614171504974365, 'learning_rate': 2.7786752827140552e-06, 'epoch': 0.02}

  0%|          | 86/41250 [13:02<99:14:42,  8.68s/it][2025-04-26 04:21:40,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 04:21:40,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.12 | bwd_microstep: 5766.49 | bwd_inner_microstep: 5699.97 | bwd_allreduce_microstep: 66.47 | step_microstep: 18.28
[2025-04-26 04:21:40,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.12 | bwd: 5766.51 | bwd_inner: 5699.97 | bwd_allreduce: 66.49 | step: 18.28

  0%|          | 87/41250 [13:11<99:18:04,  8.68s/it]
                                                     
{'loss': 0.6052, 'grad_norm': 5.089572429656982, 'learning_rate': 2.8109854604200327e-06, 'epoch': 0.02}

  0%|          | 87/41250 [13:11<99:18:04,  8.68s/it][2025-04-26 04:21:48,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 04:21:48,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.85 | bwd_microstep: 5704.39 | bwd_inner_microstep: 5644.81 | bwd_allreduce_microstep: 59.53 | step_microstep: 18.97
[2025-04-26 04:21:48,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.85 | bwd: 5704.41 | bwd_inner: 5644.81 | bwd_allreduce: 59.56 | step: 18.97

  0%|          | 88/41250 [13:19<99:01:56,  8.66s/it]
                                                     
{'loss': 0.3195, 'grad_norm': 5.6643266677856445, 'learning_rate': 2.84329563812601e-06, 'epoch': 0.02}

  0%|          | 88/41250 [13:19<99:01:56,  8.66s/it][2025-04-26 04:21:57,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 04:21:57,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.81 | bwd_microstep: 5751.87 | bwd_inner_microstep: 5649.42 | bwd_allreduce_microstep: 102.41 | step_microstep: 19.02
[2025-04-26 04:21:57,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.81 | bwd: 5751.88 | bwd_inner: 5649.42 | bwd_allreduce: 102.42 | step: 19.02

  0%|          | 89/41250 [13:28<99:04:11,  8.66s/it]
                                                     
{'loss': 0.6082, 'grad_norm': 14.211806297302246, 'learning_rate': 2.875605815831987e-06, 'epoch': 0.02}

  0%|          | 89/41250 [13:28<99:04:11,  8.66s/it][2025-04-26 04:22:06,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:22:06,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.25 | bwd_microstep: 5752.07 | bwd_inner_microstep: 5643.08 | bwd_allreduce_microstep: 108.95 | step_microstep: 17.91
[2025-04-26 04:22:06,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.25 | bwd: 5752.08 | bwd_inner: 5643.08 | bwd_allreduce: 108.97 | step: 17.92

  0%|          | 90/41250 [13:37<99:01:39,  8.66s/it]
                                                     
{'loss': 0.5377, 'grad_norm': 8.000323295593262, 'learning_rate': 2.9079159935379646e-06, 'epoch': 0.02}

  0%|          | 90/41250 [13:37<99:01:39,  8.66s/it][2025-04-26 04:22:14,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 04:22:14,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.18 | bwd_microstep: 5739.93 | bwd_inner_microstep: 5684.54 | bwd_allreduce_microstep: 55.34 | step_microstep: 18.99
[2025-04-26 04:22:14,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.18 | bwd: 5739.94 | bwd_inner: 5684.54 | bwd_allreduce: 55.36 | step: 18.99

  0%|          | 91/41250 [13:45<99:03:03,  8.66s/it]
                                                     
{'loss': 0.5202, 'grad_norm': 4.039388656616211, 'learning_rate': 2.940226171243942e-06, 'epoch': 0.02}

  0%|          | 91/41250 [13:45<99:03:03,  8.66s/it][2025-04-26 04:22:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:22:23,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.59 | bwd_microstep: 5704.63 | bwd_inner_microstep: 5654.12 | bwd_allreduce_microstep: 50.47 | step_microstep: 17.76
[2025-04-26 04:22:23,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.59 | bwd: 5704.64 | bwd_inner: 5654.12 | bwd_allreduce: 50.48 | step: 17.76

  0%|          | 92/41250 [13:54<98:50:13,  8.65s/it]
                                                     
{'loss': 0.2876, 'grad_norm': 2.7432684898376465, 'learning_rate': 2.972536348949919e-06, 'epoch': 0.02}

  0%|          | 92/41250 [13:54<98:50:13,  8.65s/it][2025-04-26 04:22:32,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:22:32,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.79 | bwd_microstep: 5723.61 | bwd_inner_microstep: 5698.90 | bwd_allreduce_microstep: 24.67 | step_microstep: 18.19
[2025-04-26 04:22:32,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5723.62 | bwd_inner: 5698.91 | bwd_allreduce: 24.68 | step: 18.20

  0%|          | 93/41250 [14:03<98:53:09,  8.65s/it]
                                                     
{'loss': 0.4543, 'grad_norm': 4.024933815002441, 'learning_rate': 3.004846526655897e-06, 'epoch': 0.02}

  0%|          | 93/41250 [14:03<98:53:09,  8.65s/it][2025-04-26 04:22:40,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:22:40,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.15 | bwd_microstep: 5698.37 | bwd_inner_microstep: 5685.65 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.34
[2025-04-26 04:22:40,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.15 | bwd: 5698.38 | bwd_inner: 5685.65 | bwd_allreduce: 12.70 | step: 18.35

  0%|          | 94/41250 [14:11<98:49:59,  8.65s/it]
                                                     
{'loss': 0.2613, 'grad_norm': 3.957373857498169, 'learning_rate': 3.0371567043618745e-06, 'epoch': 0.02}

  0%|          | 94/41250 [14:11<98:49:59,  8.65s/it][2025-04-26 04:22:49,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 04:22:49,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.99 | bwd_microstep: 5693.60 | bwd_inner_microstep: 5681.03 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.35
[2025-04-26 04:22:49,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.99 | bwd: 5693.61 | bwd_inner: 5681.02 | bwd_allreduce: 12.55 | step: 18.35

  0%|          | 95/41250 [14:20<98:46:23,  8.64s/it]
                                                     
{'loss': 0.2337, 'grad_norm': 2.4752895832061768, 'learning_rate': 3.0694668820678515e-06, 'epoch': 0.02}

  0%|          | 95/41250 [14:20<98:46:23,  8.64s/it][2025-04-26 04:22:57,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:22:57,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.67 | bwd_microstep: 5744.85 | bwd_inner_microstep: 5654.12 | bwd_allreduce_microstep: 90.68 | step_microstep: 17.74
[2025-04-26 04:22:57,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.67 | bwd: 5744.86 | bwd_inner: 5654.12 | bwd_allreduce: 90.70 | step: 17.75

  0%|          | 96/41250 [14:29<98:50:54,  8.65s/it]
                                                     
{'loss': 0.5188, 'grad_norm': 3.766841173171997, 'learning_rate': 3.101777059773829e-06, 'epoch': 0.02}

  0%|          | 96/41250 [14:29<98:50:54,  8.65s/it][2025-04-26 04:23:06,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 04:23:06,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.55 | bwd_microstep: 5777.20 | bwd_inner_microstep: 5763.99 | bwd_allreduce_microstep: 13.16 | step_microstep: 18.58
[2025-04-26 04:23:06,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.55 | bwd: 5777.21 | bwd_inner: 5763.99 | bwd_allreduce: 13.18 | step: 18.58

  0%|          | 97/41250 [14:37<99:12:14,  8.68s/it]
                                                     
{'loss': 0.3361, 'grad_norm': 2.447885036468506, 'learning_rate': 3.1340872374798064e-06, 'epoch': 0.02}

  0%|          | 97/41250 [14:37<99:12:14,  8.68s/it][2025-04-26 04:23:15,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:23:15,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.48 | bwd_microstep: 5734.69 | bwd_inner_microstep: 5709.60 | bwd_allreduce_microstep: 25.04 | step_microstep: 17.80
[2025-04-26 04:23:15,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.48 | bwd: 5734.70 | bwd_inner: 5709.60 | bwd_allreduce: 25.06 | step: 17.80

  0%|          | 98/41250 [14:46<99:14:15,  8.68s/it]
                                                     
{'loss': 0.4313, 'grad_norm': 4.194906234741211, 'learning_rate': 3.166397415185784e-06, 'epoch': 0.02}

  0%|          | 98/41250 [14:46<99:14:15,  8.68s/it][2025-04-26 04:23:24,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:23:24,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.94 | bwd_microstep: 5775.16 | bwd_inner_microstep: 5652.07 | bwd_allreduce_microstep: 123.04 | step_microstep: 18.49
[2025-04-26 04:23:24,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.94 | bwd: 5775.17 | bwd_inner: 5652.07 | bwd_allreduce: 123.06 | step: 18.49

  0%|          | 99/41250 [14:55<99:16:21,  8.68s/it]
                                                     
{'loss': 0.6032, 'grad_norm': 4.350310325622559, 'learning_rate': 3.198707592891761e-06, 'epoch': 0.02}

  0%|          | 99/41250 [14:55<99:16:21,  8.68s/it][2025-04-26 04:23:32,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:23:32,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5766.93 | bwd_inner_microstep: 5691.37 | bwd_allreduce_microstep: 75.53 | step_microstep: 18.33
[2025-04-26 04:23:32,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5766.95 | bwd_inner: 5691.37 | bwd_allreduce: 75.54 | step: 18.33

  0%|          | 100/41250 [15:03<99:18:41,  8.69s/it]
                                                      
{'loss': 0.2234, 'grad_norm': 1.70957350730896, 'learning_rate': 3.2310177705977384e-06, 'epoch': 0.02}

  0%|          | 100/41250 [15:04<99:18:41,  8.69s/it][2025-04-26 04:23:41,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:23:41,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.16 | bwd_microstep: 5763.96 | bwd_inner_microstep: 5682.59 | bwd_allreduce_microstep: 81.33 | step_microstep: 17.73
[2025-04-26 04:23:41,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.16 | bwd: 5763.97 | bwd_inner: 5682.59 | bwd_allreduce: 81.34 | step: 17.73

  0%|          | 101/41250 [15:12<99:18:20,  8.69s/it]
                                                      
{'loss': 0.3597, 'grad_norm': 3.0875742435455322, 'learning_rate': 3.2633279483037163e-06, 'epoch': 0.02}

  0%|          | 101/41250 [15:12<99:18:20,  8.69s/it][2025-04-26 04:23:50,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:23:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.90 | bwd_microstep: 5710.98 | bwd_inner_microstep: 5698.11 | bwd_allreduce_microstep: 12.83 | step_microstep: 17.81
[2025-04-26 04:23:50,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.90 | bwd: 5710.99 | bwd_inner: 5698.11 | bwd_allreduce: 12.85 | step: 17.81

  0%|          | 102/41250 [15:21<99:10:55,  8.68s/it]
                                                      
{'loss': 0.3466, 'grad_norm': 2.518275022506714, 'learning_rate': 3.2956381260096933e-06, 'epoch': 0.02}

  0%|          | 102/41250 [15:21<99:10:55,  8.68s/it][2025-04-26 04:23:58,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:23:58,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.88 | bwd_microstep: 5700.24 | bwd_inner_microstep: 5687.32 | bwd_allreduce_microstep: 12.88 | step_microstep: 17.74
[2025-04-26 04:23:58,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.88 | bwd: 5700.25 | bwd_inner: 5687.32 | bwd_allreduce: 12.89 | step: 17.74

  0%|          | 103/41250 [15:29<99:02:07,  8.66s/it]
                                                      
{'loss': 0.2117, 'grad_norm': 1.9244619607925415, 'learning_rate': 3.3279483037156708e-06, 'epoch': 0.02}

  0%|          | 103/41250 [15:29<99:02:07,  8.66s/it][2025-04-26 04:24:07,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 04:24:07,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.68 | bwd_microstep: 5750.99 | bwd_inner_microstep: 5703.31 | bwd_allreduce_microstep: 47.64 | step_microstep: 18.71
[2025-04-26 04:24:07,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.68 | bwd: 5751.00 | bwd_inner: 5703.31 | bwd_allreduce: 47.66 | step: 18.72

  0%|          | 104/41250 [15:38<99:06:03,  8.67s/it]
                                                      
{'loss': 0.4538, 'grad_norm': 4.166629314422607, 'learning_rate': 3.3602584814216482e-06, 'epoch': 0.03}

  0%|          | 104/41250 [15:38<99:06:03,  8.67s/it][2025-04-26 04:24:16,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:24:16,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5706.86 | bwd_inner_microstep: 5685.34 | bwd_allreduce_microstep: 21.48 | step_microstep: 17.94
[2025-04-26 04:24:16,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.29 | bwd: 5706.88 | bwd_inner: 5685.34 | bwd_allreduce: 21.50 | step: 17.95

  0%|          | 105/41250 [15:47<98:59:42,  8.66s/it]
                                                      
{'loss': 0.3602, 'grad_norm': 6.307094573974609, 'learning_rate': 3.3925686591276253e-06, 'epoch': 0.03}

  0%|          | 105/41250 [15:47<98:59:42,  8.66s/it][2025-04-26 04:24:24,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:24:24,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5752.36 | bwd_inner_microstep: 5709.85 | bwd_allreduce_microstep: 42.47 | step_microstep: 18.15
[2025-04-26 04:24:24,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5752.38 | bwd_inner: 5709.85 | bwd_allreduce: 42.48 | step: 18.15

  0%|          | 106/41250 [15:55<99:02:49,  8.67s/it]
                                                      
{'loss': 0.4197, 'grad_norm': 3.3919403553009033, 'learning_rate': 3.4248788368336027e-06, 'epoch': 0.03}

  0%|          | 106/41250 [15:55<99:02:49,  8.67s/it][2025-04-26 04:24:33,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:24:33,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.31 | bwd_microstep: 5711.54 | bwd_inner_microstep: 5655.94 | bwd_allreduce_microstep: 55.56 | step_microstep: 17.78
[2025-04-26 04:24:33,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.31 | bwd: 5711.56 | bwd_inner: 5655.94 | bwd_allreduce: 55.58 | step: 17.78

  0%|          | 107/41250 [16:04<98:55:58,  8.66s/it]
                                                      
{'loss': 0.4077, 'grad_norm': 3.645726442337036, 'learning_rate': 3.45718901453958e-06, 'epoch': 0.03}

  0%|          | 107/41250 [16:04<98:55:58,  8.66s/it][2025-04-26 04:24:42,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 04:24:42,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.38 | bwd_microstep: 5718.73 | bwd_inner_microstep: 5638.18 | bwd_allreduce_microstep: 80.51 | step_microstep: 18.72
[2025-04-26 04:24:42,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.38 | bwd: 5718.74 | bwd_inner: 5638.18 | bwd_allreduce: 80.53 | step: 18.72

  0%|          | 108/41250 [16:13<98:52:22,  8.65s/it]
                                                      
{'loss': 0.2183, 'grad_norm': 1.8921096324920654, 'learning_rate': 3.489499192245558e-06, 'epoch': 0.03}

  0%|          | 108/41250 [16:13<98:52:22,  8.65s/it][2025-04-26 04:24:50,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 04:24:50,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.56 | bwd_microstep: 5719.22 | bwd_inner_microstep: 5706.52 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.22
[2025-04-26 04:24:50,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.56 | bwd: 5719.23 | bwd_inner: 5706.52 | bwd_allreduce: 12.67 | step: 19.22

  0%|          | 109/41250 [16:21<98:56:03,  8.66s/it]
                                                      
{'loss': 0.4106, 'grad_norm': 3.953460931777954, 'learning_rate': 3.5218093699515347e-06, 'epoch': 0.03}

  0%|          | 109/41250 [16:21<98:56:03,  8.66s/it][2025-04-26 04:24:59,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 04:24:59,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.08 | bwd_microstep: 5793.14 | bwd_inner_microstep: 5696.70 | bwd_allreduce_microstep: 96.40 | step_microstep: 18.95
[2025-04-26 04:24:59,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.08 | bwd: 5793.15 | bwd_inner: 5696.70 | bwd_allreduce: 96.42 | step: 18.95

  0%|          | 110/41250 [16:30<99:07:58,  8.67s/it]
                                                      
{'loss': 0.3014, 'grad_norm': 2.035224199295044, 'learning_rate': 3.5541195476575126e-06, 'epoch': 0.03}

  0%|          | 110/41250 [16:30<99:07:58,  8.67s/it][2025-04-26 04:25:08,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 04:25:08,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.95 | bwd_microstep: 5767.55 | bwd_inner_microstep: 5725.97 | bwd_allreduce_microstep: 41.53 | step_microstep: 19.14
[2025-04-26 04:25:08,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.95 | bwd: 5767.56 | bwd_inner: 5725.97 | bwd_allreduce: 41.55 | step: 19.14

  0%|          | 111/41250 [16:39<99:16:04,  8.69s/it]
                                                      
{'loss': 0.4562, 'grad_norm': 3.407528877258301, 'learning_rate': 3.58642972536349e-06, 'epoch': 0.03}

  0%|          | 111/41250 [16:39<99:16:04,  8.69s/it][2025-04-26 04:25:16,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:25:16,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.71 | bwd_microstep: 5769.38 | bwd_inner_microstep: 5666.78 | bwd_allreduce_microstep: 102.55 | step_microstep: 18.45
[2025-04-26 04:25:16,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.71 | bwd: 5769.39 | bwd_inner: 5666.78 | bwd_allreduce: 102.57 | step: 18.45

  0%|          | 112/41250 [16:48<99:18:30,  8.69s/it]
                                                      
{'loss': 0.225, 'grad_norm': 2.955822467803955, 'learning_rate': 3.618739903069467e-06, 'epoch': 0.03}

  0%|          | 112/41250 [16:48<99:18:30,  8.69s/it][2025-04-26 04:25:25,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 04:25:25,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.03 | bwd_microstep: 5699.21 | bwd_inner_microstep: 5662.08 | bwd_allreduce_microstep: 37.09 | step_microstep: 18.70
[2025-04-26 04:25:25,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.03 | bwd: 5699.23 | bwd_inner: 5662.08 | bwd_allreduce: 37.10 | step: 18.71

  0%|          | 113/41250 [16:56<99:05:31,  8.67s/it]
                                                      
{'loss': 0.3745, 'grad_norm': 3.0973775386810303, 'learning_rate': 3.6510500807754445e-06, 'epoch': 0.03}

  0%|          | 113/41250 [16:56<99:05:31,  8.67s/it][2025-04-26 04:25:34,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 04:25:34,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.55 | bwd_microstep: 5740.06 | bwd_inner_microstep: 5715.16 | bwd_allreduce_microstep: 24.85 | step_microstep: 18.33
[2025-04-26 04:25:34,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.55 | bwd: 5740.08 | bwd_inner: 5715.16 | bwd_allreduce: 24.87 | step: 18.34

  0%|          | 114/41250 [17:05<99:08:11,  8.68s/it]
                                                      
{'loss': 0.6369, 'grad_norm': 7.732098579406738, 'learning_rate': 3.683360258481422e-06, 'epoch': 0.03}

  0%|          | 114/41250 [17:05<99:08:11,  8.68s/it][2025-04-26 04:25:43,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:25:43,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.41 | bwd_microstep: 6262.54 | bwd_inner_microstep: 5666.96 | bwd_allreduce_microstep: 595.53 | step_microstep: 18.37
[2025-04-26 04:25:43,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.41 | bwd: 6262.55 | bwd_inner: 5666.96 | bwd_allreduce: 595.55 | step: 18.37

  0%|          | 115/41250 [17:14<100:51:22,  8.83s/it]
                                                       
{'loss': 0.308, 'grad_norm': 3.9383223056793213, 'learning_rate': 3.715670436187399e-06, 'epoch': 0.03}

  0%|          | 115/41250 [17:14<100:51:22,  8.83s/it][2025-04-26 04:25:52,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:25:52,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.93 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5699.27 | bwd_allreduce_microstep: 65.82 | step_microstep: 18.17
[2025-04-26 04:25:52,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.94 | bwd: 5765.14 | bwd_inner: 5699.27 | bwd_allreduce: 65.84 | step: 18.17

  0%|          | 116/41250 [17:23<100:24:51,  8.79s/it]
                                                       
{'loss': 0.2151, 'grad_norm': 2.243198871612549, 'learning_rate': 3.7479806138933765e-06, 'epoch': 0.03}

  0%|          | 116/41250 [17:23<100:24:51,  8.79s/it][2025-04-26 04:26:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:26:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.44 | bwd_microstep: 5766.15 | bwd_inner_microstep: 5697.14 | bwd_allreduce_microstep: 68.97 | step_microstep: 18.47
[2025-04-26 04:26:00,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.44 | bwd: 5766.16 | bwd_inner: 5697.14 | bwd_allreduce: 68.98 | step: 18.47

  0%|          | 117/41250 [17:31<100:06:16,  8.76s/it]
                                                       
{'loss': 0.2406, 'grad_norm': 1.9895448684692383, 'learning_rate': 3.7802907915993544e-06, 'epoch': 0.03}

  0%|          | 117/41250 [17:31<100:06:16,  8.76s/it][2025-04-26 04:26:09,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 04:26:09,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.32 | bwd_microstep: 5708.64 | bwd_inner_microstep: 5653.01 | bwd_allreduce_microstep: 55.59 | step_microstep: 18.22
[2025-04-26 04:26:09,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.32 | bwd: 5708.65 | bwd_inner: 5653.01 | bwd_allreduce: 55.61 | step: 18.22

  0%|          | 118/41250 [17:40<99:35:28,  8.72s/it] 
                                                      
{'loss': 0.4413, 'grad_norm': 2.899829864501953, 'learning_rate': 3.812600969305332e-06, 'epoch': 0.03}

  0%|          | 118/41250 [17:40<99:35:28,  8.72s/it][2025-04-26 04:26:18,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:26:18,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.42 | bwd_microstep: 5759.57 | bwd_inner_microstep: 5702.66 | bwd_allreduce_microstep: 56.87 | step_microstep: 17.87
[2025-04-26 04:26:18,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.42 | bwd: 5759.58 | bwd_inner: 5702.66 | bwd_allreduce: 56.89 | step: 17.87

  0%|          | 119/41250 [17:49<99:30:37,  8.71s/it]
                                                      
{'loss': 0.2075, 'grad_norm': 1.626895546913147, 'learning_rate': 3.844911147011309e-06, 'epoch': 0.03}

  0%|          | 119/41250 [17:49<99:30:37,  8.71s/it][2025-04-26 04:26:26,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:26:26,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.28 | bwd_microstep: 5731.44 | bwd_inner_microstep: 5704.71 | bwd_allreduce_microstep: 26.69 | step_microstep: 17.79
[2025-04-26 04:26:26,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.28 | bwd: 5731.45 | bwd_inner: 5704.71 | bwd_allreduce: 26.70 | step: 17.79

  0%|          | 120/41250 [17:57<99:21:29,  8.70s/it]
                                                      
{'loss': 0.3686, 'grad_norm': 4.334173202514648, 'learning_rate': 3.877221324717286e-06, 'epoch': 0.03}

  0%|          | 120/41250 [17:57<99:21:29,  8.70s/it][2025-04-26 04:26:35,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:26:35,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.58 | bwd_microstep: 5712.58 | bwd_inner_microstep: 5699.90 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.77
[2025-04-26 04:26:35,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.58 | bwd: 5712.59 | bwd_inner: 5699.90 | bwd_allreduce: 12.66 | step: 17.77

  0%|          | 121/41250 [18:06<99:09:44,  8.68s/it]
                                                      
{'loss': 0.3441, 'grad_norm': 7.470336437225342, 'learning_rate': 3.909531502423264e-06, 'epoch': 0.03}

  0%|          | 121/41250 [18:06<99:09:44,  8.68s/it][2025-04-26 04:26:44,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:26:44,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.80 | bwd_microstep: 5722.23 | bwd_inner_microstep: 5709.55 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.50
[2025-04-26 04:26:44,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.80 | bwd: 5722.24 | bwd_inner: 5709.55 | bwd_allreduce: 12.65 | step: 18.51

  0%|          | 122/41250 [18:15<99:07:03,  8.68s/it]
                                                      
{'loss': 0.6412, 'grad_norm': 3.467392683029175, 'learning_rate': 3.941841680129241e-06, 'epoch': 0.03}

  0%|          | 122/41250 [18:15<99:07:03,  8.68s/it][2025-04-26 04:26:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 04:26:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.08 | bwd_microstep: 5861.68 | bwd_inner_microstep: 5700.29 | bwd_allreduce_microstep: 161.34 | step_microstep: 18.35
[2025-04-26 04:26:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.08 | bwd: 5861.69 | bwd_inner: 5700.29 | bwd_allreduce: 161.36 | step: 18.35

  0%|          | 123/41250 [18:24<99:30:57,  8.71s/it]
                                                      
{'loss': 0.3453, 'grad_norm': 3.1421189308166504, 'learning_rate': 3.974151857835219e-06, 'epoch': 0.03}

  0%|          | 123/41250 [18:24<99:30:57,  8.71s/it][2025-04-26 04:27:01,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:27:01,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.28 | bwd_microstep: 5709.98 | bwd_inner_microstep: 5697.41 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.26
[2025-04-26 04:27:01,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.28 | bwd: 5709.99 | bwd_inner: 5697.41 | bwd_allreduce: 12.55 | step: 18.27

  0%|          | 124/41250 [18:32<99:14:09,  8.69s/it]
                                                      
{'loss': 0.4561, 'grad_norm': 3.251512289047241, 'learning_rate': 4.006462035541196e-06, 'epoch': 0.03}

  0%|          | 124/41250 [18:32<99:14:09,  8.69s/it][2025-04-26 04:27:10,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 04:27:10,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.66 | bwd_microstep: 5771.95 | bwd_inner_microstep: 5651.24 | bwd_allreduce_microstep: 120.67 | step_microstep: 17.64
[2025-04-26 04:27:10,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.66 | bwd: 5771.96 | bwd_inner: 5651.24 | bwd_allreduce: 120.69 | step: 17.65

  0%|          | 125/41250 [18:41<99:14:37,  8.69s/it]
                                                      
{'loss': 0.2753, 'grad_norm': 2.2171993255615234, 'learning_rate': 4.038772213247173e-06, 'epoch': 0.03}

  0%|          | 125/41250 [18:41<99:14:37,  8.69s/it][2025-04-26 04:27:18,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 04:27:18,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.07 | bwd_microstep: 5754.03 | bwd_inner_microstep: 5741.37 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.48
[2025-04-26 04:27:18,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.07 | bwd: 5754.04 | bwd_inner: 5741.37 | bwd_allreduce: 12.63 | step: 18.48

  0%|          | 126/41250 [18:50<99:21:53,  8.70s/it]
                                                      
{'loss': 0.2937, 'grad_norm': 3.42175555229187, 'learning_rate': 4.071082390953151e-06, 'epoch': 0.03}

  0%|          | 126/41250 [18:50<99:21:53,  8.70s/it][2025-04-26 04:27:27,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:27:27,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.19 | bwd_microstep: 5720.36 | bwd_inner_microstep: 5684.07 | bwd_allreduce_microstep: 36.25 | step_microstep: 18.31
[2025-04-26 04:27:27,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.19 | bwd: 5720.37 | bwd_inner: 5684.07 | bwd_allreduce: 36.27 | step: 18.31

  0%|          | 127/41250 [18:58<99:12:59,  8.69s/it]
                                                      
{'loss': 0.2922, 'grad_norm': 2.4931514263153076, 'learning_rate': 4.103392568659128e-06, 'epoch': 0.03}

  0%|          | 127/41250 [18:58<99:12:59,  8.69s/it][2025-04-26 04:27:36,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-26 04:27:36,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5692.76 | bwd_inner_microstep: 5647.57 | bwd_allreduce_microstep: 45.14 | step_microstep: 18.14
[2025-04-26 04:27:36,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5692.77 | bwd_inner: 5647.57 | bwd_allreduce: 45.16 | step: 18.14

  0%|          | 128/41250 [19:07<98:54:36,  8.66s/it]
                                                      
{'loss': 0.3618, 'grad_norm': 2.7690155506134033, 'learning_rate': 4.1357027463651056e-06, 'epoch': 0.03}

  0%|          | 128/41250 [19:07<98:54:36,  8.66s/it][2025-04-26 04:27:44,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:27:44,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.28 | bwd_microstep: 5709.64 | bwd_inner_microstep: 5697.02 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.23
[2025-04-26 04:27:44,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.28 | bwd: 5709.65 | bwd_inner: 5697.02 | bwd_allreduce: 12.59 | step: 18.24

  0%|          | 129/41250 [19:15<98:51:53,  8.66s/it]
                                                      
{'loss': 0.3925, 'grad_norm': 3.5965769290924072, 'learning_rate': 4.168012924071083e-06, 'epoch': 0.03}

  0%|          | 129/41250 [19:15<98:51:53,  8.66s/it][2025-04-26 04:27:53,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:27:53,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.37 | bwd_microstep: 5760.47 | bwd_inner_microstep: 5637.86 | bwd_allreduce_microstep: 122.57 | step_microstep: 18.69
[2025-04-26 04:27:53,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.37 | bwd: 5760.49 | bwd_inner: 5637.86 | bwd_allreduce: 122.59 | step: 18.69

  0%|          | 130/41250 [19:24<98:55:16,  8.66s/it]
                                                      
{'loss': 0.3333, 'grad_norm': 2.4242610931396484, 'learning_rate': 4.2003231017770605e-06, 'epoch': 0.03}

  0%|          | 130/41250 [19:24<98:55:16,  8.66s/it][2025-04-26 04:28:02,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 04:28:02,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.47 | bwd_microstep: 5717.20 | bwd_inner_microstep: 5681.97 | bwd_allreduce_microstep: 35.19 | step_microstep: 18.57
[2025-04-26 04:28:02,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.47 | bwd: 5717.22 | bwd_inner: 5681.97 | bwd_allreduce: 35.21 | step: 18.57

  0%|          | 131/41250 [19:33<98:53:13,  8.66s/it]
                                                      
{'loss': 0.294, 'grad_norm': 2.8646066188812256, 'learning_rate': 4.2326332794830375e-06, 'epoch': 0.03}

  0%|          | 131/41250 [19:33<98:53:13,  8.66s/it][2025-04-26 04:28:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:28:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.68 | bwd_microstep: 5790.69 | bwd_inner_microstep: 5777.88 | bwd_allreduce_microstep: 12.77 | step_microstep: 17.86
[2025-04-26 04:28:10,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.68 | bwd: 5790.71 | bwd_inner: 5777.88 | bwd_allreduce: 12.78 | step: 17.86

  0%|          | 132/41250 [19:42<99:14:04,  8.69s/it]
                                                      
{'loss': 0.3909, 'grad_norm': 2.500959634780884, 'learning_rate': 4.2649434571890146e-06, 'epoch': 0.03}

  0%|          | 132/41250 [19:42<99:14:04,  8.69s/it][2025-04-26 04:28:19,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:28:19,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.98 | bwd_microstep: 5710.18 | bwd_inner_microstep: 5679.42 | bwd_allreduce_microstep: 30.71 | step_microstep: 17.82
[2025-04-26 04:28:19,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.98 | bwd: 5710.19 | bwd_inner: 5679.42 | bwd_allreduce: 30.73 | step: 17.83

  0%|          | 133/41250 [19:50<99:04:42,  8.67s/it]
                                                      
{'loss': 0.4528, 'grad_norm': 3.5860190391540527, 'learning_rate': 4.2972536348949925e-06, 'epoch': 0.03}

  0%|          | 133/41250 [19:50<99:04:42,  8.67s/it][2025-04-26 04:28:28,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 04:28:28,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.34 | bwd_microstep: 5779.15 | bwd_inner_microstep: 5766.79 | bwd_allreduce_microstep: 12.31 | step_microstep: 18.71
[2025-04-26 04:28:28,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.34 | bwd: 5779.16 | bwd_inner: 5766.79 | bwd_allreduce: 12.33 | step: 18.71

  0%|          | 134/41250 [19:59<99:20:11,  8.70s/it]
                                                      
{'loss': 0.2004, 'grad_norm': 1.3612302541732788, 'learning_rate': 4.3295638126009695e-06, 'epoch': 0.03}

  0%|          | 134/41250 [19:59<99:20:11,  8.70s/it][2025-04-26 04:28:36,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:28:36,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.16 | bwd_microstep: 5745.62 | bwd_inner_microstep: 5646.03 | bwd_allreduce_microstep: 99.55 | step_microstep: 18.20
[2025-04-26 04:28:36,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.16 | bwd: 5745.63 | bwd_inner: 5646.03 | bwd_allreduce: 99.56 | step: 18.21

  0%|          | 135/41250 [20:08<99:11:09,  8.68s/it]
                                                      
{'loss': 0.3385, 'grad_norm': 2.2139945030212402, 'learning_rate': 4.3618739903069465e-06, 'epoch': 0.03}

  0%|          | 135/41250 [20:08<99:11:09,  8.68s/it][2025-04-26 04:28:45,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:28:45,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.40 | bwd_microstep: 5713.01 | bwd_inner_microstep: 5679.81 | bwd_allreduce_microstep: 33.16 | step_microstep: 17.45
[2025-04-26 04:28:45,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.40 | bwd: 5713.02 | bwd_inner: 5679.81 | bwd_allreduce: 33.18 | step: 17.46

  0%|          | 136/41250 [20:16<99:03:23,  8.67s/it]
                                                      
{'loss': 0.2399, 'grad_norm': 2.334934711456299, 'learning_rate': 4.394184168012924e-06, 'epoch': 0.03}

  0%|          | 136/41250 [20:16<99:03:23,  8.67s/it][2025-04-26 04:28:54,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:28:54,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5794.42 | bwd_inner_microstep: 5634.52 | bwd_allreduce_microstep: 159.86 | step_microstep: 18.08
[2025-04-26 04:28:54,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5794.43 | bwd_inner: 5634.52 | bwd_allreduce: 159.88 | step: 18.08

  0%|          | 137/41250 [20:25<99:07:05,  8.68s/it]
                                                      
{'loss': 0.5224, 'grad_norm': 2.97477388381958, 'learning_rate': 4.426494345718902e-06, 'epoch': 0.03}

  0%|          | 137/41250 [20:25<99:07:05,  8.68s/it][2025-04-26 04:29:02,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:29:02,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.01 | bwd_microstep: 5750.25 | bwd_inner_microstep: 5694.92 | bwd_allreduce_microstep: 55.28 | step_microstep: 18.65
[2025-04-26 04:29:02,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.01 | bwd: 5750.26 | bwd_inner: 5694.92 | bwd_allreduce: 55.30 | step: 18.65

  0%|          | 138/41250 [20:34<99:07:21,  8.68s/it]
                                                      
{'loss': 0.2313, 'grad_norm': 1.8797695636749268, 'learning_rate': 4.458804523424879e-06, 'epoch': 0.03}

  0%|          | 138/41250 [20:34<99:07:21,  8.68s/it][2025-04-26 04:29:11,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:29:11,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.06 | bwd_microstep: 5765.22 | bwd_inner_microstep: 5752.52 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.40
[2025-04-26 04:29:11,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.06 | bwd: 5765.23 | bwd_inner: 5752.52 | bwd_allreduce: 12.67 | step: 18.40

  0%|          | 139/41250 [20:42<99:16:02,  8.69s/it]
                                                      
{'loss': 0.2306, 'grad_norm': 2.2754147052764893, 'learning_rate': 4.491114701130856e-06, 'epoch': 0.03}

  0%|          | 139/41250 [20:42<99:16:02,  8.69s/it][2025-04-26 04:29:20,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:29:20,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5797.83 | bwd_inner_microstep: 5653.33 | bwd_allreduce_microstep: 144.45 | step_microstep: 18.55
[2025-04-26 04:29:20,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5797.84 | bwd_inner: 5653.33 | bwd_allreduce: 144.47 | step: 18.56

  0%|          | 140/41250 [20:51<99:19:37,  8.70s/it]
                                                      
{'loss': 0.1567, 'grad_norm': 1.3689626455307007, 'learning_rate': 4.523424878836834e-06, 'epoch': 0.03}

  0%|          | 140/41250 [20:51<99:19:37,  8.70s/it][2025-04-26 04:29:28,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 04:29:28,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.77 | bwd_microstep: 5700.20 | bwd_inner_microstep: 5635.73 | bwd_allreduce_microstep: 64.42 | step_microstep: 19.55
[2025-04-26 04:29:28,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.77 | bwd: 5700.21 | bwd_inner: 5635.73 | bwd_allreduce: 64.44 | step: 19.55

  0%|          | 141/41250 [21:00<99:03:07,  8.67s/it]
                                                      
{'loss': 0.2389, 'grad_norm': 1.6752853393554688, 'learning_rate': 4.555735056542811e-06, 'epoch': 0.03}

  0%|          | 141/41250 [21:00<99:03:07,  8.67s/it][2025-04-26 04:29:37,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 04:29:37,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5698.79 | bwd_inner_microstep: 5686.05 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.35
[2025-04-26 04:29:37,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5698.80 | bwd_inner: 5686.05 | bwd_allreduce: 12.72 | step: 19.35

  0%|          | 142/41250 [21:08<98:53:40,  8.66s/it]
                                                      
{'loss': 0.3096, 'grad_norm': 3.655726194381714, 'learning_rate': 4.588045234248788e-06, 'epoch': 0.03}

  0%|          | 142/41250 [21:08<98:53:40,  8.66s/it][2025-04-26 04:29:46,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 04:29:46,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.95 | bwd_microstep: 5753.35 | bwd_inner_microstep: 5684.95 | bwd_allreduce_microstep: 68.35 | step_microstep: 19.09
[2025-04-26 04:29:46,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.95 | bwd: 5753.36 | bwd_inner: 5684.95 | bwd_allreduce: 68.37 | step: 19.10

  0%|          | 143/41250 [21:17<98:59:08,  8.67s/it]
                                                      
{'loss': 0.343, 'grad_norm': 2.457862615585327, 'learning_rate': 4.620355411954766e-06, 'epoch': 0.03}

  0%|          | 143/41250 [21:17<98:59:08,  8.67s/it][2025-04-26 04:29:54,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:29:54,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.46 | bwd_microstep: 5758.70 | bwd_inner_microstep: 5655.71 | bwd_allreduce_microstep: 102.94 | step_microstep: 17.98
[2025-04-26 04:29:54,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.46 | bwd: 5758.71 | bwd_inner: 5655.71 | bwd_allreduce: 102.96 | step: 17.98

  0%|          | 144/41250 [21:26<98:58:43,  8.67s/it]
                                                      
{'loss': 0.3771, 'grad_norm': 2.874499797821045, 'learning_rate': 4.652665589660743e-06, 'epoch': 0.03}

  0%|          | 144/41250 [21:26<98:58:43,  8.67s/it][2025-04-26 04:30:03,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:30:03,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5751.14 | bwd_inner_microstep: 5693.64 | bwd_allreduce_microstep: 57.45 | step_microstep: 18.12
[2025-04-26 04:30:03,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5751.15 | bwd_inner: 5693.64 | bwd_allreduce: 57.46 | step: 18.12

  0%|          | 145/41250 [21:34<99:03:29,  8.68s/it]
                                                      
{'loss': 0.2071, 'grad_norm': 3.5672600269317627, 'learning_rate': 4.68497576736672e-06, 'epoch': 0.04}

  0%|          | 145/41250 [21:34<99:03:29,  8.68s/it][2025-04-26 04:30:12,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 1.11
[2025-04-26 04:30:12,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5741.24 | bwd_inner_microstep: 5699.78 | bwd_allreduce_microstep: 41.42 | step_microstep: 18.41
[2025-04-26 04:30:12,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5741.25 | bwd_inner: 5699.78 | bwd_allreduce: 41.43 | step: 18.42

  0%|          | 146/41250 [21:43<99:00:37,  8.67s/it]
                                                      
{'loss': 0.2528, 'grad_norm': 4.147963523864746, 'learning_rate': 4.717285945072698e-06, 'epoch': 0.04}

  0%|          | 146/41250 [21:43<99:00:37,  8.67s/it][2025-04-26 04:30:20,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:30:20,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.53 | bwd_microstep: 5749.23 | bwd_inner_microstep: 5680.08 | bwd_allreduce_microstep: 69.11 | step_microstep: 18.28
[2025-04-26 04:30:20,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5749.24 | bwd_inner: 5680.08 | bwd_allreduce: 69.13 | step: 18.29

  0%|          | 147/41250 [21:52<99:01:55,  8.67s/it]
                                                      
{'loss': 0.282, 'grad_norm': 1.940940022468567, 'learning_rate': 4.749596122778676e-06, 'epoch': 0.04}

  0%|          | 147/41250 [21:52<99:01:55,  8.67s/it][2025-04-26 04:30:29,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:30:29,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.08 | bwd_microstep: 5714.56 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.51
[2025-04-26 04:30:29,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.08 | bwd: 5714.57 | bwd_inner: 5701.94 | bwd_allreduce: 12.60 | step: 18.51

  0%|          | 148/41250 [22:00<98:56:06,  8.67s/it]
                                                      
{'loss': 0.2574, 'grad_norm': 1.9515044689178467, 'learning_rate': 4.781906300484653e-06, 'epoch': 0.04}

  0%|          | 148/41250 [22:00<98:56:06,  8.67s/it][2025-04-26 04:30:38,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:30:38,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.31 | bwd_microstep: 5710.83 | bwd_inner_microstep: 5655.11 | bwd_allreduce_microstep: 55.67 | step_microstep: 18.20
[2025-04-26 04:30:38,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.31 | bwd: 5710.84 | bwd_inner: 5655.11 | bwd_allreduce: 55.69 | step: 18.20

  0%|          | 149/41250 [22:09<98:49:32,  8.66s/it]
                                                      
{'loss': 0.2261, 'grad_norm': 1.485133409500122, 'learning_rate': 4.81421647819063e-06, 'epoch': 0.04}

  0%|          | 149/41250 [22:09<98:49:32,  8.66s/it][2025-04-26 04:30:46,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:30:46,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.08 | bwd_microstep: 5699.50 | bwd_inner_microstep: 5686.87 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.05
[2025-04-26 04:30:46,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.08 | bwd: 5699.52 | bwd_inner: 5686.87 | bwd_allreduce: 12.60 | step: 19.06

  0%|          | 150/41250 [22:18<98:45:41,  8.65s/it]
                                                      
{'loss': 0.5897, 'grad_norm': 3.66837477684021, 'learning_rate': 4.846526655896608e-06, 'epoch': 0.04}

  0%|          | 150/41250 [22:18<98:45:41,  8.65s/it][2025-04-26 04:30:55,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:30:55,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.70 | bwd_microstep: 5776.42 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 115.45 | step_microstep: 17.67
[2025-04-26 04:30:55,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.70 | bwd: 5776.43 | bwd_inner: 5660.93 | bwd_allreduce: 115.47 | step: 17.67

  0%|          | 151/41250 [22:26<98:51:46,  8.66s/it]
                                                      
{'loss': 0.3587, 'grad_norm': 3.111776828765869, 'learning_rate': 4.878836833602585e-06, 'epoch': 0.04}

  0%|          | 151/41250 [22:26<98:51:46,  8.66s/it][2025-04-26 04:31:04,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:31:04,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5770.43 | bwd_inner_microstep: 5663.95 | bwd_allreduce_microstep: 106.44 | step_microstep: 17.99
[2025-04-26 04:31:04,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5770.44 | bwd_inner: 5663.95 | bwd_allreduce: 106.45 | step: 17.99

  0%|          | 152/41250 [22:35<98:57:31,  8.67s/it]
                                                      
{'loss': 0.2403, 'grad_norm': 2.071990489959717, 'learning_rate': 4.911147011308562e-06, 'epoch': 0.04}

  0%|          | 152/41250 [22:35<98:57:31,  8.67s/it][2025-04-26 04:31:12,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:31:12,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.61 | bwd_microstep: 5754.86 | bwd_inner_microstep: 5706.38 | bwd_allreduce_microstep: 48.43 | step_microstep: 17.65
[2025-04-26 04:31:12,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.61 | bwd: 5754.87 | bwd_inner: 5706.38 | bwd_allreduce: 48.45 | step: 17.66

  0%|          | 153/41250 [22:44<99:03:11,  8.68s/it]
                                                      
{'loss': 0.2781, 'grad_norm': 2.0794529914855957, 'learning_rate': 4.94345718901454e-06, 'epoch': 0.04}

  0%|          | 153/41250 [22:44<99:03:11,  8.68s/it][2025-04-26 04:31:21,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.84 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:31:21,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5713.85 | bwd_inner_microstep: 5701.17 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.45
[2025-04-26 04:31:21,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5713.86 | bwd_inner: 5701.17 | bwd_allreduce: 12.65 | step: 17.45

  0%|          | 154/41250 [22:52<98:57:03,  8.67s/it]
                                                      
{'loss': 0.3938, 'grad_norm': 2.100419759750366, 'learning_rate': 4.975767366720518e-06, 'epoch': 0.04}

  0%|          | 154/41250 [22:52<98:57:03,  8.67s/it][2025-04-26 04:31:30,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 04:31:30,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.41 | bwd_microstep: 5712.56 | bwd_inner_microstep: 5699.79 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.22
[2025-04-26 04:31:30,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.41 | bwd: 5712.57 | bwd_inner: 5699.79 | bwd_allreduce: 12.74 | step: 18.22

  0%|          | 155/41250 [23:01<98:49:29,  8.66s/it]
                                                      
{'loss': 0.2664, 'grad_norm': 2.7592861652374268, 'learning_rate': 5.008077544426495e-06, 'epoch': 0.04}

  0%|          | 155/41250 [23:01<98:49:29,  8.66s/it][2025-04-26 04:31:38,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:31:38,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.01 | bwd_microstep: 5777.36 | bwd_inner_microstep: 5692.56 | bwd_allreduce_microstep: 84.77 | step_microstep: 18.47
[2025-04-26 04:31:38,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.01 | bwd: 5777.38 | bwd_inner: 5692.56 | bwd_allreduce: 84.78 | step: 18.47

  0%|          | 156/41250 [23:10<98:59:43,  8.67s/it]
                                                      
{'loss': 0.4093, 'grad_norm': 3.331573963165283, 'learning_rate': 5.040387722132473e-06, 'epoch': 0.04}

  0%|          | 156/41250 [23:10<98:59:43,  8.67s/it][2025-04-26 04:31:47,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:31:47,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5702.58 | bwd_allreduce_microstep: 12.85 | step_microstep: 17.71
[2025-04-26 04:31:47,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5715.48 | bwd_inner: 5702.58 | bwd_allreduce: 12.86 | step: 17.71

  0%|          | 157/41250 [23:18<98:55:28,  8.67s/it]
                                                      
{'loss': 0.2213, 'grad_norm': 1.1002987623214722, 'learning_rate': 5.072697899838449e-06, 'epoch': 0.04}

  0%|          | 157/41250 [23:18<98:55:28,  8.67s/it][2025-04-26 04:31:56,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.94 | optimizer_step: 0.97
[2025-04-26 04:31:56,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.77 | bwd_microstep: 5782.93 | bwd_inner_microstep: 5657.61 | bwd_allreduce_microstep: 125.27 | step_microstep: 18.04
[2025-04-26 04:31:56,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.77 | bwd: 5782.94 | bwd_inner: 5657.61 | bwd_allreduce: 125.29 | step: 18.05

  0%|          | 158/41250 [23:27<99:01:59,  8.68s/it]
                                                      
{'loss': 0.4707, 'grad_norm': 2.5513243675231934, 'learning_rate': 5.105008077544427e-06, 'epoch': 0.04}

  0%|          | 158/41250 [23:27<99:01:59,  8.68s/it][2025-04-26 04:32:05,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:32:05,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.55 | bwd_microstep: 5787.28 | bwd_inner_microstep: 5662.77 | bwd_allreduce_microstep: 124.47 | step_microstep: 18.12
[2025-04-26 04:32:05,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.55 | bwd: 5787.29 | bwd_inner: 5662.77 | bwd_allreduce: 124.48 | step: 18.12

  0%|          | 159/41250 [23:36<99:07:17,  8.68s/it]
                                                      
{'loss': 0.2079, 'grad_norm': 1.6595091819763184, 'learning_rate': 5.137318255250404e-06, 'epoch': 0.04}

  0%|          | 159/41250 [23:36<99:07:17,  8.68s/it][2025-04-26 04:32:13,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 04:32:13,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5711.01 | bwd_inner_microstep: 5698.33 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.78
[2025-04-26 04:32:13,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5711.02 | bwd_inner: 5698.33 | bwd_allreduce: 12.66 | step: 18.79

  0%|          | 160/41250 [23:44<98:57:09,  8.67s/it]
                                                      
{'loss': 0.1197, 'grad_norm': 1.249398112297058, 'learning_rate': 5.169628432956382e-06, 'epoch': 0.04}

  0%|          | 160/41250 [23:44<98:57:09,  8.67s/it][2025-04-26 04:32:22,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.01
[2025-04-26 04:32:22,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.27 | bwd_microstep: 5840.01 | bwd_inner_microstep: 5708.28 | bwd_allreduce_microstep: 131.69 | step_microstep: 18.33
[2025-04-26 04:32:22,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.27 | bwd: 5840.03 | bwd_inner: 5708.28 | bwd_allreduce: 131.70 | step: 18.33

  0%|          | 161/41250 [23:53<99:19:07,  8.70s/it]
                                                      
{'loss': 0.2567, 'grad_norm': 2.458421468734741, 'learning_rate': 5.20193861066236e-06, 'epoch': 0.04}

  0%|          | 161/41250 [23:53<99:19:07,  8.70s/it][2025-04-26 04:32:31,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:32:31,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.37 | bwd_microstep: 5700.74 | bwd_inner_microstep: 5647.38 | bwd_allreduce_microstep: 53.32 | step_microstep: 18.20
[2025-04-26 04:32:31,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.37 | bwd: 5700.75 | bwd_inner: 5647.38 | bwd_allreduce: 53.33 | step: 18.20

  0%|          | 162/41250 [24:02<99:01:18,  8.68s/it]
                                                      
{'loss': 0.2728, 'grad_norm': 1.6445021629333496, 'learning_rate': 5.234248788368337e-06, 'epoch': 0.04}

  0%|          | 162/41250 [24:02<99:01:18,  8.68s/it][2025-04-26 04:32:39,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 04:32:39,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5719.38 | bwd_inner_microstep: 5706.92 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.27
[2025-04-26 04:32:39,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5719.39 | bwd_inner: 5706.92 | bwd_allreduce: 12.43 | step: 18.27

  0%|          | 163/41250 [24:10<98:56:19,  8.67s/it]
                                                      
{'loss': 0.2987, 'grad_norm': 2.376328468322754, 'learning_rate': 5.2665589660743146e-06, 'epoch': 0.04}

  0%|          | 163/41250 [24:10<98:56:19,  8.67s/it][2025-04-26 04:32:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:32:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.99 | bwd_microstep: 5751.20 | bwd_inner_microstep: 5640.54 | bwd_allreduce_microstep: 110.61 | step_microstep: 18.04
[2025-04-26 04:32:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.99 | bwd: 5751.21 | bwd_inner: 5640.54 | bwd_allreduce: 110.63 | step: 18.04

  0%|          | 164/41250 [24:19<98:54:11,  8.67s/it]
                                                      
{'loss': 0.3648, 'grad_norm': 3.390883445739746, 'learning_rate': 5.298869143780291e-06, 'epoch': 0.04}

  0%|          | 164/41250 [24:19<98:54:11,  8.67s/it][2025-04-26 04:32:56,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 04:32:56,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.89 | bwd_microstep: 5703.04 | bwd_inner_microstep: 5690.38 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.08
[2025-04-26 04:32:56,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.89 | bwd: 5703.05 | bwd_inner: 5690.38 | bwd_allreduce: 12.63 | step: 19.09

  0%|          | 165/41250 [24:28<98:47:49,  8.66s/it]
                                                      
{'loss': 0.23, 'grad_norm': 1.8730323314666748, 'learning_rate': 5.331179321486269e-06, 'epoch': 0.04}

  0%|          | 165/41250 [24:28<98:47:49,  8.66s/it][2025-04-26 04:33:05,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:33:05,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.60 | bwd_microstep: 5764.37 | bwd_inner_microstep: 5653.36 | bwd_allreduce_microstep: 110.97 | step_microstep: 17.74
[2025-04-26 04:33:05,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.60 | bwd: 5764.39 | bwd_inner: 5653.36 | bwd_allreduce: 110.99 | step: 17.74

  0%|          | 166/41250 [24:36<98:52:08,  8.66s/it]
                                                      
{'loss': 0.3847, 'grad_norm': 2.2566208839416504, 'learning_rate': 5.363489499192246e-06, 'epoch': 0.04}

  0%|          | 166/41250 [24:36<98:52:08,  8.66s/it][2025-04-26 04:33:14,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-26 04:33:14,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.51 | bwd_microstep: 5742.71 | bwd_inner_microstep: 5693.66 | bwd_allreduce_microstep: 49.01 | step_microstep: 17.95
[2025-04-26 04:33:14,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.51 | bwd: 5742.73 | bwd_inner: 5693.66 | bwd_allreduce: 49.03 | step: 17.96

  0%|          | 167/41250 [24:45<98:55:54,  8.67s/it]
                                                      
{'loss': 0.2946, 'grad_norm': 2.527597427368164, 'learning_rate': 5.3957996768982236e-06, 'epoch': 0.04}

  0%|          | 167/41250 [24:45<98:55:54,  8.67s/it][2025-04-26 04:33:22,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:33:22,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.03 | bwd_microstep: 5695.41 | bwd_inner_microstep: 5645.10 | bwd_allreduce_microstep: 50.26 | step_microstep: 18.46
[2025-04-26 04:33:22,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.03 | bwd: 5695.42 | bwd_inner: 5645.10 | bwd_allreduce: 50.28 | step: 18.46

  0%|          | 168/41250 [24:54<98:43:16,  8.65s/it]
                                                      
{'loss': 0.2821, 'grad_norm': 2.0323007106781006, 'learning_rate': 5.4281098546042014e-06, 'epoch': 0.04}

  0%|          | 168/41250 [24:54<98:43:16,  8.65s/it][2025-04-26 04:33:31,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:33:31,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.87 | bwd_microstep: 5714.94 | bwd_inner_microstep: 5702.31 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.47
[2025-04-26 04:33:31,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.87 | bwd: 5714.95 | bwd_inner: 5702.31 | bwd_allreduce: 12.60 | step: 18.48

  0%|          | 169/41250 [25:02<98:43:27,  8.65s/it]
                                                      
{'loss': 0.2452, 'grad_norm': 1.92365562915802, 'learning_rate': 5.4604200323101785e-06, 'epoch': 0.04}

  0%|          | 169/41250 [25:02<98:43:27,  8.65s/it][2025-04-26 04:33:40,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.09 | optimizer_step: 1.23
[2025-04-26 04:33:40,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.26 | bwd_microstep: 5793.13 | bwd_inner_microstep: 5779.95 | bwd_allreduce_microstep: 13.13 | step_microstep: 20.23
[2025-04-26 04:33:40,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.26 | bwd: 5793.15 | bwd_inner: 5779.94 | bwd_allreduce: 13.16 | step: 20.23

  0%|          | 170/41250 [25:11<99:09:58,  8.69s/it]
                                                      
{'loss': 0.3264, 'grad_norm': 1.5500853061676025, 'learning_rate': 5.492730210016155e-06, 'epoch': 0.04}

  0%|          | 170/41250 [25:11<99:09:58,  8.69s/it][2025-04-26 04:33:49,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.22 | optimizer_step: 1.04
[2025-04-26 04:33:49,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5738.20 | bwd_inner_microstep: 5650.64 | bwd_allreduce_microstep: 87.50 | step_microstep: 19.53
[2025-04-26 04:33:49,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5738.21 | bwd_inner: 5650.64 | bwd_allreduce: 87.53 | step: 19.53

  0%|          | 171/41250 [25:20<99:02:53,  8.68s/it]
                                                      
{'loss': 0.2885, 'grad_norm': 3.847388982772827, 'learning_rate': 5.5250403877221325e-06, 'epoch': 0.04}

  0%|          | 171/41250 [25:20<99:02:53,  8.68s/it][2025-04-26 04:33:57,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 04:33:57,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.39 | bwd_microstep: 5751.76 | bwd_inner_microstep: 5658.08 | bwd_allreduce_microstep: 93.63 | step_microstep: 18.65
[2025-04-26 04:33:57,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.39 | bwd: 5751.77 | bwd_inner: 5658.08 | bwd_allreduce: 93.65 | step: 18.66

  0%|          | 172/41250 [25:28<98:57:49,  8.67s/it]
                                                      
{'loss': 0.3262, 'grad_norm': 2.1804168224334717, 'learning_rate': 5.5573505654281104e-06, 'epoch': 0.04}

  0%|          | 172/41250 [25:28<98:57:49,  8.67s/it][2025-04-26 04:34:06,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.20
[2025-04-26 04:34:06,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.14 | bwd_microstep: 5752.01 | bwd_inner_microstep: 5682.84 | bwd_allreduce_microstep: 69.12 | step_microstep: 19.08
[2025-04-26 04:34:06,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.14 | bwd: 5752.02 | bwd_inner: 5682.84 | bwd_allreduce: 69.14 | step: 19.08

  0%|          | 173/41250 [25:37<98:56:14,  8.67s/it]
                                                      
{'loss': 0.2279, 'grad_norm': 1.6069263219833374, 'learning_rate': 5.5896607431340875e-06, 'epoch': 0.04}

  0%|          | 173/41250 [25:37<98:56:14,  8.67s/it][2025-04-26 04:34:15,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.00
[2025-04-26 04:34:15,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.91 | bwd_microstep: 5893.92 | bwd_inner_microstep: 5626.01 | bwd_allreduce_microstep: 267.85 | step_microstep: 19.18
[2025-04-26 04:34:15,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.91 | bwd: 5893.94 | bwd_inner: 5626.01 | bwd_allreduce: 267.88 | step: 19.19

  0%|          | 174/41250 [25:46<99:22:40,  8.71s/it]
                                                      
{'loss': 0.2048, 'grad_norm': 1.3725641965866089, 'learning_rate': 5.621970920840065e-06, 'epoch': 0.04}

  0%|          | 174/41250 [25:46<99:22:40,  8.71s/it][2025-04-26 04:34:23,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:34:23,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.19 | bwd_microstep: 5690.86 | bwd_inner_microstep: 5639.15 | bwd_allreduce_microstep: 51.66 | step_microstep: 18.51
[2025-04-26 04:34:23,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.19 | bwd: 5690.87 | bwd_inner: 5639.15 | bwd_allreduce: 51.68 | step: 18.52

  0%|          | 175/41250 [25:54<98:58:20,  8.67s/it]
                                                      
{'loss': 0.2856, 'grad_norm': 1.7269928455352783, 'learning_rate': 5.654281098546042e-06, 'epoch': 0.04}

  0%|          | 175/41250 [25:54<98:58:20,  8.67s/it][2025-04-26 04:34:32,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:34:32,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.43 | bwd_microstep: 5750.19 | bwd_inner_microstep: 5650.10 | bwd_allreduce_microstep: 100.04 | step_microstep: 17.84
[2025-04-26 04:34:32,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.43 | bwd: 5750.20 | bwd_inner: 5650.10 | bwd_allreduce: 100.06 | step: 17.84

  0%|          | 176/41250 [26:03<98:54:31,  8.67s/it]
                                                      
{'loss': 0.3479, 'grad_norm': 1.8709756135940552, 'learning_rate': 5.68659127625202e-06, 'epoch': 0.04}

  0%|          | 176/41250 [26:03<98:54:31,  8.67s/it][2025-04-26 04:34:41,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 04:34:41,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5840.00 | bwd_inner_microstep: 5694.57 | bwd_allreduce_microstep: 145.39 | step_microstep: 18.82
[2025-04-26 04:34:41,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5840.02 | bwd_inner: 5694.57 | bwd_allreduce: 145.41 | step: 18.82

  0%|          | 177/41250 [26:12<99:14:28,  8.70s/it]
                                                      
{'loss': 0.1724, 'grad_norm': 1.4564487934112549, 'learning_rate': 5.7189014539579965e-06, 'epoch': 0.04}

  0%|          | 177/41250 [26:12<99:14:28,  8.70s/it][2025-04-26 04:34:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 04:34:49,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.36 | bwd_microstep: 5746.48 | bwd_inner_microstep: 5646.96 | bwd_allreduce_microstep: 99.47 | step_microstep: 17.73
[2025-04-26 04:34:49,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.36 | bwd: 5746.49 | bwd_inner: 5646.96 | bwd_allreduce: 99.49 | step: 17.73

  0%|          | 178/41250 [26:21<99:04:26,  8.68s/it]
                                                      
{'loss': 0.2057, 'grad_norm': 1.4246015548706055, 'learning_rate': 5.751211631663974e-06, 'epoch': 0.04}

  0%|          | 178/41250 [26:21<99:04:26,  8.68s/it][2025-04-26 04:34:58,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 04:34:58,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.04 | bwd_microstep: 5720.14 | bwd_inner_microstep: 5707.31 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.25
[2025-04-26 04:34:58,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.04 | bwd: 5720.16 | bwd_inner: 5707.31 | bwd_allreduce: 12.81 | step: 18.25

  0%|          | 179/41250 [26:29<99:00:27,  8.68s/it]
                                                      
{'loss': 0.389, 'grad_norm': 2.2871415615081787, 'learning_rate': 5.783521809369952e-06, 'epoch': 0.04}

  0%|          | 179/41250 [26:29<99:00:27,  8.68s/it][2025-04-26 04:35:07,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:35:07,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.14 | bwd_microstep: 5700.29 | bwd_inner_microstep: 5633.95 | bwd_allreduce_microstep: 66.30 | step_microstep: 17.93
[2025-04-26 04:35:07,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.14 | bwd: 5700.30 | bwd_inner: 5633.95 | bwd_allreduce: 66.31 | step: 17.93

  0%|          | 180/41250 [26:38<98:46:55,  8.66s/it]
                                                      
{'loss': 0.3663, 'grad_norm': 3.562285900115967, 'learning_rate': 5.815831987075929e-06, 'epoch': 0.04}

  0%|          | 180/41250 [26:38<98:46:55,  8.66s/it][2025-04-26 04:35:15,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:35:15,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.65 | bwd_microstep: 5682.52 | bwd_inner_microstep: 5649.50 | bwd_allreduce_microstep: 32.98 | step_microstep: 17.76
[2025-04-26 04:35:15,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.65 | bwd: 5682.53 | bwd_inner: 5649.50 | bwd_allreduce: 32.99 | step: 17.77

  0%|          | 181/41250 [26:46<98:34:28,  8.64s/it]
                                                      
{'loss': 0.1978, 'grad_norm': 1.4831583499908447, 'learning_rate': 5.848142164781907e-06, 'epoch': 0.04}

  0%|          | 181/41250 [26:46<98:34:28,  8.64s/it][2025-04-26 04:35:24,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:35:24,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.51 | bwd_microstep: 5673.79 | bwd_inner_microstep: 5653.24 | bwd_allreduce_microstep: 20.51 | step_microstep: 17.77
[2025-04-26 04:35:24,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.51 | bwd: 5673.80 | bwd_inner: 5653.24 | bwd_allreduce: 20.53 | step: 17.78

  0%|          | 182/41250 [26:55<98:22:20,  8.62s/it]
                                                      
{'loss': 0.2802, 'grad_norm': 2.1121296882629395, 'learning_rate': 5.880452342487884e-06, 'epoch': 0.04}

  0%|          | 182/41250 [26:55<98:22:20,  8.62s/it][2025-04-26 04:35:32,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 04:35:32,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.60 | bwd_microstep: 5760.06 | bwd_inner_microstep: 5696.06 | bwd_allreduce_microstep: 63.96 | step_microstep: 18.95
[2025-04-26 04:35:32,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.60 | bwd: 5760.08 | bwd_inner: 5696.06 | bwd_allreduce: 63.98 | step: 18.95

  0%|          | 183/41250 [27:04<98:37:19,  8.65s/it]
                                                      
{'loss': 0.2555, 'grad_norm': 1.9859963655471802, 'learning_rate': 5.912762520193862e-06, 'epoch': 0.04}

  0%|          | 183/41250 [27:04<98:37:19,  8.65s/it][2025-04-26 04:35:41,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:35:41,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.52 | bwd_microstep: 5697.70 | bwd_inner_microstep: 5649.54 | bwd_allreduce_microstep: 48.12 | step_microstep: 18.54
[2025-04-26 04:35:41,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.52 | bwd: 5697.71 | bwd_inner: 5649.54 | bwd_allreduce: 48.13 | step: 18.54

  0%|          | 184/41250 [27:12<98:29:29,  8.63s/it]
                                                      
{'loss': 0.3527, 'grad_norm': 2.3862035274505615, 'learning_rate': 5.945072697899838e-06, 'epoch': 0.04}

  0%|          | 184/41250 [27:12<98:29:29,  8.63s/it][2025-04-26 04:35:50,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:35:50,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.83 | bwd_microstep: 5776.99 | bwd_inner_microstep: 5764.38 | bwd_allreduce_microstep: 12.56 | step_microstep: 17.83
[2025-04-26 04:35:50,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.83 | bwd: 5777.00 | bwd_inner: 5764.38 | bwd_allreduce: 12.58 | step: 17.83

  0%|          | 185/41250 [27:21<98:55:22,  8.67s/it]
                                                      
{'loss': 0.3165, 'grad_norm': 1.6179020404815674, 'learning_rate': 5.977382875605816e-06, 'epoch': 0.04}

  0%|          | 185/41250 [27:21<98:55:22,  8.67s/it][2025-04-26 04:35:59,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:35:59,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.41 | bwd_microstep: 5725.37 | bwd_inner_microstep: 5696.53 | bwd_allreduce_microstep: 28.79 | step_microstep: 18.30
[2025-04-26 04:35:59,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.41 | bwd: 5725.38 | bwd_inner: 5696.53 | bwd_allreduce: 28.81 | step: 18.30

  0%|          | 186/41250 [27:30<98:52:26,  8.67s/it]
                                                      
{'loss': 0.4014, 'grad_norm': 2.900904655456543, 'learning_rate': 6.009693053311794e-06, 'epoch': 0.05}

  0%|          | 186/41250 [27:30<98:52:26,  8.67s/it][2025-04-26 04:36:07,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 04:36:07,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.04 | bwd_microstep: 5760.18 | bwd_inner_microstep: 5645.84 | bwd_allreduce_microstep: 114.30 | step_microstep: 18.99
[2025-04-26 04:36:07,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.04 | bwd: 5760.20 | bwd_inner: 5645.84 | bwd_allreduce: 114.32 | step: 18.99

  0%|          | 187/41250 [27:38<98:53:06,  8.67s/it]
                                                      
{'loss': 0.3134, 'grad_norm': 1.917681097984314, 'learning_rate': 6.042003231017771e-06, 'epoch': 0.05}

  0%|          | 187/41250 [27:38<98:53:06,  8.67s/it][2025-04-26 04:36:16,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 04:36:16,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.57 | bwd_microstep: 5736.12 | bwd_inner_microstep: 5708.00 | bwd_allreduce_microstep: 28.09 | step_microstep: 18.29
[2025-04-26 04:36:16,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.57 | bwd: 5736.14 | bwd_inner: 5708.00 | bwd_allreduce: 28.10 | step: 18.29

  0%|          | 188/41250 [27:47<98:53:19,  8.67s/it]
                                                      
{'loss': 0.3347, 'grad_norm': 3.308549642562866, 'learning_rate': 6.074313408723749e-06, 'epoch': 0.05}

  0%|          | 188/41250 [27:47<98:53:19,  8.67s/it][2025-04-26 04:36:25,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.10 | optimizer_step: 1.19
[2025-04-26 04:36:25,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.00 | bwd_microstep: 5761.43 | bwd_inner_microstep: 5659.45 | bwd_allreduce_microstep: 101.92 | step_microstep: 19.57
[2025-04-26 04:36:25,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.00 | bwd: 5761.45 | bwd_inner: 5659.45 | bwd_allreduce: 101.95 | step: 19.57

  0%|          | 189/41250 [27:56<98:56:30,  8.67s/it]
                                                      
{'loss': 0.2523, 'grad_norm': 1.7471106052398682, 'learning_rate': 6.106623586429726e-06, 'epoch': 0.05}

  0%|          | 189/41250 [27:56<98:56:30,  8.67s/it][2025-04-26 04:36:33,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:36:33,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5706.41 | bwd_inner_microstep: 5654.77 | bwd_allreduce_microstep: 51.61 | step_microstep: 17.82
[2025-04-26 04:36:33,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5706.43 | bwd_inner: 5654.77 | bwd_allreduce: 51.62 | step: 17.82

  0%|          | 190/41250 [28:04<98:45:50,  8.66s/it]
                                                      
{'loss': 0.4295, 'grad_norm': 1.9500067234039307, 'learning_rate': 6.138933764135703e-06, 'epoch': 0.05}

  0%|          | 190/41250 [28:04<98:45:50,  8.66s/it][2025-04-26 04:36:42,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:36:42,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.63 | bwd_microstep: 5709.72 | bwd_inner_microstep: 5696.71 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.39
[2025-04-26 04:36:42,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.63 | bwd: 5709.74 | bwd_inner: 5696.71 | bwd_allreduce: 12.98 | step: 19.39

  0%|          | 191/41250 [28:13<98:42:04,  8.65s/it]
                                                      
{'loss': 0.365, 'grad_norm': 2.055774688720703, 'learning_rate': 6.17124394184168e-06, 'epoch': 0.05}

  0%|          | 191/41250 [28:13<98:42:04,  8.65s/it][2025-04-26 04:36:51,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:36:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2942.76 | bwd_microstep: 5886.06 | bwd_inner_microstep: 5873.52 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.04
[2025-04-26 04:36:51,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2942.76 | bwd: 5886.07 | bwd_inner: 5873.52 | bwd_allreduce: 12.51 | step: 18.05

  0%|          | 192/41250 [28:22<99:34:38,  8.73s/it]
                                                      
{'loss': 0.2142, 'grad_norm': 1.1912339925765991, 'learning_rate': 6.203554119547658e-06, 'epoch': 0.05}

  0%|          | 192/41250 [28:22<99:34:38,  8.73s/it][2025-04-26 04:36:59,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 04:36:59,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.57 | bwd_microstep: 5695.74 | bwd_inner_microstep: 5656.78 | bwd_allreduce_microstep: 38.92 | step_microstep: 18.97
[2025-04-26 04:36:59,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.57 | bwd: 5695.75 | bwd_inner: 5656.78 | bwd_allreduce: 38.93 | step: 18.98

  0%|          | 193/41250 [28:31<99:10:10,  8.70s/it]
                                                      
{'loss': 0.2566, 'grad_norm': 1.6277471780776978, 'learning_rate': 6.235864297253636e-06, 'epoch': 0.05}

  0%|          | 193/41250 [28:31<99:10:10,  8.70s/it][2025-04-26 04:37:08,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 04:37:08,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5703.42 | bwd_inner_microstep: 5657.82 | bwd_allreduce_microstep: 45.55 | step_microstep: 18.56
[2025-04-26 04:37:08,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5703.44 | bwd_inner: 5657.82 | bwd_allreduce: 45.57 | step: 18.56

  0%|          | 194/41250 [28:39<98:55:25,  8.67s/it]
                                                      
{'loss': 0.1793, 'grad_norm': 1.4448039531707764, 'learning_rate': 6.268174474959613e-06, 'epoch': 0.05}

  0%|          | 194/41250 [28:39<98:55:25,  8.67s/it][2025-04-26 04:37:17,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-26 04:37:17,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.12 | bwd_microstep: 5728.28 | bwd_inner_microstep: 5715.35 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.16
[2025-04-26 04:37:17,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.12 | bwd: 5728.29 | bwd_inner: 5715.35 | bwd_allreduce: 12.90 | step: 19.16

  0%|          | 195/41250 [28:48<98:54:10,  8.67s/it]
                                                      
{'loss': 0.3852, 'grad_norm': 2.1049540042877197, 'learning_rate': 6.300484652665591e-06, 'epoch': 0.05}

  0%|          | 195/41250 [28:48<98:54:10,  8.67s/it][2025-04-26 04:37:25,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:37:25,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5726.35 | bwd_inner_microstep: 5713.58 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.75
[2025-04-26 04:37:25,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5726.37 | bwd_inner: 5713.58 | bwd_allreduce: 12.74 | step: 18.75

  0%|          | 196/41250 [28:56<98:50:21,  8.67s/it]
                                                      
{'loss': 0.32, 'grad_norm': 1.7228730916976929, 'learning_rate': 6.332794830371568e-06, 'epoch': 0.05}

  0%|          | 196/41250 [28:56<98:50:21,  8.67s/it][2025-04-26 04:37:34,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.08
[2025-04-26 04:37:34,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5789.63 | bwd_inner_microstep: 5679.28 | bwd_allreduce_microstep: 110.30 | step_microstep: 19.41
[2025-04-26 04:37:34,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5789.64 | bwd_inner: 5679.28 | bwd_allreduce: 110.32 | step: 19.42

  0%|          | 197/41250 [29:05<98:59:22,  8.68s/it]
                                                      
{'loss': 0.3358, 'grad_norm': 1.9142600297927856, 'learning_rate': 6.365105008077545e-06, 'epoch': 0.05}

  0%|          | 197/41250 [29:05<98:59:22,  8.68s/it][2025-04-26 04:37:43,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:37:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.17 | bwd_microstep: 5713.78 | bwd_inner_microstep: 5689.44 | bwd_allreduce_microstep: 24.30 | step_microstep: 18.61
[2025-04-26 04:37:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.17 | bwd: 5713.80 | bwd_inner: 5689.44 | bwd_allreduce: 24.32 | step: 18.62

  0%|          | 198/41250 [29:14<98:51:17,  8.67s/it]
                                                      
{'loss': 0.2623, 'grad_norm': 1.5619583129882812, 'learning_rate': 6.397415185783522e-06, 'epoch': 0.05}

  0%|          | 198/41250 [29:14<98:51:17,  8.67s/it][2025-04-26 04:37:51,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 04:37:51,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.83 | bwd_microstep: 5764.62 | bwd_inner_microstep: 5717.12 | bwd_allreduce_microstep: 47.45 | step_microstep: 19.38
[2025-04-26 04:37:51,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.83 | bwd: 5764.64 | bwd_inner: 5717.12 | bwd_allreduce: 47.47 | step: 19.39

  0%|          | 199/41250 [29:23<98:59:01,  8.68s/it]
                                                      
{'loss': 0.4006, 'grad_norm': 2.278601884841919, 'learning_rate': 6.4297253634895e-06, 'epoch': 0.05}

  0%|          | 199/41250 [29:23<98:59:01,  8.68s/it][2025-04-26 04:38:00,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:38:00,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5707.37 | bwd_inner_microstep: 5694.56 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.64
[2025-04-26 04:38:00,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5707.38 | bwd_inner: 5694.56 | bwd_allreduce: 12.77 | step: 18.65

  0%|          | 200/41250 [29:31<98:49:01,  8.67s/it]
                                                      
{'loss': 0.1457, 'grad_norm': 1.2338846921920776, 'learning_rate': 6.462035541195477e-06, 'epoch': 0.05}

  0%|          | 200/41250 [29:31<98:49:01,  8.67s/it][2025-04-26 04:38:09,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:38:09,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5750.57 | bwd_inner_microstep: 5696.08 | bwd_allreduce_microstep: 54.44 | step_microstep: 18.64
[2025-04-26 04:38:09,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5750.59 | bwd_inner: 5696.08 | bwd_allreduce: 54.46 | step: 18.64

  0%|          | 201/41250 [29:40<98:52:41,  8.67s/it]
                                                      
{'loss': 0.2987, 'grad_norm': 2.010288953781128, 'learning_rate': 6.494345718901455e-06, 'epoch': 0.05}

  0%|          | 201/41250 [29:40<98:52:41,  8.67s/it][2025-04-26 04:38:17,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 04:38:17,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.99 | bwd_microstep: 5900.14 | bwd_inner_microstep: 5688.07 | bwd_allreduce_microstep: 212.02 | step_microstep: 19.67
[2025-04-26 04:38:17,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.99 | bwd: 5900.16 | bwd_inner: 5688.07 | bwd_allreduce: 212.04 | step: 19.67

  0%|          | 202/41250 [29:49<99:25:27,  8.72s/it]
                                                      
{'loss': 0.3706, 'grad_norm': 1.8878592252731323, 'learning_rate': 6.5266558966074325e-06, 'epoch': 0.05}

  0%|          | 202/41250 [29:49<99:25:27,  8.72s/it][2025-04-26 04:38:26,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.18
[2025-04-26 04:38:26,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.78 | bwd_microstep: 5710.27 | bwd_inner_microstep: 5651.93 | bwd_allreduce_microstep: 58.28 | step_microstep: 19.66
[2025-04-26 04:38:26,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5710.29 | bwd_inner: 5651.93 | bwd_allreduce: 58.31 | step: 19.67

  0%|          | 203/41250 [29:57<99:06:26,  8.69s/it]
                                                      
{'loss': 0.2755, 'grad_norm': 1.6627784967422485, 'learning_rate': 6.5589660743134096e-06, 'epoch': 0.05}

  0%|          | 203/41250 [29:57<99:06:26,  8.69s/it][2025-04-26 04:38:35,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:38:35,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5783.50 | bwd_inner_microstep: 5660.07 | bwd_allreduce_microstep: 123.38 | step_microstep: 18.25
[2025-04-26 04:38:35,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5783.51 | bwd_inner: 5660.07 | bwd_allreduce: 123.40 | step: 18.26

  0%|          | 204/41250 [30:06<99:08:05,  8.69s/it]
                                                      
{'loss': 0.1572, 'grad_norm': 0.9885423183441162, 'learning_rate': 6.591276252019387e-06, 'epoch': 0.05}

  0%|          | 204/41250 [30:06<99:08:05,  8.69s/it][2025-04-26 04:38:44,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.06 | optimizer_step: 1.23
[2025-04-26 04:38:44,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5806.36 | bwd_inner_microstep: 5645.24 | bwd_allreduce_microstep: 161.07 | step_microstep: 19.39
[2025-04-26 04:38:44,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5806.38 | bwd_inner: 5645.24 | bwd_allreduce: 161.09 | step: 19.39

  0%|          | 205/41250 [30:15<99:11:22,  8.70s/it]
                                                      
{'loss': 0.2526, 'grad_norm': 1.5966293811798096, 'learning_rate': 6.623586429725364e-06, 'epoch': 0.05}

  0%|          | 205/41250 [30:15<99:11:22,  8.70s/it][2025-04-26 04:38:52,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 04:38:52,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5761.84 | bwd_inner_microstep: 5646.75 | bwd_allreduce_microstep: 115.05 | step_microstep: 18.39
[2025-04-26 04:38:52,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5761.85 | bwd_inner: 5646.75 | bwd_allreduce: 115.07 | step: 18.39

  0%|          | 206/41250 [30:23<99:07:57,  8.70s/it]
                                                      
{'loss': 0.1825, 'grad_norm': 1.3031717538833618, 'learning_rate': 6.6558966074313415e-06, 'epoch': 0.05}

  0%|          | 206/41250 [30:23<99:07:57,  8.70s/it][2025-04-26 04:39:01,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:39:01,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.65 | bwd_microstep: 5717.81 | bwd_inner_microstep: 5705.07 | bwd_allreduce_microstep: 12.70 | step_microstep: 17.86
[2025-04-26 04:39:01,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.65 | bwd: 5717.82 | bwd_inner: 5705.07 | bwd_allreduce: 12.72 | step: 17.86

  1%|          | 207/41250 [30:32<98:59:10,  8.68s/it]
                                                      
{'loss': 0.2066, 'grad_norm': 1.2442219257354736, 'learning_rate': 6.6882067851373186e-06, 'epoch': 0.05}

  1%|          | 207/41250 [30:32<98:59:10,  8.68s/it][2025-04-26 04:39:10,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 04:39:10,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.70 | bwd_microstep: 5764.74 | bwd_inner_microstep: 5695.37 | bwd_allreduce_microstep: 69.31 | step_microstep: 18.71
[2025-04-26 04:39:10,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.70 | bwd: 5764.75 | bwd_inner: 5695.37 | bwd_allreduce: 69.34 | step: 18.71

  1%|          | 208/41250 [30:41<99:04:32,  8.69s/it]
                                                      
{'loss': 0.2806, 'grad_norm': 1.9887155294418335, 'learning_rate': 6.7205169628432965e-06, 'epoch': 0.05}

  1%|          | 208/41250 [30:41<99:04:32,  8.69s/it][2025-04-26 04:39:18,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-26 04:39:18,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.27 | bwd_microstep: 5781.94 | bwd_inner_microstep: 5645.44 | bwd_allreduce_microstep: 136.46 | step_microstep: 18.65
[2025-04-26 04:39:18,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.27 | bwd: 5781.95 | bwd_inner: 5645.44 | bwd_allreduce: 136.47 | step: 18.65

  1%|          | 209/41250 [30:49<99:00:59,  8.69s/it]
                                                      
{'loss': 0.1797, 'grad_norm': 1.5705664157867432, 'learning_rate': 6.752827140549274e-06, 'epoch': 0.05}

  1%|          | 209/41250 [30:49<99:00:59,  8.69s/it][2025-04-26 04:39:27,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:39:27,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.89 | bwd_microstep: 5710.69 | bwd_inner_microstep: 5698.13 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.25
[2025-04-26 04:39:27,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.89 | bwd: 5710.70 | bwd_inner: 5698.13 | bwd_allreduce: 12.54 | step: 18.25

  1%|          | 210/41250 [30:58<98:52:01,  8.67s/it]
                                                      
{'loss': 0.1484, 'grad_norm': 4.11529541015625, 'learning_rate': 6.7851373182552505e-06, 'epoch': 0.05}

  1%|          | 210/41250 [30:58<98:52:01,  8.67s/it][2025-04-26 04:39:36,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:39:36,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 5765.94 | bwd_inner_microstep: 5637.11 | bwd_allreduce_microstep: 128.79 | step_microstep: 17.83
[2025-04-26 04:39:36,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 5765.95 | bwd_inner: 5637.11 | bwd_allreduce: 128.80 | step: 17.83

  1%|          | 211/41250 [31:07<98:53:47,  8.68s/it]
                                                      
{'loss': 0.1961, 'grad_norm': 2.5238702297210693, 'learning_rate': 6.817447495961228e-06, 'epoch': 0.05}

  1%|          | 211/41250 [31:07<98:53:47,  8.68s/it][2025-04-26 04:39:44,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 04:39:44,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.21 | bwd_microstep: 5849.50 | bwd_inner_microstep: 5654.25 | bwd_allreduce_microstep: 195.19 | step_microstep: 19.65
[2025-04-26 04:39:44,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.21 | bwd: 5849.51 | bwd_inner: 5654.25 | bwd_allreduce: 195.22 | step: 19.65

  1%|          | 212/41250 [31:16<99:13:11,  8.70s/it]
                                                      
{'loss': 0.1615, 'grad_norm': 1.1589239835739136, 'learning_rate': 6.8497576736672054e-06, 'epoch': 0.05}

  1%|          | 212/41250 [31:16<99:13:11,  8.70s/it][2025-04-26 04:39:53,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:39:53,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.04 | bwd_microstep: 5758.86 | bwd_inner_microstep: 5640.09 | bwd_allreduce_microstep: 118.73 | step_microstep: 18.64
[2025-04-26 04:39:53,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.04 | bwd: 5758.87 | bwd_inner: 5640.09 | bwd_allreduce: 118.74 | step: 18.64

  1%|          | 213/41250 [31:24<99:04:09,  8.69s/it]
                                                      
{'loss': 0.2646, 'grad_norm': 1.9465585947036743, 'learning_rate': 6.882067851373183e-06, 'epoch': 0.05}

  1%|          | 213/41250 [31:24<99:04:09,  8.69s/it][2025-04-26 04:40:02,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:40:02,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.67 | bwd_microstep: 5710.47 | bwd_inner_microstep: 5637.59 | bwd_allreduce_microstep: 72.83 | step_microstep: 18.23
[2025-04-26 04:40:02,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.67 | bwd: 5710.48 | bwd_inner: 5637.59 | bwd_allreduce: 72.85 | step: 18.24

  1%|          | 214/41250 [31:33<98:46:36,  8.67s/it]
                                                      
{'loss': 0.3123, 'grad_norm': 2.2128257751464844, 'learning_rate': 6.91437802907916e-06, 'epoch': 0.05}

  1%|          | 214/41250 [31:33<98:46:36,  8.67s/it][2025-04-26 04:40:10,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 04:40:10,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.67 | bwd_microstep: 5872.31 | bwd_inner_microstep: 5703.87 | bwd_allreduce_microstep: 168.38 | step_microstep: 19.24
[2025-04-26 04:40:10,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.67 | bwd: 5872.33 | bwd_inner: 5703.87 | bwd_allreduce: 168.40 | step: 19.24

  1%|          | 215/41250 [31:42<99:15:44,  8.71s/it]
                                                      
{'loss': 0.2199, 'grad_norm': 2.0167648792266846, 'learning_rate': 6.946688206785138e-06, 'epoch': 0.05}

  1%|          | 215/41250 [31:42<99:15:44,  8.71s/it][2025-04-26 04:40:19,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:40:19,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.45 | bwd_microstep: 5839.80 | bwd_inner_microstep: 5654.40 | bwd_allreduce_microstep: 185.36 | step_microstep: 18.54
[2025-04-26 04:40:19,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.45 | bwd: 5839.82 | bwd_inner: 5654.40 | bwd_allreduce: 185.38 | step: 18.54

  1%|          | 216/41250 [31:50<99:22:43,  8.72s/it]
                                                      
{'loss': 0.2176, 'grad_norm': 1.9759330749511719, 'learning_rate': 6.978998384491116e-06, 'epoch': 0.05}

  1%|          | 216/41250 [31:50<99:22:43,  8.72s/it][2025-04-26 04:40:28,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:40:28,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.96 | bwd_microstep: 5744.97 | bwd_inner_microstep: 5645.21 | bwd_allreduce_microstep: 99.72 | step_microstep: 18.48
[2025-04-26 04:40:28,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.96 | bwd: 5744.98 | bwd_inner: 5645.21 | bwd_allreduce: 99.74 | step: 18.48

  1%|          | 217/41250 [31:59<99:07:46,  8.70s/it]
                                                      
{'loss': 0.3779, 'grad_norm': 2.0477395057678223, 'learning_rate': 7.011308562197092e-06, 'epoch': 0.05}

  1%|          | 217/41250 [31:59<99:07:46,  8.70s/it][2025-04-26 04:40:36,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:40:36,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.14 | bwd_microstep: 5729.67 | bwd_inner_microstep: 5699.73 | bwd_allreduce_microstep: 29.90 | step_microstep: 18.59
[2025-04-26 04:40:36,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.14 | bwd: 5729.68 | bwd_inner: 5699.73 | bwd_allreduce: 29.91 | step: 18.60

  1%|          | 218/41250 [32:08<98:56:56,  8.68s/it]
                                                      
{'loss': 0.358, 'grad_norm': 2.755713701248169, 'learning_rate': 7.043618739903069e-06, 'epoch': 0.05}

  1%|          | 218/41250 [32:08<98:56:56,  8.68s/it][2025-04-26 04:40:45,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-26 04:40:45,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.12 | bwd_microstep: 5685.71 | bwd_inner_microstep: 5672.71 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.70
[2025-04-26 04:40:45,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.12 | bwd: 5685.73 | bwd_inner: 5672.71 | bwd_allreduce: 12.97 | step: 18.70

  1%|          | 219/41250 [32:16<98:42:01,  8.66s/it]
                                                      
{'loss': 0.3603, 'grad_norm': 2.1685519218444824, 'learning_rate': 7.075928917609047e-06, 'epoch': 0.05}

  1%|          | 219/41250 [32:16<98:42:01,  8.66s/it][2025-04-26 04:40:54,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 04:40:54,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.20 | bwd_microstep: 5697.59 | bwd_inner_microstep: 5684.85 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.60
[2025-04-26 04:40:54,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.20 | bwd: 5697.60 | bwd_inner: 5684.85 | bwd_allreduce: 12.71 | step: 18.61

  1%|          | 220/41250 [32:25<98:34:25,  8.65s/it]
                                                      
{'loss': 0.2885, 'grad_norm': 1.6612672805786133, 'learning_rate': 7.108239095315025e-06, 'epoch': 0.05}

  1%|          | 220/41250 [32:25<98:34:25,  8.65s/it][2025-04-26 04:41:02,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:41:02,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.04 | bwd_microstep: 5747.91 | bwd_inner_microstep: 5654.25 | bwd_allreduce_microstep: 93.62 | step_microstep: 18.30
[2025-04-26 04:41:02,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.04 | bwd: 5747.93 | bwd_inner: 5654.25 | bwd_allreduce: 93.64 | step: 18.30

  1%|          | 221/41250 [32:34<98:37:59,  8.65s/it]
                                                      
{'loss': 0.2055, 'grad_norm': 1.458186149597168, 'learning_rate': 7.140549273021002e-06, 'epoch': 0.05}

  1%|          | 221/41250 [32:34<98:37:59,  8.65s/it][2025-04-26 04:41:11,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 04:41:11,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.54 | bwd_microstep: 5754.30 | bwd_inner_microstep: 5648.53 | bwd_allreduce_microstep: 105.72 | step_microstep: 18.38
[2025-04-26 04:41:11,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.54 | bwd: 5754.32 | bwd_inner: 5648.53 | bwd_allreduce: 105.75 | step: 18.38

  1%|          | 222/41250 [32:42<98:42:17,  8.66s/it]
                                                      
{'loss': 0.2423, 'grad_norm': 1.932356834411621, 'learning_rate': 7.17285945072698e-06, 'epoch': 0.05}

  1%|          | 222/41250 [32:42<98:42:17,  8.66s/it][2025-04-26 04:41:20,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 04:41:20,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.96 | bwd_microstep: 5870.65 | bwd_inner_microstep: 5858.02 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.74
[2025-04-26 04:41:20,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.96 | bwd: 5870.66 | bwd_inner: 5858.02 | bwd_allreduce: 12.61 | step: 18.74

  1%|          | 223/41250 [32:51<99:28:01,  8.73s/it]
                                                      
{'loss': 0.3148, 'grad_norm': 1.6337697505950928, 'learning_rate': 7.205169628432956e-06, 'epoch': 0.05}

  1%|          | 223/41250 [32:51<99:28:01,  8.73s/it][2025-04-26 04:41:29,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:41:29,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.17 | bwd_microstep: 5726.99 | bwd_inner_microstep: 5711.79 | bwd_allreduce_microstep: 15.16 | step_microstep: 18.42
[2025-04-26 04:41:29,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.17 | bwd: 5727.01 | bwd_inner: 5711.79 | bwd_allreduce: 15.17 | step: 18.42

  1%|          | 224/41250 [33:00<99:14:38,  8.71s/it]
                                                      
{'loss': 0.2152, 'grad_norm': 1.4786161184310913, 'learning_rate': 7.237479806138934e-06, 'epoch': 0.05}

  1%|          | 224/41250 [33:00<99:14:38,  8.71s/it][2025-04-26 04:41:37,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:41:37,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5695.84 | bwd_inner_microstep: 5637.78 | bwd_allreduce_microstep: 58.02 | step_microstep: 18.38
[2025-04-26 04:41:37,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5695.86 | bwd_inner: 5637.78 | bwd_allreduce: 58.04 | step: 18.39

  1%|          | 225/41250 [33:08<98:52:28,  8.68s/it]
                                                      
{'loss': 0.2108, 'grad_norm': 1.9917116165161133, 'learning_rate': 7.269789983844911e-06, 'epoch': 0.05}

  1%|          | 225/41250 [33:08<98:52:28,  8.68s/it][2025-04-26 04:41:46,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:41:46,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5700.33 | bwd_inner_microstep: 5687.69 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.28
[2025-04-26 04:41:46,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5700.34 | bwd_inner: 5687.69 | bwd_allreduce: 12.61 | step: 18.29

  1%|          | 226/41250 [33:17<98:42:47,  8.66s/it]
                                                      
{'loss': 0.6005, 'grad_norm': 3.5607688426971436, 'learning_rate': 7.302100161550889e-06, 'epoch': 0.05}

  1%|          | 226/41250 [33:17<98:42:47,  8.66s/it][2025-04-26 04:41:55,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:41:55,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.89 | bwd_microstep: 5972.23 | bwd_inner_microstep: 5959.34 | bwd_allreduce_microstep: 12.84 | step_microstep: 17.93
[2025-04-26 04:41:55,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5972.24 | bwd_inner: 5959.34 | bwd_allreduce: 12.86 | step: 17.93

  1%|          | 227/41250 [33:26<99:30:36,  8.73s/it]
                                                      
{'loss': 0.2463, 'grad_norm': 1.7967671155929565, 'learning_rate': 7.334410339256867e-06, 'epoch': 0.06}

  1%|          | 227/41250 [33:26<99:30:36,  8.73s/it][2025-04-26 04:42:03,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:42:03,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.31 | bwd_microstep: 5742.60 | bwd_inner_microstep: 5690.92 | bwd_allreduce_microstep: 51.64 | step_microstep: 18.39
[2025-04-26 04:42:03,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5742.62 | bwd_inner: 5690.92 | bwd_allreduce: 51.65 | step: 18.39

  1%|          | 228/41250 [33:35<99:18:31,  8.72s/it]
                                                      
{'loss': 0.198, 'grad_norm': 1.9401729106903076, 'learning_rate': 7.366720516962844e-06, 'epoch': 0.06}

  1%|          | 228/41250 [33:35<99:18:31,  8.72s/it][2025-04-26 04:42:12,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 04:42:12,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5710.76 | bwd_inner_microstep: 5697.48 | bwd_allreduce_microstep: 13.22 | step_microstep: 18.68
[2025-04-26 04:42:12,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5710.77 | bwd_inner: 5697.48 | bwd_allreduce: 13.24 | step: 18.68

  1%|          | 229/41250 [33:43<99:04:03,  8.69s/it]
                                                      
{'loss': 0.2187, 'grad_norm': 1.4544118642807007, 'learning_rate': 7.399030694668822e-06, 'epoch': 0.06}

  1%|          | 229/41250 [33:43<99:04:03,  8.69s/it][2025-04-26 04:42:21,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.14
[2025-04-26 04:42:21,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.65 | bwd_microstep: 5752.15 | bwd_inner_microstep: 5644.37 | bwd_allreduce_microstep: 107.74 | step_microstep: 19.05
[2025-04-26 04:42:21,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.65 | bwd: 5752.16 | bwd_inner: 5644.37 | bwd_allreduce: 107.76 | step: 19.05

  1%|          | 230/41250 [33:52<98:58:59,  8.69s/it]
                                                      
{'loss': 0.2984, 'grad_norm': 1.8355315923690796, 'learning_rate': 7.431340872374798e-06, 'epoch': 0.06}

  1%|          | 230/41250 [33:52<98:58:59,  8.69s/it][2025-04-26 04:42:29,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 04:42:29,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.55 | bwd_microstep: 5776.88 | bwd_inner_microstep: 5643.45 | bwd_allreduce_microstep: 133.38 | step_microstep: 19.44
[2025-04-26 04:42:29,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.55 | bwd: 5776.89 | bwd_inner: 5643.45 | bwd_allreduce: 133.40 | step: 19.44

  1%|          | 231/41250 [34:01<99:00:53,  8.69s/it]
                                                      
{'loss': 0.3244, 'grad_norm': 2.09818959236145, 'learning_rate': 7.463651050080776e-06, 'epoch': 0.06}

  1%|          | 231/41250 [34:01<99:00:53,  8.69s/it][2025-04-26 04:42:38,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-26 04:42:38,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.57 | bwd_microstep: 5802.78 | bwd_inner_microstep: 5654.95 | bwd_allreduce_microstep: 147.78 | step_microstep: 19.03
[2025-04-26 04:42:38,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.57 | bwd: 5802.79 | bwd_inner: 5654.95 | bwd_allreduce: 147.81 | step: 19.04

  1%|          | 232/41250 [34:09<99:03:36,  8.69s/it]
                                                      
{'loss': 0.3524, 'grad_norm': 2.312269926071167, 'learning_rate': 7.495961227786753e-06, 'epoch': 0.06}

  1%|          | 232/41250 [34:09<99:03:36,  8.69s/it][2025-04-26 04:42:47,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 04:42:47,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.63 | bwd_microstep: 5760.40 | bwd_inner_microstep: 5705.22 | bwd_allreduce_microstep: 55.13 | step_microstep: 18.41
[2025-04-26 04:42:47,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.63 | bwd: 5760.41 | bwd_inner: 5705.22 | bwd_allreduce: 55.15 | step: 18.41

  1%|          | 233/41250 [34:18<99:04:33,  8.70s/it]
                                                      
{'loss': 0.3547, 'grad_norm': 2.8929545879364014, 'learning_rate': 7.528271405492731e-06, 'epoch': 0.06}

  1%|          | 233/41250 [34:18<99:04:33,  8.70s/it][2025-04-26 04:42:56,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:42:56,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.25 | bwd_microstep: 5798.20 | bwd_inner_microstep: 5702.97 | bwd_allreduce_microstep: 95.18 | step_microstep: 18.28
[2025-04-26 04:42:56,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.25 | bwd: 5798.21 | bwd_inner: 5702.97 | bwd_allreduce: 95.20 | step: 18.28

  1%|          | 234/41250 [34:27<99:12:26,  8.71s/it]
                                                      
{'loss': 0.1926, 'grad_norm': 1.164900302886963, 'learning_rate': 7.560581583198709e-06, 'epoch': 0.06}

  1%|          | 234/41250 [34:27<99:12:26,  8.71s/it][2025-04-26 04:43:04,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-26 04:43:04,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.45 | bwd_microstep: 5710.01 | bwd_inner_microstep: 5697.32 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.78
[2025-04-26 04:43:04,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.45 | bwd: 5710.02 | bwd_inner: 5697.32 | bwd_allreduce: 12.66 | step: 18.78

  1%|          | 235/41250 [34:35<98:58:21,  8.69s/it]
                                                      
{'loss': 0.3662, 'grad_norm': 2.516890287399292, 'learning_rate': 7.592891760904686e-06, 'epoch': 0.06}

  1%|          | 235/41250 [34:35<98:58:21,  8.69s/it][2025-04-26 04:43:13,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 04:43:13,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.35 | bwd_microstep: 5737.67 | bwd_inner_microstep: 5694.09 | bwd_allreduce_microstep: 43.53 | step_microstep: 19.17
[2025-04-26 04:43:13,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.35 | bwd: 5737.68 | bwd_inner: 5694.09 | bwd_allreduce: 43.55 | step: 19.17

  1%|          | 236/41250 [34:44<98:54:36,  8.68s/it]
                                                      
{'loss': 0.1876, 'grad_norm': 2.3774733543395996, 'learning_rate': 7.625201938610664e-06, 'epoch': 0.06}

  1%|          | 236/41250 [34:44<98:54:36,  8.68s/it][2025-04-26 04:43:22,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:43:22,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.43 | bwd_microstep: 5742.78 | bwd_inner_microstep: 5691.57 | bwd_allreduce_microstep: 51.16 | step_microstep: 18.88
[2025-04-26 04:43:22,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.43 | bwd: 5742.79 | bwd_inner: 5691.57 | bwd_allreduce: 51.18 | step: 18.88

  1%|          | 237/41250 [34:53<98:52:46,  8.68s/it]
                                                      
{'loss': 0.2441, 'grad_norm': 2.9336211681365967, 'learning_rate': 7.65751211631664e-06, 'epoch': 0.06}

  1%|          | 237/41250 [34:53<98:52:46,  8.68s/it][2025-04-26 04:43:30,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 04:43:30,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.93 | bwd_microstep: 5744.57 | bwd_inner_microstep: 5693.05 | bwd_allreduce_microstep: 51.47 | step_microstep: 18.93
[2025-04-26 04:43:30,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.93 | bwd: 5744.59 | bwd_inner: 5693.05 | bwd_allreduce: 51.49 | step: 18.93

  1%|          | 238/41250 [35:01<98:53:17,  8.68s/it]
                                                      
{'loss': 0.3741, 'grad_norm': 2.5830860137939453, 'learning_rate': 7.689822294022618e-06, 'epoch': 0.06}

  1%|          | 238/41250 [35:01<98:53:17,  8.68s/it][2025-04-26 04:43:39,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 04:43:39,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.34 | bwd_microstep: 5710.75 | bwd_inner_microstep: 5686.50 | bwd_allreduce_microstep: 24.21 | step_microstep: 18.70
[2025-04-26 04:43:39,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.34 | bwd: 5710.76 | bwd_inner: 5686.50 | bwd_allreduce: 24.23 | step: 18.70

  1%|          | 239/41250 [35:10<98:45:05,  8.67s/it]
                                                      
{'loss': 0.2988, 'grad_norm': 1.7903717756271362, 'learning_rate': 7.722132471728596e-06, 'epoch': 0.06}

  1%|          | 239/41250 [35:10<98:45:05,  8.67s/it][2025-04-26 04:43:48,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 04:43:48,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.71 | bwd_microstep: 5761.65 | bwd_inner_microstep: 5656.22 | bwd_allreduce_microstep: 105.39 | step_microstep: 18.57
[2025-04-26 04:43:48,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.71 | bwd: 5761.67 | bwd_inner: 5656.22 | bwd_allreduce: 105.41 | step: 18.57

  1%|          | 240/41250 [35:19<98:46:08,  8.67s/it]
                                                      
{'loss': 0.2573, 'grad_norm': 2.1349687576293945, 'learning_rate': 7.754442649434572e-06, 'epoch': 0.06}

  1%|          | 240/41250 [35:19<98:46:08,  8.67s/it][2025-04-26 04:43:56,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 04:43:56,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.44 | bwd_microstep: 5769.84 | bwd_inner_microstep: 5700.55 | bwd_allreduce_microstep: 69.25 | step_microstep: 17.98
[2025-04-26 04:43:56,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.44 | bwd: 5769.85 | bwd_inner: 5700.55 | bwd_allreduce: 69.26 | step: 17.99

  1%|          | 241/41250 [35:27<98:51:08,  8.68s/it]
                                                      
{'loss': 0.3784, 'grad_norm': 2.517171621322632, 'learning_rate': 7.78675282714055e-06, 'epoch': 0.06}

  1%|          | 241/41250 [35:27<98:51:08,  8.68s/it][2025-04-26 04:44:05,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 04:44:05,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5712.08 | bwd_inner_microstep: 5653.58 | bwd_allreduce_microstep: 58.46 | step_microstep: 17.89
[2025-04-26 04:44:05,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5712.09 | bwd_inner: 5653.58 | bwd_allreduce: 58.48 | step: 17.89

  1%|          | 242/41250 [35:36<98:41:36,  8.66s/it]
                                                      
{'loss': 0.0763, 'grad_norm': 0.8474845290184021, 'learning_rate': 7.819063004846528e-06, 'epoch': 0.06}

  1%|          | 242/41250 [35:36<98:41:36,  8.66s/it][2025-04-26 04:44:14,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 04:44:14,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.71 | bwd_microstep: 5756.13 | bwd_inner_microstep: 5706.06 | bwd_allreduce_microstep: 50.02 | step_microstep: 18.99
[2025-04-26 04:44:14,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.71 | bwd: 5756.14 | bwd_inner: 5706.06 | bwd_allreduce: 50.04 | step: 19.00

  1%|          | 243/41250 [35:45<98:50:11,  8.68s/it]
                                                      
{'loss': 0.2374, 'grad_norm': 1.7711223363876343, 'learning_rate': 7.851373182552504e-06, 'epoch': 0.06}

  1%|          | 243/41250 [35:45<98:50:11,  8.68s/it][2025-04-26 04:44:22,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 04:44:22,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.69 | bwd_microstep: 5720.19 | bwd_inner_microstep: 5707.46 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.98
[2025-04-26 04:44:22,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.69 | bwd: 5720.21 | bwd_inner: 5707.46 | bwd_allreduce: 12.71 | step: 18.99

  1%|          | 244/41250 [35:53<98:46:12,  8.67s/it]
                                                      
{'loss': 0.1224, 'grad_norm': 0.9541312456130981, 'learning_rate': 7.883683360258482e-06, 'epoch': 0.06}

  1%|          | 244/41250 [35:53<98:46:12,  8.67s/it][2025-04-26 04:44:31,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 04:44:31,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.76 | bwd_microstep: 5787.14 | bwd_inner_microstep: 5711.45 | bwd_allreduce_microstep: 75.64 | step_microstep: 18.79
[2025-04-26 04:44:31,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.76 | bwd: 5787.16 | bwd_inner: 5711.45 | bwd_allreduce: 75.67 | step: 18.80

  1%|          | 245/41250 [36:02<98:54:43,  8.68s/it]
                                                      
{'loss': 0.1788, 'grad_norm': 1.60268235206604, 'learning_rate': 7.91599353796446e-06, 'epoch': 0.06}

  1%|          | 245/41250 [36:02<98:54:43,  8.68s/it][2025-04-26 04:44:40,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:44:40,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.98 | bwd_microstep: 5720.00 | bwd_inner_microstep: 5707.34 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.39
[2025-04-26 04:44:40,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.98 | bwd: 5720.01 | bwd_inner: 5707.34 | bwd_allreduce: 12.63 | step: 18.39

  1%|          | 246/41250 [36:11<98:50:27,  8.68s/it]
                                                      
{'loss': 0.1888, 'grad_norm': 1.4619486331939697, 'learning_rate': 7.948303715670437e-06, 'epoch': 0.06}

  1%|          | 246/41250 [36:11<98:50:27,  8.68s/it][2025-04-26 04:44:48,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 04:44:48,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5746.63 | bwd_inner_microstep: 5703.39 | bwd_allreduce_microstep: 43.19 | step_microstep: 18.96
[2025-04-26 04:44:48,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5746.65 | bwd_inner: 5703.39 | bwd_allreduce: 43.21 | step: 18.96

  1%|          | 247/41250 [36:19<98:52:10,  8.68s/it]
                                                      
{'loss': 0.2279, 'grad_norm': 1.9855180978775024, 'learning_rate': 7.980613893376414e-06, 'epoch': 0.06}

  1%|          | 247/41250 [36:19<98:52:10,  8.68s/it][2025-04-26 04:44:57,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 04:44:57,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5776.93 | bwd_inner_microstep: 5701.21 | bwd_allreduce_microstep: 75.67 | step_microstep: 19.32
[2025-04-26 04:44:57,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5776.94 | bwd_inner: 5701.21 | bwd_allreduce: 75.69 | step: 19.33

  1%|          | 248/41250 [36:28<98:58:01,  8.69s/it]
                                                      
{'loss': 0.3719, 'grad_norm': 2.7205312252044678, 'learning_rate': 8.012924071082391e-06, 'epoch': 0.06}

  1%|          | 248/41250 [36:28<98:58:01,  8.69s/it][2025-04-26 04:45:06,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 04:45:06,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.34 | bwd_microstep: 5779.33 | bwd_inner_microstep: 5766.16 | bwd_allreduce_microstep: 13.12 | step_microstep: 18.89
[2025-04-26 04:45:06,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.34 | bwd: 5779.34 | bwd_inner: 5766.16 | bwd_allreduce: 13.14 | step: 18.89

  1%|          | 249/41250 [36:37<99:09:36,  8.71s/it]
                                                      
{'loss': 0.3012, 'grad_norm': 1.6713253259658813, 'learning_rate': 8.04523424878837e-06, 'epoch': 0.06}

  1%|          | 249/41250 [36:37<99:09:36,  8.71s/it][2025-04-26 04:45:14,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 04:45:14,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5725.92 | bwd_allreduce_microstep: 43.13 | step_microstep: 18.92
[2025-04-26 04:45:14,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5769.12 | bwd_inner: 5725.92 | bwd_allreduce: 43.15 | step: 18.93

  1%|          | 250/41250 [36:46<99:08:18,  8.70s/it]
                                                      
{'loss': 0.3057, 'grad_norm': 1.9675062894821167, 'learning_rate': 8.077544426494346e-06, 'epoch': 0.06}

  1%|          | 250/41250 [36:46<99:08:18,  8.70s/it][2025-04-26 04:45:23,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 04:45:23,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.66 | bwd_microstep: 5694.65 | bwd_inner_microstep: 5681.92 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.75
[2025-04-26 04:45:23,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.66 | bwd: 5694.66 | bwd_inner: 5681.92 | bwd_allreduce: 12.71 | step: 18.75

  1%|          | 251/41250 [36:54<98:53:31,  8.68s/it]
                                                      
{'loss': 0.1782, 'grad_norm': 1.6511996984481812, 'learning_rate': 8.109854604200323e-06, 'epoch': 0.06}

  1%|          | 251/41250 [36:54<98:53:31,  8.68s/it][2025-04-26 04:45:32,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:45:32,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.28 | bwd_microstep: 5734.16 | bwd_inner_microstep: 5708.64 | bwd_allreduce_microstep: 25.48 | step_microstep: 18.52
[2025-04-26 04:45:32,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.28 | bwd: 5734.17 | bwd_inner: 5708.64 | bwd_allreduce: 25.50 | step: 18.52

  1%|          | 252/41250 [37:03<98:50:36,  8.68s/it]
                                                      
{'loss': 0.3204, 'grad_norm': 2.4825680255889893, 'learning_rate': 8.142164781906301e-06, 'epoch': 0.06}

  1%|          | 252/41250 [37:03<98:50:36,  8.68s/it][2025-04-26 04:45:40,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 04:45:40,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.24 | bwd_microstep: 5778.50 | bwd_inner_microstep: 5651.09 | bwd_allreduce_microstep: 127.37 | step_microstep: 18.98
[2025-04-26 04:45:40,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.24 | bwd: 5778.52 | bwd_inner: 5651.09 | bwd_allreduce: 127.38 | step: 18.98

  1%|          | 253/41250 [37:12<98:53:23,  8.68s/it]
                                                      
{'loss': 0.2762, 'grad_norm': 2.1478824615478516, 'learning_rate': 8.17447495961228e-06, 'epoch': 0.06}

  1%|          | 253/41250 [37:12<98:53:23,  8.68s/it][2025-04-26 04:45:49,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:45:49,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.20 | bwd_microstep: 5745.79 | bwd_inner_microstep: 5683.89 | bwd_allreduce_microstep: 61.85 | step_microstep: 18.54
[2025-04-26 04:45:49,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.20 | bwd: 5745.81 | bwd_inner: 5683.89 | bwd_allreduce: 61.87 | step: 18.54

  1%|          | 254/41250 [37:20<98:50:27,  8.68s/it]
                                                      
{'loss': 0.1918, 'grad_norm': 1.6824727058410645, 'learning_rate': 8.206785137318255e-06, 'epoch': 0.06}

  1%|          | 254/41250 [37:20<98:50:27,  8.68s/it][2025-04-26 04:45:58,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 04:45:58,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.94 | bwd_microstep: 5745.72 | bwd_inner_microstep: 5656.09 | bwd_allreduce_microstep: 89.58 | step_microstep: 18.74
[2025-04-26 04:45:58,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.94 | bwd: 5745.73 | bwd_inner: 5656.09 | bwd_allreduce: 89.60 | step: 18.74

  1%|          | 255/41250 [37:29<98:52:02,  8.68s/it]
                                                      
{'loss': 0.288, 'grad_norm': 1.8902924060821533, 'learning_rate': 8.239095315024233e-06, 'epoch': 0.06}

  1%|          | 255/41250 [37:29<98:52:02,  8.68s/it][2025-04-26 04:46:07,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 04:46:07,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.68 | bwd_microstep: 5772.61 | bwd_inner_microstep: 5759.79 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.96
[2025-04-26 04:46:07,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.68 | bwd: 5772.62 | bwd_inner: 5759.79 | bwd_allreduce: 12.79 | step: 18.97

  1%|          | 256/41250 [37:38<99:02:41,  8.70s/it]
                                                      
{'loss': 0.1917, 'grad_norm': 1.801606297492981, 'learning_rate': 8.271405492730211e-06, 'epoch': 0.06}

  1%|          | 256/41250 [37:38<99:02:41,  8.70s/it][2025-04-26 04:46:15,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 04:46:15,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.22 | bwd_microstep: 5728.82 | bwd_inner_microstep: 5693.99 | bwd_allreduce_microstep: 34.79 | step_microstep: 19.26
[2025-04-26 04:46:15,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.22 | bwd: 5728.84 | bwd_inner: 5693.99 | bwd_allreduce: 34.81 | step: 19.26

  1%|          | 257/41250 [37:46<98:56:47,  8.69s/it]
                                                      
{'loss': 0.1728, 'grad_norm': 1.683178186416626, 'learning_rate': 8.303715670436187e-06, 'epoch': 0.06}

  1%|          | 257/41250 [37:46<98:56:47,  8.69s/it][2025-04-26 04:46:24,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 04:46:24,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.48 | bwd_microstep: 5699.52 | bwd_inner_microstep: 5656.61 | bwd_allreduce_microstep: 42.86 | step_microstep: 18.42
[2025-04-26 04:46:24,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.48 | bwd: 5699.53 | bwd_inner: 5656.61 | bwd_allreduce: 42.88 | step: 18.43

  1%|          | 258/41250 [37:55<98:42:18,  8.67s/it]
                                                      
{'loss': 0.1742, 'grad_norm': 1.4572052955627441, 'learning_rate': 8.336025848142165e-06, 'epoch': 0.06}

  1%|          | 258/41250 [37:55<98:42:18,  8.67s/it][2025-04-26 04:46:32,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:46:32,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.99 | bwd_microstep: 5731.62 | bwd_inner_microstep: 5690.76 | bwd_allreduce_microstep: 40.81 | step_microstep: 18.64
[2025-04-26 04:46:32,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.99 | bwd: 5731.63 | bwd_inner: 5690.76 | bwd_allreduce: 40.83 | step: 18.64

  1%|          | 259/41250 [38:04<98:43:25,  8.67s/it]
                                                      
{'loss': 0.0822, 'grad_norm': 1.238423466682434, 'learning_rate': 8.368336025848143e-06, 'epoch': 0.06}

  1%|          | 259/41250 [38:04<98:43:25,  8.67s/it][2025-04-26 04:46:41,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:46:41,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.03 | bwd_microstep: 5696.31 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 46.97 | step_microstep: 18.56
[2025-04-26 04:46:41,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.03 | bwd: 5696.32 | bwd_inner: 5649.29 | bwd_allreduce: 46.99 | step: 18.57

  1%|          | 260/41250 [38:12<98:31:21,  8.65s/it]
                                                      
{'loss': 0.2295, 'grad_norm': 2.4399168491363525, 'learning_rate': 8.400646203554121e-06, 'epoch': 0.06}

  1%|          | 260/41250 [38:12<98:31:21,  8.65s/it][2025-04-26 04:46:50,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 04:46:50,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.35 | bwd_microstep: 5723.72 | bwd_inner_microstep: 5691.60 | bwd_allreduce_microstep: 32.07 | step_microstep: 18.65
[2025-04-26 04:46:50,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.35 | bwd: 5723.73 | bwd_inner: 5691.60 | bwd_allreduce: 32.09 | step: 18.65

  1%|          | 261/41250 [38:21<98:31:17,  8.65s/it]
                                                      
{'loss': 0.1751, 'grad_norm': 1.3267483711242676, 'learning_rate': 8.432956381260097e-06, 'epoch': 0.06}

  1%|          | 261/41250 [38:21<98:31:17,  8.65s/it][2025-04-26 04:46:58,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 04:46:58,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.81 | bwd_microstep: 5710.08 | bwd_inner_microstep: 5697.28 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.78
[2025-04-26 04:46:58,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.81 | bwd: 5710.10 | bwd_inner: 5697.28 | bwd_allreduce: 12.78 | step: 18.78

  1%|          | 262/41250 [38:30<98:30:37,  8.65s/it]
                                                      
{'loss': 0.3054, 'grad_norm': 4.142087459564209, 'learning_rate': 8.465266558966075e-06, 'epoch': 0.06}

  1%|          | 262/41250 [38:30<98:30:37,  8.65s/it][2025-04-26 04:47:07,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 04:47:07,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5749.14 | bwd_inner_microstep: 5683.35 | bwd_allreduce_microstep: 65.74 | step_microstep: 18.35
[2025-04-26 04:47:07,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5749.15 | bwd_inner: 5683.35 | bwd_allreduce: 65.76 | step: 18.36

  1%|          | 263/41250 [38:38<98:35:30,  8.66s/it]
                                                      
{'loss': 0.3186, 'grad_norm': 2.8628056049346924, 'learning_rate': 8.497576736672051e-06, 'epoch': 0.06}

  1%|          | 263/41250 [38:38<98:35:30,  8.66s/it][2025-04-26 04:47:16,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:47:16,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.99 | bwd_microstep: 5693.53 | bwd_inner_microstep: 5680.80 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.46
[2025-04-26 04:47:16,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.99 | bwd: 5693.54 | bwd_inner: 5680.80 | bwd_allreduce: 12.70 | step: 18.46

  1%|          | 264/41250 [38:47<98:29:14,  8.65s/it]
                                                      
{'loss': 0.2348, 'grad_norm': 3.538630485534668, 'learning_rate': 8.529886914378029e-06, 'epoch': 0.06}

  1%|          | 264/41250 [38:47<98:29:14,  8.65s/it][2025-04-26 04:47:24,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:47:24,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.45 | bwd_microstep: 5696.66 | bwd_inner_microstep: 5657.11 | bwd_allreduce_microstep: 39.50 | step_microstep: 18.37
[2025-04-26 04:47:24,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.45 | bwd: 5696.67 | bwd_inner: 5657.11 | bwd_allreduce: 39.52 | step: 18.38

  1%|          | 265/41250 [38:56<98:20:59,  8.64s/it]
                                                      
{'loss': 0.3122, 'grad_norm': 2.418339967727661, 'learning_rate': 8.562197092084007e-06, 'epoch': 0.06}

  1%|          | 265/41250 [38:56<98:20:59,  8.64s/it][2025-04-26 04:47:33,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 04:47:33,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.80 | bwd_microstep: 5747.32 | bwd_inner_microstep: 5709.40 | bwd_allreduce_microstep: 37.87 | step_microstep: 19.22
[2025-04-26 04:47:33,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.80 | bwd: 5747.33 | bwd_inner: 5709.40 | bwd_allreduce: 37.89 | step: 19.22

  1%|          | 266/41250 [39:04<98:30:06,  8.65s/it]
                                                      
{'loss': 0.2875, 'grad_norm': 2.5021111965179443, 'learning_rate': 8.594507269789985e-06, 'epoch': 0.06}

  1%|          | 266/41250 [39:04<98:30:06,  8.65s/it][2025-04-26 04:47:42,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 04:47:42,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.37 | bwd_microstep: 5774.34 | bwd_inner_microstep: 5643.58 | bwd_allreduce_microstep: 130.71 | step_microstep: 18.55
[2025-04-26 04:47:42,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.37 | bwd: 5774.35 | bwd_inner: 5643.58 | bwd_allreduce: 130.73 | step: 18.55

  1%|          | 267/41250 [39:13<98:37:11,  8.66s/it]
                                                      
{'loss': 0.2016, 'grad_norm': 3.241555690765381, 'learning_rate': 8.626817447495963e-06, 'epoch': 0.06}

  1%|          | 267/41250 [39:13<98:37:11,  8.66s/it][2025-04-26 04:47:50,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:47:50,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.85 | bwd_microstep: 5734.15 | bwd_inner_microstep: 5711.03 | bwd_allreduce_microstep: 23.07 | step_microstep: 18.49
[2025-04-26 04:47:50,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.85 | bwd: 5734.16 | bwd_inner: 5711.03 | bwd_allreduce: 23.09 | step: 18.50

  1%|          | 268/41250 [39:22<98:38:53,  8.67s/it]
                                                      
{'loss': 0.1323, 'grad_norm': 1.635090708732605, 'learning_rate': 8.659127625201939e-06, 'epoch': 0.06}

  1%|          | 268/41250 [39:22<98:38:53,  8.67s/it][2025-04-26 04:47:59,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:47:59,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.42 | bwd_microstep: 5721.10 | bwd_inner_microstep: 5683.59 | bwd_allreduce_microstep: 37.46 | step_microstep: 18.54
[2025-04-26 04:47:59,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.42 | bwd: 5721.11 | bwd_inner: 5683.59 | bwd_allreduce: 37.48 | step: 18.55

  1%|          | 269/41250 [39:30<98:38:44,  8.67s/it]
                                                      
{'loss': 0.2528, 'grad_norm': 2.084852695465088, 'learning_rate': 8.691437802907917e-06, 'epoch': 0.07}

  1%|          | 269/41250 [39:30<98:38:44,  8.67s/it][2025-04-26 04:48:08,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:48:08,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.91 | bwd_microstep: 5740.46 | bwd_inner_microstep: 5643.55 | bwd_allreduce_microstep: 96.87 | step_microstep: 18.48
[2025-04-26 04:48:08,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.91 | bwd: 5740.47 | bwd_inner: 5643.55 | bwd_allreduce: 96.88 | step: 18.48

  1%|          | 270/41250 [39:39<98:37:57,  8.66s/it]
                                                      
{'loss': 0.3612, 'grad_norm': 2.649104118347168, 'learning_rate': 8.723747980613893e-06, 'epoch': 0.07}

  1%|          | 270/41250 [39:39<98:37:57,  8.66s/it][2025-04-26 04:48:16,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 04:48:16,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.80 | bwd_microstep: 5697.24 | bwd_inner_microstep: 5632.08 | bwd_allreduce_microstep: 65.11 | step_microstep: 19.10
[2025-04-26 04:48:16,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.80 | bwd: 5697.25 | bwd_inner: 5632.08 | bwd_allreduce: 65.13 | step: 19.11

  1%|          | 271/41250 [39:47<98:27:39,  8.65s/it]
                                                      
{'loss': 0.1458, 'grad_norm': 1.843572974205017, 'learning_rate': 8.756058158319871e-06, 'epoch': 0.07}

  1%|          | 271/41250 [39:47<98:27:39,  8.65s/it][2025-04-26 04:48:25,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 1.07
[2025-04-26 04:48:25,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.55 | bwd_microstep: 5773.71 | bwd_inner_microstep: 5647.39 | bwd_allreduce_microstep: 126.27 | step_microstep: 19.53
[2025-04-26 04:48:25,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.55 | bwd: 5773.72 | bwd_inner: 5647.39 | bwd_allreduce: 126.29 | step: 19.53

  1%|          | 272/41250 [39:56<98:34:54,  8.66s/it]
                                                      
{'loss': 0.0754, 'grad_norm': 0.9079554677009583, 'learning_rate': 8.788368336025849e-06, 'epoch': 0.07}

  1%|          | 272/41250 [39:56<98:34:54,  8.66s/it][2025-04-26 04:48:34,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.09 | optimizer_step: 1.23
[2025-04-26 04:48:34,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.34 | bwd_microstep: 5742.31 | bwd_inner_microstep: 5696.45 | bwd_allreduce_microstep: 45.81 | step_microstep: 19.80
[2025-04-26 04:48:34,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.34 | bwd: 5742.32 | bwd_inner: 5696.45 | bwd_allreduce: 45.83 | step: 19.80

  1%|          | 273/41250 [40:05<98:39:58,  8.67s/it]
                                                      
{'loss': 0.3088, 'grad_norm': 2.475390911102295, 'learning_rate': 8.820678513731827e-06, 'epoch': 0.07}

  1%|          | 273/41250 [40:05<98:39:58,  8.67s/it][2025-04-26 04:48:42,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.07
[2025-04-26 04:48:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5758.45 | bwd_inner_microstep: 5665.29 | bwd_allreduce_microstep: 93.10 | step_microstep: 19.63
[2025-04-26 04:48:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5758.46 | bwd_inner: 5665.29 | bwd_allreduce: 93.12 | step: 19.63

  1%|          | 274/41250 [40:14<98:45:04,  8.68s/it]
                                                      
{'loss': 0.2829, 'grad_norm': 3.097269058227539, 'learning_rate': 8.852988691437805e-06, 'epoch': 0.07}

  1%|          | 274/41250 [40:14<98:45:04,  8.68s/it][2025-04-26 04:48:51,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:48:51,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5737.59 | bwd_inner_microstep: 5690.69 | bwd_allreduce_microstep: 46.86 | step_microstep: 18.55
[2025-04-26 04:48:51,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5737.60 | bwd_inner: 5690.69 | bwd_allreduce: 46.87 | step: 18.56

  1%|          | 275/41250 [40:22<98:44:56,  8.68s/it]
                                                      
{'loss': 0.2585, 'grad_norm': 2.7463414669036865, 'learning_rate': 8.88529886914378e-06, 'epoch': 0.07}

  1%|          | 275/41250 [40:22<98:44:56,  8.68s/it][2025-04-26 04:49:00,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 04:49:00,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5749.59 | bwd_inner_microstep: 5659.48 | bwd_allreduce_microstep: 90.06 | step_microstep: 19.27
[2025-04-26 04:49:00,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5749.60 | bwd_inner: 5659.48 | bwd_allreduce: 90.08 | step: 19.28

  1%|          | 276/41250 [40:31<98:43:36,  8.67s/it]
                                                      
{'loss': 0.2857, 'grad_norm': 4.176706314086914, 'learning_rate': 8.917609046849759e-06, 'epoch': 0.07}

  1%|          | 276/41250 [40:31<98:43:36,  8.67s/it][2025-04-26 04:49:08,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 04:49:08,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.69 | bwd_microstep: 5713.95 | bwd_inner_microstep: 5701.21 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.43
[2025-04-26 04:49:08,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.69 | bwd: 5713.96 | bwd_inner: 5701.21 | bwd_allreduce: 12.72 | step: 18.44

  1%|          | 277/41250 [40:40<98:36:37,  8.66s/it]
                                                      
{'loss': 0.1638, 'grad_norm': 2.8278772830963135, 'learning_rate': 8.949919224555735e-06, 'epoch': 0.07}

  1%|          | 277/41250 [40:40<98:36:37,  8.66s/it][2025-04-26 04:49:17,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.09 | optimizer_step: 1.18
[2025-04-26 04:49:17,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.02 | bwd_microstep: 5747.57 | bwd_inner_microstep: 5651.34 | bwd_allreduce_microstep: 96.18 | step_microstep: 19.65
[2025-04-26 04:49:17,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.02 | bwd: 5747.58 | bwd_inner: 5651.34 | bwd_allreduce: 96.20 | step: 19.66

  1%|          | 278/41250 [40:48<98:38:56,  8.67s/it]
                                                      
{'loss': 0.2701, 'grad_norm': 2.296992063522339, 'learning_rate': 8.982229402261713e-06, 'epoch': 0.07}

  1%|          | 278/41250 [40:48<98:38:56,  8.67s/it][2025-04-26 04:49:26,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 0.93
[2025-04-26 04:49:26,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.27 | bwd_microstep: 5720.29 | bwd_inner_microstep: 5706.85 | bwd_allreduce_microstep: 13.38 | step_microstep: 19.11
[2025-04-26 04:49:26,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.27 | bwd: 5720.31 | bwd_inner: 5706.85 | bwd_allreduce: 13.41 | step: 19.11

  1%|          | 279/41250 [40:57<98:37:05,  8.67s/it]
                                                      
{'loss': 0.2249, 'grad_norm': 1.8784308433532715, 'learning_rate': 9.01453957996769e-06, 'epoch': 0.07}

  1%|          | 279/41250 [40:57<98:37:05,  8.67s/it][2025-04-26 04:49:34,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:49:34,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.59 | bwd_microstep: 5727.69 | bwd_inner_microstep: 5693.37 | bwd_allreduce_microstep: 34.27 | step_microstep: 18.41
[2025-04-26 04:49:34,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.59 | bwd: 5727.70 | bwd_inner: 5693.37 | bwd_allreduce: 34.29 | step: 18.42

  1%|          | 280/41250 [41:06<98:38:04,  8.67s/it]
                                                      
{'loss': 0.3246, 'grad_norm': 2.609536647796631, 'learning_rate': 9.046849757673668e-06, 'epoch': 0.07}

  1%|          | 280/41250 [41:06<98:38:04,  8.67s/it][2025-04-26 04:49:43,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:49:43,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.50 | bwd_microstep: 5808.74 | bwd_inner_microstep: 5661.03 | bwd_allreduce_microstep: 147.65 | step_microstep: 18.59
[2025-04-26 04:49:43,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.50 | bwd: 5808.76 | bwd_inner: 5661.03 | bwd_allreduce: 147.68 | step: 18.59

  1%|          | 281/41250 [41:14<98:48:51,  8.68s/it]
                                                      
{'loss': 0.2583, 'grad_norm': 2.2472152709960938, 'learning_rate': 9.079159935379646e-06, 'epoch': 0.07}

  1%|          | 281/41250 [41:14<98:48:51,  8.68s/it][2025-04-26 04:49:52,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 04:49:52,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.94 | bwd_microstep: 5751.35 | bwd_inner_microstep: 5698.48 | bwd_allreduce_microstep: 52.81 | step_microstep: 19.36
[2025-04-26 04:49:52,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.94 | bwd: 5751.36 | bwd_inner: 5698.48 | bwd_allreduce: 52.84 | step: 19.36

  1%|          | 282/41250 [41:23<98:51:32,  8.69s/it]
                                                      
{'loss': 0.1554, 'grad_norm': 1.1901865005493164, 'learning_rate': 9.111470113085623e-06, 'epoch': 0.07}

  1%|          | 282/41250 [41:23<98:51:32,  8.69s/it][2025-04-26 04:50:00,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:50:00,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5707.89 | bwd_inner_microstep: 5694.86 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.59
[2025-04-26 04:50:00,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5707.90 | bwd_inner: 5694.86 | bwd_allreduce: 13.00 | step: 18.59

  1%|          | 283/41250 [41:32<98:41:54,  8.67s/it]
                                                      
{'loss': 0.3587, 'grad_norm': 3.5427286624908447, 'learning_rate': 9.143780290791599e-06, 'epoch': 0.07}

  1%|          | 283/41250 [41:32<98:41:54,  8.67s/it][2025-04-26 04:50:09,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:50:09,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.80 | bwd_microstep: 5765.82 | bwd_inner_microstep: 5695.28 | bwd_allreduce_microstep: 70.49 | step_microstep: 18.65
[2025-04-26 04:50:09,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.80 | bwd: 5765.83 | bwd_inner: 5695.29 | bwd_allreduce: 70.50 | step: 18.65

  1%|          | 284/41250 [41:40<98:49:38,  8.68s/it]
                                                      
{'loss': 0.1845, 'grad_norm': 1.5258549451828003, 'learning_rate': 9.176090468497577e-06, 'epoch': 0.07}

  1%|          | 284/41250 [41:40<98:49:38,  8.68s/it][2025-04-26 04:50:18,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 04:50:18,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.40 | bwd_microstep: 5689.88 | bwd_inner_microstep: 5668.19 | bwd_allreduce_microstep: 21.64 | step_microstep: 18.94
[2025-04-26 04:50:18,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.40 | bwd: 5689.89 | bwd_inner: 5668.19 | bwd_allreduce: 21.66 | step: 18.94

  1%|          | 285/41250 [41:49<98:36:17,  8.67s/it]
                                                      
{'loss': 0.2339, 'grad_norm': 3.19935941696167, 'learning_rate': 9.208400646203555e-06, 'epoch': 0.07}

  1%|          | 285/41250 [41:49<98:36:17,  8.67s/it][2025-04-26 04:50:26,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 04:50:26,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5762.37 | bwd_inner_microstep: 5700.85 | bwd_allreduce_microstep: 61.48 | step_microstep: 18.59
[2025-04-26 04:50:26,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5762.38 | bwd_inner: 5700.84 | bwd_allreduce: 61.50 | step: 18.59

  1%|          | 286/41250 [41:58<98:41:26,  8.67s/it]
                                                      
{'loss': 0.1158, 'grad_norm': 1.4099664688110352, 'learning_rate': 9.240710823909532e-06, 'epoch': 0.07}

  1%|          | 286/41250 [41:58<98:41:26,  8.67s/it][2025-04-26 04:50:35,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 04:50:35,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5753.92 | bwd_inner_microstep: 5703.03 | bwd_allreduce_microstep: 50.84 | step_microstep: 18.53
[2025-04-26 04:50:35,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5753.93 | bwd_inner: 5703.03 | bwd_allreduce: 50.86 | step: 18.53

  1%|          | 287/41250 [42:06<98:44:14,  8.68s/it]
                                                      
{'loss': 0.2864, 'grad_norm': 4.275433540344238, 'learning_rate': 9.27302100161551e-06, 'epoch': 0.07}

  1%|          | 287/41250 [42:06<98:44:14,  8.68s/it][2025-04-26 04:50:44,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:50:44,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.99 | bwd_microstep: 5782.66 | bwd_inner_microstep: 5668.43 | bwd_allreduce_microstep: 114.18 | step_microstep: 18.60
[2025-04-26 04:50:44,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.99 | bwd: 5782.67 | bwd_inner: 5668.43 | bwd_allreduce: 114.20 | step: 18.60

  1%|          | 288/41250 [42:15<98:50:42,  8.69s/it]
                                                      
{'loss': 0.2387, 'grad_norm': 1.6862984895706177, 'learning_rate': 9.305331179321486e-06, 'epoch': 0.07}

  1%|          | 288/41250 [42:15<98:50:42,  8.69s/it][2025-04-26 04:50:52,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 04:50:52,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5723.10 | bwd_inner_microstep: 5655.36 | bwd_allreduce_microstep: 67.69 | step_microstep: 18.40
[2025-04-26 04:50:52,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5723.12 | bwd_inner: 5655.36 | bwd_allreduce: 67.72 | step: 18.41

  1%|          | 289/41250 [42:24<98:40:49,  8.67s/it]
                                                      
{'loss': 0.3333, 'grad_norm': 2.6784098148345947, 'learning_rate': 9.337641357027464e-06, 'epoch': 0.07}

  1%|          | 289/41250 [42:24<98:40:49,  8.67s/it][2025-04-26 04:51:01,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 04:51:01,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.22 | bwd_microstep: 5796.31 | bwd_inner_microstep: 5642.36 | bwd_allreduce_microstep: 153.91 | step_microstep: 19.03
[2025-04-26 04:51:01,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.22 | bwd: 5796.33 | bwd_inner: 5642.36 | bwd_allreduce: 153.93 | step: 19.04

  1%|          | 290/41250 [42:32<98:44:08,  8.68s/it]
                                                      
{'loss': 0.2021, 'grad_norm': 1.8835253715515137, 'learning_rate': 9.36995153473344e-06, 'epoch': 0.07}

  1%|          | 290/41250 [42:32<98:44:08,  8.68s/it][2025-04-26 04:51:10,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:51:10,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.75 | bwd_microstep: 5719.08 | bwd_inner_microstep: 5706.27 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.53
[2025-04-26 04:51:10,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.75 | bwd: 5719.09 | bwd_inner: 5706.27 | bwd_allreduce: 12.78 | step: 18.53

  1%|          | 291/41250 [42:41<98:39:42,  8.67s/it]
                                                      
{'loss': 0.2063, 'grad_norm': 1.6320691108703613, 'learning_rate': 9.402261712439418e-06, 'epoch': 0.07}

  1%|          | 291/41250 [42:41<98:39:42,  8.67s/it][2025-04-26 04:51:19,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 04:51:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.97 | bwd_microstep: 5764.17 | bwd_inner_microstep: 5683.14 | bwd_allreduce_microstep: 80.99 | step_microstep: 18.65
[2025-04-26 04:51:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.97 | bwd: 5764.19 | bwd_inner: 5683.13 | bwd_allreduce: 81.01 | step: 18.65

  1%|          | 292/41250 [42:50<98:46:37,  8.68s/it]
                                                      
{'loss': 0.4139, 'grad_norm': 3.2363457679748535, 'learning_rate': 9.434571890145396e-06, 'epoch': 0.07}

  1%|          | 292/41250 [42:50<98:46:37,  8.68s/it][2025-04-26 04:51:27,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 04:51:27,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5710.47 | bwd_inner_microstep: 5656.34 | bwd_allreduce_microstep: 54.08 | step_microstep: 18.85
[2025-04-26 04:51:27,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5710.49 | bwd_inner: 5656.34 | bwd_allreduce: 54.10 | step: 18.85

  1%|          | 293/41250 [42:58<98:35:27,  8.67s/it]
                                                      
{'loss': 0.1815, 'grad_norm': 1.9197242259979248, 'learning_rate': 9.466882067851374e-06, 'epoch': 0.07}

  1%|          | 293/41250 [42:58<98:35:27,  8.67s/it][2025-04-26 04:51:36,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 04:51:36,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.94 | bwd_microstep: 5773.04 | bwd_inner_microstep: 5646.40 | bwd_allreduce_microstep: 126.59 | step_microstep: 19.09
[2025-04-26 04:51:36,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.94 | bwd: 5773.05 | bwd_inner: 5646.40 | bwd_allreduce: 126.61 | step: 19.09

  1%|          | 294/41250 [43:07<98:42:49,  8.68s/it]
                                                      
{'loss': 0.1488, 'grad_norm': 1.667259693145752, 'learning_rate': 9.499192245557352e-06, 'epoch': 0.07}

  1%|          | 294/41250 [43:07<98:42:49,  8.68s/it][2025-04-26 04:51:45,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 04:51:45,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.51 | bwd_microstep: 5770.61 | bwd_inner_microstep: 5679.25 | bwd_allreduce_microstep: 91.31 | step_microstep: 19.08
[2025-04-26 04:51:45,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.51 | bwd: 5770.62 | bwd_inner: 5679.25 | bwd_allreduce: 91.33 | step: 19.09

  1%|          | 295/41250 [43:16<98:47:07,  8.68s/it]
                                                      
{'loss': 0.1193, 'grad_norm': 1.4725666046142578, 'learning_rate': 9.531502423263328e-06, 'epoch': 0.07}

  1%|          | 295/41250 [43:16<98:47:07,  8.68s/it][2025-04-26 04:51:53,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 04:51:53,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5701.41 | bwd_inner_microstep: 5682.15 | bwd_allreduce_microstep: 19.22 | step_microstep: 18.61
[2025-04-26 04:51:53,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5701.43 | bwd_inner: 5682.15 | bwd_allreduce: 19.23 | step: 18.61

  1%|          | 296/41250 [43:24<98:37:35,  8.67s/it]
                                                      
{'loss': 0.155, 'grad_norm': 1.2428148984909058, 'learning_rate': 9.563812600969306e-06, 'epoch': 0.07}

  1%|          | 296/41250 [43:24<98:37:35,  8.67s/it][2025-04-26 04:52:02,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.08 | optimizer_step: 1.25
[2025-04-26 04:52:02,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.31 | bwd_microstep: 5716.49 | bwd_inner_microstep: 5632.01 | bwd_allreduce_microstep: 84.42 | step_microstep: 19.99
[2025-04-26 04:52:02,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.31 | bwd: 5716.50 | bwd_inner: 5632.01 | bwd_allreduce: 84.45 | step: 19.99

  1%|          | 297/41250 [43:33<98:28:33,  8.66s/it]
                                                      
{'loss': 0.3747, 'grad_norm': 4.114211559295654, 'learning_rate': 9.596122778675282e-06, 'epoch': 0.07}

  1%|          | 297/41250 [43:33<98:28:33,  8.66s/it][2025-04-26 04:52:10,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 04:52:10,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.63 | bwd_microstep: 5713.22 | bwd_inner_microstep: 5700.24 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.82
[2025-04-26 04:52:10,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.63 | bwd: 5713.23 | bwd_inner: 5700.24 | bwd_allreduce: 12.95 | step: 18.82

  1%|          | 298/41250 [43:42<98:24:18,  8.65s/it]
                                                      
{'loss': 0.2606, 'grad_norm': 2.4399807453155518, 'learning_rate': 9.62843295638126e-06, 'epoch': 0.07}

  1%|          | 298/41250 [43:42<98:24:18,  8.65s/it][2025-04-26 04:52:19,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 04:52:19,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.35 | bwd_microstep: 5701.87 | bwd_inner_microstep: 5689.25 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.70
[2025-04-26 04:52:19,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.35 | bwd: 5701.88 | bwd_inner: 5689.25 | bwd_allreduce: 12.59 | step: 18.70

  1%|          | 299/41250 [43:50<98:17:08,  8.64s/it]
                                                      
{'loss': 0.1819, 'grad_norm': 2.6386752128601074, 'learning_rate': 9.660743134087238e-06, 'epoch': 0.07}

  1%|          | 299/41250 [43:50<98:17:08,  8.64s/it][2025-04-26 04:52:28,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 04:52:28,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.89 | bwd_microstep: 5705.05 | bwd_inner_microstep: 5640.04 | bwd_allreduce_microstep: 64.96 | step_microstep: 18.59
[2025-04-26 04:52:28,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.89 | bwd: 5705.06 | bwd_inner: 5640.04 | bwd_allreduce: 64.98 | step: 18.60

  1%|          | 300/41250 [43:59<98:12:10,  8.63s/it]
                                                      
{'loss': 0.1465, 'grad_norm': 1.7345000505447388, 'learning_rate': 9.693053311793216e-06, 'epoch': 0.07}

  1%|          | 300/41250 [43:59<98:12:10,  8.63s/it][2025-04-26 04:52:36,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 04:52:36,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.03 | bwd_microstep: 5753.29 | bwd_inner_microstep: 5650.52 | bwd_allreduce_microstep: 102.72 | step_microstep: 19.07
[2025-04-26 04:52:36,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.03 | bwd: 5753.31 | bwd_inner: 5650.52 | bwd_allreduce: 102.75 | step: 19.07

  1%|          | 301/41250 [44:08<98:19:13,  8.64s/it]
                                                      
{'loss': 0.2563, 'grad_norm': 2.014244794845581, 'learning_rate': 9.725363489499194e-06, 'epoch': 0.07}

  1%|          | 301/41250 [44:08<98:19:13,  8.64s/it][2025-04-26 04:52:45,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-26 04:52:45,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5756.11 | bwd_inner_microstep: 5651.25 | bwd_allreduce_microstep: 104.82 | step_microstep: 18.72
[2025-04-26 04:52:45,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.48 | bwd: 5756.13 | bwd_inner: 5651.25 | bwd_allreduce: 104.84 | step: 18.73

  1%|          | 302/41250 [44:16<98:25:16,  8.65s/it]
                                                      
{'loss': 0.2382, 'grad_norm': 3.696591377258301, 'learning_rate': 9.75767366720517e-06, 'epoch': 0.07}

  1%|          | 302/41250 [44:16<98:25:16,  8.65s/it][2025-04-26 04:52:54,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:52:54,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.14 | bwd_microstep: 5704.23 | bwd_inner_microstep: 5637.85 | bwd_allreduce_microstep: 66.33 | step_microstep: 18.94
[2025-04-26 04:52:54,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.14 | bwd: 5704.24 | bwd_inner: 5637.85 | bwd_allreduce: 66.35 | step: 18.94

  1%|          | 303/41250 [44:25<98:18:52,  8.64s/it]
                                                      
{'loss': 0.1141, 'grad_norm': 1.400457739830017, 'learning_rate': 9.789983844911148e-06, 'epoch': 0.07}

  1%|          | 303/41250 [44:25<98:18:52,  8.64s/it][2025-04-26 04:53:02,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-26 04:53:02,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.37 | bwd_microstep: 5750.33 | bwd_inner_microstep: 5646.18 | bwd_allreduce_microstep: 104.10 | step_microstep: 19.21
[2025-04-26 04:53:02,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.37 | bwd: 5750.34 | bwd_inner: 5646.18 | bwd_allreduce: 104.13 | step: 19.21

  1%|          | 304/41250 [44:33<98:19:57,  8.65s/it]
                                                      
{'loss': 0.1696, 'grad_norm': 1.762178659439087, 'learning_rate': 9.822294022617124e-06, 'epoch': 0.07}

  1%|          | 304/41250 [44:34<98:19:57,  8.65s/it][2025-04-26 04:53:11,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.07 | optimizer_step: 0.92
[2025-04-26 04:53:11,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.46 | bwd_microstep: 5771.84 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 116.23 | step_microstep: 19.00
[2025-04-26 04:53:11,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.46 | bwd: 5771.86 | bwd_inner: 5655.56 | bwd_allreduce: 116.26 | step: 19.01

  1%|          | 305/41250 [44:42<98:27:33,  8.66s/it]
                                                      
{'loss': 0.2326, 'grad_norm': 3.5670766830444336, 'learning_rate': 9.854604200323102e-06, 'epoch': 0.07}

  1%|          | 305/41250 [44:42<98:27:33,  8.66s/it][2025-04-26 04:53:20,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:53:20,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.03 | bwd_microstep: 5676.01 | bwd_inner_microstep: 5648.79 | bwd_allreduce_microstep: 27.17 | step_microstep: 18.69
[2025-04-26 04:53:20,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.03 | bwd: 5676.02 | bwd_inner: 5648.79 | bwd_allreduce: 27.20 | step: 18.69

  1%|          | 306/41250 [44:51<98:13:54,  8.64s/it]
                                                      
{'loss': 0.2413, 'grad_norm': 2.6693506240844727, 'learning_rate': 9.88691437802908e-06, 'epoch': 0.07}

  1%|          | 306/41250 [44:51<98:13:54,  8.64s/it][2025-04-26 04:53:28,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.28 | optimizer_step: 1.12
[2025-04-26 04:53:28,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.09 | bwd_microstep: 5751.33 | bwd_inner_microstep: 5630.10 | bwd_allreduce_microstep: 121.17 | step_microstep: 20.21
[2025-04-26 04:53:28,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.09 | bwd: 5751.35 | bwd_inner: 5630.10 | bwd_allreduce: 121.20 | step: 20.21

  1%|          | 307/41250 [44:59<98:17:32,  8.64s/it]
                                                      
{'loss': 0.1312, 'grad_norm': 2.4616551399230957, 'learning_rate': 9.919224555735058e-06, 'epoch': 0.07}

  1%|          | 307/41250 [44:59<98:17:32,  8.64s/it][2025-04-26 04:53:37,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.04 | optimizer_step: 1.26
[2025-04-26 04:53:37,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5721.19 | bwd_inner_microstep: 5699.80 | bwd_allreduce_microstep: 21.35 | step_microstep: 19.39
[2025-04-26 04:53:37,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.83 | bwd: 5721.20 | bwd_inner: 5699.80 | bwd_allreduce: 21.36 | step: 19.39

  1%|          | 308/41250 [45:08<98:17:39,  8.64s/it]
                                                      
{'loss': 0.1935, 'grad_norm': 2.1956870555877686, 'learning_rate': 9.951534733441036e-06, 'epoch': 0.07}

  1%|          | 308/41250 [45:08<98:17:39,  8.64s/it][2025-04-26 04:53:46,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-26 04:53:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.52 | bwd_microstep: 5758.37 | bwd_inner_microstep: 5649.09 | bwd_allreduce_microstep: 109.24 | step_microstep: 19.34
[2025-04-26 04:53:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.52 | bwd: 5758.39 | bwd_inner: 5649.09 | bwd_allreduce: 109.26 | step: 19.34

  1%|          | 309/41250 [45:17<98:24:51,  8.65s/it]
                                                      
{'loss': 0.3402, 'grad_norm': 3.6729094982147217, 'learning_rate': 9.983844911147012e-06, 'epoch': 0.07}

  1%|          | 309/41250 [45:17<98:24:51,  8.65s/it][2025-04-26 04:53:54,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.11 | optimizer_step: 1.20
[2025-04-26 04:53:54,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.29 | bwd_microstep: 5763.00 | bwd_inner_microstep: 5652.48 | bwd_allreduce_microstep: 110.48 | step_microstep: 19.65
[2025-04-26 04:53:54,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.29 | bwd: 5763.02 | bwd_inner: 5652.47 | bwd_allreduce: 110.50 | step: 19.65

  1%|          | 310/41250 [45:25<98:27:41,  8.66s/it]
                                                      
{'loss': 0.3645, 'grad_norm': 3.5952794551849365, 'learning_rate': 1.001615508885299e-05, 'epoch': 0.08}

  1%|          | 310/41250 [45:25<98:27:41,  8.66s/it][2025-04-26 04:54:03,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.08 | optimizer_step: 1.22
[2025-04-26 04:54:03,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.54 | bwd_microstep: 5728.64 | bwd_inner_microstep: 5683.05 | bwd_allreduce_microstep: 45.55 | step_microstep: 19.63
[2025-04-26 04:54:03,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.54 | bwd: 5728.66 | bwd_inner: 5683.05 | bwd_allreduce: 45.57 | step: 19.64

  1%|          | 311/41250 [45:34<98:29:33,  8.66s/it]
                                                      
{'loss': 0.2319, 'grad_norm': 3.7522101402282715, 'learning_rate': 1.0048465266558966e-05, 'epoch': 0.08}

  1%|          | 311/41250 [45:34<98:29:33,  8.66s/it][2025-04-26 04:54:12,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.24 | optimizer_step: 1.06
[2025-04-26 04:54:12,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.23 | bwd_microstep: 5754.73 | bwd_inner_microstep: 5645.06 | bwd_allreduce_microstep: 109.62 | step_microstep: 20.02
[2025-04-26 04:54:12,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.23 | bwd: 5754.75 | bwd_inner: 5645.06 | bwd_allreduce: 109.64 | step: 20.02

  1%|          | 312/41250 [45:43<98:29:16,  8.66s/it]
                                                      
{'loss': 0.1638, 'grad_norm': 3.367499589920044, 'learning_rate': 1.0080775444264946e-05, 'epoch': 0.08}

  1%|          | 312/41250 [45:43<98:29:16,  8.66s/it][2025-04-26 04:54:20,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 04:54:20,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.18 | bwd_microstep: 5755.06 | bwd_inner_microstep: 5704.29 | bwd_allreduce_microstep: 50.73 | step_microstep: 18.91
[2025-04-26 04:54:20,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.18 | bwd: 5755.07 | bwd_inner: 5704.29 | bwd_allreduce: 50.74 | step: 18.91

  1%|          | 313/41250 [45:51<98:33:42,  8.67s/it]
                                                      
{'loss': 0.1866, 'grad_norm': 2.017770528793335, 'learning_rate': 1.0113085621970922e-05, 'epoch': 0.08}

  1%|          | 313/41250 [45:51<98:33:42,  8.67s/it][2025-04-26 04:54:29,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.07 | optimizer_step: 1.26
[2025-04-26 04:54:29,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 5765.37 | bwd_inner_microstep: 5652.71 | bwd_allreduce_microstep: 112.61 | step_microstep: 19.79
[2025-04-26 04:54:29,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 5765.38 | bwd_inner: 5652.71 | bwd_allreduce: 112.63 | step: 19.79

  1%|          | 314/41250 [46:00<98:36:15,  8.67s/it]
                                                      
{'loss': 0.3491, 'grad_norm': 3.1180429458618164, 'learning_rate': 1.0145395799676898e-05, 'epoch': 0.08}

  1%|          | 314/41250 [46:00<98:36:15,  8.67s/it][2025-04-26 04:54:38,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 04:54:38,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.58 | bwd_microstep: 5757.75 | bwd_inner_microstep: 5652.09 | bwd_allreduce_microstep: 105.61 | step_microstep: 18.67
[2025-04-26 04:54:38,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.58 | bwd: 5757.76 | bwd_inner: 5652.09 | bwd_allreduce: 105.63 | step: 18.68

  1%|          | 315/41250 [46:09<98:35:35,  8.67s/it]
                                                      
{'loss': 0.2641, 'grad_norm': 2.6272799968719482, 'learning_rate': 1.0177705977382877e-05, 'epoch': 0.08}

  1%|          | 315/41250 [46:09<98:35:35,  8.67s/it][2025-04-26 04:54:46,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.09 | optimizer_step: 0.89
[2025-04-26 04:54:46,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.01 | bwd_microstep: 5686.55 | bwd_inner_microstep: 5661.91 | bwd_allreduce_microstep: 24.59 | step_microstep: 19.22
[2025-04-26 04:54:46,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.01 | bwd: 5686.56 | bwd_inner: 5661.91 | bwd_allreduce: 24.61 | step: 19.22

  1%|          | 316/41250 [46:17<98:21:11,  8.65s/it]
                                                      
{'loss': 0.2712, 'grad_norm': 3.1948094367980957, 'learning_rate': 1.0210016155088854e-05, 'epoch': 0.08}

  1%|          | 316/41250 [46:17<98:21:11,  8.65s/it][2025-04-26 04:54:55,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 04:54:55,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.17 | bwd_microstep: 5766.31 | bwd_inner_microstep: 5643.83 | bwd_allreduce_microstep: 122.44 | step_microstep: 19.33
[2025-04-26 04:54:55,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.18 | bwd: 5766.33 | bwd_inner: 5643.82 | bwd_allreduce: 122.46 | step: 19.33

  1%|          | 317/41250 [46:26<98:24:20,  8.65s/it]
                                                      
{'loss': 0.0839, 'grad_norm': 1.3460108041763306, 'learning_rate': 1.0242326332794832e-05, 'epoch': 0.08}

  1%|          | 317/41250 [46:26<98:24:20,  8.65s/it][2025-04-26 04:55:04,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 04:55:04,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.63 | bwd_microstep: 5717.66 | bwd_inner_microstep: 5704.73 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.78
[2025-04-26 04:55:04,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.63 | bwd: 5717.67 | bwd_inner: 5704.73 | bwd_allreduce: 12.90 | step: 19.78

  1%|          | 318/41250 [46:35<98:23:11,  8.65s/it]
                                                      
{'loss': 0.3592, 'grad_norm': 4.437591552734375, 'learning_rate': 1.0274636510500808e-05, 'epoch': 0.08}

  1%|          | 318/41250 [46:35<98:23:11,  8.65s/it][2025-04-26 04:55:12,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 04:55:12,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.91 | bwd_microstep: 5706.94 | bwd_inner_microstep: 5651.05 | bwd_allreduce_microstep: 55.85 | step_microstep: 19.39
[2025-04-26 04:55:12,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.91 | bwd: 5706.95 | bwd_inner: 5651.05 | bwd_allreduce: 55.87 | step: 19.40

  1%|          | 319/41250 [46:43<98:15:38,  8.64s/it]
                                                      
{'loss': 0.3227, 'grad_norm': 3.464505910873413, 'learning_rate': 1.0306946688206787e-05, 'epoch': 0.08}

  1%|          | 319/41250 [46:43<98:15:38,  8.64s/it][2025-04-26 04:55:21,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 1.04 | optimizer_step: 1.15
[2025-04-26 04:55:21,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.99 | bwd_microstep: 5894.89 | bwd_inner_microstep: 5651.74 | bwd_allreduce_microstep: 243.09 | step_microstep: 18.79
[2025-04-26 04:55:21,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.99 | bwd: 5894.90 | bwd_inner: 5651.74 | bwd_allreduce: 243.12 | step: 18.79

  1%|          | 320/41250 [46:52<98:48:35,  8.69s/it]
                                                      
{'loss': 0.1132, 'grad_norm': 1.3409075736999512, 'learning_rate': 1.0339256865912764e-05, 'epoch': 0.08}

  1%|          | 320/41250 [46:52<98:48:35,  8.69s/it][2025-04-26 04:55:30,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 04:55:30,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5714.18 | bwd_inner_microstep: 5701.28 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.97
[2025-04-26 04:55:30,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5714.19 | bwd_inner: 5701.28 | bwd_allreduce: 12.86 | step: 18.97

  1%|          | 321/41250 [47:01<98:39:40,  8.68s/it]
                                                      
{'loss': 0.3184, 'grad_norm': 3.7344934940338135, 'learning_rate': 1.037156704361874e-05, 'epoch': 0.08}

  1%|          | 321/41250 [47:01<98:39:40,  8.68s/it][2025-04-26 04:55:38,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:55:38,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.71 | bwd_microstep: 5706.92 | bwd_inner_microstep: 5644.81 | bwd_allreduce_microstep: 62.06 | step_microstep: 18.53
[2025-04-26 04:55:38,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.71 | bwd: 5706.93 | bwd_inner: 5644.81 | bwd_allreduce: 62.08 | step: 18.53

  1%|          | 322/41250 [47:09<98:28:22,  8.66s/it]
                                                      
{'loss': 0.2281, 'grad_norm': 2.57902193069458, 'learning_rate': 1.040387722132472e-05, 'epoch': 0.08}

  1%|          | 322/41250 [47:09<98:28:22,  8.66s/it][2025-04-26 04:55:47,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 04:55:47,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.44 | bwd_microstep: 5790.70 | bwd_inner_microstep: 5777.89 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.65
[2025-04-26 04:55:47,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.44 | bwd: 5790.72 | bwd_inner: 5777.89 | bwd_allreduce: 12.79 | step: 19.65

  1%|          | 323/41250 [47:18<98:52:25,  8.70s/it]
                                                      
{'loss': 0.4439, 'grad_norm': 2.9428553581237793, 'learning_rate': 1.0436187399030695e-05, 'epoch': 0.08}

  1%|          | 323/41250 [47:18<98:52:25,  8.70s/it][2025-04-26 04:55:56,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:55:56,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.98 | bwd_microstep: 5765.62 | bwd_inner_microstep: 5701.45 | bwd_allreduce_microstep: 64.13 | step_microstep: 18.62
[2025-04-26 04:55:56,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.98 | bwd: 5765.64 | bwd_inner: 5701.45 | bwd_allreduce: 64.15 | step: 18.63

  1%|          | 324/41250 [47:27<98:53:19,  8.70s/it]
                                                      
{'loss': 0.4195, 'grad_norm': 3.753300905227661, 'learning_rate': 1.0468497576736673e-05, 'epoch': 0.08}

  1%|          | 324/41250 [47:27<98:53:19,  8.70s/it][2025-04-26 04:56:04,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:56:04,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.13 | bwd_microstep: 5712.18 | bwd_inner_microstep: 5699.49 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.46
[2025-04-26 04:56:04,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.13 | bwd: 5712.19 | bwd_inner: 5699.48 | bwd_allreduce: 12.66 | step: 18.47

  1%|          | 325/41250 [47:36<98:42:26,  8.68s/it]
                                                      
{'loss': 0.2401, 'grad_norm': 2.3682777881622314, 'learning_rate': 1.050080775444265e-05, 'epoch': 0.08}

  1%|          | 325/41250 [47:36<98:42:26,  8.68s/it][2025-04-26 04:56:13,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-26 04:56:13,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.13 | bwd_microstep: 5803.52 | bwd_inner_microstep: 5790.80 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.75
[2025-04-26 04:56:13,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.13 | bwd: 5803.53 | bwd_inner: 5790.80 | bwd_allreduce: 12.69 | step: 18.75

  1%|          | 326/41250 [47:44<98:59:42,  8.71s/it]
                                                      
{'loss': 0.3183, 'grad_norm': 2.8353843688964844, 'learning_rate': 1.0533117932148629e-05, 'epoch': 0.08}

  1%|          | 326/41250 [47:44<98:59:42,  8.71s/it][2025-04-26 04:56:22,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 04:56:22,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5716.05 | bwd_inner_microstep: 5664.85 | bwd_allreduce_microstep: 51.16 | step_microstep: 19.09
[2025-04-26 04:56:22,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.54 | bwd: 5716.06 | bwd_inner: 5664.85 | bwd_allreduce: 51.18 | step: 19.09

  1%|          | 327/41250 [47:53<98:45:46,  8.69s/it]
                                                      
{'loss': 0.3313, 'grad_norm': 3.701228141784668, 'learning_rate': 1.0565428109854605e-05, 'epoch': 0.08}

  1%|          | 327/41250 [47:53<98:45:46,  8.69s/it][2025-04-26 04:56:30,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 04:56:30,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.98 | bwd_microstep: 5774.14 | bwd_inner_microstep: 5681.29 | bwd_allreduce_microstep: 92.80 | step_microstep: 18.46
[2025-04-26 04:56:30,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.98 | bwd: 5774.15 | bwd_inner: 5681.29 | bwd_allreduce: 92.82 | step: 18.46

  1%|          | 328/41250 [48:02<98:48:14,  8.69s/it]
                                                      
{'loss': 0.1401, 'grad_norm': 1.593430757522583, 'learning_rate': 1.0597738287560582e-05, 'epoch': 0.08}

  1%|          | 328/41250 [48:02<98:48:14,  8.69s/it][2025-04-26 04:56:39,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:56:39,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.94 | bwd_microstep: 5769.43 | bwd_inner_microstep: 5658.05 | bwd_allreduce_microstep: 111.33 | step_microstep: 18.46
[2025-04-26 04:56:39,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.94 | bwd: 5769.44 | bwd_inner: 5658.05 | bwd_allreduce: 111.35 | step: 18.47

  1%|          | 329/41250 [48:10<98:45:32,  8.69s/it]
                                                      
{'loss': 0.0965, 'grad_norm': 1.5051320791244507, 'learning_rate': 1.0630048465266561e-05, 'epoch': 0.08}

  1%|          | 329/41250 [48:10<98:45:32,  8.69s/it][2025-04-26 04:56:48,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 04:56:48,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.23 | bwd_microstep: 5783.24 | bwd_inner_microstep: 5698.53 | bwd_allreduce_microstep: 84.67 | step_microstep: 18.45
[2025-04-26 04:56:48,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.23 | bwd: 5783.25 | bwd_inner: 5698.52 | bwd_allreduce: 84.68 | step: 18.46

  1%|          | 330/41250 [48:19<98:50:01,  8.70s/it]
                                                      
{'loss': 0.2283, 'grad_norm': 3.278073787689209, 'learning_rate': 1.0662358642972537e-05, 'epoch': 0.08}

  1%|          | 330/41250 [48:19<98:50:01,  8.70s/it][2025-04-26 04:56:57,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.24
[2025-04-26 04:56:57,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.29 | bwd_microstep: 5751.61 | bwd_inner_microstep: 5712.71 | bwd_allreduce_microstep: 38.85 | step_microstep: 19.64
[2025-04-26 04:56:57,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.29 | bwd: 5751.62 | bwd_inner: 5712.71 | bwd_allreduce: 38.87 | step: 19.64

  1%|          | 331/41250 [48:28<98:47:30,  8.69s/it]
                                                      
{'loss': 0.1721, 'grad_norm': 2.369450092315674, 'learning_rate': 1.0694668820678515e-05, 'epoch': 0.08}

  1%|          | 331/41250 [48:28<98:47:30,  8.69s/it][2025-04-26 04:57:05,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 04:57:05,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.55 | bwd_microstep: 5767.75 | bwd_inner_microstep: 5662.79 | bwd_allreduce_microstep: 104.89 | step_microstep: 19.39
[2025-04-26 04:57:05,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.55 | bwd: 5767.78 | bwd_inner: 5662.79 | bwd_allreduce: 104.92 | step: 19.39

  1%|          | 332/41250 [48:36<98:45:51,  8.69s/it]
                                                      
{'loss': 0.2228, 'grad_norm': 2.4900360107421875, 'learning_rate': 1.0726978998384491e-05, 'epoch': 0.08}

  1%|          | 332/41250 [48:36<98:45:51,  8.69s/it][2025-04-26 04:57:14,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 04:57:14,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.71 | bwd_microstep: 5717.19 | bwd_inner_microstep: 5704.32 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.13
[2025-04-26 04:57:14,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.71 | bwd: 5717.21 | bwd_inner: 5704.32 | bwd_allreduce: 12.84 | step: 19.14

  1%|          | 333/41250 [48:45<98:37:35,  8.68s/it]
                                                      
{'loss': 0.1944, 'grad_norm': 2.201319456100464, 'learning_rate': 1.0759289176090468e-05, 'epoch': 0.08}

  1%|          | 333/41250 [48:45<98:37:35,  8.68s/it][2025-04-26 04:57:23,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-26 04:57:23,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.26 | bwd_microstep: 5793.16 | bwd_inner_microstep: 5780.41 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.26
[2025-04-26 04:57:23,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.26 | bwd: 5793.17 | bwd_inner: 5780.41 | bwd_allreduce: 12.72 | step: 19.26

  1%|          | 334/41250 [48:54<98:56:02,  8.70s/it]
                                                      
{'loss': 0.1126, 'grad_norm': 1.733871340751648, 'learning_rate': 1.0791599353796447e-05, 'epoch': 0.08}

  1%|          | 334/41250 [48:54<98:56:02,  8.70s/it][2025-04-26 04:57:31,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 04:57:31,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5704.97 | bwd_inner_microstep: 5692.08 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.66
[2025-04-26 04:57:31,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5704.98 | bwd_inner: 5692.08 | bwd_allreduce: 12.85 | step: 19.66

  1%|          | 335/41250 [49:02<98:40:08,  8.68s/it]
                                                      
{'loss': 0.1757, 'grad_norm': 1.9842807054519653, 'learning_rate': 1.0823909531502423e-05, 'epoch': 0.08}

  1%|          | 335/41250 [49:02<98:40:08,  8.68s/it][2025-04-26 04:57:40,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 04:57:40,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.63 | bwd_microstep: 5764.41 | bwd_inner_microstep: 5689.96 | bwd_allreduce_microstep: 74.40 | step_microstep: 18.80
[2025-04-26 04:57:40,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.63 | bwd: 5764.42 | bwd_inner: 5689.96 | bwd_allreduce: 74.42 | step: 18.81

  1%|          | 336/41250 [49:11<98:42:31,  8.69s/it]
                                                      
{'loss': 0.2261, 'grad_norm': 2.2009730339050293, 'learning_rate': 1.0856219709208403e-05, 'epoch': 0.08}

  1%|          | 336/41250 [49:11<98:42:31,  8.69s/it][2025-04-26 04:57:49,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 04:57:49,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.48 | bwd_microstep: 5787.92 | bwd_inner_microstep: 5657.38 | bwd_allreduce_microstep: 130.49 | step_microstep: 19.22
[2025-04-26 04:57:49,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.48 | bwd: 5787.93 | bwd_inner: 5657.38 | bwd_allreduce: 130.51 | step: 19.23

  1%|          | 337/41250 [49:20<98:45:42,  8.69s/it]
                                                      
{'loss': 0.1597, 'grad_norm': 3.0101287364959717, 'learning_rate': 1.0888529886914379e-05, 'epoch': 0.08}

  1%|          | 337/41250 [49:20<98:45:42,  8.69s/it][2025-04-26 04:57:57,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:57:57,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.94 | bwd_microstep: 5771.58 | bwd_inner_microstep: 5641.73 | bwd_allreduce_microstep: 129.81 | step_microstep: 18.34
[2025-04-26 04:57:57,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.94 | bwd: 5771.59 | bwd_inner: 5641.73 | bwd_allreduce: 129.82 | step: 18.34

  1%|          | 338/41250 [49:29<98:44:26,  8.69s/it]
                                                      
{'loss': 0.0982, 'grad_norm': 1.2076185941696167, 'learning_rate': 1.0920840064620357e-05, 'epoch': 0.08}

  1%|          | 338/41250 [49:29<98:44:26,  8.69s/it][2025-04-26 04:58:06,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:58:06,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.37 | bwd_microstep: 5871.36 | bwd_inner_microstep: 5654.68 | bwd_allreduce_microstep: 216.64 | step_microstep: 18.42
[2025-04-26 04:58:06,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.37 | bwd: 5871.37 | bwd_inner: 5654.68 | bwd_allreduce: 216.65 | step: 18.42

  1%|          | 339/41250 [49:37<99:06:04,  8.72s/it]
                                                      
{'loss': 0.5235, 'grad_norm': 4.236124515533447, 'learning_rate': 1.0953150242326333e-05, 'epoch': 0.08}

  1%|          | 339/41250 [49:37<99:06:04,  8.72s/it][2025-04-26 04:58:15,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 04:58:15,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.12 | bwd_microstep: 5664.45 | bwd_inner_microstep: 5643.89 | bwd_allreduce_microstep: 20.53 | step_microstep: 17.64
[2025-04-26 04:58:15,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.12 | bwd: 5664.47 | bwd_inner: 5643.89 | bwd_allreduce: 20.54 | step: 17.64

  1%|          | 340/41250 [49:46<98:36:13,  8.68s/it]
                                                      
{'loss': 0.1748, 'grad_norm': 2.6743946075439453, 'learning_rate': 1.098546042003231e-05, 'epoch': 0.08}

  1%|          | 340/41250 [49:46<98:36:13,  8.68s/it][2025-04-26 04:58:23,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 04:58:23,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.75 | bwd_microstep: 5728.96 | bwd_inner_microstep: 5707.51 | bwd_allreduce_microstep: 21.42 | step_microstep: 18.67
[2025-04-26 04:58:23,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.75 | bwd: 5728.98 | bwd_inner: 5707.50 | bwd_allreduce: 21.43 | step: 18.67

  1%|          | 341/41250 [49:55<98:34:56,  8.68s/it]
                                                      
{'loss': 0.1457, 'grad_norm': 3.165405511856079, 'learning_rate': 1.1017770597738289e-05, 'epoch': 0.08}

  1%|          | 341/41250 [49:55<98:34:56,  8.68s/it][2025-04-26 04:58:32,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:58:32,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.60 | bwd_microstep: 5675.93 | bwd_inner_microstep: 5653.10 | bwd_allreduce_microstep: 22.79 | step_microstep: 18.16
[2025-04-26 04:58:32,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.60 | bwd: 5675.94 | bwd_inner: 5653.10 | bwd_allreduce: 22.81 | step: 18.16

  1%|          | 342/41250 [50:03<98:19:12,  8.65s/it]
                                                      
{'loss': 0.1816, 'grad_norm': 2.2201459407806396, 'learning_rate': 1.1050080775444265e-05, 'epoch': 0.08}

  1%|          | 342/41250 [50:03<98:19:12,  8.65s/it][2025-04-26 04:58:41,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 04:58:41,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.18 | bwd_microstep: 5759.82 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 113.32 | step_microstep: 18.39
[2025-04-26 04:58:41,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.19 | bwd: 5759.83 | bwd_inner: 5646.46 | bwd_allreduce: 113.34 | step: 18.40

  1%|          | 343/41250 [50:12<98:22:22,  8.66s/it]
                                                      
{'loss': 0.3232, 'grad_norm': 5.276711463928223, 'learning_rate': 1.1082390953150243e-05, 'epoch': 0.08}

  1%|          | 343/41250 [50:12<98:22:22,  8.66s/it][2025-04-26 04:58:49,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:58:49,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.70 | bwd_microstep: 5742.76 | bwd_inner_microstep: 5700.32 | bwd_allreduce_microstep: 42.39 | step_microstep: 18.16
[2025-04-26 04:58:49,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.70 | bwd: 5742.77 | bwd_inner: 5700.32 | bwd_allreduce: 42.41 | step: 18.16

  1%|          | 344/41250 [50:20<98:24:38,  8.66s/it]
                                                      
{'loss': 0.1681, 'grad_norm': 1.3014425039291382, 'learning_rate': 1.1114701130856221e-05, 'epoch': 0.08}

  1%|          | 344/41250 [50:20<98:24:38,  8.66s/it][2025-04-26 04:58:58,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:58:58,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.94 | bwd_microstep: 5786.37 | bwd_inner_microstep: 5773.56 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.83
[2025-04-26 04:58:58,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.94 | bwd: 5786.39 | bwd_inner: 5773.56 | bwd_allreduce: 12.78 | step: 18.83

  1%|          | 345/41250 [50:29<98:44:55,  8.69s/it]
                                                      
{'loss': 0.1453, 'grad_norm': 1.535555124282837, 'learning_rate': 1.1147011308562199e-05, 'epoch': 0.08}

  1%|          | 345/41250 [50:29<98:44:55,  8.69s/it][2025-04-26 04:59:07,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.15
[2025-04-26 04:59:07,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.92 | bwd_microstep: 5767.68 | bwd_inner_microstep: 5647.55 | bwd_allreduce_microstep: 120.08 | step_microstep: 19.09
[2025-04-26 04:59:07,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.92 | bwd: 5767.69 | bwd_inner: 5647.55 | bwd_allreduce: 120.10 | step: 19.09

  1%|          | 346/41250 [50:38<98:43:55,  8.69s/it]
                                                      
{'loss': 0.1703, 'grad_norm': 4.156448841094971, 'learning_rate': 1.1179321486268175e-05, 'epoch': 0.08}

  1%|          | 346/41250 [50:38<98:43:55,  8.69s/it][2025-04-26 04:59:15,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 04:59:15,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.97 | bwd_microstep: 5736.37 | bwd_inner_microstep: 5671.79 | bwd_allreduce_microstep: 64.54 | step_microstep: 18.16
[2025-04-26 04:59:15,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.97 | bwd: 5736.38 | bwd_inner: 5671.79 | bwd_allreduce: 64.55 | step: 18.16

  1%|          | 347/41250 [50:47<98:38:56,  8.68s/it]
                                                      
{'loss': 0.3466, 'grad_norm': 4.059357166290283, 'learning_rate': 1.1211631663974151e-05, 'epoch': 0.08}

  1%|          | 347/41250 [50:47<98:38:56,  8.68s/it][2025-04-26 04:59:24,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 04:59:24,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.63 | bwd_microstep: 5740.06 | bwd_inner_microstep: 5679.87 | bwd_allreduce_microstep: 60.14 | step_microstep: 18.84
[2025-04-26 04:59:24,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.63 | bwd: 5740.07 | bwd_inner: 5679.87 | bwd_allreduce: 60.16 | step: 18.84

  1%|          | 348/41250 [50:55<98:36:06,  8.68s/it]
                                                      
{'loss': 0.3081, 'grad_norm': 2.493713855743408, 'learning_rate': 1.124394184168013e-05, 'epoch': 0.08}

  1%|          | 348/41250 [50:55<98:36:06,  8.68s/it][2025-04-26 04:59:33,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 04:59:33,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.20 | bwd_microstep: 5757.41 | bwd_inner_microstep: 5699.71 | bwd_allreduce_microstep: 57.65 | step_microstep: 18.82
[2025-04-26 04:59:33,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.20 | bwd: 5757.42 | bwd_inner: 5699.71 | bwd_allreduce: 57.67 | step: 18.82

  1%|          | 349/41250 [51:04<98:35:57,  8.68s/it]
                                                      
{'loss': 0.2137, 'grad_norm': 1.8894367218017578, 'learning_rate': 1.1276252019386107e-05, 'epoch': 0.08}

  1%|          | 349/41250 [51:04<98:35:57,  8.68s/it][2025-04-26 04:59:42,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.04
[2025-04-26 04:59:42,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.46 | bwd_microstep: 5872.83 | bwd_inner_microstep: 5674.46 | bwd_allreduce_microstep: 198.33 | step_microstep: 19.43
[2025-04-26 04:59:42,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.46 | bwd: 5872.84 | bwd_inner: 5674.46 | bwd_allreduce: 198.35 | step: 19.43

  1%|          | 350/41250 [51:13<99:00:45,  8.72s/it]
                                                      
{'loss': 0.2721, 'grad_norm': 2.4367687702178955, 'learning_rate': 1.1308562197092085e-05, 'epoch': 0.08}

  1%|          | 350/41250 [51:13<99:00:45,  8.72s/it][2025-04-26 04:59:50,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 04:59:50,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.97 | bwd_microstep: 5739.73 | bwd_inner_microstep: 5683.93 | bwd_allreduce_microstep: 55.76 | step_microstep: 18.25
[2025-04-26 04:59:50,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.97 | bwd: 5739.75 | bwd_inner: 5683.93 | bwd_allreduce: 55.78 | step: 18.26

  1%|          | 351/41250 [51:21<98:50:39,  8.70s/it]
                                                      
{'loss': 0.3174, 'grad_norm': 2.6192781925201416, 'learning_rate': 1.1340872374798063e-05, 'epoch': 0.09}

  1%|          | 351/41250 [51:21<98:50:39,  8.70s/it][2025-04-26 04:59:59,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 04:59:59,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5726.87 | bwd_inner_microstep: 5696.26 | bwd_allreduce_microstep: 30.57 | step_microstep: 18.63
[2025-04-26 04:59:59,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5726.88 | bwd_inner: 5696.26 | bwd_allreduce: 30.58 | step: 18.63

  1%|          | 352/41250 [51:30<98:41:51,  8.69s/it]
                                                      
{'loss': 0.2823, 'grad_norm': 2.6434695720672607, 'learning_rate': 1.137318255250404e-05, 'epoch': 0.09}

  1%|          | 352/41250 [51:30<98:41:51,  8.69s/it][2025-04-26 05:00:08,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 05:00:08,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.33 | bwd_microstep: 5792.17 | bwd_inner_microstep: 5660.30 | bwd_allreduce_microstep: 131.82 | step_microstep: 19.01
[2025-04-26 05:00:08,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.34 | bwd: 5792.18 | bwd_inner: 5660.30 | bwd_allreduce: 131.84 | step: 19.01

  1%|          | 353/41250 [51:39<98:42:03,  8.69s/it]
                                                      
{'loss': 0.1557, 'grad_norm': 1.3619394302368164, 'learning_rate': 1.1405492730210017e-05, 'epoch': 0.09}

  1%|          | 353/41250 [51:39<98:42:03,  8.69s/it][2025-04-26 05:00:16,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 05:00:16,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5739.98 | bwd_inner_microstep: 5686.51 | bwd_allreduce_microstep: 53.43 | step_microstep: 18.36
[2025-04-26 05:00:16,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5740.00 | bwd_inner: 5686.51 | bwd_allreduce: 53.45 | step: 18.36

  1%|          | 354/41250 [51:47<98:38:41,  8.68s/it]
                                                      
{'loss': 0.191, 'grad_norm': 1.8960860967636108, 'learning_rate': 1.1437802907915993e-05, 'epoch': 0.09}

  1%|          | 354/41250 [51:47<98:38:41,  8.68s/it][2025-04-26 05:00:25,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.26 | optimizer_step: 1.08
[2025-04-26 05:00:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.59 | bwd_microstep: 5884.71 | bwd_inner_microstep: 5646.09 | bwd_allreduce_microstep: 238.57 | step_microstep: 19.96
[2025-04-26 05:00:25,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.59 | bwd: 5884.72 | bwd_inner: 5646.09 | bwd_allreduce: 238.59 | step: 19.96

  1%|          | 355/41250 [51:56<99:02:59,  8.72s/it]
                                                      
{'loss': 0.1827, 'grad_norm': 1.6383450031280518, 'learning_rate': 1.1470113085621972e-05, 'epoch': 0.09}

  1%|          | 355/41250 [51:56<99:02:59,  8.72s/it][2025-04-26 05:00:34,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.18 | optimizer_step: 1.14
[2025-04-26 05:00:34,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.61 | bwd_microstep: 5778.84 | bwd_inner_microstep: 5630.95 | bwd_allreduce_microstep: 147.84 | step_microstep: 19.72
[2025-04-26 05:00:34,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.61 | bwd: 5778.85 | bwd_inner: 5630.94 | bwd_allreduce: 147.86 | step: 19.72

  1%|          | 356/41250 [52:05<98:57:53,  8.71s/it]
                                                      
{'loss': 0.4461, 'grad_norm': 4.207483291625977, 'learning_rate': 1.1502423263327949e-05, 'epoch': 0.09}

  1%|          | 356/41250 [52:05<98:57:53,  8.71s/it][2025-04-26 05:00:42,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 05:00:42,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.60 | bwd_microstep: 5703.73 | bwd_inner_microstep: 5638.50 | bwd_allreduce_microstep: 65.18 | step_microstep: 19.26
[2025-04-26 05:00:42,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.60 | bwd: 5703.74 | bwd_inner: 5638.50 | bwd_allreduce: 65.20 | step: 19.27

  1%|          | 357/41250 [52:14<98:37:05,  8.68s/it]
                                                      
{'loss': 0.1973, 'grad_norm': 1.824783205986023, 'learning_rate': 1.1534733441033927e-05, 'epoch': 0.09}

  1%|          | 357/41250 [52:14<98:37:05,  8.68s/it][2025-04-26 05:00:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:00:51,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.75 | bwd_microstep: 5748.29 | bwd_inner_microstep: 5678.96 | bwd_allreduce_microstep: 69.27 | step_microstep: 18.91
[2025-04-26 05:00:51,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.75 | bwd: 5748.30 | bwd_inner: 5678.95 | bwd_allreduce: 69.29 | step: 18.91

  1%|          | 358/41250 [52:22<98:34:42,  8.68s/it]
                                                      
{'loss': 0.1735, 'grad_norm': 1.8313345909118652, 'learning_rate': 1.1567043618739904e-05, 'epoch': 0.09}

  1%|          | 358/41250 [52:22<98:34:42,  8.68s/it][2025-04-26 05:01:00,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 05:01:00,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.76 | bwd_microstep: 5703.12 | bwd_inner_microstep: 5647.51 | bwd_allreduce_microstep: 55.56 | step_microstep: 19.71
[2025-04-26 05:01:00,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.76 | bwd: 5703.14 | bwd_inner: 5647.51 | bwd_allreduce: 55.58 | step: 19.72

  1%|          | 359/41250 [52:31<98:21:08,  8.66s/it]
                                                      
{'loss': 0.2046, 'grad_norm': 1.8066058158874512, 'learning_rate': 1.1599353796445882e-05, 'epoch': 0.09}

  1%|          | 359/41250 [52:31<98:21:08,  8.66s/it][2025-04-26 05:01:08,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 05:01:08,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5755.52 | bwd_inner_microstep: 5650.08 | bwd_allreduce_microstep: 105.40 | step_microstep: 18.84
[2025-04-26 05:01:08,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5755.54 | bwd_inner: 5650.08 | bwd_allreduce: 105.42 | step: 18.84

  1%|          | 360/41250 [52:39<98:22:53,  8.66s/it]
                                                      
{'loss': 0.3583, 'grad_norm': 2.9145736694335938, 'learning_rate': 1.1631663974151859e-05, 'epoch': 0.09}

  1%|          | 360/41250 [52:40<98:22:53,  8.66s/it][2025-04-26 05:01:17,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:01:17,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.74 | bwd_microstep: 5761.16 | bwd_inner_microstep: 5643.03 | bwd_allreduce_microstep: 118.08 | step_microstep: 18.82
[2025-04-26 05:01:17,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.74 | bwd: 5761.17 | bwd_inner: 5643.03 | bwd_allreduce: 118.10 | step: 18.83

  1%|          | 361/41250 [52:48<98:24:05,  8.66s/it]
                                                      
{'loss': 0.3796, 'grad_norm': 2.7818315029144287, 'learning_rate': 1.1663974151857835e-05, 'epoch': 0.09}

  1%|          | 361/41250 [52:48<98:24:05,  8.66s/it][2025-04-26 05:01:26,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 05:01:26,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.60 | bwd_microstep: 5728.84 | bwd_inner_microstep: 5716.02 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.14
[2025-04-26 05:01:26,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.60 | bwd: 5728.85 | bwd_inner: 5716.02 | bwd_allreduce: 12.79 | step: 19.14

  1%|          | 362/41250 [52:57<98:22:36,  8.66s/it]
                                                      
{'loss': 0.3549, 'grad_norm': 3.640393018722534, 'learning_rate': 1.1696284329563814e-05, 'epoch': 0.09}

  1%|          | 362/41250 [52:57<98:22:36,  8.66s/it][2025-04-26 05:01:34,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-26 05:01:34,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.72 | bwd_microstep: 5765.58 | bwd_inner_microstep: 5662.21 | bwd_allreduce_microstep: 103.33 | step_microstep: 18.68
[2025-04-26 05:01:34,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.72 | bwd: 5765.59 | bwd_inner: 5662.21 | bwd_allreduce: 103.34 | step: 18.68

  1%|          | 363/41250 [53:06<98:26:04,  8.67s/it]
                                                      
{'loss': 0.1583, 'grad_norm': 1.5457130670547485, 'learning_rate': 1.172859450726979e-05, 'epoch': 0.09}

  1%|          | 363/41250 [53:06<98:26:04,  8.67s/it][2025-04-26 05:01:43,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:01:43,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5727.44 | bwd_inner_microstep: 5713.84 | bwd_allreduce_microstep: 13.56 | step_microstep: 18.65
[2025-04-26 05:01:43,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5727.46 | bwd_inner: 5713.84 | bwd_allreduce: 13.58 | step: 18.66

  1%|          | 364/41250 [53:14<98:25:51,  8.67s/it]
                                                      
{'loss': 0.2888, 'grad_norm': 2.0299646854400635, 'learning_rate': 1.1760904684975768e-05, 'epoch': 0.09}

  1%|          | 364/41250 [53:14<98:25:51,  8.67s/it][2025-04-26 05:01:52,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:01:52,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.82 | bwd_microstep: 5720.10 | bwd_inner_microstep: 5707.20 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.60
[2025-04-26 05:01:52,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.82 | bwd: 5720.12 | bwd_inner: 5707.20 | bwd_allreduce: 12.87 | step: 18.60

  1%|          | 365/41250 [53:23<98:22:17,  8.66s/it]
                                                      
{'loss': 0.0952, 'grad_norm': 1.1008167266845703, 'learning_rate': 1.1793214862681746e-05, 'epoch': 0.09}

  1%|          | 365/41250 [53:23<98:22:17,  8.66s/it][2025-04-26 05:02:00,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.92
[2025-04-26 05:02:00,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5759.71 | bwd_inner_microstep: 5690.73 | bwd_allreduce_microstep: 68.93 | step_microstep: 18.53
[2025-04-26 05:02:00,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5759.73 | bwd_inner: 5690.73 | bwd_allreduce: 68.95 | step: 18.53

  1%|          | 366/41250 [53:32<98:27:55,  8.67s/it]
                                                      
{'loss': 0.1964, 'grad_norm': 1.5694880485534668, 'learning_rate': 1.1825525040387724e-05, 'epoch': 0.09}

  1%|          | 366/41250 [53:32<98:27:55,  8.67s/it][2025-04-26 05:02:09,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-26 05:02:09,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5691.94 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 33.04 | step_microstep: 19.29
[2025-04-26 05:02:09,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5691.96 | bwd_inner: 5658.86 | bwd_allreduce: 33.05 | step: 19.30

  1%|          | 367/41250 [53:40<98:12:49,  8.65s/it]
                                                      
{'loss': 0.3274, 'grad_norm': 3.141649007797241, 'learning_rate': 1.18578352180937e-05, 'epoch': 0.09}

  1%|          | 367/41250 [53:40<98:12:49,  8.65s/it][2025-04-26 05:02:18,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 05:02:18,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.27 | bwd_microstep: 5763.72 | bwd_inner_microstep: 5658.90 | bwd_allreduce_microstep: 104.78 | step_microstep: 18.81
[2025-04-26 05:02:18,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.27 | bwd: 5763.74 | bwd_inner: 5658.90 | bwd_allreduce: 104.80 | step: 18.82

  1%|          | 368/41250 [53:49<98:19:38,  8.66s/it]
                                                      
{'loss': 0.0588, 'grad_norm': 0.9694452285766602, 'learning_rate': 1.1890145395799677e-05, 'epoch': 0.09}

  1%|          | 368/41250 [53:49<98:19:38,  8.66s/it][2025-04-26 05:02:26,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:02:26,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.87 | bwd_microstep: 5790.28 | bwd_inner_microstep: 5777.39 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.94
[2025-04-26 05:02:26,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.87 | bwd: 5790.29 | bwd_inner: 5777.39 | bwd_allreduce: 12.87 | step: 18.94

  1%|          | 369/41250 [53:58<98:40:58,  8.69s/it]
                                                      
{'loss': 0.2803, 'grad_norm': 1.700662612915039, 'learning_rate': 1.1922455573505656e-05, 'epoch': 0.09}

  1%|          | 369/41250 [53:58<98:40:58,  8.69s/it][2025-04-26 05:02:35,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 05:02:35,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 5707.76 | bwd_inner_microstep: 5694.46 | bwd_allreduce_microstep: 13.25 | step_microstep: 18.78
[2025-04-26 05:02:35,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 5707.78 | bwd_inner: 5694.46 | bwd_allreduce: 13.27 | step: 18.78

  1%|          | 370/41250 [54:06<98:29:18,  8.67s/it]
                                                      
{'loss': 0.2541, 'grad_norm': 2.786924362182617, 'learning_rate': 1.1954765751211632e-05, 'epoch': 0.09}

  1%|          | 370/41250 [54:06<98:29:18,  8.67s/it][2025-04-26 05:02:44,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.18
[2025-04-26 05:02:44,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.16 | bwd_microstep: 5725.16 | bwd_inner_microstep: 5711.99 | bwd_allreduce_microstep: 13.13 | step_microstep: 19.26
[2025-04-26 05:02:44,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.16 | bwd: 5725.17 | bwd_inner: 5711.99 | bwd_allreduce: 13.15 | step: 19.27

  1%|          | 371/41250 [54:15<98:25:02,  8.67s/it]
                                                      
{'loss': 0.2103, 'grad_norm': 2.058300495147705, 'learning_rate': 1.198707592891761e-05, 'epoch': 0.09}

  1%|          | 371/41250 [54:15<98:25:02,  8.67s/it][2025-04-26 05:02:52,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 05:02:52,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.37 | bwd_microstep: 5779.76 | bwd_inner_microstep: 5696.11 | bwd_allreduce_microstep: 83.60 | step_microstep: 18.96
[2025-04-26 05:02:52,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.37 | bwd: 5779.78 | bwd_inner: 5696.11 | bwd_allreduce: 83.62 | step: 18.96

  1%|          | 372/41250 [54:24<98:34:07,  8.68s/it]
                                                      
{'loss': 0.2513, 'grad_norm': 1.8624917268753052, 'learning_rate': 1.2019386106623588e-05, 'epoch': 0.09}

  1%|          | 372/41250 [54:24<98:34:07,  8.68s/it][2025-04-26 05:03:01,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:03:01,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.60 | bwd_microstep: 5724.75 | bwd_inner_microstep: 5712.02 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.04
[2025-04-26 05:03:01,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.60 | bwd: 5724.77 | bwd_inner: 5712.02 | bwd_allreduce: 12.70 | step: 18.04

  1%|          | 373/41250 [54:32<98:29:35,  8.67s/it]
                                                      
{'loss': 0.1501, 'grad_norm': 1.555681586265564, 'learning_rate': 1.2051696284329564e-05, 'epoch': 0.09}

  1%|          | 373/41250 [54:32<98:29:35,  8.67s/it][2025-04-26 05:03:10,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:03:10,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.41 | bwd_microstep: 5704.01 | bwd_inner_microstep: 5671.27 | bwd_allreduce_microstep: 32.70 | step_microstep: 18.11
[2025-04-26 05:03:10,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.41 | bwd: 5704.02 | bwd_inner: 5671.27 | bwd_allreduce: 32.71 | step: 18.11

  1%|          | 374/41250 [54:41<98:17:37,  8.66s/it]
                                                      
{'loss': 0.2402, 'grad_norm': 2.850234270095825, 'learning_rate': 1.2084006462035542e-05, 'epoch': 0.09}

  1%|          | 374/41250 [54:41<98:17:37,  8.66s/it][2025-04-26 05:03:19,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 05:03:19,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.52 | bwd_microstep: 5879.61 | bwd_inner_microstep: 5866.71 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.39
[2025-04-26 05:03:19,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2929.52 | bwd: 5879.62 | bwd_inner: 5866.70 | bwd_allreduce: 12.88 | step: 19.40

  1%|          | 375/41250 [54:50<99:06:20,  8.73s/it]
                                                      
{'loss': 0.3687, 'grad_norm': 3.955528974533081, 'learning_rate': 1.2116316639741518e-05, 'epoch': 0.09}

  1%|          | 375/41250 [54:50<99:06:20,  8.73s/it][2025-04-26 05:03:27,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 05:03:27,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.18 | bwd_microstep: 5706.80 | bwd_inner_microstep: 5694.00 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.00
[2025-04-26 05:03:27,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.18 | bwd: 5706.81 | bwd_inner: 5694.00 | bwd_allreduce: 12.77 | step: 19.00

  1%|          | 376/41250 [54:58<98:46:01,  8.70s/it]
                                                      
{'loss': 0.2335, 'grad_norm': 2.1271755695343018, 'learning_rate': 1.2148626817447498e-05, 'epoch': 0.09}

  1%|          | 376/41250 [54:58<98:46:01,  8.70s/it][2025-04-26 05:03:36,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:03:36,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.96 | bwd_microstep: 5778.45 | bwd_inner_microstep: 5656.21 | bwd_allreduce_microstep: 122.20 | step_microstep: 18.41
[2025-04-26 05:03:36,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.96 | bwd: 5778.47 | bwd_inner: 5656.21 | bwd_allreduce: 122.22 | step: 18.41

  1%|          | 377/41250 [55:07<98:43:34,  8.70s/it]
                                                      
{'loss': 0.1336, 'grad_norm': 1.8229703903198242, 'learning_rate': 1.2180936995153474e-05, 'epoch': 0.09}

  1%|          | 377/41250 [55:07<98:43:34,  8.70s/it][2025-04-26 05:03:45,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:03:45,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.64 | bwd_microstep: 5792.08 | bwd_inner_microstep: 5656.08 | bwd_allreduce_microstep: 135.95 | step_microstep: 18.68
[2025-04-26 05:03:45,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.64 | bwd: 5792.10 | bwd_inner: 5656.08 | bwd_allreduce: 135.97 | step: 18.68

  1%|          | 378/41250 [55:16<98:45:00,  8.70s/it]
                                                      
{'loss': 0.2098, 'grad_norm': 2.4686119556427, 'learning_rate': 1.2213247172859452e-05, 'epoch': 0.09}

  1%|          | 378/41250 [55:16<98:45:00,  8.70s/it][2025-04-26 05:03:53,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.17 | optimizer_step: 1.09
[2025-04-26 05:03:53,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.40 | bwd_microstep: 5741.11 | bwd_inner_microstep: 5698.70 | bwd_allreduce_microstep: 42.36 | step_microstep: 19.62
[2025-04-26 05:03:53,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.40 | bwd: 5741.12 | bwd_inner: 5698.70 | bwd_allreduce: 42.38 | step: 19.62

  1%|          | 379/41250 [55:24<98:39:31,  8.69s/it]
                                                      
{'loss': 0.2665, 'grad_norm': 3.1724765300750732, 'learning_rate': 1.224555735056543e-05, 'epoch': 0.09}

  1%|          | 379/41250 [55:24<98:39:31,  8.69s/it][2025-04-26 05:04:02,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:04:02,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5779.12 | bwd_inner_microstep: 5710.08 | bwd_allreduce_microstep: 68.99 | step_microstep: 18.38
[2025-04-26 05:04:02,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.26 | bwd: 5779.13 | bwd_inner: 5710.08 | bwd_allreduce: 69.01 | step: 18.39

  1%|          | 380/41250 [55:33<98:41:57,  8.69s/it]
                                                      
{'loss': 0.2011, 'grad_norm': 1.935837984085083, 'learning_rate': 1.2277867528271406e-05, 'epoch': 0.09}

  1%|          | 380/41250 [55:33<98:41:57,  8.69s/it][2025-04-26 05:04:11,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:04:11,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.60 | bwd_microstep: 5789.23 | bwd_inner_microstep: 5692.40 | bwd_allreduce_microstep: 96.78 | step_microstep: 18.48
[2025-04-26 05:04:11,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.60 | bwd: 5789.24 | bwd_inner: 5692.40 | bwd_allreduce: 96.80 | step: 18.49

  1%|          | 381/41250 [55:42<98:46:07,  8.70s/it]
                                                      
{'loss': 0.1567, 'grad_norm': 1.8867963552474976, 'learning_rate': 1.2310177705977384e-05, 'epoch': 0.09}

  1%|          | 381/41250 [55:42<98:46:07,  8.70s/it][2025-04-26 05:04:19,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:04:19,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.20 | bwd_microstep: 5741.51 | bwd_inner_microstep: 5710.17 | bwd_allreduce_microstep: 31.30 | step_microstep: 18.60
[2025-04-26 05:04:19,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.20 | bwd: 5741.52 | bwd_inner: 5710.17 | bwd_allreduce: 31.31 | step: 18.60

  1%|          | 382/41250 [55:51<98:40:36,  8.69s/it]
                                                      
{'loss': 0.3579, 'grad_norm': 3.1051278114318848, 'learning_rate': 1.234248788368336e-05, 'epoch': 0.09}

  1%|          | 382/41250 [55:51<98:40:36,  8.69s/it][2025-04-26 05:04:28,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:04:28,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.87 | bwd_microstep: 5713.02 | bwd_inner_microstep: 5700.22 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.64
[2025-04-26 05:04:28,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.87 | bwd: 5713.03 | bwd_inner: 5700.22 | bwd_allreduce: 12.77 | step: 18.64

  1%|          | 383/41250 [55:59<98:30:19,  8.68s/it]
                                                      
{'loss': 0.1316, 'grad_norm': 1.3590725660324097, 'learning_rate': 1.237479806138934e-05, 'epoch': 0.09}

  1%|          | 383/41250 [55:59<98:30:19,  8.68s/it][2025-04-26 05:04:37,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:04:37,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.63 | bwd_microstep: 5715.07 | bwd_inner_microstep: 5701.90 | bwd_allreduce_microstep: 13.12 | step_microstep: 18.86
[2025-04-26 05:04:37,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.63 | bwd: 5715.08 | bwd_inner: 5701.90 | bwd_allreduce: 13.14 | step: 18.86

  1%|          | 384/41250 [56:08<98:24:45,  8.67s/it]
                                                      
{'loss': 0.1428, 'grad_norm': 2.760929584503174, 'learning_rate': 1.2407108239095316e-05, 'epoch': 0.09}

  1%|          | 384/41250 [56:08<98:24:45,  8.67s/it][2025-04-26 05:04:45,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.47 | optimizer_step: 1.26
[2025-04-26 05:04:45,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.72 | bwd_microstep: 5857.18 | bwd_inner_microstep: 5706.48 | bwd_allreduce_microstep: 150.65 | step_microstep: 20.85
[2025-04-26 05:04:45,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.72 | bwd: 5857.20 | bwd_inner: 5706.48 | bwd_allreduce: 150.67 | step: 20.86

  1%|          | 385/41250 [56:17<98:49:28,  8.71s/it]
                                                      
{'loss': 0.193, 'grad_norm': 2.464561939239502, 'learning_rate': 1.2439418416801294e-05, 'epoch': 0.09}

  1%|          | 385/41250 [56:17<98:49:28,  8.71s/it][2025-04-26 05:04:54,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 05:04:54,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.82 | bwd_microstep: 5692.86 | bwd_inner_microstep: 5644.69 | bwd_allreduce_microstep: 48.12 | step_microstep: 18.94
[2025-04-26 05:04:54,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.82 | bwd: 5692.88 | bwd_inner: 5644.69 | bwd_allreduce: 48.15 | step: 18.94

  1%|          | 386/41250 [56:25<98:28:31,  8.68s/it]
                                                      
{'loss': 0.2122, 'grad_norm': 3.1858673095703125, 'learning_rate': 1.2471728594507272e-05, 'epoch': 0.09}

  1%|          | 386/41250 [56:25<98:28:31,  8.68s/it][2025-04-26 05:05:03,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 05:05:03,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5721.04 | bwd_inner_microstep: 5708.24 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.64
[2025-04-26 05:05:03,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5721.06 | bwd_inner: 5708.24 | bwd_allreduce: 12.77 | step: 19.64

  1%|          | 387/41250 [56:34<98:25:21,  8.67s/it]
                                                      
{'loss': 0.1726, 'grad_norm': 2.1571192741394043, 'learning_rate': 1.2504038772213248e-05, 'epoch': 0.09}

  1%|          | 387/41250 [56:34<98:25:21,  8.67s/it][2025-04-26 05:05:11,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 05:05:11,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.37 | bwd_microstep: 5723.16 | bwd_inner_microstep: 5688.67 | bwd_allreduce_microstep: 34.43 | step_microstep: 19.03
[2025-04-26 05:05:11,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.37 | bwd: 5723.17 | bwd_inner: 5688.67 | bwd_allreduce: 34.46 | step: 19.03

  1%|          | 388/41250 [56:43<98:20:41,  8.66s/it]
                                                      
{'loss': 0.2697, 'grad_norm': 2.616532564163208, 'learning_rate': 1.2536348949919226e-05, 'epoch': 0.09}

  1%|          | 388/41250 [56:43<98:20:41,  8.66s/it][2025-04-26 05:05:20,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 05:05:20,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.90 | bwd_microstep: 5737.80 | bwd_inner_microstep: 5681.29 | bwd_allreduce_microstep: 56.46 | step_microstep: 18.62
[2025-04-26 05:05:20,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.90 | bwd: 5737.81 | bwd_inner: 5681.29 | bwd_allreduce: 56.48 | step: 18.62

  1%|          | 389/41250 [56:51<98:18:10,  8.66s/it]
                                                      
{'loss': 0.3203, 'grad_norm': 3.840463399887085, 'learning_rate': 1.2568659127625202e-05, 'epoch': 0.09}

  1%|          | 389/41250 [56:51<98:18:10,  8.66s/it][2025-04-26 05:05:29,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-26 05:05:29,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.25 | bwd_microstep: 5705.78 | bwd_inner_microstep: 5692.83 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.30
[2025-04-26 05:05:29,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.25 | bwd: 5705.79 | bwd_inner: 5692.83 | bwd_allreduce: 12.92 | step: 19.31

  1%|          | 390/41250 [57:00<98:12:59,  8.65s/it]
                                                      
{'loss': 0.0558, 'grad_norm': 1.390720248222351, 'learning_rate': 1.2600969305331181e-05, 'epoch': 0.09}

  1%|          | 390/41250 [57:00<98:12:59,  8.65s/it][2025-04-26 05:05:37,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-26 05:05:37,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.22 | bwd_microstep: 5704.74 | bwd_inner_microstep: 5692.08 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.05
[2025-04-26 05:05:37,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.22 | bwd: 5704.75 | bwd_inner: 5692.08 | bwd_allreduce: 12.64 | step: 19.06

  1%|          | 391/41250 [57:08<98:12:36,  8.65s/it]
                                                      
{'loss': 0.0684, 'grad_norm': 1.1462498903274536, 'learning_rate': 1.2633279483037158e-05, 'epoch': 0.09}

  1%|          | 391/41250 [57:08<98:12:36,  8.65s/it][2025-04-26 05:05:46,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 05:05:46,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.75 | bwd_microstep: 5734.89 | bwd_inner_microstep: 5649.93 | bwd_allreduce_microstep: 84.92 | step_microstep: 18.52
[2025-04-26 05:05:46,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.75 | bwd: 5734.91 | bwd_inner: 5649.93 | bwd_allreduce: 84.94 | step: 18.52

  1%|          | 392/41250 [57:17<98:12:14,  8.65s/it]
                                                      
{'loss': 0.0952, 'grad_norm': 1.9061877727508545, 'learning_rate': 1.2665589660743136e-05, 'epoch': 0.1}

  1%|          | 392/41250 [57:17<98:12:14,  8.65s/it][2025-04-26 05:05:55,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.29 | optimizer_step: 1.02
[2025-04-26 05:05:55,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.12 | bwd_microstep: 5761.92 | bwd_inner_microstep: 5748.37 | bwd_allreduce_microstep: 13.48 | step_microstep: 19.92
[2025-04-26 05:05:55,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.12 | bwd: 5761.94 | bwd_inner: 5748.37 | bwd_allreduce: 13.51 | step: 19.92

  1%|          | 393/41250 [57:26<98:27:18,  8.68s/it]
                                                      
{'loss': 0.1322, 'grad_norm': 2.6581969261169434, 'learning_rate': 1.2697899838449112e-05, 'epoch': 0.1}

  1%|          | 393/41250 [57:26<98:27:18,  8.68s/it][2025-04-26 05:06:03,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 05:06:03,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.20 | bwd_microstep: 5670.22 | bwd_inner_microstep: 5644.50 | bwd_allreduce_microstep: 25.67 | step_microstep: 18.93
[2025-04-26 05:06:03,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.20 | bwd: 5670.24 | bwd_inner: 5644.50 | bwd_allreduce: 25.69 | step: 18.94

  1%|          | 394/41250 [57:34<98:10:46,  8.65s/it]
                                                      
{'loss': 0.1282, 'grad_norm': 1.5112422704696655, 'learning_rate': 1.273021001615509e-05, 'epoch': 0.1}

  1%|          | 394/41250 [57:34<98:10:46,  8.65s/it][2025-04-26 05:06:12,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 05:06:12,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5728.73 | bwd_inner_microstep: 5689.76 | bwd_allreduce_microstep: 38.92 | step_microstep: 19.36
[2025-04-26 05:06:12,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.70 | bwd: 5728.75 | bwd_inner: 5689.76 | bwd_allreduce: 38.94 | step: 19.37

  1%|          | 395/41250 [57:43<98:13:10,  8.65s/it]
                                                      
{'loss': 0.3754, 'grad_norm': 4.223398208618164, 'learning_rate': 1.2762520193861068e-05, 'epoch': 0.1}

  1%|          | 395/41250 [57:43<98:13:10,  8.65s/it][2025-04-26 05:06:21,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:06:21,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.88 | bwd_microstep: 5745.80 | bwd_inner_microstep: 5629.87 | bwd_allreduce_microstep: 115.89 | step_microstep: 18.48
[2025-04-26 05:06:21,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.88 | bwd: 5745.81 | bwd_inner: 5629.87 | bwd_allreduce: 115.90 | step: 18.48

  1%|          | 396/41250 [57:52<98:13:48,  8.66s/it]
                                                      
{'loss': 0.3052, 'grad_norm': 5.058220386505127, 'learning_rate': 1.2794830371567044e-05, 'epoch': 0.1}

  1%|          | 396/41250 [57:52<98:13:48,  8.66s/it][2025-04-26 05:06:29,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 1.12
[2025-04-26 05:06:29,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.15 | bwd_microstep: 5713.77 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.28
[2025-04-26 05:06:29,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.15 | bwd: 5713.78 | bwd_inner: 5700.83 | bwd_allreduce: 12.90 | step: 19.28

  1%|          | 397/41250 [58:00<98:15:17,  8.66s/it]
                                                      
{'loss': 0.1168, 'grad_norm': 2.5110366344451904, 'learning_rate': 1.2827140549273023e-05, 'epoch': 0.1}

  1%|          | 397/41250 [58:00<98:15:17,  8.66s/it][2025-04-26 05:06:38,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 05:06:38,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.47 | bwd_microstep: 5717.73 | bwd_inner_microstep: 5704.79 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.45
[2025-04-26 05:06:38,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.47 | bwd: 5717.75 | bwd_inner: 5704.79 | bwd_allreduce: 12.91 | step: 19.45

  1%|          | 398/41250 [58:09<98:16:11,  8.66s/it]
                                                      
{'loss': 0.2111, 'grad_norm': 2.5356411933898926, 'learning_rate': 1.2859450726979e-05, 'epoch': 0.1}

  1%|          | 398/41250 [58:09<98:16:11,  8.66s/it][2025-04-26 05:06:47,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:06:47,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.83 | bwd_microstep: 5744.14 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 88.53 | step_microstep: 18.69
[2025-04-26 05:06:47,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.83 | bwd: 5744.15 | bwd_inner: 5655.56 | bwd_allreduce: 88.55 | step: 18.69

  1%|          | 399/41250 [58:18<98:16:00,  8.66s/it]
                                                      
{'loss': 0.1699, 'grad_norm': 2.8370144367218018, 'learning_rate': 1.2891760904684977e-05, 'epoch': 0.1}

  1%|          | 399/41250 [58:18<98:16:00,  8.66s/it][2025-04-26 05:06:55,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.27 | optimizer_step: 1.02
[2025-04-26 05:06:55,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.79 | bwd_microstep: 5745.03 | bwd_inner_microstep: 5657.67 | bwd_allreduce_microstep: 87.30 | step_microstep: 19.37
[2025-04-26 05:06:55,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.79 | bwd: 5745.04 | bwd_inner: 5657.67 | bwd_allreduce: 87.33 | step: 19.37

  1%|          | 400/41250 [58:26<98:16:58,  8.66s/it]
                                                      
{'loss': 0.2536, 'grad_norm': 3.678739309310913, 'learning_rate': 1.2924071082390954e-05, 'epoch': 0.1}

  1%|          | 400/41250 [58:26<98:16:58,  8.66s/it][2025-04-26 05:07:04,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:07:04,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.83 | bwd_microstep: 5764.44 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 109.16 | step_microstep: 18.77
[2025-04-26 05:07:04,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.83 | bwd: 5764.46 | bwd_inner: 5655.24 | bwd_allreduce: 109.18 | step: 18.77

  1%|          | 401/41250 [58:35<98:22:30,  8.67s/it]
                                                      
{'loss': 0.3246, 'grad_norm': 3.8619418144226074, 'learning_rate': 1.2956381260096931e-05, 'epoch': 0.1}

  1%|          | 401/41250 [58:35<98:22:30,  8.67s/it][2025-04-26 05:07:13,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.16 | optimizer_step: 1.08
[2025-04-26 05:07:13,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.58 | bwd_microstep: 5699.92 | bwd_inner_microstep: 5651.90 | bwd_allreduce_microstep: 47.96 | step_microstep: 20.37
[2025-04-26 05:07:13,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.58 | bwd: 5699.94 | bwd_inner: 5651.90 | bwd_allreduce: 47.99 | step: 20.37

  1%|          | 402/41250 [58:44<98:09:58,  8.65s/it]
                                                      
{'loss': 0.2129, 'grad_norm': 3.100719928741455, 'learning_rate': 1.298869143780291e-05, 'epoch': 0.1}

  1%|          | 402/41250 [58:44<98:09:58,  8.65s/it][2025-04-26 05:07:21,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 05:07:21,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.01 | bwd_microstep: 5738.26 | bwd_inner_microstep: 5701.60 | bwd_allreduce_microstep: 36.61 | step_microstep: 18.71
[2025-04-26 05:07:21,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.01 | bwd: 5738.27 | bwd_inner: 5701.60 | bwd_allreduce: 36.63 | step: 18.71

  1%|          | 403/41250 [58:52<98:13:36,  8.66s/it]
                                                      
{'loss': 0.2352, 'grad_norm': 2.4184060096740723, 'learning_rate': 1.3021001615508886e-05, 'epoch': 0.1}

  1%|          | 403/41250 [58:52<98:13:36,  8.66s/it][2025-04-26 05:07:30,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:07:30,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.91 | bwd_microstep: 5715.45 | bwd_inner_microstep: 5698.44 | bwd_allreduce_microstep: 16.96 | step_microstep: 18.50
[2025-04-26 05:07:30,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.91 | bwd: 5715.46 | bwd_inner: 5698.44 | bwd_allreduce: 16.98 | step: 18.51

  1%|          | 404/41250 [59:01<98:15:25,  8.66s/it]
                                                      
{'loss': 0.2862, 'grad_norm': 5.245853424072266, 'learning_rate': 1.3053311793214865e-05, 'epoch': 0.1}

  1%|          | 404/41250 [59:01<98:15:25,  8.66s/it][2025-04-26 05:07:39,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:07:39,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.31 | bwd_microstep: 5756.66 | bwd_inner_microstep: 5658.36 | bwd_allreduce_microstep: 98.25 | step_microstep: 18.40
[2025-04-26 05:07:39,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.31 | bwd: 5756.67 | bwd_inner: 5658.36 | bwd_allreduce: 98.27 | step: 18.41

  1%|          | 405/41250 [59:10<98:19:31,  8.67s/it]
                                                      
{'loss': 0.2501, 'grad_norm': 5.908132076263428, 'learning_rate': 1.3085621970920841e-05, 'epoch': 0.1}

  1%|          | 405/41250 [59:10<98:19:31,  8.67s/it][2025-04-26 05:07:47,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 05:07:47,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5693.96 | bwd_inner_microstep: 5680.53 | bwd_allreduce_microstep: 13.38 | step_microstep: 18.85
[2025-04-26 05:07:47,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.37 | bwd: 5693.98 | bwd_inner: 5680.53 | bwd_allreduce: 13.40 | step: 18.85

  1%|          | 406/41250 [59:18<98:10:09,  8.65s/it]
                                                      
{'loss': 0.2521, 'grad_norm': 3.986961841583252, 'learning_rate': 1.3117932148626819e-05, 'epoch': 0.1}

  1%|          | 406/41250 [59:18<98:10:09,  8.65s/it][2025-04-26 05:07:56,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 05:07:56,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.34 | bwd_microstep: 5763.93 | bwd_inner_microstep: 5694.65 | bwd_allreduce_microstep: 69.24 | step_microstep: 18.59
[2025-04-26 05:07:56,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.34 | bwd: 5763.95 | bwd_inner: 5694.65 | bwd_allreduce: 69.26 | step: 18.60

  1%|          | 407/41250 [59:27<98:16:21,  8.66s/it]
                                                      
{'loss': 0.2121, 'grad_norm': 4.398636817932129, 'learning_rate': 1.3150242326332795e-05, 'epoch': 0.1}

  1%|          | 407/41250 [59:27<98:16:21,  8.66s/it][2025-04-26 05:08:05,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:08:05,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.33 | bwd_microstep: 5783.85 | bwd_inner_microstep: 5648.57 | bwd_allreduce_microstep: 135.24 | step_microstep: 18.51
[2025-04-26 05:08:05,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.33 | bwd: 5783.87 | bwd_inner: 5648.57 | bwd_allreduce: 135.26 | step: 18.51

  1%|          | 408/41250 [59:36<98:23:19,  8.67s/it]
                                                      
{'loss': 0.2125, 'grad_norm': 2.209606409072876, 'learning_rate': 1.3182552504038773e-05, 'epoch': 0.1}

  1%|          | 408/41250 [59:36<98:23:19,  8.67s/it][2025-04-26 05:08:13,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:08:13,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.21 | bwd_microstep: 5763.41 | bwd_inner_microstep: 5656.13 | bwd_allreduce_microstep: 107.23 | step_microstep: 18.29
[2025-04-26 05:08:13,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.21 | bwd: 5763.42 | bwd_inner: 5656.13 | bwd_allreduce: 107.25 | step: 18.30

  1%|          | 409/41250 [59:44<98:25:10,  8.68s/it]
                                                      
{'loss': 0.0874, 'grad_norm': 2.5073812007904053, 'learning_rate': 1.3214862681744751e-05, 'epoch': 0.1}

  1%|          | 409/41250 [59:44<98:25:10,  8.68s/it][2025-04-26 05:08:22,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 05:08:22,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.95 | bwd_microstep: 5885.30 | bwd_inner_microstep: 5658.41 | bwd_allreduce_microstep: 226.85 | step_microstep: 19.24
[2025-04-26 05:08:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.95 | bwd: 5885.32 | bwd_inner: 5658.41 | bwd_allreduce: 226.87 | step: 19.24

  1%|          | 410/41250 [59:53<98:51:10,  8.71s/it]
                                                      
{'loss': 0.0745, 'grad_norm': 1.4790430068969727, 'learning_rate': 1.3247172859450727e-05, 'epoch': 0.1}

  1%|          | 410/41250 [59:53<98:51:10,  8.71s/it][2025-04-26 05:08:31,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:08:31,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.60 | bwd_microstep: 5700.19 | bwd_inner_microstep: 5659.81 | bwd_allreduce_microstep: 40.34 | step_microstep: 18.48
[2025-04-26 05:08:31,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.60 | bwd: 5700.20 | bwd_inner: 5659.80 | bwd_allreduce: 40.35 | step: 18.49

  1%|          | 411/41250 [1:00:02<98:30:45,  8.68s/it]
                                                        
{'loss': 0.1253, 'grad_norm': 2.0495340824127197, 'learning_rate': 1.3279483037156707e-05, 'epoch': 0.1}

  1%|          | 411/41250 [1:00:02<98:30:45,  8.68s/it][2025-04-26 05:08:39,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:08:39,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.88 | bwd_microstep: 5718.26 | bwd_inner_microstep: 5705.31 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.45
[2025-04-26 05:08:39,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.88 | bwd: 5718.27 | bwd_inner: 5705.31 | bwd_allreduce: 12.92 | step: 18.45

  1%|          | 412/41250 [1:00:10<98:25:39,  8.68s/it]
                                                        
{'loss': 0.1008, 'grad_norm': 1.6103346347808838, 'learning_rate': 1.3311793214862683e-05, 'epoch': 0.1}

  1%|          | 412/41250 [1:00:10<98:25:39,  8.68s/it][2025-04-26 05:08:48,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:08:48,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.21 | bwd_microstep: 5800.35 | bwd_inner_microstep: 5643.42 | bwd_allreduce_microstep: 156.88 | step_microstep: 18.36
[2025-04-26 05:08:48,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.21 | bwd: 5800.36 | bwd_inner: 5643.42 | bwd_allreduce: 156.90 | step: 18.36

  1%|          | 413/41250 [1:00:19<98:32:03,  8.69s/it]
                                                        
{'loss': 0.3358, 'grad_norm': 3.717531681060791, 'learning_rate': 1.334410339256866e-05, 'epoch': 0.1}

  1%|          | 413/41250 [1:00:19<98:32:03,  8.69s/it][2025-04-26 05:08:57,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:08:57,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.06 | bwd_microstep: 5707.64 | bwd_inner_microstep: 5662.00 | bwd_allreduce_microstep: 45.59 | step_microstep: 18.40
[2025-04-26 05:08:57,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.06 | bwd: 5707.65 | bwd_inner: 5662.00 | bwd_allreduce: 45.61 | step: 18.40

  1%|          | 414/41250 [1:00:28<98:19:36,  8.67s/it]
                                                        
{'loss': 0.2319, 'grad_norm': 4.584634780883789, 'learning_rate': 1.3376413570274637e-05, 'epoch': 0.1}

  1%|          | 414/41250 [1:00:28<98:19:36,  8.67s/it][2025-04-26 05:09:05,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 05:09:05,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.26 | bwd_microstep: 5716.65 | bwd_inner_microstep: 5662.99 | bwd_allreduce_microstep: 53.61 | step_microstep: 18.87
[2025-04-26 05:09:05,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.26 | bwd: 5716.66 | bwd_inner: 5662.99 | bwd_allreduce: 53.63 | step: 18.87

  1%|          | 415/41250 [1:00:36<98:10:53,  8.66s/it]
                                                        
{'loss': 0.2424, 'grad_norm': 4.867087364196777, 'learning_rate': 1.3408723747980615e-05, 'epoch': 0.1}

  1%|          | 415/41250 [1:00:36<98:10:53,  8.66s/it][2025-04-26 05:09:14,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 05:09:14,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.05 | bwd_microstep: 5769.85 | bwd_inner_microstep: 5648.91 | bwd_allreduce_microstep: 120.89 | step_microstep: 18.70
[2025-04-26 05:09:14,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.05 | bwd: 5769.86 | bwd_inner: 5648.91 | bwd_allreduce: 120.91 | step: 18.70

  1%|          | 416/41250 [1:00:45<98:14:45,  8.66s/it]
                                                        
{'loss': 0.1508, 'grad_norm': 2.9224634170532227, 'learning_rate': 1.3441033925686593e-05, 'epoch': 0.1}

  1%|          | 416/41250 [1:00:45<98:14:45,  8.66s/it][2025-04-26 05:09:23,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:09:23,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.40 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5705.05 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.55
[2025-04-26 05:09:23,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.40 | bwd: 5717.85 | bwd_inner: 5705.05 | bwd_allreduce: 12.75 | step: 18.55

  1%|          | 417/41250 [1:00:54<98:12:08,  8.66s/it]
                                                        
{'loss': 0.6005, 'grad_norm': 6.197237491607666, 'learning_rate': 1.3473344103392569e-05, 'epoch': 0.1}

  1%|          | 417/41250 [1:00:54<98:12:08,  8.66s/it][2025-04-26 05:09:31,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:09:31,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.00 | bwd_microstep: 5862.34 | bwd_inner_microstep: 5682.04 | bwd_allreduce_microstep: 180.26 | step_microstep: 18.46
[2025-04-26 05:09:31,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.00 | bwd: 5862.35 | bwd_inner: 5682.04 | bwd_allreduce: 180.27 | step: 18.46

  1%|          | 418/41250 [1:01:03<98:38:07,  8.70s/it]
                                                        
{'loss': 0.1412, 'grad_norm': 2.3713345527648926, 'learning_rate': 1.3505654281098549e-05, 'epoch': 0.1}

  1%|          | 418/41250 [1:01:03<98:38:07,  8.70s/it][2025-04-26 05:09:40,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:09:40,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.21 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5657.23 | bwd_allreduce_microstep: 109.77 | step_microstep: 18.52
[2025-04-26 05:09:40,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.21 | bwd: 5767.06 | bwd_inner: 5657.23 | bwd_allreduce: 109.79 | step: 18.52

  1%|          | 419/41250 [1:01:11<98:37:01,  8.69s/it]
                                                        
{'loss': 0.3323, 'grad_norm': 5.672318935394287, 'learning_rate': 1.3537964458804525e-05, 'epoch': 0.1}

  1%|          | 419/41250 [1:01:11<98:37:01,  8.69s/it][2025-04-26 05:09:49,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:09:49,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.39 | bwd_microstep: 5771.35 | bwd_inner_microstep: 5649.71 | bwd_allreduce_microstep: 121.60 | step_microstep: 18.59
[2025-04-26 05:09:49,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.39 | bwd: 5771.37 | bwd_inner: 5649.71 | bwd_allreduce: 121.62 | step: 18.59

  1%|          | 420/41250 [1:01:20<98:34:06,  8.69s/it]
                                                        
{'loss': 0.2646, 'grad_norm': 3.790409803390503, 'learning_rate': 1.3570274636510501e-05, 'epoch': 0.1}

  1%|          | 420/41250 [1:01:20<98:34:06,  8.69s/it][2025-04-26 05:09:57,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 05:09:57,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5714.97 | bwd_inner_microstep: 5656.83 | bwd_allreduce_microstep: 58.09 | step_microstep: 19.49
[2025-04-26 05:09:57,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5714.98 | bwd_inner: 5656.83 | bwd_allreduce: 58.11 | step: 19.49

  1%|          | 421/41250 [1:01:29<98:19:44,  8.67s/it]
                                                        
{'loss': 0.265, 'grad_norm': 4.034595489501953, 'learning_rate': 1.3602584814216479e-05, 'epoch': 0.1}

  1%|          | 421/41250 [1:01:29<98:19:44,  8.67s/it][2025-04-26 05:10:06,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 05:10:06,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.16 | bwd_microstep: 5886.00 | bwd_inner_microstep: 5691.00 | bwd_allreduce_microstep: 194.96 | step_microstep: 18.71
[2025-04-26 05:10:06,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.16 | bwd: 5886.02 | bwd_inner: 5691.00 | bwd_allreduce: 194.98 | step: 18.71

  1%|          | 422/41250 [1:01:37<98:49:07,  8.71s/it]
                                                        
{'loss': 0.3677, 'grad_norm': 4.8485894203186035, 'learning_rate': 1.3634894991922457e-05, 'epoch': 0.1}

  1%|          | 422/41250 [1:01:37<98:49:07,  8.71s/it][2025-04-26 05:10:15,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:10:15,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.13 | bwd_microstep: 5705.13 | bwd_inner_microstep: 5682.71 | bwd_allreduce_microstep: 22.38 | step_microstep: 18.57
[2025-04-26 05:10:15,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.13 | bwd: 5705.15 | bwd_inner: 5682.71 | bwd_allreduce: 22.39 | step: 18.57

  1%|          | 423/41250 [1:01:46<98:33:43,  8.69s/it]
                                                        
{'loss': 0.1675, 'grad_norm': 3.517143726348877, 'learning_rate': 1.3667205169628435e-05, 'epoch': 0.1}

  1%|          | 423/41250 [1:01:46<98:33:43,  8.69s/it][2025-04-26 05:10:24,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:10:24,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.45 | bwd_microstep: 5761.66 | bwd_inner_microstep: 5707.35 | bwd_allreduce_microstep: 54.27 | step_microstep: 18.59
[2025-04-26 05:10:24,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.45 | bwd: 5761.68 | bwd_inner: 5707.35 | bwd_allreduce: 54.29 | step: 18.60

  1%|          | 424/41250 [1:01:55<98:35:03,  8.69s/it]
                                                        
{'loss': 0.34, 'grad_norm': 3.302319288253784, 'learning_rate': 1.3699515347334411e-05, 'epoch': 0.1}

  1%|          | 424/41250 [1:01:55<98:35:03,  8.69s/it][2025-04-26 05:10:32,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 05:10:32,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.38 | bwd_microstep: 5720.36 | bwd_inner_microstep: 5649.77 | bwd_allreduce_microstep: 70.54 | step_microstep: 19.27
[2025-04-26 05:10:32,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.38 | bwd: 5720.38 | bwd_inner: 5649.77 | bwd_allreduce: 70.56 | step: 19.27

  1%|          | 425/41250 [1:02:03<98:20:20,  8.67s/it]
                                                        
{'loss': 0.259, 'grad_norm': 3.024672508239746, 'learning_rate': 1.373182552504039e-05, 'epoch': 0.1}

  1%|          | 425/41250 [1:02:03<98:20:20,  8.67s/it][2025-04-26 05:10:41,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 05:10:41,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.76 | bwd_microstep: 5761.42 | bwd_inner_microstep: 5694.51 | bwd_allreduce_microstep: 66.86 | step_microstep: 19.43
[2025-04-26 05:10:41,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.76 | bwd: 5761.44 | bwd_inner: 5694.51 | bwd_allreduce: 66.88 | step: 19.43

  1%|          | 426/41250 [1:02:12<98:24:36,  8.68s/it]
                                                        
{'loss': 0.269, 'grad_norm': 2.499396324157715, 'learning_rate': 1.3764135702746367e-05, 'epoch': 0.1}

  1%|          | 426/41250 [1:02:12<98:24:36,  8.68s/it][2025-04-26 05:10:49,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 05:10:49,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5715.73 | bwd_inner_microstep: 5703.01 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.99
[2025-04-26 05:10:49,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.95 | bwd: 5715.75 | bwd_inner: 5703.01 | bwd_allreduce: 12.70 | step: 18.99

  1%|          | 427/41250 [1:02:21<98:19:07,  8.67s/it]
                                                        
{'loss': 0.1602, 'grad_norm': 2.7637076377868652, 'learning_rate': 1.3796445880452343e-05, 'epoch': 0.1}

  1%|          | 427/41250 [1:02:21<98:19:07,  8.67s/it][2025-04-26 05:10:58,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 05:10:58,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.40 | bwd_microstep: 5748.17 | bwd_inner_microstep: 5683.40 | bwd_allreduce_microstep: 64.73 | step_microstep: 18.80
[2025-04-26 05:10:58,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.40 | bwd: 5748.18 | bwd_inner: 5683.39 | bwd_allreduce: 64.75 | step: 18.81

  1%|          | 428/41250 [1:02:29<98:20:47,  8.67s/it]
                                                        
{'loss': 0.2056, 'grad_norm': 2.5410284996032715, 'learning_rate': 1.382875605815832e-05, 'epoch': 0.1}

  1%|          | 428/41250 [1:02:29<98:20:47,  8.67s/it][2025-04-26 05:11:07,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:11:07,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.83 | bwd_microstep: 5704.91 | bwd_inner_microstep: 5642.86 | bwd_allreduce_microstep: 62.01 | step_microstep: 18.59
[2025-04-26 05:11:07,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.83 | bwd: 5704.92 | bwd_inner: 5642.86 | bwd_allreduce: 62.02 | step: 18.59

  1%|          | 429/41250 [1:02:38<98:10:33,  8.66s/it]
                                                        
{'loss': 0.4255, 'grad_norm': 3.9576029777526855, 'learning_rate': 1.3861066235864299e-05, 'epoch': 0.1}

  1%|          | 429/41250 [1:02:38<98:10:33,  8.66s/it][2025-04-26 05:11:15,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:11:15,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.18 | bwd_microstep: 5696.39 | bwd_inner_microstep: 5654.38 | bwd_allreduce_microstep: 41.96 | step_microstep: 18.54
[2025-04-26 05:11:15,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.18 | bwd: 5696.40 | bwd_inner: 5654.38 | bwd_allreduce: 41.98 | step: 18.55

  1%|          | 430/41250 [1:02:47<97:58:15,  8.64s/it]
                                                        
{'loss': 0.1821, 'grad_norm': 1.977439284324646, 'learning_rate': 1.3893376413570277e-05, 'epoch': 0.1}

  1%|          | 430/41250 [1:02:47<97:58:15,  8.64s/it][2025-04-26 05:11:24,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:11:24,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.14 | bwd_microstep: 5707.74 | bwd_inner_microstep: 5694.99 | bwd_allreduce_microstep: 12.70 | step_microstep: 20.13
[2025-04-26 05:11:24,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.14 | bwd: 5707.75 | bwd_inner: 5694.99 | bwd_allreduce: 12.72 | step: 20.14

  1%|          | 431/41250 [1:02:55<97:57:55,  8.64s/it]
                                                        
{'loss': 0.1013, 'grad_norm': 1.2682029008865356, 'learning_rate': 1.3925686591276253e-05, 'epoch': 0.1}

  1%|          | 431/41250 [1:02:55<97:57:55,  8.64s/it][2025-04-26 05:11:33,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-26 05:11:33,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.22 | bwd_microstep: 5700.10 | bwd_inner_microstep: 5687.06 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.27
[2025-04-26 05:11:33,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.22 | bwd: 5700.12 | bwd_inner: 5687.06 | bwd_allreduce: 13.01 | step: 19.27

  1%|          | 432/41250 [1:03:04<97:54:57,  8.64s/it]
                                                        
{'loss': 0.2435, 'grad_norm': 2.832223415374756, 'learning_rate': 1.3957996768982232e-05, 'epoch': 0.1}

  1%|          | 432/41250 [1:03:04<97:54:57,  8.64s/it][2025-04-26 05:11:41,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:11:41,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.51 | bwd_microstep: 5705.28 | bwd_inner_microstep: 5692.41 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.51
[2025-04-26 05:11:41,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.51 | bwd: 5705.29 | bwd_inner: 5692.41 | bwd_allreduce: 12.85 | step: 18.51

  1%|          | 433/41250 [1:03:12<97:54:27,  8.64s/it]
                                                        
{'loss': 0.2724, 'grad_norm': 3.3474042415618896, 'learning_rate': 1.3990306946688208e-05, 'epoch': 0.1}

  1%|          | 433/41250 [1:03:12<97:54:27,  8.64s/it][2025-04-26 05:11:50,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:11:50,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.23 | bwd_microstep: 5777.13 | bwd_inner_microstep: 5639.15 | bwd_allreduce_microstep: 137.94 | step_microstep: 17.91
[2025-04-26 05:11:50,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.23 | bwd: 5777.15 | bwd_inner: 5639.15 | bwd_allreduce: 137.96 | step: 17.91

  1%|          | 434/41250 [1:03:21<98:01:59,  8.65s/it]
                                                        
{'loss': 0.2074, 'grad_norm': 2.1308698654174805, 'learning_rate': 1.4022617124394185e-05, 'epoch': 0.11}

  1%|          | 434/41250 [1:03:21<98:01:59,  8.65s/it][2025-04-26 05:11:59,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:11:59,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.19 | bwd_microstep: 5765.85 | bwd_inner_microstep: 5636.08 | bwd_allreduce_microstep: 129.73 | step_microstep: 18.28
[2025-04-26 05:11:59,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.19 | bwd: 5765.86 | bwd_inner: 5636.08 | bwd_allreduce: 129.74 | step: 18.29

  1%|          | 435/41250 [1:03:30<98:06:16,  8.65s/it]
                                                        
{'loss': 0.3018, 'grad_norm': 4.124579429626465, 'learning_rate': 1.4054927302100163e-05, 'epoch': 0.11}

  1%|          | 435/41250 [1:03:30<98:06:16,  8.65s/it][2025-04-26 05:12:07,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:12:07,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.69 | bwd_microstep: 5753.45 | bwd_inner_microstep: 5642.76 | bwd_allreduce_microstep: 110.65 | step_microstep: 18.33
[2025-04-26 05:12:07,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.69 | bwd: 5753.46 | bwd_inner: 5642.76 | bwd_allreduce: 110.66 | step: 18.33

  1%|          | 436/41250 [1:03:38<98:05:42,  8.65s/it]
                                                        
{'loss': 0.0837, 'grad_norm': 0.9370715022087097, 'learning_rate': 1.4087237479806139e-05, 'epoch': 0.11}

  1%|          | 436/41250 [1:03:38<98:05:42,  8.65s/it][2025-04-26 05:12:16,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 05:12:16,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.09 | bwd_microstep: 5770.79 | bwd_inner_microstep: 5648.12 | bwd_allreduce_microstep: 122.62 | step_microstep: 18.59
[2025-04-26 05:12:16,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.09 | bwd: 5770.80 | bwd_inner: 5648.12 | bwd_allreduce: 122.64 | step: 18.60

  1%|          | 437/41250 [1:03:47<98:09:20,  8.66s/it]
                                                        
{'loss': 0.3442, 'grad_norm': 4.28505277633667, 'learning_rate': 1.4119547657512118e-05, 'epoch': 0.11}

  1%|          | 437/41250 [1:03:47<98:09:20,  8.66s/it][2025-04-26 05:12:25,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 05:12:25,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.72 | bwd_microstep: 5711.36 | bwd_inner_microstep: 5630.83 | bwd_allreduce_microstep: 80.48 | step_microstep: 18.65
[2025-04-26 05:12:25,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.72 | bwd: 5711.37 | bwd_inner: 5630.83 | bwd_allreduce: 80.50 | step: 18.65

  1%|          | 438/41250 [1:03:56<98:00:24,  8.65s/it]
                                                        
{'loss': 0.0968, 'grad_norm': 1.1108334064483643, 'learning_rate': 1.4151857835218094e-05, 'epoch': 0.11}

  1%|          | 438/41250 [1:03:56<98:00:24,  8.65s/it][2025-04-26 05:12:33,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:12:33,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.23 | bwd_microstep: 5715.06 | bwd_inner_microstep: 5702.68 | bwd_allreduce_microstep: 12.33 | step_microstep: 18.57
[2025-04-26 05:12:33,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.23 | bwd: 5715.07 | bwd_inner: 5702.68 | bwd_allreduce: 12.34 | step: 18.58

  1%|          | 439/41250 [1:04:04<97:58:40,  8.64s/it]
                                                        
{'loss': 0.4161, 'grad_norm': 4.972020626068115, 'learning_rate': 1.4184168012924074e-05, 'epoch': 0.11}

  1%|          | 439/41250 [1:04:04<97:58:40,  8.64s/it][2025-04-26 05:12:42,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:12:42,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.57 | bwd_microstep: 5789.48 | bwd_inner_microstep: 5776.83 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.26
[2025-04-26 05:12:42,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.57 | bwd: 5789.49 | bwd_inner: 5776.83 | bwd_allreduce: 12.62 | step: 18.27

  1%|          | 440/41250 [1:04:13<98:21:02,  8.68s/it]
                                                        
{'loss': 0.1498, 'grad_norm': 1.8049044609069824, 'learning_rate': 1.421647819063005e-05, 'epoch': 0.11}

  1%|          | 440/41250 [1:04:13<98:21:02,  8.68s/it][2025-04-26 05:12:51,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:12:51,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.68 | bwd_microstep: 5691.71 | bwd_inner_microstep: 5678.89 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.91
[2025-04-26 05:12:51,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.68 | bwd: 5691.73 | bwd_inner: 5678.89 | bwd_allreduce: 12.80 | step: 18.91

  1%|          | 441/41250 [1:04:22<98:08:46,  8.66s/it]
                                                        
{'loss': 0.3553, 'grad_norm': 4.3460164070129395, 'learning_rate': 1.4248788368336026e-05, 'epoch': 0.11}

  1%|          | 441/41250 [1:04:22<98:08:46,  8.66s/it][2025-04-26 05:12:59,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 05:12:59,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.92 | bwd_microstep: 5683.43 | bwd_inner_microstep: 5634.51 | bwd_allreduce_microstep: 48.87 | step_microstep: 19.51
[2025-04-26 05:12:59,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.92 | bwd: 5683.44 | bwd_inner: 5634.51 | bwd_allreduce: 48.89 | step: 19.51

  1%|          | 442/41250 [1:04:30<97:55:12,  8.64s/it]
                                                        
{'loss': 0.3581, 'grad_norm': 4.466716289520264, 'learning_rate': 1.4281098546042004e-05, 'epoch': 0.11}

  1%|          | 442/41250 [1:04:30<97:55:12,  8.64s/it][2025-04-26 05:13:08,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:13:08,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.66 | bwd_microstep: 5773.87 | bwd_inner_microstep: 5683.17 | bwd_allreduce_microstep: 90.65 | step_microstep: 18.11
[2025-04-26 05:13:08,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.66 | bwd: 5773.88 | bwd_inner: 5683.17 | bwd_allreduce: 90.67 | step: 18.11

  1%|          | 443/41250 [1:04:39<98:04:38,  8.65s/it]
                                                        
{'loss': 0.119, 'grad_norm': 1.4349617958068848, 'learning_rate': 1.431340872374798e-05, 'epoch': 0.11}

  1%|          | 443/41250 [1:04:39<98:04:38,  8.65s/it][2025-04-26 05:13:16,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:13:16,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.03 | bwd_microstep: 5701.34 | bwd_inner_microstep: 5642.82 | bwd_allreduce_microstep: 58.48 | step_microstep: 18.57
[2025-04-26 05:13:16,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.03 | bwd: 5701.36 | bwd_inner: 5642.82 | bwd_allreduce: 58.49 | step: 18.57

  1%|          | 444/41250 [1:04:48<97:55:30,  8.64s/it]
                                                        
{'loss': 0.2877, 'grad_norm': 2.9539084434509277, 'learning_rate': 1.434571890145396e-05, 'epoch': 0.11}

  1%|          | 444/41250 [1:04:48<97:55:30,  8.64s/it][2025-04-26 05:13:25,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 05:13:25,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5740.38 | bwd_inner_microstep: 5691.68 | bwd_allreduce_microstep: 48.65 | step_microstep: 18.94
[2025-04-26 05:13:25,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5740.39 | bwd_inner: 5691.68 | bwd_allreduce: 48.67 | step: 18.94

  1%|          | 445/41250 [1:04:56<98:01:46,  8.65s/it]
                                                        
{'loss': 0.2109, 'grad_norm': 2.2045533657073975, 'learning_rate': 1.4378029079159936e-05, 'epoch': 0.11}

  1%|          | 445/41250 [1:04:56<98:01:46,  8.65s/it][2025-04-26 05:13:34,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.27 | optimizer_step: 1.24
[2025-04-26 05:13:34,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5772.91 | bwd_inner_microstep: 5645.10 | bwd_allreduce_microstep: 127.77 | step_microstep: 20.34
[2025-04-26 05:13:34,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5772.93 | bwd_inner: 5645.09 | bwd_allreduce: 127.79 | step: 20.34

  1%|          | 446/41250 [1:05:05<98:09:08,  8.66s/it]
                                                        
{'loss': 0.3563, 'grad_norm': 2.387352705001831, 'learning_rate': 1.4410339256865912e-05, 'epoch': 0.11}

  1%|          | 446/41250 [1:05:05<98:09:08,  8.66s/it][2025-04-26 05:13:42,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:13:42,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.49 | bwd_microstep: 5719.36 | bwd_inner_microstep: 5706.64 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.66
[2025-04-26 05:13:42,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.49 | bwd: 5719.38 | bwd_inner: 5706.64 | bwd_allreduce: 12.70 | step: 18.66

  1%|          | 447/41250 [1:05:14<98:06:54,  8.66s/it]
                                                        
{'loss': 0.1887, 'grad_norm': 1.9967617988586426, 'learning_rate': 1.4442649434571892e-05, 'epoch': 0.11}

  1%|          | 447/41250 [1:05:14<98:06:54,  8.66s/it][2025-04-26 05:13:52,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.28 | optimizer_step: 1.02
[2025-04-26 05:13:52,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.97 | bwd_microstep: 6233.22 | bwd_inner_microstep: 5686.65 | bwd_allreduce_microstep: 546.51 | step_microstep: 19.63
[2025-04-26 05:13:52,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.97 | bwd: 6233.24 | bwd_inner: 5686.65 | bwd_allreduce: 546.54 | step: 19.63

  1%|          | 448/41250 [1:05:23<99:49:39,  8.81s/it]
                                                        
{'loss': 0.1774, 'grad_norm': 3.3780102729797363, 'learning_rate': 1.4474959612277868e-05, 'epoch': 0.11}

  1%|          | 448/41250 [1:05:23<99:49:39,  8.81s/it][2025-04-26 05:14:00,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-26 05:14:00,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.90 | bwd_microstep: 5722.88 | bwd_inner_microstep: 5709.99 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.13
[2025-04-26 05:14:00,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.90 | bwd: 5722.90 | bwd_inner: 5709.99 | bwd_allreduce: 12.86 | step: 19.13

  1%|          | 449/41250 [1:05:31<99:19:40,  8.76s/it]
                                                        
{'loss': 0.2101, 'grad_norm': 2.7984468936920166, 'learning_rate': 1.4507269789983846e-05, 'epoch': 0.11}

  1%|          | 449/41250 [1:05:31<99:19:40,  8.76s/it][2025-04-26 05:14:09,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-26 05:14:09,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5756.96 | bwd_inner_microstep: 5718.27 | bwd_allreduce_microstep: 38.64 | step_microstep: 19.22
[2025-04-26 05:14:09,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5756.97 | bwd_inner: 5718.27 | bwd_allreduce: 38.66 | step: 19.23

  1%|          | 450/41250 [1:05:40<99:05:26,  8.74s/it]
                                                        
{'loss': 0.3161, 'grad_norm': 3.3454952239990234, 'learning_rate': 1.4539579967689822e-05, 'epoch': 0.11}

  1%|          | 450/41250 [1:05:40<99:05:26,  8.74s/it][2025-04-26 05:14:18,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 05:14:18,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5773.88 | bwd_inner_microstep: 5664.25 | bwd_allreduce_microstep: 109.58 | step_microstep: 19.24
[2025-04-26 05:14:18,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5773.89 | bwd_inner: 5664.25 | bwd_allreduce: 109.60 | step: 19.25

  1%|          | 451/41250 [1:05:49<98:54:12,  8.73s/it]
                                                        
{'loss': 0.1473, 'grad_norm': 1.6163544654846191, 'learning_rate': 1.4571890145395802e-05, 'epoch': 0.11}

  1%|          | 451/41250 [1:05:49<98:54:12,  8.73s/it][2025-04-26 05:14:26,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 05:14:26,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.80 | bwd_microstep: 5808.18 | bwd_inner_microstep: 5782.43 | bwd_allreduce_microstep: 25.70 | step_microstep: 18.83
[2025-04-26 05:14:26,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.80 | bwd: 5808.19 | bwd_inner: 5782.43 | bwd_allreduce: 25.72 | step: 18.84

  1%|          | 452/41250 [1:05:58<99:03:19,  8.74s/it]
                                                        
{'loss': 0.1834, 'grad_norm': 2.0455691814422607, 'learning_rate': 1.4604200323101778e-05, 'epoch': 0.11}

  1%|          | 452/41250 [1:05:58<99:03:19,  8.74s/it][2025-04-26 05:14:35,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 05:14:35,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.47 | bwd_microstep: 5752.70 | bwd_inner_microstep: 5717.92 | bwd_allreduce_microstep: 34.73 | step_microstep: 19.20
[2025-04-26 05:14:35,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.47 | bwd: 5752.71 | bwd_inner: 5717.92 | bwd_allreduce: 34.74 | step: 19.20

  1%|          | 453/41250 [1:06:06<98:53:13,  8.73s/it]
                                                        
{'loss': 0.2062, 'grad_norm': 2.006617546081543, 'learning_rate': 1.4636510500807754e-05, 'epoch': 0.11}

  1%|          | 453/41250 [1:06:06<98:53:13,  8.73s/it][2025-04-26 05:14:44,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:14:44,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.88 | bwd_microstep: 5803.41 | bwd_inner_microstep: 5652.18 | bwd_allreduce_microstep: 151.18 | step_microstep: 18.57
[2025-04-26 05:14:44,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.88 | bwd: 5803.42 | bwd_inner: 5652.18 | bwd_allreduce: 151.20 | step: 18.57

  1%|          | 454/41250 [1:06:15<98:49:37,  8.72s/it]
                                                        
{'loss': 0.3127, 'grad_norm': 2.814394474029541, 'learning_rate': 1.4668820678513734e-05, 'epoch': 0.11}

  1%|          | 454/41250 [1:06:15<98:49:37,  8.72s/it][2025-04-26 05:14:53,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 05:14:53,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.89 | bwd_microstep: 5754.28 | bwd_inner_microstep: 5701.18 | bwd_allreduce_microstep: 53.05 | step_microstep: 19.45
[2025-04-26 05:14:53,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.89 | bwd: 5754.30 | bwd_inner: 5701.18 | bwd_allreduce: 53.07 | step: 19.45

  1%|          | 455/41250 [1:06:24<98:42:51,  8.71s/it]
                                                        
{'loss': 0.173, 'grad_norm': 1.7852036952972412, 'learning_rate': 1.470113085621971e-05, 'epoch': 0.11}

  1%|          | 455/41250 [1:06:24<98:42:51,  8.71s/it][2025-04-26 05:15:01,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 05:15:01,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.31 | bwd_microstep: 5799.44 | bwd_inner_microstep: 5653.85 | bwd_allreduce_microstep: 145.54 | step_microstep: 19.00
[2025-04-26 05:15:01,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.31 | bwd: 5799.45 | bwd_inner: 5653.85 | bwd_allreduce: 145.56 | step: 19.00

  1%|          | 456/41250 [1:06:32<98:43:35,  8.71s/it]
                                                        
{'loss': 0.3145, 'grad_norm': 2.6049933433532715, 'learning_rate': 1.4733441033925688e-05, 'epoch': 0.11}

  1%|          | 456/41250 [1:06:32<98:43:35,  8.71s/it][2025-04-26 05:15:10,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 05:15:10,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.96 | bwd_microstep: 5712.82 | bwd_inner_microstep: 5700.03 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.18
[2025-04-26 05:15:10,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.96 | bwd: 5712.83 | bwd_inner: 5700.03 | bwd_allreduce: 12.75 | step: 19.19

  1%|          | 457/41250 [1:06:41<98:28:05,  8.69s/it]
                                                        
{'loss': 0.1994, 'grad_norm': 1.6099953651428223, 'learning_rate': 1.4765751211631664e-05, 'epoch': 0.11}

  1%|          | 457/41250 [1:06:41<98:28:05,  8.69s/it][2025-04-26 05:15:19,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 05:15:19,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5758.33 | bwd_inner_microstep: 5716.08 | bwd_allreduce_microstep: 42.20 | step_microstep: 19.24
[2025-04-26 05:15:19,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5758.34 | bwd_inner: 5716.08 | bwd_allreduce: 42.22 | step: 19.24

  1%|          | 458/41250 [1:06:50<98:29:43,  8.69s/it]
                                                        
{'loss': 0.4753, 'grad_norm': 3.169851541519165, 'learning_rate': 1.4798061389337644e-05, 'epoch': 0.11}

  1%|          | 458/41250 [1:06:50<98:29:43,  8.69s/it][2025-04-26 05:15:27,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 05:15:27,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.12 | bwd_microstep: 5726.99 | bwd_inner_microstep: 5714.24 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.67
[2025-04-26 05:15:27,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.12 | bwd: 5727.00 | bwd_inner: 5714.24 | bwd_allreduce: 12.72 | step: 18.67

  1%|          | 459/41250 [1:06:58<98:24:16,  8.68s/it]
                                                        
{'loss': 0.2936, 'grad_norm': 1.9130498170852661, 'learning_rate': 1.483037156704362e-05, 'epoch': 0.11}

  1%|          | 459/41250 [1:06:58<98:24:16,  8.68s/it][2025-04-26 05:15:36,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.19
[2025-04-26 05:15:36,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5766.68 | bwd_inner_microstep: 5703.53 | bwd_allreduce_microstep: 63.10 | step_microstep: 19.51
[2025-04-26 05:15:36,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5766.70 | bwd_inner: 5703.53 | bwd_allreduce: 63.12 | step: 19.52

  1%|          | 460/41250 [1:07:07<98:28:00,  8.69s/it]
                                                        
{'loss': 0.3367, 'grad_norm': 4.147627353668213, 'learning_rate': 1.4862681744749596e-05, 'epoch': 0.11}

  1%|          | 460/41250 [1:07:07<98:28:00,  8.69s/it][2025-04-26 05:15:45,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.09 | optimizer_step: 0.94
[2025-04-26 05:15:45,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.62 | bwd_microstep: 5712.21 | bwd_inner_microstep: 5656.86 | bwd_allreduce_microstep: 55.30 | step_microstep: 19.57
[2025-04-26 05:15:45,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5712.23 | bwd_inner: 5656.86 | bwd_allreduce: 55.32 | step: 19.57

  1%|          | 461/41250 [1:07:16<98:12:40,  8.67s/it]
                                                        
{'loss': 0.1274, 'grad_norm': 1.3727673292160034, 'learning_rate': 1.4894991922455576e-05, 'epoch': 0.11}

  1%|          | 461/41250 [1:07:16<98:12:40,  8.67s/it][2025-04-26 05:15:53,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.18 | optimizer_step: 1.13
[2025-04-26 05:15:53,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.20 | bwd_microstep: 5897.05 | bwd_inner_microstep: 5724.10 | bwd_allreduce_microstep: 172.90 | step_microstep: 19.83
[2025-04-26 05:15:53,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.20 | bwd: 5897.06 | bwd_inner: 5724.10 | bwd_allreduce: 172.92 | step: 19.83

  1%|          | 462/41250 [1:07:25<98:47:13,  8.72s/it]
                                                        
{'loss': 0.2809, 'grad_norm': 3.1105804443359375, 'learning_rate': 1.4927302100161552e-05, 'epoch': 0.11}

  1%|          | 462/41250 [1:07:25<98:47:13,  8.72s/it][2025-04-26 05:16:02,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 05:16:02,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.80 | bwd_microstep: 5693.59 | bwd_inner_microstep: 5663.28 | bwd_allreduce_microstep: 30.25 | step_microstep: 19.13
[2025-04-26 05:16:02,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.80 | bwd: 5693.60 | bwd_inner: 5663.28 | bwd_allreduce: 30.27 | step: 19.14

  1%|          | 463/41250 [1:07:33<98:26:19,  8.69s/it]
                                                        
{'loss': 0.2383, 'grad_norm': 1.6656779050827026, 'learning_rate': 1.495961227786753e-05, 'epoch': 0.11}

  1%|          | 463/41250 [1:07:33<98:26:19,  8.69s/it][2025-04-26 05:16:11,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 05:16:11,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.65 | bwd_microstep: 5696.34 | bwd_inner_microstep: 5656.55 | bwd_allreduce_microstep: 39.75 | step_microstep: 19.23
[2025-04-26 05:16:11,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.65 | bwd: 5696.36 | bwd_inner: 5656.55 | bwd_allreduce: 39.77 | step: 19.23

  1%|          | 464/41250 [1:07:42<98:13:13,  8.67s/it]
                                                        
{'loss': 0.0595, 'grad_norm': 1.1024481058120728, 'learning_rate': 1.4991922455573506e-05, 'epoch': 0.11}

  1%|          | 464/41250 [1:07:42<98:13:13,  8.67s/it][2025-04-26 05:16:19,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:16:19,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.14 | bwd_microstep: 5771.25 | bwd_inner_microstep: 5688.83 | bwd_allreduce_microstep: 82.37 | step_microstep: 18.25
[2025-04-26 05:16:19,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.14 | bwd: 5771.27 | bwd_inner: 5688.83 | bwd_allreduce: 82.39 | step: 18.26

  1%|          | 465/41250 [1:07:51<98:19:29,  8.68s/it]
                                                        
{'loss': 0.2107, 'grad_norm': 1.8787086009979248, 'learning_rate': 1.5024232633279485e-05, 'epoch': 0.11}

  1%|          | 465/41250 [1:07:51<98:19:29,  8.68s/it][2025-04-26 05:16:28,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 05:16:28,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.95 | bwd_microstep: 5710.94 | bwd_inner_microstep: 5659.53 | bwd_allreduce_microstep: 51.36 | step_microstep: 18.99
[2025-04-26 05:16:28,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.95 | bwd: 5710.95 | bwd_inner: 5659.53 | bwd_allreduce: 51.39 | step: 18.99

  1%|          | 466/41250 [1:07:59<98:07:33,  8.66s/it]
                                                        
{'loss': 0.5068, 'grad_norm': 3.667059898376465, 'learning_rate': 1.5056542810985462e-05, 'epoch': 0.11}

  1%|          | 466/41250 [1:07:59<98:07:33,  8.66s/it][2025-04-26 05:16:37,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:16:37,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.78 | bwd_microstep: 5756.53 | bwd_inner_microstep: 5690.93 | bwd_allreduce_microstep: 65.56 | step_microstep: 18.54
[2025-04-26 05:16:37,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.78 | bwd: 5756.54 | bwd_inner: 5690.93 | bwd_allreduce: 65.57 | step: 18.55

  1%|          | 467/41250 [1:08:08<98:14:05,  8.67s/it]
                                                        
{'loss': 0.1015, 'grad_norm': 1.3006775379180908, 'learning_rate': 1.5088852988691438e-05, 'epoch': 0.11}

  1%|          | 467/41250 [1:08:08<98:14:05,  8.67s/it][2025-04-26 05:16:45,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:16:45,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5759.74 | bwd_inner_microstep: 5655.19 | bwd_allreduce_microstep: 104.50 | step_microstep: 18.09
[2025-04-26 05:16:45,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5759.75 | bwd_inner: 5655.19 | bwd_allreduce: 104.52 | step: 18.09

  1%|          | 468/41250 [1:08:17<98:13:37,  8.67s/it]
                                                        
{'loss': 0.2604, 'grad_norm': 2.2397568225860596, 'learning_rate': 1.5121163166397417e-05, 'epoch': 0.11}

  1%|          | 468/41250 [1:08:17<98:13:37,  8.67s/it][2025-04-26 05:16:54,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.07 | optimizer_step: 1.05
[2025-04-26 05:16:54,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.11 | bwd_microstep: 5707.94 | bwd_inner_microstep: 5650.22 | bwd_allreduce_microstep: 57.67 | step_microstep: 19.58
[2025-04-26 05:16:54,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.11 | bwd: 5707.95 | bwd_inner: 5650.22 | bwd_allreduce: 57.69 | step: 19.59

  1%|          | 469/41250 [1:08:25<98:01:47,  8.65s/it]
                                                        
{'loss': 0.1966, 'grad_norm': 1.7040001153945923, 'learning_rate': 1.5153473344103394e-05, 'epoch': 0.11}

  1%|          | 469/41250 [1:08:25<98:01:47,  8.65s/it][2025-04-26 05:17:03,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 1.15
[2025-04-26 05:17:03,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.69 | bwd_microstep: 5759.26 | bwd_inner_microstep: 5651.08 | bwd_allreduce_microstep: 108.14 | step_microstep: 19.21
[2025-04-26 05:17:03,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.69 | bwd: 5759.28 | bwd_inner: 5651.07 | bwd_allreduce: 108.15 | step: 19.21

  1%|          | 470/41250 [1:08:34<98:04:24,  8.66s/it]
                                                        
{'loss': 0.1952, 'grad_norm': 2.092015266418457, 'learning_rate': 1.5185783521809372e-05, 'epoch': 0.11}

  1%|          | 470/41250 [1:08:34<98:04:24,  8.66s/it][2025-04-26 05:17:12,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 05:17:12,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.38 | bwd_microstep: 5988.27 | bwd_inner_microstep: 5694.32 | bwd_allreduce_microstep: 293.90 | step_microstep: 18.67
[2025-04-26 05:17:12,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.38 | bwd: 5988.28 | bwd_inner: 5694.32 | bwd_allreduce: 293.92 | step: 18.67

  1%|          | 471/41250 [1:08:43<99:00:26,  8.74s/it]
                                                        
{'loss': 0.3089, 'grad_norm': 3.361067295074463, 'learning_rate': 1.5218093699515348e-05, 'epoch': 0.11}

  1%|          | 471/41250 [1:08:43<99:00:26,  8.74s/it][2025-04-26 05:17:20,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:17:20,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.18 | bwd_microstep: 5764.45 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 60.35 | step_microstep: 18.63
[2025-04-26 05:17:20,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.18 | bwd: 5764.47 | bwd_inner: 5704.05 | bwd_allreduce: 60.37 | step: 18.63

  1%|          | 472/41250 [1:08:51<98:52:34,  8.73s/it]
                                                        
{'loss': 0.2, 'grad_norm': 2.028927803039551, 'learning_rate': 1.5250403877221327e-05, 'epoch': 0.11}

  1%|          | 472/41250 [1:08:51<98:52:34,  8.73s/it][2025-04-26 05:17:29,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 05:17:29,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.11 | bwd_microstep: 5790.79 | bwd_inner_microstep: 5777.93 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.95
[2025-04-26 05:17:29,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.11 | bwd: 5790.81 | bwd_inner: 5777.93 | bwd_allreduce: 12.84 | step: 18.96

  1%|          | 473/41250 [1:09:00<98:58:33,  8.74s/it]
                                                        
{'loss': 0.2597, 'grad_norm': 2.103203773498535, 'learning_rate': 1.5282714054927302e-05, 'epoch': 0.11}

  1%|          | 473/41250 [1:09:00<98:58:33,  8.74s/it][2025-04-26 05:17:38,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 05:17:38,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.38 | bwd_microstep: 5697.01 | bwd_inner_microstep: 5684.32 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.66
[2025-04-26 05:17:38,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.38 | bwd: 5697.03 | bwd_inner: 5684.32 | bwd_allreduce: 12.67 | step: 18.66

  1%|          | 474/41250 [1:09:09<98:34:29,  8.70s/it]
                                                        
{'loss': 0.27, 'grad_norm': 2.9734649658203125, 'learning_rate': 1.531502423263328e-05, 'epoch': 0.11}

  1%|          | 474/41250 [1:09:09<98:34:29,  8.70s/it][2025-04-26 05:17:46,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:17:46,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.65 | bwd_microstep: 5695.55 | bwd_inner_microstep: 5682.71 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.65
[2025-04-26 05:17:46,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.65 | bwd: 5695.56 | bwd_inner: 5682.71 | bwd_allreduce: 12.81 | step: 18.65

  1%|          | 475/41250 [1:09:17<98:15:36,  8.68s/it]
                                                        
{'loss': 0.2363, 'grad_norm': 2.120265245437622, 'learning_rate': 1.5347334410339258e-05, 'epoch': 0.12}

  1%|          | 475/41250 [1:09:17<98:15:36,  8.68s/it][2025-04-26 05:17:55,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:17:55,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.81 | bwd_microstep: 5708.92 | bwd_inner_microstep: 5696.17 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.66
[2025-04-26 05:17:55,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.81 | bwd: 5708.94 | bwd_inner: 5696.17 | bwd_allreduce: 12.72 | step: 18.67

  1%|          | 476/41250 [1:09:26<98:07:59,  8.66s/it]
                                                        
{'loss': 0.3009, 'grad_norm': 3.4986202716827393, 'learning_rate': 1.5379644588045235e-05, 'epoch': 0.12}

  1%|          | 476/41250 [1:09:26<98:07:59,  8.66s/it][2025-04-26 05:18:03,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 05:18:03,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.22 | bwd_microstep: 5691.29 | bwd_inner_microstep: 5678.42 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.02
[2025-04-26 05:18:03,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.22 | bwd: 5691.31 | bwd_inner: 5678.42 | bwd_allreduce: 12.84 | step: 19.02

  1%|          | 477/41250 [1:09:35<97:57:26,  8.65s/it]
                                                        
{'loss': 0.2012, 'grad_norm': 1.6855266094207764, 'learning_rate': 1.5411954765751213e-05, 'epoch': 0.12}

  1%|          | 477/41250 [1:09:35<97:57:26,  8.65s/it][2025-04-26 05:18:12,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:18:12,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.50 | bwd_microstep: 5736.92 | bwd_inner_microstep: 5646.30 | bwd_allreduce_microstep: 90.57 | step_microstep: 18.58
[2025-04-26 05:18:12,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.50 | bwd: 5736.93 | bwd_inner: 5646.30 | bwd_allreduce: 90.59 | step: 18.58

  1%|          | 478/41250 [1:09:43<97:58:57,  8.65s/it]
                                                        
{'loss': 0.0898, 'grad_norm': 1.0888867378234863, 'learning_rate': 1.544426494345719e-05, 'epoch': 0.12}

  1%|          | 478/41250 [1:09:43<97:58:57,  8.65s/it][2025-04-26 05:18:21,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:18:21,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5900.57 | bwd_inner_microstep: 5651.23 | bwd_allreduce_microstep: 249.30 | step_microstep: 18.47
[2025-04-26 05:18:21,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5900.58 | bwd_inner: 5651.23 | bwd_allreduce: 249.31 | step: 18.47

  1%|          | 479/41250 [1:09:52<98:29:31,  8.70s/it]
                                                        
{'loss': 0.3005, 'grad_norm': 2.588681936264038, 'learning_rate': 1.547657512116317e-05, 'epoch': 0.12}

  1%|          | 479/41250 [1:09:52<98:29:31,  8.70s/it][2025-04-26 05:18:30,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:18:30,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.61 | bwd_microstep: 5747.76 | bwd_inner_microstep: 5672.63 | bwd_allreduce_microstep: 75.08 | step_microstep: 18.80
[2025-04-26 05:18:30,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.61 | bwd: 5747.77 | bwd_inner: 5672.63 | bwd_allreduce: 75.10 | step: 18.80

  1%|          | 480/41250 [1:10:01<98:26:01,  8.69s/it]
                                                        
{'loss': 0.2681, 'grad_norm': 2.868882894515991, 'learning_rate': 1.5508885298869144e-05, 'epoch': 0.12}

  1%|          | 480/41250 [1:10:01<98:26:01,  8.69s/it][2025-04-26 05:18:38,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.08 | optimizer_step: 1.25
[2025-04-26 05:18:38,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5752.99 | bwd_inner_microstep: 5650.10 | bwd_allreduce_microstep: 102.84 | step_microstep: 19.64
[2025-04-26 05:18:38,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5753.00 | bwd_inner: 5650.10 | bwd_allreduce: 102.86 | step: 19.64

  1%|          | 481/41250 [1:10:09<98:20:17,  8.68s/it]
                                                        
{'loss': 0.296, 'grad_norm': 1.7007149457931519, 'learning_rate': 1.554119547657512e-05, 'epoch': 0.12}

  1%|          | 481/41250 [1:10:09<98:20:17,  8.68s/it][2025-04-26 05:18:47,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.15 | optimizer_step: 0.91
[2025-04-26 05:18:47,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.39 | bwd_microstep: 5754.97 | bwd_inner_microstep: 5649.43 | bwd_allreduce_microstep: 105.49 | step_microstep: 18.84
[2025-04-26 05:18:47,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.39 | bwd: 5754.98 | bwd_inner: 5649.43 | bwd_allreduce: 105.50 | step: 18.84

  1%|          | 482/41250 [1:10:18<98:15:59,  8.68s/it]
                                                        
{'loss': 0.1401, 'grad_norm': 1.1811606884002686, 'learning_rate': 1.55735056542811e-05, 'epoch': 0.12}

  1%|          | 482/41250 [1:10:18<98:15:59,  8.68s/it][2025-04-26 05:18:56,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 05:18:56,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.82 | bwd_microstep: 5677.73 | bwd_inner_microstep: 5652.98 | bwd_allreduce_microstep: 24.70 | step_microstep: 19.02
[2025-04-26 05:18:56,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.82 | bwd: 5677.74 | bwd_inner: 5652.98 | bwd_allreduce: 24.72 | step: 19.02

  1%|          | 483/41250 [1:10:27<97:59:14,  8.65s/it]
                                                        
{'loss': 0.1436, 'grad_norm': 1.7481467723846436, 'learning_rate': 1.5605815831987077e-05, 'epoch': 0.12}

  1%|          | 483/41250 [1:10:27<97:59:14,  8.65s/it][2025-04-26 05:19:04,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:19:04,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.29 | bwd_microstep: 5714.77 | bwd_inner_microstep: 5701.78 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.40
[2025-04-26 05:19:04,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.29 | bwd: 5714.78 | bwd_inner: 5701.78 | bwd_allreduce: 12.96 | step: 18.40

  1%|          | 484/41250 [1:10:35<97:57:59,  8.65s/it]
                                                        
{'loss': 0.219, 'grad_norm': 1.945580005645752, 'learning_rate': 1.5638126009693055e-05, 'epoch': 0.12}

  1%|          | 484/41250 [1:10:35<97:57:59,  8.65s/it][2025-04-26 05:19:13,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:19:13,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.98 | bwd_microstep: 5732.35 | bwd_inner_microstep: 5687.57 | bwd_allreduce_microstep: 44.74 | step_microstep: 18.40
[2025-04-26 05:19:13,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.98 | bwd: 5732.37 | bwd_inner: 5687.56 | bwd_allreduce: 44.76 | step: 18.40

  1%|          | 485/41250 [1:10:44<97:58:19,  8.65s/it]
                                                        
{'loss': 0.1666, 'grad_norm': 2.174227714538574, 'learning_rate': 1.5670436187399033e-05, 'epoch': 0.12}

  1%|          | 485/41250 [1:10:44<97:58:19,  8.65s/it][2025-04-26 05:19:22,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:19:22,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.29 | bwd_microstep: 5714.72 | bwd_inner_microstep: 5702.02 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.47
[2025-04-26 05:19:22,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.29 | bwd: 5714.73 | bwd_inner: 5702.02 | bwd_allreduce: 12.67 | step: 18.47

  1%|          | 486/41250 [1:10:53<98:00:40,  8.66s/it]
                                                        
{'loss': 0.1114, 'grad_norm': 2.030491828918457, 'learning_rate': 1.5702746365105007e-05, 'epoch': 0.12}

  1%|          | 486/41250 [1:10:53<98:00:40,  8.66s/it][2025-04-26 05:19:30,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 05:19:30,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5753.35 | bwd_inner_microstep: 5637.04 | bwd_allreduce_microstep: 116.25 | step_microstep: 18.83
[2025-04-26 05:19:30,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.79 | bwd: 5753.36 | bwd_inner: 5637.04 | bwd_allreduce: 116.27 | step: 18.83

  1%|          | 487/41250 [1:11:01<98:02:03,  8.66s/it]
                                                        
{'loss': 0.199, 'grad_norm': 2.5657293796539307, 'learning_rate': 1.5735056542810985e-05, 'epoch': 0.12}

  1%|          | 487/41250 [1:11:01<98:02:03,  8.66s/it][2025-04-26 05:19:39,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 05:19:39,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5695.79 | bwd_inner_microstep: 5655.81 | bwd_allreduce_microstep: 39.92 | step_microstep: 19.75
[2025-04-26 05:19:39,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5695.80 | bwd_inner: 5655.81 | bwd_allreduce: 39.94 | step: 19.75

  1%|          | 488/41250 [1:11:10<97:49:51,  8.64s/it]
                                                        
{'loss': 0.243, 'grad_norm': 1.8511440753936768, 'learning_rate': 1.5767366720516963e-05, 'epoch': 0.12}

  1%|          | 488/41250 [1:11:10<97:49:51,  8.64s/it][2025-04-26 05:19:47,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 05:19:47,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.11 | bwd_microstep: 5725.12 | bwd_inner_microstep: 5709.98 | bwd_allreduce_microstep: 15.09 | step_microstep: 18.86
[2025-04-26 05:19:47,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.11 | bwd: 5725.13 | bwd_inner: 5709.98 | bwd_allreduce: 15.11 | step: 18.86

  1%|          | 489/41250 [1:11:19<97:53:43,  8.65s/it]
                                                        
{'loss': 0.1785, 'grad_norm': 2.085629463195801, 'learning_rate': 1.579967689822294e-05, 'epoch': 0.12}

  1%|          | 489/41250 [1:11:19<97:53:43,  8.65s/it][2025-04-26 05:19:56,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:19:56,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.71 | bwd_microstep: 5903.17 | bwd_inner_microstep: 5646.41 | bwd_allreduce_microstep: 256.72 | step_microstep: 18.71
[2025-04-26 05:19:56,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.71 | bwd: 5903.19 | bwd_inner: 5646.41 | bwd_allreduce: 256.74 | step: 18.72

  1%|          | 490/41250 [1:11:27<98:26:31,  8.69s/it]
                                                        
{'loss': 0.2391, 'grad_norm': 2.6709022521972656, 'learning_rate': 1.583198707592892e-05, 'epoch': 0.12}

  1%|          | 490/41250 [1:11:27<98:26:31,  8.69s/it][2025-04-26 05:20:05,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 05:20:05,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5774.76 | bwd_inner_microstep: 5697.69 | bwd_allreduce_microstep: 77.02 | step_microstep: 19.27
[2025-04-26 05:20:05,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5774.77 | bwd_inner: 5697.69 | bwd_allreduce: 77.04 | step: 19.27

  1%|          | 491/41250 [1:11:36<98:28:34,  8.70s/it]
                                                        
{'loss': 0.2235, 'grad_norm': 2.632011890411377, 'learning_rate': 1.5864297253634897e-05, 'epoch': 0.12}

  1%|          | 491/41250 [1:11:36<98:28:34,  8.70s/it][2025-04-26 05:20:14,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 05:20:14,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 5702.93 | bwd_inner_microstep: 5690.24 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.88
[2025-04-26 05:20:14,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.17 | bwd: 5702.95 | bwd_inner: 5690.24 | bwd_allreduce: 12.66 | step: 18.88

  1%|          | 492/41250 [1:11:45<98:14:30,  8.68s/it]
                                                        
{'loss': 0.2911, 'grad_norm': 3.8667869567871094, 'learning_rate': 1.5896607431340875e-05, 'epoch': 0.12}

  1%|          | 492/41250 [1:11:45<98:14:30,  8.68s/it][2025-04-26 05:20:22,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-26 05:20:22,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.27 | bwd_microstep: 5787.15 | bwd_inner_microstep: 5649.37 | bwd_allreduce_microstep: 137.73 | step_microstep: 19.26
[2025-04-26 05:20:22,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.27 | bwd: 5787.16 | bwd_inner: 5649.37 | bwd_allreduce: 137.75 | step: 19.27

  1%|          | 493/41250 [1:11:53<98:18:04,  8.68s/it]
                                                        
{'loss': 0.1161, 'grad_norm': 1.4493578672409058, 'learning_rate': 1.592891760904685e-05, 'epoch': 0.12}

  1%|          | 493/41250 [1:11:53<98:18:04,  8.68s/it][2025-04-26 05:20:31,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 05:20:31,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.88 | bwd_microstep: 5770.83 | bwd_inner_microstep: 5656.98 | bwd_allreduce_microstep: 113.80 | step_microstep: 19.40
[2025-04-26 05:20:31,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.88 | bwd: 5770.84 | bwd_inner: 5656.98 | bwd_allreduce: 113.82 | step: 19.41

  1%|          | 494/41250 [1:12:02<98:22:12,  8.69s/it]
                                                        
{'loss': 0.284, 'grad_norm': 2.8689072132110596, 'learning_rate': 1.5961227786752827e-05, 'epoch': 0.12}

  1%|          | 494/41250 [1:12:02<98:22:12,  8.69s/it][2025-04-26 05:20:40,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.53 | optimizer_step: 0.93
[2025-04-26 05:20:40,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.86 | bwd_microstep: 5788.75 | bwd_inner_microstep: 5648.29 | bwd_allreduce_microstep: 140.41 | step_microstep: 19.66
[2025-04-26 05:20:40,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.86 | bwd: 5788.77 | bwd_inner: 5648.29 | bwd_allreduce: 140.43 | step: 19.66

  1%|          | 495/41250 [1:12:11<98:25:57,  8.69s/it]
                                                        
{'loss': 0.1203, 'grad_norm': 1.7931019067764282, 'learning_rate': 1.5993537964458805e-05, 'epoch': 0.12}

  1%|          | 495/41250 [1:12:11<98:25:57,  8.69s/it][2025-04-26 05:20:48,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:20:48,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.90 | bwd_microstep: 5703.17 | bwd_inner_microstep: 5690.36 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.63
[2025-04-26 05:20:48,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.90 | bwd: 5703.18 | bwd_inner: 5690.36 | bwd_allreduce: 12.78 | step: 18.64

  1%|          | 496/41250 [1:12:20<98:16:51,  8.68s/it]
                                                        
{'loss': 0.1551, 'grad_norm': 1.2201642990112305, 'learning_rate': 1.6025848142164783e-05, 'epoch': 0.12}

  1%|          | 496/41250 [1:12:20<98:16:51,  8.68s/it][2025-04-26 05:20:57,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 05:20:57,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5812.80 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 112.58 | step_microstep: 19.01
[2025-04-26 05:20:57,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.18 | bwd: 5812.81 | bwd_inner: 5700.17 | bwd_allreduce: 112.60 | step: 19.02

  1%|          | 497/41250 [1:12:28<98:29:21,  8.70s/it]
                                                        
{'loss': 0.2668, 'grad_norm': 2.5277554988861084, 'learning_rate': 1.605815831987076e-05, 'epoch': 0.12}

  1%|          | 497/41250 [1:12:28<98:29:21,  8.70s/it][2025-04-26 05:21:06,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 05:21:06,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.00 | bwd_microstep: 5755.08 | bwd_inner_microstep: 5709.91 | bwd_allreduce_microstep: 45.13 | step_microstep: 18.71
[2025-04-26 05:21:06,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.00 | bwd: 5755.09 | bwd_inner: 5709.91 | bwd_allreduce: 45.14 | step: 18.72

  1%|          | 498/41250 [1:12:37<98:31:14,  8.70s/it]
                                                        
{'loss': 0.4029, 'grad_norm': 3.387378215789795, 'learning_rate': 1.609046849757674e-05, 'epoch': 0.12}

  1%|          | 498/41250 [1:12:37<98:31:14,  8.70s/it][2025-04-26 05:21:15,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:21:15,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5726.38 | bwd_inner_microstep: 5713.53 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.57
[2025-04-26 05:21:15,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5726.39 | bwd_inner: 5713.53 | bwd_allreduce: 12.82 | step: 18.57

  1%|          | 499/41250 [1:12:46<98:55:27,  8.74s/it]
                                                        
{'loss': 0.1574, 'grad_norm': 1.7383604049682617, 'learning_rate': 1.6122778675282717e-05, 'epoch': 0.12}

  1%|          | 499/41250 [1:12:46<98:55:27,  8.74s/it][2025-04-26 05:21:23,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:21:23,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.26 | bwd_microstep: 5774.01 | bwd_inner_microstep: 5661.07 | bwd_allreduce_microstep: 112.89 | step_microstep: 18.44
[2025-04-26 05:21:23,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.26 | bwd: 5774.02 | bwd_inner: 5661.07 | bwd_allreduce: 112.91 | step: 18.45

  1%|          | 500/41250 [1:12:54<98:47:39,  8.73s/it]
                                                        
{'loss': 0.1333, 'grad_norm': 1.491632342338562, 'learning_rate': 1.615508885298869e-05, 'epoch': 0.12}

  1%|          | 500/41250 [1:12:54<98:47:39,  8.73s/it][2025-04-26 05:21:32,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 05:21:32,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.89 | bwd_microstep: 5751.51 | bwd_inner_microstep: 5697.15 | bwd_allreduce_microstep: 54.31 | step_microstep: 18.99
[2025-04-26 05:21:32,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.89 | bwd: 5751.53 | bwd_inner: 5697.15 | bwd_allreduce: 54.33 | step: 18.99

  1%|          | 501/41250 [1:13:03<98:41:03,  8.72s/it]
                                                        
{'loss': 0.1337, 'grad_norm': 1.3806285858154297, 'learning_rate': 1.618739903069467e-05, 'epoch': 0.12}

  1%|          | 501/41250 [1:13:03<98:41:03,  8.72s/it][2025-04-26 05:21:41,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-26 05:21:41,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5753.54 | bwd_inner_microstep: 5700.34 | bwd_allreduce_microstep: 53.14 | step_microstep: 18.94
[2025-04-26 05:21:41,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5753.55 | bwd_inner: 5700.34 | bwd_allreduce: 53.17 | step: 18.94

  1%|          | 502/41250 [1:13:12<98:33:43,  8.71s/it]
                                                        
{'loss': 0.0703, 'grad_norm': 1.0880002975463867, 'learning_rate': 1.6219709208400647e-05, 'epoch': 0.12}

  1%|          | 502/41250 [1:13:12<98:33:43,  8.71s/it][2025-04-26 05:21:49,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:21:49,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5705.50 | bwd_inner_microstep: 5692.60 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.33
[2025-04-26 05:21:49,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5705.51 | bwd_inner: 5692.60 | bwd_allreduce: 12.87 | step: 18.34

  1%|          | 503/41250 [1:13:21<98:20:29,  8.69s/it]
                                                        
{'loss': 0.3103, 'grad_norm': 3.4459614753723145, 'learning_rate': 1.6252019386106625e-05, 'epoch': 0.12}

  1%|          | 503/41250 [1:13:21<98:20:29,  8.69s/it][2025-04-26 05:21:58,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:21:58,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.72 | bwd_microstep: 5710.72 | bwd_inner_microstep: 5672.80 | bwd_allreduce_microstep: 37.87 | step_microstep: 18.57
[2025-04-26 05:21:58,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.72 | bwd: 5710.73 | bwd_inner: 5672.80 | bwd_allreduce: 37.89 | step: 18.57

  1%|          | 504/41250 [1:13:29<98:08:48,  8.67s/it]
                                                        
{'loss': 0.2147, 'grad_norm': 2.3083994388580322, 'learning_rate': 1.6284329563812603e-05, 'epoch': 0.12}

  1%|          | 504/41250 [1:13:29<98:08:48,  8.67s/it][2025-04-26 05:22:07,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:22:07,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.02 | bwd_microstep: 5711.76 | bwd_inner_microstep: 5654.97 | bwd_allreduce_microstep: 56.74 | step_microstep: 18.30
[2025-04-26 05:22:07,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.02 | bwd: 5711.77 | bwd_inner: 5654.97 | bwd_allreduce: 56.76 | step: 18.30

  1%|          | 505/41250 [1:13:38<97:59:15,  8.66s/it]
                                                        
{'loss': 0.2782, 'grad_norm': 2.7501351833343506, 'learning_rate': 1.631663974151858e-05, 'epoch': 0.12}

  1%|          | 505/41250 [1:13:38<97:59:15,  8.66s/it][2025-04-26 05:22:15,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:22:15,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.43 | bwd_microstep: 5732.10 | bwd_inner_microstep: 5647.86 | bwd_allreduce_microstep: 84.21 | step_microstep: 18.43
[2025-04-26 05:22:15,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.43 | bwd: 5732.12 | bwd_inner: 5647.86 | bwd_allreduce: 84.22 | step: 18.43

  1%|          | 506/41250 [1:13:46<97:55:38,  8.65s/it]
                                                        
{'loss': 0.1966, 'grad_norm': 2.9407644271850586, 'learning_rate': 1.634894991922456e-05, 'epoch': 0.12}

  1%|          | 506/41250 [1:13:46<97:55:38,  8.65s/it][2025-04-26 05:22:24,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-26 05:22:24,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.11 | bwd_microstep: 5722.50 | bwd_inner_microstep: 5709.58 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.63
[2025-04-26 05:22:24,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.11 | bwd: 5722.51 | bwd_inner: 5709.58 | bwd_allreduce: 12.89 | step: 19.64

  1%|          | 507/41250 [1:13:55<97:57:51,  8.66s/it]
                                                        
{'loss': 0.1183, 'grad_norm': 2.2010366916656494, 'learning_rate': 1.6381260096930533e-05, 'epoch': 0.12}

  1%|          | 507/41250 [1:13:55<97:57:51,  8.66s/it][2025-04-26 05:22:33,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:22:33,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.62 | bwd_microstep: 5760.99 | bwd_inner_microstep: 5659.27 | bwd_allreduce_microstep: 101.68 | step_microstep: 18.29
[2025-04-26 05:22:33,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.62 | bwd: 5761.00 | bwd_inner: 5659.27 | bwd_allreduce: 101.69 | step: 18.29

  1%|          | 508/41250 [1:14:04<98:01:45,  8.66s/it]
                                                        
{'loss': 0.284, 'grad_norm': 3.556820869445801, 'learning_rate': 1.641357027463651e-05, 'epoch': 0.12}

  1%|          | 508/41250 [1:14:04<98:01:45,  8.66s/it][2025-04-26 05:22:41,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:22:41,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5773.78 | bwd_inner_microstep: 5705.78 | bwd_allreduce_microstep: 67.95 | step_microstep: 18.34
[2025-04-26 05:22:41,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.26 | bwd: 5773.79 | bwd_inner: 5705.78 | bwd_allreduce: 67.97 | step: 18.35

  1%|          | 509/41250 [1:14:12<98:10:44,  8.68s/it]
                                                        
{'loss': 0.2722, 'grad_norm': 4.457894325256348, 'learning_rate': 1.644588045234249e-05, 'epoch': 0.12}

  1%|          | 509/41250 [1:14:12<98:10:44,  8.68s/it][2025-04-26 05:22:50,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:22:50,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5769.44 | bwd_inner_microstep: 5658.46 | bwd_allreduce_microstep: 110.94 | step_microstep: 17.98
[2025-04-26 05:22:50,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5769.45 | bwd_inner: 5658.46 | bwd_allreduce: 110.95 | step: 17.98

  1%|          | 510/41250 [1:14:21<98:10:44,  8.68s/it]
                                                        
{'loss': 0.1258, 'grad_norm': 2.673365831375122, 'learning_rate': 1.6478190630048467e-05, 'epoch': 0.12}

  1%|          | 510/41250 [1:14:21<98:10:44,  8.68s/it][2025-04-26 05:22:59,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-26 05:22:59,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.69 | bwd_microstep: 5853.43 | bwd_inner_microstep: 5784.12 | bwd_allreduce_microstep: 69.26 | step_microstep: 19.51
[2025-04-26 05:22:59,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.69 | bwd: 5853.44 | bwd_inner: 5784.12 | bwd_allreduce: 69.28 | step: 19.52

  1%|          | 511/41250 [1:14:30<98:40:29,  8.72s/it]
                                                        
{'loss': 0.3017, 'grad_norm': 2.7019102573394775, 'learning_rate': 1.6510500807754444e-05, 'epoch': 0.12}

  1%|          | 511/41250 [1:14:30<98:40:29,  8.72s/it][2025-04-26 05:23:07,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 05:23:07,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5710.73 | bwd_inner_microstep: 5658.82 | bwd_allreduce_microstep: 51.86 | step_microstep: 18.97
[2025-04-26 05:23:07,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5710.74 | bwd_inner: 5658.82 | bwd_allreduce: 51.88 | step: 18.97

  1%|          | 512/41250 [1:14:39<98:20:50,  8.69s/it]
                                                        
{'loss': 0.1277, 'grad_norm': 1.7072821855545044, 'learning_rate': 1.6542810985460422e-05, 'epoch': 0.12}

  1%|          | 512/41250 [1:14:39<98:20:50,  8.69s/it][2025-04-26 05:23:16,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 1.13
[2025-04-26 05:23:16,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3065.75 | bwd_microstep: 5666.19 | bwd_inner_microstep: 5653.34 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.37
[2025-04-26 05:23:16,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3065.75 | bwd: 5666.20 | bwd_inner: 5653.34 | bwd_allreduce: 12.82 | step: 19.37

  1%|          | 513/41250 [1:14:47<98:46:02,  8.73s/it]
                                                        
{'loss': 0.2493, 'grad_norm': 2.4374406337738037, 'learning_rate': 1.65751211631664e-05, 'epoch': 0.12}

  1%|          | 513/41250 [1:14:47<98:46:02,  8.73s/it][2025-04-26 05:23:25,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.29 | optimizer_step: 1.20
[2025-04-26 05:23:25,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.75 | bwd_microstep: 5707.67 | bwd_inner_microstep: 5694.35 | bwd_allreduce_microstep: 13.26 | step_microstep: 20.51
[2025-04-26 05:23:25,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.75 | bwd: 5707.68 | bwd_inner: 5694.35 | bwd_allreduce: 13.28 | step: 20.51

  1%|          | 514/41250 [1:14:56<98:30:42,  8.71s/it]
                                                        
{'loss': 0.278, 'grad_norm': 3.762022018432617, 'learning_rate': 1.6607431340872375e-05, 'epoch': 0.12}

  1%|          | 514/41250 [1:14:56<98:30:42,  8.71s/it][2025-04-26 05:23:34,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.23
[2025-04-26 05:23:34,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5714.56 | bwd_inner_microstep: 5701.80 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.29
[2025-04-26 05:23:34,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5714.58 | bwd_inner: 5701.80 | bwd_allreduce: 12.74 | step: 19.29

  1%|          | 515/41250 [1:15:05<98:17:30,  8.69s/it]
                                                        
{'loss': 0.1835, 'grad_norm': 1.9245582818984985, 'learning_rate': 1.6639741518578353e-05, 'epoch': 0.12}

  1%|          | 515/41250 [1:15:05<98:17:30,  8.69s/it][2025-04-26 05:23:42,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 05:23:42,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.21 | bwd_microstep: 5694.17 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 38.57 | step_microstep: 18.40
[2025-04-26 05:23:42,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.21 | bwd: 5694.18 | bwd_inner: 5655.56 | bwd_allreduce: 38.59 | step: 18.41

  1%|▏         | 516/41250 [1:15:13<98:00:49,  8.66s/it]
                                                        
{'loss': 0.1309, 'grad_norm': 1.7842376232147217, 'learning_rate': 1.667205169628433e-05, 'epoch': 0.13}

  1%|▏         | 516/41250 [1:15:13<98:00:49,  8.66s/it][2025-04-26 05:23:51,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 05:23:51,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.61 | bwd_microstep: 5989.64 | bwd_inner_microstep: 5687.62 | bwd_allreduce_microstep: 301.98 | step_microstep: 18.94
[2025-04-26 05:23:51,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.61 | bwd: 5989.65 | bwd_inner: 5687.62 | bwd_allreduce: 301.99 | step: 18.95

  1%|▏         | 517/41250 [1:15:22<98:54:56,  8.74s/it]
                                                        
{'loss': 0.3061, 'grad_norm': 2.7371208667755127, 'learning_rate': 1.670436187399031e-05, 'epoch': 0.13}

  1%|▏         | 517/41250 [1:15:22<98:54:56,  8.74s/it][2025-04-26 05:24:00,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:24:00,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.74 | bwd_microstep: 5749.33 | bwd_inner_microstep: 5633.22 | bwd_allreduce_microstep: 116.06 | step_microstep: 18.36
[2025-04-26 05:24:00,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.75 | bwd: 5749.34 | bwd_inner: 5633.22 | bwd_allreduce: 116.08 | step: 18.37

  1%|▏         | 518/41250 [1:15:31<98:37:39,  8.72s/it]
                                                        
{'loss': 0.1597, 'grad_norm': 2.427609920501709, 'learning_rate': 1.6736672051696286e-05, 'epoch': 0.13}

  1%|▏         | 518/41250 [1:15:31<98:37:39,  8.72s/it][2025-04-26 05:24:08,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:24:08,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.93 | bwd_microstep: 5718.13 | bwd_inner_microstep: 5705.40 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.50
[2025-04-26 05:24:08,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.94 | bwd: 5718.14 | bwd_inner: 5705.40 | bwd_allreduce: 12.70 | step: 18.50

  1%|▏         | 519/41250 [1:15:40<98:27:47,  8.70s/it]
                                                        
{'loss': 0.0911, 'grad_norm': 1.411826729774475, 'learning_rate': 1.6768982229402264e-05, 'epoch': 0.13}

  1%|▏         | 519/41250 [1:15:40<98:27:47,  8.70s/it][2025-04-26 05:24:17,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 05:24:17,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5699.28 | bwd_inner_microstep: 5648.69 | bwd_allreduce_microstep: 50.54 | step_microstep: 18.87
[2025-04-26 05:24:17,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5699.30 | bwd_inner: 5648.69 | bwd_allreduce: 50.56 | step: 18.88

  1%|▏         | 520/41250 [1:15:48<98:07:43,  8.67s/it]
                                                        
{'loss': 0.2092, 'grad_norm': 4.337844371795654, 'learning_rate': 1.6801292407108242e-05, 'epoch': 0.13}

  1%|▏         | 520/41250 [1:15:48<98:07:43,  8.67s/it][2025-04-26 05:24:26,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 05:24:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.82 | bwd_microstep: 5694.86 | bwd_inner_microstep: 5682.29 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.01
[2025-04-26 05:24:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.82 | bwd: 5694.87 | bwd_inner: 5682.29 | bwd_allreduce: 12.55 | step: 18.01

  1%|▏         | 521/41250 [1:15:57<97:56:14,  8.66s/it]
                                                        
{'loss': 0.1822, 'grad_norm': 3.5752506256103516, 'learning_rate': 1.6833602584814216e-05, 'epoch': 0.13}

  1%|▏         | 521/41250 [1:15:57<97:56:14,  8.66s/it][2025-04-26 05:24:34,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:24:34,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.37 | bwd_microstep: 5778.18 | bwd_inner_microstep: 5648.88 | bwd_allreduce_microstep: 129.26 | step_microstep: 18.08
[2025-04-26 05:24:34,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.36 | bwd: 5778.19 | bwd_inner: 5648.88 | bwd_allreduce: 129.27 | step: 18.08

  1%|▏         | 522/41250 [1:16:05<98:02:00,  8.67s/it]
                                                        
{'loss': 0.4163, 'grad_norm': 4.581690788269043, 'learning_rate': 1.6865912762520194e-05, 'epoch': 0.13}

  1%|▏         | 522/41250 [1:16:05<98:02:00,  8.67s/it][2025-04-26 05:24:43,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:24:43,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.85 | bwd_microstep: 5700.34 | bwd_inner_microstep: 5687.48 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.42
[2025-04-26 05:24:43,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5700.35 | bwd_inner: 5687.48 | bwd_allreduce: 12.83 | step: 18.43

  1%|▏         | 523/41250 [1:16:14<97:55:53,  8.66s/it]
                                                        
{'loss': 0.2292, 'grad_norm': 2.3231751918792725, 'learning_rate': 1.6898222940226172e-05, 'epoch': 0.13}

  1%|▏         | 523/41250 [1:16:14<97:55:53,  8.66s/it][2025-04-26 05:24:52,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 05:24:52,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5763.91 | bwd_inner_microstep: 5656.90 | bwd_allreduce_microstep: 106.96 | step_microstep: 18.84
[2025-04-26 05:24:52,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5763.93 | bwd_inner: 5656.90 | bwd_allreduce: 106.98 | step: 18.84

  1%|▏         | 524/41250 [1:16:23<97:57:51,  8.66s/it]
                                                        
{'loss': 0.2043, 'grad_norm': 2.9994499683380127, 'learning_rate': 1.693053311793215e-05, 'epoch': 0.13}

  1%|▏         | 524/41250 [1:16:23<97:57:51,  8.66s/it][2025-04-26 05:25:00,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:25:00,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.34 | bwd_microstep: 5726.87 | bwd_inner_microstep: 5691.50 | bwd_allreduce_microstep: 35.33 | step_microstep: 18.00
[2025-04-26 05:25:00,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5726.89 | bwd_inner: 5691.50 | bwd_allreduce: 35.34 | step: 18.01

  1%|▏         | 525/41250 [1:16:31<97:58:46,  8.66s/it]
                                                        
{'loss': 0.1992, 'grad_norm': 2.1024529933929443, 'learning_rate': 1.6962843295638128e-05, 'epoch': 0.13}

  1%|▏         | 525/41250 [1:16:31<97:58:46,  8.66s/it][2025-04-26 05:25:09,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.18
[2025-04-26 05:25:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.20 | bwd_microstep: 5748.27 | bwd_inner_microstep: 5708.73 | bwd_allreduce_microstep: 39.49 | step_microstep: 19.58
[2025-04-26 05:25:09,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.20 | bwd: 5748.29 | bwd_inner: 5708.73 | bwd_allreduce: 39.51 | step: 19.58

  1%|▏         | 526/41250 [1:16:40<98:02:36,  8.67s/it]
                                                        
{'loss': 0.1137, 'grad_norm': 1.8153009414672852, 'learning_rate': 1.6995153473344103e-05, 'epoch': 0.13}

  1%|▏         | 526/41250 [1:16:40<98:02:36,  8.67s/it][2025-04-26 05:25:18,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:25:18,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.55 | bwd_microstep: 5698.55 | bwd_inner_microstep: 5685.64 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.24
[2025-04-26 05:25:18,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.55 | bwd: 5698.56 | bwd_inner: 5685.64 | bwd_allreduce: 12.88 | step: 18.25

  1%|▏         | 527/41250 [1:16:49<97:53:23,  8.65s/it]
                                                        
{'loss': 0.0864, 'grad_norm': 1.3550256490707397, 'learning_rate': 1.7027463651050084e-05, 'epoch': 0.13}

  1%|▏         | 527/41250 [1:16:49<97:53:23,  8.65s/it][2025-04-26 05:25:26,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 05:25:26,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.55 | bwd_microstep: 5705.57 | bwd_inner_microstep: 5692.99 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.83
[2025-04-26 05:25:26,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.55 | bwd: 5705.59 | bwd_inner: 5692.99 | bwd_allreduce: 12.56 | step: 18.83

  1%|▏         | 528/41250 [1:16:57<97:51:43,  8.65s/it]
                                                        
{'loss': 0.1032, 'grad_norm': 1.8313724994659424, 'learning_rate': 1.7059773828756058e-05, 'epoch': 0.13}

  1%|▏         | 528/41250 [1:16:57<97:51:43,  8.65s/it][2025-04-26 05:25:35,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-26 05:25:35,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5753.78 | bwd_inner_microstep: 5692.00 | bwd_allreduce_microstep: 61.73 | step_microstep: 19.10
[2025-04-26 05:25:35,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5753.79 | bwd_inner: 5692.00 | bwd_allreduce: 61.75 | step: 19.10

  1%|▏         | 529/41250 [1:17:06<97:57:01,  8.66s/it]
                                                        
{'loss': 0.1944, 'grad_norm': 2.283151388168335, 'learning_rate': 1.7092084006462036e-05, 'epoch': 0.13}

  1%|▏         | 529/41250 [1:17:06<97:57:01,  8.66s/it][2025-04-26 05:25:44,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 05:25:44,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.86 | bwd_microstep: 5729.99 | bwd_inner_microstep: 5703.92 | bwd_allreduce_microstep: 26.02 | step_microstep: 19.16
[2025-04-26 05:25:44,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.86 | bwd: 5730.00 | bwd_inner: 5703.92 | bwd_allreduce: 26.04 | step: 19.17

  1%|▏         | 530/41250 [1:17:15<97:57:48,  8.66s/it]
                                                        
{'loss': 0.4326, 'grad_norm': 3.9164748191833496, 'learning_rate': 1.7124394184168014e-05, 'epoch': 0.13}

  1%|▏         | 530/41250 [1:17:15<97:57:48,  8.66s/it][2025-04-26 05:25:52,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:25:52,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.05 | bwd_microstep: 5719.97 | bwd_inner_microstep: 5707.31 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.43
[2025-04-26 05:25:52,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.05 | bwd: 5719.98 | bwd_inner: 5707.31 | bwd_allreduce: 12.63 | step: 18.43

  1%|▏         | 531/41250 [1:17:23<97:58:30,  8.66s/it]
                                                        
{'loss': 0.0825, 'grad_norm': 1.5048354864120483, 'learning_rate': 1.7156704361873992e-05, 'epoch': 0.13}

  1%|▏         | 531/41250 [1:17:23<97:58:30,  8.66s/it][2025-04-26 05:26:01,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 05:26:01,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.54 | bwd_microstep: 5693.25 | bwd_inner_microstep: 5653.65 | bwd_allreduce_microstep: 39.56 | step_microstep: 18.58
[2025-04-26 05:26:01,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.54 | bwd: 5693.27 | bwd_inner: 5653.65 | bwd_allreduce: 39.58 | step: 18.58

  1%|▏         | 532/41250 [1:17:32<97:47:10,  8.65s/it]
                                                        
{'loss': 0.1634, 'grad_norm': 1.7133415937423706, 'learning_rate': 1.718901453957997e-05, 'epoch': 0.13}

  1%|▏         | 532/41250 [1:17:32<97:47:10,  8.65s/it][2025-04-26 05:26:09,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.14 | optimizer_step: 1.05
[2025-04-26 05:26:09,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.84 | bwd_microstep: 5714.54 | bwd_inner_microstep: 5650.82 | bwd_allreduce_microstep: 63.66 | step_microstep: 19.54
[2025-04-26 05:26:09,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.84 | bwd: 5714.55 | bwd_inner: 5650.82 | bwd_allreduce: 63.69 | step: 19.55

  1%|▏         | 533/41250 [1:17:41<97:40:10,  8.64s/it]
                                                        
{'loss': 0.372, 'grad_norm': 5.490990161895752, 'learning_rate': 1.7221324717285944e-05, 'epoch': 0.13}

  1%|▏         | 533/41250 [1:17:41<97:40:10,  8.64s/it][2025-04-26 05:26:18,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:26:18,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.46 | bwd_microstep: 5760.37 | bwd_inner_microstep: 5681.07 | bwd_allreduce_microstep: 79.25 | step_microstep: 18.50
[2025-04-26 05:26:18,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.46 | bwd: 5760.38 | bwd_inner: 5681.07 | bwd_allreduce: 79.27 | step: 18.50

  1%|▏         | 534/41250 [1:17:49<97:50:12,  8.65s/it]
                                                        
{'loss': 0.1748, 'grad_norm': 2.1054093837738037, 'learning_rate': 1.7253634894991926e-05, 'epoch': 0.13}

  1%|▏         | 534/41250 [1:17:49<97:50:12,  8.65s/it][2025-04-26 05:26:27,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:26:27,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.39 | bwd_microstep: 5758.19 | bwd_inner_microstep: 5639.63 | bwd_allreduce_microstep: 118.51 | step_microstep: 18.40
[2025-04-26 05:26:27,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.39 | bwd: 5758.20 | bwd_inner: 5639.63 | bwd_allreduce: 118.53 | step: 18.41

  1%|▏         | 535/41250 [1:17:58<97:52:44,  8.65s/it]
                                                        
{'loss': 0.3051, 'grad_norm': 3.8728485107421875, 'learning_rate': 1.72859450726979e-05, 'epoch': 0.13}

  1%|▏         | 535/41250 [1:17:58<97:52:44,  8.65s/it][2025-04-26 05:26:35,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 05:26:35,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.04 | bwd_microstep: 5792.65 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 137.03 | step_microstep: 19.09
[2025-04-26 05:26:35,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.04 | bwd: 5792.66 | bwd_inner: 5655.56 | bwd_allreduce: 137.06 | step: 19.09

  1%|▏         | 536/41250 [1:18:07<98:03:09,  8.67s/it]
                                                        
{'loss': 0.1975, 'grad_norm': 1.8233387470245361, 'learning_rate': 1.7318255250403878e-05, 'epoch': 0.13}

  1%|▏         | 536/41250 [1:18:07<98:03:09,  8.67s/it][2025-04-26 05:26:44,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 05:26:44,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.37 | bwd_microstep: 5704.70 | bwd_inner_microstep: 5691.91 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.09
[2025-04-26 05:26:44,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.37 | bwd: 5704.71 | bwd_inner: 5691.91 | bwd_allreduce: 12.76 | step: 19.09

  1%|▏         | 537/41250 [1:18:15<97:57:43,  8.66s/it]
                                                        
{'loss': 0.2818, 'grad_norm': 2.9286510944366455, 'learning_rate': 1.7350565428109856e-05, 'epoch': 0.13}

  1%|▏         | 537/41250 [1:18:15<97:57:43,  8.66s/it][2025-04-26 05:26:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:26:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5753.96 | bwd_inner_microstep: 5688.50 | bwd_allreduce_microstep: 65.40 | step_microstep: 18.92
[2025-04-26 05:26:53,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5753.97 | bwd_inner: 5688.50 | bwd_allreduce: 65.42 | step: 18.92

  1%|▏         | 538/41250 [1:18:24<98:00:19,  8.67s/it]
                                                        
{'loss': 0.184, 'grad_norm': 3.4028427600860596, 'learning_rate': 1.7382875605815834e-05, 'epoch': 0.13}

  1%|▏         | 538/41250 [1:18:24<98:00:19,  8.67s/it][2025-04-26 05:27:01,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.08
[2025-04-26 05:27:01,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.11 | bwd_microstep: 5744.57 | bwd_inner_microstep: 5705.34 | bwd_allreduce_microstep: 39.18 | step_microstep: 18.92
[2025-04-26 05:27:01,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.11 | bwd: 5744.58 | bwd_inner: 5705.34 | bwd_allreduce: 39.20 | step: 18.92

  1%|▏         | 539/41250 [1:18:33<98:02:52,  8.67s/it]
                                                        
{'loss': 0.1364, 'grad_norm': 1.5558871030807495, 'learning_rate': 1.741518578352181e-05, 'epoch': 0.13}

  1%|▏         | 539/41250 [1:18:33<98:02:52,  8.67s/it][2025-04-26 05:27:10,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 05:27:10,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.28 | bwd_microstep: 5764.99 | bwd_inner_microstep: 5701.09 | bwd_allreduce_microstep: 63.86 | step_microstep: 18.94
[2025-04-26 05:27:10,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.28 | bwd: 5765.01 | bwd_inner: 5701.09 | bwd_allreduce: 63.88 | step: 18.94

  1%|▏         | 540/41250 [1:18:41<98:08:49,  8.68s/it]
                                                        
{'loss': 0.1317, 'grad_norm': 2.4580070972442627, 'learning_rate': 1.7447495961227786e-05, 'epoch': 0.13}

  1%|▏         | 540/41250 [1:18:41<98:08:49,  8.68s/it][2025-04-26 05:27:19,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 05:27:19,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.53 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5651.02 | bwd_allreduce_microstep: 114.06 | step_microstep: 18.97
[2025-04-26 05:27:19,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.53 | bwd: 5765.14 | bwd_inner: 5651.02 | bwd_allreduce: 114.08 | step: 18.98

  1%|▏         | 541/41250 [1:18:50<98:07:42,  8.68s/it]
                                                        
{'loss': 0.2681, 'grad_norm': 2.5465941429138184, 'learning_rate': 1.7479806138933767e-05, 'epoch': 0.13}

  1%|▏         | 541/41250 [1:18:50<98:07:42,  8.68s/it][2025-04-26 05:27:27,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 05:27:27,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5708.04 | bwd_inner_microstep: 5648.92 | bwd_allreduce_microstep: 59.08 | step_microstep: 18.55
[2025-04-26 05:27:27,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5708.06 | bwd_inner: 5648.92 | bwd_allreduce: 59.10 | step: 18.55

  1%|▏         | 542/41250 [1:18:59<97:52:37,  8.66s/it]
                                                        
{'loss': 0.216, 'grad_norm': 1.984389305114746, 'learning_rate': 1.7512116316639742e-05, 'epoch': 0.13}

  1%|▏         | 542/41250 [1:18:59<97:52:37,  8.66s/it][2025-04-26 05:27:36,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:27:36,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.61 | bwd_microstep: 5767.63 | bwd_inner_microstep: 5660.77 | bwd_allreduce_microstep: 106.81 | step_microstep: 18.69
[2025-04-26 05:27:36,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.61 | bwd: 5767.64 | bwd_inner: 5660.77 | bwd_allreduce: 106.83 | step: 18.69

  1%|▏         | 543/41250 [1:19:07<97:58:31,  8.66s/it]
                                                        
{'loss': 0.5249, 'grad_norm': 5.757068634033203, 'learning_rate': 1.754442649434572e-05, 'epoch': 0.13}

  1%|▏         | 543/41250 [1:19:07<97:58:31,  8.66s/it][2025-04-26 05:27:45,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:27:45,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.23 | bwd_microstep: 5767.53 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 69.04 | step_microstep: 18.32
[2025-04-26 05:27:45,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.23 | bwd: 5767.55 | bwd_inner: 5698.45 | bwd_allreduce: 69.05 | step: 18.32

  1%|▏         | 544/41250 [1:19:16<98:06:33,  8.68s/it]
                                                        
{'loss': 0.2004, 'grad_norm': 2.7255191802978516, 'learning_rate': 1.7576736672051698e-05, 'epoch': 0.13}

  1%|▏         | 544/41250 [1:19:16<98:06:33,  8.68s/it][2025-04-26 05:27:54,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 1.16
[2025-04-26 05:27:54,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.21 | bwd_microstep: 5760.90 | bwd_inner_microstep: 5693.41 | bwd_allreduce_microstep: 67.43 | step_microstep: 20.03
[2025-04-26 05:27:54,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.21 | bwd: 5760.91 | bwd_inner: 5693.41 | bwd_allreduce: 67.46 | step: 20.03

  1%|▏         | 545/41250 [1:19:25<98:11:54,  8.68s/it]
                                                        
{'loss': 0.2735, 'grad_norm': 3.189650535583496, 'learning_rate': 1.7609046849757676e-05, 'epoch': 0.13}

  1%|▏         | 545/41250 [1:19:25<98:11:54,  8.68s/it][2025-04-26 05:28:02,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 05:28:02,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.62 | bwd_microstep: 5786.56 | bwd_inner_microstep: 5659.08 | bwd_allreduce_microstep: 127.43 | step_microstep: 19.33
[2025-04-26 05:28:02,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.62 | bwd: 5786.57 | bwd_inner: 5659.08 | bwd_allreduce: 127.45 | step: 19.33

  1%|▏         | 546/41250 [1:19:33<98:15:08,  8.69s/it]
                                                        
{'loss': 0.2255, 'grad_norm': 3.2492218017578125, 'learning_rate': 1.7641357027463653e-05, 'epoch': 0.13}

  1%|▏         | 546/41250 [1:19:33<98:15:08,  8.69s/it][2025-04-26 05:28:11,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 05:28:11,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.67 | bwd_microstep: 5783.31 | bwd_inner_microstep: 5656.45 | bwd_allreduce_microstep: 126.81 | step_microstep: 19.32
[2025-04-26 05:28:11,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.67 | bwd: 5783.32 | bwd_inner: 5656.45 | bwd_allreduce: 126.83 | step: 19.33

  1%|▏         | 547/41250 [1:19:42<98:15:28,  8.69s/it]
                                                        
{'loss': 0.3142, 'grad_norm': 2.587265968322754, 'learning_rate': 1.7673667205169628e-05, 'epoch': 0.13}

  1%|▏         | 547/41250 [1:19:42<98:15:28,  8.69s/it][2025-04-26 05:28:20,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 05:28:20,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.66 | bwd_microstep: 5807.32 | bwd_inner_microstep: 5650.50 | bwd_allreduce_microstep: 156.77 | step_microstep: 19.00
[2025-04-26 05:28:20,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.66 | bwd: 5807.33 | bwd_inner: 5650.50 | bwd_allreduce: 156.79 | step: 19.01

  1%|▏         | 548/41250 [1:19:51<98:20:47,  8.70s/it]
                                                        
{'loss': 0.3343, 'grad_norm': 3.41823410987854, 'learning_rate': 1.770597738287561e-05, 'epoch': 0.13}

  1%|▏         | 548/41250 [1:19:51<98:20:47,  8.70s/it][2025-04-26 05:28:28,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-26 05:28:28,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.92 | bwd_microstep: 5800.65 | bwd_inner_microstep: 5787.87 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.38
[2025-04-26 05:28:28,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.92 | bwd: 5800.66 | bwd_inner: 5787.86 | bwd_allreduce: 12.76 | step: 19.38

  1%|▏         | 549/41250 [1:20:00<98:37:26,  8.72s/it]
                                                        
{'loss': 0.1162, 'grad_norm': 1.1793696880340576, 'learning_rate': 1.7738287560581584e-05, 'epoch': 0.13}

  1%|▏         | 549/41250 [1:20:00<98:37:26,  8.72s/it][2025-04-26 05:28:37,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 05:28:37,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5729.72 | bwd_inner_microstep: 5657.03 | bwd_allreduce_microstep: 72.64 | step_microstep: 18.68
[2025-04-26 05:28:37,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5729.73 | bwd_inner: 5657.03 | bwd_allreduce: 72.66 | step: 18.68

  1%|▏         | 550/41250 [1:20:08<98:20:53,  8.70s/it]
                                                        
{'loss': 0.151, 'grad_norm': 2.027606248855591, 'learning_rate': 1.777059773828756e-05, 'epoch': 0.13}

  1%|▏         | 550/41250 [1:20:08<98:20:53,  8.70s/it][2025-04-26 05:28:46,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.12 | optimizer_step: 0.98
[2025-04-26 05:28:46,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.96 | bwd_microstep: 5738.72 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 85.84 | step_microstep: 19.08
[2025-04-26 05:28:46,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.96 | bwd: 5738.73 | bwd_inner: 5652.83 | bwd_allreduce: 85.86 | step: 19.08

  1%|▏         | 551/41250 [1:20:17<98:08:37,  8.68s/it]
                                                        
{'loss': 0.0889, 'grad_norm': 1.3572748899459839, 'learning_rate': 1.780290791599354e-05, 'epoch': 0.13}

  1%|▏         | 551/41250 [1:20:17<98:08:37,  8.68s/it][2025-04-26 05:28:54,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 05:28:54,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.63 | bwd_microstep: 5717.88 | bwd_inner_microstep: 5673.83 | bwd_allreduce_microstep: 44.00 | step_microstep: 18.71
[2025-04-26 05:28:54,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.63 | bwd: 5717.90 | bwd_inner: 5673.83 | bwd_allreduce: 44.02 | step: 18.72

  1%|▏         | 552/41250 [1:20:26<97:57:36,  8.67s/it]
                                                        
{'loss': 0.2056, 'grad_norm': 4.58895206451416, 'learning_rate': 1.7835218093699517e-05, 'epoch': 0.13}

  1%|▏         | 552/41250 [1:20:26<97:57:36,  8.67s/it][2025-04-26 05:29:03,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 05:29:03,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.52 | bwd_microstep: 5715.25 | bwd_inner_microstep: 5702.55 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.76
[2025-04-26 05:29:03,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.52 | bwd: 5715.26 | bwd_inner: 5702.55 | bwd_allreduce: 12.67 | step: 18.77

  1%|▏         | 553/41250 [1:20:34<97:52:51,  8.66s/it]
                                                        
{'loss': 0.3927, 'grad_norm': 3.010521411895752, 'learning_rate': 1.7867528271405495e-05, 'epoch': 0.13}

  1%|▏         | 553/41250 [1:20:34<97:52:51,  8.66s/it][2025-04-26 05:29:12,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-26 05:29:12,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5763.23 | bwd_inner_microstep: 5710.73 | bwd_allreduce_microstep: 52.45 | step_microstep: 18.45
[2025-04-26 05:29:12,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5763.24 | bwd_inner: 5710.73 | bwd_allreduce: 52.47 | step: 18.45

  1%|▏         | 554/41250 [1:20:43<98:01:02,  8.67s/it]
                                                        
{'loss': 0.1354, 'grad_norm': 1.8977832794189453, 'learning_rate': 1.789983844911147e-05, 'epoch': 0.13}

  1%|▏         | 554/41250 [1:20:43<98:01:02,  8.67s/it][2025-04-26 05:29:20,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:29:20,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.53 | bwd_microstep: 5771.32 | bwd_inner_microstep: 5678.85 | bwd_allreduce_microstep: 92.44 | step_microstep: 18.40
[2025-04-26 05:29:20,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.53 | bwd: 5771.34 | bwd_inner: 5678.84 | bwd_allreduce: 92.45 | step: 18.41

  1%|▏         | 555/41250 [1:20:52<98:07:32,  8.68s/it]
                                                        
{'loss': 0.1028, 'grad_norm': 1.9520384073257446, 'learning_rate': 1.793214862681745e-05, 'epoch': 0.13}

  1%|▏         | 555/41250 [1:20:52<98:07:32,  8.68s/it][2025-04-26 05:29:29,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:29:29,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.39 | bwd_microstep: 5782.46 | bwd_inner_microstep: 5658.55 | bwd_allreduce_microstep: 123.87 | step_microstep: 18.38
[2025-04-26 05:29:29,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.39 | bwd: 5782.47 | bwd_inner: 5658.55 | bwd_allreduce: 123.89 | step: 18.38

  1%|▏         | 556/41250 [1:21:00<98:08:52,  8.68s/it]
                                                        
{'loss': 0.1977, 'grad_norm': 1.8537025451660156, 'learning_rate': 1.7964458804523425e-05, 'epoch': 0.13}

  1%|▏         | 556/41250 [1:21:00<98:08:52,  8.68s/it][2025-04-26 05:29:38,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 05:29:38,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5761.13 | bwd_inner_microstep: 5660.15 | bwd_allreduce_microstep: 100.94 | step_microstep: 19.23
[2025-04-26 05:29:38,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.93 | bwd: 5761.15 | bwd_inner: 5660.15 | bwd_allreduce: 100.96 | step: 19.24

  1%|▏         | 557/41250 [1:21:09<98:06:17,  8.68s/it]
                                                        
{'loss': 0.2537, 'grad_norm': 2.345719337463379, 'learning_rate': 1.7996768982229403e-05, 'epoch': 0.14}

  1%|▏         | 557/41250 [1:21:09<98:06:17,  8.68s/it][2025-04-26 05:29:46,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:29:46,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.28 | bwd_microstep: 5759.94 | bwd_inner_microstep: 5684.46 | bwd_allreduce_microstep: 75.43 | step_microstep: 18.47
[2025-04-26 05:29:46,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.28 | bwd: 5759.95 | bwd_inner: 5684.46 | bwd_allreduce: 75.45 | step: 18.48

  1%|▏         | 558/41250 [1:21:18<98:09:14,  8.68s/it]
                                                        
{'loss': 0.1935, 'grad_norm': 1.87367844581604, 'learning_rate': 1.802907915993538e-05, 'epoch': 0.14}

  1%|▏         | 558/41250 [1:21:18<98:09:14,  8.68s/it][2025-04-26 05:29:55,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:29:55,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.46 | bwd_microstep: 5712.03 | bwd_inner_microstep: 5663.40 | bwd_allreduce_microstep: 48.58 | step_microstep: 17.89
[2025-04-26 05:29:55,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.46 | bwd: 5712.04 | bwd_inner: 5663.40 | bwd_allreduce: 48.60 | step: 17.90

  1%|▏         | 559/41250 [1:21:26<97:56:38,  8.67s/it]
                                                        
{'loss': 0.2301, 'grad_norm': 4.064697265625, 'learning_rate': 1.806138933764136e-05, 'epoch': 0.14}

  1%|▏         | 559/41250 [1:21:26<97:56:38,  8.67s/it][2025-04-26 05:30:04,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:30:04,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.20 | bwd_microstep: 5733.89 | bwd_inner_microstep: 5700.23 | bwd_allreduce_microstep: 33.62 | step_microstep: 17.83
[2025-04-26 05:30:04,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.20 | bwd: 5733.90 | bwd_inner: 5700.23 | bwd_allreduce: 33.64 | step: 17.83

  1%|▏         | 560/41250 [1:21:35<97:55:03,  8.66s/it]
                                                        
{'loss': 0.3676, 'grad_norm': 3.230354070663452, 'learning_rate': 1.8093699515347337e-05, 'epoch': 0.14}

  1%|▏         | 560/41250 [1:21:35<97:55:03,  8.66s/it][2025-04-26 05:30:12,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:30:12,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.80 | bwd_microstep: 5772.79 | bwd_inner_microstep: 5649.52 | bwd_allreduce_microstep: 123.23 | step_microstep: 18.37
[2025-04-26 05:30:12,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.80 | bwd: 5772.80 | bwd_inner: 5649.52 | bwd_allreduce: 123.24 | step: 18.37

  1%|▏         | 561/41250 [1:21:44<97:59:43,  8.67s/it]
                                                        
{'loss': 0.1342, 'grad_norm': 1.4522126913070679, 'learning_rate': 1.812600969305331e-05, 'epoch': 0.14}

  1%|▏         | 561/41250 [1:21:44<97:59:43,  8.67s/it][2025-04-26 05:30:21,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 05:30:21,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.57 | bwd_microstep: 5705.92 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.91
[2025-04-26 05:30:21,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.57 | bwd: 5705.93 | bwd_inner: 5693.20 | bwd_allreduce: 12.69 | step: 18.91

  1%|▏         | 562/41250 [1:21:52<97:55:38,  8.66s/it]
                                                        
{'loss': 0.2507, 'grad_norm': 2.516587257385254, 'learning_rate': 1.8158319870759293e-05, 'epoch': 0.14}

  1%|▏         | 562/41250 [1:21:52<97:55:38,  8.66s/it][2025-04-26 05:30:30,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.25
[2025-04-26 05:30:30,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.99 | bwd_microstep: 5854.94 | bwd_inner_microstep: 5700.43 | bwd_allreduce_microstep: 154.46 | step_microstep: 19.52
[2025-04-26 05:30:30,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.99 | bwd: 5854.95 | bwd_inner: 5700.43 | bwd_allreduce: 154.48 | step: 19.52

  1%|▏         | 563/41250 [1:22:01<98:22:43,  8.70s/it]
                                                        
{'loss': 0.1225, 'grad_norm': 1.5950648784637451, 'learning_rate': 1.8190630048465267e-05, 'epoch': 0.14}

  1%|▏         | 563/41250 [1:22:01<98:22:43,  8.70s/it][2025-04-26 05:30:39,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.97
[2025-04-26 05:30:39,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.49 | bwd_microstep: 5735.66 | bwd_inner_microstep: 5693.63 | bwd_allreduce_microstep: 41.98 | step_microstep: 19.10
[2025-04-26 05:30:39,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.49 | bwd: 5735.67 | bwd_inner: 5693.63 | bwd_allreduce: 42.00 | step: 19.10

  1%|▏         | 564/41250 [1:22:10<98:18:33,  8.70s/it]
                                                        
{'loss': 0.2878, 'grad_norm': 3.232527732849121, 'learning_rate': 1.8222940226171245e-05, 'epoch': 0.14}

  1%|▏         | 564/41250 [1:22:10<98:18:33,  8.70s/it][2025-04-26 05:30:47,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 05:30:47,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.54 | bwd_microstep: 5764.38 | bwd_inner_microstep: 5684.83 | bwd_allreduce_microstep: 79.50 | step_microstep: 18.59
[2025-04-26 05:30:47,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.54 | bwd: 5764.40 | bwd_inner: 5684.83 | bwd_allreduce: 79.52 | step: 18.59

  1%|▏         | 565/41250 [1:22:18<98:15:45,  8.69s/it]
                                                        
{'loss': 0.2011, 'grad_norm': 2.30855131149292, 'learning_rate': 1.8255250403877223e-05, 'epoch': 0.14}

  1%|▏         | 565/41250 [1:22:18<98:15:45,  8.69s/it][2025-04-26 05:30:56,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 05:30:56,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.15 | bwd_microstep: 5749.54 | bwd_inner_microstep: 5681.88 | bwd_allreduce_microstep: 67.61 | step_microstep: 18.57
[2025-04-26 05:30:56,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.15 | bwd: 5749.55 | bwd_inner: 5681.88 | bwd_allreduce: 67.63 | step: 18.58

  1%|▏         | 566/41250 [1:22:27<98:12:40,  8.69s/it]
                                                        
{'loss': 0.0707, 'grad_norm': 1.2889379262924194, 'learning_rate': 1.8287560581583198e-05, 'epoch': 0.14}

  1%|▏         | 566/41250 [1:22:27<98:12:40,  8.69s/it][2025-04-26 05:31:05,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 05:31:05,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.83 | bwd_microstep: 5736.31 | bwd_inner_microstep: 5684.72 | bwd_allreduce_microstep: 51.54 | step_microstep: 19.01
[2025-04-26 05:31:05,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.83 | bwd: 5736.32 | bwd_inner: 5684.72 | bwd_allreduce: 51.56 | step: 19.02

  1%|▏         | 567/41250 [1:22:36<98:08:01,  8.68s/it]
                                                        
{'loss': 0.3113, 'grad_norm': 2.507721424102783, 'learning_rate': 1.831987075928918e-05, 'epoch': 0.14}

  1%|▏         | 567/41250 [1:22:36<98:08:01,  8.68s/it][2025-04-26 05:31:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 05:31:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5705.86 | bwd_inner_microstep: 5642.88 | bwd_allreduce_microstep: 62.93 | step_microstep: 18.58
[2025-04-26 05:31:13,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5705.87 | bwd_inner: 5642.88 | bwd_allreduce: 62.95 | step: 18.59

  1%|▏         | 568/41250 [1:22:44<97:54:06,  8.66s/it]
                                                        
{'loss': 0.216, 'grad_norm': 2.282418966293335, 'learning_rate': 1.8352180936995153e-05, 'epoch': 0.14}

  1%|▏         | 568/41250 [1:22:44<97:54:06,  8.66s/it][2025-04-26 05:31:22,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:31:22,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.18 | bwd_microstep: 5697.06 | bwd_inner_microstep: 5646.90 | bwd_allreduce_microstep: 50.12 | step_microstep: 18.59
[2025-04-26 05:31:22,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.18 | bwd: 5697.08 | bwd_inner: 5646.90 | bwd_allreduce: 50.14 | step: 18.59

  1%|▏         | 569/41250 [1:22:53<97:39:07,  8.64s/it]
                                                        
{'loss': 0.2282, 'grad_norm': 2.7032039165496826, 'learning_rate': 1.8384491114701135e-05, 'epoch': 0.14}

  1%|▏         | 569/41250 [1:22:53<97:39:07,  8.64s/it][2025-04-26 05:31:31,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-26 05:31:31,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.20 | bwd_microstep: 5801.09 | bwd_inner_microstep: 5774.88 | bwd_allreduce_microstep: 26.17 | step_microstep: 19.37
[2025-04-26 05:31:31,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.20 | bwd: 5801.11 | bwd_inner: 5774.88 | bwd_allreduce: 26.18 | step: 19.37

  1%|▏         | 570/41250 [1:23:02<98:06:46,  8.68s/it]
                                                        
{'loss': 0.2576, 'grad_norm': 2.2819488048553467, 'learning_rate': 1.841680129240711e-05, 'epoch': 0.14}

  1%|▏         | 570/41250 [1:23:02<98:06:46,  8.68s/it][2025-04-26 05:31:39,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 05:31:39,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.50 | bwd_microstep: 5784.44 | bwd_inner_microstep: 5644.74 | bwd_allreduce_microstep: 139.64 | step_microstep: 18.90
[2025-04-26 05:31:39,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.50 | bwd: 5784.45 | bwd_inner: 5644.74 | bwd_allreduce: 139.67 | step: 18.90

  1%|▏         | 571/41250 [1:23:10<98:09:35,  8.69s/it]
                                                        
{'loss': 0.2912, 'grad_norm': 3.0165722370147705, 'learning_rate': 1.8449111470113087e-05, 'epoch': 0.14}

  1%|▏         | 571/41250 [1:23:10<98:09:35,  8.69s/it][2025-04-26 05:31:48,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:31:48,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5735.07 | bwd_inner_microstep: 5672.08 | bwd_allreduce_microstep: 62.95 | step_microstep: 18.17
[2025-04-26 05:31:48,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.76 | bwd: 5735.08 | bwd_inner: 5672.08 | bwd_allreduce: 62.96 | step: 18.17

  1%|▏         | 572/41250 [1:23:19<98:04:40,  8.68s/it]
                                                        
{'loss': 0.2752, 'grad_norm': 3.003877639770508, 'learning_rate': 1.8481421647819065e-05, 'epoch': 0.14}

  1%|▏         | 572/41250 [1:23:19<98:04:40,  8.68s/it][2025-04-26 05:31:57,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:31:57,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.71 | bwd_microstep: 5876.47 | bwd_inner_microstep: 5636.29 | bwd_allreduce_microstep: 240.13 | step_microstep: 18.27
[2025-04-26 05:31:57,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.71 | bwd: 5876.48 | bwd_inner: 5636.29 | bwd_allreduce: 240.15 | step: 18.27

  1%|▏         | 573/41250 [1:23:28<98:24:17,  8.71s/it]
                                                        
{'loss': 0.4608, 'grad_norm': 3.84578275680542, 'learning_rate': 1.851373182552504e-05, 'epoch': 0.14}

  1%|▏         | 573/41250 [1:23:28<98:24:17,  8.71s/it][2025-04-26 05:32:05,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 05:32:05,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5728.15 | bwd_inner_microstep: 5685.99 | bwd_allreduce_microstep: 42.12 | step_microstep: 18.49
[2025-04-26 05:32:05,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5728.17 | bwd_inner: 5685.99 | bwd_allreduce: 42.14 | step: 18.49

  1%|▏         | 574/41250 [1:23:37<98:12:54,  8.69s/it]
                                                        
{'loss': 0.2887, 'grad_norm': 2.87066388130188, 'learning_rate': 1.854604200323102e-05, 'epoch': 0.14}

  1%|▏         | 574/41250 [1:23:37<98:12:54,  8.69s/it][2025-04-26 05:32:14,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 05:32:14,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.74 | bwd_microstep: 5714.95 | bwd_inner_microstep: 5702.50 | bwd_allreduce_microstep: 12.41 | step_microstep: 18.73
[2025-04-26 05:32:14,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.74 | bwd: 5714.97 | bwd_inner: 5702.50 | bwd_allreduce: 12.43 | step: 18.73

  1%|▏         | 575/41250 [1:23:45<98:04:29,  8.68s/it]
                                                        
{'loss': 0.1198, 'grad_norm': 1.9438928365707397, 'learning_rate': 1.8578352180936995e-05, 'epoch': 0.14}

  1%|▏         | 575/41250 [1:23:45<98:04:29,  8.68s/it][2025-04-26 05:32:23,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-26 05:32:23,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.63 | bwd_microstep: 5748.17 | bwd_inner_microstep: 5659.25 | bwd_allreduce_microstep: 88.87 | step_microstep: 18.96
[2025-04-26 05:32:23,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.63 | bwd: 5748.18 | bwd_inner: 5659.25 | bwd_allreduce: 88.88 | step: 18.97

  1%|▏         | 576/41250 [1:23:54<98:01:22,  8.68s/it]
                                                        
{'loss': 0.2181, 'grad_norm': 2.439406156539917, 'learning_rate': 1.8610662358642973e-05, 'epoch': 0.14}

  1%|▏         | 576/41250 [1:23:54<98:01:22,  8.68s/it][2025-04-26 05:32:31,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 05:32:31,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.65 | bwd_microstep: 5750.28 | bwd_inner_microstep: 5715.52 | bwd_allreduce_microstep: 34.72 | step_microstep: 18.64
[2025-04-26 05:32:31,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.65 | bwd: 5750.30 | bwd_inner: 5715.52 | bwd_allreduce: 34.74 | step: 18.64

  1%|▏         | 577/41250 [1:24:03<98:02:04,  8.68s/it]
                                                        
{'loss': 0.2574, 'grad_norm': 2.6375856399536133, 'learning_rate': 1.864297253634895e-05, 'epoch': 0.14}

  1%|▏         | 577/41250 [1:24:03<98:02:04,  8.68s/it][2025-04-26 05:32:40,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 05:32:40,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.38 | bwd_microstep: 5698.29 | bwd_inner_microstep: 5685.26 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.75
[2025-04-26 05:32:40,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.38 | bwd: 5698.31 | bwd_inner: 5685.27 | bwd_allreduce: 13.00 | step: 18.75

  1%|▏         | 578/41250 [1:24:11<97:51:32,  8.66s/it]
                                                        
{'loss': 0.4183, 'grad_norm': 3.058556318283081, 'learning_rate': 1.867528271405493e-05, 'epoch': 0.14}

  1%|▏         | 578/41250 [1:24:11<97:51:32,  8.66s/it][2025-04-26 05:32:49,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 05:32:49,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.77 | bwd_microstep: 5781.77 | bwd_inner_microstep: 5646.93 | bwd_allreduce_microstep: 134.78 | step_microstep: 19.68
[2025-04-26 05:32:49,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.77 | bwd: 5781.79 | bwd_inner: 5646.93 | bwd_allreduce: 134.80 | step: 19.68

  1%|▏         | 579/41250 [1:24:20<97:58:25,  8.67s/it]
                                                        
{'loss': 0.175, 'grad_norm': 1.8066166639328003, 'learning_rate': 1.8707592891760907e-05, 'epoch': 0.14}

  1%|▏         | 579/41250 [1:24:20<97:58:25,  8.67s/it][2025-04-26 05:32:57,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:32:57,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.19 | bwd_microstep: 5764.11 | bwd_inner_microstep: 5699.67 | bwd_allreduce_microstep: 64.39 | step_microstep: 18.79
[2025-04-26 05:32:57,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.19 | bwd: 5764.12 | bwd_inner: 5699.67 | bwd_allreduce: 64.41 | step: 18.79

  1%|▏         | 580/41250 [1:24:29<98:03:54,  8.68s/it]
                                                        
{'loss': 0.1396, 'grad_norm': 1.1953028440475464, 'learning_rate': 1.873990306946688e-05, 'epoch': 0.14}

  1%|▏         | 580/41250 [1:24:29<98:03:54,  8.68s/it][2025-04-26 05:33:06,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 05:33:06,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.30 | bwd_microstep: 5762.32 | bwd_inner_microstep: 5685.73 | bwd_allreduce_microstep: 76.54 | step_microstep: 18.56
[2025-04-26 05:33:06,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.30 | bwd: 5762.33 | bwd_inner: 5685.73 | bwd_allreduce: 76.56 | step: 18.56

  1%|▏         | 581/41250 [1:24:37<98:06:41,  8.68s/it]
                                                        
{'loss': 0.3713, 'grad_norm': 2.5744636058807373, 'learning_rate': 1.8772213247172862e-05, 'epoch': 0.14}

  1%|▏         | 581/41250 [1:24:37<98:06:41,  8.68s/it][2025-04-26 05:33:15,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 05:33:15,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.28 | bwd_microstep: 5771.08 | bwd_inner_microstep: 5652.49 | bwd_allreduce_microstep: 118.54 | step_microstep: 19.42
[2025-04-26 05:33:15,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.28 | bwd: 5771.09 | bwd_inner: 5652.49 | bwd_allreduce: 118.56 | step: 19.42

  1%|▏         | 582/41250 [1:24:46<98:05:18,  8.68s/it]
                                                        
{'loss': 0.1594, 'grad_norm': 1.5052738189697266, 'learning_rate': 1.8804523424878837e-05, 'epoch': 0.14}

  1%|▏         | 582/41250 [1:24:46<98:05:18,  8.68s/it][2025-04-26 05:33:23,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:33:23,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.35 | bwd_microstep: 5715.77 | bwd_inner_microstep: 5703.03 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.74
[2025-04-26 05:33:23,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.35 | bwd: 5715.78 | bwd_inner: 5703.03 | bwd_allreduce: 12.70 | step: 18.74

  1%|▏         | 583/41250 [1:24:55<97:55:49,  8.67s/it]
                                                        
{'loss': 0.1085, 'grad_norm': 1.437985897064209, 'learning_rate': 1.8836833602584815e-05, 'epoch': 0.14}

  1%|▏         | 583/41250 [1:24:55<97:55:49,  8.67s/it][2025-04-26 05:33:32,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:33:32,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2943.75 | bwd_microstep: 5883.10 | bwd_inner_microstep: 5870.29 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.33
[2025-04-26 05:33:32,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2943.75 | bwd: 5883.11 | bwd_inner: 5870.29 | bwd_allreduce: 12.78 | step: 18.33

  1%|▏         | 584/41250 [1:25:03<98:44:57,  8.74s/it]
                                                        
{'loss': 0.1499, 'grad_norm': 2.043414831161499, 'learning_rate': 1.8869143780290793e-05, 'epoch': 0.14}

  1%|▏         | 584/41250 [1:25:03<98:44:57,  8.74s/it][2025-04-26 05:33:41,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:33:41,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.73 | bwd_microstep: 5774.98 | bwd_inner_microstep: 5651.78 | bwd_allreduce_microstep: 123.15 | step_microstep: 18.83
[2025-04-26 05:33:41,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.73 | bwd: 5774.99 | bwd_inner: 5651.78 | bwd_allreduce: 123.17 | step: 18.83

  1%|▏         | 585/41250 [1:25:12<98:33:10,  8.72s/it]
                                                        
{'loss': 0.1239, 'grad_norm': 1.5361359119415283, 'learning_rate': 1.890145395799677e-05, 'epoch': 0.14}

  1%|▏         | 585/41250 [1:25:12<98:33:10,  8.72s/it][2025-04-26 05:33:50,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.36 | optimizer_step: 1.09
[2025-04-26 05:33:50,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.65 | bwd_microstep: 5751.22 | bwd_inner_microstep: 5704.21 | bwd_allreduce_microstep: 46.95 | step_microstep: 20.45
[2025-04-26 05:33:50,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.65 | bwd: 5751.24 | bwd_inner: 5704.21 | bwd_allreduce: 46.98 | step: 20.45

  1%|▏         | 586/41250 [1:25:21<98:25:31,  8.71s/it]
                                                        
{'loss': 0.4044, 'grad_norm': 3.1944615840911865, 'learning_rate': 1.893376413570275e-05, 'epoch': 0.14}

  1%|▏         | 586/41250 [1:25:21<98:25:31,  8.71s/it][2025-04-26 05:33:58,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 05:33:58,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.17 | bwd_microstep: 5788.74 | bwd_inner_microstep: 5656.76 | bwd_allreduce_microstep: 131.94 | step_microstep: 18.83
[2025-04-26 05:33:58,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.17 | bwd: 5788.75 | bwd_inner: 5656.76 | bwd_allreduce: 131.95 | step: 18.83

  1%|▏         | 587/41250 [1:25:30<98:21:05,  8.71s/it]
                                                        
{'loss': 0.2388, 'grad_norm': 2.1900205612182617, 'learning_rate': 1.8966074313408723e-05, 'epoch': 0.14}

  1%|▏         | 587/41250 [1:25:30<98:21:05,  8.71s/it][2025-04-26 05:34:07,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:34:07,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.29 | bwd_microstep: 5721.93 | bwd_inner_microstep: 5660.45 | bwd_allreduce_microstep: 61.43 | step_microstep: 18.49
[2025-04-26 05:34:07,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.29 | bwd: 5721.94 | bwd_inner: 5660.45 | bwd_allreduce: 61.45 | step: 18.49

  1%|▏         | 588/41250 [1:25:38<98:07:13,  8.69s/it]
                                                        
{'loss': 0.3265, 'grad_norm': 2.450702667236328, 'learning_rate': 1.8998384491114704e-05, 'epoch': 0.14}

  1%|▏         | 588/41250 [1:25:38<98:07:13,  8.69s/it][2025-04-26 05:34:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:34:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.38 | bwd_microstep: 5706.07 | bwd_inner_microstep: 5692.86 | bwd_allreduce_microstep: 13.17 | step_microstep: 18.65
[2025-04-26 05:34:16,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.38 | bwd: 5706.09 | bwd_inner: 5692.86 | bwd_allreduce: 13.18 | step: 18.66

  1%|▏         | 589/41250 [1:25:47<97:56:12,  8.67s/it]
                                                        
{'loss': 0.283, 'grad_norm': 2.6550302505493164, 'learning_rate': 1.903069466882068e-05, 'epoch': 0.14}

  1%|▏         | 589/41250 [1:25:47<97:56:12,  8.67s/it][2025-04-26 05:34:24,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:34:24,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.48 | bwd_microstep: 5727.54 | bwd_inner_microstep: 5714.81 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.87
[2025-04-26 05:34:24,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.48 | bwd: 5727.55 | bwd_inner: 5714.81 | bwd_allreduce: 12.70 | step: 18.87

  1%|▏         | 590/41250 [1:25:55<97:55:11,  8.67s/it]
                                                        
{'loss': 0.2436, 'grad_norm': 2.4576172828674316, 'learning_rate': 1.9063004846526657e-05, 'epoch': 0.14}

  1%|▏         | 590/41250 [1:25:55<97:55:11,  8.67s/it][2025-04-26 05:34:33,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 05:34:33,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.95 | bwd_microstep: 5785.40 | bwd_inner_microstep: 5645.49 | bwd_allreduce_microstep: 139.86 | step_microstep: 19.39
[2025-04-26 05:34:33,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.95 | bwd: 5785.42 | bwd_inner: 5645.49 | bwd_allreduce: 139.89 | step: 19.40

  1%|▏         | 591/41250 [1:26:04<98:00:23,  8.68s/it]
                                                        
{'loss': 0.2325, 'grad_norm': 1.6893880367279053, 'learning_rate': 1.9095315024232634e-05, 'epoch': 0.14}

  1%|▏         | 591/41250 [1:26:04<98:00:23,  8.68s/it][2025-04-26 05:34:42,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 05:34:42,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5711.46 | bwd_inner_microstep: 5654.79 | bwd_allreduce_microstep: 56.62 | step_microstep: 18.92
[2025-04-26 05:34:42,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5711.47 | bwd_inner: 5654.79 | bwd_allreduce: 56.64 | step: 18.92

  1%|▏         | 592/41250 [1:26:13<97:48:38,  8.66s/it]
                                                        
{'loss': 0.2178, 'grad_norm': 1.6407346725463867, 'learning_rate': 1.9127625201938612e-05, 'epoch': 0.14}

  1%|▏         | 592/41250 [1:26:13<97:48:38,  8.66s/it][2025-04-26 05:34:50,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 05:34:50,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.74 | bwd_microstep: 5779.05 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 91.46 | step_microstep: 18.56
[2025-04-26 05:34:50,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.74 | bwd: 5779.06 | bwd_inner: 5687.54 | bwd_allreduce: 91.48 | step: 18.56

  1%|▏         | 593/41250 [1:26:22<97:58:56,  8.68s/it]
                                                        
{'loss': 0.2607, 'grad_norm': 1.37513267993927, 'learning_rate': 1.915993537964459e-05, 'epoch': 0.14}

  1%|▏         | 593/41250 [1:26:22<97:58:56,  8.68s/it][2025-04-26 05:34:59,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:34:59,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5751.30 | bwd_inner_microstep: 5707.96 | bwd_allreduce_microstep: 43.29 | step_microstep: 18.72
[2025-04-26 05:34:59,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5751.31 | bwd_inner: 5707.96 | bwd_allreduce: 43.31 | step: 18.72

  1%|▏         | 594/41250 [1:26:30<98:00:02,  8.68s/it]
                                                        
{'loss': 0.2962, 'grad_norm': 2.0458662509918213, 'learning_rate': 1.9192245557350565e-05, 'epoch': 0.14}

  1%|▏         | 594/41250 [1:26:30<98:00:02,  8.68s/it][2025-04-26 05:35:08,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:35:08,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.41 | bwd_microstep: 5770.00 | bwd_inner_microstep: 5702.45 | bwd_allreduce_microstep: 67.51 | step_microstep: 18.74
[2025-04-26 05:35:08,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.41 | bwd: 5770.02 | bwd_inner: 5702.44 | bwd_allreduce: 67.53 | step: 18.74

  1%|▏         | 595/41250 [1:26:39<98:04:01,  8.68s/it]
                                                        
{'loss': 0.1991, 'grad_norm': 1.5197874307632446, 'learning_rate': 1.9224555735056546e-05, 'epoch': 0.14}

  1%|▏         | 595/41250 [1:26:39<98:04:01,  8.68s/it][2025-04-26 05:35:16,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-26 05:35:16,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.44 | bwd_microstep: 5719.78 | bwd_inner_microstep: 5656.90 | bwd_allreduce_microstep: 62.83 | step_microstep: 19.56
[2025-04-26 05:35:16,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.44 | bwd: 5719.79 | bwd_inner: 5656.90 | bwd_allreduce: 62.85 | step: 19.56

  1%|▏         | 596/41250 [1:26:48<97:52:09,  8.67s/it]
                                                        
{'loss': 0.2472, 'grad_norm': 1.4937831163406372, 'learning_rate': 1.925686591276252e-05, 'epoch': 0.14}

  1%|▏         | 596/41250 [1:26:48<97:52:09,  8.67s/it][2025-04-26 05:35:25,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 05:35:25,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.99 | bwd_microstep: 5713.33 | bwd_inner_microstep: 5650.71 | bwd_allreduce_microstep: 62.57 | step_microstep: 18.39
[2025-04-26 05:35:25,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.99 | bwd: 5713.34 | bwd_inner: 5650.71 | bwd_allreduce: 62.59 | step: 18.40

  1%|▏         | 597/41250 [1:26:56<97:45:19,  8.66s/it]
                                                        
{'loss': 0.1607, 'grad_norm': 1.4409146308898926, 'learning_rate': 1.92891760904685e-05, 'epoch': 0.14}

  1%|▏         | 597/41250 [1:26:56<97:45:19,  8.66s/it][2025-04-26 05:35:34,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 05:35:34,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.95 | bwd_microstep: 5712.78 | bwd_inner_microstep: 5662.71 | bwd_allreduce_microstep: 50.03 | step_microstep: 18.56
[2025-04-26 05:35:34,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.95 | bwd: 5712.80 | bwd_inner: 5662.71 | bwd_allreduce: 50.05 | step: 18.56

  1%|▏         | 598/41250 [1:27:05<97:40:41,  8.65s/it]
                                                        
{'loss': 0.2266, 'grad_norm': 1.9264477491378784, 'learning_rate': 1.9321486268174476e-05, 'epoch': 0.14}

  1%|▏         | 598/41250 [1:27:05<97:40:41,  8.65s/it][2025-04-26 05:35:42,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 05:35:42,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.61 | bwd_microstep: 5716.58 | bwd_inner_microstep: 5703.76 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.24
[2025-04-26 05:35:42,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.61 | bwd: 5716.59 | bwd_inner: 5703.76 | bwd_allreduce: 12.79 | step: 19.24

  1%|▏         | 599/41250 [1:27:13<97:44:33,  8.66s/it]
                                                        
{'loss': 0.1688, 'grad_norm': 1.7132024765014648, 'learning_rate': 1.9353796445880454e-05, 'epoch': 0.15}

  1%|▏         | 599/41250 [1:27:13<97:44:33,  8.66s/it][2025-04-26 05:35:51,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:35:51,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.69 | bwd_microstep: 5768.10 | bwd_inner_microstep: 5645.67 | bwd_allreduce_microstep: 122.37 | step_microstep: 18.64
[2025-04-26 05:35:51,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.69 | bwd: 5768.11 | bwd_inner: 5645.67 | bwd_allreduce: 122.40 | step: 18.64

  1%|▏         | 600/41250 [1:27:22<97:51:04,  8.67s/it]
                                                        
{'loss': 0.0823, 'grad_norm': 1.0309412479400635, 'learning_rate': 1.9386106623586432e-05, 'epoch': 0.15}

  1%|▏         | 600/41250 [1:27:22<97:51:04,  8.67s/it][2025-04-26 05:36:00,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.08
[2025-04-26 05:36:00,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.68 | bwd_microstep: 5717.68 | bwd_inner_microstep: 5704.69 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.69
[2025-04-26 05:36:00,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.68 | bwd: 5717.69 | bwd_inner: 5704.69 | bwd_allreduce: 12.96 | step: 19.70

  1%|▏         | 601/41250 [1:27:31<97:49:03,  8.66s/it]
                                                        
{'loss': 0.283, 'grad_norm': 2.125734806060791, 'learning_rate': 1.9418416801292407e-05, 'epoch': 0.15}

  1%|▏         | 601/41250 [1:27:31<97:49:03,  8.66s/it][2025-04-26 05:36:08,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 05:36:08,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.13 | bwd_microstep: 5718.81 | bwd_inner_microstep: 5705.94 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.39
[2025-04-26 05:36:08,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.13 | bwd: 5718.83 | bwd_inner: 5705.94 | bwd_allreduce: 12.85 | step: 19.39

  1%|▏         | 602/41250 [1:27:39<97:53:10,  8.67s/it]
                                                        
{'loss': 0.2589, 'grad_norm': 1.9110050201416016, 'learning_rate': 1.9450726978998388e-05, 'epoch': 0.15}

  1%|▏         | 602/41250 [1:27:39<97:53:10,  8.67s/it][2025-04-26 05:36:17,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:36:17,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.01 | bwd_microstep: 5845.30 | bwd_inner_microstep: 5683.76 | bwd_allreduce_microstep: 161.49 | step_microstep: 18.71
[2025-04-26 05:36:17,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.01 | bwd: 5845.31 | bwd_inner: 5683.76 | bwd_allreduce: 161.51 | step: 18.71

  1%|▏         | 603/41250 [1:27:48<98:18:13,  8.71s/it]
                                                        
{'loss': 0.1649, 'grad_norm': 1.2074155807495117, 'learning_rate': 1.9483037156704362e-05, 'epoch': 0.15}

  1%|▏         | 603/41250 [1:27:48<98:18:13,  8.71s/it][2025-04-26 05:36:26,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 05:36:26,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.11 | bwd_microstep: 5723.42 | bwd_inner_microstep: 5701.62 | bwd_allreduce_microstep: 21.75 | step_microstep: 19.09
[2025-04-26 05:36:26,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.11 | bwd: 5723.44 | bwd_inner: 5701.62 | bwd_allreduce: 21.77 | step: 19.09

  1%|▏         | 604/41250 [1:27:57<98:11:09,  8.70s/it]
                                                        
{'loss': 0.1118, 'grad_norm': 1.1530500650405884, 'learning_rate': 1.951534733441034e-05, 'epoch': 0.15}

  1%|▏         | 604/41250 [1:27:57<98:11:09,  8.70s/it][2025-04-26 05:36:34,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 05:36:34,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5742.30 | bwd_inner_microstep: 5707.58 | bwd_allreduce_microstep: 34.67 | step_microstep: 18.83
[2025-04-26 05:36:34,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5742.31 | bwd_inner: 5707.58 | bwd_allreduce: 34.70 | step: 18.83

  1%|▏         | 605/41250 [1:28:06<98:05:13,  8.69s/it]
                                                        
{'loss': 0.2199, 'grad_norm': 2.3455944061279297, 'learning_rate': 1.9547657512116318e-05, 'epoch': 0.15}

  1%|▏         | 605/41250 [1:28:06<98:05:13,  8.69s/it][2025-04-26 05:36:43,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:36:43,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5724.60 | bwd_inner_microstep: 5688.24 | bwd_allreduce_microstep: 36.31 | step_microstep: 18.61
[2025-04-26 05:36:43,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.69 | bwd: 5724.61 | bwd_inner: 5688.24 | bwd_allreduce: 36.33 | step: 18.61

  1%|▏         | 606/41250 [1:28:14<98:00:29,  8.68s/it]
                                                        
{'loss': 0.2016, 'grad_norm': 2.331568956375122, 'learning_rate': 1.9579967689822296e-05, 'epoch': 0.15}

  1%|▏         | 606/41250 [1:28:14<98:00:29,  8.68s/it][2025-04-26 05:36:52,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:36:52,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.25 | bwd_microstep: 5758.99 | bwd_inner_microstep: 5709.51 | bwd_allreduce_microstep: 49.43 | step_microstep: 18.79
[2025-04-26 05:36:52,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.25 | bwd: 5759.01 | bwd_inner: 5709.51 | bwd_allreduce: 49.45 | step: 18.79

  1%|▏         | 607/41250 [1:28:23<98:04:11,  8.69s/it]
                                                        
{'loss': 0.2488, 'grad_norm': 1.747820258140564, 'learning_rate': 1.9612277867528274e-05, 'epoch': 0.15}

  1%|▏         | 607/41250 [1:28:23<98:04:11,  8.69s/it][2025-04-26 05:37:00,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.28 | optimizer_step: 1.05
[2025-04-26 05:37:00,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.92 | bwd_microstep: 5710.37 | bwd_inner_microstep: 5697.37 | bwd_allreduce_microstep: 12.95 | step_microstep: 20.06
[2025-04-26 05:37:00,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.92 | bwd: 5710.39 | bwd_inner: 5697.37 | bwd_allreduce: 12.98 | step: 20.06

  1%|▏         | 608/41250 [1:28:32<97:56:29,  8.68s/it]
                                                        
{'loss': 0.245, 'grad_norm': 2.5774219036102295, 'learning_rate': 1.9644588045234248e-05, 'epoch': 0.15}

  1%|▏         | 608/41250 [1:28:32<97:56:29,  8.68s/it][2025-04-26 05:37:09,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 05:37:09,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.34 | bwd_microstep: 5695.16 | bwd_inner_microstep: 5653.74 | bwd_allreduce_microstep: 41.38 | step_microstep: 18.59
[2025-04-26 05:37:09,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.34 | bwd: 5695.17 | bwd_inner: 5653.74 | bwd_allreduce: 41.39 | step: 18.59

  1%|▏         | 609/41250 [1:28:40<97:42:41,  8.66s/it]
                                                        
{'loss': 0.0972, 'grad_norm': 1.3529924154281616, 'learning_rate': 1.967689822294023e-05, 'epoch': 0.15}

  1%|▏         | 609/41250 [1:28:40<97:42:41,  8.66s/it][2025-04-26 05:37:18,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 05:37:18,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.87 | bwd_microstep: 5698.57 | bwd_inner_microstep: 5685.84 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.07
[2025-04-26 05:37:18,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.87 | bwd: 5698.58 | bwd_inner: 5685.84 | bwd_allreduce: 12.70 | step: 19.07

  1%|▏         | 610/41250 [1:28:49<97:37:18,  8.65s/it]
                                                        
{'loss': 0.121, 'grad_norm': 1.7729226350784302, 'learning_rate': 1.9709208400646204e-05, 'epoch': 0.15}

  1%|▏         | 610/41250 [1:28:49<97:37:18,  8.65s/it][2025-04-26 05:37:26,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:37:26,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5750.61 | bwd_inner_microstep: 5641.75 | bwd_allreduce_microstep: 108.82 | step_microstep: 18.81
[2025-04-26 05:37:26,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5750.62 | bwd_inner: 5641.75 | bwd_allreduce: 108.84 | step: 18.81

  1%|▏         | 611/41250 [1:28:58<97:40:26,  8.65s/it]
                                                        
{'loss': 0.2905, 'grad_norm': 2.290283203125, 'learning_rate': 1.9741518578352182e-05, 'epoch': 0.15}

  1%|▏         | 611/41250 [1:28:58<97:40:26,  8.65s/it][2025-04-26 05:37:35,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 05:37:35,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2866.69 | bwd_microstep: 5714.96 | bwd_inner_microstep: 5701.97 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.20
[2025-04-26 05:37:35,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2866.69 | bwd: 5714.97 | bwd_inner: 5701.97 | bwd_allreduce: 12.96 | step: 19.20

  1%|▏         | 612/41250 [1:29:06<97:43:39,  8.66s/it]
                                                        
{'loss': 0.2478, 'grad_norm': 3.6335134506225586, 'learning_rate': 1.977382875605816e-05, 'epoch': 0.15}

  1%|▏         | 612/41250 [1:29:06<97:43:39,  8.66s/it][2025-04-26 05:37:44,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.23 | optimizer_step: 1.14
[2025-04-26 05:37:44,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.55 | bwd_microstep: 5741.06 | bwd_inner_microstep: 5639.86 | bwd_allreduce_microstep: 101.15 | step_microstep: 20.03
[2025-04-26 05:37:44,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.55 | bwd: 5741.07 | bwd_inner: 5639.86 | bwd_allreduce: 101.17 | step: 20.03

  1%|▏         | 613/41250 [1:29:15<97:42:31,  8.66s/it]
                                                        
{'loss': 0.3379, 'grad_norm': 3.40838623046875, 'learning_rate': 1.9806138933764138e-05, 'epoch': 0.15}

  1%|▏         | 613/41250 [1:29:15<97:42:31,  8.66s/it][2025-04-26 05:37:52,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 1.07
[2025-04-26 05:37:52,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.04 | bwd_microstep: 5693.70 | bwd_inner_microstep: 5680.81 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.62
[2025-04-26 05:37:52,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.04 | bwd: 5693.72 | bwd_inner: 5680.81 | bwd_allreduce: 12.86 | step: 19.62

  1%|▏         | 614/41250 [1:29:23<97:36:33,  8.65s/it]
                                                        
{'loss': 0.2053, 'grad_norm': 2.0256433486938477, 'learning_rate': 1.9838449111470116e-05, 'epoch': 0.15}

  1%|▏         | 614/41250 [1:29:23<97:36:33,  8.65s/it][2025-04-26 05:38:01,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 05:38:01,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.01 | bwd_microstep: 5723.09 | bwd_inner_microstep: 5704.15 | bwd_allreduce_microstep: 18.89 | step_microstep: 18.63
[2025-04-26 05:38:01,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.01 | bwd: 5723.10 | bwd_inner: 5704.15 | bwd_allreduce: 18.91 | step: 18.63

  1%|▏         | 615/41250 [1:29:32<97:39:14,  8.65s/it]
                                                        
{'loss': 0.1346, 'grad_norm': 1.4497092962265015, 'learning_rate': 1.987075928917609e-05, 'epoch': 0.15}

  1%|▏         | 615/41250 [1:29:32<97:39:14,  8.65s/it][2025-04-26 05:38:10,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 05:38:10,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.97 | bwd_microstep: 5709.84 | bwd_inner_microstep: 5691.34 | bwd_allreduce_microstep: 18.45 | step_microstep: 19.39
[2025-04-26 05:38:10,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.97 | bwd: 5709.86 | bwd_inner: 5691.34 | bwd_allreduce: 18.47 | step: 19.40

  1%|▏         | 616/41250 [1:29:41<97:39:30,  8.65s/it]
                                                        
{'loss': 0.3028, 'grad_norm': 2.614762306213379, 'learning_rate': 1.990306946688207e-05, 'epoch': 0.15}

  1%|▏         | 616/41250 [1:29:41<97:39:30,  8.65s/it][2025-04-26 05:38:18,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:38:18,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.51 | bwd_microstep: 5785.77 | bwd_inner_microstep: 5773.01 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.75
[2025-04-26 05:38:18,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.50 | bwd: 5785.78 | bwd_inner: 5773.01 | bwd_allreduce: 12.73 | step: 18.75

  1%|▏         | 617/41250 [1:29:50<98:02:23,  8.69s/it]
                                                        
{'loss': 0.2125, 'grad_norm': 2.584275722503662, 'learning_rate': 1.9935379644588046e-05, 'epoch': 0.15}

  1%|▏         | 617/41250 [1:29:50<98:02:23,  8.69s/it][2025-04-26 05:38:27,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:38:27,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.78 | bwd_microstep: 5716.84 | bwd_inner_microstep: 5649.85 | bwd_allreduce_microstep: 66.95 | step_microstep: 18.83
[2025-04-26 05:38:27,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.78 | bwd: 5716.86 | bwd_inner: 5649.85 | bwd_allreduce: 66.96 | step: 18.83

  1%|▏         | 618/41250 [1:29:58<97:56:17,  8.68s/it]
                                                        
{'loss': 0.136, 'grad_norm': 1.542458176612854, 'learning_rate': 1.9967689822294024e-05, 'epoch': 0.15}

  1%|▏         | 618/41250 [1:29:58<97:56:17,  8.68s/it][2025-04-26 05:38:36,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-26 05:38:36,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.57 | bwd_microstep: 5771.04 | bwd_inner_microstep: 5646.68 | bwd_allreduce_microstep: 124.30 | step_microstep: 19.54
[2025-04-26 05:38:36,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.57 | bwd: 5771.05 | bwd_inner: 5646.68 | bwd_allreduce: 124.33 | step: 19.54

  2%|▏         | 619/41250 [1:30:07<97:57:33,  8.68s/it]
                                                        
{'loss': 0.4513, 'grad_norm': 5.73507022857666, 'learning_rate': 2e-05, 'epoch': 0.15}

  2%|▏         | 619/41250 [1:30:07<97:57:33,  8.68s/it][2025-04-26 05:38:44,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 05:38:44,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.14 | bwd_microstep: 5711.25 | bwd_inner_microstep: 5698.80 | bwd_allreduce_microstep: 12.41 | step_microstep: 19.12
[2025-04-26 05:38:44,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.14 | bwd: 5711.27 | bwd_inner: 5698.80 | bwd_allreduce: 12.43 | step: 19.12

  2%|▏         | 620/41250 [1:30:16<97:53:47,  8.67s/it]
                                                        
{'loss': 0.0638, 'grad_norm': 1.2047359943389893, 'learning_rate': 2.003231017770598e-05, 'epoch': 0.15}

  2%|▏         | 620/41250 [1:30:16<97:53:47,  8.67s/it][2025-04-26 05:38:53,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 0.91
[2025-04-26 05:38:53,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.07 | bwd_microstep: 5744.15 | bwd_inner_microstep: 5645.96 | bwd_allreduce_microstep: 98.14 | step_microstep: 18.82
[2025-04-26 05:38:53,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.07 | bwd: 5744.17 | bwd_inner: 5645.96 | bwd_allreduce: 98.17 | step: 18.82

  2%|▏         | 621/41250 [1:30:24<97:50:50,  8.67s/it]
                                                        
{'loss': 0.3897, 'grad_norm': 5.737129211425781, 'learning_rate': 2.0064620355411954e-05, 'epoch': 0.15}

  2%|▏         | 621/41250 [1:30:24<97:50:50,  8.67s/it][2025-04-26 05:39:02,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 1.19
[2025-04-26 05:39:02,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.35 | bwd_microstep: 5709.93 | bwd_inner_microstep: 5697.06 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.25
[2025-04-26 05:39:02,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.35 | bwd: 5709.94 | bwd_inner: 5697.06 | bwd_allreduce: 12.83 | step: 19.26

  2%|▏         | 622/41250 [1:30:33<97:47:00,  8.66s/it]
                                                        
{'loss': 0.2539, 'grad_norm': 4.020270824432373, 'learning_rate': 2.0096930533117932e-05, 'epoch': 0.15}

  2%|▏         | 622/41250 [1:30:33<97:47:00,  8.66s/it][2025-04-26 05:39:10,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:39:10,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.77 | bwd_microstep: 5769.13 | bwd_inner_microstep: 5654.32 | bwd_allreduce_microstep: 114.77 | step_microstep: 18.40
[2025-04-26 05:39:10,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.77 | bwd: 5769.15 | bwd_inner: 5654.32 | bwd_allreduce: 114.79 | step: 18.41

  2%|▏         | 623/41250 [1:30:42<97:47:52,  8.67s/it]
                                                        
{'loss': 0.1422, 'grad_norm': 2.026920795440674, 'learning_rate': 2.0129240710823913e-05, 'epoch': 0.15}

  2%|▏         | 623/41250 [1:30:42<97:47:52,  8.67s/it][2025-04-26 05:39:19,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 05:39:19,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.85 | bwd_microstep: 5696.01 | bwd_inner_microstep: 5660.37 | bwd_allreduce_microstep: 35.59 | step_microstep: 18.85
[2025-04-26 05:39:19,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.85 | bwd: 5696.03 | bwd_inner: 5660.37 | bwd_allreduce: 35.62 | step: 18.86

  2%|▏         | 624/41250 [1:30:50<97:37:58,  8.65s/it]
                                                        
{'loss': 0.2337, 'grad_norm': 6.307132244110107, 'learning_rate': 2.016155088852989e-05, 'epoch': 0.15}

  2%|▏         | 624/41250 [1:30:50<97:37:58,  8.65s/it][2025-04-26 05:39:28,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 05:39:28,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.33 | bwd_microstep: 5707.29 | bwd_inner_microstep: 5694.35 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.33
[2025-04-26 05:39:28,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.33 | bwd: 5707.31 | bwd_inner: 5694.35 | bwd_allreduce: 12.92 | step: 19.34

  2%|▏         | 625/41250 [1:30:59<97:37:29,  8.65s/it]
                                                        
{'loss': 0.4353, 'grad_norm': 3.3185834884643555, 'learning_rate': 2.0193861066235866e-05, 'epoch': 0.15}

  2%|▏         | 625/41250 [1:30:59<97:37:29,  8.65s/it][2025-04-26 05:39:36,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:39:36,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.95 | bwd_microstep: 5730.52 | bwd_inner_microstep: 5717.86 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.57
[2025-04-26 05:39:36,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.95 | bwd: 5730.53 | bwd_inner: 5717.86 | bwd_allreduce: 12.62 | step: 18.57

  2%|▏         | 626/41250 [1:31:08<97:42:39,  8.66s/it]
                                                        
{'loss': 0.1866, 'grad_norm': 2.483189105987549, 'learning_rate': 2.0226171243941843e-05, 'epoch': 0.15}

  2%|▏         | 626/41250 [1:31:08<97:42:39,  8.66s/it][2025-04-26 05:39:45,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:39:45,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.89 | bwd_microstep: 5704.29 | bwd_inner_microstep: 5657.90 | bwd_allreduce_microstep: 46.35 | step_microstep: 18.40
[2025-04-26 05:39:45,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.89 | bwd: 5704.30 | bwd_inner: 5657.90 | bwd_allreduce: 46.37 | step: 18.40

  2%|▏         | 627/41250 [1:31:16<97:33:25,  8.65s/it]
                                                        
{'loss': 0.1331, 'grad_norm': 2.66581654548645, 'learning_rate': 2.025848142164782e-05, 'epoch': 0.15}

  2%|▏         | 627/41250 [1:31:16<97:33:25,  8.65s/it][2025-04-26 05:39:54,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:39:54,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.27 | bwd_microstep: 5752.92 | bwd_inner_microstep: 5655.38 | bwd_allreduce_microstep: 97.50 | step_microstep: 18.43
[2025-04-26 05:39:54,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.27 | bwd: 5752.94 | bwd_inner: 5655.38 | bwd_allreduce: 97.51 | step: 18.43

  2%|▏         | 628/41250 [1:31:25<97:38:56,  8.65s/it]
                                                        
{'loss': 0.3087, 'grad_norm': 4.5847487449646, 'learning_rate': 2.0290791599353796e-05, 'epoch': 0.15}

  2%|▏         | 628/41250 [1:31:25<97:38:56,  8.65s/it][2025-04-26 05:40:02,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 05:40:02,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5849.33 | bwd_inner_microstep: 5709.99 | bwd_allreduce_microstep: 139.29 | step_microstep: 18.97
[2025-04-26 05:40:02,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5849.34 | bwd_inner: 5709.99 | bwd_allreduce: 139.32 | step: 18.97

  2%|▏         | 629/41250 [1:31:34<98:07:00,  8.70s/it]
                                                        
{'loss': 0.3038, 'grad_norm': 3.4791958332061768, 'learning_rate': 2.0323101777059774e-05, 'epoch': 0.15}

  2%|▏         | 629/41250 [1:31:34<98:07:00,  8.70s/it][2025-04-26 05:40:11,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 05:40:11,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.46 | bwd_microstep: 5746.11 | bwd_inner_microstep: 5694.36 | bwd_allreduce_microstep: 51.71 | step_microstep: 18.81
[2025-04-26 05:40:11,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.46 | bwd: 5746.13 | bwd_inner: 5694.36 | bwd_allreduce: 51.73 | step: 18.81

  2%|▏         | 630/41250 [1:31:42<98:06:11,  8.69s/it]
                                                        
{'loss': 0.3032, 'grad_norm': 2.122748374938965, 'learning_rate': 2.0355411954765755e-05, 'epoch': 0.15}

  2%|▏         | 630/41250 [1:31:42<98:06:11,  8.69s/it][2025-04-26 05:40:20,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:40:20,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.62 | bwd_microstep: 5774.88 | bwd_inner_microstep: 5647.63 | bwd_allreduce_microstep: 127.21 | step_microstep: 18.26
[2025-04-26 05:40:20,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.62 | bwd: 5774.90 | bwd_inner: 5647.63 | bwd_allreduce: 127.23 | step: 18.26

  2%|▏         | 631/41250 [1:31:51<98:02:17,  8.69s/it]
                                                        
{'loss': 0.3846, 'grad_norm': 3.4382479190826416, 'learning_rate': 2.0387722132471733e-05, 'epoch': 0.15}

  2%|▏         | 631/41250 [1:31:51<98:02:17,  8.69s/it][2025-04-26 05:40:28,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 05:40:28,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.96 | bwd_microstep: 5781.80 | bwd_inner_microstep: 5653.02 | bwd_allreduce_microstep: 128.74 | step_microstep: 18.42
[2025-04-26 05:40:28,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.96 | bwd: 5781.81 | bwd_inner: 5653.02 | bwd_allreduce: 128.75 | step: 18.42

  2%|▏         | 632/41250 [1:32:00<98:01:54,  8.69s/it]
                                                        
{'loss': 0.1948, 'grad_norm': 4.820442199707031, 'learning_rate': 2.0420032310177707e-05, 'epoch': 0.15}

  2%|▏         | 632/41250 [1:32:00<98:01:54,  8.69s/it][2025-04-26 05:40:37,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-26 05:40:37,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.95 | bwd_microstep: 5707.16 | bwd_inner_microstep: 5660.17 | bwd_allreduce_microstep: 46.95 | step_microstep: 18.46
[2025-04-26 05:40:37,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.95 | bwd: 5707.18 | bwd_inner: 5660.17 | bwd_allreduce: 46.96 | step: 18.46

  2%|▏         | 633/41250 [1:32:08<97:48:46,  8.67s/it]
                                                        
{'loss': 0.153, 'grad_norm': 1.4712408781051636, 'learning_rate': 2.0452342487883685e-05, 'epoch': 0.15}

  2%|▏         | 633/41250 [1:32:08<97:48:46,  8.67s/it][2025-04-26 05:40:46,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 05:40:46,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.18 | bwd_microstep: 5709.20 | bwd_inner_microstep: 5670.98 | bwd_allreduce_microstep: 38.18 | step_microstep: 18.60
[2025-04-26 05:40:46,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.18 | bwd: 5709.22 | bwd_inner: 5670.98 | bwd_allreduce: 38.20 | step: 18.61

  2%|▏         | 634/41250 [1:32:17<97:42:48,  8.66s/it]
                                                        
{'loss': 0.3349, 'grad_norm': 3.910184621810913, 'learning_rate': 2.0484652665589663e-05, 'epoch': 0.15}

  2%|▏         | 634/41250 [1:32:17<97:42:48,  8.66s/it][2025-04-26 05:40:54,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 05:40:54,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.49 | bwd_microstep: 5705.43 | bwd_inner_microstep: 5662.44 | bwd_allreduce_microstep: 42.94 | step_microstep: 19.00
[2025-04-26 05:40:54,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.49 | bwd: 5705.44 | bwd_inner: 5662.44 | bwd_allreduce: 42.96 | step: 19.00

  2%|▏         | 635/41250 [1:32:26<97:35:45,  8.65s/it]
                                                        
{'loss': 0.1442, 'grad_norm': 2.4878454208374023, 'learning_rate': 2.0516962843295638e-05, 'epoch': 0.15}

  2%|▏         | 635/41250 [1:32:26<97:35:45,  8.65s/it][2025-04-26 05:41:03,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:41:03,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.73 | bwd_microstep: 5775.34 | bwd_inner_microstep: 5682.44 | bwd_allreduce_microstep: 92.86 | step_microstep: 18.67
[2025-04-26 05:41:03,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.73 | bwd: 5775.36 | bwd_inner: 5682.44 | bwd_allreduce: 92.87 | step: 18.67

  2%|▏         | 636/41250 [1:32:34<97:48:43,  8.67s/it]
                                                        
{'loss': 0.1613, 'grad_norm': 2.355635404586792, 'learning_rate': 2.0549273021001615e-05, 'epoch': 0.15}

  2%|▏         | 636/41250 [1:32:34<97:48:43,  8.67s/it][2025-04-26 05:41:12,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 05:41:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.83 | bwd_microstep: 5774.88 | bwd_inner_microstep: 5698.89 | bwd_allreduce_microstep: 75.95 | step_microstep: 19.13
[2025-04-26 05:41:12,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.83 | bwd: 5774.90 | bwd_inner: 5698.89 | bwd_allreduce: 75.96 | step: 19.13

  2%|▏         | 637/41250 [1:32:43<97:55:18,  8.68s/it]
                                                        
{'loss': 0.1974, 'grad_norm': 2.3734209537506104, 'learning_rate': 2.0581583198707597e-05, 'epoch': 0.15}

  2%|▏         | 637/41250 [1:32:43<97:55:18,  8.68s/it][2025-04-26 05:41:20,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 05:41:20,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.30 | bwd_microstep: 5687.83 | bwd_inner_microstep: 5666.44 | bwd_allreduce_microstep: 21.34 | step_microstep: 18.90
[2025-04-26 05:41:20,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.30 | bwd: 5687.84 | bwd_inner: 5666.44 | bwd_allreduce: 21.36 | step: 18.90

  2%|▏         | 638/41250 [1:32:52<97:43:20,  8.66s/it]
                                                        
{'loss': 0.0955, 'grad_norm': 3.7190334796905518, 'learning_rate': 2.0613893376413575e-05, 'epoch': 0.15}

  2%|▏         | 638/41250 [1:32:52<97:43:20,  8.66s/it][2025-04-26 05:41:29,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 05:41:29,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5755.33 | bwd_inner_microstep: 5664.96 | bwd_allreduce_microstep: 90.32 | step_microstep: 18.64
[2025-04-26 05:41:29,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5755.34 | bwd_inner: 5664.96 | bwd_allreduce: 90.34 | step: 18.65

  2%|▏         | 639/41250 [1:33:00<97:47:51,  8.67s/it]
                                                        
{'loss': 0.3002, 'grad_norm': 3.50892972946167, 'learning_rate': 2.064620355411955e-05, 'epoch': 0.15}

  2%|▏         | 639/41250 [1:33:00<97:47:51,  8.67s/it][2025-04-26 05:41:38,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 05:41:38,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.96 | bwd_microstep: 5890.02 | bwd_inner_microstep: 5709.84 | bwd_allreduce_microstep: 180.13 | step_microstep: 19.09
[2025-04-26 05:41:38,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.96 | bwd: 5890.03 | bwd_inner: 5709.84 | bwd_allreduce: 180.15 | step: 19.10

  2%|▏         | 640/41250 [1:33:09<98:19:03,  8.72s/it]
                                                        
{'loss': 0.2931, 'grad_norm': 3.8975117206573486, 'learning_rate': 2.0678513731825527e-05, 'epoch': 0.16}

  2%|▏         | 640/41250 [1:33:09<98:19:03,  8.72s/it][2025-04-26 05:41:47,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 05:41:47,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5788.89 | bwd_inner_microstep: 5648.06 | bwd_allreduce_microstep: 140.78 | step_microstep: 19.04
[2025-04-26 05:41:47,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5788.90 | bwd_inner: 5648.06 | bwd_allreduce: 140.80 | step: 19.04

  2%|▏         | 641/41250 [1:33:18<98:14:00,  8.71s/it]
                                                        
{'loss': 0.1286, 'grad_norm': 1.8504588603973389, 'learning_rate': 2.0710823909531505e-05, 'epoch': 0.16}

  2%|▏         | 641/41250 [1:33:18<98:14:00,  8.71s/it][2025-04-26 05:41:55,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:41:55,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.96 | bwd_microstep: 5777.68 | bwd_inner_microstep: 5714.06 | bwd_allreduce_microstep: 63.58 | step_microstep: 18.63
[2025-04-26 05:41:55,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.96 | bwd: 5777.69 | bwd_inner: 5714.06 | bwd_allreduce: 63.59 | step: 18.63

  2%|▏         | 642/41250 [1:33:26<98:15:45,  8.71s/it]
                                                        
{'loss': 0.4061, 'grad_norm': 3.452453136444092, 'learning_rate': 2.074313408723748e-05, 'epoch': 0.16}

  2%|▏         | 642/41250 [1:33:26<98:15:45,  8.71s/it][2025-04-26 05:42:04,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 05:42:04,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.67 | bwd_microstep: 5801.61 | bwd_inner_microstep: 5652.96 | bwd_allreduce_microstep: 148.60 | step_microstep: 19.04
[2025-04-26 05:42:04,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.67 | bwd: 5801.63 | bwd_inner: 5652.96 | bwd_allreduce: 148.63 | step: 19.04

  2%|▏         | 643/41250 [1:33:35<98:15:25,  8.71s/it]
                                                        
{'loss': 0.0976, 'grad_norm': 1.3166027069091797, 'learning_rate': 2.0775444264943457e-05, 'epoch': 0.16}

  2%|▏         | 643/41250 [1:33:35<98:15:25,  8.71s/it][2025-04-26 05:42:13,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 05:42:13,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.83 | bwd_microstep: 5709.58 | bwd_inner_microstep: 5666.78 | bwd_allreduce_microstep: 42.76 | step_microstep: 18.92
[2025-04-26 05:42:13,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.83 | bwd: 5709.59 | bwd_inner: 5666.78 | bwd_allreduce: 42.78 | step: 18.93

  2%|▏         | 644/41250 [1:33:44<97:56:55,  8.68s/it]
                                                        
{'loss': 0.0989, 'grad_norm': 2.7354960441589355, 'learning_rate': 2.080775444264944e-05, 'epoch': 0.16}

  2%|▏         | 644/41250 [1:33:44<97:56:55,  8.68s/it][2025-04-26 05:42:21,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:42:21,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.84 | bwd_microstep: 5716.84 | bwd_inner_microstep: 5704.05 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.56
[2025-04-26 05:42:21,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.84 | bwd: 5716.85 | bwd_inner: 5704.05 | bwd_allreduce: 12.77 | step: 18.57

  2%|▏         | 645/41250 [1:33:52<97:51:58,  8.68s/it]
                                                        
{'loss': 0.2754, 'grad_norm': 1.9897457361221313, 'learning_rate': 2.0840064620355416e-05, 'epoch': 0.16}

  2%|▏         | 645/41250 [1:33:52<97:51:58,  8.68s/it][2025-04-26 05:42:30,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:42:30,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.72 | bwd_microstep: 5711.64 | bwd_inner_microstep: 5656.57 | bwd_allreduce_microstep: 55.02 | step_microstep: 18.56
[2025-04-26 05:42:30,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.72 | bwd: 5711.65 | bwd_inner: 5656.57 | bwd_allreduce: 55.04 | step: 18.56

  2%|▏         | 646/41250 [1:34:01<97:41:30,  8.66s/it]
                                                        
{'loss': 0.24, 'grad_norm': 2.3430652618408203, 'learning_rate': 2.087237479806139e-05, 'epoch': 0.16}

  2%|▏         | 646/41250 [1:34:01<97:41:30,  8.66s/it][2025-04-26 05:42:39,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 05:42:39,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.54 | bwd_microstep: 5778.50 | bwd_inner_microstep: 5661.79 | bwd_allreduce_microstep: 116.66 | step_microstep: 18.64
[2025-04-26 05:42:39,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.54 | bwd: 5778.52 | bwd_inner: 5661.79 | bwd_allreduce: 116.68 | step: 18.64

  2%|▏         | 647/41250 [1:34:10<97:47:42,  8.67s/it]
                                                        
{'loss': 0.0942, 'grad_norm': 1.3910795450210571, 'learning_rate': 2.090468497576737e-05, 'epoch': 0.16}

  2%|▏         | 647/41250 [1:34:10<97:47:42,  8.67s/it][2025-04-26 05:42:47,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 05:42:47,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5652.37 | bwd_allreduce_microstep: 114.62 | step_microstep: 19.11
[2025-04-26 05:42:47,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5767.05 | bwd_inner: 5652.37 | bwd_allreduce: 114.64 | step: 19.11

  2%|▏         | 648/41250 [1:34:18<97:48:09,  8.67s/it]
                                                        
{'loss': 0.3024, 'grad_norm': 2.3532700538635254, 'learning_rate': 2.0936995153473347e-05, 'epoch': 0.16}

  2%|▏         | 648/41250 [1:34:18<97:48:09,  8.67s/it][2025-04-26 05:42:56,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 05:42:56,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.43 | bwd_microstep: 5712.15 | bwd_inner_microstep: 5699.44 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.21
[2025-04-26 05:42:56,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.43 | bwd: 5712.17 | bwd_inner: 5699.44 | bwd_allreduce: 12.68 | step: 19.21

  2%|▏         | 649/41250 [1:34:27<97:42:14,  8.66s/it]
                                                        
{'loss': 0.2906, 'grad_norm': 3.1226017475128174, 'learning_rate': 2.096930533117932e-05, 'epoch': 0.16}

  2%|▏         | 649/41250 [1:34:27<97:42:14,  8.66s/it][2025-04-26 05:43:05,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 05:43:05,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5704.87 | bwd_inner_microstep: 5692.09 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.09
[2025-04-26 05:43:05,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5704.88 | bwd_inner: 5692.09 | bwd_allreduce: 12.75 | step: 19.09

  2%|▏         | 650/41250 [1:34:36<97:35:40,  8.65s/it]
                                                        
{'loss': 0.2331, 'grad_norm': 3.7249512672424316, 'learning_rate': 2.10016155088853e-05, 'epoch': 0.16}

  2%|▏         | 650/41250 [1:34:36<97:35:40,  8.65s/it][2025-04-26 05:43:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 05:43:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.42 | bwd_microstep: 5696.68 | bwd_inner_microstep: 5683.86 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.99
[2025-04-26 05:43:13,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.42 | bwd: 5696.70 | bwd_inner: 5683.86 | bwd_allreduce: 12.79 | step: 18.99

  2%|▏         | 651/41250 [1:34:44<97:32:26,  8.65s/it]
                                                        
{'loss': 0.2575, 'grad_norm': 2.3050930500030518, 'learning_rate': 2.103392568659128e-05, 'epoch': 0.16}

  2%|▏         | 651/41250 [1:34:44<97:32:26,  8.65s/it][2025-04-26 05:43:22,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:43:22,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.60 | bwd_microstep: 5860.89 | bwd_inner_microstep: 5848.05 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.68
[2025-04-26 05:43:22,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.60 | bwd: 5860.90 | bwd_inner: 5848.05 | bwd_allreduce: 12.81 | step: 18.68

  2%|▏         | 652/41250 [1:34:53<98:18:28,  8.72s/it]
                                                        
{'loss': 0.1937, 'grad_norm': 1.7958403825759888, 'learning_rate': 2.1066235864297258e-05, 'epoch': 0.16}

  2%|▏         | 652/41250 [1:34:53<98:18:28,  8.72s/it][2025-04-26 05:43:31,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:43:31,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.70 | bwd_microstep: 5707.50 | bwd_inner_microstep: 5691.66 | bwd_allreduce_microstep: 15.79 | step_microstep: 18.70
[2025-04-26 05:43:31,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.70 | bwd: 5707.52 | bwd_inner: 5691.66 | bwd_allreduce: 15.81 | step: 18.70

  2%|▏         | 653/41250 [1:35:02<98:05:21,  8.70s/it]
                                                        
{'loss': 0.1929, 'grad_norm': 4.213986396789551, 'learning_rate': 2.1098546042003233e-05, 'epoch': 0.16}

  2%|▏         | 653/41250 [1:35:02<98:05:21,  8.70s/it][2025-04-26 05:43:39,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 05:43:39,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5691.46 | bwd_inner_microstep: 5678.42 | bwd_allreduce_microstep: 13.00 | step_microstep: 19.34
[2025-04-26 05:43:39,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5691.48 | bwd_inner: 5678.42 | bwd_allreduce: 13.02 | step: 19.35

  2%|▏         | 654/41250 [1:35:11<97:49:18,  8.67s/it]
                                                        
{'loss': 0.2484, 'grad_norm': 1.8950718641281128, 'learning_rate': 2.113085621970921e-05, 'epoch': 0.16}

  2%|▏         | 654/41250 [1:35:11<97:49:18,  8.67s/it][2025-04-26 05:43:48,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 05:43:48,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.50 | bwd_microstep: 5789.93 | bwd_inner_microstep: 5777.17 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.66
[2025-04-26 05:43:48,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.50 | bwd: 5789.94 | bwd_inner: 5777.17 | bwd_allreduce: 12.73 | step: 18.66

  2%|▏         | 655/41250 [1:35:19<98:05:13,  8.70s/it]
                                                        
{'loss': 0.2641, 'grad_norm': 2.4488027095794678, 'learning_rate': 2.116316639741519e-05, 'epoch': 0.16}

  2%|▏         | 655/41250 [1:35:19<98:05:13,  8.70s/it][2025-04-26 05:43:57,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.19
[2025-04-26 05:43:57,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.27 | bwd_microstep: 5728.19 | bwd_inner_microstep: 5685.25 | bwd_allreduce_microstep: 42.90 | step_microstep: 19.35
[2025-04-26 05:43:57,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.27 | bwd: 5728.21 | bwd_inner: 5685.25 | bwd_allreduce: 42.92 | step: 19.35

  2%|▏         | 656/41250 [1:35:28<97:59:03,  8.69s/it]
                                                        
{'loss': 0.1286, 'grad_norm': 2.7904133796691895, 'learning_rate': 2.1195476575121163e-05, 'epoch': 0.16}

  2%|▏         | 656/41250 [1:35:28<97:59:03,  8.69s/it][2025-04-26 05:44:05,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:44:05,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5712.56 | bwd_inner_microstep: 5699.71 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.86
[2025-04-26 05:44:05,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5712.57 | bwd_inner: 5699.71 | bwd_allreduce: 12.82 | step: 18.86

  2%|▏         | 657/41250 [1:35:37<97:50:20,  8.68s/it]
                                                        
{'loss': 0.3296, 'grad_norm': 2.8270065784454346, 'learning_rate': 2.122778675282714e-05, 'epoch': 0.16}

  2%|▏         | 657/41250 [1:35:37<97:50:20,  8.68s/it][2025-04-26 05:44:14,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 05:44:14,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.15 | bwd_microstep: 5693.49 | bwd_inner_microstep: 5641.64 | bwd_allreduce_microstep: 51.80 | step_microstep: 19.31
[2025-04-26 05:44:14,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.15 | bwd: 5693.50 | bwd_inner: 5641.64 | bwd_allreduce: 51.82 | step: 19.32

  2%|▏         | 658/41250 [1:35:45<97:35:04,  8.65s/it]
                                                        
{'loss': 0.0588, 'grad_norm': 1.0040849447250366, 'learning_rate': 2.1260096930533122e-05, 'epoch': 0.16}

  2%|▏         | 658/41250 [1:35:45<97:35:04,  8.65s/it][2025-04-26 05:44:23,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 05:44:23,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.84 | bwd_microstep: 5734.53 | bwd_inner_microstep: 5686.25 | bwd_allreduce_microstep: 48.23 | step_microstep: 18.83
[2025-04-26 05:44:23,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5734.54 | bwd_inner: 5686.25 | bwd_allreduce: 48.25 | step: 18.84

  2%|▏         | 659/41250 [1:35:54<97:34:30,  8.65s/it]
                                                        
{'loss': 0.1398, 'grad_norm': 1.0784105062484741, 'learning_rate': 2.1292407108239093e-05, 'epoch': 0.16}

  2%|▏         | 659/41250 [1:35:54<97:34:30,  8.65s/it][2025-04-26 05:44:31,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:44:31,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.06 | bwd_microstep: 5865.88 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 171.05 | step_microstep: 18.40
[2025-04-26 05:44:31,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.06 | bwd: 5865.90 | bwd_inner: 5694.79 | bwd_allreduce: 171.06 | step: 18.40

  2%|▏         | 660/41250 [1:36:03<98:06:00,  8.70s/it]
                                                        
{'loss': 0.2765, 'grad_norm': 2.2334649562835693, 'learning_rate': 2.1324717285945075e-05, 'epoch': 0.16}

  2%|▏         | 660/41250 [1:36:03<98:06:00,  8.70s/it][2025-04-26 05:44:40,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 05:44:40,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.52 | bwd_microstep: 5860.05 | bwd_inner_microstep: 5693.11 | bwd_allreduce_microstep: 166.89 | step_microstep: 18.56
[2025-04-26 05:44:40,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.52 | bwd: 5860.07 | bwd_inner: 5693.11 | bwd_allreduce: 166.91 | step: 18.56

  2%|▏         | 661/41250 [1:36:11<98:24:16,  8.73s/it]
                                                        
{'loss': 0.3643, 'grad_norm': 3.7845993041992188, 'learning_rate': 2.1357027463651052e-05, 'epoch': 0.16}

  2%|▏         | 661/41250 [1:36:11<98:24:16,  8.73s/it][2025-04-26 05:44:49,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 05:44:49,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5700.24 | bwd_inner_microstep: 5644.16 | bwd_allreduce_microstep: 56.03 | step_microstep: 19.35
[2025-04-26 05:44:49,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5700.25 | bwd_inner: 5644.16 | bwd_allreduce: 56.05 | step: 19.35

  2%|▏         | 662/41250 [1:36:20<98:00:37,  8.69s/it]
                                                        
{'loss': 0.1762, 'grad_norm': 3.8950014114379883, 'learning_rate': 2.138933764135703e-05, 'epoch': 0.16}

  2%|▏         | 662/41250 [1:36:20<98:00:37,  8.69s/it][2025-04-26 05:44:58,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:44:58,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.10 | bwd_microstep: 5770.37 | bwd_inner_microstep: 5648.16 | bwd_allreduce_microstep: 122.17 | step_microstep: 18.65
[2025-04-26 05:44:58,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.10 | bwd: 5770.38 | bwd_inner: 5648.16 | bwd_allreduce: 122.18 | step: 18.65

  2%|▏         | 663/41250 [1:36:29<97:57:58,  8.69s/it]
                                                        
{'loss': 0.1631, 'grad_norm': 2.304525852203369, 'learning_rate': 2.1421647819063005e-05, 'epoch': 0.16}

  2%|▏         | 663/41250 [1:36:29<97:57:58,  8.69s/it][2025-04-26 05:45:06,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 05:45:06,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5713.65 | bwd_inner_microstep: 5700.84 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.17
[2025-04-26 05:45:06,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5713.66 | bwd_inner: 5700.84 | bwd_allreduce: 12.79 | step: 18.17

  2%|▏         | 664/41250 [1:36:37<97:47:05,  8.67s/it]
                                                        
{'loss': 0.0982, 'grad_norm': 1.1089482307434082, 'learning_rate': 2.1453957996768983e-05, 'epoch': 0.16}

  2%|▏         | 664/41250 [1:36:37<97:47:05,  8.67s/it][2025-04-26 05:45:15,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:45:15,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.65 | bwd_microstep: 5705.78 | bwd_inner_microstep: 5693.09 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.10
[2025-04-26 05:45:15,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.65 | bwd: 5705.79 | bwd_inner: 5693.09 | bwd_allreduce: 12.66 | step: 18.11

  2%|▏         | 665/41250 [1:36:46<97:39:00,  8.66s/it]
                                                        
{'loss': 0.2082, 'grad_norm': 2.033320665359497, 'learning_rate': 2.1486268174474964e-05, 'epoch': 0.16}

  2%|▏         | 665/41250 [1:36:46<97:39:00,  8.66s/it][2025-04-26 05:45:24,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.20 | optimizer_step: 1.04
[2025-04-26 05:45:24,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.08 | bwd_microstep: 5776.10 | bwd_inner_microstep: 5656.88 | bwd_allreduce_microstep: 119.18 | step_microstep: 19.33
[2025-04-26 05:45:24,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.08 | bwd: 5776.12 | bwd_inner: 5656.88 | bwd_allreduce: 119.20 | step: 19.33

  2%|▏         | 666/41250 [1:36:55<97:44:24,  8.67s/it]
                                                        
{'loss': 0.2651, 'grad_norm': 2.8020505905151367, 'learning_rate': 2.1518578352180935e-05, 'epoch': 0.16}

  2%|▏         | 666/41250 [1:36:55<97:44:24,  8.67s/it][2025-04-26 05:45:32,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:45:32,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.60 | bwd_microstep: 5666.45 | bwd_inner_microstep: 5643.97 | bwd_allreduce_microstep: 22.43 | step_microstep: 18.67
[2025-04-26 05:45:32,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.60 | bwd: 5666.46 | bwd_inner: 5643.97 | bwd_allreduce: 22.45 | step: 18.67

  2%|▏         | 667/41250 [1:37:03<97:27:48,  8.65s/it]
                                                        
{'loss': 0.1355, 'grad_norm': 2.1614248752593994, 'learning_rate': 2.1550888529886916e-05, 'epoch': 0.16}

  2%|▏         | 667/41250 [1:37:03<97:27:48,  8.65s/it][2025-04-26 05:45:41,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:45:41,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5651.37 | bwd_allreduce_microstep: 66.53 | step_microstep: 18.59
[2025-04-26 05:45:41,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5717.95 | bwd_inner: 5651.37 | bwd_allreduce: 66.54 | step: 18.59

  2%|▏         | 668/41250 [1:37:12<97:21:45,  8.64s/it]
                                                        
{'loss': 0.4229, 'grad_norm': 3.4806578159332275, 'learning_rate': 2.1583198707592894e-05, 'epoch': 0.16}

  2%|▏         | 668/41250 [1:37:12<97:21:45,  8.64s/it][2025-04-26 05:45:50,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 05:45:50,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.93 | bwd_microstep: 5795.42 | bwd_inner_microstep: 5782.82 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.78
[2025-04-26 05:45:50,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.93 | bwd: 5795.44 | bwd_inner: 5782.82 | bwd_allreduce: 12.58 | step: 18.79

  2%|▏         | 669/41250 [1:37:21<97:49:53,  8.68s/it]
                                                        
{'loss': 0.2932, 'grad_norm': 2.308439016342163, 'learning_rate': 2.1615508885298872e-05, 'epoch': 0.16}

  2%|▏         | 669/41250 [1:37:21<97:49:53,  8.68s/it][2025-04-26 05:45:58,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:45:58,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.55 | bwd_microstep: 5686.94 | bwd_inner_microstep: 5672.58 | bwd_allreduce_microstep: 14.32 | step_microstep: 18.67
[2025-04-26 05:45:58,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.55 | bwd: 5686.95 | bwd_inner: 5672.58 | bwd_allreduce: 14.34 | step: 18.67

  2%|▏         | 670/41250 [1:37:29<97:36:23,  8.66s/it]
                                                        
{'loss': 0.0538, 'grad_norm': 0.8031871914863586, 'learning_rate': 2.1647819063004847e-05, 'epoch': 0.16}

  2%|▏         | 670/41250 [1:37:29<97:36:23,  8.66s/it][2025-04-26 05:46:07,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 05:46:07,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.36 | bwd_microstep: 5721.59 | bwd_inner_microstep: 5708.61 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.37
[2025-04-26 05:46:07,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.36 | bwd: 5721.60 | bwd_inner: 5708.61 | bwd_allreduce: 12.95 | step: 19.37

  2%|▏         | 671/41250 [1:37:38<97:36:11,  8.66s/it]
                                                        
{'loss': 0.4355, 'grad_norm': 7.070096015930176, 'learning_rate': 2.1680129240710824e-05, 'epoch': 0.16}

  2%|▏         | 671/41250 [1:37:38<97:36:11,  8.66s/it][2025-04-26 05:46:15,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:46:15,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.43 | bwd_microstep: 5700.18 | bwd_inner_microstep: 5670.56 | bwd_allreduce_microstep: 29.57 | step_microstep: 18.88
[2025-04-26 05:46:15,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.43 | bwd: 5700.20 | bwd_inner: 5670.56 | bwd_allreduce: 29.60 | step: 18.88

  2%|▏         | 672/41250 [1:37:47<97:27:28,  8.65s/it]
                                                        
{'loss': 0.3391, 'grad_norm': 2.6447765827178955, 'learning_rate': 2.1712439418416806e-05, 'epoch': 0.16}

  2%|▏         | 672/41250 [1:37:47<97:27:28,  8.65s/it][2025-04-26 05:46:24,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:46:24,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.74 | bwd_microstep: 5720.22 | bwd_inner_microstep: 5707.53 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.74
[2025-04-26 05:46:24,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.74 | bwd: 5720.24 | bwd_inner: 5707.53 | bwd_allreduce: 12.67 | step: 18.74

  2%|▏         | 673/41250 [1:37:55<97:28:15,  8.65s/it]
                                                        
{'loss': 0.2423, 'grad_norm': 2.3180623054504395, 'learning_rate': 2.1744749596122777e-05, 'epoch': 0.16}

  2%|▏         | 673/41250 [1:37:55<97:28:15,  8.65s/it][2025-04-26 05:46:33,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:46:33,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.53 | bwd_microstep: 5755.86 | bwd_inner_microstep: 5716.21 | bwd_allreduce_microstep: 39.61 | step_microstep: 19.01
[2025-04-26 05:46:33,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.53 | bwd: 5755.88 | bwd_inner: 5716.21 | bwd_allreduce: 39.63 | step: 19.01

  2%|▏         | 674/41250 [1:38:04<97:37:40,  8.66s/it]
                                                        
{'loss': 0.2981, 'grad_norm': 4.060444355010986, 'learning_rate': 2.1777059773828758e-05, 'epoch': 0.16}

  2%|▏         | 674/41250 [1:38:04<97:37:40,  8.66s/it][2025-04-26 05:46:41,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:46:41,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.72 | bwd_microstep: 5712.04 | bwd_inner_microstep: 5659.24 | bwd_allreduce_microstep: 52.75 | step_microstep: 18.77
[2025-04-26 05:46:41,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.72 | bwd: 5712.05 | bwd_inner: 5659.24 | bwd_allreduce: 52.77 | step: 18.78

  2%|▏         | 675/41250 [1:38:13<97:29:25,  8.65s/it]
                                                        
{'loss': 0.1518, 'grad_norm': 3.069861650466919, 'learning_rate': 2.1809369951534736e-05, 'epoch': 0.16}

  2%|▏         | 675/41250 [1:38:13<97:29:25,  8.65s/it][2025-04-26 05:46:50,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 05:46:50,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5784.07 | bwd_inner_microstep: 5693.47 | bwd_allreduce_microstep: 90.55 | step_microstep: 19.24
[2025-04-26 05:46:50,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5784.08 | bwd_inner: 5693.47 | bwd_allreduce: 90.57 | step: 19.23

  2%|▏         | 676/41250 [1:38:21<97:42:14,  8.67s/it]
                                                        
{'loss': 0.3734, 'grad_norm': 2.4189987182617188, 'learning_rate': 2.1841680129240714e-05, 'epoch': 0.16}

  2%|▏         | 676/41250 [1:38:21<97:42:14,  8.67s/it][2025-04-26 05:46:59,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 05:46:59,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 5777.25 | bwd_inner_microstep: 5666.57 | bwd_allreduce_microstep: 110.63 | step_microstep: 18.94
[2025-04-26 05:46:59,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 5777.26 | bwd_inner: 5666.57 | bwd_allreduce: 110.65 | step: 18.95

  2%|▏         | 677/41250 [1:38:30<97:43:58,  8.67s/it]
                                                        
{'loss': 0.1387, 'grad_norm': 1.8071092367172241, 'learning_rate': 2.187399030694669e-05, 'epoch': 0.16}

  2%|▏         | 677/41250 [1:38:30<97:43:58,  8.67s/it][2025-04-26 05:47:07,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:47:07,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5775.94 | bwd_inner_microstep: 5703.51 | bwd_allreduce_microstep: 72.38 | step_microstep: 18.86
[2025-04-26 05:47:07,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5775.96 | bwd_inner: 5703.51 | bwd_allreduce: 72.40 | step: 18.86

  2%|▏         | 678/41250 [1:38:39<97:51:37,  8.68s/it]
                                                        
{'loss': 0.156, 'grad_norm': 3.0539040565490723, 'learning_rate': 2.1906300484652666e-05, 'epoch': 0.16}

  2%|▏         | 678/41250 [1:38:39<97:51:37,  8.68s/it][2025-04-26 05:47:16,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 05:47:16,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.37 | bwd_microstep: 5709.08 | bwd_inner_microstep: 5653.54 | bwd_allreduce_microstep: 55.49 | step_microstep: 18.58
[2025-04-26 05:47:16,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.37 | bwd: 5709.09 | bwd_inner: 5653.54 | bwd_allreduce: 55.51 | step: 18.58

  2%|▏         | 679/41250 [1:38:47<97:38:35,  8.66s/it]
                                                        
{'loss': 0.1305, 'grad_norm': 2.3395426273345947, 'learning_rate': 2.1938610662358648e-05, 'epoch': 0.16}

  2%|▏         | 679/41250 [1:38:47<97:38:35,  8.66s/it][2025-04-26 05:47:25,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 05:47:25,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5790.99 | bwd_inner_microstep: 5657.55 | bwd_allreduce_microstep: 133.38 | step_microstep: 18.50
[2025-04-26 05:47:25,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5791.01 | bwd_inner: 5657.55 | bwd_allreduce: 133.40 | step: 18.50

  2%|▏         | 680/41250 [1:38:56<97:46:55,  8.68s/it]
                                                        
{'loss': 0.1227, 'grad_norm': 2.1238393783569336, 'learning_rate': 2.197092084006462e-05, 'epoch': 0.16}

  2%|▏         | 680/41250 [1:38:56<97:46:55,  8.68s/it][2025-04-26 05:47:33,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 05:47:33,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.83 | bwd_microstep: 5749.23 | bwd_inner_microstep: 5693.10 | bwd_allreduce_microstep: 56.09 | step_microstep: 18.99
[2025-04-26 05:47:33,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.83 | bwd: 5749.25 | bwd_inner: 5693.10 | bwd_allreduce: 56.11 | step: 19.00

  2%|▏         | 681/41250 [1:39:05<97:47:39,  8.68s/it]
                                                        
{'loss': 0.2933, 'grad_norm': 2.845865249633789, 'learning_rate': 2.20032310177706e-05, 'epoch': 0.17}

  2%|▏         | 681/41250 [1:39:05<97:47:39,  8.68s/it][2025-04-26 05:47:42,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-26 05:47:42,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.79 | bwd_microstep: 5706.73 | bwd_inner_microstep: 5661.77 | bwd_allreduce_microstep: 44.91 | step_microstep: 18.64
[2025-04-26 05:47:42,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.79 | bwd: 5706.74 | bwd_inner: 5661.77 | bwd_allreduce: 44.93 | step: 18.65

  2%|▏         | 682/41250 [1:39:13<97:33:50,  8.66s/it]
                                                        
{'loss': 0.4154, 'grad_norm': 3.549983024597168, 'learning_rate': 2.2035541195476578e-05, 'epoch': 0.17}

  2%|▏         | 682/41250 [1:39:13<97:33:50,  8.66s/it][2025-04-26 05:47:51,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:47:51,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.20 | bwd_microstep: 5775.38 | bwd_inner_microstep: 5762.64 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.63
[2025-04-26 05:47:51,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.20 | bwd: 5775.39 | bwd_inner: 5762.64 | bwd_allreduce: 12.71 | step: 18.64

  2%|▏         | 683/41250 [1:39:22<97:49:55,  8.68s/it]
                                                        
{'loss': 0.1777, 'grad_norm': 1.9705979824066162, 'learning_rate': 2.2067851373182556e-05, 'epoch': 0.17}

  2%|▏         | 683/41250 [1:39:22<97:49:55,  8.68s/it][2025-04-26 05:48:00,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:48:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.50 | bwd_microstep: 5770.76 | bwd_inner_microstep: 5681.02 | bwd_allreduce_microstep: 89.70 | step_microstep: 18.80
[2025-04-26 05:48:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.50 | bwd: 5770.77 | bwd_inner: 5681.02 | bwd_allreduce: 89.71 | step: 18.81

  2%|▏         | 684/41250 [1:39:31<97:53:30,  8.69s/it]
                                                        
{'loss': 0.2778, 'grad_norm': 2.2602381706237793, 'learning_rate': 2.210016155088853e-05, 'epoch': 0.17}

  2%|▏         | 684/41250 [1:39:31<97:53:30,  8.69s/it][2025-04-26 05:48:08,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:48:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.35 | bwd_microstep: 5752.06 | bwd_inner_microstep: 5690.02 | bwd_allreduce_microstep: 61.99 | step_microstep: 18.74
[2025-04-26 05:48:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.35 | bwd: 5752.07 | bwd_inner: 5690.02 | bwd_allreduce: 62.01 | step: 18.74

  2%|▏         | 685/41250 [1:39:39<97:51:31,  8.68s/it]
                                                        
{'loss': 0.3772, 'grad_norm': 3.0427346229553223, 'learning_rate': 2.2132471728594508e-05, 'epoch': 0.17}

  2%|▏         | 685/41250 [1:39:39<97:51:31,  8.68s/it][2025-04-26 05:48:17,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 05:48:17,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.87 | bwd_microstep: 5706.57 | bwd_inner_microstep: 5693.89 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.08
[2025-04-26 05:48:17,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.87 | bwd: 5706.59 | bwd_inner: 5693.89 | bwd_allreduce: 12.66 | step: 19.08

  2%|▏         | 686/41250 [1:39:48<97:39:08,  8.67s/it]
                                                        
{'loss': 0.089, 'grad_norm': 1.3023438453674316, 'learning_rate': 2.2164781906300486e-05, 'epoch': 0.17}

  2%|▏         | 686/41250 [1:39:48<97:39:08,  8.67s/it][2025-04-26 05:48:25,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:48:25,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.39 | bwd_microstep: 5713.22 | bwd_inner_microstep: 5700.44 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.74
[2025-04-26 05:48:25,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.39 | bwd: 5713.23 | bwd_inner: 5700.44 | bwd_allreduce: 12.76 | step: 18.75

  2%|▏         | 687/41250 [1:39:57<97:33:20,  8.66s/it]
                                                        
{'loss': 0.2075, 'grad_norm': 2.4397828578948975, 'learning_rate': 2.219709208400646e-05, 'epoch': 0.17}

  2%|▏         | 687/41250 [1:39:57<97:33:20,  8.66s/it][2025-04-26 05:48:34,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:48:34,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 5706.58 | bwd_inner_microstep: 5647.30 | bwd_allreduce_microstep: 59.24 | step_microstep: 18.28
[2025-04-26 05:48:34,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 5706.60 | bwd_inner: 5647.30 | bwd_allreduce: 59.26 | step: 18.29

  2%|▏         | 688/41250 [1:40:05<97:24:28,  8.65s/it]
                                                        
{'loss': 0.2318, 'grad_norm': 2.1816537380218506, 'learning_rate': 2.2229402261712442e-05, 'epoch': 0.17}

  2%|▏         | 688/41250 [1:40:05<97:24:28,  8.65s/it][2025-04-26 05:48:43,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.02
[2025-04-26 05:48:43,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.80 | bwd_microstep: 5778.15 | bwd_inner_microstep: 5683.74 | bwd_allreduce_microstep: 94.36 | step_microstep: 18.32
[2025-04-26 05:48:43,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.80 | bwd: 5778.16 | bwd_inner: 5683.74 | bwd_allreduce: 94.38 | step: 18.32

  2%|▏         | 689/41250 [1:40:14<97:35:34,  8.66s/it]
                                                        
{'loss': 0.2053, 'grad_norm': 1.908336877822876, 'learning_rate': 2.226171243941842e-05, 'epoch': 0.17}

  2%|▏         | 689/41250 [1:40:14<97:35:34,  8.66s/it][2025-04-26 05:48:51,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:48:51,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.92 | bwd_microstep: 5699.40 | bwd_inner_microstep: 5686.79 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.40
[2025-04-26 05:48:51,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.92 | bwd: 5699.41 | bwd_inner: 5686.79 | bwd_allreduce: 12.59 | step: 18.40

  2%|▏         | 690/41250 [1:40:23<97:29:37,  8.65s/it]
                                                        
{'loss': 0.3109, 'grad_norm': 3.2173948287963867, 'learning_rate': 2.2294022617124397e-05, 'epoch': 0.17}

  2%|▏         | 690/41250 [1:40:23<97:29:37,  8.65s/it][2025-04-26 05:49:00,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.94 | optimizer_step: 0.98
[2025-04-26 05:49:00,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.42 | bwd_microstep: 5713.39 | bwd_inner_microstep: 5641.85 | bwd_allreduce_microstep: 71.49 | step_microstep: 18.31
[2025-04-26 05:49:00,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.42 | bwd: 5713.40 | bwd_inner: 5641.85 | bwd_allreduce: 71.51 | step: 18.32

  2%|▏         | 691/41250 [1:40:31<97:20:17,  8.64s/it]
                                                        
{'loss': 0.1762, 'grad_norm': 1.8702285289764404, 'learning_rate': 2.2326332794830372e-05, 'epoch': 0.17}

  2%|▏         | 691/41250 [1:40:31<97:20:17,  8.64s/it][2025-04-26 05:49:09,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:49:09,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.73 | bwd_microstep: 5758.52 | bwd_inner_microstep: 5640.80 | bwd_allreduce_microstep: 117.68 | step_microstep: 18.16
[2025-04-26 05:49:09,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.73 | bwd: 5758.53 | bwd_inner: 5640.80 | bwd_allreduce: 117.70 | step: 18.17

  2%|▏         | 692/41250 [1:40:40<97:24:29,  8.65s/it]
                                                        
{'loss': 0.2466, 'grad_norm': 1.8006936311721802, 'learning_rate': 2.235864297253635e-05, 'epoch': 0.17}

  2%|▏         | 692/41250 [1:40:40<97:24:29,  8.65s/it][2025-04-26 05:49:17,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:49:17,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.47 | bwd_microstep: 5734.81 | bwd_inner_microstep: 5680.61 | bwd_allreduce_microstep: 54.16 | step_microstep: 18.49
[2025-04-26 05:49:17,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.48 | bwd: 5734.82 | bwd_inner: 5680.61 | bwd_allreduce: 54.17 | step: 18.49

  2%|▏         | 693/41250 [1:40:49<97:27:01,  8.65s/it]
                                                        
{'loss': 0.2115, 'grad_norm': 2.349703788757324, 'learning_rate': 2.2390953150242328e-05, 'epoch': 0.17}

  2%|▏         | 693/41250 [1:40:49<97:27:01,  8.65s/it][2025-04-26 05:49:26,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:49:26,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5744.06 | bwd_inner_microstep: 5696.54 | bwd_allreduce_microstep: 47.46 | step_microstep: 18.51
[2025-04-26 05:49:26,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5744.07 | bwd_inner: 5696.54 | bwd_allreduce: 47.48 | step: 18.52

  2%|▏         | 694/41250 [1:40:57<97:31:45,  8.66s/it]
                                                        
{'loss': 0.2271, 'grad_norm': 2.7594807147979736, 'learning_rate': 2.2423263327948302e-05, 'epoch': 0.17}

  2%|▏         | 694/41250 [1:40:57<97:31:45,  8.66s/it][2025-04-26 05:49:35,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 05:49:35,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.78 | bwd_microstep: 5696.18 | bwd_inner_microstep: 5648.14 | bwd_allreduce_microstep: 47.99 | step_microstep: 19.06
[2025-04-26 05:49:35,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.78 | bwd: 5696.20 | bwd_inner: 5648.14 | bwd_allreduce: 48.02 | step: 19.06

  2%|▏         | 695/41250 [1:41:06<97:18:42,  8.64s/it]
                                                        
{'loss': 0.2135, 'grad_norm': 2.423459529876709, 'learning_rate': 2.2455573505654284e-05, 'epoch': 0.17}

  2%|▏         | 695/41250 [1:41:06<97:18:42,  8.64s/it][2025-04-26 05:49:43,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:49:43,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.15 | bwd_microstep: 5702.15 | bwd_inner_microstep: 5689.29 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.57
[2025-04-26 05:49:43,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.15 | bwd: 5702.16 | bwd_inner: 5689.29 | bwd_allreduce: 12.83 | step: 18.57

  2%|▏         | 696/41250 [1:41:14<97:18:18,  8.64s/it]
                                                        
{'loss': 0.1274, 'grad_norm': 1.8481322526931763, 'learning_rate': 2.248788368336026e-05, 'epoch': 0.17}

  2%|▏         | 696/41250 [1:41:14<97:18:18,  8.64s/it][2025-04-26 05:49:52,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 05:49:52,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.58 | bwd_microstep: 5720.47 | bwd_inner_microstep: 5651.52 | bwd_allreduce_microstep: 68.90 | step_microstep: 18.76
[2025-04-26 05:49:52,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.58 | bwd: 5720.49 | bwd_inner: 5651.52 | bwd_allreduce: 68.92 | step: 18.76

  2%|▏         | 697/41250 [1:41:23<97:22:13,  8.64s/it]
                                                        
{'loss': 0.2325, 'grad_norm': 2.406754493713379, 'learning_rate': 2.252019386106624e-05, 'epoch': 0.17}

  2%|▏         | 697/41250 [1:41:23<97:22:13,  8.64s/it][2025-04-26 05:50:01,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:50:01,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.60 | bwd_microstep: 5693.69 | bwd_inner_microstep: 5634.92 | bwd_allreduce_microstep: 58.72 | step_microstep: 18.64
[2025-04-26 05:50:01,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.60 | bwd: 5693.70 | bwd_inner: 5634.92 | bwd_allreduce: 58.74 | step: 18.64

  2%|▏         | 698/41250 [1:41:32<97:12:18,  8.63s/it]
                                                        
{'loss': 0.421, 'grad_norm': 2.4356653690338135, 'learning_rate': 2.2552504038772214e-05, 'epoch': 0.17}

  2%|▏         | 698/41250 [1:41:32<97:12:18,  8.63s/it][2025-04-26 05:50:09,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 05:50:09,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.09 | bwd_microstep: 5714.50 | bwd_inner_microstep: 5691.08 | bwd_allreduce_microstep: 23.37 | step_microstep: 18.56
[2025-04-26 05:50:09,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.09 | bwd: 5714.51 | bwd_inner: 5691.08 | bwd_allreduce: 23.39 | step: 18.56

  2%|▏         | 699/41250 [1:41:40<97:18:07,  8.64s/it]
                                                        
{'loss': 0.145, 'grad_norm': 1.4858301877975464, 'learning_rate': 2.258481421647819e-05, 'epoch': 0.17}

  2%|▏         | 699/41250 [1:41:40<97:18:07,  8.64s/it][2025-04-26 05:50:18,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:50:18,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5750.78 | bwd_inner_microstep: 5644.24 | bwd_allreduce_microstep: 106.50 | step_microstep: 18.44
[2025-04-26 05:50:18,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.25 | bwd: 5750.79 | bwd_inner: 5644.24 | bwd_allreduce: 106.52 | step: 18.44

  2%|▏         | 700/41250 [1:41:49<97:22:09,  8.64s/it]
                                                        
{'loss': 0.1631, 'grad_norm': 1.6923125982284546, 'learning_rate': 2.261712439418417e-05, 'epoch': 0.17}

  2%|▏         | 700/41250 [1:41:49<97:22:09,  8.64s/it][2025-04-26 05:50:26,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:50:26,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.46 | bwd_microstep: 5660.87 | bwd_inner_microstep: 5637.22 | bwd_allreduce_microstep: 23.60 | step_microstep: 18.50
[2025-04-26 05:50:26,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.46 | bwd: 5660.88 | bwd_inner: 5637.22 | bwd_allreduce: 23.62 | step: 18.50

  2%|▏         | 701/41250 [1:41:58<97:07:44,  8.62s/it]
                                                        
{'loss': 0.2066, 'grad_norm': 1.5037212371826172, 'learning_rate': 2.2649434571890144e-05, 'epoch': 0.17}

  2%|▏         | 701/41250 [1:41:58<97:07:44,  8.62s/it][2025-04-26 05:50:35,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 05:50:35,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.02 | bwd_microstep: 5708.64 | bwd_inner_microstep: 5695.94 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.61
[2025-04-26 05:50:35,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.02 | bwd: 5708.65 | bwd_inner: 5695.94 | bwd_allreduce: 12.67 | step: 18.61

  2%|▏         | 702/41250 [1:42:06<97:11:56,  8.63s/it]
                                                        
{'loss': 0.1701, 'grad_norm': 1.3698161840438843, 'learning_rate': 2.2681744749596125e-05, 'epoch': 0.17}

  2%|▏         | 702/41250 [1:42:06<97:11:56,  8.63s/it][2025-04-26 05:50:44,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:50:44,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5701.24 | bwd_inner_microstep: 5688.40 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.60
[2025-04-26 05:50:44,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5701.25 | bwd_inner: 5688.40 | bwd_allreduce: 12.81 | step: 18.60

  2%|▏         | 703/41250 [1:42:15<97:14:11,  8.63s/it]
                                                        
{'loss': 0.2263, 'grad_norm': 1.9205492734909058, 'learning_rate': 2.2714054927302103e-05, 'epoch': 0.17}

  2%|▏         | 703/41250 [1:42:15<97:14:11,  8.63s/it][2025-04-26 05:50:52,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 05:50:52,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.59 | bwd_microstep: 5734.08 | bwd_inner_microstep: 5684.69 | bwd_allreduce_microstep: 49.34 | step_microstep: 18.53
[2025-04-26 05:50:52,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.59 | bwd: 5734.09 | bwd_inner: 5684.69 | bwd_allreduce: 49.36 | step: 18.54

  2%|▏         | 704/41250 [1:42:24<97:19:01,  8.64s/it]
                                                        
{'loss': 0.2196, 'grad_norm': 2.996659755706787, 'learning_rate': 2.274636510500808e-05, 'epoch': 0.17}

  2%|▏         | 704/41250 [1:42:24<97:19:01,  8.64s/it][2025-04-26 05:51:01,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-26 05:51:01,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5864.17 | bwd_inner_microstep: 5643.49 | bwd_allreduce_microstep: 220.63 | step_microstep: 18.74
[2025-04-26 05:51:01,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5864.18 | bwd_inner: 5643.49 | bwd_allreduce: 220.65 | step: 18.75

  2%|▏         | 705/41250 [1:42:32<97:49:14,  8.69s/it]
                                                        
{'loss': 0.1986, 'grad_norm': 1.2960472106933594, 'learning_rate': 2.2778675282714056e-05, 'epoch': 0.17}

  2%|▏         | 705/41250 [1:42:32<97:49:14,  8.69s/it][2025-04-26 05:51:10,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 05:51:10,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.60 | bwd_microstep: 5769.92 | bwd_inner_microstep: 5639.86 | bwd_allreduce_microstep: 130.02 | step_microstep: 18.84
[2025-04-26 05:51:10,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.60 | bwd: 5769.94 | bwd_inner: 5639.86 | bwd_allreduce: 130.03 | step: 18.84

  2%|▏         | 706/41250 [1:42:41<97:48:01,  8.68s/it]
                                                        
{'loss': 0.3037, 'grad_norm': 1.9366304874420166, 'learning_rate': 2.2810985460420033e-05, 'epoch': 0.17}

  2%|▏         | 706/41250 [1:42:41<97:48:01,  8.68s/it][2025-04-26 05:51:18,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 05:51:18,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5733.12 | bwd_inner_microstep: 5683.61 | bwd_allreduce_microstep: 49.47 | step_microstep: 18.74
[2025-04-26 05:51:18,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5733.14 | bwd_inner: 5683.61 | bwd_allreduce: 49.48 | step: 18.75

  2%|▏         | 707/41250 [1:42:50<97:46:14,  8.68s/it]
                                                        
{'loss': 0.1345, 'grad_norm': 1.8141593933105469, 'learning_rate': 2.284329563812601e-05, 'epoch': 0.17}

  2%|▏         | 707/41250 [1:42:50<97:46:14,  8.68s/it][2025-04-26 05:51:27,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 05:51:27,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.13 | bwd_microstep: 5733.34 | bwd_inner_microstep: 5719.76 | bwd_allreduce_microstep: 13.52 | step_microstep: 18.75
[2025-04-26 05:51:27,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.13 | bwd: 5733.35 | bwd_inner: 5719.76 | bwd_allreduce: 13.55 | step: 18.76

  2%|▏         | 708/41250 [1:42:58<97:45:45,  8.68s/it]
                                                        
{'loss': 0.2196, 'grad_norm': 2.8208730220794678, 'learning_rate': 2.2875605815831986e-05, 'epoch': 0.17}

  2%|▏         | 708/41250 [1:42:58<97:45:45,  8.68s/it][2025-04-26 05:51:36,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:51:36,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.31 | bwd_microstep: 5775.01 | bwd_inner_microstep: 5687.36 | bwd_allreduce_microstep: 87.60 | step_microstep: 18.41
[2025-04-26 05:51:36,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.31 | bwd: 5775.02 | bwd_inner: 5687.36 | bwd_allreduce: 87.62 | step: 18.41

  2%|▏         | 709/41250 [1:43:07<97:51:47,  8.69s/it]
                                                        
{'loss': 0.1886, 'grad_norm': 2.0754899978637695, 'learning_rate': 2.2907915993537967e-05, 'epoch': 0.17}

  2%|▏         | 709/41250 [1:43:07<97:51:47,  8.69s/it][2025-04-26 05:51:45,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:51:45,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.55 | bwd_microstep: 5722.42 | bwd_inner_microstep: 5709.71 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.15
[2025-04-26 05:51:45,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.55 | bwd: 5722.43 | bwd_inner: 5709.71 | bwd_allreduce: 12.69 | step: 18.15

  2%|▏         | 710/41250 [1:43:16<97:47:19,  8.68s/it]
                                                        
{'loss': 0.1752, 'grad_norm': 1.603049397468567, 'learning_rate': 2.2940226171243945e-05, 'epoch': 0.17}

  2%|▏         | 710/41250 [1:43:16<97:47:19,  8.68s/it][2025-04-26 05:51:53,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:51:53,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.11 | bwd_microstep: 5767.44 | bwd_inner_microstep: 5654.63 | bwd_allreduce_microstep: 112.76 | step_microstep: 18.48
[2025-04-26 05:51:53,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.11 | bwd: 5767.45 | bwd_inner: 5654.63 | bwd_allreduce: 112.78 | step: 18.48

  2%|▏         | 711/41250 [1:43:24<97:48:58,  8.69s/it]
                                                        
{'loss': 0.2764, 'grad_norm': 2.547006607055664, 'learning_rate': 2.2972536348949923e-05, 'epoch': 0.17}

  2%|▏         | 711/41250 [1:43:24<97:48:58,  8.69s/it][2025-04-26 05:52:02,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-26 05:52:02,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.49 | bwd_microstep: 5687.51 | bwd_inner_microstep: 5645.44 | bwd_allreduce_microstep: 42.02 | step_microstep: 18.17
[2025-04-26 05:52:02,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.49 | bwd: 5687.53 | bwd_inner: 5645.44 | bwd_allreduce: 42.04 | step: 18.18

  2%|▏         | 712/41250 [1:43:33<97:31:54,  8.66s/it]
                                                        
{'loss': 0.4615, 'grad_norm': 2.8932993412017822, 'learning_rate': 2.3004846526655897e-05, 'epoch': 0.17}

  2%|▏         | 712/41250 [1:43:33<97:31:54,  8.66s/it][2025-04-26 05:52:11,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:52:11,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.08 | bwd_microstep: 5767.90 | bwd_inner_microstep: 5652.32 | bwd_allreduce_microstep: 115.52 | step_microstep: 18.19
[2025-04-26 05:52:11,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.08 | bwd: 5767.91 | bwd_inner: 5652.32 | bwd_allreduce: 115.54 | step: 18.19

  2%|▏         | 713/41250 [1:43:42<97:36:29,  8.67s/it]
                                                        
{'loss': 0.0876, 'grad_norm': 0.9128193855285645, 'learning_rate': 2.3037156704361875e-05, 'epoch': 0.17}

  2%|▏         | 713/41250 [1:43:42<97:36:29,  8.67s/it][2025-04-26 05:52:19,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 05:52:19,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.34 | bwd_microstep: 5689.91 | bwd_inner_microstep: 5654.92 | bwd_allreduce_microstep: 34.94 | step_microstep: 18.48
[2025-04-26 05:52:19,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.34 | bwd: 5689.92 | bwd_inner: 5654.92 | bwd_allreduce: 34.96 | step: 18.48

  2%|▏         | 714/41250 [1:43:50<97:24:49,  8.65s/it]
                                                        
{'loss': 0.3132, 'grad_norm': 2.389589309692383, 'learning_rate': 2.3069466882067853e-05, 'epoch': 0.17}

  2%|▏         | 714/41250 [1:43:50<97:24:49,  8.65s/it][2025-04-26 05:52:28,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 05:52:28,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5719.19 | bwd_inner_microstep: 5649.32 | bwd_allreduce_microstep: 69.82 | step_microstep: 18.79
[2025-04-26 05:52:28,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5719.20 | bwd_inner: 5649.32 | bwd_allreduce: 69.84 | step: 18.80

  2%|▏         | 715/41250 [1:43:59<97:21:20,  8.65s/it]
                                                        
{'loss': 0.356, 'grad_norm': 3.47737193107605, 'learning_rate': 2.3101777059773828e-05, 'epoch': 0.17}

  2%|▏         | 715/41250 [1:43:59<97:21:20,  8.65s/it][2025-04-26 05:52:36,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 05:52:36,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.96 | bwd_microstep: 5758.95 | bwd_inner_microstep: 5704.43 | bwd_allreduce_microstep: 54.48 | step_microstep: 18.91
[2025-04-26 05:52:36,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.96 | bwd: 5758.96 | bwd_inner: 5704.43 | bwd_allreduce: 54.49 | step: 18.91

  2%|▏         | 716/41250 [1:44:08<97:33:34,  8.66s/it]
                                                        
{'loss': 0.2573, 'grad_norm': 3.0677850246429443, 'learning_rate': 2.313408723747981e-05, 'epoch': 0.17}

  2%|▏         | 716/41250 [1:44:08<97:33:34,  8.66s/it][2025-04-26 05:52:45,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:52:45,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.75 | bwd_microstep: 5715.83 | bwd_inner_microstep: 5654.36 | bwd_allreduce_microstep: 61.42 | step_microstep: 18.81
[2025-04-26 05:52:45,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.76 | bwd: 5715.85 | bwd_inner: 5654.36 | bwd_allreduce: 61.44 | step: 18.81

  2%|▏         | 717/41250 [1:44:16<97:27:17,  8.66s/it]
                                                        
{'loss': 0.2443, 'grad_norm': 4.463287830352783, 'learning_rate': 2.3166397415185787e-05, 'epoch': 0.17}

  2%|▏         | 717/41250 [1:44:16<97:27:17,  8.66s/it][2025-04-26 05:52:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:52:54,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.33 | bwd_microstep: 5760.66 | bwd_inner_microstep: 5698.41 | bwd_allreduce_microstep: 62.21 | step_microstep: 18.41
[2025-04-26 05:52:54,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.33 | bwd: 5760.68 | bwd_inner: 5698.41 | bwd_allreduce: 62.23 | step: 18.42

  2%|▏         | 718/41250 [1:44:25<97:34:44,  8.67s/it]
                                                        
{'loss': 0.1522, 'grad_norm': 2.477208137512207, 'learning_rate': 2.3198707592891765e-05, 'epoch': 0.17}

  2%|▏         | 718/41250 [1:44:25<97:34:44,  8.67s/it][2025-04-26 05:53:03,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:53:03,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2904.64 | bwd_microstep: 5834.02 | bwd_inner_microstep: 5781.12 | bwd_allreduce_microstep: 52.86 | step_microstep: 18.42
[2025-04-26 05:53:03,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2904.64 | bwd: 5834.04 | bwd_inner: 5781.12 | bwd_allreduce: 52.88 | step: 18.43

  2%|▏         | 719/41250 [1:44:34<98:06:18,  8.71s/it]
                                                        
{'loss': 0.203, 'grad_norm': 2.6109416484832764, 'learning_rate': 2.323101777059774e-05, 'epoch': 0.17}

  2%|▏         | 719/41250 [1:44:34<98:06:18,  8.71s/it][2025-04-26 05:53:11,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:53:11,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.95 | bwd_microstep: 5700.17 | bwd_inner_microstep: 5655.63 | bwd_allreduce_microstep: 44.50 | step_microstep: 18.35
[2025-04-26 05:53:11,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.95 | bwd: 5700.19 | bwd_inner: 5655.63 | bwd_allreduce: 44.52 | step: 18.36

  2%|▏         | 720/41250 [1:44:42<97:47:44,  8.69s/it]
                                                        
{'loss': 0.177, 'grad_norm': 2.0281872749328613, 'learning_rate': 2.3263327948303717e-05, 'epoch': 0.17}

  2%|▏         | 720/41250 [1:44:42<97:47:44,  8.69s/it][2025-04-26 05:53:20,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:53:20,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.43 | bwd_microstep: 5751.18 | bwd_inner_microstep: 5709.09 | bwd_allreduce_microstep: 42.04 | step_microstep: 18.53
[2025-04-26 05:53:20,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.43 | bwd: 5751.19 | bwd_inner: 5709.09 | bwd_allreduce: 42.06 | step: 18.53

  2%|▏         | 721/41250 [1:44:51<97:50:03,  8.69s/it]
                                                        
{'loss': 0.2819, 'grad_norm': 2.2966561317443848, 'learning_rate': 2.3295638126009695e-05, 'epoch': 0.17}

  2%|▏         | 721/41250 [1:44:51<97:50:03,  8.69s/it][2025-04-26 05:53:29,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:53:29,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.87 | bwd_microstep: 5721.97 | bwd_inner_microstep: 5709.14 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.47
[2025-04-26 05:53:29,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.87 | bwd: 5721.99 | bwd_inner: 5709.14 | bwd_allreduce: 12.81 | step: 18.48

  2%|▏         | 722/41250 [1:45:00<97:41:18,  8.68s/it]
                                                        
{'loss': 0.1741, 'grad_norm': 1.4250394105911255, 'learning_rate': 2.332794830371567e-05, 'epoch': 0.18}

  2%|▏         | 722/41250 [1:45:00<97:41:18,  8.68s/it][2025-04-26 05:53:37,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 05:53:37,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.16 | bwd_microstep: 5892.21 | bwd_inner_microstep: 5659.21 | bwd_allreduce_microstep: 232.93 | step_microstep: 19.59
[2025-04-26 05:53:37,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.17 | bwd: 5892.22 | bwd_inner: 5659.21 | bwd_allreduce: 232.97 | step: 19.59

  2%|▏         | 723/41250 [1:45:09<98:10:47,  8.72s/it]
                                                        
{'loss': 0.1941, 'grad_norm': 1.6301032304763794, 'learning_rate': 2.336025848142165e-05, 'epoch': 0.18}

  2%|▏         | 723/41250 [1:45:09<98:10:47,  8.72s/it][2025-04-26 05:53:46,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 05:53:46,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.26 | bwd_microstep: 5707.43 | bwd_inner_microstep: 5661.14 | bwd_allreduce_microstep: 46.24 | step_microstep: 18.91
[2025-04-26 05:53:46,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.26 | bwd: 5707.44 | bwd_inner: 5661.14 | bwd_allreduce: 46.26 | step: 18.91

  2%|▏         | 724/41250 [1:45:17<97:54:42,  8.70s/it]
                                                        
{'loss': 0.3242, 'grad_norm': 2.6313581466674805, 'learning_rate': 2.339256865912763e-05, 'epoch': 0.18}

  2%|▏         | 724/41250 [1:45:17<97:54:42,  8.70s/it][2025-04-26 05:53:55,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 05:53:55,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.88 | bwd_microstep: 5802.72 | bwd_inner_microstep: 5789.71 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.06
[2025-04-26 05:53:55,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.88 | bwd: 5802.73 | bwd_inner: 5789.71 | bwd_allreduce: 12.98 | step: 19.08

  2%|▏         | 725/41250 [1:45:26<98:14:18,  8.73s/it]
                                                        
{'loss': 0.1996, 'grad_norm': 1.6634823083877563, 'learning_rate': 2.3424878836833606e-05, 'epoch': 0.18}

  2%|▏         | 725/41250 [1:45:26<98:14:18,  8.73s/it][2025-04-26 05:54:04,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 05:54:04,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.06 | bwd_microstep: 5752.18 | bwd_inner_microstep: 5718.94 | bwd_allreduce_microstep: 33.19 | step_microstep: 19.08
[2025-04-26 05:54:04,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.06 | bwd: 5752.19 | bwd_inner: 5718.94 | bwd_allreduce: 33.21 | step: 19.08

  2%|▏         | 726/41250 [1:45:35<98:09:35,  8.72s/it]
                                                        
{'loss': 0.0973, 'grad_norm': 1.8950181007385254, 'learning_rate': 2.345718901453958e-05, 'epoch': 0.18}

  2%|▏         | 726/41250 [1:45:35<98:09:35,  8.72s/it][2025-04-26 05:54:12,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.10
[2025-04-26 05:54:12,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.89 | bwd_microstep: 5756.01 | bwd_inner_microstep: 5652.44 | bwd_allreduce_microstep: 103.52 | step_microstep: 19.33
[2025-04-26 05:54:12,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.89 | bwd: 5756.02 | bwd_inner: 5652.44 | bwd_allreduce: 103.54 | step: 19.33

  2%|▏         | 727/41250 [1:45:43<98:03:46,  8.71s/it]
                                                        
{'loss': 0.1483, 'grad_norm': 1.2454185485839844, 'learning_rate': 2.348949919224556e-05, 'epoch': 0.18}

  2%|▏         | 727/41250 [1:45:43<98:03:46,  8.71s/it][2025-04-26 05:54:21,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.19 | optimizer_step: 1.10
[2025-04-26 05:54:21,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.74 | bwd_microstep: 5774.13 | bwd_inner_microstep: 5650.43 | bwd_allreduce_microstep: 123.62 | step_microstep: 20.79
[2025-04-26 05:54:21,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.74 | bwd: 5774.15 | bwd_inner: 5650.43 | bwd_allreduce: 123.66 | step: 20.79

  2%|▏         | 728/41250 [1:45:52<98:01:17,  8.71s/it]
                                                        
{'loss': 0.2105, 'grad_norm': 2.657576084136963, 'learning_rate': 2.3521809369951537e-05, 'epoch': 0.18}

  2%|▏         | 728/41250 [1:45:52<98:01:17,  8.71s/it][2025-04-26 05:54:30,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 05:54:30,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.14 | bwd_microstep: 5779.02 | bwd_inner_microstep: 5766.18 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.01
[2025-04-26 05:54:30,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.14 | bwd: 5779.03 | bwd_inner: 5766.18 | bwd_allreduce: 12.81 | step: 19.01

  2%|▏         | 729/41250 [1:46:01<98:10:26,  8.72s/it]
                                                        
{'loss': 0.2301, 'grad_norm': 2.2420461177825928, 'learning_rate': 2.355411954765751e-05, 'epoch': 0.18}

  2%|▏         | 729/41250 [1:46:01<98:10:26,  8.72s/it][2025-04-26 05:54:38,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 05:54:38,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.09 | bwd_microstep: 5700.49 | bwd_inner_microstep: 5687.74 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.68
[2025-04-26 05:54:38,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.09 | bwd: 5700.50 | bwd_inner: 5687.74 | bwd_allreduce: 12.72 | step: 18.68

  2%|▏         | 730/41250 [1:46:10<97:50:49,  8.69s/it]
                                                        
{'loss': 0.1127, 'grad_norm': 1.3690396547317505, 'learning_rate': 2.3586429725363493e-05, 'epoch': 0.18}

  2%|▏         | 730/41250 [1:46:10<97:50:49,  8.69s/it][2025-04-26 05:54:47,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:54:47,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.83 | bwd_microstep: 5704.64 | bwd_inner_microstep: 5659.31 | bwd_allreduce_microstep: 45.29 | step_microstep: 18.65
[2025-04-26 05:54:47,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.83 | bwd: 5704.66 | bwd_inner: 5659.31 | bwd_allreduce: 45.30 | step: 18.65

  2%|▏         | 731/41250 [1:46:18<97:32:39,  8.67s/it]
                                                        
{'loss': 0.2597, 'grad_norm': 2.2614095211029053, 'learning_rate': 2.361873990306947e-05, 'epoch': 0.18}

  2%|▏         | 731/41250 [1:46:18<97:32:39,  8.67s/it][2025-04-26 05:54:56,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.95
[2025-04-26 05:54:56,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.46 | bwd_microstep: 5745.65 | bwd_inner_microstep: 5704.62 | bwd_allreduce_microstep: 40.98 | step_microstep: 18.85
[2025-04-26 05:54:56,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.46 | bwd: 5745.66 | bwd_inner: 5704.62 | bwd_allreduce: 41.00 | step: 18.85

  2%|▏         | 732/41250 [1:46:27<97:37:03,  8.67s/it]
                                                        
{'loss': 0.1493, 'grad_norm': 1.6940865516662598, 'learning_rate': 2.3651050080775448e-05, 'epoch': 0.18}

  2%|▏         | 732/41250 [1:46:27<97:37:03,  8.67s/it][2025-04-26 05:55:04,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.40 | optimizer_step: 1.07
[2025-04-26 05:55:04,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.32 | bwd_microstep: 5712.84 | bwd_inner_microstep: 5646.75 | bwd_allreduce_microstep: 66.03 | step_microstep: 22.03
[2025-04-26 05:55:04,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.32 | bwd: 5712.86 | bwd_inner: 5646.75 | bwd_allreduce: 66.05 | step: 22.03

  2%|▏         | 733/41250 [1:46:35<97:27:37,  8.66s/it]
                                                        
{'loss': 0.164, 'grad_norm': 1.2833048105239868, 'learning_rate': 2.3683360258481423e-05, 'epoch': 0.18}

  2%|▏         | 733/41250 [1:46:35<97:27:37,  8.66s/it][2025-04-26 05:55:13,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:55:13,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.48 | bwd_microstep: 5784.79 | bwd_inner_microstep: 5682.57 | bwd_allreduce_microstep: 102.17 | step_microstep: 18.58
[2025-04-26 05:55:13,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.48 | bwd: 5784.80 | bwd_inner: 5682.57 | bwd_allreduce: 102.19 | step: 18.58

  2%|▏         | 734/41250 [1:46:44<97:37:44,  8.67s/it]
                                                        
{'loss': 0.1862, 'grad_norm': 1.7821651697158813, 'learning_rate': 2.37156704361874e-05, 'epoch': 0.18}

  2%|▏         | 734/41250 [1:46:44<97:37:44,  8.67s/it][2025-04-26 05:55:22,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.94
[2025-04-26 05:55:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.98 | bwd_microstep: 5734.68 | bwd_inner_microstep: 5694.49 | bwd_allreduce_microstep: 40.14 | step_microstep: 19.17
[2025-04-26 05:55:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.98 | bwd: 5734.69 | bwd_inner: 5694.49 | bwd_allreduce: 40.16 | step: 19.17

  2%|▏         | 735/41250 [1:46:53<97:37:11,  8.67s/it]
                                                        
{'loss': 0.1996, 'grad_norm': 1.6857117414474487, 'learning_rate': 2.374798061389338e-05, 'epoch': 0.18}

  2%|▏         | 735/41250 [1:46:53<97:37:11,  8.67s/it][2025-04-26 05:55:30,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 05:55:30,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.85 | bwd_microstep: 5742.00 | bwd_inner_microstep: 5698.58 | bwd_allreduce_microstep: 43.37 | step_microstep: 18.72
[2025-04-26 05:55:30,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.85 | bwd: 5742.01 | bwd_inner: 5698.58 | bwd_allreduce: 43.39 | step: 18.72

  2%|▏         | 736/41250 [1:47:02<97:36:45,  8.67s/it]
                                                        
{'loss': 0.2777, 'grad_norm': 2.158796787261963, 'learning_rate': 2.3780290791599353e-05, 'epoch': 0.18}

  2%|▏         | 736/41250 [1:47:02<97:36:45,  8.67s/it][2025-04-26 05:55:39,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 05:55:39,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5735.52 | bwd_inner_microstep: 5692.02 | bwd_allreduce_microstep: 43.45 | step_microstep: 18.84
[2025-04-26 05:55:39,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5735.53 | bwd_inner: 5692.02 | bwd_allreduce: 43.47 | step: 18.85

  2%|▏         | 737/41250 [1:47:10<97:35:33,  8.67s/it]
                                                        
{'loss': 0.2469, 'grad_norm': 2.2211313247680664, 'learning_rate': 2.3812600969305334e-05, 'epoch': 0.18}

  2%|▏         | 737/41250 [1:47:10<97:35:33,  8.67s/it][2025-04-26 05:55:48,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:55:48,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.25 | bwd_microstep: 5692.11 | bwd_inner_microstep: 5679.40 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.76
[2025-04-26 05:55:48,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.25 | bwd: 5692.13 | bwd_inner: 5679.40 | bwd_allreduce: 12.69 | step: 18.76

  2%|▏         | 738/41250 [1:47:19<97:24:49,  8.66s/it]
                                                        
{'loss': 0.1254, 'grad_norm': 1.6422228813171387, 'learning_rate': 2.3844911147011312e-05, 'epoch': 0.18}

  2%|▏         | 738/41250 [1:47:19<97:24:49,  8.66s/it][2025-04-26 05:55:56,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 05:55:56,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.61 | bwd_microstep: 5720.25 | bwd_inner_microstep: 5698.44 | bwd_allreduce_microstep: 21.77 | step_microstep: 19.16
[2025-04-26 05:55:56,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.61 | bwd: 5720.27 | bwd_inner: 5698.44 | bwd_allreduce: 21.79 | step: 19.16

  2%|▏         | 739/41250 [1:47:27<97:26:23,  8.66s/it]
                                                        
{'loss': 0.1366, 'grad_norm': 1.8352102041244507, 'learning_rate': 2.3877221324717287e-05, 'epoch': 0.18}

  2%|▏         | 739/41250 [1:47:27<97:26:23,  8.66s/it][2025-04-26 05:56:05,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 05:56:05,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.32 | bwd_microstep: 5758.92 | bwd_inner_microstep: 5641.51 | bwd_allreduce_microstep: 117.37 | step_microstep: 18.62
[2025-04-26 05:56:05,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.32 | bwd: 5758.94 | bwd_inner: 5641.51 | bwd_allreduce: 117.39 | step: 18.62

  2%|▏         | 740/41250 [1:47:36<97:24:16,  8.66s/it]
                                                        
{'loss': 0.4237, 'grad_norm': 3.3442423343658447, 'learning_rate': 2.3909531502423265e-05, 'epoch': 0.18}

  2%|▏         | 740/41250 [1:47:36<97:24:16,  8.66s/it][2025-04-26 05:56:14,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:56:14,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.72 | bwd_microstep: 5745.72 | bwd_inner_microstep: 5693.54 | bwd_allreduce_microstep: 52.13 | step_microstep: 18.94
[2025-04-26 05:56:14,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.72 | bwd: 5745.74 | bwd_inner: 5693.54 | bwd_allreduce: 52.15 | step: 18.94

  2%|▏         | 741/41250 [1:47:45<97:27:21,  8.66s/it]
                                                        
{'loss': 0.5236, 'grad_norm': 3.0795364379882812, 'learning_rate': 2.3941841680129242e-05, 'epoch': 0.18}

  2%|▏         | 741/41250 [1:47:45<97:27:21,  8.66s/it][2025-04-26 05:56:22,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:56:22,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.15 | bwd_microstep: 5686.47 | bwd_inner_microstep: 5656.66 | bwd_allreduce_microstep: 29.77 | step_microstep: 18.45
[2025-04-26 05:56:22,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.15 | bwd: 5686.48 | bwd_inner: 5656.66 | bwd_allreduce: 29.79 | step: 18.45

  2%|▏         | 742/41250 [1:47:53<97:13:37,  8.64s/it]
                                                        
{'loss': 0.2109, 'grad_norm': 2.6723408699035645, 'learning_rate': 2.397415185783522e-05, 'epoch': 0.18}

  2%|▏         | 742/41250 [1:47:53<97:13:37,  8.64s/it][2025-04-26 05:56:31,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:56:31,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.66 | bwd_microstep: 5873.85 | bwd_inner_microstep: 5686.64 | bwd_allreduce_microstep: 187.17 | step_microstep: 18.38
[2025-04-26 05:56:31,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.66 | bwd: 5873.86 | bwd_inner: 5686.64 | bwd_allreduce: 187.18 | step: 18.38

  2%|▏         | 743/41250 [1:48:02<97:44:54,  8.69s/it]
                                                        
{'loss': 0.2186, 'grad_norm': 1.8497313261032104, 'learning_rate': 2.4006462035541195e-05, 'epoch': 0.18}

  2%|▏         | 743/41250 [1:48:02<97:44:54,  8.69s/it][2025-04-26 05:56:40,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 05:56:40,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.57 | bwd_microstep: 5748.14 | bwd_inner_microstep: 5690.50 | bwd_allreduce_microstep: 57.60 | step_microstep: 18.39
[2025-04-26 05:56:40,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.57 | bwd: 5748.15 | bwd_inner: 5690.49 | bwd_allreduce: 57.62 | step: 18.40

  2%|▏         | 744/41250 [1:48:11<97:42:38,  8.68s/it]
                                                        
{'loss': 0.2235, 'grad_norm': 1.7222286462783813, 'learning_rate': 2.4038772213247176e-05, 'epoch': 0.18}

  2%|▏         | 744/41250 [1:48:11<97:42:38,  8.68s/it][2025-04-26 05:56:48,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:56:48,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.89 | bwd_microstep: 5758.75 | bwd_inner_microstep: 5645.08 | bwd_allreduce_microstep: 113.62 | step_microstep: 18.52
[2025-04-26 05:56:48,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.89 | bwd: 5758.76 | bwd_inner: 5645.08 | bwd_allreduce: 113.63 | step: 18.52

  2%|▏         | 745/41250 [1:48:19<97:36:51,  8.68s/it]
                                                        
{'loss': 0.1482, 'grad_norm': 1.677158236503601, 'learning_rate': 2.4071082390953154e-05, 'epoch': 0.18}

  2%|▏         | 745/41250 [1:48:20<97:36:51,  8.68s/it][2025-04-26 05:56:57,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 05:56:57,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.32 | bwd_microstep: 6023.75 | bwd_inner_microstep: 5645.21 | bwd_allreduce_microstep: 378.50 | step_microstep: 19.05
[2025-04-26 05:56:57,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.32 | bwd: 6023.77 | bwd_inner: 5645.21 | bwd_allreduce: 378.52 | step: 19.05

  2%|▏         | 746/41250 [1:48:28<98:28:22,  8.75s/it]
                                                        
{'loss': 0.2653, 'grad_norm': 2.028294563293457, 'learning_rate': 2.410339256865913e-05, 'epoch': 0.18}

  2%|▏         | 746/41250 [1:48:28<98:28:22,  8.75s/it][2025-04-26 05:57:06,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 05:57:06,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.95 | bwd_microstep: 5752.54 | bwd_inner_microstep: 5640.04 | bwd_allreduce_microstep: 112.45 | step_microstep: 18.57
[2025-04-26 05:57:06,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.96 | bwd: 5752.56 | bwd_inner: 5640.04 | bwd_allreduce: 112.47 | step: 18.57

  2%|▏         | 747/41250 [1:48:37<98:08:05,  8.72s/it]
                                                        
{'loss': 0.4057, 'grad_norm': 3.5133068561553955, 'learning_rate': 2.4135702746365106e-05, 'epoch': 0.18}

  2%|▏         | 747/41250 [1:48:37<98:08:05,  8.72s/it][2025-04-26 05:57:15,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 05:57:15,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.92 | bwd_microstep: 5749.27 | bwd_inner_microstep: 5650.33 | bwd_allreduce_microstep: 98.89 | step_microstep: 18.96
[2025-04-26 05:57:15,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.92 | bwd: 5749.29 | bwd_inner: 5650.33 | bwd_allreduce: 98.91 | step: 18.96

  2%|▏         | 748/41250 [1:48:46<97:54:24,  8.70s/it]
                                                        
{'loss': 0.1809, 'grad_norm': 1.5290496349334717, 'learning_rate': 2.4168012924071084e-05, 'epoch': 0.18}

  2%|▏         | 748/41250 [1:48:46<97:54:24,  8.70s/it][2025-04-26 05:57:23,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:57:23,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5746.67 | bwd_inner_microstep: 5710.64 | bwd_allreduce_microstep: 35.98 | step_microstep: 18.60
[2025-04-26 05:57:23,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.78 | bwd: 5746.69 | bwd_inner: 5710.64 | bwd_allreduce: 36.00 | step: 18.60

  2%|▏         | 749/41250 [1:48:54<97:49:14,  8.69s/it]
                                                        
{'loss': 0.2161, 'grad_norm': 1.577711582183838, 'learning_rate': 2.4200323101777062e-05, 'epoch': 0.18}

  2%|▏         | 749/41250 [1:48:54<97:49:14,  8.69s/it][2025-04-26 05:57:32,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:57:32,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.99 | bwd_microstep: 5907.41 | bwd_inner_microstep: 5680.94 | bwd_allreduce_microstep: 226.43 | step_microstep: 18.54
[2025-04-26 05:57:32,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.99 | bwd: 5907.42 | bwd_inner: 5680.94 | bwd_allreduce: 226.45 | step: 18.55

  2%|▏         | 750/41250 [1:49:03<98:17:00,  8.74s/it]
                                                        
{'loss': 0.1942, 'grad_norm': 1.78988778591156, 'learning_rate': 2.4232633279483037e-05, 'epoch': 0.18}

  2%|▏         | 750/41250 [1:49:03<98:17:00,  8.74s/it][2025-04-26 05:57:41,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:57:41,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.70 | bwd_microstep: 5721.54 | bwd_inner_microstep: 5700.44 | bwd_allreduce_microstep: 21.05 | step_microstep: 18.68
[2025-04-26 05:57:41,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.70 | bwd: 5721.55 | bwd_inner: 5700.44 | bwd_allreduce: 21.07 | step: 18.69

  2%|▏         | 751/41250 [1:49:12<98:00:48,  8.71s/it]
                                                        
{'loss': 0.2916, 'grad_norm': 2.796858787536621, 'learning_rate': 2.4264943457189018e-05, 'epoch': 0.18}

  2%|▏         | 751/41250 [1:49:12<98:00:48,  8.71s/it][2025-04-26 05:57:49,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:57:49,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.36 | bwd_microstep: 5714.95 | bwd_inner_microstep: 5701.97 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.46
[2025-04-26 05:57:49,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.36 | bwd: 5714.97 | bwd_inner: 5701.97 | bwd_allreduce: 12.96 | step: 18.46

  2%|▏         | 752/41250 [1:49:21<97:45:37,  8.69s/it]
                                                        
{'loss': 0.3198, 'grad_norm': 2.2006261348724365, 'learning_rate': 2.4297253634894996e-05, 'epoch': 0.18}

  2%|▏         | 752/41250 [1:49:21<97:45:37,  8.69s/it][2025-04-26 05:57:58,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 05:57:58,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.82 | bwd_microstep: 5787.59 | bwd_inner_microstep: 5774.93 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.55
[2025-04-26 05:57:58,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.82 | bwd: 5787.60 | bwd_inner: 5774.93 | bwd_allreduce: 12.64 | step: 18.56

  2%|▏         | 753/41250 [1:49:29<97:58:43,  8.71s/it]
                                                        
{'loss': 0.1422, 'grad_norm': 1.4242210388183594, 'learning_rate': 2.432956381260097e-05, 'epoch': 0.18}

  2%|▏         | 753/41250 [1:49:29<97:58:43,  8.71s/it][2025-04-26 05:58:07,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 05:58:07,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.24 | bwd_microstep: 5776.94 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 124.61 | step_microstep: 19.12
[2025-04-26 05:58:07,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.24 | bwd: 5776.96 | bwd_inner: 5652.29 | bwd_allreduce: 124.63 | step: 19.13

  2%|▏         | 754/41250 [1:49:38<97:52:34,  8.70s/it]
                                                        
{'loss': 0.187, 'grad_norm': 2.033909320831299, 'learning_rate': 2.4361873990306948e-05, 'epoch': 0.18}

  2%|▏         | 754/41250 [1:49:38<97:52:34,  8.70s/it][2025-04-26 05:58:15,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 05:58:15,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5754.03 | bwd_inner_microstep: 5705.73 | bwd_allreduce_microstep: 48.26 | step_microstep: 18.87
[2025-04-26 05:58:15,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5754.05 | bwd_inner: 5705.73 | bwd_allreduce: 48.27 | step: 18.87

  2%|▏         | 755/41250 [1:49:47<97:49:28,  8.70s/it]
                                                        
{'loss': 0.2625, 'grad_norm': 1.5897154808044434, 'learning_rate': 2.4394184168012926e-05, 'epoch': 0.18}

  2%|▏         | 755/41250 [1:49:47<97:49:28,  8.70s/it][2025-04-26 05:58:24,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 05:58:24,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.31 | bwd_microstep: 5753.74 | bwd_inner_microstep: 5689.56 | bwd_allreduce_microstep: 64.13 | step_microstep: 18.19
[2025-04-26 05:58:24,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.31 | bwd: 5753.75 | bwd_inner: 5689.56 | bwd_allreduce: 64.15 | step: 18.20

  2%|▏         | 756/41250 [1:49:55<97:46:09,  8.69s/it]
                                                        
{'loss': 0.3999, 'grad_norm': 3.7735838890075684, 'learning_rate': 2.4426494345718904e-05, 'epoch': 0.18}

  2%|▏         | 756/41250 [1:49:55<97:46:09,  8.69s/it][2025-04-26 05:58:33,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 05:58:33,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.74 | bwd_microstep: 5694.06 | bwd_inner_microstep: 5679.49 | bwd_allreduce_microstep: 14.53 | step_microstep: 18.93
[2025-04-26 05:58:33,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.74 | bwd: 5694.08 | bwd_inner: 5679.48 | bwd_allreduce: 14.55 | step: 18.93

  2%|▏         | 757/41250 [1:50:04<97:31:07,  8.67s/it]
                                                        
{'loss': 0.2158, 'grad_norm': 2.221426010131836, 'learning_rate': 2.445880452342488e-05, 'epoch': 0.18}

  2%|▏         | 757/41250 [1:50:04<97:31:07,  8.67s/it][2025-04-26 05:58:41,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 05:58:41,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.54 | bwd_microstep: 5718.02 | bwd_inner_microstep: 5664.53 | bwd_allreduce_microstep: 53.44 | step_microstep: 18.65
[2025-04-26 05:58:41,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.54 | bwd: 5718.03 | bwd_inner: 5664.53 | bwd_allreduce: 53.46 | step: 18.66

  2%|▏         | 758/41250 [1:50:13<97:21:00,  8.66s/it]
                                                        
{'loss': 0.1511, 'grad_norm': 1.4359480142593384, 'learning_rate': 2.449111470113086e-05, 'epoch': 0.18}

  2%|▏         | 758/41250 [1:50:13<97:21:00,  8.66s/it][2025-04-26 05:58:50,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 05:58:50,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.58 | bwd_microstep: 5798.65 | bwd_inner_microstep: 5649.55 | bwd_allreduce_microstep: 149.05 | step_microstep: 18.61
[2025-04-26 05:58:50,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.58 | bwd: 5798.66 | bwd_inner: 5649.55 | bwd_allreduce: 149.07 | step: 18.61

  2%|▏         | 759/41250 [1:50:21<97:30:44,  8.67s/it]
                                                        
{'loss': 0.2332, 'grad_norm': 2.059046506881714, 'learning_rate': 2.4523424878836838e-05, 'epoch': 0.18}

  2%|▏         | 759/41250 [1:50:21<97:30:44,  8.67s/it][2025-04-26 05:58:59,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 05:58:59,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5776.91 | bwd_inner_microstep: 5693.62 | bwd_allreduce_microstep: 83.25 | step_microstep: 18.62
[2025-04-26 05:58:59,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5776.93 | bwd_inner: 5693.62 | bwd_allreduce: 83.27 | step: 18.62

  2%|▏         | 760/41250 [1:50:30<97:38:50,  8.68s/it]
                                                        
{'loss': 0.33, 'grad_norm': 3.2385292053222656, 'learning_rate': 2.4555735056542812e-05, 'epoch': 0.18}

  2%|▏         | 760/41250 [1:50:30<97:38:50,  8.68s/it][2025-04-26 05:59:08,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 05:59:08,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.24 | bwd_microstep: 5770.86 | bwd_inner_microstep: 5651.95 | bwd_allreduce_microstep: 118.87 | step_microstep: 18.57
[2025-04-26 05:59:08,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.24 | bwd: 5770.88 | bwd_inner: 5651.95 | bwd_allreduce: 118.89 | step: 18.58

  2%|▏         | 761/41250 [1:50:39<97:39:53,  8.68s/it]
                                                        
{'loss': 0.374, 'grad_norm': 2.6498780250549316, 'learning_rate': 2.458804523424879e-05, 'epoch': 0.18}

  2%|▏         | 761/41250 [1:50:39<97:39:53,  8.68s/it][2025-04-26 05:59:16,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:59:16,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.70 | bwd_microstep: 5779.13 | bwd_inner_microstep: 5706.64 | bwd_allreduce_microstep: 72.44 | step_microstep: 18.83
[2025-04-26 05:59:16,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.70 | bwd: 5779.15 | bwd_inner: 5706.64 | bwd_allreduce: 72.46 | step: 18.83

  2%|▏         | 762/41250 [1:50:47<97:46:18,  8.69s/it]
                                                        
{'loss': 0.1901, 'grad_norm': 2.0192277431488037, 'learning_rate': 2.4620355411954768e-05, 'epoch': 0.18}

  2%|▏         | 762/41250 [1:50:47<97:46:18,  8.69s/it][2025-04-26 05:59:25,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 05:59:25,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.63 | bwd_microstep: 5773.54 | bwd_inner_microstep: 5644.74 | bwd_allreduce_microstep: 128.76 | step_microstep: 18.94
[2025-04-26 05:59:25,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.63 | bwd: 5773.56 | bwd_inner: 5644.74 | bwd_allreduce: 128.77 | step: 18.95

  2%|▏         | 763/41250 [1:50:56<97:43:17,  8.69s/it]
                                                        
{'loss': 0.2356, 'grad_norm': 2.0369324684143066, 'learning_rate': 2.4652665589660746e-05, 'epoch': 0.18}

  2%|▏         | 763/41250 [1:50:56<97:43:17,  8.69s/it][2025-04-26 05:59:34,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 05:59:34,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.11 | bwd_microstep: 5789.70 | bwd_inner_microstep: 5672.39 | bwd_allreduce_microstep: 117.26 | step_microstep: 19.10
[2025-04-26 05:59:34,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.11 | bwd: 5789.72 | bwd_inner: 5672.39 | bwd_allreduce: 117.28 | step: 19.11

  2%|▏         | 764/41250 [1:51:05<97:47:21,  8.70s/it]
                                                        
{'loss': 0.1674, 'grad_norm': 1.561408519744873, 'learning_rate': 2.468497576736672e-05, 'epoch': 0.19}

  2%|▏         | 764/41250 [1:51:05<97:47:21,  8.70s/it][2025-04-26 05:59:42,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 05:59:42,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5767.99 | bwd_inner_microstep: 5665.97 | bwd_allreduce_microstep: 101.97 | step_microstep: 19.25
[2025-04-26 05:59:42,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5768.00 | bwd_inner: 5665.97 | bwd_allreduce: 101.99 | step: 19.25

  2%|▏         | 765/41250 [1:51:13<97:45:07,  8.69s/it]
                                                        
{'loss': 0.1697, 'grad_norm': 1.5754525661468506, 'learning_rate': 2.47172859450727e-05, 'epoch': 0.19}

  2%|▏         | 765/41250 [1:51:13<97:45:07,  8.69s/it][2025-04-26 05:59:51,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 05:59:51,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5792.92 | bwd_inner_microstep: 5661.82 | bwd_allreduce_microstep: 131.06 | step_microstep: 18.94
[2025-04-26 05:59:51,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5792.94 | bwd_inner: 5661.82 | bwd_allreduce: 131.08 | step: 18.95

  2%|▏         | 766/41250 [1:51:22<97:47:51,  8.70s/it]
                                                        
{'loss': 0.1313, 'grad_norm': 1.359161138534546, 'learning_rate': 2.474959612277868e-05, 'epoch': 0.19}

  2%|▏         | 766/41250 [1:51:22<97:47:51,  8.70s/it][2025-04-26 06:00:00,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-26 06:00:00,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.15 | bwd_microstep: 5723.85 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 64.83 | step_microstep: 18.69
[2025-04-26 06:00:00,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.15 | bwd: 5723.86 | bwd_inner: 5658.97 | bwd_allreduce: 64.85 | step: 18.70

  2%|▏         | 767/41250 [1:51:31<97:34:55,  8.68s/it]
                                                        
{'loss': 0.1858, 'grad_norm': 1.7985734939575195, 'learning_rate': 2.4781906300484654e-05, 'epoch': 0.19}

  2%|▏         | 767/41250 [1:51:31<97:34:55,  8.68s/it][2025-04-26 06:00:08,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 06:00:08,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5761.65 | bwd_inner_microstep: 5713.94 | bwd_allreduce_microstep: 47.66 | step_microstep: 19.71
[2025-04-26 06:00:08,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5761.66 | bwd_inner: 5713.94 | bwd_allreduce: 47.68 | step: 19.71

  2%|▏         | 768/41250 [1:51:40<97:39:12,  8.68s/it]
                                                        
{'loss': 0.2879, 'grad_norm': 2.414097785949707, 'learning_rate': 2.4814216478190632e-05, 'epoch': 0.19}

  2%|▏         | 768/41250 [1:51:40<97:39:12,  8.68s/it][2025-04-26 06:00:17,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 06:00:17,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.01 | bwd_microstep: 5729.01 | bwd_inner_microstep: 5660.43 | bwd_allreduce_microstep: 68.54 | step_microstep: 18.85
[2025-04-26 06:00:17,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.01 | bwd: 5729.02 | bwd_inner: 5660.43 | bwd_allreduce: 68.56 | step: 18.85

  2%|▏         | 769/41250 [1:51:48<97:29:34,  8.67s/it]
                                                        
{'loss': 0.2419, 'grad_norm': 2.2750236988067627, 'learning_rate': 2.484652665589661e-05, 'epoch': 0.19}

  2%|▏         | 769/41250 [1:51:48<97:29:34,  8.67s/it][2025-04-26 06:00:26,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:00:26,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.11 | bwd_microstep: 5715.67 | bwd_inner_microstep: 5648.25 | bwd_allreduce_microstep: 67.37 | step_microstep: 18.44
[2025-04-26 06:00:26,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.11 | bwd: 5715.68 | bwd_inner: 5648.25 | bwd_allreduce: 67.39 | step: 18.45

  2%|▏         | 770/41250 [1:51:57<97:20:39,  8.66s/it]
                                                        
{'loss': 0.101, 'grad_norm': 1.1328297853469849, 'learning_rate': 2.4878836833602588e-05, 'epoch': 0.19}

  2%|▏         | 770/41250 [1:51:57<97:20:39,  8.66s/it][2025-04-26 06:00:34,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:00:34,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.11 | bwd_microstep: 5782.17 | bwd_inner_microstep: 5643.87 | bwd_allreduce_microstep: 138.25 | step_microstep: 18.61
[2025-04-26 06:00:34,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.11 | bwd: 5782.19 | bwd_inner: 5643.87 | bwd_allreduce: 138.27 | step: 18.61

  2%|▏         | 771/41250 [1:52:05<97:27:45,  8.67s/it]
                                                        
{'loss': 0.1543, 'grad_norm': 1.9961068630218506, 'learning_rate': 2.4911147011308562e-05, 'epoch': 0.19}

  2%|▏         | 771/41250 [1:52:05<97:27:45,  8.67s/it][2025-04-26 06:00:43,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 06:00:43,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.00 | bwd_microstep: 5755.42 | bwd_inner_microstep: 5708.57 | bwd_allreduce_microstep: 46.81 | step_microstep: 19.31
[2025-04-26 06:00:43,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.00 | bwd: 5755.44 | bwd_inner: 5708.57 | bwd_allreduce: 46.83 | step: 19.31

  2%|▏         | 772/41250 [1:52:14<97:30:25,  8.67s/it]
                                                        
{'loss': 0.2458, 'grad_norm': 2.6297004222869873, 'learning_rate': 2.4943457189014543e-05, 'epoch': 0.19}

  2%|▏         | 772/41250 [1:52:14<97:30:25,  8.67s/it][2025-04-26 06:00:52,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 06:00:52,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.11 | bwd_microstep: 5786.31 | bwd_inner_microstep: 5671.04 | bwd_allreduce_microstep: 115.22 | step_microstep: 18.81
[2025-04-26 06:00:52,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.11 | bwd: 5786.32 | bwd_inner: 5671.04 | bwd_allreduce: 115.24 | step: 18.81

  2%|▏         | 773/41250 [1:52:23<97:37:01,  8.68s/it]
                                                        
{'loss': 0.276, 'grad_norm': 2.1065053939819336, 'learning_rate': 2.497576736672052e-05, 'epoch': 0.19}

  2%|▏         | 773/41250 [1:52:23<97:37:01,  8.68s/it][2025-04-26 06:01:00,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:01:00,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.40 | bwd_microstep: 5710.23 | bwd_inner_microstep: 5647.20 | bwd_allreduce_microstep: 62.98 | step_microstep: 18.65
[2025-04-26 06:01:00,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.40 | bwd: 5710.24 | bwd_inner: 5647.19 | bwd_allreduce: 63.00 | step: 18.66

  2%|▏         | 774/41250 [1:52:31<97:24:55,  8.66s/it]
                                                        
{'loss': 0.2284, 'grad_norm': 2.246361017227173, 'learning_rate': 2.5008077544426496e-05, 'epoch': 0.19}

  2%|▏         | 774/41250 [1:52:31<97:24:55,  8.66s/it][2025-04-26 06:01:09,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 06:01:09,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.04 | bwd_microstep: 5756.39 | bwd_inner_microstep: 5660.15 | bwd_allreduce_microstep: 96.19 | step_microstep: 19.23
[2025-04-26 06:01:09,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.04 | bwd: 5756.40 | bwd_inner: 5660.15 | bwd_allreduce: 96.21 | step: 19.23

  2%|▏         | 775/41250 [1:52:40<97:27:51,  8.67s/it]
                                                        
{'loss': 0.2302, 'grad_norm': 2.5165441036224365, 'learning_rate': 2.5040387722132474e-05, 'epoch': 0.19}

  2%|▏         | 775/41250 [1:52:40<97:27:51,  8.67s/it][2025-04-26 06:01:18,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 06:01:18,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5658.10 | bwd_allreduce_microstep: 54.76 | step_microstep: 19.22
[2025-04-26 06:01:18,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5712.92 | bwd_inner: 5658.10 | bwd_allreduce: 54.78 | step: 19.22

  2%|▏         | 776/41250 [1:52:49<97:17:21,  8.65s/it]
                                                        
{'loss': 0.2093, 'grad_norm': 2.7486026287078857, 'learning_rate': 2.507269789983845e-05, 'epoch': 0.19}

  2%|▏         | 776/41250 [1:52:49<97:17:21,  8.65s/it][2025-04-26 06:01:26,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 06:01:26,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.60 | bwd_microstep: 5727.58 | bwd_inner_microstep: 5714.79 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.22
[2025-04-26 06:01:26,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.60 | bwd: 5727.59 | bwd_inner: 5714.79 | bwd_allreduce: 12.76 | step: 19.23

  2%|▏         | 777/41250 [1:52:57<97:19:23,  8.66s/it]
                                                        
{'loss': 0.2046, 'grad_norm': 2.159824848175049, 'learning_rate': 2.510500807754443e-05, 'epoch': 0.19}

  2%|▏         | 777/41250 [1:52:57<97:19:23,  8.66s/it][2025-04-26 06:01:35,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:01:35,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.14 | bwd_microstep: 5710.12 | bwd_inner_microstep: 5697.21 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.01
[2025-04-26 06:01:35,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.14 | bwd: 5710.14 | bwd_inner: 5697.21 | bwd_allreduce: 12.89 | step: 19.01

  2%|▏         | 778/41250 [1:53:06<97:15:21,  8.65s/it]
                                                        
{'loss': 0.1105, 'grad_norm': 0.8467579483985901, 'learning_rate': 2.5137318255250404e-05, 'epoch': 0.19}

  2%|▏         | 778/41250 [1:53:06<97:15:21,  8.65s/it][2025-04-26 06:01:43,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:01:43,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.40 | bwd_microstep: 5684.23 | bwd_inner_microstep: 5655.58 | bwd_allreduce_microstep: 28.61 | step_microstep: 18.71
[2025-04-26 06:01:43,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.40 | bwd: 5684.25 | bwd_inner: 5655.58 | bwd_allreduce: 28.63 | step: 18.72

  2%|▏         | 779/41250 [1:53:15<97:03:34,  8.63s/it]
                                                        
{'loss': 0.1397, 'grad_norm': 1.781605839729309, 'learning_rate': 2.5169628432956382e-05, 'epoch': 0.19}

  2%|▏         | 779/41250 [1:53:15<97:03:34,  8.63s/it][2025-04-26 06:01:52,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.31 | optimizer_step: 1.06
[2025-04-26 06:01:52,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.42 | bwd_microstep: 5778.31 | bwd_inner_microstep: 5640.30 | bwd_allreduce_microstep: 137.94 | step_microstep: 19.99
[2025-04-26 06:01:52,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.42 | bwd: 5778.33 | bwd_inner: 5640.30 | bwd_allreduce: 137.98 | step: 19.99

  2%|▏         | 780/41250 [1:53:23<97:14:16,  8.65s/it]
                                                        
{'loss': 0.1369, 'grad_norm': 1.498677134513855, 'learning_rate': 2.5201938610662363e-05, 'epoch': 0.19}

  2%|▏         | 780/41250 [1:53:23<97:14:16,  8.65s/it][2025-04-26 06:02:01,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:02:01,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.90 | bwd_microstep: 5758.36 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 62.51 | step_microstep: 18.42
[2025-04-26 06:02:01,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.90 | bwd: 5758.37 | bwd_inner: 5695.81 | bwd_allreduce: 62.53 | step: 18.42

  2%|▏         | 781/41250 [1:53:32<97:20:56,  8.66s/it]
                                                        
{'loss': 0.2088, 'grad_norm': 2.303523302078247, 'learning_rate': 2.5234248788368337e-05, 'epoch': 0.19}

  2%|▏         | 781/41250 [1:53:32<97:20:56,  8.66s/it][2025-04-26 06:02:10,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-26 06:02:10,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.63 | bwd_microstep: 5729.02 | bwd_inner_microstep: 5679.66 | bwd_allreduce_microstep: 49.31 | step_microstep: 18.55
[2025-04-26 06:02:10,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.63 | bwd: 5729.03 | bwd_inner: 5679.66 | bwd_allreduce: 49.33 | step: 18.56

  2%|▏         | 782/41250 [1:53:41<97:19:52,  8.66s/it]
                                                        
{'loss': 0.2811, 'grad_norm': 2.335736036300659, 'learning_rate': 2.5266558966074315e-05, 'epoch': 0.19}

  2%|▏         | 782/41250 [1:53:41<97:19:52,  8.66s/it][2025-04-26 06:02:18,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 06:02:18,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.35 | bwd_microstep: 5741.19 | bwd_inner_microstep: 5704.65 | bwd_allreduce_microstep: 36.50 | step_microstep: 18.86
[2025-04-26 06:02:18,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.35 | bwd: 5741.20 | bwd_inner: 5704.65 | bwd_allreduce: 36.52 | step: 18.87

  2%|▏         | 783/41250 [1:53:49<97:23:04,  8.66s/it]
                                                        
{'loss': 0.231, 'grad_norm': 2.4329593181610107, 'learning_rate': 2.5298869143780293e-05, 'epoch': 0.19}

  2%|▏         | 783/41250 [1:53:49<97:23:04,  8.66s/it][2025-04-26 06:02:27,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:02:27,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.52 | bwd_microstep: 5765.20 | bwd_inner_microstep: 5649.26 | bwd_allreduce_microstep: 115.90 | step_microstep: 18.50
[2025-04-26 06:02:27,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.52 | bwd: 5765.21 | bwd_inner: 5649.26 | bwd_allreduce: 115.91 | step: 18.50

  2%|▏         | 784/41250 [1:53:58<97:23:28,  8.66s/it]
                                                        
{'loss': 0.2838, 'grad_norm': 2.798306465148926, 'learning_rate': 2.533117932148627e-05, 'epoch': 0.19}

  2%|▏         | 784/41250 [1:53:58<97:23:28,  8.66s/it][2025-04-26 06:02:36,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 06:02:36,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.46 | bwd_microstep: 5728.33 | bwd_inner_microstep: 5692.65 | bwd_allreduce_microstep: 35.63 | step_microstep: 18.29
[2025-04-26 06:02:36,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.46 | bwd: 5728.34 | bwd_inner: 5692.65 | bwd_allreduce: 35.65 | step: 18.29

  2%|▏         | 785/41250 [1:54:07<97:18:38,  8.66s/it]
                                                        
{'loss': 0.288, 'grad_norm': 2.3756728172302246, 'learning_rate': 2.5363489499192246e-05, 'epoch': 0.19}

  2%|▏         | 785/41250 [1:54:07<97:18:38,  8.66s/it][2025-04-26 06:02:44,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:02:44,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.55 | bwd_microstep: 5729.28 | bwd_inner_microstep: 5703.48 | bwd_allreduce_microstep: 25.75 | step_microstep: 18.64
[2025-04-26 06:02:44,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5729.29 | bwd_inner: 5703.48 | bwd_allreduce: 25.77 | step: 18.65

  2%|▏         | 786/41250 [1:54:15<97:19:18,  8.66s/it]
                                                        
{'loss': 0.1804, 'grad_norm': 2.164794683456421, 'learning_rate': 2.5395799676898223e-05, 'epoch': 0.19}

  2%|▏         | 786/41250 [1:54:15<97:19:18,  8.66s/it][2025-04-26 06:02:53,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 1.08
[2025-04-26 06:02:53,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.00 | bwd_microstep: 5777.86 | bwd_inner_microstep: 5635.90 | bwd_allreduce_microstep: 141.92 | step_microstep: 18.51
[2025-04-26 06:02:53,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.00 | bwd: 5777.88 | bwd_inner: 5635.90 | bwd_allreduce: 141.93 | step: 18.51

  2%|▏         | 787/41250 [1:54:24<97:22:51,  8.66s/it]
                                                        
{'loss': 0.2808, 'grad_norm': 2.444744348526001, 'learning_rate': 2.5428109854604205e-05, 'epoch': 0.19}

  2%|▏         | 787/41250 [1:54:24<97:22:51,  8.66s/it][2025-04-26 06:03:02,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 06:03:02,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.05 | bwd_microstep: 5883.45 | bwd_inner_microstep: 5641.68 | bwd_allreduce_microstep: 241.73 | step_microstep: 19.17
[2025-04-26 06:03:02,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.05 | bwd: 5883.46 | bwd_inner: 5641.68 | bwd_allreduce: 241.75 | step: 19.18

  2%|▏         | 788/41250 [1:54:33<97:48:32,  8.70s/it]
                                                        
{'loss': 0.2649, 'grad_norm': 2.4545912742614746, 'learning_rate': 2.546042003231018e-05, 'epoch': 0.19}

  2%|▏         | 788/41250 [1:54:33<97:48:32,  8.70s/it][2025-04-26 06:03:10,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:03:10,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.43 | bwd_microstep: 5694.82 | bwd_inner_microstep: 5682.11 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.58
[2025-04-26 06:03:10,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.43 | bwd: 5694.84 | bwd_inner: 5682.11 | bwd_allreduce: 12.68 | step: 18.58

  2%|▏         | 789/41250 [1:54:41<97:30:56,  8.68s/it]
                                                        
{'loss': 0.2813, 'grad_norm': 2.421290874481201, 'learning_rate': 2.5492730210016157e-05, 'epoch': 0.19}

  2%|▏         | 789/41250 [1:54:41<97:30:56,  8.68s/it][2025-04-26 06:03:19,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:03:19,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.05 | bwd_microstep: 5703.81 | bwd_inner_microstep: 5691.00 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.35
[2025-04-26 06:03:19,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.05 | bwd: 5703.82 | bwd_inner: 5691.00 | bwd_allreduce: 12.78 | step: 18.35

  2%|▏         | 790/41250 [1:54:50<97:20:47,  8.66s/it]
                                                        
{'loss': 0.258, 'grad_norm': 3.3052690029144287, 'learning_rate': 2.5525040387722135e-05, 'epoch': 0.19}

  2%|▏         | 790/41250 [1:54:50<97:20:47,  8.66s/it][2025-04-26 06:03:28,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:03:28,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.99 | bwd_microstep: 5748.04 | bwd_inner_microstep: 5649.34 | bwd_allreduce_microstep: 98.66 | step_microstep: 18.09
[2025-04-26 06:03:28,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.99 | bwd: 5748.06 | bwd_inner: 5649.34 | bwd_allreduce: 98.68 | step: 18.09

  2%|▏         | 791/41250 [1:54:59<97:19:35,  8.66s/it]
                                                        
{'loss': 0.1624, 'grad_norm': 2.3876781463623047, 'learning_rate': 2.5557350565428113e-05, 'epoch': 0.19}

  2%|▏         | 791/41250 [1:54:59<97:19:35,  8.66s/it][2025-04-26 06:03:36,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:03:36,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.22 | bwd_microstep: 5694.92 | bwd_inner_microstep: 5682.20 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.29
[2025-04-26 06:03:36,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.23 | bwd: 5694.93 | bwd_inner: 5682.20 | bwd_allreduce: 12.69 | step: 18.29

  2%|▏         | 792/41250 [1:55:07<97:11:25,  8.65s/it]
                                                        
{'loss': 0.0754, 'grad_norm': 1.4159364700317383, 'learning_rate': 2.5589660743134087e-05, 'epoch': 0.19}

  2%|▏         | 792/41250 [1:55:07<97:11:25,  8.65s/it][2025-04-26 06:03:45,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 06:03:45,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.76 | bwd_microstep: 5754.90 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 102.57 | step_microstep: 18.98
[2025-04-26 06:03:45,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.76 | bwd: 5754.91 | bwd_inner: 5652.29 | bwd_allreduce: 102.58 | step: 18.98

  2%|▏         | 793/41250 [1:55:16<97:13:38,  8.65s/it]
                                                        
{'loss': 0.2501, 'grad_norm': 4.300050258636475, 'learning_rate': 2.5621970920840065e-05, 'epoch': 0.19}

  2%|▏         | 793/41250 [1:55:16<97:13:38,  8.65s/it][2025-04-26 06:03:53,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.09 | optimizer_step: 1.00
[2025-04-26 06:03:53,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.94 | bwd_microstep: 5749.18 | bwd_inner_microstep: 5705.35 | bwd_allreduce_microstep: 43.78 | step_microstep: 18.95
[2025-04-26 06:03:53,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.94 | bwd: 5749.20 | bwd_inner: 5705.35 | bwd_allreduce: 43.80 | step: 18.95

  2%|▏         | 794/41250 [1:55:25<97:16:54,  8.66s/it]
                                                        
{'loss': 0.2814, 'grad_norm': 2.8323307037353516, 'learning_rate': 2.5654281098546047e-05, 'epoch': 0.19}

  2%|▏         | 794/41250 [1:55:25<97:16:54,  8.66s/it][2025-04-26 06:04:02,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:04:02,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.92 | bwd_microstep: 5758.22 | bwd_inner_microstep: 5642.05 | bwd_allreduce_microstep: 116.12 | step_microstep: 18.70
[2025-04-26 06:04:02,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.92 | bwd: 5758.24 | bwd_inner: 5642.05 | bwd_allreduce: 116.14 | step: 18.70

  2%|▏         | 795/41250 [1:55:33<97:17:53,  8.66s/it]
                                                        
{'loss': 0.1929, 'grad_norm': 2.121718168258667, 'learning_rate': 2.568659127625202e-05, 'epoch': 0.19}

  2%|▏         | 795/41250 [1:55:33<97:17:53,  8.66s/it][2025-04-26 06:04:11,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:04:11,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.52 | bwd_microstep: 5707.57 | bwd_inner_microstep: 5646.36 | bwd_allreduce_microstep: 61.17 | step_microstep: 18.48
[2025-04-26 06:04:11,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.52 | bwd: 5707.59 | bwd_inner: 5646.36 | bwd_allreduce: 61.18 | step: 18.48

  2%|▏         | 796/41250 [1:55:42<97:08:28,  8.64s/it]
                                                        
{'loss': 0.2035, 'grad_norm': 2.20855450630188, 'learning_rate': 2.5718901453958e-05, 'epoch': 0.19}

  2%|▏         | 796/41250 [1:55:42<97:08:28,  8.64s/it][2025-04-26 06:04:19,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.22 | optimizer_step: 0.90
[2025-04-26 06:04:19,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.16 | bwd_microstep: 5737.76 | bwd_inner_microstep: 5704.58 | bwd_allreduce_microstep: 33.14 | step_microstep: 22.16
[2025-04-26 06:04:19,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.16 | bwd: 5737.78 | bwd_inner: 5704.58 | bwd_allreduce: 33.16 | step: 22.16

  2%|▏         | 797/41250 [1:55:51<97:14:03,  8.65s/it]
                                                        
{'loss': 0.3833, 'grad_norm': 3.741170644760132, 'learning_rate': 2.5751211631663977e-05, 'epoch': 0.19}

  2%|▏         | 797/41250 [1:55:51<97:14:03,  8.65s/it][2025-04-26 06:04:28,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 06:04:28,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.11 | bwd_microstep: 5765.44 | bwd_inner_microstep: 5686.34 | bwd_allreduce_microstep: 79.06 | step_microstep: 19.08
[2025-04-26 06:04:28,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.11 | bwd: 5765.46 | bwd_inner: 5686.34 | bwd_allreduce: 79.08 | step: 19.08

  2%|▏         | 798/41250 [1:55:59<97:21:52,  8.66s/it]
                                                        
{'loss': 0.2538, 'grad_norm': 3.081200361251831, 'learning_rate': 2.5783521809369955e-05, 'epoch': 0.19}

  2%|▏         | 798/41250 [1:55:59<97:21:52,  8.66s/it][2025-04-26 06:04:37,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:04:37,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5945.62 | bwd_inner_microstep: 5708.68 | bwd_allreduce_microstep: 236.89 | step_microstep: 18.50
[2025-04-26 06:04:37,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5945.63 | bwd_inner: 5708.68 | bwd_allreduce: 236.91 | step: 18.50

  2%|▏         | 799/41250 [1:56:08<98:04:29,  8.73s/it]
                                                        
{'loss': 0.1897, 'grad_norm': 1.9720547199249268, 'learning_rate': 2.581583198707593e-05, 'epoch': 0.19}

  2%|▏         | 799/41250 [1:56:08<98:04:29,  8.73s/it][2025-04-26 06:04:46,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:04:46,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.86 | bwd_microstep: 5755.10 | bwd_inner_microstep: 5662.51 | bwd_allreduce_microstep: 92.54 | step_microstep: 18.45
[2025-04-26 06:04:46,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.87 | bwd: 5755.11 | bwd_inner: 5662.51 | bwd_allreduce: 92.56 | step: 18.46

  2%|▏         | 800/41250 [1:56:17<97:51:00,  8.71s/it]
                                                        
{'loss': 0.074, 'grad_norm': 0.9173014760017395, 'learning_rate': 2.5848142164781907e-05, 'epoch': 0.19}

  2%|▏         | 800/41250 [1:56:17<97:51:00,  8.71s/it][2025-04-26 06:04:54,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-26 06:04:54,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5744.51 | bwd_inner_microstep: 5680.88 | bwd_allreduce_microstep: 63.58 | step_microstep: 19.60
[2025-04-26 06:04:54,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5744.52 | bwd_inner: 5680.88 | bwd_allreduce: 63.60 | step: 19.60

  2%|▏         | 801/41250 [1:56:26<97:44:11,  8.70s/it]
                                                        
{'loss': 0.1314, 'grad_norm': 1.4666813611984253, 'learning_rate': 2.588045234248789e-05, 'epoch': 0.19}

  2%|▏         | 801/41250 [1:56:26<97:44:11,  8.70s/it][2025-04-26 06:05:03,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 06:05:03,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.60 | bwd_microstep: 5706.08 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.63
[2025-04-26 06:05:03,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.60 | bwd: 5706.10 | bwd_inner: 5693.20 | bwd_allreduce: 12.85 | step: 18.63

  2%|▏         | 802/41250 [1:56:34<97:31:58,  8.68s/it]
                                                        
{'loss': 0.1628, 'grad_norm': 1.8441628217697144, 'learning_rate': 2.5912762520193863e-05, 'epoch': 0.19}

  2%|▏         | 802/41250 [1:56:34<97:31:58,  8.68s/it][2025-04-26 06:05:12,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:05:12,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.81 | bwd_microstep: 5714.69 | bwd_inner_microstep: 5652.08 | bwd_allreduce_microstep: 62.56 | step_microstep: 18.17
[2025-04-26 06:05:12,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.81 | bwd: 5714.70 | bwd_inner: 5652.08 | bwd_allreduce: 62.57 | step: 18.18

  2%|▏         | 803/41250 [1:56:43<97:19:54,  8.66s/it]
                                                        
{'loss': 0.1874, 'grad_norm': 2.6326098442077637, 'learning_rate': 2.594507269789984e-05, 'epoch': 0.19}

  2%|▏         | 803/41250 [1:56:43<97:19:54,  8.66s/it][2025-04-26 06:05:20,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.09
[2025-04-26 06:05:20,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.86 | bwd_microstep: 5794.22 | bwd_inner_microstep: 5781.20 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.29
[2025-04-26 06:05:20,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.86 | bwd: 5794.24 | bwd_inner: 5781.20 | bwd_allreduce: 13.00 | step: 19.29

  2%|▏         | 804/41250 [1:56:52<97:42:02,  8.70s/it]
                                                        
{'loss': 0.1196, 'grad_norm': 1.6320369243621826, 'learning_rate': 2.597738287560582e-05, 'epoch': 0.19}

  2%|▏         | 804/41250 [1:56:52<97:42:02,  8.70s/it][2025-04-26 06:05:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 06:05:29,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.06 | bwd_microstep: 5709.66 | bwd_inner_microstep: 5696.43 | bwd_allreduce_microstep: 13.18 | step_microstep: 18.85
[2025-04-26 06:05:29,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.06 | bwd: 5709.68 | bwd_inner: 5696.43 | bwd_allreduce: 13.20 | step: 18.85

  2%|▏         | 805/41250 [1:57:00<97:30:22,  8.68s/it]
                                                        
{'loss': 0.2257, 'grad_norm': 2.422882080078125, 'learning_rate': 2.6009693053311797e-05, 'epoch': 0.2}

  2%|▏         | 805/41250 [1:57:00<97:30:22,  8.68s/it][2025-04-26 06:05:38,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 06:05:38,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.31 | bwd_microstep: 5779.55 | bwd_inner_microstep: 5661.14 | bwd_allreduce_microstep: 118.37 | step_microstep: 18.55
[2025-04-26 06:05:38,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.31 | bwd: 5779.57 | bwd_inner: 5661.14 | bwd_allreduce: 118.39 | step: 18.55

  2%|▏         | 806/41250 [1:57:09<97:33:14,  8.68s/it]
                                                        
{'loss': 0.1771, 'grad_norm': 3.033414125442505, 'learning_rate': 2.604200323101777e-05, 'epoch': 0.2}

  2%|▏         | 806/41250 [1:57:09<97:33:14,  8.68s/it][2025-04-26 06:05:46,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:05:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.32 | bwd_microstep: 5745.16 | bwd_inner_microstep: 5713.16 | bwd_allreduce_microstep: 31.96 | step_microstep: 18.51
[2025-04-26 06:05:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.32 | bwd: 5745.17 | bwd_inner: 5713.16 | bwd_allreduce: 31.98 | step: 18.51

  2%|▏         | 807/41250 [1:57:18<97:32:38,  8.68s/it]
                                                        
{'loss': 0.2347, 'grad_norm': 2.283200263977051, 'learning_rate': 2.607431340872375e-05, 'epoch': 0.2}

  2%|▏         | 807/41250 [1:57:18<97:32:38,  8.68s/it][2025-04-26 06:05:55,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:05:55,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.83 | bwd_microstep: 5727.15 | bwd_inner_microstep: 5714.45 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.52
[2025-04-26 06:05:55,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.83 | bwd: 5727.16 | bwd_inner: 5714.45 | bwd_allreduce: 12.68 | step: 18.52

  2%|▏         | 808/41250 [1:57:26<97:27:00,  8.67s/it]
                                                        
{'loss': 0.2124, 'grad_norm': 4.701636791229248, 'learning_rate': 2.610662358642973e-05, 'epoch': 0.2}

  2%|▏         | 808/41250 [1:57:26<97:27:00,  8.67s/it][2025-04-26 06:06:04,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:06:04,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5779.08 | bwd_inner_microstep: 5661.66 | bwd_allreduce_microstep: 117.37 | step_microstep: 18.31
[2025-04-26 06:06:04,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5779.09 | bwd_inner: 5661.66 | bwd_allreduce: 117.39 | step: 18.32

  2%|▏         | 809/41250 [1:57:35<97:31:47,  8.68s/it]
                                                        
{'loss': 0.109, 'grad_norm': 1.4202111959457397, 'learning_rate': 2.6138933764135705e-05, 'epoch': 0.2}

  2%|▏         | 809/41250 [1:57:35<97:31:47,  8.68s/it][2025-04-26 06:06:12,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.08
[2025-04-26 06:06:12,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.77 | bwd_microstep: 5770.51 | bwd_inner_microstep: 5699.85 | bwd_allreduce_microstep: 70.60 | step_microstep: 19.35
[2025-04-26 06:06:12,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.77 | bwd: 5770.52 | bwd_inner: 5699.85 | bwd_allreduce: 70.62 | step: 19.36

  2%|▏         | 810/41250 [1:57:44<97:37:01,  8.69s/it]
                                                        
{'loss': 0.1487, 'grad_norm': 1.785398006439209, 'learning_rate': 2.6171243941841683e-05, 'epoch': 0.2}

  2%|▏         | 810/41250 [1:57:44<97:37:01,  8.69s/it][2025-04-26 06:06:21,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:06:21,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5710.37 | bwd_inner_microstep: 5653.00 | bwd_allreduce_microstep: 57.32 | step_microstep: 17.95
[2025-04-26 06:06:21,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5710.38 | bwd_inner: 5653.00 | bwd_allreduce: 57.34 | step: 17.95

  2%|▏         | 811/41250 [1:57:52<97:22:33,  8.67s/it]
                                                        
{'loss': 0.2276, 'grad_norm': 2.1395814418792725, 'learning_rate': 2.620355411954766e-05, 'epoch': 0.2}

  2%|▏         | 811/41250 [1:57:52<97:22:33,  8.67s/it][2025-04-26 06:06:30,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:06:30,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.61 | bwd_microstep: 5721.16 | bwd_inner_microstep: 5674.88 | bwd_allreduce_microstep: 46.23 | step_microstep: 17.94
[2025-04-26 06:06:30,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.61 | bwd: 5721.17 | bwd_inner: 5674.88 | bwd_allreduce: 46.25 | step: 17.95

  2%|▏         | 812/41250 [1:58:01<97:13:02,  8.65s/it]
                                                        
{'loss': 0.1796, 'grad_norm': 1.7840440273284912, 'learning_rate': 2.6235864297253638e-05, 'epoch': 0.2}

  2%|▏         | 812/41250 [1:58:01<97:13:02,  8.65s/it][2025-04-26 06:06:38,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:06:38,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.83 | bwd_microstep: 5727.27 | bwd_inner_microstep: 5714.55 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.29
[2025-04-26 06:06:38,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.83 | bwd: 5727.28 | bwd_inner: 5714.55 | bwd_allreduce: 12.69 | step: 18.30

  2%|▏         | 813/41250 [1:58:10<97:14:59,  8.66s/it]
                                                        
{'loss': 0.2221, 'grad_norm': 1.8297414779663086, 'learning_rate': 2.6268174474959613e-05, 'epoch': 0.2}

  2%|▏         | 813/41250 [1:58:10<97:14:59,  8.66s/it][2025-04-26 06:06:47,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:06:47,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.82 | bwd_microstep: 5794.94 | bwd_inner_microstep: 5648.11 | bwd_allreduce_microstep: 146.79 | step_microstep: 17.57
[2025-04-26 06:06:47,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.82 | bwd: 5794.95 | bwd_inner: 5648.11 | bwd_allreduce: 146.80 | step: 17.58

  2%|▏         | 814/41250 [1:58:18<97:22:49,  8.67s/it]
                                                        
{'loss': 0.3949, 'grad_norm': 4.478221416473389, 'learning_rate': 2.630048465266559e-05, 'epoch': 0.2}

  2%|▏         | 814/41250 [1:58:18<97:22:49,  8.67s/it][2025-04-26 06:06:56,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:06:56,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.80 | bwd_microstep: 5780.27 | bwd_inner_microstep: 5700.33 | bwd_allreduce_microstep: 79.90 | step_microstep: 18.32
[2025-04-26 06:06:56,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.79 | bwd: 5780.28 | bwd_inner: 5700.33 | bwd_allreduce: 79.92 | step: 18.32

  2%|▏         | 815/41250 [1:58:27<97:30:13,  8.68s/it]
                                                        
{'loss': 0.1513, 'grad_norm': 1.2025611400604248, 'learning_rate': 2.6332794830371572e-05, 'epoch': 0.2}

  2%|▏         | 815/41250 [1:58:27<97:30:13,  8.68s/it][2025-04-26 06:07:04,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 06:07:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.50 | bwd_microstep: 5720.63 | bwd_inner_microstep: 5707.37 | bwd_allreduce_microstep: 13.21 | step_microstep: 18.46
[2025-04-26 06:07:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.50 | bwd: 5720.64 | bwd_inner: 5707.37 | bwd_allreduce: 13.23 | step: 18.47

  2%|▏         | 816/41250 [1:58:36<97:25:06,  8.67s/it]
                                                        
{'loss': 0.201, 'grad_norm': 4.964522361755371, 'learning_rate': 2.6365105008077546e-05, 'epoch': 0.2}

  2%|▏         | 816/41250 [1:58:36<97:25:06,  8.67s/it][2025-04-26 06:07:13,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.63 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 06:07:13,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.42 | bwd_microstep: 5712.05 | bwd_inner_microstep: 5699.46 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.70
[2025-04-26 06:07:13,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.42 | bwd: 5712.06 | bwd_inner: 5699.46 | bwd_allreduce: 12.57 | step: 18.70

  2%|▏         | 817/41250 [1:58:44<97:18:00,  8.66s/it]
                                                        
{'loss': 0.2682, 'grad_norm': 3.9852359294891357, 'learning_rate': 2.6397415185783524e-05, 'epoch': 0.2}

  2%|▏         | 817/41250 [1:58:44<97:18:00,  8.66s/it][2025-04-26 06:07:22,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 06:07:22,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.37 | bwd_microstep: 5800.44 | bwd_inner_microstep: 5646.75 | bwd_allreduce_microstep: 153.65 | step_microstep: 18.43
[2025-04-26 06:07:22,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.37 | bwd: 5800.45 | bwd_inner: 5646.75 | bwd_allreduce: 153.67 | step: 18.43

  2%|▏         | 818/41250 [1:58:53<97:27:02,  8.68s/it]
                                                        
{'loss': 0.2159, 'grad_norm': 2.5417065620422363, 'learning_rate': 2.6429725363489502e-05, 'epoch': 0.2}

  2%|▏         | 818/41250 [1:58:53<97:27:02,  8.68s/it][2025-04-26 06:07:30,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:07:30,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.28 | bwd_microstep: 5717.76 | bwd_inner_microstep: 5704.98 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.37
[2025-04-26 06:07:30,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.28 | bwd: 5717.77 | bwd_inner: 5704.98 | bwd_allreduce: 12.76 | step: 18.38

  2%|▏         | 819/41250 [1:59:02<97:22:00,  8.67s/it]
                                                        
{'loss': 0.286, 'grad_norm': 4.026177406311035, 'learning_rate': 2.6462035541195477e-05, 'epoch': 0.2}

  2%|▏         | 819/41250 [1:59:02<97:22:00,  8.67s/it][2025-04-26 06:07:39,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:07:39,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.28 | bwd_microstep: 5697.38 | bwd_inner_microstep: 5659.15 | bwd_allreduce_microstep: 38.19 | step_microstep: 18.38
[2025-04-26 06:07:39,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.28 | bwd: 5697.39 | bwd_inner: 5659.15 | bwd_allreduce: 38.21 | step: 18.38

  2%|▏         | 820/41250 [1:59:10<97:11:05,  8.65s/it]
                                                        
{'loss': 0.0798, 'grad_norm': 0.9759501814842224, 'learning_rate': 2.6494345718901455e-05, 'epoch': 0.2}

  2%|▏         | 820/41250 [1:59:10<97:11:05,  8.65s/it][2025-04-26 06:07:48,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 06:07:48,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.28 | bwd_microstep: 5725.66 | bwd_inner_microstep: 5712.67 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.65
[2025-04-26 06:07:48,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.28 | bwd: 5725.67 | bwd_inner: 5712.67 | bwd_allreduce: 12.96 | step: 18.65

  2%|▏         | 821/41250 [1:59:19<97:12:59,  8.66s/it]
                                                        
{'loss': 0.4296, 'grad_norm': 3.219937324523926, 'learning_rate': 2.6526655896607432e-05, 'epoch': 0.2}

  2%|▏         | 821/41250 [1:59:19<97:12:59,  8.66s/it][2025-04-26 06:07:56,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 06:07:56,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.59 | bwd_microstep: 5743.02 | bwd_inner_microstep: 5665.17 | bwd_allreduce_microstep: 77.80 | step_microstep: 19.03
[2025-04-26 06:07:56,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.59 | bwd: 5743.03 | bwd_inner: 5665.17 | bwd_allreduce: 77.82 | step: 19.03

  2%|▏         | 822/41250 [1:59:28<97:14:08,  8.66s/it]
                                                        
{'loss': 0.2425, 'grad_norm': 2.436447858810425, 'learning_rate': 2.6558966074313414e-05, 'epoch': 0.2}

  2%|▏         | 822/41250 [1:59:28<97:14:08,  8.66s/it][2025-04-26 06:08:05,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 06:08:05,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.04 | bwd_microstep: 5747.84 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 54.59 | step_microstep: 18.80
[2025-04-26 06:08:05,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.04 | bwd: 5747.85 | bwd_inner: 5693.20 | bwd_allreduce: 54.61 | step: 18.81

  2%|▏         | 823/41250 [1:59:36<97:18:14,  8.66s/it]
                                                        
{'loss': 0.2438, 'grad_norm': 2.419745683670044, 'learning_rate': 2.6591276252019388e-05, 'epoch': 0.2}

  2%|▏         | 823/41250 [1:59:36<97:18:14,  8.66s/it][2025-04-26 06:08:14,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:08:14,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.67 | bwd_microstep: 5701.78 | bwd_inner_microstep: 5649.81 | bwd_allreduce_microstep: 51.92 | step_microstep: 18.71
[2025-04-26 06:08:14,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.67 | bwd: 5701.80 | bwd_inner: 5649.81 | bwd_allreduce: 51.94 | step: 18.71

  2%|▏         | 824/41250 [1:59:45<97:07:30,  8.65s/it]
                                                        
{'loss': 0.086, 'grad_norm': 1.5424598455429077, 'learning_rate': 2.6623586429725366e-05, 'epoch': 0.2}

  2%|▏         | 824/41250 [1:59:45<97:07:30,  8.65s/it][2025-04-26 06:08:22,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:08:22,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.91 | bwd_microstep: 5777.50 | bwd_inner_microstep: 5696.73 | bwd_allreduce_microstep: 80.72 | step_microstep: 18.80
[2025-04-26 06:08:22,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.91 | bwd: 5777.51 | bwd_inner: 5696.73 | bwd_allreduce: 80.74 | step: 18.80

  2%|▏         | 825/41250 [1:59:54<97:19:45,  8.67s/it]
                                                        
{'loss': 0.2323, 'grad_norm': 1.851832628250122, 'learning_rate': 2.6655896607431344e-05, 'epoch': 0.2}

  2%|▏         | 825/41250 [1:59:54<97:19:45,  8.67s/it][2025-04-26 06:08:31,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:08:31,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.48 | bwd_microstep: 5735.02 | bwd_inner_microstep: 5680.96 | bwd_allreduce_microstep: 54.02 | step_microstep: 18.61
[2025-04-26 06:08:31,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.48 | bwd: 5735.03 | bwd_inner: 5680.96 | bwd_allreduce: 54.03 | step: 18.61

  2%|▏         | 826/41250 [2:00:02<97:20:52,  8.67s/it]
                                                        
{'loss': 0.2901, 'grad_norm': 1.7988249063491821, 'learning_rate': 2.668820678513732e-05, 'epoch': 0.2}

  2%|▏         | 826/41250 [2:00:02<97:20:52,  8.67s/it][2025-04-26 06:08:40,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.11 | optimizer_step: 0.97
[2025-04-26 06:08:40,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.84 | bwd_microstep: 5685.67 | bwd_inner_microstep: 5629.28 | bwd_allreduce_microstep: 56.34 | step_microstep: 19.11
[2025-04-26 06:08:40,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.84 | bwd: 5685.68 | bwd_inner: 5629.28 | bwd_allreduce: 56.36 | step: 19.12

  2%|▏         | 827/41250 [2:00:11<97:05:25,  8.65s/it]
                                                        
{'loss': 0.3457, 'grad_norm': 3.1008100509643555, 'learning_rate': 2.6720516962843296e-05, 'epoch': 0.2}

  2%|▏         | 827/41250 [2:00:11<97:05:25,  8.65s/it][2025-04-26 06:08:48,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 06:08:48,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.47 | bwd_microstep: 5695.41 | bwd_inner_microstep: 5681.97 | bwd_allreduce_microstep: 13.38 | step_microstep: 18.84
[2025-04-26 06:08:48,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.47 | bwd: 5695.42 | bwd_inner: 5681.97 | bwd_allreduce: 13.41 | step: 18.84

  2%|▏         | 828/41250 [2:00:19<97:01:20,  8.64s/it]
                                                        
{'loss': 0.1522, 'grad_norm': 1.3315818309783936, 'learning_rate': 2.6752827140549274e-05, 'epoch': 0.2}

  2%|▏         | 828/41250 [2:00:19<97:01:20,  8.64s/it][2025-04-26 06:08:57,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 06:08:57,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.80 | bwd_microstep: 5683.76 | bwd_inner_microstep: 5654.16 | bwd_allreduce_microstep: 29.56 | step_microstep: 18.69
[2025-04-26 06:08:57,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.80 | bwd: 5683.78 | bwd_inner: 5654.16 | bwd_allreduce: 29.58 | step: 18.69

  2%|▏         | 829/41250 [2:00:28<96:52:32,  8.63s/it]
                                                        
{'loss': 0.3713, 'grad_norm': 2.814002513885498, 'learning_rate': 2.6785137318255256e-05, 'epoch': 0.2}

  2%|▏         | 829/41250 [2:00:28<96:52:32,  8.63s/it][2025-04-26 06:09:06,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 06:09:06,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.80 | bwd_microstep: 5751.18 | bwd_inner_microstep: 5701.83 | bwd_allreduce_microstep: 49.31 | step_microstep: 18.47
[2025-04-26 06:09:06,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.80 | bwd: 5751.20 | bwd_inner: 5701.83 | bwd_allreduce: 49.33 | step: 18.47

  2%|▏         | 830/41250 [2:00:37<97:00:41,  8.64s/it]
                                                        
{'loss': 0.281, 'grad_norm': 1.9897451400756836, 'learning_rate': 2.681744749596123e-05, 'epoch': 0.2}

  2%|▏         | 830/41250 [2:00:37<97:00:41,  8.64s/it][2025-04-26 06:09:14,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:09:14,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.47 | bwd_microstep: 5736.91 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 38.41 | step_microstep: 18.66
[2025-04-26 06:09:14,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.47 | bwd: 5736.92 | bwd_inner: 5698.45 | bwd_allreduce: 38.43 | step: 18.67

  2%|▏         | 831/41250 [2:00:45<97:07:53,  8.65s/it]
                                                        
{'loss': 0.1366, 'grad_norm': 1.1846284866333008, 'learning_rate': 2.6849757673667208e-05, 'epoch': 0.2}

  2%|▏         | 831/41250 [2:00:45<97:07:53,  8.65s/it][2025-04-26 06:09:23,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 06:09:23,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.47 | bwd_microstep: 5699.20 | bwd_inner_microstep: 5686.47 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.41
[2025-04-26 06:09:23,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.47 | bwd: 5699.21 | bwd_inner: 5686.47 | bwd_allreduce: 12.70 | step: 18.42

  2%|▏         | 832/41250 [2:00:54<97:03:21,  8.64s/it]
                                                        
{'loss': 0.3612, 'grad_norm': 3.3154776096343994, 'learning_rate': 2.6882067851373186e-05, 'epoch': 0.2}

  2%|▏         | 832/41250 [2:00:54<97:03:21,  8.64s/it][2025-04-26 06:09:32,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:09:32,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.20 | bwd_microstep: 5751.40 | bwd_inner_microstep: 5647.54 | bwd_allreduce_microstep: 103.82 | step_microstep: 18.53
[2025-04-26 06:09:32,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.20 | bwd: 5751.41 | bwd_inner: 5647.54 | bwd_allreduce: 103.84 | step: 18.53

  2%|▏         | 833/41250 [2:01:03<97:09:18,  8.65s/it]
                                                        
{'loss': 0.1638, 'grad_norm': 1.5208185911178589, 'learning_rate': 2.691437802907916e-05, 'epoch': 0.2}

  2%|▏         | 833/41250 [2:01:03<97:09:18,  8.65s/it][2025-04-26 06:09:40,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:09:40,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.00 | bwd_microstep: 5782.43 | bwd_inner_microstep: 5653.99 | bwd_allreduce_microstep: 128.40 | step_microstep: 18.13
[2025-04-26 06:09:40,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.00 | bwd: 5782.44 | bwd_inner: 5653.99 | bwd_allreduce: 128.41 | step: 18.14

  2%|▏         | 834/41250 [2:01:11<97:15:52,  8.66s/it]
                                                        
{'loss': 0.1332, 'grad_norm': 1.4651763439178467, 'learning_rate': 2.6946688206785138e-05, 'epoch': 0.2}

  2%|▏         | 834/41250 [2:01:11<97:15:52,  8.66s/it][2025-04-26 06:09:49,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 06:09:49,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5716.92 | bwd_inner_microstep: 5703.89 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.87
[2025-04-26 06:09:49,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5716.93 | bwd_inner: 5703.89 | bwd_allreduce: 13.00 | step: 18.87

  2%|▏         | 835/41250 [2:01:20<97:12:14,  8.66s/it]
                                                        
{'loss': 0.1047, 'grad_norm': 0.9671709537506104, 'learning_rate': 2.6978998384491116e-05, 'epoch': 0.2}

  2%|▏         | 835/41250 [2:01:20<97:12:14,  8.66s/it][2025-04-26 06:09:57,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 06:09:57,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.91 | bwd_microstep: 5696.65 | bwd_inner_microstep: 5647.21 | bwd_allreduce_microstep: 49.39 | step_microstep: 19.47
[2025-04-26 06:09:57,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.91 | bwd: 5696.66 | bwd_inner: 5647.21 | bwd_allreduce: 49.41 | step: 19.47

  2%|▏         | 836/41250 [2:01:29<97:01:29,  8.64s/it]
                                                        
{'loss': 0.3688, 'grad_norm': 2.2180864810943604, 'learning_rate': 2.7011308562197097e-05, 'epoch': 0.2}

  2%|▏         | 836/41250 [2:01:29<97:01:29,  8.64s/it][2025-04-26 06:10:06,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:10:06,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.95 | bwd_microstep: 5771.32 | bwd_inner_microstep: 5648.29 | bwd_allreduce_microstep: 122.98 | step_microstep: 18.86
[2025-04-26 06:10:06,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.95 | bwd: 5771.33 | bwd_inner: 5648.29 | bwd_allreduce: 123.00 | step: 18.86

  2%|▏         | 837/41250 [2:01:37<97:08:51,  8.65s/it]
                                                        
{'loss': 0.1473, 'grad_norm': 1.7234323024749756, 'learning_rate': 2.7043618739903072e-05, 'epoch': 0.2}

  2%|▏         | 837/41250 [2:01:37<97:08:51,  8.65s/it][2025-04-26 06:10:15,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 06:10:15,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.38 | bwd_microstep: 5697.05 | bwd_inner_microstep: 5684.11 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.28
[2025-04-26 06:10:15,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.38 | bwd: 5697.06 | bwd_inner: 5684.11 | bwd_allreduce: 12.91 | step: 19.28

  2%|▏         | 838/41250 [2:01:46<97:02:08,  8.64s/it]
                                                        
{'loss': 0.1967, 'grad_norm': 2.004673957824707, 'learning_rate': 2.707592891760905e-05, 'epoch': 0.2}

  2%|▏         | 838/41250 [2:01:46<97:02:08,  8.64s/it][2025-04-26 06:10:24,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-26 06:10:24,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.61 | bwd_microstep: 6034.57 | bwd_inner_microstep: 5660.74 | bwd_allreduce_microstep: 373.79 | step_microstep: 18.78
[2025-04-26 06:10:24,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.61 | bwd: 6034.58 | bwd_inner: 5660.74 | bwd_allreduce: 373.81 | step: 18.78

  2%|▏         | 839/41250 [2:01:55<98:01:14,  8.73s/it]
                                                        
{'loss': 0.0836, 'grad_norm': 0.8617979884147644, 'learning_rate': 2.7108239095315028e-05, 'epoch': 0.2}

  2%|▏         | 839/41250 [2:01:55<98:01:14,  8.73s/it][2025-04-26 06:10:32,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.15 | optimizer_step: 0.95
[2025-04-26 06:10:32,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.50 | bwd_microstep: 5783.44 | bwd_inner_microstep: 5770.80 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.29
[2025-04-26 06:10:32,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.50 | bwd: 5783.46 | bwd_inner: 5770.80 | bwd_allreduce: 12.61 | step: 19.29

  2%|▏         | 840/41250 [2:02:04<98:04:45,  8.74s/it]
                                                        
{'loss': 0.1355, 'grad_norm': 1.5027737617492676, 'learning_rate': 2.7140549273021002e-05, 'epoch': 0.2}

  2%|▏         | 840/41250 [2:02:04<98:04:45,  8.74s/it][2025-04-26 06:10:41,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:10:41,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.66 | bwd_microstep: 5712.92 | bwd_inner_microstep: 5661.83 | bwd_allreduce_microstep: 51.05 | step_microstep: 18.84
[2025-04-26 06:10:41,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.66 | bwd: 5712.94 | bwd_inner: 5661.83 | bwd_allreduce: 51.07 | step: 18.85

  2%|▏         | 841/41250 [2:02:12<97:42:00,  8.70s/it]
                                                        
{'loss': 0.2674, 'grad_norm': 3.9435977935791016, 'learning_rate': 2.717285945072698e-05, 'epoch': 0.2}

  2%|▏         | 841/41250 [2:02:12<97:42:00,  8.70s/it][2025-04-26 06:10:50,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 06:10:50,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.56 | bwd_microstep: 5708.85 | bwd_inner_microstep: 5696.14 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.08
[2025-04-26 06:10:50,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.56 | bwd: 5708.86 | bwd_inner: 5696.14 | bwd_allreduce: 12.68 | step: 19.08

  2%|▏         | 842/41250 [2:02:21<97:28:13,  8.68s/it]
                                                        
{'loss': 0.2966, 'grad_norm': 2.5119144916534424, 'learning_rate': 2.7205169628432958e-05, 'epoch': 0.2}

  2%|▏         | 842/41250 [2:02:21<97:28:13,  8.68s/it][2025-04-26 06:10:58,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 06:10:58,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.87 | bwd_microstep: 5710.76 | bwd_inner_microstep: 5698.06 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.66
[2025-04-26 06:10:58,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.87 | bwd: 5710.77 | bwd_inner: 5698.06 | bwd_allreduce: 12.67 | step: 18.66

  2%|▏         | 843/41250 [2:02:30<97:20:44,  8.67s/it]
                                                        
{'loss': 0.302, 'grad_norm': 2.260047435760498, 'learning_rate': 2.723747980613894e-05, 'epoch': 0.2}

  2%|▏         | 843/41250 [2:02:30<97:20:44,  8.67s/it][2025-04-26 06:11:07,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:11:07,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.48 | bwd_microstep: 5698.31 | bwd_inner_microstep: 5685.41 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.52
[2025-04-26 06:11:07,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.48 | bwd: 5698.32 | bwd_inner: 5685.41 | bwd_allreduce: 12.87 | step: 18.53

  2%|▏         | 844/41250 [2:02:38<97:08:48,  8.66s/it]
                                                        
{'loss': 0.3011, 'grad_norm': 1.593686819076538, 'learning_rate': 2.7269789983844914e-05, 'epoch': 0.2}

  2%|▏         | 844/41250 [2:02:38<97:08:48,  8.66s/it][2025-04-26 06:11:16,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.38 | optimizer_step: 1.07
[2025-04-26 06:11:16,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.11 | bwd_microstep: 5703.34 | bwd_inner_microstep: 5689.68 | bwd_allreduce_microstep: 13.61 | step_microstep: 20.44
[2025-04-26 06:11:16,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.11 | bwd: 5703.36 | bwd_inner: 5689.68 | bwd_allreduce: 13.63 | step: 20.44

  2%|▏         | 845/41250 [2:02:47<97:07:07,  8.65s/it]
                                                        
{'loss': 0.214, 'grad_norm': 2.376133680343628, 'learning_rate': 2.730210016155089e-05, 'epoch': 0.2}

  2%|▏         | 845/41250 [2:02:47<97:07:07,  8.65s/it][2025-04-26 06:11:24,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.69 | optimizer_gradients: 1.10 | optimizer_step: 0.90
[2025-04-26 06:11:24,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.34 | bwd_microstep: 5744.42 | bwd_inner_microstep: 5725.56 | bwd_allreduce_microstep: 18.81 | step_microstep: 19.64
[2025-04-26 06:11:24,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.34 | bwd: 5744.44 | bwd_inner: 5725.56 | bwd_allreduce: 18.84 | step: 19.65

  2%|▏         | 846/41250 [2:02:55<97:13:45,  8.66s/it]
                                                        
{'loss': 0.2375, 'grad_norm': 2.481703996658325, 'learning_rate': 2.733441033925687e-05, 'epoch': 0.21}

  2%|▏         | 846/41250 [2:02:55<97:13:45,  8.66s/it][2025-04-26 06:11:33,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:11:33,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5769.11 | bwd_inner_microstep: 5646.84 | bwd_allreduce_microstep: 122.22 | step_microstep: 18.40
[2025-04-26 06:11:33,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5769.12 | bwd_inner: 5646.84 | bwd_allreduce: 122.24 | step: 18.41

  2%|▏         | 847/41250 [2:03:04<97:17:18,  8.67s/it]
                                                        
{'loss': 0.4158, 'grad_norm': 3.405677318572998, 'learning_rate': 2.7366720516962844e-05, 'epoch': 0.21}

  2%|▏         | 847/41250 [2:03:04<97:17:18,  8.67s/it][2025-04-26 06:11:42,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:11:42,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.55 | bwd_microstep: 5780.28 | bwd_inner_microstep: 5653.14 | bwd_allreduce_microstep: 127.10 | step_microstep: 18.40
[2025-04-26 06:11:42,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.55 | bwd: 5780.29 | bwd_inner: 5653.14 | bwd_allreduce: 127.11 | step: 18.41

  2%|▏         | 848/41250 [2:03:13<97:20:48,  8.67s/it]
                                                        
{'loss': 0.1184, 'grad_norm': 1.1000919342041016, 'learning_rate': 2.7399030694668822e-05, 'epoch': 0.21}

  2%|▏         | 848/41250 [2:03:13<97:20:48,  8.67s/it][2025-04-26 06:11:50,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 06:11:50,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.56 | bwd_microstep: 5767.21 | bwd_inner_microstep: 5710.73 | bwd_allreduce_microstep: 56.44 | step_microstep: 19.31
[2025-04-26 06:11:50,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.56 | bwd: 5767.22 | bwd_inner: 5710.73 | bwd_allreduce: 56.46 | step: 19.31

  2%|▏         | 849/41250 [2:03:22<97:27:21,  8.68s/it]
                                                        
{'loss': 0.1709, 'grad_norm': 2.531466484069824, 'learning_rate': 2.74313408723748e-05, 'epoch': 0.21}

  2%|▏         | 849/41250 [2:03:22<97:27:21,  8.68s/it][2025-04-26 06:11:59,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 06:11:59,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.75 | bwd_microstep: 5717.89 | bwd_inner_microstep: 5657.20 | bwd_allreduce_microstep: 60.64 | step_microstep: 18.89
[2025-04-26 06:11:59,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.75 | bwd: 5717.91 | bwd_inner: 5657.20 | bwd_allreduce: 60.67 | step: 18.90

  2%|▏         | 850/41250 [2:03:30<97:17:07,  8.67s/it]
                                                        
{'loss': 0.1742, 'grad_norm': 2.6680855751037598, 'learning_rate': 2.746365105008078e-05, 'epoch': 0.21}

  2%|▏         | 850/41250 [2:03:30<97:17:07,  8.67s/it][2025-04-26 06:12:08,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-26 06:12:08,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5716.42 | bwd_inner_microstep: 5653.79 | bwd_allreduce_microstep: 62.57 | step_microstep: 18.97
[2025-04-26 06:12:08,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5716.44 | bwd_inner: 5653.79 | bwd_allreduce: 62.60 | step: 18.97

  2%|▏         | 851/41250 [2:03:39<97:10:46,  8.66s/it]
                                                        
{'loss': 0.4252, 'grad_norm': 3.1083879470825195, 'learning_rate': 2.7495961227786755e-05, 'epoch': 0.21}

  2%|▏         | 851/41250 [2:03:39<97:10:46,  8.66s/it][2025-04-26 06:12:16,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 06:12:16,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5792.78 | bwd_inner_microstep: 5648.55 | bwd_allreduce_microstep: 144.18 | step_microstep: 19.12
[2025-04-26 06:12:16,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5792.79 | bwd_inner: 5648.55 | bwd_allreduce: 144.20 | step: 19.13

  2%|▏         | 852/41250 [2:03:48<97:18:59,  8.67s/it]
                                                        
{'loss': 0.2135, 'grad_norm': 2.657681941986084, 'learning_rate': 2.7528271405492733e-05, 'epoch': 0.21}

  2%|▏         | 852/41250 [2:03:48<97:18:59,  8.67s/it][2025-04-26 06:12:25,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-26 06:12:25,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.79 | bwd_microstep: 5713.91 | bwd_inner_microstep: 5701.05 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.10
[2025-04-26 06:12:25,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.79 | bwd: 5713.92 | bwd_inner: 5701.05 | bwd_allreduce: 12.83 | step: 19.10

  2%|▏         | 853/41250 [2:03:56<97:12:03,  8.66s/it]
                                                        
{'loss': 0.2009, 'grad_norm': 1.867078423500061, 'learning_rate': 2.756058158319871e-05, 'epoch': 0.21}

  2%|▏         | 853/41250 [2:03:56<97:12:03,  8.66s/it][2025-04-26 06:12:34,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.07 | optimizer_step: 1.12
[2025-04-26 06:12:34,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.37 | bwd_microstep: 5706.95 | bwd_inner_microstep: 5694.11 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.59
[2025-04-26 06:12:34,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.37 | bwd: 5706.96 | bwd_inner: 5694.11 | bwd_allreduce: 12.81 | step: 19.59

  2%|▏         | 854/41250 [2:04:05<97:07:21,  8.66s/it]
                                                        
{'loss': 0.3476, 'grad_norm': 2.7081644535064697, 'learning_rate': 2.7592891760904686e-05, 'epoch': 0.21}

  2%|▏         | 854/41250 [2:04:05<97:07:21,  8.66s/it][2025-04-26 06:12:42,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 06:12:42,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.72 | bwd_microstep: 5727.07 | bwd_inner_microstep: 5714.26 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.92
[2025-04-26 06:12:42,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.72 | bwd: 5727.08 | bwd_inner: 5714.26 | bwd_allreduce: 12.78 | step: 18.92

  2%|▏         | 855/41250 [2:04:13<97:08:20,  8.66s/it]
                                                        
{'loss': 0.1561, 'grad_norm': 2.041283130645752, 'learning_rate': 2.7625201938610664e-05, 'epoch': 0.21}

  2%|▏         | 855/41250 [2:04:13<97:08:20,  8.66s/it][2025-04-26 06:12:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:12:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.60 | bwd_microstep: 5717.58 | bwd_inner_microstep: 5644.06 | bwd_allreduce_microstep: 73.48 | step_microstep: 18.56
[2025-04-26 06:12:51,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.60 | bwd: 5717.60 | bwd_inner: 5644.06 | bwd_allreduce: 73.50 | step: 18.56

  2%|▏         | 856/41250 [2:04:22<97:01:35,  8.65s/it]
                                                        
{'loss': 0.1034, 'grad_norm': 1.0097554922103882, 'learning_rate': 2.765751211631664e-05, 'epoch': 0.21}

  2%|▏         | 856/41250 [2:04:22<97:01:35,  8.65s/it][2025-04-26 06:13:00,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 06:13:00,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.35 | bwd_microstep: 5759.85 | bwd_inner_microstep: 5709.11 | bwd_allreduce_microstep: 50.69 | step_microstep: 18.71
[2025-04-26 06:13:00,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.35 | bwd: 5759.87 | bwd_inner: 5709.11 | bwd_allreduce: 50.71 | step: 18.72

  2%|▏         | 857/41250 [2:04:31<97:09:21,  8.66s/it]
                                                        
{'loss': 0.2582, 'grad_norm': 2.3300395011901855, 'learning_rate': 2.7689822294022623e-05, 'epoch': 0.21}

  2%|▏         | 857/41250 [2:04:31<97:09:21,  8.66s/it][2025-04-26 06:13:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 06:13:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.37 | bwd_microstep: 5760.77 | bwd_inner_microstep: 5662.43 | bwd_allreduce_microstep: 98.29 | step_microstep: 19.14
[2025-04-26 06:13:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.37 | bwd: 5760.78 | bwd_inner: 5662.43 | bwd_allreduce: 98.31 | step: 19.14

  2%|▏         | 858/41250 [2:04:39<97:13:42,  8.67s/it]
                                                        
{'loss': 0.2162, 'grad_norm': 2.0226292610168457, 'learning_rate': 2.7722132471728597e-05, 'epoch': 0.21}

  2%|▏         | 858/41250 [2:04:39<97:13:42,  8.67s/it][2025-04-26 06:13:17,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:13:17,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.42 | bwd_microstep: 5875.53 | bwd_inner_microstep: 5656.83 | bwd_allreduce_microstep: 218.65 | step_microstep: 18.89
[2025-04-26 06:13:17,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.42 | bwd: 5875.54 | bwd_inner: 5656.84 | bwd_allreduce: 218.67 | step: 18.89

  2%|▏         | 859/41250 [2:04:48<97:39:08,  8.70s/it]
                                                        
{'loss': 0.1578, 'grad_norm': 2.0293078422546387, 'learning_rate': 2.7754442649434575e-05, 'epoch': 0.21}

  2%|▏         | 859/41250 [2:04:48<97:39:08,  8.70s/it][2025-04-26 06:13:26,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:13:26,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.39 | bwd_microstep: 5717.91 | bwd_inner_microstep: 5690.61 | bwd_allreduce_microstep: 27.25 | step_microstep: 18.81
[2025-04-26 06:13:26,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.39 | bwd: 5717.93 | bwd_inner: 5690.61 | bwd_allreduce: 27.27 | step: 18.81

  2%|▏         | 860/41250 [2:04:57<97:27:32,  8.69s/it]
                                                        
{'loss': 0.1304, 'grad_norm': 1.727473258972168, 'learning_rate': 2.7786752827140553e-05, 'epoch': 0.21}

  2%|▏         | 860/41250 [2:04:57<97:27:32,  8.69s/it][2025-04-26 06:13:34,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:13:34,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5707.87 | bwd_inner_microstep: 5694.97 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.95
[2025-04-26 06:13:34,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5707.88 | bwd_inner: 5694.97 | bwd_allreduce: 12.87 | step: 18.95

  2%|▏         | 861/41250 [2:05:06<97:17:59,  8.67s/it]
                                                        
{'loss': 0.1574, 'grad_norm': 1.2979527711868286, 'learning_rate': 2.7819063004846527e-05, 'epoch': 0.21}

  2%|▏         | 861/41250 [2:05:06<97:17:59,  8.67s/it][2025-04-26 06:13:43,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.19 | optimizer_step: 1.01
[2025-04-26 06:13:43,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.37 | bwd_microstep: 5760.49 | bwd_inner_microstep: 5692.95 | bwd_allreduce_microstep: 67.49 | step_microstep: 19.32
[2025-04-26 06:13:43,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.37 | bwd: 5760.50 | bwd_inner: 5692.95 | bwd_allreduce: 67.51 | step: 19.33

  2%|▏         | 862/41250 [2:05:14<97:20:20,  8.68s/it]
                                                        
{'loss': 0.3494, 'grad_norm': 3.7537589073181152, 'learning_rate': 2.7851373182552505e-05, 'epoch': 0.21}

  2%|▏         | 862/41250 [2:05:14<97:20:20,  8.68s/it][2025-04-26 06:13:52,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.11 | optimizer_step: 1.02
[2025-04-26 06:13:52,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5764.15 | bwd_inner_microstep: 5662.75 | bwd_allreduce_microstep: 101.35 | step_microstep: 19.23
[2025-04-26 06:13:52,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5764.16 | bwd_inner: 5662.75 | bwd_allreduce: 101.37 | step: 19.24

  2%|▏         | 863/41250 [2:05:23<97:20:45,  8.68s/it]
                                                        
{'loss': 0.2289, 'grad_norm': 3.0370235443115234, 'learning_rate': 2.7883683360258483e-05, 'epoch': 0.21}

  2%|▏         | 863/41250 [2:05:23<97:20:45,  8.68s/it][2025-04-26 06:14:00,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 06:14:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.25 | bwd_microstep: 5783.17 | bwd_inner_microstep: 5770.22 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.02
[2025-04-26 06:14:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.25 | bwd: 5783.19 | bwd_inner: 5770.22 | bwd_allreduce: 12.92 | step: 19.03

  2%|▏         | 864/41250 [2:05:32<97:37:04,  8.70s/it]
                                                        
{'loss': 0.3452, 'grad_norm': 2.799731731414795, 'learning_rate': 2.7915993537964465e-05, 'epoch': 0.21}

  2%|▏         | 864/41250 [2:05:32<97:37:04,  8.70s/it][2025-04-26 06:14:09,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 06:14:09,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.39 | bwd_microstep: 5759.46 | bwd_inner_microstep: 5709.01 | bwd_allreduce_microstep: 50.39 | step_microstep: 18.77
[2025-04-26 06:14:09,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.39 | bwd: 5759.47 | bwd_inner: 5709.01 | bwd_allreduce: 50.41 | step: 18.77

  2%|▏         | 865/41250 [2:05:40<97:34:42,  8.70s/it]
                                                        
{'loss': 0.2362, 'grad_norm': 1.7658227682113647, 'learning_rate': 2.7948303715670436e-05, 'epoch': 0.21}

  2%|▏         | 865/41250 [2:05:40<97:34:42,  8.70s/it][2025-04-26 06:14:18,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:14:18,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.64 | bwd_microstep: 5795.21 | bwd_inner_microstep: 5648.20 | bwd_allreduce_microstep: 146.96 | step_microstep: 18.51
[2025-04-26 06:14:18,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.64 | bwd: 5795.22 | bwd_inner: 5648.20 | bwd_allreduce: 146.98 | step: 18.52

  2%|▏         | 866/41250 [2:05:49<97:32:47,  8.70s/it]
                                                        
{'loss': 0.202, 'grad_norm': 1.7123923301696777, 'learning_rate': 2.7980613893376417e-05, 'epoch': 0.21}

  2%|▏         | 866/41250 [2:05:49<97:32:47,  8.70s/it][2025-04-26 06:14:27,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 06:14:27,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.94 | bwd_microstep: 5733.44 | bwd_inner_microstep: 5706.48 | bwd_allreduce_microstep: 26.92 | step_microstep: 19.05
[2025-04-26 06:14:27,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.94 | bwd: 5733.45 | bwd_inner: 5706.48 | bwd_allreduce: 26.93 | step: 19.06

  2%|▏         | 867/41250 [2:05:58<97:28:31,  8.69s/it]
                                                        
{'loss': 0.1247, 'grad_norm': 1.6210565567016602, 'learning_rate': 2.8012924071082395e-05, 'epoch': 0.21}

  2%|▏         | 867/41250 [2:05:58<97:28:31,  8.69s/it][2025-04-26 06:14:35,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 06:14:35,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.15 | bwd_microstep: 5780.77 | bwd_inner_microstep: 5650.44 | bwd_allreduce_microstep: 130.28 | step_microstep: 18.88
[2025-04-26 06:14:35,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.15 | bwd: 5780.78 | bwd_inner: 5650.44 | bwd_allreduce: 130.30 | step: 18.88

  2%|▏         | 868/41250 [2:06:06<97:27:27,  8.69s/it]
                                                        
{'loss': 0.1833, 'grad_norm': 1.6950420141220093, 'learning_rate': 2.804523424878837e-05, 'epoch': 0.21}

  2%|▏         | 868/41250 [2:06:06<97:27:27,  8.69s/it][2025-04-26 06:14:44,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 06:14:44,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5722.20 | bwd_inner_microstep: 5709.40 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.00
[2025-04-26 06:14:44,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5722.22 | bwd_inner: 5709.40 | bwd_allreduce: 12.77 | step: 19.00

  2%|▏         | 869/41250 [2:06:15<97:22:06,  8.68s/it]
                                                        
{'loss': 0.128, 'grad_norm': 1.4088863134384155, 'learning_rate': 2.8077544426494347e-05, 'epoch': 0.21}

  2%|▏         | 869/41250 [2:06:15<97:22:06,  8.68s/it][2025-04-26 06:14:53,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 06:14:53,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.99 | bwd_microstep: 5778.98 | bwd_inner_microstep: 5639.59 | bwd_allreduce_microstep: 139.35 | step_microstep: 18.75
[2025-04-26 06:14:53,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.99 | bwd: 5778.99 | bwd_inner: 5639.59 | bwd_allreduce: 139.37 | step: 18.75

  2%|▏         | 870/41250 [2:06:24<97:21:53,  8.68s/it]
                                                        
{'loss': 0.0821, 'grad_norm': 1.0651425123214722, 'learning_rate': 2.8109854604200325e-05, 'epoch': 0.21}

  2%|▏         | 870/41250 [2:06:24<97:21:53,  8.68s/it][2025-04-26 06:15:01,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 06:15:01,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.14 | bwd_microstep: 5768.61 | bwd_inner_microstep: 5686.79 | bwd_allreduce_microstep: 81.78 | step_microstep: 18.74
[2025-04-26 06:15:01,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.14 | bwd: 5768.63 | bwd_inner: 5686.79 | bwd_allreduce: 81.79 | step: 18.74

  2%|▏         | 871/41250 [2:06:32<97:22:55,  8.68s/it]
                                                        
{'loss': 0.0514, 'grad_norm': 0.4999789893627167, 'learning_rate': 2.8142164781906306e-05, 'epoch': 0.21}

  2%|▏         | 871/41250 [2:06:32<97:22:55,  8.68s/it][2025-04-26 06:15:10,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 06:15:10,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.04 | bwd_microstep: 5771.07 | bwd_inner_microstep: 5652.35 | bwd_allreduce_microstep: 118.68 | step_microstep: 18.91
[2025-04-26 06:15:10,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.04 | bwd: 5771.08 | bwd_inner: 5652.35 | bwd_allreduce: 118.69 | step: 18.91

  2%|▏         | 872/41250 [2:06:41<97:22:39,  8.68s/it]
                                                        
{'loss': 0.2264, 'grad_norm': 2.4175684452056885, 'learning_rate': 2.8174474959612277e-05, 'epoch': 0.21}

  2%|▏         | 872/41250 [2:06:41<97:22:39,  8.68s/it][2025-04-26 06:15:19,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:15:19,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.30 | bwd_microstep: 5735.01 | bwd_inner_microstep: 5673.12 | bwd_allreduce_microstep: 61.84 | step_microstep: 18.28
[2025-04-26 06:15:19,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.30 | bwd: 5735.02 | bwd_inner: 5673.12 | bwd_allreduce: 61.86 | step: 18.29

  2%|▏         | 873/41250 [2:06:50<97:18:36,  8.68s/it]
                                                        
{'loss': 0.3055, 'grad_norm': 2.9253547191619873, 'learning_rate': 2.820678513731826e-05, 'epoch': 0.21}

  2%|▏         | 873/41250 [2:06:50<97:18:36,  8.68s/it][2025-04-26 06:15:27,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 06:15:27,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.47 | bwd_microstep: 5923.48 | bwd_inner_microstep: 5643.62 | bwd_allreduce_microstep: 279.81 | step_microstep: 18.63
[2025-04-26 06:15:27,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.47 | bwd: 5923.49 | bwd_inner: 5643.62 | bwd_allreduce: 279.83 | step: 18.63

  2%|▏         | 874/41250 [2:06:59<97:49:36,  8.72s/it]
                                                        
{'loss': 0.0782, 'grad_norm': 1.1281582117080688, 'learning_rate': 2.8239095315024237e-05, 'epoch': 0.21}

  2%|▏         | 874/41250 [2:06:59<97:49:36,  8.72s/it][2025-04-26 06:15:36,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.99
[2025-04-26 06:15:36,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.73 | bwd_microstep: 5731.07 | bwd_inner_microstep: 5689.56 | bwd_allreduce_microstep: 41.46 | step_microstep: 18.48
[2025-04-26 06:15:36,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.73 | bwd: 5731.08 | bwd_inner: 5689.56 | bwd_allreduce: 41.48 | step: 18.48

  2%|▏         | 875/41250 [2:07:07<97:33:52,  8.70s/it]
                                                        
{'loss': 0.2129, 'grad_norm': 3.0012388229370117, 'learning_rate': 2.827140549273021e-05, 'epoch': 0.21}

  2%|▏         | 875/41250 [2:07:07<97:33:52,  8.70s/it][2025-04-26 06:15:45,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:15:45,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.47 | bwd_microstep: 5729.01 | bwd_inner_microstep: 5687.67 | bwd_allreduce_microstep: 41.30 | step_microstep: 18.45
[2025-04-26 06:15:45,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.47 | bwd: 5729.03 | bwd_inner: 5687.67 | bwd_allreduce: 41.31 | step: 18.45

  2%|▏         | 876/41250 [2:07:16<97:24:29,  8.69s/it]
                                                        
{'loss': 0.0994, 'grad_norm': 1.3044463396072388, 'learning_rate': 2.830371567043619e-05, 'epoch': 0.21}

  2%|▏         | 876/41250 [2:07:16<97:24:29,  8.69s/it][2025-04-26 06:15:53,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.07
[2025-04-26 06:15:53,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5716.76 | bwd_inner_microstep: 5683.25 | bwd_allreduce_microstep: 33.45 | step_microstep: 19.57
[2025-04-26 06:15:53,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5716.77 | bwd_inner: 5683.25 | bwd_allreduce: 33.48 | step: 19.57

  2%|▏         | 877/41250 [2:07:25<97:17:46,  8.68s/it]
                                                        
{'loss': 0.194, 'grad_norm': 1.9591940641403198, 'learning_rate': 2.8336025848142167e-05, 'epoch': 0.21}

  2%|▏         | 877/41250 [2:07:25<97:17:46,  8.68s/it][2025-04-26 06:16:02,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 06:16:02,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.98 | bwd_microstep: 5728.47 | bwd_inner_microstep: 5698.38 | bwd_allreduce_microstep: 30.04 | step_microstep: 18.84
[2025-04-26 06:16:02,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.98 | bwd: 5728.49 | bwd_inner: 5698.38 | bwd_allreduce: 30.06 | step: 18.84

  2%|▏         | 878/41250 [2:07:33<97:14:20,  8.67s/it]
                                                        
{'loss': 0.1096, 'grad_norm': 1.8695381879806519, 'learning_rate': 2.8368336025848148e-05, 'epoch': 0.21}

  2%|▏         | 878/41250 [2:07:33<97:14:20,  8.67s/it][2025-04-26 06:16:11,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:16:11,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.87 | bwd_microstep: 5732.51 | bwd_inner_microstep: 5678.07 | bwd_allreduce_microstep: 54.39 | step_microstep: 19.01
[2025-04-26 06:16:11,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.87 | bwd: 5732.52 | bwd_inner: 5678.07 | bwd_allreduce: 54.41 | step: 19.01

  2%|▏         | 879/41250 [2:07:42<97:13:17,  8.67s/it]
                                                        
{'loss': 0.167, 'grad_norm': 2.32963228225708, 'learning_rate': 2.840064620355412e-05, 'epoch': 0.21}

  2%|▏         | 879/41250 [2:07:42<97:13:17,  8.67s/it][2025-04-26 06:16:19,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 06:16:19,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.72 | bwd_microstep: 5710.50 | bwd_inner_microstep: 5697.84 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.15
[2025-04-26 06:16:19,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.72 | bwd: 5710.51 | bwd_inner: 5697.83 | bwd_allreduce: 12.63 | step: 19.15

  2%|▏         | 880/41250 [2:07:51<97:08:07,  8.66s/it]
                                                        
{'loss': 0.3626, 'grad_norm': 2.5328941345214844, 'learning_rate': 2.84329563812601e-05, 'epoch': 0.21}

  2%|▏         | 880/41250 [2:07:51<97:08:07,  8.66s/it][2025-04-26 06:16:28,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.33 | optimizer_step: 0.94
[2025-04-26 06:16:28,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.39 | bwd_microstep: 5871.13 | bwd_inner_microstep: 5651.06 | bwd_allreduce_microstep: 220.01 | step_microstep: 20.07
[2025-04-26 06:16:28,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.39 | bwd: 5871.15 | bwd_inner: 5651.06 | bwd_allreduce: 220.04 | step: 20.07

  2%|▏         | 881/41250 [2:07:59<97:32:57,  8.70s/it]
                                                        
{'loss': 0.1104, 'grad_norm': 1.7304257154464722, 'learning_rate': 2.846526655896608e-05, 'epoch': 0.21}

  2%|▏         | 881/41250 [2:07:59<97:32:57,  8.70s/it][2025-04-26 06:16:37,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:16:37,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.50 | bwd_microstep: 5788.43 | bwd_inner_microstep: 5775.54 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.87
[2025-04-26 06:16:37,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.50 | bwd: 5788.45 | bwd_inner: 5775.54 | bwd_allreduce: 12.86 | step: 18.87

  2%|▏         | 882/41250 [2:08:08<97:44:44,  8.72s/it]
                                                        
{'loss': 0.2146, 'grad_norm': 3.410471200942993, 'learning_rate': 2.8497576736672053e-05, 'epoch': 0.21}

  2%|▏         | 882/41250 [2:08:08<97:44:44,  8.72s/it][2025-04-26 06:16:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:16:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5773.49 | bwd_inner_microstep: 5653.47 | bwd_allreduce_microstep: 119.98 | step_microstep: 18.68
[2025-04-26 06:16:46,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5773.50 | bwd_inner: 5653.47 | bwd_allreduce: 120.00 | step: 18.69

  2%|▏         | 883/41250 [2:08:17<97:38:26,  8.71s/it]
                                                        
{'loss': 0.3062, 'grad_norm': 3.1754868030548096, 'learning_rate': 2.852988691437803e-05, 'epoch': 0.21}

  2%|▏         | 883/41250 [2:08:17<97:38:26,  8.71s/it][2025-04-26 06:16:54,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.01
[2025-04-26 06:16:54,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5772.80 | bwd_inner_microstep: 5654.48 | bwd_allreduce_microstep: 118.26 | step_microstep: 19.53
[2025-04-26 06:16:54,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5772.82 | bwd_inner: 5654.48 | bwd_allreduce: 118.29 | step: 19.53

  2%|▏         | 884/41250 [2:08:25<97:31:08,  8.70s/it]
                                                        
{'loss': 0.3699, 'grad_norm': 2.6383142471313477, 'learning_rate': 2.856219709208401e-05, 'epoch': 0.21}

  2%|▏         | 884/41250 [2:08:25<97:31:08,  8.70s/it][2025-04-26 06:17:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 06:17:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5724.35 | bwd_inner_microstep: 5703.96 | bwd_allreduce_microstep: 20.34 | step_microstep: 18.85
[2025-04-26 06:17:03,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5724.36 | bwd_inner: 5703.96 | bwd_allreduce: 20.36 | step: 18.85

  2%|▏         | 885/41250 [2:08:34<97:23:20,  8.69s/it]
                                                        
{'loss': 0.2643, 'grad_norm': 3.1991612911224365, 'learning_rate': 2.859450726978999e-05, 'epoch': 0.21}

  2%|▏         | 885/41250 [2:08:34<97:23:20,  8.69s/it][2025-04-26 06:17:12,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 06:17:12,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 6011.04 | bwd_inner_microstep: 5657.41 | bwd_allreduce_microstep: 353.59 | step_microstep: 18.66
[2025-04-26 06:17:12,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 6011.06 | bwd_inner: 5657.41 | bwd_allreduce: 353.61 | step: 18.67

  2%|▏         | 886/41250 [2:08:43<98:10:32,  8.76s/it]
                                                        
{'loss': 0.1814, 'grad_norm': 4.853121280670166, 'learning_rate': 2.862681744749596e-05, 'epoch': 0.21}

  2%|▏         | 886/41250 [2:08:43<98:10:32,  8.76s/it][2025-04-26 06:17:21,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:17:21,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.18 | bwd_microstep: 5771.17 | bwd_inner_microstep: 5758.15 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.88
[2025-04-26 06:17:21,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.18 | bwd: 5771.18 | bwd_inner: 5758.15 | bwd_allreduce: 12.99 | step: 18.88

  2%|▏         | 887/41250 [2:08:52<98:07:19,  8.75s/it]
                                                        
{'loss': 0.1837, 'grad_norm': 1.724569320678711, 'learning_rate': 2.8659127625201942e-05, 'epoch': 0.22}

  2%|▏         | 887/41250 [2:08:52<98:07:19,  8.75s/it][2025-04-26 06:17:29,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:17:29,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.89 | bwd_microstep: 5692.19 | bwd_inner_microstep: 5651.64 | bwd_allreduce_microstep: 40.50 | step_microstep: 18.52
[2025-04-26 06:17:29,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.89 | bwd: 5692.20 | bwd_inner: 5651.64 | bwd_allreduce: 40.52 | step: 18.53

  2%|▏         | 888/41250 [2:09:00<97:38:04,  8.71s/it]
                                                        
{'loss': 0.3173, 'grad_norm': 4.8466715812683105, 'learning_rate': 2.869143780290792e-05, 'epoch': 0.22}

  2%|▏         | 888/41250 [2:09:00<97:38:04,  8.71s/it][2025-04-26 06:17:38,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 06:17:38,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.48 | bwd_microstep: 5724.32 | bwd_inner_microstep: 5711.58 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.02
[2025-04-26 06:17:38,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.48 | bwd: 5724.34 | bwd_inner: 5711.58 | bwd_allreduce: 12.72 | step: 19.02

  2%|▏         | 889/41250 [2:09:09<97:26:06,  8.69s/it]
                                                        
{'loss': 0.2905, 'grad_norm': 2.302147150039673, 'learning_rate': 2.8723747980613895e-05, 'epoch': 0.22}

  2%|▏         | 889/41250 [2:09:09<97:26:06,  8.69s/it][2025-04-26 06:17:46,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 06:17:46,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5732.51 | bwd_inner_microstep: 5692.22 | bwd_allreduce_microstep: 40.25 | step_microstep: 19.13
[2025-04-26 06:17:46,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5732.53 | bwd_inner: 5692.22 | bwd_allreduce: 40.27 | step: 19.13

  2%|▏         | 890/41250 [2:09:18<97:21:08,  8.68s/it]
                                                        
{'loss': 0.2697, 'grad_norm': 2.411968231201172, 'learning_rate': 2.8756058158319873e-05, 'epoch': 0.22}

  2%|▏         | 890/41250 [2:09:18<97:21:08,  8.68s/it][2025-04-26 06:17:55,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:17:55,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.45 | bwd_microstep: 5707.46 | bwd_inner_microstep: 5652.14 | bwd_allreduce_microstep: 55.28 | step_microstep: 18.59
[2025-04-26 06:17:55,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.45 | bwd: 5707.48 | bwd_inner: 5652.14 | bwd_allreduce: 55.30 | step: 18.60

  2%|▏         | 891/41250 [2:09:26<97:08:17,  8.66s/it]
                                                        
{'loss': 0.2684, 'grad_norm': 2.3818016052246094, 'learning_rate': 2.878836833602585e-05, 'epoch': 0.22}

  2%|▏         | 891/41250 [2:09:26<97:08:17,  8.66s/it][2025-04-26 06:18:04,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:18:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.19 | bwd_microstep: 5707.21 | bwd_inner_microstep: 5658.02 | bwd_allreduce_microstep: 49.15 | step_microstep: 18.58
[2025-04-26 06:18:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.19 | bwd: 5707.22 | bwd_inner: 5658.02 | bwd_allreduce: 49.17 | step: 18.59

  2%|▏         | 892/41250 [2:09:35<96:58:24,  8.65s/it]
                                                        
{'loss': 0.2551, 'grad_norm': 1.8501570224761963, 'learning_rate': 2.8820678513731825e-05, 'epoch': 0.22}

  2%|▏         | 892/41250 [2:09:35<96:58:24,  8.65s/it][2025-04-26 06:18:12,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:18:12,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5719.96 | bwd_inner_microstep: 5707.23 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.78
[2025-04-26 06:18:12,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5719.97 | bwd_inner: 5707.23 | bwd_allreduce: 12.70 | step: 18.78

  2%|▏         | 893/41250 [2:09:44<96:57:03,  8.65s/it]
                                                        
{'loss': 0.0714, 'grad_norm': 1.023518681526184, 'learning_rate': 2.8852988691437803e-05, 'epoch': 0.22}

  2%|▏         | 893/41250 [2:09:44<96:57:03,  8.65s/it][2025-04-26 06:18:21,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-26 06:18:21,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.37 | bwd_microstep: 5743.84 | bwd_inner_microstep: 5696.93 | bwd_allreduce_microstep: 46.86 | step_microstep: 19.41
[2025-04-26 06:18:21,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.37 | bwd: 5743.85 | bwd_inner: 5696.93 | bwd_allreduce: 46.88 | step: 19.42

  2%|▏         | 894/41250 [2:09:52<97:02:50,  8.66s/it]
                                                        
{'loss': 0.2506, 'grad_norm': 2.7828495502471924, 'learning_rate': 2.8885298869143784e-05, 'epoch': 0.22}

  2%|▏         | 894/41250 [2:09:52<97:02:50,  8.66s/it][2025-04-26 06:18:30,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:18:30,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5705.84 | bwd_inner_microstep: 5692.99 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.87
[2025-04-26 06:18:30,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5705.86 | bwd_inner: 5692.99 | bwd_allreduce: 12.83 | step: 18.87

  2%|▏         | 895/41250 [2:10:01<96:58:19,  8.65s/it]
                                                        
{'loss': 0.3098, 'grad_norm': 2.60901141166687, 'learning_rate': 2.8917609046849762e-05, 'epoch': 0.22}

  2%|▏         | 895/41250 [2:10:01<96:58:19,  8.65s/it][2025-04-26 06:18:38,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:18:38,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.26 | bwd_microstep: 5712.83 | bwd_inner_microstep: 5700.05 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.74
[2025-04-26 06:18:38,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.26 | bwd: 5712.84 | bwd_inner: 5700.05 | bwd_allreduce: 12.74 | step: 18.75

  2%|▏         | 896/41250 [2:10:10<96:56:08,  8.65s/it]
                                                        
{'loss': 0.1549, 'grad_norm': 2.3267502784729004, 'learning_rate': 2.8949919224555736e-05, 'epoch': 0.22}

  2%|▏         | 896/41250 [2:10:10<96:56:08,  8.65s/it][2025-04-26 06:18:47,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 06:18:47,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.42 | bwd_microstep: 5894.48 | bwd_inner_microstep: 5653.28 | bwd_allreduce_microstep: 241.15 | step_microstep: 18.80
[2025-04-26 06:18:47,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.42 | bwd: 5894.49 | bwd_inner: 5653.28 | bwd_allreduce: 241.17 | step: 18.80

  2%|▏         | 897/41250 [2:10:18<97:29:15,  8.70s/it]
                                                        
{'loss': 0.254, 'grad_norm': 2.191443920135498, 'learning_rate': 2.8982229402261714e-05, 'epoch': 0.22}

  2%|▏         | 897/41250 [2:10:18<97:29:15,  8.70s/it][2025-04-26 06:18:56,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 06:18:56,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5711.50 | bwd_inner_microstep: 5698.73 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.97
[2025-04-26 06:18:56,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5711.51 | bwd_inner: 5698.73 | bwd_allreduce: 12.74 | step: 18.97

  2%|▏         | 898/41250 [2:10:27<97:19:02,  8.68s/it]
                                                        
{'loss': 0.1714, 'grad_norm': 1.4007741212844849, 'learning_rate': 2.9014539579967692e-05, 'epoch': 0.22}

  2%|▏         | 898/41250 [2:10:27<97:19:02,  8.68s/it][2025-04-26 06:19:04,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.10 | optimizer_step: 1.08
[2025-04-26 06:19:04,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5779.18 | bwd_inner_microstep: 5687.44 | bwd_allreduce_microstep: 91.68 | step_microstep: 19.48
[2025-04-26 06:19:04,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5779.19 | bwd_inner: 5687.44 | bwd_allreduce: 91.71 | step: 19.48

  2%|▏         | 899/41250 [2:10:36<97:25:11,  8.69s/it]
                                                        
{'loss': 0.1115, 'grad_norm': 2.1200942993164062, 'learning_rate': 2.9046849757673667e-05, 'epoch': 0.22}

  2%|▏         | 899/41250 [2:10:36<97:25:11,  8.69s/it][2025-04-26 06:19:13,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 06:19:13,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.71 | bwd_microstep: 5713.75 | bwd_inner_microstep: 5655.56 | bwd_allreduce_microstep: 58.14 | step_microstep: 18.55
[2025-04-26 06:19:13,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.71 | bwd: 5713.76 | bwd_inner: 5655.56 | bwd_allreduce: 58.15 | step: 18.55

  2%|▏         | 900/41250 [2:10:44<97:11:56,  8.67s/it]
                                                        
{'loss': 0.2434, 'grad_norm': 2.451662540435791, 'learning_rate': 2.9079159935379645e-05, 'epoch': 0.22}

  2%|▏         | 900/41250 [2:10:44<97:11:56,  8.67s/it][2025-04-26 06:19:22,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.33 | optimizer_step: 1.04
[2025-04-26 06:19:22,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5795.94 | bwd_inner_microstep: 5707.98 | bwd_allreduce_microstep: 87.89 | step_microstep: 19.62
[2025-04-26 06:19:22,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5795.95 | bwd_inner: 5707.98 | bwd_allreduce: 87.92 | step: 19.62

  2%|▏         | 901/41250 [2:10:53<97:23:42,  8.69s/it]
                                                        
{'loss': 0.3843, 'grad_norm': 2.14646577835083, 'learning_rate': 2.9111470113085626e-05, 'epoch': 0.22}

  2%|▏         | 901/41250 [2:10:53<97:23:42,  8.69s/it][2025-04-26 06:19:30,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 06:19:30,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.98 | bwd_microstep: 5716.50 | bwd_inner_microstep: 5655.76 | bwd_allreduce_microstep: 60.69 | step_microstep: 18.96
[2025-04-26 06:19:30,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.98 | bwd: 5716.51 | bwd_inner: 5655.76 | bwd_allreduce: 60.71 | step: 18.97

  2%|▏         | 902/41250 [2:11:02<97:10:09,  8.67s/it]
                                                        
{'loss': 0.2049, 'grad_norm': 2.4240052700042725, 'learning_rate': 2.9143780290791604e-05, 'epoch': 0.22}

  2%|▏         | 902/41250 [2:11:02<97:10:09,  8.67s/it][2025-04-26 06:19:39,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:19:39,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.31 | bwd_microstep: 5751.40 | bwd_inner_microstep: 5712.73 | bwd_allreduce_microstep: 38.62 | step_microstep: 18.80
[2025-04-26 06:19:39,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.31 | bwd: 5751.42 | bwd_inner: 5712.73 | bwd_allreduce: 38.64 | step: 18.80

  2%|▏         | 903/41250 [2:11:10<97:13:09,  8.67s/it]
                                                        
{'loss': 0.1332, 'grad_norm': 2.008817672729492, 'learning_rate': 2.9176090468497578e-05, 'epoch': 0.22}

  2%|▏         | 903/41250 [2:11:10<97:13:09,  8.67s/it][2025-04-26 06:19:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 06:19:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.44 | bwd_microstep: 5787.22 | bwd_inner_microstep: 5670.01 | bwd_allreduce_microstep: 117.16 | step_microstep: 19.00
[2025-04-26 06:19:48,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.44 | bwd: 5787.24 | bwd_inner: 5670.01 | bwd_allreduce: 117.18 | step: 19.01

  2%|▏         | 904/41250 [2:11:19<97:18:13,  8.68s/it]
                                                        
{'loss': 0.2814, 'grad_norm': 2.9014477729797363, 'learning_rate': 2.9208400646203556e-05, 'epoch': 0.22}

  2%|▏         | 904/41250 [2:11:19<97:18:13,  8.68s/it][2025-04-26 06:19:57,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-26 06:19:57,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.39 | bwd_microstep: 5774.34 | bwd_inner_microstep: 5689.36 | bwd_allreduce_microstep: 84.93 | step_microstep: 19.16
[2025-04-26 06:19:57,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.39 | bwd: 5774.36 | bwd_inner: 5689.36 | bwd_allreduce: 84.96 | step: 19.17

  2%|▏         | 905/41250 [2:11:28<97:25:05,  8.69s/it]
                                                        
{'loss': 0.2578, 'grad_norm': 1.509824514389038, 'learning_rate': 2.9240710823909534e-05, 'epoch': 0.22}

  2%|▏         | 905/41250 [2:11:28<97:25:05,  8.69s/it][2025-04-26 06:20:05,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 06:20:05,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.27 | bwd_microstep: 5770.33 | bwd_inner_microstep: 5657.44 | bwd_allreduce_microstep: 112.85 | step_microstep: 19.01
[2025-04-26 06:20:05,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.27 | bwd: 5770.34 | bwd_inner: 5657.44 | bwd_allreduce: 112.87 | step: 19.01

  2%|▏         | 906/41250 [2:11:36<97:26:43,  8.70s/it]
                                                        
{'loss': 0.1929, 'grad_norm': 1.9285846948623657, 'learning_rate': 2.927302100161551e-05, 'epoch': 0.22}

  2%|▏         | 906/41250 [2:11:36<97:26:43,  8.70s/it][2025-04-26 06:20:14,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:20:14,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.11 | bwd_microstep: 5788.45 | bwd_inner_microstep: 5640.81 | bwd_allreduce_microstep: 147.60 | step_microstep: 18.53
[2025-04-26 06:20:14,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.11 | bwd: 5788.47 | bwd_inner: 5640.81 | bwd_allreduce: 147.61 | step: 18.53

  2%|▏         | 907/41250 [2:11:45<97:26:50,  8.70s/it]
                                                        
{'loss': 0.1468, 'grad_norm': 2.7953011989593506, 'learning_rate': 2.9305331179321486e-05, 'epoch': 0.22}

  2%|▏         | 907/41250 [2:11:45<97:26:50,  8.70s/it][2025-04-26 06:20:23,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:20:23,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.40 | bwd_microstep: 5711.04 | bwd_inner_microstep: 5697.93 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.55
[2025-04-26 06:20:23,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.40 | bwd: 5711.05 | bwd_inner: 5697.93 | bwd_allreduce: 13.09 | step: 18.55

  2%|▏         | 908/41250 [2:11:54<97:17:34,  8.68s/it]
                                                        
{'loss': 0.2416, 'grad_norm': 2.0522892475128174, 'learning_rate': 2.9337641357027468e-05, 'epoch': 0.22}

  2%|▏         | 908/41250 [2:11:54<97:17:34,  8.68s/it][2025-04-26 06:20:31,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-26 06:20:31,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.69 | bwd_microstep: 5708.32 | bwd_inner_microstep: 5695.28 | bwd_allreduce_microstep: 12.99 | step_microstep: 19.18
[2025-04-26 06:20:31,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.69 | bwd: 5708.33 | bwd_inner: 5695.28 | bwd_allreduce: 13.01 | step: 19.18

  2%|▏         | 909/41250 [2:12:02<97:10:22,  8.67s/it]
                                                        
{'loss': 0.2322, 'grad_norm': 2.8971896171569824, 'learning_rate': 2.9369951534733446e-05, 'epoch': 0.22}

  2%|▏         | 909/41250 [2:12:02<97:10:22,  8.67s/it][2025-04-26 06:20:40,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 06:20:40,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.22 | bwd_microstep: 5704.50 | bwd_inner_microstep: 5691.69 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.80
[2025-04-26 06:20:40,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.22 | bwd: 5704.51 | bwd_inner: 5691.69 | bwd_allreduce: 12.78 | step: 18.80

  2%|▏         | 910/41250 [2:12:11<97:02:50,  8.66s/it]
                                                        
{'loss': 0.2237, 'grad_norm': 1.8137422800064087, 'learning_rate': 2.940226171243942e-05, 'epoch': 0.22}

  2%|▏         | 910/41250 [2:12:11<97:02:50,  8.66s/it][2025-04-26 06:20:49,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:20:49,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.66 | bwd_microstep: 5693.18 | bwd_inner_microstep: 5680.54 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.99
[2025-04-26 06:20:49,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.66 | bwd: 5693.19 | bwd_inner: 5680.54 | bwd_allreduce: 12.62 | step: 17.99

  2%|▏         | 911/41250 [2:12:20<96:53:40,  8.65s/it]
                                                        
{'loss': 0.1741, 'grad_norm': 1.8354504108428955, 'learning_rate': 2.9434571890145398e-05, 'epoch': 0.22}

  2%|▏         | 911/41250 [2:12:20<96:53:40,  8.65s/it][2025-04-26 06:20:57,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 06:20:57,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.25 | bwd_microstep: 5752.15 | bwd_inner_microstep: 5640.60 | bwd_allreduce_microstep: 111.50 | step_microstep: 18.87
[2025-04-26 06:20:57,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.25 | bwd: 5752.16 | bwd_inner: 5640.60 | bwd_allreduce: 111.52 | step: 18.87

  2%|▏         | 912/41250 [2:12:28<96:56:16,  8.65s/it]
                                                        
{'loss': 0.1911, 'grad_norm': 1.867508888244629, 'learning_rate': 2.9466882067851376e-05, 'epoch': 0.22}

  2%|▏         | 912/41250 [2:12:28<96:56:16,  8.65s/it][2025-04-26 06:21:06,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.30 | optimizer_step: 1.03
[2025-04-26 06:21:06,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.51 | bwd_microstep: 5742.46 | bwd_inner_microstep: 5680.67 | bwd_allreduce_microstep: 61.73 | step_microstep: 19.79
[2025-04-26 06:21:06,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.51 | bwd: 5742.48 | bwd_inner: 5680.67 | bwd_allreduce: 61.76 | step: 19.79

  2%|▏         | 913/41250 [2:12:37<97:00:18,  8.66s/it]
                                                        
{'loss': 0.2123, 'grad_norm': 2.4750497341156006, 'learning_rate': 2.949919224555735e-05, 'epoch': 0.22}

  2%|▏         | 913/41250 [2:12:37<97:00:18,  8.66s/it][2025-04-26 06:21:15,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 0.89
[2025-04-26 06:21:15,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.80 | bwd_microstep: 5795.90 | bwd_inner_microstep: 5782.50 | bwd_allreduce_microstep: 13.35 | step_microstep: 18.81
[2025-04-26 06:21:15,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.80 | bwd: 5795.91 | bwd_inner: 5782.50 | bwd_allreduce: 13.37 | step: 18.82

  2%|▏         | 914/41250 [2:12:46<97:21:58,  8.69s/it]
                                                        
{'loss': 0.2439, 'grad_norm': 2.337820529937744, 'learning_rate': 2.9531502423263328e-05, 'epoch': 0.22}

  2%|▏         | 914/41250 [2:12:46<97:21:58,  8.69s/it][2025-04-26 06:21:23,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 06:21:23,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.17 | bwd_microstep: 5772.02 | bwd_inner_microstep: 5759.20 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.32
[2025-04-26 06:21:23,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.17 | bwd: 5772.03 | bwd_inner: 5759.20 | bwd_allreduce: 12.79 | step: 19.32

  2%|▏         | 915/41250 [2:12:55<97:33:13,  8.71s/it]
                                                        
{'loss': 0.2372, 'grad_norm': 1.5864551067352295, 'learning_rate': 2.956381260096931e-05, 'epoch': 0.22}

  2%|▏         | 915/41250 [2:12:55<97:33:13,  8.71s/it][2025-04-26 06:21:32,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.18
[2025-04-26 06:21:32,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5912.92 | bwd_inner_microstep: 5640.26 | bwd_allreduce_microstep: 272.61 | step_microstep: 19.09
[2025-04-26 06:21:32,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5912.93 | bwd_inner: 5640.26 | bwd_allreduce: 272.63 | step: 19.09

  2%|▏         | 916/41250 [2:13:03<97:54:24,  8.74s/it]
                                                        
{'loss': 0.3409, 'grad_norm': 3.100518226623535, 'learning_rate': 2.9596122778675287e-05, 'epoch': 0.22}

  2%|▏         | 916/41250 [2:13:03<97:54:24,  8.74s/it][2025-04-26 06:21:41,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:21:41,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.85 | bwd_microstep: 5740.11 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 60.08 | step_microstep: 18.82
[2025-04-26 06:21:41,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.85 | bwd: 5740.12 | bwd_inner: 5679.98 | bwd_allreduce: 60.10 | step: 18.83

  2%|▏         | 917/41250 [2:13:12<97:38:59,  8.72s/it]
                                                        
{'loss': 0.2446, 'grad_norm': 1.582198977470398, 'learning_rate': 2.9628432956381262e-05, 'epoch': 0.22}

  2%|▏         | 917/41250 [2:13:12<97:38:59,  8.72s/it][2025-04-26 06:21:50,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 06:21:50,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2927.90 | bwd_microstep: 5873.71 | bwd_inner_microstep: 5860.77 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.64
[2025-04-26 06:21:50,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2927.90 | bwd: 5873.72 | bwd_inner: 5860.77 | bwd_allreduce: 12.91 | step: 19.64

  2%|▏         | 918/41250 [2:13:21<98:13:38,  8.77s/it]
                                                        
{'loss': 0.2322, 'grad_norm': 1.9905712604522705, 'learning_rate': 2.966074313408724e-05, 'epoch': 0.22}

  2%|▏         | 918/41250 [2:13:21<98:13:38,  8.77s/it][2025-04-26 06:21:59,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 06:21:59,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.53 | bwd_microstep: 5893.52 | bwd_inner_microstep: 5645.93 | bwd_allreduce_microstep: 247.54 | step_microstep: 18.92
[2025-04-26 06:21:59,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.53 | bwd: 5893.53 | bwd_inner: 5645.93 | bwd_allreduce: 247.56 | step: 18.93

  2%|▏         | 919/41250 [2:13:30<98:20:13,  8.78s/it]
                                                        
{'loss': 0.1543, 'grad_norm': 2.4563257694244385, 'learning_rate': 2.9693053311793218e-05, 'epoch': 0.22}

  2%|▏         | 919/41250 [2:13:30<98:20:13,  8.78s/it][2025-04-26 06:22:07,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:22:07,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.99 | bwd_microstep: 5696.80 | bwd_inner_microstep: 5683.78 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.69
[2025-04-26 06:22:07,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.99 | bwd: 5696.82 | bwd_inner: 5683.78 | bwd_allreduce: 12.99 | step: 18.70

  2%|▏         | 920/41250 [2:13:38<97:47:09,  8.73s/it]
                                                        
{'loss': 0.2272, 'grad_norm': 2.3742592334747314, 'learning_rate': 2.9725363489499192e-05, 'epoch': 0.22}

  2%|▏         | 920/41250 [2:13:38<97:47:09,  8.73s/it][2025-04-26 06:22:16,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 06:22:16,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.10 | bwd_microstep: 5873.56 | bwd_inner_microstep: 5648.11 | bwd_allreduce_microstep: 225.40 | step_microstep: 19.27
[2025-04-26 06:22:16,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.10 | bwd: 5873.57 | bwd_inner: 5648.11 | bwd_allreduce: 225.42 | step: 19.27

  2%|▏         | 921/41250 [2:13:47<97:57:29,  8.74s/it]
                                                        
{'loss': 0.2055, 'grad_norm': 2.4103243350982666, 'learning_rate': 2.975767366720517e-05, 'epoch': 0.22}

  2%|▏         | 921/41250 [2:13:47<97:57:29,  8.74s/it][2025-04-26 06:22:25,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.18 | optimizer_step: 0.93
[2025-04-26 06:22:25,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.30 | bwd_microstep: 5696.36 | bwd_inner_microstep: 5638.90 | bwd_allreduce_microstep: 57.41 | step_microstep: 19.35
[2025-04-26 06:22:25,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.30 | bwd: 5696.38 | bwd_inner: 5638.90 | bwd_allreduce: 57.43 | step: 19.35

  2%|▏         | 922/41250 [2:13:56<97:28:27,  8.70s/it]
                                                        
{'loss': 0.0773, 'grad_norm': 1.1652666330337524, 'learning_rate': 2.978998384491115e-05, 'epoch': 0.22}

  2%|▏         | 922/41250 [2:13:56<97:28:27,  8.70s/it][2025-04-26 06:22:33,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:22:33,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.46 | bwd_microstep: 5741.47 | bwd_inner_microstep: 5643.41 | bwd_allreduce_microstep: 98.00 | step_microstep: 18.65
[2025-04-26 06:22:33,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.46 | bwd: 5741.48 | bwd_inner: 5643.41 | bwd_allreduce: 98.02 | step: 18.66

  2%|▏         | 923/41250 [2:14:04<97:17:48,  8.69s/it]
                                                        
{'loss': 0.2338, 'grad_norm': 2.060788631439209, 'learning_rate': 2.982229402261713e-05, 'epoch': 0.22}

  2%|▏         | 923/41250 [2:14:04<97:17:48,  8.69s/it][2025-04-26 06:22:42,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:22:42,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5745.80 | bwd_inner_microstep: 5697.50 | bwd_allreduce_microstep: 48.26 | step_microstep: 18.59
[2025-04-26 06:22:42,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5745.81 | bwd_inner: 5697.50 | bwd_allreduce: 48.28 | step: 18.59

  2%|▏         | 924/41250 [2:14:13<97:15:55,  8.68s/it]
                                                        
{'loss': 0.114, 'grad_norm': 1.7216225862503052, 'learning_rate': 2.9854604200323104e-05, 'epoch': 0.22}

  2%|▏         | 924/41250 [2:14:13<97:15:55,  8.68s/it][2025-04-26 06:22:50,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:22:50,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.46 | bwd_microstep: 5687.60 | bwd_inner_microstep: 5634.16 | bwd_allreduce_microstep: 53.39 | step_microstep: 18.54
[2025-04-26 06:22:50,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.46 | bwd: 5687.61 | bwd_inner: 5634.16 | bwd_allreduce: 53.40 | step: 18.54

  2%|▏         | 925/41250 [2:14:22<96:56:13,  8.65s/it]
                                                        
{'loss': 0.3176, 'grad_norm': 2.858259916305542, 'learning_rate': 2.988691437802908e-05, 'epoch': 0.22}

  2%|▏         | 925/41250 [2:14:22<96:56:13,  8.65s/it][2025-04-26 06:22:59,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 06:22:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.84 | bwd_microstep: 5736.04 | bwd_inner_microstep: 5683.16 | bwd_allreduce_microstep: 52.83 | step_microstep: 18.72
[2025-04-26 06:22:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5736.05 | bwd_inner: 5683.16 | bwd_allreduce: 52.85 | step: 18.73

  2%|▏         | 926/41250 [2:14:30<96:57:11,  8.66s/it]
                                                        
{'loss': 0.2878, 'grad_norm': 3.8434979915618896, 'learning_rate': 2.991922455573506e-05, 'epoch': 0.22}

  2%|▏         | 926/41250 [2:14:30<96:57:11,  8.66s/it][2025-04-26 06:23:08,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 06:23:08,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5736.08 | bwd_inner_microstep: 5686.09 | bwd_allreduce_microstep: 49.94 | step_microstep: 18.58
[2025-04-26 06:23:08,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5736.09 | bwd_inner: 5686.09 | bwd_allreduce: 49.96 | step: 18.59

  2%|▏         | 927/41250 [2:14:39<96:59:42,  8.66s/it]
                                                        
{'loss': 0.1626, 'grad_norm': 1.2143423557281494, 'learning_rate': 2.9951534733441034e-05, 'epoch': 0.22}

  2%|▏         | 927/41250 [2:14:39<96:59:42,  8.66s/it][2025-04-26 06:23:16,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 06:23:16,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5713.63 | bwd_inner_microstep: 5700.63 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.88
[2025-04-26 06:23:16,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5713.65 | bwd_inner: 5700.63 | bwd_allreduce: 12.98 | step: 18.88

  2%|▏         | 928/41250 [2:14:48<96:55:44,  8.65s/it]
                                                        
{'loss': 0.0661, 'grad_norm': 0.6263192296028137, 'learning_rate': 2.9983844911147012e-05, 'epoch': 0.22}

  2%|▏         | 928/41250 [2:14:48<96:55:44,  8.65s/it][2025-04-26 06:23:25,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:23:25,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.80 | bwd_microstep: 5783.66 | bwd_inner_microstep: 5656.10 | bwd_allreduce_microstep: 127.51 | step_microstep: 18.49
[2025-04-26 06:23:25,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.80 | bwd: 5783.67 | bwd_inner: 5656.10 | bwd_allreduce: 127.53 | step: 18.50

  2%|▏         | 929/41250 [2:14:56<97:01:18,  8.66s/it]
                                                        
{'loss': 0.2435, 'grad_norm': 3.5060296058654785, 'learning_rate': 3.0016155088852993e-05, 'epoch': 0.23}

  2%|▏         | 929/41250 [2:14:56<97:01:18,  8.66s/it][2025-04-26 06:23:34,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:23:34,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.79 | bwd_microstep: 5788.05 | bwd_inner_microstep: 5650.88 | bwd_allreduce_microstep: 137.13 | step_microstep: 18.54
[2025-04-26 06:23:34,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.79 | bwd: 5788.07 | bwd_inner: 5650.88 | bwd_allreduce: 137.15 | step: 18.55

  2%|▏         | 930/41250 [2:15:05<97:07:04,  8.67s/it]
                                                        
{'loss': 0.1927, 'grad_norm': 2.4832894802093506, 'learning_rate': 3.004846526655897e-05, 'epoch': 0.23}

  2%|▏         | 930/41250 [2:15:05<97:07:04,  8.67s/it][2025-04-26 06:23:42,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 1.11
[2025-04-26 06:23:42,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.22 | bwd_microstep: 5684.63 | bwd_inner_microstep: 5671.78 | bwd_allreduce_microstep: 12.81 | step_microstep: 19.14
[2025-04-26 06:23:42,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.22 | bwd: 5684.65 | bwd_inner: 5671.78 | bwd_allreduce: 12.83 | step: 19.14

  2%|▏         | 931/41250 [2:15:14<96:54:45,  8.65s/it]
                                                        
{'loss': 0.2256, 'grad_norm': 2.7092981338500977, 'learning_rate': 3.0080775444264945e-05, 'epoch': 0.23}

  2%|▏         | 931/41250 [2:15:14<96:54:45,  8.65s/it][2025-04-26 06:23:51,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:23:51,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.91 | bwd_microstep: 5804.52 | bwd_inner_microstep: 5653.75 | bwd_allreduce_microstep: 150.73 | step_microstep: 18.62
[2025-04-26 06:23:51,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5804.54 | bwd_inner: 5653.75 | bwd_allreduce: 150.75 | step: 18.62

  2%|▏         | 932/41250 [2:15:22<97:06:51,  8.67s/it]
                                                        
{'loss': 0.1063, 'grad_norm': 1.165115237236023, 'learning_rate': 3.0113085621970923e-05, 'epoch': 0.23}

  2%|▏         | 932/41250 [2:15:22<97:06:51,  8.67s/it][2025-04-26 06:24:00,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:24:00,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5777.17 | bwd_inner_microstep: 5698.31 | bwd_allreduce_microstep: 78.81 | step_microstep: 18.34
[2025-04-26 06:24:00,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5777.19 | bwd_inner: 5698.31 | bwd_allreduce: 78.83 | step: 18.34

  2%|▏         | 933/41250 [2:15:31<97:14:22,  8.68s/it]
                                                        
{'loss': 0.1344, 'grad_norm': 2.1479902267456055, 'learning_rate': 3.01453957996769e-05, 'epoch': 0.23}

  2%|▏         | 933/41250 [2:15:31<97:14:22,  8.68s/it][2025-04-26 06:24:09,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:24:09,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5764.35 | bwd_inner_microstep: 5687.09 | bwd_allreduce_microstep: 77.22 | step_microstep: 18.42
[2025-04-26 06:24:09,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5764.36 | bwd_inner: 5687.09 | bwd_allreduce: 77.23 | step: 18.42

  2%|▏         | 934/41250 [2:15:40<97:15:36,  8.68s/it]
                                                        
{'loss': 0.2764, 'grad_norm': 2.485644817352295, 'learning_rate': 3.0177705977382876e-05, 'epoch': 0.23}

  2%|▏         | 934/41250 [2:15:40<97:15:36,  8.68s/it][2025-04-26 06:24:17,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:24:17,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5719.71 | bwd_inner_microstep: 5646.59 | bwd_allreduce_microstep: 73.07 | step_microstep: 18.62
[2025-04-26 06:24:17,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5719.73 | bwd_inner: 5646.59 | bwd_allreduce: 73.10 | step: 18.63

  2%|▏         | 935/41250 [2:15:48<97:05:36,  8.67s/it]
                                                        
{'loss': 0.0988, 'grad_norm': 1.4924607276916504, 'learning_rate': 3.0210016155088854e-05, 'epoch': 0.23}

  2%|▏         | 935/41250 [2:15:48<97:05:36,  8.67s/it][2025-04-26 06:24:26,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 06:24:26,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.20 | bwd_microstep: 5783.66 | bwd_inner_microstep: 5652.69 | bwd_allreduce_microstep: 130.92 | step_microstep: 18.72
[2025-04-26 06:24:26,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.20 | bwd: 5783.67 | bwd_inner: 5652.69 | bwd_allreduce: 130.93 | step: 18.72

  2%|▏         | 936/41250 [2:15:57<97:09:33,  8.68s/it]
                                                        
{'loss': 0.0658, 'grad_norm': 0.6384297609329224, 'learning_rate': 3.0242326332794835e-05, 'epoch': 0.23}

  2%|▏         | 936/41250 [2:15:57<97:09:33,  8.68s/it][2025-04-26 06:24:34,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.14
[2025-04-26 06:24:34,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.03 | bwd_microstep: 5710.65 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.42
[2025-04-26 06:24:34,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.03 | bwd: 5710.67 | bwd_inner: 5697.91 | bwd_allreduce: 12.71 | step: 19.42

  2%|▏         | 937/41250 [2:16:06<97:03:26,  8.67s/it]
                                                        
{'loss': 0.2584, 'grad_norm': 2.0422747135162354, 'learning_rate': 3.0274636510500813e-05, 'epoch': 0.23}

  2%|▏         | 937/41250 [2:16:06<97:03:26,  8.67s/it][2025-04-26 06:24:43,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 06:24:43,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.30 | bwd_microstep: 5778.16 | bwd_inner_microstep: 5654.20 | bwd_allreduce_microstep: 123.91 | step_microstep: 19.29
[2025-04-26 06:24:43,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.30 | bwd: 5778.17 | bwd_inner: 5654.20 | bwd_allreduce: 123.93 | step: 19.29

  2%|▏         | 938/41250 [2:16:14<97:06:33,  8.67s/it]
                                                        
{'loss': 0.1436, 'grad_norm': 1.7104593515396118, 'learning_rate': 3.0306946688206787e-05, 'epoch': 0.23}

  2%|▏         | 938/41250 [2:16:14<97:06:33,  8.67s/it][2025-04-26 06:24:52,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:24:52,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.96 | bwd_microstep: 5791.66 | bwd_inner_microstep: 5650.95 | bwd_allreduce_microstep: 140.66 | step_microstep: 18.88
[2025-04-26 06:24:52,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.96 | bwd: 5791.68 | bwd_inner: 5650.95 | bwd_allreduce: 140.68 | step: 18.88

  2%|▏         | 939/41250 [2:16:23<97:13:10,  8.68s/it]
                                                        
{'loss': 0.2402, 'grad_norm': 2.3501570224761963, 'learning_rate': 3.0339256865912765e-05, 'epoch': 0.23}

  2%|▏         | 939/41250 [2:16:23<97:13:10,  8.68s/it][2025-04-26 06:25:01,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:25:01,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5715.83 | bwd_inner_microstep: 5702.79 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.95
[2025-04-26 06:25:01,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5715.85 | bwd_inner: 5702.79 | bwd_allreduce: 13.01 | step: 18.95

  2%|▏         | 940/41250 [2:16:32<97:06:35,  8.67s/it]
                                                        
{'loss': 0.455, 'grad_norm': 3.4390454292297363, 'learning_rate': 3.0371567043618743e-05, 'epoch': 0.23}

  2%|▏         | 940/41250 [2:16:32<97:06:35,  8.67s/it][2025-04-26 06:25:09,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:25:09,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.52 | bwd_microstep: 5784.43 | bwd_inner_microstep: 5651.47 | bwd_allreduce_microstep: 132.91 | step_microstep: 18.72
[2025-04-26 06:25:09,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.52 | bwd: 5784.44 | bwd_inner: 5651.47 | bwd_allreduce: 132.93 | step: 18.72

  2%|▏         | 941/41250 [2:16:40<97:13:40,  8.68s/it]
                                                        
{'loss': 0.2595, 'grad_norm': 3.7595112323760986, 'learning_rate': 3.0403877221324718e-05, 'epoch': 0.23}

  2%|▏         | 941/41250 [2:16:40<97:13:40,  8.68s/it][2025-04-26 06:25:18,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 06:25:18,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.94 | bwd_microstep: 5791.46 | bwd_inner_microstep: 5657.82 | bwd_allreduce_microstep: 133.59 | step_microstep: 19.22
[2025-04-26 06:25:18,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.94 | bwd: 5791.47 | bwd_inner: 5657.82 | bwd_allreduce: 133.61 | step: 19.22

  2%|▏         | 942/41250 [2:16:49<97:17:29,  8.69s/it]
                                                        
{'loss': 0.0958, 'grad_norm': 1.4487712383270264, 'learning_rate': 3.0436187399030695e-05, 'epoch': 0.23}

  2%|▏         | 942/41250 [2:16:49<97:17:29,  8.69s/it][2025-04-26 06:25:27,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:25:27,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2906.58 | bwd_microstep: 5793.56 | bwd_inner_microstep: 5780.65 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.57
[2025-04-26 06:25:27,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2906.58 | bwd: 5793.58 | bwd_inner: 5780.65 | bwd_allreduce: 12.88 | step: 18.58

  2%|▏         | 943/41250 [2:16:58<97:35:00,  8.72s/it]
                                                        
{'loss': 0.2968, 'grad_norm': 1.9207115173339844, 'learning_rate': 3.0468497576736677e-05, 'epoch': 0.23}

  2%|▏         | 943/41250 [2:16:58<97:35:00,  8.72s/it][2025-04-26 06:25:35,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 06:25:35,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.37 | bwd_microstep: 5716.00 | bwd_inner_microstep: 5649.90 | bwd_allreduce_microstep: 66.05 | step_microstep: 19.28
[2025-04-26 06:25:35,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.37 | bwd: 5716.02 | bwd_inner: 5649.90 | bwd_allreduce: 66.08 | step: 19.28

  2%|▏         | 944/41250 [2:17:07<97:18:06,  8.69s/it]
                                                        
{'loss': 0.153, 'grad_norm': 1.6802077293395996, 'learning_rate': 3.0500807754442655e-05, 'epoch': 0.23}

  2%|▏         | 944/41250 [2:17:07<97:18:06,  8.69s/it][2025-04-26 06:25:44,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.11 | optimizer_step: 1.05
[2025-04-26 06:25:44,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.87 | bwd_microstep: 5725.29 | bwd_inner_microstep: 5645.37 | bwd_allreduce_microstep: 79.86 | step_microstep: 19.91
[2025-04-26 06:25:44,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.87 | bwd: 5725.31 | bwd_inner: 5645.37 | bwd_allreduce: 79.89 | step: 19.91

  2%|▏         | 945/41250 [2:17:15<97:06:54,  8.67s/it]
                                                        
{'loss': 0.106, 'grad_norm': 1.1537426710128784, 'learning_rate': 3.0533117932148626e-05, 'epoch': 0.23}

  2%|▏         | 945/41250 [2:17:15<97:06:54,  8.67s/it][2025-04-26 06:25:53,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.92
[2025-04-26 06:25:53,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.70 | bwd_microstep: 5702.47 | bwd_inner_microstep: 5646.99 | bwd_allreduce_microstep: 55.43 | step_microstep: 19.17
[2025-04-26 06:25:53,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.70 | bwd: 5702.48 | bwd_inner: 5646.99 | bwd_allreduce: 55.45 | step: 19.18

  2%|▏         | 946/41250 [2:17:24<96:54:25,  8.66s/it]
                                                        
{'loss': 0.1983, 'grad_norm': 2.246731758117676, 'learning_rate': 3.0565428109854604e-05, 'epoch': 0.23}

  2%|▏         | 946/41250 [2:17:24<96:54:25,  8.66s/it][2025-04-26 06:26:01,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 06:26:01,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.01 | bwd_microstep: 5693.89 | bwd_inner_microstep: 5658.52 | bwd_allreduce_microstep: 35.32 | step_microstep: 18.76
[2025-04-26 06:26:01,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.01 | bwd: 5693.91 | bwd_inner: 5658.52 | bwd_allreduce: 35.34 | step: 18.77

  2%|▏         | 947/41250 [2:17:32<96:42:29,  8.64s/it]
                                                        
{'loss': 0.1642, 'grad_norm': 1.869489312171936, 'learning_rate': 3.059773828756059e-05, 'epoch': 0.23}

  2%|▏         | 947/41250 [2:17:32<96:42:29,  8.64s/it][2025-04-26 06:26:10,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-26 06:26:10,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.89 | bwd_microstep: 5777.29 | bwd_inner_microstep: 5709.68 | bwd_allreduce_microstep: 67.56 | step_microstep: 19.08
[2025-04-26 06:26:10,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.89 | bwd: 5777.30 | bwd_inner: 5709.68 | bwd_allreduce: 67.58 | step: 19.09

  2%|▏         | 948/41250 [2:17:41<96:56:27,  8.66s/it]
                                                        
{'loss': 0.3617, 'grad_norm': 2.4827563762664795, 'learning_rate': 3.063004846526656e-05, 'epoch': 0.23}

  2%|▏         | 948/41250 [2:17:41<96:56:27,  8.66s/it][2025-04-26 06:26:19,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 06:26:19,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.63 | bwd_microstep: 5703.09 | bwd_inner_microstep: 5690.46 | bwd_allreduce_microstep: 12.59 | step_microstep: 19.31
[2025-04-26 06:26:19,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.63 | bwd: 5703.11 | bwd_inner: 5690.46 | bwd_allreduce: 12.61 | step: 19.31

  2%|▏         | 949/41250 [2:17:50<96:51:14,  8.65s/it]
                                                        
{'loss': 0.1436, 'grad_norm': 2.459615707397461, 'learning_rate': 3.066235864297254e-05, 'epoch': 0.23}

  2%|▏         | 949/41250 [2:17:50<96:51:14,  8.65s/it][2025-04-26 06:26:27,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:26:27,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2880.35 | bwd_microstep: 5781.70 | bwd_inner_microstep: 5768.82 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.80
[2025-04-26 06:26:27,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2880.35 | bwd: 5781.71 | bwd_inner: 5768.82 | bwd_allreduce: 12.85 | step: 18.80

  2%|▏         | 950/41250 [2:17:58<97:09:54,  8.68s/it]
                                                        
{'loss': 0.1827, 'grad_norm': 1.894355058670044, 'learning_rate': 3.0694668820678515e-05, 'epoch': 0.23}

  2%|▏         | 950/41250 [2:17:58<97:09:54,  8.68s/it][2025-04-26 06:26:36,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:26:36,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.75 | bwd_microstep: 5753.10 | bwd_inner_microstep: 5656.89 | bwd_allreduce_microstep: 96.16 | step_microstep: 19.02
[2025-04-26 06:26:36,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.75 | bwd: 5753.11 | bwd_inner: 5656.89 | bwd_allreduce: 96.18 | step: 19.02

  2%|▏         | 951/41250 [2:18:07<97:06:37,  8.68s/it]
                                                        
{'loss': 0.1029, 'grad_norm': 1.5247774124145508, 'learning_rate': 3.072697899838449e-05, 'epoch': 0.23}

  2%|▏         | 951/41250 [2:18:07<97:06:37,  8.68s/it][2025-04-26 06:26:45,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 1.11 | optimizer_step: 1.20
[2025-04-26 06:26:45,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.11 | bwd_microstep: 5751.58 | bwd_inner_microstep: 5706.26 | bwd_allreduce_microstep: 45.25 | step_microstep: 19.85
[2025-04-26 06:26:45,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.11 | bwd: 5751.59 | bwd_inner: 5706.26 | bwd_allreduce: 45.28 | step: 19.86

  2%|▏         | 952/41250 [2:18:16<97:08:42,  8.68s/it]
                                                        
{'loss': 0.1733, 'grad_norm': 2.47945499420166, 'learning_rate': 3.075928917609047e-05, 'epoch': 0.23}

  2%|▏         | 952/41250 [2:18:16<97:08:42,  8.68s/it][2025-04-26 06:26:53,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 0.93
[2025-04-26 06:26:53,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.06 | bwd_microstep: 5766.41 | bwd_inner_microstep: 5683.94 | bwd_allreduce_microstep: 82.42 | step_microstep: 19.22
[2025-04-26 06:26:53,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.06 | bwd: 5766.42 | bwd_inner: 5683.94 | bwd_allreduce: 82.44 | step: 19.22

  2%|▏         | 953/41250 [2:18:25<97:13:21,  8.69s/it]
                                                        
{'loss': 0.1395, 'grad_norm': 2.7221200466156006, 'learning_rate': 3.079159935379645e-05, 'epoch': 0.23}

  2%|▏         | 953/41250 [2:18:25<97:13:21,  8.69s/it][2025-04-26 06:27:02,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 06:27:02,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.99 | bwd_microstep: 5737.53 | bwd_inner_microstep: 5691.29 | bwd_allreduce_microstep: 46.18 | step_microstep: 19.00
[2025-04-26 06:27:02,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.99 | bwd: 5737.54 | bwd_inner: 5691.29 | bwd_allreduce: 46.21 | step: 19.00

  2%|▏         | 954/41250 [2:18:33<97:09:39,  8.68s/it]
                                                        
{'loss': 0.3968, 'grad_norm': 3.2627687454223633, 'learning_rate': 3.082390953150243e-05, 'epoch': 0.23}

  2%|▏         | 954/41250 [2:18:33<97:09:39,  8.68s/it][2025-04-26 06:27:11,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.35 | optimizer_step: 0.90
[2025-04-26 06:27:11,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.56 | bwd_microstep: 5699.63 | bwd_inner_microstep: 5686.82 | bwd_allreduce_microstep: 12.76 | step_microstep: 20.03
[2025-04-26 06:27:11,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.56 | bwd: 5699.65 | bwd_inner: 5686.82 | bwd_allreduce: 12.78 | step: 20.04

  2%|▏         | 955/41250 [2:18:42<96:58:46,  8.66s/it]
                                                        
{'loss': 0.0411, 'grad_norm': 0.5845869779586792, 'learning_rate': 3.0856219709208405e-05, 'epoch': 0.23}

  2%|▏         | 955/41250 [2:18:42<96:58:46,  8.66s/it][2025-04-26 06:27:19,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-26 06:27:19,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5711.30 | bwd_inner_microstep: 5697.95 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.49
[2025-04-26 06:27:19,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5711.32 | bwd_inner: 5697.95 | bwd_allreduce: 13.32 | step: 19.50

  2%|▏         | 956/41250 [2:18:50<96:52:16,  8.65s/it]
                                                        
{'loss': 0.523, 'grad_norm': 3.7805261611938477, 'learning_rate': 3.088852988691438e-05, 'epoch': 0.23}

  2%|▏         | 956/41250 [2:18:50<96:52:16,  8.65s/it][2025-04-26 06:27:28,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 06:27:28,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.86 | bwd_microstep: 5768.72 | bwd_inner_microstep: 5755.93 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.86
[2025-04-26 06:27:28,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.86 | bwd: 5768.73 | bwd_inner: 5755.93 | bwd_allreduce: 12.76 | step: 18.86

  2%|▏         | 957/41250 [2:18:59<97:08:31,  8.68s/it]
                                                        
{'loss': 0.1687, 'grad_norm': 2.3021886348724365, 'learning_rate': 3.092084006462036e-05, 'epoch': 0.23}

  2%|▏         | 957/41250 [2:18:59<97:08:31,  8.68s/it][2025-04-26 06:27:37,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:27:37,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5703.14 | bwd_inner_microstep: 5690.41 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.82
[2025-04-26 06:27:37,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5703.16 | bwd_inner: 5690.41 | bwd_allreduce: 12.71 | step: 18.83

  2%|▏         | 958/41250 [2:19:08<96:58:32,  8.66s/it]
                                                        
{'loss': 0.1605, 'grad_norm': 3.3394150733947754, 'learning_rate': 3.095315024232634e-05, 'epoch': 0.23}

  2%|▏         | 958/41250 [2:19:08<96:58:32,  8.66s/it][2025-04-26 06:27:45,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 06:27:45,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.51 | bwd_microstep: 5723.01 | bwd_inner_microstep: 5706.57 | bwd_allreduce_microstep: 16.39 | step_microstep: 19.15
[2025-04-26 06:27:45,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.51 | bwd: 5723.03 | bwd_inner: 5706.57 | bwd_allreduce: 16.41 | step: 19.15

  2%|▏         | 959/41250 [2:19:16<96:58:28,  8.66s/it]
                                                        
{'loss': 0.0629, 'grad_norm': 1.099761724472046, 'learning_rate': 3.098546042003231e-05, 'epoch': 0.23}

  2%|▏         | 959/41250 [2:19:16<96:58:28,  8.66s/it][2025-04-26 06:27:54,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 06:27:54,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.28 | bwd_microstep: 5684.29 | bwd_inner_microstep: 5648.02 | bwd_allreduce_microstep: 36.23 | step_microstep: 18.93
[2025-04-26 06:27:54,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.28 | bwd: 5684.30 | bwd_inner: 5648.02 | bwd_allreduce: 36.25 | step: 18.93

  2%|▏         | 960/41250 [2:19:25<96:43:18,  8.64s/it]
                                                        
{'loss': 0.054, 'grad_norm': 1.0721105337142944, 'learning_rate': 3.101777059773829e-05, 'epoch': 0.23}

  2%|▏         | 960/41250 [2:19:25<96:43:18,  8.64s/it][2025-04-26 06:28:02,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.07
[2025-04-26 06:28:02,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.46 | bwd_microstep: 5703.41 | bwd_inner_microstep: 5637.87 | bwd_allreduce_microstep: 65.50 | step_microstep: 19.55
[2025-04-26 06:28:02,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.46 | bwd: 5703.43 | bwd_inner: 5637.87 | bwd_allreduce: 65.52 | step: 19.55

  2%|▏         | 961/41250 [2:19:34<96:35:46,  8.63s/it]
                                                        
{'loss': 0.3162, 'grad_norm': 3.669105052947998, 'learning_rate': 3.105008077544427e-05, 'epoch': 0.23}

  2%|▏         | 961/41250 [2:19:34<96:35:46,  8.63s/it][2025-04-26 06:28:11,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.10
[2025-04-26 06:28:11,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.21 | bwd_microstep: 5696.29 | bwd_inner_microstep: 5652.55 | bwd_allreduce_microstep: 43.69 | step_microstep: 19.38
[2025-04-26 06:28:11,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.21 | bwd: 5696.30 | bwd_inner: 5652.55 | bwd_allreduce: 43.70 | step: 19.38

  2%|▏         | 962/41250 [2:19:42<96:31:04,  8.62s/it]
                                                        
{'loss': 0.3148, 'grad_norm': 1.9386731386184692, 'learning_rate': 3.108239095315024e-05, 'epoch': 0.23}

  2%|▏         | 962/41250 [2:19:42<96:31:04,  8.62s/it][2025-04-26 06:28:20,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.10 | optimizer_step: 1.04
[2025-04-26 06:28:20,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5717.84 | bwd_inner_microstep: 5704.45 | bwd_allreduce_microstep: 13.33 | step_microstep: 19.38
[2025-04-26 06:28:20,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5717.85 | bwd_inner: 5704.45 | bwd_allreduce: 13.35 | step: 19.38

  2%|▏         | 963/41250 [2:19:51<96:37:47,  8.63s/it]
                                                        
{'loss': 0.2152, 'grad_norm': 1.7981808185577393, 'learning_rate': 3.111470113085622e-05, 'epoch': 0.23}

  2%|▏         | 963/41250 [2:19:51<96:37:47,  8.63s/it][2025-04-26 06:28:29,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 06:28:29,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5876.88 | bwd_inner_microstep: 5652.86 | bwd_allreduce_microstep: 223.98 | step_microstep: 19.70
[2025-04-26 06:28:29,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5876.90 | bwd_inner: 5652.86 | bwd_allreduce: 224.00 | step: 19.70

  2%|▏         | 964/41250 [2:20:00<97:08:41,  8.68s/it]
                                                        
{'loss': 0.2446, 'grad_norm': 2.032871723175049, 'learning_rate': 3.11470113085622e-05, 'epoch': 0.23}

  2%|▏         | 964/41250 [2:20:00<97:08:41,  8.68s/it][2025-04-26 06:28:37,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:28:37,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.64 | bwd_microstep: 5780.20 | bwd_inner_microstep: 5654.53 | bwd_allreduce_microstep: 125.62 | step_microstep: 18.71
[2025-04-26 06:28:37,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.64 | bwd: 5780.21 | bwd_inner: 5654.53 | bwd_allreduce: 125.64 | step: 18.71

  2%|▏         | 965/41250 [2:20:08<97:08:04,  8.68s/it]
                                                        
{'loss': 0.1343, 'grad_norm': 1.6340726613998413, 'learning_rate': 3.1179321486268177e-05, 'epoch': 0.23}

  2%|▏         | 965/41250 [2:20:08<97:08:04,  8.68s/it][2025-04-26 06:28:46,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 06:28:46,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.39 | bwd_microstep: 5694.40 | bwd_inner_microstep: 5681.70 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.05
[2025-04-26 06:28:46,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.39 | bwd: 5694.42 | bwd_inner: 5681.70 | bwd_allreduce: 12.67 | step: 19.06

  2%|▏         | 966/41250 [2:20:17<96:56:13,  8.66s/it]
                                                        
{'loss': 0.4111, 'grad_norm': 2.703547954559326, 'learning_rate': 3.1211631663974154e-05, 'epoch': 0.23}

  2%|▏         | 966/41250 [2:20:17<96:56:13,  8.66s/it][2025-04-26 06:28:54,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.27 | optimizer_step: 0.94
[2025-04-26 06:28:54,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.75 | bwd_microstep: 5732.92 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 79.37 | step_microstep: 19.38
[2025-04-26 06:28:54,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.75 | bwd: 5732.94 | bwd_inner: 5653.50 | bwd_allreduce: 79.39 | step: 19.38

  2%|▏         | 967/41250 [2:20:26<96:54:03,  8.66s/it]
                                                        
{'loss': 0.1372, 'grad_norm': 1.861405849456787, 'learning_rate': 3.124394184168013e-05, 'epoch': 0.23}

  2%|▏         | 967/41250 [2:20:26<96:54:03,  8.66s/it][2025-04-26 06:29:03,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 06:29:03,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.26 | bwd_microstep: 5764.74 | bwd_inner_microstep: 5653.59 | bwd_allreduce_microstep: 111.10 | step_microstep: 18.68
[2025-04-26 06:29:03,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.26 | bwd: 5764.75 | bwd_inner: 5653.59 | bwd_allreduce: 111.12 | step: 18.69

  2%|▏         | 968/41250 [2:20:34<96:57:37,  8.67s/it]
                                                        
{'loss': 0.2365, 'grad_norm': 2.0165364742279053, 'learning_rate': 3.127625201938611e-05, 'epoch': 0.23}

  2%|▏         | 968/41250 [2:20:34<96:57:37,  8.67s/it][2025-04-26 06:29:12,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:29:12,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.97 | bwd_microstep: 5863.48 | bwd_inner_microstep: 5636.35 | bwd_allreduce_microstep: 227.08 | step_microstep: 18.70
[2025-04-26 06:29:12,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.97 | bwd: 5863.50 | bwd_inner: 5636.35 | bwd_allreduce: 227.11 | step: 18.71

  2%|▏         | 969/41250 [2:20:43<97:18:56,  8.70s/it]
                                                        
{'loss': 0.1074, 'grad_norm': 1.4070611000061035, 'learning_rate': 3.130856219709209e-05, 'epoch': 0.23}

  2%|▏         | 969/41250 [2:20:43<97:18:56,  8.70s/it][2025-04-26 06:29:21,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.17 | optimizer_step: 1.05
[2025-04-26 06:29:21,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.07 | bwd_microstep: 5758.33 | bwd_inner_microstep: 5641.95 | bwd_allreduce_microstep: 116.33 | step_microstep: 20.42
[2025-04-26 06:29:21,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.07 | bwd: 5758.34 | bwd_inner: 5641.95 | bwd_allreduce: 116.35 | step: 20.43

  2%|▏         | 970/41250 [2:20:52<97:13:33,  8.69s/it]
                                                        
{'loss': 0.1248, 'grad_norm': 1.8651736974716187, 'learning_rate': 3.1340872374798066e-05, 'epoch': 0.24}

  2%|▏         | 970/41250 [2:20:52<97:13:33,  8.69s/it][2025-04-26 06:29:29,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:29:29,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.18 | bwd_microstep: 5742.32 | bwd_inner_microstep: 5683.96 | bwd_allreduce_microstep: 58.31 | step_microstep: 18.74
[2025-04-26 06:29:29,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.19 | bwd: 5742.33 | bwd_inner: 5683.96 | bwd_allreduce: 58.33 | step: 18.74

  2%|▏         | 971/41250 [2:21:00<97:10:45,  8.69s/it]
                                                        
{'loss': 0.0696, 'grad_norm': 2.0586864948272705, 'learning_rate': 3.1373182552504044e-05, 'epoch': 0.24}

  2%|▏         | 971/41250 [2:21:00<97:10:45,  8.69s/it][2025-04-26 06:29:38,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:29:38,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5756.21 | bwd_inner_microstep: 5651.03 | bwd_allreduce_microstep: 105.13 | step_microstep: 18.52
[2025-04-26 06:29:38,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5756.22 | bwd_inner: 5651.03 | bwd_allreduce: 105.15 | step: 18.53

  2%|▏         | 972/41250 [2:21:09<97:06:32,  8.68s/it]
                                                        
{'loss': 0.1015, 'grad_norm': 1.3338041305541992, 'learning_rate': 3.1405492730210015e-05, 'epoch': 0.24}

  2%|▏         | 972/41250 [2:21:09<97:06:32,  8.68s/it][2025-04-26 06:29:47,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-26 06:29:47,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5748.02 | bwd_inner_microstep: 5697.66 | bwd_allreduce_microstep: 50.32 | step_microstep: 18.65
[2025-04-26 06:29:47,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5748.03 | bwd_inner: 5697.66 | bwd_allreduce: 50.33 | step: 18.65

  2%|▏         | 973/41250 [2:21:18<97:06:29,  8.68s/it]
                                                        
{'loss': 0.0919, 'grad_norm': 1.0131126642227173, 'learning_rate': 3.143780290791599e-05, 'epoch': 0.24}

  2%|▏         | 973/41250 [2:21:18<97:06:29,  8.68s/it][2025-04-26 06:29:55,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:29:55,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.76 | bwd_microstep: 5909.49 | bwd_inner_microstep: 5655.95 | bwd_allreduce_microstep: 253.49 | step_microstep: 18.76
[2025-04-26 06:29:55,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.76 | bwd: 5909.50 | bwd_inner: 5655.95 | bwd_allreduce: 253.51 | step: 18.76

  2%|▏         | 974/41250 [2:21:27<97:31:49,  8.72s/it]
                                                        
{'loss': 0.1656, 'grad_norm': 2.5995895862579346, 'learning_rate': 3.147011308562197e-05, 'epoch': 0.24}

  2%|▏         | 974/41250 [2:21:27<97:31:49,  8.72s/it][2025-04-26 06:30:04,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 06:30:04,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.76 | bwd_microstep: 5797.08 | bwd_inner_microstep: 5784.28 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.11
[2025-04-26 06:30:04,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.76 | bwd: 5797.09 | bwd_inner: 5784.28 | bwd_allreduce: 12.77 | step: 19.11

  2%|▏         | 975/41250 [2:21:35<97:42:19,  8.73s/it]
                                                        
{'loss': 0.4045, 'grad_norm': 2.7461588382720947, 'learning_rate': 3.1502423263327955e-05, 'epoch': 0.24}

  2%|▏         | 975/41250 [2:21:35<97:42:19,  8.73s/it][2025-04-26 06:30:13,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 06:30:13,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.19 | bwd_microstep: 5880.29 | bwd_inner_microstep: 5681.01 | bwd_allreduce_microstep: 199.22 | step_microstep: 19.08
[2025-04-26 06:30:13,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.19 | bwd: 5880.30 | bwd_inner: 5681.01 | bwd_allreduce: 199.24 | step: 19.08

  2%|▏         | 976/41250 [2:21:44<97:57:44,  8.76s/it]
                                                        
{'loss': 0.1337, 'grad_norm': 1.371973991394043, 'learning_rate': 3.1534733441033927e-05, 'epoch': 0.24}

  2%|▏         | 976/41250 [2:21:44<97:57:44,  8.76s/it][2025-04-26 06:30:22,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 06:30:22,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.02 | bwd_microstep: 5728.90 | bwd_inner_microstep: 5715.22 | bwd_allreduce_microstep: 13.63 | step_microstep: 19.30
[2025-04-26 06:30:22,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.02 | bwd: 5728.92 | bwd_inner: 5715.22 | bwd_allreduce: 13.66 | step: 19.30

  2%|▏         | 977/41250 [2:21:53<97:40:41,  8.73s/it]
                                                        
{'loss': 0.1662, 'grad_norm': 1.555888295173645, 'learning_rate': 3.1567043618739904e-05, 'epoch': 0.24}

  2%|▏         | 977/41250 [2:21:53<97:40:41,  8.73s/it][2025-04-26 06:30:30,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 06:30:30,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.53 | bwd_microstep: 5775.19 | bwd_inner_microstep: 5647.50 | bwd_allreduce_microstep: 127.64 | step_microstep: 18.62
[2025-04-26 06:30:30,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.53 | bwd: 5775.21 | bwd_inner: 5647.50 | bwd_allreduce: 127.66 | step: 18.62

  2%|▏         | 978/41250 [2:22:02<97:31:39,  8.72s/it]
                                                        
{'loss': 0.1375, 'grad_norm': 2.179661989212036, 'learning_rate': 3.159935379644588e-05, 'epoch': 0.24}

  2%|▏         | 978/41250 [2:22:02<97:31:39,  8.72s/it][2025-04-26 06:30:39,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 06:30:39,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5713.82 | bwd_inner_microstep: 5660.38 | bwd_allreduce_microstep: 53.39 | step_microstep: 18.55
[2025-04-26 06:30:39,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.74 | bwd: 5713.83 | bwd_inner: 5660.38 | bwd_allreduce: 53.41 | step: 18.55

  2%|▏         | 979/41250 [2:22:10<97:12:00,  8.69s/it]
                                                        
{'loss': 0.1816, 'grad_norm': 1.5766977071762085, 'learning_rate': 3.163166397415186e-05, 'epoch': 0.24}

  2%|▏         | 979/41250 [2:22:10<97:12:00,  8.69s/it][2025-04-26 06:30:48,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 06:30:48,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.75 | bwd_microstep: 5701.76 | bwd_inner_microstep: 5671.54 | bwd_allreduce_microstep: 30.17 | step_microstep: 18.59
[2025-04-26 06:30:48,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.75 | bwd: 5701.77 | bwd_inner: 5671.54 | bwd_allreduce: 30.19 | step: 18.60

  2%|▏         | 980/41250 [2:22:19<96:58:59,  8.67s/it]
                                                        
{'loss': 0.2322, 'grad_norm': 1.9759634733200073, 'learning_rate': 3.166397415185784e-05, 'epoch': 0.24}

  2%|▏         | 980/41250 [2:22:19<96:58:59,  8.67s/it][2025-04-26 06:30:56,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.25
[2025-04-26 06:30:56,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.93 | bwd_microstep: 5711.22 | bwd_inner_microstep: 5697.95 | bwd_allreduce_microstep: 13.22 | step_microstep: 19.96
[2025-04-26 06:30:56,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.93 | bwd: 5711.24 | bwd_inner: 5697.95 | bwd_allreduce: 13.24 | step: 19.96

  2%|▏         | 981/41250 [2:22:27<96:54:41,  8.66s/it]
                                                        
{'loss': 0.1246, 'grad_norm': 1.160552978515625, 'learning_rate': 3.1696284329563816e-05, 'epoch': 0.24}

  2%|▏         | 981/41250 [2:22:27<96:54:41,  8.66s/it][2025-04-26 06:31:05,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 06:31:05,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.34 | bwd_microstep: 5776.62 | bwd_inner_microstep: 5659.57 | bwd_allreduce_microstep: 117.01 | step_microstep: 18.71
[2025-04-26 06:31:05,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.34 | bwd: 5776.64 | bwd_inner: 5659.57 | bwd_allreduce: 117.02 | step: 18.72

  2%|▏         | 982/41250 [2:22:36<96:58:33,  8.67s/it]
                                                        
{'loss': 0.0373, 'grad_norm': 0.4461626410484314, 'learning_rate': 3.1728594507269794e-05, 'epoch': 0.24}

  2%|▏         | 982/41250 [2:22:36<96:58:33,  8.67s/it][2025-04-26 06:31:14,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 06:31:14,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.36 | bwd_microstep: 5790.58 | bwd_inner_microstep: 5646.90 | bwd_allreduce_microstep: 143.63 | step_microstep: 18.89
[2025-04-26 06:31:14,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.37 | bwd: 5790.60 | bwd_inner: 5646.90 | bwd_allreduce: 143.65 | step: 18.89

  2%|▏         | 983/41250 [2:22:45<97:03:02,  8.68s/it]
                                                        
{'loss': 0.3945, 'grad_norm': 3.437462091445923, 'learning_rate': 3.176090468497577e-05, 'epoch': 0.24}

  2%|▏         | 983/41250 [2:22:45<97:03:02,  8.68s/it][2025-04-26 06:31:22,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:31:22,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.32 | bwd_microstep: 5724.43 | bwd_inner_microstep: 5648.62 | bwd_allreduce_microstep: 75.76 | step_microstep: 18.77
[2025-04-26 06:31:22,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.32 | bwd: 5724.45 | bwd_inner: 5648.62 | bwd_allreduce: 75.78 | step: 18.77

  2%|▏         | 984/41250 [2:22:53<96:53:47,  8.66s/it]
                                                        
{'loss': 0.2366, 'grad_norm': 3.00465989112854, 'learning_rate': 3.179321486268175e-05, 'epoch': 0.24}

  2%|▏         | 984/41250 [2:22:53<96:53:47,  8.66s/it][2025-04-26 06:31:31,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 06:31:31,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.96 | bwd_microstep: 5736.98 | bwd_inner_microstep: 5648.93 | bwd_allreduce_microstep: 88.01 | step_microstep: 19.22
[2025-04-26 06:31:31,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.96 | bwd: 5737.00 | bwd_inner: 5648.93 | bwd_allreduce: 88.03 | step: 19.23

  2%|▏         | 985/41250 [2:23:02<96:50:34,  8.66s/it]
                                                        
{'loss': 0.2652, 'grad_norm': 2.5993006229400635, 'learning_rate': 3.182552504038773e-05, 'epoch': 0.24}

  2%|▏         | 985/41250 [2:23:02<96:50:34,  8.66s/it][2025-04-26 06:31:40,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 06:31:40,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.80 | bwd_microstep: 5799.15 | bwd_inner_microstep: 5668.79 | bwd_allreduce_microstep: 130.31 | step_microstep: 18.86
[2025-04-26 06:31:40,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.80 | bwd: 5799.16 | bwd_inner: 5668.79 | bwd_allreduce: 130.33 | step: 18.87

  2%|▏         | 986/41250 [2:23:11<97:01:43,  8.68s/it]
                                                        
{'loss': 0.1868, 'grad_norm': 1.59629487991333, 'learning_rate': 3.18578352180937e-05, 'epoch': 0.24}

  2%|▏         | 986/41250 [2:23:11<97:01:43,  8.68s/it][2025-04-26 06:31:48,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:31:48,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.49 | bwd_microstep: 5705.55 | bwd_inner_microstep: 5666.34 | bwd_allreduce_microstep: 39.16 | step_microstep: 18.97
[2025-04-26 06:31:48,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.49 | bwd: 5705.57 | bwd_inner: 5666.34 | bwd_allreduce: 39.18 | step: 18.97

  2%|▏         | 987/41250 [2:23:19<96:50:59,  8.66s/it]
                                                        
{'loss': 0.2982, 'grad_norm': 2.423521041870117, 'learning_rate': 3.1890145395799676e-05, 'epoch': 0.24}

  2%|▏         | 987/41250 [2:23:19<96:50:59,  8.66s/it][2025-04-26 06:31:57,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:31:57,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5775.24 | bwd_inner_microstep: 5655.95 | bwd_allreduce_microstep: 119.24 | step_microstep: 18.07
[2025-04-26 06:31:57,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5775.25 | bwd_inner: 5655.95 | bwd_allreduce: 119.26 | step: 18.08

  2%|▏         | 988/41250 [2:23:28<96:54:27,  8.66s/it]
                                                        
{'loss': 0.2033, 'grad_norm': 1.3256388902664185, 'learning_rate': 3.1922455573505654e-05, 'epoch': 0.24}

  2%|▏         | 988/41250 [2:23:28<96:54:27,  8.66s/it][2025-04-26 06:32:06,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.28 | optimizer_step: 1.06
[2025-04-26 06:32:06,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.14 | bwd_microstep: 5724.32 | bwd_inner_microstep: 5658.50 | bwd_allreduce_microstep: 65.75 | step_microstep: 20.08
[2025-04-26 06:32:06,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5724.34 | bwd_inner: 5658.50 | bwd_allreduce: 65.79 | step: 20.08

  2%|▏         | 989/41250 [2:23:37<96:49:20,  8.66s/it]
                                                        
{'loss': 0.081, 'grad_norm': 1.18235445022583, 'learning_rate': 3.195476575121164e-05, 'epoch': 0.24}

  2%|▏         | 989/41250 [2:23:37<96:49:20,  8.66s/it][2025-04-26 06:32:14,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 06:32:14,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.85 | bwd_microstep: 5773.89 | bwd_inner_microstep: 5693.02 | bwd_allreduce_microstep: 80.82 | step_microstep: 18.72
[2025-04-26 06:32:14,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5773.91 | bwd_inner: 5693.02 | bwd_allreduce: 80.84 | step: 18.72

  2%|▏         | 990/41250 [2:23:45<96:58:34,  8.67s/it]
                                                        
{'loss': 0.1385, 'grad_norm': 1.989792823791504, 'learning_rate': 3.198707592891761e-05, 'epoch': 0.24}

  2%|▏         | 990/41250 [2:23:45<96:58:34,  8.67s/it][2025-04-26 06:32:23,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:32:23,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.79 | bwd_microstep: 5764.75 | bwd_inner_microstep: 5660.04 | bwd_allreduce_microstep: 104.67 | step_microstep: 18.71
[2025-04-26 06:32:23,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5764.77 | bwd_inner: 5660.04 | bwd_allreduce: 104.68 | step: 18.71

  2%|▏         | 991/41250 [2:23:54<96:59:52,  8.67s/it]
                                                        
{'loss': 0.3319, 'grad_norm': 3.1015892028808594, 'learning_rate': 3.201938610662359e-05, 'epoch': 0.24}

  2%|▏         | 991/41250 [2:23:54<96:59:52,  8.67s/it][2025-04-26 06:32:32,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.96 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:32:32,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.44 | bwd_microstep: 5727.59 | bwd_inner_microstep: 5714.85 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.91
[2025-04-26 06:32:32,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.44 | bwd: 5727.60 | bwd_inner: 5714.85 | bwd_allreduce: 12.71 | step: 18.91

  2%|▏         | 992/41250 [2:24:03<96:56:59,  8.67s/it]
                                                        
{'loss': 0.1827, 'grad_norm': 2.6820342540740967, 'learning_rate': 3.2051696284329566e-05, 'epoch': 0.24}

  2%|▏         | 992/41250 [2:24:03<96:56:59,  8.67s/it][2025-04-26 06:32:40,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 06:32:40,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.65 | bwd_microstep: 5704.37 | bwd_inner_microstep: 5690.96 | bwd_allreduce_microstep: 13.36 | step_microstep: 18.67
[2025-04-26 06:32:40,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.65 | bwd: 5704.38 | bwd_inner: 5690.96 | bwd_allreduce: 13.38 | step: 18.67

  2%|▏         | 993/41250 [2:24:11<96:49:34,  8.66s/it]
                                                        
{'loss': 0.2615, 'grad_norm': 2.488646984100342, 'learning_rate': 3.2084006462035544e-05, 'epoch': 0.24}

  2%|▏         | 993/41250 [2:24:11<96:49:34,  8.66s/it][2025-04-26 06:32:49,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:32:49,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.03 | bwd_microstep: 5854.45 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 196.96 | step_microstep: 18.67
[2025-04-26 06:32:49,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.04 | bwd: 5854.47 | bwd_inner: 5657.45 | bwd_allreduce: 196.98 | step: 18.67

  2%|▏         | 994/41250 [2:24:20<97:15:13,  8.70s/it]
                                                        
{'loss': 0.1851, 'grad_norm': 3.163735866546631, 'learning_rate': 3.211631663974152e-05, 'epoch': 0.24}

  2%|▏         | 994/41250 [2:24:20<97:15:13,  8.70s/it][2025-04-26 06:32:58,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:32:58,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.95 | bwd_microstep: 5987.90 | bwd_inner_microstep: 5701.60 | bwd_allreduce_microstep: 286.25 | step_microstep: 18.61
[2025-04-26 06:32:58,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.95 | bwd: 5987.91 | bwd_inner: 5701.60 | bwd_allreduce: 286.27 | step: 18.61

  2%|▏         | 995/41250 [2:24:29<98:01:28,  8.77s/it]
                                                        
{'loss': 0.1671, 'grad_norm': 1.5333491563796997, 'learning_rate': 3.21486268174475e-05, 'epoch': 0.24}

  2%|▏         | 995/41250 [2:24:29<98:01:28,  8.77s/it][2025-04-26 06:33:07,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:33:07,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.53 | bwd_microstep: 5724.36 | bwd_inner_microstep: 5689.22 | bwd_allreduce_microstep: 35.09 | step_microstep: 19.04
[2025-04-26 06:33:07,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.53 | bwd: 5724.37 | bwd_inner: 5689.22 | bwd_allreduce: 35.11 | step: 19.04

  2%|▏         | 996/41250 [2:24:38<97:40:10,  8.73s/it]
                                                        
{'loss': 0.3489, 'grad_norm': 2.738328695297241, 'learning_rate': 3.218093699515348e-05, 'epoch': 0.24}

  2%|▏         | 996/41250 [2:24:38<97:40:10,  8.73s/it][2025-04-26 06:33:15,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:33:15,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.03 | bwd_microstep: 5785.18 | bwd_inner_microstep: 5772.46 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.51
[2025-04-26 06:33:15,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.03 | bwd: 5785.19 | bwd_inner: 5772.46 | bwd_allreduce: 12.69 | step: 18.51

  2%|▏         | 997/41250 [2:24:47<97:44:55,  8.74s/it]
                                                        
{'loss': 0.2526, 'grad_norm': 3.3603708744049072, 'learning_rate': 3.2213247172859455e-05, 'epoch': 0.24}

  2%|▏         | 997/41250 [2:24:47<97:44:55,  8.74s/it][2025-04-26 06:33:24,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:33:24,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.82 | bwd_microstep: 5768.05 | bwd_inner_microstep: 5755.22 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.44
[2025-04-26 06:33:24,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.82 | bwd: 5768.07 | bwd_inner: 5755.22 | bwd_allreduce: 12.80 | step: 18.44

  2%|▏         | 998/41250 [2:24:55<97:42:15,  8.74s/it]
                                                        
{'loss': 0.1558, 'grad_norm': 1.2097735404968262, 'learning_rate': 3.224555735056543e-05, 'epoch': 0.24}

  2%|▏         | 998/41250 [2:24:55<97:42:15,  8.74s/it][2025-04-26 06:33:33,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 06:33:33,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.18 | bwd_microstep: 5762.84 | bwd_inner_microstep: 5750.04 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.05
[2025-04-26 06:33:33,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.18 | bwd: 5762.85 | bwd_inner: 5750.04 | bwd_allreduce: 12.77 | step: 19.05

  2%|▏         | 999/41250 [2:25:04<97:42:07,  8.74s/it]
                                                        
{'loss': 0.1837, 'grad_norm': 2.2586488723754883, 'learning_rate': 3.227786752827141e-05, 'epoch': 0.24}

  2%|▏         | 999/41250 [2:25:04<97:42:07,  8.74s/it][2025-04-26 06:33:42,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 06:33:42,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.68 | bwd_microstep: 5756.88 | bwd_inner_microstep: 5654.65 | bwd_allreduce_microstep: 102.18 | step_microstep: 19.18
[2025-04-26 06:33:42,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.68 | bwd: 5756.89 | bwd_inner: 5654.65 | bwd_allreduce: 102.20 | step: 19.18

  2%|▏         | 1000/41250 [2:25:13<97:28:15,  8.72s/it]
                                                         
{'loss': 0.2874, 'grad_norm': 3.171621799468994, 'learning_rate': 3.231017770597738e-05, 'epoch': 0.24}

  2%|▏         | 1000/41250 [2:25:13<97:28:15,  8.72s/it][2025-04-26 06:33:50,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 06:33:50,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.58 | bwd_microstep: 5772.94 | bwd_inner_microstep: 5637.89 | bwd_allreduce_microstep: 135.00 | step_microstep: 18.76
[2025-04-26 06:33:50,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.58 | bwd: 5772.95 | bwd_inner: 5637.89 | bwd_allreduce: 135.02 | step: 18.77

  2%|▏         | 1001/41250 [2:25:21<97:17:23,  8.70s/it]
                                                         
{'loss': 0.1353, 'grad_norm': 1.299576759338379, 'learning_rate': 3.234248788368336e-05, 'epoch': 0.24}

  2%|▏         | 1001/41250 [2:25:21<97:17:23,  8.70s/it][2025-04-26 06:33:59,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.77 | optimizer_gradients: 1.07 | optimizer_step: 0.89
[2025-04-26 06:33:59,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.77 | bwd_microstep: 5688.71 | bwd_inner_microstep: 5652.29 | bwd_allreduce_microstep: 36.38 | step_microstep: 19.32
[2025-04-26 06:33:59,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.77 | bwd: 5688.73 | bwd_inner: 5652.29 | bwd_allreduce: 36.40 | step: 19.32

  2%|▏         | 1002/41250 [2:25:30<96:57:12,  8.67s/it]
                                                         
{'loss': 0.2236, 'grad_norm': 2.5483479499816895, 'learning_rate': 3.237479806138934e-05, 'epoch': 0.24}

  2%|▏         | 1002/41250 [2:25:30<96:57:12,  8.67s/it][2025-04-26 06:34:07,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.03
[2025-04-26 06:34:07,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.62 | bwd_microstep: 5704.04 | bwd_inner_microstep: 5690.83 | bwd_allreduce_microstep: 13.16 | step_microstep: 19.24
[2025-04-26 06:34:07,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.62 | bwd: 5704.05 | bwd_inner: 5690.83 | bwd_allreduce: 13.18 | step: 19.24

  2%|▏         | 1003/41250 [2:25:39<96:49:29,  8.66s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 2.092806577682495, 'learning_rate': 3.240710823909532e-05, 'epoch': 0.24}

  2%|▏         | 1003/41250 [2:25:39<96:49:29,  8.66s/it][2025-04-26 06:34:16,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 06:34:16,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.11 | bwd_microstep: 5708.49 | bwd_inner_microstep: 5695.75 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.20
[2025-04-26 06:34:16,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.11 | bwd: 5708.50 | bwd_inner: 5695.75 | bwd_allreduce: 12.71 | step: 19.20

  2%|▏         | 1004/41250 [2:25:47<96:49:17,  8.66s/it]
                                                         
{'loss': 0.1701, 'grad_norm': 1.6171208620071411, 'learning_rate': 3.2439418416801294e-05, 'epoch': 0.24}

  2%|▏         | 1004/41250 [2:25:47<96:49:17,  8.66s/it][2025-04-26 06:34:25,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 06:34:25,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.73 | bwd_microstep: 5699.14 | bwd_inner_microstep: 5686.29 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.96
[2025-04-26 06:34:25,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.73 | bwd: 5699.15 | bwd_inner: 5686.29 | bwd_allreduce: 12.82 | step: 18.97

  2%|▏         | 1005/41250 [2:25:56<96:44:15,  8.65s/it]
                                                         
{'loss': 0.2376, 'grad_norm': 3.0530216693878174, 'learning_rate': 3.247172859450727e-05, 'epoch': 0.24}

  2%|▏         | 1005/41250 [2:25:56<96:44:15,  8.65s/it][2025-04-26 06:34:33,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 06:34:33,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5735.49 | bwd_inner_microstep: 5674.45 | bwd_allreduce_microstep: 61.00 | step_microstep: 18.65
[2025-04-26 06:34:33,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5735.51 | bwd_inner: 5674.45 | bwd_allreduce: 61.02 | step: 18.65

  2%|▏         | 1006/41250 [2:26:05<96:43:53,  8.65s/it]
                                                         
{'loss': 0.2756, 'grad_norm': 2.1129136085510254, 'learning_rate': 3.250403877221325e-05, 'epoch': 0.24}

  2%|▏         | 1006/41250 [2:26:05<96:43:53,  8.65s/it][2025-04-26 06:34:42,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:34:42,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.33 | bwd_microstep: 5693.75 | bwd_inner_microstep: 5680.66 | bwd_allreduce_microstep: 13.04 | step_microstep: 19.10
[2025-04-26 06:34:42,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.33 | bwd: 5693.77 | bwd_inner: 5680.66 | bwd_allreduce: 13.06 | step: 19.10

  2%|▏         | 1007/41250 [2:26:13<96:37:07,  8.64s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 1.6610809564590454, 'learning_rate': 3.253634894991923e-05, 'epoch': 0.24}

  2%|▏         | 1007/41250 [2:26:13<96:37:07,  8.64s/it][2025-04-26 06:34:51,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.93 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 06:34:51,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.90 | bwd_microstep: 5689.28 | bwd_inner_microstep: 5646.55 | bwd_allreduce_microstep: 42.67 | step_microstep: 19.35
[2025-04-26 06:34:51,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.90 | bwd: 5689.29 | bwd_inner: 5646.54 | bwd_allreduce: 42.69 | step: 19.35

  2%|▏         | 1008/41250 [2:26:22<96:28:33,  8.63s/it]
                                                         
{'loss': 0.1204, 'grad_norm': 1.4752391576766968, 'learning_rate': 3.2568659127625205e-05, 'epoch': 0.24}

  2%|▏         | 1008/41250 [2:26:22<96:28:33,  8.63s/it][2025-04-26 06:34:59,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 06:34:59,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.33 | bwd_microstep: 5698.43 | bwd_inner_microstep: 5685.53 | bwd_allreduce_microstep: 12.85 | step_microstep: 19.22
[2025-04-26 06:34:59,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.33 | bwd: 5698.44 | bwd_inner: 5685.53 | bwd_allreduce: 12.87 | step: 19.22

  2%|▏         | 1009/41250 [2:26:30<96:26:27,  8.63s/it]
                                                         
{'loss': 0.1086, 'grad_norm': 2.2060701847076416, 'learning_rate': 3.260096930533118e-05, 'epoch': 0.24}

  2%|▏         | 1009/41250 [2:26:30<96:26:27,  8.63s/it][2025-04-26 06:35:08,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.07 | optimizer_step: 1.14
[2025-04-26 06:35:08,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.73 | bwd_microstep: 5875.15 | bwd_inner_microstep: 5644.95 | bwd_allreduce_microstep: 230.15 | step_microstep: 19.24
[2025-04-26 06:35:08,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.73 | bwd: 5875.16 | bwd_inner: 5644.95 | bwd_allreduce: 230.17 | step: 19.25

  2%|▏         | 1010/41250 [2:26:39<96:56:28,  8.67s/it]
                                                         
{'loss': 0.3922, 'grad_norm': 3.228013753890991, 'learning_rate': 3.263327948303716e-05, 'epoch': 0.24}

  2%|▏         | 1010/41250 [2:26:39<96:56:28,  8.67s/it][2025-04-26 06:35:17,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:35:17,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.33 | bwd_microstep: 5702.18 | bwd_inner_microstep: 5689.46 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.77
[2025-04-26 06:35:17,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.33 | bwd: 5702.20 | bwd_inner: 5689.46 | bwd_allreduce: 12.69 | step: 18.77

  2%|▏         | 1011/41250 [2:26:48<96:49:46,  8.66s/it]
                                                         
{'loss': 0.2442, 'grad_norm': 3.4654505252838135, 'learning_rate': 3.266558966074314e-05, 'epoch': 0.25}

  2%|▏         | 1011/41250 [2:26:48<96:49:46,  8.66s/it][2025-04-26 06:35:25,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:35:25,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.78 | bwd_microstep: 5798.45 | bwd_inner_microstep: 5785.40 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.01
[2025-04-26 06:35:25,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.78 | bwd: 5798.46 | bwd_inner: 5785.40 | bwd_allreduce: 13.02 | step: 19.01

  2%|▏         | 1012/41250 [2:26:57<97:12:45,  8.70s/it]
                                                         
{'loss': 0.2041, 'grad_norm': 1.9797918796539307, 'learning_rate': 3.269789983844912e-05, 'epoch': 0.25}

  2%|▏         | 1012/41250 [2:26:57<97:12:45,  8.70s/it][2025-04-26 06:35:34,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 06:35:34,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5714.05 | bwd_inner_microstep: 5701.22 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.18
[2025-04-26 06:35:34,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5714.07 | bwd_inner: 5701.22 | bwd_allreduce: 12.80 | step: 19.18

  2%|▏         | 1013/41250 [2:27:05<97:02:45,  8.68s/it]
                                                         
{'loss': 0.1142, 'grad_norm': 1.4615166187286377, 'learning_rate': 3.2730210016155095e-05, 'epoch': 0.25}

  2%|▏         | 1013/41250 [2:27:05<97:02:45,  8.68s/it][2025-04-26 06:35:43,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 06:35:43,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.75 | bwd_microstep: 5702.85 | bwd_inner_microstep: 5637.22 | bwd_allreduce_microstep: 65.58 | step_microstep: 18.83
[2025-04-26 06:35:43,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.75 | bwd: 5702.87 | bwd_inner: 5637.22 | bwd_allreduce: 65.60 | step: 18.83

  2%|▏         | 1014/41250 [2:27:14<96:48:25,  8.66s/it]
                                                         
{'loss': 0.1989, 'grad_norm': 1.8862149715423584, 'learning_rate': 3.2762520193861066e-05, 'epoch': 0.25}

  2%|▏         | 1014/41250 [2:27:14<96:48:25,  8.66s/it][2025-04-26 06:35:51,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:35:51,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.96 | bwd_microstep: 5719.14 | bwd_inner_microstep: 5648.95 | bwd_allreduce_microstep: 70.14 | step_microstep: 18.93
[2025-04-26 06:35:51,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.96 | bwd: 5719.15 | bwd_inner: 5648.94 | bwd_allreduce: 70.16 | step: 18.93

  2%|▏         | 1015/41250 [2:27:22<96:40:46,  8.65s/it]
                                                         
{'loss': 0.1843, 'grad_norm': 2.9782497882843018, 'learning_rate': 3.2794830371567044e-05, 'epoch': 0.25}

  2%|▏         | 1015/41250 [2:27:22<96:40:46,  8.65s/it][2025-04-26 06:36:00,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 06:36:00,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.17 | bwd_microstep: 5739.56 | bwd_inner_microstep: 5655.18 | bwd_allreduce_microstep: 84.33 | step_microstep: 18.86
[2025-04-26 06:36:00,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.17 | bwd: 5739.58 | bwd_inner: 5655.18 | bwd_allreduce: 84.35 | step: 18.86

  2%|▏         | 1016/41250 [2:27:31<96:43:57,  8.66s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.7077611684799194, 'learning_rate': 3.282714054927302e-05, 'epoch': 0.25}

  2%|▏         | 1016/41250 [2:27:31<96:43:57,  8.66s/it][2025-04-26 06:36:09,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:36:09,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.17 | bwd_microstep: 5747.98 | bwd_inner_microstep: 5699.00 | bwd_allreduce_microstep: 48.94 | step_microstep: 18.69
[2025-04-26 06:36:09,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.17 | bwd: 5747.99 | bwd_inner: 5699.00 | bwd_allreduce: 48.95 | step: 18.69

  2%|▏         | 1017/41250 [2:27:40<96:48:54,  8.66s/it]
                                                         
{'loss': 0.1632, 'grad_norm': 1.7545922994613647, 'learning_rate': 3.2859450726979006e-05, 'epoch': 0.25}

  2%|▏         | 1017/41250 [2:27:40<96:48:54,  8.66s/it][2025-04-26 06:36:17,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:36:17,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.03 | bwd_microstep: 5783.71 | bwd_inner_microstep: 5648.76 | bwd_allreduce_microstep: 134.90 | step_microstep: 18.60
[2025-04-26 06:36:17,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.03 | bwd: 5783.72 | bwd_inner: 5648.76 | bwd_allreduce: 134.92 | step: 18.60

  2%|▏         | 1018/41250 [2:27:49<96:53:59,  8.67s/it]
                                                         
{'loss': 0.2132, 'grad_norm': 2.4195289611816406, 'learning_rate': 3.289176090468498e-05, 'epoch': 0.25}

  2%|▏         | 1018/41250 [2:27:49<96:53:59,  8.67s/it][2025-04-26 06:36:26,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:36:26,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.57 | bwd_microstep: 5714.84 | bwd_inner_microstep: 5642.78 | bwd_allreduce_microstep: 72.02 | step_microstep: 18.54
[2025-04-26 06:36:26,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.57 | bwd: 5714.85 | bwd_inner: 5642.78 | bwd_allreduce: 72.04 | step: 18.54

  2%|▏         | 1019/41250 [2:27:57<96:41:54,  8.65s/it]
                                                         
{'loss': 0.1631, 'grad_norm': 2.2741241455078125, 'learning_rate': 3.2924071082390955e-05, 'epoch': 0.25}

  2%|▏         | 1019/41250 [2:27:57<96:41:54,  8.65s/it][2025-04-26 06:36:35,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 06:36:35,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5883.84 | bwd_inner_microstep: 5654.77 | bwd_allreduce_microstep: 229.02 | step_microstep: 18.56
[2025-04-26 06:36:35,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5883.86 | bwd_inner: 5654.77 | bwd_allreduce: 229.04 | step: 18.56

  2%|▏         | 1020/41250 [2:28:06<97:11:10,  8.70s/it]
                                                         
{'loss': 0.3679, 'grad_norm': 1.8546116352081299, 'learning_rate': 3.295638126009693e-05, 'epoch': 0.25}

  2%|▏         | 1020/41250 [2:28:06<97:11:10,  8.70s/it][2025-04-26 06:36:43,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 06:36:43,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.41 | bwd_microstep: 5768.95 | bwd_inner_microstep: 5688.77 | bwd_allreduce_microstep: 80.13 | step_microstep: 19.54
[2025-04-26 06:36:43,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.41 | bwd: 5768.97 | bwd_inner: 5688.77 | bwd_allreduce: 80.14 | step: 19.54

  2%|▏         | 1021/41250 [2:28:15<97:10:46,  8.70s/it]
                                                         
{'loss': 0.2238, 'grad_norm': 2.202698230743408, 'learning_rate': 3.298869143780291e-05, 'epoch': 0.25}

  2%|▏         | 1021/41250 [2:28:15<97:10:46,  8.70s/it][2025-04-26 06:36:52,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:36:52,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.71 | bwd_microstep: 5745.33 | bwd_inner_microstep: 5705.09 | bwd_allreduce_microstep: 40.18 | step_microstep: 18.70
[2025-04-26 06:36:52,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.71 | bwd: 5745.34 | bwd_inner: 5705.09 | bwd_allreduce: 40.20 | step: 18.70

  2%|▏         | 1022/41250 [2:28:23<97:08:39,  8.69s/it]
                                                         
{'loss': 0.1893, 'grad_norm': 2.004671096801758, 'learning_rate': 3.302100161550889e-05, 'epoch': 0.25}

  2%|▏         | 1022/41250 [2:28:23<97:08:39,  8.69s/it][2025-04-26 06:37:01,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 06:37:01,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.56 | bwd_microstep: 5771.39 | bwd_inner_microstep: 5650.18 | bwd_allreduce_microstep: 121.17 | step_microstep: 17.93
[2025-04-26 06:37:01,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.56 | bwd: 5771.40 | bwd_inner: 5650.18 | bwd_allreduce: 121.18 | step: 17.93

  2%|▏         | 1023/41250 [2:28:32<97:07:27,  8.69s/it]
                                                         
{'loss': 0.3499, 'grad_norm': 2.953758478164673, 'learning_rate': 3.305331179321487e-05, 'epoch': 0.25}

  2%|▏         | 1023/41250 [2:28:32<97:07:27,  8.69s/it][2025-04-26 06:37:09,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:37:09,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.97 | bwd_microstep: 5708.28 | bwd_inner_microstep: 5654.31 | bwd_allreduce_microstep: 53.93 | step_microstep: 18.59
[2025-04-26 06:37:09,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.97 | bwd: 5708.30 | bwd_inner: 5654.31 | bwd_allreduce: 53.95 | step: 18.59

  2%|▏         | 1024/41250 [2:28:41<96:51:19,  8.67s/it]
                                                         
{'loss': 0.3394, 'grad_norm': 2.1474075317382812, 'learning_rate': 3.3085621970920845e-05, 'epoch': 0.25}

  2%|▏         | 1024/41250 [2:28:41<96:51:19,  8.67s/it][2025-04-26 06:37:18,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 06:37:18,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.44 | bwd_microstep: 5701.62 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 46.33 | step_microstep: 18.99
[2025-04-26 06:37:18,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.44 | bwd: 5701.63 | bwd_inner: 5655.24 | bwd_allreduce: 46.35 | step: 18.99

  2%|▏         | 1025/41250 [2:28:49<96:41:00,  8.65s/it]
                                                         
{'loss': 0.2579, 'grad_norm': 2.3803598880767822, 'learning_rate': 3.3117932148626816e-05, 'epoch': 0.25}

  2%|▏         | 1025/41250 [2:28:49<96:41:00,  8.65s/it][2025-04-26 06:37:27,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.17 | optimizer_step: 0.91
[2025-04-26 06:37:27,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.27 | bwd_microstep: 5747.69 | bwd_inner_microstep: 5705.82 | bwd_allreduce_microstep: 41.83 | step_microstep: 18.93
[2025-04-26 06:37:27,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.27 | bwd: 5747.70 | bwd_inner: 5705.82 | bwd_allreduce: 41.85 | step: 18.93

  2%|▏         | 1026/41250 [2:28:58<96:46:33,  8.66s/it]
                                                         
{'loss': 0.1046, 'grad_norm': 1.1374123096466064, 'learning_rate': 3.31502423263328e-05, 'epoch': 0.25}

  2%|▏         | 1026/41250 [2:28:58<96:46:33,  8.66s/it][2025-04-26 06:37:35,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.19 | optimizer_step: 0.89
[2025-04-26 06:37:35,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 5775.00 | bwd_inner_microstep: 5692.26 | bwd_allreduce_microstep: 82.69 | step_microstep: 19.20
[2025-04-26 06:37:35,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5775.01 | bwd_inner: 5692.26 | bwd_allreduce: 82.71 | step: 19.20

  2%|▏         | 1027/41250 [2:29:07<96:54:19,  8.67s/it]
                                                         
{'loss': 0.3438, 'grad_norm': 3.7412657737731934, 'learning_rate': 3.318255250403878e-05, 'epoch': 0.25}

  2%|▏         | 1027/41250 [2:29:07<96:54:19,  8.67s/it][2025-04-26 06:37:44,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:37:44,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.02 | bwd_microstep: 5761.34 | bwd_inner_microstep: 5721.08 | bwd_allreduce_microstep: 40.21 | step_microstep: 18.66
[2025-04-26 06:37:44,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.02 | bwd: 5761.36 | bwd_inner: 5721.08 | bwd_allreduce: 40.23 | step: 18.66

  2%|▏         | 1028/41250 [2:29:15<96:57:27,  8.68s/it]
                                                         
{'loss': 0.1998, 'grad_norm': 2.5920374393463135, 'learning_rate': 3.321486268174475e-05, 'epoch': 0.25}

  2%|▏         | 1028/41250 [2:29:15<96:57:27,  8.68s/it][2025-04-26 06:37:53,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.31 | optimizer_step: 1.05
[2025-04-26 06:37:53,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2899.22 | bwd_microstep: 5801.48 | bwd_inner_microstep: 5787.69 | bwd_allreduce_microstep: 13.72 | step_microstep: 20.24
[2025-04-26 06:37:53,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2899.22 | bwd: 5801.49 | bwd_inner: 5787.69 | bwd_allreduce: 13.75 | step: 20.24

  2%|▏         | 1029/41250 [2:29:24<97:21:08,  8.71s/it]
                                                         
{'loss': 0.0896, 'grad_norm': 1.107397198677063, 'learning_rate': 3.324717285945073e-05, 'epoch': 0.25}

  2%|▏         | 1029/41250 [2:29:24<97:21:08,  8.71s/it][2025-04-26 06:38:02,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 06:38:02,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.49 | bwd_microstep: 5722.28 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 70.42 | step_microstep: 18.98
[2025-04-26 06:38:02,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.49 | bwd: 5722.30 | bwd_inner: 5651.81 | bwd_allreduce: 70.44 | step: 18.98

  2%|▏         | 1030/41250 [2:29:33<97:05:15,  8.69s/it]
                                                         
{'loss': 0.1125, 'grad_norm': 1.401999831199646, 'learning_rate': 3.3279483037156705e-05, 'epoch': 0.25}

  2%|▏         | 1030/41250 [2:29:33<97:05:15,  8.69s/it][2025-04-26 06:38:10,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:38:10,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.81 | bwd_microstep: 5869.49 | bwd_inner_microstep: 5711.04 | bwd_allreduce_microstep: 158.41 | step_microstep: 18.00
[2025-04-26 06:38:10,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.00 | bwd: 5869.51 | bwd_inner: 5711.04 | bwd_allreduce: 158.43 | step: 18.01

  2%|▏         | 1031/41250 [2:29:42<97:29:58,  8.73s/it]
                                                         
{'loss': 0.1382, 'grad_norm': 1.5554120540618896, 'learning_rate': 3.331179321486269e-05, 'epoch': 0.25}

  2%|▏         | 1031/41250 [2:29:42<97:29:58,  8.73s/it][2025-04-26 06:38:19,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:38:19,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.46 | bwd_microstep: 5729.35 | bwd_inner_microstep: 5653.36 | bwd_allreduce_microstep: 75.96 | step_microstep: 17.99
[2025-04-26 06:38:19,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.46 | bwd: 5729.37 | bwd_inner: 5653.36 | bwd_allreduce: 75.97 | step: 18.00

  3%|▎         | 1032/41250 [2:29:50<97:11:33,  8.70s/it]
                                                         
{'loss': 0.1752, 'grad_norm': 1.2576425075531006, 'learning_rate': 3.334410339256866e-05, 'epoch': 0.25}

  3%|▎         | 1032/41250 [2:29:50<97:11:33,  8.70s/it][2025-04-26 06:38:28,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:38:28,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.08 | bwd_microstep: 5713.80 | bwd_inner_microstep: 5653.53 | bwd_allreduce_microstep: 60.23 | step_microstep: 18.50
[2025-04-26 06:38:28,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.08 | bwd: 5713.82 | bwd_inner: 5653.53 | bwd_allreduce: 60.24 | step: 18.51

  3%|▎         | 1033/41250 [2:29:59<96:54:38,  8.67s/it]
                                                         
{'loss': 0.3202, 'grad_norm': 2.121727705001831, 'learning_rate': 3.337641357027464e-05, 'epoch': 0.25}

  3%|▎         | 1033/41250 [2:29:59<96:54:38,  8.67s/it][2025-04-26 06:38:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:38:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.98 | bwd_microstep: 5803.33 | bwd_inner_microstep: 5655.08 | bwd_allreduce_microstep: 148.20 | step_microstep: 18.49
[2025-04-26 06:38:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.98 | bwd: 5803.34 | bwd_inner: 5655.08 | bwd_allreduce: 148.22 | step: 18.49

  3%|▎         | 1034/41250 [2:30:08<97:02:26,  8.69s/it]
                                                         
{'loss': 0.234, 'grad_norm': 1.6703015565872192, 'learning_rate': 3.340872374798062e-05, 'epoch': 0.25}

  3%|▎         | 1034/41250 [2:30:08<97:02:26,  8.69s/it][2025-04-26 06:38:45,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:38:45,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.34 | bwd_microstep: 5724.92 | bwd_inner_microstep: 5646.37 | bwd_allreduce_microstep: 78.51 | step_microstep: 18.33
[2025-04-26 06:38:45,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.34 | bwd: 5724.94 | bwd_inner: 5646.37 | bwd_allreduce: 78.53 | step: 18.34

  3%|▎         | 1035/41250 [2:30:16<96:51:24,  8.67s/it]
                                                         
{'loss': 0.2165, 'grad_norm': 2.3534650802612305, 'learning_rate': 3.3441033925686595e-05, 'epoch': 0.25}

  3%|▎         | 1035/41250 [2:30:16<96:51:24,  8.67s/it][2025-04-26 06:38:54,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 06:38:54,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.61 | bwd_microstep: 5705.11 | bwd_inner_microstep: 5673.40 | bwd_allreduce_microstep: 31.67 | step_microstep: 18.53
[2025-04-26 06:38:54,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.61 | bwd: 5705.13 | bwd_inner: 5673.40 | bwd_allreduce: 31.69 | step: 18.53

  3%|▎         | 1036/41250 [2:30:25<96:42:12,  8.66s/it]
                                                         
{'loss': 0.2772, 'grad_norm': 2.0572566986083984, 'learning_rate': 3.347334410339257e-05, 'epoch': 0.25}

  3%|▎         | 1036/41250 [2:30:25<96:42:12,  8.66s/it][2025-04-26 06:39:02,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:39:02,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5777.99 | bwd_inner_microstep: 5712.78 | bwd_allreduce_microstep: 65.17 | step_microstep: 18.42
[2025-04-26 06:39:02,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5778.00 | bwd_inner: 5712.78 | bwd_allreduce: 65.18 | step: 18.43

  3%|▎         | 1037/41250 [2:30:33<96:50:18,  8.67s/it]
                                                         
{'loss': 0.1316, 'grad_norm': 1.4819767475128174, 'learning_rate': 3.350565428109855e-05, 'epoch': 0.25}

  3%|▎         | 1037/41250 [2:30:33<96:50:18,  8.67s/it][2025-04-26 06:39:11,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 06:39:11,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.66 | bwd_microstep: 5772.19 | bwd_inner_microstep: 5759.43 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.19
[2025-04-26 06:39:11,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.66 | bwd: 5772.21 | bwd_inner: 5759.43 | bwd_allreduce: 12.74 | step: 19.19

  3%|▎         | 1038/41250 [2:30:42<97:02:58,  8.69s/it]
                                                         
{'loss': 0.1973, 'grad_norm': 1.2635740041732788, 'learning_rate': 3.353796445880453e-05, 'epoch': 0.25}

  3%|▎         | 1038/41250 [2:30:42<97:02:58,  8.69s/it][2025-04-26 06:39:20,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 06:39:20,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.71 | bwd_microstep: 5694.60 | bwd_inner_microstep: 5653.55 | bwd_allreduce_microstep: 41.00 | step_microstep: 18.86
[2025-04-26 06:39:20,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.71 | bwd: 5694.62 | bwd_inner: 5653.55 | bwd_allreduce: 41.02 | step: 18.86

  3%|▎         | 1039/41250 [2:30:51<96:47:55,  8.67s/it]
                                                         
{'loss': 0.1882, 'grad_norm': 2.4267678260803223, 'learning_rate': 3.35702746365105e-05, 'epoch': 0.25}

  3%|▎         | 1039/41250 [2:30:51<96:47:55,  8.67s/it][2025-04-26 06:39:28,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 06:39:28,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5691.58 | bwd_inner_microstep: 5678.84 | bwd_allreduce_microstep: 12.70 | step_microstep: 19.31
[2025-04-26 06:39:28,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5691.59 | bwd_inner: 5678.84 | bwd_allreduce: 12.71 | step: 19.32

  3%|▎         | 1040/41250 [2:30:59<96:40:18,  8.66s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 1.675219178199768, 'learning_rate': 3.3602584814216484e-05, 'epoch': 0.25}

  3%|▎         | 1040/41250 [2:30:59<96:40:18,  8.66s/it][2025-04-26 06:39:37,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 06:39:37,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5754.72 | bwd_inner_microstep: 5645.77 | bwd_allreduce_microstep: 108.89 | step_microstep: 18.96
[2025-04-26 06:39:37,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5754.73 | bwd_inner: 5645.77 | bwd_allreduce: 108.91 | step: 18.96

  3%|▎         | 1041/41250 [2:31:08<96:42:16,  8.66s/it]
                                                         
{'loss': 0.2802, 'grad_norm': 2.6826579570770264, 'learning_rate': 3.363489499192246e-05, 'epoch': 0.25}

  3%|▎         | 1041/41250 [2:31:08<96:42:16,  8.66s/it][2025-04-26 06:39:46,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:39:46,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5706.04 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.85
[2025-04-26 06:39:46,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5706.05 | bwd_inner: 5693.27 | bwd_allreduce: 12.74 | step: 18.85

  3%|▎         | 1042/41250 [2:31:17<96:36:37,  8.65s/it]
                                                         
{'loss': 0.1587, 'grad_norm': 1.388665795326233, 'learning_rate': 3.366720516962843e-05, 'epoch': 0.25}

  3%|▎         | 1042/41250 [2:31:17<96:36:37,  8.65s/it][2025-04-26 06:39:54,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 06:39:54,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.84 | bwd_microstep: 5747.16 | bwd_inner_microstep: 5685.41 | bwd_allreduce_microstep: 61.69 | step_microstep: 18.98
[2025-04-26 06:39:54,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.84 | bwd: 5747.17 | bwd_inner: 5685.40 | bwd_allreduce: 61.72 | step: 18.98

  3%|▎         | 1043/41250 [2:31:25<96:42:18,  8.66s/it]
                                                         
{'loss': 0.1968, 'grad_norm': 1.5948470830917358, 'learning_rate': 3.369951534733441e-05, 'epoch': 0.25}

  3%|▎         | 1043/41250 [2:31:25<96:42:18,  8.66s/it][2025-04-26 06:40:03,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.27 | optimizer_step: 1.04
[2025-04-26 06:40:03,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.90 | bwd_microstep: 5700.42 | bwd_inner_microstep: 5687.50 | bwd_allreduce_microstep: 12.88 | step_microstep: 20.46
[2025-04-26 06:40:03,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.90 | bwd: 5700.44 | bwd_inner: 5687.50 | bwd_allreduce: 12.89 | step: 20.46

  3%|▎         | 1044/41250 [2:31:34<96:36:51,  8.65s/it]
                                                         
{'loss': 0.2823, 'grad_norm': 2.4333345890045166, 'learning_rate': 3.373182552504039e-05, 'epoch': 0.25}

  3%|▎         | 1044/41250 [2:31:34<96:36:51,  8.65s/it][2025-04-26 06:40:11,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:40:11,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.38 | bwd_microstep: 5684.42 | bwd_inner_microstep: 5671.62 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.59
[2025-04-26 06:40:11,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.38 | bwd: 5684.44 | bwd_inner: 5671.62 | bwd_allreduce: 12.78 | step: 18.59

  3%|▎         | 1045/41250 [2:31:43<96:27:57,  8.64s/it]
                                                         
{'loss': 0.3895, 'grad_norm': 2.6334455013275146, 'learning_rate': 3.3764135702746367e-05, 'epoch': 0.25}

  3%|▎         | 1045/41250 [2:31:43<96:27:57,  8.64s/it][2025-04-26 06:40:20,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:40:20,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.64 | bwd_microstep: 5709.49 | bwd_inner_microstep: 5649.96 | bwd_allreduce_microstep: 59.49 | step_microstep: 18.64
[2025-04-26 06:40:20,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5709.50 | bwd_inner: 5649.96 | bwd_allreduce: 59.50 | step: 18.65

  3%|▎         | 1046/41250 [2:31:51<96:21:40,  8.63s/it]
                                                         
{'loss': 0.1512, 'grad_norm': 1.4147772789001465, 'learning_rate': 3.3796445880452344e-05, 'epoch': 0.25}

  3%|▎         | 1046/41250 [2:31:51<96:21:40,  8.63s/it][2025-04-26 06:40:29,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:40:29,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.78 | bwd_microstep: 5703.65 | bwd_inner_microstep: 5634.62 | bwd_allreduce_microstep: 68.98 | step_microstep: 18.54
[2025-04-26 06:40:29,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.78 | bwd: 5703.66 | bwd_inner: 5634.62 | bwd_allreduce: 69.00 | step: 18.54

  3%|▎         | 1047/41250 [2:32:00<96:16:33,  8.62s/it]
                                                         
{'loss': 0.125, 'grad_norm': 1.1038998365402222, 'learning_rate': 3.382875605815832e-05, 'epoch': 0.25}

  3%|▎         | 1047/41250 [2:32:00<96:16:33,  8.62s/it][2025-04-26 06:40:37,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:40:37,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.23 | bwd_microstep: 5701.19 | bwd_inner_microstep: 5644.10 | bwd_allreduce_microstep: 57.04 | step_microstep: 18.71
[2025-04-26 06:40:37,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.23 | bwd: 5701.20 | bwd_inner: 5644.10 | bwd_allreduce: 57.06 | step: 18.71

  3%|▎         | 1048/41250 [2:32:08<96:13:55,  8.62s/it]
                                                         
{'loss': 0.4054, 'grad_norm': 2.5829639434814453, 'learning_rate': 3.38610662358643e-05, 'epoch': 0.25}

  3%|▎         | 1048/41250 [2:32:08<96:13:55,  8.62s/it][2025-04-26 06:40:46,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:40:46,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.49 | bwd_microstep: 5848.28 | bwd_inner_microstep: 5653.86 | bwd_allreduce_microstep: 194.37 | step_microstep: 18.70
[2025-04-26 06:40:46,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.49 | bwd: 5848.29 | bwd_inner: 5653.86 | bwd_allreduce: 194.39 | step: 18.71

  3%|▎         | 1049/41250 [2:32:17<96:41:21,  8.66s/it]
                                                         
{'loss': 0.267, 'grad_norm': 1.6953774690628052, 'learning_rate': 3.389337641357028e-05, 'epoch': 0.25}

  3%|▎         | 1049/41250 [2:32:17<96:41:21,  8.66s/it][2025-04-26 06:40:55,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 06:40:55,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.57 | bwd_microstep: 5738.02 | bwd_inner_microstep: 5686.31 | bwd_allreduce_microstep: 51.66 | step_microstep: 18.79
[2025-04-26 06:40:55,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.57 | bwd: 5738.03 | bwd_inner: 5686.31 | bwd_allreduce: 51.68 | step: 18.79

  3%|▎         | 1050/41250 [2:32:26<96:43:22,  8.66s/it]
                                                         
{'loss': 0.2396, 'grad_norm': 2.262817859649658, 'learning_rate': 3.3925686591276256e-05, 'epoch': 0.25}

  3%|▎         | 1050/41250 [2:32:26<96:43:22,  8.66s/it][2025-04-26 06:41:03,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 06:41:03,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.03 | bwd_microstep: 5693.35 | bwd_inner_microstep: 5644.25 | bwd_allreduce_microstep: 49.06 | step_microstep: 19.10
[2025-04-26 06:41:03,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.03 | bwd: 5693.37 | bwd_inner: 5644.25 | bwd_allreduce: 49.08 | step: 19.10

  3%|▎         | 1051/41250 [2:32:35<96:30:08,  8.64s/it]
                                                         
{'loss': 0.0883, 'grad_norm': 0.9950439929962158, 'learning_rate': 3.3957996768982234e-05, 'epoch': 0.25}

  3%|▎         | 1051/41250 [2:32:35<96:30:08,  8.64s/it][2025-04-26 06:41:12,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:41:12,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.49 | bwd_microstep: 5753.32 | bwd_inner_microstep: 5681.17 | bwd_allreduce_microstep: 72.11 | step_microstep: 18.48
[2025-04-26 06:41:12,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.49 | bwd: 5753.33 | bwd_inner: 5681.17 | bwd_allreduce: 72.13 | step: 18.48

  3%|▎         | 1052/41250 [2:32:43<96:37:06,  8.65s/it]
                                                         
{'loss': 0.2708, 'grad_norm': 2.2958967685699463, 'learning_rate': 3.3990306946688205e-05, 'epoch': 0.26}

  3%|▎         | 1052/41250 [2:32:43<96:37:06,  8.65s/it][2025-04-26 06:41:21,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 06:41:21,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.67 | bwd_microstep: 5688.53 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 29.63 | step_microstep: 18.83
[2025-04-26 06:41:21,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.67 | bwd: 5688.55 | bwd_inner: 5658.86 | bwd_allreduce: 29.64 | step: 18.84

  3%|▎         | 1053/41250 [2:32:52<96:25:44,  8.64s/it]
                                                         
{'loss': 0.255, 'grad_norm': 1.352371096611023, 'learning_rate': 3.402261712439418e-05, 'epoch': 0.26}

  3%|▎         | 1053/41250 [2:32:52<96:25:44,  8.64s/it][2025-04-26 06:41:29,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 06:41:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5688.56 | bwd_inner_microstep: 5641.16 | bwd_allreduce_microstep: 47.35 | step_microstep: 18.73
[2025-04-26 06:41:29,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5688.57 | bwd_inner: 5641.16 | bwd_allreduce: 47.37 | step: 18.73

  3%|▎         | 1054/41250 [2:33:00<96:16:19,  8.62s/it]
                                                         
{'loss': 0.1384, 'grad_norm': 2.524822235107422, 'learning_rate': 3.405492730210017e-05, 'epoch': 0.26}

  3%|▎         | 1054/41250 [2:33:00<96:16:19,  8.62s/it][2025-04-26 06:41:38,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.09 | optimizer_step: 1.19
[2025-04-26 06:41:38,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.33 | bwd_microstep: 6018.70 | bwd_inner_microstep: 5650.66 | bwd_allreduce_microstep: 367.98 | step_microstep: 19.28
[2025-04-26 06:41:38,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.33 | bwd: 6018.71 | bwd_inner: 5650.66 | bwd_allreduce: 368.01 | step: 19.28

  3%|▎         | 1055/41250 [2:33:09<97:16:32,  8.71s/it]
                                                         
{'loss': 0.2828, 'grad_norm': 2.9042208194732666, 'learning_rate': 3.4087237479806145e-05, 'epoch': 0.26}

  3%|▎         | 1055/41250 [2:33:09<97:16:32,  8.71s/it][2025-04-26 06:41:47,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 06:41:47,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.02 | bwd_microstep: 5754.26 | bwd_inner_microstep: 5698.41 | bwd_allreduce_microstep: 55.80 | step_microstep: 18.69
[2025-04-26 06:41:47,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.02 | bwd: 5754.28 | bwd_inner: 5698.41 | bwd_allreduce: 55.82 | step: 18.69

  3%|▎         | 1056/41250 [2:33:18<97:10:10,  8.70s/it]
                                                         
{'loss': 0.2316, 'grad_norm': 1.4960277080535889, 'learning_rate': 3.4119547657512117e-05, 'epoch': 0.26}

  3%|▎         | 1056/41250 [2:33:18<97:10:10,  8.70s/it][2025-04-26 06:41:55,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:41:55,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5738.00 | bwd_inner_microstep: 5698.07 | bwd_allreduce_microstep: 39.88 | step_microstep: 18.44
[2025-04-26 06:41:55,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5738.01 | bwd_inner: 5698.07 | bwd_allreduce: 39.90 | step: 18.44

  3%|▎         | 1057/41250 [2:33:27<97:04:28,  8.69s/it]
                                                         
{'loss': 0.1928, 'grad_norm': 1.254188895225525, 'learning_rate': 3.4151857835218094e-05, 'epoch': 0.26}

  3%|▎         | 1057/41250 [2:33:27<97:04:28,  8.69s/it][2025-04-26 06:42:04,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-26 06:42:04,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.07 | bwd_microstep: 5856.17 | bwd_inner_microstep: 5687.42 | bwd_allreduce_microstep: 168.69 | step_microstep: 18.96
[2025-04-26 06:42:04,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.08 | bwd: 5856.18 | bwd_inner: 5687.42 | bwd_allreduce: 168.72 | step: 18.97

  3%|▎         | 1058/41250 [2:33:35<97:24:52,  8.73s/it]
                                                         
{'loss': 0.3013, 'grad_norm': 2.8708975315093994, 'learning_rate': 3.418416801292407e-05, 'epoch': 0.26}

  3%|▎         | 1058/41250 [2:33:35<97:24:52,  8.73s/it][2025-04-26 06:42:13,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 06:42:13,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.09 | bwd_microstep: 5715.98 | bwd_inner_microstep: 5703.27 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.01
[2025-04-26 06:42:13,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.09 | bwd: 5715.99 | bwd_inner: 5703.27 | bwd_allreduce: 12.68 | step: 19.01

  3%|▎         | 1059/41250 [2:33:44<97:11:01,  8.70s/it]
                                                         
{'loss': 0.2327, 'grad_norm': 1.4784581661224365, 'learning_rate': 3.421647819063005e-05, 'epoch': 0.26}

  3%|▎         | 1059/41250 [2:33:44<97:11:01,  8.70s/it][2025-04-26 06:42:22,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:42:22,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.24 | bwd_microstep: 5697.65 | bwd_inner_microstep: 5648.42 | bwd_allreduce_microstep: 49.18 | step_microstep: 18.64
[2025-04-26 06:42:22,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.24 | bwd: 5697.66 | bwd_inner: 5648.42 | bwd_allreduce: 49.20 | step: 18.64

  3%|▎         | 1060/41250 [2:33:53<96:50:44,  8.67s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 1.7207672595977783, 'learning_rate': 3.424878836833603e-05, 'epoch': 0.26}

  3%|▎         | 1060/41250 [2:33:53<96:50:44,  8.67s/it][2025-04-26 06:42:30,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-26 06:42:30,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.75 | bwd_microstep: 5683.70 | bwd_inner_microstep: 5639.65 | bwd_allreduce_microstep: 44.00 | step_microstep: 19.12
[2025-04-26 06:42:30,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.75 | bwd: 5683.72 | bwd_inner: 5639.65 | bwd_allreduce: 44.03 | step: 19.12

  3%|▎         | 1061/41250 [2:34:01<96:35:38,  8.65s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 2.0656721591949463, 'learning_rate': 3.4281098546042006e-05, 'epoch': 0.26}

  3%|▎         | 1061/41250 [2:34:01<96:35:38,  8.65s/it][2025-04-26 06:42:39,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 06:42:39,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.36 | bwd_microstep: 5730.53 | bwd_inner_microstep: 5684.72 | bwd_allreduce_microstep: 45.77 | step_microstep: 18.55
[2025-04-26 06:42:39,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.36 | bwd: 5730.54 | bwd_inner: 5684.72 | bwd_allreduce: 45.78 | step: 18.55

  3%|▎         | 1062/41250 [2:34:10<96:39:44,  8.66s/it]
                                                         
{'loss': 0.1744, 'grad_norm': 2.2867703437805176, 'learning_rate': 3.4313408723747984e-05, 'epoch': 0.26}

  3%|▎         | 1062/41250 [2:34:10<96:39:44,  8.66s/it][2025-04-26 06:42:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:42:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.63 | bwd_microstep: 5689.47 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 35.92 | step_microstep: 18.66
[2025-04-26 06:42:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.63 | bwd: 5689.48 | bwd_inner: 5653.50 | bwd_allreduce: 35.94 | step: 18.66

  3%|▎         | 1063/41250 [2:34:19<96:30:37,  8.65s/it]
                                                         
{'loss': 0.1363, 'grad_norm': 1.573746919631958, 'learning_rate': 3.434571890145396e-05, 'epoch': 0.26}

  3%|▎         | 1063/41250 [2:34:19<96:30:37,  8.65s/it][2025-04-26 06:42:56,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 06:42:56,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.51 | bwd_microstep: 5908.12 | bwd_inner_microstep: 5695.75 | bwd_allreduce_microstep: 212.32 | step_microstep: 19.19
[2025-04-26 06:42:56,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.51 | bwd: 5908.14 | bwd_inner: 5695.75 | bwd_allreduce: 212.35 | step: 19.19

  3%|▎         | 1064/41250 [2:34:27<97:08:28,  8.70s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 2.3506968021392822, 'learning_rate': 3.437802907915994e-05, 'epoch': 0.26}

  3%|▎         | 1064/41250 [2:34:27<97:08:28,  8.70s/it][2025-04-26 06:43:05,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 06:43:05,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.86 | bwd_microstep: 5745.14 | bwd_inner_microstep: 5698.55 | bwd_allreduce_microstep: 46.55 | step_microstep: 18.42
[2025-04-26 06:43:05,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.86 | bwd: 5745.15 | bwd_inner: 5698.55 | bwd_allreduce: 46.57 | step: 18.42

  3%|▎         | 1065/41250 [2:34:36<97:03:08,  8.69s/it]
                                                         
{'loss': 0.1044, 'grad_norm': 1.0715596675872803, 'learning_rate': 3.441033925686592e-05, 'epoch': 0.26}

  3%|▎         | 1065/41250 [2:34:36<97:03:08,  8.69s/it][2025-04-26 06:43:14,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 06:43:14,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.34 | bwd_microstep: 5881.38 | bwd_inner_microstep: 5659.02 | bwd_allreduce_microstep: 222.31 | step_microstep: 18.97
[2025-04-26 06:43:14,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.34 | bwd: 5881.40 | bwd_inner: 5659.02 | bwd_allreduce: 222.33 | step: 18.97

  3%|▎         | 1066/41250 [2:34:45<97:26:01,  8.73s/it]
                                                         
{'loss': 0.1068, 'grad_norm': 2.531183958053589, 'learning_rate': 3.444264943457189e-05, 'epoch': 0.26}

  3%|▎         | 1066/41250 [2:34:45<97:26:01,  8.73s/it][2025-04-26 06:43:22,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.14
[2025-04-26 06:43:22,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.60 | bwd_microstep: 5706.56 | bwd_inner_microstep: 5662.51 | bwd_allreduce_microstep: 43.99 | step_microstep: 19.56
[2025-04-26 06:43:22,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 5706.57 | bwd_inner: 5662.51 | bwd_allreduce: 44.02 | step: 19.56

  3%|▎         | 1067/41250 [2:34:54<97:05:38,  8.70s/it]
                                                         
{'loss': 0.0803, 'grad_norm': 0.9904575943946838, 'learning_rate': 3.4474959612277866e-05, 'epoch': 0.26}

  3%|▎         | 1067/41250 [2:34:54<97:05:38,  8.70s/it][2025-04-26 06:43:31,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:43:31,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.54 | bwd_microstep: 5713.21 | bwd_inner_microstep: 5653.26 | bwd_allreduce_microstep: 59.91 | step_microstep: 19.06
[2025-04-26 06:43:31,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.54 | bwd: 5713.23 | bwd_inner: 5653.26 | bwd_allreduce: 59.93 | step: 19.06

  3%|▎         | 1068/41250 [2:35:02<96:51:52,  8.68s/it]
                                                         
{'loss': 0.2908, 'grad_norm': 1.9843250513076782, 'learning_rate': 3.450726978998385e-05, 'epoch': 0.26}

  3%|▎         | 1068/41250 [2:35:02<96:51:52,  8.68s/it][2025-04-26 06:43:40,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:43:40,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.24 | bwd_microstep: 5753.73 | bwd_inner_microstep: 5694.56 | bwd_allreduce_microstep: 59.12 | step_microstep: 18.79
[2025-04-26 06:43:40,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.24 | bwd: 5753.74 | bwd_inner: 5694.56 | bwd_allreduce: 59.14 | step: 18.79

  3%|▎         | 1069/41250 [2:35:11<96:54:22,  8.68s/it]
                                                         
{'loss': 0.0449, 'grad_norm': 0.5925006866455078, 'learning_rate': 3.453957996768983e-05, 'epoch': 0.26}

  3%|▎         | 1069/41250 [2:35:11<96:54:22,  8.68s/it][2025-04-26 06:43:48,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:43:48,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.62 | bwd_microstep: 5729.57 | bwd_inner_microstep: 5716.50 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.70
[2025-04-26 06:43:48,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.62 | bwd: 5729.59 | bwd_inner: 5716.50 | bwd_allreduce: 13.04 | step: 18.70

  3%|▎         | 1070/41250 [2:35:20<96:50:06,  8.68s/it]
                                                         
{'loss': 0.1083, 'grad_norm': 1.4640913009643555, 'learning_rate': 3.45718901453958e-05, 'epoch': 0.26}

  3%|▎         | 1070/41250 [2:35:20<96:50:06,  8.68s/it][2025-04-26 06:43:57,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:43:57,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.69 | bwd_microstep: 5750.83 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 44.58 | step_microstep: 18.67
[2025-04-26 06:43:57,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.69 | bwd: 5750.84 | bwd_inner: 5706.20 | bwd_allreduce: 44.60 | step: 18.67

  3%|▎         | 1071/41250 [2:35:28<96:51:45,  8.68s/it]
                                                         
{'loss': 0.1133, 'grad_norm': 1.2614362239837646, 'learning_rate': 3.460420032310178e-05, 'epoch': 0.26}

  3%|▎         | 1071/41250 [2:35:28<96:51:45,  8.68s/it][2025-04-26 06:44:06,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:44:06,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.23 | bwd_microstep: 5716.68 | bwd_inner_microstep: 5703.49 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.58
[2025-04-26 06:44:06,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.23 | bwd: 5716.69 | bwd_inner: 5703.49 | bwd_allreduce: 13.16 | step: 18.58

  3%|▎         | 1072/41250 [2:35:37<96:47:01,  8.67s/it]
                                                         
{'loss': 0.1056, 'grad_norm': 1.7391661405563354, 'learning_rate': 3.4636510500807756e-05, 'epoch': 0.26}

  3%|▎         | 1072/41250 [2:35:37<96:47:01,  8.67s/it][2025-04-26 06:44:14,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:44:14,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5777.07 | bwd_inner_microstep: 5688.21 | bwd_allreduce_microstep: 88.81 | step_microstep: 18.64
[2025-04-26 06:44:14,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5777.09 | bwd_inner: 5688.21 | bwd_allreduce: 88.83 | step: 18.64

  3%|▎         | 1073/41250 [2:35:46<96:52:00,  8.68s/it]
                                                         
{'loss': 0.328, 'grad_norm': 3.3365681171417236, 'learning_rate': 3.4668820678513734e-05, 'epoch': 0.26}

  3%|▎         | 1073/41250 [2:35:46<96:52:00,  8.68s/it][2025-04-26 06:44:23,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:44:23,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.65 | bwd_microstep: 5795.36 | bwd_inner_microstep: 5782.62 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.64
[2025-04-26 06:44:23,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.65 | bwd: 5795.37 | bwd_inner: 5782.62 | bwd_allreduce: 12.71 | step: 18.65

  3%|▎         | 1074/41250 [2:35:54<97:09:25,  8.71s/it]
                                                         
{'loss': 0.2044, 'grad_norm': 2.123842716217041, 'learning_rate': 3.470113085621971e-05, 'epoch': 0.26}

  3%|▎         | 1074/41250 [2:35:54<97:09:25,  8.71s/it][2025-04-26 06:44:32,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:44:32,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.87 | bwd_microstep: 5688.29 | bwd_inner_microstep: 5640.43 | bwd_allreduce_microstep: 47.80 | step_microstep: 18.64
[2025-04-26 06:44:32,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.87 | bwd: 5688.31 | bwd_inner: 5640.43 | bwd_allreduce: 47.83 | step: 18.65

  3%|▎         | 1075/41250 [2:36:03<96:51:26,  8.68s/it]
                                                         
{'loss': 0.2881, 'grad_norm': 1.9909591674804688, 'learning_rate': 3.473344103392569e-05, 'epoch': 0.26}

  3%|▎         | 1075/41250 [2:36:03<96:51:26,  8.68s/it][2025-04-26 06:44:40,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.24 | optimizer_step: 1.09
[2025-04-26 06:44:40,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.50 | bwd_microstep: 5770.53 | bwd_inner_microstep: 5706.20 | bwd_allreduce_microstep: 64.27 | step_microstep: 19.65
[2025-04-26 06:44:40,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.50 | bwd: 5770.55 | bwd_inner: 5706.20 | bwd_allreduce: 64.30 | step: 19.65

  3%|▎         | 1076/41250 [2:36:12<96:56:23,  8.69s/it]
                                                         
{'loss': 0.4235, 'grad_norm': 3.408952236175537, 'learning_rate': 3.476575121163167e-05, 'epoch': 0.26}

  3%|▎         | 1076/41250 [2:36:12<96:56:23,  8.69s/it][2025-04-26 06:44:49,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 06:44:49,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.56 | bwd_microstep: 5714.94 | bwd_inner_microstep: 5701.94 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.11
[2025-04-26 06:44:49,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.56 | bwd: 5714.95 | bwd_inner: 5701.94 | bwd_allreduce: 12.97 | step: 19.11

  3%|▎         | 1077/41250 [2:36:20<96:48:10,  8.67s/it]
                                                         
{'loss': 0.184, 'grad_norm': 2.3312525749206543, 'learning_rate': 3.4798061389337645e-05, 'epoch': 0.26}

  3%|▎         | 1077/41250 [2:36:20<96:48:10,  8.67s/it][2025-04-26 06:44:58,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:44:58,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.00 | bwd_microstep: 5726.35 | bwd_inner_microstep: 5696.13 | bwd_allreduce_microstep: 30.17 | step_microstep: 18.58
[2025-04-26 06:44:58,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.00 | bwd: 5726.36 | bwd_inner: 5696.13 | bwd_allreduce: 30.19 | step: 18.58

  3%|▎         | 1078/41250 [2:36:29<96:48:02,  8.67s/it]
                                                         
{'loss': 0.1762, 'grad_norm': 1.5899171829223633, 'learning_rate': 3.483037156704362e-05, 'epoch': 0.26}

  3%|▎         | 1078/41250 [2:36:29<96:48:02,  8.67s/it][2025-04-26 06:45:06,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 06:45:06,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.04 | bwd_microstep: 5712.53 | bwd_inner_microstep: 5699.77 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.76
[2025-04-26 06:45:06,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.04 | bwd: 5712.54 | bwd_inner: 5699.77 | bwd_allreduce: 12.74 | step: 18.76

  3%|▎         | 1079/41250 [2:36:38<96:43:09,  8.67s/it]
                                                         
{'loss': 0.3143, 'grad_norm': 4.213342666625977, 'learning_rate': 3.48626817447496e-05, 'epoch': 0.26}

  3%|▎         | 1079/41250 [2:36:38<96:43:09,  8.67s/it][2025-04-26 06:45:15,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-26 06:45:15,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.41 | bwd_microstep: 5928.94 | bwd_inner_microstep: 5663.78 | bwd_allreduce_microstep: 265.11 | step_microstep: 19.15
[2025-04-26 06:45:15,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.41 | bwd: 5928.95 | bwd_inner: 5663.78 | bwd_allreduce: 265.13 | step: 19.15

  3%|▎         | 1080/41250 [2:36:46<97:20:09,  8.72s/it]
                                                         
{'loss': 0.0615, 'grad_norm': 1.3064165115356445, 'learning_rate': 3.489499192245557e-05, 'epoch': 0.26}

  3%|▎         | 1080/41250 [2:36:46<97:20:09,  8.72s/it][2025-04-26 06:45:24,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 06:45:24,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.13 | bwd_microstep: 5687.89 | bwd_inner_microstep: 5662.96 | bwd_allreduce_microstep: 24.89 | step_microstep: 18.84
[2025-04-26 06:45:24,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.13 | bwd: 5687.91 | bwd_inner: 5662.96 | bwd_allreduce: 24.90 | step: 18.84

  3%|▎         | 1081/41250 [2:36:55<96:57:00,  8.69s/it]
                                                         
{'loss': 0.132, 'grad_norm': 1.9454872608184814, 'learning_rate': 3.492730210016155e-05, 'epoch': 0.26}

  3%|▎         | 1081/41250 [2:36:55<96:57:00,  8.69s/it][2025-04-26 06:45:33,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 06:45:33,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.09 | bwd_microstep: 5691.41 | bwd_inner_microstep: 5655.08 | bwd_allreduce_microstep: 36.28 | step_microstep: 18.56
[2025-04-26 06:45:33,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.09 | bwd: 5691.42 | bwd_inner: 5655.08 | bwd_allreduce: 36.30 | step: 18.56

  3%|▎         | 1082/41250 [2:37:04<96:39:03,  8.66s/it]
                                                         
{'loss': 0.1294, 'grad_norm': 2.050645112991333, 'learning_rate': 3.4959612277867535e-05, 'epoch': 0.26}

  3%|▎         | 1082/41250 [2:37:04<96:39:03,  8.66s/it][2025-04-26 06:45:41,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.24 | optimizer_step: 1.03
[2025-04-26 06:45:41,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.63 | bwd_microstep: 5758.64 | bwd_inner_microstep: 5656.43 | bwd_allreduce_microstep: 102.15 | step_microstep: 19.53
[2025-04-26 06:45:41,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.63 | bwd: 5758.65 | bwd_inner: 5656.43 | bwd_allreduce: 102.18 | step: 19.53

  3%|▎         | 1083/41250 [2:37:12<96:40:53,  8.67s/it]
                                                         
{'loss': 0.2969, 'grad_norm': 3.0123045444488525, 'learning_rate': 3.499192245557351e-05, 'epoch': 0.26}

  3%|▎         | 1083/41250 [2:37:12<96:40:53,  8.67s/it][2025-04-26 06:45:50,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:45:50,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.20 | bwd_microstep: 5785.87 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 133.00 | step_microstep: 18.56
[2025-04-26 06:45:50,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.20 | bwd: 5785.89 | bwd_inner: 5652.83 | bwd_allreduce: 133.02 | step: 18.56

  3%|▎         | 1084/41250 [2:37:21<96:49:07,  8.68s/it]
                                                         
{'loss': 0.074, 'grad_norm': 1.7698633670806885, 'learning_rate': 3.5024232633279484e-05, 'epoch': 0.26}

  3%|▎         | 1084/41250 [2:37:21<96:49:07,  8.68s/it][2025-04-26 06:45:59,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 06:45:59,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.83 | bwd_microstep: 5750.10 | bwd_inner_microstep: 5661.14 | bwd_allreduce_microstep: 88.91 | step_microstep: 18.86
[2025-04-26 06:45:59,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.83 | bwd: 5750.12 | bwd_inner: 5661.14 | bwd_allreduce: 88.93 | step: 18.87

  3%|▎         | 1085/41250 [2:37:30<96:48:56,  8.68s/it]
                                                         
{'loss': 0.1467, 'grad_norm': 2.386657238006592, 'learning_rate': 3.505654281098546e-05, 'epoch': 0.26}

  3%|▎         | 1085/41250 [2:37:30<96:48:56,  8.68s/it][2025-04-26 06:46:07,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 06:46:07,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5706.15 | bwd_inner_microstep: 5693.33 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.15
[2025-04-26 06:46:07,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5706.16 | bwd_inner: 5693.33 | bwd_allreduce: 12.79 | step: 19.15

  3%|▎         | 1086/41250 [2:37:38<96:40:21,  8.67s/it]
                                                         
{'loss': 0.355, 'grad_norm': 3.164583683013916, 'learning_rate': 3.508885298869144e-05, 'epoch': 0.26}

  3%|▎         | 1086/41250 [2:37:38<96:40:21,  8.67s/it][2025-04-26 06:46:16,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 06:46:16,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.38 | bwd_microstep: 5703.54 | bwd_inner_microstep: 5650.19 | bwd_allreduce_microstep: 53.30 | step_microstep: 18.66
[2025-04-26 06:46:16,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.38 | bwd: 5703.55 | bwd_inner: 5650.19 | bwd_allreduce: 53.32 | step: 18.66

  3%|▎         | 1087/41250 [2:37:47<96:28:09,  8.65s/it]
                                                         
{'loss': 0.2106, 'grad_norm': 2.749023199081421, 'learning_rate': 3.512116316639742e-05, 'epoch': 0.26}

  3%|▎         | 1087/41250 [2:37:47<96:28:09,  8.65s/it][2025-04-26 06:46:24,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:46:24,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.43 | bwd_microstep: 5694.45 | bwd_inner_microstep: 5681.63 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.63
[2025-04-26 06:46:24,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.43 | bwd: 5694.47 | bwd_inner: 5681.63 | bwd_allreduce: 12.79 | step: 18.63

  3%|▎         | 1088/41250 [2:37:56<96:23:06,  8.64s/it]
                                                         
{'loss': 0.0877, 'grad_norm': 1.2360711097717285, 'learning_rate': 3.5153473344103395e-05, 'epoch': 0.26}

  3%|▎         | 1088/41250 [2:37:56<96:23:06,  8.64s/it][2025-04-26 06:46:33,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 06:46:33,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.99 | bwd_microstep: 5711.77 | bwd_inner_microstep: 5634.82 | bwd_allreduce_microstep: 76.91 | step_microstep: 18.54
[2025-04-26 06:46:33,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.99 | bwd: 5711.79 | bwd_inner: 5634.82 | bwd_allreduce: 76.93 | step: 18.54

  3%|▎         | 1089/41250 [2:38:04<96:17:45,  8.63s/it]
                                                         
{'loss': 0.1785, 'grad_norm': 2.7466843128204346, 'learning_rate': 3.518578352180937e-05, 'epoch': 0.26}

  3%|▎         | 1089/41250 [2:38:04<96:17:45,  8.63s/it][2025-04-26 06:46:42,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:46:42,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.79 | bwd_microstep: 5861.23 | bwd_inner_microstep: 5682.41 | bwd_allreduce_microstep: 178.77 | step_microstep: 18.64
[2025-04-26 06:46:42,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.79 | bwd: 5861.25 | bwd_inner: 5682.41 | bwd_allreduce: 178.79 | step: 18.64

  3%|▎         | 1090/41250 [2:38:13<96:48:22,  8.68s/it]
                                                         
{'loss': 0.1971, 'grad_norm': 2.075368881225586, 'learning_rate': 3.521809369951535e-05, 'epoch': 0.26}

  3%|▎         | 1090/41250 [2:38:13<96:48:22,  8.68s/it][2025-04-26 06:46:50,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.79 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:46:50,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.67 | bwd_microstep: 5759.02 | bwd_inner_microstep: 5648.01 | bwd_allreduce_microstep: 110.96 | step_microstep: 17.82
[2025-04-26 06:46:50,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.67 | bwd: 5759.03 | bwd_inner: 5648.01 | bwd_allreduce: 110.98 | step: 17.82

  3%|▎         | 1091/41250 [2:38:22<96:42:43,  8.67s/it]
                                                         
{'loss': 0.183, 'grad_norm': 2.594757080078125, 'learning_rate': 3.525040387722133e-05, 'epoch': 0.26}

  3%|▎         | 1091/41250 [2:38:22<96:42:43,  8.67s/it][2025-04-26 06:46:59,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 06:46:59,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.01 | bwd_microstep: 5681.93 | bwd_inner_microstep: 5638.94 | bwd_allreduce_microstep: 42.94 | step_microstep: 19.06
[2025-04-26 06:46:59,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.98 | bwd: 5681.94 | bwd_inner: 5638.94 | bwd_allreduce: 42.96 | step: 19.07

  3%|▎         | 1092/41250 [2:38:30<96:28:17,  8.65s/it]
                                                         
{'loss': 0.0468, 'grad_norm': 0.6614208221435547, 'learning_rate': 3.528271405492731e-05, 'epoch': 0.26}

  3%|▎         | 1092/41250 [2:38:30<96:28:17,  8.65s/it][2025-04-26 06:47:08,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 06:47:08,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.96 | bwd_microstep: 5870.11 | bwd_inner_microstep: 5678.11 | bwd_allreduce_microstep: 191.95 | step_microstep: 19.00
[2025-04-26 06:47:08,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.96 | bwd: 5870.13 | bwd_inner: 5678.11 | bwd_allreduce: 191.98 | step: 19.00

  3%|▎         | 1093/41250 [2:38:39<96:56:36,  8.69s/it]
                                                         
{'loss': 0.0614, 'grad_norm': 1.1541751623153687, 'learning_rate': 3.5315024232633285e-05, 'epoch': 0.26}

  3%|▎         | 1093/41250 [2:38:39<96:56:36,  8.69s/it][2025-04-26 06:47:17,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 06:47:17,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.71 | bwd_microstep: 5729.16 | bwd_inner_microstep: 5706.34 | bwd_allreduce_microstep: 22.77 | step_microstep: 18.60
[2025-04-26 06:47:17,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.71 | bwd: 5729.17 | bwd_inner: 5706.34 | bwd_allreduce: 22.79 | step: 18.60

  3%|▎         | 1094/41250 [2:38:48<96:49:39,  8.68s/it]
                                                         
{'loss': 0.3219, 'grad_norm': 3.3954381942749023, 'learning_rate': 3.5347334410339256e-05, 'epoch': 0.27}

  3%|▎         | 1094/41250 [2:38:48<96:49:39,  8.68s/it][2025-04-26 06:47:25,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 06:47:25,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.52 | bwd_microstep: 5741.48 | bwd_inner_microstep: 5637.19 | bwd_allreduce_microstep: 104.24 | step_microstep: 18.48
[2025-04-26 06:47:25,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.52 | bwd: 5741.49 | bwd_inner: 5637.19 | bwd_allreduce: 104.26 | step: 18.48

  3%|▎         | 1095/41250 [2:38:56<96:43:10,  8.67s/it]
                                                         
{'loss': 0.0302, 'grad_norm': 0.8068628907203674, 'learning_rate': 3.5379644588045234e-05, 'epoch': 0.27}

  3%|▎         | 1095/41250 [2:38:56<96:43:10,  8.67s/it][2025-04-26 06:47:34,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:47:34,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.09 | bwd_microstep: 5700.70 | bwd_inner_microstep: 5687.87 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.68
[2025-04-26 06:47:34,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.09 | bwd: 5700.71 | bwd_inner: 5687.87 | bwd_allreduce: 12.80 | step: 18.68

  3%|▎         | 1096/41250 [2:39:05<96:34:44,  8.66s/it]
                                                         
{'loss': 0.1834, 'grad_norm': 3.996337652206421, 'learning_rate': 3.541195476575122e-05, 'epoch': 0.27}

  3%|▎         | 1096/41250 [2:39:05<96:34:44,  8.66s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x2c528040] moov atom not found
[06:47:34] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4...
[2025-04-26 06:47:42,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-26 06:47:42,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.71 | bwd_microstep: 5756.50 | bwd_inner_microstep: 5641.19 | bwd_allreduce_microstep: 115.25 | step_microstep: 18.90
[2025-04-26 06:47:42,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.71 | bwd: 5756.52 | bwd_inner: 5641.19 | bwd_allreduce: 115.28 | step: 18.90

  3%|▎         | 1097/41250 [2:39:14<96:35:36,  8.66s/it]
                                                         
{'loss': 0.6623, 'grad_norm': 6.518703460693359, 'learning_rate': 3.5444264943457196e-05, 'epoch': 0.27}

  3%|▎         | 1097/41250 [2:39:14<96:35:36,  8.66s/it][2025-04-26 06:47:51,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-26 06:47:51,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.83 | bwd_microstep: 5777.76 | bwd_inner_microstep: 5765.14 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.69
[2025-04-26 06:47:51,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.83 | bwd: 5777.77 | bwd_inner: 5765.14 | bwd_allreduce: 12.58 | step: 18.69

  3%|▎         | 1098/41250 [2:39:22<96:52:22,  8.69s/it]
                                                         
{'loss': 0.1438, 'grad_norm': 1.86737060546875, 'learning_rate': 3.547657512116317e-05, 'epoch': 0.27}

  3%|▎         | 1098/41250 [2:39:22<96:52:22,  8.69s/it][2025-04-26 06:48:00,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:48:00,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.15 | bwd_microstep: 5754.66 | bwd_inner_microstep: 5687.50 | bwd_allreduce_microstep: 67.12 | step_microstep: 18.85
[2025-04-26 06:48:00,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5754.67 | bwd_inner: 5687.50 | bwd_allreduce: 67.13 | step: 18.86

  3%|▎         | 1099/41250 [2:39:31<96:51:45,  8.68s/it]
                                                         
{'loss': 0.1501, 'grad_norm': 2.8708407878875732, 'learning_rate': 3.5508885298869145e-05, 'epoch': 0.27}

  3%|▎         | 1099/41250 [2:39:31<96:51:45,  8.68s/it][2025-04-26 06:48:09,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:48:09,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.55 | bwd_microstep: 5752.16 | bwd_inner_microstep: 5650.86 | bwd_allreduce_microstep: 101.25 | step_microstep: 18.77
[2025-04-26 06:48:09,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.55 | bwd: 5752.17 | bwd_inner: 5650.86 | bwd_allreduce: 101.27 | step: 18.78

  3%|▎         | 1100/41250 [2:39:40<96:44:09,  8.67s/it]
                                                         
{'loss': 0.4075, 'grad_norm': 4.447290420532227, 'learning_rate': 3.554119547657512e-05, 'epoch': 0.27}

  3%|▎         | 1100/41250 [2:39:40<96:44:09,  8.67s/it][2025-04-26 06:48:17,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:48:17,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.33 | bwd_microstep: 5693.92 | bwd_inner_microstep: 5681.23 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.75
[2025-04-26 06:48:17,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.33 | bwd: 5693.93 | bwd_inner: 5681.23 | bwd_allreduce: 12.66 | step: 18.75

  3%|▎         | 1101/41250 [2:39:48<96:34:07,  8.66s/it]
                                                         
{'loss': 0.4294, 'grad_norm': 8.258342742919922, 'learning_rate': 3.55735056542811e-05, 'epoch': 0.27}

  3%|▎         | 1101/41250 [2:39:48<96:34:07,  8.66s/it][2025-04-26 06:48:26,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.21
[2025-04-26 06:48:26,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.77 | bwd_microstep: 5702.85 | bwd_inner_microstep: 5653.29 | bwd_allreduce_microstep: 49.51 | step_microstep: 19.24
[2025-04-26 06:48:26,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.77 | bwd: 5702.87 | bwd_inner: 5653.29 | bwd_allreduce: 49.53 | step: 19.25

  3%|▎         | 1102/41250 [2:39:57<96:24:54,  8.65s/it]
                                                         
{'loss': 0.1599, 'grad_norm': 2.6701951026916504, 'learning_rate': 3.560581583198708e-05, 'epoch': 0.27}

  3%|▎         | 1102/41250 [2:39:57<96:24:54,  8.65s/it][2025-04-26 06:48:34,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:48:34,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5736.11 | bwd_inner_microstep: 5709.26 | bwd_allreduce_microstep: 26.80 | step_microstep: 18.81
[2025-04-26 06:48:34,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5736.13 | bwd_inner: 5709.26 | bwd_allreduce: 26.82 | step: 18.81

  3%|▎         | 1103/41250 [2:40:06<96:29:37,  8.65s/it]
                                                         
{'loss': 0.2144, 'grad_norm': 3.9593749046325684, 'learning_rate': 3.563812600969306e-05, 'epoch': 0.27}

  3%|▎         | 1103/41250 [2:40:06<96:29:37,  8.65s/it][2025-04-26 06:48:43,625] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:48:43,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.94 | bwd_microstep: 5741.60 | bwd_inner_microstep: 5686.36 | bwd_allreduce_microstep: 55.19 | step_microstep: 18.44
[2025-04-26 06:48:43,626] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.94 | bwd: 5741.61 | bwd_inner: 5686.36 | bwd_allreduce: 55.21 | step: 18.44

  3%|▎         | 1104/41250 [2:40:14<96:33:12,  8.66s/it]
                                                         
{'loss': 0.2978, 'grad_norm': 1.8881034851074219, 'learning_rate': 3.5670436187399035e-05, 'epoch': 0.27}

  3%|▎         | 1104/41250 [2:40:14<96:33:12,  8.66s/it][2025-04-26 06:48:52,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.34 | optimizer_step: 1.05
[2025-04-26 06:48:52,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.88 | bwd_microstep: 5697.24 | bwd_inner_microstep: 5683.24 | bwd_allreduce_microstep: 13.94 | step_microstep: 19.99
[2025-04-26 06:48:52,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.88 | bwd: 5697.25 | bwd_inner: 5683.24 | bwd_allreduce: 13.97 | step: 19.99

  3%|▎         | 1105/41250 [2:40:23<96:24:41,  8.65s/it]
                                                         
{'loss': 0.4022, 'grad_norm': 3.9881131649017334, 'learning_rate': 3.570274636510501e-05, 'epoch': 0.27}

  3%|▎         | 1105/41250 [2:40:23<96:24:41,  8.65s/it][2025-04-26 06:49:00,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 06:49:00,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.05 | bwd_microstep: 5760.74 | bwd_inner_microstep: 5688.40 | bwd_allreduce_microstep: 72.30 | step_microstep: 18.92
[2025-04-26 06:49:00,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.05 | bwd: 5760.75 | bwd_inner: 5688.39 | bwd_allreduce: 72.32 | step: 18.92

  3%|▎         | 1106/41250 [2:40:32<96:34:17,  8.66s/it]
                                                         
{'loss': 0.144, 'grad_norm': 2.1964111328125, 'learning_rate': 3.573505654281099e-05, 'epoch': 0.27}

  3%|▎         | 1106/41250 [2:40:32<96:34:17,  8.66s/it][2025-04-26 06:49:09,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-26 06:49:09,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.25 | bwd_microstep: 5741.32 | bwd_inner_microstep: 5691.63 | bwd_allreduce_microstep: 49.64 | step_microstep: 18.57
[2025-04-26 06:49:09,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.25 | bwd: 5741.34 | bwd_inner: 5691.64 | bwd_allreduce: 49.66 | step: 18.57

  3%|▎         | 1107/41250 [2:40:40<96:34:58,  8.66s/it]
                                                         
{'loss': 0.1638, 'grad_norm': 1.6629619598388672, 'learning_rate': 3.576736672051697e-05, 'epoch': 0.27}

  3%|▎         | 1107/41250 [2:40:40<96:34:58,  8.66s/it][2025-04-26 06:49:18,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:49:18,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.78 | bwd_microstep: 5896.50 | bwd_inner_microstep: 5699.15 | bwd_allreduce_microstep: 197.31 | step_microstep: 18.65
[2025-04-26 06:49:18,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.78 | bwd: 5896.51 | bwd_inner: 5699.15 | bwd_allreduce: 197.32 | step: 18.65

  3%|▎         | 1108/41250 [2:40:49<97:07:54,  8.71s/it]
                                                         
{'loss': 0.5142, 'grad_norm': 4.059050559997559, 'learning_rate': 3.579967689822294e-05, 'epoch': 0.27}

  3%|▎         | 1108/41250 [2:40:49<97:07:54,  8.71s/it][2025-04-26 06:49:27,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:49:27,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.02 | bwd_microstep: 5759.25 | bwd_inner_microstep: 5657.25 | bwd_allreduce_microstep: 101.95 | step_microstep: 18.58
[2025-04-26 06:49:27,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.02 | bwd: 5759.26 | bwd_inner: 5657.25 | bwd_allreduce: 101.96 | step: 18.58

  3%|▎         | 1109/41250 [2:40:58<96:56:53,  8.69s/it]
                                                         
{'loss': 0.1803, 'grad_norm': 2.0436041355133057, 'learning_rate': 3.583198707592892e-05, 'epoch': 0.27}

  3%|▎         | 1109/41250 [2:40:58<96:56:53,  8.69s/it][2025-04-26 06:49:35,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 06:49:35,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.40 | bwd_microstep: 5716.50 | bwd_inner_microstep: 5704.04 | bwd_allreduce_microstep: 12.40 | step_microstep: 19.76
[2025-04-26 06:49:35,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.40 | bwd: 5716.51 | bwd_inner: 5704.04 | bwd_allreduce: 12.42 | step: 19.76

  3%|▎         | 1110/41250 [2:41:06<96:46:49,  8.68s/it]
                                                         
{'loss': 0.176, 'grad_norm': 2.878765344619751, 'learning_rate': 3.58642972536349e-05, 'epoch': 0.27}

  3%|▎         | 1110/41250 [2:41:06<96:46:49,  8.68s/it][2025-04-26 06:49:44,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:49:44,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.42 | bwd_microstep: 5772.05 | bwd_inner_microstep: 5713.83 | bwd_allreduce_microstep: 58.18 | step_microstep: 18.59
[2025-04-26 06:49:44,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.42 | bwd: 5772.06 | bwd_inner: 5713.83 | bwd_allreduce: 58.20 | step: 18.59

  3%|▎         | 1111/41250 [2:41:15<96:52:22,  8.69s/it]
                                                         
{'loss': 0.1498, 'grad_norm': 2.6406984329223633, 'learning_rate': 3.589660743134088e-05, 'epoch': 0.27}

  3%|▎         | 1111/41250 [2:41:15<96:52:22,  8.69s/it][2025-04-26 06:49:53,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.10
[2025-04-26 06:49:53,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.45 | bwd_microstep: 5913.72 | bwd_inner_microstep: 5644.23 | bwd_allreduce_microstep: 269.45 | step_microstep: 18.90
[2025-04-26 06:49:53,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.45 | bwd: 5913.73 | bwd_inner: 5644.23 | bwd_allreduce: 269.46 | step: 18.91

  3%|▎         | 1112/41250 [2:41:24<97:19:05,  8.73s/it]
                                                         
{'loss': 0.2929, 'grad_norm': 2.682889223098755, 'learning_rate': 3.592891760904685e-05, 'epoch': 0.27}

  3%|▎         | 1112/41250 [2:41:24<97:19:05,  8.73s/it][2025-04-26 06:50:01,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:50:01,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.49 | bwd_microstep: 5729.11 | bwd_inner_microstep: 5646.54 | bwd_allreduce_microstep: 82.53 | step_microstep: 18.65
[2025-04-26 06:50:01,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.49 | bwd: 5729.12 | bwd_inner: 5646.54 | bwd_allreduce: 82.54 | step: 18.66

  3%|▎         | 1113/41250 [2:41:33<97:01:29,  8.70s/it]
                                                         
{'loss': 0.1129, 'grad_norm': 1.0576587915420532, 'learning_rate': 3.596122778675283e-05, 'epoch': 0.27}

  3%|▎         | 1113/41250 [2:41:33<97:01:29,  8.70s/it][2025-04-26 06:50:10,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:50:10,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.16 | bwd_microstep: 5783.10 | bwd_inner_microstep: 5695.59 | bwd_allreduce_microstep: 87.47 | step_microstep: 18.46
[2025-04-26 06:50:10,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.16 | bwd: 5783.12 | bwd_inner: 5695.59 | bwd_allreduce: 87.49 | step: 18.47

  3%|▎         | 1114/41250 [2:41:41<97:03:54,  8.71s/it]
                                                         
{'loss': 0.1668, 'grad_norm': 1.161884069442749, 'learning_rate': 3.599353796445881e-05, 'epoch': 0.27}

  3%|▎         | 1114/41250 [2:41:41<97:03:54,  8.71s/it][2025-04-26 06:50:19,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.11 | optimizer_step: 1.22
[2025-04-26 06:50:19,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.17 | bwd_microstep: 5766.66 | bwd_inner_microstep: 5651.39 | bwd_allreduce_microstep: 115.22 | step_microstep: 19.77
[2025-04-26 06:50:19,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.17 | bwd: 5766.68 | bwd_inner: 5651.39 | bwd_allreduce: 115.24 | step: 19.77

  3%|▎         | 1115/41250 [2:41:50<97:01:56,  8.70s/it]
                                                         
{'loss': 0.2069, 'grad_norm': 1.8874856233596802, 'learning_rate': 3.6025848142164785e-05, 'epoch': 0.27}

  3%|▎         | 1115/41250 [2:41:50<97:01:56,  8.70s/it][2025-04-26 06:50:28,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 06:50:28,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.81 | bwd_microstep: 5902.36 | bwd_inner_microstep: 5700.21 | bwd_allreduce_microstep: 202.10 | step_microstep: 18.60
[2025-04-26 06:50:28,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.81 | bwd: 5902.37 | bwd_inner: 5700.21 | bwd_allreduce: 202.12 | step: 18.60

  3%|▎         | 1116/41250 [2:41:59<97:29:43,  8.75s/it]
                                                         
{'loss': 0.2526, 'grad_norm': 2.608689546585083, 'learning_rate': 3.605815831987076e-05, 'epoch': 0.27}

  3%|▎         | 1116/41250 [2:41:59<97:29:43,  8.75s/it][2025-04-26 06:50:36,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:50:36,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.68 | bwd_microstep: 5743.57 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 47.85 | step_microstep: 19.02
[2025-04-26 06:50:36,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.68 | bwd: 5743.59 | bwd_inner: 5695.67 | bwd_allreduce: 47.87 | step: 19.03

  3%|▎         | 1117/41250 [2:42:08<97:18:24,  8.73s/it]
                                                         
{'loss': 0.2205, 'grad_norm': 1.4857627153396606, 'learning_rate': 3.609046849757674e-05, 'epoch': 0.27}

  3%|▎         | 1117/41250 [2:42:08<97:18:24,  8.73s/it][2025-04-26 06:50:45,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 06:50:45,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.59 | bwd_microstep: 5815.92 | bwd_inner_microstep: 5661.79 | bwd_allreduce_microstep: 154.09 | step_microstep: 18.88
[2025-04-26 06:50:45,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.59 | bwd: 5815.94 | bwd_inner: 5661.79 | bwd_allreduce: 154.11 | step: 18.89

  3%|▎         | 1118/41250 [2:42:16<97:16:11,  8.73s/it]
                                                         
{'loss': 0.186, 'grad_norm': 1.5220253467559814, 'learning_rate': 3.612277867528272e-05, 'epoch': 0.27}

  3%|▎         | 1118/41250 [2:42:16<97:16:11,  8.73s/it][2025-04-26 06:50:54,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 06:50:54,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.07 | bwd_microstep: 5725.94 | bwd_inner_microstep: 5654.39 | bwd_allreduce_microstep: 71.50 | step_microstep: 18.75
[2025-04-26 06:50:54,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.07 | bwd: 5725.95 | bwd_inner: 5654.39 | bwd_allreduce: 71.52 | step: 18.75

  3%|▎         | 1119/41250 [2:42:25<96:59:15,  8.70s/it]
                                                         
{'loss': 0.1445, 'grad_norm': 1.3462779521942139, 'learning_rate': 3.6155088852988696e-05, 'epoch': 0.27}

  3%|▎         | 1119/41250 [2:42:25<96:59:15,  8.70s/it][2025-04-26 06:51:02,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-26 06:51:02,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.05 | bwd_microstep: 5774.14 | bwd_inner_microstep: 5661.82 | bwd_allreduce_microstep: 112.28 | step_microstep: 18.28
[2025-04-26 06:51:02,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.05 | bwd: 5774.15 | bwd_inner: 5661.82 | bwd_allreduce: 112.30 | step: 18.29

  3%|▎         | 1120/41250 [2:42:34<96:55:49,  8.70s/it]
                                                         
{'loss': 0.1795, 'grad_norm': 1.6972169876098633, 'learning_rate': 3.6187399030694674e-05, 'epoch': 0.27}

  3%|▎         | 1120/41250 [2:42:34<96:55:49,  8.70s/it][2025-04-26 06:51:11,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:51:11,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.23 | bwd_microstep: 5767.15 | bwd_inner_microstep: 5709.20 | bwd_allreduce_microstep: 57.90 | step_microstep: 18.67
[2025-04-26 06:51:11,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.23 | bwd: 5767.16 | bwd_inner: 5709.20 | bwd_allreduce: 57.92 | step: 18.67

  3%|▎         | 1121/41250 [2:42:42<96:56:53,  8.70s/it]
                                                         
{'loss': 0.1906, 'grad_norm': 2.4672129154205322, 'learning_rate': 3.621970920840065e-05, 'epoch': 0.27}

  3%|▎         | 1121/41250 [2:42:42<96:56:53,  8.70s/it][2025-04-26 06:51:20,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 06:51:20,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.01 | bwd_microstep: 5790.79 | bwd_inner_microstep: 5655.92 | bwd_allreduce_microstep: 134.82 | step_microstep: 18.97
[2025-04-26 06:51:20,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.01 | bwd: 5790.81 | bwd_inner: 5655.92 | bwd_allreduce: 134.84 | step: 18.97

  3%|▎         | 1122/41250 [2:42:51<96:58:29,  8.70s/it]
                                                         
{'loss': 0.2014, 'grad_norm': 2.035287618637085, 'learning_rate': 3.625201938610662e-05, 'epoch': 0.27}

  3%|▎         | 1122/41250 [2:42:51<96:58:29,  8.70s/it][2025-04-26 06:51:28,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 06:51:28,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.38 | bwd_microstep: 5758.61 | bwd_inner_microstep: 5689.44 | bwd_allreduce_microstep: 69.13 | step_microstep: 19.37
[2025-04-26 06:51:28,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.38 | bwd: 5758.63 | bwd_inner: 5689.44 | bwd_allreduce: 69.14 | step: 19.38

  3%|▎         | 1123/41250 [2:43:00<96:54:54,  8.69s/it]
                                                         
{'loss': 0.2257, 'grad_norm': 2.3246114253997803, 'learning_rate': 3.62843295638126e-05, 'epoch': 0.27}

  3%|▎         | 1123/41250 [2:43:00<96:54:54,  8.69s/it][2025-04-26 06:51:37,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-26 06:51:37,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 5744.62 | bwd_inner_microstep: 5712.54 | bwd_allreduce_microstep: 32.03 | step_microstep: 18.77
[2025-04-26 06:51:37,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 5744.63 | bwd_inner: 5712.54 | bwd_allreduce: 32.05 | step: 18.77

  3%|▎         | 1124/41250 [2:43:08<96:51:10,  8.69s/it]
                                                         
{'loss': 0.2048, 'grad_norm': 1.6598303318023682, 'learning_rate': 3.6316639741518586e-05, 'epoch': 0.27}

  3%|▎         | 1124/41250 [2:43:08<96:51:10,  8.69s/it][2025-04-26 06:51:46,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 06:51:46,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.52 | bwd_microstep: 5751.52 | bwd_inner_microstep: 5684.42 | bwd_allreduce_microstep: 67.05 | step_microstep: 18.60
[2025-04-26 06:51:46,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.52 | bwd: 5751.53 | bwd_inner: 5684.42 | bwd_allreduce: 67.07 | step: 18.60

  3%|▎         | 1125/41250 [2:43:17<96:50:25,  8.69s/it]
                                                         
{'loss': 0.2074, 'grad_norm': 3.428298234939575, 'learning_rate': 3.634894991922456e-05, 'epoch': 0.27}

  3%|▎         | 1125/41250 [2:43:17<96:50:25,  8.69s/it][2025-04-26 06:51:55,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:51:55,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.58 | bwd_microstep: 5878.16 | bwd_inner_microstep: 5865.27 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.77
[2025-04-26 06:51:55,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.58 | bwd: 5878.17 | bwd_inner: 5865.27 | bwd_allreduce: 12.85 | step: 18.78

  3%|▎         | 1126/41250 [2:43:26<97:31:19,  8.75s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 2.030444383621216, 'learning_rate': 3.6381260096930535e-05, 'epoch': 0.27}

  3%|▎         | 1126/41250 [2:43:26<97:31:19,  8.75s/it][2025-04-26 06:52:03,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.05 | optimizer_step: 0.91
[2025-04-26 06:52:03,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.16 | bwd_microstep: 5792.02 | bwd_inner_microstep: 5656.66 | bwd_allreduce_microstep: 135.29 | step_microstep: 19.08
[2025-04-26 06:52:03,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.16 | bwd: 5792.04 | bwd_inner: 5656.66 | bwd_allreduce: 135.32 | step: 19.08

  3%|▎         | 1127/41250 [2:43:35<97:19:53,  8.73s/it]
                                                         
{'loss': 0.1759, 'grad_norm': 1.802786111831665, 'learning_rate': 3.641357027463651e-05, 'epoch': 0.27}

  3%|▎         | 1127/41250 [2:43:35<97:19:53,  8.73s/it][2025-04-26 06:52:12,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 06:52:12,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.20 | bwd_microstep: 5898.84 | bwd_inner_microstep: 5685.34 | bwd_allreduce_microstep: 213.44 | step_microstep: 18.95
[2025-04-26 06:52:12,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.20 | bwd: 5898.85 | bwd_inner: 5685.34 | bwd_allreduce: 213.47 | step: 18.95

  3%|▎         | 1128/41250 [2:43:43<97:38:02,  8.76s/it]
                                                         
{'loss': 0.2132, 'grad_norm': 2.6300792694091797, 'learning_rate': 3.644588045234249e-05, 'epoch': 0.27}

  3%|▎         | 1128/41250 [2:43:43<97:38:02,  8.76s/it][2025-04-26 06:52:21,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-26 06:52:21,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5753.02 | bwd_inner_microstep: 5704.54 | bwd_allreduce_microstep: 48.43 | step_microstep: 18.82
[2025-04-26 06:52:21,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5753.04 | bwd_inner: 5704.54 | bwd_allreduce: 48.45 | step: 18.83

  3%|▎         | 1129/41250 [2:43:52<97:23:17,  8.74s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 2.051879644393921, 'learning_rate': 3.647819063004847e-05, 'epoch': 0.27}

  3%|▎         | 1129/41250 [2:43:52<97:23:17,  8.74s/it][2025-04-26 06:52:30,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:52:30,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.90 | bwd_microstep: 5754.42 | bwd_inner_microstep: 5696.85 | bwd_allreduce_microstep: 57.52 | step_microstep: 18.61
[2025-04-26 06:52:30,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.91 | bwd: 5754.44 | bwd_inner: 5696.85 | bwd_allreduce: 57.55 | step: 18.61

  3%|▎         | 1130/41250 [2:44:01<97:14:16,  8.73s/it]
                                                         
{'loss': 0.2967, 'grad_norm': 2.081202507019043, 'learning_rate': 3.6510500807754446e-05, 'epoch': 0.27}

  3%|▎         | 1130/41250 [2:44:01<97:14:16,  8.73s/it][2025-04-26 06:52:38,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 06:52:38,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.99 | bwd_microstep: 5870.90 | bwd_inner_microstep: 5710.82 | bwd_allreduce_microstep: 160.02 | step_microstep: 18.75
[2025-04-26 06:52:38,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.99 | bwd: 5870.92 | bwd_inner: 5710.82 | bwd_allreduce: 160.05 | step: 18.75

  3%|▎         | 1131/41250 [2:44:10<97:33:17,  8.75s/it]
                                                         
{'loss': 0.2132, 'grad_norm': 2.2985925674438477, 'learning_rate': 3.6542810985460424e-05, 'epoch': 0.27}

  3%|▎         | 1131/41250 [2:44:10<97:33:17,  8.75s/it][2025-04-26 06:52:47,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.10
[2025-04-26 06:52:47,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.97 | bwd_microstep: 5756.17 | bwd_inner_microstep: 5653.72 | bwd_allreduce_microstep: 102.41 | step_microstep: 18.87
[2025-04-26 06:52:47,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.97 | bwd: 5756.19 | bwd_inner: 5653.72 | bwd_allreduce: 102.43 | step: 18.87

  3%|▎         | 1132/41250 [2:44:18<97:17:29,  8.73s/it]
                                                         
{'loss': 0.4179, 'grad_norm': 3.1956589221954346, 'learning_rate': 3.6575121163166395e-05, 'epoch': 0.27}

  3%|▎         | 1132/41250 [2:44:18<97:17:29,  8.73s/it][2025-04-26 06:52:56,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 06:52:56,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.66 | bwd_microstep: 5771.04 | bwd_inner_microstep: 5757.55 | bwd_allreduce_microstep: 13.43 | step_microstep: 18.90
[2025-04-26 06:52:56,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.66 | bwd: 5771.05 | bwd_inner: 5757.55 | bwd_allreduce: 13.46 | step: 18.90

  3%|▎         | 1133/41250 [2:44:27<97:19:33,  8.73s/it]
                                                         
{'loss': 0.2762, 'grad_norm': 2.6363627910614014, 'learning_rate': 3.660743134087238e-05, 'epoch': 0.27}

  3%|▎         | 1133/41250 [2:44:27<97:19:33,  8.73s/it][2025-04-26 06:53:05,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 06:53:05,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5882.75 | bwd_inner_microstep: 5649.28 | bwd_allreduce_microstep: 233.43 | step_microstep: 18.53
[2025-04-26 06:53:05,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5882.77 | bwd_inner: 5649.28 | bwd_allreduce: 233.45 | step: 18.53

  3%|▎         | 1134/41250 [2:44:36<97:32:30,  8.75s/it]
                                                         
{'loss': 0.1427, 'grad_norm': 1.1898698806762695, 'learning_rate': 3.663974151857836e-05, 'epoch': 0.27}

  3%|▎         | 1134/41250 [2:44:36<97:32:30,  8.75s/it][2025-04-26 06:53:14,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 06:53:14,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.08 | bwd_microstep: 5869.58 | bwd_inner_microstep: 5856.63 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.74
[2025-04-26 06:53:14,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.08 | bwd: 5869.59 | bwd_inner: 5856.63 | bwd_allreduce: 12.91 | step: 18.74

  3%|▎         | 1135/41250 [2:44:45<98:00:53,  8.80s/it]
                                                         
{'loss': 0.1751, 'grad_norm': 2.0073513984680176, 'learning_rate': 3.6672051696284335e-05, 'epoch': 0.28}

  3%|▎         | 1135/41250 [2:44:45<98:00:53,  8.80s/it][2025-04-26 06:53:22,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.08 | optimizer_step: 0.97
[2025-04-26 06:53:22,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.51 | bwd_microstep: 5704.25 | bwd_inner_microstep: 5691.06 | bwd_allreduce_microstep: 13.12 | step_microstep: 19.37
[2025-04-26 06:53:22,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.51 | bwd: 5704.26 | bwd_inner: 5691.06 | bwd_allreduce: 13.15 | step: 19.37

  3%|▎         | 1136/41250 [2:44:53<97:27:07,  8.75s/it]
                                                         
{'loss': 0.2216, 'grad_norm': 2.169086217880249, 'learning_rate': 3.6704361873990307e-05, 'epoch': 0.28}

  3%|▎         | 1136/41250 [2:44:53<97:27:07,  8.75s/it][2025-04-26 06:53:31,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 06:53:31,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.03 | bwd_microstep: 5745.02 | bwd_inner_microstep: 5643.75 | bwd_allreduce_microstep: 101.22 | step_microstep: 19.21
[2025-04-26 06:53:31,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.03 | bwd: 5745.04 | bwd_inner: 5643.75 | bwd_allreduce: 101.24 | step: 19.21

  3%|▎         | 1137/41250 [2:45:02<97:09:01,  8.72s/it]
                                                         
{'loss': 0.054, 'grad_norm': 0.7837592363357544, 'learning_rate': 3.6736672051696284e-05, 'epoch': 0.28}

  3%|▎         | 1137/41250 [2:45:02<97:09:01,  8.72s/it][2025-04-26 06:53:40,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 06:53:40,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5744.24 | bwd_inner_microstep: 5688.48 | bwd_allreduce_microstep: 55.71 | step_microstep: 19.02
[2025-04-26 06:53:40,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5744.25 | bwd_inner: 5688.48 | bwd_allreduce: 55.73 | step: 19.02

  3%|▎         | 1138/41250 [2:45:11<96:59:50,  8.71s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 2.155569314956665, 'learning_rate': 3.676898222940227e-05, 'epoch': 0.28}

  3%|▎         | 1138/41250 [2:45:11<96:59:50,  8.71s/it][2025-04-26 06:53:48,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 06:53:48,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.23 | bwd_microstep: 5778.49 | bwd_inner_microstep: 5650.52 | bwd_allreduce_microstep: 127.93 | step_microstep: 18.40
[2025-04-26 06:53:48,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.23 | bwd: 5778.50 | bwd_inner: 5650.52 | bwd_allreduce: 127.95 | step: 18.40

  3%|▎         | 1139/41250 [2:45:19<96:56:13,  8.70s/it]
                                                         
{'loss': 0.389, 'grad_norm': 2.6672794818878174, 'learning_rate': 3.680129240710824e-05, 'epoch': 0.28}

  3%|▎         | 1139/41250 [2:45:19<96:56:13,  8.70s/it][2025-04-26 06:53:57,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.27 | optimizer_step: 1.04
[2025-04-26 06:53:57,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.85 | bwd_microstep: 5756.84 | bwd_inner_microstep: 5677.47 | bwd_allreduce_microstep: 79.31 | step_microstep: 19.72
[2025-04-26 06:53:57,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.85 | bwd: 5756.85 | bwd_inner: 5677.47 | bwd_allreduce: 79.34 | step: 19.72

  3%|▎         | 1140/41250 [2:45:28<96:52:51,  8.70s/it]
                                                         
{'loss': 0.1011, 'grad_norm': 1.2154431343078613, 'learning_rate': 3.683360258481422e-05, 'epoch': 0.28}

  3%|▎         | 1140/41250 [2:45:28<96:52:51,  8.70s/it][2025-04-26 06:54:06,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.26 | optimizer_step: 0.89
[2025-04-26 06:54:06,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.90 | bwd_microstep: 5862.00 | bwd_inner_microstep: 5655.04 | bwd_allreduce_microstep: 206.91 | step_microstep: 19.34
[2025-04-26 06:54:06,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.90 | bwd: 5862.02 | bwd_inner: 5655.03 | bwd_allreduce: 206.93 | step: 19.34

  3%|▎         | 1141/41250 [2:45:37<97:08:13,  8.72s/it]
                                                         
{'loss': 0.2153, 'grad_norm': 2.7035155296325684, 'learning_rate': 3.6865912762520196e-05, 'epoch': 0.28}

  3%|▎         | 1141/41250 [2:45:37<97:08:13,  8.72s/it][2025-04-26 06:54:14,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:54:14,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.34 | bwd_microstep: 5698.52 | bwd_inner_microstep: 5645.75 | bwd_allreduce_microstep: 52.73 | step_microstep: 18.58
[2025-04-26 06:54:14,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.34 | bwd: 5698.53 | bwd_inner: 5645.75 | bwd_allreduce: 52.74 | step: 18.58

  3%|▎         | 1142/41250 [2:45:45<96:45:39,  8.69s/it]
                                                         
{'loss': 0.2223, 'grad_norm': 2.4131646156311035, 'learning_rate': 3.6898222940226174e-05, 'epoch': 0.28}

  3%|▎         | 1142/41250 [2:45:45<96:45:39,  8.69s/it][2025-04-26 06:54:23,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:54:23,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.46 | bwd_microstep: 5759.92 | bwd_inner_microstep: 5655.55 | bwd_allreduce_microstep: 104.32 | step_microstep: 18.52
[2025-04-26 06:54:23,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.46 | bwd: 5759.93 | bwd_inner: 5655.55 | bwd_allreduce: 104.34 | step: 18.52

  3%|▎         | 1143/41250 [2:45:54<96:42:26,  8.68s/it]
                                                         
{'loss': 0.1838, 'grad_norm': 3.052046298980713, 'learning_rate': 3.693053311793215e-05, 'epoch': 0.28}

  3%|▎         | 1143/41250 [2:45:54<96:42:26,  8.68s/it][2025-04-26 06:54:32,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:54:32,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.08 | bwd_microstep: 5773.90 | bwd_inner_microstep: 5643.21 | bwd_allreduce_microstep: 130.64 | step_microstep: 18.58
[2025-04-26 06:54:32,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.08 | bwd: 5773.92 | bwd_inner: 5643.21 | bwd_allreduce: 130.66 | step: 18.59

  3%|▎         | 1144/41250 [2:46:03<96:41:58,  8.68s/it]
                                                         
{'loss': 0.1309, 'grad_norm': 1.8452552556991577, 'learning_rate': 3.696284329563813e-05, 'epoch': 0.28}

  3%|▎         | 1144/41250 [2:46:03<96:41:58,  8.68s/it][2025-04-26 06:54:40,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.10 | optimizer_step: 1.15
[2025-04-26 06:54:40,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.30 | bwd_microstep: 5707.81 | bwd_inner_microstep: 5694.40 | bwd_allreduce_microstep: 13.36 | step_microstep: 19.61
[2025-04-26 06:54:40,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.30 | bwd: 5707.82 | bwd_inner: 5694.40 | bwd_allreduce: 13.38 | step: 19.61

  3%|▎         | 1145/41250 [2:46:11<96:32:04,  8.67s/it]
                                                         
{'loss': 0.3522, 'grad_norm': 2.613128423690796, 'learning_rate': 3.699515347334411e-05, 'epoch': 0.28}

  3%|▎         | 1145/41250 [2:46:11<96:32:04,  8.67s/it][2025-04-26 06:54:49,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:54:49,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.79 | bwd_microstep: 5699.33 | bwd_inner_microstep: 5680.02 | bwd_allreduce_microstep: 19.27 | step_microstep: 18.82
[2025-04-26 06:54:49,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.79 | bwd: 5699.35 | bwd_inner: 5680.02 | bwd_allreduce: 19.29 | step: 18.82

  3%|▎         | 1146/41250 [2:46:20<96:23:59,  8.65s/it]
                                                         
{'loss': 0.2592, 'grad_norm': 1.841620922088623, 'learning_rate': 3.702746365105008e-05, 'epoch': 0.28}

  3%|▎         | 1146/41250 [2:46:20<96:23:59,  8.65s/it][2025-04-26 06:54:57,998] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:54:57,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.25 | bwd_microstep: 5689.37 | bwd_inner_microstep: 5676.49 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.89
[2025-04-26 06:54:57,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.25 | bwd: 5689.38 | bwd_inner: 5676.49 | bwd_allreduce: 12.86 | step: 18.89

  3%|▎         | 1147/41250 [2:46:29<96:16:05,  8.64s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.7610291242599487, 'learning_rate': 3.705977382875606e-05, 'epoch': 0.28}

  3%|▎         | 1147/41250 [2:46:29<96:16:05,  8.64s/it][2025-04-26 06:55:06,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:55:06,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.88 | bwd_microstep: 5697.44 | bwd_inner_microstep: 5639.70 | bwd_allreduce_microstep: 57.69 | step_microstep: 18.65
[2025-04-26 06:55:06,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.88 | bwd: 5697.45 | bwd_inner: 5639.70 | bwd_allreduce: 57.71 | step: 18.66

  3%|▎         | 1148/41250 [2:46:37<96:09:36,  8.63s/it]
                                                         
{'loss': 0.1736, 'grad_norm': 1.5855542421340942, 'learning_rate': 3.709208400646204e-05, 'epoch': 0.28}

  3%|▎         | 1148/41250 [2:46:37<96:09:36,  8.63s/it][2025-04-26 06:55:15,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 06:55:15,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.31 | bwd_microstep: 5774.42 | bwd_inner_microstep: 5660.33 | bwd_allreduce_microstep: 114.04 | step_microstep: 18.13
[2025-04-26 06:55:15,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.31 | bwd: 5774.43 | bwd_inner: 5660.33 | bwd_allreduce: 114.06 | step: 18.14

  3%|▎         | 1149/41250 [2:46:46<96:21:40,  8.65s/it]
                                                         
{'loss': 0.1927, 'grad_norm': 3.0045087337493896, 'learning_rate': 3.712439418416802e-05, 'epoch': 0.28}

  3%|▎         | 1149/41250 [2:46:46<96:21:40,  8.65s/it][2025-04-26 06:55:23,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:55:23,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.89 | bwd_microstep: 5750.54 | bwd_inner_microstep: 5706.92 | bwd_allreduce_microstep: 43.57 | step_microstep: 18.53
[2025-04-26 06:55:23,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.89 | bwd: 5750.55 | bwd_inner: 5706.92 | bwd_allreduce: 43.59 | step: 18.53

  3%|▎         | 1150/41250 [2:46:55<96:28:36,  8.66s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 1.330962896347046, 'learning_rate': 3.715670436187399e-05, 'epoch': 0.28}

  3%|▎         | 1150/41250 [2:46:55<96:28:36,  8.66s/it][2025-04-26 06:55:32,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:55:32,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.89 | bwd_microstep: 5696.14 | bwd_inner_microstep: 5659.02 | bwd_allreduce_microstep: 37.08 | step_microstep: 18.35
[2025-04-26 06:55:32,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.89 | bwd: 5696.16 | bwd_inner: 5659.02 | bwd_allreduce: 37.09 | step: 18.36

  3%|▎         | 1151/41250 [2:47:03<96:22:16,  8.65s/it]
                                                         
{'loss': 0.0391, 'grad_norm': 0.46388688683509827, 'learning_rate': 3.718901453957997e-05, 'epoch': 0.28}

  3%|▎         | 1151/41250 [2:47:03<96:22:16,  8.65s/it][2025-04-26 06:55:41,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-26 06:55:41,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.77 | bwd_microstep: 6013.65 | bwd_inner_microstep: 5713.76 | bwd_allreduce_microstep: 299.85 | step_microstep: 18.18
[2025-04-26 06:55:41,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.77 | bwd: 6013.66 | bwd_inner: 5713.76 | bwd_allreduce: 299.86 | step: 18.18

  3%|▎         | 1152/41250 [2:47:12<97:22:33,  8.74s/it]
                                                         
{'loss': 0.3847, 'grad_norm': 3.6838979721069336, 'learning_rate': 3.7221324717285946e-05, 'epoch': 0.28}

  3%|▎         | 1152/41250 [2:47:12<97:22:33,  8.74s/it][2025-04-26 06:55:50,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:55:50,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.02 | bwd_microstep: 5695.11 | bwd_inner_microstep: 5682.19 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.25
[2025-04-26 06:55:50,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.02 | bwd: 5695.13 | bwd_inner: 5682.19 | bwd_allreduce: 12.90 | step: 18.25

  3%|▎         | 1153/41250 [2:47:21<96:59:36,  8.71s/it]
                                                         
{'loss': 0.4043, 'grad_norm': 2.914564371109009, 'learning_rate': 3.7253634894991924e-05, 'epoch': 0.28}

  3%|▎         | 1153/41250 [2:47:21<96:59:36,  8.71s/it][2025-04-26 06:55:58,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 06:55:58,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.45 | bwd_microstep: 5737.14 | bwd_inner_microstep: 5714.88 | bwd_allreduce_microstep: 22.22 | step_microstep: 17.76
[2025-04-26 06:55:58,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.45 | bwd: 5737.16 | bwd_inner: 5714.88 | bwd_allreduce: 22.24 | step: 17.77

  3%|▎         | 1154/41250 [2:47:30<96:51:10,  8.70s/it]
                                                         
{'loss': 0.1248, 'grad_norm': 2.5382609367370605, 'learning_rate': 3.72859450726979e-05, 'epoch': 0.28}

  3%|▎         | 1154/41250 [2:47:30<96:51:10,  8.70s/it][2025-04-26 06:56:07,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:56:07,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5724.74 | bwd_inner_microstep: 5712.02 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.51
[2025-04-26 06:56:07,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5724.75 | bwd_inner: 5712.02 | bwd_allreduce: 12.69 | step: 18.51

  3%|▎         | 1155/41250 [2:47:38<96:45:03,  8.69s/it]
                                                         
{'loss': 0.2111, 'grad_norm': 2.2310218811035156, 'learning_rate': 3.731825525040388e-05, 'epoch': 0.28}

  3%|▎         | 1155/41250 [2:47:38<96:45:03,  8.69s/it][2025-04-26 06:56:16,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 06:56:16,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.05 | bwd_microstep: 5743.02 | bwd_inner_microstep: 5675.92 | bwd_allreduce_microstep: 67.06 | step_microstep: 17.80
[2025-04-26 06:56:16,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.05 | bwd: 5743.03 | bwd_inner: 5675.92 | bwd_allreduce: 67.08 | step: 17.81

  3%|▎         | 1156/41250 [2:47:47<96:44:40,  8.69s/it]
                                                         
{'loss': 0.4437, 'grad_norm': 3.252730369567871, 'learning_rate': 3.735056542810986e-05, 'epoch': 0.28}

  3%|▎         | 1156/41250 [2:47:47<96:44:40,  8.69s/it][2025-04-26 06:56:25,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 06:56:25,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5905.27 | bwd_inner_microstep: 5690.04 | bwd_allreduce_microstep: 215.19 | step_microstep: 18.54
[2025-04-26 06:56:25,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5905.28 | bwd_inner: 5690.04 | bwd_allreduce: 215.21 | step: 18.54

  3%|▎         | 1157/41250 [2:47:56<97:15:14,  8.73s/it]
                                                         
{'loss': 0.219, 'grad_norm': 1.7250345945358276, 'learning_rate': 3.7382875605815835e-05, 'epoch': 0.28}

  3%|▎         | 1157/41250 [2:47:56<97:15:14,  8.73s/it][2025-04-26 06:56:33,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:56:33,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.30 | bwd_microstep: 5725.83 | bwd_inner_microstep: 5713.11 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.42
[2025-04-26 06:56:33,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.30 | bwd: 5725.84 | bwd_inner: 5713.11 | bwd_allreduce: 12.69 | step: 18.42

  3%|▎         | 1158/41250 [2:48:04<96:59:52,  8.71s/it]
                                                         
{'loss': 0.0537, 'grad_norm': 0.6322845220565796, 'learning_rate': 3.741518578352181e-05, 'epoch': 0.28}

  3%|▎         | 1158/41250 [2:48:04<96:59:52,  8.71s/it][2025-04-26 06:56:42,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:56:42,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.14 | bwd_microstep: 5737.40 | bwd_inner_microstep: 5724.60 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.77
[2025-04-26 06:56:42,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.14 | bwd: 5737.41 | bwd_inner: 5724.60 | bwd_allreduce: 12.78 | step: 17.77

  3%|▎         | 1159/41250 [2:48:13<96:52:13,  8.70s/it]
                                                         
{'loss': 0.2028, 'grad_norm': 1.7958142757415771, 'learning_rate': 3.744749596122779e-05, 'epoch': 0.28}

  3%|▎         | 1159/41250 [2:48:13<96:52:13,  8.70s/it][2025-04-26 06:56:51,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:56:51,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.28 | bwd_microstep: 5782.34 | bwd_inner_microstep: 5663.13 | bwd_allreduce_microstep: 119.16 | step_microstep: 17.75
[2025-04-26 06:56:51,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.28 | bwd: 5782.35 | bwd_inner: 5663.13 | bwd_allreduce: 119.18 | step: 17.75

  3%|▎         | 1160/41250 [2:48:22<96:51:49,  8.70s/it]
                                                         
{'loss': 0.1737, 'grad_norm': 1.394987940788269, 'learning_rate': 3.747980613893376e-05, 'epoch': 0.28}

  3%|▎         | 1160/41250 [2:48:22<96:51:49,  8.70s/it][2025-04-26 06:56:59,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:56:59,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.96 | bwd_microstep: 5723.35 | bwd_inner_microstep: 5710.53 | bwd_allreduce_microstep: 12.78 | step_microstep: 17.70
[2025-04-26 06:56:59,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.96 | bwd: 5723.36 | bwd_inner: 5710.53 | bwd_allreduce: 12.79 | step: 17.70

  3%|▎         | 1161/41250 [2:48:30<96:44:20,  8.69s/it]
                                                         
{'loss': 0.2286, 'grad_norm': 2.116246223449707, 'learning_rate': 3.751211631663975e-05, 'epoch': 0.28}

  3%|▎         | 1161/41250 [2:48:30<96:44:20,  8.69s/it][2025-04-26 06:57:08,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 06:57:08,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.55 | bwd_microstep: 5780.81 | bwd_inner_microstep: 5644.13 | bwd_allreduce_microstep: 136.64 | step_microstep: 18.59
[2025-04-26 06:57:08,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.55 | bwd: 5780.83 | bwd_inner: 5644.13 | bwd_allreduce: 136.65 | step: 18.59

  3%|▎         | 1162/41250 [2:48:39<96:44:00,  8.69s/it]
                                                         
{'loss': 0.2206, 'grad_norm': 2.906824827194214, 'learning_rate': 3.7544426494345725e-05, 'epoch': 0.28}

  3%|▎         | 1162/41250 [2:48:39<96:44:00,  8.69s/it][2025-04-26 06:57:17,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 06:57:17,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.70 | bwd_microstep: 5717.49 | bwd_inner_microstep: 5704.92 | bwd_allreduce_microstep: 12.53 | step_microstep: 17.72
[2025-04-26 06:57:17,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.70 | bwd: 5717.51 | bwd_inner: 5704.92 | bwd_allreduce: 12.55 | step: 17.72

  3%|▎         | 1163/41250 [2:48:48<96:34:22,  8.67s/it]
                                                         
{'loss': 0.0814, 'grad_norm': 0.8086070418357849, 'learning_rate': 3.75767366720517e-05, 'epoch': 0.28}

  3%|▎         | 1163/41250 [2:48:48<96:34:22,  8.67s/it][2025-04-26 06:57:25,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 06:57:25,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.78 | bwd_microstep: 5714.02 | bwd_inner_microstep: 5701.12 | bwd_allreduce_microstep: 12.86 | step_microstep: 17.74
[2025-04-26 06:57:25,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.78 | bwd: 5714.03 | bwd_inner: 5701.12 | bwd_allreduce: 12.87 | step: 17.75

  3%|▎         | 1164/41250 [2:48:56<96:28:32,  8.66s/it]
                                                         
{'loss': 0.2894, 'grad_norm': 2.47501802444458, 'learning_rate': 3.7609046849757674e-05, 'epoch': 0.28}

  3%|▎         | 1164/41250 [2:48:56<96:28:32,  8.66s/it][2025-04-26 06:57:34,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-26 06:57:34,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.21 | bwd_microstep: 5748.03 | bwd_inner_microstep: 5704.68 | bwd_allreduce_microstep: 43.30 | step_microstep: 18.46
[2025-04-26 06:57:34,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.21 | bwd: 5748.05 | bwd_inner: 5704.68 | bwd_allreduce: 43.32 | step: 18.46

  3%|▎         | 1165/41250 [2:49:05<96:32:56,  8.67s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.9923911094665527, 'learning_rate': 3.764135702746365e-05, 'epoch': 0.28}

  3%|▎         | 1165/41250 [2:49:05<96:32:56,  8.67s/it][2025-04-26 06:57:43,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:57:43,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.35 | bwd_microstep: 5711.02 | bwd_inner_microstep: 5698.08 | bwd_allreduce_microstep: 12.89 | step_microstep: 17.75
[2025-04-26 06:57:43,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.35 | bwd: 5711.03 | bwd_inner: 5698.08 | bwd_allreduce: 12.91 | step: 17.76

  3%|▎         | 1166/41250 [2:49:14<96:28:29,  8.66s/it]
                                                         
{'loss': 0.2045, 'grad_norm': 1.60890793800354, 'learning_rate': 3.767366720516963e-05, 'epoch': 0.28}

  3%|▎         | 1166/41250 [2:49:14<96:28:29,  8.66s/it][2025-04-26 06:57:51,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:57:51,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.64 | bwd_microstep: 5780.24 | bwd_inner_microstep: 5646.94 | bwd_allreduce_microstep: 133.26 | step_microstep: 18.28
[2025-04-26 06:57:51,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.64 | bwd: 5780.25 | bwd_inner: 5646.94 | bwd_allreduce: 133.27 | step: 18.29

  3%|▎         | 1167/41250 [2:49:22<96:33:25,  8.67s/it]
                                                         
{'loss': 0.2536, 'grad_norm': 1.5334917306900024, 'learning_rate': 3.770597738287561e-05, 'epoch': 0.28}

  3%|▎         | 1167/41250 [2:49:22<96:33:25,  8.67s/it][2025-04-26 06:58:00,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 06:58:00,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.02 | bwd_microstep: 5754.31 | bwd_inner_microstep: 5685.35 | bwd_allreduce_microstep: 68.92 | step_microstep: 18.37
[2025-04-26 06:58:00,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.02 | bwd: 5754.32 | bwd_inner: 5685.35 | bwd_allreduce: 68.93 | step: 18.38

  3%|▎         | 1168/41250 [2:49:31<96:34:40,  8.67s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 2.067533493041992, 'learning_rate': 3.7738287560581585e-05, 'epoch': 0.28}

  3%|▎         | 1168/41250 [2:49:31<96:34:40,  8.67s/it][2025-04-26 06:58:09,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-26 06:58:09,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.01 | bwd_microstep: 5779.79 | bwd_inner_microstep: 5642.77 | bwd_allreduce_microstep: 136.97 | step_microstep: 18.97
[2025-04-26 06:58:09,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.01 | bwd: 5779.80 | bwd_inner: 5642.77 | bwd_allreduce: 136.99 | step: 18.97

  3%|▎         | 1169/41250 [2:49:40<96:38:07,  8.68s/it]
                                                         
{'loss': 0.3623, 'grad_norm': 3.2679686546325684, 'learning_rate': 3.777059773828756e-05, 'epoch': 0.28}

  3%|▎         | 1169/41250 [2:49:40<96:38:07,  8.68s/it][2025-04-26 06:58:17,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 06:58:17,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.50 | bwd_microstep: 5693.63 | bwd_inner_microstep: 5680.87 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.19
[2025-04-26 06:58:17,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.50 | bwd: 5693.64 | bwd_inner: 5680.87 | bwd_allreduce: 12.74 | step: 18.20

  3%|▎         | 1170/41250 [2:49:48<96:27:08,  8.66s/it]
                                                         
{'loss': 0.2686, 'grad_norm': 2.5821218490600586, 'learning_rate': 3.780290791599354e-05, 'epoch': 0.28}

  3%|▎         | 1170/41250 [2:49:48<96:27:08,  8.66s/it][2025-04-26 06:58:26,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 06:58:26,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.22 | bwd_microstep: 5882.74 | bwd_inner_microstep: 5638.29 | bwd_allreduce_microstep: 244.40 | step_microstep: 18.59
[2025-04-26 06:58:26,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.22 | bwd: 5882.75 | bwd_inner: 5638.29 | bwd_allreduce: 244.42 | step: 18.59

  3%|▎         | 1171/41250 [2:49:57<96:52:38,  8.70s/it]
                                                         
{'loss': 0.2141, 'grad_norm': 2.7215609550476074, 'learning_rate': 3.783521809369952e-05, 'epoch': 0.28}

  3%|▎         | 1171/41250 [2:49:57<96:52:38,  8.70s/it][2025-04-26 06:58:35,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 06:58:35,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.85 | bwd_microstep: 5745.58 | bwd_inner_microstep: 5678.64 | bwd_allreduce_microstep: 66.91 | step_microstep: 17.78
[2025-04-26 06:58:35,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.85 | bwd: 5745.60 | bwd_inner: 5678.64 | bwd_allreduce: 66.92 | step: 17.79

  3%|▎         | 1172/41250 [2:50:06<96:44:35,  8.69s/it]
                                                         
{'loss': 0.2929, 'grad_norm': 2.0623233318328857, 'learning_rate': 3.78675282714055e-05, 'epoch': 0.28}

  3%|▎         | 1172/41250 [2:50:06<96:44:35,  8.69s/it][2025-04-26 06:58:43,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:58:43,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5770.54 | bwd_inner_microstep: 5651.61 | bwd_allreduce_microstep: 118.89 | step_microstep: 17.65
[2025-04-26 06:58:43,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5770.55 | bwd_inner: 5651.61 | bwd_allreduce: 118.91 | step: 17.65

  3%|▎         | 1173/41250 [2:50:15<96:42:52,  8.69s/it]
                                                         
{'loss': 0.31, 'grad_norm': 2.4705212116241455, 'learning_rate': 3.7899838449111475e-05, 'epoch': 0.28}

  3%|▎         | 1173/41250 [2:50:15<96:42:52,  8.69s/it][2025-04-26 06:58:52,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 06:58:52,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5755.02 | bwd_inner_microstep: 5681.34 | bwd_allreduce_microstep: 73.64 | step_microstep: 18.06
[2025-04-26 06:58:52,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5755.03 | bwd_inner: 5681.34 | bwd_allreduce: 73.65 | step: 18.06

  3%|▎         | 1174/41250 [2:50:23<96:42:11,  8.69s/it]
                                                         
{'loss': 0.2126, 'grad_norm': 1.322452187538147, 'learning_rate': 3.7932148626817446e-05, 'epoch': 0.28}

  3%|▎         | 1174/41250 [2:50:23<96:42:11,  8.69s/it][2025-04-26 06:59:01,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 06:59:01,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.97 | bwd_microstep: 5704.34 | bwd_inner_microstep: 5631.57 | bwd_allreduce_microstep: 72.71 | step_microstep: 18.23
[2025-04-26 06:59:01,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.97 | bwd: 5704.35 | bwd_inner: 5631.57 | bwd_allreduce: 72.73 | step: 18.23

  3%|▎         | 1175/41250 [2:50:32<96:25:51,  8.66s/it]
                                                         
{'loss': 0.1756, 'grad_norm': 1.2986419200897217, 'learning_rate': 3.796445880452343e-05, 'epoch': 0.28}

  3%|▎         | 1175/41250 [2:50:32<96:25:51,  8.66s/it][2025-04-26 06:59:09,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:59:09,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.61 | bwd_microstep: 5716.12 | bwd_inner_microstep: 5701.04 | bwd_allreduce_microstep: 15.04 | step_microstep: 18.36
[2025-04-26 06:59:09,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.61 | bwd: 5716.13 | bwd_inner: 5701.04 | bwd_allreduce: 15.06 | step: 18.36

  3%|▎         | 1176/41250 [2:50:41<96:24:31,  8.66s/it]
                                                         
{'loss': 0.3723, 'grad_norm': 3.1372439861297607, 'learning_rate': 3.799676898222941e-05, 'epoch': 0.29}

  3%|▎         | 1176/41250 [2:50:41<96:24:31,  8.66s/it][2025-04-26 06:59:18,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 06:59:18,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.42 | bwd_microstep: 5694.53 | bwd_inner_microstep: 5681.81 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.32
[2025-04-26 06:59:18,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.42 | bwd: 5694.55 | bwd_inner: 5681.81 | bwd_allreduce: 12.70 | step: 18.32

  3%|▎         | 1177/41250 [2:50:49<96:18:02,  8.65s/it]
                                                         
{'loss': 0.2575, 'grad_norm': 1.3378243446350098, 'learning_rate': 3.8029079159935386e-05, 'epoch': 0.29}

  3%|▎         | 1177/41250 [2:50:49<96:18:02,  8.65s/it][2025-04-26 06:59:27,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 06:59:27,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.30 | bwd_microstep: 5732.47 | bwd_inner_microstep: 5681.79 | bwd_allreduce_microstep: 50.63 | step_microstep: 19.23
[2025-04-26 06:59:27,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.30 | bwd: 5732.49 | bwd_inner: 5681.79 | bwd_allreduce: 50.65 | step: 19.24

  3%|▎         | 1178/41250 [2:50:58<96:20:40,  8.66s/it]
                                                         
{'loss': 0.0719, 'grad_norm': 1.1399931907653809, 'learning_rate': 3.806138933764136e-05, 'epoch': 0.29}

  3%|▎         | 1178/41250 [2:50:58<96:20:40,  8.66s/it][2025-04-26 06:59:35,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:59:35,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.48 | bwd_microstep: 5756.78 | bwd_inner_microstep: 5646.44 | bwd_allreduce_microstep: 110.30 | step_microstep: 18.58
[2025-04-26 06:59:35,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.47 | bwd: 5756.80 | bwd_inner: 5646.44 | bwd_allreduce: 110.32 | step: 18.58

  3%|▎         | 1179/41250 [2:51:06<96:22:14,  8.66s/it]
                                                         
{'loss': 0.1245, 'grad_norm': 0.6017708778381348, 'learning_rate': 3.8093699515347335e-05, 'epoch': 0.29}

  3%|▎         | 1179/41250 [2:51:06<96:22:14,  8.66s/it][2025-04-26 06:59:44,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 06:59:44,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.14 | bwd_microstep: 5758.74 | bwd_inner_microstep: 5630.31 | bwd_allreduce_microstep: 128.39 | step_microstep: 18.45
[2025-04-26 06:59:44,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.14 | bwd: 5758.75 | bwd_inner: 5630.31 | bwd_allreduce: 128.41 | step: 18.46

  3%|▎         | 1180/41250 [2:51:15<96:22:49,  8.66s/it]
                                                         
{'loss': 0.2422, 'grad_norm': 1.9036628007888794, 'learning_rate': 3.812600969305331e-05, 'epoch': 0.29}

  3%|▎         | 1180/41250 [2:51:15<96:22:49,  8.66s/it][2025-04-26 06:59:53,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 06:59:53,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.25 | bwd_microstep: 5931.07 | bwd_inner_microstep: 5658.96 | bwd_allreduce_microstep: 272.06 | step_microstep: 18.44
[2025-04-26 06:59:53,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.25 | bwd: 5931.08 | bwd_inner: 5658.96 | bwd_allreduce: 272.08 | step: 18.44

  3%|▎         | 1181/41250 [2:51:24<96:56:22,  8.71s/it]
                                                         
{'loss': 0.0933, 'grad_norm': 0.7739103436470032, 'learning_rate': 3.815831987075929e-05, 'epoch': 0.29}

  3%|▎         | 1181/41250 [2:51:24<96:56:22,  8.71s/it][2025-04-26 07:00:01,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:00:01,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.40 | bwd_microstep: 5725.38 | bwd_inner_microstep: 5638.61 | bwd_allreduce_microstep: 86.72 | step_microstep: 18.07
[2025-04-26 07:00:01,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.40 | bwd: 5725.39 | bwd_inner: 5638.61 | bwd_allreduce: 86.74 | step: 18.08

  3%|▎         | 1182/41250 [2:51:33<96:43:04,  8.69s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.5262091159820557, 'learning_rate': 3.819063004846527e-05, 'epoch': 0.29}

  3%|▎         | 1182/41250 [2:51:33<96:43:04,  8.69s/it][2025-04-26 07:00:10,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:00:10,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5986.86 | bwd_inner_microstep: 5683.71 | bwd_allreduce_microstep: 303.10 | step_microstep: 18.52
[2025-04-26 07:00:10,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5986.87 | bwd_inner: 5683.71 | bwd_allreduce: 303.11 | step: 18.52

  3%|▎         | 1183/41250 [2:51:42<97:28:26,  8.76s/it]
                                                         
{'loss': 0.2112, 'grad_norm': 1.79838228225708, 'learning_rate': 3.822294022617125e-05, 'epoch': 0.29}

  3%|▎         | 1183/41250 [2:51:42<97:28:26,  8.76s/it][2025-04-26 07:00:19,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:00:19,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.81 | bwd_microstep: 5862.85 | bwd_inner_microstep: 5849.90 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.45
[2025-04-26 07:00:19,714] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.81 | bwd: 5862.87 | bwd_inner: 5849.90 | bwd_allreduce: 12.93 | step: 18.45

  3%|▎         | 1184/41250 [2:51:50<97:53:05,  8.80s/it]
                                                         
{'loss': 0.3396, 'grad_norm': 2.1247429847717285, 'learning_rate': 3.8255250403877225e-05, 'epoch': 0.29}

  3%|▎         | 1184/41250 [2:51:50<97:53:05,  8.80s/it][2025-04-26 07:00:28,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 07:00:28,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.53 | bwd_microstep: 5757.44 | bwd_inner_microstep: 5653.56 | bwd_allreduce_microstep: 103.83 | step_microstep: 17.63
[2025-04-26 07:00:28,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.53 | bwd: 5757.45 | bwd_inner: 5653.56 | bwd_allreduce: 103.85 | step: 17.63

  3%|▎         | 1185/41250 [2:51:59<97:26:55,  8.76s/it]
                                                         
{'loss': 0.0569, 'grad_norm': 0.6106215119361877, 'learning_rate': 3.82875605815832e-05, 'epoch': 0.29}

  3%|▎         | 1185/41250 [2:51:59<97:26:55,  8.76s/it][2025-04-26 07:00:37,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 07:00:37,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.21 | bwd_microstep: 5750.01 | bwd_inner_microstep: 5638.88 | bwd_allreduce_microstep: 111.09 | step_microstep: 18.20
[2025-04-26 07:00:37,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.21 | bwd: 5750.03 | bwd_inner: 5638.88 | bwd_allreduce: 111.11 | step: 18.20

  3%|▎         | 1186/41250 [2:52:08<97:04:47,  8.72s/it]
                                                         
{'loss': 0.3851, 'grad_norm': 3.750494956970215, 'learning_rate': 3.831987075928918e-05, 'epoch': 0.29}

  3%|▎         | 1186/41250 [2:52:08<97:04:47,  8.72s/it][2025-04-26 07:00:45,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:00:45,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.09 | bwd_microstep: 5763.82 | bwd_inner_microstep: 5640.37 | bwd_allreduce_microstep: 123.39 | step_microstep: 18.81
[2025-04-26 07:00:45,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.09 | bwd: 5763.84 | bwd_inner: 5640.37 | bwd_allreduce: 123.42 | step: 18.81

  3%|▎         | 1187/41250 [2:52:16<96:55:47,  8.71s/it]
                                                         
{'loss': 0.0941, 'grad_norm': 1.191663146018982, 'learning_rate': 3.835218093699516e-05, 'epoch': 0.29}

  3%|▎         | 1187/41250 [2:52:16<96:55:47,  8.71s/it][2025-04-26 07:00:54,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.97
[2025-04-26 07:00:54,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.09 | bwd_microstep: 5692.53 | bwd_inner_microstep: 5679.42 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.02
[2025-04-26 07:00:54,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.09 | bwd: 5692.54 | bwd_inner: 5679.42 | bwd_allreduce: 13.07 | step: 19.02

  3%|▎         | 1188/41250 [2:52:25<96:37:43,  8.68s/it]
                                                         
{'loss': 0.1508, 'grad_norm': 1.997146487236023, 'learning_rate': 3.838449111470113e-05, 'epoch': 0.29}

  3%|▎         | 1188/41250 [2:52:25<96:37:43,  8.68s/it][2025-04-26 07:01:02,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.40 | optimizer_step: 0.95
[2025-04-26 07:01:02,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5679.86 | bwd_inner_microstep: 5651.60 | bwd_allreduce_microstep: 28.21 | step_microstep: 19.82
[2025-04-26 07:01:02,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5679.88 | bwd_inner: 5651.60 | bwd_allreduce: 28.23 | step: 19.82

  3%|▎         | 1189/41250 [2:52:34<96:20:17,  8.66s/it]
                                                         
{'loss': 0.192, 'grad_norm': 2.4016425609588623, 'learning_rate': 3.8416801292407114e-05, 'epoch': 0.29}

  3%|▎         | 1189/41250 [2:52:34<96:20:17,  8.66s/it][2025-04-26 07:01:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:01:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.91 | bwd_microstep: 5689.12 | bwd_inner_microstep: 5676.27 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.80
[2025-04-26 07:01:11,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.91 | bwd: 5689.14 | bwd_inner: 5676.27 | bwd_allreduce: 12.83 | step: 18.80

  3%|▎         | 1190/41250 [2:52:42<96:11:54,  8.64s/it]
                                                         
{'loss': 0.1718, 'grad_norm': 2.100426435470581, 'learning_rate': 3.844911147011309e-05, 'epoch': 0.29}

  3%|▎         | 1190/41250 [2:52:42<96:11:54,  8.64s/it][2025-04-26 07:01:20,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 07:01:20,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.85 | bwd_microstep: 5703.22 | bwd_inner_microstep: 5664.31 | bwd_allreduce_microstep: 38.87 | step_microstep: 18.90
[2025-04-26 07:01:20,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.85 | bwd: 5703.24 | bwd_inner: 5664.31 | bwd_allreduce: 38.89 | step: 18.91

  3%|▎         | 1191/41250 [2:52:51<96:05:55,  8.64s/it]
                                                         
{'loss': 0.2938, 'grad_norm': 2.5090575218200684, 'learning_rate': 3.848142164781907e-05, 'epoch': 0.29}

  3%|▎         | 1191/41250 [2:52:51<96:05:55,  8.64s/it][2025-04-26 07:01:28,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 0.95
[2025-04-26 07:01:28,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.54 | bwd_microstep: 5762.91 | bwd_inner_microstep: 5647.05 | bwd_allreduce_microstep: 115.81 | step_microstep: 18.87
[2025-04-26 07:01:28,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.54 | bwd: 5762.92 | bwd_inner: 5647.05 | bwd_allreduce: 115.83 | step: 18.87

  3%|▎         | 1192/41250 [2:53:00<96:13:12,  8.65s/it]
                                                         
{'loss': 0.084, 'grad_norm': 1.211259126663208, 'learning_rate': 3.851373182552504e-05, 'epoch': 0.29}

  3%|▎         | 1192/41250 [2:53:00<96:13:12,  8.65s/it][2025-04-26 07:01:37,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 07:01:37,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2940.95 | bwd_microstep: 5880.41 | bwd_inner_microstep: 5867.75 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.49
[2025-04-26 07:01:37,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2940.95 | bwd: 5880.43 | bwd_inner: 5867.75 | bwd_allreduce: 12.64 | step: 18.50

  3%|▎         | 1193/41250 [2:53:08<97:04:34,  8.72s/it]
                                                         
{'loss': 0.2852, 'grad_norm': 1.9591069221496582, 'learning_rate': 3.854604200323102e-05, 'epoch': 0.29}

  3%|▎         | 1193/41250 [2:53:08<97:04:34,  8.72s/it][2025-04-26 07:01:46,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:01:46,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.94 | bwd_microstep: 5744.04 | bwd_inner_microstep: 5650.51 | bwd_allreduce_microstep: 93.49 | step_microstep: 18.62
[2025-04-26 07:01:46,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.94 | bwd: 5744.05 | bwd_inner: 5650.51 | bwd_allreduce: 93.51 | step: 18.62

  3%|▎         | 1194/41250 [2:53:17<96:53:26,  8.71s/it]
                                                         
{'loss': 0.1826, 'grad_norm': 1.8855962753295898, 'learning_rate': 3.8578352180937e-05, 'epoch': 0.29}

  3%|▎         | 1194/41250 [2:53:17<96:53:26,  8.71s/it][2025-04-26 07:01:55,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:01:55,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.26 | bwd_microstep: 5757.15 | bwd_inner_microstep: 5709.93 | bwd_allreduce_microstep: 47.17 | step_microstep: 17.82
[2025-04-26 07:01:55,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.26 | bwd: 5757.16 | bwd_inner: 5709.93 | bwd_allreduce: 47.19 | step: 17.83

  3%|▎         | 1195/41250 [2:53:26<96:48:48,  8.70s/it]
                                                         
{'loss': 0.2649, 'grad_norm': 3.027082681655884, 'learning_rate': 3.8610662358642975e-05, 'epoch': 0.29}

  3%|▎         | 1195/41250 [2:53:26<96:48:48,  8.70s/it][2025-04-26 07:02:03,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:02:03,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.04 | bwd_microstep: 5757.04 | bwd_inner_microstep: 5692.57 | bwd_allreduce_microstep: 64.43 | step_microstep: 17.81
[2025-04-26 07:02:03,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.05 | bwd: 5757.06 | bwd_inner: 5692.57 | bwd_allreduce: 64.45 | step: 17.81

  3%|▎         | 1196/41250 [2:53:34<96:45:16,  8.70s/it]
                                                         
{'loss': 0.1781, 'grad_norm': 1.4481476545333862, 'learning_rate': 3.864297253634895e-05, 'epoch': 0.29}

  3%|▎         | 1196/41250 [2:53:34<96:45:16,  8.70s/it][2025-04-26 07:02:12,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 07:02:12,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5724.41 | bwd_inner_microstep: 5648.05 | bwd_allreduce_microstep: 76.32 | step_microstep: 19.06
[2025-04-26 07:02:12,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5724.43 | bwd_inner: 5648.05 | bwd_allreduce: 76.34 | step: 19.05

  3%|▎         | 1197/41250 [2:53:43<96:31:45,  8.68s/it]
                                                         
{'loss': 0.1082, 'grad_norm': 1.6056267023086548, 'learning_rate': 3.867528271405493e-05, 'epoch': 0.29}

  3%|▎         | 1197/41250 [2:53:43<96:31:45,  8.68s/it][2025-04-26 07:02:21,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:02:21,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.66 | bwd_microstep: 5770.04 | bwd_inner_microstep: 5662.51 | bwd_allreduce_microstep: 107.49 | step_microstep: 18.43
[2025-04-26 07:02:21,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.66 | bwd: 5770.05 | bwd_inner: 5662.51 | bwd_allreduce: 107.50 | step: 18.44

  3%|▎         | 1198/41250 [2:53:52<96:32:52,  8.68s/it]
                                                         
{'loss': 0.074, 'grad_norm': 1.951803207397461, 'learning_rate': 3.870759289176091e-05, 'epoch': 0.29}

  3%|▎         | 1198/41250 [2:53:52<96:32:52,  8.68s/it][2025-04-26 07:02:29,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.18
[2025-04-26 07:02:29,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.54 | bwd_microstep: 5708.35 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 55.47 | step_microstep: 19.31
[2025-04-26 07:02:29,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.54 | bwd: 5708.36 | bwd_inner: 5652.83 | bwd_allreduce: 55.49 | step: 19.31

  3%|▎         | 1199/41250 [2:54:00<96:19:42,  8.66s/it]
                                                         
{'loss': 0.148, 'grad_norm': 1.7798840999603271, 'learning_rate': 3.8739903069466886e-05, 'epoch': 0.29}

  3%|▎         | 1199/41250 [2:54:00<96:19:42,  8.66s/it][2025-04-26 07:02:38,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 07:02:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.57 | bwd_microstep: 5751.47 | bwd_inner_microstep: 5689.92 | bwd_allreduce_microstep: 61.51 | step_microstep: 18.70
[2025-04-26 07:02:38,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.57 | bwd: 5751.48 | bwd_inner: 5689.92 | bwd_allreduce: 61.52 | step: 18.70

  3%|▎         | 1200/41250 [2:54:09<96:25:38,  8.67s/it]
                                                         
{'loss': 0.2786, 'grad_norm': 3.435044288635254, 'learning_rate': 3.8772213247172864e-05, 'epoch': 0.29}

  3%|▎         | 1200/41250 [2:54:09<96:25:38,  8.67s/it][2025-04-26 07:02:47,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 07:02:47,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.37 | bwd_microstep: 5767.60 | bwd_inner_microstep: 5695.12 | bwd_allreduce_microstep: 72.43 | step_microstep: 18.62
[2025-04-26 07:02:47,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.37 | bwd: 5767.61 | bwd_inner: 5695.12 | bwd_allreduce: 72.45 | step: 18.62

  3%|▎         | 1201/41250 [2:54:18<96:32:13,  8.68s/it]
                                                         
{'loss': 0.4397, 'grad_norm': 4.859020709991455, 'learning_rate': 3.880452342487884e-05, 'epoch': 0.29}

  3%|▎         | 1201/41250 [2:54:18<96:32:13,  8.68s/it][2025-04-26 07:02:55,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.11 | optimizer_step: 0.90
[2025-04-26 07:02:55,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.69 | bwd_microstep: 5719.74 | bwd_inner_microstep: 5706.90 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.56
[2025-04-26 07:02:55,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.69 | bwd: 5719.75 | bwd_inner: 5706.90 | bwd_allreduce: 12.81 | step: 19.56

  3%|▎         | 1202/41250 [2:54:26<96:29:02,  8.67s/it]
                                                         
{'loss': 0.1133, 'grad_norm': 1.7874846458435059, 'learning_rate': 3.883683360258481e-05, 'epoch': 0.29}

  3%|▎         | 1202/41250 [2:54:26<96:29:02,  8.67s/it][2025-04-26 07:03:04,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 07:03:04,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.91 | bwd_microstep: 5706.01 | bwd_inner_microstep: 5690.91 | bwd_allreduce_microstep: 15.06 | step_microstep: 19.13
[2025-04-26 07:03:04,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.91 | bwd: 5706.03 | bwd_inner: 5690.91 | bwd_allreduce: 15.08 | step: 19.13

  3%|▎         | 1203/41250 [2:54:35<96:23:31,  8.67s/it]
                                                         
{'loss': 0.1092, 'grad_norm': 1.3416274785995483, 'learning_rate': 3.88691437802908e-05, 'epoch': 0.29}

  3%|▎         | 1203/41250 [2:54:35<96:23:31,  8.67s/it][2025-04-26 07:03:13,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:03:13,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.62 | bwd_microstep: 5726.50 | bwd_inner_microstep: 5664.73 | bwd_allreduce_microstep: 61.73 | step_microstep: 18.79
[2025-04-26 07:03:13,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.62 | bwd: 5726.52 | bwd_inner: 5664.73 | bwd_allreduce: 61.75 | step: 18.79

  3%|▎         | 1204/41250 [2:54:44<96:17:23,  8.66s/it]
                                                         
{'loss': 0.2498, 'grad_norm': 1.542614221572876, 'learning_rate': 3.8901453957996776e-05, 'epoch': 0.29}

  3%|▎         | 1204/41250 [2:54:44<96:17:23,  8.66s/it][2025-04-26 07:03:21,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 07:03:21,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.57 | bwd_microstep: 5757.76 | bwd_inner_microstep: 5648.39 | bwd_allreduce_microstep: 109.32 | step_microstep: 18.66
[2025-04-26 07:03:21,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.57 | bwd: 5757.77 | bwd_inner: 5648.39 | bwd_allreduce: 109.34 | step: 18.66

  3%|▎         | 1205/41250 [2:54:52<96:21:11,  8.66s/it]
                                                         
{'loss': 0.4609, 'grad_norm': 2.5778987407684326, 'learning_rate': 3.893376413570275e-05, 'epoch': 0.29}

  3%|▎         | 1205/41250 [2:54:52<96:21:11,  8.66s/it][2025-04-26 07:03:30,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:03:30,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.41 | bwd_microstep: 5718.07 | bwd_inner_microstep: 5705.34 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.32
[2025-04-26 07:03:30,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.41 | bwd: 5718.09 | bwd_inner: 5705.34 | bwd_allreduce: 12.71 | step: 18.33

  3%|▎         | 1206/41250 [2:55:01<96:20:39,  8.66s/it]
                                                         
{'loss': 0.1843, 'grad_norm': 1.8455262184143066, 'learning_rate': 3.8966074313408725e-05, 'epoch': 0.29}

  3%|▎         | 1206/41250 [2:55:01<96:20:39,  8.66s/it][2025-04-26 07:03:39,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:03:39,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.61 | bwd_microstep: 5708.65 | bwd_inner_microstep: 5695.97 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.87
[2025-04-26 07:03:39,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.61 | bwd: 5708.67 | bwd_inner: 5695.97 | bwd_allreduce: 12.66 | step: 17.87

  3%|▎         | 1207/41250 [2:55:10<96:15:58,  8.65s/it]
                                                         
{'loss': 0.2978, 'grad_norm': 2.522752046585083, 'learning_rate': 3.89983844911147e-05, 'epoch': 0.29}

  3%|▎         | 1207/41250 [2:55:10<96:15:58,  8.65s/it][2025-04-26 07:03:47,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 07:03:47,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5766.58 | bwd_inner_microstep: 5691.16 | bwd_allreduce_microstep: 75.37 | step_microstep: 18.37
[2025-04-26 07:03:47,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5766.59 | bwd_inner: 5691.16 | bwd_allreduce: 75.39 | step: 18.37

  3%|▎         | 1208/41250 [2:55:18<96:23:42,  8.67s/it]
                                                         
{'loss': 0.2553, 'grad_norm': 2.6457273960113525, 'learning_rate': 3.903069466882068e-05, 'epoch': 0.29}

  3%|▎         | 1208/41250 [2:55:18<96:23:42,  8.67s/it][2025-04-26 07:03:56,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 07:03:56,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5757.95 | bwd_inner_microstep: 5650.55 | bwd_allreduce_microstep: 107.35 | step_microstep: 18.89
[2025-04-26 07:03:56,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.46 | bwd: 5757.96 | bwd_inner: 5650.55 | bwd_allreduce: 107.37 | step: 18.89

  3%|▎         | 1209/41250 [2:55:27<96:25:20,  8.67s/it]
                                                         
{'loss': 0.3399, 'grad_norm': 3.704226493835449, 'learning_rate': 3.906300484652666e-05, 'epoch': 0.29}

  3%|▎         | 1209/41250 [2:55:27<96:25:20,  8.67s/it][2025-04-26 07:04:05,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 07:04:05,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.18 | bwd_microstep: 5723.12 | bwd_inner_microstep: 5710.44 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.49
[2025-04-26 07:04:05,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.18 | bwd: 5723.13 | bwd_inner: 5710.44 | bwd_allreduce: 12.65 | step: 18.49

  3%|▎         | 1210/41250 [2:55:36<96:22:28,  8.67s/it]
                                                         
{'loss': 0.2263, 'grad_norm': 1.7223467826843262, 'learning_rate': 3.9095315024232636e-05, 'epoch': 0.29}

  3%|▎         | 1210/41250 [2:55:36<96:22:28,  8.67s/it][2025-04-26 07:04:13,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-26 07:04:13,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.03 | bwd_microstep: 5771.46 | bwd_inner_microstep: 5653.61 | bwd_allreduce_microstep: 117.81 | step_microstep: 18.85
[2025-04-26 07:04:13,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.03 | bwd: 5771.48 | bwd_inner: 5653.61 | bwd_allreduce: 117.83 | step: 18.86

  3%|▎         | 1211/41250 [2:55:44<96:25:17,  8.67s/it]
                                                         
{'loss': 0.3383, 'grad_norm': 2.763253927230835, 'learning_rate': 3.9127625201938614e-05, 'epoch': 0.29}

  3%|▎         | 1211/41250 [2:55:44<96:25:17,  8.67s/it][2025-04-26 07:04:22,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:04:22,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.22 | bwd_microstep: 5786.13 | bwd_inner_microstep: 5773.46 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.28
[2025-04-26 07:04:22,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.22 | bwd: 5786.14 | bwd_inner: 5773.46 | bwd_allreduce: 12.64 | step: 18.28

  3%|▎         | 1212/41250 [2:55:53<96:43:44,  8.70s/it]
                                                         
{'loss': 0.1927, 'grad_norm': 3.7043654918670654, 'learning_rate': 3.915993537964459e-05, 'epoch': 0.29}

  3%|▎         | 1212/41250 [2:55:53<96:43:44,  8.70s/it][2025-04-26 07:04:31,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:04:31,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5723.24 | bwd_inner_microstep: 5710.42 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.85
[2025-04-26 07:04:31,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5723.25 | bwd_inner: 5710.42 | bwd_allreduce: 12.79 | step: 18.85

  3%|▎         | 1213/41250 [2:56:02<96:34:29,  8.68s/it]
                                                         
{'loss': 0.1214, 'grad_norm': 0.8272423148155212, 'learning_rate': 3.919224555735057e-05, 'epoch': 0.29}

  3%|▎         | 1213/41250 [2:56:02<96:34:29,  8.68s/it][2025-04-26 07:04:39,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 07:04:39,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.59 | bwd_microstep: 5724.84 | bwd_inner_microstep: 5700.32 | bwd_allreduce_microstep: 24.48 | step_microstep: 17.81
[2025-04-26 07:04:39,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.59 | bwd: 5724.85 | bwd_inner: 5700.32 | bwd_allreduce: 24.50 | step: 17.81

  3%|▎         | 1214/41250 [2:56:10<96:29:28,  8.68s/it]
                                                         
{'loss': 0.2242, 'grad_norm': 1.9822131395339966, 'learning_rate': 3.922455573505655e-05, 'epoch': 0.29}

  3%|▎         | 1214/41250 [2:56:10<96:29:28,  8.68s/it][2025-04-26 07:04:48,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 07:04:48,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.64 | bwd_microstep: 5750.44 | bwd_inner_microstep: 5698.16 | bwd_allreduce_microstep: 52.23 | step_microstep: 19.01
[2025-04-26 07:04:48,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.64 | bwd: 5750.45 | bwd_inner: 5698.16 | bwd_allreduce: 52.25 | step: 19.01

  3%|▎         | 1215/41250 [2:56:19<96:31:32,  8.68s/it]
                                                         
{'loss': 0.1551, 'grad_norm': 1.809147596359253, 'learning_rate': 3.9256865912762526e-05, 'epoch': 0.29}

  3%|▎         | 1215/41250 [2:56:19<96:31:32,  8.68s/it][2025-04-26 07:04:57,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:04:57,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5715.19 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.83
[2025-04-26 07:04:57,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5715.20 | bwd_inner: 5702.41 | bwd_allreduce: 12.74 | step: 18.83

  3%|▎         | 1216/41250 [2:56:28<96:24:09,  8.67s/it]
                                                         
{'loss': 0.0773, 'grad_norm': 0.8977442979812622, 'learning_rate': 3.9289176090468497e-05, 'epoch': 0.29}

  3%|▎         | 1216/41250 [2:56:28<96:24:09,  8.67s/it][2025-04-26 07:05:05,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 07:05:05,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.10 | bwd_microstep: 5708.61 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.73
[2025-04-26 07:05:05,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.10 | bwd: 5708.63 | bwd_inner: 5695.67 | bwd_allreduce: 12.91 | step: 18.73

  3%|▎         | 1217/41250 [2:56:36<96:15:37,  8.66s/it]
                                                         
{'loss': 0.1079, 'grad_norm': 1.9105132818222046, 'learning_rate': 3.932148626817448e-05, 'epoch': 0.3}

  3%|▎         | 1217/41250 [2:56:36<96:15:37,  8.66s/it][2025-04-26 07:05:14,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:05:14,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.83 | bwd_microstep: 5750.28 | bwd_inner_microstep: 5653.24 | bwd_allreduce_microstep: 96.99 | step_microstep: 18.94
[2025-04-26 07:05:14,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.83 | bwd: 5750.29 | bwd_inner: 5653.24 | bwd_allreduce: 97.01 | step: 18.95

  3%|▎         | 1218/41250 [2:56:45<96:16:33,  8.66s/it]
                                                         
{'loss': 0.2666, 'grad_norm': 1.747903823852539, 'learning_rate': 3.935379644588046e-05, 'epoch': 0.3}

  3%|▎         | 1218/41250 [2:56:45<96:16:33,  8.66s/it][2025-04-26 07:05:23,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:05:23,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.64 | bwd_microstep: 5767.95 | bwd_inner_microstep: 5636.00 | bwd_allreduce_microstep: 131.90 | step_microstep: 19.04
[2025-04-26 07:05:23,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.64 | bwd: 5767.96 | bwd_inner: 5636.00 | bwd_allreduce: 131.91 | step: 19.04

  3%|▎         | 1219/41250 [2:56:54<96:20:29,  8.66s/it]
                                                         
{'loss': 0.2547, 'grad_norm': 3.90028977394104, 'learning_rate': 3.938610662358643e-05, 'epoch': 0.3}

  3%|▎         | 1219/41250 [2:56:54<96:20:29,  8.66s/it][2025-04-26 07:05:31,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 1.14
[2025-04-26 07:05:31,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.60 | bwd_microstep: 5747.69 | bwd_inner_microstep: 5681.65 | bwd_allreduce_microstep: 65.99 | step_microstep: 18.80
[2025-04-26 07:05:31,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.60 | bwd: 5747.70 | bwd_inner: 5681.65 | bwd_allreduce: 66.01 | step: 18.80

  3%|▎         | 1220/41250 [2:57:02<96:21:47,  8.67s/it]
                                                         
{'loss': 0.2915, 'grad_norm': 2.4327497482299805, 'learning_rate': 3.941841680129241e-05, 'epoch': 0.3}

  3%|▎         | 1220/41250 [2:57:02<96:21:47,  8.67s/it][2025-04-26 07:05:40,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 07:05:40,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.83 | bwd_microstep: 5743.80 | bwd_inner_microstep: 5700.78 | bwd_allreduce_microstep: 42.98 | step_microstep: 18.41
[2025-04-26 07:05:40,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.83 | bwd: 5743.82 | bwd_inner: 5700.78 | bwd_allreduce: 43.00 | step: 18.41

  3%|▎         | 1221/41250 [2:57:11<96:25:48,  8.67s/it]
                                                         
{'loss': 0.4285, 'grad_norm': 2.329854726791382, 'learning_rate': 3.9450726978998386e-05, 'epoch': 0.3}

  3%|▎         | 1221/41250 [2:57:11<96:25:48,  8.67s/it][2025-04-26 07:05:49,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:05:49,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.39 | bwd_microstep: 5706.24 | bwd_inner_microstep: 5693.57 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.38
[2025-04-26 07:05:49,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.39 | bwd: 5706.25 | bwd_inner: 5693.57 | bwd_allreduce: 12.64 | step: 18.38

  3%|▎         | 1222/41250 [2:57:20<96:15:17,  8.66s/it]
                                                         
{'loss': 0.2134, 'grad_norm': 2.4853973388671875, 'learning_rate': 3.9483037156704364e-05, 'epoch': 0.3}

  3%|▎         | 1222/41250 [2:57:20<96:15:17,  8.66s/it][2025-04-26 07:05:57,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:05:57,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.23 | bwd_microstep: 5756.35 | bwd_inner_microstep: 5635.60 | bwd_allreduce_microstep: 120.71 | step_microstep: 18.10
[2025-04-26 07:05:57,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.23 | bwd: 5756.37 | bwd_inner: 5635.60 | bwd_allreduce: 120.73 | step: 18.11

  3%|▎         | 1223/41250 [2:57:28<96:17:57,  8.66s/it]
                                                         
{'loss': 0.298, 'grad_norm': 2.2784175872802734, 'learning_rate': 3.951534733441034e-05, 'epoch': 0.3}

  3%|▎         | 1223/41250 [2:57:28<96:17:57,  8.66s/it][2025-04-26 07:06:06,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:06:06,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.00 | bwd_microstep: 5695.95 | bwd_inner_microstep: 5651.83 | bwd_allreduce_microstep: 44.08 | step_microstep: 18.35
[2025-04-26 07:06:06,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.00 | bwd: 5695.96 | bwd_inner: 5651.83 | bwd_allreduce: 44.09 | step: 18.35

  3%|▎         | 1224/41250 [2:57:37<96:06:31,  8.64s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.037757396697998, 'learning_rate': 3.954765751211632e-05, 'epoch': 0.3}

  3%|▎         | 1224/41250 [2:57:37<96:06:31,  8.64s/it][2025-04-26 07:06:15,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 07:06:15,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.07 | bwd_microstep: 5770.49 | bwd_inner_microstep: 5650.34 | bwd_allreduce_microstep: 120.10 | step_microstep: 18.80
[2025-04-26 07:06:15,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.07 | bwd: 5770.50 | bwd_inner: 5650.34 | bwd_allreduce: 120.12 | step: 18.81

  3%|▎         | 1225/41250 [2:57:46<96:13:00,  8.65s/it]
                                                         
{'loss': 0.1592, 'grad_norm': 1.1239157915115356, 'learning_rate': 3.95799676898223e-05, 'epoch': 0.3}

  3%|▎         | 1225/41250 [2:57:46<96:13:00,  8.65s/it][2025-04-26 07:06:23,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:06:23,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5733.01 | bwd_inner_microstep: 5688.44 | bwd_allreduce_microstep: 44.53 | step_microstep: 18.37
[2025-04-26 07:06:23,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.68 | bwd: 5733.02 | bwd_inner: 5688.44 | bwd_allreduce: 44.54 | step: 18.38

  3%|▎         | 1226/41250 [2:57:54<96:12:06,  8.65s/it]
                                                         
{'loss': 0.2897, 'grad_norm': 1.741214394569397, 'learning_rate': 3.9612277867528275e-05, 'epoch': 0.3}

  3%|▎         | 1226/41250 [2:57:54<96:12:06,  8.65s/it][2025-04-26 07:06:32,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:06:32,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.51 | bwd_microstep: 5776.31 | bwd_inner_microstep: 5654.20 | bwd_allreduce_microstep: 122.06 | step_microstep: 18.24
[2025-04-26 07:06:32,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.51 | bwd: 5776.32 | bwd_inner: 5654.20 | bwd_allreduce: 122.08 | step: 18.24

  3%|▎         | 1227/41250 [2:58:03<96:18:16,  8.66s/it]
                                                         
{'loss': 0.3138, 'grad_norm': 1.944156527519226, 'learning_rate': 3.964458804523425e-05, 'epoch': 0.3}

  3%|▎         | 1227/41250 [2:58:03<96:18:16,  8.66s/it][2025-04-26 07:06:41,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:06:41,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.49 | bwd_microstep: 5754.52 | bwd_inner_microstep: 5703.98 | bwd_allreduce_microstep: 50.50 | step_microstep: 18.24
[2025-04-26 07:06:41,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.49 | bwd: 5754.53 | bwd_inner: 5703.98 | bwd_allreduce: 50.51 | step: 18.24

  3%|▎         | 1228/41250 [2:58:12<96:22:01,  8.67s/it]
                                                         
{'loss': 0.1799, 'grad_norm': 1.8032227754592896, 'learning_rate': 3.967689822294023e-05, 'epoch': 0.3}

  3%|▎         | 1228/41250 [2:58:12<96:22:01,  8.67s/it][2025-04-26 07:06:49,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:06:49,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.68 | bwd_microstep: 5684.71 | bwd_inner_microstep: 5648.21 | bwd_allreduce_microstep: 36.46 | step_microstep: 18.59
[2025-04-26 07:06:49,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.68 | bwd: 5684.72 | bwd_inner: 5648.21 | bwd_allreduce: 36.48 | step: 18.59

  3%|▎         | 1229/41250 [2:58:20<96:06:24,  8.65s/it]
                                                         
{'loss': 0.1322, 'grad_norm': 0.8507192730903625, 'learning_rate': 3.970920840064621e-05, 'epoch': 0.3}

  3%|▎         | 1229/41250 [2:58:20<96:06:24,  8.65s/it][2025-04-26 07:06:58,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 07:06:58,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.73 | bwd_microstep: 5729.27 | bwd_inner_microstep: 5682.22 | bwd_allreduce_microstep: 47.01 | step_microstep: 19.03
[2025-04-26 07:06:58,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.73 | bwd: 5729.29 | bwd_inner: 5682.22 | bwd_allreduce: 47.03 | step: 19.03

  3%|▎         | 1230/41250 [2:58:29<96:08:27,  8.65s/it]
                                                         
{'loss': 0.2471, 'grad_norm': 1.504062533378601, 'learning_rate': 3.974151857835218e-05, 'epoch': 0.3}

  3%|▎         | 1230/41250 [2:58:29<96:08:27,  8.65s/it][2025-04-26 07:07:06,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:07:06,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.50 | bwd_microstep: 5742.15 | bwd_inner_microstep: 5696.31 | bwd_allreduce_microstep: 45.80 | step_microstep: 18.40
[2025-04-26 07:07:06,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.50 | bwd: 5742.17 | bwd_inner: 5696.31 | bwd_allreduce: 45.82 | step: 18.41

  3%|▎         | 1231/41250 [2:58:38<96:12:32,  8.65s/it]
                                                         
{'loss': 0.1495, 'grad_norm': 1.1877516508102417, 'learning_rate': 3.9773828756058165e-05, 'epoch': 0.3}

  3%|▎         | 1231/41250 [2:58:38<96:12:32,  8.65s/it][2025-04-26 07:07:15,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 07:07:15,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.41 | bwd_microstep: 5878.18 | bwd_inner_microstep: 5656.11 | bwd_allreduce_microstep: 222.03 | step_microstep: 18.20
[2025-04-26 07:07:15,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.41 | bwd: 5878.19 | bwd_inner: 5656.11 | bwd_allreduce: 222.05 | step: 18.20

  3%|▎         | 1232/41250 [2:58:46<96:40:28,  8.70s/it]
                                                         
{'loss': 0.3474, 'grad_norm': 2.227449655532837, 'learning_rate': 3.980613893376414e-05, 'epoch': 0.3}

  3%|▎         | 1232/41250 [2:58:46<96:40:28,  8.70s/it][2025-04-26 07:07:24,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:07:24,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.29 | bwd_microstep: 5681.65 | bwd_inner_microstep: 5647.17 | bwd_allreduce_microstep: 34.44 | step_microstep: 18.77
[2025-04-26 07:07:24,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.29 | bwd: 5681.67 | bwd_inner: 5647.17 | bwd_allreduce: 34.46 | step: 18.77

  3%|▎         | 1233/41250 [2:58:55<96:20:47,  8.67s/it]
                                                         
{'loss': 0.3324, 'grad_norm': 1.9018611907958984, 'learning_rate': 3.9838449111470114e-05, 'epoch': 0.3}

  3%|▎         | 1233/41250 [2:58:55<96:20:47,  8.67s/it][2025-04-26 07:07:33,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:07:33,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.88 | bwd_microstep: 5872.25 | bwd_inner_microstep: 5652.71 | bwd_allreduce_microstep: 219.50 | step_microstep: 18.64
[2025-04-26 07:07:33,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.88 | bwd: 5872.27 | bwd_inner: 5652.71 | bwd_allreduce: 219.52 | step: 18.64

  3%|▎         | 1234/41250 [2:59:04<96:45:08,  8.70s/it]
                                                         
{'loss': 0.2935, 'grad_norm': 1.684394121170044, 'learning_rate': 3.987075928917609e-05, 'epoch': 0.3}

  3%|▎         | 1234/41250 [2:59:04<96:45:08,  8.70s/it][2025-04-26 07:07:41,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 07:07:41,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.47 | bwd_microstep: 5767.04 | bwd_inner_microstep: 5660.70 | bwd_allreduce_microstep: 106.29 | step_microstep: 18.92
[2025-04-26 07:07:41,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.47 | bwd: 5767.06 | bwd_inner: 5660.70 | bwd_allreduce: 106.31 | step: 18.93

  3%|▎         | 1235/41250 [2:59:12<96:39:20,  8.70s/it]
                                                         
{'loss': 0.1503, 'grad_norm': 1.2820775508880615, 'learning_rate': 3.990306946688207e-05, 'epoch': 0.3}

  3%|▎         | 1235/41250 [2:59:12<96:39:20,  8.70s/it][2025-04-26 07:07:50,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:07:50,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.81 | bwd_microstep: 5863.68 | bwd_inner_microstep: 5851.04 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.75
[2025-04-26 07:07:50,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.81 | bwd: 5863.69 | bwd_inner: 5851.04 | bwd_allreduce: 12.61 | step: 17.76

  3%|▎         | 1236/41250 [2:59:21<97:16:59,  8.75s/it]
                                                         
{'loss': 0.2608, 'grad_norm': 1.5520447492599487, 'learning_rate': 3.993537964458805e-05, 'epoch': 0.3}

  3%|▎         | 1236/41250 [2:59:21<97:16:59,  8.75s/it][2025-04-26 07:07:59,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.09 | optimizer_step: 0.95
[2025-04-26 07:07:59,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.03 | bwd_microstep: 5702.12 | bwd_inner_microstep: 5688.22 | bwd_allreduce_microstep: 13.84 | step_microstep: 19.83
[2025-04-26 07:07:59,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.03 | bwd: 5702.13 | bwd_inner: 5688.22 | bwd_allreduce: 13.87 | step: 19.83

  3%|▎         | 1237/41250 [2:59:30<96:53:24,  8.72s/it]
                                                         
{'loss': 0.2283, 'grad_norm': 1.201412320137024, 'learning_rate': 3.9967689822294025e-05, 'epoch': 0.3}

  3%|▎         | 1237/41250 [2:59:30<96:53:24,  8.72s/it][2025-04-26 07:08:08,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 07:08:08,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5789.41 | bwd_inner_microstep: 5687.56 | bwd_allreduce_microstep: 101.80 | step_microstep: 19.06
[2025-04-26 07:08:08,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5789.43 | bwd_inner: 5687.56 | bwd_allreduce: 101.82 | step: 19.06

  3%|▎         | 1238/41250 [2:59:39<96:53:05,  8.72s/it]
                                                         
{'loss': 0.1085, 'grad_norm': 0.6126284003257751, 'learning_rate': 4e-05, 'epoch': 0.3}

  3%|▎         | 1238/41250 [2:59:39<96:53:05,  8.72s/it][2025-04-26 07:08:16,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:08:16,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5767.91 | bwd_inner_microstep: 5702.78 | bwd_allreduce_microstep: 65.08 | step_microstep: 18.88
[2025-04-26 07:08:16,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5767.92 | bwd_inner: 5702.78 | bwd_allreduce: 65.10 | step: 18.88

  3%|▎         | 1239/41250 [2:59:47<96:49:29,  8.71s/it]
                                                         
{'loss': 0.3765, 'grad_norm': 2.3863790035247803, 'learning_rate': 3.999999993835197e-05, 'epoch': 0.3}

  3%|▎         | 1239/41250 [2:59:47<96:49:29,  8.71s/it][2025-04-26 07:08:25,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 07:08:25,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.67 | bwd_microstep: 5714.59 | bwd_inner_microstep: 5701.71 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.83
[2025-04-26 07:08:25,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.67 | bwd: 5714.60 | bwd_inner: 5701.71 | bwd_allreduce: 12.85 | step: 18.83

  3%|▎         | 1240/41250 [2:59:56<96:33:57,  8.69s/it]
                                                         
{'loss': 0.038, 'grad_norm': 0.4029396176338196, 'learning_rate': 3.999999975340787e-05, 'epoch': 0.3}

  3%|▎         | 1240/41250 [2:59:56<96:33:57,  8.69s/it][2025-04-26 07:08:34,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 07:08:34,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.20 | bwd_microstep: 5738.91 | bwd_inner_microstep: 5708.57 | bwd_allreduce_microstep: 30.28 | step_microstep: 19.11
[2025-04-26 07:08:34,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.20 | bwd: 5738.92 | bwd_inner: 5708.57 | bwd_allreduce: 30.30 | step: 19.11

  3%|▎         | 1241/41250 [3:00:05<96:33:14,  8.69s/it]
                                                         
{'loss': 0.3402, 'grad_norm': 1.916288137435913, 'learning_rate': 3.999999944516771e-05, 'epoch': 0.3}

  3%|▎         | 1241/41250 [3:00:05<96:33:14,  8.69s/it][2025-04-26 07:08:42,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:08:42,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.28 | bwd_microstep: 5800.05 | bwd_inner_microstep: 5787.18 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.89
[2025-04-26 07:08:42,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.28 | bwd: 5800.06 | bwd_inner: 5787.18 | bwd_allreduce: 12.84 | step: 18.89

  3%|▎         | 1242/41250 [3:00:14<96:50:17,  8.71s/it]
                                                         
{'loss': 0.169, 'grad_norm': 1.3454382419586182, 'learning_rate': 3.999999901363148e-05, 'epoch': 0.3}

  3%|▎         | 1242/41250 [3:00:14<96:50:17,  8.71s/it][2025-04-26 07:08:51,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:08:51,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5916.79 | bwd_inner_microstep: 5713.75 | bwd_allreduce_microstep: 202.99 | step_microstep: 18.35
[2025-04-26 07:08:51,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5916.80 | bwd_inner: 5713.75 | bwd_allreduce: 203.00 | step: 18.35

  3%|▎         | 1243/41250 [3:00:22<97:17:02,  8.75s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 1.432766079902649, 'learning_rate': 3.9999998458799195e-05, 'epoch': 0.3}

  3%|▎         | 1243/41250 [3:00:22<97:17:02,  8.75s/it][2025-04-26 07:09:00,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 07:09:00,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.76 | bwd_microstep: 5758.19 | bwd_inner_microstep: 5661.45 | bwd_allreduce_microstep: 96.70 | step_microstep: 18.13
[2025-04-26 07:09:00,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.76 | bwd: 5758.21 | bwd_inner: 5661.45 | bwd_allreduce: 96.72 | step: 18.13

  3%|▎         | 1244/41250 [3:00:31<96:58:48,  8.73s/it]
                                                         
{'loss': 0.2387, 'grad_norm': 1.508737325668335, 'learning_rate': 3.999999778067085e-05, 'epoch': 0.3}

  3%|▎         | 1244/41250 [3:00:31<96:58:48,  8.73s/it][2025-04-26 07:09:09,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 07:09:09,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5761.31 | bwd_inner_microstep: 5663.81 | bwd_allreduce_microstep: 97.47 | step_microstep: 18.95
[2025-04-26 07:09:09,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5761.33 | bwd_inner: 5663.81 | bwd_allreduce: 97.48 | step: 18.95

  3%|▎         | 1245/41250 [3:00:40<96:52:56,  8.72s/it]
                                                         
{'loss': 0.1476, 'grad_norm': 1.4873459339141846, 'learning_rate': 3.9999996979246454e-05, 'epoch': 0.3}

  3%|▎         | 1245/41250 [3:00:40<96:52:56,  8.72s/it][2025-04-26 07:09:17,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:09:17,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.81 | bwd_microstep: 5754.35 | bwd_inner_microstep: 5649.38 | bwd_allreduce_microstep: 104.93 | step_microstep: 18.83
[2025-04-26 07:09:17,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.81 | bwd: 5754.36 | bwd_inner: 5649.38 | bwd_allreduce: 104.95 | step: 18.84

  3%|▎         | 1246/41250 [3:00:48<96:44:29,  8.71s/it]
                                                         
{'loss': 0.2759, 'grad_norm': 2.458031177520752, 'learning_rate': 3.999999605452601e-05, 'epoch': 0.3}

  3%|▎         | 1246/41250 [3:00:48<96:44:29,  8.71s/it][2025-04-26 07:09:26,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 07:09:26,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.27 | bwd_microstep: 5750.16 | bwd_inner_microstep: 5711.97 | bwd_allreduce_microstep: 38.14 | step_microstep: 18.55
[2025-04-26 07:09:26,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.27 | bwd: 5750.17 | bwd_inner: 5711.97 | bwd_allreduce: 38.16 | step: 18.55

  3%|▎         | 1247/41250 [3:00:57<96:41:11,  8.70s/it]
                                                         
{'loss': 0.1791, 'grad_norm': 1.1696895360946655, 'learning_rate': 3.9999995006509526e-05, 'epoch': 0.3}

  3%|▎         | 1247/41250 [3:00:57<96:41:11,  8.70s/it][2025-04-26 07:09:35,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:09:35,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.39 | bwd_microstep: 5704.39 | bwd_inner_microstep: 5691.55 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.68
[2025-04-26 07:09:35,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.39 | bwd: 5704.40 | bwd_inner: 5691.55 | bwd_allreduce: 12.81 | step: 18.69

  3%|▎         | 1248/41250 [3:01:06<96:27:39,  8.68s/it]
                                                         
{'loss': 0.3047, 'grad_norm': 2.0075149536132812, 'learning_rate': 3.999999383519701e-05, 'epoch': 0.3}

  3%|▎         | 1248/41250 [3:01:06<96:27:39,  8.68s/it][2025-04-26 07:09:43,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:09:43,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.49 | bwd_microstep: 5723.84 | bwd_inner_microstep: 5661.29 | bwd_allreduce_microstep: 62.51 | step_microstep: 18.52
[2025-04-26 07:09:43,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.49 | bwd: 5723.86 | bwd_inner: 5661.29 | bwd_allreduce: 62.53 | step: 18.52

  3%|▎         | 1249/41250 [3:01:14<96:16:27,  8.66s/it]
                                                         
{'loss': 0.1445, 'grad_norm': 1.2009952068328857, 'learning_rate': 3.9999992540588456e-05, 'epoch': 0.3}

  3%|▎         | 1249/41250 [3:01:14<96:16:27,  8.66s/it][2025-04-26 07:09:52,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:09:52,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.54 | bwd_microstep: 5732.30 | bwd_inner_microstep: 5719.64 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.45
[2025-04-26 07:09:52,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.54 | bwd: 5732.31 | bwd_inner: 5719.64 | bwd_allreduce: 12.63 | step: 18.45

  3%|▎         | 1250/41250 [3:01:23<96:15:51,  8.66s/it]
                                                         
{'loss': 0.1478, 'grad_norm': 1.4879333972930908, 'learning_rate': 3.9999991122683885e-05, 'epoch': 0.3}

  3%|▎         | 1250/41250 [3:01:23<96:15:51,  8.66s/it][2025-04-26 07:10:00,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 07:10:00,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.23 | bwd_microstep: 5695.08 | bwd_inner_microstep: 5661.80 | bwd_allreduce_microstep: 33.23 | step_microstep: 18.86
[2025-04-26 07:10:00,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.23 | bwd: 5695.09 | bwd_inner: 5661.80 | bwd_allreduce: 33.25 | step: 18.86

  3%|▎         | 1251/41250 [3:01:32<96:04:54,  8.65s/it]
                                                         
{'loss': 0.2087, 'grad_norm': 2.4356026649475098, 'learning_rate': 3.999998958148331e-05, 'epoch': 0.3}

  3%|▎         | 1251/41250 [3:01:32<96:04:54,  8.65s/it][2025-04-26 07:10:09,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:10:09,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.93 | bwd_microstep: 5707.18 | bwd_inner_microstep: 5694.37 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.76
[2025-04-26 07:10:09,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.93 | bwd: 5707.20 | bwd_inner: 5694.37 | bwd_allreduce: 12.79 | step: 18.77

  3%|▎         | 1252/41250 [3:01:40<96:06:16,  8.65s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 1.2221331596374512, 'learning_rate': 3.999998791698672e-05, 'epoch': 0.3}

  3%|▎         | 1252/41250 [3:01:40<96:06:16,  8.65s/it][2025-04-26 07:10:18,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:10:18,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.31 | bwd_microstep: 5767.14 | bwd_inner_microstep: 5659.60 | bwd_allreduce_microstep: 107.49 | step_microstep: 18.87
[2025-04-26 07:10:18,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.31 | bwd: 5767.15 | bwd_inner: 5659.60 | bwd_allreduce: 107.51 | step: 18.87

  3%|▎         | 1253/41250 [3:01:49<96:12:58,  8.66s/it]
                                                         
{'loss': 0.2684, 'grad_norm': 1.9490809440612793, 'learning_rate': 3.9999986129194146e-05, 'epoch': 0.3}

  3%|▎         | 1253/41250 [3:01:49<96:12:58,  8.66s/it][2025-04-26 07:10:26,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 07:10:26,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.52 | bwd_microstep: 5706.27 | bwd_inner_microstep: 5644.37 | bwd_allreduce_microstep: 61.86 | step_microstep: 19.17
[2025-04-26 07:10:26,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.52 | bwd: 5706.29 | bwd_inner: 5644.37 | bwd_allreduce: 61.88 | step: 19.17

  3%|▎         | 1254/41250 [3:01:58<96:07:32,  8.65s/it]
                                                         
{'loss': 0.1668, 'grad_norm': 1.2931559085845947, 'learning_rate': 3.9999984218105596e-05, 'epoch': 0.3}

  3%|▎         | 1254/41250 [3:01:58<96:07:32,  8.65s/it][2025-04-26 07:10:35,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 07:10:35,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.16 | bwd_microstep: 5891.45 | bwd_inner_microstep: 5652.62 | bwd_allreduce_microstep: 238.77 | step_microstep: 18.87
[2025-04-26 07:10:35,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.17 | bwd: 5891.46 | bwd_inner: 5652.62 | bwd_allreduce: 238.79 | step: 18.87

  3%|▎         | 1255/41250 [3:02:06<96:38:57,  8.70s/it]
                                                         
{'loss': 0.2635, 'grad_norm': 1.7180346250534058, 'learning_rate': 3.9999982183721074e-05, 'epoch': 0.3}

  3%|▎         | 1255/41250 [3:02:06<96:38:57,  8.70s/it][2025-04-26 07:10:44,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 07:10:44,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.42 | bwd_microstep: 5761.38 | bwd_inner_microstep: 5700.13 | bwd_allreduce_microstep: 61.20 | step_microstep: 19.14
[2025-04-26 07:10:44,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.42 | bwd: 5761.39 | bwd_inner: 5700.13 | bwd_allreduce: 61.22 | step: 19.14

  3%|▎         | 1256/41250 [3:02:15<96:38:30,  8.70s/it]
                                                         
{'loss': 0.0655, 'grad_norm': 1.2096562385559082, 'learning_rate': 3.999998002604059e-05, 'epoch': 0.3}

  3%|▎         | 1256/41250 [3:02:15<96:38:30,  8.70s/it][2025-04-26 07:10:53,041] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.10
[2025-04-26 07:10:53,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.25 | bwd_microstep: 5690.62 | bwd_inner_microstep: 5645.29 | bwd_allreduce_microstep: 45.29 | step_microstep: 18.83
[2025-04-26 07:10:53,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.25 | bwd: 5690.63 | bwd_inner: 5645.29 | bwd_allreduce: 45.30 | step: 18.83

  3%|▎         | 1257/41250 [3:02:24<96:21:37,  8.67s/it]
                                                         
{'loss': 0.1766, 'grad_norm': 1.204119086265564, 'learning_rate': 3.999997774506417e-05, 'epoch': 0.3}

  3%|▎         | 1257/41250 [3:02:24<96:21:37,  8.67s/it][2025-04-26 07:11:01,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:11:01,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.47 | bwd_microstep: 5757.66 | bwd_inner_microstep: 5662.93 | bwd_allreduce_microstep: 94.69 | step_microstep: 18.39
[2025-04-26 07:11:01,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.47 | bwd: 5757.67 | bwd_inner: 5662.93 | bwd_allreduce: 94.70 | step: 18.39

  3%|▎         | 1258/41250 [3:02:32<96:19:21,  8.67s/it]
                                                         
{'loss': 0.0874, 'grad_norm': 0.9386754631996155, 'learning_rate': 3.9999975340791816e-05, 'epoch': 0.3}

  3%|▎         | 1258/41250 [3:02:32<96:19:21,  8.67s/it][2025-04-26 07:11:10,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:11:10,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.60 | bwd_microstep: 5765.98 | bwd_inner_microstep: 5752.52 | bwd_allreduce_microstep: 13.41 | step_microstep: 18.25
[2025-04-26 07:11:10,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.60 | bwd: 5766.00 | bwd_inner: 5752.52 | bwd_allreduce: 13.43 | step: 18.25

  3%|▎         | 1259/41250 [3:02:41<96:33:25,  8.69s/it]
                                                         
{'loss': 0.268, 'grad_norm': 2.1041946411132812, 'learning_rate': 3.9999972813223546e-05, 'epoch': 0.31}

  3%|▎         | 1259/41250 [3:02:41<96:33:25,  8.69s/it][2025-04-26 07:11:19,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 07:11:19,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.57 | bwd_microstep: 5778.78 | bwd_inner_microstep: 5658.17 | bwd_allreduce_microstep: 120.56 | step_microstep: 18.24
[2025-04-26 07:11:19,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.57 | bwd: 5778.79 | bwd_inner: 5658.17 | bwd_allreduce: 120.58 | step: 18.25

  3%|▎         | 1260/41250 [3:02:50<96:33:58,  8.69s/it]
                                                         
{'loss': 0.2091, 'grad_norm': 1.5307517051696777, 'learning_rate': 3.9999970162359385e-05, 'epoch': 0.31}

  3%|▎         | 1260/41250 [3:02:50<96:33:58,  8.69s/it][2025-04-26 07:11:27,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-26 07:11:27,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5733.94 | bwd_inner_microstep: 5645.71 | bwd_allreduce_microstep: 88.18 | step_microstep: 18.81
[2025-04-26 07:11:27,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5733.96 | bwd_inner: 5645.71 | bwd_allreduce: 88.20 | step: 18.81

  3%|▎         | 1261/41250 [3:02:58<96:29:14,  8.69s/it]
                                                         
{'loss': 0.1464, 'grad_norm': 1.5521256923675537, 'learning_rate': 3.999996738819934e-05, 'epoch': 0.31}

  3%|▎         | 1261/41250 [3:02:59<96:29:14,  8.69s/it][2025-04-26 07:11:36,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:11:36,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.82 | bwd_microstep: 5747.52 | bwd_inner_microstep: 5653.87 | bwd_allreduce_microstep: 93.61 | step_microstep: 18.50
[2025-04-26 07:11:36,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.82 | bwd: 5747.53 | bwd_inner: 5653.87 | bwd_allreduce: 93.63 | step: 18.51

  3%|▎         | 1262/41250 [3:03:07<96:23:08,  8.68s/it]
                                                         
{'loss': 0.2245, 'grad_norm': 1.853559136390686, 'learning_rate': 3.9999964490743423e-05, 'epoch': 0.31}

  3%|▎         | 1262/41250 [3:03:07<96:23:08,  8.68s/it][2025-04-26 07:11:45,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:11:45,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.33 | bwd_microstep: 5775.14 | bwd_inner_microstep: 5649.36 | bwd_allreduce_microstep: 125.74 | step_microstep: 18.64
[2025-04-26 07:11:45,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.33 | bwd: 5775.16 | bwd_inner: 5649.36 | bwd_allreduce: 125.76 | step: 18.64

  3%|▎         | 1263/41250 [3:03:16<96:24:33,  8.68s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 1.089159369468689, 'learning_rate': 3.9999961469991664e-05, 'epoch': 0.31}

  3%|▎         | 1263/41250 [3:03:16<96:24:33,  8.68s/it][2025-04-26 07:11:53,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:11:53,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5741.99 | bwd_inner_microstep: 5686.96 | bwd_allreduce_microstep: 54.99 | step_microstep: 18.21
[2025-04-26 07:11:53,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5742.01 | bwd_inner: 5686.96 | bwd_allreduce: 55.01 | step: 18.21

  3%|▎         | 1264/41250 [3:03:25<96:23:51,  8.68s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 2.231224536895752, 'learning_rate': 3.9999958325944073e-05, 'epoch': 0.31}

  3%|▎         | 1264/41250 [3:03:25<96:23:51,  8.68s/it][2025-04-26 07:12:02,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:12:02,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.01 | bwd_microstep: 5739.41 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 48.52 | step_microstep: 17.86
[2025-04-26 07:12:02,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.01 | bwd: 5739.42 | bwd_inner: 5690.85 | bwd_allreduce: 48.53 | step: 17.86

  3%|▎         | 1265/41250 [3:03:33<96:22:42,  8.68s/it]
                                                         
{'loss': 0.3462, 'grad_norm': 2.177887439727783, 'learning_rate': 3.9999955058600674e-05, 'epoch': 0.31}

  3%|▎         | 1265/41250 [3:03:33<96:22:42,  8.68s/it][2025-04-26 07:12:11,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 07:12:11,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.53 | bwd_microstep: 5856.83 | bwd_inner_microstep: 5640.84 | bwd_allreduce_microstep: 215.94 | step_microstep: 18.81
[2025-04-26 07:12:11,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.53 | bwd: 5856.84 | bwd_inner: 5640.84 | bwd_allreduce: 215.96 | step: 18.81

  3%|▎         | 1266/41250 [3:03:42<96:40:08,  8.70s/it]
                                                         
{'loss': 0.1207, 'grad_norm': 1.4628489017486572, 'learning_rate': 3.999995166796149e-05, 'epoch': 0.31}

  3%|▎         | 1266/41250 [3:03:42<96:40:08,  8.70s/it][2025-04-26 07:12:19,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 07:12:19,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.27 | bwd_microstep: 5715.74 | bwd_inner_microstep: 5702.98 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.71
[2025-04-26 07:12:19,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.27 | bwd: 5715.75 | bwd_inner: 5702.98 | bwd_allreduce: 12.73 | step: 18.71

  3%|▎         | 1267/41250 [3:03:51<96:30:23,  8.69s/it]
                                                         
{'loss': 0.0423, 'grad_norm': 0.4435533583164215, 'learning_rate': 3.999994815402653e-05, 'epoch': 0.31}

  3%|▎         | 1267/41250 [3:03:51<96:30:23,  8.69s/it][2025-04-26 07:12:28,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:12:28,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5708.14 | bwd_inner_microstep: 5685.20 | bwd_allreduce_microstep: 22.89 | step_microstep: 18.56
[2025-04-26 07:12:28,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5708.15 | bwd_inner: 5685.20 | bwd_allreduce: 22.91 | step: 18.57

  3%|▎         | 1268/41250 [3:03:59<96:20:02,  8.67s/it]
                                                         
{'loss': 0.1747, 'grad_norm': 2.060356378555298, 'learning_rate': 3.999994451679583e-05, 'epoch': 0.31}

  3%|▎         | 1268/41250 [3:03:59<96:20:02,  8.67s/it][2025-04-26 07:12:37,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 07:12:37,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.33 | bwd_microstep: 5691.57 | bwd_inner_microstep: 5647.07 | bwd_allreduce_microstep: 44.45 | step_microstep: 18.86
[2025-04-26 07:12:37,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.33 | bwd: 5691.58 | bwd_inner: 5647.07 | bwd_allreduce: 44.47 | step: 18.87

  3%|▎         | 1269/41250 [3:04:08<96:06:59,  8.65s/it]
                                                         
{'loss': 0.4062, 'grad_norm': 2.7555155754089355, 'learning_rate': 3.999994075626941e-05, 'epoch': 0.31}

  3%|▎         | 1269/41250 [3:04:08<96:06:59,  8.65s/it][2025-04-26 07:12:45,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 07:12:45,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5729.82 | bwd_inner_microstep: 5641.94 | bwd_allreduce_microstep: 87.83 | step_microstep: 18.87
[2025-04-26 07:12:45,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5729.83 | bwd_inner: 5641.94 | bwd_allreduce: 87.85 | step: 18.87

  3%|▎         | 1270/41250 [3:04:16<96:03:48,  8.65s/it]
                                                         
{'loss': 0.0694, 'grad_norm': 0.8310753107070923, 'learning_rate': 3.999993687244728e-05, 'epoch': 0.31}

  3%|▎         | 1270/41250 [3:04:16<96:03:48,  8.65s/it][2025-04-26 07:12:54,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:12:54,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.31 | bwd_microstep: 5721.86 | bwd_inner_microstep: 5702.44 | bwd_allreduce_microstep: 19.37 | step_microstep: 18.50
[2025-04-26 07:12:54,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.31 | bwd: 5721.88 | bwd_inner: 5702.44 | bwd_allreduce: 19.39 | step: 18.50

  3%|▎         | 1271/41250 [3:04:25<96:05:23,  8.65s/it]
                                                         
{'loss': 0.2476, 'grad_norm': 2.120335102081299, 'learning_rate': 3.999993286532947e-05, 'epoch': 0.31}

  3%|▎         | 1271/41250 [3:04:25<96:05:23,  8.65s/it][2025-04-26 07:13:03,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:13:03,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.42 | bwd_microstep: 5746.43 | bwd_inner_microstep: 5680.11 | bwd_allreduce_microstep: 66.27 | step_microstep: 18.65
[2025-04-26 07:13:03,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.43 | bwd: 5746.44 | bwd_inner: 5680.11 | bwd_allreduce: 66.29 | step: 18.66

  3%|▎         | 1272/41250 [3:04:34<96:09:17,  8.66s/it]
                                                         
{'loss': 0.137, 'grad_norm': 1.2443945407867432, 'learning_rate': 3.999992873491602e-05, 'epoch': 0.31}

  3%|▎         | 1272/41250 [3:04:34<96:09:17,  8.66s/it][2025-04-26 07:13:11,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 07:13:11,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.74 | bwd_microstep: 5691.74 | bwd_inner_microstep: 5678.93 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.29
[2025-04-26 07:13:11,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.74 | bwd: 5691.76 | bwd_inner: 5678.93 | bwd_allreduce: 12.79 | step: 19.29

  3%|▎         | 1273/41250 [3:04:42<96:02:13,  8.65s/it]
                                                         
{'loss': 0.4049, 'grad_norm': 2.9593207836151123, 'learning_rate': 3.9999924481206934e-05, 'epoch': 0.31}

  3%|▎         | 1273/41250 [3:04:42<96:02:13,  8.65s/it][2025-04-26 07:13:20,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 07:13:20,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.62 | bwd_microstep: 5744.92 | bwd_inner_microstep: 5695.32 | bwd_allreduce_microstep: 49.56 | step_microstep: 18.73
[2025-04-26 07:13:20,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.63 | bwd: 5744.94 | bwd_inner: 5695.32 | bwd_allreduce: 49.58 | step: 18.73

  3%|▎         | 1274/41250 [3:04:51<96:08:15,  8.66s/it]
                                                         
{'loss': 0.4125, 'grad_norm': 2.284959554672241, 'learning_rate': 3.999992010420225e-05, 'epoch': 0.31}

  3%|▎         | 1274/41250 [3:04:51<96:08:15,  8.66s/it][2025-04-26 07:13:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:13:29,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.45 | bwd_microstep: 5680.03 | bwd_inner_microstep: 5652.71 | bwd_allreduce_microstep: 27.28 | step_microstep: 18.20
[2025-04-26 07:13:29,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.45 | bwd: 5680.04 | bwd_inner: 5652.71 | bwd_allreduce: 27.29 | step: 18.20

  3%|▎         | 1275/41250 [3:05:00<95:55:42,  8.64s/it]
                                                         
{'loss': 0.1737, 'grad_norm': 1.1563643217086792, 'learning_rate': 3.999991560390199e-05, 'epoch': 0.31}

  3%|▎         | 1275/41250 [3:05:00<95:55:42,  8.64s/it][2025-04-26 07:13:37,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:13:37,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.34 | bwd_microstep: 5774.46 | bwd_inner_microstep: 5645.52 | bwd_allreduce_microstep: 128.90 | step_microstep: 18.74
[2025-04-26 07:13:37,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.34 | bwd: 5774.48 | bwd_inner: 5645.52 | bwd_allreduce: 128.92 | step: 18.75

  3%|▎         | 1276/41250 [3:05:08<96:04:12,  8.65s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.7270655632019043, 'learning_rate': 3.999991098030619e-05, 'epoch': 0.31}

  3%|▎         | 1276/41250 [3:05:08<96:04:12,  8.65s/it][2025-04-26 07:13:46,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:13:46,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.76 | bwd_microstep: 5751.22 | bwd_inner_microstep: 5649.03 | bwd_allreduce_microstep: 102.15 | step_microstep: 18.15
[2025-04-26 07:13:46,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.76 | bwd: 5751.23 | bwd_inner: 5649.03 | bwd_allreduce: 102.17 | step: 18.15

  3%|▎         | 1277/41250 [3:05:17<96:06:39,  8.66s/it]
                                                         
{'loss': 0.2056, 'grad_norm': 1.5725864171981812, 'learning_rate': 3.999990623341487e-05, 'epoch': 0.31}

  3%|▎         | 1277/41250 [3:05:17<96:06:39,  8.66s/it][2025-04-26 07:13:55,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:13:55,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5721.97 | bwd_inner_microstep: 5709.08 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.50
[2025-04-26 07:13:55,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5721.98 | bwd_inner: 5709.07 | bwd_allreduce: 12.87 | step: 18.51

  3%|▎         | 1278/41250 [3:05:26<96:07:43,  8.66s/it]
                                                         
{'loss': 0.1651, 'grad_norm': 1.4250401258468628, 'learning_rate': 3.9999901363228056e-05, 'epoch': 0.31}

  3%|▎         | 1278/41250 [3:05:26<96:07:43,  8.66s/it][2025-04-26 07:14:03,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:14:03,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5735.33 | bwd_inner_microstep: 5701.09 | bwd_allreduce_microstep: 34.20 | step_microstep: 18.89
[2025-04-26 07:14:03,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5735.35 | bwd_inner: 5701.09 | bwd_allreduce: 34.21 | step: 18.89

  3%|▎         | 1279/41250 [3:05:34<96:10:38,  8.66s/it]
                                                         
{'loss': 0.0561, 'grad_norm': 0.9377299547195435, 'learning_rate': 3.99998963697458e-05, 'epoch': 0.31}

  3%|▎         | 1279/41250 [3:05:34<96:10:38,  8.66s/it][2025-04-26 07:14:12,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:14:12,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.23 | bwd_microstep: 5784.25 | bwd_inner_microstep: 5652.45 | bwd_allreduce_microstep: 131.75 | step_microstep: 18.42
[2025-04-26 07:14:12,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.23 | bwd: 5784.26 | bwd_inner: 5652.45 | bwd_allreduce: 131.77 | step: 18.42

  3%|▎         | 1280/41250 [3:05:43<96:14:16,  8.67s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 1.4198805093765259, 'learning_rate': 3.99998912529681e-05, 'epoch': 0.31}

  3%|▎         | 1280/41250 [3:05:43<96:14:16,  8.67s/it][2025-04-26 07:14:21,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 07:14:21,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.88 | bwd_microstep: 5772.69 | bwd_inner_microstep: 5654.19 | bwd_allreduce_microstep: 118.46 | step_microstep: 18.40
[2025-04-26 07:14:21,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.88 | bwd: 5772.70 | bwd_inner: 5654.19 | bwd_allreduce: 118.48 | step: 18.40

  3%|▎         | 1281/41250 [3:05:52<96:19:09,  8.68s/it]
                                                         
{'loss': 0.2713, 'grad_norm': 2.5314700603485107, 'learning_rate': 3.9999886012895004e-05, 'epoch': 0.31}

  3%|▎         | 1281/41250 [3:05:52<96:19:09,  8.68s/it][2025-04-26 07:14:29,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-26 07:14:29,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.76 | bwd_microstep: 5744.13 | bwd_inner_microstep: 5706.28 | bwd_allreduce_microstep: 37.80 | step_microstep: 19.30
[2025-04-26 07:14:29,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.76 | bwd: 5744.14 | bwd_inner: 5706.28 | bwd_allreduce: 37.82 | step: 19.30

  3%|▎         | 1282/41250 [3:06:00<96:23:10,  8.68s/it]
                                                         
{'loss': 0.1089, 'grad_norm': 1.526573896408081, 'learning_rate': 3.999988064952655e-05, 'epoch': 0.31}

  3%|▎         | 1282/41250 [3:06:00<96:23:10,  8.68s/it][2025-04-26 07:14:38,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 07:14:38,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5766.00 | bwd_inner_microstep: 5667.44 | bwd_allreduce_microstep: 98.52 | step_microstep: 18.90
[2025-04-26 07:14:38,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.24 | bwd: 5766.01 | bwd_inner: 5667.43 | bwd_allreduce: 98.54 | step: 18.90

  3%|▎         | 1283/41250 [3:06:09<96:21:56,  8.68s/it]
                                                         
{'loss': 0.0906, 'grad_norm': 1.0543452501296997, 'learning_rate': 3.9999875162862767e-05, 'epoch': 0.31}

  3%|▎         | 1283/41250 [3:06:09<96:21:56,  8.68s/it][2025-04-26 07:14:47,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:14:47,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.76 | bwd_microstep: 5747.97 | bwd_inner_microstep: 5704.72 | bwd_allreduce_microstep: 43.20 | step_microstep: 18.46
[2025-04-26 07:14:47,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.76 | bwd: 5747.98 | bwd_inner: 5704.72 | bwd_allreduce: 43.22 | step: 18.46

  3%|▎         | 1284/41250 [3:06:18<96:24:06,  8.68s/it]
                                                         
{'loss': 0.0982, 'grad_norm': 1.1486916542053223, 'learning_rate': 3.999986955290369e-05, 'epoch': 0.31}

  3%|▎         | 1284/41250 [3:06:18<96:24:06,  8.68s/it][2025-04-26 07:14:55,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:14:55,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.56 | bwd_microstep: 5727.19 | bwd_inner_microstep: 5714.27 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.63
[2025-04-26 07:14:55,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.57 | bwd: 5727.20 | bwd_inner: 5714.27 | bwd_allreduce: 12.89 | step: 18.64

  3%|▎         | 1285/41250 [3:06:27<96:20:35,  8.68s/it]
                                                         
{'loss': 0.2826, 'grad_norm': 2.553966522216797, 'learning_rate': 3.999986381964934e-05, 'epoch': 0.31}

  3%|▎         | 1285/41250 [3:06:27<96:20:35,  8.68s/it][2025-04-26 07:15:04,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:15:04,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.58 | bwd_microstep: 5773.74 | bwd_inner_microstep: 5695.74 | bwd_allreduce_microstep: 77.96 | step_microstep: 18.55
[2025-04-26 07:15:04,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.59 | bwd: 5773.75 | bwd_inner: 5695.74 | bwd_allreduce: 77.97 | step: 18.55

  3%|▎         | 1286/41250 [3:06:35<96:25:59,  8.69s/it]
                                                         
{'loss': 0.1173, 'grad_norm': 1.5672107934951782, 'learning_rate': 3.999985796309977e-05, 'epoch': 0.31}

  3%|▎         | 1286/41250 [3:06:35<96:25:59,  8.69s/it][2025-04-26 07:15:13,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:15:13,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.54 | bwd_microstep: 5755.05 | bwd_inner_microstep: 5697.91 | bwd_allreduce_microstep: 57.09 | step_microstep: 18.65
[2025-04-26 07:15:13,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5755.06 | bwd_inner: 5697.91 | bwd_allreduce: 57.11 | step: 18.65

  3%|▎         | 1287/41250 [3:06:44<96:26:31,  8.69s/it]
                                                         
{'loss': 0.1828, 'grad_norm': 1.8178985118865967, 'learning_rate': 3.999985198325501e-05, 'epoch': 0.31}

  3%|▎         | 1287/41250 [3:06:44<96:26:31,  8.69s/it][2025-04-26 07:15:21,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:15:21,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.90 | bwd_microstep: 5768.40 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 109.49 | step_microstep: 18.47
[2025-04-26 07:15:21,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.90 | bwd: 5768.41 | bwd_inner: 5658.86 | bwd_allreduce: 109.51 | step: 18.48

  3%|▎         | 1288/41250 [3:06:53<96:26:16,  8.69s/it]
                                                         
{'loss': 0.5564, 'grad_norm': 3.255584955215454, 'learning_rate': 3.99998458801151e-05, 'epoch': 0.31}

  3%|▎         | 1288/41250 [3:06:53<96:26:16,  8.69s/it][2025-04-26 07:15:30,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:15:30,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5790.58 | bwd_inner_microstep: 5700.14 | bwd_allreduce_microstep: 90.39 | step_microstep: 18.45
[2025-04-26 07:15:30,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.07 | bwd: 5790.59 | bwd_inner: 5700.14 | bwd_allreduce: 90.41 | step: 18.45

  3%|▎         | 1289/41250 [3:07:01<96:30:34,  8.69s/it]
                                                         
{'loss': 0.2036, 'grad_norm': 2.985278844833374, 'learning_rate': 3.999983965368007e-05, 'epoch': 0.31}

  3%|▎         | 1289/41250 [3:07:01<96:30:34,  8.69s/it][2025-04-26 07:15:39,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 07:15:39,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.97 | bwd_microstep: 5745.05 | bwd_inner_microstep: 5715.91 | bwd_allreduce_microstep: 29.10 | step_microstep: 17.94
[2025-04-26 07:15:39,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.97 | bwd: 5745.07 | bwd_inner: 5715.91 | bwd_allreduce: 29.12 | step: 17.94

  3%|▎         | 1290/41250 [3:07:10<96:29:21,  8.69s/it]
                                                         
{'loss': 0.1429, 'grad_norm': 1.7360403537750244, 'learning_rate': 3.999983330394996e-05, 'epoch': 0.31}

  3%|▎         | 1290/41250 [3:07:10<96:29:21,  8.69s/it][2025-04-26 07:15:47,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.05 | optimizer_step: 1.19
[2025-04-26 07:15:47,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.60 | bwd_microstep: 5725.26 | bwd_inner_microstep: 5661.43 | bwd_allreduce_microstep: 63.78 | step_microstep: 19.32
[2025-04-26 07:15:47,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 5725.28 | bwd_inner: 5661.43 | bwd_allreduce: 63.81 | step: 19.33

  3%|▎         | 1291/41250 [3:07:19<96:19:41,  8.68s/it]
                                                         
{'loss': 0.1448, 'grad_norm': 2.6045639514923096, 'learning_rate': 3.9999826830924814e-05, 'epoch': 0.31}

  3%|▎         | 1291/41250 [3:07:19<96:19:41,  8.68s/it][2025-04-26 07:15:56,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 07:15:56,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.81 | bwd_microstep: 5707.58 | bwd_inner_microstep: 5694.75 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.89
[2025-04-26 07:15:56,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.81 | bwd: 5707.59 | bwd_inner: 5694.75 | bwd_allreduce: 12.80 | step: 18.89

  3%|▎         | 1292/41250 [3:07:27<96:13:55,  8.67s/it]
                                                         
{'loss': 0.3672, 'grad_norm': 3.132075309753418, 'learning_rate': 3.999982023460467e-05, 'epoch': 0.31}

  3%|▎         | 1292/41250 [3:07:27<96:13:55,  8.67s/it][2025-04-26 07:16:05,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:16:05,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.67 | bwd_microstep: 5722.12 | bwd_inner_microstep: 5709.44 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.31
[2025-04-26 07:16:05,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.67 | bwd: 5722.13 | bwd_inner: 5709.44 | bwd_allreduce: 12.65 | step: 18.32

  3%|▎         | 1293/41250 [3:07:36<96:12:45,  8.67s/it]
                                                         
{'loss': 0.1807, 'grad_norm': 2.497941017150879, 'learning_rate': 3.9999813514989566e-05, 'epoch': 0.31}

  3%|▎         | 1293/41250 [3:07:36<96:12:45,  8.67s/it][2025-04-26 07:16:14,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:16:14,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2905.28 | bwd_microstep: 5797.43 | bwd_inner_microstep: 5784.73 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.58
[2025-04-26 07:16:14,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2905.28 | bwd: 5797.44 | bwd_inner: 5784.73 | bwd_allreduce: 12.68 | step: 18.59

  3%|▎         | 1294/41250 [3:07:45<96:35:15,  8.70s/it]
                                                         
{'loss': 0.1443, 'grad_norm': 1.577728033065796, 'learning_rate': 3.999980667207955e-05, 'epoch': 0.31}

  3%|▎         | 1294/41250 [3:07:45<96:35:15,  8.70s/it][2025-04-26 07:16:22,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:16:22,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5758.27 | bwd_inner_microstep: 5700.37 | bwd_allreduce_microstep: 57.85 | step_microstep: 18.69
[2025-04-26 07:16:22,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5758.28 | bwd_inner: 5700.37 | bwd_allreduce: 57.87 | step: 18.70

  3%|▎         | 1295/41250 [3:07:53<96:33:32,  8.70s/it]
                                                         
{'loss': 0.1373, 'grad_norm': 2.2262256145477295, 'learning_rate': 3.999979970587465e-05, 'epoch': 0.31}

  3%|▎         | 1295/41250 [3:07:53<96:33:32,  8.70s/it][2025-04-26 07:16:31,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 07:16:31,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.89 | bwd_microstep: 5699.42 | bwd_inner_microstep: 5658.51 | bwd_allreduce_microstep: 40.87 | step_microstep: 19.26
[2025-04-26 07:16:31,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.89 | bwd: 5699.43 | bwd_inner: 5658.51 | bwd_allreduce: 40.89 | step: 19.27

  3%|▎         | 1296/41250 [3:08:02<96:19:50,  8.68s/it]
                                                         
{'loss': 0.4321, 'grad_norm': 2.880758762359619, 'learning_rate': 3.9999792616374926e-05, 'epoch': 0.31}

  3%|▎         | 1296/41250 [3:08:02<96:19:50,  8.68s/it][2025-04-26 07:16:40,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:16:40,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5702.64 | bwd_inner_microstep: 5689.93 | bwd_allreduce_microstep: 12.67 | step_microstep: 17.97
[2025-04-26 07:16:40,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5702.66 | bwd_inner: 5689.93 | bwd_allreduce: 12.69 | step: 17.97

  3%|▎         | 1297/41250 [3:08:11<96:11:22,  8.67s/it]
                                                         
{'loss': 0.0816, 'grad_norm': 1.0197594165802002, 'learning_rate': 3.9999785403580417e-05, 'epoch': 0.31}

  3%|▎         | 1297/41250 [3:08:11<96:11:22,  8.67s/it][2025-04-26 07:16:48,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:16:48,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.73 | bwd_microstep: 5764.91 | bwd_inner_microstep: 5704.82 | bwd_allreduce_microstep: 60.04 | step_microstep: 18.48
[2025-04-26 07:16:48,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.73 | bwd: 5764.92 | bwd_inner: 5704.82 | bwd_allreduce: 60.06 | step: 18.49

  3%|▎         | 1298/41250 [3:08:19<96:16:41,  8.68s/it]
                                                         
{'loss': 0.2569, 'grad_norm': 1.8796231746673584, 'learning_rate': 3.9999778067491154e-05, 'epoch': 0.31}

  3%|▎         | 1298/41250 [3:08:19<96:16:41,  8.68s/it][2025-04-26 07:16:57,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:16:57,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.74 | bwd_microstep: 5778.83 | bwd_inner_microstep: 5646.07 | bwd_allreduce_microstep: 132.71 | step_microstep: 18.41
[2025-04-26 07:16:57,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.74 | bwd: 5778.84 | bwd_inner: 5646.07 | bwd_allreduce: 132.73 | step: 18.42

  3%|▎         | 1299/41250 [3:08:28<96:19:17,  8.68s/it]
                                                         
{'loss': 0.1075, 'grad_norm': 0.9880445599555969, 'learning_rate': 3.99997706081072e-05, 'epoch': 0.31}

  3%|▎         | 1299/41250 [3:08:28<96:19:17,  8.68s/it][2025-04-26 07:17:06,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:17:06,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.83 | bwd_microstep: 5703.36 | bwd_inner_microstep: 5690.68 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.90
[2025-04-26 07:17:06,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.83 | bwd: 5703.38 | bwd_inner: 5690.68 | bwd_allreduce: 12.65 | step: 18.90

  3%|▎         | 1300/41250 [3:08:37<96:12:16,  8.67s/it]
                                                         
{'loss': 0.151, 'grad_norm': 1.7430602312088013, 'learning_rate': 3.9999763025428595e-05, 'epoch': 0.32}

  3%|▎         | 1300/41250 [3:08:37<96:12:16,  8.67s/it][2025-04-26 07:17:14,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 07:17:14,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.14 | bwd_microstep: 5769.81 | bwd_inner_microstep: 5643.26 | bwd_allreduce_microstep: 126.50 | step_microstep: 19.30
[2025-04-26 07:17:14,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.14 | bwd: 5769.83 | bwd_inner: 5643.26 | bwd_allreduce: 126.53 | step: 19.30

  3%|▎         | 1301/41250 [3:08:45<96:14:32,  8.67s/it]
                                                         
{'loss': 0.2374, 'grad_norm': 1.867289662361145, 'learning_rate': 3.999975531945538e-05, 'epoch': 0.32}

  3%|▎         | 1301/41250 [3:08:45<96:14:32,  8.67s/it][2025-04-26 07:17:23,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 07:17:23,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.62 | bwd_microstep: 5755.30 | bwd_inner_microstep: 5680.98 | bwd_allreduce_microstep: 74.26 | step_microstep: 18.90
[2025-04-26 07:17:23,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.62 | bwd: 5755.31 | bwd_inner: 5680.98 | bwd_allreduce: 74.28 | step: 18.90

  3%|▎         | 1302/41250 [3:08:54<96:15:32,  8.67s/it]
                                                         
{'loss': 0.2678, 'grad_norm': 2.4079606533050537, 'learning_rate': 3.9999747490187614e-05, 'epoch': 0.32}

  3%|▎         | 1302/41250 [3:08:54<96:15:32,  8.67s/it][2025-04-26 07:17:32,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:17:32,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.84 | bwd_microstep: 5707.11 | bwd_inner_microstep: 5644.24 | bwd_allreduce_microstep: 62.83 | step_microstep: 18.62
[2025-04-26 07:17:32,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.84 | bwd: 5707.12 | bwd_inner: 5644.24 | bwd_allreduce: 62.84 | step: 18.62

  3%|▎         | 1303/41250 [3:09:03<96:03:16,  8.66s/it]
                                                         
{'loss': 0.2319, 'grad_norm': 2.198697566986084, 'learning_rate': 3.999973953762533e-05, 'epoch': 0.32}

  3%|▎         | 1303/41250 [3:09:03<96:03:16,  8.66s/it][2025-04-26 07:17:40,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-26 07:17:40,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.73 | bwd_microstep: 5706.53 | bwd_inner_microstep: 5693.89 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.52
[2025-04-26 07:17:40,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.73 | bwd: 5706.55 | bwd_inner: 5693.89 | bwd_allreduce: 12.61 | step: 18.52

  3%|▎         | 1304/41250 [3:09:11<95:58:05,  8.65s/it]
                                                         
{'loss': 0.3695, 'grad_norm': 4.904691219329834, 'learning_rate': 3.999973146176859e-05, 'epoch': 0.32}

  3%|▎         | 1304/41250 [3:09:11<95:58:05,  8.65s/it][2025-04-26 07:17:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:17:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.42 | bwd_microstep: 5775.02 | bwd_inner_microstep: 5641.57 | bwd_allreduce_microstep: 133.41 | step_microstep: 18.67
[2025-04-26 07:17:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.42 | bwd: 5775.03 | bwd_inner: 5641.57 | bwd_allreduce: 133.43 | step: 18.67

  3%|▎         | 1305/41250 [3:09:20<96:04:24,  8.66s/it]
                                                         
{'loss': 0.337, 'grad_norm': 1.744345784187317, 'learning_rate': 3.999972326261744e-05, 'epoch': 0.32}

  3%|▎         | 1305/41250 [3:09:20<96:04:24,  8.66s/it][2025-04-26 07:17:57,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:17:57,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.96 | bwd_microstep: 5751.62 | bwd_inner_microstep: 5661.82 | bwd_allreduce_microstep: 89.76 | step_microstep: 18.50
[2025-04-26 07:17:57,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.96 | bwd: 5751.64 | bwd_inner: 5661.82 | bwd_allreduce: 89.78 | step: 18.50

  3%|▎         | 1306/41250 [3:09:29<96:03:52,  8.66s/it]
                                                         
{'loss': 0.0397, 'grad_norm': 0.4604303240776062, 'learning_rate': 3.999971494017193e-05, 'epoch': 0.32}

  3%|▎         | 1306/41250 [3:09:29<96:03:52,  8.66s/it][2025-04-26 07:18:06,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 07:18:06,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.60 | bwd_microstep: 5685.24 | bwd_inner_microstep: 5634.90 | bwd_allreduce_microstep: 50.29 | step_microstep: 19.15
[2025-04-26 07:18:06,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.60 | bwd: 5685.25 | bwd_inner: 5634.90 | bwd_allreduce: 50.31 | step: 19.15

  3%|▎         | 1307/41250 [3:09:37<95:47:01,  8.63s/it]
                                                         
{'loss': 0.1647, 'grad_norm': 1.0507441759109497, 'learning_rate': 3.999970649443211e-05, 'epoch': 0.32}

  3%|▎         | 1307/41250 [3:09:37<95:47:01,  8.63s/it][2025-04-26 07:18:15,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 07:18:15,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.86 | bwd_microstep: 5731.38 | bwd_inner_microstep: 5680.78 | bwd_allreduce_microstep: 50.56 | step_microstep: 19.01
[2025-04-26 07:18:15,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.86 | bwd: 5731.40 | bwd_inner: 5680.78 | bwd_allreduce: 50.58 | step: 19.02

  3%|▎         | 1308/41250 [3:09:46<95:51:28,  8.64s/it]
                                                         
{'loss': 0.1563, 'grad_norm': 1.4116178750991821, 'learning_rate': 3.999969792539804e-05, 'epoch': 0.32}

  3%|▎         | 1308/41250 [3:09:46<95:51:28,  8.64s/it][2025-04-26 07:18:23,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:18:23,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.46 | bwd_microstep: 5764.53 | bwd_inner_microstep: 5686.62 | bwd_allreduce_microstep: 77.86 | step_microstep: 19.06
[2025-04-26 07:18:23,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.46 | bwd: 5764.55 | bwd_inner: 5686.62 | bwd_allreduce: 77.88 | step: 19.07

  3%|▎         | 1309/41250 [3:09:55<96:02:37,  8.66s/it]
                                                         
{'loss': 0.1634, 'grad_norm': 1.5517829656600952, 'learning_rate': 3.9999689233069754e-05, 'epoch': 0.32}

  3%|▎         | 1309/41250 [3:09:55<96:02:37,  8.66s/it][2025-04-26 07:18:32,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:18:32,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5712.98 | bwd_inner_microstep: 5700.13 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.49
[2025-04-26 07:18:32,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5713.00 | bwd_inner: 5700.13 | bwd_allreduce: 12.82 | step: 18.49

  3%|▎         | 1310/41250 [3:10:03<96:00:48,  8.65s/it]
                                                         
{'loss': 0.1524, 'grad_norm': 2.0284931659698486, 'learning_rate': 3.999968041744732e-05, 'epoch': 0.32}

  3%|▎         | 1310/41250 [3:10:03<96:00:48,  8.65s/it][2025-04-26 07:18:41,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 07:18:41,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.64 | bwd_microstep: 5874.91 | bwd_inner_microstep: 5640.99 | bwd_allreduce_microstep: 233.87 | step_microstep: 19.11
[2025-04-26 07:18:41,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.64 | bwd: 5874.92 | bwd_inner: 5640.99 | bwd_allreduce: 233.89 | step: 19.11

  3%|▎         | 1311/41250 [3:10:12<96:29:23,  8.70s/it]
                                                         
{'loss': 0.0666, 'grad_norm': 1.0144740343093872, 'learning_rate': 3.99996714785308e-05, 'epoch': 0.32}

  3%|▎         | 1311/41250 [3:10:12<96:29:23,  8.70s/it][2025-04-26 07:18:49,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.13 | optimizer_step: 0.97
[2025-04-26 07:18:49,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.23 | bwd_microstep: 5697.76 | bwd_inner_microstep: 5684.42 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.18
[2025-04-26 07:18:49,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.23 | bwd: 5697.77 | bwd_inner: 5684.42 | bwd_allreduce: 13.32 | step: 19.18

  3%|▎         | 1312/41250 [3:10:21<96:13:45,  8.67s/it]
                                                         
{'loss': 0.2929, 'grad_norm': 2.5304384231567383, 'learning_rate': 3.999966241632023e-05, 'epoch': 0.32}

  3%|▎         | 1312/41250 [3:10:21<96:13:45,  8.67s/it][2025-04-26 07:18:58,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:18:58,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.28 | bwd_microstep: 5751.12 | bwd_inner_microstep: 5653.53 | bwd_allreduce_microstep: 97.54 | step_microstep: 18.61
[2025-04-26 07:18:58,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.28 | bwd: 5751.14 | bwd_inner: 5653.53 | bwd_allreduce: 97.56 | step: 18.62

  3%|▎         | 1313/41250 [3:10:29<96:12:07,  8.67s/it]
                                                         
{'loss': 0.0927, 'grad_norm': 1.4812546968460083, 'learning_rate': 3.999965323081568e-05, 'epoch': 0.32}

  3%|▎         | 1313/41250 [3:10:29<96:12:07,  8.67s/it][2025-04-26 07:19:07,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.02 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:19:07,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5721.42 | bwd_inner_microstep: 5692.19 | bwd_allreduce_microstep: 29.18 | step_microstep: 19.24
[2025-04-26 07:19:07,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5721.43 | bwd_inner: 5692.19 | bwd_allreduce: 29.20 | step: 19.24

  3%|▎         | 1314/41250 [3:10:38<96:07:55,  8.67s/it]
                                                         
{'loss': 0.3807, 'grad_norm': 4.347070217132568, 'learning_rate': 3.9999643922017196e-05, 'epoch': 0.32}

  3%|▎         | 1314/41250 [3:10:38<96:07:55,  8.67s/it][2025-04-26 07:19:15,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.30 | optimizer_step: 0.90
[2025-04-26 07:19:15,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.66 | bwd_microstep: 5726.69 | bwd_inner_microstep: 5696.98 | bwd_allreduce_microstep: 29.65 | step_microstep: 19.12
[2025-04-26 07:19:15,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.66 | bwd: 5726.70 | bwd_inner: 5696.98 | bwd_allreduce: 29.68 | step: 19.12

  3%|▎         | 1315/41250 [3:10:47<96:10:19,  8.67s/it]
                                                         
{'loss': 0.1938, 'grad_norm': 1.5823811292648315, 'learning_rate': 3.999963448992485e-05, 'epoch': 0.32}

  3%|▎         | 1315/41250 [3:10:47<96:10:19,  8.67s/it][2025-04-26 07:19:24,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.81 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 07:19:24,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.61 | bwd_microstep: 5701.73 | bwd_inner_microstep: 5688.19 | bwd_allreduce_microstep: 13.48 | step_microstep: 19.32
[2025-04-26 07:19:24,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.61 | bwd: 5701.74 | bwd_inner: 5688.19 | bwd_allreduce: 13.51 | step: 19.32

  3%|▎         | 1316/41250 [3:10:55<96:00:04,  8.65s/it]
                                                         
{'loss': 0.1384, 'grad_norm': 2.1073360443115234, 'learning_rate': 3.9999624934538676e-05, 'epoch': 0.32}

  3%|▎         | 1316/41250 [3:10:55<96:00:04,  8.65s/it][2025-04-26 07:19:33,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-26 07:19:33,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.12 | bwd_microstep: 5778.32 | bwd_inner_microstep: 5647.97 | bwd_allreduce_microstep: 130.30 | step_microstep: 19.08
[2025-04-26 07:19:33,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.12 | bwd: 5778.33 | bwd_inner: 5647.97 | bwd_allreduce: 130.32 | step: 19.09

  3%|▎         | 1317/41250 [3:11:04<96:05:56,  8.66s/it]
                                                         
{'loss': 0.1938, 'grad_norm': 2.088878870010376, 'learning_rate': 3.999961525585876e-05, 'epoch': 0.32}

  3%|▎         | 1317/41250 [3:11:04<96:05:56,  8.66s/it][2025-04-26 07:19:41,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 07:19:41,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.07 | bwd_microstep: 5742.88 | bwd_inner_microstep: 5706.36 | bwd_allreduce_microstep: 36.46 | step_microstep: 18.68
[2025-04-26 07:19:41,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.07 | bwd: 5742.89 | bwd_inner: 5706.36 | bwd_allreduce: 36.49 | step: 18.68

  3%|▎         | 1318/41250 [3:11:13<96:07:21,  8.67s/it]
                                                         
{'loss': 0.3269, 'grad_norm': 3.410167932510376, 'learning_rate': 3.999960545388514e-05, 'epoch': 0.32}

  3%|▎         | 1318/41250 [3:11:13<96:07:21,  8.67s/it][2025-04-26 07:19:50,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:19:50,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.89 | bwd_microstep: 5692.11 | bwd_inner_microstep: 5679.43 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.56
[2025-04-26 07:19:50,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.89 | bwd: 5692.12 | bwd_inner: 5679.43 | bwd_allreduce: 12.65 | step: 18.56

  3%|▎         | 1319/41250 [3:11:21<96:00:42,  8.66s/it]
                                                         
{'loss': 0.3647, 'grad_norm': 3.3279800415039062, 'learning_rate': 3.99995955286179e-05, 'epoch': 0.32}

  3%|▎         | 1319/41250 [3:11:21<96:00:42,  8.66s/it][2025-04-26 07:19:59,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:19:59,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5736.31 | bwd_inner_microstep: 5703.31 | bwd_allreduce_microstep: 32.96 | step_microstep: 18.68
[2025-04-26 07:19:59,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5736.33 | bwd_inner: 5703.31 | bwd_allreduce: 32.98 | step: 18.68

  3%|▎         | 1320/41250 [3:11:30<96:03:19,  8.66s/it]
                                                         
{'loss': 0.1413, 'grad_norm': 2.226857900619507, 'learning_rate': 3.9999585480057076e-05, 'epoch': 0.32}

  3%|▎         | 1320/41250 [3:11:30<96:03:19,  8.66s/it][2025-04-26 07:20:07,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 07:20:07,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5750.74 | bwd_inner_microstep: 5692.60 | bwd_allreduce_microstep: 58.08 | step_microstep: 18.83
[2025-04-26 07:20:07,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5750.75 | bwd_inner: 5692.60 | bwd_allreduce: 58.11 | step: 18.83

  3%|▎         | 1321/41250 [3:11:39<96:06:23,  8.66s/it]
                                                         
{'loss': 0.1132, 'grad_norm': 1.2248033285140991, 'learning_rate': 3.9999575308202744e-05, 'epoch': 0.32}

  3%|▎         | 1321/41250 [3:11:39<96:06:23,  8.66s/it][2025-04-26 07:20:16,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 07:20:16,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.22 | bwd_microstep: 5695.61 | bwd_inner_microstep: 5662.20 | bwd_allreduce_microstep: 33.36 | step_microstep: 18.71
[2025-04-26 07:20:16,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.22 | bwd: 5695.63 | bwd_inner: 5662.20 | bwd_allreduce: 33.39 | step: 18.72

  3%|▎         | 1322/41250 [3:11:47<95:55:33,  8.65s/it]
                                                         
{'loss': 0.2452, 'grad_norm': 1.9911760091781616, 'learning_rate': 3.9999565013054966e-05, 'epoch': 0.32}

  3%|▎         | 1322/41250 [3:11:47<95:55:33,  8.65s/it][2025-04-26 07:20:25,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:20:25,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5727.43 | bwd_inner_microstep: 5704.46 | bwd_allreduce_microstep: 22.93 | step_microstep: 18.70
[2025-04-26 07:20:25,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5727.45 | bwd_inner: 5704.46 | bwd_allreduce: 22.94 | step: 18.70

  3%|▎         | 1323/41250 [3:11:56<95:58:54,  8.65s/it]
                                                         
{'loss': 0.2095, 'grad_norm': 2.292640447616577, 'learning_rate': 3.9999554594613805e-05, 'epoch': 0.32}

  3%|▎         | 1323/41250 [3:11:56<95:58:54,  8.65s/it][2025-04-26 07:20:33,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:20:33,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.80 | bwd_microstep: 5747.07 | bwd_inner_microstep: 5697.76 | bwd_allreduce_microstep: 49.27 | step_microstep: 18.42
[2025-04-26 07:20:33,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.80 | bwd: 5747.09 | bwd_inner: 5697.76 | bwd_allreduce: 49.29 | step: 18.42

  3%|▎         | 1324/41250 [3:12:05<96:03:52,  8.66s/it]
                                                         
{'loss': 0.0552, 'grad_norm': 0.648105800151825, 'learning_rate': 3.999954405287932e-05, 'epoch': 0.32}

  3%|▎         | 1324/41250 [3:12:05<96:03:52,  8.66s/it][2025-04-26 07:20:42,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:20:42,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.10 | bwd_microstep: 5698.87 | bwd_inner_microstep: 5661.27 | bwd_allreduce_microstep: 37.55 | step_microstep: 18.52
[2025-04-26 07:20:42,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.10 | bwd: 5698.88 | bwd_inner: 5661.27 | bwd_allreduce: 37.57 | step: 18.53

  3%|▎         | 1325/41250 [3:12:13<95:50:38,  8.64s/it]
                                                         
{'loss': 0.2856, 'grad_norm': 1.8517632484436035, 'learning_rate': 3.999953338785158e-05, 'epoch': 0.32}

  3%|▎         | 1325/41250 [3:12:13<95:50:38,  8.64s/it][2025-04-26 07:20:51,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:20:51,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.65 | bwd_microstep: 5713.46 | bwd_inner_microstep: 5656.89 | bwd_allreduce_microstep: 56.52 | step_microstep: 18.61
[2025-04-26 07:20:51,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.65 | bwd: 5713.47 | bwd_inner: 5656.89 | bwd_allreduce: 56.54 | step: 18.61

  3%|▎         | 1326/41250 [3:12:22<95:46:55,  8.64s/it]
                                                         
{'loss': 0.1823, 'grad_norm': 2.578714609146118, 'learning_rate': 3.9999522599530654e-05, 'epoch': 0.32}

  3%|▎         | 1326/41250 [3:12:22<95:46:55,  8.64s/it][2025-04-26 07:20:59,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:20:59,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.58 | bwd_microstep: 5745.37 | bwd_inner_microstep: 5716.27 | bwd_allreduce_microstep: 29.05 | step_microstep: 18.51
[2025-04-26 07:20:59,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.58 | bwd: 5745.39 | bwd_inner: 5716.27 | bwd_allreduce: 29.07 | step: 18.51

  3%|▎         | 1327/41250 [3:12:30<95:56:04,  8.65s/it]
                                                         
{'loss': 0.0985, 'grad_norm': 1.573713779449463, 'learning_rate': 3.9999511687916594e-05, 'epoch': 0.32}

  3%|▎         | 1327/41250 [3:12:30<95:56:04,  8.65s/it][2025-04-26 07:21:08,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 07:21:08,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.77 | bwd_microstep: 5697.64 | bwd_inner_microstep: 5666.01 | bwd_allreduce_microstep: 31.58 | step_microstep: 18.95
[2025-04-26 07:21:08,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.77 | bwd: 5697.66 | bwd_inner: 5666.02 | bwd_allreduce: 31.60 | step: 18.96

  3%|▎         | 1328/41250 [3:12:39<95:47:58,  8.64s/it]
                                                         
{'loss': 0.1815, 'grad_norm': 1.3230191469192505, 'learning_rate': 3.999950065300949e-05, 'epoch': 0.32}

  3%|▎         | 1328/41250 [3:12:39<95:47:58,  8.64s/it][2025-04-26 07:21:17,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:21:17,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.19 | bwd_microstep: 5761.48 | bwd_inner_microstep: 5656.39 | bwd_allreduce_microstep: 105.04 | step_microstep: 18.61
[2025-04-26 07:21:17,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.19 | bwd: 5761.49 | bwd_inner: 5656.39 | bwd_allreduce: 105.06 | step: 18.61

  3%|▎         | 1329/41250 [3:12:48<95:57:05,  8.65s/it]
                                                         
{'loss': 0.2282, 'grad_norm': 1.361566424369812, 'learning_rate': 3.999948949480939e-05, 'epoch': 0.32}

  3%|▎         | 1329/41250 [3:12:48<95:57:05,  8.65s/it][2025-04-26 07:21:25,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 07:21:25,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.54 | bwd_microstep: 5726.12 | bwd_inner_microstep: 5664.54 | bwd_allreduce_microstep: 61.53 | step_microstep: 18.73
[2025-04-26 07:21:25,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.54 | bwd: 5726.13 | bwd_inner: 5664.54 | bwd_allreduce: 61.55 | step: 18.73

  3%|▎         | 1330/41250 [3:12:56<95:52:55,  8.65s/it]
                                                         
{'loss': 0.2449, 'grad_norm': 1.8392904996871948, 'learning_rate': 3.999947821331638e-05, 'epoch': 0.32}

  3%|▎         | 1330/41250 [3:12:56<95:52:55,  8.65s/it][2025-04-26 07:21:34,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:21:34,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.94 | bwd_microstep: 5743.80 | bwd_inner_microstep: 5713.61 | bwd_allreduce_microstep: 30.14 | step_microstep: 18.69
[2025-04-26 07:21:34,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.94 | bwd: 5743.81 | bwd_inner: 5713.61 | bwd_allreduce: 30.16 | step: 18.69

  3%|▎         | 1331/41250 [3:13:05<95:59:48,  8.66s/it]
                                                         
{'loss': 0.0816, 'grad_norm': 1.0962989330291748, 'learning_rate': 3.999946680853051e-05, 'epoch': 0.32}

  3%|▎         | 1331/41250 [3:13:05<95:59:48,  8.66s/it][2025-04-26 07:21:43,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:21:43,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.95 | bwd_microstep: 5714.86 | bwd_inner_microstep: 5702.01 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.55
[2025-04-26 07:21:43,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.95 | bwd: 5714.88 | bwd_inner: 5702.01 | bwd_allreduce: 12.83 | step: 18.55

  3%|▎         | 1332/41250 [3:13:14<95:57:02,  8.65s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 0.9129239320755005, 'learning_rate': 3.999945528045187e-05, 'epoch': 0.32}

  3%|▎         | 1332/41250 [3:13:14<95:57:02,  8.65s/it][2025-04-26 07:21:51,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:21:51,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.61 | bwd_microstep: 5704.14 | bwd_inner_microstep: 5659.63 | bwd_allreduce_microstep: 44.47 | step_microstep: 18.51
[2025-04-26 07:21:51,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.62 | bwd: 5704.16 | bwd_inner: 5659.63 | bwd_allreduce: 44.49 | step: 18.51

  3%|▎         | 1333/41250 [3:13:22<95:51:29,  8.65s/it]
                                                         
{'loss': 0.1779, 'grad_norm': 2.7172577381134033, 'learning_rate': 3.9999443629080514e-05, 'epoch': 0.32}

  3%|▎         | 1333/41250 [3:13:22<95:51:29,  8.65s/it][2025-04-26 07:22:00,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:22:00,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.52 | bwd_microstep: 5730.68 | bwd_inner_microstep: 5717.80 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.78
[2025-04-26 07:22:00,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.52 | bwd: 5730.69 | bwd_inner: 5717.80 | bwd_allreduce: 12.85 | step: 18.79

  3%|▎         | 1334/41250 [3:13:31<95:52:53,  8.65s/it]
                                                         
{'loss': 0.2287, 'grad_norm': 2.5030806064605713, 'learning_rate': 3.9999431854416534e-05, 'epoch': 0.32}

  3%|▎         | 1334/41250 [3:13:31<95:52:53,  8.65s/it][2025-04-26 07:22:09,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:22:09,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.70 | bwd_microstep: 5781.44 | bwd_inner_microstep: 5711.72 | bwd_allreduce_microstep: 69.67 | step_microstep: 18.72
[2025-04-26 07:22:09,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.70 | bwd: 5781.46 | bwd_inner: 5711.72 | bwd_allreduce: 69.69 | step: 18.72

  3%|▎         | 1335/41250 [3:13:40<96:07:34,  8.67s/it]
                                                         
{'loss': 0.1349, 'grad_norm': 1.078061819076538, 'learning_rate': 3.9999419956459976e-05, 'epoch': 0.32}

  3%|▎         | 1335/41250 [3:13:40<96:07:34,  8.67s/it][2025-04-26 07:22:17,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 07:22:17,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.58 | bwd_microstep: 5804.13 | bwd_inner_microstep: 5791.27 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.89
[2025-04-26 07:22:17,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.58 | bwd: 5804.14 | bwd_inner: 5791.27 | bwd_allreduce: 12.82 | step: 18.89

  3%|▎         | 1336/41250 [3:13:49<96:29:09,  8.70s/it]
                                                         
{'loss': 0.176, 'grad_norm': 2.332869291305542, 'learning_rate': 3.9999407935210935e-05, 'epoch': 0.32}

  3%|▎         | 1336/41250 [3:13:49<96:29:09,  8.70s/it][2025-04-26 07:22:26,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.10 | optimizer_step: 1.07
[2025-04-26 07:22:26,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.76 | bwd_microstep: 5710.43 | bwd_inner_microstep: 5697.21 | bwd_allreduce_microstep: 13.17 | step_microstep: 18.98
[2025-04-26 07:22:26,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.76 | bwd: 5710.45 | bwd_inner: 5697.21 | bwd_allreduce: 13.19 | step: 18.99

  3%|▎         | 1337/41250 [3:13:57<96:18:28,  8.69s/it]
                                                         
{'loss': 0.136, 'grad_norm': 1.1940910816192627, 'learning_rate': 3.999939579066948e-05, 'epoch': 0.32}

  3%|▎         | 1337/41250 [3:13:57<96:18:28,  8.69s/it][2025-04-26 07:22:35,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:22:35,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.67 | bwd_microstep: 5728.38 | bwd_inner_microstep: 5715.63 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.66
[2025-04-26 07:22:35,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.67 | bwd: 5728.39 | bwd_inner: 5715.63 | bwd_allreduce: 12.72 | step: 18.66

  3%|▎         | 1338/41250 [3:14:06<96:12:35,  8.68s/it]
                                                         
{'loss': 0.1282, 'grad_norm': 0.8398531675338745, 'learning_rate': 3.9999383522835676e-05, 'epoch': 0.32}

  3%|▎         | 1338/41250 [3:14:06<96:12:35,  8.68s/it][2025-04-26 07:22:43,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:22:43,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.88 | bwd_microstep: 5742.17 | bwd_inner_microstep: 5717.24 | bwd_allreduce_microstep: 24.89 | step_microstep: 18.42
[2025-04-26 07:22:43,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.88 | bwd: 5742.18 | bwd_inner: 5717.24 | bwd_allreduce: 24.91 | step: 18.42

  3%|▎         | 1339/41250 [3:14:14<96:10:22,  8.67s/it]
                                                         
{'loss': 0.0723, 'grad_norm': 0.9228935241699219, 'learning_rate': 3.999937113170961e-05, 'epoch': 0.32}

  3%|▎         | 1339/41250 [3:14:14<96:10:22,  8.67s/it][2025-04-26 07:22:52,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:22:52,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5797.74 | bwd_inner_microstep: 5661.85 | bwd_allreduce_microstep: 135.84 | step_microstep: 18.48
[2025-04-26 07:22:52,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5797.75 | bwd_inner: 5661.85 | bwd_allreduce: 135.85 | step: 18.48

  3%|▎         | 1340/41250 [3:14:23<96:16:59,  8.69s/it]
                                                         
{'loss': 0.2202, 'grad_norm': 2.326631784439087, 'learning_rate': 3.9999358617291354e-05, 'epoch': 0.32}

  3%|▎         | 1340/41250 [3:14:23<96:16:59,  8.69s/it][2025-04-26 07:23:01,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:23:01,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5740.95 | bwd_inner_microstep: 5699.47 | bwd_allreduce_microstep: 41.44 | step_microstep: 18.41
[2025-04-26 07:23:01,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5740.96 | bwd_inner: 5699.47 | bwd_allreduce: 41.45 | step: 18.41

  3%|▎         | 1341/41250 [3:14:32<96:14:02,  8.68s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 2.2441036701202393, 'learning_rate': 3.999934597958099e-05, 'epoch': 0.33}

  3%|▎         | 1341/41250 [3:14:32<96:14:02,  8.68s/it][2025-04-26 07:23:09,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:23:09,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.70 | bwd_microstep: 5726.14 | bwd_inner_microstep: 5713.32 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.44
[2025-04-26 07:23:09,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.70 | bwd: 5726.15 | bwd_inner: 5713.32 | bwd_allreduce: 12.79 | step: 18.44

  3%|▎         | 1342/41250 [3:14:41<96:09:37,  8.67s/it]
                                                         
{'loss': 0.6355, 'grad_norm': 3.6136369705200195, 'learning_rate': 3.9999333218578585e-05, 'epoch': 0.33}

  3%|▎         | 1342/41250 [3:14:41<96:09:37,  8.67s/it][2025-04-26 07:23:18,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:23:18,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.89 | bwd_microstep: 5700.35 | bwd_inner_microstep: 5659.81 | bwd_allreduce_microstep: 40.50 | step_microstep: 18.41
[2025-04-26 07:23:18,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.89 | bwd: 5700.37 | bwd_inner: 5659.81 | bwd_allreduce: 40.52 | step: 18.41

  3%|▎         | 1343/41250 [3:14:49<95:54:56,  8.65s/it]
                                                         
{'loss': 0.4744, 'grad_norm': 3.8031766414642334, 'learning_rate': 3.999932033428423e-05, 'epoch': 0.33}

  3%|▎         | 1343/41250 [3:14:49<95:54:56,  8.65s/it][2025-04-26 07:23:27,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:23:27,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.77 | bwd_microstep: 5686.00 | bwd_inner_microstep: 5658.96 | bwd_allreduce_microstep: 26.99 | step_microstep: 18.44
[2025-04-26 07:23:27,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.77 | bwd: 5686.01 | bwd_inner: 5658.96 | bwd_allreduce: 27.01 | step: 18.44

  3%|▎         | 1344/41250 [3:14:58<95:47:03,  8.64s/it]
                                                         
{'loss': 0.2001, 'grad_norm': 1.2958109378814697, 'learning_rate': 3.9999307326698e-05, 'epoch': 0.33}

  3%|▎         | 1344/41250 [3:14:58<95:47:03,  8.64s/it][2025-04-26 07:23:35,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:23:35,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.57 | bwd_microstep: 5708.94 | bwd_inner_microstep: 5696.25 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.43
[2025-04-26 07:23:35,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.57 | bwd: 5708.95 | bwd_inner: 5696.25 | bwd_allreduce: 12.66 | step: 18.43

  3%|▎         | 1345/41250 [3:15:06<95:47:35,  8.64s/it]
                                                         
{'loss': 0.1157, 'grad_norm': 0.9012272953987122, 'learning_rate': 3.999929419581997e-05, 'epoch': 0.33}

  3%|▎         | 1345/41250 [3:15:06<95:47:35,  8.64s/it][2025-04-26 07:23:44,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:23:44,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.69 | bwd_microstep: 5699.55 | bwd_inner_microstep: 5686.91 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.02
[2025-04-26 07:23:44,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.69 | bwd: 5699.57 | bwd_inner: 5686.91 | bwd_allreduce: 12.62 | step: 18.02

  3%|▎         | 1346/41250 [3:15:15<95:43:59,  8.64s/it]
                                                         
{'loss': 0.0772, 'grad_norm': 1.5907927751541138, 'learning_rate': 3.999928094165023e-05, 'epoch': 0.33}

  3%|▎         | 1346/41250 [3:15:15<95:43:59,  8.64s/it][2025-04-26 07:23:53,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:23:53,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.98 | bwd_microstep: 5770.49 | bwd_inner_microstep: 5757.81 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.51
[2025-04-26 07:23:53,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.98 | bwd: 5770.50 | bwd_inner: 5757.81 | bwd_allreduce: 12.65 | step: 18.51

  3%|▎         | 1347/41250 [3:15:24<96:04:31,  8.67s/it]
                                                         
{'loss': 0.2968, 'grad_norm': 1.7626407146453857, 'learning_rate': 3.9999267564188844e-05, 'epoch': 0.33}

  3%|▎         | 1347/41250 [3:15:24<96:04:31,  8.67s/it][2025-04-26 07:24:01,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.11 | optimizer_step: 1.03
[2025-04-26 07:24:01,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.23 | bwd_microstep: 5742.87 | bwd_inner_microstep: 5716.08 | bwd_allreduce_microstep: 26.73 | step_microstep: 19.95
[2025-04-26 07:24:01,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.23 | bwd: 5742.89 | bwd_inner: 5716.08 | bwd_allreduce: 26.76 | step: 19.95

  3%|▎         | 1348/41250 [3:15:32<96:06:44,  8.67s/it]
                                                         
{'loss': 0.2137, 'grad_norm': 2.5269994735717773, 'learning_rate': 3.999925406343592e-05, 'epoch': 0.33}

  3%|▎         | 1348/41250 [3:15:32<96:06:44,  8.67s/it][2025-04-26 07:24:10,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 07:24:10,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.58 | bwd_microstep: 5755.51 | bwd_inner_microstep: 5638.43 | bwd_allreduce_microstep: 117.03 | step_microstep: 18.75
[2025-04-26 07:24:10,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.58 | bwd: 5755.52 | bwd_inner: 5638.43 | bwd_allreduce: 117.05 | step: 18.76

  3%|▎         | 1349/41250 [3:15:41<96:04:46,  8.67s/it]
                                                         
{'loss': 0.3545, 'grad_norm': 1.3705276250839233, 'learning_rate': 3.999924043939152e-05, 'epoch': 0.33}

  3%|▎         | 1349/41250 [3:15:41<96:04:46,  8.67s/it][2025-04-26 07:24:19,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.34 | optimizer_step: 1.07
[2025-04-26 07:24:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.17 | bwd_microstep: 5717.16 | bwd_inner_microstep: 5662.20 | bwd_allreduce_microstep: 54.89 | step_microstep: 20.15
[2025-04-26 07:24:19,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.17 | bwd: 5717.18 | bwd_inner: 5662.20 | bwd_allreduce: 54.92 | step: 20.15

  3%|▎         | 1350/41250 [3:15:50<95:56:12,  8.66s/it]
                                                         
{'loss': 0.1128, 'grad_norm': 1.1038316488265991, 'learning_rate': 3.999922669205574e-05, 'epoch': 0.33}

  3%|▎         | 1350/41250 [3:15:50<95:56:12,  8.66s/it][2025-04-26 07:24:27,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 0.93
[2025-04-26 07:24:27,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5733.75 | bwd_inner_microstep: 5681.99 | bwd_allreduce_microstep: 51.71 | step_microstep: 18.89
[2025-04-26 07:24:27,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5733.77 | bwd_inner: 5681.99 | bwd_allreduce: 51.73 | step: 18.90

  3%|▎         | 1351/41250 [3:15:58<95:56:49,  8.66s/it]
                                                         
{'loss': 0.3655, 'grad_norm': 2.846261501312256, 'learning_rate': 3.999921282142866e-05, 'epoch': 0.33}

  3%|▎         | 1351/41250 [3:15:58<95:56:49,  8.66s/it][2025-04-26 07:24:36,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:24:36,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.50 | bwd_microstep: 5818.14 | bwd_inner_microstep: 5692.45 | bwd_allreduce_microstep: 125.64 | step_microstep: 18.57
[2025-04-26 07:24:36,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.50 | bwd: 5818.15 | bwd_inner: 5692.45 | bwd_allreduce: 125.66 | step: 18.58

  3%|▎         | 1352/41250 [3:16:07<96:11:57,  8.68s/it]
                                                         
{'loss': 0.2314, 'grad_norm': 1.587893009185791, 'learning_rate': 3.9999198827510365e-05, 'epoch': 0.33}

  3%|▎         | 1352/41250 [3:16:07<96:11:57,  8.68s/it][2025-04-26 07:24:45,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:24:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.26 | bwd_microstep: 5870.08 | bwd_inner_microstep: 5857.24 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.19
[2025-04-26 07:24:45,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.26 | bwd: 5870.09 | bwd_inner: 5857.24 | bwd_allreduce: 12.81 | step: 18.18

  3%|▎         | 1353/41250 [3:16:16<96:52:19,  8.74s/it]
                                                         
{'loss': 0.2014, 'grad_norm': 1.6934700012207031, 'learning_rate': 3.999918471030095e-05, 'epoch': 0.33}

  3%|▎         | 1353/41250 [3:16:16<96:52:19,  8.74s/it][2025-04-26 07:24:53,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:24:53,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.64 | bwd_microstep: 5746.84 | bwd_inner_microstep: 5705.22 | bwd_allreduce_microstep: 41.58 | step_microstep: 18.20
[2025-04-26 07:24:53,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.64 | bwd: 5746.86 | bwd_inner: 5705.22 | bwd_allreduce: 41.59 | step: 18.20

  3%|▎         | 1354/41250 [3:16:25<96:40:31,  8.72s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.2946372032165527, 'learning_rate': 3.999917046980049e-05, 'epoch': 0.33}

  3%|▎         | 1354/41250 [3:16:25<96:40:31,  8.72s/it][2025-04-26 07:25:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 07:25:02,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.43 | bwd_microstep: 5757.05 | bwd_inner_microstep: 5692.07 | bwd_allreduce_microstep: 64.93 | step_microstep: 19.18
[2025-04-26 07:25:02,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.43 | bwd: 5757.07 | bwd_inner: 5692.07 | bwd_allreduce: 64.95 | step: 19.19

  3%|▎         | 1355/41250 [3:16:33<96:32:45,  8.71s/it]
                                                         
{'loss': 0.1444, 'grad_norm': 1.0933741331100464, 'learning_rate': 3.9999156106009075e-05, 'epoch': 0.33}

  3%|▎         | 1355/41250 [3:16:33<96:32:45,  8.71s/it][2025-04-26 07:25:11,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 07:25:11,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.74 | bwd_microstep: 5695.37 | bwd_inner_microstep: 5654.25 | bwd_allreduce_microstep: 41.06 | step_microstep: 19.00
[2025-04-26 07:25:11,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.74 | bwd: 5695.38 | bwd_inner: 5654.25 | bwd_allreduce: 41.08 | step: 19.01

  3%|▎         | 1356/41250 [3:16:42<96:10:30,  8.68s/it]
                                                         
{'loss': 0.3251, 'grad_norm': 3.6855452060699463, 'learning_rate': 3.9999141618926805e-05, 'epoch': 0.33}

  3%|▎         | 1356/41250 [3:16:42<96:10:30,  8.68s/it][2025-04-26 07:25:19,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:25:19,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.13 | bwd_microstep: 5751.05 | bwd_inner_microstep: 5695.09 | bwd_allreduce_microstep: 55.91 | step_microstep: 18.87
[2025-04-26 07:25:19,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.13 | bwd: 5751.06 | bwd_inner: 5695.09 | bwd_allreduce: 55.93 | step: 18.87

  3%|▎         | 1357/41250 [3:16:51<96:09:07,  8.68s/it]
                                                         
{'loss': 0.2533, 'grad_norm': 1.4135422706604004, 'learning_rate': 3.9999127008553754e-05, 'epoch': 0.33}

  3%|▎         | 1357/41250 [3:16:51<96:09:07,  8.68s/it][2025-04-26 07:25:28,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:25:28,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.44 | bwd_microstep: 5745.15 | bwd_inner_microstep: 5696.71 | bwd_allreduce_microstep: 48.40 | step_microstep: 18.66
[2025-04-26 07:25:28,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.44 | bwd: 5745.17 | bwd_inner: 5696.71 | bwd_allreduce: 48.42 | step: 18.66

  3%|▎         | 1358/41250 [3:16:59<96:08:28,  8.68s/it]
                                                         
{'loss': 0.2113, 'grad_norm': 1.632498860359192, 'learning_rate': 3.9999112274890016e-05, 'epoch': 0.33}

  3%|▎         | 1358/41250 [3:16:59<96:08:28,  8.68s/it][2025-04-26 07:25:37,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 07:25:37,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.39 | bwd_microstep: 5730.31 | bwd_inner_microstep: 5703.52 | bwd_allreduce_microstep: 26.74 | step_microstep: 19.10
[2025-04-26 07:25:37,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.39 | bwd: 5730.32 | bwd_inner: 5703.52 | bwd_allreduce: 26.76 | step: 19.10

  3%|▎         | 1359/41250 [3:17:08<96:05:21,  8.67s/it]
                                                         
{'loss': 0.3405, 'grad_norm': 2.005044460296631, 'learning_rate': 3.9999097417935685e-05, 'epoch': 0.33}

  3%|▎         | 1359/41250 [3:17:08<96:05:21,  8.67s/it][2025-04-26 07:25:46,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.10
[2025-04-26 07:25:46,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.55 | bwd_microstep: 5783.64 | bwd_inner_microstep: 5770.71 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.79
[2025-04-26 07:25:46,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.55 | bwd: 5783.65 | bwd_inner: 5770.71 | bwd_allreduce: 12.90 | step: 18.79

  3%|▎         | 1360/41250 [3:17:17<96:21:20,  8.70s/it]
                                                         
{'loss': 0.377, 'grad_norm': 1.9999314546585083, 'learning_rate': 3.999908243769086e-05, 'epoch': 0.33}

  3%|▎         | 1360/41250 [3:17:17<96:21:20,  8.70s/it][2025-04-26 07:25:54,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:25:54,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.80 | bwd_microstep: 5757.45 | bwd_inner_microstep: 5704.66 | bwd_allreduce_microstep: 52.75 | step_microstep: 18.75
[2025-04-26 07:25:54,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.80 | bwd: 5757.46 | bwd_inner: 5704.66 | bwd_allreduce: 52.77 | step: 18.75

  3%|▎         | 1361/41250 [3:17:25<96:18:53,  8.69s/it]
                                                         
{'loss': 0.2134, 'grad_norm': 1.1460731029510498, 'learning_rate': 3.999906733415562e-05, 'epoch': 0.33}

  3%|▎         | 1361/41250 [3:17:25<96:18:53,  8.69s/it][2025-04-26 07:26:03,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:26:03,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5739.27 | bwd_inner_microstep: 5688.03 | bwd_allreduce_microstep: 51.19 | step_microstep: 18.41
[2025-04-26 07:26:03,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5739.29 | bwd_inner: 5688.03 | bwd_allreduce: 51.21 | step: 18.41

  3%|▎         | 1362/41250 [3:17:34<96:14:57,  8.69s/it]
                                                         
{'loss': 0.241, 'grad_norm': 1.9869256019592285, 'learning_rate': 3.999905210733006e-05, 'epoch': 0.33}

  3%|▎         | 1362/41250 [3:17:34<96:14:57,  8.69s/it][2025-04-26 07:26:12,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 07:26:12,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.34 | bwd_microstep: 5754.22 | bwd_inner_microstep: 5639.01 | bwd_allreduce_microstep: 115.16 | step_microstep: 18.91
[2025-04-26 07:26:12,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.34 | bwd: 5754.23 | bwd_inner: 5639.01 | bwd_allreduce: 115.18 | step: 18.91

  3%|▎         | 1363/41250 [3:17:43<96:08:46,  8.68s/it]
                                                         
{'loss': 0.1483, 'grad_norm': 1.238943338394165, 'learning_rate': 3.9999036757214285e-05, 'epoch': 0.33}

  3%|▎         | 1363/41250 [3:17:43<96:08:46,  8.68s/it][2025-04-26 07:26:20,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 07:26:20,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.55 | bwd_microstep: 5706.97 | bwd_inner_microstep: 5694.23 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.71
[2025-04-26 07:26:20,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.55 | bwd: 5706.99 | bwd_inner: 5694.23 | bwd_allreduce: 12.71 | step: 18.71

  3%|▎         | 1364/41250 [3:17:51<95:59:56,  8.66s/it]
                                                         
{'loss': 0.1218, 'grad_norm': 1.1067548990249634, 'learning_rate': 3.9999021283808376e-05, 'epoch': 0.33}

  3%|▎         | 1364/41250 [3:17:51<95:59:56,  8.66s/it][2025-04-26 07:26:29,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:26:29,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5719.99 | bwd_inner_microstep: 5665.06 | bwd_allreduce_microstep: 54.89 | step_microstep: 18.77
[2025-04-26 07:26:29,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5720.01 | bwd_inner: 5665.05 | bwd_allreduce: 54.91 | step: 18.77

  3%|▎         | 1365/41250 [3:18:00<95:54:45,  8.66s/it]
                                                         
{'loss': 0.2688, 'grad_norm': 1.6249967813491821, 'learning_rate': 3.999900568711244e-05, 'epoch': 0.33}

  3%|▎         | 1365/41250 [3:18:00<95:54:45,  8.66s/it][2025-04-26 07:26:38,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:26:38,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.76 | bwd_microstep: 5780.35 | bwd_inner_microstep: 5650.27 | bwd_allreduce_microstep: 130.03 | step_microstep: 18.71
[2025-04-26 07:26:38,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.76 | bwd: 5780.37 | bwd_inner: 5650.27 | bwd_allreduce: 130.05 | step: 18.72

  3%|▎         | 1366/41250 [3:18:09<95:59:51,  8.66s/it]
                                                         
{'loss': 0.1387, 'grad_norm': 1.1579458713531494, 'learning_rate': 3.9998989967126564e-05, 'epoch': 0.33}

  3%|▎         | 1366/41250 [3:18:09<95:59:51,  8.66s/it][2025-04-26 07:26:46,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 07:26:46,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5771.73 | bwd_inner_microstep: 5659.23 | bwd_allreduce_microstep: 112.45 | step_microstep: 19.39
[2025-04-26 07:26:46,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5771.74 | bwd_inner: 5659.23 | bwd_allreduce: 112.47 | step: 19.39

  3%|▎         | 1367/41250 [3:18:17<96:03:16,  8.67s/it]
                                                         
{'loss': 0.1433, 'grad_norm': 1.700866937637329, 'learning_rate': 3.9998974123850843e-05, 'epoch': 0.33}

  3%|▎         | 1367/41250 [3:18:17<96:03:16,  8.67s/it][2025-04-26 07:26:55,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:26:55,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.24 | bwd_microstep: 5717.09 | bwd_inner_microstep: 5653.60 | bwd_allreduce_microstep: 63.45 | step_microstep: 18.81
[2025-04-26 07:26:55,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.24 | bwd: 5717.10 | bwd_inner: 5653.60 | bwd_allreduce: 63.46 | step: 18.81

  3%|▎         | 1368/41250 [3:18:26<95:53:54,  8.66s/it]
                                                         
{'loss': 0.2343, 'grad_norm': 2.364234209060669, 'learning_rate': 3.9998958157285385e-05, 'epoch': 0.33}

  3%|▎         | 1368/41250 [3:18:26<95:53:54,  8.66s/it][2025-04-26 07:27:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:27:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.66 | bwd_microstep: 5672.35 | bwd_inner_microstep: 5659.48 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.87
[2025-04-26 07:27:03,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.66 | bwd: 5672.37 | bwd_inner: 5659.48 | bwd_allreduce: 12.84 | step: 18.87

  3%|▎         | 1369/41250 [3:18:35<95:51:48,  8.65s/it]
                                                         
{'loss': 0.2183, 'grad_norm': 1.4635192155838013, 'learning_rate': 3.999894206743029e-05, 'epoch': 0.33}

  3%|▎         | 1369/41250 [3:18:35<95:51:48,  8.65s/it][2025-04-26 07:27:12,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:27:12,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.74 | bwd_microstep: 5710.55 | bwd_inner_microstep: 5667.31 | bwd_allreduce_microstep: 43.20 | step_microstep: 18.46
[2025-04-26 07:27:12,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.74 | bwd: 5710.56 | bwd_inner: 5667.31 | bwd_allreduce: 43.21 | step: 18.47

  3%|▎         | 1370/41250 [3:18:43<95:43:24,  8.64s/it]
                                                         
{'loss': 0.2416, 'grad_norm': 2.118922710418701, 'learning_rate': 3.999892585428564e-05, 'epoch': 0.33}

  3%|▎         | 1370/41250 [3:18:43<95:43:24,  8.64s/it][2025-04-26 07:27:21,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.28 | optimizer_step: 1.06
[2025-04-26 07:27:21,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.92 | bwd_microstep: 5755.89 | bwd_inner_microstep: 5718.77 | bwd_allreduce_microstep: 37.07 | step_microstep: 20.01
[2025-04-26 07:27:21,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.92 | bwd: 5755.91 | bwd_inner: 5718.77 | bwd_allreduce: 37.09 | step: 20.01

  3%|▎         | 1371/41250 [3:18:52<95:54:27,  8.66s/it]
                                                         
{'loss': 0.2135, 'grad_norm': 0.7851133346557617, 'learning_rate': 3.9998909517851555e-05, 'epoch': 0.33}

  3%|▎         | 1371/41250 [3:18:52<95:54:27,  8.66s/it][2025-04-26 07:27:29,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 07:27:29,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.39 | bwd_microstep: 5774.98 | bwd_inner_microstep: 5659.67 | bwd_allreduce_microstep: 115.27 | step_microstep: 19.23
[2025-04-26 07:27:29,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.39 | bwd: 5775.00 | bwd_inner: 5659.67 | bwd_allreduce: 115.29 | step: 19.23

  3%|▎         | 1372/41250 [3:19:01<95:59:57,  8.67s/it]
                                                         
{'loss': 0.284, 'grad_norm': 3.0831241607666016, 'learning_rate': 3.999889305812812e-05, 'epoch': 0.33}

  3%|▎         | 1372/41250 [3:19:01<95:59:57,  8.67s/it][2025-04-26 07:27:38,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 07:27:38,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.42 | bwd_microstep: 5773.57 | bwd_inner_microstep: 5665.79 | bwd_allreduce_microstep: 107.74 | step_microstep: 18.92
[2025-04-26 07:27:38,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.42 | bwd: 5773.59 | bwd_inner: 5665.79 | bwd_allreduce: 107.75 | step: 18.92

  3%|▎         | 1373/41250 [3:19:09<96:05:02,  8.67s/it]
                                                         
{'loss': 0.1487, 'grad_norm': 1.3036401271820068, 'learning_rate': 3.9998876475115446e-05, 'epoch': 0.33}

  3%|▎         | 1373/41250 [3:19:09<96:05:02,  8.67s/it][2025-04-26 07:27:47,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:27:47,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.68 | bwd_microstep: 6027.85 | bwd_inner_microstep: 5705.83 | bwd_allreduce_microstep: 321.98 | step_microstep: 18.56
[2025-04-26 07:27:47,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.68 | bwd: 6027.87 | bwd_inner: 5705.83 | bwd_allreduce: 322.00 | step: 18.57

  3%|▎         | 1374/41250 [3:19:18<97:03:06,  8.76s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.1926608085632324, 'learning_rate': 3.9998859768813624e-05, 'epoch': 0.33}

  3%|▎         | 1374/41250 [3:19:18<97:03:06,  8.76s/it][2025-04-26 07:27:56,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:27:56,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.87 | bwd_microstep: 5744.10 | bwd_inner_microstep: 5718.80 | bwd_allreduce_microstep: 25.25 | step_microstep: 18.60
[2025-04-26 07:27:56,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.87 | bwd: 5744.11 | bwd_inner: 5718.80 | bwd_allreduce: 25.27 | step: 18.60

  3%|▎         | 1375/41250 [3:19:27<96:45:12,  8.74s/it]
                                                         
{'loss': 0.2513, 'grad_norm': 1.1198152303695679, 'learning_rate': 3.9998842939222775e-05, 'epoch': 0.33}

  3%|▎         | 1375/41250 [3:19:27<96:45:12,  8.74s/it][2025-04-26 07:28:05,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 07:28:05,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.29 | bwd_microstep: 6018.10 | bwd_inner_microstep: 5693.30 | bwd_allreduce_microstep: 324.75 | step_microstep: 19.64
[2025-04-26 07:28:05,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.29 | bwd: 6018.12 | bwd_inner: 5693.30 | bwd_allreduce: 324.77 | step: 19.64

  3%|▎         | 1376/41250 [3:19:36<97:28:17,  8.80s/it]
                                                         
{'loss': 0.457, 'grad_norm': 2.8842949867248535, 'learning_rate': 3.9998825986342995e-05, 'epoch': 0.33}

  3%|▎         | 1376/41250 [3:19:36<97:28:17,  8.80s/it][2025-04-26 07:28:13,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:28:13,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.77 | bwd_microstep: 5707.79 | bwd_inner_microstep: 5653.97 | bwd_allreduce_microstep: 53.78 | step_microstep: 18.67
[2025-04-26 07:28:13,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.77 | bwd: 5707.81 | bwd_inner: 5653.97 | bwd_allreduce: 53.80 | step: 18.68

  3%|▎         | 1377/41250 [3:19:45<96:52:35,  8.75s/it]
                                                         
{'loss': 0.2519, 'grad_norm': 2.0294578075408936, 'learning_rate': 3.999880891017438e-05, 'epoch': 0.33}

  3%|▎         | 1377/41250 [3:19:45<96:52:35,  8.75s/it][2025-04-26 07:28:22,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:28:22,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5768.08 | bwd_inner_microstep: 5680.85 | bwd_allreduce_microstep: 87.19 | step_microstep: 18.68
[2025-04-26 07:28:22,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5768.10 | bwd_inner: 5680.85 | bwd_allreduce: 87.21 | step: 18.68

  3%|▎         | 1378/41250 [3:19:53<96:42:36,  8.73s/it]
                                                         
{'loss': 0.1831, 'grad_norm': 1.416046142578125, 'learning_rate': 3.9998791710717044e-05, 'epoch': 0.33}

  3%|▎         | 1378/41250 [3:19:53<96:42:36,  8.73s/it][2025-04-26 07:28:31,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:28:31,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.90 | bwd_microstep: 5789.87 | bwd_inner_microstep: 5777.01 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.58
[2025-04-26 07:28:31,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.90 | bwd: 5789.88 | bwd_inner: 5777.01 | bwd_allreduce: 12.83 | step: 18.58

  3%|▎         | 1379/41250 [3:20:02<96:45:50,  8.74s/it]
                                                         
{'loss': 0.2317, 'grad_norm': 1.6707583665847778, 'learning_rate': 3.999877438797109e-05, 'epoch': 0.33}

  3%|▎         | 1379/41250 [3:20:02<96:45:50,  8.74s/it][2025-04-26 07:28:39,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 07:28:39,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.64 | bwd_microstep: 5735.05 | bwd_inner_microstep: 5702.85 | bwd_allreduce_microstep: 32.15 | step_microstep: 19.02
[2025-04-26 07:28:39,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.64 | bwd: 5735.06 | bwd_inner: 5702.85 | bwd_allreduce: 32.17 | step: 19.02

  3%|▎         | 1380/41250 [3:20:11<96:33:08,  8.72s/it]
                                                         
{'loss': 0.3546, 'grad_norm': 2.7887589931488037, 'learning_rate': 3.999875694193662e-05, 'epoch': 0.33}

  3%|▎         | 1380/41250 [3:20:11<96:33:08,  8.72s/it][2025-04-26 07:28:48,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:28:48,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.01 | bwd_microstep: 5715.89 | bwd_inner_microstep: 5703.13 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.71
[2025-04-26 07:28:48,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.01 | bwd: 5715.91 | bwd_inner: 5703.13 | bwd_allreduce: 12.73 | step: 18.72

  3%|▎         | 1381/41250 [3:20:19<96:20:39,  8.70s/it]
                                                         
{'loss': 0.2803, 'grad_norm': 2.205626964569092, 'learning_rate': 3.999873937261376e-05, 'epoch': 0.33}

  3%|▎         | 1381/41250 [3:20:19<96:20:39,  8.70s/it][2025-04-26 07:28:57,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:28:57,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.44 | bwd_microstep: 5714.10 | bwd_inner_microstep: 5698.35 | bwd_allreduce_microstep: 15.71 | step_microstep: 18.57
[2025-04-26 07:28:57,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.44 | bwd: 5714.11 | bwd_inner: 5698.35 | bwd_allreduce: 15.72 | step: 18.58

  3%|▎         | 1382/41250 [3:20:28<96:09:22,  8.68s/it]
                                                         
{'loss': 0.1138, 'grad_norm': 1.7761247158050537, 'learning_rate': 3.9998721680002595e-05, 'epoch': 0.34}

  3%|▎         | 1382/41250 [3:20:28<96:09:22,  8.68s/it][2025-04-26 07:29:05,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:29:05,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.22 | bwd_microstep: 5764.45 | bwd_inner_microstep: 5696.98 | bwd_allreduce_microstep: 67.42 | step_microstep: 18.36
[2025-04-26 07:29:05,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.22 | bwd: 5764.46 | bwd_inner: 5696.98 | bwd_allreduce: 67.44 | step: 18.36

  3%|▎         | 1383/41250 [3:20:37<96:11:03,  8.69s/it]
                                                         
{'loss': 0.1287, 'grad_norm': 0.984600841999054, 'learning_rate': 3.999870386410324e-05, 'epoch': 0.34}

  3%|▎         | 1383/41250 [3:20:37<96:11:03,  8.69s/it][2025-04-26 07:29:14,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 07:29:14,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5735.78 | bwd_inner_microstep: 5722.97 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.05
[2025-04-26 07:29:14,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5735.79 | bwd_inner: 5722.97 | bwd_allreduce: 12.78 | step: 19.05

  3%|▎         | 1384/41250 [3:20:45<96:07:28,  8.68s/it]
                                                         
{'loss': 0.2115, 'grad_norm': 1.22170090675354, 'learning_rate': 3.999868592491582e-05, 'epoch': 0.34}

  3%|▎         | 1384/41250 [3:20:45<96:07:28,  8.68s/it][2025-04-26 07:29:23,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 07:29:23,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5714.86 | bwd_inner_microstep: 5650.82 | bwd_allreduce_microstep: 63.99 | step_microstep: 18.68
[2025-04-26 07:29:23,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.60 | bwd: 5714.88 | bwd_inner: 5650.82 | bwd_allreduce: 64.01 | step: 18.69

  3%|▎         | 1385/41250 [3:20:54<95:57:54,  8.67s/it]
                                                         
{'loss': 0.3845, 'grad_norm': 2.4887616634368896, 'learning_rate': 3.9998667862440426e-05, 'epoch': 0.34}

  3%|▎         | 1385/41250 [3:20:54<95:57:54,  8.67s/it][2025-04-26 07:29:31,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:29:31,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.63 | bwd_microstep: 5783.73 | bwd_inner_microstep: 5648.03 | bwd_allreduce_microstep: 135.66 | step_microstep: 18.44
[2025-04-26 07:29:31,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.63 | bwd: 5783.75 | bwd_inner: 5648.03 | bwd_allreduce: 135.67 | step: 18.44

  3%|▎         | 1386/41250 [3:21:03<96:03:25,  8.67s/it]
                                                         
{'loss': 0.1722, 'grad_norm': 2.3547284603118896, 'learning_rate': 3.999864967667718e-05, 'epoch': 0.34}

  3%|▎         | 1386/41250 [3:21:03<96:03:25,  8.67s/it][2025-04-26 07:29:40,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:29:40,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.11 | bwd_microstep: 5763.03 | bwd_inner_microstep: 5685.77 | bwd_allreduce_microstep: 77.22 | step_microstep: 17.90
[2025-04-26 07:29:40,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.11 | bwd: 5763.04 | bwd_inner: 5685.77 | bwd_allreduce: 77.24 | step: 17.91

  3%|▎         | 1387/41250 [3:21:11<96:07:14,  8.68s/it]
                                                         
{'loss': 0.3078, 'grad_norm': 2.402164936065674, 'learning_rate': 3.9998631367626196e-05, 'epoch': 0.34}

  3%|▎         | 1387/41250 [3:21:11<96:07:14,  8.68s/it][2025-04-26 07:29:49,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:29:49,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.84 | bwd_microstep: 5753.87 | bwd_inner_microstep: 5660.26 | bwd_allreduce_microstep: 93.56 | step_microstep: 18.39
[2025-04-26 07:29:49,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.84 | bwd: 5753.88 | bwd_inner: 5660.26 | bwd_allreduce: 93.58 | step: 18.40

  3%|▎         | 1388/41250 [3:21:20<96:04:20,  8.68s/it]
                                                         
{'loss': 0.0513, 'grad_norm': 0.485429972410202, 'learning_rate': 3.9998612935287587e-05, 'epoch': 0.34}

  3%|▎         | 1388/41250 [3:21:20<96:04:20,  8.68s/it][2025-04-26 07:29:58,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:29:58,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.41 | bwd_microstep: 5773.60 | bwd_inner_microstep: 5760.18 | bwd_allreduce_microstep: 13.36 | step_microstep: 18.91
[2025-04-26 07:29:58,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.41 | bwd: 5773.61 | bwd_inner: 5760.18 | bwd_allreduce: 13.39 | step: 18.91

  3%|▎         | 1389/41250 [3:21:29<96:16:32,  8.70s/it]
                                                         
{'loss': 0.2041, 'grad_norm': 2.2063634395599365, 'learning_rate': 3.9998594379661455e-05, 'epoch': 0.34}

  3%|▎         | 1389/41250 [3:21:29<96:16:32,  8.70s/it][2025-04-26 07:30:06,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 07:30:06,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5743.57 | bwd_inner_microstep: 5703.39 | bwd_allreduce_microstep: 40.14 | step_microstep: 18.78
[2025-04-26 07:30:06,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5743.58 | bwd_inner: 5703.39 | bwd_allreduce: 40.16 | step: 18.79

  3%|▎         | 1390/41250 [3:21:37<96:13:56,  8.69s/it]
                                                         
{'loss': 0.322, 'grad_norm': 2.188572406768799, 'learning_rate': 3.999857570074792e-05, 'epoch': 0.34}

  3%|▎         | 1390/41250 [3:21:37<96:13:56,  8.69s/it][2025-04-26 07:30:15,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:30:15,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2898.95 | bwd_microstep: 5792.37 | bwd_inner_microstep: 5779.38 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.65
[2025-04-26 07:30:15,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2898.95 | bwd: 5792.38 | bwd_inner: 5779.38 | bwd_allreduce: 12.96 | step: 18.65

  3%|▎         | 1391/41250 [3:21:46<96:30:11,  8.72s/it]
                                                         
{'loss': 0.1833, 'grad_norm': 2.000635862350464, 'learning_rate': 3.999855689854711e-05, 'epoch': 0.34}

  3%|▎         | 1391/41250 [3:21:46<96:30:11,  8.72s/it][2025-04-26 07:30:24,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:30:24,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.37 | bwd_microstep: 5732.66 | bwd_inner_microstep: 5688.32 | bwd_allreduce_microstep: 44.29 | step_microstep: 18.69
[2025-04-26 07:30:24,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.37 | bwd: 5732.67 | bwd_inner: 5688.32 | bwd_allreduce: 44.31 | step: 18.70

  3%|▎         | 1392/41250 [3:21:55<96:22:10,  8.70s/it]
                                                         
{'loss': 0.2222, 'grad_norm': 1.5066583156585693, 'learning_rate': 3.9998537973059125e-05, 'epoch': 0.34}

  3%|▎         | 1392/41250 [3:21:55<96:22:10,  8.70s/it][2025-04-26 07:30:32,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 07:30:32,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.52 | bwd_microstep: 5693.47 | bwd_inner_microstep: 5680.62 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.84
[2025-04-26 07:30:32,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.52 | bwd: 5693.48 | bwd_inner: 5680.62 | bwd_allreduce: 12.82 | step: 18.84

  3%|▎         | 1393/41250 [3:22:04<96:04:22,  8.68s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 0.8865506649017334, 'learning_rate': 3.999851892428408e-05, 'epoch': 0.34}

  3%|▎         | 1393/41250 [3:22:04<96:04:22,  8.68s/it][2025-04-26 07:30:41,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 07:30:41,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5756.42 | bwd_inner_microstep: 5646.78 | bwd_allreduce_microstep: 109.60 | step_microstep: 18.62
[2025-04-26 07:30:41,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5756.44 | bwd_inner: 5646.78 | bwd_allreduce: 109.62 | step: 18.62

  3%|▎         | 1394/41250 [3:22:12<96:03:38,  8.68s/it]
                                                         
{'loss': 0.2345, 'grad_norm': 2.0084738731384277, 'learning_rate': 3.9998499752222114e-05, 'epoch': 0.34}

  3%|▎         | 1394/41250 [3:22:12<96:03:38,  8.68s/it][2025-04-26 07:30:50,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:30:50,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.92 | bwd_microstep: 5722.09 | bwd_inner_microstep: 5695.20 | bwd_allreduce_microstep: 26.84 | step_microstep: 18.93
[2025-04-26 07:30:50,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.92 | bwd: 5722.10 | bwd_inner: 5695.20 | bwd_allreduce: 26.86 | step: 18.94

  3%|▎         | 1395/41250 [3:22:21<96:00:21,  8.67s/it]
                                                         
{'loss': 0.1649, 'grad_norm': 2.510741710662842, 'learning_rate': 3.999848045687332e-05, 'epoch': 0.34}

  3%|▎         | 1395/41250 [3:22:21<96:00:21,  8.67s/it][2025-04-26 07:30:58,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:30:58,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.95 | bwd_microstep: 5750.21 | bwd_inner_microstep: 5693.63 | bwd_allreduce_microstep: 56.54 | step_microstep: 18.47
[2025-04-26 07:30:58,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.95 | bwd: 5750.23 | bwd_inner: 5693.63 | bwd_allreduce: 56.56 | step: 18.48

  3%|▎         | 1396/41250 [3:22:30<96:01:21,  8.67s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 2.3400189876556396, 'learning_rate': 3.999846103823783e-05, 'epoch': 0.34}

  3%|▎         | 1396/41250 [3:22:30<96:01:21,  8.67s/it][2025-04-26 07:31:07,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:31:07,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.05 | bwd_microstep: 5704.09 | bwd_inner_microstep: 5651.40 | bwd_allreduce_microstep: 52.64 | step_microstep: 18.78
[2025-04-26 07:31:07,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.05 | bwd: 5704.10 | bwd_inner: 5651.40 | bwd_allreduce: 52.66 | step: 18.79

  3%|▎         | 1397/41250 [3:22:38<95:47:49,  8.65s/it]
                                                         
{'loss': 0.3854, 'grad_norm': 3.158010244369507, 'learning_rate': 3.999844149631576e-05, 'epoch': 0.34}

  3%|▎         | 1397/41250 [3:22:38<95:47:49,  8.65s/it][2025-04-26 07:31:16,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 07:31:16,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.88 | bwd_microstep: 5700.27 | bwd_inner_microstep: 5646.00 | bwd_allreduce_microstep: 54.22 | step_microstep: 19.17
[2025-04-26 07:31:16,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.88 | bwd: 5700.28 | bwd_inner: 5646.00 | bwd_allreduce: 54.24 | step: 19.17

  3%|▎         | 1398/41250 [3:22:47<95:38:48,  8.64s/it]
                                                         
{'loss': 0.2041, 'grad_norm': 1.3640397787094116, 'learning_rate': 3.9998421831107234e-05, 'epoch': 0.34}

  3%|▎         | 1398/41250 [3:22:47<95:38:48,  8.64s/it][2025-04-26 07:31:24,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:31:24,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.94 | bwd_microstep: 5719.35 | bwd_inner_microstep: 5678.75 | bwd_allreduce_microstep: 40.56 | step_microstep: 18.62
[2025-04-26 07:31:24,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.94 | bwd: 5719.37 | bwd_inner: 5678.75 | bwd_allreduce: 40.58 | step: 18.63

  3%|▎         | 1399/41250 [3:22:55<95:41:55,  8.65s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.5942645072937012, 'learning_rate': 3.999840204261237e-05, 'epoch': 0.34}

  3%|▎         | 1399/41250 [3:22:55<95:41:55,  8.65s/it][2025-04-26 07:31:33,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:31:33,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.81 | bwd_microstep: 5758.64 | bwd_inner_microstep: 5746.03 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.85
[2025-04-26 07:31:33,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.81 | bwd: 5758.65 | bwd_inner: 5746.03 | bwd_allreduce: 12.58 | step: 18.86

  3%|▎         | 1400/41250 [3:23:04<95:58:27,  8.67s/it]
                                                         
{'loss': 0.2846, 'grad_norm': 1.7705985307693481, 'learning_rate': 3.999838213083129e-05, 'epoch': 0.34}

  3%|▎         | 1400/41250 [3:23:04<95:58:27,  8.67s/it][2025-04-26 07:31:42,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:31:42,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.34 | bwd_microstep: 5703.03 | bwd_inner_microstep: 5690.11 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.54
[2025-04-26 07:31:42,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.34 | bwd: 5703.04 | bwd_inner: 5690.11 | bwd_allreduce: 12.89 | step: 18.54

  3%|▎         | 1401/41250 [3:23:13<95:53:26,  8.66s/it]
                                                         
{'loss': 0.104, 'grad_norm': 0.9812477231025696, 'learning_rate': 3.999836209576412e-05, 'epoch': 0.34}

  3%|▎         | 1401/41250 [3:23:13<95:53:26,  8.66s/it][2025-04-26 07:31:50,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:31:50,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.85 | bwd_microstep: 5716.40 | bwd_inner_microstep: 5702.55 | bwd_allreduce_microstep: 13.81 | step_microstep: 18.61
[2025-04-26 07:31:50,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.85 | bwd: 5716.41 | bwd_inner: 5702.55 | bwd_allreduce: 13.83 | step: 18.61

  3%|▎         | 1402/41250 [3:23:21<95:50:20,  8.66s/it]
                                                         
{'loss': 0.2154, 'grad_norm': 2.448613405227661, 'learning_rate': 3.9998341937410975e-05, 'epoch': 0.34}

  3%|▎         | 1402/41250 [3:23:21<95:50:20,  8.66s/it][2025-04-26 07:31:59,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:31:59,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.09 | bwd_microstep: 5730.52 | bwd_inner_microstep: 5693.72 | bwd_allreduce_microstep: 36.75 | step_microstep: 18.68
[2025-04-26 07:31:59,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.09 | bwd: 5730.53 | bwd_inner: 5693.72 | bwd_allreduce: 36.77 | step: 18.69

  3%|▎         | 1403/41250 [3:23:30<95:51:38,  8.66s/it]
                                                         
{'loss': 0.4354, 'grad_norm': 3.0717146396636963, 'learning_rate': 3.999832165577199e-05, 'epoch': 0.34}

  3%|▎         | 1403/41250 [3:23:30<95:51:38,  8.66s/it][2025-04-26 07:32:08,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 07:32:08,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.95 | bwd_microstep: 5748.24 | bwd_inner_microstep: 5647.89 | bwd_allreduce_microstep: 100.31 | step_microstep: 19.34
[2025-04-26 07:32:08,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.95 | bwd: 5748.25 | bwd_inner: 5647.89 | bwd_allreduce: 100.33 | step: 19.34

  3%|▎         | 1404/41250 [3:23:39<95:50:10,  8.66s/it]
                                                         
{'loss': 0.1021, 'grad_norm': 1.427436113357544, 'learning_rate': 3.999830125084729e-05, 'epoch': 0.34}

  3%|▎         | 1404/41250 [3:23:39<95:50:10,  8.66s/it][2025-04-26 07:32:16,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:32:16,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.99 | bwd_microstep: 5715.52 | bwd_inner_microstep: 5685.42 | bwd_allreduce_microstep: 30.06 | step_microstep: 18.38
[2025-04-26 07:32:16,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.99 | bwd: 5715.53 | bwd_inner: 5685.42 | bwd_allreduce: 30.08 | step: 18.39

  3%|▎         | 1405/41250 [3:23:47<95:48:17,  8.66s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.3883006572723389, 'learning_rate': 3.999828072263699e-05, 'epoch': 0.34}

  3%|▎         | 1405/41250 [3:23:47<95:48:17,  8.66s/it][2025-04-26 07:32:25,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-26 07:32:25,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.20 | bwd_microstep: 5685.03 | bwd_inner_microstep: 5672.38 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.88
[2025-04-26 07:32:25,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.20 | bwd: 5685.04 | bwd_inner: 5672.38 | bwd_allreduce: 12.62 | step: 18.88

  3%|▎         | 1406/41250 [3:23:56<95:39:04,  8.64s/it]
                                                         
{'loss': 0.2143, 'grad_norm': 5.053143501281738, 'learning_rate': 3.999826007114122e-05, 'epoch': 0.34}

  3%|▎         | 1406/41250 [3:23:56<95:39:04,  8.64s/it][2025-04-26 07:32:33,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 07:32:33,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.04 | bwd_microstep: 5734.41 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 76.52 | step_microstep: 19.19
[2025-04-26 07:32:33,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.04 | bwd: 5734.43 | bwd_inner: 5657.85 | bwd_allreduce: 76.54 | step: 19.19

  3%|▎         | 1407/41250 [3:24:05<95:42:29,  8.65s/it]
                                                         
{'loss': 0.2815, 'grad_norm': 2.2781896591186523, 'learning_rate': 3.999823929636011e-05, 'epoch': 0.34}

  3%|▎         | 1407/41250 [3:24:05<95:42:29,  8.65s/it][2025-04-26 07:32:42,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:32:42,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.92 | bwd_microstep: 5731.34 | bwd_inner_microstep: 5697.59 | bwd_allreduce_microstep: 33.69 | step_microstep: 18.61
[2025-04-26 07:32:42,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.92 | bwd: 5731.35 | bwd_inner: 5697.59 | bwd_allreduce: 33.72 | step: 18.61

  3%|▎         | 1408/41250 [3:24:13<95:50:08,  8.66s/it]
                                                         
{'loss': 0.1273, 'grad_norm': 3.3238024711608887, 'learning_rate': 3.999821839829379e-05, 'epoch': 0.34}

  3%|▎         | 1408/41250 [3:24:13<95:50:08,  8.66s/it][2025-04-26 07:32:51,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 07:32:51,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.80 | bwd_microstep: 5769.25 | bwd_inner_microstep: 5657.41 | bwd_allreduce_microstep: 111.80 | step_microstep: 18.87
[2025-04-26 07:32:51,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.80 | bwd: 5769.26 | bwd_inner: 5657.41 | bwd_allreduce: 111.82 | step: 18.87

  3%|▎         | 1409/41250 [3:24:22<95:53:50,  8.67s/it]
                                                         
{'loss': 0.076, 'grad_norm': 0.7165248990058899, 'learning_rate': 3.99981973769424e-05, 'epoch': 0.34}

  3%|▎         | 1409/41250 [3:24:22<95:53:50,  8.67s/it][2025-04-26 07:33:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 07:33:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.20 | bwd_microstep: 5767.79 | bwd_inner_microstep: 5654.93 | bwd_allreduce_microstep: 112.82 | step_microstep: 18.54
[2025-04-26 07:33:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.20 | bwd: 5767.80 | bwd_inner: 5654.93 | bwd_allreduce: 112.83 | step: 18.54

  3%|▎         | 1410/41250 [3:24:31<95:58:41,  8.67s/it]
                                                         
{'loss': 0.0897, 'grad_norm': 1.1062217950820923, 'learning_rate': 3.9998176232306044e-05, 'epoch': 0.34}

  3%|▎         | 1410/41250 [3:24:31<95:58:41,  8.67s/it][2025-04-26 07:33:08,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:33:08,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.12 | bwd_microstep: 5715.30 | bwd_inner_microstep: 5702.49 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.48
[2025-04-26 07:33:08,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.12 | bwd: 5715.31 | bwd_inner: 5702.49 | bwd_allreduce: 12.78 | step: 18.49

  3%|▎         | 1411/41250 [3:24:39<95:54:10,  8.67s/it]
                                                         
{'loss': 0.3446, 'grad_norm': 3.7371253967285156, 'learning_rate': 3.999815496438486e-05, 'epoch': 0.34}

  3%|▎         | 1411/41250 [3:24:39<95:54:10,  8.67s/it][2025-04-26 07:33:17,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:33:17,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.34 | bwd_microstep: 5773.58 | bwd_inner_microstep: 5653.60 | bwd_allreduce_microstep: 119.93 | step_microstep: 18.88
[2025-04-26 07:33:17,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.34 | bwd: 5773.60 | bwd_inner: 5653.60 | bwd_allreduce: 119.95 | step: 18.88

  3%|▎         | 1412/41250 [3:24:48<96:00:57,  8.68s/it]
                                                         
{'loss': 0.2706, 'grad_norm': 1.8895668983459473, 'learning_rate': 3.9998133573179e-05, 'epoch': 0.34}

  3%|▎         | 1412/41250 [3:24:48<96:00:57,  8.68s/it][2025-04-26 07:33:25,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 07:33:25,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.96 | bwd_microstep: 5704.64 | bwd_inner_microstep: 5661.19 | bwd_allreduce_microstep: 43.40 | step_microstep: 18.97
[2025-04-26 07:33:25,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.96 | bwd: 5704.65 | bwd_inner: 5661.19 | bwd_allreduce: 43.42 | step: 18.97

  3%|▎         | 1413/41250 [3:24:57<95:48:55,  8.66s/it]
                                                         
{'loss': 0.1819, 'grad_norm': 2.7952985763549805, 'learning_rate': 3.999811205868857e-05, 'epoch': 0.34}

  3%|▎         | 1413/41250 [3:24:57<95:48:55,  8.66s/it][2025-04-26 07:33:34,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 07:33:34,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.59 | bwd_microstep: 5728.02 | bwd_inner_microstep: 5715.23 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.91
[2025-04-26 07:33:34,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.59 | bwd: 5728.03 | bwd_inner: 5715.23 | bwd_allreduce: 12.77 | step: 18.92

  3%|▎         | 1414/41250 [3:25:05<95:49:55,  8.66s/it]
                                                         
{'loss': 0.1321, 'grad_norm': 2.5381643772125244, 'learning_rate': 3.999809042091371e-05, 'epoch': 0.34}

  3%|▎         | 1414/41250 [3:25:05<95:49:55,  8.66s/it][2025-04-26 07:33:43,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 07:33:43,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2943.35 | bwd_microstep: 5881.70 | bwd_inner_microstep: 5868.74 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.18
[2025-04-26 07:33:43,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2943.35 | bwd: 5881.71 | bwd_inner: 5868.74 | bwd_allreduce: 12.93 | step: 19.18

  3%|▎         | 1415/41250 [3:25:14<96:37:53,  8.73s/it]
                                                         
{'loss': 0.1652, 'grad_norm': 1.1741539239883423, 'learning_rate': 3.999806865985456e-05, 'epoch': 0.34}

  3%|▎         | 1415/41250 [3:25:14<96:37:53,  8.73s/it][2025-04-26 07:33:52,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-26 07:33:52,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.93 | bwd_microstep: 5775.76 | bwd_inner_microstep: 5651.34 | bwd_allreduce_microstep: 124.37 | step_microstep: 18.81
[2025-04-26 07:33:52,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.93 | bwd: 5775.77 | bwd_inner: 5651.34 | bwd_allreduce: 124.39 | step: 18.82

  3%|▎         | 1416/41250 [3:25:23<96:29:29,  8.72s/it]
                                                         
{'loss': 0.129, 'grad_norm': 1.282066822052002, 'learning_rate': 3.999804677551125e-05, 'epoch': 0.34}

  3%|▎         | 1416/41250 [3:25:23<96:29:29,  8.72s/it][2025-04-26 07:34:00,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 07:34:00,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.75 | bwd_microstep: 5721.23 | bwd_inner_microstep: 5664.82 | bwd_allreduce_microstep: 56.35 | step_microstep: 18.98
[2025-04-26 07:34:00,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.75 | bwd: 5721.24 | bwd_inner: 5664.82 | bwd_allreduce: 56.38 | step: 18.98

  3%|▎         | 1417/41250 [3:25:32<96:12:20,  8.69s/it]
                                                         
{'loss': 0.2143, 'grad_norm': 2.1814627647399902, 'learning_rate': 3.999802476788391e-05, 'epoch': 0.34}

  3%|▎         | 1417/41250 [3:25:32<96:12:20,  8.69s/it][2025-04-26 07:34:09,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-26 07:34:09,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.42 | bwd_microstep: 5757.31 | bwd_inner_microstep: 5703.14 | bwd_allreduce_microstep: 54.12 | step_microstep: 19.31
[2025-04-26 07:34:09,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.42 | bwd: 5757.32 | bwd_inner: 5703.14 | bwd_allreduce: 54.14 | step: 19.31

  3%|▎         | 1418/41250 [3:25:40<96:11:04,  8.69s/it]
                                                         
{'loss': 0.2594, 'grad_norm': 2.446174144744873, 'learning_rate': 3.999800263697269e-05, 'epoch': 0.34}

  3%|▎         | 1418/41250 [3:25:40<96:11:04,  8.69s/it][2025-04-26 07:34:18,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 07:34:18,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.79 | bwd_microstep: 5793.84 | bwd_inner_microstep: 5652.42 | bwd_allreduce_microstep: 141.37 | step_microstep: 18.56
[2025-04-26 07:34:18,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.79 | bwd: 5793.85 | bwd_inner: 5652.42 | bwd_allreduce: 141.39 | step: 18.56

  3%|▎         | 1419/41250 [3:25:49<96:16:15,  8.70s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 1.1562961339950562, 'learning_rate': 3.999798038277771e-05, 'epoch': 0.34}

  3%|▎         | 1419/41250 [3:25:49<96:16:15,  8.70s/it][2025-04-26 07:34:26,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 07:34:26,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.99 | bwd_microstep: 5732.89 | bwd_inner_microstep: 5664.74 | bwd_allreduce_microstep: 68.11 | step_microstep: 18.67
[2025-04-26 07:34:26,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.99 | bwd: 5732.91 | bwd_inner: 5664.74 | bwd_allreduce: 68.13 | step: 18.67

  3%|▎         | 1420/41250 [3:25:58<96:02:51,  8.68s/it]
                                                         
{'loss': 0.2238, 'grad_norm': 1.6572139263153076, 'learning_rate': 3.999795800529912e-05, 'epoch': 0.34}

  3%|▎         | 1420/41250 [3:25:58<96:02:51,  8.68s/it][2025-04-26 07:34:35,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 07:34:35,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5777.83 | bwd_inner_microstep: 5651.30 | bwd_allreduce_microstep: 126.48 | step_microstep: 18.66
[2025-04-26 07:34:35,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5777.84 | bwd_inner: 5651.30 | bwd_allreduce: 126.50 | step: 18.67

  3%|▎         | 1421/41250 [3:26:06<96:04:49,  8.68s/it]
                                                         
{'loss': 0.09, 'grad_norm': 1.3042083978652954, 'learning_rate': 3.9997935504537046e-05, 'epoch': 0.34}

  3%|▎         | 1421/41250 [3:26:06<96:04:49,  8.68s/it][2025-04-26 07:34:44,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:34:44,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.92 | bwd_microstep: 5703.53 | bwd_inner_microstep: 5690.48 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.64
[2025-04-26 07:34:44,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.92 | bwd: 5703.55 | bwd_inner: 5690.48 | bwd_allreduce: 13.03 | step: 18.64

  3%|▎         | 1422/41250 [3:26:15<95:56:18,  8.67s/it]
                                                         
{'loss': 0.3185, 'grad_norm': 2.0707321166992188, 'learning_rate': 3.9997912880491635e-05, 'epoch': 0.34}

  3%|▎         | 1422/41250 [3:26:15<95:56:18,  8.67s/it][2025-04-26 07:34:52,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.99 | optimizer_step: 1.09
[2025-04-26 07:34:52,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.02 | bwd_microstep: 5782.60 | bwd_inner_microstep: 5649.91 | bwd_allreduce_microstep: 132.64 | step_microstep: 18.92
[2025-04-26 07:34:52,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.01 | bwd: 5782.61 | bwd_inner: 5649.91 | bwd_allreduce: 132.66 | step: 18.92

  3%|▎         | 1423/41250 [3:26:24<96:01:32,  8.68s/it]
                                                         
{'loss': 0.2913, 'grad_norm': 3.1486449241638184, 'learning_rate': 3.999789013316303e-05, 'epoch': 0.34}

  3%|▎         | 1423/41250 [3:26:24<96:01:32,  8.68s/it][2025-04-26 07:35:01,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 07:35:01,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.02 | bwd_microstep: 5783.91 | bwd_inner_microstep: 5771.03 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.99
[2025-04-26 07:35:01,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.02 | bwd: 5783.92 | bwd_inner: 5771.03 | bwd_allreduce: 12.85 | step: 18.99

  3%|▎         | 1424/41250 [3:26:32<96:14:35,  8.70s/it]
                                                         
{'loss': 0.0972, 'grad_norm': 1.164214849472046, 'learning_rate': 3.9997867262551355e-05, 'epoch': 0.35}

  3%|▎         | 1424/41250 [3:26:32<96:14:35,  8.70s/it][2025-04-26 07:35:10,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:35:10,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.82 | bwd_microstep: 5725.55 | bwd_inner_microstep: 5712.34 | bwd_allreduce_microstep: 13.17 | step_microstep: 19.16
[2025-04-26 07:35:10,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.82 | bwd: 5725.57 | bwd_inner: 5712.34 | bwd_allreduce: 13.19 | step: 19.17

  3%|▎         | 1425/41250 [3:26:41<96:07:48,  8.69s/it]
                                                         
{'loss': 0.3615, 'grad_norm': 2.7231788635253906, 'learning_rate': 3.999784426865677e-05, 'epoch': 0.35}

  3%|▎         | 1425/41250 [3:26:41<96:07:48,  8.69s/it][2025-04-26 07:35:19,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 07:35:19,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5820.53 | bwd_inner_microstep: 5665.22 | bwd_allreduce_microstep: 155.25 | step_microstep: 19.02
[2025-04-26 07:35:19,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5820.54 | bwd_inner: 5665.22 | bwd_allreduce: 155.28 | step: 19.02

  3%|▎         | 1426/41250 [3:26:50<96:17:41,  8.70s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 2.0048818588256836, 'learning_rate': 3.999782115147941e-05, 'epoch': 0.35}

  3%|▎         | 1426/41250 [3:26:50<96:17:41,  8.70s/it][2025-04-26 07:35:27,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.25 | optimizer_step: 1.07
[2025-04-26 07:35:27,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.07 | bwd_microstep: 5761.55 | bwd_inner_microstep: 5709.22 | bwd_allreduce_microstep: 52.26 | step_microstep: 19.69
[2025-04-26 07:35:27,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.07 | bwd: 5761.56 | bwd_inner: 5709.22 | bwd_allreduce: 52.29 | step: 19.70

  3%|▎         | 1427/41250 [3:26:59<96:18:03,  8.71s/it]
                                                         
{'loss': 0.1478, 'grad_norm': 2.272152900695801, 'learning_rate': 3.999779791101941e-05, 'epoch': 0.35}

  3%|▎         | 1427/41250 [3:26:59<96:18:03,  8.71s/it][2025-04-26 07:35:36,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:35:36,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.96 | bwd_microstep: 5793.00 | bwd_inner_microstep: 5645.00 | bwd_allreduce_microstep: 147.95 | step_microstep: 18.67
[2025-04-26 07:35:36,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.97 | bwd: 5793.01 | bwd_inner: 5645.00 | bwd_allreduce: 147.97 | step: 18.67

  3%|▎         | 1428/41250 [3:27:07<96:16:13,  8.70s/it]
                                                         
{'loss': 0.0465, 'grad_norm': 0.48385298252105713, 'learning_rate': 3.999777454727692e-05, 'epoch': 0.35}

  3%|▎         | 1428/41250 [3:27:07<96:16:13,  8.70s/it][2025-04-26 07:35:45,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:35:45,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.80 | bwd_microstep: 5727.82 | bwd_inner_microstep: 5655.15 | bwd_allreduce_microstep: 72.63 | step_microstep: 18.91
[2025-04-26 07:35:45,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.80 | bwd: 5727.84 | bwd_inner: 5655.15 | bwd_allreduce: 72.65 | step: 18.91

  3%|▎         | 1429/41250 [3:27:16<96:01:34,  8.68s/it]
                                                         
{'loss': 0.2344, 'grad_norm': 1.6796687841415405, 'learning_rate': 3.999775106025209e-05, 'epoch': 0.35}

  3%|▎         | 1429/41250 [3:27:16<96:01:34,  8.68s/it][2025-04-26 07:35:53,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:35:53,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.73 | bwd_microstep: 5747.44 | bwd_inner_microstep: 5705.95 | bwd_allreduce_microstep: 41.45 | step_microstep: 18.76
[2025-04-26 07:35:53,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.73 | bwd: 5747.46 | bwd_inner: 5705.95 | bwd_allreduce: 41.46 | step: 18.76

  3%|▎         | 1430/41250 [3:27:25<96:00:41,  8.68s/it]
                                                         
{'loss': 0.2645, 'grad_norm': 1.9619534015655518, 'learning_rate': 3.9997727449945046e-05, 'epoch': 0.35}

  3%|▎         | 1430/41250 [3:27:25<96:00:41,  8.68s/it][2025-04-26 07:36:02,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:36:02,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.19 | bwd_microstep: 5796.03 | bwd_inner_microstep: 5640.03 | bwd_allreduce_microstep: 155.96 | step_microstep: 18.64
[2025-04-26 07:36:02,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.19 | bwd: 5796.05 | bwd_inner: 5640.03 | bwd_allreduce: 155.98 | step: 18.64

  3%|▎         | 1431/41250 [3:27:33<96:04:10,  8.69s/it]
                                                         
{'loss': 0.055, 'grad_norm': 0.5623075366020203, 'learning_rate': 3.9997703716355957e-05, 'epoch': 0.35}

  3%|▎         | 1431/41250 [3:27:33<96:04:10,  8.69s/it][2025-04-26 07:36:11,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 07:36:11,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.95 | bwd_microstep: 5751.39 | bwd_inner_microstep: 5693.35 | bwd_allreduce_microstep: 57.99 | step_microstep: 18.64
[2025-04-26 07:36:11,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.95 | bwd: 5751.40 | bwd_inner: 5693.35 | bwd_allreduce: 58.01 | step: 18.64

  3%|▎         | 1432/41250 [3:27:42<96:03:50,  8.69s/it]
                                                         
{'loss': 0.2209, 'grad_norm': 2.4605021476745605, 'learning_rate': 3.999767985948495e-05, 'epoch': 0.35}

  3%|▎         | 1432/41250 [3:27:42<96:03:50,  8.69s/it][2025-04-26 07:36:19,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-26 07:36:19,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.42 | bwd_microstep: 5719.01 | bwd_inner_microstep: 5640.21 | bwd_allreduce_microstep: 78.74 | step_microstep: 19.35
[2025-04-26 07:36:19,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.42 | bwd: 5719.02 | bwd_inner: 5640.21 | bwd_allreduce: 78.77 | step: 19.35

  3%|▎         | 1433/41250 [3:27:51<95:49:22,  8.66s/it]
                                                         
{'loss': 0.2412, 'grad_norm': 1.1959060430526733, 'learning_rate': 3.9997655879332186e-05, 'epoch': 0.35}

  3%|▎         | 1433/41250 [3:27:51<95:49:22,  8.66s/it][2025-04-26 07:36:28,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 07:36:28,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.77 | bwd_microstep: 5735.67 | bwd_inner_microstep: 5700.56 | bwd_allreduce_microstep: 35.06 | step_microstep: 19.12
[2025-04-26 07:36:28,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.77 | bwd: 5735.68 | bwd_inner: 5700.56 | bwd_allreduce: 35.08 | step: 19.12

  3%|▎         | 1434/41250 [3:27:59<95:49:49,  8.66s/it]
                                                         
{'loss': 0.1531, 'grad_norm': 1.5170401334762573, 'learning_rate': 3.99976317758978e-05, 'epoch': 0.35}

  3%|▎         | 1434/41250 [3:27:59<95:49:49,  8.66s/it][2025-04-26 07:36:37,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:36:37,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.42 | bwd_microstep: 6055.03 | bwd_inner_microstep: 5646.47 | bwd_allreduce_microstep: 408.52 | step_microstep: 18.53
[2025-04-26 07:36:37,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.42 | bwd: 6055.05 | bwd_inner: 5646.47 | bwd_allreduce: 408.54 | step: 18.54

  3%|▎         | 1435/41250 [3:28:08<96:49:40,  8.76s/it]
                                                         
{'loss': 0.2661, 'grad_norm': 2.2354581356048584, 'learning_rate': 3.9997607549181954e-05, 'epoch': 0.35}

  3%|▎         | 1435/41250 [3:28:08<96:49:40,  8.76s/it][2025-04-26 07:36:46,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 07:36:46,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.72 | bwd_microstep: 5715.48 | bwd_inner_microstep: 5645.70 | bwd_allreduce_microstep: 69.72 | step_microstep: 19.25
[2025-04-26 07:36:46,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.72 | bwd: 5715.49 | bwd_inner: 5645.70 | bwd_allreduce: 69.75 | step: 19.26

  3%|▎         | 1436/41250 [3:28:17<96:23:40,  8.72s/it]
                                                         
{'loss': 0.3979, 'grad_norm': 2.247509479522705, 'learning_rate': 3.999758319918479e-05, 'epoch': 0.35}

  3%|▎         | 1436/41250 [3:28:17<96:23:40,  8.72s/it][2025-04-26 07:36:54,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:36:54,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.51 | bwd_microstep: 5784.75 | bwd_inner_microstep: 5772.11 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.58
[2025-04-26 07:36:54,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.51 | bwd: 5784.76 | bwd_inner: 5772.11 | bwd_allreduce: 12.61 | step: 18.59

  3%|▎         | 1437/41250 [3:28:26<96:30:09,  8.73s/it]
                                                         
{'loss': 0.297, 'grad_norm': 2.14667010307312, 'learning_rate': 3.999755872590646e-05, 'epoch': 0.35}

  3%|▎         | 1437/41250 [3:28:26<96:30:09,  8.73s/it][2025-04-26 07:37:03,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:37:03,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.82 | bwd_microstep: 5780.40 | bwd_inner_microstep: 5644.98 | bwd_allreduce_microstep: 135.37 | step_microstep: 18.50
[2025-04-26 07:37:03,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.82 | bwd: 5780.41 | bwd_inner: 5644.98 | bwd_allreduce: 135.39 | step: 18.50

  3%|▎         | 1438/41250 [3:28:34<96:21:05,  8.71s/it]
                                                         
{'loss': 0.2908, 'grad_norm': 1.641149640083313, 'learning_rate': 3.9997534129347106e-05, 'epoch': 0.35}

  3%|▎         | 1438/41250 [3:28:34<96:21:05,  8.71s/it][2025-04-26 07:37:12,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 07:37:12,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.77 | bwd_microstep: 5701.24 | bwd_inner_microstep: 5688.55 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.24
[2025-04-26 07:37:12,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.77 | bwd: 5701.25 | bwd_inner: 5688.55 | bwd_allreduce: 12.66 | step: 19.25

  3%|▎         | 1439/41250 [3:28:43<96:03:50,  8.69s/it]
                                                         
{'loss': 0.3695, 'grad_norm': 2.5762081146240234, 'learning_rate': 3.99975094095069e-05, 'epoch': 0.35}

  3%|▎         | 1439/41250 [3:28:43<96:03:50,  8.69s/it][2025-04-26 07:37:20,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:37:20,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.97 | bwd_microstep: 5709.48 | bwd_inner_microstep: 5696.71 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.79
[2025-04-26 07:37:20,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.97 | bwd: 5709.50 | bwd_inner: 5696.71 | bwd_allreduce: 12.74 | step: 18.79

  3%|▎         | 1440/41250 [3:28:51<95:54:34,  8.67s/it]
                                                         
{'loss': 0.0876, 'grad_norm': 0.9105509519577026, 'learning_rate': 3.999748456638597e-05, 'epoch': 0.35}

  3%|▎         | 1440/41250 [3:28:51<95:54:34,  8.67s/it][2025-04-26 07:37:29,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:37:29,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.23 | bwd_microstep: 5713.99 | bwd_inner_microstep: 5645.65 | bwd_allreduce_microstep: 68.29 | step_microstep: 18.38
[2025-04-26 07:37:29,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.23 | bwd: 5714.00 | bwd_inner: 5645.65 | bwd_allreduce: 68.31 | step: 18.38

  3%|▎         | 1441/41250 [3:29:00<95:43:46,  8.66s/it]
                                                         
{'loss': 0.1103, 'grad_norm': 1.895005702972412, 'learning_rate': 3.9997459599984484e-05, 'epoch': 0.35}

  3%|▎         | 1441/41250 [3:29:00<95:43:46,  8.66s/it][2025-04-26 07:37:38,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:37:38,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.71 | bwd_microstep: 5705.29 | bwd_inner_microstep: 5638.97 | bwd_allreduce_microstep: 66.28 | step_microstep: 17.72
[2025-04-26 07:37:38,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.71 | bwd: 5705.31 | bwd_inner: 5638.97 | bwd_allreduce: 66.30 | step: 17.73

  3%|▎         | 1442/41250 [3:29:09<95:32:16,  8.64s/it]
                                                         
{'loss': 0.2256, 'grad_norm': 1.8657420873641968, 'learning_rate': 3.999743451030259e-05, 'epoch': 0.35}

  3%|▎         | 1442/41250 [3:29:09<95:32:16,  8.64s/it][2025-04-26 07:37:46,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 07:37:46,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.81 | bwd_microstep: 5744.32 | bwd_inner_microstep: 5644.79 | bwd_allreduce_microstep: 99.48 | step_microstep: 18.33
[2025-04-26 07:37:46,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.81 | bwd: 5744.33 | bwd_inner: 5644.79 | bwd_allreduce: 99.50 | step: 18.33

  3%|▎         | 1443/41250 [3:29:17<95:37:45,  8.65s/it]
                                                         
{'loss': 0.3345, 'grad_norm': 3.1966073513031006, 'learning_rate': 3.9997409297340456e-05, 'epoch': 0.35}

  3%|▎         | 1443/41250 [3:29:17<95:37:45,  8.65s/it][2025-04-26 07:37:55,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 07:37:55,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.66 | bwd_microstep: 5716.63 | bwd_inner_microstep: 5703.53 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.88
[2025-04-26 07:37:55,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.66 | bwd: 5716.64 | bwd_inner: 5703.53 | bwd_allreduce: 13.07 | step: 18.89

  4%|▎         | 1444/41250 [3:29:26<95:38:05,  8.65s/it]
                                                         
{'loss': 0.1229, 'grad_norm': 1.1098512411117554, 'learning_rate': 3.999738396109822e-05, 'epoch': 0.35}

  4%|▎         | 1444/41250 [3:29:26<95:38:05,  8.65s/it][2025-04-26 07:38:04,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:38:04,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.45 | bwd_microstep: 5767.72 | bwd_inner_microstep: 5647.81 | bwd_allreduce_microstep: 119.86 | step_microstep: 18.57
[2025-04-26 07:38:04,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.45 | bwd: 5767.73 | bwd_inner: 5647.81 | bwd_allreduce: 119.88 | step: 18.57

  4%|▎         | 1445/41250 [3:29:35<95:43:42,  8.66s/it]
                                                         
{'loss': 0.2539, 'grad_norm': 1.7567987442016602, 'learning_rate': 3.9997358501576044e-05, 'epoch': 0.35}

  4%|▎         | 1445/41250 [3:29:35<95:43:42,  8.66s/it][2025-04-26 07:38:12,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:38:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.55 | bwd_microstep: 5693.43 | bwd_inner_microstep: 5644.12 | bwd_allreduce_microstep: 49.27 | step_microstep: 18.41
[2025-04-26 07:38:12,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.55 | bwd: 5693.44 | bwd_inner: 5644.12 | bwd_allreduce: 49.29 | step: 18.41

  4%|▎         | 1446/41250 [3:29:43<95:31:13,  8.64s/it]
                                                         
{'loss': 0.103, 'grad_norm': 1.1100419759750366, 'learning_rate': 3.999733291877409e-05, 'epoch': 0.35}

  4%|▎         | 1446/41250 [3:29:43<95:31:13,  8.64s/it][2025-04-26 07:38:21,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 07:38:21,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.37 | bwd_microstep: 5736.41 | bwd_inner_microstep: 5680.31 | bwd_allreduce_microstep: 56.05 | step_microstep: 18.60
[2025-04-26 07:38:21,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.37 | bwd: 5736.42 | bwd_inner: 5680.31 | bwd_allreduce: 56.07 | step: 18.60

  4%|▎         | 1447/41250 [3:29:52<95:34:06,  8.64s/it]
                                                         
{'loss': 0.2518, 'grad_norm': 1.8082115650177002, 'learning_rate': 3.999730721269251e-05, 'epoch': 0.35}

  4%|▎         | 1447/41250 [3:29:52<95:34:06,  8.64s/it][2025-04-26 07:38:29,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:38:29,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.07 | bwd_microstep: 5751.70 | bwd_inner_microstep: 5689.37 | bwd_allreduce_microstep: 62.28 | step_microstep: 18.68
[2025-04-26 07:38:29,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.07 | bwd: 5751.72 | bwd_inner: 5689.37 | bwd_allreduce: 62.30 | step: 18.68

  4%|▎         | 1448/41250 [3:30:01<95:40:26,  8.65s/it]
                                                         
{'loss': 0.2736, 'grad_norm': 1.80031418800354, 'learning_rate': 3.999728138333146e-05, 'epoch': 0.35}

  4%|▎         | 1448/41250 [3:30:01<95:40:26,  8.65s/it][2025-04-26 07:38:38,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 07:38:38,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.35 | bwd_microstep: 5705.98 | bwd_inner_microstep: 5693.07 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.87
[2025-04-26 07:38:38,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.35 | bwd: 5705.99 | bwd_inner: 5693.07 | bwd_allreduce: 12.88 | step: 18.87

  4%|▎         | 1449/41250 [3:30:09<95:36:32,  8.65s/it]
                                                         
{'loss': 0.2211, 'grad_norm': 1.5984536409378052, 'learning_rate': 3.999725543069111e-05, 'epoch': 0.35}

  4%|▎         | 1449/41250 [3:30:09<95:36:32,  8.65s/it][2025-04-26 07:38:47,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:38:47,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.59 | bwd_microstep: 5874.73 | bwd_inner_microstep: 5690.86 | bwd_allreduce_microstep: 183.82 | step_microstep: 18.67
[2025-04-26 07:38:47,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.59 | bwd: 5874.74 | bwd_inner: 5690.86 | bwd_allreduce: 183.84 | step: 18.68

  4%|▎         | 1450/41250 [3:30:18<96:07:22,  8.69s/it]
                                                         
{'loss': 0.2819, 'grad_norm': 1.8240289688110352, 'learning_rate': 3.9997229354771615e-05, 'epoch': 0.35}

  4%|▎         | 1450/41250 [3:30:18<96:07:22,  8.69s/it][2025-04-26 07:38:55,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:38:55,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2815.21 | bwd_microstep: 5709.35 | bwd_inner_microstep: 5631.88 | bwd_allreduce_microstep: 77.43 | step_microstep: 18.45
[2025-04-26 07:38:55,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2815.21 | bwd: 5709.37 | bwd_inner: 5631.88 | bwd_allreduce: 77.45 | step: 18.46

  4%|▎         | 1451/41250 [3:30:27<95:47:35,  8.66s/it]
                                                         
{'loss': 0.2321, 'grad_norm': 1.8302838802337646, 'learning_rate': 3.9997203155573124e-05, 'epoch': 0.35}

  4%|▎         | 1451/41250 [3:30:27<95:47:35,  8.66s/it][2025-04-26 07:39:04,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 07:39:04,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.99 | bwd_microstep: 5698.04 | bwd_inner_microstep: 5652.00 | bwd_allreduce_microstep: 46.01 | step_microstep: 17.70
[2025-04-26 07:39:04,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.99 | bwd: 5698.06 | bwd_inner: 5652.00 | bwd_allreduce: 46.02 | step: 17.70

  4%|▎         | 1452/41250 [3:30:35<95:35:08,  8.65s/it]
                                                         
{'loss': 0.1614, 'grad_norm': 1.0466374158859253, 'learning_rate': 3.999717683309582e-05, 'epoch': 0.35}

  4%|▎         | 1452/41250 [3:30:35<95:35:08,  8.65s/it][2025-04-26 07:39:13,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 07:39:13,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5760.93 | bwd_inner_microstep: 5684.82 | bwd_allreduce_microstep: 76.06 | step_microstep: 18.85
[2025-04-26 07:39:13,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5760.94 | bwd_inner: 5684.82 | bwd_allreduce: 76.08 | step: 18.85

  4%|▎         | 1453/41250 [3:30:44<95:42:39,  8.66s/it]
                                                         
{'loss': 0.1972, 'grad_norm': 1.559201717376709, 'learning_rate': 3.9997150387339846e-05, 'epoch': 0.35}

  4%|▎         | 1453/41250 [3:30:44<95:42:39,  8.66s/it][2025-04-26 07:39:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:39:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.58 | bwd_microstep: 5702.24 | bwd_inner_microstep: 5639.14 | bwd_allreduce_microstep: 63.05 | step_microstep: 18.48
[2025-04-26 07:39:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.58 | bwd: 5702.26 | bwd_inner: 5639.14 | bwd_allreduce: 63.07 | step: 18.48

  4%|▎         | 1454/41250 [3:30:53<95:32:16,  8.64s/it]
                                                         
{'loss': 0.3107, 'grad_norm': 1.8594213724136353, 'learning_rate': 3.999712381830538e-05, 'epoch': 0.35}

  4%|▎         | 1454/41250 [3:30:53<95:32:16,  8.64s/it][2025-04-26 07:39:30,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 07:39:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.82 | bwd_microstep: 5716.43 | bwd_inner_microstep: 5703.73 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.91
[2025-04-26 07:39:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.82 | bwd: 5716.44 | bwd_inner: 5703.73 | bwd_allreduce: 12.68 | step: 18.91

  4%|▎         | 1455/41250 [3:31:01<95:35:32,  8.65s/it]
                                                         
{'loss': 0.2066, 'grad_norm': 1.5064983367919922, 'learning_rate': 3.999709712599257e-05, 'epoch': 0.35}

  4%|▎         | 1455/41250 [3:31:01<95:35:32,  8.65s/it][2025-04-26 07:39:39,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.08 | optimizer_step: 1.10
[2025-04-26 07:39:39,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.81 | bwd_microstep: 5703.03 | bwd_inner_microstep: 5689.88 | bwd_allreduce_microstep: 13.10 | step_microstep: 19.20
[2025-04-26 07:39:39,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.81 | bwd: 5703.05 | bwd_inner: 5689.88 | bwd_allreduce: 13.12 | step: 19.20

  4%|▎         | 1456/41250 [3:31:10<95:30:46,  8.64s/it]
                                                         
{'loss': 0.1728, 'grad_norm': 1.2621794939041138, 'learning_rate': 3.9997070310401595e-05, 'epoch': 0.35}

  4%|▎         | 1456/41250 [3:31:10<95:30:46,  8.64s/it][2025-04-26 07:39:47,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 07:39:47,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.72 | bwd_microstep: 5712.40 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.93
[2025-04-26 07:39:47,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.72 | bwd: 5712.41 | bwd_inner: 5699.69 | bwd_allreduce: 12.68 | step: 18.93

  4%|▎         | 1457/41250 [3:31:18<95:31:39,  8.64s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 1.856050968170166, 'learning_rate': 3.999704337153261e-05, 'epoch': 0.35}

  4%|▎         | 1457/41250 [3:31:18<95:31:39,  8.64s/it][2025-04-26 07:39:56,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:39:56,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.85 | bwd_microstep: 5691.62 | bwd_inner_microstep: 5678.82 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.66
[2025-04-26 07:39:56,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.85 | bwd: 5691.63 | bwd_inner: 5678.81 | bwd_allreduce: 12.77 | step: 18.66

  4%|▎         | 1458/41250 [3:31:27<95:26:14,  8.63s/it]
                                                         
{'loss': 0.215, 'grad_norm': 1.5255622863769531, 'learning_rate': 3.999701630938579e-05, 'epoch': 0.35}

  4%|▎         | 1458/41250 [3:31:27<95:26:14,  8.63s/it][2025-04-26 07:40:05,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:40:05,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.00 | bwd_microstep: 5740.66 | bwd_inner_microstep: 5688.67 | bwd_allreduce_microstep: 51.95 | step_microstep: 18.56
[2025-04-26 07:40:05,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.00 | bwd: 5740.68 | bwd_inner: 5688.67 | bwd_allreduce: 51.96 | step: 18.56

  4%|▎         | 1459/41250 [3:31:36<95:32:46,  8.64s/it]
                                                         
{'loss': 0.2245, 'grad_norm': 1.4058496952056885, 'learning_rate': 3.99969891239613e-05, 'epoch': 0.35}

  4%|▎         | 1459/41250 [3:31:36<95:32:46,  8.64s/it][2025-04-26 07:40:13,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 07:40:13,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5773.46 | bwd_inner_microstep: 5698.10 | bwd_allreduce_microstep: 75.31 | step_microstep: 19.06
[2025-04-26 07:40:13,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5773.48 | bwd_inner: 5698.10 | bwd_allreduce: 75.33 | step: 19.06

  4%|▎         | 1460/41250 [3:31:44<95:43:19,  8.66s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 0.8514127731323242, 'learning_rate': 3.999696181525931e-05, 'epoch': 0.35}

  4%|▎         | 1460/41250 [3:31:44<95:43:19,  8.66s/it][2025-04-26 07:40:22,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:40:22,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.43 | bwd_microstep: 5712.87 | bwd_inner_microstep: 5699.83 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.78
[2025-04-26 07:40:22,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.43 | bwd: 5712.88 | bwd_inner: 5699.83 | bwd_allreduce: 13.00 | step: 18.79

  4%|▎         | 1461/41250 [3:31:53<95:42:46,  8.66s/it]
                                                         
{'loss': 0.3528, 'grad_norm': 1.8724827766418457, 'learning_rate': 3.999693438327997e-05, 'epoch': 0.35}

  4%|▎         | 1461/41250 [3:31:53<95:42:46,  8.66s/it][2025-04-26 07:40:31,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-26 07:40:31,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.45 | bwd_microstep: 5766.85 | bwd_inner_microstep: 5706.91 | bwd_allreduce_microstep: 59.89 | step_microstep: 19.04
[2025-04-26 07:40:31,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.45 | bwd: 5766.87 | bwd_inner: 5706.91 | bwd_allreduce: 59.91 | step: 19.04

  4%|▎         | 1462/41250 [3:32:02<95:50:01,  8.67s/it]
                                                         
{'loss': 0.1224, 'grad_norm': 0.8501356244087219, 'learning_rate': 3.999690682802347e-05, 'epoch': 0.35}

  4%|▎         | 1462/41250 [3:32:02<95:50:01,  8.67s/it][2025-04-26 07:40:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:40:39,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.65 | bwd_microstep: 5791.83 | bwd_inner_microstep: 5650.18 | bwd_allreduce_microstep: 141.59 | step_microstep: 18.70
[2025-04-26 07:40:39,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.65 | bwd: 5791.84 | bwd_inner: 5650.18 | bwd_allreduce: 141.62 | step: 18.70

  4%|▎         | 1463/41250 [3:32:11<95:56:52,  8.68s/it]
                                                         
{'loss': 0.2192, 'grad_norm': 1.2569706439971924, 'learning_rate': 3.999687914948997e-05, 'epoch': 0.35}

  4%|▎         | 1463/41250 [3:32:11<95:56:52,  8.68s/it][2025-04-26 07:40:48,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 07:40:48,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 5752.30 | bwd_inner_microstep: 5703.90 | bwd_allreduce_microstep: 48.34 | step_microstep: 18.83
[2025-04-26 07:40:48,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5752.32 | bwd_inner: 5703.90 | bwd_allreduce: 48.37 | step: 18.83

  4%|▎         | 1464/41250 [3:32:19<95:58:04,  8.68s/it]
                                                         
{'loss': 0.2261, 'grad_norm': 1.179273247718811, 'learning_rate': 3.9996851347679645e-05, 'epoch': 0.35}

  4%|▎         | 1464/41250 [3:32:19<95:58:04,  8.68s/it][2025-04-26 07:40:57,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 07:40:57,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2949.54 | bwd_microstep: 5886.48 | bwd_inner_microstep: 5873.74 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.28
[2025-04-26 07:40:57,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2949.54 | bwd: 5886.49 | bwd_inner: 5873.74 | bwd_allreduce: 12.71 | step: 19.28

  4%|▎         | 1465/41250 [3:32:28<96:43:16,  8.75s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 0.8367552161216736, 'learning_rate': 3.9996823422592665e-05, 'epoch': 0.36}

  4%|▎         | 1465/41250 [3:32:28<96:43:16,  8.75s/it][2025-04-26 07:41:06,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.04 | optimizer_step: 1.08
[2025-04-26 07:41:06,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.88 | bwd_microstep: 5705.56 | bwd_inner_microstep: 5692.65 | bwd_allreduce_microstep: 12.87 | step_microstep: 19.09
[2025-04-26 07:41:06,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.88 | bwd: 5705.57 | bwd_inner: 5692.65 | bwd_allreduce: 12.89 | step: 19.09

  4%|▎         | 1466/41250 [3:32:37<96:20:06,  8.72s/it]
                                                         
{'loss': 0.1841, 'grad_norm': 1.2622114419937134, 'learning_rate': 3.99967953742292e-05, 'epoch': 0.36}

  4%|▎         | 1466/41250 [3:32:37<96:20:06,  8.72s/it][2025-04-26 07:41:14,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:41:14,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.33 | bwd_microstep: 5794.28 | bwd_inner_microstep: 5781.36 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.93
[2025-04-26 07:41:14,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.33 | bwd: 5794.29 | bwd_inner: 5781.36 | bwd_allreduce: 12.88 | step: 18.93

  4%|▎         | 1467/41250 [3:32:46<96:30:19,  8.73s/it]
                                                         
{'loss': 0.2043, 'grad_norm': 1.6806117296218872, 'learning_rate': 3.999676720258942e-05, 'epoch': 0.36}

  4%|▎         | 1467/41250 [3:32:46<96:30:19,  8.73s/it][2025-04-26 07:41:23,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.13 | optimizer_step: 1.03
[2025-04-26 07:41:23,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.86 | bwd_microstep: 5717.91 | bwd_inner_microstep: 5661.49 | bwd_allreduce_microstep: 56.36 | step_microstep: 19.44
[2025-04-26 07:41:23,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.86 | bwd: 5717.92 | bwd_inner: 5661.49 | bwd_allreduce: 56.39 | step: 19.45

  4%|▎         | 1468/41250 [3:32:54<96:09:25,  8.70s/it]
                                                         
{'loss': 0.1919, 'grad_norm': 2.032365083694458, 'learning_rate': 3.999673890767352e-05, 'epoch': 0.36}

  4%|▎         | 1468/41250 [3:32:54<96:09:25,  8.70s/it][2025-04-26 07:41:32,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:41:32,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.75 | bwd_microstep: 5699.95 | bwd_inner_microstep: 5687.24 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.40
[2025-04-26 07:41:32,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.75 | bwd: 5699.96 | bwd_inner: 5687.24 | bwd_allreduce: 12.68 | step: 18.40

  4%|▎         | 1469/41250 [3:33:03<95:56:56,  8.68s/it]
                                                         
{'loss': 0.2133, 'grad_norm': 1.9367066621780396, 'learning_rate': 3.999671048948164e-05, 'epoch': 0.36}

  4%|▎         | 1469/41250 [3:33:03<95:56:56,  8.68s/it][2025-04-26 07:41:40,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:41:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.37 | bwd_microstep: 5752.21 | bwd_inner_microstep: 5700.21 | bwd_allreduce_microstep: 51.96 | step_microstep: 18.54
[2025-04-26 07:41:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.37 | bwd: 5752.23 | bwd_inner: 5700.21 | bwd_allreduce: 51.97 | step: 18.54

  4%|▎         | 1470/41250 [3:33:11<95:58:40,  8.69s/it]
                                                         
{'loss': 0.1707, 'grad_norm': 1.6863957643508911, 'learning_rate': 3.999668194801398e-05, 'epoch': 0.36}

  4%|▎         | 1470/41250 [3:33:11<95:58:40,  8.69s/it][2025-04-26 07:41:49,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:41:49,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.58 | bwd_microstep: 5707.93 | bwd_inner_microstep: 5695.13 | bwd_allreduce_microstep: 12.75 | step_microstep: 17.92
[2025-04-26 07:41:49,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.58 | bwd: 5707.94 | bwd_inner: 5695.13 | bwd_allreduce: 12.77 | step: 17.93

  4%|▎         | 1471/41250 [3:33:20<95:49:37,  8.67s/it]
                                                         
{'loss': 0.2773, 'grad_norm': 2.0024139881134033, 'learning_rate': 3.9996653283270715e-05, 'epoch': 0.36}

  4%|▎         | 1471/41250 [3:33:20<95:49:37,  8.67s/it][2025-04-26 07:41:58,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:41:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.03 | bwd_microstep: 5766.16 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 66.42 | step_microstep: 18.58
[2025-04-26 07:41:58,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.03 | bwd: 5766.18 | bwd_inner: 5699.69 | bwd_allreduce: 66.44 | step: 18.58

  4%|▎         | 1472/41250 [3:33:29<95:56:59,  8.68s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 1.3687374591827393, 'learning_rate': 3.999662449525201e-05, 'epoch': 0.36}

  4%|▎         | 1472/41250 [3:33:29<95:56:59,  8.68s/it][2025-04-26 07:42:06,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:42:06,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.08 | bwd_microstep: 5706.92 | bwd_inner_microstep: 5694.12 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.07
[2025-04-26 07:42:06,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.08 | bwd: 5706.93 | bwd_inner: 5694.12 | bwd_allreduce: 12.77 | step: 18.08

  4%|▎         | 1473/41250 [3:33:37<95:48:17,  8.67s/it]
                                                         
{'loss': 0.1698, 'grad_norm': 1.1240264177322388, 'learning_rate': 3.999659558395805e-05, 'epoch': 0.36}

  4%|▎         | 1473/41250 [3:33:37<95:48:17,  8.67s/it][2025-04-26 07:42:15,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.12
[2025-04-26 07:42:15,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.44 | bwd_microstep: 6004.64 | bwd_inner_microstep: 5691.81 | bwd_allreduce_microstep: 312.78 | step_microstep: 18.89
[2025-04-26 07:42:15,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.44 | bwd: 6004.66 | bwd_inner: 5691.81 | bwd_allreduce: 312.80 | step: 18.89

  4%|▎         | 1474/41250 [3:33:46<96:41:55,  8.75s/it]
                                                         
{'loss': 0.2368, 'grad_norm': 1.8121099472045898, 'learning_rate': 3.999656654938901e-05, 'epoch': 0.36}

  4%|▎         | 1474/41250 [3:33:46<96:41:55,  8.75s/it][2025-04-26 07:42:24,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:42:24,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.36 | bwd_microstep: 5776.65 | bwd_inner_microstep: 5658.06 | bwd_allreduce_microstep: 118.54 | step_microstep: 18.39
[2025-04-26 07:42:24,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.36 | bwd: 5776.66 | bwd_inner: 5658.06 | bwd_allreduce: 118.56 | step: 18.40

  4%|▎         | 1475/41250 [3:33:55<96:29:08,  8.73s/it]
                                                         
{'loss': 0.3083, 'grad_norm': 2.690002679824829, 'learning_rate': 3.999653739154506e-05, 'epoch': 0.36}

  4%|▎         | 1475/41250 [3:33:55<96:29:08,  8.73s/it][2025-04-26 07:42:33,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:42:33,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.15 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5645.14 | bwd_allreduce_microstep: 62.37 | step_microstep: 18.41
[2025-04-26 07:42:33,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.15 | bwd: 5707.56 | bwd_inner: 5645.14 | bwd_allreduce: 62.39 | step: 18.41

  4%|▎         | 1476/41250 [3:34:04<96:07:27,  8.70s/it]
                                                         
{'loss': 0.2203, 'grad_norm': 1.905430555343628, 'learning_rate': 3.99965081104264e-05, 'epoch': 0.36}

  4%|▎         | 1476/41250 [3:34:04<96:07:27,  8.70s/it][2025-04-26 07:42:41,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:42:41,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.83 | bwd_microstep: 5756.30 | bwd_inner_microstep: 5705.60 | bwd_allreduce_microstep: 50.66 | step_microstep: 18.55
[2025-04-26 07:42:41,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.83 | bwd: 5756.31 | bwd_inner: 5705.60 | bwd_allreduce: 50.67 | step: 18.56

  4%|▎         | 1477/41250 [3:34:12<96:05:55,  8.70s/it]
                                                         
{'loss': 0.4096, 'grad_norm': 2.81514048576355, 'learning_rate': 3.99964787060332e-05, 'epoch': 0.36}

  4%|▎         | 1477/41250 [3:34:12<96:05:55,  8.70s/it][2025-04-26 07:42:50,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:42:50,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5758.76 | bwd_inner_microstep: 5653.12 | bwd_allreduce_microstep: 105.60 | step_microstep: 18.37
[2025-04-26 07:42:50,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5758.78 | bwd_inner: 5653.12 | bwd_allreduce: 105.62 | step: 18.37

  4%|▎         | 1478/41250 [3:34:21<95:59:18,  8.69s/it]
                                                         
{'loss': 0.3761, 'grad_norm': 1.6981664896011353, 'learning_rate': 3.9996449178365646e-05, 'epoch': 0.36}

  4%|▎         | 1478/41250 [3:34:21<95:59:18,  8.69s/it][2025-04-26 07:42:59,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 07:42:59,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.49 | bwd_microstep: 5930.41 | bwd_inner_microstep: 5687.47 | bwd_allreduce_microstep: 242.89 | step_microstep: 19.23
[2025-04-26 07:42:59,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.49 | bwd: 5930.43 | bwd_inner: 5687.47 | bwd_allreduce: 242.91 | step: 19.24

  4%|▎         | 1479/41250 [3:34:30<96:33:35,  8.74s/it]
                                                         
{'loss': 0.2498, 'grad_norm': 2.331996440887451, 'learning_rate': 3.999641952742392e-05, 'epoch': 0.36}

  4%|▎         | 1479/41250 [3:34:30<96:33:35,  8.74s/it][2025-04-26 07:43:07,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 07:43:07,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.52 | bwd_microstep: 5713.76 | bwd_inner_microstep: 5700.85 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.26
[2025-04-26 07:43:07,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.52 | bwd: 5713.77 | bwd_inner: 5700.85 | bwd_allreduce: 12.88 | step: 18.26

  4%|▎         | 1480/41250 [3:34:39<96:14:09,  8.71s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 2.2072091102600098, 'learning_rate': 3.9996389753208184e-05, 'epoch': 0.36}

  4%|▎         | 1480/41250 [3:34:39<96:14:09,  8.71s/it][2025-04-26 07:43:16,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.22 | optimizer_step: 1.04
[2025-04-26 07:43:16,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.77 | bwd_microstep: 5733.11 | bwd_inner_microstep: 5685.04 | bwd_allreduce_microstep: 48.03 | step_microstep: 19.76
[2025-04-26 07:43:16,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.77 | bwd: 5733.12 | bwd_inner: 5685.04 | bwd_allreduce: 48.04 | step: 19.76

  4%|▎         | 1481/41250 [3:34:47<96:04:09,  8.70s/it]
                                                         
{'loss': 0.274, 'grad_norm': 2.425816535949707, 'learning_rate': 3.999635985571865e-05, 'epoch': 0.36}

  4%|▎         | 1481/41250 [3:34:47<96:04:09,  8.70s/it][2025-04-26 07:43:25,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:43:25,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.64 | bwd_microstep: 5718.61 | bwd_inner_microstep: 5645.01 | bwd_allreduce_microstep: 73.54 | step_microstep: 18.83
[2025-04-26 07:43:25,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.64 | bwd: 5718.62 | bwd_inner: 5645.01 | bwd_allreduce: 73.56 | step: 18.82

  4%|▎         | 1482/41250 [3:34:56<95:53:35,  8.68s/it]
                                                         
{'loss': 0.3354, 'grad_norm': 1.566833257675171, 'learning_rate': 3.9996329834955485e-05, 'epoch': 0.36}

  4%|▎         | 1482/41250 [3:34:56<95:53:35,  8.68s/it][2025-04-26 07:43:33,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:43:33,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.65 | bwd_microstep: 5728.18 | bwd_inner_microstep: 5681.51 | bwd_allreduce_microstep: 46.62 | step_microstep: 18.76
[2025-04-26 07:43:33,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.65 | bwd: 5728.19 | bwd_inner: 5681.51 | bwd_allreduce: 46.64 | step: 18.76

  4%|▎         | 1483/41250 [3:35:05<95:47:31,  8.67s/it]
                                                         
{'loss': 0.3582, 'grad_norm': 2.652193069458008, 'learning_rate': 3.9996299690918886e-05, 'epoch': 0.36}

  4%|▎         | 1483/41250 [3:35:05<95:47:31,  8.67s/it][2025-04-26 07:43:42,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:43:42,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.38 | bwd_microstep: 5776.30 | bwd_inner_microstep: 5644.43 | bwd_allreduce_microstep: 131.82 | step_microstep: 18.74
[2025-04-26 07:43:42,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.38 | bwd: 5776.31 | bwd_inner: 5644.43 | bwd_allreduce: 131.84 | step: 18.74

  4%|▎         | 1484/41250 [3:35:13<95:49:45,  8.68s/it]
                                                         
{'loss': 0.1765, 'grad_norm': 1.0759305953979492, 'learning_rate': 3.999626942360903e-05, 'epoch': 0.36}

  4%|▎         | 1484/41250 [3:35:13<95:49:45,  8.68s/it][2025-04-26 07:43:51,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 07:43:51,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.21 | bwd_microstep: 5749.30 | bwd_inner_microstep: 5642.93 | bwd_allreduce_microstep: 106.33 | step_microstep: 19.01
[2025-04-26 07:43:51,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.21 | bwd: 5749.32 | bwd_inner: 5642.93 | bwd_allreduce: 106.34 | step: 19.01

  4%|▎         | 1485/41250 [3:35:22<95:45:29,  8.67s/it]
                                                         
{'loss': 0.2075, 'grad_norm': 2.365459680557251, 'learning_rate': 3.99962390330261e-05, 'epoch': 0.36}

  4%|▎         | 1485/41250 [3:35:22<95:45:29,  8.67s/it][2025-04-26 07:43:59,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 07:43:59,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.28 | bwd_microstep: 5718.95 | bwd_inner_microstep: 5642.27 | bwd_allreduce_microstep: 76.62 | step_microstep: 19.05
[2025-04-26 07:43:59,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.28 | bwd: 5718.96 | bwd_inner: 5642.27 | bwd_allreduce: 76.64 | step: 19.05

  4%|▎         | 1486/41250 [3:35:31<95:35:12,  8.65s/it]
                                                         
{'loss': 0.1363, 'grad_norm': 1.0302506685256958, 'learning_rate': 3.9996208519170295e-05, 'epoch': 0.36}

  4%|▎         | 1486/41250 [3:35:31<95:35:12,  8.65s/it][2025-04-26 07:44:08,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 07:44:08,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.33 | bwd_microstep: 5707.02 | bwd_inner_microstep: 5640.31 | bwd_allreduce_microstep: 66.67 | step_microstep: 18.42
[2025-04-26 07:44:08,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.33 | bwd: 5707.03 | bwd_inner: 5640.31 | bwd_allreduce: 66.68 | step: 18.42

  4%|▎         | 1487/41250 [3:35:39<95:23:03,  8.64s/it]
                                                         
{'loss': 0.3115, 'grad_norm': 1.8053088188171387, 'learning_rate': 3.9996177882041793e-05, 'epoch': 0.36}

  4%|▎         | 1487/41250 [3:35:39<95:23:03,  8.64s/it][2025-04-26 07:44:17,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 07:44:17,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.21 | bwd_microstep: 5757.35 | bwd_inner_microstep: 5687.67 | bwd_allreduce_microstep: 69.63 | step_microstep: 18.51
[2025-04-26 07:44:17,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.21 | bwd: 5757.36 | bwd_inner: 5687.67 | bwd_allreduce: 69.64 | step: 18.52

  4%|▎         | 1488/41250 [3:35:48<95:32:06,  8.65s/it]
                                                         
{'loss': 0.126, 'grad_norm': 0.8862006664276123, 'learning_rate': 3.999614712164079e-05, 'epoch': 0.36}

  4%|▎         | 1488/41250 [3:35:48<95:32:06,  8.65s/it][2025-04-26 07:44:25,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-26 07:44:25,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.46 | bwd_microstep: 5744.44 | bwd_inner_microstep: 5697.79 | bwd_allreduce_microstep: 46.60 | step_microstep: 18.62
[2025-04-26 07:44:25,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.46 | bwd: 5744.45 | bwd_inner: 5697.79 | bwd_allreduce: 46.62 | step: 18.62

  4%|▎         | 1489/41250 [3:35:56<95:36:24,  8.66s/it]
                                                         
{'loss': 0.2796, 'grad_norm': 1.532922387123108, 'learning_rate': 3.9996116237967474e-05, 'epoch': 0.36}

  4%|▎         | 1489/41250 [3:35:56<95:36:24,  8.66s/it][2025-04-26 07:44:34,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-26 07:44:34,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.32 | bwd_microstep: 5792.00 | bwd_inner_microstep: 5639.33 | bwd_allreduce_microstep: 152.61 | step_microstep: 18.90
[2025-04-26 07:44:34,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.32 | bwd: 5792.02 | bwd_inner: 5639.33 | bwd_allreduce: 152.63 | step: 18.91

  4%|▎         | 1490/41250 [3:36:05<95:44:49,  8.67s/it]
                                                         
{'loss': 0.1581, 'grad_norm': 1.2733120918273926, 'learning_rate': 3.9996085231022037e-05, 'epoch': 0.36}

  4%|▎         | 1490/41250 [3:36:05<95:44:49,  8.67s/it][2025-04-26 07:44:43,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.32 | optimizer_step: 1.03
[2025-04-26 07:44:43,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.40 | bwd_microstep: 5693.84 | bwd_inner_microstep: 5680.09 | bwd_allreduce_microstep: 13.69 | step_microstep: 19.97
[2025-04-26 07:44:43,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.40 | bwd: 5693.86 | bwd_inner: 5680.09 | bwd_allreduce: 13.72 | step: 19.98

  4%|▎         | 1491/41250 [3:36:14<95:34:47,  8.65s/it]
                                                         
{'loss': 0.1441, 'grad_norm': 0.965934693813324, 'learning_rate': 3.999605410080467e-05, 'epoch': 0.36}

  4%|▎         | 1491/41250 [3:36:14<95:34:47,  8.65s/it][2025-04-26 07:44:51,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:44:51,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.22 | bwd_microstep: 5707.62 | bwd_inner_microstep: 5649.40 | bwd_allreduce_microstep: 58.18 | step_microstep: 18.72
[2025-04-26 07:44:51,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.22 | bwd: 5707.63 | bwd_inner: 5649.40 | bwd_allreduce: 58.19 | step: 18.72

  4%|▎         | 1492/41250 [3:36:22<95:25:49,  8.64s/it]
                                                         
{'loss': 0.1195, 'grad_norm': 0.9140812754631042, 'learning_rate': 3.999602284731555e-05, 'epoch': 0.36}

  4%|▎         | 1492/41250 [3:36:22<95:25:49,  8.64s/it][2025-04-26 07:45:00,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:45:00,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.58 | bwd_microstep: 5714.16 | bwd_inner_microstep: 5701.31 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.63
[2025-04-26 07:45:00,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.58 | bwd: 5714.17 | bwd_inner: 5701.31 | bwd_allreduce: 12.82 | step: 18.63

  4%|▎         | 1493/41250 [3:36:31<95:26:35,  8.64s/it]
                                                         
{'loss': 0.2856, 'grad_norm': 1.413238525390625, 'learning_rate': 3.9995991470554886e-05, 'epoch': 0.36}

  4%|▎         | 1493/41250 [3:36:31<95:26:35,  8.64s/it][2025-04-26 07:45:09,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:45:09,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.34 | bwd_microstep: 5768.37 | bwd_inner_microstep: 5755.72 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.72
[2025-04-26 07:45:09,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.34 | bwd: 5768.38 | bwd_inner: 5755.72 | bwd_allreduce: 12.62 | step: 18.72

  4%|▎         | 1494/41250 [3:36:40<95:43:49,  8.67s/it]
                                                         
{'loss': 0.1793, 'grad_norm': 1.392723560333252, 'learning_rate': 3.999595997052287e-05, 'epoch': 0.36}

  4%|▎         | 1494/41250 [3:36:40<95:43:49,  8.67s/it][2025-04-26 07:45:17,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:45:17,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.64 | bwd_microstep: 5704.39 | bwd_inner_microstep: 5647.02 | bwd_allreduce_microstep: 57.32 | step_microstep: 18.67
[2025-04-26 07:45:17,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.64 | bwd: 5704.40 | bwd_inner: 5647.02 | bwd_allreduce: 57.34 | step: 18.67

  4%|▎         | 1495/41250 [3:36:48<95:32:28,  8.65s/it]
                                                         
{'loss': 0.2429, 'grad_norm': 1.2666853666305542, 'learning_rate': 3.99959283472197e-05, 'epoch': 0.36}

  4%|▎         | 1495/41250 [3:36:48<95:32:28,  8.65s/it][2025-04-26 07:45:26,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.31 | optimizer_step: 1.04
[2025-04-26 07:45:26,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5714.95 | bwd_inner_microstep: 5639.11 | bwd_allreduce_microstep: 75.78 | step_microstep: 19.86
[2025-04-26 07:45:26,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5714.97 | bwd_inner: 5639.11 | bwd_allreduce: 75.81 | step: 19.87

  4%|▎         | 1496/41250 [3:36:57<95:24:13,  8.64s/it]
                                                         
{'loss': 0.1734, 'grad_norm': 0.9319347143173218, 'learning_rate': 3.999589660064556e-05, 'epoch': 0.36}

  4%|▎         | 1496/41250 [3:36:57<95:24:13,  8.64s/it][2025-04-26 07:45:34,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 0.93
[2025-04-26 07:45:34,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.50 | bwd_microstep: 5703.02 | bwd_inner_microstep: 5635.57 | bwd_allreduce_microstep: 67.40 | step_microstep: 19.51
[2025-04-26 07:45:34,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.50 | bwd: 5703.04 | bwd_inner: 5635.57 | bwd_allreduce: 67.43 | step: 19.51

  4%|▎         | 1497/41250 [3:37:06<95:18:00,  8.63s/it]
                                                         
{'loss': 0.0817, 'grad_norm': 0.706321656703949, 'learning_rate': 3.999586473080065e-05, 'epoch': 0.36}

  4%|▎         | 1497/41250 [3:37:06<95:18:00,  8.63s/it][2025-04-26 07:45:43,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:45:43,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5703.74 | bwd_inner_microstep: 5647.57 | bwd_allreduce_microstep: 56.12 | step_microstep: 18.72
[2025-04-26 07:45:43,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5703.75 | bwd_inner: 5647.57 | bwd_allreduce: 56.14 | step: 18.73

  4%|▎         | 1498/41250 [3:37:14<95:14:14,  8.62s/it]
                                                         
{'loss': 0.2084, 'grad_norm': 2.0975539684295654, 'learning_rate': 3.999583273768517e-05, 'epoch': 0.36}

  4%|▎         | 1498/41250 [3:37:14<95:14:14,  8.62s/it][2025-04-26 07:45:52,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:45:52,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.84 | bwd_microstep: 5763.34 | bwd_inner_microstep: 5692.04 | bwd_allreduce_microstep: 71.26 | step_microstep: 18.65
[2025-04-26 07:45:52,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.84 | bwd: 5763.36 | bwd_inner: 5692.04 | bwd_allreduce: 71.28 | step: 18.66

  4%|▎         | 1499/41250 [3:37:23<95:27:40,  8.65s/it]
                                                         
{'loss': 0.0856, 'grad_norm': 1.332549810409546, 'learning_rate': 3.9995800621299305e-05, 'epoch': 0.36}

  4%|▎         | 1499/41250 [3:37:23<95:27:40,  8.65s/it][2025-04-26 07:46:00,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:46:00,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.37 | bwd_microstep: 5788.94 | bwd_inner_microstep: 5776.22 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.60
[2025-04-26 07:46:00,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.37 | bwd: 5788.95 | bwd_inner: 5776.22 | bwd_allreduce: 12.69 | step: 18.60

  4%|▎         | 1500/41250 [3:37:32<95:50:35,  8.68s/it]
                                                         
{'loss': 0.1968, 'grad_norm': 1.4161889553070068, 'learning_rate': 3.999576838164327e-05, 'epoch': 0.36}

  4%|▎         | 1500/41250 [3:37:32<95:50:35,  8.68s/it][2025-04-26 07:46:09,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:46:09,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.73 | bwd_microstep: 5771.26 | bwd_inner_microstep: 5654.19 | bwd_allreduce_microstep: 117.02 | step_microstep: 18.65
[2025-04-26 07:46:09,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.73 | bwd: 5771.28 | bwd_inner: 5654.19 | bwd_allreduce: 117.04 | step: 18.66

  4%|▎         | 1501/41250 [3:37:40<95:49:54,  8.68s/it]
                                                         
{'loss': 0.0587, 'grad_norm': 0.6403433680534363, 'learning_rate': 3.999573601871726e-05, 'epoch': 0.36}

  4%|▎         | 1501/41250 [3:37:40<95:49:54,  8.68s/it][2025-04-26 07:46:18,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:46:18,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.53 | bwd_microstep: 5688.49 | bwd_inner_microstep: 5657.65 | bwd_allreduce_microstep: 30.79 | step_microstep: 18.71
[2025-04-26 07:46:18,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.53 | bwd: 5688.50 | bwd_inner: 5657.65 | bwd_allreduce: 30.81 | step: 18.71

  4%|▎         | 1502/41250 [3:37:49<95:36:13,  8.66s/it]
                                                         
{'loss': 0.3664, 'grad_norm': 2.4467220306396484, 'learning_rate': 3.9995703532521464e-05, 'epoch': 0.36}

  4%|▎         | 1502/41250 [3:37:49<95:36:13,  8.66s/it][2025-04-26 07:46:26,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 07:46:26,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5760.66 | bwd_inner_microstep: 5688.26 | bwd_allreduce_microstep: 72.35 | step_microstep: 19.19
[2025-04-26 07:46:26,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.55 | bwd: 5760.67 | bwd_inner: 5688.26 | bwd_allreduce: 72.37 | step: 19.19

  4%|▎         | 1503/41250 [3:37:58<95:42:27,  8.67s/it]
                                                         
{'loss': 0.2985, 'grad_norm': 2.417487859725952, 'learning_rate': 3.9995670923056084e-05, 'epoch': 0.36}

  4%|▎         | 1503/41250 [3:37:58<95:42:27,  8.67s/it][2025-04-26 07:46:35,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:46:35,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.89 | bwd_microstep: 5799.50 | bwd_inner_microstep: 5636.68 | bwd_allreduce_microstep: 162.77 | step_microstep: 18.49
[2025-04-26 07:46:35,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.89 | bwd: 5799.51 | bwd_inner: 5636.68 | bwd_allreduce: 162.79 | step: 18.49

  4%|▎         | 1504/41250 [3:38:06<95:48:29,  8.68s/it]
                                                         
{'loss': 0.1112, 'grad_norm': 1.0870672464370728, 'learning_rate': 3.9995638190321335e-05, 'epoch': 0.36}

  4%|▎         | 1504/41250 [3:38:06<95:48:29,  8.68s/it][2025-04-26 07:46:44,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 07:46:44,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.84 | bwd_microstep: 5719.42 | bwd_inner_microstep: 5691.87 | bwd_allreduce_microstep: 27.51 | step_microstep: 18.92
[2025-04-26 07:46:44,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5719.43 | bwd_inner: 5691.87 | bwd_allreduce: 27.52 | step: 18.92

  4%|▎         | 1505/41250 [3:38:15<95:39:16,  8.66s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 1.251186490058899, 'learning_rate': 3.9995605334317407e-05, 'epoch': 0.36}

  4%|▎         | 1505/41250 [3:38:15<95:39:16,  8.66s/it][2025-04-26 07:46:52,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:46:52,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5776.47 | bwd_inner_microstep: 5701.15 | bwd_allreduce_microstep: 75.27 | step_microstep: 18.69
[2025-04-26 07:46:52,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5776.48 | bwd_inner: 5701.15 | bwd_allreduce: 75.29 | step: 18.70

  4%|▎         | 1506/41250 [3:38:24<95:48:08,  8.68s/it]
                                                         
{'loss': 0.0948, 'grad_norm': 1.506079077720642, 'learning_rate': 3.99955723550445e-05, 'epoch': 0.37}

  4%|▎         | 1506/41250 [3:38:24<95:48:08,  8.68s/it][2025-04-26 07:47:01,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:47:01,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.94 | bwd_microstep: 5777.88 | bwd_inner_microstep: 5651.05 | bwd_allreduce_microstep: 126.78 | step_microstep: 18.63
[2025-04-26 07:47:01,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.94 | bwd: 5777.89 | bwd_inner: 5651.05 | bwd_allreduce: 126.80 | step: 18.63

  4%|▎         | 1507/41250 [3:38:32<95:49:21,  8.68s/it]
                                                         
{'loss': 0.1928, 'grad_norm': 1.2257496118545532, 'learning_rate': 3.9995539252502826e-05, 'epoch': 0.37}

  4%|▎         | 1507/41250 [3:38:32<95:49:21,  8.68s/it][2025-04-26 07:47:10,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:47:10,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.86 | bwd_microstep: 5900.32 | bwd_inner_microstep: 5662.27 | bwd_allreduce_microstep: 238.01 | step_microstep: 18.68
[2025-04-26 07:47:10,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.86 | bwd: 5900.34 | bwd_inner: 5662.27 | bwd_allreduce: 238.03 | step: 18.68

  4%|▎         | 1508/41250 [3:38:41<96:15:13,  8.72s/it]
                                                         
{'loss': 0.2387, 'grad_norm': 1.3967114686965942, 'learning_rate': 3.9995506026692586e-05, 'epoch': 0.37}

  4%|▎         | 1508/41250 [3:38:41<96:15:13,  8.72s/it][2025-04-26 07:47:19,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-26 07:47:19,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5714.69 | bwd_inner_microstep: 5654.89 | bwd_allreduce_microstep: 59.75 | step_microstep: 19.08
[2025-04-26 07:47:19,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5714.70 | bwd_inner: 5654.89 | bwd_allreduce: 59.77 | step: 19.08

  4%|▎         | 1509/41250 [3:38:50<95:57:07,  8.69s/it]
                                                         
{'loss': 0.0842, 'grad_norm': 0.8140724897384644, 'learning_rate': 3.999547267761398e-05, 'epoch': 0.37}

  4%|▎         | 1509/41250 [3:38:50<95:57:07,  8.69s/it][2025-04-26 07:47:27,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 07:47:27,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.64 | bwd_microstep: 5791.69 | bwd_inner_microstep: 5778.81 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.82
[2025-04-26 07:47:27,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.64 | bwd: 5791.71 | bwd_inner: 5778.81 | bwd_allreduce: 12.86 | step: 18.82

  4%|▎         | 1510/41250 [3:38:59<96:09:00,  8.71s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.5046802759170532, 'learning_rate': 3.999543920526723e-05, 'epoch': 0.37}

  4%|▎         | 1510/41250 [3:38:59<96:09:00,  8.71s/it][2025-04-26 07:47:36,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:47:36,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.22 | bwd_microstep: 5722.52 | bwd_inner_microstep: 5651.41 | bwd_allreduce_microstep: 71.07 | step_microstep: 18.84
[2025-04-26 07:47:36,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.22 | bwd: 5722.54 | bwd_inner: 5651.41 | bwd_allreduce: 71.09 | step: 18.85

  4%|▎         | 1511/41250 [3:39:07<95:52:49,  8.69s/it]
                                                         
{'loss': 0.0736, 'grad_norm': 0.9785255789756775, 'learning_rate': 3.999540560965251e-05, 'epoch': 0.37}

  4%|▎         | 1511/41250 [3:39:07<95:52:49,  8.69s/it][2025-04-26 07:47:45,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 07:47:45,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.81 | bwd_microstep: 5775.46 | bwd_inner_microstep: 5654.93 | bwd_allreduce_microstep: 120.49 | step_microstep: 18.42
[2025-04-26 07:47:45,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.81 | bwd: 5775.47 | bwd_inner: 5654.93 | bwd_allreduce: 120.51 | step: 18.42

  4%|▎         | 1512/41250 [3:39:16<95:54:09,  8.69s/it]
                                                         
{'loss': 0.2012, 'grad_norm': 1.7022404670715332, 'learning_rate': 3.999537189077006e-05, 'epoch': 0.37}

  4%|▎         | 1512/41250 [3:39:16<95:54:09,  8.69s/it][2025-04-26 07:47:53,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-26 07:47:53,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.91 | bwd_microstep: 5769.59 | bwd_inner_microstep: 5686.32 | bwd_allreduce_microstep: 83.22 | step_microstep: 18.64
[2025-04-26 07:47:53,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.91 | bwd: 5769.60 | bwd_inner: 5686.32 | bwd_allreduce: 83.24 | step: 18.64

  4%|▎         | 1513/41250 [3:39:25<95:54:32,  8.69s/it]
                                                         
{'loss': 0.035, 'grad_norm': 0.462002158164978, 'learning_rate': 3.999533804862008e-05, 'epoch': 0.37}

  4%|▎         | 1513/41250 [3:39:25<95:54:32,  8.69s/it][2025-04-26 07:48:02,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-26 07:48:02,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.30 | bwd_microstep: 5725.70 | bwd_inner_microstep: 5713.00 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.59
[2025-04-26 07:48:02,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.30 | bwd: 5725.72 | bwd_inner: 5713.00 | bwd_allreduce: 12.68 | step: 18.60

  4%|▎         | 1514/41250 [3:39:33<95:47:06,  8.68s/it]
                                                         
{'loss': 0.2237, 'grad_norm': 1.6976165771484375, 'learning_rate': 3.999530408320276e-05, 'epoch': 0.37}

  4%|▎         | 1514/41250 [3:39:33<95:47:06,  8.68s/it][2025-04-26 07:48:11,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.09 | optimizer_step: 0.98
[2025-04-26 07:48:11,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.55 | bwd_microstep: 5847.54 | bwd_inner_microstep: 5715.40 | bwd_allreduce_microstep: 132.07 | step_microstep: 19.27
[2025-04-26 07:48:11,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.55 | bwd: 5847.55 | bwd_inner: 5715.40 | bwd_allreduce: 132.10 | step: 19.27

  4%|▎         | 1515/41250 [3:39:42<96:09:11,  8.71s/it]
                                                         
{'loss': 0.1797, 'grad_norm': 1.2995394468307495, 'learning_rate': 3.999526999451833e-05, 'epoch': 0.37}

  4%|▎         | 1515/41250 [3:39:42<96:09:11,  8.71s/it][2025-04-26 07:48:20,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:48:20,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.28 | bwd_microstep: 5769.57 | bwd_inner_microstep: 5696.97 | bwd_allreduce_microstep: 72.56 | step_microstep: 18.60
[2025-04-26 07:48:20,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.28 | bwd: 5769.58 | bwd_inner: 5696.97 | bwd_allreduce: 72.58 | step: 18.61

  4%|▎         | 1516/41250 [3:39:51<96:06:44,  8.71s/it]
                                                         
{'loss': 0.1466, 'grad_norm': 1.6783506870269775, 'learning_rate': 3.9995235782567e-05, 'epoch': 0.37}

  4%|▎         | 1516/41250 [3:39:51<96:06:44,  8.71s/it][2025-04-26 07:48:28,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:48:28,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5716.83 | bwd_inner_microstep: 5659.02 | bwd_allreduce_microstep: 57.76 | step_microstep: 18.72
[2025-04-26 07:48:28,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5716.84 | bwd_inner: 5659.02 | bwd_allreduce: 57.77 | step: 18.72

  4%|▎         | 1517/41250 [3:39:59<95:50:37,  8.68s/it]
                                                         
{'loss': 0.089, 'grad_norm': 1.3413136005401611, 'learning_rate': 3.999520144734897e-05, 'epoch': 0.37}

  4%|▎         | 1517/41250 [3:39:59<95:50:37,  8.68s/it][2025-04-26 07:48:37,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:48:37,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5711.26 | bwd_inner_microstep: 5659.10 | bwd_allreduce_microstep: 52.11 | step_microstep: 18.74
[2025-04-26 07:48:37,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5711.27 | bwd_inner: 5659.10 | bwd_allreduce: 52.13 | step: 18.74

  4%|▎         | 1518/41250 [3:40:08<95:39:11,  8.67s/it]
                                                         
{'loss': 0.2833, 'grad_norm': 1.3505998849868774, 'learning_rate': 3.9995166988864454e-05, 'epoch': 0.37}

  4%|▎         | 1518/41250 [3:40:08<95:39:11,  8.67s/it][2025-04-26 07:48:45,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:48:45,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.68 | bwd_microstep: 5705.91 | bwd_inner_microstep: 5679.79 | bwd_allreduce_microstep: 26.08 | step_microstep: 18.55
[2025-04-26 07:48:45,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.68 | bwd: 5705.93 | bwd_inner: 5679.79 | bwd_allreduce: 26.10 | step: 18.55

  4%|▎         | 1519/41250 [3:40:17<95:30:04,  8.65s/it]
                                                         
{'loss': 0.0921, 'grad_norm': 1.6230792999267578, 'learning_rate': 3.999513240711367e-05, 'epoch': 0.37}

  4%|▎         | 1519/41250 [3:40:17<95:30:04,  8.65s/it][2025-04-26 07:48:54,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:48:54,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.60 | bwd_microstep: 5740.38 | bwd_inner_microstep: 5690.97 | bwd_allreduce_microstep: 49.37 | step_microstep: 18.63
[2025-04-26 07:48:54,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.60 | bwd: 5740.40 | bwd_inner: 5690.96 | bwd_allreduce: 49.39 | step: 18.63

  4%|▎         | 1520/41250 [3:40:25<95:34:11,  8.66s/it]
                                                         
{'loss': 0.1828, 'grad_norm': 1.3911571502685547, 'learning_rate': 3.999509770209682e-05, 'epoch': 0.37}

  4%|▎         | 1520/41250 [3:40:25<95:34:11,  8.66s/it][2025-04-26 07:49:03,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:49:03,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5733.83 | bwd_inner_microstep: 5685.07 | bwd_allreduce_microstep: 48.71 | step_microstep: 18.54
[2025-04-26 07:49:03,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5733.84 | bwd_inner: 5685.07 | bwd_allreduce: 48.73 | step: 18.54

  4%|▎         | 1521/41250 [3:40:34<95:35:44,  8.66s/it]
                                                         
{'loss': 0.1857, 'grad_norm': 2.6695592403411865, 'learning_rate': 3.999506287381413e-05, 'epoch': 0.37}

  4%|▎         | 1521/41250 [3:40:34<95:35:44,  8.66s/it][2025-04-26 07:49:11,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:49:11,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.12 | bwd_microstep: 5752.64 | bwd_inner_microstep: 5643.99 | bwd_allreduce_microstep: 108.61 | step_microstep: 18.53
[2025-04-26 07:49:11,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.12 | bwd: 5752.66 | bwd_inner: 5643.99 | bwd_allreduce: 108.62 | step: 18.53

  4%|▎         | 1522/41250 [3:40:43<95:33:51,  8.66s/it]
                                                         
{'loss': 0.2353, 'grad_norm': 2.761909246444702, 'learning_rate': 3.999502792226581e-05, 'epoch': 0.37}

  4%|▎         | 1522/41250 [3:40:43<95:33:51,  8.66s/it][2025-04-26 07:49:20,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 07:49:20,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.39 | bwd_microstep: 5717.26 | bwd_inner_microstep: 5704.49 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.71
[2025-04-26 07:49:20,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.39 | bwd: 5717.27 | bwd_inner: 5704.49 | bwd_allreduce: 12.73 | step: 18.71

  4%|▎         | 1523/41250 [3:40:51<95:28:49,  8.65s/it]
                                                         
{'loss': 0.1401, 'grad_norm': 2.0682520866394043, 'learning_rate': 3.999499284745208e-05, 'epoch': 0.37}

  4%|▎         | 1523/41250 [3:40:51<95:28:49,  8.65s/it][2025-04-26 07:49:29,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.24 | optimizer_step: 0.98
[2025-04-26 07:49:29,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.37 | bwd_microstep: 5737.17 | bwd_inner_microstep: 5682.48 | bwd_allreduce_microstep: 54.64 | step_microstep: 19.28
[2025-04-26 07:49:29,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.37 | bwd: 5737.19 | bwd_inner: 5682.48 | bwd_allreduce: 54.66 | step: 19.29

  4%|▎         | 1524/41250 [3:41:00<95:31:14,  8.66s/it]
                                                         
{'loss': 0.4993, 'grad_norm': 4.142135143280029, 'learning_rate': 3.999495764937314e-05, 'epoch': 0.37}

  4%|▎         | 1524/41250 [3:41:00<95:31:14,  8.66s/it][2025-04-26 07:49:37,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:49:37,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.86 | bwd_microstep: 5776.21 | bwd_inner_microstep: 5634.24 | bwd_allreduce_microstep: 141.93 | step_microstep: 18.70
[2025-04-26 07:49:37,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.86 | bwd: 5776.22 | bwd_inner: 5634.24 | bwd_allreduce: 141.95 | step: 18.70

  4%|▎         | 1525/41250 [3:41:09<95:35:41,  8.66s/it]
                                                         
{'loss': 0.1908, 'grad_norm': 2.383840560913086, 'learning_rate': 3.999492232802923e-05, 'epoch': 0.37}

  4%|▎         | 1525/41250 [3:41:09<95:35:41,  8.66s/it][2025-04-26 07:49:46,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:49:46,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5746.63 | bwd_inner_microstep: 5648.53 | bwd_allreduce_microstep: 98.05 | step_microstep: 18.43
[2025-04-26 07:49:46,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5746.65 | bwd_inner: 5648.53 | bwd_allreduce: 98.07 | step: 18.44

  4%|▎         | 1526/41250 [3:41:17<95:35:00,  8.66s/it]
                                                         
{'loss': 0.0542, 'grad_norm': 0.8349244594573975, 'learning_rate': 3.9994886883420555e-05, 'epoch': 0.37}

  4%|▎         | 1526/41250 [3:41:17<95:35:00,  8.66s/it][2025-04-26 07:49:55,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:49:55,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.97 | bwd_microstep: 5680.17 | bwd_inner_microstep: 5647.27 | bwd_allreduce_microstep: 32.85 | step_microstep: 18.30
[2025-04-26 07:49:55,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.97 | bwd: 5680.18 | bwd_inner: 5647.27 | bwd_allreduce: 32.86 | step: 18.30

  4%|▎         | 1527/41250 [3:41:26<95:21:29,  8.64s/it]
                                                         
{'loss': 0.2352, 'grad_norm': 3.0181057453155518, 'learning_rate': 3.999485131554733e-05, 'epoch': 0.37}

  4%|▎         | 1527/41250 [3:41:26<95:21:29,  8.64s/it][2025-04-26 07:50:03,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:50:03,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5704.86 | bwd_inner_microstep: 5692.06 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.43
[2025-04-26 07:50:03,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5704.87 | bwd_inner: 5692.05 | bwd_allreduce: 12.77 | step: 18.43

  4%|▎         | 1528/41250 [3:41:34<95:20:07,  8.64s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 1.339835524559021, 'learning_rate': 3.9994815624409785e-05, 'epoch': 0.37}

  4%|▎         | 1528/41250 [3:41:34<95:20:07,  8.64s/it][2025-04-26 07:50:12,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 07:50:12,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.44 | bwd_microstep: 5726.07 | bwd_inner_microstep: 5708.62 | bwd_allreduce_microstep: 17.41 | step_microstep: 18.26
[2025-04-26 07:50:12,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.44 | bwd: 5726.08 | bwd_inner: 5708.62 | bwd_allreduce: 17.42 | step: 18.26

  4%|▎         | 1529/41250 [3:41:43<95:24:55,  8.65s/it]
                                                         
{'loss': 0.2914, 'grad_norm': 3.3293869495391846, 'learning_rate': 3.999477981000813e-05, 'epoch': 0.37}

  4%|▎         | 1529/41250 [3:41:43<95:24:55,  8.65s/it][2025-04-26 07:50:21,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:50:21,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.40 | bwd_microstep: 5859.96 | bwd_inner_microstep: 5653.68 | bwd_allreduce_microstep: 206.23 | step_microstep: 18.31
[2025-04-26 07:50:21,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.40 | bwd: 5859.97 | bwd_inner: 5653.68 | bwd_allreduce: 206.25 | step: 18.31

  4%|▎         | 1530/41250 [3:41:52<95:50:37,  8.69s/it]
                                                         
{'loss': 0.1671, 'grad_norm': 1.4532431364059448, 'learning_rate': 3.999474387234259e-05, 'epoch': 0.37}

  4%|▎         | 1530/41250 [3:41:52<95:50:37,  8.69s/it][2025-04-26 07:50:29,864] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.00 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 07:50:29,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.83 | bwd_microstep: 5745.12 | bwd_inner_microstep: 5641.52 | bwd_allreduce_microstep: 103.55 | step_microstep: 17.90
[2025-04-26 07:50:29,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.83 | bwd: 5745.13 | bwd_inner: 5641.52 | bwd_allreduce: 103.57 | step: 17.91

  4%|▎         | 1531/41250 [3:42:01<95:44:24,  8.68s/it]
                                                         
{'loss': 0.3269, 'grad_norm': 3.852442502975464, 'learning_rate': 3.999470781141339e-05, 'epoch': 0.37}

  4%|▎         | 1531/41250 [3:42:01<95:44:24,  8.68s/it][2025-04-26 07:50:38,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.24 | optimizer_step: 0.95
[2025-04-26 07:50:38,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.34 | bwd_microstep: 5782.55 | bwd_inner_microstep: 5769.11 | bwd_allreduce_microstep: 13.38 | step_microstep: 19.54
[2025-04-26 07:50:38,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.34 | bwd: 5782.56 | bwd_inner: 5769.11 | bwd_allreduce: 13.40 | step: 19.55

  4%|▎         | 1532/41250 [3:42:09<95:57:46,  8.70s/it]
                                                         
{'loss': 0.2667, 'grad_norm': 2.9749257564544678, 'learning_rate': 3.999467162722075e-05, 'epoch': 0.37}

  4%|▎         | 1532/41250 [3:42:09<95:57:46,  8.70s/it][2025-04-26 07:50:47,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:50:47,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.70 | bwd_microstep: 5698.63 | bwd_inner_microstep: 5685.84 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.80
[2025-04-26 07:50:47,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.70 | bwd: 5698.65 | bwd_inner: 5685.84 | bwd_allreduce: 12.76 | step: 18.80

  4%|▎         | 1533/41250 [3:42:18<95:44:22,  8.68s/it]
                                                         
{'loss': 0.1041, 'grad_norm': 1.0432677268981934, 'learning_rate': 3.999463531976489e-05, 'epoch': 0.37}

  4%|▎         | 1533/41250 [3:42:18<95:44:22,  8.68s/it][2025-04-26 07:50:55,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 07:50:55,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.69 | bwd_microstep: 5694.37 | bwd_inner_microstep: 5681.33 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.83
[2025-04-26 07:50:55,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.69 | bwd: 5694.39 | bwd_inner: 5681.33 | bwd_allreduce: 13.01 | step: 18.83

  4%|▎         | 1534/41250 [3:42:27<95:33:27,  8.66s/it]
                                                         
{'loss': 0.1946, 'grad_norm': 0.6864929795265198, 'learning_rate': 3.999459888904603e-05, 'epoch': 0.37}

  4%|▎         | 1534/41250 [3:42:27<95:33:27,  8.66s/it][2025-04-26 07:51:04,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.27 | optimizer_step: 1.05
[2025-04-26 07:51:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.66 | bwd_microstep: 5673.19 | bwd_inner_microstep: 5642.52 | bwd_allreduce_microstep: 30.61 | step_microstep: 19.73
[2025-04-26 07:51:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.66 | bwd: 5673.21 | bwd_inner: 5642.52 | bwd_allreduce: 30.64 | step: 19.74

  4%|▎         | 1535/41250 [3:42:35<95:18:34,  8.64s/it]
                                                         
{'loss': 0.2191, 'grad_norm': 1.6716090440750122, 'learning_rate': 3.9994562335064415e-05, 'epoch': 0.37}

  4%|▎         | 1535/41250 [3:42:35<95:18:34,  8.64s/it][2025-04-26 07:51:13,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 07:51:13,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.04 | bwd_microstep: 6009.61 | bwd_inner_microstep: 5645.52 | bwd_allreduce_microstep: 364.03 | step_microstep: 19.04
[2025-04-26 07:51:13,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.04 | bwd: 6009.63 | bwd_inner: 5645.52 | bwd_allreduce: 364.05 | step: 19.04

  4%|▎         | 1536/41250 [3:42:44<96:13:55,  8.72s/it]
                                                         
{'loss': 0.2406, 'grad_norm': 2.1767807006835938, 'learning_rate': 3.999452565782025e-05, 'epoch': 0.37}

  4%|▎         | 1536/41250 [3:42:44<96:13:55,  8.72s/it][2025-04-26 07:51:22,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 07:51:22,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.20 | bwd_microstep: 5749.03 | bwd_inner_microstep: 5686.90 | bwd_allreduce_microstep: 62.09 | step_microstep: 19.09
[2025-04-26 07:51:22,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.20 | bwd: 5749.05 | bwd_inner: 5686.90 | bwd_allreduce: 62.11 | step: 19.09

  4%|▎         | 1537/41250 [3:42:53<96:02:40,  8.71s/it]
                                                         
{'loss': 0.2255, 'grad_norm': 2.0364630222320557, 'learning_rate': 3.999448885731376e-05, 'epoch': 0.37}

  4%|▎         | 1537/41250 [3:42:53<96:02:40,  8.71s/it][2025-04-26 07:51:30,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:51:30,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.96 | bwd_microstep: 5761.84 | bwd_inner_microstep: 5684.35 | bwd_allreduce_microstep: 77.44 | step_microstep: 18.70
[2025-04-26 07:51:30,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.96 | bwd: 5761.85 | bwd_inner: 5684.35 | bwd_allreduce: 77.46 | step: 18.71

  4%|▎         | 1538/41250 [3:43:01<95:58:11,  8.70s/it]
                                                         
{'loss': 0.0479, 'grad_norm': 0.511328935623169, 'learning_rate': 3.999445193354518e-05, 'epoch': 0.37}

  4%|▎         | 1538/41250 [3:43:01<95:58:11,  8.70s/it][2025-04-26 07:51:39,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 07:51:39,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.37 | bwd_microstep: 5767.36 | bwd_inner_microstep: 5680.35 | bwd_allreduce_microstep: 86.97 | step_microstep: 18.73
[2025-04-26 07:51:39,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.37 | bwd: 5767.38 | bwd_inner: 5680.35 | bwd_allreduce: 86.99 | step: 18.73

  4%|▎         | 1539/41250 [3:43:10<95:58:00,  8.70s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 2.1847283840179443, 'learning_rate': 3.999441488651474e-05, 'epoch': 0.37}

  4%|▎         | 1539/41250 [3:43:10<95:58:00,  8.70s/it][2025-04-26 07:51:48,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.19
[2025-04-26 07:51:48,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.41 | bwd_microstep: 5769.58 | bwd_inner_microstep: 5682.61 | bwd_allreduce_microstep: 86.90 | step_microstep: 20.02
[2025-04-26 07:51:48,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.41 | bwd: 5769.59 | bwd_inner: 5682.61 | bwd_allreduce: 86.93 | step: 20.02

  4%|▎         | 1540/41250 [3:43:19<95:57:38,  8.70s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 2.0137927532196045, 'learning_rate': 3.999437771622267e-05, 'epoch': 0.37}

  4%|▎         | 1540/41250 [3:43:19<95:57:38,  8.70s/it][2025-04-26 07:51:56,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 07:51:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.94 | bwd_microstep: 5763.78 | bwd_inner_microstep: 5643.46 | bwd_allreduce_microstep: 120.27 | step_microstep: 18.94
[2025-04-26 07:51:56,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.94 | bwd: 5763.79 | bwd_inner: 5643.46 | bwd_allreduce: 120.29 | step: 18.94

  4%|▎         | 1541/41250 [3:43:27<95:49:35,  8.69s/it]
                                                         
{'loss': 0.2231, 'grad_norm': 1.6479216814041138, 'learning_rate': 3.999434042266919e-05, 'epoch': 0.37}

  4%|▎         | 1541/41250 [3:43:27<95:49:35,  8.69s/it][2025-04-26 07:52:05,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:52:05,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.78 | bwd_microstep: 5879.26 | bwd_inner_microstep: 5677.47 | bwd_allreduce_microstep: 201.74 | step_microstep: 18.88
[2025-04-26 07:52:05,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.78 | bwd: 5879.27 | bwd_inner: 5677.47 | bwd_allreduce: 201.76 | step: 18.88

  4%|▎         | 1542/41250 [3:43:36<96:13:19,  8.72s/it]
                                                         
{'loss': 0.2305, 'grad_norm': 1.5375070571899414, 'learning_rate': 3.999430300585453e-05, 'epoch': 0.37}

  4%|▎         | 1542/41250 [3:43:36<96:13:19,  8.72s/it][2025-04-26 07:52:14,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:52:14,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.34 | bwd_microstep: 5715.43 | bwd_inner_microstep: 5646.38 | bwd_allreduce_microstep: 69.00 | step_microstep: 18.55
[2025-04-26 07:52:14,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.35 | bwd: 5715.44 | bwd_inner: 5646.38 | bwd_allreduce: 69.02 | step: 18.55

  4%|▎         | 1543/41250 [3:43:45<95:52:47,  8.69s/it]
                                                         
{'loss': 0.2075, 'grad_norm': 1.986948847770691, 'learning_rate': 3.9994265465778934e-05, 'epoch': 0.37}

  4%|▎         | 1543/41250 [3:43:45<95:52:47,  8.69s/it][2025-04-26 07:52:22,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 07:52:22,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.80 | bwd_microstep: 5803.09 | bwd_inner_microstep: 5649.90 | bwd_allreduce_microstep: 153.14 | step_microstep: 18.86
[2025-04-26 07:52:22,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.80 | bwd: 5803.10 | bwd_inner: 5649.90 | bwd_allreduce: 153.16 | step: 18.87

  4%|▎         | 1544/41250 [3:43:54<95:55:20,  8.70s/it]
                                                         
{'loss': 0.2135, 'grad_norm': 2.262139320373535, 'learning_rate': 3.999422780244262e-05, 'epoch': 0.37}

  4%|▎         | 1544/41250 [3:43:54<95:55:20,  8.70s/it][2025-04-26 07:52:31,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.01
[2025-04-26 07:52:31,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.86 | bwd_microstep: 5790.67 | bwd_inner_microstep: 5777.93 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.59
[2025-04-26 07:52:31,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.86 | bwd: 5790.68 | bwd_inner: 5777.93 | bwd_allreduce: 12.71 | step: 18.59

  4%|▎         | 1545/41250 [3:44:02<96:08:20,  8.72s/it]
                                                         
{'loss': 0.2298, 'grad_norm': 1.9080450534820557, 'learning_rate': 3.9994190015845815e-05, 'epoch': 0.37}

  4%|▎         | 1545/41250 [3:44:02<96:08:20,  8.72s/it][2025-04-26 07:52:40,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 07:52:40,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.03 | bwd_microstep: 5796.75 | bwd_inner_microstep: 5645.63 | bwd_allreduce_microstep: 151.06 | step_microstep: 18.73
[2025-04-26 07:52:40,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.03 | bwd: 5796.76 | bwd_inner: 5645.63 | bwd_allreduce: 151.08 | step: 18.73

  4%|▎         | 1546/41250 [3:44:11<96:04:05,  8.71s/it]
                                                         
{'loss': 0.3054, 'grad_norm': 1.4884932041168213, 'learning_rate': 3.999415210598877e-05, 'epoch': 0.37}

  4%|▎         | 1546/41250 [3:44:11<96:04:05,  8.71s/it][2025-04-26 07:52:49,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 07:52:49,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.49 | bwd_microstep: 5788.79 | bwd_inner_microstep: 5641.34 | bwd_allreduce_microstep: 147.41 | step_microstep: 19.12
[2025-04-26 07:52:49,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.49 | bwd: 5788.81 | bwd_inner: 5641.34 | bwd_allreduce: 147.43 | step: 19.12

  4%|▍         | 1547/41250 [3:44:20<96:01:06,  8.71s/it]
                                                         
{'loss': 0.0881, 'grad_norm': 0.8211141228675842, 'learning_rate': 3.999411407287171e-05, 'epoch': 0.38}

  4%|▍         | 1547/41250 [3:44:20<96:01:06,  8.71s/it][2025-04-26 07:52:57,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 07:52:57,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.59 | bwd_microstep: 5722.64 | bwd_inner_microstep: 5709.82 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.74
[2025-04-26 07:52:57,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.59 | bwd: 5722.65 | bwd_inner: 5709.82 | bwd_allreduce: 12.78 | step: 18.74

  4%|▍         | 1548/41250 [3:44:28<95:51:37,  8.69s/it]
                                                         
{'loss': 0.0734, 'grad_norm': 0.8325644135475159, 'learning_rate': 3.999407591649486e-05, 'epoch': 0.38}

  4%|▍         | 1548/41250 [3:44:28<95:51:37,  8.69s/it][2025-04-26 07:53:06,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:53:06,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.85 | bwd_microstep: 5721.70 | bwd_inner_microstep: 5652.11 | bwd_allreduce_microstep: 69.55 | step_microstep: 18.63
[2025-04-26 07:53:06,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.85 | bwd: 5721.72 | bwd_inner: 5652.11 | bwd_allreduce: 69.57 | step: 18.64

  4%|▍         | 1549/41250 [3:44:37<95:39:42,  8.67s/it]
                                                         
{'loss': 0.1361, 'grad_norm': 2.868225336074829, 'learning_rate': 3.999403763685847e-05, 'epoch': 0.38}

  4%|▍         | 1549/41250 [3:44:37<95:39:42,  8.67s/it][2025-04-26 07:53:15,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 07:53:15,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.50 | bwd_microstep: 5880.67 | bwd_inner_microstep: 5708.14 | bwd_allreduce_microstep: 172.48 | step_microstep: 18.65
[2025-04-26 07:53:15,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.50 | bwd: 5880.69 | bwd_inner: 5708.14 | bwd_allreduce: 172.50 | step: 18.65

  4%|▍         | 1550/41250 [3:44:46<96:05:55,  8.71s/it]
                                                         
{'loss': 0.3387, 'grad_norm': 1.7591174840927124, 'learning_rate': 3.999399923396278e-05, 'epoch': 0.38}

  4%|▍         | 1550/41250 [3:44:46<96:05:55,  8.71s/it][2025-04-26 07:53:23,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 07:53:23,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.06 | bwd_microstep: 5761.15 | bwd_inner_microstep: 5692.70 | bwd_allreduce_microstep: 68.40 | step_microstep: 18.59
[2025-04-26 07:53:23,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.06 | bwd: 5761.16 | bwd_inner: 5692.70 | bwd_allreduce: 68.42 | step: 18.60

  4%|▍         | 1551/41250 [3:44:55<96:01:17,  8.71s/it]
                                                         
{'loss': 0.173, 'grad_norm': 2.088082790374756, 'learning_rate': 3.999396070780801e-05, 'epoch': 0.38}

  4%|▍         | 1551/41250 [3:44:55<96:01:17,  8.71s/it][2025-04-26 07:53:32,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:53:32,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.90 | bwd_microstep: 5706.40 | bwd_inner_microstep: 5693.39 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.76
[2025-04-26 07:53:32,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.90 | bwd: 5706.41 | bwd_inner: 5693.39 | bwd_allreduce: 12.98 | step: 18.76

  4%|▍         | 1552/41250 [3:45:03<95:49:01,  8.69s/it]
                                                         
{'loss': 0.3612, 'grad_norm': 3.401329517364502, 'learning_rate': 3.99939220583944e-05, 'epoch': 0.38}

  4%|▍         | 1552/41250 [3:45:03<95:49:01,  8.69s/it][2025-04-26 07:53:41,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:53:41,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.88 | bwd_microstep: 5724.45 | bwd_inner_microstep: 5646.14 | bwd_allreduce_microstep: 78.26 | step_microstep: 18.59
[2025-04-26 07:53:41,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.88 | bwd: 5724.46 | bwd_inner: 5646.14 | bwd_allreduce: 78.28 | step: 18.60

  4%|▍         | 1553/41250 [3:45:12<95:38:07,  8.67s/it]
                                                         
{'loss': 0.2852, 'grad_norm': 1.3923684358596802, 'learning_rate': 3.99938832857222e-05, 'epoch': 0.38}

  4%|▍         | 1553/41250 [3:45:12<95:38:07,  8.67s/it][2025-04-26 07:53:49,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 07:53:49,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.52 | bwd_microstep: 5823.72 | bwd_inner_microstep: 5781.03 | bwd_allreduce_microstep: 42.65 | step_microstep: 18.67
[2025-04-26 07:53:49,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.52 | bwd: 5823.74 | bwd_inner: 5781.02 | bwd_allreduce: 42.67 | step: 18.67

  4%|▍         | 1554/41250 [3:45:21<96:02:40,  8.71s/it]
                                                         
{'loss': 0.4183, 'grad_norm': 2.4711391925811768, 'learning_rate': 3.9993844389791644e-05, 'epoch': 0.38}

  4%|▍         | 1554/41250 [3:45:21<96:02:40,  8.71s/it][2025-04-26 07:53:58,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.25 | optimizer_step: 1.07
[2025-04-26 07:53:58,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.61 | bwd_microstep: 5897.63 | bwd_inner_microstep: 5651.12 | bwd_allreduce_microstep: 246.46 | step_microstep: 20.00
[2025-04-26 07:53:58,742] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.61 | bwd: 5897.65 | bwd_inner: 5651.12 | bwd_allreduce: 246.48 | step: 20.00

  4%|▍         | 1555/41250 [3:45:29<96:21:07,  8.74s/it]
                                                         
{'loss': 0.2744, 'grad_norm': 1.9650774002075195, 'learning_rate': 3.9993805370602966e-05, 'epoch': 0.38}

  4%|▍         | 1555/41250 [3:45:29<96:21:07,  8.74s/it][2025-04-26 07:54:07,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:54:07,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.16 | bwd_microstep: 5728.20 | bwd_inner_microstep: 5662.56 | bwd_allreduce_microstep: 65.59 | step_microstep: 18.68
[2025-04-26 07:54:07,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.16 | bwd: 5728.21 | bwd_inner: 5662.56 | bwd_allreduce: 65.61 | step: 18.68

  4%|▍         | 1556/41250 [3:45:38<96:01:30,  8.71s/it]
                                                         
{'loss': 0.1413, 'grad_norm': 0.9770250916481018, 'learning_rate': 3.999376622815641e-05, 'epoch': 0.38}

  4%|▍         | 1556/41250 [3:45:38<96:01:30,  8.71s/it][2025-04-26 07:54:16,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:54:16,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.21 | bwd_microstep: 5711.07 | bwd_inner_microstep: 5698.00 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.70
[2025-04-26 07:54:16,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.21 | bwd: 5711.09 | bwd_inner: 5698.00 | bwd_allreduce: 13.04 | step: 18.70

  4%|▍         | 1557/41250 [3:45:47<95:48:18,  8.69s/it]
                                                         
{'loss': 0.4093, 'grad_norm': 2.748772621154785, 'learning_rate': 3.999372696245222e-05, 'epoch': 0.38}

  4%|▍         | 1557/41250 [3:45:47<95:48:18,  8.69s/it][2025-04-26 07:54:24,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 07:54:24,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5788.81 | bwd_inner_microstep: 5665.38 | bwd_allreduce_microstep: 123.38 | step_microstep: 18.68
[2025-04-26 07:54:24,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5788.82 | bwd_inner: 5665.38 | bwd_allreduce: 123.40 | step: 18.68

  4%|▍         | 1558/41250 [3:45:55<95:50:35,  8.69s/it]
                                                         
{'loss': 0.3622, 'grad_norm': 1.955615758895874, 'learning_rate': 3.999368757349063e-05, 'epoch': 0.38}

  4%|▍         | 1558/41250 [3:45:55<95:50:35,  8.69s/it][2025-04-26 07:54:33,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 07:54:33,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.54 | bwd_microstep: 5787.74 | bwd_inner_microstep: 5701.23 | bwd_allreduce_microstep: 86.46 | step_microstep: 18.77
[2025-04-26 07:54:33,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.54 | bwd: 5787.76 | bwd_inner: 5701.23 | bwd_allreduce: 86.48 | step: 18.77

  4%|▍         | 1559/41250 [3:46:04<95:54:00,  8.70s/it]
                                                         
{'loss': 0.1036, 'grad_norm': 1.0729230642318726, 'learning_rate': 3.999364806127189e-05, 'epoch': 0.38}

  4%|▍         | 1559/41250 [3:46:04<95:54:00,  8.70s/it][2025-04-26 07:54:42,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.03 | optimizer_step: 1.15
[2025-04-26 07:54:42,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5778.33 | bwd_inner_microstep: 5640.82 | bwd_allreduce_microstep: 137.46 | step_microstep: 18.86
[2025-04-26 07:54:42,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5778.34 | bwd_inner: 5640.82 | bwd_allreduce: 137.48 | step: 18.87

  4%|▍         | 1560/41250 [3:46:13<95:52:28,  8.70s/it]
                                                         
{'loss': 0.1082, 'grad_norm': 0.9197335243225098, 'learning_rate': 3.999360842579625e-05, 'epoch': 0.38}

  4%|▍         | 1560/41250 [3:46:13<95:52:28,  8.70s/it][2025-04-26 07:54:50,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:54:50,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5803.07 | bwd_inner_microstep: 5684.82 | bwd_allreduce_microstep: 118.20 | step_microstep: 18.46
[2025-04-26 07:54:50,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5803.08 | bwd_inner: 5684.82 | bwd_allreduce: 118.22 | step: 18.46

  4%|▍         | 1561/41250 [3:46:22<96:00:15,  8.71s/it]
                                                         
{'loss': 0.1724, 'grad_norm': 1.9154253005981445, 'learning_rate': 3.999356866706394e-05, 'epoch': 0.38}

  4%|▍         | 1561/41250 [3:46:22<96:00:15,  8.71s/it][2025-04-26 07:54:59,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:54:59,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.45 | bwd_microstep: 5875.68 | bwd_inner_microstep: 5700.23 | bwd_allreduce_microstep: 175.41 | step_microstep: 18.33
[2025-04-26 07:54:59,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.45 | bwd: 5875.69 | bwd_inner: 5700.23 | bwd_allreduce: 175.42 | step: 18.34

  4%|▍         | 1562/41250 [3:46:30<96:20:59,  8.74s/it]
                                                         
{'loss': 0.1566, 'grad_norm': 1.4319030046463013, 'learning_rate': 3.999352878507521e-05, 'epoch': 0.38}

  4%|▍         | 1562/41250 [3:46:30<96:20:59,  8.74s/it][2025-04-26 07:55:08,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 07:55:08,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.65 | bwd_microstep: 5782.70 | bwd_inner_microstep: 5689.47 | bwd_allreduce_microstep: 93.18 | step_microstep: 17.72
[2025-04-26 07:55:08,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.65 | bwd: 5782.71 | bwd_inner: 5689.47 | bwd_allreduce: 93.20 | step: 17.72

  4%|▍         | 1563/41250 [3:46:39<96:15:16,  8.73s/it]
                                                         
{'loss': 0.1994, 'grad_norm': 1.3357878923416138, 'learning_rate': 3.999348877983031e-05, 'epoch': 0.38}

  4%|▍         | 1563/41250 [3:46:39<96:15:16,  8.73s/it][2025-04-26 07:55:17,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.64 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:55:17,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.60 | bwd_microstep: 5715.25 | bwd_inner_microstep: 5651.55 | bwd_allreduce_microstep: 63.65 | step_microstep: 19.16
[2025-04-26 07:55:17,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.60 | bwd: 5715.26 | bwd_inner: 5651.55 | bwd_allreduce: 63.67 | step: 19.16

  4%|▍         | 1564/41250 [3:46:48<95:52:57,  8.70s/it]
                                                         
{'loss': 0.2058, 'grad_norm': 2.1749494075775146, 'learning_rate': 3.999344865132949e-05, 'epoch': 0.38}

  4%|▍         | 1564/41250 [3:46:48<95:52:57,  8.70s/it][2025-04-26 07:55:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 07:55:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.51 | bwd_microstep: 5695.01 | bwd_inner_microstep: 5682.26 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.83
[2025-04-26 07:55:25,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.51 | bwd: 5695.03 | bwd_inner: 5682.26 | bwd_allreduce: 12.72 | step: 18.83

  4%|▍         | 1565/41250 [3:46:56<95:39:13,  8.68s/it]
                                                         
{'loss': 0.283, 'grad_norm': 2.168722152709961, 'learning_rate': 3.999340839957299e-05, 'epoch': 0.38}

  4%|▍         | 1565/41250 [3:46:56<95:39:13,  8.68s/it][2025-04-26 07:55:34,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 07:55:34,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.13 | bwd_microstep: 5709.01 | bwd_inner_microstep: 5636.86 | bwd_allreduce_microstep: 72.11 | step_microstep: 18.83
[2025-04-26 07:55:34,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.14 | bwd: 5709.03 | bwd_inner: 5636.86 | bwd_allreduce: 72.12 | step: 18.83

  4%|▍         | 1566/41250 [3:47:05<95:26:21,  8.66s/it]
                                                         
{'loss': 0.0979, 'grad_norm': 1.1641699075698853, 'learning_rate': 3.999336802456105e-05, 'epoch': 0.38}

  4%|▍         | 1566/41250 [3:47:05<95:26:21,  8.66s/it][2025-04-26 07:55:42,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.01
[2025-04-26 07:55:42,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.10 | bwd_microstep: 5694.36 | bwd_inner_microstep: 5671.85 | bwd_allreduce_microstep: 22.47 | step_microstep: 19.08
[2025-04-26 07:55:42,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.10 | bwd: 5694.38 | bwd_inner: 5671.85 | bwd_allreduce: 22.49 | step: 19.08

  4%|▍         | 1567/41250 [3:47:14<95:18:31,  8.65s/it]
                                                         
{'loss': 0.251, 'grad_norm': 2.116384506225586, 'learning_rate': 3.9993327526293936e-05, 'epoch': 0.38}

  4%|▍         | 1567/41250 [3:47:14<95:18:31,  8.65s/it][2025-04-26 07:55:51,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 07:55:51,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5737.92 | bwd_inner_microstep: 5697.59 | bwd_allreduce_microstep: 40.27 | step_microstep: 18.82
[2025-04-26 07:55:51,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5737.94 | bwd_inner: 5697.59 | bwd_allreduce: 40.30 | step: 18.82

  4%|▍         | 1568/41250 [3:47:22<95:22:09,  8.65s/it]
                                                         
{'loss': 0.1459, 'grad_norm': 1.4303969144821167, 'learning_rate': 3.999328690477189e-05, 'epoch': 0.38}

  4%|▍         | 1568/41250 [3:47:22<95:22:09,  8.65s/it][2025-04-26 07:56:00,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.34 | optimizer_step: 1.03
[2025-04-26 07:56:00,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.06 | bwd_microstep: 5746.09 | bwd_inner_microstep: 5687.51 | bwd_allreduce_microstep: 58.52 | step_microstep: 19.88
[2025-04-26 07:56:00,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.06 | bwd: 5746.10 | bwd_inner: 5687.51 | bwd_allreduce: 58.55 | step: 19.88

  4%|▍         | 1569/41250 [3:47:31<95:28:41,  8.66s/it]
                                                         
{'loss': 0.1157, 'grad_norm': 1.2285670042037964, 'learning_rate': 3.999324615999516e-05, 'epoch': 0.38}

  4%|▍         | 1569/41250 [3:47:31<95:28:41,  8.66s/it][2025-04-26 07:56:08,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.25 | optimizer_step: 0.94
[2025-04-26 07:56:08,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.52 | bwd_microstep: 5787.84 | bwd_inner_microstep: 5774.39 | bwd_allreduce_microstep: 13.39 | step_microstep: 19.55
[2025-04-26 07:56:08,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.52 | bwd: 5787.86 | bwd_inner: 5774.39 | bwd_allreduce: 13.42 | step: 19.56

  4%|▍         | 1570/41250 [3:47:40<95:46:47,  8.69s/it]
                                                         
{'loss': 0.1929, 'grad_norm': 5.892937183380127, 'learning_rate': 3.9993205291964006e-05, 'epoch': 0.38}

  4%|▍         | 1570/41250 [3:47:40<95:46:47,  8.69s/it][2025-04-26 07:56:17,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.21 | optimizer_step: 0.91
[2025-04-26 07:56:17,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.32 | bwd_microstep: 6007.69 | bwd_inner_microstep: 5701.03 | bwd_allreduce_microstep: 306.60 | step_microstep: 19.22
[2025-04-26 07:56:17,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.32 | bwd: 6007.71 | bwd_inner: 5701.03 | bwd_allreduce: 306.63 | step: 19.22

  4%|▍         | 1571/41250 [3:47:49<96:36:41,  8.77s/it]
                                                         
{'loss': 0.2577, 'grad_norm': 2.3509435653686523, 'learning_rate': 3.999316430067867e-05, 'epoch': 0.38}

  4%|▍         | 1571/41250 [3:47:49<96:36:41,  8.77s/it][2025-04-26 07:56:26,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 07:56:26,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.40 | bwd_microstep: 5768.18 | bwd_inner_microstep: 5646.03 | bwd_allreduce_microstep: 122.10 | step_microstep: 19.28
[2025-04-26 07:56:26,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.40 | bwd: 5768.19 | bwd_inner: 5646.03 | bwd_allreduce: 122.12 | step: 19.28

  4%|▍         | 1572/41250 [3:47:57<96:17:35,  8.74s/it]
                                                         
{'loss': 0.2173, 'grad_norm': 1.0611356496810913, 'learning_rate': 3.9993123186139405e-05, 'epoch': 0.38}

  4%|▍         | 1572/41250 [3:47:57<96:17:35,  8.74s/it][2025-04-26 07:56:35,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 07:56:35,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.40 | bwd_microstep: 5700.01 | bwd_inner_microstep: 5680.77 | bwd_allreduce_microstep: 19.18 | step_microstep: 18.85
[2025-04-26 07:56:35,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.40 | bwd: 5700.02 | bwd_inner: 5680.77 | bwd_allreduce: 19.21 | step: 18.86

  4%|▍         | 1573/41250 [3:48:06<95:53:12,  8.70s/it]
                                                         
{'loss': 0.2512, 'grad_norm': 2.1197805404663086, 'learning_rate': 3.999308194834647e-05, 'epoch': 0.38}

  4%|▍         | 1573/41250 [3:48:06<95:53:12,  8.70s/it][2025-04-26 07:56:43,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 07:56:44,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.96 | bwd_microstep: 5874.52 | bwd_inner_microstep: 5672.73 | bwd_allreduce_microstep: 201.74 | step_microstep: 19.32
[2025-04-26 07:56:44,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.96 | bwd: 5874.54 | bwd_inner: 5672.73 | bwd_allreduce: 201.76 | step: 19.32

  4%|▍         | 1574/41250 [3:48:15<96:11:54,  8.73s/it]
                                                         
{'loss': 0.1492, 'grad_norm': 1.3625242710113525, 'learning_rate': 3.999304058730012e-05, 'epoch': 0.38}

  4%|▍         | 1574/41250 [3:48:15<96:11:54,  8.73s/it][2025-04-26 07:56:52,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 07:56:52,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 5701.87 | bwd_inner_microstep: 5638.80 | bwd_allreduce_microstep: 63.03 | step_microstep: 19.28
[2025-04-26 07:56:52,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 5701.89 | bwd_inner: 5638.80 | bwd_allreduce: 63.05 | step: 19.29

  4%|▍         | 1575/41250 [3:48:23<95:49:04,  8.69s/it]
                                                         
{'loss': 0.275, 'grad_norm': 1.9309241771697998, 'learning_rate': 3.99929991030006e-05, 'epoch': 0.38}

  4%|▍         | 1575/41250 [3:48:23<95:49:04,  8.69s/it][2025-04-26 07:57:01,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:57:01,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.08 | bwd_microstep: 5732.81 | bwd_inner_microstep: 5691.33 | bwd_allreduce_microstep: 41.44 | step_microstep: 18.54
[2025-04-26 07:57:01,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.08 | bwd: 5732.83 | bwd_inner: 5691.33 | bwd_allreduce: 41.45 | step: 18.55

  4%|▍         | 1576/41250 [3:48:32<95:44:44,  8.69s/it]
                                                         
{'loss': 0.1002, 'grad_norm': 1.4644147157669067, 'learning_rate': 3.999295749544819e-05, 'epoch': 0.38}

  4%|▍         | 1576/41250 [3:48:32<95:44:44,  8.69s/it][2025-04-26 07:57:10,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 07:57:10,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.66 | bwd_microstep: 5865.06 | bwd_inner_microstep: 5681.37 | bwd_allreduce_microstep: 183.63 | step_microstep: 18.75
[2025-04-26 07:57:10,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.66 | bwd: 5865.07 | bwd_inner: 5681.37 | bwd_allreduce: 183.65 | step: 18.75

  4%|▍         | 1577/41250 [3:48:41<96:04:53,  8.72s/it]
                                                         
{'loss': 0.0494, 'grad_norm': 0.43515923619270325, 'learning_rate': 3.999291576464311e-05, 'epoch': 0.38}

  4%|▍         | 1577/41250 [3:48:41<96:04:53,  8.72s/it][2025-04-26 07:57:18,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.93
[2025-04-26 07:57:18,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5728.40 | bwd_inner_microstep: 5697.05 | bwd_allreduce_microstep: 31.31 | step_microstep: 18.99
[2025-04-26 07:57:18,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5728.42 | bwd_inner: 5697.05 | bwd_allreduce: 31.33 | step: 18.99

  4%|▍         | 1578/41250 [3:48:49<95:53:34,  8.70s/it]
                                                         
{'loss': 0.1377, 'grad_norm': 1.1193110942840576, 'learning_rate': 3.999287391058564e-05, 'epoch': 0.38}

  4%|▍         | 1578/41250 [3:48:49<95:53:34,  8.70s/it][2025-04-26 07:57:27,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:57:27,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.84 | bwd_microstep: 5725.53 | bwd_inner_microstep: 5690.16 | bwd_allreduce_microstep: 35.31 | step_microstep: 18.81
[2025-04-26 07:57:27,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.84 | bwd: 5725.54 | bwd_inner: 5690.16 | bwd_allreduce: 35.34 | step: 18.82

  4%|▍         | 1579/41250 [3:48:58<95:43:59,  8.69s/it]
                                                         
{'loss': 0.2547, 'grad_norm': 1.9580682516098022, 'learning_rate': 3.9992831933276045e-05, 'epoch': 0.38}

  4%|▍         | 1579/41250 [3:48:58<95:43:59,  8.69s/it][2025-04-26 07:57:36,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 07:57:36,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.66 | bwd_microstep: 5754.25 | bwd_inner_microstep: 5682.33 | bwd_allreduce_microstep: 71.87 | step_microstep: 19.00
[2025-04-26 07:57:36,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.66 | bwd: 5754.26 | bwd_inner: 5682.33 | bwd_allreduce: 71.89 | step: 19.00

  4%|▍         | 1580/41250 [3:49:07<95:41:58,  8.68s/it]
                                                         
{'loss': 0.1124, 'grad_norm': 1.0232946872711182, 'learning_rate': 3.999278983271457e-05, 'epoch': 0.38}

  4%|▍         | 1580/41250 [3:49:07<95:41:58,  8.68s/it][2025-04-26 07:57:44,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 07:57:44,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.59 | bwd_microstep: 5704.12 | bwd_inner_microstep: 5691.28 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.85
[2025-04-26 07:57:44,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.59 | bwd: 5704.13 | bwd_inner: 5691.28 | bwd_allreduce: 12.81 | step: 18.85

  4%|▍         | 1581/41250 [3:49:15<95:31:39,  8.67s/it]
                                                         
{'loss': 0.1237, 'grad_norm': 1.1335231065750122, 'learning_rate': 3.9992747608901475e-05, 'epoch': 0.38}

  4%|▍         | 1581/41250 [3:49:15<95:31:39,  8.67s/it][2025-04-26 07:57:53,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 07:57:53,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.06 | bwd_microstep: 5704.73 | bwd_inner_microstep: 5638.60 | bwd_allreduce_microstep: 66.08 | step_microstep: 18.91
[2025-04-26 07:57:53,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.07 | bwd: 5704.75 | bwd_inner: 5638.60 | bwd_allreduce: 66.10 | step: 18.91

  4%|▍         | 1582/41250 [3:49:24<95:18:37,  8.65s/it]
                                                         
{'loss': 0.1547, 'grad_norm': 2.0688400268554688, 'learning_rate': 3.999270526183702e-05, 'epoch': 0.38}

  4%|▍         | 1582/41250 [3:49:24<95:18:37,  8.65s/it][2025-04-26 07:58:01,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 07:58:01,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.87 | bwd_microstep: 5695.11 | bwd_inner_microstep: 5682.44 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.76
[2025-04-26 07:58:01,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.87 | bwd: 5695.13 | bwd_inner: 5682.44 | bwd_allreduce: 12.65 | step: 18.76

  4%|▍         | 1583/41250 [3:49:33<95:12:23,  8.64s/it]
                                                         
{'loss': 0.0837, 'grad_norm': 0.9563020467758179, 'learning_rate': 3.9992662791521474e-05, 'epoch': 0.38}

  4%|▍         | 1583/41250 [3:49:33<95:12:23,  8.64s/it][2025-04-26 07:58:10,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:58:10,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.94 | bwd_microstep: 5701.39 | bwd_inner_microstep: 5688.61 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.06
[2025-04-26 07:58:10,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.94 | bwd: 5701.41 | bwd_inner: 5688.61 | bwd_allreduce: 12.75 | step: 19.06

  4%|▍         | 1584/41250 [3:49:41<95:10:13,  8.64s/it]
                                                         
{'loss': 0.0639, 'grad_norm': 1.0616672039031982, 'learning_rate': 3.999262019795509e-05, 'epoch': 0.38}

  4%|▍         | 1584/41250 [3:49:41<95:10:13,  8.64s/it][2025-04-26 07:58:19,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 07:58:19,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.38 | bwd_microstep: 5881.84 | bwd_inner_microstep: 5686.25 | bwd_allreduce_microstep: 195.53 | step_microstep: 19.35
[2025-04-26 07:58:19,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.38 | bwd: 5881.85 | bwd_inner: 5686.25 | bwd_allreduce: 195.55 | step: 19.36

  4%|▍         | 1585/41250 [3:49:50<95:44:24,  8.69s/it]
                                                         
{'loss': 0.0577, 'grad_norm': 0.9424113035202026, 'learning_rate': 3.999257748113815e-05, 'epoch': 0.38}

  4%|▍         | 1585/41250 [3:49:50<95:44:24,  8.69s/it][2025-04-26 07:58:28,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:58:28,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5758.47 | bwd_inner_microstep: 5676.92 | bwd_allreduce_microstep: 81.50 | step_microstep: 19.10
[2025-04-26 07:58:28,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5758.48 | bwd_inner: 5676.91 | bwd_allreduce: 81.52 | step: 19.10

  4%|▍         | 1586/41250 [3:49:59<95:42:45,  8.69s/it]
                                                         
{'loss': 0.1079, 'grad_norm': 1.4284602403640747, 'learning_rate': 3.999253464107089e-05, 'epoch': 0.38}

  4%|▍         | 1586/41250 [3:49:59<95:42:45,  8.69s/it][2025-04-26 07:58:36,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 07:58:36,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.26 | bwd_microstep: 5710.47 | bwd_inner_microstep: 5653.31 | bwd_allreduce_microstep: 57.11 | step_microstep: 18.84
[2025-04-26 07:58:36,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.26 | bwd: 5710.48 | bwd_inner: 5653.31 | bwd_allreduce: 57.13 | step: 18.85

  4%|▍         | 1587/41250 [3:50:07<95:29:03,  8.67s/it]
                                                         
{'loss': 0.0809, 'grad_norm': 1.0430599451065063, 'learning_rate': 3.9992491677753585e-05, 'epoch': 0.38}

  4%|▍         | 1587/41250 [3:50:07<95:29:03,  8.67s/it][2025-04-26 07:58:45,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 07:58:45,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.35 | bwd_microstep: 5742.96 | bwd_inner_microstep: 5691.52 | bwd_allreduce_microstep: 51.39 | step_microstep: 18.88
[2025-04-26 07:58:45,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.35 | bwd: 5742.97 | bwd_inner: 5691.52 | bwd_allreduce: 51.41 | step: 18.89

  4%|▍         | 1588/41250 [3:50:16<95:31:19,  8.67s/it]
                                                         
{'loss': 0.3402, 'grad_norm': 1.6515477895736694, 'learning_rate': 3.99924485911865e-05, 'epoch': 0.38}

  4%|▍         | 1588/41250 [3:50:16<95:31:19,  8.67s/it][2025-04-26 07:58:54,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-26 07:58:54,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5744.70 | bwd_inner_microstep: 5699.38 | bwd_allreduce_microstep: 45.28 | step_microstep: 19.06
[2025-04-26 07:58:54,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5744.72 | bwd_inner: 5699.38 | bwd_allreduce: 45.30 | step: 19.06

  4%|▍         | 1589/41250 [3:50:25<95:32:01,  8.67s/it]
                                                         
{'loss': 0.3006, 'grad_norm': 3.068065881729126, 'learning_rate': 3.999240538136991e-05, 'epoch': 0.39}

  4%|▍         | 1589/41250 [3:50:25<95:32:01,  8.67s/it][2025-04-26 07:59:02,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 07:59:02,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.06 | bwd_microstep: 5764.45 | bwd_inner_microstep: 5653.71 | bwd_allreduce_microstep: 110.69 | step_microstep: 18.56
[2025-04-26 07:59:02,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.06 | bwd: 5764.47 | bwd_inner: 5653.71 | bwd_allreduce: 110.71 | step: 18.57

  4%|▍         | 1590/41250 [3:50:33<95:31:55,  8.67s/it]
                                                         
{'loss': 0.4054, 'grad_norm': 2.275289297103882, 'learning_rate': 3.999236204830406e-05, 'epoch': 0.39}

  4%|▍         | 1590/41250 [3:50:33<95:31:55,  8.67s/it][2025-04-26 07:59:11,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 07:59:11,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.23 | bwd_microstep: 5788.40 | bwd_inner_microstep: 5696.78 | bwd_allreduce_microstep: 91.57 | step_microstep: 18.67
[2025-04-26 07:59:11,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.23 | bwd: 5788.41 | bwd_inner: 5696.78 | bwd_allreduce: 91.59 | step: 18.68

  4%|▍         | 1591/41250 [3:50:42<95:39:17,  8.68s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 2.2300758361816406, 'learning_rate': 3.999231859198924e-05, 'epoch': 0.39}

  4%|▍         | 1591/41250 [3:50:42<95:39:17,  8.68s/it][2025-04-26 07:59:20,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 07:59:20,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.90 | bwd_microstep: 5795.57 | bwd_inner_microstep: 5782.90 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.85
[2025-04-26 07:59:20,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.90 | bwd: 5795.58 | bwd_inner: 5782.90 | bwd_allreduce: 12.64 | step: 18.85

  4%|▍         | 1592/41250 [3:50:51<95:56:15,  8.71s/it]
                                                         
{'loss': 0.273, 'grad_norm': 2.9123454093933105, 'learning_rate': 3.999227501242571e-05, 'epoch': 0.39}

  4%|▍         | 1592/41250 [3:50:51<95:56:15,  8.71s/it][2025-04-26 07:59:28,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 07:59:28,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.75 | bwd_microstep: 5719.50 | bwd_inner_microstep: 5706.65 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.77
[2025-04-26 07:59:28,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.75 | bwd: 5719.52 | bwd_inner: 5706.65 | bwd_allreduce: 12.82 | step: 18.77

  4%|▍         | 1593/41250 [3:51:00<95:45:02,  8.69s/it]
                                                         
{'loss': 0.0778, 'grad_norm': 1.207398533821106, 'learning_rate': 3.9992231309613735e-05, 'epoch': 0.39}

  4%|▍         | 1593/41250 [3:51:00<95:45:02,  8.69s/it][2025-04-26 07:59:37,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 07:59:37,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5763.99 | bwd_inner_microstep: 5706.54 | bwd_allreduce_microstep: 57.41 | step_microstep: 18.74
[2025-04-26 07:59:37,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5764.01 | bwd_inner: 5706.54 | bwd_allreduce: 57.42 | step: 18.74

  4%|▍         | 1594/41250 [3:51:08<95:45:00,  8.69s/it]
                                                         
{'loss': 0.2416, 'grad_norm': 5.816880702972412, 'learning_rate': 3.9992187483553584e-05, 'epoch': 0.39}

  4%|▍         | 1594/41250 [3:51:08<95:45:00,  8.69s/it][2025-04-26 07:59:46,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 07:59:46,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.43 | bwd_microstep: 5724.13 | bwd_inner_microstep: 5711.31 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.78
[2025-04-26 07:59:46,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.43 | bwd: 5724.14 | bwd_inner: 5711.31 | bwd_allreduce: 12.79 | step: 18.79

  4%|▍         | 1595/41250 [3:51:17<95:35:56,  8.68s/it]
                                                         
{'loss': 0.2969, 'grad_norm': 2.527451515197754, 'learning_rate': 3.9992143534245524e-05, 'epoch': 0.39}

  4%|▍         | 1595/41250 [3:51:17<95:35:56,  8.68s/it][2025-04-26 07:59:54,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 07:59:54,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.71 | bwd_microstep: 5717.72 | bwd_inner_microstep: 5646.04 | bwd_allreduce_microstep: 71.63 | step_microstep: 18.78
[2025-04-26 07:59:54,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.71 | bwd: 5717.73 | bwd_inner: 5646.04 | bwd_allreduce: 71.65 | step: 18.78

  4%|▍         | 1596/41250 [3:51:25<95:25:11,  8.66s/it]
                                                         
{'loss': 0.2494, 'grad_norm': 4.280935764312744, 'learning_rate': 3.999209946168984e-05, 'epoch': 0.39}

  4%|▍         | 1596/41250 [3:51:25<95:25:11,  8.66s/it][2025-04-26 08:00:03,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 08:00:03,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5708.02 | bwd_inner_microstep: 5695.01 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.09
[2025-04-26 08:00:03,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5708.03 | bwd_inner: 5695.00 | bwd_allreduce: 12.98 | step: 19.09

  4%|▍         | 1597/41250 [3:51:34<95:20:14,  8.66s/it]
                                                         
{'loss': 0.1804, 'grad_norm': 2.514376163482666, 'learning_rate': 3.9992055265886796e-05, 'epoch': 0.39}

  4%|▍         | 1597/41250 [3:51:34<95:20:14,  8.66s/it][2025-04-26 08:00:12,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.04
[2025-04-26 08:00:12,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5789.19 | bwd_inner_microstep: 5696.79 | bwd_allreduce_microstep: 92.34 | step_microstep: 19.13
[2025-04-26 08:00:12,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5789.20 | bwd_inner: 5696.79 | bwd_allreduce: 92.36 | step: 19.14

  4%|▍         | 1598/41250 [3:51:43<95:32:29,  8.67s/it]
                                                         
{'loss': 0.1879, 'grad_norm': 1.782637119293213, 'learning_rate': 3.999201094683666e-05, 'epoch': 0.39}

  4%|▍         | 1598/41250 [3:51:43<95:32:29,  8.67s/it][2025-04-26 08:00:20,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:00:20,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.81 | bwd_microstep: 5715.25 | bwd_inner_microstep: 5702.47 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.87
[2025-04-26 08:00:20,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.81 | bwd: 5715.26 | bwd_inner: 5702.47 | bwd_allreduce: 12.75 | step: 18.88

  4%|▍         | 1599/41250 [3:51:51<95:26:27,  8.67s/it]
                                                         
{'loss': 0.0553, 'grad_norm': 1.4528071880340576, 'learning_rate': 3.999196650453971e-05, 'epoch': 0.39}

  4%|▍         | 1599/41250 [3:51:51<95:26:27,  8.67s/it][2025-04-26 08:00:29,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 08:00:29,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.80 | bwd_microstep: 5777.48 | bwd_inner_microstep: 5661.94 | bwd_allreduce_microstep: 115.49 | step_microstep: 18.95
[2025-04-26 08:00:29,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.80 | bwd: 5777.50 | bwd_inner: 5661.94 | bwd_allreduce: 115.51 | step: 18.95

  4%|▍         | 1600/41250 [3:52:00<95:29:34,  8.67s/it]
                                                         
{'loss': 0.3816, 'grad_norm': 3.059129476547241, 'learning_rate': 3.999192193899621e-05, 'epoch': 0.39}

  4%|▍         | 1600/41250 [3:52:00<95:29:34,  8.67s/it][2025-04-26 08:00:38,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:00:38,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.01 | bwd_microstep: 5780.95 | bwd_inner_microstep: 5657.64 | bwd_allreduce_microstep: 123.26 | step_microstep: 18.59
[2025-04-26 08:00:38,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.01 | bwd: 5780.97 | bwd_inner: 5657.64 | bwd_allreduce: 123.28 | step: 18.59

  4%|▍         | 1601/41250 [3:52:09<95:33:34,  8.68s/it]
                                                         
{'loss': 0.3796, 'grad_norm': 3.2101261615753174, 'learning_rate': 3.9991877250206454e-05, 'epoch': 0.39}

  4%|▍         | 1601/41250 [3:52:09<95:33:34,  8.68s/it][2025-04-26 08:00:46,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:00:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.60 | bwd_microstep: 5783.54 | bwd_inner_microstep: 5703.59 | bwd_allreduce_microstep: 79.90 | step_microstep: 18.87
[2025-04-26 08:00:46,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.60 | bwd: 5783.55 | bwd_inner: 5703.59 | bwd_allreduce: 79.92 | step: 18.87

  4%|▍         | 1602/41250 [3:52:18<95:41:17,  8.69s/it]
                                                         
{'loss': 0.2719, 'grad_norm': 1.588585615158081, 'learning_rate': 3.9991832438170706e-05, 'epoch': 0.39}

  4%|▍         | 1602/41250 [3:52:18<95:41:17,  8.69s/it][2025-04-26 08:00:55,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 08:00:55,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.85 | bwd_microstep: 5712.51 | bwd_inner_microstep: 5699.82 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.13
[2025-04-26 08:00:55,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5712.52 | bwd_inner: 5699.82 | bwd_allreduce: 12.66 | step: 19.14

  4%|▍         | 1603/41250 [3:52:26<95:31:54,  8.67s/it]
                                                         
{'loss': 0.1467, 'grad_norm': 1.5862641334533691, 'learning_rate': 3.999178750288924e-05, 'epoch': 0.39}

  4%|▍         | 1603/41250 [3:52:26<95:31:54,  8.67s/it][2025-04-26 08:01:04,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 08:01:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5792.41 | bwd_inner_microstep: 5645.62 | bwd_allreduce_microstep: 146.75 | step_microstep: 18.65
[2025-04-26 08:01:04,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5792.43 | bwd_inner: 5645.62 | bwd_allreduce: 146.76 | step: 18.65

  4%|▍         | 1604/41250 [3:52:35<95:35:48,  8.68s/it]
                                                         
{'loss': 0.281, 'grad_norm': 3.141896963119507, 'learning_rate': 3.9991742444362334e-05, 'epoch': 0.39}

  4%|▍         | 1604/41250 [3:52:35<95:35:48,  8.68s/it][2025-04-26 08:01:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 08:01:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.61 | bwd_microstep: 5761.79 | bwd_inner_microstep: 5683.00 | bwd_allreduce_microstep: 78.75 | step_microstep: 18.98
[2025-04-26 08:01:12,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.61 | bwd: 5761.81 | bwd_inner: 5683.00 | bwd_allreduce: 78.76 | step: 18.99

  4%|▍         | 1605/41250 [3:52:44<95:36:53,  8.68s/it]
                                                         
{'loss': 0.18, 'grad_norm': 1.0363930463790894, 'learning_rate': 3.9991697262590273e-05, 'epoch': 0.39}

  4%|▍         | 1605/41250 [3:52:44<95:36:53,  8.68s/it][2025-04-26 08:01:21,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 08:01:21,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.30 | bwd_microstep: 5783.12 | bwd_inner_microstep: 5699.95 | bwd_allreduce_microstep: 83.12 | step_microstep: 18.51
[2025-04-26 08:01:21,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.30 | bwd: 5783.13 | bwd_inner: 5699.95 | bwd_allreduce: 83.14 | step: 18.52

  4%|▍         | 1606/41250 [3:52:52<95:43:26,  8.69s/it]
                                                         
{'loss': 0.246, 'grad_norm': 2.150684118270874, 'learning_rate': 3.9991651957573327e-05, 'epoch': 0.39}

  4%|▍         | 1606/41250 [3:52:52<95:43:26,  8.69s/it][2025-04-26 08:01:30,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:01:30,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.44 | bwd_microstep: 5774.78 | bwd_inner_microstep: 5652.48 | bwd_allreduce_microstep: 122.26 | step_microstep: 18.61
[2025-04-26 08:01:30,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.44 | bwd: 5774.79 | bwd_inner: 5652.48 | bwd_allreduce: 122.27 | step: 18.62

  4%|▍         | 1607/41250 [3:53:01<95:41:22,  8.69s/it]
                                                         
{'loss': 0.2087, 'grad_norm': 1.7631099224090576, 'learning_rate': 3.9991606529311785e-05, 'epoch': 0.39}

  4%|▍         | 1607/41250 [3:53:01<95:41:22,  8.69s/it][2025-04-26 08:01:38,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:01:38,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.43 | bwd_microstep: 5718.46 | bwd_inner_microstep: 5654.58 | bwd_allreduce_microstep: 63.83 | step_microstep: 18.83
[2025-04-26 08:01:38,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.43 | bwd: 5718.47 | bwd_inner: 5654.58 | bwd_allreduce: 63.85 | step: 18.84

  4%|▍         | 1608/41250 [3:53:10<95:28:39,  8.67s/it]
                                                         
{'loss': 0.0545, 'grad_norm': 0.651067852973938, 'learning_rate': 3.999156097780591e-05, 'epoch': 0.39}

  4%|▍         | 1608/41250 [3:53:10<95:28:39,  8.67s/it][2025-04-26 08:01:47,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:01:47,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.94 | bwd_microstep: 5709.30 | bwd_inner_microstep: 5696.56 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.60
[2025-04-26 08:01:47,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.94 | bwd: 5709.32 | bwd_inner: 5696.56 | bwd_allreduce: 12.71 | step: 18.60

  4%|▍         | 1609/41250 [3:53:18<95:20:51,  8.66s/it]
                                                         
{'loss': 0.2212, 'grad_norm': 2.9337151050567627, 'learning_rate': 3.9991515303056e-05, 'epoch': 0.39}

  4%|▍         | 1609/41250 [3:53:18<95:20:51,  8.66s/it][2025-04-26 08:01:56,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 08:01:56,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5759.18 | bwd_inner_microstep: 5688.29 | bwd_allreduce_microstep: 70.85 | step_microstep: 19.28
[2025-04-26 08:01:56,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5759.20 | bwd_inner: 5688.29 | bwd_allreduce: 70.87 | step: 19.28

  4%|▍         | 1610/41250 [3:53:27<95:26:44,  8.67s/it]
                                                         
{'loss': 0.1696, 'grad_norm': 2.0715837478637695, 'learning_rate': 3.9991469505062335e-05, 'epoch': 0.39}

  4%|▍         | 1610/41250 [3:53:27<95:26:44,  8.67s/it][2025-04-26 08:02:05,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:02:05,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.88 | bwd_microstep: 5779.50 | bwd_inner_microstep: 5766.81 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.65
[2025-04-26 08:02:05,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.88 | bwd: 5779.51 | bwd_inner: 5766.81 | bwd_allreduce: 12.66 | step: 18.65

  4%|▍         | 1611/41250 [3:53:36<95:42:22,  8.69s/it]
                                                         
{'loss': 0.2668, 'grad_norm': 1.8503365516662598, 'learning_rate': 3.9991423583825185e-05, 'epoch': 0.39}

  4%|▍         | 1611/41250 [3:53:36<95:42:22,  8.69s/it][2025-04-26 08:02:13,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:02:13,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.08 | bwd_microstep: 5748.33 | bwd_inner_microstep: 5691.95 | bwd_allreduce_microstep: 56.33 | step_microstep: 18.73
[2025-04-26 08:02:13,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.08 | bwd: 5748.34 | bwd_inner: 5691.95 | bwd_allreduce: 56.35 | step: 18.74

  4%|▍         | 1612/41250 [3:53:44<95:40:16,  8.69s/it]
                                                         
{'loss': 0.0938, 'grad_norm': 1.0438812971115112, 'learning_rate': 3.999137753934484e-05, 'epoch': 0.39}

  4%|▍         | 1612/41250 [3:53:44<95:40:16,  8.69s/it][2025-04-26 08:02:22,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:02:22,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.48 | bwd_microstep: 5704.09 | bwd_inner_microstep: 5691.50 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.50
[2025-04-26 08:02:22,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.48 | bwd: 5704.10 | bwd_inner: 5691.50 | bwd_allreduce: 12.57 | step: 18.51

  4%|▍         | 1613/41250 [3:53:53<95:26:47,  8.67s/it]
                                                         
{'loss': 0.1705, 'grad_norm': 1.9457664489746094, 'learning_rate': 3.9991331371621587e-05, 'epoch': 0.39}

  4%|▍         | 1613/41250 [3:53:53<95:26:47,  8.67s/it][2025-04-26 08:02:30,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-26 08:02:30,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.40 | bwd_microstep: 5739.64 | bwd_inner_microstep: 5691.58 | bwd_allreduce_microstep: 48.00 | step_microstep: 20.02
[2025-04-26 08:02:30,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.40 | bwd: 5739.66 | bwd_inner: 5691.58 | bwd_allreduce: 48.03 | step: 20.03

  4%|▍         | 1614/41250 [3:54:02<95:26:57,  8.67s/it]
                                                         
{'loss': 0.2443, 'grad_norm': 1.6654198169708252, 'learning_rate': 3.9991285080655714e-05, 'epoch': 0.39}

  4%|▍         | 1614/41250 [3:54:02<95:26:57,  8.67s/it][2025-04-26 08:02:39,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.20 | optimizer_step: 0.94
[2025-04-26 08:02:39,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.31 | bwd_microstep: 5713.04 | bwd_inner_microstep: 5700.20 | bwd_allreduce_microstep: 12.80 | step_microstep: 19.27
[2025-04-26 08:02:39,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.31 | bwd: 5713.06 | bwd_inner: 5700.20 | bwd_allreduce: 12.82 | step: 19.27

  4%|▍         | 1615/41250 [3:54:10<95:21:51,  8.66s/it]
                                                         
{'loss': 0.0474, 'grad_norm': 0.5412636399269104, 'learning_rate': 3.9991238666447496e-05, 'epoch': 0.39}

  4%|▍         | 1615/41250 [3:54:10<95:21:51,  8.66s/it][2025-04-26 08:02:48,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 08:02:48,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.58 | bwd_microstep: 5789.39 | bwd_inner_microstep: 5650.52 | bwd_allreduce_microstep: 138.83 | step_microstep: 18.68
[2025-04-26 08:02:48,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5789.41 | bwd_inner: 5650.52 | bwd_allreduce: 138.85 | step: 18.68

  4%|▍         | 1616/41250 [3:54:19<95:27:54,  8.67s/it]
                                                         
{'loss': 0.2304, 'grad_norm': 1.488632321357727, 'learning_rate': 3.9991192128997225e-05, 'epoch': 0.39}

  4%|▍         | 1616/41250 [3:54:19<95:27:54,  8.67s/it][2025-04-26 08:02:56,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:02:56,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.97 | bwd_microstep: 5690.51 | bwd_inner_microstep: 5639.42 | bwd_allreduce_microstep: 51.05 | step_microstep: 18.35
[2025-04-26 08:02:56,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.97 | bwd: 5690.52 | bwd_inner: 5639.42 | bwd_allreduce: 51.06 | step: 18.36

  4%|▍         | 1617/41250 [3:54:28<95:11:37,  8.65s/it]
                                                         
{'loss': 0.286, 'grad_norm': 2.4736897945404053, 'learning_rate': 3.9991145468305184e-05, 'epoch': 0.39}

  4%|▍         | 1617/41250 [3:54:28<95:11:37,  8.65s/it][2025-04-26 08:03:05,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:03:05,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.16 | bwd_microstep: 5705.66 | bwd_inner_microstep: 5692.96 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.55
[2025-04-26 08:03:05,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.16 | bwd: 5705.67 | bwd_inner: 5692.96 | bwd_allreduce: 12.67 | step: 18.56

  4%|▍         | 1618/41250 [3:54:36<95:09:01,  8.64s/it]
                                                         
{'loss': 0.2077, 'grad_norm': 1.5263231992721558, 'learning_rate': 3.9991098684371666e-05, 'epoch': 0.39}

  4%|▍         | 1618/41250 [3:54:36<95:09:01,  8.64s/it][2025-04-26 08:03:14,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:03:14,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.95 | bwd_microstep: 5690.98 | bwd_inner_microstep: 5670.42 | bwd_allreduce_microstep: 20.51 | step_microstep: 18.48
[2025-04-26 08:03:14,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.95 | bwd: 5690.99 | bwd_inner: 5670.42 | bwd_allreduce: 20.53 | step: 18.49

  4%|▍         | 1619/41250 [3:54:45<95:04:02,  8.64s/it]
                                                         
{'loss': 0.3751, 'grad_norm': 2.0248560905456543, 'learning_rate': 3.9991051777196954e-05, 'epoch': 0.39}

  4%|▍         | 1619/41250 [3:54:45<95:04:02,  8.64s/it][2025-04-26 08:03:22,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-26 08:03:22,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.38 | bwd_microstep: 5730.35 | bwd_inner_microstep: 5701.07 | bwd_allreduce_microstep: 29.21 | step_microstep: 20.07
[2025-04-26 08:03:22,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.38 | bwd: 5730.37 | bwd_inner: 5701.07 | bwd_allreduce: 29.24 | step: 20.07

  4%|▍         | 1620/41250 [3:54:53<95:08:44,  8.64s/it]
                                                         
{'loss': 0.1107, 'grad_norm': 1.3296513557434082, 'learning_rate': 3.999100474678134e-05, 'epoch': 0.39}

  4%|▍         | 1620/41250 [3:54:54<95:08:44,  8.64s/it][2025-04-26 08:03:31,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:03:31,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.51 | bwd_microstep: 5704.53 | bwd_inner_microstep: 5639.03 | bwd_allreduce_microstep: 65.46 | step_microstep: 18.57
[2025-04-26 08:03:31,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.51 | bwd: 5704.54 | bwd_inner: 5639.03 | bwd_allreduce: 65.47 | step: 18.57

  4%|▍         | 1621/41250 [3:55:02<95:02:46,  8.63s/it]
                                                         
{'loss': 0.0878, 'grad_norm': 1.5887815952301025, 'learning_rate': 3.999095759312512e-05, 'epoch': 0.39}

  4%|▍         | 1621/41250 [3:55:02<95:02:46,  8.63s/it][2025-04-26 08:03:40,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 0.94
[2025-04-26 08:03:40,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.93 | bwd_microstep: 5751.09 | bwd_inner_microstep: 5658.60 | bwd_allreduce_microstep: 92.44 | step_microstep: 19.00
[2025-04-26 08:03:40,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.93 | bwd: 5751.10 | bwd_inner: 5658.60 | bwd_allreduce: 92.45 | step: 19.01

  4%|▍         | 1622/41250 [3:55:11<95:04:28,  8.64s/it]
                                                         
{'loss': 0.2338, 'grad_norm': 1.6094472408294678, 'learning_rate': 3.999091031622857e-05, 'epoch': 0.39}

  4%|▍         | 1622/41250 [3:55:11<95:04:28,  8.64s/it][2025-04-26 08:03:48,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:03:48,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.08 | bwd_microstep: 5687.97 | bwd_inner_microstep: 5638.72 | bwd_allreduce_microstep: 49.20 | step_microstep: 18.97
[2025-04-26 08:03:48,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.08 | bwd: 5687.99 | bwd_inner: 5638.72 | bwd_allreduce: 49.22 | step: 18.98

  4%|▍         | 1623/41250 [3:55:19<94:56:41,  8.63s/it]
                                                         
{'loss': 0.3167, 'grad_norm': 2.626150608062744, 'learning_rate': 3.999086291609199e-05, 'epoch': 0.39}

  4%|▍         | 1623/41250 [3:55:19<94:56:41,  8.63s/it][2025-04-26 08:03:57,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 08:03:57,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.33 | bwd_microstep: 5768.33 | bwd_inner_microstep: 5646.29 | bwd_allreduce_microstep: 121.98 | step_microstep: 18.80
[2025-04-26 08:03:57,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.33 | bwd: 5768.34 | bwd_inner: 5646.29 | bwd_allreduce: 122.00 | step: 18.80

  4%|▍         | 1624/41250 [3:55:28<95:07:45,  8.64s/it]
                                                         
{'loss': 0.3181, 'grad_norm': 1.9234633445739746, 'learning_rate': 3.999081539271568e-05, 'epoch': 0.39}

  4%|▍         | 1624/41250 [3:55:28<95:07:45,  8.64s/it][2025-04-26 08:04:06,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 08:04:06,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.67 | bwd_microstep: 5778.32 | bwd_inner_microstep: 5633.61 | bwd_allreduce_microstep: 144.67 | step_microstep: 19.06
[2025-04-26 08:04:06,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.67 | bwd: 5778.34 | bwd_inner: 5633.61 | bwd_allreduce: 144.69 | step: 19.07

  4%|▍         | 1625/41250 [3:55:37<95:15:06,  8.65s/it]
                                                         
{'loss': 0.2596, 'grad_norm': 2.5660924911499023, 'learning_rate': 3.999076774609992e-05, 'epoch': 0.39}

  4%|▍         | 1625/41250 [3:55:37<95:15:06,  8.65s/it][2025-04-26 08:04:14,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 08:04:14,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.74 | bwd_microstep: 5689.39 | bwd_inner_microstep: 5646.57 | bwd_allreduce_microstep: 42.77 | step_microstep: 18.79
[2025-04-26 08:04:14,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.74 | bwd: 5689.40 | bwd_inner: 5646.57 | bwd_allreduce: 42.79 | step: 18.80

  4%|▍         | 1626/41250 [3:55:45<95:03:10,  8.64s/it]
                                                         
{'loss': 0.2296, 'grad_norm': 1.5839258432388306, 'learning_rate': 3.9990719976245006e-05, 'epoch': 0.39}

  4%|▍         | 1626/41250 [3:55:45<95:03:10,  8.64s/it][2025-04-26 08:04:23,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.21 | optimizer_step: 0.93
[2025-04-26 08:04:23,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.38 | bwd_microstep: 5771.94 | bwd_inner_microstep: 5647.32 | bwd_allreduce_microstep: 124.57 | step_microstep: 19.54
[2025-04-26 08:04:23,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.38 | bwd: 5771.95 | bwd_inner: 5647.32 | bwd_allreduce: 124.58 | step: 19.54

  4%|▍         | 1627/41250 [3:55:54<95:10:10,  8.65s/it]
                                                         
{'loss': 0.1624, 'grad_norm': 1.6409492492675781, 'learning_rate': 3.999067208315124e-05, 'epoch': 0.39}

  4%|▍         | 1627/41250 [3:55:54<95:10:10,  8.65s/it][2025-04-26 08:04:31,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 08:04:31,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.59 | bwd_microstep: 5711.07 | bwd_inner_microstep: 5698.05 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.59
[2025-04-26 08:04:31,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.59 | bwd: 5711.09 | bwd_inner: 5698.05 | bwd_allreduce: 12.98 | step: 18.60

  4%|▍         | 1628/41250 [3:56:03<95:10:05,  8.65s/it]
                                                         
{'loss': 0.1811, 'grad_norm': 1.0496660470962524, 'learning_rate': 3.999062406681891e-05, 'epoch': 0.39}

  4%|▍         | 1628/41250 [3:56:03<95:10:05,  8.65s/it][2025-04-26 08:04:40,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:04:40,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.12 | bwd_microstep: 5858.97 | bwd_inner_microstep: 5846.34 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.44
[2025-04-26 08:04:40,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.12 | bwd: 5858.98 | bwd_inner: 5846.34 | bwd_allreduce: 12.61 | step: 18.44

  4%|▍         | 1629/41250 [3:56:12<95:54:42,  8.71s/it]
                                                         
{'loss': 0.1335, 'grad_norm': 1.836305022239685, 'learning_rate': 3.999057592724831e-05, 'epoch': 0.39}

  4%|▍         | 1629/41250 [3:56:12<95:54:42,  8.71s/it][2025-04-26 08:04:49,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 08:04:49,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.64 | bwd_microstep: 5708.84 | bwd_inner_microstep: 5653.07 | bwd_allreduce_microstep: 55.72 | step_microstep: 19.37
[2025-04-26 08:04:49,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.64 | bwd: 5708.85 | bwd_inner: 5653.07 | bwd_allreduce: 55.74 | step: 19.37

  4%|▍         | 1630/41250 [3:56:20<95:33:46,  8.68s/it]
                                                         
{'loss': 0.2743, 'grad_norm': 2.4043147563934326, 'learning_rate': 3.999052766443975e-05, 'epoch': 0.4}

  4%|▍         | 1630/41250 [3:56:20<95:33:46,  8.68s/it][2025-04-26 08:04:58,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 08:04:58,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.16 | bwd_microstep: 5703.06 | bwd_inner_microstep: 5651.01 | bwd_allreduce_microstep: 52.00 | step_microstep: 18.69
[2025-04-26 08:04:58,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.16 | bwd: 5703.08 | bwd_inner: 5651.01 | bwd_allreduce: 52.02 | step: 18.70

  4%|▍         | 1631/41250 [3:56:29<95:18:20,  8.66s/it]
                                                         
{'loss': 0.2034, 'grad_norm': 1.1853333711624146, 'learning_rate': 3.999047927839352e-05, 'epoch': 0.4}

  4%|▍         | 1631/41250 [3:56:29<95:18:20,  8.66s/it][2025-04-26 08:05:06,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:05:06,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.12 | bwd_microstep: 5762.80 | bwd_inner_microstep: 5643.13 | bwd_allreduce_microstep: 119.63 | step_microstep: 18.38
[2025-04-26 08:05:06,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.12 | bwd: 5762.81 | bwd_inner: 5643.13 | bwd_allreduce: 119.65 | step: 18.38

  4%|▍         | 1632/41250 [3:56:37<95:20:51,  8.66s/it]
                                                         
{'loss': 0.181, 'grad_norm': 1.2903213500976562, 'learning_rate': 3.9990430769109905e-05, 'epoch': 0.4}

  4%|▍         | 1632/41250 [3:56:37<95:20:51,  8.66s/it][2025-04-26 08:05:15,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 08:05:15,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.79 | bwd_microstep: 5770.78 | bwd_inner_microstep: 5706.79 | bwd_allreduce_microstep: 63.94 | step_microstep: 18.75
[2025-04-26 08:05:15,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.79 | bwd: 5770.79 | bwd_inner: 5706.79 | bwd_allreduce: 63.96 | step: 18.75

  4%|▍         | 1633/41250 [3:56:46<95:28:37,  8.68s/it]
                                                         
{'loss': 0.1866, 'grad_norm': 1.7699191570281982, 'learning_rate': 3.9990382136589226e-05, 'epoch': 0.4}

  4%|▍         | 1633/41250 [3:56:46<95:28:37,  8.68s/it][2025-04-26 08:05:24,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 08:05:24,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.27 | bwd_microstep: 5694.05 | bwd_inner_microstep: 5681.26 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.20
[2025-04-26 08:05:24,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.27 | bwd: 5694.07 | bwd_inner: 5681.26 | bwd_allreduce: 12.77 | step: 19.20

  4%|▍         | 1634/41250 [3:56:55<95:17:09,  8.66s/it]
                                                         
{'loss': 0.2002, 'grad_norm': 2.132615566253662, 'learning_rate': 3.999033338083177e-05, 'epoch': 0.4}

  4%|▍         | 1634/41250 [3:56:55<95:17:09,  8.66s/it][2025-04-26 08:05:32,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:05:32,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.08 | bwd_microstep: 5773.00 | bwd_inner_microstep: 5634.57 | bwd_allreduce_microstep: 138.38 | step_microstep: 18.14
[2025-04-26 08:05:32,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.08 | bwd: 5773.01 | bwd_inner: 5634.57 | bwd_allreduce: 138.40 | step: 18.14

  4%|▍         | 1635/41250 [3:57:03<95:19:51,  8.66s/it]
                                                         
{'loss': 0.2233, 'grad_norm': 1.8224610090255737, 'learning_rate': 3.999028450183784e-05, 'epoch': 0.4}

  4%|▍         | 1635/41250 [3:57:03<95:19:51,  8.66s/it][2025-04-26 08:05:41,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:05:41,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5708.57 | bwd_inner_microstep: 5695.56 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.85
[2025-04-26 08:05:41,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5708.58 | bwd_inner: 5695.56 | bwd_allreduce: 12.98 | step: 18.85

  4%|▍         | 1636/41250 [3:57:12<95:14:41,  8.66s/it]
                                                         
{'loss': 0.35, 'grad_norm': 2.1359786987304688, 'learning_rate': 3.9990235499607745e-05, 'epoch': 0.4}

  4%|▍         | 1636/41250 [3:57:12<95:14:41,  8.66s/it][2025-04-26 08:05:50,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:05:50,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.22 | bwd_microstep: 5778.49 | bwd_inner_microstep: 5646.90 | bwd_allreduce_microstep: 131.55 | step_microstep: 18.29
[2025-04-26 08:05:50,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.22 | bwd: 5778.51 | bwd_inner: 5646.90 | bwd_allreduce: 131.57 | step: 18.29

  4%|▍         | 1637/41250 [3:57:21<95:22:36,  8.67s/it]
                                                         
{'loss': 0.1868, 'grad_norm': 3.9043030738830566, 'learning_rate': 3.9990186374141775e-05, 'epoch': 0.4}

  4%|▍         | 1637/41250 [3:57:21<95:22:36,  8.67s/it][2025-04-26 08:05:58,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:05:58,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.75 | bwd_microstep: 5706.01 | bwd_inner_microstep: 5693.10 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.32
[2025-04-26 08:05:58,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.75 | bwd: 5706.02 | bwd_inner: 5693.10 | bwd_allreduce: 12.89 | step: 18.33

  4%|▍         | 1638/41250 [3:57:29<95:15:48,  8.66s/it]
                                                         
{'loss': 0.2778, 'grad_norm': 1.563856840133667, 'learning_rate': 3.9990137125440236e-05, 'epoch': 0.4}

  4%|▍         | 1638/41250 [3:57:29<95:15:48,  8.66s/it][2025-04-26 08:06:07,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 08:06:07,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.36 | bwd_microstep: 5789.10 | bwd_inner_microstep: 5651.93 | bwd_allreduce_microstep: 137.12 | step_microstep: 19.15
[2025-04-26 08:06:07,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.36 | bwd: 5789.11 | bwd_inner: 5651.93 | bwd_allreduce: 137.14 | step: 19.16

  4%|▍         | 1639/41250 [3:57:38<95:23:50,  8.67s/it]
                                                         
{'loss': 0.1815, 'grad_norm': 1.0954951047897339, 'learning_rate': 3.999008775350343e-05, 'epoch': 0.4}

  4%|▍         | 1639/41250 [3:57:38<95:23:50,  8.67s/it][2025-04-26 08:06:16,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:06:16,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5791.23 | bwd_inner_microstep: 5640.56 | bwd_allreduce_microstep: 150.63 | step_microstep: 18.42
[2025-04-26 08:06:16,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.93 | bwd: 5791.25 | bwd_inner: 5640.56 | bwd_allreduce: 150.65 | step: 18.42

  4%|▍         | 1640/41250 [3:57:47<95:27:13,  8.68s/it]
                                                         
{'loss': 0.1412, 'grad_norm': 2.331078290939331, 'learning_rate': 3.999003825833168e-05, 'epoch': 0.4}

  4%|▍         | 1640/41250 [3:57:47<95:27:13,  8.68s/it][2025-04-26 08:06:24,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:06:24,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.33 | bwd_microstep: 5805.55 | bwd_inner_microstep: 5655.16 | bwd_allreduce_microstep: 150.34 | step_microstep: 18.44
[2025-04-26 08:06:24,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.33 | bwd: 5805.56 | bwd_inner: 5655.16 | bwd_allreduce: 150.36 | step: 18.44

  4%|▍         | 1641/41250 [3:57:55<95:35:34,  8.69s/it]
                                                         
{'loss': 0.1021, 'grad_norm': 1.2777597904205322, 'learning_rate': 3.998998863992526e-05, 'epoch': 0.4}

  4%|▍         | 1641/41250 [3:57:55<95:35:34,  8.69s/it][2025-04-26 08:06:33,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 08:06:33,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5894.97 | bwd_inner_microstep: 5650.74 | bwd_allreduce_microstep: 244.18 | step_microstep: 18.64
[2025-04-26 08:06:33,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5894.98 | bwd_inner: 5650.74 | bwd_allreduce: 244.20 | step: 18.64

  4%|▍         | 1642/41250 [3:58:04<95:58:39,  8.72s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 1.1184895038604736, 'learning_rate': 3.99899388982845e-05, 'epoch': 0.4}

  4%|▍         | 1642/41250 [3:58:04<95:58:39,  8.72s/it][2025-04-26 08:06:42,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.63 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:06:42,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.62 | bwd_microstep: 5767.73 | bwd_inner_microstep: 5659.91 | bwd_allreduce_microstep: 107.78 | step_microstep: 18.81
[2025-04-26 08:06:42,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.62 | bwd: 5767.74 | bwd_inner: 5659.91 | bwd_allreduce: 107.79 | step: 18.81

  4%|▍         | 1643/41250 [3:58:13<95:49:49,  8.71s/it]
                                                         
{'loss': 0.1247, 'grad_norm': 2.7672839164733887, 'learning_rate': 3.9989889033409695e-05, 'epoch': 0.4}

  4%|▍         | 1643/41250 [3:58:13<95:49:49,  8.71s/it][2025-04-26 08:06:50,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:06:50,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.54 | bwd_microstep: 5736.10 | bwd_inner_microstep: 5646.50 | bwd_allreduce_microstep: 89.56 | step_microstep: 18.77
[2025-04-26 08:06:50,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.54 | bwd: 5736.12 | bwd_inner: 5646.50 | bwd_allreduce: 89.58 | step: 18.78

  4%|▍         | 1644/41250 [3:58:22<95:36:39,  8.69s/it]
                                                         
{'loss': 0.1379, 'grad_norm': 1.8720746040344238, 'learning_rate': 3.998983904530115e-05, 'epoch': 0.4}

  4%|▍         | 1644/41250 [3:58:22<95:36:39,  8.69s/it][2025-04-26 08:06:59,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.03 | optimizer_step: 1.10
[2025-04-26 08:06:59,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.84 | bwd_microstep: 5917.94 | bwd_inner_microstep: 5654.66 | bwd_allreduce_microstep: 263.24 | step_microstep: 19.26
[2025-04-26 08:06:59,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.84 | bwd: 5917.96 | bwd_inner: 5654.66 | bwd_allreduce: 263.26 | step: 19.26

  4%|▍         | 1645/41250 [3:58:30<96:04:23,  8.73s/it]
                                                         
{'loss': 0.2646, 'grad_norm': 2.1976635456085205, 'learning_rate': 3.998978893395919e-05, 'epoch': 0.4}

  4%|▍         | 1645/41250 [3:58:30<96:04:23,  8.73s/it][2025-04-26 08:07:08,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:07:08,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.92 | bwd_microstep: 5775.66 | bwd_inner_microstep: 5715.05 | bwd_allreduce_microstep: 60.57 | step_microstep: 18.78
[2025-04-26 08:07:08,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.92 | bwd: 5775.68 | bwd_inner: 5715.05 | bwd_allreduce: 60.58 | step: 18.78

  4%|▍         | 1646/41250 [3:58:39<95:59:55,  8.73s/it]
                                                         
{'loss': 0.1007, 'grad_norm': 0.8012254238128662, 'learning_rate': 3.998973869938411e-05, 'epoch': 0.4}

  4%|▍         | 1646/41250 [3:58:39<95:59:55,  8.73s/it][2025-04-26 08:07:17,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:07:17,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.14 | bwd_microstep: 5718.82 | bwd_inner_microstep: 5705.96 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.48
[2025-04-26 08:07:17,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.14 | bwd: 5718.84 | bwd_inner: 5705.96 | bwd_allreduce: 12.83 | step: 18.48

  4%|▍         | 1647/41250 [3:58:48<95:45:04,  8.70s/it]
                                                         
{'loss': 0.0542, 'grad_norm': 0.6151787042617798, 'learning_rate': 3.998968834157622e-05, 'epoch': 0.4}

  4%|▍         | 1647/41250 [3:58:48<95:45:04,  8.70s/it][2025-04-26 08:07:25,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 08:07:25,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.63 | bwd_microstep: 5821.41 | bwd_inner_microstep: 5654.64 | bwd_allreduce_microstep: 166.72 | step_microstep: 18.59
[2025-04-26 08:07:25,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.63 | bwd: 5821.42 | bwd_inner: 5654.64 | bwd_allreduce: 166.75 | step: 18.59

  4%|▍         | 1648/41250 [3:58:57<95:50:16,  8.71s/it]
                                                         
{'loss': 0.164, 'grad_norm': 1.8317887783050537, 'learning_rate': 3.9989637860535834e-05, 'epoch': 0.4}

  4%|▍         | 1648/41250 [3:58:57<95:50:16,  8.71s/it][2025-04-26 08:07:34,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:07:34,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.24 | bwd_microstep: 5808.27 | bwd_inner_microstep: 5642.47 | bwd_allreduce_microstep: 165.76 | step_microstep: 18.56
[2025-04-26 08:07:34,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.24 | bwd: 5808.29 | bwd_inner: 5642.47 | bwd_allreduce: 165.78 | step: 18.56

  4%|▍         | 1649/41250 [3:59:05<95:47:52,  8.71s/it]
                                                         
{'loss': 0.1256, 'grad_norm': 0.9398114085197449, 'learning_rate': 3.9989587256263264e-05, 'epoch': 0.4}

  4%|▍         | 1649/41250 [3:59:05<95:47:52,  8.71s/it][2025-04-26 08:07:43,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 08:07:43,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.58 | bwd_microstep: 5781.31 | bwd_inner_microstep: 5768.67 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.73
[2025-04-26 08:07:43,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.58 | bwd: 5781.32 | bwd_inner: 5768.67 | bwd_allreduce: 12.60 | step: 18.73

  4%|▍         | 1650/41250 [3:59:14<95:55:49,  8.72s/it]
                                                         
{'loss': 0.3248, 'grad_norm': 2.206522226333618, 'learning_rate': 3.998953652875882e-05, 'epoch': 0.4}

  4%|▍         | 1650/41250 [3:59:14<95:55:49,  8.72s/it][2025-04-26 08:07:51,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:07:51,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.12 | bwd_microstep: 5770.65 | bwd_inner_microstep: 5685.34 | bwd_allreduce_microstep: 85.27 | step_microstep: 18.55
[2025-04-26 08:07:51,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.12 | bwd: 5770.67 | bwd_inner: 5685.34 | bwd_allreduce: 85.28 | step: 18.56

  4%|▍         | 1651/41250 [3:59:23<95:51:09,  8.71s/it]
                                                         
{'loss': 0.0953, 'grad_norm': 2.416456699371338, 'learning_rate': 3.9989485678022816e-05, 'epoch': 0.4}

  4%|▍         | 1651/41250 [3:59:23<95:51:09,  8.71s/it][2025-04-26 08:08:00,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:08:00,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.20 | bwd_microstep: 5696.81 | bwd_inner_microstep: 5647.93 | bwd_allreduce_microstep: 48.84 | step_microstep: 18.61
[2025-04-26 08:08:00,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.20 | bwd: 5696.83 | bwd_inner: 5647.93 | bwd_allreduce: 48.86 | step: 18.61

  4%|▍         | 1652/41250 [3:59:31<95:30:18,  8.68s/it]
                                                         
{'loss': 0.2378, 'grad_norm': 1.6885015964508057, 'learning_rate': 3.998943470405556e-05, 'epoch': 0.4}

  4%|▍         | 1652/41250 [3:59:31<95:30:18,  8.68s/it][2025-04-26 08:08:09,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.02
[2025-04-26 08:08:09,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.53 | bwd_microstep: 5804.12 | bwd_inner_microstep: 5652.45 | bwd_allreduce_microstep: 151.61 | step_microstep: 19.48
[2025-04-26 08:08:09,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.53 | bwd: 5804.14 | bwd_inner: 5652.45 | bwd_allreduce: 151.64 | step: 19.48

  4%|▍         | 1653/41250 [3:59:40<95:36:08,  8.69s/it]
                                                         
{'loss': 0.1748, 'grad_norm': 3.9183340072631836, 'learning_rate': 3.998938360685737e-05, 'epoch': 0.4}

  4%|▍         | 1653/41250 [3:59:40<95:36:08,  8.69s/it][2025-04-26 08:08:17,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-26 08:08:17,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.82 | bwd_microstep: 5709.88 | bwd_inner_microstep: 5696.74 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.85
[2025-04-26 08:08:17,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5709.89 | bwd_inner: 5696.74 | bwd_allreduce: 13.11 | step: 18.85

  4%|▍         | 1654/41250 [3:59:49<95:25:05,  8.68s/it]
                                                         
{'loss': 0.0332, 'grad_norm': 0.40943828225135803, 'learning_rate': 3.998933238642857e-05, 'epoch': 0.4}

  4%|▍         | 1654/41250 [3:59:49<95:25:05,  8.68s/it][2025-04-26 08:08:26,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:08:26,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.84 | bwd_microstep: 5777.00 | bwd_inner_microstep: 5642.91 | bwd_allreduce_microstep: 134.05 | step_microstep: 18.39
[2025-04-26 08:08:26,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.84 | bwd: 5777.01 | bwd_inner: 5642.91 | bwd_allreduce: 134.07 | step: 18.40

  4%|▍         | 1655/41250 [3:59:57<95:25:20,  8.68s/it]
                                                         
{'loss': 0.0956, 'grad_norm': 0.8478343486785889, 'learning_rate': 3.9989281042769467e-05, 'epoch': 0.4}

  4%|▍         | 1655/41250 [3:59:57<95:25:20,  8.68s/it][2025-04-26 08:08:35,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:08:35,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.94 | bwd_microstep: 5769.92 | bwd_inner_microstep: 5692.61 | bwd_allreduce_microstep: 77.27 | step_microstep: 18.05
[2025-04-26 08:08:35,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.94 | bwd: 5769.94 | bwd_inner: 5692.61 | bwd_allreduce: 77.29 | step: 18.05

  4%|▍         | 1656/41250 [4:00:06<95:28:35,  8.68s/it]
                                                         
{'loss': 0.3892, 'grad_norm': 2.4258780479431152, 'learning_rate': 3.998922957588037e-05, 'epoch': 0.4}

  4%|▍         | 1656/41250 [4:00:06<95:28:35,  8.68s/it][2025-04-26 08:08:44,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:08:44,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5909.70 | bwd_inner_microstep: 5648.56 | bwd_allreduce_microstep: 261.09 | step_microstep: 18.40
[2025-04-26 08:08:44,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5909.71 | bwd_inner: 5648.56 | bwd_allreduce: 261.11 | step: 18.41

  4%|▍         | 1657/41250 [4:00:15<95:56:20,  8.72s/it]
                                                         
{'loss': 0.0464, 'grad_norm': 0.5734847187995911, 'learning_rate': 3.9989177985761606e-05, 'epoch': 0.4}

  4%|▍         | 1657/41250 [4:00:15<95:56:20,  8.72s/it][2025-04-26 08:08:52,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 08:08:52,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5696.18 | bwd_inner_microstep: 5683.42 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.19
[2025-04-26 08:08:52,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5696.19 | bwd_inner: 5683.42 | bwd_allreduce: 12.73 | step: 18.20

  4%|▍         | 1658/41250 [4:00:23<95:33:36,  8.69s/it]
                                                         
{'loss': 0.3051, 'grad_norm': 2.906742572784424, 'learning_rate': 3.99891262724135e-05, 'epoch': 0.4}

  4%|▍         | 1658/41250 [4:00:23<95:33:36,  8.69s/it][2025-04-26 08:09:01,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 08:09:01,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.95 | bwd_microstep: 5705.51 | bwd_inner_microstep: 5649.57 | bwd_allreduce_microstep: 55.90 | step_microstep: 19.06
[2025-04-26 08:09:01,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.95 | bwd: 5705.53 | bwd_inner: 5649.57 | bwd_allreduce: 55.92 | step: 19.06

  4%|▍         | 1659/41250 [4:00:32<95:18:15,  8.67s/it]
                                                         
{'loss': 0.2082, 'grad_norm': 2.118532180786133, 'learning_rate': 3.998907443583635e-05, 'epoch': 0.4}

  4%|▍         | 1659/41250 [4:00:32<95:18:15,  8.67s/it][2025-04-26 08:09:09,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 08:09:09,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.89 | bwd_microstep: 5727.47 | bwd_inner_microstep: 5642.37 | bwd_allreduce_microstep: 85.05 | step_microstep: 18.88
[2025-04-26 08:09:09,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.89 | bwd: 5727.48 | bwd_inner: 5642.37 | bwd_allreduce: 85.07 | step: 18.88

  4%|▍         | 1660/41250 [4:00:41<95:11:05,  8.66s/it]
                                                         
{'loss': 0.1705, 'grad_norm': 5.363543510437012, 'learning_rate': 3.99890224760305e-05, 'epoch': 0.4}

  4%|▍         | 1660/41250 [4:00:41<95:11:05,  8.66s/it][2025-04-26 08:09:18,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.26 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:09:18,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.29 | bwd_microstep: 5738.44 | bwd_inner_microstep: 5689.00 | bwd_allreduce_microstep: 49.39 | step_microstep: 19.49
[2025-04-26 08:09:18,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.29 | bwd: 5738.46 | bwd_inner: 5689.00 | bwd_allreduce: 49.41 | step: 19.50

  4%|▍         | 1661/41250 [4:00:49<95:15:23,  8.66s/it]
                                                         
{'loss': 0.1616, 'grad_norm': 1.7592010498046875, 'learning_rate': 3.998897039299625e-05, 'epoch': 0.4}

  4%|▍         | 1661/41250 [4:00:49<95:15:23,  8.66s/it][2025-04-26 08:09:27,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 08:09:27,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.19 | bwd_microstep: 5705.73 | bwd_inner_microstep: 5692.77 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.84
[2025-04-26 08:09:27,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.20 | bwd: 5705.75 | bwd_inner: 5692.77 | bwd_allreduce: 12.94 | step: 18.85

  4%|▍         | 1662/41250 [4:00:58<95:13:21,  8.66s/it]
                                                         
{'loss': 0.1863, 'grad_norm': 4.435357570648193, 'learning_rate': 3.998891818673393e-05, 'epoch': 0.4}

  4%|▍         | 1662/41250 [4:00:58<95:13:21,  8.66s/it][2025-04-26 08:09:35,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 08:09:35,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.36 | bwd_microstep: 5697.15 | bwd_inner_microstep: 5637.23 | bwd_allreduce_microstep: 59.88 | step_microstep: 18.73
[2025-04-26 08:09:35,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.36 | bwd: 5697.17 | bwd_inner: 5637.23 | bwd_allreduce: 59.89 | step: 18.74

  4%|▍         | 1663/41250 [4:01:07<95:00:17,  8.64s/it]
                                                         
{'loss': 0.13, 'grad_norm': 1.7352770566940308, 'learning_rate': 3.9988865857243865e-05, 'epoch': 0.4}

  4%|▍         | 1663/41250 [4:01:07<95:00:17,  8.64s/it][2025-04-26 08:09:44,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.31 | optimizer_step: 1.06
[2025-04-26 08:09:44,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.73 | bwd_microstep: 5697.42 | bwd_inner_microstep: 5645.42 | bwd_allreduce_microstep: 51.93 | step_microstep: 20.02
[2025-04-26 08:09:44,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.73 | bwd: 5697.43 | bwd_inner: 5645.42 | bwd_allreduce: 51.96 | step: 20.03

  4%|▍         | 1664/41250 [4:01:15<94:53:33,  8.63s/it]
                                                         
{'loss': 0.1869, 'grad_norm': 1.7056076526641846, 'learning_rate': 3.998881340452636e-05, 'epoch': 0.4}

  4%|▍         | 1664/41250 [4:01:15<94:53:33,  8.63s/it][2025-04-26 08:09:53,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 08:09:53,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5696.90 | bwd_inner_microstep: 5684.08 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.90
[2025-04-26 08:09:53,148] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5696.91 | bwd_inner: 5684.08 | bwd_allreduce: 12.79 | step: 18.90

  4%|▍         | 1665/41250 [4:01:24<94:54:07,  8.63s/it]
                                                         
{'loss': 0.4218, 'grad_norm': 6.739671230316162, 'learning_rate': 3.998876082858177e-05, 'epoch': 0.4}

  4%|▍         | 1665/41250 [4:01:24<94:54:07,  8.63s/it][2025-04-26 08:10:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:10:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.27 | bwd_microstep: 5925.67 | bwd_inner_microstep: 5650.33 | bwd_allreduce_microstep: 275.30 | step_microstep: 18.16
[2025-04-26 08:10:01,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.27 | bwd: 5925.68 | bwd_inner: 5650.32 | bwd_allreduce: 275.32 | step: 18.16

  4%|▍         | 1666/41250 [4:01:33<95:34:56,  8.69s/it]
                                                         
{'loss': 0.2315, 'grad_norm': 1.2956774234771729, 'learning_rate': 3.998870812941039e-05, 'epoch': 0.4}

  4%|▍         | 1666/41250 [4:01:33<95:34:56,  8.69s/it][2025-04-26 08:10:10,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:10:10,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.57 | bwd_microstep: 5743.24 | bwd_inner_microstep: 5682.13 | bwd_allreduce_microstep: 61.06 | step_microstep: 18.68
[2025-04-26 08:10:10,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.57 | bwd: 5743.26 | bwd_inner: 5682.13 | bwd_allreduce: 61.08 | step: 18.68

  4%|▍         | 1667/41250 [4:01:41<95:30:23,  8.69s/it]
                                                         
{'loss': 0.2775, 'grad_norm': 2.2469775676727295, 'learning_rate': 3.998865530701256e-05, 'epoch': 0.4}

  4%|▍         | 1667/41250 [4:01:41<95:30:23,  8.69s/it][2025-04-26 08:10:19,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 08:10:19,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.47 | bwd_microstep: 5723.09 | bwd_inner_microstep: 5687.89 | bwd_allreduce_microstep: 35.15 | step_microstep: 18.94
[2025-04-26 08:10:19,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.47 | bwd: 5723.10 | bwd_inner: 5687.89 | bwd_allreduce: 35.17 | step: 18.95

  4%|▍         | 1668/41250 [4:01:50<95:25:41,  8.68s/it]
                                                         
{'loss': 0.1213, 'grad_norm': 1.0472943782806396, 'learning_rate': 3.99886023613886e-05, 'epoch': 0.4}

  4%|▍         | 1668/41250 [4:01:50<95:25:41,  8.68s/it][2025-04-26 08:10:27,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 08:10:27,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.87 | bwd_microstep: 5743.67 | bwd_inner_microstep: 5672.90 | bwd_allreduce_microstep: 70.72 | step_microstep: 18.86
[2025-04-26 08:10:27,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.87 | bwd: 5743.68 | bwd_inner: 5672.90 | bwd_allreduce: 70.75 | step: 18.86

  4%|▍         | 1669/41250 [4:01:59<95:23:52,  8.68s/it]
                                                         
{'loss': 0.2218, 'grad_norm': 1.3690205812454224, 'learning_rate': 3.998854929253884e-05, 'epoch': 0.4}

  4%|▍         | 1669/41250 [4:01:59<95:23:52,  8.68s/it][2025-04-26 08:10:36,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:10:36,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.55 | bwd_microstep: 5737.68 | bwd_inner_microstep: 5645.07 | bwd_allreduce_microstep: 92.56 | step_microstep: 18.50
[2025-04-26 08:10:36,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.55 | bwd: 5737.70 | bwd_inner: 5645.07 | bwd_allreduce: 92.58 | step: 18.50

  4%|▍         | 1670/41250 [4:02:07<95:19:53,  8.67s/it]
                                                         
{'loss': 0.1745, 'grad_norm': 2.1467692852020264, 'learning_rate': 3.9988496100463604e-05, 'epoch': 0.4}

  4%|▍         | 1670/41250 [4:02:07<95:19:53,  8.67s/it][2025-04-26 08:10:45,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:10:45,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.28 | bwd_microstep: 5772.22 | bwd_inner_microstep: 5759.44 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.60
[2025-04-26 08:10:45,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.28 | bwd: 5772.24 | bwd_inner: 5759.44 | bwd_allreduce: 12.75 | step: 18.60

  4%|▍         | 1671/41250 [4:02:16<95:33:45,  8.69s/it]
                                                         
{'loss': 0.2339, 'grad_norm': 1.2587283849716187, 'learning_rate': 3.998844278516322e-05, 'epoch': 0.41}

  4%|▍         | 1671/41250 [4:02:16<95:33:45,  8.69s/it][2025-04-26 08:10:54,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 08:10:54,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.27 | bwd_microstep: 5836.23 | bwd_inner_microstep: 5677.84 | bwd_allreduce_microstep: 158.34 | step_microstep: 18.48
[2025-04-26 08:10:54,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.27 | bwd: 5836.25 | bwd_inner: 5677.84 | bwd_allreduce: 158.36 | step: 18.48

  4%|▍         | 1672/41250 [4:02:25<95:48:44,  8.72s/it]
                                                         
{'loss': 0.288, 'grad_norm': 1.8955633640289307, 'learning_rate': 3.9988389346638017e-05, 'epoch': 0.41}

  4%|▍         | 1672/41250 [4:02:25<95:48:44,  8.72s/it][2025-04-26 08:11:02,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 08:11:02,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.44 | bwd_microstep: 5691.21 | bwd_inner_microstep: 5642.12 | bwd_allreduce_microstep: 49.04 | step_microstep: 18.63
[2025-04-26 08:11:02,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.44 | bwd: 5691.23 | bwd_inner: 5642.12 | bwd_allreduce: 49.06 | step: 18.63

  4%|▍         | 1673/41250 [4:02:33<95:28:01,  8.68s/it]
                                                         
{'loss': 0.2016, 'grad_norm': 1.71510910987854, 'learning_rate': 3.9988335784888336e-05, 'epoch': 0.41}

  4%|▍         | 1673/41250 [4:02:33<95:28:01,  8.68s/it][2025-04-26 08:11:11,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 08:11:11,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.71 | bwd_microstep: 5753.44 | bwd_inner_microstep: 5643.07 | bwd_allreduce_microstep: 110.33 | step_microstep: 19.13
[2025-04-26 08:11:11,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.71 | bwd: 5753.45 | bwd_inner: 5643.07 | bwd_allreduce: 110.35 | step: 19.13

  4%|▍         | 1674/41250 [4:02:42<95:24:27,  8.68s/it]
                                                         
{'loss': 0.1126, 'grad_norm': 1.9068554639816284, 'learning_rate': 3.9988282099914484e-05, 'epoch': 0.41}

  4%|▍         | 1674/41250 [4:02:42<95:24:27,  8.68s/it][2025-04-26 08:11:20,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 08:11:20,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.57 | bwd_microstep: 5723.04 | bwd_inner_microstep: 5692.60 | bwd_allreduce_microstep: 30.40 | step_microstep: 18.71
[2025-04-26 08:11:20,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.57 | bwd: 5723.06 | bwd_inner: 5692.60 | bwd_allreduce: 30.41 | step: 18.71

  4%|▍         | 1675/41250 [4:02:51<95:21:14,  8.67s/it]
                                                         
{'loss': 0.1055, 'grad_norm': 0.8875466585159302, 'learning_rate': 3.9988228291716806e-05, 'epoch': 0.41}

  4%|▍         | 1675/41250 [4:02:51<95:21:14,  8.67s/it][2025-04-26 08:11:28,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:11:28,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.89 | bwd_microstep: 5750.82 | bwd_inner_microstep: 5689.80 | bwd_allreduce_microstep: 60.98 | step_microstep: 18.42
[2025-04-26 08:11:28,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5750.83 | bwd_inner: 5689.80 | bwd_allreduce: 60.99 | step: 18.42

  4%|▍         | 1676/41250 [4:02:59<95:20:18,  8.67s/it]
                                                         
{'loss': 0.1192, 'grad_norm': 1.4575116634368896, 'learning_rate': 3.9988174360295635e-05, 'epoch': 0.41}

  4%|▍         | 1676/41250 [4:02:59<95:20:18,  8.67s/it][2025-04-26 08:11:37,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.15 | optimizer_step: 1.08
[2025-04-26 08:11:37,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.22 | bwd_microstep: 5703.45 | bwd_inner_microstep: 5689.23 | bwd_allreduce_microstep: 14.14 | step_microstep: 20.24
[2025-04-26 08:11:37,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.22 | bwd: 5703.46 | bwd_inner: 5689.23 | bwd_allreduce: 14.18 | step: 20.24

  4%|▍         | 1677/41250 [4:03:08<95:12:23,  8.66s/it]
                                                         
{'loss': 0.1703, 'grad_norm': 1.8178467750549316, 'learning_rate': 3.9988120305651296e-05, 'epoch': 0.41}

  4%|▍         | 1677/41250 [4:03:08<95:12:23,  8.66s/it][2025-04-26 08:11:46,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.28 | optimizer_step: 0.97
[2025-04-26 08:11:46,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.17 | bwd_microstep: 5784.21 | bwd_inner_microstep: 5646.10 | bwd_allreduce_microstep: 138.05 | step_microstep: 19.81
[2025-04-26 08:11:46,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.17 | bwd: 5784.22 | bwd_inner: 5646.10 | bwd_allreduce: 138.07 | step: 19.81

  4%|▍         | 1678/41250 [4:03:17<95:18:19,  8.67s/it]
                                                         
{'loss': 0.2573, 'grad_norm': 1.9474105834960938, 'learning_rate': 3.9988066127784134e-05, 'epoch': 0.41}

  4%|▍         | 1678/41250 [4:03:17<95:18:19,  8.67s/it][2025-04-26 08:11:54,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:11:54,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.31 | bwd_microstep: 5762.56 | bwd_inner_microstep: 5703.68 | bwd_allreduce_microstep: 58.83 | step_microstep: 18.86
[2025-04-26 08:11:54,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.31 | bwd: 5762.57 | bwd_inner: 5703.68 | bwd_allreduce: 58.85 | step: 18.86

  4%|▍         | 1679/41250 [4:03:25<95:24:14,  8.68s/it]
                                                         
{'loss': 0.1072, 'grad_norm': 0.9701047539710999, 'learning_rate': 3.9988011826694474e-05, 'epoch': 0.41}

  4%|▍         | 1679/41250 [4:03:25<95:24:14,  8.68s/it][2025-04-26 08:12:03,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:12:03,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.03 | bwd_microstep: 5708.09 | bwd_inner_microstep: 5695.30 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.61
[2025-04-26 08:12:03,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.03 | bwd: 5708.10 | bwd_inner: 5695.30 | bwd_allreduce: 12.76 | step: 18.62

  4%|▍         | 1680/41250 [4:03:34<95:15:19,  8.67s/it]
                                                         
{'loss': 0.273, 'grad_norm': 2.36437726020813, 'learning_rate': 3.998795740238265e-05, 'epoch': 0.41}

  4%|▍         | 1680/41250 [4:03:34<95:15:19,  8.67s/it][2025-04-26 08:12:12,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:12:12,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.85 | bwd_microstep: 5720.60 | bwd_inner_microstep: 5659.13 | bwd_allreduce_microstep: 61.43 | step_microstep: 18.59
[2025-04-26 08:12:12,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.85 | bwd: 5720.62 | bwd_inner: 5659.13 | bwd_allreduce: 61.44 | step: 18.59

  4%|▍         | 1681/41250 [4:03:43<95:07:37,  8.65s/it]
                                                         
{'loss': 0.1925, 'grad_norm': 1.771518349647522, 'learning_rate': 3.9987902854849e-05, 'epoch': 0.41}

  4%|▍         | 1681/41250 [4:03:43<95:07:37,  8.65s/it][2025-04-26 08:12:20,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 08:12:20,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.20 | bwd_microstep: 5717.46 | bwd_inner_microstep: 5704.79 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.12
[2025-04-26 08:12:20,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.20 | bwd: 5717.47 | bwd_inner: 5704.79 | bwd_allreduce: 12.64 | step: 19.12

  4%|▍         | 1682/41250 [4:03:51<95:04:52,  8.65s/it]
                                                         
{'loss': 0.1412, 'grad_norm': 1.3491671085357666, 'learning_rate': 3.998784818409386e-05, 'epoch': 0.41}

  4%|▍         | 1682/41250 [4:03:51<95:04:52,  8.65s/it][2025-04-26 08:12:29,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-26 08:12:29,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.04 | bwd_microstep: 5750.46 | bwd_inner_microstep: 5704.07 | bwd_allreduce_microstep: 46.34 | step_microstep: 18.90
[2025-04-26 08:12:29,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.04 | bwd: 5750.47 | bwd_inner: 5704.07 | bwd_allreduce: 46.36 | step: 18.90

  4%|▍         | 1683/41250 [4:04:00<95:10:53,  8.66s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 1.1101412773132324, 'learning_rate': 3.998779339011757e-05, 'epoch': 0.41}

  4%|▍         | 1683/41250 [4:04:00<95:10:53,  8.66s/it][2025-04-26 08:12:38,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 08:12:38,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5709.92 | bwd_inner_microstep: 5696.52 | bwd_allreduce_microstep: 13.35 | step_microstep: 18.72
[2025-04-26 08:12:38,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5709.93 | bwd_inner: 5696.52 | bwd_allreduce: 13.37 | step: 18.73

  4%|▍         | 1684/41250 [4:04:09<95:07:34,  8.66s/it]
                                                         
{'loss': 0.3035, 'grad_norm': 1.6996033191680908, 'learning_rate': 3.9987738472920464e-05, 'epoch': 0.41}

  4%|▍         | 1684/41250 [4:04:09<95:07:34,  8.66s/it][2025-04-26 08:12:46,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:12:46,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.47 | bwd_microstep: 5820.27 | bwd_inner_microstep: 5650.09 | bwd_allreduce_microstep: 170.13 | step_microstep: 18.31
[2025-04-26 08:12:46,741] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.47 | bwd: 5820.28 | bwd_inner: 5650.09 | bwd_allreduce: 170.15 | step: 18.31

  4%|▍         | 1685/41250 [4:04:17<95:19:04,  8.67s/it]
                                                         
{'loss': 0.2048, 'grad_norm': 1.4209758043289185, 'learning_rate': 3.998768343250288e-05, 'epoch': 0.41}

  4%|▍         | 1685/41250 [4:04:17<95:19:04,  8.67s/it][2025-04-26 08:12:55,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:12:55,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.51 | bwd_microstep: 5719.54 | bwd_inner_microstep: 5670.87 | bwd_allreduce_microstep: 48.63 | step_microstep: 18.60
[2025-04-26 08:12:55,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.51 | bwd: 5719.55 | bwd_inner: 5670.87 | bwd_allreduce: 48.64 | step: 18.60

  4%|▍         | 1686/41250 [4:04:26<95:10:39,  8.66s/it]
                                                         
{'loss': 0.3784, 'grad_norm': 2.1783576011657715, 'learning_rate': 3.998762826886516e-05, 'epoch': 0.41}

  4%|▍         | 1686/41250 [4:04:26<95:10:39,  8.66s/it][2025-04-26 08:13:04,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 08:13:04,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.07 | bwd_microstep: 5740.79 | bwd_inner_microstep: 5661.51 | bwd_allreduce_microstep: 79.23 | step_microstep: 19.49
[2025-04-26 08:13:04,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.07 | bwd: 5740.80 | bwd_inner: 5661.51 | bwd_allreduce: 79.25 | step: 19.49

  4%|▍         | 1687/41250 [4:04:35<95:08:20,  8.66s/it]
                                                         
{'loss': 0.0952, 'grad_norm': 1.0659358501434326, 'learning_rate': 3.998757298200764e-05, 'epoch': 0.41}

  4%|▍         | 1687/41250 [4:04:35<95:08:20,  8.66s/it][2025-04-26 08:13:12,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:13:12,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5758.25 | bwd_inner_microstep: 5706.34 | bwd_allreduce_microstep: 51.86 | step_microstep: 18.70
[2025-04-26 08:13:12,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5758.26 | bwd_inner: 5706.34 | bwd_allreduce: 51.88 | step: 18.70

  4%|▍         | 1688/41250 [4:04:43<95:14:22,  8.67s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.5763458013534546, 'learning_rate': 3.998751757193067e-05, 'epoch': 0.41}

  4%|▍         | 1688/41250 [4:04:43<95:14:22,  8.67s/it][2025-04-26 08:13:21,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:13:21,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.05 | bwd_microstep: 5758.68 | bwd_inner_microstep: 5706.21 | bwd_allreduce_microstep: 52.42 | step_microstep: 18.32
[2025-04-26 08:13:21,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.05 | bwd: 5758.69 | bwd_inner: 5706.21 | bwd_allreduce: 52.44 | step: 18.32

  4%|▍         | 1689/41250 [4:04:52<95:21:08,  8.68s/it]
                                                         
{'loss': 0.4033, 'grad_norm': 2.220538854598999, 'learning_rate': 3.9987462038634574e-05, 'epoch': 0.41}

  4%|▍         | 1689/41250 [4:04:52<95:21:08,  8.68s/it][2025-04-26 08:13:30,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:13:30,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.08 | bwd_microstep: 5877.73 | bwd_inner_microstep: 5864.89 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.61
[2025-04-26 08:13:30,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.08 | bwd: 5877.74 | bwd_inner: 5864.89 | bwd_allreduce: 12.81 | step: 18.61

  4%|▍         | 1690/41250 [4:05:01<96:01:55,  8.74s/it]
                                                         
{'loss': 0.1659, 'grad_norm': 3.115443706512451, 'learning_rate': 3.998740638211972e-05, 'epoch': 0.41}

  4%|▍         | 1690/41250 [4:05:01<96:01:55,  8.74s/it][2025-04-26 08:13:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:13:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.81 | bwd_microstep: 5704.11 | bwd_inner_microstep: 5691.22 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.70
[2025-04-26 08:13:38,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.81 | bwd: 5704.13 | bwd_inner: 5691.22 | bwd_allreduce: 12.87 | step: 18.71

  4%|▍         | 1691/41250 [4:05:10<95:42:04,  8.71s/it]
                                                         
{'loss': 0.1914, 'grad_norm': 2.1398491859436035, 'learning_rate': 3.998735060238643e-05, 'epoch': 0.41}

  4%|▍         | 1691/41250 [4:05:10<95:42:04,  8.71s/it][2025-04-26 08:13:47,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:13:47,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5718.29 | bwd_inner_microstep: 5651.59 | bwd_allreduce_microstep: 66.65 | step_microstep: 18.58
[2025-04-26 08:13:47,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5718.30 | bwd_inner: 5651.59 | bwd_allreduce: 66.67 | step: 18.58

  4%|▍         | 1692/41250 [4:05:18<95:25:25,  8.68s/it]
                                                         
{'loss': 0.1834, 'grad_norm': 1.7542110681533813, 'learning_rate': 3.998729469943506e-05, 'epoch': 0.41}

  4%|▍         | 1692/41250 [4:05:18<95:25:25,  8.68s/it][2025-04-26 08:13:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:13:56,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5744.98 | bwd_inner_microstep: 5695.25 | bwd_allreduce_microstep: 49.68 | step_microstep: 18.83
[2025-04-26 08:13:56,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5745.00 | bwd_inner: 5695.25 | bwd_allreduce: 49.70 | step: 18.83

  4%|▍         | 1693/41250 [4:05:27<95:24:15,  8.68s/it]
                                                         
{'loss': 0.2071, 'grad_norm': 1.590701937675476, 'learning_rate': 3.998723867326594e-05, 'epoch': 0.41}

  4%|▍         | 1693/41250 [4:05:27<95:24:15,  8.68s/it][2025-04-26 08:14:04,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.12 | optimizer_step: 1.03
[2025-04-26 08:14:04,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.56 | bwd_microstep: 5766.63 | bwd_inner_microstep: 5655.07 | bwd_allreduce_microstep: 111.50 | step_microstep: 19.95
[2025-04-26 08:14:04,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.56 | bwd: 5766.65 | bwd_inner: 5655.07 | bwd_allreduce: 111.53 | step: 19.96

  4%|▍         | 1694/41250 [4:05:36<95:21:47,  8.68s/it]
                                                         
{'loss': 0.1472, 'grad_norm': 1.666669249534607, 'learning_rate': 3.998718252387943e-05, 'epoch': 0.41}

  4%|▍         | 1694/41250 [4:05:36<95:21:47,  8.68s/it][2025-04-26 08:14:13,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:14:13,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.62 | bwd_microstep: 5783.49 | bwd_inner_microstep: 5770.61 | bwd_allreduce_microstep: 12.84 | step_microstep: 17.92
[2025-04-26 08:14:13,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.62 | bwd: 5783.51 | bwd_inner: 5770.61 | bwd_allreduce: 12.85 | step: 17.93

  4%|▍         | 1695/41250 [4:05:44<95:36:58,  8.70s/it]
                                                         
{'loss': 0.2763, 'grad_norm': 2.780937671661377, 'learning_rate': 3.9987126251275874e-05, 'epoch': 0.41}

  4%|▍         | 1695/41250 [4:05:44<95:36:58,  8.70s/it][2025-04-26 08:14:22,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:14:22,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.95 | bwd_microstep: 5767.63 | bwd_inner_microstep: 5649.99 | bwd_allreduce_microstep: 117.59 | step_microstep: 18.79
[2025-04-26 08:14:22,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.95 | bwd: 5767.65 | bwd_inner: 5649.99 | bwd_allreduce: 117.61 | step: 18.79

  4%|▍         | 1696/41250 [4:05:53<95:34:54,  8.70s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 1.379228115081787, 'learning_rate': 3.9987069855455616e-05, 'epoch': 0.41}

  4%|▍         | 1696/41250 [4:05:53<95:34:54,  8.70s/it][2025-04-26 08:14:31,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.27 | optimizer_step: 1.28
[2025-04-26 08:14:31,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.36 | bwd_microstep: 5736.62 | bwd_inner_microstep: 5699.52 | bwd_allreduce_microstep: 37.04 | step_microstep: 20.16
[2025-04-26 08:14:31,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.36 | bwd: 5736.64 | bwd_inner: 5699.52 | bwd_allreduce: 37.07 | step: 20.16

  4%|▍         | 1697/41250 [4:06:02<95:31:12,  8.69s/it]
                                                         
{'loss': 0.3487, 'grad_norm': 2.8024983406066895, 'learning_rate': 3.9987013336419e-05, 'epoch': 0.41}

  4%|▍         | 1697/41250 [4:06:02<95:31:12,  8.69s/it][2025-04-26 08:14:39,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:14:39,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.15 | bwd_microstep: 5761.83 | bwd_inner_microstep: 5639.97 | bwd_allreduce_microstep: 121.81 | step_microstep: 18.54
[2025-04-26 08:14:39,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.15 | bwd: 5761.84 | bwd_inner: 5639.97 | bwd_allreduce: 121.83 | step: 18.54

  4%|▍         | 1698/41250 [4:06:10<95:26:55,  8.69s/it]
                                                         
{'loss': 0.09, 'grad_norm': 0.8537144660949707, 'learning_rate': 3.998695669416638e-05, 'epoch': 0.41}

  4%|▍         | 1698/41250 [4:06:10<95:26:55,  8.69s/it][2025-04-26 08:14:48,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:14:48,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.39 | bwd_microstep: 5834.26 | bwd_inner_microstep: 5691.38 | bwd_allreduce_microstep: 142.83 | step_microstep: 18.37
[2025-04-26 08:14:48,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.39 | bwd: 5834.27 | bwd_inner: 5691.38 | bwd_allreduce: 142.85 | step: 18.38

  4%|▍         | 1699/41250 [4:06:19<95:42:24,  8.71s/it]
                                                         
{'loss': 0.257, 'grad_norm': 2.389368772506714, 'learning_rate': 3.9986899928698096e-05, 'epoch': 0.41}

  4%|▍         | 1699/41250 [4:06:19<95:42:24,  8.71s/it][2025-04-26 08:14:57,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:14:57,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.62 | bwd_microstep: 5700.00 | bwd_inner_microstep: 5687.42 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.00
[2025-04-26 08:14:57,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.62 | bwd: 5700.02 | bwd_inner: 5687.42 | bwd_allreduce: 12.56 | step: 18.01

  4%|▍         | 1700/41250 [4:06:28<95:26:30,  8.69s/it]
                                                         
{'loss': 0.1227, 'grad_norm': 0.9192409515380859, 'learning_rate': 3.998684304001452e-05, 'epoch': 0.41}

  4%|▍         | 1700/41250 [4:06:28<95:26:30,  8.69s/it][2025-04-26 08:15:05,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:15:05,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.73 | bwd_microstep: 5701.62 | bwd_inner_microstep: 5688.79 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.43
[2025-04-26 08:15:05,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.74 | bwd: 5701.64 | bwd_inner: 5688.79 | bwd_allreduce: 12.81 | step: 18.43

  4%|▍         | 1701/41250 [4:06:36<95:14:49,  8.67s/it]
                                                         
{'loss': 0.1931, 'grad_norm': 1.4189798831939697, 'learning_rate': 3.9986786028115974e-05, 'epoch': 0.41}

  4%|▍         | 1701/41250 [4:06:36<95:14:49,  8.67s/it][2025-04-26 08:15:14,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 08:15:14,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.66 | bwd_microstep: 5748.94 | bwd_inner_microstep: 5645.99 | bwd_allreduce_microstep: 102.91 | step_microstep: 18.20
[2025-04-26 08:15:14,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.66 | bwd: 5748.96 | bwd_inner: 5645.99 | bwd_allreduce: 102.93 | step: 18.20

  4%|▍         | 1702/41250 [4:06:45<95:14:46,  8.67s/it]
                                                         
{'loss': 0.1428, 'grad_norm': 1.160668134689331, 'learning_rate': 3.9986728893002826e-05, 'epoch': 0.41}

  4%|▍         | 1702/41250 [4:06:45<95:14:46,  8.67s/it][2025-04-26 08:15:23,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:15:23,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.47 | bwd_microstep: 5700.32 | bwd_inner_microstep: 5687.44 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.91
[2025-04-26 08:15:23,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.47 | bwd: 5700.34 | bwd_inner: 5687.44 | bwd_allreduce: 12.85 | step: 18.92

  4%|▍         | 1703/41250 [4:06:54<95:05:04,  8.66s/it]
                                                         
{'loss': 0.0656, 'grad_norm': 0.7699446678161621, 'learning_rate': 3.998667163467543e-05, 'epoch': 0.41}

  4%|▍         | 1703/41250 [4:06:54<95:05:04,  8.66s/it][2025-04-26 08:15:31,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:15:31,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5708.65 | bwd_inner_microstep: 5673.07 | bwd_allreduce_microstep: 35.54 | step_microstep: 17.81
[2025-04-26 08:15:31,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5708.66 | bwd_inner: 5673.07 | bwd_allreduce: 35.55 | step: 17.81

  4%|▍         | 1704/41250 [4:07:02<95:03:33,  8.65s/it]
                                                         
{'loss': 0.0731, 'grad_norm': 0.882522702217102, 'learning_rate': 3.9986614253134135e-05, 'epoch': 0.41}

  4%|▍         | 1704/41250 [4:07:02<95:03:33,  8.65s/it][2025-04-26 08:15:40,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:15:40,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.17 | bwd_microstep: 5747.47 | bwd_inner_microstep: 5640.19 | bwd_allreduce_microstep: 107.24 | step_microstep: 17.87
[2025-04-26 08:15:40,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.17 | bwd: 5747.48 | bwd_inner: 5640.19 | bwd_allreduce: 107.25 | step: 17.87

  4%|▍         | 1705/41250 [4:07:11<95:04:54,  8.66s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 0.8597974181175232, 'learning_rate': 3.998655674837929e-05, 'epoch': 0.41}

  4%|▍         | 1705/41250 [4:07:11<95:04:54,  8.66s/it][2025-04-26 08:15:49,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-26 08:15:49,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5740.10 | bwd_inner_microstep: 5677.78 | bwd_allreduce_microstep: 62.27 | step_microstep: 18.08
[2025-04-26 08:15:49,011] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5740.11 | bwd_inner: 5677.78 | bwd_allreduce: 62.28 | step: 18.08

  4%|▍         | 1706/41250 [4:07:20<95:07:31,  8.66s/it]
                                                         
{'loss': 0.2611, 'grad_norm': 1.4736143350601196, 'learning_rate': 3.9986499120411254e-05, 'epoch': 0.41}

  4%|▍         | 1706/41250 [4:07:20<95:07:31,  8.66s/it][2025-04-26 08:15:57,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 08:15:57,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5688.42 | bwd_inner_microstep: 5675.51 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.31
[2025-04-26 08:15:57,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5688.43 | bwd_inner: 5675.51 | bwd_allreduce: 12.88 | step: 18.32

  4%|▍         | 1707/41250 [4:07:28<94:59:26,  8.65s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 1.189000129699707, 'learning_rate': 3.998644136923038e-05, 'epoch': 0.41}

  4%|▍         | 1707/41250 [4:07:28<94:59:26,  8.65s/it][2025-04-26 08:16:06,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:16:06,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5740.44 | bwd_inner_microstep: 5654.59 | bwd_allreduce_microstep: 85.81 | step_microstep: 18.45
[2025-04-26 08:16:06,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5740.45 | bwd_inner: 5654.59 | bwd_allreduce: 85.82 | step: 18.45

  4%|▍         | 1708/41250 [4:07:37<94:59:03,  8.65s/it]
                                                         
{'loss': 0.3108, 'grad_norm': 2.2629098892211914, 'learning_rate': 3.9986383494837035e-05, 'epoch': 0.41}

  4%|▍         | 1708/41250 [4:07:37<94:59:03,  8.65s/it][2025-04-26 08:16:14,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:16:14,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.92 | bwd_microstep: 5751.24 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 105.86 | step_microstep: 17.80
[2025-04-26 08:16:14,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.92 | bwd: 5751.25 | bwd_inner: 5645.34 | bwd_allreduce: 105.88 | step: 17.81

  4%|▍         | 1709/41250 [4:07:46<95:01:48,  8.65s/it]
                                                         
{'loss': 0.1445, 'grad_norm': 1.3411613702774048, 'learning_rate': 3.998632549723156e-05, 'epoch': 0.41}

  4%|▍         | 1709/41250 [4:07:46<95:01:48,  8.65s/it][2025-04-26 08:16:23,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:16:23,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.23 | bwd_microstep: 5738.14 | bwd_inner_microstep: 5690.04 | bwd_allreduce_microstep: 48.05 | step_microstep: 18.23
[2025-04-26 08:16:23,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.23 | bwd: 5738.15 | bwd_inner: 5690.04 | bwd_allreduce: 48.07 | step: 18.23

  4%|▍         | 1710/41250 [4:07:54<95:05:51,  8.66s/it]
                                                         
{'loss': 0.1629, 'grad_norm': 1.2981430292129517, 'learning_rate': 3.998626737641432e-05, 'epoch': 0.41}

  4%|▍         | 1710/41250 [4:07:54<95:05:51,  8.66s/it][2025-04-26 08:16:32,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:16:32,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.51 | bwd_microstep: 5747.92 | bwd_inner_microstep: 5642.77 | bwd_allreduce_microstep: 105.11 | step_microstep: 18.21
[2025-04-26 08:16:32,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.51 | bwd: 5747.94 | bwd_inner: 5642.77 | bwd_allreduce: 105.13 | step: 18.21

  4%|▍         | 1711/41250 [4:08:03<95:07:38,  8.66s/it]
                                                         
{'loss': 0.2216, 'grad_norm': 1.1489187479019165, 'learning_rate': 3.998620913238568e-05, 'epoch': 0.41}

  4%|▍         | 1711/41250 [4:08:03<95:07:38,  8.66s/it][2025-04-26 08:16:40,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 08:16:40,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.85 | bwd_microstep: 5736.97 | bwd_inner_microstep: 5686.96 | bwd_allreduce_microstep: 49.96 | step_microstep: 19.05
[2025-04-26 08:16:40,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.85 | bwd: 5736.98 | bwd_inner: 5686.96 | bwd_allreduce: 49.98 | step: 19.05

  4%|▍         | 1712/41250 [4:08:12<95:05:13,  8.66s/it]
                                                         
{'loss': 0.3289, 'grad_norm': 1.659507155418396, 'learning_rate': 3.998615076514598e-05, 'epoch': 0.42}

  4%|▍         | 1712/41250 [4:08:12<95:05:13,  8.66s/it][2025-04-26 08:16:49,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:16:49,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.08 | bwd_microstep: 5737.23 | bwd_inner_microstep: 5684.93 | bwd_allreduce_microstep: 52.25 | step_microstep: 18.52
[2025-04-26 08:16:49,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.08 | bwd: 5737.24 | bwd_inner: 5684.93 | bwd_allreduce: 52.27 | step: 18.53

  4%|▍         | 1713/41250 [4:08:20<95:06:45,  8.66s/it]
                                                         
{'loss': 0.102, 'grad_norm': 1.4322935342788696, 'learning_rate': 3.99860922746956e-05, 'epoch': 0.42}

  4%|▍         | 1713/41250 [4:08:20<95:06:45,  8.66s/it][2025-04-26 08:16:58,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:16:58,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.17 | bwd_microstep: 5748.63 | bwd_inner_microstep: 5652.10 | bwd_allreduce_microstep: 96.49 | step_microstep: 17.99
[2025-04-26 08:16:58,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.17 | bwd: 5748.64 | bwd_inner: 5652.10 | bwd_allreduce: 96.51 | step: 18.00

  4%|▍         | 1714/41250 [4:08:29<95:08:22,  8.66s/it]
                                                         
{'loss': 0.1511, 'grad_norm': 1.4555331468582153, 'learning_rate': 3.998603366103489e-05, 'epoch': 0.42}

  4%|▍         | 1714/41250 [4:08:29<95:08:22,  8.66s/it][2025-04-26 08:17:06,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:17:06,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.22 | bwd_microstep: 5776.03 | bwd_inner_microstep: 5657.14 | bwd_allreduce_microstep: 118.84 | step_microstep: 17.72
[2025-04-26 08:17:06,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.22 | bwd: 5776.04 | bwd_inner: 5657.14 | bwd_allreduce: 118.86 | step: 17.73

  4%|▍         | 1715/41250 [4:08:38<95:14:05,  8.67s/it]
                                                         
{'loss': 0.1715, 'grad_norm': 1.1867492198944092, 'learning_rate': 3.998597492416421e-05, 'epoch': 0.42}

  4%|▍         | 1715/41250 [4:08:38<95:14:05,  8.67s/it][2025-04-26 08:17:15,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 08:17:15,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5681.54 | bwd_allreduce_microstep: 83.55 | step_microstep: 18.56
[2025-04-26 08:17:15,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5765.15 | bwd_inner: 5681.53 | bwd_allreduce: 83.57 | step: 18.56

  4%|▍         | 1716/41250 [4:08:46<95:18:23,  8.68s/it]
                                                         
{'loss': 0.3484, 'grad_norm': 4.229805946350098, 'learning_rate': 3.998591606408393e-05, 'epoch': 0.42}

  4%|▍         | 1716/41250 [4:08:46<95:18:23,  8.68s/it][2025-04-26 08:17:24,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:17:24,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.93 | bwd_microstep: 5706.92 | bwd_inner_microstep: 5644.48 | bwd_allreduce_microstep: 62.40 | step_microstep: 18.17
[2025-04-26 08:17:24,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.93 | bwd: 5706.93 | bwd_inner: 5644.48 | bwd_allreduce: 62.42 | step: 18.18

  4%|▍         | 1717/41250 [4:08:55<95:03:38,  8.66s/it]
                                                         
{'loss': 0.2097, 'grad_norm': 1.5515249967575073, 'learning_rate': 3.998585708079441e-05, 'epoch': 0.42}

  4%|▍         | 1717/41250 [4:08:55<95:03:38,  8.66s/it][2025-04-26 08:17:32,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:17:32,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.06 | bwd_microstep: 5713.29 | bwd_inner_microstep: 5700.68 | bwd_allreduce_microstep: 12.57 | step_microstep: 17.90
[2025-04-26 08:17:32,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.06 | bwd: 5713.30 | bwd_inner: 5700.68 | bwd_allreduce: 12.59 | step: 17.90

  4%|▍         | 1718/41250 [4:09:04<95:04:04,  8.66s/it]
                                                         
{'loss': 0.0564, 'grad_norm': 1.0438674688339233, 'learning_rate': 3.9985797974296006e-05, 'epoch': 0.42}

  4%|▍         | 1718/41250 [4:09:04<95:04:04,  8.66s/it][2025-04-26 08:17:41,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 08:17:41,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.02 | bwd_microstep: 5759.52 | bwd_inner_microstep: 5641.14 | bwd_allreduce_microstep: 118.34 | step_microstep: 18.34
[2025-04-26 08:17:41,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.02 | bwd: 5759.53 | bwd_inner: 5641.14 | bwd_allreduce: 118.36 | step: 18.34

  4%|▍         | 1719/41250 [4:09:12<95:06:56,  8.66s/it]
                                                         
{'loss': 0.2795, 'grad_norm': 2.7227835655212402, 'learning_rate': 3.998573874458909e-05, 'epoch': 0.42}

  4%|▍         | 1719/41250 [4:09:12<95:06:56,  8.66s/it][2025-04-26 08:17:50,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 08:17:50,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.97 | bwd_microstep: 5737.71 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 40.73 | step_microstep: 17.75
[2025-04-26 08:17:50,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.97 | bwd: 5737.73 | bwd_inner: 5696.95 | bwd_allreduce: 40.74 | step: 17.75

  4%|▍         | 1720/41250 [4:09:21<95:09:22,  8.67s/it]
                                                         
{'loss': 0.2487, 'grad_norm': 2.168306827545166, 'learning_rate': 3.998567939167403e-05, 'epoch': 0.42}

  4%|▍         | 1720/41250 [4:09:21<95:09:22,  8.67s/it][2025-04-26 08:17:58,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:17:58,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5705.39 | bwd_inner_microstep: 5692.78 | bwd_allreduce_microstep: 12.56 | step_microstep: 17.71
[2025-04-26 08:17:58,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5705.40 | bwd_inner: 5692.78 | bwd_allreduce: 12.58 | step: 17.72

  4%|▍         | 1721/41250 [4:09:30<95:00:38,  8.65s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.5133609771728516, 'learning_rate': 3.998561991555118e-05, 'epoch': 0.42}

  4%|▍         | 1721/41250 [4:09:30<95:00:38,  8.65s/it][2025-04-26 08:18:07,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 08:18:07,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.61 | bwd_microstep: 5716.66 | bwd_inner_microstep: 5703.47 | bwd_allreduce_microstep: 13.15 | step_microstep: 18.80
[2025-04-26 08:18:07,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.61 | bwd: 5716.68 | bwd_inner: 5703.47 | bwd_allreduce: 13.17 | step: 18.81

  4%|▍         | 1722/41250 [4:09:38<95:00:59,  8.65s/it]
                                                         
{'loss': 0.1654, 'grad_norm': 0.9645792245864868, 'learning_rate': 3.998556031622092e-05, 'epoch': 0.42}

  4%|▍         | 1722/41250 [4:09:38<95:00:59,  8.65s/it][2025-04-26 08:18:16,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 08:18:16,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5791.72 | bwd_inner_microstep: 5657.66 | bwd_allreduce_microstep: 134.00 | step_microstep: 19.28
[2025-04-26 08:18:16,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5791.73 | bwd_inner: 5657.66 | bwd_allreduce: 134.03 | step: 19.28

  4%|▍         | 1723/41250 [4:09:47<95:10:28,  8.67s/it]
                                                         
{'loss': 0.112, 'grad_norm': 1.1745176315307617, 'learning_rate': 3.998550059368361e-05, 'epoch': 0.42}

  4%|▍         | 1723/41250 [4:09:47<95:10:28,  8.67s/it][2025-04-26 08:18:24,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:18:24,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5707.62 | bwd_inner_microstep: 5690.13 | bwd_allreduce_microstep: 17.44 | step_microstep: 18.87
[2025-04-26 08:18:24,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5707.63 | bwd_inner: 5690.13 | bwd_allreduce: 17.46 | step: 18.88

  4%|▍         | 1724/41250 [4:09:56<95:03:55,  8.66s/it]
                                                         
{'loss': 0.3013, 'grad_norm': 1.5534974336624146, 'learning_rate': 3.998544074793962e-05, 'epoch': 0.42}

  4%|▍         | 1724/41250 [4:09:56<95:03:55,  8.66s/it][2025-04-26 08:18:33,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 08:18:33,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.00 | bwd_microstep: 5708.07 | bwd_inner_microstep: 5695.34 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.61
[2025-04-26 08:18:33,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.00 | bwd: 5708.08 | bwd_inner: 5695.34 | bwd_allreduce: 12.70 | step: 18.61

  4%|▍         | 1725/41250 [4:10:04<95:01:21,  8.65s/it]
                                                         
{'loss': 0.1073, 'grad_norm': 2.667820453643799, 'learning_rate': 3.998538077898932e-05, 'epoch': 0.42}

  4%|▍         | 1725/41250 [4:10:04<95:01:21,  8.65s/it][2025-04-26 08:18:42,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:18:42,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.46 | bwd_microstep: 5726.54 | bwd_inner_microstep: 5658.43 | bwd_allreduce_microstep: 68.07 | step_microstep: 17.75
[2025-04-26 08:18:42,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.46 | bwd: 5726.55 | bwd_inner: 5658.43 | bwd_allreduce: 68.08 | step: 17.75

  4%|▍         | 1726/41250 [4:10:13<94:56:27,  8.65s/it]
                                                         
{'loss': 0.1234, 'grad_norm': 1.2138113975524902, 'learning_rate': 3.998532068683308e-05, 'epoch': 0.42}

  4%|▍         | 1726/41250 [4:10:13<94:56:27,  8.65s/it][2025-04-26 08:18:50,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 08:18:50,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.88 | bwd_microstep: 5779.30 | bwd_inner_microstep: 5653.55 | bwd_allreduce_microstep: 125.70 | step_microstep: 18.78
[2025-04-26 08:18:50,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.88 | bwd: 5779.31 | bwd_inner: 5653.55 | bwd_allreduce: 125.72 | step: 18.78

  4%|▍         | 1727/41250 [4:10:22<95:04:17,  8.66s/it]
                                                         
{'loss': 0.2756, 'grad_norm': 2.3703463077545166, 'learning_rate': 3.9985260471471266e-05, 'epoch': 0.42}

  4%|▍         | 1727/41250 [4:10:22<95:04:17,  8.66s/it][2025-04-26 08:18:59,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:18:59,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.36 | bwd_microstep: 5711.09 | bwd_inner_microstep: 5653.04 | bwd_allreduce_microstep: 58.00 | step_microstep: 18.58
[2025-04-26 08:18:59,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.36 | bwd: 5711.11 | bwd_inner: 5653.04 | bwd_allreduce: 58.02 | step: 18.58

  4%|▍         | 1728/41250 [4:10:30<94:59:17,  8.65s/it]
                                                         
{'loss': 0.0869, 'grad_norm': 1.6666587591171265, 'learning_rate': 3.998520013290425e-05, 'epoch': 0.42}

  4%|▍         | 1728/41250 [4:10:30<94:59:17,  8.65s/it][2025-04-26 08:19:08,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 08:19:08,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.75 | bwd_microstep: 5787.79 | bwd_inner_microstep: 5650.19 | bwd_allreduce_microstep: 137.55 | step_microstep: 18.04
[2025-04-26 08:19:08,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.75 | bwd: 5787.80 | bwd_inner: 5650.19 | bwd_allreduce: 137.57 | step: 18.04

  4%|▍         | 1729/41250 [4:10:39<95:08:05,  8.67s/it]
                                                         
{'loss': 0.2797, 'grad_norm': 2.149369239807129, 'learning_rate': 3.998513967113241e-05, 'epoch': 0.42}

  4%|▍         | 1729/41250 [4:10:39<95:08:05,  8.67s/it][2025-04-26 08:19:16,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:19:16,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.46 | bwd_microstep: 5766.25 | bwd_inner_microstep: 5698.81 | bwd_allreduce_microstep: 67.39 | step_microstep: 18.06
[2025-04-26 08:19:16,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.46 | bwd: 5766.26 | bwd_inner: 5698.81 | bwd_allreduce: 67.41 | step: 18.06

  4%|▍         | 1730/41250 [4:10:48<95:12:53,  8.67s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 1.6574972867965698, 'learning_rate': 3.998507908615611e-05, 'epoch': 0.42}

  4%|▍         | 1730/41250 [4:10:48<95:12:53,  8.67s/it][2025-04-26 08:19:25,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:19:25,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.19 | bwd_microstep: 5779.75 | bwd_inner_microstep: 5639.68 | bwd_allreduce_microstep: 140.02 | step_microstep: 17.92
[2025-04-26 08:19:25,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.19 | bwd: 5779.76 | bwd_inner: 5639.68 | bwd_allreduce: 140.05 | step: 17.93

  4%|▍         | 1731/41250 [4:10:56<95:18:37,  8.68s/it]
                                                         
{'loss': 0.1766, 'grad_norm': 1.370432734489441, 'learning_rate': 3.9985018377975734e-05, 'epoch': 0.42}

  4%|▍         | 1731/41250 [4:10:56<95:18:37,  8.68s/it][2025-04-26 08:19:34,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 08:19:34,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.93 | bwd_microstep: 5779.27 | bwd_inner_microstep: 5657.33 | bwd_allreduce_microstep: 121.89 | step_microstep: 18.13
[2025-04-26 08:19:34,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.93 | bwd: 5779.28 | bwd_inner: 5657.33 | bwd_allreduce: 121.91 | step: 18.14

  4%|▍         | 1732/41250 [4:11:05<95:21:28,  8.69s/it]
                                                         
{'loss': 0.1697, 'grad_norm': 2.8352320194244385, 'learning_rate': 3.998495754659165e-05, 'epoch': 0.42}

  4%|▍         | 1732/41250 [4:11:05<95:21:28,  8.69s/it][2025-04-26 08:19:42,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:19:42,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.75 | bwd_microstep: 5720.94 | bwd_inner_microstep: 5643.10 | bwd_allreduce_microstep: 77.80 | step_microstep: 18.19
[2025-04-26 08:19:42,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.75 | bwd: 5720.95 | bwd_inner: 5643.10 | bwd_allreduce: 77.81 | step: 18.20

  4%|▍         | 1733/41250 [4:11:14<95:09:15,  8.67s/it]
                                                         
{'loss': 0.1241, 'grad_norm': 0.9958112835884094, 'learning_rate': 3.998489659200423e-05, 'epoch': 0.42}

  4%|▍         | 1733/41250 [4:11:14<95:09:15,  8.67s/it][2025-04-26 08:19:51,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 08:19:51,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5711.44 | bwd_inner_microstep: 5698.31 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.27
[2025-04-26 08:19:51,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5711.45 | bwd_inner: 5698.31 | bwd_allreduce: 13.10 | step: 18.27

  4%|▍         | 1734/41250 [4:11:22<95:05:38,  8.66s/it]
                                                         
{'loss': 0.2353, 'grad_norm': 2.55314564704895, 'learning_rate': 3.9984835514213854e-05, 'epoch': 0.42}

  4%|▍         | 1734/41250 [4:11:22<95:05:38,  8.66s/it][2025-04-26 08:20:00,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 08:20:00,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.15 | bwd_microstep: 5799.13 | bwd_inner_microstep: 5659.44 | bwd_allreduce_microstep: 139.64 | step_microstep: 18.56
[2025-04-26 08:20:00,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.15 | bwd: 5799.15 | bwd_inner: 5659.44 | bwd_allreduce: 139.66 | step: 18.56

  4%|▍         | 1735/41250 [4:11:31<95:14:48,  8.68s/it]
                                                         
{'loss': 0.1891, 'grad_norm': 1.854052186012268, 'learning_rate': 3.99847743132209e-05, 'epoch': 0.42}

  4%|▍         | 1735/41250 [4:11:31<95:14:48,  8.68s/it][2025-04-26 08:20:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:20:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.13 | bwd_microstep: 5760.63 | bwd_inner_microstep: 5647.26 | bwd_allreduce_microstep: 113.33 | step_microstep: 17.82
[2025-04-26 08:20:08,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.13 | bwd: 5760.64 | bwd_inner: 5647.26 | bwd_allreduce: 113.34 | step: 17.82

  4%|▍         | 1736/41250 [4:11:40<95:15:53,  8.68s/it]
                                                         
{'loss': 0.0755, 'grad_norm': 0.8625978231430054, 'learning_rate': 3.998471298902574e-05, 'epoch': 0.42}

  4%|▍         | 1736/41250 [4:11:40<95:15:53,  8.68s/it][2025-04-26 08:20:17,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:20:17,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.12 | bwd_microstep: 5706.37 | bwd_inner_microstep: 5664.32 | bwd_allreduce_microstep: 42.00 | step_microstep: 18.53
[2025-04-26 08:20:17,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.12 | bwd: 5706.38 | bwd_inner: 5664.32 | bwd_allreduce: 42.03 | step: 18.54

  4%|▍         | 1737/41250 [4:11:48<95:03:24,  8.66s/it]
                                                         
{'loss': 0.2145, 'grad_norm': 2.3854732513427734, 'learning_rate': 3.9984651541628756e-05, 'epoch': 0.42}

  4%|▍         | 1737/41250 [4:11:48<95:03:24,  8.66s/it][2025-04-26 08:20:26,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:20:26,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5738.09 | bwd_inner_microstep: 5694.85 | bwd_allreduce_microstep: 43.20 | step_microstep: 18.15
[2025-04-26 08:20:26,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 5738.11 | bwd_inner: 5694.85 | bwd_allreduce: 43.21 | step: 18.16

  4%|▍         | 1738/41250 [4:11:57<95:05:32,  8.66s/it]
                                                         
{'loss': 0.1969, 'grad_norm': 2.129451274871826, 'learning_rate': 3.998458997103033e-05, 'epoch': 0.42}

  4%|▍         | 1738/41250 [4:11:57<95:05:32,  8.66s/it][2025-04-26 08:20:34,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:20:34,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.52 | bwd_microstep: 5700.41 | bwd_inner_microstep: 5650.20 | bwd_allreduce_microstep: 50.17 | step_microstep: 17.67
[2025-04-26 08:20:34,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.52 | bwd: 5700.42 | bwd_inner: 5650.20 | bwd_allreduce: 50.19 | step: 17.67

  4%|▍         | 1739/41250 [4:12:06<94:53:56,  8.65s/it]
                                                         
{'loss': 0.1145, 'grad_norm': 1.5475541353225708, 'learning_rate': 3.9984528277230834e-05, 'epoch': 0.42}

  4%|▍         | 1739/41250 [4:12:06<94:53:56,  8.65s/it][2025-04-26 08:20:43,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:20:43,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5688.80 | bwd_inner_microstep: 5641.58 | bwd_allreduce_microstep: 47.17 | step_microstep: 17.83
[2025-04-26 08:20:43,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5688.81 | bwd_inner: 5641.58 | bwd_allreduce: 47.19 | step: 17.83

  4%|▍         | 1740/41250 [4:12:14<94:44:11,  8.63s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.8839590549468994, 'learning_rate': 3.998446646023065e-05, 'epoch': 0.42}

  4%|▍         | 1740/41250 [4:12:14<94:44:11,  8.63s/it][2025-04-26 08:20:52,093] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:20:52,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.26 | bwd_microstep: 5728.60 | bwd_inner_microstep: 5694.69 | bwd_allreduce_microstep: 33.86 | step_microstep: 18.30
[2025-04-26 08:20:52,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.26 | bwd: 5728.61 | bwd_inner: 5694.69 | bwd_allreduce: 33.88 | step: 18.31

  4%|▍         | 1741/41250 [4:12:23<94:49:18,  8.64s/it]
                                                         
{'loss': 0.0654, 'grad_norm': 1.010994553565979, 'learning_rate': 3.998440452003016e-05, 'epoch': 0.42}

  4%|▍         | 1741/41250 [4:12:23<94:49:18,  8.64s/it][2025-04-26 08:21:00,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 08:21:00,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.38 | bwd_microstep: 5697.39 | bwd_inner_microstep: 5684.05 | bwd_allreduce_microstep: 13.29 | step_microstep: 18.90
[2025-04-26 08:21:00,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.38 | bwd: 5697.41 | bwd_inner: 5684.05 | bwd_allreduce: 13.31 | step: 18.90

  4%|▍         | 1742/41250 [4:12:31<94:46:32,  8.64s/it]
                                                         
{'loss': 0.2459, 'grad_norm': 2.1059818267822266, 'learning_rate': 3.9984342456629754e-05, 'epoch': 0.42}

  4%|▍         | 1742/41250 [4:12:31<94:46:32,  8.64s/it][2025-04-26 08:21:09,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:21:09,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.10 | bwd_microstep: 5698.99 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 55.26 | step_microstep: 18.60
[2025-04-26 08:21:09,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.10 | bwd: 5699.01 | bwd_inner: 5643.69 | bwd_allreduce: 55.28 | step: 18.60

  4%|▍         | 1743/41250 [4:12:40<94:43:24,  8.63s/it]
                                                         
{'loss': 0.2516, 'grad_norm': 2.753268003463745, 'learning_rate': 3.99842802700298e-05, 'epoch': 0.42}

  4%|▍         | 1743/41250 [4:12:40<94:43:24,  8.63s/it][2025-04-26 08:21:17,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:21:17,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.07 | bwd_microstep: 5699.19 | bwd_inner_microstep: 5633.87 | bwd_allreduce_microstep: 65.28 | step_microstep: 18.53
[2025-04-26 08:21:17,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.07 | bwd: 5699.21 | bwd_inner: 5633.87 | bwd_allreduce: 65.30 | step: 18.53

  4%|▍         | 1744/41250 [4:12:49<94:36:05,  8.62s/it]
                                                         
{'loss': 0.294, 'grad_norm': 2.3171989917755127, 'learning_rate': 3.99842179602307e-05, 'epoch': 0.42}

  4%|▍         | 1744/41250 [4:12:49<94:36:05,  8.62s/it][2025-04-26 08:21:26,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:21:26,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.98 | bwd_microstep: 5772.11 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 128.37 | step_microstep: 18.68
[2025-04-26 08:21:26,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.99 | bwd: 5772.12 | bwd_inner: 5643.69 | bwd_allreduce: 128.39 | step: 18.69

  4%|▍         | 1745/41250 [4:12:57<94:46:57,  8.64s/it]
                                                         
{'loss': 0.203, 'grad_norm': 1.4217475652694702, 'learning_rate': 3.998415552723282e-05, 'epoch': 0.42}

  4%|▍         | 1745/41250 [4:12:57<94:46:57,  8.64s/it][2025-04-26 08:21:35,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:21:35,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.22 | bwd_microstep: 5681.34 | bwd_inner_microstep: 5648.68 | bwd_allreduce_microstep: 32.62 | step_microstep: 18.32
[2025-04-26 08:21:35,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.22 | bwd: 5681.35 | bwd_inner: 5648.68 | bwd_allreduce: 32.64 | step: 18.33

  4%|▍         | 1746/41250 [4:13:06<94:41:20,  8.63s/it]
                                                         
{'loss': 0.1639, 'grad_norm': 1.7203878164291382, 'learning_rate': 3.9984092971036545e-05, 'epoch': 0.42}

  4%|▍         | 1746/41250 [4:13:06<94:41:20,  8.63s/it][2025-04-26 08:21:43,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 08:21:43,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5675.61 | bwd_inner_microstep: 5641.19 | bwd_allreduce_microstep: 34.38 | step_microstep: 19.02
[2025-04-26 08:21:43,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5675.63 | bwd_inner: 5641.19 | bwd_allreduce: 34.40 | step: 19.02

  4%|▍         | 1747/41250 [4:13:14<94:32:27,  8.62s/it]
                                                         
{'loss': 0.1397, 'grad_norm': 1.6755032539367676, 'learning_rate': 3.998403029164228e-05, 'epoch': 0.42}

  4%|▍         | 1747/41250 [4:13:14<94:32:27,  8.62s/it][2025-04-26 08:21:52,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:21:52,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.75 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5691.95 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.91
[2025-04-26 08:21:52,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.75 | bwd: 5704.89 | bwd_inner: 5691.95 | bwd_allreduce: 12.90 | step: 18.91

  4%|▍         | 1748/41250 [4:13:23<94:33:13,  8.62s/it]
                                                         
{'loss': 0.148, 'grad_norm': 1.0787557363510132, 'learning_rate': 3.9983967489050395e-05, 'epoch': 0.42}

  4%|▍         | 1748/41250 [4:13:23<94:33:13,  8.62s/it][2025-04-26 08:22:01,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:22:01,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.60 | bwd_microstep: 5780.59 | bwd_inner_microstep: 5767.77 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.33
[2025-04-26 08:22:01,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.60 | bwd: 5780.60 | bwd_inner: 5767.77 | bwd_allreduce: 12.80 | step: 18.33

  4%|▍         | 1749/41250 [4:13:32<94:59:38,  8.66s/it]
                                                         
{'loss': 0.2625, 'grad_norm': 4.030393123626709, 'learning_rate': 3.998390456326127e-05, 'epoch': 0.42}

  4%|▍         | 1749/41250 [4:13:32<94:59:38,  8.66s/it][2025-04-26 08:22:09,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:22:09,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.64 | bwd_microstep: 5740.45 | bwd_inner_microstep: 5652.86 | bwd_allreduce_microstep: 87.55 | step_microstep: 17.79
[2025-04-26 08:22:09,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.64 | bwd: 5740.47 | bwd_inner: 5652.86 | bwd_allreduce: 87.57 | step: 17.79

  4%|▍         | 1750/41250 [4:13:41<94:57:46,  8.65s/it]
                                                         
{'loss': 0.1128, 'grad_norm': 1.4836065769195557, 'learning_rate': 3.9983841514275316e-05, 'epoch': 0.42}

  4%|▍         | 1750/41250 [4:13:41<94:57:46,  8.65s/it][2025-04-26 08:22:18,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-26 08:22:18,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.49 | bwd_microstep: 5690.13 | bwd_inner_microstep: 5622.51 | bwd_allreduce_microstep: 67.56 | step_microstep: 19.13
[2025-04-26 08:22:18,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.49 | bwd: 5690.14 | bwd_inner: 5622.51 | bwd_allreduce: 67.59 | step: 19.13

  4%|▍         | 1751/41250 [4:13:49<94:46:37,  8.64s/it]
                                                         
{'loss': 0.185, 'grad_norm': 2.2692127227783203, 'learning_rate': 3.998377834209291e-05, 'epoch': 0.42}

  4%|▍         | 1751/41250 [4:13:49<94:46:37,  8.64s/it][2025-04-26 08:22:27,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 08:22:27,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.67 | bwd_microstep: 5682.78 | bwd_inner_microstep: 5637.73 | bwd_allreduce_microstep: 45.01 | step_microstep: 18.25
[2025-04-26 08:22:27,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.67 | bwd: 5682.79 | bwd_inner: 5637.73 | bwd_allreduce: 45.02 | step: 18.26

  4%|▍         | 1752/41250 [4:13:58<94:38:07,  8.63s/it]
                                                         
{'loss': 0.3132, 'grad_norm': 2.149937152862549, 'learning_rate': 3.998371504671443e-05, 'epoch': 0.42}

  4%|▍         | 1752/41250 [4:13:58<94:38:07,  8.63s/it][2025-04-26 08:22:35,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.40 | optimizer_step: 0.90
[2025-04-26 08:22:35,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.54 | bwd_microstep: 5777.79 | bwd_inner_microstep: 5764.88 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.69
[2025-04-26 08:22:35,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.54 | bwd: 5777.81 | bwd_inner: 5764.88 | bwd_allreduce: 12.88 | step: 19.69

  4%|▍         | 1753/41250 [4:14:06<95:02:03,  8.66s/it]
                                                         
{'loss': 0.3053, 'grad_norm': 2.8402040004730225, 'learning_rate': 3.998365162814029e-05, 'epoch': 0.42}

  4%|▍         | 1753/41250 [4:14:06<95:02:03,  8.66s/it][2025-04-26 08:22:44,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 08:22:44,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.04 | bwd_microstep: 5837.15 | bwd_inner_microstep: 5681.36 | bwd_allreduce_microstep: 155.74 | step_microstep: 18.45
[2025-04-26 08:22:44,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.04 | bwd: 5837.17 | bwd_inner: 5681.36 | bwd_allreduce: 155.76 | step: 18.46

  4%|▍         | 1754/41250 [4:14:15<95:22:07,  8.69s/it]
                                                         
{'loss': 0.2533, 'grad_norm': 2.742295742034912, 'learning_rate': 3.998358808637086e-05, 'epoch': 0.43}

  4%|▍         | 1754/41250 [4:14:15<95:22:07,  8.69s/it][2025-04-26 08:22:53,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 08:22:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.51 | bwd_microstep: 5725.20 | bwd_inner_microstep: 5696.74 | bwd_allreduce_microstep: 28.39 | step_microstep: 18.77
[2025-04-26 08:22:53,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.51 | bwd: 5725.21 | bwd_inner: 5696.74 | bwd_allreduce: 28.42 | step: 18.77

  4%|▍         | 1755/41250 [4:14:24<95:15:16,  8.68s/it]
                                                         
{'loss': 0.4841, 'grad_norm': 2.7329719066619873, 'learning_rate': 3.998352442140655e-05, 'epoch': 0.43}

  4%|▍         | 1755/41250 [4:14:24<95:15:16,  8.68s/it][2025-04-26 08:23:01,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 08:23:01,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.84 | bwd_microstep: 5709.91 | bwd_inner_microstep: 5696.90 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.93
[2025-04-26 08:23:01,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.84 | bwd: 5709.93 | bwd_inner: 5696.90 | bwd_allreduce: 12.98 | step: 18.93

  4%|▍         | 1756/41250 [4:14:33<95:06:37,  8.67s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 1.514112949371338, 'learning_rate': 3.998346063324773e-05, 'epoch': 0.43}

  4%|▍         | 1756/41250 [4:14:33<95:06:37,  8.67s/it][2025-04-26 08:23:10,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:23:10,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.65 | bwd_microstep: 5758.48 | bwd_inner_microstep: 5645.14 | bwd_allreduce_microstep: 113.29 | step_microstep: 18.51
[2025-04-26 08:23:10,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.65 | bwd: 5758.49 | bwd_inner: 5645.14 | bwd_allreduce: 113.31 | step: 18.51

  4%|▍         | 1757/41250 [4:14:41<95:02:43,  8.66s/it]
                                                         
{'loss': 0.0793, 'grad_norm': 1.022358775138855, 'learning_rate': 3.998339672189482e-05, 'epoch': 0.43}

  4%|▍         | 1757/41250 [4:14:41<95:02:43,  8.66s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x20840e40] moov atom not found
[08:23:13] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4...
[2025-04-26 08:23:19,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:23:19,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.24 | bwd_microstep: 5690.65 | bwd_inner_microstep: 5663.52 | bwd_allreduce_microstep: 27.08 | step_microstep: 18.36
[2025-04-26 08:23:19,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.24 | bwd: 5690.66 | bwd_inner: 5663.52 | bwd_allreduce: 27.10 | step: 18.37

  4%|▍         | 1758/41250 [4:14:50<94:50:45,  8.65s/it]
                                                         
{'loss': 0.4099, 'grad_norm': 2.182345390319824, 'learning_rate': 3.998333268734818e-05, 'epoch': 0.43}

  4%|▍         | 1758/41250 [4:14:50<94:50:45,  8.65s/it][2025-04-26 08:23:27,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:23:27,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5739.43 | bwd_inner_microstep: 5684.06 | bwd_allreduce_microstep: 55.33 | step_microstep: 18.57
[2025-04-26 08:23:27,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5739.44 | bwd_inner: 5684.06 | bwd_allreduce: 55.35 | step: 18.57

  4%|▍         | 1759/41250 [4:14:58<94:56:17,  8.65s/it]
                                                         
{'loss': 0.6182, 'grad_norm': 2.9615633487701416, 'learning_rate': 3.9983268529608244e-05, 'epoch': 0.43}

  4%|▍         | 1759/41250 [4:14:58<94:56:17,  8.65s/it][2025-04-26 08:23:36,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:23:36,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.27 | bwd_microstep: 5683.68 | bwd_inner_microstep: 5671.00 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.99
[2025-04-26 08:23:36,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.27 | bwd: 5683.69 | bwd_inner: 5671.00 | bwd_allreduce: 12.65 | step: 17.99

  4%|▍         | 1760/41250 [4:15:07<94:46:17,  8.64s/it]
                                                         
{'loss': 0.0791, 'grad_norm': 0.9827653765678406, 'learning_rate': 3.998320424867537e-05, 'epoch': 0.43}

  4%|▍         | 1760/41250 [4:15:07<94:46:17,  8.64s/it][2025-04-26 08:23:45,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:23:45,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.00 | bwd_microstep: 5746.36 | bwd_inner_microstep: 5690.94 | bwd_allreduce_microstep: 55.38 | step_microstep: 17.90
[2025-04-26 08:23:45,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.01 | bwd: 5746.37 | bwd_inner: 5690.94 | bwd_allreduce: 55.40 | step: 17.90

  4%|▍         | 1761/41250 [4:15:16<94:51:34,  8.65s/it]
                                                         
{'loss': 0.1344, 'grad_norm': 1.4806435108184814, 'learning_rate': 3.998313984454999e-05, 'epoch': 0.43}

  4%|▍         | 1761/41250 [4:15:16<94:51:34,  8.65s/it][2025-04-26 08:23:53,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:23:53,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.20 | bwd_microstep: 5740.10 | bwd_inner_microstep: 5698.37 | bwd_allreduce_microstep: 41.69 | step_microstep: 18.59
[2025-04-26 08:23:53,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.20 | bwd: 5740.11 | bwd_inner: 5698.37 | bwd_allreduce: 41.70 | step: 18.59

  4%|▍         | 1762/41250 [4:15:24<94:56:06,  8.65s/it]
                                                         
{'loss': 0.2245, 'grad_norm': 2.5587260723114014, 'learning_rate': 3.998307531723248e-05, 'epoch': 0.43}

  4%|▍         | 1762/41250 [4:15:24<94:56:06,  8.65s/it][2025-04-26 08:24:02,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:24:02,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.51 | bwd_microstep: 5729.43 | bwd_inner_microstep: 5705.75 | bwd_allreduce_microstep: 23.63 | step_microstep: 18.69
[2025-04-26 08:24:02,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.51 | bwd: 5729.45 | bwd_inner: 5705.75 | bwd_allreduce: 23.65 | step: 18.69

  4%|▍         | 1763/41250 [4:15:33<95:00:25,  8.66s/it]
                                                         
{'loss': 0.1081, 'grad_norm': 0.9904037117958069, 'learning_rate': 3.998301066672323e-05, 'epoch': 0.43}

  4%|▍         | 1763/41250 [4:15:33<95:00:25,  8.66s/it][2025-04-26 08:24:11,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.18 | optimizer_step: 1.04
[2025-04-26 08:24:11,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.98 | bwd_microstep: 6008.50 | bwd_inner_microstep: 5704.00 | bwd_allreduce_microstep: 304.45 | step_microstep: 19.01
[2025-04-26 08:24:11,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.98 | bwd: 6008.52 | bwd_inner: 5704.00 | bwd_allreduce: 304.48 | step: 19.02

  4%|▍         | 1764/41250 [4:15:42<95:56:40,  8.75s/it]
                                                         
{'loss': 0.2112, 'grad_norm': 2.5630171298980713, 'learning_rate': 3.998294589302266e-05, 'epoch': 0.43}

  4%|▍         | 1764/41250 [4:15:42<95:56:40,  8.75s/it][2025-04-26 08:24:20,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:24:20,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.32 | bwd_microstep: 5778.27 | bwd_inner_microstep: 5659.64 | bwd_allreduce_microstep: 118.58 | step_microstep: 18.22
[2025-04-26 08:24:20,016] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.32 | bwd: 5778.28 | bwd_inner: 5659.64 | bwd_allreduce: 118.60 | step: 18.22

  4%|▍         | 1765/41250 [4:15:51<95:44:05,  8.73s/it]
                                                         
{'loss': 0.1181, 'grad_norm': 0.9594731330871582, 'learning_rate': 3.998288099613115e-05, 'epoch': 0.43}

  4%|▍         | 1765/41250 [4:15:51<95:44:05,  8.73s/it][2025-04-26 08:24:28,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 08:24:28,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5706.68 | bwd_inner_microstep: 5693.46 | bwd_allreduce_microstep: 13.18 | step_microstep: 18.76
[2025-04-26 08:24:28,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5706.70 | bwd_inner: 5693.46 | bwd_allreduce: 13.20 | step: 18.76

  4%|▍         | 1766/41250 [4:15:59<95:24:51,  8.70s/it]
                                                         
{'loss': 0.0689, 'grad_norm': 0.7085291743278503, 'learning_rate': 3.998281597604912e-05, 'epoch': 0.43}

  4%|▍         | 1766/41250 [4:15:59<95:24:51,  8.70s/it][2025-04-26 08:24:37,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 08:24:37,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.44 | bwd_microstep: 5773.56 | bwd_inner_microstep: 5645.82 | bwd_allreduce_microstep: 127.69 | step_microstep: 18.86
[2025-04-26 08:24:37,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.44 | bwd: 5773.58 | bwd_inner: 5645.82 | bwd_allreduce: 127.71 | step: 18.86

  4%|▍         | 1767/41250 [4:16:08<95:23:05,  8.70s/it]
                                                         
{'loss': 0.1587, 'grad_norm': 1.0574437379837036, 'learning_rate': 3.998275083277695e-05, 'epoch': 0.43}

  4%|▍         | 1767/41250 [4:16:08<95:23:05,  8.70s/it][2025-04-26 08:24:45,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:24:45,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.98 | bwd_microstep: 5705.86 | bwd_inner_microstep: 5684.19 | bwd_allreduce_microstep: 21.62 | step_microstep: 18.29
[2025-04-26 08:24:45,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.98 | bwd: 5705.87 | bwd_inner: 5684.19 | bwd_allreduce: 21.64 | step: 18.29

  4%|▍         | 1768/41250 [4:16:17<95:12:12,  8.68s/it]
                                                         
{'loss': 0.1171, 'grad_norm': 0.818278431892395, 'learning_rate': 3.998268556631505e-05, 'epoch': 0.43}

  4%|▍         | 1768/41250 [4:16:17<95:12:12,  8.68s/it][2025-04-26 08:24:54,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:24:54,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.35 | bwd_microstep: 5765.95 | bwd_inner_microstep: 5688.82 | bwd_allreduce_microstep: 77.09 | step_microstep: 17.80
[2025-04-26 08:24:54,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.35 | bwd: 5765.96 | bwd_inner: 5688.82 | bwd_allreduce: 77.10 | step: 17.80

  4%|▍         | 1769/41250 [4:16:25<95:14:41,  8.68s/it]
                                                         
{'loss': 0.2811, 'grad_norm': 2.6457903385162354, 'learning_rate': 3.998262017666384e-05, 'epoch': 0.43}

  4%|▍         | 1769/41250 [4:16:25<95:14:41,  8.68s/it][2025-04-26 08:25:03,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:25:03,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.14 | bwd_microstep: 5709.98 | bwd_inner_microstep: 5697.20 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.28
[2025-04-26 08:25:03,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.14 | bwd: 5709.99 | bwd_inner: 5697.20 | bwd_allreduce: 12.75 | step: 18.28

  4%|▍         | 1770/41250 [4:16:34<95:08:32,  8.68s/it]
                                                         
{'loss': 0.2614, 'grad_norm': 1.9499214887619019, 'learning_rate': 3.998255466382369e-05, 'epoch': 0.43}

  4%|▍         | 1770/41250 [4:16:34<95:08:32,  8.68s/it][2025-04-26 08:25:11,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:25:11,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.73 | bwd_microstep: 5713.20 | bwd_inner_microstep: 5700.49 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.37
[2025-04-26 08:25:11,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.73 | bwd: 5713.22 | bwd_inner: 5700.49 | bwd_allreduce: 12.69 | step: 18.37

  4%|▍         | 1771/41250 [4:16:43<95:03:04,  8.67s/it]
                                                         
{'loss': 0.3308, 'grad_norm': 2.423902988433838, 'learning_rate': 3.998248902779503e-05, 'epoch': 0.43}

  4%|▍         | 1771/41250 [4:16:43<95:03:04,  8.67s/it][2025-04-26 08:25:20,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:25:20,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.67 | bwd_microstep: 5792.10 | bwd_inner_microstep: 5656.53 | bwd_allreduce_microstep: 135.53 | step_microstep: 18.52
[2025-04-26 08:25:20,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.67 | bwd: 5792.11 | bwd_inner: 5656.53 | bwd_allreduce: 135.55 | step: 18.52

  4%|▍         | 1772/41250 [4:16:51<95:10:56,  8.68s/it]
                                                         
{'loss': 0.3666, 'grad_norm': 2.698862314224243, 'learning_rate': 3.998242326857825e-05, 'epoch': 0.43}

  4%|▍         | 1772/41250 [4:16:51<95:10:56,  8.68s/it][2025-04-26 08:25:29,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.16 | optimizer_step: 1.03
[2025-04-26 08:25:29,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.94 | bwd_microstep: 5721.07 | bwd_inner_microstep: 5707.87 | bwd_allreduce_microstep: 13.15 | step_microstep: 19.37
[2025-04-26 08:25:29,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.94 | bwd: 5721.09 | bwd_inner: 5707.87 | bwd_allreduce: 13.17 | step: 19.38

  4%|▍         | 1773/41250 [4:17:00<95:05:27,  8.67s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 1.095304012298584, 'learning_rate': 3.998235738617377e-05, 'epoch': 0.43}

  4%|▍         | 1773/41250 [4:17:00<95:05:27,  8.67s/it][2025-04-26 08:25:37,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:25:37,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.77 | bwd_microstep: 5719.33 | bwd_inner_microstep: 5661.66 | bwd_allreduce_microstep: 57.63 | step_microstep: 18.44
[2025-04-26 08:25:37,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.77 | bwd: 5719.35 | bwd_inner: 5661.66 | bwd_allreduce: 57.65 | step: 18.44

  4%|▍         | 1774/41250 [4:17:09<94:58:48,  8.66s/it]
                                                         
{'loss': 0.156, 'grad_norm': 1.524095058441162, 'learning_rate': 3.998229138058197e-05, 'epoch': 0.43}

  4%|▍         | 1774/41250 [4:17:09<94:58:48,  8.66s/it][2025-04-26 08:25:46,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 08:25:46,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.44 | bwd_microstep: 5775.27 | bwd_inner_microstep: 5645.80 | bwd_allreduce_microstep: 129.42 | step_microstep: 18.80
[2025-04-26 08:25:46,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.44 | bwd: 5775.28 | bwd_inner: 5645.80 | bwd_allreduce: 129.45 | step: 18.80

  4%|▍         | 1775/41250 [4:17:17<95:00:38,  8.66s/it]
                                                         
{'loss': 0.3266, 'grad_norm': 1.556403636932373, 'learning_rate': 3.998222525180329e-05, 'epoch': 0.43}

  4%|▍         | 1775/41250 [4:17:17<95:00:38,  8.66s/it][2025-04-26 08:25:55,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:25:55,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.02 | bwd_microstep: 5782.20 | bwd_inner_microstep: 5655.04 | bwd_allreduce_microstep: 127.12 | step_microstep: 18.34
[2025-04-26 08:25:55,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.02 | bwd: 5782.21 | bwd_inner: 5655.04 | bwd_allreduce: 127.13 | step: 18.35

  4%|▍         | 1776/41250 [4:17:26<95:06:59,  8.67s/it]
                                                         
{'loss': 0.2564, 'grad_norm': 2.5343515872955322, 'learning_rate': 3.9982158999838114e-05, 'epoch': 0.43}

  4%|▍         | 1776/41250 [4:17:26<95:06:59,  8.67s/it][2025-04-26 08:26:04,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:26:04,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.28 | bwd_microstep: 5757.16 | bwd_inner_microstep: 5709.22 | bwd_allreduce_microstep: 47.90 | step_microstep: 18.38
[2025-04-26 08:26:04,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.28 | bwd: 5757.17 | bwd_inner: 5709.22 | bwd_allreduce: 47.91 | step: 18.38

  4%|▍         | 1777/41250 [4:17:35<95:11:28,  8.68s/it]
                                                         
{'loss': 0.32, 'grad_norm': 2.5761351585388184, 'learning_rate': 3.998209262468686e-05, 'epoch': 0.43}

  4%|▍         | 1777/41250 [4:17:35<95:11:28,  8.68s/it][2025-04-26 08:26:12,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:26:12,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.19 | bwd_microstep: 5756.25 | bwd_inner_microstep: 5696.02 | bwd_allreduce_microstep: 60.18 | step_microstep: 18.31
[2025-04-26 08:26:12,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.19 | bwd: 5756.26 | bwd_inner: 5696.02 | bwd_allreduce: 60.20 | step: 18.31

  4%|▍         | 1778/41250 [4:17:43<95:13:53,  8.69s/it]
                                                         
{'loss': 0.0511, 'grad_norm': 0.5761701464653015, 'learning_rate': 3.9982026126349935e-05, 'epoch': 0.43}

  4%|▍         | 1778/41250 [4:17:43<95:13:53,  8.69s/it][2025-04-26 08:26:21,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:26:21,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.71 | bwd_microstep: 5742.52 | bwd_inner_microstep: 5694.24 | bwd_allreduce_microstep: 48.24 | step_microstep: 18.41
[2025-04-26 08:26:21,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.71 | bwd: 5742.53 | bwd_inner: 5694.24 | bwd_allreduce: 48.26 | step: 18.42

  4%|▍         | 1779/41250 [4:17:52<95:12:24,  8.68s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 2.0151915550231934, 'learning_rate': 3.998195950482775e-05, 'epoch': 0.43}

  4%|▍         | 1779/41250 [4:17:52<95:12:24,  8.68s/it][2025-04-26 08:26:30,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:26:30,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.12 | bwd_microstep: 5700.55 | bwd_inner_microstep: 5687.79 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.54
[2025-04-26 08:26:30,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.12 | bwd: 5700.57 | bwd_inner: 5687.79 | bwd_allreduce: 12.73 | step: 18.55

  4%|▍         | 1780/41250 [4:18:01<95:01:21,  8.67s/it]
                                                         
{'loss': 0.1207, 'grad_norm': 1.4110183715820312, 'learning_rate': 3.9981892760120715e-05, 'epoch': 0.43}

  4%|▍         | 1780/41250 [4:18:01<95:01:21,  8.67s/it][2025-04-26 08:26:38,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:26:38,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.18 | bwd_microstep: 5714.53 | bwd_inner_microstep: 5640.45 | bwd_allreduce_microstep: 74.03 | step_microstep: 18.58
[2025-04-26 08:26:38,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.18 | bwd: 5714.54 | bwd_inner: 5640.45 | bwd_allreduce: 74.05 | step: 18.58

  4%|▍         | 1781/41250 [4:18:09<94:54:57,  8.66s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 2.228381633758545, 'learning_rate': 3.998182589222924e-05, 'epoch': 0.43}

  4%|▍         | 1781/41250 [4:18:09<94:54:57,  8.66s/it][2025-04-26 08:26:47,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:26:47,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2873.19 | bwd_microstep: 5764.11 | bwd_inner_microstep: 5751.21 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.54
[2025-04-26 08:26:47,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2873.19 | bwd: 5764.12 | bwd_inner: 5751.21 | bwd_allreduce: 12.87 | step: 18.55

  4%|▍         | 1782/41250 [4:18:18<95:06:52,  8.68s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.3426785469055176, 'learning_rate': 3.998175890115375e-05, 'epoch': 0.43}

  4%|▍         | 1782/41250 [4:18:18<95:06:52,  8.68s/it][2025-04-26 08:26:56,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:26:56,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.05 | bwd_microstep: 5743.71 | bwd_inner_microstep: 5714.81 | bwd_allreduce_microstep: 28.86 | step_microstep: 17.98
[2025-04-26 08:26:56,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.05 | bwd: 5743.72 | bwd_inner: 5714.81 | bwd_allreduce: 28.88 | step: 17.98

  4%|▍         | 1783/41250 [4:18:27<95:07:29,  8.68s/it]
                                                         
{'loss': 0.192, 'grad_norm': 1.9112476110458374, 'learning_rate': 3.998169178689463e-05, 'epoch': 0.43}

  4%|▍         | 1783/41250 [4:18:27<95:07:29,  8.68s/it][2025-04-26 08:27:04,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 08:27:04,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.67 | bwd_microstep: 5700.19 | bwd_inner_microstep: 5649.93 | bwd_allreduce_microstep: 50.22 | step_microstep: 19.01
[2025-04-26 08:27:04,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.67 | bwd: 5700.20 | bwd_inner: 5649.93 | bwd_allreduce: 50.23 | step: 19.02

  4%|▍         | 1784/41250 [4:18:35<94:54:15,  8.66s/it]
                                                         
{'loss': 0.2105, 'grad_norm': 1.6928133964538574, 'learning_rate': 3.998162454945232e-05, 'epoch': 0.43}

  4%|▍         | 1784/41250 [4:18:35<94:54:15,  8.66s/it][2025-04-26 08:27:13,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:27:13,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.38 | bwd_microstep: 5694.62 | bwd_inner_microstep: 5681.91 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.07
[2025-04-26 08:27:13,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.38 | bwd: 5694.63 | bwd_inner: 5681.91 | bwd_allreduce: 12.68 | step: 18.07

  4%|▍         | 1785/41250 [4:18:44<94:47:15,  8.65s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.604779839515686, 'learning_rate': 3.998155718882722e-05, 'epoch': 0.43}

  4%|▍         | 1785/41250 [4:18:44<94:47:15,  8.65s/it][2025-04-26 08:27:21,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 08:27:21,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.61 | bwd_microstep: 5705.31 | bwd_inner_microstep: 5642.79 | bwd_allreduce_microstep: 62.48 | step_microstep: 18.04
[2025-04-26 08:27:21,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.61 | bwd: 5705.33 | bwd_inner: 5642.79 | bwd_allreduce: 62.50 | step: 18.04

  4%|▍         | 1786/41250 [4:18:53<94:39:37,  8.64s/it]
                                                         
{'loss': 0.1084, 'grad_norm': 2.5850157737731934, 'learning_rate': 3.998148970501975e-05, 'epoch': 0.43}

  4%|▍         | 1786/41250 [4:18:53<94:39:37,  8.64s/it][2025-04-26 08:27:30,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:27:30,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.72 | bwd_microstep: 5684.89 | bwd_inner_microstep: 5634.37 | bwd_allreduce_microstep: 50.48 | step_microstep: 18.32
[2025-04-26 08:27:30,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.72 | bwd: 5684.90 | bwd_inner: 5634.37 | bwd_allreduce: 50.49 | step: 18.32

  4%|▍         | 1787/41250 [4:19:01<94:31:35,  8.62s/it]
                                                         
{'loss': 0.5156, 'grad_norm': 2.656057119369507, 'learning_rate': 3.9981422098030325e-05, 'epoch': 0.43}

  4%|▍         | 1787/41250 [4:19:01<94:31:35,  8.62s/it][2025-04-26 08:27:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:27:39,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.39 | bwd_microstep: 5755.51 | bwd_inner_microstep: 5640.75 | bwd_allreduce_microstep: 114.71 | step_microstep: 18.37
[2025-04-26 08:27:39,203] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.39 | bwd: 5755.52 | bwd_inner: 5640.75 | bwd_allreduce: 114.73 | step: 18.38

  4%|▍         | 1788/41250 [4:19:10<94:44:00,  8.64s/it]
                                                         
{'loss': 0.1547, 'grad_norm': 1.659600853919983, 'learning_rate': 3.9981354367859366e-05, 'epoch': 0.43}

  4%|▍         | 1788/41250 [4:19:10<94:44:00,  8.64s/it][2025-04-26 08:27:47,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:27:47,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.73 | bwd_microstep: 5785.67 | bwd_inner_microstep: 5772.86 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.95
[2025-04-26 08:27:47,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.73 | bwd: 5785.69 | bwd_inner: 5772.87 | bwd_allreduce: 12.78 | step: 17.96

  4%|▍         | 1789/41250 [4:19:19<95:03:26,  8.67s/it]
                                                         
{'loss': 0.1023, 'grad_norm': 0.798225998878479, 'learning_rate': 3.9981286514507285e-05, 'epoch': 0.43}

  4%|▍         | 1789/41250 [4:19:19<95:03:26,  8.67s/it][2025-04-26 08:27:56,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:27:56,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.43 | bwd_microstep: 5696.84 | bwd_inner_microstep: 5648.17 | bwd_allreduce_microstep: 48.62 | step_microstep: 18.09
[2025-04-26 08:27:56,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.43 | bwd: 5696.85 | bwd_inner: 5648.17 | bwd_allreduce: 48.64 | step: 18.09

  4%|▍         | 1790/41250 [4:19:27<94:52:54,  8.66s/it]
                                                         
{'loss': 0.103, 'grad_norm': 1.43826162815094, 'learning_rate': 3.998121853797451e-05, 'epoch': 0.43}

  4%|▍         | 1790/41250 [4:19:27<94:52:54,  8.66s/it][2025-04-26 08:28:05,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:28:05,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.48 | bwd_microstep: 5704.63 | bwd_inner_microstep: 5691.88 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.69
[2025-04-26 08:28:05,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.48 | bwd: 5704.65 | bwd_inner: 5691.88 | bwd_allreduce: 12.72 | step: 18.69

  4%|▍         | 1791/41250 [4:19:36<94:49:11,  8.65s/it]
                                                         
{'loss': 0.1485, 'grad_norm': 0.9800845384597778, 'learning_rate': 3.998115043826144e-05, 'epoch': 0.43}

  4%|▍         | 1791/41250 [4:19:36<94:49:11,  8.65s/it][2025-04-26 08:28:13,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 08:28:13,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.36 | bwd_microstep: 5695.19 | bwd_inner_microstep: 5631.13 | bwd_allreduce_microstep: 64.01 | step_microstep: 18.60
[2025-04-26 08:28:13,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.36 | bwd: 5695.20 | bwd_inner: 5631.13 | bwd_allreduce: 64.03 | step: 18.60

  4%|▍         | 1792/41250 [4:19:44<94:37:50,  8.63s/it]
                                                         
{'loss': 0.3308, 'grad_norm': 1.9323539733886719, 'learning_rate': 3.998108221536852e-05, 'epoch': 0.43}

  4%|▍         | 1792/41250 [4:19:44<94:37:50,  8.63s/it][2025-04-26 08:28:22,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.07 | optimizer_step: 1.22
[2025-04-26 08:28:22,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.67 | bwd_microstep: 5718.69 | bwd_inner_microstep: 5705.68 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.96
[2025-04-26 08:28:22,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.67 | bwd: 5718.71 | bwd_inner: 5705.68 | bwd_allreduce: 12.98 | step: 19.97

  4%|▍         | 1793/41250 [4:19:53<94:40:22,  8.64s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 1.489626407623291, 'learning_rate': 3.998101386929615e-05, 'epoch': 0.43}

  4%|▍         | 1793/41250 [4:19:53<94:40:22,  8.64s/it][2025-04-26 08:28:31,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:28:31,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.62 | bwd_microstep: 5705.86 | bwd_inner_microstep: 5693.07 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.56
[2025-04-26 08:28:31,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.62 | bwd: 5705.88 | bwd_inner: 5693.07 | bwd_allreduce: 12.76 | step: 18.56

  4%|▍         | 1794/41250 [4:20:02<94:39:53,  8.64s/it]
                                                         
{'loss': 0.1204, 'grad_norm': 1.613303542137146, 'learning_rate': 3.998094540004477e-05, 'epoch': 0.43}

  4%|▍         | 1794/41250 [4:20:02<94:39:53,  8.64s/it][2025-04-26 08:28:39,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:28:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2924.21 | bwd_microstep: 5865.63 | bwd_inner_microstep: 5852.73 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.69
[2025-04-26 08:28:39,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2924.21 | bwd: 5865.65 | bwd_inner: 5852.73 | bwd_allreduce: 12.88 | step: 18.70

  4%|▍         | 1795/41250 [4:20:11<95:26:00,  8.71s/it]
                                                         
{'loss': 0.2218, 'grad_norm': 1.5148004293441772, 'learning_rate': 3.998087680761478e-05, 'epoch': 0.44}

  4%|▍         | 1795/41250 [4:20:11<95:26:00,  8.71s/it][2025-04-26 08:28:48,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:28:48,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.19 | bwd_microstep: 5830.62 | bwd_inner_microstep: 5696.15 | bwd_allreduce_microstep: 134.42 | step_microstep: 18.71
[2025-04-26 08:28:48,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.19 | bwd: 5830.63 | bwd_inner: 5696.15 | bwd_allreduce: 134.44 | step: 18.71

  4%|▍         | 1796/41250 [4:20:19<95:34:58,  8.72s/it]
                                                         
{'loss': 0.122, 'grad_norm': 0.7592854499816895, 'learning_rate': 3.998080809200662e-05, 'epoch': 0.44}

  4%|▍         | 1796/41250 [4:20:19<95:34:58,  8.72s/it][2025-04-26 08:28:57,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:28:57,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5735.79 | bwd_inner_microstep: 5692.80 | bwd_allreduce_microstep: 42.95 | step_microstep: 18.46
[2025-04-26 08:28:57,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5735.80 | bwd_inner: 5692.80 | bwd_allreduce: 42.97 | step: 18.46

  4%|▍         | 1797/41250 [4:20:28<95:24:53,  8.71s/it]
                                                         
{'loss': 0.2137, 'grad_norm': 3.0923759937286377, 'learning_rate': 3.9980739253220704e-05, 'epoch': 0.44}

  4%|▍         | 1797/41250 [4:20:28<95:24:53,  8.71s/it][2025-04-26 08:29:05,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.15
[2025-04-26 08:29:05,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.73 | bwd_microstep: 5678.35 | bwd_inner_microstep: 5633.58 | bwd_allreduce_microstep: 44.72 | step_microstep: 19.05
[2025-04-26 08:29:05,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.73 | bwd: 5678.37 | bwd_inner: 5633.58 | bwd_allreduce: 44.74 | step: 19.06

  4%|▍         | 1798/41250 [4:20:37<94:59:36,  8.67s/it]
                                                         
{'loss': 0.3065, 'grad_norm': 1.4122267961502075, 'learning_rate': 3.998067029125746e-05, 'epoch': 0.44}

  4%|▍         | 1798/41250 [4:20:37<94:59:36,  8.67s/it][2025-04-26 08:29:14,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 08:29:14,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.57 | bwd_microstep: 5779.92 | bwd_inner_microstep: 5655.44 | bwd_allreduce_microstep: 124.44 | step_microstep: 18.76
[2025-04-26 08:29:14,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.58 | bwd: 5779.93 | bwd_inner: 5655.44 | bwd_allreduce: 124.45 | step: 18.76

  4%|▍         | 1799/41250 [4:20:45<95:03:36,  8.67s/it]
                                                         
{'loss': 0.1292, 'grad_norm': 1.2515558004379272, 'learning_rate': 3.998060120611732e-05, 'epoch': 0.44}

  4%|▍         | 1799/41250 [4:20:45<95:03:36,  8.67s/it][2025-04-26 08:29:23,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 08:29:23,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5716.10 | bwd_inner_microstep: 5703.43 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.64
[2025-04-26 08:29:23,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5716.12 | bwd_inner: 5703.43 | bwd_allreduce: 12.65 | step: 18.64

  4%|▍         | 1800/41250 [4:20:54<94:58:11,  8.67s/it]
                                                         
{'loss': 0.1464, 'grad_norm': 1.9692459106445312, 'learning_rate': 3.9980531997800705e-05, 'epoch': 0.44}

  4%|▍         | 1800/41250 [4:20:54<94:58:11,  8.67s/it][2025-04-26 08:29:31,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:29:31,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.38 | bwd_microstep: 5714.85 | bwd_inner_microstep: 5702.11 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.67
[2025-04-26 08:29:31,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.38 | bwd: 5714.87 | bwd_inner: 5702.11 | bwd_allreduce: 12.72 | step: 18.68

  4%|▍         | 1801/41250 [4:21:03<94:53:25,  8.66s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 0.9789150953292847, 'learning_rate': 3.9980462666308036e-05, 'epoch': 0.44}

  4%|▍         | 1801/41250 [4:21:03<94:53:25,  8.66s/it][2025-04-26 08:29:40,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 08:29:40,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5764.97 | bwd_inner_microstep: 5652.57 | bwd_allreduce_microstep: 112.35 | step_microstep: 18.71
[2025-04-26 08:29:40,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5764.99 | bwd_inner: 5652.57 | bwd_allreduce: 112.38 | step: 18.71

  4%|▍         | 1802/41250 [4:21:11<94:54:37,  8.66s/it]
                                                         
{'loss': 0.2806, 'grad_norm': 1.6361085176467896, 'learning_rate': 3.998039321163974e-05, 'epoch': 0.44}

  4%|▍         | 1802/41250 [4:21:11<94:54:37,  8.66s/it][2025-04-26 08:29:49,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:29:49,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.89 | bwd_microstep: 5895.07 | bwd_inner_microstep: 5648.51 | bwd_allreduce_microstep: 246.52 | step_microstep: 18.62
[2025-04-26 08:29:49,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.89 | bwd: 5895.08 | bwd_inner: 5648.51 | bwd_allreduce: 246.53 | step: 18.62

  4%|▍         | 1803/41250 [4:21:20<95:24:34,  8.71s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 1.6931920051574707, 'learning_rate': 3.9980323633796255e-05, 'epoch': 0.44}

  4%|▍         | 1803/41250 [4:21:20<95:24:34,  8.71s/it][2025-04-26 08:29:58,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:29:58,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5762.81 | bwd_inner_microstep: 5650.15 | bwd_allreduce_microstep: 112.62 | step_microstep: 18.68
[2025-04-26 08:29:58,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5762.82 | bwd_inner: 5650.15 | bwd_allreduce: 112.63 | step: 18.68

  4%|▍         | 1804/41250 [4:21:29<95:17:52,  8.70s/it]
                                                         
{'loss': 0.2043, 'grad_norm': 2.1907832622528076, 'learning_rate': 3.9980253932778e-05, 'epoch': 0.44}

  4%|▍         | 1804/41250 [4:21:29<95:17:52,  8.70s/it][2025-04-26 08:30:06,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:30:06,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5747.00 | bwd_inner_microstep: 5705.70 | bwd_allreduce_microstep: 41.25 | step_microstep: 18.42
[2025-04-26 08:30:06,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5747.01 | bwd_inner: 5705.70 | bwd_allreduce: 41.27 | step: 18.42

  4%|▍         | 1805/41250 [4:21:37<95:14:07,  8.69s/it]
                                                         
{'loss': 0.2309, 'grad_norm': 3.455983877182007, 'learning_rate': 3.998018410858542e-05, 'epoch': 0.44}

  4%|▍         | 1805/41250 [4:21:37<95:14:07,  8.69s/it][2025-04-26 08:30:15,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 08:30:15,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.46 | bwd_microstep: 5771.97 | bwd_inner_microstep: 5658.91 | bwd_allreduce_microstep: 113.02 | step_microstep: 18.62
[2025-04-26 08:30:15,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.46 | bwd: 5771.98 | bwd_inner: 5658.91 | bwd_allreduce: 113.03 | step: 18.62

  4%|▍         | 1806/41250 [4:21:46<95:11:41,  8.69s/it]
                                                         
{'loss': 0.1258, 'grad_norm': 1.2047128677368164, 'learning_rate': 3.998011416121893e-05, 'epoch': 0.44}

  4%|▍         | 1806/41250 [4:21:46<95:11:41,  8.69s/it][2025-04-26 08:30:24,072] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:30:24,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.72 | bwd_microstep: 5708.25 | bwd_inner_microstep: 5667.80 | bwd_allreduce_microstep: 40.40 | step_microstep: 18.40
[2025-04-26 08:30:24,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.73 | bwd: 5708.26 | bwd_inner: 5667.80 | bwd_allreduce: 40.42 | step: 18.40

  4%|▍         | 1807/41250 [4:21:55<94:58:59,  8.67s/it]
                                                         
{'loss': 0.0741, 'grad_norm': 1.4008817672729492, 'learning_rate': 3.9980044090678964e-05, 'epoch': 0.44}

  4%|▍         | 1807/41250 [4:21:55<94:58:59,  8.67s/it][2025-04-26 08:30:32,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:30:32,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.98 | bwd_microstep: 5790.12 | bwd_inner_microstep: 5777.42 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.16
[2025-04-26 08:30:32,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.98 | bwd: 5790.13 | bwd_inner: 5777.42 | bwd_allreduce: 12.67 | step: 18.17

  4%|▍         | 1808/41250 [4:22:04<95:18:39,  8.70s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.6159281730651855, 'learning_rate': 3.997997389696595e-05, 'epoch': 0.44}

  4%|▍         | 1808/41250 [4:22:04<95:18:39,  8.70s/it][2025-04-26 08:30:41,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.09
[2025-04-26 08:30:41,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5779.03 | bwd_inner_microstep: 5657.74 | bwd_allreduce_microstep: 121.24 | step_microstep: 18.75
[2025-04-26 08:30:41,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5779.04 | bwd_inner: 5657.74 | bwd_allreduce: 121.25 | step: 18.75

  4%|▍         | 1809/41250 [4:22:12<95:17:24,  8.70s/it]
                                                         
{'loss': 0.0948, 'grad_norm': 1.1012812852859497, 'learning_rate': 3.997990358008034e-05, 'epoch': 0.44}

  4%|▍         | 1809/41250 [4:22:12<95:17:24,  8.70s/it][2025-04-26 08:30:50,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:30:50,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.65 | bwd_microstep: 5721.56 | bwd_inner_microstep: 5659.08 | bwd_allreduce_microstep: 62.43 | step_microstep: 18.31
[2025-04-26 08:30:50,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.65 | bwd: 5721.57 | bwd_inner: 5659.08 | bwd_allreduce: 62.45 | step: 18.31

  4%|▍         | 1810/41250 [4:22:21<95:03:46,  8.68s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 2.204547166824341, 'learning_rate': 3.9979833140022545e-05, 'epoch': 0.44}

  4%|▍         | 1810/41250 [4:22:21<95:03:46,  8.68s/it][2025-04-26 08:30:58,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.18 | optimizer_step: 1.17
[2025-04-26 08:30:58,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.61 | bwd_microstep: 5769.52 | bwd_inner_microstep: 5688.91 | bwd_allreduce_microstep: 80.55 | step_microstep: 19.51
[2025-04-26 08:30:58,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.61 | bwd: 5769.54 | bwd_inner: 5688.91 | bwd_allreduce: 80.58 | step: 19.51

  4%|▍         | 1811/41250 [4:22:30<95:05:41,  8.68s/it]
                                                         
{'loss': 0.0696, 'grad_norm': 0.988802433013916, 'learning_rate': 3.997976257679301e-05, 'epoch': 0.44}

  4%|▍         | 1811/41250 [4:22:30<95:05:41,  8.68s/it][2025-04-26 08:31:07,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:31:07,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.30 | bwd_microstep: 5709.66 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.22
[2025-04-26 08:31:07,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.31 | bwd: 5709.67 | bwd_inner: 5696.99 | bwd_allreduce: 12.65 | step: 18.22

  4%|▍         | 1812/41250 [4:22:38<95:00:12,  8.67s/it]
                                                         
{'loss': 0.1717, 'grad_norm': 1.8082737922668457, 'learning_rate': 3.997969189039217e-05, 'epoch': 0.44}

  4%|▍         | 1812/41250 [4:22:38<95:00:12,  8.67s/it][2025-04-26 08:31:16,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 08:31:16,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.75 | bwd_microstep: 5705.72 | bwd_inner_microstep: 5692.37 | bwd_allreduce_microstep: 13.30 | step_microstep: 18.64
[2025-04-26 08:31:16,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.75 | bwd: 5705.73 | bwd_inner: 5692.37 | bwd_allreduce: 13.32 | step: 18.64

  4%|▍         | 1813/41250 [4:22:47<94:53:58,  8.66s/it]
                                                         
{'loss': 0.344, 'grad_norm': 1.9521198272705078, 'learning_rate': 3.997962108082046e-05, 'epoch': 0.44}

  4%|▍         | 1813/41250 [4:22:47<94:53:58,  8.66s/it][2025-04-26 08:31:24,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-26 08:31:24,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.76 | bwd_microstep: 5797.63 | bwd_inner_microstep: 5784.91 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.75
[2025-04-26 08:31:24,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.76 | bwd: 5797.65 | bwd_inner: 5784.91 | bwd_allreduce: 12.70 | step: 18.75

  4%|▍         | 1814/41250 [4:22:56<95:15:27,  8.70s/it]
                                                         
{'loss': 0.1215, 'grad_norm': 1.2008168697357178, 'learning_rate': 3.9979550148078316e-05, 'epoch': 0.44}

  4%|▍         | 1814/41250 [4:22:56<95:15:27,  8.70s/it][2025-04-26 08:31:33,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:31:33,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.04 | bwd_microstep: 5783.43 | bwd_inner_microstep: 5689.49 | bwd_allreduce_microstep: 93.90 | step_microstep: 18.45
[2025-04-26 08:31:33,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.05 | bwd: 5783.45 | bwd_inner: 5689.49 | bwd_allreduce: 93.92 | step: 18.46

  4%|▍         | 1815/41250 [4:23:04<95:18:44,  8.70s/it]
                                                         
{'loss': 0.2427, 'grad_norm': 2.1723902225494385, 'learning_rate': 3.997947909216617e-05, 'epoch': 0.44}

  4%|▍         | 1815/41250 [4:23:04<95:18:44,  8.70s/it][2025-04-26 08:31:42,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 08:31:42,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.70 | bwd_microstep: 5743.10 | bwd_inner_microstep: 5654.34 | bwd_allreduce_microstep: 88.71 | step_microstep: 18.86
[2025-04-26 08:31:42,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.70 | bwd: 5743.11 | bwd_inner: 5654.34 | bwd_allreduce: 88.73 | step: 18.86

  4%|▍         | 1816/41250 [4:23:13<95:07:27,  8.68s/it]
                                                         
{'loss': 0.1709, 'grad_norm': 2.4004969596862793, 'learning_rate': 3.997940791308447e-05, 'epoch': 0.44}

  4%|▍         | 1816/41250 [4:23:13<95:07:27,  8.68s/it][2025-04-26 08:31:50,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:31:50,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.78 | bwd_microstep: 5711.62 | bwd_inner_microstep: 5698.78 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.94
[2025-04-26 08:31:50,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.78 | bwd: 5711.64 | bwd_inner: 5698.78 | bwd_allreduce: 12.81 | step: 18.95

  4%|▍         | 1817/41250 [4:23:22<94:59:32,  8.67s/it]
                                                         
{'loss': 0.0836, 'grad_norm': 0.8479166030883789, 'learning_rate': 3.997933661083365e-05, 'epoch': 0.44}

  4%|▍         | 1817/41250 [4:23:22<94:59:32,  8.67s/it][2025-04-26 08:31:59,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:31:59,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.41 | bwd_microstep: 5713.39 | bwd_inner_microstep: 5662.32 | bwd_allreduce_microstep: 51.02 | step_microstep: 18.80
[2025-04-26 08:31:59,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.41 | bwd: 5713.41 | bwd_inner: 5662.32 | bwd_allreduce: 51.04 | step: 18.80

  4%|▍         | 1818/41250 [4:23:30<94:50:59,  8.66s/it]
                                                         
{'loss': 0.1675, 'grad_norm': 1.5596297979354858, 'learning_rate': 3.997926518541415e-05, 'epoch': 0.44}

  4%|▍         | 1818/41250 [4:23:30<94:50:59,  8.66s/it][2025-04-26 08:32:08,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 08:32:08,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.74 | bwd_microstep: 5727.98 | bwd_inner_microstep: 5715.09 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.95
[2025-04-26 08:32:08,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.74 | bwd: 5727.99 | bwd_inner: 5715.09 | bwd_allreduce: 12.86 | step: 18.95

  4%|▍         | 1819/41250 [4:23:39<94:51:33,  8.66s/it]
                                                         
{'loss': 0.1888, 'grad_norm': 1.903638482093811, 'learning_rate': 3.9979193636826406e-05, 'epoch': 0.44}

  4%|▍         | 1819/41250 [4:23:39<94:51:33,  8.66s/it][2025-04-26 08:32:16,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:32:16,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.23 | bwd_microstep: 5698.24 | bwd_inner_microstep: 5655.77 | bwd_allreduce_microstep: 42.42 | step_microstep: 18.54
[2025-04-26 08:32:16,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.23 | bwd: 5698.25 | bwd_inner: 5655.77 | bwd_allreduce: 42.44 | step: 18.55

  4%|▍         | 1820/41250 [4:23:48<94:40:22,  8.64s/it]
                                                         
{'loss': 0.0774, 'grad_norm': 1.472359299659729, 'learning_rate': 3.997912196507087e-05, 'epoch': 0.44}

  4%|▍         | 1820/41250 [4:23:48<94:40:22,  8.64s/it][2025-04-26 08:32:25,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:32:25,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.82 | bwd_microstep: 5718.81 | bwd_inner_microstep: 5706.16 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.66
[2025-04-26 08:32:25,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.82 | bwd: 5718.82 | bwd_inner: 5706.16 | bwd_allreduce: 12.62 | step: 17.67

  4%|▍         | 1821/41250 [4:23:56<94:41:45,  8.65s/it]
                                                         
{'loss': 0.3295, 'grad_norm': 2.4628520011901855, 'learning_rate': 3.997905017014796e-05, 'epoch': 0.44}

  4%|▍         | 1821/41250 [4:23:56<94:41:45,  8.65s/it][2025-04-26 08:32:34,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:32:34,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.99 | bwd_microstep: 5716.79 | bwd_inner_microstep: 5704.25 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.06
[2025-04-26 08:32:34,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.99 | bwd: 5716.80 | bwd_inner: 5704.25 | bwd_allreduce: 12.51 | step: 18.06

  4%|▍         | 1822/41250 [4:24:05<94:44:14,  8.65s/it]
                                                         
{'loss': 0.1218, 'grad_norm': 1.8370047807693481, 'learning_rate': 3.997897825205815e-05, 'epoch': 0.44}

  4%|▍         | 1822/41250 [4:24:05<94:44:14,  8.65s/it][2025-04-26 08:32:42,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:32:42,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.59 | bwd_microstep: 5743.36 | bwd_inner_microstep: 5687.46 | bwd_allreduce_microstep: 55.86 | step_microstep: 18.29
[2025-04-26 08:32:42,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.59 | bwd: 5743.38 | bwd_inner: 5687.46 | bwd_allreduce: 55.87 | step: 18.30

  4%|▍         | 1823/41250 [4:24:13<94:50:30,  8.66s/it]
                                                         
{'loss': 0.0861, 'grad_norm': 1.2508900165557861, 'learning_rate': 3.997890621080187e-05, 'epoch': 0.44}

  4%|▍         | 1823/41250 [4:24:14<94:50:30,  8.66s/it][2025-04-26 08:32:51,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 08:32:51,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.90 | bwd_microstep: 5697.21 | bwd_inner_microstep: 5647.94 | bwd_allreduce_microstep: 49.23 | step_microstep: 18.36
[2025-04-26 08:32:51,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.90 | bwd: 5697.23 | bwd_inner: 5647.94 | bwd_allreduce: 49.25 | step: 18.36

  4%|▍         | 1824/41250 [4:24:22<94:41:54,  8.65s/it]
                                                         
{'loss': 0.1512, 'grad_norm': 0.6553843021392822, 'learning_rate': 3.9978834046379564e-05, 'epoch': 0.44}

  4%|▍         | 1824/41250 [4:24:22<94:41:54,  8.65s/it][2025-04-26 08:33:00,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 08:33:00,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.03 | bwd_microstep: 5705.30 | bwd_inner_microstep: 5649.36 | bwd_allreduce_microstep: 55.90 | step_microstep: 18.68
[2025-04-26 08:33:00,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.03 | bwd: 5705.31 | bwd_inner: 5649.36 | bwd_allreduce: 55.92 | step: 18.68

  4%|▍         | 1825/41250 [4:24:31<94:35:33,  8.64s/it]
                                                         
{'loss': 0.3575, 'grad_norm': 1.9339463710784912, 'learning_rate': 3.997876175879166e-05, 'epoch': 0.44}

  4%|▍         | 1825/41250 [4:24:31<94:35:33,  8.64s/it][2025-04-26 08:33:08,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:33:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.19 | bwd_microstep: 5718.94 | bwd_inner_microstep: 5706.30 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.23
[2025-04-26 08:33:08,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5718.95 | bwd_inner: 5706.30 | bwd_allreduce: 12.61 | step: 18.23

  4%|▍         | 1826/41250 [4:24:39<94:40:04,  8.64s/it]
                                                         
{'loss': 0.1729, 'grad_norm': 1.435359001159668, 'learning_rate': 3.9978689348038635e-05, 'epoch': 0.44}

  4%|▍         | 1826/41250 [4:24:39<94:40:04,  8.64s/it][2025-04-26 08:33:17,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:33:17,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.15 | bwd_microstep: 5764.66 | bwd_inner_microstep: 5654.00 | bwd_allreduce_microstep: 110.60 | step_microstep: 18.40
[2025-04-26 08:33:17,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.15 | bwd: 5764.68 | bwd_inner: 5654.00 | bwd_allreduce: 110.62 | step: 18.40

  4%|▍         | 1827/41250 [4:24:48<94:46:04,  8.65s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 1.7618335485458374, 'learning_rate': 3.9978616814120914e-05, 'epoch': 0.44}

  4%|▍         | 1827/41250 [4:24:48<94:46:04,  8.65s/it][2025-04-26 08:33:26,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 08:33:26,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5772.49 | bwd_inner_microstep: 5660.99 | bwd_allreduce_microstep: 111.45 | step_microstep: 18.96
[2025-04-26 08:33:26,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5772.50 | bwd_inner: 5660.99 | bwd_allreduce: 111.47 | step: 18.96

  4%|▍         | 1828/41250 [4:24:57<94:51:42,  8.66s/it]
                                                         
{'loss': 0.1183, 'grad_norm': 1.3568469285964966, 'learning_rate': 3.997854415703895e-05, 'epoch': 0.44}

  4%|▍         | 1828/41250 [4:24:57<94:51:42,  8.66s/it][2025-04-26 08:33:34,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:33:34,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.78 | bwd_microstep: 5764.36 | bwd_inner_microstep: 5636.96 | bwd_allreduce_microstep: 127.36 | step_microstep: 18.48
[2025-04-26 08:33:34,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.78 | bwd: 5764.37 | bwd_inner: 5636.96 | bwd_allreduce: 127.38 | step: 18.48

  4%|▍         | 1829/41250 [4:25:05<94:50:16,  8.66s/it]
                                                         
{'loss': 0.1076, 'grad_norm': 1.3893717527389526, 'learning_rate': 3.997847137679319e-05, 'epoch': 0.44}

  4%|▍         | 1829/41250 [4:25:05<94:50:16,  8.66s/it][2025-04-26 08:33:43,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:33:43,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5732.16 | bwd_inner_microstep: 5675.37 | bwd_allreduce_microstep: 56.74 | step_microstep: 18.84
[2025-04-26 08:33:43,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5732.17 | bwd_inner: 5675.37 | bwd_allreduce: 56.76 | step: 18.84

  4%|▍         | 1830/41250 [4:25:14<94:50:55,  8.66s/it]
                                                         
{'loss': 0.0761, 'grad_norm': 0.9101067781448364, 'learning_rate': 3.9978398473384086e-05, 'epoch': 0.44}

  4%|▍         | 1830/41250 [4:25:14<94:50:55,  8.66s/it][2025-04-26 08:33:51,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 08:33:51,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.51 | bwd_microstep: 5689.56 | bwd_inner_microstep: 5645.15 | bwd_allreduce_microstep: 44.36 | step_microstep: 18.88
[2025-04-26 08:33:51,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.51 | bwd: 5689.57 | bwd_inner: 5645.15 | bwd_allreduce: 44.38 | step: 18.88

  4%|▍         | 1831/41250 [4:25:23<94:40:14,  8.65s/it]
                                                         
{'loss': 0.068, 'grad_norm': 1.4745275974273682, 'learning_rate': 3.997832544681209e-05, 'epoch': 0.44}

  4%|▍         | 1831/41250 [4:25:23<94:40:14,  8.65s/it][2025-04-26 08:34:00,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:34:00,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.41 | bwd_microstep: 5721.81 | bwd_inner_microstep: 5709.04 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.50
[2025-04-26 08:34:00,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5721.82 | bwd_inner: 5709.04 | bwd_allreduce: 12.74 | step: 18.51

  4%|▍         | 1832/41250 [4:25:31<94:42:08,  8.65s/it]
                                                         
{'loss': 0.3878, 'grad_norm': 2.8596854209899902, 'learning_rate': 3.9978252297077635e-05, 'epoch': 0.44}

  4%|▍         | 1832/41250 [4:25:31<94:42:08,  8.65s/it][2025-04-26 08:34:09,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 08:34:09,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.14 | bwd_microstep: 5694.73 | bwd_inner_microstep: 5648.88 | bwd_allreduce_microstep: 45.79 | step_microstep: 18.68
[2025-04-26 08:34:09,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.14 | bwd: 5694.74 | bwd_inner: 5648.88 | bwd_allreduce: 45.81 | step: 18.68

  4%|▍         | 1833/41250 [4:25:40<94:32:20,  8.63s/it]
                                                         
{'loss': 0.0824, 'grad_norm': 1.3799681663513184, 'learning_rate': 3.997817902418119e-05, 'epoch': 0.44}

  4%|▍         | 1833/41250 [4:25:40<94:32:20,  8.63s/it][2025-04-26 08:34:17,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:34:17,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.92 | bwd_microstep: 5763.32 | bwd_inner_microstep: 5640.92 | bwd_allreduce_microstep: 122.36 | step_microstep: 18.29
[2025-04-26 08:34:17,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.92 | bwd: 5763.33 | bwd_inner: 5640.92 | bwd_allreduce: 122.37 | step: 18.30

  4%|▍         | 1834/41250 [4:25:49<94:38:19,  8.64s/it]
                                                         
{'loss': 0.3244, 'grad_norm': 2.1319644451141357, 'learning_rate': 3.99781056281232e-05, 'epoch': 0.44}

  4%|▍         | 1834/41250 [4:25:49<94:38:19,  8.64s/it][2025-04-26 08:34:26,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-26 08:34:26,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5717.95 | bwd_inner_microstep: 5682.14 | bwd_allreduce_microstep: 35.76 | step_microstep: 18.81
[2025-04-26 08:34:26,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5717.96 | bwd_inner: 5682.14 | bwd_allreduce: 35.78 | step: 18.81

  4%|▍         | 1835/41250 [4:25:57<94:39:40,  8.65s/it]
                                                         
{'loss': 0.0393, 'grad_norm': 0.8658097386360168, 'learning_rate': 3.997803210890412e-05, 'epoch': 0.44}

  4%|▍         | 1835/41250 [4:25:57<94:39:40,  8.65s/it][2025-04-26 08:34:35,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 08:34:35,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.35 | bwd_microstep: 5740.57 | bwd_inner_microstep: 5688.35 | bwd_allreduce_microstep: 52.17 | step_microstep: 18.08
[2025-04-26 08:34:35,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.35 | bwd: 5740.58 | bwd_inner: 5688.35 | bwd_allreduce: 52.19 | step: 18.08

  4%|▍         | 1836/41250 [4:26:06<94:44:02,  8.65s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 2.781195878982544, 'learning_rate': 3.99779584665244e-05, 'epoch': 0.45}

  4%|▍         | 1836/41250 [4:26:06<94:44:02,  8.65s/it][2025-04-26 08:34:43,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:34:43,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.59 | bwd_microstep: 5716.38 | bwd_inner_microstep: 5704.20 | bwd_allreduce_microstep: 12.14 | step_microstep: 18.14
[2025-04-26 08:34:43,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.59 | bwd: 5716.39 | bwd_inner: 5704.19 | bwd_allreduce: 12.16 | step: 18.15

  4%|▍         | 1837/41250 [4:26:15<94:42:50,  8.65s/it]
                                                         
{'loss': 0.1825, 'grad_norm': 6.714719772338867, 'learning_rate': 3.99778847009845e-05, 'epoch': 0.45}

  4%|▍         | 1837/41250 [4:26:15<94:42:50,  8.65s/it][2025-04-26 08:34:52,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:34:52,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.02 | bwd_microstep: 5697.05 | bwd_inner_microstep: 5684.34 | bwd_allreduce_microstep: 12.66 | step_microstep: 17.74
[2025-04-26 08:34:52,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.02 | bwd: 5697.06 | bwd_inner: 5684.34 | bwd_allreduce: 12.68 | step: 17.74

  4%|▍         | 1838/41250 [4:26:23<94:34:46,  8.64s/it]
                                                         
{'loss': 0.0621, 'grad_norm': 0.5494499206542969, 'learning_rate': 3.997781081228486e-05, 'epoch': 0.45}

  4%|▍         | 1838/41250 [4:26:23<94:34:46,  8.64s/it][2025-04-26 08:35:01,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:35:01,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.13 | bwd_microstep: 5715.19 | bwd_inner_microstep: 5702.48 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.62
[2025-04-26 08:35:01,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.13 | bwd: 5715.20 | bwd_inner: 5702.48 | bwd_allreduce: 12.69 | step: 18.63

  4%|▍         | 1839/41250 [4:26:32<94:35:57,  8.64s/it]
                                                         
{'loss': 0.1184, 'grad_norm': 2.079223871231079, 'learning_rate': 3.9977736800425956e-05, 'epoch': 0.45}

  4%|▍         | 1839/41250 [4:26:32<94:35:57,  8.64s/it][2025-04-26 08:35:09,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:35:09,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.01 | bwd_microstep: 5726.13 | bwd_inner_microstep: 5698.20 | bwd_allreduce_microstep: 27.89 | step_microstep: 17.78
[2025-04-26 08:35:09,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.01 | bwd: 5726.14 | bwd_inner: 5698.20 | bwd_allreduce: 27.91 | step: 17.79

  4%|▍         | 1840/41250 [4:26:40<94:40:36,  8.65s/it]
                                                         
{'loss': 0.1956, 'grad_norm': 3.5252387523651123, 'learning_rate': 3.997766266540823e-05, 'epoch': 0.45}

  4%|▍         | 1840/41250 [4:26:40<94:40:36,  8.65s/it][2025-04-26 08:35:18,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:35:18,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.78 | bwd_microstep: 5770.26 | bwd_inner_microstep: 5657.70 | bwd_allreduce_microstep: 112.52 | step_microstep: 17.82
[2025-04-26 08:35:18,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.78 | bwd: 5770.27 | bwd_inner: 5657.70 | bwd_allreduce: 112.53 | step: 17.82

  4%|▍         | 1841/41250 [4:26:49<94:45:35,  8.66s/it]
                                                         
{'loss': 0.0465, 'grad_norm': 0.7534016966819763, 'learning_rate': 3.997758840723214e-05, 'epoch': 0.45}

  4%|▍         | 1841/41250 [4:26:49<94:45:35,  8.66s/it][2025-04-26 08:35:27,149] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:35:27,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.31 | bwd_microstep: 5723.71 | bwd_inner_microstep: 5692.67 | bwd_allreduce_microstep: 31.00 | step_microstep: 18.40
[2025-04-26 08:35:27,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.31 | bwd: 5723.72 | bwd_inner: 5692.67 | bwd_allreduce: 31.02 | step: 18.40

  4%|▍         | 1842/41250 [4:26:58<94:47:54,  8.66s/it]
                                                         
{'loss': 0.1047, 'grad_norm': 1.6012537479400635, 'learning_rate': 3.9977514025898154e-05, 'epoch': 0.45}

  4%|▍         | 1842/41250 [4:26:58<94:47:54,  8.66s/it][2025-04-26 08:35:35,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 08:35:35,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.88 | bwd_microstep: 5703.61 | bwd_inner_microstep: 5690.54 | bwd_allreduce_microstep: 13.02 | step_microstep: 19.00
[2025-04-26 08:35:35,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.88 | bwd: 5703.62 | bwd_inner: 5690.54 | bwd_allreduce: 13.04 | step: 19.01

  4%|▍         | 1843/41250 [4:27:06<94:40:29,  8.65s/it]
                                                         
{'loss': 0.1545, 'grad_norm': 0.8593634963035583, 'learning_rate': 3.9977439521406724e-05, 'epoch': 0.45}

  4%|▍         | 1843/41250 [4:27:06<94:40:29,  8.65s/it][2025-04-26 08:35:44,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:35:44,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.90 | bwd_microstep: 5749.39 | bwd_inner_microstep: 5652.16 | bwd_allreduce_microstep: 97.18 | step_microstep: 18.45
[2025-04-26 08:35:44,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.90 | bwd: 5749.40 | bwd_inner: 5652.16 | bwd_allreduce: 97.20 | step: 18.46

  4%|▍         | 1844/41250 [4:27:15<94:43:35,  8.65s/it]
                                                         
{'loss': 0.2902, 'grad_norm': 1.7758971452713013, 'learning_rate': 3.99773648937583e-05, 'epoch': 0.45}

  4%|▍         | 1844/41250 [4:27:15<94:43:35,  8.65s/it][2025-04-26 08:35:53,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.93 | optimizer_step: 1.05
[2025-04-26 08:35:53,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.13 | bwd_microstep: 5770.07 | bwd_inner_microstep: 5642.03 | bwd_allreduce_microstep: 128.00 | step_microstep: 18.15
[2025-04-26 08:35:53,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.13 | bwd: 5770.08 | bwd_inner: 5642.03 | bwd_allreduce: 128.02 | step: 18.16

  4%|▍         | 1845/41250 [4:27:24<94:47:51,  8.66s/it]
                                                         
{'loss': 0.3692, 'grad_norm': 2.6105926036834717, 'learning_rate': 3.997729014295336e-05, 'epoch': 0.45}

  4%|▍         | 1845/41250 [4:27:24<94:47:51,  8.66s/it][2025-04-26 08:36:01,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.96
[2025-04-26 08:36:01,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.85 | bwd_microstep: 5709.61 | bwd_inner_microstep: 5696.72 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.32
[2025-04-26 08:36:01,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.85 | bwd: 5709.62 | bwd_inner: 5696.72 | bwd_allreduce: 12.86 | step: 18.32

  4%|▍         | 1846/41250 [4:27:32<94:42:38,  8.65s/it]
                                                         
{'loss': 0.2906, 'grad_norm': 2.064579725265503, 'learning_rate': 3.997721526899235e-05, 'epoch': 0.45}

  4%|▍         | 1846/41250 [4:27:32<94:42:38,  8.65s/it][2025-04-26 08:36:10,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 08:36:10,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5717.89 | bwd_inner_microstep: 5704.59 | bwd_allreduce_microstep: 13.25 | step_microstep: 18.66
[2025-04-26 08:36:10,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5717.90 | bwd_inner: 5704.59 | bwd_allreduce: 13.27 | step: 18.66

  4%|▍         | 1847/41250 [4:27:41<94:40:45,  8.65s/it]
                                                         
{'loss': 0.1342, 'grad_norm': 1.2676481008529663, 'learning_rate': 3.997714027187574e-05, 'epoch': 0.45}

  4%|▍         | 1847/41250 [4:27:41<94:40:45,  8.65s/it][2025-04-26 08:36:19,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 08:36:19,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.83 | bwd_microstep: 5734.18 | bwd_inner_microstep: 5704.74 | bwd_allreduce_microstep: 29.38 | step_microstep: 18.97
[2025-04-26 08:36:19,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.83 | bwd: 5734.20 | bwd_inner: 5704.74 | bwd_allreduce: 29.41 | step: 18.97

  4%|▍         | 1848/41250 [4:27:50<94:45:39,  8.66s/it]
                                                         
{'loss': 0.233, 'grad_norm': 3.070117712020874, 'learning_rate': 3.997706515160398e-05, 'epoch': 0.45}

  4%|▍         | 1848/41250 [4:27:50<94:45:39,  8.66s/it][2025-04-26 08:36:27,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:36:27,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5691.42 | bwd_inner_microstep: 5678.72 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.71
[2025-04-26 08:36:27,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5691.44 | bwd_inner: 5678.72 | bwd_allreduce: 12.67 | step: 18.71

  4%|▍         | 1849/41250 [4:27:58<94:39:25,  8.65s/it]
                                                         
{'loss': 0.0977, 'grad_norm': 1.6564971208572388, 'learning_rate': 3.997698990817755e-05, 'epoch': 0.45}

  4%|▍         | 1849/41250 [4:27:58<94:39:25,  8.65s/it][2025-04-26 08:36:36,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.14 | optimizer_step: 1.17
[2025-04-26 08:36:36,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.10 | bwd_microstep: 5723.05 | bwd_inner_microstep: 5709.31 | bwd_allreduce_microstep: 13.68 | step_microstep: 19.33
[2025-04-26 08:36:36,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.10 | bwd: 5723.06 | bwd_inner: 5709.31 | bwd_allreduce: 13.71 | step: 19.33

  4%|▍         | 1850/41250 [4:28:07<94:41:32,  8.65s/it]
                                                         
{'loss': 0.0692, 'grad_norm': 1.6581979990005493, 'learning_rate': 3.997691454159691e-05, 'epoch': 0.45}

  4%|▍         | 1850/41250 [4:28:07<94:41:32,  8.65s/it][2025-04-26 08:36:45,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:36:45,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.53 | bwd_microstep: 5794.75 | bwd_inner_microstep: 5645.17 | bwd_allreduce_microstep: 149.54 | step_microstep: 18.07
[2025-04-26 08:36:45,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.53 | bwd: 5794.77 | bwd_inner: 5645.17 | bwd_allreduce: 149.56 | step: 18.07

  4%|▍         | 1851/41250 [4:28:16<94:51:10,  8.67s/it]
                                                         
{'loss': 0.05, 'grad_norm': 1.12704598903656, 'learning_rate': 3.997683905186251e-05, 'epoch': 0.45}

  4%|▍         | 1851/41250 [4:28:16<94:51:10,  8.67s/it][2025-04-26 08:36:53,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:36:53,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.88 | bwd_microstep: 5796.57 | bwd_inner_microstep: 5783.79 | bwd_allreduce_microstep: 12.73 | step_microstep: 17.84
[2025-04-26 08:36:53,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.88 | bwd: 5796.58 | bwd_inner: 5783.79 | bwd_allreduce: 12.75 | step: 17.84

  4%|▍         | 1852/41250 [4:28:25<95:10:53,  8.70s/it]
                                                         
{'loss': 0.1157, 'grad_norm': 1.1539560556411743, 'learning_rate': 3.997676343897484e-05, 'epoch': 0.45}

  4%|▍         | 1852/41250 [4:28:25<95:10:53,  8.70s/it][2025-04-26 08:37:02,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 08:37:02,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.99 | bwd_microstep: 5762.30 | bwd_inner_microstep: 5703.65 | bwd_allreduce_microstep: 58.61 | step_microstep: 18.58
[2025-04-26 08:37:02,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.99 | bwd: 5762.32 | bwd_inner: 5703.65 | bwd_allreduce: 58.63 | step: 18.58

  4%|▍         | 1853/41250 [4:28:33<95:09:25,  8.70s/it]
                                                         
{'loss': 0.2323, 'grad_norm': 1.610981822013855, 'learning_rate': 3.9976687702934344e-05, 'epoch': 0.45}

  4%|▍         | 1853/41250 [4:28:33<95:09:25,  8.70s/it][2025-04-26 08:37:11,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:37:11,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5748.63 | bwd_inner_microstep: 5699.61 | bwd_allreduce_microstep: 48.97 | step_microstep: 18.63
[2025-04-26 08:37:11,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5748.65 | bwd_inner: 5699.61 | bwd_allreduce: 48.99 | step: 18.63

  4%|▍         | 1854/41250 [4:28:42<95:08:28,  8.69s/it]
                                                         
{'loss': 0.1547, 'grad_norm': 2.755375385284424, 'learning_rate': 3.99766118437415e-05, 'epoch': 0.45}

  4%|▍         | 1854/41250 [4:28:42<95:08:28,  8.69s/it][2025-04-26 08:37:19,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 08:37:19,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5776.50 | bwd_inner_microstep: 5702.79 | bwd_allreduce_microstep: 73.66 | step_microstep: 18.69
[2025-04-26 08:37:19,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5776.51 | bwd_inner: 5702.79 | bwd_allreduce: 73.68 | step: 18.70

  4%|▍         | 1855/41250 [4:28:51<95:10:53,  8.70s/it]
                                                         
{'loss': 0.0679, 'grad_norm': 0.5985469222068787, 'learning_rate': 3.997653586139677e-05, 'epoch': 0.45}

  4%|▍         | 1855/41250 [4:28:51<95:10:53,  8.70s/it][2025-04-26 08:37:28,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 1.00
[2025-04-26 08:37:28,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.36 | bwd_microstep: 5864.22 | bwd_inner_microstep: 5710.97 | bwd_allreduce_microstep: 153.21 | step_microstep: 18.05
[2025-04-26 08:37:28,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.36 | bwd: 5864.24 | bwd_inner: 5710.97 | bwd_allreduce: 153.23 | step: 18.05

  4%|▍         | 1856/41250 [4:28:59<95:29:24,  8.73s/it]
                                                         
{'loss': 0.2748, 'grad_norm': 2.3105921745300293, 'learning_rate': 3.997645975590064e-05, 'epoch': 0.45}

  4%|▍         | 1856/41250 [4:28:59<95:29:24,  8.73s/it][2025-04-26 08:37:37,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:37:37,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5729.65 | bwd_inner_microstep: 5716.92 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.83
[2025-04-26 08:37:37,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5729.66 | bwd_inner: 5716.92 | bwd_allreduce: 12.70 | step: 17.83

  5%|▍         | 1857/41250 [4:29:08<95:17:29,  8.71s/it]
                                                         
{'loss': 0.1899, 'grad_norm': 1.8077797889709473, 'learning_rate': 3.9976383527253546e-05, 'epoch': 0.45}

  5%|▍         | 1857/41250 [4:29:08<95:17:29,  8.71s/it][2025-04-26 08:37:46,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.95 | optimizer_step: 1.01
[2025-04-26 08:37:46,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.77 | bwd_microstep: 5749.88 | bwd_inner_microstep: 5698.37 | bwd_allreduce_microstep: 51.48 | step_microstep: 18.31
[2025-04-26 08:37:46,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.77 | bwd: 5749.90 | bwd_inner: 5698.37 | bwd_allreduce: 51.49 | step: 18.31

  5%|▍         | 1858/41250 [4:29:17<95:13:10,  8.70s/it]
                                                         
{'loss': 0.0714, 'grad_norm': 0.9463362097740173, 'learning_rate': 3.997630717545598e-05, 'epoch': 0.45}

  5%|▍         | 1858/41250 [4:29:17<95:13:10,  8.70s/it][2025-04-26 08:37:54,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:37:54,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.19 | bwd_microstep: 5718.28 | bwd_inner_microstep: 5705.44 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.37
[2025-04-26 08:37:54,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.19 | bwd: 5718.29 | bwd_inner: 5705.44 | bwd_allreduce: 12.81 | step: 18.37

  5%|▍         | 1859/41250 [4:29:25<95:04:22,  8.69s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 1.9303349256515503, 'learning_rate': 3.997623070050842e-05, 'epoch': 0.45}

  5%|▍         | 1859/41250 [4:29:25<95:04:22,  8.69s/it][2025-04-26 08:38:03,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:38:03,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.92 | bwd_microstep: 5732.67 | bwd_inner_microstep: 5684.87 | bwd_allreduce_microstep: 47.77 | step_microstep: 18.37
[2025-04-26 08:38:03,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.92 | bwd: 5732.69 | bwd_inner: 5684.87 | bwd_allreduce: 47.78 | step: 18.38

  5%|▍         | 1860/41250 [4:29:34<95:01:54,  8.69s/it]
                                                         
{'loss': 0.3027, 'grad_norm': 2.7142951488494873, 'learning_rate': 3.997615410241132e-05, 'epoch': 0.45}

  5%|▍         | 1860/41250 [4:29:34<95:01:54,  8.69s/it][2025-04-26 08:38:12,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:38:12,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5722.68 | bwd_inner_microstep: 5709.96 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.44
[2025-04-26 08:38:12,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5722.69 | bwd_inner: 5709.96 | bwd_allreduce: 12.69 | step: 18.45

  5%|▍         | 1861/41250 [4:29:43<94:55:30,  8.68s/it]
                                                         
{'loss': 0.1095, 'grad_norm': 0.9930664300918579, 'learning_rate': 3.997607738116516e-05, 'epoch': 0.45}

  5%|▍         | 1861/41250 [4:29:43<94:55:30,  8.68s/it][2025-04-26 08:38:20,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 08:38:20,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.31 | bwd_microstep: 5723.93 | bwd_inner_microstep: 5711.26 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.71
[2025-04-26 08:38:20,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.31 | bwd: 5723.94 | bwd_inner: 5711.25 | bwd_allreduce: 12.65 | step: 18.71

  5%|▍         | 1862/41250 [4:29:51<94:53:13,  8.67s/it]
                                                         
{'loss': 0.0891, 'grad_norm': 1.0502208471298218, 'learning_rate': 3.9976000536770415e-05, 'epoch': 0.45}

  5%|▍         | 1862/41250 [4:29:51<94:53:13,  8.67s/it][2025-04-26 08:38:29,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:38:29,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.17 | bwd_microstep: 5781.07 | bwd_inner_microstep: 5657.45 | bwd_allreduce_microstep: 123.57 | step_microstep: 17.64
[2025-04-26 08:38:29,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.17 | bwd: 5781.08 | bwd_inner: 5657.45 | bwd_allreduce: 123.59 | step: 17.65

  5%|▍         | 1863/41250 [4:30:00<95:00:19,  8.68s/it]
                                                         
{'loss': 0.2054, 'grad_norm': 2.605221748352051, 'learning_rate': 3.997592356922755e-05, 'epoch': 0.45}

  5%|▍         | 1863/41250 [4:30:00<95:00:19,  8.68s/it][2025-04-26 08:38:38,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:38:38,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5754.55 | bwd_inner_microstep: 5683.84 | bwd_allreduce_microstep: 70.67 | step_microstep: 18.24
[2025-04-26 08:38:38,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5754.57 | bwd_inner: 5683.84 | bwd_allreduce: 70.68 | step: 18.24

  5%|▍         | 1864/41250 [4:30:09<95:00:00,  8.68s/it]
                                                         
{'loss': 0.1121, 'grad_norm': 0.9756320118904114, 'learning_rate': 3.997584647853705e-05, 'epoch': 0.45}

  5%|▍         | 1864/41250 [4:30:09<95:00:00,  8.68s/it][2025-04-26 08:38:46,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:38:46,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.20 | bwd_microstep: 5721.48 | bwd_inner_microstep: 5708.78 | bwd_allreduce_microstep: 12.66 | step_microstep: 17.66
[2025-04-26 08:38:46,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.20 | bwd: 5721.49 | bwd_inner: 5708.78 | bwd_allreduce: 12.67 | step: 17.67

  5%|▍         | 1865/41250 [4:30:17<94:53:13,  8.67s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 1.2165415287017822, 'learning_rate': 3.997576926469938e-05, 'epoch': 0.45}

  5%|▍         | 1865/41250 [4:30:17<94:53:13,  8.67s/it][2025-04-26 08:38:55,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 08:38:55,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.71 | bwd_microstep: 5692.37 | bwd_inner_microstep: 5648.19 | bwd_allreduce_microstep: 44.14 | step_microstep: 18.69
[2025-04-26 08:38:55,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.71 | bwd: 5692.38 | bwd_inner: 5648.19 | bwd_allreduce: 44.16 | step: 18.69

  5%|▍         | 1866/41250 [4:30:26<94:41:07,  8.65s/it]
                                                         
{'loss': 0.1531, 'grad_norm': 2.0150787830352783, 'learning_rate': 3.9975691927715024e-05, 'epoch': 0.45}

  5%|▍         | 1866/41250 [4:30:26<94:41:07,  8.65s/it][2025-04-26 08:39:04,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:39:04,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.64 | bwd_microstep: 5768.79 | bwd_inner_microstep: 5664.44 | bwd_allreduce_microstep: 104.31 | step_microstep: 18.56
[2025-04-26 08:39:04,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.64 | bwd: 5768.80 | bwd_inner: 5664.43 | bwd_allreduce: 104.32 | step: 18.56

  5%|▍         | 1867/41250 [4:30:35<94:49:06,  8.67s/it]
                                                         
{'loss': 0.4038, 'grad_norm': 3.0797460079193115, 'learning_rate': 3.9975614467584456e-05, 'epoch': 0.45}

  5%|▍         | 1867/41250 [4:30:35<94:49:06,  8.67s/it][2025-04-26 08:39:12,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:39:12,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.87 | bwd_microstep: 5761.27 | bwd_inner_microstep: 5644.45 | bwd_allreduce_microstep: 116.78 | step_microstep: 18.33
[2025-04-26 08:39:12,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.87 | bwd: 5761.28 | bwd_inner: 5644.45 | bwd_allreduce: 116.79 | step: 18.33

  5%|▍         | 1868/41250 [4:30:43<94:51:42,  8.67s/it]
                                                         
{'loss': 0.418, 'grad_norm': 3.666443347930908, 'learning_rate': 3.997553688430816e-05, 'epoch': 0.45}

  5%|▍         | 1868/41250 [4:30:43<94:51:42,  8.67s/it][2025-04-26 08:39:21,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:39:21,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.47 | bwd_microstep: 5786.44 | bwd_inner_microstep: 5773.68 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.59
[2025-04-26 08:39:21,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.47 | bwd: 5786.45 | bwd_inner: 5773.68 | bwd_allreduce: 12.73 | step: 18.59

  5%|▍         | 1869/41250 [4:30:52<95:10:00,  8.70s/it]
                                                         
{'loss': 0.114, 'grad_norm': 0.9498416781425476, 'learning_rate': 3.99754591778866e-05, 'epoch': 0.45}

  5%|▍         | 1869/41250 [4:30:52<95:10:00,  8.70s/it][2025-04-26 08:39:30,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 08:39:30,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.45 | bwd_microstep: 5705.96 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.63
[2025-04-26 08:39:30,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.45 | bwd: 5705.98 | bwd_inner: 5693.20 | bwd_allreduce: 12.73 | step: 18.63

  5%|▍         | 1870/41250 [4:31:01<94:56:15,  8.68s/it]
                                                         
{'loss': 0.1991, 'grad_norm': 2.0626139640808105, 'learning_rate': 3.997538134832026e-05, 'epoch': 0.45}

  5%|▍         | 1870/41250 [4:31:01<94:56:15,  8.68s/it][2025-04-26 08:39:38,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 08:39:38,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.35 | bwd_microstep: 5681.55 | bwd_inner_microstep: 5652.43 | bwd_allreduce_microstep: 29.08 | step_microstep: 18.87
[2025-04-26 08:39:38,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.35 | bwd: 5681.57 | bwd_inner: 5652.43 | bwd_allreduce: 29.10 | step: 18.88

  5%|▍         | 1871/41250 [4:31:09<94:42:10,  8.66s/it]
                                                         
{'loss': 0.1828, 'grad_norm': 0.9259169101715088, 'learning_rate': 3.997530339560963e-05, 'epoch': 0.45}

  5%|▍         | 1871/41250 [4:31:09<94:42:10,  8.66s/it][2025-04-26 08:39:47,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:39:47,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5695.77 | bwd_inner_microstep: 5648.05 | bwd_allreduce_microstep: 47.68 | step_microstep: 18.43
[2025-04-26 08:39:47,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5695.78 | bwd_inner: 5648.05 | bwd_allreduce: 47.69 | step: 18.43

  5%|▍         | 1872/41250 [4:31:18<94:32:31,  8.64s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 1.9572529792785645, 'learning_rate': 3.997522531975518e-05, 'epoch': 0.45}

  5%|▍         | 1872/41250 [4:31:18<94:32:31,  8.64s/it][2025-04-26 08:39:55,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:39:55,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.10 | bwd_microstep: 5698.40 | bwd_inner_microstep: 5685.71 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.36
[2025-04-26 08:39:55,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.10 | bwd: 5698.42 | bwd_inner: 5685.71 | bwd_allreduce: 12.66 | step: 18.37

  5%|▍         | 1873/41250 [4:31:27<94:27:31,  8.64s/it]
                                                         
{'loss': 0.0308, 'grad_norm': 0.36657729744911194, 'learning_rate': 3.9975147120757406e-05, 'epoch': 0.45}

  5%|▍         | 1873/41250 [4:31:27<94:27:31,  8.64s/it][2025-04-26 08:40:04,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:40:04,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.00 | bwd_microstep: 5696.89 | bwd_inner_microstep: 5684.11 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.53
[2025-04-26 08:40:04,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.00 | bwd: 5696.90 | bwd_inner: 5684.11 | bwd_allreduce: 12.75 | step: 18.53

  5%|▍         | 1874/41250 [4:31:35<94:24:01,  8.63s/it]
                                                         
{'loss': 0.3108, 'grad_norm': 3.1867754459381104, 'learning_rate': 3.997506879861677e-05, 'epoch': 0.45}

  5%|▍         | 1874/41250 [4:31:35<94:24:01,  8.63s/it][2025-04-26 08:40:13,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 08:40:13,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.04 | bwd_microstep: 5763.89 | bwd_inner_microstep: 5641.17 | bwd_allreduce_microstep: 122.68 | step_microstep: 18.83
[2025-04-26 08:40:13,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.04 | bwd: 5763.91 | bwd_inner: 5641.17 | bwd_allreduce: 122.70 | step: 18.84

  5%|▍         | 1875/41250 [4:31:44<94:34:07,  8.65s/it]
                                                         
{'loss': 0.215, 'grad_norm': 1.9164645671844482, 'learning_rate': 3.997499035333376e-05, 'epoch': 0.45}

  5%|▍         | 1875/41250 [4:31:44<94:34:07,  8.65s/it][2025-04-26 08:40:21,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:40:21,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.13 | bwd_microstep: 5684.82 | bwd_inner_microstep: 5672.01 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.55
[2025-04-26 08:40:21,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.13 | bwd: 5684.84 | bwd_inner: 5672.01 | bwd_allreduce: 12.78 | step: 18.55

  5%|▍         | 1876/41250 [4:31:53<94:29:14,  8.64s/it]
                                                         
{'loss': 0.128, 'grad_norm': 2.125690460205078, 'learning_rate': 3.997491178490887e-05, 'epoch': 0.45}

  5%|▍         | 1876/41250 [4:31:53<94:29:14,  8.64s/it][2025-04-26 08:40:30,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:40:30,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.59 | bwd_microstep: 5665.10 | bwd_inner_microstep: 5646.21 | bwd_allreduce_microstep: 18.85 | step_microstep: 18.84
[2025-04-26 08:40:30,487] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.59 | bwd: 5665.12 | bwd_inner: 5646.21 | bwd_allreduce: 18.87 | step: 18.85

  5%|▍         | 1877/41250 [4:32:01<94:18:36,  8.62s/it]
                                                         
{'loss': 0.0877, 'grad_norm': 1.3487093448638916, 'learning_rate': 3.997483309334257e-05, 'epoch': 0.46}

  5%|▍         | 1877/41250 [4:32:01<94:18:36,  8.62s/it][2025-04-26 08:40:39,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:40:39,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5752.63 | bwd_inner_microstep: 5648.51 | bwd_allreduce_microstep: 104.08 | step_microstep: 18.33
[2025-04-26 08:40:39,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5752.64 | bwd_inner: 5648.51 | bwd_allreduce: 104.09 | step: 18.34

  5%|▍         | 1878/41250 [4:32:10<94:26:43,  8.64s/it]
                                                         
{'loss': 0.236, 'grad_norm': 3.90739369392395, 'learning_rate': 3.997475427863536e-05, 'epoch': 0.46}

  5%|▍         | 1878/41250 [4:32:10<94:26:43,  8.64s/it][2025-04-26 08:40:47,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:40:47,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.52 | bwd_microstep: 5744.32 | bwd_inner_microstep: 5693.31 | bwd_allreduce_microstep: 50.97 | step_microstep: 18.29
[2025-04-26 08:40:47,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.52 | bwd: 5744.33 | bwd_inner: 5693.30 | bwd_allreduce: 50.99 | step: 18.29

  5%|▍         | 1879/41250 [4:32:19<94:36:38,  8.65s/it]
                                                         
{'loss': 0.0787, 'grad_norm': 1.613408088684082, 'learning_rate': 3.9974675340787715e-05, 'epoch': 0.46}

  5%|▍         | 1879/41250 [4:32:19<94:36:38,  8.65s/it][2025-04-26 08:40:56,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:40:56,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.37 | bwd_microstep: 5767.13 | bwd_inner_microstep: 5651.50 | bwd_allreduce_microstep: 115.59 | step_microstep: 18.47
[2025-04-26 08:40:56,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.37 | bwd: 5767.15 | bwd_inner: 5651.50 | bwd_allreduce: 115.60 | step: 18.47

  5%|▍         | 1880/41250 [4:32:27<94:42:21,  8.66s/it]
                                                         
{'loss': 0.1212, 'grad_norm': 1.4994746446609497, 'learning_rate': 3.997459627980013e-05, 'epoch': 0.46}

  5%|▍         | 1880/41250 [4:32:27<94:42:21,  8.66s/it][2025-04-26 08:41:05,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 08:41:05,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.10 | bwd_microstep: 5762.29 | bwd_inner_microstep: 5633.20 | bwd_allreduce_microstep: 129.04 | step_microstep: 18.88
[2025-04-26 08:41:05,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.10 | bwd: 5762.31 | bwd_inner: 5633.20 | bwd_allreduce: 129.06 | step: 18.88

  5%|▍         | 1881/41250 [4:32:36<94:44:21,  8.66s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 4.863039970397949, 'learning_rate': 3.9974517095673084e-05, 'epoch': 0.46}

  5%|▍         | 1881/41250 [4:32:36<94:44:21,  8.66s/it][2025-04-26 08:41:13,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 1.10
[2025-04-26 08:41:13,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.88 | bwd_microstep: 5717.85 | bwd_inner_microstep: 5705.10 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.85
[2025-04-26 08:41:13,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.88 | bwd: 5717.87 | bwd_inner: 5705.10 | bwd_allreduce: 12.73 | step: 18.85

  5%|▍         | 1882/41250 [4:32:45<94:43:41,  8.66s/it]
                                                         
{'loss': 0.1955, 'grad_norm': 1.5944315195083618, 'learning_rate': 3.997443778840707e-05, 'epoch': 0.46}

  5%|▍         | 1882/41250 [4:32:45<94:43:41,  8.66s/it][2025-04-26 08:41:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:41:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.12 | bwd_microstep: 5700.41 | bwd_inner_microstep: 5645.95 | bwd_allreduce_microstep: 54.42 | step_microstep: 18.49
[2025-04-26 08:41:22,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.12 | bwd: 5700.42 | bwd_inner: 5645.95 | bwd_allreduce: 54.44 | step: 18.49

  5%|▍         | 1883/41250 [4:32:53<94:32:34,  8.65s/it]
                                                         
{'loss': 0.1034, 'grad_norm': 2.791874647140503, 'learning_rate': 3.9974358358002575e-05, 'epoch': 0.46}

  5%|▍         | 1883/41250 [4:32:53<94:32:34,  8.65s/it][2025-04-26 08:41:31,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:41:31,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.29 | bwd_microstep: 5779.41 | bwd_inner_microstep: 5766.76 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.39
[2025-04-26 08:41:31,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.29 | bwd: 5779.42 | bwd_inner: 5766.76 | bwd_allreduce: 12.62 | step: 18.39

  5%|▍         | 1884/41250 [4:33:02<94:53:18,  8.68s/it]
                                                         
{'loss': 0.1155, 'grad_norm': 1.8787587881088257, 'learning_rate': 3.997427880446008e-05, 'epoch': 0.46}

  5%|▍         | 1884/41250 [4:33:02<94:53:18,  8.68s/it][2025-04-26 08:41:39,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:41:39,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.90 | bwd_microstep: 5749.91 | bwd_inner_microstep: 5704.14 | bwd_allreduce_microstep: 45.72 | step_microstep: 18.39
[2025-04-26 08:41:39,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.90 | bwd: 5749.92 | bwd_inner: 5704.14 | bwd_allreduce: 45.74 | step: 18.39

  5%|▍         | 1885/41250 [4:33:11<94:53:36,  8.68s/it]
                                                         
{'loss': 0.3239, 'grad_norm': 1.994443655014038, 'learning_rate': 3.9974199127780095e-05, 'epoch': 0.46}

  5%|▍         | 1885/41250 [4:33:11<94:53:36,  8.68s/it][2025-04-26 08:41:48,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:41:48,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5678.84 | bwd_inner_microstep: 5637.84 | bwd_allreduce_microstep: 40.96 | step_microstep: 18.17
[2025-04-26 08:41:48,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5678.86 | bwd_inner: 5637.84 | bwd_allreduce: 40.97 | step: 18.18

  5%|▍         | 1886/41250 [4:33:19<94:35:39,  8.65s/it]
                                                         
{'loss': 0.218, 'grad_norm': 3.0225274562835693, 'learning_rate': 3.9974119327963106e-05, 'epoch': 0.46}

  5%|▍         | 1886/41250 [4:33:19<94:35:39,  8.65s/it][2025-04-26 08:41:57,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:41:57,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.14 | bwd_microstep: 5772.95 | bwd_inner_microstep: 5656.55 | bwd_allreduce_microstep: 116.36 | step_microstep: 18.92
[2025-04-26 08:41:57,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.14 | bwd: 5772.96 | bwd_inner: 5656.55 | bwd_allreduce: 116.37 | step: 18.93

  5%|▍         | 1887/41250 [4:33:28<94:42:18,  8.66s/it]
                                                         
{'loss': 0.3586, 'grad_norm': 3.3631410598754883, 'learning_rate': 3.99740394050096e-05, 'epoch': 0.46}

  5%|▍         | 1887/41250 [4:33:28<94:42:18,  8.66s/it][2025-04-26 08:42:05,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 08:42:05,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.72 | bwd_microstep: 5741.04 | bwd_inner_microstep: 5694.19 | bwd_allreduce_microstep: 46.80 | step_microstep: 18.92
[2025-04-26 08:42:05,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.72 | bwd: 5741.06 | bwd_inner: 5694.19 | bwd_allreduce: 46.82 | step: 18.92

  5%|▍         | 1888/41250 [4:33:37<94:46:22,  8.67s/it]
                                                         
{'loss': 0.1401, 'grad_norm': 1.980239748954773, 'learning_rate': 3.997395935892006e-05, 'epoch': 0.46}

  5%|▍         | 1888/41250 [4:33:37<94:46:22,  8.67s/it][2025-04-26 08:42:14,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 08:42:14,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5868.58 | bwd_inner_microstep: 5700.96 | bwd_allreduce_microstep: 167.58 | step_microstep: 17.93
[2025-04-26 08:42:14,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5868.60 | bwd_inner: 5700.96 | bwd_allreduce: 167.59 | step: 17.94

  5%|▍         | 1889/41250 [4:33:45<95:12:41,  8.71s/it]
                                                         
{'loss': 0.1285, 'grad_norm': 1.606147289276123, 'learning_rate': 3.997387918969499e-05, 'epoch': 0.46}

  5%|▍         | 1889/41250 [4:33:45<95:12:41,  8.71s/it][2025-04-26 08:42:23,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 08:42:23,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.67 | bwd_microstep: 5694.62 | bwd_inner_microstep: 5651.27 | bwd_allreduce_microstep: 43.30 | step_microstep: 18.94
[2025-04-26 08:42:23,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.67 | bwd: 5694.63 | bwd_inner: 5651.27 | bwd_allreduce: 43.32 | step: 18.94

  5%|▍         | 1890/41250 [4:33:54<94:51:12,  8.68s/it]
                                                         
{'loss': 0.0645, 'grad_norm': 0.9189678430557251, 'learning_rate': 3.997379889733489e-05, 'epoch': 0.46}

  5%|▍         | 1890/41250 [4:33:54<94:51:12,  8.68s/it][2025-04-26 08:42:31,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.91
[2025-04-26 08:42:31,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5710.23 | bwd_inner_microstep: 5651.88 | bwd_allreduce_microstep: 58.31 | step_microstep: 18.44
[2025-04-26 08:42:31,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5710.24 | bwd_inner: 5651.88 | bwd_allreduce: 58.32 | step: 18.44

  5%|▍         | 1891/41250 [4:34:03<94:39:33,  8.66s/it]
                                                         
{'loss': 0.2406, 'grad_norm': 1.7845559120178223, 'learning_rate': 3.9973718481840244e-05, 'epoch': 0.46}

  5%|▍         | 1891/41250 [4:34:03<94:39:33,  8.66s/it][2025-04-26 08:42:40,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:42:40,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.57 | bwd_microstep: 5708.06 | bwd_inner_microstep: 5692.96 | bwd_allreduce_microstep: 15.05 | step_microstep: 18.35
[2025-04-26 08:42:40,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.57 | bwd: 5708.08 | bwd_inner: 5692.96 | bwd_allreduce: 15.07 | step: 18.35

  5%|▍         | 1892/41250 [4:34:11<94:31:48,  8.65s/it]
                                                         
{'loss': 0.2191, 'grad_norm': 1.9058293104171753, 'learning_rate': 3.9973637943211554e-05, 'epoch': 0.46}

  5%|▍         | 1892/41250 [4:34:11<94:31:48,  8.65s/it][2025-04-26 08:42:49,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:42:49,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.21 | bwd_microstep: 5806.42 | bwd_inner_microstep: 5662.87 | bwd_allreduce_microstep: 143.51 | step_microstep: 18.52
[2025-04-26 08:42:49,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.21 | bwd: 5806.44 | bwd_inner: 5662.87 | bwd_allreduce: 143.53 | step: 18.52

  5%|▍         | 1893/41250 [4:34:20<94:46:36,  8.67s/it]
                                                         
{'loss': 0.1541, 'grad_norm': 1.6227970123291016, 'learning_rate': 3.997355728144931e-05, 'epoch': 0.46}

  5%|▍         | 1893/41250 [4:34:20<94:46:36,  8.67s/it][2025-04-26 08:42:57,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:42:57,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.35 | bwd_microstep: 5717.10 | bwd_inner_microstep: 5704.40 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.48
[2025-04-26 08:42:57,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.35 | bwd: 5717.12 | bwd_inner: 5704.40 | bwd_allreduce: 12.68 | step: 18.48

  5%|▍         | 1894/41250 [4:34:29<94:43:38,  8.66s/it]
                                                         
{'loss': 0.2145, 'grad_norm': 2.0999600887298584, 'learning_rate': 3.9973476496554024e-05, 'epoch': 0.46}

  5%|▍         | 1894/41250 [4:34:29<94:43:38,  8.66s/it][2025-04-26 08:43:06,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:43:06,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.65 | bwd_microstep: 5778.86 | bwd_inner_microstep: 5661.08 | bwd_allreduce_microstep: 117.73 | step_microstep: 18.47
[2025-04-26 08:43:06,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.65 | bwd: 5778.87 | bwd_inner: 5661.08 | bwd_allreduce: 117.75 | step: 18.48

  5%|▍         | 1895/41250 [4:34:37<94:48:43,  8.67s/it]
                                                         
{'loss': 0.1609, 'grad_norm': 1.8228768110275269, 'learning_rate': 3.9973395588526175e-05, 'epoch': 0.46}

  5%|▍         | 1895/41250 [4:34:37<94:48:43,  8.67s/it][2025-04-26 08:43:15,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:43:15,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.57 | bwd_microstep: 5771.09 | bwd_inner_microstep: 5660.01 | bwd_allreduce_microstep: 111.03 | step_microstep: 17.86
[2025-04-26 08:43:15,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.57 | bwd: 5771.10 | bwd_inner: 5660.01 | bwd_allreduce: 111.05 | step: 17.87

  5%|▍         | 1896/41250 [4:34:46<94:50:05,  8.68s/it]
                                                         
{'loss': 0.2389, 'grad_norm': 2.819727659225464, 'learning_rate': 3.997331455736628e-05, 'epoch': 0.46}

  5%|▍         | 1896/41250 [4:34:46<94:50:05,  8.68s/it][2025-04-26 08:43:23,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:43:23,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.98 | bwd_microstep: 5778.38 | bwd_inner_microstep: 5765.67 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.33
[2025-04-26 08:43:23,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.98 | bwd: 5778.39 | bwd_inner: 5765.67 | bwd_allreduce: 12.68 | step: 18.33

  5%|▍         | 1897/41250 [4:34:55<95:01:02,  8.69s/it]
                                                         
{'loss': 0.1443, 'grad_norm': 1.5910708904266357, 'learning_rate': 3.997323340307482e-05, 'epoch': 0.46}

  5%|▍         | 1897/41250 [4:34:55<95:01:02,  8.69s/it][2025-04-26 08:43:32,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:43:32,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.14 | bwd_microstep: 5698.26 | bwd_inner_microstep: 5685.65 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.14
[2025-04-26 08:43:32,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.14 | bwd: 5698.27 | bwd_inner: 5685.65 | bwd_allreduce: 12.58 | step: 18.14

  5%|▍         | 1898/41250 [4:35:03<94:47:34,  8.67s/it]
                                                         
{'loss': 0.1472, 'grad_norm': 2.3604443073272705, 'learning_rate': 3.9973152125652304e-05, 'epoch': 0.46}

  5%|▍         | 1898/41250 [4:35:03<94:47:34,  8.67s/it][2025-04-26 08:43:41,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:43:41,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.50 | bwd_microstep: 5882.96 | bwd_inner_microstep: 5659.83 | bwd_allreduce_microstep: 223.09 | step_microstep: 17.75
[2025-04-26 08:43:41,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.50 | bwd: 5882.97 | bwd_inner: 5659.83 | bwd_allreduce: 223.10 | step: 17.76

  5%|▍         | 1899/41250 [4:35:12<95:11:24,  8.71s/it]
                                                         
{'loss': 0.1871, 'grad_norm': 1.5605220794677734, 'learning_rate': 3.997307072509924e-05, 'epoch': 0.46}

  5%|▍         | 1899/41250 [4:35:12<95:11:24,  8.71s/it][2025-04-26 08:43:50,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:43:50,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.16 | bwd_microstep: 5761.54 | bwd_inner_microstep: 5706.28 | bwd_allreduce_microstep: 55.21 | step_microstep: 18.15
[2025-04-26 08:43:50,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.16 | bwd: 5761.55 | bwd_inner: 5706.28 | bwd_allreduce: 55.23 | step: 18.15

  5%|▍         | 1900/41250 [4:35:21<95:08:00,  8.70s/it]
                                                         
{'loss': 0.1631, 'grad_norm': 2.1278533935546875, 'learning_rate': 3.997298920141612e-05, 'epoch': 0.46}

  5%|▍         | 1900/41250 [4:35:21<95:08:00,  8.70s/it][2025-04-26 08:43:58,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 08:43:58,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.04 | bwd_microstep: 5723.83 | bwd_inner_microstep: 5711.14 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.44
[2025-04-26 08:43:58,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.04 | bwd: 5723.84 | bwd_inner: 5711.14 | bwd_allreduce: 12.66 | step: 18.45

  5%|▍         | 1901/41250 [4:35:29<94:57:31,  8.69s/it]
                                                         
{'loss': 0.0916, 'grad_norm': 0.821244478225708, 'learning_rate': 3.9972907554603446e-05, 'epoch': 0.46}

  5%|▍         | 1901/41250 [4:35:29<94:57:31,  8.69s/it][2025-04-26 08:44:07,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.99
[2025-04-26 08:44:07,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.80 | bwd_microstep: 5798.02 | bwd_inner_microstep: 5648.15 | bwd_allreduce_microstep: 149.81 | step_microstep: 18.95
[2025-04-26 08:44:07,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.80 | bwd: 5798.03 | bwd_inner: 5648.15 | bwd_allreduce: 149.84 | step: 18.96

  5%|▍         | 1902/41250 [4:35:38<95:02:22,  8.70s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 1.596463918685913, 'learning_rate': 3.997282578466173e-05, 'epoch': 0.46}

  5%|▍         | 1902/41250 [4:35:38<95:02:22,  8.70s/it][2025-04-26 08:44:16,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:44:16,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.26 | bwd_microstep: 5688.75 | bwd_inner_microstep: 5658.20 | bwd_allreduce_microstep: 30.51 | step_microstep: 17.90
[2025-04-26 08:44:16,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.26 | bwd: 5688.76 | bwd_inner: 5658.20 | bwd_allreduce: 30.52 | step: 17.90

  5%|▍         | 1903/41250 [4:35:47<94:47:27,  8.67s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 2.56343674659729, 'learning_rate': 3.9972743891591477e-05, 'epoch': 0.46}

  5%|▍         | 1903/41250 [4:35:47<94:47:27,  8.67s/it][2025-04-26 08:44:24,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:44:24,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.59 | bwd_microstep: 5715.01 | bwd_inner_microstep: 5657.72 | bwd_allreduce_microstep: 57.25 | step_microstep: 18.38
[2025-04-26 08:44:24,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.59 | bwd: 5715.02 | bwd_inner: 5657.72 | bwd_allreduce: 57.26 | step: 18.38

  5%|▍         | 1904/41250 [4:35:55<94:40:02,  8.66s/it]
                                                         
{'loss': 0.1415, 'grad_norm': 3.6718404293060303, 'learning_rate': 3.997266187539319e-05, 'epoch': 0.46}

  5%|▍         | 1904/41250 [4:35:55<94:40:02,  8.66s/it][2025-04-26 08:44:33,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 08:44:33,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.64 | bwd_microstep: 5792.37 | bwd_inner_microstep: 5639.33 | bwd_allreduce_microstep: 153.00 | step_microstep: 18.37
[2025-04-26 08:44:33,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.64 | bwd: 5792.39 | bwd_inner: 5639.33 | bwd_allreduce: 153.02 | step: 18.37

  5%|▍         | 1905/41250 [4:36:04<94:47:24,  8.67s/it]
                                                         
{'loss': 0.3131, 'grad_norm': 3.091588258743286, 'learning_rate': 3.9972579736067355e-05, 'epoch': 0.46}

  5%|▍         | 1905/41250 [4:36:04<94:47:24,  8.67s/it][2025-04-26 08:44:42,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:44:42,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2928.24 | bwd_microstep: 5874.41 | bwd_inner_microstep: 5861.67 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.47
[2025-04-26 08:44:42,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2928.24 | bwd: 5874.42 | bwd_inner: 5861.67 | bwd_allreduce: 12.71 | step: 18.47

  5%|▍         | 1906/41250 [4:36:13<95:27:32,  8.73s/it]
                                                         
{'loss': 0.2399, 'grad_norm': 4.271574020385742, 'learning_rate': 3.99724974736145e-05, 'epoch': 0.46}

  5%|▍         | 1906/41250 [4:36:13<95:27:32,  8.73s/it][2025-04-26 08:44:50,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:44:50,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.77 | bwd_microstep: 5778.96 | bwd_inner_microstep: 5649.41 | bwd_allreduce_microstep: 129.50 | step_microstep: 18.42
[2025-04-26 08:44:50,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.77 | bwd: 5778.97 | bwd_inner: 5649.41 | bwd_allreduce: 129.52 | step: 18.42

  5%|▍         | 1907/41250 [4:36:22<95:18:39,  8.72s/it]
                                                         
{'loss': 0.131, 'grad_norm': 2.1669015884399414, 'learning_rate': 3.997241508803513e-05, 'epoch': 0.46}

  5%|▍         | 1907/41250 [4:36:22<95:18:39,  8.72s/it][2025-04-26 08:44:59,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:44:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.36 | bwd_microstep: 5709.39 | bwd_inner_microstep: 5696.60 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.46
[2025-04-26 08:44:59,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.36 | bwd: 5709.40 | bwd_inner: 5696.60 | bwd_allreduce: 12.77 | step: 18.46

  5%|▍         | 1908/41250 [4:36:30<95:01:37,  8.70s/it]
                                                         
{'loss': 0.2381, 'grad_norm': 2.02497935295105, 'learning_rate': 3.997233257932975e-05, 'epoch': 0.46}

  5%|▍         | 1908/41250 [4:36:30<95:01:37,  8.70s/it][2025-04-26 08:45:08,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:45:08,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.78 | bwd_microstep: 5772.52 | bwd_inner_microstep: 5649.29 | bwd_allreduce_microstep: 123.18 | step_microstep: 18.54
[2025-04-26 08:45:08,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.78 | bwd: 5772.53 | bwd_inner: 5649.29 | bwd_allreduce: 123.20 | step: 18.56

  5%|▍         | 1909/41250 [4:36:39<95:00:30,  8.69s/it]
                                                         
{'loss': 0.0392, 'grad_norm': 0.5284792184829712, 'learning_rate': 3.997224994749886e-05, 'epoch': 0.46}

  5%|▍         | 1909/41250 [4:36:39<95:00:30,  8.69s/it][2025-04-26 08:45:16,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:45:16,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.84 | bwd_microstep: 5760.27 | bwd_inner_microstep: 5686.03 | bwd_allreduce_microstep: 74.19 | step_microstep: 18.40
[2025-04-26 08:45:16,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.84 | bwd: 5760.28 | bwd_inner: 5686.02 | bwd_allreduce: 74.21 | step: 18.40

  5%|▍         | 1910/41250 [4:36:48<94:55:34,  8.69s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 2.7538058757781982, 'learning_rate': 3.997216719254298e-05, 'epoch': 0.46}

  5%|▍         | 1910/41250 [4:36:48<94:55:34,  8.69s/it][2025-04-26 08:45:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 08:45:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.88 | bwd_microstep: 5760.25 | bwd_inner_microstep: 5639.48 | bwd_allreduce_microstep: 120.73 | step_microstep: 18.83
[2025-04-26 08:45:25,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.88 | bwd: 5760.27 | bwd_inner: 5639.48 | bwd_allreduce: 120.75 | step: 18.84

  5%|▍         | 1911/41250 [4:36:56<94:54:38,  8.69s/it]
                                                         
{'loss': 0.0895, 'grad_norm': 1.6554523706436157, 'learning_rate': 3.997208431446262e-05, 'epoch': 0.46}

  5%|▍         | 1911/41250 [4:36:56<94:54:38,  8.69s/it][2025-04-26 08:45:34,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:45:34,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.82 | bwd_microstep: 5697.88 | bwd_inner_microstep: 5685.04 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.66
[2025-04-26 08:45:34,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 5697.90 | bwd_inner: 5685.04 | bwd_allreduce: 12.82 | step: 18.67

  5%|▍         | 1912/41250 [4:37:05<94:45:20,  8.67s/it]
                                                         
{'loss': 0.2767, 'grad_norm': 1.81400728225708, 'learning_rate': 3.997200131325829e-05, 'epoch': 0.46}

  5%|▍         | 1912/41250 [4:37:05<94:45:20,  8.67s/it][2025-04-26 08:45:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:45:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.48 | bwd_microstep: 5682.33 | bwd_inner_microstep: 5641.67 | bwd_allreduce_microstep: 40.62 | step_microstep: 18.45
[2025-04-26 08:45:42,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.48 | bwd: 5682.34 | bwd_inner: 5641.67 | bwd_allreduce: 40.63 | step: 18.45

  5%|▍         | 1913/41250 [4:37:14<94:29:16,  8.65s/it]
                                                         
{'loss': 0.0903, 'grad_norm': 1.5947879552841187, 'learning_rate': 3.99719181889305e-05, 'epoch': 0.46}

  5%|▍         | 1913/41250 [4:37:14<94:29:16,  8.65s/it][2025-04-26 08:45:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 08:45:51,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5733.35 | bwd_inner_microstep: 5691.30 | bwd_allreduce_microstep: 42.00 | step_microstep: 18.92
[2025-04-26 08:45:51,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5733.36 | bwd_inner: 5691.30 | bwd_allreduce: 42.02 | step: 18.93

  5%|▍         | 1914/41250 [4:37:22<94:31:40,  8.65s/it]
                                                         
{'loss': 0.1676, 'grad_norm': 1.2009427547454834, 'learning_rate': 3.997183494147976e-05, 'epoch': 0.46}

  5%|▍         | 1914/41250 [4:37:22<94:31:40,  8.65s/it][2025-04-26 08:46:00,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:46:00,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5736.32 | bwd_inner_microstep: 5683.60 | bwd_allreduce_microstep: 52.67 | step_microstep: 18.43
[2025-04-26 08:46:00,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5736.33 | bwd_inner: 5683.60 | bwd_allreduce: 52.69 | step: 18.43

  5%|▍         | 1915/41250 [4:37:31<94:35:06,  8.66s/it]
                                                         
{'loss': 0.1445, 'grad_norm': 1.0130444765090942, 'learning_rate': 3.997175157090659e-05, 'epoch': 0.46}

  5%|▍         | 1915/41250 [4:37:31<94:35:06,  8.66s/it][2025-04-26 08:46:08,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:46:08,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5691.98 | bwd_inner_microstep: 5679.56 | bwd_allreduce_microstep: 12.38 | step_microstep: 18.55
[2025-04-26 08:46:08,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5692.00 | bwd_inner: 5679.56 | bwd_allreduce: 12.39 | step: 18.55

  5%|▍         | 1916/41250 [4:37:40<94:29:09,  8.65s/it]
                                                         
{'loss': 0.1249, 'grad_norm': 1.4366776943206787, 'learning_rate': 3.99716680772115e-05, 'epoch': 0.46}

  5%|▍         | 1916/41250 [4:37:40<94:29:09,  8.65s/it][2025-04-26 08:46:17,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 08:46:17,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.14 | bwd_microstep: 5740.37 | bwd_inner_microstep: 5660.31 | bwd_allreduce_microstep: 80.01 | step_microstep: 18.50
[2025-04-26 08:46:17,481] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.14 | bwd: 5740.39 | bwd_inner: 5660.31 | bwd_allreduce: 80.03 | step: 18.50

  5%|▍         | 1917/41250 [4:37:48<94:30:17,  8.65s/it]
                                                         
{'loss': 0.2521, 'grad_norm': 2.8551223278045654, 'learning_rate': 3.9971584460395e-05, 'epoch': 0.46}

  5%|▍         | 1917/41250 [4:37:48<94:30:17,  8.65s/it][2025-04-26 08:46:26,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:46:26,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.50 | bwd_microstep: 5762.93 | bwd_inner_microstep: 5647.52 | bwd_allreduce_microstep: 115.36 | step_microstep: 18.71
[2025-04-26 08:46:26,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.50 | bwd: 5762.94 | bwd_inner: 5647.52 | bwd_allreduce: 115.38 | step: 18.71

  5%|▍         | 1918/41250 [4:37:57<94:34:38,  8.66s/it]
                                                         
{'loss': 0.1282, 'grad_norm': 1.4222979545593262, 'learning_rate': 3.997150072045761e-05, 'epoch': 0.46}

  5%|▍         | 1918/41250 [4:37:57<94:34:38,  8.66s/it][2025-04-26 08:46:34,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 08:46:34,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.21 | bwd_microstep: 5852.62 | bwd_inner_microstep: 5687.94 | bwd_allreduce_microstep: 164.63 | step_microstep: 18.67
[2025-04-26 08:46:34,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.21 | bwd: 5852.64 | bwd_inner: 5687.94 | bwd_allreduce: 164.65 | step: 18.67

  5%|▍         | 1919/41250 [4:38:06<94:57:22,  8.69s/it]
                                                         
{'loss': 0.3386, 'grad_norm': 2.304979085922241, 'learning_rate': 3.997141685739985e-05, 'epoch': 0.47}

  5%|▍         | 1919/41250 [4:38:06<94:57:22,  8.69s/it][2025-04-26 08:46:43,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 1.03
[2025-04-26 08:46:43,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5699.67 | bwd_inner_microstep: 5640.78 | bwd_allreduce_microstep: 58.85 | step_microstep: 19.01
[2025-04-26 08:46:43,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5699.69 | bwd_inner: 5640.78 | bwd_allreduce: 58.87 | step: 19.01

  5%|▍         | 1920/41250 [4:38:14<94:41:30,  8.67s/it]
                                                         
{'loss': 0.1358, 'grad_norm': 1.2035489082336426, 'learning_rate': 3.997133287122223e-05, 'epoch': 0.47}

  5%|▍         | 1920/41250 [4:38:14<94:41:30,  8.67s/it][2025-04-26 08:46:52,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:46:52,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.06 | bwd_microstep: 5757.23 | bwd_inner_microstep: 5642.26 | bwd_allreduce_microstep: 114.92 | step_microstep: 18.40
[2025-04-26 08:46:52,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.06 | bwd: 5757.24 | bwd_inner: 5642.26 | bwd_allreduce: 114.93 | step: 18.41

  5%|▍         | 1921/41250 [4:38:23<94:41:13,  8.67s/it]
                                                         
{'loss': 0.0982, 'grad_norm': 1.9951997995376587, 'learning_rate': 3.997124876192527e-05, 'epoch': 0.47}

  5%|▍         | 1921/41250 [4:38:23<94:41:13,  8.67s/it][2025-04-26 08:47:00,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:47:00,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.37 | bwd_microstep: 5779.88 | bwd_inner_microstep: 5653.89 | bwd_allreduce_microstep: 125.95 | step_microstep: 18.10
[2025-04-26 08:47:00,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.37 | bwd: 5779.90 | bwd_inner: 5653.89 | bwd_allreduce: 125.97 | step: 18.11

  5%|▍         | 1922/41250 [4:38:32<94:43:54,  8.67s/it]
                                                         
{'loss': 0.4132, 'grad_norm': 2.523801803588867, 'learning_rate': 3.997116452950949e-05, 'epoch': 0.47}

  5%|▍         | 1922/41250 [4:38:32<94:43:54,  8.67s/it][2025-04-26 08:47:09,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:47:09,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.19 | bwd_microstep: 5739.46 | bwd_inner_microstep: 5693.44 | bwd_allreduce_microstep: 45.98 | step_microstep: 18.29
[2025-04-26 08:47:09,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.19 | bwd: 5739.47 | bwd_inner: 5693.44 | bwd_allreduce: 46.00 | step: 18.29

  5%|▍         | 1923/41250 [4:38:40<94:42:22,  8.67s/it]
                                                         
{'loss': 0.5266, 'grad_norm': 3.2070136070251465, 'learning_rate': 3.997108017397542e-05, 'epoch': 0.47}

  5%|▍         | 1923/41250 [4:38:40<94:42:22,  8.67s/it][2025-04-26 08:47:18,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:47:18,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.21 | bwd_microstep: 5690.28 | bwd_inner_microstep: 5650.31 | bwd_allreduce_microstep: 39.93 | step_microstep: 17.93
[2025-04-26 08:47:18,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.21 | bwd: 5690.30 | bwd_inner: 5650.31 | bwd_allreduce: 39.95 | step: 17.94

  5%|▍         | 1924/41250 [4:38:49<94:28:45,  8.65s/it]
                                                         
{'loss': 0.2644, 'grad_norm': 1.6472595930099487, 'learning_rate': 3.997099569532355e-05, 'epoch': 0.47}

  5%|▍         | 1924/41250 [4:38:49<94:28:45,  8.65s/it][2025-04-26 08:47:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:47:26,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.41 | bwd_microstep: 5737.31 | bwd_inner_microstep: 5696.92 | bwd_allreduce_microstep: 40.34 | step_microstep: 18.48
[2025-04-26 08:47:26,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.41 | bwd: 5737.32 | bwd_inner: 5696.92 | bwd_allreduce: 40.36 | step: 18.48

  5%|▍         | 1925/41250 [4:38:58<94:33:00,  8.66s/it]
                                                         
{'loss': 0.1839, 'grad_norm': 1.8866631984710693, 'learning_rate': 3.997091109355443e-05, 'epoch': 0.47}

  5%|▍         | 1925/41250 [4:38:58<94:33:00,  8.66s/it][2025-04-26 08:47:35,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:47:35,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.38 | bwd_microstep: 5717.27 | bwd_inner_microstep: 5704.19 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.74
[2025-04-26 08:47:35,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.38 | bwd: 5717.28 | bwd_inner: 5704.19 | bwd_allreduce: 13.05 | step: 18.74

  5%|▍         | 1926/41250 [4:39:06<94:31:39,  8.65s/it]
                                                         
{'loss': 0.2658, 'grad_norm': 1.5446412563323975, 'learning_rate': 3.9970826368668574e-05, 'epoch': 0.47}

  5%|▍         | 1926/41250 [4:39:06<94:31:39,  8.65s/it][2025-04-26 08:47:44,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.31 | optimizer_step: 1.04
[2025-04-26 08:47:44,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5748.50 | bwd_inner_microstep: 5695.30 | bwd_allreduce_microstep: 53.14 | step_microstep: 19.98
[2025-04-26 08:47:44,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5748.51 | bwd_inner: 5695.30 | bwd_allreduce: 53.17 | step: 19.98

  5%|▍         | 1927/41250 [4:39:15<94:38:02,  8.66s/it]
                                                         
{'loss': 0.0976, 'grad_norm': 1.4073776006698608, 'learning_rate': 3.9970741520666496e-05, 'epoch': 0.47}

  5%|▍         | 1927/41250 [4:39:15<94:38:02,  8.66s/it][2025-04-26 08:47:52,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 08:47:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.73 | bwd_microstep: 5729.19 | bwd_inner_microstep: 5703.69 | bwd_allreduce_microstep: 25.46 | step_microstep: 18.55
[2025-04-26 08:47:52,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.73 | bwd: 5729.21 | bwd_inner: 5703.69 | bwd_allreduce: 25.48 | step: 18.55

  5%|▍         | 1928/41250 [4:39:24<94:38:06,  8.66s/it]
                                                         
{'loss': 0.2641, 'grad_norm': 1.8494008779525757, 'learning_rate': 3.997065654954872e-05, 'epoch': 0.47}

  5%|▍         | 1928/41250 [4:39:24<94:38:06,  8.66s/it][2025-04-26 08:48:01,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 08:48:01,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.51 | bwd_microstep: 5773.79 | bwd_inner_microstep: 5653.86 | bwd_allreduce_microstep: 119.88 | step_microstep: 18.94
[2025-04-26 08:48:01,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.51 | bwd: 5773.80 | bwd_inner: 5653.86 | bwd_allreduce: 119.90 | step: 18.95

  5%|▍         | 1929/41250 [4:39:32<94:41:48,  8.67s/it]
                                                         
{'loss': 0.0422, 'grad_norm': 0.6913076043128967, 'learning_rate': 3.997057145531578e-05, 'epoch': 0.47}

  5%|▍         | 1929/41250 [4:39:32<94:41:48,  8.67s/it][2025-04-26 08:48:10,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 08:48:10,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.07 | bwd_microstep: 5697.95 | bwd_inner_microstep: 5661.89 | bwd_allreduce_microstep: 36.02 | step_microstep: 18.88
[2025-04-26 08:48:10,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.07 | bwd: 5697.97 | bwd_inner: 5661.89 | bwd_allreduce: 36.04 | step: 18.88

  5%|▍         | 1930/41250 [4:39:41<94:29:49,  8.65s/it]
                                                         
{'loss': 0.2968, 'grad_norm': 2.4656968116760254, 'learning_rate': 3.99704862379682e-05, 'epoch': 0.47}

  5%|▍         | 1930/41250 [4:39:41<94:29:49,  8.65s/it][2025-04-26 08:48:18,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 08:48:18,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.31 | bwd_microstep: 5698.31 | bwd_inner_microstep: 5685.38 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.90
[2025-04-26 08:48:18,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5698.33 | bwd_inner: 5685.38 | bwd_allreduce: 12.90 | step: 18.90

  5%|▍         | 1931/41250 [4:39:49<94:25:38,  8.65s/it]
                                                         
{'loss': 0.0663, 'grad_norm': 0.8240885734558105, 'learning_rate': 3.9970400897506495e-05, 'epoch': 0.47}

  5%|▍         | 1931/41250 [4:39:49<94:25:38,  8.65s/it][2025-04-26 08:48:27,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-26 08:48:27,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.73 | bwd_microstep: 5721.09 | bwd_inner_microstep: 5661.26 | bwd_allreduce_microstep: 59.78 | step_microstep: 19.11
[2025-04-26 08:48:27,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.73 | bwd: 5721.10 | bwd_inner: 5661.26 | bwd_allreduce: 59.80 | step: 19.11

  5%|▍         | 1932/41250 [4:39:58<94:22:52,  8.64s/it]
                                                         
{'loss': 0.1818, 'grad_norm': 1.3498049974441528, 'learning_rate': 3.99703154339312e-05, 'epoch': 0.47}

  5%|▍         | 1932/41250 [4:39:58<94:22:52,  8.64s/it][2025-04-26 08:48:36,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 08:48:36,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5722.46 | bwd_inner_microstep: 5635.62 | bwd_allreduce_microstep: 86.79 | step_microstep: 18.90
[2025-04-26 08:48:36,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5722.48 | bwd_inner: 5635.62 | bwd_allreduce: 86.81 | step: 18.90

  5%|▍         | 1933/41250 [4:40:07<94:19:34,  8.64s/it]
                                                         
{'loss': 0.2422, 'grad_norm': 1.6752806901931763, 'learning_rate': 3.9970229847242833e-05, 'epoch': 0.47}

  5%|▍         | 1933/41250 [4:40:07<94:19:34,  8.64s/it][2025-04-26 08:48:44,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:48:44,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5703.55 | bwd_inner_microstep: 5690.85 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.90
[2025-04-26 08:48:44,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5703.56 | bwd_inner: 5690.85 | bwd_allreduce: 12.67 | step: 18.90

  5%|▍         | 1934/41250 [4:40:15<94:19:14,  8.64s/it]
                                                         
{'loss': 0.4199, 'grad_norm': 3.1943564414978027, 'learning_rate': 3.997014413744193e-05, 'epoch': 0.47}

  5%|▍         | 1934/41250 [4:40:15<94:19:14,  8.64s/it][2025-04-26 08:48:53,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 08:48:53,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5734.55 | bwd_inner_microstep: 5711.23 | bwd_allreduce_microstep: 23.27 | step_microstep: 18.69
[2025-04-26 08:48:53,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5734.57 | bwd_inner: 5711.23 | bwd_allreduce: 23.30 | step: 18.69

  5%|▍         | 1935/41250 [4:40:24<94:26:24,  8.65s/it]
                                                         
{'loss': 0.2778, 'grad_norm': 2.0066020488739014, 'learning_rate': 3.997005830452901e-05, 'epoch': 0.47}

  5%|▍         | 1935/41250 [4:40:24<94:26:24,  8.65s/it][2025-04-26 08:49:01,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.10
[2025-04-26 08:49:01,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.34 | bwd_microstep: 5716.67 | bwd_inner_microstep: 5649.76 | bwd_allreduce_microstep: 66.87 | step_microstep: 18.98
[2025-04-26 08:49:01,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.34 | bwd: 5716.68 | bwd_inner: 5649.75 | bwd_allreduce: 66.89 | step: 18.98

  5%|▍         | 1936/41250 [4:40:33<94:22:02,  8.64s/it]
                                                         
{'loss': 0.1488, 'grad_norm': 1.7264267206192017, 'learning_rate': 3.996997234850461e-05, 'epoch': 0.47}

  5%|▍         | 1936/41250 [4:40:33<94:22:02,  8.64s/it][2025-04-26 08:49:10,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 08:49:10,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.84 | bwd_microstep: 5710.39 | bwd_inner_microstep: 5697.47 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.44
[2025-04-26 08:49:10,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.84 | bwd: 5710.40 | bwd_inner: 5697.47 | bwd_allreduce: 12.89 | step: 18.44

  5%|▍         | 1937/41250 [4:40:41<94:20:57,  8.64s/it]
                                                         
{'loss': 0.1298, 'grad_norm': 1.2763020992279053, 'learning_rate': 3.996988626936927e-05, 'epoch': 0.47}

  5%|▍         | 1937/41250 [4:40:41<94:20:57,  8.64s/it][2025-04-26 08:49:19,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:49:19,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.47 | bwd_microstep: 5713.00 | bwd_inner_microstep: 5699.94 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.77
[2025-04-26 08:49:19,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.47 | bwd: 5713.02 | bwd_inner: 5699.94 | bwd_allreduce: 13.04 | step: 18.77

  5%|▍         | 1938/41250 [4:40:50<94:21:03,  8.64s/it]
                                                         
{'loss': 0.366, 'grad_norm': 2.041201114654541, 'learning_rate': 3.9969800067123503e-05, 'epoch': 0.47}

  5%|▍         | 1938/41250 [4:40:50<94:21:03,  8.64s/it][2025-04-26 08:49:27,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 08:49:27,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5705.39 | bwd_inner_microstep: 5653.66 | bwd_allreduce_microstep: 51.68 | step_microstep: 18.83
[2025-04-26 08:49:27,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5705.40 | bwd_inner: 5653.66 | bwd_allreduce: 51.69 | step: 18.83

  5%|▍         | 1939/41250 [4:40:59<94:15:24,  8.63s/it]
                                                         
{'loss': 0.1157, 'grad_norm': 0.9751440286636353, 'learning_rate': 3.9969713741767847e-05, 'epoch': 0.47}

  5%|▍         | 1939/41250 [4:40:59<94:15:24,  8.63s/it][2025-04-26 08:49:36,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 08:49:36,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.07 | bwd_microstep: 5731.39 | bwd_inner_microstep: 5718.62 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.67
[2025-04-26 08:49:36,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5731.40 | bwd_inner: 5718.62 | bwd_allreduce: 12.74 | step: 18.67

  5%|▍         | 1940/41250 [4:41:07<94:23:47,  8.64s/it]
                                                         
{'loss': 0.1177, 'grad_norm': 2.214425802230835, 'learning_rate': 3.996962729330283e-05, 'epoch': 0.47}

  5%|▍         | 1940/41250 [4:41:07<94:23:47,  8.64s/it][2025-04-26 08:49:45,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 08:49:45,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.16 | bwd_microstep: 5724.68 | bwd_inner_microstep: 5663.18 | bwd_allreduce_microstep: 61.45 | step_microstep: 18.98
[2025-04-26 08:49:45,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.16 | bwd: 5724.69 | bwd_inner: 5663.18 | bwd_allreduce: 61.47 | step: 18.99

  5%|▍         | 1941/41250 [4:41:16<94:24:20,  8.65s/it]
                                                         
{'loss': 0.3842, 'grad_norm': 2.7968552112579346, 'learning_rate': 3.9969540721728985e-05, 'epoch': 0.47}

  5%|▍         | 1941/41250 [4:41:16<94:24:20,  8.65s/it][2025-04-26 08:49:53,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.06 | optimizer_step: 1.04
[2025-04-26 08:49:53,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5781.12 | bwd_inner_microstep: 5701.50 | bwd_allreduce_microstep: 79.58 | step_microstep: 19.56
[2025-04-26 08:49:53,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5781.14 | bwd_inner: 5701.50 | bwd_allreduce: 79.60 | step: 19.57

  5%|▍         | 1942/41250 [4:41:25<94:37:32,  8.67s/it]
                                                         
{'loss': 0.1417, 'grad_norm': 0.9405021071434021, 'learning_rate': 3.996945402704686e-05, 'epoch': 0.47}

  5%|▍         | 1942/41250 [4:41:25<94:37:32,  8.67s/it][2025-04-26 08:50:02,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:50:02,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.21 | bwd_microstep: 5745.02 | bwd_inner_microstep: 5643.70 | bwd_allreduce_microstep: 101.27 | step_microstep: 18.81
[2025-04-26 08:50:02,526] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.21 | bwd: 5745.03 | bwd_inner: 5643.70 | bwd_allreduce: 101.29 | step: 18.81

  5%|▍         | 1943/41250 [4:41:33<94:35:18,  8.66s/it]
                                                         
{'loss': 0.4903, 'grad_norm': 2.1598002910614014, 'learning_rate': 3.996936720925697e-05, 'epoch': 0.47}

  5%|▍         | 1943/41250 [4:41:33<94:35:18,  8.66s/it][2025-04-26 08:50:11,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:50:11,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5726.86 | bwd_inner_microstep: 5714.01 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.71
[2025-04-26 08:50:11,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5726.87 | bwd_inner: 5714.01 | bwd_allreduce: 12.82 | step: 18.71

  5%|▍         | 1944/41250 [4:41:42<94:35:26,  8.66s/it]
                                                         
{'loss': 0.2727, 'grad_norm': 2.0796337127685547, 'learning_rate': 3.996928026835986e-05, 'epoch': 0.47}

  5%|▍         | 1944/41250 [4:41:42<94:35:26,  8.66s/it][2025-04-26 08:50:19,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:50:19,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.54 | bwd_microstep: 5757.79 | bwd_inner_microstep: 5706.59 | bwd_allreduce_microstep: 51.16 | step_microstep: 18.62
[2025-04-26 08:50:19,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.54 | bwd: 5757.81 | bwd_inner: 5706.59 | bwd_allreduce: 51.17 | step: 18.62

  5%|▍         | 1945/41250 [4:41:51<94:39:58,  8.67s/it]
                                                         
{'loss': 0.2913, 'grad_norm': 2.018165111541748, 'learning_rate': 3.996919320435607e-05, 'epoch': 0.47}

  5%|▍         | 1945/41250 [4:41:51<94:39:58,  8.67s/it][2025-04-26 08:50:28,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 08:50:28,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.16 | bwd_microstep: 5711.16 | bwd_inner_microstep: 5698.09 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.81
[2025-04-26 08:50:28,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.16 | bwd: 5711.18 | bwd_inner: 5698.09 | bwd_allreduce: 13.04 | step: 18.81

  5%|▍         | 1946/41250 [4:41:59<94:32:36,  8.66s/it]
                                                         
{'loss': 0.2453, 'grad_norm': 1.073030948638916, 'learning_rate': 3.9969106017246126e-05, 'epoch': 0.47}

  5%|▍         | 1946/41250 [4:41:59<94:32:36,  8.66s/it][2025-04-26 08:50:37,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-26 08:50:37,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.00 | bwd_microstep: 5757.01 | bwd_inner_microstep: 5711.65 | bwd_allreduce_microstep: 45.32 | step_microstep: 18.91
[2025-04-26 08:50:37,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.00 | bwd: 5757.02 | bwd_inner: 5711.64 | bwd_allreduce: 45.34 | step: 18.91

  5%|▍         | 1947/41250 [4:42:08<94:38:03,  8.67s/it]
                                                         
{'loss': 0.4115, 'grad_norm': 2.8522825241088867, 'learning_rate': 3.996901870703058e-05, 'epoch': 0.47}

  5%|▍         | 1947/41250 [4:42:08<94:38:03,  8.67s/it][2025-04-26 08:50:45,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 08:50:45,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.12 | bwd_microstep: 5761.11 | bwd_inner_microstep: 5663.38 | bwd_allreduce_microstep: 97.69 | step_microstep: 18.60
[2025-04-26 08:50:45,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.12 | bwd: 5761.13 | bwd_inner: 5663.38 | bwd_allreduce: 97.71 | step: 18.60

  5%|▍         | 1948/41250 [4:42:17<94:41:06,  8.67s/it]
                                                         
{'loss': 0.2133, 'grad_norm': 1.5621047019958496, 'learning_rate': 3.996893127370995e-05, 'epoch': 0.47}

  5%|▍         | 1948/41250 [4:42:17<94:41:06,  8.67s/it][2025-04-26 08:50:54,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:50:54,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.56 | bwd_microstep: 5705.42 | bwd_inner_microstep: 5692.48 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.32
[2025-04-26 08:50:54,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.56 | bwd: 5705.44 | bwd_inner: 5692.48 | bwd_allreduce: 12.91 | step: 18.32

  5%|▍         | 1949/41250 [4:42:25<94:33:48,  8.66s/it]
                                                         
{'loss': 0.2717, 'grad_norm': 1.4607081413269043, 'learning_rate': 3.996884371728479e-05, 'epoch': 0.47}

  5%|▍         | 1949/41250 [4:42:25<94:33:48,  8.66s/it][2025-04-26 08:51:03,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 08:51:03,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.73 | bwd_microstep: 5719.10 | bwd_inner_microstep: 5642.40 | bwd_allreduce_microstep: 76.66 | step_microstep: 18.18
[2025-04-26 08:51:03,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.73 | bwd: 5719.11 | bwd_inner: 5642.40 | bwd_allreduce: 76.68 | step: 18.19

  5%|▍         | 1950/41250 [4:42:34<94:25:46,  8.65s/it]
                                                         
{'loss': 0.1657, 'grad_norm': 1.2299716472625732, 'learning_rate': 3.996875603775564e-05, 'epoch': 0.47}

  5%|▍         | 1950/41250 [4:42:34<94:25:46,  8.65s/it][2025-04-26 08:51:11,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:51:11,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.33 | bwd_microstep: 5770.18 | bwd_inner_microstep: 5654.09 | bwd_allreduce_microstep: 116.05 | step_microstep: 18.03
[2025-04-26 08:51:11,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.33 | bwd: 5770.19 | bwd_inner: 5654.09 | bwd_allreduce: 116.06 | step: 18.04

  5%|▍         | 1951/41250 [4:42:43<94:30:42,  8.66s/it]
                                                         
{'loss': 0.23, 'grad_norm': 1.3521931171417236, 'learning_rate': 3.996866823512303e-05, 'epoch': 0.47}

  5%|▍         | 1951/41250 [4:42:43<94:30:42,  8.66s/it][2025-04-26 08:51:20,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:51:20,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.63 | bwd_microstep: 5792.31 | bwd_inner_microstep: 5652.20 | bwd_allreduce_microstep: 140.07 | step_microstep: 18.16
[2025-04-26 08:51:20,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.64 | bwd: 5792.32 | bwd_inner: 5652.20 | bwd_allreduce: 140.08 | step: 18.17

  5%|▍         | 1952/41250 [4:42:51<94:37:48,  8.67s/it]
                                                         
{'loss': 0.1997, 'grad_norm': 1.568192481994629, 'learning_rate': 3.996858030938752e-05, 'epoch': 0.47}

  5%|▍         | 1952/41250 [4:42:51<94:37:48,  8.67s/it][2025-04-26 08:51:29,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.87 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:51:29,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.86 | bwd_microstep: 5739.18 | bwd_inner_microstep: 5705.81 | bwd_allreduce_microstep: 33.33 | step_microstep: 17.59
[2025-04-26 08:51:29,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.86 | bwd: 5739.20 | bwd_inner: 5705.81 | bwd_allreduce: 33.34 | step: 17.59

  5%|▍         | 1953/41250 [4:43:00<94:39:42,  8.67s/it]
                                                         
{'loss': 0.3186, 'grad_norm': 2.1786067485809326, 'learning_rate': 3.9968492260549625e-05, 'epoch': 0.47}

  5%|▍         | 1953/41250 [4:43:00<94:39:42,  8.67s/it][2025-04-26 08:51:37,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:51:37,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5771.95 | bwd_inner_microstep: 5653.48 | bwd_allreduce_microstep: 118.43 | step_microstep: 18.39
[2025-04-26 08:51:37,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5771.97 | bwd_inner: 5653.48 | bwd_allreduce: 118.45 | step: 18.41

  5%|▍         | 1954/41250 [4:43:09<94:42:48,  8.68s/it]
                                                         
{'loss': 0.2067, 'grad_norm': 1.6022051572799683, 'learning_rate': 3.996840408860991e-05, 'epoch': 0.47}

  5%|▍         | 1954/41250 [4:43:09<94:42:48,  8.68s/it][2025-04-26 08:51:46,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 08:51:46,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2904.61 | bwd_microstep: 5735.54 | bwd_inner_microstep: 5700.42 | bwd_allreduce_microstep: 35.07 | step_microstep: 19.00
[2025-04-26 08:51:46,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2904.61 | bwd: 5735.55 | bwd_inner: 5700.42 | bwd_allreduce: 35.09 | step: 19.00

  5%|▍         | 1955/41250 [4:43:17<94:49:28,  8.69s/it]
                                                         
{'loss': 0.1829, 'grad_norm': 1.274886965751648, 'learning_rate': 3.9968315793568915e-05, 'epoch': 0.47}

  5%|▍         | 1955/41250 [4:43:17<94:49:28,  8.69s/it][2025-04-26 08:51:55,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.16
[2025-04-26 08:51:55,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.87 | bwd_microstep: 5766.54 | bwd_inner_microstep: 5681.36 | bwd_allreduce_microstep: 85.14 | step_microstep: 19.20
[2025-04-26 08:51:55,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.87 | bwd: 5766.56 | bwd_inner: 5681.36 | bwd_allreduce: 85.15 | step: 19.20

  5%|▍         | 1956/41250 [4:43:26<94:51:38,  8.69s/it]
                                                         
{'loss': 0.2028, 'grad_norm': 1.2281720638275146, 'learning_rate': 3.9968227375427176e-05, 'epoch': 0.47}

  5%|▍         | 1956/41250 [4:43:26<94:51:38,  8.69s/it][2025-04-26 08:52:03,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 1.00
[2025-04-26 08:52:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.62 | bwd_microstep: 5742.40 | bwd_inner_microstep: 5680.22 | bwd_allreduce_microstep: 62.13 | step_microstep: 19.24
[2025-04-26 08:52:03,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.62 | bwd: 5742.42 | bwd_inner: 5680.22 | bwd_allreduce: 62.16 | step: 19.25

  5%|▍         | 1957/41250 [4:43:35<94:49:17,  8.69s/it]
                                                         
{'loss': 0.1736, 'grad_norm': 1.2362834215164185, 'learning_rate': 3.9968138834185245e-05, 'epoch': 0.47}

  5%|▍         | 1957/41250 [4:43:35<94:49:17,  8.69s/it][2025-04-26 08:52:12,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 08:52:12,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.62 | bwd_microstep: 5683.76 | bwd_inner_microstep: 5643.52 | bwd_allreduce_microstep: 40.19 | step_microstep: 18.54
[2025-04-26 08:52:12,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.62 | bwd: 5683.77 | bwd_inner: 5643.52 | bwd_allreduce: 40.21 | step: 18.54

  5%|▍         | 1958/41250 [4:43:43<94:30:57,  8.66s/it]
                                                         
{'loss': 0.2447, 'grad_norm': 1.0954816341400146, 'learning_rate': 3.996805016984366e-05, 'epoch': 0.47}

  5%|▍         | 1958/41250 [4:43:43<94:30:57,  8.66s/it][2025-04-26 08:52:21,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 08:52:21,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.85 | bwd_microstep: 5761.77 | bwd_inner_microstep: 5646.45 | bwd_allreduce_microstep: 115.28 | step_microstep: 18.81
[2025-04-26 08:52:21,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.85 | bwd: 5761.79 | bwd_inner: 5646.45 | bwd_allreduce: 115.29 | step: 18.81

  5%|▍         | 1959/41250 [4:43:52<94:32:48,  8.66s/it]
                                                         
{'loss': 0.2602, 'grad_norm': 2.0656633377075195, 'learning_rate': 3.996796138240298e-05, 'epoch': 0.47}

  5%|▍         | 1959/41250 [4:43:52<94:32:48,  8.66s/it][2025-04-26 08:52:29,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 1.02
[2025-04-26 08:52:29,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.37 | bwd_microstep: 5739.55 | bwd_inner_microstep: 5688.12 | bwd_allreduce_microstep: 51.39 | step_microstep: 17.89
[2025-04-26 08:52:29,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.37 | bwd: 5739.56 | bwd_inner: 5688.12 | bwd_allreduce: 51.41 | step: 17.89

  5%|▍         | 1960/41250 [4:44:01<94:31:33,  8.66s/it]
                                                         
{'loss': 0.1542, 'grad_norm': 0.9887537956237793, 'learning_rate': 3.996787247186374e-05, 'epoch': 0.48}

  5%|▍         | 1960/41250 [4:44:01<94:31:33,  8.66s/it][2025-04-26 08:52:38,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 08:52:38,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.14 | bwd_microstep: 5689.37 | bwd_inner_microstep: 5675.92 | bwd_allreduce_microstep: 13.41 | step_microstep: 18.62
[2025-04-26 08:52:38,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.14 | bwd: 5689.39 | bwd_inner: 5675.92 | bwd_allreduce: 13.43 | step: 18.63

  5%|▍         | 1961/41250 [4:44:09<94:24:30,  8.65s/it]
                                                         
{'loss': 0.2352, 'grad_norm': 2.424321413040161, 'learning_rate': 3.9967783438226496e-05, 'epoch': 0.48}

  5%|▍         | 1961/41250 [4:44:09<94:24:30,  8.65s/it][2025-04-26 08:52:47,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 08:52:47,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.13 | bwd_microstep: 5718.45 | bwd_inner_microstep: 5705.74 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.52
[2025-04-26 08:52:47,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.13 | bwd: 5718.47 | bwd_inner: 5705.74 | bwd_allreduce: 12.69 | step: 18.52

  5%|▍         | 1962/41250 [4:44:18<94:26:25,  8.65s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 1.1723096370697021, 'learning_rate': 3.99676942814918e-05, 'epoch': 0.48}

  5%|▍         | 1962/41250 [4:44:18<94:26:25,  8.65s/it][2025-04-26 08:52:55,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 08:52:55,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5736.97 | bwd_inner_microstep: 5631.65 | bwd_allreduce_microstep: 105.28 | step_microstep: 18.56
[2025-04-26 08:52:55,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5736.98 | bwd_inner: 5631.65 | bwd_allreduce: 105.30 | step: 18.56

  5%|▍         | 1963/41250 [4:44:27<94:25:19,  8.65s/it]
                                                         
{'loss': 0.1773, 'grad_norm': 1.4181842803955078, 'learning_rate': 3.996760500166019e-05, 'epoch': 0.48}

  5%|▍         | 1963/41250 [4:44:27<94:25:19,  8.65s/it][2025-04-26 08:53:04,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 08:53:04,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.71 | bwd_microstep: 5701.16 | bwd_inner_microstep: 5642.89 | bwd_allreduce_microstep: 58.22 | step_microstep: 18.49
[2025-04-26 08:53:04,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.71 | bwd: 5701.17 | bwd_inner: 5642.89 | bwd_allreduce: 58.24 | step: 18.50

  5%|▍         | 1964/41250 [4:44:35<94:13:37,  8.63s/it]
                                                         
{'loss': 0.0923, 'grad_norm': 0.6252028346061707, 'learning_rate': 3.9967515598732226e-05, 'epoch': 0.48}

  5%|▍         | 1964/41250 [4:44:35<94:13:37,  8.63s/it][2025-04-26 08:53:13,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 08:53:13,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.04 | bwd_microstep: 5741.00 | bwd_inner_microstep: 5688.51 | bwd_allreduce_microstep: 52.45 | step_microstep: 17.79
[2025-04-26 08:53:13,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.04 | bwd: 5741.01 | bwd_inner: 5688.51 | bwd_allreduce: 52.47 | step: 17.79

  5%|▍         | 1965/41250 [4:44:44<94:20:47,  8.65s/it]
                                                         
{'loss': 0.0647, 'grad_norm': 0.5706114172935486, 'learning_rate': 3.996742607270845e-05, 'epoch': 0.48}

  5%|▍         | 1965/41250 [4:44:44<94:20:47,  8.65s/it][2025-04-26 08:53:21,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:53:21,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.40 | bwd_microstep: 5765.08 | bwd_inner_microstep: 5644.21 | bwd_allreduce_microstep: 120.83 | step_microstep: 18.10
[2025-04-26 08:53:21,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.40 | bwd: 5765.10 | bwd_inner: 5644.21 | bwd_allreduce: 120.85 | step: 18.10

  5%|▍         | 1966/41250 [4:44:52<94:24:13,  8.65s/it]
                                                         
{'loss': 0.1921, 'grad_norm': 1.2127909660339355, 'learning_rate': 3.9967336423589425e-05, 'epoch': 0.48}

  5%|▍         | 1966/41250 [4:44:52<94:24:13,  8.65s/it][2025-04-26 08:53:30,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:53:30,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5712.31 | bwd_inner_microstep: 5699.56 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.37
[2025-04-26 08:53:30,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.18 | bwd: 5712.32 | bwd_inner: 5699.56 | bwd_allreduce: 12.72 | step: 18.37

  5%|▍         | 1967/41250 [4:45:01<94:23:57,  8.65s/it]
                                                         
{'loss': 0.1763, 'grad_norm': 1.4382320642471313, 'learning_rate': 3.99672466513757e-05, 'epoch': 0.48}

  5%|▍         | 1967/41250 [4:45:01<94:23:57,  8.65s/it][2025-04-26 08:53:39,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 08:53:39,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.97 | bwd_microstep: 5745.84 | bwd_inner_microstep: 5704.66 | bwd_allreduce_microstep: 41.14 | step_microstep: 18.01
[2025-04-26 08:53:39,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.97 | bwd: 5745.85 | bwd_inner: 5704.66 | bwd_allreduce: 41.15 | step: 18.01

  5%|▍         | 1968/41250 [4:45:10<94:27:35,  8.66s/it]
                                                         
{'loss': 0.265, 'grad_norm': 1.689176321029663, 'learning_rate': 3.996715675606782e-05, 'epoch': 0.48}

  5%|▍         | 1968/41250 [4:45:10<94:27:35,  8.66s/it][2025-04-26 08:53:47,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 08:53:47,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.74 | bwd_microstep: 5690.60 | bwd_inner_microstep: 5635.31 | bwd_allreduce_microstep: 55.24 | step_microstep: 19.23
[2025-04-26 08:53:47,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.74 | bwd: 5690.61 | bwd_inner: 5635.31 | bwd_allreduce: 55.27 | step: 19.23

  5%|▍         | 1969/41250 [4:45:18<94:16:08,  8.64s/it]
                                                         
{'loss': 0.1074, 'grad_norm': 1.8372948169708252, 'learning_rate': 3.9967066737666356e-05, 'epoch': 0.48}

  5%|▍         | 1969/41250 [4:45:18<94:16:08,  8.64s/it][2025-04-26 08:53:56,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:53:56,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5710.19 | bwd_inner_microstep: 5639.10 | bwd_allreduce_microstep: 71.04 | step_microstep: 18.59
[2025-04-26 08:53:56,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5710.21 | bwd_inner: 5639.10 | bwd_allreduce: 71.06 | step: 18.60

  5%|▍         | 1970/41250 [4:45:27<94:11:46,  8.63s/it]
                                                         
{'loss': 0.2069, 'grad_norm': 1.4660637378692627, 'learning_rate': 3.996697659617184e-05, 'epoch': 0.48}

  5%|▍         | 1970/41250 [4:45:27<94:11:46,  8.63s/it][2025-04-26 08:54:04,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 1.05
[2025-04-26 08:54:04,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.76 | bwd_microstep: 5748.72 | bwd_inner_microstep: 5694.10 | bwd_allreduce_microstep: 54.57 | step_microstep: 18.33
[2025-04-26 08:54:04,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.76 | bwd: 5748.73 | bwd_inner: 5694.10 | bwd_allreduce: 54.59 | step: 18.33

  5%|▍         | 1971/41250 [4:45:36<94:20:35,  8.65s/it]
                                                         
{'loss': 0.121, 'grad_norm': 0.9060714244842529, 'learning_rate': 3.996688633158485e-05, 'epoch': 0.48}

  5%|▍         | 1971/41250 [4:45:36<94:20:35,  8.65s/it][2025-04-26 08:54:13,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:54:13,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5781.50 | bwd_inner_microstep: 5651.97 | bwd_allreduce_microstep: 129.48 | step_microstep: 18.43
[2025-04-26 08:54:13,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 5781.52 | bwd_inner: 5651.97 | bwd_allreduce: 129.50 | step: 18.43

  5%|▍         | 1972/41250 [4:45:44<94:28:47,  8.66s/it]
                                                         
{'loss': 0.3193, 'grad_norm': 2.540318489074707, 'learning_rate': 3.996679594390593e-05, 'epoch': 0.48}

  5%|▍         | 1972/41250 [4:45:44<94:28:47,  8.66s/it][2025-04-26 08:54:22,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 08:54:22,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.39 | bwd_microstep: 5740.98 | bwd_inner_microstep: 5716.41 | bwd_allreduce_microstep: 24.53 | step_microstep: 18.10
[2025-04-26 08:54:22,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.39 | bwd: 5740.99 | bwd_inner: 5716.41 | bwd_allreduce: 24.55 | step: 18.11

  5%|▍         | 1973/41250 [4:45:53<94:30:29,  8.66s/it]
                                                         
{'loss': 0.2484, 'grad_norm': 1.7011830806732178, 'learning_rate': 3.9966705433135636e-05, 'epoch': 0.48}

  5%|▍         | 1973/41250 [4:45:53<94:30:29,  8.66s/it][2025-04-26 08:54:30,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:54:30,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.38 | bwd_microstep: 5693.20 | bwd_inner_microstep: 5644.96 | bwd_allreduce_microstep: 48.20 | step_microstep: 18.15
[2025-04-26 08:54:30,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.38 | bwd: 5693.21 | bwd_inner: 5644.96 | bwd_allreduce: 48.22 | step: 18.16

  5%|▍         | 1974/41250 [4:46:02<94:20:34,  8.65s/it]
                                                         
{'loss': 0.1304, 'grad_norm': 1.0786221027374268, 'learning_rate': 3.9966614799274536e-05, 'epoch': 0.48}

  5%|▍         | 1974/41250 [4:46:02<94:20:34,  8.65s/it][2025-04-26 08:54:39,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 08:54:39,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.13 | bwd_microstep: 5700.13 | bwd_inner_microstep: 5687.59 | bwd_allreduce_microstep: 12.49 | step_microstep: 18.15
[2025-04-26 08:54:39,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.13 | bwd: 5700.14 | bwd_inner: 5687.59 | bwd_allreduce: 12.51 | step: 18.16

  5%|▍         | 1975/41250 [4:46:10<94:16:49,  8.64s/it]
                                                         
{'loss': 0.2993, 'grad_norm': 1.821455478668213, 'learning_rate': 3.9966524042323184e-05, 'epoch': 0.48}

  5%|▍         | 1975/41250 [4:46:10<94:16:49,  8.64s/it][2025-04-26 08:54:48,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:54:48,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.04 | bwd_microstep: 5758.95 | bwd_inner_microstep: 5681.82 | bwd_allreduce_microstep: 77.08 | step_microstep: 18.30
[2025-04-26 08:54:48,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.04 | bwd: 5758.96 | bwd_inner: 5681.82 | bwd_allreduce: 77.10 | step: 18.31

  5%|▍         | 1976/41250 [4:46:19<94:27:04,  8.66s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 2.5074715614318848, 'learning_rate': 3.9966433162282135e-05, 'epoch': 0.48}

  5%|▍         | 1976/41250 [4:46:19<94:27:04,  8.66s/it][2025-04-26 08:54:57,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:54:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5897.05 | bwd_inner_microstep: 5648.10 | bwd_allreduce_microstep: 248.91 | step_microstep: 18.31
[2025-04-26 08:54:57,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5897.07 | bwd_inner: 5648.10 | bwd_allreduce: 248.93 | step: 18.32

  5%|▍         | 1977/41250 [4:46:28<94:56:46,  8.70s/it]
                                                         
{'loss': 0.2527, 'grad_norm': 1.3314424753189087, 'learning_rate': 3.996634215915195e-05, 'epoch': 0.48}

  5%|▍         | 1977/41250 [4:46:28<94:56:46,  8.70s/it][2025-04-26 08:55:05,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 08:55:05,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5770.51 | bwd_inner_microstep: 5695.68 | bwd_allreduce_microstep: 74.79 | step_microstep: 17.72
[2025-04-26 08:55:05,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5770.52 | bwd_inner: 5695.68 | bwd_allreduce: 74.80 | step: 17.72

  5%|▍         | 1978/41250 [4:46:36<94:55:50,  8.70s/it]
                                                         
{'loss': 0.1476, 'grad_norm': 1.4631062746047974, 'learning_rate': 3.99662510329332e-05, 'epoch': 0.48}

  5%|▍         | 1978/41250 [4:46:36<94:55:50,  8.70s/it][2025-04-26 08:55:14,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:55:14,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.23 | bwd_microstep: 5879.66 | bwd_inner_microstep: 5636.32 | bwd_allreduce_microstep: 243.29 | step_microstep: 17.82
[2025-04-26 08:55:14,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.23 | bwd: 5879.67 | bwd_inner: 5636.32 | bwd_allreduce: 243.31 | step: 17.83

  5%|▍         | 1979/41250 [4:46:45<95:12:42,  8.73s/it]
                                                         
{'loss': 0.1881, 'grad_norm': 1.3068467378616333, 'learning_rate': 3.996615978362644e-05, 'epoch': 0.48}

  5%|▍         | 1979/41250 [4:46:45<95:12:42,  8.73s/it][2025-04-26 08:55:23,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:55:23,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.51 | bwd_microstep: 5712.66 | bwd_inner_microstep: 5656.22 | bwd_allreduce_microstep: 56.40 | step_microstep: 18.11
[2025-04-26 08:55:23,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.51 | bwd: 5712.68 | bwd_inner: 5656.22 | bwd_allreduce: 56.41 | step: 18.12

  5%|▍         | 1980/41250 [4:46:54<94:53:33,  8.70s/it]
                                                         
{'loss': 0.2805, 'grad_norm': 1.8707438707351685, 'learning_rate': 3.996606841123222e-05, 'epoch': 0.48}

  5%|▍         | 1980/41250 [4:46:54<94:53:33,  8.70s/it][2025-04-26 08:55:31,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:55:31,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.57 | bwd_microstep: 5721.32 | bwd_inner_microstep: 5708.78 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.41
[2025-04-26 08:55:31,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.57 | bwd: 5721.34 | bwd_inner: 5708.78 | bwd_allreduce: 12.52 | step: 18.42

  5%|▍         | 1981/41250 [4:47:03<94:48:34,  8.69s/it]
                                                         
{'loss': 0.1712, 'grad_norm': 1.6139991283416748, 'learning_rate': 3.996597691575113e-05, 'epoch': 0.48}

  5%|▍         | 1981/41250 [4:47:03<94:48:34,  8.69s/it][2025-04-26 08:55:40,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 08:55:40,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.76 | bwd_microstep: 5761.33 | bwd_inner_microstep: 5700.50 | bwd_allreduce_microstep: 60.77 | step_microstep: 18.82
[2025-04-26 08:55:40,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.76 | bwd: 5761.34 | bwd_inner: 5700.50 | bwd_allreduce: 60.79 | step: 18.82

  5%|▍         | 1982/41250 [4:47:11<94:46:16,  8.69s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 1.7323874235153198, 'learning_rate': 3.996588529718371e-05, 'epoch': 0.48}

  5%|▍         | 1982/41250 [4:47:11<94:46:16,  8.69s/it][2025-04-26 08:55:49,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:55:49,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.27 | bwd_microstep: 5784.90 | bwd_inner_microstep: 5772.23 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.35
[2025-04-26 08:55:49,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.27 | bwd: 5784.91 | bwd_inner: 5772.23 | bwd_allreduce: 12.63 | step: 18.35

  5%|▍         | 1983/41250 [4:47:20<95:01:11,  8.71s/it]
                                                         
{'loss': 0.0985, 'grad_norm': 0.7768855690956116, 'learning_rate': 3.996579355553054e-05, 'epoch': 0.48}

  5%|▍         | 1983/41250 [4:47:20<95:01:11,  8.71s/it][2025-04-26 08:55:57,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:55:57,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.22 | bwd_microstep: 5727.57 | bwd_inner_microstep: 5663.99 | bwd_allreduce_microstep: 63.54 | step_microstep: 18.65
[2025-04-26 08:55:57,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.22 | bwd: 5727.58 | bwd_inner: 5663.99 | bwd_allreduce: 63.56 | step: 18.65

  5%|▍         | 1984/41250 [4:47:29<94:47:26,  8.69s/it]
                                                         
{'loss': 0.3465, 'grad_norm': 1.4723330736160278, 'learning_rate': 3.996570169079217e-05, 'epoch': 0.48}

  5%|▍         | 1984/41250 [4:47:29<94:47:26,  8.69s/it][2025-04-26 08:56:06,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 08:56:06,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.61 | bwd_microstep: 5788.57 | bwd_inner_microstep: 5653.72 | bwd_allreduce_microstep: 134.80 | step_microstep: 18.64
[2025-04-26 08:56:06,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.61 | bwd: 5788.58 | bwd_inner: 5653.72 | bwd_allreduce: 134.82 | step: 18.64

  5%|▍         | 1985/41250 [4:47:37<94:50:43,  8.70s/it]
                                                         
{'loss': 0.1454, 'grad_norm': 1.05586838722229, 'learning_rate': 3.9965609702969184e-05, 'epoch': 0.48}

  5%|▍         | 1985/41250 [4:47:37<94:50:43,  8.70s/it][2025-04-26 08:56:15,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 08:56:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.43 | bwd_microstep: 5801.21 | bwd_inner_microstep: 5643.29 | bwd_allreduce_microstep: 157.88 | step_microstep: 18.81
[2025-04-26 08:56:15,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.43 | bwd: 5801.23 | bwd_inner: 5643.29 | bwd_allreduce: 157.90 | step: 18.81

  5%|▍         | 1986/41250 [4:47:46<94:54:20,  8.70s/it]
                                                         
{'loss': 0.1357, 'grad_norm': 1.0694254636764526, 'learning_rate': 3.996551759206214e-05, 'epoch': 0.48}

  5%|▍         | 1986/41250 [4:47:46<94:54:20,  8.70s/it][2025-04-26 08:56:24,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:56:24,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5716.87 | bwd_inner_microstep: 5704.23 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.50
[2025-04-26 08:56:24,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5716.89 | bwd_inner: 5704.23 | bwd_allreduce: 12.62 | step: 18.50

  5%|▍         | 1987/41250 [4:47:55<94:43:32,  8.69s/it]
                                                         
{'loss': 0.0764, 'grad_norm': 1.31589937210083, 'learning_rate': 3.996542535807161e-05, 'epoch': 0.48}

  5%|▍         | 1987/41250 [4:47:55<94:43:32,  8.69s/it][2025-04-26 08:56:32,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 08:56:32,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.40 | bwd_microstep: 5694.36 | bwd_inner_microstep: 5645.97 | bwd_allreduce_microstep: 48.34 | step_microstep: 18.73
[2025-04-26 08:56:32,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.40 | bwd: 5694.37 | bwd_inner: 5645.97 | bwd_allreduce: 48.36 | step: 18.74

  5%|▍         | 1988/41250 [4:48:03<94:30:50,  8.67s/it]
                                                         
{'loss': 0.1527, 'grad_norm': 1.2622250318527222, 'learning_rate': 3.9965333000998156e-05, 'epoch': 0.48}

  5%|▍         | 1988/41250 [4:48:03<94:30:50,  8.67s/it][2025-04-26 08:56:41,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 08:56:41,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.32 | bwd_microstep: 5781.70 | bwd_inner_microstep: 5687.19 | bwd_allreduce_microstep: 94.46 | step_microstep: 19.00
[2025-04-26 08:56:41,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.33 | bwd: 5781.72 | bwd_inner: 5687.19 | bwd_allreduce: 94.48 | step: 19.00

  5%|▍         | 1989/41250 [4:48:12<94:41:16,  8.68s/it]
                                                         
{'loss': 0.1721, 'grad_norm': 1.0980371236801147, 'learning_rate': 3.996524052084235e-05, 'epoch': 0.48}

  5%|▍         | 1989/41250 [4:48:12<94:41:16,  8.68s/it][2025-04-26 08:56:50,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 08:56:50,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5725.90 | bwd_inner_microstep: 5713.04 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.04
[2025-04-26 08:56:50,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5725.92 | bwd_inner: 5713.04 | bwd_allreduce: 12.84 | step: 19.04

  5%|▍         | 1990/41250 [4:48:21<94:37:07,  8.68s/it]
                                                         
{'loss': 0.1302, 'grad_norm': 1.5121772289276123, 'learning_rate': 3.996514791760477e-05, 'epoch': 0.48}

  5%|▍         | 1990/41250 [4:48:21<94:37:07,  8.68s/it][2025-04-26 08:56:58,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 08:56:58,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.68 | bwd_microstep: 5789.08 | bwd_inner_microstep: 5658.74 | bwd_allreduce_microstep: 130.30 | step_microstep: 19.07
[2025-04-26 08:56:58,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.68 | bwd: 5789.09 | bwd_inner: 5658.74 | bwd_allreduce: 130.32 | step: 19.07

  5%|▍         | 1991/41250 [4:48:29<94:40:43,  8.68s/it]
                                                         
{'loss': 0.3365, 'grad_norm': 1.6920366287231445, 'learning_rate': 3.996505519128597e-05, 'epoch': 0.48}

  5%|▍         | 1991/41250 [4:48:29<94:40:43,  8.68s/it][2025-04-26 08:57:07,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 08:57:07,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.99 | bwd_microstep: 5793.12 | bwd_inner_microstep: 5780.12 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.10
[2025-04-26 08:57:07,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.99 | bwd: 5793.14 | bwd_inner: 5780.12 | bwd_allreduce: 12.97 | step: 19.10

  5%|▍         | 1992/41250 [4:48:38<94:58:24,  8.71s/it]
                                                         
{'loss': 0.4119, 'grad_norm': 2.2139270305633545, 'learning_rate': 3.996496234188654e-05, 'epoch': 0.48}

  5%|▍         | 1992/41250 [4:48:38<94:58:24,  8.71s/it][2025-04-26 08:57:16,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 08:57:16,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.01 | bwd_microstep: 5772.66 | bwd_inner_microstep: 5644.26 | bwd_allreduce_microstep: 128.36 | step_microstep: 18.70
[2025-04-26 08:57:16,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.01 | bwd: 5772.67 | bwd_inner: 5644.26 | bwd_allreduce: 128.37 | step: 18.71

  5%|▍         | 1993/41250 [4:48:47<94:52:51,  8.70s/it]
                                                         
{'loss': 0.1548, 'grad_norm': 1.3664063215255737, 'learning_rate': 3.996486936940704e-05, 'epoch': 0.48}

  5%|▍         | 1993/41250 [4:48:47<94:52:51,  8.70s/it][2025-04-26 08:57:25,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:57:25,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.69 | bwd_microstep: 5904.94 | bwd_inner_microstep: 5654.55 | bwd_allreduce_microstep: 250.34 | step_microstep: 18.62
[2025-04-26 08:57:25,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.69 | bwd: 5904.95 | bwd_inner: 5654.55 | bwd_allreduce: 250.35 | step: 18.62

  5%|▍         | 1994/41250 [4:48:56<95:15:40,  8.74s/it]
                                                         
{'loss': 0.2124, 'grad_norm': 1.7708392143249512, 'learning_rate': 3.9964776273848044e-05, 'epoch': 0.48}

  5%|▍         | 1994/41250 [4:48:56<95:15:40,  8.74s/it][2025-04-26 08:57:33,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 08:57:33,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.61 | bwd_microstep: 5721.99 | bwd_inner_microstep: 5658.97 | bwd_allreduce_microstep: 62.97 | step_microstep: 18.84
[2025-04-26 08:57:33,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.61 | bwd: 5722.01 | bwd_inner: 5658.97 | bwd_allreduce: 62.99 | step: 18.85

  5%|▍         | 1995/41250 [4:49:04<94:55:19,  8.71s/it]
                                                         
{'loss': 0.1968, 'grad_norm': 1.4187203645706177, 'learning_rate': 3.996468305521014e-05, 'epoch': 0.48}

  5%|▍         | 1995/41250 [4:49:04<94:55:19,  8.71s/it][2025-04-26 08:57:42,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 08:57:42,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.39 | bwd_microstep: 5744.58 | bwd_inner_microstep: 5687.39 | bwd_allreduce_microstep: 57.15 | step_microstep: 18.28
[2025-04-26 08:57:42,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.39 | bwd: 5744.59 | bwd_inner: 5687.39 | bwd_allreduce: 57.16 | step: 18.29

  5%|▍         | 1996/41250 [4:49:13<94:50:28,  8.70s/it]
                                                         
{'loss': 0.109, 'grad_norm': 1.6715363264083862, 'learning_rate': 3.9964589713493877e-05, 'epoch': 0.48}

  5%|▍         | 1996/41250 [4:49:13<94:50:28,  8.70s/it][2025-04-26 08:57:50,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 08:57:50,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.45 | bwd_microstep: 5712.05 | bwd_inner_microstep: 5639.78 | bwd_allreduce_microstep: 72.21 | step_microstep: 18.68
[2025-04-26 08:57:50,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.45 | bwd: 5712.06 | bwd_inner: 5639.78 | bwd_allreduce: 72.23 | step: 18.68

  5%|▍         | 1997/41250 [4:49:22<94:36:33,  8.68s/it]
                                                         
{'loss': 0.2886, 'grad_norm': 1.69284987449646, 'learning_rate': 3.996449624869985e-05, 'epoch': 0.48}

  5%|▍         | 1997/41250 [4:49:22<94:36:33,  8.68s/it][2025-04-26 08:57:59,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 08:57:59,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.83 | bwd_microstep: 5763.37 | bwd_inner_microstep: 5689.48 | bwd_allreduce_microstep: 73.85 | step_microstep: 18.87
[2025-04-26 08:57:59,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.83 | bwd: 5763.38 | bwd_inner: 5689.48 | bwd_allreduce: 73.87 | step: 18.87

  5%|▍         | 1998/41250 [4:49:30<94:39:17,  8.68s/it]
                                                         
{'loss': 0.2163, 'grad_norm': 1.7064322233200073, 'learning_rate': 3.9964402660828636e-05, 'epoch': 0.48}

  5%|▍         | 1998/41250 [4:49:30<94:39:17,  8.68s/it][2025-04-26 08:58:08,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:58:08,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.19 | bwd_microstep: 5756.92 | bwd_inner_microstep: 5680.15 | bwd_allreduce_microstep: 76.72 | step_microstep: 18.83
[2025-04-26 08:58:08,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.19 | bwd: 5756.93 | bwd_inner: 5680.15 | bwd_allreduce: 76.74 | step: 18.84

  5%|▍         | 1999/41250 [4:49:39<94:39:47,  8.68s/it]
                                                         
{'loss': 0.0952, 'grad_norm': 0.8932092189788818, 'learning_rate': 3.996430894988081e-05, 'epoch': 0.48}

  5%|▍         | 1999/41250 [4:49:39<94:39:47,  8.68s/it][2025-04-26 08:58:16,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 08:58:16,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.01 | bwd_microstep: 5771.08 | bwd_inner_microstep: 5655.10 | bwd_allreduce_microstep: 115.93 | step_microstep: 18.84
[2025-04-26 08:58:16,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.01 | bwd: 5771.09 | bwd_inner: 5655.10 | bwd_allreduce: 115.95 | step: 18.84

  5%|▍         | 2000/41250 [4:49:48<94:37:24,  8.68s/it]
                                                         
{'loss': 0.171, 'grad_norm': 3.14416241645813, 'learning_rate': 3.9964215115856936e-05, 'epoch': 0.48}

  5%|▍         | 2000/41250 [4:49:48<94:37:24,  8.68s/it][2025-04-26 08:58:25,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.97
[2025-04-26 08:58:25,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.40 | bwd_microstep: 5755.85 | bwd_inner_microstep: 5685.78 | bwd_allreduce_microstep: 70.03 | step_microstep: 18.81
[2025-04-26 08:58:25,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.40 | bwd: 5755.86 | bwd_inner: 5685.78 | bwd_allreduce: 70.05 | step: 18.82

  5%|▍         | 2001/41250 [4:49:56<94:38:34,  8.68s/it]
                                                         
{'loss': 0.1933, 'grad_norm': 1.4783015251159668, 'learning_rate': 3.9964121158757606e-05, 'epoch': 0.49}

  5%|▍         | 2001/41250 [4:49:56<94:38:34,  8.68s/it][2025-04-26 08:58:34,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:58:34,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.85 | bwd_microstep: 5708.14 | bwd_inner_microstep: 5657.14 | bwd_allreduce_microstep: 50.96 | step_microstep: 18.52
[2025-04-26 08:58:34,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.85 | bwd: 5708.15 | bwd_inner: 5657.14 | bwd_allreduce: 50.97 | step: 18.53

  5%|▍         | 2002/41250 [4:50:05<94:24:42,  8.66s/it]
                                                         
{'loss': 0.188, 'grad_norm': 1.704674243927002, 'learning_rate': 3.996402707858339e-05, 'epoch': 0.49}

  5%|▍         | 2002/41250 [4:50:05<94:24:42,  8.66s/it][2025-04-26 08:58:42,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 08:58:42,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.75 | bwd_microstep: 5772.81 | bwd_inner_microstep: 5640.96 | bwd_allreduce_microstep: 131.81 | step_microstep: 17.77
[2025-04-26 08:58:42,966] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.75 | bwd: 5772.82 | bwd_inner: 5640.96 | bwd_allreduce: 131.82 | step: 17.77

  5%|▍         | 2003/41250 [4:50:14<94:28:03,  8.67s/it]
                                                         
{'loss': 0.1299, 'grad_norm': 0.8494198322296143, 'learning_rate': 3.9963932875334886e-05, 'epoch': 0.49}

  5%|▍         | 2003/41250 [4:50:14<94:28:03,  8.67s/it][2025-04-26 08:58:51,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 08:58:51,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5740.41 | bwd_inner_microstep: 5640.04 | bwd_allreduce_microstep: 100.32 | step_microstep: 18.45
[2025-04-26 08:58:51,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5740.42 | bwd_inner: 5640.04 | bwd_allreduce: 100.34 | step: 18.45

  5%|▍         | 2004/41250 [4:50:22<94:26:15,  8.66s/it]
                                                         
{'loss': 0.3708, 'grad_norm': 3.142864227294922, 'learning_rate': 3.9963838549012654e-05, 'epoch': 0.49}

  5%|▍         | 2004/41250 [4:50:22<94:26:15,  8.66s/it][2025-04-26 08:59:00,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 08:59:00,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.06 | bwd_microstep: 5703.97 | bwd_inner_microstep: 5691.19 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.55
[2025-04-26 08:59:00,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.06 | bwd: 5703.98 | bwd_inner: 5691.19 | bwd_allreduce: 12.75 | step: 18.55

  5%|▍         | 2005/41250 [4:50:31<94:17:45,  8.65s/it]
                                                         
{'loss': 0.2634, 'grad_norm': 3.618258237838745, 'learning_rate': 3.9963744099617294e-05, 'epoch': 0.49}

  5%|▍         | 2005/41250 [4:50:31<94:17:45,  8.65s/it][2025-04-26 08:59:08,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 08:59:08,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.02 | bwd_microstep: 5767.22 | bwd_inner_microstep: 5639.40 | bwd_allreduce_microstep: 127.77 | step_microstep: 18.55
[2025-04-26 08:59:08,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.02 | bwd: 5767.23 | bwd_inner: 5639.40 | bwd_allreduce: 127.78 | step: 18.56

  5%|▍         | 2006/41250 [4:50:40<94:20:58,  8.66s/it]
                                                         
{'loss': 0.3029, 'grad_norm': 1.3767253160476685, 'learning_rate': 3.996364952714936e-05, 'epoch': 0.49}

  5%|▍         | 2006/41250 [4:50:40<94:20:58,  8.66s/it][2025-04-26 08:59:17,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 08:59:17,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.89 | bwd_microstep: 5867.79 | bwd_inner_microstep: 5674.07 | bwd_allreduce_microstep: 193.68 | step_microstep: 18.37
[2025-04-26 08:59:17,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.89 | bwd: 5867.80 | bwd_inner: 5674.07 | bwd_allreduce: 193.69 | step: 18.38

  5%|▍         | 2007/41250 [4:50:48<94:47:29,  8.70s/it]
                                                         
{'loss': 0.1079, 'grad_norm': 2.5218918323516846, 'learning_rate': 3.996355483160947e-05, 'epoch': 0.49}

  5%|▍         | 2007/41250 [4:50:48<94:47:29,  8.70s/it][2025-04-26 08:59:26,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.08
[2025-04-26 08:59:26,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.04 | bwd_microstep: 5697.92 | bwd_inner_microstep: 5650.09 | bwd_allreduce_microstep: 47.78 | step_microstep: 18.49
[2025-04-26 08:59:26,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.04 | bwd: 5697.94 | bwd_inner: 5650.09 | bwd_allreduce: 47.80 | step: 18.49

  5%|▍         | 2008/41250 [4:50:57<94:32:35,  8.67s/it]
                                                         
{'loss': 0.2148, 'grad_norm': 2.774967908859253, 'learning_rate': 3.996346001299818e-05, 'epoch': 0.49}

  5%|▍         | 2008/41250 [4:50:57<94:32:35,  8.67s/it][2025-04-26 08:59:34,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 08:59:34,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.44 | bwd_microstep: 5709.24 | bwd_inner_microstep: 5696.76 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.51
[2025-04-26 08:59:34,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.44 | bwd: 5709.25 | bwd_inner: 5696.76 | bwd_allreduce: 12.45 | step: 18.51

  5%|▍         | 2009/41250 [4:51:06<94:24:47,  8.66s/it]
                                                         
{'loss': 0.1878, 'grad_norm': 1.2026582956314087, 'learning_rate': 3.99633650713161e-05, 'epoch': 0.49}

  5%|▍         | 2009/41250 [4:51:06<94:24:47,  8.66s/it][2025-04-26 08:59:43,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 08:59:43,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.51 | bwd_microstep: 5703.25 | bwd_inner_microstep: 5635.77 | bwd_allreduce_microstep: 67.44 | step_microstep: 18.42
[2025-04-26 08:59:43,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.51 | bwd: 5703.26 | bwd_inner: 5635.77 | bwd_allreduce: 67.45 | step: 18.42

  5%|▍         | 2010/41250 [4:51:14<94:14:19,  8.65s/it]
                                                         
{'loss': 0.1446, 'grad_norm': 1.652138590812683, 'learning_rate': 3.996327000656379e-05, 'epoch': 0.49}

  5%|▍         | 2010/41250 [4:51:14<94:14:19,  8.65s/it][2025-04-26 08:59:52,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 08:59:52,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.79 | bwd_microstep: 5706.42 | bwd_inner_microstep: 5693.62 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.90
[2025-04-26 08:59:52,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.79 | bwd: 5706.43 | bwd_inner: 5693.62 | bwd_allreduce: 12.77 | step: 17.91

  5%|▍         | 2011/41250 [4:51:23<94:11:40,  8.64s/it]
                                                         
{'loss': 0.1365, 'grad_norm': 0.7571613788604736, 'learning_rate': 3.996317481874185e-05, 'epoch': 0.49}

  5%|▍         | 2011/41250 [4:51:23<94:11:40,  8.64s/it][2025-04-26 09:00:00,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:00:00,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.24 | bwd_microstep: 5726.04 | bwd_inner_microstep: 5686.61 | bwd_allreduce_microstep: 39.38 | step_microstep: 17.60
[2025-04-26 09:00:00,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.24 | bwd: 5726.05 | bwd_inner: 5686.61 | bwd_allreduce: 39.40 | step: 17.60

  5%|▍         | 2012/41250 [4:51:32<94:14:30,  8.65s/it]
                                                         
{'loss': 0.2275, 'grad_norm': 1.592483639717102, 'learning_rate': 3.996307950785086e-05, 'epoch': 0.49}

  5%|▍         | 2012/41250 [4:51:32<94:14:30,  8.65s/it][2025-04-26 09:00:09,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:00:09,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.48 | bwd_microstep: 5766.13 | bwd_inner_microstep: 5683.72 | bwd_allreduce_microstep: 82.37 | step_microstep: 18.19
[2025-04-26 09:00:09,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.48 | bwd: 5766.14 | bwd_inner: 5683.72 | bwd_allreduce: 82.39 | step: 18.20

  5%|▍         | 2013/41250 [4:51:40<94:24:39,  8.66s/it]
                                                         
{'loss': 0.2203, 'grad_norm': 2.2225594520568848, 'learning_rate': 3.9962984073891413e-05, 'epoch': 0.49}

  5%|▍         | 2013/41250 [4:51:40<94:24:39,  8.66s/it][2025-04-26 09:00:18,216] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:00:18,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.33 | bwd_microstep: 5735.37 | bwd_inner_microstep: 5697.03 | bwd_allreduce_microstep: 38.29 | step_microstep: 18.12
[2025-04-26 09:00:18,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.33 | bwd: 5735.38 | bwd_inner: 5697.03 | bwd_allreduce: 38.31 | step: 18.13

  5%|▍         | 2014/41250 [4:51:49<94:24:38,  8.66s/it]
                                                         
{'loss': 0.4013, 'grad_norm': 2.466141700744629, 'learning_rate': 3.9962888516864096e-05, 'epoch': 0.49}

  5%|▍         | 2014/41250 [4:51:49<94:24:38,  8.66s/it][2025-04-26 09:00:26,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:00:26,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.39 | bwd_microstep: 5698.22 | bwd_inner_microstep: 5658.25 | bwd_allreduce_microstep: 39.93 | step_microstep: 17.98
[2025-04-26 09:00:26,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.39 | bwd: 5698.23 | bwd_inner: 5658.25 | bwd_allreduce: 39.95 | step: 17.99

  5%|▍         | 2015/41250 [4:51:58<94:14:03,  8.65s/it]
                                                         
{'loss': 0.1538, 'grad_norm': 1.0975210666656494, 'learning_rate': 3.9962792836769504e-05, 'epoch': 0.49}

  5%|▍         | 2015/41250 [4:51:58<94:14:03,  8.65s/it][2025-04-26 09:00:35,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 09:00:35,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.60 | bwd_microstep: 5694.04 | bwd_inner_microstep: 5648.55 | bwd_allreduce_microstep: 45.45 | step_microstep: 18.44
[2025-04-26 09:00:35,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.60 | bwd: 5694.06 | bwd_inner: 5648.55 | bwd_allreduce: 45.47 | step: 18.45

  5%|▍         | 2016/41250 [4:52:06<94:06:05,  8.63s/it]
                                                         
{'loss': 0.3247, 'grad_norm': 2.4850645065307617, 'learning_rate': 3.9962697033608215e-05, 'epoch': 0.49}

  5%|▍         | 2016/41250 [4:52:06<94:06:05,  8.63s/it][2025-04-26 09:00:44,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:00:44,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.05 | bwd_microstep: 5736.53 | bwd_inner_microstep: 5693.17 | bwd_allreduce_microstep: 43.32 | step_microstep: 18.29
[2025-04-26 09:00:44,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.05 | bwd: 5736.54 | bwd_inner: 5693.17 | bwd_allreduce: 43.33 | step: 18.29

  5%|▍         | 2017/41250 [4:52:15<94:13:54,  8.65s/it]
                                                         
{'loss': 0.2016, 'grad_norm': 0.9631589651107788, 'learning_rate': 3.996260110738082e-05, 'epoch': 0.49}

  5%|▍         | 2017/41250 [4:52:15<94:13:54,  8.65s/it][2025-04-26 09:00:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:00:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5778.98 | bwd_inner_microstep: 5646.37 | bwd_allreduce_microstep: 132.57 | step_microstep: 18.45
[2025-04-26 09:00:52,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5779.00 | bwd_inner: 5646.37 | bwd_allreduce: 132.59 | step: 18.46

  5%|▍         | 2018/41250 [4:52:23<94:19:15,  8.66s/it]
                                                         
{'loss': 0.162, 'grad_norm': 2.998462200164795, 'learning_rate': 3.996250505808792e-05, 'epoch': 0.49}

  5%|▍         | 2018/41250 [4:52:23<94:19:15,  8.66s/it][2025-04-26 09:01:01,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.93 | optimizer_step: 0.89
[2025-04-26 09:01:01,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5793.34 | bwd_inner_microstep: 5653.57 | bwd_allreduce_microstep: 139.73 | step_microstep: 17.69
[2025-04-26 09:01:01,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 5793.35 | bwd_inner: 5653.57 | bwd_allreduce: 139.75 | step: 17.70

  5%|▍         | 2019/41250 [4:52:32<94:28:12,  8.67s/it]
                                                         
{'loss': 0.2247, 'grad_norm': 2.101428985595703, 'learning_rate': 3.9962408885730106e-05, 'epoch': 0.49}

  5%|▍         | 2019/41250 [4:52:32<94:28:12,  8.67s/it][2025-04-26 09:01:10,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:01:10,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5773.85 | bwd_inner_microstep: 5650.50 | bwd_allreduce_microstep: 123.30 | step_microstep: 18.29
[2025-04-26 09:01:10,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.59 | bwd: 5773.86 | bwd_inner: 5650.50 | bwd_allreduce: 123.31 | step: 18.30

  5%|▍         | 2020/41250 [4:52:41<94:32:32,  8.68s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 1.0240217447280884, 'learning_rate': 3.996231259030797e-05, 'epoch': 0.49}

  5%|▍         | 2020/41250 [4:52:41<94:32:32,  8.68s/it][2025-04-26 09:01:18,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:01:18,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5692.94 | bwd_inner_microstep: 5680.24 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.73
[2025-04-26 09:01:18,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5692.95 | bwd_inner: 5680.24 | bwd_allreduce: 12.68 | step: 18.73

  5%|▍         | 2021/41250 [4:52:49<94:21:57,  8.66s/it]
                                                         
{'loss': 0.0763, 'grad_norm': 1.2473820447921753, 'learning_rate': 3.996221617182209e-05, 'epoch': 0.49}

  5%|▍         | 2021/41250 [4:52:49<94:21:57,  8.66s/it][2025-04-26 09:01:27,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:01:27,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.68 | bwd_microstep: 5716.96 | bwd_inner_microstep: 5653.59 | bwd_allreduce_microstep: 63.32 | step_microstep: 18.20
[2025-04-26 09:01:27,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.68 | bwd: 5716.97 | bwd_inner: 5653.59 | bwd_allreduce: 63.34 | step: 18.20

  5%|▍         | 2022/41250 [4:52:58<94:15:05,  8.65s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 2.1649248600006104, 'learning_rate': 3.996211963027309e-05, 'epoch': 0.49}

  5%|▍         | 2022/41250 [4:52:58<94:15:05,  8.65s/it][2025-04-26 09:01:36,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:01:36,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.31 | bwd_microstep: 5700.46 | bwd_inner_microstep: 5687.84 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.31
[2025-04-26 09:01:36,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.31 | bwd: 5700.47 | bwd_inner: 5687.84 | bwd_allreduce: 12.59 | step: 18.31

  5%|▍         | 2023/41250 [4:53:07<94:10:49,  8.64s/it]
                                                         
{'loss': 0.2312, 'grad_norm': 1.5122038125991821, 'learning_rate': 3.996202296566153e-05, 'epoch': 0.49}

  5%|▍         | 2023/41250 [4:53:07<94:10:49,  8.64s/it][2025-04-26 09:01:44,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:01:44,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.62 | bwd_microstep: 5712.60 | bwd_inner_microstep: 5652.54 | bwd_allreduce_microstep: 60.02 | step_microstep: 18.21
[2025-04-26 09:01:44,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.62 | bwd: 5712.61 | bwd_inner: 5652.54 | bwd_allreduce: 60.04 | step: 18.22

  5%|▍         | 2024/41250 [4:53:15<94:06:03,  8.64s/it]
                                                         
{'loss': 0.1822, 'grad_norm': 3.1059088706970215, 'learning_rate': 3.996192617798804e-05, 'epoch': 0.49}

  5%|▍         | 2024/41250 [4:53:15<94:06:03,  8.64s/it][2025-04-26 09:01:53,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:01:53,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.20 | bwd_microstep: 5796.82 | bwd_inner_microstep: 5650.52 | bwd_allreduce_microstep: 146.26 | step_microstep: 18.33
[2025-04-26 09:01:53,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.20 | bwd: 5796.84 | bwd_inner: 5650.52 | bwd_allreduce: 146.27 | step: 18.33

  5%|▍         | 2025/41250 [4:53:24<94:20:08,  8.66s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.3631577491760254, 'learning_rate': 3.99618292672532e-05, 'epoch': 0.49}

  5%|▍         | 2025/41250 [4:53:24<94:20:08,  8.66s/it][2025-04-26 09:02:01,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 09:02:01,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.54 | bwd_microstep: 5708.45 | bwd_inner_microstep: 5664.55 | bwd_allreduce_microstep: 43.85 | step_microstep: 18.00
[2025-04-26 09:02:01,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.54 | bwd: 5708.46 | bwd_inner: 5664.55 | bwd_allreduce: 43.87 | step: 18.01

  5%|▍         | 2026/41250 [4:53:33<94:12:13,  8.65s/it]
                                                         
{'loss': 0.05, 'grad_norm': 0.7160720825195312, 'learning_rate': 3.996173223345761e-05, 'epoch': 0.49}

  5%|▍         | 2026/41250 [4:53:33<94:12:13,  8.65s/it][2025-04-26 09:02:10,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:02:10,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.80 | bwd_microstep: 5711.77 | bwd_inner_microstep: 5699.13 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.07
[2025-04-26 09:02:10,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.80 | bwd: 5711.78 | bwd_inner: 5699.13 | bwd_allreduce: 12.61 | step: 18.08

  5%|▍         | 2027/41250 [4:53:41<94:11:07,  8.64s/it]
                                                         
{'loss': 0.0743, 'grad_norm': 0.7608522772789001, 'learning_rate': 3.9961635076601854e-05, 'epoch': 0.49}

  5%|▍         | 2027/41250 [4:53:41<94:11:07,  8.64s/it][2025-04-26 09:02:19,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:02:19,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.84 | bwd_microstep: 5781.82 | bwd_inner_microstep: 5769.10 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.81
[2025-04-26 09:02:19,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.84 | bwd: 5781.83 | bwd_inner: 5769.10 | bwd_allreduce: 12.70 | step: 17.82

  5%|▍         | 2028/41250 [4:53:50<94:30:59,  8.68s/it]
                                                         
{'loss': 0.1345, 'grad_norm': 0.8895780444145203, 'learning_rate': 3.996153779668656e-05, 'epoch': 0.49}

  5%|▍         | 2028/41250 [4:53:50<94:30:59,  8.68s/it][2025-04-26 09:02:28,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:02:28,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.64 | bwd_microstep: 5772.05 | bwd_inner_microstep: 5714.70 | bwd_allreduce_microstep: 57.30 | step_microstep: 18.40
[2025-04-26 09:02:28,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.65 | bwd: 5772.06 | bwd_inner: 5714.70 | bwd_allreduce: 57.32 | step: 18.40

  5%|▍         | 2029/41250 [4:53:59<94:38:27,  8.69s/it]
                                                         
{'loss': 0.1553, 'grad_norm': 1.821493148803711, 'learning_rate': 3.99614403937123e-05, 'epoch': 0.49}

  5%|▍         | 2029/41250 [4:53:59<94:38:27,  8.69s/it][2025-04-26 09:02:36,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 09:02:36,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.23 | bwd_microstep: 5716.74 | bwd_inner_microstep: 5704.08 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.41
[2025-04-26 09:02:36,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.23 | bwd: 5716.75 | bwd_inner: 5704.08 | bwd_allreduce: 12.62 | step: 18.41

  5%|▍         | 2030/41250 [4:54:07<94:34:07,  8.68s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.013396978378296, 'learning_rate': 3.9961342867679695e-05, 'epoch': 0.49}

  5%|▍         | 2030/41250 [4:54:07<94:34:07,  8.68s/it][2025-04-26 09:02:45,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:02:45,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.37 | bwd_microstep: 5703.62 | bwd_inner_microstep: 5654.32 | bwd_allreduce_microstep: 49.25 | step_microstep: 18.72
[2025-04-26 09:02:45,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.37 | bwd: 5703.63 | bwd_inner: 5654.32 | bwd_allreduce: 49.27 | step: 18.72

  5%|▍         | 2031/41250 [4:54:16<94:22:13,  8.66s/it]
                                                         
{'loss': 0.0866, 'grad_norm': 1.3145421743392944, 'learning_rate': 3.996124521858933e-05, 'epoch': 0.49}

  5%|▍         | 2031/41250 [4:54:16<94:22:13,  8.66s/it][2025-04-26 09:02:54,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 09:02:54,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.36 | bwd_microstep: 5722.00 | bwd_inner_microstep: 5639.27 | bwd_allreduce_microstep: 82.69 | step_microstep: 17.80
[2025-04-26 09:02:54,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.36 | bwd: 5722.02 | bwd_inner: 5639.27 | bwd_allreduce: 82.70 | step: 17.81

  5%|▍         | 2032/41250 [4:54:25<94:14:54,  8.65s/it]
                                                         
{'loss': 0.1985, 'grad_norm': 0.9130153656005859, 'learning_rate': 3.9961147446441815e-05, 'epoch': 0.49}

  5%|▍         | 2032/41250 [4:54:25<94:14:54,  8.65s/it][2025-04-26 09:03:02,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:03:02,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.17 | bwd_microstep: 5766.10 | bwd_inner_microstep: 5705.79 | bwd_allreduce_microstep: 60.25 | step_microstep: 18.75
[2025-04-26 09:03:02,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.17 | bwd: 5766.11 | bwd_inner: 5705.79 | bwd_allreduce: 60.27 | step: 18.75

  5%|▍         | 2033/41250 [4:54:33<94:25:48,  8.67s/it]
                                                         
{'loss': 0.1322, 'grad_norm': 1.5196824073791504, 'learning_rate': 3.996104955123776e-05, 'epoch': 0.49}

  5%|▍         | 2033/41250 [4:54:33<94:25:48,  8.67s/it][2025-04-26 09:03:11,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 09:03:11,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.65 | bwd_microstep: 5746.12 | bwd_inner_microstep: 5696.76 | bwd_allreduce_microstep: 49.31 | step_microstep: 17.77
[2025-04-26 09:03:11,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.65 | bwd: 5746.13 | bwd_inner: 5696.76 | bwd_allreduce: 49.33 | step: 17.78

  5%|▍         | 2034/41250 [4:54:42<94:28:35,  8.67s/it]
                                                         
{'loss': 0.2481, 'grad_norm': 2.3576319217681885, 'learning_rate': 3.9960951532977754e-05, 'epoch': 0.49}

  5%|▍         | 2034/41250 [4:54:42<94:28:35,  8.67s/it][2025-04-26 09:03:20,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 09:03:20,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.92 | bwd_microstep: 5721.92 | bwd_inner_microstep: 5709.27 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.51
[2025-04-26 09:03:20,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.92 | bwd: 5721.94 | bwd_inner: 5709.27 | bwd_allreduce: 12.63 | step: 18.51

  5%|▍         | 2035/41250 [4:54:51<94:26:00,  8.67s/it]
                                                         
{'loss': 0.1524, 'grad_norm': 1.7026115655899048, 'learning_rate': 3.9960853391662414e-05, 'epoch': 0.49}

  5%|▍         | 2035/41250 [4:54:51<94:26:00,  8.67s/it][2025-04-26 09:03:28,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:03:28,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.35 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5658.70 | bwd_allreduce_microstep: 54.16 | step_microstep: 18.76
[2025-04-26 09:03:28,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.35 | bwd: 5712.92 | bwd_inner: 5658.70 | bwd_allreduce: 54.18 | step: 18.76

  5%|▍         | 2036/41250 [4:54:59<94:14:51,  8.65s/it]
                                                         
{'loss': 0.265, 'grad_norm': 4.0616559982299805, 'learning_rate': 3.996075512729233e-05, 'epoch': 0.49}

  5%|▍         | 2036/41250 [4:54:59<94:14:51,  8.65s/it][2025-04-26 09:03:37,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:03:37,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.33 | bwd_microstep: 5767.63 | bwd_inner_microstep: 5693.31 | bwd_allreduce_microstep: 74.27 | step_microstep: 17.75
[2025-04-26 09:03:37,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.33 | bwd: 5767.64 | bwd_inner: 5693.31 | bwd_allreduce: 74.29 | step: 17.76

  5%|▍         | 2037/41250 [4:55:08<94:24:01,  8.67s/it]
                                                         
{'loss': 0.2164, 'grad_norm': 3.1342222690582275, 'learning_rate': 3.996065673986812e-05, 'epoch': 0.49}

  5%|▍         | 2037/41250 [4:55:08<94:24:01,  8.67s/it][2025-04-26 09:03:46,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:03:46,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.39 | bwd_microstep: 5731.41 | bwd_inner_microstep: 5643.95 | bwd_allreduce_microstep: 87.43 | step_microstep: 18.04
[2025-04-26 09:03:46,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.39 | bwd: 5731.43 | bwd_inner: 5643.95 | bwd_allreduce: 87.44 | step: 18.05

  5%|▍         | 2038/41250 [4:55:17<94:17:55,  8.66s/it]
                                                         
{'loss': 0.3393, 'grad_norm': 1.8737034797668457, 'learning_rate': 3.996055822939039e-05, 'epoch': 0.49}

  5%|▍         | 2038/41250 [4:55:17<94:17:55,  8.66s/it][2025-04-26 09:03:54,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:03:54,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.66 | bwd_microstep: 5862.59 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 172.33 | step_microstep: 17.64
[2025-04-26 09:03:54,815] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.66 | bwd: 5862.60 | bwd_inner: 5690.21 | bwd_allreduce: 172.35 | step: 17.65

  5%|▍         | 2039/41250 [4:55:25<94:46:00,  8.70s/it]
                                                         
{'loss': 0.134, 'grad_norm': 1.3193403482437134, 'learning_rate': 3.996045959585974e-05, 'epoch': 0.49}

  5%|▍         | 2039/41250 [4:55:26<94:46:00,  8.70s/it][2025-04-26 09:04:03,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:04:03,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.37 | bwd_microstep: 5780.18 | bwd_inner_microstep: 5641.51 | bwd_allreduce_microstep: 138.63 | step_microstep: 18.41
[2025-04-26 09:04:03,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.38 | bwd: 5780.19 | bwd_inner: 5641.51 | bwd_allreduce: 138.64 | step: 18.42

  5%|▍         | 2040/41250 [4:55:34<94:46:06,  8.70s/it]
                                                         
{'loss': 0.2271, 'grad_norm': 2.3750338554382324, 'learning_rate': 3.9960360839276783e-05, 'epoch': 0.49}

  5%|▍         | 2040/41250 [4:55:34<94:46:06,  8.70s/it][2025-04-26 09:04:12,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:04:12,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.53 | bwd_microstep: 5748.55 | bwd_inner_microstep: 5685.05 | bwd_allreduce_microstep: 63.45 | step_microstep: 18.44
[2025-04-26 09:04:12,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5748.57 | bwd_inner: 5685.05 | bwd_allreduce: 63.47 | step: 18.44

  5%|▍         | 2041/41250 [4:55:43<94:41:55,  8.69s/it]
                                                         
{'loss': 0.296, 'grad_norm': 5.451411247253418, 'learning_rate': 3.996026195964213e-05, 'epoch': 0.49}

  5%|▍         | 2041/41250 [4:55:43<94:41:55,  8.69s/it][2025-04-26 09:04:20,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 09:04:20,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.80 | bwd_microstep: 5706.37 | bwd_inner_microstep: 5650.75 | bwd_allreduce_microstep: 55.57 | step_microstep: 18.08
[2025-04-26 09:04:20,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.80 | bwd: 5706.38 | bwd_inner: 5650.75 | bwd_allreduce: 55.59 | step: 18.08

  5%|▍         | 2042/41250 [4:55:51<94:26:17,  8.67s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 2.2552666664123535, 'learning_rate': 3.996016295695639e-05, 'epoch': 0.5}

  5%|▍         | 2042/41250 [4:55:52<94:26:17,  8.67s/it][2025-04-26 09:04:29,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:04:29,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.50 | bwd_microstep: 5747.35 | bwd_inner_microstep: 5673.47 | bwd_allreduce_microstep: 73.84 | step_microstep: 18.06
[2025-04-26 09:04:29,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.50 | bwd: 5747.36 | bwd_inner: 5673.47 | bwd_allreduce: 73.85 | step: 18.06

  5%|▍         | 2043/41250 [4:56:00<94:27:10,  8.67s/it]
                                                         
{'loss': 0.4384, 'grad_norm': 3.3268306255340576, 'learning_rate': 3.9960063831220166e-05, 'epoch': 0.5}

  5%|▍         | 2043/41250 [4:56:00<94:27:10,  8.67s/it][2025-04-26 09:04:38,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 09:04:38,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.19 | bwd_microstep: 5677.72 | bwd_inner_microstep: 5641.56 | bwd_allreduce_microstep: 36.12 | step_microstep: 18.45
[2025-04-26 09:04:38,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.19 | bwd: 5677.73 | bwd_inner: 5641.56 | bwd_allreduce: 36.13 | step: 18.46

  5%|▍         | 2044/41250 [4:56:09<94:12:58,  8.65s/it]
                                                         
{'loss': 0.2834, 'grad_norm': 3.970862627029419, 'learning_rate': 3.9959964582434076e-05, 'epoch': 0.5}

  5%|▍         | 2044/41250 [4:56:09<94:12:58,  8.65s/it][2025-04-26 09:04:46,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 09:04:46,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.56 | bwd_microstep: 5707.10 | bwd_inner_microstep: 5694.39 | bwd_allreduce_microstep: 12.67 | step_microstep: 17.69
[2025-04-26 09:04:46,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.56 | bwd: 5707.11 | bwd_inner: 5694.39 | bwd_allreduce: 12.69 | step: 17.69

  5%|▍         | 2045/41250 [4:56:17<94:09:50,  8.65s/it]
                                                         
{'loss': 0.1775, 'grad_norm': 2.453413724899292, 'learning_rate': 3.995986521059874e-05, 'epoch': 0.5}

  5%|▍         | 2045/41250 [4:56:17<94:09:50,  8.65s/it][2025-04-26 09:04:55,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:04:55,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.16 | bwd_microstep: 5735.01 | bwd_inner_microstep: 5644.38 | bwd_allreduce_microstep: 90.59 | step_microstep: 18.77
[2025-04-26 09:04:55,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.16 | bwd: 5735.02 | bwd_inner: 5644.38 | bwd_allreduce: 90.60 | step: 18.77

  5%|▍         | 2046/41250 [4:56:26<94:11:45,  8.65s/it]
                                                         
{'loss': 0.1588, 'grad_norm': 1.7536784410476685, 'learning_rate': 3.995976571571475e-05, 'epoch': 0.5}

  5%|▍         | 2046/41250 [4:56:26<94:11:45,  8.65s/it][2025-04-26 09:05:04,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:05:04,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.33 | bwd_microstep: 5693.69 | bwd_inner_microstep: 5681.10 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.09
[2025-04-26 09:05:04,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.33 | bwd: 5693.70 | bwd_inner: 5681.10 | bwd_allreduce: 12.57 | step: 18.09

  5%|▍         | 2047/41250 [4:56:35<94:05:31,  8.64s/it]
                                                         
{'loss': 0.1343, 'grad_norm': 1.230097770690918, 'learning_rate': 3.995966609778273e-05, 'epoch': 0.5}

  5%|▍         | 2047/41250 [4:56:35<94:05:31,  8.64s/it][2025-04-26 09:05:12,652] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 09:05:12,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.97 | bwd_microstep: 5720.87 | bwd_inner_microstep: 5675.74 | bwd_allreduce_microstep: 45.08 | step_microstep: 17.76
[2025-04-26 09:05:12,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.97 | bwd: 5720.88 | bwd_inner: 5675.74 | bwd_allreduce: 45.10 | step: 17.76

  5%|▍         | 2048/41250 [4:56:43<94:07:17,  8.64s/it]
                                                         
{'loss': 0.0469, 'grad_norm': 0.6563124656677246, 'learning_rate': 3.9959566356803294e-05, 'epoch': 0.5}

  5%|▍         | 2048/41250 [4:56:43<94:07:17,  8.64s/it][2025-04-26 09:05:21,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:05:21,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.43 | bwd_microstep: 5750.22 | bwd_inner_microstep: 5644.88 | bwd_allreduce_microstep: 105.29 | step_microstep: 18.07
[2025-04-26 09:05:21,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.43 | bwd: 5750.23 | bwd_inner: 5644.88 | bwd_allreduce: 105.31 | step: 18.07

  5%|▍         | 2049/41250 [4:56:52<94:12:15,  8.65s/it]
                                                         
{'loss': 0.0937, 'grad_norm': 16.550559997558594, 'learning_rate': 3.9959466492777066e-05, 'epoch': 0.5}

  5%|▍         | 2049/41250 [4:56:52<94:12:15,  8.65s/it][2025-04-26 09:05:29,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:05:29,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.66 | bwd_microstep: 5688.83 | bwd_inner_microstep: 5640.12 | bwd_allreduce_microstep: 48.67 | step_microstep: 18.56
[2025-04-26 09:05:29,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.66 | bwd: 5688.85 | bwd_inner: 5640.12 | bwd_allreduce: 48.69 | step: 18.56

  5%|▍         | 2050/41250 [4:57:01<94:02:32,  8.64s/it]
                                                         
{'loss': 0.3078, 'grad_norm': 2.9794609546661377, 'learning_rate': 3.9959366505704646e-05, 'epoch': 0.5}

  5%|▍         | 2050/41250 [4:57:01<94:02:32,  8.64s/it][2025-04-26 09:05:38,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:05:38,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.60 | bwd_microstep: 5696.96 | bwd_inner_microstep: 5683.80 | bwd_allreduce_microstep: 13.11 | step_microstep: 18.70
[2025-04-26 09:05:38,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.60 | bwd: 5696.98 | bwd_inner: 5683.80 | bwd_allreduce: 13.14 | step: 18.70

  5%|▍         | 2051/41250 [4:57:09<93:58:51,  8.63s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 1.7872977256774902, 'learning_rate': 3.9959266395586656e-05, 'epoch': 0.5}

  5%|▍         | 2051/41250 [4:57:09<93:58:51,  8.63s/it][2025-04-26 09:05:47,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:05:47,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.54 | bwd_microstep: 5685.81 | bwd_inner_microstep: 5644.22 | bwd_allreduce_microstep: 41.54 | step_microstep: 17.57
[2025-04-26 09:05:47,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.55 | bwd: 5685.82 | bwd_inner: 5644.22 | bwd_allreduce: 41.56 | step: 17.57

  5%|▍         | 2052/41250 [4:57:18<93:53:02,  8.62s/it]
                                                         
{'loss': 0.2793, 'grad_norm': 2.784818172454834, 'learning_rate': 3.995916616242372e-05, 'epoch': 0.5}

  5%|▍         | 2052/41250 [4:57:18<93:53:02,  8.62s/it][2025-04-26 09:05:55,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 1.12
[2025-04-26 09:05:55,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.76 | bwd_microstep: 5747.47 | bwd_inner_microstep: 5680.43 | bwd_allreduce_microstep: 66.99 | step_microstep: 18.61
[2025-04-26 09:05:55,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.76 | bwd: 5747.48 | bwd_inner: 5680.43 | bwd_allreduce: 67.01 | step: 18.61

  5%|▍         | 2053/41250 [4:57:27<94:04:48,  8.64s/it]
                                                         
{'loss': 0.413, 'grad_norm': 3.3316569328308105, 'learning_rate': 3.995906580621645e-05, 'epoch': 0.5}

  5%|▍         | 2053/41250 [4:57:27<94:04:48,  8.64s/it][2025-04-26 09:06:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:06:04,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.69 | bwd_microstep: 5706.37 | bwd_inner_microstep: 5693.77 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.29
[2025-04-26 09:06:04,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.69 | bwd: 5706.38 | bwd_inner: 5693.77 | bwd_allreduce: 12.57 | step: 18.29

  5%|▍         | 2054/41250 [4:57:35<94:01:25,  8.64s/it]
                                                         
{'loss': 0.1357, 'grad_norm': 1.1199041604995728, 'learning_rate': 3.995896532696546e-05, 'epoch': 0.5}

  5%|▍         | 2054/41250 [4:57:35<94:01:25,  8.64s/it][2025-04-26 09:06:13,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:06:13,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.51 | bwd_microstep: 5737.32 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 43.99 | step_microstep: 17.76
[2025-04-26 09:06:13,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.51 | bwd: 5737.33 | bwd_inner: 5693.28 | bwd_allreduce: 44.01 | step: 17.76

  5%|▍         | 2055/41250 [4:57:44<94:11:04,  8.65s/it]
                                                         
{'loss': 0.2545, 'grad_norm': 2.0539233684539795, 'learning_rate': 3.995886472467137e-05, 'epoch': 0.5}

  5%|▍         | 2055/41250 [4:57:44<94:11:04,  8.65s/it][2025-04-26 09:06:21,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:06:21,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.48 | bwd_microstep: 5722.60 | bwd_inner_microstep: 5688.19 | bwd_allreduce_microstep: 34.36 | step_microstep: 18.26
[2025-04-26 09:06:21,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.48 | bwd: 5722.61 | bwd_inner: 5688.20 | bwd_allreduce: 34.37 | step: 18.27

  5%|▍         | 2056/41250 [4:57:52<94:12:01,  8.65s/it]
                                                         
{'loss': 0.3714, 'grad_norm': 3.939385175704956, 'learning_rate': 3.995876399933482e-05, 'epoch': 0.5}

  5%|▍         | 2056/41250 [4:57:52<94:12:01,  8.65s/it][2025-04-26 09:06:30,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:06:30,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.27 | bwd_microstep: 5752.90 | bwd_inner_microstep: 5645.50 | bwd_allreduce_microstep: 107.35 | step_microstep: 18.27
[2025-04-26 09:06:30,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.27 | bwd: 5752.91 | bwd_inner: 5645.50 | bwd_allreduce: 107.37 | step: 18.28

  5%|▍         | 2057/41250 [4:58:01<94:16:15,  8.66s/it]
                                                         
{'loss': 0.3068, 'grad_norm': 3.1850333213806152, 'learning_rate': 3.99586631509564e-05, 'epoch': 0.5}

  5%|▍         | 2057/41250 [4:58:01<94:16:15,  8.66s/it][2025-04-26 09:06:39,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:06:39,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5730.49 | bwd_inner_microstep: 5698.09 | bwd_allreduce_microstep: 32.36 | step_microstep: 18.25
[2025-04-26 09:06:39,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5730.51 | bwd_inner: 5698.09 | bwd_allreduce: 32.37 | step: 18.25

  5%|▍         | 2058/41250 [4:58:10<94:16:55,  8.66s/it]
                                                         
{'loss': 0.0958, 'grad_norm': 1.2074402570724487, 'learning_rate': 3.9958562179536754e-05, 'epoch': 0.5}

  5%|▍         | 2058/41250 [4:58:10<94:16:55,  8.66s/it][2025-04-26 09:06:47,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:06:47,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.04 | bwd_microstep: 5740.62 | bwd_inner_microstep: 5653.39 | bwd_allreduce_microstep: 87.18 | step_microstep: 17.68
[2025-04-26 09:06:47,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.04 | bwd: 5740.63 | bwd_inner: 5653.40 | bwd_allreduce: 87.20 | step: 17.69

  5%|▍         | 2059/41250 [4:58:18<94:15:12,  8.66s/it]
                                                         
{'loss': 0.2712, 'grad_norm': 3.072798252105713, 'learning_rate': 3.9958461085076505e-05, 'epoch': 0.5}

  5%|▍         | 2059/41250 [4:58:18<94:15:12,  8.66s/it][2025-04-26 09:06:56,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:06:56,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.50 | bwd_microstep: 5789.41 | bwd_inner_microstep: 5776.85 | bwd_allreduce_microstep: 12.52 | step_microstep: 17.79
[2025-04-26 09:06:56,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.50 | bwd: 5789.42 | bwd_inner: 5776.85 | bwd_allreduce: 12.53 | step: 17.79

  5%|▍         | 2060/41250 [4:58:27<94:36:25,  8.69s/it]
                                                         
{'loss': 0.3575, 'grad_norm': 1.4279924631118774, 'learning_rate': 3.9958359867576256e-05, 'epoch': 0.5}

  5%|▍         | 2060/41250 [4:58:27<94:36:25,  8.69s/it][2025-04-26 09:07:05,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 09:07:05,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.43 | bwd_microstep: 5688.48 | bwd_inner_microstep: 5638.17 | bwd_allreduce_microstep: 50.26 | step_microstep: 17.75
[2025-04-26 09:07:05,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.43 | bwd: 5688.49 | bwd_inner: 5638.17 | bwd_allreduce: 50.28 | step: 17.76

  5%|▍         | 2061/41250 [4:58:36<94:19:53,  8.67s/it]
                                                         
{'loss': 0.2659, 'grad_norm': 2.2634377479553223, 'learning_rate': 3.995825852703665e-05, 'epoch': 0.5}

  5%|▍         | 2061/41250 [4:58:36<94:19:53,  8.67s/it][2025-04-26 09:07:13,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.89 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 09:07:13,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.47 | bwd_microstep: 5677.87 | bwd_inner_microstep: 5652.68 | bwd_allreduce_microstep: 25.15 | step_microstep: 17.80
[2025-04-26 09:07:13,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.47 | bwd: 5677.88 | bwd_inner: 5652.68 | bwd_allreduce: 25.17 | step: 17.81

  5%|▍         | 2062/41250 [4:58:44<94:04:09,  8.64s/it]
                                                         
{'loss': 0.2114, 'grad_norm': 1.482538104057312, 'learning_rate': 3.9958157063458315e-05, 'epoch': 0.5}

  5%|▍         | 2062/41250 [4:58:44<94:04:09,  8.64s/it][2025-04-26 09:07:22,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 09:07:22,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.65 | bwd_microstep: 5717.98 | bwd_inner_microstep: 5649.78 | bwd_allreduce_microstep: 68.16 | step_microstep: 17.68
[2025-04-26 09:07:22,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.65 | bwd: 5717.99 | bwd_inner: 5649.78 | bwd_allreduce: 68.17 | step: 17.68

  5%|▌         | 2063/41250 [4:58:53<93:59:39,  8.63s/it]
                                                         
{'loss': 0.0752, 'grad_norm': 0.8092815279960632, 'learning_rate': 3.9958055476841856e-05, 'epoch': 0.5}

  5%|▌         | 2063/41250 [4:58:53<93:59:39,  8.63s/it][2025-04-26 09:07:31,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:07:31,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.39 | bwd_microstep: 5732.16 | bwd_inner_microstep: 5699.50 | bwd_allreduce_microstep: 32.62 | step_microstep: 18.33
[2025-04-26 09:07:31,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.39 | bwd: 5732.17 | bwd_inner: 5699.50 | bwd_allreduce: 32.64 | step: 18.34

  5%|▌         | 2064/41250 [4:59:02<94:05:55,  8.64s/it]
                                                         
{'loss': 0.1736, 'grad_norm': 2.9425668716430664, 'learning_rate': 3.995795376718791e-05, 'epoch': 0.5}

  5%|▌         | 2064/41250 [4:59:02<94:05:55,  8.64s/it][2025-04-26 09:07:39,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:07:39,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.88 | bwd_microstep: 5710.52 | bwd_inner_microstep: 5697.65 | bwd_allreduce_microstep: 12.83 | step_microstep: 17.77
[2025-04-26 09:07:39,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.88 | bwd: 5710.53 | bwd_inner: 5697.65 | bwd_allreduce: 12.85 | step: 17.78

  5%|▌         | 2065/41250 [4:59:10<94:05:42,  8.64s/it]
                                                         
{'loss': 0.3398, 'grad_norm': 10.46520709991455, 'learning_rate': 3.995785193449711e-05, 'epoch': 0.5}

  5%|▌         | 2065/41250 [4:59:10<94:05:42,  8.64s/it][2025-04-26 09:07:48,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 09:07:48,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5771.36 | bwd_inner_microstep: 5691.77 | bwd_allreduce_microstep: 79.55 | step_microstep: 18.06
[2025-04-26 09:07:48,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5771.37 | bwd_inner: 5691.77 | bwd_allreduce: 79.56 | step: 18.06

  5%|▌         | 2066/41250 [4:59:19<94:15:43,  8.66s/it]
                                                         
{'loss': 0.2031, 'grad_norm': 1.892411708831787, 'learning_rate': 3.995774997877008e-05, 'epoch': 0.5}

  5%|▌         | 2066/41250 [4:59:19<94:15:43,  8.66s/it][2025-04-26 09:07:56,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 09:07:56,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.54 | bwd_microstep: 5701.89 | bwd_inner_microstep: 5655.93 | bwd_allreduce_microstep: 45.91 | step_microstep: 18.13
[2025-04-26 09:07:56,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.54 | bwd: 5701.90 | bwd_inner: 5655.93 | bwd_allreduce: 45.93 | step: 18.14

  5%|▌         | 2067/41250 [4:59:28<94:07:31,  8.65s/it]
                                                         
{'loss': 0.4163, 'grad_norm': 2.085341215133667, 'learning_rate': 3.995764790000744e-05, 'epoch': 0.5}

  5%|▌         | 2067/41250 [4:59:28<94:07:31,  8.65s/it][2025-04-26 09:08:05,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:08:05,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.88 | bwd_microstep: 5789.33 | bwd_inner_microstep: 5655.74 | bwd_allreduce_microstep: 133.55 | step_microstep: 17.92
[2025-04-26 09:08:05,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.88 | bwd: 5789.35 | bwd_inner: 5655.74 | bwd_allreduce: 133.57 | step: 17.92

  5%|▌         | 2068/41250 [4:59:36<94:15:27,  8.66s/it]
                                                         
{'loss': 0.1256, 'grad_norm': 1.7814109325408936, 'learning_rate': 3.995754569820982e-05, 'epoch': 0.5}

  5%|▌         | 2068/41250 [4:59:36<94:15:27,  8.66s/it][2025-04-26 09:08:14,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:08:14,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.55 | bwd_microstep: 5780.70 | bwd_inner_microstep: 5638.77 | bwd_allreduce_microstep: 141.89 | step_microstep: 17.71
[2025-04-26 09:08:14,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.55 | bwd: 5780.72 | bwd_inner: 5638.77 | bwd_allreduce: 141.91 | step: 17.71

  5%|▌         | 2069/41250 [4:59:45<94:20:55,  8.67s/it]
                                                         
{'loss': 0.2108, 'grad_norm': 1.683600902557373, 'learning_rate': 3.995744337337787e-05, 'epoch': 0.5}

  5%|▌         | 2069/41250 [4:59:45<94:20:55,  8.67s/it][2025-04-26 09:08:23,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:08:23,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.67 | bwd_microstep: 5804.98 | bwd_inner_microstep: 5646.18 | bwd_allreduce_microstep: 158.75 | step_microstep: 18.41
[2025-04-26 09:08:23,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.67 | bwd: 5805.00 | bwd_inner: 5646.18 | bwd_allreduce: 158.77 | step: 18.42

  5%|▌         | 2070/41250 [4:59:54<94:28:12,  8.68s/it]
                                                         
{'loss': 0.1729, 'grad_norm': 1.4786062240600586, 'learning_rate': 3.99573409255122e-05, 'epoch': 0.5}

  5%|▌         | 2070/41250 [4:59:54<94:28:12,  8.68s/it][2025-04-26 09:08:31,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:08:31,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.30 | bwd_microstep: 5783.58 | bwd_inner_microstep: 5646.79 | bwd_allreduce_microstep: 136.75 | step_microstep: 18.37
[2025-04-26 09:08:31,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.30 | bwd: 5783.60 | bwd_inner: 5646.79 | bwd_allreduce: 136.77 | step: 18.37

  5%|▌         | 2071/41250 [5:00:02<94:31:09,  8.68s/it]
                                                         
{'loss': 0.2297, 'grad_norm': 1.458781361579895, 'learning_rate': 3.9957238354613455e-05, 'epoch': 0.5}

  5%|▌         | 2071/41250 [5:00:02<94:31:09,  8.68s/it][2025-04-26 09:08:40,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:08:40,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5716.01 | bwd_allreduce_microstep: 47.34 | step_microstep: 17.91
[2025-04-26 09:08:40,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5763.41 | bwd_inner: 5716.01 | bwd_allreduce: 47.36 | step: 17.92

  5%|▌         | 2072/41250 [5:00:11<94:30:43,  8.68s/it]
                                                         
{'loss': 0.1954, 'grad_norm': 1.9550442695617676, 'learning_rate': 3.9957135660682256e-05, 'epoch': 0.5}

  5%|▌         | 2072/41250 [5:00:11<94:30:43,  8.68s/it][2025-04-26 09:08:49,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.92
[2025-04-26 09:08:49,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.74 | bwd_microstep: 5774.81 | bwd_inner_microstep: 5695.69 | bwd_allreduce_microstep: 79.05 | step_microstep: 19.16
[2025-04-26 09:08:49,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.74 | bwd: 5774.82 | bwd_inner: 5695.69 | bwd_allreduce: 79.08 | step: 19.16

  5%|▌         | 2073/41250 [5:00:20<94:34:45,  8.69s/it]
                                                         
{'loss': 0.3059, 'grad_norm': 1.9664627313613892, 'learning_rate': 3.995703284371924e-05, 'epoch': 0.5}

  5%|▌         | 2073/41250 [5:00:20<94:34:45,  8.69s/it][2025-04-26 09:08:57,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.33 | optimizer_step: 1.02
[2025-04-26 09:08:57,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.63 | bwd_microstep: 5773.21 | bwd_inner_microstep: 5706.31 | bwd_allreduce_microstep: 66.84 | step_microstep: 19.73
[2025-04-26 09:08:57,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.63 | bwd: 5773.22 | bwd_inner: 5706.31 | bwd_allreduce: 66.86 | step: 19.73

  5%|▌         | 2074/41250 [5:00:29<94:38:00,  8.70s/it]
                                                         
{'loss': 0.1329, 'grad_norm': 1.1781948804855347, 'learning_rate': 3.995692990372504e-05, 'epoch': 0.5}

  5%|▌         | 2074/41250 [5:00:29<94:38:00,  8.70s/it][2025-04-26 09:09:06,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 09:09:06,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.96 | bwd_microstep: 5797.63 | bwd_inner_microstep: 5658.33 | bwd_allreduce_microstep: 139.24 | step_microstep: 18.29
[2025-04-26 09:09:06,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.96 | bwd: 5797.64 | bwd_inner: 5658.33 | bwd_allreduce: 139.27 | step: 18.30

  5%|▌         | 2075/41250 [5:00:37<94:40:27,  8.70s/it]
                                                         
{'loss': 0.2102, 'grad_norm': 2.2990882396698, 'learning_rate': 3.99568268407003e-05, 'epoch': 0.5}

  5%|▌         | 2075/41250 [5:00:37<94:40:27,  8.70s/it][2025-04-26 09:09:15,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 09:09:15,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.20 | bwd_microstep: 5698.43 | bwd_inner_microstep: 5658.45 | bwd_allreduce_microstep: 39.94 | step_microstep: 18.88
[2025-04-26 09:09:15,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.20 | bwd: 5698.45 | bwd_inner: 5658.45 | bwd_allreduce: 39.96 | step: 18.88

  5%|▌         | 2076/41250 [5:00:46<94:24:58,  8.68s/it]
                                                         
{'loss': 0.2043, 'grad_norm': 1.2694361209869385, 'learning_rate': 3.995672365464564e-05, 'epoch': 0.5}

  5%|▌         | 2076/41250 [5:00:46<94:24:58,  8.68s/it][2025-04-26 09:09:23,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 09:09:23,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5768.47 | bwd_inner_microstep: 5683.80 | bwd_allreduce_microstep: 84.63 | step_microstep: 18.58
[2025-04-26 09:09:23,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5768.48 | bwd_inner: 5683.80 | bwd_allreduce: 84.64 | step: 18.59

  5%|▌         | 2077/41250 [5:00:55<94:27:59,  8.68s/it]
                                                         
{'loss': 0.1509, 'grad_norm': 0.8745551109313965, 'learning_rate': 3.995662034556171e-05, 'epoch': 0.5}

  5%|▌         | 2077/41250 [5:00:55<94:27:59,  8.68s/it][2025-04-26 09:09:32,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.31 | optimizer_step: 1.04
[2025-04-26 09:09:32,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.17 | bwd_microstep: 5792.01 | bwd_inner_microstep: 5705.74 | bwd_allreduce_microstep: 86.20 | step_microstep: 19.69
[2025-04-26 09:09:32,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.17 | bwd: 5792.02 | bwd_inner: 5705.74 | bwd_allreduce: 86.23 | step: 19.69

  5%|▌         | 2078/41250 [5:01:03<94:36:24,  8.69s/it]
                                                         
{'loss': 0.1669, 'grad_norm': 1.5408246517181396, 'learning_rate': 3.995651691344914e-05, 'epoch': 0.5}

  5%|▌         | 2078/41250 [5:01:03<94:36:24,  8.69s/it][2025-04-26 09:09:41,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 09:09:41,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.40 | bwd_microstep: 5725.62 | bwd_inner_microstep: 5712.81 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.01
[2025-04-26 09:09:41,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.40 | bwd: 5725.64 | bwd_inner: 5712.81 | bwd_allreduce: 12.78 | step: 19.01

  5%|▌         | 2079/41250 [5:01:12<94:30:32,  8.69s/it]
                                                         
{'loss': 0.285, 'grad_norm': 1.873741865158081, 'learning_rate': 3.995641335830857e-05, 'epoch': 0.5}

  5%|▌         | 2079/41250 [5:01:12<94:30:32,  8.69s/it][2025-04-26 09:09:49,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:09:49,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.61 | bwd_microstep: 5714.99 | bwd_inner_microstep: 5680.15 | bwd_allreduce_microstep: 34.79 | step_microstep: 18.23
[2025-04-26 09:09:49,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.61 | bwd: 5715.00 | bwd_inner: 5680.15 | bwd_allreduce: 34.81 | step: 18.23

  5%|▌         | 2080/41250 [5:01:21<94:22:04,  8.67s/it]
                                                         
{'loss': 0.1865, 'grad_norm': 1.9048806428909302, 'learning_rate': 3.995630968014063e-05, 'epoch': 0.5}

  5%|▌         | 2080/41250 [5:01:21<94:22:04,  8.67s/it][2025-04-26 09:09:58,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:09:58,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.16 | bwd_microstep: 5756.05 | bwd_inner_microstep: 5683.82 | bwd_allreduce_microstep: 72.18 | step_microstep: 18.27
[2025-04-26 09:09:58,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.16 | bwd: 5756.06 | bwd_inner: 5683.82 | bwd_allreduce: 72.20 | step: 18.28

  5%|▌         | 2081/41250 [5:01:29<94:21:00,  8.67s/it]
                                                         
{'loss': 0.2515, 'grad_norm': 1.9962053298950195, 'learning_rate': 3.995620587894597e-05, 'epoch': 0.5}

  5%|▌         | 2081/41250 [5:01:29<94:21:00,  8.67s/it][2025-04-26 09:10:07,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:10:07,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.45 | bwd_microstep: 5689.11 | bwd_inner_microstep: 5676.36 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.46
[2025-04-26 09:10:07,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.45 | bwd: 5689.12 | bwd_inner: 5676.36 | bwd_allreduce: 12.72 | step: 18.46

  5%|▌         | 2082/41250 [5:01:38<94:08:50,  8.65s/it]
                                                         
{'loss': 0.1925, 'grad_norm': 1.5135833024978638, 'learning_rate': 3.995610195472522e-05, 'epoch': 0.5}

  5%|▌         | 2082/41250 [5:01:38<94:08:50,  8.65s/it][2025-04-26 09:10:15,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:10:15,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.56 | bwd_microstep: 5693.03 | bwd_inner_microstep: 5680.71 | bwd_allreduce_microstep: 12.27 | step_microstep: 18.48
[2025-04-26 09:10:15,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.55 | bwd: 5693.04 | bwd_inner: 5680.71 | bwd_allreduce: 12.29 | step: 18.49

  5%|▌         | 2083/41250 [5:01:47<94:05:06,  8.65s/it]
                                                         
{'loss': 0.1843, 'grad_norm': 1.0431023836135864, 'learning_rate': 3.9955997907479035e-05, 'epoch': 0.5}

  5%|▌         | 2083/41250 [5:01:47<94:05:06,  8.65s/it][2025-04-26 09:10:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:10:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.06 | bwd_microstep: 5701.23 | bwd_inner_microstep: 5688.59 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.29
[2025-04-26 09:10:24,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.06 | bwd: 5701.24 | bwd_inner: 5688.59 | bwd_allreduce: 12.61 | step: 18.30

  5%|▌         | 2084/41250 [5:01:55<94:01:54,  8.64s/it]
                                                         
{'loss': 0.1057, 'grad_norm': 1.5062333345413208, 'learning_rate': 3.995589373720805e-05, 'epoch': 0.51}

  5%|▌         | 2084/41250 [5:01:55<94:01:54,  8.64s/it][2025-04-26 09:10:33,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 09:10:33,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.96 | bwd_microstep: 5766.02 | bwd_inner_microstep: 5638.93 | bwd_allreduce_microstep: 127.04 | step_microstep: 18.59
[2025-04-26 09:10:33,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.96 | bwd: 5766.03 | bwd_inner: 5638.93 | bwd_allreduce: 127.06 | step: 18.59

  5%|▌         | 2085/41250 [5:02:04<94:08:48,  8.65s/it]
                                                         
{'loss': 0.1432, 'grad_norm': 1.6903433799743652, 'learning_rate': 3.99557894439129e-05, 'epoch': 0.51}

  5%|▌         | 2085/41250 [5:02:04<94:08:48,  8.65s/it][2025-04-26 09:10:41,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:10:41,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.04 | bwd_microstep: 5702.04 | bwd_inner_microstep: 5638.19 | bwd_allreduce_microstep: 63.81 | step_microstep: 17.83
[2025-04-26 09:10:41,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.04 | bwd: 5702.05 | bwd_inner: 5638.19 | bwd_allreduce: 63.82 | step: 17.83

  5%|▌         | 2086/41250 [5:02:12<93:58:17,  8.64s/it]
                                                         
{'loss': 0.1188, 'grad_norm': 0.9904590845108032, 'learning_rate': 3.995568502759423e-05, 'epoch': 0.51}

  5%|▌         | 2086/41250 [5:02:12<93:58:17,  8.64s/it][2025-04-26 09:10:50,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:10:50,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.74 | bwd_microstep: 5756.76 | bwd_inner_microstep: 5695.84 | bwd_allreduce_microstep: 60.88 | step_microstep: 18.64
[2025-04-26 09:10:50,437] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.74 | bwd: 5756.78 | bwd_inner: 5695.83 | bwd_allreduce: 60.89 | step: 18.64

  5%|▌         | 2087/41250 [5:02:21<94:07:16,  8.65s/it]
                                                         
{'loss': 0.0788, 'grad_norm': 0.8484229445457458, 'learning_rate': 3.9955580488252694e-05, 'epoch': 0.51}

  5%|▌         | 2087/41250 [5:02:21<94:07:16,  8.65s/it][2025-04-26 09:10:59,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:10:59,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5775.93 | bwd_inner_microstep: 5651.51 | bwd_allreduce_microstep: 124.36 | step_microstep: 18.59
[2025-04-26 09:10:59,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5775.94 | bwd_inner: 5651.51 | bwd_allreduce: 124.38 | step: 18.59

  5%|▌         | 2088/41250 [5:02:30<94:14:30,  8.66s/it]
                                                         
{'loss': 0.2503, 'grad_norm': 1.8414313793182373, 'learning_rate': 3.9955475825888923e-05, 'epoch': 0.51}

  5%|▌         | 2088/41250 [5:02:30<94:14:30,  8.66s/it][2025-04-26 09:11:07,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 09:11:07,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.36 | bwd_microstep: 5756.98 | bwd_inner_microstep: 5640.62 | bwd_allreduce_microstep: 116.31 | step_microstep: 18.63
[2025-04-26 09:11:07,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.36 | bwd: 5756.99 | bwd_inner: 5640.62 | bwd_allreduce: 116.33 | step: 18.64

  5%|▌         | 2089/41250 [5:02:38<94:13:51,  8.66s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 0.7614840269088745, 'learning_rate': 3.995537104050357e-05, 'epoch': 0.51}

  5%|▌         | 2089/41250 [5:02:38<94:13:51,  8.66s/it][2025-04-26 09:11:16,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 09:11:16,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.37 | bwd_microstep: 5775.34 | bwd_inner_microstep: 5762.62 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.73
[2025-04-26 09:11:16,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.37 | bwd: 5775.35 | bwd_inner: 5762.62 | bwd_allreduce: 12.69 | step: 18.73

  5%|▌         | 2090/41250 [5:02:47<94:25:57,  8.68s/it]
                                                         
{'loss': 0.3372, 'grad_norm': 2.5465104579925537, 'learning_rate': 3.995526613209729e-05, 'epoch': 0.51}

  5%|▌         | 2090/41250 [5:02:47<94:25:57,  8.68s/it][2025-04-26 09:11:25,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-26 09:11:25,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.38 | bwd_microstep: 5784.58 | bwd_inner_microstep: 5771.84 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.13
[2025-04-26 09:11:25,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.38 | bwd: 5784.59 | bwd_inner: 5771.84 | bwd_allreduce: 12.71 | step: 19.13

  5%|▌         | 2091/41250 [5:02:56<94:39:36,  8.70s/it]
                                                         
{'loss': 0.1707, 'grad_norm': 5.183619022369385, 'learning_rate': 3.9955161100670706e-05, 'epoch': 0.51}

  5%|▌         | 2091/41250 [5:02:56<94:39:36,  8.70s/it][2025-04-26 09:11:33,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 09:11:33,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.40 | bwd_microstep: 5774.82 | bwd_inner_microstep: 5641.63 | bwd_allreduce_microstep: 133.15 | step_microstep: 18.84
[2025-04-26 09:11:33,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.40 | bwd: 5774.84 | bwd_inner: 5641.63 | bwd_allreduce: 133.17 | step: 18.85

  5%|▌         | 2092/41250 [5:03:05<94:35:18,  8.70s/it]
                                                         
{'loss': 0.1478, 'grad_norm': 1.2246205806732178, 'learning_rate': 3.995505594622448e-05, 'epoch': 0.51}

  5%|▌         | 2092/41250 [5:03:05<94:35:18,  8.70s/it][2025-04-26 09:11:42,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:11:42,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.33 | bwd_microstep: 5779.64 | bwd_inner_microstep: 5647.73 | bwd_allreduce_microstep: 131.87 | step_microstep: 18.94
[2025-04-26 09:11:42,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.33 | bwd: 5779.66 | bwd_inner: 5647.73 | bwd_allreduce: 131.89 | step: 18.94

  5%|▌         | 2093/41250 [5:03:13<94:32:21,  8.69s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 2.43668270111084, 'learning_rate': 3.995495066875927e-05, 'epoch': 0.51}

  5%|▌         | 2093/41250 [5:03:13<94:32:21,  8.69s/it][2025-04-26 09:11:51,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:11:51,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.50 | bwd_microstep: 5759.61 | bwd_inner_microstep: 5638.99 | bwd_allreduce_microstep: 120.58 | step_microstep: 18.75
[2025-04-26 09:11:51,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.50 | bwd: 5759.63 | bwd_inner: 5638.99 | bwd_allreduce: 120.60 | step: 18.75

  5%|▌         | 2094/41250 [5:03:22<94:26:23,  8.68s/it]
                                                         
{'loss': 0.2147, 'grad_norm': 2.441793918609619, 'learning_rate': 3.99548452682757e-05, 'epoch': 0.51}

  5%|▌         | 2094/41250 [5:03:22<94:26:23,  8.68s/it][2025-04-26 09:11:59,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.07 | optimizer_step: 0.89
[2025-04-26 09:11:59,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.76 | bwd_microstep: 5733.91 | bwd_inner_microstep: 5700.93 | bwd_allreduce_microstep: 32.93 | step_microstep: 19.09
[2025-04-26 09:11:59,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.76 | bwd: 5733.93 | bwd_inner: 5700.93 | bwd_allreduce: 32.96 | step: 19.10

  5%|▌         | 2095/41250 [5:03:31<94:23:44,  8.68s/it]
                                                         
{'loss': 0.3085, 'grad_norm': 2.0307841300964355, 'learning_rate': 3.995473974477443e-05, 'epoch': 0.51}

  5%|▌         | 2095/41250 [5:03:31<94:23:44,  8.68s/it][2025-04-26 09:12:08,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 1.08 | optimizer_step: 0.99
[2025-04-26 09:12:08,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.15 | bwd_microstep: 5699.55 | bwd_inner_microstep: 5686.64 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.59
[2025-04-26 09:12:08,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.15 | bwd: 5699.56 | bwd_inner: 5686.64 | bwd_allreduce: 12.88 | step: 19.59

  5%|▌         | 2096/41250 [5:03:39<94:14:20,  8.66s/it]
                                                         
{'loss': 0.1302, 'grad_norm': 2.1577556133270264, 'learning_rate': 3.995463409825613e-05, 'epoch': 0.51}

  5%|▌         | 2096/41250 [5:03:39<94:14:20,  8.66s/it][2025-04-26 09:12:17,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 09:12:17,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.98 | bwd_microstep: 5707.41 | bwd_inner_microstep: 5637.57 | bwd_allreduce_microstep: 69.79 | step_microstep: 19.11
[2025-04-26 09:12:17,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.98 | bwd: 5707.42 | bwd_inner: 5637.57 | bwd_allreduce: 69.81 | step: 19.12

  5%|▌         | 2097/41250 [5:03:48<94:04:20,  8.65s/it]
                                                         
{'loss': 0.2653, 'grad_norm': 2.181724786758423, 'learning_rate': 3.9954528328721434e-05, 'epoch': 0.51}

  5%|▌         | 2097/41250 [5:03:48<94:04:20,  8.65s/it][2025-04-26 09:12:25,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:12:25,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.23 | bwd_microstep: 5695.78 | bwd_inner_microstep: 5682.78 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.69
[2025-04-26 09:12:25,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.23 | bwd: 5695.80 | bwd_inner: 5682.78 | bwd_allreduce: 12.98 | step: 18.69

  5%|▌         | 2098/41250 [5:03:57<93:57:11,  8.64s/it]
                                                         
{'loss': 0.0896, 'grad_norm': 1.0975264310836792, 'learning_rate': 3.995442243617099e-05, 'epoch': 0.51}

  5%|▌         | 2098/41250 [5:03:57<93:57:11,  8.64s/it][2025-04-26 09:12:34,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 09:12:34,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.84 | bwd_microstep: 5698.37 | bwd_inner_microstep: 5685.35 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.81
[2025-04-26 09:12:34,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.84 | bwd: 5698.39 | bwd_inner: 5685.35 | bwd_allreduce: 12.99 | step: 18.81

  5%|▌         | 2099/41250 [5:04:05<93:53:12,  8.63s/it]
                                                         
{'loss': 0.1545, 'grad_norm': 1.3823459148406982, 'learning_rate': 3.995431642060546e-05, 'epoch': 0.51}

  5%|▌         | 2099/41250 [5:04:05<93:53:12,  8.63s/it][2025-04-26 09:12:43,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-26 09:12:43,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.19 | bwd_microstep: 5778.09 | bwd_inner_microstep: 5683.23 | bwd_allreduce_microstep: 94.80 | step_microstep: 19.24
[2025-04-26 09:12:43,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.19 | bwd: 5778.10 | bwd_inner: 5683.23 | bwd_allreduce: 94.82 | step: 19.25

  5%|▌         | 2100/41250 [5:04:14<94:06:33,  8.65s/it]
                                                         
{'loss': 0.2848, 'grad_norm': 2.9471309185028076, 'learning_rate': 3.995421028202549e-05, 'epoch': 0.51}

  5%|▌         | 2100/41250 [5:04:14<94:06:33,  8.65s/it][2025-04-26 09:12:51,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 09:12:51,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.72 | bwd_microstep: 5710.89 | bwd_inner_microstep: 5655.11 | bwd_allreduce_microstep: 55.74 | step_microstep: 19.38
[2025-04-26 09:12:51,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.72 | bwd: 5710.91 | bwd_inner: 5655.11 | bwd_allreduce: 55.76 | step: 19.38

  5%|▌         | 2101/41250 [5:04:22<93:59:12,  8.64s/it]
                                                         
{'loss': 0.189, 'grad_norm': 2.634870767593384, 'learning_rate': 3.995410402043174e-05, 'epoch': 0.51}

  5%|▌         | 2101/41250 [5:04:22<93:59:12,  8.64s/it][2025-04-26 09:13:00,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:13:00,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.94 | bwd_microstep: 5739.02 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 41.06 | step_microstep: 18.95
[2025-04-26 09:13:00,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.94 | bwd: 5739.04 | bwd_inner: 5697.92 | bwd_allreduce: 41.08 | step: 18.96

  5%|▌         | 2102/41250 [5:04:31<94:05:22,  8.65s/it]
                                                         
{'loss': 0.0968, 'grad_norm': 1.349354863166809, 'learning_rate': 3.9953997635824864e-05, 'epoch': 0.51}

  5%|▌         | 2102/41250 [5:04:31<94:05:22,  8.65s/it][2025-04-26 09:13:09,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:13:09,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.11 | bwd_microstep: 5719.19 | bwd_inner_microstep: 5706.44 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.84
[2025-04-26 09:13:09,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.11 | bwd: 5719.21 | bwd_inner: 5706.44 | bwd_allreduce: 12.73 | step: 18.84

  5%|▌         | 2103/41250 [5:04:40<94:04:43,  8.65s/it]
                                                         
{'loss': 0.238, 'grad_norm': 2.405546188354492, 'learning_rate': 3.995389112820552e-05, 'epoch': 0.51}

  5%|▌         | 2103/41250 [5:04:40<94:04:43,  8.65s/it][2025-04-26 09:13:17,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.11 | optimizer_step: 0.99
[2025-04-26 09:13:17,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.41 | bwd_microstep: 5703.88 | bwd_inner_microstep: 5649.56 | bwd_allreduce_microstep: 54.26 | step_microstep: 19.49
[2025-04-26 09:13:17,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.41 | bwd: 5703.89 | bwd_inner: 5649.56 | bwd_allreduce: 54.29 | step: 19.50

  5%|▌         | 2104/41250 [5:04:48<93:57:13,  8.64s/it]
                                                         
{'loss': 0.3216, 'grad_norm': 1.9106911420822144, 'learning_rate': 3.995378449757436e-05, 'epoch': 0.51}

  5%|▌         | 2104/41250 [5:04:48<93:57:13,  8.64s/it][2025-04-26 09:13:26,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:13:26,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.73 | bwd_microstep: 6021.62 | bwd_inner_microstep: 5691.54 | bwd_allreduce_microstep: 330.04 | step_microstep: 18.42
[2025-04-26 09:13:26,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.73 | bwd: 6021.64 | bwd_inner: 5691.54 | bwd_allreduce: 330.06 | step: 18.42

  5%|▌         | 2105/41250 [5:04:57<94:57:26,  8.73s/it]
                                                         
{'loss': 0.0972, 'grad_norm': 1.2385579347610474, 'learning_rate': 3.995367774393204e-05, 'epoch': 0.51}

  5%|▌         | 2105/41250 [5:04:57<94:57:26,  8.73s/it][2025-04-26 09:13:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:13:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.43 | bwd_microstep: 5702.75 | bwd_inner_microstep: 5690.05 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.21
[2025-04-26 09:13:35,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.43 | bwd: 5702.76 | bwd_inner: 5690.05 | bwd_allreduce: 12.67 | step: 18.21

  5%|▌         | 2106/41250 [5:05:06<94:36:57,  8.70s/it]
                                                         
{'loss': 0.2018, 'grad_norm': 1.3252853155136108, 'learning_rate': 3.9953570867279225e-05, 'epoch': 0.51}

  5%|▌         | 2106/41250 [5:05:06<94:36:57,  8.70s/it][2025-04-26 09:13:43,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 09:13:43,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.91 | bwd_microstep: 5785.74 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 139.23 | step_microstep: 18.32
[2025-04-26 09:13:43,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.91 | bwd: 5785.75 | bwd_inner: 5646.46 | bwd_allreduce: 139.24 | step: 18.32

  5%|▌         | 2107/41250 [5:05:15<94:34:28,  8.70s/it]
                                                         
{'loss': 0.0549, 'grad_norm': 0.5488814115524292, 'learning_rate': 3.995346386761657e-05, 'epoch': 0.51}

  5%|▌         | 2107/41250 [5:05:15<94:34:28,  8.70s/it][2025-04-26 09:13:52,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:13:52,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.41 | bwd_microstep: 5764.06 | bwd_inner_microstep: 5695.91 | bwd_allreduce_microstep: 68.10 | step_microstep: 18.60
[2025-04-26 09:13:52,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5764.07 | bwd_inner: 5695.91 | bwd_allreduce: 68.12 | step: 18.60

  5%|▌         | 2108/41250 [5:05:23<94:30:31,  8.69s/it]
                                                         
{'loss': 0.1429, 'grad_norm': 1.234195590019226, 'learning_rate': 3.995335674494473e-05, 'epoch': 0.51}

  5%|▌         | 2108/41250 [5:05:23<94:30:31,  8.69s/it][2025-04-26 09:14:01,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:14:01,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.03 | bwd_microstep: 5711.69 | bwd_inner_microstep: 5649.60 | bwd_allreduce_microstep: 62.05 | step_microstep: 18.79
[2025-04-26 09:14:01,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.03 | bwd: 5711.71 | bwd_inner: 5649.60 | bwd_allreduce: 62.06 | step: 18.79

  5%|▌         | 2109/41250 [5:05:32<94:16:53,  8.67s/it]
                                                         
{'loss': 0.1621, 'grad_norm': 1.675627589225769, 'learning_rate': 3.995324949926438e-05, 'epoch': 0.51}

  5%|▌         | 2109/41250 [5:05:32<94:16:53,  8.67s/it][2025-04-26 09:14:09,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:14:09,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.64 | bwd_microstep: 5772.00 | bwd_inner_microstep: 5649.30 | bwd_allreduce_microstep: 122.66 | step_microstep: 18.50
[2025-04-26 09:14:09,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.64 | bwd: 5772.02 | bwd_inner: 5649.30 | bwd_allreduce: 122.68 | step: 18.51

  5%|▌         | 2110/41250 [5:05:41<94:18:15,  8.67s/it]
                                                         
{'loss': 0.2016, 'grad_norm': 1.9162240028381348, 'learning_rate': 3.995314213057616e-05, 'epoch': 0.51}

  5%|▌         | 2110/41250 [5:05:41<94:18:15,  8.67s/it][2025-04-26 09:14:18,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:14:18,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.65 | bwd_microstep: 5808.31 | bwd_inner_microstep: 5639.98 | bwd_allreduce_microstep: 168.28 | step_microstep: 18.69
[2025-04-26 09:14:18,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.65 | bwd: 5808.32 | bwd_inner: 5639.98 | bwd_allreduce: 168.30 | step: 18.69

  5%|▌         | 2111/41250 [5:05:49<94:26:01,  8.69s/it]
                                                         
{'loss': 0.2325, 'grad_norm': 1.5877349376678467, 'learning_rate': 3.9953034638880755e-05, 'epoch': 0.51}

  5%|▌         | 2111/41250 [5:05:49<94:26:01,  8.69s/it][2025-04-26 09:14:27,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.20 | optimizer_step: 0.99
[2025-04-26 09:14:27,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.03 | bwd_microstep: 5771.36 | bwd_inner_microstep: 5704.40 | bwd_allreduce_microstep: 66.91 | step_microstep: 19.38
[2025-04-26 09:14:27,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.03 | bwd: 5771.38 | bwd_inner: 5704.40 | bwd_allreduce: 66.93 | step: 19.38

  5%|▌         | 2112/41250 [5:05:58<94:29:20,  8.69s/it]
                                                         
{'loss': 0.1592, 'grad_norm': 2.542217969894409, 'learning_rate': 3.995292702417881e-05, 'epoch': 0.51}

  5%|▌         | 2112/41250 [5:05:58<94:29:20,  8.69s/it][2025-04-26 09:14:35,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.08
[2025-04-26 09:14:35,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.72 | bwd_microstep: 5705.65 | bwd_inner_microstep: 5692.47 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.93
[2025-04-26 09:14:35,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.72 | bwd: 5705.67 | bwd_inner: 5692.47 | bwd_allreduce: 13.15 | step: 18.93

  5%|▌         | 2113/41250 [5:06:07<94:17:02,  8.67s/it]
                                                         
{'loss': 0.3211, 'grad_norm': 2.089900493621826, 'learning_rate': 3.9952819286470995e-05, 'epoch': 0.51}

  5%|▌         | 2113/41250 [5:06:07<94:17:02,  8.67s/it][2025-04-26 09:14:44,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 09:14:44,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5711.76 | bwd_inner_microstep: 5698.69 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.84
[2025-04-26 09:14:44,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5711.77 | bwd_inner: 5698.69 | bwd_allreduce: 13.04 | step: 18.85

  5%|▌         | 2114/41250 [5:06:15<94:11:31,  8.66s/it]
                                                         
{'loss': 0.2129, 'grad_norm': 1.791334867477417, 'learning_rate': 3.995271142575797e-05, 'epoch': 0.51}

  5%|▌         | 2114/41250 [5:06:15<94:11:31,  8.66s/it][2025-04-26 09:14:53,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:14:53,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.16 | bwd_microstep: 5802.58 | bwd_inner_microstep: 5657.07 | bwd_allreduce_microstep: 145.47 | step_microstep: 18.58
[2025-04-26 09:14:53,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.16 | bwd: 5802.60 | bwd_inner: 5657.07 | bwd_allreduce: 145.48 | step: 18.58

  5%|▌         | 2115/41250 [5:06:24<94:21:01,  8.68s/it]
                                                         
{'loss': 0.0596, 'grad_norm': 0.7167279124259949, 'learning_rate': 3.995260344204041e-05, 'epoch': 0.51}

  5%|▌         | 2115/41250 [5:06:24<94:21:01,  8.68s/it][2025-04-26 09:15:02,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 09:15:02,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.47 | bwd_microstep: 5749.01 | bwd_inner_microstep: 5658.61 | bwd_allreduce_microstep: 90.35 | step_microstep: 18.61
[2025-04-26 09:15:02,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.47 | bwd: 5749.03 | bwd_inner: 5658.61 | bwd_allreduce: 90.37 | step: 18.61

  5%|▌         | 2116/41250 [5:06:33<94:16:29,  8.67s/it]
                                                         
{'loss': 0.0993, 'grad_norm': 1.336902141571045, 'learning_rate': 3.9952495335318965e-05, 'epoch': 0.51}

  5%|▌         | 2116/41250 [5:06:33<94:16:29,  8.67s/it][2025-04-26 09:15:10,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:15:10,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.60 | bwd_microstep: 5710.56 | bwd_inner_microstep: 5684.69 | bwd_allreduce_microstep: 25.82 | step_microstep: 18.71
[2025-04-26 09:15:10,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.60 | bwd: 5710.57 | bwd_inner: 5684.69 | bwd_allreduce: 25.84 | step: 18.72

  5%|▌         | 2117/41250 [5:06:41<94:08:29,  8.66s/it]
                                                         
{'loss': 0.3066, 'grad_norm': 3.4680464267730713, 'learning_rate': 3.995238710559432e-05, 'epoch': 0.51}

  5%|▌         | 2117/41250 [5:06:41<94:08:29,  8.66s/it][2025-04-26 09:15:19,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:15:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.28 | bwd_microstep: 5711.93 | bwd_inner_microstep: 5699.18 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.72
[2025-04-26 09:15:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.28 | bwd: 5711.94 | bwd_inner: 5699.18 | bwd_allreduce: 12.72 | step: 18.73

  5%|▌         | 2118/41250 [5:06:50<94:05:55,  8.66s/it]
                                                         
{'loss': 0.1254, 'grad_norm': 1.9498621225357056, 'learning_rate': 3.995227875286713e-05, 'epoch': 0.51}

  5%|▌         | 2118/41250 [5:06:50<94:05:55,  8.66s/it][2025-04-26 09:15:27,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:15:27,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.84 | bwd_microstep: 5712.94 | bwd_inner_microstep: 5690.22 | bwd_allreduce_microstep: 22.68 | step_microstep: 18.75
[2025-04-26 09:15:27,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.84 | bwd: 5712.96 | bwd_inner: 5690.22 | bwd_allreduce: 22.69 | step: 18.75

  5%|▌         | 2119/41250 [5:06:59<94:02:37,  8.65s/it]
                                                         
{'loss': 0.2654, 'grad_norm': 2.341317892074585, 'learning_rate': 3.9952170277138065e-05, 'epoch': 0.51}

  5%|▌         | 2119/41250 [5:06:59<94:02:37,  8.65s/it][2025-04-26 09:15:36,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:15:36,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5799.87 | bwd_inner_microstep: 5651.73 | bwd_allreduce_microstep: 148.09 | step_microstep: 18.77
[2025-04-26 09:15:36,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5799.88 | bwd_inner: 5651.73 | bwd_allreduce: 148.11 | step: 18.77

  5%|▌         | 2120/41250 [5:07:07<94:13:57,  8.67s/it]
                                                         
{'loss': 0.2754, 'grad_norm': 1.431293249130249, 'learning_rate': 3.9952061678407796e-05, 'epoch': 0.51}

  5%|▌         | 2120/41250 [5:07:07<94:13:57,  8.67s/it][2025-04-26 09:15:45,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 09:15:45,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.61 | bwd_microstep: 5791.69 | bwd_inner_microstep: 5778.71 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.66
[2025-04-26 09:15:45,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.61 | bwd: 5791.70 | bwd_inner: 5778.71 | bwd_allreduce: 12.95 | step: 18.67

  5%|▌         | 2121/41250 [5:07:16<94:31:21,  8.70s/it]
                                                         
{'loss': 0.1462, 'grad_norm': 3.553236961364746, 'learning_rate': 3.9951952956676987e-05, 'epoch': 0.51}

  5%|▌         | 2121/41250 [5:07:16<94:31:21,  8.70s/it][2025-04-26 09:15:54,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 1.07
[2025-04-26 09:15:54,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.69 | bwd_microstep: 5914.77 | bwd_inner_microstep: 5662.84 | bwd_allreduce_microstep: 251.89 | step_microstep: 18.17
[2025-04-26 09:15:54,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.69 | bwd: 5914.79 | bwd_inner: 5662.84 | bwd_allreduce: 251.90 | step: 18.17

  5%|▌         | 2122/41250 [5:07:25<94:57:35,  8.74s/it]
                                                         
{'loss': 0.0842, 'grad_norm': 0.9117089509963989, 'learning_rate': 3.995184411194631e-05, 'epoch': 0.51}

  5%|▌         | 2122/41250 [5:07:25<94:57:35,  8.74s/it][2025-04-26 09:16:02,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:16:02,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 5778.39 | bwd_inner_microstep: 5695.19 | bwd_allreduce_microstep: 83.16 | step_microstep: 18.51
[2025-04-26 09:16:02,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 5778.40 | bwd_inner: 5695.19 | bwd_allreduce: 83.17 | step: 18.52

  5%|▌         | 2123/41250 [5:07:34<94:52:23,  8.73s/it]
                                                         
{'loss': 0.152, 'grad_norm': 1.296645164489746, 'learning_rate': 3.9951735144216454e-05, 'epoch': 0.51}

  5%|▌         | 2123/41250 [5:07:34<94:52:23,  8.73s/it][2025-04-26 09:16:11,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 1.06
[2025-04-26 09:16:11,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.27 | bwd_microstep: 5785.11 | bwd_inner_microstep: 5688.31 | bwd_allreduce_microstep: 96.76 | step_microstep: 18.28
[2025-04-26 09:16:11,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.27 | bwd: 5785.12 | bwd_inner: 5688.31 | bwd_allreduce: 96.77 | step: 18.29

  5%|▌         | 2124/41250 [5:07:42<94:51:37,  8.73s/it]
                                                         
{'loss': 0.2469, 'grad_norm': 2.0746824741363525, 'learning_rate': 3.995162605348807e-05, 'epoch': 0.51}

  5%|▌         | 2124/41250 [5:07:42<94:51:37,  8.73s/it][2025-04-26 09:16:20,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:16:20,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.89 | bwd_microstep: 5753.83 | bwd_inner_microstep: 5700.31 | bwd_allreduce_microstep: 53.48 | step_microstep: 18.87
[2025-04-26 09:16:20,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.89 | bwd: 5753.84 | bwd_inner: 5700.31 | bwd_allreduce: 53.49 | step: 18.88

  5%|▌         | 2125/41250 [5:07:51<94:44:14,  8.72s/it]
                                                         
{'loss': 0.1415, 'grad_norm': 1.8260810375213623, 'learning_rate': 3.995151683976183e-05, 'epoch': 0.52}

  5%|▌         | 2125/41250 [5:07:51<94:44:14,  8.72s/it][2025-04-26 09:16:29,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:16:29,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.07 | bwd_microstep: 5777.72 | bwd_inner_microstep: 5652.90 | bwd_allreduce_microstep: 124.78 | step_microstep: 17.72
[2025-04-26 09:16:29,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.07 | bwd: 5777.73 | bwd_inner: 5652.90 | bwd_allreduce: 124.79 | step: 17.72

  5%|▌         | 2126/41250 [5:08:00<94:38:33,  8.71s/it]
                                                         
{'loss': 0.1471, 'grad_norm': 1.848927617073059, 'learning_rate': 3.995140750303841e-05, 'epoch': 0.52}

  5%|▌         | 2126/41250 [5:08:00<94:38:33,  8.71s/it][2025-04-26 09:16:37,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:16:37,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.87 | bwd_microstep: 5758.37 | bwd_inner_microstep: 5703.09 | bwd_allreduce_microstep: 55.23 | step_microstep: 17.85
[2025-04-26 09:16:37,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.87 | bwd: 5758.38 | bwd_inner: 5703.09 | bwd_allreduce: 55.25 | step: 17.86

  5%|▌         | 2127/41250 [5:08:08<94:33:33,  8.70s/it]
                                                         
{'loss': 0.1073, 'grad_norm': 1.1499123573303223, 'learning_rate': 3.995129804331849e-05, 'epoch': 0.52}

  5%|▌         | 2127/41250 [5:08:08<94:33:33,  8.70s/it][2025-04-26 09:16:46,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:16:46,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.36 | bwd_microstep: 6035.77 | bwd_inner_microstep: 5637.55 | bwd_allreduce_microstep: 398.18 | step_microstep: 17.77
[2025-04-26 09:16:46,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.36 | bwd: 6035.78 | bwd_inner: 5637.55 | bwd_allreduce: 398.19 | step: 17.78

  5%|▌         | 2128/41250 [5:08:17<95:23:09,  8.78s/it]
                                                         
{'loss': 0.2005, 'grad_norm': 1.989404320716858, 'learning_rate': 3.9951188460602746e-05, 'epoch': 0.52}

  5%|▌         | 2128/41250 [5:08:17<95:23:09,  8.78s/it][2025-04-26 09:16:55,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:16:55,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.94 | bwd_microstep: 5774.43 | bwd_inner_microstep: 5640.11 | bwd_allreduce_microstep: 134.28 | step_microstep: 18.08
[2025-04-26 09:16:55,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.94 | bwd: 5774.45 | bwd_inner: 5640.11 | bwd_allreduce: 134.29 | step: 18.08

  5%|▌         | 2129/41250 [5:08:26<95:05:48,  8.75s/it]
                                                         
{'loss': 0.1172, 'grad_norm': 1.4846612215042114, 'learning_rate': 3.995107875489184e-05, 'epoch': 0.52}

  5%|▌         | 2129/41250 [5:08:26<95:05:48,  8.75s/it][2025-04-26 09:17:04,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-26 09:17:04,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.05 | bwd_microstep: 5766.27 | bwd_inner_microstep: 5652.40 | bwd_allreduce_microstep: 113.83 | step_microstep: 18.88
[2025-04-26 09:17:04,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.06 | bwd: 5766.29 | bwd_inner: 5652.40 | bwd_allreduce: 113.85 | step: 18.88

  5%|▌         | 2130/41250 [5:08:35<94:54:24,  8.73s/it]
                                                         
{'loss': 0.3442, 'grad_norm': 2.9771740436553955, 'learning_rate': 3.9950968926186465e-05, 'epoch': 0.52}

  5%|▌         | 2130/41250 [5:08:35<94:54:24,  8.73s/it][2025-04-26 09:17:12,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:17:12,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5705.54 | bwd_inner_microstep: 5692.75 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.31
[2025-04-26 09:17:12,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5705.56 | bwd_inner: 5692.75 | bwd_allreduce: 12.76 | step: 18.31

  5%|▌         | 2131/41250 [5:08:43<94:33:56,  8.70s/it]
                                                         
{'loss': 0.267, 'grad_norm': 2.2577097415924072, 'learning_rate': 3.9950858974487285e-05, 'epoch': 0.52}

  5%|▌         | 2131/41250 [5:08:43<94:33:56,  8.70s/it][2025-04-26 09:17:21,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:17:21,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.64 | bwd_microstep: 5702.26 | bwd_inner_microstep: 5689.56 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.06
[2025-04-26 09:17:21,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.64 | bwd: 5702.28 | bwd_inner: 5689.56 | bwd_allreduce: 12.68 | step: 18.06

  5%|▌         | 2132/41250 [5:08:52<94:21:28,  8.68s/it]
                                                         
{'loss': 0.2107, 'grad_norm': 2.342773914337158, 'learning_rate': 3.995074889979499e-05, 'epoch': 0.52}

  5%|▌         | 2132/41250 [5:08:52<94:21:28,  8.68s/it][2025-04-26 09:17:29,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:17:29,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.79 | bwd_microstep: 5703.80 | bwd_inner_microstep: 5691.07 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.66
[2025-04-26 09:17:29,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.79 | bwd: 5703.81 | bwd_inner: 5691.07 | bwd_allreduce: 12.70 | step: 18.67

  5%|▌         | 2133/41250 [5:09:01<94:13:13,  8.67s/it]
                                                         
{'loss': 0.1787, 'grad_norm': 2.174710512161255, 'learning_rate': 3.995063870211025e-05, 'epoch': 0.52}

  5%|▌         | 2133/41250 [5:09:01<94:13:13,  8.67s/it][2025-04-26 09:17:38,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 1.06
[2025-04-26 09:17:38,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5694.64 | bwd_inner_microstep: 5643.19 | bwd_allreduce_microstep: 51.41 | step_microstep: 18.61
[2025-04-26 09:17:38,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5694.66 | bwd_inner: 5643.19 | bwd_allreduce: 51.43 | step: 18.61

  5%|▌         | 2134/41250 [5:09:09<93:59:27,  8.65s/it]
                                                         
{'loss': 0.1102, 'grad_norm': 1.2248430252075195, 'learning_rate': 3.995052838143375e-05, 'epoch': 0.52}

  5%|▌         | 2134/41250 [5:09:09<93:59:27,  8.65s/it][2025-04-26 09:17:47,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 09:17:47,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.59 | bwd_microstep: 5864.52 | bwd_inner_microstep: 5851.78 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.57
[2025-04-26 09:17:47,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.59 | bwd: 5864.54 | bwd_inner: 5851.78 | bwd_allreduce: 12.71 | step: 18.57

  5%|▌         | 2135/41250 [5:09:18<94:42:37,  8.72s/it]
                                                         
{'loss': 0.2724, 'grad_norm': 2.839749336242676, 'learning_rate': 3.995041793776616e-05, 'epoch': 0.52}

  5%|▌         | 2135/41250 [5:09:18<94:42:37,  8.72s/it][2025-04-26 09:17:56,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:17:56,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5745.40 | bwd_inner_microstep: 5642.64 | bwd_allreduce_microstep: 102.72 | step_microstep: 18.51
[2025-04-26 09:17:56,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5745.42 | bwd_inner: 5642.64 | bwd_allreduce: 102.73 | step: 18.51

  5%|▌         | 2136/41250 [5:09:27<94:31:28,  8.70s/it]
                                                         
{'loss': 0.2148, 'grad_norm': 1.9442803859710693, 'learning_rate': 3.995030737110817e-05, 'epoch': 0.52}

  5%|▌         | 2136/41250 [5:09:27<94:31:28,  8.70s/it][2025-04-26 09:18:04,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:18:04,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5718.02 | bwd_inner_microstep: 5705.36 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.23
[2025-04-26 09:18:04,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5718.03 | bwd_inner: 5705.36 | bwd_allreduce: 12.64 | step: 18.24

  5%|▌         | 2137/41250 [5:09:35<94:22:49,  8.69s/it]
                                                         
{'loss': 0.3544, 'grad_norm': 2.08184552192688, 'learning_rate': 3.9950196681460454e-05, 'epoch': 0.52}

  5%|▌         | 2137/41250 [5:09:35<94:22:49,  8.69s/it][2025-04-26 09:18:13,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:18:13,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.19 | bwd_microstep: 5743.65 | bwd_inner_microstep: 5685.88 | bwd_allreduce_microstep: 57.72 | step_microstep: 17.78
[2025-04-26 09:18:13,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.19 | bwd: 5743.66 | bwd_inner: 5685.88 | bwd_allreduce: 57.74 | step: 17.78

  5%|▌         | 2138/41250 [5:09:44<94:20:58,  8.68s/it]
                                                         
{'loss': 0.1451, 'grad_norm': 1.848618984222412, 'learning_rate': 3.995008586882371e-05, 'epoch': 0.52}

  5%|▌         | 2138/41250 [5:09:44<94:20:58,  8.68s/it][2025-04-26 09:18:22,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:18:22,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.20 | bwd_microstep: 5703.83 | bwd_inner_microstep: 5691.06 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.34
[2025-04-26 09:18:22,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.20 | bwd: 5703.84 | bwd_inner: 5691.06 | bwd_allreduce: 12.74 | step: 18.34

  5%|▌         | 2139/41250 [5:09:53<94:09:57,  8.67s/it]
                                                         
{'loss': 0.373, 'grad_norm': 2.4265782833099365, 'learning_rate': 3.9949974933198605e-05, 'epoch': 0.52}

  5%|▌         | 2139/41250 [5:09:53<94:09:57,  8.67s/it][2025-04-26 09:18:30,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 09:18:30,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.55 | bwd_microstep: 5781.05 | bwd_inner_microstep: 5768.18 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.93
[2025-04-26 09:18:30,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.55 | bwd: 5781.07 | bwd_inner: 5768.18 | bwd_allreduce: 12.85 | step: 18.93

  5%|▌         | 2140/41250 [5:10:02<94:24:34,  8.69s/it]
                                                         
{'loss': 0.1748, 'grad_norm': 1.8565860986709595, 'learning_rate': 3.9949863874585825e-05, 'epoch': 0.52}

  5%|▌         | 2140/41250 [5:10:02<94:24:34,  8.69s/it][2025-04-26 09:18:39,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:18:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5680.32 | bwd_inner_microstep: 5667.40 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.38
[2025-04-26 09:18:39,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5680.33 | bwd_inner: 5667.40 | bwd_allreduce: 12.89 | step: 18.39

  5%|▌         | 2141/41250 [5:10:10<94:08:46,  8.67s/it]
                                                         
{'loss': 0.183, 'grad_norm': 1.7540370225906372, 'learning_rate': 3.9949752692986064e-05, 'epoch': 0.52}

  5%|▌         | 2141/41250 [5:10:10<94:08:46,  8.67s/it][2025-04-26 09:18:48,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.77 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:18:48,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.76 | bwd_microstep: 5736.00 | bwd_inner_microstep: 5680.04 | bwd_allreduce_microstep: 55.91 | step_microstep: 17.47
[2025-04-26 09:18:48,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.76 | bwd: 5736.01 | bwd_inner: 5680.04 | bwd_allreduce: 55.93 | step: 17.48

  5%|▌         | 2142/41250 [5:10:19<94:08:48,  8.67s/it]
                                                         
{'loss': 0.4, 'grad_norm': 2.762627363204956, 'learning_rate': 3.9949641388400005e-05, 'epoch': 0.52}

  5%|▌         | 2142/41250 [5:10:19<94:08:48,  8.67s/it][2025-04-26 09:18:56,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:18:56,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.29 | bwd_microstep: 5701.58 | bwd_inner_microstep: 5650.01 | bwd_allreduce_microstep: 51.53 | step_microstep: 18.25
[2025-04-26 09:18:56,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.29 | bwd: 5701.59 | bwd_inner: 5650.01 | bwd_allreduce: 51.54 | step: 18.25

  5%|▌         | 2143/41250 [5:10:27<93:56:28,  8.65s/it]
                                                         
{'loss': 0.3119, 'grad_norm': 2.7472972869873047, 'learning_rate': 3.994952996082832e-05, 'epoch': 0.52}

  5%|▌         | 2143/41250 [5:10:27<93:56:28,  8.65s/it][2025-04-26 09:19:05,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 09:19:05,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.26 | bwd_microstep: 5782.67 | bwd_inner_microstep: 5769.94 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.67
[2025-04-26 09:19:05,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.26 | bwd: 5782.69 | bwd_inner: 5769.94 | bwd_allreduce: 12.70 | step: 18.67

  5%|▌         | 2144/41250 [5:10:36<94:15:38,  8.68s/it]
                                                         
{'loss': 0.18, 'grad_norm': 1.543762445449829, 'learning_rate': 3.9949418410271714e-05, 'epoch': 0.52}

  5%|▌         | 2144/41250 [5:10:36<94:15:38,  8.68s/it][2025-04-26 09:19:14,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:19:14,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.59 | bwd_microstep: 5713.92 | bwd_inner_microstep: 5655.85 | bwd_allreduce_microstep: 58.02 | step_microstep: 18.66
[2025-04-26 09:19:14,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.59 | bwd: 5713.94 | bwd_inner: 5655.85 | bwd_allreduce: 58.04 | step: 18.66

  5%|▌         | 2145/41250 [5:10:45<94:04:03,  8.66s/it]
                                                         
{'loss': 0.1215, 'grad_norm': 0.7358678579330444, 'learning_rate': 3.9949306736730864e-05, 'epoch': 0.52}

  5%|▌         | 2145/41250 [5:10:45<94:04:03,  8.66s/it][2025-04-26 09:19:22,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 09:19:22,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.75 | bwd_microstep: 5698.22 | bwd_inner_microstep: 5682.97 | bwd_allreduce_microstep: 15.20 | step_microstep: 19.34
[2025-04-26 09:19:22,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.75 | bwd: 5698.23 | bwd_inner: 5682.97 | bwd_allreduce: 15.22 | step: 19.35

  5%|▌         | 2146/41250 [5:10:53<93:56:43,  8.65s/it]
                                                         
{'loss': 0.166, 'grad_norm': 2.304785966873169, 'learning_rate': 3.994919494020647e-05, 'epoch': 0.52}

  5%|▌         | 2146/41250 [5:10:53<93:56:43,  8.65s/it][2025-04-26 09:19:31,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 09:19:31,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.99 | bwd_microstep: 5781.17 | bwd_inner_microstep: 5768.02 | bwd_allreduce_microstep: 13.11 | step_microstep: 18.99
[2025-04-26 09:19:31,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.99 | bwd: 5781.19 | bwd_inner: 5768.02 | bwd_allreduce: 13.13 | step: 18.99

  5%|▌         | 2147/41250 [5:11:02<94:16:05,  8.68s/it]
                                                         
{'loss': 0.1887, 'grad_norm': 1.9613633155822754, 'learning_rate': 3.99490830206992e-05, 'epoch': 0.52}

  5%|▌         | 2147/41250 [5:11:02<94:16:05,  8.68s/it][2025-04-26 09:19:40,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:19:40,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.44 | bwd_microstep: 5702.66 | bwd_inner_microstep: 5652.62 | bwd_allreduce_microstep: 49.99 | step_microstep: 18.86
[2025-04-26 09:19:40,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.44 | bwd: 5702.68 | bwd_inner: 5652.62 | bwd_allreduce: 50.01 | step: 18.87

  5%|▌         | 2148/41250 [5:11:11<94:04:02,  8.66s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 1.1677284240722656, 'learning_rate': 3.9948970978209766e-05, 'epoch': 0.52}

  5%|▌         | 2148/41250 [5:11:11<94:04:02,  8.66s/it][2025-04-26 09:19:48,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:19:48,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.13 | bwd_microstep: 5762.85 | bwd_inner_microstep: 5705.30 | bwd_allreduce_microstep: 57.51 | step_microstep: 18.64
[2025-04-26 09:19:48,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.13 | bwd: 5762.86 | bwd_inner: 5705.30 | bwd_allreduce: 57.52 | step: 18.64

  5%|▌         | 2149/41250 [5:11:19<94:09:14,  8.67s/it]
                                                         
{'loss': 0.0996, 'grad_norm': 1.2346856594085693, 'learning_rate': 3.9948858812738845e-05, 'epoch': 0.52}

  5%|▌         | 2149/41250 [5:11:19<94:09:14,  8.67s/it][2025-04-26 09:19:57,455] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:19:57,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.64 | bwd_microstep: 5799.87 | bwd_inner_microstep: 5662.43 | bwd_allreduce_microstep: 137.40 | step_microstep: 18.68
[2025-04-26 09:19:57,456] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.64 | bwd: 5799.88 | bwd_inner: 5662.43 | bwd_allreduce: 137.41 | step: 18.68

  5%|▌         | 2150/41250 [5:11:28<94:17:11,  8.68s/it]
                                                         
{'loss': 0.2965, 'grad_norm': 2.5093865394592285, 'learning_rate': 3.9948746524287135e-05, 'epoch': 0.52}

  5%|▌         | 2150/41250 [5:11:28<94:17:11,  8.68s/it][2025-04-26 09:20:06,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:20:06,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.15 | bwd_microstep: 5734.95 | bwd_inner_microstep: 5646.11 | bwd_allreduce_microstep: 88.79 | step_microstep: 18.72
[2025-04-26 09:20:06,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.15 | bwd: 5734.97 | bwd_inner: 5646.11 | bwd_allreduce: 88.81 | step: 18.72

  5%|▌         | 2151/41250 [5:11:37<94:09:07,  8.67s/it]
                                                         
{'loss': 0.1888, 'grad_norm': 1.8985520601272583, 'learning_rate': 3.994863411285533e-05, 'epoch': 0.52}

  5%|▌         | 2151/41250 [5:11:37<94:09:07,  8.67s/it][2025-04-26 09:20:14,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 09:20:14,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.56 | bwd_microstep: 5749.08 | bwd_inner_microstep: 5693.32 | bwd_allreduce_microstep: 55.71 | step_microstep: 19.16
[2025-04-26 09:20:14,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.56 | bwd: 5749.09 | bwd_inner: 5693.32 | bwd_allreduce: 55.73 | step: 19.16

  5%|▌         | 2152/41250 [5:11:45<94:11:57,  8.67s/it]
                                                         
{'loss': 0.1612, 'grad_norm': 2.235497236251831, 'learning_rate': 3.994852157844411e-05, 'epoch': 0.52}

  5%|▌         | 2152/41250 [5:11:45<94:11:57,  8.67s/it][2025-04-26 09:20:23,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:20:23,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.98 | bwd_microstep: 5752.60 | bwd_inner_microstep: 5711.57 | bwd_allreduce_microstep: 40.98 | step_microstep: 18.46
[2025-04-26 09:20:23,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.99 | bwd: 5752.61 | bwd_inner: 5711.57 | bwd_allreduce: 41.00 | step: 18.46

  5%|▌         | 2153/41250 [5:11:54<94:13:59,  8.68s/it]
                                                         
{'loss': 0.2562, 'grad_norm': 1.3265352249145508, 'learning_rate': 3.994840892105418e-05, 'epoch': 0.52}

  5%|▌         | 2153/41250 [5:11:54<94:13:59,  8.68s/it][2025-04-26 09:20:32,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:20:32,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.02 | bwd_microstep: 5730.03 | bwd_inner_microstep: 5717.37 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.24
[2025-04-26 09:20:32,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.02 | bwd: 5730.04 | bwd_inner: 5717.37 | bwd_allreduce: 12.64 | step: 18.24

  5%|▌         | 2154/41250 [5:12:03<94:14:45,  8.68s/it]
                                                         
{'loss': 0.2733, 'grad_norm': 2.8238439559936523, 'learning_rate': 3.994829614068624e-05, 'epoch': 0.52}

  5%|▌         | 2154/41250 [5:12:03<94:14:45,  8.68s/it][2025-04-26 09:20:40,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 09:20:40,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.49 | bwd_microstep: 5722.98 | bwd_inner_microstep: 5658.68 | bwd_allreduce_microstep: 64.25 | step_microstep: 18.83
[2025-04-26 09:20:40,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.49 | bwd: 5723.00 | bwd_inner: 5658.68 | bwd_allreduce: 64.27 | step: 18.83

  5%|▌         | 2155/41250 [5:12:11<94:07:55,  8.67s/it]
                                                         
{'loss': 0.1718, 'grad_norm': 0.9297168254852295, 'learning_rate': 3.9948183237340975e-05, 'epoch': 0.52}

  5%|▌         | 2155/41250 [5:12:11<94:07:55,  8.67s/it][2025-04-26 09:20:49,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 09:20:49,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.76 | bwd_microstep: 5713.53 | bwd_inner_microstep: 5667.06 | bwd_allreduce_microstep: 46.43 | step_microstep: 18.92
[2025-04-26 09:20:49,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.76 | bwd: 5713.55 | bwd_inner: 5667.06 | bwd_allreduce: 46.45 | step: 18.92

  5%|▌         | 2156/41250 [5:12:20<94:02:56,  8.66s/it]
                                                         
{'loss': 0.1158, 'grad_norm': 1.133448839187622, 'learning_rate': 3.9948070211019085e-05, 'epoch': 0.52}

  5%|▌         | 2156/41250 [5:12:20<94:02:56,  8.66s/it][2025-04-26 09:20:58,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.14 | optimizer_step: 0.94
[2025-04-26 09:20:58,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.08 | bwd_microstep: 5698.09 | bwd_inner_microstep: 5667.26 | bwd_allreduce_microstep: 30.78 | step_microstep: 18.61
[2025-04-26 09:20:58,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.08 | bwd: 5698.10 | bwd_inner: 5667.26 | bwd_allreduce: 30.80 | step: 18.61

  5%|▌         | 2157/41250 [5:12:29<93:54:57,  8.65s/it]
                                                         
{'loss': 0.2026, 'grad_norm': 1.230494737625122, 'learning_rate': 3.9947957061721265e-05, 'epoch': 0.52}

  5%|▌         | 2157/41250 [5:12:29<93:54:57,  8.65s/it][2025-04-26 09:21:06,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 09:21:06,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.92 | bwd_microstep: 5797.05 | bwd_inner_microstep: 5650.02 | bwd_allreduce_microstep: 146.98 | step_microstep: 18.55
[2025-04-26 09:21:06,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.92 | bwd: 5797.06 | bwd_inner: 5650.02 | bwd_allreduce: 147.00 | step: 18.55

  5%|▌         | 2158/41250 [5:12:37<94:04:15,  8.66s/it]
                                                         
{'loss': 0.1605, 'grad_norm': 1.8087912797927856, 'learning_rate': 3.994784378944821e-05, 'epoch': 0.52}

  5%|▌         | 2158/41250 [5:12:37<94:04:15,  8.66s/it][2025-04-26 09:21:15,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:21:15,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5744.19 | bwd_inner_microstep: 5701.87 | bwd_allreduce_microstep: 42.27 | step_microstep: 18.44
[2025-04-26 09:21:15,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5744.21 | bwd_inner: 5701.87 | bwd_allreduce: 42.29 | step: 18.45

  5%|▌         | 2159/41250 [5:12:46<94:08:21,  8.67s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.334389328956604, 'learning_rate': 3.994773039420062e-05, 'epoch': 0.52}

  5%|▌         | 2159/41250 [5:12:46<94:08:21,  8.67s/it][2025-04-26 09:21:24,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:21:24,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.94 | bwd_microstep: 5790.40 | bwd_inner_microstep: 5659.20 | bwd_allreduce_microstep: 131.14 | step_microstep: 18.99
[2025-04-26 09:21:24,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.94 | bwd: 5790.41 | bwd_inner: 5659.20 | bwd_allreduce: 131.17 | step: 19.00

  5%|▌         | 2160/41250 [5:12:55<94:17:22,  8.68s/it]
                                                         
{'loss': 0.1646, 'grad_norm': 1.3701236248016357, 'learning_rate': 3.99476168759792e-05, 'epoch': 0.52}

  5%|▌         | 2160/41250 [5:12:55<94:17:22,  8.68s/it][2025-04-26 09:21:32,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 09:21:32,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.79 | bwd_microstep: 5885.33 | bwd_inner_microstep: 5697.82 | bwd_allreduce_microstep: 187.46 | step_microstep: 19.21
[2025-04-26 09:21:32,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.79 | bwd: 5885.34 | bwd_inner: 5697.82 | bwd_allreduce: 187.48 | step: 19.21

  5%|▌         | 2161/41250 [5:13:04<94:46:00,  8.73s/it]
                                                         
{'loss': 0.2603, 'grad_norm': 1.993549108505249, 'learning_rate': 3.994750323478464e-05, 'epoch': 0.52}

  5%|▌         | 2161/41250 [5:13:04<94:46:00,  8.73s/it][2025-04-26 09:21:41,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:21:41,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.92 | bwd_microstep: 5757.32 | bwd_inner_microstep: 5696.58 | bwd_allreduce_microstep: 60.69 | step_microstep: 18.69
[2025-04-26 09:21:41,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.92 | bwd: 5757.33 | bwd_inner: 5696.58 | bwd_allreduce: 60.71 | step: 18.69

  5%|▌         | 2162/41250 [5:13:12<94:38:25,  8.72s/it]
                                                         
{'loss': 0.1513, 'grad_norm': 1.0775865316390991, 'learning_rate': 3.994738947061766e-05, 'epoch': 0.52}

  5%|▌         | 2162/41250 [5:13:12<94:38:25,  8.72s/it][2025-04-26 09:21:50,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:21:50,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.64 | bwd_microstep: 5779.78 | bwd_inner_microstep: 5697.87 | bwd_allreduce_microstep: 81.87 | step_microstep: 18.37
[2025-04-26 09:21:50,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.64 | bwd: 5779.79 | bwd_inner: 5697.87 | bwd_allreduce: 81.89 | step: 18.37

  5%|▌         | 2163/41250 [5:13:21<94:39:43,  8.72s/it]
                                                         
{'loss': 0.1386, 'grad_norm': 1.228963017463684, 'learning_rate': 3.994727558347894e-05, 'epoch': 0.52}

  5%|▌         | 2163/41250 [5:13:21<94:39:43,  8.72s/it][2025-04-26 09:21:59,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:21:59,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.33 | bwd_microstep: 5788.69 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 135.14 | step_microstep: 18.35
[2025-04-26 09:21:59,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.33 | bwd: 5788.70 | bwd_inner: 5653.50 | bwd_allreduce: 135.16 | step: 18.35

  5%|▌         | 2164/41250 [5:13:30<94:35:34,  8.71s/it]
                                                         
{'loss': 0.2133, 'grad_norm': 2.1212453842163086, 'learning_rate': 3.994716157336918e-05, 'epoch': 0.52}

  5%|▌         | 2164/41250 [5:13:30<94:35:34,  8.71s/it][2025-04-26 09:22:07,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:22:07,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.92 | bwd_microstep: 5716.02 | bwd_inner_microstep: 5643.61 | bwd_allreduce_microstep: 72.37 | step_microstep: 18.53
[2025-04-26 09:22:07,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.92 | bwd: 5716.04 | bwd_inner: 5643.61 | bwd_allreduce: 72.38 | step: 18.53

  5%|▌         | 2165/41250 [5:13:38<94:18:16,  8.69s/it]
                                                         
{'loss': 0.4597, 'grad_norm': 2.692957878112793, 'learning_rate': 3.9947047440289106e-05, 'epoch': 0.52}

  5%|▌         | 2165/41250 [5:13:38<94:18:16,  8.69s/it][2025-04-26 09:22:16,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:22:16,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.67 | bwd_microstep: 5700.17 | bwd_inner_microstep: 5649.52 | bwd_allreduce_microstep: 50.60 | step_microstep: 18.64
[2025-04-26 09:22:16,336] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.68 | bwd: 5700.19 | bwd_inner: 5649.52 | bwd_allreduce: 50.63 | step: 18.65

  5%|▌         | 2166/41250 [5:13:47<94:05:03,  8.67s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 0.9878154397010803, 'learning_rate': 3.9946933184239396e-05, 'epoch': 0.53}

  5%|▌         | 2166/41250 [5:13:47<94:05:03,  8.67s/it][2025-04-26 09:22:24,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 09:22:24,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.15 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.85
[2025-04-26 09:22:24,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.15 | bwd: 5707.57 | bwd_inner: 5694.79 | bwd_allreduce: 12.74 | step: 18.85

  5%|▌         | 2167/41250 [5:13:56<94:00:04,  8.66s/it]
                                                         
{'loss': 0.1579, 'grad_norm': 1.5327532291412354, 'learning_rate': 3.9946818805220776e-05, 'epoch': 0.53}

  5%|▌         | 2167/41250 [5:13:56<94:00:04,  8.66s/it][2025-04-26 09:22:33,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 09:22:33,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.10 | bwd_microstep: 5707.78 | bwd_inner_microstep: 5694.65 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.86
[2025-04-26 09:22:33,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.10 | bwd: 5707.79 | bwd_inner: 5694.65 | bwd_allreduce: 13.10 | step: 18.87

  5%|▌         | 2168/41250 [5:14:04<93:56:42,  8.65s/it]
                                                         
{'loss': 0.043, 'grad_norm': 0.4143967628479004, 'learning_rate': 3.994670430323393e-05, 'epoch': 0.53}

  5%|▌         | 2168/41250 [5:14:04<93:56:42,  8.65s/it][2025-04-26 09:22:42,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:22:42,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.69 | bwd_microstep: 5774.46 | bwd_inner_microstep: 5690.73 | bwd_allreduce_microstep: 83.68 | step_microstep: 18.35
[2025-04-26 09:22:42,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.69 | bwd: 5774.47 | bwd_inner: 5690.73 | bwd_allreduce: 83.70 | step: 18.35

  5%|▌         | 2169/41250 [5:14:13<94:05:19,  8.67s/it]
                                                         
{'loss': 0.2805, 'grad_norm': 3.374929904937744, 'learning_rate': 3.994658967827959e-05, 'epoch': 0.53}

  5%|▌         | 2169/41250 [5:14:13<94:05:19,  8.67s/it][2025-04-26 09:22:50,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:22:50,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.57 | bwd_microstep: 5742.76 | bwd_inner_microstep: 5654.01 | bwd_allreduce_microstep: 88.70 | step_microstep: 18.42
[2025-04-26 09:22:50,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.57 | bwd: 5742.77 | bwd_inner: 5654.01 | bwd_allreduce: 88.72 | step: 18.42

  5%|▌         | 2170/41250 [5:14:22<94:05:07,  8.67s/it]
                                                         
{'loss': 0.2707, 'grad_norm': 2.0625104904174805, 'learning_rate': 3.994647493035844e-05, 'epoch': 0.53}

  5%|▌         | 2170/41250 [5:14:22<94:05:07,  8.67s/it][2025-04-26 09:22:59,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:22:59,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.92 | bwd_microstep: 5764.65 | bwd_inner_microstep: 5645.77 | bwd_allreduce_microstep: 118.83 | step_microstep: 18.32
[2025-04-26 09:22:59,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.92 | bwd: 5764.66 | bwd_inner: 5645.77 | bwd_allreduce: 118.85 | step: 18.33

  5%|▌         | 2171/41250 [5:14:30<94:03:10,  8.66s/it]
                                                         
{'loss': 0.1913, 'grad_norm': 1.5062270164489746, 'learning_rate': 3.994636005947119e-05, 'epoch': 0.53}

  5%|▌         | 2171/41250 [5:14:30<94:03:10,  8.66s/it][2025-04-26 09:23:08,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:23:08,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.58 | bwd_microstep: 5729.16 | bwd_inner_microstep: 5691.25 | bwd_allreduce_microstep: 37.87 | step_microstep: 18.25
[2025-04-26 09:23:08,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.58 | bwd: 5729.17 | bwd_inner: 5691.24 | bwd_allreduce: 37.89 | step: 18.26

  5%|▌         | 2172/41250 [5:14:39<94:02:44,  8.66s/it]
                                                         
{'loss': 0.1806, 'grad_norm': 1.6828649044036865, 'learning_rate': 3.994624506561857e-05, 'epoch': 0.53}

  5%|▌         | 2172/41250 [5:14:39<94:02:44,  8.66s/it][2025-04-26 09:23:16,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:23:16,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.65 | bwd_microstep: 5751.45 | bwd_inner_microstep: 5643.74 | bwd_allreduce_microstep: 107.67 | step_microstep: 18.39
[2025-04-26 09:23:16,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.65 | bwd: 5751.46 | bwd_inner: 5643.74 | bwd_allreduce: 107.68 | step: 18.39

  5%|▌         | 2173/41250 [5:14:48<94:01:12,  8.66s/it]
                                                         
{'loss': 0.1251, 'grad_norm': 1.2512938976287842, 'learning_rate': 3.994612994880125e-05, 'epoch': 0.53}

  5%|▌         | 2173/41250 [5:14:48<94:01:12,  8.66s/it][2025-04-26 09:23:25,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 09:23:25,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.51 | bwd_microstep: 5727.73 | bwd_inner_microstep: 5684.78 | bwd_allreduce_microstep: 42.91 | step_microstep: 18.76
[2025-04-26 09:23:25,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.51 | bwd: 5727.74 | bwd_inner: 5684.78 | bwd_allreduce: 42.93 | step: 18.76

  5%|▌         | 2174/41250 [5:14:56<93:59:19,  8.66s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 2.034954309463501, 'learning_rate': 3.994601470901998e-05, 'epoch': 0.53}

  5%|▌         | 2174/41250 [5:14:56<93:59:19,  8.66s/it][2025-04-26 09:23:34,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 09:23:34,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.27 | bwd_microstep: 5724.44 | bwd_inner_microstep: 5696.72 | bwd_allreduce_microstep: 27.68 | step_microstep: 18.66
[2025-04-26 09:23:34,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.27 | bwd: 5724.45 | bwd_inner: 5696.72 | bwd_allreduce: 27.70 | step: 18.67

  5%|▌         | 2175/41250 [5:15:05<94:00:57,  8.66s/it]
                                                         
{'loss': 0.2771, 'grad_norm': 2.6084532737731934, 'learning_rate': 3.994589934627545e-05, 'epoch': 0.53}

  5%|▌         | 2175/41250 [5:15:05<94:00:57,  8.66s/it][2025-04-26 09:23:42,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:23:42,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.93 | bwd_microstep: 5700.78 | bwd_inner_microstep: 5651.49 | bwd_allreduce_microstep: 49.25 | step_microstep: 17.99
[2025-04-26 09:23:42,897] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.93 | bwd: 5700.80 | bwd_inner: 5651.49 | bwd_allreduce: 49.27 | step: 17.99

  5%|▌         | 2176/41250 [5:15:14<93:50:58,  8.65s/it]
                                                         
{'loss': 0.2859, 'grad_norm': 1.5415321588516235, 'learning_rate': 3.9945783860568364e-05, 'epoch': 0.53}

  5%|▌         | 2176/41250 [5:15:14<93:50:58,  8.65s/it][2025-04-26 09:23:51,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:23:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.45 | bwd_microstep: 5749.97 | bwd_inner_microstep: 5645.74 | bwd_allreduce_microstep: 104.18 | step_microstep: 18.57
[2025-04-26 09:23:51,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.45 | bwd: 5749.99 | bwd_inner: 5645.74 | bwd_allreduce: 104.20 | step: 18.58

  5%|▌         | 2177/41250 [5:15:22<93:54:49,  8.65s/it]
                                                         
{'loss': 0.0714, 'grad_norm': 0.7037721276283264, 'learning_rate': 3.994566825189945e-05, 'epoch': 0.53}

  5%|▌         | 2177/41250 [5:15:22<93:54:49,  8.65s/it][2025-04-26 09:24:00,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 09:24:00,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.91 | bwd_microstep: 5723.32 | bwd_inner_microstep: 5696.98 | bwd_allreduce_microstep: 26.29 | step_microstep: 18.81
[2025-04-26 09:24:00,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.91 | bwd: 5723.33 | bwd_inner: 5696.98 | bwd_allreduce: 26.31 | step: 18.81

  5%|▌         | 2178/41250 [5:15:31<93:55:16,  8.65s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 0.9004895687103271, 'learning_rate': 3.9945552520269416e-05, 'epoch': 0.53}

  5%|▌         | 2178/41250 [5:15:31<93:55:16,  8.65s/it][2025-04-26 09:24:08,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 09:24:08,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.51 | bwd_microstep: 5758.69 | bwd_inner_microstep: 5655.03 | bwd_allreduce_microstep: 103.60 | step_microstep: 18.87
[2025-04-26 09:24:08,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.51 | bwd: 5758.70 | bwd_inner: 5655.03 | bwd_allreduce: 103.62 | step: 18.87

  5%|▌         | 2179/41250 [5:15:40<93:58:06,  8.66s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.9884356260299683, 'learning_rate': 3.9945436665678976e-05, 'epoch': 0.53}

  5%|▌         | 2179/41250 [5:15:40<93:58:06,  8.66s/it][2025-04-26 09:24:17,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:24:17,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.53 | bwd_microstep: 5756.96 | bwd_inner_microstep: 5679.22 | bwd_allreduce_microstep: 77.70 | step_microstep: 18.29
[2025-04-26 09:24:17,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.53 | bwd: 5756.97 | bwd_inner: 5679.22 | bwd_allreduce: 77.71 | step: 18.29

  5%|▌         | 2180/41250 [5:15:48<94:01:15,  8.66s/it]
                                                         
{'loss': 0.277, 'grad_norm': 1.7287489175796509, 'learning_rate': 3.9945320688128834e-05, 'epoch': 0.53}

  5%|▌         | 2180/41250 [5:15:48<94:01:15,  8.66s/it][2025-04-26 09:24:26,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 09:24:26,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.53 | bwd_microstep: 5760.04 | bwd_inner_microstep: 5645.18 | bwd_allreduce_microstep: 114.81 | step_microstep: 18.49
[2025-04-26 09:24:26,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.53 | bwd: 5760.05 | bwd_inner: 5645.18 | bwd_allreduce: 114.83 | step: 18.50

  5%|▌         | 2181/41250 [5:15:57<94:04:38,  8.67s/it]
                                                         
{'loss': 0.2303, 'grad_norm': 2.015387535095215, 'learning_rate': 3.994520458761972e-05, 'epoch': 0.53}

  5%|▌         | 2181/41250 [5:15:57<94:04:38,  8.67s/it][2025-04-26 09:24:34,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:24:34,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5724.13 | bwd_inner_microstep: 5677.53 | bwd_allreduce_microstep: 46.56 | step_microstep: 18.40
[2025-04-26 09:24:34,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5724.14 | bwd_inner: 5677.53 | bwd_allreduce: 46.57 | step: 18.40

  5%|▌         | 2182/41250 [5:16:06<94:00:41,  8.66s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 1.6842859983444214, 'learning_rate': 3.994508836415234e-05, 'epoch': 0.53}

  5%|▌         | 2182/41250 [5:16:06<94:00:41,  8.66s/it][2025-04-26 09:24:43,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:24:43,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.42 | bwd_microstep: 5684.71 | bwd_inner_microstep: 5651.37 | bwd_allreduce_microstep: 33.30 | step_microstep: 17.82
[2025-04-26 09:24:43,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.42 | bwd: 5684.73 | bwd_inner: 5651.37 | bwd_allreduce: 33.32 | step: 17.83

  5%|▌         | 2183/41250 [5:16:14<93:47:19,  8.64s/it]
                                                         
{'loss': 0.0965, 'grad_norm': 1.1342467069625854, 'learning_rate': 3.994497201772741e-05, 'epoch': 0.53}

  5%|▌         | 2183/41250 [5:16:14<93:47:19,  8.64s/it][2025-04-26 09:24:52,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:24:52,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.07 | bwd_microstep: 5671.12 | bwd_inner_microstep: 5644.82 | bwd_allreduce_microstep: 26.26 | step_microstep: 18.29
[2025-04-26 09:24:52,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.07 | bwd: 5671.14 | bwd_inner: 5644.82 | bwd_allreduce: 26.27 | step: 18.29

  5%|▌         | 2184/41250 [5:16:23<93:36:03,  8.63s/it]
                                                         
{'loss': 0.0846, 'grad_norm': 0.8251906037330627, 'learning_rate': 3.994485554834565e-05, 'epoch': 0.53}

  5%|▌         | 2184/41250 [5:16:23<93:36:03,  8.63s/it][2025-04-26 09:25:00,685] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:25:00,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.97 | bwd_microstep: 5697.14 | bwd_inner_microstep: 5638.93 | bwd_allreduce_microstep: 58.17 | step_microstep: 18.62
[2025-04-26 09:25:00,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.97 | bwd: 5697.15 | bwd_inner: 5638.93 | bwd_allreduce: 58.19 | step: 18.62

  5%|▌         | 2185/41250 [5:16:31<93:33:01,  8.62s/it]
                                                         
{'loss': 0.331, 'grad_norm': 2.269655466079712, 'learning_rate': 3.994473895600778e-05, 'epoch': 0.53}

  5%|▌         | 2185/41250 [5:16:31<93:33:01,  8.62s/it][2025-04-26 09:25:09,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 09:25:09,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.51 | bwd_microstep: 5749.15 | bwd_inner_microstep: 5650.72 | bwd_allreduce_microstep: 98.39 | step_microstep: 18.74
[2025-04-26 09:25:09,349] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.51 | bwd: 5749.16 | bwd_inner: 5650.72 | bwd_allreduce: 98.41 | step: 18.75

  5%|▌         | 2186/41250 [5:16:40<93:40:54,  8.63s/it]
                                                         
{'loss': 0.1365, 'grad_norm': 2.2765848636627197, 'learning_rate': 3.9944622240714516e-05, 'epoch': 0.53}

  5%|▌         | 2186/41250 [5:16:40<93:40:54,  8.63s/it][2025-04-26 09:25:18,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 09:25:18,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.67 | bwd_microstep: 5769.59 | bwd_inner_microstep: 5678.70 | bwd_allreduce_microstep: 90.86 | step_microstep: 17.78
[2025-04-26 09:25:18,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.67 | bwd: 5769.61 | bwd_inner: 5678.70 | bwd_allreduce: 90.87 | step: 17.78

  5%|▌         | 2187/41250 [5:16:49<93:52:58,  8.65s/it]
                                                         
{'loss': 0.268, 'grad_norm': 2.03001070022583, 'learning_rate': 3.994450540246659e-05, 'epoch': 0.53}

  5%|▌         | 2187/41250 [5:16:49<93:52:58,  8.65s/it][2025-04-26 09:25:26,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:25:26,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.80 | bwd_microstep: 5696.41 | bwd_inner_microstep: 5642.55 | bwd_allreduce_microstep: 53.81 | step_microstep: 17.71
[2025-04-26 09:25:26,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.80 | bwd: 5696.42 | bwd_inner: 5642.55 | bwd_allreduce: 53.83 | step: 17.72

  5%|▌         | 2188/41250 [5:16:57<93:45:14,  8.64s/it]
                                                         
{'loss': 0.2051, 'grad_norm': 1.5624154806137085, 'learning_rate': 3.99443884412647e-05, 'epoch': 0.53}

  5%|▌         | 2188/41250 [5:16:57<93:45:14,  8.64s/it][2025-04-26 09:25:35,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:25:35,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.29 | bwd_microstep: 5752.03 | bwd_inner_microstep: 5698.93 | bwd_allreduce_microstep: 53.07 | step_microstep: 17.87
[2025-04-26 09:25:35,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.29 | bwd: 5752.04 | bwd_inner: 5698.93 | bwd_allreduce: 53.08 | step: 17.88

  5%|▌         | 2189/41250 [5:17:06<93:51:51,  8.65s/it]
                                                         
{'loss': 0.1721, 'grad_norm': 1.4085359573364258, 'learning_rate': 3.994427135710958e-05, 'epoch': 0.53}

  5%|▌         | 2189/41250 [5:17:06<93:51:51,  8.65s/it][2025-04-26 09:25:43,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:25:43,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.02 | bwd_microstep: 5708.21 | bwd_inner_microstep: 5695.58 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.61
[2025-04-26 09:25:43,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.02 | bwd: 5708.22 | bwd_inner: 5695.58 | bwd_allreduce: 12.60 | step: 18.61

  5%|▌         | 2190/41250 [5:17:15<93:49:30,  8.65s/it]
                                                         
{'loss': 0.2487, 'grad_norm': 1.5322911739349365, 'learning_rate': 3.9944154150001956e-05, 'epoch': 0.53}

  5%|▌         | 2190/41250 [5:17:15<93:49:30,  8.65s/it][2025-04-26 09:25:52,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:25:52,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.47 | bwd_microstep: 5885.14 | bwd_inner_microstep: 5872.49 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.80
[2025-04-26 09:25:52,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.47 | bwd: 5885.15 | bwd_inner: 5872.49 | bwd_allreduce: 12.62 | step: 17.80

  5%|▌         | 2191/41250 [5:17:24<94:39:18,  8.72s/it]
                                                         
{'loss': 0.1483, 'grad_norm': 1.473791241645813, 'learning_rate': 3.994403681994254e-05, 'epoch': 0.53}

  5%|▌         | 2191/41250 [5:17:24<94:39:18,  8.72s/it][2025-04-26 09:26:01,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:26:01,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.72 | bwd_microstep: 5781.08 | bwd_inner_microstep: 5768.26 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.54
[2025-04-26 09:26:01,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.72 | bwd: 5781.09 | bwd_inner: 5768.26 | bwd_allreduce: 12.79 | step: 18.54

  5%|▌         | 2192/41250 [5:17:32<94:46:31,  8.74s/it]
                                                         
{'loss': 0.1155, 'grad_norm': 1.078358769416809, 'learning_rate': 3.994391936693205e-05, 'epoch': 0.53}

  5%|▌         | 2192/41250 [5:17:32<94:46:31,  8.74s/it][2025-04-26 09:26:10,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:26:10,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5936.94 | bwd_inner_microstep: 5698.54 | bwd_allreduce_microstep: 238.36 | step_microstep: 18.33
[2025-04-26 09:26:10,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5936.95 | bwd_inner: 5698.54 | bwd_allreduce: 238.38 | step: 18.33

  5%|▌         | 2193/41250 [5:17:41<95:12:32,  8.78s/it]
                                                         
{'loss': 0.2015, 'grad_norm': 2.7610554695129395, 'learning_rate': 3.994380179097123e-05, 'epoch': 0.53}

  5%|▌         | 2193/41250 [5:17:41<95:12:32,  8.78s/it][2025-04-26 09:26:19,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:26:19,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.24 | bwd_microstep: 5764.12 | bwd_inner_microstep: 5694.14 | bwd_allreduce_microstep: 69.94 | step_microstep: 18.31
[2025-04-26 09:26:19,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.24 | bwd: 5764.14 | bwd_inner: 5694.14 | bwd_allreduce: 69.95 | step: 18.32

  5%|▌         | 2194/41250 [5:17:50<94:55:03,  8.75s/it]
                                                         
{'loss': 0.1723, 'grad_norm': 2.347090721130371, 'learning_rate': 3.99436840920608e-05, 'epoch': 0.53}

  5%|▌         | 2194/41250 [5:17:50<94:55:03,  8.75s/it][2025-04-26 09:26:27,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 09:26:27,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.44 | bwd_microstep: 5752.37 | bwd_inner_microstep: 5656.71 | bwd_allreduce_microstep: 95.61 | step_microstep: 18.89
[2025-04-26 09:26:27,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.44 | bwd: 5752.38 | bwd_inner: 5656.71 | bwd_allreduce: 95.63 | step: 18.90

  5%|▌         | 2195/41250 [5:17:59<94:41:10,  8.73s/it]
                                                         
{'loss': 0.1255, 'grad_norm': 1.915995478630066, 'learning_rate': 3.994356627020147e-05, 'epoch': 0.53}

  5%|▌         | 2195/41250 [5:17:59<94:41:10,  8.73s/it][2025-04-26 09:26:36,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:26:36,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.79 | bwd_microstep: 5726.73 | bwd_inner_microstep: 5641.25 | bwd_allreduce_microstep: 85.43 | step_microstep: 18.66
[2025-04-26 09:26:36,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.79 | bwd: 5726.74 | bwd_inner: 5641.25 | bwd_allreduce: 85.45 | step: 18.66

  5%|▌         | 2196/41250 [5:18:07<94:22:45,  8.70s/it]
                                                         
{'loss': 0.168, 'grad_norm': 1.8026440143585205, 'learning_rate': 3.994344832539398e-05, 'epoch': 0.53}

  5%|▌         | 2196/41250 [5:18:07<94:22:45,  8.70s/it][2025-04-26 09:26:45,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:26:45,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.51 | bwd_microstep: 5727.89 | bwd_inner_microstep: 5644.03 | bwd_allreduce_microstep: 83.82 | step_microstep: 18.27
[2025-04-26 09:26:45,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.51 | bwd: 5727.90 | bwd_inner: 5644.02 | bwd_allreduce: 83.84 | step: 18.27

  5%|▌         | 2197/41250 [5:18:16<94:09:12,  8.68s/it]
                                                         
{'loss': 0.2727, 'grad_norm': 2.8241524696350098, 'learning_rate': 3.994333025763905e-05, 'epoch': 0.53}

  5%|▌         | 2197/41250 [5:18:16<94:09:12,  8.68s/it][2025-04-26 09:26:53,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:26:53,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.68 | bwd_microstep: 5753.51 | bwd_inner_microstep: 5706.22 | bwd_allreduce_microstep: 47.25 | step_microstep: 17.69
[2025-04-26 09:26:53,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.68 | bwd: 5753.53 | bwd_inner: 5706.22 | bwd_allreduce: 47.26 | step: 17.70

  5%|▌         | 2198/41250 [5:18:24<94:07:39,  8.68s/it]
                                                         
{'loss': 0.1391, 'grad_norm': 1.8742077350616455, 'learning_rate': 3.994321206693742e-05, 'epoch': 0.53}

  5%|▌         | 2198/41250 [5:18:24<94:07:39,  8.68s/it][2025-04-26 09:27:02,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:27:02,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.69 | bwd_microstep: 5718.08 | bwd_inner_microstep: 5655.04 | bwd_allreduce_microstep: 62.99 | step_microstep: 18.41
[2025-04-26 09:27:02,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.69 | bwd: 5718.09 | bwd_inner: 5655.04 | bwd_allreduce: 63.01 | step: 18.41

  5%|▌         | 2199/41250 [5:18:33<93:59:19,  8.66s/it]
                                                         
{'loss': 0.1733, 'grad_norm': 2.511829137802124, 'learning_rate': 3.994309375328981e-05, 'epoch': 0.53}

  5%|▌         | 2199/41250 [5:18:33<93:59:19,  8.66s/it][2025-04-26 09:27:11,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:27:11,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5789.63 | bwd_inner_microstep: 5640.75 | bwd_allreduce_microstep: 148.84 | step_microstep: 17.93
[2025-04-26 09:27:11,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5789.64 | bwd_inner: 5640.75 | bwd_allreduce: 148.85 | step: 17.93

  5%|▌         | 2200/41250 [5:18:42<94:06:26,  8.68s/it]
                                                         
{'loss': 0.1242, 'grad_norm': 1.2353702783584595, 'learning_rate': 3.9942975316696944e-05, 'epoch': 0.53}

  5%|▌         | 2200/41250 [5:18:42<94:06:26,  8.68s/it][2025-04-26 09:27:19,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:27:19,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.36 | bwd_microstep: 5771.37 | bwd_inner_microstep: 5648.91 | bwd_allreduce_microstep: 122.41 | step_microstep: 18.45
[2025-04-26 09:27:19,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.36 | bwd: 5771.38 | bwd_inner: 5648.91 | bwd_allreduce: 122.42 | step: 18.46

  5%|▌         | 2201/41250 [5:18:51<94:07:01,  8.68s/it]
                                                         
{'loss': 0.2048, 'grad_norm': 1.566602349281311, 'learning_rate': 3.994285675715956e-05, 'epoch': 0.53}

  5%|▌         | 2201/41250 [5:18:51<94:07:01,  8.68s/it][2025-04-26 09:27:28,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 09:27:28,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.45 | bwd_microstep: 5777.86 | bwd_inner_microstep: 5706.27 | bwd_allreduce_microstep: 71.54 | step_microstep: 18.52
[2025-04-26 09:27:28,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.45 | bwd: 5777.87 | bwd_inner: 5706.27 | bwd_allreduce: 71.56 | step: 18.52

  5%|▌         | 2202/41250 [5:18:59<94:14:17,  8.69s/it]
                                                         
{'loss': 0.257, 'grad_norm': 4.2155680656433105, 'learning_rate': 3.9942738074678386e-05, 'epoch': 0.53}

  5%|▌         | 2202/41250 [5:18:59<94:14:17,  8.69s/it][2025-04-26 09:27:37,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:27:37,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.08 | bwd_microstep: 5712.55 | bwd_inner_microstep: 5699.44 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.48
[2025-04-26 09:27:37,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.08 | bwd: 5712.56 | bwd_inner: 5699.44 | bwd_allreduce: 13.08 | step: 18.48

  5%|▌         | 2203/41250 [5:19:08<94:05:30,  8.67s/it]
                                                         
{'loss': 0.2082, 'grad_norm': 1.3355356454849243, 'learning_rate': 3.9942619269254154e-05, 'epoch': 0.53}

  5%|▌         | 2203/41250 [5:19:08<94:05:30,  8.67s/it][2025-04-26 09:27:45,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 09:27:45,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.77 | bwd_microstep: 5751.67 | bwd_inner_microstep: 5688.97 | bwd_allreduce_microstep: 62.66 | step_microstep: 18.95
[2025-04-26 09:27:45,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.76 | bwd: 5751.68 | bwd_inner: 5688.97 | bwd_allreduce: 62.67 | step: 18.95

  5%|▌         | 2204/41250 [5:19:17<94:10:32,  8.68s/it]
                                                         
{'loss': 0.1324, 'grad_norm': 2.178396463394165, 'learning_rate': 3.9942500340887596e-05, 'epoch': 0.53}

  5%|▌         | 2204/41250 [5:19:17<94:10:32,  8.68s/it][2025-04-26 09:27:54,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:27:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.11 | bwd_microstep: 5744.30 | bwd_inner_microstep: 5716.37 | bwd_allreduce_microstep: 27.89 | step_microstep: 18.45
[2025-04-26 09:27:54,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.11 | bwd: 5744.31 | bwd_inner: 5716.37 | bwd_allreduce: 27.91 | step: 18.46

  5%|▌         | 2205/41250 [5:19:25<94:10:02,  8.68s/it]
                                                         
{'loss': 0.2354, 'grad_norm': 2.845848321914673, 'learning_rate': 3.994238128957944e-05, 'epoch': 0.53}

  5%|▌         | 2205/41250 [5:19:25<94:10:02,  8.68s/it][2025-04-26 09:28:03,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:28:03,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.23 | bwd_microstep: 5786.89 | bwd_inner_microstep: 5656.06 | bwd_allreduce_microstep: 130.79 | step_microstep: 18.45
[2025-04-26 09:28:03,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.23 | bwd: 5786.90 | bwd_inner: 5656.06 | bwd_allreduce: 130.80 | step: 18.45

  5%|▌         | 2206/41250 [5:19:34<94:12:56,  8.69s/it]
                                                         
{'loss': 0.1167, 'grad_norm': 1.0870307683944702, 'learning_rate': 3.994226211533044e-05, 'epoch': 0.53}

  5%|▌         | 2206/41250 [5:19:34<94:12:56,  8.69s/it][2025-04-26 09:28:11,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:28:11,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.64 | bwd_microstep: 5710.66 | bwd_inner_microstep: 5648.28 | bwd_allreduce_microstep: 62.35 | step_microstep: 18.09
[2025-04-26 09:28:11,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.64 | bwd: 5710.67 | bwd_inner: 5648.27 | bwd_allreduce: 62.36 | step: 18.09

  5%|▌         | 2207/41250 [5:19:43<94:00:01,  8.67s/it]
                                                         
{'loss': 0.3129, 'grad_norm': 2.2568132877349854, 'learning_rate': 3.994214281814131e-05, 'epoch': 0.54}

  5%|▌         | 2207/41250 [5:19:43<94:00:01,  8.67s/it][2025-04-26 09:28:20,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:28:20,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2882.67 | bwd_microstep: 5773.96 | bwd_inner_microstep: 5761.22 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.57
[2025-04-26 09:28:20,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2882.67 | bwd: 5773.97 | bwd_inner: 5761.22 | bwd_allreduce: 12.71 | step: 18.57

  5%|▌         | 2208/41250 [5:19:51<94:14:28,  8.69s/it]
                                                         
{'loss': 0.1914, 'grad_norm': 2.12096905708313, 'learning_rate': 3.994202339801279e-05, 'epoch': 0.54}

  5%|▌         | 2208/41250 [5:19:51<94:14:28,  8.69s/it][2025-04-26 09:28:29,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:28:29,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.31 | bwd_microstep: 5794.21 | bwd_inner_microstep: 5648.36 | bwd_allreduce_microstep: 145.80 | step_microstep: 18.64
[2025-04-26 09:28:29,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.31 | bwd: 5794.23 | bwd_inner: 5648.36 | bwd_allreduce: 145.82 | step: 18.65

  5%|▌         | 2209/41250 [5:20:00<94:16:51,  8.69s/it]
                                                         
{'loss': 0.2604, 'grad_norm': 1.981976866722107, 'learning_rate': 3.994190385494562e-05, 'epoch': 0.54}

  5%|▌         | 2209/41250 [5:20:00<94:16:51,  8.69s/it][2025-04-26 09:28:37,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:28:37,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.38 | bwd_microstep: 5703.42 | bwd_inner_microstep: 5651.10 | bwd_allreduce_microstep: 52.27 | step_microstep: 18.62
[2025-04-26 09:28:37,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.38 | bwd: 5703.43 | bwd_inner: 5651.10 | bwd_allreduce: 52.29 | step: 18.63

  5%|▌         | 2210/41250 [5:20:09<94:02:22,  8.67s/it]
                                                         
{'loss': 0.046, 'grad_norm': 0.6478334665298462, 'learning_rate': 3.994178418894053e-05, 'epoch': 0.54}

  5%|▌         | 2210/41250 [5:20:09<94:02:22,  8.67s/it][2025-04-26 09:28:46,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.13 | optimizer_step: 1.17
[2025-04-26 09:28:46,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.44 | bwd_microstep: 5781.77 | bwd_inner_microstep: 5656.34 | bwd_allreduce_microstep: 125.37 | step_microstep: 19.72
[2025-04-26 09:28:46,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.44 | bwd: 5781.78 | bwd_inner: 5656.34 | bwd_allreduce: 125.40 | step: 19.73

  5%|▌         | 2211/41250 [5:20:17<94:06:51,  8.68s/it]
                                                         
{'loss': 0.1915, 'grad_norm': 2.2052698135375977, 'learning_rate': 3.994166439999827e-05, 'epoch': 0.54}

  5%|▌         | 2211/41250 [5:20:17<94:06:51,  8.68s/it][2025-04-26 09:28:55,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:28:55,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5708.56 | bwd_inner_microstep: 5642.46 | bwd_allreduce_microstep: 66.05 | step_microstep: 18.70
[2025-04-26 09:28:55,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5708.57 | bwd_inner: 5642.46 | bwd_allreduce: 66.06 | step: 18.70

  5%|▌         | 2212/41250 [5:20:26<93:54:20,  8.66s/it]
                                                         
{'loss': 0.3685, 'grad_norm': 2.42421817779541, 'learning_rate': 3.9941544488119565e-05, 'epoch': 0.54}

  5%|▌         | 2212/41250 [5:20:26<93:54:20,  8.66s/it][2025-04-26 09:29:03,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.92
[2025-04-26 09:29:03,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.95 | bwd_microstep: 5756.72 | bwd_inner_microstep: 5679.56 | bwd_allreduce_microstep: 77.11 | step_microstep: 19.13
[2025-04-26 09:29:03,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.95 | bwd: 5756.73 | bwd_inner: 5679.56 | bwd_allreduce: 77.13 | step: 19.13

  5%|▌         | 2213/41250 [5:20:35<93:58:56,  8.67s/it]
                                                         
{'loss': 0.255, 'grad_norm': 2.256356954574585, 'learning_rate': 3.994142445330517e-05, 'epoch': 0.54}

  5%|▌         | 2213/41250 [5:20:35<93:58:56,  8.67s/it][2025-04-26 09:29:12,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.47 | optimizer_step: 1.04
[2025-04-26 09:29:12,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5784.07 | bwd_inner_microstep: 5646.16 | bwd_allreduce_microstep: 137.85 | step_microstep: 21.99
[2025-04-26 09:29:12,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5784.09 | bwd_inner: 5646.16 | bwd_allreduce: 137.88 | step: 21.99

  5%|▌         | 2214/41250 [5:20:43<94:05:22,  8.68s/it]
                                                         
{'loss': 0.1996, 'grad_norm': 1.313244342803955, 'learning_rate': 3.994130429555581e-05, 'epoch': 0.54}

  5%|▌         | 2214/41250 [5:20:43<94:05:22,  8.68s/it][2025-04-26 09:29:21,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 09:29:21,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.75 | bwd_microstep: 5874.98 | bwd_inner_microstep: 5679.88 | bwd_allreduce_microstep: 195.05 | step_microstep: 18.89
[2025-04-26 09:29:21,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.75 | bwd: 5874.99 | bwd_inner: 5679.88 | bwd_allreduce: 195.07 | step: 18.89

  5%|▌         | 2215/41250 [5:20:52<94:30:50,  8.72s/it]
                                                         
{'loss': 0.2115, 'grad_norm': 1.3531697988510132, 'learning_rate': 3.9941184014872224e-05, 'epoch': 0.54}

  5%|▌         | 2215/41250 [5:20:52<94:30:50,  8.72s/it][2025-04-26 09:29:30,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.23 | optimizer_step: 1.06
[2025-04-26 09:29:30,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5697.75 | bwd_inner_microstep: 5683.56 | bwd_allreduce_microstep: 14.12 | step_microstep: 20.31
[2025-04-26 09:29:30,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5697.77 | bwd_inner: 5683.56 | bwd_allreduce: 14.16 | step: 20.31

  5%|▌         | 2216/41250 [5:21:01<94:11:24,  8.69s/it]
                                                         
{'loss': 0.1549, 'grad_norm': 0.8541537523269653, 'learning_rate': 3.994106361125517e-05, 'epoch': 0.54}

  5%|▌         | 2216/41250 [5:21:01<94:11:24,  8.69s/it][2025-04-26 09:29:38,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 09:29:38,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.22 | bwd_microstep: 5712.26 | bwd_inner_microstep: 5689.12 | bwd_allreduce_microstep: 23.10 | step_microstep: 18.68
[2025-04-26 09:29:38,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.22 | bwd: 5712.27 | bwd_inner: 5689.12 | bwd_allreduce: 23.11 | step: 18.68

  5%|▌         | 2217/41250 [5:21:09<94:07:24,  8.68s/it]
                                                         
{'loss': 0.1615, 'grad_norm': 1.472719430923462, 'learning_rate': 3.994094308470538e-05, 'epoch': 0.54}

  5%|▌         | 2217/41250 [5:21:09<94:07:24,  8.68s/it][2025-04-26 09:29:47,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-26 09:29:47,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5677.15 | bwd_inner_microstep: 5648.32 | bwd_allreduce_microstep: 28.77 | step_microstep: 19.21
[2025-04-26 09:29:47,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5677.16 | bwd_inner: 5648.32 | bwd_allreduce: 28.79 | step: 19.21

  5%|▌         | 2218/41250 [5:21:18<93:54:03,  8.66s/it]
                                                         
{'loss': 0.1535, 'grad_norm': 1.789149284362793, 'learning_rate': 3.9940822435223596e-05, 'epoch': 0.54}

  5%|▌         | 2218/41250 [5:21:18<93:54:03,  8.66s/it][2025-04-26 09:29:56,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.67 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:29:56,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.04 | bwd_microstep: 5844.34 | bwd_inner_microstep: 5650.18 | bwd_allreduce_microstep: 194.12 | step_microstep: 18.94
[2025-04-26 09:29:56,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.04 | bwd: 5844.36 | bwd_inner: 5650.18 | bwd_allreduce: 194.13 | step: 18.95

  5%|▌         | 2219/41250 [5:21:27<94:13:25,  8.69s/it]
                                                         
{'loss': 0.1635, 'grad_norm': 1.4226535558700562, 'learning_rate': 3.9940701662810564e-05, 'epoch': 0.54}

  5%|▌         | 2219/41250 [5:21:27<94:13:25,  8.69s/it][2025-04-26 09:30:04,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 09:30:04,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.55 | bwd_microstep: 5706.15 | bwd_inner_microstep: 5692.75 | bwd_allreduce_microstep: 13.35 | step_microstep: 18.80
[2025-04-26 09:30:04,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.55 | bwd: 5706.17 | bwd_inner: 5692.75 | bwd_allreduce: 13.37 | step: 18.80

  5%|▌         | 2220/41250 [5:21:35<94:02:09,  8.67s/it]
                                                         
{'loss': 0.27, 'grad_norm': 1.885886311531067, 'learning_rate': 3.994058076746703e-05, 'epoch': 0.54}

  5%|▌         | 2220/41250 [5:21:35<94:02:09,  8.67s/it][2025-04-26 09:30:13,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:30:13,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5701.17 | bwd_inner_microstep: 5688.41 | bwd_allreduce_microstep: 12.71 | step_microstep: 19.02
[2025-04-26 09:30:13,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.42 | bwd: 5701.18 | bwd_inner: 5688.41 | bwd_allreduce: 12.73 | step: 19.02

  5%|▌         | 2221/41250 [5:21:44<93:51:06,  8.66s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 1.7062796354293823, 'learning_rate': 3.994045974919374e-05, 'epoch': 0.54}

  5%|▌         | 2221/41250 [5:21:44<93:51:06,  8.66s/it][2025-04-26 09:30:22,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.14 | optimizer_step: 1.04
[2025-04-26 09:30:22,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.25 | bwd_microstep: 5739.46 | bwd_inner_microstep: 5682.06 | bwd_allreduce_microstep: 57.33 | step_microstep: 19.54
[2025-04-26 09:30:22,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.25 | bwd: 5739.47 | bwd_inner: 5682.06 | bwd_allreduce: 57.36 | step: 19.54

  5%|▌         | 2222/41250 [5:21:53<93:53:57,  8.66s/it]
                                                         
{'loss': 0.1612, 'grad_norm': 1.5762073993682861, 'learning_rate': 3.994033860799143e-05, 'epoch': 0.54}

  5%|▌         | 2222/41250 [5:21:53<93:53:57,  8.66s/it][2025-04-26 09:30:30,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:30:30,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.59 | bwd_microstep: 5736.25 | bwd_inner_microstep: 5643.66 | bwd_allreduce_microstep: 92.54 | step_microstep: 18.42
[2025-04-26 09:30:30,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.59 | bwd: 5736.26 | bwd_inner: 5643.66 | bwd_allreduce: 92.56 | step: 18.42

  5%|▌         | 2223/41250 [5:22:01<93:55:28,  8.66s/it]
                                                         
{'loss': 0.1922, 'grad_norm': 1.2993451356887817, 'learning_rate': 3.994021734386086e-05, 'epoch': 0.54}

  5%|▌         | 2223/41250 [5:22:01<93:55:28,  8.66s/it][2025-04-26 09:30:39,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:30:39,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.23 | bwd_microstep: 5737.31 | bwd_inner_microstep: 5666.79 | bwd_allreduce_microstep: 70.47 | step_microstep: 18.53
[2025-04-26 09:30:39,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.23 | bwd: 5737.32 | bwd_inner: 5666.79 | bwd_allreduce: 70.50 | step: 18.53

  5%|▌         | 2224/41250 [5:22:10<93:56:28,  8.67s/it]
                                                         
{'loss': 0.2302, 'grad_norm': 1.5452443361282349, 'learning_rate': 3.994009595680277e-05, 'epoch': 0.54}

  5%|▌         | 2224/41250 [5:22:10<93:56:28,  8.67s/it][2025-04-26 09:30:48,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 09:30:48,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.15 | bwd_microstep: 5804.22 | bwd_inner_microstep: 5678.69 | bwd_allreduce_microstep: 125.49 | step_microstep: 18.07
[2025-04-26 09:30:48,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.15 | bwd: 5804.23 | bwd_inner: 5678.69 | bwd_allreduce: 125.50 | step: 18.08

  5%|▌         | 2225/41250 [5:22:19<94:05:33,  8.68s/it]
                                                         
{'loss': 0.1312, 'grad_norm': 0.9195554256439209, 'learning_rate': 3.993997444681791e-05, 'epoch': 0.54}

  5%|▌         | 2225/41250 [5:22:19<94:05:33,  8.68s/it][2025-04-26 09:30:56,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 09:30:56,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.89 | bwd_microstep: 5740.82 | bwd_inner_microstep: 5630.00 | bwd_allreduce_microstep: 110.78 | step_microstep: 18.23
[2025-04-26 09:30:56,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.89 | bwd: 5740.84 | bwd_inner: 5630.00 | bwd_allreduce: 110.80 | step: 18.24

  5%|▌         | 2226/41250 [5:22:27<93:58:29,  8.67s/it]
                                                         
{'loss': 0.1032, 'grad_norm': 0.9788561463356018, 'learning_rate': 3.9939852813907036e-05, 'epoch': 0.54}

  5%|▌         | 2226/41250 [5:22:27<93:58:29,  8.67s/it][2025-04-26 09:31:05,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:31:05,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.03 | bwd_microstep: 5705.07 | bwd_inner_microstep: 5692.23 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.06
[2025-04-26 09:31:05,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.04 | bwd: 5705.08 | bwd_inner: 5692.23 | bwd_allreduce: 12.81 | step: 18.06

  5%|▌         | 2227/41250 [5:22:36<93:53:44,  8.66s/it]
                                                         
{'loss': 0.2156, 'grad_norm': 1.712099552154541, 'learning_rate': 3.993973105807088e-05, 'epoch': 0.54}

  5%|▌         | 2227/41250 [5:22:36<93:53:44,  8.66s/it][2025-04-26 09:31:14,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:31:14,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5710.49 | bwd_inner_microstep: 5697.78 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.34
[2025-04-26 09:31:14,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5710.50 | bwd_inner: 5697.78 | bwd_allreduce: 12.68 | step: 18.35

  5%|▌         | 2228/41250 [5:22:45<93:50:38,  8.66s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.0981371402740479, 'learning_rate': 3.993960917931021e-05, 'epoch': 0.54}

  5%|▌         | 2228/41250 [5:22:45<93:50:38,  8.66s/it][2025-04-26 09:31:22,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 09:31:22,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.10 | bwd_microstep: 5699.66 | bwd_inner_microstep: 5659.39 | bwd_allreduce_microstep: 40.22 | step_microstep: 18.99
[2025-04-26 09:31:22,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.10 | bwd: 5699.67 | bwd_inner: 5659.39 | bwd_allreduce: 40.24 | step: 19.00

  5%|▌         | 2229/41250 [5:22:53<93:41:04,  8.64s/it]
                                                         
{'loss': 0.1926, 'grad_norm': 1.2291415929794312, 'learning_rate': 3.9939487177625774e-05, 'epoch': 0.54}

  5%|▌         | 2229/41250 [5:22:53<93:41:04,  8.64s/it][2025-04-26 09:31:31,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:31:31,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.53 | bwd_microstep: 5773.33 | bwd_inner_microstep: 5647.17 | bwd_allreduce_microstep: 126.11 | step_microstep: 18.63
[2025-04-26 09:31:31,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.53 | bwd: 5773.34 | bwd_inner: 5647.17 | bwd_allreduce: 126.13 | step: 18.63

  5%|▌         | 2230/41250 [5:23:02<93:47:14,  8.65s/it]
                                                         
{'loss': 0.127, 'grad_norm': 2.09724760055542, 'learning_rate': 3.9939365053018316e-05, 'epoch': 0.54}

  5%|▌         | 2230/41250 [5:23:02<93:47:14,  8.65s/it][2025-04-26 09:31:39,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:31:39,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.75 | bwd_microstep: 5717.24 | bwd_inner_microstep: 5704.35 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.40
[2025-04-26 09:31:39,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.75 | bwd: 5717.25 | bwd_inner: 5704.35 | bwd_allreduce: 12.86 | step: 18.41

  5%|▌         | 2231/41250 [5:23:11<93:47:48,  8.65s/it]
                                                         
{'loss': 0.2016, 'grad_norm': 1.6875494718551636, 'learning_rate': 3.99392428054886e-05, 'epoch': 0.54}

  5%|▌         | 2231/41250 [5:23:11<93:47:48,  8.65s/it][2025-04-26 09:31:48,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:31:48,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.51 | bwd_microstep: 5706.49 | bwd_inner_microstep: 5645.72 | bwd_allreduce_microstep: 60.72 | step_microstep: 18.40
[2025-04-26 09:31:48,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.51 | bwd: 5706.50 | bwd_inner: 5645.72 | bwd_allreduce: 60.74 | step: 18.40

  5%|▌         | 2232/41250 [5:23:19<93:38:53,  8.64s/it]
                                                         
{'loss': 0.0702, 'grad_norm': 0.7136693000793457, 'learning_rate': 3.993912043503737e-05, 'epoch': 0.54}

  5%|▌         | 2232/41250 [5:23:19<93:38:53,  8.64s/it][2025-04-26 09:31:57,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:31:57,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.73 | bwd_microstep: 5699.54 | bwd_inner_microstep: 5645.71 | bwd_allreduce_microstep: 53.78 | step_microstep: 18.30
[2025-04-26 09:31:57,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.73 | bwd: 5699.55 | bwd_inner: 5645.71 | bwd_allreduce: 53.80 | step: 18.30

  5%|▌         | 2233/41250 [5:23:28<93:31:26,  8.63s/it]
                                                         
{'loss': 0.1055, 'grad_norm': 1.1403591632843018, 'learning_rate': 3.993899794166539e-05, 'epoch': 0.54}

  5%|▌         | 2233/41250 [5:23:28<93:31:26,  8.63s/it][2025-04-26 09:32:05,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:32:05,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.93 | bwd_microstep: 5712.22 | bwd_inner_microstep: 5699.34 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.37
[2025-04-26 09:32:05,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.93 | bwd: 5712.23 | bwd_inner: 5699.34 | bwd_allreduce: 12.86 | step: 18.37

  5%|▌         | 2234/41250 [5:23:37<93:32:53,  8.63s/it]
                                                         
{'loss': 0.1533, 'grad_norm': 1.6714470386505127, 'learning_rate': 3.99388753253734e-05, 'epoch': 0.54}

  5%|▌         | 2234/41250 [5:23:37<93:32:53,  8.63s/it][2025-04-26 09:32:14,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 09:32:14,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.50 | bwd_microstep: 5889.83 | bwd_inner_microstep: 5657.08 | bwd_allreduce_microstep: 232.71 | step_microstep: 18.95
[2025-04-26 09:32:14,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.50 | bwd: 5889.85 | bwd_inner: 5657.08 | bwd_allreduce: 232.73 | step: 18.96

  5%|▌         | 2235/41250 [5:23:45<94:07:46,  8.69s/it]
                                                         
{'loss': 0.333, 'grad_norm': 2.2066478729248047, 'learning_rate': 3.993875258616217e-05, 'epoch': 0.54}

  5%|▌         | 2235/41250 [5:23:45<94:07:46,  8.69s/it][2025-04-26 09:32:23,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:32:23,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.15 | bwd_microstep: 5763.65 | bwd_inner_microstep: 5643.76 | bwd_allreduce_microstep: 119.84 | step_microstep: 18.68
[2025-04-26 09:32:23,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.15 | bwd: 5763.67 | bwd_inner: 5643.76 | bwd_allreduce: 119.86 | step: 18.68

  5%|▌         | 2236/41250 [5:23:54<94:07:20,  8.69s/it]
                                                         
{'loss': 0.4958, 'grad_norm': 2.6011524200439453, 'learning_rate': 3.993862972403246e-05, 'epoch': 0.54}

  5%|▌         | 2236/41250 [5:23:54<94:07:20,  8.69s/it][2025-04-26 09:32:32,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:32:32,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.66 | bwd_microstep: 5800.30 | bwd_inner_microstep: 5651.05 | bwd_allreduce_microstep: 149.20 | step_microstep: 18.51
[2025-04-26 09:32:32,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.66 | bwd: 5800.32 | bwd_inner: 5651.05 | bwd_allreduce: 149.23 | step: 18.51

  5%|▌         | 2237/41250 [5:24:03<94:11:54,  8.69s/it]
                                                         
{'loss': 0.1668, 'grad_norm': 2.0654871463775635, 'learning_rate': 3.9938506738985005e-05, 'epoch': 0.54}

  5%|▌         | 2237/41250 [5:24:03<94:11:54,  8.69s/it][2025-04-26 09:32:40,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:32:40,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5737.87 | bwd_inner_microstep: 5707.81 | bwd_allreduce_microstep: 30.02 | step_microstep: 18.32
[2025-04-26 09:32:40,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5737.88 | bwd_inner: 5707.81 | bwd_allreduce: 30.03 | step: 18.32

  5%|▌         | 2238/41250 [5:24:11<94:07:52,  8.69s/it]
                                                         
{'loss': 0.0577, 'grad_norm': 1.133194923400879, 'learning_rate': 3.993838363102058e-05, 'epoch': 0.54}

  5%|▌         | 2238/41250 [5:24:11<94:07:52,  8.69s/it][2025-04-26 09:32:49,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 09:32:49,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2943.34 | bwd_microstep: 5888.52 | bwd_inner_microstep: 5875.53 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.84
[2025-04-26 09:32:49,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2943.34 | bwd: 5888.54 | bwd_inner: 5875.53 | bwd_allreduce: 12.97 | step: 18.85

  5%|▌         | 2239/41250 [5:24:20<94:50:54,  8.75s/it]
                                                         
{'loss': 0.2536, 'grad_norm': 2.0818843841552734, 'learning_rate': 3.993826040013996e-05, 'epoch': 0.54}

  5%|▌         | 2239/41250 [5:24:20<94:50:54,  8.75s/it][2025-04-26 09:32:58,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:32:58,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.77 | bwd_microstep: 5702.81 | bwd_inner_microstep: 5647.67 | bwd_allreduce_microstep: 55.08 | step_microstep: 18.40
[2025-04-26 09:32:58,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.77 | bwd: 5702.82 | bwd_inner: 5647.67 | bwd_allreduce: 55.10 | step: 18.41

  5%|▌         | 2240/41250 [5:24:29<94:25:57,  8.71s/it]
                                                         
{'loss': 0.16, 'grad_norm': 1.1200056076049805, 'learning_rate': 3.993813704634386e-05, 'epoch': 0.54}

  5%|▌         | 2240/41250 [5:24:29<94:25:57,  8.71s/it][2025-04-26 09:33:06,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 09:33:06,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.26 | bwd_microstep: 5784.67 | bwd_inner_microstep: 5709.04 | bwd_allreduce_microstep: 75.59 | step_microstep: 19.03
[2025-04-26 09:33:06,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.26 | bwd: 5784.68 | bwd_inner: 5709.04 | bwd_allreduce: 75.61 | step: 19.03

  5%|▌         | 2241/41250 [5:24:38<94:27:33,  8.72s/it]
                                                         
{'loss': 0.2261, 'grad_norm': 2.5775461196899414, 'learning_rate': 3.9938013569633084e-05, 'epoch': 0.54}

  5%|▌         | 2241/41250 [5:24:38<94:27:33,  8.72s/it][2025-04-26 09:33:15,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 09:33:15,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.47 | bwd_microstep: 5795.62 | bwd_inner_microstep: 5646.65 | bwd_allreduce_microstep: 148.93 | step_microstep: 18.48
[2025-04-26 09:33:15,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.47 | bwd: 5795.64 | bwd_inner: 5646.65 | bwd_allreduce: 148.95 | step: 18.49

  5%|▌         | 2242/41250 [5:24:46<94:24:37,  8.71s/it]
                                                         
{'loss': 0.1564, 'grad_norm': 1.5292344093322754, 'learning_rate': 3.9937889970008375e-05, 'epoch': 0.54}

  5%|▌         | 2242/41250 [5:24:46<94:24:37,  8.71s/it][2025-04-26 09:33:24,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 09:33:24,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.07 | bwd_microstep: 5943.42 | bwd_inner_microstep: 5700.62 | bwd_allreduce_microstep: 242.72 | step_microstep: 18.34
[2025-04-26 09:33:24,525] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.09 | bwd: 5943.42 | bwd_inner: 5700.61 | bwd_allreduce: 242.74 | step: 18.34

  5%|▌         | 2243/41250 [5:24:55<94:54:50,  8.76s/it]
                                                         
{'loss': 0.2056, 'grad_norm': 2.2209815979003906, 'learning_rate': 3.9937766247470484e-05, 'epoch': 0.54}

  5%|▌         | 2243/41250 [5:24:55<94:54:50,  8.76s/it][2025-04-26 09:33:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:33:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5704.79 | bwd_inner_microstep: 5692.06 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.40
[2025-04-26 09:33:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5704.80 | bwd_inner: 5692.06 | bwd_allreduce: 12.70 | step: 18.40

  5%|▌         | 2244/41250 [5:25:04<94:31:08,  8.72s/it]
                                                         
{'loss': 0.1854, 'grad_norm': 1.4229644536972046, 'learning_rate': 3.99376424020202e-05, 'epoch': 0.54}

  5%|▌         | 2244/41250 [5:25:04<94:31:08,  8.72s/it][2025-04-26 09:33:41,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:33:41,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.16 | bwd_microstep: 5789.90 | bwd_inner_microstep: 5777.19 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.36
[2025-04-26 09:33:41,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.16 | bwd: 5789.92 | bwd_inner: 5777.19 | bwd_allreduce: 12.69 | step: 18.37

  5%|▌         | 2245/41250 [5:25:13<94:37:25,  8.73s/it]
                                                         
{'loss': 0.2389, 'grad_norm': 5.796565055847168, 'learning_rate': 3.993751843365826e-05, 'epoch': 0.54}

  5%|▌         | 2245/41250 [5:25:13<94:37:25,  8.73s/it][2025-04-26 09:33:50,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 09:33:50,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.51 | bwd_microstep: 5816.30 | bwd_inner_microstep: 5661.51 | bwd_allreduce_microstep: 154.74 | step_microstep: 19.36
[2025-04-26 09:33:50,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.51 | bwd: 5816.31 | bwd_inner: 5661.51 | bwd_allreduce: 154.76 | step: 19.36

  5%|▌         | 2246/41250 [5:25:21<94:36:19,  8.73s/it]
                                                         
{'loss': 0.1702, 'grad_norm': 1.3083597421646118, 'learning_rate': 3.993739434238545e-05, 'epoch': 0.54}

  5%|▌         | 2246/41250 [5:25:21<94:36:19,  8.73s/it][2025-04-26 09:33:59,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:33:59,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.21 | bwd_microstep: 5730.85 | bwd_inner_microstep: 5666.42 | bwd_allreduce_microstep: 64.38 | step_microstep: 18.09
[2025-04-26 09:33:59,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.21 | bwd: 5730.86 | bwd_inner: 5666.42 | bwd_allreduce: 64.40 | step: 18.09

  5%|▌         | 2247/41250 [5:25:30<94:20:23,  8.71s/it]
                                                         
{'loss': 0.2304, 'grad_norm': 1.8684395551681519, 'learning_rate': 3.9937270128202515e-05, 'epoch': 0.54}

  5%|▌         | 2247/41250 [5:25:30<94:20:23,  8.71s/it][2025-04-26 09:34:07,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:34:07,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.42 | bwd_microstep: 5769.48 | bwd_inner_microstep: 5656.57 | bwd_allreduce_microstep: 112.86 | step_microstep: 18.20
[2025-04-26 09:34:07,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.42 | bwd: 5769.49 | bwd_inner: 5656.57 | bwd_allreduce: 112.88 | step: 18.21

  5%|▌         | 2248/41250 [5:25:39<94:17:53,  8.70s/it]
                                                         
{'loss': 0.1944, 'grad_norm': 2.658230781555176, 'learning_rate': 3.9937145791110244e-05, 'epoch': 0.54}

  5%|▌         | 2248/41250 [5:25:39<94:17:53,  8.70s/it][2025-04-26 09:34:16,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:34:16,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5727.47 | bwd_inner_microstep: 5645.78 | bwd_allreduce_microstep: 81.64 | step_microstep: 18.40
[2025-04-26 09:34:16,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5727.48 | bwd_inner: 5645.78 | bwd_allreduce: 81.66 | step: 18.40

  5%|▌         | 2249/41250 [5:25:47<94:05:33,  8.69s/it]
                                                         
{'loss': 0.0692, 'grad_norm': 1.3101227283477783, 'learning_rate': 3.993702133110938e-05, 'epoch': 0.55}

  5%|▌         | 2249/41250 [5:25:47<94:05:33,  8.69s/it][2025-04-26 09:34:25,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 09:34:25,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.77 | bwd_microstep: 5788.40 | bwd_inner_microstep: 5682.24 | bwd_allreduce_microstep: 106.11 | step_microstep: 18.40
[2025-04-26 09:34:25,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.77 | bwd: 5788.41 | bwd_inner: 5682.24 | bwd_allreduce: 106.13 | step: 18.40

  5%|▌         | 2250/41250 [5:25:56<94:11:32,  8.69s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.7041850090026855, 'learning_rate': 3.9936896748200705e-05, 'epoch': 0.55}

  5%|▌         | 2250/41250 [5:25:56<94:11:32,  8.69s/it][2025-04-26 09:34:33,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:34:33,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.91 | bwd_microstep: 5700.95 | bwd_inner_microstep: 5642.93 | bwd_allreduce_microstep: 57.97 | step_microstep: 18.80
[2025-04-26 09:34:33,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.91 | bwd: 5700.97 | bwd_inner: 5642.93 | bwd_allreduce: 57.99 | step: 18.80

  5%|▌         | 2251/41250 [5:26:05<93:55:04,  8.67s/it]
                                                         
{'loss': 0.2891, 'grad_norm': 1.7472177743911743, 'learning_rate': 3.9936772042384983e-05, 'epoch': 0.55}

  5%|▌         | 2251/41250 [5:26:05<93:55:04,  8.67s/it][2025-04-26 09:34:42,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 09:34:42,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.38 | bwd_microstep: 5778.03 | bwd_inner_microstep: 5644.52 | bwd_allreduce_microstep: 133.46 | step_microstep: 18.87
[2025-04-26 09:34:42,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.38 | bwd: 5778.04 | bwd_inner: 5644.52 | bwd_allreduce: 133.48 | step: 18.87

  5%|▌         | 2252/41250 [5:26:13<93:56:22,  8.67s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 1.2522485256195068, 'learning_rate': 3.9936647213662985e-05, 'epoch': 0.55}

  5%|▌         | 2252/41250 [5:26:13<93:56:22,  8.67s/it][2025-04-26 09:34:51,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:34:51,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.52 | bwd_microstep: 5723.26 | bwd_inner_microstep: 5659.18 | bwd_allreduce_microstep: 64.04 | step_microstep: 18.23
[2025-04-26 09:34:51,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.52 | bwd: 5723.28 | bwd_inner: 5659.18 | bwd_allreduce: 64.06 | step: 18.23

  5%|▌         | 2253/41250 [5:26:22<93:48:46,  8.66s/it]
                                                         
{'loss': 0.2074, 'grad_norm': 1.2391897439956665, 'learning_rate': 3.9936522262035476e-05, 'epoch': 0.55}

  5%|▌         | 2253/41250 [5:26:22<93:48:46,  8.66s/it][2025-04-26 09:35:00,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:35:00,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.80 | bwd_microstep: 5868.42 | bwd_inner_microstep: 5653.07 | bwd_allreduce_microstep: 215.31 | step_microstep: 18.23
[2025-04-26 09:35:00,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.80 | bwd: 5868.43 | bwd_inner: 5653.07 | bwd_allreduce: 215.32 | step: 18.23

  5%|▌         | 2254/41250 [5:26:31<94:13:19,  8.70s/it]
                                                         
{'loss': 0.0887, 'grad_norm': 0.7454801201820374, 'learning_rate': 3.993639718750323e-05, 'epoch': 0.55}

  5%|▌         | 2254/41250 [5:26:31<94:13:19,  8.70s/it][2025-04-26 09:35:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:35:08,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5769.32 | bwd_inner_microstep: 5689.99 | bwd_allreduce_microstep: 79.29 | step_microstep: 18.37
[2025-04-26 09:35:08,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.78 | bwd: 5769.33 | bwd_inner: 5689.99 | bwd_allreduce: 79.31 | step: 18.37

  5%|▌         | 2255/41250 [5:26:39<94:14:49,  8.70s/it]
                                                         
{'loss': 0.1015, 'grad_norm': 1.2999686002731323, 'learning_rate': 3.993627199006701e-05, 'epoch': 0.55}

  5%|▌         | 2255/41250 [5:26:39<94:14:49,  8.70s/it][2025-04-26 09:35:17,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 09:35:17,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.55 | bwd_microstep: 5751.64 | bwd_inner_microstep: 5687.70 | bwd_allreduce_microstep: 63.89 | step_microstep: 18.66
[2025-04-26 09:35:17,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.55 | bwd: 5751.65 | bwd_inner: 5687.70 | bwd_allreduce: 63.91 | step: 18.67

  5%|▌         | 2256/41250 [5:26:48<94:12:20,  8.70s/it]
                                                         
{'loss': 0.2187, 'grad_norm': 1.6236017942428589, 'learning_rate': 3.99361466697276e-05, 'epoch': 0.55}

  5%|▌         | 2256/41250 [5:26:48<94:12:20,  8.70s/it][2025-04-26 09:35:26,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:35:26,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.40 | bwd_microstep: 5764.64 | bwd_inner_microstep: 5694.64 | bwd_allreduce_microstep: 69.95 | step_microstep: 18.26
[2025-04-26 09:35:26,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.40 | bwd: 5764.65 | bwd_inner: 5694.64 | bwd_allreduce: 69.97 | step: 18.26

  5%|▌         | 2257/41250 [5:26:57<94:10:32,  8.69s/it]
                                                         
{'loss': 0.0653, 'grad_norm': 1.0204347372055054, 'learning_rate': 3.993602122648577e-05, 'epoch': 0.55}

  5%|▌         | 2257/41250 [5:26:57<94:10:32,  8.69s/it][2025-04-26 09:35:34,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:35:34,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.08 | bwd_microstep: 5687.95 | bwd_inner_microstep: 5648.24 | bwd_allreduce_microstep: 39.66 | step_microstep: 18.07
[2025-04-26 09:35:34,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.08 | bwd: 5687.96 | bwd_inner: 5648.24 | bwd_allreduce: 39.68 | step: 18.07

  5%|▌         | 2258/41250 [5:27:05<93:54:33,  8.67s/it]
                                                         
{'loss': 0.1856, 'grad_norm': 1.7092043161392212, 'learning_rate': 3.993589566034228e-05, 'epoch': 0.55}

  5%|▌         | 2258/41250 [5:27:05<93:54:33,  8.67s/it][2025-04-26 09:35:43,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.24 | optimizer_gradients: 1.17 | optimizer_step: 0.90
[2025-04-26 09:35:43,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5713.54 | bwd_inner_microstep: 5649.97 | bwd_allreduce_microstep: 63.53 | step_microstep: 20.64
[2025-04-26 09:35:43,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5713.55 | bwd_inner: 5649.97 | bwd_allreduce: 63.54 | step: 20.64

  5%|▌         | 2259/41250 [5:27:14<93:47:28,  8.66s/it]
                                                         
{'loss': 0.441, 'grad_norm': 2.879664182662964, 'learning_rate': 3.9935769971297925e-05, 'epoch': 0.55}

  5%|▌         | 2259/41250 [5:27:14<93:47:28,  8.66s/it][2025-04-26 09:35:52,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:35:52,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.37 | bwd_microstep: 5764.91 | bwd_inner_microstep: 5656.04 | bwd_allreduce_microstep: 108.82 | step_microstep: 18.24
[2025-04-26 09:35:52,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.37 | bwd: 5764.92 | bwd_inner: 5656.04 | bwd_allreduce: 108.84 | step: 18.25

  5%|▌         | 2260/41250 [5:27:23<93:52:20,  8.67s/it]
                                                         
{'loss': 0.1434, 'grad_norm': 1.354970932006836, 'learning_rate': 3.9935644159353465e-05, 'epoch': 0.55}

  5%|▌         | 2260/41250 [5:27:23<93:52:20,  8.67s/it][2025-04-26 09:36:00,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:36:00,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.63 | bwd_microstep: 5769.49 | bwd_inner_microstep: 5652.33 | bwd_allreduce_microstep: 117.11 | step_microstep: 18.32
[2025-04-26 09:36:00,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.63 | bwd: 5769.50 | bwd_inner: 5652.33 | bwd_allreduce: 117.13 | step: 18.32

  5%|▌         | 2261/41250 [5:27:31<93:53:09,  8.67s/it]
                                                         
{'loss': 0.0245, 'grad_norm': 0.2836485207080841, 'learning_rate': 3.993551822450968e-05, 'epoch': 0.55}

  5%|▌         | 2261/41250 [5:27:31<93:53:09,  8.67s/it][2025-04-26 09:36:09,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.82 | optimizer_gradients: 1.04 | optimizer_step: 1.17
[2025-04-26 09:36:09,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5776.88 | bwd_inner_microstep: 5697.45 | bwd_allreduce_microstep: 79.37 | step_microstep: 19.54
[2025-04-26 09:36:09,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5776.89 | bwd_inner: 5697.45 | bwd_allreduce: 79.40 | step: 19.55

  5%|▌         | 2262/41250 [5:27:40<94:00:42,  8.68s/it]
                                                         
{'loss': 0.2728, 'grad_norm': 1.523844838142395, 'learning_rate': 3.9935392166767345e-05, 'epoch': 0.55}

  5%|▌         | 2262/41250 [5:27:40<94:00:42,  8.68s/it][2025-04-26 09:36:18,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.19 | optimizer_step: 0.93
[2025-04-26 09:36:18,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.05 | bwd_microstep: 5825.54 | bwd_inner_microstep: 5770.12 | bwd_allreduce_microstep: 55.37 | step_microstep: 19.52
[2025-04-26 09:36:18,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.05 | bwd: 5825.55 | bwd_inner: 5770.11 | bwd_allreduce: 55.39 | step: 19.52

  5%|▌         | 2263/41250 [5:27:49<94:26:25,  8.72s/it]
                                                         
{'loss': 0.2037, 'grad_norm': 1.6429111957550049, 'learning_rate': 3.9935265986127244e-05, 'epoch': 0.55}

  5%|▌         | 2263/41250 [5:27:49<94:26:25,  8.72s/it][2025-04-26 09:36:26,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.65 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 09:36:26,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.73 | bwd_microstep: 5753.44 | bwd_inner_microstep: 5686.35 | bwd_allreduce_microstep: 67.05 | step_microstep: 18.92
[2025-04-26 09:36:26,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.73 | bwd: 5753.45 | bwd_inner: 5686.35 | bwd_allreduce: 67.06 | step: 18.93

  5%|▌         | 2264/41250 [5:27:58<94:18:25,  8.71s/it]
                                                         
{'loss': 0.1293, 'grad_norm': 1.0449062585830688, 'learning_rate': 3.993513968259015e-05, 'epoch': 0.55}

  5%|▌         | 2264/41250 [5:27:58<94:18:25,  8.71s/it][2025-04-26 09:36:35,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:36:35,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.99 | bwd_microstep: 5769.49 | bwd_inner_microstep: 5651.51 | bwd_allreduce_microstep: 117.93 | step_microstep: 18.63
[2025-04-26 09:36:35,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.99 | bwd: 5769.50 | bwd_inner: 5651.51 | bwd_allreduce: 117.95 | step: 18.64

  5%|▌         | 2265/41250 [5:28:06<94:12:20,  8.70s/it]
                                                         
{'loss': 0.2535, 'grad_norm': 1.707515001296997, 'learning_rate': 3.993501325615683e-05, 'epoch': 0.55}

  5%|▌         | 2265/41250 [5:28:06<94:12:20,  8.70s/it][2025-04-26 09:36:44,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:36:44,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.29 | bwd_microstep: 5700.68 | bwd_inner_microstep: 5687.51 | bwd_allreduce_microstep: 13.11 | step_microstep: 18.78
[2025-04-26 09:36:44,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.29 | bwd: 5700.70 | bwd_inner: 5687.51 | bwd_allreduce: 13.14 | step: 18.78

  5%|▌         | 2266/41250 [5:28:15<93:58:39,  8.68s/it]
                                                         
{'loss': 0.2436, 'grad_norm': 1.6466484069824219, 'learning_rate': 3.993488670682808e-05, 'epoch': 0.55}

  5%|▌         | 2266/41250 [5:28:15<93:58:39,  8.68s/it][2025-04-26 09:36:53,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.12 | optimizer_step: 1.04
[2025-04-26 09:36:53,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5874.85 | bwd_inner_microstep: 5692.30 | bwd_allreduce_microstep: 182.49 | step_microstep: 19.54
[2025-04-26 09:36:53,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5874.86 | bwd_inner: 5692.30 | bwd_allreduce: 182.52 | step: 19.54

  5%|▌         | 2267/41250 [5:28:24<94:23:46,  8.72s/it]
                                                         
{'loss': 0.2459, 'grad_norm': 1.6869014501571655, 'learning_rate': 3.993476003460468e-05, 'epoch': 0.55}

  5%|▌         | 2267/41250 [5:28:24<94:23:46,  8.72s/it][2025-04-26 09:37:01,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 09:37:01,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.65 | bwd_microstep: 5727.33 | bwd_inner_microstep: 5676.33 | bwd_allreduce_microstep: 50.96 | step_microstep: 18.88
[2025-04-26 09:37:01,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.65 | bwd: 5727.34 | bwd_inner: 5676.33 | bwd_allreduce: 50.98 | step: 18.88

  5%|▌         | 2268/41250 [5:28:32<94:12:05,  8.70s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 1.0124177932739258, 'learning_rate': 3.99346332394874e-05, 'epoch': 0.55}

  5%|▌         | 2268/41250 [5:28:32<94:12:05,  8.70s/it][2025-04-26 09:37:10,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 09:37:10,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.24 | bwd_microstep: 5859.66 | bwd_inner_microstep: 5650.91 | bwd_allreduce_microstep: 208.70 | step_microstep: 18.87
[2025-04-26 09:37:10,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.24 | bwd: 5859.67 | bwd_inner: 5650.91 | bwd_allreduce: 208.72 | step: 18.87

  6%|▌         | 2269/41250 [5:28:41<94:26:24,  8.72s/it]
                                                         
{'loss': 0.0725, 'grad_norm': 0.7619580030441284, 'learning_rate': 3.993450632147702e-05, 'epoch': 0.55}

  6%|▌         | 2269/41250 [5:28:41<94:26:24,  8.72s/it][2025-04-26 09:37:19,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:37:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.73 | bwd_microstep: 5888.66 | bwd_inner_microstep: 5638.41 | bwd_allreduce_microstep: 250.21 | step_microstep: 18.90
[2025-04-26 09:37:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.73 | bwd: 5888.68 | bwd_inner: 5638.40 | bwd_allreduce: 250.23 | step: 18.90

  6%|▌         | 2270/41250 [5:28:50<94:38:51,  8.74s/it]
                                                         
{'loss': 0.1276, 'grad_norm': 1.5992411375045776, 'learning_rate': 3.993437928057434e-05, 'epoch': 0.55}

  6%|▌         | 2270/41250 [5:28:50<94:38:51,  8.74s/it][2025-04-26 09:37:27,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.30 | optimizer_step: 1.01
[2025-04-26 09:37:27,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.67 | bwd_microstep: 5695.95 | bwd_inner_microstep: 5682.12 | bwd_allreduce_microstep: 13.77 | step_microstep: 19.72
[2025-04-26 09:37:27,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.67 | bwd: 5695.96 | bwd_inner: 5682.12 | bwd_allreduce: 13.80 | step: 19.72

  6%|▌         | 2271/41250 [5:28:59<94:15:22,  8.71s/it]
                                                         
{'loss': 0.1724, 'grad_norm': 2.1782257556915283, 'learning_rate': 3.9934252116780124e-05, 'epoch': 0.55}

  6%|▌         | 2271/41250 [5:28:59<94:15:22,  8.71s/it][2025-04-26 09:37:36,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:37:36,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.13 | bwd_microstep: 5703.00 | bwd_inner_microstep: 5690.08 | bwd_allreduce_microstep: 12.87 | step_microstep: 17.97
[2025-04-26 09:37:36,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.13 | bwd: 5703.01 | bwd_inner: 5690.08 | bwd_allreduce: 12.89 | step: 17.97

  6%|▌         | 2272/41250 [5:29:07<94:01:27,  8.68s/it]
                                                         
{'loss': 0.062, 'grad_norm': 1.1518924236297607, 'learning_rate': 3.9934124830095175e-05, 'epoch': 0.55}

  6%|▌         | 2272/41250 [5:29:07<94:01:27,  8.68s/it][2025-04-26 09:37:45,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:37:45,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.42 | bwd_microstep: 6032.75 | bwd_inner_microstep: 5659.16 | bwd_allreduce_microstep: 373.55 | step_microstep: 18.33
[2025-04-26 09:37:45,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.42 | bwd: 6032.77 | bwd_inner: 5659.16 | bwd_allreduce: 373.57 | step: 18.33

  6%|▌         | 2273/41250 [5:29:16<94:50:55,  8.76s/it]
                                                         
{'loss': 0.2438, 'grad_norm': 2.4563517570495605, 'learning_rate': 3.993399742052026e-05, 'epoch': 0.55}

  6%|▌         | 2273/41250 [5:29:16<94:50:55,  8.76s/it][2025-04-26 09:37:54,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:37:54,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.07 | bwd_microstep: 5697.99 | bwd_inner_microstep: 5684.99 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.35
[2025-04-26 09:37:54,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.07 | bwd: 5698.01 | bwd_inner: 5684.99 | bwd_allreduce: 12.98 | step: 18.35

  6%|▌         | 2274/41250 [5:29:25<94:24:07,  8.72s/it]
                                                         
{'loss': 0.0782, 'grad_norm': 0.8369078636169434, 'learning_rate': 3.993386988805617e-05, 'epoch': 0.55}

  6%|▌         | 2274/41250 [5:29:25<94:24:07,  8.72s/it][2025-04-26 09:38:02,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:38:02,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5784.31 | bwd_inner_microstep: 5642.52 | bwd_allreduce_microstep: 141.74 | step_microstep: 18.63
[2025-04-26 09:38:02,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5784.32 | bwd_inner: 5642.52 | bwd_allreduce: 141.76 | step: 18.63

  6%|▌         | 2275/41250 [5:29:33<94:18:24,  8.71s/it]
                                                         
{'loss': 0.4344, 'grad_norm': 2.2790260314941406, 'learning_rate': 3.9933742232703696e-05, 'epoch': 0.55}

  6%|▌         | 2275/41250 [5:29:33<94:18:24,  8.71s/it][2025-04-26 09:38:11,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.91 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 09:38:11,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5708.31 | bwd_inner_microstep: 5695.27 | bwd_allreduce_microstep: 12.98 | step_microstep: 19.31
[2025-04-26 09:38:11,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5708.32 | bwd_inner: 5695.27 | bwd_allreduce: 13.01 | step: 19.32

  6%|▌         | 2276/41250 [5:29:42<94:04:57,  8.69s/it]
                                                         
{'loss': 0.2591, 'grad_norm': 1.6937267780303955, 'learning_rate': 3.993361445446363e-05, 'epoch': 0.55}

  6%|▌         | 2276/41250 [5:29:42<94:04:57,  8.69s/it][2025-04-26 09:38:20,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:38:20,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.01 | bwd_microstep: 5694.77 | bwd_inner_microstep: 5653.62 | bwd_allreduce_microstep: 41.09 | step_microstep: 18.57
[2025-04-26 09:38:20,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.01 | bwd: 5694.78 | bwd_inner: 5653.62 | bwd_allreduce: 41.11 | step: 18.57

  6%|▌         | 2277/41250 [5:29:51<93:51:52,  8.67s/it]
                                                         
{'loss': 0.1871, 'grad_norm': 1.2164087295532227, 'learning_rate': 3.993348655333674e-05, 'epoch': 0.55}

  6%|▌         | 2277/41250 [5:29:51<93:51:52,  8.67s/it][2025-04-26 09:38:28,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:38:28,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.60 | bwd_microstep: 5702.95 | bwd_inner_microstep: 5689.87 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.90
[2025-04-26 09:38:28,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.60 | bwd: 5702.96 | bwd_inner: 5689.87 | bwd_allreduce: 13.05 | step: 18.91

  6%|▌         | 2278/41250 [5:29:59<93:43:22,  8.66s/it]
                                                         
{'loss': 0.199, 'grad_norm': 1.4696791172027588, 'learning_rate': 3.9933358529323836e-05, 'epoch': 0.55}

  6%|▌         | 2278/41250 [5:29:59<93:43:22,  8.66s/it][2025-04-26 09:38:37,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:38:37,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.66 | bwd_microstep: 5718.36 | bwd_inner_microstep: 5647.97 | bwd_allreduce_microstep: 70.35 | step_microstep: 18.38
[2025-04-26 09:38:37,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.66 | bwd: 5718.37 | bwd_inner: 5647.97 | bwd_allreduce: 70.37 | step: 18.38

  6%|▌         | 2279/41250 [5:30:08<93:35:41,  8.65s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 1.1993895769119263, 'learning_rate': 3.993323038242569e-05, 'epoch': 0.55}

  6%|▌         | 2279/41250 [5:30:08<93:35:41,  8.65s/it][2025-04-26 09:38:46,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:38:46,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.49 | bwd_microstep: 5762.24 | bwd_inner_microstep: 5703.04 | bwd_allreduce_microstep: 59.16 | step_microstep: 18.25
[2025-04-26 09:38:46,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.49 | bwd: 5762.25 | bwd_inner: 5703.04 | bwd_allreduce: 59.17 | step: 18.25

  6%|▌         | 2280/41250 [5:30:17<93:47:26,  8.66s/it]
                                                         
{'loss': 0.3333, 'grad_norm': 2.2882297039031982, 'learning_rate': 3.9933102112643094e-05, 'epoch': 0.55}

  6%|▌         | 2280/41250 [5:30:17<93:47:26,  8.66s/it][2025-04-26 09:38:54,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 09:38:54,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.55 | bwd_microstep: 5715.70 | bwd_inner_microstep: 5702.35 | bwd_allreduce_microstep: 13.30 | step_microstep: 19.33
[2025-04-26 09:38:54,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.55 | bwd: 5715.72 | bwd_inner: 5702.35 | bwd_allreduce: 13.32 | step: 19.34

  6%|▌         | 2281/41250 [5:30:25<93:47:40,  8.66s/it]
                                                         
{'loss': 0.0326, 'grad_norm': 0.41744521260261536, 'learning_rate': 3.993297371997686e-05, 'epoch': 0.55}

  6%|▌         | 2281/41250 [5:30:25<93:47:40,  8.66s/it][2025-04-26 09:39:03,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:39:03,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.75 | bwd_microstep: 5748.58 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 68.56 | step_microstep: 18.16
[2025-04-26 09:39:03,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.75 | bwd: 5748.59 | bwd_inner: 5679.98 | bwd_allreduce: 68.57 | step: 18.16

  6%|▌         | 2282/41250 [5:30:34<93:49:59,  8.67s/it]
                                                         
{'loss': 0.1602, 'grad_norm': 1.3251078128814697, 'learning_rate': 3.993284520442775e-05, 'epoch': 0.55}

  6%|▌         | 2282/41250 [5:30:34<93:49:59,  8.67s/it][2025-04-26 09:39:12,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:39:12,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.03 | bwd_microstep: 5720.12 | bwd_inner_microstep: 5707.19 | bwd_allreduce_microstep: 12.88 | step_microstep: 18.49
[2025-04-26 09:39:12,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.03 | bwd: 5720.13 | bwd_inner: 5707.19 | bwd_allreduce: 12.90 | step: 18.50

  6%|▌         | 2283/41250 [5:30:43<93:46:34,  8.66s/it]
                                                         
{'loss': 0.2656, 'grad_norm': 1.809128999710083, 'learning_rate': 3.993271656599658e-05, 'epoch': 0.55}

  6%|▌         | 2283/41250 [5:30:43<93:46:34,  8.66s/it][2025-04-26 09:39:20,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 09:39:20,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.35 | bwd_microstep: 5716.44 | bwd_inner_microstep: 5652.25 | bwd_allreduce_microstep: 64.14 | step_microstep: 18.75
[2025-04-26 09:39:20,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.35 | bwd: 5716.45 | bwd_inner: 5652.25 | bwd_allreduce: 64.16 | step: 18.76

  6%|▌         | 2284/41250 [5:30:51<93:37:10,  8.65s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 1.0217130184173584, 'learning_rate': 3.993258780468412e-05, 'epoch': 0.55}

  6%|▌         | 2284/41250 [5:30:51<93:37:10,  8.65s/it][2025-04-26 09:39:29,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:39:29,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5790.21 | bwd_inner_microstep: 5693.02 | bwd_allreduce_microstep: 97.15 | step_microstep: 18.66
[2025-04-26 09:39:29,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5790.23 | bwd_inner: 5693.02 | bwd_allreduce: 97.17 | step: 18.67

  6%|▌         | 2285/41250 [5:31:00<93:50:32,  8.67s/it]
                                                         
{'loss': 0.0871, 'grad_norm': 1.0120681524276733, 'learning_rate': 3.9932458920491184e-05, 'epoch': 0.55}

  6%|▌         | 2285/41250 [5:31:00<93:50:32,  8.67s/it][2025-04-26 09:39:38,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:39:38,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.36 | bwd_microstep: 5797.86 | bwd_inner_microstep: 5648.10 | bwd_allreduce_microstep: 149.71 | step_microstep: 18.58
[2025-04-26 09:39:38,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.36 | bwd: 5797.87 | bwd_inner: 5648.10 | bwd_allreduce: 149.73 | step: 18.58

  6%|▌         | 2286/41250 [5:31:09<93:57:03,  8.68s/it]
                                                         
{'loss': 0.1168, 'grad_norm': 1.340410590171814, 'learning_rate': 3.993232991341856e-05, 'epoch': 0.55}

  6%|▌         | 2286/41250 [5:31:09<93:57:03,  8.68s/it][2025-04-26 09:39:46,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 09:39:46,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.15 | bwd_microstep: 5784.37 | bwd_inner_microstep: 5654.66 | bwd_allreduce_microstep: 129.65 | step_microstep: 18.58
[2025-04-26 09:39:46,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.15 | bwd: 5784.39 | bwd_inner: 5654.66 | bwd_allreduce: 129.68 | step: 18.58

  6%|▌         | 2287/41250 [5:31:17<94:00:56,  8.69s/it]
                                                         
{'loss': 0.273, 'grad_norm': 1.6465458869934082, 'learning_rate': 3.9932200783467045e-05, 'epoch': 0.55}

  6%|▌         | 2287/41250 [5:31:17<94:00:56,  8.69s/it][2025-04-26 09:39:55,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:39:55,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.47 | bwd_microstep: 5754.17 | bwd_inner_microstep: 5706.16 | bwd_allreduce_microstep: 47.96 | step_microstep: 18.71
[2025-04-26 09:39:55,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.47 | bwd: 5754.18 | bwd_inner: 5706.16 | bwd_allreduce: 47.98 | step: 18.72

  6%|▌         | 2288/41250 [5:31:26<94:01:19,  8.69s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 1.4126578569412231, 'learning_rate': 3.993207153063743e-05, 'epoch': 0.55}

  6%|▌         | 2288/41250 [5:31:26<94:01:19,  8.69s/it][2025-04-26 09:40:04,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:40:04,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.55 | bwd_microstep: 5709.69 | bwd_inner_microstep: 5656.19 | bwd_allreduce_microstep: 53.45 | step_microstep: 18.48
[2025-04-26 09:40:04,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.55 | bwd: 5709.70 | bwd_inner: 5656.19 | bwd_allreduce: 53.47 | step: 18.48

  6%|▌         | 2289/41250 [5:31:35<93:48:57,  8.67s/it]
                                                         
{'loss': 0.2216, 'grad_norm': 2.2133748531341553, 'learning_rate': 3.993194215493051e-05, 'epoch': 0.55}

  6%|▌         | 2289/41250 [5:31:35<93:48:57,  8.67s/it][2025-04-26 09:40:12,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 09:40:12,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.90 | bwd_microstep: 5874.59 | bwd_inner_microstep: 5657.69 | bwd_allreduce_microstep: 216.85 | step_microstep: 18.82
[2025-04-26 09:40:12,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.90 | bwd: 5874.61 | bwd_inner: 5657.69 | bwd_allreduce: 216.87 | step: 18.82

  6%|▌         | 2290/41250 [5:31:44<94:12:28,  8.71s/it]
                                                         
{'loss': 0.1832, 'grad_norm': 1.019100546836853, 'learning_rate': 3.99318126563471e-05, 'epoch': 0.56}

  6%|▌         | 2290/41250 [5:31:44<94:12:28,  8.71s/it][2025-04-26 09:40:21,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.19 | optimizer_step: 0.93
[2025-04-26 09:40:21,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.47 | bwd_microstep: 5715.17 | bwd_inner_microstep: 5661.61 | bwd_allreduce_microstep: 53.51 | step_microstep: 19.28
[2025-04-26 09:40:21,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.48 | bwd: 5715.19 | bwd_inner: 5661.61 | bwd_allreduce: 53.54 | step: 19.28

  6%|▌         | 2291/41250 [5:31:52<93:56:56,  8.68s/it]
                                                         
{'loss': 0.2345, 'grad_norm': 1.9858028888702393, 'learning_rate': 3.993168303488797e-05, 'epoch': 0.56}

  6%|▌         | 2291/41250 [5:31:52<93:56:56,  8.68s/it][2025-04-26 09:40:30,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:40:30,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.33 | bwd_microstep: 5762.33 | bwd_inner_microstep: 5693.60 | bwd_allreduce_microstep: 68.68 | step_microstep: 18.80
[2025-04-26 09:40:30,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.33 | bwd: 5762.34 | bwd_inner: 5693.60 | bwd_allreduce: 68.70 | step: 18.80

  6%|▌         | 2292/41250 [5:32:01<94:01:29,  8.69s/it]
                                                         
{'loss': 0.3831, 'grad_norm': 1.697647213935852, 'learning_rate': 3.9931553290553945e-05, 'epoch': 0.56}

  6%|▌         | 2292/41250 [5:32:01<94:01:29,  8.69s/it][2025-04-26 09:40:38,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-26 09:40:38,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.47 | bwd_microstep: 5709.89 | bwd_inner_microstep: 5696.83 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.63
[2025-04-26 09:40:38,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.47 | bwd: 5709.90 | bwd_inner: 5696.83 | bwd_allreduce: 13.03 | step: 19.63

  6%|▌         | 2293/41250 [5:32:10<93:52:43,  8.68s/it]
                                                         
{'loss': 0.2267, 'grad_norm': 1.8808366060256958, 'learning_rate': 3.9931423423345805e-05, 'epoch': 0.56}

  6%|▌         | 2293/41250 [5:32:10<93:52:43,  8.68s/it][2025-04-26 09:40:47,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:40:47,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.05 | bwd_microstep: 5715.02 | bwd_inner_microstep: 5701.74 | bwd_allreduce_microstep: 13.22 | step_microstep: 18.79
[2025-04-26 09:40:47,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.05 | bwd: 5715.03 | bwd_inner: 5701.74 | bwd_allreduce: 13.25 | step: 18.80

  6%|▌         | 2294/41250 [5:32:18<93:47:42,  8.67s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.0495010614395142, 'learning_rate': 3.993129343326437e-05, 'epoch': 0.56}

  6%|▌         | 2294/41250 [5:32:18<93:47:42,  8.67s/it][2025-04-26 09:40:56,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 09:40:56,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.88 | bwd_microstep: 5724.77 | bwd_inner_microstep: 5652.65 | bwd_allreduce_microstep: 72.07 | step_microstep: 18.75
[2025-04-26 09:40:56,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.88 | bwd: 5724.78 | bwd_inner: 5652.65 | bwd_allreduce: 72.09 | step: 18.76

  6%|▌         | 2295/41250 [5:32:27<93:42:41,  8.66s/it]
                                                         
{'loss': 0.24, 'grad_norm': 1.8792083263397217, 'learning_rate': 3.993116332031043e-05, 'epoch': 0.56}

  6%|▌         | 2295/41250 [5:32:27<93:42:41,  8.66s/it][2025-04-26 09:41:04,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:41:04,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.38 | bwd_microstep: 5718.04 | bwd_inner_microstep: 5705.19 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.60
[2025-04-26 09:41:04,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.38 | bwd: 5718.05 | bwd_inner: 5705.19 | bwd_allreduce: 12.82 | step: 18.60

  6%|▌         | 2296/41250 [5:32:35<93:42:36,  8.66s/it]
                                                         
{'loss': 0.1663, 'grad_norm': 2.2495296001434326, 'learning_rate': 3.993103308448478e-05, 'epoch': 0.56}

  6%|▌         | 2296/41250 [5:32:35<93:42:36,  8.66s/it][2025-04-26 09:41:13,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:41:13,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.24 | bwd_microstep: 5793.86 | bwd_inner_microstep: 5639.80 | bwd_allreduce_microstep: 154.01 | step_microstep: 18.08
[2025-04-26 09:41:13,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.24 | bwd: 5793.88 | bwd_inner: 5639.80 | bwd_allreduce: 154.03 | step: 18.09

  6%|▌         | 2297/41250 [5:32:44<93:49:44,  8.67s/it]
                                                         
{'loss': 0.1463, 'grad_norm': 2.3994667530059814, 'learning_rate': 3.993090272578824e-05, 'epoch': 0.56}

  6%|▌         | 2297/41250 [5:32:44<93:49:44,  8.67s/it][2025-04-26 09:41:22,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:41:22,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.94 | bwd_microstep: 5712.46 | bwd_inner_microstep: 5699.72 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.17
[2025-04-26 09:41:22,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.94 | bwd: 5712.47 | bwd_inner: 5699.72 | bwd_allreduce: 12.71 | step: 18.18

  6%|▌         | 2298/41250 [5:32:53<93:46:59,  8.67s/it]
                                                         
{'loss': 0.0658, 'grad_norm': 1.2731367349624634, 'learning_rate': 3.993077224422159e-05, 'epoch': 0.56}

  6%|▌         | 2298/41250 [5:32:53<93:46:59,  8.67s/it][2025-04-26 09:41:30,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:41:30,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5783.99 | bwd_inner_microstep: 5697.15 | bwd_allreduce_microstep: 86.79 | step_microstep: 18.73
[2025-04-26 09:41:30,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5784.00 | bwd_inner: 5697.15 | bwd_allreduce: 86.81 | step: 18.73

  6%|▌         | 2299/41250 [5:33:02<93:55:58,  8.68s/it]
                                                         
{'loss': 0.1649, 'grad_norm': 0.9181177020072937, 'learning_rate': 3.993064163978567e-05, 'epoch': 0.56}

  6%|▌         | 2299/41250 [5:33:02<93:55:58,  8.68s/it][2025-04-26 09:41:39,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 09:41:39,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5769.51 | bwd_inner_microstep: 5680.54 | bwd_allreduce_microstep: 88.92 | step_microstep: 19.21
[2025-04-26 09:41:39,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5769.52 | bwd_inner: 5680.54 | bwd_allreduce: 88.94 | step: 19.21

  6%|▌         | 2300/41250 [5:33:10<94:00:48,  8.69s/it]
                                                         
{'loss': 0.1019, 'grad_norm': 0.879704475402832, 'learning_rate': 3.9930510912481246e-05, 'epoch': 0.56}

  6%|▌         | 2300/41250 [5:33:10<94:00:48,  8.69s/it][2025-04-26 09:41:48,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 09:41:48,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.37 | bwd_microstep: 5748.20 | bwd_inner_microstep: 5702.70 | bwd_allreduce_microstep: 45.45 | step_microstep: 18.86
[2025-04-26 09:41:48,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.37 | bwd: 5748.21 | bwd_inner: 5702.70 | bwd_allreduce: 45.47 | step: 18.86

  6%|▌         | 2301/41250 [5:33:19<93:58:45,  8.69s/it]
                                                         
{'loss': 0.3155, 'grad_norm': 1.8618882894515991, 'learning_rate': 3.993038006230916e-05, 'epoch': 0.56}

  6%|▌         | 2301/41250 [5:33:19<93:58:45,  8.69s/it][2025-04-26 09:41:56,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:41:56,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.52 | bwd_microstep: 5715.33 | bwd_inner_microstep: 5659.20 | bwd_allreduce_microstep: 56.09 | step_microstep: 19.66
[2025-04-26 09:41:56,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.52 | bwd: 5715.35 | bwd_inner: 5659.20 | bwd_allreduce: 56.10 | step: 19.66

  6%|▌         | 2302/41250 [5:33:28<93:45:01,  8.67s/it]
                                                         
{'loss': 0.3152, 'grad_norm': 2.173247814178467, 'learning_rate': 3.9930249089270185e-05, 'epoch': 0.56}

  6%|▌         | 2302/41250 [5:33:28<93:45:01,  8.67s/it][2025-04-26 09:42:05,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:42:05,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.98 | bwd_microstep: 5775.72 | bwd_inner_microstep: 5655.99 | bwd_allreduce_microstep: 119.69 | step_microstep: 18.50
[2025-04-26 09:42:05,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.98 | bwd: 5775.73 | bwd_inner: 5655.99 | bwd_allreduce: 119.70 | step: 18.50

  6%|▌         | 2303/41250 [5:33:36<93:48:22,  8.67s/it]
                                                         
{'loss': 0.1791, 'grad_norm': 1.0354249477386475, 'learning_rate': 3.993011799336515e-05, 'epoch': 0.56}

  6%|▌         | 2303/41250 [5:33:36<93:48:22,  8.67s/it][2025-04-26 09:42:14,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:42:14,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.07 | bwd_microstep: 5703.25 | bwd_inner_microstep: 5690.41 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.79
[2025-04-26 09:42:14,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.07 | bwd: 5703.27 | bwd_inner: 5690.41 | bwd_allreduce: 12.81 | step: 18.80

  6%|▌         | 2304/41250 [5:33:45<93:42:24,  8.66s/it]
                                                         
{'loss': 0.166, 'grad_norm': 1.889708161354065, 'learning_rate': 3.992998677459486e-05, 'epoch': 0.56}

  6%|▌         | 2304/41250 [5:33:45<93:42:24,  8.66s/it][2025-04-26 09:42:22,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 09:42:22,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.73 | bwd_microstep: 5886.59 | bwd_inner_microstep: 5652.31 | bwd_allreduce_microstep: 234.24 | step_microstep: 18.50
[2025-04-26 09:42:22,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.73 | bwd: 5886.61 | bwd_inner: 5652.31 | bwd_allreduce: 234.26 | step: 18.51

  6%|▌         | 2305/41250 [5:33:54<94:07:48,  8.70s/it]
                                                         
{'loss': 0.1028, 'grad_norm': 1.4301046133041382, 'learning_rate': 3.992985543296012e-05, 'epoch': 0.56}

  6%|▌         | 2305/41250 [5:33:54<94:07:48,  8.70s/it][2025-04-26 09:42:31,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:42:31,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.41 | bwd_microstep: 5765.08 | bwd_inner_microstep: 5684.80 | bwd_allreduce_microstep: 80.24 | step_microstep: 18.64
[2025-04-26 09:42:31,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5765.10 | bwd_inner: 5684.80 | bwd_allreduce: 80.25 | step: 18.65

  6%|▌         | 2306/41250 [5:34:02<94:03:30,  8.69s/it]
                                                         
{'loss': 0.2069, 'grad_norm': 2.244560956954956, 'learning_rate': 3.992972396846174e-05, 'epoch': 0.56}

  6%|▌         | 2306/41250 [5:34:02<94:03:30,  8.69s/it][2025-04-26 09:42:40,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:42:40,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5705.67 | bwd_inner_microstep: 5692.94 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.29
[2025-04-26 09:42:40,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5705.68 | bwd_inner: 5692.94 | bwd_allreduce: 12.70 | step: 18.30

  6%|▌         | 2307/41250 [5:34:11<93:52:48,  8.68s/it]
                                                         
{'loss': 0.2588, 'grad_norm': 1.6091911792755127, 'learning_rate': 3.9929592381100536e-05, 'epoch': 0.56}

  6%|▌         | 2307/41250 [5:34:11<93:52:48,  8.68s/it][2025-04-26 09:42:48,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 09:42:48,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.72 | bwd_microstep: 5702.94 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.77
[2025-04-26 09:42:48,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.72 | bwd: 5702.95 | bwd_inner: 5690.21 | bwd_allreduce: 12.70 | step: 18.77

  6%|▌         | 2308/41250 [5:34:20<93:44:11,  8.67s/it]
                                                         
{'loss': 0.2429, 'grad_norm': 1.829894781112671, 'learning_rate': 3.992946067087731e-05, 'epoch': 0.56}

  6%|▌         | 2308/41250 [5:34:20<93:44:11,  8.67s/it][2025-04-26 09:42:57,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 09:42:57,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5767.12 | bwd_inner_microstep: 5639.57 | bwd_allreduce_microstep: 127.50 | step_microstep: 18.62
[2025-04-26 09:42:57,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5767.14 | bwd_inner: 5639.57 | bwd_allreduce: 127.52 | step: 18.62

  6%|▌         | 2309/41250 [5:34:28<93:46:15,  8.67s/it]
                                                         
{'loss': 0.1505, 'grad_norm': 0.9359753727912903, 'learning_rate': 3.992932883779288e-05, 'epoch': 0.56}

  6%|▌         | 2309/41250 [5:34:28<93:46:15,  8.67s/it][2025-04-26 09:43:06,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:43:06,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.73 | bwd_microstep: 5791.63 | bwd_inner_microstep: 5644.81 | bwd_allreduce_microstep: 146.77 | step_microstep: 18.88
[2025-04-26 09:43:06,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.73 | bwd: 5791.64 | bwd_inner: 5644.81 | bwd_allreduce: 146.79 | step: 18.88

  6%|▌         | 2310/41250 [5:34:37<93:55:50,  8.68s/it]
                                                         
{'loss': 0.2377, 'grad_norm': 2.5206947326660156, 'learning_rate': 3.992919688184807e-05, 'epoch': 0.56}

  6%|▌         | 2310/41250 [5:34:37<93:55:50,  8.68s/it][2025-04-26 09:43:14,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 09:43:14,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.36 | bwd_microstep: 5722.00 | bwd_inner_microstep: 5645.08 | bwd_allreduce_microstep: 76.88 | step_microstep: 18.51
[2025-04-26 09:43:14,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.36 | bwd: 5722.02 | bwd_inner: 5645.08 | bwd_allreduce: 76.90 | step: 18.51

  6%|▌         | 2311/41250 [5:34:46<93:42:37,  8.66s/it]
                                                         
{'loss': 0.2002, 'grad_norm': 1.3627086877822876, 'learning_rate': 3.9929064803043665e-05, 'epoch': 0.56}

  6%|▌         | 2311/41250 [5:34:46<93:42:37,  8.66s/it][2025-04-26 09:43:23,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:43:23,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.79 | bwd_microstep: 5697.51 | bwd_inner_microstep: 5642.25 | bwd_allreduce_microstep: 55.21 | step_microstep: 18.76
[2025-04-26 09:43:23,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.79 | bwd: 5697.53 | bwd_inner: 5642.25 | bwd_allreduce: 55.23 | step: 18.77

  6%|▌         | 2312/41250 [5:34:54<93:32:00,  8.65s/it]
                                                         
{'loss': 0.2604, 'grad_norm': 1.5242488384246826, 'learning_rate': 3.9928932601380495e-05, 'epoch': 0.56}

  6%|▌         | 2312/41250 [5:34:54<93:32:00,  8.65s/it][2025-04-26 09:43:32,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:43:32,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.74 | bwd_microstep: 5717.67 | bwd_inner_microstep: 5631.25 | bwd_allreduce_microstep: 86.37 | step_microstep: 18.49
[2025-04-26 09:43:32,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.74 | bwd: 5717.68 | bwd_inner: 5631.25 | bwd_allreduce: 86.39 | step: 18.50

  6%|▌         | 2313/41250 [5:35:03<93:26:33,  8.64s/it]
                                                         
{'loss': 0.2169, 'grad_norm': 1.3340206146240234, 'learning_rate': 3.992880027685939e-05, 'epoch': 0.56}

  6%|▌         | 2313/41250 [5:35:03<93:26:33,  8.64s/it][2025-04-26 09:43:40,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:43:40,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5694.57 | bwd_inner_microstep: 5681.33 | bwd_allreduce_microstep: 13.19 | step_microstep: 18.93
[2025-04-26 09:43:40,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.76 | bwd: 5694.58 | bwd_inner: 5681.33 | bwd_allreduce: 13.21 | step: 18.94

  6%|▌         | 2314/41250 [5:35:11<93:22:50,  8.63s/it]
                                                         
{'loss': 0.1865, 'grad_norm': 1.1738464832305908, 'learning_rate': 3.9928667829481136e-05, 'epoch': 0.56}

  6%|▌         | 2314/41250 [5:35:11<93:22:50,  8.63s/it][2025-04-26 09:43:49,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.04
[2025-04-26 09:43:49,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.69 | bwd_microstep: 5772.59 | bwd_inner_microstep: 5701.21 | bwd_allreduce_microstep: 71.33 | step_microstep: 19.41
[2025-04-26 09:43:49,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.69 | bwd: 5772.60 | bwd_inner: 5701.21 | bwd_allreduce: 71.35 | step: 19.41

  6%|▌         | 2315/41250 [5:35:20<93:33:25,  8.65s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.4368571043014526, 'learning_rate': 3.9928535259246575e-05, 'epoch': 0.56}

  6%|▌         | 2315/41250 [5:35:20<93:33:25,  8.65s/it][2025-04-26 09:43:58,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:43:58,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.26 | bwd_microstep: 5708.16 | bwd_inner_microstep: 5695.47 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.63
[2025-04-26 09:43:58,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.26 | bwd: 5708.18 | bwd_inner: 5695.47 | bwd_allreduce: 12.67 | step: 18.64

  6%|▌         | 2316/41250 [5:35:29<93:30:19,  8.65s/it]
                                                         
{'loss': 0.1531, 'grad_norm': 1.2917336225509644, 'learning_rate': 3.9928402566156504e-05, 'epoch': 0.56}

  6%|▌         | 2316/41250 [5:35:29<93:30:19,  8.65s/it][2025-04-26 09:44:06,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:44:06,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.63 | bwd_microstep: 5766.20 | bwd_inner_microstep: 5655.02 | bwd_allreduce_microstep: 111.13 | step_microstep: 18.55
[2025-04-26 09:44:06,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.63 | bwd: 5766.21 | bwd_inner: 5655.02 | bwd_allreduce: 111.15 | step: 18.55

  6%|▌         | 2317/41250 [5:35:37<93:37:52,  8.66s/it]
                                                         
{'loss': 0.2526, 'grad_norm': 1.6778136491775513, 'learning_rate': 3.992826975021176e-05, 'epoch': 0.56}

  6%|▌         | 2317/41250 [5:35:37<93:37:52,  8.66s/it][2025-04-26 09:44:15,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:44:15,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.02 | bwd_microstep: 5762.24 | bwd_inner_microstep: 5633.82 | bwd_allreduce_microstep: 128.37 | step_microstep: 18.56
[2025-04-26 09:44:15,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.02 | bwd: 5762.25 | bwd_inner: 5633.82 | bwd_allreduce: 128.39 | step: 18.57

  6%|▌         | 2318/41250 [5:35:46<93:38:29,  8.66s/it]
                                                         
{'loss': 0.1935, 'grad_norm': 1.536669135093689, 'learning_rate': 3.9928136811413145e-05, 'epoch': 0.56}

  6%|▌         | 2318/41250 [5:35:46<93:38:29,  8.66s/it][2025-04-26 09:44:24,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:44:24,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.66 | bwd_microstep: 5765.67 | bwd_inner_microstep: 5691.02 | bwd_allreduce_microstep: 74.60 | step_microstep: 18.58
[2025-04-26 09:44:24,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.66 | bwd: 5765.69 | bwd_inner: 5691.02 | bwd_allreduce: 74.62 | step: 18.58

  6%|▌         | 2319/41250 [5:35:55<93:46:01,  8.67s/it]
                                                         
{'loss': 0.1992, 'grad_norm': 2.17788028717041, 'learning_rate': 3.992800374976149e-05, 'epoch': 0.56}

  6%|▌         | 2319/41250 [5:35:55<93:46:01,  8.67s/it][2025-04-26 09:44:32,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.21 | optimizer_gradients: 1.60 | optimizer_step: 0.91
[2025-04-26 09:44:32,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.03 | bwd_microstep: 5800.46 | bwd_inner_microstep: 5786.94 | bwd_allreduce_microstep: 13.47 | step_microstep: 21.85
[2025-04-26 09:44:32,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.03 | bwd: 5800.48 | bwd_inner: 5786.94 | bwd_allreduce: 13.49 | step: 21.86

  6%|▌         | 2320/41250 [5:36:04<94:05:46,  8.70s/it]
                                                         
{'loss': 0.1317, 'grad_norm': 1.59442937374115, 'learning_rate': 3.992787056525761e-05, 'epoch': 0.56}

  6%|▌         | 2320/41250 [5:36:04<94:05:46,  8.70s/it][2025-04-26 09:44:41,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 09:44:41,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5731.23 | bwd_inner_microstep: 5680.04 | bwd_allreduce_microstep: 51.14 | step_microstep: 19.10
[2025-04-26 09:44:41,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5731.24 | bwd_inner: 5680.04 | bwd_allreduce: 51.16 | step: 19.10

  6%|▌         | 2321/41250 [5:36:12<93:59:02,  8.69s/it]
                                                         
{'loss': 0.0785, 'grad_norm': 0.7471645474433899, 'learning_rate': 3.992773725790233e-05, 'epoch': 0.56}

  6%|▌         | 2321/41250 [5:36:12<93:59:02,  8.69s/it][2025-04-26 09:44:50,300] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:44:50,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.13 | bwd_microstep: 5781.88 | bwd_inner_microstep: 5636.25 | bwd_allreduce_microstep: 145.59 | step_microstep: 18.43
[2025-04-26 09:44:50,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.13 | bwd: 5781.89 | bwd_inner: 5636.25 | bwd_allreduce: 145.61 | step: 18.43

  6%|▌         | 2322/41250 [5:36:21<93:59:17,  8.69s/it]
                                                         
{'loss': 0.326, 'grad_norm': 2.0972671508789062, 'learning_rate': 3.992760382769647e-05, 'epoch': 0.56}

  6%|▌         | 2322/41250 [5:36:21<93:59:17,  8.69s/it][2025-04-26 09:44:59,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.19 | optimizer_step: 0.90
[2025-04-26 09:44:59,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.41 | bwd_microstep: 5797.77 | bwd_inner_microstep: 5644.37 | bwd_allreduce_microstep: 153.34 | step_microstep: 18.81
[2025-04-26 09:44:59,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.42 | bwd: 5797.78 | bwd_inner: 5644.37 | bwd_allreduce: 153.36 | step: 18.83

  6%|▌         | 2323/41250 [5:36:30<94:02:05,  8.70s/it]
                                                         
{'loss': 0.2487, 'grad_norm': 1.4578099250793457, 'learning_rate': 3.992747027464085e-05, 'epoch': 0.56}

  6%|▌         | 2323/41250 [5:36:30<94:02:05,  8.70s/it][2025-04-26 09:45:07,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:45:07,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5762.94 | bwd_inner_microstep: 5694.35 | bwd_allreduce_microstep: 68.54 | step_microstep: 18.60
[2025-04-26 09:45:07,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5762.95 | bwd_inner: 5694.35 | bwd_allreduce: 68.56 | step: 18.60

  6%|▌         | 2324/41250 [5:36:38<93:59:34,  8.69s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.1769299507141113, 'learning_rate': 3.9927336598736296e-05, 'epoch': 0.56}

  6%|▌         | 2324/41250 [5:36:38<93:59:34,  8.69s/it][2025-04-26 09:45:16,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.15 | optimizer_step: 0.91
[2025-04-26 09:45:16,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.47 | bwd_microstep: 5782.91 | bwd_inner_microstep: 5643.44 | bwd_allreduce_microstep: 139.42 | step_microstep: 19.21
[2025-04-26 09:45:16,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.47 | bwd: 5782.93 | bwd_inner: 5643.44 | bwd_allreduce: 139.44 | step: 19.22

  6%|▌         | 2325/41250 [5:36:47<94:00:30,  8.69s/it]
                                                         
{'loss': 0.0635, 'grad_norm': 0.6737384796142578, 'learning_rate': 3.9927202799983626e-05, 'epoch': 0.56}

  6%|▌         | 2325/41250 [5:36:47<94:00:30,  8.69s/it][2025-04-26 09:45:25,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 09:45:25,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.88 | bwd_microstep: 5778.60 | bwd_inner_microstep: 5653.55 | bwd_allreduce_microstep: 124.99 | step_microstep: 19.11
[2025-04-26 09:45:25,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.88 | bwd: 5778.61 | bwd_inner: 5653.55 | bwd_allreduce: 125.02 | step: 19.11

  6%|▌         | 2326/41250 [5:36:56<94:01:31,  8.70s/it]
                                                         
{'loss': 0.2254, 'grad_norm': 1.852578043937683, 'learning_rate': 3.9927068878383675e-05, 'epoch': 0.56}

  6%|▌         | 2326/41250 [5:36:56<94:01:31,  8.70s/it][2025-04-26 09:45:33,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:45:33,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.51 | bwd_microstep: 5765.76 | bwd_inner_microstep: 5702.97 | bwd_allreduce_microstep: 62.73 | step_microstep: 18.67
[2025-04-26 09:45:33,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.51 | bwd: 5765.77 | bwd_inner: 5702.97 | bwd_allreduce: 62.75 | step: 18.67

  6%|▌         | 2327/41250 [5:37:04<94:01:16,  8.70s/it]
                                                         
{'loss': 0.1008, 'grad_norm': 1.2725958824157715, 'learning_rate': 3.992693483393727e-05, 'epoch': 0.56}

  6%|▌         | 2327/41250 [5:37:04<94:01:16,  8.70s/it][2025-04-26 09:45:42,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:45:42,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.49 | bwd_microstep: 5781.44 | bwd_inner_microstep: 5660.33 | bwd_allreduce_microstep: 121.06 | step_microstep: 19.00
[2025-04-26 09:45:42,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.49 | bwd: 5781.45 | bwd_inner: 5660.33 | bwd_allreduce: 121.08 | step: 19.01

  6%|▌         | 2328/41250 [5:37:13<94:02:25,  8.70s/it]
                                                         
{'loss': 0.2252, 'grad_norm': 2.0609304904937744, 'learning_rate': 3.9926800666645225e-05, 'epoch': 0.56}

  6%|▌         | 2328/41250 [5:37:13<94:02:25,  8.70s/it][2025-04-26 09:45:51,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 09:45:51,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.63 | bwd_microstep: 5804.36 | bwd_inner_microstep: 5655.86 | bwd_allreduce_microstep: 148.45 | step_microstep: 18.39
[2025-04-26 09:45:51,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.63 | bwd: 5804.37 | bwd_inner: 5655.86 | bwd_allreduce: 148.47 | step: 18.39

  6%|▌         | 2329/41250 [5:37:22<94:06:18,  8.70s/it]
                                                         
{'loss': 0.096, 'grad_norm': 0.6821066737174988, 'learning_rate': 3.992666637650837e-05, 'epoch': 0.56}

  6%|▌         | 2329/41250 [5:37:22<94:06:18,  8.70s/it][2025-04-26 09:45:59,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.20 | optimizer_step: 0.90
[2025-04-26 09:45:59,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.80 | bwd_microstep: 5744.99 | bwd_inner_microstep: 5649.48 | bwd_allreduce_microstep: 95.46 | step_microstep: 18.92
[2025-04-26 09:45:59,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.80 | bwd: 5745.01 | bwd_inner: 5649.48 | bwd_allreduce: 95.48 | step: 18.92

  6%|▌         | 2330/41250 [5:37:31<93:56:32,  8.69s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.2890223264694214, 'learning_rate': 3.992653196352754e-05, 'epoch': 0.56}

  6%|▌         | 2330/41250 [5:37:31<93:56:32,  8.69s/it][2025-04-26 09:46:08,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-26 09:46:08,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.24 | bwd_microstep: 5777.25 | bwd_inner_microstep: 5666.63 | bwd_allreduce_microstep: 110.57 | step_microstep: 19.70
[2025-04-26 09:46:08,565] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.24 | bwd: 5777.26 | bwd_inner: 5666.63 | bwd_allreduce: 110.60 | step: 19.71

  6%|▌         | 2331/41250 [5:37:39<93:59:07,  8.69s/it]
                                                         
{'loss': 0.1814, 'grad_norm': 1.2691566944122314, 'learning_rate': 3.992639742770356e-05, 'epoch': 0.57}

  6%|▌         | 2331/41250 [5:37:39<93:59:07,  8.69s/it][2025-04-26 09:46:17,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:46:17,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.76 | bwd_microstep: 5714.67 | bwd_inner_microstep: 5701.76 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.73
[2025-04-26 09:46:17,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.76 | bwd: 5714.68 | bwd_inner: 5701.76 | bwd_allreduce: 12.88 | step: 18.73

  6%|▌         | 2332/41250 [5:37:48<93:50:08,  8.68s/it]
                                                         
{'loss': 0.085, 'grad_norm': 0.8027080297470093, 'learning_rate': 3.992626276903726e-05, 'epoch': 0.57}

  6%|▌         | 2332/41250 [5:37:48<93:50:08,  8.68s/it][2025-04-26 09:46:25,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:46:25,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5792.61 | bwd_inner_microstep: 5660.45 | bwd_allreduce_microstep: 132.12 | step_microstep: 18.60
[2025-04-26 09:46:25,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5792.63 | bwd_inner: 5660.45 | bwd_allreduce: 132.14 | step: 18.60

  6%|▌         | 2333/41250 [5:37:57<93:52:04,  8.68s/it]
                                                         
{'loss': 0.1492, 'grad_norm': 1.593294382095337, 'learning_rate': 3.9926127987529466e-05, 'epoch': 0.57}

  6%|▌         | 2333/41250 [5:37:57<93:52:04,  8.68s/it][2025-04-26 09:46:34,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 09:46:34,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.58 | bwd_microstep: 5788.67 | bwd_inner_microstep: 5659.14 | bwd_allreduce_microstep: 129.48 | step_microstep: 18.33
[2025-04-26 09:46:34,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.58 | bwd: 5788.68 | bwd_inner: 5659.14 | bwd_allreduce: 129.50 | step: 18.33

  6%|▌         | 2334/41250 [5:38:05<93:55:59,  8.69s/it]
                                                         
{'loss': 0.0867, 'grad_norm': 1.443748116493225, 'learning_rate': 3.992599308318101e-05, 'epoch': 0.57}

  6%|▌         | 2334/41250 [5:38:05<93:55:59,  8.69s/it][2025-04-26 09:46:43,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:46:43,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.09 | bwd_microstep: 6051.15 | bwd_inner_microstep: 5692.84 | bwd_allreduce_microstep: 358.26 | step_microstep: 18.47
[2025-04-26 09:46:43,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.09 | bwd: 6051.16 | bwd_inner: 5692.84 | bwd_allreduce: 358.28 | step: 18.48

  6%|▌         | 2335/41250 [5:38:14<94:53:52,  8.78s/it]
                                                         
{'loss': 0.1105, 'grad_norm': 1.1451075077056885, 'learning_rate': 3.992585805599273e-05, 'epoch': 0.57}

  6%|▌         | 2335/41250 [5:38:14<94:53:52,  8.78s/it][2025-04-26 09:46:52,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:46:52,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.84 | bwd_microstep: 5726.64 | bwd_inner_microstep: 5688.97 | bwd_allreduce_microstep: 37.63 | step_microstep: 18.12
[2025-04-26 09:46:52,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.84 | bwd: 5726.65 | bwd_inner: 5688.97 | bwd_allreduce: 37.64 | step: 18.12

  6%|▌         | 2336/41250 [5:38:23<94:29:13,  8.74s/it]
                                                         
{'loss': 0.238, 'grad_norm': 1.6625334024429321, 'learning_rate': 3.992572290596545e-05, 'epoch': 0.57}

  6%|▌         | 2336/41250 [5:38:23<94:29:13,  8.74s/it][2025-04-26 09:47:00,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.08 | optimizer_step: 1.05
[2025-04-26 09:47:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.26 | bwd_microstep: 5786.07 | bwd_inner_microstep: 5706.08 | bwd_allreduce_microstep: 79.94 | step_microstep: 19.40
[2025-04-26 09:47:00,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.26 | bwd: 5786.09 | bwd_inner: 5706.08 | bwd_allreduce: 79.97 | step: 19.40

  6%|▌         | 2337/41250 [5:38:32<94:24:50,  8.73s/it]
                                                         
{'loss': 0.13, 'grad_norm': 1.3345211744308472, 'learning_rate': 3.992558763310002e-05, 'epoch': 0.57}

  6%|▌         | 2337/41250 [5:38:32<94:24:50,  8.73s/it][2025-04-26 09:47:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:47:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.75 | bwd_microstep: 5790.86 | bwd_inner_microstep: 5778.12 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.49
[2025-04-26 09:47:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.75 | bwd: 5790.88 | bwd_inner: 5778.12 | bwd_allreduce: 12.72 | step: 18.49

  6%|▌         | 2338/41250 [5:38:40<94:29:44,  8.74s/it]
                                                         
{'loss': 0.1046, 'grad_norm': 1.2974233627319336, 'learning_rate': 3.992545223739725e-05, 'epoch': 0.57}

  6%|▌         | 2338/41250 [5:38:40<94:29:44,  8.74s/it][2025-04-26 09:47:18,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 09:47:18,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.48 | bwd_microstep: 5758.30 | bwd_inner_microstep: 5704.39 | bwd_allreduce_microstep: 53.86 | step_microstep: 19.07
[2025-04-26 09:47:18,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.48 | bwd: 5758.31 | bwd_inner: 5704.39 | bwd_allreduce: 53.88 | step: 19.07

  6%|▌         | 2339/41250 [5:38:49<94:20:22,  8.73s/it]
                                                         
{'loss': 0.3397, 'grad_norm': 3.1535630226135254, 'learning_rate': 3.9925316718857996e-05, 'epoch': 0.57}

  6%|▌         | 2339/41250 [5:38:49<94:20:22,  8.73s/it][2025-04-26 09:47:27,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 09:47:27,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.40 | bwd_microstep: 5774.44 | bwd_inner_microstep: 5687.68 | bwd_allreduce_microstep: 86.71 | step_microstep: 18.74
[2025-04-26 09:47:27,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.40 | bwd: 5774.45 | bwd_inner: 5687.68 | bwd_allreduce: 86.73 | step: 18.74

  6%|▌         | 2340/41250 [5:38:58<94:15:51,  8.72s/it]
                                                         
{'loss': 0.0735, 'grad_norm': 0.8659700751304626, 'learning_rate': 3.9925181077483074e-05, 'epoch': 0.57}

  6%|▌         | 2340/41250 [5:38:58<94:15:51,  8.72s/it][2025-04-26 09:47:35,837] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 09:47:35,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.88 | bwd_microstep: 5772.88 | bwd_inner_microstep: 5692.52 | bwd_allreduce_microstep: 80.31 | step_microstep: 18.93
[2025-04-26 09:47:35,838] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.88 | bwd: 5772.89 | bwd_inner: 5692.52 | bwd_allreduce: 80.33 | step: 18.93

  6%|▌         | 2341/41250 [5:39:07<94:12:53,  8.72s/it]
                                                         
{'loss': 0.1722, 'grad_norm': 1.753294587135315, 'learning_rate': 3.992504531327333e-05, 'epoch': 0.57}

  6%|▌         | 2341/41250 [5:39:07<94:12:53,  8.72s/it][2025-04-26 09:47:44,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:47:44,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.89 | bwd_microstep: 5794.53 | bwd_inner_microstep: 5657.54 | bwd_allreduce_microstep: 136.94 | step_microstep: 18.67
[2025-04-26 09:47:44,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.89 | bwd: 5794.54 | bwd_inner: 5657.54 | bwd_allreduce: 136.96 | step: 18.67

  6%|▌         | 2342/41250 [5:39:15<94:08:26,  8.71s/it]
                                                         
{'loss': 0.0862, 'grad_norm': 1.1396636962890625, 'learning_rate': 3.992490942622961e-05, 'epoch': 0.57}

  6%|▌         | 2342/41250 [5:39:15<94:08:26,  8.71s/it][2025-04-26 09:47:53,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.73 | optimizer_gradients: 1.43 | optimizer_step: 1.03
[2025-04-26 09:47:53,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2938.62 | bwd_microstep: 5878.10 | bwd_inner_microstep: 5864.99 | bwd_allreduce_microstep: 13.06 | step_microstep: 20.97
[2025-04-26 09:47:53,436] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2938.62 | bwd: 5878.11 | bwd_inner: 5864.99 | bwd_allreduce: 13.08 | step: 20.97

  6%|▌         | 2343/41250 [5:39:24<94:45:59,  8.77s/it]
                                                         
{'loss': 0.331, 'grad_norm': 3.885988235473633, 'learning_rate': 3.9924773416352723e-05, 'epoch': 0.57}

  6%|▌         | 2343/41250 [5:39:24<94:45:59,  8.77s/it][2025-04-26 09:48:02,251] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:48:02,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.88 | bwd_microstep: 5883.46 | bwd_inner_microstep: 5691.19 | bwd_allreduce_microstep: 192.22 | step_microstep: 18.51
[2025-04-26 09:48:02,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.88 | bwd: 5883.48 | bwd_inner: 5691.19 | bwd_allreduce: 192.24 | step: 18.52

  6%|▌         | 2344/41250 [5:39:33<94:54:50,  8.78s/it]
                                                         
{'loss': 0.3899, 'grad_norm': 7.377634048461914, 'learning_rate': 3.9924637283643546e-05, 'epoch': 0.57}

  6%|▌         | 2344/41250 [5:39:33<94:54:50,  8.78s/it][2025-04-26 09:48:10,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.92 | optimizer_step: 0.89
[2025-04-26 09:48:10,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.36 | bwd_microstep: 5781.63 | bwd_inner_microstep: 5690.21 | bwd_allreduce_microstep: 91.37 | step_microstep: 18.31
[2025-04-26 09:48:10,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.36 | bwd: 5781.65 | bwd_inner: 5690.21 | bwd_allreduce: 91.39 | step: 18.31

  6%|▌         | 2345/41250 [5:39:42<94:40:53,  8.76s/it]
                                                         
{'loss': 0.1173, 'grad_norm': 0.9332043528556824, 'learning_rate': 3.9924501028102896e-05, 'epoch': 0.57}

  6%|▌         | 2345/41250 [5:39:42<94:40:53,  8.76s/it][2025-04-26 09:48:19,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:48:19,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.85 | bwd_microstep: 5784.09 | bwd_inner_microstep: 5771.30 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.22
[2025-04-26 09:48:19,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.85 | bwd: 5784.11 | bwd_inner: 5771.30 | bwd_allreduce: 12.77 | step: 18.22

  6%|▌         | 2346/41250 [5:39:50<94:39:16,  8.76s/it]
                                                         
{'loss': 0.146, 'grad_norm': 1.5918406248092651, 'learning_rate': 3.9924364649731606e-05, 'epoch': 0.57}

  6%|▌         | 2346/41250 [5:39:50<94:39:16,  8.76s/it][2025-04-26 09:48:28,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:48:28,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.51 | bwd_microstep: 5868.99 | bwd_inner_microstep: 5674.79 | bwd_allreduce_microstep: 194.16 | step_microstep: 18.63
[2025-04-26 09:48:28,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.51 | bwd: 5869.00 | bwd_inner: 5674.79 | bwd_allreduce: 194.17 | step: 18.63

  6%|▌         | 2347/41250 [5:39:59<94:47:07,  8.77s/it]
                                                         
{'loss': 0.3852, 'grad_norm': 3.972414970397949, 'learning_rate': 3.992422814853053e-05, 'epoch': 0.57}

  6%|▌         | 2347/41250 [5:39:59<94:47:07,  8.77s/it][2025-04-26 09:48:37,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 09:48:37,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.95 | bwd_microstep: 5762.63 | bwd_inner_microstep: 5654.66 | bwd_allreduce_microstep: 107.92 | step_microstep: 18.78
[2025-04-26 09:48:37,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.95 | bwd: 5762.64 | bwd_inner: 5654.66 | bwd_allreduce: 107.94 | step: 18.78

  6%|▌         | 2348/41250 [5:40:08<94:27:56,  8.74s/it]
                                                         
{'loss': 0.2173, 'grad_norm': 1.4899441003799438, 'learning_rate': 3.9924091524500516e-05, 'epoch': 0.57}

  6%|▌         | 2348/41250 [5:40:08<94:27:56,  8.74s/it][2025-04-26 09:48:46,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 09:48:46,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.86 | bwd_microstep: 6001.52 | bwd_inner_microstep: 5695.18 | bwd_allreduce_microstep: 306.29 | step_microstep: 18.77
[2025-04-26 09:48:46,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.86 | bwd: 6001.53 | bwd_inner: 5695.18 | bwd_allreduce: 306.31 | step: 18.77

  6%|▌         | 2349/41250 [5:40:17<95:05:05,  8.80s/it]
                                                         
{'loss': 0.2231, 'grad_norm': 2.1990630626678467, 'learning_rate': 3.992395477764239e-05, 'epoch': 0.57}

  6%|▌         | 2349/41250 [5:40:17<95:05:05,  8.80s/it][2025-04-26 09:48:54,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:48:54,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.29 | bwd_microstep: 5753.02 | bwd_inner_microstep: 5694.94 | bwd_allreduce_microstep: 58.04 | step_microstep: 18.92
[2025-04-26 09:48:54,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.29 | bwd: 5753.04 | bwd_inner: 5694.94 | bwd_allreduce: 58.06 | step: 18.92

  6%|▌         | 2350/41250 [5:40:25<94:40:25,  8.76s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.3781265020370483, 'learning_rate': 3.9923817907957004e-05, 'epoch': 0.57}

  6%|▌         | 2350/41250 [5:40:25<94:40:25,  8.76s/it][2025-04-26 09:49:03,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 09:49:03,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.84 | bwd_microstep: 5774.55 | bwd_inner_microstep: 5652.49 | bwd_allreduce_microstep: 122.01 | step_microstep: 18.74
[2025-04-26 09:49:03,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.84 | bwd: 5774.56 | bwd_inner: 5652.49 | bwd_allreduce: 122.03 | step: 18.75

  6%|▌         | 2351/41250 [5:40:34<94:22:16,  8.73s/it]
                                                         
{'loss': 0.1436, 'grad_norm': 1.4283522367477417, 'learning_rate': 3.9923680915445194e-05, 'epoch': 0.57}

  6%|▌         | 2351/41250 [5:40:34<94:22:16,  8.73s/it][2025-04-26 09:49:12,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.04 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:49:12,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.70 | bwd_microstep: 5699.16 | bwd_inner_microstep: 5686.24 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.07
[2025-04-26 09:49:12,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.70 | bwd: 5699.17 | bwd_inner: 5686.23 | bwd_allreduce: 12.90 | step: 19.08

  6%|▌         | 2352/41250 [5:40:43<94:03:20,  8.70s/it]
                                                         
{'loss': 0.0719, 'grad_norm': 0.6393182277679443, 'learning_rate': 3.992354380010781e-05, 'epoch': 0.57}

  6%|▌         | 2352/41250 [5:40:43<94:03:20,  8.70s/it][2025-04-26 09:49:20,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 09:49:20,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.75 | bwd_microstep: 5713.41 | bwd_inner_microstep: 5643.88 | bwd_allreduce_microstep: 69.48 | step_microstep: 19.23
[2025-04-26 09:49:20,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.75 | bwd: 5713.42 | bwd_inner: 5643.88 | bwd_allreduce: 69.50 | step: 19.22

  6%|▌         | 2353/41250 [5:40:51<93:46:49,  8.68s/it]
                                                         
{'loss': 0.219, 'grad_norm': 1.5687888860702515, 'learning_rate': 3.992340656194571e-05, 'epoch': 0.57}

  6%|▌         | 2353/41250 [5:40:51<93:46:49,  8.68s/it][2025-04-26 09:49:29,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.22 | optimizer_step: 0.89
[2025-04-26 09:49:29,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.88 | bwd_microstep: 5751.79 | bwd_inner_microstep: 5681.54 | bwd_allreduce_microstep: 70.19 | step_microstep: 19.54
[2025-04-26 09:49:29,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.88 | bwd: 5751.80 | bwd_inner: 5681.54 | bwd_allreduce: 70.22 | step: 19.54

  6%|▌         | 2354/41250 [5:41:00<93:46:35,  8.68s/it]
                                                         
{'loss': 0.0736, 'grad_norm': 1.2160611152648926, 'learning_rate': 3.992326920095971e-05, 'epoch': 0.57}

  6%|▌         | 2354/41250 [5:41:00<93:46:35,  8.68s/it][2025-04-26 09:49:38,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:49:38,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.82 | bwd_microstep: 5730.88 | bwd_inner_microstep: 5694.41 | bwd_allreduce_microstep: 36.42 | step_microstep: 18.68
[2025-04-26 09:49:38,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.82 | bwd: 5730.89 | bwd_inner: 5694.41 | bwd_allreduce: 36.44 | step: 18.68

  6%|▌         | 2355/41250 [5:41:09<93:44:05,  8.68s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 2.0482256412506104, 'learning_rate': 3.9923131717150686e-05, 'epoch': 0.57}

  6%|▌         | 2355/41250 [5:41:09<93:44:05,  8.68s/it][2025-04-26 09:49:46,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 09:49:46,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.84 | bwd_microstep: 5863.31 | bwd_inner_microstep: 5682.57 | bwd_allreduce_microstep: 180.69 | step_microstep: 18.71
[2025-04-26 09:49:46,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.84 | bwd: 5863.32 | bwd_inner: 5682.57 | bwd_allreduce: 180.71 | step: 18.71

  6%|▌         | 2356/41250 [5:41:18<94:03:58,  8.71s/it]
                                                         
{'loss': 0.4917, 'grad_norm': 3.0431089401245117, 'learning_rate': 3.9922994110519464e-05, 'epoch': 0.57}

  6%|▌         | 2356/41250 [5:41:18<94:03:58,  8.71s/it][2025-04-26 09:49:55,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:49:55,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.96 | bwd_microstep: 5713.14 | bwd_inner_microstep: 5650.38 | bwd_allreduce_microstep: 62.71 | step_microstep: 18.81
[2025-04-26 09:49:55,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.96 | bwd: 5713.15 | bwd_inner: 5650.38 | bwd_allreduce: 62.73 | step: 18.81

  6%|▌         | 2357/41250 [5:41:26<93:46:18,  8.68s/it]
                                                         
{'loss': 0.1882, 'grad_norm': 1.4982119798660278, 'learning_rate': 3.992285638106691e-05, 'epoch': 0.57}

  6%|▌         | 2357/41250 [5:41:26<93:46:18,  8.68s/it][2025-04-26 09:50:04,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 09:50:04,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.68 | bwd_microstep: 5763.75 | bwd_inner_microstep: 5644.10 | bwd_allreduce_microstep: 119.61 | step_microstep: 19.22
[2025-04-26 09:50:04,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.68 | bwd: 5763.77 | bwd_inner: 5644.10 | bwd_allreduce: 119.62 | step: 19.22

  6%|▌         | 2358/41250 [5:41:35<93:45:27,  8.68s/it]
                                                         
{'loss': 0.178, 'grad_norm': 1.5792224407196045, 'learning_rate': 3.9922718528793866e-05, 'epoch': 0.57}

  6%|▌         | 2358/41250 [5:41:35<93:45:27,  8.68s/it][2025-04-26 09:50:12,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:50:12,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.44 | bwd_microstep: 5704.95 | bwd_inner_microstep: 5692.17 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.51
[2025-04-26 09:50:12,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.44 | bwd: 5704.96 | bwd_inner: 5692.17 | bwd_allreduce: 12.75 | step: 18.52

  6%|▌         | 2359/41250 [5:41:43<93:35:57,  8.66s/it]
                                                         
{'loss': 0.0997, 'grad_norm': 1.1726454496383667, 'learning_rate': 3.992258055370117e-05, 'epoch': 0.57}

  6%|▌         | 2359/41250 [5:41:43<93:35:57,  8.66s/it][2025-04-26 09:50:21,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:50:21,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.98 | bwd_microstep: 5694.16 | bwd_inner_microstep: 5655.30 | bwd_allreduce_microstep: 38.81 | step_microstep: 18.62
[2025-04-26 09:50:21,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.98 | bwd: 5694.17 | bwd_inner: 5655.30 | bwd_allreduce: 38.83 | step: 18.62

  6%|▌         | 2360/41250 [5:41:52<93:23:03,  8.64s/it]
                                                         
{'loss': 0.1504, 'grad_norm': 1.6063841581344604, 'learning_rate': 3.9922442455789693e-05, 'epoch': 0.57}

  6%|▌         | 2360/41250 [5:41:52<93:23:03,  8.64s/it][2025-04-26 09:50:29,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:50:29,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.26 | bwd_microstep: 5697.98 | bwd_inner_microstep: 5681.93 | bwd_allreduce_microstep: 16.00 | step_microstep: 18.37
[2025-04-26 09:50:29,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.26 | bwd: 5697.99 | bwd_inner: 5681.93 | bwd_allreduce: 16.02 | step: 18.37

  6%|▌         | 2361/41250 [5:42:01<93:18:50,  8.64s/it]
                                                         
{'loss': 0.159, 'grad_norm': 2.3747410774230957, 'learning_rate': 3.992230423506027e-05, 'epoch': 0.57}

  6%|▌         | 2361/41250 [5:42:01<93:18:50,  8.64s/it][2025-04-26 09:50:38,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 09:50:38,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.74 | bwd_microstep: 5751.92 | bwd_inner_microstep: 5681.17 | bwd_allreduce_microstep: 70.70 | step_microstep: 18.48
[2025-04-26 09:50:38,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.74 | bwd: 5751.94 | bwd_inner: 5681.17 | bwd_allreduce: 70.72 | step: 18.48

  6%|▌         | 2362/41250 [5:42:09<93:26:33,  8.65s/it]
                                                         
{'loss': 0.0852, 'grad_norm': 1.7569105625152588, 'learning_rate': 3.992216589151377e-05, 'epoch': 0.57}

  6%|▌         | 2362/41250 [5:42:09<93:26:33,  8.65s/it][2025-04-26 09:50:47,346] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 09:50:47,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.58 | bwd_microstep: 5742.07 | bwd_inner_microstep: 5700.26 | bwd_allreduce_microstep: 41.74 | step_microstep: 18.66
[2025-04-26 09:50:47,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.58 | bwd: 5742.08 | bwd_inner: 5700.26 | bwd_allreduce: 41.76 | step: 18.66

  6%|▌         | 2363/41250 [5:42:18<93:30:53,  8.66s/it]
                                                         
{'loss': 0.277, 'grad_norm': 4.050884246826172, 'learning_rate': 3.9922027425151026e-05, 'epoch': 0.57}

  6%|▌         | 2363/41250 [5:42:18<93:30:53,  8.66s/it][2025-04-26 09:50:55,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 09:50:55,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.71 | bwd_microstep: 5710.02 | bwd_inner_microstep: 5649.90 | bwd_allreduce_microstep: 60.07 | step_microstep: 18.56
[2025-04-26 09:50:55,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.71 | bwd: 5710.04 | bwd_inner: 5649.90 | bwd_allreduce: 60.09 | step: 18.56

  6%|▌         | 2364/41250 [5:42:27<93:23:00,  8.65s/it]
                                                         
{'loss': 0.1666, 'grad_norm': 2.325159788131714, 'learning_rate': 3.992188883597291e-05, 'epoch': 0.57}

  6%|▌         | 2364/41250 [5:42:27<93:23:00,  8.65s/it][2025-04-26 09:51:04,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:51:04,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5712.79 | bwd_inner_microstep: 5699.68 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.49
[2025-04-26 09:51:04,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5712.81 | bwd_inner: 5699.68 | bwd_allreduce: 13.09 | step: 18.49

  6%|▌         | 2365/41250 [5:42:35<93:22:39,  8.64s/it]
                                                         
{'loss': 0.0682, 'grad_norm': 0.7380970120429993, 'learning_rate': 3.9921750123980264e-05, 'epoch': 0.57}

  6%|▌         | 2365/41250 [5:42:35<93:22:39,  8.64s/it][2025-04-26 09:51:13,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-26 09:51:13,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.84 | bwd_microstep: 5695.23 | bwd_inner_microstep: 5654.04 | bwd_allreduce_microstep: 41.14 | step_microstep: 18.90
[2025-04-26 09:51:13,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.84 | bwd: 5695.24 | bwd_inner: 5654.04 | bwd_allreduce: 41.16 | step: 18.90

  6%|▌         | 2366/41250 [5:42:44<93:17:19,  8.64s/it]
                                                         
{'loss': 0.1745, 'grad_norm': 1.3687875270843506, 'learning_rate': 3.992161128917395e-05, 'epoch': 0.57}

  6%|▌         | 2366/41250 [5:42:44<93:17:19,  8.64s/it][2025-04-26 09:51:21,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:51:21,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.60 | bwd_microstep: 5714.32 | bwd_inner_microstep: 5701.33 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.43
[2025-04-26 09:51:21,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.60 | bwd: 5714.33 | bwd_inner: 5701.33 | bwd_allreduce: 12.96 | step: 18.43

  6%|▌         | 2367/41250 [5:42:53<93:19:56,  8.64s/it]
                                                         
{'loss': 0.117, 'grad_norm': 0.9577085971832275, 'learning_rate': 3.992147233155481e-05, 'epoch': 0.57}

  6%|▌         | 2367/41250 [5:42:53<93:19:56,  8.64s/it][2025-04-26 09:51:30,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 09:51:30,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.01 | bwd_microstep: 5800.74 | bwd_inner_microstep: 5787.89 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.57
[2025-04-26 09:51:30,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.00 | bwd: 5800.75 | bwd_inner: 5787.89 | bwd_allreduce: 12.82 | step: 18.57

  6%|▌         | 2368/41250 [5:43:01<93:44:31,  8.68s/it]
                                                         
{'loss': 0.2414, 'grad_norm': 1.5242996215820312, 'learning_rate': 3.992133325112372e-05, 'epoch': 0.57}

  6%|▌         | 2368/41250 [5:43:01<93:44:31,  8.68s/it][2025-04-26 09:51:39,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 09:51:39,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.12 | bwd_microstep: 5712.28 | bwd_inner_microstep: 5650.71 | bwd_allreduce_microstep: 61.52 | step_microstep: 18.69
[2025-04-26 09:51:39,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.12 | bwd: 5712.30 | bwd_inner: 5650.71 | bwd_allreduce: 61.54 | step: 18.69

  6%|▌         | 2369/41250 [5:43:10<93:32:07,  8.66s/it]
                                                         
{'loss': 0.0764, 'grad_norm': 1.8053134679794312, 'learning_rate': 3.992119404788152e-05, 'epoch': 0.57}

  6%|▌         | 2369/41250 [5:43:10<93:32:07,  8.66s/it][2025-04-26 09:51:47,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:51:47,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.53 | bwd_microstep: 5784.91 | bwd_inner_microstep: 5648.05 | bwd_allreduce_microstep: 136.81 | step_microstep: 18.80
[2025-04-26 09:51:47,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.53 | bwd: 5784.92 | bwd_inner: 5648.05 | bwd_allreduce: 136.83 | step: 18.80

  6%|▌         | 2370/41250 [5:43:19<93:38:44,  8.67s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 1.713847279548645, 'learning_rate': 3.992105472182909e-05, 'epoch': 0.57}

  6%|▌         | 2370/41250 [5:43:19<93:38:44,  8.67s/it][2025-04-26 09:51:56,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 09:51:56,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5871.90 | bwd_inner_microstep: 5705.39 | bwd_allreduce_microstep: 166.46 | step_microstep: 18.73
[2025-04-26 09:51:56,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5871.91 | bwd_inner: 5705.39 | bwd_allreduce: 166.48 | step: 18.73

  6%|▌         | 2371/41250 [5:43:27<94:04:43,  8.71s/it]
                                                         
{'loss': 0.2573, 'grad_norm': 1.777178168296814, 'learning_rate': 3.992091527296727e-05, 'epoch': 0.57}

  6%|▌         | 2371/41250 [5:43:27<94:04:43,  8.71s/it][2025-04-26 09:52:05,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.25 | optimizer_step: 1.06
[2025-04-26 09:52:05,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.75 | bwd_microstep: 5775.04 | bwd_inner_microstep: 5694.21 | bwd_allreduce_microstep: 80.78 | step_microstep: 19.94
[2025-04-26 09:52:05,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.75 | bwd: 5775.05 | bwd_inner: 5694.21 | bwd_allreduce: 80.80 | step: 19.94

  6%|▌         | 2372/41250 [5:43:36<94:05:22,  8.71s/it]
                                                         
{'loss': 0.3989, 'grad_norm': 1.95285165309906, 'learning_rate': 3.992077570129692e-05, 'epoch': 0.58}

  6%|▌         | 2372/41250 [5:43:36<94:05:22,  8.71s/it][2025-04-26 09:52:14,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.09
[2025-04-26 09:52:14,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.54 | bwd_microstep: 5784.58 | bwd_inner_microstep: 5650.01 | bwd_allreduce_microstep: 134.52 | step_microstep: 19.04
[2025-04-26 09:52:14,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.54 | bwd: 5784.60 | bwd_inner: 5650.01 | bwd_allreduce: 134.55 | step: 19.05

  6%|▌         | 2373/41250 [5:43:45<94:01:07,  8.71s/it]
                                                         
{'loss': 0.0605, 'grad_norm': 0.8028261661529541, 'learning_rate': 3.992063600681892e-05, 'epoch': 0.58}

  6%|▌         | 2373/41250 [5:43:45<94:01:07,  8.71s/it][2025-04-26 09:52:22,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:52:22,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.06 | bwd_microstep: 5769.55 | bwd_inner_microstep: 5689.61 | bwd_allreduce_microstep: 79.90 | step_microstep: 18.60
[2025-04-26 09:52:22,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.06 | bwd: 5769.57 | bwd_inner: 5689.61 | bwd_allreduce: 79.92 | step: 18.60

  6%|▌         | 2374/41250 [5:43:54<93:59:26,  8.70s/it]
                                                         
{'loss': 0.0984, 'grad_norm': 1.2489924430847168, 'learning_rate': 3.9920496189534106e-05, 'epoch': 0.58}

  6%|▌         | 2374/41250 [5:43:54<93:59:26,  8.70s/it][2025-04-26 09:52:31,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 09:52:31,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.25 | bwd_microstep: 5781.95 | bwd_inner_microstep: 5660.19 | bwd_allreduce_microstep: 121.72 | step_microstep: 19.00
[2025-04-26 09:52:31,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5781.97 | bwd_inner: 5660.19 | bwd_allreduce: 121.74 | step: 19.00

  6%|▌         | 2375/41250 [5:44:02<93:57:45,  8.70s/it]
                                                         
{'loss': 0.4608, 'grad_norm': 1.8174066543579102, 'learning_rate': 3.9920356249443354e-05, 'epoch': 0.58}

  6%|▌         | 2375/41250 [5:44:02<93:57:45,  8.70s/it][2025-04-26 09:52:40,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.24 | optimizer_step: 0.94
[2025-04-26 09:52:40,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.98 | bwd_microstep: 5776.11 | bwd_inner_microstep: 5703.58 | bwd_allreduce_microstep: 72.48 | step_microstep: 19.55
[2025-04-26 09:52:40,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.98 | bwd: 5776.12 | bwd_inner: 5703.58 | bwd_allreduce: 72.50 | step: 19.56

  6%|▌         | 2376/41250 [5:44:11<94:01:21,  8.71s/it]
                                                         
{'loss': 0.1519, 'grad_norm': 1.4629684686660767, 'learning_rate': 3.992021618654752e-05, 'epoch': 0.58}

  6%|▌         | 2376/41250 [5:44:11<94:01:21,  8.71s/it][2025-04-26 09:52:49,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-26 09:52:49,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.49 | bwd_microstep: 5901.01 | bwd_inner_microstep: 5659.04 | bwd_allreduce_microstep: 241.90 | step_microstep: 18.65
[2025-04-26 09:52:49,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.49 | bwd: 5901.03 | bwd_inner: 5659.04 | bwd_allreduce: 241.94 | step: 18.65

  6%|▌         | 2377/41250 [5:44:20<94:25:52,  8.75s/it]
                                                         
{'loss': 0.1692, 'grad_norm': 1.3122944831848145, 'learning_rate': 3.992007600084748e-05, 'epoch': 0.58}

  6%|▌         | 2377/41250 [5:44:20<94:25:52,  8.75s/it][2025-04-26 09:52:57,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.15 | optimizer_step: 1.23
[2025-04-26 09:52:57,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.81 | bwd_microstep: 5786.13 | bwd_inner_microstep: 5641.93 | bwd_allreduce_microstep: 144.15 | step_microstep: 19.39
[2025-04-26 09:52:57,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.81 | bwd: 5786.14 | bwd_inner: 5641.93 | bwd_allreduce: 144.16 | step: 19.40

  6%|▌         | 2378/41250 [5:44:29<94:16:44,  8.73s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 3.335095167160034, 'learning_rate': 3.991993569234408e-05, 'epoch': 0.58}

  6%|▌         | 2378/41250 [5:44:29<94:16:44,  8.73s/it][2025-04-26 09:53:06,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.14 | optimizer_step: 1.01
[2025-04-26 09:53:06,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.59 | bwd_microstep: 5775.38 | bwd_inner_microstep: 5682.73 | bwd_allreduce_microstep: 92.59 | step_microstep: 18.83
[2025-04-26 09:53:06,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.59 | bwd: 5775.40 | bwd_inner: 5682.73 | bwd_allreduce: 92.61 | step: 18.83

  6%|▌         | 2379/41250 [5:44:37<94:13:41,  8.73s/it]
                                                         
{'loss': 0.1236, 'grad_norm': 1.3410199880599976, 'learning_rate': 3.9919795261038204e-05, 'epoch': 0.58}

  6%|▌         | 2379/41250 [5:44:37<94:13:41,  8.73s/it][2025-04-26 09:53:15,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.05 | optimizer_step: 0.99
[2025-04-26 09:53:15,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.67 | bwd_microstep: 5769.87 | bwd_inner_microstep: 5663.92 | bwd_allreduce_microstep: 105.89 | step_microstep: 18.83
[2025-04-26 09:53:15,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.67 | bwd: 5769.88 | bwd_inner: 5663.92 | bwd_allreduce: 105.91 | step: 18.84

  6%|▌         | 2380/41250 [5:44:46<94:06:26,  8.72s/it]
                                                         
{'loss': 0.1815, 'grad_norm': 1.2965307235717773, 'learning_rate': 3.99196547069307e-05, 'epoch': 0.58}

  6%|▌         | 2380/41250 [5:44:46<94:06:26,  8.72s/it][2025-04-26 09:53:23,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:53:23,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.68 | bwd_microstep: 5705.19 | bwd_inner_microstep: 5692.52 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.78
[2025-04-26 09:53:23,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.68 | bwd: 5705.21 | bwd_inner: 5692.52 | bwd_allreduce: 12.65 | step: 18.79

  6%|▌         | 2381/41250 [5:44:55<93:50:24,  8.69s/it]
                                                         
{'loss': 0.4515, 'grad_norm': 2.2304139137268066, 'learning_rate': 3.991951403002245e-05, 'epoch': 0.58}

  6%|▌         | 2381/41250 [5:44:55<93:50:24,  8.69s/it][2025-04-26 09:53:32,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.16 | optimizer_step: 1.01
[2025-04-26 09:53:32,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5779.06 | bwd_inner_microstep: 5689.35 | bwd_allreduce_microstep: 89.65 | step_microstep: 19.04
[2025-04-26 09:53:32,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5779.07 | bwd_inner: 5689.35 | bwd_allreduce: 89.67 | step: 19.05

  6%|▌         | 2382/41250 [5:45:03<93:55:50,  8.70s/it]
                                                         
{'loss': 0.5964, 'grad_norm': 2.55887770652771, 'learning_rate': 3.991937323031431e-05, 'epoch': 0.58}

  6%|▌         | 2382/41250 [5:45:03<93:55:50,  8.70s/it][2025-04-26 09:53:41,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:53:41,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.10 | bwd_microstep: 5763.94 | bwd_inner_microstep: 5711.47 | bwd_allreduce_microstep: 52.41 | step_microstep: 18.82
[2025-04-26 09:53:41,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.10 | bwd: 5763.95 | bwd_inner: 5711.47 | bwd_allreduce: 52.44 | step: 18.82

  6%|▌         | 2383/41250 [5:45:12<93:54:08,  8.70s/it]
                                                         
{'loss': 0.2389, 'grad_norm': 1.6490644216537476, 'learning_rate': 3.991923230780715e-05, 'epoch': 0.58}

  6%|▌         | 2383/41250 [5:45:12<93:54:08,  8.70s/it][2025-04-26 09:53:49,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 09:53:49,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.27 | bwd_microstep: 5791.37 | bwd_inner_microstep: 5649.42 | bwd_allreduce_microstep: 141.90 | step_microstep: 18.54
[2025-04-26 09:53:49,963] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.27 | bwd: 5791.38 | bwd_inner: 5649.42 | bwd_allreduce: 141.92 | step: 18.54

  6%|▌         | 2384/41250 [5:45:21<93:53:36,  8.70s/it]
                                                         
{'loss': 0.2431, 'grad_norm': 1.0916024446487427, 'learning_rate': 3.9919091262501844e-05, 'epoch': 0.58}

  6%|▌         | 2384/41250 [5:45:21<93:53:36,  8.70s/it][2025-04-26 09:53:58,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 09:53:58,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.27 | bwd_microstep: 5719.06 | bwd_inner_microstep: 5706.06 | bwd_allreduce_microstep: 12.95 | step_microstep: 19.03
[2025-04-26 09:53:58,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.27 | bwd: 5719.07 | bwd_inner: 5706.06 | bwd_allreduce: 12.97 | step: 19.03

  6%|▌         | 2385/41250 [5:45:29<93:46:00,  8.69s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.029206395149231, 'learning_rate': 3.991895009439926e-05, 'epoch': 0.58}

  6%|▌         | 2385/41250 [5:45:29<93:46:00,  8.69s/it][2025-04-26 09:54:07,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-26 09:54:07,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.96 | bwd_microstep: 5767.50 | bwd_inner_microstep: 5707.82 | bwd_allreduce_microstep: 59.63 | step_microstep: 19.09
[2025-04-26 09:54:07,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.96 | bwd: 5767.52 | bwd_inner: 5707.82 | bwd_allreduce: 59.65 | step: 19.09

  6%|▌         | 2386/41250 [5:45:38<93:48:28,  8.69s/it]
                                                         
{'loss': 0.2647, 'grad_norm': 1.449820637702942, 'learning_rate': 3.991880880350026e-05, 'epoch': 0.58}

  6%|▌         | 2386/41250 [5:45:38<93:48:28,  8.69s/it][2025-04-26 09:54:16,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 09:54:16,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.44 | bwd_microstep: 5797.95 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 144.41 | step_microstep: 18.54
[2025-04-26 09:54:16,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.44 | bwd: 5797.97 | bwd_inner: 5653.50 | bwd_allreduce: 144.42 | step: 18.55

  6%|▌         | 2387/41250 [5:45:47<93:51:07,  8.69s/it]
                                                         
{'loss': 0.1056, 'grad_norm': 1.2598676681518555, 'learning_rate': 3.9918667389805725e-05, 'epoch': 0.58}

  6%|▌         | 2387/41250 [5:45:47<93:51:07,  8.69s/it][2025-04-26 09:54:24,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-26 09:54:24,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.27 | bwd_microstep: 5876.14 | bwd_inner_microstep: 5692.28 | bwd_allreduce_microstep: 183.80 | step_microstep: 18.81
[2025-04-26 09:54:24,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.27 | bwd: 5876.15 | bwd_inner: 5692.28 | bwd_allreduce: 183.83 | step: 18.81

  6%|▌         | 2388/41250 [5:45:56<94:13:25,  8.73s/it]
                                                         
{'loss': 0.2771, 'grad_norm': 2.130638360977173, 'learning_rate': 3.9918525853316525e-05, 'epoch': 0.58}

  6%|▌         | 2388/41250 [5:45:56<94:13:25,  8.73s/it][2025-04-26 09:54:33,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 09:54:33,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.94 | bwd_microstep: 5763.47 | bwd_inner_microstep: 5678.01 | bwd_allreduce_microstep: 85.41 | step_microstep: 18.81
[2025-04-26 09:54:33,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.94 | bwd: 5763.49 | bwd_inner: 5678.01 | bwd_allreduce: 85.43 | step: 18.82

  6%|▌         | 2389/41250 [5:46:04<94:09:26,  8.72s/it]
                                                         
{'loss': 0.1673, 'grad_norm': 1.3664301633834839, 'learning_rate': 3.991838419403353e-05, 'epoch': 0.58}

  6%|▌         | 2389/41250 [5:46:04<94:09:26,  8.72s/it][2025-04-26 09:54:42,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 09:54:42,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.14 | bwd_microstep: 5772.68 | bwd_inner_microstep: 5705.76 | bwd_allreduce_microstep: 66.87 | step_microstep: 19.09
[2025-04-26 09:54:42,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.14 | bwd: 5772.70 | bwd_inner: 5705.76 | bwd_allreduce: 66.89 | step: 19.10

  6%|▌         | 2390/41250 [5:46:13<94:06:21,  8.72s/it]
                                                         
{'loss': 0.2721, 'grad_norm': 2.066336154937744, 'learning_rate': 3.9918242411957606e-05, 'epoch': 0.58}

  6%|▌         | 2390/41250 [5:46:13<94:06:21,  8.72s/it][2025-04-26 09:54:50,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 09:54:50,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.28 | bwd_microstep: 5729.20 | bwd_inner_microstep: 5705.44 | bwd_allreduce_microstep: 23.72 | step_microstep: 18.75
[2025-04-26 09:54:50,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.28 | bwd: 5729.22 | bwd_inner: 5705.44 | bwd_allreduce: 23.74 | step: 18.76

  6%|▌         | 2391/41250 [5:46:22<93:55:32,  8.70s/it]
                                                         
{'loss': 0.0695, 'grad_norm': 1.1018751859664917, 'learning_rate': 3.991810050708964e-05, 'epoch': 0.58}

  6%|▌         | 2391/41250 [5:46:22<93:55:32,  8.70s/it][2025-04-26 09:54:59,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:54:59,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.03 | bwd_microstep: 5705.25 | bwd_inner_microstep: 5636.75 | bwd_allreduce_microstep: 68.46 | step_microstep: 18.48
[2025-04-26 09:54:59,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.02 | bwd: 5705.26 | bwd_inner: 5636.75 | bwd_allreduce: 68.48 | step: 18.48

  6%|▌         | 2392/41250 [5:46:30<93:36:48,  8.67s/it]
                                                         
{'loss': 0.2753, 'grad_norm': 1.6146348714828491, 'learning_rate': 3.991795847943051e-05, 'epoch': 0.58}

  6%|▌         | 2392/41250 [5:46:30<93:36:48,  8.67s/it][2025-04-26 09:55:08,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:55:08,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.22 | bwd_microstep: 5759.08 | bwd_inner_microstep: 5695.95 | bwd_allreduce_microstep: 63.09 | step_microstep: 18.39
[2025-04-26 09:55:08,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.22 | bwd: 5759.09 | bwd_inner: 5695.95 | bwd_allreduce: 63.11 | step: 18.40

  6%|▌         | 2393/41250 [5:46:39<93:39:23,  8.68s/it]
                                                         
{'loss': 0.0627, 'grad_norm': 0.6909689903259277, 'learning_rate': 3.991781632898108e-05, 'epoch': 0.58}

  6%|▌         | 2393/41250 [5:46:39<93:39:23,  8.68s/it][2025-04-26 09:55:16,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:55:16,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.68 | bwd_microstep: 5696.89 | bwd_inner_microstep: 5684.15 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.58
[2025-04-26 09:55:16,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.68 | bwd: 5696.91 | bwd_inner: 5684.15 | bwd_allreduce: 12.72 | step: 18.58

  6%|▌         | 2394/41250 [5:46:48<93:29:07,  8.66s/it]
                                                         
{'loss': 0.1244, 'grad_norm': 1.2117836475372314, 'learning_rate': 3.991767405574223e-05, 'epoch': 0.58}

  6%|▌         | 2394/41250 [5:46:48<93:29:07,  8.66s/it][2025-04-26 09:55:25,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 09:55:25,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.02 | bwd_microstep: 5697.87 | bwd_inner_microstep: 5651.02 | bwd_allreduce_microstep: 46.81 | step_microstep: 18.43
[2025-04-26 09:55:25,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.02 | bwd: 5697.89 | bwd_inner: 5651.02 | bwd_allreduce: 46.83 | step: 18.43

  6%|▌         | 2395/41250 [5:46:56<93:18:04,  8.64s/it]
                                                         
{'loss': 0.1094, 'grad_norm': 1.1554888486862183, 'learning_rate': 3.991753165971483e-05, 'epoch': 0.58}

  6%|▌         | 2395/41250 [5:46:56<93:18:04,  8.64s/it][2025-04-26 09:55:34,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 09:55:34,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.77 | bwd_microstep: 5682.79 | bwd_inner_microstep: 5669.79 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.56
[2025-04-26 09:55:34,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.77 | bwd: 5682.80 | bwd_inner: 5669.79 | bwd_allreduce: 12.97 | step: 18.56

  6%|▌         | 2396/41250 [5:47:05<93:09:18,  8.63s/it]
                                                         
{'loss': 0.0568, 'grad_norm': 1.042087435722351, 'learning_rate': 3.991738914089977e-05, 'epoch': 0.58}

  6%|▌         | 2396/41250 [5:47:05<93:09:18,  8.63s/it][2025-04-26 09:55:42,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 09:55:42,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5726.27 | bwd_inner_microstep: 5682.50 | bwd_allreduce_microstep: 43.73 | step_microstep: 18.85
[2025-04-26 09:55:42,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5726.29 | bwd_inner: 5682.50 | bwd_allreduce: 43.74 | step: 18.86

  6%|▌         | 2397/41250 [5:47:13<93:15:00,  8.64s/it]
                                                         
{'loss': 0.2053, 'grad_norm': 2.728792428970337, 'learning_rate': 3.991724649929793e-05, 'epoch': 0.58}

  6%|▌         | 2397/41250 [5:47:13<93:15:00,  8.64s/it][2025-04-26 09:55:51,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 09:55:51,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.01 | bwd_microstep: 5679.51 | bwd_inner_microstep: 5662.88 | bwd_allreduce_microstep: 16.57 | step_microstep: 18.86
[2025-04-26 09:55:51,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.01 | bwd: 5679.52 | bwd_inner: 5662.88 | bwd_allreduce: 16.59 | step: 18.86

  6%|▌         | 2398/41250 [5:47:22<93:05:01,  8.63s/it]
                                                         
{'loss': 0.3766, 'grad_norm': 2.0126569271087646, 'learning_rate': 3.991710373491017e-05, 'epoch': 0.58}

  6%|▌         | 2398/41250 [5:47:22<93:05:01,  8.63s/it][2025-04-26 09:55:59,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:55:59,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.01 | bwd_microstep: 5688.78 | bwd_inner_microstep: 5645.34 | bwd_allreduce_microstep: 43.39 | step_microstep: 18.56
[2025-04-26 09:55:59,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.01 | bwd: 5688.80 | bwd_inner: 5645.34 | bwd_allreduce: 43.41 | step: 18.57

  6%|▌         | 2399/41250 [5:47:31<92:58:47,  8.62s/it]
                                                         
{'loss': 0.2703, 'grad_norm': 1.8189154863357544, 'learning_rate': 3.9916960847737396e-05, 'epoch': 0.58}

  6%|▌         | 2399/41250 [5:47:31<92:58:47,  8.62s/it][2025-04-26 09:56:08,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:56:08,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5697.63 | bwd_inner_microstep: 5648.99 | bwd_allreduce_microstep: 48.59 | step_microstep: 18.51
[2025-04-26 09:56:08,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5697.64 | bwd_inner: 5649.00 | bwd_allreduce: 48.60 | step: 18.51

  6%|▌         | 2400/41250 [5:47:39<92:57:07,  8.61s/it]
                                                         
{'loss': 0.3596, 'grad_norm': 2.1033544540405273, 'learning_rate': 3.9916817837780464e-05, 'epoch': 0.58}

  6%|▌         | 2400/41250 [5:47:39<92:57:07,  8.61s/it][2025-04-26 09:56:17,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.12 | optimizer_step: 0.91
[2025-04-26 09:56:17,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.98 | bwd_microstep: 5759.28 | bwd_inner_microstep: 5649.66 | bwd_allreduce_microstep: 109.57 | step_microstep: 18.68
[2025-04-26 09:56:17,147] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.99 | bwd: 5759.29 | bwd_inner: 5649.66 | bwd_allreduce: 109.59 | step: 18.69

  6%|▌         | 2401/41250 [5:47:48<93:05:17,  8.63s/it]
                                                         
{'loss': 0.0995, 'grad_norm': 1.6816380023956299, 'learning_rate': 3.991667470504028e-05, 'epoch': 0.58}

  6%|▌         | 2401/41250 [5:47:48<93:05:17,  8.63s/it][2025-04-26 09:56:25,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:56:25,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.07 | bwd_microstep: 5710.98 | bwd_inner_microstep: 5698.71 | bwd_allreduce_microstep: 12.23 | step_microstep: 18.63
[2025-04-26 09:56:25,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.07 | bwd: 5710.99 | bwd_inner: 5698.71 | bwd_allreduce: 12.25 | step: 18.64

  6%|▌         | 2402/41250 [5:47:56<93:09:15,  8.63s/it]
                                                         
{'loss': 0.1277, 'grad_norm': 1.3690251111984253, 'learning_rate': 3.99165314495177e-05, 'epoch': 0.58}

  6%|▌         | 2402/41250 [5:47:56<93:09:15,  8.63s/it][2025-04-26 09:56:34,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:56:34,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.96 | bwd_microstep: 5767.68 | bwd_inner_microstep: 5645.59 | bwd_allreduce_microstep: 122.04 | step_microstep: 18.50
[2025-04-26 09:56:34,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.96 | bwd: 5767.69 | bwd_inner: 5645.59 | bwd_allreduce: 122.06 | step: 18.51

  6%|▌         | 2403/41250 [5:48:05<93:17:11,  8.64s/it]
                                                         
{'loss': 0.2564, 'grad_norm': 2.287428855895996, 'learning_rate': 3.991638807121363e-05, 'epoch': 0.58}

  6%|▌         | 2403/41250 [5:48:05<93:17:11,  8.64s/it][2025-04-26 09:56:43,142] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 0.99
[2025-04-26 09:56:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.13 | bwd_microstep: 5744.08 | bwd_inner_microstep: 5686.40 | bwd_allreduce_microstep: 57.63 | step_microstep: 19.18
[2025-04-26 09:56:43,143] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.13 | bwd: 5744.09 | bwd_inner: 5686.40 | bwd_allreduce: 57.65 | step: 19.18

  6%|▌         | 2404/41250 [5:48:14<93:22:49,  8.65s/it]
                                                         
{'loss': 0.1577, 'grad_norm': 0.9288013577461243, 'learning_rate': 3.991624457012894e-05, 'epoch': 0.58}

  6%|▌         | 2404/41250 [5:48:14<93:22:49,  8.65s/it][2025-04-26 09:56:51,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-26 09:56:51,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.98 | bwd_microstep: 5728.93 | bwd_inner_microstep: 5636.43 | bwd_allreduce_microstep: 92.45 | step_microstep: 19.00
[2025-04-26 09:56:51,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.99 | bwd: 5728.94 | bwd_inner: 5636.43 | bwd_allreduce: 92.47 | step: 19.00

  6%|▌         | 2405/41250 [5:48:22<93:15:44,  8.64s/it]
                                                         
{'loss': 0.0571, 'grad_norm': 0.4698490798473358, 'learning_rate': 3.9916100946264526e-05, 'epoch': 0.58}

  6%|▌         | 2405/41250 [5:48:22<93:15:44,  8.64s/it][2025-04-26 09:57:00,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 09:57:00,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.09 | bwd_microstep: 5696.59 | bwd_inner_microstep: 5634.27 | bwd_allreduce_microstep: 62.28 | step_microstep: 18.84
[2025-04-26 09:57:00,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.09 | bwd: 5696.61 | bwd_inner: 5634.27 | bwd_allreduce: 62.30 | step: 18.85

  6%|▌         | 2406/41250 [5:48:31<93:07:05,  8.63s/it]
                                                         
{'loss': 0.1347, 'grad_norm': 0.8983733057975769, 'learning_rate': 3.991595719962126e-05, 'epoch': 0.58}

  6%|▌         | 2406/41250 [5:48:31<93:07:05,  8.63s/it][2025-04-26 09:57:08,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.00
[2025-04-26 09:57:08,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.25 | bwd_microstep: 5707.35 | bwd_inner_microstep: 5694.58 | bwd_allreduce_microstep: 12.72 | step_microstep: 19.43
[2025-04-26 09:57:08,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.25 | bwd: 5707.37 | bwd_inner: 5694.58 | bwd_allreduce: 12.74 | step: 19.43

  6%|▌         | 2407/41250 [5:48:40<93:09:03,  8.63s/it]
                                                         
{'loss': 0.1222, 'grad_norm': 1.7066330909729004, 'learning_rate': 3.991581333020004e-05, 'epoch': 0.58}

  6%|▌         | 2407/41250 [5:48:40<93:09:03,  8.63s/it][2025-04-26 09:57:17,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.13 | optimizer_step: 1.08
[2025-04-26 09:57:17,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.36 | bwd_microstep: 5692.95 | bwd_inner_microstep: 5649.04 | bwd_allreduce_microstep: 43.86 | step_microstep: 19.90
[2025-04-26 09:57:17,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.36 | bwd: 5692.96 | bwd_inner: 5649.04 | bwd_allreduce: 43.88 | step: 19.90

  6%|▌         | 2408/41250 [5:48:48<93:04:04,  8.63s/it]
                                                         
{'loss': 0.2568, 'grad_norm': 1.7030506134033203, 'learning_rate': 3.9915669338001745e-05, 'epoch': 0.58}

  6%|▌         | 2408/41250 [5:48:48<93:04:04,  8.63s/it][2025-04-26 09:57:26,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.08 | optimizer_step: 0.94
[2025-04-26 09:57:26,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.21 | bwd_microstep: 5738.00 | bwd_inner_microstep: 5700.95 | bwd_allreduce_microstep: 36.99 | step_microstep: 19.13
[2025-04-26 09:57:26,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.21 | bwd: 5738.01 | bwd_inner: 5700.95 | bwd_allreduce: 37.01 | step: 19.14

  6%|▌         | 2409/41250 [5:48:57<93:12:45,  8.64s/it]
                                                         
{'loss': 0.3871, 'grad_norm': 2.4623472690582275, 'learning_rate': 3.991552522302727e-05, 'epoch': 0.58}

  6%|▌         | 2409/41250 [5:48:57<93:12:45,  8.64s/it][2025-04-26 09:57:34,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:57:34,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.33 | bwd_microstep: 5750.88 | bwd_inner_microstep: 5693.36 | bwd_allreduce_microstep: 57.48 | step_microstep: 18.61
[2025-04-26 09:57:34,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.33 | bwd: 5750.90 | bwd_inner: 5693.36 | bwd_allreduce: 57.50 | step: 18.62

  6%|▌         | 2410/41250 [5:49:06<93:18:13,  8.65s/it]
                                                         
{'loss': 0.2093, 'grad_norm': 1.7871427536010742, 'learning_rate': 3.991538098527749e-05, 'epoch': 0.58}

  6%|▌         | 2410/41250 [5:49:06<93:18:13,  8.65s/it][2025-04-26 09:57:43,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 09:57:43,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.40 | bwd_microstep: 5717.74 | bwd_inner_microstep: 5705.37 | bwd_allreduce_microstep: 12.33 | step_microstep: 18.77
[2025-04-26 09:57:43,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.40 | bwd: 5717.76 | bwd_inner: 5705.37 | bwd_allreduce: 12.35 | step: 18.77

  6%|▌         | 2411/41250 [5:49:14<93:18:22,  8.65s/it]
                                                         
{'loss': 0.278, 'grad_norm': 1.2721928358078003, 'learning_rate': 3.9915236624753306e-05, 'epoch': 0.58}

  6%|▌         | 2411/41250 [5:49:14<93:18:22,  8.65s/it][2025-04-26 09:57:52,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 09:57:52,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.19 | bwd_microstep: 5712.48 | bwd_inner_microstep: 5699.59 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.29
[2025-04-26 09:57:52,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.19 | bwd: 5712.49 | bwd_inner: 5699.59 | bwd_allreduce: 12.86 | step: 19.29

  6%|▌         | 2412/41250 [5:49:23<93:16:59,  8.65s/it]
                                                         
{'loss': 0.0499, 'grad_norm': 0.5407432317733765, 'learning_rate': 3.991509214145561e-05, 'epoch': 0.58}

  6%|▌         | 2412/41250 [5:49:23<93:16:59,  8.65s/it][2025-04-26 09:58:00,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 09:58:00,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.33 | bwd_microstep: 5721.81 | bwd_inner_microstep: 5639.21 | bwd_allreduce_microstep: 82.55 | step_microstep: 18.90
[2025-04-26 09:58:00,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.33 | bwd: 5721.82 | bwd_inner: 5639.21 | bwd_allreduce: 82.57 | step: 18.91

  6%|▌         | 2413/41250 [5:49:32<93:13:23,  8.64s/it]
                                                         
{'loss': 0.1642, 'grad_norm': 1.20889151096344, 'learning_rate': 3.991494753538529e-05, 'epoch': 0.58}

  6%|▌         | 2413/41250 [5:49:32<93:13:23,  8.64s/it][2025-04-26 09:58:09,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 09:58:09,479] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5709.60 | bwd_inner_microstep: 5656.36 | bwd_allreduce_microstep: 53.19 | step_microstep: 19.27
[2025-04-26 09:58:09,480] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5709.61 | bwd_inner: 5656.36 | bwd_allreduce: 53.21 | step: 19.28

  6%|▌         | 2414/41250 [5:49:40<93:07:24,  8.63s/it]
                                                         
{'loss': 0.1392, 'grad_norm': 0.9457826614379883, 'learning_rate': 3.991480280654323e-05, 'epoch': 0.59}

  6%|▌         | 2414/41250 [5:49:40<93:07:24,  8.63s/it][2025-04-26 09:58:18,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 09:58:18,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.14 | bwd_microstep: 5704.28 | bwd_inner_microstep: 5691.61 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.48
[2025-04-26 09:58:18,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.14 | bwd: 5704.30 | bwd_inner: 5691.61 | bwd_allreduce: 12.65 | step: 18.48

  6%|▌         | 2415/41250 [5:49:49<93:08:51,  8.63s/it]
                                                         
{'loss': 0.0954, 'grad_norm': 0.8751920461654663, 'learning_rate': 3.991465795493034e-05, 'epoch': 0.59}

  6%|▌         | 2415/41250 [5:49:49<93:08:51,  8.63s/it][2025-04-26 09:58:26,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 09:58:26,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.09 | bwd_microstep: 5745.54 | bwd_inner_microstep: 5703.84 | bwd_allreduce_microstep: 41.65 | step_microstep: 17.94
[2025-04-26 09:58:26,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.10 | bwd: 5745.55 | bwd_inner: 5703.84 | bwd_allreduce: 41.67 | step: 17.94

  6%|▌         | 2416/41250 [5:49:57<93:19:05,  8.65s/it]
                                                         
{'loss': 0.2531, 'grad_norm': 1.9666448831558228, 'learning_rate': 3.991451298054748e-05, 'epoch': 0.59}

  6%|▌         | 2416/41250 [5:49:57<93:19:05,  8.65s/it][2025-04-26 09:58:35,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 09:58:35,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.62 | bwd_microstep: 5779.53 | bwd_inner_microstep: 5660.93 | bwd_allreduce_microstep: 118.55 | step_microstep: 18.37
[2025-04-26 09:58:35,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.62 | bwd: 5779.54 | bwd_inner: 5660.93 | bwd_allreduce: 118.57 | step: 18.37

  6%|▌         | 2417/41250 [5:50:06<93:30:17,  8.67s/it]
                                                         
{'loss': 0.1849, 'grad_norm': 1.2023528814315796, 'learning_rate': 3.991436788339558e-05, 'epoch': 0.59}

  6%|▌         | 2417/41250 [5:50:06<93:30:17,  8.67s/it][2025-04-26 09:58:44,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 09:58:44,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5879.18 | bwd_inner_microstep: 5664.43 | bwd_allreduce_microstep: 214.70 | step_microstep: 18.22
[2025-04-26 09:58:44,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5879.19 | bwd_inner: 5664.43 | bwd_allreduce: 214.72 | step: 18.22

  6%|▌         | 2418/41250 [5:50:15<93:54:43,  8.71s/it]
                                                         
{'loss': 0.1685, 'grad_norm': 1.2963639497756958, 'learning_rate': 3.991422266347551e-05, 'epoch': 0.59}

  6%|▌         | 2418/41250 [5:50:15<93:54:43,  8.71s/it][2025-04-26 09:58:52,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:58:52,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.62 | bwd_microstep: 5723.42 | bwd_inner_microstep: 5710.95 | bwd_allreduce_microstep: 12.43 | step_microstep: 18.09
[2025-04-26 09:58:52,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.62 | bwd: 5723.43 | bwd_inner: 5710.95 | bwd_allreduce: 12.44 | step: 18.10

  6%|▌         | 2419/41250 [5:50:24<93:47:28,  8.70s/it]
                                                         
{'loss': 0.1272, 'grad_norm': 1.4696749448776245, 'learning_rate': 3.991407732078818e-05, 'epoch': 0.59}

  6%|▌         | 2419/41250 [5:50:24<93:47:28,  8.70s/it][2025-04-26 09:59:01,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:59:01,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.21 | bwd_microstep: 5698.19 | bwd_inner_microstep: 5654.75 | bwd_allreduce_microstep: 43.39 | step_microstep: 18.49
[2025-04-26 09:59:01,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.21 | bwd: 5698.20 | bwd_inner: 5654.75 | bwd_allreduce: 43.41 | step: 18.49

  6%|▌         | 2420/41250 [5:50:32<93:34:21,  8.68s/it]
                                                         
{'loss': 0.1781, 'grad_norm': 2.358757257461548, 'learning_rate': 3.991393185533448e-05, 'epoch': 0.59}

  6%|▌         | 2420/41250 [5:50:32<93:34:21,  8.68s/it][2025-04-26 09:59:10,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-26 09:59:10,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.18 | bwd_microstep: 6046.49 | bwd_inner_microstep: 5662.65 | bwd_allreduce_microstep: 383.79 | step_microstep: 18.70
[2025-04-26 09:59:10,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.18 | bwd: 6046.50 | bwd_inner: 5662.65 | bwd_allreduce: 383.81 | step: 18.70

  6%|▌         | 2421/41250 [5:50:41<94:31:30,  8.76s/it]
                                                         
{'loss': 0.4855, 'grad_norm': 1.6757841110229492, 'learning_rate': 3.99137862671153e-05, 'epoch': 0.59}

  6%|▌         | 2421/41250 [5:50:41<94:31:30,  8.76s/it][2025-04-26 09:59:19,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 09:59:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.19 | bwd_microstep: 5753.89 | bwd_inner_microstep: 5709.15 | bwd_allreduce_microstep: 44.69 | step_microstep: 18.76
[2025-04-26 09:59:19,288] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.19 | bwd: 5753.91 | bwd_inner: 5709.15 | bwd_allreduce: 44.71 | step: 18.76

  6%|▌         | 2422/41250 [5:50:50<94:19:57,  8.75s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.0184193849563599, 'learning_rate': 3.991364055613155e-05, 'epoch': 0.59}

  6%|▌         | 2422/41250 [5:50:50<94:19:57,  8.75s/it][2025-04-26 09:59:27,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.73 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 09:59:27,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.81 | bwd_microstep: 5749.91 | bwd_inner_microstep: 5713.21 | bwd_allreduce_microstep: 36.66 | step_microstep: 18.55
[2025-04-26 09:59:27,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.81 | bwd: 5749.93 | bwd_inner: 5713.21 | bwd_allreduce: 36.67 | step: 18.56

  6%|▌         | 2423/41250 [5:50:59<94:08:45,  8.73s/it]
                                                         
{'loss': 0.1167, 'grad_norm': 1.176743984222412, 'learning_rate': 3.991349472238413e-05, 'epoch': 0.59}

  6%|▌         | 2423/41250 [5:50:59<94:08:45,  8.73s/it][2025-04-26 09:59:36,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 09:59:36,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.79 | bwd_microstep: 5766.95 | bwd_inner_microstep: 5704.68 | bwd_allreduce_microstep: 62.23 | step_microstep: 18.09
[2025-04-26 09:59:36,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5766.96 | bwd_inner: 5704.68 | bwd_allreduce: 62.24 | step: 18.09

  6%|▌         | 2424/41250 [5:51:07<94:04:03,  8.72s/it]
                                                         
{'loss': 0.0895, 'grad_norm': 0.7097205519676208, 'learning_rate': 3.9913348765873924e-05, 'epoch': 0.59}

  6%|▌         | 2424/41250 [5:51:07<94:04:03,  8.72s/it][2025-04-26 09:59:45,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.74 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 09:59:45,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.78 | bwd_microstep: 5757.54 | bwd_inner_microstep: 5713.12 | bwd_allreduce_microstep: 44.38 | step_microstep: 19.45
[2025-04-26 09:59:45,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.78 | bwd: 5757.55 | bwd_inner: 5713.12 | bwd_allreduce: 44.39 | step: 19.45

  6%|▌         | 2425/41250 [5:51:16<93:59:20,  8.72s/it]
                                                         
{'loss': 0.3212, 'grad_norm': 2.3716585636138916, 'learning_rate': 3.991320268660184e-05, 'epoch': 0.59}

  6%|▌         | 2425/41250 [5:51:16<93:59:20,  8.72s/it][2025-04-26 09:59:54,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 09:59:54,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.52 | bwd_microstep: 5877.22 | bwd_inner_microstep: 5713.44 | bwd_allreduce_microstep: 163.73 | step_microstep: 18.59
[2025-04-26 09:59:54,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.52 | bwd: 5877.24 | bwd_inner: 5713.44 | bwd_allreduce: 163.75 | step: 18.60

  6%|▌         | 2426/41250 [5:51:25<94:18:40,  8.75s/it]
                                                         
{'loss': 0.1391, 'grad_norm': 1.2480641603469849, 'learning_rate': 3.9913056484568774e-05, 'epoch': 0.59}

  6%|▌         | 2426/41250 [5:51:25<94:18:40,  8.75s/it][2025-04-26 10:00:02,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 10:00:02,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.50 | bwd_microstep: 5791.17 | bwd_inner_microstep: 5778.31 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.78
[2025-04-26 10:00:02,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.50 | bwd: 5791.19 | bwd_inner: 5778.31 | bwd_allreduce: 12.83 | step: 18.78

  6%|▌         | 2427/41250 [5:51:34<94:25:08,  8.76s/it]
                                                         
{'loss': 0.2036, 'grad_norm': 1.7315049171447754, 'learning_rate': 3.991291015977564e-05, 'epoch': 0.59}

  6%|▌         | 2427/41250 [5:51:34<94:25:08,  8.76s/it][2025-04-26 10:00:11,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 7.05 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 10:00:11,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5766.63 | bwd_inner_microstep: 5699.39 | bwd_allreduce_microstep: 67.19 | step_microstep: 20.00
[2025-04-26 10:00:11,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.70 | bwd: 5766.64 | bwd_inner: 5699.39 | bwd_allreduce: 67.21 | step: 20.01

  6%|▌         | 2428/41250 [5:51:42<94:12:10,  8.74s/it]
                                                         
{'loss': 0.1898, 'grad_norm': 0.9774007797241211, 'learning_rate': 3.9912763712223324e-05, 'epoch': 0.59}

  6%|▌         | 2428/41250 [5:51:42<94:12:10,  8.74s/it][2025-04-26 10:00:20,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:00:20,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.93 | bwd_microstep: 5783.24 | bwd_inner_microstep: 5669.13 | bwd_allreduce_microstep: 114.07 | step_microstep: 17.95
[2025-04-26 10:00:20,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.93 | bwd: 5783.26 | bwd_inner: 5669.13 | bwd_allreduce: 114.08 | step: 17.95

  6%|▌         | 2429/41250 [5:51:51<94:06:37,  8.73s/it]
                                                         
{'loss': 0.2199, 'grad_norm': 1.3737919330596924, 'learning_rate': 3.991261714191274e-05, 'epoch': 0.59}

  6%|▌         | 2429/41250 [5:51:51<94:06:37,  8.73s/it][2025-04-26 10:00:29,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:00:29,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5797.84 | bwd_inner_microstep: 5657.47 | bwd_allreduce_microstep: 140.32 | step_microstep: 17.88
[2025-04-26 10:00:29,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5797.85 | bwd_inner: 5657.47 | bwd_allreduce: 140.34 | step: 17.89

  6%|▌         | 2430/41250 [5:52:00<94:03:19,  8.72s/it]
                                                         
{'loss': 0.1153, 'grad_norm': 0.7764325141906738, 'learning_rate': 3.991247044884479e-05, 'epoch': 0.59}

  6%|▌         | 2430/41250 [5:52:00<94:03:19,  8.72s/it][2025-04-26 10:00:37,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.13 | optimizer_step: 0.90
[2025-04-26 10:00:37,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.45 | bwd_microstep: 5749.88 | bwd_inner_microstep: 5685.25 | bwd_allreduce_microstep: 64.59 | step_microstep: 19.48
[2025-04-26 10:00:37,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.45 | bwd: 5749.90 | bwd_inner: 5685.25 | bwd_allreduce: 64.61 | step: 19.49

  6%|▌         | 2431/41250 [5:52:08<93:57:25,  8.71s/it]
                                                         
{'loss': 0.0816, 'grad_norm': 1.1517322063446045, 'learning_rate': 3.9912323633020366e-05, 'epoch': 0.59}

  6%|▌         | 2431/41250 [5:52:08<93:57:25,  8.71s/it][2025-04-26 10:00:46,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:00:46,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.45 | bwd_microstep: 5791.32 | bwd_inner_microstep: 5649.24 | bwd_allreduce_microstep: 142.04 | step_microstep: 18.19
[2025-04-26 10:00:46,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.45 | bwd: 5791.33 | bwd_inner: 5649.24 | bwd_allreduce: 142.05 | step: 18.19

  6%|▌         | 2432/41250 [5:52:17<93:51:30,  8.70s/it]
                                                         
{'loss': 0.2354, 'grad_norm': 1.799546718597412, 'learning_rate': 3.99121766944404e-05, 'epoch': 0.59}

  6%|▌         | 2432/41250 [5:52:17<93:51:30,  8.70s/it][2025-04-26 10:00:55,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:00:55,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.87 | bwd_microstep: 5792.91 | bwd_inner_microstep: 5659.16 | bwd_allreduce_microstep: 133.70 | step_microstep: 18.14
[2025-04-26 10:00:55,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.87 | bwd: 5792.92 | bwd_inner: 5659.16 | bwd_allreduce: 133.72 | step: 18.14

  6%|▌         | 2433/41250 [5:52:26<93:53:14,  8.71s/it]
                                                         
{'loss': 0.0967, 'grad_norm': 0.7223762273788452, 'learning_rate': 3.991202963310577e-05, 'epoch': 0.59}

  6%|▌         | 2433/41250 [5:52:26<93:53:14,  8.71s/it][2025-04-26 10:01:03,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.64 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:01:03,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.54 | bwd_microstep: 5778.01 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 129.53 | step_microstep: 18.41
[2025-04-26 10:01:03,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.54 | bwd: 5778.03 | bwd_inner: 5648.44 | bwd_allreduce: 129.55 | step: 18.42

  6%|▌         | 2434/41250 [5:52:35<93:48:47,  8.70s/it]
                                                         
{'loss': 0.2159, 'grad_norm': 1.1387419700622559, 'learning_rate': 3.9911882449017394e-05, 'epoch': 0.59}

  6%|▌         | 2434/41250 [5:52:35<93:48:47,  8.70s/it][2025-04-26 10:01:12,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:01:12,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.95 | bwd_microstep: 5713.78 | bwd_inner_microstep: 5700.75 | bwd_allreduce_microstep: 12.99 | step_microstep: 18.06
[2025-04-26 10:01:12,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.95 | bwd: 5713.79 | bwd_inner: 5700.75 | bwd_allreduce: 13.01 | step: 18.06

  6%|▌         | 2435/41250 [5:52:43<93:40:31,  8.69s/it]
                                                         
{'loss': 0.1306, 'grad_norm': 0.9217989444732666, 'learning_rate': 3.991173514217618e-05, 'epoch': 0.59}

  6%|▌         | 2435/41250 [5:52:43<93:40:31,  8.69s/it][2025-04-26 10:01:21,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:01:21,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.26 | bwd_microstep: 5785.28 | bwd_inner_microstep: 5663.36 | bwd_allreduce_microstep: 121.87 | step_microstep: 18.37
[2025-04-26 10:01:21,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.26 | bwd: 5785.29 | bwd_inner: 5663.36 | bwd_allreduce: 121.89 | step: 18.38

  6%|▌         | 2436/41250 [5:52:52<93:44:13,  8.69s/it]
                                                         
{'loss': 0.0561, 'grad_norm': 0.5327889919281006, 'learning_rate': 3.991158771258304e-05, 'epoch': 0.59}

  6%|▌         | 2436/41250 [5:52:52<93:44:13,  8.69s/it][2025-04-26 10:01:29,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 10:01:29,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.92 | bwd_microstep: 5707.71 | bwd_inner_microstep: 5632.71 | bwd_allreduce_microstep: 74.96 | step_microstep: 18.33
[2025-04-26 10:01:29,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.92 | bwd: 5707.72 | bwd_inner: 5632.71 | bwd_allreduce: 74.97 | step: 18.33

  6%|▌         | 2437/41250 [5:53:01<93:27:32,  8.67s/it]
                                                         
{'loss': 0.1541, 'grad_norm': 1.8008726835250854, 'learning_rate': 3.9911440160238876e-05, 'epoch': 0.59}

  6%|▌         | 2437/41250 [5:53:01<93:27:32,  8.67s/it][2025-04-26 10:01:38,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:01:38,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.32 | bwd_microstep: 5696.00 | bwd_inner_microstep: 5681.26 | bwd_allreduce_microstep: 14.69 | step_microstep: 18.29
[2025-04-26 10:01:38,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.32 | bwd: 5696.01 | bwd_inner: 5681.26 | bwd_allreduce: 14.71 | step: 18.29

  6%|▌         | 2438/41250 [5:53:09<93:19:39,  8.66s/it]
                                                         
{'loss': 0.1087, 'grad_norm': 0.9472350478172302, 'learning_rate': 3.9911292485144595e-05, 'epoch': 0.59}

  6%|▌         | 2438/41250 [5:53:09<93:19:39,  8.66s/it][2025-04-26 10:01:47,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.82 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:01:47,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.19 | bwd_microstep: 5736.81 | bwd_inner_microstep: 5681.25 | bwd_allreduce_microstep: 55.51 | step_microstep: 18.63
[2025-04-26 10:01:47,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.19 | bwd: 5736.82 | bwd_inner: 5681.25 | bwd_allreduce: 55.53 | step: 18.63

  6%|▌         | 2439/41250 [5:53:18<93:23:11,  8.66s/it]
                                                         
{'loss': 0.1455, 'grad_norm': 1.7185912132263184, 'learning_rate': 3.991114468730111e-05, 'epoch': 0.59}

  6%|▌         | 2439/41250 [5:53:18<93:23:11,  8.66s/it][2025-04-26 10:01:55,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:01:55,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.07 | bwd_microstep: 5695.86 | bwd_inner_microstep: 5682.88 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.28
[2025-04-26 10:01:55,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.07 | bwd: 5695.87 | bwd_inner: 5682.88 | bwd_allreduce: 12.95 | step: 18.29

  6%|▌         | 2440/41250 [5:53:26<93:14:17,  8.65s/it]
                                                         
{'loss': 0.1049, 'grad_norm': 1.1266809701919556, 'learning_rate': 3.991099676670935e-05, 'epoch': 0.59}

  6%|▌         | 2440/41250 [5:53:26<93:14:17,  8.65s/it][2025-04-26 10:02:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:02:04,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.04 | bwd_microstep: 5773.76 | bwd_inner_microstep: 5690.38 | bwd_allreduce_microstep: 83.33 | step_microstep: 18.41
[2025-04-26 10:02:04,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.04 | bwd: 5773.77 | bwd_inner: 5690.38 | bwd_allreduce: 83.35 | step: 18.41

  6%|▌         | 2441/41250 [5:53:35<93:21:56,  8.66s/it]
                                                         
{'loss': 0.0765, 'grad_norm': 0.6683716177940369, 'learning_rate': 3.99108487233702e-05, 'epoch': 0.59}

  6%|▌         | 2441/41250 [5:53:35<93:21:56,  8.66s/it][2025-04-26 10:02:13,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:02:13,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.16 | bwd_microstep: 5768.10 | bwd_inner_microstep: 5661.33 | bwd_allreduce_microstep: 106.72 | step_microstep: 18.01
[2025-04-26 10:02:13,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.16 | bwd: 5768.11 | bwd_inner: 5661.33 | bwd_allreduce: 106.74 | step: 18.01

  6%|▌         | 2442/41250 [5:53:44<93:23:35,  8.66s/it]
                                                         
{'loss': 0.071, 'grad_norm': 0.691855251789093, 'learning_rate': 3.991070055728458e-05, 'epoch': 0.59}

  6%|▌         | 2442/41250 [5:53:44<93:23:35,  8.66s/it][2025-04-26 10:02:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:02:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.44 | bwd_microstep: 5775.62 | bwd_inner_microstep: 5762.76 | bwd_allreduce_microstep: 12.82 | step_microstep: 17.89
[2025-04-26 10:02:21,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.44 | bwd: 5775.63 | bwd_inner: 5762.76 | bwd_allreduce: 12.84 | step: 17.89

  6%|▌         | 2443/41250 [5:53:53<93:38:05,  8.69s/it]
                                                         
{'loss': 0.2211, 'grad_norm': 1.4895678758621216, 'learning_rate': 3.991055226845341e-05, 'epoch': 0.59}

  6%|▌         | 2443/41250 [5:53:53<93:38:05,  8.69s/it][2025-04-26 10:02:30,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:02:30,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.59 | bwd_microstep: 5778.74 | bwd_inner_microstep: 5652.92 | bwd_allreduce_microstep: 125.78 | step_microstep: 18.31
[2025-04-26 10:02:30,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.59 | bwd: 5778.75 | bwd_inner: 5652.92 | bwd_allreduce: 125.80 | step: 18.31

  6%|▌         | 2444/41250 [5:54:01<93:38:44,  8.69s/it]
                                                         
{'loss': 0.1408, 'grad_norm': 1.658511757850647, 'learning_rate': 3.991040385687761e-05, 'epoch': 0.59}

  6%|▌         | 2444/41250 [5:54:01<93:38:44,  8.69s/it][2025-04-26 10:02:39,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-26 10:02:39,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.15 | bwd_microstep: 5698.59 | bwd_inner_microstep: 5641.17 | bwd_allreduce_microstep: 57.37 | step_microstep: 18.16
[2025-04-26 10:02:39,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.15 | bwd: 5698.60 | bwd_inner: 5641.17 | bwd_allreduce: 57.38 | step: 18.16

  6%|▌         | 2445/41250 [5:54:10<93:21:38,  8.66s/it]
                                                         
{'loss': 0.2179, 'grad_norm': 1.0661219358444214, 'learning_rate': 3.991025532255808e-05, 'epoch': 0.59}

  6%|▌         | 2445/41250 [5:54:10<93:21:38,  8.66s/it][2025-04-26 10:02:47,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.92 | optimizer_gradients: 1.12 | optimizer_step: 1.04
[2025-04-26 10:02:47,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.24 | bwd_microstep: 5700.42 | bwd_inner_microstep: 5687.70 | bwd_allreduce_microstep: 12.66 | step_microstep: 19.35
[2025-04-26 10:02:47,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.24 | bwd: 5700.43 | bwd_inner: 5687.70 | bwd_allreduce: 12.69 | step: 19.35

  6%|▌         | 2446/41250 [5:54:18<93:14:06,  8.65s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 2.77902889251709, 'learning_rate': 3.991010666549574e-05, 'epoch': 0.59}

  6%|▌         | 2446/41250 [5:54:18<93:14:06,  8.65s/it][2025-04-26 10:02:56,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:02:56,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.92 | bwd_microstep: 5676.74 | bwd_inner_microstep: 5648.78 | bwd_allreduce_microstep: 27.92 | step_microstep: 17.67
[2025-04-26 10:02:56,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.92 | bwd: 5676.76 | bwd_inner: 5648.78 | bwd_allreduce: 27.94 | step: 17.68

  6%|▌         | 2447/41250 [5:54:27<93:02:59,  8.63s/it]
                                                         
{'loss': 0.1042, 'grad_norm': 1.0487853288650513, 'learning_rate': 3.990995788569151e-05, 'epoch': 0.59}

  6%|▌         | 2447/41250 [5:54:27<93:02:59,  8.63s/it][2025-04-26 10:03:04,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:03:04,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5689.54 | bwd_inner_microstep: 5676.99 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.24
[2025-04-26 10:03:04,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5689.55 | bwd_inner: 5676.99 | bwd_allreduce: 12.52 | step: 18.24

  6%|▌         | 2448/41250 [5:54:36<93:01:03,  8.63s/it]
                                                         
{'loss': 0.4391, 'grad_norm': 2.6941049098968506, 'learning_rate': 3.990980898314631e-05, 'epoch': 0.59}

  6%|▌         | 2448/41250 [5:54:36<93:01:03,  8.63s/it][2025-04-26 10:03:13,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:03:13,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.09 | bwd_microstep: 5738.48 | bwd_inner_microstep: 5686.67 | bwd_allreduce_microstep: 51.75 | step_microstep: 18.40
[2025-04-26 10:03:13,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.09 | bwd: 5738.50 | bwd_inner: 5686.67 | bwd_allreduce: 51.77 | step: 18.41

  6%|▌         | 2449/41250 [5:54:44<93:09:37,  8.64s/it]
                                                         
{'loss': 0.2165, 'grad_norm': 2.000103712081909, 'learning_rate': 3.9909659957861045e-05, 'epoch': 0.59}

  6%|▌         | 2449/41250 [5:54:44<93:09:37,  8.64s/it][2025-04-26 10:03:22,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:03:22,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.70 | bwd_microstep: 5715.83 | bwd_inner_microstep: 5703.11 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.89
[2025-04-26 10:03:22,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.70 | bwd: 5715.85 | bwd_inner: 5703.11 | bwd_allreduce: 12.69 | step: 18.90

  6%|▌         | 2450/41250 [5:54:53<93:07:50,  8.64s/it]
                                                         
{'loss': 0.2909, 'grad_norm': 2.2676515579223633, 'learning_rate': 3.990951080983665e-05, 'epoch': 0.59}

  6%|▌         | 2450/41250 [5:54:53<93:07:50,  8.64s/it][2025-04-26 10:03:31,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 10:03:31,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2936.00 | bwd_microstep: 5883.58 | bwd_inner_microstep: 5871.35 | bwd_allreduce_microstep: 12.19 | step_microstep: 18.78
[2025-04-26 10:03:31,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2936.00 | bwd: 5883.60 | bwd_inner: 5871.35 | bwd_allreduce: 12.21 | step: 18.78

  6%|▌         | 2451/41250 [5:55:02<93:58:31,  8.72s/it]
                                                         
{'loss': 0.0788, 'grad_norm': 1.139332890510559, 'learning_rate': 3.990936153907403e-05, 'epoch': 0.59}

  6%|▌         | 2451/41250 [5:55:02<93:58:31,  8.72s/it][2025-04-26 10:03:39,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:03:39,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5754.51 | bwd_inner_microstep: 5650.25 | bwd_allreduce_microstep: 104.21 | step_microstep: 17.76
[2025-04-26 10:03:39,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5754.52 | bwd_inner: 5650.25 | bwd_allreduce: 104.23 | step: 17.76

  6%|▌         | 2452/41250 [5:55:11<93:47:06,  8.70s/it]
                                                         
{'loss': 0.1031, 'grad_norm': 2.110301971435547, 'learning_rate': 3.9909212145574125e-05, 'epoch': 0.59}

  6%|▌         | 2452/41250 [5:55:11<93:47:06,  8.70s/it][2025-04-26 10:03:48,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:03:48,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.92 | bwd_microstep: 5767.46 | bwd_inner_microstep: 5703.76 | bwd_allreduce_microstep: 63.66 | step_microstep: 18.21
[2025-04-26 10:03:48,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.92 | bwd: 5767.47 | bwd_inner: 5703.76 | bwd_allreduce: 63.68 | step: 18.21

  6%|▌         | 2453/41250 [5:55:19<93:46:02,  8.70s/it]
                                                         
{'loss': 0.0699, 'grad_norm': 1.1182606220245361, 'learning_rate': 3.9909062629337826e-05, 'epoch': 0.59}

  6%|▌         | 2453/41250 [5:55:19<93:46:02,  8.70s/it][2025-04-26 10:03:57,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 10:03:57,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.86 | bwd_microstep: 5771.87 | bwd_inner_microstep: 5691.04 | bwd_allreduce_microstep: 80.77 | step_microstep: 18.52
[2025-04-26 10:03:57,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.86 | bwd: 5771.88 | bwd_inner: 5691.04 | bwd_allreduce: 80.79 | step: 18.52

  6%|▌         | 2454/41250 [5:55:28<93:48:18,  8.70s/it]
                                                         
{'loss': 0.092, 'grad_norm': 1.5802267789840698, 'learning_rate': 3.990891299036609e-05, 'epoch': 0.59}

  6%|▌         | 2454/41250 [5:55:28<93:48:18,  8.70s/it][2025-04-26 10:04:05,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:04:05,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5773.52 | bwd_inner_microstep: 5679.73 | bwd_allreduce_microstep: 93.74 | step_microstep: 18.58
[2025-04-26 10:04:05,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5773.53 | bwd_inner: 5679.73 | bwd_allreduce: 93.77 | step: 18.59

  6%|▌         | 2455/41250 [5:55:37<93:48:19,  8.70s/it]
                                                         
{'loss': 0.1579, 'grad_norm': 1.3919119834899902, 'learning_rate': 3.990876322865981e-05, 'epoch': 0.6}

  6%|▌         | 2455/41250 [5:55:37<93:48:19,  8.70s/it][2025-04-26 10:04:14,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 10:04:14,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.84 | bwd_microstep: 5777.62 | bwd_inner_microstep: 5645.10 | bwd_allreduce_microstep: 132.47 | step_microstep: 18.57
[2025-04-26 10:04:14,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.84 | bwd: 5777.63 | bwd_inner: 5645.10 | bwd_allreduce: 132.49 | step: 18.57

  6%|▌         | 2456/41250 [5:55:45<93:45:44,  8.70s/it]
                                                         
{'loss': 0.1404, 'grad_norm': 2.003324508666992, 'learning_rate': 3.9908613344219914e-05, 'epoch': 0.6}

  6%|▌         | 2456/41250 [5:55:45<93:45:44,  8.70s/it][2025-04-26 10:04:23,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.00
[2025-04-26 10:04:23,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.80 | bwd_microstep: 5781.00 | bwd_inner_microstep: 5648.29 | bwd_allreduce_microstep: 132.66 | step_microstep: 18.14
[2025-04-26 10:04:23,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.80 | bwd: 5781.01 | bwd_inner: 5648.29 | bwd_allreduce: 132.68 | step: 18.14

  6%|▌         | 2457/41250 [5:55:54<93:45:51,  8.70s/it]
                                                         
{'loss': 0.0421, 'grad_norm': 0.4101716876029968, 'learning_rate': 3.9908463337047344e-05, 'epoch': 0.6}

  6%|▌         | 2457/41250 [5:55:54<93:45:51,  8.70s/it][2025-04-26 10:04:32,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:04:32,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5701.60 | bwd_inner_microstep: 5688.72 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.43
[2025-04-26 10:04:32,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5701.61 | bwd_inner: 5688.72 | bwd_allreduce: 12.85 | step: 18.43

  6%|▌         | 2458/41250 [5:56:03<93:32:53,  8.68s/it]
                                                         
{'loss': 0.2638, 'grad_norm': 2.6165947914123535, 'learning_rate': 3.9908313207143006e-05, 'epoch': 0.6}

  6%|▌         | 2458/41250 [5:56:03<93:32:53,  8.68s/it][2025-04-26 10:04:40,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:04:40,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.07 | bwd_microstep: 5716.00 | bwd_inner_microstep: 5703.34 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.46
[2025-04-26 10:04:40,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.07 | bwd: 5716.01 | bwd_inner: 5703.33 | bwd_allreduce: 12.64 | step: 18.47

  6%|▌         | 2459/41250 [5:56:11<93:25:04,  8.67s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 1.9407566785812378, 'learning_rate': 3.9908162954507836e-05, 'epoch': 0.6}

  6%|▌         | 2459/41250 [5:56:11<93:25:04,  8.67s/it][2025-04-26 10:04:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:04:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5765.78 | bwd_inner_microstep: 5650.37 | bwd_allreduce_microstep: 115.37 | step_microstep: 18.14
[2025-04-26 10:04:49,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.99 | bwd: 5765.79 | bwd_inner: 5650.37 | bwd_allreduce: 115.39 | step: 18.15

  6%|▌         | 2460/41250 [5:56:20<93:28:49,  8.68s/it]
                                                         
{'loss': 0.2237, 'grad_norm': 1.9135774374008179, 'learning_rate': 3.990801257914276e-05, 'epoch': 0.6}

  6%|▌         | 2460/41250 [5:56:20<93:28:49,  8.68s/it][2025-04-26 10:04:58,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:04:58,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.39 | bwd_microstep: 5775.91 | bwd_inner_microstep: 5696.59 | bwd_allreduce_microstep: 79.28 | step_microstep: 18.36
[2025-04-26 10:04:58,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.39 | bwd: 5775.92 | bwd_inner: 5696.59 | bwd_allreduce: 79.30 | step: 18.36

  6%|▌         | 2461/41250 [5:56:29<93:34:39,  8.68s/it]
                                                         
{'loss': 0.3471, 'grad_norm': 2.056739330291748, 'learning_rate': 3.990786208104871e-05, 'epoch': 0.6}

  6%|▌         | 2461/41250 [5:56:29<93:34:39,  8.68s/it][2025-04-26 10:05:06,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 10:05:06,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5748.08 | bwd_inner_microstep: 5694.84 | bwd_allreduce_microstep: 53.20 | step_microstep: 18.39
[2025-04-26 10:05:06,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.69 | bwd: 5748.09 | bwd_inner: 5694.84 | bwd_allreduce: 53.22 | step: 18.39

  6%|▌         | 2462/41250 [5:56:37<93:34:46,  8.69s/it]
                                                         
{'loss': 0.1842, 'grad_norm': 3.4852402210235596, 'learning_rate': 3.990771146022659e-05, 'epoch': 0.6}

  6%|▌         | 2462/41250 [5:56:37<93:34:46,  8.69s/it][2025-04-26 10:05:15,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:05:15,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.02 | bwd_microstep: 5928.05 | bwd_inner_microstep: 5655.06 | bwd_allreduce_microstep: 272.95 | step_microstep: 18.29
[2025-04-26 10:05:15,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.02 | bwd: 5928.07 | bwd_inner: 5655.06 | bwd_allreduce: 272.97 | step: 18.29

  6%|▌         | 2463/41250 [5:56:46<94:05:27,  8.73s/it]
                                                         
{'loss': 0.0669, 'grad_norm': 0.6924973130226135, 'learning_rate': 3.990756071667735e-05, 'epoch': 0.6}

  6%|▌         | 2463/41250 [5:56:46<94:05:27,  8.73s/it][2025-04-26 10:05:24,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.11 | optimizer_step: 0.90
[2025-04-26 10:05:24,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.13 | bwd_microstep: 5795.70 | bwd_inner_microstep: 5661.18 | bwd_allreduce_microstep: 134.48 | step_microstep: 19.52
[2025-04-26 10:05:24,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.13 | bwd: 5795.72 | bwd_inner: 5661.18 | bwd_allreduce: 134.49 | step: 19.52

  6%|▌         | 2464/41250 [5:56:55<93:58:33,  8.72s/it]
                                                         
{'loss': 0.5045, 'grad_norm': 2.520977735519409, 'learning_rate': 3.990740985040191e-05, 'epoch': 0.6}

  6%|▌         | 2464/41250 [5:56:55<93:58:33,  8.72s/it][2025-04-26 10:05:32,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:05:32,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.73 | bwd_microstep: 5732.57 | bwd_inner_microstep: 5657.58 | bwd_allreduce_microstep: 74.94 | step_microstep: 17.73
[2025-04-26 10:05:32,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.73 | bwd: 5732.58 | bwd_inner: 5657.58 | bwd_allreduce: 74.96 | step: 17.73

  6%|▌         | 2465/41250 [5:57:04<93:42:28,  8.70s/it]
                                                         
{'loss': 0.2677, 'grad_norm': 1.8610862493515015, 'learning_rate': 3.990725886140121e-05, 'epoch': 0.6}

  6%|▌         | 2465/41250 [5:57:04<93:42:28,  8.70s/it][2025-04-26 10:05:41,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:05:41,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.78 | bwd_microstep: 5797.35 | bwd_inner_microstep: 5689.99 | bwd_allreduce_microstep: 107.32 | step_microstep: 18.20
[2025-04-26 10:05:41,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.78 | bwd: 5797.36 | bwd_inner: 5689.99 | bwd_allreduce: 107.34 | step: 18.21

  6%|▌         | 2466/41250 [5:57:12<93:47:08,  8.71s/it]
                                                         
{'loss': 0.1321, 'grad_norm': 1.551483392715454, 'learning_rate': 3.9907107749676175e-05, 'epoch': 0.6}

  6%|▌         | 2466/41250 [5:57:12<93:47:08,  8.71s/it][2025-04-26 10:05:50,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:05:50,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.10 | bwd_microstep: 5714.27 | bwd_inner_microstep: 5700.89 | bwd_allreduce_microstep: 13.33 | step_microstep: 18.49
[2025-04-26 10:05:50,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.10 | bwd: 5714.28 | bwd_inner: 5700.89 | bwd_allreduce: 13.35 | step: 18.50

  6%|▌         | 2467/41250 [5:57:21<93:36:15,  8.69s/it]
                                                         
{'loss': 0.2339, 'grad_norm': 2.358205795288086, 'learning_rate': 3.9906956515227735e-05, 'epoch': 0.6}

  6%|▌         | 2467/41250 [5:57:21<93:36:15,  8.69s/it][2025-04-26 10:05:58,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:05:58,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5715.67 | bwd_inner_microstep: 5686.36 | bwd_allreduce_microstep: 29.27 | step_microstep: 18.11
[2025-04-26 10:05:58,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5715.69 | bwd_inner: 5686.36 | bwd_allreduce: 29.29 | step: 18.11

  6%|▌         | 2468/41250 [5:57:30<93:25:39,  8.67s/it]
                                                         
{'loss': 0.2618, 'grad_norm': 2.867516279220581, 'learning_rate': 3.9906805158056825e-05, 'epoch': 0.6}

  6%|▌         | 2468/41250 [5:57:30<93:25:39,  8.67s/it][2025-04-26 10:06:07,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:06:07,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.77 | bwd_microstep: 5710.24 | bwd_inner_microstep: 5696.94 | bwd_allreduce_microstep: 13.25 | step_microstep: 18.13
[2025-04-26 10:06:07,577] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.77 | bwd: 5710.25 | bwd_inner: 5696.94 | bwd_allreduce: 13.27 | step: 18.14

  6%|▌         | 2469/41250 [5:57:38<93:22:47,  8.67s/it]
                                                         
{'loss': 0.4268, 'grad_norm': 1.786810040473938, 'learning_rate': 3.990665367816437e-05, 'epoch': 0.6}

  6%|▌         | 2469/41250 [5:57:38<93:22:47,  8.67s/it][2025-04-26 10:06:16,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 10:06:16,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.23 | bwd_microstep: 5711.96 | bwd_inner_microstep: 5671.65 | bwd_allreduce_microstep: 40.27 | step_microstep: 18.61
[2025-04-26 10:06:16,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.23 | bwd: 5711.98 | bwd_inner: 5671.65 | bwd_allreduce: 40.28 | step: 18.61

  6%|▌         | 2470/41250 [5:57:47<93:15:20,  8.66s/it]
                                                         
{'loss': 0.1538, 'grad_norm': 1.2489207983016968, 'learning_rate': 3.9906502075551314e-05, 'epoch': 0.6}

  6%|▌         | 2470/41250 [5:57:47<93:15:20,  8.66s/it][2025-04-26 10:06:24,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:06:24,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5721.58 | bwd_inner_microstep: 5650.74 | bwd_allreduce_microstep: 70.80 | step_microstep: 17.71
[2025-04-26 10:06:24,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.97 | bwd: 5721.59 | bwd_inner: 5650.74 | bwd_allreduce: 70.81 | step: 17.72

  6%|▌         | 2471/41250 [5:57:56<93:10:47,  8.65s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.9853132367134094, 'learning_rate': 3.990635035021859e-05, 'epoch': 0.6}

  6%|▌         | 2471/41250 [5:57:56<93:10:47,  8.65s/it][2025-04-26 10:06:33,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:06:33,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5776.91 | bwd_inner_microstep: 5639.33 | bwd_allreduce_microstep: 137.54 | step_microstep: 18.20
[2025-04-26 10:06:33,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.93 | bwd: 5776.92 | bwd_inner: 5639.33 | bwd_allreduce: 137.56 | step: 18.20

  6%|▌         | 2472/41250 [5:58:04<93:16:55,  8.66s/it]
                                                         
{'loss': 0.0582, 'grad_norm': 0.557584285736084, 'learning_rate': 3.990619850216713e-05, 'epoch': 0.6}

  6%|▌         | 2472/41250 [5:58:04<93:16:55,  8.66s/it][2025-04-26 10:06:42,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:06:42,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.34 | bwd_microstep: 5737.40 | bwd_inner_microstep: 5710.74 | bwd_allreduce_microstep: 26.63 | step_microstep: 17.72
[2025-04-26 10:06:42,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.34 | bwd: 5737.42 | bwd_inner: 5710.74 | bwd_allreduce: 26.64 | step: 17.73

  6%|▌         | 2473/41250 [5:58:13<93:19:22,  8.66s/it]
                                                         
{'loss': 0.2872, 'grad_norm': 2.3224589824676514, 'learning_rate': 3.9906046531397876e-05, 'epoch': 0.6}

  6%|▌         | 2473/41250 [5:58:13<93:19:22,  8.66s/it][2025-04-26 10:06:50,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:06:50,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5742.34 | bwd_inner_microstep: 5682.52 | bwd_allreduce_microstep: 59.78 | step_microstep: 18.43
[2025-04-26 10:06:50,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5742.35 | bwd_inner: 5682.52 | bwd_allreduce: 59.79 | step: 18.43

  6%|▌         | 2474/41250 [5:58:22<93:21:44,  8.67s/it]
                                                         
{'loss': 0.037, 'grad_norm': 0.4424472153186798, 'learning_rate': 3.990589443791175e-05, 'epoch': 0.6}

  6%|▌         | 2474/41250 [5:58:22<93:21:44,  8.67s/it][2025-04-26 10:06:59,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:06:59,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.40 | bwd_microstep: 5698.36 | bwd_inner_microstep: 5685.41 | bwd_allreduce_microstep: 12.91 | step_microstep: 17.88
[2025-04-26 10:06:59,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.40 | bwd: 5698.37 | bwd_inner: 5685.41 | bwd_allreduce: 12.92 | step: 17.88

  6%|▌         | 2475/41250 [5:58:30<93:14:01,  8.66s/it]
                                                         
{'loss': 0.1933, 'grad_norm': 1.587318778038025, 'learning_rate': 3.99057422217097e-05, 'epoch': 0.6}

  6%|▌         | 2475/41250 [5:58:30<93:14:01,  8.66s/it][2025-04-26 10:07:08,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:07:08,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.28 | bwd_microstep: 5772.42 | bwd_inner_microstep: 5654.52 | bwd_allreduce_microstep: 117.86 | step_microstep: 17.75
[2025-04-26 10:07:08,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.28 | bwd: 5772.43 | bwd_inner: 5654.52 | bwd_allreduce: 117.88 | step: 17.75

  6%|▌         | 2476/41250 [5:58:39<93:17:45,  8.66s/it]
                                                         
{'loss': 0.2804, 'grad_norm': 1.8940171003341675, 'learning_rate': 3.9905589882792675e-05, 'epoch': 0.6}

  6%|▌         | 2476/41250 [5:58:39<93:17:45,  8.66s/it][2025-04-26 10:07:16,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:07:16,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5770.84 | bwd_inner_microstep: 5708.69 | bwd_allreduce_microstep: 62.11 | step_microstep: 17.67
[2025-04-26 10:07:16,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5770.85 | bwd_inner: 5708.69 | bwd_allreduce: 62.13 | step: 17.67

  6%|▌         | 2477/41250 [5:58:48<93:24:17,  8.67s/it]
                                                         
{'loss': 0.1401, 'grad_norm': 1.68311607837677, 'learning_rate': 3.99054374211616e-05, 'epoch': 0.6}

  6%|▌         | 2477/41250 [5:58:48<93:24:17,  8.67s/it][2025-04-26 10:07:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:07:25,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.02 | bwd_microstep: 5785.95 | bwd_inner_microstep: 5772.91 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.24
[2025-04-26 10:07:25,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.02 | bwd: 5785.96 | bwd_inner: 5772.91 | bwd_allreduce: 13.01 | step: 18.24

  6%|▌         | 2478/41250 [5:58:56<93:42:16,  8.70s/it]
                                                         
{'loss': 0.2346, 'grad_norm': 2.041496753692627, 'learning_rate': 3.990528483681742e-05, 'epoch': 0.6}

  6%|▌         | 2478/41250 [5:58:56<93:42:16,  8.70s/it][2025-04-26 10:07:34,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.80 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 10:07:34,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.72 | bwd_microstep: 5761.52 | bwd_inner_microstep: 5647.46 | bwd_allreduce_microstep: 114.02 | step_microstep: 18.63
[2025-04-26 10:07:34,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.72 | bwd: 5761.54 | bwd_inner: 5647.46 | bwd_allreduce: 114.03 | step: 18.64

  6%|▌         | 2479/41250 [5:59:05<93:39:38,  8.70s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 1.1328402757644653, 'learning_rate': 3.990513212976107e-05, 'epoch': 0.6}

  6%|▌         | 2479/41250 [5:59:05<93:39:38,  8.70s/it][2025-04-26 10:07:42,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 10:07:42,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.79 | bwd_microstep: 5692.88 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.51
[2025-04-26 10:07:42,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.79 | bwd: 5692.89 | bwd_inner: 5679.98 | bwd_allreduce: 12.87 | step: 18.51

  6%|▌         | 2480/41250 [5:59:14<93:23:49,  8.67s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.7014107704162598, 'learning_rate': 3.99049792999935e-05, 'epoch': 0.6}

  6%|▌         | 2480/41250 [5:59:14<93:23:49,  8.67s/it][2025-04-26 10:07:51,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.95 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:07:51,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.13 | bwd_microstep: 5703.91 | bwd_inner_microstep: 5647.49 | bwd_allreduce_microstep: 56.38 | step_microstep: 18.98
[2025-04-26 10:07:51,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.13 | bwd: 5703.93 | bwd_inner: 5647.49 | bwd_allreduce: 56.39 | step: 18.98

  6%|▌         | 2481/41250 [5:59:22<93:11:18,  8.65s/it]
                                                         
{'loss': 0.1494, 'grad_norm': 2.9662811756134033, 'learning_rate': 3.9904826347515645e-05, 'epoch': 0.6}

  6%|▌         | 2481/41250 [5:59:22<93:11:18,  8.65s/it][2025-04-26 10:08:00,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:08:00,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.56 | bwd_microstep: 5734.53 | bwd_inner_microstep: 5694.63 | bwd_allreduce_microstep: 39.85 | step_microstep: 18.09
[2025-04-26 10:08:00,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.56 | bwd: 5734.54 | bwd_inner: 5694.63 | bwd_allreduce: 39.87 | step: 18.09

  6%|▌         | 2482/41250 [5:59:31<93:14:30,  8.66s/it]
                                                         
{'loss': 0.393, 'grad_norm': 1.7749072313308716, 'learning_rate': 3.990467327232845e-05, 'epoch': 0.6}

  6%|▌         | 2482/41250 [5:59:31<93:14:30,  8.66s/it][2025-04-26 10:08:08,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:08:08,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.53 | bwd_microstep: 5732.86 | bwd_inner_microstep: 5682.55 | bwd_allreduce_microstep: 50.27 | step_microstep: 18.17
[2025-04-26 10:08:08,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.53 | bwd: 5732.87 | bwd_inner: 5682.55 | bwd_allreduce: 50.28 | step: 18.17

  6%|▌         | 2483/41250 [5:59:40<93:14:02,  8.66s/it]
                                                         
{'loss': 0.1004, 'grad_norm': 1.5938000679016113, 'learning_rate': 3.9904520074432856e-05, 'epoch': 0.6}

  6%|▌         | 2483/41250 [5:59:40<93:14:02,  8.66s/it][2025-04-26 10:08:17,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:08:17,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.94 | bwd_microstep: 5737.49 | bwd_inner_microstep: 5672.40 | bwd_allreduce_microstep: 65.05 | step_microstep: 17.91
[2025-04-26 10:08:17,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.94 | bwd: 5737.50 | bwd_inner: 5672.40 | bwd_allreduce: 65.06 | step: 17.92

  6%|▌         | 2484/41250 [5:59:48<93:14:08,  8.66s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.5363187789916992, 'learning_rate': 3.9904366753829816e-05, 'epoch': 0.6}

  6%|▌         | 2484/41250 [5:59:48<93:14:08,  8.66s/it][2025-04-26 10:08:26,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:08:26,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.88 | bwd_microstep: 5711.08 | bwd_inner_microstep: 5697.63 | bwd_allreduce_microstep: 13.40 | step_microstep: 18.34
[2025-04-26 10:08:26,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.88 | bwd: 5711.09 | bwd_inner: 5697.63 | bwd_allreduce: 13.42 | step: 18.35

  6%|▌         | 2485/41250 [5:59:57<93:10:15,  8.65s/it]
                                                         
{'loss': 0.0507, 'grad_norm': 0.5693524479866028, 'learning_rate': 3.990421331052027e-05, 'epoch': 0.6}

  6%|▌         | 2485/41250 [5:59:57<93:10:15,  8.65s/it][2025-04-26 10:08:34,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:08:34,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.11 | bwd_microstep: 5788.42 | bwd_inner_microstep: 5775.19 | bwd_allreduce_microstep: 13.19 | step_microstep: 18.19
[2025-04-26 10:08:34,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.11 | bwd: 5788.43 | bwd_inner: 5775.19 | bwd_allreduce: 13.21 | step: 18.20

  6%|▌         | 2486/41250 [6:00:06<93:29:47,  8.68s/it]
                                                         
{'loss': 0.088, 'grad_norm': 1.211927056312561, 'learning_rate': 3.990405974450516e-05, 'epoch': 0.6}

  6%|▌         | 2486/41250 [6:00:06<93:29:47,  8.68s/it][2025-04-26 10:08:43,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.62 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:08:43,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.94 | bwd_microstep: 5748.62 | bwd_inner_microstep: 5647.77 | bwd_allreduce_microstep: 100.81 | step_microstep: 18.18
[2025-04-26 10:08:43,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.94 | bwd: 5748.63 | bwd_inner: 5647.77 | bwd_allreduce: 100.83 | step: 18.18

  6%|▌         | 2487/41250 [6:00:14<93:25:46,  8.68s/it]
                                                         
{'loss': 0.1508, 'grad_norm': 1.047669529914856, 'learning_rate': 3.9903906055785447e-05, 'epoch': 0.6}

  6%|▌         | 2487/41250 [6:00:14<93:25:46,  8.68s/it][2025-04-26 10:08:52,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:08:52,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.40 | bwd_microstep: 5700.11 | bwd_inner_microstep: 5687.42 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.69
[2025-04-26 10:08:52,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.40 | bwd: 5700.12 | bwd_inner: 5687.42 | bwd_allreduce: 12.66 | step: 17.70

  6%|▌         | 2488/41250 [6:00:23<93:17:45,  8.66s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 1.63936185836792, 'learning_rate': 3.990375224436206e-05, 'epoch': 0.6}

  6%|▌         | 2488/41250 [6:00:23<93:17:45,  8.66s/it][2025-04-26 10:09:00,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:09:00,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.90 | bwd_microstep: 5735.15 | bwd_inner_microstep: 5689.90 | bwd_allreduce_microstep: 45.20 | step_microstep: 17.71
[2025-04-26 10:09:00,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.90 | bwd: 5735.16 | bwd_inner: 5689.90 | bwd_allreduce: 45.22 | step: 17.71

  6%|▌         | 2489/41250 [6:00:32<93:17:55,  8.67s/it]
                                                         
{'loss': 0.1704, 'grad_norm': 1.4914116859436035, 'learning_rate': 3.990359831023595e-05, 'epoch': 0.6}

  6%|▌         | 2489/41250 [6:00:32<93:17:55,  8.67s/it][2025-04-26 10:09:09,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:09:09,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.34 | bwd_microstep: 5716.85 | bwd_inner_microstep: 5701.99 | bwd_allreduce_microstep: 14.82 | step_microstep: 17.80
[2025-04-26 10:09:09,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.34 | bwd: 5716.86 | bwd_inner: 5701.99 | bwd_allreduce: 14.84 | step: 17.80

  6%|▌         | 2490/41250 [6:00:40<93:15:56,  8.66s/it]
                                                         
{'loss': 0.2467, 'grad_norm': 1.079392671585083, 'learning_rate': 3.9903444253408075e-05, 'epoch': 0.6}

  6%|▌         | 2490/41250 [6:00:40<93:15:56,  8.66s/it][2025-04-26 10:09:18,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:09:18,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5766.20 | bwd_inner_microstep: 5646.65 | bwd_allreduce_microstep: 119.50 | step_microstep: 18.06
[2025-04-26 10:09:18,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5766.21 | bwd_inner: 5646.65 | bwd_allreduce: 119.52 | step: 18.06

  6%|▌         | 2491/41250 [6:00:49<93:17:04,  8.66s/it]
                                                         
{'loss': 0.104, 'grad_norm': 0.9291595816612244, 'learning_rate': 3.9903290073879387e-05, 'epoch': 0.6}

  6%|▌         | 2491/41250 [6:00:49<93:17:04,  8.66s/it][2025-04-26 10:09:26,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:09:26,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.40 | bwd_microstep: 5724.54 | bwd_inner_microstep: 5676.23 | bwd_allreduce_microstep: 48.27 | step_microstep: 18.09
[2025-04-26 10:09:26,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.40 | bwd: 5724.55 | bwd_inner: 5676.23 | bwd_allreduce: 48.28 | step: 18.10

  6%|▌         | 2492/41250 [6:00:58<93:16:27,  8.66s/it]
                                                         
{'loss': 0.0567, 'grad_norm': 0.8397553563117981, 'learning_rate': 3.9903135771650827e-05, 'epoch': 0.6}

  6%|▌         | 2492/41250 [6:00:58<93:16:27,  8.66s/it][2025-04-26 10:09:35,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.95 | optimizer_step: 1.02
[2025-04-26 10:09:35,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.69 | bwd_microstep: 5731.55 | bwd_inner_microstep: 5682.63 | bwd_allreduce_microstep: 48.88 | step_microstep: 18.25
[2025-04-26 10:09:35,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.69 | bwd: 5731.56 | bwd_inner: 5682.63 | bwd_allreduce: 48.89 | step: 18.26

  6%|▌         | 2493/41250 [6:01:06<93:15:56,  8.66s/it]
                                                         
{'loss': 0.339, 'grad_norm': 1.7598600387573242, 'learning_rate': 3.990298134672335e-05, 'epoch': 0.6}

  6%|▌         | 2493/41250 [6:01:06<93:15:56,  8.66s/it][2025-04-26 10:09:44,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 10:09:44,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.69 | bwd_microstep: 5723.61 | bwd_inner_microstep: 5683.77 | bwd_allreduce_microstep: 39.80 | step_microstep: 18.13
[2025-04-26 10:09:44,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.70 | bwd: 5723.62 | bwd_inner: 5683.77 | bwd_allreduce: 39.81 | step: 18.13

  6%|▌         | 2494/41250 [6:01:15<93:14:25,  8.66s/it]
                                                         
{'loss': 0.1416, 'grad_norm': 1.1188124418258667, 'learning_rate': 3.990282679909791e-05, 'epoch': 0.6}

  6%|▌         | 2494/41250 [6:01:15<93:14:25,  8.66s/it][2025-04-26 10:09:52,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:09:52,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.54 | bwd_microstep: 5731.31 | bwd_inner_microstep: 5690.17 | bwd_allreduce_microstep: 41.09 | step_microstep: 18.22
[2025-04-26 10:09:52,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.54 | bwd: 5731.32 | bwd_inner: 5690.17 | bwd_allreduce: 41.11 | step: 18.22

  6%|▌         | 2495/41250 [6:01:24<93:14:48,  8.66s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.5867352485656738, 'learning_rate': 3.9902672128775446e-05, 'epoch': 0.6}

  6%|▌         | 2495/41250 [6:01:24<93:14:48,  8.66s/it][2025-04-26 10:10:01,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:10:01,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.77 | bwd_microstep: 5688.69 | bwd_inner_microstep: 5647.52 | bwd_allreduce_microstep: 41.13 | step_microstep: 18.02
[2025-04-26 10:10:01,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.77 | bwd: 5688.71 | bwd_inner: 5647.52 | bwd_allreduce: 41.15 | step: 18.02

  6%|▌         | 2496/41250 [6:01:32<93:03:16,  8.64s/it]
                                                         
{'loss': 0.2503, 'grad_norm': 1.9915823936462402, 'learning_rate': 3.9902517335756934e-05, 'epoch': 0.61}

  6%|▌         | 2496/41250 [6:01:32<93:03:16,  8.64s/it][2025-04-26 10:10:10,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:10:10,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.56 | bwd_microstep: 5718.47 | bwd_inner_microstep: 5705.74 | bwd_allreduce_microstep: 12.69 | step_microstep: 17.66
[2025-04-26 10:10:10,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.56 | bwd: 5718.48 | bwd_inner: 5705.74 | bwd_allreduce: 12.70 | step: 17.67

  6%|▌         | 2497/41250 [6:01:41<93:07:05,  8.65s/it]
                                                         
{'loss': 0.2795, 'grad_norm': 1.3885900974273682, 'learning_rate': 3.9902362420043315e-05, 'epoch': 0.61}

  6%|▌         | 2497/41250 [6:01:41<93:07:05,  8.65s/it][2025-04-26 10:10:18,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:10:18,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.38 | bwd_microstep: 5704.20 | bwd_inner_microstep: 5642.15 | bwd_allreduce_microstep: 62.01 | step_microstep: 18.35
[2025-04-26 10:10:18,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.38 | bwd: 5704.22 | bwd_inner: 5642.15 | bwd_allreduce: 62.03 | step: 18.35

  6%|▌         | 2498/41250 [6:01:49<92:58:33,  8.64s/it]
                                                         
{'loss': 0.0722, 'grad_norm': 0.6972960829734802, 'learning_rate': 3.9902207381635544e-05, 'epoch': 0.61}

  6%|▌         | 2498/41250 [6:01:49<92:58:33,  8.64s/it][2025-04-26 10:10:27,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:10:27,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.39 | bwd_microstep: 5776.60 | bwd_inner_microstep: 5650.83 | bwd_allreduce_microstep: 125.73 | step_microstep: 17.75
[2025-04-26 10:10:27,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.39 | bwd: 5776.62 | bwd_inner: 5650.83 | bwd_allreduce: 125.75 | step: 17.75

  6%|▌         | 2499/41250 [6:01:58<93:11:36,  8.66s/it]
                                                         
{'loss': 0.2611, 'grad_norm': 1.727239727973938, 'learning_rate': 3.990205222053458e-05, 'epoch': 0.61}

  6%|▌         | 2499/41250 [6:01:58<93:11:36,  8.66s/it][2025-04-26 10:10:36,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:10:36,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.79 | bwd_microstep: 5723.54 | bwd_inner_microstep: 5710.75 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.78
[2025-04-26 10:10:36,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.79 | bwd: 5723.56 | bwd_inner: 5710.74 | bwd_allreduce: 12.77 | step: 17.79

  6%|▌         | 2500/41250 [6:02:07<93:11:59,  8.66s/it]
                                                         
{'loss': 0.0469, 'grad_norm': 0.4491344392299652, 'learning_rate': 3.990189693674138e-05, 'epoch': 0.61}

  6%|▌         | 2500/41250 [6:02:07<93:11:59,  8.66s/it][2025-04-26 10:10:44,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:10:44,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.71 | bwd_microstep: 5878.77 | bwd_inner_microstep: 5694.76 | bwd_allreduce_microstep: 183.98 | step_microstep: 17.82
[2025-04-26 10:10:44,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.71 | bwd: 5878.79 | bwd_inner: 5694.76 | bwd_allreduce: 183.99 | step: 17.82

  6%|▌         | 2501/41250 [6:02:16<93:44:06,  8.71s/it]
                                                         
{'loss': 0.2246, 'grad_norm': 2.0324113368988037, 'learning_rate': 3.99017415302569e-05, 'epoch': 0.61}

  6%|▌         | 2501/41250 [6:02:16<93:44:06,  8.71s/it][2025-04-26 10:10:53,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:10:53,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.74 | bwd_microstep: 5725.73 | bwd_inner_microstep: 5712.87 | bwd_allreduce_microstep: 12.82 | step_microstep: 17.83
[2025-04-26 10:10:53,607] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.74 | bwd: 5725.74 | bwd_inner: 5712.87 | bwd_allreduce: 12.83 | step: 17.84

  6%|▌         | 2502/41250 [6:02:24<93:36:32,  8.70s/it]
                                                         
{'loss': 0.1847, 'grad_norm': 1.3628151416778564, 'learning_rate': 3.99015860010821e-05, 'epoch': 0.61}

  6%|▌         | 2502/41250 [6:02:24<93:36:32,  8.70s/it][2025-04-26 10:11:02,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:11:02,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.42 | bwd_microstep: 5719.02 | bwd_inner_microstep: 5706.44 | bwd_allreduce_microstep: 12.55 | step_microstep: 17.84
[2025-04-26 10:11:02,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.42 | bwd: 5719.03 | bwd_inner: 5706.44 | bwd_allreduce: 12.56 | step: 17.84

  6%|▌         | 2503/41250 [6:02:33<93:29:42,  8.69s/it]
                                                         
{'loss': 0.2463, 'grad_norm': 1.5738922357559204, 'learning_rate': 3.9901430349217934e-05, 'epoch': 0.61}

  6%|▌         | 2503/41250 [6:02:33<93:29:42,  8.69s/it][2025-04-26 10:11:10,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:11:10,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.92 | bwd_microstep: 5751.80 | bwd_inner_microstep: 5719.01 | bwd_allreduce_microstep: 32.75 | step_microstep: 17.70
[2025-04-26 10:11:10,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.93 | bwd: 5751.81 | bwd_inner: 5719.01 | bwd_allreduce: 32.77 | step: 17.70

  6%|▌         | 2504/41250 [6:02:42<93:30:30,  8.69s/it]
                                                         
{'loss': 0.3613, 'grad_norm': 3.551499366760254, 'learning_rate': 3.990127457466536e-05, 'epoch': 0.61}

  6%|▌         | 2504/41250 [6:02:42<93:30:30,  8.69s/it][2025-04-26 10:11:19,667] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 10:11:19,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.52 | bwd_microstep: 5764.96 | bwd_inner_microstep: 5704.58 | bwd_allreduce_microstep: 60.34 | step_microstep: 18.45
[2025-04-26 10:11:19,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.52 | bwd: 5764.97 | bwd_inner: 5704.58 | bwd_allreduce: 60.35 | step: 18.45

  6%|▌         | 2505/41250 [6:02:50<93:34:00,  8.69s/it]
                                                         
{'loss': 0.252, 'grad_norm': 2.076812982559204, 'learning_rate': 3.9901118677425346e-05, 'epoch': 0.61}

  6%|▌         | 2505/41250 [6:02:50<93:34:00,  8.69s/it][2025-04-26 10:11:28,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:11:28,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.27 | bwd_microstep: 5758.06 | bwd_inner_microstep: 5668.37 | bwd_allreduce_microstep: 89.65 | step_microstep: 18.24
[2025-04-26 10:11:28,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.27 | bwd: 5758.08 | bwd_inner: 5668.37 | bwd_allreduce: 89.67 | step: 18.24

  6%|▌         | 2506/41250 [6:02:59<93:31:47,  8.69s/it]
                                                         
{'loss': 0.1613, 'grad_norm': 1.5944896936416626, 'learning_rate': 3.990096265749885e-05, 'epoch': 0.61}

  6%|▌         | 2506/41250 [6:02:59<93:31:47,  8.69s/it][2025-04-26 10:11:37,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:11:37,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.56 | bwd_microstep: 5746.75 | bwd_inner_microstep: 5715.78 | bwd_allreduce_microstep: 30.92 | step_microstep: 18.26
[2025-04-26 10:11:37,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.56 | bwd: 5746.76 | bwd_inner: 5715.78 | bwd_allreduce: 30.94 | step: 18.27

  6%|▌         | 2507/41250 [6:03:08<93:34:00,  8.69s/it]
                                                         
{'loss': 0.0744, 'grad_norm': 0.7545232772827148, 'learning_rate': 3.990080651488684e-05, 'epoch': 0.61}

  6%|▌         | 2507/41250 [6:03:08<93:34:00,  8.69s/it][2025-04-26 10:11:45,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:11:45,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.55 | bwd_microstep: 5887.08 | bwd_inner_microstep: 5655.88 | bwd_allreduce_microstep: 231.16 | step_microstep: 18.19
[2025-04-26 10:11:45,851] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.55 | bwd: 5887.09 | bwd_inner: 5655.88 | bwd_allreduce: 231.18 | step: 18.20

  6%|▌         | 2508/41250 [6:03:17<93:53:47,  8.73s/it]
                                                         
{'loss': 0.1805, 'grad_norm': 1.3895978927612305, 'learning_rate': 3.990065024959026e-05, 'epoch': 0.61}

  6%|▌         | 2508/41250 [6:03:17<93:53:47,  8.73s/it][2025-04-26 10:11:54,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 10:11:54,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.92 | bwd_microstep: 5792.24 | bwd_inner_microstep: 5661.81 | bwd_allreduce_microstep: 130.38 | step_microstep: 17.77
[2025-04-26 10:11:54,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.92 | bwd: 5792.25 | bwd_inner: 5661.81 | bwd_allreduce: 130.40 | step: 17.77

  6%|▌         | 2509/41250 [6:03:25<93:49:22,  8.72s/it]
                                                         
{'loss': 0.0914, 'grad_norm': 1.3448301553726196, 'learning_rate': 3.990049386161009e-05, 'epoch': 0.61}

  6%|▌         | 2509/41250 [6:03:25<93:49:22,  8.72s/it][2025-04-26 10:12:03,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:12:03,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.58 | bwd_microstep: 5729.13 | bwd_inner_microstep: 5670.74 | bwd_allreduce_microstep: 58.35 | step_microstep: 17.82
[2025-04-26 10:12:03,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.58 | bwd: 5729.14 | bwd_inner: 5670.74 | bwd_allreduce: 58.37 | step: 17.83

  6%|▌         | 2510/41250 [6:03:34<93:35:19,  8.70s/it]
                                                         
{'loss': 0.149, 'grad_norm': 2.0535621643066406, 'learning_rate': 3.9900337350947295e-05, 'epoch': 0.61}

  6%|▌         | 2510/41250 [6:03:34<93:35:19,  8.70s/it][2025-04-26 10:12:11,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:12:11,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.77 | bwd_microstep: 5745.62 | bwd_inner_microstep: 5700.83 | bwd_allreduce_microstep: 44.74 | step_microstep: 17.85
[2025-04-26 10:12:11,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.77 | bwd: 5745.63 | bwd_inner: 5700.83 | bwd_allreduce: 44.76 | step: 17.85

  6%|▌         | 2511/41250 [6:03:43<93:32:50,  8.69s/it]
                                                         
{'loss': 0.1479, 'grad_norm': 1.1964058876037598, 'learning_rate': 3.990018071760283e-05, 'epoch': 0.61}

  6%|▌         | 2511/41250 [6:03:43<93:32:50,  8.69s/it][2025-04-26 10:12:20,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 10:12:20,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.51 | bwd_microstep: 5734.21 | bwd_inner_microstep: 5721.28 | bwd_allreduce_microstep: 12.88 | step_microstep: 19.17
[2025-04-26 10:12:20,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.51 | bwd: 5734.23 | bwd_inner: 5721.28 | bwd_allreduce: 12.90 | step: 19.17

  6%|▌         | 2512/41250 [6:03:51<93:29:21,  8.69s/it]
                                                         
{'loss': 0.3047, 'grad_norm': 2.228853940963745, 'learning_rate': 3.9900023961577664e-05, 'epoch': 0.61}

  6%|▌         | 2512/41250 [6:03:51<93:29:21,  8.69s/it][2025-04-26 10:12:29,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:12:29,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.87 | bwd_microstep: 5773.52 | bwd_inner_microstep: 5714.55 | bwd_allreduce_microstep: 58.92 | step_microstep: 18.55
[2025-04-26 10:12:29,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.87 | bwd: 5773.53 | bwd_inner: 5714.55 | bwd_allreduce: 58.94 | step: 18.56

  6%|▌         | 2513/41250 [6:04:00<93:30:48,  8.69s/it]
                                                         
{'loss': 0.0783, 'grad_norm': 0.5370597243309021, 'learning_rate': 3.9899867082872765e-05, 'epoch': 0.61}

  6%|▌         | 2513/41250 [6:04:00<93:30:48,  8.69s/it][2025-04-26 10:12:37,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 10:12:37,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.36 | bwd_microstep: 5762.01 | bwd_inner_microstep: 5707.62 | bwd_allreduce_microstep: 54.35 | step_microstep: 18.32
[2025-04-26 10:12:37,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.36 | bwd: 5762.02 | bwd_inner: 5707.62 | bwd_allreduce: 54.36 | step: 18.32

  6%|▌         | 2514/41250 [6:04:09<93:33:06,  8.69s/it]
                                                         
{'loss': 0.2377, 'grad_norm': 3.2317988872528076, 'learning_rate': 3.98997100814891e-05, 'epoch': 0.61}

  6%|▌         | 2514/41250 [6:04:09<93:33:06,  8.69s/it][2025-04-26 10:12:46,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:12:46,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.85 | bwd_microstep: 5805.23 | bwd_inner_microstep: 5650.79 | bwd_allreduce_microstep: 154.39 | step_microstep: 18.39
[2025-04-26 10:12:46,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.85 | bwd: 5805.24 | bwd_inner: 5650.79 | bwd_allreduce: 154.41 | step: 18.39

  6%|▌         | 2515/41250 [6:04:17<93:38:22,  8.70s/it]
                                                         
{'loss': 0.0236, 'grad_norm': 0.19980792701244354, 'learning_rate': 3.989955295742764e-05, 'epoch': 0.61}

  6%|▌         | 2515/41250 [6:04:17<93:38:22,  8.70s/it][2025-04-26 10:12:55,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 10:12:55,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.12 | bwd_microstep: 5773.37 | bwd_inner_microstep: 5760.68 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.83
[2025-04-26 10:12:55,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.12 | bwd: 5773.38 | bwd_inner: 5760.68 | bwd_allreduce: 12.67 | step: 18.84

  6%|▌         | 2516/41250 [6:04:26<93:45:10,  8.71s/it]
                                                         
{'loss': 0.1084, 'grad_norm': 0.764387309551239, 'learning_rate': 3.9899395710689346e-05, 'epoch': 0.61}

  6%|▌         | 2516/41250 [6:04:26<93:45:10,  8.71s/it][2025-04-26 10:13:04,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:13:04,121] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.06 | bwd_microstep: 5747.39 | bwd_inner_microstep: 5722.02 | bwd_allreduce_microstep: 25.33 | step_microstep: 18.47
[2025-04-26 10:13:04,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.06 | bwd: 5747.40 | bwd_inner: 5722.03 | bwd_allreduce: 25.34 | step: 18.48

  6%|▌         | 2517/41250 [6:04:35<93:42:20,  8.71s/it]
                                                         
{'loss': 0.3109, 'grad_norm': 2.8751378059387207, 'learning_rate': 3.9899238341275195e-05, 'epoch': 0.61}

  6%|▌         | 2517/41250 [6:04:35<93:42:20,  8.71s/it][2025-04-26 10:13:12,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:13:12,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5787.18 | bwd_inner_microstep: 5650.16 | bwd_allreduce_microstep: 136.97 | step_microstep: 18.22
[2025-04-26 10:13:12,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5787.19 | bwd_inner: 5650.16 | bwd_allreduce: 136.99 | step: 18.22

  6%|▌         | 2518/41250 [6:04:44<93:39:44,  8.71s/it]
                                                         
{'loss': 0.1646, 'grad_norm': 1.511802077293396, 'learning_rate': 3.989908084918614e-05, 'epoch': 0.61}

  6%|▌         | 2518/41250 [6:04:44<93:39:44,  8.71s/it][2025-04-26 10:13:21,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:13:21,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.83 | bwd_microstep: 5710.16 | bwd_inner_microstep: 5697.24 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.42
[2025-04-26 10:13:21,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.84 | bwd: 5710.17 | bwd_inner: 5697.24 | bwd_allreduce: 12.89 | step: 18.42

  6%|▌         | 2519/41250 [6:04:52<93:29:48,  8.69s/it]
                                                         
{'loss': 0.1147, 'grad_norm': 1.3924223184585571, 'learning_rate': 3.9898923234423186e-05, 'epoch': 0.61}

  6%|▌         | 2519/41250 [6:04:52<93:29:48,  8.69s/it][2025-04-26 10:13:30,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:13:30,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.71 | bwd_microstep: 5712.06 | bwd_inner_microstep: 5699.44 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.04
[2025-04-26 10:13:30,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.71 | bwd: 5712.07 | bwd_inner: 5699.44 | bwd_allreduce: 12.59 | step: 18.04

  6%|▌         | 2520/41250 [6:05:01<93:20:08,  8.68s/it]
                                                         
{'loss': 0.2244, 'grad_norm': 2.7092931270599365, 'learning_rate': 3.9898765496987274e-05, 'epoch': 0.61}

  6%|▌         | 2520/41250 [6:05:01<93:20:08,  8.68s/it][2025-04-26 10:13:38,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:13:38,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.24 | bwd_microstep: 5759.67 | bwd_inner_microstep: 5718.69 | bwd_allreduce_microstep: 40.93 | step_microstep: 18.12
[2025-04-26 10:13:38,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.24 | bwd: 5759.68 | bwd_inner: 5718.69 | bwd_allreduce: 40.95 | step: 18.12

  6%|▌         | 2521/41250 [6:05:09<93:23:13,  8.68s/it]
                                                         
{'loss': 0.2213, 'grad_norm': 1.787296175956726, 'learning_rate': 3.989860763687938e-05, 'epoch': 0.61}

  6%|▌         | 2521/41250 [6:05:09<93:23:13,  8.68s/it][2025-04-26 10:13:47,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.21
[2025-04-26 10:13:47,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.93 | bwd_microstep: 5702.96 | bwd_inner_microstep: 5646.46 | bwd_allreduce_microstep: 56.45 | step_microstep: 19.18
[2025-04-26 10:13:47,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.93 | bwd: 5702.98 | bwd_inner: 5646.46 | bwd_allreduce: 56.47 | step: 19.18

  6%|▌         | 2522/41250 [6:05:18<93:07:43,  8.66s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 1.2608740329742432, 'learning_rate': 3.98984496541005e-05, 'epoch': 0.61}

  6%|▌         | 2522/41250 [6:05:18<93:07:43,  8.66s/it][2025-04-26 10:13:56,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:13:56,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.58 | bwd_microstep: 5697.14 | bwd_inner_microstep: 5657.97 | bwd_allreduce_microstep: 39.13 | step_microstep: 18.79
[2025-04-26 10:13:56,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.58 | bwd: 5697.15 | bwd_inner: 5657.97 | bwd_allreduce: 39.15 | step: 18.79

  6%|▌         | 2523/41250 [6:05:27<92:59:34,  8.64s/it]
                                                         
{'loss': 0.1628, 'grad_norm': 1.3783612251281738, 'learning_rate': 3.989829154865158e-05, 'epoch': 0.61}

  6%|▌         | 2523/41250 [6:05:27<92:59:34,  8.64s/it][2025-04-26 10:14:04,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 10:14:04,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.63 | bwd_microstep: 5711.45 | bwd_inner_microstep: 5697.92 | bwd_allreduce_microstep: 13.47 | step_microstep: 18.70
[2025-04-26 10:14:04,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.63 | bwd: 5711.47 | bwd_inner: 5697.91 | bwd_allreduce: 13.50 | step: 18.70

  6%|▌         | 2524/41250 [6:05:35<92:59:39,  8.64s/it]
                                                         
{'loss': 0.0598, 'grad_norm': 0.848931610584259, 'learning_rate': 3.989813332053361e-05, 'epoch': 0.61}

  6%|▌         | 2524/41250 [6:05:35<92:59:39,  8.64s/it][2025-04-26 10:14:13,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.04 | optimizer_step: 1.11
[2025-04-26 10:14:13,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5758.02 | bwd_inner_microstep: 5653.12 | bwd_allreduce_microstep: 104.85 | step_microstep: 18.86
[2025-04-26 10:14:13,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5758.03 | bwd_inner: 5653.12 | bwd_allreduce: 104.87 | step: 18.86

  6%|▌         | 2525/41250 [6:05:44<93:04:47,  8.65s/it]
                                                         
{'loss': 0.0929, 'grad_norm': 1.5040324926376343, 'learning_rate': 3.989797496974756e-05, 'epoch': 0.61}

  6%|▌         | 2525/41250 [6:05:44<93:04:47,  8.65s/it][2025-04-26 10:14:22,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-26 10:14:22,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5873.40 | bwd_inner_microstep: 5651.89 | bwd_allreduce_microstep: 221.46 | step_microstep: 18.24
[2025-04-26 10:14:22,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5873.42 | bwd_inner: 5651.89 | bwd_allreduce: 221.48 | step: 18.24

  6%|▌         | 2526/41250 [6:05:53<93:28:59,  8.69s/it]
                                                         
{'loss': 0.0891, 'grad_norm': 2.0724127292633057, 'learning_rate': 3.989781649629441e-05, 'epoch': 0.61}

  6%|▌         | 2526/41250 [6:05:53<93:28:59,  8.69s/it][2025-04-26 10:14:30,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:14:30,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.22 | bwd_microstep: 5766.92 | bwd_inner_microstep: 5653.78 | bwd_allreduce_microstep: 113.10 | step_microstep: 18.38
[2025-04-26 10:14:30,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.22 | bwd: 5766.93 | bwd_inner: 5653.78 | bwd_allreduce: 113.11 | step: 18.38

  6%|▌         | 2527/41250 [6:06:01<93:24:16,  8.68s/it]
                                                         
{'loss': 0.0963, 'grad_norm': 1.9057822227478027, 'learning_rate': 3.9897657900175135e-05, 'epoch': 0.61}

  6%|▌         | 2527/41250 [6:06:01<93:24:16,  8.68s/it][2025-04-26 10:14:39,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:14:39,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.71 | bwd_microstep: 5696.47 | bwd_inner_microstep: 5683.74 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.35
[2025-04-26 10:14:39,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.71 | bwd: 5696.48 | bwd_inner: 5683.74 | bwd_allreduce: 12.70 | step: 18.35

  6%|▌         | 2528/41250 [6:06:10<93:12:31,  8.67s/it]
                                                         
{'loss': 0.1582, 'grad_norm': 1.538362741470337, 'learning_rate': 3.989749918139071e-05, 'epoch': 0.61}

  6%|▌         | 2528/41250 [6:06:10<93:12:31,  8.67s/it][2025-04-26 10:14:48,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 10:14:48,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.17 | bwd_microstep: 5745.35 | bwd_inner_microstep: 5679.42 | bwd_allreduce_microstep: 65.88 | step_microstep: 18.72
[2025-04-26 10:14:48,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.17 | bwd: 5745.36 | bwd_inner: 5679.43 | bwd_allreduce: 65.90 | step: 18.73

  6%|▌         | 2529/41250 [6:06:19<93:13:13,  8.67s/it]
                                                         
{'loss': 0.18, 'grad_norm': 1.8291817903518677, 'learning_rate': 3.9897340339942116e-05, 'epoch': 0.61}

  6%|▌         | 2529/41250 [6:06:19<93:13:13,  8.67s/it][2025-04-26 10:14:56,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 10:14:56,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.12 | bwd_microstep: 5706.29 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.64
[2025-04-26 10:14:56,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.12 | bwd: 5706.31 | bwd_inner: 5693.28 | bwd_allreduce: 12.99 | step: 18.64

  6%|▌         | 2530/41250 [6:06:27<93:08:35,  8.66s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 1.379823923110962, 'learning_rate': 3.989718137583033e-05, 'epoch': 0.61}

  6%|▌         | 2530/41250 [6:06:27<93:08:35,  8.66s/it][2025-04-26 10:15:05,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 10:15:05,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.35 | bwd_microstep: 5765.58 | bwd_inner_microstep: 5643.29 | bwd_allreduce_microstep: 122.24 | step_microstep: 19.18
[2025-04-26 10:15:05,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.34 | bwd: 5765.59 | bwd_inner: 5643.29 | bwd_allreduce: 122.26 | step: 19.18

  6%|▌         | 2531/41250 [6:06:36<93:09:55,  8.66s/it]
                                                         
{'loss': 0.1891, 'grad_norm': 4.674919128417969, 'learning_rate': 3.9897022289056344e-05, 'epoch': 0.61}

  6%|▌         | 2531/41250 [6:06:36<93:09:55,  8.66s/it][2025-04-26 10:15:14,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:15:14,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.19 | bwd_microstep: 5752.88 | bwd_inner_microstep: 5647.96 | bwd_allreduce_microstep: 104.88 | step_microstep: 17.86
[2025-04-26 10:15:14,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.19 | bwd: 5752.89 | bwd_inner: 5647.96 | bwd_allreduce: 104.89 | step: 17.86

  6%|▌         | 2532/41250 [6:06:45<93:09:04,  8.66s/it]
                                                         
{'loss': 0.184, 'grad_norm': 2.1932213306427, 'learning_rate': 3.989686307962112e-05, 'epoch': 0.61}

  6%|▌         | 2532/41250 [6:06:45<93:09:04,  8.66s/it][2025-04-26 10:15:22,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:15:22,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.92 | bwd_microstep: 5747.04 | bwd_inner_microstep: 5675.34 | bwd_allreduce_microstep: 71.65 | step_microstep: 18.54
[2025-04-26 10:15:22,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.92 | bwd: 5747.05 | bwd_inner: 5675.34 | bwd_allreduce: 71.67 | step: 18.55

  6%|▌         | 2533/41250 [6:06:53<93:10:36,  8.66s/it]
                                                         
{'loss': 0.2581, 'grad_norm': 2.6517317295074463, 'learning_rate': 3.989670374752565e-05, 'epoch': 0.61}

  6%|▌         | 2533/41250 [6:06:53<93:10:36,  8.66s/it][2025-04-26 10:15:31,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:15:31,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.73 | bwd_microstep: 5747.88 | bwd_inner_microstep: 5708.11 | bwd_allreduce_microstep: 39.73 | step_microstep: 18.28
[2025-04-26 10:15:31,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.73 | bwd: 5747.89 | bwd_inner: 5708.11 | bwd_allreduce: 39.74 | step: 18.28

  6%|▌         | 2534/41250 [6:07:02<93:13:37,  8.67s/it]
                                                         
{'loss': 0.158, 'grad_norm': 1.5014855861663818, 'learning_rate': 3.989654429277092e-05, 'epoch': 0.61}

  6%|▌         | 2534/41250 [6:07:02<93:13:37,  8.67s/it][2025-04-26 10:15:40,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.81 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:15:40,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.72 | bwd_microstep: 5725.00 | bwd_inner_microstep: 5647.62 | bwd_allreduce_microstep: 77.33 | step_microstep: 18.70
[2025-04-26 10:15:40,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.72 | bwd: 5725.01 | bwd_inner: 5647.62 | bwd_allreduce: 77.34 | step: 18.70

  6%|▌         | 2535/41250 [6:07:11<93:06:57,  8.66s/it]
                                                         
{'loss': 0.2719, 'grad_norm': 3.6752307415008545, 'learning_rate': 3.9896384715357904e-05, 'epoch': 0.61}

  6%|▌         | 2535/41250 [6:07:11<93:06:57,  8.66s/it][2025-04-26 10:15:48,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:15:48,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.30 | bwd_microstep: 5701.42 | bwd_inner_microstep: 5636.31 | bwd_allreduce_microstep: 65.06 | step_microstep: 18.44
[2025-04-26 10:15:48,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.30 | bwd: 5701.43 | bwd_inner: 5636.31 | bwd_allreduce: 65.08 | step: 18.45

  6%|▌         | 2536/41250 [6:07:19<92:55:51,  8.64s/it]
                                                         
{'loss': 0.1396, 'grad_norm': 1.5667197704315186, 'learning_rate': 3.989622501528759e-05, 'epoch': 0.61}

  6%|▌         | 2536/41250 [6:07:19<92:55:51,  8.64s/it][2025-04-26 10:15:57,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.04 | optimizer_step: 0.98
[2025-04-26 10:15:57,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.15 | bwd_microstep: 5762.05 | bwd_inner_microstep: 5643.26 | bwd_allreduce_microstep: 118.73 | step_microstep: 20.32
[2025-04-26 10:15:57,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.14 | bwd: 5762.07 | bwd_inner: 5643.26 | bwd_allreduce: 118.76 | step: 20.32

  6%|▌         | 2537/41250 [6:07:28<93:03:05,  8.65s/it]
                                                         
{'loss': 0.6768, 'grad_norm': 4.729730606079102, 'learning_rate': 3.9896065192560964e-05, 'epoch': 0.62}

  6%|▌         | 2537/41250 [6:07:28<93:03:05,  8.65s/it][2025-04-26 10:16:06,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 10:16:06,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.41 | bwd_microstep: 5877.84 | bwd_inner_microstep: 5650.68 | bwd_allreduce_microstep: 227.11 | step_microstep: 18.24
[2025-04-26 10:16:06,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.41 | bwd: 5877.85 | bwd_inner: 5650.68 | bwd_allreduce: 227.13 | step: 18.24

  6%|▌         | 2538/41250 [6:07:37<93:30:43,  8.70s/it]
                                                         
{'loss': 0.074, 'grad_norm': 0.8202440738677979, 'learning_rate': 3.989590524717901e-05, 'epoch': 0.62}

  6%|▌         | 2538/41250 [6:07:37<93:30:43,  8.70s/it][2025-04-26 10:16:14,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:16:14,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.80 | bwd_microstep: 5774.68 | bwd_inner_microstep: 5648.47 | bwd_allreduce_microstep: 126.17 | step_microstep: 18.31
[2025-04-26 10:16:14,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.80 | bwd: 5774.69 | bwd_inner: 5648.47 | bwd_allreduce: 126.19 | step: 18.31

  6%|▌         | 2539/41250 [6:07:45<93:29:51,  8.69s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 1.9986425638198853, 'learning_rate': 3.989574517914271e-05, 'epoch': 0.62}

  6%|▌         | 2539/41250 [6:07:45<93:29:51,  8.69s/it][2025-04-26 10:16:23,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:16:23,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.51 | bwd_microstep: 5773.06 | bwd_inner_microstep: 5637.96 | bwd_allreduce_microstep: 135.05 | step_microstep: 18.28
[2025-04-26 10:16:23,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.51 | bwd: 5773.07 | bwd_inner: 5637.96 | bwd_allreduce: 135.07 | step: 18.28

  6%|▌         | 2540/41250 [6:07:54<93:23:15,  8.68s/it]
                                                         
{'loss': 0.1609, 'grad_norm': 1.4290771484375, 'learning_rate': 3.9895584988453054e-05, 'epoch': 0.62}

  6%|▌         | 2540/41250 [6:07:54<93:23:15,  8.68s/it][2025-04-26 10:16:32,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:16:32,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5713.92 | bwd_inner_microstep: 5701.17 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.28
[2025-04-26 10:16:32,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5713.93 | bwd_inner: 5701.17 | bwd_allreduce: 12.73 | step: 18.29

  6%|▌         | 2541/41250 [6:08:03<93:15:30,  8.67s/it]
                                                         
{'loss': 0.3311, 'grad_norm': 1.8537278175354004, 'learning_rate': 3.989542467511103e-05, 'epoch': 0.62}

  6%|▌         | 2541/41250 [6:08:03<93:15:30,  8.67s/it][2025-04-26 10:16:40,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:16:40,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.58 | bwd_microstep: 5747.10 | bwd_inner_microstep: 5685.46 | bwd_allreduce_microstep: 61.59 | step_microstep: 18.59
[2025-04-26 10:16:40,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.58 | bwd: 5747.12 | bwd_inner: 5685.46 | bwd_allreduce: 61.62 | step: 18.59

  6%|▌         | 2542/41250 [6:08:11<93:17:52,  8.68s/it]
                                                         
{'loss': 0.2574, 'grad_norm': 2.151534080505371, 'learning_rate': 3.989526423911764e-05, 'epoch': 0.62}

  6%|▌         | 2542/41250 [6:08:11<93:17:52,  8.68s/it][2025-04-26 10:16:49,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:16:49,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3103.42 | bwd_microstep: 5703.24 | bwd_inner_microstep: 5690.31 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.62
[2025-04-26 10:16:49,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3103.42 | bwd: 5703.25 | bwd_inner: 5690.31 | bwd_allreduce: 12.90 | step: 18.63

  6%|▌         | 2543/41250 [6:08:20<93:58:37,  8.74s/it]
                                                         
{'loss': 0.1399, 'grad_norm': 1.325495958328247, 'learning_rate': 3.9895103680473844e-05, 'epoch': 0.62}

  6%|▌         | 2543/41250 [6:08:20<93:58:37,  8.74s/it][2025-04-26 10:16:58,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.15 | optimizer_step: 0.89
[2025-04-26 10:16:58,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.71 | bwd_microstep: 5701.47 | bwd_inner_microstep: 5660.22 | bwd_allreduce_microstep: 41.21 | step_microstep: 19.15
[2025-04-26 10:16:58,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.71 | bwd: 5701.48 | bwd_inner: 5660.22 | bwd_allreduce: 41.22 | step: 19.15

  6%|▌         | 2544/41250 [6:08:29<93:35:33,  8.70s/it]
                                                         
{'loss': 0.1975, 'grad_norm': 0.7676827311515808, 'learning_rate': 3.989494299918065e-05, 'epoch': 0.62}

  6%|▌         | 2544/41250 [6:08:29<93:35:33,  8.70s/it][2025-04-26 10:17:06,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:17:06,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.82 | bwd_microstep: 5753.34 | bwd_inner_microstep: 5710.27 | bwd_allreduce_microstep: 43.02 | step_microstep: 18.02
[2025-04-26 10:17:06,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.82 | bwd: 5753.35 | bwd_inner: 5710.27 | bwd_allreduce: 43.04 | step: 18.03

  6%|▌         | 2545/41250 [6:08:38<93:31:10,  8.70s/it]
                                                         
{'loss': 0.1773, 'grad_norm': 1.6761013269424438, 'learning_rate': 3.989478219523904e-05, 'epoch': 0.62}

  6%|▌         | 2545/41250 [6:08:38<93:31:10,  8.70s/it][2025-04-26 10:17:15,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:17:15,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.51 | bwd_microstep: 5729.46 | bwd_inner_microstep: 5687.21 | bwd_allreduce_microstep: 42.21 | step_microstep: 17.93
[2025-04-26 10:17:15,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.51 | bwd: 5729.47 | bwd_inner: 5687.21 | bwd_allreduce: 42.23 | step: 17.93

  6%|▌         | 2546/41250 [6:08:46<93:24:23,  8.69s/it]
                                                         
{'loss': 0.2216, 'grad_norm': 1.7580739259719849, 'learning_rate': 3.989462126865002e-05, 'epoch': 0.62}

  6%|▌         | 2546/41250 [6:08:46<93:24:23,  8.69s/it][2025-04-26 10:17:24,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 10:17:24,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.44 | bwd_microstep: 5793.83 | bwd_inner_microstep: 5651.62 | bwd_allreduce_microstep: 142.16 | step_microstep: 18.98
[2025-04-26 10:17:24,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.44 | bwd: 5793.84 | bwd_inner: 5651.62 | bwd_allreduce: 142.18 | step: 18.98

  6%|▌         | 2547/41250 [6:08:55<93:28:12,  8.69s/it]
                                                         
{'loss': 0.1955, 'grad_norm': 1.6738865375518799, 'learning_rate': 3.989446021941457e-05, 'epoch': 0.62}

  6%|▌         | 2547/41250 [6:08:55<93:28:12,  8.69s/it][2025-04-26 10:17:33,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-26 10:17:33,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.87 | bwd_microstep: 5774.43 | bwd_inner_microstep: 5659.26 | bwd_allreduce_microstep: 115.13 | step_microstep: 18.60
[2025-04-26 10:17:33,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.87 | bwd: 5774.44 | bwd_inner: 5659.26 | bwd_allreduce: 115.15 | step: 18.61

  6%|▌         | 2548/41250 [6:09:04<93:26:43,  8.69s/it]
                                                         
{'loss': 0.19, 'grad_norm': 1.7799160480499268, 'learning_rate': 3.989429904753368e-05, 'epoch': 0.62}

  6%|▌         | 2548/41250 [6:09:04<93:26:43,  8.69s/it][2025-04-26 10:17:41,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 10:17:41,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.14 | bwd_microstep: 5703.60 | bwd_inner_microstep: 5653.95 | bwd_allreduce_microstep: 49.60 | step_microstep: 18.28
[2025-04-26 10:17:41,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.14 | bwd: 5703.61 | bwd_inner: 5653.95 | bwd_allreduce: 49.62 | step: 18.29

  6%|▌         | 2549/41250 [6:09:12<93:10:21,  8.67s/it]
                                                         
{'loss': 0.1761, 'grad_norm': 2.04819393157959, 'learning_rate': 3.9894137753008356e-05, 'epoch': 0.62}

  6%|▌         | 2549/41250 [6:09:12<93:10:21,  8.67s/it][2025-04-26 10:17:50,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 10:17:50,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5778.83 | bwd_inner_microstep: 5659.73 | bwd_allreduce_microstep: 119.06 | step_microstep: 18.16
[2025-04-26 10:17:50,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5778.84 | bwd_inner: 5659.73 | bwd_allreduce: 119.08 | step: 18.16

  6%|▌         | 2550/41250 [6:09:21<93:15:32,  8.68s/it]
                                                         
{'loss': 0.1672, 'grad_norm': 1.4286754131317139, 'learning_rate': 3.989397633583958e-05, 'epoch': 0.62}

  6%|▌         | 2550/41250 [6:09:21<93:15:32,  8.68s/it][2025-04-26 10:17:59,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 10:17:59,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.41 | bwd_microstep: 5792.61 | bwd_inner_microstep: 5779.18 | bwd_allreduce_microstep: 13.39 | step_microstep: 19.40
[2025-04-26 10:17:59,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.41 | bwd: 5792.63 | bwd_inner: 5779.18 | bwd_allreduce: 13.41 | step: 19.40

  6%|▌         | 2551/41250 [6:09:30<93:34:56,  8.71s/it]
                                                         
{'loss': 0.3129, 'grad_norm': 2.2865188121795654, 'learning_rate': 3.9893814796028356e-05, 'epoch': 0.62}

  6%|▌         | 2551/41250 [6:09:30<93:34:56,  8.71s/it][2025-04-26 10:18:07,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:18:07,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.53 | bwd_microstep: 5753.38 | bwd_inner_microstep: 5728.46 | bwd_allreduce_microstep: 24.88 | step_microstep: 18.78
[2025-04-26 10:18:07,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.53 | bwd: 5753.39 | bwd_inner: 5728.46 | bwd_allreduce: 24.89 | step: 18.78

  6%|▌         | 2552/41250 [6:09:39<93:33:44,  8.70s/it]
                                                         
{'loss': 0.1144, 'grad_norm': 1.0049810409545898, 'learning_rate': 3.989365313357568e-05, 'epoch': 0.62}

  6%|▌         | 2552/41250 [6:09:39<93:33:44,  8.70s/it][2025-04-26 10:18:16,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:18:16,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.15 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5662.97 | bwd_allreduce_microstep: 44.54 | step_microstep: 18.06
[2025-04-26 10:18:16,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.15 | bwd: 5707.57 | bwd_inner: 5662.97 | bwd_allreduce: 44.56 | step: 18.07

  6%|▌         | 2553/41250 [6:09:47<93:19:17,  8.68s/it]
                                                         
{'loss': 0.4575, 'grad_norm': 2.3716037273406982, 'learning_rate': 3.989349134848254e-05, 'epoch': 0.62}

  6%|▌         | 2553/41250 [6:09:47<93:19:17,  8.68s/it][2025-04-26 10:18:25,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:18:25,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.84 | bwd_microstep: 5734.04 | bwd_inner_microstep: 5661.23 | bwd_allreduce_microstep: 72.77 | step_microstep: 18.06
[2025-04-26 10:18:25,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.84 | bwd: 5734.05 | bwd_inner: 5661.23 | bwd_allreduce: 72.78 | step: 18.06

  6%|▌         | 2554/41250 [6:09:56<93:10:30,  8.67s/it]
                                                         
{'loss': 0.3679, 'grad_norm': 1.9869465827941895, 'learning_rate': 3.989332944074995e-05, 'epoch': 0.62}

  6%|▌         | 2554/41250 [6:09:56<93:10:30,  8.67s/it][2025-04-26 10:18:33,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 10:18:33,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.72 | bwd_microstep: 5718.13 | bwd_inner_microstep: 5687.91 | bwd_allreduce_microstep: 30.18 | step_microstep: 18.83
[2025-04-26 10:18:33,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.72 | bwd: 5718.14 | bwd_inner: 5687.91 | bwd_allreduce: 30.20 | step: 18.84

  6%|▌         | 2555/41250 [6:10:04<93:07:14,  8.66s/it]
                                                         
{'loss': 0.0773, 'grad_norm': 0.4983086585998535, 'learning_rate': 3.989316741037889e-05, 'epoch': 0.62}

  6%|▌         | 2555/41250 [6:10:04<93:07:14,  8.66s/it][2025-04-26 10:18:42,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 10:18:42,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.34 | bwd_microstep: 5715.40 | bwd_inner_microstep: 5702.66 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.67
[2025-04-26 10:18:42,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.34 | bwd: 5715.42 | bwd_inner: 5702.66 | bwd_allreduce: 12.72 | step: 18.67

  6%|▌         | 2556/41250 [6:10:13<93:05:15,  8.66s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 1.4260896444320679, 'learning_rate': 3.989300525737036e-05, 'epoch': 0.62}

  6%|▌         | 2556/41250 [6:10:13<93:05:15,  8.66s/it][2025-04-26 10:18:51,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 10:18:51,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.24 | bwd_microstep: 5791.23 | bwd_inner_microstep: 5657.84 | bwd_allreduce_microstep: 133.35 | step_microstep: 19.21
[2025-04-26 10:18:51,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.24 | bwd: 5791.25 | bwd_inner: 5657.84 | bwd_allreduce: 133.37 | step: 19.21

  6%|▌         | 2557/41250 [6:10:22<93:14:47,  8.68s/it]
                                                         
{'loss': 0.1211, 'grad_norm': 0.8911547660827637, 'learning_rate': 3.989284298172537e-05, 'epoch': 0.62}

  6%|▌         | 2557/41250 [6:10:22<93:14:47,  8.68s/it][2025-04-26 10:18:59,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:18:59,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.91 | bwd_microstep: 5794.03 | bwd_inner_microstep: 5659.78 | bwd_allreduce_microstep: 134.20 | step_microstep: 18.60
[2025-04-26 10:18:59,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.90 | bwd: 5794.04 | bwd_inner: 5659.78 | bwd_allreduce: 134.22 | step: 18.61

  6%|▌         | 2558/41250 [6:10:31<93:19:22,  8.68s/it]
                                                         
{'loss': 0.2667, 'grad_norm': 2.095531463623047, 'learning_rate': 3.989268058344491e-05, 'epoch': 0.62}

  6%|▌         | 2558/41250 [6:10:31<93:19:22,  8.68s/it][2025-04-26 10:19:08,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 10:19:08,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5695.65 | bwd_inner_microstep: 5666.42 | bwd_allreduce_microstep: 29.18 | step_microstep: 18.33
[2025-04-26 10:19:08,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5695.66 | bwd_inner: 5666.42 | bwd_allreduce: 29.20 | step: 18.33

  6%|▌         | 2559/41250 [6:10:39<93:06:03,  8.66s/it]
                                                         
{'loss': 0.0886, 'grad_norm': 1.4585931301116943, 'learning_rate': 3.989251806252999e-05, 'epoch': 0.62}

  6%|▌         | 2559/41250 [6:10:39<93:06:03,  8.66s/it][2025-04-26 10:19:17,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-26 10:19:17,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.12 | bwd_microstep: 5794.36 | bwd_inner_microstep: 5781.03 | bwd_allreduce_microstep: 13.28 | step_microstep: 18.96
[2025-04-26 10:19:17,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.12 | bwd: 5794.38 | bwd_inner: 5781.03 | bwd_allreduce: 13.30 | step: 18.96

  6%|▌         | 2560/41250 [6:10:48<93:26:49,  8.70s/it]
                                                         
{'loss': 0.1714, 'grad_norm': 0.9947657585144043, 'learning_rate': 3.9892355418981616e-05, 'epoch': 0.62}

  6%|▌         | 2560/41250 [6:10:48<93:26:49,  8.70s/it][2025-04-26 10:19:25,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.23 | optimizer_step: 0.94
[2025-04-26 10:19:25,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.57 | bwd_microstep: 5780.62 | bwd_inner_microstep: 5694.63 | bwd_allreduce_microstep: 85.91 | step_microstep: 19.75
[2025-04-26 10:19:25,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.57 | bwd: 5780.64 | bwd_inner: 5694.63 | bwd_allreduce: 85.95 | step: 19.76

  6%|▌         | 2561/41250 [6:10:57<93:31:30,  8.70s/it]
                                                         
{'loss': 0.087, 'grad_norm': 0.6405134797096252, 'learning_rate': 3.989219265280078e-05, 'epoch': 0.62}

  6%|▌         | 2561/41250 [6:10:57<93:31:30,  8.70s/it][2025-04-26 10:19:34,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.26 | optimizer_step: 1.07
[2025-04-26 10:19:34,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.64 | bwd_microstep: 5847.13 | bwd_inner_microstep: 5688.70 | bwd_allreduce_microstep: 158.36 | step_microstep: 20.11
[2025-04-26 10:19:34,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.64 | bwd: 5847.14 | bwd_inner: 5688.70 | bwd_allreduce: 158.40 | step: 20.11

  6%|▌         | 2562/41250 [6:11:05<93:49:30,  8.73s/it]
                                                         
{'loss': 0.2434, 'grad_norm': 0.9961905479431152, 'learning_rate': 3.9892029763988485e-05, 'epoch': 0.62}

  6%|▌         | 2562/41250 [6:11:05<93:49:30,  8.73s/it][2025-04-26 10:19:43,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:19:43,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.73 | bwd_microstep: 5785.44 | bwd_inner_microstep: 5647.84 | bwd_allreduce_microstep: 137.55 | step_microstep: 18.56
[2025-04-26 10:19:43,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.73 | bwd: 5785.45 | bwd_inner: 5647.84 | bwd_allreduce: 137.57 | step: 18.56

  6%|▌         | 2563/41250 [6:11:14<93:41:25,  8.72s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.1339590549468994, 'learning_rate': 3.989186675254574e-05, 'epoch': 0.62}

  6%|▌         | 2563/41250 [6:11:14<93:41:25,  8.72s/it][2025-04-26 10:19:52,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:19:52,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.68 | bwd_microstep: 5711.81 | bwd_inner_microstep: 5663.62 | bwd_allreduce_microstep: 48.15 | step_microstep: 17.87
[2025-04-26 10:19:52,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.68 | bwd: 5711.83 | bwd_inner: 5663.62 | bwd_allreduce: 48.17 | step: 17.88

  6%|▌         | 2564/41250 [6:11:23<93:23:06,  8.69s/it]
                                                         
{'loss': 0.0961, 'grad_norm': 1.2357596158981323, 'learning_rate': 3.989170361847355e-05, 'epoch': 0.62}

  6%|▌         | 2564/41250 [6:11:23<93:23:06,  8.69s/it][2025-04-26 10:20:00,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 10:20:00,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 5768.77 | bwd_inner_microstep: 5689.82 | bwd_allreduce_microstep: 78.90 | step_microstep: 18.80
[2025-04-26 10:20:00,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 5768.78 | bwd_inner: 5689.82 | bwd_allreduce: 78.92 | step: 18.81

  6%|▌         | 2565/41250 [6:11:31<93:24:49,  8.69s/it]
                                                         
{'loss': 0.1507, 'grad_norm': 1.0587358474731445, 'learning_rate': 3.989154036177292e-05, 'epoch': 0.62}

  6%|▌         | 2565/41250 [6:11:31<93:24:49,  8.69s/it][2025-04-26 10:20:09,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-26 10:20:09,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.52 | bwd_microstep: 5889.98 | bwd_inner_microstep: 5646.41 | bwd_allreduce_microstep: 243.52 | step_microstep: 18.82
[2025-04-26 10:20:09,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.52 | bwd: 5889.99 | bwd_inner: 5646.41 | bwd_allreduce: 243.54 | step: 18.83

  6%|▌         | 2566/41250 [6:11:40<93:45:57,  8.73s/it]
                                                         
{'loss': 0.1143, 'grad_norm': 0.9509855508804321, 'learning_rate': 3.9891376982444855e-05, 'epoch': 0.62}

  6%|▌         | 2566/41250 [6:11:40<93:45:57,  8.73s/it][2025-04-26 10:20:18,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.05 | optimizer_step: 1.06
[2025-04-26 10:20:18,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.53 | bwd_microstep: 5876.08 | bwd_inner_microstep: 5705.84 | bwd_allreduce_microstep: 170.19 | step_microstep: 19.59
[2025-04-26 10:20:18,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.53 | bwd: 5876.09 | bwd_inner: 5705.84 | bwd_allreduce: 170.21 | step: 19.59

  6%|▌         | 2567/41250 [6:11:49<94:00:46,  8.75s/it]
                                                         
{'loss': 0.2006, 'grad_norm': 1.4846173524856567, 'learning_rate': 3.989121348049037e-05, 'epoch': 0.62}

  6%|▌         | 2567/41250 [6:11:49<94:00:46,  8.75s/it][2025-04-26 10:20:27,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:20:27,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.18 | bwd_microstep: 5777.76 | bwd_inner_microstep: 5649.92 | bwd_allreduce_microstep: 127.80 | step_microstep: 18.49
[2025-04-26 10:20:27,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.18 | bwd: 5777.77 | bwd_inner: 5649.92 | bwd_allreduce: 127.81 | step: 18.50

  6%|▌         | 2568/41250 [6:11:58<93:49:06,  8.73s/it]
                                                         
{'loss': 0.0523, 'grad_norm': 0.742521345615387, 'learning_rate': 3.989104985591046e-05, 'epoch': 0.62}

  6%|▌         | 2568/41250 [6:11:58<93:49:06,  8.73s/it][2025-04-26 10:20:35,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:20:35,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.10 | bwd_microstep: 5718.17 | bwd_inner_microstep: 5705.33 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.52
[2025-04-26 10:20:35,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.10 | bwd: 5718.19 | bwd_inner: 5705.33 | bwd_allreduce: 12.82 | step: 18.53

  6%|▌         | 2569/41250 [6:12:06<93:35:01,  8.71s/it]
                                                         
{'loss': 0.336, 'grad_norm': 3.094630718231201, 'learning_rate': 3.989088610870614e-05, 'epoch': 0.62}

  6%|▌         | 2569/41250 [6:12:06<93:35:01,  8.71s/it][2025-04-26 10:20:44,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:20:44,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5764.46 | bwd_inner_microstep: 5680.72 | bwd_allreduce_microstep: 83.69 | step_microstep: 18.50
[2025-04-26 10:20:44,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5764.47 | bwd_inner: 5680.72 | bwd_allreduce: 83.71 | step: 18.50

  6%|▌         | 2570/41250 [6:12:15<93:31:29,  8.70s/it]
                                                         
{'loss': 0.3579, 'grad_norm': 3.1461639404296875, 'learning_rate': 3.989072223887842e-05, 'epoch': 0.62}

  6%|▌         | 2570/41250 [6:12:15<93:31:29,  8.70s/it][2025-04-26 10:20:53,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:20:53,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.51 | bwd_microstep: 5742.58 | bwd_inner_microstep: 5695.88 | bwd_allreduce_microstep: 46.65 | step_microstep: 18.73
[2025-04-26 10:20:53,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.51 | bwd: 5742.59 | bwd_inner: 5695.88 | bwd_allreduce: 46.67 | step: 18.73

  6%|▌         | 2571/41250 [6:12:24<93:24:58,  8.69s/it]
                                                         
{'loss': 0.1841, 'grad_norm': 1.8006539344787598, 'learning_rate': 3.9890558246428304e-05, 'epoch': 0.62}

  6%|▌         | 2571/41250 [6:12:24<93:24:58,  8.69s/it][2025-04-26 10:21:01,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 10:21:01,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.00 | bwd_microstep: 5784.87 | bwd_inner_microstep: 5668.47 | bwd_allreduce_microstep: 116.36 | step_microstep: 18.63
[2025-04-26 10:21:01,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.00 | bwd: 5784.88 | bwd_inner: 5668.47 | bwd_allreduce: 116.37 | step: 18.64

  6%|▌         | 2572/41250 [6:12:32<93:26:01,  8.70s/it]
                                                         
{'loss': 0.0784, 'grad_norm': 0.630366325378418, 'learning_rate': 3.9890394131356816e-05, 'epoch': 0.62}

  6%|▌         | 2572/41250 [6:12:32<93:26:01,  8.70s/it][2025-04-26 10:21:10,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:21:10,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.32 | bwd_microstep: 5697.16 | bwd_inner_microstep: 5636.66 | bwd_allreduce_microstep: 60.46 | step_microstep: 18.52
[2025-04-26 10:21:10,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.32 | bwd: 5697.18 | bwd_inner: 5636.66 | bwd_allreduce: 60.48 | step: 18.52

  6%|▌         | 2573/41250 [6:12:41<93:07:55,  8.67s/it]
                                                         
{'loss': 0.2385, 'grad_norm': 1.2365107536315918, 'learning_rate': 3.989022989366496e-05, 'epoch': 0.62}

  6%|▌         | 2573/41250 [6:12:41<93:07:55,  8.67s/it][2025-04-26 10:21:19,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:21:19,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.41 | bwd_microstep: 5768.46 | bwd_inner_microstep: 5643.64 | bwd_allreduce_microstep: 124.77 | step_microstep: 18.41
[2025-04-26 10:21:19,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.41 | bwd: 5768.47 | bwd_inner: 5643.64 | bwd_allreduce: 124.79 | step: 18.41

  6%|▌         | 2574/41250 [6:12:50<93:09:10,  8.67s/it]
                                                         
{'loss': 0.1928, 'grad_norm': 1.619955062866211, 'learning_rate': 3.989006553335375e-05, 'epoch': 0.62}

  6%|▌         | 2574/41250 [6:12:50<93:09:10,  8.67s/it][2025-04-26 10:21:27,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 10:21:27,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.58 | bwd_microstep: 5738.72 | bwd_inner_microstep: 5699.40 | bwd_allreduce_microstep: 39.27 | step_microstep: 18.51
[2025-04-26 10:21:27,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.58 | bwd: 5738.73 | bwd_inner: 5699.40 | bwd_allreduce: 39.29 | step: 18.52

  6%|▌         | 2575/41250 [6:12:58<93:09:51,  8.67s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 0.6804099082946777, 'learning_rate': 3.988990105042419e-05, 'epoch': 0.62}

  6%|▌         | 2575/41250 [6:12:58<93:09:51,  8.67s/it][2025-04-26 10:21:36,293] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:21:36,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.97 | bwd_microstep: 5699.76 | bwd_inner_microstep: 5639.58 | bwd_allreduce_microstep: 60.14 | step_microstep: 18.75
[2025-04-26 10:21:36,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.97 | bwd: 5699.78 | bwd_inner: 5639.58 | bwd_allreduce: 60.15 | step: 18.75

  6%|▌         | 2576/41250 [6:13:07<92:53:21,  8.65s/it]
                                                         
{'loss': 0.0639, 'grad_norm': 0.6999045610427856, 'learning_rate': 3.9889736444877313e-05, 'epoch': 0.62}

  6%|▌         | 2576/41250 [6:13:07<92:53:21,  8.65s/it][2025-04-26 10:21:45,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.96
[2025-04-26 10:21:45,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.28 | bwd_microstep: 5830.94 | bwd_inner_microstep: 5699.39 | bwd_allreduce_microstep: 131.49 | step_microstep: 18.77
[2025-04-26 10:21:45,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.28 | bwd: 5830.95 | bwd_inner: 5699.39 | bwd_allreduce: 131.51 | step: 18.77

  6%|▌         | 2577/41250 [6:13:16<93:16:57,  8.68s/it]
                                                         
{'loss': 0.1337, 'grad_norm': 1.377657175064087, 'learning_rate': 3.9889571716714115e-05, 'epoch': 0.62}

  6%|▌         | 2577/41250 [6:13:16<93:16:57,  8.68s/it][2025-04-26 10:21:53,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.17 | optimizer_step: 1.05
[2025-04-26 10:21:53,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.77 | bwd_microstep: 5708.80 | bwd_inner_microstep: 5647.71 | bwd_allreduce_microstep: 61.04 | step_microstep: 19.36
[2025-04-26 10:21:53,698] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.77 | bwd: 5708.81 | bwd_inner: 5647.71 | bwd_allreduce: 61.06 | step: 19.36

  6%|▌         | 2578/41250 [6:13:24<93:07:35,  8.67s/it]
                                                         
{'loss': 0.1787, 'grad_norm': 1.8413753509521484, 'learning_rate': 3.988940686593562e-05, 'epoch': 0.62}

  6%|▌         | 2578/41250 [6:13:24<93:07:35,  8.67s/it][2025-04-26 10:22:02,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 10:22:02,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.74 | bwd_microstep: 5769.12 | bwd_inner_microstep: 5632.47 | bwd_allreduce_microstep: 136.59 | step_microstep: 19.53
[2025-04-26 10:22:02,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.74 | bwd: 5769.13 | bwd_inner: 5632.47 | bwd_allreduce: 136.61 | step: 19.53

  6%|▋         | 2579/41250 [6:13:33<93:10:13,  8.67s/it]
                                                         
{'loss': 0.1205, 'grad_norm': 1.4774600267410278, 'learning_rate': 3.988924189254285e-05, 'epoch': 0.63}

  6%|▋         | 2579/41250 [6:13:33<93:10:13,  8.67s/it][2025-04-26 10:22:11,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 10:22:11,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.46 | bwd_microstep: 5757.98 | bwd_inner_microstep: 5652.69 | bwd_allreduce_microstep: 105.24 | step_microstep: 18.86
[2025-04-26 10:22:11,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.46 | bwd: 5757.99 | bwd_inner: 5652.69 | bwd_allreduce: 105.26 | step: 18.87

  6%|▋         | 2580/41250 [6:13:42<93:08:23,  8.67s/it]
                                                         
{'loss': 0.2696, 'grad_norm': 1.9006612300872803, 'learning_rate': 3.988907679653681e-05, 'epoch': 0.63}

  6%|▋         | 2580/41250 [6:13:42<93:08:23,  8.67s/it][2025-04-26 10:22:19,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:22:19,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.74 | bwd_microstep: 5859.50 | bwd_inner_microstep: 5683.48 | bwd_allreduce_microstep: 175.97 | step_microstep: 18.63
[2025-04-26 10:22:19,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.74 | bwd: 5859.51 | bwd_inner: 5683.48 | bwd_allreduce: 175.99 | step: 18.63

  6%|▋         | 2581/41250 [6:13:51<93:28:36,  8.70s/it]
                                                         
{'loss': 0.3001, 'grad_norm': 1.5910885334014893, 'learning_rate': 3.9888911577918525e-05, 'epoch': 0.63}

  6%|▋         | 2581/41250 [6:13:51<93:28:36,  8.70s/it][2025-04-26 10:22:28,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 10:22:28,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.67 | bwd_microstep: 5784.26 | bwd_inner_microstep: 5655.94 | bwd_allreduce_microstep: 128.26 | step_microstep: 18.62
[2025-04-26 10:22:28,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.67 | bwd: 5784.27 | bwd_inner: 5655.94 | bwd_allreduce: 128.29 | step: 18.62

  6%|▋         | 2582/41250 [6:13:59<93:26:20,  8.70s/it]
                                                         
{'loss': 0.1647, 'grad_norm': 2.2621548175811768, 'learning_rate': 3.9888746236689014e-05, 'epoch': 0.63}

  6%|▋         | 2582/41250 [6:13:59<93:26:20,  8.70s/it][2025-04-26 10:22:37,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:22:37,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5755.21 | bwd_inner_microstep: 5643.40 | bwd_allreduce_microstep: 111.77 | step_microstep: 18.39
[2025-04-26 10:22:37,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5755.23 | bwd_inner: 5643.40 | bwd_allreduce: 111.79 | step: 18.40

  6%|▋         | 2583/41250 [6:14:08<93:19:30,  8.69s/it]
                                                         
{'loss': 0.2466, 'grad_norm': 2.479856014251709, 'learning_rate': 3.9888580772849295e-05, 'epoch': 0.63}

  6%|▋         | 2583/41250 [6:14:08<93:19:30,  8.69s/it][2025-04-26 10:22:45,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-26 10:22:45,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.79 | bwd_microstep: 5769.99 | bwd_inner_microstep: 5655.20 | bwd_allreduce_microstep: 114.74 | step_microstep: 18.43
[2025-04-26 10:22:45,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.79 | bwd: 5770.00 | bwd_inner: 5655.20 | bwd_allreduce: 114.76 | step: 18.44

  6%|▋         | 2584/41250 [6:14:17<93:17:52,  8.69s/it]
                                                         
{'loss': 0.0772, 'grad_norm': 0.8494612574577332, 'learning_rate': 3.9888415186400385e-05, 'epoch': 0.63}

  6%|▋         | 2584/41250 [6:14:17<93:17:52,  8.69s/it][2025-04-26 10:22:54,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 10:22:54,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5708.35 | bwd_inner_microstep: 5695.88 | bwd_allreduce_microstep: 12.42 | step_microstep: 19.06
[2025-04-26 10:22:54,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5708.36 | bwd_inner: 5695.88 | bwd_allreduce: 12.44 | step: 19.06

  6%|▋         | 2585/41250 [6:14:25<93:07:25,  8.67s/it]
                                                         
{'loss': 0.1109, 'grad_norm': 1.2565304040908813, 'learning_rate': 3.9888249477343306e-05, 'epoch': 0.63}

  6%|▋         | 2585/41250 [6:14:25<93:07:25,  8.67s/it][2025-04-26 10:23:03,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 10:23:03,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.89 | bwd_microstep: 5929.78 | bwd_inner_microstep: 5689.98 | bwd_allreduce_microstep: 239.76 | step_microstep: 18.88
[2025-04-26 10:23:03,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.89 | bwd: 5929.79 | bwd_inner: 5689.98 | bwd_allreduce: 239.77 | step: 18.89

  6%|▋         | 2586/41250 [6:14:34<93:43:56,  8.73s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 2.2852272987365723, 'learning_rate': 3.988808364567908e-05, 'epoch': 0.63}

  6%|▋         | 2586/41250 [6:14:34<93:43:56,  8.73s/it][2025-04-26 10:23:12,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:23:12,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.00 | bwd_microstep: 5785.38 | bwd_inner_microstep: 5772.62 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.51
[2025-04-26 10:23:12,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.00 | bwd: 5785.39 | bwd_inner: 5772.62 | bwd_allreduce: 12.73 | step: 18.51

  6%|▋         | 2587/41250 [6:14:43<93:49:21,  8.74s/it]
                                                         
{'loss': 0.1161, 'grad_norm': 0.9665549397468567, 'learning_rate': 3.988791769140873e-05, 'epoch': 0.63}

  6%|▋         | 2587/41250 [6:14:43<93:49:21,  8.74s/it][2025-04-26 10:23:20,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 1.07
[2025-04-26 10:23:20,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.89 | bwd_microstep: 5701.56 | bwd_inner_microstep: 5643.17 | bwd_allreduce_microstep: 58.34 | step_microstep: 18.72
[2025-04-26 10:23:20,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.89 | bwd: 5701.57 | bwd_inner: 5643.17 | bwd_allreduce: 58.36 | step: 18.72

  6%|▋         | 2588/41250 [6:14:51<93:24:58,  8.70s/it]
                                                         
{'loss': 0.2592, 'grad_norm': 2.1501305103302, 'learning_rate': 3.9887751614533286e-05, 'epoch': 0.63}

  6%|▋         | 2588/41250 [6:14:51<93:24:58,  8.70s/it][2025-04-26 10:23:29,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.20
[2025-04-26 10:23:29,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.87 | bwd_microstep: 5750.97 | bwd_inner_microstep: 5710.38 | bwd_allreduce_microstep: 40.55 | step_microstep: 19.28
[2025-04-26 10:23:29,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.87 | bwd: 5750.98 | bwd_inner: 5710.38 | bwd_allreduce: 40.57 | step: 19.28

  6%|▋         | 2589/41250 [6:15:00<93:24:43,  8.70s/it]
                                                         
{'loss': 0.1017, 'grad_norm': 1.0887624025344849, 'learning_rate': 3.988758541505376e-05, 'epoch': 0.63}

  6%|▋         | 2589/41250 [6:15:00<93:24:43,  8.70s/it][2025-04-26 10:23:38,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 10:23:38,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.59 | bwd_microstep: 5903.48 | bwd_inner_microstep: 5653.87 | bwd_allreduce_microstep: 249.57 | step_microstep: 18.44
[2025-04-26 10:23:38,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.59 | bwd: 5903.50 | bwd_inner: 5653.87 | bwd_allreduce: 249.59 | step: 18.45

  6%|▋         | 2590/41250 [6:15:09<93:45:34,  8.73s/it]
                                                         
{'loss': 0.3631, 'grad_norm': 2.287863254547119, 'learning_rate': 3.988741909297118e-05, 'epoch': 0.63}

  6%|▋         | 2590/41250 [6:15:09<93:45:34,  8.73s/it][2025-04-26 10:23:46,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:23:46,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.61 | bwd_microstep: 5728.70 | bwd_inner_microstep: 5659.60 | bwd_allreduce_microstep: 69.05 | step_microstep: 18.47
[2025-04-26 10:23:46,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.61 | bwd: 5728.72 | bwd_inner: 5659.60 | bwd_allreduce: 69.07 | step: 18.48

  6%|▋         | 2591/41250 [6:15:18<93:27:01,  8.70s/it]
                                                         
{'loss': 0.0634, 'grad_norm': 0.6611658334732056, 'learning_rate': 3.9887252648286575e-05, 'epoch': 0.63}

  6%|▋         | 2591/41250 [6:15:18<93:27:01,  8.70s/it][2025-04-26 10:23:55,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.10 | optimizer_step: 1.00
[2025-04-26 10:23:55,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.39 | bwd_microstep: 5785.00 | bwd_inner_microstep: 5660.73 | bwd_allreduce_microstep: 124.21 | step_microstep: 19.44
[2025-04-26 10:23:55,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.39 | bwd: 5785.01 | bwd_inner: 5660.73 | bwd_allreduce: 124.23 | step: 19.44

  6%|▋         | 2592/41250 [6:15:26<93:26:17,  8.70s/it]
                                                         
{'loss': 0.1396, 'grad_norm': 1.9019341468811035, 'learning_rate': 3.988708608100097e-05, 'epoch': 0.63}

  6%|▋         | 2592/41250 [6:15:26<93:26:17,  8.70s/it][2025-04-26 10:24:04,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 10:24:04,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.06 | bwd_microstep: 5784.75 | bwd_inner_microstep: 5690.66 | bwd_allreduce_microstep: 94.03 | step_microstep: 18.83
[2025-04-26 10:24:04,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.06 | bwd: 5784.76 | bwd_inner: 5690.66 | bwd_allreduce: 94.05 | step: 18.84

  6%|▋         | 2593/41250 [6:15:35<93:28:40,  8.71s/it]
                                                         
{'loss': 0.3428, 'grad_norm': 2.4917337894439697, 'learning_rate': 3.9886919391115394e-05, 'epoch': 0.63}

  6%|▋         | 2593/41250 [6:15:35<93:28:40,  8.71s/it][2025-04-26 10:24:12,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 10:24:12,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.86 | bwd_microstep: 5737.57 | bwd_inner_microstep: 5724.80 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.75
[2025-04-26 10:24:12,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.86 | bwd: 5737.59 | bwd_inner: 5724.80 | bwd_allreduce: 12.74 | step: 18.75

  6%|▋         | 2594/41250 [6:15:44<93:21:21,  8.69s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.1312378644943237, 'learning_rate': 3.988675257863087e-05, 'epoch': 0.63}

  6%|▋         | 2594/41250 [6:15:44<93:21:21,  8.69s/it][2025-04-26 10:24:21,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.22 | optimizer_step: 0.99
[2025-04-26 10:24:21,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.35 | bwd_microstep: 5916.11 | bwd_inner_microstep: 5722.76 | bwd_allreduce_microstep: 193.30 | step_microstep: 19.46
[2025-04-26 10:24:21,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.35 | bwd: 5916.13 | bwd_inner: 5722.76 | bwd_allreduce: 193.33 | step: 19.47

  6%|▋         | 2595/41250 [6:15:52<93:52:56,  8.74s/it]
                                                         
{'loss': 0.0461, 'grad_norm': 1.1468679904937744, 'learning_rate': 3.988658564354842e-05, 'epoch': 0.63}

  6%|▋         | 2595/41250 [6:15:52<93:52:56,  8.74s/it][2025-04-26 10:24:30,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 10:24:30,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.16 | bwd_microstep: 5727.07 | bwd_inner_microstep: 5714.36 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.69
[2025-04-26 10:24:30,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.16 | bwd: 5727.08 | bwd_inner: 5714.35 | bwd_allreduce: 12.69 | step: 18.70

  6%|▋         | 2596/41250 [6:16:01<93:37:21,  8.72s/it]
                                                         
{'loss': 0.1888, 'grad_norm': 1.6306207180023193, 'learning_rate': 3.988641858586908e-05, 'epoch': 0.63}

  6%|▋         | 2596/41250 [6:16:01<93:37:21,  8.72s/it][2025-04-26 10:24:39,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:24:39,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5858.80 | bwd_inner_microstep: 5699.24 | bwd_allreduce_microstep: 159.52 | step_microstep: 18.49
[2025-04-26 10:24:39,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5858.82 | bwd_inner: 5699.24 | bwd_allreduce: 159.54 | step: 18.49

  6%|▋         | 2597/41250 [6:16:10<93:52:46,  8.74s/it]
                                                         
{'loss': 0.0468, 'grad_norm': 0.4573630690574646, 'learning_rate': 3.988625140559389e-05, 'epoch': 0.63}

  6%|▋         | 2597/41250 [6:16:10<93:52:46,  8.74s/it][2025-04-26 10:24:47,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.27 | optimizer_step: 1.01
[2025-04-26 10:24:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5723.71 | bwd_inner_microstep: 5693.99 | bwd_allreduce_microstep: 29.67 | step_microstep: 19.65
[2025-04-26 10:24:47,920] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5723.72 | bwd_inner: 5693.99 | bwd_allreduce: 29.69 | step: 19.66

  6%|▋         | 2598/41250 [6:16:19<93:35:29,  8.72s/it]
                                                         
{'loss': 0.0746, 'grad_norm': 0.8455091714859009, 'learning_rate': 3.9886084102723874e-05, 'epoch': 0.63}

  6%|▋         | 2598/41250 [6:16:19<93:35:29,  8.72s/it][2025-04-26 10:24:56,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 1.11
[2025-04-26 10:24:56,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.71 | bwd_microstep: 5789.55 | bwd_inner_microstep: 5671.17 | bwd_allreduce_microstep: 118.33 | step_microstep: 18.77
[2025-04-26 10:24:56,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.71 | bwd: 5789.56 | bwd_inner: 5671.17 | bwd_allreduce: 118.35 | step: 18.77

  6%|▋         | 2599/41250 [6:16:27<93:32:04,  8.71s/it]
                                                         
{'loss': 0.6879, 'grad_norm': 3.727254629135132, 'learning_rate': 3.988591667726005e-05, 'epoch': 0.63}

  6%|▋         | 2599/41250 [6:16:27<93:32:04,  8.71s/it][2025-04-26 10:25:05,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.99
[2025-04-26 10:25:05,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.12 | bwd_microstep: 5807.20 | bwd_inner_microstep: 5655.49 | bwd_allreduce_microstep: 151.66 | step_microstep: 18.85
[2025-04-26 10:25:05,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.12 | bwd: 5807.21 | bwd_inner: 5655.49 | bwd_allreduce: 151.68 | step: 18.85

  6%|▋         | 2600/41250 [6:16:36<93:33:22,  8.71s/it]
                                                         
{'loss': 0.2672, 'grad_norm': 2.2362611293792725, 'learning_rate': 3.988574912920347e-05, 'epoch': 0.63}

  6%|▋         | 2600/41250 [6:16:36<93:33:22,  8.71s/it][2025-04-26 10:25:14,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 10:25:14,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.25 | bwd_microstep: 5808.00 | bwd_inner_microstep: 5652.33 | bwd_allreduce_microstep: 155.62 | step_microstep: 18.58
[2025-04-26 10:25:14,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.25 | bwd: 5808.02 | bwd_inner: 5652.33 | bwd_allreduce: 155.64 | step: 18.58

  6%|▋         | 2601/41250 [6:16:45<93:34:05,  8.72s/it]
                                                         
{'loss': 0.0353, 'grad_norm': 0.8012112379074097, 'learning_rate': 3.988558145855515e-05, 'epoch': 0.63}

  6%|▋         | 2601/41250 [6:16:45<93:34:05,  8.72s/it][2025-04-26 10:25:22,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 1.13
[2025-04-26 10:25:22,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.12 | bwd_microstep: 5716.78 | bwd_inner_microstep: 5704.10 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.99
[2025-04-26 10:25:22,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.12 | bwd: 5716.80 | bwd_inner: 5704.10 | bwd_allreduce: 12.66 | step: 19.00

  6%|▋         | 2602/41250 [6:16:53<93:21:33,  8.70s/it]
                                                         
{'loss': 0.1438, 'grad_norm': 1.6625136137008667, 'learning_rate': 3.988541366531614e-05, 'epoch': 0.63}

  6%|▋         | 2602/41250 [6:16:53<93:21:33,  8.70s/it][2025-04-26 10:25:31,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 10:25:31,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5771.28 | bwd_inner_microstep: 5698.94 | bwd_allreduce_microstep: 72.30 | step_microstep: 18.73
[2025-04-26 10:25:31,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5771.29 | bwd_inner: 5698.94 | bwd_allreduce: 72.31 | step: 18.74

  6%|▋         | 2603/41250 [6:17:02<93:20:49,  8.70s/it]
                                                         
{'loss': 0.3193, 'grad_norm': 2.4143002033233643, 'learning_rate': 3.988524574948746e-05, 'epoch': 0.63}

  6%|▋         | 2603/41250 [6:17:02<93:20:49,  8.70s/it][2025-04-26 10:25:40,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:25:40,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.54 | bwd_microstep: 5705.77 | bwd_inner_microstep: 5693.01 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.52
[2025-04-26 10:25:40,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.54 | bwd: 5705.78 | bwd_inner: 5693.01 | bwd_allreduce: 12.73 | step: 18.52

  6%|▋         | 2604/41250 [6:17:11<93:08:27,  8.68s/it]
                                                         
{'loss': 0.109, 'grad_norm': 2.1479697227478027, 'learning_rate': 3.988507771107015e-05, 'epoch': 0.63}

  6%|▋         | 2604/41250 [6:17:11<93:08:27,  8.68s/it][2025-04-26 10:25:48,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:25:48,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.06 | bwd_microstep: 5761.88 | bwd_inner_microstep: 5702.08 | bwd_allreduce_microstep: 59.76 | step_microstep: 18.58
[2025-04-26 10:25:48,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.06 | bwd: 5761.90 | bwd_inner: 5702.08 | bwd_allreduce: 59.77 | step: 18.58

  6%|▋         | 2605/41250 [6:17:19<93:12:28,  8.68s/it]
                                                         
{'loss': 0.1417, 'grad_norm': 1.0070147514343262, 'learning_rate': 3.9884909550065255e-05, 'epoch': 0.63}

  6%|▋         | 2605/41250 [6:17:19<93:12:28,  8.68s/it][2025-04-26 10:25:57,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 10:25:57,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.33 | bwd_microstep: 5761.02 | bwd_inner_microstep: 5653.98 | bwd_allreduce_microstep: 106.99 | step_microstep: 18.89
[2025-04-26 10:25:57,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.33 | bwd: 5761.03 | bwd_inner: 5653.98 | bwd_allreduce: 107.01 | step: 18.90

  6%|▋         | 2606/41250 [6:17:28<93:11:20,  8.68s/it]
                                                         
{'loss': 0.1476, 'grad_norm': 1.1115329265594482, 'learning_rate': 3.9884741266473794e-05, 'epoch': 0.63}

  6%|▋         | 2606/41250 [6:17:28<93:11:20,  8.68s/it][2025-04-26 10:26:06,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.13 | optimizer_step: 1.05
[2025-04-26 10:26:06,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.56 | bwd_microstep: 5769.17 | bwd_inner_microstep: 5651.03 | bwd_allreduce_microstep: 118.04 | step_microstep: 20.32
[2025-04-26 10:26:06,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.56 | bwd: 5769.21 | bwd_inner: 5651.03 | bwd_allreduce: 118.09 | step: 20.31

  6%|▋         | 2607/41250 [6:17:37<93:13:48,  8.69s/it]
                                                         
{'loss': 0.1164, 'grad_norm': 0.7307649850845337, 'learning_rate': 3.9884572860296824e-05, 'epoch': 0.63}

  6%|▋         | 2607/41250 [6:17:37<93:13:48,  8.69s/it][2025-04-26 10:26:14,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.04
[2025-04-26 10:26:14,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.10 | bwd_microstep: 5720.88 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 69.02 | step_microstep: 19.00
[2025-04-26 10:26:14,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.10 | bwd: 5720.89 | bwd_inner: 5651.81 | bwd_allreduce: 69.04 | step: 19.01

  6%|▋         | 2608/41250 [6:17:45<93:01:32,  8.67s/it]
                                                         
{'loss': 0.0794, 'grad_norm': 0.8248738050460815, 'learning_rate': 3.988440433153537e-05, 'epoch': 0.63}

  6%|▋         | 2608/41250 [6:17:45<93:01:32,  8.67s/it][2025-04-26 10:26:23,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:26:23,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.00 | bwd_microstep: 5707.61 | bwd_inner_microstep: 5694.77 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.80
[2025-04-26 10:26:23,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.00 | bwd: 5707.63 | bwd_inner: 5694.77 | bwd_allreduce: 12.81 | step: 18.81

  6%|▋         | 2609/41250 [6:17:54<92:56:27,  8.66s/it]
                                                         
{'loss': 0.1683, 'grad_norm': 1.1214529275894165, 'learning_rate': 3.988423568019047e-05, 'epoch': 0.63}

  6%|▋         | 2609/41250 [6:17:54<92:56:27,  8.66s/it][2025-04-26 10:26:32,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:26:32,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.98 | bwd_microstep: 5775.60 | bwd_inner_microstep: 5645.13 | bwd_allreduce_microstep: 130.43 | step_microstep: 18.55
[2025-04-26 10:26:32,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.98 | bwd: 5775.62 | bwd_inner: 5645.13 | bwd_allreduce: 130.45 | step: 18.56

  6%|▋         | 2610/41250 [6:18:03<93:00:57,  8.67s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 0.8488182425498962, 'learning_rate': 3.988406690626317e-05, 'epoch': 0.63}

  6%|▋         | 2610/41250 [6:18:03<93:00:57,  8.67s/it][2025-04-26 10:26:40,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:26:40,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.07 | bwd_microstep: 5764.21 | bwd_inner_microstep: 5687.62 | bwd_allreduce_microstep: 76.54 | step_microstep: 18.58
[2025-04-26 10:26:40,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.07 | bwd: 5764.23 | bwd_inner: 5687.62 | bwd_allreduce: 76.56 | step: 18.58

  6%|▋         | 2611/41250 [6:18:11<93:06:22,  8.67s/it]
                                                         
{'loss': 0.4791, 'grad_norm': 2.3951635360717773, 'learning_rate': 3.988389800975451e-05, 'epoch': 0.63}

  6%|▋         | 2611/41250 [6:18:11<93:06:22,  8.67s/it][2025-04-26 10:26:49,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:26:49,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.79 | bwd_microstep: 5746.14 | bwd_inner_microstep: 5673.19 | bwd_allreduce_microstep: 72.90 | step_microstep: 18.62
[2025-04-26 10:26:49,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.79 | bwd: 5746.15 | bwd_inner: 5673.19 | bwd_allreduce: 72.92 | step: 18.63

  6%|▋         | 2612/41250 [6:18:20<93:04:05,  8.67s/it]
                                                         
{'loss': 0.4065, 'grad_norm': 2.65396785736084, 'learning_rate': 3.988372899066553e-05, 'epoch': 0.63}

  6%|▋         | 2612/41250 [6:18:20<93:04:05,  8.67s/it][2025-04-26 10:26:58,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.04 | optimizer_gradients: 1.15 | optimizer_step: 0.90
[2025-04-26 10:26:58,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.05 | bwd_microstep: 5695.55 | bwd_inner_microstep: 5682.70 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.57
[2025-04-26 10:26:58,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.05 | bwd: 5695.57 | bwd_inner: 5682.70 | bwd_allreduce: 12.83 | step: 19.58

  6%|▋         | 2613/41250 [6:18:29<92:55:04,  8.66s/it]
                                                         
{'loss': 0.1854, 'grad_norm': 1.8516303300857544, 'learning_rate': 3.988355984899727e-05, 'epoch': 0.63}

  6%|▋         | 2613/41250 [6:18:29<92:55:04,  8.66s/it][2025-04-26 10:27:06,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:27:06,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.57 | bwd_microstep: 5740.72 | bwd_inner_microstep: 5703.98 | bwd_allreduce_microstep: 36.69 | step_microstep: 18.82
[2025-04-26 10:27:06,705] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5740.73 | bwd_inner: 5703.98 | bwd_allreduce: 36.71 | step: 18.82

  6%|▋         | 2614/41250 [6:18:37<92:57:42,  8.66s/it]
                                                         
{'loss': 0.1989, 'grad_norm': 1.4451960325241089, 'learning_rate': 3.988339058475077e-05, 'epoch': 0.63}

  6%|▋         | 2614/41250 [6:18:37<92:57:42,  8.66s/it][2025-04-26 10:27:15,387] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 10:27:15,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.94 | bwd_microstep: 5757.47 | bwd_inner_microstep: 5647.02 | bwd_allreduce_microstep: 110.41 | step_microstep: 18.24
[2025-04-26 10:27:15,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.94 | bwd: 5757.48 | bwd_inner: 5647.02 | bwd_allreduce: 110.43 | step: 18.25

  6%|▋         | 2615/41250 [6:18:46<93:00:49,  8.67s/it]
                                                         
{'loss': 0.126, 'grad_norm': 0.853923499584198, 'learning_rate': 3.9883221197927085e-05, 'epoch': 0.63}

  6%|▋         | 2615/41250 [6:18:46<93:00:49,  8.67s/it][2025-04-26 10:27:24,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 10:27:24,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.97 | bwd_microstep: 5744.29 | bwd_inner_microstep: 5693.89 | bwd_allreduce_microstep: 50.35 | step_microstep: 19.22
[2025-04-26 10:27:24,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.97 | bwd: 5744.30 | bwd_inner: 5693.89 | bwd_allreduce: 50.37 | step: 19.22

  6%|▋         | 2616/41250 [6:18:55<93:04:02,  8.67s/it]
                                                         
{'loss': 0.0388, 'grad_norm': 0.4185981750488281, 'learning_rate': 3.9883051688527255e-05, 'epoch': 0.63}

  6%|▋         | 2616/41250 [6:18:55<93:04:02,  8.67s/it][2025-04-26 10:27:32,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:27:32,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.11 | bwd_microstep: 5703.64 | bwd_inner_microstep: 5690.43 | bwd_allreduce_microstep: 13.17 | step_microstep: 18.54
[2025-04-26 10:27:32,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.11 | bwd: 5703.66 | bwd_inner: 5690.43 | bwd_allreduce: 13.19 | step: 18.55

  6%|▋         | 2617/41250 [6:19:03<92:58:07,  8.66s/it]
                                                         
{'loss': 0.0921, 'grad_norm': 1.1931754350662231, 'learning_rate': 3.988288205655232e-05, 'epoch': 0.63}

  6%|▋         | 2617/41250 [6:19:03<92:58:07,  8.66s/it][2025-04-26 10:27:41,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.67 | optimizer_step: 0.92
[2025-04-26 10:27:41,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.79 | bwd_microstep: 5757.41 | bwd_inner_microstep: 5641.87 | bwd_allreduce_microstep: 115.48 | step_microstep: 20.31
[2025-04-26 10:27:41,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.79 | bwd: 5757.42 | bwd_inner: 5641.87 | bwd_allreduce: 115.51 | step: 20.31

  6%|▋         | 2618/41250 [6:19:12<92:58:39,  8.66s/it]
                                                         
{'loss': 0.121, 'grad_norm': 0.7753734588623047, 'learning_rate': 3.988271230200332e-05, 'epoch': 0.63}

  6%|▋         | 2618/41250 [6:19:12<92:58:39,  8.66s/it][2025-04-26 10:27:50,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-26 10:27:50,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.72 | bwd_microstep: 5700.14 | bwd_inner_microstep: 5687.42 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.84
[2025-04-26 10:27:50,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.72 | bwd: 5700.16 | bwd_inner: 5687.42 | bwd_allreduce: 12.69 | step: 18.84

  6%|▋         | 2619/41250 [6:19:21<92:51:00,  8.65s/it]
                                                         
{'loss': 0.2219, 'grad_norm': 2.723557710647583, 'learning_rate': 3.988254242488132e-05, 'epoch': 0.63}

  6%|▋         | 2619/41250 [6:19:21<92:51:00,  8.65s/it][2025-04-26 10:27:58,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 10:27:58,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.30 | bwd_microstep: 5681.32 | bwd_inner_microstep: 5650.13 | bwd_allreduce_microstep: 31.15 | step_microstep: 18.50
[2025-04-26 10:27:58,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.30 | bwd: 5681.33 | bwd_inner: 5650.13 | bwd_allreduce: 31.16 | step: 18.50

  6%|▋         | 2620/41250 [6:19:29<92:39:14,  8.63s/it]
                                                         
{'loss': 0.0397, 'grad_norm': 0.6580553650856018, 'learning_rate': 3.988237242518735e-05, 'epoch': 0.64}

  6%|▋         | 2620/41250 [6:19:29<92:39:14,  8.63s/it][2025-04-26 10:28:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:28:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.97 | bwd_microstep: 5763.87 | bwd_inner_microstep: 5639.35 | bwd_allreduce_microstep: 124.47 | step_microstep: 18.11
[2025-04-26 10:28:07,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.97 | bwd: 5763.88 | bwd_inner: 5639.35 | bwd_allreduce: 124.49 | step: 18.11

  6%|▋         | 2621/41250 [6:19:38<92:43:39,  8.64s/it]
                                                         
{'loss': 0.3029, 'grad_norm': 2.2930800914764404, 'learning_rate': 3.988220230292246e-05, 'epoch': 0.64}

  6%|▋         | 2621/41250 [6:19:38<92:43:39,  8.64s/it][2025-04-26 10:28:15,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:28:15,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.30 | bwd_microstep: 5734.33 | bwd_inner_microstep: 5702.61 | bwd_allreduce_microstep: 31.68 | step_microstep: 18.87
[2025-04-26 10:28:15,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.30 | bwd: 5734.34 | bwd_inner: 5702.61 | bwd_allreduce: 31.69 | step: 18.88

  6%|▋         | 2622/41250 [6:19:47<92:48:30,  8.65s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.537022352218628, 'learning_rate': 3.988203205808771e-05, 'epoch': 0.64}

  6%|▋         | 2622/41250 [6:19:47<92:48:30,  8.65s/it][2025-04-26 10:28:24,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.94
[2025-04-26 10:28:24,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.68 | bwd_microstep: 5701.66 | bwd_inner_microstep: 5640.24 | bwd_allreduce_microstep: 61.37 | step_microstep: 19.23
[2025-04-26 10:28:24,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.68 | bwd: 5701.69 | bwd_inner: 5640.24 | bwd_allreduce: 61.39 | step: 19.23

  6%|▋         | 2623/41250 [6:19:55<92:41:11,  8.64s/it]
                                                         
{'loss': 0.2462, 'grad_norm': 2.8069958686828613, 'learning_rate': 3.9881861690684145e-05, 'epoch': 0.64}

  6%|▋         | 2623/41250 [6:19:55<92:41:11,  8.64s/it][2025-04-26 10:28:33,213] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.35 | optimizer_step: 1.00
[2025-04-26 10:28:33,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.14 | bwd_microstep: 5739.76 | bwd_inner_microstep: 5698.34 | bwd_allreduce_microstep: 41.37 | step_microstep: 20.67
[2025-04-26 10:28:33,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.14 | bwd: 5739.77 | bwd_inner: 5698.34 | bwd_allreduce: 41.39 | step: 20.67

  6%|▋         | 2624/41250 [6:20:04<92:48:44,  8.65s/it]
                                                         
{'loss': 0.2057, 'grad_norm': 2.5053272247314453, 'learning_rate': 3.9881691200712815e-05, 'epoch': 0.64}

  6%|▋         | 2624/41250 [6:20:04<92:48:44,  8.65s/it][2025-04-26 10:28:41,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:28:41,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.27 | bwd_microstep: 5703.98 | bwd_inner_microstep: 5691.22 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.03
[2025-04-26 10:28:41,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.27 | bwd: 5704.00 | bwd_inner: 5691.22 | bwd_allreduce: 12.74 | step: 18.04

  6%|▋         | 2625/41250 [6:20:13<92:45:32,  8.65s/it]
                                                         
{'loss': 0.1009, 'grad_norm': 1.6067218780517578, 'learning_rate': 3.988152058817476e-05, 'epoch': 0.64}

  6%|▋         | 2625/41250 [6:20:13<92:45:32,  8.65s/it][2025-04-26 10:28:50,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:28:50,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.03 | bwd_microstep: 5710.97 | bwd_inner_microstep: 5657.60 | bwd_allreduce_microstep: 53.32 | step_microstep: 17.68
[2025-04-26 10:28:50,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.03 | bwd: 5710.98 | bwd_inner: 5657.60 | bwd_allreduce: 53.34 | step: 17.68

  6%|▋         | 2626/41250 [6:20:21<92:39:08,  8.64s/it]
                                                         
{'loss': 0.2529, 'grad_norm': 2.030853509902954, 'learning_rate': 3.9881349853071047e-05, 'epoch': 0.64}

  6%|▋         | 2626/41250 [6:20:21<92:39:08,  8.64s/it][2025-04-26 10:28:59,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.16 | optimizer_step: 0.95
[2025-04-26 10:28:59,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.21 | bwd_microstep: 5726.84 | bwd_inner_microstep: 5681.40 | bwd_allreduce_microstep: 45.39 | step_microstep: 20.17
[2025-04-26 10:28:59,126] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.21 | bwd: 5726.86 | bwd_inner: 5681.40 | bwd_allreduce: 45.41 | step: 20.17

  6%|▋         | 2627/41250 [6:20:30<92:44:29,  8.64s/it]
                                                         
{'loss': 0.2373, 'grad_norm': 5.611416339874268, 'learning_rate': 3.988117899540272e-05, 'epoch': 0.64}

  6%|▋         | 2627/41250 [6:20:30<92:44:29,  8.64s/it][2025-04-26 10:29:07,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:29:07,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.58 | bwd_microstep: 5743.65 | bwd_inner_microstep: 5703.34 | bwd_allreduce_microstep: 40.27 | step_microstep: 18.03
[2025-04-26 10:29:07,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.58 | bwd: 5743.66 | bwd_inner: 5703.34 | bwd_allreduce: 40.28 | step: 18.04

  6%|▋         | 2628/41250 [6:20:38<92:51:46,  8.66s/it]
                                                         
{'loss': 0.1288, 'grad_norm': 1.2644110918045044, 'learning_rate': 3.9881008015170836e-05, 'epoch': 0.64}

  6%|▋         | 2628/41250 [6:20:38<92:51:46,  8.66s/it][2025-04-26 10:29:16,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:29:16,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.61 | bwd_microstep: 5721.00 | bwd_inner_microstep: 5698.31 | bwd_allreduce_microstep: 22.66 | step_microstep: 18.51
[2025-04-26 10:29:16,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.61 | bwd: 5721.02 | bwd_inner: 5698.31 | bwd_allreduce: 22.67 | step: 18.51

  6%|▋         | 2629/41250 [6:20:47<92:51:45,  8.66s/it]
                                                         
{'loss': 0.1029, 'grad_norm': 0.9026946425437927, 'learning_rate': 3.988083691237645e-05, 'epoch': 0.64}

  6%|▋         | 2629/41250 [6:20:47<92:51:45,  8.66s/it][2025-04-26 10:29:25,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.04 | optimizer_step: 1.23
[2025-04-26 10:29:25,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.36 | bwd_microstep: 5762.43 | bwd_inner_microstep: 5638.37 | bwd_allreduce_microstep: 124.01 | step_microstep: 19.50
[2025-04-26 10:29:25,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.36 | bwd: 5762.44 | bwd_inner: 5638.37 | bwd_allreduce: 124.03 | step: 19.50

  6%|▋         | 2630/41250 [6:20:56<92:52:59,  8.66s/it]
                                                         
{'loss': 0.4319, 'grad_norm': 2.5532565116882324, 'learning_rate': 3.9880665687020614e-05, 'epoch': 0.64}

  6%|▋         | 2630/41250 [6:20:56<92:52:59,  8.66s/it][2025-04-26 10:29:33,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:29:33,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.05 | bwd_microstep: 5736.67 | bwd_inner_microstep: 5680.72 | bwd_allreduce_microstep: 55.90 | step_microstep: 18.26
[2025-04-26 10:29:33,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.05 | bwd: 5736.68 | bwd_inner: 5680.72 | bwd_allreduce: 55.92 | step: 18.26

  6%|▋         | 2631/41250 [6:21:04<92:53:49,  8.66s/it]
                                                         
{'loss': 0.1212, 'grad_norm': 1.3753876686096191, 'learning_rate': 3.9880494339104384e-05, 'epoch': 0.64}

  6%|▋         | 2631/41250 [6:21:04<92:53:49,  8.66s/it][2025-04-26 10:29:42,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.65 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:29:42,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.84 | bwd_microstep: 5722.95 | bwd_inner_microstep: 5710.40 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.22
[2025-04-26 10:29:42,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.84 | bwd: 5722.96 | bwd_inner: 5710.40 | bwd_allreduce: 12.52 | step: 18.22

  6%|▋         | 2632/41250 [6:21:13<92:55:20,  8.66s/it]
                                                         
{'loss': 0.204, 'grad_norm': 1.438568353652954, 'learning_rate': 3.9880322868628816e-05, 'epoch': 0.64}

  6%|▋         | 2632/41250 [6:21:13<92:55:20,  8.66s/it][2025-04-26 10:29:51,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:29:51,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.77 | bwd_microstep: 5776.05 | bwd_inner_microstep: 5652.65 | bwd_allreduce_microstep: 123.35 | step_microstep: 17.74
[2025-04-26 10:29:51,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.77 | bwd: 5776.06 | bwd_inner: 5652.65 | bwd_allreduce: 123.37 | step: 17.74

  6%|▋         | 2633/41250 [6:21:22<93:01:24,  8.67s/it]
                                                         
{'loss': 0.0984, 'grad_norm': 1.2570719718933105, 'learning_rate': 3.988015127559497e-05, 'epoch': 0.64}

  6%|▋         | 2633/41250 [6:21:22<93:01:24,  8.67s/it][2025-04-26 10:29:59,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 10:29:59,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.95 | bwd_microstep: 5797.05 | bwd_inner_microstep: 5650.19 | bwd_allreduce_microstep: 146.81 | step_microstep: 18.12
[2025-04-26 10:29:59,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.95 | bwd: 5797.06 | bwd_inner: 5650.19 | bwd_allreduce: 146.83 | step: 18.12

  6%|▋         | 2634/41250 [6:21:31<93:09:14,  8.68s/it]
                                                         
{'loss': 0.1719, 'grad_norm': 1.8916617631912231, 'learning_rate': 3.98799795600039e-05, 'epoch': 0.64}

  6%|▋         | 2634/41250 [6:21:31<93:09:14,  8.68s/it][2025-04-26 10:30:08,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:30:08,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.53 | bwd_microstep: 5777.30 | bwd_inner_microstep: 5689.16 | bwd_allreduce_microstep: 88.10 | step_microstep: 18.00
[2025-04-26 10:30:08,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.54 | bwd: 5777.31 | bwd_inner: 5689.16 | bwd_allreduce: 88.12 | step: 18.00

  6%|▋         | 2635/41250 [6:21:39<93:10:04,  8.69s/it]
                                                         
{'loss': 0.3246, 'grad_norm': 1.5099427700042725, 'learning_rate': 3.987980772185666e-05, 'epoch': 0.64}

  6%|▋         | 2635/41250 [6:21:39<93:10:04,  8.69s/it][2025-04-26 10:30:17,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:30:17,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.62 | bwd_microstep: 5779.50 | bwd_inner_microstep: 5696.26 | bwd_allreduce_microstep: 83.20 | step_microstep: 18.47
[2025-04-26 10:30:17,265] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.62 | bwd: 5779.52 | bwd_inner: 5696.26 | bwd_allreduce: 83.21 | step: 18.47

  6%|▋         | 2636/41250 [6:21:48<93:13:53,  8.69s/it]
                                                         
{'loss': 0.2118, 'grad_norm': 2.229076862335205, 'learning_rate': 3.987963576115433e-05, 'epoch': 0.64}

  6%|▋         | 2636/41250 [6:21:48<93:13:53,  8.69s/it][2025-04-26 10:30:25,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:30:25,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.69 | bwd_microstep: 5797.42 | bwd_inner_microstep: 5646.94 | bwd_allreduce_microstep: 150.43 | step_microstep: 17.76
[2025-04-26 10:30:25,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5797.43 | bwd_inner: 5646.94 | bwd_allreduce: 150.45 | step: 17.77

  6%|▋         | 2637/41250 [6:21:57<93:16:36,  8.70s/it]
                                                         
{'loss': 0.0931, 'grad_norm': 0.8282614350318909, 'learning_rate': 3.9879463677897946e-05, 'epoch': 0.64}

  6%|▋         | 2637/41250 [6:21:57<93:16:36,  8.70s/it][2025-04-26 10:30:34,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:30:34,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.48 | bwd_microstep: 5704.48 | bwd_inner_microstep: 5689.16 | bwd_allreduce_microstep: 15.28 | step_microstep: 17.90
[2025-04-26 10:30:34,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.48 | bwd: 5704.50 | bwd_inner: 5689.16 | bwd_allreduce: 15.30 | step: 17.90

  6%|▋         | 2638/41250 [6:22:05<93:04:16,  8.68s/it]
                                                         
{'loss': 0.1841, 'grad_norm': 1.4080730676651, 'learning_rate': 3.987929147208857e-05, 'epoch': 0.64}

  6%|▋         | 2638/41250 [6:22:05<93:04:16,  8.68s/it][2025-04-26 10:30:43,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:30:43,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.66 | bwd_microstep: 5765.23 | bwd_inner_microstep: 5685.62 | bwd_allreduce_microstep: 79.57 | step_microstep: 17.76
[2025-04-26 10:30:43,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.66 | bwd: 5765.24 | bwd_inner: 5685.62 | bwd_allreduce: 79.58 | step: 17.76

  6%|▋         | 2639/41250 [6:22:14<93:05:36,  8.68s/it]
                                                         
{'loss': 0.3186, 'grad_norm': 1.6245594024658203, 'learning_rate': 3.987911914372728e-05, 'epoch': 0.64}

  6%|▋         | 2639/41250 [6:22:14<93:05:36,  8.68s/it][2025-04-26 10:30:51,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:30:51,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.38 | bwd_microstep: 5731.60 | bwd_inner_microstep: 5658.53 | bwd_allreduce_microstep: 73.03 | step_microstep: 17.73
[2025-04-26 10:30:51,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.38 | bwd: 5731.61 | bwd_inner: 5658.53 | bwd_allreduce: 73.05 | step: 17.73

  6%|▋         | 2640/41250 [6:22:23<92:58:13,  8.67s/it]
                                                         
{'loss': 0.3264, 'grad_norm': 2.357840061187744, 'learning_rate': 3.987894669281514e-05, 'epoch': 0.64}

  6%|▋         | 2640/41250 [6:22:23<92:58:13,  8.67s/it][2025-04-26 10:31:00,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.96 | optimizer_step: 0.93
[2025-04-26 10:31:00,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.27 | bwd_microstep: 5706.90 | bwd_inner_microstep: 5685.46 | bwd_allreduce_microstep: 21.40 | step_microstep: 18.21
[2025-04-26 10:31:00,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.27 | bwd: 5706.91 | bwd_inner: 5685.46 | bwd_allreduce: 21.41 | step: 18.22

  6%|▋         | 2641/41250 [6:22:31<92:51:53,  8.66s/it]
                                                         
{'loss': 0.2111, 'grad_norm': 0.9141759276390076, 'learning_rate': 3.9878774119353194e-05, 'epoch': 0.64}

  6%|▋         | 2641/41250 [6:22:31<92:51:53,  8.66s/it][2025-04-26 10:31:09,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.91
[2025-04-26 10:31:09,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.83 | bwd_microstep: 5779.06 | bwd_inner_microstep: 5651.91 | bwd_allreduce_microstep: 127.11 | step_microstep: 18.34
[2025-04-26 10:31:09,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.83 | bwd: 5779.08 | bwd_inner: 5651.91 | bwd_allreduce: 127.13 | step: 18.35

  6%|▋         | 2642/41250 [6:22:40<92:59:18,  8.67s/it]
                                                         
{'loss': 0.2021, 'grad_norm': 2.1392438411712646, 'learning_rate': 3.987860142334252e-05, 'epoch': 0.64}

  6%|▋         | 2642/41250 [6:22:40<92:59:18,  8.67s/it][2025-04-26 10:31:17,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:31:17,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5762.28 | bwd_inner_microstep: 5705.50 | bwd_allreduce_microstep: 56.73 | step_microstep: 18.14
[2025-04-26 10:31:17,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5762.29 | bwd_inner: 5705.50 | bwd_allreduce: 56.75 | step: 18.15

  6%|▋         | 2643/41250 [6:22:49<93:03:37,  8.68s/it]
                                                         
{'loss': 0.1162, 'grad_norm': 0.8025223016738892, 'learning_rate': 3.987842860478417e-05, 'epoch': 0.64}

  6%|▋         | 2643/41250 [6:22:49<93:03:37,  8.68s/it][2025-04-26 10:31:26,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:31:26,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.50 | bwd_microstep: 5795.83 | bwd_inner_microstep: 5660.15 | bwd_allreduce_microstep: 135.63 | step_microstep: 18.18
[2025-04-26 10:31:26,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.50 | bwd: 5795.84 | bwd_inner: 5660.15 | bwd_allreduce: 135.65 | step: 18.19

  6%|▋         | 2644/41250 [6:22:57<93:09:41,  8.69s/it]
                                                         
{'loss': 0.2129, 'grad_norm': 1.8590567111968994, 'learning_rate': 3.9878255663679226e-05, 'epoch': 0.64}

  6%|▋         | 2644/41250 [6:22:57<93:09:41,  8.69s/it][2025-04-26 10:31:35,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:31:35,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.63 | bwd_microstep: 5714.90 | bwd_inner_microstep: 5702.20 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.28
[2025-04-26 10:31:35,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.63 | bwd: 5714.91 | bwd_inner: 5702.20 | bwd_allreduce: 12.68 | step: 18.28

  6%|▋         | 2645/41250 [6:23:06<93:02:44,  8.68s/it]
                                                         
{'loss': 0.2167, 'grad_norm': 2.5488927364349365, 'learning_rate': 3.987808260002874e-05, 'epoch': 0.64}

  6%|▋         | 2645/41250 [6:23:06<93:02:44,  8.68s/it][2025-04-26 10:31:43,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:31:43,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.22 | bwd_microstep: 5712.24 | bwd_inner_microstep: 5663.04 | bwd_allreduce_microstep: 49.16 | step_microstep: 18.32
[2025-04-26 10:31:43,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.22 | bwd: 5712.25 | bwd_inner: 5663.04 | bwd_allreduce: 49.17 | step: 18.32

  6%|▋         | 2646/41250 [6:23:15<92:54:52,  8.66s/it]
                                                         
{'loss': 0.0891, 'grad_norm': 0.9341164827346802, 'learning_rate': 3.9877909413833794e-05, 'epoch': 0.64}

  6%|▋         | 2646/41250 [6:23:15<92:54:52,  8.66s/it][2025-04-26 10:31:52,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:31:52,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.11 | bwd_microstep: 5788.78 | bwd_inner_microstep: 5776.00 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.44
[2025-04-26 10:31:52,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.12 | bwd: 5788.79 | bwd_inner: 5776.00 | bwd_allreduce: 12.75 | step: 18.44

  6%|▋         | 2647/41250 [6:23:23<93:14:44,  8.70s/it]
                                                         
{'loss': 0.3726, 'grad_norm': 2.1672892570495605, 'learning_rate': 3.9877736105095435e-05, 'epoch': 0.64}

  6%|▋         | 2647/41250 [6:23:23<93:14:44,  8.70s/it][2025-04-26 10:32:01,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:32:01,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.74 | bwd_microstep: 5770.70 | bwd_inner_microstep: 5706.41 | bwd_allreduce_microstep: 64.25 | step_microstep: 17.66
[2025-04-26 10:32:01,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.74 | bwd: 5770.71 | bwd_inner: 5706.41 | bwd_allreduce: 64.26 | step: 17.67

  6%|▋         | 2648/41250 [6:23:32<93:12:50,  8.69s/it]
                                                         
{'loss': 0.1102, 'grad_norm': 0.9790991544723511, 'learning_rate': 3.9877562673814745e-05, 'epoch': 0.64}

  6%|▋         | 2648/41250 [6:23:32<93:12:50,  8.69s/it][2025-04-26 10:32:10,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 10:32:10,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.09 | bwd_microstep: 5724.72 | bwd_inner_microstep: 5651.83 | bwd_allreduce_microstep: 72.85 | step_microstep: 18.36
[2025-04-26 10:32:10,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.09 | bwd: 5724.73 | bwd_inner: 5651.83 | bwd_allreduce: 72.86 | step: 18.37

  6%|▋         | 2649/41250 [6:23:41<93:00:48,  8.67s/it]
                                                         
{'loss': 0.255, 'grad_norm': 1.2416434288024902, 'learning_rate': 3.987738911999279e-05, 'epoch': 0.64}

  6%|▋         | 2649/41250 [6:23:41<93:00:48,  8.67s/it][2025-04-26 10:32:18,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:32:18,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.36 | bwd_microstep: 5879.78 | bwd_inner_microstep: 5650.04 | bwd_allreduce_microstep: 229.69 | step_microstep: 18.08
[2025-04-26 10:32:18,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.36 | bwd: 5879.79 | bwd_inner: 5650.04 | bwd_allreduce: 229.71 | step: 18.09

  6%|▋         | 2650/41250 [6:23:50<93:26:29,  8.71s/it]
                                                         
{'loss': 0.1701, 'grad_norm': 1.4306334257125854, 'learning_rate': 3.9877215443630646e-05, 'epoch': 0.64}

  6%|▋         | 2650/41250 [6:23:50<93:26:29,  8.71s/it][2025-04-26 10:32:27,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:32:27,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.41 | bwd_microstep: 5734.70 | bwd_inner_microstep: 5689.70 | bwd_allreduce_microstep: 44.96 | step_microstep: 18.28
[2025-04-26 10:32:27,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.41 | bwd: 5734.71 | bwd_inner: 5689.70 | bwd_allreduce: 44.98 | step: 18.28

  6%|▋         | 2651/41250 [6:23:58<93:17:31,  8.70s/it]
                                                         
{'loss': 0.2754, 'grad_norm': 1.6698585748672485, 'learning_rate': 3.987704164472938e-05, 'epoch': 0.64}

  6%|▋         | 2651/41250 [6:23:58<93:17:31,  8.70s/it][2025-04-26 10:32:36,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:32:36,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.80 | bwd_microstep: 5704.34 | bwd_inner_microstep: 5691.71 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.23
[2025-04-26 10:32:36,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.80 | bwd: 5704.36 | bwd_inner: 5691.71 | bwd_allreduce: 12.60 | step: 18.24

  6%|▋         | 2652/41250 [6:24:07<93:04:54,  8.68s/it]
                                                         
{'loss': 0.217, 'grad_norm': 1.1637556552886963, 'learning_rate': 3.9876867723290054e-05, 'epoch': 0.64}

  6%|▋         | 2652/41250 [6:24:07<93:04:54,  8.68s/it][2025-04-26 10:32:44,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 10:32:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.42 | bwd_microstep: 5697.14 | bwd_inner_microstep: 5645.38 | bwd_allreduce_microstep: 51.72 | step_microstep: 18.65
[2025-04-26 10:32:44,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.42 | bwd: 5697.15 | bwd_inner: 5645.38 | bwd_allreduce: 51.73 | step: 18.66

  6%|▋         | 2653/41250 [6:24:15<92:47:54,  8.66s/it]
                                                         
{'loss': 0.2616, 'grad_norm': 1.605472207069397, 'learning_rate': 3.9876693679313754e-05, 'epoch': 0.64}

  6%|▋         | 2653/41250 [6:24:15<92:47:54,  8.66s/it][2025-04-26 10:32:53,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.91
[2025-04-26 10:32:53,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.90 | bwd_microstep: 5783.74 | bwd_inner_microstep: 5771.04 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.38
[2025-04-26 10:32:53,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.90 | bwd: 5783.75 | bwd_inner: 5771.04 | bwd_allreduce: 12.66 | step: 18.39

  6%|▋         | 2654/41250 [6:24:24<93:09:03,  8.69s/it]
                                                         
{'loss': 0.1175, 'grad_norm': 0.8579105138778687, 'learning_rate': 3.987651951280154e-05, 'epoch': 0.64}

  6%|▋         | 2654/41250 [6:24:24<93:09:03,  8.69s/it][2025-04-26 10:33:02,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:33:02,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.62 | bwd_microstep: 5774.24 | bwd_inner_microstep: 5694.95 | bwd_allreduce_microstep: 79.24 | step_microstep: 18.26
[2025-04-26 10:33:02,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.63 | bwd: 5774.25 | bwd_inner: 5694.95 | bwd_allreduce: 79.25 | step: 18.26

  6%|▋         | 2655/41250 [6:24:33<93:11:05,  8.69s/it]
                                                         
{'loss': 0.0969, 'grad_norm': 0.841893196105957, 'learning_rate': 3.98763452237545e-05, 'epoch': 0.64}

  6%|▋         | 2655/41250 [6:24:33<93:11:05,  8.69s/it][2025-04-26 10:33:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 10:33:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.40 | bwd_microstep: 5701.61 | bwd_inner_microstep: 5688.92 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.16
[2025-04-26 10:33:10,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.40 | bwd: 5701.62 | bwd_inner: 5688.92 | bwd_allreduce: 12.66 | step: 18.17

  6%|▋         | 2656/41250 [6:24:42<92:58:40,  8.67s/it]
                                                         
{'loss': 0.215, 'grad_norm': 1.720548152923584, 'learning_rate': 3.98761708121737e-05, 'epoch': 0.64}

  6%|▋         | 2656/41250 [6:24:42<92:58:40,  8.67s/it][2025-04-26 10:33:19,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:33:19,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.90 | bwd_microstep: 5703.88 | bwd_inner_microstep: 5691.13 | bwd_allreduce_microstep: 12.70 | step_microstep: 17.83
[2025-04-26 10:33:19,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.90 | bwd: 5703.89 | bwd_inner: 5691.13 | bwd_allreduce: 12.72 | step: 17.83

  6%|▋         | 2657/41250 [6:24:50<92:47:31,  8.66s/it]
                                                         
{'loss': 0.1584, 'grad_norm': 0.8121785521507263, 'learning_rate': 3.987599627806021e-05, 'epoch': 0.64}

  6%|▋         | 2657/41250 [6:24:50<92:47:31,  8.66s/it][2025-04-26 10:33:28,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:33:28,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.96 | bwd_microstep: 5691.35 | bwd_inner_microstep: 5643.59 | bwd_allreduce_microstep: 47.72 | step_microstep: 18.02
[2025-04-26 10:33:28,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.96 | bwd: 5691.37 | bwd_inner: 5643.59 | bwd_allreduce: 47.74 | step: 18.03

  6%|▋         | 2658/41250 [6:24:59<92:39:53,  8.64s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 1.1089136600494385, 'learning_rate': 3.987582162141512e-05, 'epoch': 0.64}

  6%|▋         | 2658/41250 [6:24:59<92:39:53,  8.64s/it][2025-04-26 10:33:36,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 10:33:36,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.87 | bwd_microstep: 5738.67 | bwd_inner_microstep: 5703.10 | bwd_allreduce_microstep: 35.51 | step_microstep: 18.63
[2025-04-26 10:33:36,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.87 | bwd: 5738.68 | bwd_inner: 5703.10 | bwd_allreduce: 35.54 | step: 18.63

  6%|▋         | 2659/41250 [6:25:07<92:45:36,  8.65s/it]
                                                         
{'loss': 0.342, 'grad_norm': 1.822416067123413, 'learning_rate': 3.98756468422395e-05, 'epoch': 0.64}

  6%|▋         | 2659/41250 [6:25:07<92:45:36,  8.65s/it][2025-04-26 10:33:45,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 10:33:45,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5711.08 | bwd_inner_microstep: 5637.97 | bwd_allreduce_microstep: 73.07 | step_microstep: 18.69
[2025-04-26 10:33:45,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5711.09 | bwd_inner: 5637.97 | bwd_allreduce: 73.08 | step: 18.69

  6%|▋         | 2660/41250 [6:25:16<92:39:12,  8.64s/it]
                                                         
{'loss': 0.2163, 'grad_norm': 1.439615249633789, 'learning_rate': 3.987547194053442e-05, 'epoch': 0.64}

  6%|▋         | 2660/41250 [6:25:16<92:39:12,  8.64s/it][2025-04-26 10:33:54,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:33:54,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.68 | bwd_microstep: 5766.80 | bwd_inner_microstep: 5642.85 | bwd_allreduce_microstep: 123.91 | step_microstep: 17.79
[2025-04-26 10:33:54,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.68 | bwd: 5766.81 | bwd_inner: 5642.85 | bwd_allreduce: 123.93 | step: 17.79

  6%|▋         | 2661/41250 [6:25:25<92:45:32,  8.65s/it]
                                                         
{'loss': 0.2214, 'grad_norm': 1.7637590169906616, 'learning_rate': 3.987529691630097e-05, 'epoch': 0.65}

  6%|▋         | 2661/41250 [6:25:25<92:45:32,  8.65s/it][2025-04-26 10:34:02,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 10:34:02,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.40 | bwd_microstep: 5691.48 | bwd_inner_microstep: 5637.58 | bwd_allreduce_microstep: 53.84 | step_microstep: 18.80
[2025-04-26 10:34:02,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.40 | bwd: 5691.49 | bwd_inner: 5637.58 | bwd_allreduce: 53.86 | step: 18.80

  6%|▋         | 2662/41250 [6:25:33<92:33:40,  8.64s/it]
                                                         
{'loss': 0.2139, 'grad_norm': 1.9072366952896118, 'learning_rate': 3.987512176954022e-05, 'epoch': 0.65}

  6%|▋         | 2662/41250 [6:25:33<92:33:40,  8.64s/it][2025-04-26 10:34:11,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 10:34:11,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.59 | bwd_microstep: 5683.47 | bwd_inner_microstep: 5637.26 | bwd_allreduce_microstep: 46.16 | step_microstep: 19.05
[2025-04-26 10:34:11,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.59 | bwd: 5683.49 | bwd_inner: 5637.26 | bwd_allreduce: 46.18 | step: 19.06

  6%|▋         | 2663/41250 [6:25:42<92:25:20,  8.62s/it]
                                                         
{'loss': 0.3055, 'grad_norm': 1.3640968799591064, 'learning_rate': 3.987494650025326e-05, 'epoch': 0.65}

  6%|▋         | 2663/41250 [6:25:42<92:25:20,  8.62s/it][2025-04-26 10:34:19,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 10:34:19,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.39 | bwd_microstep: 5685.98 | bwd_inner_microstep: 5672.98 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.64
[2025-04-26 10:34:19,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.39 | bwd: 5685.99 | bwd_inner: 5672.98 | bwd_allreduce: 12.97 | step: 18.64

  6%|▋         | 2664/41250 [6:25:51<92:24:04,  8.62s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 0.6567831635475159, 'learning_rate': 3.9874771108441156e-05, 'epoch': 0.65}

  6%|▋         | 2664/41250 [6:25:51<92:24:04,  8.62s/it][2025-04-26 10:34:28,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 10:34:28,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.14 | bwd_microstep: 5710.26 | bwd_inner_microstep: 5633.23 | bwd_allreduce_microstep: 76.98 | step_microstep: 18.58
[2025-04-26 10:34:28,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.14 | bwd: 5710.27 | bwd_inner: 5633.23 | bwd_allreduce: 77.01 | step: 18.58

  6%|▋         | 2665/41250 [6:25:59<92:22:21,  8.62s/it]
                                                         
{'loss': 0.2351, 'grad_norm': 1.9713749885559082, 'learning_rate': 3.9874595594105e-05, 'epoch': 0.65}

  6%|▋         | 2665/41250 [6:25:59<92:22:21,  8.62s/it][2025-04-26 10:34:37,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 10:34:37,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.65 | bwd_microstep: 5699.35 | bwd_inner_microstep: 5686.51 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.50
[2025-04-26 10:34:37,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.65 | bwd: 5699.36 | bwd_inner: 5686.51 | bwd_allreduce: 12.81 | step: 18.51

  6%|▋         | 2666/41250 [6:26:08<92:21:43,  8.62s/it]
                                                         
{'loss': 0.2748, 'grad_norm': 1.3747646808624268, 'learning_rate': 3.987441995724587e-05, 'epoch': 0.65}

  6%|▋         | 2666/41250 [6:26:08<92:21:43,  8.62s/it][2025-04-26 10:34:45,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:34:45,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5686.98 | bwd_inner_microstep: 5674.16 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.36
[2025-04-26 10:34:45,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5687.00 | bwd_inner: 5674.16 | bwd_allreduce: 12.80 | step: 18.36

  6%|▋         | 2667/41250 [6:26:16<92:20:47,  8.62s/it]
                                                         
{'loss': 0.1286, 'grad_norm': 0.9138584733009338, 'learning_rate': 3.9874244197864856e-05, 'epoch': 0.65}

  6%|▋         | 2667/41250 [6:26:16<92:20:47,  8.62s/it][2025-04-26 10:34:54,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:34:54,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.88 | bwd_microstep: 5738.69 | bwd_inner_microstep: 5638.08 | bwd_allreduce_microstep: 100.57 | step_microstep: 18.19
[2025-04-26 10:34:54,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.88 | bwd: 5738.71 | bwd_inner: 5638.08 | bwd_allreduce: 100.58 | step: 18.20

  6%|▋         | 2668/41250 [6:26:25<92:27:46,  8.63s/it]
                                                         
{'loss': 0.1274, 'grad_norm': 0.6999167203903198, 'learning_rate': 3.987406831596303e-05, 'epoch': 0.65}

  6%|▋         | 2668/41250 [6:26:25<92:27:46,  8.63s/it][2025-04-26 10:35:02,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.03
[2025-04-26 10:35:02,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.68 | bwd_microstep: 5703.32 | bwd_inner_microstep: 5654.99 | bwd_allreduce_microstep: 48.27 | step_microstep: 19.17
[2025-04-26 10:35:02,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.68 | bwd: 5703.33 | bwd_inner: 5654.99 | bwd_allreduce: 48.30 | step: 19.18

  6%|▋         | 2669/41250 [6:26:34<92:26:10,  8.63s/it]
                                                         
{'loss': 0.2071, 'grad_norm': 1.1047444343566895, 'learning_rate': 3.9873892311541485e-05, 'epoch': 0.65}

  6%|▋         | 2669/41250 [6:26:34<92:26:10,  8.63s/it][2025-04-26 10:35:11,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:35:11,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.22 | bwd_microstep: 5713.57 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 18.74 | step_microstep: 18.25
[2025-04-26 10:35:11,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.22 | bwd: 5713.58 | bwd_inner: 5694.79 | bwd_allreduce: 18.75 | step: 18.25

  6%|▋         | 2670/41250 [6:26:42<92:32:45,  8.64s/it]
                                                         
{'loss': 0.1037, 'grad_norm': 0.7930353879928589, 'learning_rate': 3.98737161846013e-05, 'epoch': 0.65}

  6%|▋         | 2670/41250 [6:26:42<92:32:45,  8.64s/it][2025-04-26 10:35:20,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.10 | optimizer_step: 1.04
[2025-04-26 10:35:20,309] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.20 | bwd_microstep: 5761.12 | bwd_inner_microstep: 5654.25 | bwd_allreduce_microstep: 106.81 | step_microstep: 19.56
[2025-04-26 10:35:20,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.20 | bwd: 5761.14 | bwd_inner: 5654.25 | bwd_allreduce: 106.84 | step: 19.57

  6%|▋         | 2671/41250 [6:26:51<92:41:16,  8.65s/it]
                                                         
{'loss': 0.1713, 'grad_norm': 1.2170583009719849, 'learning_rate': 3.987353993514356e-05, 'epoch': 0.65}

  6%|▋         | 2671/41250 [6:26:51<92:41:16,  8.65s/it][2025-04-26 10:35:28,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 10:35:28,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.42 | bwd_microstep: 5706.26 | bwd_inner_microstep: 5639.66 | bwd_allreduce_microstep: 66.55 | step_microstep: 18.99
[2025-04-26 10:35:28,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.42 | bwd: 5706.28 | bwd_inner: 5639.66 | bwd_allreduce: 66.57 | step: 18.98

  6%|▋         | 2672/41250 [6:27:00<92:34:45,  8.64s/it]
                                                         
{'loss': 0.2345, 'grad_norm': 1.5502322912216187, 'learning_rate': 3.9873363563169366e-05, 'epoch': 0.65}

  6%|▋         | 2672/41250 [6:27:00<92:34:45,  8.64s/it][2025-04-26 10:35:37,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:35:37,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5757.35 | bwd_inner_microstep: 5639.03 | bwd_allreduce_microstep: 118.27 | step_microstep: 18.52
[2025-04-26 10:35:37,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5757.36 | bwd_inner: 5639.03 | bwd_allreduce: 118.29 | step: 18.53

  6%|▋         | 2673/41250 [6:27:08<92:39:23,  8.65s/it]
                                                         
{'loss': 0.0857, 'grad_norm': 0.8831402063369751, 'learning_rate': 3.987318706867979e-05, 'epoch': 0.65}

  6%|▋         | 2673/41250 [6:27:08<92:39:23,  8.65s/it][2025-04-26 10:35:46,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.65 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:35:46,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.11 | bwd_microstep: 5719.22 | bwd_inner_microstep: 5706.21 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.62
[2025-04-26 10:35:46,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.11 | bwd: 5719.23 | bwd_inner: 5706.21 | bwd_allreduce: 12.98 | step: 18.62

  6%|▋         | 2674/41250 [6:27:17<92:39:09,  8.65s/it]
                                                         
{'loss': 0.1179, 'grad_norm': 1.2232033014297485, 'learning_rate': 3.9873010451675925e-05, 'epoch': 0.65}

  6%|▋         | 2674/41250 [6:27:17<92:39:09,  8.65s/it][2025-04-26 10:35:54,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:35:54,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.27 | bwd_microstep: 5726.40 | bwd_inner_microstep: 5713.43 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.81
[2025-04-26 10:35:54,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.27 | bwd: 5726.42 | bwd_inner: 5713.43 | bwd_allreduce: 12.94 | step: 18.82

  6%|▋         | 2675/41250 [6:27:26<92:40:32,  8.65s/it]
                                                         
{'loss': 0.1845, 'grad_norm': 1.358315110206604, 'learning_rate': 3.987283371215885e-05, 'epoch': 0.65}

  6%|▋         | 2675/41250 [6:27:26<92:40:32,  8.65s/it][2025-04-26 10:36:03,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-26 10:36:03,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.99 | bwd_microstep: 5748.23 | bwd_inner_microstep: 5693.53 | bwd_allreduce_microstep: 54.66 | step_microstep: 19.06
[2025-04-26 10:36:03,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.99 | bwd: 5748.25 | bwd_inner: 5693.52 | bwd_allreduce: 54.68 | step: 19.07

  6%|▋         | 2676/41250 [6:27:34<92:47:50,  8.66s/it]
                                                         
{'loss': 0.3803, 'grad_norm': 1.8632863759994507, 'learning_rate': 3.9872656850129674e-05, 'epoch': 0.65}

  6%|▋         | 2676/41250 [6:27:34<92:47:50,  8.66s/it][2025-04-26 10:36:12,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:36:12,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.08 | bwd_microstep: 5716.76 | bwd_inner_microstep: 5703.96 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.54
[2025-04-26 10:36:12,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.08 | bwd: 5716.77 | bwd_inner: 5703.96 | bwd_allreduce: 12.77 | step: 18.55

  6%|▋         | 2677/41250 [6:27:43<92:45:10,  8.66s/it]
                                                         
{'loss': 0.268, 'grad_norm': 1.4225091934204102, 'learning_rate': 3.987247986558947e-05, 'epoch': 0.65}

  6%|▋         | 2677/41250 [6:27:43<92:45:10,  8.66s/it][2025-04-26 10:36:20,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:36:20,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.45 | bwd_microstep: 5784.60 | bwd_inner_microstep: 5651.43 | bwd_allreduce_microstep: 133.12 | step_microstep: 18.69
[2025-04-26 10:36:20,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.45 | bwd: 5784.61 | bwd_inner: 5651.43 | bwd_allreduce: 133.14 | step: 18.69

  6%|▋         | 2678/41250 [6:27:52<92:54:18,  8.67s/it]
                                                         
{'loss': 0.1139, 'grad_norm': 1.336014747619629, 'learning_rate': 3.987230275853934e-05, 'epoch': 0.65}

  6%|▋         | 2678/41250 [6:27:52<92:54:18,  8.67s/it][2025-04-26 10:36:29,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:36:29,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5712.00 | bwd_inner_microstep: 5657.98 | bwd_allreduce_microstep: 53.97 | step_microstep: 18.44
[2025-04-26 10:36:29,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5712.01 | bwd_inner: 5657.98 | bwd_allreduce: 53.99 | step: 18.44

  6%|▋         | 2679/41250 [6:28:00<92:45:27,  8.66s/it]
                                                         
{'loss': 0.1587, 'grad_norm': 0.9324551224708557, 'learning_rate': 3.987212552898038e-05, 'epoch': 0.65}

  6%|▋         | 2679/41250 [6:28:00<92:45:27,  8.66s/it][2025-04-26 10:36:38,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:36:38,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.90 | bwd_microstep: 5715.63 | bwd_inner_microstep: 5702.79 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.69
[2025-04-26 10:36:38,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.90 | bwd: 5715.65 | bwd_inner: 5702.79 | bwd_allreduce: 12.82 | step: 18.69

  6%|▋         | 2680/41250 [6:28:09<92:42:21,  8.65s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 5.715939521789551, 'learning_rate': 3.9871948176913664e-05, 'epoch': 0.65}

  6%|▋         | 2680/41250 [6:28:09<92:42:21,  8.65s/it][2025-04-26 10:36:46,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.01
[2025-04-26 10:36:46,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.43 | bwd_microstep: 5745.46 | bwd_inner_microstep: 5690.29 | bwd_allreduce_microstep: 55.12 | step_microstep: 19.35
[2025-04-26 10:36:46,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.43 | bwd: 5745.47 | bwd_inner: 5690.29 | bwd_allreduce: 55.14 | step: 19.35

  6%|▋         | 2681/41250 [6:28:18<92:47:15,  8.66s/it]
                                                         
{'loss': 0.0524, 'grad_norm': 0.4804858863353729, 'learning_rate': 3.9871770702340304e-05, 'epoch': 0.65}

  6%|▋         | 2681/41250 [6:28:18<92:47:15,  8.66s/it][2025-04-26 10:36:55,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:36:55,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.92 | bwd_microstep: 5708.81 | bwd_inner_microstep: 5696.11 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.59
[2025-04-26 10:36:55,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.92 | bwd: 5708.82 | bwd_inner: 5696.11 | bwd_allreduce: 12.68 | step: 18.60

  7%|▋         | 2682/41250 [6:28:26<92:42:51,  8.65s/it]
                                                         
{'loss': 0.1779, 'grad_norm': 1.3120288848876953, 'learning_rate': 3.9871593105261374e-05, 'epoch': 0.65}

  7%|▋         | 2682/41250 [6:28:26<92:42:51,  8.65s/it][2025-04-26 10:37:04,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:37:04,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.48 | bwd_microstep: 5761.03 | bwd_inner_microstep: 5690.62 | bwd_allreduce_microstep: 70.37 | step_microstep: 18.54
[2025-04-26 10:37:04,209] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.48 | bwd: 5761.05 | bwd_inner: 5690.62 | bwd_allreduce: 70.38 | step: 18.54

  7%|▋         | 2683/41250 [6:28:35<92:49:52,  8.67s/it]
                                                         
{'loss': 0.1987, 'grad_norm': 1.7291159629821777, 'learning_rate': 3.9871415385677994e-05, 'epoch': 0.65}

  7%|▋         | 2683/41250 [6:28:35<92:49:52,  8.67s/it][2025-04-26 10:37:12,829] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:37:12,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.72 | bwd_microstep: 5726.56 | bwd_inner_microstep: 5653.63 | bwd_allreduce_microstep: 72.88 | step_microstep: 18.69
[2025-04-26 10:37:12,830] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.72 | bwd: 5726.58 | bwd_inner: 5653.63 | bwd_allreduce: 72.90 | step: 18.70

  7%|▋         | 2684/41250 [6:28:44<92:41:15,  8.65s/it]
                                                         
{'loss': 0.3471, 'grad_norm': 1.7074285745620728, 'learning_rate': 3.987123754359124e-05, 'epoch': 0.65}

  7%|▋         | 2684/41250 [6:28:44<92:41:15,  8.65s/it][2025-04-26 10:37:21,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.32 | optimizer_step: 1.06
[2025-04-26 10:37:21,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.61 | bwd_microstep: 5702.17 | bwd_inner_microstep: 5688.25 | bwd_allreduce_microstep: 13.86 | step_microstep: 20.04
[2025-04-26 10:37:21,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.61 | bwd: 5702.19 | bwd_inner: 5688.25 | bwd_allreduce: 13.90 | step: 20.04

  7%|▋         | 2685/41250 [6:28:52<92:37:42,  8.65s/it]
                                                         
{'loss': 0.2511, 'grad_norm': 1.8346506357192993, 'learning_rate': 3.9871059579002225e-05, 'epoch': 0.65}

  7%|▋         | 2685/41250 [6:28:52<92:37:42,  8.65s/it][2025-04-26 10:37:30,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 10:37:30,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.33 | bwd_microstep: 5785.01 | bwd_inner_microstep: 5660.53 | bwd_allreduce_microstep: 124.43 | step_microstep: 19.06
[2025-04-26 10:37:30,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.33 | bwd: 5785.03 | bwd_inner: 5660.53 | bwd_allreduce: 124.45 | step: 19.06

  7%|▋         | 2686/41250 [6:29:01<92:49:19,  8.67s/it]
                                                         
{'loss': 0.3199, 'grad_norm': 1.3181636333465576, 'learning_rate': 3.987088149191202e-05, 'epoch': 0.65}

  7%|▋         | 2686/41250 [6:29:01<92:49:19,  8.67s/it][2025-04-26 10:37:38,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:37:38,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5780.69 | bwd_inner_microstep: 5660.11 | bwd_allreduce_microstep: 120.54 | step_microstep: 18.36
[2025-04-26 10:37:38,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5780.71 | bwd_inner: 5660.11 | bwd_allreduce: 120.56 | step: 18.37

  7%|▋         | 2687/41250 [6:29:10<92:54:52,  8.67s/it]
                                                         
{'loss': 0.2917, 'grad_norm': 2.488938093185425, 'learning_rate': 3.987070328232175e-05, 'epoch': 0.65}

  7%|▋         | 2687/41250 [6:29:10<92:54:52,  8.67s/it][2025-04-26 10:37:47,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.09 | optimizer_step: 0.97
[2025-04-26 10:37:47,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.61 | bwd_microstep: 5905.66 | bwd_inner_microstep: 5690.55 | bwd_allreduce_microstep: 215.07 | step_microstep: 18.74
[2025-04-26 10:37:47,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.61 | bwd: 5905.68 | bwd_inner: 5690.55 | bwd_allreduce: 215.09 | step: 18.74

  7%|▋         | 2688/41250 [6:29:18<93:25:36,  8.72s/it]
                                                         
{'loss': 0.3301, 'grad_norm': 2.874983310699463, 'learning_rate': 3.98705249502325e-05, 'epoch': 0.65}

  7%|▋         | 2688/41250 [6:29:18<93:25:36,  8.72s/it][2025-04-26 10:37:56,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:37:56,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.19 | bwd_microstep: 5727.79 | bwd_inner_microstep: 5688.18 | bwd_allreduce_microstep: 39.57 | step_microstep: 18.87
[2025-04-26 10:37:56,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.19 | bwd: 5727.81 | bwd_inner: 5688.18 | bwd_allreduce: 39.58 | step: 18.87

  7%|▋         | 2689/41250 [6:29:27<93:13:17,  8.70s/it]
                                                         
{'loss': 0.2976, 'grad_norm': 1.7767820358276367, 'learning_rate': 3.987034649564537e-05, 'epoch': 0.65}

  7%|▋         | 2689/41250 [6:29:27<93:13:17,  8.70s/it][2025-04-26 10:38:05,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.97
[2025-04-26 10:38:05,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.05 | bwd_microstep: 5791.39 | bwd_inner_microstep: 5703.63 | bwd_allreduce_microstep: 87.71 | step_microstep: 19.04
[2025-04-26 10:38:05,086] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.05 | bwd: 5791.41 | bwd_inner: 5703.63 | bwd_allreduce: 87.74 | step: 19.05

  7%|▋         | 2690/41250 [6:29:36<93:17:30,  8.71s/it]
                                                         
{'loss': 0.4769, 'grad_norm': 1.921228051185608, 'learning_rate': 3.9870167918561464e-05, 'epoch': 0.65}

  7%|▋         | 2690/41250 [6:29:36<93:17:30,  8.71s/it][2025-04-26 10:38:13,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.94
[2025-04-26 10:38:13,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.01 | bwd_microstep: 5765.96 | bwd_inner_microstep: 5709.87 | bwd_allreduce_microstep: 56.02 | step_microstep: 20.08
[2025-04-26 10:38:13,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.01 | bwd: 5765.97 | bwd_inner: 5709.87 | bwd_allreduce: 56.05 | step: 20.08

  7%|▋         | 2691/41250 [6:29:44<93:17:05,  8.71s/it]
                                                         
{'loss': 0.413, 'grad_norm': 1.5299625396728516, 'learning_rate': 3.986998921898188e-05, 'epoch': 0.65}

  7%|▋         | 2691/41250 [6:29:44<93:17:05,  8.71s/it][2025-04-26 10:38:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:38:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.20 | bwd_microstep: 5784.39 | bwd_inner_microstep: 5693.18 | bwd_allreduce_microstep: 91.17 | step_microstep: 18.58
[2025-04-26 10:38:22,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.20 | bwd: 5784.40 | bwd_inner: 5693.18 | bwd_allreduce: 91.19 | step: 18.59

  7%|▋         | 2692/41250 [6:29:53<93:19:03,  8.71s/it]
                                                         
{'loss': 0.0476, 'grad_norm': 0.4333950877189636, 'learning_rate': 3.986981039690772e-05, 'epoch': 0.65}

  7%|▋         | 2692/41250 [6:29:53<93:19:03,  8.71s/it][2025-04-26 10:38:31,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:38:31,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5765.13 | bwd_inner_microstep: 5704.71 | bwd_allreduce_microstep: 60.37 | step_microstep: 18.60
[2025-04-26 10:38:31,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5765.14 | bwd_inner: 5704.71 | bwd_allreduce: 60.39 | step: 18.60

  7%|▋         | 2693/41250 [6:30:02<93:13:15,  8.70s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 2.028259038925171, 'learning_rate': 3.986963145234009e-05, 'epoch': 0.65}

  7%|▋         | 2693/41250 [6:30:02<93:13:15,  8.70s/it][2025-04-26 10:38:39,825] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:38:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.83 | bwd_microstep: 5718.48 | bwd_inner_microstep: 5654.03 | bwd_allreduce_microstep: 64.40 | step_microstep: 18.36
[2025-04-26 10:38:39,826] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.83 | bwd: 5718.49 | bwd_inner: 5654.03 | bwd_allreduce: 64.42 | step: 18.36

  7%|▋         | 2694/41250 [6:30:11<92:58:27,  8.68s/it]
                                                         
{'loss': 0.2131, 'grad_norm': 1.835241675376892, 'learning_rate': 3.986945238528009e-05, 'epoch': 0.65}

  7%|▋         | 2694/41250 [6:30:11<92:58:27,  8.68s/it][2025-04-26 10:38:48,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.43 | optimizer_step: 0.91
[2025-04-26 10:38:48,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 5703.18 | bwd_inner_microstep: 5689.91 | bwd_allreduce_microstep: 13.21 | step_microstep: 19.77
[2025-04-26 10:38:48,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5703.20 | bwd_inner: 5689.92 | bwd_allreduce: 13.24 | step: 19.78

  7%|▋         | 2695/41250 [6:30:19<92:49:35,  8.67s/it]
                                                         
{'loss': 0.2496, 'grad_norm': 1.0941134691238403, 'learning_rate': 3.986927319572882e-05, 'epoch': 0.65}

  7%|▋         | 2695/41250 [6:30:19<92:49:35,  8.67s/it][2025-04-26 10:38:57,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.15 | optimizer_step: 1.04
[2025-04-26 10:38:57,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.32 | bwd_microstep: 5714.28 | bwd_inner_microstep: 5700.97 | bwd_allreduce_microstep: 13.26 | step_microstep: 19.73
[2025-04-26 10:38:57,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.32 | bwd: 5714.30 | bwd_inner: 5700.97 | bwd_allreduce: 13.28 | step: 19.73

  7%|▋         | 2696/41250 [6:30:28<92:46:53,  8.66s/it]
                                                         
{'loss': 0.1392, 'grad_norm': 0.9931756854057312, 'learning_rate': 3.986909388368739e-05, 'epoch': 0.65}

  7%|▋         | 2696/41250 [6:30:28<92:46:53,  8.66s/it][2025-04-26 10:39:05,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 10:39:05,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.87 | bwd_microstep: 5719.56 | bwd_inner_microstep: 5651.29 | bwd_allreduce_microstep: 68.21 | step_microstep: 19.01
[2025-04-26 10:39:05,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.87 | bwd: 5719.57 | bwd_inner: 5651.29 | bwd_allreduce: 68.23 | step: 19.01

  7%|▋         | 2697/41250 [6:30:36<92:40:18,  8.65s/it]
                                                         
{'loss': 0.1122, 'grad_norm': 0.8131526112556458, 'learning_rate': 3.986891444915691e-05, 'epoch': 0.65}

  7%|▋         | 2697/41250 [6:30:36<92:40:18,  8.65s/it][2025-04-26 10:39:14,474] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 10:39:14,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.37 | bwd_microstep: 5773.11 | bwd_inner_microstep: 5760.44 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.23
[2025-04-26 10:39:14,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.37 | bwd: 5773.12 | bwd_inner: 5760.44 | bwd_allreduce: 12.65 | step: 19.24

  7%|▋         | 2698/41250 [6:30:45<92:55:04,  8.68s/it]
                                                         
{'loss': 0.1752, 'grad_norm': 1.187188744544983, 'learning_rate': 3.9868734892138475e-05, 'epoch': 0.65}

  7%|▋         | 2698/41250 [6:30:45<92:55:04,  8.68s/it][2025-04-26 10:39:23,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.06 | optimizer_step: 1.27
[2025-04-26 10:39:23,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.58 | bwd_microstep: 5720.25 | bwd_inner_microstep: 5707.46 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.72
[2025-04-26 10:39:23,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.58 | bwd: 5720.27 | bwd_inner: 5707.45 | bwd_allreduce: 12.77 | step: 19.72

  7%|▋         | 2699/41250 [6:30:54<92:49:51,  8.67s/it]
                                                         
{'loss': 0.1089, 'grad_norm': 0.942938506603241, 'learning_rate': 3.9868555212633205e-05, 'epoch': 0.65}

  7%|▋         | 2699/41250 [6:30:54<92:49:51,  8.67s/it][2025-04-26 10:39:31,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.12 | optimizer_step: 0.93
[2025-04-26 10:39:31,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.12 | bwd_microstep: 5744.12 | bwd_inner_microstep: 5692.05 | bwd_allreduce_microstep: 52.03 | step_microstep: 19.00
[2025-04-26 10:39:31,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.12 | bwd: 5744.14 | bwd_inner: 5692.05 | bwd_allreduce: 52.04 | step: 19.00

  7%|▋         | 2700/41250 [6:31:02<92:51:19,  8.67s/it]
                                                         
{'loss': 0.1976, 'grad_norm': 1.095404863357544, 'learning_rate': 3.9868375410642194e-05, 'epoch': 0.65}

  7%|▋         | 2700/41250 [6:31:02<92:51:19,  8.67s/it][2025-04-26 10:39:40,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:39:40,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.74 | bwd_microstep: 5692.40 | bwd_inner_microstep: 5679.58 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.28
[2025-04-26 10:39:40,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.74 | bwd: 5692.42 | bwd_inner: 5679.58 | bwd_allreduce: 12.79 | step: 18.29

  7%|▋         | 2701/41250 [6:31:11<92:43:06,  8.66s/it]
                                                         
{'loss': 0.2299, 'grad_norm': 1.5482641458511353, 'learning_rate': 3.986819548616655e-05, 'epoch': 0.65}

  7%|▋         | 2701/41250 [6:31:11<92:43:06,  8.66s/it][2025-04-26 10:39:49,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 10:39:49,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.29 | bwd_microstep: 5699.79 | bwd_inner_microstep: 5686.91 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.97
[2025-04-26 10:39:49,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.29 | bwd: 5699.81 | bwd_inner: 5686.90 | bwd_allreduce: 12.86 | step: 18.97

  7%|▋         | 2702/41250 [6:31:20<92:36:11,  8.65s/it]
                                                         
{'loss': 0.0848, 'grad_norm': 0.7952418327331543, 'learning_rate': 3.98680154392074e-05, 'epoch': 0.66}

  7%|▋         | 2702/41250 [6:31:20<92:36:11,  8.65s/it][2025-04-26 10:39:57,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 10:39:57,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.14 | bwd_microstep: 5696.11 | bwd_inner_microstep: 5680.88 | bwd_allreduce_microstep: 15.19 | step_microstep: 19.02
[2025-04-26 10:39:57,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.14 | bwd: 5696.13 | bwd_inner: 5680.88 | bwd_allreduce: 15.21 | step: 19.02

  7%|▋         | 2703/41250 [6:31:28<92:30:43,  8.64s/it]
                                                         
{'loss': 0.1028, 'grad_norm': 0.7937155365943909, 'learning_rate': 3.986783526976584e-05, 'epoch': 0.66}

  7%|▋         | 2703/41250 [6:31:28<92:30:43,  8.64s/it][2025-04-26 10:40:06,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:40:06,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.93 | bwd_microstep: 5758.39 | bwd_inner_microstep: 5745.55 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.91
[2025-04-26 10:40:06,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.93 | bwd: 5758.41 | bwd_inner: 5745.54 | bwd_allreduce: 12.82 | step: 18.92

  7%|▋         | 2704/41250 [6:31:37<92:49:01,  8.67s/it]
                                                         
{'loss': 0.2352, 'grad_norm': 1.0262137651443481, 'learning_rate': 3.9867654977842975e-05, 'epoch': 0.66}

  7%|▋         | 2704/41250 [6:31:37<92:49:01,  8.67s/it][2025-04-26 10:40:15,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-26 10:40:15,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.18 | bwd_microstep: 5764.20 | bwd_inner_microstep: 5645.28 | bwd_allreduce_microstep: 118.86 | step_microstep: 19.01
[2025-04-26 10:40:15,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.18 | bwd: 5764.21 | bwd_inner: 5645.28 | bwd_allreduce: 118.89 | step: 19.02

  7%|▋         | 2705/41250 [6:31:46<92:50:07,  8.67s/it]
                                                         
{'loss': 0.1996, 'grad_norm': 1.493109107017517, 'learning_rate': 3.986747456343993e-05, 'epoch': 0.66}

  7%|▋         | 2705/41250 [6:31:46<92:50:07,  8.67s/it][2025-04-26 10:40:23,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:40:23,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.26 | bwd_microstep: 5757.39 | bwd_inner_microstep: 5636.17 | bwd_allreduce_microstep: 121.18 | step_microstep: 18.71
[2025-04-26 10:40:23,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.26 | bwd: 5757.40 | bwd_inner: 5636.17 | bwd_allreduce: 121.19 | step: 18.71

  7%|▋         | 2706/41250 [6:31:54<92:48:10,  8.67s/it]
                                                         
{'loss': 0.2842, 'grad_norm': 1.444923758506775, 'learning_rate': 3.986729402655782e-05, 'epoch': 0.66}

  7%|▋         | 2706/41250 [6:31:54<92:48:10,  8.67s/it][2025-04-26 10:40:32,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 10:40:32,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.48 | bwd_microstep: 5775.58 | bwd_inner_microstep: 5632.69 | bwd_allreduce_microstep: 142.84 | step_microstep: 19.21
[2025-04-26 10:40:32,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.48 | bwd: 5775.59 | bwd_inner: 5632.69 | bwd_allreduce: 142.86 | step: 19.21

  7%|▋         | 2707/41250 [6:32:03<92:48:34,  8.67s/it]
                                                         
{'loss': 0.1325, 'grad_norm': 0.8711259365081787, 'learning_rate': 3.986711336719774e-05, 'epoch': 0.66}

  7%|▋         | 2707/41250 [6:32:03<92:48:34,  8.67s/it][2025-04-26 10:40:41,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:40:41,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.26 | bwd_microstep: 5749.56 | bwd_inner_microstep: 5647.21 | bwd_allreduce_microstep: 102.31 | step_microstep: 18.76
[2025-04-26 10:40:41,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.26 | bwd: 5749.58 | bwd_inner: 5647.21 | bwd_allreduce: 102.33 | step: 18.76

  7%|▋         | 2708/41250 [6:32:12<92:45:55,  8.66s/it]
                                                         
{'loss': 0.0969, 'grad_norm': 1.4995285272598267, 'learning_rate': 3.986693258536081e-05, 'epoch': 0.66}

  7%|▋         | 2708/41250 [6:32:12<92:45:55,  8.66s/it][2025-04-26 10:40:49,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:40:49,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.34 | bwd_microstep: 5769.81 | bwd_inner_microstep: 5637.39 | bwd_allreduce_microstep: 132.37 | step_microstep: 18.38
[2025-04-26 10:40:49,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.34 | bwd: 5769.82 | bwd_inner: 5637.39 | bwd_allreduce: 132.39 | step: 18.38

  7%|▋         | 2709/41250 [6:32:20<92:47:43,  8.67s/it]
                                                         
{'loss': 0.1446, 'grad_norm': 1.0126206874847412, 'learning_rate': 3.9866751681048156e-05, 'epoch': 0.66}

  7%|▋         | 2709/41250 [6:32:20<92:47:43,  8.67s/it][2025-04-26 10:40:58,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 10:40:58,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.15 | bwd_microstep: 5699.61 | bwd_inner_microstep: 5642.17 | bwd_allreduce_microstep: 57.39 | step_microstep: 18.72
[2025-04-26 10:40:58,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.15 | bwd: 5699.62 | bwd_inner: 5642.17 | bwd_allreduce: 57.41 | step: 18.72

  7%|▋         | 2710/41250 [6:32:29<92:34:57,  8.65s/it]
                                                         
{'loss': 0.1579, 'grad_norm': 2.1892197132110596, 'learning_rate': 3.9866570654260886e-05, 'epoch': 0.66}

  7%|▋         | 2710/41250 [6:32:29<92:34:57,  8.65s/it][2025-04-26 10:41:06,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 10:41:06,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.15 | bwd_microstep: 5706.41 | bwd_inner_microstep: 5693.30 | bwd_allreduce_microstep: 13.05 | step_microstep: 18.86
[2025-04-26 10:41:06,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.15 | bwd: 5706.42 | bwd_inner: 5693.30 | bwd_allreduce: 13.07 | step: 18.86

  7%|▋         | 2711/41250 [6:32:38<92:31:11,  8.64s/it]
                                                         
{'loss': 0.146, 'grad_norm': 1.1988734006881714, 'learning_rate': 3.986638950500011e-05, 'epoch': 0.66}

  7%|▋         | 2711/41250 [6:32:38<92:31:11,  8.64s/it][2025-04-26 10:41:15,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.05 | optimizer_gradients: 1.28 | optimizer_step: 1.03
[2025-04-26 10:41:15,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.74 | bwd_microstep: 5763.67 | bwd_inner_microstep: 5653.75 | bwd_allreduce_microstep: 109.86 | step_microstep: 19.57
[2025-04-26 10:41:15,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.74 | bwd: 5763.69 | bwd_inner: 5653.75 | bwd_allreduce: 109.89 | step: 19.57

  7%|▋         | 2712/41250 [6:32:46<92:35:50,  8.65s/it]
                                                         
{'loss': 0.3127, 'grad_norm': 1.6501699686050415, 'learning_rate': 3.9866208233266954e-05, 'epoch': 0.66}

  7%|▋         | 2712/41250 [6:32:46<92:35:50,  8.65s/it][2025-04-26 10:41:24,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 10:41:24,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.59 | bwd_microstep: 5795.44 | bwd_inner_microstep: 5645.92 | bwd_allreduce_microstep: 149.48 | step_microstep: 19.06
[2025-04-26 10:41:24,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.59 | bwd: 5795.45 | bwd_inner: 5645.92 | bwd_allreduce: 149.49 | step: 19.06

  7%|▋         | 2713/41250 [6:32:55<92:46:10,  8.67s/it]
                                                         
{'loss': 0.112, 'grad_norm': 1.2687991857528687, 'learning_rate': 3.9866026839062535e-05, 'epoch': 0.66}

  7%|▋         | 2713/41250 [6:32:55<92:46:10,  8.67s/it][2025-04-26 10:41:33,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:41:33,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.50 | bwd_microstep: 5764.67 | bwd_inner_microstep: 5652.42 | bwd_allreduce_microstep: 112.20 | step_microstep: 18.57
[2025-04-26 10:41:33,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.50 | bwd: 5764.68 | bwd_inner: 5652.42 | bwd_allreduce: 112.22 | step: 18.57

  7%|▋         | 2714/41250 [6:33:04<92:46:58,  8.67s/it]
                                                         
{'loss': 0.1156, 'grad_norm': 2.1978282928466797, 'learning_rate': 3.986584532238796e-05, 'epoch': 0.66}

  7%|▋         | 2714/41250 [6:33:04<92:46:58,  8.67s/it][2025-04-26 10:41:41,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:41:41,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.05 | bwd_microstep: 5746.91 | bwd_inner_microstep: 5640.00 | bwd_allreduce_microstep: 106.87 | step_microstep: 18.45
[2025-04-26 10:41:41,688] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.05 | bwd: 5746.92 | bwd_inner: 5640.00 | bwd_allreduce: 106.89 | step: 18.46

  7%|▋         | 2715/41250 [6:33:12<92:45:48,  8.67s/it]
                                                         
{'loss': 0.0793, 'grad_norm': 0.7985302209854126, 'learning_rate': 3.986566368324437e-05, 'epoch': 0.66}

  7%|▋         | 2715/41250 [6:33:12<92:45:48,  8.67s/it][2025-04-26 10:41:50,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:41:50,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.70 | bwd_microstep: 5800.16 | bwd_inner_microstep: 5645.72 | bwd_allreduce_microstep: 154.40 | step_microstep: 18.68
[2025-04-26 10:41:50,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.70 | bwd: 5800.18 | bwd_inner: 5645.72 | bwd_allreduce: 154.42 | step: 18.68

  7%|▋         | 2716/41250 [6:33:21<92:51:43,  8.68s/it]
                                                         
{'loss': 0.0693, 'grad_norm': 0.9229581952095032, 'learning_rate': 3.9865481921632854e-05, 'epoch': 0.66}

  7%|▋         | 2716/41250 [6:33:21<92:51:43,  8.68s/it][2025-04-26 10:41:58,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 10:41:58,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.77 | bwd_microstep: 5686.58 | bwd_inner_microstep: 5655.33 | bwd_allreduce_microstep: 31.20 | step_microstep: 19.08
[2025-04-26 10:41:58,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.77 | bwd: 5686.59 | bwd_inner: 5655.33 | bwd_allreduce: 31.22 | step: 19.09

  7%|▋         | 2717/41250 [6:33:30<92:37:38,  8.65s/it]
                                                         
{'loss': 0.0809, 'grad_norm': 2.4517157077789307, 'learning_rate': 3.986530003755456e-05, 'epoch': 0.66}

  7%|▋         | 2717/41250 [6:33:30<92:37:38,  8.65s/it][2025-04-26 10:42:07,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:42:07,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.48 | bwd_microstep: 5764.42 | bwd_inner_microstep: 5751.68 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.58
[2025-04-26 10:42:07,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.48 | bwd: 5764.43 | bwd_inner: 5751.68 | bwd_allreduce: 12.71 | step: 18.58

  7%|▋         | 2718/41250 [6:33:38<92:51:49,  8.68s/it]
                                                         
{'loss': 0.2048, 'grad_norm': 1.947271704673767, 'learning_rate': 3.986511803101059e-05, 'epoch': 0.66}

  7%|▋         | 2718/41250 [6:33:38<92:51:49,  8.68s/it][2025-04-26 10:42:16,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 10:42:16,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.90 | bwd_microstep: 5706.44 | bwd_inner_microstep: 5652.89 | bwd_allreduce_microstep: 53.51 | step_microstep: 17.98
[2025-04-26 10:42:16,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.90 | bwd: 5706.46 | bwd_inner: 5652.89 | bwd_allreduce: 53.52 | step: 17.98

  7%|▋         | 2719/41250 [6:33:47<92:40:14,  8.66s/it]
                                                         
{'loss': 0.2852, 'grad_norm': 1.87856924533844, 'learning_rate': 3.9864935902002086e-05, 'epoch': 0.66}

  7%|▋         | 2719/41250 [6:33:47<92:40:14,  8.66s/it][2025-04-26 10:42:25,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.08 | optimizer_step: 0.99
[2025-04-26 10:42:25,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.42 | bwd_microstep: 5765.91 | bwd_inner_microstep: 5688.88 | bwd_allreduce_microstep: 76.98 | step_microstep: 19.08
[2025-04-26 10:42:25,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.42 | bwd: 5765.92 | bwd_inner: 5688.88 | bwd_allreduce: 77.00 | step: 19.08

  7%|▋         | 2720/41250 [6:33:56<92:44:49,  8.67s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 1.754029631614685, 'learning_rate': 3.986475365053015e-05, 'epoch': 0.66}

  7%|▋         | 2720/41250 [6:33:56<92:44:49,  8.67s/it][2025-04-26 10:42:33,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:42:33,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.19 | bwd_microstep: 5698.44 | bwd_inner_microstep: 5685.64 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.84
[2025-04-26 10:42:33,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.19 | bwd: 5698.45 | bwd_inner: 5685.64 | bwd_allreduce: 12.77 | step: 17.85

  7%|▋         | 2721/41250 [6:34:04<92:36:18,  8.65s/it]
                                                         
{'loss': 0.2385, 'grad_norm': 2.085941791534424, 'learning_rate': 3.986457127659592e-05, 'epoch': 0.66}

  7%|▋         | 2721/41250 [6:34:04<92:36:18,  8.65s/it][2025-04-26 10:42:42,284] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:42:42,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.80 | bwd_microstep: 5715.35 | bwd_inner_microstep: 5702.60 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.17
[2025-04-26 10:42:42,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.80 | bwd: 5715.36 | bwd_inner: 5702.60 | bwd_allreduce: 12.72 | step: 18.17

  7%|▋         | 2722/41250 [6:34:13<92:35:10,  8.65s/it]
                                                         
{'loss': 0.1009, 'grad_norm': 0.9518129825592041, 'learning_rate': 3.9864388780200514e-05, 'epoch': 0.66}

  7%|▋         | 2722/41250 [6:34:13<92:35:10,  8.65s/it][2025-04-26 10:42:51,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.67 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:42:51,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.97 | bwd_microstep: 5781.79 | bwd_inner_microstep: 5768.96 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.51
[2025-04-26 10:42:51,033] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.97 | bwd: 5781.80 | bwd_inner: 5768.96 | bwd_allreduce: 12.80 | step: 19.52

  7%|▋         | 2723/41250 [6:34:22<92:53:36,  8.68s/it]
                                                         
{'loss': 0.1725, 'grad_norm': 3.726203203201294, 'learning_rate': 3.986420616134505e-05, 'epoch': 0.66}

  7%|▋         | 2723/41250 [6:34:22<92:53:36,  8.68s/it][2025-04-26 10:42:59,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.04 | optimizer_step: 0.96
[2025-04-26 10:42:59,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.23 | bwd_microstep: 5780.37 | bwd_inner_microstep: 5663.76 | bwd_allreduce_microstep: 116.55 | step_microstep: 18.75
[2025-04-26 10:42:59,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.23 | bwd: 5780.38 | bwd_inner: 5663.76 | bwd_allreduce: 116.58 | step: 18.75

  7%|▋         | 2724/41250 [6:34:30<92:57:32,  8.69s/it]
                                                         
{'loss': 0.1759, 'grad_norm': 1.6263580322265625, 'learning_rate': 3.9864023420030675e-05, 'epoch': 0.66}

  7%|▋         | 2724/41250 [6:34:30<92:57:32,  8.69s/it][2025-04-26 10:43:08,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.09 | optimizer_step: 1.12
[2025-04-26 10:43:08,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 5715.74 | bwd_inner_microstep: 5654.27 | bwd_allreduce_microstep: 61.43 | step_microstep: 19.32
[2025-04-26 10:43:08,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.15 | bwd: 5715.76 | bwd_inner: 5654.27 | bwd_allreduce: 61.45 | step: 19.32

  7%|▋         | 2725/41250 [6:34:39<92:45:23,  8.67s/it]
                                                         
{'loss': 0.2967, 'grad_norm': 2.7119998931884766, 'learning_rate': 3.9863840556258495e-05, 'epoch': 0.66}

  7%|▋         | 2725/41250 [6:34:39<92:45:23,  8.67s/it][2025-04-26 10:43:17,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 10:43:17,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.92 | bwd_microstep: 5782.81 | bwd_inner_microstep: 5659.49 | bwd_allreduce_microstep: 123.27 | step_microstep: 18.36
[2025-04-26 10:43:17,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.92 | bwd: 5782.82 | bwd_inner: 5659.49 | bwd_allreduce: 123.29 | step: 18.36

  7%|▋         | 2726/41250 [6:34:48<92:51:55,  8.68s/it]
                                                         
{'loss': 0.3397, 'grad_norm': 2.0343759059906006, 'learning_rate': 3.986365757002965e-05, 'epoch': 0.66}

  7%|▋         | 2726/41250 [6:34:48<92:51:55,  8.68s/it][2025-04-26 10:43:25,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:43:25,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.71 | bwd_microstep: 5802.73 | bwd_inner_microstep: 5669.87 | bwd_allreduce_microstep: 132.82 | step_microstep: 18.40
[2025-04-26 10:43:25,773] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.71 | bwd: 5802.75 | bwd_inner: 5669.87 | bwd_allreduce: 132.83 | step: 18.41

  7%|▋         | 2727/41250 [6:34:56<92:58:28,  8.69s/it]
                                                         
{'loss': 0.1959, 'grad_norm': 1.474298357963562, 'learning_rate': 3.986347446134526e-05, 'epoch': 0.66}

  7%|▋         | 2727/41250 [6:34:56<92:58:28,  8.69s/it][2025-04-26 10:43:34,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 10:43:34,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5756.44 | bwd_inner_microstep: 5691.24 | bwd_allreduce_microstep: 65.15 | step_microstep: 18.47
[2025-04-26 10:43:34,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5756.46 | bwd_inner: 5691.24 | bwd_allreduce: 65.17 | step: 18.47

  7%|▋         | 2728/41250 [6:35:05<92:58:35,  8.69s/it]
                                                         
{'loss': 0.159, 'grad_norm': 1.1796826124191284, 'learning_rate': 3.9863291230206456e-05, 'epoch': 0.66}

  7%|▋         | 2728/41250 [6:35:05<92:58:35,  8.69s/it][2025-04-26 10:43:43,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:43:43,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.12 | bwd_microstep: 5733.74 | bwd_inner_microstep: 5666.19 | bwd_allreduce_microstep: 67.51 | step_microstep: 18.14
[2025-04-26 10:43:43,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.12 | bwd: 5733.76 | bwd_inner: 5666.19 | bwd_allreduce: 67.53 | step: 18.14

  7%|▋         | 2729/41250 [6:35:14<92:48:14,  8.67s/it]
                                                         
{'loss': 0.0629, 'grad_norm': 0.8958479762077332, 'learning_rate': 3.9863107876614374e-05, 'epoch': 0.66}

  7%|▋         | 2729/41250 [6:35:14<92:48:14,  8.67s/it][2025-04-26 10:43:51,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:43:51,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.03 | bwd_microstep: 5716.19 | bwd_inner_microstep: 5703.32 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.60
[2025-04-26 10:43:51,759] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.03 | bwd: 5716.21 | bwd_inner: 5703.32 | bwd_allreduce: 12.85 | step: 18.61

  7%|▋         | 2730/41250 [6:35:22<92:45:36,  8.67s/it]
                                                         
{'loss': 0.0875, 'grad_norm': 1.0293335914611816, 'learning_rate': 3.9862924400570134e-05, 'epoch': 0.66}

  7%|▋         | 2730/41250 [6:35:22<92:45:36,  8.67s/it][2025-04-26 10:44:00,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 10:44:00,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5779.51 | bwd_inner_microstep: 5649.70 | bwd_allreduce_microstep: 129.76 | step_microstep: 18.52
[2025-04-26 10:44:00,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5779.52 | bwd_inner: 5649.70 | bwd_allreduce: 129.78 | step: 18.52

  7%|▋         | 2731/41250 [6:35:31<92:50:25,  8.68s/it]
                                                         
{'loss': 0.5817, 'grad_norm': 6.700247764587402, 'learning_rate': 3.986274080207487e-05, 'epoch': 0.66}

  7%|▋         | 2731/41250 [6:35:31<92:50:25,  8.68s/it][2025-04-26 10:44:09,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 10:44:09,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.04 | bwd_microstep: 5703.34 | bwd_inner_microstep: 5651.61 | bwd_allreduce_microstep: 51.68 | step_microstep: 18.90
[2025-04-26 10:44:09,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.04 | bwd: 5703.35 | bwd_inner: 5651.61 | bwd_allreduce: 51.70 | step: 18.91

  7%|▋         | 2732/41250 [6:35:40<92:38:44,  8.66s/it]
                                                         
{'loss': 0.0601, 'grad_norm': 0.5600253343582153, 'learning_rate': 3.986255708112972e-05, 'epoch': 0.66}

  7%|▋         | 2732/41250 [6:35:40<92:38:44,  8.66s/it][2025-04-26 10:44:17,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.24 | optimizer_step: 0.94
[2025-04-26 10:44:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.44 | bwd_microstep: 5809.70 | bwd_inner_microstep: 5657.78 | bwd_allreduce_microstep: 151.86 | step_microstep: 19.42
[2025-04-26 10:44:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.44 | bwd: 5809.71 | bwd_inner: 5657.78 | bwd_allreduce: 151.88 | step: 19.42

  7%|▋         | 2733/41250 [6:35:48<92:51:25,  8.68s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 1.1907482147216797, 'learning_rate': 3.986237323773581e-05, 'epoch': 0.66}

  7%|▋         | 2733/41250 [6:35:48<92:51:25,  8.68s/it][2025-04-26 10:44:26,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:44:26,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.85 | bwd_microstep: 5796.39 | bwd_inner_microstep: 5783.42 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.86
[2025-04-26 10:44:26,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.85 | bwd: 5796.40 | bwd_inner: 5783.42 | bwd_allreduce: 12.93 | step: 18.86

  7%|▋         | 2734/41250 [6:35:57<93:06:55,  8.70s/it]
                                                         
{'loss': 0.2532, 'grad_norm': 2.2214722633361816, 'learning_rate': 3.986218927189427e-05, 'epoch': 0.66}

  7%|▋         | 2734/41250 [6:35:57<93:06:55,  8.70s/it][2025-04-26 10:44:35,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:44:35,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.25 | bwd_microstep: 5777.34 | bwd_inner_microstep: 5706.82 | bwd_allreduce_microstep: 70.47 | step_microstep: 18.86
[2025-04-26 10:44:35,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5777.36 | bwd_inner: 5706.82 | bwd_allreduce: 70.49 | step: 18.86

  7%|▋         | 2735/41250 [6:36:06<93:09:33,  8.71s/it]
                                                         
{'loss': 0.0747, 'grad_norm': 0.8114299178123474, 'learning_rate': 3.986200518360625e-05, 'epoch': 0.66}

  7%|▋         | 2735/41250 [6:36:06<93:09:33,  8.71s/it][2025-04-26 10:44:43,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 10:44:43,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.44 | bwd_microstep: 5706.29 | bwd_inner_microstep: 5693.28 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.20
[2025-04-26 10:44:43,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.44 | bwd: 5706.31 | bwd_inner: 5693.28 | bwd_allreduce: 12.99 | step: 19.20

  7%|▋         | 2736/41250 [6:36:15<92:57:11,  8.69s/it]
                                                         
{'loss': 0.2668, 'grad_norm': 1.971821904182434, 'learning_rate': 3.986182097287287e-05, 'epoch': 0.66}

  7%|▋         | 2736/41250 [6:36:15<92:57:11,  8.69s/it][2025-04-26 10:44:52,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 0.92
[2025-04-26 10:44:52,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.83 | bwd_microstep: 5749.50 | bwd_inner_microstep: 5690.46 | bwd_allreduce_microstep: 58.99 | step_microstep: 18.96
[2025-04-26 10:44:52,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.83 | bwd: 5749.52 | bwd_inner: 5690.46 | bwd_allreduce: 59.01 | step: 18.96

  7%|▋         | 2737/41250 [6:36:23<92:55:48,  8.69s/it]
                                                         
{'loss': 0.3895, 'grad_norm': 1.630080223083496, 'learning_rate': 3.986163663969527e-05, 'epoch': 0.66}

  7%|▋         | 2737/41250 [6:36:23<92:55:48,  8.69s/it][2025-04-26 10:45:01,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:45:01,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.58 | bwd_microstep: 5761.62 | bwd_inner_microstep: 5656.70 | bwd_allreduce_microstep: 104.88 | step_microstep: 18.54
[2025-04-26 10:45:01,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.58 | bwd: 5761.63 | bwd_inner: 5656.70 | bwd_allreduce: 104.89 | step: 18.55

  7%|▋         | 2738/41250 [6:36:32<92:52:34,  8.68s/it]
                                                         
{'loss': 0.2212, 'grad_norm': 2.1176037788391113, 'learning_rate': 3.986145218407459e-05, 'epoch': 0.66}

  7%|▋         | 2738/41250 [6:36:32<92:52:34,  8.68s/it][2025-04-26 10:45:09,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:45:09,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.00 | bwd_microstep: 5704.16 | bwd_inner_microstep: 5691.45 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.56
[2025-04-26 10:45:09,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.00 | bwd: 5704.17 | bwd_inner: 5691.45 | bwd_allreduce: 12.68 | step: 18.57

  7%|▋         | 2739/41250 [6:36:41<92:42:21,  8.67s/it]
                                                         
{'loss': 0.282, 'grad_norm': 1.6385440826416016, 'learning_rate': 3.986126760601196e-05, 'epoch': 0.66}

  7%|▋         | 2739/41250 [6:36:41<92:42:21,  8.67s/it][2025-04-26 10:45:18,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:45:18,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.94 | bwd_microstep: 5713.56 | bwd_inner_microstep: 5644.32 | bwd_allreduce_microstep: 69.19 | step_microstep: 18.55
[2025-04-26 10:45:18,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.94 | bwd: 5713.57 | bwd_inner: 5644.32 | bwd_allreduce: 69.20 | step: 18.55

  7%|▋         | 2740/41250 [6:36:49<92:33:01,  8.65s/it]
                                                         
{'loss': 0.2522, 'grad_norm': 3.1344659328460693, 'learning_rate': 3.9861082905508525e-05, 'epoch': 0.66}

  7%|▋         | 2740/41250 [6:36:49<92:33:01,  8.65s/it][2025-04-26 10:45:27,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.24 | optimizer_step: 1.07
[2025-04-26 10:45:27,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.18 | bwd_microstep: 5721.38 | bwd_inner_microstep: 5708.47 | bwd_allreduce_microstep: 12.86 | step_microstep: 19.97
[2025-04-26 10:45:27,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.18 | bwd: 5721.40 | bwd_inner: 5708.47 | bwd_allreduce: 12.88 | step: 19.97

  7%|▋         | 2741/41250 [6:36:58<92:33:08,  8.65s/it]
                                                         
{'loss': 0.2838, 'grad_norm': 1.9798461198806763, 'learning_rate': 3.9860898082565415e-05, 'epoch': 0.66}

  7%|▋         | 2741/41250 [6:36:58<92:33:08,  8.65s/it][2025-04-26 10:45:35,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:45:35,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.23 | bwd_microstep: 5723.17 | bwd_inner_microstep: 5678.69 | bwd_allreduce_microstep: 44.43 | step_microstep: 18.75
[2025-04-26 10:45:35,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.23 | bwd: 5723.18 | bwd_inner: 5678.69 | bwd_allreduce: 44.45 | step: 18.75

  7%|▋         | 2742/41250 [6:37:07<92:35:04,  8.66s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 1.0126765966415405, 'learning_rate': 3.986071313718378e-05, 'epoch': 0.66}

  7%|▋         | 2742/41250 [6:37:07<92:35:04,  8.66s/it][2025-04-26 10:45:44,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 10:45:44,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.30 | bwd_microstep: 5700.65 | bwd_inner_microstep: 5687.96 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.06
[2025-04-26 10:45:44,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5700.66 | bwd_inner: 5687.96 | bwd_allreduce: 12.66 | step: 19.06

  7%|▋         | 2743/41250 [6:37:15<92:29:20,  8.65s/it]
                                                         
{'loss': 0.4921, 'grad_norm': 4.338215351104736, 'learning_rate': 3.9860528069364753e-05, 'epoch': 0.66}

  7%|▋         | 2743/41250 [6:37:15<92:29:20,  8.65s/it][2025-04-26 10:45:53,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 10:45:53,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.37 | bwd_microstep: 5705.26 | bwd_inner_microstep: 5654.82 | bwd_allreduce_microstep: 50.39 | step_microstep: 18.96
[2025-04-26 10:45:53,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.37 | bwd: 5705.27 | bwd_inner: 5654.82 | bwd_allreduce: 50.41 | step: 18.96

  7%|▋         | 2744/41250 [6:37:24<92:22:29,  8.64s/it]
                                                         
{'loss': 0.3019, 'grad_norm': 2.21144700050354, 'learning_rate': 3.986034287910948e-05, 'epoch': 0.67}

  7%|▋         | 2744/41250 [6:37:24<92:22:29,  8.64s/it][2025-04-26 10:46:01,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:46:01,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.94 | bwd_microstep: 5691.96 | bwd_inner_microstep: 5675.31 | bwd_allreduce_microstep: 16.60 | step_microstep: 18.58
[2025-04-26 10:46:01,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.94 | bwd: 5691.97 | bwd_inner: 5675.31 | bwd_allreduce: 16.62 | step: 18.58

  7%|▋         | 2745/41250 [6:37:32<92:18:33,  8.63s/it]
                                                         
{'loss': 0.1379, 'grad_norm': 0.6217324137687683, 'learning_rate': 3.9860157566419095e-05, 'epoch': 0.67}

  7%|▋         | 2745/41250 [6:37:32<92:18:33,  8.63s/it][2025-04-26 10:46:10,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:46:10,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.60 | bwd_microstep: 5708.95 | bwd_inner_microstep: 5696.09 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.79
[2025-04-26 10:46:10,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.60 | bwd: 5708.96 | bwd_inner: 5696.09 | bwd_allreduce: 12.83 | step: 18.79

  7%|▋         | 2746/41250 [6:37:41<92:18:55,  8.63s/it]
                                                         
{'loss': 0.1148, 'grad_norm': 1.0326825380325317, 'learning_rate': 3.9859972131294746e-05, 'epoch': 0.67}

  7%|▋         | 2746/41250 [6:37:41<92:18:55,  8.63s/it][2025-04-26 10:46:19,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 10:46:19,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.54 | bwd_microstep: 5874.75 | bwd_inner_microstep: 5690.72 | bwd_allreduce_microstep: 183.98 | step_microstep: 18.76
[2025-04-26 10:46:19,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.54 | bwd: 5874.77 | bwd_inner: 5690.72 | bwd_allreduce: 184.00 | step: 18.76

  7%|▋         | 2747/41250 [6:37:50<92:50:58,  8.68s/it]
                                                         
{'loss': 0.1771, 'grad_norm': 1.936643123626709, 'learning_rate': 3.985978657373758e-05, 'epoch': 0.67}

  7%|▋         | 2747/41250 [6:37:50<92:50:58,  8.68s/it][2025-04-26 10:46:27,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.14
[2025-04-26 10:46:27,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5889.62 | bwd_inner_microstep: 5652.91 | bwd_allreduce_microstep: 236.66 | step_microstep: 19.45
[2025-04-26 10:46:27,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5889.63 | bwd_inner: 5652.91 | bwd_allreduce: 236.68 | step: 19.45

  7%|▋         | 2748/41250 [6:37:59<93:14:38,  8.72s/it]
                                                         
{'loss': 0.1971, 'grad_norm': 1.3130383491516113, 'learning_rate': 3.9859600893748726e-05, 'epoch': 0.67}

  7%|▋         | 2748/41250 [6:37:59<93:14:38,  8.72s/it][2025-04-26 10:46:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:46:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.56 | bwd_microstep: 5768.47 | bwd_inner_microstep: 5638.46 | bwd_allreduce_microstep: 129.96 | step_microstep: 18.87
[2025-04-26 10:46:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.56 | bwd: 5768.49 | bwd_inner: 5638.46 | bwd_allreduce: 129.98 | step: 18.88

  7%|▋         | 2749/41250 [6:38:07<93:05:42,  8.70s/it]
                                                         
{'loss': 0.227, 'grad_norm': 1.3105660676956177, 'learning_rate': 3.985941509132935e-05, 'epoch': 0.67}

  7%|▋         | 2749/41250 [6:38:07<93:05:42,  8.70s/it][2025-04-26 10:46:45,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:46:45,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.91 | bwd_microstep: 5744.94 | bwd_inner_microstep: 5696.44 | bwd_allreduce_microstep: 48.45 | step_microstep: 18.32
[2025-04-26 10:46:45,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.91 | bwd: 5744.95 | bwd_inner: 5696.44 | bwd_allreduce: 48.47 | step: 18.33

  7%|▋         | 2750/41250 [6:38:16<92:59:10,  8.69s/it]
                                                         
{'loss': 0.1215, 'grad_norm': 1.0309813022613525, 'learning_rate': 3.985922916648058e-05, 'epoch': 0.67}

  7%|▋         | 2750/41250 [6:38:16<92:59:10,  8.69s/it][2025-04-26 10:46:53,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 10:46:53,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5715.33 | bwd_inner_microstep: 5702.62 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.50
[2025-04-26 10:46:53,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.66 | bwd: 5715.35 | bwd_inner: 5702.62 | bwd_allreduce: 12.69 | step: 18.51

  7%|▋         | 2751/41250 [6:38:25<92:51:26,  8.68s/it]
                                                         
{'loss': 0.1734, 'grad_norm': 2.031160593032837, 'learning_rate': 3.985904311920356e-05, 'epoch': 0.67}

  7%|▋         | 2751/41250 [6:38:25<92:51:26,  8.68s/it][2025-04-26 10:47:02,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.04 | optimizer_step: 1.07
[2025-04-26 10:47:02,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2874.63 | bwd_microstep: 5765.64 | bwd_inner_microstep: 5751.39 | bwd_allreduce_microstep: 14.21 | step_microstep: 18.80
[2025-04-26 10:47:02,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2874.63 | bwd: 5765.65 | bwd_inner: 5751.39 | bwd_allreduce: 14.23 | step: 18.80

  7%|▋         | 2752/41250 [6:38:33<92:58:23,  8.69s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 1.1766390800476074, 'learning_rate': 3.9858856949499456e-05, 'epoch': 0.67}

  7%|▋         | 2752/41250 [6:38:33<92:58:23,  8.69s/it][2025-04-26 10:47:11,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 10:47:11,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.81 | bwd_microstep: 5842.08 | bwd_inner_microstep: 5704.24 | bwd_allreduce_microstep: 137.79 | step_microstep: 18.57
[2025-04-26 10:47:11,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.81 | bwd: 5842.09 | bwd_inner: 5704.24 | bwd_allreduce: 137.81 | step: 18.57

  7%|▋         | 2753/41250 [6:38:42<93:18:07,  8.73s/it]
                                                         
{'loss': 0.1648, 'grad_norm': 1.2169232368469238, 'learning_rate': 3.9858670657369405e-05, 'epoch': 0.67}

  7%|▋         | 2753/41250 [6:38:42<93:18:07,  8.73s/it][2025-04-26 10:47:20,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 10:47:20,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.72 | bwd_microstep: 5717.94 | bwd_inner_microstep: 5678.03 | bwd_allreduce_microstep: 39.87 | step_microstep: 18.16
[2025-04-26 10:47:20,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.72 | bwd: 5717.95 | bwd_inner: 5678.03 | bwd_allreduce: 39.88 | step: 18.16

  7%|▋         | 2754/41250 [6:38:51<93:05:43,  8.71s/it]
                                                         
{'loss': 0.0613, 'grad_norm': 0.5858533382415771, 'learning_rate': 3.985848424281455e-05, 'epoch': 0.67}

  7%|▋         | 2754/41250 [6:38:51<93:05:43,  8.71s/it][2025-04-26 10:47:28,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 10:47:28,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.56 | bwd_microstep: 5711.58 | bwd_inner_microstep: 5698.68 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.93
[2025-04-26 10:47:28,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.56 | bwd: 5711.59 | bwd_inner: 5698.68 | bwd_allreduce: 12.87 | step: 18.94

  7%|▋         | 2755/41250 [6:38:59<92:55:19,  8.69s/it]
                                                         
{'loss': 0.0947, 'grad_norm': 1.012740969657898, 'learning_rate': 3.9858297705836054e-05, 'epoch': 0.67}

  7%|▋         | 2755/41250 [6:38:59<92:55:19,  8.69s/it][2025-04-26 10:47:37,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 10:47:37,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.38 | bwd_microstep: 5730.45 | bwd_inner_microstep: 5640.65 | bwd_allreduce_microstep: 89.75 | step_microstep: 18.83
[2025-04-26 10:47:37,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.38 | bwd: 5730.46 | bwd_inner: 5640.65 | bwd_allreduce: 89.77 | step: 18.83

  7%|▋         | 2756/41250 [6:39:08<92:42:13,  8.67s/it]
                                                         
{'loss': 0.0864, 'grad_norm': 1.2480028867721558, 'learning_rate': 3.985811104643505e-05, 'epoch': 0.67}

  7%|▋         | 2756/41250 [6:39:08<92:42:13,  8.67s/it][2025-04-26 10:47:46,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 10:47:46,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.09 | bwd_microstep: 5760.83 | bwd_inner_microstep: 5691.12 | bwd_allreduce_microstep: 69.66 | step_microstep: 18.41
[2025-04-26 10:47:46,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.09 | bwd: 5760.84 | bwd_inner: 5691.12 | bwd_allreduce: 69.68 | step: 18.42

  7%|▋         | 2757/41250 [6:39:17<92:47:44,  8.68s/it]
                                                         
{'loss': 0.2109, 'grad_norm': 1.9978950023651123, 'learning_rate': 3.98579242646127e-05, 'epoch': 0.67}

  7%|▋         | 2757/41250 [6:39:17<92:47:44,  8.68s/it][2025-04-26 10:47:54,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-26 10:47:54,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.65 | bwd_microstep: 5714.93 | bwd_inner_microstep: 5677.59 | bwd_allreduce_microstep: 37.29 | step_microstep: 18.88
[2025-04-26 10:47:54,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.65 | bwd: 5714.95 | bwd_inner: 5677.59 | bwd_allreduce: 37.31 | step: 18.88

  7%|▋         | 2758/41250 [6:39:25<92:44:12,  8.67s/it]
                                                         
{'loss': 0.0865, 'grad_norm': 0.8700622916221619, 'learning_rate': 3.985773736037015e-05, 'epoch': 0.67}

  7%|▋         | 2758/41250 [6:39:25<92:44:12,  8.67s/it][2025-04-26 10:48:03,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.17 | optimizer_step: 1.00
[2025-04-26 10:48:03,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.98 | bwd_microstep: 5692.47 | bwd_inner_microstep: 5660.53 | bwd_allreduce_microstep: 31.88 | step_microstep: 18.87
[2025-04-26 10:48:03,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.98 | bwd: 5692.49 | bwd_inner: 5660.53 | bwd_allreduce: 31.91 | step: 18.87

  7%|▋         | 2759/41250 [6:39:34<92:32:36,  8.66s/it]
                                                         
{'loss': 0.1515, 'grad_norm': 1.4134043455123901, 'learning_rate': 3.985755033370856e-05, 'epoch': 0.67}

  7%|▋         | 2759/41250 [6:39:34<92:32:36,  8.66s/it][2025-04-26 10:48:12,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.06 | optimizer_step: 1.00
[2025-04-26 10:48:12,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.52 | bwd_microstep: 5722.18 | bwd_inner_microstep: 5708.87 | bwd_allreduce_microstep: 13.25 | step_microstep: 19.39
[2025-04-26 10:48:12,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.52 | bwd: 5722.19 | bwd_inner: 5708.87 | bwd_allreduce: 13.28 | step: 19.39

  7%|▋         | 2760/41250 [6:39:43<92:32:54,  8.66s/it]
                                                         
{'loss': 0.1229, 'grad_norm': 1.3336083889007568, 'learning_rate': 3.9857363184629074e-05, 'epoch': 0.67}

  7%|▋         | 2760/41250 [6:39:43<92:32:54,  8.66s/it][2025-04-26 10:48:20,648] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:48:20,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.16 | bwd_microstep: 5713.95 | bwd_inner_microstep: 5701.29 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.57
[2025-04-26 10:48:20,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.16 | bwd: 5713.96 | bwd_inner: 5701.29 | bwd_allreduce: 12.63 | step: 18.57

  7%|▋         | 2761/41250 [6:39:51<92:28:44,  8.65s/it]
                                                         
{'loss': 0.199, 'grad_norm': 1.9582908153533936, 'learning_rate': 3.985717591313286e-05, 'epoch': 0.67}

  7%|▋         | 2761/41250 [6:39:51<92:28:44,  8.65s/it][2025-04-26 10:48:29,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:48:29,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.94 | bwd_microstep: 5754.01 | bwd_inner_microstep: 5698.01 | bwd_allreduce_microstep: 55.96 | step_microstep: 18.33
[2025-04-26 10:48:29,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.94 | bwd: 5754.02 | bwd_inner: 5698.01 | bwd_allreduce: 55.98 | step: 18.34

  7%|▋         | 2762/41250 [6:40:00<92:33:38,  8.66s/it]
                                                         
{'loss': 0.159, 'grad_norm': 3.605834722518921, 'learning_rate': 3.985698851922105e-05, 'epoch': 0.67}

  7%|▋         | 2762/41250 [6:40:00<92:33:38,  8.66s/it][2025-04-26 10:48:38,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.66 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:48:38,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5776.28 | bwd_inner_microstep: 5689.40 | bwd_allreduce_microstep: 86.83 | step_microstep: 19.26
[2025-04-26 10:48:38,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5776.30 | bwd_inner: 5689.40 | bwd_allreduce: 86.85 | step: 19.26

  7%|▋         | 2763/41250 [6:40:09<92:42:39,  8.67s/it]
                                                         
{'loss': 0.2011, 'grad_norm': 0.8662228584289551, 'learning_rate': 3.985680100289482e-05, 'epoch': 0.67}

  7%|▋         | 2763/41250 [6:40:09<92:42:39,  8.67s/it][2025-04-26 10:48:46,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:48:46,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.66 | bwd_microstep: 5698.50 | bwd_inner_microstep: 5685.37 | bwd_allreduce_microstep: 13.09 | step_microstep: 18.71
[2025-04-26 10:48:46,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.66 | bwd: 5698.52 | bwd_inner: 5685.37 | bwd_allreduce: 13.11 | step: 18.71

  7%|▋         | 2764/41250 [6:40:17<92:32:41,  8.66s/it]
                                                         
{'loss': 0.1194, 'grad_norm': 1.2163711786270142, 'learning_rate': 3.985661336415531e-05, 'epoch': 0.67}

  7%|▋         | 2764/41250 [6:40:17<92:32:41,  8.66s/it][2025-04-26 10:48:55,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 10:48:55,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5793.02 | bwd_inner_microstep: 5648.02 | bwd_allreduce_microstep: 144.96 | step_microstep: 19.26
[2025-04-26 10:48:55,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5793.03 | bwd_inner: 5648.02 | bwd_allreduce: 144.97 | step: 19.26

  7%|▋         | 2765/41250 [6:40:26<92:40:56,  8.67s/it]
                                                         
{'loss': 0.1511, 'grad_norm': 2.5969486236572266, 'learning_rate': 3.9856425603003685e-05, 'epoch': 0.67}

  7%|▋         | 2765/41250 [6:40:26<92:40:56,  8.67s/it][2025-04-26 10:49:03,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:49:03,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.69 | bwd_microstep: 5722.21 | bwd_inner_microstep: 5653.52 | bwd_allreduce_microstep: 68.65 | step_microstep: 18.52
[2025-04-26 10:49:03,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.69 | bwd: 5722.23 | bwd_inner: 5653.52 | bwd_allreduce: 68.66 | step: 18.52

  7%|▋         | 2766/41250 [6:40:35<92:32:32,  8.66s/it]
                                                         
{'loss': 0.1443, 'grad_norm': 1.7028121948242188, 'learning_rate': 3.98562377194411e-05, 'epoch': 0.67}

  7%|▋         | 2766/41250 [6:40:35<92:32:32,  8.66s/it][2025-04-26 10:49:12,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:49:12,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.67 | bwd_microstep: 5711.53 | bwd_inner_microstep: 5698.42 | bwd_allreduce_microstep: 13.06 | step_microstep: 18.42
[2025-04-26 10:49:12,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.67 | bwd: 5711.54 | bwd_inner: 5698.42 | bwd_allreduce: 13.08 | step: 18.43

  7%|▋         | 2767/41250 [6:40:43<92:30:16,  8.65s/it]
                                                         
{'loss': 0.046, 'grad_norm': 0.6094543933868408, 'learning_rate': 3.985604971346872e-05, 'epoch': 0.67}

  7%|▋         | 2767/41250 [6:40:43<92:30:16,  8.65s/it][2025-04-26 10:49:21,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:49:21,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.35 | bwd_microstep: 5708.49 | bwd_inner_microstep: 5654.93 | bwd_allreduce_microstep: 53.52 | step_microstep: 18.35
[2025-04-26 10:49:21,259] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.35 | bwd: 5708.51 | bwd_inner: 5654.93 | bwd_allreduce: 53.54 | step: 18.37

  7%|▋         | 2768/41250 [6:40:52<92:26:33,  8.65s/it]
                                                         
{'loss': 0.351, 'grad_norm': 1.954616904258728, 'learning_rate': 3.9855861585087696e-05, 'epoch': 0.67}

  7%|▋         | 2768/41250 [6:40:52<92:26:33,  8.65s/it][2025-04-26 10:49:29,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 10:49:29,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.72 | bwd_microstep: 5719.79 | bwd_inner_microstep: 5660.17 | bwd_allreduce_microstep: 59.58 | step_microstep: 18.58
[2025-04-26 10:49:29,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.72 | bwd: 5719.80 | bwd_inner: 5660.17 | bwd_allreduce: 59.60 | step: 18.59

  7%|▋         | 2769/41250 [6:41:01<92:23:57,  8.64s/it]
                                                         
{'loss': 0.152, 'grad_norm': 1.61016047000885, 'learning_rate': 3.9855673334299196e-05, 'epoch': 0.67}

  7%|▋         | 2769/41250 [6:41:01<92:23:57,  8.64s/it][2025-04-26 10:49:38,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:49:38,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.44 | bwd_microstep: 5742.27 | bwd_inner_microstep: 5705.04 | bwd_allreduce_microstep: 37.19 | step_microstep: 18.43
[2025-04-26 10:49:38,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.44 | bwd: 5742.29 | bwd_inner: 5705.04 | bwd_allreduce: 37.21 | step: 18.43

  7%|▋         | 2770/41250 [6:41:09<92:31:15,  8.66s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 2.0833444595336914, 'learning_rate': 3.985548496110437e-05, 'epoch': 0.67}

  7%|▋         | 2770/41250 [6:41:09<92:31:15,  8.66s/it][2025-04-26 10:49:47,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 1.21
[2025-04-26 10:49:47,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.71 | bwd_microstep: 5873.35 | bwd_inner_microstep: 5699.77 | bwd_allreduce_microstep: 173.53 | step_microstep: 19.33
[2025-04-26 10:49:47,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.71 | bwd: 5873.37 | bwd_inner: 5699.77 | bwd_allreduce: 173.56 | step: 19.33

  7%|▋         | 2771/41250 [6:41:18<93:00:24,  8.70s/it]
                                                         
{'loss': 0.0492, 'grad_norm': 0.8819724321365356, 'learning_rate': 3.985529646550439e-05, 'epoch': 0.67}

  7%|▋         | 2771/41250 [6:41:18<93:00:24,  8.70s/it][2025-04-26 10:49:56,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.29 | optimizer_step: 0.90
[2025-04-26 10:49:56,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.54 | bwd_microstep: 5772.83 | bwd_inner_microstep: 5644.09 | bwd_allreduce_microstep: 128.69 | step_microstep: 19.26
[2025-04-26 10:49:56,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.54 | bwd: 5772.84 | bwd_inner: 5644.09 | bwd_allreduce: 128.71 | step: 19.27

  7%|▋         | 2772/41250 [6:41:27<92:59:43,  8.70s/it]
                                                         
{'loss': 0.2183, 'grad_norm': 1.7321561574935913, 'learning_rate': 3.985510784750041e-05, 'epoch': 0.67}

  7%|▋         | 2772/41250 [6:41:27<92:59:43,  8.70s/it][2025-04-26 10:50:04,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.94 | optimizer_gradients: 1.20 | optimizer_step: 0.89
[2025-04-26 10:50:04,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.25 | bwd_microstep: 5770.56 | bwd_inner_microstep: 5707.56 | bwd_allreduce_microstep: 62.95 | step_microstep: 20.54
[2025-04-26 10:50:04,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.25 | bwd: 5770.57 | bwd_inner: 5707.56 | bwd_allreduce: 62.97 | step: 20.55

  7%|▋         | 2773/41250 [6:41:35<93:00:54,  8.70s/it]
                                                         
{'loss': 0.0894, 'grad_norm': 1.2564022541046143, 'learning_rate': 3.985491910709359e-05, 'epoch': 0.67}

  7%|▋         | 2773/41250 [6:41:35<93:00:54,  8.70s/it][2025-04-26 10:50:13,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:50:13,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.55 | bwd_microstep: 5722.01 | bwd_inner_microstep: 5709.22 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.37
[2025-04-26 10:50:13,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.55 | bwd: 5722.02 | bwd_inner: 5709.22 | bwd_allreduce: 12.76 | step: 18.37

  7%|▋         | 2774/41250 [6:41:44<92:52:31,  8.69s/it]
                                                         
{'loss': 0.2806, 'grad_norm': 2.8148915767669678, 'learning_rate': 3.985473024428511e-05, 'epoch': 0.67}

  7%|▋         | 2774/41250 [6:41:44<92:52:31,  8.69s/it][2025-04-26 10:50:22,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.33 | optimizer_step: 1.05
[2025-04-26 10:50:22,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.78 | bwd_microstep: 5710.04 | bwd_inner_microstep: 5695.67 | bwd_allreduce_microstep: 14.30 | step_microstep: 20.70
[2025-04-26 10:50:22,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.78 | bwd: 5710.06 | bwd_inner: 5695.67 | bwd_allreduce: 14.34 | step: 20.70

  7%|▋         | 2775/41250 [6:41:53<92:43:42,  8.68s/it]
                                                         
{'loss': 0.3197, 'grad_norm': 1.900328516960144, 'learning_rate': 3.9854541259076114e-05, 'epoch': 0.67}

  7%|▋         | 2775/41250 [6:41:53<92:43:42,  8.68s/it][2025-04-26 10:50:30,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 10:50:30,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.82 | bwd_microstep: 5731.36 | bwd_inner_microstep: 5718.62 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.91
[2025-04-26 10:50:30,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.82 | bwd: 5731.38 | bwd_inner: 5718.62 | bwd_allreduce: 12.71 | step: 18.91

  7%|▋         | 2776/41250 [6:42:01<92:42:25,  8.67s/it]
                                                         
{'loss': 0.1158, 'grad_norm': 1.0361626148223877, 'learning_rate': 3.985435215146778e-05, 'epoch': 0.67}

  7%|▋         | 2776/41250 [6:42:01<92:42:25,  8.67s/it][2025-04-26 10:50:39,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:50:39,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.92 | bwd_microstep: 5783.81 | bwd_inner_microstep: 5674.08 | bwd_allreduce_microstep: 109.68 | step_microstep: 18.05
[2025-04-26 10:50:39,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.92 | bwd: 5783.83 | bwd_inner: 5674.08 | bwd_allreduce: 109.70 | step: 18.05

  7%|▋         | 2777/41250 [6:42:10<92:46:31,  8.68s/it]
                                                         
{'loss': 0.1783, 'grad_norm': 1.5707828998565674, 'learning_rate': 3.985416292146126e-05, 'epoch': 0.67}

  7%|▋         | 2777/41250 [6:42:10<92:46:31,  8.68s/it][2025-04-26 10:50:48,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.89 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-26 10:50:48,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.23 | bwd_microstep: 5768.07 | bwd_inner_microstep: 5696.33 | bwd_allreduce_microstep: 71.70 | step_microstep: 19.35
[2025-04-26 10:50:48,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.23 | bwd: 5768.08 | bwd_inner: 5696.33 | bwd_allreduce: 71.72 | step: 19.35

  7%|▋         | 2778/41250 [6:42:19<92:51:21,  8.69s/it]
                                                         
{'loss': 0.1588, 'grad_norm': 2.22880220413208, 'learning_rate': 3.985397356905774e-05, 'epoch': 0.67}

  7%|▋         | 2778/41250 [6:42:19<92:51:21,  8.69s/it][2025-04-26 10:50:56,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.18 | optimizer_step: 0.96
[2025-04-26 10:50:56,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.12 | bwd_microstep: 5803.83 | bwd_inner_microstep: 5644.46 | bwd_allreduce_microstep: 159.32 | step_microstep: 19.04
[2025-04-26 10:50:56,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.12 | bwd: 5803.85 | bwd_inner: 5644.46 | bwd_allreduce: 159.34 | step: 19.04

  7%|▋         | 2779/41250 [6:42:28<92:55:52,  8.70s/it]
                                                         
{'loss': 0.1103, 'grad_norm': 1.8606634140014648, 'learning_rate': 3.9853784094258386e-05, 'epoch': 0.67}

  7%|▋         | 2779/41250 [6:42:28<92:55:52,  8.70s/it][2025-04-26 10:51:05,845] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 0.91
[2025-04-26 10:51:05,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 6024.15 | bwd_inner_microstep: 5700.44 | bwd_allreduce_microstep: 323.65 | step_microstep: 18.91
[2025-04-26 10:51:05,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 6024.16 | bwd_inner: 5700.44 | bwd_allreduce: 323.68 | step: 18.92

  7%|▋         | 2780/41250 [6:42:37<93:47:03,  8.78s/it]
                                                         
{'loss': 0.2207, 'grad_norm': 2.447772979736328, 'learning_rate': 3.985359449706435e-05, 'epoch': 0.67}

  7%|▋         | 2780/41250 [6:42:37<93:47:03,  8.78s/it][2025-04-26 10:51:14,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:51:14,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5716.03 | bwd_inner_microstep: 5703.43 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.63
[2025-04-26 10:51:14,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5716.04 | bwd_inner: 5703.43 | bwd_allreduce: 12.57 | step: 18.63

  7%|▋         | 2781/41250 [6:42:45<93:22:44,  8.74s/it]
                                                         
{'loss': 0.4767, 'grad_norm': 3.749054193496704, 'learning_rate': 3.985340477747681e-05, 'epoch': 0.67}

  7%|▋         | 2781/41250 [6:42:45<93:22:44,  8.74s/it][2025-04-26 10:51:23,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.24 | optimizer_step: 1.15
[2025-04-26 10:51:23,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.93 | bwd_microstep: 5759.67 | bwd_inner_microstep: 5692.63 | bwd_allreduce_microstep: 66.99 | step_microstep: 19.80
[2025-04-26 10:51:23,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.93 | bwd: 5759.69 | bwd_inner: 5692.63 | bwd_allreduce: 67.01 | step: 19.80

  7%|▋         | 2782/41250 [6:42:54<93:12:38,  8.72s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 1.4418805837631226, 'learning_rate': 3.985321493549693e-05, 'epoch': 0.67}

  7%|▋         | 2782/41250 [6:42:54<93:12:38,  8.72s/it][2025-04-26 10:51:31,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:51:31,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.87 | bwd_microstep: 5789.60 | bwd_inner_microstep: 5776.65 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.98
[2025-04-26 10:51:31,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.87 | bwd: 5789.62 | bwd_inner: 5776.65 | bwd_allreduce: 12.92 | step: 18.99

  7%|▋         | 2783/41250 [6:43:03<93:18:25,  8.73s/it]
                                                         
{'loss': 0.1716, 'grad_norm': 1.8274204730987549, 'learning_rate': 3.985302497112589e-05, 'epoch': 0.67}

  7%|▋         | 2783/41250 [6:43:03<93:18:25,  8.73s/it][2025-04-26 10:51:40,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.15 | optimizer_gradients: 1.05 | optimizer_step: 1.10
[2025-04-26 10:51:40,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.83 | bwd_microstep: 5704.00 | bwd_inner_microstep: 5654.48 | bwd_allreduce_microstep: 49.48 | step_microstep: 20.10
[2025-04-26 10:51:40,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.83 | bwd: 5704.02 | bwd_inner: 5654.48 | bwd_allreduce: 49.49 | step: 20.10

  7%|▋         | 2784/41250 [6:43:11<92:56:50,  8.70s/it]
                                                         
{'loss': 0.1573, 'grad_norm': 1.5499305725097656, 'learning_rate': 3.985283488436485e-05, 'epoch': 0.67}

  7%|▋         | 2784/41250 [6:43:11<92:56:50,  8.70s/it][2025-04-26 10:51:49,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.02 | optimizer_step: 1.12
[2025-04-26 10:51:49,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.36 | bwd_microstep: 5739.98 | bwd_inner_microstep: 5707.02 | bwd_allreduce_microstep: 32.91 | step_microstep: 19.19
[2025-04-26 10:51:49,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.36 | bwd: 5739.99 | bwd_inner: 5707.02 | bwd_allreduce: 32.93 | step: 19.19

  7%|▋         | 2785/41250 [6:43:20<92:53:38,  8.69s/it]
                                                         
{'loss': 0.2641, 'grad_norm': 1.8292070627212524, 'learning_rate': 3.985264467521499e-05, 'epoch': 0.68}

  7%|▋         | 2785/41250 [6:43:20<92:53:38,  8.69s/it][2025-04-26 10:51:57,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.05
[2025-04-26 10:51:57,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.13 | bwd_microstep: 5714.29 | bwd_inner_microstep: 5701.33 | bwd_allreduce_microstep: 12.91 | step_microstep: 19.51
[2025-04-26 10:51:57,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.13 | bwd: 5714.31 | bwd_inner: 5701.33 | bwd_allreduce: 12.93 | step: 19.51

  7%|▋         | 2786/41250 [6:43:29<92:44:56,  8.68s/it]
                                                         
{'loss': 0.094, 'grad_norm': 1.227880597114563, 'learning_rate': 3.9852454343677485e-05, 'epoch': 0.68}

  7%|▋         | 2786/41250 [6:43:29<92:44:56,  8.68s/it][2025-04-26 10:52:06,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 10:52:06,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.50 | bwd_microstep: 5691.93 | bwd_inner_microstep: 5679.13 | bwd_allreduce_microstep: 12.75 | step_microstep: 19.09
[2025-04-26 10:52:06,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.50 | bwd: 5691.94 | bwd_inner: 5679.13 | bwd_allreduce: 12.77 | step: 19.10

  7%|▋         | 2787/41250 [6:43:37<92:33:39,  8.66s/it]
                                                         
{'loss': 0.0929, 'grad_norm': 0.8508692979812622, 'learning_rate': 3.98522638897535e-05, 'epoch': 0.68}

  7%|▋         | 2787/41250 [6:43:37<92:33:39,  8.66s/it][2025-04-26 10:52:15,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:52:15,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.84 | bwd_microstep: 5684.93 | bwd_inner_microstep: 5653.37 | bwd_allreduce_microstep: 31.52 | step_microstep: 18.66
[2025-04-26 10:52:15,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.84 | bwd: 5684.95 | bwd_inner: 5653.37 | bwd_allreduce: 31.54 | step: 18.66

  7%|▋         | 2788/41250 [6:43:46<92:18:30,  8.64s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 1.3700366020202637, 'learning_rate': 3.9852073313444214e-05, 'epoch': 0.68}

  7%|▋         | 2788/41250 [6:43:46<92:18:30,  8.64s/it][2025-04-26 10:52:23,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.62 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:52:23,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5693.12 | bwd_inner_microstep: 5680.22 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.91
[2025-04-26 10:52:23,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5693.14 | bwd_inner: 5680.22 | bwd_allreduce: 12.88 | step: 18.91

  7%|▋         | 2789/41250 [6:43:54<92:14:15,  8.63s/it]
                                                         
{'loss': 0.1392, 'grad_norm': 2.2186145782470703, 'learning_rate': 3.98518826147508e-05, 'epoch': 0.68}

  7%|▋         | 2789/41250 [6:43:54<92:14:15,  8.63s/it][2025-04-26 10:52:32,327] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 10:52:32,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.11 | bwd_microstep: 5693.56 | bwd_inner_microstep: 5641.39 | bwd_allreduce_microstep: 52.13 | step_microstep: 19.04
[2025-04-26 10:52:32,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.11 | bwd: 5693.58 | bwd_inner: 5641.39 | bwd_allreduce: 52.15 | step: 19.04

  7%|▋         | 2790/41250 [6:44:03<92:09:36,  8.63s/it]
                                                         
{'loss': 0.0817, 'grad_norm': 0.7175098657608032, 'learning_rate': 3.985169179367444e-05, 'epoch': 0.68}

  7%|▋         | 2790/41250 [6:44:03<92:09:36,  8.63s/it][2025-04-26 10:52:40,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.18 | optimizer_step: 0.92
[2025-04-26 10:52:40,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.09 | bwd_microstep: 5704.11 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 60.37 | step_microstep: 18.99
[2025-04-26 10:52:40,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.09 | bwd: 5704.12 | bwd_inner: 5643.69 | bwd_allreduce: 60.39 | step: 18.99

  7%|▋         | 2791/41250 [6:44:12<92:06:25,  8.62s/it]
                                                         
{'loss': 0.0262, 'grad_norm': 0.3603093922138214, 'learning_rate': 3.9851500850216296e-05, 'epoch': 0.68}

  7%|▋         | 2791/41250 [6:44:12<92:06:25,  8.62s/it][2025-04-26 10:52:49,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:52:49,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.21 | bwd_microstep: 5720.99 | bwd_inner_microstep: 5708.04 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.53
[2025-04-26 10:52:49,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.21 | bwd: 5721.00 | bwd_inner: 5708.04 | bwd_allreduce: 12.92 | step: 18.53

  7%|▋         | 2792/41250 [6:44:20<92:10:24,  8.63s/it]
                                                         
{'loss': 0.2697, 'grad_norm': 1.934485673904419, 'learning_rate': 3.985130978437756e-05, 'epoch': 0.68}

  7%|▋         | 2792/41250 [6:44:20<92:10:24,  8.63s/it][2025-04-26 10:52:58,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:52:58,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.59 | bwd_microstep: 5752.41 | bwd_inner_microstep: 5699.22 | bwd_allreduce_microstep: 53.15 | step_microstep: 18.54
[2025-04-26 10:52:58,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.59 | bwd: 5752.43 | bwd_inner: 5699.22 | bwd_allreduce: 53.17 | step: 18.54

  7%|▋         | 2793/41250 [6:44:29<92:20:08,  8.64s/it]
                                                         
{'loss': 0.1837, 'grad_norm': 1.650924563407898, 'learning_rate': 3.98511185961594e-05, 'epoch': 0.68}

  7%|▋         | 2793/41250 [6:44:29<92:20:08,  8.64s/it][2025-04-26 10:53:07,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:53:07,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.65 | bwd_microstep: 5899.95 | bwd_inner_microstep: 5639.11 | bwd_allreduce_microstep: 260.79 | step_microstep: 18.53
[2025-04-26 10:53:07,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.65 | bwd: 5899.97 | bwd_inner: 5639.11 | bwd_allreduce: 260.81 | step: 18.54

  7%|▋         | 2794/41250 [6:44:38<92:49:48,  8.69s/it]
                                                         
{'loss': 0.1749, 'grad_norm': 1.5687247514724731, 'learning_rate': 3.9850927285563e-05, 'epoch': 0.68}

  7%|▋         | 2794/41250 [6:44:38<92:49:48,  8.69s/it][2025-04-26 10:53:15,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 10:53:15,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.61 | bwd_microstep: 5769.79 | bwd_inner_microstep: 5645.78 | bwd_allreduce_microstep: 123.96 | step_microstep: 18.62
[2025-04-26 10:53:15,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.61 | bwd: 5769.81 | bwd_inner: 5645.78 | bwd_allreduce: 123.98 | step: 18.63

  7%|▋         | 2795/41250 [6:44:46<92:46:12,  8.68s/it]
                                                         
{'loss': 0.1506, 'grad_norm': 0.9923008680343628, 'learning_rate': 3.9850735852589534e-05, 'epoch': 0.68}

  7%|▋         | 2795/41250 [6:44:46<92:46:12,  8.68s/it][2025-04-26 10:53:24,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 10:53:24,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.38 | bwd_microstep: 5761.18 | bwd_inner_microstep: 5651.22 | bwd_allreduce_microstep: 109.91 | step_microstep: 19.08
[2025-04-26 10:53:24,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.38 | bwd: 5761.19 | bwd_inner: 5651.22 | bwd_allreduce: 109.93 | step: 19.08

  7%|▋         | 2796/41250 [6:44:55<92:43:00,  8.68s/it]
                                                         
{'loss': 0.0528, 'grad_norm': 0.8406686782836914, 'learning_rate': 3.9850544297240186e-05, 'epoch': 0.68}

  7%|▋         | 2796/41250 [6:44:55<92:43:00,  8.68s/it][2025-04-26 10:53:33,084] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 10:53:33,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.47 | bwd_microstep: 5780.66 | bwd_inner_microstep: 5650.23 | bwd_allreduce_microstep: 130.38 | step_microstep: 18.89
[2025-04-26 10:53:33,085] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.47 | bwd: 5780.67 | bwd_inner: 5650.23 | bwd_allreduce: 130.40 | step: 18.90

  7%|▋         | 2797/41250 [6:45:04<92:43:36,  8.68s/it]
                                                         
{'loss': 0.1846, 'grad_norm': 1.0834585428237915, 'learning_rate': 3.9850352619516146e-05, 'epoch': 0.68}

  7%|▋         | 2797/41250 [6:45:04<92:43:36,  8.68s/it][2025-04-26 10:53:41,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:53:41,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.39 | bwd_microstep: 5737.76 | bwd_inner_microstep: 5694.79 | bwd_allreduce_microstep: 42.93 | step_microstep: 18.54
[2025-04-26 10:53:41,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.39 | bwd: 5737.77 | bwd_inner: 5694.79 | bwd_allreduce: 42.95 | step: 18.54

  7%|▋         | 2798/41250 [6:45:12<92:39:50,  8.68s/it]
                                                         
{'loss': 0.0602, 'grad_norm': 0.6136064529418945, 'learning_rate': 3.985016081941858e-05, 'epoch': 0.68}

  7%|▋         | 2798/41250 [6:45:12<92:39:50,  8.68s/it][2025-04-26 10:53:50,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:53:50,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.94 | bwd_microstep: 5763.13 | bwd_inner_microstep: 5643.26 | bwd_allreduce_microstep: 119.82 | step_microstep: 18.72
[2025-04-26 10:53:50,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.94 | bwd: 5763.14 | bwd_inner: 5643.26 | bwd_allreduce: 119.84 | step: 18.72

  7%|▋         | 2799/41250 [6:45:21<92:38:25,  8.67s/it]
                                                         
{'loss': 0.1467, 'grad_norm': 1.362404227256775, 'learning_rate': 3.984996889694867e-05, 'epoch': 0.68}

  7%|▋         | 2799/41250 [6:45:21<92:38:25,  8.67s/it][2025-04-26 10:53:59,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:53:59,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.94 | bwd_microstep: 5704.37 | bwd_inner_microstep: 5691.67 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.24
[2025-04-26 10:53:59,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.94 | bwd: 5704.38 | bwd_inner: 5691.67 | bwd_allreduce: 12.67 | step: 18.24

  7%|▋         | 2800/41250 [6:45:30<92:30:24,  8.66s/it]
                                                         
{'loss': 0.1863, 'grad_norm': 1.468753695487976, 'learning_rate': 3.984977685210762e-05, 'epoch': 0.68}

  7%|▋         | 2800/41250 [6:45:30<92:30:24,  8.66s/it][2025-04-26 10:54:07,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:54:07,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.85 | bwd_microstep: 5790.93 | bwd_inner_microstep: 5642.74 | bwd_allreduce_microstep: 148.14 | step_microstep: 18.79
[2025-04-26 10:54:07,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.85 | bwd: 5790.94 | bwd_inner: 5642.74 | bwd_allreduce: 148.16 | step: 18.79

  7%|▋         | 2801/41250 [6:45:38<92:33:43,  8.67s/it]
                                                         
{'loss': 0.2875, 'grad_norm': 2.842262029647827, 'learning_rate': 3.9849584684896586e-05, 'epoch': 0.68}

  7%|▋         | 2801/41250 [6:45:38<92:33:43,  8.67s/it][2025-04-26 10:54:16,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 10:54:16,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.74 | bwd_microstep: 5801.02 | bwd_inner_microstep: 5659.50 | bwd_allreduce_microstep: 141.47 | step_microstep: 18.71
[2025-04-26 10:54:16,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.74 | bwd: 5801.04 | bwd_inner: 5659.50 | bwd_allreduce: 141.49 | step: 18.71

  7%|▋         | 2802/41250 [6:45:47<92:41:43,  8.68s/it]
                                                         
{'loss': 0.1526, 'grad_norm': 1.9498295783996582, 'learning_rate': 3.9849392395316785e-05, 'epoch': 0.68}

  7%|▋         | 2802/41250 [6:45:47<92:41:43,  8.68s/it][2025-04-26 10:54:25,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 10:54:25,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.40 | bwd_microstep: 5708.24 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 59.21 | step_microstep: 18.43
[2025-04-26 10:54:25,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.40 | bwd: 5708.25 | bwd_inner: 5648.98 | bwd_allreduce: 59.23 | step: 18.43

  7%|▋         | 2803/41250 [6:45:56<92:29:12,  8.66s/it]
                                                         
{'loss': 0.137, 'grad_norm': 1.7245631217956543, 'learning_rate': 3.9849199983369367e-05, 'epoch': 0.68}

  7%|▋         | 2803/41250 [6:45:56<92:29:12,  8.66s/it][2025-04-26 10:54:33,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 10:54:33,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.56 | bwd_microstep: 5741.02 | bwd_inner_microstep: 5709.54 | bwd_allreduce_microstep: 31.43 | step_microstep: 18.86
[2025-04-26 10:54:33,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.56 | bwd: 5741.03 | bwd_inner: 5709.54 | bwd_allreduce: 31.45 | step: 18.87

  7%|▋         | 2804/41250 [6:46:04<92:30:49,  8.66s/it]
                                                         
{'loss': 0.2024, 'grad_norm': 1.449543833732605, 'learning_rate': 3.9849007449055544e-05, 'epoch': 0.68}

  7%|▋         | 2804/41250 [6:46:04<92:30:49,  8.66s/it][2025-04-26 10:54:42,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:54:42,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.49 | bwd_microstep: 5707.16 | bwd_inner_microstep: 5637.39 | bwd_allreduce_microstep: 69.73 | step_microstep: 18.54
[2025-04-26 10:54:42,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.49 | bwd: 5707.17 | bwd_inner: 5637.39 | bwd_allreduce: 69.74 | step: 18.55

  7%|▋         | 2805/41250 [6:46:13<92:20:18,  8.65s/it]
                                                         
{'loss': 0.06, 'grad_norm': 0.6859650015830994, 'learning_rate': 3.984881479237649e-05, 'epoch': 0.68}

  7%|▋         | 2805/41250 [6:46:13<92:20:18,  8.65s/it][2025-04-26 10:54:50,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:54:50,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.27 | bwd_microstep: 5718.28 | bwd_inner_microstep: 5650.96 | bwd_allreduce_microstep: 67.28 | step_microstep: 18.42
[2025-04-26 10:54:50,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.27 | bwd: 5718.29 | bwd_inner: 5650.96 | bwd_allreduce: 67.29 | step: 18.42

  7%|▋         | 2806/41250 [6:46:22<92:14:46,  8.64s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.1811105012893677, 'learning_rate': 3.984862201333339e-05, 'epoch': 0.68}

  7%|▋         | 2806/41250 [6:46:22<92:14:46,  8.64s/it][2025-04-26 10:54:59,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 10:54:59,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.85 | bwd_microstep: 5783.49 | bwd_inner_microstep: 5706.41 | bwd_allreduce_microstep: 77.04 | step_microstep: 18.37
[2025-04-26 10:54:59,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.85 | bwd: 5783.50 | bwd_inner: 5706.41 | bwd_allreduce: 77.05 | step: 18.37

  7%|▋         | 2807/41250 [6:46:30<92:28:35,  8.66s/it]
                                                         
{'loss': 0.2591, 'grad_norm': 2.898977279663086, 'learning_rate': 3.984842911192745e-05, 'epoch': 0.68}

  7%|▋         | 2807/41250 [6:46:30<92:28:35,  8.66s/it][2025-04-26 10:55:08,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:55:08,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.09 | bwd_microstep: 5785.94 | bwd_inner_microstep: 5773.09 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.90
[2025-04-26 10:55:08,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.09 | bwd: 5785.95 | bwd_inner: 5773.09 | bwd_allreduce: 12.82 | step: 18.90

  7%|▋         | 2808/41250 [6:46:39<92:47:30,  8.69s/it]
                                                         
{'loss': 0.106, 'grad_norm': 1.0969011783599854, 'learning_rate': 3.984823608815985e-05, 'epoch': 0.68}

  7%|▋         | 2808/41250 [6:46:39<92:47:30,  8.69s/it][2025-04-26 10:55:17,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 0.91
[2025-04-26 10:55:17,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.15 | bwd_microstep: 5774.96 | bwd_inner_microstep: 5698.90 | bwd_allreduce_microstep: 75.99 | step_microstep: 18.98
[2025-04-26 10:55:17,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.15 | bwd: 5774.98 | bwd_inner: 5698.90 | bwd_allreduce: 76.02 | step: 18.98

  7%|▋         | 2809/41250 [6:46:48<92:50:05,  8.69s/it]
                                                         
{'loss': 0.1532, 'grad_norm': 1.7664053440093994, 'learning_rate': 3.9848042942031775e-05, 'epoch': 0.68}

  7%|▋         | 2809/41250 [6:46:48<92:50:05,  8.69s/it][2025-04-26 10:55:25,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.29 | optimizer_step: 1.04
[2025-04-26 10:55:25,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5783.30 | bwd_inner_microstep: 5655.13 | bwd_allreduce_microstep: 128.11 | step_microstep: 19.81
[2025-04-26 10:55:25,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5783.31 | bwd_inner: 5655.13 | bwd_allreduce: 128.14 | step: 19.81

  7%|▋         | 2810/41250 [6:46:56<92:48:20,  8.69s/it]
                                                         
{'loss': 0.1732, 'grad_norm': 1.1613188982009888, 'learning_rate': 3.9847849673544413e-05, 'epoch': 0.68}

  7%|▋         | 2810/41250 [6:46:56<92:48:20,  8.69s/it][2025-04-26 10:55:34,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 10:55:34,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.83 | bwd_microstep: 5768.96 | bwd_inner_microstep: 5704.67 | bwd_allreduce_microstep: 64.23 | step_microstep: 18.82
[2025-04-26 10:55:34,507] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.83 | bwd: 5768.97 | bwd_inner: 5704.67 | bwd_allreduce: 64.25 | step: 18.82

  7%|▋         | 2811/41250 [6:47:05<92:49:33,  8.69s/it]
                                                         
{'loss': 0.3315, 'grad_norm': 1.8509730100631714, 'learning_rate': 3.9847656282698975e-05, 'epoch': 0.68}

  7%|▋         | 2811/41250 [6:47:05<92:49:33,  8.69s/it][2025-04-26 10:55:43,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 10:55:43,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5877.33 | bwd_inner_microstep: 5712.94 | bwd_allreduce_microstep: 164.32 | step_microstep: 19.11
[2025-04-26 10:55:43,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5877.34 | bwd_inner: 5712.94 | bwd_allreduce: 164.36 | step: 19.11

  7%|▋         | 2812/41250 [6:47:14<93:11:56,  8.73s/it]
                                                         
{'loss': 0.1826, 'grad_norm': 1.4787973165512085, 'learning_rate': 3.984746276949663e-05, 'epoch': 0.68}

  7%|▋         | 2812/41250 [6:47:14<93:11:56,  8.73s/it][2025-04-26 10:55:51,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 10:55:51,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.98 | bwd_microstep: 5719.07 | bwd_inner_microstep: 5706.22 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.90
[2025-04-26 10:55:51,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.98 | bwd: 5719.08 | bwd_inner: 5706.22 | bwd_allreduce: 12.82 | step: 18.90

  7%|▋         | 2813/41250 [6:47:23<92:58:08,  8.71s/it]
                                                         
{'loss': 0.2811, 'grad_norm': 2.420597553253174, 'learning_rate': 3.984726913393858e-05, 'epoch': 0.68}

  7%|▋         | 2813/41250 [6:47:23<92:58:08,  8.71s/it][2025-04-26 10:56:00,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 10:56:00,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.01 | bwd_microstep: 5778.95 | bwd_inner_microstep: 5766.08 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.10
[2025-04-26 10:56:00,720] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.01 | bwd: 5778.96 | bwd_inner: 5766.08 | bwd_allreduce: 12.84 | step: 19.10

  7%|▋         | 2814/41250 [6:47:31<93:05:11,  8.72s/it]
                                                         
{'loss': 0.125, 'grad_norm': 1.652016282081604, 'learning_rate': 3.984707537602602e-05, 'epoch': 0.68}

  7%|▋         | 2814/41250 [6:47:31<93:05:11,  8.72s/it][2025-04-26 10:56:09,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 10:56:09,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.65 | bwd_microstep: 5712.33 | bwd_inner_microstep: 5691.82 | bwd_allreduce_microstep: 20.47 | step_microstep: 18.99
[2025-04-26 10:56:09,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.65 | bwd: 5712.35 | bwd_inner: 5691.82 | bwd_allreduce: 20.49 | step: 18.99

  7%|▋         | 2815/41250 [6:47:40<92:49:42,  8.69s/it]
                                                         
{'loss': 0.0993, 'grad_norm': 0.9698881506919861, 'learning_rate': 3.9846881495760145e-05, 'epoch': 0.68}

  7%|▋         | 2815/41250 [6:47:40<92:49:42,  8.69s/it][2025-04-26 10:56:18,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 0.89
[2025-04-26 10:56:18,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5777.99 | bwd_inner_microstep: 5712.33 | bwd_allreduce_microstep: 65.61 | step_microstep: 19.19
[2025-04-26 10:56:18,071] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5778.00 | bwd_inner: 5712.33 | bwd_allreduce: 65.63 | step: 19.19

  7%|▋         | 2816/41250 [6:47:49<92:52:47,  8.70s/it]
                                                         
{'loss': 0.3244, 'grad_norm': 1.5772206783294678, 'learning_rate': 3.984668749314216e-05, 'epoch': 0.68}

  7%|▋         | 2816/41250 [6:47:49<92:52:47,  8.70s/it][2025-04-26 10:56:26,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 10:56:26,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.68 | bwd_microstep: 5750.28 | bwd_inner_microstep: 5705.67 | bwd_allreduce_microstep: 44.56 | step_microstep: 18.56
[2025-04-26 10:56:26,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.68 | bwd: 5750.29 | bwd_inner: 5705.67 | bwd_allreduce: 44.58 | step: 18.56

  7%|▋         | 2817/41250 [6:47:57<92:49:29,  8.69s/it]
                                                         
{'loss': 0.2762, 'grad_norm': 2.0620033740997314, 'learning_rate': 3.984649336817324e-05, 'epoch': 0.68}

  7%|▋         | 2817/41250 [6:47:57<92:49:29,  8.69s/it][2025-04-26 10:56:35,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 10:56:35,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.66 | bwd_microstep: 5735.14 | bwd_inner_microstep: 5690.60 | bwd_allreduce_microstep: 44.49 | step_microstep: 19.29
[2025-04-26 10:56:35,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.66 | bwd: 5735.15 | bwd_inner: 5690.60 | bwd_allreduce: 44.51 | step: 19.29

  7%|▋         | 2818/41250 [6:48:06<92:46:14,  8.69s/it]
                                                         
{'loss': 0.4971, 'grad_norm': 3.0401062965393066, 'learning_rate': 3.984629912085459e-05, 'epoch': 0.68}

  7%|▋         | 2818/41250 [6:48:06<92:46:14,  8.69s/it][2025-04-26 10:56:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:56:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5722.98 | bwd_inner_microstep: 5646.16 | bwd_allreduce_microstep: 76.78 | step_microstep: 18.63
[2025-04-26 10:56:44,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5723.00 | bwd_inner: 5646.16 | bwd_allreduce: 76.80 | step: 18.63

  7%|▋         | 2819/41250 [6:48:15<92:33:12,  8.67s/it]
                                                         
{'loss': 0.1596, 'grad_norm': 1.0263792276382446, 'learning_rate': 3.984610475118741e-05, 'epoch': 0.68}

  7%|▋         | 2819/41250 [6:48:15<92:33:12,  8.67s/it][2025-04-26 10:56:52,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:56:52,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5715.83 | bwd_inner_microstep: 5703.01 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.67
[2025-04-26 10:56:52,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5715.84 | bwd_inner: 5703.01 | bwd_allreduce: 12.79 | step: 18.67

  7%|▋         | 2820/41250 [6:48:23<92:28:10,  8.66s/it]
                                                         
{'loss': 0.15, 'grad_norm': 0.8488461375236511, 'learning_rate': 3.9845910259172896e-05, 'epoch': 0.68}

  7%|▋         | 2820/41250 [6:48:23<92:28:10,  8.66s/it][2025-04-26 10:57:01,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 10:57:01,321] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.68 | bwd_microstep: 5711.85 | bwd_inner_microstep: 5657.75 | bwd_allreduce_microstep: 54.05 | step_microstep: 18.77
[2025-04-26 10:57:01,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.69 | bwd: 5711.87 | bwd_inner: 5657.75 | bwd_allreduce: 54.07 | step: 18.77

  7%|▋         | 2821/41250 [6:48:32<92:20:16,  8.65s/it]
                                                         
{'loss': 0.1654, 'grad_norm': 2.268467664718628, 'learning_rate': 3.984571564481225e-05, 'epoch': 0.68}

  7%|▋         | 2821/41250 [6:48:32<92:20:16,  8.65s/it][2025-04-26 10:57:09,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 10:57:09,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.39 | bwd_microstep: 5752.99 | bwd_inner_microstep: 5646.26 | bwd_allreduce_microstep: 106.68 | step_microstep: 18.66
[2025-04-26 10:57:09,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.39 | bwd: 5753.00 | bwd_inner: 5646.26 | bwd_allreduce: 106.70 | step: 18.66

  7%|▋         | 2822/41250 [6:48:41<92:23:59,  8.66s/it]
                                                         
{'loss': 0.1407, 'grad_norm': 1.3080174922943115, 'learning_rate': 3.9845520908106684e-05, 'epoch': 0.68}

  7%|▋         | 2822/41250 [6:48:41<92:23:59,  8.66s/it][2025-04-26 10:57:18,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 10:57:18,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.94 | bwd_microstep: 5707.48 | bwd_inner_microstep: 5661.36 | bwd_allreduce_microstep: 46.07 | step_microstep: 19.25
[2025-04-26 10:57:18,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.94 | bwd: 5707.50 | bwd_inner: 5661.36 | bwd_allreduce: 46.09 | step: 19.25

  7%|▋         | 2823/41250 [6:48:49<92:16:40,  8.64s/it]
                                                         
{'loss': 0.2415, 'grad_norm': 2.047919750213623, 'learning_rate': 3.984532604905737e-05, 'epoch': 0.68}

  7%|▋         | 2823/41250 [6:48:49<92:16:40,  8.64s/it][2025-04-26 10:57:27,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 10:57:27,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5736.19 | bwd_inner_microstep: 5695.10 | bwd_allreduce_microstep: 41.04 | step_microstep: 18.97
[2025-04-26 10:57:27,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5736.21 | bwd_inner: 5695.10 | bwd_allreduce: 41.06 | step: 18.97

  7%|▋         | 2824/41250 [6:48:58<92:20:44,  8.65s/it]
                                                         
{'loss': 0.1932, 'grad_norm': 1.365904450416565, 'learning_rate': 3.984513106766553e-05, 'epoch': 0.68}

  7%|▋         | 2824/41250 [6:48:58<92:20:44,  8.65s/it][2025-04-26 10:57:36,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:57:36,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.95 | bwd_microstep: 5836.99 | bwd_inner_microstep: 5703.65 | bwd_allreduce_microstep: 133.30 | step_microstep: 18.56
[2025-04-26 10:57:36,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.95 | bwd: 5837.01 | bwd_inner: 5703.65 | bwd_allreduce: 133.32 | step: 18.56

  7%|▋         | 2825/41250 [6:49:07<92:43:05,  8.69s/it]
                                                         
{'loss': 0.256, 'grad_norm': 2.8539037704467773, 'learning_rate': 3.9844935963932355e-05, 'epoch': 0.68}

  7%|▋         | 2825/41250 [6:49:07<92:43:05,  8.69s/it][2025-04-26 10:57:44,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.64 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 10:57:44,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5715.28 | bwd_inner_microstep: 5702.31 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.39
[2025-04-26 10:57:44,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5715.29 | bwd_inner: 5702.31 | bwd_allreduce: 12.94 | step: 18.39

  7%|▋         | 2826/41250 [6:49:15<92:34:51,  8.67s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 0.8119244575500488, 'learning_rate': 3.984474073785906e-05, 'epoch': 0.69}

  7%|▋         | 2826/41250 [6:49:15<92:34:51,  8.67s/it][2025-04-26 10:57:53,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:57:53,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.61 | bwd_microstep: 5716.29 | bwd_inner_microstep: 5703.52 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.29
[2025-04-26 10:57:53,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.61 | bwd: 5716.30 | bwd_inner: 5703.52 | bwd_allreduce: 12.75 | step: 18.29

  7%|▋         | 2827/41250 [6:49:24<92:31:17,  8.67s/it]
                                                         
{'loss': 0.4435, 'grad_norm': 3.018031358718872, 'learning_rate': 3.984454538944684e-05, 'epoch': 0.69}

  7%|▋         | 2827/41250 [6:49:24<92:31:17,  8.67s/it][2025-04-26 10:58:02,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:58:02,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.88 | bwd_microstep: 5752.03 | bwd_inner_microstep: 5688.05 | bwd_allreduce_microstep: 63.94 | step_microstep: 18.54
[2025-04-26 10:58:02,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.88 | bwd: 5752.04 | bwd_inner: 5688.05 | bwd_allreduce: 63.96 | step: 18.54

  7%|▋         | 2828/41250 [6:49:33<92:31:38,  8.67s/it]
                                                         
{'loss': 0.0554, 'grad_norm': 0.40057283639907837, 'learning_rate': 3.9844349918696896e-05, 'epoch': 0.69}

  7%|▋         | 2828/41250 [6:49:33<92:31:38,  8.67s/it][2025-04-26 10:58:10,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:58:10,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.85 | bwd_microstep: 5692.62 | bwd_inner_microstep: 5679.88 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.04
[2025-04-26 10:58:10,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.85 | bwd: 5692.63 | bwd_inner: 5679.88 | bwd_allreduce: 12.72 | step: 18.05

  7%|▋         | 2829/41250 [6:49:41<92:22:40,  8.66s/it]
                                                         
{'loss': 0.1502, 'grad_norm': 1.553689956665039, 'learning_rate': 3.9844154325610444e-05, 'epoch': 0.69}

  7%|▋         | 2829/41250 [6:49:41<92:22:40,  8.66s/it][2025-04-26 10:58:19,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 10:58:19,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.73 | bwd_microstep: 5705.16 | bwd_inner_microstep: 5638.88 | bwd_allreduce_microstep: 66.23 | step_microstep: 19.01
[2025-04-26 10:58:19,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.73 | bwd: 5705.17 | bwd_inner: 5638.88 | bwd_allreduce: 66.25 | step: 19.01

  7%|▋         | 2830/41250 [6:49:50<92:14:48,  8.64s/it]
                                                         
{'loss': 0.22, 'grad_norm': 1.4741517305374146, 'learning_rate': 3.9843958610188676e-05, 'epoch': 0.69}

  7%|▋         | 2830/41250 [6:49:50<92:14:48,  8.64s/it][2025-04-26 10:58:27,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 10:58:27,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.46 | bwd_microstep: 5752.11 | bwd_inner_microstep: 5655.86 | bwd_allreduce_microstep: 96.21 | step_microstep: 18.40
[2025-04-26 10:58:27,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.46 | bwd: 5752.13 | bwd_inner: 5655.86 | bwd_allreduce: 96.22 | step: 18.40

  7%|▋         | 2831/41250 [6:49:59<92:19:47,  8.65s/it]
                                                         
{'loss': 0.2645, 'grad_norm': 1.4107146263122559, 'learning_rate': 3.984376277243281e-05, 'epoch': 0.69}

  7%|▋         | 2831/41250 [6:49:59<92:19:47,  8.65s/it][2025-04-26 10:58:36,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 10:58:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.04 | bwd_microstep: 5699.50 | bwd_inner_microstep: 5654.42 | bwd_allreduce_microstep: 45.04 | step_microstep: 18.22
[2025-04-26 10:58:36,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.04 | bwd: 5699.51 | bwd_inner: 5654.42 | bwd_allreduce: 45.06 | step: 18.22

  7%|▋         | 2832/41250 [6:50:07<92:11:38,  8.64s/it]
                                                         
{'loss': 0.1632, 'grad_norm': 1.003764271736145, 'learning_rate': 3.9843566812344054e-05, 'epoch': 0.69}

  7%|▋         | 2832/41250 [6:50:07<92:11:38,  8.64s/it][2025-04-26 10:58:45,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 10:58:45,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.40 | bwd_microstep: 5718.07 | bwd_inner_microstep: 5695.81 | bwd_allreduce_microstep: 22.22 | step_microstep: 18.22
[2025-04-26 10:58:45,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.40 | bwd: 5718.08 | bwd_inner: 5695.81 | bwd_allreduce: 22.23 | step: 18.22

  7%|▋         | 2833/41250 [6:50:16<92:13:49,  8.64s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.4084219932556152, 'learning_rate': 3.984337072992361e-05, 'epoch': 0.69}

  7%|▋         | 2833/41250 [6:50:16<92:13:49,  8.64s/it][2025-04-26 10:58:53,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 10:58:53,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.53 | bwd_microstep: 5712.82 | bwd_inner_microstep: 5694.86 | bwd_allreduce_microstep: 17.91 | step_microstep: 18.04
[2025-04-26 10:58:53,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.53 | bwd: 5712.83 | bwd_inner: 5694.86 | bwd_allreduce: 17.93 | step: 18.04

  7%|▋         | 2834/41250 [6:50:25<92:16:38,  8.65s/it]
                                                         
{'loss': 0.2563, 'grad_norm': 1.4373788833618164, 'learning_rate': 3.9843174525172686e-05, 'epoch': 0.69}

  7%|▋         | 2834/41250 [6:50:25<92:16:38,  8.65s/it][2025-04-26 10:59:02,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 10:59:02,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.40 | bwd_microstep: 5708.29 | bwd_inner_microstep: 5695.50 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.31
[2025-04-26 10:59:02,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.40 | bwd: 5708.30 | bwd_inner: 5695.50 | bwd_allreduce: 12.77 | step: 18.31

  7%|▋         | 2835/41250 [6:50:33<92:15:52,  8.65s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 1.5744078159332275, 'learning_rate': 3.9842978198092496e-05, 'epoch': 0.69}

  7%|▋         | 2835/41250 [6:50:33<92:15:52,  8.65s/it][2025-04-26 10:59:11,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 10:59:11,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.48 | bwd_microstep: 5745.23 | bwd_inner_microstep: 5647.05 | bwd_allreduce_microstep: 98.13 | step_microstep: 18.49
[2025-04-26 10:59:11,154] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.48 | bwd: 5745.24 | bwd_inner: 5647.05 | bwd_allreduce: 98.15 | step: 18.49

  7%|▋         | 2836/41250 [6:50:42<92:18:47,  8.65s/it]
                                                         
{'loss': 0.2527, 'grad_norm': 1.8583061695098877, 'learning_rate': 3.984278174868425e-05, 'epoch': 0.69}

  7%|▋         | 2836/41250 [6:50:42<92:18:47,  8.65s/it][2025-04-26 10:59:19,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 10:59:19,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.85 | bwd_microstep: 5688.97 | bwd_inner_microstep: 5635.69 | bwd_allreduce_microstep: 53.24 | step_microstep: 18.45
[2025-04-26 10:59:19,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.85 | bwd: 5688.98 | bwd_inner: 5635.69 | bwd_allreduce: 53.26 | step: 18.46

  7%|▋         | 2837/41250 [6:50:50<92:07:37,  8.63s/it]
                                                         
{'loss': 0.062, 'grad_norm': 0.5359669327735901, 'learning_rate': 3.9842585176949156e-05, 'epoch': 0.69}

  7%|▋         | 2837/41250 [6:50:50<92:07:37,  8.63s/it][2025-04-26 10:59:28,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 10:59:28,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5740.58 | bwd_inner_microstep: 5685.39 | bwd_allreduce_microstep: 55.14 | step_microstep: 18.81
[2025-04-26 10:59:28,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5740.59 | bwd_inner: 5685.39 | bwd_allreduce: 55.16 | step: 18.81

  7%|▋         | 2838/41250 [6:50:59<92:16:46,  8.65s/it]
                                                         
{'loss': 0.1921, 'grad_norm': 2.5220868587493896, 'learning_rate': 3.984238848288842e-05, 'epoch': 0.69}

  7%|▋         | 2838/41250 [6:50:59<92:16:46,  8.65s/it][2025-04-26 10:59:37,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 10:59:37,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3066.50 | bwd_microstep: 5656.12 | bwd_inner_microstep: 5643.28 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.20
[2025-04-26 10:59:37,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3066.50 | bwd: 5656.13 | bwd_inner: 5643.28 | bwd_allreduce: 12.81 | step: 18.20

  7%|▋         | 2839/41250 [6:51:08<92:47:45,  8.70s/it]
                                                         
{'loss': 0.2436, 'grad_norm': 2.4040191173553467, 'learning_rate': 3.984219166650327e-05, 'epoch': 0.69}

  7%|▋         | 2839/41250 [6:51:08<92:47:45,  8.70s/it][2025-04-26 10:59:45,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 10:59:45,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.12 | bwd_microstep: 5759.08 | bwd_inner_microstep: 5650.98 | bwd_allreduce_microstep: 108.05 | step_microstep: 18.46
[2025-04-26 10:59:45,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.12 | bwd: 5759.10 | bwd_inner: 5650.99 | bwd_allreduce: 108.07 | step: 18.46

  7%|▋         | 2840/41250 [6:51:17<92:41:32,  8.69s/it]
                                                         
{'loss': 0.1769, 'grad_norm': 1.0175833702087402, 'learning_rate': 3.9841994727794906e-05, 'epoch': 0.69}

  7%|▋         | 2840/41250 [6:51:17<92:41:32,  8.69s/it][2025-04-26 10:59:54,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 10:59:54,563] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.38 | bwd_microstep: 5738.46 | bwd_inner_microstep: 5649.31 | bwd_allreduce_microstep: 89.11 | step_microstep: 17.67
[2025-04-26 10:59:54,564] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.38 | bwd: 5738.47 | bwd_inner: 5649.31 | bwd_allreduce: 89.12 | step: 17.67

  7%|▋         | 2841/41250 [6:51:25<92:35:17,  8.68s/it]
                                                         
{'loss': 0.1262, 'grad_norm': 1.2267082929611206, 'learning_rate': 3.984179766676455e-05, 'epoch': 0.69}

  7%|▋         | 2841/41250 [6:51:25<92:35:17,  8.68s/it][2025-04-26 11:00:03,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:00:03,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.95 | bwd_microstep: 5744.81 | bwd_inner_microstep: 5683.10 | bwd_allreduce_microstep: 61.66 | step_microstep: 18.08
[2025-04-26 11:00:03,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.95 | bwd: 5744.82 | bwd_inner: 5683.10 | bwd_allreduce: 61.68 | step: 18.08

  7%|▋         | 2842/41250 [6:51:34<92:35:18,  8.68s/it]
                                                         
{'loss': 0.2709, 'grad_norm': 2.9688148498535156, 'learning_rate': 3.9841600483413414e-05, 'epoch': 0.69}

  7%|▋         | 2842/41250 [6:51:34<92:35:18,  8.68s/it][2025-04-26 11:00:11,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-26 11:00:11,925] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.37 | bwd_microstep: 5759.14 | bwd_inner_microstep: 5641.31 | bwd_allreduce_microstep: 117.78 | step_microstep: 18.69
[2025-04-26 11:00:11,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.37 | bwd: 5759.16 | bwd_inner: 5641.31 | bwd_allreduce: 117.81 | step: 18.69

  7%|▋         | 2843/41250 [6:51:43<92:36:16,  8.68s/it]
                                                         
{'loss': 0.1667, 'grad_norm': 1.2240608930587769, 'learning_rate': 3.984140317774271e-05, 'epoch': 0.69}

  7%|▋         | 2843/41250 [6:51:43<92:36:16,  8.68s/it][2025-04-26 11:00:20,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 11:00:20,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.30 | bwd_microstep: 5756.17 | bwd_inner_microstep: 5647.85 | bwd_allreduce_microstep: 108.27 | step_microstep: 18.31
[2025-04-26 11:00:20,588] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.30 | bwd: 5756.18 | bwd_inner: 5647.85 | bwd_allreduce: 108.29 | step: 18.31

  7%|▋         | 2844/41250 [6:51:51<92:32:38,  8.67s/it]
                                                         
{'loss': 0.1461, 'grad_norm': 1.0702548027038574, 'learning_rate': 3.984120574975366e-05, 'epoch': 0.69}

  7%|▋         | 2844/41250 [6:51:51<92:32:38,  8.67s/it][2025-04-26 11:00:29,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 11:00:29,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.86 | bwd_microstep: 5681.53 | bwd_inner_microstep: 5648.18 | bwd_allreduce_microstep: 33.30 | step_microstep: 18.44
[2025-04-26 11:00:29,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.86 | bwd: 5681.54 | bwd_inner: 5648.18 | bwd_allreduce: 33.32 | step: 18.45

  7%|▋         | 2845/41250 [6:52:00<92:18:23,  8.65s/it]
                                                         
{'loss': 0.2844, 'grad_norm': 1.413488507270813, 'learning_rate': 3.984100819944749e-05, 'epoch': 0.69}

  7%|▋         | 2845/41250 [6:52:00<92:18:23,  8.65s/it][2025-04-26 11:00:37,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:00:37,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.29 | bwd_microstep: 5695.97 | bwd_inner_microstep: 5683.22 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.01
[2025-04-26 11:00:37,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.29 | bwd: 5695.98 | bwd_inner: 5683.22 | bwd_allreduce: 12.73 | step: 18.01

  7%|▋         | 2846/41250 [6:52:08<92:10:35,  8.64s/it]
                                                         
{'loss': 0.127, 'grad_norm': 1.215167760848999, 'learning_rate': 3.9840810526825396e-05, 'epoch': 0.69}

  7%|▋         | 2846/41250 [6:52:08<92:10:35,  8.64s/it][2025-04-26 11:00:46,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:00:46,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.77 | bwd_microstep: 5774.96 | bwd_inner_microstep: 5641.84 | bwd_allreduce_microstep: 133.07 | step_microstep: 18.57
[2025-04-26 11:00:46,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.77 | bwd: 5774.97 | bwd_inner: 5641.84 | bwd_allreduce: 133.09 | step: 18.57

  7%|▋         | 2847/41250 [6:52:17<92:18:34,  8.65s/it]
                                                         
{'loss': 0.0927, 'grad_norm': 0.8084608912467957, 'learning_rate': 3.984061273188861e-05, 'epoch': 0.69}

  7%|▋         | 2847/41250 [6:52:17<92:18:34,  8.65s/it][2025-04-26 11:00:55,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 1.15
[2025-04-26 11:00:55,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.91 | bwd_microstep: 5732.84 | bwd_inner_microstep: 5689.89 | bwd_allreduce_microstep: 42.90 | step_microstep: 19.05
[2025-04-26 11:00:55,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.91 | bwd: 5732.85 | bwd_inner: 5689.89 | bwd_allreduce: 42.93 | step: 19.05

  7%|▋         | 2848/41250 [6:52:26<92:25:13,  8.66s/it]
                                                         
{'loss': 0.2621, 'grad_norm': 2.128051996231079, 'learning_rate': 3.9840414814638346e-05, 'epoch': 0.69}

  7%|▋         | 2848/41250 [6:52:26<92:25:13,  8.66s/it][2025-04-26 11:01:03,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:01:03,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.26 | bwd_microstep: 5780.79 | bwd_inner_microstep: 5689.46 | bwd_allreduce_microstep: 91.28 | step_microstep: 18.76
[2025-04-26 11:01:03,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.26 | bwd: 5780.80 | bwd_inner: 5689.46 | bwd_allreduce: 91.30 | step: 18.76

  7%|▋         | 2849/41250 [6:52:35<92:33:42,  8.68s/it]
                                                         
{'loss': 0.1643, 'grad_norm': 1.3093867301940918, 'learning_rate': 3.9840216775075834e-05, 'epoch': 0.69}

  7%|▋         | 2849/41250 [6:52:35<92:33:42,  8.68s/it][2025-04-26 11:01:12,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.97 | optimizer_step: 0.99
[2025-04-26 11:01:12,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5758.32 | bwd_inner_microstep: 5648.96 | bwd_allreduce_microstep: 109.32 | step_microstep: 18.10
[2025-04-26 11:01:12,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5758.33 | bwd_inner: 5648.96 | bwd_allreduce: 109.34 | step: 18.10

  7%|▋         | 2850/41250 [6:52:43<92:35:43,  8.68s/it]
                                                         
{'loss': 0.2198, 'grad_norm': 2.043654203414917, 'learning_rate': 3.984001861320228e-05, 'epoch': 0.69}

  7%|▋         | 2850/41250 [6:52:43<92:35:43,  8.68s/it][2025-04-26 11:01:21,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:01:21,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.87 | bwd_microstep: 5795.94 | bwd_inner_microstep: 5783.19 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.38
[2025-04-26 11:01:21,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.87 | bwd: 5795.96 | bwd_inner: 5783.19 | bwd_allreduce: 12.72 | step: 18.38

  7%|▋         | 2851/41250 [6:52:52<92:52:38,  8.71s/it]
                                                         
{'loss': 0.2783, 'grad_norm': 1.4271483421325684, 'learning_rate': 3.9839820329018916e-05, 'epoch': 0.69}

  7%|▋         | 2851/41250 [6:52:52<92:52:38,  8.71s/it][2025-04-26 11:01:30,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:01:30,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.65 | bwd_microstep: 5760.80 | bwd_inner_microstep: 5663.03 | bwd_allreduce_microstep: 97.73 | step_microstep: 18.27
[2025-04-26 11:01:30,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.65 | bwd: 5760.82 | bwd_inner: 5663.03 | bwd_allreduce: 97.74 | step: 18.28

  7%|▋         | 2852/41250 [6:53:01<92:47:18,  8.70s/it]
                                                         
{'loss': 0.3762, 'grad_norm': 1.7787659168243408, 'learning_rate': 3.9839621922526956e-05, 'epoch': 0.69}

  7%|▋         | 2852/41250 [6:53:01<92:47:18,  8.70s/it][2025-04-26 11:01:38,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 11:01:38,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.93 | bwd_microstep: 5701.04 | bwd_inner_microstep: 5660.44 | bwd_allreduce_microstep: 40.55 | step_microstep: 18.85
[2025-04-26 11:01:38,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.93 | bwd: 5701.05 | bwd_inner: 5660.44 | bwd_allreduce: 40.57 | step: 18.85

  7%|▋         | 2853/41250 [6:53:09<92:32:12,  8.68s/it]
                                                         
{'loss': 0.1259, 'grad_norm': 2.18780779838562, 'learning_rate': 3.983942339372764e-05, 'epoch': 0.69}

  7%|▋         | 2853/41250 [6:53:09<92:32:12,  8.68s/it][2025-04-26 11:01:47,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.31 | optimizer_step: 0.91
[2025-04-26 11:01:47,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.47 | bwd_microstep: 5749.54 | bwd_inner_microstep: 5703.29 | bwd_allreduce_microstep: 46.20 | step_microstep: 20.88
[2025-04-26 11:01:47,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.47 | bwd: 5749.56 | bwd_inner: 5703.29 | bwd_allreduce: 46.22 | step: 20.88

  7%|▋         | 2854/41250 [6:53:18<92:35:13,  8.68s/it]
                                                         
{'loss': 0.2421, 'grad_norm': 1.1894851922988892, 'learning_rate': 3.983922474262217e-05, 'epoch': 0.69}

  7%|▋         | 2854/41250 [6:53:18<92:35:13,  8.68s/it][2025-04-26 11:01:55,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 11:01:55,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.74 | bwd_microstep: 5723.19 | bwd_inner_microstep: 5710.36 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.14
[2025-04-26 11:01:55,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.74 | bwd: 5723.21 | bwd_inner: 5710.36 | bwd_allreduce: 12.81 | step: 19.14

  7%|▋         | 2855/41250 [6:53:27<92:30:34,  8.67s/it]
                                                         
{'loss': 0.3578, 'grad_norm': 2.549086093902588, 'learning_rate': 3.9839025969211796e-05, 'epoch': 0.69}

  7%|▋         | 2855/41250 [6:53:27<92:30:34,  8.67s/it][2025-04-26 11:02:04,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 11:02:04,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2900.37 | bwd_microstep: 5803.81 | bwd_inner_microstep: 5790.69 | bwd_allreduce_microstep: 13.07 | step_microstep: 18.61
[2025-04-26 11:02:04,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2900.38 | bwd: 5803.83 | bwd_inner: 5790.69 | bwd_allreduce: 13.09 | step: 18.62

  7%|▋         | 2856/41250 [6:53:35<92:53:48,  8.71s/it]
                                                         
{'loss': 0.2101, 'grad_norm': 1.7528998851776123, 'learning_rate': 3.9838827073497716e-05, 'epoch': 0.69}

  7%|▋         | 2856/41250 [6:53:35<92:53:48,  8.71s/it][2025-04-26 11:02:13,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 11:02:13,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.11 | bwd_microstep: 5780.12 | bwd_inner_microstep: 5692.38 | bwd_allreduce_microstep: 87.68 | step_microstep: 19.26
[2025-04-26 11:02:13,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.11 | bwd: 5780.13 | bwd_inner: 5692.38 | bwd_allreduce: 87.71 | step: 19.27

  7%|▋         | 2857/41250 [6:53:44<92:55:56,  8.71s/it]
                                                         
{'loss': 0.12, 'grad_norm': 1.0559968948364258, 'learning_rate': 3.983862805548118e-05, 'epoch': 0.69}

  7%|▋         | 2857/41250 [6:53:44<92:55:56,  8.71s/it][2025-04-26 11:02:22,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-26 11:02:22,192] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.12 | bwd_microstep: 5738.96 | bwd_inner_microstep: 5703.26 | bwd_allreduce_microstep: 35.65 | step_microstep: 19.52
[2025-04-26 11:02:22,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.12 | bwd: 5738.98 | bwd_inner: 5703.26 | bwd_allreduce: 35.67 | step: 19.52

  7%|▋         | 2858/41250 [6:53:53<92:50:13,  8.71s/it]
                                                         
{'loss': 0.1385, 'grad_norm': 3.6837987899780273, 'learning_rate': 3.9838428915163396e-05, 'epoch': 0.69}

  7%|▋         | 2858/41250 [6:53:53<92:50:13,  8.71s/it][2025-04-26 11:02:30,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.05 | optimizer_step: 1.02
[2025-04-26 11:02:30,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5787.10 | bwd_inner_microstep: 5666.45 | bwd_allreduce_microstep: 120.60 | step_microstep: 19.09
[2025-04-26 11:02:30,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5787.11 | bwd_inner: 5666.45 | bwd_allreduce: 120.62 | step: 19.10

  7%|▋         | 2859/41250 [6:54:02<92:51:08,  8.71s/it]
                                                         
{'loss': 0.1939, 'grad_norm': 1.388331651687622, 'learning_rate': 3.9838229652545604e-05, 'epoch': 0.69}

  7%|▋         | 2859/41250 [6:54:02<92:51:08,  8.71s/it][2025-04-26 11:02:39,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:02:39,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.36 | bwd_microstep: 5895.56 | bwd_inner_microstep: 5703.85 | bwd_allreduce_microstep: 191.66 | step_microstep: 18.84
[2025-04-26 11:02:39,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.36 | bwd: 5895.57 | bwd_inner: 5703.85 | bwd_allreduce: 191.68 | step: 18.84

  7%|▋         | 2860/41250 [6:54:10<93:14:07,  8.74s/it]
                                                         
{'loss': 0.2341, 'grad_norm': 1.8731634616851807, 'learning_rate': 3.983803026762903e-05, 'epoch': 0.69}

  7%|▋         | 2860/41250 [6:54:10<93:14:07,  8.74s/it][2025-04-26 11:02:48,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.96
[2025-04-26 11:02:48,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.99 | bwd_microstep: 5707.61 | bwd_inner_microstep: 5658.26 | bwd_allreduce_microstep: 49.31 | step_microstep: 18.71
[2025-04-26 11:02:48,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.99 | bwd: 5707.62 | bwd_inner: 5658.26 | bwd_allreduce: 49.33 | step: 18.71

  7%|▋         | 2861/41250 [6:54:19<92:49:44,  8.71s/it]
                                                         
{'loss': 0.2346, 'grad_norm': 1.246190071105957, 'learning_rate': 3.983783076041489e-05, 'epoch': 0.69}

  7%|▋         | 2861/41250 [6:54:19<92:49:44,  8.71s/it][2025-04-26 11:02:57,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:02:57,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.42 | bwd_microstep: 5752.14 | bwd_inner_microstep: 5647.88 | bwd_allreduce_microstep: 104.22 | step_microstep: 18.52
[2025-04-26 11:02:57,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.43 | bwd: 5752.15 | bwd_inner: 5647.88 | bwd_allreduce: 104.23 | step: 18.53

  7%|▋         | 2862/41250 [6:54:28<92:44:26,  8.70s/it]
                                                         
{'loss': 0.2278, 'grad_norm': 1.5733424425125122, 'learning_rate': 3.983763113090444e-05, 'epoch': 0.69}

  7%|▋         | 2862/41250 [6:54:28<92:44:26,  8.70s/it][2025-04-26 11:03:05,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:03:05,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.47 | bwd_microstep: 5707.50 | bwd_inner_microstep: 5694.26 | bwd_allreduce_microstep: 13.20 | step_microstep: 18.54
[2025-04-26 11:03:05,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.47 | bwd: 5707.52 | bwd_inner: 5694.26 | bwd_allreduce: 13.22 | step: 18.55

  7%|▋         | 2863/41250 [6:54:36<92:32:36,  8.68s/it]
                                                         
{'loss': 0.3188, 'grad_norm': 1.914207100868225, 'learning_rate': 3.983743137909889e-05, 'epoch': 0.69}

  7%|▋         | 2863/41250 [6:54:36<92:32:36,  8.68s/it][2025-04-26 11:03:14,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 11:03:14,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.71 | bwd_microstep: 5783.10 | bwd_inner_microstep: 5691.94 | bwd_allreduce_microstep: 91.10 | step_microstep: 18.72
[2025-04-26 11:03:14,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.72 | bwd: 5783.12 | bwd_inner: 5691.94 | bwd_allreduce: 91.13 | step: 18.72

  7%|▋         | 2864/41250 [6:54:45<92:37:36,  8.69s/it]
                                                         
{'loss': 0.1933, 'grad_norm': 1.4628385305404663, 'learning_rate': 3.9837231504999475e-05, 'epoch': 0.69}

  7%|▋         | 2864/41250 [6:54:45<92:37:36,  8.69s/it][2025-04-26 11:03:22,978] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.12
[2025-04-26 11:03:22,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.23 | bwd_microstep: 5686.73 | bwd_inner_microstep: 5654.19 | bwd_allreduce_microstep: 32.49 | step_microstep: 19.31
[2025-04-26 11:03:22,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.23 | bwd: 5686.75 | bwd_inner: 5654.19 | bwd_allreduce: 32.51 | step: 19.32

  7%|▋         | 2865/41250 [6:54:54<92:22:23,  8.66s/it]
                                                         
{'loss': 0.099, 'grad_norm': 0.7107410430908203, 'learning_rate': 3.983703150860743e-05, 'epoch': 0.69}

  7%|▋         | 2865/41250 [6:54:54<92:22:23,  8.66s/it][2025-04-26 11:03:31,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 11:03:31,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.52 | bwd_microstep: 5691.84 | bwd_inner_microstep: 5660.19 | bwd_allreduce_microstep: 31.60 | step_microstep: 18.85
[2025-04-26 11:03:31,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.52 | bwd: 5691.85 | bwd_inner: 5660.19 | bwd_allreduce: 31.62 | step: 18.85

  7%|▋         | 2866/41250 [6:55:02<92:15:04,  8.65s/it]
                                                         
{'loss': 0.2744, 'grad_norm': 1.3625168800354004, 'learning_rate': 3.983683138992398e-05, 'epoch': 0.69}

  7%|▋         | 2866/41250 [6:55:02<92:15:04,  8.65s/it][2025-04-26 11:03:40,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:03:40,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.77 | bwd_microstep: 5738.23 | bwd_inner_microstep: 5701.54 | bwd_allreduce_microstep: 36.65 | step_microstep: 18.57
[2025-04-26 11:03:40,292] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.77 | bwd: 5738.25 | bwd_inner: 5701.54 | bwd_allreduce: 36.66 | step: 18.58

  7%|▋         | 2867/41250 [6:55:11<92:21:55,  8.66s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 0.8447490334510803, 'learning_rate': 3.983663114895038e-05, 'epoch': 0.7}

  7%|▋         | 2867/41250 [6:55:11<92:21:55,  8.66s/it][2025-04-26 11:03:49,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 11:03:49,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.59 | bwd_microstep: 5763.61 | bwd_inner_microstep: 5750.44 | bwd_allreduce_microstep: 13.12 | step_microstep: 19.11
[2025-04-26 11:03:49,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.59 | bwd: 5763.62 | bwd_inner: 5750.44 | bwd_allreduce: 13.14 | step: 19.12

  7%|▋         | 2868/41250 [6:55:20<92:35:36,  8.68s/it]
                                                         
{'loss': 0.1307, 'grad_norm': 1.027063250541687, 'learning_rate': 3.9836430785687845e-05, 'epoch': 0.7}

  7%|▋         | 2868/41250 [6:55:20<92:35:36,  8.68s/it][2025-04-26 11:03:57,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 11:03:57,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.78 | bwd_microstep: 5703.58 | bwd_inner_microstep: 5655.20 | bwd_allreduce_microstep: 48.34 | step_microstep: 18.66
[2025-04-26 11:03:57,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.78 | bwd: 5703.59 | bwd_inner: 5655.20 | bwd_allreduce: 48.36 | step: 18.67

  7%|▋         | 2869/41250 [6:55:28<92:21:39,  8.66s/it]
                                                         
{'loss': 0.1384, 'grad_norm': 1.3430143594741821, 'learning_rate': 3.983623030013762e-05, 'epoch': 0.7}

  7%|▋         | 2869/41250 [6:55:28<92:21:39,  8.66s/it][2025-04-26 11:04:06,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 11:04:06,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.72 | bwd_microstep: 5757.94 | bwd_inner_microstep: 5744.93 | bwd_allreduce_microstep: 12.97 | step_microstep: 19.02
[2025-04-26 11:04:06,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.72 | bwd: 5757.96 | bwd_inner: 5744.93 | bwd_allreduce: 12.99 | step: 19.03

  7%|▋         | 2870/41250 [6:55:37<92:34:12,  8.68s/it]
                                                         
{'loss': 0.2268, 'grad_norm': 1.5598201751708984, 'learning_rate': 3.9836029692300935e-05, 'epoch': 0.7}

  7%|▋         | 2870/41250 [6:55:37<92:34:12,  8.68s/it][2025-04-26 11:04:15,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 11:04:15,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.32 | bwd_microstep: 5784.03 | bwd_inner_microstep: 5636.14 | bwd_allreduce_microstep: 147.84 | step_microstep: 18.92
[2025-04-26 11:04:15,067] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.32 | bwd: 5784.05 | bwd_inner: 5636.14 | bwd_allreduce: 147.86 | step: 18.92

  7%|▋         | 2871/41250 [6:55:46<92:36:52,  8.69s/it]
                                                         
{'loss': 0.224, 'grad_norm': 1.1070202589035034, 'learning_rate': 3.983582896217902e-05, 'epoch': 0.7}

  7%|▋         | 2871/41250 [6:55:46<92:36:52,  8.69s/it][2025-04-26 11:04:23,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.01
[2025-04-26 11:04:23,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.00 | bwd_microstep: 5699.82 | bwd_inner_microstep: 5641.95 | bwd_allreduce_microstep: 57.82 | step_microstep: 18.67
[2025-04-26 11:04:23,673] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.00 | bwd: 5699.83 | bwd_inner: 5641.95 | bwd_allreduce: 57.84 | step: 18.67

  7%|▋         | 2872/41250 [6:55:54<92:20:45,  8.66s/it]
                                                         
{'loss': 0.1513, 'grad_norm': 1.330493450164795, 'learning_rate': 3.983562810977313e-05, 'epoch': 0.7}

  7%|▋         | 2872/41250 [6:55:54<92:20:45,  8.66s/it][2025-04-26 11:04:32,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:04:32,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.10 | bwd_microstep: 5754.44 | bwd_inner_microstep: 5698.52 | bwd_allreduce_microstep: 55.88 | step_microstep: 18.77
[2025-04-26 11:04:32,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.10 | bwd: 5754.46 | bwd_inner: 5698.52 | bwd_allreduce: 55.90 | step: 18.78

  7%|▋         | 2873/41250 [6:56:03<92:22:06,  8.66s/it]
                                                         
{'loss': 0.0939, 'grad_norm': 0.7977924942970276, 'learning_rate': 3.983542713508449e-05, 'epoch': 0.7}

  7%|▋         | 2873/41250 [6:56:03<92:22:06,  8.66s/it][2025-04-26 11:04:41,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:04:41,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.20 | bwd_microstep: 5728.19 | bwd_inner_microstep: 5688.24 | bwd_allreduce_microstep: 39.90 | step_microstep: 19.21
[2025-04-26 11:04:41,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.20 | bwd: 5728.20 | bwd_inner: 5688.24 | bwd_allreduce: 39.92 | step: 19.21

  7%|▋         | 2874/41250 [6:56:12<92:21:55,  8.66s/it]
                                                         
{'loss': 0.0633, 'grad_norm': 0.49864229559898376, 'learning_rate': 3.983522603811435e-05, 'epoch': 0.7}

  7%|▋         | 2874/41250 [6:56:12<92:21:55,  8.66s/it][2025-04-26 11:04:49,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.30 | optimizer_step: 1.01
[2025-04-26 11:04:49,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.56 | bwd_microstep: 5695.95 | bwd_inner_microstep: 5654.73 | bwd_allreduce_microstep: 41.16 | step_microstep: 19.44
[2025-04-26 11:04:49,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.56 | bwd: 5695.96 | bwd_inner: 5654.73 | bwd_allreduce: 41.18 | step: 19.44

  7%|▋         | 2875/41250 [6:56:20<92:09:33,  8.65s/it]
                                                         
{'loss': 0.2651, 'grad_norm': 1.5783146619796753, 'learning_rate': 3.983502481886393e-05, 'epoch': 0.7}

  7%|▋         | 2875/41250 [6:56:20<92:09:33,  8.65s/it][2025-04-26 11:04:58,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.62 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:04:58,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.18 | bwd_microstep: 5692.03 | bwd_inner_microstep: 5635.37 | bwd_allreduce_microstep: 56.61 | step_microstep: 18.76
[2025-04-26 11:04:58,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.18 | bwd: 5692.04 | bwd_inner: 5635.37 | bwd_allreduce: 56.63 | step: 18.76

  7%|▋         | 2876/41250 [6:56:29<92:03:06,  8.64s/it]
                                                         
{'loss': 0.4341, 'grad_norm': 2.883523941040039, 'learning_rate': 3.983482347733449e-05, 'epoch': 0.7}

  7%|▋         | 2876/41250 [6:56:29<92:03:06,  8.64s/it][2025-04-26 11:05:06,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 11:05:06,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.68 | bwd_microstep: 5666.57 | bwd_inner_microstep: 5629.94 | bwd_allreduce_microstep: 36.58 | step_microstep: 18.72
[2025-04-26 11:05:06,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.68 | bwd: 5666.58 | bwd_inner: 5629.94 | bwd_allreduce: 36.60 | step: 18.73

  7%|▋         | 2877/41250 [6:56:37<91:53:48,  8.62s/it]
                                                         
{'loss': 0.0792, 'grad_norm': 1.1270238161087036, 'learning_rate': 3.983462201352727e-05, 'epoch': 0.7}

  7%|▋         | 2877/41250 [6:56:37<91:53:48,  8.62s/it][2025-04-26 11:05:15,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:05:15,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.14 | bwd_microstep: 5710.58 | bwd_inner_microstep: 5697.82 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.23
[2025-04-26 11:05:15,454] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.14 | bwd: 5710.59 | bwd_inner: 5697.82 | bwd_allreduce: 12.73 | step: 18.23

  7%|▋         | 2878/41250 [6:56:46<91:58:06,  8.63s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.2766666412353516, 'learning_rate': 3.98344204274435e-05, 'epoch': 0.7}

  7%|▋         | 2878/41250 [6:56:46<91:58:06,  8.63s/it][2025-04-26 11:05:24,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 11:05:24,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.81 | bwd_microstep: 5882.14 | bwd_inner_microstep: 5687.82 | bwd_allreduce_microstep: 194.27 | step_microstep: 18.51
[2025-04-26 11:05:24,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.81 | bwd: 5882.15 | bwd_inner: 5687.82 | bwd_allreduce: 194.29 | step: 18.52

  7%|▋         | 2879/41250 [6:56:55<92:35:51,  8.69s/it]
                                                         
{'loss': 0.0954, 'grad_norm': 1.5888904333114624, 'learning_rate': 3.9834218719084434e-05, 'epoch': 0.7}

  7%|▋         | 2879/41250 [6:56:55<92:35:51,  8.69s/it][2025-04-26 11:05:32,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.18 | optimizer_step: 0.90
[2025-04-26 11:05:32,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.03 | bwd_microstep: 5730.70 | bwd_inner_microstep: 5678.11 | bwd_allreduce_microstep: 52.54 | step_microstep: 20.33
[2025-04-26 11:05:32,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.03 | bwd: 5730.71 | bwd_inner: 5678.11 | bwd_allreduce: 52.56 | step: 20.33

  7%|▋         | 2880/41250 [6:57:04<92:32:17,  8.68s/it]
                                                         
{'loss': 0.1484, 'grad_norm': 0.8782972693443298, 'learning_rate': 3.98340168884513e-05, 'epoch': 0.7}

  7%|▋         | 2880/41250 [6:57:04<92:32:17,  8.68s/it][2025-04-26 11:05:41,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.20 | optimizer_step: 0.96
[2025-04-26 11:05:41,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.57 | bwd_microstep: 5685.85 | bwd_inner_microstep: 5642.84 | bwd_allreduce_microstep: 42.94 | step_microstep: 19.05
[2025-04-26 11:05:41,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.57 | bwd: 5685.87 | bwd_inner: 5642.84 | bwd_allreduce: 42.96 | step: 19.05

  7%|▋         | 2881/41250 [6:57:12<92:17:15,  8.66s/it]
                                                         
{'loss': 0.1345, 'grad_norm': 1.1859416961669922, 'learning_rate': 3.9833814935545366e-05, 'epoch': 0.7}

  7%|▋         | 2881/41250 [6:57:12<92:17:15,  8.66s/it][2025-04-26 11:05:50,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:05:50,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.36 | bwd_microstep: 5929.16 | bwd_inner_microstep: 5643.59 | bwd_allreduce_microstep: 285.52 | step_microstep: 18.42
[2025-04-26 11:05:50,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.36 | bwd: 5929.17 | bwd_inner: 5643.59 | bwd_allreduce: 285.54 | step: 18.42

  7%|▋         | 2882/41250 [6:57:21<92:50:10,  8.71s/it]
                                                         
{'loss': 0.1341, 'grad_norm': 1.9782037734985352, 'learning_rate': 3.983361286036786e-05, 'epoch': 0.7}

  7%|▋         | 2882/41250 [6:57:21<92:50:10,  8.71s/it][2025-04-26 11:05:58,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 11:05:58,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.45 | bwd_microstep: 5691.47 | bwd_inner_microstep: 5650.70 | bwd_allreduce_microstep: 40.73 | step_microstep: 18.35
[2025-04-26 11:05:58,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.45 | bwd: 5691.49 | bwd_inner: 5650.70 | bwd_allreduce: 40.75 | step: 18.36

  7%|▋         | 2883/41250 [6:57:30<92:30:40,  8.68s/it]
                                                         
{'loss': 0.3728, 'grad_norm': 1.9148577451705933, 'learning_rate': 3.983341066292003e-05, 'epoch': 0.7}

  7%|▋         | 2883/41250 [6:57:30<92:30:40,  8.68s/it][2025-04-26 11:06:07,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:06:07,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.68 | bwd_microstep: 5715.93 | bwd_inner_microstep: 5691.25 | bwd_allreduce_microstep: 24.64 | step_microstep: 18.23
[2025-04-26 11:06:07,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.68 | bwd: 5715.95 | bwd_inner: 5691.25 | bwd_allreduce: 24.66 | step: 18.24

  7%|▋         | 2884/41250 [6:57:38<92:25:02,  8.67s/it]
                                                         
{'loss': 0.1389, 'grad_norm': 0.8889760375022888, 'learning_rate': 3.9833208343203126e-05, 'epoch': 0.7}

  7%|▋         | 2884/41250 [6:57:38<92:25:02,  8.67s/it][2025-04-26 11:06:16,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 11:06:16,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.80 | bwd_microstep: 5694.28 | bwd_inner_microstep: 5680.03 | bwd_allreduce_microstep: 14.21 | step_microstep: 18.59
[2025-04-26 11:06:16,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.80 | bwd: 5694.30 | bwd_inner: 5680.03 | bwd_allreduce: 14.23 | step: 18.60

  7%|▋         | 2885/41250 [6:57:47<92:16:56,  8.66s/it]
                                                         
{'loss': 0.2454, 'grad_norm': 1.8311687707901, 'learning_rate': 3.9833005901218394e-05, 'epoch': 0.7}

  7%|▋         | 2885/41250 [6:57:47<92:16:56,  8.66s/it][2025-04-26 11:06:24,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:06:24,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.39 | bwd_microstep: 5738.04 | bwd_inner_microstep: 5655.98 | bwd_allreduce_microstep: 82.02 | step_microstep: 18.28
[2025-04-26 11:06:24,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.39 | bwd: 5738.05 | bwd_inner: 5655.98 | bwd_allreduce: 82.04 | step: 18.28

  7%|▋         | 2886/41250 [6:57:56<92:16:13,  8.66s/it]
                                                         
{'loss': 0.1184, 'grad_norm': 1.198926568031311, 'learning_rate': 3.983280333696708e-05, 'epoch': 0.7}

  7%|▋         | 2886/41250 [6:57:56<92:16:13,  8.66s/it][2025-04-26 11:06:33,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:06:33,635] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.56 | bwd_microstep: 5790.19 | bwd_inner_microstep: 5658.86 | bwd_allreduce_microstep: 131.28 | step_microstep: 18.90
[2025-04-26 11:06:33,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.56 | bwd: 5790.21 | bwd_inner: 5658.86 | bwd_allreduce: 131.30 | step: 18.90

  7%|▋         | 2887/41250 [6:58:04<92:24:30,  8.67s/it]
                                                         
{'loss': 0.1712, 'grad_norm': 2.5496621131896973, 'learning_rate': 3.983260065045043e-05, 'epoch': 0.7}

  7%|▋         | 2887/41250 [6:58:04<92:24:30,  8.67s/it][2025-04-26 11:06:42,310] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 11:06:42,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.71 | bwd_microstep: 5760.84 | bwd_inner_microstep: 5650.06 | bwd_allreduce_microstep: 110.73 | step_microstep: 18.73
[2025-04-26 11:06:42,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.71 | bwd: 5760.85 | bwd_inner: 5650.06 | bwd_allreduce: 110.75 | step: 18.73

  7%|▋         | 2888/41250 [6:58:13<92:25:40,  8.67s/it]
                                                         
{'loss': 0.0859, 'grad_norm': 1.9231815338134766, 'learning_rate': 3.9832397841669704e-05, 'epoch': 0.7}

  7%|▋         | 2888/41250 [6:58:13<92:25:40,  8.67s/it][2025-04-26 11:06:51,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:06:51,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5757.00 | bwd_inner_microstep: 5692.28 | bwd_allreduce_microstep: 64.67 | step_microstep: 18.52
[2025-04-26 11:06:51,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5757.01 | bwd_inner: 5692.28 | bwd_allreduce: 64.69 | step: 18.52

  7%|▋         | 2889/41250 [6:58:22<92:28:03,  8.68s/it]
                                                         
{'loss': 0.0842, 'grad_norm': 1.0206784009933472, 'learning_rate': 3.983219491062614e-05, 'epoch': 0.7}

  7%|▋         | 2889/41250 [6:58:22<92:28:03,  8.68s/it][2025-04-26 11:06:59,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 11:06:59,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.63 | bwd_microstep: 5707.62 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 58.59 | step_microstep: 19.18
[2025-04-26 11:06:59,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.63 | bwd: 5707.63 | bwd_inner: 5648.98 | bwd_allreduce: 58.61 | step: 19.18

  7%|▋         | 2890/41250 [6:58:30<92:16:34,  8.66s/it]
                                                         
{'loss': 0.1093, 'grad_norm': 3.5543289184570312, 'learning_rate': 3.9831991857321e-05, 'epoch': 0.7}

  7%|▋         | 2890/41250 [6:58:30<92:16:34,  8.66s/it][2025-04-26 11:07:08,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:07:08,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.68 | bwd_microstep: 5788.97 | bwd_inner_microstep: 5653.78 | bwd_allreduce_microstep: 135.15 | step_microstep: 18.40
[2025-04-26 11:07:08,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.67 | bwd: 5788.98 | bwd_inner: 5653.78 | bwd_allreduce: 135.17 | step: 18.41

  7%|▋         | 2891/41250 [6:58:39<92:23:26,  8.67s/it]
                                                         
{'loss': 0.184, 'grad_norm': 1.0874009132385254, 'learning_rate': 3.9831788681755536e-05, 'epoch': 0.7}

  7%|▋         | 2891/41250 [6:58:39<92:23:26,  8.67s/it][2025-04-26 11:07:16,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.71 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:07:16,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.27 | bwd_microstep: 5722.86 | bwd_inner_microstep: 5657.39 | bwd_allreduce_microstep: 65.43 | step_microstep: 18.55
[2025-04-26 11:07:16,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.27 | bwd: 5722.87 | bwd_inner: 5657.39 | bwd_allreduce: 65.45 | step: 18.56

  7%|▋         | 2892/41250 [6:58:48<92:17:27,  8.66s/it]
                                                         
{'loss': 0.1557, 'grad_norm': 1.4827980995178223, 'learning_rate': 3.983158538393099e-05, 'epoch': 0.7}

  7%|▋         | 2892/41250 [6:58:48<92:17:27,  8.66s/it][2025-04-26 11:07:25,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:07:25,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.35 | bwd_microstep: 5797.55 | bwd_inner_microstep: 5651.25 | bwd_allreduce_microstep: 146.25 | step_microstep: 18.37
[2025-04-26 11:07:25,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.35 | bwd: 5797.56 | bwd_inner: 5651.25 | bwd_allreduce: 146.27 | step: 18.37

  7%|▋         | 2893/41250 [6:58:56<92:27:03,  8.68s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 1.2798166275024414, 'learning_rate': 3.983138196384862e-05, 'epoch': 0.7}

  7%|▋         | 2893/41250 [6:58:56<92:27:03,  8.68s/it][2025-04-26 11:07:34,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:07:34,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.62 | bwd_microstep: 5726.03 | bwd_inner_microstep: 5651.22 | bwd_allreduce_microstep: 74.77 | step_microstep: 18.46
[2025-04-26 11:07:34,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.62 | bwd: 5726.05 | bwd_inner: 5651.22 | bwd_allreduce: 74.79 | step: 18.46

  7%|▋         | 2894/41250 [6:59:05<92:19:10,  8.66s/it]
                                                         
{'loss': 0.193, 'grad_norm': 2.8696072101593018, 'learning_rate': 3.983117842150968e-05, 'epoch': 0.7}

  7%|▋         | 2894/41250 [6:59:05<92:19:10,  8.66s/it][2025-04-26 11:07:42,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 11:07:42,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.33 | bwd_microstep: 5757.45 | bwd_inner_microstep: 5697.89 | bwd_allreduce_microstep: 59.51 | step_microstep: 19.05
[2025-04-26 11:07:42,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.33 | bwd: 5757.47 | bwd_inner: 5697.89 | bwd_allreduce: 59.53 | step: 19.06

  7%|▋         | 2895/41250 [6:59:14<92:24:08,  8.67s/it]
                                                         
{'loss': 0.2046, 'grad_norm': 2.7614967823028564, 'learning_rate': 3.9830974756915426e-05, 'epoch': 0.7}

  7%|▋         | 2895/41250 [6:59:14<92:24:08,  8.67s/it][2025-04-26 11:07:51,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:07:51,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5714.31 | bwd_inner_microstep: 5689.73 | bwd_allreduce_microstep: 24.54 | step_microstep: 18.45
[2025-04-26 11:07:51,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5714.33 | bwd_inner: 5689.73 | bwd_allreduce: 24.55 | step: 18.46

  7%|▋         | 2896/41250 [6:59:22<92:17:08,  8.66s/it]
                                                         
{'loss': 0.2245, 'grad_norm': 1.4938145875930786, 'learning_rate': 3.9830770970067115e-05, 'epoch': 0.7}

  7%|▋         | 2896/41250 [6:59:22<92:17:08,  8.66s/it][2025-04-26 11:08:00,360] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:08:00,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.37 | bwd_microstep: 5813.78 | bwd_inner_microstep: 5659.83 | bwd_allreduce_microstep: 153.91 | step_microstep: 18.76
[2025-04-26 11:08:00,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.37 | bwd: 5813.79 | bwd_inner: 5659.83 | bwd_allreduce: 153.92 | step: 18.76

  7%|▋         | 2897/41250 [6:59:31<92:30:17,  8.68s/it]
                                                         
{'loss': 0.1275, 'grad_norm': 1.681527018547058, 'learning_rate': 3.9830567060966004e-05, 'epoch': 0.7}

  7%|▋         | 2897/41250 [6:59:31<92:30:17,  8.68s/it][2025-04-26 11:08:09,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:08:09,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.25 | bwd_microstep: 5785.70 | bwd_inner_microstep: 5724.24 | bwd_allreduce_microstep: 61.41 | step_microstep: 18.64
[2025-04-26 11:08:09,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.25 | bwd: 5785.71 | bwd_inner: 5724.24 | bwd_allreduce: 61.42 | step: 18.64

  7%|▋         | 2898/41250 [6:59:40<92:40:01,  8.70s/it]
                                                         
{'loss': 0.1306, 'grad_norm': 1.0738062858581543, 'learning_rate': 3.983036302961334e-05, 'epoch': 0.7}

  7%|▋         | 2898/41250 [6:59:40<92:40:01,  8.70s/it][2025-04-26 11:08:17,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:08:17,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.62 | bwd_microstep: 5700.00 | bwd_inner_microstep: 5679.98 | bwd_allreduce_microstep: 19.97 | step_microstep: 18.55
[2025-04-26 11:08:17,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.62 | bwd: 5700.01 | bwd_inner: 5679.98 | bwd_allreduce: 19.99 | step: 18.56

  7%|▋         | 2899/41250 [6:59:48<92:25:29,  8.68s/it]
                                                         
{'loss': 0.1604, 'grad_norm': 2.0117290019989014, 'learning_rate': 3.9830158876010394e-05, 'epoch': 0.7}

  7%|▋         | 2899/41250 [6:59:48<92:25:29,  8.68s/it][2025-04-26 11:08:26,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 11:08:26,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.88 | bwd_microstep: 5770.31 | bwd_inner_microstep: 5756.92 | bwd_allreduce_microstep: 13.34 | step_microstep: 18.71
[2025-04-26 11:08:26,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.88 | bwd: 5770.32 | bwd_inner: 5756.92 | bwd_allreduce: 13.36 | step: 18.72

  7%|▋         | 2900/41250 [6:59:57<92:33:55,  8.69s/it]
                                                         
{'loss': 0.2241, 'grad_norm': 3.837249517440796, 'learning_rate': 3.9829954600158416e-05, 'epoch': 0.7}

  7%|▋         | 2900/41250 [6:59:57<92:33:55,  8.69s/it][2025-04-26 11:08:35,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:08:35,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.87 | bwd_microstep: 5799.09 | bwd_inner_microstep: 5645.14 | bwd_allreduce_microstep: 153.91 | step_microstep: 18.34
[2025-04-26 11:08:35,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.87 | bwd: 5799.11 | bwd_inner: 5645.14 | bwd_allreduce: 153.93 | step: 18.34

  7%|▋         | 2901/41250 [7:00:06<92:39:27,  8.70s/it]
                                                         
{'loss': 0.2466, 'grad_norm': 2.28490948677063, 'learning_rate': 3.982975020205867e-05, 'epoch': 0.7}

  7%|▋         | 2901/41250 [7:00:06<92:39:27,  8.70s/it][2025-04-26 11:08:43,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:08:43,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.80 | bwd_microstep: 5768.37 | bwd_inner_microstep: 5690.52 | bwd_allreduce_microstep: 77.81 | step_microstep: 18.30
[2025-04-26 11:08:43,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.80 | bwd: 5768.39 | bwd_inner: 5690.52 | bwd_allreduce: 77.82 | step: 18.30

  7%|▋         | 2902/41250 [7:00:15<92:42:09,  8.70s/it]
                                                         
{'loss': 0.167, 'grad_norm': 1.4810168743133545, 'learning_rate': 3.9829545681712416e-05, 'epoch': 0.7}

  7%|▋         | 2902/41250 [7:00:15<92:42:09,  8.70s/it][2025-04-26 11:08:52,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:08:52,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.51 | bwd_microstep: 5723.14 | bwd_inner_microstep: 5710.38 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.07
[2025-04-26 11:08:52,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.51 | bwd: 5723.15 | bwd_inner: 5710.38 | bwd_allreduce: 12.73 | step: 18.07

  7%|▋         | 2903/41250 [7:00:23<92:33:25,  8.69s/it]
                                                         
{'loss': 0.28, 'grad_norm': 1.8935924768447876, 'learning_rate': 3.982934103912091e-05, 'epoch': 0.7}

  7%|▋         | 2903/41250 [7:00:23<92:33:25,  8.69s/it][2025-04-26 11:09:01,240] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:09:01,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.11 | bwd_microstep: 5775.42 | bwd_inner_microstep: 5700.58 | bwd_allreduce_microstep: 74.80 | step_microstep: 18.56
[2025-04-26 11:09:01,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.11 | bwd: 5775.44 | bwd_inner: 5700.58 | bwd_allreduce: 74.82 | step: 18.56

  7%|▋         | 2904/41250 [7:00:32<92:36:39,  8.69s/it]
                                                         
{'loss': 0.2573, 'grad_norm': 1.7403663396835327, 'learning_rate': 3.9829136274285416e-05, 'epoch': 0.7}

  7%|▋         | 2904/41250 [7:00:32<92:36:39,  8.69s/it][2025-04-26 11:09:09,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:09:09,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5706.26 | bwd_inner_microstep: 5693.25 | bwd_allreduce_microstep: 12.96 | step_microstep: 17.89
[2025-04-26 11:09:09,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5706.28 | bwd_inner: 5693.25 | bwd_allreduce: 12.98 | step: 17.90

  7%|▋         | 2905/41250 [7:00:41<92:25:22,  8.68s/it]
                                                         
{'loss': 0.4769, 'grad_norm': 4.519765377044678, 'learning_rate': 3.9828931387207206e-05, 'epoch': 0.7}

  7%|▋         | 2905/41250 [7:00:41<92:25:22,  8.68s/it][2025-04-26 11:09:18,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.39 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:09:18,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.78 | bwd_microstep: 5785.90 | bwd_inner_microstep: 5645.29 | bwd_allreduce_microstep: 140.57 | step_microstep: 19.37
[2025-04-26 11:09:18,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.78 | bwd: 5785.92 | bwd_inner: 5645.29 | bwd_allreduce: 140.59 | step: 19.38

  7%|▋         | 2906/41250 [7:00:49<92:28:40,  8.68s/it]
                                                         
{'loss': 0.1826, 'grad_norm': 1.3763362169265747, 'learning_rate': 3.9828726377887525e-05, 'epoch': 0.7}

  7%|▋         | 2906/41250 [7:00:49<92:28:40,  8.68s/it][2025-04-26 11:09:27,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.97
[2025-04-26 11:09:27,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.62 | bwd_microstep: 5779.50 | bwd_inner_microstep: 5681.25 | bwd_allreduce_microstep: 98.20 | step_microstep: 18.21
[2025-04-26 11:09:27,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.62 | bwd: 5779.51 | bwd_inner: 5681.25 | bwd_allreduce: 98.22 | step: 18.22

  7%|▋         | 2907/41250 [7:00:58<92:32:51,  8.69s/it]
                                                         
{'loss': 0.1746, 'grad_norm': 2.1559929847717285, 'learning_rate': 3.982852124632764e-05, 'epoch': 0.7}

  7%|▋         | 2907/41250 [7:00:58<92:32:51,  8.69s/it][2025-04-26 11:09:35,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.79 | optimizer_step: 0.89
[2025-04-26 11:09:35,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.21 | bwd_microstep: 5704.21 | bwd_inner_microstep: 5658.74 | bwd_allreduce_microstep: 45.43 | step_microstep: 21.27
[2025-04-26 11:09:35,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.21 | bwd: 5704.22 | bwd_inner: 5658.74 | bwd_allreduce: 45.45 | step: 21.28

  7%|▋         | 2908/41250 [7:01:07<92:19:57,  8.67s/it]
                                                         
{'loss': 0.1091, 'grad_norm': 1.03054940700531, 'learning_rate': 3.9828315992528834e-05, 'epoch': 0.7}

  7%|▋         | 2908/41250 [7:01:07<92:19:57,  8.67s/it][2025-04-26 11:09:44,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:09:44,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.68 | bwd_microstep: 5791.56 | bwd_inner_microstep: 5645.58 | bwd_allreduce_microstep: 145.93 | step_microstep: 18.51
[2025-04-26 11:09:44,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.68 | bwd: 5791.57 | bwd_inner: 5645.59 | bwd_allreduce: 145.95 | step: 18.51

  7%|▋         | 2909/41250 [7:01:15<92:25:18,  8.68s/it]
                                                         
{'loss': 0.2414, 'grad_norm': 1.7888450622558594, 'learning_rate': 3.982811061649236e-05, 'epoch': 0.71}

  7%|▋         | 2909/41250 [7:01:15<92:25:18,  8.68s/it][2025-04-26 11:09:53,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 11:09:53,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5759.26 | bwd_inner_microstep: 5709.64 | bwd_allreduce_microstep: 49.58 | step_microstep: 18.74
[2025-04-26 11:09:53,291] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5759.27 | bwd_inner: 5709.64 | bwd_allreduce: 49.59 | step: 18.75

  7%|▋         | 2910/41250 [7:01:24<92:27:51,  8.68s/it]
                                                         
{'loss': 0.0936, 'grad_norm': 1.0321941375732422, 'learning_rate': 3.982790511821947e-05, 'epoch': 0.71}

  7%|▋         | 2910/41250 [7:01:24<92:27:51,  8.68s/it][2025-04-26 11:10:01,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:10:01,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.00 | bwd_microstep: 5713.78 | bwd_inner_microstep: 5650.40 | bwd_allreduce_microstep: 63.33 | step_microstep: 18.40
[2025-04-26 11:10:01,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.00 | bwd: 5713.79 | bwd_inner: 5650.40 | bwd_allreduce: 63.35 | step: 18.40

  7%|▋         | 2911/41250 [7:01:33<92:16:33,  8.66s/it]
                                                         
{'loss': 0.1436, 'grad_norm': 1.6617473363876343, 'learning_rate': 3.982769949771146e-05, 'epoch': 0.71}

  7%|▋         | 2911/41250 [7:01:33<92:16:33,  8.66s/it][2025-04-26 11:10:10,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:10:10,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.00 | bwd_microstep: 5713.34 | bwd_inner_microstep: 5700.63 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.31
[2025-04-26 11:10:10,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.00 | bwd: 5713.36 | bwd_inner: 5700.63 | bwd_allreduce: 12.69 | step: 18.31

  7%|▋         | 2912/41250 [7:01:41<92:14:15,  8.66s/it]
                                                         
{'loss': 0.3144, 'grad_norm': 2.0139482021331787, 'learning_rate': 3.982749375496957e-05, 'epoch': 0.71}

  7%|▋         | 2912/41250 [7:01:41<92:14:15,  8.66s/it][2025-04-26 11:10:19,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:10:19,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.35 | bwd_microstep: 5885.52 | bwd_inner_microstep: 5872.74 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.38
[2025-04-26 11:10:19,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.35 | bwd: 5885.53 | bwd_inner: 5872.74 | bwd_allreduce: 12.75 | step: 18.39

  7%|▋         | 2913/41250 [7:01:50<93:00:56,  8.73s/it]
                                                         
{'loss': 0.1768, 'grad_norm': 1.1457109451293945, 'learning_rate': 3.98272878899951e-05, 'epoch': 0.71}

  7%|▋         | 2913/41250 [7:01:50<93:00:56,  8.73s/it][2025-04-26 11:10:28,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:10:28,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.38 | bwd_microstep: 5727.15 | bwd_inner_microstep: 5694.23 | bwd_allreduce_microstep: 32.88 | step_microstep: 17.80
[2025-04-26 11:10:28,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.38 | bwd: 5727.16 | bwd_inner: 5694.23 | bwd_allreduce: 32.89 | step: 17.80

  7%|▋         | 2914/41250 [7:01:59<92:47:29,  8.71s/it]
                                                         
{'loss': 0.0437, 'grad_norm': 0.46808478236198425, 'learning_rate': 3.982708190278929e-05, 'epoch': 0.71}

  7%|▋         | 2914/41250 [7:01:59<92:47:29,  8.71s/it][2025-04-26 11:10:36,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:10:36,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.24 | bwd_microstep: 5759.87 | bwd_inner_microstep: 5691.47 | bwd_allreduce_microstep: 68.36 | step_microstep: 18.55
[2025-04-26 11:10:36,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.24 | bwd: 5759.89 | bwd_inner: 5691.47 | bwd_allreduce: 68.38 | step: 18.55

  7%|▋         | 2915/41250 [7:02:08<92:44:07,  8.71s/it]
                                                         
{'loss': 0.37, 'grad_norm': 2.425483226776123, 'learning_rate': 3.9826875793353415e-05, 'epoch': 0.71}

  7%|▋         | 2915/41250 [7:02:08<92:44:07,  8.71s/it][2025-04-26 11:10:45,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 11:10:45,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.13 | bwd_microstep: 5782.52 | bwd_inner_microstep: 5644.68 | bwd_allreduce_microstep: 137.79 | step_microstep: 18.43
[2025-04-26 11:10:45,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.13 | bwd: 5782.53 | bwd_inner: 5644.68 | bwd_allreduce: 137.81 | step: 18.44

  7%|▋         | 2916/41250 [7:02:16<92:41:49,  8.71s/it]
                                                         
{'loss': 0.0594, 'grad_norm': 0.4943738877773285, 'learning_rate': 3.982666956168875e-05, 'epoch': 0.71}

  7%|▋         | 2916/41250 [7:02:16<92:41:49,  8.71s/it][2025-04-26 11:10:54,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 1.00
[2025-04-26 11:10:54,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.25 | bwd_microstep: 5700.82 | bwd_inner_microstep: 5646.58 | bwd_allreduce_microstep: 54.20 | step_microstep: 18.71
[2025-04-26 11:10:54,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.25 | bwd: 5700.84 | bwd_inner: 5646.58 | bwd_allreduce: 54.22 | step: 18.72

  7%|▋         | 2917/41250 [7:02:25<92:22:13,  8.67s/it]
                                                         
{'loss': 0.2307, 'grad_norm': 1.4435595273971558, 'learning_rate': 3.982646320779657e-05, 'epoch': 0.71}

  7%|▋         | 2917/41250 [7:02:25<92:22:13,  8.67s/it][2025-04-26 11:11:02,818] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 11:11:02,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.67 | bwd_microstep: 5785.02 | bwd_inner_microstep: 5638.02 | bwd_allreduce_microstep: 146.95 | step_microstep: 18.88
[2025-04-26 11:11:02,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.67 | bwd: 5785.03 | bwd_inner: 5638.02 | bwd_allreduce: 146.97 | step: 18.88

  7%|▋         | 2918/41250 [7:02:34<92:23:38,  8.68s/it]
                                                         
{'loss': 0.1798, 'grad_norm': 1.404689908027649, 'learning_rate': 3.982625673167814e-05, 'epoch': 0.71}

  7%|▋         | 2918/41250 [7:02:34<92:23:38,  8.68s/it][2025-04-26 11:11:11,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:11:11,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.58 | bwd_microstep: 5681.76 | bwd_inner_microstep: 5642.67 | bwd_allreduce_microstep: 39.05 | step_microstep: 18.29
[2025-04-26 11:11:11,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.58 | bwd: 5681.77 | bwd_inner: 5642.67 | bwd_allreduce: 39.07 | step: 18.29

  7%|▋         | 2919/41250 [7:02:42<92:07:48,  8.65s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.4731429815292358, 'learning_rate': 3.982605013333475e-05, 'epoch': 0.71}

  7%|▋         | 2919/41250 [7:02:42<92:07:48,  8.65s/it][2025-04-26 11:11:19,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 11:11:19,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.10 | bwd_microstep: 5671.45 | bwd_inner_microstep: 5649.35 | bwd_allreduce_microstep: 22.05 | step_microstep: 18.33
[2025-04-26 11:11:19,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.10 | bwd: 5671.46 | bwd_inner: 5649.35 | bwd_allreduce: 22.07 | step: 18.33

  7%|▋         | 2920/41250 [7:02:51<91:53:49,  8.63s/it]
                                                         
{'loss': 0.0814, 'grad_norm': 0.879435658454895, 'learning_rate': 3.9825843412767644e-05, 'epoch': 0.71}

  7%|▋         | 2920/41250 [7:02:51<91:53:49,  8.63s/it][2025-04-26 11:11:28,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:11:28,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.97 | bwd_microstep: 5753.22 | bwd_inner_microstep: 5653.89 | bwd_allreduce_microstep: 99.28 | step_microstep: 18.82
[2025-04-26 11:11:28,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.97 | bwd: 5753.23 | bwd_inner: 5653.89 | bwd_allreduce: 99.30 | step: 18.82

  7%|▋         | 2921/41250 [7:02:59<92:02:12,  8.64s/it]
                                                         
{'loss': 0.3174, 'grad_norm': 1.8986032009124756, 'learning_rate': 3.982563656997812e-05, 'epoch': 0.71}

  7%|▋         | 2921/41250 [7:02:59<92:02:12,  8.64s/it][2025-04-26 11:11:37,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 11:11:37,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.64 | bwd_microstep: 5738.62 | bwd_inner_microstep: 5684.90 | bwd_allreduce_microstep: 53.68 | step_microstep: 18.79
[2025-04-26 11:11:37,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.64 | bwd: 5738.63 | bwd_inner: 5684.90 | bwd_allreduce: 53.70 | step: 18.79

  7%|▋         | 2922/41250 [7:03:08<92:07:43,  8.65s/it]
                                                         
{'loss': 0.3974, 'grad_norm': 4.346252918243408, 'learning_rate': 3.982542960496745e-05, 'epoch': 0.71}

  7%|▋         | 2922/41250 [7:03:08<92:07:43,  8.65s/it][2025-04-26 11:11:45,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:11:45,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.13 | bwd_microstep: 5684.39 | bwd_inner_microstep: 5644.76 | bwd_allreduce_microstep: 39.59 | step_microstep: 18.36
[2025-04-26 11:11:45,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.13 | bwd: 5684.40 | bwd_inner: 5644.76 | bwd_allreduce: 39.60 | step: 18.36

  7%|▋         | 2923/41250 [7:03:17<91:54:43,  8.63s/it]
                                                         
{'loss': 0.0615, 'grad_norm': 0.632411539554596, 'learning_rate': 3.98252225177369e-05, 'epoch': 0.71}

  7%|▋         | 2923/41250 [7:03:17<91:54:43,  8.63s/it][2025-04-26 11:11:54,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:11:54,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.62 | bwd_microstep: 5726.26 | bwd_inner_microstep: 5713.31 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.82
[2025-04-26 11:11:54,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.62 | bwd: 5726.28 | bwd_inner: 5713.31 | bwd_allreduce: 12.93 | step: 18.83

  7%|▋         | 2924/41250 [7:03:25<92:00:49,  8.64s/it]
                                                         
{'loss': 0.2304, 'grad_norm': 1.8098959922790527, 'learning_rate': 3.982501530828776e-05, 'epoch': 0.71}

  7%|▋         | 2924/41250 [7:03:25<92:00:49,  8.64s/it][2025-04-26 11:12:03,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:12:03,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.91 | bwd_microstep: 5694.66 | bwd_inner_microstep: 5681.56 | bwd_allreduce_microstep: 13.05 | step_microstep: 18.77
[2025-04-26 11:12:03,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.91 | bwd: 5694.68 | bwd_inner: 5681.56 | bwd_allreduce: 13.07 | step: 18.78

  7%|▋         | 2925/41250 [7:03:34<91:58:48,  8.64s/it]
                                                         
{'loss': 0.1597, 'grad_norm': 1.3986613750457764, 'learning_rate': 3.982480797662129e-05, 'epoch': 0.71}

  7%|▋         | 2925/41250 [7:03:34<91:58:48,  8.64s/it][2025-04-26 11:12:11,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 11:12:11,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.22 | bwd_microstep: 5755.50 | bwd_inner_microstep: 5658.23 | bwd_allreduce_microstep: 97.22 | step_microstep: 19.10
[2025-04-26 11:12:11,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.22 | bwd: 5755.51 | bwd_inner: 5658.23 | bwd_allreduce: 97.25 | step: 19.11

  7%|▋         | 2926/41250 [7:03:43<92:07:07,  8.65s/it]
                                                         
{'loss': 0.2559, 'grad_norm': 1.6373296976089478, 'learning_rate': 3.9824600522738785e-05, 'epoch': 0.71}

  7%|▋         | 2926/41250 [7:03:43<92:07:07,  8.65s/it][2025-04-26 11:12:20,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 11:12:20,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.02 | bwd_microstep: 5718.76 | bwd_inner_microstep: 5706.00 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.63
[2025-04-26 11:12:20,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.02 | bwd: 5718.78 | bwd_inner: 5706.00 | bwd_allreduce: 12.74 | step: 18.63

  7%|▋         | 2927/41250 [7:03:51<92:05:14,  8.65s/it]
                                                         
{'loss': 0.3201, 'grad_norm': 2.3122191429138184, 'learning_rate': 3.982439294664152e-05, 'epoch': 0.71}

  7%|▋         | 2927/41250 [7:03:51<92:05:14,  8.65s/it][2025-04-26 11:12:29,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:12:29,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.29 | bwd_microstep: 5848.99 | bwd_inner_microstep: 5778.41 | bwd_allreduce_microstep: 70.54 | step_microstep: 18.44
[2025-04-26 11:12:29,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.29 | bwd: 5849.00 | bwd_inner: 5778.41 | bwd_allreduce: 70.55 | step: 18.44

  7%|▋         | 2928/41250 [7:04:00<92:37:59,  8.70s/it]
                                                         
{'loss': 0.1268, 'grad_norm': 1.679425835609436, 'learning_rate': 3.982418524833076e-05, 'epoch': 0.71}

  7%|▋         | 2928/41250 [7:04:00<92:37:59,  8.70s/it][2025-04-26 11:12:38,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:12:38,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.53 | bwd_microstep: 5767.74 | bwd_inner_microstep: 5690.98 | bwd_allreduce_microstep: 76.72 | step_microstep: 18.25
[2025-04-26 11:12:38,083] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.53 | bwd: 5767.75 | bwd_inner: 5690.97 | bwd_allreduce: 76.74 | step: 18.26

  7%|▋         | 2929/41250 [7:04:09<92:37:45,  8.70s/it]
                                                         
{'loss': 0.1404, 'grad_norm': 0.8953460454940796, 'learning_rate': 3.98239774278078e-05, 'epoch': 0.71}

  7%|▋         | 2929/41250 [7:04:09<92:37:45,  8.70s/it][2025-04-26 11:12:46,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.25 | optimizer_step: 0.89
[2025-04-26 11:12:46,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.16 | bwd_microstep: 5711.90 | bwd_inner_microstep: 5656.91 | bwd_allreduce_microstep: 54.94 | step_microstep: 19.06
[2025-04-26 11:12:46,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.16 | bwd: 5711.92 | bwd_inner: 5656.91 | bwd_allreduce: 54.96 | step: 19.07

  7%|▋         | 2930/41250 [7:04:17<92:22:53,  8.68s/it]
                                                         
{'loss': 0.2046, 'grad_norm': 1.3273277282714844, 'learning_rate': 3.982376948507393e-05, 'epoch': 0.71}

  7%|▋         | 2930/41250 [7:04:17<92:22:53,  8.68s/it][2025-04-26 11:12:55,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:12:55,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.30 | bwd_microstep: 5707.80 | bwd_inner_microstep: 5695.06 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.62
[2025-04-26 11:12:55,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.30 | bwd: 5707.81 | bwd_inner: 5695.06 | bwd_allreduce: 12.71 | step: 18.62

  7%|▋         | 2931/41250 [7:04:26<92:14:37,  8.67s/it]
                                                         
{'loss': 0.1965, 'grad_norm': 1.0387762784957886, 'learning_rate': 3.982356142013041e-05, 'epoch': 0.71}

  7%|▋         | 2931/41250 [7:04:26<92:14:37,  8.67s/it][2025-04-26 11:13:04,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 11:13:04,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.09 | bwd_microstep: 5783.03 | bwd_inner_microstep: 5770.21 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.21
[2025-04-26 11:13:04,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.09 | bwd: 5783.04 | bwd_inner: 5770.21 | bwd_allreduce: 12.79 | step: 19.21

  7%|▋         | 2932/41250 [7:04:35<92:29:54,  8.69s/it]
                                                         
{'loss': 0.09, 'grad_norm': 1.4853034019470215, 'learning_rate': 3.9823353232978533e-05, 'epoch': 0.71}

  7%|▋         | 2932/41250 [7:04:35<92:29:54,  8.69s/it][2025-04-26 11:13:12,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:13:12,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2934.36 | bwd_microstep: 5882.73 | bwd_inner_microstep: 5870.13 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.31
[2025-04-26 11:13:12,990] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2934.36 | bwd: 5882.74 | bwd_inner: 5870.13 | bwd_allreduce: 12.57 | step: 18.31

  7%|▋         | 2933/41250 [7:04:44<93:09:59,  8.75s/it]
                                                         
{'loss': 0.0781, 'grad_norm': 0.7636345624923706, 'learning_rate': 3.982314492361958e-05, 'epoch': 0.71}

  7%|▋         | 2933/41250 [7:04:44<93:09:59,  8.75s/it][2025-04-26 11:13:21,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:13:21,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.21 | bwd_microstep: 5753.28 | bwd_inner_microstep: 5688.09 | bwd_allreduce_microstep: 65.15 | step_microstep: 18.73
[2025-04-26 11:13:21,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.21 | bwd: 5753.29 | bwd_inner: 5688.09 | bwd_allreduce: 65.17 | step: 18.74

  7%|▋         | 2934/41250 [7:04:52<92:56:43,  8.73s/it]
                                                         
{'loss': 0.2861, 'grad_norm': 1.9837268590927124, 'learning_rate': 3.982293649205485e-05, 'epoch': 0.71}

  7%|▋         | 2934/41250 [7:04:52<92:56:43,  8.73s/it][2025-04-26 11:13:30,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:13:30,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.35 | bwd_microstep: 5792.24 | bwd_inner_microstep: 5660.42 | bwd_allreduce_microstep: 131.77 | step_microstep: 18.31
[2025-04-26 11:13:30,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.35 | bwd: 5792.25 | bwd_inner: 5660.42 | bwd_allreduce: 131.79 | step: 18.31

  7%|▋         | 2935/41250 [7:05:01<92:51:37,  8.72s/it]
                                                         
{'loss': 0.135, 'grad_norm': 1.3867872953414917, 'learning_rate': 3.9822727938285604e-05, 'epoch': 0.71}

  7%|▋         | 2935/41250 [7:05:01<92:51:37,  8.72s/it][2025-04-26 11:13:39,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:13:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.80 | bwd_microstep: 5711.99 | bwd_inner_microstep: 5699.29 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.84
[2025-04-26 11:13:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.80 | bwd: 5712.01 | bwd_inner: 5699.29 | bwd_allreduce: 12.66 | step: 18.84

  7%|▋         | 2936/41250 [7:05:10<92:35:44,  8.70s/it]
                                                         
{'loss': 0.07, 'grad_norm': 1.0784775018692017, 'learning_rate': 3.9822519262313146e-05, 'epoch': 0.71}

  7%|▋         | 2936/41250 [7:05:10<92:35:44,  8.70s/it][2025-04-26 11:13:47,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 11:13:47,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.72 | bwd_microstep: 5779.94 | bwd_inner_microstep: 5712.66 | bwd_allreduce_microstep: 67.23 | step_microstep: 19.26
[2025-04-26 11:13:47,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.72 | bwd: 5779.95 | bwd_inner: 5712.66 | bwd_allreduce: 67.25 | step: 19.27

  7%|▋         | 2937/41250 [7:05:18<92:38:16,  8.70s/it]
                                                         
{'loss': 0.1935, 'grad_norm': 2.1599605083465576, 'learning_rate': 3.982231046413875e-05, 'epoch': 0.71}

  7%|▋         | 2937/41250 [7:05:18<92:38:16,  8.70s/it][2025-04-26 11:13:56,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 11:13:56,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.48 | bwd_microstep: 5709.29 | bwd_inner_microstep: 5696.35 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.73
[2025-04-26 11:13:56,388] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.48 | bwd: 5709.30 | bwd_inner: 5696.35 | bwd_allreduce: 12.91 | step: 18.73

  7%|▋         | 2938/41250 [7:05:27<92:27:25,  8.69s/it]
                                                         
{'loss': 0.3971, 'grad_norm': 3.1662673950195312, 'learning_rate': 3.982210154376371e-05, 'epoch': 0.71}

  7%|▋         | 2938/41250 [7:05:27<92:27:25,  8.69s/it][2025-04-26 11:14:05,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:14:05,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.62 | bwd_microstep: 5770.34 | bwd_inner_microstep: 5706.26 | bwd_allreduce_microstep: 64.03 | step_microstep: 18.48
[2025-04-26 11:14:05,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.62 | bwd: 5770.35 | bwd_inner: 5706.26 | bwd_allreduce: 64.05 | step: 18.48

  7%|▋         | 2939/41250 [7:05:36<92:31:40,  8.69s/it]
                                                         
{'loss': 0.1948, 'grad_norm': 1.128915548324585, 'learning_rate': 3.982189250118932e-05, 'epoch': 0.71}

  7%|▋         | 2939/41250 [7:05:36<92:31:40,  8.69s/it][2025-04-26 11:14:13,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:14:13,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.53 | bwd_microstep: 5768.47 | bwd_inner_microstep: 5690.32 | bwd_allreduce_microstep: 78.11 | step_microstep: 18.53
[2025-04-26 11:14:13,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.53 | bwd: 5768.48 | bwd_inner: 5690.32 | bwd_allreduce: 78.12 | step: 18.54

  7%|▋         | 2940/41250 [7:05:45<92:36:47,  8.70s/it]
                                                         
{'loss': 0.1392, 'grad_norm': 1.3886032104492188, 'learning_rate': 3.982168333641685e-05, 'epoch': 0.71}

  7%|▋         | 2940/41250 [7:05:45<92:36:47,  8.70s/it][2025-04-26 11:14:22,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:14:22,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2955.82 | bwd_microstep: 5878.13 | bwd_inner_microstep: 5865.40 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.43
[2025-04-26 11:14:22,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2955.82 | bwd: 5878.14 | bwd_inner: 5865.40 | bwd_allreduce: 12.70 | step: 18.43

  7%|▋         | 2941/41250 [7:05:53<93:16:34,  8.77s/it]
                                                         
{'loss': 0.1633, 'grad_norm': 2.683838367462158, 'learning_rate': 3.982147404944761e-05, 'epoch': 0.71}

  7%|▋         | 2941/41250 [7:05:53<93:16:34,  8.77s/it][2025-04-26 11:14:31,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 11:14:31,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.56 | bwd_microstep: 5767.86 | bwd_inner_microstep: 5670.27 | bwd_allreduce_microstep: 97.55 | step_microstep: 18.54
[2025-04-26 11:14:31,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.56 | bwd: 5767.88 | bwd_inner: 5670.27 | bwd_allreduce: 97.57 | step: 18.54

  7%|▋         | 2942/41250 [7:06:02<93:00:16,  8.74s/it]
                                                         
{'loss': 0.1712, 'grad_norm': 1.1735851764678955, 'learning_rate': 3.982126464028288e-05, 'epoch': 0.71}

  7%|▋         | 2942/41250 [7:06:02<93:00:16,  8.74s/it][2025-04-26 11:14:40,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:14:40,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.41 | bwd_microstep: 5708.16 | bwd_inner_microstep: 5664.50 | bwd_allreduce_microstep: 43.62 | step_microstep: 18.11
[2025-04-26 11:14:40,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.41 | bwd: 5708.17 | bwd_inner: 5664.50 | bwd_allreduce: 43.63 | step: 18.12

  7%|▋         | 2943/41250 [7:06:11<92:40:32,  8.71s/it]
                                                         
{'loss': 0.3192, 'grad_norm': 1.7105921506881714, 'learning_rate': 3.9821055108923945e-05, 'epoch': 0.71}

  7%|▋         | 2943/41250 [7:06:11<92:40:32,  8.71s/it][2025-04-26 11:14:48,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 11:14:48,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.18 | bwd_microstep: 5725.58 | bwd_inner_microstep: 5711.58 | bwd_allreduce_microstep: 13.96 | step_microstep: 18.92
[2025-04-26 11:14:48,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.18 | bwd: 5725.60 | bwd_inner: 5711.58 | bwd_allreduce: 13.98 | step: 18.93

  7%|▋         | 2944/41250 [7:06:19<92:31:54,  8.70s/it]
                                                         
{'loss': 0.123, 'grad_norm': 1.4730076789855957, 'learning_rate': 3.982084545537211e-05, 'epoch': 0.71}

  7%|▋         | 2944/41250 [7:06:19<92:31:54,  8.70s/it][2025-04-26 11:14:57,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:14:57,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.83 | bwd_microstep: 5913.60 | bwd_inner_microstep: 5658.31 | bwd_allreduce_microstep: 255.24 | step_microstep: 18.50
[2025-04-26 11:14:57,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.83 | bwd: 5913.61 | bwd_inner: 5658.31 | bwd_allreduce: 255.26 | step: 18.51

  7%|▋         | 2945/41250 [7:06:28<92:56:02,  8.73s/it]
                                                         
{'loss': 0.1747, 'grad_norm': 2.4327685832977295, 'learning_rate': 3.982063567962866e-05, 'epoch': 0.71}

  7%|▋         | 2945/41250 [7:06:28<92:56:02,  8.73s/it][2025-04-26 11:15:06,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:15:06,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.21 | bwd_microstep: 5701.91 | bwd_inner_microstep: 5652.19 | bwd_allreduce_microstep: 49.68 | step_microstep: 18.33
[2025-04-26 11:15:06,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.21 | bwd: 5701.92 | bwd_inner: 5652.19 | bwd_allreduce: 49.69 | step: 18.34

  7%|▋         | 2946/41250 [7:06:37<92:35:31,  8.70s/it]
                                                         
{'loss': 0.2777, 'grad_norm': 1.7359577417373657, 'learning_rate': 3.982042578169489e-05, 'epoch': 0.71}

  7%|▋         | 2946/41250 [7:06:37<92:35:31,  8.70s/it][2025-04-26 11:15:14,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:15:14,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.53 | bwd_microstep: 5714.81 | bwd_inner_microstep: 5700.37 | bwd_allreduce_microstep: 14.39 | step_microstep: 18.47
[2025-04-26 11:15:14,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.53 | bwd: 5714.82 | bwd_inner: 5700.37 | bwd_allreduce: 14.41 | step: 18.47

  7%|▋         | 2947/41250 [7:06:46<92:26:18,  8.69s/it]
                                                         
{'loss': 0.2323, 'grad_norm': 2.2497475147247314, 'learning_rate': 3.982021576157209e-05, 'epoch': 0.71}

  7%|▋         | 2947/41250 [7:06:46<92:26:18,  8.69s/it][2025-04-26 11:15:23,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:15:23,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.41 | bwd_microstep: 5883.89 | bwd_inner_microstep: 5695.70 | bwd_allreduce_microstep: 188.15 | step_microstep: 18.44
[2025-04-26 11:15:23,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.41 | bwd: 5883.90 | bwd_inner: 5695.70 | bwd_allreduce: 188.17 | step: 18.44

  7%|▋         | 2948/41250 [7:06:54<92:49:36,  8.72s/it]
                                                         
{'loss': 0.1265, 'grad_norm': 1.6485085487365723, 'learning_rate': 3.982000561926156e-05, 'epoch': 0.71}

  7%|▋         | 2948/41250 [7:06:54<92:49:36,  8.72s/it][2025-04-26 11:15:32,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 11:15:32,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.40 | bwd_microstep: 5770.40 | bwd_inner_microstep: 5652.14 | bwd_allreduce_microstep: 118.22 | step_microstep: 18.36
[2025-04-26 11:15:32,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.40 | bwd: 5770.41 | bwd_inner: 5652.14 | bwd_allreduce: 118.24 | step: 18.36

  7%|▋         | 2949/41250 [7:07:03<92:41:33,  8.71s/it]
                                                         
{'loss': 0.126, 'grad_norm': 0.8429327011108398, 'learning_rate': 3.98197953547646e-05, 'epoch': 0.71}

  7%|▋         | 2949/41250 [7:07:03<92:41:33,  8.71s/it][2025-04-26 11:15:40,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.76 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:15:40,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.12 | bwd_microstep: 5704.37 | bwd_inner_microstep: 5641.99 | bwd_allreduce_microstep: 62.33 | step_microstep: 18.58
[2025-04-26 11:15:40,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.12 | bwd: 5704.38 | bwd_inner: 5641.99 | bwd_allreduce: 62.35 | step: 18.58

  7%|▋         | 2950/41250 [7:07:12<92:23:04,  8.68s/it]
                                                         
{'loss': 0.1947, 'grad_norm': 1.2316656112670898, 'learning_rate': 3.98195849680825e-05, 'epoch': 0.72}

  7%|▋         | 2950/41250 [7:07:12<92:23:04,  8.68s/it][2025-04-26 11:15:49,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 11:15:49,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.49 | bwd_microstep: 5713.44 | bwd_inner_microstep: 5700.11 | bwd_allreduce_microstep: 13.28 | step_microstep: 18.83
[2025-04-26 11:15:49,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.50 | bwd: 5713.45 | bwd_inner: 5700.11 | bwd_allreduce: 13.30 | step: 18.83

  7%|▋         | 2951/41250 [7:07:20<92:17:08,  8.67s/it]
                                                         
{'loss': 0.1672, 'grad_norm': 1.5350061655044556, 'learning_rate': 3.981937445921655e-05, 'epoch': 0.72}

  7%|▋         | 2951/41250 [7:07:20<92:17:08,  8.67s/it][2025-04-26 11:15:58,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:15:58,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.19 | bwd_microstep: 5737.29 | bwd_inner_microstep: 5691.67 | bwd_allreduce_microstep: 45.58 | step_microstep: 18.01
[2025-04-26 11:15:58,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.19 | bwd: 5737.30 | bwd_inner: 5691.67 | bwd_allreduce: 45.60 | step: 18.01

  7%|▋         | 2952/41250 [7:07:29<92:15:51,  8.67s/it]
                                                         
{'loss': 0.2734, 'grad_norm': 2.187936544418335, 'learning_rate': 3.981916382816805e-05, 'epoch': 0.72}

  7%|▋         | 2952/41250 [7:07:29<92:15:51,  8.67s/it][2025-04-26 11:16:06,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:16:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.58 | bwd_microstep: 5718.68 | bwd_inner_microstep: 5705.96 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.25
[2025-04-26 11:16:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.58 | bwd: 5718.69 | bwd_inner: 5705.96 | bwd_allreduce: 12.69 | step: 18.26

  7%|▋         | 2953/41250 [7:07:38<92:11:52,  8.67s/it]
                                                         
{'loss': 0.2058, 'grad_norm': 1.4025181531906128, 'learning_rate': 3.981895307493831e-05, 'epoch': 0.72}

  7%|▋         | 2953/41250 [7:07:38<92:11:52,  8.67s/it][2025-04-26 11:16:15,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:16:15,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.60 | bwd_microstep: 5752.32 | bwd_inner_microstep: 5694.11 | bwd_allreduce_microstep: 58.16 | step_microstep: 17.66
[2025-04-26 11:16:15,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.60 | bwd: 5752.33 | bwd_inner: 5694.11 | bwd_allreduce: 58.18 | step: 17.66

  7%|▋         | 2954/41250 [7:07:46<92:13:15,  8.67s/it]
                                                         
{'loss': 0.0598, 'grad_norm': 0.9636123180389404, 'learning_rate': 3.9818742199528626e-05, 'epoch': 0.72}

  7%|▋         | 2954/41250 [7:07:46<92:13:15,  8.67s/it][2025-04-26 11:16:24,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:16:24,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.96 | bwd_microstep: 5750.24 | bwd_inner_microstep: 5704.98 | bwd_allreduce_microstep: 45.22 | step_microstep: 18.52
[2025-04-26 11:16:24,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.96 | bwd: 5750.25 | bwd_inner: 5704.98 | bwd_allreduce: 45.24 | step: 18.53

  7%|▋         | 2955/41250 [7:07:55<92:16:29,  8.67s/it]
                                                         
{'loss': 0.2155, 'grad_norm': 1.3812179565429688, 'learning_rate': 3.9818531201940295e-05, 'epoch': 0.72}

  7%|▋         | 2955/41250 [7:07:55<92:16:29,  8.67s/it][2025-04-26 11:16:32,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:16:32,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.83 | bwd_microstep: 5699.62 | bwd_inner_microstep: 5686.80 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.50
[2025-04-26 11:16:32,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.83 | bwd: 5699.64 | bwd_inner: 5686.80 | bwd_allreduce: 12.79 | step: 18.50

  7%|▋         | 2956/41250 [7:08:04<92:08:18,  8.66s/it]
                                                         
{'loss': 0.2945, 'grad_norm': 2.4135279655456543, 'learning_rate': 3.981832008217461e-05, 'epoch': 0.72}

  7%|▋         | 2956/41250 [7:08:04<92:08:18,  8.66s/it][2025-04-26 11:16:41,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.06 | optimizer_step: 1.09
[2025-04-26 11:16:41,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.36 | bwd_microstep: 5677.10 | bwd_inner_microstep: 5650.92 | bwd_allreduce_microstep: 26.12 | step_microstep: 19.60
[2025-04-26 11:16:41,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.36 | bwd: 5677.11 | bwd_inner: 5650.92 | bwd_allreduce: 26.14 | step: 19.61

  7%|▋         | 2957/41250 [7:08:12<91:56:41,  8.64s/it]
                                                         
{'loss': 0.1398, 'grad_norm': 1.6247209310531616, 'learning_rate': 3.981810884023288e-05, 'epoch': 0.72}

  7%|▋         | 2957/41250 [7:08:12<91:56:41,  8.64s/it][2025-04-26 11:16:50,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:16:50,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.14 | bwd_microstep: 5748.95 | bwd_inner_microstep: 5675.65 | bwd_allreduce_microstep: 73.25 | step_microstep: 19.03
[2025-04-26 11:16:50,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.14 | bwd: 5748.96 | bwd_inner: 5675.65 | bwd_allreduce: 73.27 | step: 19.04

  7%|▋         | 2958/41250 [7:08:21<92:04:22,  8.66s/it]
                                                         
{'loss': 0.3445, 'grad_norm': 1.6895111799240112, 'learning_rate': 3.9817897476116415e-05, 'epoch': 0.72}

  7%|▋         | 2958/41250 [7:08:21<92:04:22,  8.66s/it][2025-04-26 11:16:58,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:16:58,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.86 | bwd_microstep: 5689.69 | bwd_inner_microstep: 5641.40 | bwd_allreduce_microstep: 48.24 | step_microstep: 18.66
[2025-04-26 11:16:58,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.86 | bwd: 5689.70 | bwd_inner: 5641.40 | bwd_allreduce: 48.26 | step: 18.66

  7%|▋         | 2959/41250 [7:08:29<91:50:48,  8.64s/it]
                                                         
{'loss': 0.3819, 'grad_norm': 3.1428048610687256, 'learning_rate': 3.98176859898265e-05, 'epoch': 0.72}

  7%|▋         | 2959/41250 [7:08:29<91:50:48,  8.64s/it][2025-04-26 11:17:07,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 11:17:07,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.46 | bwd_microstep: 5695.75 | bwd_inner_microstep: 5682.68 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.69
[2025-04-26 11:17:07,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.46 | bwd: 5695.76 | bwd_inner: 5682.68 | bwd_allreduce: 13.04 | step: 18.70

  7%|▋         | 2960/41250 [7:08:38<91:50:28,  8.63s/it]
                                                         
{'loss': 0.1149, 'grad_norm': 0.7735581994056702, 'learning_rate': 3.981747438136445e-05, 'epoch': 0.72}

  7%|▋         | 2960/41250 [7:08:38<91:50:28,  8.63s/it][2025-04-26 11:17:16,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 11:17:16,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.82 | bwd_microstep: 5781.38 | bwd_inner_microstep: 5768.02 | bwd_allreduce_microstep: 13.31 | step_microstep: 18.88
[2025-04-26 11:17:16,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.82 | bwd: 5781.39 | bwd_inner: 5768.02 | bwd_allreduce: 13.33 | step: 18.88

  7%|▋         | 2961/41250 [7:08:47<92:13:17,  8.67s/it]
                                                         
{'loss': 0.0794, 'grad_norm': 0.6267747282981873, 'learning_rate': 3.9817262650731576e-05, 'epoch': 0.72}

  7%|▋         | 2961/41250 [7:08:47<92:13:17,  8.67s/it][2025-04-26 11:17:24,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 11:17:24,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.60 | bwd_microstep: 5694.84 | bwd_inner_microstep: 5682.15 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.39
[2025-04-26 11:17:24,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.60 | bwd: 5694.85 | bwd_inner: 5682.15 | bwd_allreduce: 12.65 | step: 19.40

  7%|▋         | 2962/41250 [7:08:55<92:03:17,  8.66s/it]
                                                         
{'loss': 0.1291, 'grad_norm': 0.9450050592422485, 'learning_rate': 3.981705079792917e-05, 'epoch': 0.72}

  7%|▋         | 2962/41250 [7:08:55<92:03:17,  8.66s/it][2025-04-26 11:17:33,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-26 11:17:33,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.86 | bwd_microstep: 5771.46 | bwd_inner_microstep: 5633.90 | bwd_allreduce_microstep: 137.51 | step_microstep: 18.80
[2025-04-26 11:17:33,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.86 | bwd: 5771.47 | bwd_inner: 5633.90 | bwd_allreduce: 137.53 | step: 18.80

  7%|▋         | 2963/41250 [7:09:04<92:04:15,  8.66s/it]
                                                         
{'loss': 0.1354, 'grad_norm': 1.2846896648406982, 'learning_rate': 3.981683882295855e-05, 'epoch': 0.72}

  7%|▋         | 2963/41250 [7:09:04<92:04:15,  8.66s/it][2025-04-26 11:17:42,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:17:42,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.82 | bwd_microstep: 5699.89 | bwd_inner_microstep: 5641.92 | bwd_allreduce_microstep: 57.93 | step_microstep: 18.65
[2025-04-26 11:17:42,048] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.82 | bwd: 5699.91 | bwd_inner: 5641.92 | bwd_allreduce: 57.95 | step: 18.65

  7%|▋         | 2964/41250 [7:09:13<91:53:58,  8.64s/it]
                                                         
{'loss': 0.1242, 'grad_norm': 1.1762995719909668, 'learning_rate': 3.981662672582101e-05, 'epoch': 0.72}

  7%|▋         | 2964/41250 [7:09:13<91:53:58,  8.64s/it][2025-04-26 11:17:50,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:17:50,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.37 | bwd_microstep: 5719.14 | bwd_inner_microstep: 5706.42 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.80
[2025-04-26 11:17:50,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.37 | bwd: 5719.15 | bwd_inner: 5706.42 | bwd_allreduce: 12.68 | step: 18.80

  7%|▋         | 2965/41250 [7:09:21<91:56:53,  8.65s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 0.9942660331726074, 'learning_rate': 3.981641450651787e-05, 'epoch': 0.72}

  7%|▋         | 2965/41250 [7:09:21<91:56:53,  8.65s/it][2025-04-26 11:17:59,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 11:17:59,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.80 | bwd_microstep: 5711.16 | bwd_inner_microstep: 5651.07 | bwd_allreduce_microstep: 60.03 | step_microstep: 18.94
[2025-04-26 11:17:59,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.80 | bwd: 5711.17 | bwd_inner: 5651.07 | bwd_allreduce: 60.05 | step: 18.95

  7%|▋         | 2966/41250 [7:09:30<91:51:42,  8.64s/it]
                                                         
{'loss': 0.2428, 'grad_norm': 1.8372480869293213, 'learning_rate': 3.981620216505042e-05, 'epoch': 0.72}

  7%|▋         | 2966/41250 [7:09:30<91:51:42,  8.64s/it][2025-04-26 11:18:08,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.92
[2025-04-26 11:18:08,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.85 | bwd_microstep: 5758.94 | bwd_inner_microstep: 5663.21 | bwd_allreduce_microstep: 95.69 | step_microstep: 18.97
[2025-04-26 11:18:08,010] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.85 | bwd: 5758.95 | bwd_inner: 5663.21 | bwd_allreduce: 95.71 | step: 18.98

  7%|▋         | 2967/41250 [7:09:39<92:00:36,  8.65s/it]
                                                         
{'loss': 0.1173, 'grad_norm': 1.4606821537017822, 'learning_rate': 3.981598970141999e-05, 'epoch': 0.72}

  7%|▋         | 2967/41250 [7:09:39<92:00:36,  8.65s/it][2025-04-26 11:18:16,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:18:16,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.38 | bwd_microstep: 5773.03 | bwd_inner_microstep: 5656.80 | bwd_allreduce_microstep: 116.18 | step_microstep: 18.49
[2025-04-26 11:18:16,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.38 | bwd: 5773.05 | bwd_inner: 5656.80 | bwd_allreduce: 116.20 | step: 18.49

  7%|▋         | 2968/41250 [7:09:47<92:07:29,  8.66s/it]
                                                         
{'loss': 0.2654, 'grad_norm': 1.4319943189620972, 'learning_rate': 3.981577711562788e-05, 'epoch': 0.72}

  7%|▋         | 2968/41250 [7:09:47<92:07:29,  8.66s/it][2025-04-26 11:18:25,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 11:18:25,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.24 | bwd_microstep: 5693.58 | bwd_inner_microstep: 5647.80 | bwd_allreduce_microstep: 45.74 | step_microstep: 18.95
[2025-04-26 11:18:25,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.24 | bwd: 5693.60 | bwd_inner: 5647.80 | bwd_allreduce: 45.76 | step: 18.96

  7%|▋         | 2969/41250 [7:09:56<91:58:02,  8.65s/it]
                                                         
{'loss': 0.2003, 'grad_norm': 1.378839135169983, 'learning_rate': 3.9815564407675406e-05, 'epoch': 0.72}

  7%|▋         | 2969/41250 [7:09:56<91:58:02,  8.65s/it][2025-04-26 11:18:33,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:18:33,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.18 | bwd_microstep: 5698.79 | bwd_inner_microstep: 5636.52 | bwd_allreduce_microstep: 62.23 | step_microstep: 18.58
[2025-04-26 11:18:33,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.18 | bwd: 5698.81 | bwd_inner: 5636.52 | bwd_allreduce: 62.25 | step: 18.58

  7%|▋         | 2970/41250 [7:10:05<91:49:20,  8.64s/it]
                                                         
{'loss': 0.0948, 'grad_norm': 1.135170817375183, 'learning_rate': 3.981535157756387e-05, 'epoch': 0.72}

  7%|▋         | 2970/41250 [7:10:05<91:49:20,  8.64s/it][2025-04-26 11:18:42,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:18:42,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.88 | bwd_microstep: 5846.90 | bwd_inner_microstep: 5683.67 | bwd_allreduce_microstep: 163.18 | step_microstep: 18.52
[2025-04-26 11:18:42,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.88 | bwd: 5846.91 | bwd_inner: 5683.67 | bwd_allreduce: 163.20 | step: 18.52

  7%|▋         | 2971/41250 [7:10:13<92:17:31,  8.68s/it]
                                                         
{'loss': 0.2478, 'grad_norm': 1.0498310327529907, 'learning_rate': 3.981513862529459e-05, 'epoch': 0.72}

  7%|▋         | 2971/41250 [7:10:13<92:17:31,  8.68s/it][2025-04-26 11:18:51,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:18:51,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.57 | bwd_microstep: 5704.63 | bwd_inner_microstep: 5691.93 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.59
[2025-04-26 11:18:51,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.57 | bwd: 5704.64 | bwd_inner: 5691.92 | bwd_allreduce: 12.68 | step: 18.60

  7%|▋         | 2972/41250 [7:10:22<92:09:57,  8.67s/it]
                                                         
{'loss': 0.1359, 'grad_norm': 1.1110211610794067, 'learning_rate': 3.981492555086888e-05, 'epoch': 0.72}

  7%|▋         | 2972/41250 [7:10:22<92:09:57,  8.67s/it][2025-04-26 11:19:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:19:00,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.59 | bwd_microstep: 5760.65 | bwd_inner_microstep: 5647.29 | bwd_allreduce_microstep: 113.31 | step_microstep: 17.85
[2025-04-26 11:19:00,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.59 | bwd: 5760.66 | bwd_inner: 5647.29 | bwd_allreduce: 113.33 | step: 17.86

  7%|▋         | 2973/41250 [7:10:31<92:13:23,  8.67s/it]
                                                         
{'loss': 0.0937, 'grad_norm': 1.0874452590942383, 'learning_rate': 3.981471235428805e-05, 'epoch': 0.72}

  7%|▋         | 2973/41250 [7:10:31<92:13:23,  8.67s/it][2025-04-26 11:19:08,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 11:19:08,831] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.37 | bwd_microstep: 5885.20 | bwd_inner_microstep: 5659.66 | bwd_allreduce_microstep: 225.49 | step_microstep: 19.00
[2025-04-26 11:19:08,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.37 | bwd: 5885.21 | bwd_inner: 5659.66 | bwd_allreduce: 225.52 | step: 19.01

  7%|▋         | 2974/41250 [7:10:40<92:37:48,  8.71s/it]
                                                         
{'loss': 0.1071, 'grad_norm': 0.9340155124664307, 'learning_rate': 3.981449903555341e-05, 'epoch': 0.72}

  7%|▋         | 2974/41250 [7:10:40<92:37:48,  8.71s/it][2025-04-26 11:19:17,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:19:17,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.98 | bwd_microstep: 5734.61 | bwd_inner_microstep: 5693.07 | bwd_allreduce_microstep: 41.50 | step_microstep: 18.41
[2025-04-26 11:19:17,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.98 | bwd: 5734.63 | bwd_inner: 5693.07 | bwd_allreduce: 41.52 | step: 18.42

  7%|▋         | 2975/41250 [7:10:48<92:30:58,  8.70s/it]
                                                         
{'loss': 0.2301, 'grad_norm': 1.376815915107727, 'learning_rate': 3.9814285594666286e-05, 'epoch': 0.72}

  7%|▋         | 2975/41250 [7:10:48<92:30:58,  8.70s/it][2025-04-26 11:19:26,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:19:26,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.33 | bwd_microstep: 5770.69 | bwd_inner_microstep: 5758.08 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.17
[2025-04-26 11:19:26,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.33 | bwd: 5770.70 | bwd_inner: 5758.08 | bwd_allreduce: 12.58 | step: 18.18

  7%|▋         | 2976/41250 [7:10:57<92:38:48,  8.71s/it]
                                                         
{'loss': 0.1199, 'grad_norm': 0.9703726172447205, 'learning_rate': 3.9814072031627986e-05, 'epoch': 0.72}

  7%|▋         | 2976/41250 [7:10:57<92:38:48,  8.71s/it][2025-04-26 11:19:34,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 11:19:34,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.45 | bwd_microstep: 5722.23 | bwd_inner_microstep: 5709.41 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.20
[2025-04-26 11:19:34,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.45 | bwd: 5722.24 | bwd_inner: 5709.41 | bwd_allreduce: 12.79 | step: 18.20

  7%|▋         | 2977/41250 [7:11:06<92:27:57,  8.70s/it]
                                                         
{'loss': 0.198, 'grad_norm': 2.08417010307312, 'learning_rate': 3.981385834643983e-05, 'epoch': 0.72}

  7%|▋         | 2977/41250 [7:11:06<92:27:57,  8.70s/it][2025-04-26 11:19:43,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:19:43,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.96 | bwd_microstep: 5698.26 | bwd_inner_microstep: 5660.25 | bwd_allreduce_microstep: 37.97 | step_microstep: 17.83
[2025-04-26 11:19:43,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.96 | bwd: 5698.27 | bwd_inner: 5660.25 | bwd_allreduce: 37.98 | step: 17.84

  7%|▋         | 2978/41250 [7:11:14<92:13:34,  8.68s/it]
                                                         
{'loss': 0.3391, 'grad_norm': 2.117741584777832, 'learning_rate': 3.981364453910314e-05, 'epoch': 0.72}

  7%|▋         | 2978/41250 [7:11:14<92:13:34,  8.68s/it][2025-04-26 11:19:52,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:19:52,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.61 | bwd_microstep: 5713.34 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 64.86 | step_microstep: 18.33
[2025-04-26 11:19:52,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.61 | bwd: 5713.36 | bwd_inner: 5648.44 | bwd_allreduce: 64.88 | step: 18.34

  7%|▋         | 2979/41250 [7:11:23<92:04:02,  8.66s/it]
                                                         
{'loss': 0.1201, 'grad_norm': 1.5744943618774414, 'learning_rate': 3.9813430609619224e-05, 'epoch': 0.72}

  7%|▋         | 2979/41250 [7:11:23<92:04:02,  8.66s/it][2025-04-26 11:20:00,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-26 11:20:00,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.47 | bwd_microstep: 5786.21 | bwd_inner_microstep: 5666.25 | bwd_allreduce_microstep: 119.91 | step_microstep: 18.84
[2025-04-26 11:20:00,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.47 | bwd: 5786.22 | bwd_inner: 5666.25 | bwd_allreduce: 119.93 | step: 18.84

  7%|▋         | 2980/41250 [7:11:32<92:11:40,  8.67s/it]
                                                         
{'loss': 0.1888, 'grad_norm': 2.5558695793151855, 'learning_rate': 3.98132165579894e-05, 'epoch': 0.72}

  7%|▋         | 2980/41250 [7:11:32<92:11:40,  8.67s/it][2025-04-26 11:20:09,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 11:20:09,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.90 | bwd_microstep: 5750.46 | bwd_inner_microstep: 5705.18 | bwd_allreduce_microstep: 45.23 | step_microstep: 18.86
[2025-04-26 11:20:09,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.91 | bwd: 5750.47 | bwd_inner: 5705.18 | bwd_allreduce: 45.25 | step: 18.88

  7%|▋         | 2981/41250 [7:11:40<92:15:11,  8.68s/it]
                                                         
{'loss': 0.2138, 'grad_norm': 2.747473955154419, 'learning_rate': 3.9813002384215004e-05, 'epoch': 0.72}

  7%|▋         | 2981/41250 [7:11:40<92:15:11,  8.68s/it][2025-04-26 11:20:18,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 11:20:18,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.24 | bwd_microstep: 5761.70 | bwd_inner_microstep: 5703.41 | bwd_allreduce_microstep: 58.25 | step_microstep: 17.76
[2025-04-26 11:20:18,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.24 | bwd: 5761.72 | bwd_inner: 5703.41 | bwd_allreduce: 58.26 | step: 17.76

  7%|▋         | 2982/41250 [7:11:49<92:19:38,  8.69s/it]
                                                         
{'loss': 0.2867, 'grad_norm': 1.744069218635559, 'learning_rate': 3.981278808829734e-05, 'epoch': 0.72}

  7%|▋         | 2982/41250 [7:11:49<92:19:38,  8.69s/it][2025-04-26 11:20:26,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:20:26,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.10 | bwd_microstep: 5718.02 | bwd_inner_microstep: 5705.20 | bwd_allreduce_microstep: 12.78 | step_microstep: 17.74
[2025-04-26 11:20:26,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.10 | bwd: 5718.03 | bwd_inner: 5705.20 | bwd_allreduce: 12.80 | step: 17.74

  7%|▋         | 2983/41250 [7:11:58<92:13:27,  8.68s/it]
                                                         
{'loss': 0.2149, 'grad_norm': 1.2376747131347656, 'learning_rate': 3.9812573670237735e-05, 'epoch': 0.72}

  7%|▋         | 2983/41250 [7:11:58<92:13:27,  8.68s/it][2025-04-26 11:20:35,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:20:35,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.13 | bwd_microstep: 5706.59 | bwd_inner_microstep: 5670.03 | bwd_allreduce_microstep: 36.52 | step_microstep: 17.67
[2025-04-26 11:20:35,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.13 | bwd: 5706.60 | bwd_inner: 5670.03 | bwd_allreduce: 36.54 | step: 17.68

  7%|▋         | 2984/41250 [7:12:06<92:03:20,  8.66s/it]
                                                         
{'loss': 0.166, 'grad_norm': 1.9907515048980713, 'learning_rate': 3.981235913003751e-05, 'epoch': 0.72}

  7%|▋         | 2984/41250 [7:12:06<92:03:20,  8.66s/it][2025-04-26 11:20:44,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:20:44,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.84 | bwd_microstep: 5776.82 | bwd_inner_microstep: 5763.94 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.61
[2025-04-26 11:20:44,283] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.84 | bwd: 5776.83 | bwd_inner: 5763.94 | bwd_allreduce: 12.85 | step: 18.62

  7%|▋         | 2985/41250 [7:12:15<92:20:23,  8.69s/it]
                                                         
{'loss': 0.1402, 'grad_norm': 1.3294528722763062, 'learning_rate': 3.981214446769798e-05, 'epoch': 0.72}

  7%|▋         | 2985/41250 [7:12:15<92:20:23,  8.69s/it][2025-04-26 11:20:52,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:20:52,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5793.22 | bwd_inner_microstep: 5650.13 | bwd_allreduce_microstep: 143.05 | step_microstep: 18.60
[2025-04-26 11:20:52,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5793.23 | bwd_inner: 5650.13 | bwd_allreduce: 143.06 | step: 18.60

  7%|▋         | 2986/41250 [7:12:24<92:22:08,  8.69s/it]
                                                         
{'loss': 0.2366, 'grad_norm': 2.3678479194641113, 'learning_rate': 3.9811929683220485e-05, 'epoch': 0.72}

  7%|▋         | 2986/41250 [7:12:24<92:22:08,  8.69s/it][2025-04-26 11:21:01,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:21:01,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.83 | bwd_microstep: 5759.30 | bwd_inner_microstep: 5697.68 | bwd_allreduce_microstep: 61.58 | step_microstep: 18.20
[2025-04-26 11:21:01,684] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.83 | bwd: 5759.31 | bwd_inner: 5697.68 | bwd_allreduce: 61.60 | step: 18.20

  7%|▋         | 2987/41250 [7:12:32<92:24:38,  8.69s/it]
                                                         
{'loss': 0.158, 'grad_norm': 2.364053964614868, 'learning_rate': 3.981171477660634e-05, 'epoch': 0.72}

  7%|▋         | 2987/41250 [7:12:32<92:24:38,  8.69s/it][2025-04-26 11:21:10,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-26 11:21:10,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.29 | bwd_microstep: 5722.80 | bwd_inner_microstep: 5651.92 | bwd_allreduce_microstep: 70.83 | step_microstep: 18.65
[2025-04-26 11:21:10,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.29 | bwd: 5722.81 | bwd_inner: 5651.92 | bwd_allreduce: 70.85 | step: 18.65

  7%|▋         | 2988/41250 [7:12:41<92:12:01,  8.67s/it]
                                                         
{'loss': 0.2325, 'grad_norm': 1.2882047891616821, 'learning_rate': 3.9811499747856875e-05, 'epoch': 0.72}

  7%|▋         | 2988/41250 [7:12:41<92:12:01,  8.67s/it][2025-04-26 11:21:18,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:21:18,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.63 | bwd_microstep: 5700.28 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.65
[2025-04-26 11:21:18,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.63 | bwd: 5700.30 | bwd_inner: 5687.54 | bwd_allreduce: 12.71 | step: 18.65

  7%|▋         | 2989/41250 [7:12:50<92:03:30,  8.66s/it]
                                                         
{'loss': 0.2268, 'grad_norm': 1.293493390083313, 'learning_rate': 3.98112845969734e-05, 'epoch': 0.72}

  7%|▋         | 2989/41250 [7:12:50<92:03:30,  8.66s/it][2025-04-26 11:21:27,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 11:21:27,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.38 | bwd_microstep: 5725.71 | bwd_inner_microstep: 5713.08 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.45
[2025-04-26 11:21:27,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.38 | bwd: 5725.72 | bwd_inner: 5713.08 | bwd_allreduce: 12.61 | step: 18.46

  7%|▋         | 2990/41250 [7:12:58<92:02:36,  8.66s/it]
                                                         
{'loss': 0.3516, 'grad_norm': 2.30017352104187, 'learning_rate': 3.981106932395726e-05, 'epoch': 0.72}

  7%|▋         | 2990/41250 [7:12:58<92:02:36,  8.66s/it][2025-04-26 11:21:36,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:21:36,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5711.49 | bwd_inner_microstep: 5698.69 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.46
[2025-04-26 11:21:36,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5711.50 | bwd_inner: 5698.69 | bwd_allreduce: 12.78 | step: 18.46

  7%|▋         | 2991/41250 [7:13:07<91:58:49,  8.65s/it]
                                                         
{'loss': 0.1006, 'grad_norm': 1.0908043384552002, 'learning_rate': 3.981085392880977e-05, 'epoch': 0.73}

  7%|▋         | 2991/41250 [7:13:07<91:58:49,  8.65s/it][2025-04-26 11:21:44,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.06 | optimizer_step: 0.91
[2025-04-26 11:21:44,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.14 | bwd_microstep: 5756.90 | bwd_inner_microstep: 5702.65 | bwd_allreduce_microstep: 54.20 | step_microstep: 18.87
[2025-04-26 11:21:44,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.14 | bwd: 5756.91 | bwd_inner: 5702.65 | bwd_allreduce: 54.22 | step: 18.88

  7%|▋         | 2992/41250 [7:13:16<92:07:13,  8.67s/it]
                                                         
{'loss': 0.202, 'grad_norm': 1.5173176527023315, 'learning_rate': 3.9810638411532264e-05, 'epoch': 0.73}

  7%|▋         | 2992/41250 [7:13:16<92:07:13,  8.67s/it][2025-04-26 11:21:53,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:21:53,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.86 | bwd_microstep: 5721.43 | bwd_inner_microstep: 5644.77 | bwd_allreduce_microstep: 76.61 | step_microstep: 18.74
[2025-04-26 11:21:53,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.86 | bwd: 5721.45 | bwd_inner: 5644.77 | bwd_allreduce: 76.63 | step: 18.75

  7%|▋         | 2993/41250 [7:13:24<91:59:29,  8.66s/it]
                                                         
{'loss': 0.127, 'grad_norm': 1.403841257095337, 'learning_rate': 3.9810422772126064e-05, 'epoch': 0.73}

  7%|▋         | 2993/41250 [7:13:24<91:59:29,  8.66s/it][2025-04-26 11:22:02,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:22:02,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5762.45 | bwd_inner_microstep: 5698.20 | bwd_allreduce_microstep: 64.20 | step_microstep: 18.97
[2025-04-26 11:22:02,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5762.47 | bwd_inner: 5698.20 | bwd_allreduce: 64.22 | step: 18.97

  7%|▋         | 2994/41250 [7:13:33<92:06:49,  8.67s/it]
                                                         
{'loss': 0.1409, 'grad_norm': 0.9573098421096802, 'learning_rate': 3.981020701059251e-05, 'epoch': 0.73}

  7%|▋         | 2994/41250 [7:13:33<92:06:49,  8.67s/it][2025-04-26 11:22:10,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.94
[2025-04-26 11:22:10,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5752.13 | bwd_inner_microstep: 5687.11 | bwd_allreduce_microstep: 64.97 | step_microstep: 18.17
[2025-04-26 11:22:10,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5752.14 | bwd_inner: 5687.11 | bwd_allreduce: 64.99 | step: 18.17

  7%|▋         | 2995/41250 [7:13:42<92:08:13,  8.67s/it]
                                                         
{'loss': 0.1437, 'grad_norm': 1.3699513673782349, 'learning_rate': 3.9809991126932924e-05, 'epoch': 0.73}

  7%|▋         | 2995/41250 [7:13:42<92:08:13,  8.67s/it][2025-04-26 11:22:19,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 11:22:19,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.57 | bwd_microstep: 5757.15 | bwd_inner_microstep: 5692.29 | bwd_allreduce_microstep: 64.81 | step_microstep: 18.78
[2025-04-26 11:22:19,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.57 | bwd: 5757.17 | bwd_inner: 5692.29 | bwd_allreduce: 64.83 | step: 18.78

  7%|▋         | 2996/41250 [7:13:50<92:13:33,  8.68s/it]
                                                         
{'loss': 0.2367, 'grad_norm': 1.4889837503433228, 'learning_rate': 3.980977512114864e-05, 'epoch': 0.73}

  7%|▋         | 2996/41250 [7:13:50<92:13:33,  8.68s/it][2025-04-26 11:22:28,421] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:22:28,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5856.10 | bwd_inner_microstep: 5640.50 | bwd_allreduce_microstep: 215.55 | step_microstep: 17.81
[2025-04-26 11:22:28,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5856.11 | bwd_inner: 5640.50 | bwd_allreduce: 215.57 | step: 17.82

  7%|▋         | 2997/41250 [7:13:59<92:32:12,  8.71s/it]
                                                         
{'loss': 0.2606, 'grad_norm': 1.7115994691848755, 'learning_rate': 3.980955899324099e-05, 'epoch': 0.73}

  7%|▋         | 2997/41250 [7:13:59<92:32:12,  8.71s/it][2025-04-26 11:22:37,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:22:37,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.67 | bwd_microstep: 5755.71 | bwd_inner_microstep: 5679.69 | bwd_allreduce_microstep: 75.98 | step_microstep: 18.30
[2025-04-26 11:22:37,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.67 | bwd: 5755.73 | bwd_inner: 5679.69 | bwd_allreduce: 76.00 | step: 18.31

  7%|▋         | 2998/41250 [7:14:08<92:26:14,  8.70s/it]
                                                         
{'loss': 0.1298, 'grad_norm': 1.2323390245437622, 'learning_rate': 3.98093427432113e-05, 'epoch': 0.73}

  7%|▋         | 2998/41250 [7:14:08<92:26:14,  8.70s/it][2025-04-26 11:22:45,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.04 | optimizer_step: 1.06
[2025-04-26 11:22:45,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.40 | bwd_microstep: 5761.32 | bwd_inner_microstep: 5635.04 | bwd_allreduce_microstep: 126.23 | step_microstep: 18.93
[2025-04-26 11:22:45,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.40 | bwd: 5761.34 | bwd_inner: 5635.04 | bwd_allreduce: 126.26 | step: 18.93

  7%|▋         | 2999/41250 [7:14:16<92:19:26,  8.69s/it]
                                                         
{'loss': 0.4019, 'grad_norm': 2.0224015712738037, 'learning_rate': 3.980912637106091e-05, 'epoch': 0.73}

  7%|▋         | 2999/41250 [7:14:16<92:19:26,  8.69s/it][2025-04-26 11:22:54,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 11:22:54,369] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.54 | bwd_microstep: 5678.23 | bwd_inner_microstep: 5652.59 | bwd_allreduce_microstep: 25.58 | step_microstep: 19.34
[2025-04-26 11:22:54,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.55 | bwd: 5678.24 | bwd_inner: 5652.59 | bwd_allreduce: 25.61 | step: 19.34

  7%|▋         | 3000/41250 [7:14:25<92:03:19,  8.66s/it]
                                                         
{'loss': 0.2401, 'grad_norm': 1.2073538303375244, 'learning_rate': 3.9808909876791155e-05, 'epoch': 0.73}

  7%|▋         | 3000/41250 [7:14:25<92:03:19,  8.66s/it][2025-04-26 11:23:03,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.09 | optimizer_step: 1.01
[2025-04-26 11:23:03,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.00 | bwd_microstep: 6013.14 | bwd_inner_microstep: 5680.92 | bwd_allreduce_microstep: 332.17 | step_microstep: 18.81
[2025-04-26 11:23:03,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.00 | bwd: 6013.16 | bwd_inner: 5680.92 | bwd_allreduce: 332.20 | step: 18.82

  7%|▋         | 3001/41250 [7:14:34<92:56:01,  8.75s/it]
                                                         
{'loss': 0.1583, 'grad_norm': 1.8028879165649414, 'learning_rate': 3.9808693260403364e-05, 'epoch': 0.73}

  7%|▋         | 3001/41250 [7:14:34<92:56:01,  8.75s/it][2025-04-26 11:23:11,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:23:11,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.31 | bwd_microstep: 5676.89 | bwd_inner_microstep: 5643.82 | bwd_allreduce_microstep: 33.03 | step_microstep: 18.82
[2025-04-26 11:23:11,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.31 | bwd: 5676.90 | bwd_inner: 5643.82 | bwd_allreduce: 33.04 | step: 18.82

  7%|▋         | 3002/41250 [7:14:43<92:24:45,  8.70s/it]
                                                         
{'loss': 0.2254, 'grad_norm': 1.5566514730453491, 'learning_rate': 3.980847652189888e-05, 'epoch': 0.73}

  7%|▋         | 3002/41250 [7:14:43<92:24:45,  8.70s/it][2025-04-26 11:23:20,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 11:23:20,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.49 | bwd_microstep: 5713.57 | bwd_inner_microstep: 5700.14 | bwd_allreduce_microstep: 13.38 | step_microstep: 18.93
[2025-04-26 11:23:20,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.49 | bwd: 5713.58 | bwd_inner: 5700.14 | bwd_allreduce: 13.40 | step: 18.93

  7%|▋         | 3003/41250 [7:14:51<92:15:00,  8.68s/it]
                                                         
{'loss': 0.2425, 'grad_norm': 1.4400299787521362, 'learning_rate': 3.9808259661279026e-05, 'epoch': 0.73}

  7%|▋         | 3003/41250 [7:14:51<92:15:00,  8.68s/it][2025-04-26 11:23:29,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:23:29,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.38 | bwd_microstep: 5689.24 | bwd_inner_microstep: 5676.60 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.73
[2025-04-26 11:23:29,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.38 | bwd: 5689.26 | bwd_inner: 5676.60 | bwd_allreduce: 12.61 | step: 18.73

  7%|▋         | 3004/41250 [7:15:00<91:59:55,  8.66s/it]
                                                         
{'loss': 0.118, 'grad_norm': 0.8083534240722656, 'learning_rate': 3.9808042678545155e-05, 'epoch': 0.73}

  7%|▋         | 3004/41250 [7:15:00<91:59:55,  8.66s/it][2025-04-26 11:23:37,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.14
[2025-04-26 11:23:37,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.49 | bwd_microstep: 5784.42 | bwd_inner_microstep: 5771.66 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.97
[2025-04-26 11:23:37,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.49 | bwd: 5784.43 | bwd_inner: 5771.66 | bwd_allreduce: 12.74 | step: 18.97

  7%|▋         | 3005/41250 [7:15:09<92:17:43,  8.69s/it]
                                                         
{'loss': 0.182, 'grad_norm': 1.343223214149475, 'learning_rate': 3.98078255736986e-05, 'epoch': 0.73}

  7%|▋         | 3005/41250 [7:15:09<92:17:43,  8.69s/it][2025-04-26 11:23:46,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.09 | optimizer_step: 1.11
[2025-04-26 11:23:46,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5690.36 | bwd_inner_microstep: 5646.66 | bwd_allreduce_microstep: 43.65 | step_microstep: 19.56
[2025-04-26 11:23:46,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5690.38 | bwd_inner: 5646.66 | bwd_allreduce: 43.68 | step: 19.56

  7%|▋         | 3006/41250 [7:15:17<92:01:51,  8.66s/it]
                                                         
{'loss': 0.1909, 'grad_norm': 1.5308494567871094, 'learning_rate': 3.980760834674069e-05, 'epoch': 0.73}

  7%|▋         | 3006/41250 [7:15:17<92:01:51,  8.66s/it][2025-04-26 11:23:55,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 11:23:55,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.33 | bwd_microstep: 5749.18 | bwd_inner_microstep: 5680.04 | bwd_allreduce_microstep: 69.08 | step_microstep: 18.99
[2025-04-26 11:23:55,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.33 | bwd: 5749.19 | bwd_inner: 5680.04 | bwd_allreduce: 69.10 | step: 18.99

  7%|▋         | 3007/41250 [7:15:26<92:05:20,  8.67s/it]
                                                         
{'loss': 0.2681, 'grad_norm': 1.6481093168258667, 'learning_rate': 3.980739099767277e-05, 'epoch': 0.73}

  7%|▋         | 3007/41250 [7:15:26<92:05:20,  8.67s/it][2025-04-26 11:24:03,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:24:03,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.64 | bwd_microstep: 5765.45 | bwd_inner_microstep: 5640.53 | bwd_allreduce_microstep: 124.87 | step_microstep: 18.79
[2025-04-26 11:24:03,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.64 | bwd: 5765.46 | bwd_inner: 5640.53 | bwd_allreduce: 124.89 | step: 18.79

  7%|▋         | 3008/41250 [7:15:35<92:03:16,  8.67s/it]
                                                         
{'loss': 0.2421, 'grad_norm': 1.3494138717651367, 'learning_rate': 3.980717352649618e-05, 'epoch': 0.73}

  7%|▋         | 3008/41250 [7:15:35<92:03:16,  8.67s/it][2025-04-26 11:24:12,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:24:12,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.38 | bwd_microstep: 5702.96 | bwd_inner_microstep: 5690.10 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.92
[2025-04-26 11:24:12,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.38 | bwd: 5702.97 | bwd_inner: 5690.09 | bwd_allreduce: 12.83 | step: 18.92

  7%|▋         | 3009/41250 [7:15:43<91:56:07,  8.65s/it]
                                                         
{'loss': 0.0892, 'grad_norm': 0.718294084072113, 'learning_rate': 3.980695593321226e-05, 'epoch': 0.73}

  7%|▋         | 3009/41250 [7:15:43<91:56:07,  8.65s/it][2025-04-26 11:24:21,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.11 | optimizer_step: 1.23
[2025-04-26 11:24:21,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.72 | bwd_microstep: 5708.96 | bwd_inner_microstep: 5696.21 | bwd_allreduce_microstep: 12.69 | step_microstep: 19.93
[2025-04-26 11:24:21,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.72 | bwd: 5708.97 | bwd_inner: 5696.21 | bwd_allreduce: 12.71 | step: 19.94

  7%|▋         | 3010/41250 [7:15:52<91:55:28,  8.65s/it]
                                                         
{'loss': 0.1251, 'grad_norm': 0.9765771627426147, 'learning_rate': 3.980673821782236e-05, 'epoch': 0.73}

  7%|▋         | 3010/41250 [7:15:52<91:55:28,  8.65s/it][2025-04-26 11:24:29,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 1.05
[2025-04-26 11:24:29,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.59 | bwd_microstep: 5779.47 | bwd_inner_microstep: 5637.64 | bwd_allreduce_microstep: 141.77 | step_microstep: 19.62
[2025-04-26 11:24:29,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.59 | bwd: 5779.48 | bwd_inner: 5637.64 | bwd_allreduce: 141.79 | step: 19.62

  7%|▋         | 3011/41250 [7:16:00<92:01:37,  8.66s/it]
                                                         
{'loss': 0.1321, 'grad_norm': 0.9646005630493164, 'learning_rate': 3.980652038032781e-05, 'epoch': 0.73}

  7%|▋         | 3011/41250 [7:16:01<92:01:37,  8.66s/it][2025-04-26 11:24:38,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.06 | optimizer_step: 0.93
[2025-04-26 11:24:38,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.43 | bwd_microstep: 5759.43 | bwd_inner_microstep: 5680.43 | bwd_allreduce_microstep: 78.95 | step_microstep: 19.26
[2025-04-26 11:24:38,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.43 | bwd: 5759.44 | bwd_inner: 5680.43 | bwd_allreduce: 78.97 | step: 19.26

  7%|▋         | 3012/41250 [7:16:09<92:07:06,  8.67s/it]
                                                         
{'loss': 0.2166, 'grad_norm': 1.2494666576385498, 'learning_rate': 3.9806302420729956e-05, 'epoch': 0.73}

  7%|▋         | 3012/41250 [7:16:09<92:07:06,  8.67s/it][2025-04-26 11:24:47,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:24:47,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.68 | bwd_microstep: 5767.17 | bwd_inner_microstep: 5645.19 | bwd_allreduce_microstep: 121.92 | step_microstep: 18.98
[2025-04-26 11:24:47,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.68 | bwd: 5767.18 | bwd_inner: 5645.19 | bwd_allreduce: 121.95 | step: 18.99

  7%|▋         | 3013/41250 [7:16:18<92:06:04,  8.67s/it]
                                                         
{'loss': 0.1678, 'grad_norm': 0.8654699921607971, 'learning_rate': 3.980608433903014e-05, 'epoch': 0.73}

  7%|▋         | 3013/41250 [7:16:18<92:06:04,  8.67s/it][2025-04-26 11:24:55,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:24:55,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.32 | bwd_microstep: 5694.48 | bwd_inner_microstep: 5655.24 | bwd_allreduce_microstep: 39.19 | step_microstep: 18.98
[2025-04-26 11:24:55,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.32 | bwd: 5694.49 | bwd_inner: 5655.24 | bwd_allreduce: 39.21 | step: 18.98

  7%|▋         | 3014/41250 [7:16:26<91:54:25,  8.65s/it]
                                                         
{'loss': 0.1309, 'grad_norm': 0.9611576199531555, 'learning_rate': 3.9805866135229716e-05, 'epoch': 0.73}

  7%|▋         | 3014/41250 [7:16:26<91:54:25,  8.65s/it][2025-04-26 11:25:04,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.05 | optimizer_step: 0.94
[2025-04-26 11:25:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.74 | bwd_microstep: 5719.73 | bwd_inner_microstep: 5706.90 | bwd_allreduce_microstep: 12.77 | step_microstep: 19.12
[2025-04-26 11:25:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.74 | bwd: 5719.74 | bwd_inner: 5706.90 | bwd_allreduce: 12.79 | step: 19.12

  7%|▋         | 3015/41250 [7:16:35<91:55:53,  8.66s/it]
                                                         
{'loss': 0.0638, 'grad_norm': 0.5293380618095398, 'learning_rate': 3.9805647809330026e-05, 'epoch': 0.73}

  7%|▋         | 3015/41250 [7:16:35<91:55:53,  8.66s/it][2025-04-26 11:25:13,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.18 | optimizer_step: 1.00
[2025-04-26 11:25:13,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.77 | bwd_microstep: 5774.91 | bwd_inner_microstep: 5650.80 | bwd_allreduce_microstep: 123.87 | step_microstep: 20.35
[2025-04-26 11:25:13,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.77 | bwd: 5774.94 | bwd_inner: 5650.80 | bwd_allreduce: 124.02 | step: 20.35

  7%|▋         | 3016/41250 [7:16:44<92:02:24,  8.67s/it]
                                                         
{'loss': 0.2629, 'grad_norm': 1.6431219577789307, 'learning_rate': 3.9805429361332406e-05, 'epoch': 0.73}

  7%|▋         | 3016/41250 [7:16:44<92:02:24,  8.67s/it][2025-04-26 11:25:21,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-26 11:25:21,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.32 | bwd_microstep: 5728.05 | bwd_inner_microstep: 5644.72 | bwd_allreduce_microstep: 83.28 | step_microstep: 19.06
[2025-04-26 11:25:21,767] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.32 | bwd: 5728.06 | bwd_inner: 5644.72 | bwd_allreduce: 83.30 | step: 19.06

  7%|▋         | 3017/41250 [7:16:52<91:55:04,  8.65s/it]
                                                         
{'loss': 0.2726, 'grad_norm': 1.5679296255111694, 'learning_rate': 3.980521079123821e-05, 'epoch': 0.73}

  7%|▋         | 3017/41250 [7:16:52<91:55:04,  8.65s/it][2025-04-26 11:25:30,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:25:30,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.66 | bwd_microstep: 5713.44 | bwd_inner_microstep: 5634.71 | bwd_allreduce_microstep: 78.67 | step_microstep: 18.84
[2025-04-26 11:25:30,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.66 | bwd: 5713.45 | bwd_inner: 5634.71 | bwd_allreduce: 78.69 | step: 18.85

  7%|▋         | 3018/41250 [7:17:01<91:47:47,  8.64s/it]
                                                         
{'loss': 0.1148, 'grad_norm': 1.3746652603149414, 'learning_rate': 3.980499209904879e-05, 'epoch': 0.73}

  7%|▋         | 3018/41250 [7:17:01<91:47:47,  8.64s/it][2025-04-26 11:25:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:25:39,024] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.51 | bwd_microstep: 5712.52 | bwd_inner_microstep: 5699.61 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.71
[2025-04-26 11:25:39,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.51 | bwd: 5712.53 | bwd_inner: 5699.61 | bwd_allreduce: 12.88 | step: 18.71

  7%|▋         | 3019/41250 [7:17:10<91:46:37,  8.64s/it]
                                                         
{'loss': 0.2677, 'grad_norm': 2.1749331951141357, 'learning_rate': 3.980477328476548e-05, 'epoch': 0.73}

  7%|▋         | 3019/41250 [7:17:10<91:46:37,  8.64s/it][2025-04-26 11:25:47,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 11:25:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.68 | bwd_microstep: 5760.61 | bwd_inner_microstep: 5684.77 | bwd_allreduce_microstep: 75.79 | step_microstep: 19.19
[2025-04-26 11:25:47,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.68 | bwd: 5760.62 | bwd_inner: 5684.77 | bwd_allreduce: 75.81 | step: 19.19

  7%|▋         | 3020/41250 [7:17:18<91:55:39,  8.66s/it]
                                                         
{'loss': 0.1861, 'grad_norm': 1.3415895700454712, 'learning_rate': 3.980455434838964e-05, 'epoch': 0.73}

  7%|▋         | 3020/41250 [7:17:18<91:55:39,  8.66s/it][2025-04-26 11:25:56,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:25:56,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.31 | bwd_microstep: 5749.85 | bwd_inner_microstep: 5701.34 | bwd_allreduce_microstep: 48.46 | step_microstep: 18.10
[2025-04-26 11:25:56,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.31 | bwd: 5749.86 | bwd_inner: 5701.34 | bwd_allreduce: 48.48 | step: 18.11

  7%|▋         | 3021/41250 [7:17:27<92:01:14,  8.67s/it]
                                                         
{'loss': 0.1477, 'grad_norm': 1.1887565851211548, 'learning_rate': 3.9804335289922624e-05, 'epoch': 0.73}

  7%|▋         | 3021/41250 [7:17:27<92:01:14,  8.67s/it][2025-04-26 11:26:05,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.75 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 11:26:05,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.70 | bwd_microstep: 5777.32 | bwd_inner_microstep: 5641.01 | bwd_allreduce_microstep: 136.26 | step_microstep: 19.01
[2025-04-26 11:26:05,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.70 | bwd: 5777.33 | bwd_inner: 5641.01 | bwd_allreduce: 136.28 | step: 19.02

  7%|▋         | 3022/41250 [7:17:36<92:03:13,  8.67s/it]
                                                         
{'loss': 0.196, 'grad_norm': 1.720065951347351, 'learning_rate': 3.9804116109365775e-05, 'epoch': 0.73}

  7%|▋         | 3022/41250 [7:17:36<92:03:13,  8.67s/it][2025-04-26 11:26:13,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:26:13,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.77 | bwd_microstep: 5762.28 | bwd_inner_microstep: 5709.07 | bwd_allreduce_microstep: 53.17 | step_microstep: 18.32
[2025-04-26 11:26:13,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.77 | bwd: 5762.29 | bwd_inner: 5709.07 | bwd_allreduce: 53.19 | step: 18.32

  7%|▋         | 3023/41250 [7:17:44<92:10:09,  8.68s/it]
                                                         
{'loss': 0.228, 'grad_norm': 1.5900182723999023, 'learning_rate': 3.980389680672044e-05, 'epoch': 0.73}

  7%|▋         | 3023/41250 [7:17:44<92:10:09,  8.68s/it][2025-04-26 11:26:22,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.11 | optimizer_step: 1.16
[2025-04-26 11:26:22,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.69 | bwd_microstep: 5777.40 | bwd_inner_microstep: 5700.21 | bwd_allreduce_microstep: 77.13 | step_microstep: 19.45
[2025-04-26 11:26:22,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.70 | bwd: 5777.42 | bwd_inner: 5700.21 | bwd_allreduce: 77.15 | step: 19.46

  7%|▋         | 3024/41250 [7:17:53<92:16:01,  8.69s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 2.1455929279327393, 'learning_rate': 3.980367738198798e-05, 'epoch': 0.73}

  7%|▋         | 3024/41250 [7:17:53<92:16:01,  8.69s/it][2025-04-26 11:26:31,363] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:26:31,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.57 | bwd_microstep: 5936.34 | bwd_inner_microstep: 5690.50 | bwd_allreduce_microstep: 245.79 | step_microstep: 18.49
[2025-04-26 11:26:31,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.57 | bwd: 5936.35 | bwd_inner: 5690.50 | bwd_allreduce: 245.81 | step: 18.49

  7%|▋         | 3025/41250 [7:18:02<92:50:08,  8.74s/it]
                                                         
{'loss': 0.105, 'grad_norm': 0.9759372472763062, 'learning_rate': 3.980345783516975e-05, 'epoch': 0.73}

  7%|▋         | 3025/41250 [7:18:02<92:50:08,  8.74s/it][2025-04-26 11:26:40,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:26:40,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.18 | bwd_microstep: 5731.31 | bwd_inner_microstep: 5718.65 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.53
[2025-04-26 11:26:40,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.18 | bwd: 5731.33 | bwd_inner: 5718.65 | bwd_allreduce: 12.64 | step: 18.54

  7%|▋         | 3026/41250 [7:18:11<92:34:43,  8.72s/it]
                                                         
{'loss': 0.2126, 'grad_norm': 1.3767178058624268, 'learning_rate': 3.980323816626709e-05, 'epoch': 0.73}

  7%|▋         | 3026/41250 [7:18:11<92:34:43,  8.72s/it][2025-04-26 11:26:48,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 11:26:48,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.25 | bwd_microstep: 5707.15 | bwd_inner_microstep: 5694.44 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.47
[2025-04-26 11:26:48,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.25 | bwd: 5707.17 | bwd_inner: 5694.44 | bwd_allreduce: 12.68 | step: 18.47

  7%|▋         | 3027/41250 [7:18:19<92:17:47,  8.69s/it]
                                                         
{'loss': 0.2154, 'grad_norm': 2.507425546646118, 'learning_rate': 3.980301837528137e-05, 'epoch': 0.73}

  7%|▋         | 3027/41250 [7:18:19<92:17:47,  8.69s/it][2025-04-26 11:26:57,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.95
[2025-04-26 11:26:57,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.25 | bwd_microstep: 5711.91 | bwd_inner_microstep: 5664.41 | bwd_allreduce_microstep: 47.45 | step_microstep: 19.09
[2025-04-26 11:26:57,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.25 | bwd: 5711.92 | bwd_inner: 5664.41 | bwd_allreduce: 47.47 | step: 19.09

  7%|▋         | 3028/41250 [7:18:28<92:05:06,  8.67s/it]
                                                         
{'loss': 0.0835, 'grad_norm': 0.8561985492706299, 'learning_rate': 3.9802798462213925e-05, 'epoch': 0.73}

  7%|▋         | 3028/41250 [7:18:28<92:05:06,  8.67s/it][2025-04-26 11:27:05,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 11:27:05,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.39 | bwd_microstep: 5739.28 | bwd_inner_microstep: 5726.49 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.08
[2025-04-26 11:27:05,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.39 | bwd: 5739.29 | bwd_inner: 5726.49 | bwd_allreduce: 12.76 | step: 19.08

  7%|▋         | 3029/41250 [7:18:37<92:05:31,  8.67s/it]
                                                         
{'loss': 0.2413, 'grad_norm': 1.6968584060668945, 'learning_rate': 3.9802578427066135e-05, 'epoch': 0.73}

  7%|▋         | 3029/41250 [7:18:37<92:05:31,  8.67s/it][2025-04-26 11:27:14,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 11:27:14,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.48 | bwd_microstep: 5745.29 | bwd_inner_microstep: 5708.60 | bwd_allreduce_microstep: 36.64 | step_microstep: 18.86
[2025-04-26 11:27:14,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.48 | bwd: 5745.30 | bwd_inner: 5708.60 | bwd_allreduce: 36.66 | step: 18.86

  7%|▋         | 3030/41250 [7:18:45<92:05:56,  8.67s/it]
                                                         
{'loss': 0.0592, 'grad_norm': 0.5599287152290344, 'learning_rate': 3.980235826983933e-05, 'epoch': 0.73}

  7%|▋         | 3030/41250 [7:18:45<92:05:56,  8.67s/it][2025-04-26 11:27:23,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:27:23,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.77 | bwd_microstep: 5772.23 | bwd_inner_microstep: 5658.15 | bwd_allreduce_microstep: 114.03 | step_microstep: 18.69
[2025-04-26 11:27:23,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.77 | bwd: 5772.24 | bwd_inner: 5658.15 | bwd_allreduce: 114.05 | step: 18.69

  7%|▋         | 3031/41250 [7:18:54<92:09:30,  8.68s/it]
                                                         
{'loss': 0.1616, 'grad_norm': 1.416067123413086, 'learning_rate': 3.980213799053489e-05, 'epoch': 0.73}

  7%|▋         | 3031/41250 [7:18:54<92:09:30,  8.68s/it][2025-04-26 11:27:31,964] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-26 11:27:31,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.39 | bwd_microstep: 5714.11 | bwd_inner_microstep: 5659.42 | bwd_allreduce_microstep: 54.62 | step_microstep: 19.78
[2025-04-26 11:27:31,965] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.39 | bwd: 5714.13 | bwd_inner: 5659.42 | bwd_allreduce: 54.65 | step: 19.79

  7%|▋         | 3032/41250 [7:19:03<92:00:51,  8.67s/it]
                                                         
{'loss': 0.12, 'grad_norm': 1.0993478298187256, 'learning_rate': 3.980191758915416e-05, 'epoch': 0.74}

  7%|▋         | 3032/41250 [7:19:03<92:00:51,  8.67s/it][2025-04-26 11:27:40,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 11:27:40,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.61 | bwd_microstep: 5726.05 | bwd_inner_microstep: 5713.30 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.91
[2025-04-26 11:27:40,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.61 | bwd: 5726.06 | bwd_inner: 5713.30 | bwd_allreduce: 12.72 | step: 18.91

  7%|▋         | 3033/41250 [7:19:11<91:59:54,  8.67s/it]
                                                         
{'loss': 0.183, 'grad_norm': 1.6513828039169312, 'learning_rate': 3.980169706569851e-05, 'epoch': 0.74}

  7%|▋         | 3033/41250 [7:19:11<91:59:54,  8.67s/it][2025-04-26 11:27:49,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.28 | optimizer_step: 0.93
[2025-04-26 11:27:49,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.45 | bwd_microstep: 5772.27 | bwd_inner_microstep: 5688.09 | bwd_allreduce_microstep: 84.12 | step_microstep: 19.33
[2025-04-26 11:27:49,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.45 | bwd: 5772.28 | bwd_inner: 5688.09 | bwd_allreduce: 84.14 | step: 19.33

  7%|▋         | 3034/41250 [7:19:20<92:05:51,  8.68s/it]
                                                         
{'loss': 0.2208, 'grad_norm': 2.2275664806365967, 'learning_rate': 3.980147642016929e-05, 'epoch': 0.74}

  7%|▋         | 3034/41250 [7:19:20<92:05:51,  8.68s/it][2025-04-26 11:27:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 11:27:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5746.15 | bwd_inner_microstep: 5712.15 | bwd_allreduce_microstep: 33.94 | step_microstep: 18.83
[2025-04-26 11:27:57,997] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5746.16 | bwd_inner: 5712.15 | bwd_allreduce: 33.97 | step: 18.83

  7%|▋         | 3035/41250 [7:19:29<92:03:57,  8.67s/it]
                                                         
{'loss': 0.245, 'grad_norm': 1.5317625999450684, 'learning_rate': 3.9801255652567855e-05, 'epoch': 0.74}

  7%|▋         | 3035/41250 [7:19:29<92:03:57,  8.67s/it][2025-04-26 11:28:06,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:28:06,701] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.23 | bwd_microstep: 5795.97 | bwd_inner_microstep: 5650.93 | bwd_allreduce_microstep: 145.00 | step_microstep: 18.71
[2025-04-26 11:28:06,702] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.23 | bwd: 5795.99 | bwd_inner: 5650.93 | bwd_allreduce: 145.01 | step: 18.71

  7%|▋         | 3036/41250 [7:19:37<92:09:50,  8.68s/it]
                                                         
{'loss': 0.0352, 'grad_norm': 0.48497578501701355, 'learning_rate': 3.980103476289558e-05, 'epoch': 0.74}

  7%|▋         | 3036/41250 [7:19:37<92:09:50,  8.68s/it][2025-04-26 11:28:15,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 11:28:15,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.11 | bwd_microstep: 5724.95 | bwd_inner_microstep: 5640.79 | bwd_allreduce_microstep: 84.11 | step_microstep: 18.70
[2025-04-26 11:28:15,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.11 | bwd: 5724.96 | bwd_inner: 5640.79 | bwd_allreduce: 84.12 | step: 18.70

  7%|▋         | 3037/41250 [7:19:46<91:59:10,  8.67s/it]
                                                         
{'loss': 0.1642, 'grad_norm': 0.7249960899353027, 'learning_rate': 3.980081375115382e-05, 'epoch': 0.74}

  7%|▋         | 3037/41250 [7:19:46<91:59:10,  8.67s/it][2025-04-26 11:28:23,929] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 11:28:23,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.65 | bwd_microstep: 5696.51 | bwd_inner_microstep: 5647.64 | bwd_allreduce_microstep: 48.82 | step_microstep: 18.55
[2025-04-26 11:28:23,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.65 | bwd: 5696.53 | bwd_inner: 5647.64 | bwd_allreduce: 48.84 | step: 18.55

  7%|▋         | 3038/41250 [7:19:55<91:46:42,  8.65s/it]
                                                         
{'loss': 0.2419, 'grad_norm': 2.7981653213500977, 'learning_rate': 3.9800592617343936e-05, 'epoch': 0.74}

  7%|▋         | 3038/41250 [7:19:55<91:46:42,  8.65s/it][2025-04-26 11:28:32,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 11:28:32,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.46 | bwd_microstep: 5764.43 | bwd_inner_microstep: 5646.15 | bwd_allreduce_microstep: 118.23 | step_microstep: 18.87
[2025-04-26 11:28:32,597] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.46 | bwd: 5764.44 | bwd_inner: 5646.15 | bwd_allreduce: 118.25 | step: 18.87

  7%|▋         | 3039/41250 [7:20:03<91:50:23,  8.65s/it]
                                                         
{'loss': 0.2168, 'grad_norm': 1.759712815284729, 'learning_rate': 3.98003713614673e-05, 'epoch': 0.74}

  7%|▋         | 3039/41250 [7:20:03<91:50:23,  8.65s/it][2025-04-26 11:28:41,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 11:28:41,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.46 | bwd_microstep: 5757.22 | bwd_inner_microstep: 5646.60 | bwd_allreduce_microstep: 110.58 | step_microstep: 19.03
[2025-04-26 11:28:41,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.46 | bwd: 5757.24 | bwd_inner: 5646.60 | bwd_allreduce: 110.60 | step: 19.04

  7%|▋         | 3040/41250 [7:20:12<91:51:48,  8.66s/it]
                                                         
{'loss': 0.0579, 'grad_norm': 1.6287219524383545, 'learning_rate': 3.980014998352526e-05, 'epoch': 0.74}

  7%|▋         | 3040/41250 [7:20:12<91:51:48,  8.66s/it][2025-04-26 11:28:49,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:28:49,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.14 | bwd_microstep: 5700.15 | bwd_inner_microstep: 5655.17 | bwd_allreduce_microstep: 44.94 | step_microstep: 18.53
[2025-04-26 11:28:49,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.14 | bwd: 5700.17 | bwd_inner: 5655.17 | bwd_allreduce: 44.95 | step: 18.53

  7%|▋         | 3041/41250 [7:20:21<91:44:37,  8.64s/it]
                                                         
{'loss': 0.2898, 'grad_norm': 1.552650809288025, 'learning_rate': 3.979992848351919e-05, 'epoch': 0.74}

  7%|▋         | 3041/41250 [7:20:21<91:44:37,  8.64s/it][2025-04-26 11:28:58,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:28:58,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.87 | bwd_microstep: 5774.61 | bwd_inner_microstep: 5641.60 | bwd_allreduce_microstep: 132.96 | step_microstep: 18.78
[2025-04-26 11:28:58,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.87 | bwd: 5774.62 | bwd_inner: 5641.60 | bwd_allreduce: 132.98 | step: 18.78

  7%|▋         | 3042/41250 [7:20:29<91:51:07,  8.65s/it]
                                                         
{'loss': 0.2041, 'grad_norm': 1.450358510017395, 'learning_rate': 3.979970686145046e-05, 'epoch': 0.74}

  7%|▋         | 3042/41250 [7:20:29<91:51:07,  8.65s/it][2025-04-26 11:29:07,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:29:07,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.13 | bwd_microstep: 5758.08 | bwd_inner_microstep: 5688.17 | bwd_allreduce_microstep: 69.87 | step_microstep: 18.66
[2025-04-26 11:29:07,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.14 | bwd: 5758.10 | bwd_inner: 5688.17 | bwd_allreduce: 69.89 | step: 18.66

  7%|▋         | 3043/41250 [7:20:38<91:55:52,  8.66s/it]
                                                         
{'loss': 0.2958, 'grad_norm': 4.061823844909668, 'learning_rate': 3.979948511732043e-05, 'epoch': 0.74}

  7%|▋         | 3043/41250 [7:20:38<91:55:52,  8.66s/it][2025-04-26 11:29:15,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:29:15,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.60 | bwd_microstep: 5779.39 | bwd_inner_microstep: 5644.35 | bwd_allreduce_microstep: 134.99 | step_microstep: 18.52
[2025-04-26 11:29:15,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.60 | bwd: 5779.40 | bwd_inner: 5644.35 | bwd_allreduce: 135.01 | step: 18.52

  7%|▋         | 3044/41250 [7:20:47<91:57:44,  8.67s/it]
                                                         
{'loss': 0.1488, 'grad_norm': 0.8574129343032837, 'learning_rate': 3.979926325113047e-05, 'epoch': 0.74}

  7%|▋         | 3044/41250 [7:20:47<91:57:44,  8.67s/it][2025-04-26 11:29:24,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 11:29:24,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.37 | bwd_microstep: 5686.01 | bwd_inner_microstep: 5651.37 | bwd_allreduce_microstep: 34.60 | step_microstep: 19.00
[2025-04-26 11:29:24,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.37 | bwd: 5686.03 | bwd_inner: 5651.37 | bwd_allreduce: 34.62 | step: 19.00

  7%|▋         | 3045/41250 [7:20:55<91:43:26,  8.64s/it]
                                                         
{'loss': 0.0445, 'grad_norm': 0.5257189273834229, 'learning_rate': 3.979904126288194e-05, 'epoch': 0.74}

  7%|▋         | 3045/41250 [7:20:55<91:43:26,  8.64s/it][2025-04-26 11:29:33,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 1.12
[2025-04-26 11:29:33,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.82 | bwd_microstep: 5739.99 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 45.91 | step_microstep: 19.19
[2025-04-26 11:29:33,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.82 | bwd: 5740.00 | bwd_inner: 5694.02 | bwd_allreduce: 45.94 | step: 19.21

  7%|▋         | 3046/41250 [7:21:04<91:48:01,  8.65s/it]
                                                         
{'loss': 0.2377, 'grad_norm': 2.8529000282287598, 'learning_rate': 3.979881915257622e-05, 'epoch': 0.74}

  7%|▋         | 3046/41250 [7:21:04<91:48:01,  8.65s/it][2025-04-26 11:29:41,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:29:41,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.92 | bwd_microstep: 5699.33 | bwd_inner_microstep: 5686.25 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.64
[2025-04-26 11:29:41,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.91 | bwd: 5699.34 | bwd_inner: 5686.25 | bwd_allreduce: 13.05 | step: 18.64

  7%|▋         | 3047/41250 [7:21:12<91:42:39,  8.64s/it]
                                                         
{'loss': 0.0893, 'grad_norm': 0.9909592866897583, 'learning_rate': 3.979859692021468e-05, 'epoch': 0.74}

  7%|▋         | 3047/41250 [7:21:12<91:42:39,  8.64s/it][2025-04-26 11:29:50,451] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:29:50,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.99 | bwd_microstep: 5721.45 | bwd_inner_microstep: 5708.19 | bwd_allreduce_microstep: 13.22 | step_microstep: 18.60
[2025-04-26 11:29:50,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.99 | bwd: 5721.46 | bwd_inner: 5708.19 | bwd_allreduce: 13.23 | step: 18.60

  7%|▋         | 3048/41250 [7:21:21<91:46:19,  8.65s/it]
                                                         
{'loss': 0.1387, 'grad_norm': 1.7868287563323975, 'learning_rate': 3.979837456579867e-05, 'epoch': 0.74}

  7%|▋         | 3048/41250 [7:21:21<91:46:19,  8.65s/it][2025-04-26 11:29:59,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:29:59,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.98 | bwd_microstep: 5708.02 | bwd_inner_microstep: 5651.10 | bwd_allreduce_microstep: 56.87 | step_microstep: 18.73
[2025-04-26 11:29:59,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.98 | bwd: 5708.03 | bwd_inner: 5651.10 | bwd_allreduce: 56.89 | step: 18.73

  7%|▋         | 3049/41250 [7:21:30<91:38:33,  8.64s/it]
                                                         
{'loss': 0.0818, 'grad_norm': 0.6878287196159363, 'learning_rate': 3.979815208932959e-05, 'epoch': 0.74}

  7%|▋         | 3049/41250 [7:21:30<91:38:33,  8.64s/it][2025-04-26 11:30:07,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 11:30:07,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.81 | bwd_microstep: 5875.82 | bwd_inner_microstep: 5697.19 | bwd_allreduce_microstep: 178.57 | step_microstep: 19.34
[2025-04-26 11:30:07,863] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.81 | bwd: 5875.83 | bwd_inner: 5697.19 | bwd_allreduce: 178.59 | step: 19.34

  7%|▋         | 3050/41250 [7:21:39<92:10:30,  8.69s/it]
                                                         
{'loss': 0.2916, 'grad_norm': 1.5488001108169556, 'learning_rate': 3.979792949080879e-05, 'epoch': 0.74}

  7%|▋         | 3050/41250 [7:21:39<92:10:30,  8.69s/it][2025-04-26 11:30:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:30:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.77 | bwd_microstep: 5691.91 | bwd_inner_microstep: 5679.15 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.61
[2025-04-26 11:30:16,486] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.77 | bwd: 5691.92 | bwd_inner: 5679.15 | bwd_allreduce: 12.74 | step: 18.61

  7%|▋         | 3051/41250 [7:21:47<91:58:06,  8.67s/it]
                                                         
{'loss': 0.2613, 'grad_norm': 1.4076367616653442, 'learning_rate': 3.979770677023765e-05, 'epoch': 0.74}

  7%|▋         | 3051/41250 [7:21:47<91:58:06,  8.67s/it][2025-04-26 11:30:25,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:30:25,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.44 | bwd_microstep: 5747.47 | bwd_inner_microstep: 5693.21 | bwd_allreduce_microstep: 54.22 | step_microstep: 18.54
[2025-04-26 11:30:25,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.44 | bwd: 5747.49 | bwd_inner: 5693.21 | bwd_allreduce: 54.24 | step: 18.55

  7%|▋         | 3052/41250 [7:21:56<91:59:32,  8.67s/it]
                                                         
{'loss': 0.0709, 'grad_norm': 0.6492601633071899, 'learning_rate': 3.979748392761754e-05, 'epoch': 0.74}

  7%|▋         | 3052/41250 [7:21:56<91:59:32,  8.67s/it][2025-04-26 11:30:33,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:30:33,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.01 | bwd_microstep: 5699.90 | bwd_inner_microstep: 5644.54 | bwd_allreduce_microstep: 55.31 | step_microstep: 18.52
[2025-04-26 11:30:33,754] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.01 | bwd: 5699.92 | bwd_inner: 5644.54 | bwd_allreduce: 55.33 | step: 18.53

  7%|▋         | 3053/41250 [7:22:04<91:44:27,  8.65s/it]
                                                         
{'loss': 0.1941, 'grad_norm': 1.1763131618499756, 'learning_rate': 3.9797260962949836e-05, 'epoch': 0.74}

  7%|▋         | 3053/41250 [7:22:04<91:44:27,  8.65s/it][2025-04-26 11:30:42,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.75 | optimizer_gradients: 1.18 | optimizer_step: 0.95
[2025-04-26 11:30:42,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.30 | bwd_microstep: 5712.91 | bwd_inner_microstep: 5699.98 | bwd_allreduce_microstep: 12.88 | step_microstep: 20.34
[2025-04-26 11:30:42,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.30 | bwd: 5712.93 | bwd_inner: 5699.98 | bwd_allreduce: 12.91 | step: 20.34

  7%|▋         | 3054/41250 [7:22:13<91:45:38,  8.65s/it]
                                                         
{'loss': 0.3441, 'grad_norm': 1.6164573431015015, 'learning_rate': 3.9797037876235906e-05, 'epoch': 0.74}

  7%|▋         | 3054/41250 [7:22:13<91:45:38,  8.65s/it][2025-04-26 11:30:51,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 11:30:51,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.04 | bwd_microstep: 5714.81 | bwd_inner_microstep: 5701.66 | bwd_allreduce_microstep: 13.10 | step_microstep: 19.03
[2025-04-26 11:30:51,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.04 | bwd: 5714.82 | bwd_inner: 5701.66 | bwd_allreduce: 13.12 | step: 19.03

  7%|▋         | 3055/41250 [7:22:22<91:44:38,  8.65s/it]
                                                         
{'loss': 0.2477, 'grad_norm': 1.2675220966339111, 'learning_rate': 3.979681466747714e-05, 'epoch': 0.74}

  7%|▋         | 3055/41250 [7:22:22<91:44:38,  8.65s/it][2025-04-26 11:30:59,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 11:30:59,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.90 | bwd_microstep: 5738.74 | bwd_inner_microstep: 5710.95 | bwd_allreduce_microstep: 27.74 | step_microstep: 18.56
[2025-04-26 11:30:59,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.90 | bwd: 5738.75 | bwd_inner: 5710.95 | bwd_allreduce: 27.76 | step: 18.56

  7%|▋         | 3056/41250 [7:22:30<91:49:58,  8.66s/it]
                                                         
{'loss': 0.085, 'grad_norm': 0.8373575210571289, 'learning_rate': 3.97965913366749e-05, 'epoch': 0.74}

  7%|▋         | 3056/41250 [7:22:30<91:49:58,  8.66s/it][2025-04-26 11:31:08,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[2025-04-26 11:31:08,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.16 | bwd_microstep: 5762.12 | bwd_inner_microstep: 5704.28 | bwd_allreduce_microstep: 57.80 | step_microstep: 18.87
[2025-04-26 11:31:08,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.16 | bwd: 5762.14 | bwd_inner: 5704.28 | bwd_allreduce: 57.82 | step: 18.87

  7%|▋         | 3057/41250 [7:22:39<91:56:35,  8.67s/it]
                                                         
{'loss': 0.2255, 'grad_norm': 1.1589821577072144, 'learning_rate': 3.979636788383057e-05, 'epoch': 0.74}

  7%|▋         | 3057/41250 [7:22:39<91:56:35,  8.67s/it][2025-04-26 11:31:17,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-26 11:31:17,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.64 | bwd_microstep: 5782.79 | bwd_inner_microstep: 5697.24 | bwd_allreduce_microstep: 85.50 | step_microstep: 18.79
[2025-04-26 11:31:17,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.64 | bwd: 5782.81 | bwd_inner: 5697.24 | bwd_allreduce: 85.52 | step: 18.79

  7%|▋         | 3058/41250 [7:22:48<92:04:12,  8.68s/it]
                                                         
{'loss': 0.1378, 'grad_norm': 1.4478081464767456, 'learning_rate': 3.979614430894553e-05, 'epoch': 0.74}

  7%|▋         | 3058/41250 [7:22:48<92:04:12,  8.68s/it][2025-04-26 11:31:25,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:31:25,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.63 | bwd_microstep: 5723.68 | bwd_inner_microstep: 5710.66 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.53
[2025-04-26 11:31:25,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.63 | bwd: 5723.69 | bwd_inner: 5710.66 | bwd_allreduce: 13.00 | step: 18.54

  7%|▋         | 3059/41250 [7:22:56<92:00:09,  8.67s/it]
                                                         
{'loss': 0.1052, 'grad_norm': 1.1896754503250122, 'learning_rate': 3.979592061202115e-05, 'epoch': 0.74}

  7%|▋         | 3059/41250 [7:22:56<92:00:09,  8.67s/it][2025-04-26 11:31:34,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:31:34,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.40 | bwd_microstep: 5773.86 | bwd_inner_microstep: 5697.75 | bwd_allreduce_microstep: 76.06 | step_microstep: 18.64
[2025-04-26 11:31:34,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.40 | bwd: 5773.87 | bwd_inner: 5697.75 | bwd_allreduce: 76.08 | step: 18.64

  7%|▋         | 3060/41250 [7:23:05<92:06:27,  8.68s/it]
                                                         
{'loss': 0.0703, 'grad_norm': 0.6989370584487915, 'learning_rate': 3.979569679305881e-05, 'epoch': 0.74}

  7%|▋         | 3060/41250 [7:23:05<92:06:27,  8.68s/it][2025-04-26 11:31:43,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 11:31:43,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.40 | bwd_microstep: 5716.13 | bwd_inner_microstep: 5691.48 | bwd_allreduce_microstep: 24.61 | step_microstep: 18.51
[2025-04-26 11:31:43,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.41 | bwd: 5716.15 | bwd_inner: 5691.48 | bwd_allreduce: 24.63 | step: 18.51

  7%|▋         | 3061/41250 [7:23:14<91:59:12,  8.67s/it]
                                                         
{'loss': 0.096, 'grad_norm': 0.8939603567123413, 'learning_rate': 3.97954728520599e-05, 'epoch': 0.74}

  7%|▋         | 3061/41250 [7:23:14<91:59:12,  8.67s/it][2025-04-26 11:31:51,774] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 11:31:51,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.81 | bwd_microstep: 5717.62 | bwd_inner_microstep: 5682.81 | bwd_allreduce_microstep: 34.77 | step_microstep: 17.70
[2025-04-26 11:31:51,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.81 | bwd: 5717.63 | bwd_inner: 5682.81 | bwd_allreduce: 34.79 | step: 17.70

  7%|▋         | 3062/41250 [7:23:22<91:53:03,  8.66s/it]
                                                         
{'loss': 0.1655, 'grad_norm': 0.7968366742134094, 'learning_rate': 3.9795248789025786e-05, 'epoch': 0.74}

  7%|▋         | 3062/41250 [7:23:22<91:53:03,  8.66s/it][2025-04-26 11:32:00,460] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:32:00,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.38 | bwd_microstep: 5745.06 | bwd_inner_microstep: 5711.98 | bwd_allreduce_microstep: 33.03 | step_microstep: 18.50
[2025-04-26 11:32:00,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.38 | bwd: 5745.07 | bwd_inner: 5711.98 | bwd_allreduce: 33.05 | step: 18.50

  7%|▋         | 3063/41250 [7:23:31<91:57:20,  8.67s/it]
                                                         
{'loss': 0.1228, 'grad_norm': 0.9133422374725342, 'learning_rate': 3.9795024603957856e-05, 'epoch': 0.74}

  7%|▋         | 3063/41250 [7:23:31<91:57:20,  8.67s/it][2025-04-26 11:32:09,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.00
[2025-04-26 11:32:09,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.40 | bwd_microstep: 5730.18 | bwd_inner_microstep: 5660.88 | bwd_allreduce_microstep: 69.25 | step_microstep: 19.03
[2025-04-26 11:32:09,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.40 | bwd: 5730.20 | bwd_inner: 5660.88 | bwd_allreduce: 69.27 | step: 19.03

  7%|▋         | 3064/41250 [7:23:40<91:52:11,  8.66s/it]
                                                         
{'loss': 0.2791, 'grad_norm': 2.323957920074463, 'learning_rate': 3.97948002968575e-05, 'epoch': 0.74}

  7%|▋         | 3064/41250 [7:23:40<91:52:11,  8.66s/it][2025-04-26 11:32:17,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:32:17,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.24 | bwd_microstep: 5724.92 | bwd_inner_microstep: 5712.19 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.56
[2025-04-26 11:32:17,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.24 | bwd: 5724.94 | bwd_inner: 5712.19 | bwd_allreduce: 12.71 | step: 18.56

  7%|▋         | 3065/41250 [7:23:48<91:51:36,  8.66s/it]
                                                         
{'loss': 0.0949, 'grad_norm': 1.2563563585281372, 'learning_rate': 3.9794575867726096e-05, 'epoch': 0.74}

  7%|▋         | 3065/41250 [7:23:48<91:51:36,  8.66s/it][2025-04-26 11:32:26,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 11:32:26,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.73 | bwd_microstep: 5766.23 | bwd_inner_microstep: 5717.54 | bwd_allreduce_microstep: 48.64 | step_microstep: 18.83
[2025-04-26 11:32:26,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.73 | bwd: 5766.25 | bwd_inner: 5717.54 | bwd_allreduce: 48.66 | step: 18.83

  7%|▋         | 3066/41250 [7:23:57<92:01:15,  8.68s/it]
                                                         
{'loss': 0.3519, 'grad_norm': 1.6796784400939941, 'learning_rate': 3.979435131656502e-05, 'epoch': 0.74}

  7%|▋         | 3066/41250 [7:23:57<92:01:15,  8.68s/it][2025-04-26 11:32:35,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.28 | optimizer_step: 0.93
[2025-04-26 11:32:35,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.11 | bwd_microstep: 5780.83 | bwd_inner_microstep: 5663.31 | bwd_allreduce_microstep: 117.46 | step_microstep: 19.56
[2025-04-26 11:32:35,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.11 | bwd: 5780.84 | bwd_inner: 5663.31 | bwd_allreduce: 117.49 | step: 19.56

  7%|▋         | 3067/41250 [7:24:06<92:04:04,  8.68s/it]
                                                         
{'loss': 0.077, 'grad_norm': 0.644379734992981, 'learning_rate': 3.9794126643375656e-05, 'epoch': 0.74}

  7%|▋         | 3067/41250 [7:24:06<92:04:04,  8.68s/it][2025-04-26 11:32:43,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:32:43,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.61 | bwd_microstep: 5785.47 | bwd_inner_microstep: 5698.82 | bwd_allreduce_microstep: 86.60 | step_microstep: 18.57
[2025-04-26 11:32:43,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.61 | bwd: 5785.49 | bwd_inner: 5698.82 | bwd_allreduce: 86.62 | step: 18.57

  7%|▋         | 3068/41250 [7:24:15<92:10:19,  8.69s/it]
                                                         
{'loss': 0.0964, 'grad_norm': 0.9571952819824219, 'learning_rate': 3.97939018481594e-05, 'epoch': 0.74}

  7%|▋         | 3068/41250 [7:24:15<92:10:19,  8.69s/it][2025-04-26 11:32:52,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:32:52,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.31 | bwd_microstep: 5766.01 | bwd_inner_microstep: 5657.59 | bwd_allreduce_microstep: 108.37 | step_microstep: 18.37
[2025-04-26 11:32:52,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.31 | bwd: 5766.02 | bwd_inner: 5657.59 | bwd_allreduce: 108.38 | step: 18.37

  7%|▋         | 3069/41250 [7:24:23<92:08:16,  8.69s/it]
                                                         
{'loss': 0.3828, 'grad_norm': 2.26393723487854, 'learning_rate': 3.9793676930917635e-05, 'epoch': 0.74}

  7%|▋         | 3069/41250 [7:24:23<92:08:16,  8.69s/it][2025-04-26 11:33:01,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:33:01,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5786.23 | bwd_inner_microstep: 5702.40 | bwd_allreduce_microstep: 83.79 | step_microstep: 18.63
[2025-04-26 11:33:01,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5786.25 | bwd_inner: 5702.39 | bwd_allreduce: 83.81 | step: 18.63

  7%|▋         | 3070/41250 [7:24:32<92:13:48,  8.70s/it]
                                                         
{'loss': 0.2165, 'grad_norm': 1.4206370115280151, 'learning_rate': 3.979345189165174e-05, 'epoch': 0.74}

  7%|▋         | 3070/41250 [7:24:32<92:13:48,  8.70s/it][2025-04-26 11:33:09,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:33:09,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.23 | bwd_microstep: 5704.69 | bwd_inner_microstep: 5691.70 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.72
[2025-04-26 11:33:09,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.23 | bwd: 5704.70 | bwd_inner: 5691.70 | bwd_allreduce: 12.96 | step: 18.72

  7%|▋         | 3071/41250 [7:24:41<91:58:47,  8.67s/it]
                                                         
{'loss': 0.1182, 'grad_norm': 1.2395117282867432, 'learning_rate': 3.979322673036311e-05, 'epoch': 0.74}

  7%|▋         | 3071/41250 [7:24:41<91:58:47,  8.67s/it][2025-04-26 11:33:18,715] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 1.08 | optimizer_step: 1.12
[2025-04-26 11:33:18,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.47 | bwd_microstep: 5906.84 | bwd_inner_microstep: 5650.97 | bwd_allreduce_microstep: 255.82 | step_microstep: 20.04
[2025-04-26 11:33:18,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.47 | bwd: 5906.86 | bwd_inner: 5650.97 | bwd_allreduce: 255.84 | step: 20.04

  7%|▋         | 3072/41250 [7:24:49<92:27:21,  8.72s/it]
                                                         
{'loss': 0.1053, 'grad_norm': 0.8783794045448303, 'learning_rate': 3.979300144705313e-05, 'epoch': 0.74}

  7%|▋         | 3072/41250 [7:24:49<92:27:21,  8.72s/it][2025-04-26 11:33:27,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-26 11:33:27,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.09 | bwd_microstep: 5794.18 | bwd_inner_microstep: 5657.33 | bwd_allreduce_microstep: 136.80 | step_microstep: 19.17
[2025-04-26 11:33:27,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.09 | bwd: 5794.19 | bwd_inner: 5657.33 | bwd_allreduce: 136.82 | step: 19.18

  7%|▋         | 3073/41250 [7:24:58<92:26:08,  8.72s/it]
                                                         
{'loss': 0.1496, 'grad_norm': 1.4228144884109497, 'learning_rate': 3.9792776041723194e-05, 'epoch': 0.74}

  7%|▋         | 3073/41250 [7:24:58<92:26:08,  8.72s/it][2025-04-26 11:33:36,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:33:36,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.14 | bwd_microstep: 5763.97 | bwd_inner_microstep: 5704.12 | bwd_allreduce_microstep: 59.80 | step_microstep: 19.06
[2025-04-26 11:33:36,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.14 | bwd: 5763.98 | bwd_inner: 5704.12 | bwd_allreduce: 59.82 | step: 19.06

  7%|▋         | 3074/41250 [7:25:07<92:23:05,  8.71s/it]
                                                         
{'loss': 0.0983, 'grad_norm': 0.7883129715919495, 'learning_rate': 3.979255051437468e-05, 'epoch': 0.75}

  7%|▋         | 3074/41250 [7:25:07<92:23:05,  8.71s/it][2025-04-26 11:33:44,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:33:44,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5748.98 | bwd_inner_microstep: 5708.54 | bwd_allreduce_microstep: 40.39 | step_microstep: 18.73
[2025-04-26 11:33:44,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5748.99 | bwd_inner: 5708.54 | bwd_allreduce: 40.41 | step: 18.73

  7%|▋         | 3075/41250 [7:25:15<92:17:04,  8.70s/it]
                                                         
{'loss': 0.0579, 'grad_norm': 0.7000722885131836, 'learning_rate': 3.9792324865008986e-05, 'epoch': 0.75}

  7%|▋         | 3075/41250 [7:25:15<92:17:04,  8.70s/it][2025-04-26 11:33:53,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:33:53,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.89 | bwd_microstep: 5753.91 | bwd_inner_microstep: 5706.09 | bwd_allreduce_microstep: 47.77 | step_microstep: 18.93
[2025-04-26 11:33:53,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.89 | bwd: 5753.92 | bwd_inner: 5706.09 | bwd_allreduce: 47.79 | step: 18.94

  7%|▋         | 3076/41250 [7:25:24<92:13:31,  8.70s/it]
                                                         
{'loss': 0.1769, 'grad_norm': 1.2988818883895874, 'learning_rate': 3.9792099093627504e-05, 'epoch': 0.75}

  7%|▋         | 3076/41250 [7:25:24<92:13:31,  8.70s/it][2025-04-26 11:34:02,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.42 | optimizer_step: 1.09
[2025-04-26 11:34:02,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.74 | bwd_microstep: 5744.48 | bwd_inner_microstep: 5686.84 | bwd_allreduce_microstep: 57.56 | step_microstep: 20.54
[2025-04-26 11:34:02,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.74 | bwd: 5744.49 | bwd_inner: 5686.84 | bwd_allreduce: 57.60 | step: 20.55

  7%|▋         | 3077/41250 [7:25:33<92:09:14,  8.69s/it]
                                                         
{'loss': 0.0871, 'grad_norm': 1.3873175382614136, 'learning_rate': 3.979187320023162e-05, 'epoch': 0.75}

  7%|▋         | 3077/41250 [7:25:33<92:09:14,  8.69s/it][2025-04-26 11:34:10,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.58 | optimizer_step: 0.90
[2025-04-26 11:34:10,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.47 | bwd_microstep: 5706.04 | bwd_inner_microstep: 5649.56 | bwd_allreduce_microstep: 56.41 | step_microstep: 21.73
[2025-04-26 11:34:10,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.47 | bwd: 5706.06 | bwd_inner: 5649.56 | bwd_allreduce: 56.44 | step: 21.73

  7%|▋         | 3078/41250 [7:25:41<91:56:43,  8.67s/it]
                                                         
{'loss': 0.1858, 'grad_norm': 2.205437421798706, 'learning_rate': 3.9791647184822734e-05, 'epoch': 0.75}

  7%|▋         | 3078/41250 [7:25:41<91:56:43,  8.67s/it][2025-04-26 11:34:19,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.19 | optimizer_step: 0.93
[2025-04-26 11:34:19,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.71 | bwd_microstep: 5717.13 | bwd_inner_microstep: 5703.94 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.12
[2025-04-26 11:34:19,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.71 | bwd: 5717.14 | bwd_inner: 5703.94 | bwd_allreduce: 13.16 | step: 19.12

  7%|▋         | 3079/41250 [7:25:50<91:51:47,  8.66s/it]
                                                         
{'loss': 0.0922, 'grad_norm': 1.2687311172485352, 'learning_rate': 3.979142104740223e-05, 'epoch': 0.75}

  7%|▋         | 3079/41250 [7:25:50<91:51:47,  8.66s/it][2025-04-26 11:34:28,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.94
[2025-04-26 11:34:28,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.66 | bwd_microstep: 5686.23 | bwd_inner_microstep: 5673.20 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.76
[2025-04-26 11:34:28,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.66 | bwd: 5686.24 | bwd_inner: 5673.20 | bwd_allreduce: 13.00 | step: 18.76

  7%|▋         | 3080/41250 [7:25:59<91:40:21,  8.65s/it]
                                                         
{'loss': 0.1691, 'grad_norm': 2.5837271213531494, 'learning_rate': 3.9791194787971525e-05, 'epoch': 0.75}

  7%|▋         | 3080/41250 [7:25:59<91:40:21,  8.65s/it][2025-04-26 11:34:36,699] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:34:36,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.24 | bwd_microstep: 5717.99 | bwd_inner_microstep: 5704.55 | bwd_allreduce_microstep: 13.39 | step_microstep: 18.69
[2025-04-26 11:34:36,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.24 | bwd: 5718.01 | bwd_inner: 5704.55 | bwd_allreduce: 13.41 | step: 18.69

  7%|▋         | 3081/41250 [7:26:07<91:40:52,  8.65s/it]
                                                         
{'loss': 0.1503, 'grad_norm': 1.7217963933944702, 'learning_rate': 3.9790968406531984e-05, 'epoch': 0.75}

  7%|▋         | 3081/41250 [7:26:07<91:40:52,  8.65s/it][2025-04-26 11:34:45,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:34:45,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.47 | bwd_microstep: 5722.99 | bwd_inner_microstep: 5706.45 | bwd_allreduce_microstep: 16.50 | step_microstep: 18.57
[2025-04-26 11:34:45,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.47 | bwd: 5723.00 | bwd_inner: 5706.45 | bwd_allreduce: 16.51 | step: 18.57

  7%|▋         | 3082/41250 [7:26:16<91:42:43,  8.65s/it]
                                                         
{'loss': 0.0478, 'grad_norm': 0.627824068069458, 'learning_rate': 3.979074190308502e-05, 'epoch': 0.75}

  7%|▋         | 3082/41250 [7:26:16<91:42:43,  8.65s/it][2025-04-26 11:34:53,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.32 | optimizer_step: 1.06
[2025-04-26 11:34:53,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.72 | bwd_microstep: 5704.70 | bwd_inner_microstep: 5652.74 | bwd_allreduce_microstep: 51.89 | step_microstep: 20.01
[2025-04-26 11:34:53,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.72 | bwd: 5704.71 | bwd_inner: 5652.74 | bwd_allreduce: 51.92 | step: 20.01

  7%|▋         | 3083/41250 [7:26:25<91:36:40,  8.64s/it]
                                                         
{'loss': 0.212, 'grad_norm': 1.9206708669662476, 'learning_rate': 3.9790515277632015e-05, 'epoch': 0.75}

  7%|▋         | 3083/41250 [7:26:25<91:36:40,  8.64s/it][2025-04-26 11:35:02,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.25 | optimizer_step: 0.93
[2025-04-26 11:35:02,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.27 | bwd_microstep: 5739.79 | bwd_inner_microstep: 5698.76 | bwd_allreduce_microstep: 40.98 | step_microstep: 19.45
[2025-04-26 11:35:02,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.27 | bwd: 5739.81 | bwd_inner: 5698.76 | bwd_allreduce: 41.00 | step: 19.45

  7%|▋         | 3084/41250 [7:26:33<91:42:09,  8.65s/it]
                                                         
{'loss': 0.0542, 'grad_norm': 0.6496943831443787, 'learning_rate': 3.979028853017438e-05, 'epoch': 0.75}

  7%|▋         | 3084/41250 [7:26:33<91:42:09,  8.65s/it][2025-04-26 11:35:11,389] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:35:11,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.55 | bwd_microstep: 5844.15 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 193.95 | step_microstep: 18.76
[2025-04-26 11:35:11,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.55 | bwd: 5844.16 | bwd_inner: 5650.14 | bwd_allreduce: 193.98 | step: 18.77

  7%|▋         | 3085/41250 [7:26:42<91:59:47,  8.68s/it]
                                                         
{'loss': 0.1447, 'grad_norm': 1.7698031663894653, 'learning_rate': 3.979006166071351e-05, 'epoch': 0.75}

  7%|▋         | 3085/41250 [7:26:42<91:59:47,  8.68s/it][2025-04-26 11:35:19,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:35:19,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.56 | bwd_microstep: 5688.79 | bwd_inner_microstep: 5657.03 | bwd_allreduce_microstep: 31.71 | step_microstep: 18.69
[2025-04-26 11:35:19,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.56 | bwd: 5688.80 | bwd_inner: 5657.03 | bwd_allreduce: 31.73 | step: 18.69

  7%|▋         | 3086/41250 [7:26:51<91:44:06,  8.65s/it]
                                                         
{'loss': 0.1473, 'grad_norm': 2.0603253841400146, 'learning_rate': 3.9789834669250804e-05, 'epoch': 0.75}

  7%|▋         | 3086/41250 [7:26:51<91:44:06,  8.65s/it][2025-04-26 11:35:28,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:35:28,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.30 | bwd_microstep: 5714.02 | bwd_inner_microstep: 5701.19 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.59
[2025-04-26 11:35:28,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.30 | bwd: 5714.03 | bwd_inner: 5701.19 | bwd_allreduce: 12.80 | step: 18.59

  7%|▋         | 3087/41250 [7:26:59<91:42:42,  8.65s/it]
                                                         
{'loss': 0.0372, 'grad_norm': 1.3832926750183105, 'learning_rate': 3.9789607555787657e-05, 'epoch': 0.75}

  7%|▋         | 3087/41250 [7:26:59<91:42:42,  8.65s/it][2025-04-26 11:35:37,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:35:37,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.00 | bwd_microstep: 5718.43 | bwd_inner_microstep: 5705.41 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.80
[2025-04-26 11:35:37,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.00 | bwd: 5718.45 | bwd_inner: 5705.41 | bwd_allreduce: 12.99 | step: 18.80

  7%|▋         | 3088/41250 [7:27:08<91:43:34,  8.65s/it]
                                                         
{'loss': 0.373, 'grad_norm': 1.6912530660629272, 'learning_rate': 3.978938032032548e-05, 'epoch': 0.75}

  7%|▋         | 3088/41250 [7:27:08<91:43:34,  8.65s/it][2025-04-26 11:35:45,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:35:45,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.24 | bwd_microstep: 5737.97 | bwd_inner_microstep: 5642.59 | bwd_allreduce_microstep: 95.34 | step_microstep: 18.36
[2025-04-26 11:35:45,917] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.24 | bwd: 5737.99 | bwd_inner: 5642.59 | bwd_allreduce: 95.35 | step: 18.36

  7%|▋         | 3089/41250 [7:27:17<91:38:23,  8.65s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 0.777266263961792, 'learning_rate': 3.978915296286566e-05, 'epoch': 0.75}

  7%|▋         | 3089/41250 [7:27:17<91:38:23,  8.65s/it][2025-04-26 11:35:54,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:35:54,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.51 | bwd_microstep: 5778.35 | bwd_inner_microstep: 5677.57 | bwd_allreduce_microstep: 100.74 | step_microstep: 18.76
[2025-04-26 11:35:54,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.51 | bwd: 5778.37 | bwd_inner: 5677.57 | bwd_allreduce: 100.76 | step: 18.76

  7%|▋         | 3090/41250 [7:27:25<91:48:47,  8.66s/it]
                                                         
{'loss': 0.1218, 'grad_norm': 1.5647727251052856, 'learning_rate': 3.97889254834096e-05, 'epoch': 0.75}

  7%|▋         | 3090/41250 [7:27:25<91:48:47,  8.66s/it][2025-04-26 11:36:03,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 11:36:03,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.14 | bwd_microstep: 5747.41 | bwd_inner_microstep: 5707.50 | bwd_allreduce_microstep: 39.86 | step_microstep: 18.76
[2025-04-26 11:36:03,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.14 | bwd: 5747.42 | bwd_inner: 5707.50 | bwd_allreduce: 39.88 | step: 18.76

  7%|▋         | 3091/41250 [7:27:34<91:52:05,  8.67s/it]
                                                         
{'loss': 0.2093, 'grad_norm': 1.676342487335205, 'learning_rate': 3.978869788195871e-05, 'epoch': 0.75}

  7%|▋         | 3091/41250 [7:27:34<91:52:05,  8.67s/it][2025-04-26 11:36:11,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 11:36:11,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.99 | bwd_microstep: 5736.55 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 38.06 | step_microstep: 19.03
[2025-04-26 11:36:11,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.99 | bwd: 5736.57 | bwd_inner: 5698.45 | bwd_allreduce: 38.08 | step: 19.04

  7%|▋         | 3092/41250 [7:27:43<91:53:02,  8.67s/it]
                                                         
{'loss': 0.251, 'grad_norm': 1.3055366277694702, 'learning_rate': 3.9788470158514394e-05, 'epoch': 0.75}

  7%|▋         | 3092/41250 [7:27:43<91:53:02,  8.67s/it][2025-04-26 11:36:20,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.18 | optimizer_step: 0.92
[2025-04-26 11:36:20,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.97 | bwd_microstep: 5752.64 | bwd_inner_microstep: 5706.31 | bwd_allreduce_microstep: 46.29 | step_microstep: 18.88
[2025-04-26 11:36:20,655] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.97 | bwd: 5752.66 | bwd_inner: 5706.31 | bwd_allreduce: 46.31 | step: 18.88

  7%|▋         | 3093/41250 [7:27:51<91:56:04,  8.67s/it]
                                                         
{'loss': 0.3825, 'grad_norm': 2.3359475135803223, 'learning_rate': 3.978824231307805e-05, 'epoch': 0.75}

  7%|▋         | 3093/41250 [7:27:51<91:56:04,  8.67s/it][2025-04-26 11:36:29,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:36:29,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.27 | bwd_microstep: 5776.79 | bwd_inner_microstep: 5647.12 | bwd_allreduce_microstep: 129.62 | step_microstep: 18.63
[2025-04-26 11:36:29,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.27 | bwd: 5776.81 | bwd_inner: 5647.12 | bwd_allreduce: 129.65 | step: 18.64

  8%|▊         | 3094/41250 [7:28:00<91:57:52,  8.68s/it]
                                                         
{'loss': 0.3595, 'grad_norm': 2.2799971103668213, 'learning_rate': 3.9788014345651084e-05, 'epoch': 0.75}

  8%|▊         | 3094/41250 [7:28:00<91:57:52,  8.68s/it][2025-04-26 11:36:38,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:36:38,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.24 | bwd_microstep: 5763.84 | bwd_inner_microstep: 5650.41 | bwd_allreduce_microstep: 113.38 | step_microstep: 18.46
[2025-04-26 11:36:38,013] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.25 | bwd: 5763.85 | bwd_inner: 5650.40 | bwd_allreduce: 113.40 | step: 18.46

  8%|▊         | 3095/41250 [7:28:09<91:56:59,  8.68s/it]
                                                         
{'loss': 0.3011, 'grad_norm': 2.455904960632324, 'learning_rate': 3.9787786256234894e-05, 'epoch': 0.75}

  8%|▊         | 3095/41250 [7:28:09<91:56:59,  8.68s/it][2025-04-26 11:36:46,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:36:46,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.53 | bwd_microstep: 5761.12 | bwd_inner_microstep: 5658.02 | bwd_allreduce_microstep: 103.05 | step_microstep: 18.73
[2025-04-26 11:36:46,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.53 | bwd: 5761.13 | bwd_inner: 5658.02 | bwd_allreduce: 103.07 | step: 18.73

  8%|▊         | 3096/41250 [7:28:17<91:57:45,  8.68s/it]
                                                         
{'loss': 0.0925, 'grad_norm': 1.2094612121582031, 'learning_rate': 3.97875580448309e-05, 'epoch': 0.75}

  8%|▊         | 3096/41250 [7:28:17<91:57:45,  8.68s/it][2025-04-26 11:36:55,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 11:36:55,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.44 | bwd_microstep: 5782.01 | bwd_inner_microstep: 5650.97 | bwd_allreduce_microstep: 130.99 | step_microstep: 18.99
[2025-04-26 11:36:55,382] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.44 | bwd: 5782.02 | bwd_inner: 5650.97 | bwd_allreduce: 131.01 | step: 19.00

  8%|▊         | 3097/41250 [7:28:26<92:00:06,  8.68s/it]
                                                         
{'loss': 0.3642, 'grad_norm': 2.0672500133514404, 'learning_rate': 3.9787329711440506e-05, 'epoch': 0.75}

  8%|▊         | 3097/41250 [7:28:26<92:00:06,  8.68s/it][2025-04-26 11:37:04,049] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 11:37:04,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.07 | bwd_microstep: 5738.85 | bwd_inner_microstep: 5725.67 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.64
[2025-04-26 11:37:04,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.07 | bwd: 5738.86 | bwd_inner: 5725.67 | bwd_allreduce: 13.15 | step: 18.64

  8%|▊         | 3098/41250 [7:28:35<91:57:22,  8.68s/it]
                                                         
{'loss': 0.0254, 'grad_norm': 0.3233661353588104, 'learning_rate': 3.978710125606511e-05, 'epoch': 0.75}

  8%|▊         | 3098/41250 [7:28:35<91:57:22,  8.68s/it][2025-04-26 11:37:12,671] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:37:12,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.94 | bwd_microstep: 5712.08 | bwd_inner_microstep: 5673.34 | bwd_allreduce_microstep: 38.69 | step_microstep: 18.05
[2025-04-26 11:37:12,672] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.94 | bwd: 5712.09 | bwd_inner: 5673.34 | bwd_allreduce: 38.71 | step: 18.05

  8%|▊         | 3099/41250 [7:28:43<91:47:24,  8.66s/it]
                                                         
{'loss': 0.188, 'grad_norm': 1.5239008665084839, 'learning_rate': 3.978687267870613e-05, 'epoch': 0.75}

  8%|▊         | 3099/41250 [7:28:43<91:47:24,  8.66s/it][2025-04-26 11:37:21,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:37:21,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.43 | bwd_microstep: 5712.76 | bwd_inner_microstep: 5700.04 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.90
[2025-04-26 11:37:21,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.43 | bwd: 5712.77 | bwd_inner: 5700.04 | bwd_allreduce: 12.70 | step: 17.90

  8%|▊         | 3100/41250 [7:28:52<91:44:45,  8.66s/it]
                                                         
{'loss': 0.2595, 'grad_norm': 2.6307194232940674, 'learning_rate': 3.978664397936497e-05, 'epoch': 0.75}

  8%|▊         | 3100/41250 [7:28:52<91:44:45,  8.66s/it][2025-04-26 11:37:29,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 11:37:29,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.11 | bwd_microstep: 5717.81 | bwd_inner_microstep: 5645.79 | bwd_allreduce_microstep: 71.97 | step_microstep: 18.77
[2025-04-26 11:37:29,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.11 | bwd: 5717.82 | bwd_inner: 5645.79 | bwd_allreduce: 71.99 | step: 18.78

  8%|▊         | 3101/41250 [7:29:01<91:38:54,  8.65s/it]
                                                         
{'loss': 0.1376, 'grad_norm': 1.2840296030044556, 'learning_rate': 3.978641515804304e-05, 'epoch': 0.75}

  8%|▊         | 3101/41250 [7:29:01<91:38:54,  8.65s/it][2025-04-26 11:37:38,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 11:37:38,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.69 | bwd_microstep: 5901.14 | bwd_inner_microstep: 5662.25 | bwd_allreduce_microstep: 238.84 | step_microstep: 18.71
[2025-04-26 11:37:38,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.69 | bwd: 5901.15 | bwd_inner: 5662.25 | bwd_allreduce: 238.86 | step: 18.71

  8%|▊         | 3102/41250 [7:29:09<92:10:24,  8.70s/it]
                                                         
{'loss': 0.1326, 'grad_norm': 1.8967896699905396, 'learning_rate': 3.978618621474176e-05, 'epoch': 0.75}

  8%|▊         | 3102/41250 [7:29:09<92:10:24,  8.70s/it][2025-04-26 11:37:47,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.00
[2025-04-26 11:37:47,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.12 | bwd_microstep: 5877.90 | bwd_inner_microstep: 5667.10 | bwd_allreduce_microstep: 210.76 | step_microstep: 18.87
[2025-04-26 11:37:47,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.12 | bwd: 5877.92 | bwd_inner: 5667.10 | bwd_allreduce: 210.78 | step: 18.87

  8%|▊         | 3103/41250 [7:29:18<92:27:22,  8.73s/it]
                                                         
{'loss': 0.0675, 'grad_norm': 0.8361861705780029, 'learning_rate': 3.978595714946253e-05, 'epoch': 0.75}

  8%|▊         | 3103/41250 [7:29:18<92:27:22,  8.73s/it][2025-04-26 11:37:56,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 11:37:56,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5774.74 | bwd_inner_microstep: 5667.25 | bwd_allreduce_microstep: 107.44 | step_microstep: 19.12
[2025-04-26 11:37:56,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5774.75 | bwd_inner: 5667.25 | bwd_allreduce: 107.46 | step: 19.12

  8%|▊         | 3104/41250 [7:29:27<92:21:49,  8.72s/it]
                                                         
{'loss': 0.2202, 'grad_norm': 2.268587827682495, 'learning_rate': 3.978572796220676e-05, 'epoch': 0.75}

  8%|▊         | 3104/41250 [7:29:27<92:21:49,  8.72s/it][2025-04-26 11:38:04,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.09 | optimizer_step: 1.02
[2025-04-26 11:38:04,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.80 | bwd_microstep: 5791.08 | bwd_inner_microstep: 5718.71 | bwd_allreduce_microstep: 72.31 | step_microstep: 19.17
[2025-04-26 11:38:04,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.80 | bwd: 5791.10 | bwd_inner: 5718.71 | bwd_allreduce: 72.34 | step: 19.17

  8%|▊         | 3105/41250 [7:29:36<92:25:01,  8.72s/it]
                                                         
{'loss': 0.1624, 'grad_norm': 1.2740387916564941, 'learning_rate': 3.978549865297588e-05, 'epoch': 0.75}

  8%|▊         | 3105/41250 [7:29:36<92:25:01,  8.72s/it][2025-04-26 11:38:13,694] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 11:38:13,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.47 | bwd_microstep: 5773.94 | bwd_inner_microstep: 5716.75 | bwd_allreduce_microstep: 57.14 | step_microstep: 18.94
[2025-04-26 11:38:13,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.47 | bwd: 5773.95 | bwd_inner: 5716.75 | bwd_allreduce: 57.16 | step: 18.95

  8%|▊         | 3106/41250 [7:29:44<92:22:32,  8.72s/it]
                                                         
{'loss': 0.2502, 'grad_norm': 2.0917766094207764, 'learning_rate': 3.9785269221771286e-05, 'epoch': 0.75}

  8%|▊         | 3106/41250 [7:29:44<92:22:32,  8.72s/it][2025-04-26 11:38:22,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[2025-04-26 11:38:22,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.37 | bwd_microstep: 5799.90 | bwd_inner_microstep: 5654.40 | bwd_allreduce_microstep: 145.45 | step_microstep: 18.97
[2025-04-26 11:38:22,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.37 | bwd: 5799.92 | bwd_inner: 5654.40 | bwd_allreduce: 145.47 | step: 18.97

  8%|▊         | 3107/41250 [7:29:53<92:18:25,  8.71s/it]
                                                         
{'loss': 0.0993, 'grad_norm': 1.7968661785125732, 'learning_rate': 3.97850396685944e-05, 'epoch': 0.75}

  8%|▊         | 3107/41250 [7:29:53<92:18:25,  8.71s/it][2025-04-26 11:38:31,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.05 | optimizer_step: 1.04
[2025-04-26 11:38:31,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.46 | bwd_microstep: 5710.62 | bwd_inner_microstep: 5697.95 | bwd_allreduce_microstep: 12.62 | step_microstep: 19.48
[2025-04-26 11:38:31,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.46 | bwd: 5710.64 | bwd_inner: 5697.95 | bwd_allreduce: 12.64 | step: 19.49

  8%|▊         | 3108/41250 [7:30:02<92:07:24,  8.69s/it]
                                                         
{'loss': 0.1053, 'grad_norm': 0.8843798041343689, 'learning_rate': 3.9784809993446634e-05, 'epoch': 0.75}

  8%|▊         | 3108/41250 [7:30:02<92:07:24,  8.69s/it][2025-04-26 11:38:39,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 11:38:39,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.30 | bwd_microstep: 5723.28 | bwd_inner_microstep: 5709.33 | bwd_allreduce_microstep: 13.91 | step_microstep: 18.54
[2025-04-26 11:38:39,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.30 | bwd: 5723.30 | bwd_inner: 5709.33 | bwd_allreduce: 13.93 | step: 18.54

  8%|▊         | 3109/41250 [7:30:10<92:00:15,  8.68s/it]
                                                         
{'loss': 0.1545, 'grad_norm': 1.5462125539779663, 'learning_rate': 3.97845801963294e-05, 'epoch': 0.75}

  8%|▊         | 3109/41250 [7:30:10<92:00:15,  8.68s/it][2025-04-26 11:38:48,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-26 11:38:48,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.54 | bwd_microstep: 5750.36 | bwd_inner_microstep: 5717.09 | bwd_allreduce_microstep: 33.23 | step_microstep: 18.82
[2025-04-26 11:38:48,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.54 | bwd: 5750.38 | bwd_inner: 5717.09 | bwd_allreduce: 33.25 | step: 18.83

  8%|▊         | 3110/41250 [7:30:19<92:01:30,  8.69s/it]
                                                         
{'loss': 0.0762, 'grad_norm': 1.272457242012024, 'learning_rate': 3.978435027724413e-05, 'epoch': 0.75}

  8%|▊         | 3110/41250 [7:30:19<92:01:30,  8.69s/it][2025-04-26 11:38:57,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:38:57,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.40 | bwd_microstep: 5724.50 | bwd_inner_microstep: 5666.39 | bwd_allreduce_microstep: 58.06 | step_microstep: 18.52
[2025-04-26 11:38:57,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.40 | bwd: 5724.51 | bwd_inner: 5666.39 | bwd_allreduce: 58.08 | step: 18.52

  8%|▊         | 3111/41250 [7:30:28<91:51:28,  8.67s/it]
                                                         
{'loss': 0.1182, 'grad_norm': 0.8504347801208496, 'learning_rate': 3.978412023619223e-05, 'epoch': 0.75}

  8%|▊         | 3111/41250 [7:30:28<91:51:28,  8.67s/it][2025-04-26 11:39:05,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:39:05,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5882.34 | bwd_inner_microstep: 5648.43 | bwd_allreduce_microstep: 233.86 | step_microstep: 18.23
[2025-04-26 11:39:05,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5882.35 | bwd_inner: 5648.43 | bwd_allreduce: 233.87 | step: 18.23

  8%|▊         | 3112/41250 [7:30:37<92:17:10,  8.71s/it]
                                                         
{'loss': 0.1472, 'grad_norm': 1.5670254230499268, 'learning_rate': 3.978389007317512e-05, 'epoch': 0.75}

  8%|▊         | 3112/41250 [7:30:37<92:17:10,  8.71s/it][2025-04-26 11:39:14,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 11:39:14,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2886.32 | bwd_microstep: 5774.76 | bwd_inner_microstep: 5761.99 | bwd_allreduce_microstep: 12.73 | step_microstep: 17.93
[2025-04-26 11:39:14,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2886.32 | bwd: 5774.77 | bwd_inner: 5761.99 | bwd_allreduce: 12.74 | step: 17.93

  8%|▊         | 3113/41250 [7:30:45<92:23:23,  8.72s/it]
                                                         
{'loss': 0.0852, 'grad_norm': 0.8894575834274292, 'learning_rate': 3.9783659788194215e-05, 'epoch': 0.75}

  8%|▊         | 3113/41250 [7:30:45<92:23:23,  8.72s/it][2025-04-26 11:39:23,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 11:39:23,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.53 | bwd_microstep: 5712.55 | bwd_inner_microstep: 5699.77 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.03
[2025-04-26 11:39:23,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.53 | bwd: 5712.57 | bwd_inner: 5699.77 | bwd_allreduce: 12.75 | step: 19.04

  8%|▊         | 3114/41250 [7:30:54<92:10:48,  8.70s/it]
                                                         
{'loss': 0.2073, 'grad_norm': 1.8539167642593384, 'learning_rate': 3.978342938125094e-05, 'epoch': 0.75}

  8%|▊         | 3114/41250 [7:30:54<92:10:48,  8.70s/it][2025-04-26 11:39:31,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:39:31,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.93 | bwd_microstep: 5716.98 | bwd_inner_microstep: 5703.92 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.80
[2025-04-26 11:39:31,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.93 | bwd: 5717.00 | bwd_inner: 5703.92 | bwd_allreduce: 13.03 | step: 18.81

  8%|▊         | 3115/41250 [7:31:03<91:59:59,  8.68s/it]
                                                         
{'loss': 0.1523, 'grad_norm': 1.0866289138793945, 'learning_rate': 3.9783198852346714e-05, 'epoch': 0.76}

  8%|▊         | 3115/41250 [7:31:03<91:59:59,  8.68s/it][2025-04-26 11:39:40,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 11:39:40,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.49 | bwd_microstep: 5704.88 | bwd_inner_microstep: 5688.69 | bwd_allreduce_microstep: 16.14 | step_microstep: 18.98
[2025-04-26 11:39:40,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.49 | bwd: 5704.89 | bwd_inner: 5688.69 | bwd_allreduce: 16.16 | step: 18.98

  8%|▊         | 3116/41250 [7:31:11<91:49:29,  8.67s/it]
                                                         
{'loss': 0.154, 'grad_norm': 1.2794009447097778, 'learning_rate': 3.978296820148296e-05, 'epoch': 0.76}

  8%|▊         | 3116/41250 [7:31:11<91:49:29,  8.67s/it][2025-04-26 11:39:49,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:39:49,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.42 | bwd_microstep: 5772.87 | bwd_inner_microstep: 5760.24 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.26
[2025-04-26 11:39:49,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.42 | bwd: 5772.88 | bwd_inner: 5760.24 | bwd_allreduce: 12.60 | step: 18.26

  8%|▊         | 3117/41250 [7:31:20<92:04:36,  8.69s/it]
                                                         
{'loss': 0.197, 'grad_norm': 1.7958159446716309, 'learning_rate': 3.9782737428661103e-05, 'epoch': 0.76}

  8%|▊         | 3117/41250 [7:31:20<92:04:36,  8.69s/it][2025-04-26 11:39:58,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:39:58,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.40 | bwd_microstep: 5992.99 | bwd_inner_microstep: 5646.39 | bwd_allreduce_microstep: 346.56 | step_microstep: 18.18
[2025-04-26 11:39:58,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.40 | bwd: 5993.01 | bwd_inner: 5646.39 | bwd_allreduce: 346.58 | step: 18.18

  8%|▊         | 3118/41250 [7:31:29<92:45:31,  8.76s/it]
                                                         
{'loss': 0.0911, 'grad_norm': 1.1213428974151611, 'learning_rate': 3.978250653388256e-05, 'epoch': 0.76}

  8%|▊         | 3118/41250 [7:31:29<92:45:31,  8.76s/it][2025-04-26 11:40:06,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:40:06,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.93 | bwd_microstep: 5748.65 | bwd_inner_microstep: 5642.24 | bwd_allreduce_microstep: 106.37 | step_microstep: 18.24
[2025-04-26 11:40:06,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.93 | bwd: 5748.67 | bwd_inner: 5642.24 | bwd_allreduce: 106.39 | step: 18.24

  8%|▊         | 3119/41250 [7:31:38<92:29:06,  8.73s/it]
                                                         
{'loss': 0.2262, 'grad_norm': 1.8953495025634766, 'learning_rate': 3.978227551714875e-05, 'epoch': 0.76}

  8%|▊         | 3119/41250 [7:31:38<92:29:06,  8.73s/it][2025-04-26 11:40:15,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:40:15,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.40 | bwd_microstep: 5672.38 | bwd_inner_microstep: 5651.76 | bwd_allreduce_microstep: 20.57 | step_microstep: 18.44
[2025-04-26 11:40:15,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.40 | bwd: 5672.40 | bwd_inner: 5651.76 | bwd_allreduce: 20.59 | step: 18.45

  8%|▊         | 3120/41250 [7:31:46<92:04:03,  8.69s/it]
                                                         
{'loss': 0.0841, 'grad_norm': 1.140097975730896, 'learning_rate': 3.97820443784611e-05, 'epoch': 0.76}

  8%|▊         | 3120/41250 [7:31:46<92:04:03,  8.69s/it][2025-04-26 11:40:24,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.05 | optimizer_step: 0.93
[2025-04-26 11:40:24,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5862.05 | bwd_inner_microstep: 5689.61 | bwd_allreduce_microstep: 172.39 | step_microstep: 18.85
[2025-04-26 11:40:24,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.45 | bwd: 5862.07 | bwd_inner: 5689.61 | bwd_allreduce: 172.42 | step: 18.86

  8%|▊         | 3121/41250 [7:31:55<92:23:11,  8.72s/it]
                                                         
{'loss': 0.3244, 'grad_norm': 1.199091911315918, 'learning_rate': 3.9781813117821044e-05, 'epoch': 0.76}

  8%|▊         | 3121/41250 [7:31:55<92:23:11,  8.72s/it][2025-04-26 11:40:32,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:40:32,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.00 | bwd_microstep: 5727.88 | bwd_inner_microstep: 5700.38 | bwd_allreduce_microstep: 27.45 | step_microstep: 18.69
[2025-04-26 11:40:32,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.00 | bwd: 5727.90 | bwd_inner: 5700.38 | bwd_allreduce: 27.47 | step: 18.70

  8%|▊         | 3122/41250 [7:32:04<92:11:11,  8.70s/it]
                                                         
{'loss': 0.145, 'grad_norm': 1.9970108270645142, 'learning_rate': 3.978158173522999e-05, 'epoch': 0.76}

  8%|▊         | 3122/41250 [7:32:04<92:11:11,  8.70s/it][2025-04-26 11:40:41,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.38 | optimizer_step: 0.91
[2025-04-26 11:40:41,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3074.49 | bwd_microstep: 5652.52 | bwd_inner_microstep: 5639.52 | bwd_allreduce_microstep: 12.95 | step_microstep: 20.84
[2025-04-26 11:40:41,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3074.49 | bwd: 5652.53 | bwd_inner: 5639.52 | bwd_allreduce: 12.97 | step: 20.85

  8%|▊         | 3123/41250 [7:32:12<92:31:45,  8.74s/it]
                                                         
{'loss': 0.1415, 'grad_norm': 1.9421250820159912, 'learning_rate': 3.978135023068939e-05, 'epoch': 0.76}

  8%|▊         | 3123/41250 [7:32:12<92:31:45,  8.74s/it][2025-04-26 11:40:50,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 11:40:50,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.04 | bwd_microstep: 5754.54 | bwd_inner_microstep: 5692.85 | bwd_allreduce_microstep: 61.64 | step_microstep: 19.02
[2025-04-26 11:40:50,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.04 | bwd: 5754.55 | bwd_inner: 5692.85 | bwd_allreduce: 61.66 | step: 19.02

  8%|▊         | 3124/41250 [7:32:21<92:21:05,  8.72s/it]
                                                         
{'loss': 0.4156, 'grad_norm': 2.5078530311584473, 'learning_rate': 3.978111860420064e-05, 'epoch': 0.76}

  8%|▊         | 3124/41250 [7:32:21<92:21:05,  8.72s/it][2025-04-26 11:40:59,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.19 | optimizer_step: 0.92
[2025-04-26 11:40:59,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.66 | bwd_microstep: 5708.18 | bwd_inner_microstep: 5694.97 | bwd_allreduce_microstep: 13.15 | step_microstep: 19.23
[2025-04-26 11:40:59,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.66 | bwd: 5708.19 | bwd_inner: 5694.97 | bwd_allreduce: 13.18 | step: 19.23

  8%|▊         | 3125/41250 [7:32:30<92:03:28,  8.69s/it]
                                                         
{'loss': 0.1169, 'grad_norm': 1.2732263803482056, 'learning_rate': 3.97808868557652e-05, 'epoch': 0.76}

  8%|▊         | 3125/41250 [7:32:30<92:03:28,  8.69s/it][2025-04-26 11:41:07,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:41:07,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.19 | bwd_microstep: 5729.10 | bwd_inner_microstep: 5697.32 | bwd_allreduce_microstep: 31.73 | step_microstep: 18.56
[2025-04-26 11:41:07,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.19 | bwd: 5729.11 | bwd_inner: 5697.32 | bwd_allreduce: 31.75 | step: 18.57

  8%|▊         | 3126/41250 [7:32:38<91:57:11,  8.68s/it]
                                                         
{'loss': 0.1857, 'grad_norm': 1.2564831972122192, 'learning_rate': 3.978065498538448e-05, 'epoch': 0.76}

  8%|▊         | 3126/41250 [7:32:38<91:57:11,  8.68s/it][2025-04-26 11:41:16,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:41:16,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.25 | bwd_microstep: 5884.23 | bwd_inner_microstep: 5871.06 | bwd_allreduce_microstep: 13.13 | step_microstep: 18.62
[2025-04-26 11:41:16,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.25 | bwd: 5884.25 | bwd_inner: 5871.06 | bwd_allreduce: 13.15 | step: 18.63

  8%|▊         | 3127/41250 [7:32:47<92:39:02,  8.75s/it]
                                                         
{'loss': 0.1123, 'grad_norm': 0.9160584807395935, 'learning_rate': 3.978042299305991e-05, 'epoch': 0.76}

  8%|▊         | 3127/41250 [7:32:47<92:39:02,  8.75s/it][2025-04-26 11:41:25,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:41:25,256] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5764.09 | bwd_inner_microstep: 5663.39 | bwd_allreduce_microstep: 100.65 | step_microstep: 18.50
[2025-04-26 11:41:25,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5764.10 | bwd_inner: 5663.39 | bwd_allreduce: 100.66 | step: 18.50

  8%|▊         | 3128/41250 [7:32:56<92:23:59,  8.73s/it]
                                                         
{'loss': 0.1208, 'grad_norm': 0.7338298559188843, 'learning_rate': 3.9780190878792924e-05, 'epoch': 0.76}

  8%|▊         | 3128/41250 [7:32:56<92:23:59,  8.73s/it][2025-04-26 11:41:33,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:41:33,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.94 | bwd_microstep: 5751.84 | bwd_inner_microstep: 5687.28 | bwd_allreduce_microstep: 64.51 | step_microstep: 18.76
[2025-04-26 11:41:33,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.94 | bwd: 5751.86 | bwd_inner: 5687.28 | bwd_allreduce: 64.53 | step: 18.76

  8%|▊         | 3129/41250 [7:33:05<92:16:03,  8.71s/it]
                                                         
{'loss': 0.0913, 'grad_norm': 1.060321569442749, 'learning_rate': 3.977995864258495e-05, 'epoch': 0.76}

  8%|▊         | 3129/41250 [7:33:05<92:16:03,  8.71s/it][2025-04-26 11:41:42,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:41:42,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.77 | bwd_microstep: 5701.64 | bwd_inner_microstep: 5661.04 | bwd_allreduce_microstep: 40.56 | step_microstep: 18.52
[2025-04-26 11:41:42,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.77 | bwd: 5701.66 | bwd_inner: 5661.04 | bwd_allreduce: 40.57 | step: 18.52

  8%|▊         | 3130/41250 [7:33:13<91:55:09,  8.68s/it]
                                                         
{'loss': 0.3262, 'grad_norm': 1.4205151796340942, 'learning_rate': 3.9779726284437424e-05, 'epoch': 0.76}

  8%|▊         | 3130/41250 [7:33:13<91:55:09,  8.68s/it][2025-04-26 11:41:51,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 1.15
[2025-04-26 11:41:51,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.14 | bwd_microstep: 5868.63 | bwd_inner_microstep: 5855.91 | bwd_allreduce_microstep: 12.68 | step_microstep: 19.08
[2025-04-26 11:41:51,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2929.14 | bwd: 5868.65 | bwd_inner: 5855.91 | bwd_allreduce: 12.69 | step: 19.08

  8%|▊         | 3131/41250 [7:33:22<92:33:01,  8.74s/it]
                                                         
{'loss': 0.1501, 'grad_norm': 3.010984182357788, 'learning_rate': 3.977949380435177e-05, 'epoch': 0.76}

  8%|▊         | 3131/41250 [7:33:22<92:33:01,  8.74s/it][2025-04-26 11:42:00,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:42:00,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2878.89 | bwd_microstep: 5775.93 | bwd_inner_microstep: 5762.99 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.00
[2025-04-26 11:42:00,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2878.89 | bwd: 5775.94 | bwd_inner: 5762.99 | bwd_allreduce: 12.91 | step: 19.00

  8%|▊         | 3132/41250 [7:33:31<92:32:12,  8.74s/it]
                                                         
{'loss': 0.1227, 'grad_norm': 1.7749927043914795, 'learning_rate': 3.977926120232943e-05, 'epoch': 0.76}

  8%|▊         | 3132/41250 [7:33:31<92:32:12,  8.74s/it][2025-04-26 11:42:08,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[2025-04-26 11:42:08,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.43 | bwd_microstep: 5688.71 | bwd_inner_microstep: 5650.97 | bwd_allreduce_microstep: 37.69 | step_microstep: 19.49
[2025-04-26 11:42:08,758] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.43 | bwd: 5688.73 | bwd_inner: 5650.97 | bwd_allreduce: 37.71 | step: 19.49

  8%|▊         | 3133/41250 [7:33:39<92:04:58,  8.70s/it]
                                                         
{'loss': 0.173, 'grad_norm': 1.6919633150100708, 'learning_rate': 3.977902847837184e-05, 'epoch': 0.76}

  8%|▊         | 3133/41250 [7:33:39<92:04:58,  8.70s/it][2025-04-26 11:42:17,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.06 | optimizer_step: 1.13
[2025-04-26 11:42:17,432] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.79 | bwd_microstep: 5772.26 | bwd_inner_microstep: 5653.89 | bwd_allreduce_microstep: 118.32 | step_microstep: 19.84
[2025-04-26 11:42:17,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.79 | bwd: 5772.28 | bwd_inner: 5653.89 | bwd_allreduce: 118.34 | step: 19.84

  8%|▊         | 3134/41250 [7:33:48<92:00:47,  8.69s/it]
                                                         
{'loss': 0.0568, 'grad_norm': 0.5408487319946289, 'learning_rate': 3.977879563248043e-05, 'epoch': 0.76}

  8%|▊         | 3134/41250 [7:33:48<92:00:47,  8.69s/it][2025-04-26 11:42:26,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.05 | optimizer_step: 0.96
[2025-04-26 11:42:26,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.92 | bwd_microstep: 5761.61 | bwd_inner_microstep: 5650.33 | bwd_allreduce_microstep: 111.23 | step_microstep: 19.75
[2025-04-26 11:42:26,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.92 | bwd: 5761.63 | bwd_inner: 5650.33 | bwd_allreduce: 111.25 | step: 19.75

  8%|▊         | 3135/41250 [7:33:57<91:56:41,  8.68s/it]
                                                         
{'loss': 0.1229, 'grad_norm': 1.4908860921859741, 'learning_rate': 3.9778562664656625e-05, 'epoch': 0.76}

  8%|▊         | 3135/41250 [7:33:57<91:56:41,  8.68s/it][2025-04-26 11:42:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 11:42:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.92 | bwd_microstep: 5784.93 | bwd_inner_microstep: 5653.66 | bwd_allreduce_microstep: 131.22 | step_microstep: 19.39
[2025-04-26 11:42:34,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5784.95 | bwd_inner: 5653.66 | bwd_allreduce: 131.24 | step: 19.39

  8%|▊         | 3136/41250 [7:34:05<91:59:12,  8.69s/it]
                                                         
{'loss': 0.0942, 'grad_norm': 1.5894478559494019, 'learning_rate': 3.977832957490188e-05, 'epoch': 0.76}

  8%|▊         | 3136/41250 [7:34:05<91:59:12,  8.69s/it][2025-04-26 11:42:43,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:42:43,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.73 | bwd_microstep: 5693.71 | bwd_inner_microstep: 5656.09 | bwd_allreduce_microstep: 37.58 | step_microstep: 18.74
[2025-04-26 11:42:43,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.73 | bwd: 5693.72 | bwd_inner: 5656.09 | bwd_allreduce: 37.60 | step: 18.75

  8%|▊         | 3137/41250 [7:34:14<91:44:33,  8.67s/it]
                                                         
{'loss': 0.0543, 'grad_norm': 1.1711851358413696, 'learning_rate': 3.977809636321762e-05, 'epoch': 0.76}

  8%|▊         | 3137/41250 [7:34:14<91:44:33,  8.67s/it][2025-04-26 11:42:52,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 11:42:52,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.86 | bwd_microstep: 5800.81 | bwd_inner_microstep: 5664.00 | bwd_allreduce_microstep: 136.76 | step_microstep: 19.06
[2025-04-26 11:42:52,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.86 | bwd: 5800.82 | bwd_inner: 5664.00 | bwd_allreduce: 136.78 | step: 19.06

  8%|▊         | 3138/41250 [7:34:23<91:53:45,  8.68s/it]
                                                         
{'loss': 0.1081, 'grad_norm': 2.5783534049987793, 'learning_rate': 3.977786302960529e-05, 'epoch': 0.76}

  8%|▊         | 3138/41250 [7:34:23<91:53:45,  8.68s/it][2025-04-26 11:43:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:43:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.74 | bwd_microstep: 5859.68 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 162.68 | step_microstep: 18.82
[2025-04-26 11:43:00,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.74 | bwd: 5859.69 | bwd_inner: 5696.95 | bwd_allreduce: 162.70 | step: 18.83

  8%|▊         | 3139/41250 [7:34:32<92:14:05,  8.71s/it]
                                                         
{'loss': 0.0834, 'grad_norm': 0.9703132510185242, 'learning_rate': 3.977762957406632e-05, 'epoch': 0.76}

  8%|▊         | 3139/41250 [7:34:32<92:14:05,  8.71s/it][2025-04-26 11:43:09,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:43:09,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.48 | bwd_microstep: 5728.26 | bwd_inner_microstep: 5658.29 | bwd_allreduce_microstep: 69.92 | step_microstep: 18.99
[2025-04-26 11:43:09,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.48 | bwd: 5728.28 | bwd_inner: 5658.29 | bwd_allreduce: 69.94 | step: 18.99

  8%|▊         | 3140/41250 [7:34:40<92:00:50,  8.69s/it]
                                                         
{'loss': 0.1112, 'grad_norm': 1.3005564212799072, 'learning_rate': 3.9777395996602157e-05, 'epoch': 0.76}

  8%|▊         | 3140/41250 [7:34:40<92:00:50,  8.69s/it][2025-04-26 11:43:18,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 11:43:18,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.61 | bwd_microstep: 5777.20 | bwd_inner_microstep: 5664.24 | bwd_allreduce_microstep: 112.91 | step_microstep: 18.95
[2025-04-26 11:43:18,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.61 | bwd: 5777.21 | bwd_inner: 5664.24 | bwd_allreduce: 112.93 | step: 18.95

  8%|▊         | 3141/41250 [7:34:49<92:00:22,  8.69s/it]
                                                         
{'loss': 0.0673, 'grad_norm': 1.3321670293807983, 'learning_rate': 3.977716229721424e-05, 'epoch': 0.76}

  8%|▊         | 3141/41250 [7:34:49<92:00:22,  8.69s/it][2025-04-26 11:43:26,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 1.04 | optimizer_step: 1.25
[2025-04-26 11:43:26,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.75 | bwd_microstep: 5711.83 | bwd_inner_microstep: 5699.02 | bwd_allreduce_microstep: 12.76 | step_microstep: 19.44
[2025-04-26 11:43:26,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.75 | bwd: 5711.84 | bwd_inner: 5699.02 | bwd_allreduce: 12.78 | step: 19.45

  8%|▊         | 3142/41250 [7:34:58<91:51:00,  8.68s/it]
                                                         
{'loss': 0.2632, 'grad_norm': 1.7612566947937012, 'learning_rate': 3.9776928475904e-05, 'epoch': 0.76}

  8%|▊         | 3142/41250 [7:34:58<91:51:00,  8.68s/it][2025-04-26 11:43:35,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:43:35,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.88 | bwd_microstep: 5704.27 | bwd_inner_microstep: 5688.95 | bwd_allreduce_microstep: 15.27 | step_microstep: 18.86
[2025-04-26 11:43:35,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.88 | bwd: 5704.28 | bwd_inner: 5688.95 | bwd_allreduce: 15.29 | step: 18.86

  8%|▊         | 3143/41250 [7:35:06<91:42:24,  8.66s/it]
                                                         
{'loss': 0.1198, 'grad_norm': 1.5870474576950073, 'learning_rate': 3.977669453267289e-05, 'epoch': 0.76}

  8%|▊         | 3143/41250 [7:35:06<91:42:24,  8.66s/it][2025-04-26 11:43:44,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 11:43:44,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.75 | bwd_microstep: 5727.31 | bwd_inner_microstep: 5659.49 | bwd_allreduce_microstep: 67.77 | step_microstep: 18.86
[2025-04-26 11:43:44,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.75 | bwd: 5727.32 | bwd_inner: 5659.49 | bwd_allreduce: 67.79 | step: 18.86

  8%|▊         | 3144/41250 [7:35:15<91:38:10,  8.66s/it]
                                                         
{'loss': 0.5162, 'grad_norm': 3.1438183784484863, 'learning_rate': 3.977646046752235e-05, 'epoch': 0.76}

  8%|▊         | 3144/41250 [7:35:15<91:38:10,  8.66s/it][2025-04-26 11:43:52,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.23 | optimizer_step: 0.89
[2025-04-26 11:43:52,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.72 | bwd_microstep: 5917.75 | bwd_inner_microstep: 5653.21 | bwd_allreduce_microstep: 264.48 | step_microstep: 19.36
[2025-04-26 11:43:52,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.72 | bwd: 5917.76 | bwd_inner: 5653.21 | bwd_allreduce: 264.50 | step: 19.36

  8%|▊         | 3145/41250 [7:35:24<92:10:34,  8.71s/it]
                                                         
{'loss': 0.3683, 'grad_norm': 2.780693292617798, 'learning_rate': 3.977622628045381e-05, 'epoch': 0.76}

  8%|▊         | 3145/41250 [7:35:24<92:10:34,  8.71s/it][2025-04-26 11:44:01,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:44:01,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.86 | bwd_microstep: 5707.33 | bwd_inner_microstep: 5694.52 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.58
[2025-04-26 11:44:01,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.86 | bwd: 5707.34 | bwd_inner: 5694.52 | bwd_allreduce: 12.78 | step: 18.59

  8%|▊         | 3146/41250 [7:35:32<91:57:29,  8.69s/it]
                                                         
{'loss': 0.3804, 'grad_norm': 1.778508186340332, 'learning_rate': 3.977599197146874e-05, 'epoch': 0.76}

  8%|▊         | 3146/41250 [7:35:32<91:57:29,  8.69s/it][2025-04-26 11:44:10,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:44:10,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2935.29 | bwd_microstep: 5887.79 | bwd_inner_microstep: 5874.92 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.50
[2025-04-26 11:44:10,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2935.29 | bwd: 5887.80 | bwd_inner: 5874.92 | bwd_allreduce: 12.84 | step: 18.50

  8%|▊         | 3147/41250 [7:35:41<92:39:08,  8.75s/it]
                                                         
{'loss': 0.2168, 'grad_norm': 1.5305596590042114, 'learning_rate': 3.977575754056856e-05, 'epoch': 0.76}

  8%|▊         | 3147/41250 [7:35:41<92:39:08,  8.75s/it][2025-04-26 11:44:19,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:44:19,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.19 | bwd_microstep: 5728.96 | bwd_inner_microstep: 5716.06 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.87
[2025-04-26 11:44:19,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.19 | bwd: 5728.97 | bwd_inner: 5716.06 | bwd_allreduce: 12.87 | step: 18.87

  8%|▊         | 3148/41250 [7:35:50<92:20:49,  8.73s/it]
                                                         
{'loss': 0.0493, 'grad_norm': 0.8230340480804443, 'learning_rate': 3.977552298775473e-05, 'epoch': 0.76}

  8%|▊         | 3148/41250 [7:35:50<92:20:49,  8.73s/it][2025-04-26 11:44:27,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 11:44:27,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5715.73 | bwd_inner_microstep: 5703.14 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.73
[2025-04-26 11:44:27,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5715.75 | bwd_inner: 5703.14 | bwd_allreduce: 12.57 | step: 18.73

  8%|▊         | 3149/41250 [7:35:59<92:06:01,  8.70s/it]
                                                         
{'loss': 0.1342, 'grad_norm': 1.656947135925293, 'learning_rate': 3.977528831302869e-05, 'epoch': 0.76}

  8%|▊         | 3149/41250 [7:35:59<92:06:01,  8.70s/it][2025-04-26 11:44:36,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:44:36,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.86 | bwd_microstep: 5806.12 | bwd_inner_microstep: 5672.59 | bwd_allreduce_microstep: 133.49 | step_microstep: 18.30
[2025-04-26 11:44:36,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.86 | bwd: 5806.14 | bwd_inner: 5672.59 | bwd_allreduce: 133.51 | step: 18.30

  8%|▊         | 3150/41250 [7:36:07<92:09:02,  8.71s/it]
                                                         
{'loss': 0.0799, 'grad_norm': 1.6020393371582031, 'learning_rate': 3.9775053516391884e-05, 'epoch': 0.76}

  8%|▊         | 3150/41250 [7:36:07<92:09:02,  8.71s/it][2025-04-26 11:44:45,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.17 | optimizer_step: 0.92
[2025-04-26 11:44:45,252] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.16 | bwd_microstep: 5758.77 | bwd_inner_microstep: 5651.81 | bwd_allreduce_microstep: 106.91 | step_microstep: 19.07
[2025-04-26 11:44:45,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.16 | bwd: 5758.78 | bwd_inner: 5651.81 | bwd_allreduce: 106.93 | step: 19.07

  8%|▊         | 3151/41250 [7:36:16<92:03:58,  8.70s/it]
                                                         
{'loss': 0.3162, 'grad_norm': 2.3164222240448, 'learning_rate': 3.977481859784576e-05, 'epoch': 0.76}

  8%|▊         | 3151/41250 [7:36:16<92:03:58,  8.70s/it][2025-04-26 11:44:53,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-26 11:44:53,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5794.35 | bwd_inner_microstep: 5658.66 | bwd_allreduce_microstep: 135.63 | step_microstep: 20.79
[2025-04-26 11:44:53,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5794.37 | bwd_inner: 5658.66 | bwd_allreduce: 135.66 | step: 20.79

  8%|▊         | 3152/41250 [7:36:25<92:03:28,  8.70s/it]
                                                         
{'loss': 0.4033, 'grad_norm': 1.9362057447433472, 'learning_rate': 3.977458355739177e-05, 'epoch': 0.76}

  8%|▊         | 3152/41250 [7:36:25<92:03:28,  8.70s/it][2025-04-26 11:45:02,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 11:45:02,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.09 | bwd_microstep: 5773.24 | bwd_inner_microstep: 5704.46 | bwd_allreduce_microstep: 68.72 | step_microstep: 18.57
[2025-04-26 11:45:02,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.09 | bwd: 5773.25 | bwd_inner: 5704.46 | bwd_allreduce: 68.74 | step: 18.57

  8%|▊         | 3153/41250 [7:36:33<92:05:09,  8.70s/it]
                                                         
{'loss': 0.3103, 'grad_norm': 1.9905266761779785, 'learning_rate': 3.9774348395031366e-05, 'epoch': 0.76}

  8%|▊         | 3153/41250 [7:36:33<92:05:09,  8.70s/it][2025-04-26 11:45:11,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 11:45:11,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.10 | bwd_microstep: 5791.49 | bwd_inner_microstep: 5654.78 | bwd_allreduce_microstep: 136.66 | step_microstep: 18.45
[2025-04-26 11:45:11,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.10 | bwd: 5791.51 | bwd_inner: 5654.78 | bwd_allreduce: 136.68 | step: 18.45

  8%|▊         | 3154/41250 [7:36:42<92:06:10,  8.70s/it]
                                                         
{'loss': 0.2382, 'grad_norm': 2.5100371837615967, 'learning_rate': 3.977411311076599e-05, 'epoch': 0.76}

  8%|▊         | 3154/41250 [7:36:42<92:06:10,  8.70s/it][2025-04-26 11:45:20,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:45:20,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.82 | bwd_microstep: 5755.19 | bwd_inner_microstep: 5687.63 | bwd_allreduce_microstep: 67.52 | step_microstep: 18.53
[2025-04-26 11:45:20,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.82 | bwd: 5755.20 | bwd_inner: 5687.63 | bwd_allreduce: 67.53 | step: 18.53

  8%|▊         | 3155/41250 [7:36:51<92:02:27,  8.70s/it]
                                                         
{'loss': 0.2429, 'grad_norm': 2.6433632373809814, 'learning_rate': 3.97738777045971e-05, 'epoch': 0.76}

  8%|▊         | 3155/41250 [7:36:51<92:02:27,  8.70s/it][2025-04-26 11:45:28,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:45:28,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.39 | bwd_microstep: 5772.96 | bwd_inner_microstep: 5653.69 | bwd_allreduce_microstep: 119.22 | step_microstep: 18.65
[2025-04-26 11:45:28,734] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.39 | bwd: 5772.98 | bwd_inner: 5653.69 | bwd_allreduce: 119.24 | step: 18.66

  8%|▊         | 3156/41250 [7:36:59<91:59:45,  8.69s/it]
                                                         
{'loss': 0.3199, 'grad_norm': 2.235490083694458, 'learning_rate': 3.9773642176526135e-05, 'epoch': 0.77}

  8%|▊         | 3156/41250 [7:36:59<91:59:45,  8.69s/it][2025-04-26 11:45:37,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:45:37,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5788.14 | bwd_inner_microstep: 5699.96 | bwd_allreduce_microstep: 88.14 | step_microstep: 18.31
[2025-04-26 11:45:37,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5788.15 | bwd_inner: 5699.96 | bwd_allreduce: 88.15 | step: 18.31

  8%|▊         | 3157/41250 [7:37:08<92:03:49,  8.70s/it]
                                                         
{'loss': 0.0431, 'grad_norm': 0.562830924987793, 'learning_rate': 3.977340652655456e-05, 'epoch': 0.77}

  8%|▊         | 3157/41250 [7:37:08<92:03:49,  8.70s/it][2025-04-26 11:45:46,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 11:45:46,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.18 | bwd_microstep: 5713.75 | bwd_inner_microstep: 5701.00 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.88
[2025-04-26 11:45:46,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.18 | bwd: 5713.77 | bwd_inner: 5701.00 | bwd_allreduce: 12.73 | step: 18.88

  8%|▊         | 3158/41250 [7:37:17<91:56:01,  8.69s/it]
                                                         
{'loss': 0.1229, 'grad_norm': 0.7834312319755554, 'learning_rate': 3.977317075468383e-05, 'epoch': 0.77}

  8%|▊         | 3158/41250 [7:37:17<91:56:01,  8.69s/it][2025-04-26 11:45:55,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:45:55,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.16 | bwd_microstep: 6045.38 | bwd_inner_microstep: 5707.34 | bwd_allreduce_microstep: 338.00 | step_microstep: 17.92
[2025-04-26 11:45:55,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.16 | bwd: 6045.40 | bwd_inner: 5707.34 | bwd_allreduce: 338.02 | step: 17.93

  8%|▊         | 3159/41250 [7:37:26<92:52:07,  8.78s/it]
                                                         
{'loss': 0.1121, 'grad_norm': 1.3155546188354492, 'learning_rate': 3.977293486091538e-05, 'epoch': 0.77}

  8%|▊         | 3159/41250 [7:37:26<92:52:07,  8.78s/it][2025-04-26 11:46:03,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 11:46:03,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5770.06 | bwd_inner_microstep: 5690.49 | bwd_allreduce_microstep: 79.52 | step_microstep: 18.38
[2025-04-26 11:46:03,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5770.07 | bwd_inner: 5690.49 | bwd_allreduce: 79.54 | step: 18.38

  8%|▊         | 3160/41250 [7:37:34<92:37:35,  8.75s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 1.1922188997268677, 'learning_rate': 3.977269884525068e-05, 'epoch': 0.77}

  8%|▊         | 3160/41250 [7:37:34<92:37:35,  8.75s/it][2025-04-26 11:46:12,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.07
[2025-04-26 11:46:12,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.99 | bwd_microstep: 5712.23 | bwd_inner_microstep: 5699.50 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.60
[2025-04-26 11:46:12,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.99 | bwd: 5712.24 | bwd_inner: 5699.50 | bwd_allreduce: 12.70 | step: 18.61

  8%|▊         | 3161/41250 [7:37:43<92:14:20,  8.72s/it]
                                                         
{'loss': 0.0519, 'grad_norm': 0.5260977149009705, 'learning_rate': 3.9772462707691184e-05, 'epoch': 0.77}

  8%|▊         | 3161/41250 [7:37:43<92:14:20,  8.72s/it][2025-04-26 11:46:21,140] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:46:21,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.93 | bwd_microstep: 5766.15 | bwd_inner_microstep: 5689.52 | bwd_allreduce_microstep: 76.59 | step_microstep: 18.45
[2025-04-26 11:46:21,141] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.93 | bwd: 5766.16 | bwd_inner: 5689.52 | bwd_allreduce: 76.61 | step: 18.45

  8%|▊         | 3162/41250 [7:37:52<92:12:55,  8.72s/it]
                                                         
{'loss': 0.3816, 'grad_norm': 3.720895528793335, 'learning_rate': 3.9772226448238346e-05, 'epoch': 0.77}

  8%|▊         | 3162/41250 [7:37:52<92:12:55,  8.72s/it][2025-04-26 11:46:29,827] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 1.03
[2025-04-26 11:46:29,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.15 | bwd_microstep: 5746.17 | bwd_inner_microstep: 5694.47 | bwd_allreduce_microstep: 51.65 | step_microstep: 19.22
[2025-04-26 11:46:29,828] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.15 | bwd: 5746.18 | bwd_inner: 5694.47 | bwd_allreduce: 51.67 | step: 19.22

  8%|▊         | 3163/41250 [7:38:01<92:07:16,  8.71s/it]
                                                         
{'loss': 0.3448, 'grad_norm': 1.5949612855911255, 'learning_rate': 3.977199006689362e-05, 'epoch': 0.77}

  8%|▊         | 3163/41250 [7:38:01<92:07:16,  8.71s/it][2025-04-26 11:46:38,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 11:46:38,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.75 | bwd_microstep: 5759.63 | bwd_inner_microstep: 5700.06 | bwd_allreduce_microstep: 59.52 | step_microstep: 18.62
[2025-04-26 11:46:38,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.75 | bwd: 5759.64 | bwd_inner: 5700.06 | bwd_allreduce: 59.54 | step: 18.63

  8%|▊         | 3164/41250 [7:38:09<92:03:35,  8.70s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.23631751537323, 'learning_rate': 3.977175356365846e-05, 'epoch': 0.77}

  8%|▊         | 3164/41250 [7:38:09<92:03:35,  8.70s/it][2025-04-26 11:46:47,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:46:47,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.47 | bwd_microstep: 5740.86 | bwd_inner_microstep: 5700.54 | bwd_allreduce_microstep: 40.27 | step_microstep: 18.48
[2025-04-26 11:46:47,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.47 | bwd: 5740.87 | bwd_inner: 5700.54 | bwd_allreduce: 40.29 | step: 18.48

  8%|▊         | 3165/41250 [7:38:18<91:58:42,  8.69s/it]
                                                         
{'loss': 0.2101, 'grad_norm': 1.428354024887085, 'learning_rate': 3.9771516938534326e-05, 'epoch': 0.77}

  8%|▊         | 3165/41250 [7:38:18<91:58:42,  8.69s/it][2025-04-26 11:46:55,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 11:46:55,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.81 | bwd_microstep: 5789.86 | bwd_inner_microstep: 5777.01 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.58
[2025-04-26 11:46:55,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.81 | bwd: 5789.88 | bwd_inner: 5777.01 | bwd_allreduce: 12.83 | step: 18.59

  8%|▊         | 3166/41250 [7:38:27<92:10:33,  8.71s/it]
                                                         
{'loss': 0.1352, 'grad_norm': 0.7974644303321838, 'learning_rate': 3.977128019152268e-05, 'epoch': 0.77}

  8%|▊         | 3166/41250 [7:38:27<92:10:33,  8.71s/it][2025-04-26 11:47:04,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:47:04,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.30 | bwd_microstep: 5694.09 | bwd_inner_microstep: 5639.50 | bwd_allreduce_microstep: 54.54 | step_microstep: 18.39
[2025-04-26 11:47:04,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.30 | bwd: 5694.11 | bwd_inner: 5639.50 | bwd_allreduce: 54.56 | step: 18.39

  8%|▊         | 3167/41250 [7:38:35<91:49:42,  8.68s/it]
                                                         
{'loss': 0.3301, 'grad_norm': 2.176497459411621, 'learning_rate': 3.9771043322624984e-05, 'epoch': 0.77}

  8%|▊         | 3167/41250 [7:38:35<91:49:42,  8.68s/it][2025-04-26 11:47:13,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.07 | optimizer_step: 1.13
[2025-04-26 11:47:13,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.68 | bwd_microstep: 5697.84 | bwd_inner_microstep: 5647.39 | bwd_allreduce_microstep: 50.41 | step_microstep: 19.35
[2025-04-26 11:47:13,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.68 | bwd: 5697.85 | bwd_inner: 5647.39 | bwd_allreduce: 50.43 | step: 19.35

  8%|▊         | 3168/41250 [7:38:44<91:37:25,  8.66s/it]
                                                         
{'loss': 0.2507, 'grad_norm': 1.6513773202896118, 'learning_rate': 3.97708063318427e-05, 'epoch': 0.77}

  8%|▊         | 3168/41250 [7:38:44<91:37:25,  8.66s/it][2025-04-26 11:47:21,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:47:21,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5777.83 | bwd_inner_microstep: 5643.88 | bwd_allreduce_microstep: 133.91 | step_microstep: 17.84
[2025-04-26 11:47:21,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5777.84 | bwd_inner: 5643.88 | bwd_allreduce: 133.92 | step: 17.84

  8%|▊         | 3169/41250 [7:38:53<91:42:08,  8.67s/it]
                                                         
{'loss': 0.1918, 'grad_norm': 1.259181022644043, 'learning_rate': 3.977056921917727e-05, 'epoch': 0.77}

  8%|▊         | 3169/41250 [7:38:53<91:42:08,  8.67s/it][2025-04-26 11:47:30,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:47:30,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5741.85 | bwd_inner_microstep: 5693.70 | bwd_allreduce_microstep: 48.11 | step_microstep: 17.88
[2025-04-26 11:47:30,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5741.86 | bwd_inner: 5693.70 | bwd_allreduce: 48.13 | step: 17.88

  8%|▊         | 3170/41250 [7:39:01<91:42:05,  8.67s/it]
                                                         
{'loss': 0.0749, 'grad_norm': 0.5431756377220154, 'learning_rate': 3.9770331984630176e-05, 'epoch': 0.77}

  8%|▊         | 3170/41250 [7:39:01<91:42:05,  8.67s/it][2025-04-26 11:47:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:47:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5731.90 | bwd_inner_microstep: 5717.86 | bwd_allreduce_microstep: 14.00 | step_microstep: 18.01
[2025-04-26 11:47:39,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.65 | bwd: 5731.92 | bwd_inner: 5717.86 | bwd_allreduce: 14.02 | step: 18.01

  8%|▊         | 3171/41250 [7:39:10<91:42:31,  8.67s/it]
                                                         
{'loss': 0.0781, 'grad_norm': 0.9542376399040222, 'learning_rate': 3.977009462820288e-05, 'epoch': 0.77}

  8%|▊         | 3171/41250 [7:39:10<91:42:31,  8.67s/it][2025-04-26 11:47:47,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 11:47:47,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.93 | bwd_microstep: 5746.66 | bwd_inner_microstep: 5676.79 | bwd_allreduce_microstep: 69.83 | step_microstep: 18.20
[2025-04-26 11:47:47,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.93 | bwd: 5746.67 | bwd_inner: 5676.79 | bwd_allreduce: 69.84 | step: 18.20

  8%|▊         | 3172/41250 [7:39:19<91:44:33,  8.67s/it]
                                                         
{'loss': 0.0802, 'grad_norm': 0.9474695324897766, 'learning_rate': 3.976985714989683e-05, 'epoch': 0.77}

  8%|▊         | 3172/41250 [7:39:19<91:44:33,  8.67s/it][2025-04-26 11:47:56,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 11:47:56,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.57 | bwd_microstep: 5743.75 | bwd_inner_microstep: 5660.10 | bwd_allreduce_microstep: 83.61 | step_microstep: 18.35
[2025-04-26 11:47:56,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.57 | bwd: 5743.76 | bwd_inner: 5660.09 | bwd_allreduce: 83.63 | step: 18.36

  8%|▊         | 3173/41250 [7:39:27<91:42:24,  8.67s/it]
                                                         
{'loss': 0.1695, 'grad_norm': 1.2198421955108643, 'learning_rate': 3.9769619549713505e-05, 'epoch': 0.77}

  8%|▊         | 3173/41250 [7:39:27<91:42:24,  8.67s/it][2025-04-26 11:48:05,229] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.20 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 11:48:05,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.81 | bwd_microstep: 5746.92 | bwd_inner_microstep: 5697.13 | bwd_allreduce_microstep: 49.74 | step_microstep: 20.16
[2025-04-26 11:48:05,230] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.81 | bwd: 5746.94 | bwd_inner: 5697.13 | bwd_allreduce: 49.76 | step: 20.17

  8%|▊         | 3174/41250 [7:39:36<91:44:31,  8.67s/it]
                                                         
{'loss': 0.2066, 'grad_norm': 1.4944100379943848, 'learning_rate': 3.976938182765436e-05, 'epoch': 0.77}

  8%|▊         | 3174/41250 [7:39:36<91:44:31,  8.67s/it][2025-04-26 11:48:13,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 11:48:13,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.09 | bwd_microstep: 5765.89 | bwd_inner_microstep: 5644.40 | bwd_allreduce_microstep: 121.44 | step_microstep: 18.77
[2025-04-26 11:48:13,901] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.09 | bwd: 5765.90 | bwd_inner: 5644.40 | bwd_allreduce: 121.46 | step: 18.78

  8%|▊         | 3175/41250 [7:39:45<91:44:36,  8.67s/it]
                                                         
{'loss': 0.2446, 'grad_norm': 1.5184764862060547, 'learning_rate': 3.9769143983720866e-05, 'epoch': 0.77}

  8%|▊         | 3175/41250 [7:39:45<91:44:36,  8.67s/it][2025-04-26 11:48:22,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 11:48:22,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.18 | bwd_microstep: 5709.22 | bwd_inner_microstep: 5657.74 | bwd_allreduce_microstep: 51.43 | step_microstep: 19.08
[2025-04-26 11:48:22,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.18 | bwd: 5709.23 | bwd_inner: 5657.74 | bwd_allreduce: 51.44 | step: 19.08

  8%|▊         | 3176/41250 [7:39:53<91:36:16,  8.66s/it]
                                                         
{'loss': 0.0937, 'grad_norm': 0.6676415801048279, 'learning_rate': 3.976890601791449e-05, 'epoch': 0.77}

  8%|▊         | 3176/41250 [7:39:53<91:36:16,  8.66s/it][2025-04-26 11:48:31,226] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.94
[2025-04-26 11:48:31,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.87 | bwd_microstep: 5753.62 | bwd_inner_microstep: 5707.19 | bwd_allreduce_microstep: 46.39 | step_microstep: 19.13
[2025-04-26 11:48:31,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.87 | bwd: 5753.64 | bwd_inner: 5707.19 | bwd_allreduce: 46.41 | step: 19.13

  8%|▊         | 3177/41250 [7:40:02<91:42:28,  8.67s/it]
                                                         
{'loss': 0.2241, 'grad_norm': 1.4063016176223755, 'learning_rate': 3.9768667930236696e-05, 'epoch': 0.77}

  8%|▊         | 3177/41250 [7:40:02<91:42:28,  8.67s/it][2025-04-26 11:48:39,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.06 | optimizer_step: 0.98
[2025-04-26 11:48:39,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.41 | bwd_microstep: 5714.54 | bwd_inner_microstep: 5701.22 | bwd_allreduce_microstep: 13.27 | step_microstep: 19.28
[2025-04-26 11:48:39,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.41 | bwd: 5714.56 | bwd_inner: 5701.22 | bwd_allreduce: 13.29 | step: 19.29

  8%|▊         | 3178/41250 [7:40:11<91:37:17,  8.66s/it]
                                                         
{'loss': 0.1839, 'grad_norm': 1.2848992347717285, 'learning_rate': 3.976842972068895e-05, 'epoch': 0.77}

  8%|▊         | 3178/41250 [7:40:11<91:37:17,  8.66s/it][2025-04-26 11:48:48,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 11:48:48,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.10 | bwd_microstep: 5747.50 | bwd_inner_microstep: 5694.51 | bwd_allreduce_microstep: 52.94 | step_microstep: 18.79
[2025-04-26 11:48:48,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.10 | bwd: 5747.51 | bwd_inner: 5694.51 | bwd_allreduce: 52.96 | step: 18.79

  8%|▊         | 3179/41250 [7:40:19<91:40:58,  8.67s/it]
                                                         
{'loss': 0.2229, 'grad_norm': 2.5412025451660156, 'learning_rate': 3.976819138927272e-05, 'epoch': 0.77}

  8%|▊         | 3179/41250 [7:40:19<91:40:58,  8.67s/it][2025-04-26 11:48:57,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:48:57,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.09 | bwd_microstep: 5753.32 | bwd_inner_microstep: 5703.91 | bwd_allreduce_microstep: 49.36 | step_microstep: 18.12
[2025-04-26 11:48:57,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.09 | bwd: 5753.33 | bwd_inner: 5703.91 | bwd_allreduce: 49.38 | step: 18.12

  8%|▊         | 3180/41250 [7:40:28<91:46:41,  8.68s/it]
                                                         
{'loss': 0.1465, 'grad_norm': 1.373302698135376, 'learning_rate': 3.9767952935989484e-05, 'epoch': 0.77}

  8%|▊         | 3180/41250 [7:40:28<91:46:41,  8.68s/it][2025-04-26 11:49:05,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.42 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:49:05,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.15 | bwd_microstep: 5782.55 | bwd_inner_microstep: 5661.90 | bwd_allreduce_microstep: 120.61 | step_microstep: 18.56
[2025-04-26 11:49:05,975] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.15 | bwd: 5782.56 | bwd_inner: 5661.90 | bwd_allreduce: 120.63 | step: 18.57

  8%|▊         | 3181/41250 [7:40:37<91:53:26,  8.69s/it]
                                                         
{'loss': 0.1534, 'grad_norm': 1.2435765266418457, 'learning_rate': 3.9767714360840705e-05, 'epoch': 0.77}

  8%|▊         | 3181/41250 [7:40:37<91:53:26,  8.69s/it][2025-04-26 11:49:14,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 11:49:14,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.16 | bwd_microstep: 5754.42 | bwd_inner_microstep: 5710.48 | bwd_allreduce_microstep: 43.89 | step_microstep: 18.91
[2025-04-26 11:49:14,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.16 | bwd: 5754.43 | bwd_inner: 5710.48 | bwd_allreduce: 43.91 | step: 18.91

  8%|▊         | 3182/41250 [7:40:45<91:54:10,  8.69s/it]
                                                         
{'loss': 0.19, 'grad_norm': 2.423847198486328, 'learning_rate': 3.976747566382785e-05, 'epoch': 0.77}

  8%|▊         | 3182/41250 [7:40:45<91:54:10,  8.69s/it][2025-04-26 11:49:23,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 11:49:23,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.62 | bwd_microstep: 5717.37 | bwd_inner_microstep: 5704.94 | bwd_allreduce_microstep: 12.38 | step_microstep: 18.92
[2025-04-26 11:49:23,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.62 | bwd: 5717.38 | bwd_inner: 5704.94 | bwd_allreduce: 12.40 | step: 18.92

  8%|▊         | 3183/41250 [7:40:54<91:47:32,  8.68s/it]
                                                         
{'loss': 0.0592, 'grad_norm': 0.6885115504264832, 'learning_rate': 3.9767236844952403e-05, 'epoch': 0.77}

  8%|▊         | 3183/41250 [7:40:54<91:47:32,  8.68s/it][2025-04-26 11:49:31,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:49:31,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.56 | bwd_microstep: 5722.63 | bwd_inner_microstep: 5710.00 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.42
[2025-04-26 11:49:31,981] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.56 | bwd: 5722.64 | bwd_inner: 5710.00 | bwd_allreduce: 12.60 | step: 18.43

  8%|▊         | 3184/41250 [7:41:03<91:42:24,  8.67s/it]
                                                         
{'loss': 0.2988, 'grad_norm': 2.2925631999969482, 'learning_rate': 3.976699790421583e-05, 'epoch': 0.77}

  8%|▊         | 3184/41250 [7:41:03<91:42:24,  8.67s/it][2025-04-26 11:49:40,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.23 | optimizer_step: 0.92
[2025-04-26 11:49:40,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.68 | bwd_microstep: 5785.88 | bwd_inner_microstep: 5679.62 | bwd_allreduce_microstep: 106.21 | step_microstep: 19.04
[2025-04-26 11:49:40,697] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.68 | bwd: 5785.90 | bwd_inner: 5679.62 | bwd_allreduce: 106.24 | step: 19.04

  8%|▊         | 3185/41250 [7:41:11<91:50:29,  8.69s/it]
                                                         
{'loss': 0.0808, 'grad_norm': 1.145554542541504, 'learning_rate': 3.97667588416196e-05, 'epoch': 0.77}

  8%|▊         | 3185/41250 [7:41:11<91:50:29,  8.69s/it][2025-04-26 11:49:49,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:49:49,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2902.31 | bwd_microstep: 5806.78 | bwd_inner_microstep: 5794.12 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.42
[2025-04-26 11:49:49,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2902.31 | bwd: 5806.80 | bwd_inner: 5794.12 | bwd_allreduce: 12.64 | step: 18.42

  8%|▊         | 3186/41250 [7:41:20<92:10:30,  8.72s/it]
                                                         
{'loss': 0.1449, 'grad_norm': 1.048195719718933, 'learning_rate': 3.976651965716519e-05, 'epoch': 0.77}

  8%|▊         | 3186/41250 [7:41:20<92:10:30,  8.72s/it][2025-04-26 11:49:58,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 11:49:58,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.80 | bwd_microstep: 5728.48 | bwd_inner_microstep: 5715.82 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.36
[2025-04-26 11:49:58,159] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.80 | bwd: 5728.49 | bwd_inner: 5715.82 | bwd_allreduce: 12.63 | step: 18.36

  8%|▊         | 3187/41250 [7:41:29<92:01:47,  8.70s/it]
                                                         
{'loss': 0.2843, 'grad_norm': 1.780859351158142, 'learning_rate': 3.976628035085407e-05, 'epoch': 0.77}

  8%|▊         | 3187/41250 [7:41:29<92:01:47,  8.70s/it][2025-04-26 11:50:06,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:50:06,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.32 | bwd_microstep: 5768.65 | bwd_inner_microstep: 5664.76 | bwd_allreduce_microstep: 103.85 | step_microstep: 18.37
[2025-04-26 11:50:06,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.32 | bwd: 5768.67 | bwd_inner: 5664.75 | bwd_allreduce: 103.87 | step: 18.37

  8%|▊         | 3188/41250 [7:41:38<91:58:20,  8.70s/it]
                                                         
{'loss': 0.1913, 'grad_norm': 1.1764408349990845, 'learning_rate': 3.976604092268772e-05, 'epoch': 0.77}

  8%|▊         | 3188/41250 [7:41:38<91:58:20,  8.70s/it][2025-04-26 11:50:15,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 11:50:15,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.65 | bwd_microstep: 5803.52 | bwd_inner_microstep: 5658.78 | bwd_allreduce_microstep: 144.70 | step_microstep: 18.92
[2025-04-26 11:50:15,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.65 | bwd: 5803.54 | bwd_inner: 5658.78 | bwd_allreduce: 144.72 | step: 18.93

  8%|▊         | 3189/41250 [7:41:46<92:02:10,  8.71s/it]
                                                         
{'loss': 0.2547, 'grad_norm': 1.5555981397628784, 'learning_rate': 3.976580137266762e-05, 'epoch': 0.77}

  8%|▊         | 3189/41250 [7:41:46<92:02:10,  8.71s/it][2025-04-26 11:50:24,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:50:24,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.40 | bwd_microstep: 5708.76 | bwd_inner_microstep: 5695.85 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.68
[2025-04-26 11:50:24,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.40 | bwd: 5708.77 | bwd_inner: 5695.85 | bwd_allreduce: 12.88 | step: 18.68

  8%|▊         | 3190/41250 [7:41:55<91:53:15,  8.69s/it]
                                                         
{'loss': 0.2075, 'grad_norm': 1.2510486841201782, 'learning_rate': 3.976556170079524e-05, 'epoch': 0.77}

  8%|▊         | 3190/41250 [7:41:55<91:53:15,  8.69s/it][2025-04-26 11:50:32,865] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:50:32,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5709.46 | bwd_inner_microstep: 5696.63 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.34
[2025-04-26 11:50:32,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5709.48 | bwd_inner: 5696.63 | bwd_allreduce: 12.81 | step: 18.34

  8%|▊         | 3191/41250 [7:42:04<91:43:01,  8.68s/it]
                                                         
{'loss': 0.2488, 'grad_norm': 1.7915928363800049, 'learning_rate': 3.976532190707206e-05, 'epoch': 0.77}

  8%|▊         | 3191/41250 [7:42:04<91:43:01,  8.68s/it][2025-04-26 11:50:41,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 11:50:41,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.42 | bwd_microstep: 5770.80 | bwd_inner_microstep: 5693.64 | bwd_allreduce_microstep: 77.11 | step_microstep: 18.45
[2025-04-26 11:50:41,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.42 | bwd: 5770.81 | bwd_inner: 5693.64 | bwd_allreduce: 77.13 | step: 18.46

  8%|▊         | 3192/41250 [7:42:12<91:51:10,  8.69s/it]
                                                         
{'loss': 0.0375, 'grad_norm': 0.5019420385360718, 'learning_rate': 3.976508199149956e-05, 'epoch': 0.77}

  8%|▊         | 3192/41250 [7:42:12<91:51:10,  8.69s/it][2025-04-26 11:50:50,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 11:50:50,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.17 | bwd_microstep: 5804.24 | bwd_inner_microstep: 5648.46 | bwd_allreduce_microstep: 155.72 | step_microstep: 18.75
[2025-04-26 11:50:50,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.17 | bwd: 5804.25 | bwd_inner: 5648.46 | bwd_allreduce: 155.75 | step: 18.75

  8%|▊         | 3193/41250 [7:42:21<91:56:48,  8.70s/it]
                                                         
{'loss': 0.2883, 'grad_norm': 1.9430128335952759, 'learning_rate': 3.976484195407922e-05, 'epoch': 0.77}

  8%|▊         | 3193/41250 [7:42:21<91:56:48,  8.70s/it][2025-04-26 11:50:59,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 11:50:59,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.41 | bwd_microstep: 5855.07 | bwd_inner_microstep: 5686.83 | bwd_allreduce_microstep: 168.20 | step_microstep: 18.24
[2025-04-26 11:50:59,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.41 | bwd: 5855.09 | bwd_inner: 5686.83 | bwd_allreduce: 168.22 | step: 18.24

  8%|▊         | 3194/41250 [7:42:30<92:15:25,  8.73s/it]
                                                         
{'loss': 0.1307, 'grad_norm': 1.1716593503952026, 'learning_rate': 3.9764601794812504e-05, 'epoch': 0.77}

  8%|▊         | 3194/41250 [7:42:30<92:15:25,  8.73s/it][2025-04-26 11:51:07,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:51:07,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.28 | bwd_microstep: 5775.52 | bwd_inner_microstep: 5660.64 | bwd_allreduce_microstep: 114.84 | step_microstep: 18.23
[2025-04-26 11:51:07,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.28 | bwd: 5775.54 | bwd_inner: 5660.64 | bwd_allreduce: 114.85 | step: 18.24

  8%|▊         | 3195/41250 [7:42:38<92:09:05,  8.72s/it]
                                                         
{'loss': 0.1396, 'grad_norm': 1.1294375658035278, 'learning_rate': 3.9764361513700913e-05, 'epoch': 0.77}

  8%|▊         | 3195/41250 [7:42:38<92:09:05,  8.72s/it][2025-04-26 11:51:16,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:51:16,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 5713.14 | bwd_inner_microstep: 5693.44 | bwd_allreduce_microstep: 19.66 | step_microstep: 18.19
[2025-04-26 11:51:16,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5713.15 | bwd_inner: 5693.44 | bwd_allreduce: 19.68 | step: 18.20

  8%|▊         | 3196/41250 [7:42:47<91:55:35,  8.70s/it]
                                                         
{'loss': 0.0827, 'grad_norm': 1.0159337520599365, 'learning_rate': 3.9764121110745924e-05, 'epoch': 0.77}

  8%|▊         | 3196/41250 [7:42:47<91:55:35,  8.70s/it][2025-04-26 11:51:25,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 11:51:25,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.19 | bwd_microstep: 5707.99 | bwd_inner_microstep: 5652.09 | bwd_allreduce_microstep: 55.85 | step_microstep: 18.45
[2025-04-26 11:51:25,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.19 | bwd: 5708.00 | bwd_inner: 5652.09 | bwd_allreduce: 55.87 | step: 18.45

  8%|▊         | 3197/41250 [7:42:56<91:39:42,  8.67s/it]
                                                         
{'loss': 0.102, 'grad_norm': 0.6953459978103638, 'learning_rate': 3.976388058594901e-05, 'epoch': 0.78}

  8%|▊         | 3197/41250 [7:42:56<91:39:42,  8.67s/it][2025-04-26 11:51:33,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 11:51:33,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.60 | bwd_microstep: 5716.21 | bwd_inner_microstep: 5656.04 | bwd_allreduce_microstep: 60.13 | step_microstep: 18.37
[2025-04-26 11:51:33,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.60 | bwd: 5716.23 | bwd_inner: 5656.04 | bwd_allreduce: 60.15 | step: 18.37

  8%|▊         | 3198/41250 [7:43:04<91:33:30,  8.66s/it]
                                                         
{'loss': 0.3203, 'grad_norm': 1.9400640726089478, 'learning_rate': 3.9763639939311664e-05, 'epoch': 0.78}

  8%|▊         | 3198/41250 [7:43:04<91:33:30,  8.66s/it][2025-04-26 11:51:42,390] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 11:51:42,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.28 | bwd_microstep: 5750.70 | bwd_inner_microstep: 5706.14 | bwd_allreduce_microstep: 44.52 | step_microstep: 18.83
[2025-04-26 11:51:42,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.28 | bwd: 5750.72 | bwd_inner: 5706.14 | bwd_allreduce: 44.54 | step: 18.83

  8%|▊         | 3199/41250 [7:43:13<91:39:43,  8.67s/it]
                                                         
{'loss': 0.1795, 'grad_norm': 1.4012627601623535, 'learning_rate': 3.976339917083536e-05, 'epoch': 0.78}

  8%|▊         | 3199/41250 [7:43:13<91:39:43,  8.67s/it][2025-04-26 11:51:51,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 11:51:51,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.82 | bwd_microstep: 5756.96 | bwd_inner_microstep: 5652.79 | bwd_allreduce_microstep: 104.13 | step_microstep: 18.53
[2025-04-26 11:51:51,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.82 | bwd: 5756.97 | bwd_inner: 5652.79 | bwd_allreduce: 104.14 | step: 18.54

  8%|▊         | 3200/41250 [7:43:22<91:41:20,  8.67s/it]
                                                         
{'loss': 0.2515, 'grad_norm': 2.1229074001312256, 'learning_rate': 3.9763158280521595e-05, 'epoch': 0.78}

  8%|▊         | 3200/41250 [7:43:22<91:41:20,  8.67s/it][2025-04-26 11:51:59,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:51:59,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.58 | bwd_microstep: 5766.38 | bwd_inner_microstep: 5685.83 | bwd_allreduce_microstep: 80.51 | step_microstep: 18.38
[2025-04-26 11:51:59,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.58 | bwd: 5766.40 | bwd_inner: 5685.83 | bwd_allreduce: 80.53 | step: 18.39

  8%|▊         | 3201/41250 [7:43:30<91:46:25,  8.68s/it]
                                                         
{'loss': 0.0979, 'grad_norm': 0.7664952874183655, 'learning_rate': 3.9762917268371844e-05, 'epoch': 0.78}

  8%|▊         | 3201/41250 [7:43:30<91:46:25,  8.68s/it][2025-04-26 11:52:08,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.27 | optimizer_step: 0.90
[2025-04-26 11:52:08,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.32 | bwd_microstep: 5750.05 | bwd_inner_microstep: 5686.56 | bwd_allreduce_microstep: 63.45 | step_microstep: 20.54
[2025-04-26 11:52:08,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.32 | bwd: 5750.07 | bwd_inner: 5686.55 | bwd_allreduce: 63.47 | step: 20.54

  8%|▊         | 3202/41250 [7:43:39<91:47:26,  8.68s/it]
                                                         
{'loss': 0.2637, 'grad_norm': 1.5290392637252808, 'learning_rate': 3.976267613438759e-05, 'epoch': 0.78}

  8%|▊         | 3202/41250 [7:43:39<91:47:26,  8.68s/it][2025-04-26 11:52:17,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 11:52:17,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.26 | bwd_microstep: 5757.28 | bwd_inner_microstep: 5685.71 | bwd_allreduce_microstep: 71.53 | step_microstep: 18.39
[2025-04-26 11:52:17,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.26 | bwd: 5757.30 | bwd_inner: 5685.71 | bwd_allreduce: 71.55 | step: 18.40

  8%|▊         | 3203/41250 [7:43:48<91:47:51,  8.69s/it]
                                                         
{'loss': 0.0664, 'grad_norm': 0.5088503360748291, 'learning_rate': 3.976243487857033e-05, 'epoch': 0.78}

  8%|▊         | 3203/41250 [7:43:48<91:47:51,  8.69s/it][2025-04-26 11:52:25,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:52:25,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.36 | bwd_microstep: 5776.69 | bwd_inner_microstep: 5649.38 | bwd_allreduce_microstep: 127.27 | step_microstep: 18.05
[2025-04-26 11:52:25,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.36 | bwd: 5776.70 | bwd_inner: 5649.38 | bwd_allreduce: 127.28 | step: 18.05

  8%|▊         | 3204/41250 [7:43:57<91:49:46,  8.69s/it]
                                                         
{'loss': 0.193, 'grad_norm': 2.546577215194702, 'learning_rate': 3.976219350092154e-05, 'epoch': 0.78}

  8%|▊         | 3204/41250 [7:43:57<91:49:46,  8.69s/it][2025-04-26 11:52:34,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:52:34,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.91 | bwd_microstep: 5748.85 | bwd_inner_microstep: 5634.37 | bwd_allreduce_microstep: 114.44 | step_microstep: 17.75
[2025-04-26 11:52:34,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.91 | bwd: 5748.87 | bwd_inner: 5634.37 | bwd_allreduce: 114.46 | step: 17.76

  8%|▊         | 3205/41250 [7:44:05<91:45:32,  8.68s/it]
                                                         
{'loss': 0.1933, 'grad_norm': 1.2353733777999878, 'learning_rate': 3.9761952001442726e-05, 'epoch': 0.78}

  8%|▊         | 3205/41250 [7:44:05<91:45:32,  8.68s/it][2025-04-26 11:52:43,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:52:43,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.11 | bwd_microstep: 5736.34 | bwd_inner_microstep: 5677.65 | bwd_allreduce_microstep: 58.65 | step_microstep: 17.75
[2025-04-26 11:52:43,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.11 | bwd: 5736.35 | bwd_inner: 5677.65 | bwd_allreduce: 58.66 | step: 17.76

  8%|▊         | 3206/41250 [7:44:14<91:40:35,  8.68s/it]
                                                         
{'loss': 0.0895, 'grad_norm': 0.7334127426147461, 'learning_rate': 3.976171038013536e-05, 'epoch': 0.78}

  8%|▊         | 3206/41250 [7:44:14<91:40:35,  8.68s/it][2025-04-26 11:52:51,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:52:51,866] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.96 | bwd_microstep: 5783.39 | bwd_inner_microstep: 5643.29 | bwd_allreduce_microstep: 140.06 | step_microstep: 18.20
[2025-04-26 11:52:51,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.96 | bwd: 5783.40 | bwd_inner: 5643.29 | bwd_allreduce: 140.08 | step: 18.20

  8%|▊         | 3207/41250 [7:44:23<91:43:36,  8.68s/it]
                                                         
{'loss': 0.1323, 'grad_norm': 1.8086999654769897, 'learning_rate': 3.9761468637000924e-05, 'epoch': 0.78}

  8%|▊         | 3207/41250 [7:44:23<91:43:36,  8.68s/it][2025-04-26 11:53:00,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:53:00,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.68 | bwd_microstep: 5705.69 | bwd_inner_microstep: 5692.99 | bwd_allreduce_microstep: 12.65 | step_microstep: 19.27
[2025-04-26 11:53:00,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.68 | bwd: 5705.71 | bwd_inner: 5692.99 | bwd_allreduce: 12.67 | step: 19.27

  8%|▊         | 3208/41250 [7:44:31<91:36:00,  8.67s/it]
                                                         
{'loss': 0.2009, 'grad_norm': 1.6813452243804932, 'learning_rate': 3.976122677204093e-05, 'epoch': 0.78}

  8%|▊         | 3208/41250 [7:44:31<91:36:00,  8.67s/it][2025-04-26 11:53:09,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:53:09,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.01 | bwd_microstep: 5751.03 | bwd_inner_microstep: 5648.04 | bwd_allreduce_microstep: 102.95 | step_microstep: 18.13
[2025-04-26 11:53:09,169] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.01 | bwd: 5751.04 | bwd_inner: 5648.04 | bwd_allreduce: 102.97 | step: 18.14

  8%|▊         | 3209/41250 [7:44:40<91:34:31,  8.67s/it]
                                                         
{'loss': 0.2449, 'grad_norm': 1.4789018630981445, 'learning_rate': 3.9760984785256866e-05, 'epoch': 0.78}

  8%|▊         | 3209/41250 [7:44:40<91:34:31,  8.67s/it][2025-04-26 11:53:17,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 11:53:17,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.27 | bwd_microstep: 5697.61 | bwd_inner_microstep: 5684.77 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.69
[2025-04-26 11:53:17,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.27 | bwd: 5697.62 | bwd_inner: 5684.77 | bwd_allreduce: 12.81 | step: 18.71

  8%|▊         | 3210/41250 [7:44:48<91:29:52,  8.66s/it]
                                                         
{'loss': 0.2259, 'grad_norm': 1.9097908735275269, 'learning_rate': 3.9760742676650205e-05, 'epoch': 0.78}

  8%|▊         | 3210/41250 [7:44:49<91:29:52,  8.66s/it][2025-04-26 11:53:26,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:53:26,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.09 | bwd_microstep: 5757.52 | bwd_inner_microstep: 5688.49 | bwd_allreduce_microstep: 68.99 | step_microstep: 18.31
[2025-04-26 11:53:26,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.09 | bwd: 5757.54 | bwd_inner: 5688.49 | bwd_allreduce: 69.01 | step: 18.32

  8%|▊         | 3211/41250 [7:44:57<91:35:23,  8.67s/it]
                                                         
{'loss': 0.1054, 'grad_norm': 0.8760578036308289, 'learning_rate': 3.9760500446222466e-05, 'epoch': 0.78}

  8%|▊         | 3211/41250 [7:44:57<91:35:23,  8.67s/it][2025-04-26 11:53:35,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.11 | optimizer_step: 1.08
[2025-04-26 11:53:35,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.77 | bwd_microstep: 5754.34 | bwd_inner_microstep: 5690.42 | bwd_allreduce_microstep: 63.86 | step_microstep: 19.57
[2025-04-26 11:53:35,191] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.77 | bwd: 5754.35 | bwd_inner: 5690.42 | bwd_allreduce: 63.88 | step: 19.58

  8%|▊         | 3212/41250 [7:45:06<91:39:45,  8.68s/it]
                                                         
{'loss': 0.2984, 'grad_norm': 1.8540562391281128, 'learning_rate': 3.976025809397512e-05, 'epoch': 0.78}

  8%|▊         | 3212/41250 [7:45:06<91:39:45,  8.68s/it][2025-04-26 11:53:43,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.91 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 11:53:43,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.01 | bwd_microstep: 5752.33 | bwd_inner_microstep: 5676.17 | bwd_allreduce_microstep: 76.11 | step_microstep: 19.57
[2025-04-26 11:53:43,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.01 | bwd: 5752.34 | bwd_inner: 5676.17 | bwd_allreduce: 76.13 | step: 19.57

  8%|▊         | 3213/41250 [7:45:15<91:41:23,  8.68s/it]
                                                         
{'loss': 0.144, 'grad_norm': 1.40705144405365, 'learning_rate': 3.976001561990967e-05, 'epoch': 0.78}

  8%|▊         | 3213/41250 [7:45:15<91:41:23,  8.68s/it][2025-04-26 11:53:52,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:53:52,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.79 | bwd_microstep: 5733.36 | bwd_inner_microstep: 5690.05 | bwd_allreduce_microstep: 43.25 | step_microstep: 18.59
[2025-04-26 11:53:52,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.79 | bwd: 5733.38 | bwd_inner: 5690.05 | bwd_allreduce: 43.28 | step: 18.59

  8%|▊         | 3214/41250 [7:45:23<91:40:49,  8.68s/it]
                                                         
{'loss': 0.1842, 'grad_norm': 1.051863431930542, 'learning_rate': 3.975977302402761e-05, 'epoch': 0.78}

  8%|▊         | 3214/41250 [7:45:23<91:40:49,  8.68s/it][2025-04-26 11:54:01,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 11:54:01,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.42 | bwd_microstep: 5701.99 | bwd_inner_microstep: 5688.93 | bwd_allreduce_microstep: 13.01 | step_microstep: 18.83
[2025-04-26 11:54:01,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.42 | bwd: 5702.00 | bwd_inner: 5688.93 | bwd_allreduce: 13.03 | step: 18.83

  8%|▊         | 3215/41250 [7:45:32<91:30:44,  8.66s/it]
                                                         
{'loss': 0.1972, 'grad_norm': 1.4304686784744263, 'learning_rate': 3.9759530306330434e-05, 'epoch': 0.78}

  8%|▊         | 3215/41250 [7:45:32<91:30:44,  8.66s/it][2025-04-26 11:54:09,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 11:54:09,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.60 | bwd_microstep: 5771.20 | bwd_inner_microstep: 5690.01 | bwd_allreduce_microstep: 81.14 | step_microstep: 18.57
[2025-04-26 11:54:09,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.60 | bwd: 5771.21 | bwd_inner: 5690.01 | bwd_allreduce: 81.17 | step: 18.58

  8%|▊         | 3216/41250 [7:45:41<91:37:55,  8.67s/it]
                                                         
{'loss': 0.2828, 'grad_norm': 1.5699522495269775, 'learning_rate': 3.9759287466819646e-05, 'epoch': 0.78}

  8%|▊         | 3216/41250 [7:45:41<91:37:55,  8.67s/it][2025-04-26 11:54:18,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 11:54:18,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.92 | bwd_microstep: 5741.65 | bwd_inner_microstep: 5693.20 | bwd_allreduce_microstep: 48.39 | step_microstep: 18.97
[2025-04-26 11:54:18,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.92 | bwd: 5741.66 | bwd_inner: 5693.20 | bwd_allreduce: 48.42 | step: 18.98

  8%|▊         | 3217/41250 [7:45:49<91:38:00,  8.67s/it]
                                                         
{'loss': 0.2175, 'grad_norm': 1.5340547561645508, 'learning_rate': 3.975904450549673e-05, 'epoch': 0.78}

  8%|▊         | 3217/41250 [7:45:49<91:38:00,  8.67s/it][2025-04-26 11:54:27,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:54:27,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.67 | bwd_microstep: 5737.89 | bwd_inner_microstep: 5682.41 | bwd_allreduce_microstep: 55.43 | step_microstep: 18.56
[2025-04-26 11:54:27,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.67 | bwd: 5737.91 | bwd_inner: 5682.41 | bwd_allreduce: 55.45 | step: 18.56

  8%|▊         | 3218/41250 [7:45:58<91:38:31,  8.67s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 1.1444909572601318, 'learning_rate': 3.97588014223632e-05, 'epoch': 0.78}

  8%|▊         | 3218/41250 [7:45:58<91:38:31,  8.67s/it][2025-04-26 11:54:35,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 11:54:35,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.11 | bwd_microstep: 5773.36 | bwd_inner_microstep: 5640.49 | bwd_allreduce_microstep: 132.83 | step_microstep: 18.57
[2025-04-26 11:54:35,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.11 | bwd: 5773.37 | bwd_inner: 5640.49 | bwd_allreduce: 132.84 | step: 18.57

  8%|▊         | 3219/41250 [7:46:07<91:39:15,  8.68s/it]
                                                         
{'loss': 0.3029, 'grad_norm': 2.437070846557617, 'learning_rate': 3.975855821742054e-05, 'epoch': 0.78}

  8%|▊         | 3219/41250 [7:46:07<91:39:15,  8.68s/it][2025-04-26 11:54:44,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 11:54:44,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.46 | bwd_microstep: 5700.55 | bwd_inner_microstep: 5648.44 | bwd_allreduce_microstep: 52.07 | step_microstep: 18.62
[2025-04-26 11:54:44,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.46 | bwd: 5700.56 | bwd_inner: 5648.44 | bwd_allreduce: 52.09 | step: 18.62

  8%|▊         | 3220/41250 [7:46:15<91:24:20,  8.65s/it]
                                                         
{'loss': 0.0879, 'grad_norm': 1.1763536930084229, 'learning_rate': 3.9758314890670255e-05, 'epoch': 0.78}

  8%|▊         | 3220/41250 [7:46:15<91:24:20,  8.65s/it][2025-04-26 11:54:53,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 11:54:53,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.74 | bwd_microstep: 5769.20 | bwd_inner_microstep: 5699.66 | bwd_allreduce_microstep: 69.49 | step_microstep: 18.69
[2025-04-26 11:54:53,206] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.74 | bwd: 5769.21 | bwd_inner: 5699.66 | bwd_allreduce: 69.51 | step: 18.69

  8%|▊         | 3221/41250 [7:46:24<91:33:18,  8.67s/it]
                                                         
{'loss': 0.1018, 'grad_norm': 1.2490042448043823, 'learning_rate': 3.975807144211384e-05, 'epoch': 0.78}

  8%|▊         | 3221/41250 [7:46:24<91:33:18,  8.67s/it][2025-04-26 11:55:01,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 11:55:01,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.64 | bwd_microstep: 5718.89 | bwd_inner_microstep: 5659.65 | bwd_allreduce_microstep: 59.20 | step_microstep: 18.82
[2025-04-26 11:55:01,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.64 | bwd: 5718.90 | bwd_inner: 5659.65 | bwd_allreduce: 59.21 | step: 18.82

  8%|▊         | 3222/41250 [7:46:33<91:26:15,  8.66s/it]
                                                         
{'loss': 0.1875, 'grad_norm': 1.0012890100479126, 'learning_rate': 3.975782787175281e-05, 'epoch': 0.78}

  8%|▊         | 3222/41250 [7:46:33<91:26:15,  8.66s/it][2025-04-26 11:55:10,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 11:55:10,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.20 | bwd_microstep: 5816.75 | bwd_inner_microstep: 5783.81 | bwd_allreduce_microstep: 32.90 | step_microstep: 18.57
[2025-04-26 11:55:10,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.20 | bwd: 5816.77 | bwd_inner: 5783.81 | bwd_allreduce: 32.92 | step: 18.57

  8%|▊         | 3223/41250 [7:46:41<91:50:35,  8.69s/it]
                                                         
{'loss': 0.328, 'grad_norm': 2.929629325866699, 'learning_rate': 3.9757584179588654e-05, 'epoch': 0.78}

  8%|▊         | 3223/41250 [7:46:41<91:50:35,  8.69s/it][2025-04-26 11:55:19,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.32 | optimizer_step: 1.04
[2025-04-26 11:55:19,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.33 | bwd_microstep: 5728.58 | bwd_inner_microstep: 5660.32 | bwd_allreduce_microstep: 68.19 | step_microstep: 19.85
[2025-04-26 11:55:19,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.33 | bwd: 5728.59 | bwd_inner: 5660.32 | bwd_allreduce: 68.23 | step: 19.85

  8%|▊         | 3224/41250 [7:46:50<91:38:49,  8.68s/it]
                                                         
{'loss': 0.3319, 'grad_norm': 1.555503487586975, 'learning_rate': 3.975734036562288e-05, 'epoch': 0.78}

  8%|▊         | 3224/41250 [7:46:50<91:38:49,  8.68s/it][2025-04-26 11:55:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 11:55:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5757.08 | bwd_inner_microstep: 5688.99 | bwd_allreduce_microstep: 68.05 | step_microstep: 18.28
[2025-04-26 11:55:27,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5757.09 | bwd_inner: 5688.99 | bwd_allreduce: 68.06 | step: 18.29

  8%|▊         | 3225/41250 [7:46:59<91:41:14,  8.68s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 1.0005388259887695, 'learning_rate': 3.9757096429856984e-05, 'epoch': 0.78}

  8%|▊         | 3225/41250 [7:46:59<91:41:14,  8.68s/it][2025-04-26 11:55:36,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 11:55:36,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.19 | bwd_microstep: 5763.67 | bwd_inner_microstep: 5701.84 | bwd_allreduce_microstep: 61.79 | step_microstep: 17.63
[2025-04-26 11:55:36,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.20 | bwd: 5763.69 | bwd_inner: 5701.84 | bwd_allreduce: 61.80 | step: 17.63

  8%|▊         | 3226/41250 [7:47:07<91:44:20,  8.69s/it]
                                                         
{'loss': 0.2545, 'grad_norm': 1.86863112449646, 'learning_rate': 3.9756852372292475e-05, 'epoch': 0.78}

  8%|▊         | 3226/41250 [7:47:07<91:44:20,  8.69s/it][2025-04-26 11:55:45,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:55:45,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.95 | bwd_microstep: 5785.88 | bwd_inner_microstep: 5773.28 | bwd_allreduce_microstep: 12.56 | step_microstep: 17.69
[2025-04-26 11:55:45,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.95 | bwd: 5785.89 | bwd_inner: 5773.28 | bwd_allreduce: 12.58 | step: 17.69

  8%|▊         | 3227/41250 [7:47:16<91:59:50,  8.71s/it]
                                                         
{'loss': 0.3118, 'grad_norm': 1.7288284301757812, 'learning_rate': 3.975660819293086e-05, 'epoch': 0.78}

  8%|▊         | 3227/41250 [7:47:16<91:59:50,  8.71s/it][2025-04-26 11:55:54,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 11:55:54,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.16 | bwd_microstep: 5746.83 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 52.92 | step_microstep: 17.89
[2025-04-26 11:55:54,099] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.16 | bwd: 5746.85 | bwd_inner: 5693.87 | bwd_allreduce: 52.94 | step: 17.90

  8%|▊         | 3228/41250 [7:47:25<91:55:24,  8.70s/it]
                                                         
{'loss': 0.1767, 'grad_norm': 1.9545092582702637, 'learning_rate': 3.9756363891773644e-05, 'epoch': 0.78}

  8%|▊         | 3228/41250 [7:47:25<91:55:24,  8.70s/it][2025-04-26 11:56:02,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 11:56:02,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.97 | bwd_microstep: 5764.83 | bwd_inner_microstep: 5709.18 | bwd_allreduce_microstep: 55.60 | step_microstep: 18.06
[2025-04-26 11:56:02,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.97 | bwd: 5764.85 | bwd_inner: 5709.18 | bwd_allreduce: 55.62 | step: 18.06

  8%|▊         | 3229/41250 [7:47:33<91:54:31,  8.70s/it]
                                                         
{'loss': 0.1101, 'grad_norm': 2.383923053741455, 'learning_rate': 3.9756119468822324e-05, 'epoch': 0.78}

  8%|▊         | 3229/41250 [7:47:33<91:54:31,  8.70s/it][2025-04-26 11:56:11,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:56:11,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.12 | bwd_microstep: 5728.26 | bwd_inner_microstep: 5715.36 | bwd_allreduce_microstep: 12.86 | step_microstep: 17.83
[2025-04-26 11:56:11,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.12 | bwd: 5728.27 | bwd_inner: 5715.36 | bwd_allreduce: 12.88 | step: 17.83

  8%|▊         | 3230/41250 [7:47:42<91:47:40,  8.69s/it]
                                                         
{'loss': 0.0675, 'grad_norm': 0.465452641248703, 'learning_rate': 3.975587492407842e-05, 'epoch': 0.78}

  8%|▊         | 3230/41250 [7:47:42<91:47:40,  8.69s/it][2025-04-26 11:56:20,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:56:20,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.13 | bwd_microstep: 5808.60 | bwd_inner_microstep: 5656.28 | bwd_allreduce_microstep: 152.28 | step_microstep: 18.12
[2025-04-26 11:56:20,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.13 | bwd: 5808.61 | bwd_inner: 5656.28 | bwd_allreduce: 152.30 | step: 18.13

  8%|▊         | 3231/41250 [7:47:51<91:53:31,  8.70s/it]
                                                         
{'loss': 0.1567, 'grad_norm': 1.2312977313995361, 'learning_rate': 3.975563025754342e-05, 'epoch': 0.78}

  8%|▊         | 3231/41250 [7:47:51<91:53:31,  8.70s/it][2025-04-26 11:56:28,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:56:28,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.59 | bwd_microstep: 5782.15 | bwd_inner_microstep: 5769.17 | bwd_allreduce_microstep: 12.93 | step_microstep: 17.89
[2025-04-26 11:56:28,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.59 | bwd: 5782.16 | bwd_inner: 5769.17 | bwd_allreduce: 12.95 | step: 17.89

  8%|▊         | 3232/41250 [7:48:00<92:04:10,  8.72s/it]
                                                         
{'loss': 0.0876, 'grad_norm': 0.7201403379440308, 'learning_rate': 3.975538546921886e-05, 'epoch': 0.78}

  8%|▊         | 3232/41250 [7:48:00<92:04:10,  8.72s/it][2025-04-26 11:56:37,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:56:37,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.71 | bwd_microstep: 5750.94 | bwd_inner_microstep: 5695.58 | bwd_allreduce_microstep: 55.32 | step_microstep: 17.74
[2025-04-26 11:56:37,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.71 | bwd: 5750.95 | bwd_inner: 5695.58 | bwd_allreduce: 55.34 | step: 17.74

  8%|▊         | 3233/41250 [7:48:08<91:57:52,  8.71s/it]
                                                         
{'loss': 0.2171, 'grad_norm': 1.5191739797592163, 'learning_rate': 3.9755140559106226e-05, 'epoch': 0.78}

  8%|▊         | 3233/41250 [7:48:08<91:57:52,  8.71s/it][2025-04-26 11:56:46,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 1.09
[2025-04-26 11:56:46,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.91 | bwd_microstep: 5789.27 | bwd_inner_microstep: 5655.96 | bwd_allreduce_microstep: 133.26 | step_microstep: 18.97
[2025-04-26 11:56:46,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.91 | bwd: 5789.28 | bwd_inner: 5655.96 | bwd_allreduce: 133.28 | step: 18.98

  8%|▊         | 3234/41250 [7:48:17<91:56:07,  8.71s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 0.6134559512138367, 'learning_rate': 3.9754895527207036e-05, 'epoch': 0.78}

  8%|▊         | 3234/41250 [7:48:17<91:56:07,  8.71s/it][2025-04-26 11:56:54,976] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:56:54,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5712.04 | bwd_inner_microstep: 5699.17 | bwd_allreduce_microstep: 12.83 | step_microstep: 17.91
[2025-04-26 11:56:54,977] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5712.05 | bwd_inner: 5699.17 | bwd_allreduce: 12.84 | step: 17.92

  8%|▊         | 3235/41250 [7:48:26<91:44:08,  8.69s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 2.3951187133789062, 'learning_rate': 3.9754650373522804e-05, 'epoch': 0.78}

  8%|▊         | 3235/41250 [7:48:26<91:44:08,  8.69s/it][2025-04-26 11:57:03,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:57:03,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.08 | bwd_microstep: 5712.75 | bwd_inner_microstep: 5700.06 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.69
[2025-04-26 11:57:03,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5712.76 | bwd_inner: 5700.06 | bwd_allreduce: 12.66 | step: 17.69

  8%|▊         | 3236/41250 [7:48:34<91:37:35,  8.68s/it]
                                                         
{'loss': 0.2463, 'grad_norm': 1.880125880241394, 'learning_rate': 3.9754405098055035e-05, 'epoch': 0.78}

  8%|▊         | 3236/41250 [7:48:34<91:37:35,  8.68s/it][2025-04-26 11:57:12,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.91 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 11:57:12,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.84 | bwd_microstep: 5723.53 | bwd_inner_microstep: 5665.72 | bwd_allreduce_microstep: 57.77 | step_microstep: 17.40
[2025-04-26 11:57:12,271] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.84 | bwd: 5723.54 | bwd_inner: 5665.71 | bwd_allreduce: 57.79 | step: 17.40

  8%|▊         | 3237/41250 [7:48:43<91:30:29,  8.67s/it]
                                                         
{'loss': 0.1862, 'grad_norm': 2.1774332523345947, 'learning_rate': 3.975415970080525e-05, 'epoch': 0.78}

  8%|▊         | 3237/41250 [7:48:43<91:30:29,  8.67s/it][2025-04-26 11:57:20,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:57:20,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.03 | bwd_microstep: 5753.88 | bwd_inner_microstep: 5701.33 | bwd_allreduce_microstep: 52.51 | step_microstep: 18.41
[2025-04-26 11:57:20,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.03 | bwd: 5753.89 | bwd_inner: 5701.33 | bwd_allreduce: 52.53 | step: 18.41

  8%|▊         | 3238/41250 [7:48:52<91:35:04,  8.67s/it]
                                                         
{'loss': 0.1948, 'grad_norm': 1.1894927024841309, 'learning_rate': 3.975391418177495e-05, 'epoch': 0.78}

  8%|▊         | 3238/41250 [7:48:52<91:35:04,  8.67s/it][2025-04-26 11:57:29,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:57:29,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.88 | bwd_microstep: 5793.32 | bwd_inner_microstep: 5656.67 | bwd_allreduce_microstep: 136.61 | step_microstep: 17.84
[2025-04-26 11:57:29,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.88 | bwd: 5793.34 | bwd_inner: 5656.67 | bwd_allreduce: 136.63 | step: 17.84

  8%|▊         | 3239/41250 [7:49:00<91:40:10,  8.68s/it]
                                                         
{'loss': 0.2752, 'grad_norm': 1.9709358215332031, 'learning_rate': 3.975366854096565e-05, 'epoch': 0.79}

  8%|▊         | 3239/41250 [7:49:00<91:40:10,  8.68s/it][2025-04-26 11:57:38,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 11:57:38,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5777.67 | bwd_inner_microstep: 5693.01 | bwd_allreduce_microstep: 84.61 | step_microstep: 17.81
[2025-04-26 11:57:38,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5777.68 | bwd_inner: 5693.01 | bwd_allreduce: 84.63 | step: 17.82

  8%|▊         | 3240/41250 [7:49:09<91:45:30,  8.69s/it]
                                                         
{'loss': 0.1196, 'grad_norm': 1.4082221984863281, 'learning_rate': 3.9753422778378885e-05, 'epoch': 0.79}

  8%|▊         | 3240/41250 [7:49:09<91:45:30,  8.69s/it][2025-04-26 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.83 | bwd_microstep: 5890.57 | bwd_inner_microstep: 5682.61 | bwd_allreduce_microstep: 207.92 | step_microstep: 17.75
[2025-04-26 11:57:47,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.83 | bwd: 5890.59 | bwd_inner: 5682.61 | bwd_allreduce: 207.94 | step: 17.77

  8%|▊         | 3241/41250 [7:49:18<92:08:36,  8.73s/it]
                                                         
{'loss': 0.2177, 'grad_norm': 1.2711585760116577, 'learning_rate': 3.975317689401614e-05, 'epoch': 0.79}

  8%|▊         | 3241/41250 [7:49:18<92:08:36,  8.73s/it][2025-04-26 11:57:55,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:57:55,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.99 | bwd_microstep: 5722.00 | bwd_inner_microstep: 5653.72 | bwd_allreduce_microstep: 68.24 | step_microstep: 18.14
[2025-04-26 11:57:55,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.00 | bwd: 5722.01 | bwd_inner: 5653.71 | bwd_allreduce: 68.26 | step: 18.14

  8%|▊         | 3242/41250 [7:49:26<91:48:31,  8.70s/it]
                                                         
{'loss': 0.1305, 'grad_norm': 1.3487690687179565, 'learning_rate': 3.9752930887878955e-05, 'epoch': 0.79}

  8%|▊         | 3242/41250 [7:49:26<91:48:31,  8.70s/it][2025-04-26 11:58:04,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 11:58:04,434] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.47 | bwd_microstep: 5718.22 | bwd_inner_microstep: 5644.99 | bwd_allreduce_microstep: 73.19 | step_microstep: 18.05
[2025-04-26 11:58:04,435] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.47 | bwd: 5718.23 | bwd_inner: 5644.99 | bwd_allreduce: 73.21 | step: 18.06

  8%|▊         | 3243/41250 [7:49:35<91:35:00,  8.67s/it]
                                                         
{'loss': 0.1055, 'grad_norm': 0.8492683172225952, 'learning_rate': 3.975268475996882e-05, 'epoch': 0.79}

  8%|▊         | 3243/41250 [7:49:35<91:35:00,  8.67s/it][2025-04-26 11:58:13,062] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:58:13,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.14 | bwd_microstep: 5721.36 | bwd_inner_microstep: 5659.31 | bwd_allreduce_microstep: 61.99 | step_microstep: 17.90
[2025-04-26 11:58:13,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.14 | bwd: 5721.37 | bwd_inner: 5659.31 | bwd_allreduce: 62.01 | step: 17.90

  8%|▊         | 3244/41250 [7:49:44<91:26:02,  8.66s/it]
                                                         
{'loss': 0.1565, 'grad_norm': 1.8035744428634644, 'learning_rate': 3.975243851028729e-05, 'epoch': 0.79}

  8%|▊         | 3244/41250 [7:49:44<91:26:02,  8.66s/it][2025-04-26 11:58:21,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.68 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 11:58:21,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.67 | bwd_microstep: 5762.60 | bwd_inner_microstep: 5686.63 | bwd_allreduce_microstep: 75.93 | step_microstep: 18.49
[2025-04-26 11:58:21,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.67 | bwd: 5762.61 | bwd_inner: 5686.63 | bwd_allreduce: 75.95 | step: 18.50

  8%|▊         | 3245/41250 [7:49:52<91:31:13,  8.67s/it]
                                                         
{'loss': 0.2087, 'grad_norm': 2.023921489715576, 'learning_rate': 3.975219213883584e-05, 'epoch': 0.79}

  8%|▊         | 3245/41250 [7:49:52<91:31:13,  8.67s/it][2025-04-26 11:58:30,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:58:30,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.14 | bwd_microstep: 5692.55 | bwd_inner_microstep: 5679.34 | bwd_allreduce_microstep: 13.16 | step_microstep: 17.79
[2025-04-26 11:58:30,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5692.56 | bwd_inner: 5679.34 | bwd_allreduce: 13.18 | step: 17.79

  8%|▊         | 3246/41250 [7:50:01<91:21:35,  8.65s/it]
                                                         
{'loss': 0.172, 'grad_norm': 1.9356259107589722, 'learning_rate': 3.9751945645616025e-05, 'epoch': 0.79}

  8%|▊         | 3246/41250 [7:50:01<91:21:35,  8.65s/it][2025-04-26 11:58:38,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 11:58:38,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.84 | bwd_microstep: 5704.38 | bwd_inner_microstep: 5691.73 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.22
[2025-04-26 11:58:38,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.84 | bwd: 5704.39 | bwd_inner: 5691.73 | bwd_allreduce: 12.63 | step: 18.22

  8%|▊         | 3247/41250 [7:50:10<91:15:25,  8.64s/it]
                                                         
{'loss': 0.1383, 'grad_norm': 0.9813082814216614, 'learning_rate': 3.975169903062934e-05, 'epoch': 0.79}

  8%|▊         | 3247/41250 [7:50:10<91:15:25,  8.64s/it][2025-04-26 11:58:47,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:58:47,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.48 | bwd_microstep: 5721.34 | bwd_inner_microstep: 5691.65 | bwd_allreduce_microstep: 29.65 | step_microstep: 18.29
[2025-04-26 11:58:47,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.48 | bwd: 5721.35 | bwd_inner: 5691.65 | bwd_allreduce: 29.67 | step: 18.29

  8%|▊         | 3248/41250 [7:50:18<91:17:24,  8.65s/it]
                                                         
{'loss': 0.1593, 'grad_norm': 1.1369115114212036, 'learning_rate': 3.9751452293877325e-05, 'epoch': 0.79}

  8%|▊         | 3248/41250 [7:50:18<91:17:24,  8.65s/it][2025-04-26 11:58:56,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:58:56,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.58 | bwd_microstep: 5713.85 | bwd_inner_microstep: 5701.03 | bwd_allreduce_microstep: 12.78 | step_microstep: 17.67
[2025-04-26 11:58:56,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.58 | bwd: 5713.86 | bwd_inner: 5701.03 | bwd_allreduce: 12.79 | step: 17.67

  8%|▊         | 3249/41250 [7:50:27<91:16:36,  8.65s/it]
                                                         
{'loss': 0.1991, 'grad_norm': 1.058956503868103, 'learning_rate': 3.975120543536149e-05, 'epoch': 0.79}

  8%|▊         | 3249/41250 [7:50:27<91:16:36,  8.65s/it][2025-04-26 11:59:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:59:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.69 | bwd_microstep: 5709.56 | bwd_inner_microstep: 5650.91 | bwd_allreduce_microstep: 58.62 | step_microstep: 17.84
[2025-04-26 11:59:04,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.69 | bwd: 5709.58 | bwd_inner: 5650.90 | bwd_allreduce: 58.63 | step: 17.85

  8%|▊         | 3250/41250 [7:50:36<91:10:30,  8.64s/it]
                                                         
{'loss': 0.125, 'grad_norm': 1.0156331062316895, 'learning_rate': 3.975095845508335e-05, 'epoch': 0.79}

  8%|▊         | 3250/41250 [7:50:36<91:10:30,  8.64s/it][2025-04-26 11:59:13,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 11:59:13,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2816.10 | bwd_microstep: 5756.61 | bwd_inner_microstep: 5626.83 | bwd_allreduce_microstep: 129.74 | step_microstep: 19.02
[2025-04-26 11:59:13,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2816.10 | bwd: 5756.62 | bwd_inner: 5626.83 | bwd_allreduce: 129.76 | step: 19.03

  8%|▊         | 3251/41250 [7:50:44<91:11:49,  8.64s/it]
                                                         
{'loss': 0.1577, 'grad_norm': 0.9856624007225037, 'learning_rate': 3.9750711353044446e-05, 'epoch': 0.79}

  8%|▊         | 3251/41250 [7:50:44<91:11:49,  8.64s/it][2025-04-26 11:59:22,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 11:59:22,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.71 | bwd_microstep: 5864.73 | bwd_inner_microstep: 5656.50 | bwd_allreduce_microstep: 208.19 | step_microstep: 18.07
[2025-04-26 11:59:22,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.71 | bwd: 5864.74 | bwd_inner: 5656.50 | bwd_allreduce: 208.21 | step: 18.07

  8%|▊         | 3252/41250 [7:50:53<91:38:38,  8.68s/it]
                                                         
{'loss': 0.3572, 'grad_norm': 1.7095048427581787, 'learning_rate': 3.975046412924629e-05, 'epoch': 0.79}

  8%|▊         | 3252/41250 [7:50:53<91:38:38,  8.68s/it][2025-04-26 11:59:30,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:59:30,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.51 | bwd_microstep: 5689.91 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 39.73 | step_microstep: 18.08
[2025-04-26 11:59:30,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.51 | bwd: 5689.93 | bwd_inner: 5650.14 | bwd_allreduce: 39.75 | step: 18.09

  8%|▊         | 3253/41250 [7:51:02<91:23:03,  8.66s/it]
                                                         
{'loss': 0.1226, 'grad_norm': 0.9787582159042358, 'learning_rate': 3.97502167836904e-05, 'epoch': 0.79}

  8%|▊         | 3253/41250 [7:51:02<91:23:03,  8.66s/it][2025-04-26 11:59:39,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:59:39,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5704.58 | bwd_inner_microstep: 5691.96 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.19
[2025-04-26 11:59:39,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5704.60 | bwd_inner: 5691.96 | bwd_allreduce: 12.60 | step: 18.20

  8%|▊         | 3254/41250 [7:51:10<91:18:36,  8.65s/it]
                                                         
{'loss': 0.124, 'grad_norm': 1.1321022510528564, 'learning_rate': 3.974996931637831e-05, 'epoch': 0.79}

  8%|▊         | 3254/41250 [7:51:10<91:18:36,  8.65s/it][2025-04-26 11:59:48,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 11:59:48,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.46 | bwd_microstep: 5790.57 | bwd_inner_microstep: 5777.93 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.16
[2025-04-26 11:59:48,331] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.46 | bwd: 5790.59 | bwd_inner: 5777.93 | bwd_allreduce: 12.62 | step: 18.17

  8%|▊         | 3255/41250 [7:51:19<91:38:24,  8.68s/it]
                                                         
{'loss': 0.1916, 'grad_norm': 1.49332857131958, 'learning_rate': 3.974972172731155e-05, 'epoch': 0.79}

  8%|▊         | 3255/41250 [7:51:19<91:38:24,  8.68s/it][2025-04-26 11:59:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 11:59:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.15 | bwd_microstep: 5872.04 | bwd_inner_microstep: 5701.87 | bwd_allreduce_microstep: 170.13 | step_microstep: 17.80
[2025-04-26 11:59:57,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.16 | bwd: 5872.05 | bwd_inner: 5701.87 | bwd_allreduce: 170.14 | step: 17.81

  8%|▊         | 3256/41250 [7:51:28<92:01:46,  8.72s/it]
                                                         
{'loss': 0.2311, 'grad_norm': 1.6879104375839233, 'learning_rate': 3.9749474016491635e-05, 'epoch': 0.79}

  8%|▊         | 3256/41250 [7:51:28<92:01:46,  8.72s/it][2025-04-26 12:00:05,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.11 | optimizer_step: 0.90
[2025-04-26 12:00:05,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.99 | bwd_microstep: 5714.71 | bwd_inner_microstep: 5637.88 | bwd_allreduce_microstep: 76.80 | step_microstep: 18.55
[2025-04-26 12:00:05,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.99 | bwd: 5714.73 | bwd_inner: 5637.88 | bwd_allreduce: 76.81 | step: 18.56

  8%|▊         | 3257/41250 [7:51:36<91:45:48,  8.69s/it]
                                                         
{'loss': 0.2648, 'grad_norm': 1.11090886592865, 'learning_rate': 3.97492261839201e-05, 'epoch': 0.79}

  8%|▊         | 3257/41250 [7:51:36<91:45:48,  8.69s/it][2025-04-26 12:00:14,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:00:14,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.81 | bwd_microstep: 5708.83 | bwd_inner_microstep: 5696.22 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.07
[2025-04-26 12:00:14,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.81 | bwd: 5708.84 | bwd_inner: 5696.22 | bwd_allreduce: 12.58 | step: 18.07

  8%|▊         | 3258/41250 [7:51:45<91:34:55,  8.68s/it]
                                                         
{'loss': 0.1191, 'grad_norm': 1.0956751108169556, 'learning_rate': 3.974897822959847e-05, 'epoch': 0.79}

  8%|▊         | 3258/41250 [7:51:45<91:34:55,  8.68s/it][2025-04-26 12:00:23,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:00:23,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.63 | bwd_microstep: 5706.58 | bwd_inner_microstep: 5648.65 | bwd_allreduce_microstep: 57.89 | step_microstep: 17.72
[2025-04-26 12:00:23,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.63 | bwd: 5706.59 | bwd_inner: 5648.65 | bwd_allreduce: 57.91 | step: 17.72

  8%|▊         | 3259/41250 [7:51:54<91:25:40,  8.66s/it]
                                                         
{'loss': 0.2445, 'grad_norm': 2.11946964263916, 'learning_rate': 3.9748730153528276e-05, 'epoch': 0.79}

  8%|▊         | 3259/41250 [7:51:54<91:25:40,  8.66s/it][2025-04-26 12:00:31,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 12:00:31,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5776.53 | bwd_inner_microstep: 5631.95 | bwd_allreduce_microstep: 144.52 | step_microstep: 18.83
[2025-04-26 12:00:31,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5776.54 | bwd_inner: 5631.95 | bwd_allreduce: 144.54 | step: 18.84

  8%|▊         | 3260/41250 [7:52:02<91:28:36,  8.67s/it]
                                                         
{'loss': 0.1421, 'grad_norm': 1.6509220600128174, 'learning_rate': 3.974848195571104e-05, 'epoch': 0.79}

  8%|▊         | 3260/41250 [7:52:02<91:28:36,  8.67s/it][2025-04-26 12:00:40,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 12:00:40,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.70 | bwd_microstep: 5778.65 | bwd_inner_microstep: 5645.58 | bwd_allreduce_microstep: 133.02 | step_microstep: 18.97
[2025-04-26 12:00:40,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.70 | bwd: 5778.66 | bwd_inner: 5645.58 | bwd_allreduce: 133.04 | step: 18.97

  8%|▊         | 3261/41250 [7:52:11<91:32:22,  8.67s/it]
                                                         
{'loss': 0.2174, 'grad_norm': 1.7492573261260986, 'learning_rate': 3.9748233636148296e-05, 'epoch': 0.79}

  8%|▊         | 3261/41250 [7:52:11<91:32:22,  8.67s/it][2025-04-26 12:00:49,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 12:00:49,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.31 | bwd_microstep: 5772.90 | bwd_inner_microstep: 5709.63 | bwd_allreduce_microstep: 63.22 | step_microstep: 18.52
[2025-04-26 12:00:49,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.31 | bwd: 5772.91 | bwd_inner: 5709.63 | bwd_allreduce: 63.24 | step: 18.53

  8%|▊         | 3262/41250 [7:52:20<91:38:14,  8.68s/it]
                                                         
{'loss': 0.1855, 'grad_norm': 1.1839371919631958, 'learning_rate': 3.974798519484158e-05, 'epoch': 0.79}

  8%|▊         | 3262/41250 [7:52:20<91:38:14,  8.68s/it][2025-04-26 12:00:57,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:00:57,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.49 | bwd_microstep: 5721.00 | bwd_inner_microstep: 5708.10 | bwd_allreduce_microstep: 12.85 | step_microstep: 17.94
[2025-04-26 12:00:57,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.49 | bwd: 5721.01 | bwd_inner: 5708.10 | bwd_allreduce: 12.87 | step: 17.94

  8%|▊         | 3263/41250 [7:52:28<91:33:33,  8.68s/it]
                                                         
{'loss': 0.1515, 'grad_norm': 1.8308169841766357, 'learning_rate': 3.974773663179241e-05, 'epoch': 0.79}

  8%|▊         | 3263/41250 [7:52:28<91:33:33,  8.68s/it][2025-04-26 12:01:06,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.23 | optimizer_step: 0.90
[2025-04-26 12:01:06,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5771.44 | bwd_inner_microstep: 5705.28 | bwd_allreduce_microstep: 66.11 | step_microstep: 19.24
[2025-04-26 12:01:06,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5771.46 | bwd_inner: 5705.28 | bwd_allreduce: 66.14 | step: 19.25

  8%|▊         | 3264/41250 [7:52:37<91:40:23,  8.69s/it]
                                                         
{'loss': 0.2628, 'grad_norm': 1.5361965894699097, 'learning_rate': 3.974748794700234e-05, 'epoch': 0.79}

  8%|▊         | 3264/41250 [7:52:37<91:40:23,  8.69s/it][2025-04-26 12:01:15,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 12:01:15,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.37 | bwd_microstep: 5714.02 | bwd_inner_microstep: 5701.31 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.79
[2025-04-26 12:01:15,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.37 | bwd: 5714.03 | bwd_inner: 5701.31 | bwd_allreduce: 12.68 | step: 18.79

  8%|▊         | 3265/41250 [7:52:46<91:30:48,  8.67s/it]
                                                         
{'loss': 0.2712, 'grad_norm': 2.1816697120666504, 'learning_rate': 3.974723914047289e-05, 'epoch': 0.79}

  8%|▊         | 3265/41250 [7:52:46<91:30:48,  8.67s/it][2025-04-26 12:01:23,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:01:23,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.51 | bwd_microstep: 5716.97 | bwd_inner_microstep: 5654.17 | bwd_allreduce_microstep: 62.75 | step_microstep: 18.82
[2025-04-26 12:01:23,763] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.52 | bwd: 5716.98 | bwd_inner: 5654.17 | bwd_allreduce: 62.77 | step: 18.83

  8%|▊         | 3266/41250 [7:52:54<91:23:35,  8.66s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 2.8535828590393066, 'learning_rate': 3.974699021220559e-05, 'epoch': 0.79}

  8%|▊         | 3266/41250 [7:52:54<91:23:35,  8.66s/it][2025-04-26 12:01:32,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 12:01:32,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.34 | bwd_microstep: 5763.04 | bwd_inner_microstep: 5683.47 | bwd_allreduce_microstep: 79.53 | step_microstep: 18.80
[2025-04-26 12:01:32,459] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.34 | bwd: 5763.05 | bwd_inner: 5683.46 | bwd_allreduce: 79.55 | step: 18.80

  8%|▊         | 3267/41250 [7:53:03<91:29:29,  8.67s/it]
                                                         
{'loss': 0.0778, 'grad_norm': 0.9929301738739014, 'learning_rate': 3.974674116220198e-05, 'epoch': 0.79}

  8%|▊         | 3267/41250 [7:53:03<91:29:29,  8.67s/it][2025-04-26 12:01:41,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 12:01:41,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.61 | bwd_microstep: 5909.18 | bwd_inner_microstep: 5669.41 | bwd_allreduce_microstep: 239.72 | step_microstep: 19.14
[2025-04-26 12:01:41,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.61 | bwd: 5909.19 | bwd_inner: 5669.41 | bwd_allreduce: 239.74 | step: 19.15

  8%|▊         | 3268/41250 [7:53:12<92:00:38,  8.72s/it]
                                                         
{'loss': 0.1344, 'grad_norm': 1.2237170934677124, 'learning_rate': 3.9746491990463596e-05, 'epoch': 0.79}

  8%|▊         | 3268/41250 [7:53:12<92:00:38,  8.72s/it][2025-04-26 12:01:49,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-26 12:01:49,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.44 | bwd_microstep: 5754.49 | bwd_inner_microstep: 5707.02 | bwd_allreduce_microstep: 47.43 | step_microstep: 18.71
[2025-04-26 12:01:49,982] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.44 | bwd: 5754.51 | bwd_inner: 5707.02 | bwd_allreduce: 47.45 | step: 18.71

  8%|▊         | 3269/41250 [7:53:21<91:54:01,  8.71s/it]
                                                         
{'loss': 0.078, 'grad_norm': 0.7760350108146667, 'learning_rate': 3.974624269699197e-05, 'epoch': 0.79}

  8%|▊         | 3269/41250 [7:53:21<91:54:01,  8.71s/it][2025-04-26 12:01:58,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 12:01:58,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.05 | bwd_microstep: 5756.48 | bwd_inner_microstep: 5716.94 | bwd_allreduce_microstep: 39.49 | step_microstep: 18.07
[2025-04-26 12:01:58,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.05 | bwd: 5756.49 | bwd_inner: 5716.94 | bwd_allreduce: 39.51 | step: 18.08

  8%|▊         | 3270/41250 [7:53:29<91:50:44,  8.71s/it]
                                                         
{'loss': 0.0649, 'grad_norm': 0.6168912053108215, 'learning_rate': 3.974599328178865e-05, 'epoch': 0.79}

  8%|▊         | 3270/41250 [7:53:29<91:50:44,  8.71s/it][2025-04-26 12:02:07,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:02:07,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.58 | bwd_microstep: 5723.05 | bwd_inner_microstep: 5664.21 | bwd_allreduce_microstep: 58.80 | step_microstep: 18.54
[2025-04-26 12:02:07,314] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.58 | bwd: 5723.06 | bwd_inner: 5664.21 | bwd_allreduce: 58.81 | step: 18.55

  8%|▊         | 3271/41250 [7:53:38<91:37:51,  8.69s/it]
                                                         
{'loss': 0.1243, 'grad_norm': 1.115136742591858, 'learning_rate': 3.974574374485515e-05, 'epoch': 0.79}

  8%|▊         | 3271/41250 [7:53:38<91:37:51,  8.69s/it][2025-04-26 12:02:16,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 12:02:16,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2895.10 | bwd_microstep: 5782.02 | bwd_inner_microstep: 5768.26 | bwd_allreduce_microstep: 13.71 | step_microstep: 19.01
[2025-04-26 12:02:16,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2895.10 | bwd: 5782.03 | bwd_inner: 5768.26 | bwd_allreduce: 13.73 | step: 19.01

  8%|▊         | 3272/41250 [7:53:47<91:52:45,  8.71s/it]
                                                         
{'loss': 0.1061, 'grad_norm': 0.7323020100593567, 'learning_rate': 3.974549408619304e-05, 'epoch': 0.79}

  8%|▊         | 3272/41250 [7:53:47<91:52:45,  8.71s/it][2025-04-26 12:02:24,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 12:02:24,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.16 | bwd_microstep: 5785.98 | bwd_inner_microstep: 5661.28 | bwd_allreduce_microstep: 124.65 | step_microstep: 19.07
[2025-04-26 12:02:24,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.16 | bwd: 5786.00 | bwd_inner: 5661.28 | bwd_allreduce: 124.67 | step: 19.07

  8%|▊         | 3273/41250 [7:53:55<91:52:33,  8.71s/it]
                                                         
{'loss': 0.2655, 'grad_norm': 1.6713809967041016, 'learning_rate': 3.974524430580383e-05, 'epoch': 0.79}

  8%|▊         | 3273/41250 [7:53:55<91:52:33,  8.71s/it][2025-04-26 12:02:33,414] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:02:33,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.24 | bwd_microstep: 5710.26 | bwd_inner_microstep: 5659.27 | bwd_allreduce_microstep: 50.95 | step_microstep: 18.39
[2025-04-26 12:02:33,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.25 | bwd: 5710.27 | bwd_inner: 5659.27 | bwd_allreduce: 50.96 | step: 18.39

  8%|▊         | 3274/41250 [7:54:04<91:36:46,  8.68s/it]
                                                         
{'loss': 0.2717, 'grad_norm': 1.8622137308120728, 'learning_rate': 3.974499440368908e-05, 'epoch': 0.79}

  8%|▊         | 3274/41250 [7:54:04<91:36:46,  8.68s/it][2025-04-26 12:02:42,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:02:42,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.95 | bwd_microstep: 5758.78 | bwd_inner_microstep: 5705.37 | bwd_allreduce_microstep: 53.37 | step_microstep: 18.39
[2025-04-26 12:02:42,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.95 | bwd: 5758.80 | bwd_inner: 5705.37 | bwd_allreduce: 53.39 | step: 18.40

  8%|▊         | 3275/41250 [7:54:13<91:40:18,  8.69s/it]
                                                         
{'loss': 0.1792, 'grad_norm': 1.0682799816131592, 'learning_rate': 3.9744744379850325e-05, 'epoch': 0.79}

  8%|▊         | 3275/41250 [7:54:13<91:40:18,  8.69s/it][2025-04-26 12:02:50,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:02:50,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.52 | bwd_microstep: 5757.99 | bwd_inner_microstep: 5649.15 | bwd_allreduce_microstep: 108.80 | step_microstep: 18.44
[2025-04-26 12:02:50,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.52 | bwd: 5758.01 | bwd_inner: 5649.15 | bwd_allreduce: 108.82 | step: 18.44

  8%|▊         | 3276/41250 [7:54:21<91:39:09,  8.69s/it]
                                                         
{'loss': 0.341, 'grad_norm': 2.406899929046631, 'learning_rate': 3.97444942342891e-05, 'epoch': 0.79}

  8%|▊         | 3276/41250 [7:54:21<91:39:09,  8.69s/it][2025-04-26 12:02:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 12:02:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.68 | bwd_microstep: 5772.78 | bwd_inner_microstep: 5759.80 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.54
[2025-04-26 12:02:59,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.68 | bwd: 5772.79 | bwd_inner: 5759.80 | bwd_allreduce: 12.96 | step: 18.55

  8%|▊         | 3277/41250 [7:54:30<91:49:09,  8.70s/it]
                                                         
{'loss': 0.2153, 'grad_norm': 2.2908174991607666, 'learning_rate': 3.974424396700695e-05, 'epoch': 0.79}

  8%|▊         | 3277/41250 [7:54:30<91:49:09,  8.70s/it][2025-04-26 12:03:08,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.70 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 12:03:08,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.02 | bwd_microstep: 5754.90 | bwd_inner_microstep: 5709.69 | bwd_allreduce_microstep: 45.17 | step_microstep: 18.79
[2025-04-26 12:03:08,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.02 | bwd: 5754.91 | bwd_inner: 5709.69 | bwd_allreduce: 45.19 | step: 18.80

  8%|▊         | 3278/41250 [7:54:39<91:45:53,  8.70s/it]
                                                         
{'loss': 0.0907, 'grad_norm': 1.3226934671401978, 'learning_rate': 3.974399357800542e-05, 'epoch': 0.79}

  8%|▊         | 3278/41250 [7:54:39<91:45:53,  8.70s/it][2025-04-26 12:03:16,915] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.61 | optimizer_gradients: 1.05 | optimizer_step: 1.05
[2025-04-26 12:03:16,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.89 | bwd_microstep: 5721.89 | bwd_inner_microstep: 5708.57 | bwd_allreduce_microstep: 13.26 | step_microstep: 20.02
[2025-04-26 12:03:16,916] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.89 | bwd: 5721.91 | bwd_inner: 5708.57 | bwd_allreduce: 13.29 | step: 20.02

  8%|▊         | 3279/41250 [7:54:48<91:42:17,  8.69s/it]
                                                         
{'loss': 0.2592, 'grad_norm': 2.1979868412017822, 'learning_rate': 3.9743743067286056e-05, 'epoch': 0.79}

  8%|▊         | 3279/41250 [7:54:48<91:42:17,  8.69s/it][2025-04-26 12:03:25,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.71 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 12:03:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.00 | bwd_microstep: 5809.97 | bwd_inner_microstep: 5665.93 | bwd_allreduce_microstep: 143.99 | step_microstep: 18.87
[2025-04-26 12:03:25,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.00 | bwd: 5809.98 | bwd_inner: 5665.93 | bwd_allreduce: 144.01 | step: 18.87

  8%|▊         | 3280/41250 [7:54:56<91:48:21,  8.70s/it]
                                                         
{'loss': 0.1297, 'grad_norm': 1.1769888401031494, 'learning_rate': 3.97434924348504e-05, 'epoch': 0.8}

  8%|▊         | 3280/41250 [7:54:56<91:48:21,  8.70s/it][2025-04-26 12:03:34,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.99
[2025-04-26 12:03:34,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.93 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5704.82 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.59
[2025-04-26 12:03:34,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.93 | bwd: 5717.85 | bwd_inner: 5704.82 | bwd_allreduce: 12.99 | step: 18.59

  8%|▊         | 3281/41250 [7:55:05<91:39:04,  8.69s/it]
                                                         
{'loss': 0.3875, 'grad_norm': 3.1940531730651855, 'learning_rate': 3.974324168069999e-05, 'epoch': 0.8}

  8%|▊         | 3281/41250 [7:55:05<91:39:04,  8.69s/it][2025-04-26 12:03:42,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.95 | optimizer_gradients: 1.06 | optimizer_step: 1.04
[2025-04-26 12:03:42,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5746.33 | bwd_inner_microstep: 5694.06 | bwd_allreduce_microstep: 52.22 | step_microstep: 19.76
[2025-04-26 12:03:42,958] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5746.34 | bwd_inner: 5694.06 | bwd_allreduce: 52.24 | step: 19.76

  8%|▊         | 3282/41250 [7:55:14<91:33:02,  8.68s/it]
                                                         
{'loss': 0.2285, 'grad_norm': 1.2367849349975586, 'learning_rate': 3.974299080483638e-05, 'epoch': 0.8}

  8%|▊         | 3282/41250 [7:55:14<91:33:02,  8.68s/it][2025-04-26 12:03:51,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:03:51,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.17 | bwd_microstep: 5769.10 | bwd_inner_microstep: 5705.57 | bwd_allreduce_microstep: 63.49 | step_microstep: 17.97
[2025-04-26 12:03:51,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.17 | bwd: 5769.11 | bwd_inner: 5705.57 | bwd_allreduce: 63.50 | step: 17.97

  8%|▊         | 3283/41250 [7:55:22<91:36:56,  8.69s/it]
                                                         
{'loss': 0.1543, 'grad_norm': 6.5107035636901855, 'learning_rate': 3.974273980726113e-05, 'epoch': 0.8}

  8%|▊         | 3283/41250 [7:55:22<91:36:56,  8.69s/it][2025-04-26 12:04:00,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 12:04:00,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.76 | bwd_microstep: 5891.68 | bwd_inner_microstep: 5720.50 | bwd_allreduce_microstep: 171.13 | step_microstep: 18.57
[2025-04-26 12:04:00,485] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.76 | bwd: 5891.69 | bwd_inner: 5720.50 | bwd_allreduce: 171.15 | step: 18.58

  8%|▊         | 3284/41250 [7:55:31<92:03:01,  8.73s/it]
                                                         
{'loss': 0.2667, 'grad_norm': 2.2665960788726807, 'learning_rate': 3.9742488687975754e-05, 'epoch': 0.8}

  8%|▊         | 3284/41250 [7:55:31<92:03:01,  8.73s/it][2025-04-26 12:04:09,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:04:09,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5709.37 | bwd_inner_microstep: 5696.47 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.55
[2025-04-26 12:04:09,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5709.39 | bwd_inner: 5696.47 | bwd_allreduce: 12.88 | step: 18.55

  8%|▊         | 3285/41250 [7:55:40<91:47:37,  8.70s/it]
                                                         
{'loss': 0.161, 'grad_norm': 1.1404411792755127, 'learning_rate': 3.974223744698182e-05, 'epoch': 0.8}

  8%|▊         | 3285/41250 [7:55:40<91:47:37,  8.70s/it][2025-04-26 12:04:17,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:04:17,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.85 | bwd_microstep: 5737.22 | bwd_inner_microstep: 5685.79 | bwd_allreduce_microstep: 51.38 | step_microstep: 18.61
[2025-04-26 12:04:17,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.85 | bwd: 5737.24 | bwd_inner: 5685.79 | bwd_allreduce: 51.40 | step: 18.61

  8%|▊         | 3286/41250 [7:55:48<91:40:38,  8.69s/it]
                                                         
{'loss': 0.0909, 'grad_norm': 0.8027361631393433, 'learning_rate': 3.9741986084280884e-05, 'epoch': 0.8}

  8%|▊         | 3286/41250 [7:55:48<91:40:38,  8.69s/it][2025-04-26 12:04:26,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 12:04:26,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.07 | bwd_microstep: 5756.91 | bwd_inner_microstep: 5649.98 | bwd_allreduce_microstep: 106.89 | step_microstep: 18.96
[2025-04-26 12:04:26,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.07 | bwd: 5756.93 | bwd_inner: 5649.98 | bwd_allreduce: 106.90 | step: 18.96

  8%|▊         | 3287/41250 [7:55:57<91:35:38,  8.69s/it]
                                                         
{'loss': 0.2738, 'grad_norm': 2.3608713150024414, 'learning_rate': 3.974173459987448e-05, 'epoch': 0.8}

  8%|▊         | 3287/41250 [7:55:57<91:35:38,  8.69s/it][2025-04-26 12:04:35,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.00
[2025-04-26 12:04:35,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.79 | bwd_microstep: 5715.40 | bwd_inner_microstep: 5636.65 | bwd_allreduce_microstep: 78.71 | step_microstep: 18.53
[2025-04-26 12:04:35,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.79 | bwd: 5715.42 | bwd_inner: 5636.65 | bwd_allreduce: 78.73 | step: 18.53

  8%|▊         | 3288/41250 [7:56:06<91:22:37,  8.67s/it]
                                                         
{'loss': 0.2537, 'grad_norm': 1.828026294708252, 'learning_rate': 3.974148299376417e-05, 'epoch': 0.8}

  8%|▊         | 3288/41250 [7:56:06<91:22:37,  8.67s/it][2025-04-26 12:04:43,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 12:04:43,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.05 | bwd_microstep: 5720.70 | bwd_inner_microstep: 5707.69 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.49
[2025-04-26 12:04:43,743] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.05 | bwd: 5720.72 | bwd_inner: 5707.69 | bwd_allreduce: 12.98 | step: 18.49

  8%|▊         | 3289/41250 [7:56:14<91:20:22,  8.66s/it]
                                                         
{'loss': 0.2237, 'grad_norm': 3.516585350036621, 'learning_rate': 3.9741231265951496e-05, 'epoch': 0.8}

  8%|▊         | 3289/41250 [7:56:14<91:20:22,  8.66s/it][2025-04-26 12:04:52,364] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.07 | optimizer_step: 1.02
[2025-04-26 12:04:52,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.97 | bwd_microstep: 5693.23 | bwd_inner_microstep: 5680.04 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.36
[2025-04-26 12:04:52,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.97 | bwd: 5693.25 | bwd_inner: 5680.04 | bwd_allreduce: 13.16 | step: 19.36

  8%|▊         | 3290/41250 [7:56:23<91:13:00,  8.65s/it]
                                                         
{'loss': 0.1292, 'grad_norm': 1.0131986141204834, 'learning_rate': 3.9740979416438016e-05, 'epoch': 0.8}

  8%|▊         | 3290/41250 [7:56:23<91:13:00,  8.65s/it][2025-04-26 12:05:00,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:05:00,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.83 | bwd_microstep: 5703.79 | bwd_inner_microstep: 5691.00 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.91
[2025-04-26 12:05:00,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.83 | bwd: 5703.81 | bwd_inner: 5691.00 | bwd_allreduce: 12.76 | step: 18.92

  8%|▊         | 3291/41250 [7:56:32<91:08:23,  8.64s/it]
                                                         
{'loss': 0.36, 'grad_norm': 3.5476479530334473, 'learning_rate': 3.974072744522528e-05, 'epoch': 0.8}

  8%|▊         | 3291/41250 [7:56:32<91:08:23,  8.64s/it][2025-04-26 12:05:09,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.08 | optimizer_step: 1.02
[2025-04-26 12:05:09,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.10 | bwd_microstep: 5706.74 | bwd_inner_microstep: 5651.77 | bwd_allreduce_microstep: 54.92 | step_microstep: 19.22
[2025-04-26 12:05:09,604] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.10 | bwd: 5706.76 | bwd_inner: 5651.77 | bwd_allreduce: 54.94 | step: 19.23

  8%|▊         | 3292/41250 [7:56:40<91:02:34,  8.63s/it]
                                                         
{'loss': 0.2271, 'grad_norm': 1.5809251070022583, 'learning_rate': 3.9740475352314845e-05, 'epoch': 0.8}

  8%|▊         | 3292/41250 [7:56:40<91:02:34,  8.63s/it][2025-04-26 12:05:18,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:05:18,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5778.10 | bwd_inner_microstep: 5698.45 | bwd_allreduce_microstep: 79.60 | step_microstep: 18.45
[2025-04-26 12:05:18,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5778.11 | bwd_inner: 5698.45 | bwd_allreduce: 79.62 | step: 18.46

  8%|▊         | 3293/41250 [7:56:49<91:18:05,  8.66s/it]
                                                         
{'loss': 0.0769, 'grad_norm': 1.0547503232955933, 'learning_rate': 3.9740223137708264e-05, 'epoch': 0.8}

  8%|▊         | 3293/41250 [7:56:49<91:18:05,  8.66s/it][2025-04-26 12:05:27,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.03 | optimizer_step: 1.04
[2025-04-26 12:05:27,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.96 | bwd_microstep: 5765.35 | bwd_inner_microstep: 5652.87 | bwd_allreduce_microstep: 112.43 | step_microstep: 19.17
[2025-04-26 12:05:27,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.96 | bwd: 5765.36 | bwd_inner: 5652.87 | bwd_allreduce: 112.45 | step: 19.17

  8%|▊         | 3294/41250 [7:56:58<91:22:21,  8.67s/it]
                                                         
{'loss': 0.2006, 'grad_norm': 5.76509428024292, 'learning_rate': 3.9739970801407084e-05, 'epoch': 0.8}

  8%|▊         | 3294/41250 [7:56:58<91:22:21,  8.67s/it][2025-04-26 12:05:35,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.06 | optimizer_step: 1.15
[2025-04-26 12:05:35,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.55 | bwd_microstep: 5733.06 | bwd_inner_microstep: 5689.04 | bwd_allreduce_microstep: 43.97 | step_microstep: 19.51
[2025-04-26 12:05:35,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.56 | bwd: 5733.08 | bwd_inner: 5689.04 | bwd_allreduce: 43.99 | step: 19.51

  8%|▊         | 3295/41250 [7:57:06<91:22:49,  8.67s/it]
                                                         
{'loss': 0.214, 'grad_norm': 2.177530527114868, 'learning_rate': 3.973971834341287e-05, 'epoch': 0.8}

  8%|▊         | 3295/41250 [7:57:06<91:22:49,  8.67s/it][2025-04-26 12:05:44,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.01 | optimizer_step: 1.05
[2025-04-26 12:05:44,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.55 | bwd_microstep: 5708.15 | bwd_inner_microstep: 5695.34 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.75
[2025-04-26 12:05:44,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.55 | bwd: 5708.17 | bwd_inner: 5695.34 | bwd_allreduce: 12.79 | step: 18.75

  8%|▊         | 3296/41250 [7:57:15<91:15:37,  8.66s/it]
                                                         
{'loss': 0.2397, 'grad_norm': 1.350423812866211, 'learning_rate': 3.973946576372717e-05, 'epoch': 0.8}

  8%|▊         | 3296/41250 [7:57:15<91:15:37,  8.66s/it][2025-04-26 12:05:52,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 12:05:52,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.64 | bwd_microstep: 5737.78 | bwd_inner_microstep: 5699.64 | bwd_allreduce_microstep: 38.09 | step_microstep: 19.22
[2025-04-26 12:05:52,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.64 | bwd: 5737.79 | bwd_inner: 5699.64 | bwd_allreduce: 38.11 | step: 19.23

  8%|▊         | 3297/41250 [7:57:24<91:19:48,  8.66s/it]
                                                         
{'loss': 0.1132, 'grad_norm': 2.725808620452881, 'learning_rate': 3.973921306235155e-05, 'epoch': 0.8}

  8%|▊         | 3297/41250 [7:57:24<91:19:48,  8.66s/it][2025-04-26 12:06:01,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 0.94
[2025-04-26 12:06:01,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.91 | bwd_microstep: 5887.68 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 235.17 | step_microstep: 18.98
[2025-04-26 12:06:01,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.91 | bwd: 5887.70 | bwd_inner: 5652.46 | bwd_allreduce: 235.19 | step: 18.99

  8%|▊         | 3298/41250 [7:57:32<91:44:51,  8.70s/it]
                                                         
{'loss': 0.1741, 'grad_norm': 1.4460071325302124, 'learning_rate': 3.9738960239287564e-05, 'epoch': 0.8}

  8%|▊         | 3298/41250 [7:57:32<91:44:51,  8.70s/it][2025-04-26 12:06:10,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:06:10,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.24 | bwd_microstep: 5797.54 | bwd_inner_microstep: 5639.73 | bwd_allreduce_microstep: 157.77 | step_microstep: 18.42
[2025-04-26 12:06:10,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.24 | bwd: 5797.56 | bwd_inner: 5639.73 | bwd_allreduce: 157.79 | step: 18.42

  8%|▊         | 3299/41250 [7:57:41<91:44:07,  8.70s/it]
                                                         
{'loss': 0.1825, 'grad_norm': 2.124812602996826, 'learning_rate': 3.973870729453677e-05, 'epoch': 0.8}

  8%|▊         | 3299/41250 [7:57:41<91:44:07,  8.70s/it][2025-04-26 12:06:19,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:06:19,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.51 | bwd_microstep: 5780.44 | bwd_inner_microstep: 5678.94 | bwd_allreduce_microstep: 101.45 | step_microstep: 18.54
[2025-04-26 12:06:19,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.51 | bwd: 5780.45 | bwd_inner: 5678.94 | bwd_allreduce: 101.47 | step: 18.54

  8%|▊         | 3300/41250 [7:57:50<91:44:39,  8.70s/it]
                                                         
{'loss': 0.0863, 'grad_norm': 0.6564542651176453, 'learning_rate': 3.973845422810073e-05, 'epoch': 0.8}

  8%|▊         | 3300/41250 [7:57:50<91:44:39,  8.70s/it][2025-04-26 12:06:27,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 12:06:27,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.64 | bwd_microstep: 5701.48 | bwd_inner_microstep: 5688.41 | bwd_allreduce_microstep: 13.01 | step_microstep: 19.02
[2025-04-26 12:06:27,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.64 | bwd: 5701.49 | bwd_inner: 5688.41 | bwd_allreduce: 13.04 | step: 19.02

  8%|▊         | 3301/41250 [7:57:58<91:28:51,  8.68s/it]
                                                         
{'loss': 0.1267, 'grad_norm': 0.7850197553634644, 'learning_rate': 3.9738201039981e-05, 'epoch': 0.8}

  8%|▊         | 3301/41250 [7:57:58<91:28:51,  8.68s/it][2025-04-26 12:06:36,505] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.26 | optimizer_step: 0.98
[2025-04-26 12:06:36,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.50 | bwd_microstep: 5788.33 | bwd_inner_microstep: 5653.13 | bwd_allreduce_microstep: 135.14 | step_microstep: 19.35
[2025-04-26 12:06:36,506] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.50 | bwd: 5788.35 | bwd_inner: 5653.12 | bwd_allreduce: 135.17 | step: 19.36

  8%|▊         | 3302/41250 [7:58:07<91:33:12,  8.69s/it]
                                                         
{'loss': 0.1063, 'grad_norm': 1.138809323310852, 'learning_rate': 3.973794773017914e-05, 'epoch': 0.8}

  8%|▊         | 3302/41250 [7:58:07<91:33:12,  8.69s/it][2025-04-26 12:06:45,135] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 12:06:45,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.36 | bwd_microstep: 5721.33 | bwd_inner_microstep: 5650.75 | bwd_allreduce_microstep: 70.54 | step_microstep: 19.15
[2025-04-26 12:06:45,136] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.36 | bwd: 5721.35 | bwd_inner: 5650.75 | bwd_allreduce: 70.56 | step: 19.15

  8%|▊         | 3303/41250 [7:58:16<91:21:47,  8.67s/it]
                                                         
{'loss': 0.2972, 'grad_norm': 1.9597878456115723, 'learning_rate': 3.973769429869672e-05, 'epoch': 0.8}

  8%|▊         | 3303/41250 [7:58:16<91:21:47,  8.67s/it][2025-04-26 12:06:53,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-26 12:06:53,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.44 | bwd_microstep: 5704.16 | bwd_inner_microstep: 5644.43 | bwd_allreduce_microstep: 59.68 | step_microstep: 19.05
[2025-04-26 12:06:53,752] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.44 | bwd: 5704.17 | bwd_inner: 5644.43 | bwd_allreduce: 59.70 | step: 19.06

  8%|▊         | 3304/41250 [7:58:24<91:12:06,  8.65s/it]
                                                         
{'loss': 0.1637, 'grad_norm': 0.9647200107574463, 'learning_rate': 3.973744074553529e-05, 'epoch': 0.8}

  8%|▊         | 3304/41250 [7:58:24<91:12:06,  8.65s/it][2025-04-26 12:07:02,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 12:07:02,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5769.50 | bwd_inner_microstep: 5696.94 | bwd_allreduce_microstep: 72.51 | step_microstep: 18.85
[2025-04-26 12:07:02,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5769.52 | bwd_inner: 5696.94 | bwd_allreduce: 72.53 | step: 18.86

  8%|▊         | 3305/41250 [7:58:33<91:20:15,  8.67s/it]
                                                         
{'loss': 0.0275, 'grad_norm': 0.283722847700119, 'learning_rate': 3.9737187070696433e-05, 'epoch': 0.8}

  8%|▊         | 3305/41250 [7:58:33<91:20:15,  8.67s/it][2025-04-26 12:07:11,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:07:11,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.64 | bwd_microstep: 5702.59 | bwd_inner_microstep: 5689.67 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.80
[2025-04-26 12:07:11,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.64 | bwd: 5702.60 | bwd_inner: 5689.67 | bwd_allreduce: 12.88 | step: 18.80

  8%|▊         | 3306/41250 [7:58:42<91:13:23,  8.65s/it]
                                                         
{'loss': 0.202, 'grad_norm': 1.958146572113037, 'learning_rate': 3.9736933274181684e-05, 'epoch': 0.8}

  8%|▊         | 3306/41250 [7:58:42<91:13:23,  8.65s/it][2025-04-26 12:07:19,709] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 12:07:19,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.15 | bwd_microstep: 5702.63 | bwd_inner_microstep: 5689.76 | bwd_allreduce_microstep: 12.82 | step_microstep: 19.19
[2025-04-26 12:07:19,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.15 | bwd: 5702.65 | bwd_inner: 5689.76 | bwd_allreduce: 12.84 | step: 19.20

  8%|▊         | 3307/41250 [7:58:50<91:08:34,  8.65s/it]
                                                         
{'loss': 0.2315, 'grad_norm': 1.951789140701294, 'learning_rate': 3.973667935599263e-05, 'epoch': 0.8}

  8%|▊         | 3307/41250 [7:58:50<91:08:34,  8.65s/it][2025-04-26 12:07:28,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 12:07:28,397] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.85 | bwd_microstep: 5745.39 | bwd_inner_microstep: 5716.24 | bwd_allreduce_microstep: 29.09 | step_microstep: 18.71
[2025-04-26 12:07:28,398] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.85 | bwd: 5745.40 | bwd_inner: 5716.25 | bwd_allreduce: 29.11 | step: 18.71

  8%|▊         | 3308/41250 [7:58:59<91:16:06,  8.66s/it]
                                                         
{'loss': 0.0229, 'grad_norm': 0.2291499674320221, 'learning_rate': 3.973642531613083e-05, 'epoch': 0.8}

  8%|▊         | 3308/41250 [7:58:59<91:16:06,  8.66s/it][2025-04-26 12:07:37,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:07:37,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.40 | bwd_microstep: 5791.69 | bwd_inner_microstep: 5664.75 | bwd_allreduce_microstep: 126.90 | step_microstep: 17.76
[2025-04-26 12:07:37,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.40 | bwd: 5791.70 | bwd_inner: 5664.75 | bwd_allreduce: 126.91 | step: 17.77

  8%|▊         | 3309/41250 [7:59:08<91:24:59,  8.67s/it]
                                                         
{'loss': 0.1222, 'grad_norm': 1.259354591369629, 'learning_rate': 3.973617115459785e-05, 'epoch': 0.8}

  8%|▊         | 3309/41250 [7:59:08<91:24:59,  8.67s/it][2025-04-26 12:07:45,809] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:07:45,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.55 | bwd_microstep: 5795.22 | bwd_inner_microstep: 5644.83 | bwd_allreduce_microstep: 150.35 | step_microstep: 18.67
[2025-04-26 12:07:45,810] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.55 | bwd: 5795.24 | bwd_inner: 5644.83 | bwd_allreduce: 150.37 | step: 18.68

  8%|▊         | 3310/41250 [7:59:16<91:30:39,  8.68s/it]
                                                         
{'loss': 0.2298, 'grad_norm': 1.116227626800537, 'learning_rate': 3.973591687139526e-05, 'epoch': 0.8}

  8%|▊         | 3310/41250 [7:59:16<91:30:39,  8.68s/it][2025-04-26 12:07:54,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:07:54,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.86 | bwd_microstep: 5730.67 | bwd_inner_microstep: 5717.77 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.62
[2025-04-26 12:07:54,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.86 | bwd: 5730.69 | bwd_inner: 5717.77 | bwd_allreduce: 12.88 | step: 18.62

  8%|▊         | 3311/41250 [7:59:25<91:28:21,  8.68s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 2.3757498264312744, 'learning_rate': 3.973566246652462e-05, 'epoch': 0.8}

  8%|▊         | 3311/41250 [7:59:25<91:28:21,  8.68s/it][2025-04-26 12:08:03,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:08:03,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.42 | bwd_microstep: 5792.39 | bwd_inner_microstep: 5647.65 | bwd_allreduce_microstep: 144.69 | step_microstep: 18.41
[2025-04-26 12:08:03,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.42 | bwd: 5792.40 | bwd_inner: 5647.65 | bwd_allreduce: 144.71 | step: 18.41

  8%|▊         | 3312/41250 [7:59:34<91:32:11,  8.69s/it]
                                                         
{'loss': 0.0822, 'grad_norm': 0.8100155591964722, 'learning_rate': 3.9735407939987507e-05, 'epoch': 0.8}

  8%|▊         | 3312/41250 [7:59:34<91:32:11,  8.69s/it][2025-04-26 12:08:11,874] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 1.19
[2025-04-26 12:08:11,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.70 | bwd_microstep: 5758.02 | bwd_inner_microstep: 5695.24 | bwd_allreduce_microstep: 62.72 | step_microstep: 19.40
[2025-04-26 12:08:11,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.70 | bwd: 5758.03 | bwd_inner: 5695.24 | bwd_allreduce: 62.74 | step: 19.40

  8%|▊         | 3313/41250 [7:59:43<91:33:22,  8.69s/it]
                                                         
{'loss': 0.3333, 'grad_norm': 2.50239634513855, 'learning_rate': 3.973515329178548e-05, 'epoch': 0.8}

  8%|▊         | 3313/41250 [7:59:43<91:33:22,  8.69s/it][2025-04-26 12:08:20,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:08:20,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2867.64 | bwd_microstep: 5726.34 | bwd_inner_microstep: 5713.63 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.64
[2025-04-26 12:08:20,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2867.64 | bwd: 5726.35 | bwd_inner: 5713.63 | bwd_allreduce: 12.68 | step: 18.65

  8%|▊         | 3314/41250 [7:59:51<91:28:56,  8.68s/it]
                                                         
{'loss': 0.0903, 'grad_norm': 1.2454785108566284, 'learning_rate': 3.973489852192012e-05, 'epoch': 0.8}

  8%|▊         | 3314/41250 [7:59:51<91:28:56,  8.68s/it][2025-04-26 12:08:29,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 12:08:29,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.10 | bwd_microstep: 5787.71 | bwd_inner_microstep: 5651.66 | bwd_allreduce_microstep: 136.00 | step_microstep: 18.95
[2025-04-26 12:08:29,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.10 | bwd: 5787.73 | bwd_inner: 5651.66 | bwd_allreduce: 136.02 | step: 18.95

  8%|▊         | 3315/41250 [8:00:00<91:31:53,  8.69s/it]
                                                         
{'loss': 0.1316, 'grad_norm': 0.8326923251152039, 'learning_rate': 3.973464363039299e-05, 'epoch': 0.8}

  8%|▊         | 3315/41250 [8:00:00<91:31:53,  8.69s/it][2025-04-26 12:08:37,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 12:08:37,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.30 | bwd_microstep: 5722.14 | bwd_inner_microstep: 5709.29 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.74
[2025-04-26 12:08:37,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.30 | bwd: 5722.15 | bwd_inner: 5709.29 | bwd_allreduce: 12.82 | step: 18.74

  8%|▊         | 3316/41250 [8:00:09<91:28:30,  8.68s/it]
                                                         
{'loss': 0.2472, 'grad_norm': 4.341721534729004, 'learning_rate': 3.973438861720566e-05, 'epoch': 0.8}

  8%|▊         | 3316/41250 [8:00:09<91:28:30,  8.68s/it][2025-04-26 12:08:46,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:08:46,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.54 | bwd_microstep: 5749.55 | bwd_inner_microstep: 5695.22 | bwd_allreduce_microstep: 54.29 | step_microstep: 18.53
[2025-04-26 12:08:46,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.54 | bwd: 5749.57 | bwd_inner: 5695.22 | bwd_allreduce: 54.30 | step: 18.53

  8%|▊         | 3317/41250 [8:00:17<91:29:03,  8.68s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 1.448285698890686, 'learning_rate': 3.973413348235971e-05, 'epoch': 0.8}

  8%|▊         | 3317/41250 [8:00:17<91:29:03,  8.68s/it][2025-04-26 12:08:55,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 12:08:55,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.61 | bwd_microstep: 5775.36 | bwd_inner_microstep: 5659.21 | bwd_allreduce_microstep: 116.09 | step_microstep: 19.26
[2025-04-26 12:08:55,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.61 | bwd: 5775.37 | bwd_inner: 5659.22 | bwd_allreduce: 116.11 | step: 19.26

  8%|▊         | 3318/41250 [8:00:26<91:33:11,  8.69s/it]
                                                         
{'loss': 0.1742, 'grad_norm': 1.6802829504013062, 'learning_rate': 3.973387822585671e-05, 'epoch': 0.8}

  8%|▊         | 3318/41250 [8:00:26<91:33:11,  8.69s/it][2025-04-26 12:09:03,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:09:03,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.26 | bwd_microstep: 5733.53 | bwd_inner_microstep: 5720.65 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.67
[2025-04-26 12:09:03,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.26 | bwd: 5733.54 | bwd_inner: 5720.65 | bwd_allreduce: 12.86 | step: 18.67

  8%|▊         | 3319/41250 [8:00:35<91:30:08,  8.68s/it]
                                                         
{'loss': 0.3104, 'grad_norm': 2.5655531883239746, 'learning_rate': 3.973362284769823e-05, 'epoch': 0.8}

  8%|▊         | 3319/41250 [8:00:35<91:30:08,  8.68s/it][2025-04-26 12:09:12,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:09:12,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.06 | bwd_microstep: 5728.35 | bwd_inner_microstep: 5657.95 | bwd_allreduce_microstep: 70.35 | step_microstep: 18.46
[2025-04-26 12:09:12,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.06 | bwd: 5728.36 | bwd_inner: 5657.95 | bwd_allreduce: 70.37 | step: 18.46

  8%|▊         | 3320/41250 [8:00:43<91:21:45,  8.67s/it]
                                                         
{'loss': 0.0356, 'grad_norm': 0.5501074194908142, 'learning_rate': 3.973336734788585e-05, 'epoch': 0.8}

  8%|▊         | 3320/41250 [8:00:43<91:21:45,  8.67s/it][2025-04-26 12:09:21,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 12:09:21,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.28 | bwd_microstep: 5916.36 | bwd_inner_microstep: 5661.85 | bwd_allreduce_microstep: 254.46 | step_microstep: 18.78
[2025-04-26 12:09:21,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.28 | bwd: 5916.37 | bwd_inner: 5661.85 | bwd_allreduce: 254.48 | step: 18.78

  8%|▊         | 3321/41250 [8:00:52<91:52:23,  8.72s/it]
                                                         
{'loss': 0.2934, 'grad_norm': 2.003525495529175, 'learning_rate': 3.973311172642113e-05, 'epoch': 0.81}

  8%|▊         | 3321/41250 [8:00:52<91:52:23,  8.72s/it][2025-04-26 12:09:30,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.13 | optimizer_step: 1.05
[2025-04-26 12:09:30,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.85 | bwd_microstep: 5744.80 | bwd_inner_microstep: 5687.85 | bwd_allreduce_microstep: 56.89 | step_microstep: 20.02
[2025-04-26 12:09:30,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.85 | bwd: 5744.82 | bwd_inner: 5687.85 | bwd_allreduce: 56.92 | step: 20.02

  8%|▊         | 3322/41250 [8:01:01<91:46:20,  8.71s/it]
                                                         
{'loss': 0.1094, 'grad_norm': 1.0707422494888306, 'learning_rate': 3.973285598330567e-05, 'epoch': 0.81}

  8%|▊         | 3322/41250 [8:01:01<91:46:20,  8.71s/it][2025-04-26 12:09:38,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 12:09:38,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.51 | bwd_microstep: 5713.52 | bwd_inner_microstep: 5700.52 | bwd_allreduce_microstep: 12.95 | step_microstep: 18.97
[2025-04-26 12:09:38,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.51 | bwd: 5713.54 | bwd_inner: 5700.52 | bwd_allreduce: 12.97 | step: 18.98

  8%|▊         | 3323/41250 [8:01:09<91:33:17,  8.69s/it]
                                                         
{'loss': 0.3513, 'grad_norm': 2.737931489944458, 'learning_rate': 3.973260011854103e-05, 'epoch': 0.81}

  8%|▊         | 3323/41250 [8:01:09<91:33:17,  8.69s/it][2025-04-26 12:09:47,470] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:09:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.89 | bwd_microstep: 5784.24 | bwd_inner_microstep: 5658.13 | bwd_allreduce_microstep: 126.07 | step_microstep: 18.35
[2025-04-26 12:09:47,471] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.89 | bwd: 5784.25 | bwd_inner: 5658.13 | bwd_allreduce: 126.08 | step: 18.35

  8%|▊         | 3324/41250 [8:01:18<91:33:39,  8.69s/it]
                                                         
{'loss': 0.0541, 'grad_norm': 0.9404283761978149, 'learning_rate': 3.973234413212879e-05, 'epoch': 0.81}

  8%|▊         | 3324/41250 [8:01:18<91:33:39,  8.69s/it][2025-04-26 12:09:56,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:09:56,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.91 | bwd_microstep: 5742.54 | bwd_inner_microstep: 5657.12 | bwd_allreduce_microstep: 85.37 | step_microstep: 18.40
[2025-04-26 12:09:56,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.91 | bwd: 5742.56 | bwd_inner: 5657.12 | bwd_allreduce: 85.39 | step: 18.40

  8%|▊         | 3325/41250 [8:01:27<91:28:48,  8.68s/it]
                                                         
{'loss': 0.057, 'grad_norm': 0.7821275591850281, 'learning_rate': 3.973208802407053e-05, 'epoch': 0.81}

  8%|▊         | 3325/41250 [8:01:27<91:28:48,  8.68s/it][2025-04-26 12:10:04,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 12:10:04,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.12 | bwd_microstep: 5749.39 | bwd_inner_microstep: 5662.52 | bwd_allreduce_microstep: 86.82 | step_microstep: 18.97
[2025-04-26 12:10:04,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.12 | bwd: 5749.40 | bwd_inner: 5662.52 | bwd_allreduce: 86.84 | step: 18.97

  8%|▊         | 3326/41250 [8:01:35<91:27:10,  8.68s/it]
                                                         
{'loss': 0.1941, 'grad_norm': 1.7803696393966675, 'learning_rate': 3.973183179436783e-05, 'epoch': 0.81}

  8%|▊         | 3326/41250 [8:01:35<91:27:10,  8.68s/it][2025-04-26 12:10:13,496] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 12:10:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.98 | bwd_microstep: 5768.27 | bwd_inner_microstep: 5645.52 | bwd_allreduce_microstep: 122.70 | step_microstep: 19.24
[2025-04-26 12:10:13,497] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.98 | bwd: 5768.28 | bwd_inner: 5645.52 | bwd_allreduce: 122.72 | step: 19.24

  8%|▊         | 3327/41250 [8:01:44<91:27:40,  8.68s/it]
                                                         
{'loss': 0.1475, 'grad_norm': 1.330154538154602, 'learning_rate': 3.973157544302226e-05, 'epoch': 0.81}

  8%|▊         | 3327/41250 [8:01:44<91:27:40,  8.68s/it][2025-04-26 12:10:22,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:10:22,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.64 | bwd_microstep: 5770.68 | bwd_inner_microstep: 5675.41 | bwd_allreduce_microstep: 95.22 | step_microstep: 18.70
[2025-04-26 12:10:22,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.64 | bwd: 5770.69 | bwd_inner: 5675.41 | bwd_allreduce: 95.24 | step: 18.70

  8%|▊         | 3328/41250 [8:01:53<91:28:41,  8.68s/it]
                                                         
{'loss': 0.0987, 'grad_norm': 0.9500070810317993, 'learning_rate': 3.9731318970035426e-05, 'epoch': 0.81}

  8%|▊         | 3328/41250 [8:01:53<91:28:41,  8.68s/it][2025-04-26 12:10:30,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:10:30,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.51 | bwd_microstep: 5790.95 | bwd_inner_microstep: 5629.47 | bwd_allreduce_microstep: 161.43 | step_microstep: 18.63
[2025-04-26 12:10:30,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.51 | bwd: 5790.96 | bwd_inner: 5629.47 | bwd_allreduce: 161.44 | step: 18.64

  8%|▊         | 3329/41250 [8:02:02<91:30:28,  8.69s/it]
                                                         
{'loss': 0.0631, 'grad_norm': 0.7266091108322144, 'learning_rate': 3.973106237540887e-05, 'epoch': 0.81}

  8%|▊         | 3329/41250 [8:02:02<91:30:28,  8.69s/it][2025-04-26 12:10:39,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 12:10:39,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.48 | bwd_microstep: 5692.13 | bwd_inner_microstep: 5646.07 | bwd_allreduce_microstep: 46.02 | step_microstep: 19.19
[2025-04-26 12:10:39,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.48 | bwd: 5692.15 | bwd_inner: 5646.07 | bwd_allreduce: 46.04 | step: 19.19

  8%|▊         | 3330/41250 [8:02:10<91:13:13,  8.66s/it]
                                                         
{'loss': 0.2103, 'grad_norm': 2.1893954277038574, 'learning_rate': 3.973080565914421e-05, 'epoch': 0.81}

  8%|▊         | 3330/41250 [8:02:10<91:13:13,  8.66s/it][2025-04-26 12:10:48,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:10:48,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.39 | bwd_microstep: 5772.59 | bwd_inner_microstep: 5759.92 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.41
[2025-04-26 12:10:48,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.39 | bwd: 5772.60 | bwd_inner: 5759.92 | bwd_allreduce: 12.64 | step: 18.41

  8%|▊         | 3331/41250 [8:02:19<91:27:37,  8.68s/it]
                                                         
{'loss': 0.3385, 'grad_norm': 1.8988338708877563, 'learning_rate': 3.9730548821243016e-05, 'epoch': 0.81}

  8%|▊         | 3331/41250 [8:02:19<91:27:37,  8.68s/it][2025-04-26 12:10:56,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:10:56,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.92 | bwd_microstep: 5773.46 | bwd_inner_microstep: 5652.60 | bwd_allreduce_microstep: 120.82 | step_microstep: 18.42
[2025-04-26 12:10:56,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.92 | bwd: 5773.48 | bwd_inner: 5652.60 | bwd_allreduce: 120.84 | step: 18.43

  8%|▊         | 3332/41250 [8:02:28<91:25:51,  8.68s/it]
                                                         
{'loss': 0.3331, 'grad_norm': 2.934136152267456, 'learning_rate': 3.9730291861706855e-05, 'epoch': 0.81}

  8%|▊         | 3332/41250 [8:02:28<91:25:51,  8.68s/it][2025-04-26 12:11:05,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 1.06
[2025-04-26 12:11:05,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.79 | bwd_microstep: 5761.78 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 109.28 | step_microstep: 18.38
[2025-04-26 12:11:05,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.79 | bwd: 5761.79 | bwd_inner: 5652.46 | bwd_allreduce: 109.29 | step: 18.38

  8%|▊         | 3333/41250 [8:02:36<91:22:36,  8.68s/it]
                                                         
{'loss': 0.113, 'grad_norm': 1.0037813186645508, 'learning_rate': 3.973003478053734e-05, 'epoch': 0.81}

  8%|▊         | 3333/41250 [8:02:36<91:22:36,  8.68s/it][2025-04-26 12:11:14,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:11:14,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.82 | bwd_microstep: 5709.26 | bwd_inner_microstep: 5696.54 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.15
[2025-04-26 12:11:14,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.82 | bwd: 5709.27 | bwd_inner: 5696.54 | bwd_allreduce: 12.69 | step: 18.15

  8%|▊         | 3334/41250 [8:02:45<91:17:02,  8.67s/it]
                                                         
{'loss': 0.1602, 'grad_norm': 0.7722647190093994, 'learning_rate': 3.972977757773604e-05, 'epoch': 0.81}

  8%|▊         | 3334/41250 [8:02:45<91:17:02,  8.67s/it][2025-04-26 12:11:22,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 12:11:22,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.52 | bwd_microstep: 5750.60 | bwd_inner_microstep: 5684.99 | bwd_allreduce_microstep: 65.56 | step_microstep: 18.60
[2025-04-26 12:11:22,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.52 | bwd: 5750.61 | bwd_inner: 5684.99 | bwd_allreduce: 65.58 | step: 18.60

  8%|▊         | 3335/41250 [8:02:54<91:18:47,  8.67s/it]
                                                         
{'loss': 0.2646, 'grad_norm': 2.1816892623901367, 'learning_rate': 3.972952025330453e-05, 'epoch': 0.81}

  8%|▊         | 3335/41250 [8:02:54<91:18:47,  8.67s/it][2025-04-26 12:11:31,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 12:11:31,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.08 | bwd_microstep: 5695.26 | bwd_inner_microstep: 5682.42 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.97
[2025-04-26 12:11:31,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.08 | bwd: 5695.27 | bwd_inner: 5682.42 | bwd_allreduce: 12.80 | step: 18.97

  8%|▊         | 3336/41250 [8:03:02<91:10:28,  8.66s/it]
                                                         
{'loss': 0.2401, 'grad_norm': 1.5971126556396484, 'learning_rate': 3.972926280724442e-05, 'epoch': 0.81}

  8%|▊         | 3336/41250 [8:03:02<91:10:28,  8.66s/it][2025-04-26 12:11:40,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-26 12:11:40,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.65 | bwd_microstep: 5897.92 | bwd_inner_microstep: 5685.11 | bwd_allreduce_microstep: 212.76 | step_microstep: 18.75
[2025-04-26 12:11:40,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.65 | bwd: 5897.93 | bwd_inner: 5685.11 | bwd_allreduce: 212.78 | step: 18.75

  8%|▊         | 3337/41250 [8:03:11<91:40:14,  8.70s/it]
                                                         
{'loss': 0.3157, 'grad_norm': 1.8762946128845215, 'learning_rate': 3.9729005239557274e-05, 'epoch': 0.81}

  8%|▊         | 3337/41250 [8:03:11<91:40:14,  8.70s/it][2025-04-26 12:11:48,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.02 | optimizer_step: 1.20
[2025-04-26 12:11:48,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.20 | bwd_microstep: 5683.45 | bwd_inner_microstep: 5653.81 | bwd_allreduce_microstep: 29.58 | step_microstep: 19.44
[2025-04-26 12:11:48,914] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.20 | bwd: 5683.46 | bwd_inner: 5653.81 | bwd_allreduce: 29.60 | step: 19.45

  8%|▊         | 3338/41250 [8:03:20<91:19:53,  8.67s/it]
                                                         
{'loss': 0.2198, 'grad_norm': 2.4419963359832764, 'learning_rate': 3.97287475502447e-05, 'epoch': 0.81}

  8%|▊         | 3338/41250 [8:03:20<91:19:53,  8.67s/it][2025-04-26 12:11:57,569] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.60 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 12:11:57,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.34 | bwd_microstep: 5716.19 | bwd_inner_microstep: 5703.20 | bwd_allreduce_microstep: 12.94 | step_microstep: 19.22
[2025-04-26 12:11:57,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.35 | bwd: 5716.20 | bwd_inner: 5703.20 | bwd_allreduce: 12.96 | step: 19.22

  8%|▊         | 3339/41250 [8:03:28<91:16:07,  8.67s/it]
                                                         
{'loss': 0.088, 'grad_norm': 2.1296370029449463, 'learning_rate': 3.972848973930828e-05, 'epoch': 0.81}

  8%|▊         | 3339/41250 [8:03:28<91:16:07,  8.67s/it][2025-04-26 12:12:06,405] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 12:12:06,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.78 | bwd_microstep: 5903.86 | bwd_inner_microstep: 5690.93 | bwd_allreduce_microstep: 212.88 | step_microstep: 18.55
[2025-04-26 12:12:06,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.78 | bwd: 5903.87 | bwd_inner: 5690.93 | bwd_allreduce: 212.90 | step: 18.56

  8%|▊         | 3340/41250 [8:03:37<91:47:52,  8.72s/it]
                                                         
{'loss': 0.0908, 'grad_norm': 1.3507866859436035, 'learning_rate': 3.9728231806749593e-05, 'epoch': 0.81}

  8%|▊         | 3340/41250 [8:03:37<91:47:52,  8.72s/it][2025-04-26 12:12:15,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:12:15,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.43 | bwd_microstep: 5765.07 | bwd_inner_microstep: 5707.20 | bwd_allreduce_microstep: 57.83 | step_microstep: 18.67
[2025-04-26 12:12:15,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.43 | bwd: 5765.09 | bwd_inner: 5707.20 | bwd_allreduce: 57.85 | step: 18.68

  8%|▊         | 3341/41250 [8:03:46<91:41:06,  8.71s/it]
                                                         
{'loss': 0.0955, 'grad_norm': 1.0981297492980957, 'learning_rate': 3.9727973752570236e-05, 'epoch': 0.81}

  8%|▊         | 3341/41250 [8:03:46<91:41:06,  8.71s/it][2025-04-26 12:12:23,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:12:23,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.52 | bwd_microstep: 5883.48 | bwd_inner_microstep: 5870.71 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.81
[2025-04-26 12:12:23,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.52 | bwd: 5883.50 | bwd_inner: 5870.71 | bwd_allreduce: 12.74 | step: 18.81

  8%|▊         | 3342/41250 [8:03:55<92:17:55,  8.77s/it]
                                                         
{'loss': 0.3041, 'grad_norm': 1.4875324964523315, 'learning_rate': 3.972771557677181e-05, 'epoch': 0.81}

  8%|▊         | 3342/41250 [8:03:55<92:17:55,  8.77s/it][2025-04-26 12:12:32,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 12:12:32,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5704.01 | bwd_inner_microstep: 5691.03 | bwd_allreduce_microstep: 12.93 | step_microstep: 19.17
[2025-04-26 12:12:32,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5704.02 | bwd_inner: 5691.03 | bwd_allreduce: 12.95 | step: 19.18

  8%|▊         | 3343/41250 [8:04:03<91:53:48,  8.73s/it]
                                                         
{'loss': 0.0684, 'grad_norm': 0.9204399585723877, 'learning_rate': 3.9727457279355895e-05, 'epoch': 0.81}

  8%|▊         | 3343/41250 [8:04:03<91:53:48,  8.73s/it][2025-04-26 12:12:41,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 12:12:41,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.54 | bwd_microstep: 5695.88 | bwd_inner_microstep: 5658.42 | bwd_allreduce_microstep: 37.41 | step_microstep: 18.72
[2025-04-26 12:12:41,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.54 | bwd: 5695.89 | bwd_inner: 5658.42 | bwd_allreduce: 37.43 | step: 18.72

  8%|▊         | 3344/41250 [8:04:12<91:31:17,  8.69s/it]
                                                         
{'loss': 0.1525, 'grad_norm': 1.5920026302337646, 'learning_rate': 3.972719886032408e-05, 'epoch': 0.81}

  8%|▊         | 3344/41250 [8:04:12<91:31:17,  8.69s/it][2025-04-26 12:12:49,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 12:12:49,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.87 | bwd_microstep: 5756.67 | bwd_inner_microstep: 5708.89 | bwd_allreduce_microstep: 47.73 | step_microstep: 18.66
[2025-04-26 12:12:49,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.88 | bwd: 5756.69 | bwd_inner: 5708.89 | bwd_allreduce: 47.75 | step: 18.66

  8%|▊         | 3345/41250 [8:04:21<91:31:23,  8.69s/it]
                                                         
{'loss': 0.1897, 'grad_norm': 1.8661812543869019, 'learning_rate': 3.9726940319677976e-05, 'epoch': 0.81}

  8%|▊         | 3345/41250 [8:04:21<91:31:23,  8.69s/it][2025-04-26 12:12:58,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.02
[2025-04-26 12:12:58,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.33 | bwd_microstep: 5720.71 | bwd_inner_microstep: 5708.24 | bwd_allreduce_microstep: 12.42 | step_microstep: 18.71
[2025-04-26 12:12:58,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.33 | bwd: 5720.72 | bwd_inner: 5708.24 | bwd_allreduce: 12.44 | step: 18.72

  8%|▊         | 3346/41250 [8:04:29<91:22:41,  8.68s/it]
                                                         
{'loss': 0.2649, 'grad_norm': 2.249544143676758, 'learning_rate': 3.9726681657419155e-05, 'epoch': 0.81}

  8%|▊         | 3346/41250 [8:04:29<91:22:41,  8.68s/it][2025-04-26 12:13:07,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.20
[2025-04-26 12:13:07,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.01 | bwd_microstep: 5698.98 | bwd_inner_microstep: 5657.70 | bwd_allreduce_microstep: 41.23 | step_microstep: 19.39
[2025-04-26 12:13:07,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.01 | bwd: 5698.99 | bwd_inner: 5657.70 | bwd_allreduce: 41.25 | step: 19.39

  8%|▊         | 3347/41250 [8:04:38<91:10:59,  8.66s/it]
                                                         
{'loss': 0.09, 'grad_norm': 0.6937418580055237, 'learning_rate': 3.9726422873549225e-05, 'epoch': 0.81}

  8%|▊         | 3347/41250 [8:04:38<91:10:59,  8.66s/it][2025-04-26 12:13:15,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 12:13:15,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.82 | bwd_microstep: 5857.83 | bwd_inner_microstep: 5656.12 | bwd_allreduce_microstep: 201.66 | step_microstep: 18.66
[2025-04-26 12:13:15,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.82 | bwd: 5857.84 | bwd_inner: 5656.12 | bwd_allreduce: 201.68 | step: 18.66

  8%|▊         | 3348/41250 [8:04:47<91:35:47,  8.70s/it]
                                                         
{'loss': 0.0982, 'grad_norm': 0.7912043333053589, 'learning_rate': 3.972616396806978e-05, 'epoch': 0.81}

  8%|▊         | 3348/41250 [8:04:47<91:35:47,  8.70s/it][2025-04-26 12:13:24,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 12:13:24,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.08 | bwd_microstep: 5733.75 | bwd_inner_microstep: 5720.99 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.71
[2025-04-26 12:13:24,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5733.76 | bwd_inner: 5720.99 | bwd_allreduce: 12.73 | step: 18.72

  8%|▊         | 3349/41250 [8:04:55<91:30:54,  8.69s/it]
                                                         
{'loss': 0.03, 'grad_norm': 0.2998361885547638, 'learning_rate': 3.972590494098241e-05, 'epoch': 0.81}

  8%|▊         | 3349/41250 [8:04:55<91:30:54,  8.69s/it][2025-04-26 12:13:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 12:13:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.20 | bwd_microstep: 5762.97 | bwd_inner_microstep: 5707.56 | bwd_allreduce_microstep: 55.36 | step_microstep: 18.91
[2025-04-26 12:13:33,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.20 | bwd: 5762.99 | bwd_inner: 5707.56 | bwd_allreduce: 55.38 | step: 18.91

  8%|▊         | 3350/41250 [8:05:04<91:31:17,  8.69s/it]
                                                         
{'loss': 0.1339, 'grad_norm': 2.227112054824829, 'learning_rate': 3.9725645792288714e-05, 'epoch': 0.81}

  8%|▊         | 3350/41250 [8:05:04<91:31:17,  8.69s/it][2025-04-26 12:13:42,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:13:42,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.93 | bwd_microstep: 5771.88 | bwd_inner_microstep: 5759.08 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.25
[2025-04-26 12:13:42,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.93 | bwd: 5771.89 | bwd_inner: 5759.08 | bwd_allreduce: 12.77 | step: 18.25

  8%|▊         | 3351/41250 [8:05:13<91:40:42,  8.71s/it]
                                                         
{'loss': 0.1785, 'grad_norm': 1.0840444564819336, 'learning_rate': 3.97253865219903e-05, 'epoch': 0.81}

  8%|▊         | 3351/41250 [8:05:13<91:40:42,  8.71s/it][2025-04-26 12:13:50,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:13:50,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.80 | bwd_microstep: 5761.24 | bwd_inner_microstep: 5711.51 | bwd_allreduce_microstep: 49.69 | step_microstep: 18.41
[2025-04-26 12:13:50,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.80 | bwd: 5761.25 | bwd_inner: 5711.51 | bwd_allreduce: 49.70 | step: 18.41

  8%|▊         | 3352/41250 [8:05:21<91:38:49,  8.71s/it]
                                                         
{'loss': 0.1633, 'grad_norm': 1.3310558795928955, 'learning_rate': 3.9725127130088754e-05, 'epoch': 0.81}

  8%|▊         | 3352/41250 [8:05:21<91:38:49,  8.71s/it][2025-04-26 12:13:59,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:13:59,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.86 | bwd_microstep: 5724.63 | bwd_inner_microstep: 5655.14 | bwd_allreduce_microstep: 69.45 | step_microstep: 17.82
[2025-04-26 12:13:59,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.86 | bwd: 5724.64 | bwd_inner: 5655.14 | bwd_allreduce: 69.47 | step: 17.82

  8%|▊         | 3353/41250 [8:05:30<91:25:54,  8.69s/it]
                                                         
{'loss': 0.2871, 'grad_norm': 2.1582157611846924, 'learning_rate': 3.972486761658567e-05, 'epoch': 0.81}

  8%|▊         | 3353/41250 [8:05:30<91:25:54,  8.69s/it][2025-04-26 12:14:08,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 12:14:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.97 | bwd_microstep: 5712.97 | bwd_inner_microstep: 5700.41 | bwd_allreduce_microstep: 12.52 | step_microstep: 18.40
[2025-04-26 12:14:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.97 | bwd: 5712.98 | bwd_inner: 5700.41 | bwd_allreduce: 12.53 | step: 18.40

  8%|▊         | 3354/41250 [8:05:39<91:18:32,  8.67s/it]
                                                         
{'loss': 0.1646, 'grad_norm': 1.8375768661499023, 'learning_rate': 3.9724607981482665e-05, 'epoch': 0.81}

  8%|▊         | 3354/41250 [8:05:39<91:18:32,  8.67s/it][2025-04-26 12:14:16,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:14:16,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.32 | bwd_microstep: 5795.00 | bwd_inner_microstep: 5665.11 | bwd_allreduce_microstep: 129.85 | step_microstep: 17.93
[2025-04-26 12:14:16,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.32 | bwd: 5795.01 | bwd_inner: 5665.11 | bwd_allreduce: 129.87 | step: 17.93

  8%|▊         | 3355/41250 [8:05:47<91:26:29,  8.69s/it]
                                                         
{'loss': 0.3168, 'grad_norm': 2.876072406768799, 'learning_rate': 3.9724348224781334e-05, 'epoch': 0.81}

  8%|▊         | 3355/41250 [8:05:47<91:26:29,  8.69s/it][2025-04-26 12:14:25,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.05
[2025-04-26 12:14:25,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.50 | bwd_microstep: 5717.85 | bwd_inner_microstep: 5669.56 | bwd_allreduce_microstep: 48.24 | step_microstep: 18.66
[2025-04-26 12:14:25,446] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.50 | bwd: 5717.86 | bwd_inner: 5669.56 | bwd_allreduce: 48.26 | step: 18.66

  8%|▊         | 3356/41250 [8:05:56<91:17:43,  8.67s/it]
                                                         
{'loss': 0.2457, 'grad_norm': 1.9948822259902954, 'learning_rate': 3.9724088346483274e-05, 'epoch': 0.81}

  8%|▊         | 3356/41250 [8:05:56<91:17:43,  8.67s/it][2025-04-26 12:14:34,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:14:34,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.08 | bwd_microstep: 6026.75 | bwd_inner_microstep: 5770.15 | bwd_allreduce_microstep: 256.55 | step_microstep: 18.55
[2025-04-26 12:14:34,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.08 | bwd: 6026.76 | bwd_inner: 5770.15 | bwd_allreduce: 256.57 | step: 18.55

  8%|▊         | 3357/41250 [8:06:05<92:18:33,  8.77s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.4274243116378784, 'learning_rate': 3.972382834659009e-05, 'epoch': 0.81}

  8%|▊         | 3357/41250 [8:06:05<92:18:33,  8.77s/it][2025-04-26 12:14:43,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:14:43,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.01 | bwd_microstep: 5721.94 | bwd_inner_microstep: 5708.91 | bwd_allreduce_microstep: 12.98 | step_microstep: 18.66
[2025-04-26 12:14:43,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.01 | bwd: 5721.95 | bwd_inner: 5708.91 | bwd_allreduce: 13.00 | step: 18.66

  8%|▊         | 3358/41250 [8:06:14<91:56:35,  8.74s/it]
                                                         
{'loss': 0.1988, 'grad_norm': 1.2338238954544067, 'learning_rate': 3.972356822510338e-05, 'epoch': 0.81}

  8%|▊         | 3358/41250 [8:06:14<91:56:35,  8.74s/it][2025-04-26 12:14:51,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.10 | optimizer_step: 1.23
[2025-04-26 12:14:51,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2891.80 | bwd_microstep: 5786.68 | bwd_inner_microstep: 5773.24 | bwd_allreduce_microstep: 13.38 | step_microstep: 19.79
[2025-04-26 12:14:51,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2891.80 | bwd: 5786.70 | bwd_inner: 5773.24 | bwd_allreduce: 13.41 | step: 19.79

  8%|▊         | 3359/41250 [8:06:23<91:59:39,  8.74s/it]
                                                         
{'loss': 0.326, 'grad_norm': 2.0869054794311523, 'learning_rate': 3.9723307982024756e-05, 'epoch': 0.81}

  8%|▊         | 3359/41250 [8:06:23<91:59:39,  8.74s/it][2025-04-26 12:15:00,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 12:15:00,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.32 | bwd_microstep: 5793.44 | bwd_inner_microstep: 5693.85 | bwd_allreduce_microstep: 99.54 | step_microstep: 18.80
[2025-04-26 12:15:00,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.32 | bwd: 5793.45 | bwd_inner: 5693.85 | bwd_allreduce: 99.56 | step: 18.80

  8%|▊         | 3360/41250 [8:06:31<91:56:26,  8.74s/it]
                                                         
{'loss': 0.2643, 'grad_norm': 2.073742389678955, 'learning_rate': 3.972304761735581e-05, 'epoch': 0.81}

  8%|▊         | 3360/41250 [8:06:31<91:56:26,  8.74s/it][2025-04-26 12:15:09,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:15:09,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5722.55 | bwd_inner_microstep: 5701.14 | bwd_allreduce_microstep: 21.36 | step_microstep: 18.54
[2025-04-26 12:15:09,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5722.56 | bwd_inner: 5701.14 | bwd_allreduce: 21.38 | step: 18.54

  8%|▊         | 3361/41250 [8:06:40<91:40:30,  8.71s/it]
                                                         
{'loss': 0.0946, 'grad_norm': 1.3499486446380615, 'learning_rate': 3.972278713109816e-05, 'epoch': 0.81}

  8%|▊         | 3361/41250 [8:06:40<91:40:30,  8.71s/it][2025-04-26 12:15:17,933] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:15:17,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.80 | bwd_microstep: 5776.11 | bwd_inner_microstep: 5705.82 | bwd_allreduce_microstep: 70.25 | step_microstep: 18.70
[2025-04-26 12:15:17,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.80 | bwd: 5776.12 | bwd_inner: 5705.82 | bwd_allreduce: 70.26 | step: 18.70

  8%|▊         | 3362/41250 [8:06:49<91:40:26,  8.71s/it]
                                                         
{'loss': 0.159, 'grad_norm': 1.3514454364776611, 'learning_rate': 3.9722526523253406e-05, 'epoch': 0.82}

  8%|▊         | 3362/41250 [8:06:49<91:40:26,  8.71s/it][2025-04-26 12:15:26,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.14 | optimizer_step: 1.18
[2025-04-26 12:15:26,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.33 | bwd_microstep: 5752.91 | bwd_inner_microstep: 5715.13 | bwd_allreduce_microstep: 37.71 | step_microstep: 19.82
[2025-04-26 12:15:26,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.33 | bwd: 5752.92 | bwd_inner: 5715.13 | bwd_allreduce: 37.74 | step: 19.82

  8%|▊         | 3363/41250 [8:06:57<91:37:14,  8.71s/it]
                                                         
{'loss': 0.2336, 'grad_norm': 1.3153469562530518, 'learning_rate': 3.972226579382316e-05, 'epoch': 0.82}

  8%|▊         | 3363/41250 [8:06:57<91:37:14,  8.71s/it][2025-04-26 12:15:35,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:15:35,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.53 | bwd_microstep: 5780.52 | bwd_inner_microstep: 5659.24 | bwd_allreduce_microstep: 121.23 | step_microstep: 18.27
[2025-04-26 12:15:35,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.53 | bwd: 5780.53 | bwd_inner: 5659.24 | bwd_allreduce: 121.25 | step: 18.27

  8%|▊         | 3364/41250 [8:07:06<91:33:23,  8.70s/it]
                                                         
{'loss': 0.2271, 'grad_norm': 1.6875090599060059, 'learning_rate': 3.972200494280902e-05, 'epoch': 0.82}

  8%|▊         | 3364/41250 [8:07:06<91:33:23,  8.70s/it][2025-04-26 12:15:43,937] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:15:43,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.17 | bwd_microstep: 5707.61 | bwd_inner_microstep: 5664.51 | bwd_allreduce_microstep: 43.06 | step_microstep: 18.38
[2025-04-26 12:15:43,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.17 | bwd: 5707.63 | bwd_inner: 5664.51 | bwd_allreduce: 43.08 | step: 18.39

  8%|▊         | 3365/41250 [8:07:15<91:18:35,  8.68s/it]
                                                         
{'loss': 0.1744, 'grad_norm': 2.5832436084747314, 'learning_rate': 3.97217439702126e-05, 'epoch': 0.82}

  8%|▊         | 3365/41250 [8:07:15<91:18:35,  8.68s/it][2025-04-26 12:15:52,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:15:52,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.20 | bwd_microstep: 5703.32 | bwd_inner_microstep: 5661.10 | bwd_allreduce_microstep: 42.17 | step_microstep: 18.85
[2025-04-26 12:15:52,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.20 | bwd: 5703.33 | bwd_inner: 5661.10 | bwd_allreduce: 42.18 | step: 18.86

  8%|▊         | 3366/41250 [8:07:23<91:06:22,  8.66s/it]
                                                         
{'loss': 0.0984, 'grad_norm': 0.7690770030021667, 'learning_rate': 3.97214828760355e-05, 'epoch': 0.82}

  8%|▊         | 3366/41250 [8:07:23<91:06:22,  8.66s/it][2025-04-26 12:16:01,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 12:16:01,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5782.83 | bwd_inner_microstep: 5661.33 | bwd_allreduce_microstep: 121.44 | step_microstep: 18.70
[2025-04-26 12:16:01,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5782.85 | bwd_inner: 5661.33 | bwd_allreduce: 121.46 | step: 18.70

  8%|▊         | 3367/41250 [8:07:32<91:13:32,  8.67s/it]
                                                         
{'loss': 0.0491, 'grad_norm': 0.8003374934196472, 'learning_rate': 3.972122166027934e-05, 'epoch': 0.82}

  8%|▊         | 3367/41250 [8:07:32<91:13:32,  8.67s/it][2025-04-26 12:16:09,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.13
[2025-04-26 12:16:09,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.48 | bwd_microstep: 5711.89 | bwd_inner_microstep: 5661.40 | bwd_allreduce_microstep: 50.45 | step_microstep: 19.17
[2025-04-26 12:16:09,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.48 | bwd: 5711.91 | bwd_inner: 5661.40 | bwd_allreduce: 50.47 | step: 19.17

  8%|▊         | 3368/41250 [8:07:41<91:02:20,  8.65s/it]
                                                         
{'loss': 0.121, 'grad_norm': 1.0259681940078735, 'learning_rate': 3.972096032294573e-05, 'epoch': 0.82}

  8%|▊         | 3368/41250 [8:07:41<91:02:20,  8.65s/it][2025-04-26 12:16:18,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 12:16:18,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.95 | bwd_microstep: 5709.01 | bwd_inner_microstep: 5696.12 | bwd_allreduce_microstep: 12.85 | step_microstep: 18.75
[2025-04-26 12:16:18,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.95 | bwd: 5709.03 | bwd_inner: 5696.12 | bwd_allreduce: 12.87 | step: 18.75

  8%|▊         | 3369/41250 [8:07:49<91:00:47,  8.65s/it]
                                                         
{'loss': 0.3008, 'grad_norm': 1.5485414266586304, 'learning_rate': 3.972069886403628e-05, 'epoch': 0.82}

  8%|▊         | 3369/41250 [8:07:49<91:00:47,  8.65s/it][2025-04-26 12:16:27,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 12:16:27,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.91 | bwd_microstep: 5748.45 | bwd_inner_microstep: 5664.39 | bwd_allreduce_microstep: 84.01 | step_microstep: 18.74
[2025-04-26 12:16:27,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.92 | bwd: 5748.47 | bwd_inner: 5664.39 | bwd_allreduce: 84.03 | step: 18.75

  8%|▊         | 3370/41250 [8:07:58<91:03:05,  8.65s/it]
                                                         
{'loss': 0.0706, 'grad_norm': 0.713882327079773, 'learning_rate': 3.97204372835526e-05, 'epoch': 0.82}

  8%|▊         | 3370/41250 [8:07:58<91:03:05,  8.65s/it][2025-04-26 12:16:35,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 12:16:35,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.21 | bwd_microstep: 5692.58 | bwd_inner_microstep: 5643.69 | bwd_allreduce_microstep: 48.83 | step_microstep: 18.92
[2025-04-26 12:16:35,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.21 | bwd: 5692.59 | bwd_inner: 5643.69 | bwd_allreduce: 48.86 | step: 18.92

  8%|▊         | 3371/41250 [8:08:06<90:54:14,  8.64s/it]
                                                         
{'loss': 0.1736, 'grad_norm': 1.0232453346252441, 'learning_rate': 3.97201755814963e-05, 'epoch': 0.82}

  8%|▊         | 3371/41250 [8:08:06<90:54:14,  8.64s/it][2025-04-26 12:16:44,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.95
[2025-04-26 12:16:44,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.00 | bwd_microstep: 5747.14 | bwd_inner_microstep: 5655.57 | bwd_allreduce_microstep: 91.53 | step_microstep: 18.65
[2025-04-26 12:16:44,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.00 | bwd: 5747.15 | bwd_inner: 5655.57 | bwd_allreduce: 91.54 | step: 18.66

  8%|▊         | 3372/41250 [8:08:15<90:57:33,  8.64s/it]
                                                         
{'loss': 0.153, 'grad_norm': 1.0673459768295288, 'learning_rate': 3.9719913757868995e-05, 'epoch': 0.82}

  8%|▊         | 3372/41250 [8:08:15<90:57:33,  8.64s/it][2025-04-26 12:16:53,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:16:53,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2883.36 | bwd_microstep: 5776.35 | bwd_inner_microstep: 5763.60 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.74
[2025-04-26 12:16:53,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2883.36 | bwd: 5776.36 | bwd_inner: 5763.60 | bwd_allreduce: 12.72 | step: 18.74

  8%|▊         | 3373/41250 [8:08:24<91:15:39,  8.67s/it]
                                                         
{'loss': 0.182, 'grad_norm': 1.4746092557907104, 'learning_rate': 3.971965181267231e-05, 'epoch': 0.82}

  8%|▊         | 3373/41250 [8:08:24<91:15:39,  8.67s/it][2025-04-26 12:17:01,849] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 12:17:01,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.11 | bwd_microstep: 5771.55 | bwd_inner_microstep: 5649.73 | bwd_allreduce_microstep: 121.76 | step_microstep: 19.03
[2025-04-26 12:17:01,850] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.11 | bwd: 5771.56 | bwd_inner: 5649.73 | bwd_allreduce: 121.79 | step: 19.03

  8%|▊         | 3374/41250 [8:08:33<91:16:31,  8.68s/it]
                                                         
{'loss': 0.2426, 'grad_norm': 2.089052438735962, 'learning_rate': 3.971938974590784e-05, 'epoch': 0.82}

  8%|▊         | 3374/41250 [8:08:33<91:16:31,  8.68s/it][2025-04-26 12:17:10,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 12:17:10,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2879.08 | bwd_microstep: 5781.41 | bwd_inner_microstep: 5768.52 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.84
[2025-04-26 12:17:10,594] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2879.08 | bwd: 5781.42 | bwd_inner: 5768.52 | bwd_allreduce: 12.86 | step: 18.84

  8%|▊         | 3375/41250 [8:08:41<91:29:47,  8.70s/it]
                                                         
{'loss': 0.0797, 'grad_norm': 0.6196321249008179, 'learning_rate': 3.9719127557577216e-05, 'epoch': 0.82}

  8%|▊         | 3375/41250 [8:08:41<91:29:47,  8.70s/it][2025-04-26 12:17:19,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:17:19,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.91 | bwd_microstep: 5694.80 | bwd_inner_microstep: 5652.90 | bwd_allreduce_microstep: 41.85 | step_microstep: 18.82
[2025-04-26 12:17:19,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.91 | bwd: 5694.82 | bwd_inner: 5652.90 | bwd_allreduce: 41.87 | step: 18.82

  8%|▊         | 3376/41250 [8:08:50<91:12:16,  8.67s/it]
                                                         
{'loss': 0.1876, 'grad_norm': 2.2318241596221924, 'learning_rate': 3.971886524768205e-05, 'epoch': 0.82}

  8%|▊         | 3376/41250 [8:08:50<91:12:16,  8.67s/it][2025-04-26 12:17:27,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 1.06 | optimizer_step: 1.16
[2025-04-26 12:17:27,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.45 | bwd_microstep: 5771.78 | bwd_inner_microstep: 5644.45 | bwd_allreduce_microstep: 127.28 | step_microstep: 19.16
[2025-04-26 12:17:27,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.45 | bwd: 5771.79 | bwd_inner: 5644.45 | bwd_allreduce: 127.30 | step: 19.16

  8%|▊         | 3377/41250 [8:08:59<91:12:13,  8.67s/it]
                                                         
{'loss': 0.16, 'grad_norm': 1.2193156480789185, 'learning_rate': 3.9718602816223955e-05, 'epoch': 0.82}

  8%|▊         | 3377/41250 [8:08:59<91:12:13,  8.67s/it][2025-04-26 12:17:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 12:17:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.13 | bwd_microstep: 5764.81 | bwd_inner_microstep: 5642.31 | bwd_allreduce_microstep: 122.45 | step_microstep: 19.02
[2025-04-26 12:17:36,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.13 | bwd: 5764.82 | bwd_inner: 5642.31 | bwd_allreduce: 122.47 | step: 19.02

  8%|▊         | 3378/41250 [8:09:07<91:14:26,  8.67s/it]
                                                         
{'loss': 0.1715, 'grad_norm': 1.9083882570266724, 'learning_rate': 3.9718340263204556e-05, 'epoch': 0.82}

  8%|▊         | 3378/41250 [8:09:07<91:14:26,  8.67s/it][2025-04-26 12:17:45,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.06
[2025-04-26 12:17:45,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.14 | bwd_microstep: 5691.49 | bwd_inner_microstep: 5654.73 | bwd_allreduce_microstep: 36.71 | step_microstep: 18.81
[2025-04-26 12:17:45,153] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.14 | bwd: 5691.51 | bwd_inner: 5654.73 | bwd_allreduce: 36.73 | step: 18.81

  8%|▊         | 3379/41250 [8:09:16<91:00:40,  8.65s/it]
                                                         
{'loss': 0.1163, 'grad_norm': 0.7631301879882812, 'learning_rate': 3.971807758862546e-05, 'epoch': 0.82}

  8%|▊         | 3379/41250 [8:09:16<91:00:40,  8.65s/it][2025-04-26 12:17:53,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 1.07
[2025-04-26 12:17:53,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.25 | bwd_microstep: 5796.37 | bwd_inner_microstep: 5649.35 | bwd_allreduce_microstep: 146.97 | step_microstep: 18.80
[2025-04-26 12:17:53,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.25 | bwd: 5796.38 | bwd_inner: 5649.35 | bwd_allreduce: 146.99 | step: 18.80

  8%|▊         | 3380/41250 [8:09:25<91:10:10,  8.67s/it]
                                                         
{'loss': 0.2399, 'grad_norm': 1.040600299835205, 'learning_rate': 3.97178147924883e-05, 'epoch': 0.82}

  8%|▊         | 3380/41250 [8:09:25<91:10:10,  8.67s/it][2025-04-26 12:18:02,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.30 | optimizer_step: 1.01
[2025-04-26 12:18:02,656] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.18 | bwd_microstep: 5868.81 | bwd_inner_microstep: 5706.15 | bwd_allreduce_microstep: 162.61 | step_microstep: 19.46
[2025-04-26 12:18:02,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.18 | bwd: 5868.82 | bwd_inner: 5706.15 | bwd_allreduce: 162.63 | step: 19.46

  8%|▊         | 3381/41250 [8:09:33<91:35:34,  8.71s/it]
                                                         
{'loss': 0.1528, 'grad_norm': 0.6331652402877808, 'learning_rate': 3.971755187479469e-05, 'epoch': 0.82}

  8%|▊         | 3381/41250 [8:09:33<91:35:34,  8.71s/it][2025-04-26 12:18:11,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 1.03
[2025-04-26 12:18:11,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.91 | bwd_microstep: 5706.80 | bwd_inner_microstep: 5659.84 | bwd_allreduce_microstep: 46.91 | step_microstep: 18.44
[2025-04-26 12:18:11,270] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.91 | bwd: 5706.81 | bwd_inner: 5659.84 | bwd_allreduce: 46.93 | step: 18.44

  8%|▊         | 3382/41250 [8:09:42<91:17:24,  8.68s/it]
                                                         
{'loss': 0.1855, 'grad_norm': 1.7855098247528076, 'learning_rate': 3.971728883554625e-05, 'epoch': 0.82}

  8%|▊         | 3382/41250 [8:09:42<91:17:24,  8.68s/it][2025-04-26 12:18:19,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:18:19,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.83 | bwd_microstep: 5774.76 | bwd_inner_microstep: 5649.89 | bwd_allreduce_microstep: 124.82 | step_microstep: 18.32
[2025-04-26 12:18:19,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.83 | bwd: 5774.77 | bwd_inner: 5649.89 | bwd_allreduce: 124.84 | step: 18.32

  8%|▊         | 3383/41250 [8:09:51<91:18:31,  8.68s/it]
                                                         
{'loss': 0.306, 'grad_norm': 3.708641529083252, 'learning_rate': 3.9717025674744604e-05, 'epoch': 0.82}

  8%|▊         | 3383/41250 [8:09:51<91:18:31,  8.68s/it][2025-04-26 12:18:28,640] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 12:18:28,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.16 | bwd_microstep: 5754.14 | bwd_inner_microstep: 5717.43 | bwd_allreduce_microstep: 36.66 | step_microstep: 18.69
[2025-04-26 12:18:28,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.16 | bwd: 5754.15 | bwd_inner: 5717.43 | bwd_allreduce: 36.68 | step: 18.69

  8%|▊         | 3384/41250 [8:09:59<91:19:14,  8.68s/it]
                                                         
{'loss': 0.094, 'grad_norm': 0.9962443709373474, 'learning_rate': 3.971676239239137e-05, 'epoch': 0.82}

  8%|▊         | 3384/41250 [8:09:59<91:19:14,  8.68s/it][2025-04-26 12:18:37,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:18:37,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.07 | bwd_microstep: 5755.95 | bwd_inner_microstep: 5653.37 | bwd_allreduce_microstep: 102.54 | step_microstep: 18.77
[2025-04-26 12:18:37,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.07 | bwd: 5755.96 | bwd_inner: 5653.37 | bwd_allreduce: 102.55 | step: 18.77

  8%|▊         | 3385/41250 [8:10:08<91:15:32,  8.68s/it]
                                                         
{'loss': 0.1677, 'grad_norm': 1.5614005327224731, 'learning_rate': 3.9716498988488174e-05, 'epoch': 0.82}

  8%|▊         | 3385/41250 [8:10:08<91:15:32,  8.68s/it][2025-04-26 12:18:45,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:18:45,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.38 | bwd_microstep: 5707.30 | bwd_inner_microstep: 5651.12 | bwd_allreduce_microstep: 56.14 | step_microstep: 18.67
[2025-04-26 12:18:45,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.38 | bwd: 5707.32 | bwd_inner: 5651.12 | bwd_allreduce: 56.16 | step: 18.67

  8%|▊         | 3386/41250 [8:10:17<91:02:14,  8.66s/it]
                                                         
{'loss': 0.1419, 'grad_norm': 1.4853589534759521, 'learning_rate': 3.971623546303664e-05, 'epoch': 0.82}

  8%|▊         | 3386/41250 [8:10:17<91:02:14,  8.66s/it][2025-04-26 12:18:54,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 12:18:54,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.99 | bwd_microstep: 5775.26 | bwd_inner_microstep: 5655.35 | bwd_allreduce_microstep: 119.87 | step_microstep: 18.89
[2025-04-26 12:18:54,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.99 | bwd: 5775.28 | bwd_inner: 5655.35 | bwd_allreduce: 119.89 | step: 18.89

  8%|▊         | 3387/41250 [8:10:25<91:07:27,  8.66s/it]
                                                         
{'loss': 0.2323, 'grad_norm': 4.855961322784424, 'learning_rate': 3.971597181603839e-05, 'epoch': 0.82}

  8%|▊         | 3387/41250 [8:10:25<91:07:27,  8.66s/it][2025-04-26 12:19:03,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:19:03,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.04 | bwd_microstep: 5712.60 | bwd_inner_microstep: 5699.69 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.65
[2025-04-26 12:19:03,239] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.04 | bwd: 5712.61 | bwd_inner: 5699.69 | bwd_allreduce: 12.88 | step: 18.66

  8%|▊         | 3388/41250 [8:10:34<91:03:39,  8.66s/it]
                                                         
{'loss': 0.2146, 'grad_norm': 2.2048683166503906, 'learning_rate': 3.971570804749506e-05, 'epoch': 0.82}

  8%|▊         | 3388/41250 [8:10:34<91:03:39,  8.66s/it][2025-04-26 12:19:11,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:19:11,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.96 | bwd_microstep: 5714.22 | bwd_inner_microstep: 5701.36 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.68
[2025-04-26 12:19:11,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.96 | bwd: 5714.23 | bwd_inner: 5701.36 | bwd_allreduce: 12.83 | step: 18.69

  8%|▊         | 3389/41250 [8:10:43<91:03:14,  8.66s/it]
                                                         
{'loss': 0.3488, 'grad_norm': 2.810239791870117, 'learning_rate': 3.9715444157408256e-05, 'epoch': 0.82}

  8%|▊         | 3389/41250 [8:10:43<91:03:14,  8.66s/it][2025-04-26 12:19:20,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 12:19:20,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.34 | bwd_microstep: 5731.24 | bwd_inner_microstep: 5657.58 | bwd_allreduce_microstep: 73.61 | step_microstep: 18.82
[2025-04-26 12:19:20,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.34 | bwd: 5731.25 | bwd_inner: 5657.58 | bwd_allreduce: 73.63 | step: 18.82

  8%|▊         | 3390/41250 [8:10:51<91:00:27,  8.65s/it]
                                                         
{'loss': 0.1338, 'grad_norm': 0.7800670862197876, 'learning_rate': 3.971518014577963e-05, 'epoch': 0.82}

  8%|▊         | 3390/41250 [8:10:51<91:00:27,  8.65s/it][2025-04-26 12:19:29,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 12:19:29,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5750.44 | bwd_inner_microstep: 5695.51 | bwd_allreduce_microstep: 54.88 | step_microstep: 18.63
[2025-04-26 12:19:29,221] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5750.45 | bwd_inner: 5695.51 | bwd_allreduce: 54.90 | step: 18.63

  8%|▊         | 3391/41250 [8:11:00<91:05:30,  8.66s/it]
                                                         
{'loss': 0.1847, 'grad_norm': 1.5069729089736938, 'learning_rate': 3.971491601261079e-05, 'epoch': 0.82}

  8%|▊         | 3391/41250 [8:11:00<91:05:30,  8.66s/it][2025-04-26 12:19:37,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:19:37,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.38 | bwd_microstep: 5725.32 | bwd_inner_microstep: 5712.62 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.41
[2025-04-26 12:19:37,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.38 | bwd: 5725.33 | bwd_inner: 5712.62 | bwd_allreduce: 12.67 | step: 18.41

  8%|▊         | 3392/41250 [8:11:09<91:06:10,  8.66s/it]
                                                         
{'loss': 0.0469, 'grad_norm': 0.879763126373291, 'learning_rate': 3.9714651757903364e-05, 'epoch': 0.82}

  8%|▊         | 3392/41250 [8:11:09<91:06:10,  8.66s/it][2025-04-26 12:19:46,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 1.19
[2025-04-26 12:19:46,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.42 | bwd_microstep: 5781.87 | bwd_inner_microstep: 5769.00 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.16
[2025-04-26 12:19:46,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.42 | bwd: 5781.88 | bwd_inner: 5769.00 | bwd_allreduce: 12.85 | step: 19.17

  8%|▊         | 3393/41250 [8:11:17<91:23:13,  8.69s/it]
                                                         
{'loss': 0.1795, 'grad_norm': 2.158912181854248, 'learning_rate': 3.9714387381659e-05, 'epoch': 0.82}

  8%|▊         | 3393/41250 [8:11:17<91:23:13,  8.69s/it][2025-04-26 12:19:55,328] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 12:19:55,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.93 | bwd_microstep: 5741.79 | bwd_inner_microstep: 5716.31 | bwd_allreduce_microstep: 25.44 | step_microstep: 18.50
[2025-04-26 12:19:55,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.93 | bwd: 5741.80 | bwd_inner: 5716.30 | bwd_allreduce: 25.45 | step: 18.50

  8%|▊         | 3394/41250 [8:11:26<91:22:52,  8.69s/it]
                                                         
{'loss': 0.3785, 'grad_norm': 1.8441439867019653, 'learning_rate': 3.971412288387931e-05, 'epoch': 0.82}

  8%|▊         | 3394/41250 [8:11:26<91:22:52,  8.69s/it][2025-04-26 12:20:04,050] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 12:20:04,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.21 | bwd_microstep: 5795.94 | bwd_inner_microstep: 5712.60 | bwd_allreduce_microstep: 83.30 | step_microstep: 18.81
[2025-04-26 12:20:04,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.21 | bwd: 5795.96 | bwd_inner: 5712.60 | bwd_allreduce: 83.32 | step: 18.81

  8%|▊         | 3395/41250 [8:11:35<91:28:42,  8.70s/it]
                                                         
{'loss': 0.1458, 'grad_norm': 1.740136981010437, 'learning_rate': 3.971385826456592e-05, 'epoch': 0.82}

  8%|▊         | 3395/41250 [8:11:35<91:28:42,  8.70s/it][2025-04-26 12:20:12,768] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.06
[2025-04-26 12:20:12,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.53 | bwd_microstep: 5804.55 | bwd_inner_microstep: 5665.58 | bwd_allreduce_microstep: 138.92 | step_microstep: 18.37
[2025-04-26 12:20:12,769] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.53 | bwd: 5804.56 | bwd_inner: 5665.58 | bwd_allreduce: 138.94 | step: 18.38

  8%|▊         | 3396/41250 [8:11:43<91:32:08,  8.71s/it]
                                                         
{'loss': 0.2092, 'grad_norm': 1.9128671884536743, 'learning_rate': 3.9713593523720484e-05, 'epoch': 0.82}

  8%|▊         | 3396/41250 [8:11:43<91:32:08,  8.71s/it][2025-04-26 12:20:21,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:20:21,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.00 | bwd_microstep: 5901.57 | bwd_inner_microstep: 5711.78 | bwd_allreduce_microstep: 189.75 | step_microstep: 18.34
[2025-04-26 12:20:21,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.00 | bwd: 5901.58 | bwd_inner: 5711.78 | bwd_allreduce: 189.77 | step: 18.34

  8%|▊         | 3397/41250 [8:11:52<91:57:50,  8.75s/it]
                                                         
{'loss': 0.154, 'grad_norm': 1.4321345090866089, 'learning_rate': 3.971332866134462e-05, 'epoch': 0.82}

  8%|▊         | 3397/41250 [8:11:52<91:57:50,  8.75s/it][2025-04-26 12:20:30,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:20:30,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.67 | bwd_microstep: 5755.93 | bwd_inner_microstep: 5718.77 | bwd_allreduce_microstep: 37.10 | step_microstep: 18.71
[2025-04-26 12:20:30,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.67 | bwd: 5755.94 | bwd_inner: 5718.77 | bwd_allreduce: 37.13 | step: 18.71

  8%|▊         | 3398/41250 [8:12:01<91:49:37,  8.73s/it]
                                                         
{'loss': 0.2144, 'grad_norm': 1.5624691247940063, 'learning_rate': 3.971306367743996e-05, 'epoch': 0.82}

  8%|▊         | 3398/41250 [8:12:01<91:49:37,  8.73s/it][2025-04-26 12:20:38,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 12:20:38,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5716.50 | bwd_inner_microstep: 5659.70 | bwd_allreduce_microstep: 56.75 | step_microstep: 18.58
[2025-04-26 12:20:38,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.37 | bwd: 5716.52 | bwd_inner: 5659.70 | bwd_allreduce: 56.78 | step: 18.58

  8%|▊         | 3399/41250 [8:12:10<91:32:37,  8.71s/it]
                                                         
{'loss': 0.2358, 'grad_norm': 1.671422004699707, 'learning_rate': 3.971279857200814e-05, 'epoch': 0.82}

  8%|▊         | 3399/41250 [8:12:10<91:32:37,  8.71s/it][2025-04-26 12:20:47,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.06 | optimizer_step: 0.94
[2025-04-26 12:20:47,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5722.16 | bwd_inner_microstep: 5660.26 | bwd_allreduce_microstep: 61.86 | step_microstep: 19.28
[2025-04-26 12:20:47,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5722.18 | bwd_inner: 5660.26 | bwd_allreduce: 61.88 | step: 19.28

  8%|▊         | 3400/41250 [8:12:18<91:17:45,  8.68s/it]
                                                         
{'loss': 0.2472, 'grad_norm': 1.2339074611663818, 'learning_rate': 3.971253334505079e-05, 'epoch': 0.82}

  8%|▊         | 3400/41250 [8:12:18<91:17:45,  8.68s/it][2025-04-26 12:20:56,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:20:56,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.96 | bwd_microstep: 5800.80 | bwd_inner_microstep: 5663.32 | bwd_allreduce_microstep: 137.44 | step_microstep: 18.53
[2025-04-26 12:20:56,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.96 | bwd: 5800.81 | bwd_inner: 5663.32 | bwd_allreduce: 137.45 | step: 18.54

  8%|▊         | 3401/41250 [8:12:27<91:23:45,  8.69s/it]
                                                         
{'loss': 0.0297, 'grad_norm': 0.4269160330295563, 'learning_rate': 3.971226799656955e-05, 'epoch': 0.82}

  8%|▊         | 3401/41250 [8:12:27<91:23:45,  8.69s/it][2025-04-26 12:21:04,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:21:04,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.17 | bwd_microstep: 5736.95 | bwd_inner_microstep: 5651.93 | bwd_allreduce_microstep: 84.98 | step_microstep: 18.44
[2025-04-26 12:21:04,954] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.17 | bwd: 5736.97 | bwd_inner: 5651.93 | bwd_allreduce: 85.00 | step: 18.44

  8%|▊         | 3402/41250 [8:12:36<91:15:35,  8.68s/it]
                                                         
{'loss': 0.2208, 'grad_norm': 2.470121145248413, 'learning_rate': 3.9712002526566056e-05, 'epoch': 0.82}

  8%|▊         | 3402/41250 [8:12:36<91:15:35,  8.68s/it][2025-04-26 12:21:13,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.22 | optimizer_step: 0.89
[2025-04-26 12:21:13,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.74 | bwd_microstep: 5776.46 | bwd_inner_microstep: 5656.88 | bwd_allreduce_microstep: 119.52 | step_microstep: 18.81
[2025-04-26 12:21:13,659] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.74 | bwd: 5776.47 | bwd_inner: 5656.88 | bwd_allreduce: 119.55 | step: 18.81

  8%|▊         | 3403/41250 [8:12:44<91:20:17,  8.69s/it]
                                                         
{'loss': 0.1384, 'grad_norm': 1.8131229877471924, 'learning_rate': 3.9711736935041954e-05, 'epoch': 0.82}

  8%|▊         | 3403/41250 [8:12:44<91:20:17,  8.69s/it][2025-04-26 12:21:22,347] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.13
[2025-04-26 12:21:22,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.61 | bwd_microstep: 5770.70 | bwd_inner_microstep: 5685.00 | bwd_allreduce_microstep: 85.65 | step_microstep: 18.96
[2025-04-26 12:21:22,348] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.61 | bwd: 5770.71 | bwd_inner: 5685.00 | bwd_allreduce: 85.67 | step: 18.97

  8%|▊         | 3404/41250 [8:12:53<91:20:10,  8.69s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 1.7541781663894653, 'learning_rate': 3.971147122199885e-05, 'epoch': 0.83}

  8%|▊         | 3404/41250 [8:12:53<91:20:10,  8.69s/it][2025-04-26 12:21:31,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.07 | optimizer_step: 0.91
[2025-04-26 12:21:31,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.42 | bwd_microstep: 5797.77 | bwd_inner_microstep: 5658.26 | bwd_allreduce_microstep: 139.46 | step_microstep: 18.93
[2025-04-26 12:21:31,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.42 | bwd: 5797.79 | bwd_inner: 5658.26 | bwd_allreduce: 139.48 | step: 18.93

  8%|▊         | 3405/41250 [8:13:02<91:24:22,  8.70s/it]
                                                         
{'loss': 0.0847, 'grad_norm': 0.7767948508262634, 'learning_rate': 3.971120538743842e-05, 'epoch': 0.83}

  8%|▊         | 3405/41250 [8:13:02<91:24:22,  8.70s/it][2025-04-26 12:21:39,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.01
[2025-04-26 12:21:39,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.39 | bwd_microstep: 5714.25 | bwd_inner_microstep: 5701.45 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.79
[2025-04-26 12:21:39,708] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.39 | bwd: 5714.27 | bwd_inner: 5701.45 | bwd_allreduce: 12.78 | step: 18.79

  8%|▊         | 3406/41250 [8:13:10<91:15:57,  8.68s/it]
                                                         
{'loss': 0.1348, 'grad_norm': 0.8246582746505737, 'learning_rate': 3.971093943136228e-05, 'epoch': 0.83}

  8%|▊         | 3406/41250 [8:13:10<91:15:57,  8.68s/it][2025-04-26 12:21:48,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.05
[2025-04-26 12:21:48,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.71 | bwd_microstep: 5717.98 | bwd_inner_microstep: 5648.23 | bwd_allreduce_microstep: 69.70 | step_microstep: 19.01
[2025-04-26 12:21:48,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.71 | bwd: 5717.99 | bwd_inner: 5648.23 | bwd_allreduce: 69.72 | step: 19.01

  8%|▊         | 3407/41250 [8:13:19<91:06:28,  8.67s/it]
                                                         
{'loss': 0.1368, 'grad_norm': 1.0124311447143555, 'learning_rate': 3.971067335377208e-05, 'epoch': 0.83}

  8%|▊         | 3407/41250 [8:13:19<91:06:28,  8.67s/it][2025-04-26 12:21:57,110] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.93
[2025-04-26 12:21:57,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.25 | bwd_microstep: 5793.67 | bwd_inner_microstep: 5780.38 | bwd_allreduce_microstep: 13.24 | step_microstep: 19.00
[2025-04-26 12:21:57,111] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.25 | bwd: 5793.69 | bwd_inner: 5780.38 | bwd_allreduce: 13.26 | step: 19.00

  8%|▊         | 3408/41250 [8:13:28<91:25:04,  8.70s/it]
                                                         
{'loss': 0.2472, 'grad_norm': 1.9469425678253174, 'learning_rate': 3.971040715466945e-05, 'epoch': 0.83}

  8%|▊         | 3408/41250 [8:13:28<91:25:04,  8.70s/it][2025-04-26 12:22:05,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:22:05,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.69 | bwd_microstep: 5736.06 | bwd_inner_microstep: 5700.64 | bwd_allreduce_microstep: 35.38 | step_microstep: 18.69
[2025-04-26 12:22:05,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.69 | bwd: 5736.08 | bwd_inner: 5700.64 | bwd_allreduce: 35.39 | step: 18.69

  8%|▊         | 3409/41250 [8:13:36<91:19:36,  8.69s/it]
                                                         
{'loss': 0.0668, 'grad_norm': 0.5994327664375305, 'learning_rate': 3.971014083405604e-05, 'epoch': 0.83}

  8%|▊         | 3409/41250 [8:13:36<91:19:36,  8.69s/it][2025-04-26 12:22:14,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 12:22:14,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.33 | bwd_microstep: 5746.31 | bwd_inner_microstep: 5684.81 | bwd_allreduce_microstep: 61.46 | step_microstep: 18.62
[2025-04-26 12:22:14,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.33 | bwd: 5746.32 | bwd_inner: 5684.81 | bwd_allreduce: 61.47 | step: 18.63

  8%|▊         | 3410/41250 [8:13:45<91:15:57,  8.68s/it]
                                                         
{'loss': 0.0887, 'grad_norm': 1.413616418838501, 'learning_rate': 3.970987439193349e-05, 'epoch': 0.83}

  8%|▊         | 3410/41250 [8:13:45<91:15:57,  8.68s/it][2025-04-26 12:22:23,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:22:23,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.82 | bwd_microstep: 5694.78 | bwd_inner_microstep: 5681.36 | bwd_allreduce_microstep: 13.37 | step_microstep: 18.80
[2025-04-26 12:22:23,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.82 | bwd: 5694.79 | bwd_inner: 5681.36 | bwd_allreduce: 13.39 | step: 18.80

  8%|▊         | 3411/41250 [8:13:54<91:05:22,  8.67s/it]
                                                         
{'loss': 0.1981, 'grad_norm': 1.701534628868103, 'learning_rate': 3.970960782830344e-05, 'epoch': 0.83}

  8%|▊         | 3411/41250 [8:13:54<91:05:22,  8.67s/it][2025-04-26 12:22:31,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.95
[2025-04-26 12:22:31,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.52 | bwd_microstep: 5981.45 | bwd_inner_microstep: 5683.69 | bwd_allreduce_microstep: 297.71 | step_microstep: 18.92
[2025-04-26 12:22:31,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.52 | bwd: 5981.46 | bwd_inner: 5683.69 | bwd_allreduce: 297.73 | step: 18.92

  8%|▊         | 3412/41250 [8:14:03<91:50:51,  8.74s/it]
                                                         
{'loss': 0.1174, 'grad_norm': 1.2613098621368408, 'learning_rate': 3.970934114316752e-05, 'epoch': 0.83}

  8%|▊         | 3412/41250 [8:14:03<91:50:51,  8.74s/it][2025-04-26 12:22:40,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 12:22:40,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.72 | bwd_microstep: 5738.36 | bwd_inner_microstep: 5679.15 | bwd_allreduce_microstep: 59.15 | step_microstep: 19.10
[2025-04-26 12:22:40,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.72 | bwd: 5738.37 | bwd_inner: 5679.15 | bwd_allreduce: 59.17 | step: 19.10

  8%|▊         | 3413/41250 [8:14:11<91:36:15,  8.72s/it]
                                                         
{'loss': 0.3941, 'grad_norm': 3.485966682434082, 'learning_rate': 3.9709074336527414e-05, 'epoch': 0.83}

  8%|▊         | 3413/41250 [8:14:11<91:36:15,  8.72s/it][2025-04-26 12:22:49,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:22:49,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.83 | bwd_microstep: 5859.95 | bwd_inner_microstep: 5847.00 | bwd_allreduce_microstep: 12.90 | step_microstep: 18.54
[2025-04-26 12:22:49,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.83 | bwd: 5859.96 | bwd_inner: 5847.00 | bwd_allreduce: 12.92 | step: 18.54

  8%|▊         | 3414/41250 [8:14:20<91:44:49,  8.73s/it]
                                                         
{'loss': 0.1301, 'grad_norm': 0.9740304946899414, 'learning_rate': 3.9708807408384726e-05, 'epoch': 0.83}

  8%|▊         | 3414/41250 [8:14:20<91:44:49,  8.73s/it][2025-04-26 12:22:58,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-26 12:22:58,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.48 | bwd_microstep: 5872.04 | bwd_inner_microstep: 5693.78 | bwd_allreduce_microstep: 178.21 | step_microstep: 18.63
[2025-04-26 12:22:58,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.48 | bwd: 5872.05 | bwd_inner: 5693.78 | bwd_allreduce: 178.23 | step: 18.63

  8%|▊         | 3415/41250 [8:14:29<91:57:44,  8.75s/it]
                                                         
{'loss': 0.1486, 'grad_norm': 1.191343903541565, 'learning_rate': 3.970854035874112e-05, 'epoch': 0.83}

  8%|▊         | 3415/41250 [8:14:29<91:57:44,  8.75s/it][2025-04-26 12:23:07,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 12:23:07,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.95 | bwd_microstep: 5892.94 | bwd_inner_microstep: 5643.95 | bwd_allreduce_microstep: 248.94 | step_microstep: 18.77
[2025-04-26 12:23:07,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.95 | bwd: 5892.95 | bwd_inner: 5643.95 | bwd_allreduce: 248.96 | step: 18.77

  8%|▊         | 3416/41250 [8:14:38<92:07:35,  8.77s/it]
                                                         
{'loss': 0.344, 'grad_norm': 1.3717519044876099, 'learning_rate': 3.9708273187598235e-05, 'epoch': 0.83}

  8%|▊         | 3416/41250 [8:14:38<92:07:35,  8.77s/it][2025-04-26 12:23:15,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:23:15,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.58 | bwd_microstep: 5745.73 | bwd_inner_microstep: 5648.88 | bwd_allreduce_microstep: 96.80 | step_microstep: 18.26
[2025-04-26 12:23:15,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.58 | bwd: 5745.74 | bwd_inner: 5648.88 | bwd_allreduce: 96.82 | step: 18.26

  8%|▊         | 3417/41250 [8:14:46<91:48:08,  8.74s/it]
                                                         
{'loss': 0.2313, 'grad_norm': 1.8663533926010132, 'learning_rate': 3.9708005894957726e-05, 'epoch': 0.83}

  8%|▊         | 3417/41250 [8:14:46<91:48:08,  8.74s/it][2025-04-26 12:23:24,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 12:23:24,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.41 | bwd_microstep: 5689.39 | bwd_inner_microstep: 5635.49 | bwd_allreduce_microstep: 53.84 | step_microstep: 18.66
[2025-04-26 12:23:24,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.41 | bwd: 5689.41 | bwd_inner: 5635.49 | bwd_allreduce: 53.87 | step: 18.67

  8%|▊         | 3418/41250 [8:14:55<91:22:15,  8.69s/it]
                                                         
{'loss': 0.078, 'grad_norm': 1.0098018646240234, 'learning_rate': 3.970773848082124e-05, 'epoch': 0.83}

  8%|▊         | 3418/41250 [8:14:55<91:22:15,  8.69s/it][2025-04-26 12:23:32,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:23:32,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.74 | bwd_microstep: 5725.26 | bwd_inner_microstep: 5658.51 | bwd_allreduce_microstep: 66.70 | step_microstep: 18.83
[2025-04-26 12:23:32,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.74 | bwd: 5725.27 | bwd_inner: 5658.51 | bwd_allreduce: 66.72 | step: 18.83

  8%|▊         | 3419/41250 [8:15:04<91:10:30,  8.68s/it]
                                                         
{'loss': 0.1869, 'grad_norm': 1.1556540727615356, 'learning_rate': 3.970747094519041e-05, 'epoch': 0.83}

  8%|▊         | 3419/41250 [8:15:04<91:10:30,  8.68s/it][2025-04-26 12:23:41,518] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:23:41,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.32 | bwd_microstep: 5701.82 | bwd_inner_microstep: 5660.94 | bwd_allreduce_microstep: 40.83 | step_microstep: 18.90
[2025-04-26 12:23:41,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.32 | bwd: 5701.83 | bwd_inner: 5660.94 | bwd_allreduce: 40.85 | step: 18.90

  8%|▊         | 3420/41250 [8:15:12<90:58:24,  8.66s/it]
                                                         
{'loss': 0.2075, 'grad_norm': 1.5352296829223633, 'learning_rate': 3.970720328806691e-05, 'epoch': 0.83}

  8%|▊         | 3420/41250 [8:15:12<90:58:24,  8.66s/it][2025-04-26 12:23:50,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 12:23:50,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.87 | bwd_microstep: 5702.26 | bwd_inner_microstep: 5651.30 | bwd_allreduce_microstep: 50.92 | step_microstep: 18.75
[2025-04-26 12:23:50,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.87 | bwd: 5702.28 | bwd_inner: 5651.30 | bwd_allreduce: 50.94 | step: 18.75

  8%|▊         | 3421/41250 [8:15:21<90:48:59,  8.64s/it]
                                                         
{'loss': 0.0915, 'grad_norm': 1.4023932218551636, 'learning_rate': 3.970693550945238e-05, 'epoch': 0.83}

  8%|▊         | 3421/41250 [8:15:21<90:48:59,  8.64s/it][2025-04-26 12:23:58,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:23:58,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.08 | bwd_microstep: 5770.88 | bwd_inner_microstep: 5758.00 | bwd_allreduce_microstep: 12.84 | step_microstep: 19.28
[2025-04-26 12:23:58,848] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.08 | bwd: 5770.90 | bwd_inner: 5758.00 | bwd_allreduce: 12.86 | step: 19.28

  8%|▊         | 3422/41250 [8:15:30<91:04:07,  8.67s/it]
                                                         
{'loss': 0.4868, 'grad_norm': 3.5780858993530273, 'learning_rate': 3.970666760934846e-05, 'epoch': 0.83}

  8%|▊         | 3422/41250 [8:15:30<91:04:07,  8.67s/it][2025-04-26 12:24:07,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:24:07,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.76 | bwd_microstep: 5711.35 | bwd_inner_microstep: 5662.45 | bwd_allreduce_microstep: 48.85 | step_microstep: 18.84
[2025-04-26 12:24:07,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.76 | bwd: 5711.36 | bwd_inner: 5662.45 | bwd_allreduce: 48.87 | step: 18.85

  8%|▊         | 3423/41250 [8:15:38<90:55:15,  8.65s/it]
                                                         
{'loss': 0.2194, 'grad_norm': 2.211178779602051, 'learning_rate': 3.970639958775682e-05, 'epoch': 0.83}

  8%|▊         | 3423/41250 [8:15:38<90:55:15,  8.65s/it][2025-04-26 12:24:16,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:24:16,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.13 | bwd_microstep: 5747.39 | bwd_inner_microstep: 5650.46 | bwd_allreduce_microstep: 96.88 | step_microstep: 18.23
[2025-04-26 12:24:16,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.13 | bwd: 5747.41 | bwd_inner: 5650.46 | bwd_allreduce: 96.90 | step: 18.23

  8%|▊         | 3424/41250 [8:15:47<90:56:16,  8.65s/it]
                                                         
{'loss': 0.0893, 'grad_norm': 0.862045168876648, 'learning_rate': 3.970613144467909e-05, 'epoch': 0.83}

  8%|▊         | 3424/41250 [8:15:47<90:56:16,  8.65s/it][2025-04-26 12:24:24,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.05 | optimizer_step: 1.18
[2025-04-26 12:24:24,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.29 | bwd_microstep: 5763.46 | bwd_inner_microstep: 5644.80 | bwd_allreduce_microstep: 118.62 | step_microstep: 19.37
[2025-04-26 12:24:24,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.29 | bwd: 5763.48 | bwd_inner: 5644.80 | bwd_allreduce: 118.64 | step: 19.37

  8%|▊         | 3425/41250 [8:15:55<91:00:02,  8.66s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.0809073448181152, 'learning_rate': 3.970586318011694e-05, 'epoch': 0.83}

  8%|▊         | 3425/41250 [8:15:55<91:00:02,  8.66s/it][2025-04-26 12:24:33,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.02
[2025-04-26 12:24:33,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.75 | bwd_microstep: 5768.95 | bwd_inner_microstep: 5663.96 | bwd_allreduce_microstep: 104.95 | step_microstep: 18.53
[2025-04-26 12:24:33,494] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.75 | bwd: 5768.97 | bwd_inner: 5663.96 | bwd_allreduce: 104.97 | step: 18.53

  8%|▊         | 3426/41250 [8:16:04<91:04:47,  8.67s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 0.8510642051696777, 'learning_rate': 3.970559479407203e-05, 'epoch': 0.83}

  8%|▊         | 3426/41250 [8:16:04<91:04:47,  8.67s/it][2025-04-26 12:24:42,193] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:24:42,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.08 | bwd_microstep: 5773.69 | bwd_inner_microstep: 5694.62 | bwd_allreduce_microstep: 79.03 | step_microstep: 18.60
[2025-04-26 12:24:42,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.08 | bwd: 5773.71 | bwd_inner: 5694.62 | bwd_allreduce: 79.05 | step: 18.61

  8%|▊         | 3427/41250 [8:16:13<91:10:34,  8.68s/it]
                                                         
{'loss': 0.5138, 'grad_norm': 3.097623586654663, 'learning_rate': 3.970532628654599e-05, 'epoch': 0.83}

  8%|▊         | 3427/41250 [8:16:13<91:10:34,  8.68s/it][2025-04-26 12:24:50,839] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:24:50,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.28 | bwd_microstep: 5707.85 | bwd_inner_microstep: 5695.10 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.72
[2025-04-26 12:24:50,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.28 | bwd: 5707.86 | bwd_inner: 5695.10 | bwd_allreduce: 12.73 | step: 18.73

  8%|▊         | 3428/41250 [8:16:22<91:04:21,  8.67s/it]
                                                         
{'loss': 0.135, 'grad_norm': 0.8657055497169495, 'learning_rate': 3.97050576575405e-05, 'epoch': 0.83}

  8%|▊         | 3428/41250 [8:16:22<91:04:21,  8.67s/it][2025-04-26 12:24:59,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 12:24:59,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.73 | bwd_microstep: 5728.20 | bwd_inner_microstep: 5715.54 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.48
[2025-04-26 12:24:59,499] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.73 | bwd: 5728.21 | bwd_inner: 5715.54 | bwd_allreduce: 12.63 | step: 18.49

  8%|▊         | 3429/41250 [8:16:30<91:02:29,  8.67s/it]
                                                         
{'loss': 0.1223, 'grad_norm': 4.055278301239014, 'learning_rate': 3.970478890705719e-05, 'epoch': 0.83}

  8%|▊         | 3429/41250 [8:16:30<91:02:29,  8.67s/it][2025-04-26 12:25:08,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-26 12:25:08,180] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.31 | bwd_microstep: 5768.09 | bwd_inner_microstep: 5665.34 | bwd_allreduce_microstep: 102.70 | step_microstep: 19.09
[2025-04-26 12:25:08,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.31 | bwd: 5768.11 | bwd_inner: 5665.34 | bwd_allreduce: 102.72 | step: 19.09

  8%|▊         | 3430/41250 [8:16:39<91:05:13,  8.67s/it]
                                                         
{'loss': 0.1579, 'grad_norm': 1.3302381038665771, 'learning_rate': 3.9704520035097746e-05, 'epoch': 0.83}

  8%|▊         | 3430/41250 [8:16:39<91:05:13,  8.67s/it][2025-04-26 12:25:16,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.98 | optimizer_step: 1.08
[2025-04-26 12:25:16,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.90 | bwd_microstep: 5703.20 | bwd_inner_microstep: 5683.93 | bwd_allreduce_microstep: 19.22 | step_microstep: 18.49
[2025-04-26 12:25:16,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.90 | bwd: 5703.21 | bwd_inner: 5683.93 | bwd_allreduce: 19.24 | step: 18.50

  8%|▊         | 3431/41250 [8:16:47<90:55:52,  8.66s/it]
                                                         
{'loss': 0.3084, 'grad_norm': 1.865166425704956, 'learning_rate': 3.970425104166381e-05, 'epoch': 0.83}

  8%|▊         | 3431/41250 [8:16:47<90:55:52,  8.66s/it][2025-04-26 12:25:25,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:25:25,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5772.66 | bwd_inner_microstep: 5712.46 | bwd_allreduce_microstep: 60.15 | step_microstep: 18.63
[2025-04-26 12:25:25,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5772.67 | bwd_inner: 5712.46 | bwd_allreduce: 60.17 | step: 18.63

  8%|▊         | 3432/41250 [8:16:56<91:05:40,  8.67s/it]
                                                         
{'loss': 0.0739, 'grad_norm': 1.2604058980941772, 'learning_rate': 3.9703981926757036e-05, 'epoch': 0.83}

  8%|▊         | 3432/41250 [8:16:56<91:05:40,  8.67s/it][2025-04-26 12:25:34,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:25:34,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.11 | bwd_microstep: 5724.44 | bwd_inner_microstep: 5711.64 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.63
[2025-04-26 12:25:34,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.11 | bwd: 5724.45 | bwd_inner: 5711.64 | bwd_allreduce: 12.77 | step: 18.63

  8%|▊         | 3433/41250 [8:17:05<91:03:29,  8.67s/it]
                                                         
{'loss': 0.1852, 'grad_norm': 1.29091477394104, 'learning_rate': 3.97037126903791e-05, 'epoch': 0.83}

  8%|▊         | 3433/41250 [8:17:05<91:03:29,  8.67s/it][2025-04-26 12:25:42,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:25:42,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.66 | bwd_microstep: 5701.56 | bwd_inner_microstep: 5664.15 | bwd_allreduce_microstep: 37.36 | step_microstep: 18.76
[2025-04-26 12:25:42,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.66 | bwd: 5701.58 | bwd_inner: 5664.15 | bwd_allreduce: 37.38 | step: 18.76

  8%|▊         | 3434/41250 [8:17:13<90:53:32,  8.65s/it]
                                                         
{'loss': 0.0847, 'grad_norm': 0.7871275544166565, 'learning_rate': 3.970344333253164e-05, 'epoch': 0.83}

  8%|▊         | 3434/41250 [8:17:13<90:53:32,  8.65s/it][2025-04-26 12:25:51,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:25:51,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.64 | bwd_microstep: 5941.34 | bwd_inner_microstep: 5669.10 | bwd_allreduce_microstep: 272.18 | step_microstep: 18.75
[2025-04-26 12:25:51,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.64 | bwd: 5941.35 | bwd_inner: 5669.10 | bwd_allreduce: 272.21 | step: 18.75

  8%|▊         | 3435/41250 [8:17:22<91:31:35,  8.71s/it]
                                                         
{'loss': 0.095, 'grad_norm': 1.1979397535324097, 'learning_rate': 3.970317385321633e-05, 'epoch': 0.83}

  8%|▊         | 3435/41250 [8:17:22<91:31:35,  8.71s/it][2025-04-26 12:26:00,355] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.92 | optimizer_gradients: 0.99 | optimizer_step: 0.93
[2025-04-26 12:26:00,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.87 | bwd_microstep: 5784.20 | bwd_inner_microstep: 5720.66 | bwd_allreduce_microstep: 63.49 | step_microstep: 17.95
[2025-04-26 12:26:00,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.87 | bwd: 5784.21 | bwd_inner: 5720.66 | bwd_allreduce: 63.51 | step: 17.95

  8%|▊         | 3436/41250 [8:17:31<91:31:15,  8.71s/it]
                                                         
{'loss': 0.1867, 'grad_norm': 1.3337154388427734, 'learning_rate': 3.9702904252434826e-05, 'epoch': 0.83}

  8%|▊         | 3436/41250 [8:17:31<91:31:15,  8.71s/it][2025-04-26 12:26:09,073] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:26:09,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.09 | bwd_microstep: 5792.19 | bwd_inner_microstep: 5658.67 | bwd_allreduce_microstep: 133.49 | step_microstep: 18.04
[2025-04-26 12:26:09,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.09 | bwd: 5792.20 | bwd_inner: 5658.67 | bwd_allreduce: 133.50 | step: 18.04

  8%|▊         | 3437/41250 [8:17:40<91:32:06,  8.71s/it]
                                                         
{'loss': 0.1348, 'grad_norm': 1.42471182346344, 'learning_rate': 3.9702634530188796e-05, 'epoch': 0.83}

  8%|▊         | 3437/41250 [8:17:40<91:32:06,  8.71s/it][2025-04-26 12:26:17,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:26:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.63 | bwd_microstep: 5805.77 | bwd_inner_microstep: 5654.04 | bwd_allreduce_microstep: 151.69 | step_microstep: 17.67
[2025-04-26 12:26:17,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.63 | bwd: 5805.78 | bwd_inner: 5654.04 | bwd_allreduce: 151.70 | step: 17.68

  8%|▊         | 3438/41250 [8:17:48<91:33:25,  8.72s/it]
                                                         
{'loss': 0.0581, 'grad_norm': 0.5613118410110474, 'learning_rate': 3.97023646864799e-05, 'epoch': 0.83}

  8%|▊         | 3438/41250 [8:17:48<91:33:25,  8.72s/it][2025-04-26 12:26:26,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:26:26,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.08 | bwd_microstep: 5772.13 | bwd_inner_microstep: 5706.23 | bwd_allreduce_microstep: 65.86 | step_microstep: 17.70
[2025-04-26 12:26:26,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.08 | bwd: 5772.14 | bwd_inner: 5706.23 | bwd_allreduce: 65.88 | step: 17.71

  8%|▊         | 3439/41250 [8:17:57<91:32:17,  8.72s/it]
                                                         
{'loss': 0.2395, 'grad_norm': 2.223970890045166, 'learning_rate': 3.97020947213098e-05, 'epoch': 0.83}

  8%|▊         | 3439/41250 [8:17:57<91:32:17,  8.72s/it][2025-04-26 12:26:35,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:26:35,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.31 | bwd_microstep: 5758.87 | bwd_inner_microstep: 5704.85 | bwd_allreduce_microstep: 53.98 | step_microstep: 17.66
[2025-04-26 12:26:35,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.31 | bwd: 5758.88 | bwd_inner: 5704.85 | bwd_allreduce: 53.99 | step: 17.66

  8%|▊         | 3440/41250 [8:18:06<91:26:40,  8.71s/it]
                                                         
{'loss': 0.334, 'grad_norm': 2.9559526443481445, 'learning_rate': 3.970182463468017e-05, 'epoch': 0.83}

  8%|▊         | 3440/41250 [8:18:06<91:26:40,  8.71s/it][2025-04-26 12:26:43,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:26:43,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.34 | bwd_microstep: 5788.15 | bwd_inner_microstep: 5655.06 | bwd_allreduce_microstep: 133.05 | step_microstep: 17.68
[2025-04-26 12:26:43,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.34 | bwd: 5788.16 | bwd_inner: 5655.06 | bwd_allreduce: 133.06 | step: 17.69

  8%|▊         | 3441/41250 [8:18:15<91:26:04,  8.71s/it]
                                                         
{'loss': 0.2541, 'grad_norm': 2.8044326305389404, 'learning_rate': 3.9701554426592654e-05, 'epoch': 0.83}

  8%|▊         | 3441/41250 [8:18:15<91:26:04,  8.71s/it][2025-04-26 12:26:52,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:26:52,619] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.89 | bwd_microstep: 5805.76 | bwd_inner_microstep: 5674.33 | bwd_allreduce_microstep: 131.39 | step_microstep: 18.75
[2025-04-26 12:26:52,620] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.89 | bwd: 5805.78 | bwd_inner: 5674.33 | bwd_allreduce: 131.41 | step: 18.75

  8%|▊         | 3442/41250 [8:18:23<91:28:43,  8.71s/it]
                                                         
{'loss': 0.28, 'grad_norm': 1.531684160232544, 'learning_rate': 3.970128409704893e-05, 'epoch': 0.83}

  8%|▊         | 3442/41250 [8:18:23<91:28:43,  8.71s/it][2025-04-26 12:27:01,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:27:01,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.02 | bwd_microstep: 5805.33 | bwd_inner_microstep: 5643.07 | bwd_allreduce_microstep: 162.22 | step_microstep: 18.20
[2025-04-26 12:27:01,335] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.02 | bwd: 5805.34 | bwd_inner: 5643.07 | bwd_allreduce: 162.23 | step: 18.20

  8%|▊         | 3443/41250 [8:18:32<91:29:25,  8.71s/it]
                                                         
{'loss': 0.3414, 'grad_norm': 2.6357452869415283, 'learning_rate': 3.970101364605067e-05, 'epoch': 0.83}

  8%|▊         | 3443/41250 [8:18:32<91:29:25,  8.71s/it][2025-04-26 12:27:09,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:27:09,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.00 | bwd_microstep: 5707.55 | bwd_inner_microstep: 5660.04 | bwd_allreduce_microstep: 47.47 | step_microstep: 18.51
[2025-04-26 12:27:09,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.00 | bwd: 5707.57 | bwd_inner: 5660.04 | bwd_allreduce: 47.48 | step: 18.51

  8%|▊         | 3444/41250 [8:18:41<91:13:04,  8.69s/it]
                                                         
{'loss': 0.2304, 'grad_norm': 2.96748685836792, 'learning_rate': 3.9700743073599535e-05, 'epoch': 0.83}

  8%|▊         | 3444/41250 [8:18:41<91:13:04,  8.69s/it][2025-04-26 12:27:18,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 12:27:18,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.04 | bwd_microstep: 5776.44 | bwd_inner_microstep: 5697.30 | bwd_allreduce_microstep: 79.08 | step_microstep: 18.60
[2025-04-26 12:27:18,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.04 | bwd: 5776.45 | bwd_inner: 5697.30 | bwd_allreduce: 79.11 | step: 18.61

  8%|▊         | 3445/41250 [8:18:49<91:16:09,  8.69s/it]
                                                         
{'loss': 0.1346, 'grad_norm': 2.263948678970337, 'learning_rate': 3.9700472379697197e-05, 'epoch': 0.84}

  8%|▊         | 3445/41250 [8:18:49<91:16:09,  8.69s/it][2025-04-26 12:27:27,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:27:27,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.41 | bwd_microstep: 5705.89 | bwd_inner_microstep: 5691.40 | bwd_allreduce_microstep: 14.44 | step_microstep: 18.46
[2025-04-26 12:27:27,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.41 | bwd: 5705.90 | bwd_inner: 5691.40 | bwd_allreduce: 14.46 | step: 18.47

  8%|▊         | 3446/41250 [8:18:58<91:06:29,  8.68s/it]
                                                         
{'loss': 0.3087, 'grad_norm': 1.6646935939788818, 'learning_rate': 3.9700201564345316e-05, 'epoch': 0.84}

  8%|▊         | 3446/41250 [8:18:58<91:06:29,  8.68s/it][2025-04-26 12:27:35,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 12:27:35,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.47 | bwd_microstep: 5740.02 | bwd_inner_microstep: 5696.31 | bwd_allreduce_microstep: 43.67 | step_microstep: 18.54
[2025-04-26 12:27:35,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.47 | bwd: 5740.03 | bwd_inner: 5696.31 | bwd_allreduce: 43.69 | step: 18.55

  8%|▊         | 3447/41250 [8:19:07<91:06:09,  8.68s/it]
                                                         
{'loss': 0.148, 'grad_norm': 1.1985771656036377, 'learning_rate': 3.969993062754557e-05, 'epoch': 0.84}

  8%|▊         | 3447/41250 [8:19:07<91:06:09,  8.68s/it][2025-04-26 12:27:44,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:27:44,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.20 | bwd_microstep: 5761.05 | bwd_inner_microstep: 5710.12 | bwd_allreduce_microstep: 50.88 | step_microstep: 18.59
[2025-04-26 12:27:44,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.20 | bwd: 5761.06 | bwd_inner: 5710.12 | bwd_allreduce: 50.90 | step: 18.60

  8%|▊         | 3448/41250 [8:19:15<91:10:45,  8.68s/it]
                                                         
{'loss': 0.094, 'grad_norm': 0.7942913174629211, 'learning_rate': 3.969965956929962e-05, 'epoch': 0.84}

  8%|▊         | 3448/41250 [8:19:15<91:10:45,  8.68s/it][2025-04-26 12:27:53,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:27:53,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.67 | bwd_microstep: 5753.78 | bwd_inner_microstep: 5695.41 | bwd_allreduce_microstep: 58.33 | step_microstep: 18.66
[2025-04-26 12:27:53,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.67 | bwd: 5753.80 | bwd_inner: 5695.41 | bwd_allreduce: 58.35 | step: 18.67

  8%|▊         | 3449/41250 [8:19:24<91:09:26,  8.68s/it]
                                                         
{'loss': 0.5617, 'grad_norm': 2.1496188640594482, 'learning_rate': 3.969938838960915e-05, 'epoch': 0.84}

  8%|▊         | 3449/41250 [8:19:24<91:09:26,  8.68s/it][2025-04-26 12:28:02,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.96 | optimizer_step: 0.99
[2025-04-26 12:28:02,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.36 | bwd_microstep: 5767.53 | bwd_inner_microstep: 5689.78 | bwd_allreduce_microstep: 77.70 | step_microstep: 18.42
[2025-04-26 12:28:02,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.36 | bwd: 5767.54 | bwd_inner: 5689.78 | bwd_allreduce: 77.72 | step: 18.42

  8%|▊         | 3450/41250 [8:19:33<91:12:17,  8.69s/it]
                                                         
{'loss': 0.4035, 'grad_norm': 2.98067569732666, 'learning_rate': 3.969911708847583e-05, 'epoch': 0.84}

  8%|▊         | 3450/41250 [8:19:33<91:12:17,  8.69s/it][2025-04-26 12:28:10,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 1.27 | optimizer_step: 1.01
[2025-04-26 12:28:10,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.38 | bwd_microstep: 5689.33 | bwd_inner_microstep: 5645.55 | bwd_allreduce_microstep: 43.71 | step_microstep: 19.22
[2025-04-26 12:28:10,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.38 | bwd: 5689.34 | bwd_inner: 5645.55 | bwd_allreduce: 43.74 | step: 19.22

  8%|▊         | 3451/41250 [8:19:41<90:57:13,  8.66s/it]
                                                         
{'loss': 0.0753, 'grad_norm': 0.8004536628723145, 'learning_rate': 3.969884566590132e-05, 'epoch': 0.84}

  8%|▊         | 3451/41250 [8:19:41<90:57:13,  8.66s/it][2025-04-26 12:28:19,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:28:19,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5749.90 | bwd_inner_microstep: 5695.68 | bwd_allreduce_microstep: 54.17 | step_microstep: 18.05
[2025-04-26 12:28:19,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5749.91 | bwd_inner: 5695.68 | bwd_allreduce: 54.19 | step: 18.06

  8%|▊         | 3452/41250 [8:19:50<91:00:18,  8.67s/it]
                                                         
{'loss': 0.103, 'grad_norm': 1.6426340341567993, 'learning_rate': 3.9698574121887304e-05, 'epoch': 0.84}

  8%|▊         | 3452/41250 [8:19:50<91:00:18,  8.67s/it][2025-04-26 12:28:27,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:28:27,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.60 | bwd_microstep: 5707.83 | bwd_inner_microstep: 5695.17 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.07
[2025-04-26 12:28:27,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.60 | bwd: 5707.84 | bwd_inner: 5695.17 | bwd_allreduce: 12.63 | step: 18.07

  8%|▊         | 3453/41250 [8:19:59<90:56:09,  8.66s/it]
                                                         
{'loss': 0.1881, 'grad_norm': 2.16548490524292, 'learning_rate': 3.969830245643545e-05, 'epoch': 0.84}

  8%|▊         | 3453/41250 [8:19:59<90:56:09,  8.66s/it][2025-04-26 12:28:36,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.09
[2025-04-26 12:28:36,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5777.92 | bwd_inner_microstep: 5699.28 | bwd_allreduce_microstep: 78.60 | step_microstep: 18.67
[2025-04-26 12:28:36,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.26 | bwd: 5777.93 | bwd_inner: 5699.28 | bwd_allreduce: 78.62 | step: 18.67

  8%|▊         | 3454/41250 [8:20:07<91:04:28,  8.67s/it]
                                                         
{'loss': 0.1358, 'grad_norm': 1.8026214838027954, 'learning_rate': 3.969803066954744e-05, 'epoch': 0.84}

  8%|▊         | 3454/41250 [8:20:07<91:04:28,  8.67s/it][2025-04-26 12:28:45,395] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.24 | optimizer_step: 0.95
[2025-04-26 12:28:45,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.14 | bwd_microstep: 5794.58 | bwd_inner_microstep: 5646.08 | bwd_allreduce_microstep: 148.45 | step_microstep: 19.13
[2025-04-26 12:28:45,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.14 | bwd: 5794.60 | bwd_inner: 5646.08 | bwd_allreduce: 148.47 | step: 19.13

  8%|▊         | 3455/41250 [8:20:16<91:09:38,  8.68s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 1.215600609779358, 'learning_rate': 3.969775876122494e-05, 'epoch': 0.84}

  8%|▊         | 3455/41250 [8:20:16<91:09:38,  8.68s/it][2025-04-26 12:28:54,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:28:54,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.31 | bwd_microstep: 5781.65 | bwd_inner_microstep: 5656.54 | bwd_allreduce_microstep: 125.06 | step_microstep: 18.24
[2025-04-26 12:28:54,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.31 | bwd: 5781.66 | bwd_inner: 5656.54 | bwd_allreduce: 125.08 | step: 18.25

  8%|▊         | 3456/41250 [8:20:25<91:11:22,  8.69s/it]
                                                         
{'loss': 0.1419, 'grad_norm': 1.8562575578689575, 'learning_rate': 3.969748673146964e-05, 'epoch': 0.84}

  8%|▊         | 3456/41250 [8:20:25<91:11:22,  8.69s/it][2025-04-26 12:29:02,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:29:02,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.18 | bwd_microstep: 5712.51 | bwd_inner_microstep: 5699.66 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.47
[2025-04-26 12:29:02,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.18 | bwd: 5712.52 | bwd_inner: 5699.66 | bwd_allreduce: 12.83 | step: 18.47

  8%|▊         | 3457/41250 [8:20:33<91:03:02,  8.67s/it]
                                                         
{'loss': 0.183, 'grad_norm': 1.109954595565796, 'learning_rate': 3.9697214580283205e-05, 'epoch': 0.84}

  8%|▊         | 3457/41250 [8:20:33<91:03:02,  8.67s/it][2025-04-26 12:29:11,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:29:11,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.52 | bwd_microstep: 5711.97 | bwd_inner_microstep: 5652.41 | bwd_allreduce_microstep: 59.52 | step_microstep: 17.67
[2025-04-26 12:29:11,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.52 | bwd: 5711.98 | bwd_inner: 5652.41 | bwd_allreduce: 59.54 | step: 17.68

  8%|▊         | 3458/41250 [8:20:42<90:51:18,  8.65s/it]
                                                         
{'loss': 0.2126, 'grad_norm': 1.3224555253982544, 'learning_rate': 3.969694230766731e-05, 'epoch': 0.84}

  8%|▊         | 3458/41250 [8:20:42<90:51:18,  8.65s/it][2025-04-26 12:29:20,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:29:20,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.58 | bwd_microstep: 5735.75 | bwd_inner_microstep: 5683.94 | bwd_allreduce_microstep: 51.77 | step_microstep: 18.16
[2025-04-26 12:29:20,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.58 | bwd: 5735.76 | bwd_inner: 5683.94 | bwd_allreduce: 51.78 | step: 18.17

  8%|▊         | 3459/41250 [8:20:51<90:54:57,  8.66s/it]
                                                         
{'loss': 0.0774, 'grad_norm': 0.6270085573196411, 'learning_rate': 3.9696669913623646e-05, 'epoch': 0.84}

  8%|▊         | 3459/41250 [8:20:51<90:54:57,  8.66s/it][2025-04-26 12:29:28,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.03 | optimizer_step: 0.97
[2025-04-26 12:29:28,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5709.81 | bwd_inner_microstep: 5696.71 | bwd_allreduce_microstep: 13.05 | step_microstep: 19.08
[2025-04-26 12:29:28,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.76 | bwd: 5709.83 | bwd_inner: 5696.71 | bwd_allreduce: 13.07 | step: 19.08

  8%|▊         | 3460/41250 [8:20:59<90:49:26,  8.65s/it]
                                                         
{'loss': 0.1634, 'grad_norm': 1.2623649835586548, 'learning_rate': 3.969639739815389e-05, 'epoch': 0.84}

  8%|▊         | 3460/41250 [8:20:59<90:49:26,  8.65s/it][2025-04-26 12:29:37,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:29:37,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.41 | bwd_microstep: 5758.68 | bwd_inner_microstep: 5692.24 | bwd_allreduce_microstep: 66.40 | step_microstep: 18.34
[2025-04-26 12:29:37,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.41 | bwd: 5758.69 | bwd_inner: 5692.24 | bwd_allreduce: 66.41 | step: 18.34

  8%|▊         | 3461/41250 [8:21:08<90:54:44,  8.66s/it]
                                                         
{'loss': 0.1353, 'grad_norm': 1.0913478136062622, 'learning_rate': 3.969612476125971e-05, 'epoch': 0.84}

  8%|▊         | 3461/41250 [8:21:08<90:54:44,  8.66s/it][2025-04-26 12:29:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:29:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.77 | bwd_microstep: 5699.18 | bwd_inner_microstep: 5686.40 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.63
[2025-04-26 12:29:45,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.77 | bwd: 5699.20 | bwd_inner: 5686.40 | bwd_allreduce: 12.76 | step: 18.64

  8%|▊         | 3462/41250 [8:21:17<90:48:11,  8.65s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 1.4923303127288818, 'learning_rate': 3.96958520029428e-05, 'epoch': 0.84}

  8%|▊         | 3462/41250 [8:21:17<90:48:11,  8.65s/it][2025-04-26 12:29:54,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:29:54,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.90 | bwd_microstep: 5720.20 | bwd_inner_microstep: 5698.19 | bwd_allreduce_microstep: 21.96 | step_microstep: 18.29
[2025-04-26 12:29:54,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.90 | bwd: 5720.21 | bwd_inner: 5698.19 | bwd_allreduce: 21.98 | step: 18.29

  8%|▊         | 3463/41250 [8:21:25<90:48:47,  8.65s/it]
                                                         
{'loss': 0.2693, 'grad_norm': 1.3998783826828003, 'learning_rate': 3.969557912320483e-05, 'epoch': 0.84}

  8%|▊         | 3463/41250 [8:21:25<90:48:47,  8.65s/it][2025-04-26 12:30:03,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 12:30:03,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.94 | bwd_microstep: 5723.73 | bwd_inner_microstep: 5710.86 | bwd_allreduce_microstep: 12.83 | step_microstep: 19.04
[2025-04-26 12:30:03,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.94 | bwd: 5723.75 | bwd_inner: 5710.86 | bwd_allreduce: 12.85 | step: 19.04

  8%|▊         | 3464/41250 [8:21:34<90:51:49,  8.66s/it]
                                                         
{'loss': 0.2198, 'grad_norm': 1.6266205310821533, 'learning_rate': 3.969530612204749e-05, 'epoch': 0.84}

  8%|▊         | 3464/41250 [8:21:34<90:51:49,  8.66s/it][2025-04-26 12:30:11,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:30:11,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.31 | bwd_microstep: 5718.85 | bwd_inner_microstep: 5705.88 | bwd_allreduce_microstep: 12.92 | step_microstep: 18.81
[2025-04-26 12:30:11,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.31 | bwd: 5718.86 | bwd_inner: 5705.88 | bwd_allreduce: 12.93 | step: 18.81

  8%|▊         | 3465/41250 [8:21:43<90:51:00,  8.66s/it]
                                                         
{'loss': 0.1636, 'grad_norm': 1.5145385265350342, 'learning_rate': 3.969503299947246e-05, 'epoch': 0.84}

  8%|▊         | 3465/41250 [8:21:43<90:51:00,  8.66s/it][2025-04-26 12:30:20,615] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:30:20,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.07 | bwd_microstep: 5752.26 | bwd_inner_microstep: 5696.82 | bwd_allreduce_microstep: 55.39 | step_microstep: 19.14
[2025-04-26 12:30:20,616] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.07 | bwd: 5752.27 | bwd_inner: 5696.82 | bwd_allreduce: 55.41 | step: 19.14

  8%|▊         | 3466/41250 [8:21:51<90:55:38,  8.66s/it]
                                                         
{'loss': 0.1791, 'grad_norm': 0.9346274733543396, 'learning_rate': 3.969475975548144e-05, 'epoch': 0.84}

  8%|▊         | 3466/41250 [8:21:51<90:55:38,  8.66s/it][2025-04-26 12:30:29,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:30:29,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5741.96 | bwd_inner_microstep: 5705.61 | bwd_allreduce_microstep: 36.30 | step_microstep: 18.86
[2025-04-26 12:30:29,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5741.97 | bwd_inner: 5705.61 | bwd_allreduce: 36.32 | step: 18.86

  8%|▊         | 3467/41250 [8:22:00<90:56:04,  8.66s/it]
                                                         
{'loss': 0.1205, 'grad_norm': 1.0351123809814453, 'learning_rate': 3.9694486390076086e-05, 'epoch': 0.84}

  8%|▊         | 3467/41250 [8:22:00<90:56:04,  8.66s/it][2025-04-26 12:30:37,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.11 | optimizer_step: 1.21
[2025-04-26 12:30:37,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.77 | bwd_microstep: 5768.94 | bwd_inner_microstep: 5645.88 | bwd_allreduce_microstep: 123.00 | step_microstep: 19.69
[2025-04-26 12:30:37,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.77 | bwd: 5768.96 | bwd_inner: 5645.88 | bwd_allreduce: 123.03 | step: 19.69

  8%|▊         | 3468/41250 [8:22:09<90:57:36,  8.67s/it]
                                                         
{'loss': 0.0685, 'grad_norm': 0.7973301410675049, 'learning_rate': 3.96942129032581e-05, 'epoch': 0.84}

  8%|▊         | 3468/41250 [8:22:09<90:57:36,  8.67s/it][2025-04-26 12:30:46,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:30:46,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.55 | bwd_microstep: 5761.29 | bwd_inner_microstep: 5693.10 | bwd_allreduce_microstep: 68.14 | step_microstep: 17.93
[2025-04-26 12:30:46,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.55 | bwd: 5761.30 | bwd_inner: 5693.10 | bwd_allreduce: 68.16 | step: 17.94

  8%|▊         | 3469/41250 [8:22:17<91:01:43,  8.67s/it]
                                                         
{'loss': 0.1934, 'grad_norm': 1.3958967924118042, 'learning_rate': 3.969393929502917e-05, 'epoch': 0.84}

  8%|▊         | 3469/41250 [8:22:17<91:01:43,  8.67s/it][2025-04-26 12:30:55,344] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:30:55,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.87 | bwd_microstep: 5781.66 | bwd_inner_microstep: 5662.25 | bwd_allreduce_microstep: 119.37 | step_microstep: 18.52
[2025-04-26 12:30:55,345] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.87 | bwd: 5781.68 | bwd_inner: 5662.25 | bwd_allreduce: 119.39 | step: 18.52

  8%|▊         | 3470/41250 [8:22:26<91:06:20,  8.68s/it]
                                                         
{'loss': 0.1846, 'grad_norm': 1.932759165763855, 'learning_rate': 3.969366556539097e-05, 'epoch': 0.84}

  8%|▊         | 3470/41250 [8:22:26<91:06:20,  8.68s/it][2025-04-26 12:31:03,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 12:31:03,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.76 | bwd_microstep: 5705.54 | bwd_inner_microstep: 5657.41 | bwd_allreduce_microstep: 48.07 | step_microstep: 18.73
[2025-04-26 12:31:03,962] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.76 | bwd: 5705.55 | bwd_inner: 5657.41 | bwd_allreduce: 48.10 | step: 18.74

  8%|▊         | 3471/41250 [8:22:35<90:54:12,  8.66s/it]
                                                         
{'loss': 0.2034, 'grad_norm': 1.2867385149002075, 'learning_rate': 3.96933917143452e-05, 'epoch': 0.84}

  8%|▊         | 3471/41250 [8:22:35<90:54:12,  8.66s/it][2025-04-26 12:31:12,610] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-26 12:31:12,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.23 | bwd_microstep: 5721.89 | bwd_inner_microstep: 5708.57 | bwd_allreduce_microstep: 13.27 | step_microstep: 19.08
[2025-04-26 12:31:12,611] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.23 | bwd: 5721.90 | bwd_inner: 5708.57 | bwd_allreduce: 13.29 | step: 19.08

  8%|▊         | 3472/41250 [8:22:43<90:51:23,  8.66s/it]
                                                         
{'loss': 0.0829, 'grad_norm': 1.090951919555664, 'learning_rate': 3.969311774189354e-05, 'epoch': 0.84}

  8%|▊         | 3472/41250 [8:22:43<90:51:23,  8.66s/it][2025-04-26 12:31:21,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:31:21,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.50 | bwd_microstep: 5771.29 | bwd_inner_microstep: 5707.85 | bwd_allreduce_microstep: 63.40 | step_microstep: 18.77
[2025-04-26 12:31:21,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.50 | bwd: 5771.31 | bwd_inner: 5707.85 | bwd_allreduce: 63.42 | step: 18.77

  8%|▊         | 3473/41250 [8:22:52<91:00:17,  8.67s/it]
                                                         
{'loss': 0.2544, 'grad_norm': 2.0894715785980225, 'learning_rate': 3.9692843648037686e-05, 'epoch': 0.84}

  8%|▊         | 3473/41250 [8:22:52<91:00:17,  8.67s/it][2025-04-26 12:31:30,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 12:31:30,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.32 | bwd_microstep: 5775.80 | bwd_inner_microstep: 5712.99 | bwd_allreduce_microstep: 62.76 | step_microstep: 18.79
[2025-04-26 12:31:30,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.32 | bwd: 5775.82 | bwd_inner: 5712.99 | bwd_allreduce: 62.78 | step: 18.80

  8%|▊         | 3474/41250 [8:23:01<91:07:25,  8.68s/it]
                                                         
{'loss': 0.1653, 'grad_norm': 1.344338059425354, 'learning_rate': 3.969256943277932e-05, 'epoch': 0.84}

  8%|▊         | 3474/41250 [8:23:01<91:07:25,  8.68s/it][2025-04-26 12:31:38,674] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 12:31:38,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5716.74 | bwd_inner_microstep: 5703.67 | bwd_allreduce_microstep: 13.03 | step_microstep: 18.61
[2025-04-26 12:31:38,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5716.76 | bwd_inner: 5703.67 | bwd_allreduce: 13.05 | step: 18.62

  8%|▊         | 3475/41250 [8:23:09<91:00:20,  8.67s/it]
                                                         
{'loss': 0.2266, 'grad_norm': 1.6476861238479614, 'learning_rate': 3.9692295096120146e-05, 'epoch': 0.84}

  8%|▊         | 3475/41250 [8:23:09<91:00:20,  8.67s/it][2025-04-26 12:31:47,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:31:47,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.82 | bwd_microstep: 5714.80 | bwd_inner_microstep: 5702.13 | bwd_allreduce_microstep: 12.64 | step_microstep: 17.71
[2025-04-26 12:31:47,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.82 | bwd: 5714.82 | bwd_inner: 5702.13 | bwd_allreduce: 12.65 | step: 17.71

  8%|▊         | 3476/41250 [8:23:18<90:53:09,  8.66s/it]
                                                         
{'loss': 0.1761, 'grad_norm': 1.5900686979293823, 'learning_rate': 3.969202063806184e-05, 'epoch': 0.84}

  8%|▊         | 3476/41250 [8:23:18<90:53:09,  8.66s/it][2025-04-26 12:31:56,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:31:56,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.50 | bwd_microstep: 5800.25 | bwd_inner_microstep: 5654.12 | bwd_allreduce_microstep: 146.09 | step_microstep: 18.11
[2025-04-26 12:31:56,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.50 | bwd: 5800.26 | bwd_inner: 5654.12 | bwd_allreduce: 146.11 | step: 18.11

  8%|▊         | 3477/41250 [8:23:27<91:02:18,  8.68s/it]
                                                         
{'loss': 0.0956, 'grad_norm': 0.7758203744888306, 'learning_rate': 3.96917460586061e-05, 'epoch': 0.84}

  8%|▊         | 3477/41250 [8:23:27<91:02:18,  8.68s/it][2025-04-26 12:32:04,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 12:32:04,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.06 | bwd_microstep: 5773.95 | bwd_inner_microstep: 5697.19 | bwd_allreduce_microstep: 76.71 | step_microstep: 18.83
[2025-04-26 12:32:04,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.06 | bwd: 5773.96 | bwd_inner: 5697.19 | bwd_allreduce: 76.73 | step: 18.83

  8%|▊         | 3478/41250 [8:23:35<91:08:29,  8.69s/it]
                                                         
{'loss': 0.1328, 'grad_norm': 1.2866120338439941, 'learning_rate': 3.9691471357754616e-05, 'epoch': 0.84}

  8%|▊         | 3478/41250 [8:23:35<91:08:29,  8.69s/it][2025-04-26 12:32:13,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:32:13,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.06 | bwd_microstep: 5756.87 | bwd_inner_microstep: 5663.38 | bwd_allreduce_microstep: 93.44 | step_microstep: 18.75
[2025-04-26 12:32:13,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.06 | bwd: 5756.89 | bwd_inner: 5663.38 | bwd_allreduce: 93.46 | step: 18.76

  8%|▊         | 3479/41250 [8:23:44<91:08:14,  8.69s/it]
                                                         
{'loss': 0.1672, 'grad_norm': 0.843311607837677, 'learning_rate': 3.969119653550909e-05, 'epoch': 0.84}

  8%|▊         | 3479/41250 [8:23:44<91:08:14,  8.69s/it][2025-04-26 12:32:22,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:32:22,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.18 | bwd_microstep: 5780.31 | bwd_inner_microstep: 5767.52 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.71
[2025-04-26 12:32:22,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.18 | bwd: 5780.33 | bwd_inner: 5767.52 | bwd_allreduce: 12.77 | step: 18.71

  8%|▊         | 3480/41250 [8:23:53<91:21:24,  8.71s/it]
                                                         
{'loss': 0.2833, 'grad_norm': 1.802015781402588, 'learning_rate': 3.9690921591871205e-05, 'epoch': 0.84}

  8%|▊         | 3480/41250 [8:23:53<91:21:24,  8.71s/it][2025-04-26 12:32:30,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:32:30,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.67 | bwd_microstep: 5752.33 | bwd_inner_microstep: 5712.08 | bwd_allreduce_microstep: 40.20 | step_microstep: 18.78
[2025-04-26 12:32:30,856] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.67 | bwd: 5752.35 | bwd_inner: 5712.08 | bwd_allreduce: 40.22 | step: 18.79

  8%|▊         | 3481/41250 [8:24:02<91:16:49,  8.70s/it]
                                                         
{'loss': 0.0403, 'grad_norm': 0.4569554626941681, 'learning_rate': 3.9690646526842663e-05, 'epoch': 0.84}

  8%|▊         | 3481/41250 [8:24:02<91:16:49,  8.70s/it][2025-04-26 12:32:39,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:32:39,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.53 | bwd_microstep: 5747.57 | bwd_inner_microstep: 5719.15 | bwd_allreduce_microstep: 28.37 | step_microstep: 18.68
[2025-04-26 12:32:39,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.53 | bwd: 5747.58 | bwd_inner: 5719.15 | bwd_allreduce: 28.39 | step: 18.68

  8%|▊         | 3482/41250 [8:24:10<91:14:10,  8.70s/it]
                                                         
{'loss': 0.1533, 'grad_norm': 1.5129274129867554, 'learning_rate': 3.969037134042516e-05, 'epoch': 0.84}

  8%|▊         | 3482/41250 [8:24:10<91:14:10,  8.70s/it][2025-04-26 12:32:48,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 12:32:48,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.99 | bwd_microstep: 5788.17 | bwd_inner_microstep: 5648.91 | bwd_allreduce_microstep: 139.21 | step_microstep: 18.59
[2025-04-26 12:32:48,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.99 | bwd: 5788.18 | bwd_inner: 5648.91 | bwd_allreduce: 139.23 | step: 18.60

  8%|▊         | 3483/41250 [8:24:19<91:14:16,  8.70s/it]
                                                         
{'loss': 0.1921, 'grad_norm': 1.0217217206954956, 'learning_rate': 3.969009603262039e-05, 'epoch': 0.84}

  8%|▊         | 3483/41250 [8:24:19<91:14:16,  8.70s/it][2025-04-26 12:32:56,949] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:32:56,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.15 | bwd_microstep: 5780.28 | bwd_inner_microstep: 5669.74 | bwd_allreduce_microstep: 110.49 | step_microstep: 18.32
[2025-04-26 12:32:56,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.15 | bwd: 5780.29 | bwd_inner: 5669.74 | bwd_allreduce: 110.51 | step: 18.32

  8%|▊         | 3484/41250 [8:24:28<91:15:48,  8.70s/it]
                                                         
{'loss': 0.2828, 'grad_norm': 3.202249050140381, 'learning_rate': 3.968982060343005e-05, 'epoch': 0.84}

  8%|▊         | 3484/41250 [8:24:28<91:15:48,  8.70s/it][2025-04-26 12:33:05,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:33:05,641] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.98 | bwd_microstep: 5756.82 | bwd_inner_microstep: 5728.08 | bwd_allreduce_microstep: 28.69 | step_microstep: 18.25
[2025-04-26 12:33:05,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.98 | bwd: 5756.83 | bwd_inner: 5728.08 | bwd_allreduce: 28.71 | step: 18.25

  8%|▊         | 3485/41250 [8:24:36<91:14:06,  8.70s/it]
                                                         
{'loss': 0.2807, 'grad_norm': 2.4836971759796143, 'learning_rate': 3.9689545052855835e-05, 'epoch': 0.84}

  8%|▊         | 3485/41250 [8:24:36<91:14:06,  8.70s/it][2025-04-26 12:33:14,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:33:14,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.90 | bwd_microstep: 5752.55 | bwd_inner_microstep: 5692.46 | bwd_allreduce_microstep: 60.04 | step_microstep: 17.82
[2025-04-26 12:33:14,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.90 | bwd: 5752.56 | bwd_inner: 5692.46 | bwd_allreduce: 60.06 | step: 17.83

  8%|▊         | 3486/41250 [8:24:45<91:11:32,  8.69s/it]
                                                         
{'loss': 0.1358, 'grad_norm': 0.987447202205658, 'learning_rate': 3.9689269380899456e-05, 'epoch': 0.85}

  8%|▊         | 3486/41250 [8:24:45<91:11:32,  8.69s/it][2025-04-26 12:33:22,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:33:22,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.14 | bwd_microstep: 5698.30 | bwd_inner_microstep: 5644.61 | bwd_allreduce_microstep: 53.65 | step_microstep: 18.17
[2025-04-26 12:33:22,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.14 | bwd: 5698.32 | bwd_inner: 5644.61 | bwd_allreduce: 53.67 | step: 18.17

  8%|▊         | 3487/41250 [8:24:54<90:57:30,  8.67s/it]
                                                         
{'loss': 0.1612, 'grad_norm': 3.951578140258789, 'learning_rate': 3.96889935875626e-05, 'epoch': 0.85}

  8%|▊         | 3487/41250 [8:24:54<90:57:30,  8.67s/it][2025-04-26 12:33:31,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:33:31,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.95 | bwd_microstep: 5739.72 | bwd_inner_microstep: 5701.53 | bwd_allreduce_microstep: 38.14 | step_microstep: 18.72
[2025-04-26 12:33:31,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.95 | bwd: 5739.73 | bwd_inner: 5701.53 | bwd_allreduce: 38.16 | step: 18.72

  8%|▊         | 3488/41250 [8:25:02<90:58:49,  8.67s/it]
                                                         
{'loss': 0.1482, 'grad_norm': 1.6189484596252441, 'learning_rate': 3.968871767284697e-05, 'epoch': 0.85}

  8%|▊         | 3488/41250 [8:25:02<90:58:49,  8.67s/it][2025-04-26 12:33:40,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:33:40,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.52 | bwd_microstep: 5785.22 | bwd_inner_microstep: 5654.30 | bwd_allreduce_microstep: 130.87 | step_microstep: 18.54
[2025-04-26 12:33:40,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.52 | bwd: 5785.23 | bwd_inner: 5654.30 | bwd_allreduce: 130.89 | step: 18.54

  8%|▊         | 3489/41250 [8:25:11<91:03:05,  8.68s/it]
                                                         
{'loss': 0.1714, 'grad_norm': 1.659438133239746, 'learning_rate': 3.968844163675426e-05, 'epoch': 0.85}

  8%|▊         | 3489/41250 [8:25:11<91:03:05,  8.68s/it][2025-04-26 12:33:48,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.01 | optimizer_step: 1.06
[2025-04-26 12:33:48,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.89 | bwd_microstep: 5719.91 | bwd_inner_microstep: 5706.71 | bwd_allreduce_microstep: 13.15 | step_microstep: 18.77
[2025-04-26 12:33:48,961] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.89 | bwd: 5719.92 | bwd_inner: 5706.71 | bwd_allreduce: 13.17 | step: 18.77

  8%|▊         | 3490/41250 [8:25:20<90:55:24,  8.67s/it]
                                                         
{'loss': 0.0374, 'grad_norm': 0.5767075419425964, 'learning_rate': 3.9688165479286195e-05, 'epoch': 0.85}

  8%|▊         | 3490/41250 [8:25:20<90:55:24,  8.67s/it][2025-04-26 12:33:57,571] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:33:57,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.59 | bwd_microstep: 5700.81 | bwd_inner_microstep: 5659.86 | bwd_allreduce_microstep: 40.91 | step_microstep: 18.61
[2025-04-26 12:33:57,572] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.59 | bwd: 5700.83 | bwd_inner: 5659.86 | bwd_allreduce: 40.92 | step: 18.61

  8%|▊         | 3491/41250 [8:25:28<90:44:24,  8.65s/it]
                                                         
{'loss': 0.2664, 'grad_norm': 1.5014662742614746, 'learning_rate': 3.968788920044445e-05, 'epoch': 0.85}

  8%|▊         | 3491/41250 [8:25:28<90:44:24,  8.65s/it][2025-04-26 12:34:06,227] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:34:06,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.39 | bwd_microstep: 5720.90 | bwd_inner_microstep: 5707.96 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.79
[2025-04-26 12:34:06,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.39 | bwd: 5720.92 | bwd_inner: 5707.96 | bwd_allreduce: 12.91 | step: 18.79

  8%|▊         | 3492/41250 [8:25:37<90:45:37,  8.65s/it]
                                                         
{'loss': 0.1441, 'grad_norm': 1.5604757070541382, 'learning_rate': 3.968761280023075e-05, 'epoch': 0.85}

  8%|▊         | 3492/41250 [8:25:37<90:45:37,  8.65s/it][2025-04-26 12:34:14,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:34:14,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.59 | bwd_microstep: 5706.36 | bwd_inner_microstep: 5656.50 | bwd_allreduce_microstep: 49.82 | step_microstep: 18.48
[2025-04-26 12:34:14,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.59 | bwd: 5706.38 | bwd_inner: 5656.50 | bwd_allreduce: 49.84 | step: 18.48

  8%|▊         | 3493/41250 [8:25:46<90:38:26,  8.64s/it]
                                                         
{'loss': 0.3881, 'grad_norm': 1.8108460903167725, 'learning_rate': 3.968733627864679e-05, 'epoch': 0.85}

  8%|▊         | 3493/41250 [8:25:46<90:38:26,  8.64s/it][2025-04-26 12:34:23,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 12:34:23,457] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.84 | bwd_microstep: 5708.87 | bwd_inner_microstep: 5657.15 | bwd_allreduce_microstep: 51.68 | step_microstep: 18.94
[2025-04-26 12:34:23,458] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.84 | bwd: 5708.88 | bwd_inner: 5657.15 | bwd_allreduce: 51.70 | step: 18.94

  8%|▊         | 3494/41250 [8:25:54<90:32:24,  8.63s/it]
                                                         
{'loss': 0.104, 'grad_norm': 0.9669821262359619, 'learning_rate': 3.9687059635694266e-05, 'epoch': 0.85}

  8%|▊         | 3494/41250 [8:25:54<90:32:24,  8.63s/it][2025-04-26 12:34:32,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 12:34:32,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.01 | bwd_microstep: 5731.92 | bwd_inner_microstep: 5698.14 | bwd_allreduce_microstep: 33.74 | step_microstep: 18.77
[2025-04-26 12:34:32,120] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.01 | bwd: 5731.93 | bwd_inner: 5698.14 | bwd_allreduce: 33.76 | step: 18.77

  8%|▊         | 3495/41250 [8:26:03<90:37:53,  8.64s/it]
                                                         
{'loss': 0.2843, 'grad_norm': 1.1691313982009888, 'learning_rate': 3.96867828713749e-05, 'epoch': 0.85}

  8%|▊         | 3495/41250 [8:26:03<90:37:53,  8.64s/it][2025-04-26 12:34:40,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:34:40,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.87 | bwd_microstep: 5709.66 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.52
[2025-04-26 12:34:40,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.87 | bwd: 5709.67 | bwd_inner: 5696.99 | bwd_allreduce: 12.64 | step: 18.53

  8%|▊         | 3496/41250 [8:26:11<90:37:29,  8.64s/it]
                                                         
{'loss': 0.1851, 'grad_norm': 1.6518586874008179, 'learning_rate': 3.968650598569038e-05, 'epoch': 0.85}

  8%|▊         | 3496/41250 [8:26:11<90:37:29,  8.64s/it][2025-04-26 12:34:49,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 12:34:49,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.63 | bwd_microstep: 5869.92 | bwd_inner_microstep: 5660.06 | bwd_allreduce_microstep: 209.81 | step_microstep: 18.69
[2025-04-26 12:34:49,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.63 | bwd: 5869.93 | bwd_inner: 5660.06 | bwd_allreduce: 209.83 | step: 18.69

  8%|▊         | 3497/41250 [8:26:20<91:02:56,  8.68s/it]
                                                         
{'loss': 0.1185, 'grad_norm': 1.1621544361114502, 'learning_rate': 3.968622897864243e-05, 'epoch': 0.85}

  8%|▊         | 3497/41250 [8:26:20<91:02:56,  8.68s/it][2025-04-26 12:34:58,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.00
[2025-04-26 12:34:58,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.56 | bwd_microstep: 5749.07 | bwd_inner_microstep: 5680.13 | bwd_allreduce_microstep: 68.90 | step_microstep: 18.58
[2025-04-26 12:34:58,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.56 | bwd: 5749.08 | bwd_inner: 5680.13 | bwd_allreduce: 68.92 | step: 18.58

  8%|▊         | 3498/41250 [8:26:29<91:02:29,  8.68s/it]
                                                         
{'loss': 0.2062, 'grad_norm': 1.8551859855651855, 'learning_rate': 3.9685951850232745e-05, 'epoch': 0.85}

  8%|▊         | 3498/41250 [8:26:29<91:02:29,  8.68s/it][2025-04-26 12:35:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:35:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.96 | bwd_microstep: 5766.67 | bwd_inner_microstep: 5696.01 | bwd_allreduce_microstep: 70.61 | step_microstep: 18.78
[2025-04-26 12:35:06,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.96 | bwd: 5766.68 | bwd_inner: 5696.01 | bwd_allreduce: 70.63 | step: 18.78

  8%|▊         | 3499/41250 [8:26:38<91:04:03,  8.68s/it]
                                                         
{'loss': 0.2447, 'grad_norm': 1.5398681163787842, 'learning_rate': 3.968567460046304e-05, 'epoch': 0.85}

  8%|▊         | 3499/41250 [8:26:38<91:04:03,  8.68s/it][2025-04-26 12:35:15,598] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:35:15,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.89 | bwd_microstep: 5787.12 | bwd_inner_microstep: 5641.14 | bwd_allreduce_microstep: 145.93 | step_microstep: 19.04
[2025-04-26 12:35:15,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.89 | bwd: 5787.13 | bwd_inner: 5641.14 | bwd_allreduce: 145.95 | step: 19.05

  8%|▊         | 3500/41250 [8:26:46<91:05:01,  8.69s/it]
                                                         
{'loss': 0.0374, 'grad_norm': 0.43228423595428467, 'learning_rate': 3.968539722933503e-05, 'epoch': 0.85}

  8%|▊         | 3500/41250 [8:26:46<91:05:01,  8.69s/it][2025-04-26 12:35:24,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 1.09
[2025-04-26 12:35:24,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.79 | bwd_microstep: 5685.37 | bwd_inner_microstep: 5644.00 | bwd_allreduce_microstep: 41.33 | step_microstep: 19.01
[2025-04-26 12:35:24,197] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.79 | bwd: 5685.39 | bwd_inner: 5644.00 | bwd_allreduce: 41.35 | step: 19.01

  8%|▊         | 3501/41250 [8:26:55<90:48:02,  8.66s/it]
                                                         
{'loss': 0.3128, 'grad_norm': 1.9850846529006958, 'learning_rate': 3.968511973685041e-05, 'epoch': 0.85}

  8%|▊         | 3501/41250 [8:26:55<90:48:02,  8.66s/it][2025-04-26 12:35:32,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:35:32,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.59 | bwd_microstep: 5736.05 | bwd_inner_microstep: 5696.60 | bwd_allreduce_microstep: 39.40 | step_microstep: 18.65
[2025-04-26 12:35:32,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.59 | bwd: 5736.06 | bwd_inner: 5696.60 | bwd_allreduce: 39.42 | step: 18.65

  8%|▊         | 3502/41250 [8:27:04<90:48:24,  8.66s/it]
                                                         
{'loss': 0.2144, 'grad_norm': 2.4844844341278076, 'learning_rate': 3.96848421230109e-05, 'epoch': 0.85}

  8%|▊         | 3502/41250 [8:27:04<90:48:24,  8.66s/it][2025-04-26 12:35:41,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:35:41,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.71 | bwd_microstep: 5800.30 | bwd_inner_microstep: 5645.94 | bwd_allreduce_microstep: 154.31 | step_microstep: 18.59
[2025-04-26 12:35:41,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.71 | bwd: 5800.31 | bwd_inner: 5645.94 | bwd_allreduce: 154.33 | step: 18.59

  8%|▊         | 3503/41250 [8:27:12<90:54:05,  8.67s/it]
                                                         
{'loss': 0.0951, 'grad_norm': 1.1700522899627686, 'learning_rate': 3.9684564387818213e-05, 'epoch': 0.85}

  8%|▊         | 3503/41250 [8:27:12<90:54:05,  8.67s/it][2025-04-26 12:35:50,186] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:35:50,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.63 | bwd_microstep: 5705.34 | bwd_inner_microstep: 5692.54 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.52
[2025-04-26 12:35:50,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.63 | bwd: 5705.35 | bwd_inner: 5692.54 | bwd_allreduce: 12.77 | step: 18.53

  8%|▊         | 3504/41250 [8:27:21<90:47:48,  8.66s/it]
                                                         
{'loss': 0.0688, 'grad_norm': 0.5509632229804993, 'learning_rate': 3.9684286531274064e-05, 'epoch': 0.85}

  8%|▊         | 3504/41250 [8:27:21<90:47:48,  8.66s/it][2025-04-26 12:35:58,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:35:58,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.67 | bwd_microstep: 5759.74 | bwd_inner_microstep: 5648.84 | bwd_allreduce_microstep: 110.84 | step_microstep: 18.30
[2025-04-26 12:35:58,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.67 | bwd: 5759.75 | bwd_inner: 5648.84 | bwd_allreduce: 110.87 | step: 18.31

  8%|▊         | 3505/41250 [8:27:30<90:50:27,  8.66s/it]
                                                         
{'loss': 0.138, 'grad_norm': 1.5855473279953003, 'learning_rate': 3.9684008553380147e-05, 'epoch': 0.85}

  8%|▊         | 3505/41250 [8:27:30<90:50:27,  8.66s/it][2025-04-26 12:36:07,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 12:36:07,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.15 | bwd_microstep: 5733.10 | bwd_inner_microstep: 5694.54 | bwd_allreduce_microstep: 38.52 | step_microstep: 18.73
[2025-04-26 12:36:07,524] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.15 | bwd: 5733.12 | bwd_inner: 5694.54 | bwd_allreduce: 38.54 | step: 18.74

  8%|▊         | 3506/41250 [8:27:38<90:50:11,  8.66s/it]
                                                         
{'loss': 0.3031, 'grad_norm': 2.589695930480957, 'learning_rate': 3.9683730454138195e-05, 'epoch': 0.85}

  8%|▊         | 3506/41250 [8:27:38<90:50:11,  8.66s/it][2025-04-26 12:36:16,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.06 | optimizer_step: 0.92
[2025-04-26 12:36:16,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.00 | bwd_microstep: 5749.94 | bwd_inner_microstep: 5687.23 | bwd_allreduce_microstep: 62.65 | step_microstep: 18.57
[2025-04-26 12:36:16,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.00 | bwd: 5749.95 | bwd_inner: 5687.23 | bwd_allreduce: 62.67 | step: 18.57

  9%|▊         | 3507/41250 [8:27:47<90:52:13,  8.67s/it]
                                                         
{'loss': 0.2249, 'grad_norm': 1.7398067712783813, 'learning_rate': 3.968345223354992e-05, 'epoch': 0.85}

  9%|▊         | 3507/41250 [8:27:47<90:52:13,  8.67s/it][2025-04-26 12:36:24,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:36:24,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2862.29 | bwd_microstep: 5722.78 | bwd_inner_microstep: 5710.14 | bwd_allreduce_microstep: 12.61 | step_microstep: 18.21
[2025-04-26 12:36:24,860] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2862.29 | bwd: 5722.80 | bwd_inner: 5710.13 | bwd_allreduce: 12.62 | step: 18.21

  9%|▊         | 3508/41250 [8:27:56<90:50:31,  8.66s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.3450639247894287, 'learning_rate': 3.9683173891617034e-05, 'epoch': 0.85}

  9%|▊         | 3508/41250 [8:27:56<90:50:31,  8.66s/it][2025-04-26 12:36:33,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:36:33,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.81 | bwd_microstep: 5709.20 | bwd_inner_microstep: 5696.87 | bwd_allreduce_microstep: 12.28 | step_microstep: 18.25
[2025-04-26 12:36:33,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.81 | bwd: 5709.21 | bwd_inner: 5696.87 | bwd_allreduce: 12.30 | step: 18.26

  9%|▊         | 3509/41250 [8:28:04<90:46:08,  8.66s/it]
                                                         
{'loss': 0.1343, 'grad_norm': 0.5782833099365234, 'learning_rate': 3.968289542834124e-05, 'epoch': 0.85}

  9%|▊         | 3509/41250 [8:28:04<90:46:08,  8.66s/it][2025-04-26 12:36:42,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:36:42,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.37 | bwd_microstep: 5702.42 | bwd_inner_microstep: 5689.86 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.28
[2025-04-26 12:36:42,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.38 | bwd: 5702.44 | bwd_inner: 5689.86 | bwd_allreduce: 12.54 | step: 18.28

  9%|▊         | 3510/41250 [8:28:13<90:40:30,  8.65s/it]
                                                         
{'loss': 0.2405, 'grad_norm': 1.349311113357544, 'learning_rate': 3.9682616843724276e-05, 'epoch': 0.85}

  9%|▊         | 3510/41250 [8:28:13<90:40:30,  8.65s/it][2025-04-26 12:36:50,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:36:50,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.71 | bwd_microstep: 5724.14 | bwd_inner_microstep: 5711.42 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.38
[2025-04-26 12:36:50,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.71 | bwd: 5724.16 | bwd_inner: 5711.42 | bwd_allreduce: 12.70 | step: 18.39

  9%|▊         | 3511/41250 [8:28:21<90:40:38,  8.65s/it]
                                                         
{'loss': 0.2496, 'grad_norm': 1.47836434841156, 'learning_rate': 3.968233813776785e-05, 'epoch': 0.85}

  9%|▊         | 3511/41250 [8:28:21<90:40:38,  8.65s/it][2025-04-26 12:36:59,399] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 12:36:59,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.11 | bwd_microstep: 5718.90 | bwd_inner_microstep: 5654.24 | bwd_allreduce_microstep: 64.61 | step_microstep: 18.68
[2025-04-26 12:36:59,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.11 | bwd: 5718.91 | bwd_inner: 5654.24 | bwd_allreduce: 64.63 | step: 18.68

  9%|▊         | 3512/41250 [8:28:30<90:34:20,  8.64s/it]
                                                         
{'loss': 0.3779, 'grad_norm': 2.272956609725952, 'learning_rate': 3.9682059310473684e-05, 'epoch': 0.85}

  9%|▊         | 3512/41250 [8:28:30<90:34:20,  8.64s/it][2025-04-26 12:37:08,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:37:08,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.64 | bwd_microstep: 5725.85 | bwd_inner_microstep: 5643.20 | bwd_allreduce_microstep: 82.61 | step_microstep: 17.87
[2025-04-26 12:37:08,028] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.64 | bwd: 5725.86 | bwd_inner: 5643.20 | bwd_allreduce: 82.62 | step: 17.88

  9%|▊         | 3513/41250 [8:28:39<90:32:02,  8.64s/it]
                                                         
{'loss': 0.1012, 'grad_norm': 1.0715670585632324, 'learning_rate': 3.968178036184349e-05, 'epoch': 0.85}

  9%|▊         | 3513/41250 [8:28:39<90:32:02,  8.64s/it][2025-04-26 12:37:16,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.05
[2025-04-26 12:37:16,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5796.88 | bwd_inner_microstep: 5666.18 | bwd_allreduce_microstep: 130.65 | step_microstep: 18.80
[2025-04-26 12:37:16,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5796.89 | bwd_inner: 5666.18 | bwd_allreduce: 130.67 | step: 18.80

  9%|▊         | 3514/41250 [8:28:47<90:45:49,  8.66s/it]
                                                         
{'loss': 0.1429, 'grad_norm': 1.6836128234863281, 'learning_rate': 3.968150129187899e-05, 'epoch': 0.85}

  9%|▊         | 3514/41250 [8:28:47<90:45:49,  8.66s/it][2025-04-26 12:37:25,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:37:25,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.50 | bwd_microstep: 5740.58 | bwd_inner_microstep: 5718.35 | bwd_allreduce_microstep: 22.18 | step_microstep: 18.50
[2025-04-26 12:37:25,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.50 | bwd: 5740.59 | bwd_inner: 5718.35 | bwd_allreduce: 22.20 | step: 18.50

  9%|▊         | 3515/41250 [8:28:56<90:49:35,  8.67s/it]
                                                         
{'loss': 0.176, 'grad_norm': 0.8108009696006775, 'learning_rate': 3.9681222100581905e-05, 'epoch': 0.85}

  9%|▊         | 3515/41250 [8:28:56<90:49:35,  8.67s/it][2025-04-26 12:37:34,046] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:37:34,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.38 | bwd_microstep: 5706.02 | bwd_inner_microstep: 5658.36 | bwd_allreduce_microstep: 47.61 | step_microstep: 17.67
[2025-04-26 12:37:34,047] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.38 | bwd: 5706.03 | bwd_inner: 5658.36 | bwd_allreduce: 47.63 | step: 17.68

  9%|▊         | 3516/41250 [8:29:05<90:42:28,  8.65s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 2.2793643474578857, 'learning_rate': 3.968094278795396e-05, 'epoch': 0.85}

  9%|▊         | 3516/41250 [8:29:05<90:42:28,  8.65s/it][2025-04-26 12:37:42,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:37:42,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.01 | bwd_microstep: 5713.62 | bwd_inner_microstep: 5700.92 | bwd_allreduce_microstep: 12.66 | step_microstep: 17.74
[2025-04-26 12:37:42,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.01 | bwd: 5713.63 | bwd_inner: 5700.92 | bwd_allreduce: 12.68 | step: 17.75

  9%|▊         | 3517/41250 [8:29:13<90:40:25,  8.65s/it]
                                                         
{'loss': 0.4186, 'grad_norm': 2.424947738647461, 'learning_rate': 3.9680663353996875e-05, 'epoch': 0.85}

  9%|▊         | 3517/41250 [8:29:13<90:40:25,  8.65s/it][2025-04-26 12:37:51,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:37:51,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.51 | bwd_microstep: 5779.71 | bwd_inner_microstep: 5649.81 | bwd_allreduce_microstep: 129.85 | step_microstep: 18.44
[2025-04-26 12:37:51,378] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.51 | bwd: 5779.72 | bwd_inner: 5649.81 | bwd_allreduce: 129.87 | step: 18.44

  9%|▊         | 3518/41250 [8:29:22<90:47:21,  8.66s/it]
                                                         
{'loss': 0.2025, 'grad_norm': 1.7066082954406738, 'learning_rate': 3.968038379871237e-05, 'epoch': 0.85}

  9%|▊         | 3518/41250 [8:29:22<90:47:21,  8.66s/it][2025-04-26 12:38:00,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.98 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:38:00,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.00 | bwd_microstep: 5883.72 | bwd_inner_microstep: 5713.77 | bwd_allreduce_microstep: 169.91 | step_microstep: 17.75
[2025-04-26 12:38:00,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.00 | bwd: 5883.73 | bwd_inner: 5713.77 | bwd_allreduce: 169.92 | step: 17.75

  9%|▊         | 3519/41250 [8:29:31<91:17:42,  8.71s/it]
                                                         
{'loss': 0.1421, 'grad_norm': 1.0008732080459595, 'learning_rate': 3.968010412210217e-05, 'epoch': 0.85}

  9%|▊         | 3519/41250 [8:29:31<91:17:42,  8.71s/it][2025-04-26 12:38:08,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:38:08,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.03 | bwd_microstep: 5726.83 | bwd_inner_microstep: 5713.87 | bwd_allreduce_microstep: 12.91 | step_microstep: 17.75
[2025-04-26 12:38:08,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.03 | bwd: 5726.84 | bwd_inner: 5713.87 | bwd_allreduce: 12.93 | step: 17.75

  9%|▊         | 3520/41250 [8:29:40<91:09:02,  8.70s/it]
                                                         
{'loss': 0.1663, 'grad_norm': 1.1060799360275269, 'learning_rate': 3.9679824324168e-05, 'epoch': 0.85}

  9%|▊         | 3520/41250 [8:29:40<91:09:02,  8.70s/it][2025-04-26 12:38:17,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 12:38:17,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.06 | bwd_microstep: 5880.84 | bwd_inner_microstep: 5717.69 | bwd_allreduce_microstep: 163.11 | step_microstep: 18.10
[2025-04-26 12:38:17,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.06 | bwd: 5880.85 | bwd_inner: 5717.69 | bwd_allreduce: 163.13 | step: 18.10

  9%|▊         | 3521/41250 [8:29:48<91:30:21,  8.73s/it]
                                                         
{'loss': 0.1605, 'grad_norm': 1.0323333740234375, 'learning_rate': 3.967954440491158e-05, 'epoch': 0.85}

  9%|▊         | 3521/41250 [8:29:48<91:30:21,  8.73s/it][2025-04-26 12:38:26,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 12:38:26,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.17 | bwd_microstep: 5713.10 | bwd_inner_microstep: 5700.42 | bwd_allreduce_microstep: 12.63 | step_microstep: 19.03
[2025-04-26 12:38:26,325] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.17 | bwd: 5713.11 | bwd_inner: 5700.42 | bwd_allreduce: 12.65 | step: 19.04

  9%|▊         | 3522/41250 [8:29:57<91:14:12,  8.71s/it]
                                                         
{'loss': 0.259, 'grad_norm': 1.2134625911712646, 'learning_rate': 3.967926436433464e-05, 'epoch': 0.85}

  9%|▊         | 3522/41250 [8:29:57<91:14:12,  8.71s/it][2025-04-26 12:38:35,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:38:35,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.90 | bwd_microstep: 5795.94 | bwd_inner_microstep: 5657.60 | bwd_allreduce_microstep: 138.30 | step_microstep: 18.69
[2025-04-26 12:38:35,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.90 | bwd: 5795.96 | bwd_inner: 5657.60 | bwd_allreduce: 138.31 | step: 18.69

  9%|▊         | 3523/41250 [8:30:06<91:14:59,  8.71s/it]
                                                         
{'loss': 0.1261, 'grad_norm': 1.5931528806686401, 'learning_rate': 3.96789842024389e-05, 'epoch': 0.85}

  9%|▊         | 3523/41250 [8:30:06<91:14:59,  8.71s/it][2025-04-26 12:38:43,746] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:38:43,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.04 | bwd_microstep: 5787.00 | bwd_inner_microstep: 5664.62 | bwd_allreduce_microstep: 122.34 | step_microstep: 18.63
[2025-04-26 12:38:43,747] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.04 | bwd: 5787.02 | bwd_inner: 5664.62 | bwd_allreduce: 122.36 | step: 18.63

  9%|▊         | 3524/41250 [8:30:14<91:15:30,  8.71s/it]
                                                         
{'loss': 0.1647, 'grad_norm': 1.5328747034072876, 'learning_rate': 3.9678703919226106e-05, 'epoch': 0.85}

  9%|▊         | 3524/41250 [8:30:14<91:15:30,  8.71s/it][2025-04-26 12:38:52,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:38:52,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.63 | bwd_microstep: 5746.69 | bwd_inner_microstep: 5703.87 | bwd_allreduce_microstep: 42.78 | step_microstep: 18.49
[2025-04-26 12:38:52,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.63 | bwd: 5746.70 | bwd_inner: 5703.87 | bwd_allreduce: 42.79 | step: 18.50

  9%|▊         | 3525/41250 [8:30:23<91:09:49,  8.70s/it]
                                                         
{'loss': 0.2882, 'grad_norm': 1.5879619121551514, 'learning_rate': 3.967842351469797e-05, 'epoch': 0.85}

  9%|▊         | 3525/41250 [8:30:23<91:09:49,  8.70s/it][2025-04-26 12:39:01,122] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 1.11
[2025-04-26 12:39:01,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.28 | bwd_microstep: 5790.29 | bwd_inner_microstep: 5674.11 | bwd_allreduce_microstep: 116.12 | step_microstep: 18.54
[2025-04-26 12:39:01,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.28 | bwd: 5790.30 | bwd_inner: 5674.11 | bwd_allreduce: 116.15 | step: 18.54

  9%|▊         | 3526/41250 [8:30:32<91:09:07,  8.70s/it]
                                                         
{'loss': 0.2534, 'grad_norm': 0.9161572456359863, 'learning_rate': 3.967814298885622e-05, 'epoch': 0.85}

  9%|▊         | 3526/41250 [8:30:32<91:09:07,  8.70s/it][2025-04-26 12:39:09,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 12:39:09,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.85 | bwd_microstep: 5771.70 | bwd_inner_microstep: 5657.16 | bwd_allreduce_microstep: 114.49 | step_microstep: 18.61
[2025-04-26 12:39:09,805] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.85 | bwd: 5771.71 | bwd_inner: 5657.16 | bwd_allreduce: 114.51 | step: 18.61

  9%|▊         | 3527/41250 [8:30:40<91:06:00,  8.69s/it]
                                                         
{'loss': 0.1011, 'grad_norm': 1.2198410034179688, 'learning_rate': 3.967786234170259e-05, 'epoch': 0.86}

  9%|▊         | 3527/41250 [8:30:40<91:06:00,  8.69s/it][2025-04-26 12:39:18,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.90 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:39:18,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5773.90 | bwd_inner_microstep: 5689.98 | bwd_allreduce_microstep: 83.88 | step_microstep: 17.51
[2025-04-26 12:39:18,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5773.91 | bwd_inner: 5689.98 | bwd_allreduce: 83.89 | step: 17.51

  9%|▊         | 3528/41250 [8:30:49<91:08:16,  8.70s/it]
                                                         
{'loss': 0.2433, 'grad_norm': 1.596645474433899, 'learning_rate': 3.9677581573238806e-05, 'epoch': 0.86}

  9%|▊         | 3528/41250 [8:30:49<91:08:16,  8.70s/it][2025-04-26 12:39:27,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:39:27,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.47 | bwd_microstep: 5682.28 | bwd_inner_microstep: 5652.34 | bwd_allreduce_microstep: 29.89 | step_microstep: 17.71
[2025-04-26 12:39:27,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.47 | bwd: 5682.29 | bwd_inner: 5652.34 | bwd_allreduce: 29.90 | step: 17.71

  9%|▊         | 3529/41250 [8:30:58<90:48:51,  8.67s/it]
                                                         
{'loss': 0.0427, 'grad_norm': 0.36881616711616516, 'learning_rate': 3.967730068346661e-05, 'epoch': 0.86}

  9%|▊         | 3529/41250 [8:30:58<90:48:51,  8.67s/it][2025-04-26 12:39:35,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:39:35,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.05 | bwd_microstep: 5771.07 | bwd_inner_microstep: 5711.86 | bwd_allreduce_microstep: 59.17 | step_microstep: 18.41
[2025-04-26 12:39:35,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.05 | bwd: 5771.09 | bwd_inner: 5711.86 | bwd_allreduce: 59.19 | step: 18.41

  9%|▊         | 3530/41250 [8:31:06<90:54:48,  8.68s/it]
                                                         
{'loss': 0.1038, 'grad_norm': 1.1730544567108154, 'learning_rate': 3.967701967238772e-05, 'epoch': 0.86}

  9%|▊         | 3530/41250 [8:31:06<90:54:48,  8.68s/it][2025-04-26 12:39:44,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:39:44,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5742.07 | bwd_inner_microstep: 5677.39 | bwd_allreduce_microstep: 64.64 | step_microstep: 18.44
[2025-04-26 12:39:44,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5742.09 | bwd_inner: 5677.39 | bwd_allreduce: 64.66 | step: 18.45

  9%|▊         | 3531/41250 [8:31:15<90:54:26,  8.68s/it]
                                                         
{'loss': 0.2902, 'grad_norm': 2.0671207904815674, 'learning_rate': 3.967673854000387e-05, 'epoch': 0.86}

  9%|▊         | 3531/41250 [8:31:15<90:54:26,  8.68s/it][2025-04-26 12:39:53,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:39:53,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.30 | bwd_microstep: 5758.39 | bwd_inner_microstep: 5701.74 | bwd_allreduce_microstep: 56.60 | step_microstep: 18.19
[2025-04-26 12:39:53,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.30 | bwd: 5758.40 | bwd_inner: 5701.74 | bwd_allreduce: 56.62 | step: 18.19

  9%|▊         | 3532/41250 [8:31:24<90:57:28,  8.68s/it]
                                                         
{'loss': 0.118, 'grad_norm': 1.4210810661315918, 'learning_rate': 3.967645728631681e-05, 'epoch': 0.86}

  9%|▊         | 3532/41250 [8:31:24<90:57:28,  8.68s/it][2025-04-26 12:40:01,931] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:40:01,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.73 | bwd_microstep: 5780.15 | bwd_inner_microstep: 5767.78 | bwd_allreduce_microstep: 12.32 | step_microstep: 18.03
[2025-04-26 12:40:01,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.73 | bwd: 5780.16 | bwd_inner: 5767.78 | bwd_allreduce: 12.34 | step: 18.04

  9%|▊         | 3533/41250 [8:31:33<91:11:27,  8.70s/it]
                                                         
{'loss': 0.1273, 'grad_norm': 1.0918574333190918, 'learning_rate': 3.9676175911328247e-05, 'epoch': 0.86}

  9%|▊         | 3533/41250 [8:31:33<91:11:27,  8.70s/it][2025-04-26 12:40:10,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:40:10,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.23 | bwd_microstep: 5712.19 | bwd_inner_microstep: 5699.44 | bwd_allreduce_microstep: 12.72 | step_microstep: 17.71
[2025-04-26 12:40:10,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.23 | bwd: 5712.21 | bwd_inner: 5699.44 | bwd_allreduce: 12.73 | step: 17.71

  9%|▊         | 3534/41250 [8:31:41<91:00:32,  8.69s/it]
                                                         
{'loss': 0.21, 'grad_norm': 1.4244999885559082, 'learning_rate': 3.967589441503994e-05, 'epoch': 0.86}

  9%|▊         | 3534/41250 [8:31:41<91:00:32,  8.69s/it][2025-04-26 12:40:19,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 12:40:19,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.92 | bwd_microstep: 5770.13 | bwd_inner_microstep: 5650.82 | bwd_allreduce_microstep: 119.28 | step_microstep: 18.68
[2025-04-26 12:40:19,269] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.92 | bwd: 5770.15 | bwd_inner: 5650.82 | bwd_allreduce: 119.29 | step: 18.68

  9%|▊         | 3535/41250 [8:31:50<91:00:57,  8.69s/it]
                                                         
{'loss': 0.3201, 'grad_norm': 1.8159255981445312, 'learning_rate': 3.9675612797453605e-05, 'epoch': 0.86}

  9%|▊         | 3535/41250 [8:31:50<91:00:57,  8.69s/it][2025-04-26 12:40:27,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:40:27,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.95 | bwd_microstep: 5698.16 | bwd_inner_microstep: 5685.57 | bwd_allreduce_microstep: 12.55 | step_microstep: 17.83
[2025-04-26 12:40:27,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.95 | bwd: 5698.17 | bwd_inner: 5685.57 | bwd_allreduce: 12.56 | step: 17.83

  9%|▊         | 3536/41250 [8:31:59<90:49:43,  8.67s/it]
                                                         
{'loss': 0.1988, 'grad_norm': 0.9484725594520569, 'learning_rate': 3.967533105857099e-05, 'epoch': 0.86}

  9%|▊         | 3536/41250 [8:31:59<90:49:43,  8.67s/it][2025-04-26 12:40:36,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:40:36,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.08 | bwd_microstep: 5752.46 | bwd_inner_microstep: 5686.67 | bwd_allreduce_microstep: 65.75 | step_microstep: 18.11
[2025-04-26 12:40:36,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.08 | bwd: 5752.47 | bwd_inner: 5686.67 | bwd_allreduce: 65.77 | step: 18.12

  9%|▊         | 3537/41250 [8:32:07<90:51:56,  8.67s/it]
                                                         
{'loss': 0.1329, 'grad_norm': 1.466301679611206, 'learning_rate': 3.967504919839383e-05, 'epoch': 0.86}

  9%|▊         | 3537/41250 [8:32:07<90:51:56,  8.67s/it][2025-04-26 12:40:45,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:40:45,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5749.31 | bwd_inner_microstep: 5689.80 | bwd_allreduce_microstep: 59.47 | step_microstep: 18.08
[2025-04-26 12:40:45,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5749.32 | bwd_inner: 5689.80 | bwd_allreduce: 59.49 | step: 18.08

  9%|▊         | 3538/41250 [8:32:16<90:52:51,  8.68s/it]
                                                         
{'loss': 0.3604, 'grad_norm': 2.338857412338257, 'learning_rate': 3.967476721692386e-05, 'epoch': 0.86}

  9%|▊         | 3538/41250 [8:32:16<90:52:51,  8.68s/it][2025-04-26 12:40:53,888] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:40:53,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.82 | bwd_microstep: 5704.87 | bwd_inner_microstep: 5692.20 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.30
[2025-04-26 12:40:53,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.82 | bwd: 5704.88 | bwd_inner: 5692.20 | bwd_allreduce: 12.64 | step: 18.30

  9%|▊         | 3539/41250 [8:32:25<90:43:49,  8.66s/it]
                                                         
{'loss': 0.2019, 'grad_norm': 1.4154154062271118, 'learning_rate': 3.967448511416282e-05, 'epoch': 0.86}

  9%|▊         | 3539/41250 [8:32:25<90:43:49,  8.66s/it][2025-04-26 12:41:02,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:41:02,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.74 | bwd_microstep: 5727.84 | bwd_inner_microstep: 5705.49 | bwd_allreduce_microstep: 22.31 | step_microstep: 18.76
[2025-04-26 12:41:02,562] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.74 | bwd: 5727.86 | bwd_inner: 5705.49 | bwd_allreduce: 22.33 | step: 18.76

  9%|▊         | 3540/41250 [8:32:33<90:46:16,  8.67s/it]
                                                         
{'loss': 0.1228, 'grad_norm': 0.8785567879676819, 'learning_rate': 3.967420289011245e-05, 'epoch': 0.86}

  9%|▊         | 3540/41250 [8:32:33<90:46:16,  8.67s/it][2025-04-26 12:41:11,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:41:11,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.06 | bwd_microstep: 5754.46 | bwd_inner_microstep: 5649.16 | bwd_allreduce_microstep: 105.26 | step_microstep: 18.44
[2025-04-26 12:41:11,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.06 | bwd: 5754.48 | bwd_inner: 5649.16 | bwd_allreduce: 105.28 | step: 18.44

  9%|▊         | 3541/41250 [8:32:42<90:45:05,  8.66s/it]
                                                         
{'loss': 0.1659, 'grad_norm': 2.0386595726013184, 'learning_rate': 3.967392054477449e-05, 'epoch': 0.86}

  9%|▊         | 3541/41250 [8:32:42<90:45:05,  8.66s/it][2025-04-26 12:41:19,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.97 | optimizer_step: 1.05
[2025-04-26 12:41:19,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.12 | bwd_microstep: 5778.41 | bwd_inner_microstep: 5642.11 | bwd_allreduce_microstep: 136.25 | step_microstep: 18.27
[2025-04-26 12:41:19,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.12 | bwd: 5778.42 | bwd_inner: 5642.11 | bwd_allreduce: 136.27 | step: 18.28

  9%|▊         | 3542/41250 [8:32:51<90:49:42,  8.67s/it]
                                                         
{'loss': 0.2417, 'grad_norm': 1.8604108095169067, 'learning_rate': 3.9673638078150685e-05, 'epoch': 0.86}

  9%|▊         | 3542/41250 [8:32:51<90:49:42,  8.67s/it][2025-04-26 12:41:28,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:41:28,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.64 | bwd_microstep: 5719.12 | bwd_inner_microstep: 5706.31 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.76
[2025-04-26 12:41:28,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.64 | bwd: 5719.14 | bwd_inner: 5706.31 | bwd_allreduce: 12.78 | step: 18.77

  9%|▊         | 3543/41250 [8:32:59<90:45:16,  8.66s/it]
                                                         
{'loss': 0.1463, 'grad_norm': 1.8961924314498901, 'learning_rate': 3.9673355490242763e-05, 'epoch': 0.86}

  9%|▊         | 3543/41250 [8:32:59<90:45:16,  8.66s/it][2025-04-26 12:41:37,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 0.90
[2025-04-26 12:41:37,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.89 | bwd_microstep: 5780.25 | bwd_inner_microstep: 5647.96 | bwd_allreduce_microstep: 132.24 | step_microstep: 19.04
[2025-04-26 12:41:37,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.89 | bwd: 5780.27 | bwd_inner: 5647.96 | bwd_allreduce: 132.26 | step: 19.04

  9%|▊         | 3544/41250 [8:33:08<90:48:34,  8.67s/it]
                                                         
{'loss': 0.3384, 'grad_norm': 1.4127624034881592, 'learning_rate': 3.967307278105248e-05, 'epoch': 0.86}

  9%|▊         | 3544/41250 [8:33:08<90:48:34,  8.67s/it][2025-04-26 12:41:45,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 12:41:45,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.60 | bwd_microstep: 5696.01 | bwd_inner_microstep: 5683.26 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.70
[2025-04-26 12:41:45,868] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.60 | bwd: 5696.02 | bwd_inner: 5683.26 | bwd_allreduce: 12.72 | step: 18.71

  9%|▊         | 3545/41250 [8:33:17<90:40:01,  8.66s/it]
                                                         
{'loss': 0.273, 'grad_norm': 3.1351866722106934, 'learning_rate': 3.967278995058157e-05, 'epoch': 0.86}

  9%|▊         | 3545/41250 [8:33:17<90:40:01,  8.66s/it][2025-04-26 12:41:54,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:41:54,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.26 | bwd_microstep: 5727.11 | bwd_inner_microstep: 5697.63 | bwd_allreduce_microstep: 29.44 | step_microstep: 18.68
[2025-04-26 12:41:54,530] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.26 | bwd: 5727.13 | bwd_inner: 5697.63 | bwd_allreduce: 29.45 | step: 18.69

  9%|▊         | 3546/41250 [8:33:25<90:41:01,  8.66s/it]
                                                         
{'loss': 0.3136, 'grad_norm': 1.8433281183242798, 'learning_rate': 3.9672506998831775e-05, 'epoch': 0.86}

  9%|▊         | 3546/41250 [8:33:25<90:41:01,  8.66s/it][2025-04-26 12:42:03,144] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.94
[2025-04-26 12:42:03,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.70 | bwd_microstep: 5679.41 | bwd_inner_microstep: 5656.77 | bwd_allreduce_microstep: 22.60 | step_microstep: 18.71
[2025-04-26 12:42:03,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.70 | bwd: 5679.42 | bwd_inner: 5656.77 | bwd_allreduce: 22.61 | step: 18.71

  9%|▊         | 3547/41250 [8:33:34<90:32:16,  8.64s/it]
                                                         
{'loss': 0.2152, 'grad_norm': 1.347225308418274, 'learning_rate': 3.967222392580484e-05, 'epoch': 0.86}

  9%|▊         | 3547/41250 [8:33:34<90:32:16,  8.64s/it][2025-04-26 12:42:11,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:42:11,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.97 | bwd_microstep: 5757.58 | bwd_inner_microstep: 5658.60 | bwd_allreduce_microstep: 98.94 | step_microstep: 17.75
[2025-04-26 12:42:11,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.97 | bwd: 5757.59 | bwd_inner: 5658.60 | bwd_allreduce: 98.96 | step: 17.75

  9%|▊         | 3548/41250 [8:33:42<90:35:13,  8.65s/it]
                                                         
{'loss': 0.1996, 'grad_norm': 2.8845887184143066, 'learning_rate': 3.9671940731502524e-05, 'epoch': 0.86}

  9%|▊         | 3548/41250 [8:33:42<90:35:13,  8.65s/it][2025-04-26 12:42:20,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:42:20,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.49 | bwd_microstep: 5771.97 | bwd_inner_microstep: 5646.85 | bwd_allreduce_microstep: 125.07 | step_microstep: 17.93
[2025-04-26 12:42:20,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.49 | bwd: 5771.98 | bwd_inner: 5646.85 | bwd_allreduce: 125.09 | step: 17.93

  9%|▊         | 3549/41250 [8:33:51<90:41:29,  8.66s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 1.0170331001281738, 'learning_rate': 3.967165741592656e-05, 'epoch': 0.86}

  9%|▊         | 3549/41250 [8:33:51<90:41:29,  8.66s/it][2025-04-26 12:42:29,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:42:29,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.91 | bwd_microstep: 5762.55 | bwd_inner_microstep: 5654.05 | bwd_allreduce_microstep: 108.45 | step_microstep: 18.30
[2025-04-26 12:42:29,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.91 | bwd: 5762.56 | bwd_inner: 5654.05 | bwd_allreduce: 108.47 | step: 18.30

  9%|▊         | 3550/41250 [8:34:00<90:44:15,  8.66s/it]
                                                         
{'loss': 0.2702, 'grad_norm': 3.035503387451172, 'learning_rate': 3.9671373979078696e-05, 'epoch': 0.86}

  9%|▊         | 3550/41250 [8:34:00<90:44:15,  8.66s/it][2025-04-26 12:42:37,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:42:37,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.26 | bwd_microstep: 5695.09 | bwd_inner_microstep: 5682.33 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.03
[2025-04-26 12:42:37,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.26 | bwd: 5695.10 | bwd_inner: 5682.33 | bwd_allreduce: 12.73 | step: 18.03

  9%|▊         | 3551/41250 [8:34:08<90:36:13,  8.65s/it]
                                                         
{'loss': 0.2252, 'grad_norm': 3.574868679046631, 'learning_rate': 3.9671090420960684e-05, 'epoch': 0.86}

  9%|▊         | 3551/41250 [8:34:08<90:36:13,  8.65s/it][2025-04-26 12:42:46,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:42:46,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.46 | bwd_microstep: 5703.74 | bwd_inner_microstep: 5690.46 | bwd_allreduce_microstep: 13.24 | step_microstep: 18.44
[2025-04-26 12:42:46,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.46 | bwd: 5703.76 | bwd_inner: 5690.46 | bwd_allreduce: 13.26 | step: 18.44

  9%|▊         | 3552/41250 [8:34:17<90:33:22,  8.65s/it]
                                                         
{'loss': 0.3141, 'grad_norm': 2.2444968223571777, 'learning_rate': 3.9670806741574265e-05, 'epoch': 0.86}

  9%|▊         | 3552/41250 [8:34:17<90:33:22,  8.65s/it][2025-04-26 12:42:55,123] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 12:42:55,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.42 | bwd_microstep: 5776.61 | bwd_inner_microstep: 5689.93 | bwd_allreduce_microstep: 86.64 | step_microstep: 18.20
[2025-04-26 12:42:55,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.42 | bwd: 5776.63 | bwd_inner: 5689.93 | bwd_allreduce: 86.66 | step: 18.20

  9%|▊         | 3553/41250 [8:34:26<90:42:42,  8.66s/it]
                                                         
{'loss': 0.0891, 'grad_norm': 1.1515023708343506, 'learning_rate': 3.967052294092119e-05, 'epoch': 0.86}

  9%|▊         | 3553/41250 [8:34:26<90:42:42,  8.66s/it][2025-04-26 12:43:03,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:43:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.05 | bwd_microstep: 5728.40 | bwd_inner_microstep: 5696.35 | bwd_allreduce_microstep: 32.01 | step_microstep: 17.90
[2025-04-26 12:43:03,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.05 | bwd: 5728.41 | bwd_inner: 5696.35 | bwd_allreduce: 32.03 | step: 17.90

  9%|▊         | 3554/41250 [8:34:34<90:43:04,  8.66s/it]
                                                         
{'loss': 0.3578, 'grad_norm': 1.395309329032898, 'learning_rate': 3.967023901900321e-05, 'epoch': 0.86}

  9%|▊         | 3554/41250 [8:34:34<90:43:04,  8.66s/it][2025-04-26 12:43:12,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 1.11
[2025-04-26 12:43:12,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.69 | bwd_microstep: 5737.76 | bwd_inner_microstep: 5702.17 | bwd_allreduce_microstep: 35.54 | step_microstep: 19.18
[2025-04-26 12:43:12,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.69 | bwd: 5737.77 | bwd_inner: 5702.17 | bwd_allreduce: 35.56 | step: 19.18

  9%|▊         | 3555/41250 [8:34:43<90:45:06,  8.67s/it]
                                                         
{'loss': 0.3076, 'grad_norm': 0.9638082981109619, 'learning_rate': 3.966995497582208e-05, 'epoch': 0.86}

  9%|▊         | 3555/41250 [8:34:43<90:45:06,  8.67s/it][2025-04-26 12:43:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:43:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.47 | bwd_microstep: 5760.88 | bwd_inner_microstep: 5693.95 | bwd_allreduce_microstep: 66.89 | step_microstep: 18.39
[2025-04-26 12:43:21,157] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.47 | bwd: 5760.90 | bwd_inner: 5693.95 | bwd_allreduce: 66.91 | step: 18.39

  9%|▊         | 3556/41250 [8:34:52<90:49:45,  8.67s/it]
                                                         
{'loss': 0.1393, 'grad_norm': 1.2215760946273804, 'learning_rate': 3.9669670811379544e-05, 'epoch': 0.86}

  9%|▊         | 3556/41250 [8:34:52<90:49:45,  8.67s/it][2025-04-26 12:43:29,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 12:43:29,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.53 | bwd_microstep: 5725.22 | bwd_inner_microstep: 5644.53 | bwd_allreduce_microstep: 80.65 | step_microstep: 18.90
[2025-04-26 12:43:29,780] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.53 | bwd: 5725.24 | bwd_inner: 5644.53 | bwd_allreduce: 80.66 | step: 18.90

  9%|▊         | 3557/41250 [8:35:00<90:39:53,  8.66s/it]
                                                         
{'loss': 0.169, 'grad_norm': 1.3637518882751465, 'learning_rate': 3.966938652567735e-05, 'epoch': 0.86}

  9%|▊         | 3557/41250 [8:35:00<90:39:53,  8.66s/it][2025-04-26 12:43:38,419] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:43:38,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.09 | bwd_microstep: 5709.44 | bwd_inner_microstep: 5696.58 | bwd_allreduce_microstep: 12.81 | step_microstep: 18.80
[2025-04-26 12:43:38,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.09 | bwd: 5709.45 | bwd_inner: 5696.58 | bwd_allreduce: 12.83 | step: 18.81

  9%|▊         | 3558/41250 [8:35:09<90:36:02,  8.65s/it]
                                                         
{'loss': 0.1701, 'grad_norm': 0.9783226847648621, 'learning_rate': 3.966910211871727e-05, 'epoch': 0.86}

  9%|▊         | 3558/41250 [8:35:09<90:36:02,  8.65s/it][2025-04-26 12:43:47,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:43:47,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5769.37 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 72.33 | step_microstep: 18.42
[2025-04-26 12:43:47,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5769.38 | bwd_inner: 5696.99 | bwd_allreduce: 72.34 | step: 18.42

  9%|▊         | 3559/41250 [8:35:18<90:44:14,  8.67s/it]
                                                         
{'loss': 0.1615, 'grad_norm': 0.8071712851524353, 'learning_rate': 3.966881759050104e-05, 'epoch': 0.86}

  9%|▊         | 3559/41250 [8:35:18<90:44:14,  8.67s/it][2025-04-26 12:43:55,802] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:43:55,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.21 | bwd_microstep: 5749.85 | bwd_inner_microstep: 5718.03 | bwd_allreduce_microstep: 31.78 | step_microstep: 18.64
[2025-04-26 12:43:55,803] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.21 | bwd: 5749.86 | bwd_inner: 5718.03 | bwd_allreduce: 31.79 | step: 18.65

  9%|▊         | 3560/41250 [8:35:26<90:48:00,  8.67s/it]
                                                         
{'loss': 0.1204, 'grad_norm': 0.7550594806671143, 'learning_rate': 3.966853294103041e-05, 'epoch': 0.86}

  9%|▊         | 3560/41250 [8:35:26<90:48:00,  8.67s/it][2025-04-26 12:44:04,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 12:44:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.95 | bwd_microstep: 5726.93 | bwd_inner_microstep: 5658.08 | bwd_allreduce_microstep: 68.80 | step_microstep: 18.92
[2025-04-26 12:44:04,445] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.95 | bwd: 5726.94 | bwd_inner: 5658.08 | bwd_allreduce: 68.82 | step: 18.92

  9%|▊         | 3561/41250 [8:35:35<90:41:46,  8.66s/it]
                                                         
{'loss': 0.1106, 'grad_norm': 0.9584838151931763, 'learning_rate': 3.9668248170307154e-05, 'epoch': 0.86}

  9%|▊         | 3561/41250 [8:35:35<90:41:46,  8.66s/it][2025-04-26 12:44:13,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:44:13,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.69 | bwd_microstep: 5780.52 | bwd_inner_microstep: 5660.63 | bwd_allreduce_microstep: 119.85 | step_microstep: 18.15
[2025-04-26 12:44:13,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.69 | bwd: 5780.53 | bwd_inner: 5660.63 | bwd_allreduce: 119.86 | step: 18.15

  9%|▊         | 3562/41250 [8:35:44<90:46:16,  8.67s/it]
                                                         
{'loss': 0.1366, 'grad_norm': 1.819603681564331, 'learning_rate': 3.9667963278333006e-05, 'epoch': 0.86}

  9%|▊         | 3562/41250 [8:35:44<90:46:16,  8.67s/it][2025-04-26 12:44:21,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:44:21,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.41 | bwd_microstep: 5733.10 | bwd_inner_microstep: 5671.81 | bwd_allreduce_microstep: 61.24 | step_microstep: 18.44
[2025-04-26 12:44:21,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.41 | bwd: 5733.11 | bwd_inner: 5671.81 | bwd_allreduce: 61.26 | step: 18.45

  9%|▊         | 3563/41250 [8:35:52<90:42:25,  8.66s/it]
                                                         
{'loss': 0.1129, 'grad_norm': 1.8063324689865112, 'learning_rate': 3.966767826510974e-05, 'epoch': 0.86}

  9%|▊         | 3563/41250 [8:35:52<90:42:25,  8.66s/it][2025-04-26 12:44:30,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:44:30,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.65 | bwd_microstep: 5740.33 | bwd_inner_microstep: 5702.29 | bwd_allreduce_microstep: 38.00 | step_microstep: 17.94
[2025-04-26 12:44:30,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.65 | bwd: 5740.34 | bwd_inner: 5702.29 | bwd_allreduce: 38.02 | step: 17.95

  9%|▊         | 3564/41250 [8:36:01<90:45:49,  8.67s/it]
                                                         
{'loss': 0.111, 'grad_norm': 1.2336176633834839, 'learning_rate': 3.9667393130639104e-05, 'epoch': 0.86}

  9%|▊         | 3564/41250 [8:36:01<90:45:49,  8.67s/it][2025-04-26 12:44:39,129] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.18 | optimizer_step: 0.93
[2025-04-26 12:44:39,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.64 | bwd_microstep: 5726.89 | bwd_inner_microstep: 5713.67 | bwd_allreduce_microstep: 13.17 | step_microstep: 19.08
[2025-04-26 12:44:39,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.64 | bwd: 5726.91 | bwd_inner: 5713.67 | bwd_allreduce: 13.19 | step: 19.08

  9%|▊         | 3565/41250 [8:36:10<90:44:10,  8.67s/it]
                                                         
{'loss': 0.1724, 'grad_norm': 1.3883556127548218, 'learning_rate': 3.966710787492286e-05, 'epoch': 0.86}

  9%|▊         | 3565/41250 [8:36:10<90:44:10,  8.67s/it][2025-04-26 12:44:47,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 1.03
[2025-04-26 12:44:47,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.52 | bwd_microstep: 5870.35 | bwd_inner_microstep: 5717.00 | bwd_allreduce_microstep: 153.31 | step_microstep: 18.55
[2025-04-26 12:44:47,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.52 | bwd: 5870.36 | bwd_inner: 5717.00 | bwd_allreduce: 153.33 | step: 18.55

  9%|▊         | 3566/41250 [8:36:19<91:08:29,  8.71s/it]
                                                         
{'loss': 0.117, 'grad_norm': 0.9642267823219299, 'learning_rate': 3.966682249796276e-05, 'epoch': 0.86}

  9%|▊         | 3566/41250 [8:36:19<91:08:29,  8.71s/it][2025-04-26 12:44:56,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.99
[2025-04-26 12:44:56,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.34 | bwd_microstep: 5743.52 | bwd_inner_microstep: 5716.17 | bwd_allreduce_microstep: 27.31 | step_microstep: 18.65
[2025-04-26 12:44:56,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.34 | bwd: 5743.54 | bwd_inner: 5716.17 | bwd_allreduce: 27.33 | step: 18.66

  9%|▊         | 3567/41250 [8:36:27<91:03:03,  8.70s/it]
                                                         
{'loss': 0.2391, 'grad_norm': 1.604455828666687, 'learning_rate': 3.966653699976057e-05, 'epoch': 0.86}

  9%|▊         | 3567/41250 [8:36:27<91:03:03,  8.70s/it][2025-04-26 12:45:05,311] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 1.07
[2025-04-26 12:45:05,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.96 | bwd_microstep: 5774.95 | bwd_inner_microstep: 5686.04 | bwd_allreduce_microstep: 88.87 | step_microstep: 18.65
[2025-04-26 12:45:05,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.96 | bwd: 5774.97 | bwd_inner: 5686.04 | bwd_allreduce: 88.88 | step: 18.65

  9%|▊         | 3568/41250 [8:36:36<91:04:09,  8.70s/it]
                                                         
{'loss': 0.2197, 'grad_norm': 2.518606185913086, 'learning_rate': 3.966625138031805e-05, 'epoch': 0.86}

  9%|▊         | 3568/41250 [8:36:36<91:04:09,  8.70s/it][2025-04-26 12:45:14,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 12:45:14,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.04 | bwd_microstep: 5797.11 | bwd_inner_microstep: 5663.61 | bwd_allreduce_microstep: 133.46 | step_microstep: 18.73
[2025-04-26 12:45:14,023] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.04 | bwd: 5797.12 | bwd_inner: 5663.61 | bwd_allreduce: 133.48 | step: 18.73

  9%|▊         | 3569/41250 [8:36:45<91:06:26,  8.70s/it]
                                                         
{'loss': 0.0755, 'grad_norm': 0.7149373888969421, 'learning_rate': 3.966596563963695e-05, 'epoch': 0.87}

  9%|▊         | 3569/41250 [8:36:45<91:06:26,  8.70s/it][2025-04-26 12:45:22,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 0.99
[2025-04-26 12:45:22,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.21 | bwd_microstep: 5706.85 | bwd_inner_microstep: 5651.38 | bwd_allreduce_microstep: 55.42 | step_microstep: 18.93
[2025-04-26 12:45:22,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.21 | bwd: 5706.86 | bwd_inner: 5651.38 | bwd_allreduce: 55.44 | step: 18.94

  9%|▊         | 3570/41250 [8:36:53<90:49:04,  8.68s/it]
                                                         
{'loss': 0.2717, 'grad_norm': 3.787489175796509, 'learning_rate': 3.9665679777719046e-05, 'epoch': 0.87}

  9%|▊         | 3570/41250 [8:36:53<90:49:04,  8.68s/it][2025-04-26 12:45:31,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:45:31,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.60 | bwd_microstep: 5743.84 | bwd_inner_microstep: 5696.69 | bwd_allreduce_microstep: 47.10 | step_microstep: 18.28
[2025-04-26 12:45:31,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.60 | bwd: 5743.85 | bwd_inner: 5696.69 | bwd_allreduce: 47.12 | step: 18.28

  9%|▊         | 3571/41250 [8:37:02<90:49:29,  8.68s/it]
                                                         
{'loss': 0.376, 'grad_norm': 2.9605331420898438, 'learning_rate': 3.9665393794566096e-05, 'epoch': 0.87}

  9%|▊         | 3571/41250 [8:37:02<90:49:29,  8.68s/it][2025-04-26 12:45:39,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:45:39,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.82 | bwd_microstep: 5697.75 | bwd_inner_microstep: 5656.35 | bwd_allreduce_microstep: 41.36 | step_microstep: 18.33
[2025-04-26 12:45:39,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.82 | bwd: 5697.76 | bwd_inner: 5656.35 | bwd_allreduce: 41.38 | step: 18.33

  9%|▊         | 3572/41250 [8:37:11<90:38:55,  8.66s/it]
                                                         
{'loss': 0.1278, 'grad_norm': 1.0052270889282227, 'learning_rate': 3.9665107690179854e-05, 'epoch': 0.87}

  9%|▊         | 3572/41250 [8:37:11<90:38:55,  8.66s/it][2025-04-26 12:45:48,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:45:48,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.91 | bwd_microstep: 5781.03 | bwd_inner_microstep: 5651.74 | bwd_allreduce_microstep: 129.25 | step_microstep: 17.71
[2025-04-26 12:45:48,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.91 | bwd: 5781.04 | bwd_inner: 5651.73 | bwd_allreduce: 129.27 | step: 17.71

  9%|▊         | 3573/41250 [8:37:19<90:45:25,  8.67s/it]
                                                         
{'loss': 0.3502, 'grad_norm': 2.3124425411224365, 'learning_rate': 3.9664821464562095e-05, 'epoch': 0.87}

  9%|▊         | 3573/41250 [8:37:19<90:45:25,  8.67s/it][2025-04-26 12:45:57,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:45:57,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.91 | bwd_microstep: 5766.89 | bwd_inner_microstep: 5703.12 | bwd_allreduce_microstep: 63.72 | step_microstep: 18.24
[2025-04-26 12:45:57,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.91 | bwd: 5766.90 | bwd_inner: 5703.12 | bwd_allreduce: 63.74 | step: 18.24

  9%|▊         | 3574/41250 [8:37:28<90:51:23,  8.68s/it]
                                                         
{'loss': 0.0588, 'grad_norm': 0.4750567674636841, 'learning_rate': 3.966453511771458e-05, 'epoch': 0.87}

  9%|▊         | 3574/41250 [8:37:28<90:51:23,  8.68s/it][2025-04-26 12:46:05,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:46:05,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.67 | bwd_microstep: 5718.56 | bwd_inner_microstep: 5705.43 | bwd_allreduce_microstep: 13.08 | step_microstep: 18.69
[2025-04-26 12:46:05,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.67 | bwd: 5718.57 | bwd_inner: 5705.43 | bwd_allreduce: 13.10 | step: 18.70

  9%|▊         | 3575/41250 [8:37:37<90:44:11,  8.67s/it]
                                                         
{'loss': 0.3069, 'grad_norm': 1.7498903274536133, 'learning_rate': 3.966424864963908e-05, 'epoch': 0.87}

  9%|▊         | 3575/41250 [8:37:37<90:44:11,  8.67s/it][2025-04-26 12:46:14,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:46:14,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.83 | bwd_microstep: 5759.18 | bwd_inner_microstep: 5693.06 | bwd_allreduce_microstep: 66.08 | step_microstep: 18.37
[2025-04-26 12:46:14,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.83 | bwd: 5759.19 | bwd_inner: 5693.06 | bwd_allreduce: 66.10 | step: 18.37

  9%|▊         | 3576/41250 [8:37:45<90:47:59,  8.68s/it]
                                                         
{'loss': 0.0427, 'grad_norm': 0.5622393488883972, 'learning_rate': 3.966396206033734e-05, 'epoch': 0.87}

  9%|▊         | 3576/41250 [8:37:45<90:47:59,  8.68s/it][2025-04-26 12:46:23,304] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:46:23,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.99 | bwd_microstep: 5711.88 | bwd_inner_microstep: 5653.11 | bwd_allreduce_microstep: 58.72 | step_microstep: 19.10
[2025-04-26 12:46:23,305] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.99 | bwd: 5711.89 | bwd_inner: 5653.11 | bwd_allreduce: 58.74 | step: 19.11

  9%|▊         | 3577/41250 [8:37:54<90:39:08,  8.66s/it]
                                                         
{'loss': 0.3256, 'grad_norm': 12.387682914733887, 'learning_rate': 3.966367534981116e-05, 'epoch': 0.87}

  9%|▊         | 3577/41250 [8:37:54<90:39:08,  8.66s/it][2025-04-26 12:46:31,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.05 | optimizer_step: 1.03
[2025-04-26 12:46:31,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.99 | bwd_microstep: 5700.50 | bwd_inner_microstep: 5687.56 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.61
[2025-04-26 12:46:31,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.99 | bwd: 5700.52 | bwd_inner: 5687.55 | bwd_allreduce: 12.92 | step: 19.61

  9%|▊         | 3578/41250 [8:38:03<90:33:47,  8.65s/it]
                                                         
{'loss': 0.1943, 'grad_norm': 1.5620157718658447, 'learning_rate': 3.9663388518062275e-05, 'epoch': 0.87}

  9%|▊         | 3578/41250 [8:38:03<90:33:47,  8.65s/it][2025-04-26 12:46:40,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 12:46:40,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.18 | bwd_microstep: 5768.49 | bwd_inner_microstep: 5662.26 | bwd_allreduce_microstep: 106.19 | step_microstep: 18.38
[2025-04-26 12:46:40,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.18 | bwd: 5768.51 | bwd_inner: 5662.26 | bwd_allreduce: 106.21 | step: 18.38

  9%|▊         | 3579/41250 [8:38:11<90:38:59,  8.66s/it]
                                                         
{'loss': 0.0341, 'grad_norm': 0.2728898525238037, 'learning_rate': 3.9663101565092473e-05, 'epoch': 0.87}

  9%|▊         | 3579/41250 [8:38:11<90:38:59,  8.66s/it][2025-04-26 12:46:49,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:46:49,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.32 | bwd_microstep: 5848.09 | bwd_inner_microstep: 5710.65 | bwd_allreduce_microstep: 137.40 | step_microstep: 18.68
[2025-04-26 12:46:49,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.32 | bwd: 5848.10 | bwd_inner: 5710.65 | bwd_allreduce: 137.42 | step: 18.68

  9%|▊         | 3580/41250 [8:38:20<91:00:57,  8.70s/it]
                                                         
{'loss': 0.3717, 'grad_norm': 2.3116002082824707, 'learning_rate': 3.9662814490903515e-05, 'epoch': 0.87}

  9%|▊         | 3580/41250 [8:38:20<91:00:57,  8.70s/it][2025-04-26 12:46:58,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:46:58,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.60 | bwd_microstep: 5709.78 | bwd_inner_microstep: 5656.29 | bwd_allreduce_microstep: 53.44 | step_microstep: 18.76
[2025-04-26 12:46:58,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.60 | bwd: 5709.79 | bwd_inner: 5656.29 | bwd_allreduce: 53.46 | step: 18.77

  9%|▊         | 3581/41250 [8:38:29<90:45:45,  8.67s/it]
                                                         
{'loss': 0.1166, 'grad_norm': 1.2405261993408203, 'learning_rate': 3.966252729549718e-05, 'epoch': 0.87}

  9%|▊         | 3581/41250 [8:38:29<90:45:45,  8.67s/it][2025-04-26 12:47:06,642] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:47:06,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.03 | bwd_microstep: 5713.88 | bwd_inner_microstep: 5641.01 | bwd_allreduce_microstep: 72.82 | step_microstep: 18.76
[2025-04-26 12:47:06,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.04 | bwd: 5713.89 | bwd_inner: 5641.01 | bwd_allreduce: 72.84 | step: 18.76

  9%|▊         | 3582/41250 [8:38:37<90:35:29,  8.66s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 1.8396427631378174, 'learning_rate': 3.966223997887522e-05, 'epoch': 0.87}

  9%|▊         | 3582/41250 [8:38:37<90:35:29,  8.66s/it][2025-04-26 12:47:15,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:47:15,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.42 | bwd_microstep: 5754.41 | bwd_inner_microstep: 5683.35 | bwd_allreduce_microstep: 71.01 | step_microstep: 18.85
[2025-04-26 12:47:15,320] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.42 | bwd: 5754.42 | bwd_inner: 5683.35 | bwd_allreduce: 71.03 | step: 18.86

  9%|▊         | 3583/41250 [8:38:46<90:39:05,  8.66s/it]
                                                         
{'loss': 0.2147, 'grad_norm': 1.172221064567566, 'learning_rate': 3.9661952541039425e-05, 'epoch': 0.87}

  9%|▊         | 3583/41250 [8:38:46<90:39:05,  8.66s/it][2025-04-26 12:47:23,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 12:47:23,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.53 | bwd_microstep: 5780.37 | bwd_inner_microstep: 5641.05 | bwd_allreduce_microstep: 139.27 | step_microstep: 18.86
[2025-04-26 12:47:23,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.53 | bwd: 5780.39 | bwd_inner: 5641.05 | bwd_allreduce: 139.29 | step: 18.86

  9%|▊         | 3584/41250 [8:38:55<90:40:12,  8.67s/it]
                                                         
{'loss': 0.2001, 'grad_norm': 1.5039080381393433, 'learning_rate': 3.966166498199156e-05, 'epoch': 0.87}

  9%|▊         | 3584/41250 [8:38:55<90:40:12,  8.67s/it][2025-04-26 12:47:32,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:47:32,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.68 | bwd_microstep: 5691.14 | bwd_inner_microstep: 5643.23 | bwd_allreduce_microstep: 47.87 | step_microstep: 17.73
[2025-04-26 12:47:32,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.68 | bwd: 5691.15 | bwd_inner: 5643.23 | bwd_allreduce: 47.88 | step: 17.74

  9%|▊         | 3585/41250 [8:39:03<90:27:35,  8.65s/it]
                                                         
{'loss': 0.138, 'grad_norm': 1.4861199855804443, 'learning_rate': 3.96613773017334e-05, 'epoch': 0.87}

  9%|▊         | 3585/41250 [8:39:03<90:27:35,  8.65s/it][2025-04-26 12:47:41,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:47:41,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.41 | bwd_microstep: 5714.75 | bwd_inner_microstep: 5702.11 | bwd_allreduce_microstep: 12.60 | step_microstep: 17.98
[2025-04-26 12:47:41,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.41 | bwd: 5714.77 | bwd_inner: 5702.11 | bwd_allreduce: 12.62 | step: 17.98

  9%|▊         | 3586/41250 [8:39:12<90:26:59,  8.65s/it]
                                                         
{'loss': 0.0407, 'grad_norm': 0.6911059021949768, 'learning_rate': 3.966108950026671e-05, 'epoch': 0.87}

  9%|▊         | 3586/41250 [8:39:12<90:26:59,  8.65s/it][2025-04-26 12:47:49,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:47:49,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5685.49 | bwd_inner_microstep: 5653.48 | bwd_allreduce_microstep: 31.96 | step_microstep: 18.52
[2025-04-26 12:47:49,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5685.50 | bwd_inner: 5653.48 | bwd_allreduce: 31.98 | step: 18.52

  9%|▊         | 3587/41250 [8:39:21<90:18:14,  8.63s/it]
                                                         
{'loss': 0.1338, 'grad_norm': 1.4896091222763062, 'learning_rate': 3.966080157759327e-05, 'epoch': 0.87}

  9%|▊         | 3587/41250 [8:39:21<90:18:14,  8.63s/it][2025-04-26 12:47:58,492] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:47:58,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.98 | bwd_microstep: 5734.71 | bwd_inner_microstep: 5681.36 | bwd_allreduce_microstep: 53.31 | step_microstep: 18.60
[2025-04-26 12:47:58,493] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.98 | bwd: 5734.72 | bwd_inner: 5681.36 | bwd_allreduce: 53.32 | step: 18.60

  9%|▊         | 3588/41250 [8:39:29<90:23:13,  8.64s/it]
                                                         
{'loss': 0.1521, 'grad_norm': 1.7067352533340454, 'learning_rate': 3.966051353371486e-05, 'epoch': 0.87}

  9%|▊         | 3588/41250 [8:39:29<90:23:13,  8.64s/it][2025-04-26 12:48:07,162] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 12:48:07,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.31 | bwd_microstep: 5772.38 | bwd_inner_microstep: 5638.83 | bwd_allreduce_microstep: 133.51 | step_microstep: 18.57
[2025-04-26 12:48:07,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.31 | bwd: 5772.40 | bwd_inner: 5638.83 | bwd_allreduce: 133.53 | step: 18.57

  9%|▊         | 3589/41250 [8:39:38<90:28:46,  8.65s/it]
                                                         
{'loss': 0.0473, 'grad_norm': 0.5897782444953918, 'learning_rate': 3.9660225368633243e-05, 'epoch': 0.87}

  9%|▊         | 3589/41250 [8:39:38<90:28:46,  8.65s/it][2025-04-26 12:48:15,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:48:15,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.40 | bwd_microstep: 5750.68 | bwd_inner_microstep: 5647.13 | bwd_allreduce_microstep: 103.51 | step_microstep: 18.54
[2025-04-26 12:48:15,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.40 | bwd: 5750.70 | bwd_inner: 5647.13 | bwd_allreduce: 103.53 | step: 18.55

  9%|▊         | 3590/41250 [8:39:47<90:31:04,  8.65s/it]
                                                         
{'loss': 0.2712, 'grad_norm': 1.7770226001739502, 'learning_rate': 3.9659937082350214e-05, 'epoch': 0.87}

  9%|▊         | 3590/41250 [8:39:47<90:31:04,  8.65s/it][2025-04-26 12:48:24,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 12:48:24,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.30 | bwd_microstep: 5758.87 | bwd_inner_microstep: 5682.52 | bwd_allreduce_microstep: 76.30 | step_microstep: 18.79
[2025-04-26 12:48:24,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.30 | bwd: 5758.88 | bwd_inner: 5682.52 | bwd_allreduce: 76.32 | step: 18.79

  9%|▊         | 3591/41250 [8:39:55<90:37:11,  8.66s/it]
                                                         
{'loss': 0.157, 'grad_norm': 1.7079894542694092, 'learning_rate': 3.965964867486754e-05, 'epoch': 0.87}

  9%|▊         | 3591/41250 [8:39:55<90:37:11,  8.66s/it][2025-04-26 12:48:33,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:48:33,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.80 | bwd_microstep: 5700.91 | bwd_inner_microstep: 5688.08 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.65
[2025-04-26 12:48:33,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.80 | bwd: 5700.92 | bwd_inner: 5688.08 | bwd_allreduce: 12.80 | step: 18.66

  9%|▊         | 3592/41250 [8:40:04<90:32:54,  8.66s/it]
                                                         
{'loss': 0.1227, 'grad_norm': 1.5338642597198486, 'learning_rate': 3.965936014618699e-05, 'epoch': 0.87}

  9%|▊         | 3592/41250 [8:40:04<90:32:54,  8.66s/it][2025-04-26 12:48:42,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:48:42,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2938.55 | bwd_microstep: 5867.73 | bwd_inner_microstep: 5854.69 | bwd_allreduce_microstep: 13.00 | step_microstep: 18.09
[2025-04-26 12:48:42,029] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2938.55 | bwd: 5867.74 | bwd_inner: 5854.69 | bwd_allreduce: 13.02 | step: 18.10

  9%|▊         | 3593/41250 [8:40:13<91:14:15,  8.72s/it]
                                                         
{'loss': 0.2005, 'grad_norm': 2.3931891918182373, 'learning_rate': 3.965907149631036e-05, 'epoch': 0.87}

  9%|▊         | 3593/41250 [8:40:13<91:14:15,  8.72s/it][2025-04-26 12:48:50,613] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:48:50,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.21 | bwd_microstep: 5667.90 | bwd_inner_microstep: 5649.43 | bwd_allreduce_microstep: 18.41 | step_microstep: 18.83
[2025-04-26 12:48:50,614] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.21 | bwd: 5667.91 | bwd_inner: 5649.43 | bwd_allreduce: 18.44 | step: 18.84

  9%|▊         | 3594/41250 [8:40:21<90:48:21,  8.68s/it]
                                                         
{'loss': 0.0843, 'grad_norm': 0.8359384536743164, 'learning_rate': 3.965878272523941e-05, 'epoch': 0.87}

  9%|▊         | 3594/41250 [8:40:21<90:48:21,  8.68s/it][2025-04-26 12:48:59,253] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:48:59,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.98 | bwd_microstep: 5703.01 | bwd_inner_microstep: 5690.14 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.87
[2025-04-26 12:48:59,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.98 | bwd: 5703.02 | bwd_inner: 5690.14 | bwd_allreduce: 12.84 | step: 18.87

  9%|▊         | 3595/41250 [8:40:30<90:40:28,  8.67s/it]
                                                         
{'loss': 0.1234, 'grad_norm': 1.9477595090866089, 'learning_rate': 3.965849383297594e-05, 'epoch': 0.87}

  9%|▊         | 3595/41250 [8:40:30<90:40:28,  8.67s/it][2025-04-26 12:49:07,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:49:07,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2884.38 | bwd_microstep: 5763.64 | bwd_inner_microstep: 5750.93 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.91
[2025-04-26 12:49:07,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2884.38 | bwd: 5763.65 | bwd_inner: 5750.93 | bwd_allreduce: 12.68 | step: 18.91

  9%|▊         | 3596/41250 [8:40:39<90:52:10,  8.69s/it]
                                                         
{'loss': 0.2857, 'grad_norm': 2.0374672412872314, 'learning_rate': 3.965820481952172e-05, 'epoch': 0.87}

  9%|▊         | 3596/41250 [8:40:39<90:52:10,  8.69s/it][2025-04-26 12:49:16,657] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:49:16,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.11 | bwd_microstep: 5740.07 | bwd_inner_microstep: 5648.77 | bwd_allreduce_microstep: 91.25 | step_microstep: 17.95
[2025-04-26 12:49:16,658] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.11 | bwd: 5740.08 | bwd_inner: 5648.77 | bwd_allreduce: 91.27 | step: 17.95

  9%|▊         | 3597/41250 [8:40:47<90:49:22,  8.68s/it]
                                                         
{'loss': 0.1318, 'grad_norm': 1.2080806493759155, 'learning_rate': 3.965791568487853e-05, 'epoch': 0.87}

  9%|▊         | 3597/41250 [8:40:47<90:49:22,  8.68s/it][2025-04-26 12:49:25,332] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:49:25,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.45 | bwd_microstep: 5756.85 | bwd_inner_microstep: 5675.77 | bwd_allreduce_microstep: 81.04 | step_microstep: 18.49
[2025-04-26 12:49:25,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.45 | bwd: 5756.86 | bwd_inner: 5675.77 | bwd_allreduce: 81.05 | step: 18.49

  9%|▊         | 3598/41250 [8:40:56<90:47:07,  8.68s/it]
                                                         
{'loss': 0.2532, 'grad_norm': 1.5402425527572632, 'learning_rate': 3.9657626429048164e-05, 'epoch': 0.87}

  9%|▊         | 3598/41250 [8:40:56<90:47:07,  8.68s/it][2025-04-26 12:49:34,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 12:49:34,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.21 | bwd_microstep: 5865.87 | bwd_inner_microstep: 5718.16 | bwd_allreduce_microstep: 147.66 | step_microstep: 18.24
[2025-04-26 12:49:34,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.21 | bwd: 5865.88 | bwd_inner: 5718.16 | bwd_allreduce: 147.68 | step: 18.24

  9%|▊         | 3599/41250 [8:41:05<91:10:40,  8.72s/it]
                                                         
{'loss': 0.2302, 'grad_norm': 1.198240876197815, 'learning_rate': 3.965733705203239e-05, 'epoch': 0.87}

  9%|▊         | 3599/41250 [8:41:05<91:10:40,  8.72s/it][2025-04-26 12:49:42,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:49:42,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.64 | bwd_microstep: 5744.64 | bwd_inner_microstep: 5704.81 | bwd_allreduce_microstep: 39.79 | step_microstep: 18.14
[2025-04-26 12:49:42,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.64 | bwd: 5744.66 | bwd_inner: 5704.81 | bwd_allreduce: 39.81 | step: 18.14

  9%|▊         | 3600/41250 [8:41:14<91:03:30,  8.71s/it]
                                                         
{'loss': 0.2513, 'grad_norm': 2.077467918395996, 'learning_rate': 3.9657047553833006e-05, 'epoch': 0.87}

  9%|▊         | 3600/41250 [8:41:14<91:03:30,  8.71s/it][2025-04-26 12:49:51,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:49:51,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.92 | bwd_microstep: 5715.47 | bwd_inner_microstep: 5702.85 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.35
[2025-04-26 12:49:51,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.92 | bwd: 5715.48 | bwd_inner: 5702.85 | bwd_allreduce: 12.59 | step: 18.35

  9%|▊         | 3601/41250 [8:41:22<90:51:39,  8.69s/it]
                                                         
{'loss': 0.2391, 'grad_norm': 4.4136457443237305, 'learning_rate': 3.9656757934451794e-05, 'epoch': 0.87}

  9%|▊         | 3601/41250 [8:41:22<90:51:39,  8.69s/it][2025-04-26 12:50:00,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:50:00,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.95 | bwd_microstep: 5708.57 | bwd_inner_microstep: 5695.87 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.16
[2025-04-26 12:50:00,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.95 | bwd: 5708.59 | bwd_inner: 5695.87 | bwd_allreduce: 12.68 | step: 18.16

  9%|▊         | 3602/41250 [8:41:31<90:40:07,  8.67s/it]
                                                         
{'loss': 0.1174, 'grad_norm': 1.8803530931472778, 'learning_rate': 3.9656468193890525e-05, 'epoch': 0.87}

  9%|▊         | 3602/41250 [8:41:31<90:40:07,  8.67s/it][2025-04-26 12:50:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:50:08,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.72 | bwd_microstep: 5714.37 | bwd_inner_microstep: 5701.67 | bwd_allreduce_microstep: 12.66 | step_microstep: 17.86
[2025-04-26 12:50:08,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.72 | bwd: 5714.38 | bwd_inner: 5701.67 | bwd_allreduce: 12.68 | step: 17.86

  9%|▊         | 3603/41250 [8:41:39<90:35:43,  8.66s/it]
                                                         
{'loss': 0.1107, 'grad_norm': 1.651233434677124, 'learning_rate': 3.9656178332151e-05, 'epoch': 0.87}

  9%|▊         | 3603/41250 [8:41:39<90:35:43,  8.66s/it][2025-04-26 12:50:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:50:17,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.55 | bwd_microstep: 5703.08 | bwd_inner_microstep: 5665.36 | bwd_allreduce_microstep: 37.67 | step_microstep: 18.44
[2025-04-26 12:50:17,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.55 | bwd: 5703.09 | bwd_inner: 5665.36 | bwd_allreduce: 37.69 | step: 18.44

  9%|▊         | 3604/41250 [8:41:48<90:29:02,  8.65s/it]
                                                         
{'loss': 0.2008, 'grad_norm': 2.092782974243164, 'learning_rate': 3.9655888349235e-05, 'epoch': 0.87}

  9%|▊         | 3604/41250 [8:41:48<90:29:02,  8.65s/it][2025-04-26 12:50:26,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:50:26,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.99 | bwd_microstep: 5769.12 | bwd_inner_microstep: 5722.28 | bwd_allreduce_microstep: 46.79 | step_microstep: 17.88
[2025-04-26 12:50:26,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.99 | bwd: 5769.13 | bwd_inner: 5722.28 | bwd_allreduce: 46.81 | step: 17.89

  9%|▊         | 3605/41250 [8:41:57<90:39:45,  8.67s/it]
                                                         
{'loss': 0.1913, 'grad_norm': 2.1455583572387695, 'learning_rate': 3.965559824514432e-05, 'epoch': 0.87}

  9%|▊         | 3605/41250 [8:41:57<90:39:45,  8.67s/it][2025-04-26 12:50:34,731] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.18 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:50:34,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.15 | bwd_microstep: 5712.38 | bwd_inner_microstep: 5699.63 | bwd_allreduce_microstep: 12.71 | step_microstep: 17.86
[2025-04-26 12:50:34,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.15 | bwd: 5712.40 | bwd_inner: 5699.64 | bwd_allreduce: 12.72 | step: 17.86

  9%|▊         | 3606/41250 [8:42:05<90:36:30,  8.67s/it]
                                                         
{'loss': 0.0434, 'grad_norm': 0.6381401419639587, 'learning_rate': 3.965530801988074e-05, 'epoch': 0.87}

  9%|▊         | 3606/41250 [8:42:05<90:36:30,  8.67s/it][2025-04-26 12:50:43,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 12:50:43,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.65 | bwd_microstep: 5772.57 | bwd_inner_microstep: 5700.77 | bwd_allreduce_microstep: 71.75 | step_microstep: 19.06
[2025-04-26 12:50:43,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.65 | bwd: 5772.58 | bwd_inner: 5700.77 | bwd_allreduce: 71.77 | step: 19.07

  9%|▊         | 3607/41250 [8:42:14<90:44:08,  8.68s/it]
                                                         
{'loss': 0.1097, 'grad_norm': 1.4569236040115356, 'learning_rate': 3.9655017673446046e-05, 'epoch': 0.87}

  9%|▊         | 3607/41250 [8:42:14<90:44:08,  8.68s/it][2025-04-26 12:50:52,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:50:52,266] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.14 | bwd_microstep: 5906.31 | bwd_inner_microstep: 5664.95 | bwd_allreduce_microstep: 241.32 | step_microstep: 17.85
[2025-04-26 12:50:52,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.14 | bwd: 5906.32 | bwd_inner: 5664.95 | bwd_allreduce: 241.33 | step: 17.86

  9%|▊         | 3608/41250 [8:42:23<91:12:15,  8.72s/it]
                                                         
{'loss': 0.0806, 'grad_norm': 2.2688090801239014, 'learning_rate': 3.965472720584204e-05, 'epoch': 0.87}

  9%|▊         | 3608/41250 [8:42:23<91:12:15,  8.72s/it][2025-04-26 12:51:00,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:51:00,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.50 | bwd_microstep: 5705.31 | bwd_inner_microstep: 5669.36 | bwd_allreduce_microstep: 35.90 | step_microstep: 18.61
[2025-04-26 12:51:00,892] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.50 | bwd: 5705.32 | bwd_inner: 5669.36 | bwd_allreduce: 35.91 | step: 18.61

  9%|▊         | 3609/41250 [8:42:32<90:54:17,  8.69s/it]
                                                         
{'loss': 0.2343, 'grad_norm': 1.5291045904159546, 'learning_rate': 3.965443661707051e-05, 'epoch': 0.87}

  9%|▊         | 3609/41250 [8:42:32<90:54:17,  8.69s/it][2025-04-26 12:51:09,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.07 | optimizer_step: 0.90
[2025-04-26 12:51:09,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.79 | bwd_microstep: 5725.51 | bwd_inner_microstep: 5649.37 | bwd_allreduce_microstep: 76.09 | step_microstep: 18.79
[2025-04-26 12:51:09,529] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.79 | bwd: 5725.53 | bwd_inner: 5649.37 | bwd_allreduce: 76.11 | step: 18.80

  9%|▉         | 3610/41250 [8:42:40<90:43:33,  8.68s/it]
                                                         
{'loss': 0.2631, 'grad_norm': 2.5069692134857178, 'learning_rate': 3.9654145907133235e-05, 'epoch': 0.88}

  9%|▉         | 3610/41250 [8:42:40<90:43:33,  8.68s/it][2025-04-26 12:51:18,199] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.01 | optimizer_gradients: 1.06 | optimizer_step: 1.23
[2025-04-26 12:51:18,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.97 | bwd_microstep: 5736.91 | bwd_inner_microstep: 5721.92 | bwd_allreduce_microstep: 14.94 | step_microstep: 19.56
[2025-04-26 12:51:18,200] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.97 | bwd: 5736.92 | bwd_inner: 5721.92 | bwd_allreduce: 14.96 | step: 19.56

  9%|▉         | 3611/41250 [8:42:49<90:42:34,  8.68s/it]
                                                         
{'loss': 0.1015, 'grad_norm': 0.9709692597389221, 'learning_rate': 3.965385507603202e-05, 'epoch': 0.88}

  9%|▉         | 3611/41250 [8:42:49<90:42:34,  8.68s/it][2025-04-26 12:51:26,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 0.93
[2025-04-26 12:51:26,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.49 | bwd_microstep: 5729.27 | bwd_inner_microstep: 5716.51 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.87
[2025-04-26 12:51:26,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.50 | bwd: 5729.28 | bwd_inner: 5716.51 | bwd_allreduce: 12.73 | step: 18.87

  9%|▉         | 3612/41250 [8:42:58<90:42:30,  8.68s/it]
                                                         
{'loss': 0.189, 'grad_norm': 2.7085506916046143, 'learning_rate': 3.9653564123768655e-05, 'epoch': 0.88}

  9%|▉         | 3612/41250 [8:42:58<90:42:30,  8.68s/it][2025-04-26 12:51:35,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:51:35,527] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.86 | bwd_microstep: 5714.98 | bwd_inner_microstep: 5700.95 | bwd_allreduce_microstep: 13.98 | step_microstep: 18.35
[2025-04-26 12:51:35,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.86 | bwd: 5714.99 | bwd_inner: 5700.95 | bwd_allreduce: 14.00 | step: 18.35

  9%|▉         | 3613/41250 [8:43:06<90:36:55,  8.67s/it]
                                                         
{'loss': 0.2802, 'grad_norm': 2.003708839416504, 'learning_rate': 3.9653273050344936e-05, 'epoch': 0.88}

  9%|▉         | 3613/41250 [8:43:06<90:36:55,  8.67s/it][2025-04-26 12:51:44,297] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 12:51:44,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.21 | bwd_microstep: 5794.76 | bwd_inner_microstep: 5781.40 | bwd_allreduce_microstep: 13.31 | step_microstep: 18.79
[2025-04-26 12:51:44,298] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.21 | bwd: 5794.77 | bwd_inner: 5781.40 | bwd_allreduce: 13.33 | step: 18.79

  9%|▉         | 3614/41250 [8:43:15<90:56:15,  8.70s/it]
                                                         
{'loss': 0.1497, 'grad_norm': 1.5520535707473755, 'learning_rate': 3.965298185576265e-05, 'epoch': 0.88}

  9%|▉         | 3614/41250 [8:43:15<90:56:15,  8.70s/it][2025-04-26 12:51:52,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 12:51:52,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.02 | bwd_microstep: 5713.91 | bwd_inner_microstep: 5647.11 | bwd_allreduce_microstep: 66.76 | step_microstep: 18.79
[2025-04-26 12:51:52,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.02 | bwd: 5713.93 | bwd_inner: 5647.11 | bwd_allreduce: 66.78 | step: 18.79

  9%|▉         | 3615/41250 [8:43:24<90:42:11,  8.68s/it]
                                                         
{'loss': 0.261, 'grad_norm': 2.063485860824585, 'learning_rate': 3.965269054002359e-05, 'epoch': 0.88}

  9%|▉         | 3615/41250 [8:43:24<90:42:11,  8.68s/it][2025-04-26 12:52:01,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 12:52:01,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.08 | bwd_microstep: 5779.32 | bwd_inner_microstep: 5657.79 | bwd_allreduce_microstep: 121.48 | step_microstep: 18.72
[2025-04-26 12:52:01,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.08 | bwd: 5779.34 | bwd_inner: 5657.79 | bwd_allreduce: 121.50 | step: 18.73

  9%|▉         | 3616/41250 [8:43:32<90:42:46,  8.68s/it]
                                                         
{'loss': 0.2349, 'grad_norm': 3.7484652996063232, 'learning_rate': 3.965239910312957e-05, 'epoch': 0.88}

  9%|▉         | 3616/41250 [8:43:32<90:42:46,  8.68s/it][2025-04-26 12:52:10,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 12:52:10,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.35 | bwd_microstep: 5775.58 | bwd_inner_microstep: 5647.35 | bwd_allreduce_microstep: 128.19 | step_microstep: 18.95
[2025-04-26 12:52:10,287] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.35 | bwd: 5775.60 | bwd_inner: 5647.34 | bwd_allreduce: 128.21 | step: 18.96

  9%|▉         | 3617/41250 [8:43:41<90:43:52,  8.68s/it]
                                                         
{'loss': 0.2898, 'grad_norm': 2.0135865211486816, 'learning_rate': 3.9652107545082376e-05, 'epoch': 0.88}

  9%|▉         | 3617/41250 [8:43:41<90:43:52,  8.68s/it][2025-04-26 12:52:18,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:52:18,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.04 | bwd_microstep: 5765.36 | bwd_inner_microstep: 5717.44 | bwd_allreduce_microstep: 47.88 | step_microstep: 18.56
[2025-04-26 12:52:18,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.04 | bwd: 5765.38 | bwd_inner: 5717.44 | bwd_allreduce: 47.90 | step: 18.56

  9%|▉         | 3618/41250 [8:43:50<90:48:26,  8.69s/it]
                                                         
{'loss': 0.1014, 'grad_norm': 1.3246794939041138, 'learning_rate': 3.9651815865883794e-05, 'epoch': 0.88}

  9%|▉         | 3618/41250 [8:43:50<90:48:26,  8.69s/it][2025-04-26 12:52:27,695] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.91
[2025-04-26 12:52:27,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.16 | bwd_microstep: 5789.31 | bwd_inner_microstep: 5663.00 | bwd_allreduce_microstep: 126.27 | step_microstep: 18.72
[2025-04-26 12:52:27,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.16 | bwd: 5789.32 | bwd_inner: 5663.00 | bwd_allreduce: 126.28 | step: 18.72

  9%|▉         | 3619/41250 [8:43:58<90:51:31,  8.69s/it]
                                                         
{'loss': 0.1468, 'grad_norm': 1.6353139877319336, 'learning_rate': 3.965152406553564e-05, 'epoch': 0.88}

  9%|▉         | 3619/41250 [8:43:58<90:51:31,  8.69s/it][2025-04-26 12:52:36,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:52:36,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.93 | bwd_microstep: 5715.55 | bwd_inner_microstep: 5648.09 | bwd_allreduce_microstep: 67.42 | step_microstep: 18.85
[2025-04-26 12:52:36,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.94 | bwd: 5715.57 | bwd_inner: 5648.09 | bwd_allreduce: 67.43 | step: 18.86

  9%|▉         | 3620/41250 [8:44:07<90:36:20,  8.67s/it]
                                                         
{'loss': 0.0767, 'grad_norm': 0.7133233547210693, 'learning_rate': 3.96512321440397e-05, 'epoch': 0.88}

  9%|▉         | 3620/41250 [8:44:07<90:36:20,  8.67s/it][2025-04-26 12:52:44,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:52:44,942] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.59 | bwd_microstep: 5704.96 | bwd_inner_microstep: 5692.14 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.32
[2025-04-26 12:52:44,943] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.59 | bwd: 5704.98 | bwd_inner: 5692.14 | bwd_allreduce: 12.79 | step: 18.32

  9%|▉         | 3621/41250 [8:44:16<90:29:58,  8.66s/it]
                                                         
{'loss': 0.1156, 'grad_norm': 0.8640238046646118, 'learning_rate': 3.9650940101397785e-05, 'epoch': 0.88}

  9%|▉         | 3621/41250 [8:44:16<90:29:58,  8.66s/it][2025-04-26 12:52:53,566] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:52:53,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.62 | bwd_microstep: 5718.98 | bwd_inner_microstep: 5648.36 | bwd_allreduce_microstep: 70.57 | step_microstep: 18.52
[2025-04-26 12:52:53,567] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.62 | bwd: 5719.00 | bwd_inner: 5648.36 | bwd_allreduce: 70.59 | step: 18.52

  9%|▉         | 3622/41250 [8:44:24<90:23:30,  8.65s/it]
                                                         
{'loss': 0.0839, 'grad_norm': 0.615356981754303, 'learning_rate': 3.965064793761169e-05, 'epoch': 0.88}

  9%|▉         | 3622/41250 [8:44:24<90:23:30,  8.65s/it][2025-04-26 12:53:02,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.98 | optimizer_step: 0.95
[2025-04-26 12:53:02,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.61 | bwd_microstep: 5704.47 | bwd_inner_microstep: 5639.17 | bwd_allreduce_microstep: 65.25 | step_microstep: 18.48
[2025-04-26 12:53:02,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.61 | bwd: 5704.48 | bwd_inner: 5639.17 | bwd_allreduce: 65.27 | step: 18.48

  9%|▉         | 3623/41250 [8:44:33<90:15:40,  8.64s/it]
                                                         
{'loss': 0.1721, 'grad_norm': 2.157346248626709, 'learning_rate': 3.965035565268321e-05, 'epoch': 0.88}

  9%|▉         | 3623/41250 [8:44:33<90:15:40,  8.64s/it][2025-04-26 12:53:10,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 12:53:10,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.84 | bwd_microstep: 5792.38 | bwd_inner_microstep: 5779.54 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.77
[2025-04-26 12:53:10,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.84 | bwd: 5792.39 | bwd_inner: 5779.54 | bwd_allreduce: 12.81 | step: 18.77

  9%|▉         | 3624/41250 [8:44:42<90:41:09,  8.68s/it]
                                                         
{'loss': 0.1006, 'grad_norm': 1.1013916730880737, 'learning_rate': 3.965006324661415e-05, 'epoch': 0.88}

  9%|▉         | 3624/41250 [8:44:42<90:41:09,  8.68s/it][2025-04-26 12:53:19,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.32 | optimizer_step: 1.03
[2025-04-26 12:53:19,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.31 | bwd_microstep: 5708.93 | bwd_inner_microstep: 5695.25 | bwd_allreduce_microstep: 13.62 | step_microstep: 19.87
[2025-04-26 12:53:19,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.31 | bwd: 5708.94 | bwd_inner: 5695.25 | bwd_allreduce: 13.64 | step: 19.88

  9%|▉         | 3625/41250 [8:44:50<90:32:43,  8.66s/it]
                                                         
{'loss': 0.3246, 'grad_norm': 2.6050713062286377, 'learning_rate': 3.9649770719406325e-05, 'epoch': 0.88}

  9%|▉         | 3625/41250 [8:44:50<90:32:43,  8.66s/it][2025-04-26 12:53:28,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.02 | optimizer_step: 0.93
[2025-04-26 12:53:28,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5753.03 | bwd_inner_microstep: 5698.66 | bwd_allreduce_microstep: 54.32 | step_microstep: 18.93
[2025-04-26 12:53:28,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5753.05 | bwd_inner: 5698.66 | bwd_allreduce: 54.35 | step: 18.93

  9%|▉         | 3626/41250 [8:44:59<90:36:36,  8.67s/it]
                                                         
{'loss': 0.1434, 'grad_norm': 2.630709409713745, 'learning_rate': 3.964947807106152e-05, 'epoch': 0.88}

  9%|▉         | 3626/41250 [8:44:59<90:36:36,  8.67s/it][2025-04-26 12:53:37,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 12:53:37,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.42 | bwd_microstep: 6042.05 | bwd_inner_microstep: 5639.38 | bwd_allreduce_microstep: 402.62 | step_microstep: 19.10
[2025-04-26 12:53:37,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.42 | bwd: 6042.06 | bwd_inner: 5639.38 | bwd_allreduce: 402.64 | step: 19.10

  9%|▉         | 3627/41250 [8:45:08<91:28:41,  8.75s/it]
                                                         
{'loss': 0.0432, 'grad_norm': 1.2712669372558594, 'learning_rate': 3.964918530158155e-05, 'epoch': 0.88}

  9%|▉         | 3627/41250 [8:45:08<91:28:41,  8.75s/it][2025-04-26 12:53:45,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:53:45,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.47 | bwd_microstep: 5688.62 | bwd_inner_microstep: 5640.66 | bwd_allreduce_microstep: 47.91 | step_microstep: 18.46
[2025-04-26 12:53:45,804] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.47 | bwd: 5688.63 | bwd_inner: 5640.66 | bwd_allreduce: 47.93 | step: 18.47

  9%|▉         | 3628/41250 [8:45:16<90:58:13,  8.70s/it]
                                                         
{'loss': 0.1963, 'grad_norm': 2.0487890243530273, 'learning_rate': 3.964889241096821e-05, 'epoch': 0.88}

  9%|▉         | 3628/41250 [8:45:16<90:58:13,  8.70s/it][2025-04-26 12:53:54,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:53:54,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.79 | bwd_microstep: 5757.64 | bwd_inner_microstep: 5635.92 | bwd_allreduce_microstep: 121.68 | step_microstep: 18.40
[2025-04-26 12:53:54,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.79 | bwd: 5757.66 | bwd_inner: 5635.92 | bwd_allreduce: 121.70 | step: 18.40

  9%|▉         | 3629/41250 [8:45:25<90:47:29,  8.69s/it]
                                                         
{'loss': 0.2011, 'grad_norm': 1.5700080394744873, 'learning_rate': 3.964859939922333e-05, 'epoch': 0.88}

  9%|▉         | 3629/41250 [8:45:25<90:47:29,  8.69s/it][2025-04-26 12:54:03,117] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:54:03,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.03 | bwd_microstep: 5731.48 | bwd_inner_microstep: 5690.55 | bwd_allreduce_microstep: 40.89 | step_microstep: 18.68
[2025-04-26 12:54:03,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.03 | bwd: 5731.50 | bwd_inner: 5690.55 | bwd_allreduce: 40.91 | step: 18.68

  9%|▉         | 3630/41250 [8:45:34<90:43:01,  8.68s/it]
                                                         
{'loss': 0.1154, 'grad_norm': 1.1562426090240479, 'learning_rate': 3.964830626634869e-05, 'epoch': 0.88}

  9%|▉         | 3630/41250 [8:45:34<90:43:01,  8.68s/it][2025-04-26 12:54:11,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:54:11,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.70 | bwd_microstep: 5871.33 | bwd_inner_microstep: 5635.97 | bwd_allreduce_microstep: 235.32 | step_microstep: 18.62
[2025-04-26 12:54:11,908] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.70 | bwd: 5871.35 | bwd_inner: 5635.97 | bwd_allreduce: 235.33 | step: 18.62

  9%|▉         | 3631/41250 [8:45:43<91:03:22,  8.71s/it]
                                                         
{'loss': 0.1168, 'grad_norm': 1.1557261943817139, 'learning_rate': 3.964801301234611e-05, 'epoch': 0.88}

  9%|▉         | 3631/41250 [8:45:43<91:03:22,  8.71s/it][2025-04-26 12:54:20,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.13 | optimizer_step: 1.24
[2025-04-26 12:54:20,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.15 | bwd_microstep: 5690.96 | bwd_inner_microstep: 5637.67 | bwd_allreduce_microstep: 53.22 | step_microstep: 20.04
[2025-04-26 12:54:20,504] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.16 | bwd: 5690.97 | bwd_inner: 5637.67 | bwd_allreduce: 53.25 | step: 20.04

  9%|▉         | 3632/41250 [8:45:51<90:41:24,  8.68s/it]
                                                         
{'loss': 0.1895, 'grad_norm': 2.189404249191284, 'learning_rate': 3.9647719637217386e-05, 'epoch': 0.88}

  9%|▉         | 3632/41250 [8:45:51<90:41:24,  8.68s/it][2025-04-26 12:54:29,393] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.01 | optimizer_step: 1.03
[2025-04-26 12:54:29,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2933.14 | bwd_microstep: 5868.63 | bwd_inner_microstep: 5855.55 | bwd_allreduce_microstep: 13.04 | step_microstep: 18.87
[2025-04-26 12:54:29,394] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2933.14 | bwd: 5868.65 | bwd_inner: 5855.55 | bwd_allreduce: 13.06 | step: 18.87

  9%|▉         | 3633/41250 [8:46:00<91:20:49,  8.74s/it]
                                                         
{'loss': 0.2025, 'grad_norm': 1.4896928071975708, 'learning_rate': 3.9647426140964335e-05, 'epoch': 0.88}

  9%|▉         | 3633/41250 [8:46:00<91:20:49,  8.74s/it][2025-04-26 12:54:38,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.17 | optimizer_step: 0.90
[2025-04-26 12:54:38,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.69 | bwd_microstep: 5743.94 | bwd_inner_microstep: 5702.24 | bwd_allreduce_microstep: 41.65 | step_microstep: 19.34
[2025-04-26 12:54:38,061] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.69 | bwd: 5743.96 | bwd_inner: 5702.24 | bwd_allreduce: 41.67 | step: 19.35

  9%|▉         | 3634/41250 [8:46:09<91:06:35,  8.72s/it]
                                                         
{'loss': 0.1181, 'grad_norm': 1.1303198337554932, 'learning_rate': 3.964713252358877e-05, 'epoch': 0.88}

  9%|▉         | 3634/41250 [8:46:09<91:06:35,  8.72s/it][2025-04-26 12:54:46,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 12:54:46,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.73 | bwd_microstep: 5693.13 | bwd_inner_microstep: 5655.52 | bwd_allreduce_microstep: 37.56 | step_microstep: 18.76
[2025-04-26 12:54:46,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.73 | bwd: 5693.15 | bwd_inner: 5655.52 | bwd_allreduce: 37.58 | step: 18.76

  9%|▉         | 3635/41250 [8:46:17<90:44:25,  8.68s/it]
                                                         
{'loss': 0.1641, 'grad_norm': 2.3217451572418213, 'learning_rate': 3.96468387850925e-05, 'epoch': 0.88}

  9%|▉         | 3635/41250 [8:46:17<90:44:25,  8.68s/it][2025-04-26 12:54:55,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 12:54:55,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.47 | bwd_microstep: 5780.56 | bwd_inner_microstep: 5767.48 | bwd_allreduce_microstep: 13.02 | step_microstep: 18.81
[2025-04-26 12:54:55,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.47 | bwd: 5780.59 | bwd_inner: 5767.48 | bwd_allreduce: 13.05 | step: 18.81

  9%|▉         | 3636/41250 [8:46:26<90:59:08,  8.71s/it]
                                                         
{'loss': 0.1578, 'grad_norm': 1.9558831453323364, 'learning_rate': 3.9646544925477334e-05, 'epoch': 0.88}

  9%|▉         | 3636/41250 [8:46:26<90:59:08,  8.71s/it][2025-04-26 12:55:04,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.32 | optimizer_step: 1.02
[2025-04-26 12:55:04,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.95 | bwd_microstep: 5683.17 | bwd_inner_microstep: 5652.46 | bwd_allreduce_microstep: 30.65 | step_microstep: 19.75
[2025-04-26 12:55:04,020] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.95 | bwd: 5683.18 | bwd_inner: 5652.46 | bwd_allreduce: 30.68 | step: 19.75

  9%|▉         | 3637/41250 [8:46:35<90:37:08,  8.67s/it]
                                                         
{'loss': 0.0748, 'grad_norm': 0.890609622001648, 'learning_rate': 3.9646250944745086e-05, 'epoch': 0.88}

  9%|▉         | 3637/41250 [8:46:35<90:37:08,  8.67s/it][2025-04-26 12:55:12,662] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.05 | optimizer_step: 0.92
[2025-04-26 12:55:12,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.90 | bwd_microstep: 5750.20 | bwd_inner_microstep: 5649.05 | bwd_allreduce_microstep: 101.10 | step_microstep: 18.78
[2025-04-26 12:55:12,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.90 | bwd: 5750.21 | bwd_inner: 5649.05 | bwd_allreduce: 101.12 | step: 18.78

  9%|▉         | 3638/41250 [8:46:43<90:31:21,  8.66s/it]
                                                         
{'loss': 0.1715, 'grad_norm': 1.0807138681411743, 'learning_rate': 3.9645956842897555e-05, 'epoch': 0.88}

  9%|▉         | 3638/41250 [8:46:43<90:31:21,  8.66s/it][2025-04-26 12:55:21,400] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:55:21,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.11 | bwd_microstep: 5767.75 | bwd_inner_microstep: 5754.88 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.87
[2025-04-26 12:55:21,401] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.11 | bwd: 5767.77 | bwd_inner: 5754.88 | bwd_allreduce: 12.85 | step: 18.86

  9%|▉         | 3639/41250 [8:46:52<90:45:01,  8.69s/it]
                                                         
{'loss': 0.1389, 'grad_norm': 0.9437944293022156, 'learning_rate': 3.964566261993657e-05, 'epoch': 0.88}

  9%|▉         | 3639/41250 [8:46:52<90:45:01,  8.69s/it][2025-04-26 12:55:30,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 12:55:30,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.13 | bwd_microstep: 5748.88 | bwd_inner_microstep: 5682.05 | bwd_allreduce_microstep: 66.78 | step_microstep: 18.42
[2025-04-26 12:55:30,076] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.13 | bwd: 5748.89 | bwd_inner: 5682.05 | bwd_allreduce: 66.80 | step: 18.42

  9%|▉         | 3640/41250 [8:47:01<90:42:40,  8.68s/it]
                                                         
{'loss': 0.2331, 'grad_norm': 2.0971977710723877, 'learning_rate': 3.964536827586393e-05, 'epoch': 0.88}

  9%|▉         | 3640/41250 [8:47:01<90:42:40,  8.68s/it][2025-04-26 12:55:38,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 12:55:38,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.81 | bwd_microstep: 5777.10 | bwd_inner_microstep: 5687.99 | bwd_allreduce_microstep: 89.06 | step_microstep: 18.33
[2025-04-26 12:55:38,775] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.81 | bwd: 5777.11 | bwd_inner: 5687.99 | bwd_allreduce: 89.08 | step: 18.33

  9%|▉         | 3641/41250 [8:47:09<90:45:40,  8.69s/it]
                                                         
{'loss': 0.1549, 'grad_norm': 1.0073835849761963, 'learning_rate': 3.9645073810681475e-05, 'epoch': 0.88}

  9%|▉         | 3641/41250 [8:47:09<90:45:40,  8.69s/it][2025-04-26 12:55:47,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.93 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:55:47,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.00 | bwd_microstep: 5795.32 | bwd_inner_microstep: 5782.55 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.03
[2025-04-26 12:55:47,538] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.00 | bwd: 5795.33 | bwd_inner: 5782.55 | bwd_allreduce: 12.74 | step: 18.03

  9%|▉         | 3642/41250 [8:47:18<90:59:49,  8.71s/it]
                                                         
{'loss': 0.1817, 'grad_norm': 1.4615908861160278, 'learning_rate': 3.9644779224390994e-05, 'epoch': 0.88}

  9%|▉         | 3642/41250 [8:47:18<90:59:49,  8.71s/it][2025-04-26 12:55:56,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.20 | optimizer_gradients: 1.06 | optimizer_step: 1.22
[2025-04-26 12:55:56,163] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.89 | bwd_microstep: 5703.80 | bwd_inner_microstep: 5690.12 | bwd_allreduce_microstep: 13.63 | step_microstep: 19.65
[2025-04-26 12:55:56,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.89 | bwd: 5703.82 | bwd_inner: 5690.12 | bwd_allreduce: 13.65 | step: 19.65

  9%|▉         | 3643/41250 [8:47:27<90:43:47,  8.69s/it]
                                                         
{'loss': 0.4484, 'grad_norm': 2.2988553047180176, 'learning_rate': 3.964448451699432e-05, 'epoch': 0.88}

  9%|▉         | 3643/41250 [8:47:27<90:43:47,  8.69s/it][2025-04-26 12:56:04,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 12:56:04,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.83 | bwd_microstep: 5765.08 | bwd_inner_microstep: 5652.40 | bwd_allreduce_microstep: 112.64 | step_microstep: 19.08
[2025-04-26 12:56:04,836] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.83 | bwd: 5765.10 | bwd_inner: 5652.40 | bwd_allreduce: 112.66 | step: 19.08

  9%|▉         | 3644/41250 [8:47:36<90:41:03,  8.68s/it]
                                                         
{'loss': 0.17, 'grad_norm': 1.4315887689590454, 'learning_rate': 3.964418968849325e-05, 'epoch': 0.88}

  9%|▉         | 3644/41250 [8:47:36<90:41:03,  8.68s/it][2025-04-26 12:56:13,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.98 | optimizer_step: 0.93
[2025-04-26 12:56:13,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.70 | bwd_microstep: 5745.16 | bwd_inner_microstep: 5682.80 | bwd_allreduce_microstep: 62.32 | step_microstep: 18.54
[2025-04-26 12:56:13,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.70 | bwd: 5745.18 | bwd_inner: 5682.80 | bwd_allreduce: 62.33 | step: 18.54

  9%|▉         | 3645/41250 [8:47:44<90:39:47,  8.68s/it]
                                                         
{'loss': 0.0726, 'grad_norm': 1.1402463912963867, 'learning_rate': 3.964389473888962e-05, 'epoch': 0.88}

  9%|▉         | 3645/41250 [8:47:44<90:39:47,  8.68s/it][2025-04-26 12:56:22,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.02 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:56:22,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.31 | bwd_microstep: 5722.88 | bwd_inner_microstep: 5659.33 | bwd_allreduce_microstep: 63.51 | step_microstep: 18.02
[2025-04-26 12:56:22,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.31 | bwd: 5722.90 | bwd_inner: 5659.33 | bwd_allreduce: 63.53 | step: 18.03

  9%|▉         | 3646/41250 [8:47:53<90:29:58,  8.66s/it]
                                                         
{'loss': 0.3171, 'grad_norm': 2.121727228164673, 'learning_rate': 3.9643599668185246e-05, 'epoch': 0.88}

  9%|▉         | 3646/41250 [8:47:53<90:29:58,  8.66s/it][2025-04-26 12:56:30,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 12:56:30,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.65 | bwd_microstep: 5755.45 | bwd_inner_microstep: 5709.62 | bwd_allreduce_microstep: 45.79 | step_microstep: 18.63
[2025-04-26 12:56:30,822] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.65 | bwd: 5755.46 | bwd_inner: 5709.62 | bwd_allreduce: 45.80 | step: 18.64

  9%|▉         | 3647/41250 [8:48:02<90:33:29,  8.67s/it]
                                                         
{'loss': 0.1, 'grad_norm': 0.8721223473548889, 'learning_rate': 3.9643304476381946e-05, 'epoch': 0.88}

  9%|▉         | 3647/41250 [8:48:02<90:33:29,  8.67s/it][2025-04-26 12:56:39,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:56:39,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.42 | bwd_microstep: 5774.64 | bwd_inner_microstep: 5697.24 | bwd_allreduce_microstep: 77.36 | step_microstep: 18.26
[2025-04-26 12:56:39,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.42 | bwd: 5774.65 | bwd_inner: 5697.24 | bwd_allreduce: 77.38 | step: 18.27

  9%|▉         | 3648/41250 [8:48:10<90:40:33,  8.68s/it]
                                                         
{'loss': 0.2492, 'grad_norm': 1.4699524641036987, 'learning_rate': 3.964300916348153e-05, 'epoch': 0.88}

  9%|▉         | 3648/41250 [8:48:10<90:40:33,  8.68s/it][2025-04-26 12:56:48,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:56:48,164] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.05 | bwd_microstep: 5712.36 | bwd_inner_microstep: 5653.74 | bwd_allreduce_microstep: 58.57 | step_microstep: 18.93
[2025-04-26 12:56:48,165] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.05 | bwd: 5712.37 | bwd_inner: 5653.74 | bwd_allreduce: 58.59 | step: 18.93

  9%|▉         | 3649/41250 [8:48:19<90:31:41,  8.67s/it]
                                                         
{'loss': 0.3286, 'grad_norm': 3.9610345363616943, 'learning_rate': 3.964271372948583e-05, 'epoch': 0.88}

  9%|▉         | 3649/41250 [8:48:19<90:31:41,  8.67s/it][2025-04-26 12:56:56,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 12:56:56,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.24 | bwd_microstep: 5748.47 | bwd_inner_microstep: 5719.08 | bwd_allreduce_microstep: 29.34 | step_microstep: 18.98
[2025-04-26 12:56:56,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.24 | bwd: 5748.48 | bwd_inner: 5719.08 | bwd_allreduce: 29.36 | step: 18.99

  9%|▉         | 3650/41250 [8:48:28<90:35:28,  8.67s/it]
                                                         
{'loss': 0.3849, 'grad_norm': 6.542270183563232, 'learning_rate': 3.9642418174396665e-05, 'epoch': 0.88}

  9%|▉         | 3650/41250 [8:48:28<90:35:28,  8.67s/it][2025-04-26 12:57:05,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 12:57:05,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.24 | bwd_microstep: 5760.96 | bwd_inner_microstep: 5671.12 | bwd_allreduce_microstep: 89.80 | step_microstep: 18.67
[2025-04-26 12:57:05,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.24 | bwd: 5760.98 | bwd_inner: 5671.12 | bwd_allreduce: 89.82 | step: 18.68

  9%|▉         | 3651/41250 [8:48:36<90:36:36,  8.68s/it]
                                                         
{'loss': 0.1738, 'grad_norm': 1.017675518989563, 'learning_rate': 3.9642122498215846e-05, 'epoch': 0.89}

  9%|▉         | 3651/41250 [8:48:36<90:36:36,  8.68s/it][2025-04-26 12:57:14,232] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 1.00
[2025-04-26 12:57:14,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.51 | bwd_microstep: 5781.81 | bwd_inner_microstep: 5670.22 | bwd_allreduce_microstep: 111.54 | step_microstep: 19.10
[2025-04-26 12:57:14,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.51 | bwd: 5781.82 | bwd_inner: 5670.22 | bwd_allreduce: 111.55 | step: 19.11

  9%|▉         | 3652/41250 [8:48:45<90:41:08,  8.68s/it]
                                                         
{'loss': 0.2291, 'grad_norm': 1.7564500570297241, 'learning_rate': 3.964182670094522e-05, 'epoch': 0.89}

  9%|▉         | 3652/41250 [8:48:45<90:41:08,  8.68s/it][2025-04-26 12:57:22,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 12:57:22,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.57 | bwd_microstep: 5713.46 | bwd_inner_microstep: 5695.38 | bwd_allreduce_microstep: 18.04 | step_microstep: 19.02
[2025-04-26 12:57:22,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.57 | bwd: 5713.48 | bwd_inner: 5695.38 | bwd_allreduce: 18.06 | step: 19.02

  9%|▉         | 3653/41250 [8:48:54<90:33:46,  8.67s/it]
                                                         
{'loss': 0.2059, 'grad_norm': 1.2785347700119019, 'learning_rate': 3.9641530782586586e-05, 'epoch': 0.89}

  9%|▉         | 3653/41250 [8:48:54<90:33:46,  8.67s/it][2025-04-26 12:57:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 12:57:31,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.45 | bwd_microstep: 5728.37 | bwd_inner_microstep: 5658.78 | bwd_allreduce_microstep: 69.54 | step_microstep: 18.74
[2025-04-26 12:57:31,523] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.45 | bwd: 5728.38 | bwd_inner: 5658.78 | bwd_allreduce: 69.56 | step: 18.75

  9%|▉         | 3654/41250 [8:49:02<90:28:22,  8.66s/it]
                                                         
{'loss': 0.2322, 'grad_norm': 1.2851516008377075, 'learning_rate': 3.9641234743141784e-05, 'epoch': 0.89}

  9%|▉         | 3654/41250 [8:49:02<90:28:22,  8.66s/it][2025-04-26 12:57:40,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:57:40,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.35 | bwd_microstep: 5758.14 | bwd_inner_microstep: 5726.06 | bwd_allreduce_microstep: 32.03 | step_microstep: 18.62
[2025-04-26 12:57:40,225] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.35 | bwd: 5758.15 | bwd_inner: 5726.06 | bwd_allreduce: 32.05 | step: 18.62

  9%|▉         | 3655/41250 [8:49:11<90:35:32,  8.67s/it]
                                                         
{'loss': 0.1618, 'grad_norm': 1.3923133611679077, 'learning_rate': 3.9640938582612625e-05, 'epoch': 0.89}

  9%|▉         | 3655/41250 [8:49:11<90:35:32,  8.67s/it][2025-04-26 12:57:48,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 12:57:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.33 | bwd_microstep: 5772.28 | bwd_inner_microstep: 5663.31 | bwd_allreduce_microstep: 108.93 | step_microstep: 18.58
[2025-04-26 12:57:48,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.33 | bwd: 5772.29 | bwd_inner: 5663.31 | bwd_allreduce: 108.94 | step: 18.58

  9%|▉         | 3656/41250 [8:49:20<90:36:34,  8.68s/it]
                                                         
{'loss': 0.2459, 'grad_norm': 1.5746830701828003, 'learning_rate': 3.9640642301000947e-05, 'epoch': 0.89}

  9%|▉         | 3656/41250 [8:49:20<90:36:34,  8.68s/it][2025-04-26 12:57:57,531] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 1.02
[2025-04-26 12:57:57,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.89 | bwd_microstep: 5712.37 | bwd_inner_microstep: 5663.66 | bwd_allreduce_microstep: 48.66 | step_microstep: 18.67
[2025-04-26 12:57:57,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.89 | bwd: 5712.38 | bwd_inner: 5663.66 | bwd_allreduce: 48.68 | step: 18.67

  9%|▉         | 3657/41250 [8:49:28<90:27:02,  8.66s/it]
                                                         
{'loss': 0.2064, 'grad_norm': 1.5370564460754395, 'learning_rate': 3.964034589830857e-05, 'epoch': 0.89}

  9%|▉         | 3657/41250 [8:49:28<90:27:02,  8.66s/it][2025-04-26 12:58:06,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.95
[2025-04-26 12:58:06,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.03 | bwd_microstep: 5741.39 | bwd_inner_microstep: 5703.20 | bwd_allreduce_microstep: 38.14 | step_microstep: 18.80
[2025-04-26 12:58:06,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.03 | bwd: 5741.40 | bwd_inner: 5703.20 | bwd_allreduce: 38.16 | step: 18.80

  9%|▉         | 3658/41250 [8:49:37<90:30:15,  8.67s/it]
                                                         
{'loss': 0.2266, 'grad_norm': 1.6925432682037354, 'learning_rate': 3.964004937453733e-05, 'epoch': 0.89}

  9%|▉         | 3658/41250 [8:49:37<90:30:15,  8.67s/it][2025-04-26 12:58:14,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 12:58:14,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.26 | bwd_microstep: 5737.86 | bwd_inner_microstep: 5712.13 | bwd_allreduce_microstep: 25.68 | step_microstep: 19.12
[2025-04-26 12:58:14,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.26 | bwd: 5737.87 | bwd_inner: 5712.13 | bwd_allreduce: 25.70 | step: 19.13

  9%|▉         | 3659/41250 [8:49:46<90:30:54,  8.67s/it]
                                                         
{'loss': 0.2398, 'grad_norm': 2.475907564163208, 'learning_rate': 3.963975272968904e-05, 'epoch': 0.89}

  9%|▉         | 3659/41250 [8:49:46<90:30:54,  8.67s/it][2025-04-26 12:58:23,553] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 12:58:23,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.46 | bwd_microstep: 5726.68 | bwd_inner_microstep: 5713.90 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.77
[2025-04-26 12:58:23,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.46 | bwd: 5726.70 | bwd_inner: 5713.90 | bwd_allreduce: 12.75 | step: 18.77

  9%|▉         | 3660/41250 [8:49:54<90:31:00,  8.67s/it]
                                                         
{'loss': 0.2204, 'grad_norm': 1.9510211944580078, 'learning_rate': 3.963945596376555e-05, 'epoch': 0.89}

  9%|▉         | 3660/41250 [8:49:54<90:31:00,  8.67s/it][2025-04-26 12:58:32,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 12:58:32,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.44 | bwd_microstep: 5764.14 | bwd_inner_microstep: 5687.60 | bwd_allreduce_microstep: 76.50 | step_microstep: 18.74
[2025-04-26 12:58:32,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.44 | bwd: 5764.15 | bwd_inner: 5687.60 | bwd_allreduce: 76.52 | step: 18.74

  9%|▉         | 3661/41250 [8:50:03<90:34:35,  8.67s/it]
                                                         
{'loss': 0.1955, 'grad_norm': 1.398438811302185, 'learning_rate': 3.963915907676866e-05, 'epoch': 0.89}

  9%|▉         | 3661/41250 [8:50:03<90:34:35,  8.67s/it][2025-04-26 12:58:40,858] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 12:58:40,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5704.46 | bwd_inner_microstep: 5656.32 | bwd_allreduce_microstep: 48.09 | step_microstep: 19.06
[2025-04-26 12:58:40,859] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5704.48 | bwd_inner: 5656.32 | bwd_allreduce: 48.11 | step: 19.06

  9%|▉         | 3662/41250 [8:50:12<90:24:19,  8.66s/it]
                                                         
{'loss': 0.1074, 'grad_norm': 0.958021342754364, 'learning_rate': 3.9638862068700234e-05, 'epoch': 0.89}

  9%|▉         | 3662/41250 [8:50:12<90:24:19,  8.66s/it][2025-04-26 12:58:49,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 12:58:49,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.10 | bwd_microstep: 5752.72 | bwd_inner_microstep: 5654.63 | bwd_allreduce_microstep: 98.04 | step_microstep: 18.67
[2025-04-26 12:58:49,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.10 | bwd: 5752.74 | bwd_inner: 5654.63 | bwd_allreduce: 98.06 | step: 18.67

  9%|▉         | 3663/41250 [8:50:20<90:26:21,  8.66s/it]
                                                         
{'loss': 0.1202, 'grad_norm': 0.9067596793174744, 'learning_rate': 3.963856493956208e-05, 'epoch': 0.89}

  9%|▉         | 3663/41250 [8:50:20<90:26:21,  8.66s/it][2025-04-26 12:58:58,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 12:58:58,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.34 | bwd_microstep: 5739.36 | bwd_inner_microstep: 5695.63 | bwd_allreduce_microstep: 43.69 | step_microstep: 18.36
[2025-04-26 12:58:58,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.34 | bwd: 5739.38 | bwd_inner: 5695.63 | bwd_allreduce: 43.71 | step: 18.37

  9%|▉         | 3664/41250 [8:50:29<90:28:18,  8.67s/it]
                                                         
{'loss': 0.0959, 'grad_norm': 1.1810766458511353, 'learning_rate': 3.963826768935603e-05, 'epoch': 0.89}

  9%|▉         | 3664/41250 [8:50:29<90:28:18,  8.67s/it][2025-04-26 12:59:06,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 12:59:06,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.41 | bwd_microstep: 5695.09 | bwd_inner_microstep: 5682.51 | bwd_allreduce_microstep: 12.54 | step_microstep: 17.91
[2025-04-26 12:59:06,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.41 | bwd: 5695.10 | bwd_inner: 5682.51 | bwd_allreduce: 12.56 | step: 17.92

  9%|▉         | 3665/41250 [8:50:38<90:18:12,  8.65s/it]
                                                         
{'loss': 0.1519, 'grad_norm': 1.747149109840393, 'learning_rate': 3.9637970318083925e-05, 'epoch': 0.89}

  9%|▉         | 3665/41250 [8:50:38<90:18:12,  8.65s/it][2025-04-26 12:59:15,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 12:59:15,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.76 | bwd_microstep: 5724.61 | bwd_inner_microstep: 5711.75 | bwd_allreduce_microstep: 12.82 | step_microstep: 17.88
[2025-04-26 12:59:15,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.76 | bwd: 5724.62 | bwd_inner: 5711.75 | bwd_allreduce: 12.83 | step: 17.88

  9%|▉         | 3666/41250 [8:50:46<90:21:43,  8.66s/it]
                                                         
{'loss': 0.1629, 'grad_norm': 1.0929336547851562, 'learning_rate': 3.96376728257476e-05, 'epoch': 0.89}

  9%|▉         | 3666/41250 [8:50:46<90:21:43,  8.66s/it][2025-04-26 12:59:24,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 12:59:24,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.59 | bwd_microstep: 5742.32 | bwd_inner_microstep: 5703.11 | bwd_allreduce_microstep: 39.16 | step_microstep: 18.42
[2025-04-26 12:59:24,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.59 | bwd: 5742.33 | bwd_inner: 5703.11 | bwd_allreduce: 39.18 | step: 18.42

  9%|▉         | 3667/41250 [8:50:55<90:27:33,  8.66s/it]
                                                         
{'loss': 0.2026, 'grad_norm': 1.3290425539016724, 'learning_rate': 3.9637375212348876e-05, 'epoch': 0.89}

  9%|▉         | 3667/41250 [8:50:55<90:27:33,  8.66s/it][2025-04-26 12:59:32,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 12:59:32,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.37 | bwd_microstep: 5757.69 | bwd_inner_microstep: 5656.00 | bwd_allreduce_microstep: 101.64 | step_microstep: 18.51
[2025-04-26 12:59:32,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.37 | bwd: 5757.70 | bwd_inner: 5656.00 | bwd_allreduce: 101.66 | step: 18.52

  9%|▉         | 3668/41250 [8:51:04<90:29:42,  8.67s/it]
                                                         
{'loss': 0.1894, 'grad_norm': 1.2013109922409058, 'learning_rate': 3.96370774778896e-05, 'epoch': 0.89}

  9%|▉         | 3668/41250 [8:51:04<90:29:42,  8.67s/it][2025-04-26 12:59:41,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.10 | optimizer_step: 1.03
[2025-04-26 12:59:41,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.20 | bwd_microstep: 5793.72 | bwd_inner_microstep: 5780.13 | bwd_allreduce_microstep: 13.54 | step_microstep: 19.38
[2025-04-26 12:59:41,618] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.20 | bwd: 5793.73 | bwd_inner: 5780.13 | bwd_allreduce: 13.56 | step: 19.38

  9%|▉         | 3669/41250 [8:51:12<90:47:49,  8.70s/it]
                                                         
{'loss': 0.1525, 'grad_norm': 1.1677159070968628, 'learning_rate': 3.963677962237161e-05, 'epoch': 0.89}

  9%|▉         | 3669/41250 [8:51:12<90:47:49,  8.70s/it][2025-04-26 12:59:50,279] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 12:59:50,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.89 | bwd_microstep: 5734.76 | bwd_inner_microstep: 5703.60 | bwd_allreduce_microstep: 31.12 | step_microstep: 18.46
[2025-04-26 12:59:50,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.89 | bwd: 5734.77 | bwd_inner: 5703.60 | bwd_allreduce: 31.14 | step: 18.46

  9%|▉         | 3670/41250 [8:51:21<90:40:51,  8.69s/it]
                                                         
{'loss': 0.0877, 'grad_norm': 0.9551190733909607, 'learning_rate': 3.963648164579673e-05, 'epoch': 0.89}

  9%|▉         | 3670/41250 [8:51:21<90:40:51,  8.69s/it][2025-04-26 12:59:58,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.11 | optimizer_step: 1.01
[2025-04-26 12:59:58,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.67 | bwd_microstep: 5754.12 | bwd_inner_microstep: 5704.67 | bwd_allreduce_microstep: 49.39 | step_microstep: 18.96
[2025-04-26 12:59:58,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.67 | bwd: 5754.13 | bwd_inner: 5704.67 | bwd_allreduce: 49.42 | step: 18.96

  9%|▉         | 3671/41250 [8:51:30<90:41:42,  8.69s/it]
                                                         
{'loss': 0.2235, 'grad_norm': 1.2191550731658936, 'learning_rate': 3.963618354816681e-05, 'epoch': 0.89}

  9%|▉         | 3671/41250 [8:51:30<90:41:42,  8.69s/it][2025-04-26 13:00:07,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:00:07,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.12 | bwd_microstep: 5712.41 | bwd_inner_microstep: 5693.32 | bwd_allreduce_microstep: 19.05 | step_microstep: 18.65
[2025-04-26 13:00:07,617] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.12 | bwd: 5712.43 | bwd_inner: 5693.32 | bwd_allreduce: 19.07 | step: 18.65

  9%|▉         | 3672/41250 [8:51:38<90:33:28,  8.68s/it]
                                                         
{'loss': 0.2079, 'grad_norm': 2.011317729949951, 'learning_rate': 3.963588532948367e-05, 'epoch': 0.89}

  9%|▉         | 3672/41250 [8:51:38<90:33:28,  8.68s/it][2025-04-26 13:00:16,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.24 | optimizer_step: 0.90
[2025-04-26 13:00:16,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.65 | bwd_microstep: 5751.72 | bwd_inner_microstep: 5633.39 | bwd_allreduce_microstep: 118.28 | step_microstep: 19.01
[2025-04-26 13:00:16,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.65 | bwd: 5751.74 | bwd_inner: 5633.39 | bwd_allreduce: 118.31 | step: 19.02

  9%|▉         | 3673/41250 [8:51:47<90:29:43,  8.67s/it]
                                                         
{'loss': 0.1367, 'grad_norm': 1.4623184204101562, 'learning_rate': 3.963558698974916e-05, 'epoch': 0.89}

  9%|▉         | 3673/41250 [8:51:47<90:29:43,  8.67s/it][2025-04-26 13:00:24,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.57 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:00:24,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.72 | bwd_microstep: 5709.98 | bwd_inner_microstep: 5697.22 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.89
[2025-04-26 13:00:24,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.72 | bwd: 5709.99 | bwd_inner: 5697.22 | bwd_allreduce: 12.73 | step: 18.89

  9%|▉         | 3674/41250 [8:51:56<90:22:16,  8.66s/it]
                                                         
{'loss': 0.337, 'grad_norm': 2.1903419494628906, 'learning_rate': 3.963528852896513e-05, 'epoch': 0.89}

  9%|▉         | 3674/41250 [8:51:56<90:22:16,  8.66s/it][2025-04-26 13:00:33,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 13:00:33,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2894.06 | bwd_microstep: 5779.06 | bwd_inner_microstep: 5766.23 | bwd_allreduce_microstep: 12.79 | step_microstep: 19.09
[2025-04-26 13:00:33,663] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2894.06 | bwd: 5779.07 | bwd_inner: 5766.23 | bwd_allreduce: 12.80 | step: 19.09

  9%|▉         | 3675/41250 [8:52:04<90:41:05,  8.69s/it]
                                                         
{'loss': 0.3561, 'grad_norm': 1.7535276412963867, 'learning_rate': 3.96349899471334e-05, 'epoch': 0.89}

  9%|▉         | 3675/41250 [8:52:04<90:41:05,  8.69s/it][2025-04-26 13:00:42,376] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 13:00:42,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2876.02 | bwd_microstep: 5754.85 | bwd_inner_microstep: 5742.06 | bwd_allreduce_microstep: 12.74 | step_microstep: 19.03
[2025-04-26 13:00:42,377] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2876.02 | bwd: 5754.86 | bwd_inner: 5742.06 | bwd_allreduce: 12.76 | step: 19.04

  9%|▉         | 3676/41250 [8:52:13<90:45:44,  8.70s/it]
                                                         
{'loss': 0.085, 'grad_norm': 1.41110360622406, 'learning_rate': 3.963469124425582e-05, 'epoch': 0.89}

  9%|▉         | 3676/41250 [8:52:13<90:45:44,  8.70s/it][2025-04-26 13:00:50,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.02 | optimizer_step: 0.96
[2025-04-26 13:00:50,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.91 | bwd_microstep: 5695.77 | bwd_inner_microstep: 5682.48 | bwd_allreduce_microstep: 13.24 | step_microstep: 18.65
[2025-04-26 13:00:51,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.91 | bwd: 5695.80 | bwd_inner: 5682.48 | bwd_allreduce: 13.27 | step: 18.65

  9%|▉         | 3677/41250 [8:52:22<90:31:31,  8.67s/it]
                                                         
{'loss': 0.2797, 'grad_norm': 1.6067008972167969, 'learning_rate': 3.9634392420334224e-05, 'epoch': 0.89}

  9%|▉         | 3677/41250 [8:52:22<90:31:31,  8.67s/it][2025-04-26 13:00:59,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:00:59,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.94 | bwd_microstep: 5739.70 | bwd_inner_microstep: 5708.91 | bwd_allreduce_microstep: 30.75 | step_microstep: 18.03
[2025-04-26 13:00:59,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.94 | bwd: 5739.71 | bwd_inner: 5708.91 | bwd_allreduce: 30.77 | step: 18.03

  9%|▉         | 3678/41250 [8:52:30<90:33:14,  8.68s/it]
                                                         
{'loss': 0.1756, 'grad_norm': 1.3658210039138794, 'learning_rate': 3.963409347537047e-05, 'epoch': 0.89}

  9%|▉         | 3678/41250 [8:52:30<90:33:14,  8.68s/it][2025-04-26 13:01:08,589] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:01:08,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.14 | bwd_microstep: 5980.68 | bwd_inner_microstep: 5684.25 | bwd_allreduce_microstep: 296.40 | step_microstep: 18.04
[2025-04-26 13:01:08,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.14 | bwd: 5980.70 | bwd_inner: 5684.25 | bwd_allreduce: 296.41 | step: 18.05

  9%|▉         | 3679/41250 [8:52:39<91:16:30,  8.75s/it]
                                                         
{'loss': 0.0414, 'grad_norm': 0.38592809438705444, 'learning_rate': 3.9633794409366385e-05, 'epoch': 0.89}

  9%|▉         | 3679/41250 [8:52:39<91:16:30,  8.75s/it][2025-04-26 13:01:17,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 13:01:17,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.13 | bwd_microstep: 5680.86 | bwd_inner_microstep: 5648.04 | bwd_allreduce_microstep: 32.78 | step_microstep: 18.89
[2025-04-26 13:01:17,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.13 | bwd: 5680.88 | bwd_inner: 5648.04 | bwd_allreduce: 32.80 | step: 18.89

  9%|▉         | 3680/41250 [8:52:48<90:48:49,  8.70s/it]
                                                         
{'loss': 0.0859, 'grad_norm': 0.8157440423965454, 'learning_rate': 3.9633495222323827e-05, 'epoch': 0.89}

  9%|▉         | 3680/41250 [8:52:48<90:48:49,  8.70s/it][2025-04-26 13:01:25,795] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:01:25,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.87 | bwd_microstep: 5696.36 | bwd_inner_microstep: 5650.14 | bwd_allreduce_microstep: 46.17 | step_microstep: 18.85
[2025-04-26 13:01:25,796] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.87 | bwd: 5696.37 | bwd_inner: 5650.14 | bwd_allreduce: 46.19 | step: 18.85

  9%|▉         | 3681/41250 [8:52:56<90:30:54,  8.67s/it]
                                                         
{'loss': 0.1332, 'grad_norm': 1.604629635810852, 'learning_rate': 3.963319591424463e-05, 'epoch': 0.89}

  9%|▉         | 3681/41250 [8:52:56<90:30:54,  8.67s/it][2025-04-26 13:01:34,429] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:01:34,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.64 | bwd_microstep: 5706.17 | bwd_inner_microstep: 5693.43 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.61
[2025-04-26 13:01:34,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.64 | bwd: 5706.18 | bwd_inner: 5693.43 | bwd_allreduce: 12.71 | step: 18.61

  9%|▉         | 3682/41250 [8:53:05<90:23:18,  8.66s/it]
                                                         
{'loss': 0.0543, 'grad_norm': 0.9135642051696777, 'learning_rate': 3.963289648513063e-05, 'epoch': 0.89}

  9%|▉         | 3682/41250 [8:53:05<90:23:18,  8.66s/it][2025-04-26 13:01:43,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:01:43,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.12 | bwd_microstep: 5889.83 | bwd_inner_microstep: 5651.03 | bwd_allreduce_microstep: 238.75 | step_microstep: 18.50
[2025-04-26 13:01:43,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.12 | bwd: 5889.84 | bwd_inner: 5651.03 | bwd_allreduce: 238.77 | step: 18.50

  9%|▉         | 3683/41250 [8:53:14<90:46:48,  8.70s/it]
                                                         
{'loss': 0.2915, 'grad_norm': 1.7180991172790527, 'learning_rate': 3.96325969349837e-05, 'epoch': 0.89}

  9%|▉         | 3683/41250 [8:53:14<90:46:48,  8.70s/it][2025-04-26 13:01:51,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:01:51,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.60 | bwd_microstep: 5764.01 | bwd_inner_microstep: 5691.08 | bwd_allreduce_microstep: 72.89 | step_microstep: 18.60
[2025-04-26 13:01:51,911] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.60 | bwd: 5764.03 | bwd_inner: 5691.08 | bwd_allreduce: 72.91 | step: 18.60

  9%|▉         | 3684/41250 [8:53:23<90:45:30,  8.70s/it]
                                                         
{'loss': 0.1217, 'grad_norm': 1.2841622829437256, 'learning_rate': 3.963229726380567e-05, 'epoch': 0.89}

  9%|▉         | 3684/41250 [8:53:23<90:45:30,  8.70s/it][2025-04-26 13:02:00,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.92
[2025-04-26 13:02:00,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.24 | bwd_microstep: 5748.66 | bwd_inner_microstep: 5688.03 | bwd_allreduce_microstep: 60.57 | step_microstep: 18.58
[2025-04-26 13:02:00,585] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.24 | bwd: 5748.67 | bwd_inner: 5688.03 | bwd_allreduce: 60.59 | step: 18.58

  9%|▉         | 3685/41250 [8:53:31<90:41:02,  8.69s/it]
                                                         
{'loss': 0.2536, 'grad_norm': 1.7361013889312744, 'learning_rate': 3.963199747159838e-05, 'epoch': 0.89}

  9%|▉         | 3685/41250 [8:53:31<90:41:02,  8.69s/it][2025-04-26 13:02:09,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.04 | optimizer_step: 1.25
[2025-04-26 13:02:09,356] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.07 | bwd_microstep: 5792.68 | bwd_inner_microstep: 5780.23 | bwd_allreduce_microstep: 12.40 | step_microstep: 19.57
[2025-04-26 13:02:09,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.07 | bwd: 5792.70 | bwd_inner: 5780.23 | bwd_allreduce: 12.42 | step: 19.57

  9%|▉         | 3686/41250 [8:53:40<90:56:16,  8.72s/it]
                                                         
{'loss': 0.1455, 'grad_norm': 1.106796383857727, 'learning_rate': 3.963169755836369e-05, 'epoch': 0.89}

  9%|▉         | 3686/41250 [8:53:40<90:56:16,  8.72s/it][2025-04-26 13:02:18,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 1.02 | optimizer_step: 0.89
[2025-04-26 13:02:18,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.16 | bwd_microstep: 5767.85 | bwd_inner_microstep: 5649.87 | bwd_allreduce_microstep: 117.93 | step_microstep: 18.96
[2025-04-26 13:02:18,036] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.16 | bwd: 5767.87 | bwd_inner: 5649.87 | bwd_allreduce: 117.95 | step: 18.96

  9%|▉         | 3687/41250 [8:53:49<90:49:12,  8.70s/it]
                                                         
{'loss': 0.1085, 'grad_norm': 1.1801862716674805, 'learning_rate': 3.963139752410345e-05, 'epoch': 0.89}

  9%|▉         | 3687/41250 [8:53:49<90:49:12,  8.70s/it][2025-04-26 13:02:26,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:02:26,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.43 | bwd_microstep: 5771.88 | bwd_inner_microstep: 5649.98 | bwd_allreduce_microstep: 121.86 | step_microstep: 18.42
[2025-04-26 13:02:26,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.43 | bwd: 5771.89 | bwd_inner: 5649.98 | bwd_allreduce: 121.88 | step: 18.42

  9%|▉         | 3688/41250 [8:53:57<90:43:45,  8.70s/it]
                                                         
{'loss': 0.1263, 'grad_norm': 0.791940450668335, 'learning_rate': 3.963109736881951e-05, 'epoch': 0.89}

  9%|▉         | 3688/41250 [8:53:57<90:43:45,  8.70s/it][2025-04-26 13:02:35,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:02:35,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.52 | bwd_microstep: 5786.90 | bwd_inner_microstep: 5774.05 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.81
[2025-04-26 13:02:35,463] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.52 | bwd: 5786.91 | bwd_inner: 5774.05 | bwd_allreduce: 12.82 | step: 18.82

  9%|▉         | 3689/41250 [8:54:06<90:54:02,  8.71s/it]
                                                         
{'loss': 0.2928, 'grad_norm': 3.05311918258667, 'learning_rate': 3.96307970925137e-05, 'epoch': 0.89}

  9%|▉         | 3689/41250 [8:54:06<90:54:02,  8.71s/it][2025-04-26 13:02:44,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 1.14
[2025-04-26 13:02:44,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.85 | bwd_microstep: 5801.33 | bwd_inner_microstep: 5664.71 | bwd_allreduce_microstep: 136.55 | step_microstep: 19.96
[2025-04-26 13:02:44,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.85 | bwd: 5801.35 | bwd_inner: 5664.71 | bwd_allreduce: 136.58 | step: 19.96

  9%|▉         | 3690/41250 [8:54:15<90:54:19,  8.71s/it]
                                                         
{'loss': 0.2903, 'grad_norm': 2.661752700805664, 'learning_rate': 3.96304966951879e-05, 'epoch': 0.89}

  9%|▉         | 3690/41250 [8:54:15<90:54:19,  8.71s/it][2025-04-26 13:02:52,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:02:52,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.55 | bwd_microstep: 5698.72 | bwd_inner_microstep: 5656.07 | bwd_allreduce_microstep: 42.61 | step_microstep: 18.68
[2025-04-26 13:02:52,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5698.73 | bwd_inner: 5656.07 | bwd_allreduce: 42.62 | step: 18.68

  9%|▉         | 3691/41250 [8:54:23<90:35:20,  8.68s/it]
                                                         
{'loss': 0.0376, 'grad_norm': 0.7828084230422974, 'learning_rate': 3.9630196176843947e-05, 'epoch': 0.89}

  9%|▉         | 3691/41250 [8:54:23<90:35:20,  8.68s/it][2025-04-26 13:03:01,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 13:03:01,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.45 | bwd_microstep: 5715.05 | bwd_inner_microstep: 5702.41 | bwd_allreduce_microstep: 12.60 | step_microstep: 18.63
[2025-04-26 13:03:01,428] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.45 | bwd: 5715.07 | bwd_inner: 5702.41 | bwd_allreduce: 12.62 | step: 18.64

  9%|▉         | 3692/41250 [8:54:32<90:26:36,  8.67s/it]
                                                         
{'loss': 0.2446, 'grad_norm': 2.2261838912963867, 'learning_rate': 3.962989553748369e-05, 'epoch': 0.9}

  9%|▉         | 3692/41250 [8:54:32<90:26:36,  8.67s/it][2025-04-26 13:03:10,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:03:10,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.29 | bwd_microstep: 5713.60 | bwd_inner_microstep: 5700.10 | bwd_allreduce_microstep: 13.46 | step_microstep: 18.58
[2025-04-26 13:03:10,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.29 | bwd: 5713.62 | bwd_inner: 5700.10 | bwd_allreduce: 13.47 | step: 18.58

  9%|▉         | 3693/41250 [8:54:41<90:21:14,  8.66s/it]
                                                         
{'loss': 0.0446, 'grad_norm': 0.47679638862609863, 'learning_rate': 3.9629594777108996e-05, 'epoch': 0.9}

  9%|▉         | 3693/41250 [8:54:41<90:21:14,  8.66s/it][2025-04-26 13:03:18,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 13:03:18,706] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.81 | bwd_microstep: 5708.81 | bwd_inner_microstep: 5689.43 | bwd_allreduce_microstep: 19.34 | step_microstep: 18.53
[2025-04-26 13:03:18,707] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.81 | bwd: 5708.83 | bwd_inner: 5689.42 | bwd_allreduce: 19.36 | step: 18.53

  9%|▉         | 3694/41250 [8:54:49<90:16:41,  8.65s/it]
                                                         
{'loss': 0.0799, 'grad_norm': 0.7282202839851379, 'learning_rate': 3.9629293895721707e-05, 'epoch': 0.9}

  9%|▉         | 3694/41250 [8:54:49<90:16:41,  8.65s/it][2025-04-26 13:03:27,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:03:27,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.87 | bwd_microstep: 5724.23 | bwd_inner_microstep: 5663.28 | bwd_allreduce_microstep: 60.90 | step_microstep: 18.68
[2025-04-26 13:03:27,343] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.87 | bwd: 5724.25 | bwd_inner: 5663.28 | bwd_allreduce: 60.92 | step: 18.68

  9%|▉         | 3695/41250 [8:54:58<90:13:31,  8.65s/it]
                                                         
{'loss': 0.1763, 'grad_norm': 1.5440056324005127, 'learning_rate': 3.9628992893323685e-05, 'epoch': 0.9}

  9%|▉         | 3695/41250 [8:54:58<90:13:31,  8.65s/it][2025-04-26 13:03:36,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.31 | optimizer_step: 1.02
[2025-04-26 13:03:36,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.14 | bwd_microstep: 5912.55 | bwd_inner_microstep: 5659.62 | bwd_allreduce_microstep: 252.88 | step_microstep: 19.83
[2025-04-26 13:03:36,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.14 | bwd: 5912.57 | bwd_inner: 5659.61 | bwd_allreduce: 252.91 | step: 19.83

  9%|▉         | 3696/41250 [8:55:07<90:47:42,  8.70s/it]
                                                         
{'loss': 0.1988, 'grad_norm': 2.1524972915649414, 'learning_rate': 3.9628691769916775e-05, 'epoch': 0.9}

  9%|▉         | 3696/41250 [8:55:07<90:47:42,  8.70s/it][2025-04-26 13:03:44,869] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:03:44,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.71 | bwd_microstep: 5768.66 | bwd_inner_microstep: 5691.36 | bwd_allreduce_microstep: 77.25 | step_microstep: 18.62
[2025-04-26 13:03:44,870] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.71 | bwd: 5768.67 | bwd_inner: 5691.36 | bwd_allreduce: 77.27 | step: 18.62

  9%|▉         | 3697/41250 [8:55:16<90:45:44,  8.70s/it]
                                                         
{'loss': 0.1727, 'grad_norm': 2.004037618637085, 'learning_rate': 3.9628390525502846e-05, 'epoch': 0.9}

  9%|▉         | 3697/41250 [8:55:16<90:45:44,  8.70s/it][2025-04-26 13:03:53,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:03:53,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2897.91 | bwd_microstep: 5786.10 | bwd_inner_microstep: 5773.23 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.61
[2025-04-26 13:03:53,639] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2897.91 | bwd: 5786.11 | bwd_inner: 5773.23 | bwd_allreduce: 12.84 | step: 18.61

  9%|▉         | 3698/41250 [8:55:24<90:58:15,  8.72s/it]
                                                         
{'loss': 0.1662, 'grad_norm': 1.7298142910003662, 'learning_rate': 3.962808916008375e-05, 'epoch': 0.9}

  9%|▉         | 3698/41250 [8:55:24<90:58:15,  8.72s/it][2025-04-26 13:04:02,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:04:02,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.76 | bwd_microstep: 5701.07 | bwd_inner_microstep: 5679.72 | bwd_allreduce_microstep: 21.31 | step_microstep: 18.71
[2025-04-26 13:04:02,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.76 | bwd: 5701.09 | bwd_inner: 5679.72 | bwd_allreduce: 21.32 | step: 18.71

  9%|▉         | 3699/41250 [8:55:33<90:40:53,  8.69s/it]
                                                         
{'loss': 0.0855, 'grad_norm': 1.2148009538650513, 'learning_rate': 3.962778767366135e-05, 'epoch': 0.9}

  9%|▉         | 3699/41250 [8:55:33<90:40:53,  8.69s/it][2025-04-26 13:04:10,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 13:04:10,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.87 | bwd_microstep: 5700.24 | bwd_inner_microstep: 5664.84 | bwd_allreduce_microstep: 35.35 | step_microstep: 19.03
[2025-04-26 13:04:10,885] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.87 | bwd: 5700.25 | bwd_inner: 5664.84 | bwd_allreduce: 35.37 | step: 19.03

  9%|▉         | 3700/41250 [8:55:42<90:26:27,  8.67s/it]
                                                         
{'loss': 0.1834, 'grad_norm': 2.05731463432312, 'learning_rate': 3.962748606623749e-05, 'epoch': 0.9}

  9%|▉         | 3700/41250 [8:55:42<90:26:27,  8.67s/it][2025-04-26 13:04:19,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:04:19,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.13 | bwd_microstep: 5725.43 | bwd_inner_microstep: 5712.51 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.65
[2025-04-26 13:04:19,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5725.44 | bwd_inner: 5712.51 | bwd_allreduce: 12.89 | step: 18.66

  9%|▉         | 3701/41250 [8:55:50<90:22:36,  8.66s/it]
                                                         
{'loss': 0.2232, 'grad_norm': 1.622084140777588, 'learning_rate': 3.962718433781404e-05, 'epoch': 0.9}

  9%|▉         | 3701/41250 [8:55:50<90:22:36,  8.66s/it][2025-04-26 13:04:28,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:04:28,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.11 | bwd_microstep: 5719.60 | bwd_inner_microstep: 5665.76 | bwd_allreduce_microstep: 53.80 | step_microstep: 18.92
[2025-04-26 13:04:28,172] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.11 | bwd: 5719.62 | bwd_inner: 5665.76 | bwd_allreduce: 53.82 | step: 18.93

  9%|▉         | 3702/41250 [8:55:59<90:17:20,  8.66s/it]
                                                         
{'loss': 0.1687, 'grad_norm': 2.4147934913635254, 'learning_rate': 3.9626882488392864e-05, 'epoch': 0.9}

  9%|▉         | 3702/41250 [8:55:59<90:17:20,  8.66s/it][2025-04-26 13:04:36,861] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.28 | optimizer_step: 1.03
[2025-04-26 13:04:36,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.76 | bwd_microstep: 5772.50 | bwd_inner_microstep: 5659.87 | bwd_allreduce_microstep: 112.56 | step_microstep: 19.86
[2025-04-26 13:04:36,862] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.76 | bwd: 5772.51 | bwd_inner: 5659.87 | bwd_allreduce: 112.59 | step: 19.86

  9%|▉         | 3703/41250 [8:56:08<90:23:12,  8.67s/it]
                                                         
{'loss': 0.1739, 'grad_norm': 1.7519874572753906, 'learning_rate': 3.9626580517975816e-05, 'epoch': 0.9}

  9%|▉         | 3703/41250 [8:56:08<90:23:12,  8.67s/it][2025-04-26 13:04:45,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 13:04:45,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.89 | bwd_microstep: 5730.19 | bwd_inner_microstep: 5702.13 | bwd_allreduce_microstep: 28.02 | step_microstep: 18.75
[2025-04-26 13:04:45,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.89 | bwd: 5730.20 | bwd_inner: 5702.13 | bwd_allreduce: 28.04 | step: 18.76

  9%|▉         | 3704/41250 [8:56:16<90:24:14,  8.67s/it]
                                                         
{'loss': 0.2155, 'grad_norm': 1.2907466888427734, 'learning_rate': 3.962627842656476e-05, 'epoch': 0.9}

  9%|▉         | 3704/41250 [8:56:16<90:24:14,  8.67s/it][2025-04-26 13:04:54,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.07 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:04:54,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2875.12 | bwd_microstep: 5959.02 | bwd_inner_microstep: 5723.52 | bwd_allreduce_microstep: 235.46 | step_microstep: 18.13
[2025-04-26 13:04:54,450] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2875.12 | bwd: 5959.04 | bwd_inner: 5723.52 | bwd_allreduce: 235.48 | step: 18.13

  9%|▉         | 3705/41250 [8:56:25<91:10:22,  8.74s/it]
                                                         
{'loss': 0.3795, 'grad_norm': 2.883849620819092, 'learning_rate': 3.962597621416156e-05, 'epoch': 0.9}

  9%|▉         | 3705/41250 [8:56:25<91:10:22,  8.74s/it][2025-04-26 13:05:03,081] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:05:03,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.28 | bwd_microstep: 5707.53 | bwd_inner_microstep: 5694.91 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.43
[2025-04-26 13:05:03,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.28 | bwd: 5707.55 | bwd_inner: 5694.91 | bwd_allreduce: 12.60 | step: 18.44

  9%|▉         | 3706/41250 [8:56:34<90:49:34,  8.71s/it]
                                                         
{'loss': 0.2051, 'grad_norm': 3.283194065093994, 'learning_rate': 3.9625673880768076e-05, 'epoch': 0.9}

  9%|▉         | 3706/41250 [8:56:34<90:49:34,  8.71s/it][2025-04-26 13:05:11,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:05:11,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.26 | bwd_microstep: 5714.34 | bwd_inner_microstep: 5701.55 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.57
[2025-04-26 13:05:11,725] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.26 | bwd: 5714.35 | bwd_inner: 5701.55 | bwd_allreduce: 12.76 | step: 18.58

  9%|▉         | 3707/41250 [8:56:42<90:37:02,  8.69s/it]
                                                         
{'loss': 0.0544, 'grad_norm': 1.3132755756378174, 'learning_rate': 3.962537142638616e-05, 'epoch': 0.9}

  9%|▉         | 3707/41250 [8:56:42<90:37:02,  8.69s/it][2025-04-26 13:05:20,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 13:05:20,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.84 | bwd_microstep: 5899.23 | bwd_inner_microstep: 5662.12 | bwd_allreduce_microstep: 237.06 | step_microstep: 18.83
[2025-04-26 13:05:20,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.84 | bwd: 5899.24 | bwd_inner: 5662.12 | bwd_allreduce: 237.08 | step: 18.83

  9%|▉         | 3708/41250 [8:56:51<91:02:35,  8.73s/it]
                                                         
{'loss': 0.2404, 'grad_norm': 2.4122486114501953, 'learning_rate': 3.962506885101771e-05, 'epoch': 0.9}

  9%|▉         | 3708/41250 [8:56:51<91:02:35,  8.73s/it][2025-04-26 13:05:29,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:05:29,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.27 | bwd_microstep: 5710.68 | bwd_inner_microstep: 5650.79 | bwd_allreduce_microstep: 59.84 | step_microstep: 18.79
[2025-04-26 13:05:29,175] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.27 | bwd: 5710.70 | bwd_inner: 5650.79 | bwd_allreduce: 59.86 | step: 18.79

  9%|▉         | 3709/41250 [8:57:00<90:42:37,  8.70s/it]
                                                         
{'loss': 0.0826, 'grad_norm': 1.3586608171463013, 'learning_rate': 3.962476615466456e-05, 'epoch': 0.9}

  9%|▉         | 3709/41250 [8:57:00<90:42:37,  8.70s/it][2025-04-26 13:05:37,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.99
[2025-04-26 13:05:37,819] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5720.74 | bwd_inner_microstep: 5708.06 | bwd_allreduce_microstep: 12.64 | step_microstep: 19.26
[2025-04-26 13:05:37,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5720.76 | bwd_inner: 5708.06 | bwd_allreduce: 12.66 | step: 19.27

  9%|▉         | 3710/41250 [8:57:09<90:32:17,  8.68s/it]
                                                         
{'loss': 0.264, 'grad_norm': 1.564235806465149, 'learning_rate': 3.9624463337328586e-05, 'epoch': 0.9}

  9%|▉         | 3710/41250 [8:57:09<90:32:17,  8.68s/it][2025-04-26 13:05:46,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.01 | optimizer_step: 1.12
[2025-04-26 13:05:46,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.93 | bwd_microstep: 5706.20 | bwd_inner_microstep: 5663.70 | bwd_allreduce_microstep: 42.45 | step_microstep: 18.95
[2025-04-26 13:05:46,441] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.93 | bwd: 5706.21 | bwd_inner: 5663.70 | bwd_allreduce: 42.47 | step: 18.95

  9%|▉         | 3711/41250 [8:57:17<90:20:37,  8.66s/it]
                                                         
{'loss': 0.3657, 'grad_norm': 2.065976142883301, 'learning_rate': 3.962416039901166e-05, 'epoch': 0.9}

  9%|▉         | 3711/41250 [8:57:17<90:20:37,  8.66s/it][2025-04-26 13:05:55,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:05:55,127] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.09 | bwd_microstep: 5755.77 | bwd_inner_microstep: 5651.14 | bwd_allreduce_microstep: 104.58 | step_microstep: 18.73
[2025-04-26 13:05:55,128] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.09 | bwd: 5755.78 | bwd_inner: 5651.14 | bwd_allreduce: 104.60 | step: 18.74

  9%|▉         | 3712/41250 [8:57:26<90:24:35,  8.67s/it]
                                                         
{'loss': 0.0446, 'grad_norm': 0.7078949809074402, 'learning_rate': 3.9623857339715636e-05, 'epoch': 0.9}

  9%|▉         | 3712/41250 [8:57:26<90:24:35,  8.67s/it][2025-04-26 13:06:03,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:06:03,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.67 | bwd_microstep: 5755.68 | bwd_inner_microstep: 5691.61 | bwd_allreduce_microstep: 64.01 | step_microstep: 18.74
[2025-04-26 13:06:03,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.67 | bwd: 5755.69 | bwd_inner: 5691.61 | bwd_allreduce: 64.04 | step: 18.73

  9%|▉         | 3713/41250 [8:57:34<90:27:22,  8.68s/it]
                                                         
{'loss': 0.4801, 'grad_norm': 2.12404203414917, 'learning_rate': 3.96235541594424e-05, 'epoch': 0.9}

  9%|▉         | 3713/41250 [8:57:34<90:27:22,  8.68s/it][2025-04-26 13:06:12,490] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 13:06:12,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.27 | bwd_microstep: 5767.44 | bwd_inner_microstep: 5657.19 | bwd_allreduce_microstep: 110.21 | step_microstep: 19.04
[2025-04-26 13:06:12,491] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.27 | bwd: 5767.45 | bwd_inner: 5657.19 | bwd_allreduce: 110.22 | step: 19.05

  9%|▉         | 3714/41250 [8:57:43<90:27:52,  8.68s/it]
                                                         
{'loss': 0.0751, 'grad_norm': 0.7255618572235107, 'learning_rate': 3.962325085819381e-05, 'epoch': 0.9}

  9%|▉         | 3714/41250 [8:57:43<90:27:52,  8.68s/it][2025-04-26 13:06:21,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:06:21,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.74 | bwd_microstep: 5761.58 | bwd_inner_microstep: 5644.63 | bwd_allreduce_microstep: 116.91 | step_microstep: 18.54
[2025-04-26 13:06:21,158] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.74 | bwd: 5761.59 | bwd_inner: 5644.62 | bwd_allreduce: 116.93 | step: 18.55

  9%|▉         | 3715/41250 [8:57:52<90:25:47,  8.67s/it]
                                                         
{'loss': 0.1116, 'grad_norm': 1.4677939414978027, 'learning_rate': 3.962294743597173e-05, 'epoch': 0.9}

  9%|▉         | 3715/41250 [8:57:52<90:25:47,  8.67s/it][2025-04-26 13:06:29,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:06:29,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.73 | bwd_microstep: 5707.74 | bwd_inner_microstep: 5695.04 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.35
[2025-04-26 13:06:29,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.73 | bwd: 5707.76 | bwd_inner: 5695.04 | bwd_allreduce: 12.68 | step: 18.36

  9%|▉         | 3716/41250 [8:58:00<90:21:54,  8.67s/it]
                                                         
{'loss': 0.0618, 'grad_norm': 0.7234595417976379, 'learning_rate': 3.962264389277805e-05, 'epoch': 0.9}

  9%|▉         | 3716/41250 [8:58:01<90:21:54,  8.67s/it][2025-04-26 13:06:38,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.17 | optimizer_step: 0.90
[2025-04-26 13:06:38,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.84 | bwd_microstep: 5750.69 | bwd_inner_microstep: 5653.23 | bwd_allreduce_microstep: 97.40 | step_microstep: 18.90
[2025-04-26 13:06:38,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.84 | bwd: 5750.70 | bwd_inner: 5653.23 | bwd_allreduce: 97.42 | step: 18.91

  9%|▉         | 3717/41250 [8:58:09<90:21:39,  8.67s/it]
                                                         
{'loss': 0.0648, 'grad_norm': 0.7134460210800171, 'learning_rate': 3.962234022861463e-05, 'epoch': 0.9}

  9%|▉         | 3717/41250 [8:58:09<90:21:39,  8.67s/it][2025-04-26 13:06:47,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:06:47,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.28 | bwd_microstep: 5680.84 | bwd_inner_microstep: 5649.37 | bwd_allreduce_microstep: 31.42 | step_microstep: 18.73
[2025-04-26 13:06:47,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.28 | bwd: 5680.85 | bwd_inner: 5649.37 | bwd_allreduce: 31.44 | step: 18.73

  9%|▉         | 3718/41250 [8:58:18<90:07:44,  8.65s/it]
                                                         
{'loss': 0.0479, 'grad_norm': 0.5783421993255615, 'learning_rate': 3.962203644348334e-05, 'epoch': 0.9}

  9%|▉         | 3718/41250 [8:58:18<90:07:44,  8.65s/it][2025-04-26 13:06:55,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 13:06:55,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.49 | bwd_microstep: 5788.80 | bwd_inner_microstep: 5656.62 | bwd_allreduce_microstep: 132.13 | step_microstep: 19.10
[2025-04-26 13:06:55,757] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.49 | bwd: 5788.81 | bwd_inner: 5656.62 | bwd_allreduce: 132.15 | step: 19.10

  9%|▉         | 3719/41250 [8:58:26<90:15:23,  8.66s/it]
                                                         
{'loss': 0.1093, 'grad_norm': 1.4233487844467163, 'learning_rate': 3.962173253738605e-05, 'epoch': 0.9}

  9%|▉         | 3719/41250 [8:58:26<90:15:23,  8.66s/it][2025-04-26 13:07:04,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 13:07:04,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.91 | bwd_microstep: 5736.63 | bwd_inner_microstep: 5698.82 | bwd_allreduce_microstep: 37.77 | step_microstep: 18.99
[2025-04-26 13:07:04,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.91 | bwd: 5736.65 | bwd_inner: 5698.82 | bwd_allreduce: 37.79 | step: 18.99

  9%|▉         | 3720/41250 [8:58:35<90:17:24,  8.66s/it]
                                                         
{'loss': 0.1994, 'grad_norm': 1.9849313497543335, 'learning_rate': 3.962142851032464e-05, 'epoch': 0.9}

  9%|▉         | 3720/41250 [8:58:35<90:17:24,  8.66s/it][2025-04-26 13:07:13,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:07:13,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.81 | bwd_microstep: 5711.30 | bwd_inner_microstep: 5698.48 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.92
[2025-04-26 13:07:13,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.81 | bwd: 5711.32 | bwd_inner: 5698.48 | bwd_allreduce: 12.79 | step: 18.92

  9%|▉         | 3721/41250 [8:58:44<90:13:01,  8.65s/it]
                                                         
{'loss': 0.207, 'grad_norm': 2.3015494346618652, 'learning_rate': 3.962112436230099e-05, 'epoch': 0.9}

  9%|▉         | 3721/41250 [8:58:44<90:13:01,  8.65s/it][2025-04-26 13:07:21,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:07:21,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.64 | bwd_microstep: 5730.93 | bwd_inner_microstep: 5699.57 | bwd_allreduce_microstep: 31.31 | step_microstep: 18.53
[2025-04-26 13:07:21,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.64 | bwd: 5730.94 | bwd_inner: 5699.57 | bwd_allreduce: 31.33 | step: 18.53

  9%|▉         | 3722/41250 [8:58:52<90:14:45,  8.66s/it]
                                                         
{'loss': 0.1976, 'grad_norm': 1.6972086429595947, 'learning_rate': 3.9620820093316956e-05, 'epoch': 0.9}

  9%|▉         | 3722/41250 [8:58:52<90:14:45,  8.66s/it][2025-04-26 13:07:30,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:07:30,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.12 | bwd_microstep: 5702.51 | bwd_inner_microstep: 5689.71 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.59
[2025-04-26 13:07:30,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.12 | bwd: 5702.52 | bwd_inner: 5689.71 | bwd_allreduce: 12.77 | step: 18.60

  9%|▉         | 3723/41250 [8:59:01<90:09:20,  8.65s/it]
                                                         
{'loss': 0.086, 'grad_norm': 1.3323042392730713, 'learning_rate': 3.962051570337443e-05, 'epoch': 0.9}

  9%|▉         | 3723/41250 [8:59:01<90:09:20,  8.65s/it][2025-04-26 13:07:38,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.04 | optimizer_step: 1.04
[2025-04-26 13:07:38,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.76 | bwd_microstep: 5708.79 | bwd_inner_microstep: 5695.60 | bwd_allreduce_microstep: 13.14 | step_microstep: 19.61
[2025-04-26 13:07:38,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.76 | bwd: 5708.80 | bwd_inner: 5695.60 | bwd_allreduce: 13.16 | step: 19.61

  9%|▉         | 3724/41250 [8:59:10<90:07:22,  8.65s/it]
                                                         
{'loss': 0.1322, 'grad_norm': 1.4316918849945068, 'learning_rate': 3.962021119247529e-05, 'epoch': 0.9}

  9%|▉         | 3724/41250 [8:59:10<90:07:22,  8.65s/it][2025-04-26 13:07:47,692] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:07:47,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.22 | bwd_microstep: 5763.60 | bwd_inner_microstep: 5705.57 | bwd_allreduce_microstep: 57.98 | step_microstep: 18.98
[2025-04-26 13:07:47,693] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.22 | bwd: 5763.61 | bwd_inner: 5705.57 | bwd_allreduce: 58.00 | step: 18.98

  9%|▉         | 3725/41250 [8:59:18<90:16:33,  8.66s/it]
                                                         
{'loss': 0.0576, 'grad_norm': 0.7301201820373535, 'learning_rate': 3.96199065606214e-05, 'epoch': 0.9}

  9%|▉         | 3725/41250 [8:59:18<90:16:33,  8.66s/it][2025-04-26 13:07:56,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.97 | optimizer_step: 0.98
[2025-04-26 13:07:56,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.28 | bwd_microstep: 5710.33 | bwd_inner_microstep: 5697.56 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.55
[2025-04-26 13:07:56,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.28 | bwd: 5710.34 | bwd_inner: 5697.56 | bwd_allreduce: 12.74 | step: 18.56

  9%|▉         | 3726/41250 [8:59:27<90:12:35,  8.65s/it]
                                                         
{'loss': 0.4571, 'grad_norm': 2.5254600048065186, 'learning_rate': 3.961960180781464e-05, 'epoch': 0.9}

  9%|▉         | 3726/41250 [8:59:27<90:12:35,  8.65s/it][2025-04-26 13:08:05,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:08:05,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.54 | bwd_microstep: 5724.75 | bwd_inner_microstep: 5700.34 | bwd_allreduce_microstep: 24.36 | step_microstep: 18.66
[2025-04-26 13:08:05,002] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.54 | bwd: 5724.76 | bwd_inner: 5700.34 | bwd_allreduce: 24.38 | step: 18.66

  9%|▉         | 3727/41250 [8:59:36<90:15:00,  8.66s/it]
                                                         
{'loss': 0.1832, 'grad_norm': 2.5448989868164062, 'learning_rate': 3.9619296934056907e-05, 'epoch': 0.9}

  9%|▉         | 3727/41250 [8:59:36<90:15:00,  8.66s/it][2025-04-26 13:08:13,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:08:13,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.90 | bwd_microstep: 5718.35 | bwd_inner_microstep: 5648.78 | bwd_allreduce_microstep: 69.52 | step_microstep: 18.62
[2025-04-26 13:08:13,622] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.90 | bwd: 5718.37 | bwd_inner: 5648.78 | bwd_allreduce: 69.54 | step: 18.62

  9%|▉         | 3728/41250 [8:59:44<90:07:40,  8.65s/it]
                                                         
{'loss': 0.1203, 'grad_norm': 2.5177927017211914, 'learning_rate': 3.961899193935006e-05, 'epoch': 0.9}

  9%|▉         | 3728/41250 [8:59:44<90:07:40,  8.65s/it][2025-04-26 13:08:22,329] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:08:22,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.86 | bwd_microstep: 5796.56 | bwd_inner_microstep: 5655.77 | bwd_allreduce_microstep: 140.74 | step_microstep: 18.37
[2025-04-26 13:08:22,330] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.86 | bwd: 5796.57 | bwd_inner: 5655.77 | bwd_allreduce: 140.76 | step: 18.38

  9%|▉         | 3729/41250 [8:59:53<90:18:57,  8.67s/it]
                                                         
{'loss': 0.1679, 'grad_norm': 2.049468994140625, 'learning_rate': 3.9618686823695986e-05, 'epoch': 0.9}

  9%|▉         | 3729/41250 [8:59:53<90:18:57,  8.67s/it][2025-04-26 13:08:30,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 13:08:30,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.37 | bwd_microstep: 5705.80 | bwd_inner_microstep: 5693.16 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.03
[2025-04-26 13:08:30,960] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.37 | bwd: 5705.81 | bwd_inner: 5693.15 | bwd_allreduce: 12.61 | step: 19.04

  9%|▉         | 3730/41250 [9:00:02<90:12:03,  8.65s/it]
                                                         
{'loss': 0.0726, 'grad_norm': 1.62516450881958, 'learning_rate': 3.961838158709657e-05, 'epoch': 0.9}

  9%|▉         | 3730/41250 [9:00:02<90:12:03,  8.65s/it][2025-04-26 13:08:39,643] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:08:39,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.47 | bwd_microstep: 5743.07 | bwd_inner_microstep: 5713.05 | bwd_allreduce_microstep: 29.98 | step_microstep: 18.45
[2025-04-26 13:08:39,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.47 | bwd: 5743.09 | bwd_inner: 5713.05 | bwd_allreduce: 30.00 | step: 18.46

  9%|▉         | 3731/41250 [9:00:10<90:17:23,  8.66s/it]
                                                         
{'loss': 0.2659, 'grad_norm': 3.5965683460235596, 'learning_rate': 3.961807622955369e-05, 'epoch': 0.9}

  9%|▉         | 3731/41250 [9:00:10<90:17:23,  8.66s/it][2025-04-26 13:08:48,312] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:08:48,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.14 | bwd_microstep: 5734.23 | bwd_inner_microstep: 5721.51 | bwd_allreduce_microstep: 12.67 | step_microstep: 18.89
[2025-04-26 13:08:48,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.14 | bwd: 5734.24 | bwd_inner: 5721.51 | bwd_allreduce: 12.69 | step: 18.89

  9%|▉         | 3732/41250 [9:00:19<90:18:21,  8.67s/it]
                                                         
{'loss': 0.2326, 'grad_norm': 1.999025821685791, 'learning_rate': 3.961777075106923e-05, 'epoch': 0.9}

  9%|▉         | 3732/41250 [9:00:19<90:18:21,  8.67s/it][2025-04-26 13:08:57,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:08:57,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.24 | bwd_microstep: 5746.89 | bwd_inner_microstep: 5724.27 | bwd_allreduce_microstep: 22.58 | step_microstep: 18.48
[2025-04-26 13:08:57,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.24 | bwd: 5746.91 | bwd_inner: 5724.27 | bwd_allreduce: 22.60 | step: 18.49

  9%|▉         | 3733/41250 [9:00:28<90:22:32,  8.67s/it]
                                                         
{'loss': 0.1603, 'grad_norm': 2.0448668003082275, 'learning_rate': 3.961746515164506e-05, 'epoch': 0.9}

  9%|▉         | 3733/41250 [9:00:28<90:22:32,  8.67s/it][2025-04-26 13:09:05,653] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:09:05,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.92 | bwd_microstep: 5718.46 | bwd_inner_microstep: 5705.64 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.05
[2025-04-26 13:09:05,654] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.92 | bwd: 5718.47 | bwd_inner: 5705.64 | bwd_allreduce: 12.79 | step: 18.05

  9%|▉         | 3734/41250 [9:00:36<90:18:39,  8.67s/it]
                                                         
{'loss': 0.1409, 'grad_norm': 1.0104036331176758, 'learning_rate': 3.961715943128309e-05, 'epoch': 0.91}

  9%|▉         | 3734/41250 [9:00:36<90:18:39,  8.67s/it][2025-04-26 13:09:14,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:09:14,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.68 | bwd_microstep: 5888.90 | bwd_inner_microstep: 5876.15 | bwd_allreduce_microstep: 12.70 | step_microstep: 18.54
[2025-04-26 13:09:14,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.68 | bwd: 5888.91 | bwd_inner: 5876.15 | bwd_allreduce: 12.72 | step: 18.55

  9%|▉         | 3735/41250 [9:00:45<91:04:11,  8.74s/it]
                                                         
{'loss': 0.4082, 'grad_norm': 2.44724178314209, 'learning_rate': 3.961685358998518e-05, 'epoch': 0.91}

  9%|▉         | 3735/41250 [9:00:45<91:04:11,  8.74s/it][2025-04-26 13:09:23,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 13:09:23,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.88 | bwd_microstep: 5705.47 | bwd_inner_microstep: 5673.78 | bwd_allreduce_microstep: 31.65 | step_microstep: 18.39
[2025-04-26 13:09:23,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.89 | bwd: 5705.49 | bwd_inner: 5673.78 | bwd_allreduce: 31.67 | step: 18.39

  9%|▉         | 3736/41250 [9:00:54<90:42:57,  8.71s/it]
                                                         
{'loss': 0.2765, 'grad_norm': 1.6601859331130981, 'learning_rate': 3.961654762775323e-05, 'epoch': 0.91}

  9%|▉         | 3736/41250 [9:00:54<90:42:57,  8.71s/it][2025-04-26 13:09:31,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:09:31,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.15 | bwd_microstep: 5721.52 | bwd_inner_microstep: 5708.95 | bwd_allreduce_microstep: 12.54 | step_microstep: 18.18
[2025-04-26 13:09:31,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.15 | bwd: 5721.54 | bwd_inner: 5708.95 | bwd_allreduce: 12.55 | step: 18.18

  9%|▉         | 3737/41250 [9:01:03<90:32:25,  8.69s/it]
                                                         
{'loss': 0.0593, 'grad_norm': 0.5482847094535828, 'learning_rate': 3.961624154458913e-05, 'epoch': 0.91}

  9%|▉         | 3737/41250 [9:01:03<90:32:25,  8.69s/it][2025-04-26 13:09:40,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:09:40,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.87 | bwd_microstep: 5793.27 | bwd_inner_microstep: 5668.38 | bwd_allreduce_microstep: 124.85 | step_microstep: 18.11
[2025-04-26 13:09:40,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.87 | bwd: 5793.28 | bwd_inner: 5668.38 | bwd_allreduce: 124.87 | step: 18.12

  9%|▉         | 3738/41250 [9:01:11<90:37:13,  8.70s/it]
                                                         
{'loss': 0.119, 'grad_norm': 1.8732824325561523, 'learning_rate': 3.9615935340494746e-05, 'epoch': 0.91}

  9%|▉         | 3738/41250 [9:01:11<90:37:13,  8.70s/it][2025-04-26 13:09:49,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.95 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:09:49,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.55 | bwd_microstep: 5753.76 | bwd_inner_microstep: 5718.85 | bwd_allreduce_microstep: 34.87 | step_microstep: 17.54
[2025-04-26 13:09:49,244] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.55 | bwd: 5753.77 | bwd_inner: 5718.84 | bwd_allreduce: 34.89 | step: 17.55

  9%|▉         | 3739/41250 [9:01:20<90:35:30,  8.69s/it]
                                                         
{'loss': 0.3073, 'grad_norm': 2.6954081058502197, 'learning_rate': 3.9615629015471985e-05, 'epoch': 0.91}

  9%|▉         | 3739/41250 [9:01:20<90:35:30,  8.69s/it][2025-04-26 13:09:58,069] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:09:58,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2908.28 | bwd_microstep: 5836.22 | bwd_inner_microstep: 5793.18 | bwd_allreduce_microstep: 42.99 | step_microstep: 18.34
[2025-04-26 13:09:58,070] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2908.28 | bwd: 5836.23 | bwd_inner: 5793.18 | bwd_allreduce: 43.01 | step: 18.34

  9%|▉         | 3740/41250 [9:01:29<90:59:58,  8.73s/it]
                                                         
{'loss': 0.1996, 'grad_norm': 1.973943829536438, 'learning_rate': 3.9615322569522735e-05, 'epoch': 0.91}

  9%|▉         | 3740/41250 [9:01:29<90:59:58,  8.73s/it][2025-04-26 13:10:06,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:10:06,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.43 | bwd_microstep: 5768.76 | bwd_inner_microstep: 5666.57 | bwd_allreduce_microstep: 102.15 | step_microstep: 17.96
[2025-04-26 13:10:06,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.43 | bwd: 5768.77 | bwd_inner: 5666.57 | bwd_allreduce: 102.16 | step: 17.97

  9%|▉         | 3741/41250 [9:01:37<90:51:55,  8.72s/it]
                                                         
{'loss': 0.1793, 'grad_norm': 1.7895227670669556, 'learning_rate': 3.961501600264887e-05, 'epoch': 0.91}

  9%|▉         | 3741/41250 [9:01:37<90:51:55,  8.72s/it][2025-04-26 13:10:15,535] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 13:10:15,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.77 | bwd_microstep: 5836.69 | bwd_inner_microstep: 5718.52 | bwd_allreduce_microstep: 118.12 | step_microstep: 18.12
[2025-04-26 13:10:15,536] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.77 | bwd: 5836.70 | bwd_inner: 5718.52 | bwd_allreduce: 118.14 | step: 18.13

  9%|▉         | 3742/41250 [9:01:46<91:01:47,  8.74s/it]
                                                         
{'loss': 0.2757, 'grad_norm': 1.7508307695388794, 'learning_rate': 3.96147093148523e-05, 'epoch': 0.91}

  9%|▉         | 3742/41250 [9:01:46<91:01:47,  8.74s/it][2025-04-26 13:10:24,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:10:24,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.15 | bwd_microstep: 5707.90 | bwd_inner_microstep: 5695.11 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.32
[2025-04-26 13:10:24,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.15 | bwd: 5707.91 | bwd_inner: 5695.11 | bwd_allreduce: 12.76 | step: 18.32

  9%|▉         | 3743/41250 [9:01:55<90:44:00,  8.71s/it]
                                                         
{'loss': 0.215, 'grad_norm': 4.1994194984436035, 'learning_rate': 3.96144025061349e-05, 'epoch': 0.91}

  9%|▉         | 3743/41250 [9:01:55<90:44:00,  8.71s/it][2025-04-26 13:10:32,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:10:32,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.42 | bwd_microstep: 5755.20 | bwd_inner_microstep: 5723.17 | bwd_allreduce_microstep: 31.98 | step_microstep: 18.90
[2025-04-26 13:10:32,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.42 | bwd: 5755.22 | bwd_inner: 5723.17 | bwd_allreduce: 32.00 | step: 18.91

  9%|▉         | 3744/41250 [9:02:04<90:41:43,  8.71s/it]
                                                         
{'loss': 0.0723, 'grad_norm': 0.9869248867034912, 'learning_rate': 3.961409557649856e-05, 'epoch': 0.91}

  9%|▉         | 3744/41250 [9:02:04<90:41:43,  8.71s/it][2025-04-26 13:10:41,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 13:10:41,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.14 | bwd_microstep: 5748.17 | bwd_inner_microstep: 5664.96 | bwd_allreduce_microstep: 83.16 | step_microstep: 18.87
[2025-04-26 13:10:41,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.14 | bwd: 5748.18 | bwd_inner: 5664.96 | bwd_allreduce: 83.18 | step: 18.87

  9%|▉         | 3745/41250 [9:02:12<90:32:49,  8.69s/it]
                                                         
{'loss': 0.3459, 'grad_norm': 2.30454683303833, 'learning_rate': 3.961378852594519e-05, 'epoch': 0.91}

  9%|▉         | 3745/41250 [9:02:12<90:32:49,  8.69s/it][2025-04-26 13:10:50,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:10:50,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.08 | bwd_microstep: 5715.43 | bwd_inner_microstep: 5702.86 | bwd_allreduce_microstep: 12.53 | step_microstep: 18.28
[2025-04-26 13:10:50,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.08 | bwd: 5715.45 | bwd_inner: 5702.86 | bwd_allreduce: 12.55 | step: 18.28

  9%|▉         | 3746/41250 [9:02:21<90:24:40,  8.68s/it]
                                                         
{'loss': 0.1382, 'grad_norm': 1.104875087738037, 'learning_rate': 3.961348135447666e-05, 'epoch': 0.91}

  9%|▉         | 3746/41250 [9:02:21<90:24:40,  8.68s/it][2025-04-26 13:10:58,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:10:58,843] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.18 | bwd_microstep: 5722.09 | bwd_inner_microstep: 5709.37 | bwd_allreduce_microstep: 12.68 | step_microstep: 17.73
[2025-04-26 13:10:58,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.19 | bwd: 5722.11 | bwd_inner: 5709.37 | bwd_allreduce: 12.70 | step: 17.73

  9%|▉         | 3747/41250 [9:02:30<90:21:01,  8.67s/it]
                                                         
{'loss': 0.1477, 'grad_norm': 1.4344209432601929, 'learning_rate': 3.9613174062094885e-05, 'epoch': 0.91}

  9%|▉         | 3747/41250 [9:02:30<90:21:01,  8.67s/it][2025-04-26 13:11:07,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 13:11:07,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.39 | bwd_microstep: 5722.76 | bwd_inner_microstep: 5666.46 | bwd_allreduce_microstep: 56.26 | step_microstep: 17.67
[2025-04-26 13:11:07,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.39 | bwd: 5722.77 | bwd_inner: 5666.46 | bwd_allreduce: 56.27 | step: 17.68

  9%|▉         | 3748/41250 [9:02:38<90:13:36,  8.66s/it]
                                                         
{'loss': 0.1205, 'grad_norm': 0.8232057690620422, 'learning_rate': 3.961286664880174e-05, 'epoch': 0.91}

  9%|▉         | 3748/41250 [9:02:38<90:13:36,  8.66s/it][2025-04-26 13:11:16,107] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.21 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:11:16,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.08 | bwd_microstep: 5714.59 | bwd_inner_microstep: 5647.35 | bwd_allreduce_microstep: 67.20 | step_microstep: 17.82
[2025-04-26 13:11:16,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.08 | bwd: 5714.60 | bwd_inner: 5647.35 | bwd_allreduce: 67.21 | step: 17.82

  9%|▉         | 3749/41250 [9:02:47<90:07:36,  8.65s/it]
                                                         
{'loss': 0.2505, 'grad_norm': 1.3652056455612183, 'learning_rate': 3.9612559114599135e-05, 'epoch': 0.91}

  9%|▉         | 3749/41250 [9:02:47<90:07:36,  8.65s/it][2025-04-26 13:11:24,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:11:24,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.62 | bwd_microstep: 5771.18 | bwd_inner_microstep: 5704.04 | bwd_allreduce_microstep: 67.10 | step_microstep: 18.24
[2025-04-26 13:11:24,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.62 | bwd: 5771.19 | bwd_inner: 5704.04 | bwd_allreduce: 67.11 | step: 18.25

  9%|▉         | 3750/41250 [9:02:55<90:17:28,  8.67s/it]
                                                         
{'loss': 0.2107, 'grad_norm': 0.9867593050003052, 'learning_rate': 3.9612251459488966e-05, 'epoch': 0.91}

  9%|▉         | 3750/41250 [9:02:55<90:17:28,  8.67s/it][2025-04-26 13:11:33,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:11:33,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.45 | bwd_microstep: 5760.69 | bwd_inner_microstep: 5689.01 | bwd_allreduce_microstep: 71.64 | step_microstep: 18.45
[2025-04-26 13:11:33,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.46 | bwd: 5760.71 | bwd_inner: 5689.01 | bwd_allreduce: 71.66 | step: 18.46

  9%|▉         | 3751/41250 [9:03:04<90:21:24,  8.67s/it]
                                                         
{'loss': 0.1187, 'grad_norm': 1.1223429441452026, 'learning_rate': 3.961194368347312e-05, 'epoch': 0.91}

  9%|▉         | 3751/41250 [9:03:04<90:21:24,  8.67s/it][2025-04-26 13:11:42,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:11:42,182] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.21 | bwd_microstep: 5748.19 | bwd_inner_microstep: 5692.14 | bwd_allreduce_microstep: 56.00 | step_microstep: 17.89
[2025-04-26 13:11:42,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.21 | bwd: 5748.20 | bwd_inner: 5692.14 | bwd_allreduce: 56.02 | step: 17.89

  9%|▉         | 3752/41250 [9:03:13<90:22:19,  8.68s/it]
                                                         
{'loss': 0.1245, 'grad_norm': 1.5959521532058716, 'learning_rate': 3.9611635786553495e-05, 'epoch': 0.91}

  9%|▉         | 3752/41250 [9:03:13<90:22:19,  8.68s/it][2025-04-26 13:11:50,852] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:11:50,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.30 | bwd_microstep: 5736.65 | bwd_inner_microstep: 5702.07 | bwd_allreduce_microstep: 34.54 | step_microstep: 17.71
[2025-04-26 13:11:50,853] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.30 | bwd: 5736.66 | bwd_inner: 5702.07 | bwd_allreduce: 34.55 | step: 17.72

  9%|▉         | 3753/41250 [9:03:22<90:21:00,  8.67s/it]
                                                         
{'loss': 0.0318, 'grad_norm': 0.3395027220249176, 'learning_rate': 3.9611327768731994e-05, 'epoch': 0.91}

  9%|▉         | 3753/41250 [9:03:22<90:21:00,  8.67s/it][2025-04-26 13:11:59,539] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 13:11:59,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.90 | bwd_microstep: 5756.96 | bwd_inner_microstep: 5694.31 | bwd_allreduce_microstep: 62.60 | step_microstep: 18.98
[2025-04-26 13:11:59,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.90 | bwd: 5756.97 | bwd_inner: 5694.31 | bwd_allreduce: 62.62 | step: 18.98

  9%|▉         | 3754/41250 [9:03:30<90:23:14,  8.68s/it]
                                                         
{'loss': 0.1406, 'grad_norm': 0.9916378855705261, 'learning_rate': 3.961101963001052e-05, 'epoch': 0.91}

  9%|▉         | 3754/41250 [9:03:30<90:23:14,  8.68s/it][2025-04-26 13:12:08,214] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:12:08,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.12 | bwd_microstep: 5750.16 | bwd_inner_microstep: 5687.01 | bwd_allreduce_microstep: 63.11 | step_microstep: 18.72
[2025-04-26 13:12:08,215] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.12 | bwd: 5750.17 | bwd_inner: 5687.01 | bwd_allreduce: 63.12 | step: 18.72

  9%|▉         | 3755/41250 [9:03:39<90:22:36,  8.68s/it]
                                                         
{'loss': 0.151, 'grad_norm': 1.2269022464752197, 'learning_rate': 3.961071137039097e-05, 'epoch': 0.91}

  9%|▉         | 3755/41250 [9:03:39<90:22:36,  8.68s/it][2025-04-26 13:12:16,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.09 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:12:16,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.70 | bwd_microstep: 5733.43 | bwd_inner_microstep: 5682.63 | bwd_allreduce_microstep: 50.75 | step_microstep: 17.76
[2025-04-26 13:12:16,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.70 | bwd: 5733.45 | bwd_inner: 5682.63 | bwd_allreduce: 50.77 | step: 17.77

  9%|▉         | 3756/41250 [9:03:48<90:19:39,  8.67s/it]
                                                         
{'loss': 0.3587, 'grad_norm': 2.3603296279907227, 'learning_rate': 3.961040298987523e-05, 'epoch': 0.91}

  9%|▉         | 3756/41250 [9:03:48<90:19:39,  8.67s/it][2025-04-26 13:12:25,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:12:25,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.93 | bwd_microstep: 5742.72 | bwd_inner_microstep: 5691.84 | bwd_allreduce_microstep: 50.84 | step_microstep: 18.43
[2025-04-26 13:12:25,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.93 | bwd: 5742.73 | bwd_inner: 5691.84 | bwd_allreduce: 50.86 | step: 18.43

  9%|▉         | 3757/41250 [9:03:56<90:19:00,  8.67s/it]
                                                         
{'loss': 0.1314, 'grad_norm': 1.0207600593566895, 'learning_rate': 3.961009448846521e-05, 'epoch': 0.91}

  9%|▉         | 3757/41250 [9:03:56<90:19:00,  8.67s/it][2025-04-26 13:12:34,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:12:34,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.34 | bwd_microstep: 5679.18 | bwd_inner_microstep: 5655.02 | bwd_allreduce_microstep: 24.12 | step_microstep: 18.56
[2025-04-26 13:12:34,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.34 | bwd: 5679.20 | bwd_inner: 5655.02 | bwd_allreduce: 24.14 | step: 18.56

  9%|▉         | 3758/41250 [9:04:05<90:03:36,  8.65s/it]
                                                         
{'loss': 0.2133, 'grad_norm': 1.995503306388855, 'learning_rate': 3.960978586616283e-05, 'epoch': 0.91}

  9%|▉         | 3758/41250 [9:04:05<90:03:36,  8.65s/it][2025-04-26 13:12:42,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:12:42,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.87 | bwd_microstep: 5722.35 | bwd_inner_microstep: 5709.43 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.93
[2025-04-26 13:12:42,797] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.87 | bwd: 5722.37 | bwd_inner: 5709.43 | bwd_allreduce: 12.89 | step: 18.94

  9%|▉         | 3759/41250 [9:04:13<90:05:51,  8.65s/it]
                                                         
{'loss': 0.1221, 'grad_norm': 0.8477162718772888, 'learning_rate': 3.960947712296997e-05, 'epoch': 0.91}

  9%|▉         | 3759/41250 [9:04:13<90:05:51,  8.65s/it][2025-04-26 13:12:51,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 13:12:51,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.77 | bwd_microstep: 5759.68 | bwd_inner_microstep: 5643.44 | bwd_allreduce_microstep: 116.19 | step_microstep: 18.62
[2025-04-26 13:12:51,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.77 | bwd: 5759.69 | bwd_inner: 5643.44 | bwd_allreduce: 116.21 | step: 18.62

  9%|▉         | 3760/41250 [9:04:22<90:06:43,  8.65s/it]
                                                         
{'loss': 0.1797, 'grad_norm': 1.9883856773376465, 'learning_rate': 3.960916825888854e-05, 'epoch': 0.91}

  9%|▉         | 3760/41250 [9:04:22<90:06:43,  8.65s/it][2025-04-26 13:13:00,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:13:00,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.56 | bwd_microstep: 5707.39 | bwd_inner_microstep: 5694.78 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.46
[2025-04-26 13:13:00,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.56 | bwd: 5707.41 | bwd_inner: 5694.78 | bwd_allreduce: 12.59 | step: 18.46

  9%|▉         | 3761/41250 [9:04:31<90:03:18,  8.65s/it]
                                                         
{'loss': 0.1674, 'grad_norm': 1.0331026315689087, 'learning_rate': 3.960885927392044e-05, 'epoch': 0.91}

  9%|▉         | 3761/41250 [9:04:31<90:03:18,  8.65s/it][2025-04-26 13:13:08,765] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:13:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.66 | bwd_microstep: 5744.84 | bwd_inner_microstep: 5684.30 | bwd_allreduce_microstep: 60.50 | step_microstep: 18.60
[2025-04-26 13:13:08,766] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.66 | bwd: 5744.86 | bwd_inner: 5684.30 | bwd_allreduce: 60.51 | step: 18.60

  9%|▉         | 3762/41250 [9:04:39<90:08:22,  8.66s/it]
                                                         
{'loss': 0.1206, 'grad_norm': 1.5231859683990479, 'learning_rate': 3.960855016806759e-05, 'epoch': 0.91}

  9%|▉         | 3762/41250 [9:04:39<90:08:22,  8.66s/it][2025-04-26 13:13:17,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:13:17,365] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.55 | bwd_microstep: 5691.67 | bwd_inner_microstep: 5646.05 | bwd_allreduce_microstep: 45.57 | step_microstep: 18.54
[2025-04-26 13:13:17,366] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.55 | bwd: 5691.68 | bwd_inner: 5646.05 | bwd_allreduce: 45.58 | step: 18.54

  9%|▉         | 3763/41250 [9:04:48<89:57:32,  8.64s/it]
                                                         
{'loss': 0.0784, 'grad_norm': 1.006325125694275, 'learning_rate': 3.960824094133187e-05, 'epoch': 0.91}

  9%|▉         | 3763/41250 [9:04:48<89:57:32,  8.64s/it][2025-04-26 13:13:26,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 13:13:26,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.47 | bwd_microstep: 5765.12 | bwd_inner_microstep: 5752.24 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.81
[2025-04-26 13:13:26,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.47 | bwd: 5765.13 | bwd_inner: 5752.24 | bwd_allreduce: 12.85 | step: 18.81

  9%|▉         | 3764/41250 [9:04:57<90:12:59,  8.66s/it]
                                                         
{'loss': 0.3433, 'grad_norm': 4.969546794891357, 'learning_rate': 3.960793159371522e-05, 'epoch': 0.91}

  9%|▉         | 3764/41250 [9:04:57<90:12:59,  8.66s/it][2025-04-26 13:13:34,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.02
[2025-04-26 13:13:34,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.88 | bwd_microstep: 5693.11 | bwd_inner_microstep: 5644.41 | bwd_allreduce_microstep: 48.65 | step_microstep: 18.72
[2025-04-26 13:13:34,689] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.88 | bwd: 5693.13 | bwd_inner: 5644.42 | bwd_allreduce: 48.67 | step: 18.72

  9%|▉         | 3765/41250 [9:05:05<90:01:08,  8.65s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 1.260972023010254, 'learning_rate': 3.9607622125219513e-05, 'epoch': 0.91}

  9%|▉         | 3765/41250 [9:05:05<90:01:08,  8.65s/it][2025-04-26 13:13:43,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 1.06
[2025-04-26 13:13:43,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.54 | bwd_microstep: 5755.70 | bwd_inner_microstep: 5699.64 | bwd_allreduce_microstep: 56.01 | step_microstep: 19.01
[2025-04-26 13:13:43,386] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.54 | bwd: 5755.72 | bwd_inner: 5699.64 | bwd_allreduce: 56.03 | step: 19.01

  9%|▉         | 3766/41250 [9:05:14<90:11:28,  8.66s/it]
                                                         
{'loss': 0.1176, 'grad_norm': 1.2703691720962524, 'learning_rate': 3.960731253584668e-05, 'epoch': 0.91}

  9%|▉         | 3766/41250 [9:05:14<90:11:28,  8.66s/it][2025-04-26 13:13:52,005] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 1.16
[2025-04-26 13:13:52,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.95 | bwd_microstep: 5705.97 | bwd_inner_microstep: 5649.82 | bwd_allreduce_microstep: 56.09 | step_microstep: 19.20
[2025-04-26 13:13:52,006] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.95 | bwd: 5705.98 | bwd_inner: 5649.82 | bwd_allreduce: 56.11 | step: 19.21

  9%|▉         | 3767/41250 [9:05:23<90:02:42,  8.65s/it]
                                                         
{'loss': 0.3141, 'grad_norm': 1.8596237897872925, 'learning_rate': 3.960700282559862e-05, 'epoch': 0.91}

  9%|▉         | 3767/41250 [9:05:23<90:02:42,  8.65s/it][2025-04-26 13:14:00,623] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:14:00,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.07 | bwd_microstep: 5706.11 | bwd_inner_microstep: 5660.23 | bwd_allreduce_microstep: 45.84 | step_microstep: 18.74
[2025-04-26 13:14:00,624] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.07 | bwd: 5706.12 | bwd_inner: 5660.23 | bwd_allreduce: 45.86 | step: 18.75

  9%|▉         | 3768/41250 [9:05:31<89:56:49,  8.64s/it]
                                                         
{'loss': 0.3726, 'grad_norm': 1.8019342422485352, 'learning_rate': 3.960669299447724e-05, 'epoch': 0.91}

  9%|▉         | 3768/41250 [9:05:31<89:56:49,  8.64s/it][2025-04-26 13:14:09,315] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:14:09,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.48 | bwd_microstep: 5764.89 | bwd_inner_microstep: 5698.99 | bwd_allreduce_microstep: 65.85 | step_microstep: 19.16
[2025-04-26 13:14:09,316] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.48 | bwd: 5764.91 | bwd_inner: 5698.99 | bwd_allreduce: 65.87 | step: 19.16

  9%|▉         | 3769/41250 [9:05:40<90:06:35,  8.65s/it]
                                                         
{'loss': 0.2586, 'grad_norm': 3.4355015754699707, 'learning_rate': 3.9606383042484456e-05, 'epoch': 0.91}

  9%|▉         | 3769/41250 [9:05:40<90:06:35,  8.65s/it][2025-04-26 13:14:18,000] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:14:18,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5750.43 | bwd_inner_microstep: 5703.81 | bwd_allreduce_microstep: 46.57 | step_microstep: 18.89
[2025-04-26 13:14:18,001] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.26 | bwd: 5750.45 | bwd_inner: 5703.81 | bwd_allreduce: 46.59 | step: 18.89

  9%|▉         | 3770/41250 [9:05:49<90:12:06,  8.66s/it]
                                                         
{'loss': 0.1669, 'grad_norm': 1.3569881916046143, 'learning_rate': 3.960607296962218e-05, 'epoch': 0.91}

  9%|▉         | 3770/41250 [9:05:49<90:12:06,  8.66s/it][2025-04-26 13:14:26,646] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.33 | optimizer_step: 1.06
[2025-04-26 13:14:26,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.61 | bwd_microstep: 5722.67 | bwd_inner_microstep: 5655.82 | bwd_allreduce_microstep: 66.78 | step_microstep: 20.03
[2025-04-26 13:14:26,647] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.61 | bwd: 5722.69 | bwd_inner: 5655.82 | bwd_allreduce: 66.82 | step: 20.03

  9%|▉         | 3771/41250 [9:05:57<90:08:48,  8.66s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.423379898071289, 'learning_rate': 3.9605762775892326e-05, 'epoch': 0.91}

  9%|▉         | 3771/41250 [9:05:57<90:08:48,  8.66s/it][2025-04-26 13:14:35,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.25 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 13:14:35,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.38 | bwd_microstep: 5733.27 | bwd_inner_microstep: 5707.50 | bwd_allreduce_microstep: 25.72 | step_microstep: 18.68
[2025-04-26 13:14:35,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.38 | bwd: 5733.28 | bwd_inner: 5707.50 | bwd_allreduce: 25.74 | step: 18.68

  9%|▉         | 3772/41250 [9:06:06<90:11:37,  8.66s/it]
                                                         
{'loss': 0.6054, 'grad_norm': 2.2972006797790527, 'learning_rate': 3.960545246129679e-05, 'epoch': 0.91}

  9%|▉         | 3772/41250 [9:06:06<90:11:37,  8.66s/it][2025-04-26 13:14:43,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:14:43,921] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.96 | bwd_microstep: 5701.73 | bwd_inner_microstep: 5654.94 | bwd_allreduce_microstep: 46.75 | step_microstep: 18.80
[2025-04-26 13:14:43,922] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.96 | bwd: 5701.75 | bwd_inner: 5654.94 | bwd_allreduce: 46.77 | step: 18.80

  9%|▉         | 3773/41250 [9:06:15<89:59:14,  8.64s/it]
                                                         
{'loss': 0.2212, 'grad_norm': 1.3612147569656372, 'learning_rate': 3.96051420258375e-05, 'epoch': 0.91}

  9%|▉         | 3773/41250 [9:06:15<89:59:14,  8.64s/it][2025-04-26 13:14:52,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 1.00 | optimizer_step: 1.04
[2025-04-26 13:14:52,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.47 | bwd_microstep: 5944.78 | bwd_inner_microstep: 5639.39 | bwd_allreduce_microstep: 305.34 | step_microstep: 18.58
[2025-04-26 13:14:52,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.47 | bwd: 5944.80 | bwd_inner: 5639.39 | bwd_allreduce: 305.37 | step: 18.58

  9%|▉         | 3774/41250 [9:06:23<90:37:35,  8.71s/it]
                                                         
{'loss': 0.2162, 'grad_norm': 2.68264102935791, 'learning_rate': 3.960483146951636e-05, 'epoch': 0.91}

  9%|▉         | 3774/41250 [9:06:23<90:37:35,  8.71s/it][2025-04-26 13:15:01,412] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 13:15:01,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.50 | bwd_microstep: 5730.84 | bwd_inner_microstep: 5650.85 | bwd_allreduce_microstep: 79.95 | step_microstep: 18.52
[2025-04-26 13:15:01,413] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.50 | bwd: 5730.85 | bwd_inner: 5650.85 | bwd_allreduce: 79.96 | step: 18.52

  9%|▉         | 3775/41250 [9:06:32<90:25:25,  8.69s/it]
                                                         
{'loss': 0.1259, 'grad_norm': 1.3529375791549683, 'learning_rate': 3.9604520792335305e-05, 'epoch': 0.92}

  9%|▉         | 3775/41250 [9:06:32<90:25:25,  8.69s/it][2025-04-26 13:15:10,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.08
[2025-04-26 13:15:10,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.46 | bwd_microstep: 5780.62 | bwd_inner_microstep: 5696.95 | bwd_allreduce_microstep: 83.63 | step_microstep: 18.69
[2025-04-26 13:15:10,125] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.46 | bwd: 5780.63 | bwd_inner: 5696.95 | bwd_allreduce: 83.64 | step: 18.70

  9%|▉         | 3776/41250 [9:06:41<90:30:12,  8.69s/it]
                                                         
{'loss': 0.1678, 'grad_norm': 2.2323055267333984, 'learning_rate': 3.960420999429622e-05, 'epoch': 0.92}

  9%|▉         | 3776/41250 [9:06:41<90:30:12,  8.69s/it][2025-04-26 13:15:18,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.94
[2025-04-26 13:15:18,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.86 | bwd_microstep: 5765.41 | bwd_inner_microstep: 5658.37 | bwd_allreduce_microstep: 107.00 | step_microstep: 18.60
[2025-04-26 13:15:18,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.86 | bwd: 5765.43 | bwd_inner: 5658.37 | bwd_allreduce: 107.02 | step: 18.61

  9%|▉         | 3777/41250 [9:06:50<90:29:31,  8.69s/it]
                                                         
{'loss': 0.1075, 'grad_norm': 0.7978087663650513, 'learning_rate': 3.960389907540104e-05, 'epoch': 0.92}

  9%|▉         | 3777/41250 [9:06:50<90:29:31,  8.69s/it][2025-04-26 13:15:27,532] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:15:27,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.36 | bwd_microstep: 5779.49 | bwd_inner_microstep: 5708.22 | bwd_allreduce_microstep: 71.23 | step_microstep: 18.33
[2025-04-26 13:15:27,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.36 | bwd: 5779.51 | bwd_inner: 5708.22 | bwd_allreduce: 71.25 | step: 18.33

  9%|▉         | 3778/41250 [9:06:58<90:33:30,  8.70s/it]
                                                         
{'loss': 0.0907, 'grad_norm': 1.694434642791748, 'learning_rate': 3.9603588035651686e-05, 'epoch': 0.92}

  9%|▉         | 3778/41250 [9:06:58<90:33:30,  8.70s/it][2025-04-26 13:15:36,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:15:36,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.36 | bwd_microstep: 5776.73 | bwd_inner_microstep: 5721.66 | bwd_allreduce_microstep: 55.03 | step_microstep: 18.35
[2025-04-26 13:15:36,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.36 | bwd: 5776.74 | bwd_inner: 5721.66 | bwd_allreduce: 55.05 | step: 18.35

  9%|▉         | 3779/41250 [9:07:07<90:35:50,  8.70s/it]
                                                         
{'loss': 0.0751, 'grad_norm': 0.7815191149711609, 'learning_rate': 3.9603276875050055e-05, 'epoch': 0.92}

  9%|▉         | 3779/41250 [9:07:07<90:35:50,  8.70s/it][2025-04-26 13:15:44,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:15:44,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.96 | bwd_microstep: 5729.78 | bwd_inner_microstep: 5657.58 | bwd_allreduce_microstep: 72.16 | step_microstep: 17.89
[2025-04-26 13:15:44,891] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.95 | bwd: 5729.79 | bwd_inner: 5657.58 | bwd_allreduce: 72.17 | step: 17.89

  9%|▉         | 3780/41250 [9:07:16<90:24:41,  8.69s/it]
                                                         
{'loss': 0.218, 'grad_norm': 1.0130491256713867, 'learning_rate': 3.960296559359809e-05, 'epoch': 0.92}

  9%|▉         | 3780/41250 [9:07:16<90:24:41,  8.69s/it][2025-04-26 13:15:53,578] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:15:53,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.65 | bwd_microstep: 5751.28 | bwd_inner_microstep: 5699.93 | bwd_allreduce_microstep: 51.30 | step_microstep: 18.40
[2025-04-26 13:15:53,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.65 | bwd: 5751.29 | bwd_inner: 5699.93 | bwd_allreduce: 51.32 | step: 18.40

  9%|▉         | 3781/41250 [9:07:24<90:24:42,  8.69s/it]
                                                         
{'loss': 0.124, 'grad_norm': 0.8486618995666504, 'learning_rate': 3.960265419129769e-05, 'epoch': 0.92}

  9%|▉         | 3781/41250 [9:07:24<90:24:42,  8.69s/it][2025-04-26 13:16:02,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:16:02,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.78 | bwd_microstep: 5762.72 | bwd_inner_microstep: 5719.58 | bwd_allreduce_microstep: 43.09 | step_microstep: 18.35
[2025-04-26 13:16:02,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.78 | bwd: 5762.73 | bwd_inner: 5719.58 | bwd_allreduce: 43.11 | step: 18.35

  9%|▉         | 3782/41250 [9:07:33<90:26:36,  8.69s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 4.429182052612305, 'learning_rate': 3.960234266815078e-05, 'epoch': 0.92}

  9%|▉         | 3782/41250 [9:07:33<90:26:36,  8.69s/it][2025-04-26 13:16:11,059] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:16:11,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2904.50 | bwd_microstep: 5797.02 | bwd_inner_microstep: 5784.34 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.37
[2025-04-26 13:16:11,060] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2904.50 | bwd: 5797.04 | bwd_inner: 5784.34 | bwd_allreduce: 12.66 | step: 18.37

  9%|▉         | 3783/41250 [9:07:42<90:43:59,  8.72s/it]
                                                         
{'loss': 0.2477, 'grad_norm': 1.3373639583587646, 'learning_rate': 3.960203102415929e-05, 'epoch': 0.92}

  9%|▉         | 3783/41250 [9:07:42<90:43:59,  8.72s/it][2025-04-26 13:16:19,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.98
[2025-04-26 13:16:19,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.08 | bwd_microstep: 5777.62 | bwd_inner_microstep: 5673.51 | bwd_allreduce_microstep: 104.07 | step_microstep: 18.85
[2025-04-26 13:16:19,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.08 | bwd: 5777.64 | bwd_inner: 5673.50 | bwd_allreduce: 104.09 | step: 18.85

  9%|▉         | 3784/41250 [9:07:50<90:40:55,  8.71s/it]
                                                         
{'loss': 0.1518, 'grad_norm': 1.150631308555603, 'learning_rate': 3.9601719259325136e-05, 'epoch': 0.92}

  9%|▉         | 3784/41250 [9:07:50<90:40:55,  8.71s/it][2025-04-26 13:16:28,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 13:16:28,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.65 | bwd_microstep: 5746.51 | bwd_inner_microstep: 5696.30 | bwd_allreduce_microstep: 50.17 | step_microstep: 18.23
[2025-04-26 13:16:28,443] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.65 | bwd: 5746.53 | bwd_inner: 5696.30 | bwd_allreduce: 50.18 | step: 18.24

  9%|▉         | 3785/41250 [9:07:59<90:34:49,  8.70s/it]
                                                         
{'loss': 0.1605, 'grad_norm': 1.1639281511306763, 'learning_rate': 3.960140737365024e-05, 'epoch': 0.92}

  9%|▉         | 3785/41250 [9:07:59<90:34:49,  8.70s/it][2025-04-26 13:16:37,103] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 0.96
[2025-04-26 13:16:37,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.61 | bwd_microstep: 5722.57 | bwd_inner_microstep: 5709.84 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.34
[2025-04-26 13:16:37,104] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.61 | bwd: 5722.58 | bwd_inner: 5709.84 | bwd_allreduce: 12.71 | step: 18.34

  9%|▉         | 3786/41250 [9:08:08<90:26:32,  8.69s/it]
                                                         
{'loss': 0.1099, 'grad_norm': 1.6125434637069702, 'learning_rate': 3.9601095367136506e-05, 'epoch': 0.92}

  9%|▉         | 3786/41250 [9:08:08<90:26:32,  8.69s/it][2025-04-26 13:16:45,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:16:45,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.23 | bwd_microstep: 5765.37 | bwd_inner_microstep: 5658.17 | bwd_allreduce_microstep: 107.15 | step_microstep: 18.32
[2025-04-26 13:16:45,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.23 | bwd: 5765.38 | bwd_inner: 5658.18 | bwd_allreduce: 107.17 | step: 18.33

  9%|▉         | 3787/41250 [9:08:16<90:25:01,  8.69s/it]
                                                         
{'loss': 0.1728, 'grad_norm': 0.9941138625144958, 'learning_rate': 3.9600783239785895e-05, 'epoch': 0.92}

  9%|▉         | 3787/41250 [9:08:16<90:25:01,  8.69s/it][2025-04-26 13:16:54,472] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:16:54,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.44 | bwd_microstep: 5768.54 | bwd_inner_microstep: 5655.31 | bwd_allreduce_microstep: 113.18 | step_microstep: 18.33
[2025-04-26 13:16:54,473] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.44 | bwd: 5768.55 | bwd_inner: 5655.31 | bwd_allreduce: 113.20 | step: 18.33

  9%|▉         | 3788/41250 [9:08:25<90:24:20,  8.69s/it]
                                                         
{'loss': 0.1984, 'grad_norm': 1.1230857372283936, 'learning_rate': 3.9600470991600295e-05, 'epoch': 0.92}

  9%|▉         | 3788/41250 [9:08:25<90:24:20,  8.69s/it][2025-04-26 13:17:03,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:17:03,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.18 | bwd_microstep: 5717.47 | bwd_inner_microstep: 5704.73 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.58
[2025-04-26 13:17:03,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.18 | bwd: 5717.48 | bwd_inner: 5704.73 | bwd_allreduce: 12.71 | step: 18.58

  9%|▉         | 3789/41250 [9:08:34<90:17:20,  8.68s/it]
                                                         
{'loss': 0.2765, 'grad_norm': 1.683679223060608, 'learning_rate': 3.960015862258166e-05, 'epoch': 0.92}

  9%|▉         | 3789/41250 [9:08:34<90:17:20,  8.68s/it][2025-04-26 13:17:11,806] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:17:11,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.00 | bwd_microstep: 5753.78 | bwd_inner_microstep: 5695.05 | bwd_allreduce_microstep: 58.69 | step_microstep: 18.17
[2025-04-26 13:17:11,807] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.00 | bwd: 5753.79 | bwd_inner: 5695.05 | bwd_allreduce: 58.70 | step: 18.17

  9%|▉         | 3790/41250 [9:08:42<90:18:17,  8.68s/it]
                                                         
{'loss': 0.1607, 'grad_norm': 2.0458145141601562, 'learning_rate': 3.959984613273189e-05, 'epoch': 0.92}

  9%|▉         | 3790/41250 [9:08:42<90:18:17,  8.68s/it][2025-04-26 13:17:20,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:17:20,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.22 | bwd_microstep: 5757.67 | bwd_inner_microstep: 5700.92 | bwd_allreduce_microstep: 56.71 | step_microstep: 18.05
[2025-04-26 13:17:20,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.22 | bwd: 5757.69 | bwd_inner: 5700.92 | bwd_allreduce: 56.73 | step: 18.05

  9%|▉         | 3791/41250 [9:08:51<90:21:00,  8.68s/it]
                                                         
{'loss': 0.2983, 'grad_norm': 1.8895741701126099, 'learning_rate': 3.959953352205293e-05, 'epoch': 0.92}

  9%|▉         | 3791/41250 [9:08:51<90:21:00,  8.68s/it][2025-04-26 13:17:29,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:17:29,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.55 | bwd_microstep: 5762.25 | bwd_inner_microstep: 5656.57 | bwd_allreduce_microstep: 105.64 | step_microstep: 17.96
[2025-04-26 13:17:29,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.55 | bwd: 5762.27 | bwd_inner: 5656.57 | bwd_allreduce: 105.65 | step: 17.96

  9%|▉         | 3792/41250 [9:09:00<90:20:41,  8.68s/it]
                                                         
{'loss': 0.167, 'grad_norm': 1.1689777374267578, 'learning_rate': 3.95992207905467e-05, 'epoch': 0.92}

  9%|▉         | 3792/41250 [9:09:00<90:20:41,  8.68s/it][2025-04-26 13:17:37,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:17:37,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.80 | bwd_microstep: 5689.14 | bwd_inner_microstep: 5648.60 | bwd_allreduce_microstep: 40.49 | step_microstep: 17.85
[2025-04-26 13:17:37,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.80 | bwd: 5689.15 | bwd_inner: 5648.60 | bwd_allreduce: 40.51 | step: 17.85

  9%|▉         | 3793/41250 [9:09:08<90:04:53,  8.66s/it]
                                                         
{'loss': 0.088, 'grad_norm': 1.4247181415557861, 'learning_rate': 3.9598907938215126e-05, 'epoch': 0.92}

  9%|▉         | 3793/41250 [9:09:08<90:04:53,  8.66s/it][2025-04-26 13:17:46,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:17:46,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5754.11 | bwd_inner_microstep: 5693.02 | bwd_allreduce_microstep: 61.05 | step_microstep: 18.42
[2025-04-26 13:17:46,469] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5754.13 | bwd_inner: 5693.02 | bwd_allreduce: 61.06 | step: 18.43

  9%|▉         | 3794/41250 [9:09:17<90:10:16,  8.67s/it]
                                                         
{'loss': 0.2527, 'grad_norm': 2.0643439292907715, 'learning_rate': 3.9598594965060145e-05, 'epoch': 0.92}

  9%|▉         | 3794/41250 [9:09:17<90:10:16,  8.67s/it][2025-04-26 13:17:55,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 13:17:55,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.78 | bwd_microstep: 5747.00 | bwd_inner_microstep: 5651.60 | bwd_allreduce_microstep: 95.36 | step_microstep: 18.92
[2025-04-26 13:17:55,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.78 | bwd: 5747.02 | bwd_inner: 5651.60 | bwd_allreduce: 95.37 | step: 18.92

  9%|▉         | 3795/41250 [9:09:26<90:10:44,  8.67s/it]
                                                         
{'loss': 0.1439, 'grad_norm': 2.761568784713745, 'learning_rate': 3.959828187108367e-05, 'epoch': 0.92}

  9%|▉         | 3795/41250 [9:09:26<90:10:44,  8.67s/it][2025-04-26 13:18:03,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:18:03,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.63 | bwd_microstep: 5710.22 | bwd_inner_microstep: 5657.53 | bwd_allreduce_microstep: 52.65 | step_microstep: 18.24
[2025-04-26 13:18:03,755] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.63 | bwd: 5710.23 | bwd_inner: 5657.53 | bwd_allreduce: 52.66 | step: 18.24

  9%|▉         | 3796/41250 [9:09:34<90:00:48,  8.65s/it]
                                                         
{'loss': 0.1283, 'grad_norm': 0.8696309328079224, 'learning_rate': 3.9597968656287655e-05, 'epoch': 0.92}

  9%|▉         | 3796/41250 [9:09:34<90:00:48,  8.65s/it][2025-04-26 13:18:12,415] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:18:12,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.48 | bwd_microstep: 5730.87 | bwd_inner_microstep: 5683.88 | bwd_allreduce_microstep: 46.95 | step_microstep: 18.06
[2025-04-26 13:18:12,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.48 | bwd: 5730.88 | bwd_inner: 5683.88 | bwd_allreduce: 46.96 | step: 18.06

  9%|▉         | 3797/41250 [9:09:43<90:02:12,  8.65s/it]
                                                         
{'loss': 0.0865, 'grad_norm': 1.3196665048599243, 'learning_rate': 3.959765532067401e-05, 'epoch': 0.92}

  9%|▉         | 3797/41250 [9:09:43<90:02:12,  8.65s/it][2025-04-26 13:18:21,087] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:18:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.81 | bwd_microstep: 5761.09 | bwd_inner_microstep: 5652.85 | bwd_allreduce_microstep: 108.20 | step_microstep: 18.20
[2025-04-26 13:18:21,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.81 | bwd: 5761.11 | bwd_inner: 5652.85 | bwd_allreduce: 108.21 | step: 18.20

  9%|▉         | 3798/41250 [9:09:52<90:05:27,  8.66s/it]
                                                         
{'loss': 0.1287, 'grad_norm': 0.6396167278289795, 'learning_rate': 3.9597341864244684e-05, 'epoch': 0.92}

  9%|▉         | 3798/41250 [9:09:52<90:05:27,  8.66s/it][2025-04-26 13:18:29,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:18:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.06 | bwd_microstep: 5760.73 | bwd_inner_microstep: 5659.50 | bwd_allreduce_microstep: 101.18 | step_microstep: 17.80
[2025-04-26 13:18:29,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.06 | bwd: 5760.75 | bwd_inner: 5659.50 | bwd_allreduce: 101.20 | step: 17.81

  9%|▉         | 3799/41250 [9:10:00<90:10:30,  8.67s/it]
                                                         
{'loss': 0.1212, 'grad_norm': 1.0548087358474731, 'learning_rate': 3.959702828700159e-05, 'epoch': 0.92}

  9%|▉         | 3799/41250 [9:10:00<90:10:30,  8.67s/it][2025-04-26 13:18:38,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:18:38,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.74 | bwd_microstep: 5724.04 | bwd_inner_microstep: 5683.60 | bwd_allreduce_microstep: 40.40 | step_microstep: 18.00
[2025-04-26 13:18:38,420] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.74 | bwd: 5724.05 | bwd_inner: 5683.60 | bwd_allreduce: 40.42 | step: 18.01

  9%|▉         | 3800/41250 [9:10:09<90:05:56,  8.66s/it]
                                                         
{'loss': 0.2431, 'grad_norm': 1.1578569412231445, 'learning_rate': 3.9596714588946675e-05, 'epoch': 0.92}

  9%|▉         | 3800/41250 [9:10:09<90:05:56,  8.66s/it][2025-04-26 13:18:47,026] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.99
[2025-04-26 13:18:47,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.35 | bwd_microstep: 5696.36 | bwd_inner_microstep: 5657.94 | bwd_allreduce_microstep: 38.37 | step_microstep: 18.77
[2025-04-26 13:18:47,027] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.35 | bwd: 5696.37 | bwd_inner: 5657.94 | bwd_allreduce: 38.39 | step: 18.78

  9%|▉         | 3801/41250 [9:10:18<89:55:39,  8.64s/it]
                                                         
{'loss': 0.1485, 'grad_norm': 1.2388664484024048, 'learning_rate': 3.959640077008187e-05, 'epoch': 0.92}

  9%|▉         | 3801/41250 [9:10:18<89:55:39,  8.64s/it][2025-04-26 13:18:55,690] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.99 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:18:55,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.77 | bwd_microstep: 5753.12 | bwd_inner_microstep: 5654.52 | bwd_allreduce_microstep: 98.56 | step_microstep: 17.48
[2025-04-26 13:18:55,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.77 | bwd: 5753.14 | bwd_inner: 5654.52 | bwd_allreduce: 98.58 | step: 17.47

  9%|▉         | 3802/41250 [9:10:26<89:59:03,  8.65s/it]
                                                         
{'loss': 0.1558, 'grad_norm': 0.8712134957313538, 'learning_rate': 3.9596086830409106e-05, 'epoch': 0.92}

  9%|▉         | 3802/41250 [9:10:26<89:59:03,  8.65s/it][2025-04-26 13:19:04,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:19:04,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.32 | bwd_microstep: 5768.43 | bwd_inner_microstep: 5646.40 | bwd_allreduce_microstep: 121.99 | step_microstep: 17.99
[2025-04-26 13:19:04,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.32 | bwd: 5768.45 | bwd_inner: 5646.40 | bwd_allreduce: 122.01 | step: 17.99

  9%|▉         | 3803/41250 [9:10:35<90:05:06,  8.66s/it]
                                                         
{'loss': 0.0887, 'grad_norm': 1.0312607288360596, 'learning_rate': 3.959577276993032e-05, 'epoch': 0.92}

  9%|▉         | 3803/41250 [9:10:35<90:05:06,  8.66s/it][2025-04-26 13:19:13,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:19:13,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.13 | bwd_microstep: 5887.67 | bwd_inner_microstep: 5655.31 | bwd_allreduce_microstep: 232.32 | step_microstep: 18.69
[2025-04-26 13:19:13,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.13 | bwd: 5887.69 | bwd_inner: 5655.30 | bwd_allreduce: 232.34 | step: 18.69

  9%|▉         | 3804/41250 [9:10:44<90:30:38,  8.70s/it]
                                                         
{'loss': 0.1295, 'grad_norm': 1.232110857963562, 'learning_rate': 3.959545858864745e-05, 'epoch': 0.92}

  9%|▉         | 3804/41250 [9:10:44<90:30:38,  8.70s/it][2025-04-26 13:19:21,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:19:21,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.56 | bwd_microstep: 5792.52 | bwd_inner_microstep: 5779.71 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.74
[2025-04-26 13:19:21,928] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2889.56 | bwd: 5792.53 | bwd_inner: 5779.71 | bwd_allreduce: 12.78 | step: 18.74

  9%|▉         | 3805/41250 [9:10:53<90:41:15,  8.72s/it]
                                                         
{'loss': 0.1277, 'grad_norm': 0.91877281665802, 'learning_rate': 3.959514428656244e-05, 'epoch': 0.92}

  9%|▉         | 3805/41250 [9:10:53<90:41:15,  8.72s/it][2025-04-26 13:19:30,599] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.07
[2025-04-26 13:19:30,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.07 | bwd_microstep: 5759.65 | bwd_inner_microstep: 5658.20 | bwd_allreduce_microstep: 101.40 | step_microstep: 18.74
[2025-04-26 13:19:30,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.07 | bwd: 5759.66 | bwd_inner: 5658.20 | bwd_allreduce: 101.42 | step: 18.74

  9%|▉         | 3806/41250 [9:11:01<90:31:45,  8.70s/it]
                                                         
{'loss': 0.1858, 'grad_norm': 1.1141709089279175, 'learning_rate': 3.959482986367721e-05, 'epoch': 0.92}

  9%|▉         | 3806/41250 [9:11:01<90:31:45,  8.70s/it][2025-04-26 13:19:39,260] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:19:39,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.42 | bwd_microstep: 5745.39 | bwd_inner_microstep: 5643.84 | bwd_allreduce_microstep: 101.51 | step_microstep: 18.51
[2025-04-26 13:19:39,261] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.42 | bwd: 5745.41 | bwd_inner: 5643.83 | bwd_allreduce: 101.53 | step: 18.52

  9%|▉         | 3807/41250 [9:11:10<90:24:03,  8.69s/it]
                                                         
{'loss': 0.2361, 'grad_norm': 0.9727767109870911, 'learning_rate': 3.959451531999371e-05, 'epoch': 0.92}

  9%|▉         | 3807/41250 [9:11:10<90:24:03,  8.69s/it][2025-04-26 13:19:47,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:19:47,912] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.42 | bwd_microstep: 5714.61 | bwd_inner_microstep: 5695.59 | bwd_allreduce_microstep: 18.98 | step_microstep: 17.75
[2025-04-26 13:19:47,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.42 | bwd: 5714.62 | bwd_inner: 5695.59 | bwd_allreduce: 19.00 | step: 17.75

  9%|▉         | 3808/41250 [9:11:19<90:15:57,  8.68s/it]
                                                         
{'loss': 0.0919, 'grad_norm': 0.7903253436088562, 'learning_rate': 3.959420065551387e-05, 'epoch': 0.92}

  9%|▉         | 3808/41250 [9:11:19<90:15:57,  8.68s/it][2025-04-26 13:19:56,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:19:56,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.33 | bwd_microstep: 5770.30 | bwd_inner_microstep: 5653.43 | bwd_allreduce_microstep: 116.82 | step_microstep: 18.58
[2025-04-26 13:19:56,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.33 | bwd: 5770.31 | bwd_inner: 5653.43 | bwd_allreduce: 116.84 | step: 18.59

  9%|▉         | 3809/41250 [9:11:27<90:15:05,  8.68s/it]
                                                         
{'loss': 0.1583, 'grad_norm': 1.4814010858535767, 'learning_rate': 3.959388587023965e-05, 'epoch': 0.92}

  9%|▉         | 3809/41250 [9:11:27<90:15:05,  8.68s/it][2025-04-26 13:20:05,237] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:20:05,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.94 | bwd_microstep: 5718.29 | bwd_inner_microstep: 5705.53 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.55
[2025-04-26 13:20:05,238] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.94 | bwd: 5718.30 | bwd_inner: 5705.53 | bwd_allreduce: 12.74 | step: 18.55

  9%|▉         | 3810/41250 [9:11:36<90:10:01,  8.67s/it]
                                                         
{'loss': 0.2409, 'grad_norm': 2.1562037467956543, 'learning_rate': 3.9593570964172973e-05, 'epoch': 0.92}

  9%|▉         | 3810/41250 [9:11:36<90:10:01,  8.67s/it][2025-04-26 13:20:13,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:20:13,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.53 | bwd_microstep: 5706.78 | bwd_inner_microstep: 5694.02 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.68
[2025-04-26 13:20:13,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.53 | bwd: 5706.79 | bwd_inner: 5694.02 | bwd_allreduce: 12.73 | step: 18.68

  9%|▉         | 3811/41250 [9:11:45<90:04:26,  8.66s/it]
                                                         
{'loss': 0.177, 'grad_norm': 1.4411187171936035, 'learning_rate': 3.9593255937315786e-05, 'epoch': 0.92}

  9%|▉         | 3811/41250 [9:11:45<90:04:26,  8.66s/it][2025-04-26 13:20:22,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:20:22,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.62 | bwd_microstep: 5750.10 | bwd_inner_microstep: 5718.45 | bwd_allreduce_microstep: 31.60 | step_microstep: 18.61
[2025-04-26 13:20:22,568] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.62 | bwd: 5750.12 | bwd_inner: 5718.45 | bwd_allreduce: 31.62 | step: 18.61

  9%|▉         | 3812/41250 [9:11:53<90:09:30,  8.67s/it]
                                                         
{'loss': 0.0731, 'grad_norm': 2.622356653213501, 'learning_rate': 3.959294078967003e-05, 'epoch': 0.92}

  9%|▉         | 3812/41250 [9:11:53<90:09:30,  8.67s/it][2025-04-26 13:20:31,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:20:31,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.58 | bwd_microstep: 5770.22 | bwd_inner_microstep: 5703.99 | bwd_allreduce_microstep: 66.19 | step_microstep: 18.70
[2025-04-26 13:20:31,268] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.58 | bwd: 5770.23 | bwd_inner: 5703.98 | bwd_allreduce: 66.20 | step: 18.71

  9%|▉         | 3813/41250 [9:12:02<90:15:10,  8.68s/it]
                                                         
{'loss': 0.1502, 'grad_norm': 1.0599185228347778, 'learning_rate': 3.959262552123764e-05, 'epoch': 0.92}

  9%|▉         | 3813/41250 [9:12:02<90:15:10,  8.68s/it][2025-04-26 13:20:40,222] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:20:40,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.34 | bwd_microstep: 6025.77 | bwd_inner_microstep: 5698.43 | bwd_allreduce_microstep: 327.29 | step_microstep: 18.77
[2025-04-26 13:20:40,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.34 | bwd: 6025.78 | bwd_inner: 5698.43 | bwd_allreduce: 327.31 | step: 18.77

  9%|▉         | 3814/41250 [9:12:11<91:06:40,  8.76s/it]
                                                         
{'loss': 0.3555, 'grad_norm': 2.278977394104004, 'learning_rate': 3.959231013202057e-05, 'epoch': 0.92}

  9%|▉         | 3814/41250 [9:12:11<91:06:40,  8.76s/it][2025-04-26 13:20:48,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.03 | optimizer_step: 1.15
[2025-04-26 13:20:48,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.18 | bwd_microstep: 5693.64 | bwd_inner_microstep: 5644.83 | bwd_allreduce_microstep: 48.76 | step_microstep: 19.26
[2025-04-26 13:20:48,824] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.18 | bwd: 5693.65 | bwd_inner: 5644.83 | bwd_allreduce: 48.78 | step: 19.26

  9%|▉         | 3815/41250 [9:12:20<90:36:27,  8.71s/it]
                                                         
{'loss': 0.086, 'grad_norm': 0.8154548406600952, 'learning_rate': 3.9591994622020766e-05, 'epoch': 0.92}

  9%|▉         | 3815/41250 [9:12:20<90:36:27,  8.71s/it][2025-04-26 13:20:57,543] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 13:20:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.14 | bwd_microstep: 5804.21 | bwd_inner_microstep: 5653.48 | bwd_allreduce_microstep: 150.68 | step_microstep: 18.77
[2025-04-26 13:20:57,544] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.14 | bwd: 5804.22 | bwd_inner: 5653.48 | bwd_allreduce: 150.70 | step: 18.78

  9%|▉         | 3816/41250 [9:12:28<90:37:35,  8.72s/it]
                                                         
{'loss': 0.14, 'grad_norm': 1.2665152549743652, 'learning_rate': 3.959167899124017e-05, 'epoch': 0.93}

  9%|▉         | 3816/41250 [9:12:28<90:37:35,  8.72s/it][2025-04-26 13:21:06,187] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 13:21:06,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.81 | bwd_microstep: 5712.57 | bwd_inner_microstep: 5699.25 | bwd_allreduce_microstep: 13.27 | step_microstep: 18.84
[2025-04-26 13:21:06,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.81 | bwd: 5712.58 | bwd_inner: 5699.25 | bwd_allreduce: 13.29 | step: 18.84

  9%|▉         | 3817/41250 [9:12:37<90:23:58,  8.69s/it]
                                                         
{'loss': 0.1931, 'grad_norm': 1.8000714778900146, 'learning_rate': 3.9591363239680726e-05, 'epoch': 0.93}

  9%|▉         | 3817/41250 [9:12:37<90:23:58,  8.69s/it][2025-04-26 13:21:15,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:21:15,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2945.00 | bwd_microstep: 5899.31 | bwd_inner_microstep: 5886.33 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.72
[2025-04-26 13:21:15,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2945.01 | bwd: 5899.32 | bwd_inner: 5886.33 | bwd_allreduce: 12.95 | step: 18.73

  9%|▉         | 3818/41250 [9:12:46<91:05:57,  8.76s/it]
                                                         
{'loss': 0.0982, 'grad_norm': 0.9841403365135193, 'learning_rate': 3.959104736734438e-05, 'epoch': 0.93}

  9%|▉         | 3818/41250 [9:12:46<91:05:57,  8.76s/it][2025-04-26 13:21:23,732] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 13:21:23,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.96 | bwd_microstep: 5711.48 | bwd_inner_microstep: 5670.47 | bwd_allreduce_microstep: 40.96 | step_microstep: 18.66
[2025-04-26 13:21:23,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.96 | bwd: 5711.49 | bwd_inner: 5670.47 | bwd_allreduce: 40.98 | step: 18.67

  9%|▉         | 3819/41250 [9:12:54<90:40:32,  8.72s/it]
                                                         
{'loss': 0.2409, 'grad_norm': 2.6104745864868164, 'learning_rate': 3.959073137423308e-05, 'epoch': 0.93}

  9%|▉         | 3819/41250 [9:12:54<90:40:32,  8.72s/it][2025-04-26 13:21:32,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:21:32,422] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.69 | bwd_microstep: 5774.69 | bwd_inner_microstep: 5661.24 | bwd_allreduce_microstep: 113.40 | step_microstep: 18.65
[2025-04-26 13:21:32,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.69 | bwd: 5774.70 | bwd_inner: 5661.24 | bwd_allreduce: 113.42 | step: 18.65

  9%|▉         | 3820/41250 [9:13:03<90:34:27,  8.71s/it]
                                                         
{'loss': 0.1332, 'grad_norm': 1.145171880722046, 'learning_rate': 3.959041526034877e-05, 'epoch': 0.93}

  9%|▉         | 3820/41250 [9:13:03<90:34:27,  8.71s/it][2025-04-26 13:21:41,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.97 | optimizer_step: 0.94
[2025-04-26 13:21:41,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.26 | bwd_microstep: 5720.63 | bwd_inner_microstep: 5668.80 | bwd_allreduce_microstep: 51.79 | step_microstep: 18.52
[2025-04-26 13:21:41,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.26 | bwd: 5720.64 | bwd_inner: 5668.79 | bwd_allreduce: 51.81 | step: 18.53

  9%|▉         | 3821/41250 [9:13:12<90:20:05,  8.69s/it]
                                                         
{'loss': 0.0707, 'grad_norm': 0.822140634059906, 'learning_rate': 3.959009902569341e-05, 'epoch': 0.93}

  9%|▉         | 3821/41250 [9:13:12<90:20:05,  8.69s/it][2025-04-26 13:21:49,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:21:49,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.81 | bwd_microstep: 5762.73 | bwd_inner_microstep: 5696.99 | bwd_allreduce_microstep: 65.69 | step_microstep: 18.83
[2025-04-26 13:21:49,750] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.81 | bwd: 5762.74 | bwd_inner: 5696.99 | bwd_allreduce: 65.71 | step: 18.83

  9%|▉         | 3822/41250 [9:13:20<90:20:31,  8.69s/it]
                                                         
{'loss': 0.1538, 'grad_norm': 1.641648292541504, 'learning_rate': 3.958978267026894e-05, 'epoch': 0.93}

  9%|▉         | 3822/41250 [9:13:20<90:20:31,  8.69s/it][2025-04-26 13:21:58,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:21:58,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.49 | bwd_microstep: 5792.73 | bwd_inner_microstep: 5699.58 | bwd_allreduce_microstep: 93.10 | step_microstep: 18.56
[2025-04-26 13:21:58,466] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.49 | bwd: 5792.74 | bwd_inner: 5699.58 | bwd_allreduce: 93.12 | step: 18.57

  9%|▉         | 3823/41250 [9:13:29<90:25:23,  8.70s/it]
                                                         
{'loss': 0.2233, 'grad_norm': 1.719826340675354, 'learning_rate': 3.958946619407731e-05, 'epoch': 0.93}

  9%|▉         | 3823/41250 [9:13:29<90:25:23,  8.70s/it][2025-04-26 13:22:07,091] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:22:07,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.93 | bwd_microstep: 5703.28 | bwd_inner_microstep: 5670.44 | bwd_allreduce_microstep: 32.80 | step_microstep: 18.41
[2025-04-26 13:22:07,092] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.93 | bwd: 5703.29 | bwd_inner: 5670.44 | bwd_allreduce: 32.82 | step: 18.41

  9%|▉         | 3824/41250 [9:13:38<90:11:43,  8.68s/it]
                                                         
{'loss': 0.1708, 'grad_norm': 1.622233510017395, 'learning_rate': 3.958914959712048e-05, 'epoch': 0.93}

  9%|▉         | 3824/41250 [9:13:38<90:11:43,  8.68s/it][2025-04-26 13:22:15,792] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.26 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 13:22:15,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.06 | bwd_microstep: 5766.89 | bwd_inner_microstep: 5685.63 | bwd_allreduce_microstep: 81.22 | step_microstep: 18.21
[2025-04-26 13:22:15,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.06 | bwd: 5766.91 | bwd_inner: 5685.63 | bwd_allreduce: 81.24 | step: 18.21

  9%|▉         | 3825/41250 [9:13:46<90:16:14,  8.68s/it]
                                                         
{'loss': 0.1265, 'grad_norm': 1.7918044328689575, 'learning_rate': 3.958883287940039e-05, 'epoch': 0.93}

  9%|▉         | 3825/41250 [9:13:46<90:16:14,  8.68s/it][2025-04-26 13:22:24,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:22:24,482] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.90 | bwd_microstep: 5767.49 | bwd_inner_microstep: 5654.87 | bwd_allreduce_microstep: 112.57 | step_microstep: 18.62
[2025-04-26 13:22:24,483] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.90 | bwd: 5767.50 | bwd_inner: 5654.87 | bwd_allreduce: 112.58 | step: 18.63

  9%|▉         | 3826/41250 [9:13:55<90:17:33,  8.69s/it]
                                                         
{'loss': 0.3604, 'grad_norm': 2.4139199256896973, 'learning_rate': 3.9588516040919e-05, 'epoch': 0.93}

  9%|▉         | 3826/41250 [9:13:55<90:17:33,  8.69s/it][2025-04-26 13:22:33,100] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:22:33,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.97 | bwd_microstep: 5697.74 | bwd_inner_microstep: 5685.12 | bwd_allreduce_microstep: 12.57 | step_microstep: 18.06
[2025-04-26 13:22:33,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.97 | bwd: 5697.76 | bwd_inner: 5685.12 | bwd_allreduce: 12.59 | step: 18.06

  9%|▉         | 3827/41250 [9:14:04<90:04:41,  8.67s/it]
                                                         
{'loss': 0.1777, 'grad_norm': 2.1113996505737305, 'learning_rate': 3.958819908167826e-05, 'epoch': 0.93}

  9%|▉         | 3827/41250 [9:14:04<90:04:41,  8.67s/it][2025-04-26 13:22:41,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:22:41,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.08 | bwd_microstep: 5730.74 | bwd_inner_microstep: 5717.80 | bwd_allreduce_microstep: 12.89 | step_microstep: 18.45
[2025-04-26 13:22:41,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.08 | bwd: 5730.75 | bwd_inner: 5717.80 | bwd_allreduce: 12.91 | step: 18.46

  9%|▉         | 3828/41250 [9:14:12<90:06:31,  8.67s/it]
                                                         
{'loss': 0.1606, 'grad_norm': 1.6029356718063354, 'learning_rate': 3.958788200168013e-05, 'epoch': 0.93}

  9%|▉         | 3828/41250 [9:14:12<90:06:31,  8.67s/it][2025-04-26 13:22:50,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.10 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:22:50,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.42 | bwd_microstep: 5796.99 | bwd_inner_microstep: 5666.33 | bwd_allreduce_microstep: 130.62 | step_microstep: 18.11
[2025-04-26 13:22:50,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.42 | bwd: 5797.00 | bwd_inner: 5666.33 | bwd_allreduce: 130.63 | step: 18.12

  9%|▉         | 3829/41250 [9:14:21<90:14:20,  8.68s/it]
                                                         
{'loss': 0.2466, 'grad_norm': 1.6446526050567627, 'learning_rate': 3.958756480092656e-05, 'epoch': 0.93}

  9%|▉         | 3829/41250 [9:14:21<90:14:20,  8.68s/it][2025-04-26 13:22:59,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.03
[2025-04-26 13:22:59,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.76 | bwd_microstep: 5747.23 | bwd_inner_microstep: 5711.33 | bwd_allreduce_microstep: 35.86 | step_microstep: 18.39
[2025-04-26 13:22:59,181] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.76 | bwd: 5747.25 | bwd_inner: 5711.33 | bwd_allreduce: 35.88 | step: 18.39

  9%|▉         | 3830/41250 [9:14:30<90:16:33,  8.69s/it]
                                                         
{'loss': 0.1219, 'grad_norm': 1.5991140604019165, 'learning_rate': 3.9587247479419504e-05, 'epoch': 0.93}

  9%|▉         | 3830/41250 [9:14:30<90:16:33,  8.69s/it][2025-04-26 13:23:07,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.96 | optimizer_step: 0.94
[2025-04-26 13:23:07,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.53 | bwd_microstep: 5754.34 | bwd_inner_microstep: 5712.39 | bwd_allreduce_microstep: 41.91 | step_microstep: 18.24
[2025-04-26 13:23:07,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.53 | bwd: 5754.36 | bwd_inner: 5712.39 | bwd_allreduce: 41.92 | step: 18.25

  9%|▉         | 3831/41250 [9:14:39<90:19:24,  8.69s/it]
                                                         
{'loss': 0.3126, 'grad_norm': 2.679800033569336, 'learning_rate': 3.9586930037160915e-05, 'epoch': 0.93}

  9%|▉         | 3831/41250 [9:14:39<90:19:24,  8.69s/it][2025-04-26 13:23:16,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:23:16,586] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.17 | bwd_microstep: 5779.39 | bwd_inner_microstep: 5692.68 | bwd_allreduce_microstep: 86.66 | step_microstep: 18.90
[2025-04-26 13:23:16,587] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.17 | bwd: 5779.40 | bwd_inner: 5692.68 | bwd_allreduce: 86.68 | step: 18.90

  9%|▉         | 3832/41250 [9:14:47<90:22:27,  8.69s/it]
                                                         
{'loss': 0.1015, 'grad_norm': 1.339727759361267, 'learning_rate': 3.958661247415276e-05, 'epoch': 0.93}

  9%|▉         | 3832/41250 [9:14:47<90:22:27,  8.69s/it][2025-04-26 13:23:25,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:23:25,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.71 | bwd_microstep: 5761.64 | bwd_inner_microstep: 5685.62 | bwd_allreduce_microstep: 75.97 | step_microstep: 17.69
[2025-04-26 13:23:25,286] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.71 | bwd: 5761.65 | bwd_inner: 5685.62 | bwd_allreduce: 75.99 | step: 17.70

  9%|▉         | 3833/41250 [9:14:56<90:22:14,  8.69s/it]
                                                         
{'loss': 0.0954, 'grad_norm': 1.0844635963439941, 'learning_rate': 3.958629479039699e-05, 'epoch': 0.93}

  9%|▉         | 3833/41250 [9:14:56<90:22:14,  8.69s/it][2025-04-26 13:23:33,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 1.00
[2025-04-26 13:23:33,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.53 | bwd_microstep: 5765.77 | bwd_inner_microstep: 5700.98 | bwd_allreduce_microstep: 64.75 | step_microstep: 18.83
[2025-04-26 13:23:33,987] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.53 | bwd: 5765.79 | bwd_inner: 5700.98 | bwd_allreduce: 64.76 | step: 18.83

  9%|▉         | 3834/41250 [9:15:05<90:23:31,  8.70s/it]
                                                         
{'loss': 0.3152, 'grad_norm': 1.684949278831482, 'learning_rate': 3.958597698589557e-05, 'epoch': 0.93}

  9%|▉         | 3834/41250 [9:15:05<90:23:31,  8.70s/it][2025-04-26 13:23:42,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:23:42,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.87 | bwd_microstep: 5787.82 | bwd_inner_microstep: 5653.15 | bwd_allreduce_microstep: 134.62 | step_microstep: 18.03
[2025-04-26 13:23:42,681] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.87 | bwd: 5787.83 | bwd_inner: 5653.15 | bwd_allreduce: 134.64 | step: 18.03

  9%|▉         | 3835/41250 [9:15:13<90:22:43,  8.70s/it]
                                                         
{'loss': 0.1278, 'grad_norm': 1.8608980178833008, 'learning_rate': 3.958565906065045e-05, 'epoch': 0.93}

  9%|▉         | 3835/41250 [9:15:13<90:22:43,  8.70s/it][2025-04-26 13:23:51,337] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:23:51,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.26 | bwd_microstep: 5756.54 | bwd_inner_microstep: 5662.53 | bwd_allreduce_microstep: 93.97 | step_microstep: 18.20
[2025-04-26 13:23:51,338] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.26 | bwd: 5756.55 | bwd_inner: 5662.53 | bwd_allreduce: 93.98 | step: 18.21

  9%|▉         | 3836/41250 [9:15:22<90:15:07,  8.68s/it]
                                                         
{'loss': 0.2094, 'grad_norm': 1.8219414949417114, 'learning_rate': 3.958534101466359e-05, 'epoch': 0.93}

  9%|▉         | 3836/41250 [9:15:22<90:15:07,  8.68s/it][2025-04-26 13:24:00,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:24:00,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.35 | bwd_microstep: 5779.16 | bwd_inner_microstep: 5642.19 | bwd_allreduce_microstep: 136.93 | step_microstep: 18.07
[2025-04-26 13:24:00,019] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.35 | bwd: 5779.17 | bwd_inner: 5642.19 | bwd_allreduce: 136.94 | step: 18.07

  9%|▉         | 3837/41250 [9:15:31<90:14:29,  8.68s/it]
                                                         
{'loss': 0.2575, 'grad_norm': 1.9315129518508911, 'learning_rate': 3.9585022847936957e-05, 'epoch': 0.93}

  9%|▉         | 3837/41250 [9:15:31<90:14:29,  8.68s/it][2025-04-26 13:24:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:24:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.22 | bwd_microstep: 5771.44 | bwd_inner_microstep: 5648.75 | bwd_allreduce_microstep: 122.65 | step_microstep: 17.69
[2025-04-26 13:24:08,700] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.22 | bwd: 5771.46 | bwd_inner: 5648.75 | bwd_allreduce: 122.67 | step: 17.69

  9%|▉         | 3838/41250 [9:15:39<90:13:56,  8.68s/it]
                                                         
{'loss': 0.2081, 'grad_norm': 1.6561096906661987, 'learning_rate': 3.958470456047251e-05, 'epoch': 0.93}

  9%|▉         | 3838/41250 [9:15:39<90:13:56,  8.68s/it][2025-04-26 13:24:17,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:24:17,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.43 | bwd_microstep: 5754.13 | bwd_inner_microstep: 5694.70 | bwd_allreduce_microstep: 59.38 | step_microstep: 17.82
[2025-04-26 13:24:17,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.43 | bwd: 5754.14 | bwd_inner: 5694.70 | bwd_allreduce: 59.40 | step: 17.83

  9%|▉         | 3839/41250 [9:15:48<90:14:27,  8.68s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.563578724861145, 'learning_rate': 3.958438615227222e-05, 'epoch': 0.93}

  9%|▉         | 3839/41250 [9:15:48<90:14:27,  8.68s/it][2025-04-26 13:24:26,063] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.95
[2025-04-26 13:24:26,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.83 | bwd_microstep: 5749.66 | bwd_inner_microstep: 5691.22 | bwd_allreduce_microstep: 58.40 | step_microstep: 18.21
[2025-04-26 13:24:26,064] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.83 | bwd: 5749.67 | bwd_inner: 5691.22 | bwd_allreduce: 58.42 | step: 18.21

  9%|▉         | 3840/41250 [9:15:57<90:13:05,  8.68s/it]
                                                         
{'loss': 0.1199, 'grad_norm': 1.9315063953399658, 'learning_rate': 3.958406762333803e-05, 'epoch': 0.93}

  9%|▉         | 3840/41250 [9:15:57<90:13:05,  8.68s/it][2025-04-26 13:24:34,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:24:34,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.66 | bwd_microstep: 5762.81 | bwd_inner_microstep: 5700.30 | bwd_allreduce_microstep: 62.47 | step_microstep: 18.34
[2025-04-26 13:24:34,764] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.66 | bwd: 5762.82 | bwd_inner: 5700.30 | bwd_allreduce: 62.48 | step: 18.34

  9%|▉         | 3841/41250 [9:16:05<90:16:31,  8.69s/it]
                                                         
{'loss': 0.2369, 'grad_norm': 1.239337682723999, 'learning_rate': 3.9583748973671925e-05, 'epoch': 0.93}

  9%|▉         | 3841/41250 [9:16:05<90:16:31,  8.69s/it][2025-04-26 13:24:43,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:24:43,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.20 | bwd_microstep: 5735.80 | bwd_inner_microstep: 5684.86 | bwd_allreduce_microstep: 50.89 | step_microstep: 18.22
[2025-04-26 13:24:43,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.20 | bwd: 5735.81 | bwd_inner: 5684.86 | bwd_allreduce: 50.91 | step: 18.22

  9%|▉         | 3842/41250 [9:16:14<90:11:31,  8.68s/it]
                                                         
{'loss': 0.0321, 'grad_norm': 0.3669835925102234, 'learning_rate': 3.958343020327585e-05, 'epoch': 0.93}

  9%|▉         | 3842/41250 [9:16:14<90:11:31,  8.68s/it][2025-04-26 13:24:52,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:24:52,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5718.89 | bwd_inner_microstep: 5706.35 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.19
[2025-04-26 13:24:52,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5718.90 | bwd_inner: 5706.35 | bwd_allreduce: 12.52 | step: 18.19

  9%|▉         | 3843/41250 [9:16:23<90:06:26,  8.67s/it]
                                                         
{'loss': 0.0314, 'grad_norm': 0.3875788748264313, 'learning_rate': 3.958311131215179e-05, 'epoch': 0.93}

  9%|▉         | 3843/41250 [9:16:23<90:06:26,  8.67s/it][2025-04-26 13:25:00,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:25:00,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.36 | bwd_microstep: 5732.31 | bwd_inner_microstep: 5719.70 | bwd_allreduce_microstep: 12.56 | step_microstep: 18.13
[2025-04-26 13:25:00,740] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.36 | bwd: 5732.32 | bwd_inner: 5719.70 | bwd_allreduce: 12.58 | step: 18.13

  9%|▉         | 3844/41250 [9:16:31<90:04:10,  8.67s/it]
                                                         
{'loss': 0.2105, 'grad_norm': 4.560171604156494, 'learning_rate': 3.958279230030169e-05, 'epoch': 0.93}

  9%|▉         | 3844/41250 [9:16:31<90:04:10,  8.67s/it][2025-04-26 13:25:09,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:25:09,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2817.46 | bwd_microstep: 5770.60 | bwd_inner_microstep: 5637.44 | bwd_allreduce_microstep: 133.12 | step_microstep: 18.14
[2025-04-26 13:25:09,396] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2817.46 | bwd: 5770.61 | bwd_inner: 5637.44 | bwd_allreduce: 133.13 | step: 18.14

  9%|▉         | 3845/41250 [9:16:40<90:01:44,  8.66s/it]
                                                         
{'loss': 0.2581, 'grad_norm': 1.2964566946029663, 'learning_rate': 3.9582473167727535e-05, 'epoch': 0.93}

  9%|▉         | 3845/41250 [9:16:40<90:01:44,  8.66s/it][2025-04-26 13:25:18,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:25:18,037] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.95 | bwd_microstep: 5710.78 | bwd_inner_microstep: 5698.10 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.26
[2025-04-26 13:25:18,038] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.95 | bwd: 5710.79 | bwd_inner: 5698.10 | bwd_allreduce: 12.66 | step: 18.27

  9%|▉         | 3846/41250 [9:16:49<89:57:13,  8.66s/it]
                                                         
{'loss': 0.0568, 'grad_norm': 0.6106880903244019, 'learning_rate': 3.958215391443128e-05, 'epoch': 0.93}

  9%|▉         | 3846/41250 [9:16:49<89:57:13,  8.66s/it][2025-04-26 13:25:26,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.99 | optimizer_step: 1.01
[2025-04-26 13:25:26,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.26 | bwd_microstep: 5691.83 | bwd_inner_microstep: 5652.97 | bwd_allreduce_microstep: 38.81 | step_microstep: 18.57
[2025-04-26 13:25:26,638] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.26 | bwd: 5691.84 | bwd_inner: 5652.97 | bwd_allreduce: 38.83 | step: 18.57

  9%|▉         | 3847/41250 [9:16:57<89:46:45,  8.64s/it]
                                                         
{'loss': 0.1424, 'grad_norm': 1.882243037223816, 'learning_rate': 3.95818345404149e-05, 'epoch': 0.93}

  9%|▉         | 3847/41250 [9:16:57<89:46:45,  8.64s/it][2025-04-26 13:25:35,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:25:35,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.85 | bwd_microstep: 5723.75 | bwd_inner_microstep: 5710.87 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.88
[2025-04-26 13:25:35,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.85 | bwd: 5723.76 | bwd_inner: 5710.87 | bwd_allreduce: 12.85 | step: 18.88

  9%|▉         | 3848/41250 [9:17:06<89:52:00,  8.65s/it]
                                                         
{'loss': 0.1934, 'grad_norm': 1.586477279663086, 'learning_rate': 3.958151504568036e-05, 'epoch': 0.93}

  9%|▉         | 3848/41250 [9:17:06<89:52:00,  8.65s/it][2025-04-26 13:25:43,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 13:25:43,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.06 | bwd_microstep: 5745.69 | bwd_inner_microstep: 5695.09 | bwd_allreduce_microstep: 50.56 | step_microstep: 18.73
[2025-04-26 13:25:43,989] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.06 | bwd: 5745.70 | bwd_inner: 5695.09 | bwd_allreduce: 50.58 | step: 18.74

  9%|▉         | 3849/41250 [9:17:15<89:57:22,  8.66s/it]
                                                         
{'loss': 0.133, 'grad_norm': 1.1759577989578247, 'learning_rate': 3.9581195430229636e-05, 'epoch': 0.93}

  9%|▉         | 3849/41250 [9:17:15<89:57:22,  8.66s/it][2025-04-26 13:25:52,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:25:52,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.08 | bwd_microstep: 5719.45 | bwd_inner_microstep: 5706.64 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.67
[2025-04-26 13:25:52,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.08 | bwd: 5719.46 | bwd_inner: 5706.64 | bwd_allreduce: 12.78 | step: 18.67

  9%|▉         | 3850/41250 [9:17:23<89:57:00,  8.66s/it]
                                                         
{'loss': 0.3488, 'grad_norm': 3.2459797859191895, 'learning_rate': 3.958087569406469e-05, 'epoch': 0.93}

  9%|▉         | 3850/41250 [9:17:23<89:57:00,  8.66s/it][2025-04-26 13:26:01,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:26:01,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.69 | bwd_microstep: 5716.48 | bwd_inner_microstep: 5645.69 | bwd_allreduce_microstep: 70.74 | step_microstep: 18.90
[2025-04-26 13:26:01,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.69 | bwd: 5716.49 | bwd_inner: 5645.69 | bwd_allreduce: 70.76 | step: 18.91

  9%|▉         | 3851/41250 [9:17:32<89:51:45,  8.65s/it]
                                                         
{'loss': 0.2043, 'grad_norm': 2.7484042644500732, 'learning_rate': 3.958055583718749e-05, 'epoch': 0.93}

  9%|▉         | 3851/41250 [9:17:32<89:51:45,  8.65s/it][2025-04-26 13:26:09,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:26:09,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.27 | bwd_microstep: 5709.86 | bwd_inner_microstep: 5658.64 | bwd_allreduce_microstep: 51.17 | step_microstep: 18.82
[2025-04-26 13:26:09,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.27 | bwd: 5709.88 | bwd_inner: 5658.64 | bwd_allreduce: 51.19 | step: 18.82

  9%|▉         | 3852/41250 [9:17:41<89:46:48,  8.64s/it]
                                                         
{'loss': 0.2566, 'grad_norm': 2.2843985557556152, 'learning_rate': 3.958023585960002e-05, 'epoch': 0.93}

  9%|▉         | 3852/41250 [9:17:41<89:46:48,  8.64s/it][2025-04-26 13:26:18,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:26:18,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.05 | bwd_microstep: 5781.59 | bwd_inner_microstep: 5660.62 | bwd_allreduce_microstep: 120.92 | step_microstep: 18.55
[2025-04-26 13:26:18,593] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.05 | bwd: 5781.60 | bwd_inner: 5660.62 | bwd_allreduce: 120.94 | step: 18.56

  9%|▉         | 3853/41250 [9:17:49<89:55:54,  8.66s/it]
                                                         
{'loss': 0.3267, 'grad_norm': 2.1201114654541016, 'learning_rate': 3.957991576130425e-05, 'epoch': 0.93}

  9%|▉         | 3853/41250 [9:17:49<89:55:54,  8.66s/it][2025-04-26 13:26:27,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 13:26:27,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2877.60 | bwd_microstep: 5776.33 | bwd_inner_microstep: 5763.56 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.02
[2025-04-26 13:26:27,319] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2877.60 | bwd: 5776.35 | bwd_inner: 5763.56 | bwd_allreduce: 12.75 | step: 19.03

  9%|▉         | 3854/41250 [9:17:58<90:08:41,  8.68s/it]
                                                         
{'loss': 0.2789, 'grad_norm': 1.613060712814331, 'learning_rate': 3.957959554230215e-05, 'epoch': 0.93}

  9%|▉         | 3854/41250 [9:17:58<90:08:41,  8.68s/it][2025-04-26 13:26:35,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:26:35,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.84 | bwd_microstep: 5720.11 | bwd_inner_microstep: 5707.30 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.70
[2025-04-26 13:26:35,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.84 | bwd: 5720.12 | bwd_inner: 5707.30 | bwd_allreduce: 12.78 | step: 18.70

  9%|▉         | 3855/41250 [9:18:07<90:05:11,  8.67s/it]
                                                         
{'loss': 0.1712, 'grad_norm': 3.0326015949249268, 'learning_rate': 3.957927520259569e-05, 'epoch': 0.93}

  9%|▉         | 3855/41250 [9:18:07<90:05:11,  8.67s/it][2025-04-26 13:26:44,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:26:44,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.30 | bwd_microstep: 5760.88 | bwd_inner_microstep: 5688.53 | bwd_allreduce_microstep: 72.31 | step_microstep: 18.63
[2025-04-26 13:26:44,666] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.30 | bwd: 5760.90 | bwd_inner: 5688.53 | bwd_allreduce: 72.32 | step: 18.63

  9%|▉         | 3856/41250 [9:18:15<90:07:31,  8.68s/it]
                                                         
{'loss': 0.1664, 'grad_norm': 1.1631827354431152, 'learning_rate': 3.957895474218686e-05, 'epoch': 0.93}

  9%|▉         | 3856/41250 [9:18:15<90:07:31,  8.68s/it][2025-04-26 13:26:53,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:26:53,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.05 | bwd_microstep: 5782.47 | bwd_inner_microstep: 5667.31 | bwd_allreduce_microstep: 115.12 | step_microstep: 18.98
[2025-04-26 13:26:53,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.05 | bwd: 5782.49 | bwd_inner: 5667.31 | bwd_allreduce: 115.14 | step: 18.98

  9%|▉         | 3857/41250 [9:18:24<90:13:10,  8.69s/it]
                                                         
{'loss': 0.254, 'grad_norm': 2.016425371170044, 'learning_rate': 3.957863416107761e-05, 'epoch': 0.94}

  9%|▉         | 3857/41250 [9:18:24<90:13:10,  8.69s/it][2025-04-26 13:27:02,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:27:02,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.27 | bwd_microstep: 5785.27 | bwd_inner_microstep: 5663.64 | bwd_allreduce_microstep: 121.59 | step_microstep: 18.64
[2025-04-26 13:27:02,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.27 | bwd: 5785.28 | bwd_inner: 5663.64 | bwd_allreduce: 121.61 | step: 18.65

  9%|▉         | 3858/41250 [9:18:33<90:16:05,  8.69s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 2.4527392387390137, 'learning_rate': 3.957831345926994e-05, 'epoch': 0.94}

  9%|▉         | 3858/41250 [9:18:33<90:16:05,  8.69s/it][2025-04-26 13:27:10,728] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.03 | optimizer_step: 1.03
[2025-04-26 13:27:10,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.71 | bwd_microstep: 5725.49 | bwd_inner_microstep: 5712.29 | bwd_allreduce_microstep: 13.15 | step_microstep: 19.32
[2025-04-26 13:27:10,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.71 | bwd: 5725.50 | bwd_inner: 5712.29 | bwd_allreduce: 13.17 | step: 19.33

  9%|▉         | 3859/41250 [9:18:41<90:09:08,  8.68s/it]
                                                         
{'loss': 0.1998, 'grad_norm': 1.1513899564743042, 'learning_rate': 3.957799263676582e-05, 'epoch': 0.94}

  9%|▉         | 3859/41250 [9:18:41<90:09:08,  8.68s/it][2025-04-26 13:27:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:27:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.65 | bwd_microstep: 5754.35 | bwd_inner_microstep: 5694.88 | bwd_allreduce_microstep: 59.43 | step_microstep: 18.68
[2025-04-26 13:27:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.65 | bwd: 5754.36 | bwd_inner: 5694.88 | bwd_allreduce: 59.44 | step: 18.69

  9%|▉         | 3860/41250 [9:18:50<90:12:02,  8.68s/it]
                                                         
{'loss': 0.1037, 'grad_norm': 0.8479385375976562, 'learning_rate': 3.9577671693567226e-05, 'epoch': 0.94}

  9%|▉         | 3860/41250 [9:18:50<90:12:02,  8.68s/it][2025-04-26 13:27:28,201] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:27:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.34 | bwd_microstep: 5801.81 | bwd_inner_microstep: 5788.85 | bwd_allreduce_microstep: 12.91 | step_microstep: 18.64
[2025-04-26 13:27:28,202] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.34 | bwd: 5801.83 | bwd_inner: 5788.85 | bwd_allreduce: 12.93 | step: 18.64

  9%|▉         | 3861/41250 [9:18:59<90:28:51,  8.71s/it]
                                                         
{'loss': 0.1795, 'grad_norm': 1.4649169445037842, 'learning_rate': 3.957735062967613e-05, 'epoch': 0.94}

  9%|▉         | 3861/41250 [9:18:59<90:28:51,  8.71s/it][2025-04-26 13:27:36,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:27:36,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.81 | bwd_microstep: 5850.17 | bwd_inner_microstep: 5709.58 | bwd_allreduce_microstep: 140.54 | step_microstep: 18.48
[2025-04-26 13:27:36,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.81 | bwd: 5850.18 | bwd_inner: 5709.58 | bwd_allreduce: 140.56 | step: 18.48

  9%|▉         | 3862/41250 [9:19:08<90:43:34,  8.74s/it]
                                                         
{'loss': 0.1624, 'grad_norm': 1.4828386306762695, 'learning_rate': 3.9577029445094526e-05, 'epoch': 0.94}

  9%|▉         | 3862/41250 [9:19:08<90:43:34,  8.74s/it][2025-04-26 13:27:45,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.06 | optimizer_step: 0.90
[2025-04-26 13:27:45,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.45 | bwd_microstep: 5739.90 | bwd_inner_microstep: 5651.72 | bwd_allreduce_microstep: 88.12 | step_microstep: 18.93
[2025-04-26 13:27:45,629] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.45 | bwd: 5739.92 | bwd_inner: 5651.72 | bwd_allreduce: 88.15 | step: 18.93

  9%|▉         | 3863/41250 [9:19:16<90:25:25,  8.71s/it]
                                                         
{'loss': 0.159, 'grad_norm': 1.924075961112976, 'learning_rate': 3.957670813982438e-05, 'epoch': 0.94}

  9%|▉         | 3863/41250 [9:19:16<90:25:25,  8.71s/it][2025-04-26 13:27:54,333] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:27:54,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.70 | bwd_microstep: 5769.14 | bwd_inner_microstep: 5694.88 | bwd_allreduce_microstep: 74.22 | step_microstep: 18.55
[2025-04-26 13:27:54,334] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.70 | bwd: 5769.15 | bwd_inner: 5694.88 | bwd_allreduce: 74.24 | step: 18.55

  9%|▉         | 3864/41250 [9:19:25<90:24:13,  8.71s/it]
                                                         
{'loss': 0.1279, 'grad_norm': 1.3502980470657349, 'learning_rate': 3.9576386713867683e-05, 'epoch': 0.94}

  9%|▉         | 3864/41250 [9:19:25<90:24:13,  8.71s/it][2025-04-26 13:28:02,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.04 | optimizer_step: 1.12
[2025-04-26 13:28:02,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.23 | bwd_microstep: 5722.85 | bwd_inner_microstep: 5664.78 | bwd_allreduce_microstep: 58.03 | step_microstep: 19.04
[2025-04-26 13:28:02,974] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.23 | bwd: 5722.87 | bwd_inner: 5664.78 | bwd_allreduce: 58.05 | step: 19.04

  9%|▉         | 3865/41250 [9:19:34<90:12:07,  8.69s/it]
                                                         
{'loss': 0.1834, 'grad_norm': 1.7385274171829224, 'learning_rate': 3.957606516722641e-05, 'epoch': 0.94}

  9%|▉         | 3865/41250 [9:19:34<90:12:07,  8.69s/it][2025-04-26 13:28:11,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:28:11,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.74 | bwd_microstep: 5918.28 | bwd_inner_microstep: 5722.18 | bwd_allreduce_microstep: 196.05 | step_microstep: 18.69
[2025-04-26 13:28:11,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.74 | bwd: 5918.29 | bwd_inner: 5722.18 | bwd_allreduce: 196.07 | step: 18.69

  9%|▉         | 3866/41250 [9:19:43<90:44:22,  8.74s/it]
                                                         
{'loss': 0.1882, 'grad_norm': 0.9486144185066223, 'learning_rate': 3.957574349990254e-05, 'epoch': 0.94}

  9%|▉         | 3866/41250 [9:19:43<90:44:22,  8.74s/it][2025-04-26 13:28:20,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.64 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:28:20,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2907.16 | bwd_microstep: 5798.24 | bwd_inner_microstep: 5785.37 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.88
[2025-04-26 13:28:20,621] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2907.16 | bwd: 5798.25 | bwd_inner: 5785.37 | bwd_allreduce: 12.84 | step: 18.88

  9%|▉         | 3867/41250 [9:19:51<90:53:29,  8.75s/it]
                                                         
{'loss': 0.162, 'grad_norm': 0.8097183108329773, 'learning_rate': 3.9575421711898075e-05, 'epoch': 0.94}

  9%|▉         | 3867/41250 [9:19:51<90:53:29,  8.75s/it][2025-04-26 13:28:29,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 13:28:29,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.44 | bwd_microstep: 5717.55 | bwd_inner_microstep: 5645.25 | bwd_allreduce_microstep: 72.26 | step_microstep: 18.81
[2025-04-26 13:28:29,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.44 | bwd: 5717.57 | bwd_inner: 5645.25 | bwd_allreduce: 72.27 | step: 18.82

  9%|▉         | 3868/41250 [9:20:00<90:28:41,  8.71s/it]
                                                         
{'loss': 0.1989, 'grad_norm': 2.695732831954956, 'learning_rate': 3.9575099803214975e-05, 'epoch': 0.94}

  9%|▉         | 3868/41250 [9:20:00<90:28:41,  8.71s/it][2025-04-26 13:28:37,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 13:28:37,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.82 | bwd_microstep: 5783.27 | bwd_inner_microstep: 5704.27 | bwd_allreduce_microstep: 78.96 | step_microstep: 18.53
[2025-04-26 13:28:37,970] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.82 | bwd: 5783.29 | bwd_inner: 5704.27 | bwd_allreduce: 78.98 | step: 18.53

  9%|▉         | 3869/41250 [9:20:09<90:31:11,  8.72s/it]
                                                         
{'loss': 0.2098, 'grad_norm': 1.484981894493103, 'learning_rate': 3.957477777385524e-05, 'epoch': 0.94}

  9%|▉         | 3869/41250 [9:20:09<90:31:11,  8.72s/it][2025-04-26 13:28:46,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:28:46,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.75 | bwd_microstep: 5722.78 | bwd_inner_microstep: 5709.89 | bwd_allreduce_microstep: 12.84 | step_microstep: 18.80
[2025-04-26 13:28:46,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.75 | bwd: 5722.79 | bwd_inner: 5709.89 | bwd_allreduce: 12.86 | step: 18.81

  9%|▉         | 3870/41250 [9:20:17<90:19:55,  8.70s/it]
                                                         
{'loss': 0.1766, 'grad_norm': 1.7741172313690186, 'learning_rate': 3.957445562382085e-05, 'epoch': 0.94}

  9%|▉         | 3870/41250 [9:20:17<90:19:55,  8.70s/it][2025-04-26 13:28:55,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:28:55,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.61 | bwd_microstep: 5758.36 | bwd_inner_microstep: 5656.72 | bwd_allreduce_microstep: 101.59 | step_microstep: 18.79
[2025-04-26 13:28:55,299] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.61 | bwd: 5758.37 | bwd_inner: 5656.72 | bwd_allreduce: 101.61 | step: 18.79

  9%|▉         | 3871/41250 [9:20:26<90:14:41,  8.69s/it]
                                                         
{'loss': 0.2266, 'grad_norm': 2.095305919647217, 'learning_rate': 3.957413335311379e-05, 'epoch': 0.94}

  9%|▉         | 3871/41250 [9:20:26<90:14:41,  8.69s/it][2025-04-26 13:29:03,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 13:29:03,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.30 | bwd_microstep: 5771.04 | bwd_inner_microstep: 5700.63 | bwd_allreduce_microstep: 70.36 | step_microstep: 18.78
[2025-04-26 13:29:03,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.31 | bwd: 5771.05 | bwd_inner: 5700.63 | bwd_allreduce: 70.38 | step: 18.78

  9%|▉         | 3872/41250 [9:20:35<90:14:39,  8.69s/it]
                                                         
{'loss': 0.186, 'grad_norm': 1.026180624961853, 'learning_rate': 3.9573810961736046e-05, 'epoch': 0.94}

  9%|▉         | 3872/41250 [9:20:35<90:14:39,  8.69s/it][2025-04-26 13:29:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:29:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.19 | bwd_microstep: 5760.98 | bwd_inner_microstep: 5652.12 | bwd_allreduce_microstep: 108.81 | step_microstep: 18.71
[2025-04-26 13:29:12,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.19 | bwd: 5760.99 | bwd_inner: 5652.12 | bwd_allreduce: 108.83 | step: 18.71

  9%|▉         | 3873/41250 [9:20:43<90:10:04,  8.68s/it]
                                                         
{'loss': 0.0959, 'grad_norm': 0.8003941774368286, 'learning_rate': 3.9573488449689606e-05, 'epoch': 0.94}

  9%|▉         | 3873/41250 [9:20:43<90:10:04,  8.68s/it][2025-04-26 13:29:21,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 13:29:21,341] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.36 | bwd_microstep: 5763.40 | bwd_inner_microstep: 5655.14 | bwd_allreduce_microstep: 108.22 | step_microstep: 19.23
[2025-04-26 13:29:21,342] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.36 | bwd: 5763.42 | bwd_inner: 5655.14 | bwd_allreduce: 108.24 | step: 19.23

  9%|▉         | 3874/41250 [9:20:52<90:09:20,  8.68s/it]
                                                         
{'loss': 0.0871, 'grad_norm': 0.8476047515869141, 'learning_rate': 3.9573165816976464e-05, 'epoch': 0.94}

  9%|▉         | 3874/41250 [9:20:52<90:09:20,  8.68s/it][2025-04-26 13:29:29,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:29:29,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.03 | bwd_microstep: 5706.97 | bwd_inner_microstep: 5694.20 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.54
[2025-04-26 13:29:29,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.03 | bwd: 5706.99 | bwd_inner: 5694.20 | bwd_allreduce: 12.75 | step: 18.54

  9%|▉         | 3875/41250 [9:21:01<90:03:03,  8.67s/it]
                                                         
{'loss': 0.1149, 'grad_norm': 2.048750162124634, 'learning_rate': 3.95728430635986e-05, 'epoch': 0.94}

  9%|▉         | 3875/41250 [9:21:01<90:03:03,  8.67s/it][2025-04-26 13:29:38,877] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.92
[2025-04-26 13:29:38,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2930.82 | bwd_microstep: 5871.38 | bwd_inner_microstep: 5858.83 | bwd_allreduce_microstep: 12.50 | step_microstep: 18.80
[2025-04-26 13:29:38,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2930.82 | bwd: 5871.39 | bwd_inner: 5858.83 | bwd_allreduce: 12.52 | step: 18.80

  9%|▉         | 3876/41250 [9:21:10<90:42:22,  8.74s/it]
                                                         
{'loss': 0.0784, 'grad_norm': 0.8257723450660706, 'learning_rate': 3.9572520189558014e-05, 'epoch': 0.94}

  9%|▉         | 3876/41250 [9:21:10<90:42:22,  8.74s/it][2025-04-26 13:29:47,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.03
[2025-04-26 13:29:47,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.35 | bwd_microstep: 5755.90 | bwd_inner_microstep: 5691.69 | bwd_allreduce_microstep: 64.17 | step_microstep: 18.55
[2025-04-26 13:29:47,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5755.92 | bwd_inner: 5691.69 | bwd_allreduce: 64.18 | step: 18.56

  9%|▉         | 3877/41250 [9:21:18<90:32:01,  8.72s/it]
                                                         
{'loss': 0.179, 'grad_norm': 1.677220106124878, 'learning_rate': 3.957219719485669e-05, 'epoch': 0.94}

  9%|▉         | 3877/41250 [9:21:18<90:32:01,  8.72s/it][2025-04-26 13:29:56,233] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.99 | optimizer_step: 1.05
[2025-04-26 13:29:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.35 | bwd_microstep: 5749.21 | bwd_inner_microstep: 5688.78 | bwd_allreduce_microstep: 60.39 | step_microstep: 18.45
[2025-04-26 13:29:56,234] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.35 | bwd: 5749.23 | bwd_inner: 5688.78 | bwd_allreduce: 60.41 | step: 18.46

  9%|▉         | 3878/41250 [9:21:27<90:22:55,  8.71s/it]
                                                         
{'loss': 0.1442, 'grad_norm': 1.591147541999817, 'learning_rate': 3.957187407949661e-05, 'epoch': 0.94}

  9%|▉         | 3878/41250 [9:21:27<90:22:55,  8.71s/it][2025-04-26 13:30:04,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:30:04,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.17 | bwd_microstep: 5710.15 | bwd_inner_microstep: 5697.37 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.54
[2025-04-26 13:30:04,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.17 | bwd: 5710.17 | bwd_inner: 5697.37 | bwd_allreduce: 12.76 | step: 18.55

  9%|▉         | 3879/41250 [9:21:36<90:09:52,  8.69s/it]
                                                         
{'loss': 0.1682, 'grad_norm': 1.3542540073394775, 'learning_rate': 3.957155084347979e-05, 'epoch': 0.94}

  9%|▉         | 3879/41250 [9:21:36<90:09:52,  8.69s/it][2025-04-26 13:30:13,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 13:30:13,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.74 | bwd_microstep: 5698.98 | bwd_inner_microstep: 5665.23 | bwd_allreduce_microstep: 33.69 | step_microstep: 18.68
[2025-04-26 13:30:13,475] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.74 | bwd: 5699.00 | bwd_inner: 5665.23 | bwd_allreduce: 33.72 | step: 18.69

  9%|▉         | 3880/41250 [9:21:44<89:55:20,  8.66s/it]
                                                         
{'loss': 0.1319, 'grad_norm': 1.296600580215454, 'learning_rate': 3.95712274868082e-05, 'epoch': 0.94}

  9%|▉         | 3880/41250 [9:21:44<89:55:20,  8.66s/it][2025-04-26 13:30:22,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 0.98 | optimizer_step: 0.97
[2025-04-26 13:30:22,114] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.91 | bwd_microstep: 5719.36 | bwd_inner_microstep: 5706.66 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.51
[2025-04-26 13:30:22,115] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.91 | bwd: 5719.38 | bwd_inner: 5706.66 | bwd_allreduce: 12.67 | step: 18.51

  9%|▉         | 3881/41250 [9:21:53<89:50:08,  8.65s/it]
                                                         
{'loss': 0.1893, 'grad_norm': 2.2976877689361572, 'learning_rate': 3.9570904009483844e-05, 'epoch': 0.94}

  9%|▉         | 3881/41250 [9:21:53<89:50:08,  8.65s/it][2025-04-26 13:30:30,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:30:30,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.10 | bwd_microstep: 5691.20 | bwd_inner_microstep: 5678.43 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.54
[2025-04-26 13:30:30,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.10 | bwd: 5691.21 | bwd_inner: 5678.43 | bwd_allreduce: 12.74 | step: 18.55

  9%|▉         | 3882/41250 [9:22:01<89:41:56,  8.64s/it]
                                                         
{'loss': 0.2372, 'grad_norm': 1.6144860982894897, 'learning_rate': 3.957058041150871e-05, 'epoch': 0.94}

  9%|▉         | 3882/41250 [9:22:01<89:41:56,  8.64s/it][2025-04-26 13:30:39,384] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 1.01
[2025-04-26 13:30:39,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.07 | bwd_microstep: 5713.90 | bwd_inner_microstep: 5701.12 | bwd_allreduce_microstep: 12.73 | step_microstep: 19.26
[2025-04-26 13:30:39,385] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.07 | bwd: 5713.92 | bwd_inner: 5701.12 | bwd_allreduce: 12.75 | step: 19.27

  9%|▉         | 3883/41250 [9:22:10<89:45:04,  8.65s/it]
                                                         
{'loss': 0.0681, 'grad_norm': 0.6147465109825134, 'learning_rate': 3.9570256692884805e-05, 'epoch': 0.94}

  9%|▉         | 3883/41250 [9:22:10<89:45:04,  8.65s/it][2025-04-26 13:30:48,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.35 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:30:48,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.11 | bwd_microstep: 5754.24 | bwd_inner_microstep: 5701.32 | bwd_allreduce_microstep: 52.88 | step_microstep: 18.56
[2025-04-26 13:30:48,068] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.11 | bwd: 5754.26 | bwd_inner: 5701.32 | bwd_allreduce: 52.90 | step: 18.56

  9%|▉         | 3884/41250 [9:22:19<89:51:38,  8.66s/it]
                                                         
{'loss': 0.0528, 'grad_norm': 0.9793967008590698, 'learning_rate': 3.956993285361411e-05, 'epoch': 0.94}

  9%|▉         | 3884/41250 [9:22:19<89:51:38,  8.66s/it][2025-04-26 13:30:56,723] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:30:56,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.22 | bwd_microstep: 5723.77 | bwd_inner_microstep: 5710.96 | bwd_allreduce_microstep: 12.76 | step_microstep: 17.92
[2025-04-26 13:30:56,724] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.22 | bwd: 5723.78 | bwd_inner: 5710.96 | bwd_allreduce: 12.78 | step: 17.92

  9%|▉         | 3885/41250 [9:22:27<89:51:04,  8.66s/it]
                                                         
{'loss': 0.0785, 'grad_norm': 1.1025749444961548, 'learning_rate': 3.956960889369864e-05, 'epoch': 0.94}

  9%|▉         | 3885/41250 [9:22:27<89:51:04,  8.66s/it][2025-04-26 13:31:05,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 13:31:05,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.55 | bwd_microstep: 5724.15 | bwd_inner_microstep: 5711.60 | bwd_allreduce_microstep: 12.51 | step_microstep: 18.67
[2025-04-26 13:31:05,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.56 | bwd: 5724.17 | bwd_inner: 5711.60 | bwd_allreduce: 12.53 | step: 18.67

  9%|▉         | 3886/41250 [9:22:36<89:49:11,  8.65s/it]
                                                         
{'loss': 0.3618, 'grad_norm': 1.6531257629394531, 'learning_rate': 3.9569284813140364e-05, 'epoch': 0.94}

  9%|▉         | 3886/41250 [9:22:36<89:49:11,  8.65s/it][2025-04-26 13:31:14,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:31:14,039] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.93 | bwd_microstep: 5740.13 | bwd_inner_microstep: 5692.48 | bwd_allreduce_microstep: 47.61 | step_microstep: 18.41
[2025-04-26 13:31:14,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.93 | bwd: 5740.15 | bwd_inner: 5692.48 | bwd_allreduce: 47.63 | step: 18.41

  9%|▉         | 3887/41250 [9:22:45<89:51:38,  8.66s/it]
                                                         
{'loss': 0.0909, 'grad_norm': 1.274551510810852, 'learning_rate': 3.956896061194131e-05, 'epoch': 0.94}

  9%|▉         | 3887/41250 [9:22:45<89:51:38,  8.66s/it][2025-04-26 13:31:22,660] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:31:22,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.85 | bwd_microstep: 5696.08 | bwd_inner_microstep: 5683.29 | bwd_allreduce_microstep: 12.75 | step_microstep: 18.69
[2025-04-26 13:31:22,661] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.85 | bwd: 5696.10 | bwd_inner: 5683.29 | bwd_allreduce: 12.77 | step: 18.69

  9%|▉         | 3888/41250 [9:22:53<89:44:45,  8.65s/it]
                                                         
{'loss': 0.0419, 'grad_norm': 0.3823777735233307, 'learning_rate': 3.9568636290103456e-05, 'epoch': 0.94}

  9%|▉         | 3888/41250 [9:22:53<89:44:45,  8.65s/it][2025-04-26 13:31:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.18 | optimizer_step: 0.95
[2025-04-26 13:31:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.84 | bwd_microstep: 5992.59 | bwd_inner_microstep: 5705.14 | bwd_allreduce_microstep: 287.39 | step_microstep: 19.02
[2025-04-26 13:31:31,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.84 | bwd: 5992.61 | bwd_inner: 5705.14 | bwd_allreduce: 287.42 | step: 19.03

  9%|▉         | 3889/41250 [9:23:02<90:36:18,  8.73s/it]
                                                         
{'loss': 0.1765, 'grad_norm': 3.438838243484497, 'learning_rate': 3.956831184762881e-05, 'epoch': 0.94}

  9%|▉         | 3889/41250 [9:23:02<90:36:18,  8.73s/it][2025-04-26 13:31:40,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.04 | optimizer_step: 0.89
[2025-04-26 13:31:40,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.35 | bwd_microstep: 5767.67 | bwd_inner_microstep: 5691.24 | bwd_allreduce_microstep: 76.38 | step_microstep: 18.50
[2025-04-26 13:31:40,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.35 | bwd: 5767.68 | bwd_inner: 5691.24 | bwd_allreduce: 76.40 | step: 18.50

  9%|▉         | 3890/41250 [9:23:11<90:28:11,  8.72s/it]
                                                         
{'loss': 0.1574, 'grad_norm': 0.9563020467758179, 'learning_rate': 3.9567987284519366e-05, 'epoch': 0.94}

  9%|▉         | 3890/41250 [9:23:11<90:28:11,  8.72s/it][2025-04-26 13:31:48,950] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:31:48,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.72 | bwd_microstep: 5747.84 | bwd_inner_microstep: 5694.11 | bwd_allreduce_microstep: 53.68 | step_microstep: 18.63
[2025-04-26 13:31:48,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.72 | bwd: 5747.85 | bwd_inner: 5694.11 | bwd_allreduce: 53.70 | step: 18.64

  9%|▉         | 3891/41250 [9:23:20<90:20:35,  8.71s/it]
                                                         
{'loss': 0.179, 'grad_norm': 1.8810240030288696, 'learning_rate': 3.956766260077713e-05, 'epoch': 0.94}

  9%|▉         | 3891/41250 [9:23:20<90:20:35,  8.71s/it][2025-04-26 13:31:57,602] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.22 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 13:31:57,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.14 | bwd_microstep: 5714.08 | bwd_inner_microstep: 5701.42 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.05
[2025-04-26 13:31:57,603] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.14 | bwd: 5714.09 | bwd_inner: 5701.42 | bwd_allreduce: 12.63 | step: 19.05

  9%|▉         | 3892/41250 [9:23:28<90:10:28,  8.69s/it]
                                                         
{'loss': 0.1149, 'grad_norm': 0.852031946182251, 'learning_rate': 3.956733779640411e-05, 'epoch': 0.94}

  9%|▉         | 3892/41250 [9:23:28<90:10:28,  8.69s/it][2025-04-26 13:32:06,274] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:32:06,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.25 | bwd_microstep: 5744.39 | bwd_inner_microstep: 5710.42 | bwd_allreduce_microstep: 33.93 | step_microstep: 18.35
[2025-04-26 13:32:06,275] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.25 | bwd: 5744.40 | bwd_inner: 5710.42 | bwd_allreduce: 33.95 | step: 18.35

  9%|▉         | 3893/41250 [9:23:37<90:06:57,  8.68s/it]
                                                         
{'loss': 0.1332, 'grad_norm': 2.2162530422210693, 'learning_rate': 3.95670128714023e-05, 'epoch': 0.94}

  9%|▉         | 3893/41250 [9:23:37<90:06:57,  8.68s/it][2025-04-26 13:32:14,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:32:14,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.26 | bwd_microstep: 5714.35 | bwd_inner_microstep: 5657.12 | bwd_allreduce_microstep: 57.19 | step_microstep: 18.54
[2025-04-26 13:32:14,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.26 | bwd: 5714.37 | bwd_inner: 5657.12 | bwd_allreduce: 57.21 | step: 18.54

  9%|▉         | 3894/41250 [9:23:46<89:54:51,  8.67s/it]
                                                         
{'loss': 0.1213, 'grad_norm': 1.220577597618103, 'learning_rate': 3.95666878257737e-05, 'epoch': 0.94}

  9%|▉         | 3894/41250 [9:23:46<89:54:51,  8.67s/it][2025-04-26 13:32:23,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 13:32:23,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.26 | bwd_microstep: 5881.27 | bwd_inner_microstep: 5665.28 | bwd_allreduce_microstep: 215.94 | step_microstep: 18.81
[2025-04-26 13:32:23,680] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.26 | bwd: 5881.28 | bwd_inner: 5665.28 | bwd_allreduce: 215.95 | step: 18.81

  9%|▉         | 3895/41250 [9:23:54<90:17:10,  8.70s/it]
                                                         
{'loss': 0.2253, 'grad_norm': 1.6472628116607666, 'learning_rate': 3.9566362659520314e-05, 'epoch': 0.94}

  9%|▉         | 3895/41250 [9:23:54<90:17:10,  8.70s/it][2025-04-26 13:32:32,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:32:32,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.22 | bwd_microstep: 5744.89 | bwd_inner_microstep: 5680.03 | bwd_allreduce_microstep: 64.81 | step_microstep: 18.74
[2025-04-26 13:32:32,354] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.22 | bwd: 5744.91 | bwd_inner: 5680.03 | bwd_allreduce: 64.83 | step: 18.74

  9%|▉         | 3896/41250 [9:24:03<90:11:54,  8.69s/it]
                                                         
{'loss': 0.1966, 'grad_norm': 1.6636621952056885, 'learning_rate': 3.956603737264416e-05, 'epoch': 0.94}

  9%|▉         | 3896/41250 [9:24:03<90:11:54,  8.69s/it][2025-04-26 13:32:41,074] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:32:41,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.36 | bwd_microstep: 5782.34 | bwd_inner_microstep: 5706.74 | bwd_allreduce_microstep: 75.56 | step_microstep: 18.71
[2025-04-26 13:32:41,075] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.36 | bwd: 5782.36 | bwd_inner: 5706.74 | bwd_allreduce: 75.58 | step: 18.71

  9%|▉         | 3897/41250 [9:24:12<90:17:14,  8.70s/it]
                                                         
{'loss': 0.1318, 'grad_norm': 0.9462710022926331, 'learning_rate': 3.956571196514722e-05, 'epoch': 0.94}

  9%|▉         | 3897/41250 [9:24:12<90:17:14,  8.70s/it][2025-04-26 13:32:49,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.02 | optimizer_step: 1.03
[2025-04-26 13:32:49,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.73 | bwd_microstep: 5756.56 | bwd_inner_microstep: 5683.07 | bwd_allreduce_microstep: 73.44 | step_microstep: 19.42
[2025-04-26 13:32:49,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.73 | bwd: 5756.57 | bwd_inner: 5683.07 | bwd_allreduce: 73.46 | step: 19.42

  9%|▉         | 3898/41250 [9:24:20<90:14:09,  8.70s/it]
                                                         
{'loss': 0.0721, 'grad_norm': 0.8761919140815735, 'learning_rate': 3.956538643703153e-05, 'epoch': 0.94}

  9%|▉         | 3898/41250 [9:24:20<90:14:09,  8.70s/it][2025-04-26 13:32:58,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.21 | optimizer_step: 0.90
[2025-04-26 13:32:58,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.60 | bwd_microstep: 5755.80 | bwd_inner_microstep: 5703.83 | bwd_allreduce_microstep: 51.92 | step_microstep: 19.11
[2025-04-26 13:32:58,440] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.60 | bwd: 5755.81 | bwd_inner: 5703.82 | bwd_allreduce: 51.94 | step: 19.11

  9%|▉         | 3899/41250 [9:24:29<90:10:24,  8.69s/it]
                                                         
{'loss': 0.1738, 'grad_norm': 1.849104404449463, 'learning_rate': 3.9565060788299075e-05, 'epoch': 0.95}

  9%|▉         | 3899/41250 [9:24:29<90:10:24,  8.69s/it][2025-04-26 13:33:07,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:33:07,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.46 | bwd_microstep: 5701.74 | bwd_inner_microstep: 5688.97 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.66
[2025-04-26 13:33:07,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.46 | bwd: 5701.75 | bwd_inner: 5688.97 | bwd_allreduce: 12.74 | step: 18.66

  9%|▉         | 3900/41250 [9:24:38<90:00:41,  8.68s/it]
                                                         
{'loss': 0.1549, 'grad_norm': 0.7621117830276489, 'learning_rate': 3.956473501895187e-05, 'epoch': 0.95}

  9%|▉         | 3900/41250 [9:24:38<90:00:41,  8.68s/it][2025-04-26 13:33:15,716] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.05 | optimizer_step: 0.89
[2025-04-26 13:33:15,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.12 | bwd_microstep: 5700.32 | bwd_inner_microstep: 5687.54 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.86
[2025-04-26 13:33:15,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.13 | bwd: 5700.33 | bwd_inner: 5687.54 | bwd_allreduce: 12.75 | step: 18.87

  9%|▉         | 3901/41250 [9:24:46<89:53:46,  8.66s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.2861604690551758, 'learning_rate': 3.956440912899192e-05, 'epoch': 0.95}

  9%|▉         | 3901/41250 [9:24:46<89:53:46,  8.66s/it][2025-04-26 13:33:24,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 1.05 | optimizer_step: 0.90
[2025-04-26 13:33:24,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.34 | bwd_microstep: 5718.91 | bwd_inner_microstep: 5706.49 | bwd_allreduce_microstep: 12.38 | step_microstep: 19.14
[2025-04-26 13:33:24,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.34 | bwd: 5718.93 | bwd_inner: 5706.49 | bwd_allreduce: 12.40 | step: 19.14

  9%|▉         | 3902/41250 [9:24:55<89:52:37,  8.66s/it]
                                                         
{'loss': 0.0293, 'grad_norm': 0.4265589416027069, 'learning_rate': 3.956408311842124e-05, 'epoch': 0.95}

  9%|▉         | 3902/41250 [9:24:55<89:52:37,  8.66s/it][2025-04-26 13:33:33,021] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.25 | optimizer_step: 1.09
[2025-04-26 13:33:33,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.07 | bwd_microstep: 5708.58 | bwd_inner_microstep: 5694.94 | bwd_allreduce_microstep: 13.58 | step_microstep: 19.87
[2025-04-26 13:33:33,022] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.07 | bwd: 5708.59 | bwd_inner: 5694.94 | bwd_allreduce: 13.60 | step: 19.87

  9%|▉         | 3903/41250 [9:25:04<89:48:47,  8.66s/it]
                                                         
{'loss': 0.1057, 'grad_norm': 1.542784571647644, 'learning_rate': 3.9563756987241825e-05, 'epoch': 0.95}

  9%|▉         | 3903/41250 [9:25:04<89:48:47,  8.66s/it][2025-04-26 13:33:41,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:33:41,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.23 | bwd_microstep: 5851.70 | bwd_inner_microstep: 5783.56 | bwd_allreduce_microstep: 68.09 | step_microstep: 18.60
[2025-04-26 13:33:41,842] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2892.23 | bwd: 5851.71 | bwd_inner: 5783.56 | bwd_allreduce: 68.11 | step: 18.61

  9%|▉         | 3904/41250 [9:25:13<90:18:54,  8.71s/it]
                                                         
{'loss': 0.0361, 'grad_norm': 0.4723818004131317, 'learning_rate': 3.9563430735455706e-05, 'epoch': 0.95}

  9%|▉         | 3904/41250 [9:25:13<90:18:54,  8.71s/it][2025-04-26 13:33:50,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:33:50,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.26 | bwd_microstep: 5765.48 | bwd_inner_microstep: 5704.75 | bwd_allreduce_microstep: 60.68 | step_microstep: 18.53
[2025-04-26 13:33:50,541] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.25 | bwd: 5765.49 | bwd_inner: 5704.75 | bwd_allreduce: 60.69 | step: 18.53

  9%|▉         | 3905/41250 [9:25:21<90:17:24,  8.70s/it]
                                                         
{'loss': 0.0581, 'grad_norm': 0.7070940732955933, 'learning_rate': 3.9563104363064876e-05, 'epoch': 0.95}

  9%|▉         | 3905/41250 [9:25:21<90:17:24,  8.70s/it][2025-04-26 13:33:59,235] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.93
[2025-04-26 13:33:59,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5779.72 | bwd_inner_microstep: 5670.51 | bwd_allreduce_microstep: 109.17 | step_microstep: 18.79
[2025-04-26 13:33:59,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5779.74 | bwd_inner: 5670.51 | bwd_allreduce: 109.19 | step: 18.79

  9%|▉         | 3906/41250 [9:25:30<90:15:39,  8.70s/it]
                                                         
{'loss': 0.0221, 'grad_norm': 0.21447503566741943, 'learning_rate': 3.956277787007136e-05, 'epoch': 0.95}

  9%|▉         | 3906/41250 [9:25:30<90:15:39,  8.70s/it][2025-04-26 13:34:08,088] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 1.03 | optimizer_step: 0.89
[2025-04-26 13:34:08,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.47 | bwd_microstep: 5933.45 | bwd_inner_microstep: 5671.80 | bwd_allreduce_microstep: 261.59 | step_microstep: 18.92
[2025-04-26 13:34:08,089] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.47 | bwd: 5933.47 | bwd_inner: 5671.80 | bwd_allreduce: 261.61 | step: 18.92

  9%|▉         | 3907/41250 [9:25:39<90:43:48,  8.75s/it]
                                                         
{'loss': 0.1827, 'grad_norm': 1.2638639211654663, 'learning_rate': 3.9562451256477164e-05, 'epoch': 0.95}

  9%|▉         | 3907/41250 [9:25:39<90:43:48,  8.75s/it][2025-04-26 13:34:16,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.41 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:34:16,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.70 | bwd_microstep: 5718.03 | bwd_inner_microstep: 5705.04 | bwd_allreduce_microstep: 12.94 | step_microstep: 18.80
[2025-04-26 13:34:16,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.70 | bwd: 5718.04 | bwd_inner: 5705.04 | bwd_allreduce: 12.96 | step: 18.80

  9%|▉         | 3908/41250 [9:25:47<90:25:33,  8.72s/it]
                                                         
{'loss': 0.2205, 'grad_norm': 2.0362017154693604, 'learning_rate': 3.956212452228431e-05, 'epoch': 0.95}

  9%|▉         | 3908/41250 [9:25:47<90:25:33,  8.72s/it][2025-04-26 13:34:25,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 13:34:25,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.30 | bwd_microstep: 5792.26 | bwd_inner_microstep: 5669.28 | bwd_allreduce_microstep: 122.93 | step_microstep: 19.08
[2025-04-26 13:34:25,449] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.30 | bwd: 5792.27 | bwd_inner: 5669.28 | bwd_allreduce: 122.95 | step: 19.08

  9%|▉         | 3909/41250 [9:25:56<90:24:09,  8.72s/it]
                                                         
{'loss': 0.0709, 'grad_norm': 1.1487394571304321, 'learning_rate': 3.95617976674948e-05, 'epoch': 0.95}

  9%|▉         | 3909/41250 [9:25:56<90:24:09,  8.72s/it][2025-04-26 13:34:34,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:34:34,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.85 | bwd_microstep: 5710.66 | bwd_inner_microstep: 5697.76 | bwd_allreduce_microstep: 12.86 | step_microstep: 18.62
[2025-04-26 13:34:34,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.85 | bwd: 5710.68 | bwd_inner: 5697.76 | bwd_allreduce: 12.87 | step: 18.63

  9%|▉         | 3910/41250 [9:26:05<90:10:58,  8.69s/it]
                                                         
{'loss': 0.1949, 'grad_norm': 1.3049328327178955, 'learning_rate': 3.9561470692110664e-05, 'epoch': 0.95}

  9%|▉         | 3910/41250 [9:26:05<90:10:58,  8.69s/it][2025-04-26 13:34:42,760] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.93
[2025-04-26 13:34:42,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.43 | bwd_microstep: 5727.00 | bwd_inner_microstep: 5714.30 | bwd_allreduce_microstep: 12.65 | step_microstep: 18.47
[2025-04-26 13:34:42,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.43 | bwd: 5727.01 | bwd_inner: 5714.30 | bwd_allreduce: 12.67 | step: 18.47

  9%|▉         | 3911/41250 [9:26:13<90:05:23,  8.69s/it]
                                                         
{'loss': 0.0687, 'grad_norm': 1.233359694480896, 'learning_rate': 3.95611435961339e-05, 'epoch': 0.95}

  9%|▉         | 3911/41250 [9:26:13<90:05:23,  8.69s/it][2025-04-26 13:34:51,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:34:51,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.44 | bwd_microstep: 5702.21 | bwd_inner_microstep: 5664.58 | bwd_allreduce_microstep: 37.58 | step_microstep: 18.64
[2025-04-26 13:34:51,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.44 | bwd: 5702.22 | bwd_inner: 5664.58 | bwd_allreduce: 37.60 | step: 18.65

  9%|▉         | 3912/41250 [9:26:22<89:52:50,  8.67s/it]
                                                         
{'loss': 0.039, 'grad_norm': 0.3802572190761566, 'learning_rate': 3.9560816379566536e-05, 'epoch': 0.95}

  9%|▉         | 3912/41250 [9:26:22<89:52:50,  8.67s/it][2025-04-26 13:35:00,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.03
[2025-04-26 13:35:00,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.79 | bwd_microstep: 5797.49 | bwd_inner_microstep: 5784.83 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.74
[2025-04-26 13:35:00,150] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.79 | bwd: 5797.50 | bwd_inner: 5784.83 | bwd_allreduce: 12.64 | step: 18.74

  9%|▉         | 3913/41250 [9:26:31<90:12:10,  8.70s/it]
                                                         
{'loss': 0.1971, 'grad_norm': 1.611221194267273, 'learning_rate': 3.9560489042410586e-05, 'epoch': 0.95}

  9%|▉         | 3913/41250 [9:26:31<90:12:10,  8.70s/it][2025-04-26 13:35:08,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.98
[2025-04-26 13:35:08,832] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5750.10 | bwd_inner_microstep: 5696.53 | bwd_allreduce_microstep: 53.51 | step_microstep: 19.02
[2025-04-26 13:35:08,833] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5750.11 | bwd_inner: 5696.53 | bwd_allreduce: 53.53 | step: 19.03

  9%|▉         | 3914/41250 [9:26:40<90:09:14,  8.69s/it]
                                                         
{'loss': 0.1509, 'grad_norm': 1.9881577491760254, 'learning_rate': 3.9560161584668064e-05, 'epoch': 0.95}

  9%|▉         | 3914/41250 [9:26:40<90:09:14,  8.69s/it][2025-04-26 13:35:17,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:35:17,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.22 | bwd_microstep: 5732.64 | bwd_inner_microstep: 5720.18 | bwd_allreduce_microstep: 12.42 | step_microstep: 18.61
[2025-04-26 13:35:17,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.22 | bwd: 5732.66 | bwd_inner: 5720.18 | bwd_allreduce: 12.44 | step: 18.61

  9%|▉         | 3915/41250 [9:26:48<90:06:24,  8.69s/it]
                                                         
{'loss': 0.188, 'grad_norm': 1.43052339553833, 'learning_rate': 3.9559834006341e-05, 'epoch': 0.95}

  9%|▉         | 3915/41250 [9:26:48<90:06:24,  8.69s/it][2025-04-26 13:35:26,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:35:26,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.08 | bwd_microstep: 5763.14 | bwd_inner_microstep: 5699.58 | bwd_allreduce_microstep: 63.51 | step_microstep: 18.57
[2025-04-26 13:35:26,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.07 | bwd: 5763.15 | bwd_inner: 5699.58 | bwd_allreduce: 63.53 | step: 18.58

  9%|▉         | 3916/41250 [9:26:57<90:08:38,  8.69s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 1.252131700515747, 'learning_rate': 3.9559506307431404e-05, 'epoch': 0.95}

  9%|▉         | 3916/41250 [9:26:57<90:08:38,  8.69s/it][2025-04-26 13:35:34,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:35:34,902] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.00 | bwd_microstep: 5768.83 | bwd_inner_microstep: 5701.03 | bwd_allreduce_microstep: 67.76 | step_microstep: 18.63
[2025-04-26 13:35:34,903] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.00 | bwd: 5768.84 | bwd_inner: 5701.03 | bwd_allreduce: 67.77 | step: 18.64

  9%|▉         | 3917/41250 [9:27:06<90:08:10,  8.69s/it]
                                                         
{'loss': 0.2544, 'grad_norm': 2.6628453731536865, 'learning_rate': 3.95591784879413e-05, 'epoch': 0.95}

  9%|▉         | 3917/41250 [9:27:06<90:08:10,  8.69s/it][2025-04-26 13:35:43,558] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:35:43,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.81 | bwd_microstep: 5721.78 | bwd_inner_microstep: 5709.00 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.90
[2025-04-26 13:35:43,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.82 | bwd: 5721.80 | bwd_inner: 5709.00 | bwd_allreduce: 12.75 | step: 18.90

  9%|▉         | 3918/41250 [9:27:14<90:01:37,  8.68s/it]
                                                         
{'loss': 0.5161, 'grad_norm': 3.881168842315674, 'learning_rate': 3.95588505478727e-05, 'epoch': 0.95}

  9%|▉         | 3918/41250 [9:27:14<90:01:37,  8.68s/it][2025-04-26 13:35:52,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.03 | optimizer_step: 0.95
[2025-04-26 13:35:52,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.71 | bwd_microstep: 5822.82 | bwd_inner_microstep: 5670.93 | bwd_allreduce_microstep: 151.84 | step_microstep: 18.81
[2025-04-26 13:35:52,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.71 | bwd: 5822.84 | bwd_inner: 5670.93 | bwd_allreduce: 151.86 | step: 18.81

 10%|▉         | 3919/41250 [9:27:23<90:11:38,  8.70s/it]
                                                         
{'loss': 0.2523, 'grad_norm': 3.129020929336548, 'learning_rate': 3.955852248722764e-05, 'epoch': 0.95}

 10%|▉         | 3919/41250 [9:27:23<90:11:38,  8.70s/it][2025-04-26 13:36:00,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:36:00,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.05 | bwd_microstep: 5715.87 | bwd_inner_microstep: 5685.95 | bwd_allreduce_microstep: 29.87 | step_microstep: 17.96
[2025-04-26 13:36:00,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.05 | bwd: 5715.88 | bwd_inner: 5685.95 | bwd_allreduce: 29.89 | step: 17.97

 10%|▉         | 3920/41250 [9:27:32<90:03:27,  8.68s/it]
                                                         
{'loss': 0.0811, 'grad_norm': 1.7916805744171143, 'learning_rate': 3.955819430600814e-05, 'epoch': 0.95}

 10%|▉         | 3920/41250 [9:27:32<90:03:27,  8.68s/it][2025-04-26 13:36:09,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:36:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2896.18 | bwd_microstep: 5797.94 | bwd_inner_microstep: 5785.21 | bwd_allreduce_microstep: 12.69 | step_microstep: 17.73
[2025-04-26 13:36:09,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2896.18 | bwd: 5797.96 | bwd_inner: 5785.21 | bwd_allreduce: 12.70 | step: 17.73

 10%|▉         | 3921/41250 [9:27:40<90:20:35,  8.71s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 1.6292128562927246, 'learning_rate': 3.955786600421621e-05, 'epoch': 0.95}

 10%|▉         | 3921/41250 [9:27:40<90:20:35,  8.71s/it][2025-04-26 13:36:18,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:36:18,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5748.97 | bwd_inner_microstep: 5697.95 | bwd_allreduce_microstep: 50.98 | step_microstep: 18.28
[2025-04-26 13:36:18,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5748.98 | bwd_inner: 5697.95 | bwd_allreduce: 51.00 | step: 18.28

 10%|▉         | 3922/41250 [9:27:49<90:14:26,  8.70s/it]
                                                         
{'loss': 0.1498, 'grad_norm': 1.747310996055603, 'learning_rate': 3.955753758185389e-05, 'epoch': 0.95}

 10%|▉         | 3922/41250 [9:27:49<90:14:26,  8.70s/it][2025-04-26 13:36:27,105] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.96 | optimizer_step: 1.04
[2025-04-26 13:36:27,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2863.63 | bwd_microstep: 5745.23 | bwd_inner_microstep: 5693.87 | bwd_allreduce_microstep: 51.32 | step_microstep: 18.27
[2025-04-26 13:36:27,106] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2863.63 | bwd: 5745.24 | bwd_inner: 5693.87 | bwd_allreduce: 51.33 | step: 18.27

 10%|▉         | 3923/41250 [9:27:58<90:12:57,  8.70s/it]
                                                         
{'loss': 0.1835, 'grad_norm': 1.7420192956924438, 'learning_rate': 3.9557209038923196e-05, 'epoch': 0.95}

 10%|▉         | 3923/41250 [9:27:58<90:12:57,  8.70s/it][2025-04-26 13:36:35,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:36:35,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2868.76 | bwd_microstep: 5719.96 | bwd_inner_microstep: 5707.29 | bwd_allreduce_microstep: 12.62 | step_microstep: 18.28
[2025-04-26 13:36:35,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2868.76 | bwd: 5719.97 | bwd_inner: 5707.29 | bwd_allreduce: 12.64 | step: 18.28

 10%|▉         | 3924/41250 [9:28:06<90:08:19,  8.69s/it]
                                                         
{'loss': 0.1865, 'grad_norm': 1.848414421081543, 'learning_rate': 3.955688037542615e-05, 'epoch': 0.95}

 10%|▉         | 3924/41250 [9:28:06<90:08:19,  8.69s/it][2025-04-26 13:36:44,438] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:36:44,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.64 | bwd_microstep: 5712.04 | bwd_inner_microstep: 5699.45 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.27
[2025-04-26 13:36:44,439] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.64 | bwd: 5712.05 | bwd_inner: 5699.45 | bwd_allreduce: 12.56 | step: 18.27

 10%|▉         | 3925/41250 [9:28:15<90:01:09,  8.68s/it]
                                                         
{'loss': 0.1197, 'grad_norm': 1.4399551153182983, 'learning_rate': 3.9556551591364786e-05, 'epoch': 0.95}

 10%|▉         | 3925/41250 [9:28:15<90:01:09,  8.68s/it][2025-04-26 13:36:53,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:36:53,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.84 | bwd_microstep: 5692.03 | bwd_inner_microstep: 5657.51 | bwd_allreduce_microstep: 34.48 | step_microstep: 18.31
[2025-04-26 13:36:53,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.84 | bwd: 5692.05 | bwd_inner: 5657.50 | bwd_allreduce: 34.50 | step: 18.32

 10%|▉         | 3926/41250 [9:28:24<89:44:19,  8.66s/it]
                                                         
{'loss': 0.1545, 'grad_norm': 2.7335853576660156, 'learning_rate': 3.9556222686741136e-05, 'epoch': 0.95}

 10%|▉         | 3926/41250 [9:28:24<89:44:19,  8.66s/it][2025-04-26 13:37:01,703] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:37:01,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.86 | bwd_microstep: 5726.25 | bwd_inner_microstep: 5692.97 | bwd_allreduce_microstep: 33.24 | step_microstep: 18.26
[2025-04-26 13:37:01,704] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.85 | bwd: 5726.26 | bwd_inner: 5692.97 | bwd_allreduce: 33.25 | step: 18.26

 10%|▉         | 3927/41250 [9:28:32<89:47:14,  8.66s/it]
                                                         
{'loss': 0.1449, 'grad_norm': 1.4671763181686401, 'learning_rate': 3.955589366155721e-05, 'epoch': 0.95}

 10%|▉         | 3927/41250 [9:28:32<89:47:14,  8.66s/it][2025-04-26 13:37:10,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:37:10,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.08 | bwd_microstep: 5766.22 | bwd_inner_microstep: 5638.67 | bwd_allreduce_microstep: 127.51 | step_microstep: 18.23
[2025-04-26 13:37:10,381] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.08 | bwd: 5766.23 | bwd_inner: 5638.67 | bwd_allreduce: 127.52 | step: 18.23

 10%|▉         | 3928/41250 [9:28:41<89:50:12,  8.67s/it]
                                                         
{'loss': 0.1095, 'grad_norm': 1.0647850036621094, 'learning_rate': 3.9555564515815044e-05, 'epoch': 0.95}

 10%|▉         | 3928/41250 [9:28:41<89:50:12,  8.67s/it][2025-04-26 13:37:19,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:37:19,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.67 | bwd_microstep: 5688.23 | bwd_inner_microstep: 5675.40 | bwd_allreduce_microstep: 12.79 | step_microstep: 17.81
[2025-04-26 13:37:19,008] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.68 | bwd: 5688.25 | bwd_inner: 5675.40 | bwd_allreduce: 12.81 | step: 17.81

 10%|▉         | 3929/41250 [9:28:50<89:42:54,  8.65s/it]
                                                         
{'loss': 0.1372, 'grad_norm': 1.853834629058838, 'learning_rate': 3.9555235249516675e-05, 'epoch': 0.95}

 10%|▉         | 3929/41250 [9:28:50<89:42:54,  8.65s/it][2025-04-26 13:37:27,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.94 | optimizer_step: 0.89
[2025-04-26 13:37:27,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.94 | bwd_microstep: 5675.56 | bwd_inner_microstep: 5651.36 | bwd_allreduce_microstep: 24.16 | step_microstep: 18.20
[2025-04-26 13:37:27,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.94 | bwd: 5675.57 | bwd_inner: 5651.36 | bwd_allreduce: 24.18 | step: 18.21

 10%|▉         | 3930/41250 [9:28:58<89:29:40,  8.63s/it]
                                                         
{'loss': 0.0568, 'grad_norm': 0.7463034391403198, 'learning_rate': 3.955490586266413e-05, 'epoch': 0.95}

 10%|▉         | 3930/41250 [9:28:58<89:29:40,  8.63s/it][2025-04-26 13:37:36,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:37:36,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.63 | bwd_microstep: 5687.89 | bwd_inner_microstep: 5645.50 | bwd_allreduce_microstep: 42.35 | step_microstep: 17.85
[2025-04-26 13:37:36,190] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.63 | bwd: 5687.90 | bwd_inner: 5645.50 | bwd_allreduce: 42.36 | step: 17.85

 10%|▉         | 3931/41250 [9:29:07<89:23:06,  8.62s/it]
                                                         
{'loss': 0.3804, 'grad_norm': 4.055363655090332, 'learning_rate': 3.955457635525943e-05, 'epoch': 0.95}

 10%|▉         | 3931/41250 [9:29:07<89:23:06,  8.62s/it][2025-04-26 13:37:44,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 13:37:44,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2870.59 | bwd_microstep: 5807.95 | bwd_inner_microstep: 5686.60 | bwd_allreduce_microstep: 121.31 | step_microstep: 19.06
[2025-04-26 13:37:44,951] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2870.59 | bwd: 5807.97 | bwd_inner: 5686.60 | bwd_allreduce: 121.33 | step: 19.06

 10%|▉         | 3932/41250 [9:29:16<89:48:57,  8.66s/it]
                                                         
{'loss': 0.0646, 'grad_norm': 0.772676944732666, 'learning_rate': 3.955424672730462e-05, 'epoch': 0.95}

 10%|▉         | 3932/41250 [9:29:16<89:48:57,  8.66s/it][2025-04-26 13:37:53,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:37:53,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.23 | bwd_microstep: 5701.22 | bwd_inner_microstep: 5658.45 | bwd_allreduce_microstep: 42.73 | step_microstep: 18.34
[2025-04-26 13:37:53,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.23 | bwd: 5701.24 | bwd_inner: 5658.44 | bwd_allreduce: 42.75 | step: 18.35

 10%|▉         | 3933/41250 [9:29:24<89:38:16,  8.65s/it]
                                                         
{'loss': 0.3976, 'grad_norm': 3.4187216758728027, 'learning_rate': 3.9553916978801714e-05, 'epoch': 0.95}

 10%|▉         | 3933/41250 [9:29:24<89:38:16,  8.65s/it][2025-04-26 13:38:02,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.00 | optimizer_step: 1.07
[2025-04-26 13:38:02,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.34 | bwd_microstep: 5760.51 | bwd_inner_microstep: 5702.66 | bwd_allreduce_microstep: 57.80 | step_microstep: 18.90
[2025-04-26 13:38:02,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.34 | bwd: 5760.53 | bwd_inner: 5702.66 | bwd_allreduce: 57.82 | step: 18.90

 10%|▉         | 3934/41250 [9:29:33<89:45:38,  8.66s/it]
                                                         
{'loss': 0.278, 'grad_norm': 4.429871559143066, 'learning_rate': 3.955358710975277e-05, 'epoch': 0.95}

 10%|▉         | 3934/41250 [9:29:33<89:45:38,  8.66s/it][2025-04-26 13:38:10,938] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:38:10,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.93 | bwd_microstep: 5773.67 | bwd_inner_microstep: 5704.17 | bwd_allreduce_microstep: 69.46 | step_microstep: 18.82
[2025-04-26 13:38:10,939] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.93 | bwd: 5773.69 | bwd_inner: 5704.17 | bwd_allreduce: 69.48 | step: 18.83

 10%|▉         | 3935/41250 [9:29:42<89:51:33,  8.67s/it]
                                                         
{'loss': 0.1068, 'grad_norm': 1.6182993650436401, 'learning_rate': 3.955325712015979e-05, 'epoch': 0.95}

 10%|▉         | 3935/41250 [9:29:42<89:51:33,  8.67s/it][2025-04-26 13:38:19,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:38:19,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.10 | bwd_microstep: 5697.77 | bwd_inner_microstep: 5684.94 | bwd_allreduce_microstep: 12.79 | step_microstep: 18.77
[2025-04-26 13:38:19,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.10 | bwd: 5697.78 | bwd_inner: 5684.94 | bwd_allreduce: 12.80 | step: 18.77

 10%|▉         | 3936/41250 [9:29:50<89:42:18,  8.65s/it]
                                                         
{'loss': 0.1246, 'grad_norm': 1.4997937679290771, 'learning_rate': 3.955292701002484e-05, 'epoch': 0.95}

 10%|▉         | 3936/41250 [9:29:50<89:42:18,  8.65s/it][2025-04-26 13:38:28,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.46 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:38:28,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.64 | bwd_microstep: 5767.44 | bwd_inner_microstep: 5650.35 | bwd_allreduce_microstep: 117.04 | step_microstep: 18.68
[2025-04-26 13:38:28,236] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.64 | bwd: 5767.45 | bwd_inner: 5650.35 | bwd_allreduce: 117.06 | step: 18.68

 10%|▉         | 3937/41250 [9:29:59<89:46:19,  8.66s/it]
                                                         
{'loss': 0.1145, 'grad_norm': 1.8665140867233276, 'learning_rate': 3.955259677934994e-05, 'epoch': 0.95}

 10%|▉         | 3937/41250 [9:29:59<89:46:19,  8.66s/it][2025-04-26 13:38:36,940] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.08
[2025-04-26 13:38:36,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.19 | bwd_microstep: 5771.48 | bwd_inner_microstep: 5717.32 | bwd_allreduce_microstep: 54.12 | step_microstep: 18.84
[2025-04-26 13:38:36,941] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.19 | bwd: 5771.49 | bwd_inner: 5717.31 | bwd_allreduce: 54.14 | step: 18.84

 10%|▉         | 3938/41250 [9:30:08<89:54:15,  8.67s/it]
                                                         
{'loss': 0.2286, 'grad_norm': 1.3608431816101074, 'learning_rate': 3.955226642813712e-05, 'epoch': 0.95}

 10%|▉         | 3938/41250 [9:30:08<89:54:15,  8.67s/it][2025-04-26 13:38:45,712] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:38:45,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.51 | bwd_microstep: 5796.03 | bwd_inner_microstep: 5783.18 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.79
[2025-04-26 13:38:45,713] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2893.51 | bwd: 5796.04 | bwd_inner: 5783.18 | bwd_allreduce: 12.82 | step: 18.79

 10%|▉         | 3939/41250 [9:30:16<90:12:50,  8.70s/it]
                                                         
{'loss': 0.4144, 'grad_norm': 2.2488930225372314, 'learning_rate': 3.955193595638842e-05, 'epoch': 0.95}

 10%|▉         | 3939/41250 [9:30:16<90:12:50,  8.70s/it][2025-04-26 13:38:54,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:38:54,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.97 | bwd_microstep: 5761.57 | bwd_inner_microstep: 5708.26 | bwd_allreduce_microstep: 53.26 | step_microstep: 18.65
[2025-04-26 13:38:54,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.97 | bwd: 5761.58 | bwd_inner: 5708.26 | bwd_allreduce: 53.28 | step: 18.65

 10%|▉         | 3940/41250 [9:30:25<90:10:26,  8.70s/it]
                                                         
{'loss': 0.1894, 'grad_norm': 1.1358977556228638, 'learning_rate': 3.955160536410589e-05, 'epoch': 0.96}

 10%|▉         | 3940/41250 [9:30:25<90:10:26,  8.70s/it][2025-04-26 13:39:03,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 13:39:03,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.16 | bwd_microstep: 5740.96 | bwd_inner_microstep: 5699.97 | bwd_allreduce_microstep: 40.94 | step_microstep: 18.73
[2025-04-26 13:39:03,080] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.16 | bwd: 5740.98 | bwd_inner: 5699.97 | bwd_allreduce: 40.97 | step: 18.74

 10%|▉         | 3941/41250 [9:30:34<90:04:47,  8.69s/it]
                                                         
{'loss': 0.0287, 'grad_norm': 0.3669043183326721, 'learning_rate': 3.955127465129156e-05, 'epoch': 0.96}

 10%|▉         | 3941/41250 [9:30:34<90:04:47,  8.69s/it][2025-04-26 13:39:11,798] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:39:11,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.28 | bwd_microstep: 5788.39 | bwd_inner_microstep: 5690.61 | bwd_allreduce_microstep: 97.75 | step_microstep: 18.09
[2025-04-26 13:39:11,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.28 | bwd: 5788.41 | bwd_inner: 5690.61 | bwd_allreduce: 97.76 | step: 18.09

 10%|▉         | 3942/41250 [9:30:42<90:09:34,  8.70s/it]
                                                         
{'loss': 0.1467, 'grad_norm': 1.5216158628463745, 'learning_rate': 3.955094381794746e-05, 'epoch': 0.96}

 10%|▉         | 3942/41250 [9:30:42<90:09:34,  8.70s/it][2025-04-26 13:39:20,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:39:20,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.73 | bwd_microstep: 5790.87 | bwd_inner_microstep: 5651.58 | bwd_allreduce_microstep: 139.24 | step_microstep: 18.07
[2025-04-26 13:39:20,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.73 | bwd: 5790.88 | bwd_inner: 5651.58 | bwd_allreduce: 139.26 | step: 18.08

 10%|▉         | 3943/41250 [9:30:51<90:10:12,  8.70s/it]
                                                         
{'loss': 0.0676, 'grad_norm': 0.7810387015342712, 'learning_rate': 3.9550612864075636e-05, 'epoch': 0.96}

 10%|▉         | 3943/41250 [9:30:51<90:10:12,  8.70s/it][2025-04-26 13:39:29,183] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:39:29,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.73 | bwd_microstep: 5754.72 | bwd_inner_microstep: 5711.29 | bwd_allreduce_microstep: 43.38 | step_microstep: 18.19
[2025-04-26 13:39:29,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.73 | bwd: 5754.73 | bwd_inner: 5711.29 | bwd_allreduce: 43.40 | step: 18.19

 10%|▉         | 3944/41250 [9:31:00<90:06:21,  8.70s/it]
                                                         
{'loss': 0.0212, 'grad_norm': 0.23197762668132782, 'learning_rate': 3.955028178967813e-05, 'epoch': 0.96}

 10%|▉         | 3944/41250 [9:31:00<90:06:21,  8.70s/it][2025-04-26 13:39:37,871] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.95 | optimizer_step: 0.90
[2025-04-26 13:39:37,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.34 | bwd_microstep: 5779.35 | bwd_inner_microstep: 5658.76 | bwd_allreduce_microstep: 120.55 | step_microstep: 18.17
[2025-04-26 13:39:37,872] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.34 | bwd: 5779.37 | bwd_inner: 5658.76 | bwd_allreduce: 120.57 | step: 18.17

 10%|▉         | 3945/41250 [9:31:09<90:04:50,  8.69s/it]
                                                         
{'loss': 0.1715, 'grad_norm': 1.4532771110534668, 'learning_rate': 3.954995059475699e-05, 'epoch': 0.96}

 10%|▉         | 3945/41250 [9:31:09<90:04:50,  8.69s/it][2025-04-26 13:39:46,533] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.03 | optimizer_step: 0.98
[2025-04-26 13:39:46,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.42 | bwd_microstep: 5717.53 | bwd_inner_microstep: 5704.59 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.18
[2025-04-26 13:39:46,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.42 | bwd: 5717.54 | bwd_inner: 5704.59 | bwd_allreduce: 12.91 | step: 19.18

 10%|▉         | 3946/41250 [9:31:17<89:59:15,  8.68s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 1.3134430646896362, 'learning_rate': 3.954961927931424e-05, 'epoch': 0.96}

 10%|▉         | 3946/41250 [9:31:17<89:59:15,  8.68s/it][2025-04-26 13:39:55,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 0.95
[2025-04-26 13:39:55,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.06 | bwd_microstep: 5796.13 | bwd_inner_microstep: 5663.38 | bwd_allreduce_microstep: 132.71 | step_microstep: 18.15
[2025-04-26 13:39:55,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.06 | bwd: 5796.14 | bwd_inner: 5663.38 | bwd_allreduce: 132.73 | step: 18.15

 10%|▉         | 3947/41250 [9:31:26<90:03:23,  8.69s/it]
                                                         
{'loss': 0.2177, 'grad_norm': 1.2351436614990234, 'learning_rate': 3.954928784335194e-05, 'epoch': 0.96}

 10%|▉         | 3947/41250 [9:31:26<90:03:23,  8.69s/it][2025-04-26 13:40:03,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 13:40:03,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.06 | bwd_microstep: 5761.23 | bwd_inner_microstep: 5685.13 | bwd_allreduce_microstep: 76.06 | step_microstep: 17.71
[2025-04-26 13:40:03,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.06 | bwd: 5761.24 | bwd_inner: 5685.13 | bwd_allreduce: 76.08 | step: 17.72

 10%|▉         | 3948/41250 [9:31:35<90:05:46,  8.70s/it]
                                                         
{'loss': 0.0799, 'grad_norm': 2.040501832962036, 'learning_rate': 3.954895628687212e-05, 'epoch': 0.96}

 10%|▉         | 3948/41250 [9:31:35<90:05:46,  8.70s/it][2025-04-26 13:40:12,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:40:12,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2841.22 | bwd_microstep: 5713.01 | bwd_inner_microstep: 5670.40 | bwd_allreduce_microstep: 42.57 | step_microstep: 18.65
[2025-04-26 13:40:12,579] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2841.22 | bwd: 5713.03 | bwd_inner: 5670.40 | bwd_allreduce: 42.59 | step: 18.65

 10%|▉         | 3949/41250 [9:31:43<89:53:51,  8.68s/it]
                                                         
{'loss': 0.2007, 'grad_norm': 2.6134653091430664, 'learning_rate': 3.9548624609876836e-05, 'epoch': 0.96}

 10%|▉         | 3949/41250 [9:31:43<89:53:51,  8.68s/it][2025-04-26 13:40:21,281] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:40:21,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.87 | bwd_microstep: 5766.68 | bwd_inner_microstep: 5695.94 | bwd_allreduce_microstep: 70.70 | step_microstep: 17.92
[2025-04-26 13:40:21,282] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.87 | bwd: 5766.69 | bwd_inner: 5695.94 | bwd_allreduce: 70.71 | step: 17.93

 10%|▉         | 3950/41250 [9:31:52<89:58:36,  8.68s/it]
                                                         
{'loss': 0.5383, 'grad_norm': 2.4419093132019043, 'learning_rate': 3.9548292812368124e-05, 'epoch': 0.96}

 10%|▉         | 3950/41250 [9:31:52<89:58:36,  8.68s/it][2025-04-26 13:40:29,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.03 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:40:29,991] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.85 | bwd_microstep: 5775.66 | bwd_inner_microstep: 5691.16 | bwd_allreduce_microstep: 84.45 | step_microstep: 17.68
[2025-04-26 13:40:29,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.85 | bwd: 5775.67 | bwd_inner: 5691.16 | bwd_allreduce: 84.47 | step: 17.69

 10%|▉         | 3951/41250 [9:32:01<90:03:13,  8.69s/it]
                                                         
{'loss': 0.6655, 'grad_norm': 2.8831591606140137, 'learning_rate': 3.9547960894348023e-05, 'epoch': 0.96}

 10%|▉         | 3951/41250 [9:32:01<90:03:13,  8.69s/it][2025-04-26 13:40:38,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.06 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:40:38,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.97 | bwd_microstep: 5789.09 | bwd_inner_microstep: 5664.58 | bwd_allreduce_microstep: 124.34 | step_microstep: 17.59
[2025-04-26 13:40:38,696] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.97 | bwd: 5789.10 | bwd_inner: 5664.58 | bwd_allreduce: 124.49 | step: 17.60

 10%|▉         | 3952/41250 [9:32:09<90:05:34,  8.70s/it]
                                                         
{'loss': 0.2594, 'grad_norm': 2.877962350845337, 'learning_rate': 3.9547628855818594e-05, 'epoch': 0.96}

 10%|▉         | 3952/41250 [9:32:09<90:05:34,  8.70s/it][2025-04-26 13:40:47,306] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:40:47,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2840.58 | bwd_microstep: 5699.98 | bwd_inner_microstep: 5664.18 | bwd_allreduce_microstep: 35.76 | step_microstep: 17.86
[2025-04-26 13:40:47,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2840.58 | bwd: 5699.99 | bwd_inner: 5664.17 | bwd_allreduce: 35.78 | step: 17.86

 10%|▉         | 3953/41250 [9:32:18<89:49:30,  8.67s/it]
                                                         
{'loss': 0.1109, 'grad_norm': 1.730372667312622, 'learning_rate': 3.954729669678188e-05, 'epoch': 0.96}

 10%|▉         | 3953/41250 [9:32:18<89:49:30,  8.67s/it][2025-04-26 13:40:55,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:40:55,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.01 | bwd_microstep: 5704.59 | bwd_inner_microstep: 5668.35 | bwd_allreduce_microstep: 36.20 | step_microstep: 18.00
[2025-04-26 13:40:55,936] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.02 | bwd: 5704.61 | bwd_inner: 5668.35 | bwd_allreduce: 36.21 | step: 18.00

 10%|▉         | 3954/41250 [9:32:27<89:41:51,  8.66s/it]
                                                         
{'loss': 0.2914, 'grad_norm': 2.215815544128418, 'learning_rate': 3.9546964417239926e-05, 'epoch': 0.96}

 10%|▉         | 3954/41250 [9:32:27<89:41:51,  8.66s/it][2025-04-26 13:41:04,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.23 | optimizer_step: 1.18
[2025-04-26 13:41:04,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.97 | bwd_microstep: 5786.70 | bwd_inner_microstep: 5663.12 | bwd_allreduce_microstep: 123.55 | step_microstep: 20.18
[2025-04-26 13:41:04,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.97 | bwd: 5786.71 | bwd_inner: 5663.12 | bwd_allreduce: 123.56 | step: 20.18

 10%|▉         | 3955/41250 [9:32:35<89:51:59,  8.67s/it]
                                                         
{'loss': 0.2586, 'grad_norm': 1.65129816532135, 'learning_rate': 3.954663201719478e-05, 'epoch': 0.96}

 10%|▉         | 3955/41250 [9:32:35<89:51:59,  8.67s/it][2025-04-26 13:41:13,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.02
[2025-04-26 13:41:13,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.63 | bwd_microstep: 5796.63 | bwd_inner_microstep: 5714.42 | bwd_allreduce_microstep: 82.16 | step_microstep: 18.08
[2025-04-26 13:41:13,392] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.64 | bwd: 5796.64 | bwd_inner: 5714.42 | bwd_allreduce: 82.18 | step: 18.09

 10%|▉         | 3956/41250 [9:32:44<90:04:31,  8.70s/it]
                                                         
{'loss': 0.2105, 'grad_norm': 2.7151200771331787, 'learning_rate': 3.9546299496648494e-05, 'epoch': 0.96}

 10%|▉         | 3956/41250 [9:32:44<90:04:31,  8.70s/it][2025-04-26 13:41:22,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:41:22,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.15 | bwd_microstep: 5773.89 | bwd_inner_microstep: 5661.42 | bwd_allreduce_microstep: 112.42 | step_microstep: 18.04
[2025-04-26 13:41:22,090] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.15 | bwd: 5773.90 | bwd_inner: 5661.42 | bwd_allreduce: 112.44 | step: 18.04

 10%|▉         | 3957/41250 [9:32:53<90:04:51,  8.70s/it]
                                                         
{'loss': 0.144, 'grad_norm': 3.382842779159546, 'learning_rate': 3.9545966855603116e-05, 'epoch': 0.96}

 10%|▉         | 3957/41250 [9:32:53<90:04:51,  8.70s/it][2025-04-26 13:41:30,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:41:30,875] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.27 | bwd_microstep: 5850.16 | bwd_inner_microstep: 5709.50 | bwd_allreduce_microstep: 140.61 | step_microstep: 18.93
[2025-04-26 13:41:30,876] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.27 | bwd: 5850.17 | bwd_inner: 5709.50 | bwd_allreduce: 140.63 | step: 18.93

 10%|▉         | 3958/41250 [9:33:02<90:21:40,  8.72s/it]
                                                         
{'loss': 0.1391, 'grad_norm': 1.186943769454956, 'learning_rate': 3.9545634094060694e-05, 'epoch': 0.96}

 10%|▉         | 3958/41250 [9:33:02<90:21:40,  8.72s/it][2025-04-26 13:41:39,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:41:39,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.85 | bwd_microstep: 5721.35 | bwd_inner_microstep: 5708.58 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.47
[2025-04-26 13:41:39,528] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.85 | bwd: 5721.36 | bwd_inner: 5708.58 | bwd_allreduce: 12.74 | step: 18.47

 10%|▉         | 3959/41250 [9:33:10<90:08:14,  8.70s/it]
                                                         
{'loss': 0.3402, 'grad_norm': 2.17181134223938, 'learning_rate': 3.954530121202328e-05, 'epoch': 0.96}

 10%|▉         | 3959/41250 [9:33:10<90:08:14,  8.70s/it][2025-04-26 13:41:48,170] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 13:41:48,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.68 | bwd_microstep: 5729.36 | bwd_inner_microstep: 5657.50 | bwd_allreduce_microstep: 71.81 | step_microstep: 18.71
[2025-04-26 13:41:48,171] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.68 | bwd: 5729.38 | bwd_inner: 5657.50 | bwd_allreduce: 71.83 | step: 18.71

 10%|▉         | 3960/41250 [9:33:19<89:57:05,  8.68s/it]
                                                         
{'loss': 0.1275, 'grad_norm': 0.9961316585540771, 'learning_rate': 3.954496820949293e-05, 'epoch': 0.96}

 10%|▉         | 3960/41250 [9:33:19<89:57:05,  8.68s/it][2025-04-26 13:41:56,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.29 | optimizer_step: 0.92
[2025-04-26 13:41:56,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.31 | bwd_microstep: 5881.45 | bwd_inner_microstep: 5700.20 | bwd_allreduce_microstep: 181.18 | step_microstep: 19.20
[2025-04-26 13:41:56,984] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.31 | bwd: 5881.47 | bwd_inner: 5700.20 | bwd_allreduce: 181.22 | step: 19.20

 10%|▉         | 3961/41250 [9:33:28<90:22:01,  8.72s/it]
                                                         
{'loss': 0.206, 'grad_norm': 1.8458534479141235, 'learning_rate': 3.95446350864717e-05, 'epoch': 0.96}

 10%|▉         | 3961/41250 [9:33:28<90:22:01,  8.72s/it][2025-04-26 13:42:05,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.38 | optimizer_gradients: 1.16 | optimizer_step: 1.05
[2025-04-26 13:42:05,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.76 | bwd_microstep: 5769.75 | bwd_inner_microstep: 5705.83 | bwd_allreduce_microstep: 63.86 | step_microstep: 20.23
[2025-04-26 13:42:05,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.76 | bwd: 5769.77 | bwd_inner: 5705.83 | bwd_allreduce: 63.89 | step: 20.23

 10%|▉         | 3962/41250 [9:33:36<90:17:07,  8.72s/it]
                                                         
{'loss': 0.0602, 'grad_norm': 0.5792210102081299, 'learning_rate': 3.9544301842961636e-05, 'epoch': 0.96}

 10%|▉         | 3962/41250 [9:33:36<90:17:07,  8.72s/it][2025-04-26 13:42:14,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.03 | optimizer_step: 0.94
[2025-04-26 13:42:14,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.38 | bwd_microstep: 5716.69 | bwd_inner_microstep: 5703.68 | bwd_allreduce_microstep: 12.96 | step_microstep: 18.76
[2025-04-26 13:42:14,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.38 | bwd: 5716.70 | bwd_inner: 5703.68 | bwd_allreduce: 12.98 | step: 18.77

 10%|▉         | 3963/41250 [9:33:45<90:04:54,  8.70s/it]
                                                         
{'loss': 0.2052, 'grad_norm': 1.5984808206558228, 'learning_rate': 3.9543968478964794e-05, 'epoch': 0.96}

 10%|▉         | 3963/41250 [9:33:45<90:04:54,  8.70s/it][2025-04-26 13:42:22,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:42:22,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.80 | bwd_microstep: 5700.55 | bwd_inner_microstep: 5665.04 | bwd_allreduce_microstep: 35.47 | step_microstep: 18.70
[2025-04-26 13:42:22,959] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.80 | bwd: 5700.57 | bwd_inner: 5665.04 | bwd_allreduce: 35.49 | step: 18.71

 10%|▉         | 3964/41250 [9:33:54<89:50:15,  8.67s/it]
                                                         
{'loss': 0.0991, 'grad_norm': 0.7611533403396606, 'learning_rate': 3.9543634994483235e-05, 'epoch': 0.96}

 10%|▉         | 3964/41250 [9:33:54<89:50:15,  8.67s/it][2025-04-26 13:42:31,627] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:42:31,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.05 | bwd_microstep: 5758.10 | bwd_inner_microstep: 5664.11 | bwd_allreduce_microstep: 93.94 | step_microstep: 18.68
[2025-04-26 13:42:31,628] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.05 | bwd: 5758.11 | bwd_inner: 5664.11 | bwd_allreduce: 93.96 | step: 18.68

 10%|▉         | 3965/41250 [9:34:02<89:49:37,  8.67s/it]
                                                         
{'loss': 0.1951, 'grad_norm': 1.7444639205932617, 'learning_rate': 3.9543301389519005e-05, 'epoch': 0.96}

 10%|▉         | 3965/41250 [9:34:02<89:49:37,  8.67s/it][2025-04-26 13:42:40,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.01 | optimizer_step: 1.16
[2025-04-26 13:42:40,276] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.56 | bwd_microstep: 5710.09 | bwd_inner_microstep: 5697.44 | bwd_allreduce_microstep: 12.60 | step_microstep: 19.13
[2025-04-26 13:42:40,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.56 | bwd: 5710.10 | bwd_inner: 5697.44 | bwd_allreduce: 12.61 | step: 19.13

 10%|▉         | 3966/41250 [9:34:11<89:44:24,  8.66s/it]
                                                         
{'loss': 0.0535, 'grad_norm': 0.462112158536911, 'learning_rate': 3.954296766407417e-05, 'epoch': 0.96}

 10%|▉         | 3966/41250 [9:34:11<89:44:24,  8.66s/it][2025-04-26 13:42:48,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:42:48,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.20 | bwd_microstep: 5714.38 | bwd_inner_microstep: 5655.19 | bwd_allreduce_microstep: 59.14 | step_microstep: 19.10
[2025-04-26 13:42:48,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.20 | bwd: 5714.40 | bwd_inner: 5655.19 | bwd_allreduce: 59.16 | step: 19.11

 10%|▉         | 3967/41250 [9:34:20<89:35:32,  8.65s/it]
                                                         
{'loss': 0.2772, 'grad_norm': 1.727891206741333, 'learning_rate': 3.9542633818150784e-05, 'epoch': 0.96}

 10%|▉         | 3967/41250 [9:34:20<89:35:32,  8.65s/it][2025-04-26 13:42:57,556] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.01 | optimizer_step: 1.04
[2025-04-26 13:42:57,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2819.08 | bwd_microstep: 5759.92 | bwd_inner_microstep: 5642.74 | bwd_allreduce_microstep: 117.13 | step_microstep: 18.75
[2025-04-26 13:42:57,557] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2819.08 | bwd: 5759.94 | bwd_inner: 5642.74 | bwd_allreduce: 117.15 | step: 18.75

 10%|▉         | 3968/41250 [9:34:28<89:37:24,  8.65s/it]
                                                         
{'loss': 0.2379, 'grad_norm': 1.834281086921692, 'learning_rate': 3.95422998517509e-05, 'epoch': 0.96}

 10%|▉         | 3968/41250 [9:34:28<89:37:24,  8.65s/it][2025-04-26 13:43:06,160] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:43:06,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.87 | bwd_microstep: 5697.05 | bwd_inner_microstep: 5651.53 | bwd_allreduce_microstep: 45.48 | step_microstep: 18.45
[2025-04-26 13:43:06,161] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.87 | bwd: 5697.07 | bwd_inner: 5651.53 | bwd_allreduce: 45.50 | step: 18.45

 10%|▉         | 3969/41250 [9:34:37<89:27:54,  8.64s/it]
                                                         
{'loss': 0.1437, 'grad_norm': 1.019405484199524, 'learning_rate': 3.9541965764876585e-05, 'epoch': 0.96}

 10%|▉         | 3969/41250 [9:34:37<89:27:54,  8.64s/it][2025-04-26 13:43:14,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.02 | optimizer_step: 0.99
[2025-04-26 13:43:14,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.15 | bwd_microstep: 5833.33 | bwd_inner_microstep: 5705.02 | bwd_allreduce_microstep: 128.25 | step_microstep: 18.80
[2025-04-26 13:43:14,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.15 | bwd: 5833.34 | bwd_inner: 5705.02 | bwd_allreduce: 128.28 | step: 18.82

 10%|▉         | 3970/41250 [9:34:46<89:51:42,  8.68s/it]
                                                         
{'loss': 0.2124, 'grad_norm': 1.524857997894287, 'learning_rate': 3.954163155752989e-05, 'epoch': 0.96}

 10%|▉         | 3970/41250 [9:34:46<89:51:42,  8.68s/it][2025-04-26 13:43:23,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.24 | optimizer_step: 0.99
[2025-04-26 13:43:23,591] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5760.76 | bwd_inner_microstep: 5652.97 | bwd_allreduce_microstep: 107.73 | step_microstep: 19.39
[2025-04-26 13:43:23,592] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5760.77 | bwd_inner: 5652.97 | bwd_allreduce: 107.75 | step: 19.39

 10%|▉         | 3971/41250 [9:34:54<89:48:58,  8.67s/it]
                                                         
{'loss': 0.1508, 'grad_norm': 1.696575403213501, 'learning_rate': 3.954129722971289e-05, 'epoch': 0.96}

 10%|▉         | 3971/41250 [9:34:54<89:48:58,  8.67s/it][2025-04-26 13:43:32,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.56 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:43:32,219] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.70 | bwd_microstep: 5701.47 | bwd_inner_microstep: 5688.79 | bwd_allreduce_microstep: 12.63 | step_microstep: 18.62
[2025-04-26 13:43:32,220] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.70 | bwd: 5701.49 | bwd_inner: 5688.79 | bwd_allreduce: 12.65 | step: 18.62

 10%|▉         | 3972/41250 [9:35:03<89:40:18,  8.66s/it]
                                                         
{'loss': 0.1975, 'grad_norm': 1.6460227966308594, 'learning_rate': 3.9540962781427626e-05, 'epoch': 0.96}

 10%|▉         | 3972/41250 [9:35:03<89:40:18,  8.66s/it][2025-04-26 13:43:40,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:43:40,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.04 | bwd_microstep: 5744.83 | bwd_inner_microstep: 5653.84 | bwd_allreduce_microstep: 90.94 | step_microstep: 18.65
[2025-04-26 13:43:40,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.04 | bwd: 5744.84 | bwd_inner: 5653.84 | bwd_allreduce: 90.96 | step: 18.65

 10%|▉         | 3973/41250 [9:35:12<89:40:28,  8.66s/it]
                                                         
{'loss': 0.1769, 'grad_norm': 2.264585256576538, 'learning_rate': 3.954062821267617e-05, 'epoch': 0.96}

 10%|▉         | 3973/41250 [9:35:12<89:40:28,  8.66s/it][2025-04-26 13:43:49,636] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:43:49,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.39 | bwd_microstep: 5782.80 | bwd_inner_microstep: 5769.99 | bwd_allreduce_microstep: 12.76 | step_microstep: 18.57
[2025-04-26 13:43:49,637] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2890.39 | bwd: 5782.81 | bwd_inner: 5769.99 | bwd_allreduce: 12.78 | step: 18.57

 10%|▉         | 3974/41250 [9:35:20<89:58:08,  8.69s/it]
                                                         
{'loss': 0.1386, 'grad_norm': 1.4417364597320557, 'learning_rate': 3.954029352346059e-05, 'epoch': 0.96}

 10%|▉         | 3974/41250 [9:35:20<89:58:08,  8.69s/it][2025-04-26 13:43:58,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.91
[2025-04-26 13:43:58,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.42 | bwd_microstep: 5696.63 | bwd_inner_microstep: 5683.76 | bwd_allreduce_microstep: 12.82 | step_microstep: 18.83
[2025-04-26 13:43:58,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.42 | bwd: 5696.64 | bwd_inner: 5683.76 | bwd_allreduce: 12.84 | step: 18.83

 10%|▉         | 3975/41250 [9:35:29<89:45:20,  8.67s/it]
                                                         
{'loss': 0.1556, 'grad_norm': 1.3021634817123413, 'learning_rate': 3.953995871378294e-05, 'epoch': 0.96}

 10%|▉         | 3975/41250 [9:35:29<89:45:20,  8.67s/it][2025-04-26 13:44:06,854] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:44:06,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.26 | bwd_microstep: 5684.73 | bwd_inner_microstep: 5651.22 | bwd_allreduce_microstep: 33.47 | step_microstep: 18.82
[2025-04-26 13:44:06,855] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.26 | bwd: 5684.75 | bwd_inner: 5651.22 | bwd_allreduce: 33.49 | step: 18.82

 10%|▉         | 3976/41250 [9:35:38<89:32:03,  8.65s/it]
                                                         
{'loss': 0.2545, 'grad_norm': 1.5828659534454346, 'learning_rate': 3.9539623783645284e-05, 'epoch': 0.96}

 10%|▉         | 3976/41250 [9:35:38<89:32:03,  8.65s/it][2025-04-26 13:44:15,512] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.01 | optimizer_step: 0.97
[2025-04-26 13:44:15,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2855.56 | bwd_microstep: 5715.62 | bwd_inner_microstep: 5702.97 | bwd_allreduce_microstep: 12.61 | step_microstep: 19.01
[2025-04-26 13:44:15,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2855.56 | bwd: 5715.63 | bwd_inner: 5702.97 | bwd_allreduce: 12.62 | step: 19.01

 10%|▉         | 3977/41250 [9:35:46<89:33:42,  8.65s/it]
                                                         
{'loss': 0.1051, 'grad_norm': 0.6880919933319092, 'learning_rate': 3.95392887330497e-05, 'epoch': 0.96}

 10%|▉         | 3977/41250 [9:35:46<89:33:42,  8.65s/it][2025-04-26 13:44:24,194] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:44:24,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.09 | bwd_microstep: 5741.96 | bwd_inner_microstep: 5693.80 | bwd_allreduce_microstep: 48.12 | step_microstep: 18.37
[2025-04-26 13:44:24,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.09 | bwd: 5741.98 | bwd_inner: 5693.80 | bwd_allreduce: 48.13 | step: 18.38

 10%|▉         | 3978/41250 [9:35:55<89:39:24,  8.66s/it]
                                                         
{'loss': 0.212, 'grad_norm': 1.9505375623703003, 'learning_rate': 3.9538953561998236e-05, 'epoch': 0.96}

 10%|▉         | 3978/41250 [9:35:55<89:39:24,  8.66s/it][2025-04-26 13:44:32,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:44:32,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2818.98 | bwd_microstep: 5694.36 | bwd_inner_microstep: 5642.67 | bwd_allreduce_microstep: 51.65 | step_microstep: 18.46
[2025-04-26 13:44:32,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2818.98 | bwd: 5694.38 | bwd_inner: 5642.67 | bwd_allreduce: 51.67 | step: 18.46

 10%|▉         | 3979/41250 [9:36:03<89:27:21,  8.64s/it]
                                                         
{'loss': 0.1008, 'grad_norm': 0.6537830829620361, 'learning_rate': 3.953861827049297e-05, 'epoch': 0.96}

 10%|▉         | 3979/41250 [9:36:03<89:27:21,  8.64s/it][2025-04-26 13:44:41,416] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.13 | optimizer_gradients: 0.96 | optimizer_step: 0.90
[2025-04-26 13:44:41,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.13 | bwd_microstep: 5708.89 | bwd_inner_microstep: 5695.91 | bwd_allreduce_microstep: 12.93 | step_microstep: 17.89
[2025-04-26 13:44:41,417] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.14 | bwd: 5708.90 | bwd_inner: 5695.91 | bwd_allreduce: 12.95 | step: 17.89

 10%|▉         | 3980/41250 [9:36:12<89:24:30,  8.64s/it]
                                                         
{'loss': 0.2265, 'grad_norm': 1.0462090969085693, 'learning_rate': 3.953828285853597e-05, 'epoch': 0.96}

 10%|▉         | 3980/41250 [9:36:12<89:24:30,  8.64s/it][2025-04-26 13:44:50,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-26 13:44:50,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.81 | bwd_microstep: 5704.32 | bwd_inner_microstep: 5647.24 | bwd_allreduce_microstep: 57.03 | step_microstep: 18.26
[2025-04-26 13:44:50,032] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.81 | bwd: 5704.33 | bwd_inner: 5647.24 | bwd_allreduce: 57.05 | step: 18.26

 10%|▉         | 3981/41250 [9:36:21<89:20:27,  8.63s/it]
                                                         
{'loss': 0.0902, 'grad_norm': 0.649511456489563, 'learning_rate': 3.953794732612929e-05, 'epoch': 0.97}

 10%|▉         | 3981/41250 [9:36:21<89:20:27,  8.63s/it][2025-04-26 13:44:58,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:44:58,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.45 | bwd_microstep: 5720.01 | bwd_inner_microstep: 5707.31 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.51
[2025-04-26 13:44:58,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.45 | bwd: 5720.03 | bwd_inner: 5707.31 | bwd_allreduce: 12.68 | step: 18.52

 10%|▉         | 3982/41250 [9:36:29<89:24:34,  8.64s/it]
                                                         
{'loss': 0.0834, 'grad_norm': 0.6215733289718628, 'learning_rate': 3.9537611673275017e-05, 'epoch': 0.97}

 10%|▉         | 3982/41250 [9:36:29<89:24:34,  8.64s/it][2025-04-26 13:45:07,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 13:45:07,508] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.11 | bwd_microstep: 5855.52 | bwd_inner_microstep: 5780.04 | bwd_allreduce_microstep: 75.42 | step_microstep: 18.77
[2025-04-26 13:45:07,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.11 | bwd: 5855.53 | bwd_inner: 5780.05 | bwd_allreduce: 75.45 | step: 18.77

 10%|▉         | 3983/41250 [9:36:38<89:59:21,  8.69s/it]
                                                         
{'loss': 0.1942, 'grad_norm': 1.6185386180877686, 'learning_rate': 3.95372758999752e-05, 'epoch': 0.97}

 10%|▉         | 3983/41250 [9:36:38<89:59:21,  8.69s/it][2025-04-26 13:45:16,166] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:45:16,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.64 | bwd_microstep: 5720.02 | bwd_inner_microstep: 5707.39 | bwd_allreduce_microstep: 12.59 | step_microstep: 18.34
[2025-04-26 13:45:16,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.64 | bwd: 5720.04 | bwd_inner: 5707.39 | bwd_allreduce: 12.60 | step: 18.34

 10%|▉         | 3984/41250 [9:36:47<89:52:44,  8.68s/it]
                                                         
{'loss': 0.0531, 'grad_norm': 0.5171935558319092, 'learning_rate': 3.953694000623193e-05, 'epoch': 0.97}

 10%|▉         | 3984/41250 [9:36:47<89:52:44,  8.68s/it][2025-04-26 13:45:24,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:45:24,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.45 | bwd_microstep: 5744.59 | bwd_inner_microstep: 5684.26 | bwd_allreduce_microstep: 60.29 | step_microstep: 18.35
[2025-04-26 13:45:24,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.45 | bwd: 5744.60 | bwd_inner: 5684.26 | bwd_allreduce: 60.31 | step: 18.35

 10%|▉         | 3985/41250 [9:36:56<89:50:54,  8.68s/it]
                                                         
{'loss': 0.1176, 'grad_norm': 0.8410338163375854, 'learning_rate': 3.953660399204726e-05, 'epoch': 0.97}

 10%|▉         | 3985/41250 [9:36:56<89:50:54,  8.68s/it][2025-04-26 13:45:33,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:45:33,605] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2888.75 | bwd_microstep: 5795.17 | bwd_inner_microstep: 5782.57 | bwd_allreduce_microstep: 12.55 | step_microstep: 18.33
[2025-04-26 13:45:33,606] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2888.75 | bwd: 5795.18 | bwd_inner: 5782.57 | bwd_allreduce: 12.57 | step: 18.34

 10%|▉         | 3986/41250 [9:37:04<90:06:38,  8.71s/it]
                                                         
{'loss': 0.1306, 'grad_norm': 0.8943285346031189, 'learning_rate': 3.9536267857423276e-05, 'epoch': 0.97}

 10%|▉         | 3986/41250 [9:37:04<90:06:38,  8.71s/it][2025-04-26 13:45:42,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.00 | optimizer_step: 1.02
[2025-04-26 13:45:42,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.84 | bwd_microstep: 5865.44 | bwd_inner_microstep: 5697.07 | bwd_allreduce_microstep: 168.32 | step_microstep: 18.88
[2025-04-26 13:45:42,404] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.84 | bwd: 5865.45 | bwd_inner: 5697.07 | bwd_allreduce: 168.34 | step: 18.88

 10%|▉         | 3987/41250 [9:37:13<90:23:47,  8.73s/it]
                                                         
{'loss': 0.2225, 'grad_norm': 1.8623102903366089, 'learning_rate': 3.953593160236204e-05, 'epoch': 0.97}

 10%|▉         | 3987/41250 [9:37:13<90:23:47,  8.73s/it][2025-04-26 13:45:51,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:45:51,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.25 | bwd_microstep: 5710.03 | bwd_inner_microstep: 5697.19 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.42
[2025-04-26 13:45:51,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.25 | bwd: 5710.04 | bwd_inner: 5697.19 | bwd_allreduce: 12.82 | step: 18.43

 10%|▉         | 3988/41250 [9:37:22<90:08:23,  8.71s/it]
                                                         
{'loss': 0.0661, 'grad_norm': 0.81551194190979, 'learning_rate': 3.953559522686563e-05, 'epoch': 0.97}

 10%|▉         | 3988/41250 [9:37:22<90:08:23,  8.71s/it][2025-04-26 13:45:59,729] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.43 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:45:59,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.83 | bwd_microstep: 5767.03 | bwd_inner_microstep: 5658.17 | bwd_allreduce_microstep: 108.81 | step_microstep: 18.57
[2025-04-26 13:45:59,730] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.83 | bwd: 5767.04 | bwd_inner: 5658.17 | bwd_allreduce: 108.83 | step: 18.57

 10%|▉         | 3989/41250 [9:37:30<90:01:49,  8.70s/it]
                                                         
{'loss': 0.0629, 'grad_norm': 1.0839014053344727, 'learning_rate': 3.953525873093611e-05, 'epoch': 0.97}

 10%|▉         | 3989/41250 [9:37:30<90:01:49,  8.70s/it][2025-04-26 13:46:08,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:46:08,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.14 | bwd_microstep: 5770.25 | bwd_inner_microstep: 5699.92 | bwd_allreduce_microstep: 70.28 | step_microstep: 18.75
[2025-04-26 13:46:08,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.14 | bwd: 5770.26 | bwd_inner: 5699.92 | bwd_allreduce: 70.30 | step: 18.75

 10%|▉         | 3990/41250 [9:37:39<90:02:16,  8.70s/it]
                                                         
{'loss': 0.0769, 'grad_norm': 0.9333319664001465, 'learning_rate': 3.953492211457557e-05, 'epoch': 0.97}

 10%|▉         | 3990/41250 [9:37:39<90:02:16,  8.70s/it][2025-04-26 13:46:17,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:46:17,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.58 | bwd_microstep: 5724.32 | bwd_inner_microstep: 5711.60 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.54
[2025-04-26 13:46:17,098] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.58 | bwd: 5724.34 | bwd_inner: 5711.60 | bwd_allreduce: 12.70 | step: 18.54

 10%|▉         | 3991/41250 [9:37:48<89:56:17,  8.69s/it]
                                                         
{'loss': 0.1127, 'grad_norm': 1.5643404722213745, 'learning_rate': 3.953458537778608e-05, 'epoch': 0.97}

 10%|▉         | 3991/41250 [9:37:48<89:56:17,  8.69s/it][2025-04-26 13:46:25,783] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.97
[2025-04-26 13:46:25,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2865.38 | bwd_microstep: 5738.48 | bwd_inner_microstep: 5725.76 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.75
[2025-04-26 13:46:25,784] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2865.38 | bwd: 5738.50 | bwd_inner: 5725.76 | bwd_allreduce: 12.70 | step: 18.76

 10%|▉         | 3992/41250 [9:37:56<89:55:15,  8.69s/it]
                                                         
{'loss': 0.1612, 'grad_norm': 1.7132909297943115, 'learning_rate': 3.953424852056971e-05, 'epoch': 0.97}

 10%|▉         | 3992/41250 [9:37:56<89:55:15,  8.69s/it][2025-04-26 13:46:34,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:46:34,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 3129.44 | bwd_microstep: 5676.67 | bwd_inner_microstep: 5663.92 | bwd_allreduce_microstep: 12.71 | step_microstep: 18.40
[2025-04-26 13:46:34,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 3129.44 | bwd: 5676.69 | bwd_inner: 5663.92 | bwd_allreduce: 12.73 | step: 18.41

 10%|▉         | 3993/41250 [9:38:05<90:32:55,  8.75s/it]
                                                         
{'loss': 0.1136, 'grad_norm': 1.9441242218017578, 'learning_rate': 3.953391154292854e-05, 'epoch': 0.97}

 10%|▉         | 3993/41250 [9:38:05<90:32:55,  8.75s/it][2025-04-26 13:46:43,374] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 0.97 | optimizer_step: 1.04
[2025-04-26 13:46:43,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.66 | bwd_microstep: 5800.95 | bwd_inner_microstep: 5656.09 | bwd_allreduce_microstep: 144.82 | step_microstep: 18.11
[2025-04-26 13:46:43,375] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.66 | bwd: 5800.96 | bwd_inner: 5656.09 | bwd_allreduce: 144.84 | step: 18.12

 10%|▉         | 3994/41250 [9:38:14<90:23:28,  8.73s/it]
                                                         
{'loss': 0.0371, 'grad_norm': 0.41506820917129517, 'learning_rate': 3.9533574444864644e-05, 'epoch': 0.97}

 10%|▉         | 3994/41250 [9:38:14<90:23:28,  8.73s/it][2025-04-26 13:46:51,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:46:51,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.82 | bwd_microstep: 5684.52 | bwd_inner_microstep: 5658.76 | bwd_allreduce_microstep: 25.72 | step_microstep: 17.80
[2025-04-26 13:46:51,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.82 | bwd: 5684.54 | bwd_inner: 5658.76 | bwd_allreduce: 25.74 | step: 17.81

 10%|▉         | 3995/41250 [9:38:23<89:59:02,  8.70s/it]
                                                         
{'loss': 0.3795, 'grad_norm': 1.6951014995574951, 'learning_rate': 3.95332372263801e-05, 'epoch': 0.97}

 10%|▉         | 3995/41250 [9:38:23<89:59:02,  8.70s/it][2025-04-26 13:47:00,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 13:47:00,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.89 | bwd_microstep: 5995.56 | bwd_inner_microstep: 5720.01 | bwd_allreduce_microstep: 275.51 | step_microstep: 18.22
[2025-04-26 13:47:00,913] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.89 | bwd: 5995.57 | bwd_inner: 5720.00 | bwd_allreduce: 275.53 | step: 18.22

 10%|▉         | 3996/41250 [9:38:32<90:43:24,  8.77s/it]
                                                         
{'loss': 0.0377, 'grad_norm': 0.4442313611507416, 'learning_rate': 3.9532899887477e-05, 'epoch': 0.97}

 10%|▉         | 3996/41250 [9:38:32<90:43:24,  8.77s/it][2025-04-26 13:47:09,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 13:47:09,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.67 | bwd_microstep: 5704.85 | bwd_inner_microstep: 5692.17 | bwd_allreduce_microstep: 12.64 | step_microstep: 18.18
[2025-04-26 13:47:09,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.67 | bwd: 5704.86 | bwd_inner: 5692.17 | bwd_allreduce: 12.66 | step: 18.19

 10%|▉         | 3997/41250 [9:38:40<90:18:21,  8.73s/it]
                                                         
{'loss': 0.3224, 'grad_norm': 2.257087230682373, 'learning_rate': 3.9532562428157405e-05, 'epoch': 0.97}

 10%|▉         | 3997/41250 [9:38:40<90:18:21,  8.73s/it][2025-04-26 13:47:18,178] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.16 | optimizer_gradients: 0.94 | optimizer_step: 0.90
[2025-04-26 13:47:18,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.90 | bwd_microstep: 5713.50 | bwd_inner_microstep: 5700.84 | bwd_allreduce_microstep: 12.62 | step_microstep: 17.73
[2025-04-26 13:47:18,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.90 | bwd: 5713.51 | bwd_inner: 5700.84 | bwd_allreduce: 12.63 | step: 17.73

 10%|▉         | 3998/41250 [9:38:49<90:00:31,  8.70s/it]
                                                         
{'loss': 0.0231, 'grad_norm': 0.22854970395565033, 'learning_rate': 3.953222484842342e-05, 'epoch': 0.97}

 10%|▉         | 3998/41250 [9:38:49<90:00:31,  8.70s/it][2025-04-26 13:47:26,840] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 13:47:26,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.13 | bwd_microstep: 5718.38 | bwd_inner_microstep: 5705.76 | bwd_allreduce_microstep: 12.58 | step_microstep: 18.17
[2025-04-26 13:47:26,841] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.13 | bwd: 5718.40 | bwd_inner: 5705.76 | bwd_allreduce: 12.60 | step: 18.17

 10%|▉         | 3999/41250 [9:38:58<89:53:37,  8.69s/it]
                                                         
{'loss': 0.1297, 'grad_norm': 1.002255916595459, 'learning_rate': 3.95318871482771e-05, 'epoch': 0.97}

 10%|▉         | 3999/41250 [9:38:58<89:53:37,  8.69s/it][2025-04-26 13:47:35,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:47:35,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2881.18 | bwd_microstep: 5775.18 | bwd_inner_microstep: 5762.26 | bwd_allreduce_microstep: 12.87 | step_microstep: 18.60
[2025-04-26 13:47:35,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2881.18 | bwd: 5775.19 | bwd_inner: 5762.26 | bwd_allreduce: 12.89 | step: 18.61

 10%|▉         | 4000/41250 [9:39:06<90:03:40,  8.70s/it]
                                                         
{'loss': 0.0377, 'grad_norm': 0.8456867933273315, 'learning_rate': 3.953154932772053e-05, 'epoch': 0.97}

 10%|▉         | 4000/41250 [9:39:06<90:03:40,  8.70s/it][2025-04-26 13:47:44,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:47:44,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.04 | bwd_microstep: 5686.96 | bwd_inner_microstep: 5651.61 | bwd_allreduce_microstep: 35.31 | step_microstep: 18.49
[2025-04-26 13:47:44,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.04 | bwd: 5686.98 | bwd_inner: 5651.61 | bwd_allreduce: 35.33 | step: 18.49

 10%|▉         | 4001/41250 [9:39:15<89:44:41,  8.67s/it]
                                                         
{'loss': 0.2035, 'grad_norm': 1.8615033626556396, 'learning_rate': 3.9531211386755815e-05, 'epoch': 0.97}

 10%|▉         | 4001/41250 [9:39:15<89:44:41,  8.67s/it][2025-04-26 13:47:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:47:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.60 | bwd_microstep: 5692.32 | bwd_inner_microstep: 5658.82 | bwd_allreduce_microstep: 33.46 | step_microstep: 18.00
[2025-04-26 13:47:52,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.60 | bwd: 5692.34 | bwd_inner: 5658.82 | bwd_allreduce: 33.48 | step: 18.00

 10%|▉         | 4002/41250 [9:39:23<89:30:07,  8.65s/it]
                                                         
{'loss': 0.2278, 'grad_norm': 0.9157285094261169, 'learning_rate': 3.9530873325385014e-05, 'epoch': 0.97}

 10%|▉         | 4002/41250 [9:39:23<89:30:07,  8.65s/it][2025-04-26 13:48:01,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:48:01,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.05 | bwd_microstep: 5740.78 | bwd_inner_microstep: 5699.09 | bwd_allreduce_microstep: 41.64 | step_microstep: 18.46
[2025-04-26 13:48:01,448] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.05 | bwd: 5740.79 | bwd_inner: 5699.09 | bwd_allreduce: 41.66 | step: 18.47

 10%|▉         | 4003/41250 [9:39:32<89:33:31,  8.66s/it]
                                                         
{'loss': 0.061, 'grad_norm': 0.7892138361930847, 'learning_rate': 3.953053514361022e-05, 'epoch': 0.97}

 10%|▉         | 4003/41250 [9:39:32<89:33:31,  8.66s/it][2025-04-26 13:48:10,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:48:10,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.17 | bwd_microstep: 5707.64 | bwd_inner_microstep: 5650.28 | bwd_allreduce_microstep: 57.32 | step_microstep: 18.40
[2025-04-26 13:48:10,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.17 | bwd: 5707.65 | bwd_inner: 5650.28 | bwd_allreduce: 57.33 | step: 18.40

 10%|▉         | 4004/41250 [9:39:41<89:25:49,  8.64s/it]
                                                         
{'loss': 0.0767, 'grad_norm': 0.9125062823295593, 'learning_rate': 3.953019684143352e-05, 'epoch': 0.97}

 10%|▉         | 4004/41250 [9:39:41<89:25:49,  8.64s/it][2025-04-26 13:48:18,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.12
[2025-04-26 13:48:18,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.68 | bwd_microstep: 5759.45 | bwd_inner_microstep: 5647.11 | bwd_allreduce_microstep: 112.29 | step_microstep: 18.77
[2025-04-26 13:48:18,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2837.68 | bwd: 5759.46 | bwd_inner: 5647.11 | bwd_allreduce: 112.31 | step: 18.78

 10%|▉         | 4005/41250 [9:39:49<89:32:12,  8.65s/it]
                                                         
{'loss': 0.3106, 'grad_norm': 2.753019332885742, 'learning_rate': 3.952985841885701e-05, 'epoch': 0.97}

 10%|▉         | 4005/41250 [9:39:49<89:32:12,  8.65s/it][2025-04-26 13:48:27,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 13:48:27,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.85 | bwd_microstep: 5702.68 | bwd_inner_microstep: 5655.63 | bwd_allreduce_microstep: 47.00 | step_microstep: 18.63
[2025-04-26 13:48:27,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.85 | bwd: 5702.69 | bwd_inner: 5655.63 | bwd_allreduce: 47.02 | step: 18.63

 10%|▉         | 4006/41250 [9:39:58<89:24:24,  8.64s/it]
                                                         
{'loss': 0.075, 'grad_norm': 1.3841177225112915, 'learning_rate': 3.9529519875882754e-05, 'epoch': 0.97}

 10%|▉         | 4006/41250 [9:39:58<89:24:24,  8.64s/it][2025-04-26 13:48:35,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:48:35,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.37 | bwd_microstep: 5691.53 | bwd_inner_microstep: 5652.25 | bwd_allreduce_microstep: 39.24 | step_microstep: 18.32
[2025-04-26 13:48:35,957] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.37 | bwd: 5691.55 | bwd_inner: 5652.25 | bwd_allreduce: 39.26 | step: 18.32

 10%|▉         | 4007/41250 [9:40:07<89:16:06,  8.63s/it]
                                                         
{'loss': 0.1198, 'grad_norm': 1.2649286985397339, 'learning_rate': 3.9529181212512844e-05, 'epoch': 0.97}

 10%|▉         | 4007/41250 [9:40:07<89:16:06,  8.63s/it][2025-04-26 13:48:44,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:48:44,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.11 | bwd_microstep: 5692.57 | bwd_inner_microstep: 5657.57 | bwd_allreduce_microstep: 34.96 | step_microstep: 18.44
[2025-04-26 13:48:44,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.11 | bwd: 5692.59 | bwd_inner: 5657.57 | bwd_allreduce: 34.98 | step: 18.44

 10%|▉         | 4008/41250 [9:40:15<89:11:21,  8.62s/it]
                                                         
{'loss': 0.0648, 'grad_norm': 1.1178088188171387, 'learning_rate': 3.952884242874938e-05, 'epoch': 0.97}

 10%|▉         | 4008/41250 [9:40:15<89:11:21,  8.62s/it][2025-04-26 13:48:53,249] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:48:53,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.54 | bwd_microstep: 5741.29 | bwd_inner_microstep: 5688.52 | bwd_allreduce_microstep: 52.73 | step_microstep: 18.31
[2025-04-26 13:48:53,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.54 | bwd: 5741.31 | bwd_inner: 5688.52 | bwd_allreduce: 52.75 | step: 18.31

 10%|▉         | 4009/41250 [9:40:24<89:24:00,  8.64s/it]
                                                         
{'loss': 0.165, 'grad_norm': 1.8735748529434204, 'learning_rate': 3.952850352459444e-05, 'epoch': 0.97}

 10%|▉         | 4009/41250 [9:40:24<89:24:00,  8.64s/it][2025-04-26 13:49:01,934] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 1.00
[2025-04-26 13:49:01,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.88 | bwd_microstep: 5751.16 | bwd_inner_microstep: 5701.63 | bwd_allreduce_microstep: 49.49 | step_microstep: 18.84
[2025-04-26 13:49:01,935] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.88 | bwd: 5751.18 | bwd_inner: 5701.63 | bwd_allreduce: 49.51 | step: 18.84

 10%|▉         | 4010/41250 [9:40:33<89:31:35,  8.65s/it]
                                                         
{'loss': 0.1901, 'grad_norm': 1.5795470476150513, 'learning_rate': 3.9528164500050116e-05, 'epoch': 0.97}

 10%|▉         | 4010/41250 [9:40:33<89:31:35,  8.65s/it][2025-04-26 13:49:10,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.07 | optimizer_step: 1.10
[2025-04-26 13:49:10,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2861.81 | bwd_microstep: 5720.85 | bwd_inner_microstep: 5707.90 | bwd_allreduce_microstep: 12.90 | step_microstep: 19.42
[2025-04-26 13:49:10,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2861.81 | bwd: 5720.86 | bwd_inner: 5707.90 | bwd_allreduce: 12.92 | step: 19.42

 10%|▉         | 4011/41250 [9:40:41<89:34:24,  8.66s/it]
                                                         
{'loss': 0.0744, 'grad_norm': 1.3142893314361572, 'learning_rate': 3.95278253551185e-05, 'epoch': 0.97}

 10%|▉         | 4011/41250 [9:40:41<89:34:24,  8.66s/it][2025-04-26 13:49:19,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:49:19,301] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.25 | bwd_microstep: 5785.28 | bwd_inner_microstep: 5656.01 | bwd_allreduce_microstep: 129.22 | step_microstep: 18.63
[2025-04-26 13:49:19,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.25 | bwd: 5785.29 | bwd_inner: 5656.01 | bwd_allreduce: 129.24 | step: 18.63

 10%|▉         | 4012/41250 [9:40:50<89:41:05,  8.67s/it]
                                                         
{'loss': 0.0737, 'grad_norm': 0.7223451733589172, 'learning_rate': 3.952748608980169e-05, 'epoch': 0.97}

 10%|▉         | 4012/41250 [9:40:50<89:41:05,  8.67s/it][2025-04-26 13:49:27,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.02 | optimizer_step: 1.01
[2025-04-26 13:49:27,979] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.03 | bwd_microstep: 5738.45 | bwd_inner_microstep: 5688.56 | bwd_allreduce_microstep: 49.84 | step_microstep: 19.24
[2025-04-26 13:49:27,980] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.03 | bwd: 5738.47 | bwd_inner: 5688.56 | bwd_allreduce: 49.86 | step: 19.24

 10%|▉         | 4013/41250 [9:40:59<89:43:03,  8.67s/it]
                                                         
{'loss': 0.0338, 'grad_norm': 0.5705196261405945, 'learning_rate': 3.952714670410176e-05, 'epoch': 0.97}

 10%|▉         | 4013/41250 [9:40:59<89:43:03,  8.67s/it][2025-04-26 13:49:36,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.07 | optimizer_step: 1.28
[2025-04-26 13:49:36,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.31 | bwd_microstep: 5766.23 | bwd_inner_microstep: 5648.94 | bwd_allreduce_microstep: 117.24 | step_microstep: 19.81
[2025-04-26 13:49:36,670] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2835.31 | bwd: 5766.25 | bwd_inner: 5648.94 | bwd_allreduce: 117.26 | step: 19.81

 10%|▉         | 4014/41250 [9:41:07<89:46:19,  8.68s/it]
                                                         
{'loss': 0.448, 'grad_norm': 3.5147523880004883, 'learning_rate': 3.952680719802081e-05, 'epoch': 0.97}

 10%|▉         | 4014/41250 [9:41:07<89:46:19,  8.68s/it][2025-04-26 13:49:45,322] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:49:45,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.12 | bwd_microstep: 5718.40 | bwd_inner_microstep: 5705.55 | bwd_allreduce_microstep: 12.80 | step_microstep: 18.80
[2025-04-26 13:49:45,323] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.12 | bwd: 5718.41 | bwd_inner: 5705.55 | bwd_allreduce: 12.82 | step: 18.80

 10%|▉         | 4015/41250 [9:41:16<89:40:35,  8.67s/it]
                                                         
{'loss': 0.3972, 'grad_norm': 3.45249605178833, 'learning_rate': 3.952646757156093e-05, 'epoch': 0.97}

 10%|▉         | 4015/41250 [9:41:16<89:40:35,  8.67s/it][2025-04-26 13:49:53,967] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.98 | optimizer_step: 0.98
[2025-04-26 13:49:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.07 | bwd_microstep: 5724.17 | bwd_inner_microstep: 5711.44 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.64
[2025-04-26 13:49:53,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.07 | bwd: 5724.18 | bwd_inner: 5711.44 | bwd_allreduce: 12.70 | step: 18.64

 10%|▉         | 4016/41250 [9:41:25<89:35:35,  8.66s/it]
                                                         
{'loss': 0.2659, 'grad_norm': 2.812976360321045, 'learning_rate': 3.952612782472423e-05, 'epoch': 0.97}

 10%|▉         | 4016/41250 [9:41:25<89:35:35,  8.66s/it][2025-04-26 13:50:02,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.33 | optimizer_gradients: 0.97 | optimizer_step: 0.92
[2025-04-26 13:50:02,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.39 | bwd_microstep: 5683.87 | bwd_inner_microstep: 5647.52 | bwd_allreduce_microstep: 36.31 | step_microstep: 18.47
[2025-04-26 13:50:02,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.39 | bwd: 5683.88 | bwd_inner: 5647.52 | bwd_allreduce: 36.33 | step: 18.48

 10%|▉         | 4017/41250 [9:41:33<89:22:16,  8.64s/it]
                                                         
{'loss': 0.1368, 'grad_norm': 0.9622504711151123, 'learning_rate': 3.952578795751279e-05, 'epoch': 0.97}

 10%|▉         | 4017/41250 [9:41:33<89:22:16,  8.64s/it][2025-04-26 13:50:11,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.96 | optimizer_step: 0.98
[2025-04-26 13:50:11,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.29 | bwd_microstep: 5779.70 | bwd_inner_microstep: 5654.99 | bwd_allreduce_microstep: 124.67 | step_microstep: 18.40
[2025-04-26 13:50:11,247] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.29 | bwd: 5779.72 | bwd_inner: 5654.99 | bwd_allreduce: 124.69 | step: 18.41

 10%|▉         | 4018/41250 [9:41:42<89:30:47,  8.66s/it]
                                                         
{'loss': 0.2764, 'grad_norm': 1.674586534500122, 'learning_rate': 3.95254479699287e-05, 'epoch': 0.97}

 10%|▉         | 4018/41250 [9:41:42<89:30:47,  8.66s/it][2025-04-26 13:50:19,904] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 13:50:19,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.92 | bwd_microstep: 5724.20 | bwd_inner_microstep: 5680.58 | bwd_allreduce_microstep: 43.58 | step_microstep: 18.73
[2025-04-26 13:50:19,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.92 | bwd: 5724.22 | bwd_inner: 5680.58 | bwd_allreduce: 43.59 | step: 18.74

 10%|▉         | 4019/41250 [9:41:51<89:31:34,  8.66s/it]
                                                         
{'loss': 0.4768, 'grad_norm': 2.611588478088379, 'learning_rate': 3.952510786197407e-05, 'epoch': 0.97}

 10%|▉         | 4019/41250 [9:41:51<89:31:34,  8.66s/it][2025-04-26 13:50:28,513] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 13:50:28,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.24 | bwd_microstep: 5699.22 | bwd_inner_microstep: 5645.12 | bwd_allreduce_microstep: 54.05 | step_microstep: 19.13
[2025-04-26 13:50:28,514] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.24 | bwd: 5699.23 | bwd_inner: 5645.12 | bwd_allreduce: 54.07 | step: 19.13

 10%|▉         | 4020/41250 [9:41:59<89:22:17,  8.64s/it]
                                                         
{'loss': 0.1364, 'grad_norm': 1.4670435190200806, 'learning_rate': 3.952476763365099e-05, 'epoch': 0.97}

 10%|▉         | 4020/41250 [9:41:59<89:22:17,  8.64s/it][2025-04-26 13:50:37,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 0.99 | optimizer_step: 0.91
[2025-04-26 13:50:37,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.34 | bwd_microstep: 5752.20 | bwd_inner_microstep: 5688.86 | bwd_allreduce_microstep: 63.29 | step_microstep: 18.80
[2025-04-26 13:50:37,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.34 | bwd: 5752.21 | bwd_inner: 5688.86 | bwd_allreduce: 63.31 | step: 18.80

 10%|▉         | 4021/41250 [9:42:08<89:28:12,  8.65s/it]
                                                         
{'loss': 0.1679, 'grad_norm': 1.5866522789001465, 'learning_rate': 3.952442728496156e-05, 'epoch': 0.97}

 10%|▉         | 4021/41250 [9:42:08<89:28:12,  8.65s/it][2025-04-26 13:50:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:50:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.70 | bwd_microstep: 5756.15 | bwd_inner_microstep: 5687.44 | bwd_allreduce_microstep: 68.66 | step_microstep: 18.88
[2025-04-26 13:50:45,873] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.70 | bwd: 5756.16 | bwd_inner: 5687.44 | bwd_allreduce: 68.68 | step: 18.88

 10%|▉         | 4022/41250 [9:42:17<89:34:10,  8.66s/it]
                                                         
{'loss': 0.2137, 'grad_norm': 3.0139620304107666, 'learning_rate': 3.952408681590787e-05, 'epoch': 0.98}

 10%|▉         | 4022/41250 [9:42:17<89:34:10,  8.66s/it][2025-04-26 13:50:54,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.23 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:50:54,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.42 | bwd_microstep: 5740.56 | bwd_inner_microstep: 5690.89 | bwd_allreduce_microstep: 49.62 | step_microstep: 18.59
[2025-04-26 13:50:54,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.42 | bwd: 5740.57 | bwd_inner: 5690.89 | bwd_allreduce: 49.64 | step: 18.59

 10%|▉         | 4023/41250 [9:42:25<89:36:25,  8.67s/it]
                                                         
{'loss': 0.281, 'grad_norm': 1.2320427894592285, 'learning_rate': 3.952374622649204e-05, 'epoch': 0.98}

 10%|▉         | 4023/41250 [9:42:25<89:36:25,  8.67s/it][2025-04-26 13:51:03,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:51:03,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.98 | bwd_microstep: 5766.14 | bwd_inner_microstep: 5659.93 | bwd_allreduce_microstep: 106.17 | step_microstep: 18.78
[2025-04-26 13:51:03,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.98 | bwd: 5766.15 | bwd_inner: 5659.93 | bwd_allreduce: 106.18 | step: 18.79

 10%|▉         | 4024/41250 [9:42:34<89:39:14,  8.67s/it]
                                                         
{'loss': 0.0956, 'grad_norm': 1.876631259918213, 'learning_rate': 3.9523405516716134e-05, 'epoch': 0.98}

 10%|▉         | 4024/41250 [9:42:34<89:39:14,  8.67s/it][2025-04-26 13:51:11,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.08 | optimizer_gradients: 1.03 | optimizer_step: 1.17
[2025-04-26 13:51:11,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.06 | bwd_microstep: 5791.54 | bwd_inner_microstep: 5647.65 | bwd_allreduce_microstep: 143.84 | step_microstep: 19.20
[2025-04-26 13:51:11,924] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.06 | bwd: 5791.55 | bwd_inner: 5647.64 | bwd_allreduce: 143.86 | step: 19.20

 10%|▉         | 4025/41250 [9:42:43<89:43:40,  8.68s/it]
                                                         
{'loss': 0.2109, 'grad_norm': 1.394536018371582, 'learning_rate': 3.9523064686582284e-05, 'epoch': 0.98}

 10%|▉         | 4025/41250 [9:42:43<89:43:40,  8.68s/it][2025-04-26 13:51:20,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 13:51:20,608] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.05 | bwd_microstep: 5758.00 | bwd_inner_microstep: 5701.88 | bwd_allreduce_microstep: 56.08 | step_microstep: 18.88
[2025-04-26 13:51:20,609] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.05 | bwd: 5758.01 | bwd_inner: 5701.88 | bwd_allreduce: 56.09 | step: 18.88

 10%|▉         | 4026/41250 [9:42:51<89:44:55,  8.68s/it]
                                                         
{'loss': 0.2069, 'grad_norm': 2.419267416000366, 'learning_rate': 3.952272373609258e-05, 'epoch': 0.98}

 10%|▉         | 4026/41250 [9:42:51<89:44:55,  8.68s/it][2025-04-26 13:51:29,317] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:51:29,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.04 | bwd_microstep: 5801.30 | bwd_inner_microstep: 5639.86 | bwd_allreduce_microstep: 161.39 | step_microstep: 18.74
[2025-04-26 13:51:29,318] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.04 | bwd: 5801.32 | bwd_inner: 5639.86 | bwd_allreduce: 161.41 | step: 18.74

 10%|▉         | 4027/41250 [9:43:00<89:50:17,  8.69s/it]
                                                         
{'loss': 0.0371, 'grad_norm': 0.546177327632904, 'learning_rate': 3.952238266524912e-05, 'epoch': 0.98}

 10%|▉         | 4027/41250 [9:43:00<89:50:17,  8.69s/it][2025-04-26 13:51:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.89
[2025-04-26 13:51:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.79 | bwd_microstep: 5900.72 | bwd_inner_microstep: 5660.26 | bwd_allreduce_microstep: 240.42 | step_microstep: 18.53
[2025-04-26 13:51:38,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.79 | bwd: 5900.74 | bwd_inner: 5660.26 | bwd_allreduce: 240.43 | step: 18.53

 10%|▉         | 4028/41250 [9:43:09<90:13:53,  8.73s/it]
                                                         
{'loss': 0.2283, 'grad_norm': 1.2556965351104736, 'learning_rate': 3.9522041474054014e-05, 'epoch': 0.98}

 10%|▉         | 4028/41250 [9:43:09<90:13:53,  8.73s/it][2025-04-26 13:51:46,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.05
[2025-04-26 13:51:46,770] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.10 | bwd_microstep: 5720.90 | bwd_inner_microstep: 5660.09 | bwd_allreduce_microstep: 60.76 | step_microstep: 19.53
[2025-04-26 13:51:46,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.10 | bwd: 5720.91 | bwd_inner: 5660.09 | bwd_allreduce: 60.78 | step: 19.53

 10%|▉         | 4029/41250 [9:43:17<89:57:02,  8.70s/it]
                                                         
{'loss': 0.2746, 'grad_norm': 2.970210552215576, 'learning_rate': 3.952170016250936e-05, 'epoch': 0.98}

 10%|▉         | 4029/41250 [9:43:17<89:57:02,  8.70s/it][2025-04-26 13:51:55,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:51:55,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.89 | bwd_microstep: 5758.99 | bwd_inner_microstep: 5652.32 | bwd_allreduce_microstep: 106.63 | step_microstep: 19.04
[2025-04-26 13:51:55,442] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.89 | bwd: 5759.01 | bwd_inner: 5652.32 | bwd_allreduce: 106.65 | step: 19.04

 10%|▉         | 4030/41250 [9:43:26<89:51:22,  8.69s/it]
                                                         
{'loss': 0.1318, 'grad_norm': 1.0692298412322998, 'learning_rate': 3.952135873061726e-05, 'epoch': 0.98}

 10%|▉         | 4030/41250 [9:43:26<89:51:22,  8.69s/it][2025-04-26 13:52:04,112] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:52:04,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.44 | bwd_microstep: 5762.55 | bwd_inner_microstep: 5654.16 | bwd_allreduce_microstep: 108.35 | step_microstep: 18.63
[2025-04-26 13:52:04,113] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.44 | bwd: 5762.56 | bwd_inner: 5654.16 | bwd_allreduce: 108.37 | step: 18.63

 10%|▉         | 4031/41250 [9:43:35<89:47:23,  8.68s/it]
                                                         
{'loss': 0.1929, 'grad_norm': 1.7482672929763794, 'learning_rate': 3.9521017178379836e-05, 'epoch': 0.98}

 10%|▉         | 4031/41250 [9:43:35<89:47:23,  8.68s/it][2025-04-26 13:52:12,834] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.03 | optimizer_step: 1.02
[2025-04-26 13:52:12,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.65 | bwd_microstep: 5783.35 | bwd_inner_microstep: 5715.18 | bwd_allreduce_microstep: 68.12 | step_microstep: 18.84
[2025-04-26 13:52:12,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.65 | bwd: 5783.37 | bwd_inner: 5715.18 | bwd_allreduce: 68.14 | step: 18.84

 10%|▉         | 4032/41250 [9:43:44<89:54:13,  8.70s/it]
                                                         
{'loss': 0.246, 'grad_norm': 1.9742727279663086, 'learning_rate': 3.952067550579917e-05, 'epoch': 0.98}

 10%|▉         | 4032/41250 [9:43:44<89:54:13,  8.70s/it][2025-04-26 13:52:21,476] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:52:21,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.93 | bwd_microstep: 5709.63 | bwd_inner_microstep: 5696.81 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.64
[2025-04-26 13:52:21,477] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.93 | bwd: 5709.65 | bwd_inner: 5696.81 | bwd_allreduce: 12.79 | step: 18.65

 10%|▉         | 4033/41250 [9:43:52<89:43:56,  8.68s/it]
                                                         
{'loss': 0.1297, 'grad_norm': 1.449610710144043, 'learning_rate': 3.952033371287739e-05, 'epoch': 0.98}

 10%|▉         | 4033/41250 [9:43:52<89:43:56,  8.68s/it][2025-04-26 13:52:30,151] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 1.02
[2025-04-26 13:52:30,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.45 | bwd_microstep: 5777.30 | bwd_inner_microstep: 5649.21 | bwd_allreduce_microstep: 128.04 | step_microstep: 18.96
[2025-04-26 13:52:30,152] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.45 | bwd: 5777.32 | bwd_inner: 5649.22 | bwd_allreduce: 128.06 | step: 18.97

 10%|▉         | 4034/41250 [9:44:01<89:42:57,  8.68s/it]
                                                         
{'loss': 0.0781, 'grad_norm': 0.8394472002983093, 'learning_rate': 3.951999179961658e-05, 'epoch': 0.98}

 10%|▉         | 4034/41250 [9:44:01<89:42:57,  8.68s/it][2025-04-26 13:52:38,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.19 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 13:52:38,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.61 | bwd_microstep: 5729.47 | bwd_inner_microstep: 5708.10 | bwd_allreduce_microstep: 21.32 | step_microstep: 18.74
[2025-04-26 13:52:38,823] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.61 | bwd: 5729.48 | bwd_inner: 5708.10 | bwd_allreduce: 21.34 | step: 18.74

 10%|▉         | 4035/41250 [9:44:10<89:41:27,  8.68s/it]
                                                         
{'loss': 0.072, 'grad_norm': 0.9813722372055054, 'learning_rate': 3.951964976601886e-05, 'epoch': 0.98}

 10%|▉         | 4035/41250 [9:44:10<89:41:27,  8.68s/it][2025-04-26 13:52:47,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 13:52:47,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.91 | bwd_microstep: 5934.08 | bwd_inner_microstep: 5664.82 | bwd_allreduce_microstep: 269.20 | step_microstep: 18.98
[2025-04-26 13:52:47,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.91 | bwd: 5934.09 | bwd_inner: 5664.82 | bwd_allreduce: 269.23 | step: 18.98

 10%|▉         | 4036/41250 [9:44:18<90:14:41,  8.73s/it]
                                                         
{'loss': 0.3801, 'grad_norm': 2.043745994567871, 'learning_rate': 3.951930761208635e-05, 'epoch': 0.98}

 10%|▉         | 4036/41250 [9:44:18<90:14:41,  8.73s/it][2025-04-26 13:52:56,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 13:52:56,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.78 | bwd_microstep: 5749.72 | bwd_inner_microstep: 5695.18 | bwd_allreduce_microstep: 54.49 | step_microstep: 18.73
[2025-04-26 13:52:56,368] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.78 | bwd: 5749.73 | bwd_inner: 5695.18 | bwd_allreduce: 54.51 | step: 18.73

 10%|▉         | 4037/41250 [9:44:27<90:06:57,  8.72s/it]
                                                         
{'loss': 0.0833, 'grad_norm': 0.9537338018417358, 'learning_rate': 3.951896533782114e-05, 'epoch': 0.98}

 10%|▉         | 4037/41250 [9:44:27<90:06:57,  8.72s/it][2025-04-26 13:53:05,043] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.13 | optimizer_step: 1.18
[2025-04-26 13:53:05,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.10 | bwd_microstep: 5760.38 | bwd_inner_microstep: 5655.99 | bwd_allreduce_microstep: 104.33 | step_microstep: 19.72
[2025-04-26 13:53:05,045] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.10 | bwd: 5760.40 | bwd_inner: 5655.99 | bwd_allreduce: 104.35 | step: 19.72

 10%|▉         | 4038/41250 [9:44:36<89:59:32,  8.71s/it]
                                                         
{'loss': 0.1788, 'grad_norm': 0.8347290754318237, 'learning_rate': 3.951862294322535e-05, 'epoch': 0.98}

 10%|▉         | 4038/41250 [9:44:36<89:59:32,  8.71s/it][2025-04-26 13:53:13,686] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 13:53:13,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.56 | bwd_microstep: 5713.63 | bwd_inner_microstep: 5700.69 | bwd_allreduce_microstep: 12.89 | step_microstep: 19.07
[2025-04-26 13:53:13,687] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.57 | bwd: 5713.64 | bwd_inner: 5700.69 | bwd_allreduce: 12.91 | step: 19.07

 10%|▉         | 4039/41250 [9:44:44<89:47:06,  8.69s/it]
                                                         
{'loss': 0.1131, 'grad_norm': 1.7544869184494019, 'learning_rate': 3.951828042830109e-05, 'epoch': 0.98}

 10%|▉         | 4039/41250 [9:44:44<89:47:06,  8.69s/it][2025-04-26 13:53:22,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:53:22,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.85 | bwd_microstep: 5937.47 | bwd_inner_microstep: 5664.16 | bwd_allreduce_microstep: 273.27 | step_microstep: 18.78
[2025-04-26 13:53:22,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.85 | bwd: 5937.48 | bwd_inner: 5664.16 | bwd_allreduce: 273.29 | step: 18.79

 10%|▉         | 4040/41250 [9:44:53<90:18:01,  8.74s/it]
                                                         
{'loss': 0.0616, 'grad_norm': 0.5996928811073303, 'learning_rate': 3.951793779305046e-05, 'epoch': 0.98}

 10%|▉         | 4040/41250 [9:44:53<90:18:01,  8.74s/it][2025-04-26 13:53:31,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.00 | optimizer_step: 1.08
[2025-04-26 13:53:31,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.80 | bwd_microstep: 5736.50 | bwd_inner_microstep: 5699.81 | bwd_allreduce_microstep: 36.64 | step_microstep: 18.64
[2025-04-26 13:53:31,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.80 | bwd: 5736.51 | bwd_inner: 5699.81 | bwd_allreduce: 36.66 | step: 18.65

 10%|▉         | 4041/41250 [9:45:02<90:05:02,  8.72s/it]
                                                         
{'loss': 0.0698, 'grad_norm': 1.0288363695144653, 'learning_rate': 3.9517595037475586e-05, 'epoch': 0.98}

 10%|▉         | 4041/41250 [9:45:02<90:05:02,  8.72s/it][2025-04-26 13:53:39,992] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:53:39,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.35 | bwd_microstep: 5844.47 | bwd_inner_microstep: 5708.34 | bwd_allreduce_microstep: 136.09 | step_microstep: 18.92
[2025-04-26 13:53:39,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.35 | bwd: 5844.49 | bwd_inner: 5708.33 | bwd_allreduce: 136.10 | step: 18.92

 10%|▉         | 4042/41250 [9:45:11<90:18:15,  8.74s/it]
                                                         
{'loss': 0.1281, 'grad_norm': 1.1252471208572388, 'learning_rate': 3.9517252161578586e-05, 'epoch': 0.98}

 10%|▉         | 4042/41250 [9:45:11<90:18:15,  8.74s/it][2025-04-26 13:53:48,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.24 | optimizer_step: 1.02
[2025-04-26 13:53:48,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.62 | bwd_microstep: 5769.55 | bwd_inner_microstep: 5652.90 | bwd_allreduce_microstep: 116.59 | step_microstep: 19.76
[2025-04-26 13:53:48,669] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.62 | bwd: 5769.57 | bwd_inner: 5652.90 | bwd_allreduce: 116.62 | step: 19.77

 10%|▉         | 4043/41250 [9:45:19<90:06:21,  8.72s/it]
                                                         
{'loss': 0.2732, 'grad_norm': 3.429990768432617, 'learning_rate': 3.951690916536156e-05, 'epoch': 0.98}

 10%|▉         | 4043/41250 [9:45:19<90:06:21,  8.72s/it][2025-04-26 13:53:57,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:53:57,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.35 | bwd_microstep: 5740.31 | bwd_inner_microstep: 5688.03 | bwd_allreduce_microstep: 52.23 | step_microstep: 18.67
[2025-04-26 13:53:57,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.35 | bwd: 5740.32 | bwd_inner: 5688.03 | bwd_allreduce: 52.25 | step: 18.68

 10%|▉         | 4044/41250 [9:45:28<89:57:18,  8.70s/it]
                                                         
{'loss': 0.1154, 'grad_norm': 2.031339406967163, 'learning_rate': 3.9516566048826625e-05, 'epoch': 0.98}

 10%|▉         | 4044/41250 [9:45:28<89:57:18,  8.70s/it][2025-04-26 13:54:05,955] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 13:54:05,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.20 | bwd_microstep: 5709.31 | bwd_inner_microstep: 5648.03 | bwd_allreduce_microstep: 61.23 | step_microstep: 18.47
[2025-04-26 13:54:05,956] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.20 | bwd: 5709.32 | bwd_inner: 5648.03 | bwd_allreduce: 61.25 | step: 18.47

 10%|▉         | 4045/41250 [9:45:37<89:40:45,  8.68s/it]
                                                         
{'loss': 0.2102, 'grad_norm': 1.5438196659088135, 'learning_rate': 3.951622281197591e-05, 'epoch': 0.98}

 10%|▉         | 4045/41250 [9:45:37<89:40:45,  8.68s/it][2025-04-26 13:54:14,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.55 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:54:14,710] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2885.95 | bwd_microstep: 5785.14 | bwd_inner_microstep: 5772.42 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.78
[2025-04-26 13:54:14,711] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2885.95 | bwd: 5785.15 | bwd_inner: 5772.42 | bwd_allreduce: 12.70 | step: 18.78

 10%|▉         | 4046/41250 [9:45:45<89:55:05,  8.70s/it]
                                                         
{'loss': 0.1803, 'grad_norm': 6.941871643066406, 'learning_rate': 3.95158794548115e-05, 'epoch': 0.98}

 10%|▉         | 4046/41250 [9:45:45<89:55:05,  8.70s/it][2025-04-26 13:54:23,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:54:23,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.56 | bwd_microstep: 5785.16 | bwd_inner_microstep: 5660.83 | bwd_allreduce_microstep: 124.28 | step_microstep: 18.71
[2025-04-26 13:54:23,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.56 | bwd: 5785.17 | bwd_inner: 5660.83 | bwd_allreduce: 124.30 | step: 18.72

 10%|▉         | 4047/41250 [9:45:54<89:54:39,  8.70s/it]
                                                         
{'loss': 0.1283, 'grad_norm': 1.2022249698638916, 'learning_rate': 3.951553597733555e-05, 'epoch': 0.98}

 10%|▉         | 4047/41250 [9:45:54<89:54:39,  8.70s/it][2025-04-26 13:54:32,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.11 | optimizer_gradients: 0.97 | optimizer_step: 0.96
[2025-04-26 13:54:32,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.50 | bwd_microstep: 5741.90 | bwd_inner_microstep: 5705.23 | bwd_allreduce_microstep: 36.62 | step_microstep: 18.15
[2025-04-26 13:54:32,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.50 | bwd: 5741.91 | bwd_inner: 5705.23 | bwd_allreduce: 36.64 | step: 18.16

 10%|▉         | 4048/41250 [9:46:03<89:48:15,  8.69s/it]
                                                         
{'loss': 0.2986, 'grad_norm': 2.179609775543213, 'learning_rate': 3.951519237955015e-05, 'epoch': 0.98}

 10%|▉         | 4048/41250 [9:46:03<89:48:15,  8.69s/it][2025-04-26 13:54:40,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:54:40,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.68 | bwd_microstep: 5758.69 | bwd_inner_microstep: 5647.88 | bwd_allreduce_microstep: 110.77 | step_microstep: 18.59
[2025-04-26 13:54:40,744] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.68 | bwd: 5758.70 | bwd_inner: 5647.88 | bwd_allreduce: 110.78 | step: 18.59

 10%|▉         | 4049/41250 [9:46:11<89:43:46,  8.68s/it]
                                                         
{'loss': 0.0927, 'grad_norm': 1.907683253288269, 'learning_rate': 3.9514848661457435e-05, 'epoch': 0.98}

 10%|▉         | 4049/41250 [9:46:11<89:43:46,  8.68s/it][2025-04-26 13:54:49,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.96 | optimizer_step: 1.07
[2025-04-26 13:54:49,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2844.13 | bwd_microstep: 5735.50 | bwd_inner_microstep: 5706.36 | bwd_allreduce_microstep: 29.10 | step_microstep: 18.20
[2025-04-26 13:54:49,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2844.13 | bwd: 5735.51 | bwd_inner: 5706.36 | bwd_allreduce: 29.12 | step: 18.20

 10%|▉         | 4050/41250 [9:46:20<89:40:11,  8.68s/it]
                                                         
{'loss': 0.1673, 'grad_norm': 1.4608404636383057, 'learning_rate': 3.9514504823059516e-05, 'epoch': 0.98}

 10%|▉         | 4050/41250 [9:46:20<89:40:11,  8.68s/it][2025-04-26 13:54:58,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.39 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:54:58,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.32 | bwd_microstep: 5757.78 | bwd_inner_microstep: 5646.89 | bwd_allreduce_microstep: 110.85 | step_microstep: 18.42
[2025-04-26 13:54:58,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.32 | bwd: 5757.80 | bwd_inner: 5646.89 | bwd_allreduce: 110.87 | step: 18.43

 10%|▉         | 4051/41250 [9:46:29<89:39:13,  8.68s/it]
                                                         
{'loss': 0.2795, 'grad_norm': 1.9848030805587769, 'learning_rate': 3.951416086435851e-05, 'epoch': 0.98}

 10%|▉         | 4051/41250 [9:46:29<89:39:13,  8.68s/it][2025-04-26 13:55:06,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.04
[2025-04-26 13:55:06,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.63 | bwd_microstep: 5700.04 | bwd_inner_microstep: 5649.44 | bwd_allreduce_microstep: 50.55 | step_microstep: 18.53
[2025-04-26 13:55:06,683] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.63 | bwd: 5700.05 | bwd_inner: 5649.44 | bwd_allreduce: 50.57 | step: 18.53

 10%|▉         | 4052/41250 [9:46:37<89:24:50,  8.65s/it]
                                                         
{'loss': 0.2065, 'grad_norm': 1.149560809135437, 'learning_rate': 3.951381678535653e-05, 'epoch': 0.98}

 10%|▉         | 4052/41250 [9:46:37<89:24:50,  8.65s/it][2025-04-26 13:55:15,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:55:15,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.97 | bwd_microstep: 5688.54 | bwd_inner_microstep: 5640.83 | bwd_allreduce_microstep: 47.67 | step_microstep: 18.46
[2025-04-26 13:55:15,285] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.97 | bwd: 5688.56 | bwd_inner: 5640.83 | bwd_allreduce: 47.69 | step: 18.46

 10%|▉         | 4053/41250 [9:46:46<89:15:22,  8.64s/it]
                                                         
{'loss': 0.201, 'grad_norm': 1.6861310005187988, 'learning_rate': 3.9513472586055724e-05, 'epoch': 0.98}

 10%|▉         | 4053/41250 [9:46:46<89:15:22,  8.64s/it][2025-04-26 13:55:23,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.58 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:55:23,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2836.84 | bwd_microstep: 5684.71 | bwd_inner_microstep: 5650.68 | bwd_allreduce_microstep: 33.99 | step_microstep: 18.42
[2025-04-26 13:55:23,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2836.84 | bwd: 5684.73 | bwd_inner: 5650.68 | bwd_allreduce: 34.01 | step: 18.43

 10%|▉         | 4054/41250 [9:46:55<89:08:51,  8.63s/it]
                                                         
{'loss': 0.2084, 'grad_norm': 1.8499786853790283, 'learning_rate': 3.951312826645819e-05, 'epoch': 0.98}

 10%|▉         | 4054/41250 [9:46:55<89:08:51,  8.63s/it][2025-04-26 13:55:32,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 0.97
[2025-04-26 13:55:32,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.66 | bwd_microstep: 5768.10 | bwd_inner_microstep: 5650.66 | bwd_allreduce_microstep: 117.39 | step_microstep: 18.69
[2025-04-26 13:55:32,570] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.66 | bwd: 5768.12 | bwd_inner: 5650.66 | bwd_allreduce: 117.41 | step: 18.69

 10%|▉         | 4055/41250 [9:47:03<89:19:09,  8.64s/it]
                                                         
{'loss': 0.1399, 'grad_norm': 1.3644459247589111, 'learning_rate': 3.9512783826566055e-05, 'epoch': 0.98}

 10%|▉         | 4055/41250 [9:47:03<89:19:09,  8.64s/it][2025-04-26 13:55:41,204] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 1.04 | optimizer_step: 1.02
[2025-04-26 13:55:41,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.55 | bwd_microstep: 5702.72 | bwd_inner_microstep: 5689.90 | bwd_allreduce_microstep: 12.78 | step_microstep: 19.29
[2025-04-26 13:55:41,205] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.55 | bwd: 5702.74 | bwd_inner: 5689.90 | bwd_allreduce: 12.80 | step: 19.29

 10%|▉         | 4056/41250 [9:47:12<89:16:32,  8.64s/it]
                                                         
{'loss': 0.1705, 'grad_norm': 1.4203472137451172, 'learning_rate': 3.951243926638145e-05, 'epoch': 0.98}

 10%|▉         | 4056/41250 [9:47:12<89:16:32,  8.64s/it][2025-04-26 13:55:49,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:55:49,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2859.98 | bwd_microstep: 5703.81 | bwd_inner_microstep: 5691.03 | bwd_allreduce_microstep: 12.74 | step_microstep: 18.77
[2025-04-26 13:55:49,847] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2859.98 | bwd: 5703.83 | bwd_inner: 5691.03 | bwd_allreduce: 12.76 | step: 18.78

 10%|▉         | 4057/41250 [9:47:21<89:16:53,  8.64s/it]
                                                         
{'loss': 0.3666, 'grad_norm': 2.3417298793792725, 'learning_rate': 3.95120945859065e-05, 'epoch': 0.98}

 10%|▉         | 4057/41250 [9:47:21<89:16:53,  8.64s/it][2025-04-26 13:55:58,790] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 13:55:58,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2820.84 | bwd_microstep: 6038.71 | bwd_inner_microstep: 5640.54 | bwd_allreduce_microstep: 398.11 | step_microstep: 19.16
[2025-04-26 13:55:58,791] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2820.84 | bwd: 6038.72 | bwd_inner: 5640.54 | bwd_allreduce: 398.14 | step: 19.16

 10%|▉         | 4058/41250 [9:47:29<90:12:42,  8.73s/it]
                                                         
{'loss': 0.3351, 'grad_norm': 4.280261993408203, 'learning_rate': 3.9511749785143317e-05, 'epoch': 0.98}

 10%|▉         | 4058/41250 [9:47:29<90:12:42,  8.73s/it][2025-04-26 13:56:07,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.02 | optimizer_step: 1.06
[2025-04-26 13:56:07,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.68 | bwd_microstep: 5774.60 | bwd_inner_microstep: 5656.64 | bwd_allreduce_microstep: 117.91 | step_microstep: 19.09
[2025-04-26 13:56:07,478] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.68 | bwd: 5774.61 | bwd_inner: 5656.64 | bwd_allreduce: 117.93 | step: 19.09

 10%|▉         | 4059/41250 [9:47:38<90:04:15,  8.72s/it]
                                                         
{'loss': 0.2333, 'grad_norm': 1.6645808219909668, 'learning_rate': 3.9511404864094034e-05, 'epoch': 0.98}

 10%|▉         | 4059/41250 [9:47:38<90:04:15,  8.72s/it][2025-04-26 13:56:16,262] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.11 | optimizer_step: 1.00
[2025-04-26 13:56:16,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.71 | bwd_microstep: 5869.73 | bwd_inner_microstep: 5656.52 | bwd_allreduce_microstep: 213.16 | step_microstep: 19.07
[2025-04-26 13:56:16,263] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.71 | bwd: 5869.74 | bwd_inner: 5656.52 | bwd_allreduce: 213.18 | step: 19.07

 10%|▉         | 4060/41250 [9:47:47<90:16:52,  8.74s/it]
                                                         
{'loss': 0.2245, 'grad_norm': 1.9143973588943481, 'learning_rate': 3.951105982276078e-05, 'epoch': 0.98}

 10%|▉         | 4060/41250 [9:47:47<90:16:52,  8.74s/it][2025-04-26 13:56:24,947] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.21 | optimizer_step: 0.99
[2025-04-26 13:56:24,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.51 | bwd_microstep: 5760.06 | bwd_inner_microstep: 5700.17 | bwd_allreduce_microstep: 59.84 | step_microstep: 19.27
[2025-04-26 13:56:24,948] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.51 | bwd: 5760.07 | bwd_inner: 5700.17 | bwd_allreduce: 59.86 | step: 19.27

 10%|▉         | 4061/41250 [9:47:56<90:06:14,  8.72s/it]
                                                         
{'loss': 0.0694, 'grad_norm': 1.3838132619857788, 'learning_rate': 3.9510714661145684e-05, 'epoch': 0.98}

 10%|▉         | 4061/41250 [9:47:56<90:06:14,  8.72s/it][2025-04-26 13:56:33,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.29 | optimizer_gradients: 1.19 | optimizer_step: 0.93
[2025-04-26 13:56:33,550] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.65 | bwd_microstep: 5692.60 | bwd_inner_microstep: 5648.14 | bwd_allreduce_microstep: 44.41 | step_microstep: 19.36
[2025-04-26 13:56:33,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2824.65 | bwd: 5692.62 | bwd_inner: 5648.14 | bwd_allreduce: 44.43 | step: 19.36

 10%|▉         | 4062/41250 [9:48:04<89:43:42,  8.69s/it]
                                                         
{'loss': 0.1092, 'grad_norm': 1.074299931526184, 'learning_rate': 3.9510369379250866e-05, 'epoch': 0.98}

 10%|▉         | 4062/41250 [9:48:04<89:43:42,  8.69s/it][2025-04-26 13:56:42,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 13:56:42,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.55 | bwd_microstep: 5749.61 | bwd_inner_microstep: 5657.89 | bwd_allreduce_microstep: 91.68 | step_microstep: 18.79
[2025-04-26 13:56:42,212] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.55 | bwd: 5749.62 | bwd_inner: 5657.89 | bwd_allreduce: 91.70 | step: 18.80

 10%|▉         | 4063/41250 [9:48:13<89:38:52,  8.68s/it]
                                                         
{'loss': 0.1302, 'grad_norm': 1.2229033708572388, 'learning_rate': 3.9510023977078453e-05, 'epoch': 0.98}

 10%|▉         | 4063/41250 [9:48:13<89:38:52,  8.68s/it][2025-04-26 13:56:50,898] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.44 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:56:50,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.96 | bwd_microstep: 5773.38 | bwd_inner_microstep: 5648.35 | bwd_allreduce_microstep: 124.98 | step_microstep: 18.48
[2025-04-26 13:56:50,899] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.96 | bwd: 5773.39 | bwd_inner: 5648.35 | bwd_allreduce: 125.00 | step: 18.49

 10%|▉         | 4064/41250 [9:48:22<89:40:10,  8.68s/it]
                                                         
{'loss': 0.0671, 'grad_norm': 0.6930499076843262, 'learning_rate': 3.950967845463059e-05, 'epoch': 0.99}

 10%|▉         | 4064/41250 [9:48:22<89:40:10,  8.68s/it][2025-04-26 13:56:59,580] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:56:59,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.92 | bwd_microstep: 5747.24 | bwd_inner_microstep: 5710.26 | bwd_allreduce_microstep: 36.92 | step_microstep: 18.74
[2025-04-26 13:56:59,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.92 | bwd: 5747.25 | bwd_inner: 5710.26 | bwd_allreduce: 36.94 | step: 18.74

 10%|▉         | 4065/41250 [9:48:30<89:40:33,  8.68s/it]
                                                         
{'loss': 0.3222, 'grad_norm': 1.8405601978302002, 'learning_rate': 3.950933281190939e-05, 'epoch': 0.99}

 10%|▉         | 4065/41250 [9:48:30<89:40:33,  8.68s/it][2025-04-26 13:57:08,207] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.12 | optimizer_gradients: 1.08 | optimizer_step: 1.01
[2025-04-26 13:57:08,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.25 | bwd_microstep: 5722.15 | bwd_inner_microstep: 5653.44 | bwd_allreduce_microstep: 68.66 | step_microstep: 19.10
[2025-04-26 13:57:08,208] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.25 | bwd: 5722.17 | bwd_inner: 5653.44 | bwd_allreduce: 68.68 | step: 19.11

 10%|▉         | 4066/41250 [9:48:39<89:30:03,  8.67s/it]
                                                         
{'loss': 0.1546, 'grad_norm': 1.2935965061187744, 'learning_rate': 3.950898704891699e-05, 'epoch': 0.99}

 10%|▉         | 4066/41250 [9:48:39<89:30:03,  8.67s/it][2025-04-26 13:57:16,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.37 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 13:57:16,886] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.78 | bwd_microstep: 5746.42 | bwd_inner_microstep: 5708.55 | bwd_allreduce_microstep: 37.82 | step_microstep: 18.52
[2025-04-26 13:57:16,887] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.78 | bwd: 5746.43 | bwd_inner: 5708.55 | bwd_allreduce: 37.84 | step: 18.53

 10%|▉         | 4067/41250 [9:48:48<89:32:21,  8.67s/it]
                                                         
{'loss': 0.1625, 'grad_norm': 0.8438771963119507, 'learning_rate': 3.950864116565552e-05, 'epoch': 0.99}

 10%|▉         | 4067/41250 [9:48:48<89:32:21,  8.67s/it][2025-04-26 13:57:25,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 13:57:25,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2842.90 | bwd_microstep: 5748.90 | bwd_inner_microstep: 5688.26 | bwd_allreduce_microstep: 60.59 | step_microstep: 18.81
[2025-04-26 13:57:25,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2842.90 | bwd: 5748.91 | bwd_inner: 5688.26 | bwd_allreduce: 60.61 | step: 18.82

 10%|▉         | 4068/41250 [9:48:56<89:33:07,  8.67s/it]
                                                         
{'loss': 0.0516, 'grad_norm': 0.6313653588294983, 'learning_rate': 3.950829516212712e-05, 'epoch': 0.99}

 10%|▉         | 4068/41250 [9:48:56<89:33:07,  8.67s/it][2025-04-26 13:57:34,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.36 | optimizer_gradients: 1.04 | optimizer_step: 0.90
[2025-04-26 13:57:34,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.65 | bwd_microstep: 5787.99 | bwd_inner_microstep: 5658.28 | bwd_allreduce_microstep: 129.67 | step_microstep: 19.05
[2025-04-26 13:57:34,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.65 | bwd: 5788.01 | bwd_inner: 5658.28 | bwd_allreduce: 129.69 | step: 19.05

 10%|▉         | 4069/41250 [9:49:05<89:38:23,  8.68s/it]
                                                         
{'loss': 0.0384, 'grad_norm': 0.6556083559989929, 'learning_rate': 3.9507949038333915e-05, 'epoch': 0.99}

 10%|▉         | 4069/41250 [9:49:05<89:38:23,  8.68s/it][2025-04-26 13:57:42,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.03 | optimizer_step: 0.99
[2025-04-26 13:57:42,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.81 | bwd_microstep: 5717.02 | bwd_inner_microstep: 5671.57 | bwd_allreduce_microstep: 45.40 | step_microstep: 19.24
[2025-04-26 13:57:42,882] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.81 | bwd: 5717.03 | bwd_inner: 5671.57 | bwd_allreduce: 45.42 | step: 19.25

 10%|▉         | 4070/41250 [9:49:14<89:27:57,  8.66s/it]
                                                         
{'loss': 0.064, 'grad_norm': 0.8362647294998169, 'learning_rate': 3.950760279427805e-05, 'epoch': 0.99}

 10%|▉         | 4070/41250 [9:49:14<89:27:57,  8.66s/it][2025-04-26 13:57:51,573] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 1.02 | optimizer_step: 0.91
[2025-04-26 13:57:51,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.30 | bwd_microstep: 5749.84 | bwd_inner_microstep: 5710.48 | bwd_allreduce_microstep: 39.32 | step_microstep: 19.22
[2025-04-26 13:57:51,574] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.30 | bwd: 5749.86 | bwd_inner: 5710.48 | bwd_allreduce: 39.33 | step: 19.23

 10%|▉         | 4071/41250 [9:49:22<89:33:28,  8.67s/it]
                                                         
{'loss': 0.2348, 'grad_norm': 3.3372714519500732, 'learning_rate': 3.9507256429961635e-05, 'epoch': 0.99}

 10%|▉         | 4071/41250 [9:49:22<89:33:28,  8.67s/it][2025-04-26 13:58:00,257] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.04 | optimizer_step: 1.05
[2025-04-26 13:58:00,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.05 | bwd_microstep: 5741.02 | bwd_inner_microstep: 5697.63 | bwd_allreduce_microstep: 43.33 | step_microstep: 19.37
[2025-04-26 13:58:00,258] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.05 | bwd: 5741.03 | bwd_inner: 5697.63 | bwd_allreduce: 43.35 | step: 19.38

 10%|▉         | 4072/41250 [9:49:31<89:35:06,  8.67s/it]
                                                         
{'loss': 0.0248, 'grad_norm': 0.5553785562515259, 'learning_rate': 3.950690994538683e-05, 'epoch': 0.99}

 10%|▉         | 4072/41250 [9:49:31<89:35:06,  8.67s/it][2025-04-26 13:58:09,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 13:58:09,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.18 | bwd_microstep: 5862.13 | bwd_inner_microstep: 5710.82 | bwd_allreduce_microstep: 151.25 | step_microstep: 18.77
[2025-04-26 13:58:09,057] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.19 | bwd: 5862.14 | bwd_inner: 5710.82 | bwd_allreduce: 151.27 | step: 18.77

 10%|▉         | 4073/41250 [9:49:40<89:58:06,  8.71s/it]
                                                         
{'loss': 0.1607, 'grad_norm': 1.9259010553359985, 'learning_rate': 3.950656334055576e-05, 'epoch': 0.99}

 10%|▉         | 4073/41250 [9:49:40<89:58:06,  8.71s/it][2025-04-26 13:58:17,718] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.34 | optimizer_gradients: 1.00 | optimizer_step: 1.01
[2025-04-26 13:58:17,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2857.06 | bwd_microstep: 5721.68 | bwd_inner_microstep: 5708.66 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.88
[2025-04-26 13:58:17,719] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2857.07 | bwd: 5721.70 | bwd_inner: 5708.66 | bwd_allreduce: 12.99 | step: 18.89

 10%|▉         | 4074/41250 [9:49:48<89:48:37,  8.70s/it]
                                                         
{'loss': 0.3433, 'grad_norm': 1.5876619815826416, 'learning_rate': 3.950621661547057e-05, 'epoch': 0.99}

 10%|▉         | 4074/41250 [9:49:48<89:48:37,  8.70s/it][2025-04-26 13:58:26,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 13:58:26,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.33 | bwd_microstep: 5733.86 | bwd_inner_microstep: 5721.12 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.57
[2025-04-26 13:58:26,383] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.33 | bwd: 5733.87 | bwd_inner: 5721.12 | bwd_allreduce: 12.71 | step: 18.57

 10%|▉         | 4075/41250 [9:49:57<89:42:27,  8.69s/it]
                                                         
{'loss': 0.038, 'grad_norm': 0.5875356197357178, 'learning_rate': 3.9505869770133387e-05, 'epoch': 0.99}

 10%|▉         | 4075/41250 [9:49:57<89:42:27,  8.69s/it][2025-04-26 13:58:35,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:58:35,034] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.37 | bwd_microstep: 5719.69 | bwd_inner_microstep: 5706.82 | bwd_allreduce_microstep: 12.83 | step_microstep: 18.66
[2025-04-26 13:58:35,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.37 | bwd: 5719.70 | bwd_inner: 5706.82 | bwd_allreduce: 12.85 | step: 18.66

 10%|▉         | 4076/41250 [9:50:06<89:35:34,  8.68s/it]
                                                         
{'loss': 0.053, 'grad_norm': 0.7738544940948486, 'learning_rate': 3.950552280454635e-05, 'epoch': 0.99}

 10%|▉         | 4076/41250 [9:50:06<89:35:34,  8.68s/it][2025-04-26 13:58:43,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:58:43,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.39 | bwd_microstep: 5748.33 | bwd_inner_microstep: 5712.29 | bwd_allreduce_microstep: 35.98 | step_microstep: 18.67
[2025-04-26 13:58:43,721] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.39 | bwd: 5748.34 | bwd_inner: 5712.29 | bwd_allreduce: 36.00 | step: 18.68

 10%|▉         | 4077/41250 [9:50:14<89:37:43,  8.68s/it]
                                                         
{'loss': 0.2652, 'grad_norm': 2.495845079421997, 'learning_rate': 3.9505175718711604e-05, 'epoch': 0.99}

 10%|▉         | 4077/41250 [9:50:14<89:37:43,  8.68s/it][2025-04-26 13:58:52,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.01 | optimizer_step: 0.98
[2025-04-26 13:58:52,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.69 | bwd_microstep: 5772.85 | bwd_inner_microstep: 5665.34 | bwd_allreduce_microstep: 107.46 | step_microstep: 19.22
[2025-04-26 13:58:52,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.69 | bwd: 5772.87 | bwd_inner: 5665.34 | bwd_allreduce: 107.49 | step: 19.23

 10%|▉         | 4078/41250 [9:50:23<89:39:10,  8.68s/it]
                                                         
{'loss': 0.2178, 'grad_norm': 3.5360374450683594, 'learning_rate': 3.950482851263128e-05, 'epoch': 0.99}

 10%|▉         | 4078/41250 [9:50:23<89:39:10,  8.68s/it][2025-04-26 13:59:01,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.54 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:59:01,101] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.47 | bwd_microstep: 5764.92 | bwd_inner_microstep: 5705.36 | bwd_allreduce_microstep: 59.51 | step_microstep: 18.92
[2025-04-26 13:59:01,102] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.47 | bwd: 5764.93 | bwd_inner: 5705.36 | bwd_allreduce: 59.53 | step: 18.92

 10%|▉         | 4079/41250 [9:50:32<89:40:31,  8.69s/it]
                                                         
{'loss': 0.184, 'grad_norm': 1.6228458881378174, 'learning_rate': 3.9504481186307535e-05, 'epoch': 0.99}

 10%|▉         | 4079/41250 [9:50:32<89:40:31,  8.69s/it][2025-04-26 13:59:09,748] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.03 | optimizer_step: 0.90
[2025-04-26 13:59:09,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.90 | bwd_microstep: 5708.46 | bwd_inner_microstep: 5695.23 | bwd_allreduce_microstep: 13.18 | step_microstep: 19.16
[2025-04-26 13:59:09,749] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.90 | bwd: 5708.47 | bwd_inner: 5695.23 | bwd_allreduce: 13.19 | step: 19.16

 10%|▉         | 4080/41250 [9:50:40<89:33:15,  8.67s/it]
                                                         
{'loss': 0.1855, 'grad_norm': 1.504977822303772, 'learning_rate': 3.950413373974249e-05, 'epoch': 0.99}

 10%|▉         | 4080/41250 [9:50:40<89:33:15,  8.67s/it][2025-04-26 13:59:18,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 1.02 | optimizer_step: 0.90
[2025-04-26 13:59:18,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.69 | bwd_microstep: 5773.19 | bwd_inner_microstep: 5715.97 | bwd_allreduce_microstep: 57.18 | step_microstep: 19.04
[2025-04-26 13:59:18,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.69 | bwd: 5773.21 | bwd_inner: 5715.97 | bwd_allreduce: 57.19 | step: 19.04

 10%|▉         | 4081/41250 [9:50:49<89:40:27,  8.69s/it]
                                                         
{'loss': 0.0507, 'grad_norm': 0.6179854869842529, 'learning_rate': 3.9503786172938305e-05, 'epoch': 0.99}

 10%|▉         | 4081/41250 [9:50:49<89:40:27,  8.69s/it][2025-04-26 13:59:27,094] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.59 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 13:59:27,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2830.87 | bwd_microstep: 5719.20 | bwd_inner_microstep: 5659.68 | bwd_allreduce_microstep: 59.47 | step_microstep: 19.02
[2025-04-26 13:59:27,095] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2830.87 | bwd: 5719.22 | bwd_inner: 5659.68 | bwd_allreduce: 59.49 | step: 19.02

 10%|▉         | 4082/41250 [9:50:58<89:30:26,  8.67s/it]
                                                         
{'loss': 0.1854, 'grad_norm': 3.7350099086761475, 'learning_rate': 3.950343848589711e-05, 'epoch': 0.99}

 10%|▉         | 4082/41250 [9:50:58<89:30:26,  8.67s/it][2025-04-26 13:59:35,771] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 0.98 | optimizer_step: 1.14
[2025-04-26 13:59:35,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2860.25 | bwd_microstep: 5733.78 | bwd_inner_microstep: 5721.21 | bwd_allreduce_microstep: 12.53 | step_microstep: 19.03
[2025-04-26 13:59:35,772] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2860.25 | bwd: 5733.80 | bwd_inner: 5721.21 | bwd_allreduce: 12.54 | step: 19.03

 10%|▉         | 4083/41250 [9:51:06<89:31:47,  8.67s/it]
                                                         
{'loss': 0.5358, 'grad_norm': 2.726242780685425, 'learning_rate': 3.9503090678621056e-05, 'epoch': 0.99}

 10%|▉         | 4083/41250 [9:51:06<89:31:47,  8.67s/it][2025-04-26 13:59:44,361] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 13:59:44,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.90 | bwd_microstep: 5684.60 | bwd_inner_microstep: 5663.87 | bwd_allreduce_microstep: 20.68 | step_microstep: 18.73
[2025-04-26 13:59:44,362] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.90 | bwd: 5684.61 | bwd_inner: 5663.87 | bwd_allreduce: 20.70 | step: 18.74

 10%|▉         | 4084/41250 [9:51:15<89:16:22,  8.65s/it]
                                                         
{'loss': 0.0929, 'grad_norm': 1.8224893808364868, 'learning_rate': 3.950274275111228e-05, 'epoch': 0.99}

 10%|▉         | 4084/41250 [9:51:15<89:16:22,  8.65s/it][2025-04-26 13:59:53,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 13:59:53,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.57 | bwd_microstep: 5712.16 | bwd_inner_microstep: 5699.39 | bwd_allreduce_microstep: 12.73 | step_microstep: 18.69
[2025-04-26 13:59:53,009] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.57 | bwd: 5712.17 | bwd_inner: 5699.39 | bwd_allreduce: 12.74 | step: 18.69

 10%|▉         | 4085/41250 [9:51:24<89:16:15,  8.65s/it]
                                                         
{'loss': 0.1499, 'grad_norm': 1.2115182876586914, 'learning_rate': 3.9502394703372934e-05, 'epoch': 0.99}

 10%|▉         | 4085/41250 [9:51:24<89:16:15,  8.65s/it][2025-04-26 14:00:01,675] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.21 | optimizer_step: 0.93
[2025-04-26 14:00:01,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.28 | bwd_microstep: 5751.71 | bwd_inner_microstep: 5661.34 | bwd_allreduce_microstep: 90.32 | step_microstep: 19.11
[2025-04-26 14:00:01,676] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2833.28 | bwd: 5751.73 | bwd_inner: 5661.34 | bwd_allreduce: 90.35 | step: 19.12

 10%|▉         | 4086/41250 [9:51:32<89:19:50,  8.65s/it]
                                                         
{'loss': 0.1361, 'grad_norm': 2.4830899238586426, 'learning_rate': 3.950204653540516e-05, 'epoch': 0.99}

 10%|▉         | 4086/41250 [9:51:32<89:19:50,  8.65s/it][2025-04-26 14:00:10,352] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.06 | optimizer_step: 1.01
[2025-04-26 14:00:10,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2823.09 | bwd_microstep: 5771.94 | bwd_inner_microstep: 5652.83 | bwd_allreduce_microstep: 119.06 | step_microstep: 19.06
[2025-04-26 14:00:10,353] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2823.09 | bwd: 5771.96 | bwd_inner: 5652.83 | bwd_allreduce: 119.08 | step: 19.07

 10%|▉         | 4087/41250 [9:51:41<89:23:56,  8.66s/it]
                                                         
{'loss': 0.1418, 'grad_norm': 3.534104585647583, 'learning_rate': 3.95016982472111e-05, 'epoch': 0.99}

 10%|▉         | 4087/41250 [9:51:41<89:23:56,  8.66s/it][2025-04-26 14:00:18,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 14:00:18,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.27 | bwd_microstep: 5717.93 | bwd_inner_microstep: 5663.44 | bwd_allreduce_microstep: 54.45 | step_microstep: 18.70
[2025-04-26 14:00:18,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.27 | bwd: 5717.94 | bwd_inner: 5663.44 | bwd_allreduce: 54.46 | step: 18.70

 10%|▉         | 4088/41250 [9:51:50<89:15:33,  8.65s/it]
                                                         
{'loss': 0.2506, 'grad_norm': 2.3259565830230713, 'learning_rate': 3.950134983879292e-05, 'epoch': 0.99}

 10%|▉         | 4088/41250 [9:51:50<89:15:33,  8.65s/it][2025-04-26 14:00:27,649] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.17 | optimizer_gradients: 1.08 | optimizer_step: 1.00
[2025-04-26 14:00:27,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.93 | bwd_microstep: 5751.85 | bwd_inner_microstep: 5696.77 | bwd_allreduce_microstep: 55.02 | step_microstep: 19.13
[2025-04-26 14:00:27,650] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.93 | bwd: 5751.86 | bwd_inner: 5696.77 | bwd_allreduce: 55.05 | step: 19.13

 10%|▉         | 4089/41250 [9:51:58<89:22:05,  8.66s/it]
                                                         
{'loss': 0.3183, 'grad_norm': 3.140249729156494, 'learning_rate': 3.950100131015274e-05, 'epoch': 0.99}

 10%|▉         | 4089/41250 [9:51:58<89:22:05,  8.66s/it][mov,mp4,m4a,3gp,3g2,mj2 @ 0x4b6ee000] moov atom not found
[14:00:29] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, Invalid data found when processing input
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4...
[2025-04-26 14:00:36,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.92
[2025-04-26 14:00:36,423] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.21 | bwd_microstep: 5800.69 | bwd_inner_microstep: 5758.01 | bwd_allreduce_microstep: 42.64 | step_microstep: 18.87
[2025-04-26 14:00:36,424] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2887.21 | bwd: 5800.71 | bwd_inner: 5758.01 | bwd_allreduce: 42.65 | step: 18.87

 10%|▉         | 4090/41250 [9:52:07<89:43:42,  8.69s/it]
                                                         
{'loss': 0.214, 'grad_norm': 1.5808570384979248, 'learning_rate': 3.9500652661292725e-05, 'epoch': 0.99}

 10%|▉         | 4090/41250 [9:52:07<89:43:42,  8.69s/it][2025-04-26 14:00:45,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.45 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 14:00:45,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2849.57 | bwd_microstep: 5697.08 | bwd_inner_microstep: 5684.32 | bwd_allreduce_microstep: 12.72 | step_microstep: 18.94
[2025-04-26 14:00:45,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2849.57 | bwd: 5697.09 | bwd_inner: 5684.32 | bwd_allreduce: 12.73 | step: 18.94

 10%|▉         | 4091/41250 [9:52:16<89:31:50,  8.67s/it]
                                                         
{'loss': 0.3075, 'grad_norm': 1.9493638277053833, 'learning_rate': 3.9500303892215026e-05, 'epoch': 0.99}

 10%|▉         | 4091/41250 [9:52:16<89:31:50,  8.67s/it][2025-04-26 14:00:53,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 14:00:53,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.06 | bwd_microstep: 5767.71 | bwd_inner_microstep: 5654.00 | bwd_allreduce_microstep: 113.67 | step_microstep: 18.65
[2025-04-26 14:00:53,733] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2827.06 | bwd: 5767.73 | bwd_inner: 5654.00 | bwd_allreduce: 113.69 | step: 18.65

 10%|▉         | 4092/41250 [9:52:24<89:32:15,  8.67s/it]
                                                         
{'loss': 0.244, 'grad_norm': 1.4395318031311035, 'learning_rate': 3.9499955002921784e-05, 'epoch': 0.99}

 10%|▉         | 4092/41250 [9:52:24<89:32:15,  8.67s/it][2025-04-26 14:01:02,630] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.98 | optimizer_step: 0.90
[2025-04-26 14:01:02,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.71 | bwd_microstep: 5890.67 | bwd_inner_microstep: 5877.85 | bwd_allreduce_microstep: 12.78 | step_microstep: 18.78
[2025-04-26 14:01:02,631] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2932.71 | bwd: 5890.69 | bwd_inner: 5877.85 | bwd_allreduce: 12.80 | step: 18.78

 10%|▉         | 4093/41250 [9:52:33<90:13:34,  8.74s/it]
                                                         
{'loss': 0.2087, 'grad_norm': 1.6231790781021118, 'learning_rate': 3.949960599341516e-05, 'epoch': 0.99}

 10%|▉         | 4093/41250 [9:52:33<90:13:34,  8.74s/it][2025-04-26 14:01:11,307] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 14:01:11,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2852.11 | bwd_microstep: 5739.96 | bwd_inner_microstep: 5695.25 | bwd_allreduce_microstep: 44.66 | step_microstep: 18.61
[2025-04-26 14:01:11,308] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2852.11 | bwd: 5739.98 | bwd_inner: 5695.25 | bwd_allreduce: 44.68 | step: 18.61

 10%|▉         | 4094/41250 [9:52:42<90:01:32,  8.72s/it]
                                                         
{'loss': 0.2077, 'grad_norm': 1.6672494411468506, 'learning_rate': 3.94992568636973e-05, 'epoch': 0.99}

 10%|▉         | 4094/41250 [9:52:42<90:01:32,  8.72s/it][2025-04-26 14:01:20,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 6.67 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 14:01:20,108] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2826.82 | bwd_microstep: 5885.70 | bwd_inner_microstep: 5643.88 | bwd_allreduce_microstep: 241.78 | step_microstep: 19.85
[2025-04-26 14:01:20,109] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2826.82 | bwd: 5885.72 | bwd_inner: 5643.88 | bwd_allreduce: 241.79 | step: 19.86

 10%|▉         | 4095/41250 [9:52:51<90:16:05,  8.75s/it]
                                                         
{'loss': 0.1602, 'grad_norm': 1.2559350728988647, 'learning_rate': 3.949890761377035e-05, 'epoch': 0.99}

 10%|▉         | 4095/41250 [9:52:51<90:16:05,  8.75s/it][2025-04-26 14:01:28,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 14:01:28,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2846.37 | bwd_microstep: 5857.16 | bwd_inner_microstep: 5684.10 | bwd_allreduce_microstep: 173.01 | step_microstep: 19.02
[2025-04-26 14:01:28,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2846.36 | bwd: 5857.17 | bwd_inner: 5684.10 | bwd_allreduce: 173.03 | step: 19.02

 10%|▉         | 4096/41250 [9:53:00<90:23:02,  8.76s/it]
                                                         
{'loss': 0.0739, 'grad_norm': 1.0193445682525635, 'learning_rate': 3.949855824363647e-05, 'epoch': 0.99}

 10%|▉         | 4096/41250 [9:53:00<90:23:02,  8.76s/it][2025-04-26 14:01:37,548] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.27 | optimizer_gradients: 1.02 | optimizer_step: 0.92
[2025-04-26 14:01:37,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.53 | bwd_microstep: 5737.31 | bwd_inner_microstep: 5690.59 | bwd_allreduce_microstep: 46.67 | step_microstep: 19.04
[2025-04-26 14:01:37,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.53 | bwd: 5737.32 | bwd_inner: 5690.59 | bwd_allreduce: 46.69 | step: 19.04

 10%|▉         | 4097/41250 [9:53:08<90:03:54,  8.73s/it]
                                                         
{'loss': 0.1284, 'grad_norm': 0.9981158971786499, 'learning_rate': 3.9498208753297824e-05, 'epoch': 0.99}

 10%|▉         | 4097/41250 [9:53:08<90:03:54,  8.73s/it][2025-04-26 14:01:46,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.04 | optimizer_gradients: 1.01 | optimizer_step: 0.96
[2025-04-26 14:01:46,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.57 | bwd_microstep: 5871.03 | bwd_inner_microstep: 5719.46 | bwd_allreduce_microstep: 151.52 | step_microstep: 18.77
[2025-04-26 14:01:46,359] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.57 | bwd: 5871.05 | bwd_inner: 5719.46 | bwd_allreduce: 151.54 | step: 18.77

 10%|▉         | 4098/41250 [9:53:17<90:19:03,  8.75s/it]
                                                         
{'loss': 0.0512, 'grad_norm': 0.532365620136261, 'learning_rate': 3.949785914275655e-05, 'epoch': 0.99}

 10%|▉         | 4098/41250 [9:53:17<90:19:03,  8.75s/it][2025-04-26 14:01:54,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 14:01:54,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2843.80 | bwd_microstep: 5700.90 | bwd_inner_microstep: 5687.91 | bwd_allreduce_microstep: 12.93 | step_microstep: 18.81
[2025-04-26 14:01:54,986] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2843.80 | bwd: 5700.91 | bwd_inner: 5687.91 | bwd_allreduce: 12.95 | step: 18.81

 10%|▉         | 4099/41250 [9:53:26<89:55:53,  8.71s/it]
                                                         
{'loss': 0.1656, 'grad_norm': 1.2679983377456665, 'learning_rate': 3.949750941201481e-05, 'epoch': 0.99}

 10%|▉         | 4099/41250 [9:53:26<89:55:53,  8.71s/it][2025-04-26 14:02:03,581] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.24 | optimizer_gradients: 0.99 | optimizer_step: 1.04
[2025-04-26 14:02:03,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.55 | bwd_microstep: 5691.28 | bwd_inner_microstep: 5642.73 | bwd_allreduce_microstep: 48.50 | step_microstep: 18.75
[2025-04-26 14:02:03,582] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2821.55 | bwd: 5691.30 | bwd_inner: 5642.73 | bwd_allreduce: 48.52 | step: 18.76

 10%|▉         | 4100/41250 [9:53:34<89:34:08,  8.68s/it]
                                                         
{'loss': 0.2931, 'grad_norm': 2.0953116416931152, 'learning_rate': 3.949715956107475e-05, 'epoch': 0.99}

 10%|▉         | 4100/41250 [9:53:34<89:34:08,  8.68s/it][2025-04-26 14:02:12,245] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.40 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 14:02:12,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2832.75 | bwd_microstep: 5746.03 | bwd_inner_microstep: 5648.98 | bwd_allreduce_microstep: 97.00 | step_microstep: 18.81
[2025-04-26 14:02:12,246] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2832.75 | bwd: 5746.04 | bwd_inner: 5648.98 | bwd_allreduce: 97.02 | step: 18.81

 10%|▉         | 4101/41250 [9:53:43<89:30:33,  8.67s/it]
                                                         
{'loss': 0.1448, 'grad_norm': 1.2438780069351196, 'learning_rate': 3.9496809589938555e-05, 'epoch': 0.99}

 10%|▉         | 4101/41250 [9:53:43<89:30:33,  8.67s/it][2025-04-26 14:02:21,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.99 | optimizer_step: 0.92
[2025-04-26 14:02:21,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2931.58 | bwd_microstep: 5877.55 | bwd_inner_microstep: 5864.82 | bwd_allreduce_microstep: 12.69 | step_microstep: 18.66
[2025-04-26 14:02:21,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2931.58 | bwd: 5877.57 | bwd_inner: 5864.82 | bwd_allreduce: 12.71 | step: 18.66

 10%|▉         | 4102/41250 [9:53:52<90:09:21,  8.74s/it]
                                                         
{'loss': 0.093, 'grad_norm': 1.0715811252593994, 'learning_rate': 3.949645949860835e-05, 'epoch': 0.99}

 10%|▉         | 4102/41250 [9:53:52<90:09:21,  8.74s/it][2025-04-26 14:02:29,793] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 14:02:29,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.74 | bwd_microstep: 5747.85 | bwd_inner_microstep: 5652.05 | bwd_allreduce_microstep: 95.75 | step_microstep: 18.38
[2025-04-26 14:02:29,794] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.74 | bwd: 5747.86 | bwd_inner: 5652.05 | bwd_allreduce: 95.77 | step: 18.38

 10%|▉         | 4103/41250 [9:54:00<89:55:38,  8.72s/it]
                                                         
{'loss': 0.2554, 'grad_norm': 2.616133213043213, 'learning_rate': 3.9496109287086314e-05, 'epoch': 0.99}

 10%|▉         | 4103/41250 [9:54:00<89:55:38,  8.72s/it][2025-04-26 14:02:38,402] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 4.96 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 14:02:38,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2838.74 | bwd_microstep: 5685.75 | bwd_inner_microstep: 5661.29 | bwd_allreduce_microstep: 24.42 | step_microstep: 17.80
[2025-04-26 14:02:38,403] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2838.75 | bwd: 5685.77 | bwd_inner: 5661.29 | bwd_allreduce: 24.44 | step: 17.81

 10%|▉         | 4104/41250 [9:54:09<89:35:42,  8.68s/it]
                                                         
{'loss': 0.2819, 'grad_norm': 1.9544668197631836, 'learning_rate': 3.94957589553746e-05, 'epoch': 0.99}

 10%|▉         | 4104/41250 [9:54:09<89:35:42,  8.68s/it][2025-04-26 14:02:47,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.49 | optimizer_gradients: 1.00 | optimizer_step: 0.90
[2025-04-26 14:02:47,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2822.18 | bwd_microstep: 5748.82 | bwd_inner_microstep: 5660.57 | bwd_allreduce_microstep: 88.21 | step_microstep: 18.49
[2025-04-26 14:02:47,056] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2822.18 | bwd: 5748.83 | bwd_inner: 5660.56 | bwd_allreduce: 88.23 | step: 18.49

 10%|▉         | 4105/41250 [9:54:18<89:30:07,  8.67s/it]
                                                         
{'loss': 0.2021, 'grad_norm': 2.1308095455169678, 'learning_rate': 3.949540850347536e-05, 'epoch': 1.0}

 10%|▉         | 4105/41250 [9:54:18<89:30:07,  8.67s/it][2025-04-26 14:02:55,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.99 | optimizer_step: 0.89
[2025-04-26 14:02:55,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2845.86 | bwd_microstep: 5705.72 | bwd_inner_microstep: 5692.70 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.53
[2025-04-26 14:02:55,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2845.86 | bwd: 5705.73 | bwd_inner: 5692.70 | bwd_allreduce: 12.99 | step: 18.53

 10%|▉         | 4106/41250 [9:54:26<89:20:50,  8.66s/it]
                                                         
{'loss': 0.3069, 'grad_norm': 1.8637737035751343, 'learning_rate': 3.9495057931390764e-05, 'epoch': 1.0}

 10%|▉         | 4106/41250 [9:54:26<89:20:50,  8.66s/it][2025-04-26 14:03:04,350] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.96
[2025-04-26 14:03:04,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2864.68 | bwd_microstep: 5717.83 | bwd_inner_microstep: 5704.81 | bwd_allreduce_microstep: 12.97 | step_microstep: 18.88
[2025-04-26 14:03:04,351] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2864.68 | bwd: 5717.84 | bwd_inner: 5704.81 | bwd_allreduce: 12.99 | step: 18.89

 10%|▉         | 4107/41250 [9:54:35<89:22:39,  8.66s/it]
                                                         
{'loss': 0.1211, 'grad_norm': 2.0844857692718506, 'learning_rate': 3.949470723912297e-05, 'epoch': 1.0}

 10%|▉         | 4107/41250 [9:54:35<89:22:39,  8.66s/it][2025-04-26 14:03:13,030] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.32 | optimizer_gradients: 1.00 | optimizer_step: 0.98
[2025-04-26 14:03:13,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2858.32 | bwd_microstep: 5738.69 | bwd_inner_microstep: 5708.46 | bwd_allreduce_microstep: 30.18 | step_microstep: 18.48
[2025-04-26 14:03:13,031] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2858.32 | bwd: 5738.71 | bwd_inner: 5708.46 | bwd_allreduce: 30.20 | step: 18.49

 10%|▉         | 4108/41250 [9:54:44<89:25:44,  8.67s/it]
                                                         
{'loss': 0.1682, 'grad_norm': 2.0710866451263428, 'learning_rate': 3.9494356426674143e-05, 'epoch': 1.0}

 10%|▉         | 4108/41250 [9:54:44<89:25:44,  8.67s/it][2025-04-26 14:03:21,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.28 | optimizer_gradients: 1.08 | optimizer_step: 0.90
[2025-04-26 14:03:21,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.67 | bwd_microstep: 5846.43 | bwd_inner_microstep: 5699.25 | bwd_allreduce_microstep: 147.12 | step_microstep: 19.13
[2025-04-26 14:03:21,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.67 | bwd: 5846.44 | bwd_inner: 5699.25 | bwd_allreduce: 147.14 | step: 19.14

 10%|▉         | 4109/41250 [9:54:52<89:47:14,  8.70s/it]
                                                         
{'loss': 0.1115, 'grad_norm': 1.2967761754989624, 'learning_rate': 3.949400549404644e-05, 'epoch': 1.0}

 10%|▉         | 4109/41250 [9:54:53<89:47:14,  8.70s/it][2025-04-26 14:03:30,430] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.99 | optimizer_step: 0.90
[2025-04-26 14:03:30,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.17 | bwd_microstep: 5704.77 | bwd_inner_microstep: 5650.54 | bwd_allreduce_microstep: 54.18 | step_microstep: 18.65
[2025-04-26 14:03:30,431] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.17 | bwd: 5704.78 | bwd_inner: 5650.54 | bwd_allreduce: 54.20 | step: 18.65

 10%|▉         | 4110/41250 [9:55:01<89:30:44,  8.68s/it]
                                                         
{'loss': 0.1244, 'grad_norm': 1.59711754322052, 'learning_rate': 3.9493654441242026e-05, 'epoch': 1.0}

 10%|▉         | 4110/41250 [9:55:01<89:30:44,  8.68s/it][2025-04-26 14:03:39,130] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.30 | optimizer_gradients: 0.99 | optimizer_step: 1.15
[2025-04-26 14:03:39,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.28 | bwd_microstep: 5790.73 | bwd_inner_microstep: 5657.76 | bwd_allreduce_microstep: 132.93 | step_microstep: 18.95
[2025-04-26 14:03:39,131] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2834.28 | bwd: 5790.74 | bwd_inner: 5657.76 | bwd_allreduce: 132.94 | step: 18.96

 10%|▉         | 4111/41250 [9:55:10<89:35:04,  8.68s/it]
                                                         
{'loss': 0.2524, 'grad_norm': 1.6704260110855103, 'learning_rate': 3.9493303268263064e-05, 'epoch': 1.0}

 10%|▉         | 4111/41250 [9:55:10<89:35:04,  8.68s/it][2025-04-26 14:03:47,778] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.92
[2025-04-26 14:03:47,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.35 | bwd_microstep: 5717.04 | bwd_inner_microstep: 5703.60 | bwd_allreduce_microstep: 13.39 | step_microstep: 19.05
[2025-04-26 14:03:47,779] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.35 | bwd: 5717.06 | bwd_inner: 5703.60 | bwd_allreduce: 13.42 | step: 19.05

 10%|▉         | 4112/41250 [9:55:18<89:28:19,  8.67s/it]
                                                         
{'loss': 0.2042, 'grad_norm': 1.8191859722137451, 'learning_rate': 3.949295197511173e-05, 'epoch': 1.0}

 10%|▉         | 4112/41250 [9:55:18<89:28:19,  8.67s/it][2025-04-26 14:03:56,488] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.14 | optimizer_gradients: 1.05 | optimizer_step: 1.08
[2025-04-26 14:03:56,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.95 | bwd_microstep: 5793.83 | bwd_inner_microstep: 5671.53 | bwd_allreduce_microstep: 122.25 | step_microstep: 18.72
[2025-04-26 14:03:56,489] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2831.95 | bwd: 5793.85 | bwd_inner: 5671.53 | bwd_allreduce: 122.27 | step: 18.72

 10%|▉         | 4113/41250 [9:55:27<89:34:52,  8.68s/it]
                                                         
{'loss': 0.0874, 'grad_norm': 0.7500478625297546, 'learning_rate': 3.949260056179017e-05, 'epoch': 1.0}

 10%|▉         | 4113/41250 [9:55:27<89:34:52,  8.68s/it][2025-04-26 14:04:05,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 0.97 | optimizer_step: 0.90
[2025-04-26 14:04:05,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.36 | bwd_microstep: 5713.90 | bwd_inner_microstep: 5701.19 | bwd_allreduce_microstep: 12.66 | step_microstep: 18.58
[2025-04-26 14:04:05,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.36 | bwd: 5713.92 | bwd_inner: 5701.19 | bwd_allreduce: 12.68 | step: 18.58

 10%|▉         | 4114/41250 [9:55:36<89:27:18,  8.67s/it]
                                                         
{'loss': 0.1781, 'grad_norm': 1.12101149559021, 'learning_rate': 3.949224902830057e-05, 'epoch': 1.0}

 10%|▉         | 4114/41250 [9:55:36<89:27:18,  8.67s/it][2025-04-26 14:04:13,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.15 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 14:04:13,786] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2850.89 | bwd_microstep: 5732.65 | bwd_inner_microstep: 5711.84 | bwd_allreduce_microstep: 20.76 | step_microstep: 18.06
[2025-04-26 14:04:13,787] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2850.89 | bwd: 5732.66 | bwd_inner: 5711.84 | bwd_allreduce: 20.78 | step: 18.06

 10%|▉         | 4115/41250 [9:55:44<89:23:42,  8.67s/it]
                                                         
{'loss': 0.2785, 'grad_norm': 3.2205309867858887, 'learning_rate': 3.949189737464509e-05, 'epoch': 1.0}

 10%|▉         | 4115/41250 [9:55:44<89:23:42,  8.67s/it][2025-04-26 14:04:22,461] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.51 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 14:04:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2853.78 | bwd_microstep: 5738.87 | bwd_inner_microstep: 5703.50 | bwd_allreduce_microstep: 35.32 | step_microstep: 19.00
[2025-04-26 14:04:22,462] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2853.78 | bwd: 5738.88 | bwd_inner: 5703.50 | bwd_allreduce: 35.34 | step: 19.01

 10%|▉         | 4116/41250 [9:55:53<89:25:28,  8.67s/it]
                                                         
{'loss': 0.1578, 'grad_norm': 1.429623007774353, 'learning_rate': 3.94915456008259e-05, 'epoch': 1.0}

 10%|▉         | 4116/41250 [9:55:53<89:25:28,  8.67s/it][2025-04-26 14:04:31,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.53 | optimizer_gradients: 0.96 | optimizer_step: 0.89
[2025-04-26 14:04:31,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2856.17 | bwd_microstep: 5732.13 | bwd_inner_microstep: 5719.40 | bwd_allreduce_microstep: 12.68 | step_microstep: 18.39
[2025-04-26 14:04:31,134] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2856.17 | bwd: 5732.14 | bwd_inner: 5719.40 | bwd_allreduce: 12.70 | step: 18.39

 10%|▉         | 4117/41250 [9:56:02<89:25:36,  8.67s/it]
                                                         
{'loss': 0.2452, 'grad_norm': 3.0433366298675537, 'learning_rate': 3.949119370684515e-05, 'epoch': 1.0}

 10%|▉         | 4117/41250 [9:56:02<89:25:36,  8.67s/it][2025-04-26 14:04:39,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 1.01 | optimizer_step: 0.90
[2025-04-26 14:04:39,816] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2825.57 | bwd_microstep: 5774.67 | bwd_inner_microstep: 5653.50 | bwd_allreduce_microstep: 121.12 | step_microstep: 18.61
[2025-04-26 14:04:39,817] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2825.57 | bwd: 5774.68 | bwd_inner: 5653.50 | bwd_allreduce: 121.14 | step: 18.61

 10%|▉         | 4118/41250 [9:56:10<89:27:51,  8.67s/it]
                                                         
{'loss': 0.3086, 'grad_norm': 4.19312858581543, 'learning_rate': 3.949084169270504e-05, 'epoch': 1.0}

 10%|▉         | 4118/41250 [9:56:11<89:27:51,  8.67s/it][2025-04-26 14:04:48,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.31 | optimizer_gradients: 0.98 | optimizer_step: 1.07
[2025-04-26 14:04:48,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2848.87 | bwd_microstep: 5768.95 | bwd_inner_microstep: 5719.63 | bwd_allreduce_microstep: 49.28 | step_microstep: 18.50
[2025-04-26 14:04:48,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2848.87 | bwd: 5768.96 | bwd_inner: 5719.63 | bwd_allreduce: 49.30 | step: 18.50

 10%|▉         | 4119/41250 [9:56:19<89:32:28,  8.68s/it]
                                                         
{'loss': 0.1528, 'grad_norm': 1.2572075128555298, 'learning_rate': 3.949048955840772e-05, 'epoch': 1.0}

 10%|▉         | 4119/41250 [9:56:19<89:32:28,  8.68s/it][2025-04-26 14:04:57,176] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 0.95 | optimizer_step: 0.89
[2025-04-26 14:04:57,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2854.39 | bwd_microstep: 5730.54 | bwd_inner_microstep: 5718.01 | bwd_allreduce_microstep: 12.48 | step_microstep: 18.43
[2025-04-26 14:04:57,177] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2854.39 | bwd: 5730.55 | bwd_inner: 5718.01 | bwd_allreduce: 12.50 | step: 18.43

 10%|▉         | 4120/41250 [9:56:28<89:28:33,  8.68s/it]
                                                         
{'loss': 0.2871, 'grad_norm': 1.7137424945831299, 'learning_rate': 3.949013730395535e-05, 'epoch': 1.0}

 10%|▉         | 4120/41250 [9:56:28<89:28:33,  8.68s/it][2025-04-26 14:05:05,944] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.48 | optimizer_gradients: 0.98 | optimizer_step: 0.89
[2025-04-26 14:05:05,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2851.54 | bwd_microstep: 5834.32 | bwd_inner_microstep: 5722.36 | bwd_allreduce_microstep: 111.91 | step_microstep: 18.50
[2025-04-26 14:05:05,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2851.54 | bwd: 5834.33 | bwd_inner: 5722.36 | bwd_allreduce: 111.93 | step: 18.51

 10%|▉         | 4121/41250 [9:56:37<89:45:58,  8.70s/it]
                                                         
{'loss': 0.4277, 'grad_norm': 2.72971510887146, 'learning_rate': 3.9489784929350135e-05, 'epoch': 1.0}

 10%|▉         | 4121/41250 [9:56:37<89:45:58,  8.70s/it][2025-04-26 14:05:14,545] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 1.00 | optimizer_step: 0.97
[2025-04-26 14:05:14,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.45 | bwd_microstep: 5687.88 | bwd_inner_microstep: 5665.10 | bwd_allreduce_microstep: 22.73 | step_microstep: 18.92
[2025-04-26 14:05:14,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2829.45 | bwd: 5687.89 | bwd_inner: 5665.10 | bwd_allreduce: 22.75 | step: 18.92

 10%|▉         | 4122/41250 [9:56:45<89:26:35,  8.67s/it]
                                                         
{'loss': 0.1097, 'grad_norm': 1.065871238708496, 'learning_rate': 3.9489432434594224e-05, 'epoch': 1.0}

 10%|▉         | 4122/41250 [9:56:45<89:26:35,  8.67s/it][2025-04-26 14:05:23,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.47 | optimizer_gradients: 1.00 | optimizer_step: 0.91
[2025-04-26 14:05:23,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.75 | bwd_microstep: 5793.70 | bwd_inner_microstep: 5664.91 | bwd_allreduce_microstep: 128.75 | step_microstep: 18.63
[2025-04-26 14:05:23,248] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2828.75 | bwd: 5793.72 | bwd_inner: 5664.91 | bwd_allreduce: 128.76 | step: 18.63

 10%|▉         | 4123/41250 [9:56:54<89:32:01,  8.68s/it]
                                                         
{'loss': 0.1373, 'grad_norm': 1.3465911149978638, 'learning_rate': 3.94890798196898e-05, 'epoch': 1.0}

 10%|▉         | 4123/41250 [9:56:54<89:32:01,  8.68s/it][2025-04-26 14:05:31,909] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.52 | optimizer_gradients: 1.01 | optimizer_step: 0.89
[2025-04-26 14:05:31,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2847.00 | bwd_microstep: 5743.56 | bwd_inner_microstep: 5700.48 | bwd_allreduce_microstep: 43.04 | step_microstep: 18.83
[2025-04-26 14:05:31,910] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2847.00 | bwd: 5743.58 | bwd_inner: 5700.48 | bwd_allreduce: 43.06 | step: 18.83

 10%|▉         | 4124/41250 [9:57:03<89:28:09,  8.68s/it]
                                                         
{'loss': 0.2182, 'grad_norm': 1.1854397058486938, 'learning_rate': 3.948872708463903e-05, 'epoch': 1.0}

 10%|▉         | 4124/41250 [9:57:03<89:28:09,  8.68s/it]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Pyramid/01174.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00778.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00776.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/Animate/00777.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Failed to load video: /home/wangjiarui/AIGV6K/Allvideos/ZeroScope/02039.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
[2025-04-26 14:05:42,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 5.50 | optimizer_gradients: 0.97 | optimizer_step: 0.89
[2025-04-26 14:05:42,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.05 | bwd_microstep: 5695.93 | bwd_inner_microstep: 5683.11 | bwd_allreduce_microstep: 12.77 | step_microstep: 18.32
[2025-04-26 14:05:42,634] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 2839.05 | bwd: 5695.94 | bwd_inner: 5683.11 | bwd_allreduce: 12.79 | step: 18.32

 10%|█         | 4125/41250 [9:57:13<95:48:03,  9.29s/it]evaluate!
evaluate!

                                                         
{'loss': 0.2961, 'grad_norm': 2.8895785808563232, 'learning_rate': 3.948837422944409e-05, 'epoch': 1.0}

 10%|█         | 4125/41250 [9:57:13<95:48:03,  9.29s/it]evaluate!
1
1
1
[WARNING|trainer.py:803] 2025-04-26 14:05:45,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:05:45,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:05:45,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2
2
2
[WARNING|trainer.py:803] 2025-04-26 14:05:47,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:47,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:47,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
3
3
[WARNING|trainer.py:803] 2025-04-26 14:05:50,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:50,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:50,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4
4
4
[WARNING|trainer.py:803] 2025-04-26 14:05:52,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:05:52,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:05:52,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5
5
5
[WARNING|trainer.py:803] 2025-04-26 14:05:55,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:05:55,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:05:55,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6
6
6
[WARNING|trainer.py:803] 2025-04-26 14:05:57,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:57,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:05:57,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7
7
7
[WARNING|trainer.py:803] 2025-04-26 14:06:00,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:00,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:00,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8
8
8
[WARNING|trainer.py:803] 2025-04-26 14:06:02,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:06:02,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:06:02,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
9
9
9
[WARNING|trainer.py:803] 2025-04-26 14:06:05,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:05,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:05,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
10
10
10
[WARNING|trainer.py:803] 2025-04-26 14:06:07,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:07,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:07,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
11
11
11
[WARNING|trainer.py:803] 2025-04-26 14:06:10,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:10,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:10,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
12
12
12
[WARNING|trainer.py:803] 2025-04-26 14:06:12,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:12,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:06:12,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
13
13
13
[WARNING|trainer.py:803] 2025-04-26 14:06:15,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:15,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:15,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
14
14
14
[WARNING|trainer.py:803] 2025-04-26 14:06:17,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:17,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:17,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
15
15
15
[WARNING|trainer.py:803] 2025-04-26 14:06:20,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:20,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:20,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
16
16
16
[WARNING|trainer.py:803] 2025-04-26 14:06:22,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:22,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:22,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
17
17
17
[WARNING|trainer.py:803] 2025-04-26 14:06:25,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:25,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:25,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
18
18
18
[WARNING|trainer.py:803] 2025-04-26 14:06:27,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:27,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:27,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
19
19
19
[WARNING|trainer.py:803] 2025-04-26 14:06:29,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:29,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:29,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
20
20
20
[WARNING|trainer.py:803] 2025-04-26 14:06:32,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:32,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:32,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
21
21
21
[WARNING|trainer.py:803] 2025-04-26 14:06:34,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:34,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:34,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
22
22
22
[WARNING|trainer.py:803] 2025-04-26 14:06:37,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:37,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:37,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
23
23
23
[WARNING|trainer.py:803] 2025-04-26 14:06:39,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:39,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:06:39,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
24
24
24
[WARNING|trainer.py:803] 2025-04-26 14:06:42,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:42,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:42,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
25
25
25
[WARNING|trainer.py:803] 2025-04-26 14:06:45,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:45,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:45,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
26
26
26
[WARNING|trainer.py:803] 2025-04-26 14:06:47,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:47,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:47,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
27
27
27
[WARNING|trainer.py:803] 2025-04-26 14:06:50,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:50,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:50,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
28
28
28
[WARNING|trainer.py:803] 2025-04-26 14:06:52,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:52,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:06:53,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
29
29
29
[WARNING|trainer.py:803] 2025-04-26 14:06:55,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:55,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:55,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
30
30
30
[WARNING|trainer.py:803] 2025-04-26 14:06:57,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:57,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:06:57,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
31
31
31
[WARNING|trainer.py:803] 2025-04-26 14:07:00,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:07:00,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:07:00,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
32
32
32
[WARNING|trainer.py:803] 2025-04-26 14:07:02,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:03,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:03,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
33
33
33
[WARNING|trainer.py:803] 2025-04-26 14:07:05,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:05,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:05,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
34
34
34
[WARNING|trainer.py:803] 2025-04-26 14:07:07,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:07,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:08,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
35
35
35
[WARNING|trainer.py:803] 2025-04-26 14:07:10,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:10,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:10,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
36
36
36
[WARNING|trainer.py:803] 2025-04-26 14:07:13,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:13,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:13,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
37
37
37
[WARNING|trainer.py:803] 2025-04-26 14:07:15,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:15,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:15,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
38
38
38
[WARNING|trainer.py:803] 2025-04-26 14:07:18,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:18,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:18,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
39
39
39
[WARNING|trainer.py:803] 2025-04-26 14:07:20,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:20,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:20,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
40
40
40
[WARNING|trainer.py:803] 2025-04-26 14:07:22,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:23,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:23,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
41
41
41
[WARNING|trainer.py:803] 2025-04-26 14:07:25,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:25,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:25,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
42
42
42
[WARNING|trainer.py:803] 2025-04-26 14:07:27,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:07:28,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:07:28,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
43
43
43
[WARNING|trainer.py:803] 2025-04-26 14:07:30,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:30,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:30,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
44
44
44
[WARNING|trainer.py:803] 2025-04-26 14:07:33,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:33,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:33,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
45
45
45
[WARNING|trainer.py:803] 2025-04-26 14:07:35,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:35,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:36,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
46
46
46
[WARNING|trainer.py:803] 2025-04-26 14:07:38,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:38,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:38,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
47
47
47
[WARNING|trainer.py:803] 2025-04-26 14:07:41,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:41,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:41,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
48
48
48
[WARNING|trainer.py:803] 2025-04-26 14:07:43,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:43,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:43,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
49
49
49
[WARNING|trainer.py:803] 2025-04-26 14:07:46,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:46,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:46,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
50
50
50
[WARNING|trainer.py:803] 2025-04-26 14:07:49,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:49,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:07:49,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
51
51
51
[WARNING|trainer.py:803] 2025-04-26 14:07:51,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:51,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:52,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
52
52
52
[WARNING|trainer.py:803] 2025-04-26 14:07:54,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:54,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:54,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
53
53
53
[WARNING|trainer.py:803] 2025-04-26 14:07:57,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:57,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:07:57,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
54
54
54
[WARNING|trainer.py:803] 2025-04-26 14:07:59,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:59,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:07:59,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
55
55
55
[WARNING|trainer.py:803] 2025-04-26 14:08:02,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:02,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:02,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
56
56
56
[WARNING|trainer.py:803] 2025-04-26 14:08:04,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:04,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:04,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
57
57
57
[WARNING|trainer.py:803] 2025-04-26 14:08:06,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:06,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:06,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
58
58
58
[WARNING|trainer.py:803] 2025-04-26 14:08:08,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:09,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:09,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
59
59
59
[WARNING|trainer.py:803] 2025-04-26 14:08:11,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:11,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:11,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
60
60
60
[WARNING|trainer.py:803] 2025-04-26 14:08:13,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:14,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:14,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
61
61
61
[WARNING|trainer.py:803] 2025-04-26 14:08:16,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:16,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:16,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
62
62
62
[WARNING|trainer.py:803] 2025-04-26 14:08:19,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:19,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:19,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
63
63
63
[WARNING|trainer.py:803] 2025-04-26 14:08:21,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:21,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:21,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
64
64
64
[WARNING|trainer.py:803] 2025-04-26 14:08:23,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:23,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:23,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
65
65
65
[WARNING|trainer.py:803] 2025-04-26 14:08:26,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:26,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:26,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
66
66
66
[WARNING|trainer.py:803] 2025-04-26 14:08:28,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:28,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:28,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
67
67
67
[WARNING|trainer.py:803] 2025-04-26 14:08:31,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:31,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:31,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
68
68
68
[WARNING|trainer.py:803] 2025-04-26 14:08:33,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:34,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:34,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
69
69
69
[WARNING|trainer.py:803] 2025-04-26 14:08:36,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:36,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:36,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
70
70
70
[WARNING|trainer.py:803] 2025-04-26 14:08:39,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:39,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:08:39,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
71
71
71
[WARNING|trainer.py:803] 2025-04-26 14:08:41,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:42,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:08:42,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
72
72
72
[WARNING|trainer.py:803] 2025-04-26 14:08:44,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:44,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:44,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
73
73
73
[WARNING|trainer.py:803] 2025-04-26 14:08:47,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:08:47,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:08:47,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
74
74
74
[WARNING|trainer.py:803] 2025-04-26 14:08:49,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:49,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:49,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
75
75
75
[WARNING|trainer.py:803] 2025-04-26 14:08:52,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:53,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:53,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
76
76
76
[WARNING|trainer.py:803] 2025-04-26 14:08:55,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:08:55,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:08:55,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
77
77
77
[WARNING|trainer.py:803] 2025-04-26 14:08:58,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:58,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:08:58,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
78
78
78
[WARNING|trainer.py:803] 2025-04-26 14:09:00,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:00,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:00,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
79
79
79
[WARNING|trainer.py:803] 2025-04-26 14:09:02,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:03,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:03,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
80
80
80
[WARNING|trainer.py:803] 2025-04-26 14:09:05,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:05,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:05,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
81
81
81
[WARNING|trainer.py:803] 2025-04-26 14:09:08,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:08,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:08,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
82
82
82
[WARNING|trainer.py:803] 2025-04-26 14:09:10,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:10,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:10,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
83
83
83
[WARNING|trainer.py:803] 2025-04-26 14:09:13,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:13,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:13,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
84
84
84
[WARNING|trainer.py:803] 2025-04-26 14:09:16,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:16,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:16,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
85
85
85
[WARNING|trainer.py:803] 2025-04-26 14:09:19,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:19,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:19,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
86
86
86
[WARNING|trainer.py:803] 2025-04-26 14:09:21,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:21,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:21,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
87
87
87
[WARNING|trainer.py:803] 2025-04-26 14:09:24,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:24,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:24,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
88
88
88
[WARNING|trainer.py:803] 2025-04-26 14:09:26,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:26,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:26,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
89
89
89
[WARNING|trainer.py:803] 2025-04-26 14:09:28,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:28,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:28,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
90
90
90
[WARNING|trainer.py:803] 2025-04-26 14:09:30,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:30,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:31,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
91
91
91
[WARNING|trainer.py:803] 2025-04-26 14:09:33,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:09:33,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:09:33,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
92
92
92
[WARNING|trainer.py:803] 2025-04-26 14:09:36,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:36,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:09:36,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
93
93
93
[WARNING|trainer.py:803] 2025-04-26 14:09:38,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:39,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:39,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
94
94
94
[WARNING|trainer.py:803] 2025-04-26 14:09:41,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:09:41,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:09:41,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
95
95
95
[WARNING|trainer.py:803] 2025-04-26 14:09:43,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:43,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:44,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
96
96
96
[WARNING|trainer.py:803] 2025-04-26 14:09:46,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:46,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:46,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
97
97
97
[WARNING|trainer.py:803] 2025-04-26 14:09:48,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:49,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:49,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
98
98
98
[WARNING|trainer.py:803] 2025-04-26 14:09:51,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:51,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:52,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
99
99
99
[WARNING|trainer.py:803] 2025-04-26 14:09:53,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:54,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:54,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
100
100
100
[WARNING|trainer.py:803] 2025-04-26 14:09:56,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:56,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:09:56,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
101
101
101
[WARNING|trainer.py:803] 2025-04-26 14:09:59,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:59,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:09:59,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
102
102
102
[WARNING|trainer.py:803] 2025-04-26 14:10:01,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:02,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:02,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
103
103
103
[WARNING|trainer.py:803] 2025-04-26 14:10:04,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:04,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:05,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
104
104
104
[WARNING|trainer.py:803] 2025-04-26 14:10:07,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:07,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:07,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
105
105
105
[WARNING|trainer.py:803] 2025-04-26 14:10:10,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:10,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:10,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
106
106
106
[WARNING|trainer.py:803] 2025-04-26 14:10:12,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:13,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:13,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
107
107
107
[WARNING|trainer.py:803] 2025-04-26 14:10:15,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:16,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:16,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
108
108
108
[WARNING|trainer.py:803] 2025-04-26 14:10:18,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:18,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:18,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
109
109
109
[WARNING|trainer.py:803] 2025-04-26 14:10:20,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:20,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:20,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
110
110
110
[WARNING|trainer.py:803] 2025-04-26 14:10:23,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:24,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:24,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
111
111
111
[WARNING|trainer.py:803] 2025-04-26 14:10:26,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:26,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:26,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
112
112
112
[WARNING|trainer.py:803] 2025-04-26 14:10:28,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:29,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:29,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
113
113
113
[WARNING|trainer.py:803] 2025-04-26 14:10:31,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:31,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:31,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
114
114
114
[WARNING|trainer.py:803] 2025-04-26 14:10:33,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:10:34,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:10:34,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
115
115
115
[WARNING|trainer.py:803] 2025-04-26 14:10:36,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:36,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:36,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
116
116
116
[WARNING|trainer.py:803] 2025-04-26 14:10:39,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:39,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:39,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
117
117
117
[WARNING|trainer.py:803] 2025-04-26 14:10:41,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:41,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:41,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
118
118
118
[WARNING|trainer.py:803] 2025-04-26 14:10:44,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:45,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:45,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
119
119
119
[WARNING|trainer.py:803] 2025-04-26 14:10:47,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:47,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:47,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
120
120
120
[WARNING|trainer.py:803] 2025-04-26 14:10:50,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:50,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:10:50,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
121
121
121
[WARNING|trainer.py:803] 2025-04-26 14:10:53,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:53,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:10:53,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
122
122
122
[WARNING|trainer.py:803] 2025-04-26 14:10:56,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:56,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:10:56,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
123
123
123
[WARNING|trainer.py:803] 2025-04-26 14:10:59,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:10:59,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:10:59,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
124
124
124
[WARNING|trainer.py:803] 2025-04-26 14:11:02,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:02,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:02,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
125
125
125
[WARNING|trainer.py:803] 2025-04-26 14:11:04,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:04,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:04,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
126
126
126
[WARNING|trainer.py:803] 2025-04-26 14:11:07,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:07,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:07,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
127
127
127
[WARNING|trainer.py:803] 2025-04-26 14:11:10,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:10,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:10,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
128
128
128
[WARNING|trainer.py:803] 2025-04-26 14:11:12,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:12,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:12,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
129
129
129
[WARNING|trainer.py:803] 2025-04-26 14:11:15,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:15,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:15,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
130
130
130
[WARNING|trainer.py:803] 2025-04-26 14:11:17,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:17,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:17,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
131
131
131
[WARNING|trainer.py:803] 2025-04-26 14:11:20,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:20,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:20,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
132
132
132
[WARNING|trainer.py:803] 2025-04-26 14:11:22,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:22,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:23,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
133
133
133
[WARNING|trainer.py:803] 2025-04-26 14:11:25,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:25,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:25,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
134
134
134
[WARNING|trainer.py:803] 2025-04-26 14:11:28,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:28,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:28,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
135
135
135
[WARNING|trainer.py:803] 2025-04-26 14:11:30,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:11:30,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:11:30,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
136
136
136
[WARNING|trainer.py:803] 2025-04-26 14:11:33,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:11:33,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:11:33,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
137
137
137
[WARNING|trainer.py:803] 2025-04-26 14:11:36,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:36,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:11:36,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
138
138
138
[WARNING|trainer.py:803] 2025-04-26 14:11:39,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:39,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:39,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
139
139
139
[WARNING|trainer.py:803] 2025-04-26 14:11:41,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:41,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:41,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
140
140
140
[WARNING|trainer.py:803] 2025-04-26 14:11:44,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:44,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:44,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
141
141
141
[WARNING|trainer.py:803] 2025-04-26 14:11:46,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:46,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:46,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
142
142
142
[WARNING|trainer.py:803] 2025-04-26 14:11:49,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:49,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:11:49,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
143
143
143
[WARNING|trainer.py:803] 2025-04-26 14:11:52,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:52,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:52,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
144
144
144
[WARNING|trainer.py:803] 2025-04-26 14:11:54,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:54,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:55,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
145
145
145
[WARNING|trainer.py:803] 2025-04-26 14:11:58,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:58,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:11:58,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
146
146
146
[WARNING|trainer.py:803] 2025-04-26 14:12:00,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:00,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:00,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
147
147
147
[WARNING|trainer.py:803] 2025-04-26 14:12:03,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:12:03,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:12:03,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
148
148
148
[WARNING|trainer.py:803] 2025-04-26 14:12:06,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:06,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:06,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
149
149
149
[WARNING|trainer.py:803] 2025-04-26 14:12:09,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:09,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:09,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
150
150
150
[WARNING|trainer.py:803] 2025-04-26 14:12:11,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:11,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:11,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
151
151
151
[WARNING|trainer.py:803] 2025-04-26 14:12:14,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:14,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:14,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
152
152
152
[WARNING|trainer.py:803] 2025-04-26 14:12:17,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:17,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:17,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
153
153
153
[WARNING|trainer.py:803] 2025-04-26 14:12:20,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:20,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:20,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
154
154
154
[WARNING|trainer.py:803] 2025-04-26 14:12:22,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:22,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:22,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
155
155
155
[WARNING|trainer.py:803] 2025-04-26 14:12:25,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:25,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:25,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
156
156
156
[WARNING|trainer.py:803] 2025-04-26 14:12:27,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:27,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:27,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
157
157
157
[WARNING|trainer.py:803] 2025-04-26 14:12:30,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:30,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:30,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
158
158
158
[WARNING|trainer.py:803] 2025-04-26 14:12:33,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:33,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:33,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
159
159
159
[WARNING|trainer.py:803] 2025-04-26 14:12:35,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:35,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:35,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
160
160
160
[WARNING|trainer.py:803] 2025-04-26 14:12:38,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:38,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:38,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
161
161
161
[WARNING|trainer.py:803] 2025-04-26 14:12:40,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:40,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:41,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
162
162
162
[WARNING|trainer.py:803] 2025-04-26 14:12:43,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:43,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:43,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
163
163
163
[WARNING|trainer.py:803] 2025-04-26 14:12:46,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:46,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:46,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
164
164
164
[WARNING|trainer.py:803] 2025-04-26 14:12:48,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:48,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:49,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
165
165
165
[WARNING|trainer.py:803] 2025-04-26 14:12:51,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:51,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:12:51,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
166
166
166
[WARNING|trainer.py:803] 2025-04-26 14:12:53,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:54,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:54,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
167
167
167
[WARNING|trainer.py:803] 2025-04-26 14:12:56,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:56,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:12:57,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
168
168
168
[WARNING|trainer.py:803] 2025-04-26 14:12:59,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:59,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:12:59,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
169
169
169
[WARNING|trainer.py:803] 2025-04-26 14:13:03,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:03,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:03,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
170
170
170
[WARNING|trainer.py:803] 2025-04-26 14:13:05,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:05,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:05,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
171
171
171
[WARNING|trainer.py:803] 2025-04-26 14:13:07,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:07,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:08,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
172
172
172
[WARNING|trainer.py:803] 2025-04-26 14:13:10,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:10,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:10,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
173
173
173
[WARNING|trainer.py:803] 2025-04-26 14:13:12,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:12,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:12,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
174
174
174
[WARNING|trainer.py:803] 2025-04-26 14:13:15,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:15,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:15,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
175
175
175
[WARNING|trainer.py:803] 2025-04-26 14:13:17,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:17,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:17,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
176
176
176
[WARNING|trainer.py:803] 2025-04-26 14:13:20,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:20,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:20,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
177
177
177
[WARNING|trainer.py:803] 2025-04-26 14:13:22,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:22,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:23,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
178
178
178
[WARNING|trainer.py:803] 2025-04-26 14:13:25,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:25,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:26,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
179
179
179
[WARNING|trainer.py:803] 2025-04-26 14:13:28,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:28,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:29,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
180
180
180
[WARNING|trainer.py:803] 2025-04-26 14:13:31,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:31,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:32,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
181
181
181
[WARNING|trainer.py:803] 2025-04-26 14:13:33,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:34,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:34,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
182
182
182
[WARNING|trainer.py:803] 2025-04-26 14:13:36,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:36,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:37,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
183
183
183
[WARNING|trainer.py:803] 2025-04-26 14:13:39,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:39,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:13:39,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
184
184
184
[WARNING|trainer.py:803] 2025-04-26 14:13:41,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:42,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:42,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
185
185
185
[WARNING|trainer.py:803] 2025-04-26 14:13:44,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:44,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:45,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
186
186
186
[WARNING|trainer.py:803] 2025-04-26 14:13:47,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:47,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:13:47,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
187
187
187
[WARNING|trainer.py:803] 2025-04-26 14:13:49,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:50,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:13:50,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
188
188
188
[WARNING|trainer.py:803] 2025-04-26 14:13:53,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:53,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:53,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
189
189
189
[WARNING|trainer.py:803] 2025-04-26 14:13:55,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:55,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:56,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
190
190
190
[WARNING|trainer.py:803] 2025-04-26 14:13:58,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:58,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:13:58,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
191
191
191
[WARNING|trainer.py:803] 2025-04-26 14:14:00,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:01,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:01,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
192
192
192
[WARNING|trainer.py:803] 2025-04-26 14:14:03,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:03,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:04,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
193
193
193
[WARNING|trainer.py:803] 2025-04-26 14:14:06,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:06,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:06,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
194
194
194
[WARNING|trainer.py:803] 2025-04-26 14:14:09,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:09,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:09,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
195
195
195
[WARNING|trainer.py:803] 2025-04-26 14:14:11,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:11,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:11,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
196
196
196
[WARNING|trainer.py:803] 2025-04-26 14:14:14,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:14,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:14,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
197
197
197
[WARNING|trainer.py:803] 2025-04-26 14:14:16,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:16,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:17,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
198
198
198
[WARNING|trainer.py:803] 2025-04-26 14:14:19,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:19,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:19,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
199
199
199
[WARNING|trainer.py:803] 2025-04-26 14:14:21,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:21,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:21,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
200
200
200
[WARNING|trainer.py:803] 2025-04-26 14:14:24,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:24,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:24,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
201
201
201
[WARNING|trainer.py:803] 2025-04-26 14:14:27,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:27,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:27,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
202
202
202
[WARNING|trainer.py:803] 2025-04-26 14:14:30,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:30,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:30,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
203
203
203
[WARNING|trainer.py:803] 2025-04-26 14:14:32,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:32,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:32,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
204
204
204
[WARNING|trainer.py:803] 2025-04-26 14:14:35,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:35,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:35,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
205
205
205
[WARNING|trainer.py:803] 2025-04-26 14:14:37,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:37,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:38,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
206
206
206
[WARNING|trainer.py:803] 2025-04-26 14:14:40,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:40,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:40,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
207
207
207
[WARNING|trainer.py:803] 2025-04-26 14:14:43,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:43,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:43,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
208
208
208
[WARNING|trainer.py:803] 2025-04-26 14:14:45,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:46,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:14:46,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
209
209
209
[WARNING|trainer.py:803] 2025-04-26 14:14:48,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:48,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:48,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
210
210
210
[WARNING|trainer.py:803] 2025-04-26 14:14:50,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:50,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:50,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
211
211
211
[WARNING|trainer.py:803] 2025-04-26 14:14:52,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:52,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:52,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
212
212
212
[WARNING|trainer.py:803] 2025-04-26 14:14:54,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:54,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:14:54,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
213
213
213
[WARNING|trainer.py:803] 2025-04-26 14:14:56,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:56,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:14:56,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
214
214
214
[WARNING|trainer.py:803] 2025-04-26 14:14:58,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:58,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:14:58,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
215
215
215
[WARNING|trainer.py:803] 2025-04-26 14:15:00,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:15:00,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:15:00,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
216
216
216
[WARNING|trainer.py:803] 2025-04-26 14:15:02,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:02,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:02,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
217
217
217
[WARNING|trainer.py:803] 2025-04-26 14:15:04,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:04,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:04,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
218
218
218
[WARNING|trainer.py:803] 2025-04-26 14:15:06,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:06,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:06,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
219
219
219
[WARNING|trainer.py:803] 2025-04-26 14:15:08,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:08,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:09,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
220
220
220
[WARNING|trainer.py:803] 2025-04-26 14:15:10,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:10,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:11,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
221
221
221
[WARNING|trainer.py:803] 2025-04-26 14:15:12,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:12,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:13,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
222
222
222
[WARNING|trainer.py:803] 2025-04-26 14:15:14,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:14,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:15,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
223
223
223
[WARNING|trainer.py:803] 2025-04-26 14:15:16,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:17,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:17,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
224
224
224
[WARNING|trainer.py:803] 2025-04-26 14:15:18,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:19,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:19,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
225
225
225
[WARNING|trainer.py:803] 2025-04-26 14:15:20,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:21,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:21,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
226
226
226
[WARNING|trainer.py:803] 2025-04-26 14:15:22,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:23,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:23,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
227
227
227
[WARNING|trainer.py:803] 2025-04-26 14:15:24,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:25,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:25,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
228
228
228
[WARNING|trainer.py:803] 2025-04-26 14:15:26,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:27,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:27,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
229
229
229
[WARNING|trainer.py:803] 2025-04-26 14:15:28,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:29,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:29,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
230
230
230
[WARNING|trainer.py:803] 2025-04-26 14:15:30,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:31,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:31,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
231
231
231
[WARNING|trainer.py:803] 2025-04-26 14:15:32,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:33,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:33,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
232
232
232
[WARNING|trainer.py:803] 2025-04-26 14:15:34,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:35,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:35,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
233
233
233
[WARNING|trainer.py:803] 2025-04-26 14:15:36,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:37,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:37,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
234
234
[WARNING|trainer.py:803] 2025-04-26 14:15:38,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
234
[WARNING|trainer.py:803] 2025-04-26 14:15:39,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:39,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
235
[WARNING|trainer.py:803] 2025-04-26 14:15:40,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
235
235
[WARNING|trainer.py:803] 2025-04-26 14:15:41,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:41,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
236
[WARNING|trainer.py:803] 2025-04-26 14:15:42,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
236
236
[WARNING|trainer.py:803] 2025-04-26 14:15:43,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:43,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
237
[WARNING|trainer.py:803] 2025-04-26 14:15:44,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
237
237
[WARNING|trainer.py:803] 2025-04-26 14:15:45,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:45,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
238
[WARNING|trainer.py:803] 2025-04-26 14:15:46,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
238
238
[WARNING|trainer.py:803] 2025-04-26 14:15:47,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:47,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
239
[WARNING|trainer.py:803] 2025-04-26 14:15:48,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
239
239
[WARNING|trainer.py:803] 2025-04-26 14:15:49,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:49,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
240
[WARNING|trainer.py:803] 2025-04-26 14:15:50,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
240
240
[WARNING|trainer.py:803] 2025-04-26 14:15:51,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:15:51,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
241
[WARNING|trainer.py:803] 2025-04-26 14:15:52,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
241
241
[WARNING|trainer.py:803] 2025-04-26 14:15:53,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:53,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
242
[WARNING|trainer.py:803] 2025-04-26 14:15:54,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
242
242
[WARNING|trainer.py:803] 2025-04-26 14:15:55,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
243
[WARNING|trainer.py:803] 2025-04-26 14:15:55,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:15:56,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
243
243
244
[WARNING|trainer.py:803] 2025-04-26 14:15:57,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:57,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:15:58,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
244
244
245
[WARNING|trainer.py:803] 2025-04-26 14:15:59,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:15:59,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:00,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
245
245
246
[WARNING|trainer.py:803] 2025-04-26 14:16:01,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:01,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:02,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
246
246
247
[WARNING|trainer.py:803] 2025-04-26 14:16:03,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:03,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:03,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
247
247
248
[WARNING|trainer.py:803] 2025-04-26 14:16:05,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:05,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:05,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
248
248
249
[WARNING|trainer.py:803] 2025-04-26 14:16:07,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:07,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:07,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
249
249
[WARNING|trainer.py:803] 2025-04-26 14:16:09,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
250
[WARNING|trainer.py:803] 2025-04-26 14:16:09,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:10,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
250
250
251
[WARNING|trainer.py:803] 2025-04-26 14:16:11,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:11,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:12,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
251
251
252
[WARNING|trainer.py:803] 2025-04-26 14:16:13,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:13,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:14,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
252
252
253
[WARNING|trainer.py:803] 2025-04-26 14:16:15,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:15,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:15,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
253
253
254
[WARNING|trainer.py:803] 2025-04-26 14:16:17,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:17,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:17,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
254
254
255
[WARNING|trainer.py:803] 2025-04-26 14:16:19,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:19,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:19,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
255
255
256
[WARNING|trainer.py:803] 2025-04-26 14:16:21,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:21,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:22,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
256
256
257
[WARNING|trainer.py:803] 2025-04-26 14:16:23,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:24,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:24,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
257
257
[WARNING|trainer.py:803] 2025-04-26 14:16:26,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
258
[WARNING|trainer.py:803] 2025-04-26 14:16:26,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:27,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
258
258
259
[WARNING|trainer.py:803] 2025-04-26 14:16:28,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:28,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:29,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
259
260
259
[WARNING|trainer.py:803] 2025-04-26 14:16:31,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:31,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:31,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
260
260
261
[WARNING|trainer.py:803] 2025-04-26 14:16:33,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:33,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:33,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
261
262
261
[WARNING|trainer.py:803] 2025-04-26 14:16:35,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:16:35,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:35,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
263
262
262
[WARNING|trainer.py:803] 2025-04-26 14:16:37,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:37,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:37,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
263
264
263
[WARNING|trainer.py:803] 2025-04-26 14:16:39,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:39,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:39,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
264
264
265
[WARNING|trainer.py:803] 2025-04-26 14:16:41,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:41,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:42,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
265
265
266
[WARNING|trainer.py:803] 2025-04-26 14:16:43,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:44,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:44,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
266
266
267
[WARNING|trainer.py:803] 2025-04-26 14:16:46,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:46,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:46,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
267
268
267
[WARNING|trainer.py:803] 2025-04-26 14:16:48,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:48,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:48,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
268
268
269
[WARNING|trainer.py:803] 2025-04-26 14:16:50,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:51,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:51,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
269
269
270
[WARNING|trainer.py:803] 2025-04-26 14:16:53,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:53,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:16:53,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
270
271
270
[WARNING|trainer.py:803] 2025-04-26 14:16:55,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:16:55,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:55,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
271
271
272
[WARNING|trainer.py:803] 2025-04-26 14:16:58,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:58,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:16:58,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
272
273
272
[WARNING|trainer.py:803] 2025-04-26 14:17:00,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:00,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:00,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
273
273
274
[WARNING|trainer.py:803] 2025-04-26 14:17:03,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:03,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:03,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
274
274
275
[WARNING|trainer.py:803] 2025-04-26 14:17:05,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:05,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:05,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
275
276
275
[WARNING|trainer.py:803] 2025-04-26 14:17:08,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:08,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:08,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
276
277
276
[WARNING|trainer.py:803] 2025-04-26 14:17:10,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:10,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:10,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
277
278
277
[WARNING|trainer.py:803] 2025-04-26 14:17:13,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:13,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:13,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
278
279
278
[WARNING|trainer.py:803] 2025-04-26 14:17:15,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:15,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:15,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
279
280
279
[WARNING|trainer.py:803] 2025-04-26 14:17:17,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:17,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:17:18,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
280
281
280
[WARNING|trainer.py:803] 2025-04-26 14:17:20,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:17:20,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:20,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
281
282
281
[WARNING|trainer.py:803] 2025-04-26 14:17:22,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:22,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:22,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
282
283
282
[WARNING|trainer.py:803] 2025-04-26 14:17:25,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:25,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:25,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
283
284
283
[WARNING|trainer.py:803] 2025-04-26 14:17:27,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:27,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:17:27,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
284
285
284
[WARNING|trainer.py:803] 2025-04-26 14:17:30,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:17:30,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:30,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
285
285
286
[WARNING|trainer.py:803] 2025-04-26 14:17:32,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:33,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:17:33,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
286
286
[WARNING|trainer.py:803] 2025-04-26 14:17:35,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
287
[WARNING|trainer.py:803] 2025-04-26 14:17:35,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:36,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
287
288
287
[WARNING|trainer.py:803] 2025-04-26 14:17:38,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:38,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:38,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
288
289
288
[WARNING|trainer.py:803] 2025-04-26 14:17:40,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:40,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:17:41,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
289
289
290
[WARNING|trainer.py:803] 2025-04-26 14:17:43,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:17:43,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:17:43,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
290
291
290
[WARNING|trainer.py:803] 2025-04-26 14:17:45,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:46,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:46,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
291
292
291
[WARNING|trainer.py:803] 2025-04-26 14:17:48,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:48,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:48,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
292
293
292
[WARNING|trainer.py:803] 2025-04-26 14:17:50,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:50,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:51,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
293
293
294
[WARNING|trainer.py:803] 2025-04-26 14:17:53,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:53,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:53,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
294
294
295
[WARNING|trainer.py:803] 2025-04-26 14:17:56,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:56,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:56,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
296
295
295
[WARNING|trainer.py:803] 2025-04-26 14:17:58,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:17:58,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:17:59,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
296
297
296
[WARNING|trainer.py:803] 2025-04-26 14:18:00,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:01,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:01,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
297
298
297
[WARNING|trainer.py:803] 2025-04-26 14:18:03,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:03,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:03,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
298
298
299
[WARNING|trainer.py:803] 2025-04-26 14:18:05,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:06,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:06,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
299
300
299
[WARNING|trainer.py:803] 2025-04-26 14:18:08,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:08,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:08,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
301
[WARNING|trainer.py:803] 2025-04-26 14:18:09,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
300
300
302
[WARNING|trainer.py:803] 2025-04-26 14:18:10,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:11,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:11,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
301
301
303
[WARNING|trainer.py:803] 2025-04-26 14:18:12,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:12,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:12,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
302
302
304
[WARNING|trainer.py:803] 2025-04-26 14:18:13,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:13,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:13,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
303
305
303
[WARNING|trainer.py:803] 2025-04-26 14:18:14,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:15,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:15,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
304
306
304
[WARNING|trainer.py:803] 2025-04-26 14:18:16,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:16,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:16,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
305
307
305
[WARNING|trainer.py:803] 2025-04-26 14:18:17,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:18,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
306
[WARNING|trainer.py:803] 2025-04-26 14:18:18,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
308
306
[WARNING|trainer.py:803] 2025-04-26 14:18:18,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:19,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
307
[WARNING|trainer.py:803] 2025-04-26 14:18:19,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
309
307
[WARNING|trainer.py:803] 2025-04-26 14:18:20,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:20,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
308
[WARNING|trainer.py:803] 2025-04-26 14:18:20,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
310
308
[WARNING|trainer.py:803] 2025-04-26 14:18:21,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:22,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
309
[WARNING|trainer.py:803] 2025-04-26 14:18:22,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
311
309
[WARNING|trainer.py:803] 2025-04-26 14:18:22,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:23,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
310
[WARNING|trainer.py:803] 2025-04-26 14:18:23,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
312
310
[WARNING|trainer.py:803] 2025-04-26 14:18:24,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:24,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
311
[WARNING|trainer.py:803] 2025-04-26 14:18:25,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
313
311
[WARNING|trainer.py:803] 2025-04-26 14:18:25,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:26,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
312
[WARNING|trainer.py:803] 2025-04-26 14:18:26,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
314
312
[WARNING|trainer.py:803] 2025-04-26 14:18:26,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:27,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
313
[WARNING|trainer.py:803] 2025-04-26 14:18:27,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
315
313
[WARNING|trainer.py:803] 2025-04-26 14:18:28,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:18:28,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
314
[WARNING|trainer.py:803] 2025-04-26 14:18:29,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
316
[WARNING|trainer.py:803] 2025-04-26 14:18:29,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
314
[WARNING|trainer.py:803] 2025-04-26 14:18:29,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
315
[WARNING|trainer.py:803] 2025-04-26 14:18:30,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
317
[WARNING|trainer.py:803] 2025-04-26 14:18:30,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
315
[WARNING|trainer.py:803] 2025-04-26 14:18:31,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
316
[WARNING|trainer.py:803] 2025-04-26 14:18:31,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
318
[WARNING|trainer.py:803] 2025-04-26 14:18:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
316
[WARNING|trainer.py:803] 2025-04-26 14:18:32,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
317
[WARNING|trainer.py:803] 2025-04-26 14:18:33,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
319
[WARNING|trainer.py:803] 2025-04-26 14:18:33,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
317
[WARNING|trainer.py:803] 2025-04-26 14:18:34,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
318
[WARNING|trainer.py:803] 2025-04-26 14:18:34,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
320
[WARNING|trainer.py:803] 2025-04-26 14:18:34,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
318
[WARNING|trainer.py:803] 2025-04-26 14:18:35,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
319
[WARNING|trainer.py:803] 2025-04-26 14:18:35,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
321
[WARNING|trainer.py:803] 2025-04-26 14:18:36,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
319
[WARNING|trainer.py:803] 2025-04-26 14:18:36,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
320
[WARNING|trainer.py:803] 2025-04-26 14:18:37,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
322
[WARNING|trainer.py:803] 2025-04-26 14:18:37,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
320
[WARNING|trainer.py:803] 2025-04-26 14:18:38,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
321
[WARNING|trainer.py:803] 2025-04-26 14:18:38,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
323
[WARNING|trainer.py:803] 2025-04-26 14:18:38,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
321
322
[WARNING|trainer.py:803] 2025-04-26 14:18:39,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:18:39,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
324
[WARNING|trainer.py:803] 2025-04-26 14:18:40,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
322
323
[WARNING|trainer.py:803] 2025-04-26 14:18:40,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:41,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
325
[WARNING|trainer.py:803] 2025-04-26 14:18:41,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
323
[WARNING|trainer.py:803] 2025-04-26 14:18:42,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
324
[WARNING|trainer.py:803] 2025-04-26 14:18:42,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
326
[WARNING|trainer.py:803] 2025-04-26 14:18:43,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
324
[WARNING|trainer.py:803] 2025-04-26 14:18:43,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
325
[WARNING|trainer.py:803] 2025-04-26 14:18:44,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
327
[WARNING|trainer.py:803] 2025-04-26 14:18:44,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
325
[WARNING|trainer.py:803] 2025-04-26 14:18:45,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
326
[WARNING|trainer.py:803] 2025-04-26 14:18:45,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
328
[WARNING|trainer.py:803] 2025-04-26 14:18:45,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
326
327
[WARNING|trainer.py:803] 2025-04-26 14:18:46,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:46,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
329
[WARNING|trainer.py:803] 2025-04-26 14:18:47,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
327
328
[WARNING|trainer.py:803] 2025-04-26 14:18:47,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:48,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
330
[WARNING|trainer.py:803] 2025-04-26 14:18:48,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
328
329
[WARNING|trainer.py:803] 2025-04-26 14:18:49,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:49,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
331
[WARNING|trainer.py:803] 2025-04-26 14:18:49,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
329
330
[WARNING|trainer.py:803] 2025-04-26 14:18:50,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:51,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
332
[WARNING|trainer.py:803] 2025-04-26 14:18:51,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
330
331
[WARNING|trainer.py:803] 2025-04-26 14:18:52,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:52,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
333
[WARNING|trainer.py:803] 2025-04-26 14:18:52,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
331
332
[WARNING|trainer.py:803] 2025-04-26 14:18:53,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:18:53,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
334
[WARNING|trainer.py:803] 2025-04-26 14:18:54,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
332
333
[WARNING|trainer.py:803] 2025-04-26 14:18:54,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:18:55,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
335
[WARNING|trainer.py:803] 2025-04-26 14:18:55,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
333
334
[WARNING|trainer.py:803] 2025-04-26 14:18:56,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:56,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
336
[WARNING|trainer.py:803] 2025-04-26 14:18:56,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
334
335
[WARNING|trainer.py:803] 2025-04-26 14:18:57,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
337
[WARNING|trainer.py:803] 2025-04-26 14:18:58,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:18:58,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
335
336
[WARNING|trainer.py:803] 2025-04-26 14:18:58,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
338
[WARNING|trainer.py:803] 2025-04-26 14:18:59,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:18:59,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
336
337
[WARNING|trainer.py:803] 2025-04-26 14:19:00,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
339
[WARNING|trainer.py:803] 2025-04-26 14:19:00,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:00,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
337
338
[WARNING|trainer.py:803] 2025-04-26 14:19:01,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
340
[WARNING|trainer.py:803] 2025-04-26 14:19:02,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:02,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
338
339
[WARNING|trainer.py:803] 2025-04-26 14:19:02,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
341
[WARNING|trainer.py:803] 2025-04-26 14:19:03,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:03,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
339
340
[WARNING|trainer.py:803] 2025-04-26 14:19:04,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
342
[WARNING|trainer.py:803] 2025-04-26 14:19:04,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:04,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
340
341
[WARNING|trainer.py:803] 2025-04-26 14:19:05,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
343
[WARNING|trainer.py:803] 2025-04-26 14:19:06,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:06,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
341
342
[WARNING|trainer.py:803] 2025-04-26 14:19:06,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
344
[WARNING|trainer.py:803] 2025-04-26 14:19:07,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:07,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
343
[WARNING|trainer.py:803] 2025-04-26 14:19:08,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
342
345
[WARNING|trainer.py:803] 2025-04-26 14:19:09,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:09,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
344
[WARNING|trainer.py:803] 2025-04-26 14:19:09,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
343
346
[WARNING|trainer.py:803] 2025-04-26 14:19:10,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:10,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
345
344
[WARNING|trainer.py:803] 2025-04-26 14:19:11,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
347
[WARNING|trainer.py:803] 2025-04-26 14:19:11,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:11,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
346
[WARNING|trainer.py:803] 2025-04-26 14:19:12,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
345
348
[WARNING|trainer.py:803] 2025-04-26 14:19:13,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:13,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
347
[WARNING|trainer.py:803] 2025-04-26 14:19:13,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
346
349
[WARNING|trainer.py:803] 2025-04-26 14:19:14,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:14,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
348
[WARNING|trainer.py:803] 2025-04-26 14:19:15,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
347
350
[WARNING|trainer.py:803] 2025-04-26 14:19:15,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:16,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
349
[WARNING|trainer.py:803] 2025-04-26 14:19:16,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
348
351
[WARNING|trainer.py:803] 2025-04-26 14:19:17,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:17,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
350
[WARNING|trainer.py:803] 2025-04-26 14:19:17,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
349
352
[WARNING|trainer.py:803] 2025-04-26 14:19:18,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:18,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:19,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
351
350
353
[WARNING|trainer.py:803] 2025-04-26 14:19:20,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:20,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:20,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
352
351
354
[WARNING|trainer.py:803] 2025-04-26 14:19:21,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:21,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:21,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
353
352
355
[WARNING|trainer.py:803] 2025-04-26 14:19:22,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:23,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
354
[WARNING|trainer.py:803] 2025-04-26 14:19:23,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
353
356
[WARNING|trainer.py:803] 2025-04-26 14:19:24,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:24,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:24,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
355
354
357
[WARNING|trainer.py:803] 2025-04-26 14:19:25,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:25,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:26,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
356
355
358
[WARNING|trainer.py:803] 2025-04-26 14:19:26,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:27,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:27,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
357
356
359
[WARNING|trainer.py:803] 2025-04-26 14:19:28,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:28,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:28,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
358
357
360
[WARNING|trainer.py:803] 2025-04-26 14:19:29,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:29,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:29,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
359
358
361
[WARNING|trainer.py:803] 2025-04-26 14:19:30,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:19:31,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:31,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
360
362
359
[WARNING|trainer.py:803] 2025-04-26 14:19:32,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:19:32,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:32,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
361
363
360
[WARNING|trainer.py:803] 2025-04-26 14:19:33,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:33,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:33,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
362
364
361
[WARNING|trainer.py:803] 2025-04-26 14:19:34,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:35,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
363
[WARNING|trainer.py:803] 2025-04-26 14:19:35,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
365
362
[WARNING|trainer.py:803] 2025-04-26 14:19:36,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:36,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
364
[WARNING|trainer.py:803] 2025-04-26 14:19:36,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
366
363
[WARNING|trainer.py:803] 2025-04-26 14:19:37,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:37,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:19:38,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
365
367
364
[WARNING|trainer.py:803] 2025-04-26 14:19:38,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
366
[WARNING|trainer.py:803] 2025-04-26 14:19:39,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:39,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
368
365
[WARNING|trainer.py:803] 2025-04-26 14:19:40,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
367
[WARNING|trainer.py:803] 2025-04-26 14:19:40,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:40,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
369
366
[WARNING|trainer.py:803] 2025-04-26 14:19:41,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
368
[WARNING|trainer.py:803] 2025-04-26 14:19:42,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:42,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
370
367
[WARNING|trainer.py:803] 2025-04-26 14:19:42,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
369
[WARNING|trainer.py:803] 2025-04-26 14:19:43,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:43,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
368
371
[WARNING|trainer.py:803] 2025-04-26 14:19:44,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
370
[WARNING|trainer.py:803] 2025-04-26 14:19:44,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:44,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
369
372
[WARNING|trainer.py:803] 2025-04-26 14:19:45,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
371
[WARNING|trainer.py:803] 2025-04-26 14:19:46,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:46,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
370
373
[WARNING|trainer.py:803] 2025-04-26 14:19:46,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
372
[WARNING|trainer.py:803] 2025-04-26 14:19:47,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:19:47,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
371
374
[WARNING|trainer.py:803] 2025-04-26 14:19:48,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
373
[WARNING|trainer.py:803] 2025-04-26 14:19:49,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:49,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
372
[WARNING|trainer.py:803] 2025-04-26 14:19:49,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
375
374
[WARNING|trainer.py:803] 2025-04-26 14:19:50,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:50,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
373
[WARNING|trainer.py:803] 2025-04-26 14:19:51,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
376
375
[WARNING|trainer.py:803] 2025-04-26 14:19:51,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:51,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
374
377
[WARNING|trainer.py:803] 2025-04-26 14:19:52,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
376
[WARNING|trainer.py:803] 2025-04-26 14:19:53,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:53,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
378
[WARNING|trainer.py:803] 2025-04-26 14:19:53,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
375
377
[WARNING|trainer.py:803] 2025-04-26 14:19:54,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:54,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:55,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
379
376
378
[WARNING|trainer.py:803] 2025-04-26 14:19:55,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:19:56,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:56,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
380
377
379
[WARNING|trainer.py:803] 2025-04-26 14:19:57,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:57,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:19:57,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
381
378
380
[WARNING|trainer.py:803] 2025-04-26 14:19:58,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:19:58,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
382
379
[WARNING|trainer.py:803] 2025-04-26 14:19:59,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
381
[WARNING|trainer.py:803] 2025-04-26 14:19:59,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:00,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
383
380
[WARNING|trainer.py:803] 2025-04-26 14:20:00,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
382
[WARNING|trainer.py:803] 2025-04-26 14:20:01,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:01,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
384
[WARNING|trainer.py:803] 2025-04-26 14:20:02,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
381
383
[WARNING|trainer.py:803] 2025-04-26 14:20:02,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:02,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
385
[WARNING|trainer.py:803] 2025-04-26 14:20:03,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
382
384
[WARNING|trainer.py:803] 2025-04-26 14:20:04,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:04,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
386
[WARNING|trainer.py:803] 2025-04-26 14:20:04,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
383
385
[WARNING|trainer.py:803] 2025-04-26 14:20:05,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:05,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
387
[WARNING|trainer.py:803] 2025-04-26 14:20:06,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
384
386
[WARNING|trainer.py:803] 2025-04-26 14:20:06,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:07,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
388
[WARNING|trainer.py:803] 2025-04-26 14:20:07,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
385
387
[WARNING|trainer.py:803] 2025-04-26 14:20:08,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:08,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
389
[WARNING|trainer.py:803] 2025-04-26 14:20:08,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
386
388
[WARNING|trainer.py:803] 2025-04-26 14:20:09,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:09,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
390
[WARNING|trainer.py:803] 2025-04-26 14:20:10,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
387
389
[WARNING|trainer.py:803] 2025-04-26 14:20:10,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:11,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
391
[WARNING|trainer.py:803] 2025-04-26 14:20:11,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
388
390
[WARNING|trainer.py:803] 2025-04-26 14:20:12,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:12,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
392
[WARNING|trainer.py:803] 2025-04-26 14:20:12,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
389
391
[WARNING|trainer.py:803] 2025-04-26 14:20:13,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:14,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
393
[WARNING|trainer.py:803] 2025-04-26 14:20:14,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
390
392
[WARNING|trainer.py:803] 2025-04-26 14:20:14,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:15,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
394
[WARNING|trainer.py:803] 2025-04-26 14:20:15,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
391
393
[WARNING|trainer.py:803] 2025-04-26 14:20:16,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:16,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
395
[WARNING|trainer.py:803] 2025-04-26 14:20:17,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
392
394
[WARNING|trainer.py:803] 2025-04-26 14:20:17,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
396
[WARNING|trainer.py:803] 2025-04-26 14:20:18,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:18,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
393
395
[WARNING|trainer.py:803] 2025-04-26 14:20:19,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
397
[WARNING|trainer.py:803] 2025-04-26 14:20:19,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:19,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
394
396
[WARNING|trainer.py:803] 2025-04-26 14:20:20,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:20,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
398
[WARNING|trainer.py:803] 2025-04-26 14:20:21,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
395
397
[WARNING|trainer.py:803] 2025-04-26 14:20:21,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
399
[WARNING|trainer.py:803] 2025-04-26 14:20:22,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:22,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
396
398
[WARNING|trainer.py:803] 2025-04-26 14:20:23,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
400
[WARNING|trainer.py:803] 2025-04-26 14:20:23,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:23,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
397
399
[WARNING|trainer.py:803] 2025-04-26 14:20:24,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
401
[WARNING|trainer.py:803] 2025-04-26 14:20:25,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:25,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
398
400
[WARNING|trainer.py:803] 2025-04-26 14:20:25,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
402
[WARNING|trainer.py:803] 2025-04-26 14:20:26,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:26,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
399
401
[WARNING|trainer.py:803] 2025-04-26 14:20:27,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
403
[WARNING|trainer.py:803] 2025-04-26 14:20:27,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:27,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
400
402
[WARNING|trainer.py:803] 2025-04-26 14:20:28,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
404
[WARNING|trainer.py:803] 2025-04-26 14:20:29,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:29,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
401
403
[WARNING|trainer.py:803] 2025-04-26 14:20:29,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
405
[WARNING|trainer.py:803] 2025-04-26 14:20:30,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:30,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
402
404
[WARNING|trainer.py:803] 2025-04-26 14:20:31,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
406
[WARNING|trainer.py:803] 2025-04-26 14:20:31,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:32,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
403
405
[WARNING|trainer.py:803] 2025-04-26 14:20:32,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
407
[WARNING|trainer.py:803] 2025-04-26 14:20:33,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:33,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
404
406
[WARNING|trainer.py:803] 2025-04-26 14:20:34,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
408
[WARNING|trainer.py:803] 2025-04-26 14:20:34,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:34,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
405
407
[WARNING|trainer.py:803] 2025-04-26 14:20:35,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
409
[WARNING|trainer.py:803] 2025-04-26 14:20:36,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:36,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:36,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
406
408
410
[WARNING|trainer.py:803] 2025-04-26 14:20:37,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:37,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:38,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
409
407
411
[WARNING|trainer.py:803] 2025-04-26 14:20:38,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:38,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:39,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
408
410
412
[WARNING|trainer.py:803] 2025-04-26 14:20:40,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:40,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:40,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
411
409
413
[WARNING|trainer.py:803] 2025-04-26 14:20:41,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:41,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:42,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
412
410
414
[WARNING|trainer.py:803] 2025-04-26 14:20:43,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:43,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:43,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
413
411
415
[WARNING|trainer.py:803] 2025-04-26 14:20:44,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:44,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:44,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
414
412
416
[WARNING|trainer.py:803] 2025-04-26 14:20:45,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:45,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:46,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
415
413
417
[WARNING|trainer.py:803] 2025-04-26 14:20:47,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:47,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:47,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
416
414
418
[WARNING|trainer.py:803] 2025-04-26 14:20:48,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:48,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:49,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
417
415
419
[WARNING|trainer.py:803] 2025-04-26 14:20:49,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:50,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:50,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
418
416
420
[WARNING|trainer.py:803] 2025-04-26 14:20:51,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:51,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:51,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
419
417
421
[WARNING|trainer.py:803] 2025-04-26 14:20:52,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:52,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:53,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
420
418
422
[WARNING|trainer.py:803] 2025-04-26 14:20:54,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:54,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:20:54,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
421
419
423
[WARNING|trainer.py:803] 2025-04-26 14:20:55,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:55,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:20:55,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
422
420
424
[WARNING|trainer.py:803] 2025-04-26 14:20:56,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:57,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:57,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
423
421
425
[WARNING|trainer.py:803] 2025-04-26 14:20:58,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:20:58,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:20:58,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
424
422
426
[WARNING|trainer.py:803] 2025-04-26 14:20:59,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:00,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:00,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
425
423
427
[WARNING|trainer.py:803] 2025-04-26 14:21:01,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:01,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:01,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
426
424
428
[WARNING|trainer.py:803] 2025-04-26 14:21:02,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:02,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:02,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
427
429
425
[WARNING|trainer.py:803] 2025-04-26 14:21:04,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:04,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:04,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
428
430
426
[WARNING|trainer.py:803] 2025-04-26 14:21:05,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:05,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:05,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
429
431
427
[WARNING|trainer.py:803] 2025-04-26 14:21:06,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:06,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
430
[WARNING|trainer.py:803] 2025-04-26 14:21:07,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
432
428
[WARNING|trainer.py:803] 2025-04-26 14:21:08,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:08,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
431
[WARNING|trainer.py:803] 2025-04-26 14:21:08,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
433
429
[WARNING|trainer.py:803] 2025-04-26 14:21:09,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:09,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
432
[WARNING|trainer.py:803] 2025-04-26 14:21:09,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
434
430
[WARNING|trainer.py:803] 2025-04-26 14:21:10,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:10,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
433
[WARNING|trainer.py:803] 2025-04-26 14:21:11,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
435
431
[WARNING|trainer.py:803] 2025-04-26 14:21:12,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:12,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
434
[WARNING|trainer.py:803] 2025-04-26 14:21:12,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
436
432
[WARNING|trainer.py:803] 2025-04-26 14:21:13,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:13,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
435
[WARNING|trainer.py:803] 2025-04-26 14:21:14,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
437
433
[WARNING|trainer.py:803] 2025-04-26 14:21:14,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:15,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
436
[WARNING|trainer.py:803] 2025-04-26 14:21:15,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
438
434
[WARNING|trainer.py:803] 2025-04-26 14:21:16,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:16,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
437
[WARNING|trainer.py:803] 2025-04-26 14:21:16,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
439
435
[WARNING|trainer.py:803] 2025-04-26 14:21:17,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:17,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
438
[WARNING|trainer.py:803] 2025-04-26 14:21:18,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
440
436
[WARNING|trainer.py:803] 2025-04-26 14:21:19,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:19,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
439
441
[WARNING|trainer.py:803] 2025-04-26 14:21:19,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
437
[WARNING|trainer.py:803] 2025-04-26 14:21:20,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:20,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
440
[WARNING|trainer.py:803] 2025-04-26 14:21:21,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
442
438
[WARNING|trainer.py:803] 2025-04-26 14:21:21,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:21,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
441
[WARNING|trainer.py:803] 2025-04-26 14:21:22,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
443
439
[WARNING|trainer.py:803] 2025-04-26 14:21:23,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:23,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:23,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
442
444
440
[WARNING|trainer.py:803] 2025-04-26 14:21:24,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:24,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:25,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
443
445
441
[WARNING|trainer.py:803] 2025-04-26 14:21:25,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:26,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:26,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
446
444
442
[WARNING|trainer.py:803] 2025-04-26 14:21:27,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:27,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:27,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
447
445
443
[WARNING|trainer.py:803] 2025-04-26 14:21:28,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:28,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:29,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
446
448
444
[WARNING|trainer.py:803] 2025-04-26 14:21:30,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:30,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:30,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
449
447
445
[WARNING|trainer.py:803] 2025-04-26 14:21:31,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:31,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:31,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
450
448
446
[WARNING|trainer.py:803] 2025-04-26 14:21:32,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:32,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:21:33,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
451
449
447
[WARNING|trainer.py:803] 2025-04-26 14:21:34,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:34,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:34,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
452
450
448
[WARNING|trainer.py:803] 2025-04-26 14:21:35,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:35,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:36,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
453
451
449
[WARNING|trainer.py:803] 2025-04-26 14:21:36,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:37,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
454
[WARNING|trainer.py:803] 2025-04-26 14:21:37,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
452
450
[WARNING|trainer.py:803] 2025-04-26 14:21:38,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:38,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
455
[WARNING|trainer.py:803] 2025-04-26 14:21:38,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
453
451
[WARNING|trainer.py:803] 2025-04-26 14:21:39,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:39,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
456
[WARNING|trainer.py:803] 2025-04-26 14:21:40,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
454
452
[WARNING|trainer.py:803] 2025-04-26 14:21:40,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:41,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
457
[WARNING|trainer.py:803] 2025-04-26 14:21:41,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
455
453
[WARNING|trainer.py:803] 2025-04-26 14:21:42,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:42,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
458
[WARNING|trainer.py:803] 2025-04-26 14:21:42,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
456
454
[WARNING|trainer.py:803] 2025-04-26 14:21:43,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:43,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
459
[WARNING|trainer.py:803] 2025-04-26 14:21:44,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
457
455
[WARNING|trainer.py:803] 2025-04-26 14:21:44,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:45,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
460
[WARNING|trainer.py:803] 2025-04-26 14:21:45,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
458
456
[WARNING|trainer.py:803] 2025-04-26 14:21:46,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:46,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
461
[WARNING|trainer.py:803] 2025-04-26 14:21:47,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
459
457
[WARNING|trainer.py:803] 2025-04-26 14:21:47,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:48,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
462
[WARNING|trainer.py:803] 2025-04-26 14:21:48,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
460
[WARNING|trainer.py:803] 2025-04-26 14:21:49,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
458
463
[WARNING|trainer.py:803] 2025-04-26 14:21:49,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:21:49,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
461
[WARNING|trainer.py:803] 2025-04-26 14:21:50,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
459
[WARNING|trainer.py:803] 2025-04-26 14:21:50,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
464
[WARNING|trainer.py:803] 2025-04-26 14:21:51,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
462
[WARNING|trainer.py:803] 2025-04-26 14:21:51,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
460
[WARNING|trainer.py:803] 2025-04-26 14:21:52,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
465
[WARNING|trainer.py:803] 2025-04-26 14:21:52,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
463
[WARNING|trainer.py:803] 2025-04-26 14:21:53,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
461
[WARNING|trainer.py:803] 2025-04-26 14:21:53,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
466
[WARNING|trainer.py:803] 2025-04-26 14:21:54,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
464
[WARNING|trainer.py:803] 2025-04-26 14:21:54,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
462
[WARNING|trainer.py:803] 2025-04-26 14:21:55,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
467
[WARNING|trainer.py:803] 2025-04-26 14:21:55,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
465
[WARNING|trainer.py:803] 2025-04-26 14:21:55,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
463
[WARNING|trainer.py:803] 2025-04-26 14:21:56,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
468
[WARNING|trainer.py:803] 2025-04-26 14:21:56,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
466
[WARNING|trainer.py:803] 2025-04-26 14:21:57,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
464
469
[WARNING|trainer.py:803] 2025-04-26 14:21:57,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:21:58,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
467
[WARNING|trainer.py:803] 2025-04-26 14:21:58,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
465
470
[WARNING|trainer.py:803] 2025-04-26 14:21:59,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:21:59,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
468
[WARNING|trainer.py:803] 2025-04-26 14:21:59,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
466
471
[WARNING|trainer.py:803] 2025-04-26 14:22:00,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:01,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
469
[WARNING|trainer.py:803] 2025-04-26 14:22:01,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
467
472
[WARNING|trainer.py:803] 2025-04-26 14:22:01,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:02,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
470
[WARNING|trainer.py:803] 2025-04-26 14:22:02,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
468
473
[WARNING|trainer.py:803] 2025-04-26 14:22:03,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
471
[WARNING|trainer.py:803] 2025-04-26 14:22:03,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:03,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
474
469
[WARNING|trainer.py:803] 2025-04-26 14:22:04,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
472
[WARNING|trainer.py:803] 2025-04-26 14:22:05,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:05,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
475
470
[WARNING|trainer.py:803] 2025-04-26 14:22:05,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:06,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
473
[WARNING|trainer.py:803] 2025-04-26 14:22:06,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
476
471
[WARNING|trainer.py:803] 2025-04-26 14:22:07,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
474
[WARNING|trainer.py:803] 2025-04-26 14:22:07,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:07,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
477
472
[WARNING|trainer.py:803] 2025-04-26 14:22:08,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
475
[WARNING|trainer.py:803] 2025-04-26 14:22:09,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:09,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
478
473
[WARNING|trainer.py:803] 2025-04-26 14:22:09,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:10,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
476
[WARNING|trainer.py:803] 2025-04-26 14:22:10,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
479
474
[WARNING|trainer.py:803] 2025-04-26 14:22:11,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
477
[WARNING|trainer.py:803] 2025-04-26 14:22:11,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:11,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
480
475
[WARNING|trainer.py:803] 2025-04-26 14:22:12,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:13,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
478
[WARNING|trainer.py:803] 2025-04-26 14:22:13,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
481
476
[WARNING|trainer.py:803] 2025-04-26 14:22:14,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:14,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
479
[WARNING|trainer.py:803] 2025-04-26 14:22:14,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
482
477
[WARNING|trainer.py:803] 2025-04-26 14:22:15,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:15,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
480
[WARNING|trainer.py:803] 2025-04-26 14:22:16,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
483
478
[WARNING|trainer.py:803] 2025-04-26 14:22:16,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:17,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
481
[WARNING|trainer.py:803] 2025-04-26 14:22:17,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
484
479
[WARNING|trainer.py:803] 2025-04-26 14:22:18,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:18,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
482
[WARNING|trainer.py:803] 2025-04-26 14:22:18,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
485
480
[WARNING|trainer.py:803] 2025-04-26 14:22:19,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:19,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
483
[WARNING|trainer.py:803] 2025-04-26 14:22:20,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
486
481
[WARNING|trainer.py:803] 2025-04-26 14:22:20,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:21,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
484
[WARNING|trainer.py:803] 2025-04-26 14:22:21,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
487
482
[WARNING|trainer.py:803] 2025-04-26 14:22:22,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:22,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
485
[WARNING|trainer.py:803] 2025-04-26 14:22:22,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
488
483
[WARNING|trainer.py:803] 2025-04-26 14:22:23,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:23,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
486
[WARNING|trainer.py:803] 2025-04-26 14:22:24,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
489
484
[WARNING|trainer.py:803] 2025-04-26 14:22:24,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:25,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
487
[WARNING|trainer.py:803] 2025-04-26 14:22:25,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
490
485
[WARNING|trainer.py:803] 2025-04-26 14:22:26,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:26,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
488
[WARNING|trainer.py:803] 2025-04-26 14:22:27,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
491
486
[WARNING|trainer.py:803] 2025-04-26 14:22:27,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:28,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
489
[WARNING|trainer.py:803] 2025-04-26 14:22:28,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
492
487
[WARNING|trainer.py:803] 2025-04-26 14:22:29,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:29,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
490
[WARNING|trainer.py:803] 2025-04-26 14:22:29,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
493
488
[WARNING|trainer.py:803] 2025-04-26 14:22:30,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:30,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
491
[WARNING|trainer.py:803] 2025-04-26 14:22:31,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
494
489
[WARNING|trainer.py:803] 2025-04-26 14:22:31,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:32,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
492
[WARNING|trainer.py:803] 2025-04-26 14:22:32,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
495
490
[WARNING|trainer.py:803] 2025-04-26 14:22:33,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:33,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
493
[WARNING|trainer.py:803] 2025-04-26 14:22:34,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
496
491
[WARNING|trainer.py:803] 2025-04-26 14:22:34,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:35,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
494
[WARNING|trainer.py:803] 2025-04-26 14:22:35,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
497
492
[WARNING|trainer.py:803] 2025-04-26 14:22:36,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:36,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
495
[WARNING|trainer.py:803] 2025-04-26 14:22:36,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
498
493
[WARNING|trainer.py:803] 2025-04-26 14:22:37,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:37,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
496
499
[WARNING|trainer.py:803] 2025-04-26 14:22:38,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
494
[WARNING|trainer.py:803] 2025-04-26 14:22:38,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:39,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
497
500
[WARNING|trainer.py:803] 2025-04-26 14:22:39,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
495
[WARNING|trainer.py:803] 2025-04-26 14:22:40,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:40,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
498
501
[WARNING|trainer.py:803] 2025-04-26 14:22:41,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:41,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
496
[WARNING|trainer.py:803] 2025-04-26 14:22:41,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
499
502
[WARNING|trainer.py:803] 2025-04-26 14:22:42,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:43,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:43,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
497
500
503
[WARNING|trainer.py:803] 2025-04-26 14:22:43,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:44,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:44,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
498
504
501
[WARNING|trainer.py:803] 2025-04-26 14:22:45,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:22:45,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:45,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
499
505
502
[WARNING|trainer.py:803] 2025-04-26 14:22:46,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:47,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
500
[WARNING|trainer.py:803] 2025-04-26 14:22:47,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
506
503
[WARNING|trainer.py:803] 2025-04-26 14:22:48,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:48,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:48,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
501
507
504
[WARNING|trainer.py:803] 2025-04-26 14:22:49,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:49,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:49,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
502
508
505
[WARNING|trainer.py:803] 2025-04-26 14:22:50,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:50,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:51,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
503
509
506
[WARNING|trainer.py:803] 2025-04-26 14:22:52,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:52,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:52,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
504
510
507
[WARNING|trainer.py:803] 2025-04-26 14:22:53,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:53,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:53,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
505
511
508
[WARNING|trainer.py:803] 2025-04-26 14:22:54,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:22:54,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:55,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
506
512
509
[WARNING|trainer.py:803] 2025-04-26 14:22:56,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:56,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:56,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
507
513
510
[WARNING|trainer.py:803] 2025-04-26 14:22:57,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:22:57,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:57,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
508
514
511
[WARNING|trainer.py:803] 2025-04-26 14:22:58,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:58,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:22:59,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
509
515
512
[WARNING|trainer.py:803] 2025-04-26 14:23:00,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:00,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:00,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
516
510
513
[WARNING|trainer.py:803] 2025-04-26 14:23:01,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:01,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:01,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
517
511
514
[WARNING|trainer.py:803] 2025-04-26 14:23:02,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:02,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:03,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
518
512
515
[WARNING|trainer.py:803] 2025-04-26 14:23:04,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:04,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:23:04,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
519
513
516
[WARNING|trainer.py:803] 2025-04-26 14:23:05,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:05,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:05,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
520
517
514
[WARNING|trainer.py:803] 2025-04-26 14:23:07,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:07,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:07,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
521
518
515
[WARNING|trainer.py:803] 2025-04-26 14:23:08,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:08,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:08,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
522
519
516
[WARNING|trainer.py:803] 2025-04-26 14:23:09,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:09,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:09,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
523
517
520
[WARNING|trainer.py:803] 2025-04-26 14:23:11,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:11,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:11,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
518
524
521
[WARNING|trainer.py:803] 2025-04-26 14:23:12,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:12,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:12,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
519
522
525
[WARNING|trainer.py:803] 2025-04-26 14:23:13,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:13,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:14,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
520
523
526
[WARNING|trainer.py:803] 2025-04-26 14:23:15,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:15,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:15,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
524
521
527
[WARNING|trainer.py:803] 2025-04-26 14:23:16,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:16,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:16,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
525
522
528
[WARNING|trainer.py:803] 2025-04-26 14:23:17,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:17,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:18,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
526
523
529
[WARNING|trainer.py:803] 2025-04-26 14:23:19,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:19,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:19,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
527
524
530
[WARNING|trainer.py:803] 2025-04-26 14:23:20,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:20,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:20,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
528
525
531
[WARNING|trainer.py:803] 2025-04-26 14:23:21,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:22,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:22,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
529
526
532
[WARNING|trainer.py:803] 2025-04-26 14:23:23,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:23,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:23,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
530
527
533
[WARNING|trainer.py:803] 2025-04-26 14:23:24,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:24,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:24,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
531
528
534
[WARNING|trainer.py:803] 2025-04-26 14:23:25,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:26,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:26,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
532
529
535
[WARNING|trainer.py:803] 2025-04-26 14:23:27,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:27,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
533
[WARNING|trainer.py:803] 2025-04-26 14:23:27,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
530
536
[WARNING|trainer.py:803] 2025-04-26 14:23:28,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:28,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:28,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
534
NoYes
531
537
[WARNING|trainer.py:803] 2025-04-26 14:23:29,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:23:30,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
535
[WARNING|trainer.py:803] 2025-04-26 14:23:30,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
532
538
[WARNING|trainer.py:803] 2025-04-26 14:23:31,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:31,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
536
[WARNING|trainer.py:803] 2025-04-26 14:23:31,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
533
539
[WARNING|trainer.py:803] 2025-04-26 14:23:32,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:32,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
537
[WARNING|trainer.py:803] 2025-04-26 14:23:32,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
534
540
[WARNING|trainer.py:803] 2025-04-26 14:23:33,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:34,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
538
[WARNING|trainer.py:803] 2025-04-26 14:23:34,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
535
541
[WARNING|trainer.py:803] 2025-04-26 14:23:35,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
539
[WARNING|trainer.py:803] 2025-04-26 14:23:35,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:35,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
536
542
[WARNING|trainer.py:803] 2025-04-26 14:23:36,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:36,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
540
[WARNING|trainer.py:803] 2025-04-26 14:23:36,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
537
543
[WARNING|trainer.py:803] 2025-04-26 14:23:37,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:38,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
541
[WARNING|trainer.py:803] 2025-04-26 14:23:38,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
538
544
[WARNING|trainer.py:803] 2025-04-26 14:23:39,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:39,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
542
[WARNING|trainer.py:803] 2025-04-26 14:23:39,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
539
545
[WARNING|trainer.py:803] 2025-04-26 14:23:40,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:40,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
543
[WARNING|trainer.py:803] 2025-04-26 14:23:40,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
540
546
[WARNING|trainer.py:803] 2025-04-26 14:23:41,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:42,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
544
[WARNING|trainer.py:803] 2025-04-26 14:23:42,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
541
547
[WARNING|trainer.py:803] 2025-04-26 14:23:43,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
545
[WARNING|trainer.py:803] 2025-04-26 14:23:43,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:43,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
542
548
[WARNING|trainer.py:803] 2025-04-26 14:23:44,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
546
[WARNING|trainer.py:803] 2025-04-26 14:23:44,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:44,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
543
549
[WARNING|trainer.py:803] 2025-04-26 14:23:45,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
547
[WARNING|trainer.py:803] 2025-04-26 14:23:46,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
544
550
[WARNING|trainer.py:803] 2025-04-26 14:23:46,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
548
[WARNING|trainer.py:803] 2025-04-26 14:23:47,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:47,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
545
551
[WARNING|trainer.py:803] 2025-04-26 14:23:48,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
549
[WARNING|trainer.py:803] 2025-04-26 14:23:48,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:48,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
546
552
[WARNING|trainer.py:803] 2025-04-26 14:23:49,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
550
[WARNING|trainer.py:803] 2025-04-26 14:23:50,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:50,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
547
553
[WARNING|trainer.py:803] 2025-04-26 14:23:50,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
551
[WARNING|trainer.py:803] 2025-04-26 14:23:51,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:51,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
548
554
[WARNING|trainer.py:803] 2025-04-26 14:23:52,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
552
[WARNING|trainer.py:803] 2025-04-26 14:23:52,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:23:52,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
549
555
[WARNING|trainer.py:803] 2025-04-26 14:23:53,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
553
[WARNING|trainer.py:803] 2025-04-26 14:23:54,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:23:54,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
550
556
[WARNING|trainer.py:803] 2025-04-26 14:23:54,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
554
[WARNING|trainer.py:803] 2025-04-26 14:23:55,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:23:55,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
551
[WARNING|trainer.py:803] 2025-04-26 14:23:55,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
557
555
[WARNING|trainer.py:803] 2025-04-26 14:23:56,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:56,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:57,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
552
558
556
[WARNING|trainer.py:803] 2025-04-26 14:23:58,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:58,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:58,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
553
559
557
[WARNING|trainer.py:803] 2025-04-26 14:23:59,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:59,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:23:59,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
554
560
558
[WARNING|trainer.py:803] 2025-04-26 14:24:00,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:00,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:01,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
555
561
559
[WARNING|trainer.py:803] 2025-04-26 14:24:02,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:02,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:02,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
556
562
560
[WARNING|trainer.py:803] 2025-04-26 14:24:03,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:03,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:03,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
557
563
561
[WARNING|trainer.py:803] 2025-04-26 14:24:04,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:04,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:05,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
558
564
562
[WARNING|trainer.py:803] 2025-04-26 14:24:06,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:06,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:06,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
565
559
563
[WARNING|trainer.py:803] 2025-04-26 14:24:07,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:07,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:07,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
566
560
564
[WARNING|trainer.py:803] 2025-04-26 14:24:08,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:08,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:09,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
567
561
565
[WARNING|trainer.py:803] 2025-04-26 14:24:10,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:10,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:10,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
568
562
566
[WARNING|trainer.py:803] 2025-04-26 14:24:11,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:11,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:11,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
569
563
567
[WARNING|trainer.py:803] 2025-04-26 14:24:12,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:13,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:13,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
570
564
568
[WARNING|trainer.py:803] 2025-04-26 14:24:14,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:14,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:14,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
571
565
569
[WARNING|trainer.py:803] 2025-04-26 14:24:15,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:15,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:15,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
572
566
570
[WARNING|trainer.py:803] 2025-04-26 14:24:17,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:17,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:17,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
573
567
571
[WARNING|trainer.py:803] 2025-04-26 14:24:18,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:18,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:18,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
574
568
572
[WARNING|trainer.py:803] 2025-04-26 14:24:19,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:19,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:20,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
575
569
573
[WARNING|trainer.py:803] 2025-04-26 14:24:21,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:21,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:21,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
576
570
574
[WARNING|trainer.py:803] 2025-04-26 14:24:22,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:22,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:22,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
577
571
575
[WARNING|trainer.py:803] 2025-04-26 14:24:23,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:23,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:24,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
578
572
576
[WARNING|trainer.py:803] 2025-04-26 14:24:25,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:25,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:25,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
579
573
577
[WARNING|trainer.py:803] 2025-04-26 14:24:26,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:26,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:26,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
580
578
574
[WARNING|trainer.py:803] 2025-04-26 14:24:27,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:28,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:28,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
581
579
575
[WARNING|trainer.py:803] 2025-04-26 14:24:29,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:29,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:29,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
582
580
576
[WARNING|trainer.py:803] 2025-04-26 14:24:30,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:30,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:30,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
583
581
577
[WARNING|trainer.py:803] 2025-04-26 14:24:31,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:32,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
584
[WARNING|trainer.py:803] 2025-04-26 14:24:32,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
582
578
[WARNING|trainer.py:803] 2025-04-26 14:24:33,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:33,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
585
[WARNING|trainer.py:803] 2025-04-26 14:24:33,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
583
579
[WARNING|trainer.py:803] 2025-04-26 14:24:34,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:34,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:34,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
586
584
580
[WARNING|trainer.py:803] 2025-04-26 14:24:35,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:36,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:36,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
587
585
581
[WARNING|trainer.py:803] 2025-04-26 14:24:37,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:37,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:37,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
588
586
582
[WARNING|trainer.py:803] 2025-04-26 14:24:38,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:38,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:39,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
589
587
583
[WARNING|trainer.py:803] 2025-04-26 14:24:39,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:40,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:40,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
590
588
584
[WARNING|trainer.py:803] 2025-04-26 14:24:41,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:41,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:41,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
591
589
585
[WARNING|trainer.py:803] 2025-04-26 14:24:42,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:42,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:24:43,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
592
590
586
[WARNING|trainer.py:803] 2025-04-26 14:24:44,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:44,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:24:44,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
593
591
587
[WARNING|trainer.py:803] 2025-04-26 14:24:45,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:45,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
594
[WARNING|trainer.py:803] 2025-04-26 14:24:46,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
592
588
[WARNING|trainer.py:803] 2025-04-26 14:24:46,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:47,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:47,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
595
593
589
[WARNING|trainer.py:803] 2025-04-26 14:24:48,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:48,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:48,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
596
594
590
[WARNING|trainer.py:803] 2025-04-26 14:24:49,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:49,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
597
[WARNING|trainer.py:803] 2025-04-26 14:24:50,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
595
591
[WARNING|trainer.py:803] 2025-04-26 14:24:50,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:51,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
598
[WARNING|trainer.py:803] 2025-04-26 14:24:51,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
596
592
[WARNING|trainer.py:803] 2025-04-26 14:24:52,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:52,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
599
[WARNING|trainer.py:803] 2025-04-26 14:24:52,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
597
593
[WARNING|trainer.py:803] 2025-04-26 14:24:53,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:53,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
600
[WARNING|trainer.py:803] 2025-04-26 14:24:54,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
598
594
[WARNING|trainer.py:803] 2025-04-26 14:24:54,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:55,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:55,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
599
595
601
[WARNING|trainer.py:803] 2025-04-26 14:24:56,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:56,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
600
[WARNING|trainer.py:803] 2025-04-26 14:24:57,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
596
[WARNING|trainer.py:803] 2025-04-26 14:24:57,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:24:58,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
602
597
601
[WARNING|trainer.py:803] 2025-04-26 14:24:59,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:24:59,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
598
[WARNING|trainer.py:803] 2025-04-26 14:25:00,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
603
[WARNING|trainer.py:803] 2025-04-26 14:25:01,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
599
602
[WARNING|trainer.py:803] 2025-04-26 14:25:01,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:02,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:02,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
600
604
[WARNING|trainer.py:803] 2025-04-26 14:25:03,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
603
[WARNING|trainer.py:803] 2025-04-26 14:25:04,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:04,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
601
605
[WARNING|trainer.py:803] 2025-04-26 14:25:06,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
604
[WARNING|trainer.py:803] 2025-04-26 14:25:06,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:07,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
602
606
[WARNING|trainer.py:803] 2025-04-26 14:25:08,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:25:08,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
605
[WARNING|trainer.py:803] 2025-04-26 14:25:09,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
603
607
[WARNING|trainer.py:803] 2025-04-26 14:25:10,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
606
[WARNING|trainer.py:803] 2025-04-26 14:25:10,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:25:11,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
604
608
[WARNING|trainer.py:803] 2025-04-26 14:25:12,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
607
[WARNING|trainer.py:803] 2025-04-26 14:25:13,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:13,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
605
609
[WARNING|trainer.py:803] 2025-04-26 14:25:15,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
608
[WARNING|trainer.py:803] 2025-04-26 14:25:15,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:25:16,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
606
610
[WARNING|trainer.py:803] 2025-04-26 14:25:17,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:17,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
609
607
[WARNING|trainer.py:803] 2025-04-26 14:25:18,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
611
[WARNING|trainer.py:803] 2025-04-26 14:25:19,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:25:19,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
610
[WARNING|trainer.py:803] 2025-04-26 14:25:20,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
608
612
[WARNING|trainer.py:803] 2025-04-26 14:25:22,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
611
[WARNING|trainer.py:803] 2025-04-26 14:25:22,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:23,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
609
613
[WARNING|trainer.py:803] 2025-04-26 14:25:24,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
612
[WARNING|trainer.py:803] 2025-04-26 14:25:25,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
610
[WARNING|trainer.py:803] 2025-04-26 14:25:25,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:26,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
614
613
[WARNING|trainer.py:803] 2025-04-26 14:25:27,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
611
[WARNING|trainer.py:803] 2025-04-26 14:25:28,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
615
[WARNING|trainer.py:803] 2025-04-26 14:25:28,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:25:29,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
614
[WARNING|trainer.py:803] 2025-04-26 14:25:30,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
612
616
[WARNING|trainer.py:803] 2025-04-26 14:25:31,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:31,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
615
[WARNING|trainer.py:803] 2025-04-26 14:25:32,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
617
613
[WARNING|trainer.py:803] 2025-04-26 14:25:33,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
616
[WARNING|trainer.py:803] 2025-04-26 14:25:33,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:34,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
618
614
[WARNING|trainer.py:803] 2025-04-26 14:25:36,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
617
[WARNING|trainer.py:803] 2025-04-26 14:25:36,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:36,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
615
619
[WARNING|trainer.py:803] 2025-04-26 14:25:38,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:25:38,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
618
[WARNING|trainer.py:803] 2025-04-26 14:25:39,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
620
616
[WARNING|trainer.py:803] 2025-04-26 14:25:40,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:25:40,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
619
[WARNING|trainer.py:803] 2025-04-26 14:25:41,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
621
617
[WARNING|trainer.py:803] 2025-04-26 14:25:42,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:25:42,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
620
[WARNING|trainer.py:803] 2025-04-26 14:25:43,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
618
622
621
[WARNING|trainer.py:803] 2025-04-26 14:25:44,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:44,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:45,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
619
623
622
[WARNING|trainer.py:803] 2025-04-26 14:25:47,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:25:47,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:48,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
620
624
[WARNING|trainer.py:803] 2025-04-26 14:25:49,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
623
[WARNING|trainer.py:803] 2025-04-26 14:25:49,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:50,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
621
625
[WARNING|trainer.py:803] 2025-04-26 14:25:51,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
624
[WARNING|trainer.py:803] 2025-04-26 14:25:52,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:53,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
622
626
[WARNING|trainer.py:803] 2025-04-26 14:25:54,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
625
[WARNING|trainer.py:803] 2025-04-26 14:25:54,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:25:55,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
623
627
[WARNING|trainer.py:803] 2025-04-26 14:25:56,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
626
[WARNING|trainer.py:803] 2025-04-26 14:25:57,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:25:57,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
624
628
[WARNING|trainer.py:803] 2025-04-26 14:25:59,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
627
[WARNING|trainer.py:803] 2025-04-26 14:25:59,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:00,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
629
625
628
[WARNING|trainer.py:803] 2025-04-26 14:26:01,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:01,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:02,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
626
630
629
[WARNING|trainer.py:803] 2025-04-26 14:26:03,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:04,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:04,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
631
627
630
[WARNING|trainer.py:803] 2025-04-26 14:26:06,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:06,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:06,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
632
628
631
[WARNING|trainer.py:803] 2025-04-26 14:26:08,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:08,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:08,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
629
633
632
[WARNING|trainer.py:803] 2025-04-26 14:26:10,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:10,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:11,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
633
630
634
[WARNING|trainer.py:803] 2025-04-26 14:26:13,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:13,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:13,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
631
635
634
[WARNING|trainer.py:803] 2025-04-26 14:26:15,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:15,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:16,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
632
636
635
[WARNING|trainer.py:803] 2025-04-26 14:26:17,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:17,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:18,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
633
637
636
[WARNING|trainer.py:803] 2025-04-26 14:26:20,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:20,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:20,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
638
637
634
[WARNING|trainer.py:803] 2025-04-26 14:26:22,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:22,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:22,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
639
638
635
[WARNING|trainer.py:803] 2025-04-26 14:26:24,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:24,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:25,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
640
639
636
[WARNING|trainer.py:803] 2025-04-26 14:26:26,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:27,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:27,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
641
640
637
[WARNING|trainer.py:803] 2025-04-26 14:26:29,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:29,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:29,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
642
641
638
[WARNING|trainer.py:803] 2025-04-26 14:26:31,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:31,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:31,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
642
643
639
[WARNING|trainer.py:803] 2025-04-26 14:26:34,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:34,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:26:34,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
640
643
644
[WARNING|trainer.py:803] 2025-04-26 14:26:36,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:36,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:26:36,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
641
644
645
[WARNING|trainer.py:803] 2025-04-26 14:26:38,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:39,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:39,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
642
646
645
[WARNING|trainer.py:803] 2025-04-26 14:26:40,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:41,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:41,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
643
647
646
[WARNING|trainer.py:803] 2025-04-26 14:26:43,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:26:43,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:43,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
648
644
647
[WARNING|trainer.py:803] 2025-04-26 14:26:46,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:26:46,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:46,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
649
645
648
[WARNING|trainer.py:803] 2025-04-26 14:26:48,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:48,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:48,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
646
649
650
[WARNING|trainer.py:803] 2025-04-26 14:26:50,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:50,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:50,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
651
650
647
[WARNING|trainer.py:803] 2025-04-26 14:26:52,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:26:53,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:53,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
652
651
648
[WARNING|trainer.py:803] 2025-04-26 14:26:55,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:55,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:26:55,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
653
652
649
[WARNING|trainer.py:803] 2025-04-26 14:26:57,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:26:57,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:26:57,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
653
654
650
[WARNING|trainer.py:803] 2025-04-26 14:26:59,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:00,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:00,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
651
655
654
[WARNING|trainer.py:803] 2025-04-26 14:27:02,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:02,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:02,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
656
652
655
[WARNING|trainer.py:803] 2025-04-26 14:27:04,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:04,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:27:04,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
657
653
656
[WARNING|trainer.py:803] 2025-04-26 14:27:06,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:07,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:07,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
658
657
654
[WARNING|trainer.py:803] 2025-04-26 14:27:09,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:09,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:09,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
659
658
655
[WARNING|trainer.py:803] 2025-04-26 14:27:11,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:11,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:11,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
660
656
659
[WARNING|trainer.py:803] 2025-04-26 14:27:13,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:13,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:14,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
661
660
657
[WARNING|trainer.py:803] 2025-04-26 14:27:15,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:16,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:16,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
661
658
662
[WARNING|trainer.py:803] 2025-04-26 14:27:18,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:18,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:18,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
659
663
662
[WARNING|trainer.py:803] 2025-04-26 14:27:20,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:21,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:21,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
660
663
664
[WARNING|trainer.py:803] 2025-04-26 14:27:23,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:23,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:23,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
661
665
[WARNING|trainer.py:803] 2025-04-26 14:27:25,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
664
[WARNING|trainer.py:803] 2025-04-26 14:27:26,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:26,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
666
662
665
[WARNING|trainer.py:803] 2025-04-26 14:27:28,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:28,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:28,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
667
666
663
[WARNING|trainer.py:803] 2025-04-26 14:27:30,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:30,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:30,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
668
667
664
[WARNING|trainer.py:803] 2025-04-26 14:27:32,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:32,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:33,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
669
668
665
[WARNING|trainer.py:803] 2025-04-26 14:27:34,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:35,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:35,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
670
669
666
[WARNING|trainer.py:803] 2025-04-26 14:27:37,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:37,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:37,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
667
670
671
[WARNING|trainer.py:803] 2025-04-26 14:27:39,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:40,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:40,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
668
672
671
[WARNING|trainer.py:803] 2025-04-26 14:27:42,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:42,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:42,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
673
669
672
[WARNING|trainer.py:803] 2025-04-26 14:27:44,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:27:44,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:44,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
674
670
673
[WARNING|trainer.py:803] 2025-04-26 14:27:46,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:46,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:46,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
675
674
671
[WARNING|trainer.py:803] 2025-04-26 14:27:49,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:49,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:49,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
676
672
675
[WARNING|trainer.py:803] 2025-04-26 14:27:51,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:51,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:51,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
677
673
676
[WARNING|trainer.py:803] 2025-04-26 14:27:53,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:53,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:27:54,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
678
674
677
[WARNING|trainer.py:803] 2025-04-26 14:27:55,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:56,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:27:56,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
679
678
675
[WARNING|trainer.py:803] 2025-04-26 14:27:58,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:27:58,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:27:58,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
680
679
676
[WARNING|trainer.py:803] 2025-04-26 14:28:00,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:00,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:00,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
681
680
677
[WARNING|trainer.py:803] 2025-04-26 14:28:02,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:03,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:03,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
682
681
678
[WARNING|trainer.py:803] 2025-04-26 14:28:04,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:05,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:05,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
682
683
679
[WARNING|trainer.py:803] 2025-04-26 14:28:07,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:07,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:07,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
680
683
684
[WARNING|trainer.py:803] 2025-04-26 14:28:09,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:10,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:10,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
681
[WARNING|trainer.py:803] 2025-04-26 14:28:12,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
685
684
[WARNING|trainer.py:803] 2025-04-26 14:28:12,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
682
[WARNING|trainer.py:803] 2025-04-26 14:28:13,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:14,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
686
685
[WARNING|trainer.py:803] 2025-04-26 14:28:15,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:15,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
683
[WARNING|trainer.py:803] 2025-04-26 14:28:16,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
687
686
[WARNING|trainer.py:803] 2025-04-26 14:28:17,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:18,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
684
688
687
[WARNING|trainer.py:803] 2025-04-26 14:28:19,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:20,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:20,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
685
689
688
[WARNING|trainer.py:803] 2025-04-26 14:28:22,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:22,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:22,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
690
689
686
[WARNING|trainer.py:803] 2025-04-26 14:28:24,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:24,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:24,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
690
691
687
[WARNING|trainer.py:803] 2025-04-26 14:28:27,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:27,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:28:27,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
688
691
692
[WARNING|trainer.py:803] 2025-04-26 14:28:29,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:29,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:28:29,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
689
693
692
[WARNING|trainer.py:803] 2025-04-26 14:28:31,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:31,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:32,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
690
694
693
[WARNING|trainer.py:803] 2025-04-26 14:28:33,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:33,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:34,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
694
691
695
[WARNING|trainer.py:803] 2025-04-26 14:28:36,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:36,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:28:36,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
696
695
692
[WARNING|trainer.py:803] 2025-04-26 14:28:38,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:39,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:39,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
697
693
696
[WARNING|trainer.py:803] 2025-04-26 14:28:41,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:41,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:41,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
694
697
698
[WARNING|trainer.py:803] 2025-04-26 14:28:43,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:43,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:44,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
695
699
698
[WARNING|trainer.py:803] 2025-04-26 14:28:46,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:46,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:46,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
696
700
699
[WARNING|trainer.py:803] 2025-04-26 14:28:48,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:48,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:49,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
697
701
700
[WARNING|trainer.py:803] 2025-04-26 14:28:50,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:28:50,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:28:51,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
702
701
698
[WARNING|trainer.py:803] 2025-04-26 14:28:53,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:53,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:28:53,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
703
702
699
[WARNING|trainer.py:803] 2025-04-26 14:28:55,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 14:28:56,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:28:56,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
704
703
700
[WARNING|trainer.py:803] 2025-04-26 14:28:57,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:28:58,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 14:28:58,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
705
704
701
[WARNING|trainer.py:803] 2025-04-26 14:29:00,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:00,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:00,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
705
706
702
[WARNING|trainer.py:803] 2025-04-26 14:29:02,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:02,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:03,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
707
703
706
[WARNING|trainer.py:803] 2025-04-26 14:29:05,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:05,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 14:29:05,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
704
708
707
[WARNING|trainer.py:803] 2025-04-26 14:29:07,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:07,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:07,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
705
709
708
[WARNING|trainer.py:803] 2025-04-26 14:29:09,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:09,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:10,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
709
710
706
[WARNING|trainer.py:803] 2025-04-26 14:29:12,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:12,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:12,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
707
711
710
[WARNING|trainer.py:803] 2025-04-26 14:29:14,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:14,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:14,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
712
711
708
[WARNING|trainer.py:803] 2025-04-26 14:29:17,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:17,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:17,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
709
712
713
[WARNING|trainer.py:803] 2025-04-26 14:29:19,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:19,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:19,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
710
713
714
[WARNING|trainer.py:803] 2025-04-26 14:29:22,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:22,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:22,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
711
715
714
[WARNING|trainer.py:803] 2025-04-26 14:29:24,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:24,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:24,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
712
715
716
[WARNING|trainer.py:803] 2025-04-26 14:29:26,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:27,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:27,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
716
713
717
[WARNING|trainer.py:803] 2025-04-26 14:29:29,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:29,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:29,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
717
718
714
[WARNING|trainer.py:803] 2025-04-26 14:29:32,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:32,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:32,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
715
718
719
[WARNING|trainer.py:803] 2025-04-26 14:29:34,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:34,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:34,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
716
720
719
[WARNING|trainer.py:803] 2025-04-26 14:29:36,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:37,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:37,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
721
720
717
[WARNING|trainer.py:803] 2025-04-26 14:29:39,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:39,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:39,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
722
721
718
[WARNING|trainer.py:803] 2025-04-26 14:29:41,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:42,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:42,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
723
722
719
[WARNING|trainer.py:803] 2025-04-26 14:29:44,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:44,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:44,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
723
720
724
[WARNING|trainer.py:803] 2025-04-26 14:29:46,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:47,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:47,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
721
724
725
[WARNING|trainer.py:803] 2025-04-26 14:29:49,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:49,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:29:49,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
722
726
725
[WARNING|trainer.py:803] 2025-04-26 14:29:52,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:52,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:52,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
723
727
726
[WARNING|trainer.py:803] 2025-04-26 14:29:54,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:54,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:54,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
728
727
724
[WARNING|trainer.py:803] 2025-04-26 14:29:56,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:29:57,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:29:57,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
728
729
[WARNING|trainer.py:803] 2025-04-26 14:29:59,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
725
[WARNING|trainer.py:803] 2025-04-26 14:29:59,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:00,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
730
729
[WARNING|trainer.py:803] 2025-04-26 14:30:01,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
726
[WARNING|trainer.py:803] 2025-04-26 14:30:01,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:02,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
731
730
[WARNING|trainer.py:803] 2025-04-26 14:30:03,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
727
[WARNING|trainer.py:803] 2025-04-26 14:30:03,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:04,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
732
731
[WARNING|trainer.py:803] 2025-04-26 14:30:05,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
728
[WARNING|trainer.py:803] 2025-04-26 14:30:06,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:06,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
732
733
[WARNING|trainer.py:803] 2025-04-26 14:30:08,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:08,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
729
[WARNING|trainer.py:803] 2025-04-26 14:30:09,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
733
734
[WARNING|trainer.py:803] 2025-04-26 14:30:10,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:30:10,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
730
[WARNING|trainer.py:803] 2025-04-26 14:30:11,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
734
735
731
[WARNING|trainer.py:803] 2025-04-26 14:30:13,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:13,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:30:13,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
736
735
732
[WARNING|trainer.py:803] 2025-04-26 14:30:15,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:15,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:30:15,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
736
737
733
[WARNING|trainer.py:803] 2025-04-26 14:30:18,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:18,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:30:18,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
738
737
734
[WARNING|trainer.py:803] 2025-04-26 14:30:20,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:20,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:30:20,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
739
738
735
[WARNING|trainer.py:803] 2025-04-26 14:30:22,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:23,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:23,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
740
739
736
[WARNING|trainer.py:803] 2025-04-26 14:30:25,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:25,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:25,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
741
740
[WARNING|trainer.py:803] 2025-04-26 14:30:27,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
737
[WARNING|trainer.py:803] 2025-04-26 14:30:27,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:28,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
742
741
[WARNING|trainer.py:803] 2025-04-26 14:30:29,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
738
[WARNING|trainer.py:803] 2025-04-26 14:30:29,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:30,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
742
743
[WARNING|trainer.py:803] 2025-04-26 14:30:32,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:32,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
739
[WARNING|trainer.py:803] 2025-04-26 14:30:33,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
744
743
[WARNING|trainer.py:803] 2025-04-26 14:30:34,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:34,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
740
[WARNING|trainer.py:803] 2025-04-26 14:30:35,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
744
745
741
[WARNING|trainer.py:803] 2025-04-26 14:30:36,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:36,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:37,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
746
745
742
[WARNING|trainer.py:803] 2025-04-26 14:30:39,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:39,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:39,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
746
747
743
[WARNING|trainer.py:803] 2025-04-26 14:30:41,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:41,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:30:42,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
748
747
744
[WARNING|trainer.py:803] 2025-04-26 14:30:44,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:44,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:30:44,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
748
749
745
[WARNING|trainer.py:803] 2025-04-26 14:30:46,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:46,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:47,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
750
746
749
[WARNING|trainer.py:803] 2025-04-26 14:30:48,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:49,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:49,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
750
751
747
[WARNING|trainer.py:803] 2025-04-26 14:30:51,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:51,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:51,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
752
751
748
[WARNING|trainer.py:803] 2025-04-26 14:30:53,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:53,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:54,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
753
752
749
[WARNING|trainer.py:803] 2025-04-26 14:30:56,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:30:56,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:30:56,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
753
754
750
[WARNING|trainer.py:803] 2025-04-26 14:30:58,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:30:58,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:30:59,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
754
755
[WARNING|trainer.py:803] 2025-04-26 14:31:00,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:00,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
751
[WARNING|trainer.py:803] 2025-04-26 14:31:01,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
756
755
[WARNING|trainer.py:803] 2025-04-26 14:31:02,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:31:02,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
752
[WARNING|trainer.py:803] 2025-04-26 14:31:03,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
756
757
[WARNING|trainer.py:803] 2025-04-26 14:31:05,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
753
[WARNING|trainer.py:803] 2025-04-26 14:31:05,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:05,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
757
[WARNING|trainer.py:803] 2025-04-26 14:31:07,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
758
754
[WARNING|trainer.py:803] 2025-04-26 14:31:08,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:08,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
758
759
755
[WARNING|trainer.py:803] 2025-04-26 14:31:10,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:10,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:10,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
759
756
760
[WARNING|trainer.py:803] 2025-04-26 14:31:12,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:12,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:31:12,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
757
760
761
[WARNING|trainer.py:803] 2025-04-26 14:31:14,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:15,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:15,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
761
762
758
[WARNING|trainer.py:803] 2025-04-26 14:31:17,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:17,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:17,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
762
763
759
[WARNING|trainer.py:803] 2025-04-26 14:31:19,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:19,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:20,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
763
764
760
[WARNING|trainer.py:803] 2025-04-26 14:31:22,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:22,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:31:22,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
764
765
761
[WARNING|trainer.py:803] 2025-04-26 14:31:24,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:31:24,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:25,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
765
766
762
[WARNING|trainer.py:803] 2025-04-26 14:31:26,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:27,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:27,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
766
767
763
[WARNING|trainer.py:803] 2025-04-26 14:31:29,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:29,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:29,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
767
768
764
[WARNING|trainer.py:803] 2025-04-26 14:31:31,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:32,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:32,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
768
765
769
[WARNING|trainer.py:803] 2025-04-26 14:31:34,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:34,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:35,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
766
770
769
[WARNING|trainer.py:803] 2025-04-26 14:31:37,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:37,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:37,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
771
770
767
[WARNING|trainer.py:803] 2025-04-26 14:31:39,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:31:39,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:39,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
772
771
768
[WARNING|trainer.py:803] 2025-04-26 14:31:41,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:41,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:31:42,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
773
772
[WARNING|trainer.py:803] 2025-04-26 14:31:43,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:31:43,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
769
[WARNING|trainer.py:803] 2025-04-26 14:31:45,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
774
773
[WARNING|trainer.py:803] 2025-04-26 14:31:46,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:46,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
770
[WARNING|trainer.py:803] 2025-04-26 14:31:47,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
775
774
[WARNING|trainer.py:803] 2025-04-26 14:31:48,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
771
[WARNING|trainer.py:803] 2025-04-26 14:31:48,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:31:49,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
776
775
[WARNING|trainer.py:803] 2025-04-26 14:31:50,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:50,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
772
777
[WARNING|trainer.py:803] 2025-04-26 14:31:51,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
776
[WARNING|trainer.py:803] 2025-04-26 14:31:52,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:31:52,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
773
[WARNING|trainer.py:803] 2025-04-26 14:31:54,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
778
777
[WARNING|trainer.py:803] 2025-04-26 14:31:55,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:55,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
774
[WARNING|trainer.py:803] 2025-04-26 14:31:56,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
779
778
[WARNING|trainer.py:803] 2025-04-26 14:31:57,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
775
[WARNING|trainer.py:803] 2025-04-26 14:31:57,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:31:58,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
779
780
[WARNING|trainer.py:803] 2025-04-26 14:31:59,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:00,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
776
[WARNING|trainer.py:803] 2025-04-26 14:32:00,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
781
780
[WARNING|trainer.py:803] 2025-04-26 14:32:02,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
777
[WARNING|trainer.py:803] 2025-04-26 14:32:02,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:03,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
782
781
[WARNING|trainer.py:803] 2025-04-26 14:32:04,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:04,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
778
[WARNING|trainer.py:803] 2025-04-26 14:32:05,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
783
782
[WARNING|trainer.py:803] 2025-04-26 14:32:07,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:07,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
779
[WARNING|trainer.py:803] 2025-04-26 14:32:08,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
784
783
[WARNING|trainer.py:803] 2025-04-26 14:32:09,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:09,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
780
[WARNING|trainer.py:803] 2025-04-26 14:32:10,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
784
785
[WARNING|trainer.py:803] 2025-04-26 14:32:11,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:11,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
781
[WARNING|trainer.py:803] 2025-04-26 14:32:13,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
786
785
[WARNING|trainer.py:803] 2025-04-26 14:32:14,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:14,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
782
[WARNING|trainer.py:803] 2025-04-26 14:32:15,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
786
787
[WARNING|trainer.py:803] 2025-04-26 14:32:16,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:16,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
783
[WARNING|trainer.py:803] 2025-04-26 14:32:17,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
787
788
[WARNING|trainer.py:803] 2025-04-26 14:32:19,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
784
[WARNING|trainer.py:803] 2025-04-26 14:32:19,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:20,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
789
788
785
[WARNING|trainer.py:803] 2025-04-26 14:32:21,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:21,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:22,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
790
789
[WARNING|trainer.py:803] 2025-04-26 14:32:23,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
786
[WARNING|trainer.py:803] 2025-04-26 14:32:24,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:24,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
791
790
[WARNING|trainer.py:803] 2025-04-26 14:32:25,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:26,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
787
[WARNING|trainer.py:803] 2025-04-26 14:32:27,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
791
792
[WARNING|trainer.py:803] 2025-04-26 14:32:28,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:28,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
788
792
793
[WARNING|trainer.py:803] 2025-04-26 14:32:30,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:32:30,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:31,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
789
793
[WARNING|trainer.py:803] 2025-04-26 14:32:32,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
794
[WARNING|trainer.py:803] 2025-04-26 14:32:33,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:32:33,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
790
795
[WARNING|trainer.py:803] 2025-04-26 14:32:34,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
794
[WARNING|trainer.py:803] 2025-04-26 14:32:35,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:35,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
791
[WARNING|trainer.py:803] 2025-04-26 14:32:36,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
795
796
[WARNING|trainer.py:803] 2025-04-26 14:32:37,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:37,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
792
[WARNING|trainer.py:803] 2025-04-26 14:32:39,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
796
797
[WARNING|trainer.py:803] 2025-04-26 14:32:40,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:32:40,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
793
[WARNING|trainer.py:803] 2025-04-26 14:32:41,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
797
798
[WARNING|trainer.py:803] 2025-04-26 14:32:42,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:42,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
794
799
798
[WARNING|trainer.py:803] 2025-04-26 14:32:44,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:32:44,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:32:45,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
795
[WARNING|trainer.py:803] 2025-04-26 14:32:46,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
799
800
[WARNING|trainer.py:803] 2025-04-26 14:32:47,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:32:47,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
796
[WARNING|trainer.py:803] 2025-04-26 14:32:48,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
800
801
[WARNING|trainer.py:803] 2025-04-26 14:32:49,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:32:49,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
797
[WARNING|trainer.py:803] 2025-04-26 14:32:51,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
801
802
[WARNING|trainer.py:803] 2025-04-26 14:32:52,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:32:52,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
798
[WARNING|trainer.py:803] 2025-04-26 14:32:53,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
803
802
[WARNING|trainer.py:803] 2025-04-26 14:32:54,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:32:54,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
799
[WARNING|trainer.py:803] 2025-04-26 14:32:55,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
804
803
[WARNING|trainer.py:803] 2025-04-26 14:32:56,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:32:57,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
800
805
[WARNING|trainer.py:803] 2025-04-26 14:32:58,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
804
[WARNING|trainer.py:803] 2025-04-26 14:32:59,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:32:59,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
801
806
805
[WARNING|trainer.py:803] 2025-04-26 14:33:00,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:01,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:01,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
802
806
807
[WARNING|trainer.py:803] 2025-04-26 14:33:03,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:03,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:04,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
803
808
807
[WARNING|trainer.py:803] 2025-04-26 14:33:06,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:06,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:06,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
804
809
[WARNING|trainer.py:803] 2025-04-26 14:33:08,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
808
[WARNING|trainer.py:803] 2025-04-26 14:33:08,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:08,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
805
810
809
[WARNING|trainer.py:803] 2025-04-26 14:33:10,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:10,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:11,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
806
811
810
[WARNING|trainer.py:803] 2025-04-26 14:33:12,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:12,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:13,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
812
807
811
[WARNING|trainer.py:803] 2025-04-26 14:33:15,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:15,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:15,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
813
808
812
[WARNING|trainer.py:803] 2025-04-26 14:33:17,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:33:17,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:17,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
809
814
813
[WARNING|trainer.py:803] 2025-04-26 14:33:19,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:20,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:33:20,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
815
810
814
[WARNING|trainer.py:803] 2025-04-26 14:33:22,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:22,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:22,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
816
811
815
[WARNING|trainer.py:803] 2025-04-26 14:33:24,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:24,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:24,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
817
812
816
[WARNING|trainer.py:803] 2025-04-26 14:33:26,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:33:26,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:27,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
818
813
[WARNING|trainer.py:803] 2025-04-26 14:33:28,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
817
[WARNING|trainer.py:803] 2025-04-26 14:33:28,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:33:29,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
819
814
[WARNING|trainer.py:803] 2025-04-26 14:33:30,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
818
[WARNING|trainer.py:803] 2025-04-26 14:33:31,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:33:31,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
820
815
819
[WARNING|trainer.py:803] 2025-04-26 14:33:33,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:33,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:33,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
821
816
[WARNING|trainer.py:803] 2025-04-26 14:33:35,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
820
[WARNING|trainer.py:803] 2025-04-26 14:33:35,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:36,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
822
817
821
[WARNING|trainer.py:803] 2025-04-26 14:33:37,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:38,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:33:38,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
823
818
822
[WARNING|trainer.py:803] 2025-04-26 14:33:40,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:40,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:40,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
819
824
[WARNING|trainer.py:803] 2025-04-26 14:33:42,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
823
[WARNING|trainer.py:803] 2025-04-26 14:33:42,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:43,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
825
820
[WARNING|trainer.py:803] 2025-04-26 14:33:44,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
824
[WARNING|trainer.py:803] 2025-04-26 14:33:44,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:45,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
826
821
825
[WARNING|trainer.py:803] 2025-04-26 14:33:46,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:47,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:47,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
827
822
[WARNING|trainer.py:803] 2025-04-26 14:33:48,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
826
[WARNING|trainer.py:803] 2025-04-26 14:33:49,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:49,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
828
[WARNING|trainer.py:803] 2025-04-26 14:33:50,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
827
823
[WARNING|trainer.py:803] 2025-04-26 14:33:51,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:33:52,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
829
828
[WARNING|trainer.py:803] 2025-04-26 14:33:53,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
824
[WARNING|trainer.py:803] 2025-04-26 14:33:53,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:54,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
830
829
[WARNING|trainer.py:803] 2025-04-26 14:33:55,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
825
[WARNING|trainer.py:803] 2025-04-26 14:33:56,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:33:56,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
831
830
826
[WARNING|trainer.py:803] 2025-04-26 14:33:57,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:33:58,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:33:58,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
832
831
827
[WARNING|trainer.py:803] 2025-04-26 14:34:00,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:00,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:00,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
833
828
[WARNING|trainer.py:803] 2025-04-26 14:34:02,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
832
[WARNING|trainer.py:803] 2025-04-26 14:34:02,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:03,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
834
829
833
[WARNING|trainer.py:803] 2025-04-26 14:34:04,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:05,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:05,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
835
834
830
[WARNING|trainer.py:803] 2025-04-26 14:34:06,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:07,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:07,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
836
[WARNING|trainer.py:803] 2025-04-26 14:34:08,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
835
831
[WARNING|trainer.py:803] 2025-04-26 14:34:09,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:09,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
836
837
832
[WARNING|trainer.py:803] 2025-04-26 14:34:11,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:11,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:12,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
838
833
837
[WARNING|trainer.py:803] 2025-04-26 14:34:13,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:14,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:14,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
839
834
838
[WARNING|trainer.py:803] 2025-04-26 14:34:16,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:16,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:16,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
840
835
839
[WARNING|trainer.py:803] 2025-04-26 14:34:18,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:18,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:18,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
841
836
840
[WARNING|trainer.py:803] 2025-04-26 14:34:20,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:20,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:21,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
842
841
[WARNING|trainer.py:803] 2025-04-26 14:34:22,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
837
[WARNING|trainer.py:803] 2025-04-26 14:34:23,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:23,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
843
842
[WARNING|trainer.py:803] 2025-04-26 14:34:25,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
838
[WARNING|trainer.py:803] 2025-04-26 14:34:25,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:26,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
844
843
[WARNING|trainer.py:803] 2025-04-26 14:34:27,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
839
[WARNING|trainer.py:803] 2025-04-26 14:34:28,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:28,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
845
844
[WARNING|trainer.py:803] 2025-04-26 14:34:29,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
840
[WARNING|trainer.py:803] 2025-04-26 14:34:30,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:30,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
846
845
[WARNING|trainer.py:803] 2025-04-26 14:34:31,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
841
[WARNING|trainer.py:803] 2025-04-26 14:34:32,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
847
[WARNING|trainer.py:803] 2025-04-26 14:34:32,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:33,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
846
842
[WARNING|trainer.py:803] 2025-04-26 14:34:34,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
848
[WARNING|trainer.py:803] 2025-04-26 14:34:35,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
847
[WARNING|trainer.py:803] 2025-04-26 14:34:35,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:36,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
843
849
[WARNING|trainer.py:803] 2025-04-26 14:34:37,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:38,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
848
844
[WARNING|trainer.py:803] 2025-04-26 14:34:38,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:39,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
850
849
[WARNING|trainer.py:803] 2025-04-26 14:34:40,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
845
[WARNING|trainer.py:803] 2025-04-26 14:34:41,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:41,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
851
[WARNING|trainer.py:803] 2025-04-26 14:34:42,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
850
846
[WARNING|trainer.py:803] 2025-04-26 14:34:43,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:43,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
852
[WARNING|trainer.py:803] 2025-04-26 14:34:44,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
851
847
[WARNING|trainer.py:803] 2025-04-26 14:34:45,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:46,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
853
[WARNING|trainer.py:803] 2025-04-26 14:34:47,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
852
848
[WARNING|trainer.py:803] 2025-04-26 14:34:47,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
854
[WARNING|trainer.py:803] 2025-04-26 14:34:48,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:34:49,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
853
849
[WARNING|trainer.py:803] 2025-04-26 14:34:50,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
855
[WARNING|trainer.py:803] 2025-04-26 14:34:50,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:34:51,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
854
[WARNING|trainer.py:803] 2025-04-26 14:34:52,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
856
850
[WARNING|trainer.py:803] 2025-04-26 14:34:53,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 14:34:53,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
855
[WARNING|trainer.py:803] 2025-04-26 14:34:54,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
851
857
[WARNING|trainer.py:803] 2025-04-26 14:34:55,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:34:55,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
856
[WARNING|trainer.py:803] 2025-04-26 14:34:56,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
852
858
[WARNING|trainer.py:803] 2025-04-26 14:34:57,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
857
[WARNING|trainer.py:803] 2025-04-26 14:34:58,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:34:58,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
853
859
[WARNING|trainer.py:803] 2025-04-26 14:34:59,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:35:00,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
858
854
[WARNING|trainer.py:803] 2025-04-26 14:35:01,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
860
[WARNING|trainer.py:803] 2025-04-26 14:35:02,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:35:02,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
859
855
[WARNING|trainer.py:803] 2025-04-26 14:35:03,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
861
[WARNING|trainer.py:803] 2025-04-26 14:35:04,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:35:04,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
860
856
[WARNING|trainer.py:803] 2025-04-26 14:35:05,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
862
[WARNING|trainer.py:803] 2025-04-26 14:35:06,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 14:35:06,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
861
857
[WARNING|trainer.py:803] 2025-04-26 14:35:07,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
863
[WARNING|trainer.py:803] 2025-04-26 14:35:08,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:35:08,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
862
[WARNING|trainer.py:803] 2025-04-26 14:35:10,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
864
858
[WARNING|trainer.py:803] 2025-04-26 14:35:11,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:11,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
863
[WARNING|trainer.py:803] 2025-04-26 14:35:12,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
865
859
[WARNING|trainer.py:803] 2025-04-26 14:35:13,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
864
[WARNING|trainer.py:803] 2025-04-26 14:35:13,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:14,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
866
860
[WARNING|trainer.py:803] 2025-04-26 14:35:15,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:15,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
865
867
[WARNING|trainer.py:803] 2025-04-26 14:35:16,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
861
[WARNING|trainer.py:803] 2025-04-26 14:35:17,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:17,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
866
[WARNING|trainer.py:803] 2025-04-26 14:35:18,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
868
862
[WARNING|trainer.py:803] 2025-04-26 14:35:19,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
867
[WARNING|trainer.py:803] 2025-04-26 14:35:20,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:20,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
869
863
[WARNING|trainer.py:803] 2025-04-26 14:35:21,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
868
[WARNING|trainer.py:803] 2025-04-26 14:35:22,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:22,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
870
864
[WARNING|trainer.py:803] 2025-04-26 14:35:23,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
869
[WARNING|trainer.py:803] 2025-04-26 14:35:24,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:24,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
871
865
[WARNING|trainer.py:803] 2025-04-26 14:35:25,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
870
[WARNING|trainer.py:803] 2025-04-26 14:35:26,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:26,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
866
872
871
[WARNING|trainer.py:803] 2025-04-26 14:35:28,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:28,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:29,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
867
873
[WARNING|trainer.py:803] 2025-04-26 14:35:30,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:30,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
872
[WARNING|trainer.py:803] 2025-04-26 14:35:31,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
868
874
[WARNING|trainer.py:803] 2025-04-26 14:35:32,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:35:33,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
873
869
[WARNING|trainer.py:803] 2025-04-26 14:35:34,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
875
[WARNING|trainer.py:803] 2025-04-26 14:35:35,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
874
[WARNING|trainer.py:803] 2025-04-26 14:35:35,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
870
[WARNING|trainer.py:803] 2025-04-26 14:35:36,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
876
[WARNING|trainer.py:803] 2025-04-26 14:35:37,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:37,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
875
871
[WARNING|trainer.py:803] 2025-04-26 14:35:38,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
877
[WARNING|trainer.py:803] 2025-04-26 14:35:39,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:39,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
876
[WARNING|trainer.py:803] 2025-04-26 14:35:41,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
878
872
[WARNING|trainer.py:803] 2025-04-26 14:35:41,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:35:42,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
877
879
[WARNING|trainer.py:803] 2025-04-26 14:35:43,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
873
[WARNING|trainer.py:803] 2025-04-26 14:35:43,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
878
[WARNING|trainer.py:803] 2025-04-26 14:35:44,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:45,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
880
874
[WARNING|trainer.py:803] 2025-04-26 14:35:46,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
879
[WARNING|trainer.py:803] 2025-04-26 14:35:46,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:35:47,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
881
875
[WARNING|trainer.py:803] 2025-04-26 14:35:48,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
880
[WARNING|trainer.py:803] 2025-04-26 14:35:49,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:49,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
882
876
881
[WARNING|trainer.py:803] 2025-04-26 14:35:50,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:51,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:51,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
883
877
[WARNING|trainer.py:803] 2025-04-26 14:35:53,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
882
[WARNING|trainer.py:803] 2025-04-26 14:35:53,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:53,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
884
878
[WARNING|trainer.py:803] 2025-04-26 14:35:55,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
883
[WARNING|trainer.py:803] 2025-04-26 14:35:55,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:35:56,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
885
879
[WARNING|trainer.py:803] 2025-04-26 14:35:57,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
884
[WARNING|trainer.py:803] 2025-04-26 14:35:57,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:35:58,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
886
880
885
[WARNING|trainer.py:803] 2025-04-26 14:35:59,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:00,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:00,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
887
881
886
[WARNING|trainer.py:803] 2025-04-26 14:36:02,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:02,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:36:02,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
888
882
887
[WARNING|trainer.py:803] 2025-04-26 14:36:04,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:04,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:36:05,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
889
883
[WARNING|trainer.py:803] 2025-04-26 14:36:06,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:06,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
888
[WARNING|trainer.py:803] 2025-04-26 14:36:07,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
884
890
889
[WARNING|trainer.py:803] 2025-04-26 14:36:08,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:36:09,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:09,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
885
891
[WARNING|trainer.py:803] 2025-04-26 14:36:11,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
890
[WARNING|trainer.py:803] 2025-04-26 14:36:11,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:12,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
886
892
[WARNING|trainer.py:803] 2025-04-26 14:36:13,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:13,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
891
[WARNING|trainer.py:803] 2025-04-26 14:36:14,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
893
887
[WARNING|trainer.py:803] 2025-04-26 14:36:15,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
892
[WARNING|trainer.py:803] 2025-04-26 14:36:16,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:16,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
894
888
893
[WARNING|trainer.py:803] 2025-04-26 14:36:18,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:18,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:18,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
889
895
894
[WARNING|trainer.py:803] 2025-04-26 14:36:20,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:20,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:21,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
896
890
895
[WARNING|trainer.py:803] 2025-04-26 14:36:23,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:36:23,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:23,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
897
891
896
[WARNING|trainer.py:803] 2025-04-26 14:36:25,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:25,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:26,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
892
898
897
[WARNING|trainer.py:803] 2025-04-26 14:36:27,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:27,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:28,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
893
899
898
[WARNING|trainer.py:803] 2025-04-26 14:36:29,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:30,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:30,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
894
900
899
[WARNING|trainer.py:803] 2025-04-26 14:36:32,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:32,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:33,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
901
895
[WARNING|trainer.py:803] 2025-04-26 14:36:34,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
900
[WARNING|trainer.py:803] 2025-04-26 14:36:34,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
902
[WARNING|trainer.py:803] 2025-04-26 14:36:35,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:35,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
901
896
[WARNING|trainer.py:803] 2025-04-26 14:36:36,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:36:37,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
903
902
[WARNING|trainer.py:803] 2025-04-26 14:36:38,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:38,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
897
904
[WARNING|trainer.py:803] 2025-04-26 14:36:39,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:40,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
903
905
[WARNING|trainer.py:803] 2025-04-26 14:36:41,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
898
[WARNING|trainer.py:803] 2025-04-26 14:36:41,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
904
[WARNING|trainer.py:803] 2025-04-26 14:36:42,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:42,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
906
899
905
[WARNING|trainer.py:803] 2025-04-26 14:36:43,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:44,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:44,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
907
906
[WARNING|trainer.py:803] 2025-04-26 14:36:45,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
900
[WARNING|trainer.py:803] 2025-04-26 14:36:46,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
908
[WARNING|trainer.py:803] 2025-04-26 14:36:46,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
907
[WARNING|trainer.py:803] 2025-04-26 14:36:47,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
901
[WARNING|trainer.py:803] 2025-04-26 14:36:47,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
909
[WARNING|trainer.py:803] 2025-04-26 14:36:48,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:36:48,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
908
902
910
[WARNING|trainer.py:803] 2025-04-26 14:36:49,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 14:36:50,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:36:50,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
909
911
[WARNING|trainer.py:803] 2025-04-26 14:36:51,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
903
[WARNING|trainer.py:803] 2025-04-26 14:36:52,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
910
[WARNING|trainer.py:803] 2025-04-26 14:36:52,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:53,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
912
904
911
[WARNING|trainer.py:803] 2025-04-26 14:36:54,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:54,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:36:54,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
913
905
912
[WARNING|trainer.py:803] 2025-04-26 14:36:55,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:36:55,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:56,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
914
906
913
[WARNING|trainer.py:803] 2025-04-26 14:36:57,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:57,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:36:58,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
915
907
914
[WARNING|trainer.py:803] 2025-04-26 14:36:59,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:36:59,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:00,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
916
908
915
[WARNING|trainer.py:803] 2025-04-26 14:37:01,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:01,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 14:37:01,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
917
909
916
[WARNING|trainer.py:803] 2025-04-26 14:37:02,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:03,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:03,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
918
910
917
[WARNING|trainer.py:803] 2025-04-26 14:37:04,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:04,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:05,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
919
911
918
[WARNING|trainer.py:803] 2025-04-26 14:37:06,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:06,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:07,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
920
912
919
[WARNING|trainer.py:803] 2025-04-26 14:37:08,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:08,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:08,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
921
913
920
[WARNING|trainer.py:803] 2025-04-26 14:37:09,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:10,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:10,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
914
922
921
[WARNING|trainer.py:803] 2025-04-26 14:37:11,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:11,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:12,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
915
923
922
[WARNING|trainer.py:803] 2025-04-26 14:37:13,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:13,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:13,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
916
923
924
[WARNING|trainer.py:803] 2025-04-26 14:37:15,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:15,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:15,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
917
925
[WARNING|trainer.py:803] 2025-04-26 14:37:17,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
924
[WARNING|trainer.py:803] 2025-04-26 14:37:17,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
918
[WARNING|trainer.py:803] 2025-04-26 14:37:18,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
926
[WARNING|trainer.py:803] 2025-04-26 14:37:18,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
925
[WARNING|trainer.py:803] 2025-04-26 14:37:19,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
919
[WARNING|trainer.py:803] 2025-04-26 14:37:19,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
927
[WARNING|trainer.py:803] 2025-04-26 14:37:20,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
926
[WARNING|trainer.py:803] 2025-04-26 14:37:21,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
920
[WARNING|trainer.py:803] 2025-04-26 14:37:21,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:22,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
928
927
921
[WARNING|trainer.py:803] 2025-04-26 14:37:23,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:23,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:23,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
929
928
922
[WARNING|trainer.py:803] 2025-04-26 14:37:24,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:25,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:25,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
930
929
[WARNING|trainer.py:803] 2025-04-26 14:37:26,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
923
[WARNING|trainer.py:803] 2025-04-26 14:37:26,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:27,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
931
930
[WARNING|trainer.py:803] 2025-04-26 14:37:28,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:28,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
924
932
931
[WARNING|trainer.py:803] 2025-04-26 14:37:29,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:30,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:30,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
925
933
932
[WARNING|trainer.py:803] 2025-04-26 14:37:31,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:32,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:32,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
926
934
933
[WARNING|trainer.py:803] 2025-04-26 14:37:33,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:33,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:33,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
927
935
934
[WARNING|trainer.py:803] 2025-04-26 14:37:35,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:35,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:35,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
928
936
935
[WARNING|trainer.py:803] 2025-04-26 14:37:36,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:37,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:37,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
929
937
936
[WARNING|trainer.py:803] 2025-04-26 14:37:38,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:39,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:39,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
930
938
937
[WARNING|trainer.py:803] 2025-04-26 14:37:40,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:40,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:41,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
931
939
938
[WARNING|trainer.py:803] 2025-04-26 14:37:42,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:37:42,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:42,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
932
940
939
[WARNING|trainer.py:803] 2025-04-26 14:37:44,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:44,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:44,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
933
941
940
[WARNING|trainer.py:803] 2025-04-26 14:37:46,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:46,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:46,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
934
941
942
[WARNING|trainer.py:803] 2025-04-26 14:37:47,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:48,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:48,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
935
943
942
[WARNING|trainer.py:803] 2025-04-26 14:37:49,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:49,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:50,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
936
944
943
[WARNING|trainer.py:803] 2025-04-26 14:37:51,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:51,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:52,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
937
945
944
[WARNING|trainer.py:803] 2025-04-26 14:37:53,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:53,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:54,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
938
946
945
[WARNING|trainer.py:803] 2025-04-26 14:37:55,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:37:55,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:55,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
939
946
947
[WARNING|trainer.py:803] 2025-04-26 14:37:56,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:37:57,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:37:57,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
940
948
[WARNING|trainer.py:803] 2025-04-26 14:37:58,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
947
[WARNING|trainer.py:803] 2025-04-26 14:37:59,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
941
[WARNING|trainer.py:803] 2025-04-26 14:37:59,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
949
[WARNING|trainer.py:803] 2025-04-26 14:38:00,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
948
[WARNING|trainer.py:803] 2025-04-26 14:38:01,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:01,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
942
950
[WARNING|trainer.py:803] 2025-04-26 14:38:02,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
949
[WARNING|trainer.py:803] 2025-04-26 14:38:03,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:03,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
943
[WARNING|trainer.py:803] 2025-04-26 14:38:04,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
951
950
[WARNING|trainer.py:803] 2025-04-26 14:38:05,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:05,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
944
[WARNING|trainer.py:803] 2025-04-26 14:38:06,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
952
951
[WARNING|trainer.py:803] 2025-04-26 14:38:07,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
945
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:07,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:07,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
953
952
[WARNING|trainer.py:803] 2025-04-26 14:38:08,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
946
[WARNING|trainer.py:803] 2025-04-26 14:38:09,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:09,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
954
953
[WARNING|trainer.py:803] 2025-04-26 14:38:10,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:10,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
947
955
954
[WARNING|trainer.py:803] 2025-04-26 14:38:12,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:12,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:12,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
948
956
[WARNING|trainer.py:803] 2025-04-26 14:38:13,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
955
[WARNING|trainer.py:803] 2025-04-26 14:38:14,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:14,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
949
957
[WARNING|trainer.py:803] 2025-04-26 14:38:15,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
956
[WARNING|trainer.py:803] 2025-04-26 14:38:15,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:16,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
950
958
957
[WARNING|trainer.py:803] 2025-04-26 14:38:17,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:17,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:18,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
959
951
958
[WARNING|trainer.py:803] 2025-04-26 14:38:19,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:19,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:19,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
960
952
959
[WARNING|trainer.py:803] 2025-04-26 14:38:21,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:21,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:21,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
961
953
960
[WARNING|trainer.py:803] 2025-04-26 14:38:22,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:23,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:23,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
962
961
954
[WARNING|trainer.py:803] 2025-04-26 14:38:24,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:25,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:25,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
963
962
955
[WARNING|trainer.py:803] 2025-04-26 14:38:26,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:26,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:26,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
964
963
956
[WARNING|trainer.py:803] 2025-04-26 14:38:27,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:28,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:28,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
965
964
957
[WARNING|trainer.py:803] 2025-04-26 14:38:29,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:30,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:30,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
966
965
958
[WARNING|trainer.py:803] 2025-04-26 14:38:31,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:31,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:32,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
967
966
959
[WARNING|trainer.py:803] 2025-04-26 14:38:33,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:33,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:33,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
968
967
960
[WARNING|trainer.py:803] 2025-04-26 14:38:34,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:35,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:35,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
969
961
968
[WARNING|trainer.py:803] 2025-04-26 14:38:36,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:37,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:37,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
970
962
969
[WARNING|trainer.py:803] 2025-04-26 14:38:38,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:38,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:39,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
971
963
970
[WARNING|trainer.py:803] 2025-04-26 14:38:40,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:40,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:41,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
972
964
[WARNING|trainer.py:803] 2025-04-26 14:38:42,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
971
[WARNING|trainer.py:803] 2025-04-26 14:38:42,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
973
965
[WARNING|trainer.py:803] 2025-04-26 14:38:43,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:38:44,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:44,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
972
974
966
[WARNING|trainer.py:803] 2025-04-26 14:38:45,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:45,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:45,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
973
967
[WARNING|trainer.py:803] 2025-04-26 14:38:46,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
975
[WARNING|trainer.py:803] 2025-04-26 14:38:47,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
974
[WARNING|trainer.py:803] 2025-04-26 14:38:48,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:48,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
968
976
[WARNING|trainer.py:803] 2025-04-26 14:38:49,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:38:50,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
975
969
[WARNING|trainer.py:803] 2025-04-26 14:38:51,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
977
[WARNING|trainer.py:803] 2025-04-26 14:38:51,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:51,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
976
970
[WARNING|trainer.py:803] 2025-04-26 14:38:52,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
978
[WARNING|trainer.py:803] 2025-04-26 14:38:53,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:53,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
977
971
[WARNING|trainer.py:803] 2025-04-26 14:38:54,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
979
[WARNING|trainer.py:803] 2025-04-26 14:38:55,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
978
[WARNING|trainer.py:803] 2025-04-26 14:38:55,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:38:56,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
972
980
[WARNING|trainer.py:803] 2025-04-26 14:38:57,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
979
[WARNING|trainer.py:803] 2025-04-26 14:38:57,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:57,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
973
981
[WARNING|trainer.py:803] 2025-04-26 14:38:58,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:38:59,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
980
974
[WARNING|trainer.py:803] 2025-04-26 14:39:00,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
982
[WARNING|trainer.py:803] 2025-04-26 14:39:00,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
981
[WARNING|trainer.py:803] 2025-04-26 14:39:00,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:01,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
975
983
982
[WARNING|trainer.py:803] 2025-04-26 14:39:03,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:03,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:03,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
976
984
[WARNING|trainer.py:803] 2025-04-26 14:39:04,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
983
[WARNING|trainer.py:803] 2025-04-26 14:39:05,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
977
[WARNING|trainer.py:803] 2025-04-26 14:39:05,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
985
[WARNING|trainer.py:803] 2025-04-26 14:39:06,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
984
[WARNING|trainer.py:803] 2025-04-26 14:39:07,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
978
[WARNING|trainer.py:803] 2025-04-26 14:39:07,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:08,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
985
986
979
[WARNING|trainer.py:803] 2025-04-26 14:39:09,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:09,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:10,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
987
986
980
[WARNING|trainer.py:803] 2025-04-26 14:39:11,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:11,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:12,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
988
987
981
[WARNING|trainer.py:803] 2025-04-26 14:39:13,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:39:13,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:13,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
989
988
[WARNING|trainer.py:803] 2025-04-26 14:39:14,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
982
[WARNING|trainer.py:803] 2025-04-26 14:39:15,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:39:15,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
990
989
[WARNING|trainer.py:803] 2025-04-26 14:39:16,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:17,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
983
991
990
[WARNING|trainer.py:803] 2025-04-26 14:39:18,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:18,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:18,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
984
992
991
[WARNING|trainer.py:803] 2025-04-26 14:39:20,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:20,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:20,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
993
985
992
[WARNING|trainer.py:803] 2025-04-26 14:39:22,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:22,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:22,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
994
993
986
[WARNING|trainer.py:803] 2025-04-26 14:39:23,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:24,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:24,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
995
994
987
[WARNING|trainer.py:803] 2025-04-26 14:39:25,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:25,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:26,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
996
988
995
[WARNING|trainer.py:803] 2025-04-26 14:39:27,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:28,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:39:28,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
997
989
996
[WARNING|trainer.py:803] 2025-04-26 14:39:29,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:29,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:29,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
998
990
997
[WARNING|trainer.py:803] 2025-04-26 14:39:31,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:31,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:31,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
999
991
998
[WARNING|trainer.py:803] 2025-04-26 14:39:32,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:33,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:33,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1000
999
992
[WARNING|trainer.py:803] 2025-04-26 14:39:34,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:35,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:35,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1001
993
1000
[WARNING|trainer.py:803] 2025-04-26 14:39:36,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:36,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:36,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1002
994
1001
[WARNING|trainer.py:803] 2025-04-26 14:39:38,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:38,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:39:39,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1003
995
1002
[WARNING|trainer.py:803] 2025-04-26 14:39:40,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:40,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:41,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
996
1004
1003
[WARNING|trainer.py:803] 2025-04-26 14:39:42,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:42,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:42,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1005
997
1004
[WARNING|trainer.py:803] 2025-04-26 14:39:44,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:44,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:45,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1006
998
1005
[WARNING|trainer.py:803] 2025-04-26 14:39:46,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:46,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:46,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1007
999
1006
[WARNING|trainer.py:803] 2025-04-26 14:39:47,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:47,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:48,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1008
1000
[WARNING|trainer.py:803] 2025-04-26 14:39:49,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1007
[WARNING|trainer.py:803] 2025-04-26 14:39:49,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1009
[WARNING|trainer.py:803] 2025-04-26 14:39:50,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1001
[WARNING|trainer.py:803] 2025-04-26 14:39:51,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1008
[WARNING|trainer.py:803] 2025-04-26 14:39:51,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:52,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1010
1002
1009
[WARNING|trainer.py:803] 2025-04-26 14:39:53,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:53,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:54,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1011
1003
[WARNING|trainer.py:803] 2025-04-26 14:39:54,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1010
[WARNING|trainer.py:803] 2025-04-26 14:39:55,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1012
[WARNING|trainer.py:803] 2025-04-26 14:39:55,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:39:56,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1011
1004
1013
[WARNING|trainer.py:803] 2025-04-26 14:39:57,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:39:57,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:39:58,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1012
1005
[WARNING|trainer.py:803] 2025-04-26 14:39:59,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1014
[WARNING|trainer.py:803] 2025-04-26 14:39:59,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1013
[WARNING|trainer.py:803] 2025-04-26 14:40:00,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1006
[WARNING|trainer.py:803] 2025-04-26 14:40:01,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1015
[WARNING|trainer.py:803] 2025-04-26 14:40:01,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:02,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1014
1007
1016
[WARNING|trainer.py:803] 2025-04-26 14:40:03,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:40:03,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:03,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1015
1008
[WARNING|trainer.py:803] 2025-04-26 14:40:04,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:05,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1017
1016
1009
[WARNING|trainer.py:803] 2025-04-26 14:40:06,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:06,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:40:06,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1018
1017
1010
[WARNING|trainer.py:803] 2025-04-26 14:40:08,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:08,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:08,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1019
1011
[WARNING|trainer.py:803] 2025-04-26 14:40:10,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1018
[WARNING|trainer.py:803] 2025-04-26 14:40:10,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:11,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1012
1020
1019
[WARNING|trainer.py:803] 2025-04-26 14:40:12,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:12,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:12,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1013
1021
[WARNING|trainer.py:803] 2025-04-26 14:40:14,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1020
[WARNING|trainer.py:803] 2025-04-26 14:40:14,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:40:15,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1014
1022
[WARNING|trainer.py:803] 2025-04-26 14:40:16,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1021
[WARNING|trainer.py:803] 2025-04-26 14:40:16,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1015
[WARNING|trainer.py:803] 2025-04-26 14:40:17,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1023
[WARNING|trainer.py:803] 2025-04-26 14:40:17,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:18,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1022
1016
[WARNING|trainer.py:803] 2025-04-26 14:40:19,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:19,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1024
1023
[WARNING|trainer.py:803] 2025-04-26 14:40:20,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1017
[WARNING|trainer.py:803] 2025-04-26 14:40:21,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:21,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1025
[WARNING|trainer.py:803] 2025-04-26 14:40:22,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1024
1018
1026
[WARNING|trainer.py:803] 2025-04-26 14:40:23,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:40:23,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:24,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1025
1019
[WARNING|trainer.py:803] 2025-04-26 14:40:25,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1027
[WARNING|trainer.py:803] 2025-04-26 14:40:25,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:26,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1026
1020
1028
[WARNING|trainer.py:803] 2025-04-26 14:40:27,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:40:28,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:28,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1027
1029
1021
[WARNING|trainer.py:803] 2025-04-26 14:40:29,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:29,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:30,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1028
1030
[WARNING|trainer.py:803] 2025-04-26 14:40:31,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1022
[WARNING|trainer.py:803] 2025-04-26 14:40:31,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1029
[WARNING|trainer.py:803] 2025-04-26 14:40:32,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1031
[WARNING|trainer.py:803] 2025-04-26 14:40:32,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1023
[WARNING|trainer.py:803] 2025-04-26 14:40:33,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1030
[WARNING|trainer.py:803] 2025-04-26 14:40:34,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1032
[WARNING|trainer.py:803] 2025-04-26 14:40:34,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:35,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1031
1024
1033
[WARNING|trainer.py:803] 2025-04-26 14:40:36,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:36,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:40:36,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1032
1025
1034
[WARNING|trainer.py:803] 2025-04-26 14:40:38,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:38,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:40:38,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1033
1026
[WARNING|trainer.py:803] 2025-04-26 14:40:39,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1035
[WARNING|trainer.py:803] 2025-04-26 14:40:40,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:40:40,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1034
1027
[WARNING|trainer.py:803] 2025-04-26 14:40:41,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1036
[WARNING|trainer.py:803] 2025-04-26 14:40:42,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:42,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1035
1028
1037
[WARNING|trainer.py:803] 2025-04-26 14:40:44,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:44,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:44,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1029
1036
1038
[WARNING|trainer.py:803] 2025-04-26 14:40:45,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:45,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:46,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1030
1037
1039
[WARNING|trainer.py:803] 2025-04-26 14:40:47,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:47,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:48,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1031
1038
1040
[WARNING|trainer.py:803] 2025-04-26 14:40:49,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:49,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:49,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1032
1039
1041
[WARNING|trainer.py:803] 2025-04-26 14:40:51,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:51,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:40:51,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1033
1040
1042
[WARNING|trainer.py:803] 2025-04-26 14:40:52,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:53,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:53,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1034
1041
[WARNING|trainer.py:803] 2025-04-26 14:40:54,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1043
[WARNING|trainer.py:803] 2025-04-26 14:40:55,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:55,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1035
1042
1044
[WARNING|trainer.py:803] 2025-04-26 14:40:57,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:40:57,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:40:57,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1036
1043
[WARNING|trainer.py:803] 2025-04-26 14:40:58,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1045
[WARNING|trainer.py:803] 2025-04-26 14:40:59,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1037
[WARNING|trainer.py:803] 2025-04-26 14:41:00,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1044
[WARNING|trainer.py:803] 2025-04-26 14:41:00,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1046
[WARNING|trainer.py:803] 2025-04-26 14:41:01,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:01,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1038
1047
[WARNING|trainer.py:803] 2025-04-26 14:41:02,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1045
[WARNING|trainer.py:803] 2025-04-26 14:41:03,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:03,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1039
1046
[WARNING|trainer.py:803] 2025-04-26 14:41:04,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1048
[WARNING|trainer.py:803] 2025-04-26 14:41:05,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1040
[WARNING|trainer.py:803] 2025-04-26 14:41:05,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:06,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1047
1049
1041
[WARNING|trainer.py:803] 2025-04-26 14:41:07,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:07,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:08,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1050
1048
1042
[WARNING|trainer.py:803] 2025-04-26 14:41:09,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:09,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:10,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1049
1051
[WARNING|trainer.py:803] 2025-04-26 14:41:11,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:11,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1043
[WARNING|trainer.py:803] 2025-04-26 14:41:12,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1052
1050
[WARNING|trainer.py:803] 2025-04-26 14:41:13,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:13,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1044
[WARNING|trainer.py:803] 2025-04-26 14:41:14,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1053
1051
[WARNING|trainer.py:803] 2025-04-26 14:41:15,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:15,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1045
1054
1052
[WARNING|trainer.py:803] 2025-04-26 14:41:16,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:17,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:17,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1046
1055
[WARNING|trainer.py:803] 2025-04-26 14:41:18,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1053
[WARNING|trainer.py:803] 2025-04-26 14:41:18,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:19,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1047
1056
[WARNING|trainer.py:803] 2025-04-26 14:41:20,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1054
[WARNING|trainer.py:803] 2025-04-26 14:41:20,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:20,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1057
1048
1055
[WARNING|trainer.py:803] 2025-04-26 14:41:22,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:22,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:22,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1049
1056
1058
[WARNING|trainer.py:803] 2025-04-26 14:41:24,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:24,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:24,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1050
1057
1059
[WARNING|trainer.py:803] 2025-04-26 14:41:26,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:26,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:26,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1051
1060
1058
[WARNING|trainer.py:803] 2025-04-26 14:41:28,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:28,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:28,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1052
1059
1061
[WARNING|trainer.py:803] 2025-04-26 14:41:30,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:30,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:30,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1053
1062
1060
[WARNING|trainer.py:803] 2025-04-26 14:41:32,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:32,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:33,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1054
1063
1061
[WARNING|trainer.py:803] 2025-04-26 14:41:33,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:34,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:34,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1055
1064
1062
[WARNING|trainer.py:803] 2025-04-26 14:41:35,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:36,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:36,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1056
1065
1063
[WARNING|trainer.py:803] 2025-04-26 14:41:37,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:37,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:38,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1057
1066
[WARNING|trainer.py:803] 2025-04-26 14:41:39,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1064
[WARNING|trainer.py:803] 2025-04-26 14:41:39,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:40,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1058
1065
1067
[WARNING|trainer.py:803] 2025-04-26 14:41:41,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:42,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:42,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1059
1066
1068
[WARNING|trainer.py:803] 2025-04-26 14:41:43,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:43,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:44,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1060
1067
1069
[WARNING|trainer.py:803] 2025-04-26 14:41:46,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:46,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:46,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1061
1070
1068
[WARNING|trainer.py:803] 2025-04-26 14:41:48,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:48,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:48,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1062
1071
[WARNING|trainer.py:803] 2025-04-26 14:41:49,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:41:49,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1069
1063
1072
[WARNING|trainer.py:803] 2025-04-26 14:41:50,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:41:51,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:41:51,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1070
1073
[WARNING|trainer.py:803] 2025-04-26 14:41:52,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1064
[WARNING|trainer.py:803] 2025-04-26 14:41:53,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1071
[WARNING|trainer.py:803] 2025-04-26 14:41:53,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1074
[WARNING|trainer.py:803] 2025-04-26 14:41:54,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1065
[WARNING|trainer.py:803] 2025-04-26 14:41:54,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1072
[WARNING|trainer.py:803] 2025-04-26 14:41:55,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1075
[WARNING|trainer.py:803] 2025-04-26 14:41:56,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1066
[WARNING|trainer.py:803] 2025-04-26 14:41:56,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1073
[WARNING|trainer.py:803] 2025-04-26 14:41:57,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:41:57,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1076
1074
[WARNING|trainer.py:803] 2025-04-26 14:41:58,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1067
[WARNING|trainer.py:803] 2025-04-26 14:41:59,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1077
[WARNING|trainer.py:803] 2025-04-26 14:41:59,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1075
[WARNING|trainer.py:803] 2025-04-26 14:42:00,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1068
[WARNING|trainer.py:803] 2025-04-26 14:42:01,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1078
[WARNING|trainer.py:803] 2025-04-26 14:42:01,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:02,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1076
1069
1079
[WARNING|trainer.py:803] 2025-04-26 14:42:03,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:04,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:42:04,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1077
1070
[WARNING|trainer.py:803] 2025-04-26 14:42:05,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1080
[WARNING|trainer.py:803] 2025-04-26 14:42:05,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1078
[WARNING|trainer.py:803] 2025-04-26 14:42:06,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1071
[WARNING|trainer.py:803] 2025-04-26 14:42:06,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:07,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1081
1079
[WARNING|trainer.py:803] 2025-04-26 14:42:08,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1072
[WARNING|trainer.py:803] 2025-04-26 14:42:09,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:09,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1082
[WARNING|trainer.py:803] 2025-04-26 14:42:10,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1073
1080
[WARNING|trainer.py:803] 2025-04-26 14:42:11,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1083
[WARNING|trainer.py:803] 2025-04-26 14:42:11,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1074
[WARNING|trainer.py:803] 2025-04-26 14:42:12,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1081
[WARNING|trainer.py:803] 2025-04-26 14:42:12,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1084
[WARNING|trainer.py:803] 2025-04-26 14:42:13,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1075
[WARNING|trainer.py:803] 2025-04-26 14:42:13,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1082
[WARNING|trainer.py:803] 2025-04-26 14:42:14,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:42:14,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1085
1076
[WARNING|trainer.py:803] 2025-04-26 14:42:15,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1083
1086
[WARNING|trainer.py:803] 2025-04-26 14:42:16,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:16,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:17,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1077
1084
[WARNING|trainer.py:803] 2025-04-26 14:42:18,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1087
[WARNING|trainer.py:803] 2025-04-26 14:42:18,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:19,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1078
1085
[WARNING|trainer.py:803] 2025-04-26 14:42:20,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:20,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1088
1086
1079
[WARNING|trainer.py:803] 2025-04-26 14:42:21,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:22,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:22,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1089
1087
[WARNING|trainer.py:803] 2025-04-26 14:42:23,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1080
[WARNING|trainer.py:803] 2025-04-26 14:42:24,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1090
[WARNING|trainer.py:803] 2025-04-26 14:42:24,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:42:25,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1088
1081
1091
[WARNING|trainer.py:803] 2025-04-26 14:42:26,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:26,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:26,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1089
1082
1092
[WARNING|trainer.py:803] 2025-04-26 14:42:28,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:28,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:29,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1083
1090
[WARNING|trainer.py:803] 2025-04-26 14:42:30,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:30,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1093
1084
1091
[WARNING|trainer.py:803] 2025-04-26 14:42:31,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:31,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:31,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1094
1085
1092
[WARNING|trainer.py:803] 2025-04-26 14:42:33,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:42:33,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:34,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1095
1086
[WARNING|trainer.py:803] 2025-04-26 14:42:35,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:35,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1093
1096
[WARNING|trainer.py:803] 2025-04-26 14:42:36,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1087
[WARNING|trainer.py:803] 2025-04-26 14:42:36,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1094
[WARNING|trainer.py:803] 2025-04-26 14:42:37,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:38,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1097
1088
[WARNING|trainer.py:803] 2025-04-26 14:42:39,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1095
[WARNING|trainer.py:803] 2025-04-26 14:42:39,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1098
[WARNING|trainer.py:803] 2025-04-26 14:42:39,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1089
[WARNING|trainer.py:803] 2025-04-26 14:42:40,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1096
[WARNING|trainer.py:803] 2025-04-26 14:42:41,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1099
[WARNING|trainer.py:803] 2025-04-26 14:42:41,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:42,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1090
1097
[WARNING|trainer.py:803] 2025-04-26 14:42:43,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1100
[WARNING|trainer.py:803] 2025-04-26 14:42:44,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:44,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1091
1098
[WARNING|trainer.py:803] 2025-04-26 14:42:45,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1101
[WARNING|trainer.py:803] 2025-04-26 14:42:45,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:46,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1092
1099
[WARNING|trainer.py:803] 2025-04-26 14:42:47,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:47,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1102
1100
[WARNING|trainer.py:803] 2025-04-26 14:42:48,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1093
[WARNING|trainer.py:803] 2025-04-26 14:42:49,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1103
[WARNING|trainer.py:803] 2025-04-26 14:42:49,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:50,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1101
1094
1104
[WARNING|trainer.py:803] 2025-04-26 14:42:51,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:51,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:42:52,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1095
1102
1105
[WARNING|trainer.py:803] 2025-04-26 14:42:53,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:53,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:53,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1103
1096
1106
[WARNING|trainer.py:803] 2025-04-26 14:42:55,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:55,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:55,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1104
1107
1097
[WARNING|trainer.py:803] 2025-04-26 14:42:57,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:42:57,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:57,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1105
1098
1108
[WARNING|trainer.py:803] 2025-04-26 14:42:59,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:42:59,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:42:59,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1106
1099
[WARNING|trainer.py:803] 2025-04-26 14:43:00,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1109
[WARNING|trainer.py:803] 2025-04-26 14:43:01,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1107
[WARNING|trainer.py:803] 2025-04-26 14:43:01,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1100
[WARNING|trainer.py:803] 2025-04-26 14:43:02,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1110
[WARNING|trainer.py:803] 2025-04-26 14:43:03,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:03,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1108
1111
1101
[WARNING|trainer.py:803] 2025-04-26 14:43:04,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:05,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:05,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1109
1112
[WARNING|trainer.py:803] 2025-04-26 14:43:06,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1102
[WARNING|trainer.py:803] 2025-04-26 14:43:06,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1110
[WARNING|trainer.py:803] 2025-04-26 14:43:07,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1113
[WARNING|trainer.py:803] 2025-04-26 14:43:08,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1103
[WARNING|trainer.py:803] 2025-04-26 14:43:08,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:09,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1111
1114
[WARNING|trainer.py:803] 2025-04-26 14:43:10,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:10,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1104
1112
1115
[WARNING|trainer.py:803] 2025-04-26 14:43:11,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:11,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:12,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1105
1113
1116
[WARNING|trainer.py:803] 2025-04-26 14:43:13,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:43:13,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:13,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1106
1114
[WARNING|trainer.py:803] 2025-04-26 14:43:14,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1117
[WARNING|trainer.py:803] 2025-04-26 14:43:15,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1107
[WARNING|trainer.py:803] 2025-04-26 14:43:15,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1115
[WARNING|trainer.py:803] 2025-04-26 14:43:16,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1118
[WARNING|trainer.py:803] 2025-04-26 14:43:17,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:17,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1116
1108
1119
[WARNING|trainer.py:803] 2025-04-26 14:43:18,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:18,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:19,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1117
1109
1120
[WARNING|trainer.py:803] 2025-04-26 14:43:20,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:20,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:43:21,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1118
1110
1121
[WARNING|trainer.py:803] 2025-04-26 14:43:22,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:22,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:23,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1119
1111
[WARNING|trainer.py:803] 2025-04-26 14:43:24,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1122
[WARNING|trainer.py:803] 2025-04-26 14:43:24,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:25,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1112
1120
[WARNING|trainer.py:803] 2025-04-26 14:43:26,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:26,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1123
1113
1121
[WARNING|trainer.py:803] 2025-04-26 14:43:27,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:28,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:28,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1124
1114
[WARNING|trainer.py:803] 2025-04-26 14:43:29,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1122
[WARNING|trainer.py:803] 2025-04-26 14:43:29,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1125
[WARNING|trainer.py:803] 2025-04-26 14:43:30,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1115
[WARNING|trainer.py:803] 2025-04-26 14:43:31,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:31,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1123
1126
1116
[WARNING|trainer.py:803] 2025-04-26 14:43:32,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:32,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:33,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1124
1127
[WARNING|trainer.py:803] 2025-04-26 14:43:34,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:34,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1117
1128
1125
[WARNING|trainer.py:803] 2025-04-26 14:43:35,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:36,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:36,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1118
1126
1129
[WARNING|trainer.py:803] 2025-04-26 14:43:37,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:37,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:37,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1119
1127
1130
[WARNING|trainer.py:803] 2025-04-26 14:43:38,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:39,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:39,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1131
1128
1120
[WARNING|trainer.py:803] 2025-04-26 14:43:41,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:41,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:41,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1132
1129
1121
[WARNING|trainer.py:803] 2025-04-26 14:43:42,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:43:42,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:43,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1133
1130
1122
[WARNING|trainer.py:803] 2025-04-26 14:43:44,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:44,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:45,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1134
1131
[WARNING|trainer.py:803] 2025-04-26 14:43:46,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:46,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1123
1135
1132
[WARNING|trainer.py:803] 2025-04-26 14:43:47,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:47,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:48,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1124
1136
1133
[WARNING|trainer.py:803] 2025-04-26 14:43:49,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:49,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:49,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1125
1134
1137
[WARNING|trainer.py:803] 2025-04-26 14:43:50,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:51,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:51,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1126
1135
1138
[WARNING|trainer.py:803] 2025-04-26 14:43:52,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:53,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:43:53,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1127
1136
1139
[WARNING|trainer.py:803] 2025-04-26 14:43:54,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:54,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:43:55,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1128
1140
[WARNING|trainer.py:803] 2025-04-26 14:43:56,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1137
[WARNING|trainer.py:803] 2025-04-26 14:43:56,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:43:56,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1129
1141
[WARNING|trainer.py:803] 2025-04-26 14:43:57,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1138
[WARNING|trainer.py:803] 2025-04-26 14:43:58,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1130
[WARNING|trainer.py:803] 2025-04-26 14:43:58,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1142
[WARNING|trainer.py:803] 2025-04-26 14:43:59,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1139
[WARNING|trainer.py:803] 2025-04-26 14:44:00,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1131
[WARNING|trainer.py:803] 2025-04-26 14:44:00,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1143
[WARNING|trainer.py:803] 2025-04-26 14:44:01,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1140
[WARNING|trainer.py:803] 2025-04-26 14:44:01,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:02,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
1132
YesYes
1144
[WARNING|trainer.py:803] 2025-04-26 14:44:02,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1141
[WARNING|trainer.py:803] 2025-04-26 14:44:03,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:03,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1133
1145
[WARNING|trainer.py:803] 2025-04-26 14:44:04,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1142
[WARNING|trainer.py:803] 2025-04-26 14:44:05,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1134
[WARNING|trainer.py:803] 2025-04-26 14:44:05,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1146
[WARNING|trainer.py:803] 2025-04-26 14:44:06,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1143
[WARNING|trainer.py:803] 2025-04-26 14:44:07,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1135
[WARNING|trainer.py:803] 2025-04-26 14:44:07,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1147
1144
[WARNING|trainer.py:803] 2025-04-26 14:44:08,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:08,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:08,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1136
1145
1148
[WARNING|trainer.py:803] 2025-04-26 14:44:09,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:10,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:10,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1137
1149
1146
[WARNING|trainer.py:803] 2025-04-26 14:44:11,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:12,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:12,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1138
1147
1150
[WARNING|trainer.py:803] 2025-04-26 14:44:13,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:14,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:14,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1139
1151
1148
[WARNING|trainer.py:803] 2025-04-26 14:44:15,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:15,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:15,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1140
1152
1149
[WARNING|trainer.py:803] 2025-04-26 14:44:16,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:17,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:17,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1141
1153
1150
[WARNING|trainer.py:803] 2025-04-26 14:44:18,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:19,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:19,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1142
1154
1151
[WARNING|trainer.py:803] 2025-04-26 14:44:20,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:20,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:21,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1143
1152
1155
[WARNING|trainer.py:803] 2025-04-26 14:44:22,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:22,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:22,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1144
1153
1156
[WARNING|trainer.py:803] 2025-04-26 14:44:24,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:24,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:24,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1145
1154
1157
[WARNING|trainer.py:803] 2025-04-26 14:44:25,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:26,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:26,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1146
1155
1158
[WARNING|trainer.py:803] 2025-04-26 14:44:27,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:27,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:28,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1147
1156
1159
[WARNING|trainer.py:803] 2025-04-26 14:44:29,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:29,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:29,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1157
1148
1160
[WARNING|trainer.py:803] 2025-04-26 14:44:31,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:31,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:31,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1149
1158
1161
[WARNING|trainer.py:803] 2025-04-26 14:44:32,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:33,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:33,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1150
1159
1162
[WARNING|trainer.py:803] 2025-04-26 14:44:34,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:34,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:34,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1151
1160
1163
[WARNING|trainer.py:803] 2025-04-26 14:44:36,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:36,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:36,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1152
1161
1164
[WARNING|trainer.py:803] 2025-04-26 14:44:38,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:38,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:38,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1153
1162
1165
[WARNING|trainer.py:803] 2025-04-26 14:44:39,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:40,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:40,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1154
1163
1166
[WARNING|trainer.py:803] 2025-04-26 14:44:41,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:41,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:41,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1155
1164
1167
[WARNING|trainer.py:803] 2025-04-26 14:44:43,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:43,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:43,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1156
1165
1168
[WARNING|trainer.py:803] 2025-04-26 14:44:45,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:45,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:45,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1157
1166
1169
[WARNING|trainer.py:803] 2025-04-26 14:44:46,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:46,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:47,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1167
1158
1170
[WARNING|trainer.py:803] 2025-04-26 14:44:48,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:48,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:48,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1168
1159
1171
[WARNING|trainer.py:803] 2025-04-26 14:44:50,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:50,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:50,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1169
1160
1172
[WARNING|trainer.py:803] 2025-04-26 14:44:52,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:52,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:52,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1170
1161
1173
[WARNING|trainer.py:803] 2025-04-26 14:44:53,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:53,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:54,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1171
1162
1174
[WARNING|trainer.py:803] 2025-04-26 14:44:55,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:55,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:56,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1163
1172
1175
[WARNING|trainer.py:803] 2025-04-26 14:44:57,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:44:57,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:44:58,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1164
1173
1176
[WARNING|trainer.py:803] 2025-04-26 14:44:59,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:44:59,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:44:59,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1165
1174
1177
[WARNING|trainer.py:803] 2025-04-26 14:45:01,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:01,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:01,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1166
1175
1178
[WARNING|trainer.py:803] 2025-04-26 14:45:02,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:45:03,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:03,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1167
1179
1176
[WARNING|trainer.py:803] 2025-04-26 14:45:04,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:05,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:05,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1168
1180
1177
[WARNING|trainer.py:803] 2025-04-26 14:45:06,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:06,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:06,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1169
1178
1181
[WARNING|trainer.py:803] 2025-04-26 14:45:07,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:08,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:08,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1170
1179
1182
[WARNING|trainer.py:803] 2025-04-26 14:45:09,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:10,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:10,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1171
1180
1183
[WARNING|trainer.py:803] 2025-04-26 14:45:11,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:45:12,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:12,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1172
1184
1181
[WARNING|trainer.py:803] 2025-04-26 14:45:13,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:13,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:13,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1173
1182
1185
[WARNING|trainer.py:803] 2025-04-26 14:45:15,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:15,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:15,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1174
1183
1186
[WARNING|trainer.py:803] 2025-04-26 14:45:17,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:17,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:17,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1184
1175
1187
[WARNING|trainer.py:803] 2025-04-26 14:45:19,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:19,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:19,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1176
1185
1188
[WARNING|trainer.py:803] 2025-04-26 14:45:21,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:45:21,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:21,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1177
1186
1189
[WARNING|trainer.py:803] 2025-04-26 14:45:22,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:23,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:23,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1178
1187
[WARNING|trainer.py:803] 2025-04-26 14:45:24,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1190
[WARNING|trainer.py:803] 2025-04-26 14:45:25,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1179
[WARNING|trainer.py:803] 2025-04-26 14:45:25,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1188
[WARNING|trainer.py:803] 2025-04-26 14:45:26,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1191
[WARNING|trainer.py:803] 2025-04-26 14:45:26,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1180
[WARNING|trainer.py:803] 2025-04-26 14:45:27,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1189
[WARNING|trainer.py:803] 2025-04-26 14:45:27,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1192
[WARNING|trainer.py:803] 2025-04-26 14:45:28,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1181
[WARNING|trainer.py:803] 2025-04-26 14:45:29,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:29,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1193
1190
1182
[WARNING|trainer.py:803] 2025-04-26 14:45:30,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:31,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:31,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1194
1191
1183
[WARNING|trainer.py:803] 2025-04-26 14:45:32,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:32,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:33,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1195
1192
1184
[WARNING|trainer.py:803] 2025-04-26 14:45:34,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:34,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:35,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1196
1193
1185
[WARNING|trainer.py:803] 2025-04-26 14:45:36,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:36,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:36,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1197
1194
[WARNING|trainer.py:803] 2025-04-26 14:45:37,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:45:37,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1186
1198
1195
[WARNING|trainer.py:803] 2025-04-26 14:45:38,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:39,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:39,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1187
1196
1199
[WARNING|trainer.py:803] 2025-04-26 14:45:40,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:41,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:41,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1188
1197
1200
[WARNING|trainer.py:803] 2025-04-26 14:45:42,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:43,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:45:43,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1189
1198
[WARNING|trainer.py:803] 2025-04-26 14:45:44,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:44,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1201
[WARNING|trainer.py:803] 2025-04-26 14:45:45,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1199
1190
[WARNING|trainer.py:803] 2025-04-26 14:45:46,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:46,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1202
1200
1191
[WARNING|trainer.py:803] 2025-04-26 14:45:48,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:48,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:48,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1192
1203
1201
[WARNING|trainer.py:803] 2025-04-26 14:45:50,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:50,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:51,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1193
1204
[WARNING|trainer.py:803] 2025-04-26 14:45:52,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1202
[WARNING|trainer.py:803] 2025-04-26 14:45:52,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1194
[WARNING|trainer.py:803] 2025-04-26 14:45:53,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:45:53,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1205
1195
[WARNING|trainer.py:803] 2025-04-26 14:45:54,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1203
[WARNING|trainer.py:803] 2025-04-26 14:45:55,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:55,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1206
1196
1204
[WARNING|trainer.py:803] 2025-04-26 14:45:57,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:45:57,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:45:58,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1197
1207
[WARNING|trainer.py:803] 2025-04-26 14:45:59,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1205
[WARNING|trainer.py:803] 2025-04-26 14:45:59,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1198
[WARNING|trainer.py:803] 2025-04-26 14:46:00,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1208
[WARNING|trainer.py:803] 2025-04-26 14:46:00,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:01,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1206
1199
[WARNING|trainer.py:803] 2025-04-26 14:46:02,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:02,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1209
1200
1207
[WARNING|trainer.py:803] 2025-04-26 14:46:04,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:04,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:04,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1210
1208
1201
[WARNING|trainer.py:803] 2025-04-26 14:46:06,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:06,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:07,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1211
[WARNING|trainer.py:803] 2025-04-26 14:46:08,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1209
1202
[WARNING|trainer.py:803] 2025-04-26 14:46:09,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:09,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1212
1210
[WARNING|trainer.py:803] 2025-04-26 14:46:10,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1203
[WARNING|trainer.py:803] 2025-04-26 14:46:11,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:11,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1213
1211
[WARNING|trainer.py:803] 2025-04-26 14:46:13,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1204
[WARNING|trainer.py:803] 2025-04-26 14:46:13,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:46:14,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1214
1212
[WARNING|trainer.py:803] 2025-04-26 14:46:15,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1205
[WARNING|trainer.py:803] 2025-04-26 14:46:16,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:16,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1215
[WARNING|trainer.py:803] 2025-04-26 14:46:17,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1213
1206
[WARNING|trainer.py:803] 2025-04-26 14:46:18,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:18,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1216
1214
[WARNING|trainer.py:803] 2025-04-26 14:46:19,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1207
[WARNING|trainer.py:803] 2025-04-26 14:46:20,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:20,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1217
[WARNING|trainer.py:803] 2025-04-26 14:46:22,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1215
1208
[WARNING|trainer.py:803] 2025-04-26 14:46:22,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:46:23,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1218
[WARNING|trainer.py:803] 2025-04-26 14:46:24,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1216
1209
[WARNING|trainer.py:803] 2025-04-26 14:46:25,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1219
[WARNING|trainer.py:803] 2025-04-26 14:46:25,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:26,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1217
1210
[WARNING|trainer.py:803] 2025-04-26 14:46:27,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1220
[WARNING|trainer.py:803] 2025-04-26 14:46:27,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:28,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1218
1211
[WARNING|trainer.py:803] 2025-04-26 14:46:29,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1221
[WARNING|trainer.py:803] 2025-04-26 14:46:30,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:46:30,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1219
1212
[WARNING|trainer.py:803] 2025-04-26 14:46:31,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1222
[WARNING|trainer.py:803] 2025-04-26 14:46:32,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:32,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1220
1213
[WARNING|trainer.py:803] 2025-04-26 14:46:34,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1223
[WARNING|trainer.py:803] 2025-04-26 14:46:34,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1221
[WARNING|trainer.py:803] 2025-04-26 14:46:35,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:36,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1214
[WARNING|trainer.py:803] 2025-04-26 14:46:37,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1224
1222
[WARNING|trainer.py:803] 2025-04-26 14:46:38,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:46:38,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1215
[WARNING|trainer.py:803] 2025-04-26 14:46:39,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1225
1223
[WARNING|trainer.py:803] 2025-04-26 14:46:40,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:46:40,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1216
[WARNING|trainer.py:803] 2025-04-26 14:46:41,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1226
1224
[WARNING|trainer.py:803] 2025-04-26 14:46:42,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1217
[WARNING|trainer.py:803] 2025-04-26 14:46:43,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:46:43,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1227
[WARNING|trainer.py:803] 2025-04-26 14:46:44,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1225
1218
[WARNING|trainer.py:803] 2025-04-26 14:46:45,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:46:46,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1228
1226
[WARNING|trainer.py:803] 2025-04-26 14:46:47,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1219
[WARNING|trainer.py:803] 2025-04-26 14:46:48,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:48,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1229
1227
1220
[WARNING|trainer.py:803] 2025-04-26 14:46:49,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:50,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:50,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1230
1221
1228
[WARNING|trainer.py:803] 2025-04-26 14:46:52,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:52,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:52,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1231
1222
1229
[WARNING|trainer.py:803] 2025-04-26 14:46:54,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:54,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:46:55,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1232
1223
[WARNING|trainer.py:803] 2025-04-26 14:46:56,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1230
[WARNING|trainer.py:803] 2025-04-26 14:46:57,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:46:57,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1233
[WARNING|trainer.py:803] 2025-04-26 14:46:59,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1231
1224
[WARNING|trainer.py:803] 2025-04-26 14:46:59,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:00,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1234
1232
[WARNING|trainer.py:803] 2025-04-26 14:47:01,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1225
[WARNING|trainer.py:803] 2025-04-26 14:47:02,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:02,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1235
1233
[WARNING|trainer.py:803] 2025-04-26 14:47:03,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1226
[WARNING|trainer.py:803] 2025-04-26 14:47:04,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:04,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1236
1234
1227
[WARNING|trainer.py:803] 2025-04-26 14:47:06,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:06,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:06,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1237
1235
[WARNING|trainer.py:803] 2025-04-26 14:47:08,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1228
[WARNING|trainer.py:803] 2025-04-26 14:47:09,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:09,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1238
1236
[WARNING|trainer.py:803] 2025-04-26 14:47:10,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1229
[WARNING|trainer.py:803] 2025-04-26 14:47:11,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:11,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1239
[WARNING|trainer.py:803] 2025-04-26 14:47:13,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1237
1230
[WARNING|trainer.py:803] 2025-04-26 14:47:13,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1240
[WARNING|trainer.py:803] 2025-04-26 14:47:14,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:15,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1238
1231
[WARNING|trainer.py:803] 2025-04-26 14:47:16,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:16,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1241
1239
[WARNING|trainer.py:803] 2025-04-26 14:47:17,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1232
[WARNING|trainer.py:803] 2025-04-26 14:47:18,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:18,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1242
1240
1233
[WARNING|trainer.py:803] 2025-04-26 14:47:20,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:20,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:21,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1243
1241
1234
[WARNING|trainer.py:803] 2025-04-26 14:47:22,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:23,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:23,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1244
1242
1235
[WARNING|trainer.py:803] 2025-04-26 14:47:25,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:25,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:25,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1245
1243
1236
[WARNING|trainer.py:803] 2025-04-26 14:47:27,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:28,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:28,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1246
1244
1237
[WARNING|trainer.py:803] 2025-04-26 14:47:29,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:30,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:30,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1247
1238
1245
[WARNING|trainer.py:803] 2025-04-26 14:47:32,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:32,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:33,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1248
1239
1246
[WARNING|trainer.py:803] 2025-04-26 14:47:34,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:35,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:35,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1249
1240
[WARNING|trainer.py:803] 2025-04-26 14:47:36,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1247
[WARNING|trainer.py:803] 2025-04-26 14:47:37,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:37,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1250
1241
[WARNING|trainer.py:803] 2025-04-26 14:47:39,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1248
[WARNING|trainer.py:803] 2025-04-26 14:47:39,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:40,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1251
1242
1249
[WARNING|trainer.py:803] 2025-04-26 14:47:41,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:42,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:42,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1252
1243
1250
[WARNING|trainer.py:803] 2025-04-26 14:47:44,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:44,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:44,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1253
1244
1251
[WARNING|trainer.py:803] 2025-04-26 14:47:46,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:46,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:47,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1254
1245
1252
[WARNING|trainer.py:803] 2025-04-26 14:47:48,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:49,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:47:49,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1255
1246
1253
[WARNING|trainer.py:803] 2025-04-26 14:47:51,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:51,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:51,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1256
1254
1247
[WARNING|trainer.py:803] 2025-04-26 14:47:53,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:54,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:47:54,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1257
1255
1248
[WARNING|trainer.py:803] 2025-04-26 14:47:55,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:56,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:47:56,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1258
1256
1249
[WARNING|trainer.py:803] 2025-04-26 14:47:58,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:58,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:47:58,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1259
1257
1250
[WARNING|trainer.py:803] 2025-04-26 14:48:00,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:00,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:00,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1260
1258
1251
[WARNING|trainer.py:803] 2025-04-26 14:48:03,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:03,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:03,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1261
1259
[WARNING|trainer.py:803] 2025-04-26 14:48:05,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1252
[WARNING|trainer.py:803] 2025-04-26 14:48:05,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:05,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1262
1260
1253
[WARNING|trainer.py:803] 2025-04-26 14:48:07,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:08,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:08,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1263
1261
[WARNING|trainer.py:803] 2025-04-26 14:48:09,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1254
[WARNING|trainer.py:803] 2025-04-26 14:48:10,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:10,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1264
1262
1255
[WARNING|trainer.py:803] 2025-04-26 14:48:12,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:12,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:12,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1265
1263
1256
[WARNING|trainer.py:803] 2025-04-26 14:48:14,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:14,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:15,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1266
1264
1257
[WARNING|trainer.py:803] 2025-04-26 14:48:16,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:17,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:17,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1267
1265
1258
[WARNING|trainer.py:803] 2025-04-26 14:48:19,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:19,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:19,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1268
1259
1266
[WARNING|trainer.py:803] 2025-04-26 14:48:21,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:21,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:21,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1269
1267
1260
[WARNING|trainer.py:803] 2025-04-26 14:48:24,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:24,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:24,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1270
1268
1261
[WARNING|trainer.py:803] 2025-04-26 14:48:26,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:26,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:26,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1271
1269
1262
[WARNING|trainer.py:803] 2025-04-26 14:48:28,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:28,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:29,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1272
1263
1270
[WARNING|trainer.py:803] 2025-04-26 14:48:31,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:31,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:31,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1273
1264
1271
[WARNING|trainer.py:803] 2025-04-26 14:48:33,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:48:33,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:33,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1274
1272
1265
[WARNING|trainer.py:803] 2025-04-26 14:48:35,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:35,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:35,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1275
1273
1266
[WARNING|trainer.py:803] 2025-04-26 14:48:38,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:38,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:48:38,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1274
1276
1267
[WARNING|trainer.py:803] 2025-04-26 14:48:40,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:40,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:40,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1277
1268
1275
[WARNING|trainer.py:803] 2025-04-26 14:48:42,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:42,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:43,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1278
1276
1269
[WARNING|trainer.py:803] 2025-04-26 14:48:44,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:45,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:45,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1279
1277
1270
[WARNING|trainer.py:803] 2025-04-26 14:48:46,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:47,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:47,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1278
1280
1271
[WARNING|trainer.py:803] 2025-04-26 14:48:49,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:49,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:49,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1281
1279
1272
[WARNING|trainer.py:803] 2025-04-26 14:48:51,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:51,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:52,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1282
1280
1273
[WARNING|trainer.py:803] 2025-04-26 14:48:53,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:54,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:48:54,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1281
1274
1283
[WARNING|trainer.py:803] 2025-04-26 14:48:56,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:56,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:56,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1282
1284
1275
[WARNING|trainer.py:803] 2025-04-26 14:48:58,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:48:59,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:48:59,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1276
1283
1285
[WARNING|trainer.py:803] 2025-04-26 14:49:01,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:01,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:01,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1277
1286
1284
[WARNING|trainer.py:803] 2025-04-26 14:49:03,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:03,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:04,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1278
1285
[WARNING|trainer.py:803] 2025-04-26 14:49:05,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1287
[WARNING|trainer.py:803] 2025-04-26 14:49:06,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:06,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1279
1288
[WARNING|trainer.py:803] 2025-04-26 14:49:07,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1286
[WARNING|trainer.py:803] 2025-04-26 14:49:08,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:08,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1280
1289
[WARNING|trainer.py:803] 2025-04-26 14:49:10,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:10,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1287
[WARNING|trainer.py:803] 2025-04-26 14:49:11,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1281
1290
[WARNING|trainer.py:803] 2025-04-26 14:49:12,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:12,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1288
[WARNING|trainer.py:803] 2025-04-26 14:49:13,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
1282
1291
[WARNING|trainer.py:803] 2025-04-26 14:49:14,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1289
[WARNING|trainer.py:803] 2025-04-26 14:49:15,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:49:15,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1292
1283
1290
[WARNING|trainer.py:803] 2025-04-26 14:49:17,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:17,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:17,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1293
1291
1284
[WARNING|trainer.py:803] 2025-04-26 14:49:19,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:20,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:49:20,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1294
1292
[WARNING|trainer.py:803] 2025-04-26 14:49:21,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1285
[WARNING|trainer.py:803] 2025-04-26 14:49:22,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:22,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1295
1293
[WARNING|trainer.py:803] 2025-04-26 14:49:24,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1286
[WARNING|trainer.py:803] 2025-04-26 14:49:24,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:25,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1296
1294
[WARNING|trainer.py:803] 2025-04-26 14:49:26,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:26,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1287
[WARNING|trainer.py:803] 2025-04-26 14:49:27,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1297
1295
[WARNING|trainer.py:803] 2025-04-26 14:49:28,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:29,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1288
[WARNING|trainer.py:803] 2025-04-26 14:49:30,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
1298
1296
1289
[WARNING|trainer.py:803] 2025-04-26 14:49:31,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:31,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:32,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1299
1297
1290
[WARNING|trainer.py:803] 2025-04-26 14:49:33,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:33,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:34,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1300
1298
1291
[WARNING|trainer.py:803] 2025-04-26 14:49:35,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:36,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:36,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1299
1301
1292
[WARNING|trainer.py:803] 2025-04-26 14:49:38,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:38,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:38,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1300
1293
1302
[WARNING|trainer.py:803] 2025-04-26 14:49:40,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:40,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:41,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1294
1303
1301
[WARNING|trainer.py:803] 2025-04-26 14:49:43,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:43,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:49:43,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1295
1304
1302
[WARNING|trainer.py:803] 2025-04-26 14:49:45,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:45,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:45,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1305
1296
1303
[WARNING|trainer.py:803] 2025-04-26 14:49:47,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:47,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:47,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1297
1306
1304
[WARNING|trainer.py:803] 2025-04-26 14:49:50,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:50,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:50,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1307
1305
1298
[WARNING|trainer.py:803] 2025-04-26 14:49:52,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:52,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:52,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1308
1299
1306
[WARNING|trainer.py:803] 2025-04-26 14:49:54,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:49:54,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:49:54,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1309
1307
1300
[WARNING|trainer.py:803] 2025-04-26 14:49:56,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:57,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:49:57,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1310
1308
[WARNING|trainer.py:803] 2025-04-26 14:49:59,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1301
[WARNING|trainer.py:803] 2025-04-26 14:49:59,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:00,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1311
1309
[WARNING|trainer.py:803] 2025-04-26 14:50:01,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:01,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1302
[WARNING|trainer.py:803] 2025-04-26 14:50:02,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1312
1310
[WARNING|trainer.py:803] 2025-04-26 14:50:03,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:03,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1303
[WARNING|trainer.py:803] 2025-04-26 14:50:04,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1311
1313
[WARNING|trainer.py:803] 2025-04-26 14:50:06,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1304
[WARNING|trainer.py:803] 2025-04-26 14:50:06,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:07,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1312
1314
1305
[WARNING|trainer.py:803] 2025-04-26 14:50:08,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:08,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:09,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1315
1313
1306
[WARNING|trainer.py:803] 2025-04-26 14:50:10,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:10,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:11,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1314
1316
1307
[WARNING|trainer.py:803] 2025-04-26 14:50:13,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:13,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:13,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1315
1317
1308
[WARNING|trainer.py:803] 2025-04-26 14:50:15,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:15,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:50:15,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1316
1318
1309
[WARNING|trainer.py:803] 2025-04-26 14:50:17,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:17,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:18,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1317
1310
1319
[WARNING|trainer.py:803] 2025-04-26 14:50:20,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:50:20,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:20,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1318
1311
1320
[WARNING|trainer.py:803] 2025-04-26 14:50:22,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:22,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:22,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1312
1319
1321
[WARNING|trainer.py:803] 2025-04-26 14:50:25,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:25,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:25,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1313
1320
1322
[WARNING|trainer.py:803] 2025-04-26 14:50:27,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:27,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:27,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1314
1323
1321
[WARNING|trainer.py:803] 2025-04-26 14:50:29,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:30,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:50:30,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1315
1324
1322
[WARNING|trainer.py:803] 2025-04-26 14:50:32,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:32,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:32,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1316
1323
1325
[WARNING|trainer.py:803] 2025-04-26 14:50:34,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:34,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:50:35,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1317
1324
1326
[WARNING|trainer.py:803] 2025-04-26 14:50:36,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:50:37,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:37,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1318
1327
1325
[WARNING|trainer.py:803] 2025-04-26 14:50:39,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:39,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:39,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1328
1319
1326
[WARNING|trainer.py:803] 2025-04-26 14:50:41,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:41,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:42,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1329
1320
1327
[WARNING|trainer.py:803] 2025-04-26 14:50:43,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:44,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:44,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1330
1328
1321
[WARNING|trainer.py:803] 2025-04-26 14:50:46,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:46,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:46,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1331
1329
[WARNING|trainer.py:803] 2025-04-26 14:50:48,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1322
[WARNING|trainer.py:803] 2025-04-26 14:50:48,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:49,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1332
1330
[WARNING|trainer.py:803] 2025-04-26 14:50:50,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1323
[WARNING|trainer.py:803] 2025-04-26 14:50:51,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:51,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1333
1331
[WARNING|trainer.py:803] 2025-04-26 14:50:53,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1324
[WARNING|trainer.py:803] 2025-04-26 14:50:53,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:50:53,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1334
1332
[WARNING|trainer.py:803] 2025-04-26 14:50:55,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:50:55,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1325
[WARNING|trainer.py:803] 2025-04-26 14:50:56,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1333
1335
[WARNING|trainer.py:803] 2025-04-26 14:50:57,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:50:57,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1326
[WARNING|trainer.py:803] 2025-04-26 14:50:58,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1334
1336
[WARNING|trainer.py:803] 2025-04-26 14:51:00,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:00,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1327
[WARNING|trainer.py:803] 2025-04-26 14:51:01,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1337
1335
[WARNING|trainer.py:803] 2025-04-26 14:51:02,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:02,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1328
[WARNING|trainer.py:803] 2025-04-26 14:51:03,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1338
1336
[WARNING|trainer.py:803] 2025-04-26 14:51:04,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1329
[WARNING|trainer.py:803] 2025-04-26 14:51:05,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:05,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1337
1339
1330
[WARNING|trainer.py:803] 2025-04-26 14:51:07,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:07,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:08,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1338
1340
1331
[WARNING|trainer.py:803] 2025-04-26 14:51:09,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:09,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:10,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1341
1339
1332
[WARNING|trainer.py:803] 2025-04-26 14:51:11,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:12,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:12,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1342
1340
1333
[WARNING|trainer.py:803] 2025-04-26 14:51:14,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:14,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:14,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1343
1341
1334
[WARNING|trainer.py:803] 2025-04-26 14:51:16,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:16,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:17,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1344
1342
1335
[WARNING|trainer.py:803] 2025-04-26 14:51:18,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:19,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:19,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1345
1343
[WARNING|trainer.py:803] 2025-04-26 14:51:21,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1336
[WARNING|trainer.py:803] 2025-04-26 14:51:21,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:22,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1346
1344
1337
[WARNING|trainer.py:803] 2025-04-26 14:51:23,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:51:23,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:24,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1345
1347
1338
[WARNING|trainer.py:803] 2025-04-26 14:51:26,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:26,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:26,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1346
1348
1339
[WARNING|trainer.py:803] 2025-04-26 14:51:28,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:51:28,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:29,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1349
1347
[WARNING|trainer.py:803] 2025-04-26 14:51:30,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1340
[WARNING|trainer.py:803] 2025-04-26 14:51:30,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:31,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1348
1341
[WARNING|trainer.py:803] 2025-04-26 14:51:33,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1350
[WARNING|trainer.py:803] 2025-04-26 14:51:33,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:34,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1349
1342
[WARNING|trainer.py:803] 2025-04-26 14:51:35,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:36,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1351
1343
[WARNING|trainer.py:803] 2025-04-26 14:51:37,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:38,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1350
[WARNING|trainer.py:803] 2025-04-26 14:51:39,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1344
1352
[WARNING|trainer.py:803] 2025-04-26 14:51:40,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:40,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1351
1345
[WARNING|trainer.py:803] 2025-04-26 14:51:42,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1353
[WARNING|trainer.py:803] 2025-04-26 14:51:43,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:43,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1346
1352
1354
[WARNING|trainer.py:803] 2025-04-26 14:51:45,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:51:45,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1347
[WARNING|trainer.py:803] 2025-04-26 14:51:48,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1353
1355
[WARNING|trainer.py:803] 2025-04-26 14:51:48,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:49,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1348
[WARNING|trainer.py:803] 2025-04-26 14:51:50,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1354
1356
[WARNING|trainer.py:803] 2025-04-26 14:51:51,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1349
[WARNING|trainer.py:803] 2025-04-26 14:51:52,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:52,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1355
1357
[WARNING|trainer.py:803] 2025-04-26 14:51:54,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:51:54,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1350
1356
[WARNING|trainer.py:803] 2025-04-26 14:51:56,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:51:57,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1358
[WARNING|trainer.py:803] 2025-04-26 14:51:58,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1351
1357
1359
[WARNING|trainer.py:803] 2025-04-26 14:51:59,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:51:59,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:00,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1352
1358
[WARNING|trainer.py:803] 2025-04-26 14:52:02,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:52:03,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1360
[WARNING|trainer.py:803] 2025-04-26 14:52:03,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1359
1353
[WARNING|trainer.py:803] 2025-04-26 14:52:05,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1361
[WARNING|trainer.py:803] 2025-04-26 14:52:05,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:06,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1354
1362
[WARNING|trainer.py:803] 2025-04-26 14:52:08,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1360
[WARNING|trainer.py:803] 2025-04-26 14:52:09,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:09,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1355
1361
[WARNING|trainer.py:803] 2025-04-26 14:52:11,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1363
[WARNING|trainer.py:803] 2025-04-26 14:52:11,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:12,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1356
1362
[WARNING|trainer.py:803] 2025-04-26 14:52:14,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:14,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1364
[WARNING|trainer.py:803] 2025-04-26 14:52:15,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1357
1363
[WARNING|trainer.py:803] 2025-04-26 14:52:16,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:17,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1365
[WARNING|trainer.py:803] 2025-04-26 14:52:18,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1358
1364
[WARNING|trainer.py:803] 2025-04-26 14:52:20,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1366
[WARNING|trainer.py:803] 2025-04-26 14:52:21,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:21,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1359
[WARNING|trainer.py:803] 2025-04-26 14:52:22,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1365
1367
[WARNING|trainer.py:803] 2025-04-26 14:52:23,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:24,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1360
1368
1366
[WARNING|trainer.py:803] 2025-04-26 14:52:26,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:26,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:26,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1361
1367
[WARNING|trainer.py:803] 2025-04-26 14:52:28,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1369
[WARNING|trainer.py:803] 2025-04-26 14:52:29,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:29,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1362
1368
[WARNING|trainer.py:803] 2025-04-26 14:52:31,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1370
[WARNING|trainer.py:803] 2025-04-26 14:52:31,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:32,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1363
1371
1369
[WARNING|trainer.py:803] 2025-04-26 14:52:34,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:35,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:35,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1372
1370
1364
[WARNING|trainer.py:803] 2025-04-26 14:52:37,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:37,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:52:38,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1373
1371
[WARNING|trainer.py:803] 2025-04-26 14:52:40,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1365
[WARNING|trainer.py:803] 2025-04-26 14:52:41,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1372
1374
[WARNING|trainer.py:803] 2025-04-26 14:52:42,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:42,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1366
[WARNING|trainer.py:803] 2025-04-26 14:52:44,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1375
1373
[WARNING|trainer.py:803] 2025-04-26 14:52:45,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:45,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1367
[WARNING|trainer.py:803] 2025-04-26 14:52:46,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1374
1376
[WARNING|trainer.py:803] 2025-04-26 14:52:48,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1368
[WARNING|trainer.py:803] 2025-04-26 14:52:48,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:49,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1375
1377
[WARNING|trainer.py:803] 2025-04-26 14:52:50,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:52:51,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1369
[WARNING|trainer.py:803] 2025-04-26 14:52:52,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1376
[WARNING|trainer.py:803] 2025-04-26 14:52:53,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1378
1370
[WARNING|trainer.py:803] 2025-04-26 14:52:54,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:55,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1377
[WARNING|trainer.py:803] 2025-04-26 14:52:56,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1379
1371
[WARNING|trainer.py:803] 2025-04-26 14:52:57,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:52:58,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1378
[WARNING|trainer.py:803] 2025-04-26 14:52:59,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1372
1380
[WARNING|trainer.py:803] 2025-04-26 14:53:00,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:00,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1379
1373
[WARNING|trainer.py:803] 2025-04-26 14:53:02,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1381
[WARNING|trainer.py:803] 2025-04-26 14:53:03,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:03,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1380
1374
[WARNING|trainer.py:803] 2025-04-26 14:53:05,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:05,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1382
[WARNING|trainer.py:803] 2025-04-26 14:53:06,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1375
1381
[WARNING|trainer.py:803] 2025-04-26 14:53:08,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1383
[WARNING|trainer.py:803] 2025-04-26 14:53:08,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:53:09,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1382
1376
1384
[WARNING|trainer.py:803] 2025-04-26 14:53:11,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:53:11,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:11,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1383
1377
[WARNING|trainer.py:803] 2025-04-26 14:53:13,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:53:13,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1385
[WARNING|trainer.py:803] 2025-04-26 14:53:14,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1384
[WARNING|trainer.py:803] 2025-04-26 14:53:16,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1378
1386
[WARNING|trainer.py:803] 2025-04-26 14:53:17,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:17,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1385
1379
1387
[WARNING|trainer.py:803] 2025-04-26 14:53:19,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:20,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:20,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1386
[WARNING|trainer.py:803] 2025-04-26 14:53:22,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1380
[WARNING|trainer.py:803] 2025-04-26 14:53:23,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1388
[WARNING|trainer.py:803] 2025-04-26 14:53:24,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1387
[WARNING|trainer.py:803] 2025-04-26 14:53:25,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1381
[WARNING|trainer.py:803] 2025-04-26 14:53:26,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1389
[WARNING|trainer.py:803] 2025-04-26 14:53:27,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1388
1382
1390
[WARNING|trainer.py:803] 2025-04-26 14:53:29,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:29,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:53:30,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1383
1389
[WARNING|trainer.py:803] 2025-04-26 14:53:32,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1391
[WARNING|trainer.py:803] 2025-04-26 14:53:32,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:53:32,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1384
1390
[WARNING|trainer.py:803] 2025-04-26 14:53:34,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:53:35,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1392
[WARNING|trainer.py:803] 2025-04-26 14:53:35,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1385
1391
[WARNING|trainer.py:803] 2025-04-26 14:53:37,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:37,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1393
[WARNING|trainer.py:803] 2025-04-26 14:53:39,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1386
1392
[WARNING|trainer.py:803] 2025-04-26 14:53:40,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:53:41,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1394
[WARNING|trainer.py:803] 2025-04-26 14:53:42,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1387
1393
[WARNING|trainer.py:803] 2025-04-26 14:53:43,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:53:44,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1395
[WARNING|trainer.py:803] 2025-04-26 14:53:45,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1394
1388
[WARNING|trainer.py:803] 2025-04-26 14:53:47,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:53:47,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1396
[WARNING|trainer.py:803] 2025-04-26 14:53:49,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1389
1395
[WARNING|trainer.py:803] 2025-04-26 14:53:50,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:53:50,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1397
1390
[WARNING|trainer.py:803] 2025-04-26 14:53:53,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:53:53,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1396
[WARNING|trainer.py:803] 2025-04-26 14:53:54,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1391
1398
[WARNING|trainer.py:803] 2025-04-26 14:53:56,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:53:56,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1397
[WARNING|trainer.py:803] 2025-04-26 14:53:58,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1399
1392
[WARNING|trainer.py:803] 2025-04-26 14:53:58,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:53:59,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1398
1400
[WARNING|trainer.py:803] 2025-04-26 14:54:01,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:01,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1393
[WARNING|trainer.py:803] 2025-04-26 14:54:02,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1401
1399
[WARNING|trainer.py:803] 2025-04-26 14:54:03,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:03,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1394
1402
1400
[WARNING|trainer.py:803] 2025-04-26 14:54:05,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:05,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:54:06,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1403
1401
[WARNING|trainer.py:803] 2025-04-26 14:54:08,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1395
[WARNING|trainer.py:803] 2025-04-26 14:54:08,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:09,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1404
1402
[WARNING|trainer.py:803] 2025-04-26 14:54:10,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:54:10,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1405
1403
1396
[WARNING|trainer.py:803] 2025-04-26 14:54:12,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:54:12,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:12,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1406
1404
[WARNING|trainer.py:803] 2025-04-26 14:54:14,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:15,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1397
1407
[WARNING|trainer.py:803] 2025-04-26 14:54:16,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1405
[WARNING|trainer.py:803] 2025-04-26 14:54:17,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:17,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1408
1406
[WARNING|trainer.py:803] 2025-04-26 14:54:19,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1398
[WARNING|trainer.py:803] 2025-04-26 14:54:19,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:20,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1409
1407
1399
[WARNING|trainer.py:803] 2025-04-26 14:54:21,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:21,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:22,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1410
1408
[WARNING|trainer.py:803] 2025-04-26 14:54:23,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1400
[WARNING|trainer.py:803] 2025-04-26 14:54:24,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:24,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1411
1409
[WARNING|trainer.py:803] 2025-04-26 14:54:25,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1401
[WARNING|trainer.py:803] 2025-04-26 14:54:26,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:26,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1412
1410
[WARNING|trainer.py:803] 2025-04-26 14:54:28,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1402
[WARNING|trainer.py:803] 2025-04-26 14:54:28,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:54:29,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1413
1411
[WARNING|trainer.py:803] 2025-04-26 14:54:30,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1403
[WARNING|trainer.py:803] 2025-04-26 14:54:30,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:54:31,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1414
1412
[WARNING|trainer.py:803] 2025-04-26 14:54:32,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1404
[WARNING|trainer.py:803] 2025-04-26 14:54:33,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:33,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1415
1413
[WARNING|trainer.py:803] 2025-04-26 14:54:34,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1405
[WARNING|trainer.py:803] 2025-04-26 14:54:35,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:35,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1416
1414
[WARNING|trainer.py:803] 2025-04-26 14:54:36,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1406
[WARNING|trainer.py:803] 2025-04-26 14:54:37,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:38,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1417
1415
[WARNING|trainer.py:803] 2025-04-26 14:54:39,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1407
[WARNING|trainer.py:803] 2025-04-26 14:54:39,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:54:40,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1418
1416
[WARNING|trainer.py:803] 2025-04-26 14:54:41,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1408
[WARNING|trainer.py:803] 2025-04-26 14:54:41,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:54:42,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1419
1417
[WARNING|trainer.py:803] 2025-04-26 14:54:43,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1409
[WARNING|trainer.py:803] 2025-04-26 14:54:44,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:54:44,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1420
1418
[WARNING|trainer.py:803] 2025-04-26 14:54:45,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1410
[WARNING|trainer.py:803] 2025-04-26 14:54:46,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:46,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1421
1419
[WARNING|trainer.py:803] 2025-04-26 14:54:47,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1411
[WARNING|trainer.py:803] 2025-04-26 14:54:48,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:54:49,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1422
1420
[WARNING|trainer.py:803] 2025-04-26 14:54:50,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1412
[WARNING|trainer.py:803] 2025-04-26 14:54:50,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:54:51,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1423
1421
[WARNING|trainer.py:803] 2025-04-26 14:54:52,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1413
[WARNING|trainer.py:803] 2025-04-26 14:54:52,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:53,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1424
1422
[WARNING|trainer.py:803] 2025-04-26 14:54:54,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:54:54,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1414
1425
[WARNING|trainer.py:803] 2025-04-26 14:54:55,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1423
[WARNING|trainer.py:803] 2025-04-26 14:54:56,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1415
[WARNING|trainer.py:803] 2025-04-26 14:54:57,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1426
[WARNING|trainer.py:803] 2025-04-26 14:54:57,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1424
[WARNING|trainer.py:803] 2025-04-26 14:54:58,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1416
[WARNING|trainer.py:803] 2025-04-26 14:54:59,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:00,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1427
1425
[WARNING|trainer.py:803] 2025-04-26 14:55:00,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:01,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1417
1428
[WARNING|trainer.py:803] 2025-04-26 14:55:02,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1426
[WARNING|trainer.py:803] 2025-04-26 14:55:02,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1418
[WARNING|trainer.py:803] 2025-04-26 14:55:03,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1429
[WARNING|trainer.py:803] 2025-04-26 14:55:04,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1427
[WARNING|trainer.py:803] 2025-04-26 14:55:05,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:05,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1419
1430
[WARNING|trainer.py:803] 2025-04-26 14:55:06,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1428
[WARNING|trainer.py:803] 2025-04-26 14:55:07,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:07,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1420
1431
[WARNING|trainer.py:803] 2025-04-26 14:55:08,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1429
[WARNING|trainer.py:803] 2025-04-26 14:55:09,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1421
[WARNING|trainer.py:803] 2025-04-26 14:55:10,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1432
[WARNING|trainer.py:803] 2025-04-26 14:55:11,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1430
[WARNING|trainer.py:803] 2025-04-26 14:55:11,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:55:12,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1422
1433
[WARNING|trainer.py:803] 2025-04-26 14:55:13,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1431
[WARNING|trainer.py:803] 2025-04-26 14:55:13,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:14,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1423
1434
[WARNING|trainer.py:803] 2025-04-26 14:55:15,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1432
[WARNING|trainer.py:803] 2025-04-26 14:55:16,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:16,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1424
1435
[WARNING|trainer.py:803] 2025-04-26 14:55:17,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:18,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1433
[WARNING|trainer.py:803] 2025-04-26 14:55:19,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1425
1436
[WARNING|trainer.py:803] 2025-04-26 14:55:19,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1434
[WARNING|trainer.py:803] 2025-04-26 14:55:20,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1426
[WARNING|trainer.py:803] 2025-04-26 14:55:21,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:22,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1437
1435
[WARNING|trainer.py:803] 2025-04-26 14:55:23,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1427
[WARNING|trainer.py:803] 2025-04-26 14:55:23,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:24,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1438
1436
[WARNING|trainer.py:803] 2025-04-26 14:55:25,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1428
[WARNING|trainer.py:803] 2025-04-26 14:55:25,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1439
[WARNING|trainer.py:803] 2025-04-26 14:55:26,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:27,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1437
1429
[WARNING|trainer.py:803] 2025-04-26 14:55:28,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:55:28,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1440
1438
[WARNING|trainer.py:803] 2025-04-26 14:55:29,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1430
[WARNING|trainer.py:803] 2025-04-26 14:55:30,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:55:30,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1441
1439
[WARNING|trainer.py:803] 2025-04-26 14:55:31,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1431
[WARNING|trainer.py:803] 2025-04-26 14:55:32,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:32,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1442
1440
[WARNING|trainer.py:803] 2025-04-26 14:55:34,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1432
[WARNING|trainer.py:803] 2025-04-26 14:55:34,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:55:35,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1443
1441
[WARNING|trainer.py:803] 2025-04-26 14:55:36,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1433
[WARNING|trainer.py:803] 2025-04-26 14:55:36,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:37,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1444
[WARNING|trainer.py:803] 2025-04-26 14:55:38,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1442
1434
[WARNING|trainer.py:803] 2025-04-26 14:55:39,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1445
[WARNING|trainer.py:803] 2025-04-26 14:55:39,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:40,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1443
1435
[WARNING|trainer.py:803] 2025-04-26 14:55:41,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:55:41,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1446
[WARNING|trainer.py:803] 2025-04-26 14:55:42,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1444
1436
[WARNING|trainer.py:803] 2025-04-26 14:55:43,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1447
[WARNING|trainer.py:803] 2025-04-26 14:55:44,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:55:44,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1445
1437
[WARNING|trainer.py:803] 2025-04-26 14:55:45,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1448
[WARNING|trainer.py:803] 2025-04-26 14:55:46,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1446
[WARNING|trainer.py:803] 2025-04-26 14:55:47,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1438
[WARNING|trainer.py:803] 2025-04-26 14:55:47,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1449
[WARNING|trainer.py:803] 2025-04-26 14:55:48,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1447
[WARNING|trainer.py:803] 2025-04-26 14:55:49,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1439
[WARNING|trainer.py:803] 2025-04-26 14:55:49,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:50,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1450
1448
[WARNING|trainer.py:803] 2025-04-26 14:55:51,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1440
[WARNING|trainer.py:803] 2025-04-26 14:55:52,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1451
[WARNING|trainer.py:803] 2025-04-26 14:55:53,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1449
[WARNING|trainer.py:803] 2025-04-26 14:55:53,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1441
[WARNING|trainer.py:803] 2025-04-26 14:55:54,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1452
[WARNING|trainer.py:803] 2025-04-26 14:55:55,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:56,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1450
[WARNING|trainer.py:803] 2025-04-26 14:55:56,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1442
1453
[WARNING|trainer.py:803] 2025-04-26 14:55:57,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1451
[WARNING|trainer.py:803] 2025-04-26 14:55:58,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:55:59,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1443
1454
[WARNING|trainer.py:803] 2025-04-26 14:56:00,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1452
[WARNING|trainer.py:803] 2025-04-26 14:56:00,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:01,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1444
1455
[WARNING|trainer.py:803] 2025-04-26 14:56:02,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:56:02,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1453
[WARNING|trainer.py:803] 2025-04-26 14:56:03,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1445
1456
[WARNING|trainer.py:803] 2025-04-26 14:56:04,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:04,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1454
[WARNING|trainer.py:803] 2025-04-26 14:56:05,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1446
1457
[WARNING|trainer.py:803] 2025-04-26 14:56:06,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:56:06,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1455
[WARNING|trainer.py:803] 2025-04-26 14:56:07,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1447
1458
[WARNING|trainer.py:803] 2025-04-26 14:56:08,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1456
[WARNING|trainer.py:803] 2025-04-26 14:56:09,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:10,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1448
1459
[WARNING|trainer.py:803] 2025-04-26 14:56:11,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1457
[WARNING|trainer.py:803] 2025-04-26 14:56:11,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:12,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1449
1460
[WARNING|trainer.py:803] 2025-04-26 14:56:13,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:56:13,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1458
[WARNING|trainer.py:803] 2025-04-26 14:56:14,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1450
1461
[WARNING|trainer.py:803] 2025-04-26 14:56:15,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:15,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1459
[WARNING|trainer.py:803] 2025-04-26 14:56:17,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1451
1462
[WARNING|trainer.py:803] 2025-04-26 14:56:17,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:18,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1460
[WARNING|trainer.py:803] 2025-04-26 14:56:19,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1452
1463
[WARNING|trainer.py:803] 2025-04-26 14:56:20,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:20,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1461
[WARNING|trainer.py:803] 2025-04-26 14:56:21,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1464
1453
[WARNING|trainer.py:803] 2025-04-26 14:56:22,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1462
[WARNING|trainer.py:803] 2025-04-26 14:56:22,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:23,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1465
1454
[WARNING|trainer.py:803] 2025-04-26 14:56:24,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:24,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1463
[WARNING|trainer.py:803] 2025-04-26 14:56:25,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1466
1455
[WARNING|trainer.py:803] 2025-04-26 14:56:26,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:26,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1464
[WARNING|trainer.py:803] 2025-04-26 14:56:27,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1467
1456
[WARNING|trainer.py:803] 2025-04-26 14:56:28,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:56:29,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1465
[WARNING|trainer.py:803] 2025-04-26 14:56:30,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1468
1457
[WARNING|trainer.py:803] 2025-04-26 14:56:31,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:31,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1466
[WARNING|trainer.py:803] 2025-04-26 14:56:32,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1469
1458
[WARNING|trainer.py:803] 2025-04-26 14:56:33,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1467
[WARNING|trainer.py:803] 2025-04-26 14:56:33,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:34,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1470
1459
[WARNING|trainer.py:803] 2025-04-26 14:56:35,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1468
[WARNING|trainer.py:803] 2025-04-26 14:56:36,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:36,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1471
1460
[WARNING|trainer.py:803] 2025-04-26 14:56:37,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1469
[WARNING|trainer.py:803] 2025-04-26 14:56:38,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:38,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1472
1461
[WARNING|trainer.py:803] 2025-04-26 14:56:39,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1470
[WARNING|trainer.py:803] 2025-04-26 14:56:40,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:40,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1473
1462
[WARNING|trainer.py:803] 2025-04-26 14:56:42,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1471
[WARNING|trainer.py:803] 2025-04-26 14:56:42,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:42,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1474
1463
1472
[WARNING|trainer.py:803] 2025-04-26 14:56:44,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:56:44,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:56:45,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1464
1475
1473
[WARNING|trainer.py:803] 2025-04-26 14:56:46,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:47,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:47,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1465
1476
1474
[WARNING|trainer.py:803] 2025-04-26 14:56:49,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:49,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:56:49,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1466
1477
[WARNING|trainer.py:803] 2025-04-26 14:56:51,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1475
[WARNING|trainer.py:803] 2025-04-26 14:56:51,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:52,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1467
1478
[WARNING|trainer.py:803] 2025-04-26 14:56:53,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:56:53,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1476
[WARNING|trainer.py:803] 2025-04-26 14:56:54,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1479
1468
[WARNING|trainer.py:803] 2025-04-26 14:56:55,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:56:55,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1477
[WARNING|trainer.py:803] 2025-04-26 14:56:56,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1469
1480
[WARNING|trainer.py:803] 2025-04-26 14:56:58,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:56:58,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1478
[WARNING|trainer.py:803] 2025-04-26 14:56:58,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1470
1481
1479
[WARNING|trainer.py:803] 2025-04-26 14:57:00,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:00,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:01,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1471
1482
1480
[WARNING|trainer.py:803] 2025-04-26 14:57:02,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:02,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:03,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1472
1483
[WARNING|trainer.py:803] 2025-04-26 14:57:04,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1481
[WARNING|trainer.py:803] 2025-04-26 14:57:05,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:05,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1473
1484
[WARNING|trainer.py:803] 2025-04-26 14:57:07,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1482
[WARNING|trainer.py:803] 2025-04-26 14:57:07,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:08,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1474
1485
[WARNING|trainer.py:803] 2025-04-26 14:57:09,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1483
[WARNING|trainer.py:803] 2025-04-26 14:57:09,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:10,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1475
1486
1484
[WARNING|trainer.py:803] 2025-04-26 14:57:12,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:57:12,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:12,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1476
1485
1487
[WARNING|trainer.py:803] 2025-04-26 14:57:14,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:14,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:14,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1477
1488
[WARNING|trainer.py:803] 2025-04-26 14:57:16,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1486
[WARNING|trainer.py:803] 2025-04-26 14:57:16,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:57:17,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1478
1489
[WARNING|trainer.py:803] 2025-04-26 14:57:18,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:57:19,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1487
1479
[WARNING|trainer.py:803] 2025-04-26 14:57:20,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:20,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1488
1490
[WARNING|trainer.py:803] 2025-04-26 14:57:22,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:57:22,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1480
[WARNING|trainer.py:803] 2025-04-26 14:57:23,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1491
1489
[WARNING|trainer.py:803] 2025-04-26 14:57:24,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:24,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1481
1492
[WARNING|trainer.py:803] 2025-04-26 14:57:25,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:26,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1490
1482
[WARNING|trainer.py:803] 2025-04-26 14:57:27,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1493
[WARNING|trainer.py:803] 2025-04-26 14:57:28,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1491
[WARNING|trainer.py:803] 2025-04-26 14:57:28,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:29,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1494
1483
[WARNING|trainer.py:803] 2025-04-26 14:57:30,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1492
[WARNING|trainer.py:803] 2025-04-26 14:57:30,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:31,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1495
1484
1493
[WARNING|trainer.py:803] 2025-04-26 14:57:32,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:57:32,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:33,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1496
1485
1494
[WARNING|trainer.py:803] 2025-04-26 14:57:34,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:57:35,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:57:35,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1497
1486
1495
[WARNING|trainer.py:803] 2025-04-26 14:57:37,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:37,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:37,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1498
1496
1487
[WARNING|trainer.py:803] 2025-04-26 14:57:39,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:39,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:57:40,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1499
1497
1488
[WARNING|trainer.py:803] 2025-04-26 14:57:41,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:42,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:42,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1500
1498
1489
[WARNING|trainer.py:803] 2025-04-26 14:57:44,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:57:44,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:44,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1499
1501
[WARNING|trainer.py:803] 2025-04-26 14:57:46,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1490
[WARNING|trainer.py:803] 2025-04-26 14:57:47,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:57:47,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1500
1491
[WARNING|trainer.py:803] 2025-04-26 14:57:48,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 14:57:49,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1502
[WARNING|trainer.py:803] 2025-04-26 14:57:50,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1492
1501
[WARNING|trainer.py:803] 2025-04-26 14:57:51,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:52,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1493
1503
[WARNING|trainer.py:803] 2025-04-26 14:57:53,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:54,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1502
1494
[WARNING|trainer.py:803] 2025-04-26 14:57:55,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:57:56,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1504
1495
[WARNING|trainer.py:803] 2025-04-26 14:57:57,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:57:58,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1503
[WARNING|trainer.py:803] 2025-04-26 14:57:59,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1496
1505
[WARNING|trainer.py:803] 2025-04-26 14:58:00,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:00,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1504
1497
[WARNING|trainer.py:803] 2025-04-26 14:58:02,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:02,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1506
[WARNING|trainer.py:803] 2025-04-26 14:58:04,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1498
1505
[WARNING|trainer.py:803] 2025-04-26 14:58:05,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:05,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1499
1507
[WARNING|trainer.py:803] 2025-04-26 14:58:07,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:07,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1506
1500
[WARNING|trainer.py:803] 2025-04-26 14:58:09,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:09,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1508
[WARNING|trainer.py:803] 2025-04-26 14:58:10,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1507
1501
[WARNING|trainer.py:803] 2025-04-26 14:58:12,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:58:13,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1509
[WARNING|trainer.py:803] 2025-04-26 14:58:14,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1508
1502
[WARNING|trainer.py:803] 2025-04-26 14:58:16,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1510
[WARNING|trainer.py:803] 2025-04-26 14:58:16,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:17,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1509
1503
[WARNING|trainer.py:803] 2025-04-26 14:58:19,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1511
[WARNING|trainer.py:803] 2025-04-26 14:58:20,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:20,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1510
1504
[WARNING|trainer.py:803] 2025-04-26 14:58:22,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1512
[WARNING|trainer.py:803] 2025-04-26 14:58:23,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:24,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1511
[WARNING|trainer.py:803] 2025-04-26 14:58:26,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1505
1513
[WARNING|trainer.py:803] 2025-04-26 14:58:26,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:27,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1512
[WARNING|trainer.py:803] 2025-04-26 14:58:29,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1506
1514
[WARNING|trainer.py:803] 2025-04-26 14:58:30,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:30,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1513
[WARNING|trainer.py:803] 2025-04-26 14:58:32,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1507
1515
[WARNING|trainer.py:803] 2025-04-26 14:58:33,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:58:34,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1514
[WARNING|trainer.py:803] 2025-04-26 14:58:36,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1508
1516
[WARNING|trainer.py:803] 2025-04-26 14:58:37,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:37,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1515
[WARNING|trainer.py:803] 2025-04-26 14:58:39,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1509
1517
[WARNING|trainer.py:803] 2025-04-26 14:58:40,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:58:41,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1516
[WARNING|trainer.py:803] 2025-04-26 14:58:43,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1510
1518
[WARNING|trainer.py:803] 2025-04-26 14:58:44,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:44,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1517
[WARNING|trainer.py:803] 2025-04-26 14:58:46,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1511
1519
[WARNING|trainer.py:803] 2025-04-26 14:58:47,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 14:58:47,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1518
[WARNING|trainer.py:803] 2025-04-26 14:58:49,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1512
1520
[WARNING|trainer.py:803] 2025-04-26 14:58:50,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:51,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1519
[WARNING|trainer.py:803] 2025-04-26 14:58:53,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1513
1521
[WARNING|trainer.py:803] 2025-04-26 14:58:54,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:54,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1520
[WARNING|trainer.py:803] 2025-04-26 14:58:56,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1514
1522
[WARNING|trainer.py:803] 2025-04-26 14:58:57,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:58:58,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1521
[WARNING|trainer.py:803] 2025-04-26 14:58:59,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1515
[WARNING|trainer.py:803] 2025-04-26 14:59:00,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1523
[WARNING|trainer.py:803] 2025-04-26 14:59:01,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1522
[WARNING|trainer.py:803] 2025-04-26 14:59:03,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1516
[WARNING|trainer.py:803] 2025-04-26 14:59:04,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1524
[WARNING|trainer.py:803] 2025-04-26 14:59:05,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1523
1517
[WARNING|trainer.py:803] 2025-04-26 14:59:07,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:59:07,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1525
[WARNING|trainer.py:803] 2025-04-26 14:59:08,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1524
1518
[WARNING|trainer.py:803] 2025-04-26 14:59:10,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:11,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1526
[WARNING|trainer.py:803] 2025-04-26 14:59:12,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1525
1519
[WARNING|trainer.py:803] 2025-04-26 14:59:14,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 14:59:14,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1527
[WARNING|trainer.py:803] 2025-04-26 14:59:16,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1526
1520
[WARNING|trainer.py:803] 2025-04-26 14:59:17,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:18,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1528
[WARNING|trainer.py:803] 2025-04-26 14:59:19,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1521
1527
[WARNING|trainer.py:803] 2025-04-26 14:59:21,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:21,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1529
[WARNING|trainer.py:803] 2025-04-26 14:59:22,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1522
1528
[WARNING|trainer.py:803] 2025-04-26 14:59:24,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:24,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1530
[WARNING|trainer.py:803] 2025-04-26 14:59:26,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1529
1523
[WARNING|trainer.py:803] 2025-04-26 14:59:28,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:28,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1531
[WARNING|trainer.py:803] 2025-04-26 14:59:30,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1530
1524
[WARNING|trainer.py:803] 2025-04-26 14:59:31,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:31,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1532
[WARNING|trainer.py:803] 2025-04-26 14:59:33,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1531
1525
[WARNING|trainer.py:803] 2025-04-26 14:59:35,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:35,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1533
[WARNING|trainer.py:803] 2025-04-26 14:59:37,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1532
1526
[WARNING|trainer.py:803] 2025-04-26 14:59:38,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 14:59:39,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1534
[WARNING|trainer.py:803] 2025-04-26 14:59:40,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1533
1527
[WARNING|trainer.py:803] 2025-04-26 14:59:42,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:42,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1535
[WARNING|trainer.py:803] 2025-04-26 14:59:44,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1534
1528
[WARNING|trainer.py:803] 2025-04-26 14:59:46,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:46,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1536
[WARNING|trainer.py:803] 2025-04-26 14:59:48,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1529
1535
[WARNING|trainer.py:803] 2025-04-26 14:59:49,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:49,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1537
[WARNING|trainer.py:803] 2025-04-26 14:59:51,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1530
[WARNING|trainer.py:803] 2025-04-26 14:59:52,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1536
[WARNING|trainer.py:803] 2025-04-26 14:59:53,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1538
[WARNING|trainer.py:803] 2025-04-26 14:59:54,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1531
1537
[WARNING|trainer.py:803] 2025-04-26 14:59:56,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 14:59:57,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1539
[WARNING|trainer.py:803] 2025-04-26 14:59:58,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1532
1538
[WARNING|trainer.py:803] 2025-04-26 14:59:59,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:00,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1540
[WARNING|trainer.py:803] 2025-04-26 15:00:01,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1533
1539
[WARNING|trainer.py:803] 2025-04-26 15:00:03,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:00:04,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1541
[WARNING|trainer.py:803] 2025-04-26 15:00:05,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1534
1540
[WARNING|trainer.py:803] 2025-04-26 15:00:06,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:00:07,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1542
[WARNING|trainer.py:803] 2025-04-26 15:00:08,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1535
1541
[WARNING|trainer.py:803] 2025-04-26 15:00:10,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:10,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1543
[WARNING|trainer.py:803] 2025-04-26 15:00:11,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1542
1536
[WARNING|trainer.py:803] 2025-04-26 15:00:14,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:00:14,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1544
[WARNING|trainer.py:803] 2025-04-26 15:00:15,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1543
1537
[WARNING|trainer.py:803] 2025-04-26 15:00:17,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:17,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1545
[WARNING|trainer.py:803] 2025-04-26 15:00:18,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1544
1538
1546
[WARNING|trainer.py:803] 2025-04-26 15:00:21,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:21,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:22,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1545
1539
1547
[WARNING|trainer.py:803] 2025-04-26 15:00:24,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:24,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:25,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1546
1540
1548
[WARNING|trainer.py:803] 2025-04-26 15:00:28,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:28,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:28,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1541
1547
1549
[WARNING|trainer.py:803] 2025-04-26 15:00:31,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:31,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:00:31,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1548
1542
1550
[WARNING|trainer.py:803] 2025-04-26 15:00:34,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:35,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:00:35,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1549
1543
[WARNING|trainer.py:803] 2025-04-26 15:00:38,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1551
[WARNING|trainer.py:803] 2025-04-26 15:00:38,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:38,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1550
1544
1552
[WARNING|trainer.py:803] 2025-04-26 15:00:41,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:41,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:42,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1551
1553
1545
[WARNING|trainer.py:803] 2025-04-26 15:00:45,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:00:45,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:45,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1552
1546
1554
[WARNING|trainer.py:803] 2025-04-26 15:00:48,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:48,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:48,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1553
1547
1555
[WARNING|trainer.py:803] 2025-04-26 15:00:51,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:52,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:52,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1554
1548
1556
[WARNING|trainer.py:803] 2025-04-26 15:00:55,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:00:55,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:00:55,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1555
1549
1557
[WARNING|trainer.py:803] 2025-04-26 15:00:58,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:00:58,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:00:58,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1556
1558
1550
[WARNING|trainer.py:803] 2025-04-26 15:01:01,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:02,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:02,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1557
1559
1551
[WARNING|trainer.py:803] 2025-04-26 15:01:05,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:05,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:05,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1558
1560
1552
[WARNING|trainer.py:803] 2025-04-26 15:01:08,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:08,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:09,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1559
1561
1553
[WARNING|trainer.py:803] 2025-04-26 15:01:12,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:12,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:12,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1560
1554
1562
[WARNING|trainer.py:803] 2025-04-26 15:01:15,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:15,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:15,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1561
1555
1563
[WARNING|trainer.py:803] 2025-04-26 15:01:18,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:19,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:19,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1564
1562
1556
[WARNING|trainer.py:803] 2025-04-26 15:01:22,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:22,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:22,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1565
1563
1557
[WARNING|trainer.py:803] 2025-04-26 15:01:25,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:25,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:25,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1564
1566
1558
[WARNING|trainer.py:803] 2025-04-26 15:01:28,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:29,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:29,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1565
1567
1559
[WARNING|trainer.py:803] 2025-04-26 15:01:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:32,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:32,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1566
1568
1560
[WARNING|trainer.py:803] 2025-04-26 15:01:35,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:36,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:36,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1567
1561
1569
[WARNING|trainer.py:803] 2025-04-26 15:01:39,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:39,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:39,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1568
1570
1562
[WARNING|trainer.py:803] 2025-04-26 15:01:42,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:43,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:43,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1569
1571
1563
[WARNING|trainer.py:803] 2025-04-26 15:01:46,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:01:46,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:01:46,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1570
1564
1572
[WARNING|trainer.py:803] 2025-04-26 15:01:49,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:50,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:50,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1571
1573
1565
[WARNING|trainer.py:803] 2025-04-26 15:01:53,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:01:53,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:01:53,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1572
1574
1566
[WARNING|trainer.py:803] 2025-04-26 15:01:56,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:01:56,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:01:57,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1573
1575
1567
[WARNING|trainer.py:803] 2025-04-26 15:02:00,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:02:00,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:00,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1574
1576
[WARNING|trainer.py:803] 2025-04-26 15:02:03,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1568
[WARNING|trainer.py:803] 2025-04-26 15:02:03,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:02:04,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1575
1577
[WARNING|trainer.py:803] 2025-04-26 15:02:06,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:06,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1569
[WARNING|trainer.py:803] 2025-04-26 15:02:07,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1576
1578
[WARNING|trainer.py:803] 2025-04-26 15:02:10,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:02:10,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1570
[WARNING|trainer.py:803] 2025-04-26 15:02:11,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1579
1577
[WARNING|trainer.py:803] 2025-04-26 15:02:13,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:02:13,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1571
[WARNING|trainer.py:803] 2025-04-26 15:02:14,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1578
1580
[WARNING|trainer.py:803] 2025-04-26 15:02:16,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:17,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1572
[WARNING|trainer.py:803] 2025-04-26 15:02:18,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1579
1581
[WARNING|trainer.py:803] 2025-04-26 15:02:20,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:02:20,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1573
[WARNING|trainer.py:803] 2025-04-26 15:02:21,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1582
1580
[WARNING|trainer.py:803] 2025-04-26 15:02:23,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:23,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1574
[WARNING|trainer.py:803] 2025-04-26 15:02:25,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1581
1583
[WARNING|trainer.py:803] 2025-04-26 15:02:26,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:27,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1575
[WARNING|trainer.py:803] 2025-04-26 15:02:28,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1582
1584
[WARNING|trainer.py:803] 2025-04-26 15:02:30,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:31,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1576
[WARNING|trainer.py:803] 2025-04-26 15:02:32,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1583
1585
[WARNING|trainer.py:803] 2025-04-26 15:02:34,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:34,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1577
[WARNING|trainer.py:803] 2025-04-26 15:02:35,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1584
1586
[WARNING|trainer.py:803] 2025-04-26 15:02:37,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:37,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1578
[WARNING|trainer.py:803] 2025-04-26 15:02:38,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1585
1587
[WARNING|trainer.py:803] 2025-04-26 15:02:40,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1579
[WARNING|trainer.py:803] 2025-04-26 15:02:41,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:02:41,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1588
1586
[WARNING|trainer.py:803] 2025-04-26 15:02:44,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:02:44,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1580
[WARNING|trainer.py:803] 2025-04-26 15:02:45,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1589
1587
[WARNING|trainer.py:803] 2025-04-26 15:02:47,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:02:48,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1581
[WARNING|trainer.py:803] 2025-04-26 15:02:49,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1590
1588
[WARNING|trainer.py:803] 2025-04-26 15:02:50,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:02:51,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1582
[WARNING|trainer.py:803] 2025-04-26 15:02:52,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1591
1589
[WARNING|trainer.py:803] 2025-04-26 15:02:54,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:02:54,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1583
[WARNING|trainer.py:803] 2025-04-26 15:02:56,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1590
1592
[WARNING|trainer.py:803] 2025-04-26 15:02:57,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:02:57,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1584
[WARNING|trainer.py:803] 2025-04-26 15:02:59,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1591
1593
[WARNING|trainer.py:803] 2025-04-26 15:03:01,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:03:01,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1585
[WARNING|trainer.py:803] 2025-04-26 15:03:03,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1594
1592
[WARNING|trainer.py:803] 2025-04-26 15:03:04,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:03:04,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1586
1593
[WARNING|trainer.py:803] 2025-04-26 15:03:07,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1595
[WARNING|trainer.py:803] 2025-04-26 15:03:07,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:08,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1587
1596
1594
[WARNING|trainer.py:803] 2025-04-26 15:03:10,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:03:11,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:11,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1588
1597
[WARNING|trainer.py:803] 2025-04-26 15:03:13,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1595
[WARNING|trainer.py:803] 2025-04-26 15:03:14,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:15,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1589
[WARNING|trainer.py:803] 2025-04-26 15:03:17,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1596
1598
[WARNING|trainer.py:803] 2025-04-26 15:03:18,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:18,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1590
[WARNING|trainer.py:803] 2025-04-26 15:03:20,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1597
1599
[WARNING|trainer.py:803] 2025-04-26 15:03:21,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:21,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1591
1600
[WARNING|trainer.py:803] 2025-04-26 15:03:23,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1598
[WARNING|trainer.py:803] 2025-04-26 15:03:24,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:24,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1592
1599
[WARNING|trainer.py:803] 2025-04-26 15:03:27,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1601
[WARNING|trainer.py:803] 2025-04-26 15:03:28,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:28,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1593
1600
1602
[WARNING|trainer.py:803] 2025-04-26 15:03:31,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:31,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:31,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1594
1603
1601
[WARNING|trainer.py:803] 2025-04-26 15:03:34,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:03:35,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:35,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1595
1604
1602
[WARNING|trainer.py:803] 2025-04-26 15:03:38,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:03:38,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:38,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1605
1596
1603
[WARNING|trainer.py:803] 2025-04-26 15:03:41,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:41,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:41,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1597
1606
1604
[WARNING|trainer.py:803] 2025-04-26 15:03:44,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:45,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:45,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1598
1605
1607
[WARNING|trainer.py:803] 2025-04-26 15:03:48,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:48,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:48,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1599
1606
[WARNING|trainer.py:803] 2025-04-26 15:03:51,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1608
[WARNING|trainer.py:803] 2025-04-26 15:03:51,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:52,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1600
1609
1607
[WARNING|trainer.py:803] 2025-04-26 15:03:55,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:03:55,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:55,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1601
1610
1608
[WARNING|trainer.py:803] 2025-04-26 15:03:58,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:03:59,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:03:59,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1602
1609
1611
[WARNING|trainer.py:803] 2025-04-26 15:04:02,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:02,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:03,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1603
1612
[WARNING|trainer.py:803] 2025-04-26 15:04:05,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1610
[WARNING|trainer.py:803] 2025-04-26 15:04:05,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:04:06,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1604
1613
1611
[WARNING|trainer.py:803] 2025-04-26 15:04:09,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:09,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:10,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1605
1614
1612
[WARNING|trainer.py:803] 2025-04-26 15:04:12,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:12,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:04:12,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1606
1613
1615
[WARNING|trainer.py:803] 2025-04-26 15:04:16,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:16,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:16,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1614
1607
1616
[WARNING|trainer.py:803] 2025-04-26 15:04:19,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:04:19,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:20,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1617
1615
1608
[WARNING|trainer.py:803] 2025-04-26 15:04:23,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:23,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:23,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1609
1618
1616
[WARNING|trainer.py:803] 2025-04-26 15:04:27,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:27,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:27,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1619
1617
1610
[WARNING|trainer.py:803] 2025-04-26 15:04:30,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:30,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:30,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1618
1611
1620
[WARNING|trainer.py:803] 2025-04-26 15:04:34,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:34,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:04:34,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1612
1619
1621
[WARNING|trainer.py:803] 2025-04-26 15:04:37,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:37,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:04:38,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1613
1622
1620
[WARNING|trainer.py:803] 2025-04-26 15:04:41,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:41,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:41,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1614
1623
1621
[WARNING|trainer.py:803] 2025-04-26 15:04:44,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:04:44,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:04:45,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1624
1615
1622
[WARNING|trainer.py:803] 2025-04-26 15:04:47,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:48,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:48,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1625
1623
1616
[WARNING|trainer.py:803] 2025-04-26 15:04:51,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:52,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:04:52,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1626
1624
1617
[WARNING|trainer.py:803] 2025-04-26 15:04:55,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:55,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:04:55,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1618
1627
1625
[WARNING|trainer.py:803] 2025-04-26 15:04:59,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:59,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:04:59,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1628
1619
1626
[WARNING|trainer.py:803] 2025-04-26 15:05:02,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:05:02,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:05:02,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1629
[WARNING|trainer.py:803] 2025-04-26 15:05:05,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1620
1627
[WARNING|trainer.py:803] 2025-04-26 15:05:06,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:05:06,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1630
1628
[WARNING|trainer.py:803] 2025-04-26 15:05:08,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:05:09,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1621
[WARNING|trainer.py:803] 2025-04-26 15:05:10,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1631
1629
[WARNING|trainer.py:803] 2025-04-26 15:05:12,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:05:12,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1622
[WARNING|trainer.py:803] 2025-04-26 15:05:13,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1632
1630
[WARNING|trainer.py:803] 2025-04-26 15:05:15,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:05:16,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1623
[WARNING|trainer.py:803] 2025-04-26 15:05:17,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1633
1631
[WARNING|trainer.py:803] 2025-04-26 15:05:19,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:05:19,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1624
[WARNING|trainer.py:803] 2025-04-26 15:05:20,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1634
1632
[WARNING|trainer.py:803] 2025-04-26 15:05:22,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:05:23,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1625
[WARNING|trainer.py:803] 2025-04-26 15:05:24,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1635
1633
[WARNING|trainer.py:803] 2025-04-26 15:05:25,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:05:26,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1626
[WARNING|trainer.py:803] 2025-04-26 15:05:27,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1636
1634
[WARNING|trainer.py:803] 2025-04-26 15:05:29,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:05:29,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1627
[WARNING|trainer.py:803] 2025-04-26 15:05:31,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1637
1635
[WARNING|trainer.py:803] 2025-04-26 15:05:32,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:05:33,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1628
[WARNING|trainer.py:803] 2025-04-26 15:05:34,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1638
[WARNING|trainer.py:803] 2025-04-26 15:05:35,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1636
[WARNING|trainer.py:803] 2025-04-26 15:05:36,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1629
[WARNING|trainer.py:803] 2025-04-26 15:05:37,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1639
[WARNING|trainer.py:803] 2025-04-26 15:05:39,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1637
[WARNING|trainer.py:803] 2025-04-26 15:05:40,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1630
[WARNING|trainer.py:803] 2025-04-26 15:05:41,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1640
[WARNING|trainer.py:803] 2025-04-26 15:05:42,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1638
[WARNING|trainer.py:803] 2025-04-26 15:05:43,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1631
[WARNING|trainer.py:803] 2025-04-26 15:05:44,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1641
[WARNING|trainer.py:803] 2025-04-26 15:05:45,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1639
[WARNING|trainer.py:803] 2025-04-26 15:05:46,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1632
[WARNING|trainer.py:803] 2025-04-26 15:05:48,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1642
[WARNING|trainer.py:803] 2025-04-26 15:05:49,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1640
[WARNING|trainer.py:803] 2025-04-26 15:05:50,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1633
[WARNING|trainer.py:803] 2025-04-26 15:05:51,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1643
[WARNING|trainer.py:803] 2025-04-26 15:05:52,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1641
[WARNING|trainer.py:803] 2025-04-26 15:05:53,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1634
[WARNING|trainer.py:803] 2025-04-26 15:05:54,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1644
[WARNING|trainer.py:803] 2025-04-26 15:05:55,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1642
[WARNING|trainer.py:803] 2025-04-26 15:05:57,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1635
[WARNING|trainer.py:803] 2025-04-26 15:05:58,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1645
[WARNING|trainer.py:803] 2025-04-26 15:05:59,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1643
[WARNING|trainer.py:803] 2025-04-26 15:06:00,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1636
1646
[WARNING|trainer.py:803] 2025-04-26 15:06:01,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:06:02,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1644
[WARNING|trainer.py:803] 2025-04-26 15:06:03,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1637
1647
[WARNING|trainer.py:803] 2025-04-26 15:06:05,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:06:05,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1645
[WARNING|trainer.py:803] 2025-04-26 15:06:07,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1638
1648
[WARNING|trainer.py:803] 2025-04-26 15:06:08,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:09,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1646
[WARNING|trainer.py:803] 2025-04-26 15:06:10,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1649
1639
[WARNING|trainer.py:803] 2025-04-26 15:06:11,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:12,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1647
1650
[WARNING|trainer.py:803] 2025-04-26 15:06:13,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1640
[WARNING|trainer.py:803] 2025-04-26 15:06:14,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:15,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1648
1651
[WARNING|trainer.py:803] 2025-04-26 15:06:17,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:17,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1641
[WARNING|trainer.py:803] 2025-04-26 15:06:19,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1649
1652
[WARNING|trainer.py:803] 2025-04-26 15:06:20,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:20,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1642
1650
[WARNING|trainer.py:803] 2025-04-26 15:06:22,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1653
[WARNING|trainer.py:803] 2025-04-26 15:06:22,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:23,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1643
1654
1651
[WARNING|trainer.py:803] 2025-04-26 15:06:25,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:25,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:25,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1655
1652
1644
[WARNING|trainer.py:803] 2025-04-26 15:06:28,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:28,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:29,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1656
1653
[WARNING|trainer.py:803] 2025-04-26 15:06:31,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:31,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1645
[WARNING|trainer.py:803] 2025-04-26 15:06:32,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1654
1657
[WARNING|trainer.py:803] 2025-04-26 15:06:34,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:34,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1646
[WARNING|trainer.py:803] 2025-04-26 15:06:35,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1655
1658
[WARNING|trainer.py:803] 2025-04-26 15:06:37,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:37,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1647
1656
1659
[WARNING|trainer.py:803] 2025-04-26 15:06:39,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:06:39,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:40,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1657
1648
1660
[WARNING|trainer.py:803] 2025-04-26 15:06:42,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:42,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:42,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1649
1661
1658
[WARNING|trainer.py:803] 2025-04-26 15:06:45,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:45,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:45,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1662
1659
1650
[WARNING|trainer.py:803] 2025-04-26 15:06:48,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:48,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:48,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1663
1660
1651
[WARNING|trainer.py:803] 2025-04-26 15:06:51,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:06:51,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:51,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1664
1661
1652
[WARNING|trainer.py:803] 2025-04-26 15:06:54,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:06:54,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:54,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1665
1662
[WARNING|trainer.py:803] 2025-04-26 15:06:56,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1653
[WARNING|trainer.py:803] 2025-04-26 15:06:57,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:06:57,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1666
1663
[WARNING|trainer.py:803] 2025-04-26 15:06:59,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1654
[WARNING|trainer.py:803] 2025-04-26 15:06:59,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:00,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1667
1664
[WARNING|trainer.py:803] 2025-04-26 15:07:02,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1655
[WARNING|trainer.py:803] 2025-04-26 15:07:02,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:07:03,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1668
1665
[WARNING|trainer.py:803] 2025-04-26 15:07:04,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1656
[WARNING|trainer.py:803] 2025-04-26 15:07:05,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:05,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1669
1666
1657
[WARNING|trainer.py:803] 2025-04-26 15:07:08,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:08,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:08,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1670
1667
[WARNING|trainer.py:803] 2025-04-26 15:07:10,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1658
[WARNING|trainer.py:803] 2025-04-26 15:07:10,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:11,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1671
1668
[WARNING|trainer.py:803] 2025-04-26 15:07:13,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1659
[WARNING|trainer.py:803] 2025-04-26 15:07:13,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:14,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1672
[WARNING|trainer.py:803] 2025-04-26 15:07:15,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1669
1660
[WARNING|trainer.py:803] 2025-04-26 15:07:16,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:17,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1673
1670
[WARNING|trainer.py:803] 2025-04-26 15:07:18,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:07:19,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1661
[WARNING|trainer.py:803] 2025-04-26 15:07:20,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1674
1671
[WARNING|trainer.py:803] 2025-04-26 15:07:21,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:21,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1662
[WARNING|trainer.py:803] 2025-04-26 15:07:22,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1675
1672
[WARNING|trainer.py:803] 2025-04-26 15:07:23,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:24,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1663
[WARNING|trainer.py:803] 2025-04-26 15:07:25,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1676
1673
[WARNING|trainer.py:803] 2025-04-26 15:07:26,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:07:26,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1664
1677
[WARNING|trainer.py:803] 2025-04-26 15:07:28,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1674
[WARNING|trainer.py:803] 2025-04-26 15:07:29,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:29,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1665
1678
[WARNING|trainer.py:803] 2025-04-26 15:07:31,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1675
[WARNING|trainer.py:803] 2025-04-26 15:07:32,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:32,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1666
[WARNING|trainer.py:803] 2025-04-26 15:07:33,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1676
1679
[WARNING|trainer.py:803] 2025-04-26 15:07:34,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:07:34,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1667
1677
[WARNING|trainer.py:803] 2025-04-26 15:07:36,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1680
[WARNING|trainer.py:803] 2025-04-26 15:07:37,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:37,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1668
1678
1681
[WARNING|trainer.py:803] 2025-04-26 15:07:39,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:40,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:40,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1669
1682
1679
[WARNING|trainer.py:803] 2025-04-26 15:07:42,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:07:43,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:43,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1670
1683
1680
[WARNING|trainer.py:803] 2025-04-26 15:07:45,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:45,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:45,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1671
1684
1681
[WARNING|trainer.py:803] 2025-04-26 15:07:48,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:48,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:48,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1672
1682
[WARNING|trainer.py:803] 2025-04-26 15:07:50,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1685
[WARNING|trainer.py:803] 2025-04-26 15:07:51,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:51,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1673
1683
[WARNING|trainer.py:803] 2025-04-26 15:07:53,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1686
[WARNING|trainer.py:803] 2025-04-26 15:07:54,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:07:54,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1674
[WARNING|trainer.py:803] 2025-04-26 15:07:56,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1684
1687
[WARNING|trainer.py:803] 2025-04-26 15:07:57,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:07:57,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1675
[WARNING|trainer.py:803] 2025-04-26 15:07:58,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1685
1688
[WARNING|trainer.py:803] 2025-04-26 15:08:00,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:08:00,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1676
[WARNING|trainer.py:803] 2025-04-26 15:08:01,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1686
1689
[WARNING|trainer.py:803] 2025-04-26 15:08:02,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:08:02,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1677
[WARNING|trainer.py:803] 2025-04-26 15:08:04,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1687
1690
[WARNING|trainer.py:803] 2025-04-26 15:08:05,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:08:05,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1678
[WARNING|trainer.py:803] 2025-04-26 15:08:07,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1688
1691
[WARNING|trainer.py:803] 2025-04-26 15:08:08,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:08:08,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1679
[WARNING|trainer.py:803] 2025-04-26 15:08:09,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1689
1692
[WARNING|trainer.py:803] 2025-04-26 15:08:11,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:11,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1680
[WARNING|trainer.py:803] 2025-04-26 15:08:12,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1690
1693
[WARNING|trainer.py:803] 2025-04-26 15:08:14,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:14,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1681
[WARNING|trainer.py:803] 2025-04-26 15:08:15,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1691
1694
[WARNING|trainer.py:803] 2025-04-26 15:08:16,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:08:17,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1682
[WARNING|trainer.py:803] 2025-04-26 15:08:18,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1692
1695
[WARNING|trainer.py:803] 2025-04-26 15:08:19,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:20,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1683
[WARNING|trainer.py:803] 2025-04-26 15:08:21,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1693
1696
[WARNING|trainer.py:803] 2025-04-26 15:08:22,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:08:22,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1684
[WARNING|trainer.py:803] 2025-04-26 15:08:23,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1694
1697
[WARNING|trainer.py:803] 2025-04-26 15:08:25,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:25,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1685
[WARNING|trainer.py:803] 2025-04-26 15:08:27,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1695
1698
[WARNING|trainer.py:803] 2025-04-26 15:08:28,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:28,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1686
[WARNING|trainer.py:803] 2025-04-26 15:08:29,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1699
1696
[WARNING|trainer.py:803] 2025-04-26 15:08:31,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:31,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1687
[WARNING|trainer.py:803] 2025-04-26 15:08:32,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1700
1697
[WARNING|trainer.py:803] 2025-04-26 15:08:34,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:34,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1688
[WARNING|trainer.py:803] 2025-04-26 15:08:35,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1701
1698
[WARNING|trainer.py:803] 2025-04-26 15:08:36,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:37,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1689
[WARNING|trainer.py:803] 2025-04-26 15:08:38,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1702
1699
[WARNING|trainer.py:803] 2025-04-26 15:08:39,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:40,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1690
[WARNING|trainer.py:803] 2025-04-26 15:08:41,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1703
1700
[WARNING|trainer.py:803] 2025-04-26 15:08:42,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:42,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1691
[WARNING|trainer.py:803] 2025-04-26 15:08:44,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1704
1701
[WARNING|trainer.py:803] 2025-04-26 15:08:45,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:45,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1692
[WARNING|trainer.py:803] 2025-04-26 15:08:47,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
1705
Yes

1702
[WARNING|trainer.py:803] 2025-04-26 15:08:47,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:48,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1693
1706
[WARNING|trainer.py:803] 2025-04-26 15:08:50,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1703
[WARNING|trainer.py:803] 2025-04-26 15:08:50,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:51,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1694
1707
[WARNING|trainer.py:803] 2025-04-26 15:08:52,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1704
[WARNING|trainer.py:803] 2025-04-26 15:08:53,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:53,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1695
1708
1705
[WARNING|trainer.py:803] 2025-04-26 15:08:55,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:56,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:08:56,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1696
1709
1706
[WARNING|trainer.py:803] 2025-04-26 15:08:58,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:08:58,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:08:59,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1697
1710
1707
[WARNING|trainer.py:803] 2025-04-26 15:09:01,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:09:01,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:02,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1711
1698
1708
[WARNING|trainer.py:803] 2025-04-26 15:09:04,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:04,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:04,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1712
1699
1709
[WARNING|trainer.py:803] 2025-04-26 15:09:07,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:07,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:07,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1713
1710
1700
[WARNING|trainer.py:803] 2025-04-26 15:09:09,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:10,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:10,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1714
1711
1701
[WARNING|trainer.py:803] 2025-04-26 15:09:12,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:12,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:13,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1715
1712
1702
[WARNING|trainer.py:803] 2025-04-26 15:09:15,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:15,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:16,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1716
1713
[WARNING|trainer.py:803] 2025-04-26 15:09:17,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:18,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1703
[WARNING|trainer.py:803] 2025-04-26 15:09:19,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1717
1714
[WARNING|trainer.py:803] 2025-04-26 15:09:20,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1704
[WARNING|trainer.py:803] 2025-04-26 15:09:21,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:21,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1718
1715
[WARNING|trainer.py:803] 2025-04-26 15:09:22,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1705
[WARNING|trainer.py:803] 2025-04-26 15:09:23,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:24,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1719
1716
[WARNING|trainer.py:803] 2025-04-26 15:09:25,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1706
[WARNING|trainer.py:803] 2025-04-26 15:09:26,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:26,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1720
1717
[WARNING|trainer.py:803] 2025-04-26 15:09:28,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:28,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1707
[WARNING|trainer.py:803] 2025-04-26 15:09:29,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1721
1718
[WARNING|trainer.py:803] 2025-04-26 15:09:30,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:09:31,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1708
[WARNING|trainer.py:803] 2025-04-26 15:09:32,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1722
1719
[WARNING|trainer.py:803] 2025-04-26 15:09:33,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:09:33,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1709
[WARNING|trainer.py:803] 2025-04-26 15:09:35,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1723
1720
[WARNING|trainer.py:803] 2025-04-26 15:09:36,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:36,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1710
[WARNING|trainer.py:803] 2025-04-26 15:09:37,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1724
1721
[WARNING|trainer.py:803] 2025-04-26 15:09:39,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:09:39,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1711
[WARNING|trainer.py:803] 2025-04-26 15:09:40,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1725
1722
[WARNING|trainer.py:803] 2025-04-26 15:09:41,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:41,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1712
[WARNING|trainer.py:803] 2025-04-26 15:09:43,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1726
1723
[WARNING|trainer.py:803] 2025-04-26 15:09:44,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:44,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1713
[WARNING|trainer.py:803] 2025-04-26 15:09:45,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1727
1724
[WARNING|trainer.py:803] 2025-04-26 15:09:47,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:09:47,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1714
[WARNING|trainer.py:803] 2025-04-26 15:09:48,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1728
1725
[WARNING|trainer.py:803] 2025-04-26 15:09:49,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:09:50,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1715
[WARNING|trainer.py:803] 2025-04-26 15:09:51,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1729
1726
[WARNING|trainer.py:803] 2025-04-26 15:09:52,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:52,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1716
1730
[WARNING|trainer.py:803] 2025-04-26 15:09:53,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1727
[WARNING|trainer.py:803] 2025-04-26 15:09:54,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:55,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1717
1731
[WARNING|trainer.py:803] 2025-04-26 15:09:56,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1728
[WARNING|trainer.py:803] 2025-04-26 15:09:57,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:09:57,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1718
1732
[WARNING|trainer.py:803] 2025-04-26 15:09:59,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1729
[WARNING|trainer.py:803] 2025-04-26 15:10:00,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:00,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1719
1730
1733
[WARNING|trainer.py:803] 2025-04-26 15:10:02,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:02,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:02,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1720
1731
1734
[WARNING|trainer.py:803] 2025-04-26 15:10:04,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:10:05,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:05,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1721
1732
1735
[WARNING|trainer.py:803] 2025-04-26 15:10:07,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:07,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:08,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1722
1733
1736
[WARNING|trainer.py:803] 2025-04-26 15:10:10,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:10,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:10,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1723
1734
1737
[WARNING|trainer.py:803] 2025-04-26 15:10:13,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:13,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:13,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1724
1735
1738
[WARNING|trainer.py:803] 2025-04-26 15:10:16,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:16,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:16,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1725
1739
1736
[WARNING|trainer.py:803] 2025-04-26 15:10:19,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:19,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:19,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1726
1740
1737
[WARNING|trainer.py:803] 2025-04-26 15:10:21,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:21,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:22,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1727
1741
1738
[WARNING|trainer.py:803] 2025-04-26 15:10:24,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:24,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:24,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1728
1742
1739
[WARNING|trainer.py:803] 2025-04-26 15:10:27,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:10:27,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:27,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1743
1729
1740
[WARNING|trainer.py:803] 2025-04-26 15:10:30,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:30,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:30,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1730
1744
1741
[WARNING|trainer.py:803] 2025-04-26 15:10:32,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:32,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:32,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1731
1745
1742
[WARNING|trainer.py:803] 2025-04-26 15:10:35,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:35,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:35,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1732
1743
1746
[WARNING|trainer.py:803] 2025-04-26 15:10:38,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:38,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:38,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1733
1747
1744
[WARNING|trainer.py:803] 2025-04-26 15:10:41,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:41,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:41,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1745
1734
1748
[WARNING|trainer.py:803] 2025-04-26 15:10:43,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:44,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:44,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1735
1746
1749
[WARNING|trainer.py:803] 2025-04-26 15:10:46,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:46,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:47,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1747
1736
1750
[WARNING|trainer.py:803] 2025-04-26 15:10:49,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:49,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:10:50,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1737
1748
1751
[WARNING|trainer.py:803] 2025-04-26 15:10:52,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:52,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:52,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1738
1749
1752
[WARNING|trainer.py:803] 2025-04-26 15:10:55,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:55,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:55,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1739
1753
1750
[WARNING|trainer.py:803] 2025-04-26 15:10:58,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:10:58,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:10:58,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1754
1740
1751
[WARNING|trainer.py:803] 2025-04-26 15:11:00,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:00,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:11:01,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1755
1741
1752
[WARNING|trainer.py:803] 2025-04-26 15:11:03,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:03,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:03,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1756
1753
1742
[WARNING|trainer.py:803] 2025-04-26 15:11:06,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:06,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:11:06,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1757
1754
1743
[WARNING|trainer.py:803] 2025-04-26 15:11:09,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:09,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:09,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1755
1744
1758
[WARNING|trainer.py:803] 2025-04-26 15:11:11,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:12,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:11:12,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1745
1756
1759
[WARNING|trainer.py:803] 2025-04-26 15:11:14,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:11:14,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:15,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1760
1757
1746
[WARNING|trainer.py:803] 2025-04-26 15:11:17,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:17,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:17,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1761
1747
1758
[WARNING|trainer.py:803] 2025-04-26 15:11:20,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:20,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:20,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1762
1759
[WARNING|trainer.py:803] 2025-04-26 15:11:22,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1748
[WARNING|trainer.py:803] 2025-04-26 15:11:23,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:23,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1763
1760
[WARNING|trainer.py:803] 2025-04-26 15:11:25,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1749
[WARNING|trainer.py:803] 2025-04-26 15:11:25,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:26,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1764
1761
[WARNING|trainer.py:803] 2025-04-26 15:11:27,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:28,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1750
[WARNING|trainer.py:803] 2025-04-26 15:11:29,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1765
1762
[WARNING|trainer.py:803] 2025-04-26 15:11:30,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:30,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1751
[WARNING|trainer.py:803] 2025-04-26 15:11:32,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1766
1763
[WARNING|trainer.py:803] 2025-04-26 15:11:33,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:11:33,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1752
1767
[WARNING|trainer.py:803] 2025-04-26 15:11:35,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1764
[WARNING|trainer.py:803] 2025-04-26 15:11:35,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:11:36,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1753
1768
[WARNING|trainer.py:803] 2025-04-26 15:11:37,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1765
[WARNING|trainer.py:803] 2025-04-26 15:11:38,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:38,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1754
1769
[WARNING|trainer.py:803] 2025-04-26 15:11:40,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1766
[WARNING|trainer.py:803] 2025-04-26 15:11:40,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:41,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1755
1770
1767
[WARNING|trainer.py:803] 2025-04-26 15:11:43,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:43,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:43,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1756
1771
1768
[WARNING|trainer.py:803] 2025-04-26 15:11:45,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:46,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:46,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1757
1772
1769
[WARNING|trainer.py:803] 2025-04-26 15:11:48,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:49,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:11:49,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1773
1770
1758
[WARNING|trainer.py:803] 2025-04-26 15:11:51,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:11:51,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:52,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1774
1759
1771
[WARNING|trainer.py:803] 2025-04-26 15:11:54,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:11:54,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:54,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1760
1772
1775
[WARNING|trainer.py:803] 2025-04-26 15:11:57,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:11:57,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:11:57,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1761
1773
1776
[WARNING|trainer.py:803] 2025-04-26 15:12:00,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:00,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:00,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1762
1777
1774
[WARNING|trainer.py:803] 2025-04-26 15:12:02,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:03,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:03,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1778
1763
1775
[WARNING|trainer.py:803] 2025-04-26 15:12:05,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:05,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:06,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1779
1764
[WARNING|trainer.py:803] 2025-04-26 15:12:08,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1776
[WARNING|trainer.py:803] 2025-04-26 15:12:08,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:09,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1780
1765
1777
[WARNING|trainer.py:803] 2025-04-26 15:12:10,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:11,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:11,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1781
1766
1778
[WARNING|trainer.py:803] 2025-04-26 15:12:13,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:13,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:14,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1782
1767
1779
[WARNING|trainer.py:803] 2025-04-26 15:12:16,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:16,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:16,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1768
1780
1783
[WARNING|trainer.py:803] 2025-04-26 15:12:19,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:19,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:19,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1784
1781
1769
[WARNING|trainer.py:803] 2025-04-26 15:12:21,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:22,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:22,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1785
1782
1770
[WARNING|trainer.py:803] 2025-04-26 15:12:24,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:24,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:25,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1771
1786
1783
[WARNING|trainer.py:803] 2025-04-26 15:12:27,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:27,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:27,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1772
1784
1787
[WARNING|trainer.py:803] 2025-04-26 15:12:30,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:30,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:30,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1773
1788
1785
[WARNING|trainer.py:803] 2025-04-26 15:12:33,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:12:33,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:33,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1789
1774
1786
[WARNING|trainer.py:803] 2025-04-26 15:12:36,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:36,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:36,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1790
1775
1787
[WARNING|trainer.py:803] 2025-04-26 15:12:39,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:12:39,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:39,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1791
1788
[WARNING|trainer.py:803] 2025-04-26 15:12:41,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1776
[WARNING|trainer.py:803] 2025-04-26 15:12:42,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:42,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1792
1789
[WARNING|trainer.py:803] 2025-04-26 15:12:44,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1777
[WARNING|trainer.py:803] 2025-04-26 15:12:45,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:45,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1793
[WARNING|trainer.py:803] 2025-04-26 15:12:46,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1778
1790
[WARNING|trainer.py:803] 2025-04-26 15:12:47,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:47,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1794
[WARNING|trainer.py:803] 2025-04-26 15:12:49,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1779
1791
[WARNING|trainer.py:803] 2025-04-26 15:12:50,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:12:50,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1795
[WARNING|trainer.py:803] 2025-04-26 15:12:51,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1792
1780
[WARNING|trainer.py:803] 2025-04-26 15:12:53,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:53,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1796
[WARNING|trainer.py:803] 2025-04-26 15:12:54,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1793
1781
[WARNING|trainer.py:803] 2025-04-26 15:12:56,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:56,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1797
[WARNING|trainer.py:803] 2025-04-26 15:12:57,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1794
1782
[WARNING|trainer.py:803] 2025-04-26 15:12:58,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:12:58,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1798
[WARNING|trainer.py:803] 2025-04-26 15:12:59,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1795
1783
[WARNING|trainer.py:803] 2025-04-26 15:13:01,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1799
[WARNING|trainer.py:803] 2025-04-26 15:13:01,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:13:02,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1796
1784
[WARNING|trainer.py:803] 2025-04-26 15:13:04,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1800
[WARNING|trainer.py:803] 2025-04-26 15:13:04,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:05,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1797
1801
1785
[WARNING|trainer.py:803] 2025-04-26 15:13:06,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:07,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:07,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1798
1802
1786
[WARNING|trainer.py:803] 2025-04-26 15:13:09,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:09,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:13:10,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1803
1799
[WARNING|trainer.py:803] 2025-04-26 15:13:11,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:12,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1787
[WARNING|trainer.py:803] 2025-04-26 15:13:13,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1804
1800
[WARNING|trainer.py:803] 2025-04-26 15:13:14,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:14,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1788
[WARNING|trainer.py:803] 2025-04-26 15:13:15,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1805
1801
[WARNING|trainer.py:803] 2025-04-26 15:13:16,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:16,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1789
1806
1802
[WARNING|trainer.py:803] 2025-04-26 15:13:18,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:18,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:19,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1807
1790
[WARNING|trainer.py:803] 2025-04-26 15:13:20,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1803
[WARNING|trainer.py:803] 2025-04-26 15:13:21,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:21,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1808
[WARNING|trainer.py:803] 2025-04-26 15:13:23,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1791
1804
[WARNING|trainer.py:803] 2025-04-26 15:13:23,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:24,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1809
[WARNING|trainer.py:803] 2025-04-26 15:13:25,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1805
1792
[WARNING|trainer.py:803] 2025-04-26 15:13:26,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1810
[WARNING|trainer.py:803] 2025-04-26 15:13:26,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:27,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1806
[WARNING|trainer.py:803] 2025-04-26 15:13:28,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1793
1811
[WARNING|trainer.py:803] 2025-04-26 15:13:29,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:29,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1807
[WARNING|trainer.py:803] 2025-04-26 15:13:30,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1812
1794
[WARNING|trainer.py:803] 2025-04-26 15:13:31,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1808
[WARNING|trainer.py:803] 2025-04-26 15:13:31,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:32,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1813
[WARNING|trainer.py:803] 2025-04-26 15:13:33,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1795
1809
[WARNING|trainer.py:803] 2025-04-26 15:13:34,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1814
[WARNING|trainer.py:803] 2025-04-26 15:13:34,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:35,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1810
1796
1815
[WARNING|trainer.py:803] 2025-04-26 15:13:36,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:37,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:37,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1811
[WARNING|trainer.py:803] 2025-04-26 15:13:38,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1797
1816
[WARNING|trainer.py:803] 2025-04-26 15:13:39,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:40,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1812
[WARNING|trainer.py:803] 2025-04-26 15:13:41,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1817
1798
[WARNING|trainer.py:803] 2025-04-26 15:13:42,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1813
[WARNING|trainer.py:803] 2025-04-26 15:13:42,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:43,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1818
1799
1814
[WARNING|trainer.py:803] 2025-04-26 15:13:44,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:45,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:45,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1819
1815
[WARNING|trainer.py:803] 2025-04-26 15:13:46,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1800
[WARNING|trainer.py:803] 2025-04-26 15:13:47,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:47,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1820
[WARNING|trainer.py:803] 2025-04-26 15:13:49,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1816
1801
[WARNING|trainer.py:803] 2025-04-26 15:13:49,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:50,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1821
[WARNING|trainer.py:803] 2025-04-26 15:13:51,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1817
1802
[WARNING|trainer.py:803] 2025-04-26 15:13:51,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:13:52,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1822
1818
[WARNING|trainer.py:803] 2025-04-26 15:13:53,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1803
[WARNING|trainer.py:803] 2025-04-26 15:13:54,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:13:54,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1823
1819
[WARNING|trainer.py:803] 2025-04-26 15:13:56,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:13:56,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1804
[WARNING|trainer.py:803] 2025-04-26 15:13:57,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1820
1824
[WARNING|trainer.py:803] 2025-04-26 15:13:58,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:13:58,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1805
[WARNING|trainer.py:803] 2025-04-26 15:13:59,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1821
1825
[WARNING|trainer.py:803] 2025-04-26 15:14:00,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:00,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1806
[WARNING|trainer.py:803] 2025-04-26 15:14:01,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1822
1826
1807
[WARNING|trainer.py:803] 2025-04-26 15:14:02,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:02,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:03,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1827
1823
[WARNING|trainer.py:803] 2025-04-26 15:14:04,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1808
[WARNING|trainer.py:803] 2025-04-26 15:14:05,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:14:05,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1828
1824
[WARNING|trainer.py:803] 2025-04-26 15:14:07,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1809
[WARNING|trainer.py:803] 2025-04-26 15:14:07,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:08,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1829
1825
1810
[WARNING|trainer.py:803] 2025-04-26 15:14:09,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:09,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:10,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1830
1811
1826
[WARNING|trainer.py:803] 2025-04-26 15:14:12,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:12,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:12,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1827
1812
1831
[WARNING|trainer.py:803] 2025-04-26 15:14:14,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:14,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:14:14,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1813
1832
1828
[WARNING|trainer.py:803] 2025-04-26 15:14:16,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:16,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:16,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1814
1833
1829
[WARNING|trainer.py:803] 2025-04-26 15:14:18,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:18,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:18,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1815
1834
1830
[WARNING|trainer.py:803] 2025-04-26 15:14:20,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:20,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:21,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1835
1816
1831
[WARNING|trainer.py:803] 2025-04-26 15:14:23,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:23,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:23,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1836
1817
1832
[WARNING|trainer.py:803] 2025-04-26 15:14:25,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:25,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:25,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1837
1818
1833
[WARNING|trainer.py:803] 2025-04-26 15:14:27,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:27,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:27,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1838
1819
1834
[WARNING|trainer.py:803] 2025-04-26 15:14:29,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:29,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:30,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1839
1820
1835
[WARNING|trainer.py:803] 2025-04-26 15:14:31,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:32,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:32,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1840
1836
[WARNING|trainer.py:803] 2025-04-26 15:14:33,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1821
[WARNING|trainer.py:803] 2025-04-26 15:14:34,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:14:34,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1841
1837
[WARNING|trainer.py:803] 2025-04-26 15:14:35,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1822
[WARNING|trainer.py:803] 2025-04-26 15:14:36,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:36,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1842
1838
[WARNING|trainer.py:803] 2025-04-26 15:14:37,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1823
[WARNING|trainer.py:803] 2025-04-26 15:14:38,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:39,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1843
1839
[WARNING|trainer.py:803] 2025-04-26 15:14:40,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:40,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1824
1840
[WARNING|trainer.py:803] 2025-04-26 15:14:41,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1844
[WARNING|trainer.py:803] 2025-04-26 15:14:42,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:42,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1825
1841
[WARNING|trainer.py:803] 2025-04-26 15:14:43,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1845
[WARNING|trainer.py:803] 2025-04-26 15:14:44,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:44,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1826
[WARNING|trainer.py:803] 2025-04-26 15:14:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1842
1846
[WARNING|trainer.py:803] 2025-04-26 15:14:47,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:47,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1827
[WARNING|trainer.py:803] 2025-04-26 15:14:48,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1847
1843
[WARNING|trainer.py:803] 2025-04-26 15:14:49,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:14:49,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1828
1848
[WARNING|trainer.py:803] 2025-04-26 15:14:50,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1844
[WARNING|trainer.py:803] 2025-04-26 15:14:51,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1829
[WARNING|trainer.py:803] 2025-04-26 15:14:51,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1849
[WARNING|trainer.py:803] 2025-04-26 15:14:52,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1845
[WARNING|trainer.py:803] 2025-04-26 15:14:53,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:54,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1830
1850
[WARNING|trainer.py:803] 2025-04-26 15:14:55,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:14:55,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1846
[WARNING|trainer.py:803] 2025-04-26 15:14:56,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1831
1851
[WARNING|trainer.py:803] 2025-04-26 15:14:57,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1847
[WARNING|trainer.py:803] 2025-04-26 15:14:58,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:14:58,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1832
1852
[WARNING|trainer.py:803] 2025-04-26 15:14:59,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1848
[WARNING|trainer.py:803] 2025-04-26 15:15:00,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:15:00,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1833
[WARNING|trainer.py:803] 2025-04-26 15:15:01,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1853
1849
[WARNING|trainer.py:803] 2025-04-26 15:15:02,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:03,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1834
[WARNING|trainer.py:803] 2025-04-26 15:15:04,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1854
1850
[WARNING|trainer.py:803] 2025-04-26 15:15:05,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:05,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1835
[WARNING|trainer.py:803] 2025-04-26 15:15:06,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1855
1851
1836
[WARNING|trainer.py:803] 2025-04-26 15:15:07,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:07,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:08,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1856
1852
1837
[WARNING|trainer.py:803] 2025-04-26 15:15:09,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:15:10,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:15:10,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1857
1853
[WARNING|trainer.py:803] 2025-04-26 15:15:11,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1838
[WARNING|trainer.py:803] 2025-04-26 15:15:12,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:12,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1858
1854
1839
[WARNING|trainer.py:803] 2025-04-26 15:15:13,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:14,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:14,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1859
1840
1855
[WARNING|trainer.py:803] 2025-04-26 15:15:16,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:16,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:16,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1860
1841
1856
[WARNING|trainer.py:803] 2025-04-26 15:15:18,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:15:19,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:19,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1861
1857
[WARNING|trainer.py:803] 2025-04-26 15:15:20,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1842
[WARNING|trainer.py:803] 2025-04-26 15:15:21,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:15:21,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1862
1858
[WARNING|trainer.py:803] 2025-04-26 15:15:22,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1843
[WARNING|trainer.py:803] 2025-04-26 15:15:23,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:23,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1863
1859
[WARNING|trainer.py:803] 2025-04-26 15:15:24,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1844
[WARNING|trainer.py:803] 2025-04-26 15:15:25,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1864
[WARNING|trainer.py:803] 2025-04-26 15:15:26,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1860
[WARNING|trainer.py:803] 2025-04-26 15:15:27,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:27,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1845
1865
1861
[WARNING|trainer.py:803] 2025-04-26 15:15:28,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:29,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:29,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1846
1866
1862
[WARNING|trainer.py:803] 2025-04-26 15:15:31,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:31,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:31,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1847
1867
1863
[WARNING|trainer.py:803] 2025-04-26 15:15:33,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:15:33,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:15:33,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1848
1868
1864
[WARNING|trainer.py:803] 2025-04-26 15:15:35,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:35,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:36,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1849
1869
1865
[WARNING|trainer.py:803] 2025-04-26 15:15:37,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:37,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:38,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1850
1870
1866
[WARNING|trainer.py:803] 2025-04-26 15:15:40,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:40,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:40,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1871
1867
1851
[WARNING|trainer.py:803] 2025-04-26 15:15:42,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:42,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:15:42,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1872
1868
1852
[WARNING|trainer.py:803] 2025-04-26 15:15:44,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:44,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:45,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1873
1869
1853
[WARNING|trainer.py:803] 2025-04-26 15:15:46,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:46,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:47,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1874
1870
1854
[WARNING|trainer.py:803] 2025-04-26 15:15:48,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:49,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:49,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1875
1871
[WARNING|trainer.py:803] 2025-04-26 15:15:51,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1855
[WARNING|trainer.py:803] 2025-04-26 15:15:51,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:15:52,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1876
1872
[WARNING|trainer.py:803] 2025-04-26 15:15:53,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1856
[WARNING|trainer.py:803] 2025-04-26 15:15:53,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:54,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1877
1873
[WARNING|trainer.py:803] 2025-04-26 15:15:55,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1857
[WARNING|trainer.py:803] 2025-04-26 15:15:55,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:56,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1878
1874
[WARNING|trainer.py:803] 2025-04-26 15:15:57,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1858
[WARNING|trainer.py:803] 2025-04-26 15:15:57,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:15:58,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1879
1875
[WARNING|trainer.py:803] 2025-04-26 15:15:59,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1859
[WARNING|trainer.py:803] 2025-04-26 15:16:00,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:16:00,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1880
1876
[WARNING|trainer.py:803] 2025-04-26 15:16:01,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:02,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1860
1881
[WARNING|trainer.py:803] 2025-04-26 15:16:03,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1877
[WARNING|trainer.py:803] 2025-04-26 15:16:03,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1861
[WARNING|trainer.py:803] 2025-04-26 15:16:04,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1882
[WARNING|trainer.py:803] 2025-04-26 15:16:04,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1878
[WARNING|trainer.py:803] 2025-04-26 15:16:05,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:06,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1862
[WARNING|trainer.py:803] 2025-04-26 15:16:07,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1883
1879
[WARNING|trainer.py:803] 2025-04-26 15:16:08,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1863
[WARNING|trainer.py:803] 2025-04-26 15:16:08,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:09,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1880
1884
1864
[WARNING|trainer.py:803] 2025-04-26 15:16:10,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:11,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:11,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1881
1885
[WARNING|trainer.py:803] 2025-04-26 15:16:12,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1865
[WARNING|trainer.py:803] 2025-04-26 15:16:13,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:16:13,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1882
1886
[WARNING|trainer.py:803] 2025-04-26 15:16:14,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1866
[WARNING|trainer.py:803] 2025-04-26 15:16:15,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:15,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1883
1867
1887
[WARNING|trainer.py:803] 2025-04-26 15:16:17,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:18,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:18,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1884
1888
1868
[WARNING|trainer.py:803] 2025-04-26 15:16:20,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:20,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:20,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1889
1869
1885
[WARNING|trainer.py:803] 2025-04-26 15:16:22,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:22,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:22,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1890
1870
1886
[WARNING|trainer.py:803] 2025-04-26 15:16:24,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:24,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:25,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1891
1871
[WARNING|trainer.py:803] 2025-04-26 15:16:26,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1887
[WARNING|trainer.py:803] 2025-04-26 15:16:26,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:27,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1892
1872
1888
[WARNING|trainer.py:803] 2025-04-26 15:16:29,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:29,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:16:29,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1893
1889
1873
[WARNING|trainer.py:803] 2025-04-26 15:16:31,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:31,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:31,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1894
1890
1874
[WARNING|trainer.py:803] 2025-04-26 15:16:33,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:33,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:33,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1895
1891
1875
[WARNING|trainer.py:803] 2025-04-26 15:16:35,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:35,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:16:35,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1896
1876
1892
[WARNING|trainer.py:803] 2025-04-26 15:16:37,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:38,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:16:38,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1897
1877
1893
[WARNING|trainer.py:803] 2025-04-26 15:16:39,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:40,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:40,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1878
1894
1898
[WARNING|trainer.py:803] 2025-04-26 15:16:42,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:42,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:42,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1879
1895
1899
[WARNING|trainer.py:803] 2025-04-26 15:16:44,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:44,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:45,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1880
1896
1900
[WARNING|trainer.py:803] 2025-04-26 15:16:46,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:47,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:47,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1881
1897
1901
[WARNING|trainer.py:803] 2025-04-26 15:16:48,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:49,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1882
1902
[WARNING|trainer.py:803] 2025-04-26 15:16:50,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1898
[WARNING|trainer.py:803] 2025-04-26 15:16:51,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:52,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1883
1903
[WARNING|trainer.py:803] 2025-04-26 15:16:53,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1899
[WARNING|trainer.py:803] 2025-04-26 15:16:53,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:54,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1884
1904
1900
[WARNING|trainer.py:803] 2025-04-26 15:16:56,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:56,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:16:56,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1905
1885
1901
[WARNING|trainer.py:803] 2025-04-26 15:16:58,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:16:58,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:16:59,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1906
1886
1902
[WARNING|trainer.py:803] 2025-04-26 15:17:00,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:01,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:01,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1907
1903
1887
[WARNING|trainer.py:803] 2025-04-26 15:17:02,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:03,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:03,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1908
1888
1904
[WARNING|trainer.py:803] 2025-04-26 15:17:05,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:05,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:05,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1909
1889
1905
[WARNING|trainer.py:803] 2025-04-26 15:17:07,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:07,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:08,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1910
1890
[WARNING|trainer.py:803] 2025-04-26 15:17:09,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1906
[WARNING|trainer.py:803] 2025-04-26 15:17:09,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:10,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1911
1891
[WARNING|trainer.py:803] 2025-04-26 15:17:11,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1907
[WARNING|trainer.py:803] 2025-04-26 15:17:12,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:17:12,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1912
[WARNING|trainer.py:803] 2025-04-26 15:17:13,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1908
1892
[WARNING|trainer.py:803] 2025-04-26 15:17:14,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:14,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1913
1909
[WARNING|trainer.py:803] 2025-04-26 15:17:15,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1893
[WARNING|trainer.py:803] 2025-04-26 15:17:16,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:16,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1914
1910
1894
[WARNING|trainer.py:803] 2025-04-26 15:17:18,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:17:18,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:19,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1915
1911
[WARNING|trainer.py:803] 2025-04-26 15:17:20,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1895
[WARNING|trainer.py:803] 2025-04-26 15:17:20,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:21,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1916
1912
[WARNING|trainer.py:803] 2025-04-26 15:17:22,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1896
[WARNING|trainer.py:803] 2025-04-26 15:17:22,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:23,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1917
1913
1897
[WARNING|trainer.py:803] 2025-04-26 15:17:25,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:25,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:17:25,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1918
1914
[WARNING|trainer.py:803] 2025-04-26 15:17:27,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:27,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1898
[WARNING|trainer.py:803] 2025-04-26 15:17:28,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1919
1915
[WARNING|trainer.py:803] 2025-04-26 15:17:29,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:29,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1899
[WARNING|trainer.py:803] 2025-04-26 15:17:30,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1916
1920
[WARNING|trainer.py:803] 2025-04-26 15:17:31,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:32,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1900
[WARNING|trainer.py:803] 2025-04-26 15:17:33,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1917
1921
[WARNING|trainer.py:803] 2025-04-26 15:17:34,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:34,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1901
[WARNING|trainer.py:803] 2025-04-26 15:17:35,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1918
1922
[WARNING|trainer.py:803] 2025-04-26 15:17:36,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:36,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1902
[WARNING|trainer.py:803] 2025-04-26 15:17:37,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1919
1923
[WARNING|trainer.py:803] 2025-04-26 15:17:38,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:39,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1903
[WARNING|trainer.py:803] 2025-04-26 15:17:40,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1920
1924
[WARNING|trainer.py:803] 2025-04-26 15:17:41,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1904
[WARNING|trainer.py:803] 2025-04-26 15:17:41,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:42,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1921
1925
[WARNING|trainer.py:803] 2025-04-26 15:17:43,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1905
[WARNING|trainer.py:803] 2025-04-26 15:17:44,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:44,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1922
1926
1906
[WARNING|trainer.py:803] 2025-04-26 15:17:46,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:46,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:46,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1923
1927
1907
[WARNING|trainer.py:803] 2025-04-26 15:17:48,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:48,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:49,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1924
1928
1908
[WARNING|trainer.py:803] 2025-04-26 15:17:50,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:50,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:17:51,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1925
1929
1909
[WARNING|trainer.py:803] 2025-04-26 15:17:53,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:17:53,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:53,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1926
1930
1910
[WARNING|trainer.py:803] 2025-04-26 15:17:55,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:55,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:55,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1931
1911
1927
[WARNING|trainer.py:803] 2025-04-26 15:17:57,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:57,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:57,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1928
1912
1932
[WARNING|trainer.py:803] 2025-04-26 15:17:59,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:17:59,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:17:59,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1929
1933
1913
[WARNING|trainer.py:803] 2025-04-26 15:18:02,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:02,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:02,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1930
1934
1914
[WARNING|trainer.py:803] 2025-04-26 15:18:04,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:04,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:04,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1931
1935
1915
[WARNING|trainer.py:803] 2025-04-26 15:18:06,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:06,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:06,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1932
1936
1916
[WARNING|trainer.py:803] 2025-04-26 15:18:08,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:08,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:18:08,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1933
1937
1917
[WARNING|trainer.py:803] 2025-04-26 15:18:10,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:10,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:11,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1934
1938
1918
[WARNING|trainer.py:803] 2025-04-26 15:18:13,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:13,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:13,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1939
1935
1919
[WARNING|trainer.py:803] 2025-04-26 15:18:15,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:15,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:15,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1940
1936
[WARNING|trainer.py:803] 2025-04-26 15:18:17,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:17,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1920
[WARNING|trainer.py:803] 2025-04-26 15:18:18,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1941
1937
[WARNING|trainer.py:803] 2025-04-26 15:18:19,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:19,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1921
[WARNING|trainer.py:803] 2025-04-26 15:18:20,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1938
1942
[WARNING|trainer.py:803] 2025-04-26 15:18:21,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:21,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1922
1939
1943
[WARNING|trainer.py:803] 2025-04-26 15:18:23,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:23,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:24,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1923
1940
1944
[WARNING|trainer.py:803] 2025-04-26 15:18:25,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:25,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:26,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1924
1941
1945
[WARNING|trainer.py:803] 2025-04-26 15:18:28,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:28,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:28,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1942
1925
1946
[WARNING|trainer.py:803] 2025-04-26 15:18:30,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:30,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:30,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1943
1926
1947
[WARNING|trainer.py:803] 2025-04-26 15:18:32,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:32,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:32,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1944
1927
1948
[WARNING|trainer.py:803] 2025-04-26 15:18:34,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:35,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:35,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1928
1949
1945
[WARNING|trainer.py:803] 2025-04-26 15:18:37,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:37,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:37,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1946
1929
1950
[WARNING|trainer.py:803] 2025-04-26 15:18:39,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:39,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:39,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1947
1930
1951
[WARNING|trainer.py:803] 2025-04-26 15:18:41,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:41,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:41,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1948
1931
1952
[WARNING|trainer.py:803] 2025-04-26 15:18:43,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:44,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:44,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1949
1932
[WARNING|trainer.py:803] 2025-04-26 15:18:45,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1953
[WARNING|trainer.py:803] 2025-04-26 15:18:46,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:46,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1950
1933
1954
[WARNING|trainer.py:803] 2025-04-26 15:18:48,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:48,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:48,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1951
1934
1955
[WARNING|trainer.py:803] 2025-04-26 15:18:50,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:50,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:18:51,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1952
1935
1956
[WARNING|trainer.py:803] 2025-04-26 15:18:53,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:53,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:53,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1936
1953
1957
[WARNING|trainer.py:803] 2025-04-26 15:18:55,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:18:55,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:18:55,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1937
1954
[WARNING|trainer.py:803] 2025-04-26 15:18:57,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:18:57,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1958
1938
[WARNING|trainer.py:803] 2025-04-26 15:18:58,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1955
[WARNING|trainer.py:803] 2025-04-26 15:18:59,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:18:59,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1959
[WARNING|trainer.py:803] 2025-04-26 15:19:00,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1939
1956
[WARNING|trainer.py:803] 2025-04-26 15:19:01,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:02,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1960
1940
[WARNING|trainer.py:803] 2025-04-26 15:19:03,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1957
[WARNING|trainer.py:803] 2025-04-26 15:19:03,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:04,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1961
1941
[WARNING|trainer.py:803] 2025-04-26 15:19:05,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:06,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1958
1962
[WARNING|trainer.py:803] 2025-04-26 15:19:07,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1942
[WARNING|trainer.py:803] 2025-04-26 15:19:07,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:08,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1959
1963
[WARNING|trainer.py:803] 2025-04-26 15:19:09,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1943
[WARNING|trainer.py:803] 2025-04-26 15:19:09,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:10,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1960
1964
[WARNING|trainer.py:803] 2025-04-26 15:19:11,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1944
[WARNING|trainer.py:803] 2025-04-26 15:19:12,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:12,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1961
1965
[WARNING|trainer.py:803] 2025-04-26 15:19:14,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1945
[WARNING|trainer.py:803] 2025-04-26 15:19:14,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:15,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1962
1966
[WARNING|trainer.py:803] 2025-04-26 15:19:16,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1946
[WARNING|trainer.py:803] 2025-04-26 15:19:16,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:17,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1963
1967
1947
[WARNING|trainer.py:803] 2025-04-26 15:19:18,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:19,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:19,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1964
1968
1948
[WARNING|trainer.py:803] 2025-04-26 15:19:21,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:21,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:21,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1965
1949
[WARNING|trainer.py:803] 2025-04-26 15:19:23,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1969
[WARNING|trainer.py:803] 2025-04-26 15:19:23,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:24,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1966
[WARNING|trainer.py:803] 2025-04-26 15:19:25,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1970
1950
[WARNING|trainer.py:803] 2025-04-26 15:19:26,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:26,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1967
1971
1951
[WARNING|trainer.py:803] 2025-04-26 15:19:27,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:28,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:28,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1968
1972
1952
[WARNING|trainer.py:803] 2025-04-26 15:19:30,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:30,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:31,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1973
1969
[WARNING|trainer.py:803] 2025-04-26 15:19:32,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1953
[WARNING|trainer.py:803] 2025-04-26 15:19:33,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:33,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1974
1970
[WARNING|trainer.py:803] 2025-04-26 15:19:34,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1954
[WARNING|trainer.py:803] 2025-04-26 15:19:35,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:35,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1975
1971
[WARNING|trainer.py:803] 2025-04-26 15:19:36,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:37,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1955
[WARNING|trainer.py:803] 2025-04-26 15:19:38,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1976
1972
[WARNING|trainer.py:803] 2025-04-26 15:19:38,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:39,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1956
1977
1973
[WARNING|trainer.py:803] 2025-04-26 15:19:40,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:40,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:41,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1957
1974
[WARNING|trainer.py:803] 2025-04-26 15:19:42,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1978
[WARNING|trainer.py:803] 2025-04-26 15:19:43,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:43,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1975
1958
1979
[WARNING|trainer.py:803] 2025-04-26 15:19:45,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:45,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:45,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1976
1959
1980
[WARNING|trainer.py:803] 2025-04-26 15:19:47,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:47,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:48,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1977
1960
1981
[WARNING|trainer.py:803] 2025-04-26 15:19:49,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:50,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:50,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1978
1961
1982
[WARNING|trainer.py:803] 2025-04-26 15:19:52,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:52,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:52,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1979
1962
1983
[WARNING|trainer.py:803] 2025-04-26 15:19:54,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:54,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:55,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1980
1963
1984
[WARNING|trainer.py:803] 2025-04-26 15:19:56,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:56,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:19:57,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1981
1964
1985
[WARNING|trainer.py:803] 2025-04-26 15:19:59,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:19:59,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:19:59,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1982
1965
1986
[WARNING|trainer.py:803] 2025-04-26 15:20:01,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:01,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:01,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1983
1966
1987
[WARNING|trainer.py:803] 2025-04-26 15:20:03,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:20:03,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:04,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1984
1967
[WARNING|trainer.py:803] 2025-04-26 15:20:05,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1988
[WARNING|trainer.py:803] 2025-04-26 15:20:06,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:06,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1985
1968
1989
[WARNING|trainer.py:803] 2025-04-26 15:20:08,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:08,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:09,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1986
[WARNING|trainer.py:803] 2025-04-26 15:20:10,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1990
1969
[WARNING|trainer.py:803] 2025-04-26 15:20:11,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:11,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1987
1991
1970
[WARNING|trainer.py:803] 2025-04-26 15:20:12,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:13,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:20:13,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1988
1971
1992
[WARNING|trainer.py:803] 2025-04-26 15:20:15,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:15,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:16,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1972
1989
1993
[WARNING|trainer.py:803] 2025-04-26 15:20:17,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:17,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:18,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1973
1990
[WARNING|trainer.py:803] 2025-04-26 15:20:19,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1994
[WARNING|trainer.py:803] 2025-04-26 15:20:20,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:20,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1974
1991
[WARNING|trainer.py:803] 2025-04-26 15:20:22,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:22,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1995
[WARNING|trainer.py:803] 2025-04-26 15:20:23,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1975
1992
[WARNING|trainer.py:803] 2025-04-26 15:20:24,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:24,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1996
1976
[WARNING|trainer.py:803] 2025-04-26 15:20:25,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1993
[WARNING|trainer.py:803] 2025-04-26 15:20:26,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:26,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1997
1977
1994
[WARNING|trainer.py:803] 2025-04-26 15:20:28,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:28,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:20:29,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1998
1978
[WARNING|trainer.py:803] 2025-04-26 15:20:30,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1995
[WARNING|trainer.py:803] 2025-04-26 15:20:30,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:31,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1999
1979
[WARNING|trainer.py:803] 2025-04-26 15:20:32,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:33,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1996
[WARNING|trainer.py:803] 2025-04-26 15:20:34,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2000
1980
[WARNING|trainer.py:803] 2025-04-26 15:20:35,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:35,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1997
2001
[WARNING|trainer.py:803] 2025-04-26 15:20:36,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1981
[WARNING|trainer.py:803] 2025-04-26 15:20:37,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:37,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1998
2002
[WARNING|trainer.py:803] 2025-04-26 15:20:39,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1982
[WARNING|trainer.py:803] 2025-04-26 15:20:39,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:40,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1999
[WARNING|trainer.py:803] 2025-04-26 15:20:41,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2003
1983
[WARNING|trainer.py:803] 2025-04-26 15:20:42,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:20:42,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2000
[WARNING|trainer.py:803] 2025-04-26 15:20:43,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2004
1984
[WARNING|trainer.py:803] 2025-04-26 15:20:44,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:20:44,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2001
[WARNING|trainer.py:803] 2025-04-26 15:20:45,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2005
1985
[WARNING|trainer.py:803] 2025-04-26 15:20:47,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:20:47,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2002
[WARNING|trainer.py:803] 2025-04-26 15:20:48,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2006
1986
[WARNING|trainer.py:803] 2025-04-26 15:20:49,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:49,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2003
2007
[WARNING|trainer.py:803] 2025-04-26 15:20:50,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1987
[WARNING|trainer.py:803] 2025-04-26 15:20:51,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:20:51,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2004
2008
[WARNING|trainer.py:803] 2025-04-26 15:20:53,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:20:53,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1988
[WARNING|trainer.py:803] 2025-04-26 15:20:54,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2005
2009
[WARNING|trainer.py:803] 2025-04-26 15:20:55,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:20:55,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1989
2006
[WARNING|trainer.py:803] 2025-04-26 15:20:56,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2010
[WARNING|trainer.py:803] 2025-04-26 15:20:57,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:20:57,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1990
2007
2011
[WARNING|trainer.py:803] 2025-04-26 15:20:59,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:20:59,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:20:59,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1991
2008
2012
[WARNING|trainer.py:803] 2025-04-26 15:21:01,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:21:01,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:01,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1992
2009
2013
[WARNING|trainer.py:803] 2025-04-26 15:21:03,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:03,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:03,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2010
1993
2014
[WARNING|trainer.py:803] 2025-04-26 15:21:05,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:06,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:06,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2011
1994
2015
[WARNING|trainer.py:803] 2025-04-26 15:21:08,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:08,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:08,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2012
2016
1995
[WARNING|trainer.py:803] 2025-04-26 15:21:10,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:10,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:10,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2013
2017
[WARNING|trainer.py:803] 2025-04-26 15:21:12,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1996
[WARNING|trainer.py:803] 2025-04-26 15:21:13,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:13,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2014
2018
[WARNING|trainer.py:803] 2025-04-26 15:21:14,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1997
[WARNING|trainer.py:803] 2025-04-26 15:21:15,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2015
[WARNING|trainer.py:803] 2025-04-26 15:21:16,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2019
[WARNING|trainer.py:803] 2025-04-26 15:21:16,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:17,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1998
2016
[WARNING|trainer.py:803] 2025-04-26 15:21:18,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:18,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2020
1999
[WARNING|trainer.py:803] 2025-04-26 15:21:19,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:20,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2017
2021
[WARNING|trainer.py:803] 2025-04-26 15:21:21,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:21,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2000
[WARNING|trainer.py:803] 2025-04-26 15:21:22,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2018
2022
[WARNING|trainer.py:803] 2025-04-26 15:21:23,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:24,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2001
2019
[WARNING|trainer.py:803] 2025-04-26 15:21:25,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:25,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2023
[WARNING|trainer.py:803] 2025-04-26 15:21:26,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2002
2020
[WARNING|trainer.py:803] 2025-04-26 15:21:27,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2024
[WARNING|trainer.py:803] 2025-04-26 15:21:28,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:28,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2003
2021
2025
[WARNING|trainer.py:803] 2025-04-26 15:21:29,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:30,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:30,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2004
2022
2026
[WARNING|trainer.py:803] 2025-04-26 15:21:32,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:32,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:32,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2005
2027
2023
[WARNING|trainer.py:803] 2025-04-26 15:21:34,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:35,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:35,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2006
2024
2028
[WARNING|trainer.py:803] 2025-04-26 15:21:36,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:37,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:37,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2007
2025
2029
[WARNING|trainer.py:803] 2025-04-26 15:21:38,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:39,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:39,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2008
2030
2026
[WARNING|trainer.py:803] 2025-04-26 15:21:41,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:41,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:41,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2009
2031
2027
[WARNING|trainer.py:803] 2025-04-26 15:21:43,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:43,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:43,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2010
2032
2028
[WARNING|trainer.py:803] 2025-04-26 15:21:45,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:45,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:45,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2033
2011
2029
[WARNING|trainer.py:803] 2025-04-26 15:21:47,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:47,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:48,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2034
2012
2030
[WARNING|trainer.py:803] 2025-04-26 15:21:49,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:49,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:50,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2013
2035
2031
[WARNING|trainer.py:803] 2025-04-26 15:21:52,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:52,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:52,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2036
2032
2014
[WARNING|trainer.py:803] 2025-04-26 15:21:54,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:54,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:21:54,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2033
2015
2037
[WARNING|trainer.py:803] 2025-04-26 15:21:56,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:21:56,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:56,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2034
2038
2016
[WARNING|trainer.py:803] 2025-04-26 15:21:58,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:21:58,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:21:58,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2035
2039
2017
[WARNING|trainer.py:803] 2025-04-26 15:22:00,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:01,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:01,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2036
2040
[WARNING|trainer.py:803] 2025-04-26 15:22:02,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2018
[WARNING|trainer.py:803] 2025-04-26 15:22:03,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:03,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2037
2041
2019
[WARNING|trainer.py:803] 2025-04-26 15:22:05,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:22:05,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:05,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2038
2042
[WARNING|trainer.py:803] 2025-04-26 15:22:07,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2020
[WARNING|trainer.py:803] 2025-04-26 15:22:07,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:08,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2039
2043
[WARNING|trainer.py:803] 2025-04-26 15:22:09,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:09,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2021
2040
[WARNING|trainer.py:803] 2025-04-26 15:22:10,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2044
[WARNING|trainer.py:803] 2025-04-26 15:22:11,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:12,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2022
2041
[WARNING|trainer.py:803] 2025-04-26 15:22:13,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2045
[WARNING|trainer.py:803] 2025-04-26 15:22:13,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:14,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2023
2042
2046
[WARNING|trainer.py:803] 2025-04-26 15:22:15,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:16,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:16,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2024
2043
2047
[WARNING|trainer.py:803] 2025-04-26 15:22:17,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:18,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:18,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2025
2048
[WARNING|trainer.py:803] 2025-04-26 15:22:19,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2044
[WARNING|trainer.py:803] 2025-04-26 15:22:20,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:20,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2026
2049
2045
[WARNING|trainer.py:803] 2025-04-26 15:22:21,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:22,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:22:22,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2027
2050
2046
[WARNING|trainer.py:803] 2025-04-26 15:22:24,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:24,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:24,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2028
2051
2047
[WARNING|trainer.py:803] 2025-04-26 15:22:26,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:26,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:26,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2029
2052
2048
[WARNING|trainer.py:803] 2025-04-26 15:22:28,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:28,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:29,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2030
2049
2053
[WARNING|trainer.py:803] 2025-04-26 15:22:30,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:31,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:22:31,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2031
2054
2050
[WARNING|trainer.py:803] 2025-04-26 15:22:33,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:33,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:33,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2032
2051
2055
[WARNING|trainer.py:803] 2025-04-26 15:22:35,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:35,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:35,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2033
2056
2052
[WARNING|trainer.py:803] 2025-04-26 15:22:37,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:37,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:37,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2034
2057
2053
[WARNING|trainer.py:803] 2025-04-26 15:22:39,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:22:39,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:39,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2035
2054
2058
[WARNING|trainer.py:803] 2025-04-26 15:22:41,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:42,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:42,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2036
2055
2059
[WARNING|trainer.py:803] 2025-04-26 15:22:43,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:22:44,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:22:44,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2056
2037
2060
[WARNING|trainer.py:803] 2025-04-26 15:22:46,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:46,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:22:46,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2038
2061
2057
[WARNING|trainer.py:803] 2025-04-26 15:22:48,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:48,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:48,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2039
2062
2058
[WARNING|trainer.py:803] 2025-04-26 15:22:50,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:50,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:50,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2040
2063
2059
[WARNING|trainer.py:803] 2025-04-26 15:22:52,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:52,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:52,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2041
2064
2060
[WARNING|trainer.py:803] 2025-04-26 15:22:54,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:54,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:54,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2061
2042
2065
[WARNING|trainer.py:803] 2025-04-26 15:22:57,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:57,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:57,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2062
2043
2066
[WARNING|trainer.py:803] 2025-04-26 15:22:59,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:59,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:22:59,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2067
2063
2044
[WARNING|trainer.py:803] 2025-04-26 15:23:01,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:01,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:01,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2068
2064
2045
[WARNING|trainer.py:803] 2025-04-26 15:23:03,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:03,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:03,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2069
2065
2046
[WARNING|trainer.py:803] 2025-04-26 15:23:05,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:05,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:06,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2070
2066
2047
[WARNING|trainer.py:803] 2025-04-26 15:23:07,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:08,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:08,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2071
2067
2048
[WARNING|trainer.py:803] 2025-04-26 15:23:09,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:10,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:10,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2068
2049
2072
[WARNING|trainer.py:803] 2025-04-26 15:23:12,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:12,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:12,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2069
2050
2073
[WARNING|trainer.py:803] 2025-04-26 15:23:14,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:14,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:14,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2070
2051
2074
[WARNING|trainer.py:803] 2025-04-26 15:23:16,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:16,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:16,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2071
2052
2075
[WARNING|trainer.py:803] 2025-04-26 15:23:18,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:18,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:19,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2072
2053
2076
[WARNING|trainer.py:803] 2025-04-26 15:23:21,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:21,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:21,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2054
2077
2073
[WARNING|trainer.py:803] 2025-04-26 15:23:23,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:23,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:23,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2078
2055
2074
[WARNING|trainer.py:803] 2025-04-26 15:23:25,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:25,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:25,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2056
2079
2075
[WARNING|trainer.py:803] 2025-04-26 15:23:27,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:27,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:27,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2080
2057
2076
[WARNING|trainer.py:803] 2025-04-26 15:23:29,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:29,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:30,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2081
2058
2077
[WARNING|trainer.py:803] 2025-04-26 15:23:32,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:32,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:32,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2078
2059
2082
[WARNING|trainer.py:803] 2025-04-26 15:23:34,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:34,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:34,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2060
2079
2083
[WARNING|trainer.py:803] 2025-04-26 15:23:36,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:36,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:36,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2061
2080
2084
[WARNING|trainer.py:803] 2025-04-26 15:23:38,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:38,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:39,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2062
2081
2085
[WARNING|trainer.py:803] 2025-04-26 15:23:40,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:41,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:23:41,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2063
2082
[WARNING|trainer.py:803] 2025-04-26 15:23:42,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2086
[WARNING|trainer.py:803] 2025-04-26 15:23:43,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:23:44,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2064
2083
[WARNING|trainer.py:803] 2025-04-26 15:23:44,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:45,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2087
2065
[WARNING|trainer.py:803] 2025-04-26 15:23:46,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2084
[WARNING|trainer.py:803] 2025-04-26 15:23:47,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2088
[WARNING|trainer.py:803] 2025-04-26 15:23:47,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2066
[WARNING|trainer.py:803] 2025-04-26 15:23:48,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:49,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2085
2089
[WARNING|trainer.py:803] 2025-04-26 15:23:50,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2067
[WARNING|trainer.py:803] 2025-04-26 15:23:50,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:51,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2086
2068
2090
[WARNING|trainer.py:803] 2025-04-26 15:23:52,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:53,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:53,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2087
2069
2091
[WARNING|trainer.py:803] 2025-04-26 15:23:55,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:55,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:55,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2088
2070
2092
[WARNING|trainer.py:803] 2025-04-26 15:23:57,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:57,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:57,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2089
2093
2071
[WARNING|trainer.py:803] 2025-04-26 15:23:59,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:23:59,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:23:59,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2094
2072
2090
[WARNING|trainer.py:803] 2025-04-26 15:24:01,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:02,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:02,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2095
2091
2073
[WARNING|trainer.py:803] 2025-04-26 15:24:04,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:04,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:04,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2096
2092
[WARNING|trainer.py:803] 2025-04-26 15:24:06,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2074
[WARNING|trainer.py:803] 2025-04-26 15:24:06,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:07,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2097
2093
[WARNING|trainer.py:803] 2025-04-26 15:24:08,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2075
[WARNING|trainer.py:803] 2025-04-26 15:24:08,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:09,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2098
2094
[WARNING|trainer.py:803] 2025-04-26 15:24:10,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2076
[WARNING|trainer.py:803] 2025-04-26 15:24:10,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:11,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2099
2095
2077
[WARNING|trainer.py:803] 2025-04-26 15:24:13,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:13,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:13,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2100
2096
2078
[WARNING|trainer.py:803] 2025-04-26 15:24:15,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:24:15,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2101
[WARNING|trainer.py:803] 2025-04-26 15:24:15,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:16,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2097
2102
2079
[WARNING|trainer.py:803] 2025-04-26 15:24:17,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:24:17,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:17,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2103
2098
2080
[WARNING|trainer.py:803] 2025-04-26 15:24:19,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2104
[WARNING|trainer.py:803] 2025-04-26 15:24:19,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:19,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:20,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2105
2099
2081
[WARNING|trainer.py:803] 2025-04-26 15:24:21,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2106
[WARNING|trainer.py:803] 2025-04-26 15:24:22,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:22,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:23,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2107
2100
2082
[WARNING|trainer.py:803] 2025-04-26 15:24:24,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:24,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2108
[WARNING|trainer.py:803] 2025-04-26 15:24:24,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2101
[WARNING|trainer.py:803] 2025-04-26 15:24:25,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:25,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2109
2083
2102
[WARNING|trainer.py:803] 2025-04-26 15:24:26,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:27,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:27,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2110
2103
[WARNING|trainer.py:803] 2025-04-26 15:24:28,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2084
[WARNING|trainer.py:803] 2025-04-26 15:24:28,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2111
2104
[WARNING|trainer.py:803] 2025-04-26 15:24:29,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:29,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:29,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2112
2105
2085
[WARNING|trainer.py:803] 2025-04-26 15:24:30,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:31,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2113
[WARNING|trainer.py:803] 2025-04-26 15:24:31,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2106
[WARNING|trainer.py:803] 2025-04-26 15:24:32,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:24:32,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2114
2107
2086
[WARNING|trainer.py:803] 2025-04-26 15:24:33,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:33,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2115
[WARNING|trainer.py:803] 2025-04-26 15:24:33,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2108
[WARNING|trainer.py:803] 2025-04-26 15:24:34,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:35,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2116
2109
2087
[WARNING|trainer.py:803] 2025-04-26 15:24:35,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:36,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2117
[WARNING|trainer.py:803] 2025-04-26 15:24:36,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2110
[WARNING|trainer.py:803] 2025-04-26 15:24:37,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:37,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2118
2088
2111
[WARNING|trainer.py:803] 2025-04-26 15:24:38,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:38,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:38,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2119
2112
[WARNING|trainer.py:803] 2025-04-26 15:24:39,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2089
[WARNING|trainer.py:803] 2025-04-26 15:24:40,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2120
2113
[WARNING|trainer.py:803] 2025-04-26 15:24:40,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:24:41,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:41,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2121
2114
[WARNING|trainer.py:803] 2025-04-26 15:24:42,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2090
[WARNING|trainer.py:803] 2025-04-26 15:24:42,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2122
2115
[WARNING|trainer.py:803] 2025-04-26 15:24:43,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:43,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:44,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2123
2116
2091
[WARNING|trainer.py:803] 2025-04-26 15:24:44,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:45,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2124
[WARNING|trainer.py:803] 2025-04-26 15:24:45,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2117
[WARNING|trainer.py:803] 2025-04-26 15:24:46,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:46,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2125
2092
2118
[WARNING|trainer.py:803] 2025-04-26 15:24:47,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:24:47,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:48,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2126
2119
[WARNING|trainer.py:803] 2025-04-26 15:24:48,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2093
[WARNING|trainer.py:803] 2025-04-26 15:24:49,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2127
2120
[WARNING|trainer.py:803] 2025-04-26 15:24:49,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:50,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:50,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2128
2094
2121
[WARNING|trainer.py:803] 2025-04-26 15:24:51,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:51,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:51,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2129
2122
[WARNING|trainer.py:803] 2025-04-26 15:24:52,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2095
[WARNING|trainer.py:803] 2025-04-26 15:24:53,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2130
2123
[WARNING|trainer.py:803] 2025-04-26 15:24:54,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:24:54,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2131
[WARNING|trainer.py:803] 2025-04-26 15:24:54,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2124
2096
[WARNING|trainer.py:803] 2025-04-26 15:24:55,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2132
[WARNING|trainer.py:803] 2025-04-26 15:24:56,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:24:56,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2125
[WARNING|trainer.py:803] 2025-04-26 15:24:56,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2133
[WARNING|trainer.py:803] 2025-04-26 15:24:57,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2097
2126
[WARNING|trainer.py:803] 2025-04-26 15:24:58,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:24:58,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2134
[WARNING|trainer.py:803] 2025-04-26 15:24:58,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2127
[WARNING|trainer.py:803] 2025-04-26 15:24:59,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2098
2135
[WARNING|trainer.py:803] 2025-04-26 15:25:00,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2128
[WARNING|trainer.py:803] 2025-04-26 15:25:00,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:00,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2136
[WARNING|trainer.py:803] 2025-04-26 15:25:01,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2129
[WARNING|trainer.py:803] 2025-04-26 15:25:02,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2099
2137
[WARNING|trainer.py:803] 2025-04-26 15:25:02,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:03,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2130
[WARNING|trainer.py:803] 2025-04-26 15:25:03,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2138
[WARNING|trainer.py:803] 2025-04-26 15:25:03,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2100
2131
[WARNING|trainer.py:803] 2025-04-26 15:25:04,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:25:05,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2139
[WARNING|trainer.py:803] 2025-04-26 15:25:05,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2101
2132
[WARNING|trainer.py:803] 2025-04-26 15:25:06,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:06,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2140
[WARNING|trainer.py:803] 2025-04-26 15:25:06,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2102
2133
[WARNING|trainer.py:803] 2025-04-26 15:25:07,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:07,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2141
[WARNING|trainer.py:803] 2025-04-26 15:25:07,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2103
2134
[WARNING|trainer.py:803] 2025-04-26 15:25:08,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:09,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2142
[WARNING|trainer.py:803] 2025-04-26 15:25:09,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2104
2135
[WARNING|trainer.py:803] 2025-04-26 15:25:09,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:10,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2143
[WARNING|trainer.py:803] 2025-04-26 15:25:10,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2105
2136
[WARNING|trainer.py:803] 2025-04-26 15:25:11,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:25:11,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2144
[WARNING|trainer.py:803] 2025-04-26 15:25:12,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2106
2137
[WARNING|trainer.py:803] 2025-04-26 15:25:12,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:13,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2145
[WARNING|trainer.py:803] 2025-04-26 15:25:13,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2107
2138
[WARNING|trainer.py:803] 2025-04-26 15:25:13,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:14,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2146
[WARNING|trainer.py:803] 2025-04-26 15:25:14,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2108
2139
[WARNING|trainer.py:803] 2025-04-26 15:25:15,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:15,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2147
[WARNING|trainer.py:803] 2025-04-26 15:25:16,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2109
2140
[WARNING|trainer.py:803] 2025-04-26 15:25:16,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:16,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2148
[WARNING|trainer.py:803] 2025-04-26 15:25:17,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2110
2141
[WARNING|trainer.py:803] 2025-04-26 15:25:17,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:18,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2149
[WARNING|trainer.py:803] 2025-04-26 15:25:18,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2111
2142
[WARNING|trainer.py:803] 2025-04-26 15:25:19,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:19,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2150
[WARNING|trainer.py:803] 2025-04-26 15:25:19,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2112
2143
[WARNING|trainer.py:803] 2025-04-26 15:25:20,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:20,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2151
[WARNING|trainer.py:803] 2025-04-26 15:25:21,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2113
2144
[WARNING|trainer.py:803] 2025-04-26 15:25:21,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:22,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2152
2114
[WARNING|trainer.py:803] 2025-04-26 15:25:22,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2145
[WARNING|trainer.py:803] 2025-04-26 15:25:23,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:23,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2153
2115
[WARNING|trainer.py:803] 2025-04-26 15:25:23,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2146
[WARNING|trainer.py:803] 2025-04-26 15:25:24,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:24,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2154
2116
[WARNING|trainer.py:803] 2025-04-26 15:25:25,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:25,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2147
[WARNING|trainer.py:803] 2025-04-26 15:25:26,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2155
2117
[WARNING|trainer.py:803] 2025-04-26 15:25:26,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:27,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2148
[WARNING|trainer.py:803] 2025-04-26 15:25:27,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2156
2118
[WARNING|trainer.py:803] 2025-04-26 15:25:27,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:28,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2149
[WARNING|trainer.py:803] 2025-04-26 15:25:28,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2157
2119
[WARNING|trainer.py:803] 2025-04-26 15:25:29,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:29,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2150
[WARNING|trainer.py:803] 2025-04-26 15:25:29,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2158
2120
[WARNING|trainer.py:803] 2025-04-26 15:25:30,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:30,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2151
[WARNING|trainer.py:803] 2025-04-26 15:25:31,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2159
2121
[WARNING|trainer.py:803] 2025-04-26 15:25:31,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:32,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2152
[WARNING|trainer.py:803] 2025-04-26 15:25:32,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2160
2122
[WARNING|trainer.py:803] 2025-04-26 15:25:33,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:33,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2153
[WARNING|trainer.py:803] 2025-04-26 15:25:33,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2161
2123
[WARNING|trainer.py:803] 2025-04-26 15:25:34,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:34,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2154
[WARNING|trainer.py:803] 2025-04-26 15:25:35,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2162
2124
[WARNING|trainer.py:803] 2025-04-26 15:25:35,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:25:36,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2155
[WARNING|trainer.py:803] 2025-04-26 15:25:36,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2163
2125
[WARNING|trainer.py:803] 2025-04-26 15:25:37,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:37,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2156
[WARNING|trainer.py:803] 2025-04-26 15:25:37,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2164
2126
[WARNING|trainer.py:803] 2025-04-26 15:25:38,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:38,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2157
[WARNING|trainer.py:803] 2025-04-26 15:25:39,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2165
2127
[WARNING|trainer.py:803] 2025-04-26 15:25:39,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:40,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2158
[WARNING|trainer.py:803] 2025-04-26 15:25:40,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2166
2128
[WARNING|trainer.py:803] 2025-04-26 15:25:41,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:41,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2159
[WARNING|trainer.py:803] 2025-04-26 15:25:41,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2167
2129
[WARNING|trainer.py:803] 2025-04-26 15:25:42,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:42,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2160
[WARNING|trainer.py:803] 2025-04-26 15:25:43,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2168
2130
[WARNING|trainer.py:803] 2025-04-26 15:25:43,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:43,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2161
[WARNING|trainer.py:803] 2025-04-26 15:25:44,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2169
2131
[WARNING|trainer.py:803] 2025-04-26 15:25:44,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:45,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2162
[WARNING|trainer.py:803] 2025-04-26 15:25:45,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2170
2132
[WARNING|trainer.py:803] 2025-04-26 15:25:46,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:46,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2163
[WARNING|trainer.py:803] 2025-04-26 15:25:47,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2171
2133
[WARNING|trainer.py:803] 2025-04-26 15:25:47,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:25:47,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2164
[WARNING|trainer.py:803] 2025-04-26 15:25:48,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2172
2134
[WARNING|trainer.py:803] 2025-04-26 15:25:48,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:49,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2165
[WARNING|trainer.py:803] 2025-04-26 15:25:49,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2173
2135
[WARNING|trainer.py:803] 2025-04-26 15:25:50,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:50,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2166
2174
[WARNING|trainer.py:803] 2025-04-26 15:25:51,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2136
[WARNING|trainer.py:803] 2025-04-26 15:25:51,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:51,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2167
[WARNING|trainer.py:803] 2025-04-26 15:25:52,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2175
2137
[WARNING|trainer.py:803] 2025-04-26 15:25:52,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:25:53,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2168
[WARNING|trainer.py:803] 2025-04-26 15:25:53,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2176
2138
[WARNING|trainer.py:803] 2025-04-26 15:25:54,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:54,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2169
[WARNING|trainer.py:803] 2025-04-26 15:25:54,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2177
2139
[WARNING|trainer.py:803] 2025-04-26 15:25:55,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:55,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2170
[WARNING|trainer.py:803] 2025-04-26 15:25:56,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2178
2140
[WARNING|trainer.py:803] 2025-04-26 15:25:56,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:57,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2171
[WARNING|trainer.py:803] 2025-04-26 15:25:57,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2179
2141
[WARNING|trainer.py:803] 2025-04-26 15:25:58,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:25:58,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2172
[WARNING|trainer.py:803] 2025-04-26 15:25:58,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2180
2142
[WARNING|trainer.py:803] 2025-04-26 15:25:59,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:25:59,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2173
[WARNING|trainer.py:803] 2025-04-26 15:26:00,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2181
2143
[WARNING|trainer.py:803] 2025-04-26 15:26:00,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:00,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2174
2182
[WARNING|trainer.py:803] 2025-04-26 15:26:01,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2144
[WARNING|trainer.py:803] 2025-04-26 15:26:02,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:02,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:02,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2175
2183
2145
[WARNING|trainer.py:803] 2025-04-26 15:26:03,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:03,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:04,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2176
2184
2146
[WARNING|trainer.py:803] 2025-04-26 15:26:04,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:04,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:05,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2177
2185
2147
[WARNING|trainer.py:803] 2025-04-26 15:26:06,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:06,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2178
[WARNING|trainer.py:803] 2025-04-26 15:26:06,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2186
2148
[WARNING|trainer.py:803] 2025-04-26 15:26:07,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:07,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2179
[WARNING|trainer.py:803] 2025-04-26 15:26:07,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2187
2149
[WARNING|trainer.py:803] 2025-04-26 15:26:08,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:08,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:09,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2180
2188
2150
[WARNING|trainer.py:803] 2025-04-26 15:26:10,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:10,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:10,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2181
2189
2151
[WARNING|trainer.py:803] 2025-04-26 15:26:11,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:11,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2182
[WARNING|trainer.py:803] 2025-04-26 15:26:11,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2190
2152
[WARNING|trainer.py:803] 2025-04-26 15:26:12,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:12,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:13,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2183
2191
2153
[WARNING|trainer.py:803] 2025-04-26 15:26:14,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:14,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:14,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2192
2184
2154
[WARNING|trainer.py:803] 2025-04-26 15:26:15,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:15,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:15,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2193
2185
2155
[WARNING|trainer.py:803] 2025-04-26 15:26:16,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:16,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:17,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2194
2186
2156
[WARNING|trainer.py:803] 2025-04-26 15:26:18,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:18,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:18,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2195
2187
2157
[WARNING|trainer.py:803] 2025-04-26 15:26:19,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:19,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:19,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2196
2188
2158
[WARNING|trainer.py:803] 2025-04-26 15:26:20,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:20,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:21,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2197
2189
2159
[WARNING|trainer.py:803] 2025-04-26 15:26:21,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:22,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:22,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2198
2190
2160
[WARNING|trainer.py:803] 2025-04-26 15:26:23,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:23,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:23,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2199
2191
2161
[WARNING|trainer.py:803] 2025-04-26 15:26:24,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:24,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2200
2192
2162
[WARNING|trainer.py:803] 2025-04-26 15:26:25,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:26,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2201
[WARNING|trainer.py:803] 2025-04-26 15:26:26,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2193
2163
[WARNING|trainer.py:803] 2025-04-26 15:26:27,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:26:27,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:27,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2202
2194
2164
[WARNING|trainer.py:803] 2025-04-26 15:26:28,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:28,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:28,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2203
2195
2165
[WARNING|trainer.py:803] 2025-04-26 15:26:29,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:30,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:30,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2204
2196
2166
[WARNING|trainer.py:803] 2025-04-26 15:26:31,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:31,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:31,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2205
2197
2167
[WARNING|trainer.py:803] 2025-04-26 15:26:32,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:32,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:32,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2206
2198
2168
[WARNING|trainer.py:803] 2025-04-26 15:26:33,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:34,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:34,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2207
2199
2169
[WARNING|trainer.py:803] 2025-04-26 15:26:35,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:35,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:35,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2208
2200
2170
[WARNING|trainer.py:803] 2025-04-26 15:26:36,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:36,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:36,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2209
2201
2171
[WARNING|trainer.py:803] 2025-04-26 15:26:37,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:38,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:26:38,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2210
2202
2172
[WARNING|trainer.py:803] 2025-04-26 15:26:39,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:39,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:39,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2211
2173
2203
[WARNING|trainer.py:803] 2025-04-26 15:26:40,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:40,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:40,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2212
2174
2204
[WARNING|trainer.py:803] 2025-04-26 15:26:41,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:42,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2213
[WARNING|trainer.py:803] 2025-04-26 15:26:42,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2175
2205
[WARNING|trainer.py:803] 2025-04-26 15:26:43,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:43,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2214
[WARNING|trainer.py:803] 2025-04-26 15:26:43,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2176
2206
[WARNING|trainer.py:803] 2025-04-26 15:26:44,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:44,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2215
[WARNING|trainer.py:803] 2025-04-26 15:26:45,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2177
2207
[WARNING|trainer.py:803] 2025-04-26 15:26:45,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2216
[WARNING|trainer.py:803] 2025-04-26 15:26:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:46,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2178
2208
[WARNING|trainer.py:803] 2025-04-26 15:26:47,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:26:47,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2217
[WARNING|trainer.py:803] 2025-04-26 15:26:47,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2179
2209
[WARNING|trainer.py:803] 2025-04-26 15:26:48,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:48,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2218
[WARNING|trainer.py:803] 2025-04-26 15:26:49,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2180
2210
[WARNING|trainer.py:803] 2025-04-26 15:26:49,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:50,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2219
[WARNING|trainer.py:803] 2025-04-26 15:26:50,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2181
2211
[WARNING|trainer.py:803] 2025-04-26 15:26:51,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:51,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2220
2182
[WARNING|trainer.py:803] 2025-04-26 15:26:51,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2212
[WARNING|trainer.py:803] 2025-04-26 15:26:52,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:52,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2221
[WARNING|trainer.py:803] 2025-04-26 15:26:53,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2183
2213
[WARNING|trainer.py:803] 2025-04-26 15:26:53,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:54,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2222
[WARNING|trainer.py:803] 2025-04-26 15:26:54,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2184
2214
[WARNING|trainer.py:803] 2025-04-26 15:26:55,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:26:55,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2223
[WARNING|trainer.py:803] 2025-04-26 15:26:55,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2185
2215
[WARNING|trainer.py:803] 2025-04-26 15:26:56,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:26:56,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2224
[WARNING|trainer.py:803] 2025-04-26 15:26:57,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2186
2216
[WARNING|trainer.py:803] 2025-04-26 15:26:57,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:58,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2225
[WARNING|trainer.py:803] 2025-04-26 15:26:58,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2187
2217
[WARNING|trainer.py:803] 2025-04-26 15:26:59,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:26:59,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2226
[WARNING|trainer.py:803] 2025-04-26 15:26:59,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2188
2218
[WARNING|trainer.py:803] 2025-04-26 15:27:00,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:00,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2227
2189
[WARNING|trainer.py:803] 2025-04-26 15:27:01,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2219
[WARNING|trainer.py:803] 2025-04-26 15:27:01,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:02,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2228
2190
[WARNING|trainer.py:803] 2025-04-26 15:27:02,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2220
[WARNING|trainer.py:803] 2025-04-26 15:27:03,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:03,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2229
2191
[WARNING|trainer.py:803] 2025-04-26 15:27:04,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:04,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
2221
[WARNING|trainer.py:803] 2025-04-26 15:27:04,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2230
2192
[WARNING|trainer.py:803] 2025-04-26 15:27:05,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:05,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2222
[WARNING|trainer.py:803] 2025-04-26 15:27:06,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2231
2193
[WARNING|trainer.py:803] 2025-04-26 15:27:06,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:07,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2223
[WARNING|trainer.py:803] 2025-04-26 15:27:07,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2232
2194
[WARNING|trainer.py:803] 2025-04-26 15:27:08,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:08,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:08,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2224
2233
2195
[WARNING|trainer.py:803] 2025-04-26 15:27:09,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:09,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:10,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2225
2234
2196
[WARNING|trainer.py:803] 2025-04-26 15:27:10,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:11,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:11,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2226
2235
2197
[WARNING|trainer.py:803] 2025-04-26 15:27:12,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:12,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:12,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2227
2236
2198
[WARNING|trainer.py:803] 2025-04-26 15:27:13,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:13,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:13,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2228
2199
2237
[WARNING|trainer.py:803] 2025-04-26 15:27:14,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:15,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:15,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2229
2200
2238
[WARNING|trainer.py:803] 2025-04-26 15:27:16,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 15:27:16,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:16,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2230
2201
2239
[WARNING|trainer.py:803] 2025-04-26 15:27:17,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:27:17,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:27:17,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2231
2240
2202
[WARNING|trainer.py:803] 2025-04-26 15:27:18,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:19,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:19,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2232
2241
2203
[WARNING|trainer.py:803] 2025-04-26 15:27:20,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:20,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:20,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2233
2242
2204
[WARNING|trainer.py:803] 2025-04-26 15:27:21,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:21,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2234
[WARNING|trainer.py:803] 2025-04-26 15:27:22,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2243
2205
[WARNING|trainer.py:803] 2025-04-26 15:27:22,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:23,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:23,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2235
2244
2206
[WARNING|trainer.py:803] 2025-04-26 15:27:24,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:24,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2236
[WARNING|trainer.py:803] 2025-04-26 15:27:24,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2245
2207
[WARNING|trainer.py:803] 2025-04-26 15:27:25,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:26,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2237
[WARNING|trainer.py:803] 2025-04-26 15:27:26,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2246
2208
[WARNING|trainer.py:803] 2025-04-26 15:27:26,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:27,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2238
[WARNING|trainer.py:803] 2025-04-26 15:27:27,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2247
2209
[WARNING|trainer.py:803] 2025-04-26 15:27:28,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:28,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:28,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2239
2248
2210
[WARNING|trainer.py:803] 2025-04-26 15:27:29,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:30,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2240
[WARNING|trainer.py:803] 2025-04-26 15:27:30,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2249
2211
[WARNING|trainer.py:803] 2025-04-26 15:27:30,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:31,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2241
[WARNING|trainer.py:803] 2025-04-26 15:27:31,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2250
2212
[WARNING|trainer.py:803] 2025-04-26 15:27:32,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:32,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2242
[WARNING|trainer.py:803] 2025-04-26 15:27:32,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2251
2213
[WARNING|trainer.py:803] 2025-04-26 15:27:33,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:33,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2243
[WARNING|trainer.py:803] 2025-04-26 15:27:34,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2252
2214
[WARNING|trainer.py:803] 2025-04-26 15:27:34,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:35,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2244
[WARNING|trainer.py:803] 2025-04-26 15:27:35,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2253
2215
[WARNING|trainer.py:803] 2025-04-26 15:27:36,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:36,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2245
[WARNING|trainer.py:803] 2025-04-26 15:27:36,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2254
2216
[WARNING|trainer.py:803] 2025-04-26 15:27:37,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:37,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2246
[WARNING|trainer.py:803] 2025-04-26 15:27:38,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2255
2217
[WARNING|trainer.py:803] 2025-04-26 15:27:38,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:38,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2256
2247
[WARNING|trainer.py:803] 2025-04-26 15:27:39,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2218
[WARNING|trainer.py:803] 2025-04-26 15:27:40,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:40,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2257
2248
[WARNING|trainer.py:803] 2025-04-26 15:27:40,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2219
[WARNING|trainer.py:803] 2025-04-26 15:27:41,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:41,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2258
2249
[WARNING|trainer.py:803] 2025-04-26 15:27:42,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:42,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2220
[WARNING|trainer.py:803] 2025-04-26 15:27:42,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2259
2250
[WARNING|trainer.py:803] 2025-04-26 15:27:43,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:43,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2221
[WARNING|trainer.py:803] 2025-04-26 15:27:44,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2260
2251
[WARNING|trainer.py:803] 2025-04-26 15:27:45,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:45,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:45,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2222
2261
2252
[WARNING|trainer.py:803] 2025-04-26 15:27:46,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:46,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:46,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2223
2262
2253
[WARNING|trainer.py:803] 2025-04-26 15:27:47,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:47,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:48,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2263
2224
2254
[WARNING|trainer.py:803] 2025-04-26 15:27:49,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:49,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:49,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2264
2225
2255
[WARNING|trainer.py:803] 2025-04-26 15:27:50,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:50,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:50,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2265
2226
2256
[WARNING|trainer.py:803] 2025-04-26 15:27:51,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:51,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2266
[WARNING|trainer.py:803] 2025-04-26 15:27:51,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2227
2257
[WARNING|trainer.py:803] 2025-04-26 15:27:52,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2267
[WARNING|trainer.py:803] 2025-04-26 15:27:53,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:27:53,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2228
2258
[WARNING|trainer.py:803] 2025-04-26 15:27:54,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2268
[WARNING|trainer.py:803] 2025-04-26 15:27:54,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:27:54,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2229
2259
[WARNING|trainer.py:803] 2025-04-26 15:27:55,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2269
[WARNING|trainer.py:803] 2025-04-26 15:27:55,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 15:27:55,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2260
2230
[WARNING|trainer.py:803] 2025-04-26 15:27:56,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2270
[WARNING|trainer.py:803] 2025-04-26 15:27:57,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:27:57,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2261
2231
[WARNING|trainer.py:803] 2025-04-26 15:27:57,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2271
[WARNING|trainer.py:803] 2025-04-26 15:27:58,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:58,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2262
[WARNING|trainer.py:803] 2025-04-26 15:27:59,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2232
2272
[WARNING|trainer.py:803] 2025-04-26 15:27:59,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:27:59,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2263
[WARNING|trainer.py:803] 2025-04-26 15:28:00,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2233
2273
[WARNING|trainer.py:803] 2025-04-26 15:28:01,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:01,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2264
[WARNING|trainer.py:803] 2025-04-26 15:28:01,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2234
2274
[WARNING|trainer.py:803] 2025-04-26 15:28:02,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:02,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2265
[WARNING|trainer.py:803] 2025-04-26 15:28:02,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2235
2275
[WARNING|trainer.py:803] 2025-04-26 15:28:03,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:03,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2266
[WARNING|trainer.py:803] 2025-04-26 15:28:04,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2236
2276
[WARNING|trainer.py:803] 2025-04-26 15:28:04,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:05,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:05,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2267
2277
2237
[WARNING|trainer.py:803] 2025-04-26 15:28:06,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:06,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2268
[WARNING|trainer.py:803] 2025-04-26 15:28:06,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2278
2238
[WARNING|trainer.py:803] 2025-04-26 15:28:07,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:07,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2269
[WARNING|trainer.py:803] 2025-04-26 15:28:07,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2279
2239
[WARNING|trainer.py:803] 2025-04-26 15:28:08,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:09,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2270
[WARNING|trainer.py:803] 2025-04-26 15:28:09,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2280
2240
[WARNING|trainer.py:803] 2025-04-26 15:28:10,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:10,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2271
[WARNING|trainer.py:803] 2025-04-26 15:28:10,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2281
2241
[WARNING|trainer.py:803] 2025-04-26 15:28:11,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:11,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2272
[WARNING|trainer.py:803] 2025-04-26 15:28:11,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2282
2242
[WARNING|trainer.py:803] 2025-04-26 15:28:12,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:12,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2273
[WARNING|trainer.py:803] 2025-04-26 15:28:13,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2283
2243
[WARNING|trainer.py:803] 2025-04-26 15:28:13,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:14,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2274
2284
[WARNING|trainer.py:803] 2025-04-26 15:28:14,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2244
[WARNING|trainer.py:803] 2025-04-26 15:28:15,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:15,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2275
2285
[WARNING|trainer.py:803] 2025-04-26 15:28:15,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2245
[WARNING|trainer.py:803] 2025-04-26 15:28:16,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:16,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2276
2286
[WARNING|trainer.py:803] 2025-04-26 15:28:17,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2246
[WARNING|trainer.py:803] 2025-04-26 15:28:17,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:17,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2277
2287
[WARNING|trainer.py:803] 2025-04-26 15:28:18,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:18,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2247
[WARNING|trainer.py:803] 2025-04-26 15:28:19,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2278
2288
[WARNING|trainer.py:803] 2025-04-26 15:28:19,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:20,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2248
[WARNING|trainer.py:803] 2025-04-26 15:28:20,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2279
2289
[WARNING|trainer.py:803] 2025-04-26 15:28:21,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:28:21,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2249
[WARNING|trainer.py:803] 2025-04-26 15:28:21,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2280
2290
[WARNING|trainer.py:803] 2025-04-26 15:28:22,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:28:22,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2250
[WARNING|trainer.py:803] 2025-04-26 15:28:22,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2281
2291
[WARNING|trainer.py:803] 2025-04-26 15:28:23,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:24,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2251
[WARNING|trainer.py:803] 2025-04-26 15:28:24,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2282
2292
[WARNING|trainer.py:803] 2025-04-26 15:28:24,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:25,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2252
[WARNING|trainer.py:803] 2025-04-26 15:28:25,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2283
2293
[WARNING|trainer.py:803] 2025-04-26 15:28:26,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2253
[WARNING|trainer.py:803] 2025-04-26 15:28:26,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:26,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2284
2294
[WARNING|trainer.py:803] 2025-04-26 15:28:27,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2254
[WARNING|trainer.py:803] 2025-04-26 15:28:27,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:28,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2285
2295
[WARNING|trainer.py:803] 2025-04-26 15:28:28,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2255
[WARNING|trainer.py:803] 2025-04-26 15:28:29,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:29,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2286
2296
[WARNING|trainer.py:803] 2025-04-26 15:28:30,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2256
[WARNING|trainer.py:803] 2025-04-26 15:28:30,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:30,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2287
2297
[WARNING|trainer.py:803] 2025-04-26 15:28:31,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2257
[WARNING|trainer.py:803] 2025-04-26 15:28:31,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:31,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2298
2288
[WARNING|trainer.py:803] 2025-04-26 15:28:32,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2258
[WARNING|trainer.py:803] 2025-04-26 15:28:33,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:33,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2299
2289
[WARNING|trainer.py:803] 2025-04-26 15:28:33,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2259
[WARNING|trainer.py:803] 2025-04-26 15:28:34,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:34,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2300
2290
[WARNING|trainer.py:803] 2025-04-26 15:28:35,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2260
[WARNING|trainer.py:803] 2025-04-26 15:28:35,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:35,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2301
2291
[WARNING|trainer.py:803] 2025-04-26 15:28:36,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2261
[WARNING|trainer.py:803] 2025-04-26 15:28:36,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:36,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2302
2292
[WARNING|trainer.py:803] 2025-04-26 15:28:37,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2262
[WARNING|trainer.py:803] 2025-04-26 15:28:38,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:38,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2303
2293
[WARNING|trainer.py:803] 2025-04-26 15:28:38,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2263
[WARNING|trainer.py:803] 2025-04-26 15:28:39,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:39,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2304
2294
[WARNING|trainer.py:803] 2025-04-26 15:28:40,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:40,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2264
[WARNING|trainer.py:803] 2025-04-26 15:28:40,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2305
2295
[WARNING|trainer.py:803] 2025-04-26 15:28:41,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:41,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2265
[WARNING|trainer.py:803] 2025-04-26 15:28:42,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2306
2296
[WARNING|trainer.py:803] 2025-04-26 15:28:42,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:43,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2266
[WARNING|trainer.py:803] 2025-04-26 15:28:43,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2307
2297
[WARNING|trainer.py:803] 2025-04-26 15:28:44,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:44,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2267
[WARNING|trainer.py:803] 2025-04-26 15:28:44,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2308
2298
[WARNING|trainer.py:803] 2025-04-26 15:28:45,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:45,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2268
[WARNING|trainer.py:803] 2025-04-26 15:28:45,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2309
2299
[WARNING|trainer.py:803] 2025-04-26 15:28:46,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:46,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2269
[WARNING|trainer.py:803] 2025-04-26 15:28:47,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2310
2300
[WARNING|trainer.py:803] 2025-04-26 15:28:47,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:48,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2270
[WARNING|trainer.py:803] 2025-04-26 15:28:48,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2311
2301
[WARNING|trainer.py:803] 2025-04-26 15:28:49,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:49,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2271
[WARNING|trainer.py:803] 2025-04-26 15:28:49,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2312
2302
[WARNING|trainer.py:803] 2025-04-26 15:28:50,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:50,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2272
[WARNING|trainer.py:803] 2025-04-26 15:28:51,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2313
2303
[WARNING|trainer.py:803] 2025-04-26 15:28:51,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:52,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2273
[WARNING|trainer.py:803] 2025-04-26 15:28:52,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2314
2304
[WARNING|trainer.py:803] 2025-04-26 15:28:53,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:53,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:53,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2274
2315
2305
[WARNING|trainer.py:803] 2025-04-26 15:28:54,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:54,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:54,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2275
2316
2306
[WARNING|trainer.py:803] 2025-04-26 15:28:55,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:55,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:56,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2276
2317
2307
[WARNING|trainer.py:803] 2025-04-26 15:28:56,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:28:57,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:57,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2277
2318
2308
[WARNING|trainer.py:803] 2025-04-26 15:28:58,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:28:58,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:58,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2278
2319
2309
[WARNING|trainer.py:803] 2025-04-26 15:28:59,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:28:59,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:28:59,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2279
2320
2310
[WARNING|trainer.py:803] 2025-04-26 15:29:00,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:00,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:01,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2321
2280
2311
[WARNING|trainer.py:803] 2025-04-26 15:29:02,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:02,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:02,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2322
2281
2312
[WARNING|trainer.py:803] 2025-04-26 15:29:03,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:03,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:03,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2323
2282
2313
[WARNING|trainer.py:803] 2025-04-26 15:29:04,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:04,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:05,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2324
2283
2314
[WARNING|trainer.py:803] 2025-04-26 15:29:05,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:06,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:06,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2325
2284
2315
[WARNING|trainer.py:803] 2025-04-26 15:29:07,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:07,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:29:07,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2326
2285
2316
[WARNING|trainer.py:803] 2025-04-26 15:29:08,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:08,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:08,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2327
2286
2317
[WARNING|trainer.py:803] 2025-04-26 15:29:09,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:09,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:10,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2328
2287
2318
[WARNING|trainer.py:803] 2025-04-26 15:29:10,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:11,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2329
[WARNING|trainer.py:803] 2025-04-26 15:29:11,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2288
2319
[WARNING|trainer.py:803] 2025-04-26 15:29:12,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:12,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2330
[WARNING|trainer.py:803] 2025-04-26 15:29:12,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2289
2320
[WARNING|trainer.py:803] 2025-04-26 15:29:13,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:13,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2331
[WARNING|trainer.py:803] 2025-04-26 15:29:14,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2290
2321
[WARNING|trainer.py:803] 2025-04-26 15:29:14,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2332
[WARNING|trainer.py:803] 2025-04-26 15:29:15,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:15,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2291
2322
[WARNING|trainer.py:803] 2025-04-26 15:29:15,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2333
[WARNING|trainer.py:803] 2025-04-26 15:29:16,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:16,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2292
2323
[WARNING|trainer.py:803] 2025-04-26 15:29:17,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2334
[WARNING|trainer.py:803] 2025-04-26 15:29:17,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:17,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2293
2324
[WARNING|trainer.py:803] 2025-04-26 15:29:18,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2335
[WARNING|trainer.py:803] 2025-04-26 15:29:19,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:19,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2294
2325
[WARNING|trainer.py:803] 2025-04-26 15:29:19,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2336
[WARNING|trainer.py:803] 2025-04-26 15:29:20,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:29:20,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2295
2326
[WARNING|trainer.py:803] 2025-04-26 15:29:20,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2337
[WARNING|trainer.py:803] 2025-04-26 15:29:21,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:21,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2296
2327
[WARNING|trainer.py:803] 2025-04-26 15:29:22,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2338
[WARNING|trainer.py:803] 2025-04-26 15:29:22,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:22,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2297
[WARNING|trainer.py:803] 2025-04-26 15:29:23,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2328
2339
[WARNING|trainer.py:803] 2025-04-26 15:29:24,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:24,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:24,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2298
2329
2340
[WARNING|trainer.py:803] 2025-04-26 15:29:25,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:25,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:25,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2299
2330
2341
[WARNING|trainer.py:803] 2025-04-26 15:29:26,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:26,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:27,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2300
2331
2342
[WARNING|trainer.py:803] 2025-04-26 15:29:28,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:28,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:28,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2301
2332
2343
[WARNING|trainer.py:803] 2025-04-26 15:29:29,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 15:29:29,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:29,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2302
2333
2344
[WARNING|trainer.py:803] 2025-04-26 15:29:30,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:30,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:30,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2303
2334
2345
[WARNING|trainer.py:803] 2025-04-26 15:29:32,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:32,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:32,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2304
2335
2346
[WARNING|trainer.py:803] 2025-04-26 15:29:33,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:33,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:33,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2305
2336
2347
[WARNING|trainer.py:803] 2025-04-26 15:29:34,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:34,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:34,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2306
2337
2348
[WARNING|trainer.py:803] 2025-04-26 15:29:35,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:35,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:36,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2307
2338
2349
[WARNING|trainer.py:803] 2025-04-26 15:29:37,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:37,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:37,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2339
2308
2350
[WARNING|trainer.py:803] 2025-04-26 15:29:38,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:38,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:38,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2340
2309
2351
[WARNING|trainer.py:803] 2025-04-26 15:29:39,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:39,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:40,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2341
2310
2352
[WARNING|trainer.py:803] 2025-04-26 15:29:41,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:41,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:41,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2342
2311
2353
[WARNING|trainer.py:803] 2025-04-26 15:29:42,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:42,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:42,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2343
2312
2354
[WARNING|trainer.py:803] 2025-04-26 15:29:43,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:43,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:29:43,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2344
2313
2355
[WARNING|trainer.py:803] 2025-04-26 15:29:44,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:45,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:45,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2345
2314
2356
[WARNING|trainer.py:803] 2025-04-26 15:29:46,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:46,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:46,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2346
2315
2357
[WARNING|trainer.py:803] 2025-04-26 15:29:47,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:47,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:47,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2347
2316
2358
[WARNING|trainer.py:803] 2025-04-26 15:29:48,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:48,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:48,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2348
2317
2359
[WARNING|trainer.py:803] 2025-04-26 15:29:50,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:50,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:50,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2349
2318
2360
[WARNING|trainer.py:803] 2025-04-26 15:29:51,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:51,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:51,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2350
2319
2361
[WARNING|trainer.py:803] 2025-04-26 15:29:52,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:52,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:52,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2351
2362
2320
[WARNING|trainer.py:803] 2025-04-26 15:29:53,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:54,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:54,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2352
2363
2321
[WARNING|trainer.py:803] 2025-04-26 15:29:55,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:55,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:29:55,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2353
2364
2322
[WARNING|trainer.py:803] 2025-04-26 15:29:56,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:56,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:29:56,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2354
2365
2323
[WARNING|trainer.py:803] 2025-04-26 15:29:57,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:57,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:29:58,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2355
2366
2324
[WARNING|trainer.py:803] 2025-04-26 15:29:59,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:59,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:29:59,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2356
2367
2325
[WARNING|trainer.py:803] 2025-04-26 15:30:00,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:00,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:00,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2357
2368
2326
[WARNING|trainer.py:803] 2025-04-26 15:30:01,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:01,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:02,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2369
2358
2327
[WARNING|trainer.py:803] 2025-04-26 15:30:02,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:03,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:03,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2370
2359
2328
[WARNING|trainer.py:803] 2025-04-26 15:30:04,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:04,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:04,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2371
2360
2329
[WARNING|trainer.py:803] 2025-04-26 15:30:05,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:05,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:05,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2372
2361
2330
[WARNING|trainer.py:803] 2025-04-26 15:30:06,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:06,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:07,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2373
2362
2331
[WARNING|trainer.py:803] 2025-04-26 15:30:08,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:08,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:08,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2374
2363
2332
[WARNING|trainer.py:803] 2025-04-26 15:30:09,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:09,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2375
[WARNING|trainer.py:803] 2025-04-26 15:30:09,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2364
2333
[WARNING|trainer.py:803] 2025-04-26 15:30:10,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:10,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2376
[WARNING|trainer.py:803] 2025-04-26 15:30:11,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2365
2334
[WARNING|trainer.py:803] 2025-04-26 15:30:11,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:12,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2377
[WARNING|trainer.py:803] 2025-04-26 15:30:12,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2366
2335
[WARNING|trainer.py:803] 2025-04-26 15:30:13,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:13,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2378
[WARNING|trainer.py:803] 2025-04-26 15:30:13,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2367
2336
[WARNING|trainer.py:803] 2025-04-26 15:30:14,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:14,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2379
2368
[WARNING|trainer.py:803] 2025-04-26 15:30:15,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2337
[WARNING|trainer.py:803] 2025-04-26 15:30:15,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:15,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2380
2369
[WARNING|trainer.py:803] 2025-04-26 15:30:16,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2338
[WARNING|trainer.py:803] 2025-04-26 15:30:16,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:17,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2381
2370
[WARNING|trainer.py:803] 2025-04-26 15:30:17,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2339
[WARNING|trainer.py:803] 2025-04-26 15:30:18,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:18,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2382
2371
[WARNING|trainer.py:803] 2025-04-26 15:30:18,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:19,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2340
[WARNING|trainer.py:803] 2025-04-26 15:30:19,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2383
2372
[WARNING|trainer.py:803] 2025-04-26 15:30:20,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:20,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2341
[WARNING|trainer.py:803] 2025-04-26 15:30:20,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2384
2373
[WARNING|trainer.py:803] 2025-04-26 15:30:21,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:21,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2342
[WARNING|trainer.py:803] 2025-04-26 15:30:22,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2385
2374
[WARNING|trainer.py:803] 2025-04-26 15:30:22,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:23,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2343
[WARNING|trainer.py:803] 2025-04-26 15:30:23,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2386
2375
[WARNING|trainer.py:803] 2025-04-26 15:30:24,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:24,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2344
[WARNING|trainer.py:803] 2025-04-26 15:30:24,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2387
2376
[WARNING|trainer.py:803] 2025-04-26 15:30:25,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:25,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2345
2388
[WARNING|trainer.py:803] 2025-04-26 15:30:26,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2377
[WARNING|trainer.py:803] 2025-04-26 15:30:26,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:26,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2346
2389
[WARNING|trainer.py:803] 2025-04-26 15:30:27,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2378
[WARNING|trainer.py:803] 2025-04-26 15:30:27,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:28,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2347
2390
[WARNING|trainer.py:803] 2025-04-26 15:30:28,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:29,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2379
[WARNING|trainer.py:803] 2025-04-26 15:30:29,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2348
2391
[WARNING|trainer.py:803] 2025-04-26 15:30:30,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:30,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2380
[WARNING|trainer.py:803] 2025-04-26 15:30:30,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2349
2392
[WARNING|trainer.py:803] 2025-04-26 15:30:31,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:31,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2381
[WARNING|trainer.py:803] 2025-04-26 15:30:31,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2350
2393
[WARNING|trainer.py:803] 2025-04-26 15:30:32,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:32,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2382
[WARNING|trainer.py:803] 2025-04-26 15:30:33,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2351
2394
[WARNING|trainer.py:803] 2025-04-26 15:30:34,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:34,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2383
[WARNING|trainer.py:803] 2025-04-26 15:30:34,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2352
2395
[WARNING|trainer.py:803] 2025-04-26 15:30:35,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:35,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2384
[WARNING|trainer.py:803] 2025-04-26 15:30:35,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2353
2396
[WARNING|trainer.py:803] 2025-04-26 15:30:36,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:36,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2385
[WARNING|trainer.py:803] 2025-04-26 15:30:37,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2354
2397
[WARNING|trainer.py:803] 2025-04-26 15:30:37,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:38,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2386
[WARNING|trainer.py:803] 2025-04-26 15:30:38,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2355
2398
[WARNING|trainer.py:803] 2025-04-26 15:30:39,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:39,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2387
2356
[WARNING|trainer.py:803] 2025-04-26 15:30:39,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2399
[WARNING|trainer.py:803] 2025-04-26 15:30:40,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:40,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2388
2357
[WARNING|trainer.py:803] 2025-04-26 15:30:41,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2400
[WARNING|trainer.py:803] 2025-04-26 15:30:41,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:41,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2389
2358
[WARNING|trainer.py:803] 2025-04-26 15:30:42,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:43,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:43,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2390
2359
2401
[WARNING|trainer.py:803] 2025-04-26 15:30:44,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:44,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:44,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2391
2360
[WARNING|trainer.py:803] 2025-04-26 15:30:45,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:45,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2392
2361
2402
[WARNING|trainer.py:803] 2025-04-26 15:30:46,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:46,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:47,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2362
2393
[WARNING|trainer.py:803] 2025-04-26 15:30:48,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:48,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2403
2363
2394
[WARNING|trainer.py:803] 2025-04-26 15:30:49,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:49,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:30:49,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2364
2395
[WARNING|trainer.py:803] 2025-04-26 15:30:50,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:50,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2404
2365
2396
[WARNING|trainer.py:803] 2025-04-26 15:30:51,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:52,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:52,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2366
2397
2405
[WARNING|trainer.py:803] 2025-04-26 15:30:53,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:53,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2367
[WARNING|trainer.py:803] 2025-04-26 15:30:54,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2398
[WARNING|trainer.py:803] 2025-04-26 15:30:54,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:54,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2368
2399
2406
[WARNING|trainer.py:803] 2025-04-26 15:30:55,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:30:56,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:30:56,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2369
2400
[WARNING|trainer.py:803] 2025-04-26 15:30:57,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:57,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2370
[WARNING|trainer.py:803] 2025-04-26 15:30:58,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2371
2401
[WARNING|trainer.py:803] 2025-04-26 15:30:59,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:30:59,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2372
[WARNING|trainer.py:803] 2025-04-26 15:31:01,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2407
2373
2402
[WARNING|trainer.py:803] 2025-04-26 15:31:01,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:02,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:02,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2374
2408
[WARNING|trainer.py:803] 2025-04-26 15:31:03,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2403
2375
[WARNING|trainer.py:803] 2025-04-26 15:31:04,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:04,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:04,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2376
2409
[WARNING|trainer.py:803] 2025-04-26 15:31:06,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2404
2377
[WARNING|trainer.py:803] 2025-04-26 15:31:06,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:07,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:07,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2378
2410
2405
[WARNING|trainer.py:803] 2025-04-26 15:31:08,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:08,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2379
[WARNING|trainer.py:803] 2025-04-26 15:31:09,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:10,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2411
2380
[WARNING|trainer.py:803] 2025-04-26 15:31:10,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2406
[WARNING|trainer.py:803] 2025-04-26 15:31:11,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2381
[WARNING|trainer.py:803] 2025-04-26 15:31:11,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:12,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2412
2382
[WARNING|trainer.py:803] 2025-04-26 15:31:13,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:31:13,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2383
2413
[WARNING|trainer.py:803] 2025-04-26 15:31:15,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2384
[WARNING|trainer.py:803] 2025-04-26 15:31:15,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:16,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2407
2385
2414
[WARNING|trainer.py:803] 2025-04-26 15:31:17,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:17,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2386
[WARNING|trainer.py:803] 2025-04-26 15:31:18,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2408
[WARNING|trainer.py:803] 2025-04-26 15:31:19,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2387
2415
[WARNING|trainer.py:803] 2025-04-26 15:31:19,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:20,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:20,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2388
2409
[WARNING|trainer.py:803] 2025-04-26 15:31:21,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2389
[WARNING|trainer.py:803] 2025-04-26 15:31:22,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2416
[WARNING|trainer.py:803] 2025-04-26 15:31:22,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:23,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2390
2410
[WARNING|trainer.py:803] 2025-04-26 15:31:24,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:24,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2417
2391
[WARNING|trainer.py:803] 2025-04-26 15:31:25,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:25,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2411
2392
[WARNING|trainer.py:803] 2025-04-26 15:31:26,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2418
[WARNING|trainer.py:803] 2025-04-26 15:31:26,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2393
[WARNING|trainer.py:803] 2025-04-26 15:31:27,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:28,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2412
2394
2419
[WARNING|trainer.py:803] 2025-04-26 15:31:28,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:31:29,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:29,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2395
2413
[WARNING|trainer.py:803] 2025-04-26 15:31:30,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2396
2420
[WARNING|trainer.py:803] 2025-04-26 15:31:31,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:32,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:32,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2397
2414
2421
[WARNING|trainer.py:803] 2025-04-26 15:31:33,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:33,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2398
[WARNING|trainer.py:803] 2025-04-26 15:31:34,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:31:34,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2415
2399
2422
[WARNING|trainer.py:803] 2025-04-26 15:31:36,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:31:36,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2400
[WARNING|trainer.py:803] 2025-04-26 15:31:36,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:37,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2416
2423
[WARNING|trainer.py:803] 2025-04-26 15:31:38,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2401
[WARNING|trainer.py:803] 2025-04-26 15:31:38,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:39,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2417
[WARNING|trainer.py:803] 2025-04-26 15:31:41,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2402
2424
[WARNING|trainer.py:803] 2025-04-26 15:31:42,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2418
[WARNING|trainer.py:803] 2025-04-26 15:31:42,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:43,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2425
2403
2419
[WARNING|trainer.py:803] 2025-04-26 15:31:44,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:31:44,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:45,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2404
2426
2420
[WARNING|trainer.py:803] 2025-04-26 15:31:47,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:47,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:31:47,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2427
2405
2421
[WARNING|trainer.py:803] 2025-04-26 15:31:49,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:49,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:49,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2406
2428
2422
[WARNING|trainer.py:803] 2025-04-26 15:31:51,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:31:52,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:52,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2429
2423
[WARNING|trainer.py:803] 2025-04-26 15:31:54,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:31:54,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2430
2407
[WARNING|trainer.py:803] 2025-04-26 15:31:56,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2424
[WARNING|trainer.py:803] 2025-04-26 15:31:57,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:31:58,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2431
2408
[WARNING|trainer.py:803] 2025-04-26 15:31:59,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2425
[WARNING|trainer.py:803] 2025-04-26 15:31:59,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:32:00,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2432
2409
[WARNING|trainer.py:803] 2025-04-26 15:32:01,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2426
[WARNING|trainer.py:803] 2025-04-26 15:32:02,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:32:02,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2433
2410
[WARNING|trainer.py:803] 2025-04-26 15:32:03,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2427
[WARNING|trainer.py:803] 2025-04-26 15:32:04,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:04,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2411
2434
[WARNING|trainer.py:803] 2025-04-26 15:32:06,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:06,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2428
[WARNING|trainer.py:803] 2025-04-26 15:32:07,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2435
2412
[WARNING|trainer.py:803] 2025-04-26 15:32:08,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:08,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2429
2436
[WARNING|trainer.py:803] 2025-04-26 15:32:10,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2413
[WARNING|trainer.py:803] 2025-04-26 15:32:10,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:32:11,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2430
2437
[WARNING|trainer.py:803] 2025-04-26 15:32:12,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:12,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2414
[WARNING|trainer.py:803] 2025-04-26 15:32:13,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2431
2438
[WARNING|trainer.py:803] 2025-04-26 15:32:14,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2415
[WARNING|trainer.py:803] 2025-04-26 15:32:15,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:32:15,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2432
2439
[WARNING|trainer.py:803] 2025-04-26 15:32:17,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:17,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2416
2433
[WARNING|trainer.py:803] 2025-04-26 15:32:18,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2440
[WARNING|trainer.py:803] 2025-04-26 15:32:19,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:20,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2417
[WARNING|trainer.py:803] 2025-04-26 15:32:20,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2434
2441
[WARNING|trainer.py:803] 2025-04-26 15:32:22,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:22,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2418
[WARNING|trainer.py:803] 2025-04-26 15:32:23,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2435
[WARNING|trainer.py:803] 2025-04-26 15:32:24,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2419
[WARNING|trainer.py:803] 2025-04-26 15:32:25,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2436
[WARNING|trainer.py:803] 2025-04-26 15:32:26,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2420
[WARNING|trainer.py:803] 2025-04-26 15:32:27,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2437
2442
[WARNING|trainer.py:803] 2025-04-26 15:32:28,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2421
[WARNING|trainer.py:803] 2025-04-26 15:32:29,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:32:29,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2443
2438
[WARNING|trainer.py:803] 2025-04-26 15:32:31,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:32:31,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2422
[WARNING|trainer.py:803] 2025-04-26 15:32:32,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2439
2444
[WARNING|trainer.py:803] 2025-04-26 15:32:33,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2423
[WARNING|trainer.py:803] 2025-04-26 15:32:34,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 15:32:34,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2445
2440
[WARNING|trainer.py:803] 2025-04-26 15:32:36,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:36,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2424
2441
2446
[WARNING|trainer.py:803] 2025-04-26 15:32:38,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:38,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:38,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2425
[WARNING|trainer.py:803] 2025-04-26 15:32:40,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2447
2426
[WARNING|trainer.py:803] 2025-04-26 15:32:42,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:32:42,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2448
2427
2442
[WARNING|trainer.py:803] 2025-04-26 15:32:44,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:32:44,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:32:45,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2443
2428
[WARNING|trainer.py:803] 2025-04-26 15:32:47,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:32:47,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2444
2429
[WARNING|trainer.py:803] 2025-04-26 15:32:50,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 15:32:50,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2449
[WARNING|trainer.py:803] 2025-04-26 15:32:51,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2445
2430
[WARNING|trainer.py:803] 2025-04-26 15:32:52,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:52,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2450
[WARNING|trainer.py:803] 2025-04-26 15:32:54,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2446
2431
[WARNING|trainer.py:803] 2025-04-26 15:32:54,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:32:55,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2451
[WARNING|trainer.py:803] 2025-04-26 15:32:56,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2432
[WARNING|trainer.py:803] 2025-04-26 15:32:57,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2447
2452
[WARNING|trainer.py:803] 2025-04-26 15:32:58,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:32:58,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2433
[WARNING|trainer.py:803] 2025-04-26 15:32:59,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2448
2453
[WARNING|trainer.py:803] 2025-04-26 15:33:00,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:01,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2434
[WARNING|trainer.py:803] 2025-04-26 15:33:02,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2454
[WARNING|trainer.py:803] 2025-04-26 15:33:03,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2435
[WARNING|trainer.py:803] 2025-04-26 15:33:04,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2455
[WARNING|trainer.py:803] 2025-04-26 15:33:05,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2436
[WARNING|trainer.py:803] 2025-04-26 15:33:06,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2449
[WARNING|trainer.py:803] 2025-04-26 15:33:07,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2437
[WARNING|trainer.py:803] 2025-04-26 15:33:08,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2450
2456
[WARNING|trainer.py:803] 2025-04-26 15:33:10,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:10,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2438
[WARNING|trainer.py:803] 2025-04-26 15:33:11,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2457
2451
[WARNING|trainer.py:803] 2025-04-26 15:33:12,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2439
[WARNING|trainer.py:803] 2025-04-26 15:33:12,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:13,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2458
2452
[WARNING|trainer.py:803] 2025-04-26 15:33:14,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:33:15,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2440
[WARNING|trainer.py:803] 2025-04-26 15:33:16,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2459
2453
[WARNING|trainer.py:803] 2025-04-26 15:33:17,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2441
[WARNING|trainer.py:803] 2025-04-26 15:33:17,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:33:18,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2454
2460
[WARNING|trainer.py:803] 2025-04-26 15:33:19,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:19,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2461
2455
[WARNING|trainer.py:803] 2025-04-26 15:33:21,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:22,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2462
2442
[WARNING|trainer.py:803] 2025-04-26 15:33:24,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:25,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2456
2443
[WARNING|trainer.py:803] 2025-04-26 15:33:26,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:27,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2457
2463
[WARNING|trainer.py:803] 2025-04-26 15:33:29,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2444
[WARNING|trainer.py:803] 2025-04-26 15:33:29,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:30,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
2458
[WARNING|trainer.py:803] 2025-04-26 15:33:31,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2445
[WARNING|trainer.py:803] 2025-04-26 15:33:32,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2459
[WARNING|trainer.py:803] 2025-04-26 15:33:33,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2446
2464
[WARNING|trainer.py:803] 2025-04-26 15:33:34,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:34,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2460
[WARNING|trainer.py:803] 2025-04-26 15:33:36,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2461
2447
[WARNING|trainer.py:803] 2025-04-26 15:33:38,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:38,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2448
2465
2462
[WARNING|trainer.py:803] 2025-04-26 15:33:40,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:33:40,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:41,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2466
2463
[WARNING|trainer.py:803] 2025-04-26 15:33:46,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:33:46,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2449
[WARNING|trainer.py:803] 2025-04-26 15:33:47,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2450
[WARNING|trainer.py:803] 2025-04-26 15:33:50,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2464
2467
[WARNING|trainer.py:803] 2025-04-26 15:33:51,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:51,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2451
[WARNING|trainer.py:803] 2025-04-26 15:33:52,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2452
[WARNING|trainer.py:803] 2025-04-26 15:33:55,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2453
2465
2468
[WARNING|trainer.py:803] 2025-04-26 15:33:57,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:33:57,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:33:57,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2454
[WARNING|trainer.py:803] 2025-04-26 15:33:59,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2455
[WARNING|trainer.py:803] 2025-04-26 15:34:02,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2469
2466
[WARNING|trainer.py:803] 2025-04-26 15:34:03,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:03,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2456
[WARNING|trainer.py:803] 2025-04-26 15:34:07,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2470
2467
2457
[WARNING|trainer.py:803] 2025-04-26 15:34:08,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:08,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:34:09,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2458
[WARNING|trainer.py:803] 2025-04-26 15:34:11,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2459
2471
[WARNING|trainer.py:803] 2025-04-26 15:34:14,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2468
[WARNING|trainer.py:803] 2025-04-26 15:34:14,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:15,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2460
2472
[WARNING|trainer.py:803] 2025-04-26 15:34:16,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:17,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2461
2473
[WARNING|trainer.py:803] 2025-04-26 15:34:18,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:34:19,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2469
[WARNING|trainer.py:803] 2025-04-26 15:34:20,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2474
2462
[WARNING|trainer.py:803] 2025-04-26 15:34:21,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:21,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2475
2470
[WARNING|trainer.py:803] 2025-04-26 15:34:25,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:25,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2463
2476
[WARNING|trainer.py:803] 2025-04-26 15:34:26,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:34:27,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2477
[WARNING|trainer.py:803] 2025-04-26 15:34:29,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2471
2478
2464
[WARNING|trainer.py:803] 2025-04-26 15:34:31,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:31,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:31,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2472
2479
[WARNING|trainer.py:803] 2025-04-26 15:34:33,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:33,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2473
2480
[WARNING|trainer.py:803] 2025-04-26 15:34:35,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:34:36,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2474
2465
2481
[WARNING|trainer.py:803] 2025-04-26 15:34:37,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:38,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:38,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2482
[WARNING|trainer.py:803] 2025-04-26 15:34:40,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2475
[WARNING|trainer.py:803] 2025-04-26 15:34:41,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2466
2476
2483
[WARNING|trainer.py:803] 2025-04-26 15:34:43,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:43,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:43,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2477
[WARNING|trainer.py:803] 2025-04-26 15:34:45,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2484
[WARNING|trainer.py:803] 2025-04-26 15:34:47,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2478
2467
[WARNING|trainer.py:803] 2025-04-26 15:34:48,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2485
[WARNING|trainer.py:803] 2025-04-26 15:34:48,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:34:49,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2479
[WARNING|trainer.py:803] 2025-04-26 15:34:50,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2486
2480
[WARNING|trainer.py:803] 2025-04-26 15:34:51,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:52,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2487
2468
2481
[WARNING|trainer.py:803] 2025-04-26 15:34:54,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:54,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:34:54,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2488
2482
[WARNING|trainer.py:803] 2025-04-26 15:34:56,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:34:56,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2489
[WARNING|trainer.py:803] 2025-04-26 15:34:58,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2469
2483
2490
[WARNING|trainer.py:803] 2025-04-26 15:35:00,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:00,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:00,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2491
2484
[WARNING|trainer.py:803] 2025-04-26 15:35:03,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:35:03,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2470
2492
2485
[WARNING|trainer.py:803] 2025-04-26 15:35:05,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:05,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:06,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2493
2486
[WARNING|trainer.py:803] 2025-04-26 15:35:08,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:08,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2494
[WARNING|trainer.py:803] 2025-04-26 15:35:10,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2471
2487
[WARNING|trainer.py:803] 2025-04-26 15:35:11,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:35:11,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2488
2495
2472
[WARNING|trainer.py:803] 2025-04-26 15:35:13,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:13,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:13,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2489
2496
2473
[WARNING|trainer.py:803] 2025-04-26 15:35:15,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:15,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:15,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2490
2497
2474
[WARNING|trainer.py:803] 2025-04-26 15:35:17,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:17,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:17,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2491
2498
[WARNING|trainer.py:803] 2025-04-26 15:35:20,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:35:20,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2475
2499
[WARNING|trainer.py:803] 2025-04-26 15:35:21,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2492
[WARNING|trainer.py:803] 2025-04-26 15:35:22,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2476
[WARNING|trainer.py:803] 2025-04-26 15:35:22,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:23,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2493
2477
[WARNING|trainer.py:803] 2025-04-26 15:35:25,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:25,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2494
2500
2478
[WARNING|trainer.py:803] 2025-04-26 15:35:27,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:35:27,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:35:28,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2501
2479
2495
[WARNING|trainer.py:803] 2025-04-26 15:35:30,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:30,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:35:30,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2496
2480
2502
[WARNING|trainer.py:803] 2025-04-26 15:35:32,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:32,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:35:32,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2497
2481
2503
[WARNING|trainer.py:803] 2025-04-26 15:35:34,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:34,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:34,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2482
2498
[WARNING|trainer.py:803] 2025-04-26 15:35:36,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2504
[WARNING|trainer.py:803] 2025-04-26 15:35:37,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:35:37,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2499
2505
[WARNING|trainer.py:803] 2025-04-26 15:35:39,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2483
[WARNING|trainer.py:803] 2025-04-26 15:35:40,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:40,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2506
[WARNING|trainer.py:803] 2025-04-26 15:35:42,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2484
[WARNING|trainer.py:803] 2025-04-26 15:35:43,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2500
2507
[WARNING|trainer.py:803] 2025-04-26 15:35:44,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:35:45,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2485
[WARNING|trainer.py:803] 2025-04-26 15:35:45,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2508
2501
[WARNING|trainer.py:803] 2025-04-26 15:35:47,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:47,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2486
2509
[WARNING|trainer.py:803] 2025-04-26 15:35:48,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:49,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2502
[WARNING|trainer.py:803] 2025-04-26 15:35:50,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2487
2510
[WARNING|trainer.py:803] 2025-04-26 15:35:51,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2503
[WARNING|trainer.py:803] 2025-04-26 15:35:51,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:35:52,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2488
2511
[WARNING|trainer.py:803] 2025-04-26 15:35:53,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:53,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2489
2504
2512
[WARNING|trainer.py:803] 2025-04-26 15:35:55,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:55,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:35:55,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2490
2505
[WARNING|trainer.py:803] 2025-04-26 15:35:57,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2513
[WARNING|trainer.py:803] 2025-04-26 15:35:57,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:58,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2506
2491
2514
[WARNING|trainer.py:803] 2025-04-26 15:35:59,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:35:59,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:36:00,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2507
2492
2515
[WARNING|trainer.py:803] 2025-04-26 15:36:02,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:02,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:03,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2508
2493
2516
[WARNING|trainer.py:803] 2025-04-26 15:36:04,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:04,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:05,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2509
2494
[WARNING|trainer.py:803] 2025-04-26 15:36:06,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:07,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2517
[WARNING|trainer.py:803] 2025-04-26 15:36:08,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2510
[WARNING|trainer.py:803] 2025-04-26 15:36:09,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2495
[WARNING|trainer.py:803] 2025-04-26 15:36:10,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2518
2511
[WARNING|trainer.py:803] 2025-04-26 15:36:11,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2496
[WARNING|trainer.py:803] 2025-04-26 15:36:11,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:12,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2512
2519
2497
[WARNING|trainer.py:803] 2025-04-26 15:36:13,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:36:13,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:36:14,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2513
2520
[WARNING|trainer.py:803] 2025-04-26 15:36:16,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2498
[WARNING|trainer.py:803] 2025-04-26 15:36:16,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:17,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2514
2499
[WARNING|trainer.py:803] 2025-04-26 15:36:18,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2521
[WARNING|trainer.py:803] 2025-04-26 15:36:19,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:19,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2515
[WARNING|trainer.py:803] 2025-04-26 15:36:21,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2516
[WARNING|trainer.py:803] 2025-04-26 15:36:23,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2522
2500
[WARNING|trainer.py:803] 2025-04-26 15:36:25,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:25,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2517
[WARNING|trainer.py:803] 2025-04-26 15:36:26,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2501
[WARNING|trainer.py:803] 2025-04-26 15:36:27,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2518
2502
[WARNING|trainer.py:803] 2025-04-26 15:36:29,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2523
[WARNING|trainer.py:803] 2025-04-26 15:36:30,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:30,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2519
2503
[WARNING|trainer.py:803] 2025-04-26 15:36:32,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:36:32,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2520
2504
[WARNING|trainer.py:803] 2025-04-26 15:36:34,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:35,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2524
[WARNING|trainer.py:803] 2025-04-26 15:36:36,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2521
2505
[WARNING|trainer.py:803] 2025-04-26 15:36:37,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:36:37,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2506
[WARNING|trainer.py:803] 2025-04-26 15:36:40,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2507
2522
2525
[WARNING|trainer.py:803] 2025-04-26 15:36:42,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:43,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:43,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2508
[WARNING|trainer.py:803] 2025-04-26 15:36:45,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2509
[WARNING|trainer.py:803] 2025-04-26 15:36:47,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2523
2526
2510
[WARNING|trainer.py:803] 2025-04-26 15:36:48,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:36:49,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:36:49,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2511
[WARNING|trainer.py:803] 2025-04-26 15:36:51,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2512
[WARNING|trainer.py:803] 2025-04-26 15:36:54,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2524
[WARNING|trainer.py:803] 2025-04-26 15:36:55,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2513
[WARNING|trainer.py:803] 2025-04-26 15:36:56,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2514
2527
[WARNING|trainer.py:803] 2025-04-26 15:36:58,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:36:59,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2515
2525
[WARNING|trainer.py:803] 2025-04-26 15:37:01,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:01,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2516
2528
[WARNING|trainer.py:803] 2025-04-26 15:37:03,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:37:04,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2517
[WARNING|trainer.py:803] 2025-04-26 15:37:06,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2526
[WARNING|trainer.py:803] 2025-04-26 15:37:08,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2529
2518
[WARNING|trainer.py:803] 2025-04-26 15:37:09,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:09,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2519
[WARNING|trainer.py:803] 2025-04-26 15:37:12,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2530
2520
[WARNING|trainer.py:803] 2025-04-26 15:37:14,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:15,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2527
2521
2531
[WARNING|trainer.py:803] 2025-04-26 15:37:17,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:18,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:37:18,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2532
[WARNING|trainer.py:803] 2025-04-26 15:37:21,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2528
2522
2533
[WARNING|trainer.py:803] 2025-04-26 15:37:23,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:23,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:23,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2534
[WARNING|trainer.py:803] 2025-04-26 15:37:26,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2529
[WARNING|trainer.py:803] 2025-04-26 15:37:28,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2535
2523
[WARNING|trainer.py:803] 2025-04-26 15:37:29,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:37:29,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2536
[WARNING|trainer.py:803] 2025-04-26 15:37:31,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2530
2537
[WARNING|trainer.py:803] 2025-04-26 15:37:33,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:37:33,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2524
2538
[WARNING|trainer.py:803] 2025-04-26 15:37:35,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:37:35,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2531
[WARNING|trainer.py:803] 2025-04-26 15:37:37,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2539
[WARNING|trainer.py:803] 2025-04-26 15:37:38,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2532
[WARNING|trainer.py:803] 2025-04-26 15:37:40,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2540
[WARNING|trainer.py:803] 2025-04-26 15:37:41,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2525
2533
[WARNING|trainer.py:803] 2025-04-26 15:37:42,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2541
[WARNING|trainer.py:803] 2025-04-26 15:37:43,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:37:43,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2534
2542
[WARNING|trainer.py:803] 2025-04-26 15:37:45,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:37:45,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2535
2543
2526
[WARNING|trainer.py:803] 2025-04-26 15:37:48,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:37:48,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:37:48,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2536
2544
[WARNING|trainer.py:803] 2025-04-26 15:37:50,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:37:51,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2537
[WARNING|trainer.py:803] 2025-04-26 15:37:52,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2538
2545
[WARNING|trainer.py:803] 2025-04-26 15:37:54,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:37:54,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2546
2539
[WARNING|trainer.py:803] 2025-04-26 15:37:57,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2527
[WARNING|trainer.py:803] 2025-04-26 15:37:57,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:37:58,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2547
2540
[WARNING|trainer.py:803] 2025-04-26 15:37:59,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:00,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2548
2541
[WARNING|trainer.py:803] 2025-04-26 15:38:01,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:02,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2528
2549
[WARNING|trainer.py:803] 2025-04-26 15:38:03,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2542
[WARNING|trainer.py:803] 2025-04-26 15:38:04,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:04,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2550
2543
[WARNING|trainer.py:803] 2025-04-26 15:38:06,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2529
[WARNING|trainer.py:803] 2025-04-26 15:38:07,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2551
[WARNING|trainer.py:803] 2025-04-26 15:38:08,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:09,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2544
[WARNING|trainer.py:803] 2025-04-26 15:38:10,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2552
[WARNING|trainer.py:803] 2025-04-26 15:38:11,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2530
2545
2553
[WARNING|trainer.py:803] 2025-04-26 15:38:13,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:14,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:14,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2546
[WARNING|trainer.py:803] 2025-04-26 15:38:16,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2554
2531
[WARNING|trainer.py:803] 2025-04-26 15:38:17,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2547
[WARNING|trainer.py:803] 2025-04-26 15:38:17,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2555
[WARNING|trainer.py:803] 2025-04-26 15:38:18,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:19,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2532
2548
[WARNING|trainer.py:803] 2025-04-26 15:38:20,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2556
[WARNING|trainer.py:803] 2025-04-26 15:38:21,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:21,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2549
2533
2557
[WARNING|trainer.py:803] 2025-04-26 15:38:23,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:23,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:23,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2534
2550
[WARNING|trainer.py:803] 2025-04-26 15:38:25,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:26,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2558
2551
2535
[WARNING|trainer.py:803] 2025-04-26 15:38:28,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:28,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:28,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2559
2536
2552
[WARNING|trainer.py:803] 2025-04-26 15:38:30,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:30,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:38:30,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2560
2537
2553
[WARNING|trainer.py:803] 2025-04-26 15:38:33,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:33,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:38:33,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2538
2561
[WARNING|trainer.py:803] 2025-04-26 15:38:35,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2554
[WARNING|trainer.py:803] 2025-04-26 15:38:35,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:36,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2562
2539
2555
[WARNING|trainer.py:803] 2025-04-26 15:38:37,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:38,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:38,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2563
2540
2556
[WARNING|trainer.py:803] 2025-04-26 15:38:40,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:40,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:40,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2541
2557
[WARNING|trainer.py:803] 2025-04-26 15:38:42,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:43,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2564
[WARNING|trainer.py:803] 2025-04-26 15:38:44,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2542
[WARNING|trainer.py:803] 2025-04-26 15:38:45,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2565
[WARNING|trainer.py:803] 2025-04-26 15:38:46,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2558
2543
[WARNING|trainer.py:803] 2025-04-26 15:38:47,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:48,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2566
2559
[WARNING|trainer.py:803] 2025-04-26 15:38:49,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:49,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2544
[WARNING|trainer.py:803] 2025-04-26 15:38:50,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2560
2567
[WARNING|trainer.py:803] 2025-04-26 15:38:52,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:53,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2545
2561
[WARNING|trainer.py:803] 2025-04-26 15:38:54,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:38:54,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2568
2546
2562
[WARNING|trainer.py:803] 2025-04-26 15:38:56,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:38:56,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:38:57,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2569
2547
2563
[WARNING|trainer.py:803] 2025-04-26 15:38:58,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:59,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:38:59,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2570
[WARNING|trainer.py:803] 2025-04-26 15:39:00,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2548
[WARNING|trainer.py:803] 2025-04-26 15:39:01,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2571
2564
[WARNING|trainer.py:803] 2025-04-26 15:39:03,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2549
[WARNING|trainer.py:803] 2025-04-26 15:39:03,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:03,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2572
2565
[WARNING|trainer.py:803] 2025-04-26 15:39:05,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:05,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2550
2573
[WARNING|trainer.py:803] 2025-04-26 15:39:06,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:07,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2566
2551
[WARNING|trainer.py:803] 2025-04-26 15:39:08,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2574
[WARNING|trainer.py:803] 2025-04-26 15:39:09,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:09,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2552
2575
2567
[WARNING|trainer.py:803] 2025-04-26 15:39:11,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:39:11,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:12,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2553
2576
[WARNING|trainer.py:803] 2025-04-26 15:39:14,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:14,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2568
[WARNING|trainer.py:803] 2025-04-26 15:39:15,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2577
2554
[WARNING|trainer.py:803] 2025-04-26 15:39:16,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2569
[WARNING|trainer.py:803] 2025-04-26 15:39:17,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:18,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2555
2578
2570
[WARNING|trainer.py:803] 2025-04-26 15:39:19,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:19,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:20,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2556
2571
2579
[WARNING|trainer.py:803] 2025-04-26 15:39:21,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:22,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:22,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2557
2572
[WARNING|trainer.py:803] 2025-04-26 15:39:24,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:24,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2580
[WARNING|trainer.py:803] 2025-04-26 15:39:25,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2573
[WARNING|trainer.py:803] 2025-04-26 15:39:26,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2581
2558
2574
[WARNING|trainer.py:803] 2025-04-26 15:39:27,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:28,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:28,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2582
2559
2575
[WARNING|trainer.py:803] 2025-04-26 15:39:30,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:30,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:31,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2583
2560
2576
[WARNING|trainer.py:803] 2025-04-26 15:39:33,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:33,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:33,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2584
2577
2561
[WARNING|trainer.py:803] 2025-04-26 15:39:35,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:36,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:39:36,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2562
2585
2578
[WARNING|trainer.py:803] 2025-04-26 15:39:38,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:38,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:38,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2586
2563
[WARNING|trainer.py:803] 2025-04-26 15:39:40,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:40,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2579
[WARNING|trainer.py:803] 2025-04-26 15:39:41,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2587
[WARNING|trainer.py:803] 2025-04-26 15:39:43,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2580
2564
[WARNING|trainer.py:803] 2025-04-26 15:39:44,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:39:44,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2588
2565
2581
[WARNING|trainer.py:803] 2025-04-26 15:39:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:46,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:39:46,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2589
2582
[WARNING|trainer.py:803] 2025-04-26 15:39:48,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2566
[WARNING|trainer.py:803] 2025-04-26 15:39:49,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:49,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2590
2583
[WARNING|trainer.py:803] 2025-04-26 15:39:51,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:52,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2567
2591
[WARNING|trainer.py:803] 2025-04-26 15:39:53,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2584
[WARNING|trainer.py:803] 2025-04-26 15:39:54,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:39:54,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2592
2568
2585
[WARNING|trainer.py:803] 2025-04-26 15:39:56,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:56,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:39:57,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2569
2586
2593
[WARNING|trainer.py:803] 2025-04-26 15:39:59,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:39:59,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:39:59,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2570
2594
2587
[WARNING|trainer.py:803] 2025-04-26 15:40:01,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:02,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:02,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2571
2595
[WARNING|trainer.py:803] 2025-04-26 15:40:03,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:40:04,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2588
2572
[WARNING|trainer.py:803] 2025-04-26 15:40:05,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:40:05,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2596
[WARNING|trainer.py:803] 2025-04-26 15:40:06,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2589
2573
[WARNING|trainer.py:803] 2025-04-26 15:40:07,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:40:07,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2597
2574
[WARNING|trainer.py:803] 2025-04-26 15:40:09,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:10,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2590
[WARNING|trainer.py:803] 2025-04-26 15:40:10,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2598
2575
[WARNING|trainer.py:803] 2025-04-26 15:40:11,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:12,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2591
2599
[WARNING|trainer.py:803] 2025-04-26 15:40:13,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:40:13,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2576
2592
[WARNING|trainer.py:803] 2025-04-26 15:40:15,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2600
[WARNING|trainer.py:803] 2025-04-26 15:40:16,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:16,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2577
[WARNING|trainer.py:803] 2025-04-26 15:40:17,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2593
2601
[WARNING|trainer.py:803] 2025-04-26 15:40:18,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:19,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2578
[WARNING|trainer.py:803] 2025-04-26 15:40:20,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2594
2602
[WARNING|trainer.py:803] 2025-04-26 15:40:21,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:21,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2579
2595
[WARNING|trainer.py:803] 2025-04-26 15:40:22,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:23,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2603
[WARNING|trainer.py:803] 2025-04-26 15:40:24,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2596
2580
2604
[WARNING|trainer.py:803] 2025-04-26 15:40:25,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:26,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:26,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2581
2597
[WARNING|trainer.py:803] 2025-04-26 15:40:28,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2605
[WARNING|trainer.py:803] 2025-04-26 15:40:28,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:29,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2582
2598
2606
[WARNING|trainer.py:803] 2025-04-26 15:40:30,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:40:30,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:31,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2599
2607
[WARNING|trainer.py:803] 2025-04-26 15:40:32,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2583
[WARNING|trainer.py:803] 2025-04-26 15:40:33,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:34,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2600
2608
[WARNING|trainer.py:803] 2025-04-26 15:40:35,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2584
[WARNING|trainer.py:803] 2025-04-26 15:40:35,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:36,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2609
2601
[WARNING|trainer.py:803] 2025-04-26 15:40:38,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2585
[WARNING|trainer.py:803] 2025-04-26 15:40:38,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:39,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2602
2586
[WARNING|trainer.py:803] 2025-04-26 15:40:40,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:41,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2610
[WARNING|trainer.py:803] 2025-04-26 15:40:42,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2603
2587
[WARNING|trainer.py:803] 2025-04-26 15:40:43,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:43,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2604
[WARNING|trainer.py:803] 2025-04-26 15:40:45,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2588
[WARNING|trainer.py:803] 2025-04-26 15:40:46,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2611
2605
[WARNING|trainer.py:803] 2025-04-26 15:40:48,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:48,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2589
[WARNING|trainer.py:803] 2025-04-26 15:40:49,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2612
2606
[WARNING|trainer.py:803] 2025-04-26 15:40:50,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:40:50,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2590
2613
2607
[WARNING|trainer.py:803] 2025-04-26 15:40:52,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:52,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:52,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2614
2591
2608
[WARNING|trainer.py:803] 2025-04-26 15:40:54,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:40:55,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:40:55,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2615
2609
2592
[WARNING|trainer.py:803] 2025-04-26 15:40:56,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:57,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:40:57,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2616
[WARNING|trainer.py:803] 2025-04-26 15:40:59,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2593
2617
2610
[WARNING|trainer.py:803] 2025-04-26 15:41:00,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:01,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:01,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2594
[WARNING|trainer.py:803] 2025-04-26 15:41:03,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2618
[WARNING|trainer.py:803] 2025-04-26 15:41:04,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2595
[WARNING|trainer.py:803] 2025-04-26 15:41:05,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2619
2611
[WARNING|trainer.py:803] 2025-04-26 15:41:06,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:06,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2596
[WARNING|trainer.py:803] 2025-04-26 15:41:07,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2620
2612
[WARNING|trainer.py:803] 2025-04-26 15:41:09,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:09,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2597
2621
2613
[WARNING|trainer.py:803] 2025-04-26 15:41:10,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:11,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:11,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2598
2622
[WARNING|trainer.py:803] 2025-04-26 15:41:12,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2614
[WARNING|trainer.py:803] 2025-04-26 15:41:13,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:13,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2599
[WARNING|trainer.py:803] 2025-04-26 15:41:14,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2615
2623
[WARNING|trainer.py:803] 2025-04-26 15:41:15,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:16,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2600
2616
[WARNING|trainer.py:803] 2025-04-26 15:41:17,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:17,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2624
2617
2601
[WARNING|trainer.py:803] 2025-04-26 15:41:19,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:20,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:20,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2625
2602
2618
[WARNING|trainer.py:803] 2025-04-26 15:41:22,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:22,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:22,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2619
2626
2603
[WARNING|trainer.py:803] 2025-04-26 15:41:25,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:25,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:25,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2627
2604
2620
[WARNING|trainer.py:803] 2025-04-26 15:41:27,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:27,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:27,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2628
2621
2605
[WARNING|trainer.py:803] 2025-04-26 15:41:29,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:29,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:30,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2629
2622
2606
[WARNING|trainer.py:803] 2025-04-26 15:41:32,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:32,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:32,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2630
2607
2623
[WARNING|trainer.py:803] 2025-04-26 15:41:34,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:34,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:35,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2608
2631
[WARNING|trainer.py:803] 2025-04-26 15:41:37,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:41:37,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2624
2609
[WARNING|trainer.py:803] 2025-04-26 15:41:38,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2632
[WARNING|trainer.py:803] 2025-04-26 15:41:39,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:39,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2625
[WARNING|trainer.py:803] 2025-04-26 15:41:40,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2633
[WARNING|trainer.py:803] 2025-04-26 15:41:42,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2626
2610
2634
[WARNING|trainer.py:803] 2025-04-26 15:41:43,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:43,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:44,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2627
2635
[WARNING|trainer.py:803] 2025-04-26 15:41:46,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:46,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2628
2636
[WARNING|trainer.py:803] 2025-04-26 15:41:48,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2611
[WARNING|trainer.py:803] 2025-04-26 15:41:49,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:49,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2629
2637
[WARNING|trainer.py:803] 2025-04-26 15:41:50,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:41:51,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2612
[WARNING|trainer.py:803] 2025-04-26 15:41:52,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2630
2638
[WARNING|trainer.py:803] 2025-04-26 15:41:53,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:53,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2613
[WARNING|trainer.py:803] 2025-04-26 15:41:54,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2639
[WARNING|trainer.py:803] 2025-04-26 15:41:55,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2631
2614
[WARNING|trainer.py:803] 2025-04-26 15:41:56,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2640
[WARNING|trainer.py:803] 2025-04-26 15:41:56,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:41:57,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2632
2615
[WARNING|trainer.py:803] 2025-04-26 15:41:58,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:41:58,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2641
[WARNING|trainer.py:803] 2025-04-26 15:41:59,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2633
2616
[WARNING|trainer.py:803] 2025-04-26 15:42:00,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:00,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2642
[WARNING|trainer.py:803] 2025-04-26 15:42:02,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2634
2617
[WARNING|trainer.py:803] 2025-04-26 15:42:02,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:03,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2635
2618
[WARNING|trainer.py:803] 2025-04-26 15:42:05,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:05,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2643
[WARNING|trainer.py:803] 2025-04-26 15:42:06,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2636
2619
[WARNING|trainer.py:803] 2025-04-26 15:42:07,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:42:07,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2637
2620
[WARNING|trainer.py:803] 2025-04-26 15:42:09,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:42:10,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2644
2638
[WARNING|trainer.py:803] 2025-04-26 15:42:11,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2621
[WARNING|trainer.py:803] 2025-04-26 15:42:11,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:12,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2639
[WARNING|trainer.py:803] 2025-04-26 15:42:13,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2622
[WARNING|trainer.py:803] 2025-04-26 15:42:14,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2640
2645
[WARNING|trainer.py:803] 2025-04-26 15:42:16,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:16,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2623
2641
[WARNING|trainer.py:803] 2025-04-26 15:42:18,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:18,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2642
2646
2624
[WARNING|trainer.py:803] 2025-04-26 15:42:20,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:21,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:42:21,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2625
[WARNING|trainer.py:803] 2025-04-26 15:42:23,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2647
2643
[WARNING|trainer.py:803] 2025-04-26 15:42:25,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:42:25,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2626
[WARNING|trainer.py:803] 2025-04-26 15:42:26,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2627
2648
[WARNING|trainer.py:803] 2025-04-26 15:42:29,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2644
[WARNING|trainer.py:803] 2025-04-26 15:42:29,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2628
[WARNING|trainer.py:803] 2025-04-26 15:42:30,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:31,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2629
2649
[WARNING|trainer.py:803] 2025-04-26 15:42:33,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:34,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2645
2630
[WARNING|trainer.py:803] 2025-04-26 15:42:35,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:35,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2631
2650
2646
[WARNING|trainer.py:803] 2025-04-26 15:42:39,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:39,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:40,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2632
2651
[WARNING|trainer.py:803] 2025-04-26 15:42:41,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:41,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2633
2652
[WARNING|trainer.py:803] 2025-04-26 15:42:43,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2647
[WARNING|trainer.py:803] 2025-04-26 15:42:43,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:44,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2653
2634
[WARNING|trainer.py:803] 2025-04-26 15:42:45,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:46,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2654
2635
2648
[WARNING|trainer.py:803] 2025-04-26 15:42:48,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:42:48,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:42:48,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2655
2636
[WARNING|trainer.py:803] 2025-04-26 15:42:50,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:42:50,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2656
2649
2637
[WARNING|trainer.py:803] 2025-04-26 15:42:52,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:52,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:42:52,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2657
2638
[WARNING|trainer.py:803] 2025-04-26 15:42:54,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:55,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2639
2658
[WARNING|trainer.py:803] 2025-04-26 15:42:57,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:57,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2650
[WARNING|trainer.py:803] 2025-04-26 15:42:58,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2640
2659
[WARNING|trainer.py:803] 2025-04-26 15:42:59,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:42:59,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2651
[WARNING|trainer.py:803] 2025-04-26 15:43:00,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2660
2641
[WARNING|trainer.py:803] 2025-04-26 15:43:01,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:01,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2652
[WARNING|trainer.py:803] 2025-04-26 15:43:02,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2661
2642
[WARNING|trainer.py:803] 2025-04-26 15:43:03,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2653
[WARNING|trainer.py:803] 2025-04-26 15:43:04,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:04,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2662
[WARNING|trainer.py:803] 2025-04-26 15:43:05,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2654
[WARNING|trainer.py:803] 2025-04-26 15:43:06,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2643
2663
2655
[WARNING|trainer.py:803] 2025-04-26 15:43:08,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:43:08,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:43:09,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2656
2664
[WARNING|trainer.py:803] 2025-04-26 15:43:11,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:11,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2657
2644
[WARNING|trainer.py:803] 2025-04-26 15:43:13,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2665
[WARNING|trainer.py:803] 2025-04-26 15:43:13,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:14,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2658
[WARNING|trainer.py:803] 2025-04-26 15:43:16,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2659
2645
[WARNING|trainer.py:803] 2025-04-26 15:43:18,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2666
[WARNING|trainer.py:803] 2025-04-26 15:43:18,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:19,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2660
[WARNING|trainer.py:803] 2025-04-26 15:43:20,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2667
[WARNING|trainer.py:803] 2025-04-26 15:43:21,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2661
[WARNING|trainer.py:803] 2025-04-26 15:43:22,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2646
2668
[WARNING|trainer.py:803] 2025-04-26 15:43:23,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:43:23,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2662
[WARNING|trainer.py:803] 2025-04-26 15:43:24,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2669
[WARNING|trainer.py:803] 2025-04-26 15:43:26,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2663
2647
2670
[WARNING|trainer.py:803] 2025-04-26 15:43:27,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:43:28,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:43:28,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2671
2664
[WARNING|trainer.py:803] 2025-04-26 15:43:30,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:43:30,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2648
2665
[WARNING|trainer.py:803] 2025-04-26 15:43:32,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2672
[WARNING|trainer.py:803] 2025-04-26 15:43:33,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:33,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2673
[WARNING|trainer.py:803] 2025-04-26 15:43:35,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2649
[WARNING|trainer.py:803] 2025-04-26 15:43:36,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2666
2674
[WARNING|trainer.py:803] 2025-04-26 15:43:38,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:43:39,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2667
[WARNING|trainer.py:803] 2025-04-26 15:43:40,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2650
2668
[WARNING|trainer.py:803] 2025-04-26 15:43:42,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:42,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2651
2669
[WARNING|trainer.py:803] 2025-04-26 15:43:44,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:45,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2652
2670
[WARNING|trainer.py:803] 2025-04-26 15:43:46,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:47,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2653
2675
2671
[WARNING|trainer.py:803] 2025-04-26 15:43:48,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:49,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:49,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2654
2676
[WARNING|trainer.py:803] 2025-04-26 15:43:51,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2672
[WARNING|trainer.py:803] 2025-04-26 15:43:51,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:43:52,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2655
2677
[WARNING|trainer.py:803] 2025-04-26 15:43:53,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2673
[WARNING|trainer.py:803] 2025-04-26 15:43:53,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:54,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2656
2678
[WARNING|trainer.py:803] 2025-04-26 15:43:55,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:55,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2657
2679
2674
[WARNING|trainer.py:803] 2025-04-26 15:43:57,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:43:57,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:43:57,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2680
2658
[WARNING|trainer.py:803] 2025-04-26 15:43:59,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:00,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2681
2659
[WARNING|trainer.py:803] 2025-04-26 15:44:02,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:02,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2660
2682
[WARNING|trainer.py:803] 2025-04-26 15:44:04,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:44:04,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2683
2661
2675
[WARNING|trainer.py:803] 2025-04-26 15:44:06,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:07,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:07,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2684
2662
[WARNING|trainer.py:803] 2025-04-26 15:44:08,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2676
[WARNING|trainer.py:803] 2025-04-26 15:44:09,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:09,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2685
2677
2663
[WARNING|trainer.py:803] 2025-04-26 15:44:11,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:44:11,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:12,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2678
2686
[WARNING|trainer.py:803] 2025-04-26 15:44:13,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:14,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2664
2679
[WARNING|trainer.py:803] 2025-04-26 15:44:15,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2687
[WARNING|trainer.py:803] 2025-04-26 15:44:15,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:16,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2665
2680
[WARNING|trainer.py:803] 2025-04-26 15:44:17,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2688
[WARNING|trainer.py:803] 2025-04-26 15:44:17,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:18,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2681
2689
[WARNING|trainer.py:803] 2025-04-26 15:44:20,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:20,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2666
2682
2690
[WARNING|trainer.py:803] 2025-04-26 15:44:22,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:44:22,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:23,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2667
2683
2691
[WARNING|trainer.py:803] 2025-04-26 15:44:24,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:25,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:25,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2684
2668
2692
[WARNING|trainer.py:803] 2025-04-26 15:44:27,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:27,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:44:27,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2669
2685
2693
[WARNING|trainer.py:803] 2025-04-26 15:44:29,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:44:29,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:44:29,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2670
2694
2686
[WARNING|trainer.py:803] 2025-04-26 15:44:31,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:44:31,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:32,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2671
2695
2687
[WARNING|trainer.py:803] 2025-04-26 15:44:33,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:34,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:34,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2672
2688
2696
[WARNING|trainer.py:803] 2025-04-26 15:44:36,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:36,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:36,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2689
2673
2697
[WARNING|trainer.py:803] 2025-04-26 15:44:38,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:44:38,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:39,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2690
2698
[WARNING|trainer.py:803] 2025-04-26 15:44:41,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:41,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2674
[WARNING|trainer.py:803] 2025-04-26 15:44:42,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2691
2699
[WARNING|trainer.py:803] 2025-04-26 15:44:43,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:43,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2692
2700
[WARNING|trainer.py:803] 2025-04-26 15:44:45,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:45,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2693
2701
[WARNING|trainer.py:803] 2025-04-26 15:44:47,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:48,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2694
[WARNING|trainer.py:803] 2025-04-26 15:44:50,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2702
[WARNING|trainer.py:803] 2025-04-26 15:44:51,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2675
2695
[WARNING|trainer.py:803] 2025-04-26 15:44:52,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:52,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2703
[WARNING|trainer.py:803] 2025-04-26 15:44:53,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2676
2696
[WARNING|trainer.py:803] 2025-04-26 15:44:54,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:44:54,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2704
2677
[WARNING|trainer.py:803] 2025-04-26 15:44:56,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2697
[WARNING|trainer.py:803] 2025-04-26 15:44:56,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:57,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2678
2705
2698
[WARNING|trainer.py:803] 2025-04-26 15:44:58,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:58,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:44:59,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2679
2706
[WARNING|trainer.py:803] 2025-04-26 15:45:00,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2699
[WARNING|trainer.py:803] 2025-04-26 15:45:01,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:01,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2680
[WARNING|trainer.py:803] 2025-04-26 15:45:02,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2707
2700
[WARNING|trainer.py:803] 2025-04-26 15:45:03,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:04,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2681
[WARNING|trainer.py:803] 2025-04-26 15:45:05,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2708
2701
[WARNING|trainer.py:803] 2025-04-26 15:45:06,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:06,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2682
[WARNING|trainer.py:803] 2025-04-26 15:45:08,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2709
2702
[WARNING|trainer.py:803] 2025-04-26 15:45:09,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:09,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2683
[WARNING|trainer.py:803] 2025-04-26 15:45:10,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2710
2703
2684
[WARNING|trainer.py:803] 2025-04-26 15:45:11,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:11,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:12,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2711
2704
2685
[WARNING|trainer.py:803] 2025-04-26 15:45:14,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:14,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:14,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2712
2705
2686
[WARNING|trainer.py:803] 2025-04-26 15:45:16,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:17,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:17,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2713
2706
2687
[WARNING|trainer.py:803] 2025-04-26 15:45:19,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:19,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:19,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2714
2688
2707
[WARNING|trainer.py:803] 2025-04-26 15:45:21,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:21,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:22,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2689
2715
[WARNING|trainer.py:803] 2025-04-26 15:45:23,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2708
[WARNING|trainer.py:803] 2025-04-26 15:45:24,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:45:24,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2690
2716
2709
[WARNING|trainer.py:803] 2025-04-26 15:45:26,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:27,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:27,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2691
2717
[WARNING|trainer.py:803] 2025-04-26 15:45:28,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2710
[WARNING|trainer.py:803] 2025-04-26 15:45:29,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:29,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2692
[WARNING|trainer.py:803] 2025-04-26 15:45:31,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2718
2711
[WARNING|trainer.py:803] 2025-04-26 15:45:32,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2693
[WARNING|trainer.py:803] 2025-04-26 15:45:32,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:33,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2719
2712
2694
[WARNING|trainer.py:803] 2025-04-26 15:45:34,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:34,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:35,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2720
2713
2695
[WARNING|trainer.py:803] 2025-04-26 15:45:37,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:45:37,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:37,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2721
2714
2696
[WARNING|trainer.py:803] 2025-04-26 15:45:39,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:45:40,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:40,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2722
2715
2697
[WARNING|trainer.py:803] 2025-04-26 15:45:42,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:42,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:45:42,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2698
2723
2716
[WARNING|trainer.py:803] 2025-04-26 15:45:45,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:45,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:45,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2699
2717
2724
[WARNING|trainer.py:803] 2025-04-26 15:45:47,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:47,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:47,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2700
2718
2725
[WARNING|trainer.py:803] 2025-04-26 15:45:49,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:45:50,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:50,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2701
2719
2726
[WARNING|trainer.py:803] 2025-04-26 15:45:52,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:52,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:52,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2702
2727
2720
[WARNING|trainer.py:803] 2025-04-26 15:45:54,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:55,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:45:55,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2703
2721
2728
[WARNING|trainer.py:803] 2025-04-26 15:45:57,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:45:58,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:45:58,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2704
2722
2729
[WARNING|trainer.py:803] 2025-04-26 15:46:00,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:00,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:00,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2705
2730
2723
[WARNING|trainer.py:803] 2025-04-26 15:46:02,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:03,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:03,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2706
2731
2724
[WARNING|trainer.py:803] 2025-04-26 15:46:05,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:05,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:05,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2707
2725
2732
[WARNING|trainer.py:803] 2025-04-26 15:46:07,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:08,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:08,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2708
2726
2733
[WARNING|trainer.py:803] 2025-04-26 15:46:10,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:10,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:11,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2709
2727
2734
[WARNING|trainer.py:803] 2025-04-26 15:46:13,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:13,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:13,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2710
2735
2728
[WARNING|trainer.py:803] 2025-04-26 15:46:15,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:16,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:16,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2711
2729
2736
[WARNING|trainer.py:803] 2025-04-26 15:46:18,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:18,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:18,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2712
2730
2737
[WARNING|trainer.py:803] 2025-04-26 15:46:20,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:21,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:21,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2713
2738
2731
[WARNING|trainer.py:803] 2025-04-26 15:46:23,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:23,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:23,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2714
2739
2732
[WARNING|trainer.py:803] 2025-04-26 15:46:26,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:26,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:26,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2715
2740
2733
[WARNING|trainer.py:803] 2025-04-26 15:46:28,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:46:28,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:29,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2716
2741
2734
[WARNING|trainer.py:803] 2025-04-26 15:46:31,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:31,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:31,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2717
2742
2735
[WARNING|trainer.py:803] 2025-04-26 15:46:33,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:34,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:34,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2718
2743
2736
[WARNING|trainer.py:803] 2025-04-26 15:46:36,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:36,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:36,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2719
2737
2744
[WARNING|trainer.py:803] 2025-04-26 15:46:39,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:39,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:39,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2720
2745
2738
[WARNING|trainer.py:803] 2025-04-26 15:46:41,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:41,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:41,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2721
2739
2746
[WARNING|trainer.py:803] 2025-04-26 15:46:44,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:46:44,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:44,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2740
2722
2747
[WARNING|trainer.py:803] 2025-04-26 15:46:47,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:47,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:47,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2741
2748
2723
[WARNING|trainer.py:803] 2025-04-26 15:46:49,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:46:49,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 15:46:49,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2742
2749
2724
[WARNING|trainer.py:803] 2025-04-26 15:46:52,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:52,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:52,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2743
2725
2750
[WARNING|trainer.py:803] 2025-04-26 15:46:54,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:54,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:54,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2744
2726
2751
[WARNING|trainer.py:803] 2025-04-26 15:46:57,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:46:57,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:46:57,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2745
2727
2752
[WARNING|trainer.py:803] 2025-04-26 15:46:59,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:00,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:00,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2746
2728
2753
[WARNING|trainer.py:803] 2025-04-26 15:47:02,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:02,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:02,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2747
2729
2754
[WARNING|trainer.py:803] 2025-04-26 15:47:05,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:05,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:05,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2748
2730
2755
[WARNING|trainer.py:803] 2025-04-26 15:47:07,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 15:47:07,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:07,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2749
2731
2756
[WARNING|trainer.py:803] 2025-04-26 15:47:10,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:10,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:10,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2750
2732
2757
[WARNING|trainer.py:803] 2025-04-26 15:47:12,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:12,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:13,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2758
2733
2751
[WARNING|trainer.py:803] 2025-04-26 15:47:15,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:15,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:15,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2759
2734
2752
[WARNING|trainer.py:803] 2025-04-26 15:47:18,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:18,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:18,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2760
2735
2753
[WARNING|trainer.py:803] 2025-04-26 15:47:20,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:20,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:20,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2761
2754
2736
[WARNING|trainer.py:803] 2025-04-26 15:47:23,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:23,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:23,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2762
2755
2737
[WARNING|trainer.py:803] 2025-04-26 15:47:25,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:26,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:26,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2763
2738
2756
[WARNING|trainer.py:803] 2025-04-26 15:47:28,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:47:28,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:28,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2764
2757
2739
[WARNING|trainer.py:803] 2025-04-26 15:47:31,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:31,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:31,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2758
2740
2765
[WARNING|trainer.py:803] 2025-04-26 15:47:33,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:33,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:33,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2766
2741
2759
[WARNING|trainer.py:803] 2025-04-26 15:47:36,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:36,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:36,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2767
2760
2742
[WARNING|trainer.py:803] 2025-04-26 15:47:38,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:47:38,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:38,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2761
2743
2768
[WARNING|trainer.py:803] 2025-04-26 15:47:41,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:41,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:41,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2762
2769
2744
[WARNING|trainer.py:803] 2025-04-26 15:47:43,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:44,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:44,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2745
2763
2770
[WARNING|trainer.py:803] 2025-04-26 15:47:46,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:46,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:46,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2746
2764
2771
[WARNING|trainer.py:803] 2025-04-26 15:47:49,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:49,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:49,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2747
2772
2765
[WARNING|trainer.py:803] 2025-04-26 15:47:51,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:51,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:51,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2766
2773
2748
[WARNING|trainer.py:803] 2025-04-26 15:47:54,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:54,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:47:54,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2767
2774
2749
[WARNING|trainer.py:803] 2025-04-26 15:47:57,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:47:57,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:57,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2768
2750
2775
[WARNING|trainer.py:803] 2025-04-26 15:47:59,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:47:59,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:47:59,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2769
2751
2776
[WARNING|trainer.py:803] 2025-04-26 15:48:02,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:02,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:02,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2752
2777
2770
[WARNING|trainer.py:803] 2025-04-26 15:48:04,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:48:05,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:05,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2771
2778
2753
[WARNING|trainer.py:803] 2025-04-26 15:48:07,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:07,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:07,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2754
2779
2772
[WARNING|trainer.py:803] 2025-04-26 15:48:10,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:10,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:10,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2773
2755
2780
[WARNING|trainer.py:803] 2025-04-26 15:48:12,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:48:12,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:12,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2781
2756
2774
[WARNING|trainer.py:803] 2025-04-26 15:48:15,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:15,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:48:15,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2757
2782
2775
[WARNING|trainer.py:803] 2025-04-26 15:48:18,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:18,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:18,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2758
2783
2776
[WARNING|trainer.py:803] 2025-04-26 15:48:20,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:20,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:20,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2759
2784
2777
[WARNING|trainer.py:803] 2025-04-26 15:48:23,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:23,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:23,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2778
2785
2760
[WARNING|trainer.py:803] 2025-04-26 15:48:25,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:25,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:25,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2761
2779
2786
[WARNING|trainer.py:803] 2025-04-26 15:48:28,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:28,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:28,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2762
2780
2787
[WARNING|trainer.py:803] 2025-04-26 15:48:30,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:31,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:48:31,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2781
2763
2788
[WARNING|trainer.py:803] 2025-04-26 15:48:33,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:33,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:48:33,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2789
2782
2764
[WARNING|trainer.py:803] 2025-04-26 15:48:36,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:36,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:36,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2790
2783
2765
[WARNING|trainer.py:803] 2025-04-26 15:48:38,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:38,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:39,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2766
2784
2791
[WARNING|trainer.py:803] 2025-04-26 15:48:41,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:41,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:48:41,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2785
2767
2792
[WARNING|trainer.py:803] 2025-04-26 15:48:44,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:44,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:48:44,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2786
2768
2793
[WARNING|trainer.py:803] 2025-04-26 15:48:46,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:46,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:46,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2787
2769
2794
[WARNING|trainer.py:803] 2025-04-26 15:48:49,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:49,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:49,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2788
2795
2770
[WARNING|trainer.py:803] 2025-04-26 15:48:52,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:52,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:52,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2789
2771
2796
[WARNING|trainer.py:803] 2025-04-26 15:48:54,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:54,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:54,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2790
2772
2797
[WARNING|trainer.py:803] 2025-04-26 15:48:57,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:48:57,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:48:57,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2791
2773
2798
[WARNING|trainer.py:803] 2025-04-26 15:48:59,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:48:59,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:48:59,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2792
2774
2799
[WARNING|trainer.py:803] 2025-04-26 15:49:02,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:02,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:02,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2793
2775
2800
[WARNING|trainer.py:803] 2025-04-26 15:49:04,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:05,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:05,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2776
2794
2801
[WARNING|trainer.py:803] 2025-04-26 15:49:07,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:07,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:07,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2777
2795
2802
[WARNING|trainer.py:803] 2025-04-26 15:49:10,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:10,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:10,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2778
2796
2803
[WARNING|trainer.py:803] 2025-04-26 15:49:12,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:12,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:12,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2779
2797
2804
[WARNING|trainer.py:803] 2025-04-26 15:49:15,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:15,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:15,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2780
2805
2798
[WARNING|trainer.py:803] 2025-04-26 15:49:17,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:18,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:18,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2781
2806
2799
[WARNING|trainer.py:803] 2025-04-26 15:49:20,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:20,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:20,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2782
2807
2800
[WARNING|trainer.py:803] 2025-04-26 15:49:23,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:23,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:23,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2783
2808
2801
[WARNING|trainer.py:803] 2025-04-26 15:49:25,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:25,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:26,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2784
2809
2802
[WARNING|trainer.py:803] 2025-04-26 15:49:28,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:28,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:28,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2785
2810
2803
[WARNING|trainer.py:803] 2025-04-26 15:49:30,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:30,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:31,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2786
2811
2804
[WARNING|trainer.py:803] 2025-04-26 15:49:33,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:33,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:33,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2812
2787
2805
[WARNING|trainer.py:803] 2025-04-26 15:49:36,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:36,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:36,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2788
2813
2806
[WARNING|trainer.py:803] 2025-04-26 15:49:38,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:38,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:39,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2789
2814
2807
[WARNING|trainer.py:803] 2025-04-26 15:49:41,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:41,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:41,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2790
2815
2808
[WARNING|trainer.py:803] 2025-04-26 15:49:43,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:44,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:44,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2791
2809
2816
[WARNING|trainer.py:803] 2025-04-26 15:49:46,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:49:46,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:46,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2792
2817
2810
[WARNING|trainer.py:803] 2025-04-26 15:49:49,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:49,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:49,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2793
2811
2818
[WARNING|trainer.py:803] 2025-04-26 15:49:51,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:51,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:52,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2794
2812
2819
[WARNING|trainer.py:803] 2025-04-26 15:49:54,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:54,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:54,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2795
2813
2820
[WARNING|trainer.py:803] 2025-04-26 15:49:57,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:57,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:49:57,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2796
2814
2821
[WARNING|trainer.py:803] 2025-04-26 15:49:59,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:49:59,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:49:59,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2815
2797
2822
[WARNING|trainer.py:803] 2025-04-26 15:50:02,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:02,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:02,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2816
2798
2823
[WARNING|trainer.py:803] 2025-04-26 15:50:05,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:50:05,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:05,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2817
2799
2824
[WARNING|trainer.py:803] 2025-04-26 15:50:07,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:50:07,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:07,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2800
2825
2818
[WARNING|trainer.py:803] 2025-04-26 15:50:10,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:10,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:10,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2801
2826
2819
[WARNING|trainer.py:803] 2025-04-26 15:50:13,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:13,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:13,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2827
2802
2820
[WARNING|trainer.py:803] 2025-04-26 15:50:15,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:15,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:15,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2828
2803
2821
[WARNING|trainer.py:803] 2025-04-26 15:50:18,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:50:18,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:18,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2829
2804
2822
[WARNING|trainer.py:803] 2025-04-26 15:50:20,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:21,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:21,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2830
2805
2823
[WARNING|trainer.py:803] 2025-04-26 15:50:23,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:23,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:23,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2831
2806
2824
[WARNING|trainer.py:803] 2025-04-26 15:50:26,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:26,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:26,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2807
2832
2825
[WARNING|trainer.py:803] 2025-04-26 15:50:28,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:28,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:29,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2808
2833
2826
[WARNING|trainer.py:803] 2025-04-26 15:50:31,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:31,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:31,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2809
2834
2827
[WARNING|trainer.py:803] 2025-04-26 15:50:33,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:34,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:34,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2810
2835
2828
[WARNING|trainer.py:803] 2025-04-26 15:50:36,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:36,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:50:36,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2811
2836
2829
[WARNING|trainer.py:803] 2025-04-26 15:50:39,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:39,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:50:39,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2812
2837
2830
[WARNING|trainer.py:803] 2025-04-26 15:50:41,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:41,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:50:41,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2813
2838
2831
[WARNING|trainer.py:803] 2025-04-26 15:50:44,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:44,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:44,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2814
2839
2832
[WARNING|trainer.py:803] 2025-04-26 15:50:46,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:50:46,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:50:47,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2815
2840
2833
[WARNING|trainer.py:803] 2025-04-26 15:50:49,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:49,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:49,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2816
2841
2834
[WARNING|trainer.py:803] 2025-04-26 15:50:52,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:50:52,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:52,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2817
2842
2835
[WARNING|trainer.py:803] 2025-04-26 15:50:54,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:54,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:50:54,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2818
2843
2836
[WARNING|trainer.py:803] 2025-04-26 15:50:57,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:50:57,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:50:57,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2837
2844
2819
[WARNING|trainer.py:803] 2025-04-26 15:50:59,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:00,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:00,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2838
2845
2820
[WARNING|trainer.py:803] 2025-04-26 15:51:02,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:02,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:02,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2839
2846
2821
[WARNING|trainer.py:803] 2025-04-26 15:51:05,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:51:05,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:05,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2840
2847
2822
[WARNING|trainer.py:803] 2025-04-26 15:51:07,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:07,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:08,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2841
2848
2823
[WARNING|trainer.py:803] 2025-04-26 15:51:10,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:10,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:10,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2842
2849
2824
[WARNING|trainer.py:803] 2025-04-26 15:51:13,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:13,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:13,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2850
2843
2825
[WARNING|trainer.py:803] 2025-04-26 15:51:15,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:51:15,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:16,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2844
2851
2826
[WARNING|trainer.py:803] 2025-04-26 15:51:18,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:18,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:18,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2845
2852
2827
[WARNING|trainer.py:803] 2025-04-26 15:51:21,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:21,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:21,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2846
2828
2853
[WARNING|trainer.py:803] 2025-04-26 15:51:23,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:23,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:51:23,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2847
2854
2829
[WARNING|trainer.py:803] 2025-04-26 15:51:26,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:26,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:26,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2848
2855
2830
[WARNING|trainer.py:803] 2025-04-26 15:51:29,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:29,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:29,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2849
2831
2856
[WARNING|trainer.py:803] 2025-04-26 15:51:31,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:31,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:31,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2857
2832
2850
[WARNING|trainer.py:803] 2025-04-26 15:51:34,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:34,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:34,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2858
2833
2851
[WARNING|trainer.py:803] 2025-04-26 15:51:37,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:37,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:37,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2834
2859
2852
[WARNING|trainer.py:803] 2025-04-26 15:51:39,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:39,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:51:39,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2835
2860
2853
[WARNING|trainer.py:803] 2025-04-26 15:51:42,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:42,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:42,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2836
2861
2854
[WARNING|trainer.py:803] 2025-04-26 15:51:44,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:45,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:45,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2837
2862
2855
[WARNING|trainer.py:803] 2025-04-26 15:51:47,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:51:47,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:47,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2838
2863
2856
[WARNING|trainer.py:803] 2025-04-26 15:51:50,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:51:50,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:50,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2839
2857
2864
[WARNING|trainer.py:803] 2025-04-26 15:51:52,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:51:52,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:52,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2840
2865
2858
[WARNING|trainer.py:803] 2025-04-26 15:51:55,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:55,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:55,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2841
2866
2859
[WARNING|trainer.py:803] 2025-04-26 15:51:58,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:58,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:51:58,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2842
2867
2860
[WARNING|trainer.py:803] 2025-04-26 15:52:00,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:00,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:01,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2843
2861
2868
[WARNING|trainer.py:803] 2025-04-26 15:52:03,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:03,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:03,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2862
2844
2869
[WARNING|trainer.py:803] 2025-04-26 15:52:06,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:06,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:06,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2870
2863
2845
[WARNING|trainer.py:803] 2025-04-26 15:52:08,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:08,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:08,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2871
2846
2864
[WARNING|trainer.py:803] 2025-04-26 15:52:11,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:11,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:11,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2872
2865
2847
[WARNING|trainer.py:803] 2025-04-26 15:52:14,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:14,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:14,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2866
2848
2873
[WARNING|trainer.py:803] 2025-04-26 15:52:16,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:16,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:16,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2849
2874
2867
[WARNING|trainer.py:803] 2025-04-26 15:52:19,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:19,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:19,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2875
2850
2868
[WARNING|trainer.py:803] 2025-04-26 15:52:22,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:22,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:52:22,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2876
2851
2869
[WARNING|trainer.py:803] 2025-04-26 15:52:24,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:24,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:25,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2877
2852
2870
[WARNING|trainer.py:803] 2025-04-26 15:52:27,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:27,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:27,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2878
2853
2871
[WARNING|trainer.py:803] 2025-04-26 15:52:29,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:30,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:30,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2879
2854
2872
[WARNING|trainer.py:803] 2025-04-26 15:52:32,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:32,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:32,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2880
2873
2855
[WARNING|trainer.py:803] 2025-04-26 15:52:35,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:35,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:35,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2881
2856
2874
[WARNING|trainer.py:803] 2025-04-26 15:52:37,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:38,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 15:52:38,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2882
2857
2875
[WARNING|trainer.py:803] 2025-04-26 15:52:40,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:40,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:40,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2883
2858
2876
[WARNING|trainer.py:803] 2025-04-26 15:52:43,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:43,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:43,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2884
2877
2859
[WARNING|trainer.py:803] 2025-04-26 15:52:45,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:45,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:45,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2878
2885
2860
[WARNING|trainer.py:803] 2025-04-26 15:52:48,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:48,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:52:48,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2879
2861
2886
[WARNING|trainer.py:803] 2025-04-26 15:52:51,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:52:51,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:51,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2880
2862
2887
[WARNING|trainer.py:803] 2025-04-26 15:52:53,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:53,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:53,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2881
2863
2888
[WARNING|trainer.py:803] 2025-04-26 15:52:56,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:56,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:56,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2882
2864
2889
[WARNING|trainer.py:803] 2025-04-26 15:52:59,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:52:59,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:52:59,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2890
2883
2865
[WARNING|trainer.py:803] 2025-04-26 15:53:01,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:01,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:01,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2884
2866
2891
[WARNING|trainer.py:803] 2025-04-26 15:53:04,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:04,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:04,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2867
2885
2892
[WARNING|trainer.py:803] 2025-04-26 15:53:07,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:53:07,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:07,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2893
2886
2868
[WARNING|trainer.py:803] 2025-04-26 15:53:09,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:09,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:09,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2894
2887
2869
[WARNING|trainer.py:803] 2025-04-26 15:53:12,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:12,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:12,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2870
2895
2888
[WARNING|trainer.py:803] 2025-04-26 15:53:15,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:15,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:15,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2896
2871
2889
[WARNING|trainer.py:803] 2025-04-26 15:53:17,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:17,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:17,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2897
2872
2890
[WARNING|trainer.py:803] 2025-04-26 15:53:20,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:20,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:20,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2898
2891
2873
[WARNING|trainer.py:803] 2025-04-26 15:53:23,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:23,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:23,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2899
2892
2874
[WARNING|trainer.py:803] 2025-04-26 15:53:25,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:25,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:25,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2900
2893
2875
[WARNING|trainer.py:803] 2025-04-26 15:53:28,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:28,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:28,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2894
2901
2876
[WARNING|trainer.py:803] 2025-04-26 15:53:31,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:31,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:31,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2895
2877
2902
[WARNING|trainer.py:803] 2025-04-26 15:53:33,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:33,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:33,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2903
2878
2896
[WARNING|trainer.py:803] 2025-04-26 15:53:36,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:36,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:36,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2904
2879
2897
[WARNING|trainer.py:803] 2025-04-26 15:53:38,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 15:53:39,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:39,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2905
2898
2880
[WARNING|trainer.py:803] 2025-04-26 15:53:41,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:41,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:41,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2906
2899
2881
[WARNING|trainer.py:803] 2025-04-26 15:53:44,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:44,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:44,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2907
2882
2900
[WARNING|trainer.py:803] 2025-04-26 15:53:46,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:46,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:46,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2908
2883
2901
[WARNING|trainer.py:803] 2025-04-26 15:53:49,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:53:49,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:49,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2909
2884
2902
[WARNING|trainer.py:803] 2025-04-26 15:53:52,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:53:52,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:52,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2910
2903
2885
[WARNING|trainer.py:803] 2025-04-26 15:53:54,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:54,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:54,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2911
2904
2886
[WARNING|trainer.py:803] 2025-04-26 15:53:57,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:57,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 15:53:57,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2912
2905
2887
[WARNING|trainer.py:803] 2025-04-26 15:53:59,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:53:59,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:00,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2913
2906
2888
[WARNING|trainer.py:803] 2025-04-26 15:54:02,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:02,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:02,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2914
2907
2889
[WARNING|trainer.py:803] 2025-04-26 15:54:05,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:05,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:05,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2915
2908
2890
[WARNING|trainer.py:803] 2025-04-26 15:54:07,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:07,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:08,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2916
2909
2891
[WARNING|trainer.py:803] 2025-04-26 15:54:10,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:54:10,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:10,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2917
2910
2892
[WARNING|trainer.py:803] 2025-04-26 15:54:12,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:13,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:13,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2918
2911
2893
[WARNING|trainer.py:803] 2025-04-26 15:54:15,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:15,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:15,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2919
2912
2894
[WARNING|trainer.py:803] 2025-04-26 15:54:17,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:54:18,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:18,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2920
2913
2895
[WARNING|trainer.py:803] 2025-04-26 15:54:20,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:20,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:21,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2921
2914
2896
[WARNING|trainer.py:803] 2025-04-26 15:54:23,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:23,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:23,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2922
2915
2897
[WARNING|trainer.py:803] 2025-04-26 15:54:25,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:26,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:26,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2923
2916
2898
[WARNING|trainer.py:803] 2025-04-26 15:54:28,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:28,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:54:28,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2924
2917
2899
[WARNING|trainer.py:803] 2025-04-26 15:54:30,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:31,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:31,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2925
2918
2900
[WARNING|trainer.py:803] 2025-04-26 15:54:33,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:33,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:34,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2926
2919
2901
[WARNING|trainer.py:803] 2025-04-26 15:54:36,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:36,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:54:36,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2927
2920
2902
[WARNING|trainer.py:803] 2025-04-26 15:54:38,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:39,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:39,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2928
2921
2903
[WARNING|trainer.py:803] 2025-04-26 15:54:41,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:41,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:42,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2929
2922
2904
[WARNING|trainer.py:803] 2025-04-26 15:54:43,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:54:44,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:44,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2930
2923
2905
[WARNING|trainer.py:803] 2025-04-26 15:54:46,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:54:46,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:47,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2931
2924
2906
[WARNING|trainer.py:803] 2025-04-26 15:54:49,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:49,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:49,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2932
2925
2907
[WARNING|trainer.py:803] 2025-04-26 15:54:51,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:52,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:52,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2933
2926
2908
[WARNING|trainer.py:803] 2025-04-26 15:54:54,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:54,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:55,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2934
2927
[WARNING|trainer.py:803] 2025-04-26 15:54:56,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2909
[WARNING|trainer.py:803] 2025-04-26 15:54:57,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:54:57,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2935
2928
2910
[WARNING|trainer.py:803] 2025-04-26 15:54:59,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:54:59,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:00,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2936
2929
[WARNING|trainer.py:803] 2025-04-26 15:55:02,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2911
[WARNING|trainer.py:803] 2025-04-26 15:55:02,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:55:03,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2937
2930
2912
[WARNING|trainer.py:803] 2025-04-26 15:55:04,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:05,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:55:05,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2938
2931
2913
[WARNING|trainer.py:803] 2025-04-26 15:55:07,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:07,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:08,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2939
2932
2914
[WARNING|trainer.py:803] 2025-04-26 15:55:10,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:10,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:10,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2940
2933
2915
[WARNING|trainer.py:803] 2025-04-26 15:55:12,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:12,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:13,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2941
2934
2916
[WARNING|trainer.py:803] 2025-04-26 15:55:15,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:15,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:15,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2942
2935
2917
[WARNING|trainer.py:803] 2025-04-26 15:55:17,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:18,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:18,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2943
2936
2918
[WARNING|trainer.py:803] 2025-04-26 15:55:20,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:20,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:55:20,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2944
2937
2919
[WARNING|trainer.py:803] 2025-04-26 15:55:23,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:23,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:23,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2945
2938
2920
[WARNING|trainer.py:803] 2025-04-26 15:55:25,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:26,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:26,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2946
2939
2921
[WARNING|trainer.py:803] 2025-04-26 15:55:28,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:28,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:28,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2947
2940
2922
[WARNING|trainer.py:803] 2025-04-26 15:55:30,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:55:31,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:31,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2948
2941
2923
[WARNING|trainer.py:803] 2025-04-26 15:55:33,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:55:33,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:34,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2949
2942
2924
[WARNING|trainer.py:803] 2025-04-26 15:55:36,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:36,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:36,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2950
2943
2925
[WARNING|trainer.py:803] 2025-04-26 15:55:38,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:39,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:39,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2944
2951
2926
[WARNING|trainer.py:803] 2025-04-26 15:55:41,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:41,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:41,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2952
2927
2945
[WARNING|trainer.py:803] 2025-04-26 15:55:44,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:55:44,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:44,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2953
2946
2928
[WARNING|trainer.py:803] 2025-04-26 15:55:46,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:46,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:46,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2954
2947
2929
[WARNING|trainer.py:803] 2025-04-26 15:55:49,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:49,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:55:49,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2955
2930
2948
[WARNING|trainer.py:803] 2025-04-26 15:55:51,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:55:52,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:55:52,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2956
2931
2949
[WARNING|trainer.py:803] 2025-04-26 15:55:54,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:54,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:54,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2957
2932
2950
[WARNING|trainer.py:803] 2025-04-26 15:55:57,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:57,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:57,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2958
2933
2951
[WARNING|trainer.py:803] 2025-04-26 15:55:59,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:55:59,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:00,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2959
2934
2952
[WARNING|trainer.py:803] 2025-04-26 15:56:02,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:02,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:02,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2960
2953
2935
[WARNING|trainer.py:803] 2025-04-26 15:56:04,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:56:05,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:05,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2954
2961
2936
[WARNING|trainer.py:803] 2025-04-26 15:56:07,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:07,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:07,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2955
2962
2937
[WARNING|trainer.py:803] 2025-04-26 15:56:10,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:10,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:10,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2956
2963
2938
[WARNING|trainer.py:803] 2025-04-26 15:56:12,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:12,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:56:13,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2957
2964
2939
[WARNING|trainer.py:803] 2025-04-26 15:56:15,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:15,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:15,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2958
2965
2940
[WARNING|trainer.py:803] 2025-04-26 15:56:18,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:18,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:18,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2966
2959
2941
[WARNING|trainer.py:803] 2025-04-26 15:56:20,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:56:20,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:20,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2960
2967
2942
[WARNING|trainer.py:803] 2025-04-26 15:56:23,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:56:23,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:23,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2968
2943
2961
[WARNING|trainer.py:803] 2025-04-26 15:56:26,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:26,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:26,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2944
2969
2962
[WARNING|trainer.py:803] 2025-04-26 15:56:28,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:28,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:28,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2945
2970
2963
[WARNING|trainer.py:803] 2025-04-26 15:56:31,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:31,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:31,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2946
2971
2964
[WARNING|trainer.py:803] 2025-04-26 15:56:34,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:34,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:34,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2947
2972
2965
[WARNING|trainer.py:803] 2025-04-26 15:56:36,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:56:36,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:56:36,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2966
2973
2948
[WARNING|trainer.py:803] 2025-04-26 15:56:39,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:56:39,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:56:39,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2967
2974
2949
[WARNING|trainer.py:803] 2025-04-26 15:56:41,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:42,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:42,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2968
2975
2950
[WARNING|trainer.py:803] 2025-04-26 15:56:44,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:44,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:44,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2969
2976
2951
[WARNING|trainer.py:803] 2025-04-26 15:56:47,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:47,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:56:47,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2970
2977
2952
[WARNING|trainer.py:803] 2025-04-26 15:56:49,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:49,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:49,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2953
2971
2978
[WARNING|trainer.py:803] 2025-04-26 15:56:52,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:52,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:56:52,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2954
2979
2972
[WARNING|trainer.py:803] 2025-04-26 15:56:55,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:55,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:56:55,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2955
2980
2973
[WARNING|trainer.py:803] 2025-04-26 15:56:57,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:56:57,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:56:57,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2956
2981
2974
[WARNING|trainer.py:803] 2025-04-26 15:57:00,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:00,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:00,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2957
2982
2975
[WARNING|trainer.py:803] 2025-04-26 15:57:02,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:03,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:03,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2983
2958
2976
[WARNING|trainer.py:803] 2025-04-26 15:57:05,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:05,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:05,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2984
2959
2977
[WARNING|trainer.py:803] 2025-04-26 15:57:08,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:08,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:08,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2985
2960
2978
[WARNING|trainer.py:803] 2025-04-26 15:57:10,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:10,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:57:10,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2979
2986
2961
[WARNING|trainer.py:803] 2025-04-26 15:57:13,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:13,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:13,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2980
2962
2987
[WARNING|trainer.py:803] 2025-04-26 15:57:16,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 15:57:16,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:16,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2981
2988
2963
[WARNING|trainer.py:803] 2025-04-26 15:57:18,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:18,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:18,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2989
2982
2964
[WARNING|trainer.py:803] 2025-04-26 15:57:21,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:57:21,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:21,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2990
2983
2965
[WARNING|trainer.py:803] 2025-04-26 15:57:23,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:23,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:24,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2991
2984
2966
[WARNING|trainer.py:803] 2025-04-26 15:57:26,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:26,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:26,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2985
2992
2967
[WARNING|trainer.py:803] 2025-04-26 15:57:28,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:29,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:57:29,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2993
2986
2968
[WARNING|trainer.py:803] 2025-04-26 15:57:31,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:31,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:31,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2994
2987
2969
[WARNING|trainer.py:803] 2025-04-26 15:57:34,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:34,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:34,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2995
2970
2988
[WARNING|trainer.py:803] 2025-04-26 15:57:36,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:37,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:37,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2989
2996
2971
[WARNING|trainer.py:803] 2025-04-26 15:57:39,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:57:39,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:39,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2997
2990
2972
[WARNING|trainer.py:803] 2025-04-26 15:57:42,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:42,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:42,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2998
2991
2973
[WARNING|trainer.py:803] 2025-04-26 15:57:44,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:44,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:44,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2992
2999
2974
[WARNING|trainer.py:803] 2025-04-26 15:57:47,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:47,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:47,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2993
3000
2975
[WARNING|trainer.py:803] 2025-04-26 15:57:50,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:50,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:50,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3001
2994
2976
[WARNING|trainer.py:803] 2025-04-26 15:57:52,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:57:52,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:52,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3002
[WARNING|trainer.py:803] 2025-04-26 15:57:54,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2995
2977
[WARNING|trainer.py:803] 2025-04-26 15:57:55,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:57:55,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3003
[WARNING|trainer.py:803] 2025-04-26 15:57:56,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2996
2978
[WARNING|trainer.py:803] 2025-04-26 15:57:57,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3004
[WARNING|trainer.py:803] 2025-04-26 15:57:58,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:57:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2997
2979
3005
[WARNING|trainer.py:803] 2025-04-26 15:58:00,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:00,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:58:00,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3006
2998
2980
[WARNING|trainer.py:803] 2025-04-26 15:58:03,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:03,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:03,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3007
2999
[WARNING|trainer.py:803] 2025-04-26 15:58:05,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2981
[WARNING|trainer.py:803] 2025-04-26 15:58:05,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:06,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3008
[WARNING|trainer.py:803] 2025-04-26 15:58:07,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3000
2982
[WARNING|trainer.py:803] 2025-04-26 15:58:08,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:08,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3009
[WARNING|trainer.py:803] 2025-04-26 15:58:09,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3001
2983
[WARNING|trainer.py:803] 2025-04-26 15:58:10,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3010
[WARNING|trainer.py:803] 2025-04-26 15:58:11,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:58:11,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3002
[WARNING|trainer.py:803] 2025-04-26 15:58:12,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3011
2984
[WARNING|trainer.py:803] 2025-04-26 15:58:13,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:13,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3003
3012
[WARNING|trainer.py:803] 2025-04-26 15:58:15,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2985
[WARNING|trainer.py:803] 2025-04-26 15:58:15,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:16,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3004
3013
[WARNING|trainer.py:803] 2025-04-26 15:58:17,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:17,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2986
3005
[WARNING|trainer.py:803] 2025-04-26 15:58:18,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3014
[WARNING|trainer.py:803] 2025-04-26 15:58:19,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:20,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3006
2987
3015
[WARNING|trainer.py:803] 2025-04-26 15:58:21,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:21,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:22,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3007
2988
3016
[WARNING|trainer.py:803] 2025-04-26 15:58:23,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:24,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:24,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3008
3017
[WARNING|trainer.py:803] 2025-04-26 15:58:25,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2989
[WARNING|trainer.py:803] 2025-04-26 15:58:26,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:26,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3009
3018
[WARNING|trainer.py:803] 2025-04-26 15:58:27,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2990
[WARNING|trainer.py:803] 2025-04-26 15:58:28,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3010
[WARNING|trainer.py:803] 2025-04-26 15:58:29,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3019
[WARNING|trainer.py:803] 2025-04-26 15:58:30,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:30,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2991
3011
[WARNING|trainer.py:803] 2025-04-26 15:58:31,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:32,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3020
[WARNING|trainer.py:803] 2025-04-26 15:58:32,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3012
2992
[WARNING|trainer.py:803] 2025-04-26 15:58:34,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3021
[WARNING|trainer.py:803] 2025-04-26 15:58:34,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:35,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3013
[WARNING|trainer.py:803] 2025-04-26 15:58:36,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2993
3022
[WARNING|trainer.py:803] 2025-04-26 15:58:36,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:37,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3014
[WARNING|trainer.py:803] 2025-04-26 15:58:38,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3023
2994
[WARNING|trainer.py:803] 2025-04-26 15:58:39,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3015
[WARNING|trainer.py:803] 2025-04-26 15:58:39,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:40,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3024
2995
[WARNING|trainer.py:803] 2025-04-26 15:58:41,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3016
[WARNING|trainer.py:803] 2025-04-26 15:58:42,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:42,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3025
[WARNING|trainer.py:803] 2025-04-26 15:58:43,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3017
2996
[WARNING|trainer.py:803] 2025-04-26 15:58:44,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3026
[WARNING|trainer.py:803] 2025-04-26 15:58:44,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:45,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3018
2997
[WARNING|trainer.py:803] 2025-04-26 15:58:46,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3027
[WARNING|trainer.py:803] 2025-04-26 15:58:47,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:47,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3019
[WARNING|trainer.py:803] 2025-04-26 15:58:48,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3028
2998
[WARNING|trainer.py:803] 2025-04-26 15:58:50,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:50,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3020
[WARNING|trainer.py:803] 2025-04-26 15:58:51,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3029
2999
[WARNING|trainer.py:803] 2025-04-26 15:58:52,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3021
[WARNING|trainer.py:803] 2025-04-26 15:58:52,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:53,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3030
[WARNING|trainer.py:803] 2025-04-26 15:58:54,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3022
3000
[WARNING|trainer.py:803] 2025-04-26 15:58:55,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:58:55,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3031
[WARNING|trainer.py:803] 2025-04-26 15:58:56,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3023
3001
[WARNING|trainer.py:803] 2025-04-26 15:58:57,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:58:57,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3032
[WARNING|trainer.py:803] 2025-04-26 15:58:58,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3024
3002
[WARNING|trainer.py:803] 2025-04-26 15:58:59,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:58:59,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3033
[WARNING|trainer.py:803] 2025-04-26 15:59:00,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3025
3003
[WARNING|trainer.py:803] 2025-04-26 15:59:01,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:01,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3034
3026
[WARNING|trainer.py:803] 2025-04-26 15:59:03,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3004
[WARNING|trainer.py:803] 2025-04-26 15:59:03,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:04,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3035
3027
[WARNING|trainer.py:803] 2025-04-26 15:59:05,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3005
[WARNING|trainer.py:803] 2025-04-26 15:59:05,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:06,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3036
[WARNING|trainer.py:803] 2025-04-26 15:59:07,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3028
3006
[WARNING|trainer.py:803] 2025-04-26 15:59:08,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:08,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3037
[WARNING|trainer.py:803] 2025-04-26 15:59:09,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3029
3007
[WARNING|trainer.py:803] 2025-04-26 15:59:10,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:10,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3038
[WARNING|trainer.py:803] 2025-04-26 15:59:11,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3030
3008
[WARNING|trainer.py:803] 2025-04-26 15:59:12,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:12,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3039
[WARNING|trainer.py:803] 2025-04-26 15:59:13,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3031
3009
[WARNING|trainer.py:803] 2025-04-26 15:59:14,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:14,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3040
[WARNING|trainer.py:803] 2025-04-26 15:59:15,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3032
3010
[WARNING|trainer.py:803] 2025-04-26 15:59:16,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:17,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3041
[WARNING|trainer.py:803] 2025-04-26 15:59:17,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3011
3033
[WARNING|trainer.py:803] 2025-04-26 15:59:19,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:19,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3042
[WARNING|trainer.py:803] 2025-04-26 15:59:20,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3012
3034
[WARNING|trainer.py:803] 2025-04-26 15:59:21,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:59:21,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3043
[WARNING|trainer.py:803] 2025-04-26 15:59:22,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3013
3035
[WARNING|trainer.py:803] 2025-04-26 15:59:23,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:23,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3044
[WARNING|trainer.py:803] 2025-04-26 15:59:24,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3014
3036
[WARNING|trainer.py:803] 2025-04-26 15:59:25,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:25,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3045
[WARNING|trainer.py:803] 2025-04-26 15:59:26,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3037
3015
[WARNING|trainer.py:803] 2025-04-26 15:59:27,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:27,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3046
[WARNING|trainer.py:803] 2025-04-26 15:59:28,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3038
3016
[WARNING|trainer.py:803] 2025-04-26 15:59:29,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:29,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3047
[WARNING|trainer.py:803] 2025-04-26 15:59:30,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3039
3017
[WARNING|trainer.py:803] 2025-04-26 15:59:31,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:32,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3048
[WARNING|trainer.py:803] 2025-04-26 15:59:33,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3040
3018
[WARNING|trainer.py:803] 2025-04-26 15:59:34,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:59:34,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3049
[WARNING|trainer.py:803] 2025-04-26 15:59:35,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3041
3019
[WARNING|trainer.py:803] 2025-04-26 15:59:36,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:36,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3050
[WARNING|trainer.py:803] 2025-04-26 15:59:37,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3020
3042
[WARNING|trainer.py:803] 2025-04-26 15:59:38,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:38,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3051
[WARNING|trainer.py:803] 2025-04-26 15:59:39,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3021
3043
[WARNING|trainer.py:803] 2025-04-26 15:59:40,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:40,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3052
[WARNING|trainer.py:803] 2025-04-26 15:59:41,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3022
3044
[WARNING|trainer.py:803] 2025-04-26 15:59:42,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:42,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3053
[WARNING|trainer.py:803] 2025-04-26 15:59:43,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3023
3045
[WARNING|trainer.py:803] 2025-04-26 15:59:44,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 15:59:44,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3054
[WARNING|trainer.py:803] 2025-04-26 15:59:45,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3024
3046
[WARNING|trainer.py:803] 2025-04-26 15:59:47,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:47,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3055
[WARNING|trainer.py:803] 2025-04-26 15:59:47,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3025
3047
[WARNING|trainer.py:803] 2025-04-26 15:59:49,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:49,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3056
[WARNING|trainer.py:803] 2025-04-26 15:59:50,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3026
3048
[WARNING|trainer.py:803] 2025-04-26 15:59:51,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:51,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3057
[WARNING|trainer.py:803] 2025-04-26 15:59:52,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3027
3049
[WARNING|trainer.py:803] 2025-04-26 15:59:53,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:53,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3058
[WARNING|trainer.py:803] 2025-04-26 15:59:54,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3050
3028
3059
[WARNING|trainer.py:803] 2025-04-26 15:59:55,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:55,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 15:59:56,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3051
3029
3060
[WARNING|trainer.py:803] 2025-04-26 15:59:57,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 15:59:57,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 15:59:58,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3052
3030
3061
[WARNING|trainer.py:803] 2025-04-26 15:59:59,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:00:00,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:00,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3031
3053
3062
[WARNING|trainer.py:803] 2025-04-26 16:00:02,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:02,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:02,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3032
3054
3063
[WARNING|trainer.py:803] 2025-04-26 16:00:04,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:04,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:04,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3055
3033
3064
[WARNING|trainer.py:803] 2025-04-26 16:00:06,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:06,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:06,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3056
3034
3065
[WARNING|trainer.py:803] 2025-04-26 16:00:08,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:08,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:08,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3057
3035
3066
[WARNING|trainer.py:803] 2025-04-26 16:00:10,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:10,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:11,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3058
3036
3067
[WARNING|trainer.py:803] 2025-04-26 16:00:12,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:13,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:13,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3059
3037
3068
[WARNING|trainer.py:803] 2025-04-26 16:00:15,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:15,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:15,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3060
3069
3038
[WARNING|trainer.py:803] 2025-04-26 16:00:17,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:17,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:17,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3061
3070
3039
[WARNING|trainer.py:803] 2025-04-26 16:00:19,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:19,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:19,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3062
3071
3040
[WARNING|trainer.py:803] 2025-04-26 16:00:21,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:21,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:21,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3063
3072
3041
[WARNING|trainer.py:803] 2025-04-26 16:00:23,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:23,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:23,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3064
3073
3042
[WARNING|trainer.py:803] 2025-04-26 16:00:25,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:00:25,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:00:26,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3065
3074
3043
[WARNING|trainer.py:803] 2025-04-26 16:00:27,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:28,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:28,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3066
3075
[WARNING|trainer.py:803] 2025-04-26 16:00:29,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3044
[WARNING|trainer.py:803] 2025-04-26 16:00:30,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:30,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3067
3076
3045
[WARNING|trainer.py:803] 2025-04-26 16:00:31,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:32,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:32,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3068
3077
3046
[WARNING|trainer.py:803] 2025-04-26 16:00:34,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:34,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:34,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3069
3078
3047
[WARNING|trainer.py:803] 2025-04-26 16:00:36,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:36,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:36,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3070
3079
3048
[WARNING|trainer.py:803] 2025-04-26 16:00:38,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:38,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:39,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3071
3080
3049
[WARNING|trainer.py:803] 2025-04-26 16:00:40,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:41,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:41,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3072
3081
3050
[WARNING|trainer.py:803] 2025-04-26 16:00:42,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:43,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:43,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3073
3082
3051
[WARNING|trainer.py:803] 2025-04-26 16:00:44,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:00:45,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:45,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3074
3083
3052
[WARNING|trainer.py:803] 2025-04-26 16:00:46,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:47,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:47,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3075
3084
3053
[WARNING|trainer.py:803] 2025-04-26 16:00:49,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:49,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:49,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3076
3085
3054
[WARNING|trainer.py:803] 2025-04-26 16:00:51,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:51,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:51,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3077
3055
3086
[WARNING|trainer.py:803] 2025-04-26 16:00:53,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:54,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:54,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3078
3087
3056
[WARNING|trainer.py:803] 2025-04-26 16:00:55,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:56,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:00:56,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3079
3088
3057
[WARNING|trainer.py:803] 2025-04-26 16:00:57,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:00:58,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:00:58,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3080
3089
3058
[WARNING|trainer.py:803] 2025-04-26 16:00:59,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:00,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:01:00,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3081
3090
3059
[WARNING|trainer.py:803] 2025-04-26 16:01:01,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:02,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:02,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3082
3091
3060
[WARNING|trainer.py:803] 2025-04-26 16:01:04,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:04,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:01:04,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3083
3092
3061
[WARNING|trainer.py:803] 2025-04-26 16:01:06,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:06,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:06,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3084
3093
3062
[WARNING|trainer.py:803] 2025-04-26 16:01:08,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:08,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:08,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3085
3094
3063
[WARNING|trainer.py:803] 2025-04-26 16:01:10,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:10,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:11,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3086
3095
3064
[WARNING|trainer.py:803] 2025-04-26 16:01:12,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:12,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:13,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3087
3096
3065
[WARNING|trainer.py:803] 2025-04-26 16:01:14,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:15,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:15,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3088
3097
3066
[WARNING|trainer.py:803] 2025-04-26 16:01:16,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:17,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:17,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3089
3098
3067
[WARNING|trainer.py:803] 2025-04-26 16:01:19,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:01:19,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:19,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3090
3099
3068
[WARNING|trainer.py:803] 2025-04-26 16:01:21,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:21,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:21,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3091
3100
3069
[WARNING|trainer.py:803] 2025-04-26 16:01:23,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:01:23,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:23,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3092
3070
[WARNING|trainer.py:803] 2025-04-26 16:01:25,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:26,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3101
3093
[WARNING|trainer.py:803] 2025-04-26 16:01:26,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3071
[WARNING|trainer.py:803] 2025-04-26 16:01:27,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:28,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3094
3102
3072
[WARNING|trainer.py:803] 2025-04-26 16:01:29,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:29,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:30,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3095
3103
3073
[WARNING|trainer.py:803] 2025-04-26 16:01:31,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:32,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:01:32,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3096
3074
[WARNING|trainer.py:803] 2025-04-26 16:01:33,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3104
[WARNING|trainer.py:803] 2025-04-26 16:01:34,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:34,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3097
3075
[WARNING|trainer.py:803] 2025-04-26 16:01:36,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3105
[WARNING|trainer.py:803] 2025-04-26 16:01:36,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:37,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3098
3076
[WARNING|trainer.py:803] 2025-04-26 16:01:38,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:38,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3106
3099
3077
[WARNING|trainer.py:803] 2025-04-26 16:01:40,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:40,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3107
3100
3078
[WARNING|trainer.py:803] 2025-04-26 16:01:42,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:42,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:43,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3079
3108
3101
[WARNING|trainer.py:803] 2025-04-26 16:01:45,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:01:45,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:45,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3080
3109
[WARNING|trainer.py:803] 2025-04-26 16:01:47,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3102
[WARNING|trainer.py:803] 2025-04-26 16:01:47,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:48,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3081
3110
[WARNING|trainer.py:803] 2025-04-26 16:01:49,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:50,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3103
3082
[WARNING|trainer.py:803] 2025-04-26 16:01:51,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:01:51,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3111
[WARNING|trainer.py:803] 2025-04-26 16:01:52,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3083
3104
[WARNING|trainer.py:803] 2025-04-26 16:01:53,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:01:53,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3112
[WARNING|trainer.py:803] 2025-04-26 16:01:54,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3084
3105
[WARNING|trainer.py:803] 2025-04-26 16:01:55,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:56,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3113
3085
[WARNING|trainer.py:803] 2025-04-26 16:01:57,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3106
[WARNING|trainer.py:803] 2025-04-26 16:01:58,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:01:58,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3114
3086
[WARNING|trainer.py:803] 2025-04-26 16:01:59,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:00,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3107
[WARNING|trainer.py:803] 2025-04-26 16:02:01,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3087
3115
[WARNING|trainer.py:803] 2025-04-26 16:02:02,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:02,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3108
3088
[WARNING|trainer.py:803] 2025-04-26 16:02:04,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:04,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3116
[WARNING|trainer.py:803] 2025-04-26 16:02:05,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3109
3089
[WARNING|trainer.py:803] 2025-04-26 16:02:06,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:06,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3117
3090
[WARNING|trainer.py:803] 2025-04-26 16:02:07,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3110
[WARNING|trainer.py:803] 2025-04-26 16:02:08,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:08,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3091
3118
3111
[WARNING|trainer.py:803] 2025-04-26 16:02:10,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:02:10,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:11,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3092
3119
3112
[WARNING|trainer.py:803] 2025-04-26 16:02:12,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:13,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:13,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3093
[WARNING|trainer.py:803] 2025-04-26 16:02:14,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3113
3120
[WARNING|trainer.py:803] 2025-04-26 16:02:15,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3094
[WARNING|trainer.py:803] 2025-04-26 16:02:16,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:02:17,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3114
3121
3095
[WARNING|trainer.py:803] 2025-04-26 16:02:18,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:19,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:02:19,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3115
3096
[WARNING|trainer.py:803] 2025-04-26 16:02:21,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:21,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3122
[WARNING|trainer.py:803] 2025-04-26 16:02:22,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3097
3116
[WARNING|trainer.py:803] 2025-04-26 16:02:23,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3123
[WARNING|trainer.py:803] 2025-04-26 16:02:23,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:24,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3098
3117
[WARNING|trainer.py:803] 2025-04-26 16:02:25,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:26,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3124
3099
[WARNING|trainer.py:803] 2025-04-26 16:02:27,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:02:27,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3118
3100
[WARNING|trainer.py:803] 2025-04-26 16:02:29,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3125
[WARNING|trainer.py:803] 2025-04-26 16:02:30,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:02:30,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3119
[WARNING|trainer.py:803] 2025-04-26 16:02:32,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3126
3101
[WARNING|trainer.py:803] 2025-04-26 16:02:33,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:33,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3120
[WARNING|trainer.py:803] 2025-04-26 16:02:34,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3102
3127
[WARNING|trainer.py:803] 2025-04-26 16:02:35,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:36,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3121
[WARNING|trainer.py:803] 2025-04-26 16:02:37,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3103
3128
[WARNING|trainer.py:803] 2025-04-26 16:02:38,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:38,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3122
3104
3129
[WARNING|trainer.py:803] 2025-04-26 16:02:41,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:41,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:41,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3123
3105
3130
[WARNING|trainer.py:803] 2025-04-26 16:02:43,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:02:43,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:43,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3131
3124
3106
[WARNING|trainer.py:803] 2025-04-26 16:02:46,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:46,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:02:46,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3132
3107
3125
[WARNING|trainer.py:803] 2025-04-26 16:02:48,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:48,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:49,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3133
3108
[WARNING|trainer.py:803] 2025-04-26 16:02:50,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3126
[WARNING|trainer.py:803] 2025-04-26 16:02:51,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:02:51,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3134
3109
[WARNING|trainer.py:803] 2025-04-26 16:02:53,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:53,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3127
[WARNING|trainer.py:803] 2025-04-26 16:02:54,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3135
3110
[WARNING|trainer.py:803] 2025-04-26 16:02:56,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3128
[WARNING|trainer.py:803] 2025-04-26 16:02:56,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:02:57,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3136
3111
[WARNING|trainer.py:803] 2025-04-26 16:02:58,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:02:58,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3129
[WARNING|trainer.py:803] 2025-04-26 16:02:59,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3137
3112
[WARNING|trainer.py:803] 2025-04-26 16:03:01,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:01,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3130
[WARNING|trainer.py:803] 2025-04-26 16:03:02,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3113
3138
[WARNING|trainer.py:803] 2025-04-26 16:03:03,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:03,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3131
[WARNING|trainer.py:803] 2025-04-26 16:03:04,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3114
3139
[WARNING|trainer.py:803] 2025-04-26 16:03:06,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:06,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3132
[WARNING|trainer.py:803] 2025-04-26 16:03:07,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3115
3140
3133
[WARNING|trainer.py:803] 2025-04-26 16:03:09,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:09,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:09,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3116
3141
3134
[WARNING|trainer.py:803] 2025-04-26 16:03:11,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:11,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:12,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3142
3117
[WARNING|trainer.py:803] 2025-04-26 16:03:14,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3135
[WARNING|trainer.py:803] 2025-04-26 16:03:14,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:15,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3143
3136
3118
[WARNING|trainer.py:803] 2025-04-26 16:03:16,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:17,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:03:17,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3144
3137
[WARNING|trainer.py:803] 2025-04-26 16:03:19,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3119
[WARNING|trainer.py:803] 2025-04-26 16:03:19,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:20,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3145
3138
3120
[WARNING|trainer.py:803] 2025-04-26 16:03:22,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:22,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:22,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3146
3139
[WARNING|trainer.py:803] 2025-04-26 16:03:24,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3121
[WARNING|trainer.py:803] 2025-04-26 16:03:25,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:25,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3147
3140
[WARNING|trainer.py:803] 2025-04-26 16:03:27,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3122
[WARNING|trainer.py:803] 2025-04-26 16:03:28,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:28,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3148
3141
[WARNING|trainer.py:803] 2025-04-26 16:03:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3123
[WARNING|trainer.py:803] 2025-04-26 16:03:30,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3149
[WARNING|trainer.py:803] 2025-04-26 16:03:31,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:03:31,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3142
[WARNING|trainer.py:803] 2025-04-26 16:03:33,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3150
3124
[WARNING|trainer.py:803] 2025-04-26 16:03:34,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:34,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3143
3151
[WARNING|trainer.py:803] 2025-04-26 16:03:35,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3125
[WARNING|trainer.py:803] 2025-04-26 16:03:36,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:36,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3144
3152
[WARNING|trainer.py:803] 2025-04-26 16:03:38,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:38,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3126
[WARNING|trainer.py:803] 2025-04-26 16:03:39,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3153
[WARNING|trainer.py:803] 2025-04-26 16:03:40,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3145
[WARNING|trainer.py:803] 2025-04-26 16:03:41,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3154
3127
[WARNING|trainer.py:803] 2025-04-26 16:03:42,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:42,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3146
3155
[WARNING|trainer.py:803] 2025-04-26 16:03:43,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3128
[WARNING|trainer.py:803] 2025-04-26 16:03:44,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:45,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3147
3156
[WARNING|trainer.py:803] 2025-04-26 16:03:46,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:03:46,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3129
[WARNING|trainer.py:803] 2025-04-26 16:03:47,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3157
3148
[WARNING|trainer.py:803] 2025-04-26 16:03:48,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:49,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3130
3158
[WARNING|trainer.py:803] 2025-04-26 16:03:50,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3149
[WARNING|trainer.py:803] 2025-04-26 16:03:50,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:51,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3131
3159
3150
[WARNING|trainer.py:803] 2025-04-26 16:03:52,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:52,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:53,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3160
3151
3132
[WARNING|trainer.py:803] 2025-04-26 16:03:55,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:55,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:03:55,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3161
3152
3133
[WARNING|trainer.py:803] 2025-04-26 16:03:57,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:57,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:57,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3162
3153
3134
[WARNING|trainer.py:803] 2025-04-26 16:03:59,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:03:59,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:03:59,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3163
3154
[WARNING|trainer.py:803] 2025-04-26 16:04:01,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:01,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3135
3155
3164
[WARNING|trainer.py:803] 2025-04-26 16:04:02,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:03,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:03,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3136
3156
3165
[WARNING|trainer.py:803] 2025-04-26 16:04:04,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:05,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:05,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3166
3137
3157
[WARNING|trainer.py:803] 2025-04-26 16:04:07,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:07,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:07,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3167
3158
3138
[WARNING|trainer.py:803] 2025-04-26 16:04:09,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:09,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:10,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3168
3159
[WARNING|trainer.py:803] 2025-04-26 16:04:11,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:11,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3139
[WARNING|trainer.py:803] 2025-04-26 16:04:12,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3169
3160
[WARNING|trainer.py:803] 2025-04-26 16:04:13,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:14,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3140
3170
3161
[WARNING|trainer.py:803] 2025-04-26 16:04:15,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:15,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:16,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3171
3162
3141
[WARNING|trainer.py:803] 2025-04-26 16:04:17,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:18,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:18,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3172
3163
3142
[WARNING|trainer.py:803] 2025-04-26 16:04:19,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:20,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:20,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3173
3164
[WARNING|trainer.py:803] 2025-04-26 16:04:21,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3143
[WARNING|trainer.py:803] 2025-04-26 16:04:22,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3174
[WARNING|trainer.py:803] 2025-04-26 16:04:23,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3165
[WARNING|trainer.py:803] 2025-04-26 16:04:24,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:24,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3144
3175
3166
[WARNING|trainer.py:803] 2025-04-26 16:04:25,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:26,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:26,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3176
3167
[WARNING|trainer.py:803] 2025-04-26 16:04:28,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3145
[WARNING|trainer.py:803] 2025-04-26 16:04:28,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:29,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3177
3168
[WARNING|trainer.py:803] 2025-04-26 16:04:30,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:30,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3146
[WARNING|trainer.py:803] 2025-04-26 16:04:31,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3178
3169
[WARNING|trainer.py:803] 2025-04-26 16:04:32,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:32,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3147
3170
3179
[WARNING|trainer.py:803] 2025-04-26 16:04:34,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:34,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:34,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3171
3148
3180
[WARNING|trainer.py:803] 2025-04-26 16:04:36,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:36,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:36,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3172
3149
3181
[WARNING|trainer.py:803] 2025-04-26 16:04:38,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:38,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:38,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3173
3150
3182
[WARNING|trainer.py:803] 2025-04-26 16:04:40,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:40,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:04:40,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
NoNo
3174
3183
3151
[WARNING|trainer.py:803] 2025-04-26 16:04:42,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:42,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:43,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3175
3184
3152
[WARNING|trainer.py:803] 2025-04-26 16:04:44,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:45,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:45,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3176
3185
3153
[WARNING|trainer.py:803] 2025-04-26 16:04:47,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:47,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:47,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3186
3177
3154
[WARNING|trainer.py:803] 2025-04-26 16:04:49,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:49,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:49,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3187
3155
3178
[WARNING|trainer.py:803] 2025-04-26 16:04:51,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:51,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:51,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3188
3156
3179
[WARNING|trainer.py:803] 2025-04-26 16:04:53,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:53,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:53,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3189
3157
3180
[WARNING|trainer.py:803] 2025-04-26 16:04:55,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:55,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:04:55,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3190
3158
3181
[WARNING|trainer.py:803] 2025-04-26 16:04:57,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:04:57,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:04:57,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3191
3182
3159
[WARNING|trainer.py:803] 2025-04-26 16:04:59,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:04:59,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:04:59,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3183
3192
3160
[WARNING|trainer.py:803] 2025-04-26 16:05:01,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:01,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:02,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3184
3193
3161
[WARNING|trainer.py:803] 2025-04-26 16:05:03,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:03,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:04,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3194
3185
3162
[WARNING|trainer.py:803] 2025-04-26 16:05:06,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:06,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:06,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3186
3195
3163
[WARNING|trainer.py:803] 2025-04-26 16:05:08,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:08,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:08,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3196
3187
3164
[WARNING|trainer.py:803] 2025-04-26 16:05:10,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:10,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:10,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3197
3188
3165
[WARNING|trainer.py:803] 2025-04-26 16:05:12,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:12,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:12,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3166
3198
3189
[WARNING|trainer.py:803] 2025-04-26 16:05:14,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:14,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:14,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3167
3199
3190
[WARNING|trainer.py:803] 2025-04-26 16:05:16,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:16,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:16,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3191
3168
3200
[WARNING|trainer.py:803] 2025-04-26 16:05:18,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:05:18,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:18,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3201
3169
3192
[WARNING|trainer.py:803] 2025-04-26 16:05:20,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:20,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:20,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3202
3170
3193
[WARNING|trainer.py:803] 2025-04-26 16:05:22,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:22,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:23,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3171
3203
3194
[WARNING|trainer.py:803] 2025-04-26 16:05:24,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:24,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:25,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3204
3172
3195
[WARNING|trainer.py:803] 2025-04-26 16:05:26,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:27,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:27,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3205
3173
3196
[WARNING|trainer.py:803] 2025-04-26 16:05:28,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:29,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:29,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3206
3174
3197
[WARNING|trainer.py:803] 2025-04-26 16:05:31,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:31,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:31,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3207
3175
3198
[WARNING|trainer.py:803] 2025-04-26 16:05:33,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:33,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:33,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3208
3176
3199
[WARNING|trainer.py:803] 2025-04-26 16:05:35,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:35,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:35,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3209
3177
3200
[WARNING|trainer.py:803] 2025-04-26 16:05:37,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:37,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:37,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3210
3201
3178
[WARNING|trainer.py:803] 2025-04-26 16:05:39,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:39,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:39,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3211
3202
3179
[WARNING|trainer.py:803] 2025-04-26 16:05:41,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:41,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:41,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3212
3203
3180
[WARNING|trainer.py:803] 2025-04-26 16:05:43,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:43,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:44,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3213
3204
3181
[WARNING|trainer.py:803] 2025-04-26 16:05:45,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:45,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:46,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3214
3205
3182
[WARNING|trainer.py:803] 2025-04-26 16:05:47,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:47,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:05:48,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3215
3206
3183
[WARNING|trainer.py:803] 2025-04-26 16:05:49,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:49,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:50,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3216
3207
3184
[WARNING|trainer.py:803] 2025-04-26 16:05:51,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:52,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:52,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3217
3208
3185
[WARNING|trainer.py:803] 2025-04-26 16:05:53,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:05:54,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:05:54,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3218
3209
3186
[WARNING|trainer.py:803] 2025-04-26 16:05:55,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:56,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:56,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3219
3210
3187
[WARNING|trainer.py:803] 2025-04-26 16:05:57,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:05:58,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:05:58,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3220
3211
3188
[WARNING|trainer.py:803] 2025-04-26 16:05:59,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:00,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:00,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3221
3212
3189
[WARNING|trainer.py:803] 2025-04-26 16:06:02,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:02,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:02,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3222
3213
[WARNING|trainer.py:803] 2025-04-26 16:06:04,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3190
[WARNING|trainer.py:803] 2025-04-26 16:06:04,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:04,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3223
3214
[WARNING|trainer.py:803] 2025-04-26 16:06:06,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3191
[WARNING|trainer.py:803] 2025-04-26 16:06:06,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:06,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3224
3215
[WARNING|trainer.py:803] 2025-04-26 16:06:08,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3192
[WARNING|trainer.py:803] 2025-04-26 16:06:08,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:09,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3225
3216
[WARNING|trainer.py:803] 2025-04-26 16:06:10,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3193
[WARNING|trainer.py:803] 2025-04-26 16:06:10,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:11,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3226
3217
[WARNING|trainer.py:803] 2025-04-26 16:06:12,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3194
[WARNING|trainer.py:803] 2025-04-26 16:06:12,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3227
[WARNING|trainer.py:803] 2025-04-26 16:06:13,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3218
[WARNING|trainer.py:803] 2025-04-26 16:06:14,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3195
[WARNING|trainer.py:803] 2025-04-26 16:06:14,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3228
[WARNING|trainer.py:803] 2025-04-26 16:06:15,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3219
[WARNING|trainer.py:803] 2025-04-26 16:06:16,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:16,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3196
3229
[WARNING|trainer.py:803] 2025-04-26 16:06:17,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3220
[WARNING|trainer.py:803] 2025-04-26 16:06:18,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:18,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3197
3230
[WARNING|trainer.py:803] 2025-04-26 16:06:19,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3221
[WARNING|trainer.py:803] 2025-04-26 16:06:20,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:20,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3198
3231
[WARNING|trainer.py:803] 2025-04-26 16:06:21,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3222
[WARNING|trainer.py:803] 2025-04-26 16:06:22,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:22,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3199
3232
[WARNING|trainer.py:803] 2025-04-26 16:06:24,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3223
[WARNING|trainer.py:803] 2025-04-26 16:06:24,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:24,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3200
3233
3224
[WARNING|trainer.py:803] 2025-04-26 16:06:26,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:26,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:26,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3201
3234
3225
[WARNING|trainer.py:803] 2025-04-26 16:06:28,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:28,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:28,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3235
3202
3226
[WARNING|trainer.py:803] 2025-04-26 16:06:30,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:30,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:30,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3236
3203
3227
[WARNING|trainer.py:803] 2025-04-26 16:06:32,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:32,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:32,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3237
3204
3228
[WARNING|trainer.py:803] 2025-04-26 16:06:34,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:34,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:34,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3205
3238
3229
[WARNING|trainer.py:803] 2025-04-26 16:06:36,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:36,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:36,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3239
3206
3230
[WARNING|trainer.py:803] 2025-04-26 16:06:38,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:38,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:38,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3231
3240
3207
[WARNING|trainer.py:803] 2025-04-26 16:06:40,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:40,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:40,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3241
3232
3208
[WARNING|trainer.py:803] 2025-04-26 16:06:42,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:42,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:42,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3242
3233
3209
[WARNING|trainer.py:803] 2025-04-26 16:06:44,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:44,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:44,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3234
3243
3210
[WARNING|trainer.py:803] 2025-04-26 16:06:46,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:46,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:46,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3244
3235
3211
[WARNING|trainer.py:803] 2025-04-26 16:06:48,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:06:48,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:49,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3236
3245
3212
[WARNING|trainer.py:803] 2025-04-26 16:06:50,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:50,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:51,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3237
3246
3213
[WARNING|trainer.py:803] 2025-04-26 16:06:52,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:52,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:53,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3247
3238
3214
[WARNING|trainer.py:803] 2025-04-26 16:06:54,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:54,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:55,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3239
3248
3215
[WARNING|trainer.py:803] 2025-04-26 16:06:56,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:06:56,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:57,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3249
3240
3216
[WARNING|trainer.py:803] 2025-04-26 16:06:58,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:06:58,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:06:59,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3241
3250
3217
[WARNING|trainer.py:803] 2025-04-26 16:07:00,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:00,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:01,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3251
3242
3218
[WARNING|trainer.py:803] 2025-04-26 16:07:02,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:02,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:03,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3252
3243
3219
[WARNING|trainer.py:803] 2025-04-26 16:07:04,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:04,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:07:05,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3253
3244
3220
[WARNING|trainer.py:803] 2025-04-26 16:07:06,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:06,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:07,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3254
3245
[WARNING|trainer.py:803] 2025-04-26 16:07:08,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3221
[WARNING|trainer.py:803] 2025-04-26 16:07:09,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:09,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3255
3246
[WARNING|trainer.py:803] 2025-04-26 16:07:10,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3222
[WARNING|trainer.py:803] 2025-04-26 16:07:11,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:07:11,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3256
3247
[WARNING|trainer.py:803] 2025-04-26 16:07:12,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3223
[WARNING|trainer.py:803] 2025-04-26 16:07:12,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3257
[WARNING|trainer.py:803] 2025-04-26 16:07:13,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3248
[WARNING|trainer.py:803] 2025-04-26 16:07:14,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3224
[WARNING|trainer.py:803] 2025-04-26 16:07:14,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:15,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3258
3249
[WARNING|trainer.py:803] 2025-04-26 16:07:16,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3225
[WARNING|trainer.py:803] 2025-04-26 16:07:16,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:17,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3259
3250
[WARNING|trainer.py:803] 2025-04-26 16:07:18,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3226
[WARNING|trainer.py:803] 2025-04-26 16:07:18,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3260
[WARNING|trainer.py:803] 2025-04-26 16:07:19,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3251
[WARNING|trainer.py:803] 2025-04-26 16:07:20,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3227
[WARNING|trainer.py:803] 2025-04-26 16:07:20,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:21,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3261
3252
[WARNING|trainer.py:803] 2025-04-26 16:07:22,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3228
[WARNING|trainer.py:803] 2025-04-26 16:07:22,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3262
[WARNING|trainer.py:803] 2025-04-26 16:07:23,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3253
[WARNING|trainer.py:803] 2025-04-26 16:07:24,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:24,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3229
3263
[WARNING|trainer.py:803] 2025-04-26 16:07:25,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3254
[WARNING|trainer.py:803] 2025-04-26 16:07:26,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:07:26,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3230
3255
3264
[WARNING|trainer.py:803] 2025-04-26 16:07:27,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:28,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:28,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3231
[WARNING|trainer.py:803] 2025-04-26 16:07:29,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3256
3265
[WARNING|trainer.py:803] 2025-04-26 16:07:30,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:07:30,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3232
3257
[WARNING|trainer.py:803] 2025-04-26 16:07:31,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3266
[WARNING|trainer.py:803] 2025-04-26 16:07:32,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:07:32,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3233
3258
[WARNING|trainer.py:803] 2025-04-26 16:07:33,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3267
[WARNING|trainer.py:803] 2025-04-26 16:07:34,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:07:34,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3234
3259
[WARNING|trainer.py:803] 2025-04-26 16:07:35,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3268
[WARNING|trainer.py:803] 2025-04-26 16:07:36,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:07:36,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3235
3260
[WARNING|trainer.py:803] 2025-04-26 16:07:37,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3269
[WARNING|trainer.py:803] 2025-04-26 16:07:38,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:07:38,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3236
3261
[WARNING|trainer.py:803] 2025-04-26 16:07:39,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3270
[WARNING|trainer.py:803] 2025-04-26 16:07:40,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:40,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3237
3262
[WARNING|trainer.py:803] 2025-04-26 16:07:41,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3271
[WARNING|trainer.py:803] 2025-04-26 16:07:42,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:42,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3238
3263
[WARNING|trainer.py:803] 2025-04-26 16:07:43,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3272
[WARNING|trainer.py:803] 2025-04-26 16:07:44,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:07:44,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3239
3264
[WARNING|trainer.py:803] 2025-04-26 16:07:45,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3273
[WARNING|trainer.py:803] 2025-04-26 16:07:46,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:46,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3240
3265
[WARNING|trainer.py:803] 2025-04-26 16:07:47,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3274
[WARNING|trainer.py:803] 2025-04-26 16:07:48,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:48,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3241
3266
[WARNING|trainer.py:803] 2025-04-26 16:07:49,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3275
[WARNING|trainer.py:803] 2025-04-26 16:07:50,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:07:50,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3242
3267
[WARNING|trainer.py:803] 2025-04-26 16:07:51,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3276
[WARNING|trainer.py:803] 2025-04-26 16:07:52,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:52,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3243
3268
[WARNING|trainer.py:803] 2025-04-26 16:07:53,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3277
[WARNING|trainer.py:803] 2025-04-26 16:07:54,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:54,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3244
[WARNING|trainer.py:803] 2025-04-26 16:07:55,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3269
3278
[WARNING|trainer.py:803] 2025-04-26 16:07:56,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:07:56,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3245
3270
[WARNING|trainer.py:803] 2025-04-26 16:07:57,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3279
[WARNING|trainer.py:803] 2025-04-26 16:07:58,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:07:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3246
3271
[WARNING|trainer.py:803] 2025-04-26 16:08:00,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3280
[WARNING|trainer.py:803] 2025-04-26 16:08:00,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:00,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3247
3272
[WARNING|trainer.py:803] 2025-04-26 16:08:01,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3281
[WARNING|trainer.py:803] 2025-04-26 16:08:02,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:02,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3248
3273
[WARNING|trainer.py:803] 2025-04-26 16:08:04,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3282
[WARNING|trainer.py:803] 2025-04-26 16:08:04,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:05,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3249
[WARNING|trainer.py:803] 2025-04-26 16:08:05,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3274
3283
[WARNING|trainer.py:803] 2025-04-26 16:08:06,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:07,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3250
[WARNING|trainer.py:803] 2025-04-26 16:08:08,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3275
3284
3251
[WARNING|trainer.py:803] 2025-04-26 16:08:08,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:09,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:09,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3276
3285
3252
[WARNING|trainer.py:803] 2025-04-26 16:08:11,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 16:08:11,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:11,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3277
3286
3253
[WARNING|trainer.py:803] 2025-04-26 16:08:13,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:13,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:13,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3278
3254
3287
[WARNING|trainer.py:803] 2025-04-26 16:08:15,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:15,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:15,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3279
3255
3288
[WARNING|trainer.py:803] 2025-04-26 16:08:16,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:17,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:17,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3280
3256
3289
[WARNING|trainer.py:803] 2025-04-26 16:08:19,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:19,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:08:19,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3281
3257
3290
[WARNING|trainer.py:803] 2025-04-26 16:08:21,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:21,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:08:21,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3282
3258
3291
[WARNING|trainer.py:803] 2025-04-26 16:08:23,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:23,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:08:23,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3283
3259
3292
[WARNING|trainer.py:803] 2025-04-26 16:08:25,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:25,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:08:25,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3284
3260
3293
[WARNING|trainer.py:803] 2025-04-26 16:08:27,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:27,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:27,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3285
3261
3294
[WARNING|trainer.py:803] 2025-04-26 16:08:29,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:29,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:29,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3262
3286
3295
[WARNING|trainer.py:803] 2025-04-26 16:08:31,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:31,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:31,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3263
3287
3296
[WARNING|trainer.py:803] 2025-04-26 16:08:33,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:08:33,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:33,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3288
3264
3297
[WARNING|trainer.py:803] 2025-04-26 16:08:35,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:35,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:35,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3289
3265
3298
[WARNING|trainer.py:803] 2025-04-26 16:08:37,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:37,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:38,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3290
3266
3299
[WARNING|trainer.py:803] 2025-04-26 16:08:39,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:39,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:40,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3291
3267
3300
[WARNING|trainer.py:803] 2025-04-26 16:08:41,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:42,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:42,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3301
3292
3268
[WARNING|trainer.py:803] 2025-04-26 16:08:43,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:08:43,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3302
[WARNING|trainer.py:803] 2025-04-26 16:08:44,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:44,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3293
3303
3269
[WARNING|trainer.py:803] 2025-04-26 16:08:45,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:46,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:46,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3304
3294
[WARNING|trainer.py:803] 2025-04-26 16:08:47,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3270
3305
[WARNING|trainer.py:803] 2025-04-26 16:08:47,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:48,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:48,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3295
3306
3271
[WARNING|trainer.py:803] 2025-04-26 16:08:50,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:50,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:50,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3307
3296
[WARNING|trainer.py:803] 2025-04-26 16:08:51,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3272
3308
[WARNING|trainer.py:803] 2025-04-26 16:08:52,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:08:52,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:08:52,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3309
3297
3273
[WARNING|trainer.py:803] 2025-04-26 16:08:53,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:53,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3310
[WARNING|trainer.py:803] 2025-04-26 16:08:54,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3298
[WARNING|trainer.py:803] 2025-04-26 16:08:55,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3274
3311
[WARNING|trainer.py:803] 2025-04-26 16:08:56,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:56,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:56,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3312
3299
3275
[WARNING|trainer.py:803] 2025-04-26 16:08:57,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:58,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3313
[WARNING|trainer.py:803] 2025-04-26 16:08:58,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:08:59,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3300
3314
3276
[WARNING|trainer.py:803] 2025-04-26 16:09:00,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:09:00,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:00,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3301
3315
[WARNING|trainer.py:803] 2025-04-26 16:09:01,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3277
[WARNING|trainer.py:803] 2025-04-26 16:09:01,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3302
3316
[WARNING|trainer.py:803] 2025-04-26 16:09:02,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:02,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:03,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3303
3317
3278
[WARNING|trainer.py:803] 2025-04-26 16:09:04,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:04,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:04,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3304
3318
[WARNING|trainer.py:803] 2025-04-26 16:09:05,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3279
[WARNING|trainer.py:803] 2025-04-26 16:09:05,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3305
3319
[WARNING|trainer.py:803] 2025-04-26 16:09:06,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:06,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:06,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3306
3320
3280
[WARNING|trainer.py:803] 2025-04-26 16:09:08,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:08,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:08,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3307
3321
[WARNING|trainer.py:803] 2025-04-26 16:09:09,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:09,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3281
3308
3322
[WARNING|trainer.py:803] 2025-04-26 16:09:10,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:10,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:10,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3309
3323
3282
[WARNING|trainer.py:803] 2025-04-26 16:09:12,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:12,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3310
3324
[WARNING|trainer.py:803] 2025-04-26 16:09:12,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:13,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:13,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3311
3283
3325
[WARNING|trainer.py:803] 2025-04-26 16:09:14,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:14,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:14,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3312
3326
[WARNING|trainer.py:803] 2025-04-26 16:09:15,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3284
[WARNING|trainer.py:803] 2025-04-26 16:09:16,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3313
3327
[WARNING|trainer.py:803] 2025-04-26 16:09:16,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:17,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:17,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3314
3328
3285
[WARNING|trainer.py:803] 2025-04-26 16:09:18,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:18,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3315
[WARNING|trainer.py:803] 2025-04-26 16:09:18,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3329
[WARNING|trainer.py:803] 2025-04-26 16:09:19,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:20,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3316
3286
3330
[WARNING|trainer.py:803] 2025-04-26 16:09:21,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:21,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:21,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3317
3331
3287
[WARNING|trainer.py:803] 2025-04-26 16:09:22,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:22,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3318
3332
[WARNING|trainer.py:803] 2025-04-26 16:09:23,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:23,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:23,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3319
3288
3333
[WARNING|trainer.py:803] 2025-04-26 16:09:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:25,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:25,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3320
3334
[WARNING|trainer.py:803] 2025-04-26 16:09:26,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3289
[WARNING|trainer.py:803] 2025-04-26 16:09:26,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3321
3335
[WARNING|trainer.py:803] 2025-04-26 16:09:27,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:27,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:27,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3322
3336
3290
[WARNING|trainer.py:803] 2025-04-26 16:09:28,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:29,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3323
[WARNING|trainer.py:803] 2025-04-26 16:09:29,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3337
[WARNING|trainer.py:803] 2025-04-26 16:09:30,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:30,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3291
3324
3338
[WARNING|trainer.py:803] 2025-04-26 16:09:31,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:31,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:31,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3325
3339
3292
[WARNING|trainer.py:803] 2025-04-26 16:09:32,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:09:33,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3326
3340
[WARNING|trainer.py:803] 2025-04-26 16:09:33,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:34,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:34,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3327
3293
3341
[WARNING|trainer.py:803] 2025-04-26 16:09:35,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:35,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:35,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3328
3342
[WARNING|trainer.py:803] 2025-04-26 16:09:36,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3294
3329
[WARNING|trainer.py:803] 2025-04-26 16:09:37,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:37,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3343
[WARNING|trainer.py:803] 2025-04-26 16:09:37,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3330
[WARNING|trainer.py:803] 2025-04-26 16:09:38,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3295
3344
[WARNING|trainer.py:803] 2025-04-26 16:09:39,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:39,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3331
[WARNING|trainer.py:803] 2025-04-26 16:09:39,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3345
[WARNING|trainer.py:803] 2025-04-26 16:09:40,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3296
3332
[WARNING|trainer.py:803] 2025-04-26 16:09:41,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3346
[WARNING|trainer.py:803] 2025-04-26 16:09:41,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:41,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3333
[WARNING|trainer.py:803] 2025-04-26 16:09:42,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3297
3347
[WARNING|trainer.py:803] 2025-04-26 16:09:43,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3334
[WARNING|trainer.py:803] 2025-04-26 16:09:43,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:43,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3348
[WARNING|trainer.py:803] 2025-04-26 16:09:44,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3335
3298
[WARNING|trainer.py:803] 2025-04-26 16:09:44,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3349
[WARNING|trainer.py:803] 2025-04-26 16:09:45,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:45,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3336
[WARNING|trainer.py:803] 2025-04-26 16:09:46,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3350
[WARNING|trainer.py:803] 2025-04-26 16:09:46,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3299
3337
[WARNING|trainer.py:803] 2025-04-26 16:09:47,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:47,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3351
[WARNING|trainer.py:803] 2025-04-26 16:09:48,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3338
[WARNING|trainer.py:803] 2025-04-26 16:09:48,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3300
3352
[WARNING|trainer.py:803] 2025-04-26 16:09:49,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:49,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3339
[WARNING|trainer.py:803] 2025-04-26 16:09:50,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3301
3353
[WARNING|trainer.py:803] 2025-04-26 16:09:50,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:09:51,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3340
[WARNING|trainer.py:803] 2025-04-26 16:09:51,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3302
3354
[WARNING|trainer.py:803] 2025-04-26 16:09:52,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:52,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3341
[WARNING|trainer.py:803] 2025-04-26 16:09:52,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3303
3355
[WARNING|trainer.py:803] 2025-04-26 16:09:53,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:53,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3342
[WARNING|trainer.py:803] 2025-04-26 16:09:54,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3304
3356
[WARNING|trainer.py:803] 2025-04-26 16:09:54,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:09:55,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3343
[WARNING|trainer.py:803] 2025-04-26 16:09:55,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3305
3357
[WARNING|trainer.py:803] 2025-04-26 16:09:55,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:56,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3344
[WARNING|trainer.py:803] 2025-04-26 16:09:56,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3306
3358
[WARNING|trainer.py:803] 2025-04-26 16:09:57,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:57,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3345
[WARNING|trainer.py:803] 2025-04-26 16:09:57,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3307
3359
[WARNING|trainer.py:803] 2025-04-26 16:09:58,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:09:58,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3346
[WARNING|trainer.py:803] 2025-04-26 16:09:59,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3308
3360
[WARNING|trainer.py:803] 2025-04-26 16:09:59,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:00,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3347
[WARNING|trainer.py:803] 2025-04-26 16:10:00,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3309
3361
[WARNING|trainer.py:803] 2025-04-26 16:10:01,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:01,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3348
[WARNING|trainer.py:803] 2025-04-26 16:10:01,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3310
3362
[WARNING|trainer.py:803] 2025-04-26 16:10:02,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:02,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3349
[WARNING|trainer.py:803] 2025-04-26 16:10:03,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3311
3363
[WARNING|trainer.py:803] 2025-04-26 16:10:03,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:04,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3350
[WARNING|trainer.py:803] 2025-04-26 16:10:04,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3312
3364
[WARNING|trainer.py:803] 2025-04-26 16:10:05,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:05,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3351
[WARNING|trainer.py:803] 2025-04-26 16:10:05,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3313
3365
[WARNING|trainer.py:803] 2025-04-26 16:10:06,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:06,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3352
[WARNING|trainer.py:803] 2025-04-26 16:10:06,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3314
3366
[WARNING|trainer.py:803] 2025-04-26 16:10:07,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:07,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3353
[WARNING|trainer.py:803] 2025-04-26 16:10:08,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3315
3367
[WARNING|trainer.py:803] 2025-04-26 16:10:09,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:09,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3354
[WARNING|trainer.py:803] 2025-04-26 16:10:09,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3316
3368
[WARNING|trainer.py:803] 2025-04-26 16:10:10,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:10,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3355
[WARNING|trainer.py:803] 2025-04-26 16:10:10,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3317
3369
[WARNING|trainer.py:803] 2025-04-26 16:10:11,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:11,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3356
[WARNING|trainer.py:803] 2025-04-26 16:10:12,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3318
3370
[WARNING|trainer.py:803] 2025-04-26 16:10:12,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:13,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3357
[WARNING|trainer.py:803] 2025-04-26 16:10:13,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3319
3371
[WARNING|trainer.py:803] 2025-04-26 16:10:14,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:14,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3358
[WARNING|trainer.py:803] 2025-04-26 16:10:14,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3320
3372
[WARNING|trainer.py:803] 2025-04-26 16:10:15,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:15,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3359
[WARNING|trainer.py:803] 2025-04-26 16:10:15,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3321
3373
[WARNING|trainer.py:803] 2025-04-26 16:10:16,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:17,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3360
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:17,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3322
3374
[WARNING|trainer.py:803] 2025-04-26 16:10:18,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3361
[WARNING|trainer.py:803] 2025-04-26 16:10:18,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:18,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3323
3375
[WARNING|trainer.py:803] 2025-04-26 16:10:19,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3362
[WARNING|trainer.py:803] 2025-04-26 16:10:19,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:19,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3376
3324
[WARNING|trainer.py:803] 2025-04-26 16:10:20,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3363
[WARNING|trainer.py:803] 2025-04-26 16:10:21,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:21,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3377
3325
[WARNING|trainer.py:803] 2025-04-26 16:10:21,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3364
[WARNING|trainer.py:803] 2025-04-26 16:10:22,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:22,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3378
3326
[WARNING|trainer.py:803] 2025-04-26 16:10:23,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3365
[WARNING|trainer.py:803] 2025-04-26 16:10:23,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:23,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3379
3327
[WARNING|trainer.py:803] 2025-04-26 16:10:24,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3366
[WARNING|trainer.py:803] 2025-04-26 16:10:25,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:25,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3380
3328
[WARNING|trainer.py:803] 2025-04-26 16:10:25,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3367
[WARNING|trainer.py:803] 2025-04-26 16:10:26,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:26,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3381
3329
[WARNING|trainer.py:803] 2025-04-26 16:10:27,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3368
[WARNING|trainer.py:803] 2025-04-26 16:10:27,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:27,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3382
3330
[WARNING|trainer.py:803] 2025-04-26 16:10:28,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3369
[WARNING|trainer.py:803] 2025-04-26 16:10:28,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:29,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3383
3331
[WARNING|trainer.py:803] 2025-04-26 16:10:29,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3370
[WARNING|trainer.py:803] 2025-04-26 16:10:30,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:30,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3384
3332
[WARNING|trainer.py:803] 2025-04-26 16:10:30,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3371
[WARNING|trainer.py:803] 2025-04-26 16:10:31,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:31,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3333
3385
[WARNING|trainer.py:803] 2025-04-26 16:10:32,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3372
[WARNING|trainer.py:803] 2025-04-26 16:10:32,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:32,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3334
3386
[WARNING|trainer.py:803] 2025-04-26 16:10:33,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3373
[WARNING|trainer.py:803] 2025-04-26 16:10:34,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:34,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3335
3387
[WARNING|trainer.py:803] 2025-04-26 16:10:34,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3374
[WARNING|trainer.py:803] 2025-04-26 16:10:35,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:35,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3336
3388
[WARNING|trainer.py:803] 2025-04-26 16:10:36,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3375
[WARNING|trainer.py:803] 2025-04-26 16:10:36,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:36,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3389
3337
[WARNING|trainer.py:803] 2025-04-26 16:10:37,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3376
[WARNING|trainer.py:803] 2025-04-26 16:10:38,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:38,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3390
3338
[WARNING|trainer.py:803] 2025-04-26 16:10:38,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3377
[WARNING|trainer.py:803] 2025-04-26 16:10:39,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:39,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3391
3339
[WARNING|trainer.py:803] 2025-04-26 16:10:40,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3378
[WARNING|trainer.py:803] 2025-04-26 16:10:40,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:40,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3392
3340
[WARNING|trainer.py:803] 2025-04-26 16:10:41,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3379
[WARNING|trainer.py:803] 2025-04-26 16:10:41,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:10:42,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3393
3341
[WARNING|trainer.py:803] 2025-04-26 16:10:42,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3380
[WARNING|trainer.py:803] 2025-04-26 16:10:43,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:43,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3394
3342
[WARNING|trainer.py:803] 2025-04-26 16:10:43,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3381
[WARNING|trainer.py:803] 2025-04-26 16:10:44,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:44,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3395
3343
[WARNING|trainer.py:803] 2025-04-26 16:10:45,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3382
[WARNING|trainer.py:803] 2025-04-26 16:10:45,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:45,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3396
3344
[WARNING|trainer.py:803] 2025-04-26 16:10:46,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3383
[WARNING|trainer.py:803] 2025-04-26 16:10:47,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:10:47,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3397
3345
[WARNING|trainer.py:803] 2025-04-26 16:10:47,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3384
[WARNING|trainer.py:803] 2025-04-26 16:10:48,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:48,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3398
3346
[WARNING|trainer.py:803] 2025-04-26 16:10:49,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3385
[WARNING|trainer.py:803] 2025-04-26 16:10:49,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:49,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3399
3347
[WARNING|trainer.py:803] 2025-04-26 16:10:50,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3386
[WARNING|trainer.py:803] 2025-04-26 16:10:51,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:51,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3400
3348
[WARNING|trainer.py:803] 2025-04-26 16:10:51,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3387
[WARNING|trainer.py:803] 2025-04-26 16:10:52,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:10:52,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3401
3349
[WARNING|trainer.py:803] 2025-04-26 16:10:52,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3388
[WARNING|trainer.py:803] 2025-04-26 16:10:53,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:10:53,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3402
3350
[WARNING|trainer.py:803] 2025-04-26 16:10:54,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3389
[WARNING|trainer.py:803] 2025-04-26 16:10:54,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:55,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3403
[WARNING|trainer.py:803] 2025-04-26 16:10:55,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3351
3390
[WARNING|trainer.py:803] 2025-04-26 16:10:56,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:10:56,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3404
[WARNING|trainer.py:803] 2025-04-26 16:10:56,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3352
3391
[WARNING|trainer.py:803] 2025-04-26 16:10:57,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:10:57,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3405
[WARNING|trainer.py:803] 2025-04-26 16:10:58,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3353
3392
[WARNING|trainer.py:803] 2025-04-26 16:10:58,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:10:58,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3406
[WARNING|trainer.py:803] 2025-04-26 16:10:59,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3354
3393
[WARNING|trainer.py:803] 2025-04-26 16:11:00,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:00,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:00,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3407
3355
3394
[WARNING|trainer.py:803] 2025-04-26 16:11:01,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:01,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:02,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3356
3408
3395
[WARNING|trainer.py:803] 2025-04-26 16:11:02,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:02,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:03,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3357
3409
3396
[WARNING|trainer.py:803] 2025-04-26 16:11:04,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:04,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:04,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3358
3410
3397
[WARNING|trainer.py:803] 2025-04-26 16:11:05,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:05,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:05,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3359
3411
3398
[WARNING|trainer.py:803] 2025-04-26 16:11:06,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:06,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:07,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3360
3412
3399
[WARNING|trainer.py:803] 2025-04-26 16:11:08,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:08,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:08,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3361
3413
3400
[WARNING|trainer.py:803] 2025-04-26 16:11:09,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:09,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3362
[WARNING|trainer.py:803] 2025-04-26 16:11:09,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3414
3401
[WARNING|trainer.py:803] 2025-04-26 16:11:10,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:10,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3363
[WARNING|trainer.py:803] 2025-04-26 16:11:11,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3415
3402
[WARNING|trainer.py:803] 2025-04-26 16:11:11,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:11,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3364
[WARNING|trainer.py:803] 2025-04-26 16:11:12,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3416
3403
[WARNING|trainer.py:803] 2025-04-26 16:11:13,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:13,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3365
[WARNING|trainer.py:803] 2025-04-26 16:11:13,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3417
3404
[WARNING|trainer.py:803] 2025-04-26 16:11:14,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:14,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3366
[WARNING|trainer.py:803] 2025-04-26 16:11:15,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3418
3405
[WARNING|trainer.py:803] 2025-04-26 16:11:15,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:15,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3367
[WARNING|trainer.py:803] 2025-04-26 16:11:16,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3419
3406
[WARNING|trainer.py:803] 2025-04-26 16:11:17,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:17,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3368
[WARNING|trainer.py:803] 2025-04-26 16:11:17,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3420
3407
[WARNING|trainer.py:803] 2025-04-26 16:11:18,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:18,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3369
[WARNING|trainer.py:803] 2025-04-26 16:11:19,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3421
3408
[WARNING|trainer.py:803] 2025-04-26 16:11:19,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:19,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3370
[WARNING|trainer.py:803] 2025-04-26 16:11:20,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3422
3409
[WARNING|trainer.py:803] 2025-04-26 16:11:21,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:21,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3371
[WARNING|trainer.py:803] 2025-04-26 16:11:21,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3423
3410
[WARNING|trainer.py:803] 2025-04-26 16:11:22,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:22,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3372
[WARNING|trainer.py:803] 2025-04-26 16:11:22,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3424
3411
[WARNING|trainer.py:803] 2025-04-26 16:11:23,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:23,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3373
[WARNING|trainer.py:803] 2025-04-26 16:11:24,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3425
3412
[WARNING|trainer.py:803] 2025-04-26 16:11:25,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:25,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3374
3426
[WARNING|trainer.py:803] 2025-04-26 16:11:25,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3413
[WARNING|trainer.py:803] 2025-04-26 16:11:26,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:26,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3375
3427
[WARNING|trainer.py:803] 2025-04-26 16:11:27,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3414
[WARNING|trainer.py:803] 2025-04-26 16:11:27,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:27,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3376
3428
[WARNING|trainer.py:803] 2025-04-26 16:11:28,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3415
[WARNING|trainer.py:803] 2025-04-26 16:11:28,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:28,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3377
3429
[WARNING|trainer.py:803] 2025-04-26 16:11:29,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3416
[WARNING|trainer.py:803] 2025-04-26 16:11:30,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:30,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3378
3430
[WARNING|trainer.py:803] 2025-04-26 16:11:30,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3417
[WARNING|trainer.py:803] 2025-04-26 16:11:31,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:31,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3379
3431
[WARNING|trainer.py:803] 2025-04-26 16:11:32,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3418
[WARNING|trainer.py:803] 2025-04-26 16:11:32,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:32,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3380
3432
[WARNING|trainer.py:803] 2025-04-26 16:11:33,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:34,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3419
[WARNING|trainer.py:803] 2025-04-26 16:11:34,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3381
3433
[WARNING|trainer.py:803] 2025-04-26 16:11:35,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:35,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:35,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3420
3434
3382
[WARNING|trainer.py:803] 2025-04-26 16:11:36,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:36,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:36,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3421
3435
3383
[WARNING|trainer.py:803] 2025-04-26 16:11:37,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:38,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:38,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3422
3436
3384
[WARNING|trainer.py:803] 2025-04-26 16:11:38,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:39,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3423
[WARNING|trainer.py:803] 2025-04-26 16:11:39,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3437
3385
[WARNING|trainer.py:803] 2025-04-26 16:11:40,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:40,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3424
[WARNING|trainer.py:803] 2025-04-26 16:11:40,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3438
3386
[WARNING|trainer.py:803] 2025-04-26 16:11:41,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:41,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3425
[WARNING|trainer.py:803] 2025-04-26 16:11:42,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3439
3387
[WARNING|trainer.py:803] 2025-04-26 16:11:42,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:43,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3426
[WARNING|trainer.py:803] 2025-04-26 16:11:43,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3440
3388
[WARNING|trainer.py:803] 2025-04-26 16:11:44,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:44,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3427
[WARNING|trainer.py:803] 2025-04-26 16:11:44,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3441
3389
[WARNING|trainer.py:803] 2025-04-26 16:11:45,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:45,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3428
[WARNING|trainer.py:803] 2025-04-26 16:11:45,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3442
3390
[WARNING|trainer.py:803] 2025-04-26 16:11:46,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:47,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3429
[WARNING|trainer.py:803] 2025-04-26 16:11:47,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3443
3391
[WARNING|trainer.py:803] 2025-04-26 16:11:48,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:48,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:48,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3430
3444
3392
[WARNING|trainer.py:803] 2025-04-26 16:11:49,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:49,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:49,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3431
3445
3393
[WARNING|trainer.py:803] 2025-04-26 16:11:50,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:51,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:51,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3432
3446
3394
[WARNING|trainer.py:803] 2025-04-26 16:11:52,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:52,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:52,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3433
3447
3395
[WARNING|trainer.py:803] 2025-04-26 16:11:53,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:53,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:53,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3434
3396
3448
[WARNING|trainer.py:803] 2025-04-26 16:11:54,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:55,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:11:55,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3435
3397
3449
[WARNING|trainer.py:803] 2025-04-26 16:11:56,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:56,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:11:56,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3436
3398
3450
[WARNING|trainer.py:803] 2025-04-26 16:11:57,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:57,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:11:57,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3437
3399
3451
[WARNING|trainer.py:803] 2025-04-26 16:11:58,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:11:59,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:11:59,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3438
3400
3452
[WARNING|trainer.py:803] 2025-04-26 16:11:59,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:00,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:00,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3439
3401
3453
[WARNING|trainer.py:803] 2025-04-26 16:12:01,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:01,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3440
[WARNING|trainer.py:803] 2025-04-26 16:12:01,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3402
3454
[WARNING|trainer.py:803] 2025-04-26 16:12:02,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:02,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3441
[WARNING|trainer.py:803] 2025-04-26 16:12:03,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3403
3455
[WARNING|trainer.py:803] 2025-04-26 16:12:03,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3442
[WARNING|trainer.py:803] 2025-04-26 16:12:04,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:12:04,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3404
3456
[WARNING|trainer.py:803] 2025-04-26 16:12:05,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3443
[WARNING|trainer.py:803] 2025-04-26 16:12:05,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:05,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3457
3405
[WARNING|trainer.py:803] 2025-04-26 16:12:06,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3444
[WARNING|trainer.py:803] 2025-04-26 16:12:06,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:07,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3458
3406
[WARNING|trainer.py:803] 2025-04-26 16:12:07,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3445
[WARNING|trainer.py:803] 2025-04-26 16:12:08,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:08,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3459
3407
[WARNING|trainer.py:803] 2025-04-26 16:12:09,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3446
[WARNING|trainer.py:803] 2025-04-26 16:12:09,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:09,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3460
3408
[WARNING|trainer.py:803] 2025-04-26 16:12:10,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3447
[WARNING|trainer.py:803] 2025-04-26 16:12:10,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:11,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3461
3409
[WARNING|trainer.py:803] 2025-04-26 16:12:11,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3448
[WARNING|trainer.py:803] 2025-04-26 16:12:12,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:12,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3462
3410
[WARNING|trainer.py:803] 2025-04-26 16:12:12,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3449
[WARNING|trainer.py:803] 2025-04-26 16:12:13,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:13,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3463
[WARNING|trainer.py:803] 2025-04-26 16:12:14,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3411
3450
[WARNING|trainer.py:803] 2025-04-26 16:12:14,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:15,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3464
[WARNING|trainer.py:803] 2025-04-26 16:12:15,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3412
3451
[WARNING|trainer.py:803] 2025-04-26 16:12:16,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:16,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3465
[WARNING|trainer.py:803] 2025-04-26 16:12:16,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3413
3452
[WARNING|trainer.py:803] 2025-04-26 16:12:17,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:17,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3466
[WARNING|trainer.py:803] 2025-04-26 16:12:18,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3414
3453
[WARNING|trainer.py:803] 2025-04-26 16:12:18,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:18,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3467
[WARNING|trainer.py:803] 2025-04-26 16:12:19,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3415
3454
[WARNING|trainer.py:803] 2025-04-26 16:12:20,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:20,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3468
[WARNING|trainer.py:803] 2025-04-26 16:12:20,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3416
3455
[WARNING|trainer.py:803] 2025-04-26 16:12:21,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:21,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3469
[WARNING|trainer.py:803] 2025-04-26 16:12:21,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3417
3456
[WARNING|trainer.py:803] 2025-04-26 16:12:22,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:22,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3470
[WARNING|trainer.py:803] 2025-04-26 16:12:23,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3418
3457
[WARNING|trainer.py:803] 2025-04-26 16:12:23,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:24,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3471
[WARNING|trainer.py:803] 2025-04-26 16:12:24,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3419
3458
[WARNING|trainer.py:803] 2025-04-26 16:12:25,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:25,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3472
[WARNING|trainer.py:803] 2025-04-26 16:12:25,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3420
3459
[WARNING|trainer.py:803] 2025-04-26 16:12:26,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:26,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3473
[WARNING|trainer.py:803] 2025-04-26 16:12:27,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3421
3460
[WARNING|trainer.py:803] 2025-04-26 16:12:27,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3474
[WARNING|trainer.py:803] 2025-04-26 16:12:28,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:28,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3422
3461
[WARNING|trainer.py:803] 2025-04-26 16:12:29,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3475
[WARNING|trainer.py:803] 2025-04-26 16:12:29,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:29,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3423
3462
[WARNING|trainer.py:803] 2025-04-26 16:12:30,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3476
[WARNING|trainer.py:803] 2025-04-26 16:12:30,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:31,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3424
3463
[WARNING|trainer.py:803] 2025-04-26 16:12:31,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3477
[WARNING|trainer.py:803] 2025-04-26 16:12:32,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:32,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3425
3464
[WARNING|trainer.py:803] 2025-04-26 16:12:32,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3478
[WARNING|trainer.py:803] 2025-04-26 16:12:33,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:33,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3465
3426
[WARNING|trainer.py:803] 2025-04-26 16:12:34,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3479
[WARNING|trainer.py:803] 2025-04-26 16:12:34,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:34,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3466
3427
[WARNING|trainer.py:803] 2025-04-26 16:12:35,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3480
[WARNING|trainer.py:803] 2025-04-26 16:12:36,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:36,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3467
3428
[WARNING|trainer.py:803] 2025-04-26 16:12:36,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3481
[WARNING|trainer.py:803] 2025-04-26 16:12:37,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:37,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3468
3429
[WARNING|trainer.py:803] 2025-04-26 16:12:38,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3482
[WARNING|trainer.py:803] 2025-04-26 16:12:38,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:38,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3469
3430
[WARNING|trainer.py:803] 2025-04-26 16:12:39,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3483
[WARNING|trainer.py:803] 2025-04-26 16:12:40,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:40,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3470
[WARNING|trainer.py:803] 2025-04-26 16:12:40,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3431
3484
[WARNING|trainer.py:803] 2025-04-26 16:12:41,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:41,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3471
[WARNING|trainer.py:803] 2025-04-26 16:12:42,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3432
3485
[WARNING|trainer.py:803] 2025-04-26 16:12:42,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:42,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3472
[WARNING|trainer.py:803] 2025-04-26 16:12:43,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3433
3486
[WARNING|trainer.py:803] 2025-04-26 16:12:43,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:44,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3473
[WARNING|trainer.py:803] 2025-04-26 16:12:44,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3434
3487
[WARNING|trainer.py:803] 2025-04-26 16:12:45,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:45,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3474
[WARNING|trainer.py:803] 2025-04-26 16:12:45,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3435
3488
[WARNING|trainer.py:803] 2025-04-26 16:12:46,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:46,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3475
[WARNING|trainer.py:803] 2025-04-26 16:12:47,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3436
3489
[WARNING|trainer.py:803] 2025-04-26 16:12:47,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:48,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3476
[WARNING|trainer.py:803] 2025-04-26 16:12:48,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3437
3490
[WARNING|trainer.py:803] 2025-04-26 16:12:49,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:49,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3477
[WARNING|trainer.py:803] 2025-04-26 16:12:49,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3438
3491
[WARNING|trainer.py:803] 2025-04-26 16:12:50,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:12:50,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3478
[WARNING|trainer.py:803] 2025-04-26 16:12:51,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3439
3492
[WARNING|trainer.py:803] 2025-04-26 16:12:51,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:52,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3479
[WARNING|trainer.py:803] 2025-04-26 16:12:52,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3440
3493
[WARNING|trainer.py:803] 2025-04-26 16:12:52,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:53,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3480
[WARNING|trainer.py:803] 2025-04-26 16:12:53,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3441
3494
[WARNING|trainer.py:803] 2025-04-26 16:12:54,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:12:54,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3481
[WARNING|trainer.py:803] 2025-04-26 16:12:54,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3442
3495
[WARNING|trainer.py:803] 2025-04-26 16:12:55,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:55,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3482
[WARNING|trainer.py:803] 2025-04-26 16:12:56,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3443
3496
[WARNING|trainer.py:803] 2025-04-26 16:12:56,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:57,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3483
[WARNING|trainer.py:803] 2025-04-26 16:12:57,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3444
3497
[WARNING|trainer.py:803] 2025-04-26 16:12:58,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:12:58,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3484
[WARNING|trainer.py:803] 2025-04-26 16:12:58,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3445
3498
[WARNING|trainer.py:803] 2025-04-26 16:12:59,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:12:59,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3485
[WARNING|trainer.py:803] 2025-04-26 16:13:00,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3446
3499
[WARNING|trainer.py:803] 2025-04-26 16:13:00,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:01,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3486
[WARNING|trainer.py:803] 2025-04-26 16:13:01,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3447
3500
[WARNING|trainer.py:803] 2025-04-26 16:13:02,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:02,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3487
[WARNING|trainer.py:803] 2025-04-26 16:13:02,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3448
3501
[WARNING|trainer.py:803] 2025-04-26 16:13:03,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:03,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3488
[WARNING|trainer.py:803] 2025-04-26 16:13:04,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3449
3502
[WARNING|trainer.py:803] 2025-04-26 16:13:04,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:05,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3489
[WARNING|trainer.py:803] 2025-04-26 16:13:05,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3450
3503
[WARNING|trainer.py:803] 2025-04-26 16:13:06,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3490
[WARNING|trainer.py:803] 2025-04-26 16:13:06,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:06,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3504
3451
[WARNING|trainer.py:803] 2025-04-26 16:13:07,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3491
[WARNING|trainer.py:803] 2025-04-26 16:13:07,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:07,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3505
3452
[WARNING|trainer.py:803] 2025-04-26 16:13:08,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:09,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3492
[WARNING|trainer.py:803] 2025-04-26 16:13:09,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3506
3453
[WARNING|trainer.py:803] 2025-04-26 16:13:09,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:10,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3493
[WARNING|trainer.py:803] 2025-04-26 16:13:10,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3507
3454
[WARNING|trainer.py:803] 2025-04-26 16:13:11,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:11,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3494
[WARNING|trainer.py:803] 2025-04-26 16:13:11,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3508
3455
[WARNING|trainer.py:803] 2025-04-26 16:13:12,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:12,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3495
[WARNING|trainer.py:803] 2025-04-26 16:13:13,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3509
3456
[WARNING|trainer.py:803] 2025-04-26 16:13:13,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:14,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3496
[WARNING|trainer.py:803] 2025-04-26 16:13:14,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3510
3457
[WARNING|trainer.py:803] 2025-04-26 16:13:15,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:15,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3497
[WARNING|trainer.py:803] 2025-04-26 16:13:15,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3511
3458
[WARNING|trainer.py:803] 2025-04-26 16:13:16,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:16,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3498
[WARNING|trainer.py:803] 2025-04-26 16:13:17,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3512
3459
[WARNING|trainer.py:803] 2025-04-26 16:13:17,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:18,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3499
[WARNING|trainer.py:803] 2025-04-26 16:13:18,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3513
3460
[WARNING|trainer.py:803] 2025-04-26 16:13:19,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:19,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3500
[WARNING|trainer.py:803] 2025-04-26 16:13:19,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3514
3461
[WARNING|trainer.py:803] 2025-04-26 16:13:20,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:20,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3501
[WARNING|trainer.py:803] 2025-04-26 16:13:21,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3515
3462
[WARNING|trainer.py:803] 2025-04-26 16:13:21,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:21,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3502
3516
[WARNING|trainer.py:803] 2025-04-26 16:13:22,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3463
[WARNING|trainer.py:803] 2025-04-26 16:13:23,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:23,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3503
3517
[WARNING|trainer.py:803] 2025-04-26 16:13:23,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3464
[WARNING|trainer.py:803] 2025-04-26 16:13:24,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:24,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3504
3518
[WARNING|trainer.py:803] 2025-04-26 16:13:25,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3465
[WARNING|trainer.py:803] 2025-04-26 16:13:25,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:25,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3505
3519
[WARNING|trainer.py:803] 2025-04-26 16:13:26,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3466
[WARNING|trainer.py:803] 2025-04-26 16:13:26,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:27,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3506
3520
[WARNING|trainer.py:803] 2025-04-26 16:13:27,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3467
[WARNING|trainer.py:803] 2025-04-26 16:13:28,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:28,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3507
3521
[WARNING|trainer.py:803] 2025-04-26 16:13:28,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:29,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3468
[WARNING|trainer.py:803] 2025-04-26 16:13:29,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3508
3522
[WARNING|trainer.py:803] 2025-04-26 16:13:30,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:30,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3469
[WARNING|trainer.py:803] 2025-04-26 16:13:30,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3509
3523
[WARNING|trainer.py:803] 2025-04-26 16:13:31,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:13:32,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3470
[WARNING|trainer.py:803] 2025-04-26 16:13:32,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3510
3524
[WARNING|trainer.py:803] 2025-04-26 16:13:32,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:33,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3471
[WARNING|trainer.py:803] 2025-04-26 16:13:33,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3511
3525
[WARNING|trainer.py:803] 2025-04-26 16:13:34,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:34,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3472
[WARNING|trainer.py:803] 2025-04-26 16:13:34,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3512
3526
[WARNING|trainer.py:803] 2025-04-26 16:13:35,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:35,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3473
[WARNING|trainer.py:803] 2025-04-26 16:13:36,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3513
3527
[WARNING|trainer.py:803] 2025-04-26 16:13:36,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:37,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3474
[WARNING|trainer.py:803] 2025-04-26 16:13:37,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3514
3528
[WARNING|trainer.py:803] 2025-04-26 16:13:38,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3475
[WARNING|trainer.py:803] 2025-04-26 16:13:38,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:38,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3515
3529
[WARNING|trainer.py:803] 2025-04-26 16:13:39,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3476
[WARNING|trainer.py:803] 2025-04-26 16:13:39,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:39,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3516
3530
[WARNING|trainer.py:803] 2025-04-26 16:13:40,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3477
[WARNING|trainer.py:803] 2025-04-26 16:13:41,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:41,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x89737980] moov atom not found
[16:13:41] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
3517
[WARNING|trainer.py:803] 2025-04-26 16:13:41,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3531
3478
[WARNING|trainer.py:803] 2025-04-26 16:13:42,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3518
[WARNING|trainer.py:803] 2025-04-26 16:13:43,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:13:43,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3532
3479
[WARNING|trainer.py:803] 2025-04-26 16:13:43,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3519
[WARNING|trainer.py:803] 2025-04-26 16:13:44,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:44,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3533
[WARNING|trainer.py:803] 2025-04-26 16:13:45,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3480
3520
[WARNING|trainer.py:803] 2025-04-26 16:13:45,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:46,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3534
[WARNING|trainer.py:803] 2025-04-26 16:13:46,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3481
3521
[WARNING|trainer.py:803] 2025-04-26 16:13:47,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:47,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3535
[WARNING|trainer.py:803] 2025-04-26 16:13:47,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3482
3522
[WARNING|trainer.py:803] 2025-04-26 16:13:48,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:48,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3536
[WARNING|trainer.py:803] 2025-04-26 16:13:49,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3483
3523
[WARNING|trainer.py:803] 2025-04-26 16:13:49,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:50,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3537
[WARNING|trainer.py:803] 2025-04-26 16:13:50,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3484
3524
[WARNING|trainer.py:803] 2025-04-26 16:13:50,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:51,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3538
[WARNING|trainer.py:803] 2025-04-26 16:13:51,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3485
3525
[WARNING|trainer.py:803] 2025-04-26 16:13:52,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:52,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3539
[WARNING|trainer.py:803] 2025-04-26 16:13:52,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3486
3526
[WARNING|trainer.py:803] 2025-04-26 16:13:53,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:13:53,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3540
[WARNING|trainer.py:803] 2025-04-26 16:13:54,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3487
3527
[WARNING|trainer.py:803] 2025-04-26 16:13:54,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:13:55,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3541
[WARNING|trainer.py:803] 2025-04-26 16:13:55,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3488
3528
[WARNING|trainer.py:803] 2025-04-26 16:13:56,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:13:56,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3542
[WARNING|trainer.py:803] 2025-04-26 16:13:56,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3489
3529
[WARNING|trainer.py:803] 2025-04-26 16:13:57,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:13:57,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3543
[WARNING|trainer.py:803] 2025-04-26 16:13:58,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3490
3530
[WARNING|trainer.py:803] 2025-04-26 16:13:58,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:13:59,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3544
[WARNING|trainer.py:803] 2025-04-26 16:13:59,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3b8651c0] moov atom not found
[16:13:59] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
3491
[WARNING|trainer.py:803] 2025-04-26 16:14:00,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:00,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3545
3531
3492
[WARNING|trainer.py:803] 2025-04-26 16:14:01,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:01,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:01,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3546
3532
3493
[WARNING|trainer.py:803] 2025-04-26 16:14:02,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:02,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:03,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3547
3533
3494
[WARNING|trainer.py:803] 2025-04-26 16:14:03,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:04,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3548
[WARNING|trainer.py:803] 2025-04-26 16:14:04,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3534
3495
[WARNING|trainer.py:803] 2025-04-26 16:14:05,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:05,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3549
[WARNING|trainer.py:803] 2025-04-26 16:14:05,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3535
3496
[WARNING|trainer.py:803] 2025-04-26 16:14:06,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:06,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3550
[WARNING|trainer.py:803] 2025-04-26 16:14:07,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3536
3497
[WARNING|trainer.py:803] 2025-04-26 16:14:07,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:07,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3551
[WARNING|trainer.py:803] 2025-04-26 16:14:08,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3537
3498
[WARNING|trainer.py:803] 2025-04-26 16:14:09,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:09,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3552
[WARNING|trainer.py:803] 2025-04-26 16:14:09,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3538
3499
[WARNING|trainer.py:803] 2025-04-26 16:14:10,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:10,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3553
[WARNING|trainer.py:803] 2025-04-26 16:14:10,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3539
3500
[WARNING|trainer.py:803] 2025-04-26 16:14:11,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:11,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3554
[WARNING|trainer.py:803] 2025-04-26 16:14:12,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3540
3501
[WARNING|trainer.py:803] 2025-04-26 16:14:12,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:13,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3555
[WARNING|trainer.py:803] 2025-04-26 16:14:13,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3541
3502
[WARNING|trainer.py:803] 2025-04-26 16:14:14,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:14,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3556
[WARNING|trainer.py:803] 2025-04-26 16:14:14,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3542
3503
[WARNING|trainer.py:803] 2025-04-26 16:14:15,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:15,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3557
[WARNING|trainer.py:803] 2025-04-26 16:14:16,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3543
3504
[WARNING|trainer.py:803] 2025-04-26 16:14:16,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:17,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3558
[WARNING|trainer.py:803] 2025-04-26 16:14:17,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3544
3505
[WARNING|trainer.py:803] 2025-04-26 16:14:18,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:18,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3559
[WARNING|trainer.py:803] 2025-04-26 16:14:18,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3545
3506
[WARNING|trainer.py:803] 2025-04-26 16:14:19,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3560
[WARNING|trainer.py:803] 2025-04-26 16:14:19,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:19,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3546
3507
[WARNING|trainer.py:803] 2025-04-26 16:14:20,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3561
[WARNING|trainer.py:803] 2025-04-26 16:14:21,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:21,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3547
3508
[WARNING|trainer.py:803] 2025-04-26 16:14:21,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3562
[WARNING|trainer.py:803] 2025-04-26 16:14:22,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:22,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3548
3509
[WARNING|trainer.py:803] 2025-04-26 16:14:23,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3563
[WARNING|trainer.py:803] 2025-04-26 16:14:23,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:23,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3549
3510
[WARNING|trainer.py:803] 2025-04-26 16:14:24,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3564
[WARNING|trainer.py:803] 2025-04-26 16:14:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:25,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3550
3511
[WARNING|trainer.py:803] 2025-04-26 16:14:25,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3565
[WARNING|trainer.py:803] 2025-04-26 16:14:26,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:26,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3551
3512
[WARNING|trainer.py:803] 2025-04-26 16:14:26,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3566
[WARNING|trainer.py:803] 2025-04-26 16:14:27,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:27,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3552
3513
[WARNING|trainer.py:803] 2025-04-26 16:14:28,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3567
[WARNING|trainer.py:803] 2025-04-26 16:14:28,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:28,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3553
3514
[WARNING|trainer.py:803] 2025-04-26 16:14:29,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3568
[WARNING|trainer.py:803] 2025-04-26 16:14:30,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:30,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3554
3515
[WARNING|trainer.py:803] 2025-04-26 16:14:30,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3569
[WARNING|trainer.py:803] 2025-04-26 16:14:31,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:31,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3555
3516
[WARNING|trainer.py:803] 2025-04-26 16:14:32,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3570
[WARNING|trainer.py:803] 2025-04-26 16:14:32,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:32,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3556
3517
[WARNING|trainer.py:803] 2025-04-26 16:14:33,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3571
[WARNING|trainer.py:803] 2025-04-26 16:14:34,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:34,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3557
3518
[WARNING|trainer.py:803] 2025-04-26 16:14:34,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3572
[WARNING|trainer.py:803] 2025-04-26 16:14:35,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:35,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3558
3519
[WARNING|trainer.py:803] 2025-04-26 16:14:36,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3573
[WARNING|trainer.py:803] 2025-04-26 16:14:36,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:36,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3559
3520
[WARNING|trainer.py:803] 2025-04-26 16:14:37,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3574
[WARNING|trainer.py:803] 2025-04-26 16:14:37,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:38,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3560
3521
[WARNING|trainer.py:803] 2025-04-26 16:14:38,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3575
[WARNING|trainer.py:803] 2025-04-26 16:14:39,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:39,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3561
3522
[WARNING|trainer.py:803] 2025-04-26 16:14:39,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3576
[WARNING|trainer.py:803] 2025-04-26 16:14:40,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:14:40,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3562
3523
[WARNING|trainer.py:803] 2025-04-26 16:14:41,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3577
[WARNING|trainer.py:803] 2025-04-26 16:14:41,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:41,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3563
3524
[WARNING|trainer.py:803] 2025-04-26 16:14:42,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3578
[WARNING|trainer.py:803] 2025-04-26 16:14:43,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:43,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3564
3525
[WARNING|trainer.py:803] 2025-04-26 16:14:43,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3579
[WARNING|trainer.py:803] 2025-04-26 16:14:44,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:14:44,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3565
3526
[WARNING|trainer.py:803] 2025-04-26 16:14:45,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3580
[WARNING|trainer.py:803] 2025-04-26 16:14:45,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:45,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3527
3566
[WARNING|trainer.py:803] 2025-04-26 16:14:46,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3581
[WARNING|trainer.py:803] 2025-04-26 16:14:47,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:14:47,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3528
3567
[WARNING|trainer.py:803] 2025-04-26 16:14:47,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3582
[WARNING|trainer.py:803] 2025-04-26 16:14:48,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:48,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3568
3529
[WARNING|trainer.py:803] 2025-04-26 16:14:49,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3583
[WARNING|trainer.py:803] 2025-04-26 16:14:49,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:49,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3530
3569
[WARNING|trainer.py:803] 2025-04-26 16:14:50,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3584
[WARNING|trainer.py:803] 2025-04-26 16:14:51,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x589b8940] moov atom not found
[16:14:51] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
[WARNING|trainer.py:803] 2025-04-26 16:14:51,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3570
[WARNING|trainer.py:803] 2025-04-26 16:14:51,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3585
3531
[WARNING|trainer.py:803] 2025-04-26 16:14:52,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3571
[WARNING|trainer.py:803] 2025-04-26 16:14:52,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:14:53,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3586
3532
[WARNING|trainer.py:803] 2025-04-26 16:14:53,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3572
[WARNING|trainer.py:803] 2025-04-26 16:14:54,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:54,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3587
3533
[WARNING|trainer.py:803] 2025-04-26 16:14:54,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3573
[WARNING|trainer.py:803] 2025-04-26 16:14:55,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:55,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3588
3534
[WARNING|trainer.py:803] 2025-04-26 16:14:56,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3574
[WARNING|trainer.py:803] 2025-04-26 16:14:56,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:56,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3589
3535
[WARNING|trainer.py:803] 2025-04-26 16:14:57,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3575
[WARNING|trainer.py:803] 2025-04-26 16:14:58,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:58,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3590
3536
[WARNING|trainer.py:803] 2025-04-26 16:14:58,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3576
[WARNING|trainer.py:803] 2025-04-26 16:14:59,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:14:59,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3591
3537
[WARNING|trainer.py:803] 2025-04-26 16:15:00,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3577
[WARNING|trainer.py:803] 2025-04-26 16:15:00,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:00,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3592
3538
[WARNING|trainer.py:803] 2025-04-26 16:15:01,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3578
[WARNING|trainer.py:803] 2025-04-26 16:15:02,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:02,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3593
3539
[WARNING|trainer.py:803] 2025-04-26 16:15:02,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3579
[WARNING|trainer.py:803] 2025-04-26 16:15:03,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:03,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3594
3540
[WARNING|trainer.py:803] 2025-04-26 16:15:03,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3580
[WARNING|trainer.py:803] 2025-04-26 16:15:04,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:04,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3595
3541
[WARNING|trainer.py:803] 2025-04-26 16:15:05,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3581
[WARNING|trainer.py:803] 2025-04-26 16:15:06,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:06,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3596
[WARNING|trainer.py:803] 2025-04-26 16:15:06,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3542
3582
[WARNING|trainer.py:803] 2025-04-26 16:15:07,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:07,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3597
[WARNING|trainer.py:803] 2025-04-26 16:15:07,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3543
3583
[WARNING|trainer.py:803] 2025-04-26 16:15:08,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:08,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3598
[WARNING|trainer.py:803] 2025-04-26 16:15:09,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3544
3584
[WARNING|trainer.py:803] 2025-04-26 16:15:09,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:10,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:10,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3599
3545
3585
[WARNING|trainer.py:803] 2025-04-26 16:15:11,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:11,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:11,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3600
3546
3586
[WARNING|trainer.py:803] 2025-04-26 16:15:12,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:12,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:13,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3601
3547
3587
[WARNING|trainer.py:803] 2025-04-26 16:15:13,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:13,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:14,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3548
3602
3588
[WARNING|trainer.py:803] 2025-04-26 16:15:15,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:15,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:15,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3549
3603
3589
[WARNING|trainer.py:803] 2025-04-26 16:15:16,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:16,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:16,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3550
3604
3590
[WARNING|trainer.py:803] 2025-04-26 16:15:17,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:17,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:18,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3551
3605
3591
[WARNING|trainer.py:803] 2025-04-26 16:15:19,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:19,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:19,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3552
3606
3592
[WARNING|trainer.py:803] 2025-04-26 16:15:20,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:20,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:20,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3553
3607
3593
[WARNING|trainer.py:803] 2025-04-26 16:15:21,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:21,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:22,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3554
NoNo
3608
3594
[WARNING|trainer.py:803] 2025-04-26 16:15:22,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:23,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3555
[WARNING|trainer.py:803] 2025-04-26 16:15:23,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3609
3595
[WARNING|trainer.py:803] 2025-04-26 16:15:24,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:24,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3556
[WARNING|trainer.py:803] 2025-04-26 16:15:24,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3610
3596
[WARNING|trainer.py:803] 2025-04-26 16:15:25,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:25,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3557
[WARNING|trainer.py:803] 2025-04-26 16:15:26,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3611
3597
[WARNING|trainer.py:803] 2025-04-26 16:15:26,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:27,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3558
[WARNING|trainer.py:803] 2025-04-26 16:15:27,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3612
3598
[WARNING|trainer.py:803] 2025-04-26 16:15:28,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:28,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3559
[WARNING|trainer.py:803] 2025-04-26 16:15:28,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3613
3599
[WARNING|trainer.py:803] 2025-04-26 16:15:29,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3560
[WARNING|trainer.py:803] 2025-04-26 16:15:29,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:29,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3614
3600
[WARNING|trainer.py:803] 2025-04-26 16:15:30,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3561
[WARNING|trainer.py:803] 2025-04-26 16:15:31,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:31,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3615
3601
[WARNING|trainer.py:803] 2025-04-26 16:15:32,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3562
[WARNING|trainer.py:803] 2025-04-26 16:15:32,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:32,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3602
3616
[WARNING|trainer.py:803] 2025-04-26 16:15:33,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3563
[WARNING|trainer.py:803] 2025-04-26 16:15:33,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:33,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3603
3617
[WARNING|trainer.py:803] 2025-04-26 16:15:34,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3564
[WARNING|trainer.py:803] 2025-04-26 16:15:35,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:35,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3604
[WARNING|trainer.py:803] 2025-04-26 16:15:35,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3618
3565
[WARNING|trainer.py:803] 2025-04-26 16:15:36,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:36,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3605
[WARNING|trainer.py:803] 2025-04-26 16:15:37,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3619
3566
[WARNING|trainer.py:803] 2025-04-26 16:15:37,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:37,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3606
[WARNING|trainer.py:803] 2025-04-26 16:15:38,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3620
3567
[WARNING|trainer.py:803] 2025-04-26 16:15:39,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:39,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3607
[WARNING|trainer.py:803] 2025-04-26 16:15:39,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3621
3568
[WARNING|trainer.py:803] 2025-04-26 16:15:40,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:40,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3608
[WARNING|trainer.py:803] 2025-04-26 16:15:41,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3622
3569
[WARNING|trainer.py:803] 2025-04-26 16:15:41,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:41,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:42,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3609
3623
3570
[WARNING|trainer.py:803] 2025-04-26 16:15:43,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:43,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:43,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3610
3624
3571
[WARNING|trainer.py:803] 2025-04-26 16:15:44,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:44,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:44,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3611
3625
3572
[WARNING|trainer.py:803] 2025-04-26 16:15:45,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:46,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3612
3626
3573
[WARNING|trainer.py:803] 2025-04-26 16:15:47,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:47,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:47,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3613
3627
3574
[WARNING|trainer.py:803] 2025-04-26 16:15:48,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:48,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:48,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3614
3575
3628
[WARNING|trainer.py:803] 2025-04-26 16:15:49,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:50,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:50,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3615
3576
3629
[WARNING|trainer.py:803] 2025-04-26 16:15:51,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:51,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:51,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3577
3616
3630
[WARNING|trainer.py:803] 2025-04-26 16:15:52,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:15:52,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:15:53,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3578
3617
3631
[WARNING|trainer.py:803] 2025-04-26 16:15:54,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:54,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:54,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3579
3618
3632
[WARNING|trainer.py:803] 2025-04-26 16:15:55,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:55,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:55,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3580
3619
3633
[WARNING|trainer.py:803] 2025-04-26 16:15:56,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:56,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:56,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3581
3620
3634
[WARNING|trainer.py:803] 2025-04-26 16:15:57,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:15:58,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:15:58,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3582
3621
3635
[WARNING|trainer.py:803] 2025-04-26 16:15:59,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:59,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:15:59,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3583
3622
3636
[WARNING|trainer.py:803] 2025-04-26 16:16:00,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:00,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3584
[WARNING|trainer.py:803] 2025-04-26 16:16:01,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3623
3637
[WARNING|trainer.py:803] 2025-04-26 16:16:01,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:02,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3585
[WARNING|trainer.py:803] 2025-04-26 16:16:02,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3624
3638
[WARNING|trainer.py:803] 2025-04-26 16:16:03,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3586
[WARNING|trainer.py:803] 2025-04-26 16:16:03,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:03,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3625
[WARNING|trainer.py:803] 2025-04-26 16:16:04,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3639
3587
[WARNING|trainer.py:803] 2025-04-26 16:16:05,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:05,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3626
[WARNING|trainer.py:803] 2025-04-26 16:16:05,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3640
3588
[WARNING|trainer.py:803] 2025-04-26 16:16:06,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:06,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3627
[WARNING|trainer.py:803] 2025-04-26 16:16:06,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3641
3589
[WARNING|trainer.py:803] 2025-04-26 16:16:07,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:07,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:08,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3628
3642
3590
[WARNING|trainer.py:803] 2025-04-26 16:16:09,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:09,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:09,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3629
3643
3591
[WARNING|trainer.py:803] 2025-04-26 16:16:10,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:10,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:10,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3630
3644
3592
[WARNING|trainer.py:803] 2025-04-26 16:16:11,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:12,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:12,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3631
3645
3593
[WARNING|trainer.py:803] 2025-04-26 16:16:13,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:13,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:13,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3632
3594
3646
[WARNING|trainer.py:803] 2025-04-26 16:16:14,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:14,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:14,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3633
3595
3647
[WARNING|trainer.py:803] 2025-04-26 16:16:15,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:16,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:16,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3634
3596
3648
[WARNING|trainer.py:803] 2025-04-26 16:16:17,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:17,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:17,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3635
3597
3649
[WARNING|trainer.py:803] 2025-04-26 16:16:18,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:18,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:18,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3636
3598
3650
[WARNING|trainer.py:803] 2025-04-26 16:16:20,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:20,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:20,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3599
3637
3651
[WARNING|trainer.py:803] 2025-04-26 16:16:21,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:21,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:21,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3600
3638
3652
[WARNING|trainer.py:803] 2025-04-26 16:16:22,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:22,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:23,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3601
3639
3653
[WARNING|trainer.py:803] 2025-04-26 16:16:23,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:24,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3602
[WARNING|trainer.py:803] 2025-04-26 16:16:24,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3640
3654
[WARNING|trainer.py:803] 2025-04-26 16:16:25,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:25,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:25,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3603
3641
3655
[WARNING|trainer.py:803] 2025-04-26 16:16:26,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:26,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3604
[WARNING|trainer.py:803] 2025-04-26 16:16:27,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3642
3656
[WARNING|trainer.py:803] 2025-04-26 16:16:27,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:28,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3605
[WARNING|trainer.py:803] 2025-04-26 16:16:28,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3643
3657
[WARNING|trainer.py:803] 2025-04-26 16:16:29,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3606
[WARNING|trainer.py:803] 2025-04-26 16:16:29,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:29,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3658
3644
[WARNING|trainer.py:803] 2025-04-26 16:16:30,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3607
[WARNING|trainer.py:803] 2025-04-26 16:16:31,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:31,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3659
3645
[WARNING|trainer.py:803] 2025-04-26 16:16:31,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3608
[WARNING|trainer.py:803] 2025-04-26 16:16:32,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:32,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3660
[WARNING|trainer.py:803] 2025-04-26 16:16:33,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3646
3609
[WARNING|trainer.py:803] 2025-04-26 16:16:33,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:33,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:34,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3661
3647
3610
[WARNING|trainer.py:803] 2025-04-26 16:16:35,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:35,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:35,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3662
3648
3611
[WARNING|trainer.py:803] 2025-04-26 16:16:36,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:36,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:37,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3663
3649
3612
[WARNING|trainer.py:803] 2025-04-26 16:16:38,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:38,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:38,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3664
3650
3613
[WARNING|trainer.py:803] 2025-04-26 16:16:39,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:39,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:39,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3665
3651
3614
[WARNING|trainer.py:803] 2025-04-26 16:16:40,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:40,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:41,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3666
3652
3615
[WARNING|trainer.py:803] 2025-04-26 16:16:42,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:42,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:42,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3667
3653
3616
[WARNING|trainer.py:803] 2025-04-26 16:16:43,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:43,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:43,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3668
3654
3617
[WARNING|trainer.py:803] 2025-04-26 16:16:44,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:45,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:45,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3669
3655
3618
[WARNING|trainer.py:803] 2025-04-26 16:16:46,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:46,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:46,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3670
3656
3619
[WARNING|trainer.py:803] 2025-04-26 16:16:47,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:47,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:47,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3671
3657
3620
[WARNING|trainer.py:803] 2025-04-26 16:16:49,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:49,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:49,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3672
3658
3621
[WARNING|trainer.py:803] 2025-04-26 16:16:50,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:50,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:50,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3673
3659
3622
[WARNING|trainer.py:803] 2025-04-26 16:16:51,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:51,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:51,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3674
3660
3623
[WARNING|trainer.py:803] 2025-04-26 16:16:52,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:53,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:53,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3675
3661
3624
[WARNING|trainer.py:803] 2025-04-26 16:16:54,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:54,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:54,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3676
3662
3625
[WARNING|trainer.py:803] 2025-04-26 16:16:55,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:56,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:16:56,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3677
3663
3626
[WARNING|trainer.py:803] 2025-04-26 16:16:57,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:57,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:16:57,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3678
3664
3627
[WARNING|trainer.py:803] 2025-04-26 16:16:58,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:16:58,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:16:58,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3679
3665
3628
[WARNING|trainer.py:803] 2025-04-26 16:16:59,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:00,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:00,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3680
3666
3629
[WARNING|trainer.py:803] 2025-04-26 16:17:01,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:01,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:01,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3681
3667
3630
[WARNING|trainer.py:803] 2025-04-26 16:17:02,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:02,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3682
[WARNING|trainer.py:803] 2025-04-26 16:17:03,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3668
3631
[WARNING|trainer.py:803] 2025-04-26 16:17:03,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:04,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:04,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3683
3669
3632
[WARNING|trainer.py:803] 2025-04-26 16:17:05,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:05,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:05,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3684
3633
3670
[WARNING|trainer.py:803] 2025-04-26 16:17:06,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:07,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:07,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3685
3634
3671
[WARNING|trainer.py:803] 2025-04-26 16:17:08,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:08,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:08,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3686
3635
3672
[WARNING|trainer.py:803] 2025-04-26 16:17:09,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:09,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:09,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3687
3636
3673
[WARNING|trainer.py:803] 2025-04-26 16:17:10,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:11,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:11,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3688
3674
3637
[WARNING|trainer.py:803] 2025-04-26 16:17:12,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:12,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:12,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3689
3675
3638
[WARNING|trainer.py:803] 2025-04-26 16:17:13,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:14,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3690
[WARNING|trainer.py:803] 2025-04-26 16:17:14,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3676
3639
[WARNING|trainer.py:803] 2025-04-26 16:17:14,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:15,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:15,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3691
3677
3640
[WARNING|trainer.py:803] 2025-04-26 16:17:16,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:16,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:16,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3692
3678
3641
[WARNING|trainer.py:803] 2025-04-26 16:17:17,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:18,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:18,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3693
3679
3642
[WARNING|trainer.py:803] 2025-04-26 16:17:19,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:19,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:19,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3694
3680
3643
[WARNING|trainer.py:803] 2025-04-26 16:17:20,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:20,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:20,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3695
3681
3644
[WARNING|trainer.py:803] 2025-04-26 16:17:21,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:22,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3696
[WARNING|trainer.py:803] 2025-04-26 16:17:22,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3682
3645
[WARNING|trainer.py:803] 2025-04-26 16:17:23,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3697
[WARNING|trainer.py:803] 2025-04-26 16:17:23,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:23,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3646
3683
[WARNING|trainer.py:803] 2025-04-26 16:17:24,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3698
[WARNING|trainer.py:803] 2025-04-26 16:17:25,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:25,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3647
[WARNING|trainer.py:803] 2025-04-26 16:17:25,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3684
3699
[WARNING|trainer.py:803] 2025-04-26 16:17:26,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:26,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3648
[WARNING|trainer.py:803] 2025-04-26 16:17:27,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3685
3700
[WARNING|trainer.py:803] 2025-04-26 16:17:27,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:28,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3649
[WARNING|trainer.py:803] 2025-04-26 16:17:28,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3686
3701
[WARNING|trainer.py:803] 2025-04-26 16:17:29,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:29,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3650
[WARNING|trainer.py:803] 2025-04-26 16:17:29,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3687
3702
[WARNING|trainer.py:803] 2025-04-26 16:17:30,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:30,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3651
[WARNING|trainer.py:803] 2025-04-26 16:17:31,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3688
3703
[WARNING|trainer.py:803] 2025-04-26 16:17:31,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:32,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3652
[WARNING|trainer.py:803] 2025-04-26 16:17:32,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3689
[WARNING|trainer.py:803] 2025-04-26 16:17:33,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3704
[WARNING|trainer.py:803] 2025-04-26 16:17:33,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3653
3690
[WARNING|trainer.py:803] 2025-04-26 16:17:34,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:34,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3705
[WARNING|trainer.py:803] 2025-04-26 16:17:34,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3654
3691
[WARNING|trainer.py:803] 2025-04-26 16:17:35,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:17:36,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3706
[WARNING|trainer.py:803] 2025-04-26 16:17:36,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3655
3692
[WARNING|trainer.py:803] 2025-04-26 16:17:37,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:37,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3707
[WARNING|trainer.py:803] 2025-04-26 16:17:37,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3656
3693
[WARNING|trainer.py:803] 2025-04-26 16:17:38,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:38,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3708
[WARNING|trainer.py:803] 2025-04-26 16:17:39,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3657
3694
[WARNING|trainer.py:803] 2025-04-26 16:17:39,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:40,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3709
[WARNING|trainer.py:803] 2025-04-26 16:17:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3658
3695
[WARNING|trainer.py:803] 2025-04-26 16:17:41,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:41,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:41,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3710
3659
3696
[WARNING|trainer.py:803] 2025-04-26 16:17:42,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:42,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:43,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3711
3660
3697
[WARNING|trainer.py:803] 2025-04-26 16:17:44,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:44,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:44,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3712
3661
3698
[WARNING|trainer.py:803] 2025-04-26 16:17:45,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:45,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:45,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3713
3662
3699
[WARNING|trainer.py:803] 2025-04-26 16:17:46,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:47,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:47,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3714
3663
3700
[WARNING|trainer.py:803] 2025-04-26 16:17:48,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:48,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:48,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3715
3664
3701
[WARNING|trainer.py:803] 2025-04-26 16:17:49,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:49,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:50,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3716
3665
3702
[WARNING|trainer.py:803] 2025-04-26 16:17:50,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:51,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3717
[WARNING|trainer.py:803] 2025-04-26 16:17:51,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3666
3703
[WARNING|trainer.py:803] 2025-04-26 16:17:52,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:17:52,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:52,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3667
3718
3704
[WARNING|trainer.py:803] 2025-04-26 16:17:53,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:53,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:17:54,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3719
3668
3705
[WARNING|trainer.py:803] 2025-04-26 16:17:55,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:55,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:55,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3669
3720
3706
[WARNING|trainer.py:803] 2025-04-26 16:17:56,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:56,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:57,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3670
3721
3707
[WARNING|trainer.py:803] 2025-04-26 16:17:58,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:17:58,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:17:58,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3671
3722
3708
[WARNING|trainer.py:803] 2025-04-26 16:17:59,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:59,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:17:59,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3672
3723
3709
[WARNING|trainer.py:803] 2025-04-26 16:18:00,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:01,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:01,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3673
3724
3710
[WARNING|trainer.py:803] 2025-04-26 16:18:02,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3674
[WARNING|trainer.py:803] 2025-04-26 16:18:02,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:02,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3725
3711
[WARNING|trainer.py:803] 2025-04-26 16:18:03,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:04,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3675
[WARNING|trainer.py:803] 2025-04-26 16:18:04,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3726
3712
[WARNING|trainer.py:803] 2025-04-26 16:18:05,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:05,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3676
[WARNING|trainer.py:803] 2025-04-26 16:18:05,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3727
3713
[WARNING|trainer.py:803] 2025-04-26 16:18:06,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3677
[WARNING|trainer.py:803] 2025-04-26 16:18:06,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:07,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3728
3714
[WARNING|trainer.py:803] 2025-04-26 16:18:07,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3678
[WARNING|trainer.py:803] 2025-04-26 16:18:08,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:08,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3729
3715
[WARNING|trainer.py:803] 2025-04-26 16:18:09,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:09,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3679
[WARNING|trainer.py:803] 2025-04-26 16:18:09,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3730
3716
[WARNING|trainer.py:803] 2025-04-26 16:18:10,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:11,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3680
[WARNING|trainer.py:803] 2025-04-26 16:18:11,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3731
3717
[WARNING|trainer.py:803] 2025-04-26 16:18:11,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:12,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3681
[WARNING|trainer.py:803] 2025-04-26 16:18:12,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3732
3718
[WARNING|trainer.py:803] 2025-04-26 16:18:13,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:13,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3682
[WARNING|trainer.py:803] 2025-04-26 16:18:14,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3733
[WARNING|trainer.py:803] 2025-04-26 16:18:14,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3719
[WARNING|trainer.py:803] 2025-04-26 16:18:15,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3683
3734
[WARNING|trainer.py:803] 2025-04-26 16:18:15,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3720
[WARNING|trainer.py:803] 2025-04-26 16:18:16,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:16,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3684
3735
[WARNING|trainer.py:803] 2025-04-26 16:18:16,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:17,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3721
[WARNING|trainer.py:803] 2025-04-26 16:18:17,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3736
3685
[WARNING|trainer.py:803] 2025-04-26 16:18:18,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:19,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:19,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3722
3686
3737
[WARNING|trainer.py:803] 2025-04-26 16:18:20,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:20,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:20,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3723
3687
3738
[WARNING|trainer.py:803] 2025-04-26 16:18:21,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:21,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:21,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3724
3688
3739
[WARNING|trainer.py:803] 2025-04-26 16:18:23,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:23,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:23,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3725
3689
3740
[WARNING|trainer.py:803] 2025-04-26 16:18:24,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:24,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:24,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3726
3690
3741
[WARNING|trainer.py:803] 2025-04-26 16:18:25,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:25,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:25,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3727
3691
3742
[WARNING|trainer.py:803] 2025-04-26 16:18:27,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:27,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:27,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3728
3692
3743
[WARNING|trainer.py:803] 2025-04-26 16:18:28,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:28,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:28,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3729
3693
3744
[WARNING|trainer.py:803] 2025-04-26 16:18:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:29,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:30,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3694
3730
3745
[WARNING|trainer.py:803] 2025-04-26 16:18:31,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:31,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:18:31,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3695
3731
3746
[WARNING|trainer.py:803] 2025-04-26 16:18:32,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:32,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:32,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3696
3732
3747
[WARNING|trainer.py:803] 2025-04-26 16:18:34,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:34,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:34,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3697
3733
3748
[WARNING|trainer.py:803] 2025-04-26 16:18:35,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:35,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:35,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3734
3698
3749
[WARNING|trainer.py:803] 2025-04-26 16:18:36,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:36,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:37,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3699
3735
3750
[WARNING|trainer.py:803] 2025-04-26 16:18:38,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:38,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:38,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3700
3736
3751
[WARNING|trainer.py:803] 2025-04-26 16:18:39,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:39,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:39,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3701
3737
3752
[WARNING|trainer.py:803] 2025-04-26 16:18:40,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:18:41,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:41,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3702
3738
3753
[WARNING|trainer.py:803] 2025-04-26 16:18:42,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:42,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:42,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3703
3739
3754
[WARNING|trainer.py:803] 2025-04-26 16:18:43,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:43,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:43,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3755
3704
3740
[WARNING|trainer.py:803] 2025-04-26 16:18:45,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:45,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:45,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3756
3705
3741
[WARNING|trainer.py:803] 2025-04-26 16:18:46,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:46,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:18:46,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3757
3742
3706
[WARNING|trainer.py:803] 2025-04-26 16:18:48,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:48,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:48,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3707
3743
3758
[WARNING|trainer.py:803] 2025-04-26 16:18:49,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:49,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:49,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3759
3708
3744
[WARNING|trainer.py:803] 2025-04-26 16:18:51,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:51,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:51,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3709
3745
3760
[WARNING|trainer.py:803] 2025-04-26 16:18:52,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:52,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:52,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3746
3761
3710
[WARNING|trainer.py:803] 2025-04-26 16:18:53,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:18:53,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:54,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3747
3762
3711
[WARNING|trainer.py:803] 2025-04-26 16:18:55,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:55,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:55,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3748
3763
3712
[WARNING|trainer.py:803] 2025-04-26 16:18:56,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:56,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:18:56,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3749
3764
3713
[WARNING|trainer.py:803] 2025-04-26 16:18:57,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:58,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:18:58,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3750
3765
3714
[WARNING|trainer.py:803] 2025-04-26 16:18:59,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:59,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:18:59,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3751
3766
3715
[WARNING|trainer.py:803] 2025-04-26 16:19:00,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:00,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:01,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3767
3752
3716
[WARNING|trainer.py:803] 2025-04-26 16:19:02,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:02,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:02,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3753
3768
3717
[WARNING|trainer.py:803] 2025-04-26 16:19:03,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:03,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:03,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3769
3754
3718
[WARNING|trainer.py:803] 2025-04-26 16:19:04,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:04,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:05,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3770
3755
3719
[WARNING|trainer.py:803] 2025-04-26 16:19:06,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:06,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:06,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3771
3756
3720
[WARNING|trainer.py:803] 2025-04-26 16:19:07,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:07,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3772
3757
[WARNING|trainer.py:803] 2025-04-26 16:19:08,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3721
[WARNING|trainer.py:803] 2025-04-26 16:19:08,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:08,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3773
3758
[WARNING|trainer.py:803] 2025-04-26 16:19:09,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:10,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3722
[WARNING|trainer.py:803] 2025-04-26 16:19:10,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3774
3759
[WARNING|trainer.py:803] 2025-04-26 16:19:11,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:11,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3723
[WARNING|trainer.py:803] 2025-04-26 16:19:11,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3775
[WARNING|trainer.py:803] 2025-04-26 16:19:12,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3760
[WARNING|trainer.py:803] 2025-04-26 16:19:12,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3724
[WARNING|trainer.py:803] 2025-04-26 16:19:13,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3776
3761
[WARNING|trainer.py:803] 2025-04-26 16:19:14,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:14,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3725
[WARNING|trainer.py:803] 2025-04-26 16:19:14,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3777
3762
[WARNING|trainer.py:803] 2025-04-26 16:19:15,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:15,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3726
[WARNING|trainer.py:803] 2025-04-26 16:19:16,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3778
[WARNING|trainer.py:803] 2025-04-26 16:19:16,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3763
[WARNING|trainer.py:803] 2025-04-26 16:19:17,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3727
[WARNING|trainer.py:803] 2025-04-26 16:19:17,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3779
[WARNING|trainer.py:803] 2025-04-26 16:19:18,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3764
[WARNING|trainer.py:803] 2025-04-26 16:19:18,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3728
[WARNING|trainer.py:803] 2025-04-26 16:19:19,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3780
[WARNING|trainer.py:803] 2025-04-26 16:19:19,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3765
3729
[WARNING|trainer.py:803] 2025-04-26 16:19:20,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:20,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3781
[WARNING|trainer.py:803] 2025-04-26 16:19:20,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3766
[WARNING|trainer.py:803] 2025-04-26 16:19:21,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3730
[WARNING|trainer.py:803] 2025-04-26 16:19:21,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3782
3767
[WARNING|trainer.py:803] 2025-04-26 16:19:22,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3731
[WARNING|trainer.py:803] 2025-04-26 16:19:22,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:23,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3783
[WARNING|trainer.py:803] 2025-04-26 16:19:23,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3768
3732
[WARNING|trainer.py:803] 2025-04-26 16:19:24,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:24,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:24,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3769
3784
3733
[WARNING|trainer.py:803] 2025-04-26 16:19:25,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:25,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3770
[WARNING|trainer.py:803] 2025-04-26 16:19:26,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3785
3734
[WARNING|trainer.py:803] 2025-04-26 16:19:27,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:27,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:27,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3771
3786
3735
[WARNING|trainer.py:803] 2025-04-26 16:19:28,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:28,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3772
[WARNING|trainer.py:803] 2025-04-26 16:19:29,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3787
3736
[WARNING|trainer.py:803] 2025-04-26 16:19:29,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:29,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3773
[WARNING|trainer.py:803] 2025-04-26 16:19:30,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3788
3737
[WARNING|trainer.py:803] 2025-04-26 16:19:31,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:31,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3774
[WARNING|trainer.py:803] 2025-04-26 16:19:31,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3789
3738
[WARNING|trainer.py:803] 2025-04-26 16:19:32,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:32,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3775
[WARNING|trainer.py:803] 2025-04-26 16:19:33,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3790
3739
[WARNING|trainer.py:803] 2025-04-26 16:19:33,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:34,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3776
[WARNING|trainer.py:803] 2025-04-26 16:19:34,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3791
3740
[WARNING|trainer.py:803] 2025-04-26 16:19:35,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:35,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3777
[WARNING|trainer.py:803] 2025-04-26 16:19:35,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3792
3741
[WARNING|trainer.py:803] 2025-04-26 16:19:36,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:36,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3778
[WARNING|trainer.py:803] 2025-04-26 16:19:37,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3793
[WARNING|trainer.py:803] 2025-04-26 16:19:37,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3742
[WARNING|trainer.py:803] 2025-04-26 16:19:38,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3779
[WARNING|trainer.py:803] 2025-04-26 16:19:38,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3794
[WARNING|trainer.py:803] 2025-04-26 16:19:39,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3743
[WARNING|trainer.py:803] 2025-04-26 16:19:39,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3780
3795
[WARNING|trainer.py:803] 2025-04-26 16:19:40,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:40,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3744
[WARNING|trainer.py:803] 2025-04-26 16:19:41,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3781
3796
[WARNING|trainer.py:803] 2025-04-26 16:19:42,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:42,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:42,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3745
3782
3797
[WARNING|trainer.py:803] 2025-04-26 16:19:43,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:43,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:43,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3746
3783
3798
[WARNING|trainer.py:803] 2025-04-26 16:19:44,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:45,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3747
[WARNING|trainer.py:803] 2025-04-26 16:19:45,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3784
3799
[WARNING|trainer.py:803] 2025-04-26 16:19:46,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:46,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:46,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3748
3785
3800
[WARNING|trainer.py:803] 2025-04-26 16:19:47,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:47,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:47,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3749
3786
3801
[WARNING|trainer.py:803] 2025-04-26 16:19:48,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:49,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:49,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3750
3787
3802
[WARNING|trainer.py:803] 2025-04-26 16:19:50,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:50,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:19:50,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3751
3803
3788
[WARNING|trainer.py:803] 2025-04-26 16:19:51,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:51,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:52,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3752
3804
3789
[WARNING|trainer.py:803] 2025-04-26 16:19:53,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:53,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:53,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3753
3805
3790
[WARNING|trainer.py:803] 2025-04-26 16:19:54,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:19:54,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:54,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3806
3754
3791
[WARNING|trainer.py:803] 2025-04-26 16:19:55,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:55,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:19:56,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3807
3755
3792
[WARNING|trainer.py:803] 2025-04-26 16:19:57,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:57,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3808
[WARNING|trainer.py:803] 2025-04-26 16:19:57,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3756
3793
[WARNING|trainer.py:803] 2025-04-26 16:19:58,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:58,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3809
[WARNING|trainer.py:803] 2025-04-26 16:19:58,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3757
3794
[WARNING|trainer.py:803] 2025-04-26 16:19:59,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:19:59,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3810
[WARNING|trainer.py:803] 2025-04-26 16:20:00,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3758
3795
[WARNING|trainer.py:803] 2025-04-26 16:20:00,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3811
[WARNING|trainer.py:803] 2025-04-26 16:20:01,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:01,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3759
[WARNING|trainer.py:803] 2025-04-26 16:20:02,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3796
3812
[WARNING|trainer.py:803] 2025-04-26 16:20:02,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:03,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3760
[WARNING|trainer.py:803] 2025-04-26 16:20:03,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3797
3813
[WARNING|trainer.py:803] 2025-04-26 16:20:04,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:04,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:04,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3761
3798
3814
[WARNING|trainer.py:803] 2025-04-26 16:20:05,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:05,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:06,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3762
3799
3815
[WARNING|trainer.py:803] 2025-04-26 16:20:07,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:07,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:07,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3763
3800
3816
[WARNING|trainer.py:803] 2025-04-26 16:20:08,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:08,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:08,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3817
3764
3801
[WARNING|trainer.py:803] 2025-04-26 16:20:09,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:10,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:10,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3818
3765
3802
[WARNING|trainer.py:803] 2025-04-26 16:20:11,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:11,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:11,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3819
3766
3803
[WARNING|trainer.py:803] 2025-04-26 16:20:12,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:12,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:12,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3820
3804
3767
[WARNING|trainer.py:803] 2025-04-26 16:20:14,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:14,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:14,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3821
3805
3768
[WARNING|trainer.py:803] 2025-04-26 16:20:15,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:15,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:15,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3822
3806
3769
[WARNING|trainer.py:803] 2025-04-26 16:20:16,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:16,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:16,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3823
3807
3770
[WARNING|trainer.py:803] 2025-04-26 16:20:18,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:18,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:18,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3824
3808
3771
[WARNING|trainer.py:803] 2025-04-26 16:20:19,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:19,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:19,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3825
3809
3772
[WARNING|trainer.py:803] 2025-04-26 16:20:20,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:20,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:21,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3826
3810
3773
[WARNING|trainer.py:803] 2025-04-26 16:20:21,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:21,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3827
[WARNING|trainer.py:803] 2025-04-26 16:20:22,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3811
3774
[WARNING|trainer.py:803] 2025-04-26 16:20:23,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:23,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3828
3812
[WARNING|trainer.py:803] 2025-04-26 16:20:23,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3775
[WARNING|trainer.py:803] 2025-04-26 16:20:24,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:24,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3829
3813
[WARNING|trainer.py:803] 2025-04-26 16:20:25,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:25,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3776
[WARNING|trainer.py:803] 2025-04-26 16:20:25,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3830
3814
[WARNING|trainer.py:803] 2025-04-26 16:20:26,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:26,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3777
[WARNING|trainer.py:803] 2025-04-26 16:20:27,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3831
3815
[WARNING|trainer.py:803] 2025-04-26 16:20:27,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:28,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:28,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3778
3832
3816
[WARNING|trainer.py:803] 2025-04-26 16:20:29,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:29,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:29,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3833
3779
3817
[WARNING|trainer.py:803] 2025-04-26 16:20:30,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:30,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:31,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3834
3780
3818
[WARNING|trainer.py:803] 2025-04-26 16:20:31,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:32,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3835
[WARNING|trainer.py:803] 2025-04-26 16:20:32,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3781
3819
[WARNING|trainer.py:803] 2025-04-26 16:20:33,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:33,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3836
[WARNING|trainer.py:803] 2025-04-26 16:20:33,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3782
3820
[WARNING|trainer.py:803] 2025-04-26 16:20:34,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3837
[WARNING|trainer.py:803] 2025-04-26 16:20:35,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:35,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3821
3783
[WARNING|trainer.py:803] 2025-04-26 16:20:35,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3838
[WARNING|trainer.py:803] 2025-04-26 16:20:36,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:36,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3822
[WARNING|trainer.py:803] 2025-04-26 16:20:37,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3784
3839
[WARNING|trainer.py:803] 2025-04-26 16:20:37,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:38,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:38,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3823
3785
3840
[WARNING|trainer.py:803] 2025-04-26 16:20:39,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:39,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:39,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3824
3786
3841
[WARNING|trainer.py:803] 2025-04-26 16:20:40,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:40,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3825
[WARNING|trainer.py:803] 2025-04-26 16:20:40,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3842
3787
[WARNING|trainer.py:803] 2025-04-26 16:20:41,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:42,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3826
[WARNING|trainer.py:803] 2025-04-26 16:20:42,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3843
3788
[WARNING|trainer.py:803] 2025-04-26 16:20:43,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3827
[WARNING|trainer.py:803] 2025-04-26 16:20:43,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:43,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3844
[WARNING|trainer.py:803] 2025-04-26 16:20:44,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3789
3828
[WARNING|trainer.py:803] 2025-04-26 16:20:44,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:45,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3845
[WARNING|trainer.py:803] 2025-04-26 16:20:45,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3790
3829
[WARNING|trainer.py:803] 2025-04-26 16:20:46,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:46,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3846
[WARNING|trainer.py:803] 2025-04-26 16:20:46,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3791
3830
[WARNING|trainer.py:803] 2025-04-26 16:20:47,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3847
[WARNING|trainer.py:803] 2025-04-26 16:20:47,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:20:48,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3792
3831
[WARNING|trainer.py:803] 2025-04-26 16:20:48,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3848
[WARNING|trainer.py:803] 2025-04-26 16:20:49,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:49,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3793
3832
[WARNING|trainer.py:803] 2025-04-26 16:20:49,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3849
[WARNING|trainer.py:803] 2025-04-26 16:20:50,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:20:50,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3833
3794
[WARNING|trainer.py:803] 2025-04-26 16:20:51,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3850
[WARNING|trainer.py:803] 2025-04-26 16:20:51,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:52,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3834
[WARNING|trainer.py:803] 2025-04-26 16:20:52,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3795
3851
[WARNING|trainer.py:803] 2025-04-26 16:20:53,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:53,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3835
[WARNING|trainer.py:803] 2025-04-26 16:20:53,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3796
3852
[WARNING|trainer.py:803] 2025-04-26 16:20:54,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:20:54,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3836
[WARNING|trainer.py:803] 2025-04-26 16:20:55,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3797
3853
[WARNING|trainer.py:803] 2025-04-26 16:20:55,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:56,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3837
[WARNING|trainer.py:803] 2025-04-26 16:20:56,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3854
3798
[WARNING|trainer.py:803] 2025-04-26 16:20:57,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:57,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3838
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:20:57,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3855
3799
[WARNING|trainer.py:803] 2025-04-26 16:20:58,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:20:58,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3839
[WARNING|trainer.py:803] 2025-04-26 16:20:58,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3856
3800
[WARNING|trainer.py:803] 2025-04-26 16:20:59,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:00,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3840
[WARNING|trainer.py:803] 2025-04-26 16:21:00,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3857
3801
[WARNING|trainer.py:803] 2025-04-26 16:21:00,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:01,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3841
[WARNING|trainer.py:803] 2025-04-26 16:21:01,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3858
3802
[WARNING|trainer.py:803] 2025-04-26 16:21:02,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:02,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3842
[WARNING|trainer.py:803] 2025-04-26 16:21:03,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3859
3803
[WARNING|trainer.py:803] 2025-04-26 16:21:03,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:04,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3843
[WARNING|trainer.py:803] 2025-04-26 16:21:04,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3860
3804
[WARNING|trainer.py:803] 2025-04-26 16:21:05,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:05,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3844
[WARNING|trainer.py:803] 2025-04-26 16:21:05,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3861
3805
[WARNING|trainer.py:803] 2025-04-26 16:21:06,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:06,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3845
[WARNING|trainer.py:803] 2025-04-26 16:21:06,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3862
3806
[WARNING|trainer.py:803] 2025-04-26 16:21:07,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:07,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3846
[WARNING|trainer.py:803] 2025-04-26 16:21:08,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3863
3807
[WARNING|trainer.py:803] 2025-04-26 16:21:08,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:09,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3847
[WARNING|trainer.py:803] 2025-04-26 16:21:09,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3864
[WARNING|trainer.py:803] 2025-04-26 16:21:10,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3808
[WARNING|trainer.py:803] 2025-04-26 16:21:10,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3848
[WARNING|trainer.py:803] 2025-04-26 16:21:10,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3865
[WARNING|trainer.py:803] 2025-04-26 16:21:11,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3809
3849
[WARNING|trainer.py:803] 2025-04-26 16:21:11,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:12,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3866
[WARNING|trainer.py:803] 2025-04-26 16:21:12,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3810
[WARNING|trainer.py:803] 2025-04-26 16:21:13,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3850
[WARNING|trainer.py:803] 2025-04-26 16:21:13,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3867
3811
[WARNING|trainer.py:803] 2025-04-26 16:21:13,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3851
[WARNING|trainer.py:803] 2025-04-26 16:21:14,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:14,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3868
[WARNING|trainer.py:803] 2025-04-26 16:21:15,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3812
[WARNING|trainer.py:803] 2025-04-26 16:21:15,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3852
[WARNING|trainer.py:803] 2025-04-26 16:21:16,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3869
3813
[WARNING|trainer.py:803] 2025-04-26 16:21:16,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:16,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3853
[WARNING|trainer.py:803] 2025-04-26 16:21:17,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3870
[WARNING|trainer.py:803] 2025-04-26 16:21:17,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3814
[WARNING|trainer.py:803] 2025-04-26 16:21:18,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3854
[WARNING|trainer.py:803] 2025-04-26 16:21:18,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3871
[WARNING|trainer.py:803] 2025-04-26 16:21:19,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3815
3855
[WARNING|trainer.py:803] 2025-04-26 16:21:19,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:19,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3872
[WARNING|trainer.py:803] 2025-04-26 16:21:20,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3816
3856
[WARNING|trainer.py:803] 2025-04-26 16:21:20,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:21,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3873
[WARNING|trainer.py:803] 2025-04-26 16:21:21,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3817
3857
[WARNING|trainer.py:803] 2025-04-26 16:21:22,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:22,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3874
[WARNING|trainer.py:803] 2025-04-26 16:21:22,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3818
3858
[WARNING|trainer.py:803] 2025-04-26 16:21:23,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:24,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:24,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3875
3819
3859
[WARNING|trainer.py:803] 2025-04-26 16:21:25,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:25,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:25,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3876
3860
3820
[WARNING|trainer.py:803] 2025-04-26 16:21:26,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:26,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:26,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3877
3821
3861
[WARNING|trainer.py:803] 2025-04-26 16:21:27,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:28,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:28,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3878
3822
3862
[WARNING|trainer.py:803] 2025-04-26 16:21:29,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:29,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:29,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3879
3863
3823
[WARNING|trainer.py:803] 2025-04-26 16:21:30,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:30,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:30,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3880
3864
3824
[WARNING|trainer.py:803] 2025-04-26 16:21:31,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:32,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:32,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3881
3865
3825
[WARNING|trainer.py:803] 2025-04-26 16:21:33,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:33,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:33,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3882
3866
3826
[WARNING|trainer.py:803] 2025-04-26 16:21:34,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:34,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:34,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3883
3867
3827
[WARNING|trainer.py:803] 2025-04-26 16:21:35,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:35,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:36,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3884
3868
3828
[WARNING|trainer.py:803] 2025-04-26 16:21:37,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:37,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:37,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3885
3869
3829
[WARNING|trainer.py:803] 2025-04-26 16:21:38,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:38,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:38,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3886
3870
3830
[WARNING|trainer.py:803] 2025-04-26 16:21:39,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:39,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:39,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3831
3871
3887
[WARNING|trainer.py:803] 2025-04-26 16:21:41,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:41,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:41,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3832
3888
3872
[WARNING|trainer.py:803] 2025-04-26 16:21:42,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:42,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:42,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3889
3833
3873
[WARNING|trainer.py:803] 2025-04-26 16:21:43,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:43,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:44,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3834
3890
3874
[WARNING|trainer.py:803] 2025-04-26 16:21:45,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:45,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:45,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3835
3891
3875
[WARNING|trainer.py:803] 2025-04-26 16:21:46,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:21:46,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3836
[WARNING|trainer.py:803] 2025-04-26 16:21:46,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3892
3876
[WARNING|trainer.py:803] 2025-04-26 16:21:47,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:48,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3837
[WARNING|trainer.py:803] 2025-04-26 16:21:48,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3893
3877
[WARNING|trainer.py:803] 2025-04-26 16:21:49,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:49,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3838
[WARNING|trainer.py:803] 2025-04-26 16:21:49,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3894
3878
[WARNING|trainer.py:803] 2025-04-26 16:21:50,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:50,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3839
[WARNING|trainer.py:803] 2025-04-26 16:21:51,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3895
3879
[WARNING|trainer.py:803] 2025-04-26 16:21:51,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:52,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3840
[WARNING|trainer.py:803] 2025-04-26 16:21:52,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3896
3880
[WARNING|trainer.py:803] 2025-04-26 16:21:52,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:21:53,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3841
[WARNING|trainer.py:803] 2025-04-26 16:21:53,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3897
3881
[WARNING|trainer.py:803] 2025-04-26 16:21:54,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3842
[WARNING|trainer.py:803] 2025-04-26 16:21:54,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:55,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3898
[WARNING|trainer.py:803] 2025-04-26 16:21:55,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3882
3843
[WARNING|trainer.py:803] 2025-04-26 16:21:56,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:56,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3899
[WARNING|trainer.py:803] 2025-04-26 16:21:56,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3883
3844
[WARNING|trainer.py:803] 2025-04-26 16:21:57,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:57,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3900
[WARNING|trainer.py:803] 2025-04-26 16:21:58,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3884
3845
[WARNING|trainer.py:803] 2025-04-26 16:21:58,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:21:59,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:21:59,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3901
3885
3846
[WARNING|trainer.py:803] 2025-04-26 16:22:00,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:00,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:00,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3902
3886
3847
[WARNING|trainer.py:803] 2025-04-26 16:22:01,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:01,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:02,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3887
3848
3903
[WARNING|trainer.py:803] 2025-04-26 16:22:03,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:03,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:03,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3888
3849
3904
[WARNING|trainer.py:803] 2025-04-26 16:22:04,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:04,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:04,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3889
3850
3905
[WARNING|trainer.py:803] 2025-04-26 16:22:05,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:05,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3851
[WARNING|trainer.py:803] 2025-04-26 16:22:06,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3890
3906
[WARNING|trainer.py:803] 2025-04-26 16:22:07,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:07,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3852
3891
[WARNING|trainer.py:803] 2025-04-26 16:22:07,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:08,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3907
[WARNING|trainer.py:803] 2025-04-26 16:22:08,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3853
3892
[WARNING|trainer.py:803] 2025-04-26 16:22:09,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:09,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:10,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3854
3908
3893
[WARNING|trainer.py:803] 2025-04-26 16:22:10,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:10,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3855
[WARNING|trainer.py:803] 2025-04-26 16:22:11,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3909
3894
[WARNING|trainer.py:803] 2025-04-26 16:22:12,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3856
[WARNING|trainer.py:803] 2025-04-26 16:22:12,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:12,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3895
3910
[WARNING|trainer.py:803] 2025-04-26 16:22:13,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3857
[WARNING|trainer.py:803] 2025-04-26 16:22:14,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:14,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3896
[WARNING|trainer.py:803] 2025-04-26 16:22:14,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3911
3858
[WARNING|trainer.py:803] 2025-04-26 16:22:15,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:15,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3897
[WARNING|trainer.py:803] 2025-04-26 16:22:15,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3912
3859
[WARNING|trainer.py:803] 2025-04-26 16:22:16,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:17,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3898
[WARNING|trainer.py:803] 2025-04-26 16:22:17,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3860
3913
[WARNING|trainer.py:803] 2025-04-26 16:22:18,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:18,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3899
[WARNING|trainer.py:803] 2025-04-26 16:22:18,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3861
[WARNING|trainer.py:803] 2025-04-26 16:22:19,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3914
[WARNING|trainer.py:803] 2025-04-26 16:22:19,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3900
3862
[WARNING|trainer.py:803] 2025-04-26 16:22:20,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:20,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3915
[WARNING|trainer.py:803] 2025-04-26 16:22:21,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3901
3863
[WARNING|trainer.py:803] 2025-04-26 16:22:22,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:22,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:22,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3902
3864
3916
[WARNING|trainer.py:803] 2025-04-26 16:22:23,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:23,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:22:23,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3865
3917
3903
[WARNING|trainer.py:803] 2025-04-26 16:22:25,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:22:25,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:25,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3866
3904
3918
[WARNING|trainer.py:803] 2025-04-26 16:22:26,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3867
[WARNING|trainer.py:803] 2025-04-26 16:22:26,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:26,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3919
3905
[WARNING|trainer.py:803] 2025-04-26 16:22:27,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3868
[WARNING|trainer.py:803] 2025-04-26 16:22:28,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:28,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:28,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3906
3920
3869
[WARNING|trainer.py:803] 2025-04-26 16:22:29,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:29,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:30,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3907
3921
3870
[WARNING|trainer.py:803] 2025-04-26 16:22:31,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:31,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:31,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3908
3922
3871
[WARNING|trainer.py:803] 2025-04-26 16:22:33,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:33,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:33,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3872
3923
3909
[WARNING|trainer.py:803] 2025-04-26 16:22:34,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:34,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:34,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3873
3910
3924
[WARNING|trainer.py:803] 2025-04-26 16:22:35,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:36,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:36,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3874
3911
3925
[WARNING|trainer.py:803] 2025-04-26 16:22:37,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:37,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:37,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3875
3912
3926
[WARNING|trainer.py:803] 2025-04-26 16:22:38,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3876
[WARNING|trainer.py:803] 2025-04-26 16:22:39,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:39,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3913
[WARNING|trainer.py:803] 2025-04-26 16:22:40,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3927
3877
[WARNING|trainer.py:803] 2025-04-26 16:22:40,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:40,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:41,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3914
3928
3878
[WARNING|trainer.py:803] 2025-04-26 16:22:42,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:42,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:42,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3915
3879
3929
[WARNING|trainer.py:803] 2025-04-26 16:22:43,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:44,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:44,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3880
3916
3930
[WARNING|trainer.py:803] 2025-04-26 16:22:45,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:22:45,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:45,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3881
3917
3931
[WARNING|trainer.py:803] 2025-04-26 16:22:46,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:46,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:47,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3882
3918
3932
[WARNING|trainer.py:803] 2025-04-26 16:22:48,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:48,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3883
[WARNING|trainer.py:803] 2025-04-26 16:22:48,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3919
3933
[WARNING|trainer.py:803] 2025-04-26 16:22:49,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:50,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3884
[WARNING|trainer.py:803] 2025-04-26 16:22:50,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3920
[WARNING|trainer.py:803] 2025-04-26 16:22:50,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3934
3885
[WARNING|trainer.py:803] 2025-04-26 16:22:51,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:52,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:52,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3921
3886
3935
[WARNING|trainer.py:803] 2025-04-26 16:22:53,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:22:53,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:53,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3922
3887
3936
[WARNING|trainer.py:803] 2025-04-26 16:22:54,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:55,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:55,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3923
3888
3937
[WARNING|trainer.py:803] 2025-04-26 16:22:56,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:56,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:56,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3889
3924
3938
[WARNING|trainer.py:803] 2025-04-26 16:22:57,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:22:58,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:58,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3890
3925
3939
[WARNING|trainer.py:803] 2025-04-26 16:22:59,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:22:59,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3891
[WARNING|trainer.py:803] 2025-04-26 16:22:59,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3926
3940
[WARNING|trainer.py:803] 2025-04-26 16:23:00,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3892
[WARNING|trainer.py:803] 2025-04-26 16:23:01,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:01,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:01,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3927
3941
3893
[WARNING|trainer.py:803] 2025-04-26 16:23:02,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:02,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:03,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3928
3942
3894
[WARNING|trainer.py:803] 2025-04-26 16:23:04,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:04,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:04,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3929
3895
3943
[WARNING|trainer.py:803] 2025-04-26 16:23:05,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:05,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:06,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3896
3930
3944
[WARNING|trainer.py:803] 2025-04-26 16:23:07,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:07,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:23:07,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3897
3931
3945
[WARNING|trainer.py:803] 2025-04-26 16:23:08,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:08,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3898
[WARNING|trainer.py:803] 2025-04-26 16:23:09,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3932
[WARNING|trainer.py:803] 2025-04-26 16:23:09,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3946
3899
[WARNING|trainer.py:803] 2025-04-26 16:23:10,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:10,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3933
[WARNING|trainer.py:803] 2025-04-26 16:23:11,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3947
3900
[WARNING|trainer.py:803] 2025-04-26 16:23:12,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:12,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:12,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3934
3948
3901
[WARNING|trainer.py:803] 2025-04-26 16:23:13,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:13,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:14,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3935
3949
3902
[WARNING|trainer.py:803] 2025-04-26 16:23:15,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:15,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:15,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3936
3950
3903
[WARNING|trainer.py:803] 2025-04-26 16:23:16,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:16,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:17,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3937
3951
3904
[WARNING|trainer.py:803] 2025-04-26 16:23:18,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:18,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:18,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3938
3952
3905
[WARNING|trainer.py:803] 2025-04-26 16:23:19,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:20,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:20,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3939
3953
3906
[WARNING|trainer.py:803] 2025-04-26 16:23:21,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:21,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:21,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3940
3954
3907
[WARNING|trainer.py:803] 2025-04-26 16:23:23,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:23,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:23,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3941
3955
3908
[WARNING|trainer.py:803] 2025-04-26 16:23:24,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:24,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:24,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3942
3956
3909
[WARNING|trainer.py:803] 2025-04-26 16:23:26,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:26,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:26,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3957
3943
3910
[WARNING|trainer.py:803] 2025-04-26 16:23:27,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:27,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:28,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3958
3944
3911
[WARNING|trainer.py:803] 2025-04-26 16:23:29,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:23:29,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:29,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3959
3945
3912
[WARNING|trainer.py:803] 2025-04-26 16:23:30,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:31,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:31,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3960
3913
3946
[WARNING|trainer.py:803] 2025-04-26 16:23:32,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:23:32,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:32,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3961
3914
3947
[WARNING|trainer.py:803] 2025-04-26 16:23:33,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:34,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:34,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3962
3915
3948
[WARNING|trainer.py:803] 2025-04-26 16:23:35,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:35,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3963
[WARNING|trainer.py:803] 2025-04-26 16:23:35,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3916
3949
[WARNING|trainer.py:803] 2025-04-26 16:23:36,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3964
[WARNING|trainer.py:803] 2025-04-26 16:23:37,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:37,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3917
[WARNING|trainer.py:803] 2025-04-26 16:23:38,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3950
3965
[WARNING|trainer.py:803] 2025-04-26 16:23:38,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:38,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3951
3918
[WARNING|trainer.py:803] 2025-04-26 16:23:39,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3966
[WARNING|trainer.py:803] 2025-04-26 16:23:40,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:40,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:41,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3919
3952
3967
[WARNING|trainer.py:803] 2025-04-26 16:23:42,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:42,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:23:42,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3953
3920
3968
[WARNING|trainer.py:803] 2025-04-26 16:23:43,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:43,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3921
[WARNING|trainer.py:803] 2025-04-26 16:23:44,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3954
3969
[WARNING|trainer.py:803] 2025-04-26 16:23:45,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:45,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3922
3955
[WARNING|trainer.py:803] 2025-04-26 16:23:45,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3970
[WARNING|trainer.py:803] 2025-04-26 16:23:46,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:23:46,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3956
[WARNING|trainer.py:803] 2025-04-26 16:23:47,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3923
[WARNING|trainer.py:803] 2025-04-26 16:23:48,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3971
[WARNING|trainer.py:803] 2025-04-26 16:23:48,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3957
[WARNING|trainer.py:803] 2025-04-26 16:23:48,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3924
3972
[WARNING|trainer.py:803] 2025-04-26 16:23:49,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:50,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3958
[WARNING|trainer.py:803] 2025-04-26 16:23:50,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3925
[WARNING|trainer.py:803] 2025-04-26 16:23:51,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3973
[WARNING|trainer.py:803] 2025-04-26 16:23:51,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3959
[WARNING|trainer.py:803] 2025-04-26 16:23:51,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3926
3974
[WARNING|trainer.py:803] 2025-04-26 16:23:52,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:23:53,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3960
[WARNING|trainer.py:803] 2025-04-26 16:23:53,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3927
[WARNING|trainer.py:803] 2025-04-26 16:23:54,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3975
[WARNING|trainer.py:803] 2025-04-26 16:23:54,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3961
[WARNING|trainer.py:803] 2025-04-26 16:23:55,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3928
[WARNING|trainer.py:803] 2025-04-26 16:23:55,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3976
[WARNING|trainer.py:803] 2025-04-26 16:23:56,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3962
[WARNING|trainer.py:803] 2025-04-26 16:23:56,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3929
[WARNING|trainer.py:803] 2025-04-26 16:23:57,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3977
[WARNING|trainer.py:803] 2025-04-26 16:23:57,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3963
[WARNING|trainer.py:803] 2025-04-26 16:23:58,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3930
[WARNING|trainer.py:803] 2025-04-26 16:23:58,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3978
[WARNING|trainer.py:803] 2025-04-26 16:23:59,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3964
[WARNING|trainer.py:803] 2025-04-26 16:23:59,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3931
[WARNING|trainer.py:803] 2025-04-26 16:24:00,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3979
[WARNING|trainer.py:803] 2025-04-26 16:24:00,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3965
[WARNING|trainer.py:803] 2025-04-26 16:24:01,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3932
[WARNING|trainer.py:803] 2025-04-26 16:24:01,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3980
3966
[WARNING|trainer.py:803] 2025-04-26 16:24:02,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:02,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3933
[WARNING|trainer.py:803] 2025-04-26 16:24:03,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3981
3967
[WARNING|trainer.py:803] 2025-04-26 16:24:04,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:04,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:04,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3982
3934
3968
[WARNING|trainer.py:803] 2025-04-26 16:24:05,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:24:05,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:06,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3935
3983
3969
[WARNING|trainer.py:803] 2025-04-26 16:24:07,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:07,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:07,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3936
3984
3970
[WARNING|trainer.py:803] 2025-04-26 16:24:08,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:08,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:09,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3937
3985
3971
[WARNING|trainer.py:803] 2025-04-26 16:24:10,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:10,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:10,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3938
3986
3972
[WARNING|trainer.py:803] 2025-04-26 16:24:11,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:24:11,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:12,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3939
3987
3973
[WARNING|trainer.py:803] 2025-04-26 16:24:13,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:24:13,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3988
[WARNING|trainer.py:803] 2025-04-26 16:24:14,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3940
3974
[WARNING|trainer.py:803] 2025-04-26 16:24:14,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:15,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3989
[WARNING|trainer.py:803] 2025-04-26 16:24:15,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3941
[WARNING|trainer.py:803] 2025-04-26 16:24:16,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3975
[WARNING|trainer.py:803] 2025-04-26 16:24:16,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3990
[WARNING|trainer.py:803] 2025-04-26 16:24:17,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3942
[WARNING|trainer.py:803] 2025-04-26 16:24:17,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3976
[WARNING|trainer.py:803] 2025-04-26 16:24:18,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3991
[WARNING|trainer.py:803] 2025-04-26 16:24:18,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3943
[WARNING|trainer.py:803] 2025-04-26 16:24:19,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3977
[WARNING|trainer.py:803] 2025-04-26 16:24:19,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3992
[WARNING|trainer.py:803] 2025-04-26 16:24:20,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3944
3978
[WARNING|trainer.py:803] 2025-04-26 16:24:21,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:21,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3993
[WARNING|trainer.py:803] 2025-04-26 16:24:21,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3945
3979
[WARNING|trainer.py:803] 2025-04-26 16:24:22,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:23,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3994
[WARNING|trainer.py:803] 2025-04-26 16:24:23,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3946
[WARNING|trainer.py:803] 2025-04-26 16:24:23,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3980
[WARNING|trainer.py:803] 2025-04-26 16:24:24,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3995
[WARNING|trainer.py:803] 2025-04-26 16:24:24,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3947
3981
[WARNING|trainer.py:803] 2025-04-26 16:24:25,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:26,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3996
[WARNING|trainer.py:803] 2025-04-26 16:24:26,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3948
3982
[WARNING|trainer.py:803] 2025-04-26 16:24:27,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:27,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:27,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3997
3949
[WARNING|trainer.py:803] 2025-04-26 16:24:28,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3983
[WARNING|trainer.py:803] 2025-04-26 16:24:29,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3998
[WARNING|trainer.py:803] 2025-04-26 16:24:29,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3950
[WARNING|trainer.py:803] 2025-04-26 16:24:30,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3984
[WARNING|trainer.py:803] 2025-04-26 16:24:30,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3999
[WARNING|trainer.py:803] 2025-04-26 16:24:31,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3951
[WARNING|trainer.py:803] 2025-04-26 16:24:31,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3985
4000
[WARNING|trainer.py:803] 2025-04-26 16:24:32,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:32,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3952
[WARNING|trainer.py:803] 2025-04-26 16:24:33,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3986
[WARNING|trainer.py:803] 2025-04-26 16:24:33,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4001
[WARNING|trainer.py:803] 2025-04-26 16:24:34,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3953
[WARNING|trainer.py:803] 2025-04-26 16:24:34,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3987
[WARNING|trainer.py:803] 2025-04-26 16:24:35,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4002
[WARNING|trainer.py:803] 2025-04-26 16:24:35,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3954
[WARNING|trainer.py:803] 2025-04-26 16:24:36,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3988
[WARNING|trainer.py:803] 2025-04-26 16:24:36,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4003
[WARNING|trainer.py:803] 2025-04-26 16:24:37,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3955
[WARNING|trainer.py:803] 2025-04-26 16:24:37,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3989
[WARNING|trainer.py:803] 2025-04-26 16:24:38,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4004
[WARNING|trainer.py:803] 2025-04-26 16:24:38,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3956
3990
[WARNING|trainer.py:803] 2025-04-26 16:24:39,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:39,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4005
[WARNING|trainer.py:803] 2025-04-26 16:24:40,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3957
3991
[WARNING|trainer.py:803] 2025-04-26 16:24:41,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:41,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:41,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4006
3958
3992
[WARNING|trainer.py:803] 2025-04-26 16:24:42,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:42,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4007
[WARNING|trainer.py:803] 2025-04-26 16:24:43,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3959
3993
[WARNING|trainer.py:803] 2025-04-26 16:24:44,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:44,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:44,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4008
3960
3994
[WARNING|trainer.py:803] 2025-04-26 16:24:45,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:45,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:46,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4009
3961
3995
[WARNING|trainer.py:803] 2025-04-26 16:24:47,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:24:47,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4010
[WARNING|trainer.py:803] 2025-04-26 16:24:48,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3962
3996
[WARNING|trainer.py:803] 2025-04-26 16:24:48,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:48,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4011
3963
[WARNING|trainer.py:803] 2025-04-26 16:24:49,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3997
[WARNING|trainer.py:803] 2025-04-26 16:24:50,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:50,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4012
[WARNING|trainer.py:803] 2025-04-26 16:24:51,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3964
3998
[WARNING|trainer.py:803] 2025-04-26 16:24:51,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:51,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4013
[WARNING|trainer.py:803] 2025-04-26 16:24:52,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3965
3999
[WARNING|trainer.py:803] 2025-04-26 16:24:53,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:53,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4014
[WARNING|trainer.py:803] 2025-04-26 16:24:54,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3966
4000
[WARNING|trainer.py:803] 2025-04-26 16:24:54,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:55,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:55,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4015
3967
4001
[WARNING|trainer.py:803] 2025-04-26 16:24:56,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:56,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:24:57,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4016
3968
4002
[WARNING|trainer.py:803] 2025-04-26 16:24:58,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:58,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:24:58,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4017
3969
4003
[WARNING|trainer.py:803] 2025-04-26 16:24:59,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:24:59,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:00,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3970
4018
4004
[WARNING|trainer.py:803] 2025-04-26 16:25:01,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:01,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:01,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3971
4019
4005
[WARNING|trainer.py:803] 2025-04-26 16:25:02,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:02,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:03,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3972
4020
4006
[WARNING|trainer.py:803] 2025-04-26 16:25:04,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:04,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:04,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3973
4021
4007
[WARNING|trainer.py:803] 2025-04-26 16:25:05,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:06,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:06,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3974
4022
4008
[WARNING|trainer.py:803] 2025-04-26 16:25:07,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:07,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:08,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3975
4023
4009
[WARNING|trainer.py:803] 2025-04-26 16:25:09,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:09,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:09,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3976
4024
4010
[WARNING|trainer.py:803] 2025-04-26 16:25:10,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:10,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:11,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3977
4025
4011
[WARNING|trainer.py:803] 2025-04-26 16:25:12,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:12,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:12,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3978
4012
4026
[WARNING|trainer.py:803] 2025-04-26 16:25:13,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:14,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:14,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3979
4013
4027
[WARNING|trainer.py:803] 2025-04-26 16:25:15,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:15,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:15,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3980
4014
4028
[WARNING|trainer.py:803] 2025-04-26 16:25:16,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:17,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:17,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3981
4015
4029
[WARNING|trainer.py:803] 2025-04-26 16:25:18,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:18,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3982
[WARNING|trainer.py:803] 2025-04-26 16:25:18,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4016
4030
[WARNING|trainer.py:803] 2025-04-26 16:25:19,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:20,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:20,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3983
4017
4031
[WARNING|trainer.py:803] 2025-04-26 16:25:21,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:21,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:21,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3984
4032
4018
[WARNING|trainer.py:803] 2025-04-26 16:25:22,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:23,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3985
[WARNING|trainer.py:803] 2025-04-26 16:25:23,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4033
4019
[WARNING|trainer.py:803] 2025-04-26 16:25:24,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:24,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3986
[WARNING|trainer.py:803] 2025-04-26 16:25:25,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4034
4020
[WARNING|trainer.py:803] 2025-04-26 16:25:26,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:26,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3987
[WARNING|trainer.py:803] 2025-04-26 16:25:26,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4035
[WARNING|trainer.py:803] 2025-04-26 16:25:27,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4021
[WARNING|trainer.py:803] 2025-04-26 16:25:28,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3988
[WARNING|trainer.py:803] 2025-04-26 16:25:28,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4036
[WARNING|trainer.py:803] 2025-04-26 16:25:28,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4022
3989
[WARNING|trainer.py:803] 2025-04-26 16:25:29,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:30,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4037
[WARNING|trainer.py:803] 2025-04-26 16:25:30,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4023
3990
[WARNING|trainer.py:803] 2025-04-26 16:25:31,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:31,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:31,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4038
4024
3991
[WARNING|trainer.py:803] 2025-04-26 16:25:32,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:33,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:33,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4039
4025
3992
[WARNING|trainer.py:803] 2025-04-26 16:25:34,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:34,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4040
[WARNING|trainer.py:803] 2025-04-26 16:25:35,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3993
4026
[WARNING|trainer.py:803] 2025-04-26 16:25:35,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4041
[WARNING|trainer.py:803] 2025-04-26 16:25:36,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:36,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3994
[WARNING|trainer.py:803] 2025-04-26 16:25:37,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4027
4042
[WARNING|trainer.py:803] 2025-04-26 16:25:38,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:38,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:38,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3995
4028
4043
[WARNING|trainer.py:803] 2025-04-26 16:25:39,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:39,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3996
[WARNING|trainer.py:803] 2025-04-26 16:25:40,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4029
4044
[WARNING|trainer.py:803] 2025-04-26 16:25:41,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:41,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:41,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3997
4030
4045
[WARNING|trainer.py:803] 2025-04-26 16:25:42,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:42,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4031
[WARNING|trainer.py:803] 2025-04-26 16:25:43,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3998
4046
[WARNING|trainer.py:803] 2025-04-26 16:25:44,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:44,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4032
3999
[WARNING|trainer.py:803] 2025-04-26 16:25:45,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4047
[WARNING|trainer.py:803] 2025-04-26 16:25:45,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:45,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4033
4000
[WARNING|trainer.py:803] 2025-04-26 16:25:46,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:47,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:25:47,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4048
4034
4001
[WARNING|trainer.py:803] 2025-04-26 16:25:48,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:48,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4049
[WARNING|trainer.py:803] 2025-04-26 16:25:48,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4035
4002
[WARNING|trainer.py:803] 2025-04-26 16:25:49,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:50,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4050
[WARNING|trainer.py:803] 2025-04-26 16:25:50,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4003
4036
[WARNING|trainer.py:803] 2025-04-26 16:25:51,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4051
[WARNING|trainer.py:803] 2025-04-26 16:25:52,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:52,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4037
4004
[WARNING|trainer.py:803] 2025-04-26 16:25:52,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4052
[WARNING|trainer.py:803] 2025-04-26 16:25:53,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:53,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4005
4038
[WARNING|trainer.py:803] 2025-04-26 16:25:54,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4053
[WARNING|trainer.py:803] 2025-04-26 16:25:55,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:55,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4039
4006
[WARNING|trainer.py:803] 2025-04-26 16:25:55,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4054
[WARNING|trainer.py:803] 2025-04-26 16:25:56,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:25:56,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:25:57,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4040
4007
4055
[WARNING|trainer.py:803] 2025-04-26 16:25:58,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:58,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4041
[WARNING|trainer.py:803] 2025-04-26 16:25:58,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4008
4056
[WARNING|trainer.py:803] 2025-04-26 16:25:59,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:25:59,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4042
[WARNING|trainer.py:803] 2025-04-26 16:26:00,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4009
[WARNING|trainer.py:803] 2025-04-26 16:26:01,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4057
[WARNING|trainer.py:803] 2025-04-26 16:26:01,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:02,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4010
4043
4058
[WARNING|trainer.py:803] 2025-04-26 16:26:02,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:02,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4011
4044
[WARNING|trainer.py:803] 2025-04-26 16:26:03,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4059
[WARNING|trainer.py:803] 2025-04-26 16:26:04,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:04,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4012
[WARNING|trainer.py:803] 2025-04-26 16:26:05,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4045
[WARNING|trainer.py:803] 2025-04-26 16:26:05,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4060
[WARNING|trainer.py:803] 2025-04-26 16:26:06,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4013
4046
[WARNING|trainer.py:803] 2025-04-26 16:26:06,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4061
[WARNING|trainer.py:803] 2025-04-26 16:26:07,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:07,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4014
[WARNING|trainer.py:803] 2025-04-26 16:26:08,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4047
[WARNING|trainer.py:803] 2025-04-26 16:26:08,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4062
[WARNING|trainer.py:803] 2025-04-26 16:26:09,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4015
[WARNING|trainer.py:803] 2025-04-26 16:26:09,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4048
[WARNING|trainer.py:803] 2025-04-26 16:26:10,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4063
[WARNING|trainer.py:803] 2025-04-26 16:26:10,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4016
[WARNING|trainer.py:803] 2025-04-26 16:26:11,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4049
[WARNING|trainer.py:803] 2025-04-26 16:26:12,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4064
[WARNING|trainer.py:803] 2025-04-26 16:26:12,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4017
4050
[WARNING|trainer.py:803] 2025-04-26 16:26:13,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:13,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4065
[WARNING|trainer.py:803] 2025-04-26 16:26:13,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4018
4051
[WARNING|trainer.py:803] 2025-04-26 16:26:14,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4066
[WARNING|trainer.py:803] 2025-04-26 16:26:15,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:15,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4052
[WARNING|trainer.py:803] 2025-04-26 16:26:16,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4019
[WARNING|trainer.py:803] 2025-04-26 16:26:16,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4067
[WARNING|trainer.py:803] 2025-04-26 16:26:16,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4053
4020
[WARNING|trainer.py:803] 2025-04-26 16:26:17,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4068
[WARNING|trainer.py:803] 2025-04-26 16:26:18,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:18,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4054
[WARNING|trainer.py:803] 2025-04-26 16:26:19,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4021
[WARNING|trainer.py:803] 2025-04-26 16:26:19,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4069
[WARNING|trainer.py:803] 2025-04-26 16:26:20,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4055
[WARNING|trainer.py:803] 2025-04-26 16:26:20,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4022
4070
[WARNING|trainer.py:803] 2025-04-26 16:26:21,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:21,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4056
[WARNING|trainer.py:803] 2025-04-26 16:26:22,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4023
4071
[WARNING|trainer.py:803] 2025-04-26 16:26:23,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:23,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:23,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4057
4024
4072
[WARNING|trainer.py:803] 2025-04-26 16:26:24,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:25,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:25,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4058
4073
4025
[WARNING|trainer.py:803] 2025-04-26 16:26:26,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:26,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:26,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4059
4074
4026
[WARNING|trainer.py:803] 2025-04-26 16:26:27,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:28,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:28,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4060
4075
4027
[WARNING|trainer.py:803] 2025-04-26 16:26:29,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:29,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:29,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4061
4076
4028
[WARNING|trainer.py:803] 2025-04-26 16:26:31,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:31,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:31,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4062
4077
4029
[WARNING|trainer.py:803] 2025-04-26 16:26:32,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:32,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:33,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4063
4078
4030
[WARNING|trainer.py:803] 2025-04-26 16:26:34,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:34,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:34,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4064
4079
4031
[WARNING|trainer.py:803] 2025-04-26 16:26:35,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:36,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:36,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4065
4032
4080
[WARNING|trainer.py:803] 2025-04-26 16:26:37,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:37,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:37,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4066
4033
4081
[WARNING|trainer.py:803] 2025-04-26 16:26:38,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:39,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:39,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4067
4034
4082
[WARNING|trainer.py:803] 2025-04-26 16:26:40,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:40,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:40,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4068
4035
4083
[WARNING|trainer.py:803] 2025-04-26 16:26:42,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:42,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:42,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4069
4036
4084
[WARNING|trainer.py:803] 2025-04-26 16:26:43,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:44,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:44,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4070
4037
4085
[WARNING|trainer.py:803] 2025-04-26 16:26:45,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:45,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:45,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4071
4038
4086
[WARNING|trainer.py:803] 2025-04-26 16:26:46,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:26:47,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:47,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4072
4039
4087
[WARNING|trainer.py:803] 2025-04-26 16:26:48,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:48,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4073
[WARNING|trainer.py:803] 2025-04-26 16:26:48,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4040
[WARNING|trainer.py:803] 2025-04-26 16:26:49,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4088
4074
[WARNING|trainer.py:803] 2025-04-26 16:26:50,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:50,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4041
[WARNING|trainer.py:803] 2025-04-26 16:26:50,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4089
4075
[WARNING|trainer.py:803] 2025-04-26 16:26:51,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:26:51,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4042
[WARNING|trainer.py:803] 2025-04-26 16:26:52,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4090
[WARNING|trainer.py:803] 2025-04-26 16:26:53,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4076
[WARNING|trainer.py:803] 2025-04-26 16:26:53,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4043
[WARNING|trainer.py:803] 2025-04-26 16:26:54,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4091
4077
[WARNING|trainer.py:803] 2025-04-26 16:26:54,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:26:55,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4044
[WARNING|trainer.py:803] 2025-04-26 16:26:55,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4092
4078
[WARNING|trainer.py:803] 2025-04-26 16:26:56,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:56,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4045
[WARNING|trainer.py:803] 2025-04-26 16:26:57,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4093
4079
[WARNING|trainer.py:803] 2025-04-26 16:26:58,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:26:58,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4046
[WARNING|trainer.py:803] 2025-04-26 16:26:58,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4094
[WARNING|trainer.py:803] 2025-04-26 16:26:59,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4080
[WARNING|trainer.py:803] 2025-04-26 16:26:59,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4047
[WARNING|trainer.py:803] 2025-04-26 16:27:00,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4095
[WARNING|trainer.py:803] 2025-04-26 16:27:01,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4081
[WARNING|trainer.py:803] 2025-04-26 16:27:01,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4048
[WARNING|trainer.py:803] 2025-04-26 16:27:01,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4096
4082
[WARNING|trainer.py:803] 2025-04-26 16:27:02,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:02,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4049
[WARNING|trainer.py:803] 2025-04-26 16:27:03,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4097
4083
[WARNING|trainer.py:803] 2025-04-26 16:27:04,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:04,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:05,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4050
4098
4084
[WARNING|trainer.py:803] 2025-04-26 16:27:05,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:27:06,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:27:06,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4051
4099
4085
[WARNING|trainer.py:803] 2025-04-26 16:27:07,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:07,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4052
[WARNING|trainer.py:803] 2025-04-26 16:27:08,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4100
4086
[WARNING|trainer.py:803] 2025-04-26 16:27:08,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:09,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:09,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4053
4101
4087
[WARNING|trainer.py:803] 2025-04-26 16:27:10,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:10,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4054
[WARNING|trainer.py:803] 2025-04-26 16:27:11,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4102
[WARNING|trainer.py:803] 2025-04-26 16:27:11,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4088
[WARNING|trainer.py:803] 2025-04-26 16:27:12,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4055
[WARNING|trainer.py:803] 2025-04-26 16:27:12,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4103
4089
[WARNING|trainer.py:803] 2025-04-26 16:27:13,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:13,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4056
[WARNING|trainer.py:803] 2025-04-26 16:27:14,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4104
4090
[WARNING|trainer.py:803] 2025-04-26 16:27:15,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:15,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4057
[WARNING|trainer.py:803] 2025-04-26 16:27:15,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4105
4091
[WARNING|trainer.py:803] 2025-04-26 16:27:16,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:16,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:17,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4058
4106
4092
[WARNING|trainer.py:803] 2025-04-26 16:27:18,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:18,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4059
[WARNING|trainer.py:803] 2025-04-26 16:27:18,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4107
4093
[WARNING|trainer.py:803] 2025-04-26 16:27:19,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:19,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4108
[WARNING|trainer.py:803] 2025-04-26 16:27:20,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4060
4094
[WARNING|trainer.py:803] 2025-04-26 16:27:21,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:27:21,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4109
[WARNING|trainer.py:803] 2025-04-26 16:27:22,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4061
4095
[WARNING|trainer.py:803] 2025-04-26 16:27:22,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:22,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4110
4062
[WARNING|trainer.py:803] 2025-04-26 16:27:23,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:24,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:24,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4096
4111
4063
[WARNING|trainer.py:803] 2025-04-26 16:27:25,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:25,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:26,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4097
4112
4064
[WARNING|trainer.py:803] 2025-04-26 16:27:27,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:27,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:27,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4098
4113
4065
[WARNING|trainer.py:803] 2025-04-26 16:27:28,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:27:28,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:29,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4099
4114
4066
[WARNING|trainer.py:803] 2025-04-26 16:27:30,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:30,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:31,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4100
4115
4067
[WARNING|trainer.py:803] 2025-04-26 16:27:31,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:32,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:32,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4116
4101
4068
[WARNING|trainer.py:803] 2025-04-26 16:27:33,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:33,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4117
[WARNING|trainer.py:803] 2025-04-26 16:27:34,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4102
[WARNING|trainer.py:803] 2025-04-26 16:27:35,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4069
[WARNING|trainer.py:803] 2025-04-26 16:27:35,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4118
4103
[WARNING|trainer.py:803] 2025-04-26 16:27:35,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4070
[WARNING|trainer.py:803] 2025-04-26 16:27:36,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:36,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4119
[WARNING|trainer.py:803] 2025-04-26 16:27:37,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4104
4071
[WARNING|trainer.py:803] 2025-04-26 16:27:38,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:38,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:38,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4105
4120
4072
[WARNING|trainer.py:803] 2025-04-26 16:27:39,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:39,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:40,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4106
4121
4073
[WARNING|trainer.py:803] 2025-04-26 16:27:41,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:41,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:41,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4107
4122
4074
[WARNING|trainer.py:803] 2025-04-26 16:27:42,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:42,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:43,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4108
4123
4075
[WARNING|trainer.py:803] 2025-04-26 16:27:44,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:27:44,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:44,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4109
4124
4076
[WARNING|trainer.py:803] 2025-04-26 16:27:45,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:45,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:46,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4110
4125
4077
[WARNING|trainer.py:803] 2025-04-26 16:27:47,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:47,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:47,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4111
4126
4078
[WARNING|trainer.py:803] 2025-04-26 16:27:48,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:48,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:49,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4112
4127
4079
[WARNING|trainer.py:803] 2025-04-26 16:27:50,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:50,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4113
[WARNING|trainer.py:803] 2025-04-26 16:27:51,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4128
4080
[WARNING|trainer.py:803] 2025-04-26 16:27:51,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:51,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:52,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4114
4129
4081
[WARNING|trainer.py:803] 2025-04-26 16:27:53,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:53,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4115
4130
[WARNING|trainer.py:803] 2025-04-26 16:27:54,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4082
[WARNING|trainer.py:803] 2025-04-26 16:27:54,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:54,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4116
[WARNING|trainer.py:803] 2025-04-26 16:27:55,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4131
4083
[WARNING|trainer.py:803] 2025-04-26 16:27:56,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:56,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4117
4132
[WARNING|trainer.py:803] 2025-04-26 16:27:57,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:57,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4084
[WARNING|trainer.py:803] 2025-04-26 16:27:57,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4118
4133
[WARNING|trainer.py:803] 2025-04-26 16:27:58,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:27:59,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:27:59,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4085
4119
4134
[WARNING|trainer.py:803] 2025-04-26 16:28:00,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:00,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:00,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4086
4135
4120
[WARNING|trainer.py:803] 2025-04-26 16:28:01,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:02,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4087
[WARNING|trainer.py:803] 2025-04-26 16:28:02,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4136
4121
[WARNING|trainer.py:803] 2025-04-26 16:28:03,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:03,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:04,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4088
4137
4122
[WARNING|trainer.py:803] 2025-04-26 16:28:05,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:05,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:05,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4089
4138
4123
[WARNING|trainer.py:803] 2025-04-26 16:28:06,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:07,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:07,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4090
4139
4124
[WARNING|trainer.py:803] 2025-04-26 16:28:08,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:08,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:08,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4091
4140
4125
[WARNING|trainer.py:803] 2025-04-26 16:28:09,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:10,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:10,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4092
4141
4126
[WARNING|trainer.py:803] 2025-04-26 16:28:11,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:11,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:11,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4093
4142
4127
[WARNING|trainer.py:803] 2025-04-26 16:28:12,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:13,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:13,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4094
4143
4128
[WARNING|trainer.py:803] 2025-04-26 16:28:14,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:28:14,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:14,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4095
4144
4129
[WARNING|trainer.py:803] 2025-04-26 16:28:16,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:16,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:16,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4145
4096
4130
[WARNING|trainer.py:803] 2025-04-26 16:28:17,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:17,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:17,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4146
4097
4131
[WARNING|trainer.py:803] 2025-04-26 16:28:19,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:19,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:19,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4147
4132
4098
[WARNING|trainer.py:803] 2025-04-26 16:28:20,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:20,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:20,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4148
4133
4099
[WARNING|trainer.py:803] 2025-04-26 16:28:22,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:22,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:22,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4134
4149
4100
[WARNING|trainer.py:803] 2025-04-26 16:28:23,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:23,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:24,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4135
4150
4101
[WARNING|trainer.py:803] 2025-04-26 16:28:25,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:25,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:25,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4151
4136
4102
[WARNING|trainer.py:803] 2025-04-26 16:28:26,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:26,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:27,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4152
4137
4103
[WARNING|trainer.py:803] 2025-04-26 16:28:28,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:28,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:28,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4153
4138
4104
[WARNING|trainer.py:803] 2025-04-26 16:28:29,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:29,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:30,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4154
4139
4105
[WARNING|trainer.py:803] 2025-04-26 16:28:31,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:31,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:31,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4155
4140
4106
[WARNING|trainer.py:803] 2025-04-26 16:28:32,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:32,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:33,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4156
4141
4107
[WARNING|trainer.py:803] 2025-04-26 16:28:34,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:34,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:34,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4157
4142
4108
[WARNING|trainer.py:803] 2025-04-26 16:28:35,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:35,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:36,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4143
4158
4109
[WARNING|trainer.py:803] 2025-04-26 16:28:37,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:37,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:37,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4144
4159
4110
[WARNING|trainer.py:803] 2025-04-26 16:28:38,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:38,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:39,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4160
4145
4111
[WARNING|trainer.py:803] 2025-04-26 16:28:40,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:40,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:40,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4161
4146
4112
[WARNING|trainer.py:803] 2025-04-26 16:28:41,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:41,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:42,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4147
4162
4113
[WARNING|trainer.py:803] 2025-04-26 16:28:43,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:43,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:43,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4148
4163
4114
[WARNING|trainer.py:803] 2025-04-26 16:28:45,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:45,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:45,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4164
4149
4115
[WARNING|trainer.py:803] 2025-04-26 16:28:46,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:46,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:46,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4165
4150
4116
[WARNING|trainer.py:803] 2025-04-26 16:28:48,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:48,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:28:48,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4151
4166
4117
[WARNING|trainer.py:803] 2025-04-26 16:28:49,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:49,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:49,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4152
4167
4118
[WARNING|trainer.py:803] 2025-04-26 16:28:50,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:51,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:51,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4153
4168
4119
[WARNING|trainer.py:803] 2025-04-26 16:28:52,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:52,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:28:53,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4154
4169
[WARNING|trainer.py:803] 2025-04-26 16:28:53,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4120
[WARNING|trainer.py:803] 2025-04-26 16:28:54,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4155
[WARNING|trainer.py:803] 2025-04-26 16:28:54,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4170
4121
[WARNING|trainer.py:803] 2025-04-26 16:28:55,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:55,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4156
[WARNING|trainer.py:803] 2025-04-26 16:28:56,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4171
[WARNING|trainer.py:803] 2025-04-26 16:28:56,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4122
[WARNING|trainer.py:803] 2025-04-26 16:28:57,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4157
4172
[WARNING|trainer.py:803] 2025-04-26 16:28:57,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:28:58,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4123
[WARNING|trainer.py:803] 2025-04-26 16:28:58,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4158
4173
[WARNING|trainer.py:803] 2025-04-26 16:28:59,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4124
[WARNING|trainer.py:803] 2025-04-26 16:29:00,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:00,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4159
4174
[WARNING|trainer.py:803] 2025-04-26 16:29:00,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4125
[WARNING|trainer.py:803] 2025-04-26 16:29:01,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:01,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4160
[WARNING|trainer.py:803] 2025-04-26 16:29:02,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4175
[WARNING|trainer.py:803] 2025-04-26 16:29:03,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4126
[WARNING|trainer.py:803] 2025-04-26 16:29:03,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4161
[WARNING|trainer.py:803] 2025-04-26 16:29:03,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4176
[WARNING|trainer.py:803] 2025-04-26 16:29:04,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4127
[WARNING|trainer.py:803] 2025-04-26 16:29:04,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4162
4177
[WARNING|trainer.py:803] 2025-04-26 16:29:05,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:06,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4128
[WARNING|trainer.py:803] 2025-04-26 16:29:06,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4163
4178
[WARNING|trainer.py:803] 2025-04-26 16:29:06,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:07,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:07,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4129
4179
4164
[WARNING|trainer.py:803] 2025-04-26 16:29:08,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:09,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4130
[WARNING|trainer.py:803] 2025-04-26 16:29:09,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4180
4165
[WARNING|trainer.py:803] 2025-04-26 16:29:09,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:10,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4131
[WARNING|trainer.py:803] 2025-04-26 16:29:10,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4181
4166
[WARNING|trainer.py:803] 2025-04-26 16:29:11,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:12,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4132
[WARNING|trainer.py:803] 2025-04-26 16:29:12,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4182
4167
[WARNING|trainer.py:803] 2025-04-26 16:29:13,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:13,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4133
[WARNING|trainer.py:803] 2025-04-26 16:29:13,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4183
4168
[WARNING|trainer.py:803] 2025-04-26 16:29:14,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:15,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4134
[WARNING|trainer.py:803] 2025-04-26 16:29:15,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4184
4169
[WARNING|trainer.py:803] 2025-04-26 16:29:16,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:16,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4135
[WARNING|trainer.py:803] 2025-04-26 16:29:16,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4185
4170
[WARNING|trainer.py:803] 2025-04-26 16:29:17,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:18,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4136
[WARNING|trainer.py:803] 2025-04-26 16:29:18,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4186
4171
[WARNING|trainer.py:803] 2025-04-26 16:29:19,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:19,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:19,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4137
4172
4187
[WARNING|trainer.py:803] 2025-04-26 16:29:20,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4138
[WARNING|trainer.py:803] 2025-04-26 16:29:21,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:21,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4173
4188
[WARNING|trainer.py:803] 2025-04-26 16:29:22,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:22,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4139
[WARNING|trainer.py:803] 2025-04-26 16:29:22,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4174
4189
[WARNING|trainer.py:803] 2025-04-26 16:29:23,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4140
[WARNING|trainer.py:803] 2025-04-26 16:29:24,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:24,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4175
[WARNING|trainer.py:803] 2025-04-26 16:29:25,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4190
4141
[WARNING|trainer.py:803] 2025-04-26 16:29:25,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:26,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4176
[WARNING|trainer.py:803] 2025-04-26 16:29:26,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4191
4142
[WARNING|trainer.py:803] 2025-04-26 16:29:27,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:27,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4177
[WARNING|trainer.py:803] 2025-04-26 16:29:28,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4192
[WARNING|trainer.py:803] 2025-04-26 16:29:28,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4143
[WARNING|trainer.py:803] 2025-04-26 16:29:29,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4178
[WARNING|trainer.py:803] 2025-04-26 16:29:29,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4193
[WARNING|trainer.py:803] 2025-04-26 16:29:30,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4144
[WARNING|trainer.py:803] 2025-04-26 16:29:30,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4179
[WARNING|trainer.py:803] 2025-04-26 16:29:31,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4194
[WARNING|trainer.py:803] 2025-04-26 16:29:31,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4145
[WARNING|trainer.py:803] 2025-04-26 16:29:32,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4180
[WARNING|trainer.py:803] 2025-04-26 16:29:32,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4195
[WARNING|trainer.py:803] 2025-04-26 16:29:33,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4146
[WARNING|trainer.py:803] 2025-04-26 16:29:33,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4181
4196
[WARNING|trainer.py:803] 2025-04-26 16:29:34,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:34,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4147
[WARNING|trainer.py:803] 2025-04-26 16:29:35,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4182
4197
[WARNING|trainer.py:803] 2025-04-26 16:29:35,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:36,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4148
[WARNING|trainer.py:803] 2025-04-26 16:29:36,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4183
4198
[WARNING|trainer.py:803] 2025-04-26 16:29:37,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:37,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4149
[WARNING|trainer.py:803] 2025-04-26 16:29:38,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4184
4199
[WARNING|trainer.py:803] 2025-04-26 16:29:38,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:39,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4150
[WARNING|trainer.py:803] 2025-04-26 16:29:39,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4185
4200
[WARNING|trainer.py:803] 2025-04-26 16:29:40,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:40,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4151
[WARNING|trainer.py:803] 2025-04-26 16:29:41,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4186
4201
[WARNING|trainer.py:803] 2025-04-26 16:29:41,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:42,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:42,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4152
4202
4187
[WARNING|trainer.py:803] 2025-04-26 16:29:43,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:43,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:43,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4153
4203
4188
[WARNING|trainer.py:803] 2025-04-26 16:29:44,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:44,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:45,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4204
4154
4189
[WARNING|trainer.py:803] 2025-04-26 16:29:46,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:29:46,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4205
[WARNING|trainer.py:803] 2025-04-26 16:29:46,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4155
[WARNING|trainer.py:803] 2025-04-26 16:29:47,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4190
[WARNING|trainer.py:803] 2025-04-26 16:29:47,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4206
4156
[WARNING|trainer.py:803] 2025-04-26 16:29:48,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:48,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4207
4191
[WARNING|trainer.py:803] 2025-04-26 16:29:49,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4157
[WARNING|trainer.py:803] 2025-04-26 16:29:49,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:50,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4208
4192
[WARNING|trainer.py:803] 2025-04-26 16:29:50,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:51,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:29:51,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4158
4209
4193
[WARNING|trainer.py:803] 2025-04-26 16:29:52,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:52,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4210
[WARNING|trainer.py:803] 2025-04-26 16:29:53,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4159
[WARNING|trainer.py:803] 2025-04-26 16:29:53,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4194
[WARNING|trainer.py:803] 2025-04-26 16:29:53,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4211
4160
[WARNING|trainer.py:803] 2025-04-26 16:29:54,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:54,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4195
[WARNING|trainer.py:803] 2025-04-26 16:29:55,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4212
4161
[WARNING|trainer.py:803] 2025-04-26 16:29:56,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:56,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4213
4196
[WARNING|trainer.py:803] 2025-04-26 16:29:56,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:57,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4162
[WARNING|trainer.py:803] 2025-04-26 16:29:57,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4214
4197
[WARNING|trainer.py:803] 2025-04-26 16:29:58,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:29:58,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4215
4163
[WARNING|trainer.py:803] 2025-04-26 16:29:59,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4198
[WARNING|trainer.py:803] 2025-04-26 16:29:59,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:29:59,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4216
4164
[WARNING|trainer.py:803] 2025-04-26 16:30:00,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:01,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4199
[WARNING|trainer.py:803] 2025-04-26 16:30:01,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4217
4165
[WARNING|trainer.py:803] 2025-04-26 16:30:02,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:02,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4218
4200
[WARNING|trainer.py:803] 2025-04-26 16:30:02,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4166
[WARNING|trainer.py:803] 2025-04-26 16:30:03,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:03,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4219
4201
[WARNING|trainer.py:803] 2025-04-26 16:30:04,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:04,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:04,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4167
4220
4202
[WARNING|trainer.py:803] 2025-04-26 16:30:05,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:06,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:06,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4221
4203
4168
[WARNING|trainer.py:803] 2025-04-26 16:30:07,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:07,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:30:07,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4222
4204
4169
[WARNING|trainer.py:803] 2025-04-26 16:30:08,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:08,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4223
[WARNING|trainer.py:803] 2025-04-26 16:30:09,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4205
[WARNING|trainer.py:803] 2025-04-26 16:30:09,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4170
[WARNING|trainer.py:803] 2025-04-26 16:30:10,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4224
4206
[WARNING|trainer.py:803] 2025-04-26 16:30:10,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:11,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4171
[WARNING|trainer.py:803] 2025-04-26 16:30:11,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4225
4207
[WARNING|trainer.py:803] 2025-04-26 16:30:12,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:12,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:30:12,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4226
4172
4208
[WARNING|trainer.py:803] 2025-04-26 16:30:13,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:13,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:13,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4227
4209
4173
[WARNING|trainer.py:803] 2025-04-26 16:30:14,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:15,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4228
[WARNING|trainer.py:803] 2025-04-26 16:30:15,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4210
4174
[WARNING|trainer.py:803] 2025-04-26 16:30:15,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4229
[WARNING|trainer.py:803] 2025-04-26 16:30:16,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4211
[WARNING|trainer.py:803] 2025-04-26 16:30:16,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:17,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4175
4230
[WARNING|trainer.py:803] 2025-04-26 16:30:17,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4212
[WARNING|trainer.py:803] 2025-04-26 16:30:18,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:18,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4231
[WARNING|trainer.py:803] 2025-04-26 16:30:18,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4176
4213
[WARNING|trainer.py:803] 2025-04-26 16:30:19,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:19,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4232
[WARNING|trainer.py:803] 2025-04-26 16:30:20,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4177
4214
[WARNING|trainer.py:803] 2025-04-26 16:30:20,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4233
[WARNING|trainer.py:803] 2025-04-26 16:30:21,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:21,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4215
4178
[WARNING|trainer.py:803] 2025-04-26 16:30:22,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4234
[WARNING|trainer.py:803] 2025-04-26 16:30:22,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:22,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4216
[WARNING|trainer.py:803] 2025-04-26 16:30:23,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4179
4235
[WARNING|trainer.py:803] 2025-04-26 16:30:23,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:24,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4217
[WARNING|trainer.py:803] 2025-04-26 16:30:24,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4180
4236
[WARNING|trainer.py:803] 2025-04-26 16:30:25,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4218
[WARNING|trainer.py:803] 2025-04-26 16:30:25,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:25,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4237
[WARNING|trainer.py:803] 2025-04-26 16:30:26,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4181
4219
[WARNING|trainer.py:803] 2025-04-26 16:30:27,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:27,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4238
[WARNING|trainer.py:803] 2025-04-26 16:30:27,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4182
4220
[WARNING|trainer.py:803] 2025-04-26 16:30:28,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4239
[WARNING|trainer.py:803] 2025-04-26 16:30:28,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:30:28,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4221
4183
[WARNING|trainer.py:803] 2025-04-26 16:30:29,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4240
[WARNING|trainer.py:803] 2025-04-26 16:30:30,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:30,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4222
[WARNING|trainer.py:803] 2025-04-26 16:30:30,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4184
4241
[WARNING|trainer.py:803] 2025-04-26 16:30:31,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4223
[WARNING|trainer.py:803] 2025-04-26 16:30:31,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:31,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4242
4185
[WARNING|trainer.py:803] 2025-04-26 16:30:32,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4224
[WARNING|trainer.py:803] 2025-04-26 16:30:33,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:33,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4243
[WARNING|trainer.py:803] 2025-04-26 16:30:33,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4186
4225
[WARNING|trainer.py:803] 2025-04-26 16:30:34,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4244
[WARNING|trainer.py:803] 2025-04-26 16:30:35,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:35,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4226
[WARNING|trainer.py:803] 2025-04-26 16:30:35,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4187
4245
[WARNING|trainer.py:803] 2025-04-26 16:30:36,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:36,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4227
[WARNING|trainer.py:803] 2025-04-26 16:30:36,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4188
4246
[WARNING|trainer.py:803] 2025-04-26 16:30:37,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4228
[WARNING|trainer.py:803] 2025-04-26 16:30:38,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:38,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4247
4189
[WARNING|trainer.py:803] 2025-04-26 16:30:38,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4229
[WARNING|trainer.py:803] 2025-04-26 16:30:39,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:39,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4248
[WARNING|trainer.py:803] 2025-04-26 16:30:40,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4230
4190
[WARNING|trainer.py:803] 2025-04-26 16:30:40,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4249
[WARNING|trainer.py:803] 2025-04-26 16:30:41,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:41,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4231
[WARNING|trainer.py:803] 2025-04-26 16:30:41,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4191
4250
[WARNING|trainer.py:803] 2025-04-26 16:30:42,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:42,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4232
[WARNING|trainer.py:803] 2025-04-26 16:30:43,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4192
4251
[WARNING|trainer.py:803] 2025-04-26 16:30:43,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4233
[WARNING|trainer.py:803] 2025-04-26 16:30:44,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:44,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4252
4193
[WARNING|trainer.py:803] 2025-04-26 16:30:45,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4234
[WARNING|trainer.py:803] 2025-04-26 16:30:45,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:45,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4253
[WARNING|trainer.py:803] 2025-04-26 16:30:46,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4194
4235
[WARNING|trainer.py:803] 2025-04-26 16:30:46,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4254
[WARNING|trainer.py:803] 2025-04-26 16:30:47,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:47,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4236
4195
[WARNING|trainer.py:803] 2025-04-26 16:30:48,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4255
[WARNING|trainer.py:803] 2025-04-26 16:30:48,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:48,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4237
[WARNING|trainer.py:803] 2025-04-26 16:30:49,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4196
4256
[WARNING|trainer.py:803] 2025-04-26 16:30:50,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:50,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4238
[WARNING|trainer.py:803] 2025-04-26 16:30:50,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4257
4197
[WARNING|trainer.py:803] 2025-04-26 16:30:51,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4239
[WARNING|trainer.py:803] 2025-04-26 16:30:51,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:52,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4258
[WARNING|trainer.py:803] 2025-04-26 16:30:52,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4198
4240
[WARNING|trainer.py:803] 2025-04-26 16:30:53,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4259
[WARNING|trainer.py:803] 2025-04-26 16:30:53,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:53,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:54,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4199
4241
4260
[WARNING|trainer.py:803] 2025-04-26 16:30:55,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:55,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:55,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4242
4200
4261
[WARNING|trainer.py:803] 2025-04-26 16:30:56,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:56,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:56,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4243
4201
4262
[WARNING|trainer.py:803] 2025-04-26 16:30:57,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:57,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:57,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4244
4202
4263
[WARNING|trainer.py:803] 2025-04-26 16:30:58,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:30:59,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:30:59,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4245
4203
4264
[WARNING|trainer.py:803] 2025-04-26 16:31:00,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:00,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:00,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4246
4204
4265
[WARNING|trainer.py:803] 2025-04-26 16:31:01,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:01,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:31:01,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4247
4205
4266
[WARNING|trainer.py:803] 2025-04-26 16:31:02,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:02,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:02,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4248
4206
4267
[WARNING|trainer.py:803] 2025-04-26 16:31:03,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:04,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:04,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4249
4207
4268
[WARNING|trainer.py:803] 2025-04-26 16:31:05,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:05,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:05,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4250
4208
4269
[WARNING|trainer.py:803] 2025-04-26 16:31:06,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:06,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:31:06,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4251
4209
4270
[WARNING|trainer.py:803] 2025-04-26 16:31:07,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:07,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:07,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4252
4210
4271
[WARNING|trainer.py:803] 2025-04-26 16:31:08,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:09,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:09,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4253
4211
4272
[WARNING|trainer.py:803] 2025-04-26 16:31:10,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:10,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:10,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4254
4212
4273
[WARNING|trainer.py:803] 2025-04-26 16:31:11,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:11,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:11,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4255
4213
4274
[WARNING|trainer.py:803] 2025-04-26 16:31:12,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:12,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:12,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4256
4214
4275
[WARNING|trainer.py:803] 2025-04-26 16:31:13,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:14,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:14,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4257
4215
4276
[WARNING|trainer.py:803] 2025-04-26 16:31:15,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:15,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:15,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4258
4216
4277
[WARNING|trainer.py:803] 2025-04-26 16:31:16,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:16,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:16,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4259
4278
4217
[WARNING|trainer.py:803] 2025-04-26 16:31:17,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:17,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:17,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4260
4279
4218
[WARNING|trainer.py:803] 2025-04-26 16:31:18,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:19,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:19,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4261
4280
4219
[WARNING|trainer.py:803] 2025-04-26 16:31:20,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:20,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:20,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4262
4281
4220
[WARNING|trainer.py:803] 2025-04-26 16:31:21,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:21,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:21,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4263
4282
4221
[WARNING|trainer.py:803] 2025-04-26 16:31:22,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:22,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:22,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4264
4283
4222
[WARNING|trainer.py:803] 2025-04-26 16:31:23,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:24,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:24,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4265
4284
4223
[WARNING|trainer.py:803] 2025-04-26 16:31:24,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:25,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:25,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4266
4285
4224
[WARNING|trainer.py:803] 2025-04-26 16:31:26,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:26,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:26,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4267
4286
4225
[WARNING|trainer.py:803] 2025-04-26 16:31:27,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:27,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:31:27,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4268
4287
4226
[WARNING|trainer.py:803] 2025-04-26 16:31:28,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:28,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:29,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4269
4288
4227
[WARNING|trainer.py:803] 2025-04-26 16:31:29,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:30,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:31:30,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4270
4289
4228
[WARNING|trainer.py:803] 2025-04-26 16:31:31,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:31,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:31,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4271
4290
4229
[WARNING|trainer.py:803] 2025-04-26 16:31:32,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:32,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:32,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4272
4291
4230
[WARNING|trainer.py:803] 2025-04-26 16:31:33,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:33,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:34,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4273
4292
4231
[WARNING|trainer.py:803] 2025-04-26 16:31:34,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:35,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:35,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4274
4293
4232
[WARNING|trainer.py:803] 2025-04-26 16:31:36,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:36,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:36,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4275
4294
4233
[WARNING|trainer.py:803] 2025-04-26 16:31:37,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:37,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:37,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4276
4295
4234
[WARNING|trainer.py:803] 2025-04-26 16:31:38,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:38,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4277
[WARNING|trainer.py:803] 2025-04-26 16:31:39,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4296
4235
[WARNING|trainer.py:803] 2025-04-26 16:31:39,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:40,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4278
[WARNING|trainer.py:803] 2025-04-26 16:31:40,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4297
4236
[WARNING|trainer.py:803] 2025-04-26 16:31:41,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:41,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4279
[WARNING|trainer.py:803] 2025-04-26 16:31:41,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4298
4237
[WARNING|trainer.py:803] 2025-04-26 16:31:42,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:42,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4280
[WARNING|trainer.py:803] 2025-04-26 16:31:42,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4299
4238
[WARNING|trainer.py:803] 2025-04-26 16:31:43,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:43,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:44,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4281
4300
4239
[WARNING|trainer.py:803] 2025-04-26 16:31:44,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:45,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:45,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4282
4301
4240
[WARNING|trainer.py:803] 2025-04-26 16:31:46,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:46,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:46,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4283
4302
4241
[WARNING|trainer.py:803] 2025-04-26 16:31:47,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:47,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:47,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4284
4303
4242
[WARNING|trainer.py:803] 2025-04-26 16:31:48,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:48,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:48,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4285
4304
4243
[WARNING|trainer.py:803] 2025-04-26 16:31:49,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:50,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:50,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4286
4305
4244
[WARNING|trainer.py:803] 2025-04-26 16:31:51,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:31:51,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:51,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4287
4306
4245
[WARNING|trainer.py:803] 2025-04-26 16:31:52,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:31:52,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4288
[WARNING|trainer.py:803] 2025-04-26 16:31:52,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4307
4246
[WARNING|trainer.py:803] 2025-04-26 16:31:53,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:31:53,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4289
[WARNING|trainer.py:803] 2025-04-26 16:31:53,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4308
4247
[WARNING|trainer.py:803] 2025-04-26 16:31:54,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:55,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4290
[WARNING|trainer.py:803] 2025-04-26 16:31:55,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4309
4248
[WARNING|trainer.py:803] 2025-04-26 16:31:55,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:56,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4291
[WARNING|trainer.py:803] 2025-04-26 16:31:56,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4310
4249
[WARNING|trainer.py:803] 2025-04-26 16:31:57,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:57,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4292
[WARNING|trainer.py:803] 2025-04-26 16:31:57,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4311
4250
[WARNING|trainer.py:803] 2025-04-26 16:31:58,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:31:58,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4293
[WARNING|trainer.py:803] 2025-04-26 16:31:58,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4312
4251
[WARNING|trainer.py:803] 2025-04-26 16:31:59,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:00,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4294
[WARNING|trainer.py:803] 2025-04-26 16:32:00,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4313
4252
[WARNING|trainer.py:803] 2025-04-26 16:32:00,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:01,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4295
[WARNING|trainer.py:803] 2025-04-26 16:32:01,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4314
4253
[WARNING|trainer.py:803] 2025-04-26 16:32:02,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4296
[WARNING|trainer.py:803] 2025-04-26 16:32:02,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:02,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4315
4254
[WARNING|trainer.py:803] 2025-04-26 16:32:03,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:03,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4297
[WARNING|trainer.py:803] 2025-04-26 16:32:03,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4316
4255
[WARNING|trainer.py:803] 2025-04-26 16:32:04,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:32:04,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4298
[WARNING|trainer.py:803] 2025-04-26 16:32:05,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4317
4256
[WARNING|trainer.py:803] 2025-04-26 16:32:05,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4299
[WARNING|trainer.py:803] 2025-04-26 16:32:06,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:06,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4318
4257
[WARNING|trainer.py:803] 2025-04-26 16:32:07,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4300
[WARNING|trainer.py:803] 2025-04-26 16:32:07,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:07,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4319
4258
[WARNING|trainer.py:803] 2025-04-26 16:32:08,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4301
[WARNING|trainer.py:803] 2025-04-26 16:32:08,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:08,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4320
4259
[WARNING|trainer.py:803] 2025-04-26 16:32:09,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4302
[WARNING|trainer.py:803] 2025-04-26 16:32:10,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:10,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4260
4321
[WARNING|trainer.py:803] 2025-04-26 16:32:10,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4303
[WARNING|trainer.py:803] 2025-04-26 16:32:11,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:11,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4261
4322
[WARNING|trainer.py:803] 2025-04-26 16:32:12,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:12,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4304
[WARNING|trainer.py:803] 2025-04-26 16:32:12,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4262
4323
[WARNING|trainer.py:803] 2025-04-26 16:32:13,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4305
[WARNING|trainer.py:803] 2025-04-26 16:32:13,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:13,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4263
4324
[WARNING|trainer.py:803] 2025-04-26 16:32:14,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:14,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:15,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4306
4264
4325
[WARNING|trainer.py:803] 2025-04-26 16:32:15,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:16,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:16,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4307
4265
4326
[WARNING|trainer.py:803] 2025-04-26 16:32:17,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:17,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:17,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4308
4266
4327
[WARNING|trainer.py:803] 2025-04-26 16:32:18,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:18,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:18,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4309
4267
4328
[WARNING|trainer.py:803] 2025-04-26 16:32:19,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:32:19,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:20,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4310
4268
4329
[WARNING|trainer.py:803] 2025-04-26 16:32:20,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:21,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:21,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4311
4269
4330
[WARNING|trainer.py:803] 2025-04-26 16:32:22,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:22,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:22,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4312
4270
4331
[WARNING|trainer.py:803] 2025-04-26 16:32:23,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:23,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:23,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4313
4271
4332
[WARNING|trainer.py:803] 2025-04-26 16:32:24,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:24,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:24,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4314
4272
4333
[WARNING|trainer.py:803] 2025-04-26 16:32:25,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:26,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:26,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4315
4273
4334
[WARNING|trainer.py:803] 2025-04-26 16:32:27,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:27,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:27,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4316
4274
4335
[WARNING|trainer.py:803] 2025-04-26 16:32:28,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:28,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:28,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4317
4275
4336
[WARNING|trainer.py:803] 2025-04-26 16:32:29,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:29,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:29,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4318
4276
4337
[WARNING|trainer.py:803] 2025-04-26 16:32:30,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:31,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:31,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4319
4338
4277
[WARNING|trainer.py:803] 2025-04-26 16:32:32,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:32,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:32:32,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4320
4339
4278
[WARNING|trainer.py:803] 2025-04-26 16:32:33,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:33,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:33,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4321
4340
4279
[WARNING|trainer.py:803] 2025-04-26 16:32:34,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:34,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:34,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4322
4341
4280
[WARNING|trainer.py:803] 2025-04-26 16:32:35,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:36,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:36,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4323
4342
4281
[WARNING|trainer.py:803] 2025-04-26 16:32:37,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:37,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:37,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4324
4343
4282
[WARNING|trainer.py:803] 2025-04-26 16:32:38,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:38,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:38,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4325
4344
4283
[WARNING|trainer.py:803] 2025-04-26 16:32:39,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:39,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:39,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4326
4345
4284
[WARNING|trainer.py:803] 2025-04-26 16:32:40,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:32:41,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:32:41,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4327
4346
4285
[WARNING|trainer.py:803] 2025-04-26 16:32:42,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:42,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:42,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4328
4347
4286
[WARNING|trainer.py:803] 2025-04-26 16:32:43,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:43,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:43,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4329
4348
4287
[WARNING|trainer.py:803] 2025-04-26 16:32:44,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:44,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4330
[WARNING|trainer.py:803] 2025-04-26 16:32:45,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4288
4349
[WARNING|trainer.py:803] 2025-04-26 16:32:45,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:46,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:32:46,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4331
4289
4350
[WARNING|trainer.py:803] 2025-04-26 16:32:47,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:47,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:47,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4332
4351
4290
[WARNING|trainer.py:803] 2025-04-26 16:32:48,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:32:48,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:48,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4333
4352
4291
[WARNING|trainer.py:803] 2025-04-26 16:32:49,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:49,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:49,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4334
4353
4292
[WARNING|trainer.py:803] 2025-04-26 16:32:50,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:51,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:51,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4335
4354
4293
[WARNING|trainer.py:803] 2025-04-26 16:32:52,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:52,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:52,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4336
4355
4294
[WARNING|trainer.py:803] 2025-04-26 16:32:53,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:53,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:53,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4337
4356
4295
[WARNING|trainer.py:803] 2025-04-26 16:32:54,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:54,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:54,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4338
4357
4296
[WARNING|trainer.py:803] 2025-04-26 16:32:55,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:55,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:32:56,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4358
4339
4297
[WARNING|trainer.py:803] 2025-04-26 16:32:56,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:32:57,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4359
[WARNING|trainer.py:803] 2025-04-26 16:32:57,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4340
4298
[WARNING|trainer.py:803] 2025-04-26 16:32:58,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:58,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4360
[WARNING|trainer.py:803] 2025-04-26 16:32:58,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4341
4299
[WARNING|trainer.py:803] 2025-04-26 16:32:59,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:32:59,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4361
[WARNING|trainer.py:803] 2025-04-26 16:32:59,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4342
4300
[WARNING|trainer.py:803] 2025-04-26 16:33:00,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:00,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4362
[WARNING|trainer.py:803] 2025-04-26 16:33:01,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4343
4301
[WARNING|trainer.py:803] 2025-04-26 16:33:01,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:02,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4363
[WARNING|trainer.py:803] 2025-04-26 16:33:02,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4344
4302
[WARNING|trainer.py:803] 2025-04-26 16:33:02,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4364
[WARNING|trainer.py:803] 2025-04-26 16:33:03,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:03,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4345
4303
[WARNING|trainer.py:803] 2025-04-26 16:33:04,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4365
[WARNING|trainer.py:803] 2025-04-26 16:33:04,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:33:04,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4346
4304
[WARNING|trainer.py:803] 2025-04-26 16:33:05,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4366
[WARNING|trainer.py:803] 2025-04-26 16:33:05,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:06,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4347
[WARNING|trainer.py:803] 2025-04-26 16:33:06,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4305
4367
[WARNING|trainer.py:803] 2025-04-26 16:33:07,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:07,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4348
[WARNING|trainer.py:803] 2025-04-26 16:33:07,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4306
4368
[WARNING|trainer.py:803] 2025-04-26 16:33:08,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4349
[WARNING|trainer.py:803] 2025-04-26 16:33:08,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:08,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4307
4369
[WARNING|trainer.py:803] 2025-04-26 16:33:09,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4350
[WARNING|trainer.py:803] 2025-04-26 16:33:09,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:10,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4308
4370
[WARNING|trainer.py:803] 2025-04-26 16:33:10,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4351
[WARNING|trainer.py:803] 2025-04-26 16:33:11,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:11,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4309
4371
[WARNING|trainer.py:803] 2025-04-26 16:33:11,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4352
[WARNING|trainer.py:803] 2025-04-26 16:33:12,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:33:12,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4372
4310
[WARNING|trainer.py:803] 2025-04-26 16:33:13,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4353
[WARNING|trainer.py:803] 2025-04-26 16:33:13,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:13,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4373
4311
[WARNING|trainer.py:803] 2025-04-26 16:33:14,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4354
[WARNING|trainer.py:803] 2025-04-26 16:33:14,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:14,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4374
4312
[WARNING|trainer.py:803] 2025-04-26 16:33:15,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4355
[WARNING|trainer.py:803] 2025-04-26 16:33:15,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:16,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4375
4313
[WARNING|trainer.py:803] 2025-04-26 16:33:16,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4356
[WARNING|trainer.py:803] 2025-04-26 16:33:17,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:17,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4376
[WARNING|trainer.py:803] 2025-04-26 16:33:17,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4314
4357
[WARNING|trainer.py:803] 2025-04-26 16:33:18,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:18,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4377
[WARNING|trainer.py:803] 2025-04-26 16:33:18,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4315
4358
[WARNING|trainer.py:803] 2025-04-26 16:33:19,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:19,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4378
[WARNING|trainer.py:803] 2025-04-26 16:33:20,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4316
4359
[WARNING|trainer.py:803] 2025-04-26 16:33:20,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4379
[WARNING|trainer.py:803] 2025-04-26 16:33:21,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:21,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4317
4360
[WARNING|trainer.py:803] 2025-04-26 16:33:21,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4380
[WARNING|trainer.py:803] 2025-04-26 16:33:22,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:22,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4318
4361
[WARNING|trainer.py:803] 2025-04-26 16:33:23,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4381
[WARNING|trainer.py:803] 2025-04-26 16:33:23,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:23,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4319
4362
[WARNING|trainer.py:803] 2025-04-26 16:33:24,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4382
[WARNING|trainer.py:803] 2025-04-26 16:33:24,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:24,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4320
4363
[WARNING|trainer.py:803] 2025-04-26 16:33:25,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4383
[WARNING|trainer.py:803] 2025-04-26 16:33:26,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:26,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4364
[WARNING|trainer.py:803] 2025-04-26 16:33:26,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4321
4384
[WARNING|trainer.py:803] 2025-04-26 16:33:27,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:27,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4365
[WARNING|trainer.py:803] 2025-04-26 16:33:27,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4322
4385
[WARNING|trainer.py:803] 2025-04-26 16:33:28,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:28,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4366
[WARNING|trainer.py:803] 2025-04-26 16:33:28,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4323
4386
[WARNING|trainer.py:803] 2025-04-26 16:33:29,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:29,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:30,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4367
4324
4387
[WARNING|trainer.py:803] 2025-04-26 16:33:30,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:31,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:31,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4368
4325
4388
[WARNING|trainer.py:803] 2025-04-26 16:33:32,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:33:32,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:32,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4369
4389
4326
[WARNING|trainer.py:803] 2025-04-26 16:33:33,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:33,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4370
[WARNING|trainer.py:803] 2025-04-26 16:33:33,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4390
4327
[WARNING|trainer.py:803] 2025-04-26 16:33:34,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4371
[WARNING|trainer.py:803] 2025-04-26 16:33:34,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:35,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4391
4328
[WARNING|trainer.py:803] 2025-04-26 16:33:35,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4372
[WARNING|trainer.py:803] 2025-04-26 16:33:36,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:36,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4392
4329
[WARNING|trainer.py:803] 2025-04-26 16:33:36,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4373
[WARNING|trainer.py:803] 2025-04-26 16:33:37,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:37,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4393
4330
[WARNING|trainer.py:803] 2025-04-26 16:33:38,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4374
[WARNING|trainer.py:803] 2025-04-26 16:33:38,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:38,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4394
4331
[WARNING|trainer.py:803] 2025-04-26 16:33:39,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4375
[WARNING|trainer.py:803] 2025-04-26 16:33:39,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:39,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4395
4332
[WARNING|trainer.py:803] 2025-04-26 16:33:40,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4376
[WARNING|trainer.py:803] 2025-04-26 16:33:40,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:41,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4396
4333
[WARNING|trainer.py:803] 2025-04-26 16:33:41,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4377
[WARNING|trainer.py:803] 2025-04-26 16:33:42,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4397
[WARNING|trainer.py:803] 2025-04-26 16:33:42,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:42,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4334
4378
[WARNING|trainer.py:803] 2025-04-26 16:33:43,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4398
[WARNING|trainer.py:803] 2025-04-26 16:33:43,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:43,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4335
4379
[WARNING|trainer.py:803] 2025-04-26 16:33:44,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
4399
[WARNING|trainer.py:803] 2025-04-26 16:33:44,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:45,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4336
4380
[WARNING|trainer.py:803] 2025-04-26 16:33:45,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4400
[WARNING|trainer.py:803] 2025-04-26 16:33:46,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:46,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4337
4381
[WARNING|trainer.py:803] 2025-04-26 16:33:46,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4401
[WARNING|trainer.py:803] 2025-04-26 16:33:47,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:47,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4338
4382
[WARNING|trainer.py:803] 2025-04-26 16:33:47,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4402
[WARNING|trainer.py:803] 2025-04-26 16:33:48,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:33:48,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4339
4383
[WARNING|trainer.py:803] 2025-04-26 16:33:49,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4403
[WARNING|trainer.py:803] 2025-04-26 16:33:49,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:49,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4340
4384
[WARNING|trainer.py:803] 2025-04-26 16:33:50,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4404
[WARNING|trainer.py:803] 2025-04-26 16:33:51,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:51,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4341
4385
[WARNING|trainer.py:803] 2025-04-26 16:33:51,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4405
[WARNING|trainer.py:803] 2025-04-26 16:33:52,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:52,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4386
[WARNING|trainer.py:803] 2025-04-26 16:33:52,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4342
4406
[WARNING|trainer.py:803] 2025-04-26 16:33:53,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:53,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4387
[WARNING|trainer.py:803] 2025-04-26 16:33:53,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4343
4407
[WARNING|trainer.py:803] 2025-04-26 16:33:54,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:54,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4388
[WARNING|trainer.py:803] 2025-04-26 16:33:55,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4344
4408
[WARNING|trainer.py:803] 2025-04-26 16:33:55,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:56,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4389
[WARNING|trainer.py:803] 2025-04-26 16:33:56,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4345
4409
[WARNING|trainer.py:803] 2025-04-26 16:33:57,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:57,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4390
[WARNING|trainer.py:803] 2025-04-26 16:33:57,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4346
4410
[WARNING|trainer.py:803] 2025-04-26 16:33:58,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:33:58,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4391
[WARNING|trainer.py:803] 2025-04-26 16:33:58,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4347
4411
[WARNING|trainer.py:803] 2025-04-26 16:33:59,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:33:59,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4392
[WARNING|trainer.py:803] 2025-04-26 16:33:59,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4412
4348
[WARNING|trainer.py:803] 2025-04-26 16:34:00,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:01,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4393
[WARNING|trainer.py:803] 2025-04-26 16:34:01,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4413
4349
[WARNING|trainer.py:803] 2025-04-26 16:34:01,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:02,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4394
[WARNING|trainer.py:803] 2025-04-26 16:34:02,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4414
4350
[WARNING|trainer.py:803] 2025-04-26 16:34:03,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:03,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4395
[WARNING|trainer.py:803] 2025-04-26 16:34:03,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4415
4351
[WARNING|trainer.py:803] 2025-04-26 16:34:04,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:04,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4396
[WARNING|trainer.py:803] 2025-04-26 16:34:04,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4416
4352
[WARNING|trainer.py:803] 2025-04-26 16:34:05,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:05,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4397
[WARNING|trainer.py:803] 2025-04-26 16:34:05,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4417
4353
[WARNING|trainer.py:803] 2025-04-26 16:34:06,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:06,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4398
[WARNING|trainer.py:803] 2025-04-26 16:34:07,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4418
4354
[WARNING|trainer.py:803] 2025-04-26 16:34:07,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
[WARNING|trainer.py:803] 2025-04-26 16:34:08,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4399
[WARNING|trainer.py:803] 2025-04-26 16:34:08,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4419
4355
[WARNING|trainer.py:803] 2025-04-26 16:34:09,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:09,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4400
[WARNING|trainer.py:803] 2025-04-26 16:34:09,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4420
4356
[WARNING|trainer.py:803] 2025-04-26 16:34:10,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:10,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4401
[WARNING|trainer.py:803] 2025-04-26 16:34:10,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4421
4357
[WARNING|trainer.py:803] 2025-04-26 16:34:11,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:11,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4402
[WARNING|trainer.py:803] 2025-04-26 16:34:11,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4422
4358
[WARNING|trainer.py:803] 2025-04-26 16:34:12,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:12,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4403
[WARNING|trainer.py:803] 2025-04-26 16:34:13,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4423
4359
[WARNING|trainer.py:803] 2025-04-26 16:34:13,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:14,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4404
[WARNING|trainer.py:803] 2025-04-26 16:34:14,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4424
4360
[WARNING|trainer.py:803] 2025-04-26 16:34:15,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:15,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4405
[WARNING|trainer.py:803] 2025-04-26 16:34:15,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4425
4361
[WARNING|trainer.py:803] 2025-04-26 16:34:16,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:16,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4406
[WARNING|trainer.py:803] 2025-04-26 16:34:16,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4426
4362
[WARNING|trainer.py:803] 2025-04-26 16:34:17,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:17,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4407
[WARNING|trainer.py:803] 2025-04-26 16:34:17,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4427
4363
[WARNING|trainer.py:803] 2025-04-26 16:34:18,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:18,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4408
[WARNING|trainer.py:803] 2025-04-26 16:34:19,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4428
4364
[WARNING|trainer.py:803] 2025-04-26 16:34:19,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:20,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4409
[WARNING|trainer.py:803] 2025-04-26 16:34:20,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4429
4365
[WARNING|trainer.py:803] 2025-04-26 16:34:21,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:21,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4410
[WARNING|trainer.py:803] 2025-04-26 16:34:21,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4430
4366
[WARNING|trainer.py:803] 2025-04-26 16:34:22,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:22,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4411
[WARNING|trainer.py:803] 2025-04-26 16:34:22,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4431
4367
[WARNING|trainer.py:803] 2025-04-26 16:34:23,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:23,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4412
[WARNING|trainer.py:803] 2025-04-26 16:34:23,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4432
4368
[WARNING|trainer.py:803] 2025-04-26 16:34:24,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:24,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4413
[WARNING|trainer.py:803] 2025-04-26 16:34:25,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4433
4369
[WARNING|trainer.py:803] 2025-04-26 16:34:25,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:26,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4414
[WARNING|trainer.py:803] 2025-04-26 16:34:26,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4434
4370
[WARNING|trainer.py:803] 2025-04-26 16:34:27,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:27,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4415
[WARNING|trainer.py:803] 2025-04-26 16:34:27,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4435
4371
[WARNING|trainer.py:803] 2025-04-26 16:34:28,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:28,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4416
[WARNING|trainer.py:803] 2025-04-26 16:34:28,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4436
4372
[WARNING|trainer.py:803] 2025-04-26 16:34:29,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:29,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4417
[WARNING|trainer.py:803] 2025-04-26 16:34:29,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4437
4373
[WARNING|trainer.py:803] 2025-04-26 16:34:30,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:30,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4418
[WARNING|trainer.py:803] 2025-04-26 16:34:31,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4438
4374
[WARNING|trainer.py:803] 2025-04-26 16:34:31,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:32,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4419
[WARNING|trainer.py:803] 2025-04-26 16:34:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4439
4375
[WARNING|trainer.py:803] 2025-04-26 16:34:33,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:33,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4420
[WARNING|trainer.py:803] 2025-04-26 16:34:33,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4440
4376
[WARNING|trainer.py:803] 2025-04-26 16:34:34,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:34,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4421
[WARNING|trainer.py:803] 2025-04-26 16:34:34,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4441
4377
[WARNING|trainer.py:803] 2025-04-26 16:34:35,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:35,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4422
[WARNING|trainer.py:803] 2025-04-26 16:34:35,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4442
4378
[WARNING|trainer.py:803] 2025-04-26 16:34:36,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:36,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4423
[WARNING|trainer.py:803] 2025-04-26 16:34:37,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4443
4379
[WARNING|trainer.py:803] 2025-04-26 16:34:37,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:38,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4424
[WARNING|trainer.py:803] 2025-04-26 16:34:38,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4444
4380
[WARNING|trainer.py:803] 2025-04-26 16:34:38,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:39,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4425
[WARNING|trainer.py:803] 2025-04-26 16:34:39,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4445
4381
[WARNING|trainer.py:803] 2025-04-26 16:34:40,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4426
[WARNING|trainer.py:803] 2025-04-26 16:34:40,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4446
4382
[WARNING|trainer.py:803] 2025-04-26 16:34:41,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:41,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4427
[WARNING|trainer.py:803] 2025-04-26 16:34:41,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4447
4383
[WARNING|trainer.py:803] 2025-04-26 16:34:42,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:42,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4428
[WARNING|trainer.py:803] 2025-04-26 16:34:43,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4448
4384
[WARNING|trainer.py:803] 2025-04-26 16:34:43,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:43,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4429
[WARNING|trainer.py:803] 2025-04-26 16:34:44,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4449
4385
[WARNING|trainer.py:803] 2025-04-26 16:34:44,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:45,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4430
[WARNING|trainer.py:803] 2025-04-26 16:34:45,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4450
4386
[WARNING|trainer.py:803] 2025-04-26 16:34:46,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:46,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4431
[WARNING|trainer.py:803] 2025-04-26 16:34:46,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4451
4387
[WARNING|trainer.py:803] 2025-04-26 16:34:47,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:47,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4432
[WARNING|trainer.py:803] 2025-04-26 16:34:47,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4452
4388
[WARNING|trainer.py:803] 2025-04-26 16:34:48,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:48,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4433
4453
[WARNING|trainer.py:803] 2025-04-26 16:34:49,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4389
[WARNING|trainer.py:803] 2025-04-26 16:34:49,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:49,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4434
4454
[WARNING|trainer.py:803] 2025-04-26 16:34:50,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4390
[WARNING|trainer.py:803] 2025-04-26 16:34:50,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:51,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4435
4455
[WARNING|trainer.py:803] 2025-04-26 16:34:51,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4391
[WARNING|trainer.py:803] 2025-04-26 16:34:52,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:52,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4436
4456
[WARNING|trainer.py:803] 2025-04-26 16:34:52,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4392
[WARNING|trainer.py:803] 2025-04-26 16:34:53,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:53,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4437
4457
[WARNING|trainer.py:803] 2025-04-26 16:34:53,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4393
[WARNING|trainer.py:803] 2025-04-26 16:34:54,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:54,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4438
4458
[WARNING|trainer.py:803] 2025-04-26 16:34:55,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4394
[WARNING|trainer.py:803] 2025-04-26 16:34:55,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:55,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4439
4459
[WARNING|trainer.py:803] 2025-04-26 16:34:56,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4395
[WARNING|trainer.py:803] 2025-04-26 16:34:56,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:56,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4440
4460
[WARNING|trainer.py:803] 2025-04-26 16:34:57,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4396
[WARNING|trainer.py:803] 2025-04-26 16:34:58,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:34:58,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4441
4461
[WARNING|trainer.py:803] 2025-04-26 16:34:58,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4397
[WARNING|trainer.py:803] 2025-04-26 16:34:59,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:34:59,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4442
4462
[WARNING|trainer.py:803] 2025-04-26 16:34:59,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4398
[WARNING|trainer.py:803] 2025-04-26 16:35:00,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:00,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4443
4463
[WARNING|trainer.py:803] 2025-04-26 16:35:01,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
4399
[WARNING|trainer.py:803] 2025-04-26 16:35:01,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:01,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4444
4464
[WARNING|trainer.py:803] 2025-04-26 16:35:02,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4400
[WARNING|trainer.py:803] 2025-04-26 16:35:02,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:02,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4445
4465
[WARNING|trainer.py:803] 2025-04-26 16:35:03,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4401
[WARNING|trainer.py:803] 2025-04-26 16:35:04,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:35:04,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
NoNo
4446
4466
[WARNING|trainer.py:803] 2025-04-26 16:35:04,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4402
[WARNING|trainer.py:803] 2025-04-26 16:35:05,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:05,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4447
4467
[WARNING|trainer.py:803] 2025-04-26 16:35:05,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4403
[WARNING|trainer.py:803] 2025-04-26 16:35:06,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:06,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4448
4468
[WARNING|trainer.py:803] 2025-04-26 16:35:06,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4404
[WARNING|trainer.py:803] 2025-04-26 16:35:07,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:07,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4449
4469
[WARNING|trainer.py:803] 2025-04-26 16:35:08,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4405
[WARNING|trainer.py:803] 2025-04-26 16:35:08,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:08,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4450
4470
[WARNING|trainer.py:803] 2025-04-26 16:35:09,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4406
[WARNING|trainer.py:803] 2025-04-26 16:35:10,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:10,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4471
4451
[WARNING|trainer.py:803] 2025-04-26 16:35:10,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4407
[WARNING|trainer.py:803] 2025-04-26 16:35:11,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:11,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:11,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4472
4452
4408
[WARNING|trainer.py:803] 2025-04-26 16:35:12,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:35:12,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:12,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4473
4453
4409
[WARNING|trainer.py:803] 2025-04-26 16:35:13,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:13,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:14,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4474
4454
4410
[WARNING|trainer.py:803] 2025-04-26 16:35:14,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:14,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:15,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4475
4455
4411
[WARNING|trainer.py:803] 2025-04-26 16:35:16,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:16,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:16,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4476
4456
4412
[WARNING|trainer.py:803] 2025-04-26 16:35:17,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:17,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:17,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4477
4457
4413
[WARNING|trainer.py:803] 2025-04-26 16:35:18,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:18,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:18,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4478
4458
4414
[WARNING|trainer.py:803] 2025-04-26 16:35:19,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:19,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:35:19,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4479
4459
4415
[WARNING|trainer.py:803] 2025-04-26 16:35:20,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:20,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:21,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4480
4460
4416
[WARNING|trainer.py:803] 2025-04-26 16:35:22,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:22,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:22,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4481
4461
4417
[WARNING|trainer.py:803] 2025-04-26 16:35:23,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:23,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:23,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4482
4462
4418
[WARNING|trainer.py:803] 2025-04-26 16:35:24,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:24,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:24,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4483
4463
4419
[WARNING|trainer.py:803] 2025-04-26 16:35:25,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:25,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:25,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4484
4464
4420
[WARNING|trainer.py:803] 2025-04-26 16:35:26,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:26,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4485
[WARNING|trainer.py:803] 2025-04-26 16:35:27,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4465
4421
[WARNING|trainer.py:803] 2025-04-26 16:35:27,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:28,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4486
[WARNING|trainer.py:803] 2025-04-26 16:35:28,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4466
4422
[WARNING|trainer.py:803] 2025-04-26 16:35:29,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:29,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4487
[WARNING|trainer.py:803] 2025-04-26 16:35:29,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4467
4423
[WARNING|trainer.py:803] 2025-04-26 16:35:30,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:30,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4488
[WARNING|trainer.py:803] 2025-04-26 16:35:30,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4468
4424
[WARNING|trainer.py:803] 2025-04-26 16:35:31,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:31,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4489
[WARNING|trainer.py:803] 2025-04-26 16:35:31,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4469
4425
[WARNING|trainer.py:803] 2025-04-26 16:35:32,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:32,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4490
[WARNING|trainer.py:803] 2025-04-26 16:35:33,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4470
4426
[WARNING|trainer.py:803] 2025-04-26 16:35:33,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:34,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4491
[WARNING|trainer.py:803] 2025-04-26 16:35:34,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4471
4427
[WARNING|trainer.py:803] 2025-04-26 16:35:35,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:35,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:35,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4492
4472
4428
[WARNING|trainer.py:803] 2025-04-26 16:35:36,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:36,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:35:36,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4493
4473
4429
[WARNING|trainer.py:803] 2025-04-26 16:35:37,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:37,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:37,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4494
4474
4430
[WARNING|trainer.py:803] 2025-04-26 16:35:38,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:38,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:39,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4495
4475
4431
[WARNING|trainer.py:803] 2025-04-26 16:35:40,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:40,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:40,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4496
4476
4432
[WARNING|trainer.py:803] 2025-04-26 16:35:41,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:41,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:41,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4497
4477
4433
[WARNING|trainer.py:803] 2025-04-26 16:35:42,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:42,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:42,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4498
4478
4434
[WARNING|trainer.py:803] 2025-04-26 16:35:43,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:43,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:43,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4479
4499
4435
[WARNING|trainer.py:803] 2025-04-26 16:35:44,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:45,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:45,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4480
4500
4436
[WARNING|trainer.py:803] 2025-04-26 16:35:46,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:46,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4481
4501
4437
[WARNING|trainer.py:803] 2025-04-26 16:35:47,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:47,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:35:47,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4482
4502
4438
[WARNING|trainer.py:803] 2025-04-26 16:35:48,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:48,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:48,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4483
4503
4439
[WARNING|trainer.py:803] 2025-04-26 16:35:49,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:49,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:49,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4484
4504
4440
[WARNING|trainer.py:803] 2025-04-26 16:35:50,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:51,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:51,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4485
4505
4441
[WARNING|trainer.py:803] 2025-04-26 16:35:52,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:52,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:52,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4486
4506
4442
[WARNING|trainer.py:803] 2025-04-26 16:35:53,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:53,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:53,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4487
4507
4443
[WARNING|trainer.py:803] 2025-04-26 16:35:54,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:54,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:54,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4488
4508
4444
[WARNING|trainer.py:803] 2025-04-26 16:35:55,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:55,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:55,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4489
4509
4445
[WARNING|trainer.py:803] 2025-04-26 16:35:56,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:57,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:57,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4490
4510
4446
[WARNING|trainer.py:803] 2025-04-26 16:35:58,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:58,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:58,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4491
4511
4447
[WARNING|trainer.py:803] 2025-04-26 16:35:59,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:35:59,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:35:59,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4492
4512
4448
[WARNING|trainer.py:803] 2025-04-26 16:36:00,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:00,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:00,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4493
4513
4449
[WARNING|trainer.py:803] 2025-04-26 16:36:01,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:01,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:01,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4494
4514
4450
[WARNING|trainer.py:803] 2025-04-26 16:36:02,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:03,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:03,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4495
4515
4451
[WARNING|trainer.py:803] 2025-04-26 16:36:04,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:04,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:36:04,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4496
4516
4452
[WARNING|trainer.py:803] 2025-04-26 16:36:05,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:05,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:05,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4497
4517
4453
[WARNING|trainer.py:803] 2025-04-26 16:36:06,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:06,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:06,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4498
4518
4454
[WARNING|trainer.py:803] 2025-04-26 16:36:07,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:07,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:07,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4519
4455
4499
[WARNING|trainer.py:803] 2025-04-26 16:36:09,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:09,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:09,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4520
4456
4500
[WARNING|trainer.py:803] 2025-04-26 16:36:10,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:10,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:10,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4521
4457
4501
[WARNING|trainer.py:803] 2025-04-26 16:36:11,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:36:11,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:11,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4522
4458
4502
[WARNING|trainer.py:803] 2025-04-26 16:36:12,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:12,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:36:12,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4523
4459
4503
[WARNING|trainer.py:803] 2025-04-26 16:36:13,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:14,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:14,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4524
4460
4504
[WARNING|trainer.py:803] 2025-04-26 16:36:15,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:15,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:15,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4525
4461
4505
[WARNING|trainer.py:803] 2025-04-26 16:36:16,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:16,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:16,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4526
4462
4506
[WARNING|trainer.py:803] 2025-04-26 16:36:17,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:17,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:17,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4527
4463
4507
[WARNING|trainer.py:803] 2025-04-26 16:36:18,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:18,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:18,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4528
4464
4508
[WARNING|trainer.py:803] 2025-04-26 16:36:20,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:20,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:20,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4529
4465
4509
[WARNING|trainer.py:803] 2025-04-26 16:36:21,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:21,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:21,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4530
4466
4510
[WARNING|trainer.py:803] 2025-04-26 16:36:22,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:22,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:22,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4531
4467
4511
[WARNING|trainer.py:803] 2025-04-26 16:36:23,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:23,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:23,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4532
4468
4512
[WARNING|trainer.py:803] 2025-04-26 16:36:24,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:24,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:25,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4533
4469
4513
[WARNING|trainer.py:803] 2025-04-26 16:36:26,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:26,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4470
4534
[WARNING|trainer.py:803] 2025-04-26 16:36:26,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4514
[WARNING|trainer.py:803] 2025-04-26 16:36:27,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:27,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4471
[WARNING|trainer.py:803] 2025-04-26 16:36:27,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4535
4515
[WARNING|trainer.py:803] 2025-04-26 16:36:28,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:28,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4472
[WARNING|trainer.py:803] 2025-04-26 16:36:28,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4536
4516
[WARNING|trainer.py:803] 2025-04-26 16:36:29,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:36:29,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4473
[WARNING|trainer.py:803] 2025-04-26 16:36:30,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4537
4517
[WARNING|trainer.py:803] 2025-04-26 16:36:30,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:30,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4474
[WARNING|trainer.py:803] 2025-04-26 16:36:31,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4538
4518
[WARNING|trainer.py:803] 2025-04-26 16:36:32,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:32,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4475
[WARNING|trainer.py:803] 2025-04-26 16:36:32,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4539
4519
[WARNING|trainer.py:803] 2025-04-26 16:36:33,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:33,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4476
[WARNING|trainer.py:803] 2025-04-26 16:36:33,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4540
4520
[WARNING|trainer.py:803] 2025-04-26 16:36:34,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:34,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4477
[WARNING|trainer.py:803] 2025-04-26 16:36:34,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4541
4521
[WARNING|trainer.py:803] 2025-04-26 16:36:35,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:35,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4478
[WARNING|trainer.py:803] 2025-04-26 16:36:36,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4542
4522
[WARNING|trainer.py:803] 2025-04-26 16:36:36,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:36,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4479
[WARNING|trainer.py:803] 2025-04-26 16:36:37,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4543
4523
[WARNING|trainer.py:803] 2025-04-26 16:36:38,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:38,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4480
[WARNING|trainer.py:803] 2025-04-26 16:36:38,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4544
4524
[WARNING|trainer.py:803] 2025-04-26 16:36:39,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:39,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4481
[WARNING|trainer.py:803] 2025-04-26 16:36:39,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4545
4525
[WARNING|trainer.py:803] 2025-04-26 16:36:40,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:40,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4482
[WARNING|trainer.py:803] 2025-04-26 16:36:40,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4546
4526
[WARNING|trainer.py:803] 2025-04-26 16:36:41,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:41,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4483
[WARNING|trainer.py:803] 2025-04-26 16:36:42,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4547
4527
[WARNING|trainer.py:803] 2025-04-26 16:36:42,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:43,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4484
[WARNING|trainer.py:803] 2025-04-26 16:36:43,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4548
4528
[WARNING|trainer.py:803] 2025-04-26 16:36:43,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:44,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4485
[WARNING|trainer.py:803] 2025-04-26 16:36:44,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4549
4529
[WARNING|trainer.py:803] 2025-04-26 16:36:45,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:36:45,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4486
[WARNING|trainer.py:803] 2025-04-26 16:36:45,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4550
4530
[WARNING|trainer.py:803] 2025-04-26 16:36:46,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:46,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4487
[WARNING|trainer.py:803] 2025-04-26 16:36:46,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4551
4531
[WARNING|trainer.py:803] 2025-04-26 16:36:47,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:47,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4488
[WARNING|trainer.py:803] 2025-04-26 16:36:48,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4552
4532
[WARNING|trainer.py:803] 2025-04-26 16:36:48,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:49,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4489
[WARNING|trainer.py:803] 2025-04-26 16:36:49,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4553
4533
[WARNING|trainer.py:803] 2025-04-26 16:36:49,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4490
[WARNING|trainer.py:803] 2025-04-26 16:36:50,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:50,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4554
4534
[WARNING|trainer.py:803] 2025-04-26 16:36:51,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:51,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4491
[WARNING|trainer.py:803] 2025-04-26 16:36:51,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4555
4535
[WARNING|trainer.py:803] 2025-04-26 16:36:52,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:52,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4492
[WARNING|trainer.py:803] 2025-04-26 16:36:52,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4556
4536
[WARNING|trainer.py:803] 2025-04-26 16:36:53,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:53,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4493
[WARNING|trainer.py:803] 2025-04-26 16:36:54,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4557
4537
[WARNING|trainer.py:803] 2025-04-26 16:36:54,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:55,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4494
[WARNING|trainer.py:803] 2025-04-26 16:36:55,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4558
4538
[WARNING|trainer.py:803] 2025-04-26 16:36:55,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:56,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4495
[WARNING|trainer.py:803] 2025-04-26 16:36:56,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4559
4539
[WARNING|trainer.py:803] 2025-04-26 16:36:57,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:57,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4496
[WARNING|trainer.py:803] 2025-04-26 16:36:57,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4560
4540
[WARNING|trainer.py:803] 2025-04-26 16:36:58,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:58,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4497
[WARNING|trainer.py:803] 2025-04-26 16:36:58,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4561
4541
[WARNING|trainer.py:803] 2025-04-26 16:36:59,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:36:59,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4498
[WARNING|trainer.py:803] 2025-04-26 16:37:00,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4562
4542
[WARNING|trainer.py:803] 2025-04-26 16:37:00,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:01,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4499
[WARNING|trainer.py:803] 2025-04-26 16:37:01,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4563
4543
[WARNING|trainer.py:803] 2025-04-26 16:37:02,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:02,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4500
[WARNING|trainer.py:803] 2025-04-26 16:37:02,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4564
4544
[WARNING|trainer.py:803] 2025-04-26 16:37:03,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:03,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4501
[WARNING|trainer.py:803] 2025-04-26 16:37:03,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4565
4545
[WARNING|trainer.py:803] 2025-04-26 16:37:04,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:37:04,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4502
[WARNING|trainer.py:803] 2025-04-26 16:37:04,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4566
4546
[WARNING|trainer.py:803] 2025-04-26 16:37:05,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:05,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4503
[WARNING|trainer.py:803] 2025-04-26 16:37:06,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4567
4547
[WARNING|trainer.py:803] 2025-04-26 16:37:06,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:07,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4504
[WARNING|trainer.py:803] 2025-04-26 16:37:07,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4568
4548
[WARNING|trainer.py:803] 2025-04-26 16:37:08,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:08,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4505
[WARNING|trainer.py:803] 2025-04-26 16:37:08,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4569
4549
[WARNING|trainer.py:803] 2025-04-26 16:37:09,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:09,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4506
[WARNING|trainer.py:803] 2025-04-26 16:37:09,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4570
4550
[WARNING|trainer.py:803] 2025-04-26 16:37:10,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:10,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4507
[WARNING|trainer.py:803] 2025-04-26 16:37:10,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4571
4551
[WARNING|trainer.py:803] 2025-04-26 16:37:11,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:11,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4508
[WARNING|trainer.py:803] 2025-04-26 16:37:12,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4572
4552
[WARNING|trainer.py:803] 2025-04-26 16:37:12,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:13,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4509
[WARNING|trainer.py:803] 2025-04-26 16:37:13,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4573
4553
[WARNING|trainer.py:803] 2025-04-26 16:37:14,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:14,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4510
[WARNING|trainer.py:803] 2025-04-26 16:37:14,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4574
4554
[WARNING|trainer.py:803] 2025-04-26 16:37:15,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:15,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4511
[WARNING|trainer.py:803] 2025-04-26 16:37:15,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4575
4555
[WARNING|trainer.py:803] 2025-04-26 16:37:16,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:16,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4512
[WARNING|trainer.py:803] 2025-04-26 16:37:17,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4576
4556
[WARNING|trainer.py:803] 2025-04-26 16:37:17,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:17,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4513
[WARNING|trainer.py:803] 2025-04-26 16:37:18,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4577
4557
[WARNING|trainer.py:803] 2025-04-26 16:37:18,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:19,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4514
[WARNING|trainer.py:803] 2025-04-26 16:37:19,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4578
4558
[WARNING|trainer.py:803] 2025-04-26 16:37:20,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:20,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4515
[WARNING|trainer.py:803] 2025-04-26 16:37:20,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4579
4559
[WARNING|trainer.py:803] 2025-04-26 16:37:21,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:37:21,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4516
[WARNING|trainer.py:803] 2025-04-26 16:37:21,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4580
4560
[WARNING|trainer.py:803] 2025-04-26 16:37:22,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:22,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4517
[WARNING|trainer.py:803] 2025-04-26 16:37:23,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4581
4561
[WARNING|trainer.py:803] 2025-04-26 16:37:23,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:23,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4518
[WARNING|trainer.py:803] 2025-04-26 16:37:24,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4582
4562
[WARNING|trainer.py:803] 2025-04-26 16:37:24,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:25,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4519
[WARNING|trainer.py:803] 2025-04-26 16:37:25,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4583
4563
[WARNING|trainer.py:803] 2025-04-26 16:37:25,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:26,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4520
[WARNING|trainer.py:803] 2025-04-26 16:37:26,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4584
4564
[WARNING|trainer.py:803] 2025-04-26 16:37:27,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4521
[WARNING|trainer.py:803] 2025-04-26 16:37:27,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:27,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4585
4565
[WARNING|trainer.py:803] 2025-04-26 16:37:28,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4522
[WARNING|trainer.py:803] 2025-04-26 16:37:28,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:37:29,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4586
4566
[WARNING|trainer.py:803] 2025-04-26 16:37:29,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4523
[WARNING|trainer.py:803] 2025-04-26 16:37:29,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:37:30,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4587
4567
[WARNING|trainer.py:803] 2025-04-26 16:37:30,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4524
[WARNING|trainer.py:803] 2025-04-26 16:37:31,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4588
[WARNING|trainer.py:803] 2025-04-26 16:37:31,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4568
[WARNING|trainer.py:803] 2025-04-26 16:37:31,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4525
[WARNING|trainer.py:803] 2025-04-26 16:37:32,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4589
[WARNING|trainer.py:803] 2025-04-26 16:37:32,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4569
[WARNING|trainer.py:803] 2025-04-26 16:37:33,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4526
[WARNING|trainer.py:803] 2025-04-26 16:37:33,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4590
[WARNING|trainer.py:803] 2025-04-26 16:37:33,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4570
[WARNING|trainer.py:803] 2025-04-26 16:37:34,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4527
[WARNING|trainer.py:803] 2025-04-26 16:37:34,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4591
[WARNING|trainer.py:803] 2025-04-26 16:37:35,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4571
[WARNING|trainer.py:803] 2025-04-26 16:37:35,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:35,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4528
4592
[WARNING|trainer.py:803] 2025-04-26 16:37:36,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4572
[WARNING|trainer.py:803] 2025-04-26 16:37:36,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:37,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4529
4593
[WARNING|trainer.py:803] 2025-04-26 16:37:37,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4573
[WARNING|trainer.py:803] 2025-04-26 16:37:37,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:38,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4530
4594
[WARNING|trainer.py:803] 2025-04-26 16:37:38,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4574
[WARNING|trainer.py:803] 2025-04-26 16:37:39,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:39,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4531
4595
[WARNING|trainer.py:803] 2025-04-26 16:37:39,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4575
[WARNING|trainer.py:803] 2025-04-26 16:37:40,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:40,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4532
4596
[WARNING|trainer.py:803] 2025-04-26 16:37:41,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4576
[WARNING|trainer.py:803] 2025-04-26 16:37:41,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:41,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4533
4597
[WARNING|trainer.py:803] 2025-04-26 16:37:42,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4577
[WARNING|trainer.py:803] 2025-04-26 16:37:42,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:43,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4534
4598
[WARNING|trainer.py:803] 2025-04-26 16:37:43,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4578
[WARNING|trainer.py:803] 2025-04-26 16:37:43,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:44,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4535
4599
[WARNING|trainer.py:803] 2025-04-26 16:37:44,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4579
[WARNING|trainer.py:803] 2025-04-26 16:37:45,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:45,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4536
4600
[WARNING|trainer.py:803] 2025-04-26 16:37:45,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4580
[WARNING|trainer.py:803] 2025-04-26 16:37:46,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:46,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4537
4601
[WARNING|trainer.py:803] 2025-04-26 16:37:47,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4581
[WARNING|trainer.py:803] 2025-04-26 16:37:47,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:47,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4538
4602
[WARNING|trainer.py:803] 2025-04-26 16:37:48,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4582
[WARNING|trainer.py:803] 2025-04-26 16:37:48,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4539
[WARNING|trainer.py:803] 2025-04-26 16:37:49,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4603
[WARNING|trainer.py:803] 2025-04-26 16:37:49,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4583
[WARNING|trainer.py:803] 2025-04-26 16:37:49,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4540
[WARNING|trainer.py:803] 2025-04-26 16:37:50,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4604
[WARNING|trainer.py:803] 2025-04-26 16:37:50,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4584
[WARNING|trainer.py:803] 2025-04-26 16:37:51,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4541
[WARNING|trainer.py:803] 2025-04-26 16:37:51,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4605
[WARNING|trainer.py:803] 2025-04-26 16:37:51,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4585
[WARNING|trainer.py:803] 2025-04-26 16:37:52,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4542
[WARNING|trainer.py:803] 2025-04-26 16:37:52,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4606
[WARNING|trainer.py:803] 2025-04-26 16:37:53,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4586
[WARNING|trainer.py:803] 2025-04-26 16:37:53,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4543
[WARNING|trainer.py:803] 2025-04-26 16:37:53,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4607
[WARNING|trainer.py:803] 2025-04-26 16:37:54,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4587
[WARNING|trainer.py:803] 2025-04-26 16:37:54,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4544
[WARNING|trainer.py:803] 2025-04-26 16:37:55,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4608
[WARNING|trainer.py:803] 2025-04-26 16:37:55,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:37:55,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
4588
NoNo
4545
[WARNING|trainer.py:803] 2025-04-26 16:37:56,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4609
[WARNING|trainer.py:803] 2025-04-26 16:37:56,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:57,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4589
4546
[WARNING|trainer.py:803] 2025-04-26 16:37:57,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4610
[WARNING|trainer.py:803] 2025-04-26 16:37:57,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4590
[WARNING|trainer.py:803] 2025-04-26 16:37:58,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:58,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4547
4611
[WARNING|trainer.py:803] 2025-04-26 16:37:59,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4591
[WARNING|trainer.py:803] 2025-04-26 16:37:59,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:37:59,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4548
4612
[WARNING|trainer.py:803] 2025-04-26 16:38:00,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4592
[WARNING|trainer.py:803] 2025-04-26 16:38:00,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:01,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4549
4613
[WARNING|trainer.py:803] 2025-04-26 16:38:01,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4593
[WARNING|trainer.py:803] 2025-04-26 16:38:02,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:02,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4550
4614
[WARNING|trainer.py:803] 2025-04-26 16:38:02,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4594
[WARNING|trainer.py:803] 2025-04-26 16:38:03,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:03,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4551
4615
[WARNING|trainer.py:803] 2025-04-26 16:38:03,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4595
[WARNING|trainer.py:803] 2025-04-26 16:38:04,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:04,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4552
4616
[WARNING|trainer.py:803] 2025-04-26 16:38:05,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4596
[WARNING|trainer.py:803] 2025-04-26 16:38:05,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:05,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4553
4617
[WARNING|trainer.py:803] 2025-04-26 16:38:06,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4597
[WARNING|trainer.py:803] 2025-04-26 16:38:06,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:07,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4554
[WARNING|trainer.py:803] 2025-04-26 16:38:07,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4618
4598
[WARNING|trainer.py:803] 2025-04-26 16:38:08,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:08,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4555
[WARNING|trainer.py:803] 2025-04-26 16:38:08,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4619
4599
[WARNING|trainer.py:803] 2025-04-26 16:38:09,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:09,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4556
[WARNING|trainer.py:803] 2025-04-26 16:38:09,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4620
4600
[WARNING|trainer.py:803] 2025-04-26 16:38:10,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:10,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4557
[WARNING|trainer.py:803] 2025-04-26 16:38:11,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4621
4601
[WARNING|trainer.py:803] 2025-04-26 16:38:11,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4558
[WARNING|trainer.py:803] 2025-04-26 16:38:11,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:12,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4622
4602
[WARNING|trainer.py:803] 2025-04-26 16:38:12,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4559
[WARNING|trainer.py:803] 2025-04-26 16:38:13,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4623
[WARNING|trainer.py:803] 2025-04-26 16:38:13,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4603
[WARNING|trainer.py:803] 2025-04-26 16:38:13,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4560
[WARNING|trainer.py:803] 2025-04-26 16:38:14,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 16:38:14,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4604
[WARNING|trainer.py:803] 2025-04-26 16:38:15,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4561
[WARNING|trainer.py:803] 2025-04-26 16:38:15,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4625
[WARNING|trainer.py:803] 2025-04-26 16:38:15,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4605
[WARNING|trainer.py:803] 2025-04-26 16:38:16,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4562
[WARNING|trainer.py:803] 2025-04-26 16:38:16,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4626
[WARNING|trainer.py:803] 2025-04-26 16:38:17,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:17,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4606
4563
[WARNING|trainer.py:803] 2025-04-26 16:38:17,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4627
[WARNING|trainer.py:803] 2025-04-26 16:38:18,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:38:18,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4607
4564
[WARNING|trainer.py:803] 2025-04-26 16:38:19,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4628
[WARNING|trainer.py:803] 2025-04-26 16:38:19,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:19,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4608
4565
[WARNING|trainer.py:803] 2025-04-26 16:38:20,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4629
[WARNING|trainer.py:803] 2025-04-26 16:38:20,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:21,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4609
4566
[WARNING|trainer.py:803] 2025-04-26 16:38:21,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4630
[WARNING|trainer.py:803] 2025-04-26 16:38:22,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:22,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4610
4567
[WARNING|trainer.py:803] 2025-04-26 16:38:22,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4631
[WARNING|trainer.py:803] 2025-04-26 16:38:23,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:23,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4611
4568
[WARNING|trainer.py:803] 2025-04-26 16:38:23,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4632
[WARNING|trainer.py:803] 2025-04-26 16:38:24,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:24,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4612
4569
[WARNING|trainer.py:803] 2025-04-26 16:38:25,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4633
[WARNING|trainer.py:803] 2025-04-26 16:38:25,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:25,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4613
4570
[WARNING|trainer.py:803] 2025-04-26 16:38:26,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4634
[WARNING|trainer.py:803] 2025-04-26 16:38:26,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:27,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4614
4571
[WARNING|trainer.py:803] 2025-04-26 16:38:27,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4635
[WARNING|trainer.py:803] 2025-04-26 16:38:28,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:28,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4615
4572
[WARNING|trainer.py:803] 2025-04-26 16:38:28,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4636
[WARNING|trainer.py:803] 2025-04-26 16:38:29,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:29,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4616
4573
[WARNING|trainer.py:803] 2025-04-26 16:38:29,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4637
[WARNING|trainer.py:803] 2025-04-26 16:38:30,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:30,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4617
4574
[WARNING|trainer.py:803] 2025-04-26 16:38:31,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4638
[WARNING|trainer.py:803] 2025-04-26 16:38:31,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:31,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4618
4575
[WARNING|trainer.py:803] 2025-04-26 16:38:32,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4639
[WARNING|trainer.py:803] 2025-04-26 16:38:33,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:33,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4619
4576
[WARNING|trainer.py:803] 2025-04-26 16:38:33,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4640
[WARNING|trainer.py:803] 2025-04-26 16:38:34,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:34,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4620
4577
[WARNING|trainer.py:803] 2025-04-26 16:38:34,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
[WARNING|trainer.py:803] 2025-04-26 16:38:35,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:35,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4621
4578
[WARNING|trainer.py:803] 2025-04-26 16:38:35,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4642
[WARNING|trainer.py:803] 2025-04-26 16:38:36,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:36,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4622
4579
[WARNING|trainer.py:803] 2025-04-26 16:38:37,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4643
[WARNING|trainer.py:803] 2025-04-26 16:38:37,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:37,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4623
4580
[WARNING|trainer.py:803] 2025-04-26 16:38:38,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4644
[WARNING|trainer.py:803] 2025-04-26 16:38:39,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:39,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 16:38:39,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
4581
NoNo
4645
[WARNING|trainer.py:803] 2025-04-26 16:38:40,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:40,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4625
4582
[WARNING|trainer.py:803] 2025-04-26 16:38:40,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4646
[WARNING|trainer.py:803] 2025-04-26 16:38:41,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:41,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4626
[WARNING|trainer.py:803] 2025-04-26 16:38:41,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4583
4647
[WARNING|trainer.py:803] 2025-04-26 16:38:42,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:38:42,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4627
[WARNING|trainer.py:803] 2025-04-26 16:38:43,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4584
4648
[WARNING|trainer.py:803] 2025-04-26 16:38:43,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:44,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:44,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4628
4585
4649
[WARNING|trainer.py:803] 2025-04-26 16:38:45,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:45,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:38:45,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4629
4586
4650
[WARNING|trainer.py:803] 2025-04-26 16:38:46,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:46,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:38:46,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4630
4587
4651
[WARNING|trainer.py:803] 2025-04-26 16:38:47,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:47,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:38:47,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4588
4631
4652
[WARNING|trainer.py:803] 2025-04-26 16:38:48,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:48,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:49,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4589
4632
4653
[WARNING|trainer.py:803] 2025-04-26 16:38:50,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:50,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:50,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4590
4633
4654
[WARNING|trainer.py:803] 2025-04-26 16:38:51,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:51,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:51,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4591
4634
4655
[WARNING|trainer.py:803] 2025-04-26 16:38:52,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:52,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:52,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4592
4635
4656
[WARNING|trainer.py:803] 2025-04-26 16:38:53,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:53,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:54,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4593
4636
4657
[WARNING|trainer.py:803] 2025-04-26 16:38:54,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:54,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4594
4637
[WARNING|trainer.py:803] 2025-04-26 16:38:55,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4658
[WARNING|trainer.py:803] 2025-04-26 16:38:56,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:56,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4595
4638
[WARNING|trainer.py:803] 2025-04-26 16:38:56,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4659
[WARNING|trainer.py:803] 2025-04-26 16:38:57,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:57,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4596
4639
[WARNING|trainer.py:803] 2025-04-26 16:38:57,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4660
[WARNING|trainer.py:803] 2025-04-26 16:38:58,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:38:58,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4597
4640
[WARNING|trainer.py:803] 2025-04-26 16:38:58,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4661
[WARNING|trainer.py:803] 2025-04-26 16:38:59,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:38:59,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4598
[WARNING|trainer.py:803] 2025-04-26 16:39:00,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
4662
[WARNING|trainer.py:803] 2025-04-26 16:39:00,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:00,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4599
[WARNING|trainer.py:803] 2025-04-26 16:39:01,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4642
4663
[WARNING|trainer.py:803] 2025-04-26 16:39:02,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:02,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:02,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4600
4643
4664
[WARNING|trainer.py:803] 2025-04-26 16:39:03,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:03,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:03,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4601
4644
4665
[WARNING|trainer.py:803] 2025-04-26 16:39:04,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:39:04,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:04,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4602
4645
4666
[WARNING|trainer.py:803] 2025-04-26 16:39:05,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:05,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:06,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4603
4646
4667
[WARNING|trainer.py:803] 2025-04-26 16:39:06,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:06,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:07,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4604
4647
4668
[WARNING|trainer.py:803] 2025-04-26 16:39:08,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:08,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:08,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4648
4605
4669
[WARNING|trainer.py:803] 2025-04-26 16:39:09,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:09,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:09,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4649
4606
4670
[WARNING|trainer.py:803] 2025-04-26 16:39:10,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:10,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:39:10,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4650
4607
4671
[WARNING|trainer.py:803] 2025-04-26 16:39:11,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:11,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:11,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4651
4608
4672
[WARNING|trainer.py:803] 2025-04-26 16:39:12,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:13,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:13,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4652
4609
4673
[WARNING|trainer.py:803] 2025-04-26 16:39:14,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:14,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:14,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4653
4610
4674
[WARNING|trainer.py:803] 2025-04-26 16:39:15,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:15,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:15,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4654
4611
4675
[WARNING|trainer.py:803] 2025-04-26 16:39:16,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:16,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:16,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4655
4612
4676
[WARNING|trainer.py:803] 2025-04-26 16:39:17,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:17,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:17,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4656
4677
4613
[WARNING|trainer.py:803] 2025-04-26 16:39:19,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:19,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:19,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4657
4678
4614
[WARNING|trainer.py:803] 2025-04-26 16:39:20,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:20,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:20,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4658
4679
4615
[WARNING|trainer.py:803] 2025-04-26 16:39:21,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:21,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:21,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4659
4680
4616
[WARNING|trainer.py:803] 2025-04-26 16:39:22,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:22,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:22,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4660
4681
4617
[WARNING|trainer.py:803] 2025-04-26 16:39:23,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:23,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:24,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4661
4682
4618
[WARNING|trainer.py:803] 2025-04-26 16:39:25,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:25,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4662
[WARNING|trainer.py:803] 2025-04-26 16:39:25,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4683
4619
[WARNING|trainer.py:803] 2025-04-26 16:39:26,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:26,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4663
[WARNING|trainer.py:803] 2025-04-26 16:39:26,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4684
4620
[WARNING|trainer.py:803] 2025-04-26 16:39:27,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:27,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4664
[WARNING|trainer.py:803] 2025-04-26 16:39:27,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4685
4621
[WARNING|trainer.py:803] 2025-04-26 16:39:28,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:28,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4665
4686
[WARNING|trainer.py:803] 2025-04-26 16:39:29,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4622
[WARNING|trainer.py:803] 2025-04-26 16:39:29,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:29,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4666
4687
[WARNING|trainer.py:803] 2025-04-26 16:39:30,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4623
[WARNING|trainer.py:803] 2025-04-26 16:39:31,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:31,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4667
4688
[WARNING|trainer.py:803] 2025-04-26 16:39:31,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 16:39:32,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:32,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4668
4689
[WARNING|trainer.py:803] 2025-04-26 16:39:32,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4625
[WARNING|trainer.py:803] 2025-04-26 16:39:33,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:33,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4690
4669
[WARNING|trainer.py:803] 2025-04-26 16:39:33,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4626
[WARNING|trainer.py:803] 2025-04-26 16:39:34,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:34,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4691
4670
[WARNING|trainer.py:803] 2025-04-26 16:39:35,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4627
[WARNING|trainer.py:803] 2025-04-26 16:39:35,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:35,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4692
4671
[WARNING|trainer.py:803] 2025-04-26 16:39:36,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4628
[WARNING|trainer.py:803] 2025-04-26 16:39:37,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:37,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4693
[WARNING|trainer.py:803] 2025-04-26 16:39:37,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4672
4629
[WARNING|trainer.py:803] 2025-04-26 16:39:38,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:38,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4694
[WARNING|trainer.py:803] 2025-04-26 16:39:38,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4673
4630
[WARNING|trainer.py:803] 2025-04-26 16:39:39,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:39,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4695
[WARNING|trainer.py:803] 2025-04-26 16:39:40,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4674
4631
[WARNING|trainer.py:803] 2025-04-26 16:39:40,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:40,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4696
[WARNING|trainer.py:803] 2025-04-26 16:39:41,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4675
4632
[WARNING|trainer.py:803] 2025-04-26 16:39:41,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4697
[WARNING|trainer.py:803] 2025-04-26 16:39:42,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:42,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4676
4633
[WARNING|trainer.py:803] 2025-04-26 16:39:42,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4698
[WARNING|trainer.py:803] 2025-04-26 16:39:43,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:43,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4677
4634
[WARNING|trainer.py:803] 2025-04-26 16:39:44,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4699
[WARNING|trainer.py:803] 2025-04-26 16:39:44,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:44,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4678
4635
[WARNING|trainer.py:803] 2025-04-26 16:39:45,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4700
[WARNING|trainer.py:803] 2025-04-26 16:39:45,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:46,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4679
4636
[WARNING|trainer.py:803] 2025-04-26 16:39:46,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4701
[WARNING|trainer.py:803] 2025-04-26 16:39:47,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:47,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4680
4637
[WARNING|trainer.py:803] 2025-04-26 16:39:47,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4702
[WARNING|trainer.py:803] 2025-04-26 16:39:48,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:48,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4681
[WARNING|trainer.py:803] 2025-04-26 16:39:48,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4638
4703
[WARNING|trainer.py:803] 2025-04-26 16:39:49,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:49,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4682
[WARNING|trainer.py:803] 2025-04-26 16:39:50,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4639
4704
[WARNING|trainer.py:803] 2025-04-26 16:39:50,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:50,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4683
[WARNING|trainer.py:803] 2025-04-26 16:39:51,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4640
4705
[WARNING|trainer.py:803] 2025-04-26 16:39:51,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:52,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4684
[WARNING|trainer.py:803] 2025-04-26 16:39:52,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
4706
[WARNING|trainer.py:803] 2025-04-26 16:39:53,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:53,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4685
[WARNING|trainer.py:803] 2025-04-26 16:39:53,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4642
4707
[WARNING|trainer.py:803] 2025-04-26 16:39:54,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:39:54,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4686
[WARNING|trainer.py:803] 2025-04-26 16:39:54,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4643
4708
[WARNING|trainer.py:803] 2025-04-26 16:39:55,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:55,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4687
[WARNING|trainer.py:803] 2025-04-26 16:39:55,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4644
4709
[WARNING|trainer.py:803] 2025-04-26 16:39:56,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:56,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4688
[WARNING|trainer.py:803] 2025-04-26 16:39:57,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4645
4710
[WARNING|trainer.py:803] 2025-04-26 16:39:57,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:58,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4689
[WARNING|trainer.py:803] 2025-04-26 16:39:58,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4646
4711
[WARNING|trainer.py:803] 2025-04-26 16:39:59,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:39:59,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4690
[WARNING|trainer.py:803] 2025-04-26 16:39:59,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4647
4712
[WARNING|trainer.py:803] 2025-04-26 16:40:00,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:00,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4691
[WARNING|trainer.py:803] 2025-04-26 16:40:00,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4648
4713
[WARNING|trainer.py:803] 2025-04-26 16:40:01,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:01,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4692
[WARNING|trainer.py:803] 2025-04-26 16:40:01,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4649
4714
[WARNING|trainer.py:803] 2025-04-26 16:40:02,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:02,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4693
[WARNING|trainer.py:803] 2025-04-26 16:40:03,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4650
4715
[WARNING|trainer.py:803] 2025-04-26 16:40:03,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:40:04,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:04,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4694
4716
4651
[WARNING|trainer.py:803] 2025-04-26 16:40:05,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:05,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:05,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4695
4717
4652
[WARNING|trainer.py:803] 2025-04-26 16:40:06,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:06,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4696
[WARNING|trainer.py:803] 2025-04-26 16:40:06,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4718
4653
[WARNING|trainer.py:803] 2025-04-26 16:40:07,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:07,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4697
[WARNING|trainer.py:803] 2025-04-26 16:40:07,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4719
4654
[WARNING|trainer.py:803] 2025-04-26 16:40:08,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:08,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4698
[WARNING|trainer.py:803] 2025-04-26 16:40:09,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4720
4655
[WARNING|trainer.py:803] 2025-04-26 16:40:09,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:10,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4699
[WARNING|trainer.py:803] 2025-04-26 16:40:10,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4721
4656
[WARNING|trainer.py:803] 2025-04-26 16:40:11,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:11,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4700
[WARNING|trainer.py:803] 2025-04-26 16:40:11,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4722
4657
[WARNING|trainer.py:803] 2025-04-26 16:40:12,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:40:12,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4701
[WARNING|trainer.py:803] 2025-04-26 16:40:12,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4723
4658
[WARNING|trainer.py:803] 2025-04-26 16:40:13,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:13,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:13,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4702
4724
4659
[WARNING|trainer.py:803] 2025-04-26 16:40:14,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:14,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:15,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4703
4725
4660
[WARNING|trainer.py:803] 2025-04-26 16:40:15,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:16,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4704
[WARNING|trainer.py:803] 2025-04-26 16:40:16,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4726
4661
[WARNING|trainer.py:803] 2025-04-26 16:40:17,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:17,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4705
[WARNING|trainer.py:803] 2025-04-26 16:40:17,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4727
4662
[WARNING|trainer.py:803] 2025-04-26 16:40:18,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:18,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4706
[WARNING|trainer.py:803] 2025-04-26 16:40:18,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4728
4663
[WARNING|trainer.py:803] 2025-04-26 16:40:19,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:19,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4707
[WARNING|trainer.py:803] 2025-04-26 16:40:19,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4729
4664
[WARNING|trainer.py:803] 2025-04-26 16:40:20,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:21,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4708
[WARNING|trainer.py:803] 2025-04-26 16:40:21,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4730
4665
[WARNING|trainer.py:803] 2025-04-26 16:40:21,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:40:22,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4709
[WARNING|trainer.py:803] 2025-04-26 16:40:22,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4731
4666
[WARNING|trainer.py:803] 2025-04-26 16:40:23,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:23,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4710
[WARNING|trainer.py:803] 2025-04-26 16:40:23,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4732
4667
[WARNING|trainer.py:803] 2025-04-26 16:40:24,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:24,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4711
[WARNING|trainer.py:803] 2025-04-26 16:40:24,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4733
4668
[WARNING|trainer.py:803] 2025-04-26 16:40:25,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:25,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4712
[WARNING|trainer.py:803] 2025-04-26 16:40:26,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4734
4669
[WARNING|trainer.py:803] 2025-04-26 16:40:26,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:27,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4713
[WARNING|trainer.py:803] 2025-04-26 16:40:27,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4735
4670
[WARNING|trainer.py:803] 2025-04-26 16:40:27,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:28,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4714
[WARNING|trainer.py:803] 2025-04-26 16:40:28,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4736
4671
[WARNING|trainer.py:803] 2025-04-26 16:40:29,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:29,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4715
[WARNING|trainer.py:803] 2025-04-26 16:40:29,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4737
4672
[WARNING|trainer.py:803] 2025-04-26 16:40:30,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:30,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4716
[WARNING|trainer.py:803] 2025-04-26 16:40:30,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4738
4673
[WARNING|trainer.py:803] 2025-04-26 16:40:31,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:31,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4717
[WARNING|trainer.py:803] 2025-04-26 16:40:32,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4739
4674
[WARNING|trainer.py:803] 2025-04-26 16:40:32,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:33,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4718
[WARNING|trainer.py:803] 2025-04-26 16:40:33,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4740
4675
[WARNING|trainer.py:803] 2025-04-26 16:40:33,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:34,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4719
[WARNING|trainer.py:803] 2025-04-26 16:40:34,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4741
4676
[WARNING|trainer.py:803] 2025-04-26 16:40:35,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:35,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4720
[WARNING|trainer.py:803] 2025-04-26 16:40:35,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4742
4677
[WARNING|trainer.py:803] 2025-04-26 16:40:36,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:36,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4721
[WARNING|trainer.py:803] 2025-04-26 16:40:36,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4743
4678
[WARNING|trainer.py:803] 2025-04-26 16:40:37,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:37,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4722
[WARNING|trainer.py:803] 2025-04-26 16:40:38,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4744
4679
[WARNING|trainer.py:803] 2025-04-26 16:40:38,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:39,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4723
[WARNING|trainer.py:803] 2025-04-26 16:40:39,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4745
4680
[WARNING|trainer.py:803] 2025-04-26 16:40:39,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4724
[WARNING|trainer.py:803] 2025-04-26 16:40:40,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4746
4681
[WARNING|trainer.py:803] 2025-04-26 16:40:41,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:41,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4725
[WARNING|trainer.py:803] 2025-04-26 16:40:41,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4747
4682
[WARNING|trainer.py:803] 2025-04-26 16:40:42,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4726
[WARNING|trainer.py:803] 2025-04-26 16:40:42,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:42,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4748
4683
[WARNING|trainer.py:803] 2025-04-26 16:40:43,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4727
[WARNING|trainer.py:803] 2025-04-26 16:40:43,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:44,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4749
4684
[WARNING|trainer.py:803] 2025-04-26 16:40:44,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4728
[WARNING|trainer.py:803] 2025-04-26 16:40:45,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:45,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4750
4685
[WARNING|trainer.py:803] 2025-04-26 16:40:45,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4729
[WARNING|trainer.py:803] 2025-04-26 16:40:46,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:46,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4751
4686
[WARNING|trainer.py:803] 2025-04-26 16:40:47,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4730
[WARNING|trainer.py:803] 2025-04-26 16:40:47,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:47,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4752
4687
[WARNING|trainer.py:803] 2025-04-26 16:40:48,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4731
[WARNING|trainer.py:803] 2025-04-26 16:40:48,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:48,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4753
4688
[WARNING|trainer.py:803] 2025-04-26 16:40:49,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4732
[WARNING|trainer.py:803] 2025-04-26 16:40:49,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:50,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4754
4689
[WARNING|trainer.py:803] 2025-04-26 16:40:50,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4733
[WARNING|trainer.py:803] 2025-04-26 16:40:51,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:51,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4755
4690
[WARNING|trainer.py:803] 2025-04-26 16:40:51,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4734
[WARNING|trainer.py:803] 2025-04-26 16:40:52,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:52,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4756
4691
[WARNING|trainer.py:803] 2025-04-26 16:40:53,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4735
[WARNING|trainer.py:803] 2025-04-26 16:40:53,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:53,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4757
4692
[WARNING|trainer.py:803] 2025-04-26 16:40:54,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4736
[WARNING|trainer.py:803] 2025-04-26 16:40:54,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:54,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4758
4693
[WARNING|trainer.py:803] 2025-04-26 16:40:55,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4737
[WARNING|trainer.py:803] 2025-04-26 16:40:55,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:40:56,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4759
4694
[WARNING|trainer.py:803] 2025-04-26 16:40:56,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4738
[WARNING|trainer.py:803] 2025-04-26 16:40:57,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:57,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4760
4695
[WARNING|trainer.py:803] 2025-04-26 16:40:57,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4739
[WARNING|trainer.py:803] 2025-04-26 16:40:58,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:58,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4761
4696
[WARNING|trainer.py:803] 2025-04-26 16:40:59,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4740
[WARNING|trainer.py:803] 2025-04-26 16:40:59,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:40:59,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4762
4697
[WARNING|trainer.py:803] 2025-04-26 16:41:00,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4741
[WARNING|trainer.py:803] 2025-04-26 16:41:00,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:00,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4763
4698
[WARNING|trainer.py:803] 2025-04-26 16:41:01,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4742
[WARNING|trainer.py:803] 2025-04-26 16:41:01,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:02,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4764
4699
[WARNING|trainer.py:803] 2025-04-26 16:41:02,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4743
[WARNING|trainer.py:803] 2025-04-26 16:41:03,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:03,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4765
4700
[WARNING|trainer.py:803] 2025-04-26 16:41:03,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4744
[WARNING|trainer.py:803] 2025-04-26 16:41:04,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:04,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4766
4701
[WARNING|trainer.py:803] 2025-04-26 16:41:04,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4745
[WARNING|trainer.py:803] 2025-04-26 16:41:05,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:05,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4767
4702
[WARNING|trainer.py:803] 2025-04-26 16:41:06,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4746
[WARNING|trainer.py:803] 2025-04-26 16:41:06,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:06,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4768
4703
[WARNING|trainer.py:803] 2025-04-26 16:41:07,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4747
[WARNING|trainer.py:803] 2025-04-26 16:41:07,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:08,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4769
4704
[WARNING|trainer.py:803] 2025-04-26 16:41:08,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4748
[WARNING|trainer.py:803] 2025-04-26 16:41:09,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:09,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4770
4705
[WARNING|trainer.py:803] 2025-04-26 16:41:09,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4749
[WARNING|trainer.py:803] 2025-04-26 16:41:10,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:10,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4771
4706
[WARNING|trainer.py:803] 2025-04-26 16:41:10,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4750
[WARNING|trainer.py:803] 2025-04-26 16:41:11,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:41:11,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4772
4707
[WARNING|trainer.py:803] 2025-04-26 16:41:12,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4751
[WARNING|trainer.py:803] 2025-04-26 16:41:12,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:12,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4773
4708
[WARNING|trainer.py:803] 2025-04-26 16:41:13,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4752
[WARNING|trainer.py:803] 2025-04-26 16:41:13,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:14,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4774
4709
[WARNING|trainer.py:803] 2025-04-26 16:41:14,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4753
[WARNING|trainer.py:803] 2025-04-26 16:41:15,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:15,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4775
4710
[WARNING|trainer.py:803] 2025-04-26 16:41:15,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4754
[WARNING|trainer.py:803] 2025-04-26 16:41:16,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:16,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4776
4711
[WARNING|trainer.py:803] 2025-04-26 16:41:16,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4755
[WARNING|trainer.py:803] 2025-04-26 16:41:17,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:17,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4777
4712
[WARNING|trainer.py:803] 2025-04-26 16:41:18,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4756
[WARNING|trainer.py:803] 2025-04-26 16:41:18,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:18,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4778
4713
[WARNING|trainer.py:803] 2025-04-26 16:41:19,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4757
[WARNING|trainer.py:803] 2025-04-26 16:41:19,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:20,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4779
4714
[WARNING|trainer.py:803] 2025-04-26 16:41:20,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4758
[WARNING|trainer.py:803] 2025-04-26 16:41:21,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:21,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4780
4715
[WARNING|trainer.py:803] 2025-04-26 16:41:21,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4759
[WARNING|trainer.py:803] 2025-04-26 16:41:22,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:22,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4781
4716
[WARNING|trainer.py:803] 2025-04-26 16:41:22,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4760
[WARNING|trainer.py:803] 2025-04-26 16:41:23,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:23,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4782
4717
[WARNING|trainer.py:803] 2025-04-26 16:41:24,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4761
[WARNING|trainer.py:803] 2025-04-26 16:41:24,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:24,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4783
4718
[WARNING|trainer.py:803] 2025-04-26 16:41:25,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4762
[WARNING|trainer.py:803] 2025-04-26 16:41:25,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:26,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4784
4719
[WARNING|trainer.py:803] 2025-04-26 16:41:26,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4763
[WARNING|trainer.py:803] 2025-04-26 16:41:26,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:27,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4785
4720
[WARNING|trainer.py:803] 2025-04-26 16:41:27,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4764
[WARNING|trainer.py:803] 2025-04-26 16:41:28,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:28,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4786
4721
[WARNING|trainer.py:803] 2025-04-26 16:41:28,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4765
[WARNING|trainer.py:803] 2025-04-26 16:41:29,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:29,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4787
4722
[WARNING|trainer.py:803] 2025-04-26 16:41:30,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4766
[WARNING|trainer.py:803] 2025-04-26 16:41:30,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:30,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4788
4723
[WARNING|trainer.py:803] 2025-04-26 16:41:31,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4767
[WARNING|trainer.py:803] 2025-04-26 16:41:31,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:32,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4789
4724
[WARNING|trainer.py:803] 2025-04-26 16:41:32,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4768
[WARNING|trainer.py:803] 2025-04-26 16:41:32,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:33,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4790
4725
[WARNING|trainer.py:803] 2025-04-26 16:41:33,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4769
[WARNING|trainer.py:803] 2025-04-26 16:41:34,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:34,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
4726
[WARNING|trainer.py:803] 2025-04-26 16:41:34,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4770
[WARNING|trainer.py:803] 2025-04-26 16:41:35,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:35,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4792
[WARNING|trainer.py:803] 2025-04-26 16:41:36,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4727
4771
[WARNING|trainer.py:803] 2025-04-26 16:41:36,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:36,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4793
4728
[WARNING|trainer.py:803] 2025-04-26 16:41:37,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4772
[WARNING|trainer.py:803] 2025-04-26 16:41:37,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:38,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4794
4729
[WARNING|trainer.py:803] 2025-04-26 16:41:38,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4773
[WARNING|trainer.py:803] 2025-04-26 16:41:38,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:39,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4795
4730
[WARNING|trainer.py:803] 2025-04-26 16:41:39,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4774
[WARNING|trainer.py:803] 2025-04-26 16:41:40,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:40,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4796
4731
[WARNING|trainer.py:803] 2025-04-26 16:41:40,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4775
[WARNING|trainer.py:803] 2025-04-26 16:41:41,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:41,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4797
4732
[WARNING|trainer.py:803] 2025-04-26 16:41:42,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4776
[WARNING|trainer.py:803] 2025-04-26 16:41:42,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:42,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4798
4733
[WARNING|trainer.py:803] 2025-04-26 16:41:43,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:43,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4777
[WARNING|trainer.py:803] 2025-04-26 16:41:44,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4799
4734
[WARNING|trainer.py:803] 2025-04-26 16:41:44,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:44,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4778
[WARNING|trainer.py:803] 2025-04-26 16:41:45,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4800
4735
[WARNING|trainer.py:803] 2025-04-26 16:41:45,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:46,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4779
[WARNING|trainer.py:803] 2025-04-26 16:41:46,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4801
4736
[WARNING|trainer.py:803] 2025-04-26 16:41:47,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:47,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4780
[WARNING|trainer.py:803] 2025-04-26 16:41:47,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4802
4737
[WARNING|trainer.py:803] 2025-04-26 16:41:48,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:48,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4781
[WARNING|trainer.py:803] 2025-04-26 16:41:48,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4803
4738
[WARNING|trainer.py:803] 2025-04-26 16:41:49,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:49,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4782
[WARNING|trainer.py:803] 2025-04-26 16:41:50,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4804
4739
[WARNING|trainer.py:803] 2025-04-26 16:41:50,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:50,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4783
[WARNING|trainer.py:803] 2025-04-26 16:41:51,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4805
4740
[WARNING|trainer.py:803] 2025-04-26 16:41:52,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:52,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:52,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4784
4806
4741
[WARNING|trainer.py:803] 2025-04-26 16:41:53,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:53,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:53,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4785
4807
4742
[WARNING|trainer.py:803] 2025-04-26 16:41:54,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:54,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:54,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4808
4786
4743
[WARNING|trainer.py:803] 2025-04-26 16:41:55,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:55,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:55,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4809
4787
4744
[WARNING|trainer.py:803] 2025-04-26 16:41:56,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:56,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:57,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4810
4788
4745
[WARNING|trainer.py:803] 2025-04-26 16:41:58,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:41:58,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4811
[WARNING|trainer.py:803] 2025-04-26 16:41:58,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4789
4746
[WARNING|trainer.py:803] 2025-04-26 16:41:59,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:41:59,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4812
[WARNING|trainer.py:803] 2025-04-26 16:41:59,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4790
4747
[WARNING|trainer.py:803] 2025-04-26 16:42:00,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:00,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4813
[WARNING|trainer.py:803] 2025-04-26 16:42:00,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
4748
[WARNING|trainer.py:803] 2025-04-26 16:42:01,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:01,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4814
[WARNING|trainer.py:803] 2025-04-26 16:42:01,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4792
4749
[WARNING|trainer.py:803] 2025-04-26 16:42:02,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:03,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:03,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4793
4815
4750
[WARNING|trainer.py:803] 2025-04-26 16:42:04,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:04,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:04,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4794
4816
4751
[WARNING|trainer.py:803] 2025-04-26 16:42:05,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:05,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:05,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4795
4817
4752
[WARNING|trainer.py:803] 2025-04-26 16:42:06,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:06,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:06,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4796
4818
4753
[WARNING|trainer.py:803] 2025-04-26 16:42:07,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:07,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:07,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4797
4819
4754
[WARNING|trainer.py:803] 2025-04-26 16:42:08,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:09,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:09,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4798
4820
4755
[WARNING|trainer.py:803] 2025-04-26 16:42:10,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:10,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:10,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4799
4821
4756
[WARNING|trainer.py:803] 2025-04-26 16:42:11,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:11,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:11,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4800
4822
4757
[WARNING|trainer.py:803] 2025-04-26 16:42:12,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:12,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:12,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4801
4823
4758
[WARNING|trainer.py:803] 2025-04-26 16:42:13,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:13,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:14,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4802
4824
4759
[WARNING|trainer.py:803] 2025-04-26 16:42:14,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:15,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:15,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4803
4825
4760
[WARNING|trainer.py:803] 2025-04-26 16:42:16,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:16,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:16,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4804
4826
4761
[WARNING|trainer.py:803] 2025-04-26 16:42:17,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:17,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:17,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4805
4827
4762
[WARNING|trainer.py:803] 2025-04-26 16:42:18,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:18,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:18,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4806
4828
4763
[WARNING|trainer.py:803] 2025-04-26 16:42:19,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:19,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:20,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4807
4829
4764
[WARNING|trainer.py:803] 2025-04-26 16:42:20,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:21,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:21,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4808
4830
4765
[WARNING|trainer.py:803] 2025-04-26 16:42:22,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:22,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4809
[WARNING|trainer.py:803] 2025-04-26 16:42:22,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4831
4766
[WARNING|trainer.py:803] 2025-04-26 16:42:23,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:23,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4810
[WARNING|trainer.py:803] 2025-04-26 16:42:23,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4832
4767
[WARNING|trainer.py:803] 2025-04-26 16:42:24,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:24,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4811
[WARNING|trainer.py:803] 2025-04-26 16:42:24,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4833
4768
[WARNING|trainer.py:803] 2025-04-26 16:42:25,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:25,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4812
[WARNING|trainer.py:803] 2025-04-26 16:42:26,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4834
4769
[WARNING|trainer.py:803] 2025-04-26 16:42:26,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:27,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4813
[WARNING|trainer.py:803] 2025-04-26 16:42:27,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4835
4770
[WARNING|trainer.py:803] 2025-04-26 16:42:28,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:28,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4814
[WARNING|trainer.py:803] 2025-04-26 16:42:28,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4836
4771
[WARNING|trainer.py:803] 2025-04-26 16:42:29,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:29,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4815
[WARNING|trainer.py:803] 2025-04-26 16:42:29,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4837
4772
[WARNING|trainer.py:803] 2025-04-26 16:42:30,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:30,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4816
[WARNING|trainer.py:803] 2025-04-26 16:42:30,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4838
4773
[WARNING|trainer.py:803] 2025-04-26 16:42:31,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:31,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4817
[WARNING|trainer.py:803] 2025-04-26 16:42:32,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4839
4774
[WARNING|trainer.py:803] 2025-04-26 16:42:32,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:32,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4818
[WARNING|trainer.py:803] 2025-04-26 16:42:33,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4840
4775
[WARNING|trainer.py:803] 2025-04-26 16:42:33,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:34,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4819
[WARNING|trainer.py:803] 2025-04-26 16:42:34,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4841
4776
[WARNING|trainer.py:803] 2025-04-26 16:42:35,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:35,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4820
[WARNING|trainer.py:803] 2025-04-26 16:42:35,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4842
4777
[WARNING|trainer.py:803] 2025-04-26 16:42:36,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:36,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4821
[WARNING|trainer.py:803] 2025-04-26 16:42:36,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4843
4778
[WARNING|trainer.py:803] 2025-04-26 16:42:37,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:37,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4822
[WARNING|trainer.py:803] 2025-04-26 16:42:37,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4844
4779
[WARNING|trainer.py:803] 2025-04-26 16:42:38,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:38,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4823
[WARNING|trainer.py:803] 2025-04-26 16:42:39,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4845
4780
[WARNING|trainer.py:803] 2025-04-26 16:42:39,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:40,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4824
[WARNING|trainer.py:803] 2025-04-26 16:42:40,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4846
4781
[WARNING|trainer.py:803] 2025-04-26 16:42:41,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:41,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4825
[WARNING|trainer.py:803] 2025-04-26 16:42:41,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4847
4782
[WARNING|trainer.py:803] 2025-04-26 16:42:42,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:42,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4826
[WARNING|trainer.py:803] 2025-04-26 16:42:42,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4848
4783
[WARNING|trainer.py:803] 2025-04-26 16:42:43,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:43,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4827
[WARNING|trainer.py:803] 2025-04-26 16:42:43,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4849
4784
[WARNING|trainer.py:803] 2025-04-26 16:42:44,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:44,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4828
[WARNING|trainer.py:803] 2025-04-26 16:42:45,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4850
4785
[WARNING|trainer.py:803] 2025-04-26 16:42:45,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:46,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4829
[WARNING|trainer.py:803] 2025-04-26 16:42:46,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4851
4786
[WARNING|trainer.py:803] 2025-04-26 16:42:47,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:47,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4830
4852
[WARNING|trainer.py:803] 2025-04-26 16:42:47,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4787
[WARNING|trainer.py:803] 2025-04-26 16:42:48,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:48,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4831
4853
[WARNING|trainer.py:803] 2025-04-26 16:42:48,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4788
[WARNING|trainer.py:803] 2025-04-26 16:42:49,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:49,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4832
4854
[WARNING|trainer.py:803] 2025-04-26 16:42:49,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4789
[WARNING|trainer.py:803] 2025-04-26 16:42:50,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:50,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4833
4855
[WARNING|trainer.py:803] 2025-04-26 16:42:51,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4790
[WARNING|trainer.py:803] 2025-04-26 16:42:51,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:51,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4834
4856
[WARNING|trainer.py:803] 2025-04-26 16:42:52,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
[WARNING|trainer.py:803] 2025-04-26 16:42:52,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:53,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4835
4857
[WARNING|trainer.py:803] 2025-04-26 16:42:53,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4792
[WARNING|trainer.py:803] 2025-04-26 16:42:54,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:54,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4836
4858
[WARNING|trainer.py:803] 2025-04-26 16:42:54,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4793
[WARNING|trainer.py:803] 2025-04-26 16:42:55,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:42:55,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4837
4859
[WARNING|trainer.py:803] 2025-04-26 16:42:55,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4794
[WARNING|trainer.py:803] 2025-04-26 16:42:56,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:56,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4838
4860
[WARNING|trainer.py:803] 2025-04-26 16:42:57,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4795
[WARNING|trainer.py:803] 2025-04-26 16:42:57,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:42:57,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4839
4861
[WARNING|trainer.py:803] 2025-04-26 16:42:58,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4796
[WARNING|trainer.py:803] 2025-04-26 16:42:58,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:42:59,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4840
4862
[WARNING|trainer.py:803] 2025-04-26 16:42:59,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4797
[WARNING|trainer.py:803] 2025-04-26 16:43:00,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:00,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4841
4863
[WARNING|trainer.py:803] 2025-04-26 16:43:00,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4798
[WARNING|trainer.py:803] 2025-04-26 16:43:01,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:01,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4842
4864
[WARNING|trainer.py:803] 2025-04-26 16:43:01,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4799
[WARNING|trainer.py:803] 2025-04-26 16:43:02,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:02,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4843
4865
[WARNING|trainer.py:803] 2025-04-26 16:43:03,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4800
[WARNING|trainer.py:803] 2025-04-26 16:43:03,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:03,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4844
4866
[WARNING|trainer.py:803] 2025-04-26 16:43:04,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4801
[WARNING|trainer.py:803] 2025-04-26 16:43:04,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:04,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4845
4867
[WARNING|trainer.py:803] 2025-04-26 16:43:05,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4802
[WARNING|trainer.py:803] 2025-04-26 16:43:06,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:06,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4846
4868
[WARNING|trainer.py:803] 2025-04-26 16:43:06,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4803
[WARNING|trainer.py:803] 2025-04-26 16:43:07,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:07,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4847
4869
[WARNING|trainer.py:803] 2025-04-26 16:43:07,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4804
[WARNING|trainer.py:803] 2025-04-26 16:43:08,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:08,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4848
4870
[WARNING|trainer.py:803] 2025-04-26 16:43:08,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4805
[WARNING|trainer.py:803] 2025-04-26 16:43:09,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:09,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4871
4849
[WARNING|trainer.py:803] 2025-04-26 16:43:10,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4806
[WARNING|trainer.py:803] 2025-04-26 16:43:10,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:10,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4872
4850
[WARNING|trainer.py:803] 2025-04-26 16:43:11,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4807
[WARNING|trainer.py:803] 2025-04-26 16:43:12,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:12,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4873
[WARNING|trainer.py:803] 2025-04-26 16:43:12,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4851
4808
[WARNING|trainer.py:803] 2025-04-26 16:43:13,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:13,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4874
[WARNING|trainer.py:803] 2025-04-26 16:43:13,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4852
4809
[WARNING|trainer.py:803] 2025-04-26 16:43:14,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:14,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:14,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4875
4853
4810
[WARNING|trainer.py:803] 2025-04-26 16:43:15,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:15,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:16,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4876
4854
4811
[WARNING|trainer.py:803] 2025-04-26 16:43:16,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:16,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:17,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4877
4855
4812
[WARNING|trainer.py:803] 2025-04-26 16:43:18,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:18,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:18,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4878
4856
4813
[WARNING|trainer.py:803] 2025-04-26 16:43:19,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:19,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:19,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4879
4857
4814
[WARNING|trainer.py:803] 2025-04-26 16:43:20,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:20,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:20,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4880
4858
4815
[WARNING|trainer.py:803] 2025-04-26 16:43:21,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:21,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:21,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4881
4859
4816
[WARNING|trainer.py:803] 2025-04-26 16:43:22,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:22,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:23,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4882
4860
4817
[WARNING|trainer.py:803] 2025-04-26 16:43:24,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:24,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:24,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4883
4861
4818
[WARNING|trainer.py:803] 2025-04-26 16:43:25,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:25,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:25,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4884
4862
4819
[WARNING|trainer.py:803] 2025-04-26 16:43:26,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:26,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:26,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4885
4863
4820
[WARNING|trainer.py:803] 2025-04-26 16:43:27,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:27,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:43:27,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4886
4864
4821
[WARNING|trainer.py:803] 2025-04-26 16:43:28,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:28,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:29,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4887
4865
4822
[WARNING|trainer.py:803] 2025-04-26 16:43:29,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:30,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:30,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4888
4866
4823
[WARNING|trainer.py:803] 2025-04-26 16:43:31,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:43:31,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:31,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4889
4867
4824
[WARNING|trainer.py:803] 2025-04-26 16:43:32,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:32,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:43:32,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4890
4868
4825
[WARNING|trainer.py:803] 2025-04-26 16:43:33,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:33,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:33,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4891
4869
4826
[WARNING|trainer.py:803] 2025-04-26 16:43:34,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:34,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:35,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4892
4870
4827
[WARNING|trainer.py:803] 2025-04-26 16:43:35,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:35,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:36,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4871
4893
4828
[WARNING|trainer.py:803] 2025-04-26 16:43:37,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:37,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:37,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4872
4894
4829
[WARNING|trainer.py:803] 2025-04-26 16:43:38,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:38,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:38,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4873
4895
4830
[WARNING|trainer.py:803] 2025-04-26 16:43:39,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:39,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:39,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4874
4896
4831
[WARNING|trainer.py:803] 2025-04-26 16:43:40,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:40,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:43:40,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4875
4897
4832
[WARNING|trainer.py:803] 2025-04-26 16:43:41,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:41,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:43:42,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4876
4898
4833
[WARNING|trainer.py:803] 2025-04-26 16:43:43,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:43,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:43,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4877
4899
4834
[WARNING|trainer.py:803] 2025-04-26 16:43:44,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:44,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:44,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4878
4900
4835
[WARNING|trainer.py:803] 2025-04-26 16:43:45,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:45,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:43:45,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4879
4901
4836
[WARNING|trainer.py:803] 2025-04-26 16:43:46,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:46,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:43:46,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4880
4902
4837
[WARNING|trainer.py:803] 2025-04-26 16:43:47,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:47,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:48,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4881
4903
4838
[WARNING|trainer.py:803] 2025-04-26 16:43:49,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:49,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:49,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4882
4904
4839
[WARNING|trainer.py:803] 2025-04-26 16:43:50,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:50,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:50,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4883
4905
4840
[WARNING|trainer.py:803] 2025-04-26 16:43:51,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:51,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:51,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4884
4906
4841
[WARNING|trainer.py:803] 2025-04-26 16:43:52,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:52,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:52,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4885
4907
4842
[WARNING|trainer.py:803] 2025-04-26 16:43:53,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:53,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:54,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4886
4908
4843
[WARNING|trainer.py:803] 2025-04-26 16:43:54,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:55,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:55,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4887
4909
4844
[WARNING|trainer.py:803] 2025-04-26 16:43:56,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:43:56,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:43:56,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4888
4910
4845
[WARNING|trainer.py:803] 2025-04-26 16:43:57,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:43:57,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:57,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4889
4911
4846
[WARNING|trainer.py:803] 2025-04-26 16:43:58,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:58,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:43:58,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4890
4912
4847
[WARNING|trainer.py:803] 2025-04-26 16:43:59,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:43:59,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:00,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4891
4913
4848
[WARNING|trainer.py:803] 2025-04-26 16:44:00,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:01,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:01,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4892
4914
4849
[WARNING|trainer.py:803] 2025-04-26 16:44:02,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:02,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:02,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4893
4915
4850
[WARNING|trainer.py:803] 2025-04-26 16:44:03,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:03,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:03,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4894
4916
4851
[WARNING|trainer.py:803] 2025-04-26 16:44:04,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:04,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:04,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4895
4917
4852
[WARNING|trainer.py:803] 2025-04-26 16:44:05,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:06,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:06,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4896
4853
4918
[WARNING|trainer.py:803] 2025-04-26 16:44:06,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:44:07,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4897
[WARNING|trainer.py:803] 2025-04-26 16:44:07,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4854
4919
[WARNING|trainer.py:803] 2025-04-26 16:44:08,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:44:08,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4898
[WARNING|trainer.py:803] 2025-04-26 16:44:08,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4855
4920
[WARNING|trainer.py:803] 2025-04-26 16:44:09,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4899
[WARNING|trainer.py:803] 2025-04-26 16:44:09,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:09,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4856
4921
[WARNING|trainer.py:803] 2025-04-26 16:44:10,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4900
[WARNING|trainer.py:803] 2025-04-26 16:44:10,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:10,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4857
4922
[WARNING|trainer.py:803] 2025-04-26 16:44:11,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4901
[WARNING|trainer.py:803] 2025-04-26 16:44:11,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:12,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4858
4923
[WARNING|trainer.py:803] 2025-04-26 16:44:12,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:44:13,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4902
[WARNING|trainer.py:803] 2025-04-26 16:44:13,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4859
4924
[WARNING|trainer.py:803] 2025-04-26 16:44:13,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:14,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4903
[WARNING|trainer.py:803] 2025-04-26 16:44:14,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4860
4925
[WARNING|trainer.py:803] 2025-04-26 16:44:15,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:15,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4904
[WARNING|trainer.py:803] 2025-04-26 16:44:15,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4861
4926
[WARNING|trainer.py:803] 2025-04-26 16:44:16,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:16,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4905
[WARNING|trainer.py:803] 2025-04-26 16:44:16,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4862
4927
[WARNING|trainer.py:803] 2025-04-26 16:44:17,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:17,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4906
[WARNING|trainer.py:803] 2025-04-26 16:44:18,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4863
4928
[WARNING|trainer.py:803] 2025-04-26 16:44:18,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:19,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4907
[WARNING|trainer.py:803] 2025-04-26 16:44:19,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4864
4929
[WARNING|trainer.py:803] 2025-04-26 16:44:19,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:20,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4908
[WARNING|trainer.py:803] 2025-04-26 16:44:20,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4865
4930
[WARNING|trainer.py:803] 2025-04-26 16:44:21,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:21,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4909
[WARNING|trainer.py:803] 2025-04-26 16:44:21,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4866
4931
[WARNING|trainer.py:803] 2025-04-26 16:44:22,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:44:22,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4910
[WARNING|trainer.py:803] 2025-04-26 16:44:22,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4867
4932
[WARNING|trainer.py:803] 2025-04-26 16:44:23,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:23,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4911
[WARNING|trainer.py:803] 2025-04-26 16:44:24,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4868
4933
[WARNING|trainer.py:803] 2025-04-26 16:44:24,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:44:24,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4912
[WARNING|trainer.py:803] 2025-04-26 16:44:25,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4869
4934
[WARNING|trainer.py:803] 2025-04-26 16:44:25,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:26,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4913
[WARNING|trainer.py:803] 2025-04-26 16:44:26,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4870
4935
[WARNING|trainer.py:803] 2025-04-26 16:44:27,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:27,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4914
[WARNING|trainer.py:803] 2025-04-26 16:44:27,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4871
4936
[WARNING|trainer.py:803] 2025-04-26 16:44:28,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:28,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4915
[WARNING|trainer.py:803] 2025-04-26 16:44:28,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4872
4937
[WARNING|trainer.py:803] 2025-04-26 16:44:29,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:29,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4916
[WARNING|trainer.py:803] 2025-04-26 16:44:29,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4873
4938
[WARNING|trainer.py:803] 2025-04-26 16:44:30,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:30,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4917
[WARNING|trainer.py:803] 2025-04-26 16:44:31,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4874
4939
[WARNING|trainer.py:803] 2025-04-26 16:44:31,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:32,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4918
[WARNING|trainer.py:803] 2025-04-26 16:44:32,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4875
4940
[WARNING|trainer.py:803] 2025-04-26 16:44:33,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:33,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4919
[WARNING|trainer.py:803] 2025-04-26 16:44:33,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4876
4941
[WARNING|trainer.py:803] 2025-04-26 16:44:34,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:34,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4920
[WARNING|trainer.py:803] 2025-04-26 16:44:34,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4877
4942
[WARNING|trainer.py:803] 2025-04-26 16:44:35,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:35,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4921
[WARNING|trainer.py:803] 2025-04-26 16:44:35,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4878
4943
[WARNING|trainer.py:803] 2025-04-26 16:44:36,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:36,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:36,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4922
4879
4944
[WARNING|trainer.py:803] 2025-04-26 16:44:37,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:37,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:38,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4923
4880
4945
[WARNING|trainer.py:803] 2025-04-26 16:44:39,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:39,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:39,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4924
4881
4946
[WARNING|trainer.py:803] 2025-04-26 16:44:40,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:40,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:40,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4925
4882
4947
[WARNING|trainer.py:803] 2025-04-26 16:44:41,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:41,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:41,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4926
4883
4948
[WARNING|trainer.py:803] 2025-04-26 16:44:42,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:42,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:42,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4927
4884
4949
[WARNING|trainer.py:803] 2025-04-26 16:44:43,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:43,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:44,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4928
4885
4950
[WARNING|trainer.py:803] 2025-04-26 16:44:44,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:45,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:45,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4929
4886
4951
[WARNING|trainer.py:803] 2025-04-26 16:44:46,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:46,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:46,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4930
4887
4952
[WARNING|trainer.py:803] 2025-04-26 16:44:47,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:47,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:47,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4931
4888
4953
[WARNING|trainer.py:803] 2025-04-26 16:44:48,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:48,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:44:48,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4932
4889
4954
[WARNING|trainer.py:803] 2025-04-26 16:44:49,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:49,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:49,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4933
4890
4955
[WARNING|trainer.py:803] 2025-04-26 16:44:50,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:51,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:51,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4934
4891
4956
[WARNING|trainer.py:803] 2025-04-26 16:44:52,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:52,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:52,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4935
4892
4957
[WARNING|trainer.py:803] 2025-04-26 16:44:53,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:53,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:53,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4936
4893
4958
[WARNING|trainer.py:803] 2025-04-26 16:44:54,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:54,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:54,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4937
4894
4959
[WARNING|trainer.py:803] 2025-04-26 16:44:55,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:55,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:44:55,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4938
4895
4960
[WARNING|trainer.py:803] 2025-04-26 16:44:56,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:44:57,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:57,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4939
4896
4961
[WARNING|trainer.py:803] 2025-04-26 16:44:58,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:44:58,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:44:58,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4940
4897
4962
[WARNING|trainer.py:803] 2025-04-26 16:44:59,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:44:59,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:44:59,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4941
4898
4963
[WARNING|trainer.py:803] 2025-04-26 16:45:00,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:00,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:00,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4942
4899
4964
[WARNING|trainer.py:803] 2025-04-26 16:45:01,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:01,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:01,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4943
4900
4965
[WARNING|trainer.py:803] 2025-04-26 16:45:03,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:03,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:45:03,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4944
4901
4966
[WARNING|trainer.py:803] 2025-04-26 16:45:04,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:04,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:04,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4945
4967
4902
[WARNING|trainer.py:803] 2025-04-26 16:45:05,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:05,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:05,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4946
4968
4903
[WARNING|trainer.py:803] 2025-04-26 16:45:06,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:06,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:06,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4947
4969
4904
[WARNING|trainer.py:803] 2025-04-26 16:45:07,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:45:07,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:07,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4970
4948
4905
[WARNING|trainer.py:803] 2025-04-26 16:45:09,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:09,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:09,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4971
4949
4906
[WARNING|trainer.py:803] 2025-04-26 16:45:10,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:10,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:10,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4972
4950
4907
[WARNING|trainer.py:803] 2025-04-26 16:45:11,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:11,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:11,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4973
4951
4908
[WARNING|trainer.py:803] 2025-04-26 16:45:12,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:12,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:12,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4974
4952
4909
[WARNING|trainer.py:803] 2025-04-26 16:45:13,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:13,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:13,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4975
4953
4910
[WARNING|trainer.py:803] 2025-04-26 16:45:14,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:15,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:15,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4976
4954
4911
[WARNING|trainer.py:803] 2025-04-26 16:45:16,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:16,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:16,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4977
4955
4912
[WARNING|trainer.py:803] 2025-04-26 16:45:17,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:17,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:17,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4978
4956
4913
[WARNING|trainer.py:803] 2025-04-26 16:45:18,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:18,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:18,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4979
4957
4914
[WARNING|trainer.py:803] 2025-04-26 16:45:19,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:19,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:19,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4980
4958
4915
[WARNING|trainer.py:803] 2025-04-26 16:45:20,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:21,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:21,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4981
4959
4916
[WARNING|trainer.py:803] 2025-04-26 16:45:21,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:22,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:22,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4982
4960
4917
[WARNING|trainer.py:803] 2025-04-26 16:45:23,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:23,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4983
[WARNING|trainer.py:803] 2025-04-26 16:45:23,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4961
4918
[WARNING|trainer.py:803] 2025-04-26 16:45:24,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4984
[WARNING|trainer.py:803] 2025-04-26 16:45:24,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:24,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4962
4919
[WARNING|trainer.py:803] 2025-04-26 16:45:25,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4985
[WARNING|trainer.py:803] 2025-04-26 16:45:25,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:25,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4963
4920
[WARNING|trainer.py:803] 2025-04-26 16:45:26,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4986
[WARNING|trainer.py:803] 2025-04-26 16:45:27,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:27,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4964
4921
[WARNING|trainer.py:803] 2025-04-26 16:45:27,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4987
[WARNING|trainer.py:803] 2025-04-26 16:45:28,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:28,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4965
4922
[WARNING|trainer.py:803] 2025-04-26 16:45:29,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4988
[WARNING|trainer.py:803] 2025-04-26 16:45:29,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:29,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4966
4923
[WARNING|trainer.py:803] 2025-04-26 16:45:30,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4989
[WARNING|trainer.py:803] 2025-04-26 16:45:30,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:30,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4967
4924
[WARNING|trainer.py:803] 2025-04-26 16:45:31,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4990
[WARNING|trainer.py:803] 2025-04-26 16:45:31,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:31,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4968
4925
[WARNING|trainer.py:803] 2025-04-26 16:45:32,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4991
[WARNING|trainer.py:803] 2025-04-26 16:45:33,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:33,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4969
4926
[WARNING|trainer.py:803] 2025-04-26 16:45:33,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4992
[WARNING|trainer.py:803] 2025-04-26 16:45:34,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:45:34,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4970
4927
[WARNING|trainer.py:803] 2025-04-26 16:45:35,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4993
[WARNING|trainer.py:803] 2025-04-26 16:45:35,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:35,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4971
4928
[WARNING|trainer.py:803] 2025-04-26 16:45:36,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4994
[WARNING|trainer.py:803] 2025-04-26 16:45:36,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:36,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4972
4929
[WARNING|trainer.py:803] 2025-04-26 16:45:37,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4995
[WARNING|trainer.py:803] 2025-04-26 16:45:37,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:37,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4930
4973
[WARNING|trainer.py:803] 2025-04-26 16:45:38,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4996
[WARNING|trainer.py:803] 2025-04-26 16:45:39,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:39,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4974
4931
[WARNING|trainer.py:803] 2025-04-26 16:45:39,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4997
[WARNING|trainer.py:803] 2025-04-26 16:45:40,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:40,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4975
4932
[WARNING|trainer.py:803] 2025-04-26 16:45:40,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4998
[WARNING|trainer.py:803] 2025-04-26 16:45:41,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:41,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4976
4933
[WARNING|trainer.py:803] 2025-04-26 16:45:42,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4999
[WARNING|trainer.py:803] 2025-04-26 16:45:42,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:42,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4977
4934
[WARNING|trainer.py:803] 2025-04-26 16:45:43,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5000
[WARNING|trainer.py:803] 2025-04-26 16:45:43,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:43,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4935
4978
[WARNING|trainer.py:803] 2025-04-26 16:45:44,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5001
[WARNING|trainer.py:803] 2025-04-26 16:45:45,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:45,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4936
4979
[WARNING|trainer.py:803] 2025-04-26 16:45:45,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5002
[WARNING|trainer.py:803] 2025-04-26 16:45:46,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:46,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4937
4980
[WARNING|trainer.py:803] 2025-04-26 16:45:46,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5003
[WARNING|trainer.py:803] 2025-04-26 16:45:47,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:47,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4938
4981
[WARNING|trainer.py:803] 2025-04-26 16:45:48,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5004
[WARNING|trainer.py:803] 2025-04-26 16:45:48,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:48,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4939
4982
[WARNING|trainer.py:803] 2025-04-26 16:45:49,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5005
[WARNING|trainer.py:803] 2025-04-26 16:45:49,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:49,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4940
4983
[WARNING|trainer.py:803] 2025-04-26 16:45:50,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5006
[WARNING|trainer.py:803] 2025-04-26 16:45:51,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:45:51,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4941
4984
[WARNING|trainer.py:803] 2025-04-26 16:45:51,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5007
[WARNING|trainer.py:803] 2025-04-26 16:45:52,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:45:52,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4942
4985
[WARNING|trainer.py:803] 2025-04-26 16:45:52,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5008
[WARNING|trainer.py:803] 2025-04-26 16:45:53,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:53,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4943
4986
[WARNING|trainer.py:803] 2025-04-26 16:45:53,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5009
[WARNING|trainer.py:803] 2025-04-26 16:45:54,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:54,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4944
4987
[WARNING|trainer.py:803] 2025-04-26 16:45:55,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5010
[WARNING|trainer.py:803] 2025-04-26 16:45:55,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:55,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4945
4988
[WARNING|trainer.py:803] 2025-04-26 16:45:56,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5011
[WARNING|trainer.py:803] 2025-04-26 16:45:56,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:45:57,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4946
4989
[WARNING|trainer.py:803] 2025-04-26 16:45:57,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5012
[WARNING|trainer.py:803] 2025-04-26 16:45:58,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:58,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4947
4990
[WARNING|trainer.py:803] 2025-04-26 16:45:58,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5013
[WARNING|trainer.py:803] 2025-04-26 16:45:59,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:45:59,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4948
4991
[WARNING|trainer.py:803] 2025-04-26 16:45:59,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5014
[WARNING|trainer.py:803] 2025-04-26 16:46:00,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:00,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4949
[WARNING|trainer.py:803] 2025-04-26 16:46:01,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4992
5015
[WARNING|trainer.py:803] 2025-04-26 16:46:01,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:01,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4950
[WARNING|trainer.py:803] 2025-04-26 16:46:02,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4993
5016
[WARNING|trainer.py:803] 2025-04-26 16:46:02,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:03,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4951
[WARNING|trainer.py:803] 2025-04-26 16:46:03,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4994
5017
[WARNING|trainer.py:803] 2025-04-26 16:46:04,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:04,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4952
[WARNING|trainer.py:803] 2025-04-26 16:46:04,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4995
5018
[WARNING|trainer.py:803] 2025-04-26 16:46:05,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:05,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4953
[WARNING|trainer.py:803] 2025-04-26 16:46:05,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4996
5019
[WARNING|trainer.py:803] 2025-04-26 16:46:06,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:06,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4954
[WARNING|trainer.py:803] 2025-04-26 16:46:07,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4997
5020
[WARNING|trainer.py:803] 2025-04-26 16:46:07,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:07,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4955
[WARNING|trainer.py:803] 2025-04-26 16:46:08,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4998
5021
[WARNING|trainer.py:803] 2025-04-26 16:46:08,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:09,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4956
[WARNING|trainer.py:803] 2025-04-26 16:46:09,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4999
5022
[WARNING|trainer.py:803] 2025-04-26 16:46:09,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:10,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4957
[WARNING|trainer.py:803] 2025-04-26 16:46:10,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5000
5023
[WARNING|trainer.py:803] 2025-04-26 16:46:11,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:11,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4958
[WARNING|trainer.py:803] 2025-04-26 16:46:11,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5001
5024
[WARNING|trainer.py:803] 2025-04-26 16:46:12,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:12,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4959
[WARNING|trainer.py:803] 2025-04-26 16:46:13,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5002
5025
[WARNING|trainer.py:803] 2025-04-26 16:46:13,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:13,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4960
[WARNING|trainer.py:803] 2025-04-26 16:46:14,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5003
5026
[WARNING|trainer.py:803] 2025-04-26 16:46:14,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:15,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4961
[WARNING|trainer.py:803] 2025-04-26 16:46:15,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5004
5027
[WARNING|trainer.py:803] 2025-04-26 16:46:15,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4962
[WARNING|trainer.py:803] 2025-04-26 16:46:16,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:16,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5005
5028
[WARNING|trainer.py:803] 2025-04-26 16:46:17,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4963
[WARNING|trainer.py:803] 2025-04-26 16:46:17,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:17,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5006
5029
[WARNING|trainer.py:803] 2025-04-26 16:46:18,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:18,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4964
[WARNING|trainer.py:803] 2025-04-26 16:46:18,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5007
5030
[WARNING|trainer.py:803] 2025-04-26 16:46:19,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:19,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4965
[WARNING|trainer.py:803] 2025-04-26 16:46:20,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5008
5031
[WARNING|trainer.py:803] 2025-04-26 16:46:20,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:21,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4966
[WARNING|trainer.py:803] 2025-04-26 16:46:21,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5009
5032
[WARNING|trainer.py:803] 2025-04-26 16:46:22,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:22,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4967
[WARNING|trainer.py:803] 2025-04-26 16:46:22,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5010
5033
[WARNING|trainer.py:803] 2025-04-26 16:46:23,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:23,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4968
[WARNING|trainer.py:803] 2025-04-26 16:46:23,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5011
5034
[WARNING|trainer.py:803] 2025-04-26 16:46:24,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:46:24,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4969
[WARNING|trainer.py:803] 2025-04-26 16:46:24,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5012
5035
[WARNING|trainer.py:803] 2025-04-26 16:46:25,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:46:25,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4970
[WARNING|trainer.py:803] 2025-04-26 16:46:26,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5013
5036
[WARNING|trainer.py:803] 2025-04-26 16:46:26,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:26,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4971
[WARNING|trainer.py:803] 2025-04-26 16:46:27,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5014
5037
[WARNING|trainer.py:803] 2025-04-26 16:46:27,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:28,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4972
[WARNING|trainer.py:803] 2025-04-26 16:46:28,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5015
5038
[WARNING|trainer.py:803] 2025-04-26 16:46:29,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:29,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4973
[WARNING|trainer.py:803] 2025-04-26 16:46:29,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5016
5039
[WARNING|trainer.py:803] 2025-04-26 16:46:30,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:30,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4974
[WARNING|trainer.py:803] 2025-04-26 16:46:30,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5017
5040
[WARNING|trainer.py:803] 2025-04-26 16:46:31,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:31,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4975
[WARNING|trainer.py:803] 2025-04-26 16:46:32,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5018
5041
[WARNING|trainer.py:803] 2025-04-26 16:46:32,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:32,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4976
[WARNING|trainer.py:803] 2025-04-26 16:46:33,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5019
5042
[WARNING|trainer.py:803] 2025-04-26 16:46:33,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:34,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4977
[WARNING|trainer.py:803] 2025-04-26 16:46:34,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5020
5043
[WARNING|trainer.py:803] 2025-04-26 16:46:35,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:35,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4978
[WARNING|trainer.py:803] 2025-04-26 16:46:35,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5021
5044
[WARNING|trainer.py:803] 2025-04-26 16:46:36,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:36,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4979
[WARNING|trainer.py:803] 2025-04-26 16:46:36,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5022
5045
[WARNING|trainer.py:803] 2025-04-26 16:46:37,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:37,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4980
[WARNING|trainer.py:803] 2025-04-26 16:46:37,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5023
5046
[WARNING|trainer.py:803] 2025-04-26 16:46:38,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:38,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4981
[WARNING|trainer.py:803] 2025-04-26 16:46:39,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5024
5047
[WARNING|trainer.py:803] 2025-04-26 16:46:39,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:40,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4982
[WARNING|trainer.py:803] 2025-04-26 16:46:40,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5025
5048
[WARNING|trainer.py:803] 2025-04-26 16:46:40,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:41,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4983
[WARNING|trainer.py:803] 2025-04-26 16:46:41,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5026
5049
[WARNING|trainer.py:803] 2025-04-26 16:46:42,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:42,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4984
[WARNING|trainer.py:803] 2025-04-26 16:46:42,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5027
5050
[WARNING|trainer.py:803] 2025-04-26 16:46:43,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:43,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4985
[WARNING|trainer.py:803] 2025-04-26 16:46:43,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5028
5051
[WARNING|trainer.py:803] 2025-04-26 16:46:44,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:46:44,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4986
[WARNING|trainer.py:803] 2025-04-26 16:46:45,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5029
5052
[WARNING|trainer.py:803] 2025-04-26 16:46:45,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4987
[WARNING|trainer.py:803] 2025-04-26 16:46:45,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:46,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5030
5053
[WARNING|trainer.py:803] 2025-04-26 16:46:46,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4988
[WARNING|trainer.py:803] 2025-04-26 16:46:47,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:47,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5031
5054
[WARNING|trainer.py:803] 2025-04-26 16:46:47,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4989
[WARNING|trainer.py:803] 2025-04-26 16:46:48,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:48,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5032
5055
[WARNING|trainer.py:803] 2025-04-26 16:46:49,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4990
[WARNING|trainer.py:803] 2025-04-26 16:46:49,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:49,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5033
5056
[WARNING|trainer.py:803] 2025-04-26 16:46:50,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4991
[WARNING|trainer.py:803] 2025-04-26 16:46:50,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:51,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5034
5057
[WARNING|trainer.py:803] 2025-04-26 16:46:51,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4992
[WARNING|trainer.py:803] 2025-04-26 16:46:51,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:52,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5035
5058
[WARNING|trainer.py:803] 2025-04-26 16:46:52,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4993
[WARNING|trainer.py:803] 2025-04-26 16:46:53,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:53,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5036
5059
[WARNING|trainer.py:803] 2025-04-26 16:46:53,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4994
[WARNING|trainer.py:803] 2025-04-26 16:46:54,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:54,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5037
5060
[WARNING|trainer.py:803] 2025-04-26 16:46:55,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4995
[WARNING|trainer.py:803] 2025-04-26 16:46:55,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:55,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5038
5061
[WARNING|trainer.py:803] 2025-04-26 16:46:56,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4996
[WARNING|trainer.py:803] 2025-04-26 16:46:56,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:56,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5039
5062
[WARNING|trainer.py:803] 2025-04-26 16:46:57,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4997
[WARNING|trainer.py:803] 2025-04-26 16:46:57,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:46:58,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5040
5063
[WARNING|trainer.py:803] 2025-04-26 16:46:58,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4998
[WARNING|trainer.py:803] 2025-04-26 16:46:59,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:46:59,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5041
5064
[WARNING|trainer.py:803] 2025-04-26 16:46:59,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4999
[WARNING|trainer.py:803] 2025-04-26 16:47:00,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:00,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5042
5065
[WARNING|trainer.py:803] 2025-04-26 16:47:00,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5000
[WARNING|trainer.py:803] 2025-04-26 16:47:01,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:01,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5043
5066
[WARNING|trainer.py:803] 2025-04-26 16:47:02,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5001
[WARNING|trainer.py:803] 2025-04-26 16:47:02,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:02,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5044
5067
[WARNING|trainer.py:803] 2025-04-26 16:47:03,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5002
[WARNING|trainer.py:803] 2025-04-26 16:47:03,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:04,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5045
5068
[WARNING|trainer.py:803] 2025-04-26 16:47:04,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5003
[WARNING|trainer.py:803] 2025-04-26 16:47:04,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:05,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5046
[WARNING|trainer.py:803] 2025-04-26 16:47:05,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5069
5004
[WARNING|trainer.py:803] 2025-04-26 16:47:06,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:06,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5047
[WARNING|trainer.py:803] 2025-04-26 16:47:06,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5070
5005
[WARNING|trainer.py:803] 2025-04-26 16:47:07,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:07,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5048
[WARNING|trainer.py:803] 2025-04-26 16:47:08,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5071
5006
[WARNING|trainer.py:803] 2025-04-26 16:47:08,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:47:08,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5049
[WARNING|trainer.py:803] 2025-04-26 16:47:09,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5072
5007
[WARNING|trainer.py:803] 2025-04-26 16:47:09,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:10,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5050
[WARNING|trainer.py:803] 2025-04-26 16:47:10,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5073
5008
[WARNING|trainer.py:803] 2025-04-26 16:47:10,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5051
[WARNING|trainer.py:803] 2025-04-26 16:47:11,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:11,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5074
5009
[WARNING|trainer.py:803] 2025-04-26 16:47:12,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5052
[WARNING|trainer.py:803] 2025-04-26 16:47:12,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:12,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5075
5010
[WARNING|trainer.py:803] 2025-04-26 16:47:13,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5053
[WARNING|trainer.py:803] 2025-04-26 16:47:13,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:13,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5076
5011
[WARNING|trainer.py:803] 2025-04-26 16:47:14,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:14,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5054
[WARNING|trainer.py:803] 2025-04-26 16:47:15,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5077
5012
[WARNING|trainer.py:803] 2025-04-26 16:47:15,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5055
[WARNING|trainer.py:803] 2025-04-26 16:47:16,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:16,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5078
5013
[WARNING|trainer.py:803] 2025-04-26 16:47:16,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5056
[WARNING|trainer.py:803] 2025-04-26 16:47:17,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:17,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5079
5014
[WARNING|trainer.py:803] 2025-04-26 16:47:18,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5057
[WARNING|trainer.py:803] 2025-04-26 16:47:18,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:18,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5080
5015
[WARNING|trainer.py:803] 2025-04-26 16:47:19,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:19,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5058
[WARNING|trainer.py:803] 2025-04-26 16:47:19,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5081
5016
[WARNING|trainer.py:803] 2025-04-26 16:47:20,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:20,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5059
[WARNING|trainer.py:803] 2025-04-26 16:47:21,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5082
5017
[WARNING|trainer.py:803] 2025-04-26 16:47:21,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:21,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5060
[WARNING|trainer.py:803] 2025-04-26 16:47:22,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5083
5018
[WARNING|trainer.py:803] 2025-04-26 16:47:22,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:23,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5061
[WARNING|trainer.py:803] 2025-04-26 16:47:23,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5084
5019
[WARNING|trainer.py:803] 2025-04-26 16:47:24,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:24,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5062
[WARNING|trainer.py:803] 2025-04-26 16:47:24,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5085
5020
[WARNING|trainer.py:803] 2025-04-26 16:47:25,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:25,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5063
[WARNING|trainer.py:803] 2025-04-26 16:47:25,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5086
5021
[WARNING|trainer.py:803] 2025-04-26 16:47:26,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:26,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5064
[WARNING|trainer.py:803] 2025-04-26 16:47:26,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5087
5022
[WARNING|trainer.py:803] 2025-04-26 16:47:27,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:27,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5065
[WARNING|trainer.py:803] 2025-04-26 16:47:28,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5088
5023
[WARNING|trainer.py:803] 2025-04-26 16:47:28,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:47:29,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5066
[WARNING|trainer.py:803] 2025-04-26 16:47:29,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5089
5024
[WARNING|trainer.py:803] 2025-04-26 16:47:29,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:30,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5067
[WARNING|trainer.py:803] 2025-04-26 16:47:30,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5090
5025
[WARNING|trainer.py:803] 2025-04-26 16:47:31,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:31,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5068
[WARNING|trainer.py:803] 2025-04-26 16:47:31,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5091
5026
[WARNING|trainer.py:803] 2025-04-26 16:47:32,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:32,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5069
[WARNING|trainer.py:803] 2025-04-26 16:47:32,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5092
5027
[WARNING|trainer.py:803] 2025-04-26 16:47:33,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:33,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5070
[WARNING|trainer.py:803] 2025-04-26 16:47:34,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5093
5028
[WARNING|trainer.py:803] 2025-04-26 16:47:34,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:35,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5071
[WARNING|trainer.py:803] 2025-04-26 16:47:35,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5094
5029
[WARNING|trainer.py:803] 2025-04-26 16:47:35,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:47:36,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5072
[WARNING|trainer.py:803] 2025-04-26 16:47:36,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5095
5030
[WARNING|trainer.py:803] 2025-04-26 16:47:37,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:37,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5073
[WARNING|trainer.py:803] 2025-04-26 16:47:37,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5096
5031
[WARNING|trainer.py:803] 2025-04-26 16:47:38,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:38,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5074
[WARNING|trainer.py:803] 2025-04-26 16:47:38,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5097
5032
[WARNING|trainer.py:803] 2025-04-26 16:47:39,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5075
[WARNING|trainer.py:803] 2025-04-26 16:47:39,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:40,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5098
5033
[WARNING|trainer.py:803] 2025-04-26 16:47:40,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5076
[WARNING|trainer.py:803] 2025-04-26 16:47:41,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:41,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5099
5034
[WARNING|trainer.py:803] 2025-04-26 16:47:41,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5077
[WARNING|trainer.py:803] 2025-04-26 16:47:42,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:42,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5100
5035
[WARNING|trainer.py:803] 2025-04-26 16:47:42,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5078
[WARNING|trainer.py:803] 2025-04-26 16:47:43,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:43,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5101
5036
[WARNING|trainer.py:803] 2025-04-26 16:47:44,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5079
[WARNING|trainer.py:803] 2025-04-26 16:47:44,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:47:44,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5037
5102
[WARNING|trainer.py:803] 2025-04-26 16:47:45,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5080
[WARNING|trainer.py:803] 2025-04-26 16:47:45,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:46,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5038
[WARNING|trainer.py:803] 2025-04-26 16:47:46,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5103
5081
[WARNING|trainer.py:803] 2025-04-26 16:47:47,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5039
[WARNING|trainer.py:803] 2025-04-26 16:47:47,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:47,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5104
5082
[WARNING|trainer.py:803] 2025-04-26 16:47:48,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5040
[WARNING|trainer.py:803] 2025-04-26 16:47:48,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:47:48,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5083
5105
[WARNING|trainer.py:803] 2025-04-26 16:47:49,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5041
[WARNING|trainer.py:803] 2025-04-26 16:47:50,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:50,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5084
[WARNING|trainer.py:803] 2025-04-26 16:47:50,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5106
5042
[WARNING|trainer.py:803] 2025-04-26 16:47:51,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:51,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5085
[WARNING|trainer.py:803] 2025-04-26 16:47:51,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5107
5043
[WARNING|trainer.py:803] 2025-04-26 16:47:52,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5086
[WARNING|trainer.py:803] 2025-04-26 16:47:53,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:53,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5044
5108
[WARNING|trainer.py:803] 2025-04-26 16:47:53,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5087
[WARNING|trainer.py:803] 2025-04-26 16:47:54,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:54,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5045
5109
[WARNING|trainer.py:803] 2025-04-26 16:47:54,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5088
[WARNING|trainer.py:803] 2025-04-26 16:47:55,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:55,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5046
[WARNING|trainer.py:803] 2025-04-26 16:47:56,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5110
5089
[WARNING|trainer.py:803] 2025-04-26 16:47:56,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5047
[WARNING|trainer.py:803] 2025-04-26 16:47:57,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:47:57,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5111
5090
[WARNING|trainer.py:803] 2025-04-26 16:47:57,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5048
[WARNING|trainer.py:803] 2025-04-26 16:47:58,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:47:58,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5091
[WARNING|trainer.py:803] 2025-04-26 16:47:58,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5112
5049
[WARNING|trainer.py:803] 2025-04-26 16:47:59,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:47:59,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5092
[WARNING|trainer.py:803] 2025-04-26 16:48:00,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5113
5050
[WARNING|trainer.py:803] 2025-04-26 16:48:00,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:01,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5093
[WARNING|trainer.py:803] 2025-04-26 16:48:01,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5051
5114
[WARNING|trainer.py:803] 2025-04-26 16:48:02,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5094
[WARNING|trainer.py:803] 2025-04-26 16:48:02,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:02,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5052
5115
[WARNING|trainer.py:803] 2025-04-26 16:48:03,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5095
[WARNING|trainer.py:803] 2025-04-26 16:48:03,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:04,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5053
5116
[WARNING|trainer.py:803] 2025-04-26 16:48:04,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5096
[WARNING|trainer.py:803] 2025-04-26 16:48:04,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5054
[WARNING|trainer.py:803] 2025-04-26 16:48:05,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:48:05,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5117
5097
[WARNING|trainer.py:803] 2025-04-26 16:48:06,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5055
[WARNING|trainer.py:803] 2025-04-26 16:48:06,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:06,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5098
[WARNING|trainer.py:803] 2025-04-26 16:48:07,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5118
5056
[WARNING|trainer.py:803] 2025-04-26 16:48:08,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:08,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:08,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
5099
NoYes
5119
5057
[WARNING|trainer.py:803] 2025-04-26 16:48:09,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:09,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5100
[WARNING|trainer.py:803] 2025-04-26 16:48:09,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5058
5120
[WARNING|trainer.py:803] 2025-04-26 16:48:10,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:10,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:10,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5101
5059
5121
[WARNING|trainer.py:803] 2025-04-26 16:48:11,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:12,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:12,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5102
5060
5122
[WARNING|trainer.py:803] 2025-04-26 16:48:13,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:13,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5061
[WARNING|trainer.py:803] 2025-04-26 16:48:13,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5103
5123
[WARNING|trainer.py:803] 2025-04-26 16:48:14,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:14,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5062
[WARNING|trainer.py:803] 2025-04-26 16:48:15,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5104
[WARNING|trainer.py:803] 2025-04-26 16:48:15,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5124
5063
[WARNING|trainer.py:803] 2025-04-26 16:48:16,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:16,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5105
[WARNING|trainer.py:803] 2025-04-26 16:48:16,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5125
5064
[WARNING|trainer.py:803] 2025-04-26 16:48:17,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:17,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5106
[WARNING|trainer.py:803] 2025-04-26 16:48:18,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5065
5126
[WARNING|trainer.py:803] 2025-04-26 16:48:18,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:19,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:19,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5107
5066
5127
[WARNING|trainer.py:803] 2025-04-26 16:48:20,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:20,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:20,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5108
5067
5128
[WARNING|trainer.py:803] 2025-04-26 16:48:21,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:21,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:21,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5068
5109
5129
[WARNING|trainer.py:803] 2025-04-26 16:48:22,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:22,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5069
[WARNING|trainer.py:803] 2025-04-26 16:48:23,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5110
5130
[WARNING|trainer.py:803] 2025-04-26 16:48:24,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:24,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5070
[WARNING|trainer.py:803] 2025-04-26 16:48:24,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5111
[WARNING|trainer.py:803] 2025-04-26 16:48:25,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5131
5071
[WARNING|trainer.py:803] 2025-04-26 16:48:25,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:26,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5112
[WARNING|trainer.py:803] 2025-04-26 16:48:26,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5132
5072
[WARNING|trainer.py:803] 2025-04-26 16:48:26,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:27,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5113
[WARNING|trainer.py:803] 2025-04-26 16:48:27,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5073
5133
[WARNING|trainer.py:803] 2025-04-26 16:48:28,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:28,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:28,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5114
5074
5134
[WARNING|trainer.py:803] 2025-04-26 16:48:29,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:48:29,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:30,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5115
5075
5135
[WARNING|trainer.py:803] 2025-04-26 16:48:31,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:48:31,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5076
[WARNING|trainer.py:803] 2025-04-26 16:48:31,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5116
5136
[WARNING|trainer.py:803] 2025-04-26 16:48:32,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:32,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5077
[WARNING|trainer.py:803] 2025-04-26 16:48:32,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5117
5137
[WARNING|trainer.py:803] 2025-04-26 16:48:33,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:33,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5078
[WARNING|trainer.py:803] 2025-04-26 16:48:34,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5118
[WARNING|trainer.py:803] 2025-04-26 16:48:34,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5138
5079
[WARNING|trainer.py:803] 2025-04-26 16:48:35,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:35,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5119
[WARNING|trainer.py:803] 2025-04-26 16:48:35,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5139
5080
[WARNING|trainer.py:803] 2025-04-26 16:48:36,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:36,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:37,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5120
5081
5140
[WARNING|trainer.py:803] 2025-04-26 16:48:37,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:38,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:38,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5121
5082
5141
[WARNING|trainer.py:803] 2025-04-26 16:48:39,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:39,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:39,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5083
5122
5142
[WARNING|trainer.py:803] 2025-04-26 16:48:40,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:40,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5084
[WARNING|trainer.py:803] 2025-04-26 16:48:41,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5123
5143
[WARNING|trainer.py:803] 2025-04-26 16:48:41,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:42,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5085
[WARNING|trainer.py:803] 2025-04-26 16:48:42,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5124
5144
[WARNING|trainer.py:803] 2025-04-26 16:48:43,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5086
[WARNING|trainer.py:803] 2025-04-26 16:48:43,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:43,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5125
[WARNING|trainer.py:803] 2025-04-26 16:48:44,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5145
5087
[WARNING|trainer.py:803] 2025-04-26 16:48:44,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:48:45,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5126
[WARNING|trainer.py:803] 2025-04-26 16:48:45,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5088
5146
[WARNING|trainer.py:803] 2025-04-26 16:48:46,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:46,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:46,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5127
5089
5147
[WARNING|trainer.py:803] 2025-04-26 16:48:47,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:48:47,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:48,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5128
5090
5148
[WARNING|trainer.py:803] 2025-04-26 16:48:48,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:48,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5091
[WARNING|trainer.py:803] 2025-04-26 16:48:49,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5129
5149
[WARNING|trainer.py:803] 2025-04-26 16:48:50,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:50,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5092
[WARNING|trainer.py:803] 2025-04-26 16:48:50,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5130
[WARNING|trainer.py:803] 2025-04-26 16:48:51,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5150
[WARNING|trainer.py:803] 2025-04-26 16:48:51,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5093
5131
[WARNING|trainer.py:803] 2025-04-26 16:48:52,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:52,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5151
5094
[WARNING|trainer.py:803] 2025-04-26 16:48:52,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5132
[WARNING|trainer.py:803] 2025-04-26 16:48:53,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:53,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5095
5152
[WARNING|trainer.py:803] 2025-04-26 16:48:54,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5133
[WARNING|trainer.py:803] 2025-04-26 16:48:54,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:54,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5096
5153
[WARNING|trainer.py:803] 2025-04-26 16:48:55,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:56,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5134
[WARNING|trainer.py:803] 2025-04-26 16:48:56,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5097
5154
[WARNING|trainer.py:803] 2025-04-26 16:48:57,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:48:57,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5135
5098
[WARNING|trainer.py:803] 2025-04-26 16:48:57,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5155
[WARNING|trainer.py:803] 2025-04-26 16:48:58,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:48:58,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5099
5136
[WARNING|trainer.py:803] 2025-04-26 16:48:59,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5156
[WARNING|trainer.py:803] 2025-04-26 16:48:59,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:48:59,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5100
5137
[WARNING|trainer.py:803] 2025-04-26 16:49:00,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:00,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5157
[WARNING|trainer.py:803] 2025-04-26 16:49:01,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5101
5138
[WARNING|trainer.py:803] 2025-04-26 16:49:01,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:02,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5158
[WARNING|trainer.py:803] 2025-04-26 16:49:02,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5102
5139
[WARNING|trainer.py:803] 2025-04-26 16:49:03,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:03,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5159
[WARNING|trainer.py:803] 2025-04-26 16:49:03,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5103
5140
[WARNING|trainer.py:803] 2025-04-26 16:49:04,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:04,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5160
[WARNING|trainer.py:803] 2025-04-26 16:49:05,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5104
5141
[WARNING|trainer.py:803] 2025-04-26 16:49:05,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:06,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5161
[WARNING|trainer.py:803] 2025-04-26 16:49:06,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5105
5142
[WARNING|trainer.py:803] 2025-04-26 16:49:07,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:07,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5162
[WARNING|trainer.py:803] 2025-04-26 16:49:07,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5106
5143
[WARNING|trainer.py:803] 2025-04-26 16:49:08,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:09,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5163
[WARNING|trainer.py:803] 2025-04-26 16:49:09,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5107
5144
[WARNING|trainer.py:803] 2025-04-26 16:49:09,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:10,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5164
[WARNING|trainer.py:803] 2025-04-26 16:49:10,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5108
5145
[WARNING|trainer.py:803] 2025-04-26 16:49:11,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:11,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5165
[WARNING|trainer.py:803] 2025-04-26 16:49:12,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5109
5146
[WARNING|trainer.py:803] 2025-04-26 16:49:12,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:13,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5166
[WARNING|trainer.py:803] 2025-04-26 16:49:13,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5110
5147
[WARNING|trainer.py:803] 2025-04-26 16:49:14,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:14,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5167
[WARNING|trainer.py:803] 2025-04-26 16:49:14,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5111
5148
[WARNING|trainer.py:803] 2025-04-26 16:49:15,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5168
[WARNING|trainer.py:803] 2025-04-26 16:49:15,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:16,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5112
5149
[WARNING|trainer.py:803] 2025-04-26 16:49:16,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5169
[WARNING|trainer.py:803] 2025-04-26 16:49:17,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:17,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5113
5150
[WARNING|trainer.py:803] 2025-04-26 16:49:18,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:18,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5170
[WARNING|trainer.py:803] 2025-04-26 16:49:18,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5114
5151
[WARNING|trainer.py:803] 2025-04-26 16:49:19,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:20,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5171
[WARNING|trainer.py:803] 2025-04-26 16:49:20,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5115
5152
[WARNING|trainer.py:803] 2025-04-26 16:49:20,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:21,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5172
[WARNING|trainer.py:803] 2025-04-26 16:49:21,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5116
5153
[WARNING|trainer.py:803] 2025-04-26 16:49:22,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:22,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5173
[WARNING|trainer.py:803] 2025-04-26 16:49:23,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5117
5154
[WARNING|trainer.py:803] 2025-04-26 16:49:23,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:24,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5174
[WARNING|trainer.py:803] 2025-04-26 16:49:24,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5118
5155
[WARNING|trainer.py:803] 2025-04-26 16:49:25,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:25,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5175
[WARNING|trainer.py:803] 2025-04-26 16:49:25,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5119
5156
[WARNING|trainer.py:803] 2025-04-26 16:49:26,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:26,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5176
[WARNING|trainer.py:803] 2025-04-26 16:49:27,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5120
5157
[WARNING|trainer.py:803] 2025-04-26 16:49:27,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:28,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5177
[WARNING|trainer.py:803] 2025-04-26 16:49:28,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5121
5158
[WARNING|trainer.py:803] 2025-04-26 16:49:29,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5178
[WARNING|trainer.py:803] 2025-04-26 16:49:29,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:29,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5159
5122
[WARNING|trainer.py:803] 2025-04-26 16:49:30,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5179
[WARNING|trainer.py:803] 2025-04-26 16:49:31,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:31,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5160
5123
[WARNING|trainer.py:803] 2025-04-26 16:49:31,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5180
[WARNING|trainer.py:803] 2025-04-26 16:49:32,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:32,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5161
5124
[WARNING|trainer.py:803] 2025-04-26 16:49:33,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:33,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5181
[WARNING|trainer.py:803] 2025-04-26 16:49:34,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5162
5125
[WARNING|trainer.py:803] 2025-04-26 16:49:34,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5182
[WARNING|trainer.py:803] 2025-04-26 16:49:35,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:35,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5163
5126
[WARNING|trainer.py:803] 2025-04-26 16:49:36,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:36,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5183
[WARNING|trainer.py:803] 2025-04-26 16:49:36,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5164
5127
[WARNING|trainer.py:803] 2025-04-26 16:49:37,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:37,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5184
[WARNING|trainer.py:803] 2025-04-26 16:49:38,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5165
5128
[WARNING|trainer.py:803] 2025-04-26 16:49:39,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:39,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5185
[WARNING|trainer.py:803] 2025-04-26 16:49:39,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5166
5129
[WARNING|trainer.py:803] 2025-04-26 16:49:40,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:40,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5186
[WARNING|trainer.py:803] 2025-04-26 16:49:41,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5167
5130
[WARNING|trainer.py:803] 2025-04-26 16:49:41,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:42,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5187
[WARNING|trainer.py:803] 2025-04-26 16:49:42,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5168
5131
[WARNING|trainer.py:803] 2025-04-26 16:49:43,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:43,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5188
[WARNING|trainer.py:803] 2025-04-26 16:49:43,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5169
5132
[WARNING|trainer.py:803] 2025-04-26 16:49:44,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:44,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:45,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5189
5170
5133
[WARNING|trainer.py:803] 2025-04-26 16:49:45,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:46,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:46,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5190
5171
5134
[WARNING|trainer.py:803] 2025-04-26 16:49:47,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:47,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:47,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5191
5172
5135
[WARNING|trainer.py:803] 2025-04-26 16:49:48,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:48,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:49,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5192
5173
5136
[WARNING|trainer.py:803] 2025-04-26 16:49:50,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:50,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5193
[WARNING|trainer.py:803] 2025-04-26 16:49:50,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5174
5137
[WARNING|trainer.py:803] 2025-04-26 16:49:51,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:51,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5194
[WARNING|trainer.py:803] 2025-04-26 16:49:51,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5175
5138
[WARNING|trainer.py:803] 2025-04-26 16:49:52,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:53,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5195
[WARNING|trainer.py:803] 2025-04-26 16:49:53,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5176
5139
[WARNING|trainer.py:803] 2025-04-26 16:49:54,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:54,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:49:54,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5196
5177
5140
[WARNING|trainer.py:803] 2025-04-26 16:49:55,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:55,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:56,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5197
5178
5141
[WARNING|trainer.py:803] 2025-04-26 16:49:56,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:57,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:57,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5198
5179
5142
[WARNING|trainer.py:803] 2025-04-26 16:49:58,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:49:58,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:49:58,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5199
5180
5143
[WARNING|trainer.py:803] 2025-04-26 16:49:59,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:49:59,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:00,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5200
5181
5144
[WARNING|trainer.py:803] 2025-04-26 16:50:01,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:01,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:01,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5201
5182
5145
[WARNING|trainer.py:803] 2025-04-26 16:50:02,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:02,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:02,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5202
5183
5146
[WARNING|trainer.py:803] 2025-04-26 16:50:03,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:04,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:04,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5203
5184
5147
[WARNING|trainer.py:803] 2025-04-26 16:50:05,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:05,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:05,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5204
5185
5148
[WARNING|trainer.py:803] 2025-04-26 16:50:06,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:06,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:07,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5205
5186
5149
[WARNING|trainer.py:803] 2025-04-26 16:50:08,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:08,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:08,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5206
5187
5150
[WARNING|trainer.py:803] 2025-04-26 16:50:09,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:09,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:09,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5188
5207
5151
[WARNING|trainer.py:803] 2025-04-26 16:50:11,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:11,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:11,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5189
5208
5152
[WARNING|trainer.py:803] 2025-04-26 16:50:12,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:12,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:50:12,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5190
5209
5153
[WARNING|trainer.py:803] 2025-04-26 16:50:13,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:13,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:14,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5191
5210
5154
[WARNING|trainer.py:803] 2025-04-26 16:50:15,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:15,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:15,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5211
5192
5155
[WARNING|trainer.py:803] 2025-04-26 16:50:16,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:50:16,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:16,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5212
5193
5156
[WARNING|trainer.py:803] 2025-04-26 16:50:18,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:18,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:18,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5213
5194
5157
[WARNING|trainer.py:803] 2025-04-26 16:50:19,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:19,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:19,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5214
5195
5158
[WARNING|trainer.py:803] 2025-04-26 16:50:20,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:20,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:20,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5196
5215
5159
[WARNING|trainer.py:803] 2025-04-26 16:50:22,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:22,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:22,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5197
5160
5216
[WARNING|trainer.py:803] 2025-04-26 16:50:23,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:23,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:23,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5198
5161
5217
[WARNING|trainer.py:803] 2025-04-26 16:50:25,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:25,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:25,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5199
5162
5218
[WARNING|trainer.py:803] 2025-04-26 16:50:26,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:26,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:26,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5200
5163
5219
[WARNING|trainer.py:803] 2025-04-26 16:50:27,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:27,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:28,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5201
5164
5220
[WARNING|trainer.py:803] 2025-04-26 16:50:29,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:29,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:29,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5202
5165
5221
[WARNING|trainer.py:803] 2025-04-26 16:50:30,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:30,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:30,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5166
5203
5222
[WARNING|trainer.py:803] 2025-04-26 16:50:31,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:31,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:32,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5167
5204
5223
[WARNING|trainer.py:803] 2025-04-26 16:50:33,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:33,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:33,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5168
5205
5224
[WARNING|trainer.py:803] 2025-04-26 16:50:34,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:34,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:35,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5169
5206
5225
[WARNING|trainer.py:803] 2025-04-26 16:50:36,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:36,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:36,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5170
5207
5226
[WARNING|trainer.py:803] 2025-04-26 16:50:37,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:37,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:37,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5171
5208
5227
[WARNING|trainer.py:803] 2025-04-26 16:50:38,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:38,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:50:39,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5172
5209
5228
[WARNING|trainer.py:803] 2025-04-26 16:50:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:40,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:40,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5210
5173
5229
[WARNING|trainer.py:803] 2025-04-26 16:50:41,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:41,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:42,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5211
5174
5230
[WARNING|trainer.py:803] 2025-04-26 16:50:43,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:50:43,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:43,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5212
5175
5231
[WARNING|trainer.py:803] 2025-04-26 16:50:44,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:44,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:44,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5213
5176
5232
[WARNING|trainer.py:803] 2025-04-26 16:50:45,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:45,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5214
[WARNING|trainer.py:803] 2025-04-26 16:50:46,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5177
5233
[WARNING|trainer.py:803] 2025-04-26 16:50:47,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:50:47,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:47,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5215
5178
5234
[WARNING|trainer.py:803] 2025-04-26 16:50:48,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:48,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:49,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5216
5179
5235
[WARNING|trainer.py:803] 2025-04-26 16:50:50,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:50,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:50,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5217
5180
5236
[WARNING|trainer.py:803] 2025-04-26 16:50:51,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:51,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:51,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5181
5218
5237
[WARNING|trainer.py:803] 2025-04-26 16:50:52,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:52,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:53,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5182
5219
5238
[WARNING|trainer.py:803] 2025-04-26 16:50:54,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:54,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:54,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5220
5183
5239
[WARNING|trainer.py:803] 2025-04-26 16:50:55,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:55,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:56,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5221
5184
5240
[WARNING|trainer.py:803] 2025-04-26 16:50:57,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:50:57,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:57,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5185
5222
5241
[WARNING|trainer.py:803] 2025-04-26 16:50:58,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:58,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:50:58,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5186
5223
5242
[WARNING|trainer.py:803] 2025-04-26 16:50:59,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:50:59,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:00,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5187
5224
5243
[WARNING|trainer.py:803] 2025-04-26 16:51:01,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:01,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:01,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5188
5225
5244
[WARNING|trainer.py:803] 2025-04-26 16:51:02,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:02,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:03,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5189
5226
5245
[WARNING|trainer.py:803] 2025-04-26 16:51:04,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:04,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:04,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5190
5227
5246
[WARNING|trainer.py:803] 2025-04-26 16:51:05,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:05,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5191
[WARNING|trainer.py:803] 2025-04-26 16:51:05,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5228
5247
[WARNING|trainer.py:803] 2025-04-26 16:51:06,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:06,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:07,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5192
5229
5248
[WARNING|trainer.py:803] 2025-04-26 16:51:08,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:08,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:08,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5193
5230
5249
[WARNING|trainer.py:803] 2025-04-26 16:51:09,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:09,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:10,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5194
5231
5250
[WARNING|trainer.py:803] 2025-04-26 16:51:10,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:11,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:11,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5195
5232
5251
[WARNING|trainer.py:803] 2025-04-26 16:51:12,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:12,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:12,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5196
5233
5252
[WARNING|trainer.py:803] 2025-04-26 16:51:13,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:13,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:14,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5234
5197
5253
[WARNING|trainer.py:803] 2025-04-26 16:51:15,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:15,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:15,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5235
5198
5254
[WARNING|trainer.py:803] 2025-04-26 16:51:16,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:16,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:16,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5236
5199
5255
[WARNING|trainer.py:803] 2025-04-26 16:51:17,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:17,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:18,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5200
5237
5256
[WARNING|trainer.py:803] 2025-04-26 16:51:19,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:19,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:19,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5201
5238
5257
[WARNING|trainer.py:803] 2025-04-26 16:51:20,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:20,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:21,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5239
5202
5258
[WARNING|trainer.py:803] 2025-04-26 16:51:22,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:22,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:22,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5240
5203
5259
[WARNING|trainer.py:803] 2025-04-26 16:51:23,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:23,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:23,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5241
5204
5260
[WARNING|trainer.py:803] 2025-04-26 16:51:24,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:25,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:25,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5242
5205
5261
[WARNING|trainer.py:803] 2025-04-26 16:51:26,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:26,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:26,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5243
5206
5262
[WARNING|trainer.py:803] 2025-04-26 16:51:27,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:27,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:28,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5244
5207
5263
[WARNING|trainer.py:803] 2025-04-26 16:51:29,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:29,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:29,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5245
5208
5264
[WARNING|trainer.py:803] 2025-04-26 16:51:30,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:30,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:51:30,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5246
5209
5265
[WARNING|trainer.py:803] 2025-04-26 16:51:32,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:32,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:32,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5247
5210
5266
[WARNING|trainer.py:803] 2025-04-26 16:51:33,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:33,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:33,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5248
5211
5267
[WARNING|trainer.py:803] 2025-04-26 16:51:34,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:34,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:51:34,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5249
5212
5268
[WARNING|trainer.py:803] 2025-04-26 16:51:36,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:36,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:36,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5250
5213
5269
[WARNING|trainer.py:803] 2025-04-26 16:51:37,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:37,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:37,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5251
5270
5214
[WARNING|trainer.py:803] 2025-04-26 16:51:39,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:39,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:39,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5252
5271
5215
[WARNING|trainer.py:803] 2025-04-26 16:51:40,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:40,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:40,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5253
5272
5216
[WARNING|trainer.py:803] 2025-04-26 16:51:41,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:41,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:42,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5254
5273
5217
[WARNING|trainer.py:803] 2025-04-26 16:51:43,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:43,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:43,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5255
5274
5218
[WARNING|trainer.py:803] 2025-04-26 16:51:44,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:44,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:44,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5256
5275
5219
[WARNING|trainer.py:803] 2025-04-26 16:51:45,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:45,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:46,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5257
5276
5220
[WARNING|trainer.py:803] 2025-04-26 16:51:47,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:47,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:47,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5277
5258
5221
[WARNING|trainer.py:803] 2025-04-26 16:51:48,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:48,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:49,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5278
5259
5222
[WARNING|trainer.py:803] 2025-04-26 16:51:50,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:50,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:50,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5279
5260
5223
[WARNING|trainer.py:803] 2025-04-26 16:51:51,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:51,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:51,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5280
5261
5224
[WARNING|trainer.py:803] 2025-04-26 16:51:52,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:52,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:53,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5262
5281
5225
[WARNING|trainer.py:803] 2025-04-26 16:51:54,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:54,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:54,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5263
5282
5226
[WARNING|trainer.py:803] 2025-04-26 16:51:55,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:55,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:56,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5264
5283
5227
[WARNING|trainer.py:803] 2025-04-26 16:51:57,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:51:57,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:51:57,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5265
5284
5228
[WARNING|trainer.py:803] 2025-04-26 16:51:58,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:51:58,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5285
5266
[WARNING|trainer.py:803] 2025-04-26 16:51:59,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5229
[WARNING|trainer.py:803] 2025-04-26 16:51:59,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:51:59,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5286
5267
[WARNING|trainer.py:803] 2025-04-26 16:52:00,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5230
[WARNING|trainer.py:803] 2025-04-26 16:52:01,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:01,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5287
5268
[WARNING|trainer.py:803] 2025-04-26 16:52:02,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5231
[WARNING|trainer.py:803] 2025-04-26 16:52:02,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:02,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5288
5269
[WARNING|trainer.py:803] 2025-04-26 16:52:03,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5232
[WARNING|trainer.py:803] 2025-04-26 16:52:04,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:04,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5270
5289
[WARNING|trainer.py:803] 2025-04-26 16:52:04,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5233
[WARNING|trainer.py:803] 2025-04-26 16:52:05,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:05,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5271
5290
[WARNING|trainer.py:803] 2025-04-26 16:52:06,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5234
[WARNING|trainer.py:803] 2025-04-26 16:52:06,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:06,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5272
5291
[WARNING|trainer.py:803] 2025-04-26 16:52:07,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5235
[WARNING|trainer.py:803] 2025-04-26 16:52:08,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:08,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5273
5292
[WARNING|trainer.py:803] 2025-04-26 16:52:08,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5236
[WARNING|trainer.py:803] 2025-04-26 16:52:09,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:09,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5274
5293
[WARNING|trainer.py:803] 2025-04-26 16:52:10,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5237
[WARNING|trainer.py:803] 2025-04-26 16:52:10,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:10,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5275
5294
[WARNING|trainer.py:803] 2025-04-26 16:52:11,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5238
[WARNING|trainer.py:803] 2025-04-26 16:52:12,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:12,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5276
5295
[WARNING|trainer.py:803] 2025-04-26 16:52:12,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5239
[WARNING|trainer.py:803] 2025-04-26 16:52:13,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:13,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5277
5296
[WARNING|trainer.py:803] 2025-04-26 16:52:14,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5240
[WARNING|trainer.py:803] 2025-04-26 16:52:14,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:15,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5278
5297
[WARNING|trainer.py:803] 2025-04-26 16:52:15,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5241
[WARNING|trainer.py:803] 2025-04-26 16:52:16,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:16,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5279
5298
[WARNING|trainer.py:803] 2025-04-26 16:52:17,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5242
[WARNING|trainer.py:803] 2025-04-26 16:52:17,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:17,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5280
5299
[WARNING|trainer.py:803] 2025-04-26 16:52:18,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5243
[WARNING|trainer.py:803] 2025-04-26 16:52:19,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:19,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5281
5300
[WARNING|trainer.py:803] 2025-04-26 16:52:19,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:20,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5244
[WARNING|trainer.py:803] 2025-04-26 16:52:20,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5282
5301
[WARNING|trainer.py:803] 2025-04-26 16:52:21,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:21,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5245
[WARNING|trainer.py:803] 2025-04-26 16:52:22,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5283
5302
[WARNING|trainer.py:803] 2025-04-26 16:52:22,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:23,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5246
[WARNING|trainer.py:803] 2025-04-26 16:52:23,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5284
5303
[WARNING|trainer.py:803] 2025-04-26 16:52:24,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:24,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5247
[WARNING|trainer.py:803] 2025-04-26 16:52:24,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5285
5304
[WARNING|trainer.py:803] 2025-04-26 16:52:25,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:26,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5248
[WARNING|trainer.py:803] 2025-04-26 16:52:26,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5286
5305
[WARNING|trainer.py:803] 2025-04-26 16:52:27,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:27,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5249
[WARNING|trainer.py:803] 2025-04-26 16:52:27,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5287
5306
[WARNING|trainer.py:803] 2025-04-26 16:52:28,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:28,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5250
[WARNING|trainer.py:803] 2025-04-26 16:52:28,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5288
5307
[WARNING|trainer.py:803] 2025-04-26 16:52:29,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:30,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5251
[WARNING|trainer.py:803] 2025-04-26 16:52:30,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5289
5308
[WARNING|trainer.py:803] 2025-04-26 16:52:31,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:31,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5252
[WARNING|trainer.py:803] 2025-04-26 16:52:31,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5290
5309
[WARNING|trainer.py:803] 2025-04-26 16:52:32,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:33,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5253
[WARNING|trainer.py:803] 2025-04-26 16:52:33,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5291
5310
[WARNING|trainer.py:803] 2025-04-26 16:52:33,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:34,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5254
[WARNING|trainer.py:803] 2025-04-26 16:52:34,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5292
5311
[WARNING|trainer.py:803] 2025-04-26 16:52:35,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5255
[WARNING|trainer.py:803] 2025-04-26 16:52:35,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:35,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5293
5312
[WARNING|trainer.py:803] 2025-04-26 16:52:36,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:37,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5256
[WARNING|trainer.py:803] 2025-04-26 16:52:37,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5294
5313
[WARNING|trainer.py:803] 2025-04-26 16:52:38,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:38,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5257
[WARNING|trainer.py:803] 2025-04-26 16:52:38,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5295
5314
[WARNING|trainer.py:803] 2025-04-26 16:52:39,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:39,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5258
[WARNING|trainer.py:803] 2025-04-26 16:52:40,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5296
5315
[WARNING|trainer.py:803] 2025-04-26 16:52:40,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:41,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5259
[WARNING|trainer.py:803] 2025-04-26 16:52:41,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5297
5316
[WARNING|trainer.py:803] 2025-04-26 16:52:42,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:42,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5260
[WARNING|trainer.py:803] 2025-04-26 16:52:42,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5298
5317
[WARNING|trainer.py:803] 2025-04-26 16:52:43,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:44,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5261
[WARNING|trainer.py:803] 2025-04-26 16:52:44,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5299
5318
[WARNING|trainer.py:803] 2025-04-26 16:52:45,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:45,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5262
[WARNING|trainer.py:803] 2025-04-26 16:52:45,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5300
5319
[WARNING|trainer.py:803] 2025-04-26 16:52:46,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:46,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5263
[WARNING|trainer.py:803] 2025-04-26 16:52:47,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5301
5320
[WARNING|trainer.py:803] 2025-04-26 16:52:47,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5264
[WARNING|trainer.py:803] 2025-04-26 16:52:48,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:48,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5302
5321
[WARNING|trainer.py:803] 2025-04-26 16:52:49,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5265
[WARNING|trainer.py:803] 2025-04-26 16:52:49,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:49,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5303
5322
[WARNING|trainer.py:803] 2025-04-26 16:52:50,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5266
[WARNING|trainer.py:803] 2025-04-26 16:52:51,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:51,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5304
5323
[WARNING|trainer.py:803] 2025-04-26 16:52:51,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5267
[WARNING|trainer.py:803] 2025-04-26 16:52:52,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:52,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5305
5324
[WARNING|trainer.py:803] 2025-04-26 16:52:53,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5268
[WARNING|trainer.py:803] 2025-04-26 16:52:53,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:54,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5306
5325
[WARNING|trainer.py:803] 2025-04-26 16:52:54,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:52:55,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5269
[WARNING|trainer.py:803] 2025-04-26 16:52:55,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5307
5326
[WARNING|trainer.py:803] 2025-04-26 16:52:56,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:52:56,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5270
[WARNING|trainer.py:803] 2025-04-26 16:52:56,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5308
5327
[WARNING|trainer.py:803] 2025-04-26 16:52:57,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5271
[WARNING|trainer.py:803] 2025-04-26 16:52:58,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:52:58,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5309
5328
[WARNING|trainer.py:803] 2025-04-26 16:52:58,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5272
[WARNING|trainer.py:803] 2025-04-26 16:52:59,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:52:59,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5310
5329
[WARNING|trainer.py:803] 2025-04-26 16:53:00,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5273
[WARNING|trainer.py:803] 2025-04-26 16:53:00,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:00,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5311
5330
[WARNING|trainer.py:803] 2025-04-26 16:53:01,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5274
[WARNING|trainer.py:803] 2025-04-26 16:53:02,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:02,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5312
5331
[WARNING|trainer.py:803] 2025-04-26 16:53:02,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5275
[WARNING|trainer.py:803] 2025-04-26 16:53:03,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:03,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5332
5313
[WARNING|trainer.py:803] 2025-04-26 16:53:04,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5276
[WARNING|trainer.py:803] 2025-04-26 16:53:04,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:04,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5333
5314
[WARNING|trainer.py:803] 2025-04-26 16:53:05,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5277
[WARNING|trainer.py:803] 2025-04-26 16:53:06,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:06,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5334
5315
[WARNING|trainer.py:803] 2025-04-26 16:53:07,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5278
[WARNING|trainer.py:803] 2025-04-26 16:53:07,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:07,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5335
5316
[WARNING|trainer.py:803] 2025-04-26 16:53:08,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:08,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5279
[WARNING|trainer.py:803] 2025-04-26 16:53:09,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5336
5317
[WARNING|trainer.py:803] 2025-04-26 16:53:09,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:10,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5280
[WARNING|trainer.py:803] 2025-04-26 16:53:10,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5337
5318
[WARNING|trainer.py:803] 2025-04-26 16:53:11,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:11,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5281
[WARNING|trainer.py:803] 2025-04-26 16:53:11,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5338
5319
[WARNING|trainer.py:803] 2025-04-26 16:53:12,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5282
[WARNING|trainer.py:803] 2025-04-26 16:53:13,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:13,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5339
5320
[WARNING|trainer.py:803] 2025-04-26 16:53:13,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5283
[WARNING|trainer.py:803] 2025-04-26 16:53:14,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:14,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5340
5321
[WARNING|trainer.py:803] 2025-04-26 16:53:15,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5284
[WARNING|trainer.py:803] 2025-04-26 16:53:15,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:15,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5341
5322
[WARNING|trainer.py:803] 2025-04-26 16:53:16,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5285
[WARNING|trainer.py:803] 2025-04-26 16:53:17,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:17,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5342
5323
[WARNING|trainer.py:803] 2025-04-26 16:53:18,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5286
[WARNING|trainer.py:803] 2025-04-26 16:53:18,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:18,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5343
5324
[WARNING|trainer.py:803] 2025-04-26 16:53:19,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:20,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5287
[WARNING|trainer.py:803] 2025-04-26 16:53:20,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5344
5325
[WARNING|trainer.py:803] 2025-04-26 16:53:20,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:21,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:21,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5288
5345
5326
[WARNING|trainer.py:803] 2025-04-26 16:53:22,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:22,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:22,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5289
5327
5346
[WARNING|trainer.py:803] 2025-04-26 16:53:23,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:24,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:24,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5290
5328
5347
[WARNING|trainer.py:803] 2025-04-26 16:53:25,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:25,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:25,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5291
5329
5348
[WARNING|trainer.py:803] 2025-04-26 16:53:26,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:26,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:27,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5292
5330
5349
[WARNING|trainer.py:803] 2025-04-26 16:53:28,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:28,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:28,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5293
5331
5350
[WARNING|trainer.py:803] 2025-04-26 16:53:29,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:29,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:29,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5294
5332
5351
[WARNING|trainer.py:803] 2025-04-26 16:53:30,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:31,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:31,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5295
5333
5352
[WARNING|trainer.py:803] 2025-04-26 16:53:32,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:32,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:32,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5296
5334
5353
[WARNING|trainer.py:803] 2025-04-26 16:53:33,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:33,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:33,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5297
5335
5354
[WARNING|trainer.py:803] 2025-04-26 16:53:35,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:35,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:35,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5298
5336
5355
[WARNING|trainer.py:803] 2025-04-26 16:53:36,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:36,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:36,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5299
5337
5356
[WARNING|trainer.py:803] 2025-04-26 16:53:37,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:38,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:38,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
5300
5338
5357
[WARNING|trainer.py:803] 2025-04-26 16:53:39,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:39,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:39,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5301
5339
5358
[WARNING|trainer.py:803] 2025-04-26 16:53:40,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:40,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:40,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5340
5359
5302
[WARNING|trainer.py:803] 2025-04-26 16:53:42,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:42,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:42,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5360
5341
5303
[WARNING|trainer.py:803] 2025-04-26 16:53:43,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:43,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:43,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5361
5342
5304
[WARNING|trainer.py:803] 2025-04-26 16:53:44,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:44,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:45,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5362
5343
5305
[WARNING|trainer.py:803] 2025-04-26 16:53:46,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:46,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:46,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5363
5344
5306
[WARNING|trainer.py:803] 2025-04-26 16:53:47,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:47,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:47,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5364
5345
5307
[WARNING|trainer.py:803] 2025-04-26 16:53:49,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:49,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:49,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5365
5346
5308
[WARNING|trainer.py:803] 2025-04-26 16:53:50,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:50,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:53:50,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5366
5347
5309
[WARNING|trainer.py:803] 2025-04-26 16:53:51,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:51,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:52,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5367
5348
5310
[WARNING|trainer.py:803] 2025-04-26 16:53:53,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:53,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:53,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5368
5349
5311
[WARNING|trainer.py:803] 2025-04-26 16:53:54,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:54,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:54,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5369
5350
5312
[WARNING|trainer.py:803] 2025-04-26 16:53:55,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:56,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:53:56,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5370
5351
5313
[WARNING|trainer.py:803] 2025-04-26 16:53:57,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:53:57,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:57,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5371
5352
5314
[WARNING|trainer.py:803] 2025-04-26 16:53:58,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:58,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:53:59,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5372
5353
5315
[WARNING|trainer.py:803] 2025-04-26 16:54:00,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:00,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:00,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5373
5354
5316
[WARNING|trainer.py:803] 2025-04-26 16:54:01,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:01,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:01,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5374
5355
5317
[WARNING|trainer.py:803] 2025-04-26 16:54:02,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:03,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:03,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5375
5356
5318
[WARNING|trainer.py:803] 2025-04-26 16:54:04,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:04,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 16:54:04,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5376
5357
5319
[WARNING|trainer.py:803] 2025-04-26 16:54:05,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:05,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:06,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5377
5358
5320
[WARNING|trainer.py:803] 2025-04-26 16:54:06,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:07,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5378
[WARNING|trainer.py:803] 2025-04-26 16:54:07,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5359
5321
[WARNING|trainer.py:803] 2025-04-26 16:54:08,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:08,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5379
[WARNING|trainer.py:803] 2025-04-26 16:54:08,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5360
5322
[WARNING|trainer.py:803] 2025-04-26 16:54:09,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:09,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5380
[WARNING|trainer.py:803] 2025-04-26 16:54:10,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5361
5323
[WARNING|trainer.py:803] 2025-04-26 16:54:11,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:11,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5381
[WARNING|trainer.py:803] 2025-04-26 16:54:11,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5362
5324
[WARNING|trainer.py:803] 2025-04-26 16:54:12,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:12,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5382
[WARNING|trainer.py:803] 2025-04-26 16:54:13,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5363
5325
[WARNING|trainer.py:803] 2025-04-26 16:54:13,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:14,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5383
[WARNING|trainer.py:803] 2025-04-26 16:54:14,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5364
5326
[WARNING|trainer.py:803] 2025-04-26 16:54:15,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:15,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5384
[WARNING|trainer.py:803] 2025-04-26 16:54:15,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5365
5327
[WARNING|trainer.py:803] 2025-04-26 16:54:16,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:16,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5385
[WARNING|trainer.py:803] 2025-04-26 16:54:17,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5366
5328
[WARNING|trainer.py:803] 2025-04-26 16:54:17,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:18,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5386
[WARNING|trainer.py:803] 2025-04-26 16:54:18,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5367
5329
[WARNING|trainer.py:803] 2025-04-26 16:54:19,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:19,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5387
[WARNING|trainer.py:803] 2025-04-26 16:54:19,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5368
5330
[WARNING|trainer.py:803] 2025-04-26 16:54:20,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:20,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5388
[WARNING|trainer.py:803] 2025-04-26 16:54:21,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5369
5331
[WARNING|trainer.py:803] 2025-04-26 16:54:21,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:22,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5389
[WARNING|trainer.py:803] 2025-04-26 16:54:22,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5370
5332
[WARNING|trainer.py:803] 2025-04-26 16:54:23,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:23,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5390
[WARNING|trainer.py:803] 2025-04-26 16:54:23,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5371
5333
[WARNING|trainer.py:803] 2025-04-26 16:54:24,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:25,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:25,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5391
5372
5334
[WARNING|trainer.py:803] 2025-04-26 16:54:26,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:26,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:26,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5392
5373
5335
[WARNING|trainer.py:803] 2025-04-26 16:54:27,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:27,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:27,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5393
5374
5336
[WARNING|trainer.py:803] 2025-04-26 16:54:28,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:29,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:29,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5394
5375
5337
[WARNING|trainer.py:803] 2025-04-26 16:54:30,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:30,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:30,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5395
5376
5338
[WARNING|trainer.py:803] 2025-04-26 16:54:31,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:32,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:32,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5396
5377
5339
[WARNING|trainer.py:803] 2025-04-26 16:54:33,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:33,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:33,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5397
5378
5340
[WARNING|trainer.py:803] 2025-04-26 16:54:34,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:34,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:34,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5398
5379
5341
[WARNING|trainer.py:803] 2025-04-26 16:54:35,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:36,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:36,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5399
5380
5342
[WARNING|trainer.py:803] 2025-04-26 16:54:37,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:37,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:37,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5400
5381
5343
[WARNING|trainer.py:803] 2025-04-26 16:54:38,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:38,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:39,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5401
5382
5344
[WARNING|trainer.py:803] 2025-04-26 16:54:40,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:40,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:40,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5402
5383
5345
[WARNING|trainer.py:803] 2025-04-26 16:54:41,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:41,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:41,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5384
5403
5346
[WARNING|trainer.py:803] 2025-04-26 16:54:43,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:43,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:43,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5385
5404
5347
[WARNING|trainer.py:803] 2025-04-26 16:54:44,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:44,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:44,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5386
5348
5405
[WARNING|trainer.py:803] 2025-04-26 16:54:45,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:46,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:46,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5387
5349
5406
[WARNING|trainer.py:803] 2025-04-26 16:54:47,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:47,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:47,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5388
5350
5407
[WARNING|trainer.py:803] 2025-04-26 16:54:48,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:48,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5389
[WARNING|trainer.py:803] 2025-04-26 16:54:49,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5351
5408
[WARNING|trainer.py:803] 2025-04-26 16:54:49,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:50,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5390
[WARNING|trainer.py:803] 2025-04-26 16:54:50,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5352
[WARNING|trainer.py:803] 2025-04-26 16:54:51,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5409
[WARNING|trainer.py:803] 2025-04-26 16:54:51,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5391
[WARNING|trainer.py:803] 2025-04-26 16:54:52,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5353
[WARNING|trainer.py:803] 2025-04-26 16:54:52,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5410
[WARNING|trainer.py:803] 2025-04-26 16:54:53,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5392
[WARNING|trainer.py:803] 2025-04-26 16:54:53,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5354
[WARNING|trainer.py:803] 2025-04-26 16:54:54,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5411
[WARNING|trainer.py:803] 2025-04-26 16:54:54,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5393
5355
[WARNING|trainer.py:803] 2025-04-26 16:54:55,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:54:55,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5412
[WARNING|trainer.py:803] 2025-04-26 16:54:55,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5394
5356
[WARNING|trainer.py:803] 2025-04-26 16:54:56,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:54:56,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:57,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
5413
5395
5357
[WARNING|trainer.py:803] 2025-04-26 16:54:58,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:58,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:58,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5396
5414
5358
[WARNING|trainer.py:803] 2025-04-26 16:54:59,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:54:59,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:54:59,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5397
5415
5359
[WARNING|trainer.py:803] 2025-04-26 16:55:00,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:01,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:01,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5398
5416
5360
[WARNING|trainer.py:803] 2025-04-26 16:55:02,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:02,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:02,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5399
5417
5361
[WARNING|trainer.py:803] 2025-04-26 16:55:03,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:04,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5400
[WARNING|trainer.py:803] 2025-04-26 16:55:04,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5362
5418
[WARNING|trainer.py:803] 2025-04-26 16:55:04,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:05,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:55:05,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
:Yes
5401
5363
5419
[WARNING|trainer.py:803] 2025-04-26 16:55:06,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:06,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:07,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5402
5364
5420
[WARNING|trainer.py:803] 2025-04-26 16:55:07,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:08,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:08,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5403
5365
5421
[WARNING|trainer.py:803] 2025-04-26 16:55:09,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:09,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:10,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5366
5404
5422
[WARNING|trainer.py:803] 2025-04-26 16:55:10,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:10,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5367
[WARNING|trainer.py:803] 2025-04-26 16:55:11,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5405
5423
[WARNING|trainer.py:803] 2025-04-26 16:55:12,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:12,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5368
[WARNING|trainer.py:803] 2025-04-26 16:55:13,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5406
[WARNING|trainer.py:803] 2025-04-26 16:55:13,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5424
[WARNING|trainer.py:803] 2025-04-26 16:55:13,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5369
[WARNING|trainer.py:803] 2025-04-26 16:55:14,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5407
[WARNING|trainer.py:803] 2025-04-26 16:55:15,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5425
[WARNING|trainer.py:803] 2025-04-26 16:55:15,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5370
[WARNING|trainer.py:803] 2025-04-26 16:55:16,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5408
[WARNING|trainer.py:803] 2025-04-26 16:55:16,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5426
5371
[WARNING|trainer.py:803] 2025-04-26 16:55:16,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:17,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5409
[WARNING|trainer.py:803] 2025-04-26 16:55:17,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5427
5372
[WARNING|trainer.py:803] 2025-04-26 16:55:18,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:19,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5410
[WARNING|trainer.py:803] 2025-04-26 16:55:19,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5373
5428
[WARNING|trainer.py:803] 2025-04-26 16:55:19,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5411
[WARNING|trainer.py:803] 2025-04-26 16:55:20,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:20,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5374
5429
[WARNING|trainer.py:803] 2025-04-26 16:55:21,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:21,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5412
[WARNING|trainer.py:803] 2025-04-26 16:55:22,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5375
5430
[WARNING|trainer.py:803] 2025-04-26 16:55:22,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:23,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5413
[WARNING|trainer.py:803] 2025-04-26 16:55:23,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5376
[WARNING|trainer.py:803] 2025-04-26 16:55:24,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5431
[WARNING|trainer.py:803] 2025-04-26 16:55:24,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5414
[WARNING|trainer.py:803] 2025-04-26 16:55:25,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5377
[WARNING|trainer.py:803] 2025-04-26 16:55:25,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
5432:Yes

[WARNING|trainer.py:803] 2025-04-26 16:55:26,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5415
5378
[WARNING|trainer.py:803] 2025-04-26 16:55:26,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5433
[WARNING|trainer.py:803] 2025-04-26 16:55:27,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:27,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5379
5416
[WARNING|trainer.py:803] 2025-04-26 16:55:28,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5434
[WARNING|trainer.py:803] 2025-04-26 16:55:28,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:28,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5380
5417
[WARNING|trainer.py:803] 2025-04-26 16:55:29,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:30,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5435
[WARNING|trainer.py:803] 2025-04-26 16:55:30,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5381
5418
[WARNING|trainer.py:803] 2025-04-26 16:55:31,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:31,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5436
[WARNING|trainer.py:803] 2025-04-26 16:55:31,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5382
5419
[WARNING|trainer.py:803] 2025-04-26 16:55:32,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:32,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5437
[WARNING|trainer.py:803] 2025-04-26 16:55:33,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5383
5420
[WARNING|trainer.py:803] 2025-04-26 16:55:34,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:34,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:34,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5438
5384
5421
[WARNING|trainer.py:803] 2025-04-26 16:55:35,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:35,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5385
[WARNING|trainer.py:803] 2025-04-26 16:55:36,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5439
5422
[WARNING|trainer.py:803] 2025-04-26 16:55:37,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:37,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5386
[WARNING|trainer.py:803] 2025-04-26 16:55:37,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5440
[WARNING|trainer.py:803] 2025-04-26 16:55:38,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5423
[WARNING|trainer.py:803] 2025-04-26 16:55:38,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5387
5441
[WARNING|trainer.py:803] 2025-04-26 16:55:39,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:39,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5424
[WARNING|trainer.py:803] 2025-04-26 16:55:40,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5388
5442
[WARNING|trainer.py:803] 2025-04-26 16:55:40,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:41,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5425
[WARNING|trainer.py:803] 2025-04-26 16:55:41,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5389
[WARNING|trainer.py:803] 2025-04-26 16:55:42,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5443
[WARNING|trainer.py:803] 2025-04-26 16:55:42,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5426
5390
[WARNING|trainer.py:803] 2025-04-26 16:55:43,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:43,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5444
[WARNING|trainer.py:803] 2025-04-26 16:55:43,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5391
5427
[WARNING|trainer.py:803] 2025-04-26 16:55:44,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:45,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:45,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5445
5392
5428
[WARNING|trainer.py:803] 2025-04-26 16:55:46,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:46,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5446
[WARNING|trainer.py:803] 2025-04-26 16:55:46,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5393
5429
[WARNING|trainer.py:803] 2025-04-26 16:55:47,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:48,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5447
[WARNING|trainer.py:803] 2025-04-26 16:55:48,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5394
5430
[WARNING|trainer.py:803] 2025-04-26 16:55:49,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:49,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5448
[WARNING|trainer.py:803] 2025-04-26 16:55:50,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5395
5431
[WARNING|trainer.py:803] 2025-04-26 16:55:50,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:50,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5396
5449
[WARNING|trainer.py:803] 2025-04-26 16:55:51,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5432
[WARNING|trainer.py:803] 2025-04-26 16:55:52,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:55:52,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5397
5450
[WARNING|trainer.py:803] 2025-04-26 16:55:53,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:53,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5433
[WARNING|trainer.py:803] 2025-04-26 16:55:53,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5398
[WARNING|trainer.py:803] 2025-04-26 16:55:54,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5451
[WARNING|trainer.py:803] 2025-04-26 16:55:55,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5434
[WARNING|trainer.py:803] 2025-04-26 16:55:55,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5399
5452
[WARNING|trainer.py:803] 2025-04-26 16:55:56,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:55:56,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5435
[WARNING|trainer.py:803] 2025-04-26 16:55:56,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5400
5453
[WARNING|trainer.py:803] 2025-04-26 16:55:57,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:55:57,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5436
[WARNING|trainer.py:803] 2025-04-26 16:55:58,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5401
5454
[WARNING|trainer.py:803] 2025-04-26 16:55:59,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:55:59,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5437
[WARNING|trainer.py:803] 2025-04-26 16:55:59,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5402
5455
[WARNING|trainer.py:803] 2025-04-26 16:56:00,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:00,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5438
[WARNING|trainer.py:803] 2025-04-26 16:56:01,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5403
5456
[WARNING|trainer.py:803] 2025-04-26 16:56:02,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:02,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5439
[WARNING|trainer.py:803] 2025-04-26 16:56:02,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5404
5457
[WARNING|trainer.py:803] 2025-04-26 16:56:03,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:03,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:04,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5440
5405
5458
[WARNING|trainer.py:803] 2025-04-26 16:56:05,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:05,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:05,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5441
5406
5459
[WARNING|trainer.py:803] 2025-04-26 16:56:06,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:07,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:07,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5442
5407
5460
[WARNING|trainer.py:803] 2025-04-26 16:56:08,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:08,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:08,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5443
5408
5461
[WARNING|trainer.py:803] 2025-04-26 16:56:09,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:10,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:10,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5444
5409
5462
[WARNING|trainer.py:803] 2025-04-26 16:56:11,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:11,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:11,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5445
5410
5463
[WARNING|trainer.py:803] 2025-04-26 16:56:12,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:13,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:13,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5446
5464
5411
[WARNING|trainer.py:803] 2025-04-26 16:56:14,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:14,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:14,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5447
5412
5465
[WARNING|trainer.py:803] 2025-04-26 16:56:16,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:16,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:16,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5448
5466
5413
[WARNING|trainer.py:803] 2025-04-26 16:56:17,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:17,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:17,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5467
5449
5414
[WARNING|trainer.py:803] 2025-04-26 16:56:19,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:19,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:19,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5450
5468
5415
[WARNING|trainer.py:803] 2025-04-26 16:56:20,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:20,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:20,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5469
5451
5416
[WARNING|trainer.py:803] 2025-04-26 16:56:22,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:22,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:22,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5470
5452
5417
[WARNING|trainer.py:803] 2025-04-26 16:56:23,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:23,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:23,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5471
5453
5418
[WARNING|trainer.py:803] 2025-04-26 16:56:25,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:25,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:25,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5472
5454
5419
[WARNING|trainer.py:803] 2025-04-26 16:56:26,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:26,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:26,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5473
5420
5455
[WARNING|trainer.py:803] 2025-04-26 16:56:28,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:28,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:28,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5474
5421
5456
[WARNING|trainer.py:803] 2025-04-26 16:56:29,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:29,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:29,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5457
5422
5475
[WARNING|trainer.py:803] 2025-04-26 16:56:31,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:31,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:31,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5458
5423
5476
[WARNING|trainer.py:803] 2025-04-26 16:56:32,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:32,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:32,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5459
5477
5424
[WARNING|trainer.py:803] 2025-04-26 16:56:34,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:34,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:34,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5478
5460
5425
[WARNING|trainer.py:803] 2025-04-26 16:56:35,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:35,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:36,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5461
5479
5426
[WARNING|trainer.py:803] 2025-04-26 16:56:37,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:37,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:37,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5480
5462
5427
[WARNING|trainer.py:803] 2025-04-26 16:56:38,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:38,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:39,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5481
5463
5428
[WARNING|trainer.py:803] 2025-04-26 16:56:40,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:40,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:40,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5482
5464
5429
[WARNING|trainer.py:803] 2025-04-26 16:56:41,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:41,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:42,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5483
5465
5430
[WARNING|trainer.py:803] 2025-04-26 16:56:43,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:43,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:43,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5466
5484
5431
[WARNING|trainer.py:803] 2025-04-26 16:56:44,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:44,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:45,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5467
5485
5432
[WARNING|trainer.py:803] 2025-04-26 16:56:46,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:46,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:46,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5468
5486
5433
[WARNING|trainer.py:803] 2025-04-26 16:56:47,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:48,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:48,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5469
5487
5434
[WARNING|trainer.py:803] 2025-04-26 16:56:49,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:49,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:49,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5488
5470
5435
[WARNING|trainer.py:803] 2025-04-26 16:56:51,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:51,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:51,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5489
5471
5436
[WARNING|trainer.py:803] 2025-04-26 16:56:52,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:52,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:52,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5490
5472
5437
[WARNING|trainer.py:803] 2025-04-26 16:56:53,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:54,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 16:56:54,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
NoYes
5491
5473
5438
[WARNING|trainer.py:803] 2025-04-26 16:56:55,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:56:55,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:55,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5492
5474
5439
[WARNING|trainer.py:803] 2025-04-26 16:56:57,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:57,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:56:57,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5493
5440
5475
[WARNING|trainer.py:803] 2025-04-26 16:56:58,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:56:58,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:56:58,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5494
5441
5476
[WARNING|trainer.py:803] 2025-04-26 16:56:59,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:00,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:00,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5495
5477
5442
[WARNING|trainer.py:803] 2025-04-26 16:57:01,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:01,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:01,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5496
5478
5443
[WARNING|trainer.py:803] 2025-04-26 16:57:02,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:03,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:03,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5497
5479
5444
[WARNING|trainer.py:803] 2025-04-26 16:57:04,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:04,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:04,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5498
5480
5445
[WARNING|trainer.py:803] 2025-04-26 16:57:06,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:06,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:06,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5499
5481
5446
[WARNING|trainer.py:803] 2025-04-26 16:57:07,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:07,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:08,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5482
5500
5447
[WARNING|trainer.py:803] 2025-04-26 16:57:09,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:09,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:09,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5501
5483
5448
[WARNING|trainer.py:803] 2025-04-26 16:57:10,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:10,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:11,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5502
5484
5449
[WARNING|trainer.py:803] 2025-04-26 16:57:12,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:12,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:12,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5503
5485
5450
[WARNING|trainer.py:803] 2025-04-26 16:57:13,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:13,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5504
[WARNING|trainer.py:803] 2025-04-26 16:57:14,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5486
5451
[WARNING|trainer.py:803] 2025-04-26 16:57:14,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:15,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5505
[WARNING|trainer.py:803] 2025-04-26 16:57:15,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5487
[WARNING|trainer.py:803] 2025-04-26 16:57:16,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5452
[WARNING|trainer.py:803] 2025-04-26 16:57:16,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5506
[WARNING|trainer.py:803] 2025-04-26 16:57:17,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5488
[WARNING|trainer.py:803] 2025-04-26 16:57:17,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5453
[WARNING|trainer.py:803] 2025-04-26 16:57:18,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5507
5489
[WARNING|trainer.py:803] 2025-04-26 16:57:18,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:19,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5454
[WARNING|trainer.py:803] 2025-04-26 16:57:19,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5508
5490
[WARNING|trainer.py:803] 2025-04-26 16:57:20,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:20,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5455
[WARNING|trainer.py:803] 2025-04-26 16:57:21,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5509
5491
[WARNING|trainer.py:803] 2025-04-26 16:57:21,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:22,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5456
[WARNING|trainer.py:803] 2025-04-26 16:57:22,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5510
5492
[WARNING|trainer.py:803] 2025-04-26 16:57:23,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:23,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5457
5511
[WARNING|trainer.py:803] 2025-04-26 16:57:24,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5493
[WARNING|trainer.py:803] 2025-04-26 16:57:24,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:24,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5512
5458
[WARNING|trainer.py:803] 2025-04-26 16:57:25,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5494
[WARNING|trainer.py:803] 2025-04-26 16:57:26,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:26,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5513
5459
[WARNING|trainer.py:803] 2025-04-26 16:57:27,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:27,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5495
[WARNING|trainer.py:803] 2025-04-26 16:57:27,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5514
5460
[WARNING|trainer.py:803] 2025-04-26 16:57:28,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:29,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5496
[WARNING|trainer.py:803] 2025-04-26 16:57:29,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5515
5461
[WARNING|trainer.py:803] 2025-04-26 16:57:30,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:30,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:30,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5497
5516
5462
[WARNING|trainer.py:803] 2025-04-26 16:57:31,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:32,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:32,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5498
5517
5463
[WARNING|trainer.py:803] 2025-04-26 16:57:33,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:33,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:33,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5499
5518
5464
[WARNING|trainer.py:803] 2025-04-26 16:57:34,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:35,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:35,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5500
5519
5465
[WARNING|trainer.py:803] 2025-04-26 16:57:36,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:36,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:57:36,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5501
5520
5466
[WARNING|trainer.py:803] 2025-04-26 16:57:37,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:38,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:38,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5502
5521
5467
[WARNING|trainer.py:803] 2025-04-26 16:57:39,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:39,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:39,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5503
5522
5468
[WARNING|trainer.py:803] 2025-04-26 16:57:40,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:40,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:41,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5504
5523
5469
[WARNING|trainer.py:803] 2025-04-26 16:57:42,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:42,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5505
[WARNING|trainer.py:803] 2025-04-26 16:57:42,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5524
5470
[WARNING|trainer.py:803] 2025-04-26 16:57:43,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:43,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5506
5525
[WARNING|trainer.py:803] 2025-04-26 16:57:44,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5471
[WARNING|trainer.py:803] 2025-04-26 16:57:45,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:45,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5507
[WARNING|trainer.py:803] 2025-04-26 16:57:45,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5526
[WARNING|trainer.py:803] 2025-04-26 16:57:46,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5472
[WARNING|trainer.py:803] 2025-04-26 16:57:46,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5508
5527
[WARNING|trainer.py:803] 2025-04-26 16:57:47,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:48,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5473
[WARNING|trainer.py:803] 2025-04-26 16:57:48,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5509
5528
[WARNING|trainer.py:803] 2025-04-26 16:57:48,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:49,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5474
[WARNING|trainer.py:803] 2025-04-26 16:57:49,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5510
5529
[WARNING|trainer.py:803] 2025-04-26 16:57:50,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:50,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:51,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5475
5511
5530
[WARNING|trainer.py:803] 2025-04-26 16:57:52,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:52,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:52,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5476
5512
5531
[WARNING|trainer.py:803] 2025-04-26 16:57:53,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:53,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:54,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5477
5513
5532
[WARNING|trainer.py:803] 2025-04-26 16:57:55,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:55,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:55,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5478
5514
5533
[WARNING|trainer.py:803] 2025-04-26 16:57:56,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:56,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:56,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5479
5515
5534
[WARNING|trainer.py:803] 2025-04-26 16:57:58,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:58,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:57:58,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5516
5480
5535
[WARNING|trainer.py:803] 2025-04-26 16:57:59,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:57:59,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:57:59,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5517
5481
5536
[WARNING|trainer.py:803] 2025-04-26 16:58:00,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:01,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:01,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5518
5482
5537
[WARNING|trainer.py:803] 2025-04-26 16:58:02,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:02,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:02,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5519
5483
5538
[WARNING|trainer.py:803] 2025-04-26 16:58:03,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:04,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:04,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5520
5484
[WARNING|trainer.py:803] 2025-04-26 16:58:05,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5539
[WARNING|trainer.py:803] 2025-04-26 16:58:05,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5521
[WARNING|trainer.py:803] 2025-04-26 16:58:06,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5485
[WARNING|trainer.py:803] 2025-04-26 16:58:06,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5540
[WARNING|trainer.py:803] 2025-04-26 16:58:07,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5522
5486
[WARNING|trainer.py:803] 2025-04-26 16:58:08,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:08,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:08,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5523
5541
5487
[WARNING|trainer.py:803] 2025-04-26 16:58:09,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:09,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:10,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5524
5542
5488
[WARNING|trainer.py:803] 2025-04-26 16:58:11,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:11,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:11,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5525
5489
5543
[WARNING|trainer.py:803] 2025-04-26 16:58:12,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:13,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5526
[WARNING|trainer.py:803] 2025-04-26 16:58:13,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5490
[WARNING|trainer.py:803] 2025-04-26 16:58:14,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5544
[WARNING|trainer.py:803] 2025-04-26 16:58:14,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5527
[WARNING|trainer.py:803] 2025-04-26 16:58:15,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5491
[WARNING|trainer.py:803] 2025-04-26 16:58:15,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5545
5528
[WARNING|trainer.py:803] 2025-04-26 16:58:16,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:16,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5492
[WARNING|trainer.py:803] 2025-04-26 16:58:16,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5529
[WARNING|trainer.py:803] 2025-04-26 16:58:17,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5546
5493
[WARNING|trainer.py:803] 2025-04-26 16:58:18,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:18,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5530
[WARNING|trainer.py:803] 2025-04-26 16:58:19,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5547
[WARNING|trainer.py:803] 2025-04-26 16:58:19,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5494
[WARNING|trainer.py:803] 2025-04-26 16:58:20,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
5531
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:20,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:21,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5495
5548
5532
[WARNING|trainer.py:803] 2025-04-26 16:58:22,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:22,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:22,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5496
5549
5533
[WARNING|trainer.py:803] 2025-04-26 16:58:23,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:23,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:24,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5497
5550
5534
[WARNING|trainer.py:803] 2025-04-26 16:58:25,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:25,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:25,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5498
5551
5535
[WARNING|trainer.py:803] 2025-04-26 16:58:26,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:26,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:27,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5499
5552
5536
[WARNING|trainer.py:803] 2025-04-26 16:58:28,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:28,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:28,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5500
5553
5537
[WARNING|trainer.py:803] 2025-04-26 16:58:29,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:30,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:30,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5501
5554
5538
[WARNING|trainer.py:803] 2025-04-26 16:58:31,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:31,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:31,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5502
5555
5539
[WARNING|trainer.py:803] 2025-04-26 16:58:32,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:33,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5503
[WARNING|trainer.py:803] 2025-04-26 16:58:33,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5556
[WARNING|trainer.py:803] 2025-04-26 16:58:34,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:34,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5540
5504
5557
[WARNING|trainer.py:803] 2025-04-26 16:58:35,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:35,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:36,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5541
5505
5558
[WARNING|trainer.py:803] 2025-04-26 16:58:37,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:37,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:37,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5506
5542
5559
[WARNING|trainer.py:803] 2025-04-26 16:58:38,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:38,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:39,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5507
5560
5543
[WARNING|trainer.py:803] 2025-04-26 16:58:40,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:40,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:40,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5508
5561
[WARNING|trainer.py:803] 2025-04-26 16:58:41,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5544
[WARNING|trainer.py:803] 2025-04-26 16:58:42,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5509
[WARNING|trainer.py:803] 2025-04-26 16:58:42,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5562
[WARNING|trainer.py:803] 2025-04-26 16:58:43,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5545
5510
[WARNING|trainer.py:803] 2025-04-26 16:58:43,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:44,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5563
[WARNING|trainer.py:803] 2025-04-26 16:58:44,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5511
5546
[WARNING|trainer.py:803] 2025-04-26 16:58:45,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:45,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5564
[WARNING|trainer.py:803] 2025-04-26 16:58:46,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5512
[WARNING|trainer.py:803] 2025-04-26 16:58:46,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5547
[WARNING|trainer.py:803] 2025-04-26 16:58:47,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5565
5513
[WARNING|trainer.py:803] 2025-04-26 16:58:47,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:48,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:48,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5548
5566
5514
[WARNING|trainer.py:803] 2025-04-26 16:58:49,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:49,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:50,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5549
5567
5515
[WARNING|trainer.py:803] 2025-04-26 16:58:51,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:51,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:51,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5550
5568
5516
[WARNING|trainer.py:803] 2025-04-26 16:58:52,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:53,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:53,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5551
5517
5569
[WARNING|trainer.py:803] 2025-04-26 16:58:54,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:54,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:54,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5552
5570
5518
[WARNING|trainer.py:803] 2025-04-26 16:58:55,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:56,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:58:56,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5553
5571
5519
[WARNING|trainer.py:803] 2025-04-26 16:58:57,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:57,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:58:57,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5554
5572
5520
[WARNING|trainer.py:803] 2025-04-26 16:58:58,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:58:59,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:58:59,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5555
5573
5521
[WARNING|trainer.py:803] 2025-04-26 16:59:00,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:00,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:00,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5556
5574
5522
[WARNING|trainer.py:803] 2025-04-26 16:59:01,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:02,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:02,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5557
5575
5523
[WARNING|trainer.py:803] 2025-04-26 16:59:03,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:03,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:03,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5576
5524
5558
[WARNING|trainer.py:803] 2025-04-26 16:59:05,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:05,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:05,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5525
5577
5559
[WARNING|trainer.py:803] 2025-04-26 16:59:06,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:06,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:06,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5526
5578
5560
[WARNING|trainer.py:803] 2025-04-26 16:59:07,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:08,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:08,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5527
5561
5579
[WARNING|trainer.py:803] 2025-04-26 16:59:09,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:09,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:09,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5528
5562
5580
[WARNING|trainer.py:803] 2025-04-26 16:59:10,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:11,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:11,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5529
5563
5581
[WARNING|trainer.py:803] 2025-04-26 16:59:12,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:12,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:12,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5530
5582
5564
[WARNING|trainer.py:803] 2025-04-26 16:59:13,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5531
[WARNING|trainer.py:803] 2025-04-26 16:59:14,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:14,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5583
5565
[WARNING|trainer.py:803] 2025-04-26 16:59:15,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5532
[WARNING|trainer.py:803] 2025-04-26 16:59:15,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:59:15,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5566
5584
[WARNING|trainer.py:803] 2025-04-26 16:59:16,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5533
[WARNING|trainer.py:803] 2025-04-26 16:59:17,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:17,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:18,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5567
5585
5534
[WARNING|trainer.py:803] 2025-04-26 16:59:18,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:18,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:19,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5586
5568
5535
[WARNING|trainer.py:803] 2025-04-26 16:59:20,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:20,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:20,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5587
5569
5536
[WARNING|trainer.py:803] 2025-04-26 16:59:21,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:21,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:22,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5570
5588
5537
[WARNING|trainer.py:803] 2025-04-26 16:59:23,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:23,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:59:23,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5571
5589
5538
[WARNING|trainer.py:803] 2025-04-26 16:59:24,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:59:25,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:25,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5572
5590
[WARNING|trainer.py:803] 2025-04-26 16:59:26,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5539
[WARNING|trainer.py:803] 2025-04-26 16:59:26,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5573
5591
[WARNING|trainer.py:803] 2025-04-26 16:59:27,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:27,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:27,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5540
5574
5592
[WARNING|trainer.py:803] 2025-04-26 16:59:29,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:29,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:29,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5575
5541
5593
[WARNING|trainer.py:803] 2025-04-26 16:59:30,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:30,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:31,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5576
5542
5594
[WARNING|trainer.py:803] 2025-04-26 16:59:32,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:32,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:32,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5577
5595
5543
[WARNING|trainer.py:803] 2025-04-26 16:59:33,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:34,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:34,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5578
5596
5544
[WARNING|trainer.py:803] 2025-04-26 16:59:35,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:35,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5579
[WARNING|trainer.py:803] 2025-04-26 16:59:36,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5597
[WARNING|trainer.py:803] 2025-04-26 16:59:37,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5545
[WARNING|trainer.py:803] 2025-04-26 16:59:37,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5580
[WARNING|trainer.py:803] 2025-04-26 16:59:37,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5598
[WARNING|trainer.py:803] 2025-04-26 16:59:38,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5546
[WARNING|trainer.py:803] 2025-04-26 16:59:38,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5581
5599
[WARNING|trainer.py:803] 2025-04-26 16:59:39,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:40,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:40,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5547
5582
5600
[WARNING|trainer.py:803] 2025-04-26 16:59:41,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:41,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:41,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5583
5548
5601
[WARNING|trainer.py:803] 2025-04-26 16:59:43,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:59:43,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:43,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5584
5549
5602
[WARNING|trainer.py:803] 2025-04-26 16:59:44,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:44,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:44,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5585
5603
5550
[WARNING|trainer.py:803] 2025-04-26 16:59:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:46,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:46,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5586
5604
5551
[WARNING|trainer.py:803] 2025-04-26 16:59:47,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:47,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:47,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5605
5587
5552
[WARNING|trainer.py:803] 2025-04-26 16:59:49,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:49,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:49,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5606
5588
5553
[WARNING|trainer.py:803] 2025-04-26 16:59:50,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:50,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 16:59:51,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5607
5589
5554
[WARNING|trainer.py:803] 2025-04-26 16:59:52,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:52,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:52,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5608
5590
5555
[WARNING|trainer.py:803] 2025-04-26 16:59:53,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:54,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:54,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5609
5591
5556
[WARNING|trainer.py:803] 2025-04-26 16:59:55,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:55,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 16:59:55,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5610
5592
5557
[WARNING|trainer.py:803] 2025-04-26 16:59:56,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 16:59:57,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 16:59:57,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5611
5593
5558
[WARNING|trainer.py:803] 2025-04-26 16:59:58,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 16:59:58,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5612
[WARNING|trainer.py:803] 2025-04-26 16:59:58,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5594
5559
[WARNING|trainer.py:803] 2025-04-26 16:59:59,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5613
[WARNING|trainer.py:803] 2025-04-26 17:00:00,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:00,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5595
[WARNING|trainer.py:803] 2025-04-26 17:00:00,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5560
[WARNING|trainer.py:803] 2025-04-26 17:00:01,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5614
[WARNING|trainer.py:803] 2025-04-26 17:00:01,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5596
[WARNING|trainer.py:803] 2025-04-26 17:00:02,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5561
5615
[WARNING|trainer.py:803] 2025-04-26 17:00:03,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:03,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5597
[WARNING|trainer.py:803] 2025-04-26 17:00:03,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5562
5616
[WARNING|trainer.py:803] 2025-04-26 17:00:04,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:04,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:05,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5598
5563
5617
[WARNING|trainer.py:803] 2025-04-26 17:00:06,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:06,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5599
[WARNING|trainer.py:803] 2025-04-26 17:00:06,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5564
5618
[WARNING|trainer.py:803] 2025-04-26 17:00:07,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:00:08,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5600
[WARNING|trainer.py:803] 2025-04-26 17:00:08,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5565
5619
[WARNING|trainer.py:803] 2025-04-26 17:00:09,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:09,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5601
[WARNING|trainer.py:803] 2025-04-26 17:00:09,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5566
5620
[WARNING|trainer.py:803] 2025-04-26 17:00:10,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:11,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5602
[WARNING|trainer.py:803] 2025-04-26 17:00:11,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5567
5621
[WARNING|trainer.py:803] 2025-04-26 17:00:12,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:12,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5603
[WARNING|trainer.py:803] 2025-04-26 17:00:12,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5568
5622
[WARNING|trainer.py:803] 2025-04-26 17:00:13,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:14,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5604
[WARNING|trainer.py:803] 2025-04-26 17:00:14,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5569
5623
[WARNING|trainer.py:803] 2025-04-26 17:00:15,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5605
[WARNING|trainer.py:803] 2025-04-26 17:00:15,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:00:15,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5570
5624
[WARNING|trainer.py:803] 2025-04-26 17:00:16,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5606
[WARNING|trainer.py:803] 2025-04-26 17:00:17,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:17,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5571
5625
[WARNING|trainer.py:803] 2025-04-26 17:00:18,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5607
[WARNING|trainer.py:803] 2025-04-26 17:00:18,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:18,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5572
5626
[WARNING|trainer.py:803] 2025-04-26 17:00:19,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5608
[WARNING|trainer.py:803] 2025-04-26 17:00:20,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:20,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5573
5627
[WARNING|trainer.py:803] 2025-04-26 17:00:21,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5609
[WARNING|trainer.py:803] 2025-04-26 17:00:21,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:21,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5574
5628
[WARNING|trainer.py:803] 2025-04-26 17:00:22,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:23,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5610
[WARNING|trainer.py:803] 2025-04-26 17:00:23,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5575
5629
[WARNING|trainer.py:803] 2025-04-26 17:00:24,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:00:24,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5611
[WARNING|trainer.py:803] 2025-04-26 17:00:24,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5630
5576
[WARNING|trainer.py:803] 2025-04-26 17:00:25,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5612
[WARNING|trainer.py:803] 2025-04-26 17:00:26,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:26,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5631
5577
[WARNING|trainer.py:803] 2025-04-26 17:00:27,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5613
[WARNING|trainer.py:803] 2025-04-26 17:00:27,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:27,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5632
[WARNING|trainer.py:803] 2025-04-26 17:00:28,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5578
5614
[WARNING|trainer.py:803] 2025-04-26 17:00:29,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:29,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5633
[WARNING|trainer.py:803] 2025-04-26 17:00:29,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5579
5615
[WARNING|trainer.py:803] 2025-04-26 17:00:30,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:30,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5634
[WARNING|trainer.py:803] 2025-04-26 17:00:31,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5580
5616
[WARNING|trainer.py:803] 2025-04-26 17:00:32,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:32,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5635
[WARNING|trainer.py:803] 2025-04-26 17:00:32,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5581
[WARNING|trainer.py:803] 2025-04-26 17:00:33,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5617
[WARNING|trainer.py:803] 2025-04-26 17:00:34,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5636
[WARNING|trainer.py:803] 2025-04-26 17:00:34,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5582
[WARNING|trainer.py:803] 2025-04-26 17:00:35,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5618
[WARNING|trainer.py:803] 2025-04-26 17:00:35,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5637
[WARNING|trainer.py:803] 2025-04-26 17:00:35,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5583
[WARNING|trainer.py:803] 2025-04-26 17:00:36,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5619
[WARNING|trainer.py:803] 2025-04-26 17:00:37,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5638
[WARNING|trainer.py:803] 2025-04-26 17:00:37,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5584
[WARNING|trainer.py:803] 2025-04-26 17:00:38,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5620
[WARNING|trainer.py:803] 2025-04-26 17:00:38,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5639
[WARNING|trainer.py:803] 2025-04-26 17:00:39,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5585
[WARNING|trainer.py:803] 2025-04-26 17:00:39,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5621
5640
[WARNING|trainer.py:803] 2025-04-26 17:00:40,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:40,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5586
[WARNING|trainer.py:803] 2025-04-26 17:00:40,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5622
5641
[WARNING|trainer.py:803] 2025-04-26 17:00:41,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:42,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:42,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5587
5623
5642
[WARNING|trainer.py:803] 2025-04-26 17:00:43,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:43,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:00:43,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5588
5624
5643
[WARNING|trainer.py:803] 2025-04-26 17:00:44,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:44,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:45,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5625
5589
5644
[WARNING|trainer.py:803] 2025-04-26 17:00:46,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:46,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:46,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5626
5590
5645
[WARNING|trainer.py:803] 2025-04-26 17:00:47,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:48,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:48,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5627
5591
5646
[WARNING|trainer.py:803] 2025-04-26 17:00:49,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:49,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:00:49,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5628
5592
5647
[WARNING|trainer.py:803] 2025-04-26 17:00:50,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:51,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:51,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5629
5593
5648
[WARNING|trainer.py:803] 2025-04-26 17:00:52,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:52,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:00:52,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5630
5649
5594
[WARNING|trainer.py:803] 2025-04-26 17:00:53,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:00:54,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:54,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5631
5650
5595
[WARNING|trainer.py:803] 2025-04-26 17:00:55,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:55,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:00:55,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5632
5651
5596
[WARNING|trainer.py:803] 2025-04-26 17:00:56,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:00:57,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5633
[WARNING|trainer.py:803] 2025-04-26 17:00:57,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5652
[WARNING|trainer.py:803] 2025-04-26 17:00:58,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5597
[WARNING|trainer.py:803] 2025-04-26 17:00:58,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5634
[WARNING|trainer.py:803] 2025-04-26 17:00:58,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5653
5598
[WARNING|trainer.py:803] 2025-04-26 17:00:59,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:00,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5635
[WARNING|trainer.py:803] 2025-04-26 17:01:00,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5654
5599
[WARNING|trainer.py:803] 2025-04-26 17:01:01,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:01,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5636
[WARNING|trainer.py:803] 2025-04-26 17:01:01,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5655
5600
[WARNING|trainer.py:803] 2025-04-26 17:01:02,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:02,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:03,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5637
5656
5601
[WARNING|trainer.py:803] 2025-04-26 17:01:04,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:04,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5638
[WARNING|trainer.py:803] 2025-04-26 17:01:04,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5657
5602
[WARNING|trainer.py:803] 2025-04-26 17:01:05,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:05,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5639
[WARNING|trainer.py:803] 2025-04-26 17:01:06,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5658
5603
[WARNING|trainer.py:803] 2025-04-26 17:01:07,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:07,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5640
[WARNING|trainer.py:803] 2025-04-26 17:01:07,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5659
5604
[WARNING|trainer.py:803] 2025-04-26 17:01:08,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:08,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5641
[WARNING|trainer.py:803] 2025-04-26 17:01:09,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5660
5605
[WARNING|trainer.py:803] 2025-04-26 17:01:10,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:10,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5642
[WARNING|trainer.py:803] 2025-04-26 17:01:10,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5661
5606
[WARNING|trainer.py:803] 2025-04-26 17:01:11,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:11,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:12,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5643
5662
5607
[WARNING|trainer.py:803] 2025-04-26 17:01:13,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:13,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:13,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5663
5644
5608
[WARNING|trainer.py:803] 2025-04-26 17:01:14,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:14,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:15,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5645
5664
5609
[WARNING|trainer.py:803] 2025-04-26 17:01:16,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:16,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:16,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5665
5646
5610
[WARNING|trainer.py:803] 2025-04-26 17:01:17,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:17,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5666
[WARNING|trainer.py:803] 2025-04-26 17:01:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5647
5611
[WARNING|trainer.py:803] 2025-04-26 17:01:19,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:19,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:19,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5667
5648
5612
[WARNING|trainer.py:803] 2025-04-26 17:01:20,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:20,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:21,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5668
5649
5613
[WARNING|trainer.py:803] 2025-04-26 17:01:21,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:22,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5669
[WARNING|trainer.py:803] 2025-04-26 17:01:22,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5650
5614
[WARNING|trainer.py:803] 2025-04-26 17:01:23,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:23,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5670
[WARNING|trainer.py:803] 2025-04-26 17:01:24,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5651
5615
[WARNING|trainer.py:803] 2025-04-26 17:01:24,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:25,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5671
[WARNING|trainer.py:803] 2025-04-26 17:01:25,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5652
5616
[WARNING|trainer.py:803] 2025-04-26 17:01:26,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:26,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5672
[WARNING|trainer.py:803] 2025-04-26 17:01:27,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5653
5617
[WARNING|trainer.py:803] 2025-04-26 17:01:27,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:28,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5673
[WARNING|trainer.py:803] 2025-04-26 17:01:28,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5654
5618
[WARNING|trainer.py:803] 2025-04-26 17:01:29,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:29,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5674
[WARNING|trainer.py:803] 2025-04-26 17:01:30,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5655
5619
[WARNING|trainer.py:803] 2025-04-26 17:01:30,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:30,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5675
[WARNING|trainer.py:803] 2025-04-26 17:01:31,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5656
5620
[WARNING|trainer.py:803] 2025-04-26 17:01:32,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:32,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5676
5657
[WARNING|trainer.py:803] 2025-04-26 17:01:33,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5621
[WARNING|trainer.py:803] 2025-04-26 17:01:33,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:33,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5677
[WARNING|trainer.py:803] 2025-04-26 17:01:34,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5658
5622
[WARNING|trainer.py:803] 2025-04-26 17:01:35,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:35,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5678
[WARNING|trainer.py:803] 2025-04-26 17:01:36,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5659
5623
[WARNING|trainer.py:803] 2025-04-26 17:01:36,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:36,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5679
[WARNING|trainer.py:803] 2025-04-26 17:01:37,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5660
5624
[WARNING|trainer.py:803] 2025-04-26 17:01:38,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:38,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5680
[WARNING|trainer.py:803] 2025-04-26 17:01:38,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5661
5625
[WARNING|trainer.py:803] 2025-04-26 17:01:39,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:39,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5681
[WARNING|trainer.py:803] 2025-04-26 17:01:40,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5662
5626
[WARNING|trainer.py:803] 2025-04-26 17:01:41,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:41,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5682
[WARNING|trainer.py:803] 2025-04-26 17:01:41,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5663
5627
[WARNING|trainer.py:803] 2025-04-26 17:01:42,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:42,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5683
[WARNING|trainer.py:803] 2025-04-26 17:01:43,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5664
5628
[WARNING|trainer.py:803] 2025-04-26 17:01:44,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:44,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5684
[WARNING|trainer.py:803] 2025-04-26 17:01:44,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5665
5629
[WARNING|trainer.py:803] 2025-04-26 17:01:45,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:45,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5685
[WARNING|trainer.py:803] 2025-04-26 17:01:46,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5666
5630
[WARNING|trainer.py:803] 2025-04-26 17:01:47,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:47,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:47,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5686
5667
5631
[WARNING|trainer.py:803] 2025-04-26 17:01:48,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:48,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:49,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5668
5687
5632
[WARNING|trainer.py:803] 2025-04-26 17:01:50,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:50,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5688
5669
[WARNING|trainer.py:803] 2025-04-26 17:01:50,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5633
[WARNING|trainer.py:803] 2025-04-26 17:01:51,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:51,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5689
5670
[WARNING|trainer.py:803] 2025-04-26 17:01:52,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5634
[WARNING|trainer.py:803] 2025-04-26 17:01:53,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:01:53,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:53,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
5671
NoYes
5690
5635
[WARNING|trainer.py:803] 2025-04-26 17:01:54,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:01:54,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5672
5691
[WARNING|trainer.py:803] 2025-04-26 17:01:55,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5636
[WARNING|trainer.py:803] 2025-04-26 17:01:56,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:01:56,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5692
5673
[WARNING|trainer.py:803] 2025-04-26 17:01:56,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5637
[WARNING|trainer.py:803] 2025-04-26 17:01:57,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:57,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5693
5674
[WARNING|trainer.py:803] 2025-04-26 17:01:58,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5638
[WARNING|trainer.py:803] 2025-04-26 17:01:58,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:01:59,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5694
[WARNING|trainer.py:803] 2025-04-26 17:01:59,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5675
5639
[WARNING|trainer.py:803] 2025-04-26 17:02:00,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:00,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5695
[WARNING|trainer.py:803] 2025-04-26 17:02:01,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5676
5640
[WARNING|trainer.py:803] 2025-04-26 17:02:01,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:02,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5696
[WARNING|trainer.py:803] 2025-04-26 17:02:02,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5677
[WARNING|trainer.py:803] 2025-04-26 17:02:03,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x8975f900] moov atom not found
[17:02:03] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
5641
[WARNING|trainer.py:803] 2025-04-26 17:02:03,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5697
[WARNING|trainer.py:803] 2025-04-26 17:02:04,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5678
[WARNING|trainer.py:803] 2025-04-26 17:02:04,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5642
[WARNING|trainer.py:803] 2025-04-26 17:02:05,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5698
[WARNING|trainer.py:803] 2025-04-26 17:02:05,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5679
[WARNING|trainer.py:803] 2025-04-26 17:02:06,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5643
[WARNING|trainer.py:803] 2025-04-26 17:02:06,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5699
[WARNING|trainer.py:803] 2025-04-26 17:02:07,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5680
[WARNING|trainer.py:803] 2025-04-26 17:02:07,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5644
[WARNING|trainer.py:803] 2025-04-26 17:02:08,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5700
[WARNING|trainer.py:803] 2025-04-26 17:02:08,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5681
[WARNING|trainer.py:803] 2025-04-26 17:02:09,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5645
[WARNING|trainer.py:803] 2025-04-26 17:02:09,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:10,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5682
5701
5646
[WARNING|trainer.py:803] 2025-04-26 17:02:11,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:11,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:11,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5683
5647
5702
[WARNING|trainer.py:803] 2025-04-26 17:02:12,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:12,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5684
[WARNING|trainer.py:803] 2025-04-26 17:02:13,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5648
[WARNING|trainer.py:803] 2025-04-26 17:02:14,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:14,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5685
5703
5649
[WARNING|trainer.py:803] 2025-04-26 17:02:15,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:15,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:15,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5686
5650
5704
[WARNING|trainer.py:803] 2025-04-26 17:02:17,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:17,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5687
[WARNING|trainer.py:803] 2025-04-26 17:02:17,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5651
[WARNING|trainer.py:803] 2025-04-26 17:02:18,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:18,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5705
5688
5652
[WARNING|trainer.py:803] 2025-04-26 17:02:20,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:20,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:20,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5689
5653
5706
[WARNING|trainer.py:803] 2025-04-26 17:02:21,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:21,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:22,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5690
5654
[WARNING|trainer.py:803] 2025-04-26 17:02:23,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:23,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5707
5691
5655
[WARNING|trainer.py:803] 2025-04-26 17:02:24,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:24,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:24,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5656
5708
5692
[WARNING|trainer.py:803] 2025-04-26 17:02:26,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:26,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:26,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5693
5657
5709
[WARNING|trainer.py:803] 2025-04-26 17:02:27,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:27,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:28,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5694
5658
[WARNING|trainer.py:803] 2025-04-26 17:02:29,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:29,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5710
5695
5659
[WARNING|trainer.py:803] 2025-04-26 17:02:30,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:30,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:30,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5696
5660
5711
[WARNING|trainer.py:803] 2025-04-26 17:02:32,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3b93e0c0] moov atom not found
[17:02:32] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
[WARNING|trainer.py:803] 2025-04-26 17:02:32,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:32,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5697
5661
[WARNING|trainer.py:803] 2025-04-26 17:02:33,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5712
[WARNING|trainer.py:803] 2025-04-26 17:02:33,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5662
5698
[WARNING|trainer.py:803] 2025-04-26 17:02:34,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:35,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:35,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5663
5713
5699
[WARNING|trainer.py:803] 2025-04-26 17:02:36,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:36,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:36,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5664
5700
5714
[WARNING|trainer.py:803] 2025-04-26 17:02:38,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:38,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5665
[WARNING|trainer.py:803] 2025-04-26 17:02:38,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5701
[WARNING|trainer.py:803] 2025-04-26 17:02:39,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5715
5666
[WARNING|trainer.py:803] 2025-04-26 17:02:40,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:40,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:40,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5702
5667
5716
[WARNING|trainer.py:803] 2025-04-26 17:02:42,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:42,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:42,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5668
5703
[WARNING|trainer.py:803] 2025-04-26 17:02:43,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5717
5669
[WARNING|trainer.py:803] 2025-04-26 17:02:44,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:44,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:45,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5704
5670
5718
[WARNING|trainer.py:803] 2025-04-26 17:02:46,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:46,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:02:47,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5671
5705
5719
[WARNING|trainer.py:803] 2025-04-26 17:02:48,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:48,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5672
[WARNING|trainer.py:803] 2025-04-26 17:02:49,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:49,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5706
5720
5673
[WARNING|trainer.py:803] 2025-04-26 17:02:50,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:51,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:51,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5707
5674
5721
[WARNING|trainer.py:803] 2025-04-26 17:02:52,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:52,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:53,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5675
5708
[WARNING|trainer.py:803] 2025-04-26 17:02:54,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5722
[WARNING|trainer.py:803] 2025-04-26 17:02:54,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5676
[WARNING|trainer.py:803] 2025-04-26 17:02:55,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:02:55,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5709
5677
5723
[WARNING|trainer.py:803] 2025-04-26 17:02:56,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:02:57,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:02:57,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5678
5710
[WARNING|trainer.py:803] 2025-04-26 17:02:58,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5724
[WARNING|trainer.py:803] 2025-04-26 17:02:59,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5679
[WARNING|trainer.py:803] 2025-04-26 17:02:59,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5711
[WARNING|trainer.py:803] 2025-04-26 17:03:00,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5725
5680
[WARNING|trainer.py:803] 2025-04-26 17:03:01,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:01,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:01,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5712
5681
5726
[WARNING|trainer.py:803] 2025-04-26 17:03:03,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:03,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:03,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5682
5713
[WARNING|trainer.py:803] 2025-04-26 17:03:04,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5727
[WARNING|trainer.py:803] 2025-04-26 17:03:05,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5683
[WARNING|trainer.py:803] 2025-04-26 17:03:05,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:06,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5714
5684
5728
[WARNING|trainer.py:803] 2025-04-26 17:03:07,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:07,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:08,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5685
5715
5729
[WARNING|trainer.py:803] 2025-04-26 17:03:09,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:09,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:10,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5686
5716
[WARNING|trainer.py:803] 2025-04-26 17:03:11,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5730
[WARNING|trainer.py:803] 2025-04-26 17:03:11,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5687
[WARNING|trainer.py:803] 2025-04-26 17:03:12,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:12,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5717
5688
5731
[WARNING|trainer.py:803] 2025-04-26 17:03:13,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:14,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:14,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5689
5718
5732
[WARNING|trainer.py:803] 2025-04-26 17:03:15,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:15,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5690
[WARNING|trainer.py:803] 2025-04-26 17:03:16,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:17,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5719
5733
5691
[WARNING|trainer.py:803] 2025-04-26 17:03:17,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:18,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:18,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5720
5692
5734
[WARNING|trainer.py:803] 2025-04-26 17:03:19,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:19,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:20,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5693
5721
[WARNING|trainer.py:803] 2025-04-26 17:03:21,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5735
[WARNING|trainer.py:803] 2025-04-26 17:03:21,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5694
[WARNING|trainer.py:803] 2025-04-26 17:03:22,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:22,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5722
5695
5736
[WARNING|trainer.py:803] 2025-04-26 17:03:24,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:24,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:24,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5696
5723
5737
[WARNING|trainer.py:803] 2025-04-26 17:03:25,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x6612ad40] moov atom not found
[17:03:25] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
[WARNING|trainer.py:803] 2025-04-26 17:03:26,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5697
[WARNING|trainer.py:803] 2025-04-26 17:03:26,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:27,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5724
5698
5738
[WARNING|trainer.py:803] 2025-04-26 17:03:28,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:28,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:28,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5699
5725
5739
[WARNING|trainer.py:803] 2025-04-26 17:03:30,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:30,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:30,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5700
5726
[WARNING|trainer.py:803] 2025-04-26 17:03:31,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5740
[WARNING|trainer.py:803] 2025-04-26 17:03:32,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:33,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5701
[WARNING|trainer.py:803] 2025-04-26 17:03:33,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5727
5741
[WARNING|trainer.py:803] 2025-04-26 17:03:34,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:34,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5702
5728
5742
[WARNING|trainer.py:803] 2025-04-26 17:03:36,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:37,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:37,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5703
5729
5743
[WARNING|trainer.py:803] 2025-04-26 17:03:38,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:39,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:39,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5704
5730
5744
[WARNING|trainer.py:803] 2025-04-26 17:03:40,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:41,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:41,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5705
5731
5745
[WARNING|trainer.py:803] 2025-04-26 17:03:42,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:43,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:43,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5706
5732
5746
[WARNING|trainer.py:803] 2025-04-26 17:03:44,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:45,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:45,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5707
5733
[WARNING|trainer.py:803] 2025-04-26 17:03:46,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5747
[WARNING|trainer.py:803] 2025-04-26 17:03:47,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:47,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5708
5734
[WARNING|trainer.py:803] 2025-04-26 17:03:49,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5748
[WARNING|trainer.py:803] 2025-04-26 17:03:49,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:49,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5709
5735
5749
[WARNING|trainer.py:803] 2025-04-26 17:03:51,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:51,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:51,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5710
5736
5750
[WARNING|trainer.py:803] 2025-04-26 17:03:53,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:53,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:03:54,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5711
5737
5751
[WARNING|trainer.py:803] 2025-04-26 17:03:55,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:55,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:03:56,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5712
5738
5752
[WARNING|trainer.py:803] 2025-04-26 17:03:57,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:03:58,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:03:58,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5713
5739
5753
[WARNING|trainer.py:803] 2025-04-26 17:03:59,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:00,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:00,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5714
5740
5754
[WARNING|trainer.py:803] 2025-04-26 17:04:01,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:02,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:02,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5715
5741
5755
[WARNING|trainer.py:803] 2025-04-26 17:04:04,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:04,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:04,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5742
5716
5756
[WARNING|trainer.py:803] 2025-04-26 17:04:06,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:06,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:06,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5717
5757
5743
[WARNING|trainer.py:803] 2025-04-26 17:04:08,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:08,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:08,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5758
5718
5744
[WARNING|trainer.py:803] 2025-04-26 17:04:10,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:10,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:10,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5759
5745
5719
[WARNING|trainer.py:803] 2025-04-26 17:04:12,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:12,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:12,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5760
5720
5746
[WARNING|trainer.py:803] 2025-04-26 17:04:14,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:14,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:14,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5761
5721
5747
[WARNING|trainer.py:803] 2025-04-26 17:04:16,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:16,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:17,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5762
5722
5748
[WARNING|trainer.py:803] 2025-04-26 17:04:18,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:18,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:19,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5763
5723
5749
[WARNING|trainer.py:803] 2025-04-26 17:04:20,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:20,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:21,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5764
5724
5750
[WARNING|trainer.py:803] 2025-04-26 17:04:22,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:23,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:23,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5765
5725
5751
[WARNING|trainer.py:803] 2025-04-26 17:04:24,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:25,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:25,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5766
5726
5752
[WARNING|trainer.py:803] 2025-04-26 17:04:26,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:27,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:27,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5767
5727
5753
[WARNING|trainer.py:803] 2025-04-26 17:04:28,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:29,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:29,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5768
5754
[WARNING|trainer.py:803] 2025-04-26 17:04:30,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5728
[WARNING|trainer.py:803] 2025-04-26 17:04:31,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:31,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5769
[WARNING|trainer.py:803] 2025-04-26 17:04:32,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5755
5729
[WARNING|trainer.py:803] 2025-04-26 17:04:33,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:33,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5770
5756
[WARNING|trainer.py:803] 2025-04-26 17:04:34,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5730
[WARNING|trainer.py:803] 2025-04-26 17:04:35,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:35,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5771
5757
[WARNING|trainer.py:803] 2025-04-26 17:04:36,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5731
[WARNING|trainer.py:803] 2025-04-26 17:04:37,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:37,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5772
5758
5732
[WARNING|trainer.py:803] 2025-04-26 17:04:39,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:39,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:39,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5773
5759
5733
[WARNING|trainer.py:803] 2025-04-26 17:04:41,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:04:41,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:41,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5774
5760
[WARNING|trainer.py:803] 2025-04-26 17:04:43,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5734
[WARNING|trainer.py:803] 2025-04-26 17:04:43,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:43,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5775
5761
5735
[WARNING|trainer.py:803] 2025-04-26 17:04:45,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:45,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:46,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5776
5762
5736
[WARNING|trainer.py:803] 2025-04-26 17:04:47,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:47,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:48,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5777
5763
5737
[WARNING|trainer.py:803] 2025-04-26 17:04:49,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:49,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:50,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5778
5764
[WARNING|trainer.py:803] 2025-04-26 17:04:51,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5738
[WARNING|trainer.py:803] 2025-04-26 17:04:51,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:52,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5779
5765
[WARNING|trainer.py:803] 2025-04-26 17:04:53,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5739
[WARNING|trainer.py:803] 2025-04-26 17:04:53,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:04:54,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5766
5780
5740
[WARNING|trainer.py:803] 2025-04-26 17:04:55,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:55,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:56,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5781
5767
5741
[WARNING|trainer.py:803] 2025-04-26 17:04:57,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:04:58,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:04:58,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5782
5768
5742
[WARNING|trainer.py:803] 2025-04-26 17:04:59,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:00,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:00,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5769
5783
5743
[WARNING|trainer.py:803] 2025-04-26 17:05:02,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:02,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:02,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5770
5784
5744
[WARNING|trainer.py:803] 2025-04-26 17:05:04,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:04,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:04,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5771
5785
5745
[WARNING|trainer.py:803] 2025-04-26 17:05:06,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:06,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:06,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5772
5786
5746
[WARNING|trainer.py:803] 2025-04-26 17:05:08,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:08,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:09,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5773
5787
[WARNING|trainer.py:803] 2025-04-26 17:05:10,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5747
[WARNING|trainer.py:803] 2025-04-26 17:05:10,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:11,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5774
5788
[WARNING|trainer.py:803] 2025-04-26 17:05:12,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5748
[WARNING|trainer.py:803] 2025-04-26 17:05:13,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:13,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5775
5789
[WARNING|trainer.py:803] 2025-04-26 17:05:14,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5749
[WARNING|trainer.py:803] 2025-04-26 17:05:14,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:15,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5776
5790
[WARNING|trainer.py:803] 2025-04-26 17:05:16,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5750
[WARNING|trainer.py:803] 2025-04-26 17:05:16,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:17,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5777
5791
[WARNING|trainer.py:803] 2025-04-26 17:05:18,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5751
[WARNING|trainer.py:803] 2025-04-26 17:05:19,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:05:19,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5778
5792
[WARNING|trainer.py:803] 2025-04-26 17:05:20,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:21,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5752
[WARNING|trainer.py:803] 2025-04-26 17:05:22,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5779
5793
[WARNING|trainer.py:803] 2025-04-26 17:05:23,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5753
[WARNING|trainer.py:803] 2025-04-26 17:05:23,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:05:23,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5794
5780
5754
[WARNING|trainer.py:803] 2025-04-26 17:05:25,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:25,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:25,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5795
5781
5755
[WARNING|trainer.py:803] 2025-04-26 17:05:27,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:27,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:28,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5796
5782
5756
[WARNING|trainer.py:803] 2025-04-26 17:05:29,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:29,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:30,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5797
5783
5757
[WARNING|trainer.py:803] 2025-04-26 17:05:31,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:32,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:32,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5798
5758
5784
[WARNING|trainer.py:803] 2025-04-26 17:05:33,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:34,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:34,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5799
5759
5785
[WARNING|trainer.py:803] 2025-04-26 17:05:35,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:36,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:36,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5800
5760
[WARNING|trainer.py:803] 2025-04-26 17:05:37,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5786
[WARNING|trainer.py:803] 2025-04-26 17:05:38,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:38,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5801
5761
[WARNING|trainer.py:803] 2025-04-26 17:05:39,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:40,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5787
[WARNING|trainer.py:803] 2025-04-26 17:05:40,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5802
5762
[WARNING|trainer.py:803] 2025-04-26 17:05:41,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5788
[WARNING|trainer.py:803] 2025-04-26 17:05:42,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:42,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5763
5803
5789
[WARNING|trainer.py:803] 2025-04-26 17:05:44,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:44,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:44,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5764
5804
5790
[WARNING|trainer.py:803] 2025-04-26 17:05:46,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:46,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:46,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5765
5805
5791
[WARNING|trainer.py:803] 2025-04-26 17:05:48,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:48,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:48,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5766
5806
5792
[WARNING|trainer.py:803] 2025-04-26 17:05:50,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:50,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:50,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5767
5807
5793
[WARNING|trainer.py:803] 2025-04-26 17:05:52,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:52,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:52,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5768
5808
5794
[WARNING|trainer.py:803] 2025-04-26 17:05:54,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:05:54,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:05:54,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5769
5809
5795
[WARNING|trainer.py:803] 2025-04-26 17:05:56,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:56,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:56,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5770
5810
5796
[WARNING|trainer.py:803] 2025-04-26 17:05:58,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:05:58,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:05:59,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5771
5811
5797
[WARNING|trainer.py:803] 2025-04-26 17:06:00,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:00,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:01,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5812
5772
5798
[WARNING|trainer.py:803] 2025-04-26 17:06:02,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:02,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:03,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5813
5773
5799
[WARNING|trainer.py:803] 2025-04-26 17:06:04,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:04,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:05,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5774
5814
5800
[WARNING|trainer.py:803] 2025-04-26 17:06:06,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:06,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:07,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5815
5775
5801
[WARNING|trainer.py:803] 2025-04-26 17:06:08,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:09,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:09,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5816
5776
5802
[WARNING|trainer.py:803] 2025-04-26 17:06:11,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:11,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:11,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5817
5777
5803
[WARNING|trainer.py:803] 2025-04-26 17:06:13,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:13,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:13,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5778
5818
5804
[WARNING|trainer.py:803] 2025-04-26 17:06:15,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:15,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:16,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5779
5819
5805
[WARNING|trainer.py:803] 2025-04-26 17:06:17,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:17,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:18,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5820
5780
5806
[WARNING|trainer.py:803] 2025-04-26 17:06:19,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:19,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:20,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5781
5821
5807
[WARNING|trainer.py:803] 2025-04-26 17:06:21,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:22,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:22,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5782
5808
5822
[WARNING|trainer.py:803] 2025-04-26 17:06:24,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:24,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:24,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5809
5783
5823
[WARNING|trainer.py:803] 2025-04-26 17:06:26,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:26,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:26,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5810
5824
5784
[WARNING|trainer.py:803] 2025-04-26 17:06:28,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:28,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:28,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5811
5785
5825
[WARNING|trainer.py:803] 2025-04-26 17:06:30,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:30,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:30,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5812
5826
5786
[WARNING|trainer.py:803] 2025-04-26 17:06:32,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:32,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:32,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5813
5827
5787
[WARNING|trainer.py:803] 2025-04-26 17:06:34,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:35,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:35,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5814
5828
5788
[WARNING|trainer.py:803] 2025-04-26 17:06:36,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:37,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:37,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5815
5829
5789
[WARNING|trainer.py:803] 2025-04-26 17:06:38,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:39,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:39,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5816
5790
5830
[WARNING|trainer.py:803] 2025-04-26 17:06:40,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:41,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:41,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5817
5831
5791
[WARNING|trainer.py:803] 2025-04-26 17:06:43,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:43,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:43,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5818
5792
5832
[WARNING|trainer.py:803] 2025-04-26 17:06:45,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:45,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:45,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5793
5833
5819
[WARNING|trainer.py:803] 2025-04-26 17:06:47,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:47,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:47,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5794
5834
5820
[WARNING|trainer.py:803] 2025-04-26 17:06:49,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:49,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:49,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5795
5835
5821
[WARNING|trainer.py:803] 2025-04-26 17:06:51,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:51,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:52,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5796
5836
5822
[WARNING|trainer.py:803] 2025-04-26 17:06:53,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:53,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:06:54,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5797
5837
5823
[WARNING|trainer.py:803] 2025-04-26 17:06:55,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:55,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:06:56,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5838
5798
5824
[WARNING|trainer.py:803] 2025-04-26 17:06:57,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:06:57,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:06:58,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5799
5839
5825
[WARNING|trainer.py:803] 2025-04-26 17:06:59,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:00,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:00,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5800
5840
5826
[WARNING|trainer.py:803] 2025-04-26 17:07:02,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:02,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:02,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5841
5801
[WARNING|trainer.py:803] 2025-04-26 17:07:04,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5827
[WARNING|trainer.py:803] 2025-04-26 17:07:04,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:05,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5842
5802
5828
[WARNING|trainer.py:803] 2025-04-26 17:07:06,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:06,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:06,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5843
5803
5829
[WARNING|trainer.py:803] 2025-04-26 17:07:08,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:08,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:08,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5844
5804
5830
[WARNING|trainer.py:803] 2025-04-26 17:07:10,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:10,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:11,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5845
5805
5831
[WARNING|trainer.py:803] 2025-04-26 17:07:12,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:12,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:13,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5846
5806
5832
[WARNING|trainer.py:803] 2025-04-26 17:07:14,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:14,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:15,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5847
5807
5833
[WARNING|trainer.py:803] 2025-04-26 17:07:16,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:16,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:17,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5848
5808
5834
[WARNING|trainer.py:803] 2025-04-26 17:07:18,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:18,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:19,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5849
5809
5835
[WARNING|trainer.py:803] 2025-04-26 17:07:20,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:20,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:21,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5810
5850
5836
[WARNING|trainer.py:803] 2025-04-26 17:07:22,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:22,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:23,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5811
5851
5837
[WARNING|trainer.py:803] 2025-04-26 17:07:24,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:25,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:25,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5812
5852
5838
[WARNING|trainer.py:803] 2025-04-26 17:07:27,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:27,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:27,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5813
5853
5839
[WARNING|trainer.py:803] 2025-04-26 17:07:29,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:29,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:29,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5814
5854
5840
[WARNING|trainer.py:803] 2025-04-26 17:07:31,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:07:31,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:31,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5815
5855
5841
[WARNING|trainer.py:803] 2025-04-26 17:07:33,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:33,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:33,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5816
5856
5842
[WARNING|trainer.py:803] 2025-04-26 17:07:35,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:07:35,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:35,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5857
5817
5843
[WARNING|trainer.py:803] 2025-04-26 17:07:37,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:37,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:38,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5858
5818
5844
[WARNING|trainer.py:803] 2025-04-26 17:07:39,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:39,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:40,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5859
5819
5845
[WARNING|trainer.py:803] 2025-04-26 17:07:41,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:42,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:42,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5860
5846
5820
[WARNING|trainer.py:803] 2025-04-26 17:07:44,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:44,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:44,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5861
5847
5821
[WARNING|trainer.py:803] 2025-04-26 17:07:46,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:46,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:46,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5862
5848
5822
[WARNING|trainer.py:803] 2025-04-26 17:07:48,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:48,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:48,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5863
5849
5823
[WARNING|trainer.py:803] 2025-04-26 17:07:50,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:50,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:51,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5850
5864
5824
[WARNING|trainer.py:803] 2025-04-26 17:07:52,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:52,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:53,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5851
5865
5825
[WARNING|trainer.py:803] 2025-04-26 17:07:54,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:54,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:55,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5866
5852
5826
[WARNING|trainer.py:803] 2025-04-26 17:07:56,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:57,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:07:57,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5853
5867
5827
[WARNING|trainer.py:803] 2025-04-26 17:07:59,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:07:59,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:07:59,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5854
5868
5828
[WARNING|trainer.py:803] 2025-04-26 17:08:01,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:01,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:01,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5855
5869
5829
[WARNING|trainer.py:803] 2025-04-26 17:08:03,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:03,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:03,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5856
5870
5830
[WARNING|trainer.py:803] 2025-04-26 17:08:05,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:05,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:05,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5857
5871
5831
[WARNING|trainer.py:803] 2025-04-26 17:08:07,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:07,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:07,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5858
5872
5832
[WARNING|trainer.py:803] 2025-04-26 17:08:09,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:09,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:10,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5859
5873
5833
[WARNING|trainer.py:803] 2025-04-26 17:08:11,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:11,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:12,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5874
5860
5834
[WARNING|trainer.py:803] 2025-04-26 17:08:13,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:13,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:14,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5875
5861
5835
[WARNING|trainer.py:803] 2025-04-26 17:08:15,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:15,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:16,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5876
5862
5836
[WARNING|trainer.py:803] 2025-04-26 17:08:17,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:17,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:18,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5877
5863
5837
[WARNING|trainer.py:803] 2025-04-26 17:08:19,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:20,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:20,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5878
5864
5838
[WARNING|trainer.py:803] 2025-04-26 17:08:21,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:22,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:22,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5879
5865
5839
[WARNING|trainer.py:803] 2025-04-26 17:08:24,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:08:24,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:24,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5880
5866
5840
[WARNING|trainer.py:803] 2025-04-26 17:08:26,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:26,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:26,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5881
5867
5841
[WARNING|trainer.py:803] 2025-04-26 17:08:28,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:28,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:28,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5882
5868
5842
[WARNING|trainer.py:803] 2025-04-26 17:08:30,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:30,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:31,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5883
5869
5843
[WARNING|trainer.py:803] 2025-04-26 17:08:33,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:33,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:33,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5884
5844
5870
[WARNING|trainer.py:803] 2025-04-26 17:08:35,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:08:35,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:35,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5885
5845
5871
[WARNING|trainer.py:803] 2025-04-26 17:08:37,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:37,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:37,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5886
5846
5872
[WARNING|trainer.py:803] 2025-04-26 17:08:39,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:08:39,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:39,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5873
5847
5887
[WARNING|trainer.py:803] 2025-04-26 17:08:41,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:41,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:41,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5874
5848
5888
[WARNING|trainer.py:803] 2025-04-26 17:08:43,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:43,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:44,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5875
5849
5889
[WARNING|trainer.py:803] 2025-04-26 17:08:45,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:45,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:46,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5876
5850
[WARNING|trainer.py:803] 2025-04-26 17:08:47,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5890
[WARNING|trainer.py:803] 2025-04-26 17:08:47,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:48,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5877
5851
5891
[WARNING|trainer.py:803] 2025-04-26 17:08:49,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:50,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:50,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5878
5852
[WARNING|trainer.py:803] 2025-04-26 17:08:51,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5892
[WARNING|trainer.py:803] 2025-04-26 17:08:52,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:52,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5879
5853
[WARNING|trainer.py:803] 2025-04-26 17:08:53,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5893
[WARNING|trainer.py:803] 2025-04-26 17:08:54,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:08:54,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5880
5854
5894
[WARNING|trainer.py:803] 2025-04-26 17:08:56,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:56,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:56,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5881
5855
5895
[WARNING|trainer.py:803] 2025-04-26 17:08:58,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:08:58,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:08:59,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5882
5856
5896
[WARNING|trainer.py:803] 2025-04-26 17:09:00,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:00,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:01,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5883
5857
[WARNING|trainer.py:803] 2025-04-26 17:09:02,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:02,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5897
[WARNING|trainer.py:803] 2025-04-26 17:09:03,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5884
5858
[WARNING|trainer.py:803] 2025-04-26 17:09:04,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5898
[WARNING|trainer.py:803] 2025-04-26 17:09:05,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:05,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5885
5859
[WARNING|trainer.py:803] 2025-04-26 17:09:06,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5899
[WARNING|trainer.py:803] 2025-04-26 17:09:07,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:07,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5886
5860
[WARNING|trainer.py:803] 2025-04-26 17:09:08,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5900
[WARNING|trainer.py:803] 2025-04-26 17:09:09,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:09,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5887
5861
[WARNING|trainer.py:803] 2025-04-26 17:09:11,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5901
[WARNING|trainer.py:803] 2025-04-26 17:09:11,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:11,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5862
5888
5902
[WARNING|trainer.py:803] 2025-04-26 17:09:13,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:13,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:09:13,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5863
5889
5903
[WARNING|trainer.py:803] 2025-04-26 17:09:15,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:15,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:15,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5864
5904
5890
[WARNING|trainer.py:803] 2025-04-26 17:09:17,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:17,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:18,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5865
5891
5905
[WARNING|trainer.py:803] 2025-04-26 17:09:19,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:19,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:20,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5866
5906
5892
[WARNING|trainer.py:803] 2025-04-26 17:09:22,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:22,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:22,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5907
5867
5893
[WARNING|trainer.py:803] 2025-04-26 17:09:24,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:24,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:24,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5908
5868
5894
[WARNING|trainer.py:803] 2025-04-26 17:09:26,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:26,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:26,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5909
5869
5895
[WARNING|trainer.py:803] 2025-04-26 17:09:28,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:28,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:28,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5910
5870
5896
[WARNING|trainer.py:803] 2025-04-26 17:09:30,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:30,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:30,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5911
5871
[WARNING|trainer.py:803] 2025-04-26 17:09:32,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5897
[WARNING|trainer.py:803] 2025-04-26 17:09:32,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:33,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5912
5872
[WARNING|trainer.py:803] 2025-04-26 17:09:34,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5898
[WARNING|trainer.py:803] 2025-04-26 17:09:34,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:35,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5913
5873
[WARNING|trainer.py:803] 2025-04-26 17:09:36,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5899
[WARNING|trainer.py:803] 2025-04-26 17:09:36,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:37,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5914
5874
[WARNING|trainer.py:803] 2025-04-26 17:09:38,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5900
[WARNING|trainer.py:803] 2025-04-26 17:09:38,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5915
[WARNING|trainer.py:803] 2025-04-26 17:09:39,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:09:40,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5875
5901
[WARNING|trainer.py:803] 2025-04-26 17:09:40,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5916
[WARNING|trainer.py:803] 2025-04-26 17:09:41,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:42,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5876
5902
[WARNING|trainer.py:803] 2025-04-26 17:09:43,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5917
[WARNING|trainer.py:803] 2025-04-26 17:09:43,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:44,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5877
5903
[WARNING|trainer.py:803] 2025-04-26 17:09:45,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5918
[WARNING|trainer.py:803] 2025-04-26 17:09:45,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:46,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5878
5904
[WARNING|trainer.py:803] 2025-04-26 17:09:47,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5919
[WARNING|trainer.py:803] 2025-04-26 17:09:47,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:48,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5879
5905
5920
[WARNING|trainer.py:803] 2025-04-26 17:09:49,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:09:49,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:50,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5906
5880
5921
[WARNING|trainer.py:803] 2025-04-26 17:09:51,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:51,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:52,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5907
5881
5922
[WARNING|trainer.py:803] 2025-04-26 17:09:53,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:54,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:09:54,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5908
5882
5923
[WARNING|trainer.py:803] 2025-04-26 17:09:55,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:56,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:56,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5909
5883
5924
[WARNING|trainer.py:803] 2025-04-26 17:09:57,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:09:58,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:09:58,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5910
5925
[WARNING|trainer.py:803] 2025-04-26 17:09:59,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5884
[WARNING|trainer.py:803] 2025-04-26 17:10:00,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:00,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5911
5926
[WARNING|trainer.py:803] 2025-04-26 17:10:01,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5885
[WARNING|trainer.py:803] 2025-04-26 17:10:02,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:02,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5912
5927
[WARNING|trainer.py:803] 2025-04-26 17:10:03,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5886
[WARNING|trainer.py:803] 2025-04-26 17:10:04,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:04,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5913
5928
[WARNING|trainer.py:803] 2025-04-26 17:10:05,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5887
[WARNING|trainer.py:803] 2025-04-26 17:10:06,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5914
[WARNING|trainer.py:803] 2025-04-26 17:10:07,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5929
[WARNING|trainer.py:803] 2025-04-26 17:10:07,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:08,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5888
5915
5930
[WARNING|trainer.py:803] 2025-04-26 17:10:09,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:09,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:10:10,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5916
5889
5931
[WARNING|trainer.py:803] 2025-04-26 17:10:11,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:11,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:12,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5917
5890
5932
[WARNING|trainer.py:803] 2025-04-26 17:10:13,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:10:14,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:14,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5918
5933
5891
[WARNING|trainer.py:803] 2025-04-26 17:10:15,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:16,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:16,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5919
5934
5892
[WARNING|trainer.py:803] 2025-04-26 17:10:17,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:18,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:18,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5920
5935
[WARNING|trainer.py:803] 2025-04-26 17:10:20,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5893
[WARNING|trainer.py:803] 2025-04-26 17:10:20,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:20,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5921
5936
[WARNING|trainer.py:803] 2025-04-26 17:10:21,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:22,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5894
[WARNING|trainer.py:803] 2025-04-26 17:10:23,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5922
5937
[WARNING|trainer.py:803] 2025-04-26 17:10:24,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:24,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5895
[WARNING|trainer.py:803] 2025-04-26 17:10:25,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5938
5923
[WARNING|trainer.py:803] 2025-04-26 17:10:26,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:26,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5896
5939
[WARNING|trainer.py:803] 2025-04-26 17:10:27,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5924
[WARNING|trainer.py:803] 2025-04-26 17:10:28,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:28,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5897
5940
5925
[WARNING|trainer.py:803] 2025-04-26 17:10:29,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:30,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:30,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5898
5941
5926
[WARNING|trainer.py:803] 2025-04-26 17:10:31,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:32,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:32,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5899
5942
5927
[WARNING|trainer.py:803] 2025-04-26 17:10:33,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:34,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:34,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5900
5943
5928
[WARNING|trainer.py:803] 2025-04-26 17:10:36,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:36,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:36,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5944
5901
5929
[WARNING|trainer.py:803] 2025-04-26 17:10:38,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:38,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:38,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5945
5930
5902
[WARNING|trainer.py:803] 2025-04-26 17:10:39,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:40,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:40,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5946
5931
5903
[WARNING|trainer.py:803] 2025-04-26 17:10:42,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:42,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:42,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5947
5932
5904
[WARNING|trainer.py:803] 2025-04-26 17:10:43,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:44,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:44,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5948
5933
5905
[WARNING|trainer.py:803] 2025-04-26 17:10:45,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:46,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:46,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5934
5949
5906
[WARNING|trainer.py:803] 2025-04-26 17:10:48,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:48,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:48,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5935
5907
5950
[WARNING|trainer.py:803] 2025-04-26 17:10:49,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:50,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 17:10:50,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
YesYes
5936
5951
5908
[WARNING|trainer.py:803] 2025-04-26 17:10:51,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:52,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:52,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5937
5952
5909
[WARNING|trainer.py:803] 2025-04-26 17:10:54,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:10:54,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:54,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5938
5953
5910
[WARNING|trainer.py:803] 2025-04-26 17:10:56,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:10:56,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:56,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5939
5954
5911
[WARNING|trainer.py:803] 2025-04-26 17:10:57,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:10:58,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:10:58,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5940
5955
5912
[WARNING|trainer.py:803] 2025-04-26 17:11:00,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:00,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:00,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5956
5941
5913
[WARNING|trainer.py:803] 2025-04-26 17:11:02,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:02,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:02,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5957
5942
5914
[WARNING|trainer.py:803] 2025-04-26 17:11:04,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:04,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:04,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5958
5943
5915
[WARNING|trainer.py:803] 2025-04-26 17:11:06,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:06,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:06,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5944
5959
5916
[WARNING|trainer.py:803] 2025-04-26 17:11:08,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:08,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:08,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5945
5960
5917
[WARNING|trainer.py:803] 2025-04-26 17:11:10,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:10,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:11:10,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5961
5946
5918
[WARNING|trainer.py:803] 2025-04-26 17:11:12,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:12,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:12,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5947
5962
5919
[WARNING|trainer.py:803] 2025-04-26 17:11:14,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:14,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:14,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5948
5963
5920
[WARNING|trainer.py:803] 2025-04-26 17:11:16,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:16,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:16,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5949
5964
5921
[WARNING|trainer.py:803] 2025-04-26 17:11:18,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:18,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:18,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5965
5950
5922
[WARNING|trainer.py:803] 2025-04-26 17:11:20,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:20,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:20,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5966
5951
5923
[WARNING|trainer.py:803] 2025-04-26 17:11:22,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:22,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:22,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5967
5952
5924
[WARNING|trainer.py:803] 2025-04-26 17:11:23,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:24,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:24,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5968
5953
5925
[WARNING|trainer.py:803] 2025-04-26 17:11:25,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:26,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:26,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5954
5969
5926
[WARNING|trainer.py:803] 2025-04-26 17:11:28,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:28,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:28,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5955
5970
5927
[WARNING|trainer.py:803] 2025-04-26 17:11:30,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:30,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:30,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5956
5971
5928
[WARNING|trainer.py:803] 2025-04-26 17:11:32,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:32,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:32,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5957
5972
5929
[WARNING|trainer.py:803] 2025-04-26 17:11:34,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:34,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:34,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5958
5973
5930
[WARNING|trainer.py:803] 2025-04-26 17:11:36,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:36,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:36,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5959
5974
5931
[WARNING|trainer.py:803] 2025-04-26 17:11:38,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:38,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:38,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5960
5932
5975
[WARNING|trainer.py:803] 2025-04-26 17:11:40,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:11:40,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:40,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5961
5933
5976
[WARNING|trainer.py:803] 2025-04-26 17:11:42,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:42,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:42,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5962
5934
5977
[WARNING|trainer.py:803] 2025-04-26 17:11:44,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:44,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:44,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5963
5935
5978
[WARNING|trainer.py:803] 2025-04-26 17:11:46,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:46,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:46,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5964
5979
5936
[WARNING|trainer.py:803] 2025-04-26 17:11:48,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:48,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:48,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5965
5937
5980
[WARNING|trainer.py:803] 2025-04-26 17:11:50,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:50,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:50,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5966
5938
5981
[WARNING|trainer.py:803] 2025-04-26 17:11:52,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:11:52,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:52,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5967
5939
5982
[WARNING|trainer.py:803] 2025-04-26 17:11:54,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:54,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:54,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5968
5940
5983
[WARNING|trainer.py:803] 2025-04-26 17:11:56,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:56,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:11:56,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5969
5941
5984
[WARNING|trainer.py:803] 2025-04-26 17:11:58,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:11:58,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:11:58,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5970
5942
5985
[WARNING|trainer.py:803] 2025-04-26 17:12:00,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:00,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:00,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5971
5943
[WARNING|trainer.py:803] 2025-04-26 17:12:02,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:12:02,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5986
[WARNING|trainer.py:803] 2025-04-26 17:12:03,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5944
5972
[WARNING|trainer.py:803] 2025-04-26 17:12:04,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:04,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5987
[WARNING|trainer.py:803] 2025-04-26 17:12:05,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5945
5973
[WARNING|trainer.py:803] 2025-04-26 17:12:06,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:06,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5988
[WARNING|trainer.py:803] 2025-04-26 17:12:07,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5946
5974
[WARNING|trainer.py:803] 2025-04-26 17:12:08,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:08,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5989
[WARNING|trainer.py:803] 2025-04-26 17:12:09,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5947
5975
[WARNING|trainer.py:803] 2025-04-26 17:12:10,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:10,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5990
5948
[WARNING|trainer.py:803] 2025-04-26 17:12:11,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5976
[WARNING|trainer.py:803] 2025-04-26 17:12:12,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:12:12,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5991
5949
[WARNING|trainer.py:803] 2025-04-26 17:12:13,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5977
[WARNING|trainer.py:803] 2025-04-26 17:12:14,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:14,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5992
5950
[WARNING|trainer.py:803] 2025-04-26 17:12:15,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5978
[WARNING|trainer.py:803] 2025-04-26 17:12:16,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:16,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5993
5951
5979
[WARNING|trainer.py:803] 2025-04-26 17:12:17,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:18,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:18,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5994
5952
5980
[WARNING|trainer.py:803] 2025-04-26 17:12:19,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:20,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:20,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5995
5953
[WARNING|trainer.py:803] 2025-04-26 17:12:21,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5981
[WARNING|trainer.py:803] 2025-04-26 17:12:22,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:22,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5996
5954
[WARNING|trainer.py:803] 2025-04-26 17:12:23,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5982
[WARNING|trainer.py:803] 2025-04-26 17:12:24,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:24,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5997
5955
[WARNING|trainer.py:803] 2025-04-26 17:12:25,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5983
[WARNING|trainer.py:803] 2025-04-26 17:12:26,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:26,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5998
5956
[WARNING|trainer.py:803] 2025-04-26 17:12:27,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5984
[WARNING|trainer.py:803] 2025-04-26 17:12:28,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:28,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5999
5957
[WARNING|trainer.py:803] 2025-04-26 17:12:29,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5985
[WARNING|trainer.py:803] 2025-04-26 17:12:30,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6000
[WARNING|trainer.py:803] 2025-04-26 17:12:31,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5958
[WARNING|trainer.py:803] 2025-04-26 17:12:31,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5986
[WARNING|trainer.py:803] 2025-04-26 17:12:32,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6001
[WARNING|trainer.py:803] 2025-04-26 17:12:33,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5959
[WARNING|trainer.py:803] 2025-04-26 17:12:33,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:34,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5987
6002
[WARNING|trainer.py:803] 2025-04-26 17:12:35,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5960
[WARNING|trainer.py:803] 2025-04-26 17:12:35,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:12:36,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5988
6003
[WARNING|trainer.py:803] 2025-04-26 17:12:37,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5961
[WARNING|trainer.py:803] 2025-04-26 17:12:38,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:38,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5989
6004
[WARNING|trainer.py:803] 2025-04-26 17:12:39,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5962
[WARNING|trainer.py:803] 2025-04-26 17:12:40,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:40,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5990
6005
5963
[WARNING|trainer.py:803] 2025-04-26 17:12:41,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:42,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:42,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5991
6006
5964
[WARNING|trainer.py:803] 2025-04-26 17:12:43,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:44,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:44,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5992
6007
5965
[WARNING|trainer.py:803] 2025-04-26 17:12:45,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:45,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:46,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5993
6008
5966
[WARNING|trainer.py:803] 2025-04-26 17:12:47,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:47,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:48,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5994
6009
5967
[WARNING|trainer.py:803] 2025-04-26 17:12:50,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:50,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:50,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6010
5995
5968
[WARNING|trainer.py:803] 2025-04-26 17:12:51,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:52,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:52,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6011
5996
5969
[WARNING|trainer.py:803] 2025-04-26 17:12:53,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:54,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:54,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6012
5997
5970
[WARNING|trainer.py:803] 2025-04-26 17:12:55,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:56,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:12:56,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6013
5998
5971
[WARNING|trainer.py:803] 2025-04-26 17:12:57,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:12:58,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:12:58,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6014
5999
5972
[WARNING|trainer.py:803] 2025-04-26 17:13:00,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:00,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:00,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6015
6000
5973
[WARNING|trainer.py:803] 2025-04-26 17:13:02,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:02,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:02,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6016
6001
5974
[WARNING|trainer.py:803] 2025-04-26 17:13:04,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:04,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:04,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6017
6002
[WARNING|trainer.py:803] 2025-04-26 17:13:05,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5975
[WARNING|trainer.py:803] 2025-04-26 17:13:06,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:06,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6018
6003
[WARNING|trainer.py:803] 2025-04-26 17:13:07,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5976
[WARNING|trainer.py:803] 2025-04-26 17:13:08,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:08,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6019
6004
[WARNING|trainer.py:803] 2025-04-26 17:13:09,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5977
[WARNING|trainer.py:803] 2025-04-26 17:13:10,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:10,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6020
6005
[WARNING|trainer.py:803] 2025-04-26 17:13:11,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5978
[WARNING|trainer.py:803] 2025-04-26 17:13:12,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:12,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6021
6006
[WARNING|trainer.py:803] 2025-04-26 17:13:13,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5979
[WARNING|trainer.py:803] 2025-04-26 17:13:14,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:14,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6022
6007
[WARNING|trainer.py:803] 2025-04-26 17:13:15,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5980
[WARNING|trainer.py:803] 2025-04-26 17:13:16,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:16,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6023
6008
[WARNING|trainer.py:803] 2025-04-26 17:13:17,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5981
[WARNING|trainer.py:803] 2025-04-26 17:13:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:18,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6024
6009
[WARNING|trainer.py:803] 2025-04-26 17:13:20,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5982
[WARNING|trainer.py:803] 2025-04-26 17:13:20,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:21,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6025
6010
[WARNING|trainer.py:803] 2025-04-26 17:13:22,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:22,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5983
[WARNING|trainer.py:803] 2025-04-26 17:13:23,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6011
6026
[WARNING|trainer.py:803] 2025-04-26 17:13:23,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:24,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5984
6027
[WARNING|trainer.py:803] 2025-04-26 17:13:25,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6012
[WARNING|trainer.py:803] 2025-04-26 17:13:25,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:26,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5985
6013
6028
[WARNING|trainer.py:803] 2025-04-26 17:13:27,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:28,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:28,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5986
6014
6029
[WARNING|trainer.py:803] 2025-04-26 17:13:29,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:30,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:30,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5987
6015
[WARNING|trainer.py:803] 2025-04-26 17:13:31,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6030
[WARNING|trainer.py:803] 2025-04-26 17:13:32,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:32,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5988
6016
[WARNING|trainer.py:803] 2025-04-26 17:13:33,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6031
[WARNING|trainer.py:803] 2025-04-26 17:13:34,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:34,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5989
6017
[WARNING|trainer.py:803] 2025-04-26 17:13:35,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6032
[WARNING|trainer.py:803] 2025-04-26 17:13:36,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:36,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6018
5990
6033
[WARNING|trainer.py:803] 2025-04-26 17:13:38,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:38,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:38,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6019
5991
6034
[WARNING|trainer.py:803] 2025-04-26 17:13:40,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:40,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:40,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6020
5992
6035
[WARNING|trainer.py:803] 2025-04-26 17:13:42,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:42,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:42,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6021
5993
6036
[WARNING|trainer.py:803] 2025-04-26 17:13:44,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:44,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:44,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6022
5994
6037
[WARNING|trainer.py:803] 2025-04-26 17:13:46,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:46,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:46,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5995
6023
6038
[WARNING|trainer.py:803] 2025-04-26 17:13:48,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:48,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:48,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5996
6039
6024
[WARNING|trainer.py:803] 2025-04-26 17:13:50,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:50,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:50,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5997
6040
6025
[WARNING|trainer.py:803] 2025-04-26 17:13:52,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:52,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:13:52,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5998
6026
6041
[WARNING|trainer.py:803] 2025-04-26 17:13:54,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:54,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:54,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5999
6027
6042
[WARNING|trainer.py:803] 2025-04-26 17:13:56,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:56,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:13:56,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6000
6028
6043
[WARNING|trainer.py:803] 2025-04-26 17:13:58,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:13:58,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:13:58,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6001
6029
6044
[WARNING|trainer.py:803] 2025-04-26 17:14:00,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:00,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:00,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6002
6045
6030
[WARNING|trainer.py:803] 2025-04-26 17:14:02,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:02,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:03,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6046
6003
6031
[WARNING|trainer.py:803] 2025-04-26 17:14:04,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:04,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:05,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6004
6047
6032
[WARNING|trainer.py:803] 2025-04-26 17:14:06,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:07,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:07,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6005
6048
6033
[WARNING|trainer.py:803] 2025-04-26 17:14:08,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:09,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:09,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6006
6049
6034
[WARNING|trainer.py:803] 2025-04-26 17:14:10,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:11,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:11,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6007
6050
6035
[WARNING|trainer.py:803] 2025-04-26 17:14:12,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:13,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:13,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6008
6051
6036
[WARNING|trainer.py:803] 2025-04-26 17:14:14,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:15,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:15,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6009
6037
6052
[WARNING|trainer.py:803] 2025-04-26 17:14:16,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:17,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:17,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6010
[WARNING|trainer.py:803] 2025-04-26 17:14:18,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6053
6038
[WARNING|trainer.py:803] 2025-04-26 17:14:19,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:19,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6011
[WARNING|trainer.py:803] 2025-04-26 17:14:20,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6054
6039
[WARNING|trainer.py:803] 2025-04-26 17:14:21,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:21,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6012
6055
6040
[WARNING|trainer.py:803] 2025-04-26 17:14:22,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:23,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:23,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6013
6056
[WARNING|trainer.py:803] 2025-04-26 17:14:24,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6041
[WARNING|trainer.py:803] 2025-04-26 17:14:25,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:25,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6014
6057
[WARNING|trainer.py:803] 2025-04-26 17:14:26,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6042
[WARNING|trainer.py:803] 2025-04-26 17:14:27,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:27,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6015
6058
[WARNING|trainer.py:803] 2025-04-26 17:14:28,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6043
[WARNING|trainer.py:803] 2025-04-26 17:14:29,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:29,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6016
6059
[WARNING|trainer.py:803] 2025-04-26 17:14:30,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:31,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6044
6017
[WARNING|trainer.py:803] 2025-04-26 17:14:31,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6060
[WARNING|trainer.py:803] 2025-04-26 17:14:32,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:32,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6045
[WARNING|trainer.py:803] 2025-04-26 17:14:33,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6061
6018
[WARNING|trainer.py:803] 2025-04-26 17:14:34,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:34,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6046
6062
[WARNING|trainer.py:803] 2025-04-26 17:14:35,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6019
[WARNING|trainer.py:803] 2025-04-26 17:14:36,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:36,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6047
6063
6020
[WARNING|trainer.py:803] 2025-04-26 17:14:38,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:38,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:38,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6048
6064
6021
[WARNING|trainer.py:803] 2025-04-26 17:14:40,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:40,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:40,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6049
6022
6065
[WARNING|trainer.py:803] 2025-04-26 17:14:42,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:42,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:42,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6050
6066
6023
[WARNING|trainer.py:803] 2025-04-26 17:14:44,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:44,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:44,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6067
6051
[WARNING|trainer.py:803] 2025-04-26 17:14:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6024
[WARNING|trainer.py:803] 2025-04-26 17:14:46,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:47,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6068
6052
[WARNING|trainer.py:803] 2025-04-26 17:14:48,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6025
[WARNING|trainer.py:803] 2025-04-26 17:14:48,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:48,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6069
6053
6026
[WARNING|trainer.py:803] 2025-04-26 17:14:50,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:50,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:50,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6070
6054
6027
[WARNING|trainer.py:803] 2025-04-26 17:14:52,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:52,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:14:52,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6071
6055
6028
[WARNING|trainer.py:803] 2025-04-26 17:14:54,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:54,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:14:54,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6072
6056
[WARNING|trainer.py:803] 2025-04-26 17:14:56,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6029
[WARNING|trainer.py:803] 2025-04-26 17:14:56,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:14:57,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6057
6073
[WARNING|trainer.py:803] 2025-04-26 17:14:58,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:14:58,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6030
6074
6058
[WARNING|trainer.py:803] 2025-04-26 17:14:59,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:00,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:00,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6031
6059
[WARNING|trainer.py:803] 2025-04-26 17:15:01,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6075
[WARNING|trainer.py:803] 2025-04-26 17:15:02,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:02,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6032
6060
[WARNING|trainer.py:803] 2025-04-26 17:15:03,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6076
[WARNING|trainer.py:803] 2025-04-26 17:15:04,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:04,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6033
6061
[WARNING|trainer.py:803] 2025-04-26 17:15:05,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6077
[WARNING|trainer.py:803] 2025-04-26 17:15:05,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:06,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6034
6062
6078
[WARNING|trainer.py:803] 2025-04-26 17:15:07,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:07,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:08,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6063
6035
6079
[WARNING|trainer.py:803] 2025-04-26 17:15:09,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:09,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:10,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6064
6036
6080
[WARNING|trainer.py:803] 2025-04-26 17:15:11,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:11,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:12,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6037
6081
6065
[WARNING|trainer.py:803] 2025-04-26 17:15:13,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:13,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:13,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6038
6082
6066
[WARNING|trainer.py:803] 2025-04-26 17:15:15,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:15,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:15,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6067
6039
6083
[WARNING|trainer.py:803] 2025-04-26 17:15:17,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:17,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:18,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6068
6040
[WARNING|trainer.py:803] 2025-04-26 17:15:19,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:19,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6084
[WARNING|trainer.py:803] 2025-04-26 17:15:20,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6069
6041
[WARNING|trainer.py:803] 2025-04-26 17:15:21,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6085
[WARNING|trainer.py:803] 2025-04-26 17:15:21,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:22,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6070
6042
[WARNING|trainer.py:803] 2025-04-26 17:15:23,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:23,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6086
[WARNING|trainer.py:803] 2025-04-26 17:15:24,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6071
6043
[WARNING|trainer.py:803] 2025-04-26 17:15:25,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:25,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6087
[WARNING|trainer.py:803] 2025-04-26 17:15:26,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6072
6044
6088
[WARNING|trainer.py:803] 2025-04-26 17:15:27,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:28,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:28,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6073
6045
6089
[WARNING|trainer.py:803] 2025-04-26 17:15:29,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:29,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:30,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6074
6046
6090
[WARNING|trainer.py:803] 2025-04-26 17:15:31,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:32,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:32,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6075
6091
6047
[WARNING|trainer.py:803] 2025-04-26 17:15:34,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:34,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:34,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6076
6048
6092
[WARNING|trainer.py:803] 2025-04-26 17:15:35,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:36,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:36,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6077
6049
6093
[WARNING|trainer.py:803] 2025-04-26 17:15:37,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:38,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:38,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6078
6094
6050
[WARNING|trainer.py:803] 2025-04-26 17:15:39,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:40,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:40,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6079
6095
6051
[WARNING|trainer.py:803] 2025-04-26 17:15:41,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:42,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:42,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6080
6096
[WARNING|trainer.py:803] 2025-04-26 17:15:43,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6052
[WARNING|trainer.py:803] 2025-04-26 17:15:44,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:44,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6081
6097
[WARNING|trainer.py:803] 2025-04-26 17:15:45,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6053
[WARNING|trainer.py:803] 2025-04-26 17:15:46,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:46,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6082
[WARNING|trainer.py:803] 2025-04-26 17:15:47,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6054
6098
[WARNING|trainer.py:803] 2025-04-26 17:15:48,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:15:48,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6083
6099
6055
[WARNING|trainer.py:803] 2025-04-26 17:15:49,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:50,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:50,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6084
6100
6056
[WARNING|trainer.py:803] 2025-04-26 17:15:52,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:52,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:15:52,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6085
6057
6101
[WARNING|trainer.py:803] 2025-04-26 17:15:54,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:54,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:54,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6058
6086
6102
[WARNING|trainer.py:803] 2025-04-26 17:15:56,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:56,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:56,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6059
6103
6087
[WARNING|trainer.py:803] 2025-04-26 17:15:58,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:15:58,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:15:58,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6060
6088
6104
[WARNING|trainer.py:803] 2025-04-26 17:16:00,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:00,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:00,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6061
6089
[WARNING|trainer.py:803] 2025-04-26 17:16:01,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6105
[WARNING|trainer.py:803] 2025-04-26 17:16:02,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:02,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6062
6090
[WARNING|trainer.py:803] 2025-04-26 17:16:03,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:04,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6106
6063
[WARNING|trainer.py:803] 2025-04-26 17:16:04,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6091
[WARNING|trainer.py:803] 2025-04-26 17:16:05,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:06,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6107
6064
[WARNING|trainer.py:803] 2025-04-26 17:16:07,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:07,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6092
6108
[WARNING|trainer.py:803] 2025-04-26 17:16:08,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6065
[WARNING|trainer.py:803] 2025-04-26 17:16:09,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6093
[WARNING|trainer.py:803] 2025-04-26 17:16:09,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6109
[WARNING|trainer.py:803] 2025-04-26 17:16:10,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6066
[WARNING|trainer.py:803] 2025-04-26 17:16:10,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6094
[WARNING|trainer.py:803] 2025-04-26 17:16:11,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:12,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6110
6067
[WARNING|trainer.py:803] 2025-04-26 17:16:13,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6095
[WARNING|trainer.py:803] 2025-04-26 17:16:13,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6111
[WARNING|trainer.py:803] 2025-04-26 17:16:14,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6068
[WARNING|trainer.py:803] 2025-04-26 17:16:14,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:15,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6096
6112
[WARNING|trainer.py:803] 2025-04-26 17:16:16,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6069
[WARNING|trainer.py:803] 2025-04-26 17:16:16,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:17,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6097
6113
[WARNING|trainer.py:803] 2025-04-26 17:16:18,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6070
[WARNING|trainer.py:803] 2025-04-26 17:16:19,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:19,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6098
6114
6071
[WARNING|trainer.py:803] 2025-04-26 17:16:20,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:21,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:21,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6099
6115
[WARNING|trainer.py:803] 2025-04-26 17:16:22,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6072
[WARNING|trainer.py:803] 2025-04-26 17:16:23,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:23,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6100
6116
[WARNING|trainer.py:803] 2025-04-26 17:16:24,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6073
[WARNING|trainer.py:803] 2025-04-26 17:16:25,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:25,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6101
6074
6117
[WARNING|trainer.py:803] 2025-04-26 17:16:26,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:27,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:27,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6102
[WARNING|trainer.py:803] 2025-04-26 17:16:28,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6075
6118
[WARNING|trainer.py:803] 2025-04-26 17:16:29,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:29,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6103
6076
[WARNING|trainer.py:803] 2025-04-26 17:16:30,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6119
[WARNING|trainer.py:803] 2025-04-26 17:16:31,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:31,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6104
6077
[WARNING|trainer.py:803] 2025-04-26 17:16:32,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6120
[WARNING|trainer.py:803] 2025-04-26 17:16:33,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6105
[WARNING|trainer.py:803] 2025-04-26 17:16:34,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6078
[WARNING|trainer.py:803] 2025-04-26 17:16:34,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6121
[WARNING|trainer.py:803] 2025-04-26 17:16:35,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6106
[WARNING|trainer.py:803] 2025-04-26 17:16:36,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6079
[WARNING|trainer.py:803] 2025-04-26 17:16:37,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:37,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6122
6107
[WARNING|trainer.py:803] 2025-04-26 17:16:38,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6080
[WARNING|trainer.py:803] 2025-04-26 17:16:39,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:39,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6123
6108
6081
[WARNING|trainer.py:803] 2025-04-26 17:16:40,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:16:41,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:41,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6124
6109
6082
[WARNING|trainer.py:803] 2025-04-26 17:16:42,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:43,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:43,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6125
6110
6083
[WARNING|trainer.py:803] 2025-04-26 17:16:45,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:16:45,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:16:45,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6111
6126
[WARNING|trainer.py:803] 2025-04-26 17:16:47,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:47,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6084
[WARNING|trainer.py:803] 2025-04-26 17:16:47,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6112
6127
6085
[WARNING|trainer.py:803] 2025-04-26 17:16:49,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:49,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:49,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6128
6113
[WARNING|trainer.py:803] 2025-04-26 17:16:51,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:16:51,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6086
[WARNING|trainer.py:803] 2025-04-26 17:16:52,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6129
6114
[WARNING|trainer.py:803] 2025-04-26 17:16:53,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:53,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6087
[WARNING|trainer.py:803] 2025-04-26 17:16:54,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6130
6115
[WARNING|trainer.py:803] 2025-04-26 17:16:55,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6088
[WARNING|trainer.py:803] 2025-04-26 17:16:55,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6131
[WARNING|trainer.py:803] 2025-04-26 17:16:56,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6116
[WARNING|trainer.py:803] 2025-04-26 17:16:56,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6089
[WARNING|trainer.py:803] 2025-04-26 17:16:57,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:57,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6132
6117
[WARNING|trainer.py:803] 2025-04-26 17:16:58,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6090
[WARNING|trainer.py:803] 2025-04-26 17:16:59,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:16:59,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6133
6091
[WARNING|trainer.py:803] 2025-04-26 17:17:01,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6118
[WARNING|trainer.py:803] 2025-04-26 17:17:01,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:17:02,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6134
[WARNING|trainer.py:803] 2025-04-26 17:17:03,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6119
6092
[WARNING|trainer.py:803] 2025-04-26 17:17:04,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:04,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6135
6093
[WARNING|trainer.py:803] 2025-04-26 17:17:05,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6120
[WARNING|trainer.py:803] 2025-04-26 17:17:06,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:06,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6136
6094
[WARNING|trainer.py:803] 2025-04-26 17:17:07,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6121
[WARNING|trainer.py:803] 2025-04-26 17:17:07,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:08,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6137
6095
[WARNING|trainer.py:803] 2025-04-26 17:17:09,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:10,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6122
6138
[WARNING|trainer.py:803] 2025-04-26 17:17:11,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6096
[WARNING|trainer.py:803] 2025-04-26 17:17:11,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:11,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6123
6139
[WARNING|trainer.py:803] 2025-04-26 17:17:12,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6097
[WARNING|trainer.py:803] 2025-04-26 17:17:13,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:13,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6124
[WARNING|trainer.py:803] 2025-04-26 17:17:14,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6140
6098
[WARNING|trainer.py:803] 2025-04-26 17:17:16,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:16,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6125
6141
[WARNING|trainer.py:803] 2025-04-26 17:17:17,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6099
[WARNING|trainer.py:803] 2025-04-26 17:17:18,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:18,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6126
6142
[WARNING|trainer.py:803] 2025-04-26 17:17:19,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6100
[WARNING|trainer.py:803] 2025-04-26 17:17:20,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:20,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6127
6143
[WARNING|trainer.py:803] 2025-04-26 17:17:21,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6101
[WARNING|trainer.py:803] 2025-04-26 17:17:22,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:22,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6128
[WARNING|trainer.py:803] 2025-04-26 17:17:23,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6144
6102
[WARNING|trainer.py:803] 2025-04-26 17:17:24,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:24,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6129
[WARNING|trainer.py:803] 2025-04-26 17:17:25,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6103
6145
[WARNING|trainer.py:803] 2025-04-26 17:17:26,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:26,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6130
[WARNING|trainer.py:803] 2025-04-26 17:17:27,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6146
6104
[WARNING|trainer.py:803] 2025-04-26 17:17:28,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6131
[WARNING|trainer.py:803] 2025-04-26 17:17:28,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:29,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6105
6147
6132
[WARNING|trainer.py:803] 2025-04-26 17:17:30,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:30,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:31,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6148
6106
6133
[WARNING|trainer.py:803] 2025-04-26 17:17:32,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:32,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:33,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6149
6107
6134
[WARNING|trainer.py:803] 2025-04-26 17:17:34,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:34,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:35,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6108
6150
[WARNING|trainer.py:803] 2025-04-26 17:17:37,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6135
[WARNING|trainer.py:803] 2025-04-26 17:17:37,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:37,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6109
6151
[WARNING|trainer.py:803] 2025-04-26 17:17:38,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:39,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6136
[WARNING|trainer.py:803] 2025-04-26 17:17:39,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6110
6152
[WARNING|trainer.py:803] 2025-04-26 17:17:41,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6137
[WARNING|trainer.py:803] 2025-04-26 17:17:41,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6111
[WARNING|trainer.py:803] 2025-04-26 17:17:41,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6153
[WARNING|trainer.py:803] 2025-04-26 17:17:42,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6138
[WARNING|trainer.py:803] 2025-04-26 17:17:43,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:17:43,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6112
6154
[WARNING|trainer.py:803] 2025-04-26 17:17:44,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6139
[WARNING|trainer.py:803] 2025-04-26 17:17:45,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6113
[WARNING|trainer.py:803] 2025-04-26 17:17:46,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6155
[WARNING|trainer.py:803] 2025-04-26 17:17:47,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:17:47,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6140
[WARNING|trainer.py:803] 2025-04-26 17:17:48,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6114
6156
[WARNING|trainer.py:803] 2025-04-26 17:17:49,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:17:49,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6141
[WARNING|trainer.py:803] 2025-04-26 17:17:50,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6157
6115
[WARNING|trainer.py:803] 2025-04-26 17:17:51,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:51,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6142
[WARNING|trainer.py:803] 2025-04-26 17:17:52,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6116
6158
[WARNING|trainer.py:803] 2025-04-26 17:17:53,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6143
[WARNING|trainer.py:803] 2025-04-26 17:17:53,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6117
[WARNING|trainer.py:803] 2025-04-26 17:17:54,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6159
[WARNING|trainer.py:803] 2025-04-26 17:17:55,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6144
[WARNING|trainer.py:803] 2025-04-26 17:17:55,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:56,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6118
6160
[WARNING|trainer.py:803] 2025-04-26 17:17:57,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:17:58,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6145
[WARNING|trainer.py:803] 2025-04-26 17:17:59,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6161
6119
6146
[WARNING|trainer.py:803] 2025-04-26 17:17:59,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:00,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:00,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6162
6120
[WARNING|trainer.py:803] 2025-04-26 17:18:01,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:02,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6147
[WARNING|trainer.py:803] 2025-04-26 17:18:03,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6163
6121
[WARNING|trainer.py:803] 2025-04-26 17:18:04,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6148
[WARNING|trainer.py:803] 2025-04-26 17:18:04,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:05,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6164
6122
[WARNING|trainer.py:803] 2025-04-26 17:18:06,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6149
[WARNING|trainer.py:803] 2025-04-26 17:18:07,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:07,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6165
6123
[WARNING|trainer.py:803] 2025-04-26 17:18:08,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6150
[WARNING|trainer.py:803] 2025-04-26 17:18:08,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6166
[WARNING|trainer.py:803] 2025-04-26 17:18:09,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:10,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6124
6151
[WARNING|trainer.py:803] 2025-04-26 17:18:10,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6167
[WARNING|trainer.py:803] 2025-04-26 17:18:11,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:12,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6125
6152
6168
[WARNING|trainer.py:803] 2025-04-26 17:18:13,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:13,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:14,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6126
6153
6169
[WARNING|trainer.py:803] 2025-04-26 17:18:15,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:15,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:15,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6127
6170
6154
[WARNING|trainer.py:803] 2025-04-26 17:18:17,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:17,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:17,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6128
6171
6155
[WARNING|trainer.py:803] 2025-04-26 17:18:19,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:19,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:19,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6129
6172
6156
[WARNING|trainer.py:803] 2025-04-26 17:18:21,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:21,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:21,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6130
6173
6157
[WARNING|trainer.py:803] 2025-04-26 17:18:23,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:23,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:23,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6131
6174
[WARNING|trainer.py:803] 2025-04-26 17:18:25,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:25,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6158
[WARNING|trainer.py:803] 2025-04-26 17:18:26,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6175
6132
[WARNING|trainer.py:803] 2025-04-26 17:18:27,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6159
[WARNING|trainer.py:803] 2025-04-26 17:18:27,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:28,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6176
6133
6160
[WARNING|trainer.py:803] 2025-04-26 17:18:29,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:29,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:30,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6177
6134
[WARNING|trainer.py:803] 2025-04-26 17:18:31,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6161
[WARNING|trainer.py:803] 2025-04-26 17:18:31,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:32,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6178
6135
6162
[WARNING|trainer.py:803] 2025-04-26 17:18:33,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 17:18:33,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:34,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6179
6136
6163
[WARNING|trainer.py:803] 2025-04-26 17:18:35,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:36,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:36,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6180
6137
6164
[WARNING|trainer.py:803] 2025-04-26 17:18:37,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:38,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:38,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6181
6138
6165
[WARNING|trainer.py:803] 2025-04-26 17:18:40,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:40,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:18:40,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6182
6166
6139
[WARNING|trainer.py:803] 2025-04-26 17:18:42,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:42,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:42,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6183
6167
6140
[WARNING|trainer.py:803] 2025-04-26 17:18:44,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:44,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:44,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6184
6168
6141
[WARNING|trainer.py:803] 2025-04-26 17:18:46,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:46,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:46,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6169
6185
6142
[WARNING|trainer.py:803] 2025-04-26 17:18:48,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:48,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:48,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6170
6186
[WARNING|trainer.py:803] 2025-04-26 17:18:49,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:49,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6143
[WARNING|trainer.py:803] 2025-04-26 17:18:50,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6171
6187
[WARNING|trainer.py:803] 2025-04-26 17:18:51,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:18:52,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6144
[WARNING|trainer.py:803] 2025-04-26 17:18:52,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6172
6188
[WARNING|trainer.py:803] 2025-04-26 17:18:53,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:54,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6145
6173
[WARNING|trainer.py:803] 2025-04-26 17:18:55,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6189
[WARNING|trainer.py:803] 2025-04-26 17:18:55,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6146
[WARNING|trainer.py:803] 2025-04-26 17:18:56,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6174
[WARNING|trainer.py:803] 2025-04-26 17:18:56,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6190
[WARNING|trainer.py:803] 2025-04-26 17:18:57,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:58,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6147
6175
6191
[WARNING|trainer.py:803] 2025-04-26 17:18:59,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:18:59,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:18:59,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6148
6176
[WARNING|trainer.py:803] 2025-04-26 17:19:01,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6192
[WARNING|trainer.py:803] 2025-04-26 17:19:01,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:02,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6149
6177
[WARNING|trainer.py:803] 2025-04-26 17:19:03,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:03,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6193
[WARNING|trainer.py:803] 2025-04-26 17:19:04,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6150
6178
[WARNING|trainer.py:803] 2025-04-26 17:19:05,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6194
[WARNING|trainer.py:803] 2025-04-26 17:19:05,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 17:19:06,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6151
6179
[WARNING|trainer.py:803] 2025-04-26 17:19:07,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6195
[WARNING|trainer.py:803] 2025-04-26 17:19:07,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:08,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6152
6180
[WARNING|trainer.py:803] 2025-04-26 17:19:09,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6196
[WARNING|trainer.py:803] 2025-04-26 17:19:10,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:10,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6153
6181
[WARNING|trainer.py:803] 2025-04-26 17:19:12,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:12,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6197
[WARNING|trainer.py:803] 2025-04-26 17:19:13,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6154
6182
[WARNING|trainer.py:803] 2025-04-26 17:19:13,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:14,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6198
[WARNING|trainer.py:803] 2025-04-26 17:19:15,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6155
6183
6199
[WARNING|trainer.py:803] 2025-04-26 17:19:16,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:16,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:16,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6156
6184
[WARNING|trainer.py:803] 2025-04-26 17:19:18,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6200
[WARNING|trainer.py:803] 2025-04-26 17:19:18,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:18,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6157
6185
[WARNING|trainer.py:803] 2025-04-26 17:19:19,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6201
[WARNING|trainer.py:803] 2025-04-26 17:19:20,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:21,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6186
6158
[WARNING|trainer.py:803] 2025-04-26 17:19:22,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6202
[WARNING|trainer.py:803] 2025-04-26 17:19:22,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:23,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6187
6159
[WARNING|trainer.py:803] 2025-04-26 17:19:24,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:24,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6203
[WARNING|trainer.py:803] 2025-04-26 17:19:25,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6188
6160
6204
[WARNING|trainer.py:803] 2025-04-26 17:19:26,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:26,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:27,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6189
6161
6205
[WARNING|trainer.py:803] 2025-04-26 17:19:28,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:28,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:28,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6162
6190
6206
[WARNING|trainer.py:803] 2025-04-26 17:19:30,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:30,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:30,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6191
6163
6207
[WARNING|trainer.py:803] 2025-04-26 17:19:32,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:19:32,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:32,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6192
6164
6208
[WARNING|trainer.py:803] 2025-04-26 17:19:34,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:34,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:34,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6165
6193
6209
[WARNING|trainer.py:803] 2025-04-26 17:19:36,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:36,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:37,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6166
6210
6194
[WARNING|trainer.py:803] 2025-04-26 17:19:38,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:38,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:39,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6211
6167
6195
[WARNING|trainer.py:803] 2025-04-26 17:19:40,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:40,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:40,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6212
6168
6196
[WARNING|trainer.py:803] 2025-04-26 17:19:42,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:42,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:43,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6213
6169
[WARNING|trainer.py:803] 2025-04-26 17:19:44,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:19:44,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6197
[WARNING|trainer.py:803] 2025-04-26 17:19:45,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6214
6170
[WARNING|trainer.py:803] 2025-04-26 17:19:46,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:46,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6198
6215
6171
[WARNING|trainer.py:803] 2025-04-26 17:19:47,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:48,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:48,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6199
[WARNING|trainer.py:803] 2025-04-26 17:19:49,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6172
6216
[WARNING|trainer.py:803] 2025-04-26 17:19:50,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:50,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6200
6173
6217
[WARNING|trainer.py:803] 2025-04-26 17:19:51,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:52,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:52,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6201
6174
6218
[WARNING|trainer.py:803] 2025-04-26 17:19:53,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:54,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:54,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6202
6175
6219
[WARNING|trainer.py:803] 2025-04-26 17:19:55,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:19:56,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:19:56,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6203
6220
6176
[WARNING|trainer.py:803] 2025-04-26 17:19:57,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:58,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:58,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6204
6221
6177
[WARNING|trainer.py:803] 2025-04-26 17:19:59,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:19:59,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:00,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6205
6222
[WARNING|trainer.py:803] 2025-04-26 17:20:01,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6178
[WARNING|trainer.py:803] 2025-04-26 17:20:01,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:02,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
6206
6223
[WARNING|trainer.py:803] 2025-04-26 17:20:03,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6179
[WARNING|trainer.py:803] 2025-04-26 17:20:03,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6207
[WARNING|trainer.py:803] 2025-04-26 17:20:04,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6224
[WARNING|trainer.py:803] 2025-04-26 17:20:05,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:05,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6180
6208
6225
[WARNING|trainer.py:803] 2025-04-26 17:20:06,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:07,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:07,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6181
6226
6209
[WARNING|trainer.py:803] 2025-04-26 17:20:08,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:09,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:09,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6182
6227
6210
[WARNING|trainer.py:803] 2025-04-26 17:20:10,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:11,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:11,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6228
6183
6211
[WARNING|trainer.py:803] 2025-04-26 17:20:12,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:13,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:13,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6229
6212
6184
[WARNING|trainer.py:803] 2025-04-26 17:20:14,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:14,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:15,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6230
6213
6185
[WARNING|trainer.py:803] 2025-04-26 17:20:16,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:16,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:17,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6231
6214
6186
[WARNING|trainer.py:803] 2025-04-26 17:20:18,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:18,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:19,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6232
6215
6187
[WARNING|trainer.py:803] 2025-04-26 17:20:20,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:20,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:21,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6233
6216
[WARNING|trainer.py:803] 2025-04-26 17:20:22,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6188
[WARNING|trainer.py:803] 2025-04-26 17:20:22,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6234
[WARNING|trainer.py:803] 2025-04-26 17:20:23,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6217
[WARNING|trainer.py:803] 2025-04-26 17:20:23,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:24,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6189
6235
[WARNING|trainer.py:803] 2025-04-26 17:20:25,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6218
[WARNING|trainer.py:803] 2025-04-26 17:20:25,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:26,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6190
6236
[WARNING|trainer.py:803] 2025-04-26 17:20:27,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6219
[WARNING|trainer.py:803] 2025-04-26 17:20:27,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:28,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6191
6237
[WARNING|trainer.py:803] 2025-04-26 17:20:29,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6220
[WARNING|trainer.py:803] 2025-04-26 17:20:29,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:30,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6192
6238
6221
[WARNING|trainer.py:803] 2025-04-26 17:20:31,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:31,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:31,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6239
6193
6222
[WARNING|trainer.py:803] 2025-04-26 17:20:33,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:33,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:33,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6240
6194
6223
[WARNING|trainer.py:803] 2025-04-26 17:20:35,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:35,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:35,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6241
6224
6195
[WARNING|trainer.py:803] 2025-04-26 17:20:37,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:37,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:37,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6242
6225
[WARNING|trainer.py:803] 2025-04-26 17:20:39,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6196
[WARNING|trainer.py:803] 2025-04-26 17:20:39,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:40,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6243
6226
[WARNING|trainer.py:803] 2025-04-26 17:20:41,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:41,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6197
6244
6227
[WARNING|trainer.py:803] 2025-04-26 17:20:42,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:42,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:42,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6198
6245
6228
[WARNING|trainer.py:803] 2025-04-26 17:20:44,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:44,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:44,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6199
6246
6229
[WARNING|trainer.py:803] 2025-04-26 17:20:46,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:46,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:46,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6247
6200
6230
[WARNING|trainer.py:803] 2025-04-26 17:20:48,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:48,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:48,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6248
6201
6231
[WARNING|trainer.py:803] 2025-04-26 17:20:50,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:50,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:50,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6249
6232
6202
[WARNING|trainer.py:803] 2025-04-26 17:20:52,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:52,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:52,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6233
6250
6203
[WARNING|trainer.py:803] 2025-04-26 17:20:53,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:54,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:20:54,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6251
6234
6204
[WARNING|trainer.py:803] 2025-04-26 17:20:55,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:55,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:56,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6252
6235
6205
[WARNING|trainer.py:803] 2025-04-26 17:20:57,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:57,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:20:58,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6253
6236
6206
[WARNING|trainer.py:803] 2025-04-26 17:20:59,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:20:59,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:20:59,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6254
6237
[WARNING|trainer.py:803] 2025-04-26 17:21:01,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6207
[WARNING|trainer.py:803] 2025-04-26 17:21:01,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6255
[WARNING|trainer.py:803] 2025-04-26 17:21:02,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6238
[WARNING|trainer.py:803] 2025-04-26 17:21:02,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6208
[WARNING|trainer.py:803] 2025-04-26 17:21:03,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6256
[WARNING|trainer.py:803] 2025-04-26 17:21:04,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6239
[WARNING|trainer.py:803] 2025-04-26 17:21:04,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:05,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6209
6257
[WARNING|trainer.py:803] 2025-04-26 17:21:06,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6240
[WARNING|trainer.py:803] 2025-04-26 17:21:06,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:07,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6210
6258
[WARNING|trainer.py:803] 2025-04-26 17:21:08,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6241
[WARNING|trainer.py:803] 2025-04-26 17:21:08,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:08,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6211
6259
6242
[WARNING|trainer.py:803] 2025-04-26 17:21:09,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:10,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:10,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6212
6260
6243
[WARNING|trainer.py:803] 2025-04-26 17:21:11,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:12,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:12,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6213
6261
6244
[WARNING|trainer.py:803] 2025-04-26 17:21:13,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:14,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:14,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6214
6262
6245
[WARNING|trainer.py:803] 2025-04-26 17:21:15,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:21:16,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:16,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6215
6246
6263
[WARNING|trainer.py:803] 2025-04-26 17:21:17,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:21:17,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:17,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6216
6247
6264
[WARNING|trainer.py:803] 2025-04-26 17:21:19,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:21:19,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:19,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6217
6248
6265
[WARNING|trainer.py:803] 2025-04-26 17:21:21,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:21,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:21,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6218
6249
6266
[WARNING|trainer.py:803] 2025-04-26 17:21:23,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:23,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:23,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6219
6250
6267
[WARNING|trainer.py:803] 2025-04-26 17:21:25,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:25,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:25,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6220
6251
6268
[WARNING|trainer.py:803] 2025-04-26 17:21:27,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:27,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:27,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6221
6252
6269
[WARNING|trainer.py:803] 2025-04-26 17:21:29,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:21:29,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:29,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6253
6222
6270
[WARNING|trainer.py:803] 2025-04-26 17:21:30,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:30,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:31,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6254
6223
[WARNING|trainer.py:803] 2025-04-26 17:21:32,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6271
[WARNING|trainer.py:803] 2025-04-26 17:21:33,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6255
[WARNING|trainer.py:803] 2025-04-26 17:21:33,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6224
[WARNING|trainer.py:803] 2025-04-26 17:21:34,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:34,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6272
6256
[WARNING|trainer.py:803] 2025-04-26 17:21:35,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6225
[WARNING|trainer.py:803] 2025-04-26 17:21:36,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:36,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6273
6257
6226
[WARNING|trainer.py:803] 2025-04-26 17:21:37,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:37,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:38,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6258
6274
6227
[WARNING|trainer.py:803] 2025-04-26 17:21:39,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:39,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:40,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6259
6275
6228
[WARNING|trainer.py:803] 2025-04-26 17:21:41,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:42,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:42,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6260
6276
6229
[WARNING|trainer.py:803] 2025-04-26 17:21:43,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:43,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:44,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6261
6277
6230
[WARNING|trainer.py:803] 2025-04-26 17:21:45,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:45,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:45,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6278
6262
6231
[WARNING|trainer.py:803] 2025-04-26 17:21:47,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:47,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:47,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6279
6263
6232
[WARNING|trainer.py:803] 2025-04-26 17:21:49,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:49,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:49,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6280
6264
6233
[WARNING|trainer.py:803] 2025-04-26 17:21:51,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:51,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:51,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6265
6234
6281
[WARNING|trainer.py:803] 2025-04-26 17:21:53,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:21:53,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:53,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6235
6266
6282
[WARNING|trainer.py:803] 2025-04-26 17:21:55,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:21:55,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:55,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6236
6267
6283
[WARNING|trainer.py:803] 2025-04-26 17:21:57,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:57,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:57,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6268
6237
6284
[WARNING|trainer.py:803] 2025-04-26 17:21:59,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:21:59,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:21:59,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6269
6238
6285
[WARNING|trainer.py:803] 2025-04-26 17:22:01,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:01,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:01,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6270
6239
6286
[WARNING|trainer.py:803] 2025-04-26 17:22:03,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:03,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:03,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6271
6240
[WARNING|trainer.py:803] 2025-04-26 17:22:04,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:05,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6287
6241
[WARNING|trainer.py:803] 2025-04-26 17:22:06,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6272
[WARNING|trainer.py:803] 2025-04-26 17:22:06,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6288
[WARNING|trainer.py:803] 2025-04-26 17:22:07,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:07,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6242
6273
6289
[WARNING|trainer.py:803] 2025-04-26 17:22:08,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:09,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:09,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6243
6274
[WARNING|trainer.py:803] 2025-04-26 17:22:10,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6290
[WARNING|trainer.py:803] 2025-04-26 17:22:11,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6244
[WARNING|trainer.py:803] 2025-04-26 17:22:12,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:12,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6275
6291
[WARNING|trainer.py:803] 2025-04-26 17:22:13,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6245
[WARNING|trainer.py:803] 2025-04-26 17:22:14,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:14,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6276
6292
6246
[WARNING|trainer.py:803] 2025-04-26 17:22:15,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:15,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:16,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6277
6293
6247
[WARNING|trainer.py:803] 2025-04-26 17:22:17,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:17,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:17,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6278
6294
[WARNING|trainer.py:803] 2025-04-26 17:22:18,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6248
[WARNING|trainer.py:803] 2025-04-26 17:22:19,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:19,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6279
[WARNING|trainer.py:803] 2025-04-26 17:22:20,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6295
6249
[WARNING|trainer.py:803] 2025-04-26 17:22:21,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:21,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6280
[WARNING|trainer.py:803] 2025-04-26 17:22:22,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6296
6250
[WARNING|trainer.py:803] 2025-04-26 17:22:23,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:23,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6281
6251
6297
[WARNING|trainer.py:803] 2025-04-26 17:22:24,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:25,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:25,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6282
6252
6298
[WARNING|trainer.py:803] 2025-04-26 17:22:26,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:27,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:27,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6283
6253
6299
[WARNING|trainer.py:803] 2025-04-26 17:22:28,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:28,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:29,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6254
6284
6300
[WARNING|trainer.py:803] 2025-04-26 17:22:30,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:30,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:31,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6255
6285
6301
[WARNING|trainer.py:803] 2025-04-26 17:22:32,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:32,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:32,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6302
6256
[WARNING|trainer.py:803] 2025-04-26 17:22:34,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6286
[WARNING|trainer.py:803] 2025-04-26 17:22:34,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6303
[WARNING|trainer.py:803] 2025-04-26 17:22:35,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6257
[WARNING|trainer.py:803] 2025-04-26 17:22:35,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6304
[WARNING|trainer.py:803] 2025-04-26 17:22:36,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6287
[WARNING|trainer.py:803] 2025-04-26 17:22:36,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6305
6258
[WARNING|trainer.py:803] 2025-04-26 17:22:37,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:38,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:38,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6288
6306
[WARNING|trainer.py:803] 2025-04-26 17:22:39,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6259
[WARNING|trainer.py:803] 2025-04-26 17:22:39,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6307
[WARNING|trainer.py:803] 2025-04-26 17:22:40,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6289
[WARNING|trainer.py:803] 2025-04-26 17:22:40,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6260
[WARNING|trainer.py:803] 2025-04-26 17:22:41,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6308
[WARNING|trainer.py:803] 2025-04-26 17:22:41,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:42,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6309
6290
6261
[WARNING|trainer.py:803] 2025-04-26 17:22:43,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:43,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6310
[WARNING|trainer.py:803] 2025-04-26 17:22:44,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6291
[WARNING|trainer.py:803] 2025-04-26 17:22:44,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6262
6311
[WARNING|trainer.py:803] 2025-04-26 17:22:45,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:46,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:46,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6312
6292
6263
[WARNING|trainer.py:803] 2025-04-26 17:22:47,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:47,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6313
[WARNING|trainer.py:803] 2025-04-26 17:22:48,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6293
[WARNING|trainer.py:803] 2025-04-26 17:22:48,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6264
6314
[WARNING|trainer.py:803] 2025-04-26 17:22:49,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:49,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:50,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6294
6315
6265
[WARNING|trainer.py:803] 2025-04-26 17:22:51,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:51,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:51,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6316
6295
[WARNING|trainer.py:803] 2025-04-26 17:22:52,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6266
[WARNING|trainer.py:803] 2025-04-26 17:22:53,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6317
[WARNING|trainer.py:803] 2025-04-26 17:22:53,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:22:54,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6296
6318
6267
[WARNING|trainer.py:803] 2025-04-26 17:22:55,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:22:55,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:55,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6319
6297
[WARNING|trainer.py:803] 2025-04-26 17:22:56,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6268
[WARNING|trainer.py:803] 2025-04-26 17:22:57,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6320
[WARNING|trainer.py:803] 2025-04-26 17:22:57,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:58,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6298
6321
6269
[WARNING|trainer.py:803] 2025-04-26 17:22:59,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:22:59,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:22:59,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6322
6299
6270
[WARNING|trainer.py:803] 2025-04-26 17:23:00,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:01,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6323
[WARNING|trainer.py:803] 2025-04-26 17:23:01,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:02,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6300
6271
6324
[WARNING|trainer.py:803] 2025-04-26 17:23:03,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:03,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:03,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6301
6325
[WARNING|trainer.py:803] 2025-04-26 17:23:04,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:04,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6272
6302
6326
[WARNING|trainer.py:803] 2025-04-26 17:23:05,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:05,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:06,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6303
6327
6273
[WARNING|trainer.py:803] 2025-04-26 17:23:07,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:07,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6304
[WARNING|trainer.py:803] 2025-04-26 17:23:07,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6328
[WARNING|trainer.py:803] 2025-04-26 17:23:08,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:08,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6305
6274
6329
[WARNING|trainer.py:803] 2025-04-26 17:23:09,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:09,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:10,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6306
6330
[WARNING|trainer.py:803] 2025-04-26 17:23:11,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6275
[WARNING|trainer.py:803] 2025-04-26 17:23:11,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6307
[WARNING|trainer.py:803] 2025-04-26 17:23:11,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6331
[WARNING|trainer.py:803] 2025-04-26 17:23:12,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6308
[WARNING|trainer.py:803] 2025-04-26 17:23:12,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6276
6332
[WARNING|trainer.py:803] 2025-04-26 17:23:13,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:13,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:14,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6309
6333
6277
[WARNING|trainer.py:803] 2025-04-26 17:23:15,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6310
[WARNING|trainer.py:803] 2025-04-26 17:23:15,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:15,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6334
[WARNING|trainer.py:803] 2025-04-26 17:23:16,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6278
6311
[WARNING|trainer.py:803] 2025-04-26 17:23:16,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6335
[WARNING|trainer.py:803] 2025-04-26 17:23:17,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:17,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6312
[WARNING|trainer.py:803] 2025-04-26 17:23:18,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6279
6336
[WARNING|trainer.py:803] 2025-04-26 17:23:18,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:19,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6313
[WARNING|trainer.py:803] 2025-04-26 17:23:19,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6337
[WARNING|trainer.py:803] 2025-04-26 17:23:20,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6280
6314
[WARNING|trainer.py:803] 2025-04-26 17:23:20,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6338
[WARNING|trainer.py:803] 2025-04-26 17:23:21,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:21,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6315
[WARNING|trainer.py:803] 2025-04-26 17:23:22,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6339
6281
[WARNING|trainer.py:803] 2025-04-26 17:23:22,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6316
[WARNING|trainer.py:803] 2025-04-26 17:23:23,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:23,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6340
[WARNING|trainer.py:803] 2025-04-26 17:23:24,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6317
6282
[WARNING|trainer.py:803] 2025-04-26 17:23:24,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6341
[WARNING|trainer.py:803] 2025-04-26 17:23:25,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:25,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6318
[WARNING|trainer.py:803] 2025-04-26 17:23:26,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6342
6283
[WARNING|trainer.py:803] 2025-04-26 17:23:26,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6319
[WARNING|trainer.py:803] 2025-04-26 17:23:27,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:27,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6343
[WARNING|trainer.py:803] 2025-04-26 17:23:28,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6320
6284
[WARNING|trainer.py:803] 2025-04-26 17:23:28,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6344
[WARNING|trainer.py:803] 2025-04-26 17:23:29,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:29,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6321
[WARNING|trainer.py:803] 2025-04-26 17:23:30,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6345
6285
[WARNING|trainer.py:803] 2025-04-26 17:23:30,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6322
[WARNING|trainer.py:803] 2025-04-26 17:23:31,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:31,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6346
[WARNING|trainer.py:803] 2025-04-26 17:23:32,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6323
[WARNING|trainer.py:803] 2025-04-26 17:23:32,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6286
6347
[WARNING|trainer.py:803] 2025-04-26 17:23:33,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6324
[WARNING|trainer.py:803] 2025-04-26 17:23:33,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:34,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:34,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6348
6325
6287
[WARNING|trainer.py:803] 2025-04-26 17:23:35,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:35,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6349
[WARNING|trainer.py:803] 2025-04-26 17:23:36,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6326
[WARNING|trainer.py:803] 2025-04-26 17:23:36,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6288
[WARNING|trainer.py:803] 2025-04-26 17:23:37,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6350
6327
[WARNING|trainer.py:803] 2025-04-26 17:23:37,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:38,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:38,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6351
6289
6328
[WARNING|trainer.py:803] 2025-04-26 17:23:39,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:39,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:39,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6352
6329
[WARNING|trainer.py:803] 2025-04-26 17:23:40,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:41,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6353
6290
6330
[WARNING|trainer.py:803] 2025-04-26 17:23:42,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:42,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:42,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6354
6331
6291
[WARNING|trainer.py:803] 2025-04-26 17:23:43,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:43,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6355
[WARNING|trainer.py:803] 2025-04-26 17:23:44,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6332
[WARNING|trainer.py:803] 2025-04-26 17:23:44,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:45,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6356
6292
6333
[WARNING|trainer.py:803] 2025-04-26 17:23:46,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:46,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:46,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6357
6334
6293
[WARNING|trainer.py:803] 2025-04-26 17:23:47,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:47,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6358
[WARNING|trainer.py:803] 2025-04-26 17:23:48,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6335
[WARNING|trainer.py:803] 2025-04-26 17:23:48,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6294
[WARNING|trainer.py:803] 2025-04-26 17:23:49,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6359
6336
[WARNING|trainer.py:803] 2025-04-26 17:23:49,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:50,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:50,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6360
6337
6295
[WARNING|trainer.py:803] 2025-04-26 17:23:51,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:51,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6361
[WARNING|trainer.py:803] 2025-04-26 17:23:51,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6338
[WARNING|trainer.py:803] 2025-04-26 17:23:52,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6296
[WARNING|trainer.py:803] 2025-04-26 17:23:53,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6362
6339
[WARNING|trainer.py:803] 2025-04-26 17:23:53,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:23:54,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:54,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6363
6340
6297
[WARNING|trainer.py:803] 2025-04-26 17:23:55,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:55,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6364
[WARNING|trainer.py:803] 2025-04-26 17:23:55,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6341
[WARNING|trainer.py:803] 2025-04-26 17:23:56,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:57,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6298
6365
6342
[WARNING|trainer.py:803] 2025-04-26 17:23:57,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:23:57,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:23:58,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6366
6343
6299
[WARNING|trainer.py:803] 2025-04-26 17:23:59,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:23:59,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6367
[WARNING|trainer.py:803] 2025-04-26 17:23:59,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6344
[WARNING|trainer.py:803] 2025-04-26 17:24:00,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6300
[WARNING|trainer.py:803] 2025-04-26 17:24:01,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6368
6345
[WARNING|trainer.py:803] 2025-04-26 17:24:01,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:01,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6301
[WARNING|trainer.py:803] 2025-04-26 17:24:02,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6369
6346
[WARNING|trainer.py:803] 2025-04-26 17:24:03,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:03,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6302
[WARNING|trainer.py:803] 2025-04-26 17:24:03,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6370
6347
[WARNING|trainer.py:803] 2025-04-26 17:24:04,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:04,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6303
[WARNING|trainer.py:803] 2025-04-26 17:24:05,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6371
6348
[WARNING|trainer.py:803] 2025-04-26 17:24:05,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:06,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6304
[WARNING|trainer.py:803] 2025-04-26 17:24:06,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6372
6349
[WARNING|trainer.py:803] 2025-04-26 17:24:07,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:07,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6305
[WARNING|trainer.py:803] 2025-04-26 17:24:07,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6373
6350
[WARNING|trainer.py:803] 2025-04-26 17:24:08,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:08,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6306
[WARNING|trainer.py:803] 2025-04-26 17:24:09,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6374
6351
[WARNING|trainer.py:803] 2025-04-26 17:24:09,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:10,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6307
[WARNING|trainer.py:803] 2025-04-26 17:24:10,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6375
6352
[WARNING|trainer.py:803] 2025-04-26 17:24:11,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:11,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6308
[WARNING|trainer.py:803] 2025-04-26 17:24:11,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6376
6353
[WARNING|trainer.py:803] 2025-04-26 17:24:12,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:12,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6309
6377
[WARNING|trainer.py:803] 2025-04-26 17:24:13,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:13,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6354
[WARNING|trainer.py:803] 2025-04-26 17:24:14,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6310
6378
[WARNING|trainer.py:803] 2025-04-26 17:24:14,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6355
[WARNING|trainer.py:803] 2025-04-26 17:24:15,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:15,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6311
6379
[WARNING|trainer.py:803] 2025-04-26 17:24:15,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:16,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6356
[WARNING|trainer.py:803] 2025-04-26 17:24:16,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6312
6380
[WARNING|trainer.py:803] 2025-04-26 17:24:17,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6357
[WARNING|trainer.py:803] 2025-04-26 17:24:17,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:18,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6313
6381
[WARNING|trainer.py:803] 2025-04-26 17:24:18,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6358
[WARNING|trainer.py:803] 2025-04-26 17:24:19,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:19,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6314
6382
[WARNING|trainer.py:803] 2025-04-26 17:24:19,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6359
[WARNING|trainer.py:803] 2025-04-26 17:24:20,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:20,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6315
[WARNING|trainer.py:803] 2025-04-26 17:24:21,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6383
6360
[WARNING|trainer.py:803] 2025-04-26 17:24:21,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:22,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6316
[WARNING|trainer.py:803] 2025-04-26 17:24:22,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6384
6361
[WARNING|trainer.py:803] 2025-04-26 17:24:23,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:23,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6317
[WARNING|trainer.py:803] 2025-04-26 17:24:23,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6385
6362
[WARNING|trainer.py:803] 2025-04-26 17:24:24,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:24,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6318
[WARNING|trainer.py:803] 2025-04-26 17:24:25,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6386
6363
[WARNING|trainer.py:803] 2025-04-26 17:24:25,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:26,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6319
[WARNING|trainer.py:803] 2025-04-26 17:24:26,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6387
6364
[WARNING|trainer.py:803] 2025-04-26 17:24:27,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:27,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6320
[WARNING|trainer.py:803] 2025-04-26 17:24:27,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6388
6365
[WARNING|trainer.py:803] 2025-04-26 17:24:28,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:28,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6321
[WARNING|trainer.py:803] 2025-04-26 17:24:29,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6389
6366
[WARNING|trainer.py:803] 2025-04-26 17:24:29,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:29,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6322
[WARNING|trainer.py:803] 2025-04-26 17:24:30,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6390
6367
[WARNING|trainer.py:803] 2025-04-26 17:24:31,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:31,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6323
[WARNING|trainer.py:803] 2025-04-26 17:24:31,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6391
6368
[WARNING|trainer.py:803] 2025-04-26 17:24:32,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:32,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6324
[WARNING|trainer.py:803] 2025-04-26 17:24:33,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6392
6369
[WARNING|trainer.py:803] 2025-04-26 17:24:33,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:33,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6325
[WARNING|trainer.py:803] 2025-04-26 17:24:34,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6393
6370
[WARNING|trainer.py:803] 2025-04-26 17:24:34,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:35,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6326
[WARNING|trainer.py:803] 2025-04-26 17:24:35,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6394
6371
[WARNING|trainer.py:803] 2025-04-26 17:24:36,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:36,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6327
[WARNING|trainer.py:803] 2025-04-26 17:24:36,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6395
6372
[WARNING|trainer.py:803] 2025-04-26 17:24:37,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:37,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6328
[WARNING|trainer.py:803] 2025-04-26 17:24:38,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6396
6373
[WARNING|trainer.py:803] 2025-04-26 17:24:38,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:39,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6329
[WARNING|trainer.py:803] 2025-04-26 17:24:39,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6397
6374
[WARNING|trainer.py:803] 2025-04-26 17:24:40,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:40,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6330
[WARNING|trainer.py:803] 2025-04-26 17:24:40,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6398
6375
[WARNING|trainer.py:803] 2025-04-26 17:24:41,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:41,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6331
[WARNING|trainer.py:803] 2025-04-26 17:24:42,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6399
6376
[WARNING|trainer.py:803] 2025-04-26 17:24:42,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:24:43,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6332
[WARNING|trainer.py:803] 2025-04-26 17:24:43,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6400
6377
[WARNING|trainer.py:803] 2025-04-26 17:24:44,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:44,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6333
[WARNING|trainer.py:803] 2025-04-26 17:24:44,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6401
6378
[WARNING|trainer.py:803] 2025-04-26 17:24:45,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:45,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6334
[WARNING|trainer.py:803] 2025-04-26 17:24:46,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6402
6379
[WARNING|trainer.py:803] 2025-04-26 17:24:46,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:24:47,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6335
[WARNING|trainer.py:803] 2025-04-26 17:24:47,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6403
6380
[WARNING|trainer.py:803] 2025-04-26 17:24:48,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:48,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6336
[WARNING|trainer.py:803] 2025-04-26 17:24:48,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6404
6381
[WARNING|trainer.py:803] 2025-04-26 17:24:49,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:49,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6337
[WARNING|trainer.py:803] 2025-04-26 17:24:50,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6405
6382
[WARNING|trainer.py:803] 2025-04-26 17:24:50,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:51,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6338
[WARNING|trainer.py:803] 2025-04-26 17:24:51,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6406
6383
[WARNING|trainer.py:803] 2025-04-26 17:24:52,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:52,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6339
[WARNING|trainer.py:803] 2025-04-26 17:24:52,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6407
6384
[WARNING|trainer.py:803] 2025-04-26 17:24:53,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6340
[WARNING|trainer.py:803] 2025-04-26 17:24:53,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:54,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6408
6385
[WARNING|trainer.py:803] 2025-04-26 17:24:54,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6341
[WARNING|trainer.py:803] 2025-04-26 17:24:55,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:55,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6409
6386
[WARNING|trainer.py:803] 2025-04-26 17:24:56,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6342
[WARNING|trainer.py:803] 2025-04-26 17:24:56,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:24:56,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6410
6387
[WARNING|trainer.py:803] 2025-04-26 17:24:57,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6343
[WARNING|trainer.py:803] 2025-04-26 17:24:57,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:24:58,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6411
6388
[WARNING|trainer.py:803] 2025-04-26 17:24:58,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6344
[WARNING|trainer.py:803] 2025-04-26 17:24:59,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:24:59,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6412
6389
[WARNING|trainer.py:803] 2025-04-26 17:25:00,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6345
[WARNING|trainer.py:803] 2025-04-26 17:25:00,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:00,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6413
6390
[WARNING|trainer.py:803] 2025-04-26 17:25:01,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6346
[WARNING|trainer.py:803] 2025-04-26 17:25:01,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:02,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6414
6391
[WARNING|trainer.py:803] 2025-04-26 17:25:02,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6347
[WARNING|trainer.py:803] 2025-04-26 17:25:03,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:03,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6415
6392
[WARNING|trainer.py:803] 2025-04-26 17:25:04,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:04,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6348
[WARNING|trainer.py:803] 2025-04-26 17:25:04,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6416
6393
[WARNING|trainer.py:803] 2025-04-26 17:25:05,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6349
[WARNING|trainer.py:803] 2025-04-26 17:25:05,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:06,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6417
6394
[WARNING|trainer.py:803] 2025-04-26 17:25:06,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6350
[WARNING|trainer.py:803] 2025-04-26 17:25:07,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:25:07,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6395
6418
[WARNING|trainer.py:803] 2025-04-26 17:25:08,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6351
[WARNING|trainer.py:803] 2025-04-26 17:25:08,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:08,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6396
6419
[WARNING|trainer.py:803] 2025-04-26 17:25:09,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6352
[WARNING|trainer.py:803] 2025-04-26 17:25:09,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:10,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6397
6420
[WARNING|trainer.py:803] 2025-04-26 17:25:10,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6353
[WARNING|trainer.py:803] 2025-04-26 17:25:11,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:11,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6398
6421
[WARNING|trainer.py:803] 2025-04-26 17:25:12,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6354
[WARNING|trainer.py:803] 2025-04-26 17:25:12,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:12,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6399
6422
[WARNING|trainer.py:803] 2025-04-26 17:25:13,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6355
[WARNING|trainer.py:803] 2025-04-26 17:25:13,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:14,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6400
6423
[WARNING|trainer.py:803] 2025-04-26 17:25:14,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6356
[WARNING|trainer.py:803] 2025-04-26 17:25:15,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:15,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6401
6424
[WARNING|trainer.py:803] 2025-04-26 17:25:15,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6357
[WARNING|trainer.py:803] 2025-04-26 17:25:16,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:16,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6402
6425
[WARNING|trainer.py:803] 2025-04-26 17:25:17,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6358
[WARNING|trainer.py:803] 2025-04-26 17:25:17,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:18,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6403
[WARNING|trainer.py:803] 2025-04-26 17:25:18,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6426
6359
[WARNING|trainer.py:803] 2025-04-26 17:25:19,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:19,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6404
[WARNING|trainer.py:803] 2025-04-26 17:25:19,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6427
6360
[WARNING|trainer.py:803] 2025-04-26 17:25:20,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:20,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6405
[WARNING|trainer.py:803] 2025-04-26 17:25:21,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6428
6361
[WARNING|trainer.py:803] 2025-04-26 17:25:21,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:25:22,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6406
[WARNING|trainer.py:803] 2025-04-26 17:25:22,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6429
6362
[WARNING|trainer.py:803] 2025-04-26 17:25:23,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:23,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6407
[WARNING|trainer.py:803] 2025-04-26 17:25:23,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6430
6363
[WARNING|trainer.py:803] 2025-04-26 17:25:24,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:24,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6408
[WARNING|trainer.py:803] 2025-04-26 17:25:25,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6431
6364
[WARNING|trainer.py:803] 2025-04-26 17:25:25,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:26,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6409
[WARNING|trainer.py:803] 2025-04-26 17:25:26,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6432
6365
[WARNING|trainer.py:803] 2025-04-26 17:25:27,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:25:27,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6410
[WARNING|trainer.py:803] 2025-04-26 17:25:27,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6433
6366
[WARNING|trainer.py:803] 2025-04-26 17:25:28,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:28,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6411
[WARNING|trainer.py:803] 2025-04-26 17:25:29,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6434
6367
[WARNING|trainer.py:803] 2025-04-26 17:25:29,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:30,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6412
[WARNING|trainer.py:803] 2025-04-26 17:25:30,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6435
6368
[WARNING|trainer.py:803] 2025-04-26 17:25:31,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:31,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6413
[WARNING|trainer.py:803] 2025-04-26 17:25:31,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6436
6369
[WARNING|trainer.py:803] 2025-04-26 17:25:32,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:32,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6414
[WARNING|trainer.py:803] 2025-04-26 17:25:33,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6437
6370
[WARNING|trainer.py:803] 2025-04-26 17:25:33,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:34,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6415
[WARNING|trainer.py:803] 2025-04-26 17:25:34,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6438
6371
[WARNING|trainer.py:803] 2025-04-26 17:25:35,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:35,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6416
[WARNING|trainer.py:803] 2025-04-26 17:25:35,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6439
6372
[WARNING|trainer.py:803] 2025-04-26 17:25:36,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:36,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6417
[WARNING|trainer.py:803] 2025-04-26 17:25:37,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6440
6373
[WARNING|trainer.py:803] 2025-04-26 17:25:37,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:25:38,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6418
[WARNING|trainer.py:803] 2025-04-26 17:25:38,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6441
6374
[WARNING|trainer.py:803] 2025-04-26 17:25:39,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:39,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6419
[WARNING|trainer.py:803] 2025-04-26 17:25:39,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6442
6375
[WARNING|trainer.py:803] 2025-04-26 17:25:40,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:40,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:40,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6420
6443
6376
[WARNING|trainer.py:803] 2025-04-26 17:25:41,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:42,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:42,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6421
6444
6377
[WARNING|trainer.py:803] 2025-04-26 17:25:43,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:43,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:43,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6422
6445
6378
[WARNING|trainer.py:803] 2025-04-26 17:25:44,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:44,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6423
[WARNING|trainer.py:803] 2025-04-26 17:25:44,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6446
6379
[WARNING|trainer.py:803] 2025-04-26 17:25:45,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:46,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:46,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6424
6447
6380
[WARNING|trainer.py:803] 2025-04-26 17:25:47,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:47,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:47,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6425
6448
6381
[WARNING|trainer.py:803] 2025-04-26 17:25:48,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:48,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:48,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6426
6449
6382
[WARNING|trainer.py:803] 2025-04-26 17:25:49,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:25:50,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:50,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6427
6450
6383
[WARNING|trainer.py:803] 2025-04-26 17:25:51,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:51,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6428
[WARNING|trainer.py:803] 2025-04-26 17:25:51,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6451
6384
[WARNING|trainer.py:803] 2025-04-26 17:25:52,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:52,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6429
[WARNING|trainer.py:803] 2025-04-26 17:25:53,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6452
6385
[WARNING|trainer.py:803] 2025-04-26 17:25:53,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:54,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6430
[WARNING|trainer.py:803] 2025-04-26 17:25:54,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6453
6386
[WARNING|trainer.py:803] 2025-04-26 17:25:55,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:25:55,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6431
[WARNING|trainer.py:803] 2025-04-26 17:25:55,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6454
6387
[WARNING|trainer.py:803] 2025-04-26 17:25:56,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:25:56,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6432
[WARNING|trainer.py:803] 2025-04-26 17:25:56,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6455
6388
[WARNING|trainer.py:803] 2025-04-26 17:25:57,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:25:58,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6433
[WARNING|trainer.py:803] 2025-04-26 17:25:58,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6456
6389
[WARNING|trainer.py:803] 2025-04-26 17:25:59,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:25:59,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6434
[WARNING|trainer.py:803] 2025-04-26 17:25:59,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6457
6390
[WARNING|trainer.py:803] 2025-04-26 17:26:00,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:00,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6435
[WARNING|trainer.py:803] 2025-04-26 17:26:00,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6458
6391
[WARNING|trainer.py:803] 2025-04-26 17:26:01,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6436
[WARNING|trainer.py:803] 2025-04-26 17:26:02,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:02,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6459
6392
[WARNING|trainer.py:803] 2025-04-26 17:26:02,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6437
[WARNING|trainer.py:803] 2025-04-26 17:26:03,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:03,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6460
6393
[WARNING|trainer.py:803] 2025-04-26 17:26:04,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6438
[WARNING|trainer.py:803] 2025-04-26 17:26:04,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:26:04,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6461
6394
[WARNING|trainer.py:803] 2025-04-26 17:26:05,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6439
[WARNING|trainer.py:803] 2025-04-26 17:26:06,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:06,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6462
6395
[WARNING|trainer.py:803] 2025-04-26 17:26:06,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6440
[WARNING|trainer.py:803] 2025-04-26 17:26:07,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:07,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6463
6396
[WARNING|trainer.py:803] 2025-04-26 17:26:08,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6441
[WARNING|trainer.py:803] 2025-04-26 17:26:08,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:08,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6397
6464
[WARNING|trainer.py:803] 2025-04-26 17:26:09,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6442
[WARNING|trainer.py:803] 2025-04-26 17:26:10,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:10,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6465
6398
[WARNING|trainer.py:803] 2025-04-26 17:26:10,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6443
[WARNING|trainer.py:803] 2025-04-26 17:26:11,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:11,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6399
6466
[WARNING|trainer.py:803] 2025-04-26 17:26:12,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6444
[WARNING|trainer.py:803] 2025-04-26 17:26:12,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:12,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6400
6467
[WARNING|trainer.py:803] 2025-04-26 17:26:13,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6445
[WARNING|trainer.py:803] 2025-04-26 17:26:14,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:14,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6401
[WARNING|trainer.py:803] 2025-04-26 17:26:14,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6468
6446
[WARNING|trainer.py:803] 2025-04-26 17:26:15,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:15,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6402
[WARNING|trainer.py:803] 2025-04-26 17:26:16,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6469
6447
[WARNING|trainer.py:803] 2025-04-26 17:26:16,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:17,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6403
[WARNING|trainer.py:803] 2025-04-26 17:26:17,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6470
6448
[WARNING|trainer.py:803] 2025-04-26 17:26:18,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:18,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6404
[WARNING|trainer.py:803] 2025-04-26 17:26:18,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6471
6449
[WARNING|trainer.py:803] 2025-04-26 17:26:19,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:19,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6405
6472
[WARNING|trainer.py:803] 2025-04-26 17:26:20,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6450
[WARNING|trainer.py:803] 2025-04-26 17:26:20,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:26:21,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6406
6473
[WARNING|trainer.py:803] 2025-04-26 17:26:21,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6451
[WARNING|trainer.py:803] 2025-04-26 17:26:22,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:22,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6407
6474
[WARNING|trainer.py:803] 2025-04-26 17:26:22,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6452
[WARNING|trainer.py:803] 2025-04-26 17:26:23,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:23,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6475
6408
[WARNING|trainer.py:803] 2025-04-26 17:26:24,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6453
[WARNING|trainer.py:803] 2025-04-26 17:26:24,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:25,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6409
6476
[WARNING|trainer.py:803] 2025-04-26 17:26:25,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6454
[WARNING|trainer.py:803] 2025-04-26 17:26:26,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:26,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6410
6477
[WARNING|trainer.py:803] 2025-04-26 17:26:26,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6455
[WARNING|trainer.py:803] 2025-04-26 17:26:27,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:27,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6478
6411
[WARNING|trainer.py:803] 2025-04-26 17:26:28,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6456
[WARNING|trainer.py:803] 2025-04-26 17:26:28,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:28,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6412
6479
[WARNING|trainer.py:803] 2025-04-26 17:26:29,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6457
[WARNING|trainer.py:803] 2025-04-26 17:26:30,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:30,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6413
6480
[WARNING|trainer.py:803] 2025-04-26 17:26:30,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6458
[WARNING|trainer.py:803] 2025-04-26 17:26:31,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:31,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6414
6481
[WARNING|trainer.py:803] 2025-04-26 17:26:32,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6459
[WARNING|trainer.py:803] 2025-04-26 17:26:32,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:33,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6415
6482
[WARNING|trainer.py:803] 2025-04-26 17:26:33,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6460
[WARNING|trainer.py:803] 2025-04-26 17:26:34,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:34,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6416
6483
[WARNING|trainer.py:803] 2025-04-26 17:26:35,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6461
[WARNING|trainer.py:803] 2025-04-26 17:26:35,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:35,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6417
[WARNING|trainer.py:803] 2025-04-26 17:26:36,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6484
6462
[WARNING|trainer.py:803] 2025-04-26 17:26:36,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:26:37,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6418
[WARNING|trainer.py:803] 2025-04-26 17:26:37,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6485
6463
[WARNING|trainer.py:803] 2025-04-26 17:26:38,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:38,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6419
[WARNING|trainer.py:803] 2025-04-26 17:26:39,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6486
6464
[WARNING|trainer.py:803] 2025-04-26 17:26:39,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:39,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6420
6487
[WARNING|trainer.py:803] 2025-04-26 17:26:40,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6465
[WARNING|trainer.py:803] 2025-04-26 17:26:40,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:41,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6421
[WARNING|trainer.py:803] 2025-04-26 17:26:41,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6488
6466
[WARNING|trainer.py:803] 2025-04-26 17:26:42,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:42,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6422
[WARNING|trainer.py:803] 2025-04-26 17:26:43,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6489
[WARNING|trainer.py:803] 2025-04-26 17:26:43,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6467
[WARNING|trainer.py:803] 2025-04-26 17:26:43,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6423
6490
[WARNING|trainer.py:803] 2025-04-26 17:26:44,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:44,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6468
[WARNING|trainer.py:803] 2025-04-26 17:26:45,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6424
6491
[WARNING|trainer.py:803] 2025-04-26 17:26:45,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:46,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6469
[WARNING|trainer.py:803] 2025-04-26 17:26:46,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6425
6492
[WARNING|trainer.py:803] 2025-04-26 17:26:47,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:47,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6470
[WARNING|trainer.py:803] 2025-04-26 17:26:47,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6426
6493
[WARNING|trainer.py:803] 2025-04-26 17:26:48,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:48,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6471
[WARNING|trainer.py:803] 2025-04-26 17:26:49,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6427
6494
[WARNING|trainer.py:803] 2025-04-26 17:26:49,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:50,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6472
[WARNING|trainer.py:803] 2025-04-26 17:26:50,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6428
6495
[WARNING|trainer.py:803] 2025-04-26 17:26:51,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:51,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6473
[WARNING|trainer.py:803] 2025-04-26 17:26:51,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6429
6496
[WARNING|trainer.py:803] 2025-04-26 17:26:52,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:52,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6474
[WARNING|trainer.py:803] 2025-04-26 17:26:53,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6430
6497
[WARNING|trainer.py:803] 2025-04-26 17:26:53,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:26:54,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6475
[WARNING|trainer.py:803] 2025-04-26 17:26:54,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6431
6498
[WARNING|trainer.py:803] 2025-04-26 17:26:55,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:55,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6476
[WARNING|trainer.py:803] 2025-04-26 17:26:55,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6432
6499
[WARNING|trainer.py:803] 2025-04-26 17:26:56,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:26:56,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6477
[WARNING|trainer.py:803] 2025-04-26 17:26:57,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6433
6500
[WARNING|trainer.py:803] 2025-04-26 17:26:57,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:26:58,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6478
[WARNING|trainer.py:803] 2025-04-26 17:26:58,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6434
6501
[WARNING|trainer.py:803] 2025-04-26 17:26:59,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:26:59,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6479
[WARNING|trainer.py:803] 2025-04-26 17:26:59,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6435
6502
[WARNING|trainer.py:803] 2025-04-26 17:27:00,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:00,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6480
[WARNING|trainer.py:803] 2025-04-26 17:27:01,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6436
6503
[WARNING|trainer.py:803] 2025-04-26 17:27:01,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:02,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6481
[WARNING|trainer.py:803] 2025-04-26 17:27:02,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6437
6504
[WARNING|trainer.py:803] 2025-04-26 17:27:03,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:03,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6482
[WARNING|trainer.py:803] 2025-04-26 17:27:03,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6438
6505
[WARNING|trainer.py:803] 2025-04-26 17:27:04,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:04,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6483
[WARNING|trainer.py:803] 2025-04-26 17:27:04,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6439
6506
[WARNING|trainer.py:803] 2025-04-26 17:27:05,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:06,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6484
[WARNING|trainer.py:803] 2025-04-26 17:27:06,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6440
6507
[WARNING|trainer.py:803] 2025-04-26 17:27:07,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:07,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6485
[WARNING|trainer.py:803] 2025-04-26 17:27:07,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6441
6508
[WARNING|trainer.py:803] 2025-04-26 17:27:08,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:08,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:08,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6486
6442
6509
[WARNING|trainer.py:803] 2025-04-26 17:27:09,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:10,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:10,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6487
6443
6510
[WARNING|trainer.py:803] 2025-04-26 17:27:11,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:11,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:11,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6488
6444
6511
[WARNING|trainer.py:803] 2025-04-26 17:27:12,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:12,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:12,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6489
6445
6512
[WARNING|trainer.py:803] 2025-04-26 17:27:13,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:14,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:14,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6490
6446
6513
[WARNING|trainer.py:803] 2025-04-26 17:27:15,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:15,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:15,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6491
6447
6514
[WARNING|trainer.py:803] 2025-04-26 17:27:16,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:16,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:16,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6492
6515
6448
[WARNING|trainer.py:803] 2025-04-26 17:27:17,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:18,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:18,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6493
6516
6449
[WARNING|trainer.py:803] 2025-04-26 17:27:19,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:19,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:19,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6494
6517
6450
[WARNING|trainer.py:803] 2025-04-26 17:27:20,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:20,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:20,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6495
6518
6451
[WARNING|trainer.py:803] 2025-04-26 17:27:21,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:22,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:22,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6496
6519
6452
[WARNING|trainer.py:803] 2025-04-26 17:27:23,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:23,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:23,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6497
6520
6453
[WARNING|trainer.py:803] 2025-04-26 17:27:24,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:24,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:24,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6498
6521
6454
[WARNING|trainer.py:803] 2025-04-26 17:27:25,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:26,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:26,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6499
6522
6455
[WARNING|trainer.py:803] 2025-04-26 17:27:27,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:27,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:27,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6500
6523
6456
[WARNING|trainer.py:803] 2025-04-26 17:27:28,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:28,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:28,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6501
6524
6457
[WARNING|trainer.py:803] 2025-04-26 17:27:29,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:30,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:30,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6502
6525
6458
[WARNING|trainer.py:803] 2025-04-26 17:27:31,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:31,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:31,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6503
6526
6459
[WARNING|trainer.py:803] 2025-04-26 17:27:32,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:32,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6504
[WARNING|trainer.py:803] 2025-04-26 17:27:33,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6527
6460
[WARNING|trainer.py:803] 2025-04-26 17:27:33,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:34,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6505
[WARNING|trainer.py:803] 2025-04-26 17:27:34,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6528
6461
[WARNING|trainer.py:803] 2025-04-26 17:27:35,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:35,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:35,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6506
6529
6462
[WARNING|trainer.py:803] 2025-04-26 17:27:36,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:36,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:37,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6507
6530
6463
[WARNING|trainer.py:803] 2025-04-26 17:27:37,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:37,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:38,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6508
6531
6464
[WARNING|trainer.py:803] 2025-04-26 17:27:39,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:39,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:39,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6509
6532
6465
[WARNING|trainer.py:803] 2025-04-26 17:27:40,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:40,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6533
[WARNING|trainer.py:803] 2025-04-26 17:27:41,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6510
6466
[WARNING|trainer.py:803] 2025-04-26 17:27:41,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:41,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6534
[WARNING|trainer.py:803] 2025-04-26 17:27:42,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6511
6467
[WARNING|trainer.py:803] 2025-04-26 17:27:43,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:43,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6535
[WARNING|trainer.py:803] 2025-04-26 17:27:43,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6512
6468
[WARNING|trainer.py:803] 2025-04-26 17:27:44,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:44,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6536
[WARNING|trainer.py:803] 2025-04-26 17:27:45,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6513
6469
[WARNING|trainer.py:803] 2025-04-26 17:27:45,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:45,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6537
[WARNING|trainer.py:803] 2025-04-26 17:27:46,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6514
6470
[WARNING|trainer.py:803] 2025-04-26 17:27:47,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:47,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6538
[WARNING|trainer.py:803] 2025-04-26 17:27:47,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6515
6471
[WARNING|trainer.py:803] 2025-04-26 17:27:48,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:48,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6539
[WARNING|trainer.py:803] 2025-04-26 17:27:49,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6516
6472
[WARNING|trainer.py:803] 2025-04-26 17:27:49,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:27:49,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6540
[WARNING|trainer.py:803] 2025-04-26 17:27:50,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6517
6473
[WARNING|trainer.py:803] 2025-04-26 17:27:50,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:27:51,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6541
[WARNING|trainer.py:803] 2025-04-26 17:27:51,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6518
6474
[WARNING|trainer.py:803] 2025-04-26 17:27:52,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:27:52,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6542
[WARNING|trainer.py:803] 2025-04-26 17:27:53,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6519
6475
[WARNING|trainer.py:803] 2025-04-26 17:27:53,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:53,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6543
[WARNING|trainer.py:803] 2025-04-26 17:27:54,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6520
6476
[WARNING|trainer.py:803] 2025-04-26 17:27:54,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6544
[WARNING|trainer.py:803] 2025-04-26 17:27:55,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:55,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6521
[WARNING|trainer.py:803] 2025-04-26 17:27:56,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6477
6545
[WARNING|trainer.py:803] 2025-04-26 17:27:56,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:56,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6522
[WARNING|trainer.py:803] 2025-04-26 17:27:57,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6478
6546
[WARNING|trainer.py:803] 2025-04-26 17:27:57,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:58,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6523
[WARNING|trainer.py:803] 2025-04-26 17:27:58,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6479
6547
[WARNING|trainer.py:803] 2025-04-26 17:27:59,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:27:59,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6524
[WARNING|trainer.py:803] 2025-04-26 17:28:00,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6480
6548
[WARNING|trainer.py:803] 2025-04-26 17:28:00,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:00,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6525
[WARNING|trainer.py:803] 2025-04-26 17:28:01,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6481
6549
[WARNING|trainer.py:803] 2025-04-26 17:28:01,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:02,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6526
[WARNING|trainer.py:803] 2025-04-26 17:28:02,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6482
6550
[WARNING|trainer.py:803] 2025-04-26 17:28:03,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:03,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6527
[WARNING|trainer.py:803] 2025-04-26 17:28:04,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6483
6551
[WARNING|trainer.py:803] 2025-04-26 17:28:04,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:04,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6528
[WARNING|trainer.py:803] 2025-04-26 17:28:05,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6484
6552
[WARNING|trainer.py:803] 2025-04-26 17:28:05,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:06,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6529
[WARNING|trainer.py:803] 2025-04-26 17:28:06,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6485
6553
[WARNING|trainer.py:803] 2025-04-26 17:28:07,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:07,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6530
[WARNING|trainer.py:803] 2025-04-26 17:28:07,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6486
6554
[WARNING|trainer.py:803] 2025-04-26 17:28:08,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:08,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6531
[WARNING|trainer.py:803] 2025-04-26 17:28:09,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6487
6555
[WARNING|trainer.py:803] 2025-04-26 17:28:09,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:10,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6532
[WARNING|trainer.py:803] 2025-04-26 17:28:10,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6488
[WARNING|trainer.py:803] 2025-04-26 17:28:11,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6556
6533
[WARNING|trainer.py:803] 2025-04-26 17:28:11,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:11,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6489
[WARNING|trainer.py:803] 2025-04-26 17:28:12,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6557
6534
[WARNING|trainer.py:803] 2025-04-26 17:28:12,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:13,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6490
[WARNING|trainer.py:803] 2025-04-26 17:28:13,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6558
6535
[WARNING|trainer.py:803] 2025-04-26 17:28:14,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:14,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6491
[WARNING|trainer.py:803] 2025-04-26 17:28:15,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6559
6536
[WARNING|trainer.py:803] 2025-04-26 17:28:15,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:15,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6492
[WARNING|trainer.py:803] 2025-04-26 17:28:16,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6560
6537
[WARNING|trainer.py:803] 2025-04-26 17:28:16,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:17,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6493
[WARNING|trainer.py:803] 2025-04-26 17:28:17,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6561
6538
[WARNING|trainer.py:803] 2025-04-26 17:28:18,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:18,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6494
[WARNING|trainer.py:803] 2025-04-26 17:28:18,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6562
6539
[WARNING|trainer.py:803] 2025-04-26 17:28:19,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:19,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6495
[WARNING|trainer.py:803] 2025-04-26 17:28:20,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6563
6540
[WARNING|trainer.py:803] 2025-04-26 17:28:20,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:21,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6496
[WARNING|trainer.py:803] 2025-04-26 17:28:21,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6564
6541
[WARNING|trainer.py:803] 2025-04-26 17:28:22,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:22,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6497
[WARNING|trainer.py:803] 2025-04-26 17:28:22,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6565
6542
[WARNING|trainer.py:803] 2025-04-26 17:28:23,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:23,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6498
[WARNING|trainer.py:803] 2025-04-26 17:28:24,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6566
6543
[WARNING|trainer.py:803] 2025-04-26 17:28:24,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:25,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6499
[WARNING|trainer.py:803] 2025-04-26 17:28:25,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6567
6544
[WARNING|trainer.py:803] 2025-04-26 17:28:26,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:26,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6500
[WARNING|trainer.py:803] 2025-04-26 17:28:26,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6568
6545
[WARNING|trainer.py:803] 2025-04-26 17:28:27,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:27,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6501
[WARNING|trainer.py:803] 2025-04-26 17:28:28,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6569
6546
[WARNING|trainer.py:803] 2025-04-26 17:28:28,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:29,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6502
[WARNING|trainer.py:803] 2025-04-26 17:28:29,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6570
6547
[WARNING|trainer.py:803] 2025-04-26 17:28:30,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:30,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6503
[WARNING|trainer.py:803] 2025-04-26 17:28:30,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6571
6548
[WARNING|trainer.py:803] 2025-04-26 17:28:31,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:31,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6504
[WARNING|trainer.py:803] 2025-04-26 17:28:32,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6572
6549
[WARNING|trainer.py:803] 2025-04-26 17:28:32,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:33,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6505
[WARNING|trainer.py:803] 2025-04-26 17:28:33,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6573
6550
[WARNING|trainer.py:803] 2025-04-26 17:28:34,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:34,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6506
[WARNING|trainer.py:803] 2025-04-26 17:28:34,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6574
6551
[WARNING|trainer.py:803] 2025-04-26 17:28:35,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6507
[WARNING|trainer.py:803] 2025-04-26 17:28:35,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:36,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6575
6552
[WARNING|trainer.py:803] 2025-04-26 17:28:36,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6508
[WARNING|trainer.py:803] 2025-04-26 17:28:37,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:37,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6576
6553
[WARNING|trainer.py:803] 2025-04-26 17:28:37,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6509
[WARNING|trainer.py:803] 2025-04-26 17:28:38,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:38,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6577
6554
[WARNING|trainer.py:803] 2025-04-26 17:28:39,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6510
[WARNING|trainer.py:803] 2025-04-26 17:28:39,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:40,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6578
6555
[WARNING|trainer.py:803] 2025-04-26 17:28:40,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6511
[WARNING|trainer.py:803] 2025-04-26 17:28:41,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:41,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6579
6556
[WARNING|trainer.py:803] 2025-04-26 17:28:41,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6512
[WARNING|trainer.py:803] 2025-04-26 17:28:42,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:42,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6580
[WARNING|trainer.py:803] 2025-04-26 17:28:43,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6557
6513
[WARNING|trainer.py:803] 2025-04-26 17:28:43,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:44,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6581
[WARNING|trainer.py:803] 2025-04-26 17:28:44,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6558
6514
[WARNING|trainer.py:803] 2025-04-26 17:28:45,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:45,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6582
[WARNING|trainer.py:803] 2025-04-26 17:28:45,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6559
6515
[WARNING|trainer.py:803] 2025-04-26 17:28:46,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:46,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6583
[WARNING|trainer.py:803] 2025-04-26 17:28:47,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6560
6516
[WARNING|trainer.py:803] 2025-04-26 17:28:47,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:48,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6584
[WARNING|trainer.py:803] 2025-04-26 17:28:48,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6561
6517
[WARNING|trainer.py:803] 2025-04-26 17:28:49,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:49,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6585
[WARNING|trainer.py:803] 2025-04-26 17:28:49,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6562
6518
[WARNING|trainer.py:803] 2025-04-26 17:28:50,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:50,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:28:51,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6586
6563
6519
[WARNING|trainer.py:803] 2025-04-26 17:28:51,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:52,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:52,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6587
6564
6520
[WARNING|trainer.py:803] 2025-04-26 17:28:53,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:53,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6588
[WARNING|trainer.py:803] 2025-04-26 17:28:53,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6565
6521
[WARNING|trainer.py:803] 2025-04-26 17:28:54,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:54,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6589
[WARNING|trainer.py:803] 2025-04-26 17:28:55,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6566
6522
[WARNING|trainer.py:803] 2025-04-26 17:28:55,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:56,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:56,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6590
6567
6523
[WARNING|trainer.py:803] 2025-04-26 17:28:57,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:28:57,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:28:57,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6591
6568
6524
[WARNING|trainer.py:803] 2025-04-26 17:28:58,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:58,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:28:59,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6592
6569
6525
[WARNING|trainer.py:803] 2025-04-26 17:28:59,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:00,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:00,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6593
6570
6526
[WARNING|trainer.py:803] 2025-04-26 17:29:01,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:01,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:01,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6594
6571
6527
[WARNING|trainer.py:803] 2025-04-26 17:29:02,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:02,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:03,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6595
6572
6528
[WARNING|trainer.py:803] 2025-04-26 17:29:03,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:04,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:04,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6596
6573
6529
[WARNING|trainer.py:803] 2025-04-26 17:29:05,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:05,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:05,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6597
6574
6530
[WARNING|trainer.py:803] 2025-04-26 17:29:06,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:06,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:06,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6598
6575
6531
[WARNING|trainer.py:803] 2025-04-26 17:29:07,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:08,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:08,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6599
6576
6532
[WARNING|trainer.py:803] 2025-04-26 17:29:09,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:09,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:09,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6600
6577
6533
[WARNING|trainer.py:803] 2025-04-26 17:29:10,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:10,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:10,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6601
6578
6534
[WARNING|trainer.py:803] 2025-04-26 17:29:11,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:12,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:12,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6602
6579
6535
[WARNING|trainer.py:803] 2025-04-26 17:29:13,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:13,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:13,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6603
6580
6536
[WARNING|trainer.py:803] 2025-04-26 17:29:14,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:14,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:14,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6604
6581
6537
[WARNING|trainer.py:803] 2025-04-26 17:29:15,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:16,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:16,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6605
6538
6582
[WARNING|trainer.py:803] 2025-04-26 17:29:17,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:17,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:17,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6606
6539
6583
[WARNING|trainer.py:803] 2025-04-26 17:29:18,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:18,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:18,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6607
6540
6584
[WARNING|trainer.py:803] 2025-04-26 17:29:19,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:20,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:20,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
6541
6585
[WARNING|trainer.py:803] 2025-04-26 17:29:21,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:21,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:21,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6609
6542
6586
[WARNING|trainer.py:803] 2025-04-26 17:29:22,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:22,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:22,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6610
6543
6587
[WARNING|trainer.py:803] 2025-04-26 17:29:23,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:24,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6611
[WARNING|trainer.py:803] 2025-04-26 17:29:24,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6544
6588
[WARNING|trainer.py:803] 2025-04-26 17:29:25,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:25,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6612
[WARNING|trainer.py:803] 2025-04-26 17:29:25,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6545
6589
[WARNING|trainer.py:803] 2025-04-26 17:29:26,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:26,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6613
[WARNING|trainer.py:803] 2025-04-26 17:29:26,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6546
6590
[WARNING|trainer.py:803] 2025-04-26 17:29:27,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:28,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6614
[WARNING|trainer.py:803] 2025-04-26 17:29:28,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6547
6591
[WARNING|trainer.py:803] 2025-04-26 17:29:28,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:29,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6615
[WARNING|trainer.py:803] 2025-04-26 17:29:29,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6548
6592
[WARNING|trainer.py:803] 2025-04-26 17:29:30,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:30,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6616
[WARNING|trainer.py:803] 2025-04-26 17:29:30,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6549
6593
[WARNING|trainer.py:803] 2025-04-26 17:29:31,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:29:31,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6617
[WARNING|trainer.py:803] 2025-04-26 17:29:32,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6550
6594
[WARNING|trainer.py:803] 2025-04-26 17:29:32,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:33,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6618
[WARNING|trainer.py:803] 2025-04-26 17:29:33,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6551
6595
[WARNING|trainer.py:803] 2025-04-26 17:29:34,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:34,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6619
[WARNING|trainer.py:803] 2025-04-26 17:29:34,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6552
[WARNING|trainer.py:803] 2025-04-26 17:29:35,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6596
6620
[WARNING|trainer.py:803] 2025-04-26 17:29:36,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:36,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6553
[WARNING|trainer.py:803] 2025-04-26 17:29:36,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6597
6621
[WARNING|trainer.py:803] 2025-04-26 17:29:37,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:37,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6554
[WARNING|trainer.py:803] 2025-04-26 17:29:38,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6598
6622
[WARNING|trainer.py:803] 2025-04-26 17:29:38,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:38,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6555
[WARNING|trainer.py:803] 2025-04-26 17:29:39,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6599
6623
[WARNING|trainer.py:803] 2025-04-26 17:29:39,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:40,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6556
[WARNING|trainer.py:803] 2025-04-26 17:29:40,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6600
6624
[WARNING|trainer.py:803] 2025-04-26 17:29:41,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:41,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6557
[WARNING|trainer.py:803] 2025-04-26 17:29:41,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6601
6625
[WARNING|trainer.py:803] 2025-04-26 17:29:42,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:42,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6558
[WARNING|trainer.py:803] 2025-04-26 17:29:43,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6602
6626
[WARNING|trainer.py:803] 2025-04-26 17:29:43,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:44,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6559
[WARNING|trainer.py:803] 2025-04-26 17:29:44,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6603
6627
[WARNING|trainer.py:803] 2025-04-26 17:29:45,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:45,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6560
[WARNING|trainer.py:803] 2025-04-26 17:29:45,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6604
6628
[WARNING|trainer.py:803] 2025-04-26 17:29:46,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:46,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6561
[WARNING|trainer.py:803] 2025-04-26 17:29:47,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6605
6629
[WARNING|trainer.py:803] 2025-04-26 17:29:47,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:48,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6562
[WARNING|trainer.py:803] 2025-04-26 17:29:48,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6606
6630
[WARNING|trainer.py:803] 2025-04-26 17:29:49,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:29:49,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6563
[WARNING|trainer.py:803] 2025-04-26 17:29:49,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6607
6631
[WARNING|trainer.py:803] 2025-04-26 17:29:50,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:50,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6564
[WARNING|trainer.py:803] 2025-04-26 17:29:51,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
6632
[WARNING|trainer.py:803] 2025-04-26 17:29:51,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6565
[WARNING|trainer.py:803] 2025-04-26 17:29:52,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:52,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6609
6633
[WARNING|trainer.py:803] 2025-04-26 17:29:53,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6566
[WARNING|trainer.py:803] 2025-04-26 17:29:53,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:53,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6610
6634
[WARNING|trainer.py:803] 2025-04-26 17:29:54,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6567
[WARNING|trainer.py:803] 2025-04-26 17:29:54,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:55,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6611
[WARNING|trainer.py:803] 2025-04-26 17:29:55,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6635
6568
[WARNING|trainer.py:803] 2025-04-26 17:29:56,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:56,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6612
[WARNING|trainer.py:803] 2025-04-26 17:29:56,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6636
6569
[WARNING|trainer.py:803] 2025-04-26 17:29:57,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:29:57,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6613
[WARNING|trainer.py:803] 2025-04-26 17:29:58,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6637
6570
[WARNING|trainer.py:803] 2025-04-26 17:29:58,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:29:59,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6614
[WARNING|trainer.py:803] 2025-04-26 17:29:59,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6638
6571
[WARNING|trainer.py:803] 2025-04-26 17:30:00,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:00,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6615
[WARNING|trainer.py:803] 2025-04-26 17:30:00,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6639
6572
[WARNING|trainer.py:803] 2025-04-26 17:30:01,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:01,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6616
[WARNING|trainer.py:803] 2025-04-26 17:30:02,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6640
6573
[WARNING|trainer.py:803] 2025-04-26 17:30:02,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:03,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6617
[WARNING|trainer.py:803] 2025-04-26 17:30:03,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6641
6574
[WARNING|trainer.py:803] 2025-04-26 17:30:04,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:04,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6618
[WARNING|trainer.py:803] 2025-04-26 17:30:04,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6642
6575
[WARNING|trainer.py:803] 2025-04-26 17:30:05,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:05,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6619
[WARNING|trainer.py:803] 2025-04-26 17:30:06,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6643
6576
[WARNING|trainer.py:803] 2025-04-26 17:30:06,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:07,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6620
[WARNING|trainer.py:803] 2025-04-26 17:30:07,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6644
6577
[WARNING|trainer.py:803] 2025-04-26 17:30:08,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:08,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6621
[WARNING|trainer.py:803] 2025-04-26 17:30:08,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6645
6578
[WARNING|trainer.py:803] 2025-04-26 17:30:09,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:09,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6622
[WARNING|trainer.py:803] 2025-04-26 17:30:10,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6646
6579
[WARNING|trainer.py:803] 2025-04-26 17:30:10,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:11,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6623
[WARNING|trainer.py:803] 2025-04-26 17:30:11,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6647
6580
[WARNING|trainer.py:803] 2025-04-26 17:30:12,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:12,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6624
[WARNING|trainer.py:803] 2025-04-26 17:30:12,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6648
6581
[WARNING|trainer.py:803] 2025-04-26 17:30:13,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:13,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6625
[WARNING|trainer.py:803] 2025-04-26 17:30:14,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6649
6582
[WARNING|trainer.py:803] 2025-04-26 17:30:14,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:15,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6626
[WARNING|trainer.py:803] 2025-04-26 17:30:15,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6650
6583
[WARNING|trainer.py:803] 2025-04-26 17:30:16,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:16,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6627
[WARNING|trainer.py:803] 2025-04-26 17:30:16,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6651
6584
[WARNING|trainer.py:803] 2025-04-26 17:30:17,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:17,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6628
[WARNING|trainer.py:803] 2025-04-26 17:30:18,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6652
6585
[WARNING|trainer.py:803] 2025-04-26 17:30:18,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:19,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6629
[WARNING|trainer.py:803] 2025-04-26 17:30:19,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6653
6586
[WARNING|trainer.py:803] 2025-04-26 17:30:20,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:20,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6630
[WARNING|trainer.py:803] 2025-04-26 17:30:20,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6654
6587
[WARNING|trainer.py:803] 2025-04-26 17:30:21,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:30:21,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6631
[WARNING|trainer.py:803] 2025-04-26 17:30:22,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6655
6588
[WARNING|trainer.py:803] 2025-04-26 17:30:22,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:23,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6632
[WARNING|trainer.py:803] 2025-04-26 17:30:23,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6656
6589
[WARNING|trainer.py:803] 2025-04-26 17:30:23,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:24,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6633
[WARNING|trainer.py:803] 2025-04-26 17:30:24,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6657
[WARNING|trainer.py:803] 2025-04-26 17:30:25,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6590
[WARNING|trainer.py:803] 2025-04-26 17:30:25,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6634
6658
[WARNING|trainer.py:803] 2025-04-26 17:30:26,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:26,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6591
[WARNING|trainer.py:803] 2025-04-26 17:30:27,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6635
6659
[WARNING|trainer.py:803] 2025-04-26 17:30:27,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:27,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6592
[WARNING|trainer.py:803] 2025-04-26 17:30:28,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6636
[WARNING|trainer.py:803] 2025-04-26 17:30:28,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6660
[WARNING|trainer.py:803] 2025-04-26 17:30:29,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6593
[WARNING|trainer.py:803] 2025-04-26 17:30:29,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6637
[WARNING|trainer.py:803] 2025-04-26 17:30:30,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6661
[WARNING|trainer.py:803] 2025-04-26 17:30:30,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6594
[WARNING|trainer.py:803] 2025-04-26 17:30:30,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6638
[WARNING|trainer.py:803] 2025-04-26 17:30:31,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6662
[WARNING|trainer.py:803] 2025-04-26 17:30:31,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6595
[WARNING|trainer.py:803] 2025-04-26 17:30:32,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6639
[WARNING|trainer.py:803] 2025-04-26 17:30:32,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6663
[WARNING|trainer.py:803] 2025-04-26 17:30:33,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6596
[WARNING|trainer.py:803] 2025-04-26 17:30:33,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6640
[WARNING|trainer.py:803] 2025-04-26 17:30:34,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6664
[WARNING|trainer.py:803] 2025-04-26 17:30:34,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6597
[WARNING|trainer.py:803] 2025-04-26 17:30:34,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
6641
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:35,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6665
[WARNING|trainer.py:803] 2025-04-26 17:30:35,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6598
6642
[WARNING|trainer.py:803] 2025-04-26 17:30:36,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:36,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6666
[WARNING|trainer.py:803] 2025-04-26 17:30:37,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6599
[WARNING|trainer.py:803] 2025-04-26 17:30:37,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6643
[WARNING|trainer.py:803] 2025-04-26 17:30:37,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6667
[WARNING|trainer.py:803] 2025-04-26 17:30:38,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6600
[WARNING|trainer.py:803] 2025-04-26 17:30:38,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6644
[WARNING|trainer.py:803] 2025-04-26 17:30:39,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6668
[WARNING|trainer.py:803] 2025-04-26 17:30:39,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6601
6645
[WARNING|trainer.py:803] 2025-04-26 17:30:40,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:40,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6669
[WARNING|trainer.py:803] 2025-04-26 17:30:41,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6602
6646
[WARNING|trainer.py:803] 2025-04-26 17:30:41,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:30:41,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6670
[WARNING|trainer.py:803] 2025-04-26 17:30:42,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6603
6647
[WARNING|trainer.py:803] 2025-04-26 17:30:42,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:43,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6671
[WARNING|trainer.py:803] 2025-04-26 17:30:43,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6604
6648
[WARNING|trainer.py:803] 2025-04-26 17:30:44,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:44,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6672
[WARNING|trainer.py:803] 2025-04-26 17:30:44,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6605
6649
[WARNING|trainer.py:803] 2025-04-26 17:30:45,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:45,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6673
[WARNING|trainer.py:803] 2025-04-26 17:30:46,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6606
6650
[WARNING|trainer.py:803] 2025-04-26 17:30:46,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:30:47,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6674
[WARNING|trainer.py:803] 2025-04-26 17:30:47,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6607
[WARNING|trainer.py:803] 2025-04-26 17:30:48,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6651
[WARNING|trainer.py:803] 2025-04-26 17:30:48,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6675
[WARNING|trainer.py:803] 2025-04-26 17:30:48,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
6652
[WARNING|trainer.py:803] 2025-04-26 17:30:49,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:49,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6676
[WARNING|trainer.py:803] 2025-04-26 17:30:50,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6609
6653
[WARNING|trainer.py:803] 2025-04-26 17:30:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:51,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6677
[WARNING|trainer.py:803] 2025-04-26 17:30:51,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6610
[WARNING|trainer.py:803] 2025-04-26 17:30:52,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6654
[WARNING|trainer.py:803] 2025-04-26 17:30:52,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6678
[WARNING|trainer.py:803] 2025-04-26 17:30:52,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6611
[WARNING|trainer.py:803] 2025-04-26 17:30:53,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6655
[WARNING|trainer.py:803] 2025-04-26 17:30:53,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6679
[WARNING|trainer.py:803] 2025-04-26 17:30:54,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6612
[WARNING|trainer.py:803] 2025-04-26 17:30:54,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6656
[WARNING|trainer.py:803] 2025-04-26 17:30:55,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6680
6613
[WARNING|trainer.py:803] 2025-04-26 17:30:55,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6657
[WARNING|trainer.py:803] 2025-04-26 17:30:56,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:56,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6681
[WARNING|trainer.py:803] 2025-04-26 17:30:56,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6614
6658
[WARNING|trainer.py:803] 2025-04-26 17:30:57,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:30:57,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6682
[WARNING|trainer.py:803] 2025-04-26 17:30:58,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6615
[WARNING|trainer.py:803] 2025-04-26 17:30:58,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6659
[WARNING|trainer.py:803] 2025-04-26 17:30:59,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6683
[WARNING|trainer.py:803] 2025-04-26 17:30:59,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6616
6660
[WARNING|trainer.py:803] 2025-04-26 17:31:00,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:00,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6684
[WARNING|trainer.py:803] 2025-04-26 17:31:00,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6617
6661
[WARNING|trainer.py:803] 2025-04-26 17:31:01,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:01,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6685
[WARNING|trainer.py:803] 2025-04-26 17:31:02,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6618
6662
[WARNING|trainer.py:803] 2025-04-26 17:31:02,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:03,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6686
[WARNING|trainer.py:803] 2025-04-26 17:31:03,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6619
6663
[WARNING|trainer.py:803] 2025-04-26 17:31:04,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:04,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6687
[WARNING|trainer.py:803] 2025-04-26 17:31:04,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6620
6664
[WARNING|trainer.py:803] 2025-04-26 17:31:05,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:05,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6688
[WARNING|trainer.py:803] 2025-04-26 17:31:06,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6621
6665
[WARNING|trainer.py:803] 2025-04-26 17:31:06,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:06,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6689
[WARNING|trainer.py:803] 2025-04-26 17:31:07,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6622
6666
[WARNING|trainer.py:803] 2025-04-26 17:31:08,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:08,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6690
[WARNING|trainer.py:803] 2025-04-26 17:31:08,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6623
6667
[WARNING|trainer.py:803] 2025-04-26 17:31:09,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:09,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6691
[WARNING|trainer.py:803] 2025-04-26 17:31:10,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6624
6668
[WARNING|trainer.py:803] 2025-04-26 17:31:10,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:10,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6692
[WARNING|trainer.py:803] 2025-04-26 17:31:11,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6625
6669
[WARNING|trainer.py:803] 2025-04-26 17:31:12,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:12,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6693
[WARNING|trainer.py:803] 2025-04-26 17:31:12,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6626
6670
[WARNING|trainer.py:803] 2025-04-26 17:31:13,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:13,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6694
[WARNING|trainer.py:803] 2025-04-26 17:31:13,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6627
6671
[WARNING|trainer.py:803] 2025-04-26 17:31:14,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:14,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6695
[WARNING|trainer.py:803] 2025-04-26 17:31:15,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6628
6672
[WARNING|trainer.py:803] 2025-04-26 17:31:15,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:16,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6696
[WARNING|trainer.py:803] 2025-04-26 17:31:16,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6629
6673
[WARNING|trainer.py:803] 2025-04-26 17:31:17,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:17,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6697
[WARNING|trainer.py:803] 2025-04-26 17:31:17,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6630
6674
[WARNING|trainer.py:803] 2025-04-26 17:31:18,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:18,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6698
[WARNING|trainer.py:803] 2025-04-26 17:31:19,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6631
6675
[WARNING|trainer.py:803] 2025-04-26 17:31:19,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:20,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6699
[WARNING|trainer.py:803] 2025-04-26 17:31:20,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6632
6676
[WARNING|trainer.py:803] 2025-04-26 17:31:21,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:21,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6700
[WARNING|trainer.py:803] 2025-04-26 17:31:21,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6633
6677
[WARNING|trainer.py:803] 2025-04-26 17:31:22,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:22,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6701
[WARNING|trainer.py:803] 2025-04-26 17:31:23,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6634
6678
[WARNING|trainer.py:803] 2025-04-26 17:31:23,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:31:24,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6702
[WARNING|trainer.py:803] 2025-04-26 17:31:24,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6635
6679
[WARNING|trainer.py:803] 2025-04-26 17:31:25,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:25,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6703
[WARNING|trainer.py:803] 2025-04-26 17:31:25,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6636
6680
[WARNING|trainer.py:803] 2025-04-26 17:31:26,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:31:26,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6704
[WARNING|trainer.py:803] 2025-04-26 17:31:27,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6637
6681
[WARNING|trainer.py:803] 2025-04-26 17:31:27,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:28,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6705
[WARNING|trainer.py:803] 2025-04-26 17:31:28,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6638
6682
[WARNING|trainer.py:803] 2025-04-26 17:31:29,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:29,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:29,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6706
6639
6683
[WARNING|trainer.py:803] 2025-04-26 17:31:30,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:30,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:31,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6707
6640
6684
[WARNING|trainer.py:803] 2025-04-26 17:31:31,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:32,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6708
[WARNING|trainer.py:803] 2025-04-26 17:31:32,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6641
6685
[WARNING|trainer.py:803] 2025-04-26 17:31:33,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:33,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6709
[WARNING|trainer.py:803] 2025-04-26 17:31:33,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6642
6686
[WARNING|trainer.py:803] 2025-04-26 17:31:34,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:34,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6710
[WARNING|trainer.py:803] 2025-04-26 17:31:35,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6643
6687
[WARNING|trainer.py:803] 2025-04-26 17:31:35,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:36,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6711
[WARNING|trainer.py:803] 2025-04-26 17:31:36,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6644
6688
[WARNING|trainer.py:803] 2025-04-26 17:31:37,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:37,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6712
[WARNING|trainer.py:803] 2025-04-26 17:31:37,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6645
6689
[WARNING|trainer.py:803] 2025-04-26 17:31:38,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:38,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6713
[WARNING|trainer.py:803] 2025-04-26 17:31:39,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6646
6690
[WARNING|trainer.py:803] 2025-04-26 17:31:39,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:40,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6714
[WARNING|trainer.py:803] 2025-04-26 17:31:40,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6647
6691
[WARNING|trainer.py:803] 2025-04-26 17:31:41,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:41,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6715
[WARNING|trainer.py:803] 2025-04-26 17:31:41,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6648
6692
[WARNING|trainer.py:803] 2025-04-26 17:31:42,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:42,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6716
[WARNING|trainer.py:803] 2025-04-26 17:31:43,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6649
6693
[WARNING|trainer.py:803] 2025-04-26 17:31:43,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:44,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6717
[WARNING|trainer.py:803] 2025-04-26 17:31:44,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6650
6694
[WARNING|trainer.py:803] 2025-04-26 17:31:45,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:45,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:45,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6718
6651
6695
[WARNING|trainer.py:803] 2025-04-26 17:31:46,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:31:46,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:46,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6719
6652
6696
[WARNING|trainer.py:803] 2025-04-26 17:31:47,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:48,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:48,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6720
6653
6697
[WARNING|trainer.py:803] 2025-04-26 17:31:49,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:49,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:49,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6721
6654
6698
[WARNING|trainer.py:803] 2025-04-26 17:31:50,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:50,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:50,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6722
6655
6699
[WARNING|trainer.py:803] 2025-04-26 17:31:51,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:52,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:52,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6723
6656
6700
[WARNING|trainer.py:803] 2025-04-26 17:31:53,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:53,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:53,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6724
6657
6701
[WARNING|trainer.py:803] 2025-04-26 17:31:54,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:54,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:54,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6725
6658
6702
[WARNING|trainer.py:803] 2025-04-26 17:31:55,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:56,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:56,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6726
6659
6703
[WARNING|trainer.py:803] 2025-04-26 17:31:57,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:31:57,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:57,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6727
6660
6704
[WARNING|trainer.py:803] 2025-04-26 17:31:58,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:31:58,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:31:58,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6728
6661
6705
[WARNING|trainer.py:803] 2025-04-26 17:31:59,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:00,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:00,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6729
6662
6706
[WARNING|trainer.py:803] 2025-04-26 17:32:01,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:01,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:01,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6730
6663
6707
[WARNING|trainer.py:803] 2025-04-26 17:32:02,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 17:32:02,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:02,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6731
6664
6708
[WARNING|trainer.py:803] 2025-04-26 17:32:03,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:04,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:04,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6732
6665
6709
[WARNING|trainer.py:803] 2025-04-26 17:32:05,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:05,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:05,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6733
6666
6710
[WARNING|trainer.py:803] 2025-04-26 17:32:06,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:06,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:06,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6734
6667
6711
[WARNING|trainer.py:803] 2025-04-26 17:32:07,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:08,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:08,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6735
6668
6712
[WARNING|trainer.py:803] 2025-04-26 17:32:09,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:09,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:09,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6736
6669
6713
[WARNING|trainer.py:803] 2025-04-26 17:32:10,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:10,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:10,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6737
6670
6714
[WARNING|trainer.py:803] 2025-04-26 17:32:11,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:12,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:12,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6738
6671
6715
[WARNING|trainer.py:803] 2025-04-26 17:32:13,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:13,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:13,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6739
6672
6716
[WARNING|trainer.py:803] 2025-04-26 17:32:14,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:14,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:14,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6740
6673
6717
[WARNING|trainer.py:803] 2025-04-26 17:32:15,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:16,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:16,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6741
6674
6718
[WARNING|trainer.py:803] 2025-04-26 17:32:17,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:17,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6742
[WARNING|trainer.py:803] 2025-04-26 17:32:17,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6675
6719
[WARNING|trainer.py:803] 2025-04-26 17:32:18,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:18,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6743
[WARNING|trainer.py:803] 2025-04-26 17:32:18,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6676
6720
[WARNING|trainer.py:803] 2025-04-26 17:32:19,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:20,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6744
[WARNING|trainer.py:803] 2025-04-26 17:32:20,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6677
6721
[WARNING|trainer.py:803] 2025-04-26 17:32:21,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:21,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6745
[WARNING|trainer.py:803] 2025-04-26 17:32:21,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6678
6722
[WARNING|trainer.py:803] 2025-04-26 17:32:22,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:22,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6746
[WARNING|trainer.py:803] 2025-04-26 17:32:23,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6679
6723
[WARNING|trainer.py:803] 2025-04-26 17:32:23,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:24,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6747
[WARNING|trainer.py:803] 2025-04-26 17:32:24,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6680
6724
[WARNING|trainer.py:803] 2025-04-26 17:32:25,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:25,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6748
[WARNING|trainer.py:803] 2025-04-26 17:32:25,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6681
6725
[WARNING|trainer.py:803] 2025-04-26 17:32:26,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:26,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6749
[WARNING|trainer.py:803] 2025-04-26 17:32:27,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6682
6726
[WARNING|trainer.py:803] 2025-04-26 17:32:27,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:28,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6750
[WARNING|trainer.py:803] 2025-04-26 17:32:28,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6683
6727
[WARNING|trainer.py:803] 2025-04-26 17:32:29,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6751
[WARNING|trainer.py:803] 2025-04-26 17:32:29,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:29,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6684
6728
[WARNING|trainer.py:803] 2025-04-26 17:32:30,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6752
[WARNING|trainer.py:803] 2025-04-26 17:32:30,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:31,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6685
6729
[WARNING|trainer.py:803] 2025-04-26 17:32:31,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6753
[WARNING|trainer.py:803] 2025-04-26 17:32:32,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:32,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6686
6730
[WARNING|trainer.py:803] 2025-04-26 17:32:32,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6754
[WARNING|trainer.py:803] 2025-04-26 17:32:33,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:33,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6687
6731
[WARNING|trainer.py:803] 2025-04-26 17:32:34,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6755
[WARNING|trainer.py:803] 2025-04-26 17:32:34,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:35,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6688
[WARNING|trainer.py:803] 2025-04-26 17:32:35,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6732
6756
[WARNING|trainer.py:803] 2025-04-26 17:32:36,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:36,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6689
[WARNING|trainer.py:803] 2025-04-26 17:32:36,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6733
6757
[WARNING|trainer.py:803] 2025-04-26 17:32:37,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:37,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6690
[WARNING|trainer.py:803] 2025-04-26 17:32:38,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6734
6758
[WARNING|trainer.py:803] 2025-04-26 17:32:38,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:38,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6691
6735
[WARNING|trainer.py:803] 2025-04-26 17:32:39,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6759
[WARNING|trainer.py:803] 2025-04-26 17:32:40,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:40,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6692
6736
[WARNING|trainer.py:803] 2025-04-26 17:32:40,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6760
[WARNING|trainer.py:803] 2025-04-26 17:32:41,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:41,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6693
6737
[WARNING|trainer.py:803] 2025-04-26 17:32:42,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6761
[WARNING|trainer.py:803] 2025-04-26 17:32:42,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:42,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6694
6738
[WARNING|trainer.py:803] 2025-04-26 17:32:43,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6762
[WARNING|trainer.py:803] 2025-04-26 17:32:44,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:44,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6695
6739
[WARNING|trainer.py:803] 2025-04-26 17:32:44,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6763
[WARNING|trainer.py:803] 2025-04-26 17:32:45,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:45,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6696
6740
[WARNING|trainer.py:803] 2025-04-26 17:32:46,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6764
[WARNING|trainer.py:803] 2025-04-26 17:32:46,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:46,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6697
6741
[WARNING|trainer.py:803] 2025-04-26 17:32:47,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6765
[WARNING|trainer.py:803] 2025-04-26 17:32:48,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:48,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6698
6742
[WARNING|trainer.py:803] 2025-04-26 17:32:48,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6766
[WARNING|trainer.py:803] 2025-04-26 17:32:49,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:49,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6699
6743
[WARNING|trainer.py:803] 2025-04-26 17:32:50,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6767
[WARNING|trainer.py:803] 2025-04-26 17:32:50,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:50,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6700
6744
[WARNING|trainer.py:803] 2025-04-26 17:32:51,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6768
[WARNING|trainer.py:803] 2025-04-26 17:32:52,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:52,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6701
[WARNING|trainer.py:803] 2025-04-26 17:32:52,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6745
6769
[WARNING|trainer.py:803] 2025-04-26 17:32:53,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:32:53,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6702
[WARNING|trainer.py:803] 2025-04-26 17:32:53,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6746
6770
[WARNING|trainer.py:803] 2025-04-26 17:32:54,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:54,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:55,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6703
6747
6771
[WARNING|trainer.py:803] 2025-04-26 17:32:56,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:32:56,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:32:56,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6704
6748
6772
[WARNING|trainer.py:803] 2025-04-26 17:32:57,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:57,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:32:57,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6705
6749
6773
[WARNING|trainer.py:803] 2025-04-26 17:32:58,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:32:58,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:32:59,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6750
6706
6774
[WARNING|trainer.py:803] 2025-04-26 17:33:00,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:00,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:00,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6751
6707
6775
[WARNING|trainer.py:803] 2025-04-26 17:33:01,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:01,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:01,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6752
6708
6776
[WARNING|trainer.py:803] 2025-04-26 17:33:02,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:02,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:03,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6753
6709
6777
[WARNING|trainer.py:803] 2025-04-26 17:33:04,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:04,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:04,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6754
6710
6778
[WARNING|trainer.py:803] 2025-04-26 17:33:05,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:05,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6755
[WARNING|trainer.py:803] 2025-04-26 17:33:05,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6779
6711
[WARNING|trainer.py:803] 2025-04-26 17:33:06,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6756
[WARNING|trainer.py:803] 2025-04-26 17:33:07,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:07,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6780
6712
[WARNING|trainer.py:803] 2025-04-26 17:33:07,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6757
[WARNING|trainer.py:803] 2025-04-26 17:33:08,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:08,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6781
6713
[WARNING|trainer.py:803] 2025-04-26 17:33:09,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:09,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6758
[WARNING|trainer.py:803] 2025-04-26 17:33:09,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6782
6714
[WARNING|trainer.py:803] 2025-04-26 17:33:10,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6759
[WARNING|trainer.py:803] 2025-04-26 17:33:11,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:11,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6783
6715
[WARNING|trainer.py:803] 2025-04-26 17:33:11,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:12,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6760
[WARNING|trainer.py:803] 2025-04-26 17:33:12,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6784
6716
[WARNING|trainer.py:803] 2025-04-26 17:33:13,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:13,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6761
[WARNING|trainer.py:803] 2025-04-26 17:33:13,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6785
6717
[WARNING|trainer.py:803] 2025-04-26 17:33:14,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6762
[WARNING|trainer.py:803] 2025-04-26 17:33:14,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:15,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6786
[WARNING|trainer.py:803] 2025-04-26 17:33:15,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6718
6763
[WARNING|trainer.py:803] 2025-04-26 17:33:16,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:16,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6787
[WARNING|trainer.py:803] 2025-04-26 17:33:17,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6719
6764
[WARNING|trainer.py:803] 2025-04-26 17:33:17,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:18,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6788
[WARNING|trainer.py:803] 2025-04-26 17:33:18,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6720
6765
[WARNING|trainer.py:803] 2025-04-26 17:33:18,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6789
[WARNING|trainer.py:803] 2025-04-26 17:33:19,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:33:19,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6721
6766
[WARNING|trainer.py:803] 2025-04-26 17:33:20,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6790
[WARNING|trainer.py:803] 2025-04-26 17:33:20,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:21,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6722
6767
[WARNING|trainer.py:803] 2025-04-26 17:33:21,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6791
[WARNING|trainer.py:803] 2025-04-26 17:33:22,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:22,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6723
6768
[WARNING|trainer.py:803] 2025-04-26 17:33:22,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6792
[WARNING|trainer.py:803] 2025-04-26 17:33:23,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:23,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6724
6769
[WARNING|trainer.py:803] 2025-04-26 17:33:24,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6793
[WARNING|trainer.py:803] 2025-04-26 17:33:24,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:25,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6725
6770
[WARNING|trainer.py:803] 2025-04-26 17:33:25,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:26,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:26,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6794
6726
6771
[WARNING|trainer.py:803] 2025-04-26 17:33:27,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:27,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6795
[WARNING|trainer.py:803] 2025-04-26 17:33:27,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6727
6772
[WARNING|trainer.py:803] 2025-04-26 17:33:28,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:28,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:33:28,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6796
6728
6773
[WARNING|trainer.py:803] 2025-04-26 17:33:29,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:30,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:30,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6797
6729
6774
[WARNING|trainer.py:803] 2025-04-26 17:33:31,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:31,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:31,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6798
6730
6775
[WARNING|trainer.py:803] 2025-04-26 17:33:32,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:32,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:32,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6799
6731
6776
[WARNING|trainer.py:803] 2025-04-26 17:33:33,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:33:34,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:34,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6800
6777
6732
[WARNING|trainer.py:803] 2025-04-26 17:33:35,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6801
[WARNING|trainer.py:803] 2025-04-26 17:33:35,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:35,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6778
6733
[WARNING|trainer.py:803] 2025-04-26 17:33:36,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6802
[WARNING|trainer.py:803] 2025-04-26 17:33:36,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:36,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6734
6779
[WARNING|trainer.py:803] 2025-04-26 17:33:37,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:38,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 17:33:38,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
NoYes
6803
6780
6735
[WARNING|trainer.py:803] 2025-04-26 17:33:39,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6804
[WARNING|trainer.py:803] 2025-04-26 17:33:39,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:39,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6736
6781
[WARNING|trainer.py:803] 2025-04-26 17:33:40,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6805
[WARNING|trainer.py:803] 2025-04-26 17:33:40,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:40,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6782
6737
[WARNING|trainer.py:803] 2025-04-26 17:33:41,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6806
[WARNING|trainer.py:803] 2025-04-26 17:33:42,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:42,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6783
6738
[WARNING|trainer.py:803] 2025-04-26 17:33:42,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6807
[WARNING|trainer.py:803] 2025-04-26 17:33:43,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:43,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6784
6739
[WARNING|trainer.py:803] 2025-04-26 17:33:44,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6808
[WARNING|trainer.py:803] 2025-04-26 17:33:44,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:44,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6785
6740
[WARNING|trainer.py:803] 2025-04-26 17:33:45,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6809
[WARNING|trainer.py:803] 2025-04-26 17:33:46,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:46,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6786
6741
[WARNING|trainer.py:803] 2025-04-26 17:33:46,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6810
[WARNING|trainer.py:803] 2025-04-26 17:33:47,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:47,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6787
6742
[WARNING|trainer.py:803] 2025-04-26 17:33:48,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6811
[WARNING|trainer.py:803] 2025-04-26 17:33:48,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:48,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6788
[WARNING|trainer.py:803] 2025-04-26 17:33:49,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6743
6812
[WARNING|trainer.py:803] 2025-04-26 17:33:50,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:50,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6789
[WARNING|trainer.py:803] 2025-04-26 17:33:50,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6744
6813
[WARNING|trainer.py:803] 2025-04-26 17:33:51,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:51,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6790
[WARNING|trainer.py:803] 2025-04-26 17:33:51,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6745
6814
[WARNING|trainer.py:803] 2025-04-26 17:33:52,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:52,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6791
[WARNING|trainer.py:803] 2025-04-26 17:33:53,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6746
6815
[WARNING|trainer.py:803] 2025-04-26 17:33:54,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:54,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6792
[WARNING|trainer.py:803] 2025-04-26 17:33:54,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6747
6816
[WARNING|trainer.py:803] 2025-04-26 17:33:55,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:55,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6793
[WARNING|trainer.py:803] 2025-04-26 17:33:55,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6748
6817
[WARNING|trainer.py:803] 2025-04-26 17:33:56,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:33:57,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6794
[WARNING|trainer.py:803] 2025-04-26 17:33:57,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6749
6818
[WARNING|trainer.py:803] 2025-04-26 17:33:57,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:33:58,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6795
[WARNING|trainer.py:803] 2025-04-26 17:33:58,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6750
6819
[WARNING|trainer.py:803] 2025-04-26 17:33:59,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:33:59,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6796
[WARNING|trainer.py:803] 2025-04-26 17:33:59,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6751
6820
[WARNING|trainer.py:803] 2025-04-26 17:34:00,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:01,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6797
[WARNING|trainer.py:803] 2025-04-26 17:34:01,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6752
6821
[WARNING|trainer.py:803] 2025-04-26 17:34:01,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6798
[WARNING|trainer.py:803] 2025-04-26 17:34:02,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:02,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6753
6822
[WARNING|trainer.py:803] 2025-04-26 17:34:03,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6799
[WARNING|trainer.py:803] 2025-04-26 17:34:03,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:03,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6754
6823
[WARNING|trainer.py:803] 2025-04-26 17:34:04,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6800
[WARNING|trainer.py:803] 2025-04-26 17:34:05,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:05,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6755
6824
[WARNING|trainer.py:803] 2025-04-26 17:34:05,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6801
[WARNING|trainer.py:803] 2025-04-26 17:34:06,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:06,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6756
6825
[WARNING|trainer.py:803] 2025-04-26 17:34:07,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6802
[WARNING|trainer.py:803] 2025-04-26 17:34:07,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:07,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6757
6826
[WARNING|trainer.py:803] 2025-04-26 17:34:08,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6803
[WARNING|trainer.py:803] 2025-04-26 17:34:08,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:09,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6758
6827
[WARNING|trainer.py:803] 2025-04-26 17:34:09,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6804
[WARNING|trainer.py:803] 2025-04-26 17:34:10,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:10,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6759
6828
[WARNING|trainer.py:803] 2025-04-26 17:34:10,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6805
[WARNING|trainer.py:803] 2025-04-26 17:34:11,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:11,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6760
6829
[WARNING|trainer.py:803] 2025-04-26 17:34:12,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6806
[WARNING|trainer.py:803] 2025-04-26 17:34:12,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:12,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6761
6830
[WARNING|trainer.py:803] 2025-04-26 17:34:13,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6807
[WARNING|trainer.py:803] 2025-04-26 17:34:14,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:14,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6762
6831
[WARNING|trainer.py:803] 2025-04-26 17:34:14,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6808
[WARNING|trainer.py:803] 2025-04-26 17:34:15,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:34:15,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6763
6832
[WARNING|trainer.py:803] 2025-04-26 17:34:16,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6809
[WARNING|trainer.py:803] 2025-04-26 17:34:16,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:16,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6833
6764
[WARNING|trainer.py:803] 2025-04-26 17:34:17,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6810
[WARNING|trainer.py:803] 2025-04-26 17:34:18,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:18,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6834
[WARNING|trainer.py:803] 2025-04-26 17:34:18,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6765
6811
[WARNING|trainer.py:803] 2025-04-26 17:34:19,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:19,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:19,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
6835
NoNo
6766
6812
[WARNING|trainer.py:803] 2025-04-26 17:34:20,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:20,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:21,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6836
6767
6813
[WARNING|trainer.py:803] 2025-04-26 17:34:22,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:22,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:34:22,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6837
6768
6814
[WARNING|trainer.py:803] 2025-04-26 17:34:23,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:23,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:34:23,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6838
6769
6815
[WARNING|trainer.py:803] 2025-04-26 17:34:24,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:24,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:25,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6839
6770
6816
[WARNING|trainer.py:803] 2025-04-26 17:34:26,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:26,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:26,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6840
6771
6817
[WARNING|trainer.py:803] 2025-04-26 17:34:27,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:27,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:27,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6841
6772
6818
[WARNING|trainer.py:803] 2025-04-26 17:34:28,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:28,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6842
[WARNING|trainer.py:803] 2025-04-26 17:34:29,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6773
6819
[WARNING|trainer.py:803] 2025-04-26 17:34:29,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:30,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6843
[WARNING|trainer.py:803] 2025-04-26 17:34:30,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6774
6820
[WARNING|trainer.py:803] 2025-04-26 17:34:31,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:31,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6844
[WARNING|trainer.py:803] 2025-04-26 17:34:31,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6775
6821
[WARNING|trainer.py:803] 2025-04-26 17:34:32,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:32,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6845
[WARNING|trainer.py:803] 2025-04-26 17:34:33,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6776
6822
[WARNING|trainer.py:803] 2025-04-26 17:34:33,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:34,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:34,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6846
6777
6823
[WARNING|trainer.py:803] 2025-04-26 17:34:35,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6847
[WARNING|trainer.py:803] 2025-04-26 17:34:35,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:35,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6778
6824
[WARNING|trainer.py:803] 2025-04-26 17:34:36,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6848
[WARNING|trainer.py:803] 2025-04-26 17:34:36,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:37,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6825
6779
[WARNING|trainer.py:803] 2025-04-26 17:34:37,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6849
[WARNING|trainer.py:803] 2025-04-26 17:34:38,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:38,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6826
6780
[WARNING|trainer.py:803] 2025-04-26 17:34:39,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6850
[WARNING|trainer.py:803] 2025-04-26 17:34:39,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:39,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6827
6781
[WARNING|trainer.py:803] 2025-04-26 17:34:40,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6851
[WARNING|trainer.py:803] 2025-04-26 17:34:40,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:40,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6828
6782
[WARNING|trainer.py:803] 2025-04-26 17:34:41,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6852
[WARNING|trainer.py:803] 2025-04-26 17:34:42,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:42,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6829
6783
[WARNING|trainer.py:803] 2025-04-26 17:34:42,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6853
[WARNING|trainer.py:803] 2025-04-26 17:34:43,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:43,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6830
6784
[WARNING|trainer.py:803] 2025-04-26 17:34:44,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6854
[WARNING|trainer.py:803] 2025-04-26 17:34:44,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:44,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6831
6785
[WARNING|trainer.py:803] 2025-04-26 17:34:45,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:46,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6855
[WARNING|trainer.py:803] 2025-04-26 17:34:46,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6832
6786
[WARNING|trainer.py:803] 2025-04-26 17:34:46,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6856
[WARNING|trainer.py:803] 2025-04-26 17:34:47,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:47,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6833
6787
[WARNING|trainer.py:803] 2025-04-26 17:34:48,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6857
[WARNING|trainer.py:803] 2025-04-26 17:34:48,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:48,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6834
6788
[WARNING|trainer.py:803] 2025-04-26 17:34:49,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6858
[WARNING|trainer.py:803] 2025-04-26 17:34:49,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:50,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6835
6789
[WARNING|trainer.py:803] 2025-04-26 17:34:50,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6859
[WARNING|trainer.py:803] 2025-04-26 17:34:51,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:51,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6836
[WARNING|trainer.py:803] 2025-04-26 17:34:52,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6790
6860
[WARNING|trainer.py:803] 2025-04-26 17:34:52,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:52,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6837
[WARNING|trainer.py:803] 2025-04-26 17:34:53,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6791
6861
[WARNING|trainer.py:803] 2025-04-26 17:34:53,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:54,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6838
[WARNING|trainer.py:803] 2025-04-26 17:34:54,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6792
6862
[WARNING|trainer.py:803] 2025-04-26 17:34:55,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:34:55,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6839
[WARNING|trainer.py:803] 2025-04-26 17:34:55,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6793
6863
[WARNING|trainer.py:803] 2025-04-26 17:34:56,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:56,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6840
[WARNING|trainer.py:803] 2025-04-26 17:34:57,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6794
6864
[WARNING|trainer.py:803] 2025-04-26 17:34:57,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:34:58,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6841
[WARNING|trainer.py:803] 2025-04-26 17:34:58,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6795
6865
[WARNING|trainer.py:803] 2025-04-26 17:34:59,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6842
[WARNING|trainer.py:803] 2025-04-26 17:34:59,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:34:59,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6796
6866
[WARNING|trainer.py:803] 2025-04-26 17:35:00,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6843
[WARNING|trainer.py:803] 2025-04-26 17:35:00,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:00,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6797
6867
[WARNING|trainer.py:803] 2025-04-26 17:35:01,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6844
[WARNING|trainer.py:803] 2025-04-26 17:35:02,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:02,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6868
6798
[WARNING|trainer.py:803] 2025-04-26 17:35:03,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6845
[WARNING|trainer.py:803] 2025-04-26 17:35:03,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:03,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6869
6799
[WARNING|trainer.py:803] 2025-04-26 17:35:04,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6846
[WARNING|trainer.py:803] 2025-04-26 17:35:04,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:04,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6870
6800
[WARNING|trainer.py:803] 2025-04-26 17:35:05,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6847
[WARNING|trainer.py:803] 2025-04-26 17:35:06,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:06,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6871
6801
[WARNING|trainer.py:803] 2025-04-26 17:35:06,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6848
[WARNING|trainer.py:803] 2025-04-26 17:35:07,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:07,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6872
6802
[WARNING|trainer.py:803] 2025-04-26 17:35:08,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6849
[WARNING|trainer.py:803] 2025-04-26 17:35:08,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:08,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6873
6803
[WARNING|trainer.py:803] 2025-04-26 17:35:09,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6850
[WARNING|trainer.py:803] 2025-04-26 17:35:10,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:10,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6874
6804
[WARNING|trainer.py:803] 2025-04-26 17:35:10,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6851
[WARNING|trainer.py:803] 2025-04-26 17:35:11,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:11,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6875
6805
[WARNING|trainer.py:803] 2025-04-26 17:35:12,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6852
[WARNING|trainer.py:803] 2025-04-26 17:35:12,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:12,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6876
6806
[WARNING|trainer.py:803] 2025-04-26 17:35:13,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6853
[WARNING|trainer.py:803] 2025-04-26 17:35:14,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:14,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6877
[WARNING|trainer.py:803] 2025-04-26 17:35:14,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6807
6854
[WARNING|trainer.py:803] 2025-04-26 17:35:15,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:15,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:15,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6878
6808
6855
[WARNING|trainer.py:803] 2025-04-26 17:35:16,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:16,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:17,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6879
6809
6856
[WARNING|trainer.py:803] 2025-04-26 17:35:18,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:18,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:18,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6880
6810
6857
[WARNING|trainer.py:803] 2025-04-26 17:35:19,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:19,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:19,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6881
6811
6858
[WARNING|trainer.py:803] 2025-04-26 17:35:20,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:20,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:21,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6812
6882
6859
[WARNING|trainer.py:803] 2025-04-26 17:35:22,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:22,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:22,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6813
6883
6860
[WARNING|trainer.py:803] 2025-04-26 17:35:23,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:23,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:23,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6814
6884
6861
[WARNING|trainer.py:803] 2025-04-26 17:35:24,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:24,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:25,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6815
6885
6862
[WARNING|trainer.py:803] 2025-04-26 17:35:25,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:26,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:26,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6816
6886
6863
[WARNING|trainer.py:803] 2025-04-26 17:35:27,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:27,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:27,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6817
6887
6864
[WARNING|trainer.py:803] 2025-04-26 17:35:28,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:28,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:29,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6888
6818
6865
[WARNING|trainer.py:803] 2025-04-26 17:35:29,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:30,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:30,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6889
6819
6866
[WARNING|trainer.py:803] 2025-04-26 17:35:31,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:31,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:31,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6890
6820
6867
[WARNING|trainer.py:803] 2025-04-26 17:35:32,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:32,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:32,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6891
6821
6868
[WARNING|trainer.py:803] 2025-04-26 17:35:33,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:34,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:34,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6892
6822
6869
[WARNING|trainer.py:803] 2025-04-26 17:35:35,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:35,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:35,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6893
6823
6870
[WARNING|trainer.py:803] 2025-04-26 17:35:36,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:36,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:36,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6894
6824
6871
[WARNING|trainer.py:803] 2025-04-26 17:35:37,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:38,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:38,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6895
6825
6872
[WARNING|trainer.py:803] 2025-04-26 17:35:39,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:39,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:39,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6896
6826
6873
[WARNING|trainer.py:803] 2025-04-26 17:35:40,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:40,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:40,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6897
6827
6874
[WARNING|trainer.py:803] 2025-04-26 17:35:41,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:42,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:42,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6898
6828
6875
[WARNING|trainer.py:803] 2025-04-26 17:35:43,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:43,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:43,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6899
6829
6876
[WARNING|trainer.py:803] 2025-04-26 17:35:44,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:44,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:44,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6900
6830
6877
[WARNING|trainer.py:803] 2025-04-26 17:35:45,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:46,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:46,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6901
6831
6878
[WARNING|trainer.py:803] 2025-04-26 17:35:47,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:47,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:47,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6902
6832
6879
[WARNING|trainer.py:803] 2025-04-26 17:35:48,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:48,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:48,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6903
6833
6880
[WARNING|trainer.py:803] 2025-04-26 17:35:49,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:49,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:50,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
6904
NoNo
6834
6881
[WARNING|trainer.py:803] 2025-04-26 17:35:50,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:35:51,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6905
[WARNING|trainer.py:803] 2025-04-26 17:35:51,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6835
6882
[WARNING|trainer.py:803] 2025-04-26 17:35:52,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6906
[WARNING|trainer.py:803] 2025-04-26 17:35:52,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:52,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6836
6883
[WARNING|trainer.py:803] 2025-04-26 17:35:53,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6907
[WARNING|trainer.py:803] 2025-04-26 17:35:53,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:54,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6837
6884
[WARNING|trainer.py:803] 2025-04-26 17:35:54,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6908
[WARNING|trainer.py:803] 2025-04-26 17:35:55,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:55,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6838
6885
[WARNING|trainer.py:803] 2025-04-26 17:35:55,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6909
[WARNING|trainer.py:803] 2025-04-26 17:35:56,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:35:56,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6839
[WARNING|trainer.py:803] 2025-04-26 17:35:57,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6886
6910
[WARNING|trainer.py:803] 2025-04-26 17:35:57,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:57,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6840
[WARNING|trainer.py:803] 2025-04-26 17:35:58,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6887
6911
[WARNING|trainer.py:803] 2025-04-26 17:35:59,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:35:59,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:35:59,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6841
6888
6912
[WARNING|trainer.py:803] 2025-04-26 17:36:00,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:00,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:00,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6842
6889
6913
[WARNING|trainer.py:803] 2025-04-26 17:36:01,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:01,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:02,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6843
6890
6914
[WARNING|trainer.py:803] 2025-04-26 17:36:03,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:03,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:03,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6844
6891
6915
[WARNING|trainer.py:803] 2025-04-26 17:36:04,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:04,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:04,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6845
6916
6892
[WARNING|trainer.py:803] 2025-04-26 17:36:05,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:05,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:05,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6846
6917
6893
[WARNING|trainer.py:803] 2025-04-26 17:36:07,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:07,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:07,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6918
6847
6894
[WARNING|trainer.py:803] 2025-04-26 17:36:08,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:08,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:08,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6919
6848
6895
[WARNING|trainer.py:803] 2025-04-26 17:36:09,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:09,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:09,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6920
6849
6896
[WARNING|trainer.py:803] 2025-04-26 17:36:10,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:11,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:11,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6921
6850
6897
[WARNING|trainer.py:803] 2025-04-26 17:36:12,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:12,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6922
[WARNING|trainer.py:803] 2025-04-26 17:36:12,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6851
6898
[WARNING|trainer.py:803] 2025-04-26 17:36:13,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:13,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6923
[WARNING|trainer.py:803] 2025-04-26 17:36:13,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6852
6899
[WARNING|trainer.py:803] 2025-04-26 17:36:14,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6924
[WARNING|trainer.py:803] 2025-04-26 17:36:14,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:15,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6853
6900
[WARNING|trainer.py:803] 2025-04-26 17:36:15,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6925
[WARNING|trainer.py:803] 2025-04-26 17:36:16,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:16,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6854
6901
[WARNING|trainer.py:803] 2025-04-26 17:36:16,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6926
[WARNING|trainer.py:803] 2025-04-26 17:36:17,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:17,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6855
6902
[WARNING|trainer.py:803] 2025-04-26 17:36:18,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6927
[WARNING|trainer.py:803] 2025-04-26 17:36:18,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:18,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6903
6856
[WARNING|trainer.py:803] 2025-04-26 17:36:19,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6928
[WARNING|trainer.py:803] 2025-04-26 17:36:20,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:20,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6904
6857
[WARNING|trainer.py:803] 2025-04-26 17:36:20,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6929
[WARNING|trainer.py:803] 2025-04-26 17:36:21,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:36:21,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6905
[WARNING|trainer.py:803] 2025-04-26 17:36:21,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6858
6930
[WARNING|trainer.py:803] 2025-04-26 17:36:22,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:22,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6906
[WARNING|trainer.py:803] 2025-04-26 17:36:23,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6859
6931
[WARNING|trainer.py:803] 2025-04-26 17:36:23,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:24,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6907
[WARNING|trainer.py:803] 2025-04-26 17:36:24,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6860
6932
[WARNING|trainer.py:803] 2025-04-26 17:36:25,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:25,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6908
[WARNING|trainer.py:803] 2025-04-26 17:36:25,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6861
6933
[WARNING|trainer.py:803] 2025-04-26 17:36:26,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6909
[WARNING|trainer.py:803] 2025-04-26 17:36:26,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:26,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6862
6934
[WARNING|trainer.py:803] 2025-04-26 17:36:27,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6910
[WARNING|trainer.py:803] 2025-04-26 17:36:27,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:36:28,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6863
6935
[WARNING|trainer.py:803] 2025-04-26 17:36:28,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6911
[WARNING|trainer.py:803] 2025-04-26 17:36:29,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:29,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6936
6864
[WARNING|trainer.py:803] 2025-04-26 17:36:29,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6912
[WARNING|trainer.py:803] 2025-04-26 17:36:30,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:30,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6937
6865
[WARNING|trainer.py:803] 2025-04-26 17:36:31,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6913
[WARNING|trainer.py:803] 2025-04-26 17:36:31,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:31,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6938
[WARNING|trainer.py:803] 2025-04-26 17:36:32,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6866
6914
[WARNING|trainer.py:803] 2025-04-26 17:36:33,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:33,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6939
[WARNING|trainer.py:803] 2025-04-26 17:36:33,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6867
6915
[WARNING|trainer.py:803] 2025-04-26 17:36:34,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:34,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6940
[WARNING|trainer.py:803] 2025-04-26 17:36:34,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6868
6916
[WARNING|trainer.py:803] 2025-04-26 17:36:35,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:35,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6941
[WARNING|trainer.py:803] 2025-04-26 17:36:36,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6869
6917
[WARNING|trainer.py:803] 2025-04-26 17:36:36,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:37,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6942
[WARNING|trainer.py:803] 2025-04-26 17:36:37,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6870
6918
[WARNING|trainer.py:803] 2025-04-26 17:36:38,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:38,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6943
[WARNING|trainer.py:803] 2025-04-26 17:36:38,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6871
6919
[WARNING|trainer.py:803] 2025-04-26 17:36:39,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:39,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6944
[WARNING|trainer.py:803] 2025-04-26 17:36:39,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6920
6872
[WARNING|trainer.py:803] 2025-04-26 17:36:40,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6945
[WARNING|trainer.py:803] 2025-04-26 17:36:40,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:41,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6921
6873
[WARNING|trainer.py:803] 2025-04-26 17:36:41,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6946
[WARNING|trainer.py:803] 2025-04-26 17:36:42,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:42,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6922
6874
[WARNING|trainer.py:803] 2025-04-26 17:36:43,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6947
[WARNING|trainer.py:803] 2025-04-26 17:36:43,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:43,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6923
6875
[WARNING|trainer.py:803] 2025-04-26 17:36:44,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6948
[WARNING|trainer.py:803] 2025-04-26 17:36:44,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:45,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6924
[WARNING|trainer.py:803] 2025-04-26 17:36:45,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6876
[WARNING|trainer.py:803] 2025-04-26 17:36:45,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6949
6925
[WARNING|trainer.py:803] 2025-04-26 17:36:46,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:46,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6877
[WARNING|trainer.py:803] 2025-04-26 17:36:47,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6950
6926
[WARNING|trainer.py:803] 2025-04-26 17:36:47,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:48,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6878
[WARNING|trainer.py:803] 2025-04-26 17:36:48,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6951
6927
[WARNING|trainer.py:803] 2025-04-26 17:36:49,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:49,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6879
[WARNING|trainer.py:803] 2025-04-26 17:36:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6952
6928
[WARNING|trainer.py:803] 2025-04-26 17:36:50,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:50,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6880
6953
[WARNING|trainer.py:803] 2025-04-26 17:36:51,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6929
[WARNING|trainer.py:803] 2025-04-26 17:36:51,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:51,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6881
6954
[WARNING|trainer.py:803] 2025-04-26 17:36:52,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6930
[WARNING|trainer.py:803] 2025-04-26 17:36:52,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:53,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6882
[WARNING|trainer.py:803] 2025-04-26 17:36:53,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6955
6931
[WARNING|trainer.py:803] 2025-04-26 17:36:54,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:54,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:54,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6883
6956
6932
[WARNING|trainer.py:803] 2025-04-26 17:36:55,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:36:55,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:55,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6957
6884
6933
[WARNING|trainer.py:803] 2025-04-26 17:36:56,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:56,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:57,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6958
6885
6934
[WARNING|trainer.py:803] 2025-04-26 17:36:58,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:58,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:58,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6959
6886
6935
[WARNING|trainer.py:803] 2025-04-26 17:36:59,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:36:59,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:36:59,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6960
6887
6936
[WARNING|trainer.py:803] 2025-04-26 17:37:00,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:37:00,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:00,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6961
6888
6937
[WARNING|trainer.py:803] 2025-04-26 17:37:01,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:02,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:02,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6962
6938
6889
[WARNING|trainer.py:803] 2025-04-26 17:37:03,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:03,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:03,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6963
6939
6890
[WARNING|trainer.py:803] 2025-04-26 17:37:04,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:04,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6964
[WARNING|trainer.py:803] 2025-04-26 17:37:04,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6940
6891
[WARNING|trainer.py:803] 2025-04-26 17:37:05,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:05,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6965
[WARNING|trainer.py:803] 2025-04-26 17:37:06,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6941
6892
[WARNING|trainer.py:803] 2025-04-26 17:37:06,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:07,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6966
[WARNING|trainer.py:803] 2025-04-26 17:37:07,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6942
6893
[WARNING|trainer.py:803] 2025-04-26 17:37:08,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:08,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6967
6943
[WARNING|trainer.py:803] 2025-04-26 17:37:08,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6894
[WARNING|trainer.py:803] 2025-04-26 17:37:09,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:37:09,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6968
6944
[WARNING|trainer.py:803] 2025-04-26 17:37:10,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6895
[WARNING|trainer.py:803] 2025-04-26 17:37:10,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:10,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6969
6945
[WARNING|trainer.py:803] 2025-04-26 17:37:11,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:11,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6896
[WARNING|trainer.py:803] 2025-04-26 17:37:12,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6970
6946
[WARNING|trainer.py:803] 2025-04-26 17:37:12,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:13,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6897
[WARNING|trainer.py:803] 2025-04-26 17:37:13,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6971
6947
[WARNING|trainer.py:803] 2025-04-26 17:37:14,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:14,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6898
[WARNING|trainer.py:803] 2025-04-26 17:37:14,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6972
6948
[WARNING|trainer.py:803] 2025-04-26 17:37:15,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:15,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:15,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6899
6973
6949
[WARNING|trainer.py:803] 2025-04-26 17:37:16,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:16,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:17,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6900
6974
6950
[WARNING|trainer.py:803] 2025-04-26 17:37:18,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:18,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:18,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6901
6975
6951
[WARNING|trainer.py:803] 2025-04-26 17:37:19,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:19,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:19,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6902
6976
6952
[WARNING|trainer.py:803] 2025-04-26 17:37:20,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:20,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:20,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6903
6977
6953
[WARNING|trainer.py:803] 2025-04-26 17:37:21,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:21,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:22,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6904
6978
6954
[WARNING|trainer.py:803] 2025-04-26 17:37:23,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:37:23,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:23,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6905
6979
6955
[WARNING|trainer.py:803] 2025-04-26 17:37:24,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:24,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:24,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6906
6980
6956
[WARNING|trainer.py:803] 2025-04-26 17:37:25,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:25,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:25,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6907
6981
6957
[WARNING|trainer.py:803] 2025-04-26 17:37:26,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:26,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:27,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6908
6982
6958
[WARNING|trainer.py:803] 2025-04-26 17:37:28,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:28,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:28,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6909
6983
6959
[WARNING|trainer.py:803] 2025-04-26 17:37:29,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:29,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:29,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6910
6984
6960
[WARNING|trainer.py:803] 2025-04-26 17:37:30,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:30,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:30,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6985
6911
6961
[WARNING|trainer.py:803] 2025-04-26 17:37:31,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:32,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:32,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6986
6912
6962
[WARNING|trainer.py:803] 2025-04-26 17:37:33,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:37:33,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:33,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6987
6913
6963
[WARNING|trainer.py:803] 2025-04-26 17:37:34,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:34,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:34,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6988
6914
6964
[WARNING|trainer.py:803] 2025-04-26 17:37:35,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:35,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:35,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6989
6915
6965
[WARNING|trainer.py:803] 2025-04-26 17:37:37,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:37,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:37,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6990
6916
6966
[WARNING|trainer.py:803] 2025-04-26 17:37:38,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:38,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:38,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6991
6917
6967
[WARNING|trainer.py:803] 2025-04-26 17:37:39,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:39,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:37:39,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6992
6968
6918
[WARNING|trainer.py:803] 2025-04-26 17:37:40,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:40,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:40,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6993
6969
6919
[WARNING|trainer.py:803] 2025-04-26 17:37:42,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:42,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:42,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6994
6970
6920
[WARNING|trainer.py:803] 2025-04-26 17:37:43,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:43,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:43,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6995
6971
6921
[WARNING|trainer.py:803] 2025-04-26 17:37:44,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:44,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:44,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6996
6972
6922
[WARNING|trainer.py:803] 2025-04-26 17:37:45,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:45,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:46,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6997
6973
6923
[WARNING|trainer.py:803] 2025-04-26 17:37:47,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:37:47,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:47,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6998
6974
6924
[WARNING|trainer.py:803] 2025-04-26 17:37:48,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:48,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:48,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6999
6975
6925
[WARNING|trainer.py:803] 2025-04-26 17:37:49,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:49,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:49,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7000
6976
6926
[WARNING|trainer.py:803] 2025-04-26 17:37:50,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:50,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:51,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7001
6977
6927
[WARNING|trainer.py:803] 2025-04-26 17:37:52,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:37:52,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:52,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7002
6978
6928
[WARNING|trainer.py:803] 2025-04-26 17:37:53,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:37:53,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:53,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7003
6979
6929
[WARNING|trainer.py:803] 2025-04-26 17:37:54,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:54,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:54,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6980
7004
6930
[WARNING|trainer.py:803] 2025-04-26 17:37:55,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:55,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:56,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6981
7005
6931
[WARNING|trainer.py:803] 2025-04-26 17:37:57,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:57,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:57,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6982
7006
6932
[WARNING|trainer.py:803] 2025-04-26 17:37:58,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:37:58,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:37:58,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6983
7007
6933
[WARNING|trainer.py:803] 2025-04-26 17:37:59,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:59,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:37:59,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6984
7008
6934
[WARNING|trainer.py:803] 2025-04-26 17:38:00,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:00,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:01,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6985
7009
6935
[WARNING|trainer.py:803] 2025-04-26 17:38:02,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:02,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:02,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6986
7010
6936
[WARNING|trainer.py:803] 2025-04-26 17:38:03,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:03,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6987
[WARNING|trainer.py:803] 2025-04-26 17:38:03,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7011
6937
[WARNING|trainer.py:803] 2025-04-26 17:38:04,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:04,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6988
[WARNING|trainer.py:803] 2025-04-26 17:38:05,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7012
6938
[WARNING|trainer.py:803] 2025-04-26 17:38:05,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:05,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6989
[WARNING|trainer.py:803] 2025-04-26 17:38:06,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7013
6939
[WARNING|trainer.py:803] 2025-04-26 17:38:07,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:07,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6990
[WARNING|trainer.py:803] 2025-04-26 17:38:07,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7014
6940
[WARNING|trainer.py:803] 2025-04-26 17:38:08,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:08,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6991
[WARNING|trainer.py:803] 2025-04-26 17:38:08,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7015
6941
[WARNING|trainer.py:803] 2025-04-26 17:38:09,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:09,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6992
[WARNING|trainer.py:803] 2025-04-26 17:38:10,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7016
6942
[WARNING|trainer.py:803] 2025-04-26 17:38:10,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:11,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6993
[WARNING|trainer.py:803] 2025-04-26 17:38:11,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7017
6943
[WARNING|trainer.py:803] 2025-04-26 17:38:11,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:12,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6994
[WARNING|trainer.py:803] 2025-04-26 17:38:12,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7018
6944
[WARNING|trainer.py:803] 2025-04-26 17:38:13,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6995
[WARNING|trainer.py:803] 2025-04-26 17:38:13,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:13,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7019
6945
[WARNING|trainer.py:803] 2025-04-26 17:38:14,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6996
[WARNING|trainer.py:803] 2025-04-26 17:38:14,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:14,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7020
6946
[WARNING|trainer.py:803] 2025-04-26 17:38:15,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6997
[WARNING|trainer.py:803] 2025-04-26 17:38:16,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:16,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7021
6947
[WARNING|trainer.py:803] 2025-04-26 17:38:16,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6998
[WARNING|trainer.py:803] 2025-04-26 17:38:17,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:17,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7022
6948
[WARNING|trainer.py:803] 2025-04-26 17:38:18,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6999
[WARNING|trainer.py:803] 2025-04-26 17:38:18,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:18,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7023
6949
[WARNING|trainer.py:803] 2025-04-26 17:38:19,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7000
[WARNING|trainer.py:803] 2025-04-26 17:38:19,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:19,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7024
6950
[WARNING|trainer.py:803] 2025-04-26 17:38:20,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7001
[WARNING|trainer.py:803] 2025-04-26 17:38:21,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:21,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7025
6951
[WARNING|trainer.py:803] 2025-04-26 17:38:21,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7002
[WARNING|trainer.py:803] 2025-04-26 17:38:22,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:22,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7026
6952
[WARNING|trainer.py:803] 2025-04-26 17:38:23,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7003
[WARNING|trainer.py:803] 2025-04-26 17:38:23,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:23,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7027
6953
[WARNING|trainer.py:803] 2025-04-26 17:38:24,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7004
[WARNING|trainer.py:803] 2025-04-26 17:38:24,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:25,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7028
6954
[WARNING|trainer.py:803] 2025-04-26 17:38:25,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7005
[WARNING|trainer.py:803] 2025-04-26 17:38:26,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:26,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7029
6955
[WARNING|trainer.py:803] 2025-04-26 17:38:26,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7006
[WARNING|trainer.py:803] 2025-04-26 17:38:27,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:27,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7030
6956
[WARNING|trainer.py:803] 2025-04-26 17:38:27,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7007
[WARNING|trainer.py:803] 2025-04-26 17:38:28,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:38:28,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7031
6957
[WARNING|trainer.py:803] 2025-04-26 17:38:29,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7008
[WARNING|trainer.py:803] 2025-04-26 17:38:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:29,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7032
6958
[WARNING|trainer.py:803] 2025-04-26 17:38:30,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7009
[WARNING|trainer.py:803] 2025-04-26 17:38:31,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:31,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7033
6959
[WARNING|trainer.py:803] 2025-04-26 17:38:31,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7010
[WARNING|trainer.py:803] 2025-04-26 17:38:32,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:32,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7034
6960
[WARNING|trainer.py:803] 2025-04-26 17:38:32,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7011
[WARNING|trainer.py:803] 2025-04-26 17:38:33,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:33,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7035
[WARNING|trainer.py:803] 2025-04-26 17:38:34,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6961
7012
[WARNING|trainer.py:803] 2025-04-26 17:38:34,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:34,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:35,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7036
6962
7013
[WARNING|trainer.py:803] 2025-04-26 17:38:36,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:36,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:36,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7037
6963
7014
[WARNING|trainer.py:803] 2025-04-26 17:38:37,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:37,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:37,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7038
6964
7015
[WARNING|trainer.py:803] 2025-04-26 17:38:38,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:38,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:39,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7039
6965
7016
[WARNING|trainer.py:803] 2025-04-26 17:38:39,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:39,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7040
6966
7017
[WARNING|trainer.py:803] 2025-04-26 17:38:41,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:41,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:41,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7041
6967
7018
[WARNING|trainer.py:803] 2025-04-26 17:38:42,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:42,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:42,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7042
6968
7019
[WARNING|trainer.py:803] 2025-04-26 17:38:43,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:43,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:44,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7043
6969
7020
[WARNING|trainer.py:803] 2025-04-26 17:38:44,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:44,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:45,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7044
6970
7021
[WARNING|trainer.py:803] 2025-04-26 17:38:46,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:46,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:46,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7045
6971
7022
[WARNING|trainer.py:803] 2025-04-26 17:38:47,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:47,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:47,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7046
6972
7023
[WARNING|trainer.py:803] 2025-04-26 17:38:48,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:48,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:49,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7047
6973
7024
[WARNING|trainer.py:803] 2025-04-26 17:38:49,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:49,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:50,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6974
7048
7025
[WARNING|trainer.py:803] 2025-04-26 17:38:51,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:51,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:38:51,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6975
7049
7026
[WARNING|trainer.py:803] 2025-04-26 17:38:52,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:52,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:38:52,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6976
7050
7027
[WARNING|trainer.py:803] 2025-04-26 17:38:53,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:38:53,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:54,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6977
7051
7028
[WARNING|trainer.py:803] 2025-04-26 17:38:54,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:55,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:55,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6978
7052
7029
[WARNING|trainer.py:803] 2025-04-26 17:38:56,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:56,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:56,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6979
7053
7030
[WARNING|trainer.py:803] 2025-04-26 17:38:57,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:57,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:57,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6980
7054
7031
[WARNING|trainer.py:803] 2025-04-26 17:38:58,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:38:59,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6981
7055
7032
[WARNING|trainer.py:803] 2025-04-26 17:38:59,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:00,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:00,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6982
7056
7033
[WARNING|trainer.py:803] 2025-04-26 17:39:01,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:01,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:01,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6983
7057
7034
[WARNING|trainer.py:803] 2025-04-26 17:39:02,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:02,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:02,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6984
7058
7035
[WARNING|trainer.py:803] 2025-04-26 17:39:03,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:03,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:04,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6985
7059
7036
[WARNING|trainer.py:803] 2025-04-26 17:39:05,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:05,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:05,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6986
7060
7037
[WARNING|trainer.py:803] 2025-04-26 17:39:06,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:06,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:06,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6987
7061
7038
[WARNING|trainer.py:803] 2025-04-26 17:39:07,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:07,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:07,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6988
7062
7039
[WARNING|trainer.py:803] 2025-04-26 17:39:08,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:08,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:09,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6989
7063
7040
[WARNING|trainer.py:803] 2025-04-26 17:39:10,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:10,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:10,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6990
7064
7041
[WARNING|trainer.py:803] 2025-04-26 17:39:11,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:11,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:11,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6991
7065
7042
[WARNING|trainer.py:803] 2025-04-26 17:39:12,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:12,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:12,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6992
7066
7043
[WARNING|trainer.py:803] 2025-04-26 17:39:13,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:13,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:14,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6993
7067
7044
[WARNING|trainer.py:803] 2025-04-26 17:39:15,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:15,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:15,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6994
7068
7045
[WARNING|trainer.py:803] 2025-04-26 17:39:16,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:16,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:16,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6995
7069
7046
[WARNING|trainer.py:803] 2025-04-26 17:39:17,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:17,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:17,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6996
7070
7047
[WARNING|trainer.py:803] 2025-04-26 17:39:18,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:18,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:19,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6997
7071
7048
[WARNING|trainer.py:803] 2025-04-26 17:39:20,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:20,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:20,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6998
7072
7049
[WARNING|trainer.py:803] 2025-04-26 17:39:21,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:21,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:21,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6999
7073
7050
[WARNING|trainer.py:803] 2025-04-26 17:39:22,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:22,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:22,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7000
7074
7051
[WARNING|trainer.py:803] 2025-04-26 17:39:23,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:23,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:24,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7001
7075
7052
[WARNING|trainer.py:803] 2025-04-26 17:39:25,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:39:25,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:25,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7002
7076
7053
[WARNING|trainer.py:803] 2025-04-26 17:39:26,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:26,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:26,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7003
7077
7054
[WARNING|trainer.py:803] 2025-04-26 17:39:27,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:27,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:27,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7004
7078
7055
[WARNING|trainer.py:803] 2025-04-26 17:39:28,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:28,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:29,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7005
7079
7056
[WARNING|trainer.py:803] 2025-04-26 17:39:30,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:30,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:30,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7080
7006
7057
[WARNING|trainer.py:803] 2025-04-26 17:39:31,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:31,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:31,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7081
7058
7007
[WARNING|trainer.py:803] 2025-04-26 17:39:32,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:32,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:32,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7082
7059
7008
[WARNING|trainer.py:803] 2025-04-26 17:39:34,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:34,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:34,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7083
7060
7009
[WARNING|trainer.py:803] 2025-04-26 17:39:35,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:35,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:35,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7084
7061
7010
[WARNING|trainer.py:803] 2025-04-26 17:39:36,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 17:39:36,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:36,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7085
7062
7011
[WARNING|trainer.py:803] 2025-04-26 17:39:37,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:37,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:37,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7086
7063
7012
[WARNING|trainer.py:803] 2025-04-26 17:39:39,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:39,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:39,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7087
7064
7013
[WARNING|trainer.py:803] 2025-04-26 17:39:40,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:40,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:40,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7088
7065
7014
[WARNING|trainer.py:803] 2025-04-26 17:39:41,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:41,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:41,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7089
7066
7015
[WARNING|trainer.py:803] 2025-04-26 17:39:42,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:42,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:42,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7067
7090
7016
[WARNING|trainer.py:803] 2025-04-26 17:39:44,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:44,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:44,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7068
7091
7017
[WARNING|trainer.py:803] 2025-04-26 17:39:45,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:39:45,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:45,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7069
7092
7018
[WARNING|trainer.py:803] 2025-04-26 17:39:46,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:39:46,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:46,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7070
7093
7019
[WARNING|trainer.py:803] 2025-04-26 17:39:47,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:47,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:48,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7071
7094
7020
[WARNING|trainer.py:803] 2025-04-26 17:39:49,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:49,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:49,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7072
7095
7021
[WARNING|trainer.py:803] 2025-04-26 17:39:50,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:50,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:50,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7073
7096
7022
[WARNING|trainer.py:803] 2025-04-26 17:39:51,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:51,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:51,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7074
7097
7023
[WARNING|trainer.py:803] 2025-04-26 17:39:52,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:52,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:53,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7075
7098
7024
[WARNING|trainer.py:803] 2025-04-26 17:39:53,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:54,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7076
[WARNING|trainer.py:803] 2025-04-26 17:39:54,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7099
7025
[WARNING|trainer.py:803] 2025-04-26 17:39:55,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:55,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7077
[WARNING|trainer.py:803] 2025-04-26 17:39:55,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7100
7026
[WARNING|trainer.py:803] 2025-04-26 17:39:56,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:56,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7078
[WARNING|trainer.py:803] 2025-04-26 17:39:56,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7101
7027
[WARNING|trainer.py:803] 2025-04-26 17:39:57,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:39:57,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7079
[WARNING|trainer.py:803] 2025-04-26 17:39:58,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7102
7028
[WARNING|trainer.py:803] 2025-04-26 17:39:58,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:39:59,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7080
[WARNING|trainer.py:803] 2025-04-26 17:39:59,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7103
7029
[WARNING|trainer.py:803] 2025-04-26 17:40:00,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:00,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7081
[WARNING|trainer.py:803] 2025-04-26 17:40:00,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7104
7030
[WARNING|trainer.py:803] 2025-04-26 17:40:01,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:01,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7082
[WARNING|trainer.py:803] 2025-04-26 17:40:01,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7105
7031
[WARNING|trainer.py:803] 2025-04-26 17:40:02,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:02,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7083
[WARNING|trainer.py:803] 2025-04-26 17:40:03,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7106
7032
[WARNING|trainer.py:803] 2025-04-26 17:40:03,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7084
[WARNING|trainer.py:803] 2025-04-26 17:40:04,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:04,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7107
7033
[WARNING|trainer.py:803] 2025-04-26 17:40:04,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
7085
[WARNING|trainer.py:803] 2025-04-26 17:40:05,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:05,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7108
7034
[WARNING|trainer.py:803] 2025-04-26 17:40:06,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7086
[WARNING|trainer.py:803] 2025-04-26 17:40:06,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:06,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7109
7035
[WARNING|trainer.py:803] 2025-04-26 17:40:07,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7087
[WARNING|trainer.py:803] 2025-04-26 17:40:07,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:08,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7110
7036
[WARNING|trainer.py:803] 2025-04-26 17:40:08,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7088
[WARNING|trainer.py:803] 2025-04-26 17:40:09,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:09,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7111
7037
[WARNING|trainer.py:803] 2025-04-26 17:40:09,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7089
[WARNING|trainer.py:803] 2025-04-26 17:40:10,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:10,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7112
7038
[WARNING|trainer.py:803] 2025-04-26 17:40:11,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7090
[WARNING|trainer.py:803] 2025-04-26 17:40:11,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:11,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7113
7039
[WARNING|trainer.py:803] 2025-04-26 17:40:12,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7091
[WARNING|trainer.py:803] 2025-04-26 17:40:12,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:13,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7114
7040
[WARNING|trainer.py:803] 2025-04-26 17:40:13,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7092
[WARNING|trainer.py:803] 2025-04-26 17:40:14,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:14,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7115
[WARNING|trainer.py:803] 2025-04-26 17:40:14,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7041
7093
[WARNING|trainer.py:803] 2025-04-26 17:40:15,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7116
[WARNING|trainer.py:803] 2025-04-26 17:40:15,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:40:16,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7042
7094
[WARNING|trainer.py:803] 2025-04-26 17:40:16,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7117
[WARNING|trainer.py:803] 2025-04-26 17:40:16,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:17,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7043
7095
[WARNING|trainer.py:803] 2025-04-26 17:40:17,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7118
[WARNING|trainer.py:803] 2025-04-26 17:40:18,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:18,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7044
7096
[WARNING|trainer.py:803] 2025-04-26 17:40:18,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7119
[WARNING|trainer.py:803] 2025-04-26 17:40:19,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:19,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7045
7097
[WARNING|trainer.py:803] 2025-04-26 17:40:20,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7120
[WARNING|trainer.py:803] 2025-04-26 17:40:20,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:21,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7046
7098
[WARNING|trainer.py:803] 2025-04-26 17:40:21,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7121
[WARNING|trainer.py:803] 2025-04-26 17:40:22,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:22,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7047
[WARNING|trainer.py:803] 2025-04-26 17:40:22,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7099
7122
[WARNING|trainer.py:803] 2025-04-26 17:40:23,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:23,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7048
[WARNING|trainer.py:803] 2025-04-26 17:40:23,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7100
7123
[WARNING|trainer.py:803] 2025-04-26 17:40:24,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:24,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7049
7101
[WARNING|trainer.py:803] 2025-04-26 17:40:25,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7124
[WARNING|trainer.py:803] 2025-04-26 17:40:25,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:40:25,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7050
7102
[WARNING|trainer.py:803] 2025-04-26 17:40:26,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7125
[WARNING|trainer.py:803] 2025-04-26 17:40:26,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:27,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7051
7103
[WARNING|trainer.py:803] 2025-04-26 17:40:27,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7126
[WARNING|trainer.py:803] 2025-04-26 17:40:28,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:28,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7052
7104
[WARNING|trainer.py:803] 2025-04-26 17:40:28,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7127
[WARNING|trainer.py:803] 2025-04-26 17:40:29,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:29,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7053
7105
[WARNING|trainer.py:803] 2025-04-26 17:40:30,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7128
[WARNING|trainer.py:803] 2025-04-26 17:40:30,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:30,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7054
7106
[WARNING|trainer.py:803] 2025-04-26 17:40:31,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7129
[WARNING|trainer.py:803] 2025-04-26 17:40:31,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:32,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7055
7107
[WARNING|trainer.py:803] 2025-04-26 17:40:32,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7130
[WARNING|trainer.py:803] 2025-04-26 17:40:33,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:33,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7056
7108
[WARNING|trainer.py:803] 2025-04-26 17:40:33,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7131
[WARNING|trainer.py:803] 2025-04-26 17:40:34,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:34,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7057
7109
[WARNING|trainer.py:803] 2025-04-26 17:40:35,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7132
[WARNING|trainer.py:803] 2025-04-26 17:40:35,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:35,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7058
7110
[WARNING|trainer.py:803] 2025-04-26 17:40:36,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7133
[WARNING|trainer.py:803] 2025-04-26 17:40:36,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:37,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7059
7111
[WARNING|trainer.py:803] 2025-04-26 17:40:37,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7134
[WARNING|trainer.py:803] 2025-04-26 17:40:38,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:38,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7060
7112
[WARNING|trainer.py:803] 2025-04-26 17:40:38,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7135
[WARNING|trainer.py:803] 2025-04-26 17:40:39,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:39,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7061
7113
[WARNING|trainer.py:803] 2025-04-26 17:40:39,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7136
[WARNING|trainer.py:803] 2025-04-26 17:40:40,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:40,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7062
7114
[WARNING|trainer.py:803] 2025-04-26 17:40:41,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7137
[WARNING|trainer.py:803] 2025-04-26 17:40:41,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:41,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7063
7115
[WARNING|trainer.py:803] 2025-04-26 17:40:42,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7138
[WARNING|trainer.py:803] 2025-04-26 17:40:42,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:43,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7064
7116
[WARNING|trainer.py:803] 2025-04-26 17:40:43,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7139
[WARNING|trainer.py:803] 2025-04-26 17:40:44,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:44,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7065
7117
[WARNING|trainer.py:803] 2025-04-26 17:40:44,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7140
[WARNING|trainer.py:803] 2025-04-26 17:40:45,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:45,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7066
7118
[WARNING|trainer.py:803] 2025-04-26 17:40:46,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7141
[WARNING|trainer.py:803] 2025-04-26 17:40:46,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:46,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7067
7119
[WARNING|trainer.py:803] 2025-04-26 17:40:47,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7142
[WARNING|trainer.py:803] 2025-04-26 17:40:47,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:40:48,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7068
7120
[WARNING|trainer.py:803] 2025-04-26 17:40:48,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7143
[WARNING|trainer.py:803] 2025-04-26 17:40:49,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:40:49,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7069
7121
[WARNING|trainer.py:803] 2025-04-26 17:40:49,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7144
[WARNING|trainer.py:803] 2025-04-26 17:40:50,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:50,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7070
7122
[WARNING|trainer.py:803] 2025-04-26 17:40:51,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7145
[WARNING|trainer.py:803] 2025-04-26 17:40:51,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:51,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7071
7123
[WARNING|trainer.py:803] 2025-04-26 17:40:52,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7146
[WARNING|trainer.py:803] 2025-04-26 17:40:52,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:53,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7072
7124
[WARNING|trainer.py:803] 2025-04-26 17:40:53,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:40:54,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7147
[WARNING|trainer.py:803] 2025-04-26 17:40:54,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7073
7125
[WARNING|trainer.py:803] 2025-04-26 17:40:54,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:55,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7148
[WARNING|trainer.py:803] 2025-04-26 17:40:55,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7074
7126
[WARNING|trainer.py:803] 2025-04-26 17:40:56,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:56,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7149
[WARNING|trainer.py:803] 2025-04-26 17:40:56,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7075
7127
[WARNING|trainer.py:803] 2025-04-26 17:40:57,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:40:57,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7150
[WARNING|trainer.py:803] 2025-04-26 17:40:58,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7076
7128
[WARNING|trainer.py:803] 2025-04-26 17:40:58,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7151
[WARNING|trainer.py:803] 2025-04-26 17:40:59,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:40:59,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7077
7129
[WARNING|trainer.py:803] 2025-04-26 17:40:59,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7152
[WARNING|trainer.py:803] 2025-04-26 17:41:00,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:00,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7078
7130
[WARNING|trainer.py:803] 2025-04-26 17:41:01,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7153
[WARNING|trainer.py:803] 2025-04-26 17:41:01,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:01,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7079
7131
[WARNING|trainer.py:803] 2025-04-26 17:41:02,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7154
[WARNING|trainer.py:803] 2025-04-26 17:41:02,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:02,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7080
7132
[WARNING|trainer.py:803] 2025-04-26 17:41:03,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7155
[WARNING|trainer.py:803] 2025-04-26 17:41:04,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:04,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7081
7133
[WARNING|trainer.py:803] 2025-04-26 17:41:04,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7156
[WARNING|trainer.py:803] 2025-04-26 17:41:05,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:05,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7082
7134
[WARNING|trainer.py:803] 2025-04-26 17:41:06,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7157
[WARNING|trainer.py:803] 2025-04-26 17:41:06,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:06,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7083
7135
[WARNING|trainer.py:803] 2025-04-26 17:41:07,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7158
[WARNING|trainer.py:803] 2025-04-26 17:41:07,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:07,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7084
7136
[WARNING|trainer.py:803] 2025-04-26 17:41:08,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7159
[WARNING|trainer.py:803] 2025-04-26 17:41:08,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 17:41:09,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7085
7137
[WARNING|trainer.py:803] 2025-04-26 17:41:09,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7160
[WARNING|trainer.py:803] 2025-04-26 17:41:10,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:10,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7086
7138
[WARNING|trainer.py:803] 2025-04-26 17:41:10,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7161
[WARNING|trainer.py:803] 2025-04-26 17:41:11,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:11,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7087
7139
[WARNING|trainer.py:803] 2025-04-26 17:41:12,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7162
[WARNING|trainer.py:803] 2025-04-26 17:41:12,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:12,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7088
7140
[WARNING|trainer.py:803] 2025-04-26 17:41:13,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7163
[WARNING|trainer.py:803] 2025-04-26 17:41:13,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:14,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7089
7141
[WARNING|trainer.py:803] 2025-04-26 17:41:14,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7164
[WARNING|trainer.py:803] 2025-04-26 17:41:15,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:15,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7090
7142
[WARNING|trainer.py:803] 2025-04-26 17:41:15,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7165
[WARNING|trainer.py:803] 2025-04-26 17:41:16,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:16,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7091
7143
[WARNING|trainer.py:803] 2025-04-26 17:41:17,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7166
[WARNING|trainer.py:803] 2025-04-26 17:41:17,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:17,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7092
7144
[WARNING|trainer.py:803] 2025-04-26 17:41:18,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7167
[WARNING|trainer.py:803] 2025-04-26 17:41:18,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:19,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7093
7145
[WARNING|trainer.py:803] 2025-04-26 17:41:19,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7168
[WARNING|trainer.py:803] 2025-04-26 17:41:20,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:20,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7094
7146
[WARNING|trainer.py:803] 2025-04-26 17:41:20,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7169
[WARNING|trainer.py:803] 2025-04-26 17:41:21,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:21,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7095
7147
[WARNING|trainer.py:803] 2025-04-26 17:41:22,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7170
[WARNING|trainer.py:803] 2025-04-26 17:41:22,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:22,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7096
7148
[WARNING|trainer.py:803] 2025-04-26 17:41:23,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7171
[WARNING|trainer.py:803] 2025-04-26 17:41:23,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:24,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7097
7149
[WARNING|trainer.py:803] 2025-04-26 17:41:24,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7172
[WARNING|trainer.py:803] 2025-04-26 17:41:25,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:25,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7098
7150
[WARNING|trainer.py:803] 2025-04-26 17:41:25,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7173
[WARNING|trainer.py:803] 2025-04-26 17:41:26,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:26,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7099
7151
[WARNING|trainer.py:803] 2025-04-26 17:41:27,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7174
[WARNING|trainer.py:803] 2025-04-26 17:41:27,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:27,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7100
7152
[WARNING|trainer.py:803] 2025-04-26 17:41:28,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7175
[WARNING|trainer.py:803] 2025-04-26 17:41:28,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:41:28,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7101
7153
[WARNING|trainer.py:803] 2025-04-26 17:41:29,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7176
[WARNING|trainer.py:803] 2025-04-26 17:41:30,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:30,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7102
7154
[WARNING|trainer.py:803] 2025-04-26 17:41:30,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7177
[WARNING|trainer.py:803] 2025-04-26 17:41:31,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:31,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7103
7155
[WARNING|trainer.py:803] 2025-04-26 17:41:32,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7178
[WARNING|trainer.py:803] 2025-04-26 17:41:32,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:32,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7104
7156
[WARNING|trainer.py:803] 2025-04-26 17:41:33,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7179
[WARNING|trainer.py:803] 2025-04-26 17:41:33,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:34,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7105
7157
[WARNING|trainer.py:803] 2025-04-26 17:41:34,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7180
[WARNING|trainer.py:803] 2025-04-26 17:41:35,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:35,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7106
7158
[WARNING|trainer.py:803] 2025-04-26 17:41:35,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7181
[WARNING|trainer.py:803] 2025-04-26 17:41:36,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:36,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7107
7159
[WARNING|trainer.py:803] 2025-04-26 17:41:36,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7182
[WARNING|trainer.py:803] 2025-04-26 17:41:37,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:37,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7108
[WARNING|trainer.py:803] 2025-04-26 17:41:38,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7160
7183
[WARNING|trainer.py:803] 2025-04-26 17:41:38,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:39,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7109
[WARNING|trainer.py:803] 2025-04-26 17:41:39,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7161
7184
[WARNING|trainer.py:803] 2025-04-26 17:41:40,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:41:40,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7110
[WARNING|trainer.py:803] 2025-04-26 17:41:40,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7162
7185
[WARNING|trainer.py:803] 2025-04-26 17:41:41,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:41,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7111
[WARNING|trainer.py:803] 2025-04-26 17:41:41,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7163
7186
[WARNING|trainer.py:803] 2025-04-26 17:41:42,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:42,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7112
[WARNING|trainer.py:803] 2025-04-26 17:41:43,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7164
7187
[WARNING|trainer.py:803] 2025-04-26 17:41:43,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:44,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7113
[WARNING|trainer.py:803] 2025-04-26 17:41:44,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7165
7188
[WARNING|trainer.py:803] 2025-04-26 17:41:44,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:45,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7114
[WARNING|trainer.py:803] 2025-04-26 17:41:45,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7166
7189
[WARNING|trainer.py:803] 2025-04-26 17:41:46,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:46,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7115
[WARNING|trainer.py:803] 2025-04-26 17:41:46,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7167
7190
[WARNING|trainer.py:803] 2025-04-26 17:41:47,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:47,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7116
[WARNING|trainer.py:803] 2025-04-26 17:41:48,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7168
7191
[WARNING|trainer.py:803] 2025-04-26 17:41:48,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:49,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7117
[WARNING|trainer.py:803] 2025-04-26 17:41:49,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7169
7192
[WARNING|trainer.py:803] 2025-04-26 17:41:49,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:50,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7118
[WARNING|trainer.py:803] 2025-04-26 17:41:50,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7170
7193
[WARNING|trainer.py:803] 2025-04-26 17:41:51,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:51,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7119
[WARNING|trainer.py:803] 2025-04-26 17:41:51,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7171
7194
[WARNING|trainer.py:803] 2025-04-26 17:41:52,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7120
[WARNING|trainer.py:803] 2025-04-26 17:41:52,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:53,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7172
7195
[WARNING|trainer.py:803] 2025-04-26 17:41:53,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7121
[WARNING|trainer.py:803] 2025-04-26 17:41:54,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:41:54,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7173
7196
[WARNING|trainer.py:803] 2025-04-26 17:41:54,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7122
[WARNING|trainer.py:803] 2025-04-26 17:41:55,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:55,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7174
7197
[WARNING|trainer.py:803] 2025-04-26 17:41:56,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7123
[WARNING|trainer.py:803] 2025-04-26 17:41:56,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:41:56,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7175
7198
[WARNING|trainer.py:803] 2025-04-26 17:41:57,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7124
[WARNING|trainer.py:803] 2025-04-26 17:41:57,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:41:58,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7176
7199
[WARNING|trainer.py:803] 2025-04-26 17:41:58,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7125
[WARNING|trainer.py:803] 2025-04-26 17:41:58,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:41:59,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7177
7200
[WARNING|trainer.py:803] 2025-04-26 17:41:59,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7126
[WARNING|trainer.py:803] 2025-04-26 17:42:00,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:00,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7178
7201
[WARNING|trainer.py:803] 2025-04-26 17:42:00,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7127
[WARNING|trainer.py:803] 2025-04-26 17:42:01,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:01,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7179
7202
[WARNING|trainer.py:803] 2025-04-26 17:42:02,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7128
[WARNING|trainer.py:803] 2025-04-26 17:42:02,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:03,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7180
7203
[WARNING|trainer.py:803] 2025-04-26 17:42:03,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7129
[WARNING|trainer.py:803] 2025-04-26 17:42:03,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:04,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7181
7204
[WARNING|trainer.py:803] 2025-04-26 17:42:04,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:05,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7130
[WARNING|trainer.py:803] 2025-04-26 17:42:05,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7182
7205
[WARNING|trainer.py:803] 2025-04-26 17:42:06,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:06,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7131
[WARNING|trainer.py:803] 2025-04-26 17:42:06,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7183
7206
[WARNING|trainer.py:803] 2025-04-26 17:42:07,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:07,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7132
[WARNING|trainer.py:803] 2025-04-26 17:42:07,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7184
7207
[WARNING|trainer.py:803] 2025-04-26 17:42:08,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:08,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7133
[WARNING|trainer.py:803] 2025-04-26 17:42:09,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7185
7208
[WARNING|trainer.py:803] 2025-04-26 17:42:09,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:10,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7134
[WARNING|trainer.py:803] 2025-04-26 17:42:10,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7186
7209
[WARNING|trainer.py:803] 2025-04-26 17:42:11,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:11,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7135
[WARNING|trainer.py:803] 2025-04-26 17:42:11,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7187
7210
[WARNING|trainer.py:803] 2025-04-26 17:42:12,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:12,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7136
[WARNING|trainer.py:803] 2025-04-26 17:42:12,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7188
7211
[WARNING|trainer.py:803] 2025-04-26 17:42:13,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:13,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7137
[WARNING|trainer.py:803] 2025-04-26 17:42:14,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7189
7212
[WARNING|trainer.py:803] 2025-04-26 17:42:14,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:15,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7138
[WARNING|trainer.py:803] 2025-04-26 17:42:15,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7190
7213
[WARNING|trainer.py:803] 2025-04-26 17:42:16,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:16,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7139
[WARNING|trainer.py:803] 2025-04-26 17:42:16,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7191
7214
[WARNING|trainer.py:803] 2025-04-26 17:42:17,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:17,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7140
[WARNING|trainer.py:803] 2025-04-26 17:42:17,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7192
7215
[WARNING|trainer.py:803] 2025-04-26 17:42:18,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:18,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7141
[WARNING|trainer.py:803] 2025-04-26 17:42:19,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7193
7216
[WARNING|trainer.py:803] 2025-04-26 17:42:19,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:20,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:20,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7142
7194
7217
[WARNING|trainer.py:803] 2025-04-26 17:42:21,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:21,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:21,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7143
7195
7218
[WARNING|trainer.py:803] 2025-04-26 17:42:22,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:22,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:22,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7144
7196
7219
[WARNING|trainer.py:803] 2025-04-26 17:42:23,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:23,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:24,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7145
7197
7220
[WARNING|trainer.py:803] 2025-04-26 17:42:25,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:25,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:25,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7146
7198
7221
[WARNING|trainer.py:803] 2025-04-26 17:42:26,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:26,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:26,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7147
7199
7222
[WARNING|trainer.py:803] 2025-04-26 17:42:27,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:27,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:27,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7148
7200
7223
[WARNING|trainer.py:803] 2025-04-26 17:42:28,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:28,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:29,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7149
7201
7224
[WARNING|trainer.py:803] 2025-04-26 17:42:30,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:30,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:30,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7150
7202
7225
[WARNING|trainer.py:803] 2025-04-26 17:42:31,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:31,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:31,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7151
7203
7226
[WARNING|trainer.py:803] 2025-04-26 17:42:32,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:32,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:32,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7152
7227
7204
[WARNING|trainer.py:803] 2025-04-26 17:42:33,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:33,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:33,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7153
7228
7205
[WARNING|trainer.py:803] 2025-04-26 17:42:34,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:35,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:35,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7154
7229
7206
[WARNING|trainer.py:803] 2025-04-26 17:42:36,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:36,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:36,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7155
7230
7207
[WARNING|trainer.py:803] 2025-04-26 17:42:37,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:37,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:37,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7156
7231
7208
[WARNING|trainer.py:803] 2025-04-26 17:42:38,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:38,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:39,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7157
7232
7209
[WARNING|trainer.py:803] 2025-04-26 17:42:39,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:40,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:40,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7158
7233
7210
[WARNING|trainer.py:803] 2025-04-26 17:42:41,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:41,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:41,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7159
7234
7211
[WARNING|trainer.py:803] 2025-04-26 17:42:42,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:42,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:42,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7160
7235
7212
[WARNING|trainer.py:803] 2025-04-26 17:42:43,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:43,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:44,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7161
7236
7213
[WARNING|trainer.py:803] 2025-04-26 17:42:45,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:45,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:45,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7162
7237
7214
[WARNING|trainer.py:803] 2025-04-26 17:42:46,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:46,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:46,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7163
7238
7215
[WARNING|trainer.py:803] 2025-04-26 17:42:47,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:47,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:47,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7164
7239
7216
[WARNING|trainer.py:803] 2025-04-26 17:42:48,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:48,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:42:49,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7165
7240
7217
[WARNING|trainer.py:803] 2025-04-26 17:42:50,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:50,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:50,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7241
7166
7218
[WARNING|trainer.py:803] 2025-04-26 17:42:51,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:51,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:51,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7242
7167
7219
[WARNING|trainer.py:803] 2025-04-26 17:42:52,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:52,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:52,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7243
7168
7220
[WARNING|trainer.py:803] 2025-04-26 17:42:53,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:53,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:54,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7244
7169
7221
[WARNING|trainer.py:803] 2025-04-26 17:42:55,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:55,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:55,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7245
7170
7222
[WARNING|trainer.py:803] 2025-04-26 17:42:56,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:56,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:56,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7246
7171
7223
[WARNING|trainer.py:803] 2025-04-26 17:42:57,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:57,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:42:57,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7247
7172
7224
[WARNING|trainer.py:803] 2025-04-26 17:42:58,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:42:59,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:42:59,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7248
7173
7225
[WARNING|trainer.py:803] 2025-04-26 17:42:59,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:00,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:00,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7249
7226
7174
[WARNING|trainer.py:803] 2025-04-26 17:43:01,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:01,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:01,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7250
7227
7175
[WARNING|trainer.py:803] 2025-04-26 17:43:02,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:02,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7251
[WARNING|trainer.py:803] 2025-04-26 17:43:02,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7228
7176
[WARNING|trainer.py:803] 2025-04-26 17:43:03,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7252
[WARNING|trainer.py:803] 2025-04-26 17:43:04,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:04,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7229
7177
[WARNING|trainer.py:803] 2025-04-26 17:43:04,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7253
[WARNING|trainer.py:803] 2025-04-26 17:43:05,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:05,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7230
7178
[WARNING|trainer.py:803] 2025-04-26 17:43:06,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7254
[WARNING|trainer.py:803] 2025-04-26 17:43:06,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:06,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7231
7179
[WARNING|trainer.py:803] 2025-04-26 17:43:07,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7255
[WARNING|trainer.py:803] 2025-04-26 17:43:07,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:07,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7232
7180
[WARNING|trainer.py:803] 2025-04-26 17:43:08,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7256
[WARNING|trainer.py:803] 2025-04-26 17:43:08,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:09,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7233
7181
[WARNING|trainer.py:803] 2025-04-26 17:43:09,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7257
[WARNING|trainer.py:803] 2025-04-26 17:43:10,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:10,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7234
7182
[WARNING|trainer.py:803] 2025-04-26 17:43:10,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7258
[WARNING|trainer.py:803] 2025-04-26 17:43:11,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:11,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7235
[WARNING|trainer.py:803] 2025-04-26 17:43:12,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7183
7259
[WARNING|trainer.py:803] 2025-04-26 17:43:12,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:12,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7236
[WARNING|trainer.py:803] 2025-04-26 17:43:13,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7184
7260
[WARNING|trainer.py:803] 2025-04-26 17:43:13,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:14,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7237
[WARNING|trainer.py:803] 2025-04-26 17:43:14,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7185
7261
[WARNING|trainer.py:803] 2025-04-26 17:43:15,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:15,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7238
[WARNING|trainer.py:803] 2025-04-26 17:43:15,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7186
7262
[WARNING|trainer.py:803] 2025-04-26 17:43:16,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7239
[WARNING|trainer.py:803] 2025-04-26 17:43:16,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:17,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7187
7263
[WARNING|trainer.py:803] 2025-04-26 17:43:17,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7240
[WARNING|trainer.py:803] 2025-04-26 17:43:17,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:18,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7188
7264
[WARNING|trainer.py:803] 2025-04-26 17:43:18,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7241
[WARNING|trainer.py:803] 2025-04-26 17:43:19,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:19,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7189
[WARNING|trainer.py:803] 2025-04-26 17:43:19,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7265
7242
[WARNING|trainer.py:803] 2025-04-26 17:43:20,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:20,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7190
[WARNING|trainer.py:803] 2025-04-26 17:43:21,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7266
7243
[WARNING|trainer.py:803] 2025-04-26 17:43:21,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:22,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7191
[WARNING|trainer.py:803] 2025-04-26 17:43:22,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7267
7244
[WARNING|trainer.py:803] 2025-04-26 17:43:23,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:23,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7192
[WARNING|trainer.py:803] 2025-04-26 17:43:23,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7268
7245
[WARNING|trainer.py:803] 2025-04-26 17:43:24,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:24,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7193
[WARNING|trainer.py:803] 2025-04-26 17:43:24,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7269
7246
[WARNING|trainer.py:803] 2025-04-26 17:43:25,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:25,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7194
[WARNING|trainer.py:803] 2025-04-26 17:43:26,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7270
7247
[WARNING|trainer.py:803] 2025-04-26 17:43:26,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:27,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7195
[WARNING|trainer.py:803] 2025-04-26 17:43:27,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7271
7248
[WARNING|trainer.py:803] 2025-04-26 17:43:27,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:28,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7196
[WARNING|trainer.py:803] 2025-04-26 17:43:28,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7272
7249
[WARNING|trainer.py:803] 2025-04-26 17:43:29,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:29,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7197
[WARNING|trainer.py:803] 2025-04-26 17:43:29,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7273
7250
[WARNING|trainer.py:803] 2025-04-26 17:43:30,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:30,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7198
[WARNING|trainer.py:803] 2025-04-26 17:43:31,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7274
7251
[WARNING|trainer.py:803] 2025-04-26 17:43:31,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:32,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7199
[WARNING|trainer.py:803] 2025-04-26 17:43:32,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7275
7252
[WARNING|trainer.py:803] 2025-04-26 17:43:32,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:33,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7200
[WARNING|trainer.py:803] 2025-04-26 17:43:33,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7276
7253
[WARNING|trainer.py:803] 2025-04-26 17:43:34,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:34,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7201
[WARNING|trainer.py:803] 2025-04-26 17:43:34,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7277
7254
[WARNING|trainer.py:803] 2025-04-26 17:43:35,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:35,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7202
[WARNING|trainer.py:803] 2025-04-26 17:43:35,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7278
7255
[WARNING|trainer.py:803] 2025-04-26 17:43:36,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:36,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7203
[WARNING|trainer.py:803] 2025-04-26 17:43:37,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7279
7256
[WARNING|trainer.py:803] 2025-04-26 17:43:37,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:38,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7204
[WARNING|trainer.py:803] 2025-04-26 17:43:38,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7280
7257
[WARNING|trainer.py:803] 2025-04-26 17:43:39,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:39,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7205
[WARNING|trainer.py:803] 2025-04-26 17:43:39,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7281
7258
[WARNING|trainer.py:803] 2025-04-26 17:43:40,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7206
[WARNING|trainer.py:803] 2025-04-26 17:43:40,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:40,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7282
7259
[WARNING|trainer.py:803] 2025-04-26 17:43:41,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7207
[WARNING|trainer.py:803] 2025-04-26 17:43:42,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:42,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7283
7260
[WARNING|trainer.py:803] 2025-04-26 17:43:42,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7208
[WARNING|trainer.py:803] 2025-04-26 17:43:43,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7284
7261
[WARNING|trainer.py:803] 2025-04-26 17:43:44,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7209
[WARNING|trainer.py:803] 2025-04-26 17:43:44,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:44,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7285
7262
[WARNING|trainer.py:803] 2025-04-26 17:43:45,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7210
[WARNING|trainer.py:803] 2025-04-26 17:43:45,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:45,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7286
7263
[WARNING|trainer.py:803] 2025-04-26 17:43:46,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7211
[WARNING|trainer.py:803] 2025-04-26 17:43:47,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:47,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7287
7264
[WARNING|trainer.py:803] 2025-04-26 17:43:47,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7212
[WARNING|trainer.py:803] 2025-04-26 17:43:48,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:48,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7288
7265
[WARNING|trainer.py:803] 2025-04-26 17:43:48,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7213
[WARNING|trainer.py:803] 2025-04-26 17:43:49,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:49,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7289
7266
[WARNING|trainer.py:803] 2025-04-26 17:43:50,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7214
[WARNING|trainer.py:803] 2025-04-26 17:43:50,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:50,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7290
7267
[WARNING|trainer.py:803] 2025-04-26 17:43:51,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7215
[WARNING|trainer.py:803] 2025-04-26 17:43:51,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:52,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7291
7268
[WARNING|trainer.py:803] 2025-04-26 17:43:52,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7216
[WARNING|trainer.py:803] 2025-04-26 17:43:53,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:53,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7292
7269
[WARNING|trainer.py:803] 2025-04-26 17:43:53,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7217
[WARNING|trainer.py:803] 2025-04-26 17:43:54,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:54,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7293
7270
[WARNING|trainer.py:803] 2025-04-26 17:43:55,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7218
[WARNING|trainer.py:803] 2025-04-26 17:43:55,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:43:55,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7294
7271
[WARNING|trainer.py:803] 2025-04-26 17:43:56,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7219
[WARNING|trainer.py:803] 2025-04-26 17:43:56,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:43:57,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7295
7272
[WARNING|trainer.py:803] 2025-04-26 17:43:57,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7220
[WARNING|trainer.py:803] 2025-04-26 17:43:58,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:43:58,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7296
7273
[WARNING|trainer.py:803] 2025-04-26 17:43:59,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7221
[WARNING|trainer.py:803] 2025-04-26 17:43:59,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:43:59,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7297
7274
[WARNING|trainer.py:803] 2025-04-26 17:44:00,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7222
[WARNING|trainer.py:803] 2025-04-26 17:44:00,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:00,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7298
7275
[WARNING|trainer.py:803] 2025-04-26 17:44:01,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7223
[WARNING|trainer.py:803] 2025-04-26 17:44:01,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:02,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7299
7276
[WARNING|trainer.py:803] 2025-04-26 17:44:02,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7224
[WARNING|trainer.py:803] 2025-04-26 17:44:03,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:03,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7300
7277
[WARNING|trainer.py:803] 2025-04-26 17:44:04,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7225
[WARNING|trainer.py:803] 2025-04-26 17:44:04,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:04,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7301
7278
[WARNING|trainer.py:803] 2025-04-26 17:44:05,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7226
[WARNING|trainer.py:803] 2025-04-26 17:44:05,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:05,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7302
7279
[WARNING|trainer.py:803] 2025-04-26 17:44:06,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7227
[WARNING|trainer.py:803] 2025-04-26 17:44:06,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:07,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7303
7280
[WARNING|trainer.py:803] 2025-04-26 17:44:07,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7228
[WARNING|trainer.py:803] 2025-04-26 17:44:08,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 17:44:08,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7304
7281
[WARNING|trainer.py:803] 2025-04-26 17:44:09,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7229
[WARNING|trainer.py:803] 2025-04-26 17:44:09,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:44:09,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7305
7282
[WARNING|trainer.py:803] 2025-04-26 17:44:10,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7230
[WARNING|trainer.py:803] 2025-04-26 17:44:10,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:10,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7283
7306
[WARNING|trainer.py:803] 2025-04-26 17:44:11,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7231
[WARNING|trainer.py:803] 2025-04-26 17:44:12,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:12,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7284
7307
[WARNING|trainer.py:803] 2025-04-26 17:44:12,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7232
[WARNING|trainer.py:803] 2025-04-26 17:44:13,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:44:13,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7285
7308
[WARNING|trainer.py:803] 2025-04-26 17:44:13,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7233
[WARNING|trainer.py:803] 2025-04-26 17:44:14,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:14,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7286
7309
[WARNING|trainer.py:803] 2025-04-26 17:44:15,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7234
[WARNING|trainer.py:803] 2025-04-26 17:44:15,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:15,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7287
7310
[WARNING|trainer.py:803] 2025-04-26 17:44:16,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7235
[WARNING|trainer.py:803] 2025-04-26 17:44:16,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:17,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7288
7311
[WARNING|trainer.py:803] 2025-04-26 17:44:17,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7236
[WARNING|trainer.py:803] 2025-04-26 17:44:18,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:18,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7289
7312
[WARNING|trainer.py:803] 2025-04-26 17:44:18,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7237
[WARNING|trainer.py:803] 2025-04-26 17:44:19,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:19,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7290
7313
[WARNING|trainer.py:803] 2025-04-26 17:44:20,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7238
[WARNING|trainer.py:803] 2025-04-26 17:44:20,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:20,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7291
7314
[WARNING|trainer.py:803] 2025-04-26 17:44:21,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7239
[WARNING|trainer.py:803] 2025-04-26 17:44:21,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:22,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7292
7315
[WARNING|trainer.py:803] 2025-04-26 17:44:22,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7240
[WARNING|trainer.py:803] 2025-04-26 17:44:23,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:44:23,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7293
7316
[WARNING|trainer.py:803] 2025-04-26 17:44:23,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7241
[WARNING|trainer.py:803] 2025-04-26 17:44:24,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:24,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7294
7317
[WARNING|trainer.py:803] 2025-04-26 17:44:25,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7242
[WARNING|trainer.py:803] 2025-04-26 17:44:25,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:44:25,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7295
7318
[WARNING|trainer.py:803] 2025-04-26 17:44:26,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7243
[WARNING|trainer.py:803] 2025-04-26 17:44:26,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:27,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7296
7319
[WARNING|trainer.py:803] 2025-04-26 17:44:27,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7244
[WARNING|trainer.py:803] 2025-04-26 17:44:28,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:28,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7297
7320
[WARNING|trainer.py:803] 2025-04-26 17:44:28,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7245
[WARNING|trainer.py:803] 2025-04-26 17:44:29,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:29,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7298
7321
[WARNING|trainer.py:803] 2025-04-26 17:44:30,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7246
[WARNING|trainer.py:803] 2025-04-26 17:44:30,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:30,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7299
7322
[WARNING|trainer.py:803] 2025-04-26 17:44:31,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7247
[WARNING|trainer.py:803] 2025-04-26 17:44:31,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:32,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7300
7323
[WARNING|trainer.py:803] 2025-04-26 17:44:32,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7248
[WARNING|trainer.py:803] 2025-04-26 17:44:33,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:33,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7301
7324
[WARNING|trainer.py:803] 2025-04-26 17:44:33,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7249
[WARNING|trainer.py:803] 2025-04-26 17:44:34,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:34,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7302
7325
[WARNING|trainer.py:803] 2025-04-26 17:44:35,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7250
[WARNING|trainer.py:803] 2025-04-26 17:44:35,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:35,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7303
7326
[WARNING|trainer.py:803] 2025-04-26 17:44:36,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7251
[WARNING|trainer.py:803] 2025-04-26 17:44:36,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 17:44:37,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7304
7327
[WARNING|trainer.py:803] 2025-04-26 17:44:37,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7252
[WARNING|trainer.py:803] 2025-04-26 17:44:38,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:44:38,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7305
7328
[WARNING|trainer.py:803] 2025-04-26 17:44:38,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7253
[WARNING|trainer.py:803] 2025-04-26 17:44:39,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:39,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7306
7329
[WARNING|trainer.py:803] 2025-04-26 17:44:40,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7254
[WARNING|trainer.py:803] 2025-04-26 17:44:40,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:40,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7307
7330
[WARNING|trainer.py:803] 2025-04-26 17:44:41,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7255
[WARNING|trainer.py:803] 2025-04-26 17:44:41,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:42,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7308
[WARNING|trainer.py:803] 2025-04-26 17:44:42,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7331
7256
[WARNING|trainer.py:803] 2025-04-26 17:44:42,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:43,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7309
[WARNING|trainer.py:803] 2025-04-26 17:44:43,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7332
7257
[WARNING|trainer.py:803] 2025-04-26 17:44:44,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:44,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
7310
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:44,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7333
7258
[WARNING|trainer.py:803] 2025-04-26 17:44:45,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7311
[WARNING|trainer.py:803] 2025-04-26 17:44:45,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:46,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7334
7259
[WARNING|trainer.py:803] 2025-04-26 17:44:46,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7312
[WARNING|trainer.py:803] 2025-04-26 17:44:47,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:47,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7335
7260
[WARNING|trainer.py:803] 2025-04-26 17:44:47,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7313
[WARNING|trainer.py:803] 2025-04-26 17:44:48,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:48,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7336
7261
[WARNING|trainer.py:803] 2025-04-26 17:44:49,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7314
[WARNING|trainer.py:803] 2025-04-26 17:44:49,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:49,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7337
7262
[WARNING|trainer.py:803] 2025-04-26 17:44:50,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7315
[WARNING|trainer.py:803] 2025-04-26 17:44:50,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:44:51,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7338
7263
[WARNING|trainer.py:803] 2025-04-26 17:44:51,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7316
[WARNING|trainer.py:803] 2025-04-26 17:44:52,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:52,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7339
7264
[WARNING|trainer.py:803] 2025-04-26 17:44:52,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7317
[WARNING|trainer.py:803] 2025-04-26 17:44:53,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:53,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7340
7265
[WARNING|trainer.py:803] 2025-04-26 17:44:54,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7318
[WARNING|trainer.py:803] 2025-04-26 17:44:54,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:54,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7341
7266
[WARNING|trainer.py:803] 2025-04-26 17:44:55,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7319
[WARNING|trainer.py:803] 2025-04-26 17:44:55,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:56,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7342
7267
[WARNING|trainer.py:803] 2025-04-26 17:44:56,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7320
[WARNING|trainer.py:803] 2025-04-26 17:44:56,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:44:57,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7343
7268
[WARNING|trainer.py:803] 2025-04-26 17:44:57,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7321
[WARNING|trainer.py:803] 2025-04-26 17:44:58,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:58,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7344
7269
[WARNING|trainer.py:803] 2025-04-26 17:44:58,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7322
[WARNING|trainer.py:803] 2025-04-26 17:44:59,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:44:59,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7345
7270
[WARNING|trainer.py:803] 2025-04-26 17:45:00,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7323
[WARNING|trainer.py:803] 2025-04-26 17:45:00,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:01,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7346
7271
[WARNING|trainer.py:803] 2025-04-26 17:45:01,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7324
[WARNING|trainer.py:803] 2025-04-26 17:45:01,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:45:02,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7347
7272
[WARNING|trainer.py:803] 2025-04-26 17:45:02,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7325
[WARNING|trainer.py:803] 2025-04-26 17:45:03,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:03,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7348
7273
[WARNING|trainer.py:803] 2025-04-26 17:45:03,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7326
[WARNING|trainer.py:803] 2025-04-26 17:45:04,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:04,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7349
7274
[WARNING|trainer.py:803] 2025-04-26 17:45:05,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7327
[WARNING|trainer.py:803] 2025-04-26 17:45:05,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:05,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7350
7275
[WARNING|trainer.py:803] 2025-04-26 17:45:06,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7328
[WARNING|trainer.py:803] 2025-04-26 17:45:06,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:07,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7351
[WARNING|trainer.py:803] 2025-04-26 17:45:07,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7276
7329
[WARNING|trainer.py:803] 2025-04-26 17:45:08,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:08,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7352
[WARNING|trainer.py:803] 2025-04-26 17:45:08,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7277
7330
[WARNING|trainer.py:803] 2025-04-26 17:45:09,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:09,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7353
7278
[WARNING|trainer.py:803] 2025-04-26 17:45:10,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7331
[WARNING|trainer.py:803] 2025-04-26 17:45:10,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:10,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7354
7279
[WARNING|trainer.py:803] 2025-04-26 17:45:11,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7332
[WARNING|trainer.py:803] 2025-04-26 17:45:11,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:12,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7355
[WARNING|trainer.py:803] 2025-04-26 17:45:12,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
7280
:Yes
7333
[WARNING|trainer.py:803] 2025-04-26 17:45:13,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:13,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7356
[WARNING|trainer.py:803] 2025-04-26 17:45:13,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7281
7334
[WARNING|trainer.py:803] 2025-04-26 17:45:14,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:14,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7357
[WARNING|trainer.py:803] 2025-04-26 17:45:14,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7282
7335
[WARNING|trainer.py:803] 2025-04-26 17:45:15,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:15,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7358
[WARNING|trainer.py:803] 2025-04-26 17:45:16,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7283
7336
[WARNING|trainer.py:803] 2025-04-26 17:45:16,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:17,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7359
[WARNING|trainer.py:803] 2025-04-26 17:45:17,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7284
7337
[WARNING|trainer.py:803] 2025-04-26 17:45:18,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:18,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7360
[WARNING|trainer.py:803] 2025-04-26 17:45:18,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7285
7338
[WARNING|trainer.py:803] 2025-04-26 17:45:19,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:19,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7361
[WARNING|trainer.py:803] 2025-04-26 17:45:19,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7286
7339
[WARNING|trainer.py:803] 2025-04-26 17:45:20,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:20,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7362
[WARNING|trainer.py:803] 2025-04-26 17:45:21,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7287
7340
[WARNING|trainer.py:803] 2025-04-26 17:45:21,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:22,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7363
[WARNING|trainer.py:803] 2025-04-26 17:45:22,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7288
7341
[WARNING|trainer.py:803] 2025-04-26 17:45:23,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7364
[WARNING|trainer.py:803] 2025-04-26 17:45:23,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:23,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7289
7342
[WARNING|trainer.py:803] 2025-04-26 17:45:24,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7365
[WARNING|trainer.py:803] 2025-04-26 17:45:24,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:24,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7290
7343
[WARNING|trainer.py:803] 2025-04-26 17:45:25,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7366
[WARNING|trainer.py:803] 2025-04-26 17:45:25,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:26,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7291
7344
[WARNING|trainer.py:803] 2025-04-26 17:45:26,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7367
[WARNING|trainer.py:803] 2025-04-26 17:45:27,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:27,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7292
7345
[WARNING|trainer.py:803] 2025-04-26 17:45:27,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7368
[WARNING|trainer.py:803] 2025-04-26 17:45:28,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:28,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7293
7346
[WARNING|trainer.py:803] 2025-04-26 17:45:29,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7369
[WARNING|trainer.py:803] 2025-04-26 17:45:29,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:29,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7294
7347
[WARNING|trainer.py:803] 2025-04-26 17:45:30,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7370
[WARNING|trainer.py:803] 2025-04-26 17:45:30,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:31,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7295
7348
[WARNING|trainer.py:803] 2025-04-26 17:45:31,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7371
[WARNING|trainer.py:803] 2025-04-26 17:45:32,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:32,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7296
7349
[WARNING|trainer.py:803] 2025-04-26 17:45:33,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:33,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7372
[WARNING|trainer.py:803] 2025-04-26 17:45:33,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7297
7350
[WARNING|trainer.py:803] 2025-04-26 17:45:34,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:34,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7373
[WARNING|trainer.py:803] 2025-04-26 17:45:34,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7298
7351
[WARNING|trainer.py:803] 2025-04-26 17:45:35,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:35,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7374
[WARNING|trainer.py:803] 2025-04-26 17:45:36,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7299
7352
[WARNING|trainer.py:803] 2025-04-26 17:45:36,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:37,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7375
[WARNING|trainer.py:803] 2025-04-26 17:45:37,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7300
7353
[WARNING|trainer.py:803] 2025-04-26 17:45:37,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:38,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7376
[WARNING|trainer.py:803] 2025-04-26 17:45:38,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7301
7354
[WARNING|trainer.py:803] 2025-04-26 17:45:39,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:39,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7377
[WARNING|trainer.py:803] 2025-04-26 17:45:39,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7302
7355
[WARNING|trainer.py:803] 2025-04-26 17:45:40,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:40,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7378
[WARNING|trainer.py:803] 2025-04-26 17:45:41,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7303
7356
[WARNING|trainer.py:803] 2025-04-26 17:45:41,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7379
[WARNING|trainer.py:803] 2025-04-26 17:45:42,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 17:45:42,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7304
7357
[WARNING|trainer.py:803] 2025-04-26 17:45:42,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7380
[WARNING|trainer.py:803] 2025-04-26 17:45:43,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:43,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7305
7358
[WARNING|trainer.py:803] 2025-04-26 17:45:44,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7381
[WARNING|trainer.py:803] 2025-04-26 17:45:44,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:44,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7306
7359
[WARNING|trainer.py:803] 2025-04-26 17:45:45,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7382
[WARNING|trainer.py:803] 2025-04-26 17:45:45,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:46,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7307
7360
[WARNING|trainer.py:803] 2025-04-26 17:45:46,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7383
[WARNING|trainer.py:803] 2025-04-26 17:45:47,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:47,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7308
7361
[WARNING|trainer.py:803] 2025-04-26 17:45:47,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7384
[WARNING|trainer.py:803] 2025-04-26 17:45:48,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:48,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7362
7309
[WARNING|trainer.py:803] 2025-04-26 17:45:49,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7385
[WARNING|trainer.py:803] 2025-04-26 17:45:49,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:49,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7363
7310
[WARNING|trainer.py:803] 2025-04-26 17:45:50,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7386
[WARNING|trainer.py:803] 2025-04-26 17:45:50,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:51,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7364
7311
[WARNING|trainer.py:803] 2025-04-26 17:45:51,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7387
[WARNING|trainer.py:803] 2025-04-26 17:45:52,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:52,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7365
7312
[WARNING|trainer.py:803] 2025-04-26 17:45:52,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7388
[WARNING|trainer.py:803] 2025-04-26 17:45:53,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:53,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7366
7313
[WARNING|trainer.py:803] 2025-04-26 17:45:54,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7389
[WARNING|trainer.py:803] 2025-04-26 17:45:54,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:54,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7367
7314
[WARNING|trainer.py:803] 2025-04-26 17:45:55,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7390
[WARNING|trainer.py:803] 2025-04-26 17:45:55,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:45:56,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7368
7315
[WARNING|trainer.py:803] 2025-04-26 17:45:56,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7391
[WARNING|trainer.py:803] 2025-04-26 17:45:57,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:45:57,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7369
7316
[WARNING|trainer.py:803] 2025-04-26 17:45:57,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7392
[WARNING|trainer.py:803] 2025-04-26 17:45:58,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:45:58,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7370
[WARNING|trainer.py:803] 2025-04-26 17:45:59,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7317
7393
[WARNING|trainer.py:803] 2025-04-26 17:45:59,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:45:59,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7371
[WARNING|trainer.py:803] 2025-04-26 17:46:00,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7318
7394
[WARNING|trainer.py:803] 2025-04-26 17:46:00,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:01,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7372
[WARNING|trainer.py:803] 2025-04-26 17:46:01,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7319
7395
[WARNING|trainer.py:803] 2025-04-26 17:46:02,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7373
[WARNING|trainer.py:803] 2025-04-26 17:46:02,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:02,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7320
7396
[WARNING|trainer.py:803] 2025-04-26 17:46:03,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7374
[WARNING|trainer.py:803] 2025-04-26 17:46:03,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:04,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7321
7397
[WARNING|trainer.py:803] 2025-04-26 17:46:04,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7375
[WARNING|trainer.py:803] 2025-04-26 17:46:05,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:05,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7322
7398
[WARNING|trainer.py:803] 2025-04-26 17:46:05,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7376
[WARNING|trainer.py:803] 2025-04-26 17:46:06,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:06,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7323
7399
[WARNING|trainer.py:803] 2025-04-26 17:46:07,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7377
[WARNING|trainer.py:803] 2025-04-26 17:46:07,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:07,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7324
7400
[WARNING|trainer.py:803] 2025-04-26 17:46:08,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7378
[WARNING|trainer.py:803] 2025-04-26 17:46:08,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:09,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7325
7401
[WARNING|trainer.py:803] 2025-04-26 17:46:09,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7379
[WARNING|trainer.py:803] 2025-04-26 17:46:09,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:10,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7326
7402
[WARNING|trainer.py:803] 2025-04-26 17:46:10,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7380
[WARNING|trainer.py:803] 2025-04-26 17:46:11,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:11,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7327
[WARNING|trainer.py:803] 2025-04-26 17:46:11,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7403
7381
[WARNING|trainer.py:803] 2025-04-26 17:46:12,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:12,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7328
[WARNING|trainer.py:803] 2025-04-26 17:46:13,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7404
7382
[WARNING|trainer.py:803] 2025-04-26 17:46:13,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:14,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7329
[WARNING|trainer.py:803] 2025-04-26 17:46:14,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7405
7383
[WARNING|trainer.py:803] 2025-04-26 17:46:14,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:15,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7330
[WARNING|trainer.py:803] 2025-04-26 17:46:15,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7406
7384
[WARNING|trainer.py:803] 2025-04-26 17:46:16,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:46:16,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7331
[WARNING|trainer.py:803] 2025-04-26 17:46:16,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7407
7385
[WARNING|trainer.py:803] 2025-04-26 17:46:17,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:17,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7332
[WARNING|trainer.py:803] 2025-04-26 17:46:18,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7408
7386
[WARNING|trainer.py:803] 2025-04-26 17:46:18,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:19,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7333
[WARNING|trainer.py:803] 2025-04-26 17:46:19,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7409
7387
[WARNING|trainer.py:803] 2025-04-26 17:46:19,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:20,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7334
[WARNING|trainer.py:803] 2025-04-26 17:46:20,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7410
7388
[WARNING|trainer.py:803] 2025-04-26 17:46:21,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:21,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7335
[WARNING|trainer.py:803] 2025-04-26 17:46:21,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7411
7389
[WARNING|trainer.py:803] 2025-04-26 17:46:22,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:22,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7336
[WARNING|trainer.py:803] 2025-04-26 17:46:23,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7412
7390
[WARNING|trainer.py:803] 2025-04-26 17:46:23,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:23,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7337
[WARNING|trainer.py:803] 2025-04-26 17:46:24,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7413
7391
[WARNING|trainer.py:803] 2025-04-26 17:46:24,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:25,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7338
[WARNING|trainer.py:803] 2025-04-26 17:46:25,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7414
7392
[WARNING|trainer.py:803] 2025-04-26 17:46:26,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:26,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7339
[WARNING|trainer.py:803] 2025-04-26 17:46:26,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7415
7393
[WARNING|trainer.py:803] 2025-04-26 17:46:27,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:27,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7340
[WARNING|trainer.py:803] 2025-04-26 17:46:27,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7416
7394
[WARNING|trainer.py:803] 2025-04-26 17:46:28,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:28,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7341
[WARNING|trainer.py:803] 2025-04-26 17:46:29,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7417
7395
[WARNING|trainer.py:803] 2025-04-26 17:46:29,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:30,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7342
[WARNING|trainer.py:803] 2025-04-26 17:46:30,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7418
7396
[WARNING|trainer.py:803] 2025-04-26 17:46:31,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:31,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
7343
[WARNING|trainer.py:803] 2025-04-26 17:46:31,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7419
7397
[WARNING|trainer.py:803] 2025-04-26 17:46:32,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:32,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7344
[WARNING|trainer.py:803] 2025-04-26 17:46:32,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7420
7398
[WARNING|trainer.py:803] 2025-04-26 17:46:33,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:33,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7345
[WARNING|trainer.py:803] 2025-04-26 17:46:34,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7421
7399
[WARNING|trainer.py:803] 2025-04-26 17:46:34,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:35,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7346
[WARNING|trainer.py:803] 2025-04-26 17:46:35,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7422
7400
[WARNING|trainer.py:803] 2025-04-26 17:46:36,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:46:36,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7347
[WARNING|trainer.py:803] 2025-04-26 17:46:36,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7423
7401
[WARNING|trainer.py:803] 2025-04-26 17:46:37,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:37,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7348
[WARNING|trainer.py:803] 2025-04-26 17:46:37,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7424
7402
[WARNING|trainer.py:803] 2025-04-26 17:46:38,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:38,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7349
[WARNING|trainer.py:803] 2025-04-26 17:46:39,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7425
7403
[WARNING|trainer.py:803] 2025-04-26 17:46:39,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:40,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7350
[WARNING|trainer.py:803] 2025-04-26 17:46:40,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7426
7404
[WARNING|trainer.py:803] 2025-04-26 17:46:41,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:41,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7351
[WARNING|trainer.py:803] 2025-04-26 17:46:41,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7427
7405
[WARNING|trainer.py:803] 2025-04-26 17:46:42,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:42,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7352
[WARNING|trainer.py:803] 2025-04-26 17:46:42,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7428
7406
[WARNING|trainer.py:803] 2025-04-26 17:46:43,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:43,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7353
[WARNING|trainer.py:803] 2025-04-26 17:46:44,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7429
7407
[WARNING|trainer.py:803] 2025-04-26 17:46:44,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:45,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7354
[WARNING|trainer.py:803] 2025-04-26 17:46:45,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7430
7408
[WARNING|trainer.py:803] 2025-04-26 17:46:46,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:46,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7355
[WARNING|trainer.py:803] 2025-04-26 17:46:46,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7431
7409
[WARNING|trainer.py:803] 2025-04-26 17:46:47,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:47,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7356
[WARNING|trainer.py:803] 2025-04-26 17:46:47,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7432
7410
[WARNING|trainer.py:803] 2025-04-26 17:46:48,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7357
[WARNING|trainer.py:803] 2025-04-26 17:46:48,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:49,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7433
7411
[WARNING|trainer.py:803] 2025-04-26 17:46:49,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7358
[WARNING|trainer.py:803] 2025-04-26 17:46:50,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:50,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7434
7412
[WARNING|trainer.py:803] 2025-04-26 17:46:51,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7359
[WARNING|trainer.py:803] 2025-04-26 17:46:51,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:51,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7435
7413
[WARNING|trainer.py:803] 2025-04-26 17:46:52,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7360
[WARNING|trainer.py:803] 2025-04-26 17:46:52,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:52,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7436
7414
[WARNING|trainer.py:803] 2025-04-26 17:46:53,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7361
[WARNING|trainer.py:803] 2025-04-26 17:46:53,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:53,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7437
7415
[WARNING|trainer.py:803] 2025-04-26 17:46:54,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7362
[WARNING|trainer.py:803] 2025-04-26 17:46:55,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:46:55,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7438
7416
[WARNING|trainer.py:803] 2025-04-26 17:46:55,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7363
[WARNING|trainer.py:803] 2025-04-26 17:46:56,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:46:56,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7439
7417
[WARNING|trainer.py:803] 2025-04-26 17:46:57,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7364
[WARNING|trainer.py:803] 2025-04-26 17:46:57,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:46:57,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7440
7418
[WARNING|trainer.py:803] 2025-04-26 17:46:58,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7365
[WARNING|trainer.py:803] 2025-04-26 17:46:58,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:46:58,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
7441
7419
[WARNING|trainer.py:803] 2025-04-26 17:46:59,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7366
[WARNING|trainer.py:803] 2025-04-26 17:47:00,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:00,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7442
7420
[WARNING|trainer.py:803] 2025-04-26 17:47:00,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7367
[WARNING|trainer.py:803] 2025-04-26 17:47:01,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:01,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7443
7421
[WARNING|trainer.py:803] 2025-04-26 17:47:02,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7368
[WARNING|trainer.py:803] 2025-04-26 17:47:02,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:02,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7444
7422
[WARNING|trainer.py:803] 2025-04-26 17:47:03,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7369
[WARNING|trainer.py:803] 2025-04-26 17:47:03,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:03,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7423
7445
[WARNING|trainer.py:803] 2025-04-26 17:47:04,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7370
[WARNING|trainer.py:803] 2025-04-26 17:47:05,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:05,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7424
7446
[WARNING|trainer.py:803] 2025-04-26 17:47:05,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7371
[WARNING|trainer.py:803] 2025-04-26 17:47:06,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:06,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7425
7447
[WARNING|trainer.py:803] 2025-04-26 17:47:07,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7372
[WARNING|trainer.py:803] 2025-04-26 17:47:07,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:07,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7426
7448
[WARNING|trainer.py:803] 2025-04-26 17:47:08,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7373
[WARNING|trainer.py:803] 2025-04-26 17:47:08,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:08,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7427
7449
[WARNING|trainer.py:803] 2025-04-26 17:47:09,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7374
[WARNING|trainer.py:803] 2025-04-26 17:47:10,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:10,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7428
7450
[WARNING|trainer.py:803] 2025-04-26 17:47:10,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7375
[WARNING|trainer.py:803] 2025-04-26 17:47:11,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:11,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7429
7451
[WARNING|trainer.py:803] 2025-04-26 17:47:11,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7376
[WARNING|trainer.py:803] 2025-04-26 17:47:12,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:12,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7430
7452
[WARNING|trainer.py:803] 2025-04-26 17:47:13,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7377
[WARNING|trainer.py:803] 2025-04-26 17:47:13,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:13,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7431
7453
[WARNING|trainer.py:803] 2025-04-26 17:47:14,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7378
[WARNING|trainer.py:803] 2025-04-26 17:47:15,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:15,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7432
7454
[WARNING|trainer.py:803] 2025-04-26 17:47:15,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7379
[WARNING|trainer.py:803] 2025-04-26 17:47:16,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:16,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7433
7455
[WARNING|trainer.py:803] 2025-04-26 17:47:16,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7380
[WARNING|trainer.py:803] 2025-04-26 17:47:17,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:17,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7434
7456
[WARNING|trainer.py:803] 2025-04-26 17:47:18,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7381
[WARNING|trainer.py:803] 2025-04-26 17:47:18,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:18,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7435
7457
[WARNING|trainer.py:803] 2025-04-26 17:47:19,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7382
[WARNING|trainer.py:803] 2025-04-26 17:47:19,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:20,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7436
7458
[WARNING|trainer.py:803] 2025-04-26 17:47:20,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7383
[WARNING|trainer.py:803] 2025-04-26 17:47:21,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:21,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7437
7459
[WARNING|trainer.py:803] 2025-04-26 17:47:21,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7384
[WARNING|trainer.py:803] 2025-04-26 17:47:22,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:22,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7438
[WARNING|trainer.py:803] 2025-04-26 17:47:23,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7460
7385
[WARNING|trainer.py:803] 2025-04-26 17:47:23,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:23,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7439
[WARNING|trainer.py:803] 2025-04-26 17:47:24,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7461
7386
[WARNING|trainer.py:803] 2025-04-26 17:47:24,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:25,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7440
[WARNING|trainer.py:803] 2025-04-26 17:47:25,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7462
7387
[WARNING|trainer.py:803] 2025-04-26 17:47:26,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:26,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7441
[WARNING|trainer.py:803] 2025-04-26 17:47:26,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7463
7388
[WARNING|trainer.py:803] 2025-04-26 17:47:27,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:27,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7442
[WARNING|trainer.py:803] 2025-04-26 17:47:27,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7464
7389
[WARNING|trainer.py:803] 2025-04-26 17:47:28,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:28,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7443
[WARNING|trainer.py:803] 2025-04-26 17:47:29,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7465
7390
[WARNING|trainer.py:803] 2025-04-26 17:47:29,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:30,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7444
[WARNING|trainer.py:803] 2025-04-26 17:47:30,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7466
7391
[WARNING|trainer.py:803] 2025-04-26 17:47:30,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:31,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7445
[WARNING|trainer.py:803] 2025-04-26 17:47:31,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7467
7392
[WARNING|trainer.py:803] 2025-04-26 17:47:32,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:32,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7446
[WARNING|trainer.py:803] 2025-04-26 17:47:32,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7468
7393
[WARNING|trainer.py:803] 2025-04-26 17:47:33,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:33,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7447
[WARNING|trainer.py:803] 2025-04-26 17:47:34,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7469
7394
[WARNING|trainer.py:803] 2025-04-26 17:47:34,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:35,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7448
[WARNING|trainer.py:803] 2025-04-26 17:47:35,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7470
7395
[WARNING|trainer.py:803] 2025-04-26 17:47:35,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7449
[WARNING|trainer.py:803] 2025-04-26 17:47:36,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7471
[WARNING|trainer.py:803] 2025-04-26 17:47:36,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7396
[WARNING|trainer.py:803] 2025-04-26 17:47:37,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7450
[WARNING|trainer.py:803] 2025-04-26 17:47:37,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:37,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7472
[WARNING|trainer.py:803] 2025-04-26 17:47:38,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
7397:Yes

7451
[WARNING|trainer.py:803] 2025-04-26 17:47:38,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:39,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7473
7398
[WARNING|trainer.py:803] 2025-04-26 17:47:39,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7452
[WARNING|trainer.py:803] 2025-04-26 17:47:39,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:40,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7474
[WARNING|trainer.py:803] 2025-04-26 17:47:40,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7399
7453
[WARNING|trainer.py:803] 2025-04-26 17:47:41,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:41,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7475
[WARNING|trainer.py:803] 2025-04-26 17:47:41,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7400
7454
[WARNING|trainer.py:803] 2025-04-26 17:47:42,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:42,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7476
[WARNING|trainer.py:803] 2025-04-26 17:47:43,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7401
7455
[WARNING|trainer.py:803] 2025-04-26 17:47:43,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:44,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7477
[WARNING|trainer.py:803] 2025-04-26 17:47:44,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7402
7456
[WARNING|trainer.py:803] 2025-04-26 17:47:44,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:45,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7478
[WARNING|trainer.py:803] 2025-04-26 17:47:45,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7403
7457
[WARNING|trainer.py:803] 2025-04-26 17:47:46,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:46,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7479
[WARNING|trainer.py:803] 2025-04-26 17:47:46,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7404
7458
[WARNING|trainer.py:803] 2025-04-26 17:47:47,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:47,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7480
[WARNING|trainer.py:803] 2025-04-26 17:47:48,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7405
7459
[WARNING|trainer.py:803] 2025-04-26 17:47:48,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:47:48,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7481
[WARNING|trainer.py:803] 2025-04-26 17:47:49,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7406
7460
[WARNING|trainer.py:803] 2025-04-26 17:47:49,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:50,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7482
[WARNING|trainer.py:803] 2025-04-26 17:47:50,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7407
7461
[WARNING|trainer.py:803] 2025-04-26 17:47:51,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:51,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7483
[WARNING|trainer.py:803] 2025-04-26 17:47:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7408
7462
[WARNING|trainer.py:803] 2025-04-26 17:47:52,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:47:52,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7484
[WARNING|trainer.py:803] 2025-04-26 17:47:53,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7409
7463
[WARNING|trainer.py:803] 2025-04-26 17:47:53,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:53,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7485
[WARNING|trainer.py:803] 2025-04-26 17:47:54,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7410
7464
[WARNING|trainer.py:803] 2025-04-26 17:47:54,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:47:55,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7486
[WARNING|trainer.py:803] 2025-04-26 17:47:55,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7411
7465
[WARNING|trainer.py:803] 2025-04-26 17:47:56,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:56,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7487
[WARNING|trainer.py:803] 2025-04-26 17:47:56,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7412
7466
[WARNING|trainer.py:803] 2025-04-26 17:47:57,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:57,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7488
[WARNING|trainer.py:803] 2025-04-26 17:47:57,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7413
7467
[WARNING|trainer.py:803] 2025-04-26 17:47:58,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:47:58,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7489
[WARNING|trainer.py:803] 2025-04-26 17:47:59,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7414
7468
[WARNING|trainer.py:803] 2025-04-26 17:47:59,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:00,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7490
[WARNING|trainer.py:803] 2025-04-26 17:48:00,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7415
7469
[WARNING|trainer.py:803] 2025-04-26 17:48:01,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:01,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7491
[WARNING|trainer.py:803] 2025-04-26 17:48:01,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7416
7470
[WARNING|trainer.py:803] 2025-04-26 17:48:02,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:02,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7492
[WARNING|trainer.py:803] 2025-04-26 17:48:02,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7417
7471
[WARNING|trainer.py:803] 2025-04-26 17:48:03,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:03,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7493
[WARNING|trainer.py:803] 2025-04-26 17:48:04,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7418
7472
[WARNING|trainer.py:803] 2025-04-26 17:48:04,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:05,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
7494
[WARNING|trainer.py:803] 2025-04-26 17:48:05,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7419
7473
[WARNING|trainer.py:803] 2025-04-26 17:48:06,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:06,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7495
[WARNING|trainer.py:803] 2025-04-26 17:48:06,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7420
7474
[WARNING|trainer.py:803] 2025-04-26 17:48:07,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:07,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7496
[WARNING|trainer.py:803] 2025-04-26 17:48:07,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7421
7475
[WARNING|trainer.py:803] 2025-04-26 17:48:08,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:08,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7497
[WARNING|trainer.py:803] 2025-04-26 17:48:09,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7422
7476
[WARNING|trainer.py:803] 2025-04-26 17:48:09,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:10,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7498
[WARNING|trainer.py:803] 2025-04-26 17:48:10,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7423
7477
[WARNING|trainer.py:803] 2025-04-26 17:48:10,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:11,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7499
[WARNING|trainer.py:803] 2025-04-26 17:48:11,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7424
7478
[WARNING|trainer.py:803] 2025-04-26 17:48:12,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:12,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7500
[WARNING|trainer.py:803] 2025-04-26 17:48:12,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7425
7479
[WARNING|trainer.py:803] 2025-04-26 17:48:13,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:13,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:13,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7501
7426
7480
[WARNING|trainer.py:803] 2025-04-26 17:48:14,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:15,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:15,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7502
7427
7481
[WARNING|trainer.py:803] 2025-04-26 17:48:16,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:16,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:16,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7428
7482
7503
[WARNING|trainer.py:803] 2025-04-26 17:48:17,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:17,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:17,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7429
7483
7504
[WARNING|trainer.py:803] 2025-04-26 17:48:18,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:18,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:19,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7430
7484
7505
[WARNING|trainer.py:803] 2025-04-26 17:48:20,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:20,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:20,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7431
7485
7506
[WARNING|trainer.py:803] 2025-04-26 17:48:21,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:21,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7486
7432
[WARNING|trainer.py:803] 2025-04-26 17:48:21,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7507
[WARNING|trainer.py:803] 2025-04-26 17:48:22,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:22,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7487
7433
[WARNING|trainer.py:803] 2025-04-26 17:48:23,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7508
[WARNING|trainer.py:803] 2025-04-26 17:48:23,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:24,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7488
7434
[WARNING|trainer.py:803] 2025-04-26 17:48:24,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:25,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:25,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7509
7489
7435
[WARNING|trainer.py:803] 2025-04-26 17:48:26,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:26,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:26,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7510
7490
7436
[WARNING|trainer.py:803] 2025-04-26 17:48:27,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:27,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:27,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7491
7511
7437
[WARNING|trainer.py:803] 2025-04-26 17:48:28,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:29,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:29,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7492
7438
7512
[WARNING|trainer.py:803] 2025-04-26 17:48:30,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:30,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:30,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7493
7439
7513
[WARNING|trainer.py:803] 2025-04-26 17:48:31,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:31,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7494
[WARNING|trainer.py:803] 2025-04-26 17:48:31,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7440
7514
[WARNING|trainer.py:803] 2025-04-26 17:48:32,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:32,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7495
[WARNING|trainer.py:803] 2025-04-26 17:48:33,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7441
7515
[WARNING|trainer.py:803] 2025-04-26 17:48:33,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:34,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7496
7442
[WARNING|trainer.py:803] 2025-04-26 17:48:34,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:35,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7516
[WARNING|trainer.py:803] 2025-04-26 17:48:35,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7497
7443
[WARNING|trainer.py:803] 2025-04-26 17:48:36,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:36,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7517
[WARNING|trainer.py:803] 2025-04-26 17:48:36,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7498
7444
[WARNING|trainer.py:803] 2025-04-26 17:48:37,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:37,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:37,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7518
7499
7445
[WARNING|trainer.py:803] 2025-04-26 17:48:38,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:38,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:39,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7500
7519
7446
[WARNING|trainer.py:803] 2025-04-26 17:48:40,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:40,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:40,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7501
7447
7520
[WARNING|trainer.py:803] 2025-04-26 17:48:41,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:41,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:41,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7448
7502
7521
[WARNING|trainer.py:803] 2025-04-26 17:48:42,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:42,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:43,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7449
7503
7522
[WARNING|trainer.py:803] 2025-04-26 17:48:44,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:44,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:44,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7450
7504
7523
[WARNING|trainer.py:803] 2025-04-26 17:48:45,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7451
[WARNING|trainer.py:803] 2025-04-26 17:48:45,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:46,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7505
[WARNING|trainer.py:803] 2025-04-26 17:48:46,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7524
7452
[WARNING|trainer.py:803] 2025-04-26 17:48:47,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:47,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7506
[WARNING|trainer.py:803] 2025-04-26 17:48:47,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7525
7453
[WARNING|trainer.py:803] 2025-04-26 17:48:48,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:48,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:49,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7507
7454
7526
[WARNING|trainer.py:803] 2025-04-26 17:48:50,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:50,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:50,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7508
7455
7527
[WARNING|trainer.py:803] 2025-04-26 17:48:51,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:48:51,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:51,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7456
7509
7528
[WARNING|trainer.py:803] 2025-04-26 17:48:52,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:52,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:53,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7457
7510
7529
[WARNING|trainer.py:803] 2025-04-26 17:48:54,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:54,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7458
[WARNING|trainer.py:803] 2025-04-26 17:48:54,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7511
7530
[WARNING|trainer.py:803] 2025-04-26 17:48:55,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7459
[WARNING|trainer.py:803] 2025-04-26 17:48:55,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:56,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7512
[WARNING|trainer.py:803] 2025-04-26 17:48:56,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7531
7460
[WARNING|trainer.py:803] 2025-04-26 17:48:57,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:48:57,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:57,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7513
7461
7532
[WARNING|trainer.py:803] 2025-04-26 17:48:58,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:48:59,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:48:59,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7514
7462
7533
[WARNING|trainer.py:803] 2025-04-26 17:49:00,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:00,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:00,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7515
7463
7534
[WARNING|trainer.py:803] 2025-04-26 17:49:01,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:01,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7464
[WARNING|trainer.py:803] 2025-04-26 17:49:01,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7516
7535
[WARNING|trainer.py:803] 2025-04-26 17:49:02,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:02,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7465
[WARNING|trainer.py:803] 2025-04-26 17:49:03,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7517
7536
[WARNING|trainer.py:803] 2025-04-26 17:49:04,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:04,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7466
[WARNING|trainer.py:803] 2025-04-26 17:49:04,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7518
[WARNING|trainer.py:803] 2025-04-26 17:49:05,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7537
7467
[WARNING|trainer.py:803] 2025-04-26 17:49:05,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:06,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7519
[WARNING|trainer.py:803] 2025-04-26 17:49:06,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7538
7468
[WARNING|trainer.py:803] 2025-04-26 17:49:07,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:07,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:07,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7520
7469
7539
[WARNING|trainer.py:803] 2025-04-26 17:49:08,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:09,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:09,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7521
7470
7540
[WARNING|trainer.py:803] 2025-04-26 17:49:10,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:10,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:10,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7471
7522
7541
[WARNING|trainer.py:803] 2025-04-26 17:49:11,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:11,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:11,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7472
7523
7542
[WARNING|trainer.py:803] 2025-04-26 17:49:12,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:12,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7473
[WARNING|trainer.py:803] 2025-04-26 17:49:13,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7524
7543
[WARNING|trainer.py:803] 2025-04-26 17:49:14,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7474
[WARNING|trainer.py:803] 2025-04-26 17:49:14,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:14,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7525
[WARNING|trainer.py:803] 2025-04-26 17:49:15,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7544
7475
[WARNING|trainer.py:803] 2025-04-26 17:49:15,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:16,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:16,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7526
7545
7476
[WARNING|trainer.py:803] 2025-04-26 17:49:17,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:17,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:17,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7527
7546
7477
[WARNING|trainer.py:803] 2025-04-26 17:49:18,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:18,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:18,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7478
7528
7547
[WARNING|trainer.py:803] 2025-04-26 17:49:20,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:20,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:20,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7479
7529
7548
[WARNING|trainer.py:803] 2025-04-26 17:49:21,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:21,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:21,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7480
7549
7530
[WARNING|trainer.py:803] 2025-04-26 17:49:22,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7481
[WARNING|trainer.py:803] 2025-04-26 17:49:23,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:23,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7550
7531
[WARNING|trainer.py:803] 2025-04-26 17:49:23,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7482
[WARNING|trainer.py:803] 2025-04-26 17:49:24,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:24,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:25,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7551
7532
7483
[WARNING|trainer.py:803] 2025-04-26 17:49:25,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:26,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:26,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7552
7533
7484
[WARNING|trainer.py:803] 2025-04-26 17:49:27,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:27,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:27,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7553
7485
7534
[WARNING|trainer.py:803] 2025-04-26 17:49:28,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:28,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:29,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7486
7554
7535
[WARNING|trainer.py:803] 2025-04-26 17:49:30,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:30,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:30,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7487
7555
7536
[WARNING|trainer.py:803] 2025-04-26 17:49:31,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:31,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:31,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7488
7556
7537
[WARNING|trainer.py:803] 2025-04-26 17:49:32,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7489
[WARNING|trainer.py:803] 2025-04-26 17:49:33,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:33,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7557
7538
[WARNING|trainer.py:803] 2025-04-26 17:49:33,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7490
[WARNING|trainer.py:803] 2025-04-26 17:49:34,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:34,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7558
7539
[WARNING|trainer.py:803] 2025-04-26 17:49:35,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7491
[WARNING|trainer.py:803] 2025-04-26 17:49:35,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:36,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7559
[WARNING|trainer.py:803] 2025-04-26 17:49:36,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7540
7492
[WARNING|trainer.py:803] 2025-04-26 17:49:37,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:37,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7560
[WARNING|trainer.py:803] 2025-04-26 17:49:37,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7541
7493
[WARNING|trainer.py:803] 2025-04-26 17:49:38,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:38,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:38,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7561
7494
7542
[WARNING|trainer.py:803] 2025-04-26 17:49:39,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:40,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7562
[WARNING|trainer.py:803] 2025-04-26 17:49:40,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7495
7543
[WARNING|trainer.py:803] 2025-04-26 17:49:41,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:41,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7563
[WARNING|trainer.py:803] 2025-04-26 17:49:41,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7496
7544
[WARNING|trainer.py:803] 2025-04-26 17:49:42,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:42,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7564
7497
[WARNING|trainer.py:803] 2025-04-26 17:49:43,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7545
[WARNING|trainer.py:803] 2025-04-26 17:49:43,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:43,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7498
7565
[WARNING|trainer.py:803] 2025-04-26 17:49:44,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:45,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7546
[WARNING|trainer.py:803] 2025-04-26 17:49:45,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7499
7566
[WARNING|trainer.py:803] 2025-04-26 17:49:46,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:46,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:46,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7547
7500
7567
[WARNING|trainer.py:803] 2025-04-26 17:49:47,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:47,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:48,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7548
7501
7568
[WARNING|trainer.py:803] 2025-04-26 17:49:48,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:49,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:49,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7549
7502
7569
[WARNING|trainer.py:803] 2025-04-26 17:49:50,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:50,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:50,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7550
7503
7570
[WARNING|trainer.py:803] 2025-04-26 17:49:51,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:52,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:52,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7551
7504
7571
[WARNING|trainer.py:803] 2025-04-26 17:49:53,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:49:53,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:49:53,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7552
7505
7572
[WARNING|trainer.py:803] 2025-04-26 17:49:54,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:54,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:55,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7553
7506
7573
[WARNING|trainer.py:803] 2025-04-26 17:49:56,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:56,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:56,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7554
7507
7574
[WARNING|trainer.py:803] 2025-04-26 17:49:57,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:57,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:57,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7555
7508
7575
[WARNING|trainer.py:803] 2025-04-26 17:49:58,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:49:58,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:49:59,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7556
7509
7576
[WARNING|trainer.py:803] 2025-04-26 17:50:00,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:00,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:00,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7557
7510
7577
[WARNING|trainer.py:803] 2025-04-26 17:50:01,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:01,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:02,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7558
7511
7578
[WARNING|trainer.py:803] 2025-04-26 17:50:03,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:03,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:03,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7559
7512
7579
[WARNING|trainer.py:803] 2025-04-26 17:50:04,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:04,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:04,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7560
7513
7580
[WARNING|trainer.py:803] 2025-04-26 17:50:05,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:06,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:06,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7561
7514
7581
[WARNING|trainer.py:803] 2025-04-26 17:50:07,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:07,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:07,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7562
7515
7582
[WARNING|trainer.py:803] 2025-04-26 17:50:08,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:08,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:09,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7563
7516
7583
[WARNING|trainer.py:803] 2025-04-26 17:50:10,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:10,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:10,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7564
7517
7584
[WARNING|trainer.py:803] 2025-04-26 17:50:11,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:11,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:12,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7565
7518
7585
[WARNING|trainer.py:803] 2025-04-26 17:50:12,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:13,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:13,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7566
7519
7586
[WARNING|trainer.py:803] 2025-04-26 17:50:14,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:14,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7567
[WARNING|trainer.py:803] 2025-04-26 17:50:14,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7520
7587
[WARNING|trainer.py:803] 2025-04-26 17:50:15,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:15,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7568
[WARNING|trainer.py:803] 2025-04-26 17:50:16,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7521
7588
[WARNING|trainer.py:803] 2025-04-26 17:50:17,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:17,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:17,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7569
7522
7589
[WARNING|trainer.py:803] 2025-04-26 17:50:18,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:18,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7570
[WARNING|trainer.py:803] 2025-04-26 17:50:19,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7523
7590
[WARNING|trainer.py:803] 2025-04-26 17:50:19,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:20,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7571
[WARNING|trainer.py:803] 2025-04-26 17:50:20,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7524
7591
[WARNING|trainer.py:803] 2025-04-26 17:50:21,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:21,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7572
[WARNING|trainer.py:803] 2025-04-26 17:50:21,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7525
7592
[WARNING|trainer.py:803] 2025-04-26 17:50:22,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:23,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7573
[WARNING|trainer.py:803] 2025-04-26 17:50:23,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7526
7593
[WARNING|trainer.py:803] 2025-04-26 17:50:24,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:50:24,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7574
[WARNING|trainer.py:803] 2025-04-26 17:50:24,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7527
7594
[WARNING|trainer.py:803] 2025-04-26 17:50:25,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:25,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:26,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7575
7528
7595
[WARNING|trainer.py:803] 2025-04-26 17:50:26,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:27,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7576
[WARNING|trainer.py:803] 2025-04-26 17:50:27,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7529
7596
[WARNING|trainer.py:803] 2025-04-26 17:50:28,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:28,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:28,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7577
7530
7597
[WARNING|trainer.py:803] 2025-04-26 17:50:29,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:30,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7578
[WARNING|trainer.py:803] 2025-04-26 17:50:30,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7531
7598
[WARNING|trainer.py:803] 2025-04-26 17:50:31,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:31,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:31,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7579
7532
7599
[WARNING|trainer.py:803] 2025-04-26 17:50:32,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:33,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7580
[WARNING|trainer.py:803] 2025-04-26 17:50:33,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7533
7600
[WARNING|trainer.py:803] 2025-04-26 17:50:34,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:50:34,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7581
[WARNING|trainer.py:803] 2025-04-26 17:50:34,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7534
7601
[WARNING|trainer.py:803] 2025-04-26 17:50:35,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7582
[WARNING|trainer.py:803] 2025-04-26 17:50:35,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:36,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7535
7602
[WARNING|trainer.py:803] 2025-04-26 17:50:36,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:37,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7583
[WARNING|trainer.py:803] 2025-04-26 17:50:37,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7536
7603
[WARNING|trainer.py:803] 2025-04-26 17:50:38,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:50:38,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7584
[WARNING|trainer.py:803] 2025-04-26 17:50:39,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7537
[WARNING|trainer.py:803] 2025-04-26 17:50:39,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7604
[WARNING|trainer.py:803] 2025-04-26 17:50:40,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7585
[WARNING|trainer.py:803] 2025-04-26 17:50:40,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7538
[WARNING|trainer.py:803] 2025-04-26 17:50:41,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7605
[WARNING|trainer.py:803] 2025-04-26 17:50:41,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7586
[WARNING|trainer.py:803] 2025-04-26 17:50:41,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7539
[WARNING|trainer.py:803] 2025-04-26 17:50:42,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7606
[WARNING|trainer.py:803] 2025-04-26 17:50:42,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7587
[WARNING|trainer.py:803] 2025-04-26 17:50:43,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7540
7607
[WARNING|trainer.py:803] 2025-04-26 17:50:44,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:44,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7588
[WARNING|trainer.py:803] 2025-04-26 17:50:44,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7541
[WARNING|trainer.py:803] 2025-04-26 17:50:45,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7608
[WARNING|trainer.py:803] 2025-04-26 17:50:45,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7589
[WARNING|trainer.py:803] 2025-04-26 17:50:46,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7542
[WARNING|trainer.py:803] 2025-04-26 17:50:46,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7609
[WARNING|trainer.py:803] 2025-04-26 17:50:47,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7590
[WARNING|trainer.py:803] 2025-04-26 17:50:47,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7543
[WARNING|trainer.py:803] 2025-04-26 17:50:48,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7610
[WARNING|trainer.py:803] 2025-04-26 17:50:48,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7591
[WARNING|trainer.py:803] 2025-04-26 17:50:49,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7544
[WARNING|trainer.py:803] 2025-04-26 17:50:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7611
[WARNING|trainer.py:803] 2025-04-26 17:50:50,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7592
[WARNING|trainer.py:803] 2025-04-26 17:50:50,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7545
[WARNING|trainer.py:803] 2025-04-26 17:50:51,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7612
[WARNING|trainer.py:803] 2025-04-26 17:50:51,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7593
[WARNING|trainer.py:803] 2025-04-26 17:50:51,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7546
[WARNING|trainer.py:803] 2025-04-26 17:50:52,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7613
[WARNING|trainer.py:803] 2025-04-26 17:50:52,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7594
[WARNING|trainer.py:803] 2025-04-26 17:50:53,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7547
[WARNING|trainer.py:803] 2025-04-26 17:50:53,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7614
7595
[WARNING|trainer.py:803] 2025-04-26 17:50:54,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:50:54,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7548
[WARNING|trainer.py:803] 2025-04-26 17:50:55,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7615
[WARNING|trainer.py:803] 2025-04-26 17:50:55,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7596
[WARNING|trainer.py:803] 2025-04-26 17:50:56,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7549
[WARNING|trainer.py:803] 2025-04-26 17:50:56,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7616
7597
[WARNING|trainer.py:803] 2025-04-26 17:50:57,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:50:57,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7550
[WARNING|trainer.py:803] 2025-04-26 17:50:58,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7617
7598
[WARNING|trainer.py:803] 2025-04-26 17:50:58,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:50:59,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7551
[WARNING|trainer.py:803] 2025-04-26 17:50:59,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7618
7599
[WARNING|trainer.py:803] 2025-04-26 17:51:00,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:00,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7552
[WARNING|trainer.py:803] 2025-04-26 17:51:00,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7619
7600
[WARNING|trainer.py:803] 2025-04-26 17:51:01,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:51:02,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7553
[WARNING|trainer.py:803] 2025-04-26 17:51:02,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7620
7601
[WARNING|trainer.py:803] 2025-04-26 17:51:02,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:03,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7554
[WARNING|trainer.py:803] 2025-04-26 17:51:03,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7621
7602
[WARNING|trainer.py:803] 2025-04-26 17:51:04,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:04,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7555
[WARNING|trainer.py:803] 2025-04-26 17:51:05,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7622
7603
[WARNING|trainer.py:803] 2025-04-26 17:51:05,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:06,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7556
[WARNING|trainer.py:803] 2025-04-26 17:51:06,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7623
[WARNING|trainer.py:803] 2025-04-26 17:51:07,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7604
[WARNING|trainer.py:803] 2025-04-26 17:51:07,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7557
[WARNING|trainer.py:803] 2025-04-26 17:51:08,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7624
7605
[WARNING|trainer.py:803] 2025-04-26 17:51:08,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:09,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7558
[WARNING|trainer.py:803] 2025-04-26 17:51:09,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7625
[WARNING|trainer.py:803] 2025-04-26 17:51:10,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7606
7559
[WARNING|trainer.py:803] 2025-04-26 17:51:10,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:11,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7626
[WARNING|trainer.py:803] 2025-04-26 17:51:11,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7607
7560
[WARNING|trainer.py:803] 2025-04-26 17:51:12,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:12,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7627
[WARNING|trainer.py:803] 2025-04-26 17:51:12,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7608
7561
[WARNING|trainer.py:803] 2025-04-26 17:51:13,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:13,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7628
[WARNING|trainer.py:803] 2025-04-26 17:51:14,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7609
7562
[WARNING|trainer.py:803] 2025-04-26 17:51:14,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:51:15,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7629
[WARNING|trainer.py:803] 2025-04-26 17:51:15,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7610
7563
[WARNING|trainer.py:803] 2025-04-26 17:51:16,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:16,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7630
[WARNING|trainer.py:803] 2025-04-26 17:51:16,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7611
7564
[WARNING|trainer.py:803] 2025-04-26 17:51:17,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:18,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7631
[WARNING|trainer.py:803] 2025-04-26 17:51:18,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7612
7565
[WARNING|trainer.py:803] 2025-04-26 17:51:19,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:19,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7632
[WARNING|trainer.py:803] 2025-04-26 17:51:19,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7613
7566
[WARNING|trainer.py:803] 2025-04-26 17:51:20,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:21,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7633
[WARNING|trainer.py:803] 2025-04-26 17:51:21,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7567
7614
[WARNING|trainer.py:803] 2025-04-26 17:51:21,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:22,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:22,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7634
7568
7615
[WARNING|trainer.py:803] 2025-04-26 17:51:23,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:23,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:24,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7635
7569
7616
[WARNING|trainer.py:803] 2025-04-26 17:51:24,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:51:25,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:25,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7636
7570
7617
[WARNING|trainer.py:803] 2025-04-26 17:51:26,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:51:26,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7637
[WARNING|trainer.py:803] 2025-04-26 17:51:27,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7571
7618
[WARNING|trainer.py:803] 2025-04-26 17:51:28,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:28,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7638
[WARNING|trainer.py:803] 2025-04-26 17:51:28,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7572
7619
[WARNING|trainer.py:803] 2025-04-26 17:51:29,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:29,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:30,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7639
7573
7620
[WARNING|trainer.py:803] 2025-04-26 17:51:30,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:30,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7574
7640
[WARNING|trainer.py:803] 2025-04-26 17:51:31,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7621
[WARNING|trainer.py:803] 2025-04-26 17:51:32,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:32,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7641
[WARNING|trainer.py:803] 2025-04-26 17:51:32,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7575
7622
[WARNING|trainer.py:803] 2025-04-26 17:51:33,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:33,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7576
7642
[WARNING|trainer.py:803] 2025-04-26 17:51:34,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7623
[WARNING|trainer.py:803] 2025-04-26 17:51:35,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:35,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:35,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7577
7643
7624
[WARNING|trainer.py:803] 2025-04-26 17:51:36,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:36,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7578
[WARNING|trainer.py:803] 2025-04-26 17:51:37,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7644
7625
[WARNING|trainer.py:803] 2025-04-26 17:51:38,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:38,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7579
[WARNING|trainer.py:803] 2025-04-26 17:51:38,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7645
7626
[WARNING|trainer.py:803] 2025-04-26 17:51:39,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:39,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7580
[WARNING|trainer.py:803] 2025-04-26 17:51:40,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7646
7627
[WARNING|trainer.py:803] 2025-04-26 17:51:40,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:40,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7581
[WARNING|trainer.py:803] 2025-04-26 17:51:41,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7647
7628
[WARNING|trainer.py:803] 2025-04-26 17:51:42,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:42,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7582
[WARNING|trainer.py:803] 2025-04-26 17:51:42,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7648
7629
[WARNING|trainer.py:803] 2025-04-26 17:51:43,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:43,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7583
[WARNING|trainer.py:803] 2025-04-26 17:51:44,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7649
7630
[WARNING|trainer.py:803] 2025-04-26 17:51:45,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:45,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7584
[WARNING|trainer.py:803] 2025-04-26 17:51:45,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7650
7631
[WARNING|trainer.py:803] 2025-04-26 17:51:46,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:46,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7585
[WARNING|trainer.py:803] 2025-04-26 17:51:47,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7651
7632
[WARNING|trainer.py:803] 2025-04-26 17:51:47,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:48,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7586
[WARNING|trainer.py:803] 2025-04-26 17:51:48,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7652
7633
[WARNING|trainer.py:803] 2025-04-26 17:51:49,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:49,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7587
[WARNING|trainer.py:803] 2025-04-26 17:51:50,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7653
7634
[WARNING|trainer.py:803] 2025-04-26 17:51:50,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:51,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7588
[WARNING|trainer.py:803] 2025-04-26 17:51:51,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7654
7635
[WARNING|trainer.py:803] 2025-04-26 17:51:52,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7589
[WARNING|trainer.py:803] 2025-04-26 17:51:52,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:53,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7655
[WARNING|trainer.py:803] 2025-04-26 17:51:53,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7636
7590
[WARNING|trainer.py:803] 2025-04-26 17:51:54,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:54,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7656
[WARNING|trainer.py:803] 2025-04-26 17:51:55,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7637
7591
[WARNING|trainer.py:803] 2025-04-26 17:51:55,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:55,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7657
[WARNING|trainer.py:803] 2025-04-26 17:51:56,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7638
7592
[WARNING|trainer.py:803] 2025-04-26 17:51:57,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:51:57,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:51:57,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7658
7639
7593
[WARNING|trainer.py:803] 2025-04-26 17:51:58,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:58,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:51:59,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7659
7640
7594
[WARNING|trainer.py:803] 2025-04-26 17:52:00,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:00,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:00,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7641
7660
7595
[WARNING|trainer.py:803] 2025-04-26 17:52:01,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:01,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:02,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7642
7661
7596
[WARNING|trainer.py:803] 2025-04-26 17:52:03,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:03,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:03,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7643
7662
7597
[WARNING|trainer.py:803] 2025-04-26 17:52:04,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:04,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:04,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7644
7663
7598
[WARNING|trainer.py:803] 2025-04-26 17:52:05,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:06,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:06,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7645
7599
7664
[WARNING|trainer.py:803] 2025-04-26 17:52:07,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:07,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:07,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7646
7600
7665
[WARNING|trainer.py:803] 2025-04-26 17:52:08,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:09,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:09,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7647
7601
7666
[WARNING|trainer.py:803] 2025-04-26 17:52:10,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:52:10,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:10,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7648
7602
7667
[WARNING|trainer.py:803] 2025-04-26 17:52:11,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:12,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:12,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7649
7603
7668
[WARNING|trainer.py:803] 2025-04-26 17:52:13,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:13,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:13,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7650
7604
7669
[WARNING|trainer.py:803] 2025-04-26 17:52:14,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:14,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7651
[WARNING|trainer.py:803] 2025-04-26 17:52:15,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7605
7670
[WARNING|trainer.py:803] 2025-04-26 17:52:16,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:16,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7652
[WARNING|trainer.py:803] 2025-04-26 17:52:16,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7606
7671
[WARNING|trainer.py:803] 2025-04-26 17:52:17,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:17,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:18,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7653
7607
7672
[WARNING|trainer.py:803] 2025-04-26 17:52:19,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:19,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:19,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7654
7608
7673
[WARNING|trainer.py:803] 2025-04-26 17:52:20,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:20,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:21,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7609
7655
7674
[WARNING|trainer.py:803] 2025-04-26 17:52:22,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:22,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:22,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7656
7610
7675
[WARNING|trainer.py:803] 2025-04-26 17:52:23,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:23,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:23,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7611
7657
7676
[WARNING|trainer.py:803] 2025-04-26 17:52:24,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:25,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:25,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7612
7658
7677
[WARNING|trainer.py:803] 2025-04-26 17:52:26,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:26,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:26,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7613
7659
7678
[WARNING|trainer.py:803] 2025-04-26 17:52:27,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:28,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:28,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7614
7660
7679
[WARNING|trainer.py:803] 2025-04-26 17:52:29,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:29,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7615
[WARNING|trainer.py:803] 2025-04-26 17:52:29,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7661
7680
[WARNING|trainer.py:803] 2025-04-26 17:52:30,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:31,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7616
[WARNING|trainer.py:803] 2025-04-26 17:52:31,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7662
7681
[WARNING|trainer.py:803] 2025-04-26 17:52:32,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:32,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7617
[WARNING|trainer.py:803] 2025-04-26 17:52:32,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7663
7682
[WARNING|trainer.py:803] 2025-04-26 17:52:33,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:34,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7618
[WARNING|trainer.py:803] 2025-04-26 17:52:34,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7664
7683
[WARNING|trainer.py:803] 2025-04-26 17:52:35,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:35,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7619
[WARNING|trainer.py:803] 2025-04-26 17:52:35,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7665
7684
[WARNING|trainer.py:803] 2025-04-26 17:52:36,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:37,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7620
[WARNING|trainer.py:803] 2025-04-26 17:52:37,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7666
7685
[WARNING|trainer.py:803] 2025-04-26 17:52:38,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:38,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7621
[WARNING|trainer.py:803] 2025-04-26 17:52:38,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7667
7686
[WARNING|trainer.py:803] 2025-04-26 17:52:39,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7622
[WARNING|trainer.py:803] 2025-04-26 17:52:40,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:40,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7687
7668
[WARNING|trainer.py:803] 2025-04-26 17:52:40,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7623
[WARNING|trainer.py:803] 2025-04-26 17:52:41,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:41,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:42,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7688
7669
7624
[WARNING|trainer.py:803] 2025-04-26 17:52:43,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:43,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:43,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7689
7670
7625
[WARNING|trainer.py:803] 2025-04-26 17:52:44,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:44,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:45,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7690
7671
7626
[WARNING|trainer.py:803] 2025-04-26 17:52:46,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:46,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:52:46,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7672
7691
7627
[WARNING|trainer.py:803] 2025-04-26 17:52:47,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:47,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:48,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7673
7692
7628
[WARNING|trainer.py:803] 2025-04-26 17:52:49,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:52:49,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:49,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7674
7693
7629
[WARNING|trainer.py:803] 2025-04-26 17:52:50,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:50,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7675
[WARNING|trainer.py:803] 2025-04-26 17:52:51,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7694
7630
[WARNING|trainer.py:803] 2025-04-26 17:52:51,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:52,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:52,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7676
7695
7631
[WARNING|trainer.py:803] 2025-04-26 17:52:53,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:53,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:53,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7677
7696
7632
[WARNING|trainer.py:803] 2025-04-26 17:52:54,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:55,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:55,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7678
7633
7697
[WARNING|trainer.py:803] 2025-04-26 17:52:56,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:56,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:52:56,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7679
7634
7698
[WARNING|trainer.py:803] 2025-04-26 17:52:57,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:58,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:58,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7680
7699
7635
[WARNING|trainer.py:803] 2025-04-26 17:52:59,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:52:59,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:52:59,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7681
7636
7700
[WARNING|trainer.py:803] 2025-04-26 17:53:00,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:01,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:01,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7682
7637
7701
[WARNING|trainer.py:803] 2025-04-26 17:53:02,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:02,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:02,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7683
7638
7702
[WARNING|trainer.py:803] 2025-04-26 17:53:03,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:04,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:04,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7684
7639
7703
[WARNING|trainer.py:803] 2025-04-26 17:53:05,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:05,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:05,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7685
7640
7704
[WARNING|trainer.py:803] 2025-04-26 17:53:06,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:06,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:07,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7686
7641
7705
[WARNING|trainer.py:803] 2025-04-26 17:53:08,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:08,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:08,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7687
7642
7706
[WARNING|trainer.py:803] 2025-04-26 17:53:09,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:09,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:10,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7688
7643
7707
[WARNING|trainer.py:803] 2025-04-26 17:53:11,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:11,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:11,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7689
7644
7708
[WARNING|trainer.py:803] 2025-04-26 17:53:12,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:12,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:12,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7690
7645
7709
[WARNING|trainer.py:803] 2025-04-26 17:53:13,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:14,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7691
[WARNING|trainer.py:803] 2025-04-26 17:53:14,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7646
7710
[WARNING|trainer.py:803] 2025-04-26 17:53:15,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:15,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:16,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7692
7647
7711
[WARNING|trainer.py:803] 2025-04-26 17:53:16,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:17,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:17,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7693
7648
7712
[WARNING|trainer.py:803] 2025-04-26 17:53:18,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:18,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:18,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7694
7649
7713
[WARNING|trainer.py:803] 2025-04-26 17:53:19,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:20,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:20,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7695
7650
7714
[WARNING|trainer.py:803] 2025-04-26 17:53:21,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:21,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:21,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7696
7715
7651
[WARNING|trainer.py:803] 2025-04-26 17:53:22,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:23,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 17:53:23,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7716
7697
7652
[WARNING|trainer.py:803] 2025-04-26 17:53:24,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:24,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:24,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7717
7698
7653
[WARNING|trainer.py:803] 2025-04-26 17:53:25,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:26,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:26,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7718
7699
7654
[WARNING|trainer.py:803] 2025-04-26 17:53:27,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:27,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:27,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7719
7700
7655
[WARNING|trainer.py:803] 2025-04-26 17:53:28,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:28,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:29,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7720
7701
7656
[WARNING|trainer.py:803] 2025-04-26 17:53:30,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:30,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7721
[WARNING|trainer.py:803] 2025-04-26 17:53:30,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7702
7657
[WARNING|trainer.py:803] 2025-04-26 17:53:31,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:31,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7722
[WARNING|trainer.py:803] 2025-04-26 17:53:32,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7703
7658
[WARNING|trainer.py:803] 2025-04-26 17:53:32,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:33,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7723
[WARNING|trainer.py:803] 2025-04-26 17:53:33,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7704
[WARNING|trainer.py:803] 2025-04-26 17:53:34,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7659
[WARNING|trainer.py:803] 2025-04-26 17:53:34,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7724
[WARNING|trainer.py:803] 2025-04-26 17:53:35,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7705
[WARNING|trainer.py:803] 2025-04-26 17:53:35,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7660
[WARNING|trainer.py:803] 2025-04-26 17:53:36,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7725
7706
[WARNING|trainer.py:803] 2025-04-26 17:53:36,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:37,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7661
[WARNING|trainer.py:803] 2025-04-26 17:53:37,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7726
7707
[WARNING|trainer.py:803] 2025-04-26 17:53:38,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:38,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7662
[WARNING|trainer.py:803] 2025-04-26 17:53:38,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7727
7708
[WARNING|trainer.py:803] 2025-04-26 17:53:39,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:39,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7663
[WARNING|trainer.py:803] 2025-04-26 17:53:40,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7728
7709
[WARNING|trainer.py:803] 2025-04-26 17:53:41,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:41,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7729
7664
[WARNING|trainer.py:803] 2025-04-26 17:53:42,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7710
[WARNING|trainer.py:803] 2025-04-26 17:53:42,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:42,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7730
7665
[WARNING|trainer.py:803] 2025-04-26 17:53:43,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7711
[WARNING|trainer.py:803] 2025-04-26 17:53:44,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:44,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7731
7666
[WARNING|trainer.py:803] 2025-04-26 17:53:44,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7712
[WARNING|trainer.py:803] 2025-04-26 17:53:45,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:45,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7732
[WARNING|trainer.py:803] 2025-04-26 17:53:46,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7667
7713
[WARNING|trainer.py:803] 2025-04-26 17:53:46,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:47,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7733
[WARNING|trainer.py:803] 2025-04-26 17:53:47,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7668
7714
[WARNING|trainer.py:803] 2025-04-26 17:53:48,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:53:48,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7734
[WARNING|trainer.py:803] 2025-04-26 17:53:49,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7669
7715
[WARNING|trainer.py:803] 2025-04-26 17:53:49,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7735
[WARNING|trainer.py:803] 2025-04-26 17:53:50,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:50,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7670
7716
[WARNING|trainer.py:803] 2025-04-26 17:53:51,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7736
[WARNING|trainer.py:803] 2025-04-26 17:53:51,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:51,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7671
7717
[WARNING|trainer.py:803] 2025-04-26 17:53:52,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7737
[WARNING|trainer.py:803] 2025-04-26 17:53:53,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:53,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7672
7718
[WARNING|trainer.py:803] 2025-04-26 17:53:53,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7738
[WARNING|trainer.py:803] 2025-04-26 17:53:54,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:54,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7719
7673
[WARNING|trainer.py:803] 2025-04-26 17:53:55,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7739
[WARNING|trainer.py:803] 2025-04-26 17:53:56,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:56,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7720
[WARNING|trainer.py:803] 2025-04-26 17:53:56,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7674
7740
[WARNING|trainer.py:803] 2025-04-26 17:53:57,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:53:57,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:58,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7721
7675
7741
[WARNING|trainer.py:803] 2025-04-26 17:53:59,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:53:59,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:53:59,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7722
7676
7742
[WARNING|trainer.py:803] 2025-04-26 17:54:00,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:00,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:01,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7723
7677
7743
[WARNING|trainer.py:803] 2025-04-26 17:54:01,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:02,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:02,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7724
7678
7744
[WARNING|trainer.py:803] 2025-04-26 17:54:03,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:03,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:03,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7725
7679
7745
[WARNING|trainer.py:803] 2025-04-26 17:54:04,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:04,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:05,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7726
7680
7746
[WARNING|trainer.py:803] 2025-04-26 17:54:06,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:06,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:06,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7727
7681
7747
[WARNING|trainer.py:803] 2025-04-26 17:54:07,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:08,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:08,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7728
7748
7682
[WARNING|trainer.py:803] 2025-04-26 17:54:08,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7729
[WARNING|trainer.py:803] 2025-04-26 17:54:09,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:09,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7749
7683
[WARNING|trainer.py:803] 2025-04-26 17:54:10,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7730
[WARNING|trainer.py:803] 2025-04-26 17:54:10,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:11,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7750
[WARNING|trainer.py:803] 2025-04-26 17:54:11,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7684
7731
[WARNING|trainer.py:803] 2025-04-26 17:54:12,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:12,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7751
[WARNING|trainer.py:803] 2025-04-26 17:54:13,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7685
7732
[WARNING|trainer.py:803] 2025-04-26 17:54:13,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:14,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7752
[WARNING|trainer.py:803] 2025-04-26 17:54:14,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7686
[WARNING|trainer.py:803] 2025-04-26 17:54:15,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7733
[WARNING|trainer.py:803] 2025-04-26 17:54:15,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7753
[WARNING|trainer.py:803] 2025-04-26 17:54:15,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7687
[WARNING|trainer.py:803] 2025-04-26 17:54:16,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7734
[WARNING|trainer.py:803] 2025-04-26 17:54:16,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7754
[WARNING|trainer.py:803] 2025-04-26 17:54:17,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7688
[WARNING|trainer.py:803] 2025-04-26 17:54:17,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7735
7755
[WARNING|trainer.py:803] 2025-04-26 17:54:18,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:18,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7689
[WARNING|trainer.py:803] 2025-04-26 17:54:19,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7736
7756
[WARNING|trainer.py:803] 2025-04-26 17:54:19,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:20,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7690
[WARNING|trainer.py:803] 2025-04-26 17:54:20,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7737
7757
[WARNING|trainer.py:803] 2025-04-26 17:54:21,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:21,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:22,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7738
7691
7758
[WARNING|trainer.py:803] 2025-04-26 17:54:22,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:22,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:23,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7739
7692
7759
[WARNING|trainer.py:803] 2025-04-26 17:54:24,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:24,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:24,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7740
7693
7760
[WARNING|trainer.py:803] 2025-04-26 17:54:25,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:26,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:26,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7741
7761
7694
[WARNING|trainer.py:803] 2025-04-26 17:54:27,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:27,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:27,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7742
7762
7695
[WARNING|trainer.py:803] 2025-04-26 17:54:28,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:28,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:29,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7743
7763
7696
[WARNING|trainer.py:803] 2025-04-26 17:54:30,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:30,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:30,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7744
7764
7697
[WARNING|trainer.py:803] 2025-04-26 17:54:31,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:31,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7745
[WARNING|trainer.py:803] 2025-04-26 17:54:32,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7765
7698
[WARNING|trainer.py:803] 2025-04-26 17:54:32,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:33,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7746
7766
[WARNING|trainer.py:803] 2025-04-26 17:54:33,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7699
[WARNING|trainer.py:803] 2025-04-26 17:54:34,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:34,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7747
7767
[WARNING|trainer.py:803] 2025-04-26 17:54:35,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7700
[WARNING|trainer.py:803] 2025-04-26 17:54:35,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:35,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7748
7768
[WARNING|trainer.py:803] 2025-04-26 17:54:36,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:37,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:37,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7701
7769
7749
[WARNING|trainer.py:803] 2025-04-26 17:54:38,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:38,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:38,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7702
7770
7750
[WARNING|trainer.py:803] 2025-04-26 17:54:39,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:40,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:40,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7703
7771
7751
[WARNING|trainer.py:803] 2025-04-26 17:54:41,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:41,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:41,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7704
7772
7752
[WARNING|trainer.py:803] 2025-04-26 17:54:42,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:42,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:42,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7705
7773
7753
[WARNING|trainer.py:803] 2025-04-26 17:54:43,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:44,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:44,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7706
7774
7754
[WARNING|trainer.py:803] 2025-04-26 17:54:45,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:45,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:45,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7707
7775
7755
[WARNING|trainer.py:803] 2025-04-26 17:54:46,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:47,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:47,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7708
7776
7756
[WARNING|trainer.py:803] 2025-04-26 17:54:48,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:48,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:48,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7709
7777
7757
[WARNING|trainer.py:803] 2025-04-26 17:54:49,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:49,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:50,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7778
7710
7758
[WARNING|trainer.py:803] 2025-04-26 17:54:51,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:51,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:51,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7779
7711
7759
[WARNING|trainer.py:803] 2025-04-26 17:54:52,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:52,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:52,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7780
7712
7760
[WARNING|trainer.py:803] 2025-04-26 17:54:54,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:54:54,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:54,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7781
7713
7761
[WARNING|trainer.py:803] 2025-04-26 17:54:55,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:55,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:54:55,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7782
7762
7714
[WARNING|trainer.py:803] 2025-04-26 17:54:57,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:57,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:54:57,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7783
7763
7715
[WARNING|trainer.py:803] 2025-04-26 17:54:58,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:58,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:58,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7784
7764
7716
[WARNING|trainer.py:803] 2025-04-26 17:54:59,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:54:59,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:00,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7785
7765
7717
[WARNING|trainer.py:803] 2025-04-26 17:55:01,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:01,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:01,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7766
7786
7718
[WARNING|trainer.py:803] 2025-04-26 17:55:02,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:02,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:02,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7767
7787
7719
[WARNING|trainer.py:803] 2025-04-26 17:55:04,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:04,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:04,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7768
7788
7720
[WARNING|trainer.py:803] 2025-04-26 17:55:05,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:05,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:05,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7769
7789
7721
[WARNING|trainer.py:803] 2025-04-26 17:55:06,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:06,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:07,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7770
7790
7722
[WARNING|trainer.py:803] 2025-04-26 17:55:08,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:08,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:08,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7771
7791
7723
[WARNING|trainer.py:803] 2025-04-26 17:55:09,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:09,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:10,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7772
7792
7724
[WARNING|trainer.py:803] 2025-04-26 17:55:11,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:11,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:11,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7773
7793
7725
[WARNING|trainer.py:803] 2025-04-26 17:55:12,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:12,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:12,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7794
7774
7726
[WARNING|trainer.py:803] 2025-04-26 17:55:14,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:14,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:14,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7775
7795
7727
[WARNING|trainer.py:803] 2025-04-26 17:55:15,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:15,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:15,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7776
7728
7796
[WARNING|trainer.py:803] 2025-04-26 17:55:17,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:17,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:17,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7729
7797
7777
[WARNING|trainer.py:803] 2025-04-26 17:55:18,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:18,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:18,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7730
7798
7778
[WARNING|trainer.py:803] 2025-04-26 17:55:19,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:19,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:20,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7731
7799
7779
[WARNING|trainer.py:803] 2025-04-26 17:55:21,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:21,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:21,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7732
7780
7800
[WARNING|trainer.py:803] 2025-04-26 17:55:22,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:22,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:22,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7733
7781
7801
[WARNING|trainer.py:803] 2025-04-26 17:55:24,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:24,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:24,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7734
7782
7802
[WARNING|trainer.py:803] 2025-04-26 17:55:25,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:25,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7735
7783
[WARNING|trainer.py:803] 2025-04-26 17:55:26,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:26,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:27,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7736
7803
7784
[WARNING|trainer.py:803] 2025-04-26 17:55:28,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:28,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:28,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7737
7785
7804
[WARNING|trainer.py:803] 2025-04-26 17:55:29,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:29,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:30,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7738
7786
7805
[WARNING|trainer.py:803] 2025-04-26 17:55:31,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:31,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7739
[WARNING|trainer.py:803] 2025-04-26 17:55:31,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7787
[WARNING|trainer.py:803] 2025-04-26 17:55:32,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7806
[WARNING|trainer.py:803] 2025-04-26 17:55:32,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7740
7788
[WARNING|trainer.py:803] 2025-04-26 17:55:33,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:34,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:34,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7807
7741
7789
[WARNING|trainer.py:803] 2025-04-26 17:55:35,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:35,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:35,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7742
7790
7808
[WARNING|trainer.py:803] 2025-04-26 17:55:36,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:37,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:37,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7743
7791
7809
[WARNING|trainer.py:803] 2025-04-26 17:55:38,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:38,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7744
[WARNING|trainer.py:803] 2025-04-26 17:55:39,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7792
[WARNING|trainer.py:803] 2025-04-26 17:55:39,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7810
[WARNING|trainer.py:803] 2025-04-26 17:55:39,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7745
7793
[WARNING|trainer.py:803] 2025-04-26 17:55:40,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:41,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:41,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7811
7746
7794
[WARNING|trainer.py:803] 2025-04-26 17:55:42,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:42,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:42,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7747
7812
7795
[WARNING|trainer.py:803] 2025-04-26 17:55:43,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:44,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:44,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7748
7796
7813
[WARNING|trainer.py:803] 2025-04-26 17:55:45,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:45,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7749
[WARNING|trainer.py:803] 2025-04-26 17:55:46,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7797
[WARNING|trainer.py:803] 2025-04-26 17:55:46,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7814
[WARNING|trainer.py:803] 2025-04-26 17:55:47,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7750
7798
[WARNING|trainer.py:803] 2025-04-26 17:55:47,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:48,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:48,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7815
7751
7799
[WARNING|trainer.py:803] 2025-04-26 17:55:49,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:49,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:50,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7752
7816
7800
[WARNING|trainer.py:803] 2025-04-26 17:55:51,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:51,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:51,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7753
7817
7801
[WARNING|trainer.py:803] 2025-04-26 17:55:52,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:55:53,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7754
[WARNING|trainer.py:803] 2025-04-26 17:55:53,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:53,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7818
7802
7755
[WARNING|trainer.py:803] 2025-04-26 17:55:54,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:54,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:55:55,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7819
7756
7803
[WARNING|trainer.py:803] 2025-04-26 17:55:56,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:56,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:56,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7757
7820
7804
[WARNING|trainer.py:803] 2025-04-26 17:55:58,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:55:58,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:55:58,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7758
7821
[WARNING|trainer.py:803] 2025-04-26 17:55:59,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7805
[WARNING|trainer.py:803] 2025-04-26 17:56:00,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7759
[WARNING|trainer.py:803] 2025-04-26 17:56:00,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:01,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7822
7806
7760
[WARNING|trainer.py:803] 2025-04-26 17:56:01,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:02,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:02,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7761
7807
7823
[WARNING|trainer.py:803] 2025-04-26 17:56:03,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:03,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:03,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7762
7808
7824
[WARNING|trainer.py:803] 2025-04-26 17:56:05,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:05,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7763
[WARNING|trainer.py:803] 2025-04-26 17:56:05,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:06,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7825
7809
7764
[WARNING|trainer.py:803] 2025-04-26 17:56:07,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:07,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:07,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7826
7810
7765
[WARNING|trainer.py:803] 2025-04-26 17:56:09,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:09,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:09,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7766
7827
7811
[WARNING|trainer.py:803] 2025-04-26 17:56:10,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:10,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:11,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7767
7812
[WARNING|trainer.py:803] 2025-04-26 17:56:12,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7828
7768
[WARNING|trainer.py:803] 2025-04-26 17:56:12,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:12,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:13,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7813
7769
7829
[WARNING|trainer.py:803] 2025-04-26 17:56:14,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:14,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:14,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7770
7814
7830
[WARNING|trainer.py:803] 2025-04-26 17:56:16,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:16,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:16,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7771
7815
[WARNING|trainer.py:803] 2025-04-26 17:56:17,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7831
[WARNING|trainer.py:803] 2025-04-26 17:56:18,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7772
[WARNING|trainer.py:803] 2025-04-26 17:56:18,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:18,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7816
7832
7773
[WARNING|trainer.py:803] 2025-04-26 17:56:19,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:20,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:20,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7817
7833
7774
[WARNING|trainer.py:803] 2025-04-26 17:56:21,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:21,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:21,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7775
7818
7834
[WARNING|trainer.py:803] 2025-04-26 17:56:23,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:23,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:23,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7776
7819
7835
[WARNING|trainer.py:803] 2025-04-26 17:56:24,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7777
[WARNING|trainer.py:803] 2025-04-26 17:56:25,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:25,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:25,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7836
7820
7778
[WARNING|trainer.py:803] 2025-04-26 17:56:27,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:27,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:27,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7837
7779
7821
[WARNING|trainer.py:803] 2025-04-26 17:56:28,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:28,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:28,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7780
7822
7838
[WARNING|trainer.py:803] 2025-04-26 17:56:30,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:30,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:30,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7781
7839
[WARNING|trainer.py:803] 2025-04-26 17:56:31,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7823
7782
[WARNING|trainer.py:803] 2025-04-26 17:56:32,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:32,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:33,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7840
7783
7824
[WARNING|trainer.py:803] 2025-04-26 17:56:33,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:34,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7841
[WARNING|trainer.py:803] 2025-04-26 17:56:34,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7784
7825
[WARNING|trainer.py:803] 2025-04-26 17:56:35,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:36,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:36,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7785
7842
7826
[WARNING|trainer.py:803] 2025-04-26 17:56:37,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:37,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:37,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7786
7827
7843
[WARNING|trainer.py:803] 2025-04-26 17:56:38,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7787
[WARNING|trainer.py:803] 2025-04-26 17:56:39,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:39,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:40,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7828
7788
7844
[WARNING|trainer.py:803] 2025-04-26 17:56:41,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:56:41,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:41,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7789
7845
7829
[WARNING|trainer.py:803] 2025-04-26 17:56:43,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:43,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:43,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7790
7830
[WARNING|trainer.py:803] 2025-04-26 17:56:44,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7846
[WARNING|trainer.py:803] 2025-04-26 17:56:45,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7791
[WARNING|trainer.py:803] 2025-04-26 17:56:45,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:46,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7831
7792
7847
[WARNING|trainer.py:803] 2025-04-26 17:56:47,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:47,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:47,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7793
7832
7848
[WARNING|trainer.py:803] 2025-04-26 17:56:49,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:49,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:49,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7794
7833
7849
[WARNING|trainer.py:803] 2025-04-26 17:56:50,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:50,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7795
[WARNING|trainer.py:803] 2025-04-26 17:56:51,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7834
[WARNING|trainer.py:803] 2025-04-26 17:56:51,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7850
[WARNING|trainer.py:803] 2025-04-26 17:56:52,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7796
[WARNING|trainer.py:803] 2025-04-26 17:56:53,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:53,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7835
7797
[WARNING|trainer.py:803] 2025-04-26 17:56:54,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7851
[WARNING|trainer.py:803] 2025-04-26 17:56:54,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7836
[WARNING|trainer.py:803] 2025-04-26 17:56:55,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7798
[WARNING|trainer.py:803] 2025-04-26 17:56:55,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:56,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7852
7837
7799
[WARNING|trainer.py:803] 2025-04-26 17:56:57,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:56:57,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:56:57,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7800
7853
7838
[WARNING|trainer.py:803] 2025-04-26 17:56:59,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:59,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:56:59,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7801
7854
7839
[WARNING|trainer.py:803] 2025-04-26 17:57:01,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:01,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:01,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7855
7840
7802
[WARNING|trainer.py:803] 2025-04-26 17:57:02,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:02,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:02,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7841
7856
7803
[WARNING|trainer.py:803] 2025-04-26 17:57:04,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:04,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:04,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7857
7804
7842
[WARNING|trainer.py:803] 2025-04-26 17:57:06,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:06,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:06,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7858
7805
7843
[WARNING|trainer.py:803] 2025-04-26 17:57:07,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:08,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:08,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7859
7806
[WARNING|trainer.py:803] 2025-04-26 17:57:09,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7844
[WARNING|trainer.py:803] 2025-04-26 17:57:10,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7860
[WARNING|trainer.py:803] 2025-04-26 17:57:10,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7807
[WARNING|trainer.py:803] 2025-04-26 17:57:11,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7845
7861
[WARNING|trainer.py:803] 2025-04-26 17:57:11,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:12,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:12,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7808
7862
[WARNING|trainer.py:803] 2025-04-26 17:57:13,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7846
[WARNING|trainer.py:803] 2025-04-26 17:57:14,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:14,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7809
7863
[WARNING|trainer.py:803] 2025-04-26 17:57:15,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7847
[WARNING|trainer.py:803] 2025-04-26 17:57:15,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7810
[WARNING|trainer.py:803] 2025-04-26 17:57:16,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7864
[WARNING|trainer.py:803] 2025-04-26 17:57:17,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7848
[WARNING|trainer.py:803] 2025-04-26 17:57:17,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7811
[WARNING|trainer.py:803] 2025-04-26 17:57:18,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7865
[WARNING|trainer.py:803] 2025-04-26 17:57:18,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7849
[WARNING|trainer.py:803] 2025-04-26 17:57:19,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7812
[WARNING|trainer.py:803] 2025-04-26 17:57:19,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7866
[WARNING|trainer.py:803] 2025-04-26 17:57:20,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:21,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7850
7813
7867
[WARNING|trainer.py:803] 2025-04-26 17:57:22,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:22,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:22,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7851
7814
[WARNING|trainer.py:803] 2025-04-26 17:57:24,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:24,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7868
[WARNING|trainer.py:803] 2025-04-26 17:57:25,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7815
7852
[WARNING|trainer.py:803] 2025-04-26 17:57:25,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7869
[WARNING|trainer.py:803] 2025-04-26 17:57:26,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:26,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7816
7853
[WARNING|trainer.py:803] 2025-04-26 17:57:27,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:28,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7870
7817
[WARNING|trainer.py:803] 2025-04-26 17:57:28,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7854
[WARNING|trainer.py:803] 2025-04-26 17:57:29,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:29,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7871
7818
[WARNING|trainer.py:803] 2025-04-26 17:57:30,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7855
[WARNING|trainer.py:803] 2025-04-26 17:57:31,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7872
[WARNING|trainer.py:803] 2025-04-26 17:57:31,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7819
[WARNING|trainer.py:803] 2025-04-26 17:57:32,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7856
[WARNING|trainer.py:803] 2025-04-26 17:57:33,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7873
[WARNING|trainer.py:803] 2025-04-26 17:57:33,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:33,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7820
7857
7874
[WARNING|trainer.py:803] 2025-04-26 17:57:34,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:57:34,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:35,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7858
7821
[WARNING|trainer.py:803] 2025-04-26 17:57:36,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:36,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7875
7859
7822
[WARNING|trainer.py:803] 2025-04-26 17:57:37,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:38,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:38,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7876
7860
[WARNING|trainer.py:803] 2025-04-26 17:57:39,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7823
[WARNING|trainer.py:803] 2025-04-26 17:57:39,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:40,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7861
7877
[WARNING|trainer.py:803] 2025-04-26 17:57:41,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:41,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7824
7862
7878
[WARNING|trainer.py:803] 2025-04-26 17:57:42,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:43,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:43,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7825
7863
[WARNING|trainer.py:803] 2025-04-26 17:57:44,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7879
[WARNING|trainer.py:803] 2025-04-26 17:57:44,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7826
[WARNING|trainer.py:803] 2025-04-26 17:57:45,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7864
[WARNING|trainer.py:803] 2025-04-26 17:57:45,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7880
[WARNING|trainer.py:803] 2025-04-26 17:57:46,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7827
[WARNING|trainer.py:803] 2025-04-26 17:57:47,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:47,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7865
7881
[WARNING|trainer.py:803] 2025-04-26 17:57:48,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:48,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7828
7866
7882
[WARNING|trainer.py:803] 2025-04-26 17:57:49,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:50,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:50,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7829
7867
[WARNING|trainer.py:803] 2025-04-26 17:57:51,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:51,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7883
7830
[WARNING|trainer.py:803] 2025-04-26 17:57:52,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7868
[WARNING|trainer.py:803] 2025-04-26 17:57:53,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:54,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7884
7831
7869
[WARNING|trainer.py:803] 2025-04-26 17:57:55,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:55,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:57:55,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7885
7832
[WARNING|trainer.py:803] 2025-04-26 17:57:56,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:57:57,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7870
7833
[WARNING|trainer.py:803] 2025-04-26 17:57:57,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7886
[WARNING|trainer.py:803] 2025-04-26 17:57:58,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:57:58,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7871
7834
[WARNING|trainer.py:803] 2025-04-26 17:57:59,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7887
[WARNING|trainer.py:803] 2025-04-26 17:58:00,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7872
[WARNING|trainer.py:803] 2025-04-26 17:58:00,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:01,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7835
7888
7873
[WARNING|trainer.py:803] 2025-04-26 17:58:02,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:02,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:03,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7889
7836
7874
[WARNING|trainer.py:803] 2025-04-26 17:58:03,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:04,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:04,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7890
7837
[WARNING|trainer.py:803] 2025-04-26 17:58:05,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:05,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7875
7891
7838
[WARNING|trainer.py:803] 2025-04-26 17:58:07,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:07,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:07,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7876
7839
7892
[WARNING|trainer.py:803] 2025-04-26 17:58:08,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:09,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:09,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7877
7840
7893
[WARNING|trainer.py:803] 2025-04-26 17:58:10,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:10,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:11,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7878
7841
[WARNING|trainer.py:803] 2025-04-26 17:58:12,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:12,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7894
7879
[WARNING|trainer.py:803] 2025-04-26 17:58:13,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7842
[WARNING|trainer.py:803] 2025-04-26 17:58:14,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7895
[WARNING|trainer.py:803] 2025-04-26 17:58:14,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:15,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7880
7843
[WARNING|trainer.py:803] 2025-04-26 17:58:16,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7896
[WARNING|trainer.py:803] 2025-04-26 17:58:16,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7881
[WARNING|trainer.py:803] 2025-04-26 17:58:17,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:18,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7844
7897
[WARNING|trainer.py:803] 2025-04-26 17:58:18,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7882
[WARNING|trainer.py:803] 2025-04-26 17:58:19,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7845
[WARNING|trainer.py:803] 2025-04-26 17:58:19,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7898
[WARNING|trainer.py:803] 2025-04-26 17:58:20,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:21,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7883
7846
[WARNING|trainer.py:803] 2025-04-26 17:58:22,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7899
[WARNING|trainer.py:803] 2025-04-26 17:58:22,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:23,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7884
7847
7900
[WARNING|trainer.py:803] 2025-04-26 17:58:24,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:24,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:24,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7885
7848
7901
[WARNING|trainer.py:803] 2025-04-26 17:58:26,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:26,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:26,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7849
7886
7902
[WARNING|trainer.py:803] 2025-04-26 17:58:28,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:28,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:28,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7887
7903
7850
[WARNING|trainer.py:803] 2025-04-26 17:58:30,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:58:30,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:30,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7888
7851
7904
[WARNING|trainer.py:803] 2025-04-26 17:58:31,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:32,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7889
[WARNING|trainer.py:803] 2025-04-26 17:58:32,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7905
[WARNING|trainer.py:803] 2025-04-26 17:58:33,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7852
7890
[WARNING|trainer.py:803] 2025-04-26 17:58:34,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:34,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:34,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7906
7853
[WARNING|trainer.py:803] 2025-04-26 17:58:35,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7891
[WARNING|trainer.py:803] 2025-04-26 17:58:36,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:36,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7907
7854
[WARNING|trainer.py:803] 2025-04-26 17:58:37,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7892
[WARNING|trainer.py:803] 2025-04-26 17:58:38,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7908
[WARNING|trainer.py:803] 2025-04-26 17:58:38,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7855
[WARNING|trainer.py:803] 2025-04-26 17:58:39,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7893
[WARNING|trainer.py:803] 2025-04-26 17:58:39,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7909
[WARNING|trainer.py:803] 2025-04-26 17:58:40,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7856
[WARNING|trainer.py:803] 2025-04-26 17:58:41,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:41,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7894
7910
7857
[WARNING|trainer.py:803] 2025-04-26 17:58:42,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:42,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:43,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7911
7858
7895
[WARNING|trainer.py:803] 2025-04-26 17:58:44,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:44,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:44,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7912
7859
7896
[WARNING|trainer.py:803] 2025-04-26 17:58:46,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:46,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:46,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7913
7860
7897
[WARNING|trainer.py:803] 2025-04-26 17:58:47,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:48,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:48,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7861
7914
[WARNING|trainer.py:803] 2025-04-26 17:58:49,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:49,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7898
7862
[WARNING|trainer.py:803] 2025-04-26 17:58:50,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7915
[WARNING|trainer.py:803] 2025-04-26 17:58:51,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:51,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7899
7863
[WARNING|trainer.py:803] 2025-04-26 17:58:52,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7916
[WARNING|trainer.py:803] 2025-04-26 17:58:52,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7900
[WARNING|trainer.py:803] 2025-04-26 17:58:53,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7864
[WARNING|trainer.py:803] 2025-04-26 17:58:54,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7917
[WARNING|trainer.py:803] 2025-04-26 17:58:54,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:55,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7901
7865
[WARNING|trainer.py:803] 2025-04-26 17:58:56,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7918
[WARNING|trainer.py:803] 2025-04-26 17:58:56,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7866
7902
[WARNING|trainer.py:803] 2025-04-26 17:58:57,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:58:58,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:58:58,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7919
7903
7867
[WARNING|trainer.py:803] 2025-04-26 17:58:59,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:59,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:58:59,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7920
[WARNING|trainer.py:803] 2025-04-26 17:59:01,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7904
7868
[WARNING|trainer.py:803] 2025-04-26 17:59:02,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:02,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7921
7905
7869
[WARNING|trainer.py:803] 2025-04-26 17:59:03,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:03,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:03,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7906
7922
7870
[WARNING|trainer.py:803] 2025-04-26 17:59:05,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:05,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:06,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7923
7907
7871
[WARNING|trainer.py:803] 2025-04-26 17:59:07,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:07,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:07,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7908
7872
7924
[WARNING|trainer.py:803] 2025-04-26 17:59:09,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:09,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:09,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7909
7873
7925
[WARNING|trainer.py:803] 2025-04-26 17:59:10,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:11,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:11,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7874
7910
7926
[WARNING|trainer.py:803] 2025-04-26 17:59:12,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:12,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:13,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7911
7875
[WARNING|trainer.py:803] 2025-04-26 17:59:14,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7927
[WARNING|trainer.py:803] 2025-04-26 17:59:15,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7912
[WARNING|trainer.py:803] 2025-04-26 17:59:15,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:16,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7876
7928
7913
[WARNING|trainer.py:803] 2025-04-26 17:59:16,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:17,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:17,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7877
7929
7914
[WARNING|trainer.py:803] 2025-04-26 17:59:18,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:18,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:19,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7878
7930
7915
[WARNING|trainer.py:803] 2025-04-26 17:59:20,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:20,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:21,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7931
7879
[WARNING|trainer.py:803] 2025-04-26 17:59:22,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7916
[WARNING|trainer.py:803] 2025-04-26 17:59:22,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:23,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7932
7880
7917
[WARNING|trainer.py:803] 2025-04-26 17:59:24,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:24,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:24,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7881
7933
[WARNING|trainer.py:803] 2025-04-26 17:59:26,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7918
[WARNING|trainer.py:803] 2025-04-26 17:59:26,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7882
[WARNING|trainer.py:803] 2025-04-26 17:59:27,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7934
[WARNING|trainer.py:803] 2025-04-26 17:59:27,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7919
[WARNING|trainer.py:803] 2025-04-26 17:59:28,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:28,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7883
7935
7920
[WARNING|trainer.py:803] 2025-04-26 17:59:30,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:30,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:30,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7936
7884
[WARNING|trainer.py:803] 2025-04-26 17:59:32,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7921
[WARNING|trainer.py:803] 2025-04-26 17:59:32,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:33,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7937
7885
[WARNING|trainer.py:803] 2025-04-26 17:59:34,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:34,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7922
7938
[WARNING|trainer.py:803] 2025-04-26 17:59:35,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7886
[WARNING|trainer.py:803] 2025-04-26 17:59:35,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:36,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7923
7939
[WARNING|trainer.py:803] 2025-04-26 17:59:37,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7887
[WARNING|trainer.py:803] 2025-04-26 17:59:37,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:38,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7940
7924
7888
[WARNING|trainer.py:803] 2025-04-26 17:59:39,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:39,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:40,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7925
7941
7889
[WARNING|trainer.py:803] 2025-04-26 17:59:41,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:41,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:41,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7926
7942
7890
[WARNING|trainer.py:803] 2025-04-26 17:59:43,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:43,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:43,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7891
7943
7927
[WARNING|trainer.py:803] 2025-04-26 17:59:45,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:45,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:45,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7944
7928
7892
[WARNING|trainer.py:803] 2025-04-26 17:59:46,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:47,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:47,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7929
7893
7945
[WARNING|trainer.py:803] 2025-04-26 17:59:48,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:48,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:49,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7930
7946
7894
[WARNING|trainer.py:803] 2025-04-26 17:59:50,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:50,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 17:59:51,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7931
7947
[WARNING|trainer.py:803] 2025-04-26 17:59:52,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7895
[WARNING|trainer.py:803] 2025-04-26 17:59:52,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 17:59:53,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7932
7896
7948
[WARNING|trainer.py:803] 2025-04-26 17:59:54,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:54,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:54,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7933
7949
7897
[WARNING|trainer.py:803] 2025-04-26 17:59:56,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:56,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 17:59:56,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7934
7950
7898
[WARNING|trainer.py:803] 2025-04-26 17:59:58,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 17:59:58,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 17:59:58,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7935
7951
7899
[WARNING|trainer.py:803] 2025-04-26 18:00:00,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:00:00,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:00,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7936
7900
[WARNING|trainer.py:803] 2025-04-26 18:00:01,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7952
[WARNING|trainer.py:803] 2025-04-26 18:00:02,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:02,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7937
7901
[WARNING|trainer.py:803] 2025-04-26 18:00:03,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7953
[WARNING|trainer.py:803] 2025-04-26 18:00:04,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7938
[WARNING|trainer.py:803] 2025-04-26 18:00:04,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7902
[WARNING|trainer.py:803] 2025-04-26 18:00:05,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7954
[WARNING|trainer.py:803] 2025-04-26 18:00:06,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:06,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7939
7903
[WARNING|trainer.py:803] 2025-04-26 18:00:07,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7955
[WARNING|trainer.py:803] 2025-04-26 18:00:08,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7940
[WARNING|trainer.py:803] 2025-04-26 18:00:08,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:09,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7956
7904
[WARNING|trainer.py:803] 2025-04-26 18:00:10,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7941
[WARNING|trainer.py:803] 2025-04-26 18:00:10,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7905
7957
[WARNING|trainer.py:803] 2025-04-26 18:00:11,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:11,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 18:00:11,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes

Yes
7942
7906
[WARNING|trainer.py:803] 2025-04-26 18:00:13,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7958
[WARNING|trainer.py:803] 2025-04-26 18:00:13,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7943
[WARNING|trainer.py:803] 2025-04-26 18:00:14,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7907
[WARNING|trainer.py:803] 2025-04-26 18:00:14,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7959
[WARNING|trainer.py:803] 2025-04-26 18:00:15,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7944
[WARNING|trainer.py:803] 2025-04-26 18:00:15,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7908
[WARNING|trainer.py:803] 2025-04-26 18:00:16,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7960
[WARNING|trainer.py:803] 2025-04-26 18:00:17,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:18,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7909
7945
7961
[WARNING|trainer.py:803] 2025-04-26 18:00:19,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:19,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:19,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7946
7910
7962
[WARNING|trainer.py:803] 2025-04-26 18:00:20,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:00:20,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:21,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7911
7947
7963
[WARNING|trainer.py:803] 2025-04-26 18:00:22,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:22,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:23,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7912
7948
[WARNING|trainer.py:803] 2025-04-26 18:00:24,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7964
[WARNING|trainer.py:803] 2025-04-26 18:00:24,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7913
[WARNING|trainer.py:803] 2025-04-26 18:00:25,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7949
[WARNING|trainer.py:803] 2025-04-26 18:00:26,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:26,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7965
7914
[WARNING|trainer.py:803] 2025-04-26 18:00:27,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7950
[WARNING|trainer.py:803] 2025-04-26 18:00:27,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7966
[WARNING|trainer.py:803] 2025-04-26 18:00:28,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7915
[WARNING|trainer.py:803] 2025-04-26 18:00:29,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:29,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7951
[WARNING|trainer.py:803] 2025-04-26 18:00:30,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7916
7967
[WARNING|trainer.py:803] 2025-04-26 18:00:31,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:31,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7952
7917
7968
[WARNING|trainer.py:803] 2025-04-26 18:00:32,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:33,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:33,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7953
7969
7918
[WARNING|trainer.py:803] 2025-04-26 18:00:34,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:35,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:35,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7954
7970
7919
[WARNING|trainer.py:803] 2025-04-26 18:00:36,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:37,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:37,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7955
7971
[WARNING|trainer.py:803] 2025-04-26 18:00:38,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7920
[WARNING|trainer.py:803] 2025-04-26 18:00:38,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7956
[WARNING|trainer.py:803] 2025-04-26 18:00:39,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7972
[WARNING|trainer.py:803] 2025-04-26 18:00:40,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:40,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7921
7957
7973
[WARNING|trainer.py:803] 2025-04-26 18:00:41,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:41,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:42,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7974
7922
7958
[WARNING|trainer.py:803] 2025-04-26 18:00:43,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:43,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:44,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7975
7923
7959
[WARNING|trainer.py:803] 2025-04-26 18:00:45,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:45,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:00:45,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7976
7960
7924
[WARNING|trainer.py:803] 2025-04-26 18:00:47,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:00:47,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:48,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7977
7961
7925
[WARNING|trainer.py:803] 2025-04-26 18:00:49,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:49,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:49,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7978
7926
7962
[WARNING|trainer.py:803] 2025-04-26 18:00:51,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:51,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:51,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7963
7979
7927
[WARNING|trainer.py:803] 2025-04-26 18:00:53,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:53,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:53,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7928
7980
7964
[WARNING|trainer.py:803] 2025-04-26 18:00:55,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:55,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:55,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7929
7981
7965
[WARNING|trainer.py:803] 2025-04-26 18:00:57,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:57,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:00:57,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7982
7930
7966
[WARNING|trainer.py:803] 2025-04-26 18:00:58,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:00:59,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:00:59,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7931
7983
[WARNING|trainer.py:803] 2025-04-26 18:01:00,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7967
[WARNING|trainer.py:803] 2025-04-26 18:01:00,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:01,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7984
7932
7968
[WARNING|trainer.py:803] 2025-04-26 18:01:02,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:02,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:03,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7933
7985
7969
[WARNING|trainer.py:803] 2025-04-26 18:01:04,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:04,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:05,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7934
7986
7970
[WARNING|trainer.py:803] 2025-04-26 18:01:06,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:06,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:06,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7987
7935
7971
[WARNING|trainer.py:803] 2025-04-26 18:01:08,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:08,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:08,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7972
7936
7988
[WARNING|trainer.py:803] 2025-04-26 18:01:10,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:10,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:10,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7973
7937
7989
[WARNING|trainer.py:803] 2025-04-26 18:01:12,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:12,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:12,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7974
7938
7990
[WARNING|trainer.py:803] 2025-04-26 18:01:13,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:14,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:14,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7975
7939
[WARNING|trainer.py:803] 2025-04-26 18:01:15,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7991
[WARNING|trainer.py:803] 2025-04-26 18:01:15,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:16,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7976
7940
[WARNING|trainer.py:803] 2025-04-26 18:01:17,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:17,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7992
7977
[WARNING|trainer.py:803] 2025-04-26 18:01:18,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7941
[WARNING|trainer.py:803] 2025-04-26 18:01:19,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:19,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7993
7978
7942
[WARNING|trainer.py:803] 2025-04-26 18:01:20,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:21,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:21,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7994
7979
7943
[WARNING|trainer.py:803] 2025-04-26 18:01:22,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:23,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:23,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7995
7944
[WARNING|trainer.py:803] 2025-04-26 18:01:24,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7980
[WARNING|trainer.py:803] 2025-04-26 18:01:25,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:25,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7996
7981
[WARNING|trainer.py:803] 2025-04-26 18:01:26,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7945
[WARNING|trainer.py:803] 2025-04-26 18:01:27,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:27,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7997
7982
7946
[WARNING|trainer.py:803] 2025-04-26 18:01:28,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:29,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:29,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7998
7983
7947
[WARNING|trainer.py:803] 2025-04-26 18:01:30,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:31,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:31,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7999
7984
7948
[WARNING|trainer.py:803] 2025-04-26 18:01:32,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:32,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:33,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7985
8000
7949
[WARNING|trainer.py:803] 2025-04-26 18:01:34,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:34,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:34,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7986
8001
7950
[WARNING|trainer.py:803] 2025-04-26 18:01:36,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:36,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:36,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7987
8002
7951
[WARNING|trainer.py:803] 2025-04-26 18:01:38,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:38,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:39,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7988
8003
7952
[WARNING|trainer.py:803] 2025-04-26 18:01:40,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 18:01:40,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:41,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8004
7989
[WARNING|trainer.py:803] 2025-04-26 18:01:42,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7953
[WARNING|trainer.py:803] 2025-04-26 18:01:42,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:43,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8005
7990
7954
[WARNING|trainer.py:803] 2025-04-26 18:01:44,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:44,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:45,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8006
7991
[WARNING|trainer.py:803] 2025-04-26 18:01:45,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7955
[WARNING|trainer.py:803] 2025-04-26 18:01:46,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:46,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8007
7956
[WARNING|trainer.py:803] 2025-04-26 18:01:47,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7992
[WARNING|trainer.py:803] 2025-04-26 18:01:48,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:48,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8008
7957
[WARNING|trainer.py:803] 2025-04-26 18:01:49,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7993
[WARNING|trainer.py:803] 2025-04-26 18:01:50,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:50,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8009
[WARNING|trainer.py:803] 2025-04-26 18:01:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7994
7958
8010
[WARNING|trainer.py:803] 2025-04-26 18:01:52,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:52,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:01:53,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7959
7995
8011
[WARNING|trainer.py:803] 2025-04-26 18:01:54,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:54,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:01:55,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7960
7996
8012
[WARNING|trainer.py:803] 2025-04-26 18:01:56,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:01:56,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:56,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7961
8013
[WARNING|trainer.py:803] 2025-04-26 18:01:58,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7997
[WARNING|trainer.py:803] 2025-04-26 18:01:58,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:01:58,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7962
8014
7998
[WARNING|trainer.py:803] 2025-04-26 18:01:59,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:00,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:00,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8015
7963
7999
[WARNING|trainer.py:803] 2025-04-26 18:02:01,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:01,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:02:02,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8016
7964
[WARNING|trainer.py:803] 2025-04-26 18:02:03,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8000
[WARNING|trainer.py:803] 2025-04-26 18:02:04,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8017
[WARNING|trainer.py:803] 2025-04-26 18:02:04,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7965
[WARNING|trainer.py:803] 2025-04-26 18:02:05,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:05,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8001
8018
7966
[WARNING|trainer.py:803] 2025-04-26 18:02:06,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:07,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:07,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8019
8002
[WARNING|trainer.py:803] 2025-04-26 18:02:08,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7967
[WARNING|trainer.py:803] 2025-04-26 18:02:09,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:02:09,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8020
8003
[WARNING|trainer.py:803] 2025-04-26 18:02:10,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7968
[WARNING|trainer.py:803] 2025-04-26 18:02:10,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8021
8004
[WARNING|trainer.py:803] 2025-04-26 18:02:11,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:12,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:12,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7969
8005
[WARNING|trainer.py:803] 2025-04-26 18:02:13,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8022
[WARNING|trainer.py:803] 2025-04-26 18:02:14,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:02:14,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7970
8006
[WARNING|trainer.py:803] 2025-04-26 18:02:15,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8023
[WARNING|trainer.py:803] 2025-04-26 18:02:15,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7971
[WARNING|trainer.py:803] 2025-04-26 18:02:16,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:16,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8024
8007
7972
[WARNING|trainer.py:803] 2025-04-26 18:02:17,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:17,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:18,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8025
8008
7973
[WARNING|trainer.py:803] 2025-04-26 18:02:19,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:19,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:20,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8026
8009
7974
[WARNING|trainer.py:803] 2025-04-26 18:02:21,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:21,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:21,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8027
8010
7975
[WARNING|trainer.py:803] 2025-04-26 18:02:23,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:23,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:23,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8028
8011
[WARNING|trainer.py:803] 2025-04-26 18:02:24,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7976
[WARNING|trainer.py:803] 2025-04-26 18:02:25,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8029
[WARNING|trainer.py:803] 2025-04-26 18:02:25,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8012
[WARNING|trainer.py:803] 2025-04-26 18:02:26,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:26,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7977
8030
8013
[WARNING|trainer.py:803] 2025-04-26 18:02:27,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:02:28,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:28,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7978
8031
8014
[WARNING|trainer.py:803] 2025-04-26 18:02:29,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:29,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:30,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7979
8032
8015
[WARNING|trainer.py:803] 2025-04-26 18:02:31,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:31,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:31,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8016
8033
7980
[WARNING|trainer.py:803] 2025-04-26 18:02:33,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:02:33,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:33,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8017
8034
7981
[WARNING|trainer.py:803] 2025-04-26 18:02:35,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:35,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:35,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8035
8018
7982
[WARNING|trainer.py:803] 2025-04-26 18:02:36,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:37,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:37,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8036
8019
7983
[WARNING|trainer.py:803] 2025-04-26 18:02:38,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:38,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:39,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
8020
7984
[WARNING|trainer.py:803] 2025-04-26 18:02:40,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:40,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:40,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8038
8021
[WARNING|trainer.py:803] 2025-04-26 18:02:41,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7985
[WARNING|trainer.py:803] 2025-04-26 18:02:42,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8039
[WARNING|trainer.py:803] 2025-04-26 18:02:42,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:43,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8022
7986
[WARNING|trainer.py:803] 2025-04-26 18:02:44,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8040
[WARNING|trainer.py:803] 2025-04-26 18:02:44,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:45,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8023
8041
7987
[WARNING|trainer.py:803] 2025-04-26 18:02:46,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:46,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:47,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8024
8042
[WARNING|trainer.py:803] 2025-04-26 18:02:48,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7988
[WARNING|trainer.py:803] 2025-04-26 18:02:48,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8025
[WARNING|trainer.py:803] 2025-04-26 18:02:49,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8043
[WARNING|trainer.py:803] 2025-04-26 18:02:49,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7989
[WARNING|trainer.py:803] 2025-04-26 18:02:50,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8026
8044
[WARNING|trainer.py:803] 2025-04-26 18:02:51,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:51,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7990
8027
8045
[WARNING|trainer.py:803] 2025-04-26 18:02:52,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:02:53,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:02:53,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7991
8028
[WARNING|trainer.py:803] 2025-04-26 18:02:54,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:02:55,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8046
8029
[WARNING|trainer.py:803] 2025-04-26 18:02:56,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7992
[WARNING|trainer.py:803] 2025-04-26 18:02:56,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8047
[WARNING|trainer.py:803] 2025-04-26 18:02:57,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8030
[WARNING|trainer.py:803] 2025-04-26 18:02:57,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7993
[WARNING|trainer.py:803] 2025-04-26 18:02:58,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8048
[WARNING|trainer.py:803] 2025-04-26 18:02:59,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8031
[WARNING|trainer.py:803] 2025-04-26 18:02:59,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:00,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7994
8049
8032
[WARNING|trainer.py:803] 2025-04-26 18:03:01,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:01,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:01,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8050
7995
8033
[WARNING|trainer.py:803] 2025-04-26 18:03:02,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:02,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:03,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8051
7996
8034
[WARNING|trainer.py:803] 2025-04-26 18:03:04,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:05,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:05,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8052
8035
[WARNING|trainer.py:803] 2025-04-26 18:03:06,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7997
[WARNING|trainer.py:803] 2025-04-26 18:03:06,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8053
[WARNING|trainer.py:803] 2025-04-26 18:03:07,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8036
[WARNING|trainer.py:803] 2025-04-26 18:03:07,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7998
8054
[WARNING|trainer.py:803] 2025-04-26 18:03:08,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:09,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:09,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
8055
7999
[WARNING|trainer.py:803] 2025-04-26 18:03:10,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:11,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:11,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8038
8056
[WARNING|trainer.py:803] 2025-04-26 18:03:12,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8000
[WARNING|trainer.py:803] 2025-04-26 18:03:12,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8039
[WARNING|trainer.py:803] 2025-04-26 18:03:13,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8057
[WARNING|trainer.py:803] 2025-04-26 18:03:13,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:14,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8001
8040
8058
[WARNING|trainer.py:803] 2025-04-26 18:03:15,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:15,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:15,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8041
8002
8059
[WARNING|trainer.py:803] 2025-04-26 18:03:17,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:17,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:17,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8042
8060
8003
[WARNING|trainer.py:803] 2025-04-26 18:03:19,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:19,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:19,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8043
8061
8004
[WARNING|trainer.py:803] 2025-04-26 18:03:20,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:20,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:20,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8044
8062
8005
[WARNING|trainer.py:803] 2025-04-26 18:03:22,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:22,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:22,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8045
8063
8006
[WARNING|trainer.py:803] 2025-04-26 18:03:24,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:24,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:24,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8064
8007
8046
[WARNING|trainer.py:803] 2025-04-26 18:03:26,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:26,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:26,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8065
8047
8008
[WARNING|trainer.py:803] 2025-04-26 18:03:27,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:28,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:28,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8066
8048
8009
[WARNING|trainer.py:803] 2025-04-26 18:03:29,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:30,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:30,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8067
8049
8010
[WARNING|trainer.py:803] 2025-04-26 18:03:31,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:31,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:31,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8068
8050
8011
[WARNING|trainer.py:803] 2025-04-26 18:03:32,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:33,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:33,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8069
8051
8012
[WARNING|trainer.py:803] 2025-04-26 18:03:34,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:35,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:35,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8070
8013
8052
[WARNING|trainer.py:803] 2025-04-26 18:03:36,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:36,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:36,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8071
8053
8014
[WARNING|trainer.py:803] 2025-04-26 18:03:38,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:38,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:38,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8072
8054
8015
[WARNING|trainer.py:803] 2025-04-26 18:03:39,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:39,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:40,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8055
8016
8073
[WARNING|trainer.py:803] 2025-04-26 18:03:41,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:41,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:41,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8056
8017
[WARNING|trainer.py:803] 2025-04-26 18:03:43,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8074
[WARNING|trainer.py:803] 2025-04-26 18:03:43,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8057
[WARNING|trainer.py:803] 2025-04-26 18:03:43,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8018
[WARNING|trainer.py:803] 2025-04-26 18:03:44,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8075
[WARNING|trainer.py:803] 2025-04-26 18:03:45,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8058
[WARNING|trainer.py:803] 2025-04-26 18:03:46,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8019
[WARNING|trainer.py:803] 2025-04-26 18:03:46,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8076
[WARNING|trainer.py:803] 2025-04-26 18:03:47,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8059
[WARNING|trainer.py:803] 2025-04-26 18:03:47,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:48,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8020
8077
8060
[WARNING|trainer.py:803] 2025-04-26 18:03:49,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:03:49,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:49,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8021
8078
8061
[WARNING|trainer.py:803] 2025-04-26 18:03:50,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:51,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:51,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8079
8022
8062
[WARNING|trainer.py:803] 2025-04-26 18:03:52,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:52,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:53,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8080
8023
8063
[WARNING|trainer.py:803] 2025-04-26 18:03:54,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:54,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:54,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8081
8024
8064
[WARNING|trainer.py:803] 2025-04-26 18:03:56,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:56,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:03:56,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8082
8065
8025
[WARNING|trainer.py:803] 2025-04-26 18:03:58,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:03:58,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:03:58,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8083
8066
8026
[WARNING|trainer.py:803] 2025-04-26 18:03:59,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:00,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:00,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8084
8067
8027
[WARNING|trainer.py:803] 2025-04-26 18:04:01,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:01,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:01,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8085
8068
8028
[WARNING|trainer.py:803] 2025-04-26 18:04:03,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:03,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:03,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8029
8086
8069
[WARNING|trainer.py:803] 2025-04-26 18:04:05,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:05,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:05,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8030
8070
8087
[WARNING|trainer.py:803] 2025-04-26 18:04:07,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:07,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:07,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8031
8071
8088
[WARNING|trainer.py:803] 2025-04-26 18:04:08,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:04:08,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:08,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8032
8072
8089
[WARNING|trainer.py:803] 2025-04-26 18:04:10,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:10,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:10,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8033
8073
[WARNING|trainer.py:803] 2025-04-26 18:04:12,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8090
[WARNING|trainer.py:803] 2025-04-26 18:04:12,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8034
[WARNING|trainer.py:803] 2025-04-26 18:04:13,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8074
8091
[WARNING|trainer.py:803] 2025-04-26 18:04:13,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:04:14,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8035
[WARNING|trainer.py:803] 2025-04-26 18:04:14,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:15,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8092
8075
8036
[WARNING|trainer.py:803] 2025-04-26 18:04:16,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:16,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:17,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8093
8076
[WARNING|trainer.py:803] 2025-04-26 18:04:18,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:18,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
8094
8077
[WARNING|trainer.py:803] 2025-04-26 18:04:19,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:20,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:20,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8038
8095
8078
[WARNING|trainer.py:803] 2025-04-26 18:04:20,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:21,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:21,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8039
8096
8079
[WARNING|trainer.py:803] 2025-04-26 18:04:22,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:23,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:23,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8040
8097
8080
[WARNING|trainer.py:803] 2025-04-26 18:04:24,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:24,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:24,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8041
8098
[WARNING|trainer.py:803] 2025-04-26 18:04:25,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8081
[WARNING|trainer.py:803] 2025-04-26 18:04:26,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8042
[WARNING|trainer.py:803] 2025-04-26 18:04:26,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8099
[WARNING|trainer.py:803] 2025-04-26 18:04:27,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8082
[WARNING|trainer.py:803] 2025-04-26 18:04:28,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8043
[WARNING|trainer.py:803] 2025-04-26 18:04:28,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8100
[WARNING|trainer.py:803] 2025-04-26 18:04:29,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8083
[WARNING|trainer.py:803] 2025-04-26 18:04:29,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8044
[WARNING|trainer.py:803] 2025-04-26 18:04:30,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:30,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8084
8101
8045
[WARNING|trainer.py:803] 2025-04-26 18:04:32,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:32,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:32,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8085
8102
[WARNING|trainer.py:803] 2025-04-26 18:04:33,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8046
[WARNING|trainer.py:803] 2025-04-26 18:04:34,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:34,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8086
8103
8047
[WARNING|trainer.py:803] 2025-04-26 18:04:35,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:36,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:36,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8087
8104
8048
[WARNING|trainer.py:803] 2025-04-26 18:04:37,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:38,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:38,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8088
8049
[WARNING|trainer.py:803] 2025-04-26 18:04:39,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8105
[WARNING|trainer.py:803] 2025-04-26 18:04:40,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8089
[WARNING|trainer.py:803] 2025-04-26 18:04:40,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8050
[WARNING|trainer.py:803] 2025-04-26 18:04:41,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8106
[WARNING|trainer.py:803] 2025-04-26 18:04:41,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:42,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8051
8090
[WARNING|trainer.py:803] 2025-04-26 18:04:43,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:04:43,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8107
[WARNING|trainer.py:803] 2025-04-26 18:04:44,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8091
8052
[WARNING|trainer.py:803] 2025-04-26 18:04:45,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:45,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8108
8053
8092
[WARNING|trainer.py:803] 2025-04-26 18:04:46,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:04:46,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:04:47,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8054
8093
8109
[WARNING|trainer.py:803] 2025-04-26 18:04:48,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:48,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:48,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8055
8094
8110
[WARNING|trainer.py:803] 2025-04-26 18:04:50,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:50,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:50,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8056
8095
[WARNING|trainer.py:803] 2025-04-26 18:04:51,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8111
[WARNING|trainer.py:803] 2025-04-26 18:04:52,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8057
[WARNING|trainer.py:803] 2025-04-26 18:04:52,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8096
[WARNING|trainer.py:803] 2025-04-26 18:04:53,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:53,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8112
8058
8097
[WARNING|trainer.py:803] 2025-04-26 18:04:55,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:04:55,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:04:55,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8059
8113
8098
[WARNING|trainer.py:803] 2025-04-26 18:04:56,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:04:57,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8060
[WARNING|trainer.py:803] 2025-04-26 18:04:57,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8114
[WARNING|trainer.py:803] 2025-04-26 18:04:58,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8099
[WARNING|trainer.py:803] 2025-04-26 18:04:59,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:04:59,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8061
8100
[WARNING|trainer.py:803] 2025-04-26 18:05:00,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8115
[WARNING|trainer.py:803] 2025-04-26 18:05:00,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8062
[WARNING|trainer.py:803] 2025-04-26 18:05:01,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:05:01,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8101
8116
8063
[WARNING|trainer.py:803] 2025-04-26 18:05:03,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:03,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:03,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8102
8064
8117
[WARNING|trainer.py:803] 2025-04-26 18:05:05,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:05,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:05,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8065
8103
8118
[WARNING|trainer.py:803] 2025-04-26 18:05:07,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:07,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:07,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8066
8104
8119
[WARNING|trainer.py:803] 2025-04-26 18:05:09,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:05:09,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:09,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8067
[WARNING|trainer.py:803] 2025-04-26 18:05:10,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8105
8120
8068
[WARNING|trainer.py:803] 2025-04-26 18:05:11,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:11,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:12,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8121
8106
8069
[WARNING|trainer.py:803] 2025-04-26 18:05:13,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:13,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:14,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8107
8122
8070
[WARNING|trainer.py:803] 2025-04-26 18:05:15,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:16,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:16,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8071
8108
8123
[WARNING|trainer.py:803] 2025-04-26 18:05:17,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:18,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:05:18,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8072
8109
8124
[WARNING|trainer.py:803] 2025-04-26 18:05:19,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:20,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:20,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8073
8110
[WARNING|trainer.py:803] 2025-04-26 18:05:21,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8125
[WARNING|trainer.py:803] 2025-04-26 18:05:22,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:22,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8074
8111
8126
[WARNING|trainer.py:803] 2025-04-26 18:05:23,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:24,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:24,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8075
8127
8112
[WARNING|trainer.py:803] 2025-04-26 18:05:25,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:26,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:26,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8076
[WARNING|trainer.py:803] 2025-04-26 18:05:27,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8113
8128
8077
[WARNING|trainer.py:803] 2025-04-26 18:05:28,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:28,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:29,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8114
8129
8078
[WARNING|trainer.py:803] 2025-04-26 18:05:30,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:05:30,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:30,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8079
8115
8130
[WARNING|trainer.py:803] 2025-04-26 18:05:32,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:32,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:05:33,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8080
8116
[WARNING|trainer.py:803] 2025-04-26 18:05:34,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8131
[WARNING|trainer.py:803] 2025-04-26 18:05:34,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:35,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8081
8117
[WARNING|trainer.py:803] 2025-04-26 18:05:36,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8132
[WARNING|trainer.py:803] 2025-04-26 18:05:37,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8082
[WARNING|trainer.py:803] 2025-04-26 18:05:37,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:38,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8118
8133
8083
[WARNING|trainer.py:803] 2025-04-26 18:05:39,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:39,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:39,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8119
8084
8134
[WARNING|trainer.py:803] 2025-04-26 18:05:41,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:41,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:41,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8120
8085
8135
[WARNING|trainer.py:803] 2025-04-26 18:05:43,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:43,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:05:43,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8121
8086
8136
[WARNING|trainer.py:803] 2025-04-26 18:05:45,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:45,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:45,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8087
8122
8137
[WARNING|trainer.py:803] 2025-04-26 18:05:47,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:47,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:47,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8088
8123
[WARNING|trainer.py:803] 2025-04-26 18:05:49,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8138
[WARNING|trainer.py:803] 2025-04-26 18:05:49,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8089
[WARNING|trainer.py:803] 2025-04-26 18:05:50,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:50,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8124
8139
[WARNING|trainer.py:803] 2025-04-26 18:05:51,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:52,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8090
8125
[WARNING|trainer.py:803] 2025-04-26 18:05:53,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8140
[WARNING|trainer.py:803] 2025-04-26 18:05:53,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8091
[WARNING|trainer.py:803] 2025-04-26 18:05:54,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:54,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8126
8141
8092
[WARNING|trainer.py:803] 2025-04-26 18:05:55,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:56,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:05:56,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8127
8093
8142
[WARNING|trainer.py:803] 2025-04-26 18:05:58,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:05:58,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:05:58,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8094
8128
8143
[WARNING|trainer.py:803] 2025-04-26 18:06:00,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:06:00,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:00,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8095
8129
[WARNING|trainer.py:803] 2025-04-26 18:06:01,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8144
[WARNING|trainer.py:803] 2025-04-26 18:06:02,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8096
[WARNING|trainer.py:803] 2025-04-26 18:06:02,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:06:03,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8130
8145
8097
[WARNING|trainer.py:803] 2025-04-26 18:06:04,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:05,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:05,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8131
8146
8098
[WARNING|trainer.py:803] 2025-04-26 18:06:06,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:07,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:07,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8099
8132
8147
[WARNING|trainer.py:803] 2025-04-26 18:06:08,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:06:09,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:09,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8100
[WARNING|trainer.py:803] 2025-04-26 18:06:10,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8148
8133
[WARNING|trainer.py:803] 2025-04-26 18:06:11,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:11,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8101
8149
[WARNING|trainer.py:803] 2025-04-26 18:06:12,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8134
[WARNING|trainer.py:803] 2025-04-26 18:06:13,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:13,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8102
[WARNING|trainer.py:803] 2025-04-26 18:06:14,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8150
8135
[WARNING|trainer.py:803] 2025-04-26 18:06:15,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:15,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8103
[WARNING|trainer.py:803] 2025-04-26 18:06:16,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8136
8151
[WARNING|trainer.py:803] 2025-04-26 18:06:17,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:06:17,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8104
8137
[WARNING|trainer.py:803] 2025-04-26 18:06:18,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8152
[WARNING|trainer.py:803] 2025-04-26 18:06:19,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:19,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8105
[WARNING|trainer.py:803] 2025-04-26 18:06:21,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8138
8153
[WARNING|trainer.py:803] 2025-04-26 18:06:21,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:22,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8106
8139
[WARNING|trainer.py:803] 2025-04-26 18:06:23,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8154
[WARNING|trainer.py:803] 2025-04-26 18:06:23,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:24,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8107
[WARNING|trainer.py:803] 2025-04-26 18:06:25,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8140
8155
[WARNING|trainer.py:803] 2025-04-26 18:06:26,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:26,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8108
[WARNING|trainer.py:803] 2025-04-26 18:06:27,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8141
8156
[WARNING|trainer.py:803] 2025-04-26 18:06:28,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:28,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8109
[WARNING|trainer.py:803] 2025-04-26 18:06:29,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8157
8142
[WARNING|trainer.py:803] 2025-04-26 18:06:30,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:30,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8110
[WARNING|trainer.py:803] 2025-04-26 18:06:31,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8158
8143
[WARNING|trainer.py:803] 2025-04-26 18:06:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:32,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8111
[WARNING|trainer.py:803] 2025-04-26 18:06:33,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8159
8144
[WARNING|trainer.py:803] 2025-04-26 18:06:34,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:34,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8112
8160
[WARNING|trainer.py:803] 2025-04-26 18:06:35,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8145
[WARNING|trainer.py:803] 2025-04-26 18:06:36,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:36,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8113
8161
[WARNING|trainer.py:803] 2025-04-26 18:06:37,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8146
[WARNING|trainer.py:803] 2025-04-26 18:06:38,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:38,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8114
8162
[WARNING|trainer.py:803] 2025-04-26 18:06:39,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8147
[WARNING|trainer.py:803] 2025-04-26 18:06:40,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:06:41,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8115
8163
[WARNING|trainer.py:803] 2025-04-26 18:06:42,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8148
[WARNING|trainer.py:803] 2025-04-26 18:06:42,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:06:43,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8116
8164
[WARNING|trainer.py:803] 2025-04-26 18:06:44,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8149
[WARNING|trainer.py:803] 2025-04-26 18:06:44,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:45,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8117
8165
[WARNING|trainer.py:803] 2025-04-26 18:06:46,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8150
[WARNING|trainer.py:803] 2025-04-26 18:06:46,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:47,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8118
8166
[WARNING|trainer.py:803] 2025-04-26 18:06:48,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8151
[WARNING|trainer.py:803] 2025-04-26 18:06:48,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:49,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8119
8167
[WARNING|trainer.py:803] 2025-04-26 18:06:50,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:50,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8152
[WARNING|trainer.py:803] 2025-04-26 18:06:51,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8168
8120
[WARNING|trainer.py:803] 2025-04-26 18:06:52,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:06:52,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8153
[WARNING|trainer.py:803] 2025-04-26 18:06:53,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8121
8169
[WARNING|trainer.py:803] 2025-04-26 18:06:54,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:54,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8154
[WARNING|trainer.py:803] 2025-04-26 18:06:55,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8170
8122
[WARNING|trainer.py:803] 2025-04-26 18:06:56,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:57,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8155
[WARNING|trainer.py:803] 2025-04-26 18:06:57,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8171
8123
[WARNING|trainer.py:803] 2025-04-26 18:06:58,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:06:59,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8156
[WARNING|trainer.py:803] 2025-04-26 18:07:00,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8172
8124
[WARNING|trainer.py:803] 2025-04-26 18:07:01,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8157
[WARNING|trainer.py:803] 2025-04-26 18:07:01,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:02,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8173
8125
[WARNING|trainer.py:803] 2025-04-26 18:07:03,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8158
[WARNING|trainer.py:803] 2025-04-26 18:07:03,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:04,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8174
8126
[WARNING|trainer.py:803] 2025-04-26 18:07:05,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8159
[WARNING|trainer.py:803] 2025-04-26 18:07:05,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:06,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8127
8175
8160
[WARNING|trainer.py:803] 2025-04-26 18:07:07,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:07,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:08,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8176
8128
8161
[WARNING|trainer.py:803] 2025-04-26 18:07:09,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:09,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:10,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8177
8129
8162
[WARNING|trainer.py:803] 2025-04-26 18:07:11,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:12,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:12,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8178
8130
8163
[WARNING|trainer.py:803] 2025-04-26 18:07:13,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:14,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:14,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8179
8164
8131
[WARNING|trainer.py:803] 2025-04-26 18:07:15,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:16,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:16,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8180
8132
8165
[WARNING|trainer.py:803] 2025-04-26 18:07:18,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:07:18,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:18,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8181
8166
8133
[WARNING|trainer.py:803] 2025-04-26 18:07:20,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:20,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:20,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8182
8167
[WARNING|trainer.py:803] 2025-04-26 18:07:22,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8134
[WARNING|trainer.py:803] 2025-04-26 18:07:22,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:23,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8183
8168
8135
[WARNING|trainer.py:803] 2025-04-26 18:07:24,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:24,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:25,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8184
8169
8136
[WARNING|trainer.py:803] 2025-04-26 18:07:26,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:26,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:27,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8185
8170
8137
[WARNING|trainer.py:803] 2025-04-26 18:07:28,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:28,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:29,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8171
8186
8138
[WARNING|trainer.py:803] 2025-04-26 18:07:30,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:31,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:31,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8172
8187
8139
[WARNING|trainer.py:803] 2025-04-26 18:07:33,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:33,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:07:33,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8173
8188
8140
[WARNING|trainer.py:803] 2025-04-26 18:07:35,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:07:35,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:35,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8174
8189
8141
[WARNING|trainer.py:803] 2025-04-26 18:07:37,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:37,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:37,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8190
8175
8142
[WARNING|trainer.py:803] 2025-04-26 18:07:39,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:39,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:40,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8191
8176
8143
[WARNING|trainer.py:803] 2025-04-26 18:07:41,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:07:41,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:42,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8177
8192
8144
[WARNING|trainer.py:803] 2025-04-26 18:07:44,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:44,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:44,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8178
8193
8145
[WARNING|trainer.py:803] 2025-04-26 18:07:46,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:46,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:46,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8179
8194
8146
[WARNING|trainer.py:803] 2025-04-26 18:07:48,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:48,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:07:48,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8180
8195
8147
[WARNING|trainer.py:803] 2025-04-26 18:07:50,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:07:50,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:50,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8181
8196
8148
[WARNING|trainer.py:803] 2025-04-26 18:07:52,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:52,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:52,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8182
8197
8149
[WARNING|trainer.py:803] 2025-04-26 18:07:54,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:54,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:07:54,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8183
8198
8150
[WARNING|trainer.py:803] 2025-04-26 18:07:56,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:56,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:07:57,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8184
8199
8151
[WARNING|trainer.py:803] 2025-04-26 18:07:58,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:59,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:07:59,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8185
8200
8152
[WARNING|trainer.py:803] 2025-04-26 18:08:01,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:01,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:01,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8186
8201
8153
[WARNING|trainer.py:803] 2025-04-26 18:08:03,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:03,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:03,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8202
8187
8154
[WARNING|trainer.py:803] 2025-04-26 18:08:05,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:05,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:05,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8188
8203
8155
[WARNING|trainer.py:803] 2025-04-26 18:08:07,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:07,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:08:07,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8189
8156
8204
[WARNING|trainer.py:803] 2025-04-26 18:08:09,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:09,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:10,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8190
8157
8205
[WARNING|trainer.py:803] 2025-04-26 18:08:11,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:11,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:12,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8191
8158
8206
[WARNING|trainer.py:803] 2025-04-26 18:08:13,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:08:14,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:14,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8159
8192
8207
[WARNING|trainer.py:803] 2025-04-26 18:08:16,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:16,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:16,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8160
8193
8208
[WARNING|trainer.py:803] 2025-04-26 18:08:18,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:18,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:18,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8161
8194
8209
[WARNING|trainer.py:803] 2025-04-26 18:08:20,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:20,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:20,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8162
8195
8210
[WARNING|trainer.py:803] 2025-04-26 18:08:22,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:22,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:22,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8163
8196
8211
[WARNING|trainer.py:803] 2025-04-26 18:08:24,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:24,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:24,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8197
8164
8212
[WARNING|trainer.py:803] 2025-04-26 18:08:26,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:26,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:26,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8165
8213
8198
[WARNING|trainer.py:803] 2025-04-26 18:08:28,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:29,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:29,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8166
8214
8199
[WARNING|trainer.py:803] 2025-04-26 18:08:30,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:31,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:31,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8167
8215
8200
[WARNING|trainer.py:803] 2025-04-26 18:08:33,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:33,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:33,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8168
8216
8201
[WARNING|trainer.py:803] 2025-04-26 18:08:35,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:35,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:35,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8169
8217
8202
[WARNING|trainer.py:803] 2025-04-26 18:08:37,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:37,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:37,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8170
8203
8218
[WARNING|trainer.py:803] 2025-04-26 18:08:39,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:39,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:08:40,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8171
8204
8219
[WARNING|trainer.py:803] 2025-04-26 18:08:41,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:42,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:08:42,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8172
8205
8220
[WARNING|trainer.py:803] 2025-04-26 18:08:43,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:44,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:44,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8173
8206
8221
[WARNING|trainer.py:803] 2025-04-26 18:08:45,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:08:46,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:46,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8174
8207
[WARNING|trainer.py:803] 2025-04-26 18:08:47,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8222
[WARNING|trainer.py:803] 2025-04-26 18:08:48,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:49,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8175
8208
[WARNING|trainer.py:803] 2025-04-26 18:08:50,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8223
[WARNING|trainer.py:803] 2025-04-26 18:08:50,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:51,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8176
8209
[WARNING|trainer.py:803] 2025-04-26 18:08:52,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8224
[WARNING|trainer.py:803] 2025-04-26 18:08:52,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:08:53,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8177
8210
[WARNING|trainer.py:803] 2025-04-26 18:08:54,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8225
[WARNING|trainer.py:803] 2025-04-26 18:08:55,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:08:55,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8178
8211
[WARNING|trainer.py:803] 2025-04-26 18:08:56,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8226
[WARNING|trainer.py:803] 2025-04-26 18:08:56,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8179
[WARNING|trainer.py:803] 2025-04-26 18:08:57,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8212
[WARNING|trainer.py:803] 2025-04-26 18:08:58,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:08:58,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8227
8180
[WARNING|trainer.py:803] 2025-04-26 18:08:59,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8213
[WARNING|trainer.py:803] 2025-04-26 18:09:00,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:01,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8228
8181
[WARNING|trainer.py:803] 2025-04-26 18:09:01,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8214
[WARNING|trainer.py:803] 2025-04-26 18:09:02,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:03,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8229
8182
[WARNING|trainer.py:803] 2025-04-26 18:09:04,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8215
[WARNING|trainer.py:803] 2025-04-26 18:09:04,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:05,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8230
8183
[WARNING|trainer.py:803] 2025-04-26 18:09:06,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8216
[WARNING|trainer.py:803] 2025-04-26 18:09:07,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:07,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8231
[WARNING|trainer.py:803] 2025-04-26 18:09:08,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8184
8217
[WARNING|trainer.py:803] 2025-04-26 18:09:09,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:09,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8232
8185
[WARNING|trainer.py:803] 2025-04-26 18:09:10,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8218
[WARNING|trainer.py:803] 2025-04-26 18:09:11,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8233
[WARNING|trainer.py:803] 2025-04-26 18:09:11,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:12,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8186
8219
[WARNING|trainer.py:803] 2025-04-26 18:09:13,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8234
[WARNING|trainer.py:803] 2025-04-26 18:09:14,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:14,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8187
8220
[WARNING|trainer.py:803] 2025-04-26 18:09:15,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8235
[WARNING|trainer.py:803] 2025-04-26 18:09:16,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:17,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8188
8221
[WARNING|trainer.py:803] 2025-04-26 18:09:17,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8236
[WARNING|trainer.py:803] 2025-04-26 18:09:18,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8189
[WARNING|trainer.py:803] 2025-04-26 18:09:19,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:19,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8222
8237
[WARNING|trainer.py:803] 2025-04-26 18:09:20,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8190
[WARNING|trainer.py:803] 2025-04-26 18:09:21,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:09:21,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8223
8238
[WARNING|trainer.py:803] 2025-04-26 18:09:23,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8191
[WARNING|trainer.py:803] 2025-04-26 18:09:23,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:24,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8224
8239
[WARNING|trainer.py:803] 2025-04-26 18:09:25,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:25,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8192
[WARNING|trainer.py:803] 2025-04-26 18:09:26,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8225
8240
[WARNING|trainer.py:803] 2025-04-26 18:09:27,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:27,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8193
[WARNING|trainer.py:803] 2025-04-26 18:09:28,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8226
8241
[WARNING|trainer.py:803] 2025-04-26 18:09:29,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:29,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8194
[WARNING|trainer.py:803] 2025-04-26 18:09:30,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8227
8242
[WARNING|trainer.py:803] 2025-04-26 18:09:31,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:31,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8195
[WARNING|trainer.py:803] 2025-04-26 18:09:32,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8228
8243
[WARNING|trainer.py:803] 2025-04-26 18:09:33,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:09:33,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8196
[WARNING|trainer.py:803] 2025-04-26 18:09:34,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8244
8229
[WARNING|trainer.py:803] 2025-04-26 18:09:36,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8197
[WARNING|trainer.py:803] 2025-04-26 18:09:36,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:36,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8245
8230
[WARNING|trainer.py:803] 2025-04-26 18:09:38,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:38,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8198
8246
8231
[WARNING|trainer.py:803] 2025-04-26 18:09:39,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:09:40,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:40,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8199
8232
8247
[WARNING|trainer.py:803] 2025-04-26 18:09:41,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:42,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:42,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8200
8233
[WARNING|trainer.py:803] 2025-04-26 18:09:43,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8248
[WARNING|trainer.py:803] 2025-04-26 18:09:44,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:44,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8201
8234
8249
[WARNING|trainer.py:803] 2025-04-26 18:09:45,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:46,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:09:46,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8202
8250
8235
[WARNING|trainer.py:803] 2025-04-26 18:09:48,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:48,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:48,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8203
8251
[WARNING|trainer.py:803] 2025-04-26 18:09:50,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8236
[WARNING|trainer.py:803] 2025-04-26 18:09:50,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:51,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8204
8237
8252
[WARNING|trainer.py:803] 2025-04-26 18:09:52,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:53,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:09:53,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8205
8238
8253
[WARNING|trainer.py:803] 2025-04-26 18:09:54,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:55,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:55,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8206
8239
8254
[WARNING|trainer.py:803] 2025-04-26 18:09:57,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:57,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:09:57,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8207
8240
8255
[WARNING|trainer.py:803] 2025-04-26 18:09:59,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:59,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:09:59,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8208
8241
8256
[WARNING|trainer.py:803] 2025-04-26 18:10:01,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:01,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:01,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8209
8242
8257
[WARNING|trainer.py:803] 2025-04-26 18:10:03,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:10:03,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:04,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8210
8243
8258
[WARNING|trainer.py:803] 2025-04-26 18:10:05,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:05,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:06,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8211
8244
8259
[WARNING|trainer.py:803] 2025-04-26 18:10:07,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:07,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:08,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8212
8245
8260
[WARNING|trainer.py:803] 2025-04-26 18:10:09,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:10,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:10,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8213
8246
[WARNING|trainer.py:803] 2025-04-26 18:10:11,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8261
[WARNING|trainer.py:803] 2025-04-26 18:10:12,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:12,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8214
8247
8262
[WARNING|trainer.py:803] 2025-04-26 18:10:13,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:14,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:14,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8215
8248
8263
[WARNING|trainer.py:803] 2025-04-26 18:10:16,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:16,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:16,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8216
8249
[WARNING|trainer.py:803] 2025-04-26 18:10:18,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8264
[WARNING|trainer.py:803] 2025-04-26 18:10:18,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:19,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8217
8250
8265
[WARNING|trainer.py:803] 2025-04-26 18:10:20,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:20,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:21,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8218
8251
8266
[WARNING|trainer.py:803] 2025-04-26 18:10:22,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:22,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:23,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8219
8252
8267
[WARNING|trainer.py:803] 2025-04-26 18:10:25,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:25,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:25,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8253
8220
8268
[WARNING|trainer.py:803] 2025-04-26 18:10:27,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:27,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:27,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8254
8221
8269
[WARNING|trainer.py:803] 2025-04-26 18:10:29,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:29,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:29,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8270
8255
8222
[WARNING|trainer.py:803] 2025-04-26 18:10:31,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:10:31,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:32,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8271
8256
8223
[WARNING|trainer.py:803] 2025-04-26 18:10:33,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:10:34,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:34,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8272
8257
8224
[WARNING|trainer.py:803] 2025-04-26 18:10:35,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:36,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:36,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8273
8258
8225
[WARNING|trainer.py:803] 2025-04-26 18:10:38,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:38,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:38,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8274
8259
8226
[WARNING|trainer.py:803] 2025-04-26 18:10:40,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:40,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:40,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8275
8260
[WARNING|trainer.py:803] 2025-04-26 18:10:42,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8227
[WARNING|trainer.py:803] 2025-04-26 18:10:42,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:42,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8276
8261
8228
[WARNING|trainer.py:803] 2025-04-26 18:10:44,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:44,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:44,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8277
8262
8229
[WARNING|trainer.py:803] 2025-04-26 18:10:46,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:46,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:47,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8278
8263
8230
[WARNING|trainer.py:803] 2025-04-26 18:10:48,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:49,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:49,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8279
8231
8264
[WARNING|trainer.py:803] 2025-04-26 18:10:50,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:51,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:51,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8280
8232
8265
[WARNING|trainer.py:803] 2025-04-26 18:10:53,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:53,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:53,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8281
8233
8266
[WARNING|trainer.py:803] 2025-04-26 18:10:55,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:55,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:10:55,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8282
8267
8234
[WARNING|trainer.py:803] 2025-04-26 18:10:57,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:10:57,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:10:57,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8283
8268
8235
[WARNING|trainer.py:803] 2025-04-26 18:10:59,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:10:59,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:00,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8284
8269
8236
[WARNING|trainer.py:803] 2025-04-26 18:11:01,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:02,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:02,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8285
8270
8237
[WARNING|trainer.py:803] 2025-04-26 18:11:04,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:04,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:11:04,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8286
8271
8238
[WARNING|trainer.py:803] 2025-04-26 18:11:06,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:06,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:11:06,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8272
8287
8239
[WARNING|trainer.py:803] 2025-04-26 18:11:08,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:08,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:08,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8273
8288
8240
[WARNING|trainer.py:803] 2025-04-26 18:11:10,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:10,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:10,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8274
8289
8241
[WARNING|trainer.py:803] 2025-04-26 18:11:12,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:12,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:11:12,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8275
8290
8242
[WARNING|trainer.py:803] 2025-04-26 18:11:14,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:14,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:15,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8276
8291
8243
[WARNING|trainer.py:803] 2025-04-26 18:11:16,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:16,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:17,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8277
8244
8292
[WARNING|trainer.py:803] 2025-04-26 18:11:18,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:19,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:19,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8278
8245
8293
[WARNING|trainer.py:803] 2025-04-26 18:11:21,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:21,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:21,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8246
8279
8294
[WARNING|trainer.py:803] 2025-04-26 18:11:23,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:23,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:23,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8247
8280
8295
[WARNING|trainer.py:803] 2025-04-26 18:11:25,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:25,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:25,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8281
8248
8296
[WARNING|trainer.py:803] 2025-04-26 18:11:27,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:27,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:28,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8249
8282
8297
[WARNING|trainer.py:803] 2025-04-26 18:11:29,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:29,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:30,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8250
8283
8298
[WARNING|trainer.py:803] 2025-04-26 18:11:32,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:32,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:11:32,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8284
8251
8299
[WARNING|trainer.py:803] 2025-04-26 18:11:34,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:34,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:34,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8285
8252
8300
[WARNING|trainer.py:803] 2025-04-26 18:11:36,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:36,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:36,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8286
8253
8301
[WARNING|trainer.py:803] 2025-04-26 18:11:38,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:38,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:38,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8287
8254
8302
[WARNING|trainer.py:803] 2025-04-26 18:11:40,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:41,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:41,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8288
8303
8255
[WARNING|trainer.py:803] 2025-04-26 18:11:43,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:43,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:43,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8289
8304
8256
[WARNING|trainer.py:803] 2025-04-26 18:11:45,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:11:45,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:45,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8290
8305
8257
[WARNING|trainer.py:803] 2025-04-26 18:11:47,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:47,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:11:47,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8291
8306
8258
[WARNING|trainer.py:803] 2025-04-26 18:11:49,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:49,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:50,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8307
8292
8259
[WARNING|trainer.py:803] 2025-04-26 18:11:51,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:51,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:51,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8308
8260
8293
[WARNING|trainer.py:803] 2025-04-26 18:11:53,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:11:54,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:54,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8309
8261
8294
[WARNING|trainer.py:803] 2025-04-26 18:11:55,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:56,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:56,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8310
8262
[WARNING|trainer.py:803] 2025-04-26 18:11:57,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8295
[WARNING|trainer.py:803] 2025-04-26 18:11:58,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:11:58,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8311
[WARNING|trainer.py:803] 2025-04-26 18:11:59,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8263
8296
[WARNING|trainer.py:803] 2025-04-26 18:12:00,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8312
[WARNING|trainer.py:803] 2025-04-26 18:12:00,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:12:01,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8264
8297
8313
[WARNING|trainer.py:803] 2025-04-26 18:12:02,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:12:03,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:12:03,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8265
8298
8314
[WARNING|trainer.py:803] 2025-04-26 18:12:04,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:05,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:05,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8266
8299
8315
[WARNING|trainer.py:803] 2025-04-26 18:12:07,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:07,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:07,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8267
8316
8300
[WARNING|trainer.py:803] 2025-04-26 18:12:09,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:09,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:09,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8268
8317
8301
[WARNING|trainer.py:803] 2025-04-26 18:12:11,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:11,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:11,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8269
8318
8302
[WARNING|trainer.py:803] 2025-04-26 18:12:13,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:12:13,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:14,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8270
8319
8303
[WARNING|trainer.py:803] 2025-04-26 18:12:15,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:15,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:16,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8271
8320
8304
[WARNING|trainer.py:803] 2025-04-26 18:12:17,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:17,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:12:18,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8272
8321
8305
[WARNING|trainer.py:803] 2025-04-26 18:12:19,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:19,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:20,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8322
8273
8306
[WARNING|trainer.py:803] 2025-04-26 18:12:21,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:21,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:12:22,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8274
8323
8307
[WARNING|trainer.py:803] 2025-04-26 18:12:23,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:23,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:24,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8324
8275
8308
[WARNING|trainer.py:803] 2025-04-26 18:12:25,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:25,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:26,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8325
8276
8309
[WARNING|trainer.py:803] 2025-04-26 18:12:27,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:28,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:28,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8326
8277
8310
[WARNING|trainer.py:803] 2025-04-26 18:12:29,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:30,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:12:30,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8327
8311
8278
[WARNING|trainer.py:803] 2025-04-26 18:12:31,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:32,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:32,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8328
8312
8279
[WARNING|trainer.py:803] 2025-04-26 18:12:33,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:34,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:34,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8329
8313
[WARNING|trainer.py:803] 2025-04-26 18:12:35,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8280
[WARNING|trainer.py:803] 2025-04-26 18:12:36,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:36,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8330
8314
[WARNING|trainer.py:803] 2025-04-26 18:12:37,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8281
[WARNING|trainer.py:803] 2025-04-26 18:12:38,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8331
[WARNING|trainer.py:803] 2025-04-26 18:12:39,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8315
[WARNING|trainer.py:803] 2025-04-26 18:12:39,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8282
[WARNING|trainer.py:803] 2025-04-26 18:12:40,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8332
[WARNING|trainer.py:803] 2025-04-26 18:12:41,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8316
[WARNING|trainer.py:803] 2025-04-26 18:12:41,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:42,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8283
8333
[WARNING|trainer.py:803] 2025-04-26 18:12:43,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:43,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8317
[WARNING|trainer.py:803] 2025-04-26 18:12:44,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8284
8334
[WARNING|trainer.py:803] 2025-04-26 18:12:45,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:45,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8318
[WARNING|trainer.py:803] 2025-04-26 18:12:46,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8335
8285
[WARNING|trainer.py:803] 2025-04-26 18:12:47,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:12:47,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8319
[WARNING|trainer.py:803] 2025-04-26 18:12:48,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8336
8286
[WARNING|trainer.py:803] 2025-04-26 18:12:49,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:49,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8320
[WARNING|trainer.py:803] 2025-04-26 18:12:50,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8337
8287
[WARNING|trainer.py:803] 2025-04-26 18:12:51,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8321
[WARNING|trainer.py:803] 2025-04-26 18:12:52,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:52,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8338
8288
[WARNING|trainer.py:803] 2025-04-26 18:12:53,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8322
[WARNING|trainer.py:803] 2025-04-26 18:12:54,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:12:54,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8339
8289
[WARNING|trainer.py:803] 2025-04-26 18:12:55,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8323
[WARNING|trainer.py:803] 2025-04-26 18:12:56,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:12:56,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8340
8290
[WARNING|trainer.py:803] 2025-04-26 18:12:58,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8324
[WARNING|trainer.py:803] 2025-04-26 18:12:58,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:12:58,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8341
8291
[WARNING|trainer.py:803] 2025-04-26 18:13:00,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8325
[WARNING|trainer.py:803] 2025-04-26 18:13:00,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:00,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8342
8326
[WARNING|trainer.py:803] 2025-04-26 18:13:02,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8292
[WARNING|trainer.py:803] 2025-04-26 18:13:02,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:03,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8343
8327
[WARNING|trainer.py:803] 2025-04-26 18:13:04,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8293
[WARNING|trainer.py:803] 2025-04-26 18:13:04,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8344
[WARNING|trainer.py:803] 2025-04-26 18:13:05,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8328
[WARNING|trainer.py:803] 2025-04-26 18:13:06,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:06,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8294
8345
[WARNING|trainer.py:803] 2025-04-26 18:13:07,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:08,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8329
[WARNING|trainer.py:803] 2025-04-26 18:13:08,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8295
8346
8330
[WARNING|trainer.py:803] 2025-04-26 18:13:10,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:10,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:10,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8347
8296
8331
[WARNING|trainer.py:803] 2025-04-26 18:13:12,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:12,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:12,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8348
8297
8332
[WARNING|trainer.py:803] 2025-04-26 18:13:14,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:14,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:14,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8349
8333
8298
[WARNING|trainer.py:803] 2025-04-26 18:13:16,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:16,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:13:16,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8350
8334
8299
[WARNING|trainer.py:803] 2025-04-26 18:13:18,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:18,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:19,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8351
8335
8300
[WARNING|trainer.py:803] 2025-04-26 18:13:20,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:20,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:21,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8352
8336
[WARNING|trainer.py:803] 2025-04-26 18:13:22,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8301
[WARNING|trainer.py:803] 2025-04-26 18:13:22,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:23,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8353
8337
[WARNING|trainer.py:803] 2025-04-26 18:13:24,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8302
[WARNING|trainer.py:803] 2025-04-26 18:13:24,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:25,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8354
8338
[WARNING|trainer.py:803] 2025-04-26 18:13:26,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8303
[WARNING|trainer.py:803] 2025-04-26 18:13:27,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:27,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8355
8339
[WARNING|trainer.py:803] 2025-04-26 18:13:28,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8304
[WARNING|trainer.py:803] 2025-04-26 18:13:29,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:29,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8356
8340
[WARNING|trainer.py:803] 2025-04-26 18:13:30,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8305
[WARNING|trainer.py:803] 2025-04-26 18:13:31,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:31,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8357
8341
[WARNING|trainer.py:803] 2025-04-26 18:13:32,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8306
[WARNING|trainer.py:803] 2025-04-26 18:13:33,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:33,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8358
8342
[WARNING|trainer.py:803] 2025-04-26 18:13:35,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8307
[WARNING|trainer.py:803] 2025-04-26 18:13:35,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:36,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8359
8343
[WARNING|trainer.py:803] 2025-04-26 18:13:37,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8308
[WARNING|trainer.py:803] 2025-04-26 18:13:37,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:38,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8360
8344
[WARNING|trainer.py:803] 2025-04-26 18:13:39,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8309
[WARNING|trainer.py:803] 2025-04-26 18:13:39,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:40,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8361
8345
[WARNING|trainer.py:803] 2025-04-26 18:13:41,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8310
[WARNING|trainer.py:803] 2025-04-26 18:13:41,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:42,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8362
8346
[WARNING|trainer.py:803] 2025-04-26 18:13:43,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8311
[WARNING|trainer.py:803] 2025-04-26 18:13:43,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:44,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8363
8347
8312
[WARNING|trainer.py:803] 2025-04-26 18:13:45,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:45,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:46,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8364
8348
8313
[WARNING|trainer.py:803] 2025-04-26 18:13:47,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:47,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:48,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8365
8349
8314
[WARNING|trainer.py:803] 2025-04-26 18:13:49,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:49,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:50,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8366
8350
8315
[WARNING|trainer.py:803] 2025-04-26 18:13:51,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:51,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:13:52,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8367
8351
8316
[WARNING|trainer.py:803] 2025-04-26 18:13:53,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:53,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:54,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8368
8352
[WARNING|trainer.py:803] 2025-04-26 18:13:55,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8317
[WARNING|trainer.py:803] 2025-04-26 18:13:55,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:13:56,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8369
8353
[WARNING|trainer.py:803] 2025-04-26 18:13:57,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8318
[WARNING|trainer.py:803] 2025-04-26 18:13:57,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:13:58,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8370
8354
[WARNING|trainer.py:803] 2025-04-26 18:13:59,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8319
[WARNING|trainer.py:803] 2025-04-26 18:13:59,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:00,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8371
8355
[WARNING|trainer.py:803] 2025-04-26 18:14:01,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8320
[WARNING|trainer.py:803] 2025-04-26 18:14:01,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:02,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8372
8356
8321
[WARNING|trainer.py:803] 2025-04-26 18:14:03,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:04,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:04,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8373
8357
8322
[WARNING|trainer.py:803] 2025-04-26 18:14:05,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:06,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:06,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8374
8358
8323
[WARNING|trainer.py:803] 2025-04-26 18:14:08,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:08,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:08,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8359
8375
8324
[WARNING|trainer.py:803] 2025-04-26 18:14:10,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:10,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:10,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8360
8376
8325
[WARNING|trainer.py:803] 2025-04-26 18:14:12,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:12,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:12,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8361
8377
8326
[WARNING|trainer.py:803] 2025-04-26 18:14:14,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:14,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:14,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8362
8378
8327
[WARNING|trainer.py:803] 2025-04-26 18:14:16,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:16,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:16,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8363
8379
8328
[WARNING|trainer.py:803] 2025-04-26 18:14:18,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:18,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:18,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8364
8380
8329
[WARNING|trainer.py:803] 2025-04-26 18:14:20,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:20,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:20,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8365
8381
8330
[WARNING|trainer.py:803] 2025-04-26 18:14:22,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:22,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:22,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8366
8331
8382
[WARNING|trainer.py:803] 2025-04-26 18:14:24,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:24,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:24,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8367
8332
8383
[WARNING|trainer.py:803] 2025-04-26 18:14:26,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:26,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:26,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8333
8368
8384
[WARNING|trainer.py:803] 2025-04-26 18:14:28,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:28,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:28,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8369
8334
8385
[WARNING|trainer.py:803] 2025-04-26 18:14:30,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:30,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:31,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8335
8370
8386
[WARNING|trainer.py:803] 2025-04-26 18:14:32,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:32,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:33,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8336
8371
8387
[WARNING|trainer.py:803] 2025-04-26 18:14:34,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:35,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:35,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8337
8372
8388
[WARNING|trainer.py:803] 2025-04-26 18:14:37,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:37,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:37,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8338
8373
8389
[WARNING|trainer.py:803] 2025-04-26 18:14:39,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:39,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:39,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8339
8374
[WARNING|trainer.py:803] 2025-04-26 18:14:41,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8390
[WARNING|trainer.py:803] 2025-04-26 18:14:41,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:42,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8340
8375
8391
[WARNING|trainer.py:803] 2025-04-26 18:14:43,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:43,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:44,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8341
8376
[WARNING|trainer.py:803] 2025-04-26 18:14:45,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8392
[WARNING|trainer.py:803] 2025-04-26 18:14:45,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:14:46,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8342
8377
8393
[WARNING|trainer.py:803] 2025-04-26 18:14:47,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:47,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:48,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8378
8343
8394
[WARNING|trainer.py:803] 2025-04-26 18:14:49,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:49,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:50,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8379
8344
8395
[WARNING|trainer.py:803] 2025-04-26 18:14:51,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:51,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:52,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8345
8380
[WARNING|trainer.py:803] 2025-04-26 18:14:53,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:14:53,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8396
[WARNING|trainer.py:803] 2025-04-26 18:14:54,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8346
8381
8397
[WARNING|trainer.py:803] 2025-04-26 18:14:55,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:55,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:14:56,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8347
8382
[WARNING|trainer.py:803] 2025-04-26 18:14:57,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:14:57,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8398
[WARNING|trainer.py:803] 2025-04-26 18:14:58,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8348
8383
[WARNING|trainer.py:803] 2025-04-26 18:14:59,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8399
[WARNING|trainer.py:803] 2025-04-26 18:14:59,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:00,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8349
8384
8400
[WARNING|trainer.py:803] 2025-04-26 18:15:01,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:01,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:02,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8385
8350
8401
[WARNING|trainer.py:803] 2025-04-26 18:15:03,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:04,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:04,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8351
8386
8402
[WARNING|trainer.py:803] 2025-04-26 18:15:06,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:06,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:06,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8352
8403
8387
[WARNING|trainer.py:803] 2025-04-26 18:15:08,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:08,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:08,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8404
8353
8388
[WARNING|trainer.py:803] 2025-04-26 18:15:09,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:10,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:10,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8405
8354
[WARNING|trainer.py:803] 2025-04-26 18:15:11,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8389
[WARNING|trainer.py:803] 2025-04-26 18:15:12,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:12,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8406
8355
[WARNING|trainer.py:803] 2025-04-26 18:15:13,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8390
[WARNING|trainer.py:803] 2025-04-26 18:15:14,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8407
[WARNING|trainer.py:803] 2025-04-26 18:15:14,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:15,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8356
8391
8408
[WARNING|trainer.py:803] 2025-04-26 18:15:16,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:17,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:17,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8357
8409
8392
[WARNING|trainer.py:803] 2025-04-26 18:15:18,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:18,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:19,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8358
8410
8393
[WARNING|trainer.py:803] 2025-04-26 18:15:20,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:20,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:21,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8411
8359
8394
[WARNING|trainer.py:803] 2025-04-26 18:15:22,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:22,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:23,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8412
8360
8395
[WARNING|trainer.py:803] 2025-04-26 18:15:24,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:24,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:25,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8413
8361
[WARNING|trainer.py:803] 2025-04-26 18:15:26,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8396
[WARNING|trainer.py:803] 2025-04-26 18:15:26,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8414
[WARNING|trainer.py:803] 2025-04-26 18:15:27,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8362
[WARNING|trainer.py:803] 2025-04-26 18:15:27,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8397
[WARNING|trainer.py:803] 2025-04-26 18:15:28,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8415
[WARNING|trainer.py:803] 2025-04-26 18:15:29,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:29,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8363
8398
[WARNING|trainer.py:803] 2025-04-26 18:15:30,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8416
[WARNING|trainer.py:803] 2025-04-26 18:15:31,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:31,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8364
8417
8399
[WARNING|trainer.py:803] 2025-04-26 18:15:32,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:33,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:33,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8365
8418
8400
[WARNING|trainer.py:803] 2025-04-26 18:15:34,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:35,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:35,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8366
8419
8401
[WARNING|trainer.py:803] 2025-04-26 18:15:36,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:37,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:37,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8420
8367
8402
[WARNING|trainer.py:803] 2025-04-26 18:15:38,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:38,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:39,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8421
8403
8368
[WARNING|trainer.py:803] 2025-04-26 18:15:40,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:40,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:41,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8422
8404
8369
[WARNING|trainer.py:803] 2025-04-26 18:15:42,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:42,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:43,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8423
8405
8370
[WARNING|trainer.py:803] 2025-04-26 18:15:44,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:44,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:15:45,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8424
8406
[WARNING|trainer.py:803] 2025-04-26 18:15:46,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:46,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8371
[WARNING|trainer.py:803] 2025-04-26 18:15:47,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8407
8425
[WARNING|trainer.py:803] 2025-04-26 18:15:48,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:48,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8372
8426
8408
[WARNING|trainer.py:803] 2025-04-26 18:15:49,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:49,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:49,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8373
8427
8409
[WARNING|trainer.py:803] 2025-04-26 18:15:51,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:15:51,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:51,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8374
8428
8410
[WARNING|trainer.py:803] 2025-04-26 18:15:53,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:53,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:53,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8429
8411
8375
[WARNING|trainer.py:803] 2025-04-26 18:15:55,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:55,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:55,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8412
8430
8376
[WARNING|trainer.py:803] 2025-04-26 18:15:57,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:15:57,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:57,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8413
8431
8377
[WARNING|trainer.py:803] 2025-04-26 18:15:58,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:59,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:15:59,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8414
8432
[WARNING|trainer.py:803] 2025-04-26 18:16:00,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:00,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8378
8415
[WARNING|trainer.py:803] 2025-04-26 18:16:01,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8433
[WARNING|trainer.py:803] 2025-04-26 18:16:02,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:02,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8379
8416
8434
[WARNING|trainer.py:803] 2025-04-26 18:16:03,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:04,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:04,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8380
8417
8435
[WARNING|trainer.py:803] 2025-04-26 18:16:05,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:05,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:06,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8418
8381
8436
[WARNING|trainer.py:803] 2025-04-26 18:16:07,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:07,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:08,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8419
8437
8382
[WARNING|trainer.py:803] 2025-04-26 18:16:09,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:09,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:09,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8420
8438
8383
[WARNING|trainer.py:803] 2025-04-26 18:16:11,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:11,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:11,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8421
8439
[WARNING|trainer.py:803] 2025-04-26 18:16:13,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8384
[WARNING|trainer.py:803] 2025-04-26 18:16:13,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:14,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8422
8440
[WARNING|trainer.py:803] 2025-04-26 18:16:14,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:15,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8385
8423
[WARNING|trainer.py:803] 2025-04-26 18:16:16,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8441
[WARNING|trainer.py:803] 2025-04-26 18:16:16,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:17,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8386
8424
8442
[WARNING|trainer.py:803] 2025-04-26 18:16:18,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:18,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:18,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8425
8387
8443
[WARNING|trainer.py:803] 2025-04-26 18:16:20,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:20,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:20,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8426
8388
8444
[WARNING|trainer.py:803] 2025-04-26 18:16:22,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:22,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:22,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8427
8445
8389
[WARNING|trainer.py:803] 2025-04-26 18:16:24,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:24,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:24,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8428
8446
[WARNING|trainer.py:803] 2025-04-26 18:16:25,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8390
[WARNING|trainer.py:803] 2025-04-26 18:16:26,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8429
[WARNING|trainer.py:803] 2025-04-26 18:16:27,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8447
[WARNING|trainer.py:803] 2025-04-26 18:16:27,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:28,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8391
8430
8448
[WARNING|trainer.py:803] 2025-04-26 18:16:29,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:29,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:29,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8392
8431
8449
[WARNING|trainer.py:803] 2025-04-26 18:16:31,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:31,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:31,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8432
8393
8450
[WARNING|trainer.py:803] 2025-04-26 18:16:33,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:16:33,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:33,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8433
8394
8451
[WARNING|trainer.py:803] 2025-04-26 18:16:34,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:35,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:35,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8434
8452
8395
[WARNING|trainer.py:803] 2025-04-26 18:16:36,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:37,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:37,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8435
8453
[WARNING|trainer.py:803] 2025-04-26 18:16:38,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8396
[WARNING|trainer.py:803] 2025-04-26 18:16:39,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:39,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8436
8454
[WARNING|trainer.py:803] 2025-04-26 18:16:40,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8397
[WARNING|trainer.py:803] 2025-04-26 18:16:40,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8437
[WARNING|trainer.py:803] 2025-04-26 18:16:41,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8455
[WARNING|trainer.py:803] 2025-04-26 18:16:42,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:42,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8398
8438
8456
[WARNING|trainer.py:803] 2025-04-26 18:16:43,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:44,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:44,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8399
8439
8457
[WARNING|trainer.py:803] 2025-04-26 18:16:45,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:45,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:46,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8440
8458
8400
[WARNING|trainer.py:803] 2025-04-26 18:16:47,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:47,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:47,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8441
8459
8401
[WARNING|trainer.py:803] 2025-04-26 18:16:49,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:49,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:49,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8442
8460
8402
[WARNING|trainer.py:803] 2025-04-26 18:16:51,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:51,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:51,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8443
8461
8403
[WARNING|trainer.py:803] 2025-04-26 18:16:53,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:53,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:53,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8462
8444
8404
[WARNING|trainer.py:803] 2025-04-26 18:16:55,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:55,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:16:55,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8463
8445
8405
[WARNING|trainer.py:803] 2025-04-26 18:16:56,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:16:56,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:56,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8464
8446
8406
[WARNING|trainer.py:803] 2025-04-26 18:16:58,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:16:58,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:16:58,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8407
8465
8447
[WARNING|trainer.py:803] 2025-04-26 18:17:00,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:00,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:00,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8466
8448
8408
[WARNING|trainer.py:803] 2025-04-26 18:17:02,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:02,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:02,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8467
8449
8409
[WARNING|trainer.py:803] 2025-04-26 18:17:03,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:04,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:04,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8468
8410
8450
[WARNING|trainer.py:803] 2025-04-26 18:17:05,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:05,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:05,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8469
8411
8451
[WARNING|trainer.py:803] 2025-04-26 18:17:07,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:07,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:07,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8470
8412
8452
[WARNING|trainer.py:803] 2025-04-26 18:17:09,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:09,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:09,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8471
8413
8453
[WARNING|trainer.py:803] 2025-04-26 18:17:11,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:11,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:11,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8472
8454
8414
[WARNING|trainer.py:803] 2025-04-26 18:17:12,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:13,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:13,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8473
8455
8415
[WARNING|trainer.py:803] 2025-04-26 18:17:14,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:14,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:15,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8474
8456
8416
[WARNING|trainer.py:803] 2025-04-26 18:17:16,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:16,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:16,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8475
8457
8417
[WARNING|trainer.py:803] 2025-04-26 18:17:18,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:18,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:18,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8476
8458
8418
[WARNING|trainer.py:803] 2025-04-26 18:17:20,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:20,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:20,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8477
8459
8419
[WARNING|trainer.py:803] 2025-04-26 18:17:21,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:22,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:22,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8478
8460
8420
[WARNING|trainer.py:803] 2025-04-26 18:17:23,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:23,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:23,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8461
8479
8421
[WARNING|trainer.py:803] 2025-04-26 18:17:25,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:25,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:25,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8480
8462
8422
[WARNING|trainer.py:803] 2025-04-26 18:17:27,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:27,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:27,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8481
8463
8423
[WARNING|trainer.py:803] 2025-04-26 18:17:29,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:29,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:29,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8482
8464
8424
[WARNING|trainer.py:803] 2025-04-26 18:17:30,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:30,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:31,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8483
8465
8425
[WARNING|trainer.py:803] 2025-04-26 18:17:32,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:32,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:33,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8484
8466
8426
[WARNING|trainer.py:803] 2025-04-26 18:17:34,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:34,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:34,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8485
8467
8427
[WARNING|trainer.py:803] 2025-04-26 18:17:36,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:36,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:36,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8468
8486
8428
[WARNING|trainer.py:803] 2025-04-26 18:17:38,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:38,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:17:38,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8469
8487
8429
[WARNING|trainer.py:803] 2025-04-26 18:17:39,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:40,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:40,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8470
8488
8430
[WARNING|trainer.py:803] 2025-04-26 18:17:41,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:41,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:42,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8471
8489
[WARNING|trainer.py:803] 2025-04-26 18:17:43,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:43,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8431
[WARNING|trainer.py:803] 2025-04-26 18:17:44,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8490
8472
8432
[WARNING|trainer.py:803] 2025-04-26 18:17:45,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:45,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:45,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8473
8491
8433
[WARNING|trainer.py:803] 2025-04-26 18:17:46,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:47,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:47,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8474
8492
[WARNING|trainer.py:803] 2025-04-26 18:17:48,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8434
[WARNING|trainer.py:803] 2025-04-26 18:17:48,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:49,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8493
8475
8435
[WARNING|trainer.py:803] 2025-04-26 18:17:50,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:50,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:51,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8494
8476
8436
[WARNING|trainer.py:803] 2025-04-26 18:17:52,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:52,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:53,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8477
8495
8437
[WARNING|trainer.py:803] 2025-04-26 18:17:54,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:54,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:54,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8478
8496
8438
[WARNING|trainer.py:803] 2025-04-26 18:17:55,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:17:55,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:56,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8479
8497
8439
[WARNING|trainer.py:803] 2025-04-26 18:17:57,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:57,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:17:58,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8480
8498
8440
[WARNING|trainer.py:803] 2025-04-26 18:17:59,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:17:59,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:17:59,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8481
8499
8441
[WARNING|trainer.py:803] 2025-04-26 18:18:01,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:01,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:01,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8482
8500
8442
[WARNING|trainer.py:803] 2025-04-26 18:18:02,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:03,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:03,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8483
8501
8443
[WARNING|trainer.py:803] 2025-04-26 18:18:04,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:04,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:05,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8484
8502
8444
[WARNING|trainer.py:803] 2025-04-26 18:18:06,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:06,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:07,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8485
8503
8445
[WARNING|trainer.py:803] 2025-04-26 18:18:08,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:08,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:09,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8486
8504
8446
[WARNING|trainer.py:803] 2025-04-26 18:18:10,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:18:10,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:11,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8505
8487
8447
[WARNING|trainer.py:803] 2025-04-26 18:18:12,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:18:12,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:12,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8488
8506
8448
[WARNING|trainer.py:803] 2025-04-26 18:18:13,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:14,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:14,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8489
8507
8449
[WARNING|trainer.py:803] 2025-04-26 18:18:15,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:15,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:16,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8490
8508
[WARNING|trainer.py:803] 2025-04-26 18:18:17,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8450
[WARNING|trainer.py:803] 2025-04-26 18:18:17,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:18,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8491
8509
[WARNING|trainer.py:803] 2025-04-26 18:18:19,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8451
[WARNING|trainer.py:803] 2025-04-26 18:18:19,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:20,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8492
8510
[WARNING|trainer.py:803] 2025-04-26 18:18:20,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8452
[WARNING|trainer.py:803] 2025-04-26 18:18:21,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8493
[WARNING|trainer.py:803] 2025-04-26 18:18:21,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8511
[WARNING|trainer.py:803] 2025-04-26 18:18:22,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8453
[WARNING|trainer.py:803] 2025-04-26 18:18:23,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8494
[WARNING|trainer.py:803] 2025-04-26 18:18:23,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8512
[WARNING|trainer.py:803] 2025-04-26 18:18:24,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8454
[WARNING|trainer.py:803] 2025-04-26 18:18:24,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8495
[WARNING|trainer.py:803] 2025-04-26 18:18:25,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8513
[WARNING|trainer.py:803] 2025-04-26 18:18:26,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8455
[WARNING|trainer.py:803] 2025-04-26 18:18:26,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8496
[WARNING|trainer.py:803] 2025-04-26 18:18:27,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8514
[WARNING|trainer.py:803] 2025-04-26 18:18:28,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8456
[WARNING|trainer.py:803] 2025-04-26 18:18:28,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:28,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8497
8515
[WARNING|trainer.py:803] 2025-04-26 18:18:29,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8457
[WARNING|trainer.py:803] 2025-04-26 18:18:30,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:18:30,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8498
8516
8458
[WARNING|trainer.py:803] 2025-04-26 18:18:31,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:32,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:32,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8499
8517
8459
[WARNING|trainer.py:803] 2025-04-26 18:18:33,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:34,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:34,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8500
8518
8460
[WARNING|trainer.py:803] 2025-04-26 18:18:35,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:35,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:36,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8501
8461
8519
[WARNING|trainer.py:803] 2025-04-26 18:18:37,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:37,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:37,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8502
8462
8520
[WARNING|trainer.py:803] 2025-04-26 18:18:38,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:39,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:39,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8503
8463
[WARNING|trainer.py:803] 2025-04-26 18:18:40,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8521
[WARNING|trainer.py:803] 2025-04-26 18:18:41,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:41,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8504
8464
[WARNING|trainer.py:803] 2025-04-26 18:18:42,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8522
[WARNING|trainer.py:803] 2025-04-26 18:18:43,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:43,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8505
8465
8523
[WARNING|trainer.py:803] 2025-04-26 18:18:44,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:18:45,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:45,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8506
8466
8524
[WARNING|trainer.py:803] 2025-04-26 18:18:46,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:46,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:47,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8507
8467
[WARNING|trainer.py:803] 2025-04-26 18:18:48,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8525
[WARNING|trainer.py:803] 2025-04-26 18:18:48,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:18:49,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8508
8468
8526
[WARNING|trainer.py:803] 2025-04-26 18:18:50,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:50,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:50,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8509
8469
8527
[WARNING|trainer.py:803] 2025-04-26 18:18:51,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:51,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:52,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8510
8470
8528
[WARNING|trainer.py:803] 2025-04-26 18:18:53,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:53,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:54,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8511
8471
8529
[WARNING|trainer.py:803] 2025-04-26 18:18:55,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:18:55,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:56,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8512
8472
8530
[WARNING|trainer.py:803] 2025-04-26 18:18:57,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:57,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:57,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8513
8473
8531
[WARNING|trainer.py:803] 2025-04-26 18:18:59,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:18:59,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:18:59,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8474
8514
8532
[WARNING|trainer.py:803] 2025-04-26 18:19:01,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:01,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:01,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8515
8475
8533
[WARNING|trainer.py:803] 2025-04-26 18:19:02,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:19:02,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:03,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8516
8476
8534
[WARNING|trainer.py:803] 2025-04-26 18:19:04,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:04,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:05,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8517
8477
8535
[WARNING|trainer.py:803] 2025-04-26 18:19:06,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:06,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:07,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8478
8518
8536
[WARNING|trainer.py:803] 2025-04-26 18:19:08,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:08,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:08,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8479
8519
8537
[WARNING|trainer.py:803] 2025-04-26 18:19:10,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:10,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:10,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8480
8520
8538
[WARNING|trainer.py:803] 2025-04-26 18:19:11,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:12,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:12,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8481
8521
8539
[WARNING|trainer.py:803] 2025-04-26 18:19:13,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:13,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:14,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8482
8522
8540
[WARNING|trainer.py:803] 2025-04-26 18:19:15,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:15,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:15,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8483
8523
8541
[WARNING|trainer.py:803] 2025-04-26 18:19:17,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:17,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:17,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8484
8524
8542
[WARNING|trainer.py:803] 2025-04-26 18:19:19,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:19,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:19,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8485
8543
8525
[WARNING|trainer.py:803] 2025-04-26 18:19:21,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:21,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:21,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8486
8544
8526
[WARNING|trainer.py:803] 2025-04-26 18:19:22,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:19:23,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:23,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8487
8527
8545
[WARNING|trainer.py:803] 2025-04-26 18:19:24,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:25,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:25,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8488
8528
8546
[WARNING|trainer.py:803] 2025-04-26 18:19:26,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:26,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:26,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8489
8529
8547
[WARNING|trainer.py:803] 2025-04-26 18:19:28,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:28,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:28,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8490
8530
8548
[WARNING|trainer.py:803] 2025-04-26 18:19:30,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:30,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:30,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8491
8531
8549
[WARNING|trainer.py:803] 2025-04-26 18:19:31,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:32,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:32,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8492
8532
8550
[WARNING|trainer.py:803] 2025-04-26 18:19:33,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:34,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:34,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8493
8551
8533
[WARNING|trainer.py:803] 2025-04-26 18:19:35,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:36,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:36,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8494
[WARNING|trainer.py:803] 2025-04-26 18:19:37,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8552
8534
[WARNING|trainer.py:803] 2025-04-26 18:19:37,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:37,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8495
[WARNING|trainer.py:803] 2025-04-26 18:19:38,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8553
8535
8496
[WARNING|trainer.py:803] 2025-04-26 18:19:39,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:39,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:40,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8536
8554
8497
[WARNING|trainer.py:803] 2025-04-26 18:19:41,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:41,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:42,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8537
8555
8498
[WARNING|trainer.py:803] 2025-04-26 18:19:43,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:43,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:44,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8538
8556
8499
[WARNING|trainer.py:803] 2025-04-26 18:19:45,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:45,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:45,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8557
8539
8500
[WARNING|trainer.py:803] 2025-04-26 18:19:47,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:47,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:47,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8540
8558
8501
[WARNING|trainer.py:803] 2025-04-26 18:19:48,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:48,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:49,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8559
8541
8502
[WARNING|trainer.py:803] 2025-04-26 18:19:50,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:19:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:51,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8560
8542
8503
[WARNING|trainer.py:803] 2025-04-26 18:19:52,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:52,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:53,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8561
8543
8504
[WARNING|trainer.py:803] 2025-04-26 18:19:54,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:54,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:55,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8562
8544
8505
[WARNING|trainer.py:803] 2025-04-26 18:19:56,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:19:56,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:56,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8563
8545
[WARNING|trainer.py:803] 2025-04-26 18:19:57,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8506
[WARNING|trainer.py:803] 2025-04-26 18:19:58,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:19:58,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8564
8546
8507
[WARNING|trainer.py:803] 2025-04-26 18:19:59,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:19:59,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:20:00,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8565
8547
8508
[WARNING|trainer.py:803] 2025-04-26 18:20:01,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:01,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:02,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8566
8548
8509
[WARNING|trainer.py:803] 2025-04-26 18:20:03,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:03,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:04,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8567
8549
8510
[WARNING|trainer.py:803] 2025-04-26 18:20:05,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:05,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:06,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8568
8550
8511
[WARNING|trainer.py:803] 2025-04-26 18:20:07,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:07,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:07,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8569
8551
8512
[WARNING|trainer.py:803] 2025-04-26 18:20:08,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:09,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8570
[WARNING|trainer.py:803] 2025-04-26 18:20:09,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8552
[WARNING|trainer.py:803] 2025-04-26 18:20:10,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8513
[WARNING|trainer.py:803] 2025-04-26 18:20:11,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8571
[WARNING|trainer.py:803] 2025-04-26 18:20:11,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:12,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8553
8514
[WARNING|trainer.py:803] 2025-04-26 18:20:13,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8572
[WARNING|trainer.py:803] 2025-04-26 18:20:13,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:13,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8554
8515
[WARNING|trainer.py:803] 2025-04-26 18:20:14,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8573
[WARNING|trainer.py:803] 2025-04-26 18:20:15,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:20:15,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8555
8516
8574
[WARNING|trainer.py:803] 2025-04-26 18:20:16,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:16,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:17,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8556
8517
8575
[WARNING|trainer.py:803] 2025-04-26 18:20:18,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:18,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:19,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8557
8518
8576
[WARNING|trainer.py:803] 2025-04-26 18:20:20,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:20,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:20,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8558
8519
8577
[WARNING|trainer.py:803] 2025-04-26 18:20:22,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:22,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:22,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8559
8520
8578
[WARNING|trainer.py:803] 2025-04-26 18:20:24,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:24,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:24,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8560
8521
8579
[WARNING|trainer.py:803] 2025-04-26 18:20:25,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:26,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:26,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8561
8580
8522
[WARNING|trainer.py:803] 2025-04-26 18:20:27,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:28,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:28,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8562
8523
8581
[WARNING|trainer.py:803] 2025-04-26 18:20:29,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:29,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:30,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8563
8524
8582
[WARNING|trainer.py:803] 2025-04-26 18:20:31,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:31,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:32,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8564
8525
8583
[WARNING|trainer.py:803] 2025-04-26 18:20:33,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:33,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:34,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8565
8526
8584
[WARNING|trainer.py:803] 2025-04-26 18:20:35,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:35,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:35,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8566
8527
8585
[WARNING|trainer.py:803] 2025-04-26 18:20:36,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:37,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:37,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8567
8528
8586
[WARNING|trainer.py:803] 2025-04-26 18:20:38,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:39,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:39,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8568
8529
8587
[WARNING|trainer.py:803] 2025-04-26 18:20:40,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:41,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:41,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8569
8530
8588
[WARNING|trainer.py:803] 2025-04-26 18:20:42,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:42,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:43,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8570
8531
8589
[WARNING|trainer.py:803] 2025-04-26 18:20:44,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:44,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:44,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8571
8590
8532
[WARNING|trainer.py:803] 2025-04-26 18:20:45,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:20:46,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:20:46,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8572
8591
[WARNING|trainer.py:803] 2025-04-26 18:20:47,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8533
[WARNING|trainer.py:803] 2025-04-26 18:20:48,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:20:48,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8573
8592
[WARNING|trainer.py:803] 2025-04-26 18:20:49,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8534
[WARNING|trainer.py:803] 2025-04-26 18:20:50,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8574
[WARNING|trainer.py:803] 2025-04-26 18:20:50,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8593
[WARNING|trainer.py:803] 2025-04-26 18:20:51,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8535
[WARNING|trainer.py:803] 2025-04-26 18:20:51,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8575
[WARNING|trainer.py:803] 2025-04-26 18:20:52,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8594
[WARNING|trainer.py:803] 2025-04-26 18:20:52,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8536
[WARNING|trainer.py:803] 2025-04-26 18:20:53,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8576
[WARNING|trainer.py:803] 2025-04-26 18:20:54,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8595
[WARNING|trainer.py:803] 2025-04-26 18:20:54,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8537
[WARNING|trainer.py:803] 2025-04-26 18:20:55,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8577
[WARNING|trainer.py:803] 2025-04-26 18:20:55,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8596
[WARNING|trainer.py:803] 2025-04-26 18:20:56,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8538
[WARNING|trainer.py:803] 2025-04-26 18:20:57,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8578
[WARNING|trainer.py:803] 2025-04-26 18:20:57,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:20:58,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8597
8539
8579
[WARNING|trainer.py:803] 2025-04-26 18:20:59,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:20:59,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:00,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8598
8540
[WARNING|trainer.py:803] 2025-04-26 18:21:00,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8580
[WARNING|trainer.py:803] 2025-04-26 18:21:01,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:01,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8599
8541
[WARNING|trainer.py:803] 2025-04-26 18:21:02,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8581
[WARNING|trainer.py:803] 2025-04-26 18:21:03,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8600
[WARNING|trainer.py:803] 2025-04-26 18:21:03,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8542
[WARNING|trainer.py:803] 2025-04-26 18:21:04,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8582
[WARNING|trainer.py:803] 2025-04-26 18:21:05,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8601
[WARNING|trainer.py:803] 2025-04-26 18:21:05,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8543
[WARNING|trainer.py:803] 2025-04-26 18:21:06,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8583
[WARNING|trainer.py:803] 2025-04-26 18:21:06,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8602
[WARNING|trainer.py:803] 2025-04-26 18:21:07,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:07,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8544
8584
8603
[WARNING|trainer.py:803] 2025-04-26 18:21:08,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:09,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:09,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8545
8585
8604
[WARNING|trainer.py:803] 2025-04-26 18:21:10,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:11,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:21:11,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8546
8605
8586
[WARNING|trainer.py:803] 2025-04-26 18:21:12,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:21:13,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:13,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8547
8606
8587
[WARNING|trainer.py:803] 2025-04-26 18:21:14,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:14,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:21:14,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8548
8607
8588
[WARNING|trainer.py:803] 2025-04-26 18:21:16,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:16,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:16,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8549
8608
8589
[WARNING|trainer.py:803] 2025-04-26 18:21:18,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:18,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:18,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8550
8609
8590
[WARNING|trainer.py:803] 2025-04-26 18:21:20,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:20,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:20,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8551
8610
8591
[WARNING|trainer.py:803] 2025-04-26 18:21:21,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:21,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:22,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8552
8611
8592
[WARNING|trainer.py:803] 2025-04-26 18:21:23,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:23,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:23,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8612
8553
8593
[WARNING|trainer.py:803] 2025-04-26 18:21:25,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:25,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:21:25,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8613
8554
8594
[WARNING|trainer.py:803] 2025-04-26 18:21:27,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:27,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:27,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8614
8595
8555
[WARNING|trainer.py:803] 2025-04-26 18:21:29,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:29,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:29,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8615
8556
8596
[WARNING|trainer.py:803] 2025-04-26 18:21:30,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:31,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:31,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8616
8557
8597
[WARNING|trainer.py:803] 2025-04-26 18:21:32,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:33,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:33,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8617
8598
8558
[WARNING|trainer.py:803] 2025-04-26 18:21:34,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:34,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:35,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8618
8599
8559
[WARNING|trainer.py:803] 2025-04-26 18:21:36,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:36,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:36,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8619
8600
8560
[WARNING|trainer.py:803] 2025-04-26 18:21:37,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:38,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:38,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8620
8601
[WARNING|trainer.py:803] 2025-04-26 18:21:39,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8561
[WARNING|trainer.py:803] 2025-04-26 18:21:40,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:40,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8621
8602
[WARNING|trainer.py:803] 2025-04-26 18:21:41,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8562
[WARNING|trainer.py:803] 2025-04-26 18:21:42,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8622
[WARNING|trainer.py:803] 2025-04-26 18:21:42,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8603
[WARNING|trainer.py:803] 2025-04-26 18:21:43,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8563
[WARNING|trainer.py:803] 2025-04-26 18:21:43,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8623
[WARNING|trainer.py:803] 2025-04-26 18:21:44,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8604
[WARNING|trainer.py:803] 2025-04-26 18:21:44,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8564
[WARNING|trainer.py:803] 2025-04-26 18:21:45,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8624
[WARNING|trainer.py:803] 2025-04-26 18:21:46,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8605
[WARNING|trainer.py:803] 2025-04-26 18:21:46,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8565
[WARNING|trainer.py:803] 2025-04-26 18:21:47,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8625
[WARNING|trainer.py:803] 2025-04-26 18:21:47,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8606
[WARNING|trainer.py:803] 2025-04-26 18:21:48,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8566
[WARNING|trainer.py:803] 2025-04-26 18:21:48,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8626
[WARNING|trainer.py:803] 2025-04-26 18:21:49,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8607
[WARNING|trainer.py:803] 2025-04-26 18:21:50,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8567
[WARNING|trainer.py:803] 2025-04-26 18:21:50,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8627
[WARNING|trainer.py:803] 2025-04-26 18:21:51,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8608
[WARNING|trainer.py:803] 2025-04-26 18:21:51,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8568
[WARNING|trainer.py:803] 2025-04-26 18:21:52,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8628
[WARNING|trainer.py:803] 2025-04-26 18:21:53,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:53,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8609
8569
8629
[WARNING|trainer.py:803] 2025-04-26 18:21:54,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:55,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:55,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8610
8570
8630
[WARNING|trainer.py:803] 2025-04-26 18:21:56,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:21:56,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:21:56,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8611
8631
8571
[WARNING|trainer.py:803] 2025-04-26 18:21:57,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:21:58,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:21:58,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8612
8632
8572
[WARNING|trainer.py:803] 2025-04-26 18:21:59,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:00,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:00,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8613
8633
8573
[WARNING|trainer.py:803] 2025-04-26 18:22:01,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:02,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:02,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8614
8634
8574
[WARNING|trainer.py:803] 2025-04-26 18:22:03,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:03,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:03,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8615
8635
8575
[WARNING|trainer.py:803] 2025-04-26 18:22:05,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:05,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:05,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8616
8636
8576
[WARNING|trainer.py:803] 2025-04-26 18:22:06,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:07,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:07,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8617
8637
8577
[WARNING|trainer.py:803] 2025-04-26 18:22:08,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:09,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:09,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8618
8578
8638
[WARNING|trainer.py:803] 2025-04-26 18:22:10,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:11,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:11,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8619
8639
8579
[WARNING|trainer.py:803] 2025-04-26 18:22:12,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:12,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:12,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8620
8640
8580
[WARNING|trainer.py:803] 2025-04-26 18:22:13,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:14,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:14,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8621
8641
[WARNING|trainer.py:803] 2025-04-26 18:22:15,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8581
[WARNING|trainer.py:803] 2025-04-26 18:22:16,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8622
[WARNING|trainer.py:803] 2025-04-26 18:22:16,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8642
[WARNING|trainer.py:803] 2025-04-26 18:22:17,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8582
[WARNING|trainer.py:803] 2025-04-26 18:22:17,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8623
[WARNING|trainer.py:803] 2025-04-26 18:22:18,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8643
[WARNING|trainer.py:803] 2025-04-26 18:22:19,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8583
[WARNING|trainer.py:803] 2025-04-26 18:22:19,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8624
[WARNING|trainer.py:803] 2025-04-26 18:22:20,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8644
[WARNING|trainer.py:803] 2025-04-26 18:22:21,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:21,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8584
8625
[WARNING|trainer.py:803] 2025-04-26 18:22:22,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8645
[WARNING|trainer.py:803] 2025-04-26 18:22:22,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:23,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8585
8626
8646
[WARNING|trainer.py:803] 2025-04-26 18:22:24,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:24,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:25,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8586
8627
8647
[WARNING|trainer.py:803] 2025-04-26 18:22:26,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:26,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:26,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8587
8628
8648
[WARNING|trainer.py:803] 2025-04-26 18:22:27,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:28,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:28,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8629
8588
8649
[WARNING|trainer.py:803] 2025-04-26 18:22:29,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:29,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:30,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8630
8589
8650
[WARNING|trainer.py:803] 2025-04-26 18:22:31,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:31,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:31,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8631
8590
8651
[WARNING|trainer.py:803] 2025-04-26 18:22:33,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:33,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:33,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8632
8591
8652
[WARNING|trainer.py:803] 2025-04-26 18:22:34,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:35,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:22:35,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8633
8592
8653
[WARNING|trainer.py:803] 2025-04-26 18:22:36,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:36,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:22:37,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8634
8593
8654
[WARNING|trainer.py:803] 2025-04-26 18:22:38,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:38,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:22:38,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8635
8655
8594
[WARNING|trainer.py:803] 2025-04-26 18:22:40,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:40,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:40,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8636
8656
8595
[WARNING|trainer.py:803] 2025-04-26 18:22:41,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:42,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:42,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8637
8657
8596
[WARNING|trainer.py:803] 2025-04-26 18:22:43,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:44,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:22:44,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8638
8658
[WARNING|trainer.py:803] 2025-04-26 18:22:45,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8597
[WARNING|trainer.py:803] 2025-04-26 18:22:45,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8639
[WARNING|trainer.py:803] 2025-04-26 18:22:46,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8659
[WARNING|trainer.py:803] 2025-04-26 18:22:47,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8598
[WARNING|trainer.py:803] 2025-04-26 18:22:47,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8640
[WARNING|trainer.py:803] 2025-04-26 18:22:48,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8660
[WARNING|trainer.py:803] 2025-04-26 18:22:48,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8599
[WARNING|trainer.py:803] 2025-04-26 18:22:49,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8641
[WARNING|trainer.py:803] 2025-04-26 18:22:49,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8661
[WARNING|trainer.py:803] 2025-04-26 18:22:50,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8600
[WARNING|trainer.py:803] 2025-04-26 18:22:50,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8642
[WARNING|trainer.py:803] 2025-04-26 18:22:51,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8662
[WARNING|trainer.py:803] 2025-04-26 18:22:52,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8601
[WARNING|trainer.py:803] 2025-04-26 18:22:52,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8643
[WARNING|trainer.py:803] 2025-04-26 18:22:53,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8663
[WARNING|trainer.py:803] 2025-04-26 18:22:54,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:54,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8602
8644
[WARNING|trainer.py:803] 2025-04-26 18:22:55,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8664
[WARNING|trainer.py:803] 2025-04-26 18:22:55,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:22:55,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8603
8645
[WARNING|trainer.py:803] 2025-04-26 18:22:56,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8665
[WARNING|trainer.py:803] 2025-04-26 18:22:57,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8604
[WARNING|trainer.py:803] 2025-04-26 18:22:57,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:22:58,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8666
8646
[WARNING|trainer.py:803] 2025-04-26 18:22:59,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8605
[WARNING|trainer.py:803] 2025-04-26 18:22:59,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:00,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8667
8647
8606
[WARNING|trainer.py:803] 2025-04-26 18:23:01,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:01,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:01,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8648
8668
[WARNING|trainer.py:803] 2025-04-26 18:23:02,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:23:02,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8607
[WARNING|trainer.py:803] 2025-04-26 18:23:03,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8649
8669
[WARNING|trainer.py:803] 2025-04-26 18:23:04,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:04,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8608
8650
[WARNING|trainer.py:803] 2025-04-26 18:23:05,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8670
[WARNING|trainer.py:803] 2025-04-26 18:23:06,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:06,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8609
8651
[WARNING|trainer.py:803] 2025-04-26 18:23:07,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8671
[WARNING|trainer.py:803] 2025-04-26 18:23:08,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:08,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
8610
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:09,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8652
8672
[WARNING|trainer.py:803] 2025-04-26 18:23:10,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:10,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8611
8653
[WARNING|trainer.py:803] 2025-04-26 18:23:10,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8673
[WARNING|trainer.py:803] 2025-04-26 18:23:11,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:11,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8612
8654
[WARNING|trainer.py:803] 2025-04-26 18:23:12,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8674
[WARNING|trainer.py:803] 2025-04-26 18:23:13,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8613
[WARNING|trainer.py:803] 2025-04-26 18:23:13,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8655
[WARNING|trainer.py:803] 2025-04-26 18:23:14,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8675
[WARNING|trainer.py:803] 2025-04-26 18:23:15,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:15,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8614
8656
8676
[WARNING|trainer.py:803] 2025-04-26 18:23:16,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:17,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:17,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8615
8657
8677
[WARNING|trainer.py:803] 2025-04-26 18:23:18,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:18,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:18,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8616
8658
8678
[WARNING|trainer.py:803] 2025-04-26 18:23:19,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:20,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:20,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8617
8659
8679
[WARNING|trainer.py:803] 2025-04-26 18:23:21,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:22,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:22,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8618
8660
8680
[WARNING|trainer.py:803] 2025-04-26 18:23:23,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:23,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:24,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8619
8661
8681
[WARNING|trainer.py:803] 2025-04-26 18:23:25,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:25,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:25,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8620
8662
[WARNING|trainer.py:803] 2025-04-26 18:23:26,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8682
[WARNING|trainer.py:803] 2025-04-26 18:23:27,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:23:27,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8621
8663
[WARNING|trainer.py:803] 2025-04-26 18:23:28,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8683
[WARNING|trainer.py:803] 2025-04-26 18:23:29,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:23:29,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8622
8664
8684
[WARNING|trainer.py:803] 2025-04-26 18:23:30,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:30,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:23:31,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8623
8665
[WARNING|trainer.py:803] 2025-04-26 18:23:32,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8685
[WARNING|trainer.py:803] 2025-04-26 18:23:32,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:23:33,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8624
8666
[WARNING|trainer.py:803] 2025-04-26 18:23:33,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8686
[WARNING|trainer.py:803] 2025-04-26 18:23:34,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8625
[WARNING|trainer.py:803] 2025-04-26 18:23:34,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8667
[WARNING|trainer.py:803] 2025-04-26 18:23:35,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8687
[WARNING|trainer.py:803] 2025-04-26 18:23:36,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8626
[WARNING|trainer.py:803] 2025-04-26 18:23:36,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8668
[WARNING|trainer.py:803] 2025-04-26 18:23:37,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8688
[WARNING|trainer.py:803] 2025-04-26 18:23:37,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:23:38,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8627
8669
[WARNING|trainer.py:803] 2025-04-26 18:23:39,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8689
[WARNING|trainer.py:803] 2025-04-26 18:23:39,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8628
[WARNING|trainer.py:803] 2025-04-26 18:23:40,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8670
[WARNING|trainer.py:803] 2025-04-26 18:23:40,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8690
[WARNING|trainer.py:803] 2025-04-26 18:23:41,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8629
[WARNING|trainer.py:803] 2025-04-26 18:23:41,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8671
[WARNING|trainer.py:803] 2025-04-26 18:23:42,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8691
[WARNING|trainer.py:803] 2025-04-26 18:23:43,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8630
[WARNING|trainer.py:803] 2025-04-26 18:23:43,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8672
[WARNING|trainer.py:803] 2025-04-26 18:23:44,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8692
[WARNING|trainer.py:803] 2025-04-26 18:23:44,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8631
[WARNING|trainer.py:803] 2025-04-26 18:23:45,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8673
[WARNING|trainer.py:803] 2025-04-26 18:23:46,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8693
[WARNING|trainer.py:803] 2025-04-26 18:23:46,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8632
[WARNING|trainer.py:803] 2025-04-26 18:23:47,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8674
[WARNING|trainer.py:803] 2025-04-26 18:23:47,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8694
[WARNING|trainer.py:803] 2025-04-26 18:23:48,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8633
[WARNING|trainer.py:803] 2025-04-26 18:23:48,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:23:49,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8675
8695
[WARNING|trainer.py:803] 2025-04-26 18:23:50,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8634
[WARNING|trainer.py:803] 2025-04-26 18:23:50,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8676
[WARNING|trainer.py:803] 2025-04-26 18:23:51,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8696
[WARNING|trainer.py:803] 2025-04-26 18:23:51,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8635
[WARNING|trainer.py:803] 2025-04-26 18:23:52,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8677
[WARNING|trainer.py:803] 2025-04-26 18:23:52,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8697
[WARNING|trainer.py:803] 2025-04-26 18:23:53,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8636
[WARNING|trainer.py:803] 2025-04-26 18:23:54,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8678
[WARNING|trainer.py:803] 2025-04-26 18:23:54,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8698
[WARNING|trainer.py:803] 2025-04-26 18:23:55,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8637
[WARNING|trainer.py:803] 2025-04-26 18:23:56,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8679
[WARNING|trainer.py:803] 2025-04-26 18:23:56,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8699
[WARNING|trainer.py:803] 2025-04-26 18:23:57,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8638
[WARNING|trainer.py:803] 2025-04-26 18:23:57,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8680
[WARNING|trainer.py:803] 2025-04-26 18:23:58,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8700
[WARNING|trainer.py:803] 2025-04-26 18:23:58,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8639
[WARNING|trainer.py:803] 2025-04-26 18:23:59,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8681
[WARNING|trainer.py:803] 2025-04-26 18:23:59,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8701
[WARNING|trainer.py:803] 2025-04-26 18:24:00,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8640
[WARNING|trainer.py:803] 2025-04-26 18:24:00,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8702
8682
[WARNING|trainer.py:803] 2025-04-26 18:24:01,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:24:02,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:02,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8641
8703
[WARNING|trainer.py:803] 2025-04-26 18:24:03,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8683
[WARNING|trainer.py:803] 2025-04-26 18:24:03,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8704
8642
[WARNING|trainer.py:803] 2025-04-26 18:24:04,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:04,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:04,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8684
8705
[WARNING|trainer.py:803] 2025-04-26 18:24:05,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8643
[WARNING|trainer.py:803] 2025-04-26 18:24:05,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8706
[WARNING|trainer.py:803] 2025-04-26 18:24:06,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8685
[WARNING|trainer.py:803] 2025-04-26 18:24:07,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8644
[WARNING|trainer.py:803] 2025-04-26 18:24:07,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8707
[WARNING|trainer.py:803] 2025-04-26 18:24:08,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:08,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8686
8708
8645
[WARNING|trainer.py:803] 2025-04-26 18:24:09,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:09,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:10,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8709
8687
8646
[WARNING|trainer.py:803] 2025-04-26 18:24:11,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:11,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8710
[WARNING|trainer.py:803] 2025-04-26 18:24:11,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8688
[WARNING|trainer.py:803] 2025-04-26 18:24:12,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:12,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8647
8711
[WARNING|trainer.py:803] 2025-04-26 18:24:13,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:13,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8689
8712
[WARNING|trainer.py:803] 2025-04-26 18:24:14,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8648
[WARNING|trainer.py:803] 2025-04-26 18:24:15,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:15,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8713
8690
[WARNING|trainer.py:803] 2025-04-26 18:24:16,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8649
[WARNING|trainer.py:803] 2025-04-26 18:24:16,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8714
[WARNING|trainer.py:803] 2025-04-26 18:24:17,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8691
[WARNING|trainer.py:803] 2025-04-26 18:24:17,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8650
[WARNING|trainer.py:803] 2025-04-26 18:24:18,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8715
[WARNING|trainer.py:803] 2025-04-26 18:24:19,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:19,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8692
8716
8651
[WARNING|trainer.py:803] 2025-04-26 18:24:19,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:20,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:20,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8717
8693
[WARNING|trainer.py:803] 2025-04-26 18:24:21,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8652
[WARNING|trainer.py:803] 2025-04-26 18:24:21,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8718
[WARNING|trainer.py:803] 2025-04-26 18:24:22,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8694
[WARNING|trainer.py:803] 2025-04-26 18:24:22,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8653
8719
[WARNING|trainer.py:803] 2025-04-26 18:24:23,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:24,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:24,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8695
8720
8654
[WARNING|trainer.py:803] 2025-04-26 18:24:25,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:25,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:25,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8721
8696
[WARNING|trainer.py:803] 2025-04-26 18:24:26,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8655
[WARNING|trainer.py:803] 2025-04-26 18:24:26,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8722
[WARNING|trainer.py:803] 2025-04-26 18:24:27,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8697
[WARNING|trainer.py:803] 2025-04-26 18:24:28,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8656
8723
[WARNING|trainer.py:803] 2025-04-26 18:24:28,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:29,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:24:29,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8698
8724
8657
[WARNING|trainer.py:803] 2025-04-26 18:24:30,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:30,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:31,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8699
8725
8658
[WARNING|trainer.py:803] 2025-04-26 18:24:32,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:32,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8726
[WARNING|trainer.py:803] 2025-04-26 18:24:32,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8700
[WARNING|trainer.py:803] 2025-04-26 18:24:33,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8659
[WARNING|trainer.py:803] 2025-04-26 18:24:33,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8727
8701
[WARNING|trainer.py:803] 2025-04-26 18:24:34,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:24:34,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:35,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8728
8660
8702
[WARNING|trainer.py:803] 2025-04-26 18:24:36,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:36,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:24:36,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8729
8703
8661
[WARNING|trainer.py:803] 2025-04-26 18:24:37,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:37,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8730
[WARNING|trainer.py:803] 2025-04-26 18:24:38,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8704
[WARNING|trainer.py:803] 2025-04-26 18:24:38,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8662
[WARNING|trainer.py:803] 2025-04-26 18:24:39,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8731
8705
[WARNING|trainer.py:803] 2025-04-26 18:24:39,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:40,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:40,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8732
8663
8706
[WARNING|trainer.py:803] 2025-04-26 18:24:41,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:41,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:41,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8733
8707
8664
[WARNING|trainer.py:803] 2025-04-26 18:24:42,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:42,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8734
[WARNING|trainer.py:803] 2025-04-26 18:24:43,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8708
[WARNING|trainer.py:803] 2025-04-26 18:24:44,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8665
[WARNING|trainer.py:803] 2025-04-26 18:24:44,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8735
8709
[WARNING|trainer.py:803] 2025-04-26 18:24:44,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:45,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:45,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8736
8666
8710
[WARNING|trainer.py:803] 2025-04-26 18:24:46,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:46,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:46,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8737
8711
8667
[WARNING|trainer.py:803] 2025-04-26 18:24:47,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:48,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8738
[WARNING|trainer.py:803] 2025-04-26 18:24:48,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8712
[WARNING|trainer.py:803] 2025-04-26 18:24:49,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:49,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8668
8739
8713
[WARNING|trainer.py:803] 2025-04-26 18:24:50,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:50,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:50,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8740
8714
8669
[WARNING|trainer.py:803] 2025-04-26 18:24:51,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:51,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:52,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8741
8715
8670
[WARNING|trainer.py:803] 2025-04-26 18:24:53,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:53,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8742
8716
[WARNING|trainer.py:803] 2025-04-26 18:24:53,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:54,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:54,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8671
8743
8717
[WARNING|trainer.py:803] 2025-04-26 18:24:55,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:24:55,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:55,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8744
8718
8672
[WARNING|trainer.py:803] 2025-04-26 18:24:57,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:57,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:24:57,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8745
8719
8673
[WARNING|trainer.py:803] 2025-04-26 18:24:58,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:58,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8720
8746
[WARNING|trainer.py:803] 2025-04-26 18:24:59,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:24:59,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:24:59,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8674
8721
8747
[WARNING|trainer.py:803] 2025-04-26 18:25:00,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:01,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:01,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8722
8748
8675
[WARNING|trainer.py:803] 2025-04-26 18:25:02,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:02,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:02,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8723
8749
[WARNING|trainer.py:803] 2025-04-26 18:25:03,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8676
[WARNING|trainer.py:803] 2025-04-26 18:25:03,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8750
8724
[WARNING|trainer.py:803] 2025-04-26 18:25:04,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:05,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:05,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8677
8725
8751
[WARNING|trainer.py:803] 2025-04-26 18:25:06,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:06,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:06,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8726
8752
8678
[WARNING|trainer.py:803] 2025-04-26 18:25:07,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:07,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:08,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8727
8753
8679
[WARNING|trainer.py:803] 2025-04-26 18:25:09,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:09,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8728
8754
[WARNING|trainer.py:803] 2025-04-26 18:25:09,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:10,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:10,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8680
8729
8755
[WARNING|trainer.py:803] 2025-04-26 18:25:11,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:25:11,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:11,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8730
8756
8681
[WARNING|trainer.py:803] 2025-04-26 18:25:12,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:13,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:13,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8731
8757
8682
[WARNING|trainer.py:803] 2025-04-26 18:25:14,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:14,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8732
8758
[WARNING|trainer.py:803] 2025-04-26 18:25:15,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:15,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:15,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8683
8733
8759
[WARNING|trainer.py:803] 2025-04-26 18:25:16,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:16,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:16,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8734
8760
8684
[WARNING|trainer.py:803] 2025-04-26 18:25:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:18,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:18,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8735
8761
[WARNING|trainer.py:803] 2025-04-26 18:25:19,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8685
[WARNING|trainer.py:803] 2025-04-26 18:25:19,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8736
8762
[WARNING|trainer.py:803] 2025-04-26 18:25:20,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:25:20,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:20,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8737
8686
8763
[WARNING|trainer.py:803] 2025-04-26 18:25:22,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:22,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:25:22,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8738
8764
8687
[WARNING|trainer.py:803] 2025-04-26 18:25:23,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:23,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8739
[WARNING|trainer.py:803] 2025-04-26 18:25:24,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8765
[WARNING|trainer.py:803] 2025-04-26 18:25:24,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8688
[WARNING|trainer.py:803] 2025-04-26 18:25:24,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8740
8766
[WARNING|trainer.py:803] 2025-04-26 18:25:25,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:26,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:26,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8689
8741
8767
[WARNING|trainer.py:803] 2025-04-26 18:25:27,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:27,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:27,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8742
8768
8690
[WARNING|trainer.py:803] 2025-04-26 18:25:28,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:28,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:29,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8743
8769
8691
[WARNING|trainer.py:803] 2025-04-26 18:25:30,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:30,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8744
8770
[WARNING|trainer.py:803] 2025-04-26 18:25:30,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:31,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:31,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8692
8745
8771
[WARNING|trainer.py:803] 2025-04-26 18:25:32,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:32,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:32,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8772
8746
8693
[WARNING|trainer.py:803] 2025-04-26 18:25:34,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:34,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:34,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8773
8747
8694
[WARNING|trainer.py:803] 2025-04-26 18:25:35,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:35,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8774
8748
[WARNING|trainer.py:803] 2025-04-26 18:25:36,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:36,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:36,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8775
8695
8749
[WARNING|trainer.py:803] 2025-04-26 18:25:37,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:38,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:38,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8776
8750
8696
[WARNING|trainer.py:803] 2025-04-26 18:25:39,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:39,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8777
[WARNING|trainer.py:803] 2025-04-26 18:25:39,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8751
[WARNING|trainer.py:803] 2025-04-26 18:25:40,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8697
[WARNING|trainer.py:803] 2025-04-26 18:25:40,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8778
8752
[WARNING|trainer.py:803] 2025-04-26 18:25:41,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:41,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:42,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8779
8698
8753
[WARNING|trainer.py:803] 2025-04-26 18:25:43,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:43,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:43,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8780
8754
8699
[WARNING|trainer.py:803] 2025-04-26 18:25:44,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:44,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8781
[WARNING|trainer.py:803] 2025-04-26 18:25:45,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8755
[WARNING|trainer.py:803] 2025-04-26 18:25:45,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:46,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8700
8782
8756
[WARNING|trainer.py:803] 2025-04-26 18:25:47,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:47,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8701
[WARNING|trainer.py:803] 2025-04-26 18:25:47,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8783
8757
[WARNING|trainer.py:803] 2025-04-26 18:25:48,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:48,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:48,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8702
8784
8758
[WARNING|trainer.py:803] 2025-04-26 18:25:49,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:49,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:50,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8703
8785
8759
[WARNING|trainer.py:803] 2025-04-26 18:25:51,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:51,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:51,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8704
8786
8760
[WARNING|trainer.py:803] 2025-04-26 18:25:52,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:52,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:25:52,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8705
8787
8761
[WARNING|trainer.py:803] 2025-04-26 18:25:53,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:53,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:25:54,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8706
8788
8762
[WARNING|trainer.py:803] 2025-04-26 18:25:55,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:55,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:55,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8707
8789
8763
[WARNING|trainer.py:803] 2025-04-26 18:25:56,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:56,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:56,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8708
8790
8764
[WARNING|trainer.py:803] 2025-04-26 18:25:57,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:57,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:58,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8709
8791
8765
[WARNING|trainer.py:803] 2025-04-26 18:25:58,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:25:59,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8710
[WARNING|trainer.py:803] 2025-04-26 18:25:59,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8792
8766
[WARNING|trainer.py:803] 2025-04-26 18:26:00,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:00,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8711
[WARNING|trainer.py:803] 2025-04-26 18:26:00,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8793
8767
[WARNING|trainer.py:803] 2025-04-26 18:26:01,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:01,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8712
[WARNING|trainer.py:803] 2025-04-26 18:26:02,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8794
8768
[WARNING|trainer.py:803] 2025-04-26 18:26:02,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:03,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8713
[WARNING|trainer.py:803] 2025-04-26 18:26:03,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8795
8769
[WARNING|trainer.py:803] 2025-04-26 18:26:04,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8714
[WARNING|trainer.py:803] 2025-04-26 18:26:04,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:04,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8796
8770
[WARNING|trainer.py:803] 2025-04-26 18:26:05,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8715
[WARNING|trainer.py:803] 2025-04-26 18:26:05,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:26:06,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8797
8771
[WARNING|trainer.py:803] 2025-04-26 18:26:06,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8716
[WARNING|trainer.py:803] 2025-04-26 18:26:07,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:26:07,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8798
8772
[WARNING|trainer.py:803] 2025-04-26 18:26:07,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8717
[WARNING|trainer.py:803] 2025-04-26 18:26:08,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:08,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8799
8773
[WARNING|trainer.py:803] 2025-04-26 18:26:09,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8718
[WARNING|trainer.py:803] 2025-04-26 18:26:09,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:09,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8800
8774
[WARNING|trainer.py:803] 2025-04-26 18:26:10,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8719
[WARNING|trainer.py:803] 2025-04-26 18:26:11,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:26:11,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8801
8775
[WARNING|trainer.py:803] 2025-04-26 18:26:11,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8720
[WARNING|trainer.py:803] 2025-04-26 18:26:12,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:12,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:13,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8776
8802
8721
[WARNING|trainer.py:803] 2025-04-26 18:26:13,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:13,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8803
[WARNING|trainer.py:803] 2025-04-26 18:26:14,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8777
8722
[WARNING|trainer.py:803] 2025-04-26 18:26:15,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:15,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:15,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8778
8804
8723
[WARNING|trainer.py:803] 2025-04-26 18:26:16,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:16,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:17,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8805
8779
8724
[WARNING|trainer.py:803] 2025-04-26 18:26:17,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:17,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:18,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8806
8780
8725
[WARNING|trainer.py:803] 2025-04-26 18:26:19,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:26:19,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:19,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8807
8781
8726
[WARNING|trainer.py:803] 2025-04-26 18:26:20,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:26:20,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:21,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8808
8782
8727
[WARNING|trainer.py:803] 2025-04-26 18:26:21,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:22,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8809
[WARNING|trainer.py:803] 2025-04-26 18:26:22,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8783
8728
[WARNING|trainer.py:803] 2025-04-26 18:26:23,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:26:23,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8810
[WARNING|trainer.py:803] 2025-04-26 18:26:23,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8729
8784
[WARNING|trainer.py:803] 2025-04-26 18:26:24,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:24,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8811
[WARNING|trainer.py:803] 2025-04-26 18:26:24,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8785
8730
[WARNING|trainer.py:803] 2025-04-26 18:26:25,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8812
[WARNING|trainer.py:803] 2025-04-26 18:26:26,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:26,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8786
8731
[WARNING|trainer.py:803] 2025-04-26 18:26:27,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8813
[WARNING|trainer.py:803] 2025-04-26 18:26:27,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:26:27,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8787
8732
[WARNING|trainer.py:803] 2025-04-26 18:26:28,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8814
[WARNING|trainer.py:803] 2025-04-26 18:26:28,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:28,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8788
8733
[WARNING|trainer.py:803] 2025-04-26 18:26:29,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8815
[WARNING|trainer.py:803] 2025-04-26 18:26:30,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:30,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8789
8734
[WARNING|trainer.py:803] 2025-04-26 18:26:30,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8816
[WARNING|trainer.py:803] 2025-04-26 18:26:31,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:31,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8790
8735
[WARNING|trainer.py:803] 2025-04-26 18:26:32,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8817
[WARNING|trainer.py:803] 2025-04-26 18:26:32,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:32,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8736
8791
[WARNING|trainer.py:803] 2025-04-26 18:26:33,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8818
[WARNING|trainer.py:803] 2025-04-26 18:26:34,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:34,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8737
8792
[WARNING|trainer.py:803] 2025-04-26 18:26:35,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8819
[WARNING|trainer.py:803] 2025-04-26 18:26:35,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:35,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8738
8793
[WARNING|trainer.py:803] 2025-04-26 18:26:36,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8820
[WARNING|trainer.py:803] 2025-04-26 18:26:36,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:37,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8739
8794
[WARNING|trainer.py:803] 2025-04-26 18:26:37,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:38,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8821
[WARNING|trainer.py:803] 2025-04-26 18:26:38,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8740
8795
[WARNING|trainer.py:803] 2025-04-26 18:26:39,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:39,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8822
[WARNING|trainer.py:803] 2025-04-26 18:26:39,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8741
8796
[WARNING|trainer.py:803] 2025-04-26 18:26:40,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8823
[WARNING|trainer.py:803] 2025-04-26 18:26:40,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:40,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8742
8797
[WARNING|trainer.py:803] 2025-04-26 18:26:41,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:42,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8824
[WARNING|trainer.py:803] 2025-04-26 18:26:42,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8743
8798
[WARNING|trainer.py:803] 2025-04-26 18:26:43,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8825
[WARNING|trainer.py:803] 2025-04-26 18:26:43,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:43,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8744
8799
[WARNING|trainer.py:803] 2025-04-26 18:26:44,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8826
[WARNING|trainer.py:803] 2025-04-26 18:26:44,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:44,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8745
8800
[WARNING|trainer.py:803] 2025-04-26 18:26:45,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8827
[WARNING|trainer.py:803] 2025-04-26 18:26:46,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:46,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8746
8801
[WARNING|trainer.py:803] 2025-04-26 18:26:46,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8828
[WARNING|trainer.py:803] 2025-04-26 18:26:47,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:47,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8747
8802
[WARNING|trainer.py:803] 2025-04-26 18:26:48,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8829
[WARNING|trainer.py:803] 2025-04-26 18:26:48,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:49,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8748
8803
[WARNING|trainer.py:803] 2025-04-26 18:26:49,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8830
[WARNING|trainer.py:803] 2025-04-26 18:26:50,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:50,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8749
[WARNING|trainer.py:803] 2025-04-26 18:26:50,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8804
8831
[WARNING|trainer.py:803] 2025-04-26 18:26:51,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:51,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8750
[WARNING|trainer.py:803] 2025-04-26 18:26:52,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8805
8832
[WARNING|trainer.py:803] 2025-04-26 18:26:52,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:53,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8751
[WARNING|trainer.py:803] 2025-04-26 18:26:53,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8806
8833
[WARNING|trainer.py:803] 2025-04-26 18:26:54,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:54,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8752
[WARNING|trainer.py:803] 2025-04-26 18:26:54,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8807
8834
[WARNING|trainer.py:803] 2025-04-26 18:26:55,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:55,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8753
[WARNING|trainer.py:803] 2025-04-26 18:26:56,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8808
8835
[WARNING|trainer.py:803] 2025-04-26 18:26:56,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:57,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8754
[WARNING|trainer.py:803] 2025-04-26 18:26:57,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8809
8836
[WARNING|trainer.py:803] 2025-04-26 18:26:58,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:26:58,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8755
[WARNING|trainer.py:803] 2025-04-26 18:26:58,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8810
8837
[WARNING|trainer.py:803] 2025-04-26 18:26:59,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:26:59,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8756
[WARNING|trainer.py:803] 2025-04-26 18:27:00,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8811
8838
[WARNING|trainer.py:803] 2025-04-26 18:27:00,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:01,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8757
[WARNING|trainer.py:803] 2025-04-26 18:27:01,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8812
8839
[WARNING|trainer.py:803] 2025-04-26 18:27:02,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:02,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8758
[WARNING|trainer.py:803] 2025-04-26 18:27:02,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8813
8840
[WARNING|trainer.py:803] 2025-04-26 18:27:03,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:03,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8759
[WARNING|trainer.py:803] 2025-04-26 18:27:03,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8814
8841
[WARNING|trainer.py:803] 2025-04-26 18:27:04,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:05,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8760
[WARNING|trainer.py:803] 2025-04-26 18:27:05,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8815
8842
[WARNING|trainer.py:803] 2025-04-26 18:27:06,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:06,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:06,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8761
8816
8843
[WARNING|trainer.py:803] 2025-04-26 18:27:07,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:07,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:07,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8762
8817
8844
[WARNING|trainer.py:803] 2025-04-26 18:27:08,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:09,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:27:09,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8763
8818
8845
[WARNING|trainer.py:803] 2025-04-26 18:27:09,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8764
[WARNING|trainer.py:803] 2025-04-26 18:27:10,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:10,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8819
8846
[WARNING|trainer.py:803] 2025-04-26 18:27:11,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8765
[WARNING|trainer.py:803] 2025-04-26 18:27:11,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:11,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8847
8820
[WARNING|trainer.py:803] 2025-04-26 18:27:12,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:13,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8766
[WARNING|trainer.py:803] 2025-04-26 18:27:13,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8848
8821
[WARNING|trainer.py:803] 2025-04-26 18:27:13,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8767
[WARNING|trainer.py:803] 2025-04-26 18:27:14,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:14,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8849
8822
[WARNING|trainer.py:803] 2025-04-26 18:27:15,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:15,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8768
[WARNING|trainer.py:803] 2025-04-26 18:27:15,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8850
8823
[WARNING|trainer.py:803] 2025-04-26 18:27:16,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:17,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8769
[WARNING|trainer.py:803] 2025-04-26 18:27:17,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8851
8824
[WARNING|trainer.py:803] 2025-04-26 18:27:17,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:18,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8770
[WARNING|trainer.py:803] 2025-04-26 18:27:18,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8852
8825
[WARNING|trainer.py:803] 2025-04-26 18:27:19,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:19,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8771
[WARNING|trainer.py:803] 2025-04-26 18:27:19,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8853
8826
[WARNING|trainer.py:803] 2025-04-26 18:27:20,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8772
[WARNING|trainer.py:803] 2025-04-26 18:27:21,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:21,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8854
8827
[WARNING|trainer.py:803] 2025-04-26 18:27:21,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8773
[WARNING|trainer.py:803] 2025-04-26 18:27:22,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:22,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8855
8828
[WARNING|trainer.py:803] 2025-04-26 18:27:23,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8774
[WARNING|trainer.py:803] 2025-04-26 18:27:23,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:23,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8856
8829
[WARNING|trainer.py:803] 2025-04-26 18:27:24,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8775
[WARNING|trainer.py:803] 2025-04-26 18:27:25,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:25,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8857
8830
[WARNING|trainer.py:803] 2025-04-26 18:27:25,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8776
[WARNING|trainer.py:803] 2025-04-26 18:27:26,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:26,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8831
8858
[WARNING|trainer.py:803] 2025-04-26 18:27:27,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8777
[WARNING|trainer.py:803] 2025-04-26 18:27:27,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:27,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8859
8832
[WARNING|trainer.py:803] 2025-04-26 18:27:28,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8778
[WARNING|trainer.py:803] 2025-04-26 18:27:29,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:29,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8860
8833
[WARNING|trainer.py:803] 2025-04-26 18:27:29,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8779
[WARNING|trainer.py:803] 2025-04-26 18:27:30,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:30,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8861
8834
[WARNING|trainer.py:803] 2025-04-26 18:27:31,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8780
[WARNING|trainer.py:803] 2025-04-26 18:27:31,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:31,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8862
8835
[WARNING|trainer.py:803] 2025-04-26 18:27:32,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8781
[WARNING|trainer.py:803] 2025-04-26 18:27:32,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:33,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8863
8836
[WARNING|trainer.py:803] 2025-04-26 18:27:33,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8782
[WARNING|trainer.py:803] 2025-04-26 18:27:34,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:34,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8837
8864
[WARNING|trainer.py:803] 2025-04-26 18:27:34,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8783
[WARNING|trainer.py:803] 2025-04-26 18:27:35,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:35,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8838
8865
[WARNING|trainer.py:803] 2025-04-26 18:27:36,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8784
[WARNING|trainer.py:803] 2025-04-26 18:27:36,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:36,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8839
8866
[WARNING|trainer.py:803] 2025-04-26 18:27:37,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8785
[WARNING|trainer.py:803] 2025-04-26 18:27:38,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:38,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8840
8867
[WARNING|trainer.py:803] 2025-04-26 18:27:39,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8786
[WARNING|trainer.py:803] 2025-04-26 18:27:39,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:27:39,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8841
8868
[WARNING|trainer.py:803] 2025-04-26 18:27:40,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8787
[WARNING|trainer.py:803] 2025-04-26 18:27:40,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:41,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8842
8869
[WARNING|trainer.py:803] 2025-04-26 18:27:41,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8788
[WARNING|trainer.py:803] 2025-04-26 18:27:42,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:27:42,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8843
8870
[WARNING|trainer.py:803] 2025-04-26 18:27:42,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8789
[WARNING|trainer.py:803] 2025-04-26 18:27:43,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:43,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8844
8871
[WARNING|trainer.py:803] 2025-04-26 18:27:44,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8790
[WARNING|trainer.py:803] 2025-04-26 18:27:44,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:44,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8845
8872
[WARNING|trainer.py:803] 2025-04-26 18:27:45,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8791
[WARNING|trainer.py:803] 2025-04-26 18:27:46,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:46,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8846
8873
[WARNING|trainer.py:803] 2025-04-26 18:27:46,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:47,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8792
[WARNING|trainer.py:803] 2025-04-26 18:27:47,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8847
8874
[WARNING|trainer.py:803] 2025-04-26 18:27:48,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:48,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:48,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8793
8875
8848
[WARNING|trainer.py:803] 2025-04-26 18:27:49,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:50,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8794
[WARNING|trainer.py:803] 2025-04-26 18:27:50,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8876
8849
[WARNING|trainer.py:803] 2025-04-26 18:27:50,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:51,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8795
[WARNING|trainer.py:803] 2025-04-26 18:27:51,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8877
8850
[WARNING|trainer.py:803] 2025-04-26 18:27:52,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:27:52,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8796
[WARNING|trainer.py:803] 2025-04-26 18:27:52,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8878
8851
[WARNING|trainer.py:803] 2025-04-26 18:27:53,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:27:54,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8797
[WARNING|trainer.py:803] 2025-04-26 18:27:54,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8879
8852
[WARNING|trainer.py:803] 2025-04-26 18:27:54,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:55,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8798
[WARNING|trainer.py:803] 2025-04-26 18:27:55,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8880
8853
[WARNING|trainer.py:803] 2025-04-26 18:27:56,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:56,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8799
[WARNING|trainer.py:803] 2025-04-26 18:27:56,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8881
8854
[WARNING|trainer.py:803] 2025-04-26 18:27:57,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:27:58,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8800
[WARNING|trainer.py:803] 2025-04-26 18:27:58,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8882
8855
[WARNING|trainer.py:803] 2025-04-26 18:27:58,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:27:59,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8801
[WARNING|trainer.py:803] 2025-04-26 18:27:59,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8883
8856
[WARNING|trainer.py:803] 2025-04-26 18:28:00,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:00,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8802
[WARNING|trainer.py:803] 2025-04-26 18:28:00,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8884
8857
[WARNING|trainer.py:803] 2025-04-26 18:28:01,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:01,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8803
[WARNING|trainer.py:803] 2025-04-26 18:28:02,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8885
8858
[WARNING|trainer.py:803] 2025-04-26 18:28:02,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:03,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8804
[WARNING|trainer.py:803] 2025-04-26 18:28:03,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8886
8859
[WARNING|trainer.py:803] 2025-04-26 18:28:04,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:04,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8805
[WARNING|trainer.py:803] 2025-04-26 18:28:04,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8887
8860
[WARNING|trainer.py:803] 2025-04-26 18:28:05,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:05,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8806
[WARNING|trainer.py:803] 2025-04-26 18:28:06,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8888
8861
[WARNING|trainer.py:803] 2025-04-26 18:28:06,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:07,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8807
[WARNING|trainer.py:803] 2025-04-26 18:28:07,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8889
8862
[WARNING|trainer.py:803] 2025-04-26 18:28:08,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:08,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8808
[WARNING|trainer.py:803] 2025-04-26 18:28:08,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8890
8863
[WARNING|trainer.py:803] 2025-04-26 18:28:09,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:09,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:10,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8809
8891
8864
[WARNING|trainer.py:803] 2025-04-26 18:28:11,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:11,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:11,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8810
8892
8865
[WARNING|trainer.py:803] 2025-04-26 18:28:12,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:12,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:12,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8811
8893
8866
[WARNING|trainer.py:803] 2025-04-26 18:28:13,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:13,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:14,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8812
8894
8867
[WARNING|trainer.py:803] 2025-04-26 18:28:15,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:15,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:15,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8813
8895
8868
[WARNING|trainer.py:803] 2025-04-26 18:28:16,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:16,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:16,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8814
8896
8869
[WARNING|trainer.py:803] 2025-04-26 18:28:17,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:17,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:18,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8815
8897
8870
[WARNING|trainer.py:803] 2025-04-26 18:28:19,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:19,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:19,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8816
8898
8871
[WARNING|trainer.py:803] 2025-04-26 18:28:20,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:20,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:20,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8817
8899
8872
[WARNING|trainer.py:803] 2025-04-26 18:28:21,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:28:21,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:22,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8818
8900
8873
[WARNING|trainer.py:803] 2025-04-26 18:28:23,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:23,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:23,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8901
8819
8874
[WARNING|trainer.py:803] 2025-04-26 18:28:24,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:24,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:24,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8902
8820
8875
[WARNING|trainer.py:803] 2025-04-26 18:28:25,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:25,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:26,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8903
8821
8876
[WARNING|trainer.py:803] 2025-04-26 18:28:27,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:27,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:27,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8904
8877
8822
[WARNING|trainer.py:803] 2025-04-26 18:28:28,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:28,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:28,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8905
8823
8878
[WARNING|trainer.py:803] 2025-04-26 18:28:29,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:30,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:30,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8906
8879
8824
[WARNING|trainer.py:803] 2025-04-26 18:28:31,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:31,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:31,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8907
8825
8880
[WARNING|trainer.py:803] 2025-04-26 18:28:32,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:32,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:32,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8908
8826
8881
[WARNING|trainer.py:803] 2025-04-26 18:28:33,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:34,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:34,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8909
8882
8827
[WARNING|trainer.py:803] 2025-04-26 18:28:35,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:35,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:35,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8910
8883
8828
[WARNING|trainer.py:803] 2025-04-26 18:28:36,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:36,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:36,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8911
8884
8829
[WARNING|trainer.py:803] 2025-04-26 18:28:37,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:38,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:38,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8912
8885
8830
[WARNING|trainer.py:803] 2025-04-26 18:28:38,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:39,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:39,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8913
8886
8831
[WARNING|trainer.py:803] 2025-04-26 18:28:40,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:40,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:40,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8914
8887
8832
[WARNING|trainer.py:803] 2025-04-26 18:28:41,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:41,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:42,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8915
8888
8833
[WARNING|trainer.py:803] 2025-04-26 18:28:43,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:43,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:43,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8916
8889
8834
[WARNING|trainer.py:803] 2025-04-26 18:28:44,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:44,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:44,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8917
8890
8835
[WARNING|trainer.py:803] 2025-04-26 18:28:45,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:45,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:46,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8918
8891
8836
[WARNING|trainer.py:803] 2025-04-26 18:28:46,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:47,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:47,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8919
8837
8892
[WARNING|trainer.py:803] 2025-04-26 18:28:48,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:48,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:48,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8920
8838
8893
[WARNING|trainer.py:803] 2025-04-26 18:28:49,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:49,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:28:50,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8921
8839
8894
[WARNING|trainer.py:803] 2025-04-26 18:28:51,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:51,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:51,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8922
8840
8895
[WARNING|trainer.py:803] 2025-04-26 18:28:52,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:52,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:52,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8923
8841
8896
[WARNING|trainer.py:803] 2025-04-26 18:28:53,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:53,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:54,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8924
8842
8897
[WARNING|trainer.py:803] 2025-04-26 18:28:55,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:55,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:28:55,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8925
8898
8843
[WARNING|trainer.py:803] 2025-04-26 18:28:56,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:56,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:28:56,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8926
8899
8844
[WARNING|trainer.py:803] 2025-04-26 18:28:57,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:58,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:58,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8927
8900
8845
[WARNING|trainer.py:803] 2025-04-26 18:28:59,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:28:59,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:28:59,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8928
8901
8846
[WARNING|trainer.py:803] 2025-04-26 18:29:00,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:00,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:29:00,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8929
8847
8902
[WARNING|trainer.py:803] 2025-04-26 18:29:01,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:02,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:02,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8930
8903
8848
[WARNING|trainer.py:803] 2025-04-26 18:29:03,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:03,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:03,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8931
8904
8849
[WARNING|trainer.py:803] 2025-04-26 18:29:04,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:04,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:04,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8932
8850
8905
[WARNING|trainer.py:803] 2025-04-26 18:29:05,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:06,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:06,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8933
8851
8906
[WARNING|trainer.py:803] 2025-04-26 18:29:07,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:07,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:29:07,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8934
8907
8852
[WARNING|trainer.py:803] 2025-04-26 18:29:08,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:08,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:08,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8935
8908
8853
[WARNING|trainer.py:803] 2025-04-26 18:29:09,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:10,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:10,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8936
8854
8909
[WARNING|trainer.py:803] 2025-04-26 18:29:11,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:11,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:11,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8937
8855
8910
[WARNING|trainer.py:803] 2025-04-26 18:29:12,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:12,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:12,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8938
8856
8911
[WARNING|trainer.py:803] 2025-04-26 18:29:13,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:14,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:14,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8939
8912
8857
[WARNING|trainer.py:803] 2025-04-26 18:29:15,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:15,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:15,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8940
8913
8858
[WARNING|trainer.py:803] 2025-04-26 18:29:16,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:16,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:16,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8941
8914
8859
[WARNING|trainer.py:803] 2025-04-26 18:29:17,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:18,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:18,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8942
8915
8860
[WARNING|trainer.py:803] 2025-04-26 18:29:19,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:19,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:19,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8943
8916
8861
[WARNING|trainer.py:803] 2025-04-26 18:29:20,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:20,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:20,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8944
8862
8917
[WARNING|trainer.py:803] 2025-04-26 18:29:21,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:22,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:22,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8945
8918
8863
[WARNING|trainer.py:803] 2025-04-26 18:29:23,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:23,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:23,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8946
8919
8864
[WARNING|trainer.py:803] 2025-04-26 18:29:24,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:29:24,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:24,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8947
8920
8865
[WARNING|trainer.py:803] 2025-04-26 18:29:25,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:26,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:26,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8948
8866
8921
[WARNING|trainer.py:803] 2025-04-26 18:29:27,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:29:27,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:27,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8949
8867
8922
[WARNING|trainer.py:803] 2025-04-26 18:29:28,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:28,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:28,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8950
8868
8923
[WARNING|trainer.py:803] 2025-04-26 18:29:29,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:30,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:29:30,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8951
8869
8924
[WARNING|trainer.py:803] 2025-04-26 18:29:31,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:31,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:29:31,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8952
8870
8925
[WARNING|trainer.py:803] 2025-04-26 18:29:32,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:32,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:32,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8953
8871
8926
[WARNING|trainer.py:803] 2025-04-26 18:29:33,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:34,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:34,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8954
8872
8927
[WARNING|trainer.py:803] 2025-04-26 18:29:35,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:35,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8955
[WARNING|trainer.py:803] 2025-04-26 18:29:35,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8873
8928
[WARNING|trainer.py:803] 2025-04-26 18:29:36,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:29:36,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8956
[WARNING|trainer.py:803] 2025-04-26 18:29:36,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8874
8929
[WARNING|trainer.py:803] 2025-04-26 18:29:37,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:37,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8957
[WARNING|trainer.py:803] 2025-04-26 18:29:38,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8875
8930
[WARNING|trainer.py:803] 2025-04-26 18:29:39,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:39,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8958
[WARNING|trainer.py:803] 2025-04-26 18:29:39,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8876
8931
[WARNING|trainer.py:803] 2025-04-26 18:29:40,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:40,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8959
[WARNING|trainer.py:803] 2025-04-26 18:29:40,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8877
8932
[WARNING|trainer.py:803] 2025-04-26 18:29:41,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:41,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8960
[WARNING|trainer.py:803] 2025-04-26 18:29:42,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8878
8933
[WARNING|trainer.py:803] 2025-04-26 18:29:42,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:43,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8961
[WARNING|trainer.py:803] 2025-04-26 18:29:43,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8879
8934
[WARNING|trainer.py:803] 2025-04-26 18:29:44,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:44,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8962
[WARNING|trainer.py:803] 2025-04-26 18:29:44,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8880
8935
[WARNING|trainer.py:803] 2025-04-26 18:29:45,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:45,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8963
[WARNING|trainer.py:803] 2025-04-26 18:29:46,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8881
8936
[WARNING|trainer.py:803] 2025-04-26 18:29:46,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:47,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8964
[WARNING|trainer.py:803] 2025-04-26 18:29:47,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8882
8937
[WARNING|trainer.py:803] 2025-04-26 18:29:48,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:48,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8965
[WARNING|trainer.py:803] 2025-04-26 18:29:48,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8883
[WARNING|trainer.py:803] 2025-04-26 18:29:49,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8938
[WARNING|trainer.py:803] 2025-04-26 18:29:49,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8966
[WARNING|trainer.py:803] 2025-04-26 18:29:50,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8884
[WARNING|trainer.py:803] 2025-04-26 18:29:50,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8939
[WARNING|trainer.py:803] 2025-04-26 18:29:51,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8967
[WARNING|trainer.py:803] 2025-04-26 18:29:51,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8885
[WARNING|trainer.py:803] 2025-04-26 18:29:52,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8940
[WARNING|trainer.py:803] 2025-04-26 18:29:52,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8968
8886
[WARNING|trainer.py:803] 2025-04-26 18:29:53,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:53,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8941
[WARNING|trainer.py:803] 2025-04-26 18:29:53,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8969
8887
[WARNING|trainer.py:803] 2025-04-26 18:29:54,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:54,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8942
8970
[WARNING|trainer.py:803] 2025-04-26 18:29:55,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8888
[WARNING|trainer.py:803] 2025-04-26 18:29:55,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:55,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8943
[WARNING|trainer.py:803] 2025-04-26 18:29:56,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8971
8889
[WARNING|trainer.py:803] 2025-04-26 18:29:57,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:57,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8944
[WARNING|trainer.py:803] 2025-04-26 18:29:57,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8972
8890
[WARNING|trainer.py:803] 2025-04-26 18:29:58,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:29:58,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8945
[WARNING|trainer.py:803] 2025-04-26 18:29:59,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8973
8891
[WARNING|trainer.py:803] 2025-04-26 18:29:59,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:00,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8946
[WARNING|trainer.py:803] 2025-04-26 18:30:00,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8974
8892
[WARNING|trainer.py:803] 2025-04-26 18:30:01,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:30:01,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8947
[WARNING|trainer.py:803] 2025-04-26 18:30:01,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8975
8893
[WARNING|trainer.py:803] 2025-04-26 18:30:02,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:02,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8948
[WARNING|trainer.py:803] 2025-04-26 18:30:03,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8976
8894
[WARNING|trainer.py:803] 2025-04-26 18:30:03,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:30:04,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8949
[WARNING|trainer.py:803] 2025-04-26 18:30:04,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8977
8895
[WARNING|trainer.py:803] 2025-04-26 18:30:05,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:05,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8950
[WARNING|trainer.py:803] 2025-04-26 18:30:05,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8978
8896
[WARNING|trainer.py:803] 2025-04-26 18:30:06,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:06,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8951
[WARNING|trainer.py:803] 2025-04-26 18:30:07,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8979
8897
[WARNING|trainer.py:803] 2025-04-26 18:30:07,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:08,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8952
[WARNING|trainer.py:803] 2025-04-26 18:30:08,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8980
8898
[WARNING|trainer.py:803] 2025-04-26 18:30:09,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:09,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8953
[WARNING|trainer.py:803] 2025-04-26 18:30:09,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8981
8899
[WARNING|trainer.py:803] 2025-04-26 18:30:10,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:10,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8954
[WARNING|trainer.py:803] 2025-04-26 18:30:11,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8982
8900
[WARNING|trainer.py:803] 2025-04-26 18:30:11,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:12,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8955
[WARNING|trainer.py:803] 2025-04-26 18:30:12,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8983
8901
[WARNING|trainer.py:803] 2025-04-26 18:30:13,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:30:13,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8956
[WARNING|trainer.py:803] 2025-04-26 18:30:13,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8984
8902
[WARNING|trainer.py:803] 2025-04-26 18:30:14,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:14,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8957
[WARNING|trainer.py:803] 2025-04-26 18:30:14,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8985
8903
[WARNING|trainer.py:803] 2025-04-26 18:30:15,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:15,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8958
[WARNING|trainer.py:803] 2025-04-26 18:30:16,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8986
8904
[WARNING|trainer.py:803] 2025-04-26 18:30:16,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:17,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8959
[WARNING|trainer.py:803] 2025-04-26 18:30:17,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8987
8905
[WARNING|trainer.py:803] 2025-04-26 18:30:18,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:18,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8960
[WARNING|trainer.py:803] 2025-04-26 18:30:18,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8988
8906
[WARNING|trainer.py:803] 2025-04-26 18:30:19,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:20,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8961
[WARNING|trainer.py:803] 2025-04-26 18:30:20,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8989
8907
[WARNING|trainer.py:803] 2025-04-26 18:30:20,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:21,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8962
[WARNING|trainer.py:803] 2025-04-26 18:30:21,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8990
8908
[WARNING|trainer.py:803] 2025-04-26 18:30:22,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8963
[WARNING|trainer.py:803] 2025-04-26 18:30:22,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:22,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8991
8909
[WARNING|trainer.py:803] 2025-04-26 18:30:23,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:23,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8964
[WARNING|trainer.py:803] 2025-04-26 18:30:24,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8992
8910
[WARNING|trainer.py:803] 2025-04-26 18:30:24,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:25,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8965
[WARNING|trainer.py:803] 2025-04-26 18:30:25,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8993
8911
[WARNING|trainer.py:803] 2025-04-26 18:30:26,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:26,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8966
[WARNING|trainer.py:803] 2025-04-26 18:30:26,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8994
8912
[WARNING|trainer.py:803] 2025-04-26 18:30:27,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:27,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8967
[WARNING|trainer.py:803] 2025-04-26 18:30:28,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8995
8913
[WARNING|trainer.py:803] 2025-04-26 18:30:28,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:29,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8968
[WARNING|trainer.py:803] 2025-04-26 18:30:29,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8996
8914
[WARNING|trainer.py:803] 2025-04-26 18:30:30,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:30,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8969
[WARNING|trainer.py:803] 2025-04-26 18:30:30,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8997
[WARNING|trainer.py:803] 2025-04-26 18:30:31,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8915
[WARNING|trainer.py:803] 2025-04-26 18:30:31,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8970
8998
[WARNING|trainer.py:803] 2025-04-26 18:30:32,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:32,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8916
[WARNING|trainer.py:803] 2025-04-26 18:30:32,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8971
8999
[WARNING|trainer.py:803] 2025-04-26 18:30:33,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8917
[WARNING|trainer.py:803] 2025-04-26 18:30:34,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:34,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8972
9000
[WARNING|trainer.py:803] 2025-04-26 18:30:34,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:35,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8918
[WARNING|trainer.py:803] 2025-04-26 18:30:35,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.7871111111111111
New best accuracy: 0.7871111111111111. Saving model...
8973
[WARNING|trainer.py:803] 2025-04-26 18:30:36,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:36,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8919
8974
[WARNING|trainer.py:803] 2025-04-26 18:30:37,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8920
[WARNING|trainer.py:803] 2025-04-26 18:30:37,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8975
[WARNING|trainer.py:803] 2025-04-26 18:30:38,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:39,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8921
8976
[WARNING|trainer.py:803] 2025-04-26 18:30:40,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:40,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8922
8977
[WARNING|trainer.py:803] 2025-04-26 18:30:41,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:42,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8923
8978
[WARNING|trainer.py:803] 2025-04-26 18:30:42,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:43,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8924
8979
[WARNING|trainer.py:803] 2025-04-26 18:30:44,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8925
[WARNING|trainer.py:803] 2025-04-26 18:30:44,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8980
[WARNING|trainer.py:803] 2025-04-26 18:30:45,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8926
[WARNING|trainer.py:803] 2025-04-26 18:30:45,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8981
[WARNING|trainer.py:803] 2025-04-26 18:30:46,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8927
[WARNING|trainer.py:803] 2025-04-26 18:30:47,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8982
evaluate!
[WARNING|trainer.py:803] 2025-04-26 18:30:48,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8928
[WARNING|trainer.py:803] 2025-04-26 18:30:48,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8983
[WARNING|trainer.py:803] 2025-04-26 18:30:49,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1
8929
[WARNING|trainer.py:803] 2025-04-26 18:30:50,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8984
[WARNING|trainer.py:803] 2025-04-26 18:30:50,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:30:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8930
[WARNING|trainer.py:803] 2025-04-26 18:30:51,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8985
[WARNING|trainer.py:803] 2025-04-26 18:30:52,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2
8931
[WARNING|trainer.py:803] 2025-04-26 18:30:52,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:30:53,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8986
[WARNING|trainer.py:803] 2025-04-26 18:30:53,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8932
[WARNING|trainer.py:803] 2025-04-26 18:30:54,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8987
[WARNING|trainer.py:803] 2025-04-26 18:30:54,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
8933
[WARNING|trainer.py:803] 2025-04-26 18:30:55,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:55,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8988
[WARNING|trainer.py:803] 2025-04-26 18:30:56,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8934
[WARNING|trainer.py:803] 2025-04-26 18:30:56,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8989
4
[WARNING|trainer.py:803] 2025-04-26 18:30:57,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8935
[WARNING|trainer.py:803] 2025-04-26 18:30:58,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:30:58,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8990
[WARNING|trainer.py:803] 2025-04-26 18:30:58,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8936
[WARNING|trainer.py:803] 2025-04-26 18:30:59,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5
8991
[WARNING|trainer.py:803] 2025-04-26 18:31:00,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:00,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8937
[WARNING|trainer.py:803] 2025-04-26 18:31:00,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8992
[WARNING|trainer.py:803] 2025-04-26 18:31:01,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:01,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8938
6
8993
[WARNING|trainer.py:803] 2025-04-26 18:31:02,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:03,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:03,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8939
8994
[WARNING|trainer.py:803] 2025-04-26 18:31:04,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:04,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7
8940
8995
[WARNING|trainer.py:803] 2025-04-26 18:31:05,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:05,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:05,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8941
8996
[WARNING|trainer.py:803] 2025-04-26 18:31:06,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8
[WARNING|trainer.py:803] 2025-04-26 18:31:07,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8942
8997
[WARNING|trainer.py:803] 2025-04-26 18:31:07,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:31:08,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:08,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8943
8998
[WARNING|trainer.py:803] 2025-04-26 18:31:09,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
9
[WARNING|trainer.py:803] 2025-04-26 18:31:09,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8944
8999
[WARNING|trainer.py:803] 2025-04-26 18:31:10,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:10,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:11,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8945
9000
[WARNING|trainer.py:803] 2025-04-26 18:31:12,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
10
[WARNING|trainer.py:803] 2025-04-26 18:31:12,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.7871111111111111
New best accuracy: 0.7871111111111111. Saving model...
8946
[WARNING|trainer.py:803] 2025-04-26 18:31:12,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:13,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8947
11
[WARNING|trainer.py:803] 2025-04-26 18:31:14,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8948
[WARNING|trainer.py:803] 2025-04-26 18:31:15,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:16,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8949
12
[WARNING|trainer.py:803] 2025-04-26 18:31:17,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:17,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8950
[WARNING|trainer.py:803] 2025-04-26 18:31:18,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8951
13
[WARNING|trainer.py:803] 2025-04-26 18:31:20,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:20,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8952
[WARNING|trainer.py:803] 2025-04-26 18:31:21,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
14
8953
[WARNING|trainer.py:803] 2025-04-26 18:31:22,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:22,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8954
[WARNING|trainer.py:803] 2025-04-26 18:31:23,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
15
8955
[WARNING|trainer.py:803] 2025-04-26 18:31:24,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
evaluate!
[WARNING|trainer.py:803] 2025-04-26 18:31:25,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8956
16
[WARNING|trainer.py:803] 2025-04-26 18:31:26,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8957
1
[WARNING|trainer.py:803] 2025-04-26 18:31:27,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:27,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:31:27,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8958
17
[WARNING|trainer.py:803] 2025-04-26 18:31:29,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2
8959
[WARNING|trainer.py:803] 2025-04-26 18:31:29,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:30,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:30,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8960
18
[WARNING|trainer.py:803] 2025-04-26 18:31:31,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3
[WARNING|trainer.py:803] 2025-04-26 18:31:32,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8961
[WARNING|trainer.py:803] 2025-04-26 18:31:32,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:33,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8962
19
4
[WARNING|trainer.py:803] 2025-04-26 18:31:34,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:34,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8963
[WARNING|trainer.py:803] 2025-04-26 18:31:35,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:31:35,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
20
8964
5
[WARNING|trainer.py:803] 2025-04-26 18:31:37,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:37,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8965
[WARNING|trainer.py:803] 2025-04-26 18:31:37,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:38,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
21
8966
6
[WARNING|trainer.py:803] 2025-04-26 18:31:39,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:39,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:40,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8967
22
[WARNING|trainer.py:803] 2025-04-26 18:31:41,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8968
7
[WARNING|trainer.py:803] 2025-04-26 18:31:41,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:42,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:42,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8969
23
[WARNING|trainer.py:803] 2025-04-26 18:31:43,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8
8970
[WARNING|trainer.py:803] 2025-04-26 18:31:44,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:44,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:31:45,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8971
[WARNING|trainer.py:803] 2025-04-26 18:31:46,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
24
9
8972
[WARNING|trainer.py:803] 2025-04-26 18:31:47,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:31:47,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:47,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8973
25
10
[WARNING|trainer.py:803] 2025-04-26 18:31:49,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8974
[WARNING|trainer.py:803] 2025-04-26 18:31:49,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:50,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:50,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8975
26
11
[WARNING|trainer.py:803] 2025-04-26 18:31:51,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:52,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:31:52,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8976
[WARNING|trainer.py:803] 2025-04-26 18:31:53,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8977
27
12
[WARNING|trainer.py:803] 2025-04-26 18:31:54,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:31:54,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:31:54,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8978
[WARNING|trainer.py:803] 2025-04-26 18:31:56,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
13
8979
28
[WARNING|trainer.py:803] 2025-04-26 18:31:57,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:57,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:31:57,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8980
[WARNING|trainer.py:803] 2025-04-26 18:31:58,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
14
29
8981
[WARNING|trainer.py:803] 2025-04-26 18:31:59,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:31:59,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:00,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8982
15
[WARNING|trainer.py:803] 2025-04-26 18:32:01,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
30
8983
[WARNING|trainer.py:803] 2025-04-26 18:32:02,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:32:02,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:02,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8984
16
[WARNING|trainer.py:803] 2025-04-26 18:32:04,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
31
8985
[WARNING|trainer.py:803] 2025-04-26 18:32:04,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:32:05,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:05,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8986
17
32
[WARNING|trainer.py:803] 2025-04-26 18:32:06,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:07,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8987
[WARNING|trainer.py:803] 2025-04-26 18:32:07,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:32:08,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8988
18
33
[WARNING|trainer.py:803] 2025-04-26 18:32:09,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:09,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:09,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8989
[WARNING|trainer.py:803] 2025-04-26 18:32:10,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
19
8990
34
[WARNING|trainer.py:803] 2025-04-26 18:32:11,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:12,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:12,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8991
[WARNING|trainer.py:803] 2025-04-26 18:32:13,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
20
8992
35
[WARNING|trainer.py:803] 2025-04-26 18:32:14,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:14,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:15,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8993
21
[WARNING|trainer.py:803] 2025-04-26 18:32:16,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8994
[WARNING|trainer.py:803] 2025-04-26 18:32:16,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
36
[WARNING|trainer.py:803] 2025-04-26 18:32:17,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:17,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8995
22
[WARNING|trainer.py:803] 2025-04-26 18:32:18,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:19,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8996
37
[WARNING|trainer.py:803] 2025-04-26 18:32:19,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:19,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8997
23
[WARNING|trainer.py:803] 2025-04-26 18:32:21,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
38
8998
[WARNING|trainer.py:803] 2025-04-26 18:32:21,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:22,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:32:22,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8999
24
[WARNING|trainer.py:803] 2025-04-26 18:32:23,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
39
9000
[WARNING|trainer.py:803] 2025-04-26 18:32:24,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:24,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:25,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.7871111111111111
New best accuracy: 0.7871111111111111. Saving model...
25
40
[WARNING|trainer.py:803] 2025-04-26 18:32:27,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:32:27,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
41
26
[WARNING|trainer.py:803] 2025-04-26 18:32:29,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:32:29,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
27
42
[WARNING|trainer.py:803] 2025-04-26 18:32:32,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:32,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
43
28
[WARNING|trainer.py:803] 2025-04-26 18:32:34,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:32:34,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[INFO|trainer.py:3910] 2025-04-26 18:32:35,434 >> Saving model checkpoint to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast
[INFO|configuration_utils.py:420] 2025-04-26 18:32:35,438 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/config.json
[INFO|configuration_utils.py:909] 2025-04-26 18:32:35,439 >> Configuration saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/generation_config.json
29
44
[WARNING|trainer.py:803] 2025-04-26 18:32:37,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:37,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
30
45
[WARNING|trainer.py:803] 2025-04-26 18:32:39,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:39,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
31
46
[WARNING|trainer.py:803] 2025-04-26 18:32:42,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:32:42,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
32
47
[WARNING|trainer.py:803] 2025-04-26 18:32:44,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:32:45,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
33
48
[WARNING|trainer.py:803] 2025-04-26 18:32:46,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:32:47,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
34
49
[WARNING|trainer.py:803] 2025-04-26 18:32:49,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:32:49,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
35
50
[WARNING|trainer.py:803] 2025-04-26 18:32:52,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:32:52,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
36
[WARNING|trainer.py:803] 2025-04-26 18:32:54,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
51
[WARNING|trainer.py:803] 2025-04-26 18:32:55,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
37
[WARNING|trainer.py:803] 2025-04-26 18:32:56,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
52
[WARNING|trainer.py:803] 2025-04-26 18:32:57,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
38
[WARNING|trainer.py:803] 2025-04-26 18:32:59,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
53
[WARNING|trainer.py:803] 2025-04-26 18:33:00,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
39
[WARNING|trainer.py:803] 2025-04-26 18:33:01,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
54
[WARNING|trainer.py:803] 2025-04-26 18:33:02,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
40
[WARNING|trainer.py:803] 2025-04-26 18:33:04,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
55
[WARNING|trainer.py:803] 2025-04-26 18:33:05,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
41
[WARNING|trainer.py:803] 2025-04-26 18:33:06,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
56
[WARNING|trainer.py:803] 2025-04-26 18:33:07,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
42
57
[WARNING|trainer.py:803] 2025-04-26 18:33:09,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:09,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
43
58
[WARNING|trainer.py:803] 2025-04-26 18:33:11,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:33:11,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
59
44
[WARNING|trainer.py:803] 2025-04-26 18:33:14,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:14,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
60
45
[WARNING|trainer.py:803] 2025-04-26 18:33:16,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:16,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
61
[WARNING|trainer.py:803] 2025-04-26 18:33:18,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
46
[WARNING|trainer.py:803] 2025-04-26 18:33:19,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
62
47
[WARNING|trainer.py:803] 2025-04-26 18:33:21,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:21,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
63
48
[WARNING|trainer.py:803] 2025-04-26 18:33:23,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:24,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
64
[INFO|modeling_utils.py:2996] 2025-04-26 18:33:25,484 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2491] 2025-04-26 18:33:25,487 >> tokenizer config file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/tokenizer_config.json
[INFO|tokenization_utils_base.py:2500] 2025-04-26 18:33:25,487 >> Special tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/special_tokens_map.json
[INFO|tokenization_utils_base.py:2553] 2025-04-26 18:33:25,487 >> added tokens file saved in /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/added_tokens.json
[WARNING|trainer.py:803] 2025-04-26 18:33:25,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
49
[WARNING|trainer.py:803] 2025-04-26 18:33:26,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
04/26/2025 18:33:27 - INFO - __main__ - Saved LoRA weights to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/lora_weights.pth
evaluate!
65
[WARNING|trainer.py:803] 2025-04-26 18:33:28,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
50
1
[WARNING|trainer.py:803] 2025-04-26 18:33:29,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:30,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
66
[WARNING|trainer.py:803] 2025-04-26 18:33:31,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2
51
[WARNING|trainer.py:803] 2025-04-26 18:33:32,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:33:32,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
67
3
[WARNING|trainer.py:803] 2025-04-26 18:33:34,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:34,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
52
[WARNING|trainer.py:803] 2025-04-26 18:33:35,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4
68
[WARNING|trainer.py:803] 2025-04-26 18:33:36,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:36,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
53
[WARNING|trainer.py:803] 2025-04-26 18:33:38,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5
69
[WARNING|trainer.py:803] 2025-04-26 18:33:39,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:39,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
54
6
[WARNING|trainer.py:803] 2025-04-26 18:33:40,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:41,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
70
7
55
[WARNING|trainer.py:803] 2025-04-26 18:33:42,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:43,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:43,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
71
8
56
[WARNING|trainer.py:803] 2025-04-26 18:33:45,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:45,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:33:46,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
9
72
57
[WARNING|trainer.py:803] 2025-04-26 18:33:48,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:48,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:48,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
10
58
73
[WARNING|trainer.py:803] 2025-04-26 18:33:50,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:33:51,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:51,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
11
[WARNING|trainer.py:803] 2025-04-26 18:33:52,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
59
74
[WARNING|trainer.py:803] 2025-04-26 18:33:53,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:33:53,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
12
[WARNING|trainer.py:803] 2025-04-26 18:33:54,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
60
13
[WARNING|trainer.py:803] 2025-04-26 18:33:56,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
75
[WARNING|trainer.py:803] 2025-04-26 18:33:57,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:57,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
61
14
[WARNING|trainer.py:803] 2025-04-26 18:33:58,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
76
[WARNING|trainer.py:803] 2025-04-26 18:33:59,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:33:59,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
15
62
[WARNING|trainer.py:803] 2025-04-26 18:34:01,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:01,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
77
[WARNING|trainer.py:803] 2025-04-26 18:34:02,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
16
63
[WARNING|trainer.py:803] 2025-04-26 18:34:03,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:04,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
78
17
[WARNING|trainer.py:803] 2025-04-26 18:34:05,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
64
[WARNING|trainer.py:803] 2025-04-26 18:34:05,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:06,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
79
18
[WARNING|trainer.py:803] 2025-04-26 18:34:07,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:08,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
65
[WARNING|trainer.py:803] 2025-04-26 18:34:09,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
19
80
[WARNING|trainer.py:803] 2025-04-26 18:34:10,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
66
[WARNING|trainer.py:803] 2025-04-26 18:34:10,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:11,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
20
81
[WARNING|trainer.py:803] 2025-04-26 18:34:12,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:34:12,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
67
21
[WARNING|trainer.py:803] 2025-04-26 18:34:14,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:14,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
82
[WARNING|trainer.py:803] 2025-04-26 18:34:15,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
22
68
[WARNING|trainer.py:803] 2025-04-26 18:34:16,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:16,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
83
23
69
[WARNING|trainer.py:803] 2025-04-26 18:34:18,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:18,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:34:19,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
84
24
70
[WARNING|trainer.py:803] 2025-04-26 18:34:21,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:21,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:21,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
25
85
71
[WARNING|trainer.py:803] 2025-04-26 18:34:23,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:23,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:24,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
26
86
[WARNING|trainer.py:803] 2025-04-26 18:34:25,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
72
[WARNING|trainer.py:803] 2025-04-26 18:34:26,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:34:26,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
27
[WARNING|trainer.py:803] 2025-04-26 18:34:28,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
87
73
[WARNING|trainer.py:803] 2025-04-26 18:34:29,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:34:29,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
28
88
[WARNING|trainer.py:803] 2025-04-26 18:34:30,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
74
[WARNING|trainer.py:803] 2025-04-26 18:34:31,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
29
[WARNING|trainer.py:803] 2025-04-26 18:34:31,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
89
[WARNING|trainer.py:803] 2025-04-26 18:34:32,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:33,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
30
75
90
[WARNING|trainer.py:803] 2025-04-26 18:34:35,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:35,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:34:35,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
31
76
[WARNING|trainer.py:803] 2025-04-26 18:34:37,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:37,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
91
32
[WARNING|trainer.py:803] 2025-04-26 18:34:38,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:34:39,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
77
[WARNING|trainer.py:803] 2025-04-26 18:34:40,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
92
33
[WARNING|trainer.py:803] 2025-04-26 18:34:41,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:34:41,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
78
[WARNING|trainer.py:803] 2025-04-26 18:34:42,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
93
34
[WARNING|trainer.py:803] 2025-04-26 18:34:43,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:43,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
79
94
[WARNING|trainer.py:803] 2025-04-26 18:34:45,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
35
[WARNING|trainer.py:803] 2025-04-26 18:34:46,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:34:46,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
80
36
95
[WARNING|trainer.py:803] 2025-04-26 18:34:47,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:48,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:48,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
81
37
96
[WARNING|trainer.py:803] 2025-04-26 18:34:50,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:50,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:34:50,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
82
38
97
[WARNING|trainer.py:803] 2025-04-26 18:34:52,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:52,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:53,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
39
[WARNING|trainer.py:803] 2025-04-26 18:34:55,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
83
98
[WARNING|trainer.py:803] 2025-04-26 18:34:55,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
40
[WARNING|trainer.py:803] 2025-04-26 18:34:56,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:34:57,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
84
99
41
[WARNING|trainer.py:803] 2025-04-26 18:34:58,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:58,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:34:59,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
100
85
42
[WARNING|trainer.py:803] 2025-04-26 18:35:01,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:01,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:01,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
43
86
101
[WARNING|trainer.py:803] 2025-04-26 18:35:03,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:35:03,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:04,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
44
87
102
[WARNING|trainer.py:803] 2025-04-26 18:35:06,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:35:06,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:06,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
45
88
103
[WARNING|trainer.py:803] 2025-04-26 18:35:08,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:08,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:09,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
89
46
[WARNING|trainer.py:803] 2025-04-26 18:35:10,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
104
[WARNING|trainer.py:803] 2025-04-26 18:35:11,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:12,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
90
47
[WARNING|trainer.py:803] 2025-04-26 18:35:12,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:13,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
105
[WARNING|trainer.py:803] 2025-04-26 18:35:14,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
48
91
[WARNING|trainer.py:803] 2025-04-26 18:35:15,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:35:15,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
106
49
[WARNING|trainer.py:803] 2025-04-26 18:35:17,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
92
[WARNING|trainer.py:803] 2025-04-26 18:35:18,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:18,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
107
50
93
[WARNING|trainer.py:803] 2025-04-26 18:35:20,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:20,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:20,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
108
51
94
[WARNING|trainer.py:803] 2025-04-26 18:35:22,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:23,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:23,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
109
95
[WARNING|trainer.py:803] 2025-04-26 18:35:24,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
52
[WARNING|trainer.py:803] 2025-04-26 18:35:25,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:25,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
53
110
96
[WARNING|trainer.py:803] 2025-04-26 18:35:27,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:35:28,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:28,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
54
111
97
[WARNING|trainer.py:803] 2025-04-26 18:35:29,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:30,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:30,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
55
112
[WARNING|trainer.py:803] 2025-04-26 18:35:32,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
98
[WARNING|trainer.py:803] 2025-04-26 18:35:33,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
56
[WARNING|trainer.py:803] 2025-04-26 18:35:33,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:34,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
113
99
57
[WARNING|trainer.py:803] 2025-04-26 18:35:35,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:35,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:36,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
114
58
100
[WARNING|trainer.py:803] 2025-04-26 18:35:37,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:35:38,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:38,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
115
59
[WARNING|trainer.py:803] 2025-04-26 18:35:40,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:40,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
101
[WARNING|trainer.py:803] 2025-04-26 18:35:41,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
60
116
[WARNING|trainer.py:803] 2025-04-26 18:35:42,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
102
[WARNING|trainer.py:803] 2025-04-26 18:35:43,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
61
[WARNING|trainer.py:803] 2025-04-26 18:35:43,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:44,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
117
[WARNING|trainer.py:803] 2025-04-26 18:35:45,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
103
62
[WARNING|trainer.py:803] 2025-04-26 18:35:46,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:47,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
118
63
104
[WARNING|trainer.py:803] 2025-04-26 18:35:48,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:49,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:49,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
64
119
[WARNING|trainer.py:803] 2025-04-26 18:35:51,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
105
[WARNING|trainer.py:803] 2025-04-26 18:35:51,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:35:52,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
65
[WARNING|trainer.py:803] 2025-04-26 18:35:53,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
120
106
[WARNING|trainer.py:803] 2025-04-26 18:35:54,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
66
[WARNING|trainer.py:803] 2025-04-26 18:35:54,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:55,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
121
107
67
[WARNING|trainer.py:803] 2025-04-26 18:35:57,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:35:57,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:35:57,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
108
68
122
[WARNING|trainer.py:803] 2025-04-26 18:36:00,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:00,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:00,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
109
69
123
[WARNING|trainer.py:803] 2025-04-26 18:36:02,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:02,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:02,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
70
110
124
[WARNING|trainer.py:803] 2025-04-26 18:36:05,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:05,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:05,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
71
111
[WARNING|trainer.py:803] 2025-04-26 18:36:07,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
125
[WARNING|trainer.py:803] 2025-04-26 18:36:08,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:08,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
72
112
[WARNING|trainer.py:803] 2025-04-26 18:36:09,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
126
[WARNING|trainer.py:803] 2025-04-26 18:36:10,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:10,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
73
113
[WARNING|trainer.py:803] 2025-04-26 18:36:12,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:36:12,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
127
74
[WARNING|trainer.py:803] 2025-04-26 18:36:14,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:14,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
114
[WARNING|trainer.py:803] 2025-04-26 18:36:15,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
128
[WARNING|trainer.py:803] 2025-04-26 18:36:16,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
75
115
[WARNING|trainer.py:803] 2025-04-26 18:36:17,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:17,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
129
76
[WARNING|trainer.py:803] 2025-04-26 18:36:18,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
116
[WARNING|trainer.py:803] 2025-04-26 18:36:19,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:36:20,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
130
77
[WARNING|trainer.py:803] 2025-04-26 18:36:21,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
117
[WARNING|trainer.py:803] 2025-04-26 18:36:22,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
131
[WARNING|trainer.py:803] 2025-04-26 18:36:22,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
78
[WARNING|trainer.py:803] 2025-04-26 18:36:23,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:24,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
118
79
132
[WARNING|trainer.py:803] 2025-04-26 18:36:26,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:26,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:26,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
119
80
133
[WARNING|trainer.py:803] 2025-04-26 18:36:28,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:28,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:29,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
81
134
[WARNING|trainer.py:803] 2025-04-26 18:36:30,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
120
[WARNING|trainer.py:803] 2025-04-26 18:36:31,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:31,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
82
[WARNING|trainer.py:803] 2025-04-26 18:36:33,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
135
121
[WARNING|trainer.py:803] 2025-04-26 18:36:34,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:36:34,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
83
[WARNING|trainer.py:803] 2025-04-26 18:36:36,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
136
122
[WARNING|trainer.py:803] 2025-04-26 18:36:37,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
84
[WARNING|trainer.py:803] 2025-04-26 18:36:37,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:38,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
137
123
85
[WARNING|trainer.py:803] 2025-04-26 18:36:40,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:40,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:36:40,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
138
86
124
[WARNING|trainer.py:803] 2025-04-26 18:36:42,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:43,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:43,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
139
87
125
[WARNING|trainer.py:803] 2025-04-26 18:36:45,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:45,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:45,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
88
140
[WARNING|trainer.py:803] 2025-04-26 18:36:47,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
126
[WARNING|trainer.py:803] 2025-04-26 18:36:47,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
89
[WARNING|trainer.py:803] 2025-04-26 18:36:48,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:49,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
141
90
[WARNING|trainer.py:803] 2025-04-26 18:36:50,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
127
[WARNING|trainer.py:803] 2025-04-26 18:36:51,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:51,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
142
91
128
[WARNING|trainer.py:803] 2025-04-26 18:36:53,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:53,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:36:53,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
92
143
129
[WARNING|trainer.py:803] 2025-04-26 18:36:56,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:36:56,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:56,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
93
144
130
[WARNING|trainer.py:803] 2025-04-26 18:36:58,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:36:58,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:36:59,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
94
131
[WARNING|trainer.py:803] 2025-04-26 18:37:00,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
145
[WARNING|trainer.py:803] 2025-04-26 18:37:01,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
95
[WARNING|trainer.py:803] 2025-04-26 18:37:01,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:37:02,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
132
146
[WARNING|trainer.py:803] 2025-04-26 18:37:04,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
96
[WARNING|trainer.py:803] 2025-04-26 18:37:04,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:04,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
133
147
97
[WARNING|trainer.py:803] 2025-04-26 18:37:06,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:37:06,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:37:07,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
134
98
[WARNING|trainer.py:803] 2025-04-26 18:37:09,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
148
[WARNING|trainer.py:803] 2025-04-26 18:37:09,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:10,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
99
135
[WARNING|trainer.py:803] 2025-04-26 18:37:11,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
149
[WARNING|trainer.py:803] 2025-04-26 18:37:12,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:37:12,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
100
136
[WARNING|trainer.py:803] 2025-04-26 18:37:14,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
150
[WARNING|trainer.py:803] 2025-04-26 18:37:14,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:37:15,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
101
[WARNING|trainer.py:803] 2025-04-26 18:37:16,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
137
151
[WARNING|trainer.py:803] 2025-04-26 18:37:17,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:17,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
102
[WARNING|trainer.py:803] 2025-04-26 18:37:18,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
138
152
[WARNING|trainer.py:803] 2025-04-26 18:37:20,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
103
[WARNING|trainer.py:803] 2025-04-26 18:37:20,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:37:21,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
139
153
[WARNING|trainer.py:803] 2025-04-26 18:37:23,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
104
[WARNING|trainer.py:803] 2025-04-26 18:37:23,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:24,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
140
154
[WARNING|trainer.py:803] 2025-04-26 18:37:25,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
105
[WARNING|trainer.py:803] 2025-04-26 18:37:25,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:26,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
141
155
[WARNING|trainer.py:803] 2025-04-26 18:37:28,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
106
[WARNING|trainer.py:803] 2025-04-26 18:37:28,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:29,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
142
156
107
[WARNING|trainer.py:803] 2025-04-26 18:37:31,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:31,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:31,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
157
108
143
[WARNING|trainer.py:803] 2025-04-26 18:37:33,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:33,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:33,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
109
144
[WARNING|trainer.py:803] 2025-04-26 18:37:35,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
158
[WARNING|trainer.py:803] 2025-04-26 18:37:36,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:37:36,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
110
159
[WARNING|trainer.py:803] 2025-04-26 18:37:38,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
145
[WARNING|trainer.py:803] 2025-04-26 18:37:38,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:39,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
111
160
[WARNING|trainer.py:803] 2025-04-26 18:37:40,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
146
[WARNING|trainer.py:803] 2025-04-26 18:37:41,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:41,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
112
[WARNING|trainer.py:803] 2025-04-26 18:37:43,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
161
147
[WARNING|trainer.py:803] 2025-04-26 18:37:44,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
113
[WARNING|trainer.py:803] 2025-04-26 18:37:44,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:37:45,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
162
[WARNING|trainer.py:803] 2025-04-26 18:37:46,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
114
148
[WARNING|trainer.py:803] 2025-04-26 18:37:47,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:37:47,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
163
115
[WARNING|trainer.py:803] 2025-04-26 18:37:49,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
149
[WARNING|trainer.py:803] 2025-04-26 18:37:49,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:37:50,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
116
164
[WARNING|trainer.py:803] 2025-04-26 18:37:52,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:52,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
150
[WARNING|trainer.py:803] 2025-04-26 18:37:53,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
117
165
[WARNING|trainer.py:803] 2025-04-26 18:37:54,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:37:54,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
151
[WARNING|trainer.py:803] 2025-04-26 18:37:55,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
166
118
[WARNING|trainer.py:803] 2025-04-26 18:37:57,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:57,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
152
[WARNING|trainer.py:803] 2025-04-26 18:37:58,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
119
167
[WARNING|trainer.py:803] 2025-04-26 18:37:59,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:37:59,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
153
[WARNING|trainer.py:803] 2025-04-26 18:38:01,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
120
168
[WARNING|trainer.py:803] 2025-04-26 18:38:02,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:38:02,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
154
[WARNING|trainer.py:803] 2025-04-26 18:38:03,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
121
[WARNING|trainer.py:803] 2025-04-26 18:38:05,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
169
155
[WARNING|trainer.py:803] 2025-04-26 18:38:06,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:38:06,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
122
[WARNING|trainer.py:803] 2025-04-26 18:38:07,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
170
156
[WARNING|trainer.py:803] 2025-04-26 18:38:08,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:38:09,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
123
[WARNING|trainer.py:803] 2025-04-26 18:38:09,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
171
157
[WARNING|trainer.py:803] 2025-04-26 18:38:11,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:38:11,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
124
172
[WARNING|trainer.py:803] 2025-04-26 18:38:12,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:13,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
158
125
[WARNING|trainer.py:803] 2025-04-26 18:38:14,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:14,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
173
[WARNING|trainer.py:803] 2025-04-26 18:38:15,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
126
159
[WARNING|trainer.py:803] 2025-04-26 18:38:17,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:17,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
174
[WARNING|trainer.py:803] 2025-04-26 18:38:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
160
127
175
[WARNING|trainer.py:803] 2025-04-26 18:38:19,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:20,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:20,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
128
161
[WARNING|trainer.py:803] 2025-04-26 18:38:22,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:22,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
176
[WARNING|trainer.py:803] 2025-04-26 18:38:23,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
129
162
[WARNING|trainer.py:803] 2025-04-26 18:38:24,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:24,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
177
130
[WARNING|trainer.py:803] 2025-04-26 18:38:25,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:38:26,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
163
[WARNING|trainer.py:803] 2025-04-26 18:38:27,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
131
178
[WARNING|trainer.py:803] 2025-04-26 18:38:28,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:28,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
164
[WARNING|trainer.py:803] 2025-04-26 18:38:30,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
132
179
[WARNING|trainer.py:803] 2025-04-26 18:38:31,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:31,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
165
133
[WARNING|trainer.py:803] 2025-04-26 18:38:32,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:38:33,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
180
166
[WARNING|trainer.py:803] 2025-04-26 18:38:34,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
134
[WARNING|trainer.py:803] 2025-04-26 18:38:35,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:35,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
181
[WARNING|trainer.py:803] 2025-04-26 18:38:37,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
135
167
[WARNING|trainer.py:803] 2025-04-26 18:38:38,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:38:38,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
182
136
[WARNING|trainer.py:803] 2025-04-26 18:38:39,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
168
[WARNING|trainer.py:803] 2025-04-26 18:38:40,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:38:41,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
183
[WARNING|trainer.py:803] 2025-04-26 18:38:42,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
137
[WARNING|trainer.py:803] 2025-04-26 18:38:43,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
169
184
[WARNING|trainer.py:803] 2025-04-26 18:38:44,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
138
[WARNING|trainer.py:803] 2025-04-26 18:38:45,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:45,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
170
185
[WARNING|trainer.py:803] 2025-04-26 18:38:46,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
139
[WARNING|trainer.py:803] 2025-04-26 18:38:47,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:38:47,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
171
[WARNING|trainer.py:803] 2025-04-26 18:38:49,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
140
186
[WARNING|trainer.py:803] 2025-04-26 18:38:50,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:50,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
172
[WARNING|trainer.py:803] 2025-04-26 18:38:51,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
141
187
[WARNING|trainer.py:803] 2025-04-26 18:38:52,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:52,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
173
[WARNING|trainer.py:803] 2025-04-26 18:38:54,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
142
[WARNING|trainer.py:803] 2025-04-26 18:38:55,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
188
174
[WARNING|trainer.py:803] 2025-04-26 18:38:56,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:56,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
143
[WARNING|trainer.py:803] 2025-04-26 18:38:57,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
189
175
[WARNING|trainer.py:803] 2025-04-26 18:38:58,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:38:58,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
144
[WARNING|trainer.py:803] 2025-04-26 18:39:00,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
190
176
[WARNING|trainer.py:803] 2025-04-26 18:39:01,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:01,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
145
191
[WARNING|trainer.py:803] 2025-04-26 18:39:03,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
177
[WARNING|trainer.py:803] 2025-04-26 18:39:03,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:04,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
146
[WARNING|trainer.py:803] 2025-04-26 18:39:05,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
192
178
[WARNING|trainer.py:803] 2025-04-26 18:39:06,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
147
[WARNING|trainer.py:803] 2025-04-26 18:39:07,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:39:07,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
193
179
[WARNING|trainer.py:803] 2025-04-26 18:39:09,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
148
[WARNING|trainer.py:803] 2025-04-26 18:39:09,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:10,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
194
[WARNING|trainer.py:803] 2025-04-26 18:39:11,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
149
180
[WARNING|trainer.py:803] 2025-04-26 18:39:12,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:13,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
195
[WARNING|trainer.py:803] 2025-04-26 18:39:14,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
150
181
[WARNING|trainer.py:803] 2025-04-26 18:39:15,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:15,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
196
151
[WARNING|trainer.py:803] 2025-04-26 18:39:16,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
182
[WARNING|trainer.py:803] 2025-04-26 18:39:17,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:18,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
197
152
[WARNING|trainer.py:803] 2025-04-26 18:39:19,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
183
[WARNING|trainer.py:803] 2025-04-26 18:39:20,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:39:20,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
198
153
[WARNING|trainer.py:803] 2025-04-26 18:39:22,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
184
[WARNING|trainer.py:803] 2025-04-26 18:39:22,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:23,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
199
154
[WARNING|trainer.py:803] 2025-04-26 18:39:24,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:25,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
185
[WARNING|trainer.py:803] 2025-04-26 18:39:26,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
200
155
[WARNING|trainer.py:803] 2025-04-26 18:39:27,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:27,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
186
[WARNING|trainer.py:803] 2025-04-26 18:39:28,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
156
201
[WARNING|trainer.py:803] 2025-04-26 18:39:29,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:29,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
187
157
[WARNING|trainer.py:803] 2025-04-26 18:39:31,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:39:31,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
202
[WARNING|trainer.py:803] 2025-04-26 18:39:32,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
188
158
203
[WARNING|trainer.py:803] 2025-04-26 18:39:34,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:34,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:35,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
159
189
[WARNING|trainer.py:803] 2025-04-26 18:39:36,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
204
[WARNING|trainer.py:803] 2025-04-26 18:39:37,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:37,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
160
190
[WARNING|trainer.py:803] 2025-04-26 18:39:39,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:39,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
205
[WARNING|trainer.py:803] 2025-04-26 18:39:40,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
161
191
[WARNING|trainer.py:803] 2025-04-26 18:39:41,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:39:42,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
206
162
[WARNING|trainer.py:803] 2025-04-26 18:39:43,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:39:43,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
192
[WARNING|trainer.py:803] 2025-04-26 18:39:44,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
207
163
[WARNING|trainer.py:803] 2025-04-26 18:39:45,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:46,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
193
[WARNING|trainer.py:803] 2025-04-26 18:39:47,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
208
164
[WARNING|trainer.py:803] 2025-04-26 18:39:48,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:39:48,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
194
[WARNING|trainer.py:803] 2025-04-26 18:39:50,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
165
209
[WARNING|trainer.py:803] 2025-04-26 18:39:51,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:51,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
195
210
166
[WARNING|trainer.py:803] 2025-04-26 18:39:52,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:53,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:53,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
211
196
[WARNING|trainer.py:803] 2025-04-26 18:39:54,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
167
[WARNING|trainer.py:803] 2025-04-26 18:39:55,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:55,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
212
[WARNING|trainer.py:803] 2025-04-26 18:39:56,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
197
168
213
[WARNING|trainer.py:803] 2025-04-26 18:39:58,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:58,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:39:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
198
214
[WARNING|trainer.py:803] 2025-04-26 18:40:00,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:00,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
169
[WARNING|trainer.py:803] 2025-04-26 18:40:01,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
215
199
[WARNING|trainer.py:803] 2025-04-26 18:40:02,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:40:02,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
170
216
[WARNING|trainer.py:803] 2025-04-26 18:40:03,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
200
[WARNING|trainer.py:803] 2025-04-26 18:40:04,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
171
[WARNING|trainer.py:803] 2025-04-26 18:40:05,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
217
[WARNING|trainer.py:803] 2025-04-26 18:40:05,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:06,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
172
201
[WARNING|trainer.py:803] 2025-04-26 18:40:07,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
218
[WARNING|trainer.py:803] 2025-04-26 18:40:08,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:08,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
173
219
[WARNING|trainer.py:803] 2025-04-26 18:40:10,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
202
[WARNING|trainer.py:803] 2025-04-26 18:40:10,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:11,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
174
220
[WARNING|trainer.py:803] 2025-04-26 18:40:12,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:12,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
203
175
[WARNING|trainer.py:803] 2025-04-26 18:40:13,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
221
[WARNING|trainer.py:803] 2025-04-26 18:40:14,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:14,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
204
176
222
[WARNING|trainer.py:803] 2025-04-26 18:40:16,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:16,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:16,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
223
205
177
[WARNING|trainer.py:803] 2025-04-26 18:40:18,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:19,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:19,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
224
206
[WARNING|trainer.py:803] 2025-04-26 18:40:20,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
178
[WARNING|trainer.py:803] 2025-04-26 18:40:21,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:21,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
225
[WARNING|trainer.py:803] 2025-04-26 18:40:22,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
207
179
226
[WARNING|trainer.py:803] 2025-04-26 18:40:24,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:24,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:24,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
227
208
180
[WARNING|trainer.py:803] 2025-04-26 18:40:26,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:27,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:40:27,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
228
181
209
[WARNING|trainer.py:803] 2025-04-26 18:40:28,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:29,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:40:29,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
229
210
[WARNING|trainer.py:803] 2025-04-26 18:40:30,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
182
[WARNING|trainer.py:803] 2025-04-26 18:40:31,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:31,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
230
211
[WARNING|trainer.py:803] 2025-04-26 18:40:32,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
183
[WARNING|trainer.py:803] 2025-04-26 18:40:33,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
231
[WARNING|trainer.py:803] 2025-04-26 18:40:34,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
212
[WARNING|trainer.py:803] 2025-04-26 18:40:34,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
184
[WARNING|trainer.py:803] 2025-04-26 18:40:35,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
232
[WARNING|trainer.py:803] 2025-04-26 18:40:36,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
213
[WARNING|trainer.py:803] 2025-04-26 18:40:36,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:37,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
233
185
214
[WARNING|trainer.py:803] 2025-04-26 18:40:38,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:38,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:39,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
234
186
215
[WARNING|trainer.py:803] 2025-04-26 18:40:40,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:41,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:41,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
235
187
216
[WARNING|trainer.py:803] 2025-04-26 18:40:42,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:43,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:40:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
236
[WARNING|trainer.py:803] 2025-04-26 18:40:44,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
217
[WARNING|trainer.py:803] 2025-04-26 18:40:45,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
188
237
[WARNING|trainer.py:803] 2025-04-26 18:40:46,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:46,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
218
[WARNING|trainer.py:803] 2025-04-26 18:40:47,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
238
189
[WARNING|trainer.py:803] 2025-04-26 18:40:48,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
219
[WARNING|trainer.py:803] 2025-04-26 18:40:49,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:49,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
239
190
[WARNING|trainer.py:803] 2025-04-26 18:40:50,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
220
[WARNING|trainer.py:803] 2025-04-26 18:40:51,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:40:51,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
240
[WARNING|trainer.py:803] 2025-04-26 18:40:52,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
221
191
[WARNING|trainer.py:803] 2025-04-26 18:40:53,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:40:53,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
241
[WARNING|trainer.py:803] 2025-04-26 18:40:54,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
222
192
[WARNING|trainer.py:803] 2025-04-26 18:40:55,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
242
[WARNING|trainer.py:803] 2025-04-26 18:40:56,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:40:56,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
223
193
[WARNING|trainer.py:803] 2025-04-26 18:40:57,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
243
[WARNING|trainer.py:803] 2025-04-26 18:40:58,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:40:58,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
224
[WARNING|trainer.py:803] 2025-04-26 18:40:59,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
244
194
[WARNING|trainer.py:803] 2025-04-26 18:41:00,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:00,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
225
245
[WARNING|trainer.py:803] 2025-04-26 18:41:01,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
195
[WARNING|trainer.py:803] 2025-04-26 18:41:02,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
226
[WARNING|trainer.py:803] 2025-04-26 18:41:03,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
246
[WARNING|trainer.py:803] 2025-04-26 18:41:03,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
196
[WARNING|trainer.py:803] 2025-04-26 18:41:04,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
227
[WARNING|trainer.py:803] 2025-04-26 18:41:05,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
247
[WARNING|trainer.py:803] 2025-04-26 18:41:05,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:06,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
228
197
248
[WARNING|trainer.py:803] 2025-04-26 18:41:07,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:41:07,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:08,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
229
198
249
[WARNING|trainer.py:803] 2025-04-26 18:41:09,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:09,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:41:10,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
230
199
250
[WARNING|trainer.py:803] 2025-04-26 18:41:11,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:12,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:12,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
231
251
[WARNING|trainer.py:803] 2025-04-26 18:41:13,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
200
[WARNING|trainer.py:803] 2025-04-26 18:41:14,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:14,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
232
[WARNING|trainer.py:803] 2025-04-26 18:41:15,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
252
201
[WARNING|trainer.py:803] 2025-04-26 18:41:16,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
233
[WARNING|trainer.py:803] 2025-04-26 18:41:17,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:17,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
253
[WARNING|trainer.py:803] 2025-04-26 18:41:18,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
234
202
[WARNING|trainer.py:803] 2025-04-26 18:41:19,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
254
[WARNING|trainer.py:803] 2025-04-26 18:41:19,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:20,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
235
203
[WARNING|trainer.py:803] 2025-04-26 18:41:21,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
255
[WARNING|trainer.py:803] 2025-04-26 18:41:21,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:22,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
236
[WARNING|trainer.py:803] 2025-04-26 18:41:23,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
204
256
[WARNING|trainer.py:803] 2025-04-26 18:41:24,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
237
[WARNING|trainer.py:803] 2025-04-26 18:41:24,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:25,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
205
257
238
[WARNING|trainer.py:803] 2025-04-26 18:41:26,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:26,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:27,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
239
206
258
[WARNING|trainer.py:803] 2025-04-26 18:41:29,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:29,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:29,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
240
207
259
[WARNING|trainer.py:803] 2025-04-26 18:41:31,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:31,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:31,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
241
260
208
[WARNING|trainer.py:803] 2025-04-26 18:41:33,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:33,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:33,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
242
209
[WARNING|trainer.py:803] 2025-04-26 18:41:35,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
261
[WARNING|trainer.py:803] 2025-04-26 18:41:35,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:36,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
243
210
[WARNING|trainer.py:803] 2025-04-26 18:41:37,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
262
[WARNING|trainer.py:803] 2025-04-26 18:41:37,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:38,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
244
211
[WARNING|trainer.py:803] 2025-04-26 18:41:38,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
263
[WARNING|trainer.py:803] 2025-04-26 18:41:39,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:40,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
245
212
[WARNING|trainer.py:803] 2025-04-26 18:41:40,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
264
[WARNING|trainer.py:803] 2025-04-26 18:41:41,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:41:41,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
213
246
[WARNING|trainer.py:803] 2025-04-26 18:41:42,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:41:42,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
265
214
247
[WARNING|trainer.py:803] 2025-04-26 18:41:44,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:44,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:44,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
215
266
248
[WARNING|trainer.py:803] 2025-04-26 18:41:46,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:41:46,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:46,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
216
267
249
[WARNING|trainer.py:803] 2025-04-26 18:41:48,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:41:48,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:48,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
217
[WARNING|trainer.py:803] 2025-04-26 18:41:50,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
268
250
218
[WARNING|trainer.py:803] 2025-04-26 18:41:51,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:51,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:41:51,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
251
269
219
[WARNING|trainer.py:803] 2025-04-26 18:41:53,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:53,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:53,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
252
220
[WARNING|trainer.py:803] 2025-04-26 18:41:54,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
270
[WARNING|trainer.py:803] 2025-04-26 18:41:55,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:41:55,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
253
221
[WARNING|trainer.py:803] 2025-04-26 18:41:56,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:57,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
271
254
[WARNING|trainer.py:803] 2025-04-26 18:41:58,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
222
[WARNING|trainer.py:803] 2025-04-26 18:41:58,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:41:59,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
272
255
223
[WARNING|trainer.py:803] 2025-04-26 18:42:00,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:00,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:01,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
224
273
256
[WARNING|trainer.py:803] 2025-04-26 18:42:02,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:03,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:03,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
225
[WARNING|trainer.py:803] 2025-04-26 18:42:04,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
257
274
[WARNING|trainer.py:803] 2025-04-26 18:42:05,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:05,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
226
[WARNING|trainer.py:803] 2025-04-26 18:42:06,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
258
227
275
[WARNING|trainer.py:803] 2025-04-26 18:42:08,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:08,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:08,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
228
259
276
[WARNING|trainer.py:803] 2025-04-26 18:42:10,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:10,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:10,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
229
260
[WARNING|trainer.py:803] 2025-04-26 18:42:11,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
277
[WARNING|trainer.py:803] 2025-04-26 18:42:12,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
230
[WARNING|trainer.py:803] 2025-04-26 18:42:13,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:13,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
261
231
278
[WARNING|trainer.py:803] 2025-04-26 18:42:14,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:15,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:15,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
262
232
[WARNING|trainer.py:803] 2025-04-26 18:42:16,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
279
[WARNING|trainer.py:803] 2025-04-26 18:42:17,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:17,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
233
263
[WARNING|trainer.py:803] 2025-04-26 18:42:18,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:18,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
280
234
264
[WARNING|trainer.py:803] 2025-04-26 18:42:20,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:20,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:20,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
235
281
265
[WARNING|trainer.py:803] 2025-04-26 18:42:22,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:22,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:23,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
236
[WARNING|trainer.py:803] 2025-04-26 18:42:24,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
282
266
237
[WARNING|trainer.py:803] 2025-04-26 18:42:25,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:25,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:25,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
283
238
267
[WARNING|trainer.py:803] 2025-04-26 18:42:27,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:27,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:27,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
239
268
284
[WARNING|trainer.py:803] 2025-04-26 18:42:29,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:30,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:30,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
240
[WARNING|trainer.py:803] 2025-04-26 18:42:31,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
269
285
241
[WARNING|trainer.py:803] 2025-04-26 18:42:32,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:32,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:33,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
270
242
286
[WARNING|trainer.py:803] 2025-04-26 18:42:34,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:34,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:35,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
243
271
[WARNING|trainer.py:803] 2025-04-26 18:42:36,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:37,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
244
287
[WARNING|trainer.py:803] 2025-04-26 18:42:38,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:38,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
272
245
[WARNING|trainer.py:803] 2025-04-26 18:42:39,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
288
[WARNING|trainer.py:803] 2025-04-26 18:42:40,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:40,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
246
273
[WARNING|trainer.py:803] 2025-04-26 18:42:42,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:42:42,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
289
247
[WARNING|trainer.py:803] 2025-04-26 18:42:43,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:42:43,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
274
[WARNING|trainer.py:803] 2025-04-26 18:42:44,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
248
290
[WARNING|trainer.py:803] 2025-04-26 18:42:45,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:42:45,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
275
249
[WARNING|trainer.py:803] 2025-04-26 18:42:47,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:47,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
291
[WARNING|trainer.py:803] 2025-04-26 18:42:48,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
250
276
[WARNING|trainer.py:803] 2025-04-26 18:42:49,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:42:49,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
292
251
[WARNING|trainer.py:803] 2025-04-26 18:42:50,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:51,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
277
252
[WARNING|trainer.py:803] 2025-04-26 18:42:52,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
293
[WARNING|trainer.py:803] 2025-04-26 18:42:52,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:53,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
278
253
[WARNING|trainer.py:803] 2025-04-26 18:42:54,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:54,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
294
254
[WARNING|trainer.py:803] 2025-04-26 18:42:55,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
279
[WARNING|trainer.py:803] 2025-04-26 18:42:56,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:56,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
255
295
[WARNING|trainer.py:803] 2025-04-26 18:42:58,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
280
[WARNING|trainer.py:803] 2025-04-26 18:42:58,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:42:59,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
256
296
[WARNING|trainer.py:803] 2025-04-26 18:43:00,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
281
[WARNING|trainer.py:803] 2025-04-26 18:43:00,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
257
[WARNING|trainer.py:803] 2025-04-26 18:43:01,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:02,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
297
[WARNING|trainer.py:803] 2025-04-26 18:43:03,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
282
258
[WARNING|trainer.py:803] 2025-04-26 18:43:04,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:04,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
298
283
[WARNING|trainer.py:803] 2025-04-26 18:43:05,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
259
[WARNING|trainer.py:803] 2025-04-26 18:43:06,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:06,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
260
299
[WARNING|trainer.py:803] 2025-04-26 18:43:08,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
284
[WARNING|trainer.py:803] 2025-04-26 18:43:08,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:09,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
261
300
[WARNING|trainer.py:803] 2025-04-26 18:43:10,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:43:10,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
285
301
262
[WARNING|trainer.py:803] 2025-04-26 18:43:11,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:11,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:12,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
302
263
[WARNING|trainer.py:803] 2025-04-26 18:43:13,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
286
[WARNING|trainer.py:803] 2025-04-26 18:43:13,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
303
[WARNING|trainer.py:803] 2025-04-26 18:43:14,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:14,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
264
304
[WARNING|trainer.py:803] 2025-04-26 18:43:15,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:16,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
305
287
265
[WARNING|trainer.py:803] 2025-04-26 18:43:17,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:17,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:17,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
306
[WARNING|trainer.py:803] 2025-04-26 18:43:18,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
266
288
307
[WARNING|trainer.py:803] 2025-04-26 18:43:19,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:19,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:20,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
308
267
289
[WARNING|trainer.py:803] 2025-04-26 18:43:21,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:21,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
309
[WARNING|trainer.py:803] 2025-04-26 18:43:22,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:43:22,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
268
310
[WARNING|trainer.py:803] 2025-04-26 18:43:23,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:24,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
290
311
[WARNING|trainer.py:803] 2025-04-26 18:43:25,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
269
[WARNING|trainer.py:803] 2025-04-26 18:43:25,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
312
[WARNING|trainer.py:803] 2025-04-26 18:43:25,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
291
[WARNING|trainer.py:803] 2025-04-26 18:43:26,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
270
313
[WARNING|trainer.py:803] 2025-04-26 18:43:27,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:28,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:28,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
314
292
271
[WARNING|trainer.py:803] 2025-04-26 18:43:29,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
315
[WARNING|trainer.py:803] 2025-04-26 18:43:29,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:30,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:30,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
316
293
272
[WARNING|trainer.py:803] 2025-04-26 18:43:32,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:32,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:32,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
317
[WARNING|trainer.py:803] 2025-04-26 18:43:33,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
273
318
294
[WARNING|trainer.py:803] 2025-04-26 18:43:34,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:34,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:35,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
319
274
[WARNING|trainer.py:803] 2025-04-26 18:43:36,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
320
[WARNING|trainer.py:803] 2025-04-26 18:43:36,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
295
[WARNING|trainer.py:803] 2025-04-26 18:43:37,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:37,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
321
275
[WARNING|trainer.py:803] 2025-04-26 18:43:38,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:39,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
296
322
[WARNING|trainer.py:803] 2025-04-26 18:43:39,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:40,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
276
323
[WARNING|trainer.py:803] 2025-04-26 18:43:41,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:41,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
297
324
277
[WARNING|trainer.py:803] 2025-04-26 18:43:42,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:42,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:43:43,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
325
298
[WARNING|trainer.py:803] 2025-04-26 18:43:44,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
278
326
[WARNING|trainer.py:803] 2025-04-26 18:43:44,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:45,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:45,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
327
279
299
[WARNING|trainer.py:803] 2025-04-26 18:43:46,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:47,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
328
[WARNING|trainer.py:803] 2025-04-26 18:43:47,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:43:48,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
280
329
300
[WARNING|trainer.py:803] 2025-04-26 18:43:49,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:49,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
330
301
281
[WARNING|trainer.py:803] 2025-04-26 18:43:51,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:51,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
331
[WARNING|trainer.py:803] 2025-04-26 18:43:51,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
302
[WARNING|trainer.py:803] 2025-04-26 18:43:52,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:43:52,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
332
282
303
[WARNING|trainer.py:803] 2025-04-26 18:43:53,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:43:53,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:54,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
333
304
[WARNING|trainer.py:803] 2025-04-26 18:43:55,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
283
[WARNING|trainer.py:803] 2025-04-26 18:43:55,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
334
305
[WARNING|trainer.py:803] 2025-04-26 18:43:56,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:56,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:56,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
335
306
284
[WARNING|trainer.py:803] 2025-04-26 18:43:57,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:43:58,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
336
[WARNING|trainer.py:803] 2025-04-26 18:43:58,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
307
[WARNING|trainer.py:803] 2025-04-26 18:43:59,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:43:59,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
337
285
308
[WARNING|trainer.py:803] 2025-04-26 18:44:00,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:00,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:00,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
338
309
[WARNING|trainer.py:803] 2025-04-26 18:44:01,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
339
[WARNING|trainer.py:803] 2025-04-26 18:44:02,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
286
310
[WARNING|trainer.py:803] 2025-04-26 18:44:03,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:03,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:03,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
340
311
[WARNING|trainer.py:803] 2025-04-26 18:44:04,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
341
[WARNING|trainer.py:803] 2025-04-26 18:44:05,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
287
312
[WARNING|trainer.py:803] 2025-04-26 18:44:05,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:05,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:06,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
342
313
288
[WARNING|trainer.py:803] 2025-04-26 18:44:07,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
343
[WARNING|trainer.py:803] 2025-04-26 18:44:07,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:07,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
314
[WARNING|trainer.py:803] 2025-04-26 18:44:08,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:09,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
344
289
315
[WARNING|trainer.py:803] 2025-04-26 18:44:09,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:10,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:44:10,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
345
316
[WARNING|trainer.py:803] 2025-04-26 18:44:11,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:11,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
290
346
317
[WARNING|trainer.py:803] 2025-04-26 18:44:12,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:12,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:13,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
347
318
291
[WARNING|trainer.py:803] 2025-04-26 18:44:14,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:14,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
348
[WARNING|trainer.py:803] 2025-04-26 18:44:14,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
319
[WARNING|trainer.py:803] 2025-04-26 18:44:15,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:15,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
349
292
320
[WARNING|trainer.py:803] 2025-04-26 18:44:16,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:16,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:17,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
350
321
293
[WARNING|trainer.py:803] 2025-04-26 18:44:18,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:18,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
351
[WARNING|trainer.py:803] 2025-04-26 18:44:18,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
322
[WARNING|trainer.py:803] 2025-04-26 18:44:19,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:19,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
352
323
294
[WARNING|trainer.py:803] 2025-04-26 18:44:20,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:21,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
353
[WARNING|trainer.py:803] 2025-04-26 18:44:21,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
324
[WARNING|trainer.py:803] 2025-04-26 18:44:22,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
354
[WARNING|trainer.py:803] 2025-04-26 18:44:22,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
295
325
[WARNING|trainer.py:803] 2025-04-26 18:44:23,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:23,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
355
[WARNING|trainer.py:803] 2025-04-26 18:44:24,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
326
296
[WARNING|trainer.py:803] 2025-04-26 18:44:24,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
356
[WARNING|trainer.py:803] 2025-04-26 18:44:25,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:25,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
327
[WARNING|trainer.py:803] 2025-04-26 18:44:26,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
357
[WARNING|trainer.py:803] 2025-04-26 18:44:26,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
297
328
[WARNING|trainer.py:803] 2025-04-26 18:44:27,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:27,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
358
[WARNING|trainer.py:803] 2025-04-26 18:44:28,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
329
[WARNING|trainer.py:803] 2025-04-26 18:44:28,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
298
359
[WARNING|trainer.py:803] 2025-04-26 18:44:29,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:44:30,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:30,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
330
360
[WARNING|trainer.py:803] 2025-04-26 18:44:31,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:44:31,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
331
299
361
[WARNING|trainer.py:803] 2025-04-26 18:44:32,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:32,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:32,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
332
362
300
[WARNING|trainer.py:803] 2025-04-26 18:44:33,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:34,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
333
363
[WARNING|trainer.py:803] 2025-04-26 18:44:34,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
301
[WARNING|trainer.py:803] 2025-04-26 18:44:35,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:35,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
334
[WARNING|trainer.py:803] 2025-04-26 18:44:35,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
364
302
[WARNING|trainer.py:803] 2025-04-26 18:44:36,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:36,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:36,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
335
365
303
[WARNING|trainer.py:803] 2025-04-26 18:44:37,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:44:38,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:38,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
336
304
366
[WARNING|trainer.py:803] 2025-04-26 18:44:39,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:39,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:44:39,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
305
337
367
[WARNING|trainer.py:803] 2025-04-26 18:44:40,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:40,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:40,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
306
338
368
[WARNING|trainer.py:803] 2025-04-26 18:44:41,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
307
[WARNING|trainer.py:803] 2025-04-26 18:44:42,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:42,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
339
369
[WARNING|trainer.py:803] 2025-04-26 18:44:42,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
308
[WARNING|trainer.py:803] 2025-04-26 18:44:43,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:43,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
340
[WARNING|trainer.py:803] 2025-04-26 18:44:43,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
370
309
[WARNING|trainer.py:803] 2025-04-26 18:44:44,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:44,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:45,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
341
310
371
[WARNING|trainer.py:803] 2025-04-26 18:44:46,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:46,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:46,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
342
311
372
[WARNING|trainer.py:803] 2025-04-26 18:44:47,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:47,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
312
[WARNING|trainer.py:803] 2025-04-26 18:44:47,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
343
373
[WARNING|trainer.py:803] 2025-04-26 18:44:48,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:48,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
313
[WARNING|trainer.py:803] 2025-04-26 18:44:49,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
344
374
[WARNING|trainer.py:803] 2025-04-26 18:44:49,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
314
[WARNING|trainer.py:803] 2025-04-26 18:44:50,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:50,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
345
[WARNING|trainer.py:803] 2025-04-26 18:44:50,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
375
315
[WARNING|trainer.py:803] 2025-04-26 18:44:51,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:51,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:51,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
346
316
376
[WARNING|trainer.py:803] 2025-04-26 18:44:52,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:53,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:53,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
317
347
377
[WARNING|trainer.py:803] 2025-04-26 18:44:54,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:54,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
318
[WARNING|trainer.py:803] 2025-04-26 18:44:54,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
348
378
[WARNING|trainer.py:803] 2025-04-26 18:44:55,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:44:55,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
319
[WARNING|trainer.py:803] 2025-04-26 18:44:55,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
349
379
[WARNING|trainer.py:803] 2025-04-26 18:44:56,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
320
[WARNING|trainer.py:803] 2025-04-26 18:44:56,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:57,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
350
[WARNING|trainer.py:803] 2025-04-26 18:44:57,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
380
321
[WARNING|trainer.py:803] 2025-04-26 18:44:58,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:58,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:58,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
351
381
322
[WARNING|trainer.py:803] 2025-04-26 18:44:59,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:44:59,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:44:59,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
352
323
382
[WARNING|trainer.py:803] 2025-04-26 18:45:01,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:01,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:01,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
324
353
383
[WARNING|trainer.py:803] 2025-04-26 18:45:02,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:02,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:02,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
325
354
384
[WARNING|trainer.py:803] 2025-04-26 18:45:03,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
326
[WARNING|trainer.py:803] 2025-04-26 18:45:03,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:04,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
355
385
[WARNING|trainer.py:803] 2025-04-26 18:45:04,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
327
[WARNING|trainer.py:803] 2025-04-26 18:45:05,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:05,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
356
[WARNING|trainer.py:803] 2025-04-26 18:45:05,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
386
328
[WARNING|trainer.py:803] 2025-04-26 18:45:06,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:06,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
357
[WARNING|trainer.py:803] 2025-04-26 18:45:07,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
329
387
[WARNING|trainer.py:803] 2025-04-26 18:45:07,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:08,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:08,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
358
330
388
[WARNING|trainer.py:803] 2025-04-26 18:45:09,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:09,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:09,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
359
331
389
[WARNING|trainer.py:803] 2025-04-26 18:45:10,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:10,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:10,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
332
360
390
[WARNING|trainer.py:803] 2025-04-26 18:45:11,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:12,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
333
[WARNING|trainer.py:803] 2025-04-26 18:45:12,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
361
391
[WARNING|trainer.py:803] 2025-04-26 18:45:13,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:13,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
334
[WARNING|trainer.py:803] 2025-04-26 18:45:13,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
362
392
[WARNING|trainer.py:803] 2025-04-26 18:45:14,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:14,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
335
[WARNING|trainer.py:803] 2025-04-26 18:45:15,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
363
[WARNING|trainer.py:803] 2025-04-26 18:45:15,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
393
336
[WARNING|trainer.py:803] 2025-04-26 18:45:15,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:16,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
364
[WARNING|trainer.py:803] 2025-04-26 18:45:16,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
394
337
[WARNING|trainer.py:803] 2025-04-26 18:45:17,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:17,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:17,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
365
338
395
[WARNING|trainer.py:803] 2025-04-26 18:45:18,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:18,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:19,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
339
366
396
[WARNING|trainer.py:803] 2025-04-26 18:45:19,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:19,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
340
[WARNING|trainer.py:803] 2025-04-26 18:45:20,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
367
397
[WARNING|trainer.py:803] 2025-04-26 18:45:21,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:21,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
341
[WARNING|trainer.py:803] 2025-04-26 18:45:21,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
368
[WARNING|trainer.py:803] 2025-04-26 18:45:22,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
398
[WARNING|trainer.py:803] 2025-04-26 18:45:22,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
342
369
[WARNING|trainer.py:803] 2025-04-26 18:45:23,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:23,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
399
343
[WARNING|trainer.py:803] 2025-04-26 18:45:24,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:24,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
370
[WARNING|trainer.py:803] 2025-04-26 18:45:24,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
344
400
[WARNING|trainer.py:803] 2025-04-26 18:45:25,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:25,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:25,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
371
345
401
[WARNING|trainer.py:803] 2025-04-26 18:45:26,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:26,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:27,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
372
346
402
[WARNING|trainer.py:803] 2025-04-26 18:45:28,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:28,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
347
[WARNING|trainer.py:803] 2025-04-26 18:45:28,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
373
403
[WARNING|trainer.py:803] 2025-04-26 18:45:29,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:29,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
348
[WARNING|trainer.py:803] 2025-04-26 18:45:30,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
374
404
[WARNING|trainer.py:803] 2025-04-26 18:45:30,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
349
[WARNING|trainer.py:803] 2025-04-26 18:45:30,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:31,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
375
[WARNING|trainer.py:803] 2025-04-26 18:45:31,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
405
350
[WARNING|trainer.py:803] 2025-04-26 18:45:32,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:32,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
376
[WARNING|trainer.py:803] 2025-04-26 18:45:32,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
351
406
[WARNING|trainer.py:803] 2025-04-26 18:45:33,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:34,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:34,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
377
352
407
[WARNING|trainer.py:803] 2025-04-26 18:45:35,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:35,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:35,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
378
353
408
[WARNING|trainer.py:803] 2025-04-26 18:45:36,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:36,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
354
[WARNING|trainer.py:803] 2025-04-26 18:45:36,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
379
409
[WARNING|trainer.py:803] 2025-04-26 18:45:37,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:37,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
355
[WARNING|trainer.py:803] 2025-04-26 18:45:38,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
380
410
[WARNING|trainer.py:803] 2025-04-26 18:45:38,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
356
[WARNING|trainer.py:803] 2025-04-26 18:45:39,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:39,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
381
[WARNING|trainer.py:803] 2025-04-26 18:45:39,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
411
357
[WARNING|trainer.py:803] 2025-04-26 18:45:40,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:40,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
382
[WARNING|trainer.py:803] 2025-04-26 18:45:41,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
358
412
[WARNING|trainer.py:803] 2025-04-26 18:45:41,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:42,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:42,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
383
359
413
[WARNING|trainer.py:803] 2025-04-26 18:45:43,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:43,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:43,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
360
384
414
[WARNING|trainer.py:803] 2025-04-26 18:45:44,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:44,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
361
[WARNING|trainer.py:803] 2025-04-26 18:45:44,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
385
415
[WARNING|trainer.py:803] 2025-04-26 18:45:45,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
362
[WARNING|trainer.py:803] 2025-04-26 18:45:46,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:46,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
386
[WARNING|trainer.py:803] 2025-04-26 18:45:46,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
416
363
[WARNING|trainer.py:803] 2025-04-26 18:45:47,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:47,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:47,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
387
364
417
[WARNING|trainer.py:803] 2025-04-26 18:45:48,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:48,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:49,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
388
365
418
[WARNING|trainer.py:803] 2025-04-26 18:45:50,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:50,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
366
[WARNING|trainer.py:803] 2025-04-26 18:45:50,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
389
419
[WARNING|trainer.py:803] 2025-04-26 18:45:51,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:51,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
367
[WARNING|trainer.py:803] 2025-04-26 18:45:51,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
390
[WARNING|trainer.py:803] 2025-04-26 18:45:52,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
420
368
[WARNING|trainer.py:803] 2025-04-26 18:45:52,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
391
[WARNING|trainer.py:803] 2025-04-26 18:45:53,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:45:53,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
369
421
[WARNING|trainer.py:803] 2025-04-26 18:45:54,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:54,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:54,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
392
370
422
[WARNING|trainer.py:803] 2025-04-26 18:45:55,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:45:55,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
371
[WARNING|trainer.py:803] 2025-04-26 18:45:56,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
393
423
[WARNING|trainer.py:803] 2025-04-26 18:45:56,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:45:56,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
372
[WARNING|trainer.py:803] 2025-04-26 18:45:57,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
394
424
[WARNING|trainer.py:803] 2025-04-26 18:45:58,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:45:58,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
373
[WARNING|trainer.py:803] 2025-04-26 18:45:58,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
395
[WARNING|trainer.py:803] 2025-04-26 18:45:59,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
425
374
[WARNING|trainer.py:803] 2025-04-26 18:45:59,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
396
[WARNING|trainer.py:803] 2025-04-26 18:46:00,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:00,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
375
426
[WARNING|trainer.py:803] 2025-04-26 18:46:01,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:01,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
397
[WARNING|trainer.py:803] 2025-04-26 18:46:01,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
376
427
[WARNING|trainer.py:803] 2025-04-26 18:46:02,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:02,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
377
398
[WARNING|trainer.py:803] 2025-04-26 18:46:03,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
428
[WARNING|trainer.py:803] 2025-04-26 18:46:03,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:03,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
378
399
[WARNING|trainer.py:803] 2025-04-26 18:46:04,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:04,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
429
379
[WARNING|trainer.py:803] 2025-04-26 18:46:05,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:05,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
400
[WARNING|trainer.py:803] 2025-04-26 18:46:05,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
430
380
[WARNING|trainer.py:803] 2025-04-26 18:46:06,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:07,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
401
[WARNING|trainer.py:803] 2025-04-26 18:46:07,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
381
431
[WARNING|trainer.py:803] 2025-04-26 18:46:08,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:08,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:08,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
402
382
432
[WARNING|trainer.py:803] 2025-04-26 18:46:09,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:09,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:09,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
383
403
433
[WARNING|trainer.py:803] 2025-04-26 18:46:10,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:10,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:11,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
384
404
434
[WARNING|trainer.py:803] 2025-04-26 18:46:11,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:12,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
385
[WARNING|trainer.py:803] 2025-04-26 18:46:12,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
405
435
[WARNING|trainer.py:803] 2025-04-26 18:46:13,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:13,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
386
[WARNING|trainer.py:803] 2025-04-26 18:46:13,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
406
[WARNING|trainer.py:803] 2025-04-26 18:46:14,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
436
387
[WARNING|trainer.py:803] 2025-04-26 18:46:14,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:15,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
407
[WARNING|trainer.py:803] 2025-04-26 18:46:15,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
437
388
[WARNING|trainer.py:803] 2025-04-26 18:46:16,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:16,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:16,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
408
389
438
[WARNING|trainer.py:803] 2025-04-26 18:46:17,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:17,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:17,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
390
409
439
[WARNING|trainer.py:803] 2025-04-26 18:46:18,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:19,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
391
[WARNING|trainer.py:803] 2025-04-26 18:46:19,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
410
440
[WARNING|trainer.py:803] 2025-04-26 18:46:20,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:20,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
392
[WARNING|trainer.py:803] 2025-04-26 18:46:20,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
411
441
[WARNING|trainer.py:803] 2025-04-26 18:46:21,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
393
[WARNING|trainer.py:803] 2025-04-26 18:46:21,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:21,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
412
[WARNING|trainer.py:803] 2025-04-26 18:46:22,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
442
394
[WARNING|trainer.py:803] 2025-04-26 18:46:23,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:23,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:23,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
413
395
443
[WARNING|trainer.py:803] 2025-04-26 18:46:24,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:24,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:24,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
414
396
444
[WARNING|trainer.py:803] 2025-04-26 18:46:25,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:25,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
397
[WARNING|trainer.py:803] 2025-04-26 18:46:26,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
415
445
[WARNING|trainer.py:803] 2025-04-26 18:46:27,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:27,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
398
[WARNING|trainer.py:803] 2025-04-26 18:46:27,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
416
446
[WARNING|trainer.py:803] 2025-04-26 18:46:28,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
399
[WARNING|trainer.py:803] 2025-04-26 18:46:28,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:28,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
417
[WARNING|trainer.py:803] 2025-04-26 18:46:29,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
447
400
[WARNING|trainer.py:803] 2025-04-26 18:46:30,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:30,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:30,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
418
401
448
[WARNING|trainer.py:803] 2025-04-26 18:46:31,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:31,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:31,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
419
402
449
[WARNING|trainer.py:803] 2025-04-26 18:46:32,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:32,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:33,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
403
420
450
[WARNING|trainer.py:803] 2025-04-26 18:46:34,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:34,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
404
[WARNING|trainer.py:803] 2025-04-26 18:46:34,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
421
451
[WARNING|trainer.py:803] 2025-04-26 18:46:35,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
405
[WARNING|trainer.py:803] 2025-04-26 18:46:35,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:35,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
422
452
[WARNING|trainer.py:803] 2025-04-26 18:46:36,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
406
[WARNING|trainer.py:803] 2025-04-26 18:46:36,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:37,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
423
[WARNING|trainer.py:803] 2025-04-26 18:46:37,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
453
407
[WARNING|trainer.py:803] 2025-04-26 18:46:38,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:38,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:38,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
424
454
408
[WARNING|trainer.py:803] 2025-04-26 18:46:39,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:39,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:39,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
409
425
455
[WARNING|trainer.py:803] 2025-04-26 18:46:41,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:41,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:41,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
410
426
456
[WARNING|trainer.py:803] 2025-04-26 18:46:42,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
411
[WARNING|trainer.py:803] 2025-04-26 18:46:42,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:42,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
457
427
[WARNING|trainer.py:803] 2025-04-26 18:46:43,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
412
[WARNING|trainer.py:803] 2025-04-26 18:46:43,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:43,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:44,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
428
458
413
[WARNING|trainer.py:803] 2025-04-26 18:46:45,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:45,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:45,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
459
429
414
[WARNING|trainer.py:803] 2025-04-26 18:46:46,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:46,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:46,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
415
430
460
[WARNING|trainer.py:803] 2025-04-26 18:46:47,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:48,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:48,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
416
431
461
[WARNING|trainer.py:803] 2025-04-26 18:46:49,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:49,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
417
[WARNING|trainer.py:803] 2025-04-26 18:46:49,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
432
462
[WARNING|trainer.py:803] 2025-04-26 18:46:50,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
418
[WARNING|trainer.py:803] 2025-04-26 18:46:50,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:50,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
433
463
[WARNING|trainer.py:803] 2025-04-26 18:46:51,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
419
[WARNING|trainer.py:803] 2025-04-26 18:46:52,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:52,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
434
[WARNING|trainer.py:803] 2025-04-26 18:46:52,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
464
420
[WARNING|trainer.py:803] 2025-04-26 18:46:53,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:53,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:46:53,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
435
465
421
[WARNING|trainer.py:803] 2025-04-26 18:46:54,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:55,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:46:55,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
436
422
466
[WARNING|trainer.py:803] 2025-04-26 18:46:56,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:56,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:56,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
423
437
467
[WARNING|trainer.py:803] 2025-04-26 18:46:57,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:57,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:46:57,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
424
438
468
[WARNING|trainer.py:803] 2025-04-26 18:46:58,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:46:59,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
425
[WARNING|trainer.py:803] 2025-04-26 18:46:59,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
439
469
[WARNING|trainer.py:803] 2025-04-26 18:47:00,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
426
[WARNING|trainer.py:803] 2025-04-26 18:47:00,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:00,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
440
470
[WARNING|trainer.py:803] 2025-04-26 18:47:01,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
427
[WARNING|trainer.py:803] 2025-04-26 18:47:01,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:01,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
441
[WARNING|trainer.py:803] 2025-04-26 18:47:02,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
471
428
[WARNING|trainer.py:803] 2025-04-26 18:47:03,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:03,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:03,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
442
472
429
[WARNING|trainer.py:803] 2025-04-26 18:47:04,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:04,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:04,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
430
443
473
[WARNING|trainer.py:803] 2025-04-26 18:47:05,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:05,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:06,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
431
474
444
[WARNING|trainer.py:803] 2025-04-26 18:47:07,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:07,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:07,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
432
475
445
[WARNING|trainer.py:803] 2025-04-26 18:47:08,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
433
[WARNING|trainer.py:803] 2025-04-26 18:47:08,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:08,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
446
476
[WARNING|trainer.py:803] 2025-04-26 18:47:09,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
434
[WARNING|trainer.py:803] 2025-04-26 18:47:10,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:10,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:10,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
477
447
435
[WARNING|trainer.py:803] 2025-04-26 18:47:11,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:11,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:11,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
478
448
436
[WARNING|trainer.py:803] 2025-04-26 18:47:12,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:12,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:13,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
437
479
449
[WARNING|trainer.py:803] 2025-04-26 18:47:14,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:14,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:14,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
438
480
450
[WARNING|trainer.py:803] 2025-04-26 18:47:15,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
439
[WARNING|trainer.py:803] 2025-04-26 18:47:15,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:15,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
481
451
[WARNING|trainer.py:803] 2025-04-26 18:47:16,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
440
[WARNING|trainer.py:803] 2025-04-26 18:47:16,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:17,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
482
[WARNING|trainer.py:803] 2025-04-26 18:47:17,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
452
441
[WARNING|trainer.py:803] 2025-04-26 18:47:18,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:18,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:18,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
483
453
442
[WARNING|trainer.py:803] 2025-04-26 18:47:19,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:19,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:19,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
484
443
454
[WARNING|trainer.py:803] 2025-04-26 18:47:21,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:21,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:21,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
444
485
455
[WARNING|trainer.py:803] 2025-04-26 18:47:22,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:22,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:22,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
445
486
456
[WARNING|trainer.py:803] 2025-04-26 18:47:23,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
446
[WARNING|trainer.py:803] 2025-04-26 18:47:23,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:23,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
487
457
[WARNING|trainer.py:803] 2025-04-26 18:47:24,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
447
[WARNING|trainer.py:803] 2025-04-26 18:47:25,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:25,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:25,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
488
458
448
[WARNING|trainer.py:803] 2025-04-26 18:47:26,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:26,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:26,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
489
449
459
[WARNING|trainer.py:803] 2025-04-26 18:47:27,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:28,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:28,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
450
490
460
[WARNING|trainer.py:803] 2025-04-26 18:47:29,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:29,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
451
[WARNING|trainer.py:803] 2025-04-26 18:47:29,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
491
461
[WARNING|trainer.py:803] 2025-04-26 18:47:30,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
452
[WARNING|trainer.py:803] 2025-04-26 18:47:30,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:31,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
492
[WARNING|trainer.py:803] 2025-04-26 18:47:31,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
462
453
[WARNING|trainer.py:803] 2025-04-26 18:47:32,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:32,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:32,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
493
454
463
[WARNING|trainer.py:803] 2025-04-26 18:47:33,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:33,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:33,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
494
455
464
[WARNING|trainer.py:803] 2025-04-26 18:47:34,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:35,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
456
[WARNING|trainer.py:803] 2025-04-26 18:47:35,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
495
465
[WARNING|trainer.py:803] 2025-04-26 18:47:36,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:36,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
457
[WARNING|trainer.py:803] 2025-04-26 18:47:36,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
496
466
[WARNING|trainer.py:803] 2025-04-26 18:47:37,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:37,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
458
[WARNING|trainer.py:803] 2025-04-26 18:47:38,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
497
[WARNING|trainer.py:803] 2025-04-26 18:47:38,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
467
459
[WARNING|trainer.py:803] 2025-04-26 18:47:39,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:39,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
498
[WARNING|trainer.py:803] 2025-04-26 18:47:39,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
468
460
[WARNING|trainer.py:803] 2025-04-26 18:47:40,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:40,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
499
[WARNING|trainer.py:803] 2025-04-26 18:47:40,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
461
469
[WARNING|trainer.py:803] 2025-04-26 18:47:41,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:42,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:42,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
500
462
470
[WARNING|trainer.py:803] 2025-04-26 18:47:43,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:43,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:43,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
463
501
471
[WARNING|trainer.py:803] 2025-04-26 18:47:44,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:44,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
464
[WARNING|trainer.py:803] 2025-04-26 18:47:44,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
502
472
[WARNING|trainer.py:803] 2025-04-26 18:47:45,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:45,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
465
[WARNING|trainer.py:803] 2025-04-26 18:47:46,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
503
473
[WARNING|trainer.py:803] 2025-04-26 18:47:46,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:47,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
466
[WARNING|trainer.py:803] 2025-04-26 18:47:47,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
504
474
[WARNING|trainer.py:803] 2025-04-26 18:47:47,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
467
[WARNING|trainer.py:803] 2025-04-26 18:47:48,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:48,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
505
[WARNING|trainer.py:803] 2025-04-26 18:47:49,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
475
468
[WARNING|trainer.py:803] 2025-04-26 18:47:50,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:50,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:50,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
506
469
476
[WARNING|trainer.py:803] 2025-04-26 18:47:51,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:51,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:51,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
470
507
477
[WARNING|trainer.py:803] 2025-04-26 18:47:52,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:52,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:52,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
471
508
478
[WARNING|trainer.py:803] 2025-04-26 18:47:53,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
472
[WARNING|trainer.py:803] 2025-04-26 18:47:54,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:54,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
509
479
[WARNING|trainer.py:803] 2025-04-26 18:47:54,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
473
[WARNING|trainer.py:803] 2025-04-26 18:47:55,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:55,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
510
[WARNING|trainer.py:803] 2025-04-26 18:47:55,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
480
474
[WARNING|trainer.py:803] 2025-04-26 18:47:56,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:56,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:47:57,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
511
475
481
[WARNING|trainer.py:803] 2025-04-26 18:47:58,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:47:58,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:47:58,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
476
512
482
[WARNING|trainer.py:803] 2025-04-26 18:47:59,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:47:59,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
477
[WARNING|trainer.py:803] 2025-04-26 18:47:59,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
513
483
[WARNING|trainer.py:803] 2025-04-26 18:48:00,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:00,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
478
[WARNING|trainer.py:803] 2025-04-26 18:48:01,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
514
484
[WARNING|trainer.py:803] 2025-04-26 18:48:01,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
479
[WARNING|trainer.py:803] 2025-04-26 18:48:02,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:02,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
515
[WARNING|trainer.py:803] 2025-04-26 18:48:02,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
485
480
[WARNING|trainer.py:803] 2025-04-26 18:48:03,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
516
[WARNING|trainer.py:803] 2025-04-26 18:48:03,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:04,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
481
486
[WARNING|trainer.py:803] 2025-04-26 18:48:04,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
517
[WARNING|trainer.py:803] 2025-04-26 18:48:05,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:05,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
482
487
[WARNING|trainer.py:803] 2025-04-26 18:48:05,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:06,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
518
[WARNING|trainer.py:803] 2025-04-26 18:48:06,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
483
488
[WARNING|trainer.py:803] 2025-04-26 18:48:07,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:07,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
519
484
[WARNING|trainer.py:803] 2025-04-26 18:48:07,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
489
[WARNING|trainer.py:803] 2025-04-26 18:48:08,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:08,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
485
520
[WARNING|trainer.py:803] 2025-04-26 18:48:09,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
490
[WARNING|trainer.py:803] 2025-04-26 18:48:09,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:09,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
486
521
[WARNING|trainer.py:803] 2025-04-26 18:48:10,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:10,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:11,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
491
487
522
[WARNING|trainer.py:803] 2025-04-26 18:48:12,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:12,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:12,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
488
492
523
[WARNING|trainer.py:803] 2025-04-26 18:48:13,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:13,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
489
[WARNING|trainer.py:803] 2025-04-26 18:48:13,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
493
524
[WARNING|trainer.py:803] 2025-04-26 18:48:14,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
490
[WARNING|trainer.py:803] 2025-04-26 18:48:15,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:15,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
494
525
[WARNING|trainer.py:803] 2025-04-26 18:48:15,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
491
[WARNING|trainer.py:803] 2025-04-26 18:48:16,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:16,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
495
[WARNING|trainer.py:803] 2025-04-26 18:48:16,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
526
492
[WARNING|trainer.py:803] 2025-04-26 18:48:17,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:17,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:18,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
527
496
493
[WARNING|trainer.py:803] 2025-04-26 18:48:19,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:19,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:19,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
528
494
497
[WARNING|trainer.py:803] 2025-04-26 18:48:20,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:20,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:20,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
529
495
498
[WARNING|trainer.py:803] 2025-04-26 18:48:21,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:21,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:22,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
496
530
499
[WARNING|trainer.py:803] 2025-04-26 18:48:22,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:23,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
497
[WARNING|trainer.py:803] 2025-04-26 18:48:23,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
531
500
[WARNING|trainer.py:803] 2025-04-26 18:48:24,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:24,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
498
[WARNING|trainer.py:803] 2025-04-26 18:48:24,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
532
501
[WARNING|trainer.py:803] 2025-04-26 18:48:25,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
499
[WARNING|trainer.py:803] 2025-04-26 18:48:25,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:26,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
533
[WARNING|trainer.py:803] 2025-04-26 18:48:26,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
502
500
[WARNING|trainer.py:803] 2025-04-26 18:48:26,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:27,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
534
[WARNING|trainer.py:803] 2025-04-26 18:48:27,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
501
503
[WARNING|trainer.py:803] 2025-04-26 18:48:28,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:28,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:28,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
535
502
504
[WARNING|trainer.py:803] 2025-04-26 18:48:29,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:29,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
503
536
[WARNING|trainer.py:803] 2025-04-26 18:48:30,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
505
[WARNING|trainer.py:803] 2025-04-26 18:48:30,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:30,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
504
[WARNING|trainer.py:803] 2025-04-26 18:48:31,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
537
506
[WARNING|trainer.py:803] 2025-04-26 18:48:32,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:32,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
505
[WARNING|trainer.py:803] 2025-04-26 18:48:32,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
538
507
[WARNING|trainer.py:803] 2025-04-26 18:48:33,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
506
[WARNING|trainer.py:803] 2025-04-26 18:48:33,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
539
[WARNING|trainer.py:803] 2025-04-26 18:48:34,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:34,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
508
507
[WARNING|trainer.py:803] 2025-04-26 18:48:34,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
540
[WARNING|trainer.py:803] 2025-04-26 18:48:35,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:35,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
508
509
[WARNING|trainer.py:803] 2025-04-26 18:48:36,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:36,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
541
[WARNING|trainer.py:803] 2025-04-26 18:48:36,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
509
510
[WARNING|trainer.py:803] 2025-04-26 18:48:37,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:37,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
542
510
[WARNING|trainer.py:803] 2025-04-26 18:48:38,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
511
[WARNING|trainer.py:803] 2025-04-26 18:48:38,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:38,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
511
543
[WARNING|trainer.py:803] 2025-04-26 18:48:39,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
512
[WARNING|trainer.py:803] 2025-04-26 18:48:39,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:40,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
512
544
[WARNING|trainer.py:803] 2025-04-26 18:48:40,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:41,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
513
513
[WARNING|trainer.py:803] 2025-04-26 18:48:41,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
545
[WARNING|trainer.py:803] 2025-04-26 18:48:42,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:42,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
514
514
[WARNING|trainer.py:803] 2025-04-26 18:48:42,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
546
[WARNING|trainer.py:803] 2025-04-26 18:48:43,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:48:43,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
515
515
[WARNING|trainer.py:803] 2025-04-26 18:48:44,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:44,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
547
516
[WARNING|trainer.py:803] 2025-04-26 18:48:44,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
516
[WARNING|trainer.py:803] 2025-04-26 18:48:45,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:48:45,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
517
548
[WARNING|trainer.py:803] 2025-04-26 18:48:46,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
517
[WARNING|trainer.py:803] 2025-04-26 18:48:46,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:46,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
518
549
[WARNING|trainer.py:803] 2025-04-26 18:48:47,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:47,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
518
[WARNING|trainer.py:803] 2025-04-26 18:48:48,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
519
550
[WARNING|trainer.py:803] 2025-04-26 18:48:48,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:48,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
520
519
[WARNING|trainer.py:803] 2025-04-26 18:48:49,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
551
[WARNING|trainer.py:803] 2025-04-26 18:48:50,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:50,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
521
[WARNING|trainer.py:803] 2025-04-26 18:48:50,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
520
[WARNING|trainer.py:803] 2025-04-26 18:48:51,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
552
522
[WARNING|trainer.py:803] 2025-04-26 18:48:51,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
521
[WARNING|trainer.py:803] 2025-04-26 18:48:51,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:52,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
553
523
[WARNING|trainer.py:803] 2025-04-26 18:48:52,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
522
[WARNING|trainer.py:803] 2025-04-26 18:48:53,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:53,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
524
554
[WARNING|trainer.py:803] 2025-04-26 18:48:54,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:54,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:54,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
523
525
555
[WARNING|trainer.py:803] 2025-04-26 18:48:55,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:55,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:55,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
526
524
556
[WARNING|trainer.py:803] 2025-04-26 18:48:56,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:56,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
527
[WARNING|trainer.py:803] 2025-04-26 18:48:57,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
525
557
[WARNING|trainer.py:803] 2025-04-26 18:48:57,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:48:58,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
528
[WARNING|trainer.py:803] 2025-04-26 18:48:58,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
526
[WARNING|trainer.py:803] 2025-04-26 18:48:58,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
558
529
[WARNING|trainer.py:803] 2025-04-26 18:48:59,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:48:59,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
527
[WARNING|trainer.py:803] 2025-04-26 18:49:00,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
530
559
[WARNING|trainer.py:803] 2025-04-26 18:49:00,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:01,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:01,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
528
531
560
[WARNING|trainer.py:803] 2025-04-26 18:49:02,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:02,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
532
[WARNING|trainer.py:803] 2025-04-26 18:49:02,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
529
561
[WARNING|trainer.py:803] 2025-04-26 18:49:03,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:03,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
533
[WARNING|trainer.py:803] 2025-04-26 18:49:03,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
530
562
[WARNING|trainer.py:803] 2025-04-26 18:49:04,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
534
[WARNING|trainer.py:803] 2025-04-26 18:49:04,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:05,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
531
[WARNING|trainer.py:803] 2025-04-26 18:49:05,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
563
535
[WARNING|trainer.py:803] 2025-04-26 18:49:06,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:06,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:49:06,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
532
536
564
[WARNING|trainer.py:803] 2025-04-26 18:49:07,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:07,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:07,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
533
537
565
[WARNING|trainer.py:803] 2025-04-26 18:49:08,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:08,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
538
[WARNING|trainer.py:803] 2025-04-26 18:49:09,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
534
566
[WARNING|trainer.py:803] 2025-04-26 18:49:10,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:49:10,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
539
[WARNING|trainer.py:803] 2025-04-26 18:49:10,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
535
567
[WARNING|trainer.py:803] 2025-04-26 18:49:11,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
540
[WARNING|trainer.py:803] 2025-04-26 18:49:11,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:11,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
536
[WARNING|trainer.py:803] 2025-04-26 18:49:12,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
568
541
[WARNING|trainer.py:803] 2025-04-26 18:49:12,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:13,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
537
[WARNING|trainer.py:803] 2025-04-26 18:49:13,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
542
569
[WARNING|trainer.py:803] 2025-04-26 18:49:14,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:14,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:14,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
538
543
570
[WARNING|trainer.py:803] 2025-04-26 18:49:15,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:49:15,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
544
[WARNING|trainer.py:803] 2025-04-26 18:49:15,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
539
571
[WARNING|trainer.py:803] 2025-04-26 18:49:16,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:16,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
545
[WARNING|trainer.py:803] 2025-04-26 18:49:17,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
540
[WARNING|trainer.py:803] 2025-04-26 18:49:17,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
572
546
[WARNING|trainer.py:803] 2025-04-26 18:49:18,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:18,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
541
[WARNING|trainer.py:803] 2025-04-26 18:49:18,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
547
573
[WARNING|trainer.py:803] 2025-04-26 18:49:19,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:19,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:19,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
542
548
574
[WARNING|trainer.py:803] 2025-04-26 18:49:20,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:21,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
549
[WARNING|trainer.py:803] 2025-04-26 18:49:21,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
543
575
[WARNING|trainer.py:803] 2025-04-26 18:49:22,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:22,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
550
544
[WARNING|trainer.py:803] 2025-04-26 18:49:22,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:23,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
576
551
[WARNING|trainer.py:803] 2025-04-26 18:49:23,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
545
[WARNING|trainer.py:803] 2025-04-26 18:49:24,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:24,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
552
577
[WARNING|trainer.py:803] 2025-04-26 18:49:24,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
546
[WARNING|trainer.py:803] 2025-04-26 18:49:25,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:25,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
553
578
[WARNING|trainer.py:803] 2025-04-26 18:49:26,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:26,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
554
[WARNING|trainer.py:803] 2025-04-26 18:49:26,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
547
579
[WARNING|trainer.py:803] 2025-04-26 18:49:27,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:27,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
555
[WARNING|trainer.py:803] 2025-04-26 18:49:28,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
548
580
[WARNING|trainer.py:803] 2025-04-26 18:49:28,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:28,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
556
549
[WARNING|trainer.py:803] 2025-04-26 18:49:29,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:29,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
581
557
[WARNING|trainer.py:803] 2025-04-26 18:49:30,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
550
[WARNING|trainer.py:803] 2025-04-26 18:49:30,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:49:30,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
558
582
[WARNING|trainer.py:803] 2025-04-26 18:49:31,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
551
[WARNING|trainer.py:803] 2025-04-26 18:49:31,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:32,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
559
583
[WARNING|trainer.py:803] 2025-04-26 18:49:32,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:33,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
552
560
[WARNING|trainer.py:803] 2025-04-26 18:49:33,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
584
[WARNING|trainer.py:803] 2025-04-26 18:49:34,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:34,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
561
553
[WARNING|trainer.py:803] 2025-04-26 18:49:34,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
585
[WARNING|trainer.py:803] 2025-04-26 18:49:35,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:35,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
562
554
[WARNING|trainer.py:803] 2025-04-26 18:49:36,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:49:36,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
586
563
[WARNING|trainer.py:803] 2025-04-26 18:49:36,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
555
[WARNING|trainer.py:803] 2025-04-26 18:49:37,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:37,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
564
587
[WARNING|trainer.py:803] 2025-04-26 18:49:38,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
556
[WARNING|trainer.py:803] 2025-04-26 18:49:38,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
565
[WARNING|trainer.py:803] 2025-04-26 18:49:38,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
588
[WARNING|trainer.py:803] 2025-04-26 18:49:39,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:39,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
557
566
[WARNING|trainer.py:803] 2025-04-26 18:49:40,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
589
[WARNING|trainer.py:803] 2025-04-26 18:49:40,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:49:40,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
567
558
[WARNING|trainer.py:803] 2025-04-26 18:49:41,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:49:41,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
568
[WARNING|trainer.py:803] 2025-04-26 18:49:42,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
590
559
[WARNING|trainer.py:803] 2025-04-26 18:49:42,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:49:43,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
569
[WARNING|trainer.py:803] 2025-04-26 18:49:43,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
591
560
[WARNING|trainer.py:803] 2025-04-26 18:49:44,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
570
[WARNING|trainer.py:803] 2025-04-26 18:49:44,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:44,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
592
[WARNING|trainer.py:803] 2025-04-26 18:49:45,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
561
571
[WARNING|trainer.py:803] 2025-04-26 18:49:45,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:46,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
593
[WARNING|trainer.py:803] 2025-04-26 18:49:46,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
562
572
[WARNING|trainer.py:803] 2025-04-26 18:49:47,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:47,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
594
[WARNING|trainer.py:803] 2025-04-26 18:49:47,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
573
563
[WARNING|trainer.py:803] 2025-04-26 18:49:48,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:48,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:48,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
595
574
564
[WARNING|trainer.py:803] 2025-04-26 18:49:49,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:49,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:50,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
575
596
565
[WARNING|trainer.py:803] 2025-04-26 18:49:51,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:51,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
576
[WARNING|trainer.py:803] 2025-04-26 18:49:51,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
597
566
[WARNING|trainer.py:803] 2025-04-26 18:49:52,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
577
[WARNING|trainer.py:803] 2025-04-26 18:49:52,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:52,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
598
[WARNING|trainer.py:803] 2025-04-26 18:49:53,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
567
578
[WARNING|trainer.py:803] 2025-04-26 18:49:53,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:54,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
599
[WARNING|trainer.py:803] 2025-04-26 18:49:54,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
568
579
[WARNING|trainer.py:803] 2025-04-26 18:49:55,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:55,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:49:55,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
600
580
569
[WARNING|trainer.py:803] 2025-04-26 18:49:56,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:56,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:56,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
581
570
[WARNING|trainer.py:803] 2025-04-26 18:49:57,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
601
582
[WARNING|trainer.py:803] 2025-04-26 18:49:58,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:49:58,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
571
[WARNING|trainer.py:803] 2025-04-26 18:49:58,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
583
[WARNING|trainer.py:803] 2025-04-26 18:49:59,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:49:59,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
602
572
584
[WARNING|trainer.py:803] 2025-04-26 18:50:00,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:00,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:01,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
585
573
[WARNING|trainer.py:803] 2025-04-26 18:50:02,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
603
[WARNING|trainer.py:803] 2025-04-26 18:50:02,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
586
574
[WARNING|trainer.py:803] 2025-04-26 18:50:03,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:03,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:03,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
587
575
604
[WARNING|trainer.py:803] 2025-04-26 18:50:04,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
588
[WARNING|trainer.py:803] 2025-04-26 18:50:05,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:05,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
576
[WARNING|trainer.py:803] 2025-04-26 18:50:05,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
589
[WARNING|trainer.py:803] 2025-04-26 18:50:06,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
605
577
[WARNING|trainer.py:803] 2025-04-26 18:50:06,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
590
[WARNING|trainer.py:803] 2025-04-26 18:50:07,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:07,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
578
[WARNING|trainer.py:803] 2025-04-26 18:50:08,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
591
606
[WARNING|trainer.py:803] 2025-04-26 18:50:09,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:09,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
579
[WARNING|trainer.py:803] 2025-04-26 18:50:09,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
592
[WARNING|trainer.py:803] 2025-04-26 18:50:10,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:10,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
580
593
607
[WARNING|trainer.py:803] 2025-04-26 18:50:11,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:11,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
594
[WARNING|trainer.py:803] 2025-04-26 18:50:12,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
581
[WARNING|trainer.py:803] 2025-04-26 18:50:12,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:13,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
595
608
582
[WARNING|trainer.py:803] 2025-04-26 18:50:14,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:14,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
596
[WARNING|trainer.py:803] 2025-04-26 18:50:14,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
583
[WARNING|trainer.py:803] 2025-04-26 18:50:15,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
597
[WARNING|trainer.py:803] 2025-04-26 18:50:15,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
609
584
[WARNING|trainer.py:803] 2025-04-26 18:50:16,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
598
[WARNING|trainer.py:803] 2025-04-26 18:50:16,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:17,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:17,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
585
599
610
[WARNING|trainer.py:803] 2025-04-26 18:50:18,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:18,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:18,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
600
586
[WARNING|trainer.py:803] 2025-04-26 18:50:19,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:19,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
611
587
601
[WARNING|trainer.py:803] 2025-04-26 18:50:20,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:21,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:21,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
588
602
612
[WARNING|trainer.py:803] 2025-04-26 18:50:22,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
589
[WARNING|trainer.py:803] 2025-04-26 18:50:23,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:23,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:24,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
590
603
613
[WARNING|trainer.py:803] 2025-04-26 18:50:25,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:25,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:26,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
591
604
[WARNING|trainer.py:803] 2025-04-26 18:50:26,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
614
592
[WARNING|trainer.py:803] 2025-04-26 18:50:27,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:28,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:28,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
605
593
[WARNING|trainer.py:803] 2025-04-26 18:50:29,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
615
[WARNING|trainer.py:803] 2025-04-26 18:50:29,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
594
[WARNING|trainer.py:803] 2025-04-26 18:50:30,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
606
[WARNING|trainer.py:803] 2025-04-26 18:50:31,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:31,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
595
616
[WARNING|trainer.py:803] 2025-04-26 18:50:32,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:32,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
607
596
[WARNING|trainer.py:803] 2025-04-26 18:50:33,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:33,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
617
597
608
[WARNING|trainer.py:803] 2025-04-26 18:50:34,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:35,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:35,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
598
618
[WARNING|trainer.py:803] 2025-04-26 18:50:36,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
609
[WARNING|trainer.py:803] 2025-04-26 18:50:36,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
599
[WARNING|trainer.py:803] 2025-04-26 18:50:37,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:37,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
600
619
610
[WARNING|trainer.py:803] 2025-04-26 18:50:39,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:39,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:39,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
620
611
601
[WARNING|trainer.py:803] 2025-04-26 18:50:41,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:41,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:41,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
621
602
612
[WARNING|trainer.py:803] 2025-04-26 18:50:43,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:43,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:50:43,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
622
603
613
[WARNING|trainer.py:803] 2025-04-26 18:50:45,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:45,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:46,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
614
604
623
[WARNING|trainer.py:803] 2025-04-26 18:50:48,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:48,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:48,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
615
605
[WARNING|trainer.py:803] 2025-04-26 18:50:50,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
624
[WARNING|trainer.py:803] 2025-04-26 18:50:50,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:50,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
616
606
[WARNING|trainer.py:803] 2025-04-26 18:50:51,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
625
[WARNING|trainer.py:803] 2025-04-26 18:50:52,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
617
[WARNING|trainer.py:803] 2025-04-26 18:50:53,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:50:53,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
607
626
[WARNING|trainer.py:803] 2025-04-26 18:50:54,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
618
[WARNING|trainer.py:803] 2025-04-26 18:50:55,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:55,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
608
619
627
[WARNING|trainer.py:803] 2025-04-26 18:50:57,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:57,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:50:57,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
609
620
628
[WARNING|trainer.py:803] 2025-04-26 18:50:59,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:50:59,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:00,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
621
610
629
[WARNING|trainer.py:803] 2025-04-26 18:51:01,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:01,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:02,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
622
611
[WARNING|trainer.py:803] 2025-04-26 18:51:03,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
630
[WARNING|trainer.py:803] 2025-04-26 18:51:04,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:51:04,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
623
612
631
[WARNING|trainer.py:803] 2025-04-26 18:51:05,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:06,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:06,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
624
[WARNING|trainer.py:803] 2025-04-26 18:51:08,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
632
613
[WARNING|trainer.py:803] 2025-04-26 18:51:09,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:09,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
625
[WARNING|trainer.py:803] 2025-04-26 18:51:10,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
633
614
[WARNING|trainer.py:803] 2025-04-26 18:51:11,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:11,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
626
[WARNING|trainer.py:803] 2025-04-26 18:51:12,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
615
634
[WARNING|trainer.py:803] 2025-04-26 18:51:13,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
627
[WARNING|trainer.py:803] 2025-04-26 18:51:13,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:14,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
616
635
628
[WARNING|trainer.py:803] 2025-04-26 18:51:15,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:51:16,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:16,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
617
636
629
[WARNING|trainer.py:803] 2025-04-26 18:51:17,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:51:18,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:18,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
618
637
630
[WARNING|trainer.py:803] 2025-04-26 18:51:20,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:20,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:20,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
631
619
638
[WARNING|trainer.py:803] 2025-04-26 18:51:22,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:22,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:22,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
632
620
639
[WARNING|trainer.py:803] 2025-04-26 18:51:24,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:24,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:24,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
633
621
[WARNING|trainer.py:803] 2025-04-26 18:51:26,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
640
[WARNING|trainer.py:803] 2025-04-26 18:51:26,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:27,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
634
641
622
[WARNING|trainer.py:803] 2025-04-26 18:51:28,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:29,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:29,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
635
[WARNING|trainer.py:803] 2025-04-26 18:51:30,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
642
623
[WARNING|trainer.py:803] 2025-04-26 18:51:31,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
636
[WARNING|trainer.py:803] 2025-04-26 18:51:31,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:32,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
643
624
637
[WARNING|trainer.py:803] 2025-04-26 18:51:34,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:51:34,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:34,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
638
625
644
[WARNING|trainer.py:803] 2025-04-26 18:51:36,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:36,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:36,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
639
626
645
[WARNING|trainer.py:803] 2025-04-26 18:51:38,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:38,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:38,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
640
646
[WARNING|trainer.py:803] 2025-04-26 18:51:40,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
627
[WARNING|trainer.py:803] 2025-04-26 18:51:41,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:41,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
641
[WARNING|trainer.py:803] 2025-04-26 18:51:42,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
628
647
[WARNING|trainer.py:803] 2025-04-26 18:51:43,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
642
[WARNING|trainer.py:803] 2025-04-26 18:51:43,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:44,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
629
648
[WARNING|trainer.py:803] 2025-04-26 18:51:45,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
643
[WARNING|trainer.py:803] 2025-04-26 18:51:45,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:46,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
649
630
644
[WARNING|trainer.py:803] 2025-04-26 18:51:47,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:48,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:48,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
631
650
645
[WARNING|trainer.py:803] 2025-04-26 18:51:50,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:50,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:50,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
632
651
646
[WARNING|trainer.py:803] 2025-04-26 18:51:52,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:52,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:51:52,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
633
652
647
[WARNING|trainer.py:803] 2025-04-26 18:51:54,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:54,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:54,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
648
653
634
[WARNING|trainer.py:803] 2025-04-26 18:51:56,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:51:56,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:57,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
649
635
654
[WARNING|trainer.py:803] 2025-04-26 18:51:58,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:51:59,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:51:59,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
650
636
655
[WARNING|trainer.py:803] 2025-04-26 18:52:01,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:01,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:01,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
651
[WARNING|trainer.py:803] 2025-04-26 18:52:02,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
637
656
[WARNING|trainer.py:803] 2025-04-26 18:52:03,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:03,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
652
[WARNING|trainer.py:803] 2025-04-26 18:52:04,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
638
657
[WARNING|trainer.py:803] 2025-04-26 18:52:06,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
653
[WARNING|trainer.py:803] 2025-04-26 18:52:06,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:06,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
658
639
[WARNING|trainer.py:803] 2025-04-26 18:52:08,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
654
[WARNING|trainer.py:803] 2025-04-26 18:52:08,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:09,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
640
659
655
[WARNING|trainer.py:803] 2025-04-26 18:52:10,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:10,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:11,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
641
660
656
[WARNING|trainer.py:803] 2025-04-26 18:52:12,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:12,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:12,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
661
657
642
[WARNING|trainer.py:803] 2025-04-26 18:52:14,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:14,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:14,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
658
643
662
[WARNING|trainer.py:803] 2025-04-26 18:52:16,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:17,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:52:17,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
659
[WARNING|trainer.py:803] 2025-04-26 18:52:18,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
644
663
660
[WARNING|trainer.py:803] 2025-04-26 18:52:19,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:20,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:52:20,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
645
661
664
[WARNING|trainer.py:803] 2025-04-26 18:52:22,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:22,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:22,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
646
665
662
[WARNING|trainer.py:803] 2025-04-26 18:52:24,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:24,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:25,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
666
647
663
[WARNING|trainer.py:803] 2025-04-26 18:52:27,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:27,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:27,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
667
648
664
[WARNING|trainer.py:803] 2025-04-26 18:52:29,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:29,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:29,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
665
668
649
[WARNING|trainer.py:803] 2025-04-26 18:52:31,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:31,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:31,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
666
669
650
[WARNING|trainer.py:803] 2025-04-26 18:52:33,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:33,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:52:33,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
667
651
670
[WARNING|trainer.py:803] 2025-04-26 18:52:35,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:35,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:52:36,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
668
[WARNING|trainer.py:803] 2025-04-26 18:52:37,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
652
671
[WARNING|trainer.py:803] 2025-04-26 18:52:38,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
669
[WARNING|trainer.py:803] 2025-04-26 18:52:38,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:39,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
653
672
[WARNING|trainer.py:803] 2025-04-26 18:52:40,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
670
[WARNING|trainer.py:803] 2025-04-26 18:52:40,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:41,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
673
654
671
[WARNING|trainer.py:803] 2025-04-26 18:52:42,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:42,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:43,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
674
655
672
[WARNING|trainer.py:803] 2025-04-26 18:52:45,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:45,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:45,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
656
673
675
[WARNING|trainer.py:803] 2025-04-26 18:52:47,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:47,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:52:47,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
674
657
676
[WARNING|trainer.py:803] 2025-04-26 18:52:49,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:49,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:49,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
658
675
677
[WARNING|trainer.py:803] 2025-04-26 18:52:51,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:51,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:52,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
676
659
678
[WARNING|trainer.py:803] 2025-04-26 18:52:53,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:52:54,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:54,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
677
660
[WARNING|trainer.py:803] 2025-04-26 18:52:55,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
679
[WARNING|trainer.py:803] 2025-04-26 18:52:56,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:52:56,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
678
661
[WARNING|trainer.py:803] 2025-04-26 18:52:57,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
680
[WARNING|trainer.py:803] 2025-04-26 18:52:58,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
679
[WARNING|trainer.py:803] 2025-04-26 18:52:58,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:52:59,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
681
662
680
[WARNING|trainer.py:803] 2025-04-26 18:53:00,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:01,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:01,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
682
681
663
[WARNING|trainer.py:803] 2025-04-26 18:53:02,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:03,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:03,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
682
683
[WARNING|trainer.py:803] 2025-04-26 18:53:05,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
664
[WARNING|trainer.py:803] 2025-04-26 18:53:05,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:06,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
683
665
[WARNING|trainer.py:803] 2025-04-26 18:53:07,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
684
[WARNING|trainer.py:803] 2025-04-26 18:53:08,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:08,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
684
666
685
[WARNING|trainer.py:803] 2025-04-26 18:53:10,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:10,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:11,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
685
667
[WARNING|trainer.py:803] 2025-04-26 18:53:12,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:12,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
686
[WARNING|trainer.py:803] 2025-04-26 18:53:13,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
686
668
[WARNING|trainer.py:803] 2025-04-26 18:53:14,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:14,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
687
[WARNING|trainer.py:803] 2025-04-26 18:53:16,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
687
669
[WARNING|trainer.py:803] 2025-04-26 18:53:16,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:17,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
688
688
[WARNING|trainer.py:803] 2025-04-26 18:53:18,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
670
[WARNING|trainer.py:803] 2025-04-26 18:53:18,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:19,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
689
689
[WARNING|trainer.py:803] 2025-04-26 18:53:20,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:20,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
671
690
690
[WARNING|trainer.py:803] 2025-04-26 18:53:22,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:22,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:22,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
672
691
691
[WARNING|trainer.py:803] 2025-04-26 18:53:24,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:24,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:53:25,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
673
[WARNING|trainer.py:803] 2025-04-26 18:53:26,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
692
692
[WARNING|trainer.py:803] 2025-04-26 18:53:27,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
674
[WARNING|trainer.py:803] 2025-04-26 18:53:27,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
693
[WARNING|trainer.py:803] 2025-04-26 18:53:28,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
693
[WARNING|trainer.py:803] 2025-04-26 18:53:29,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:53:29,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
694
675
694
[WARNING|trainer.py:803] 2025-04-26 18:53:30,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:30,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:31,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
676
695
[WARNING|trainer.py:803] 2025-04-26 18:53:33,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:33,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
695
[WARNING|trainer.py:803] 2025-04-26 18:53:34,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
696
677
[WARNING|trainer.py:803] 2025-04-26 18:53:35,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:35,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
696
[WARNING|trainer.py:803] 2025-04-26 18:53:36,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
678
697
[WARNING|trainer.py:803] 2025-04-26 18:53:37,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:37,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
697
679
[WARNING|trainer.py:803] 2025-04-26 18:53:38,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
698
[WARNING|trainer.py:803] 2025-04-26 18:53:39,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:53:40,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
698
680
699
[WARNING|trainer.py:803] 2025-04-26 18:53:41,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:41,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:53:41,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
700
681
699
[WARNING|trainer.py:803] 2025-04-26 18:53:43,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:44,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:44,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
682
701
700
[WARNING|trainer.py:803] 2025-04-26 18:53:46,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:46,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:53:46,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
702
701
683
[WARNING|trainer.py:803] 2025-04-26 18:53:48,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:48,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 18:53:49,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
703
[WARNING|trainer.py:803] 2025-04-26 18:53:50,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
702
[WARNING|trainer.py:803] 2025-04-26 18:53:51,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
684
704
[WARNING|trainer.py:803] 2025-04-26 18:53:52,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:53:52,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
703
705
[WARNING|trainer.py:803] 2025-04-26 18:53:53,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
685
[WARNING|trainer.py:803] 2025-04-26 18:53:54,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:54,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
704
706
[WARNING|trainer.py:803] 2025-04-26 18:53:55,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
686
[WARNING|trainer.py:803] 2025-04-26 18:53:56,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
705
[WARNING|trainer.py:803] 2025-04-26 18:53:57,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
707
[WARNING|trainer.py:803] 2025-04-26 18:53:57,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
687
[WARNING|trainer.py:803] 2025-04-26 18:53:58,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:53:59,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
706
708
[WARNING|trainer.py:803] 2025-04-26 18:54:00,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
688
[WARNING|trainer.py:803] 2025-04-26 18:54:00,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:01,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
709
707
[WARNING|trainer.py:803] 2025-04-26 18:54:02,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:02,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
689
[WARNING|trainer.py:803] 2025-04-26 18:54:03,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
710
708
690
[WARNING|trainer.py:803] 2025-04-26 18:54:04,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:05,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:05,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
711
709
[WARNING|trainer.py:803] 2025-04-26 18:54:07,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:07,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
691
712
[WARNING|trainer.py:803] 2025-04-26 18:54:08,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:54:08,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
710
[WARNING|trainer.py:803] 2025-04-26 18:54:09,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
692
713
[WARNING|trainer.py:803] 2025-04-26 18:54:10,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
711
[WARNING|trainer.py:803] 2025-04-26 18:54:11,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
693
[WARNING|trainer.py:803] 2025-04-26 18:54:12,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
714
[WARNING|trainer.py:803] 2025-04-26 18:54:12,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
712
[WARNING|trainer.py:803] 2025-04-26 18:54:13,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
694
[WARNING|trainer.py:803] 2025-04-26 18:54:14,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:14,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
715
[WARNING|trainer.py:803] 2025-04-26 18:54:15,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
713
695
716
[WARNING|trainer.py:803] 2025-04-26 18:54:17,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:54:17,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:17,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
696
714
717
[WARNING|trainer.py:803] 2025-04-26 18:54:19,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:19,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:20,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
715
697
718
[WARNING|trainer.py:803] 2025-04-26 18:54:22,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:22,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:54:22,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
716
719
698
[WARNING|trainer.py:803] 2025-04-26 18:54:24,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:24,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:25,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
720
717
699
[WARNING|trainer.py:803] 2025-04-26 18:54:26,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:54:27,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:27,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
721
700
718
[WARNING|trainer.py:803] 2025-04-26 18:54:28,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:54:29,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:29,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
722
[WARNING|trainer.py:803] 2025-04-26 18:54:30,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
719
701
[WARNING|trainer.py:803] 2025-04-26 18:54:31,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:31,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
723
[WARNING|trainer.py:803] 2025-04-26 18:54:32,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
720
702
[WARNING|trainer.py:803] 2025-04-26 18:54:34,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:54:34,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
724
703
[WARNING|trainer.py:803] 2025-04-26 18:54:35,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
721
[WARNING|trainer.py:803] 2025-04-26 18:54:36,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 18:54:36,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
725
704
[WARNING|trainer.py:803] 2025-04-26 18:54:38,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
722
[WARNING|trainer.py:803] 2025-04-26 18:54:38,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:38,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
726
705
[WARNING|trainer.py:803] 2025-04-26 18:54:40,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
723
[WARNING|trainer.py:803] 2025-04-26 18:54:40,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:41,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
727
[WARNING|trainer.py:803] 2025-04-26 18:54:42,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
706
728
724
[WARNING|trainer.py:803] 2025-04-26 18:54:43,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:44,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:44,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
707
729
[WARNING|trainer.py:803] 2025-04-26 18:54:45,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
725
[WARNING|trainer.py:803] 2025-04-26 18:54:46,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:46,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
730
708
[WARNING|trainer.py:803] 2025-04-26 18:54:48,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:48,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
726
[WARNING|trainer.py:803] 2025-04-26 18:54:49,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
731
709
[WARNING|trainer.py:803] 2025-04-26 18:54:49,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:50,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
727
732
[WARNING|trainer.py:803] 2025-04-26 18:54:51,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:51,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
710
728
[WARNING|trainer.py:803] 2025-04-26 18:54:52,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
733
[WARNING|trainer.py:803] 2025-04-26 18:54:53,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:54:54,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
711
729
[WARNING|trainer.py:803] 2025-04-26 18:54:55,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
734
[WARNING|trainer.py:803] 2025-04-26 18:54:55,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:56,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
712
730
[WARNING|trainer.py:803] 2025-04-26 18:54:57,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
735
[WARNING|trainer.py:803] 2025-04-26 18:54:58,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:54:58,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
731
713
736
[WARNING|trainer.py:803] 2025-04-26 18:55:00,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:00,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:00,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
732
714
737
[WARNING|trainer.py:803] 2025-04-26 18:55:02,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:02,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:03,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
738
733
715
[WARNING|trainer.py:803] 2025-04-26 18:55:04,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:04,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:05,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
739
734
716
[WARNING|trainer.py:803] 2025-04-26 18:55:07,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:07,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:07,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
740
735
[WARNING|trainer.py:803] 2025-04-26 18:55:09,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
717
[WARNING|trainer.py:803] 2025-04-26 18:55:10,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:10,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
741
[WARNING|trainer.py:803] 2025-04-26 18:55:11,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
736
718
742
[WARNING|trainer.py:803] 2025-04-26 18:55:12,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:12,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:13,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
737
719
743
[WARNING|trainer.py:803] 2025-04-26 18:55:15,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:15,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:15,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
738
744
720
[WARNING|trainer.py:803] 2025-04-26 18:55:17,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:17,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:17,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
745
739
721
[WARNING|trainer.py:803] 2025-04-26 18:55:19,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:19,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:19,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
746
740
[WARNING|trainer.py:803] 2025-04-26 18:55:21,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
722
[WARNING|trainer.py:803] 2025-04-26 18:55:21,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:22,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
747
741
[WARNING|trainer.py:803] 2025-04-26 18:55:23,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
723
[WARNING|trainer.py:803] 2025-04-26 18:55:24,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:24,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
748
742
[WARNING|trainer.py:803] 2025-04-26 18:55:25,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:26,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
724
749
[WARNING|trainer.py:803] 2025-04-26 18:55:27,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:27,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
743
[WARNING|trainer.py:803] 2025-04-26 18:55:28,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
750
725
[WARNING|trainer.py:803] 2025-04-26 18:55:29,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:30,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
744
[WARNING|trainer.py:803] 2025-04-26 18:55:31,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
751
726
[WARNING|trainer.py:803] 2025-04-26 18:55:31,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:32,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
745
752
[WARNING|trainer.py:803] 2025-04-26 18:55:33,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:33,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
727
753
[WARNING|trainer.py:803] 2025-04-26 18:55:34,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
746
[WARNING|trainer.py:803] 2025-04-26 18:55:35,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:55:35,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
728
754
[WARNING|trainer.py:803] 2025-04-26 18:55:36,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
747
[WARNING|trainer.py:803] 2025-04-26 18:55:37,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:38,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
729
755
[WARNING|trainer.py:803] 2025-04-26 18:55:39,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:39,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
748
756
[WARNING|trainer.py:803] 2025-04-26 18:55:40,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
730
[WARNING|trainer.py:803] 2025-04-26 18:55:41,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:42,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
749
757
[WARNING|trainer.py:803] 2025-04-26 18:55:43,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
731
[WARNING|trainer.py:803] 2025-04-26 18:55:43,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:44,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
750
758
732
[WARNING|trainer.py:803] 2025-04-26 18:55:45,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:46,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:46,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
751
759
733
[WARNING|trainer.py:803] 2025-04-26 18:55:48,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:48,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:48,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
752
760
[WARNING|trainer.py:803] 2025-04-26 18:55:50,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
734
[WARNING|trainer.py:803] 2025-04-26 18:55:50,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:51,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
753
761
[WARNING|trainer.py:803] 2025-04-26 18:55:52,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:55:52,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
735
762
[WARNING|trainer.py:803] 2025-04-26 18:55:53,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
754
[WARNING|trainer.py:803] 2025-04-26 18:55:54,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:54,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
736
763
755
[WARNING|trainer.py:803] 2025-04-26 18:55:56,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:56,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:55:56,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
737
764
756
[WARNING|trainer.py:803] 2025-04-26 18:55:58,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:55:58,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:55:59,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
738
765
757
[WARNING|trainer.py:803] 2025-04-26 18:56:00,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:56:01,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:01,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
766
739
[WARNING|trainer.py:803] 2025-04-26 18:56:03,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
758
[WARNING|trainer.py:803] 2025-04-26 18:56:03,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:56:04,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
767
740
759
[WARNING|trainer.py:803] 2025-04-26 18:56:05,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:05,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:06,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
768
741
[WARNING|trainer.py:803] 2025-04-26 18:56:07,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:07,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
760
[WARNING|trainer.py:803] 2025-04-26 18:56:08,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
742
769
[WARNING|trainer.py:803] 2025-04-26 18:56:10,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:10,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
761
770
[WARNING|trainer.py:803] 2025-04-26 18:56:11,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
743
[WARNING|trainer.py:803] 2025-04-26 18:56:12,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:12,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
762
771
[WARNING|trainer.py:803] 2025-04-26 18:56:13,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:13,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
744
[WARNING|trainer.py:803] 2025-04-26 18:56:14,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
763
772
[WARNING|trainer.py:803] 2025-04-26 18:56:15,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:16,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
745
773
[WARNING|trainer.py:803] 2025-04-26 18:56:17,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
764
[WARNING|trainer.py:803] 2025-04-26 18:56:18,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:56:18,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
746
774
[WARNING|trainer.py:803] 2025-04-26 18:56:19,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
765
[WARNING|trainer.py:803] 2025-04-26 18:56:20,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:20,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
775
747
[WARNING|trainer.py:803] 2025-04-26 18:56:21,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:22,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
766
776
[WARNING|trainer.py:803] 2025-04-26 18:56:23,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
748
[WARNING|trainer.py:803] 2025-04-26 18:56:23,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:24,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
767
777
[WARNING|trainer.py:803] 2025-04-26 18:56:25,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:26,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
749
[WARNING|trainer.py:803] 2025-04-26 18:56:27,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
768
778
[WARNING|trainer.py:803] 2025-04-26 18:56:28,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:28,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
750
[WARNING|trainer.py:803] 2025-04-26 18:56:29,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
779
[WARNING|trainer.py:803] 2025-04-26 18:56:30,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
769
751
[WARNING|trainer.py:803] 2025-04-26 18:56:31,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
780
[WARNING|trainer.py:803] 2025-04-26 18:56:31,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
770
[WARNING|trainer.py:803] 2025-04-26 18:56:32,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
752
[WARNING|trainer.py:803] 2025-04-26 18:56:33,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
781
[WARNING|trainer.py:803] 2025-04-26 18:56:34,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
771
[WARNING|trainer.py:803] 2025-04-26 18:56:34,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:35,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
753
782
[WARNING|trainer.py:803] 2025-04-26 18:56:36,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:56:36,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
772
[WARNING|trainer.py:803] 2025-04-26 18:56:37,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
754
783
[WARNING|trainer.py:803] 2025-04-26 18:56:38,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:56:38,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
773
755
784
[WARNING|trainer.py:803] 2025-04-26 18:56:40,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:56:40,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:56:40,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
774
785
756
[WARNING|trainer.py:803] 2025-04-26 18:56:42,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:42,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:42,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
775
786
[WARNING|trainer.py:803] 2025-04-26 18:56:44,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
757
[WARNING|trainer.py:803] 2025-04-26 18:56:45,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:45,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
776
787
[WARNING|trainer.py:803] 2025-04-26 18:56:46,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:47,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
758
777
[WARNING|trainer.py:803] 2025-04-26 18:56:48,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:48,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
788
759
[WARNING|trainer.py:803] 2025-04-26 18:56:49,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:56:50,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
778
789
[WARNING|trainer.py:803] 2025-04-26 18:56:51,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:51,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
760
779
790
[WARNING|trainer.py:803] 2025-04-26 18:56:52,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:53,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:56:53,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
761
791
[WARNING|trainer.py:803] 2025-04-26 18:56:55,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
780
[WARNING|trainer.py:803] 2025-04-26 18:56:55,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:56:56,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
762
792
[WARNING|trainer.py:803] 2025-04-26 18:56:57,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:56:57,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
781
[WARNING|trainer.py:803] 2025-04-26 18:56:58,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
763
793
[WARNING|trainer.py:803] 2025-04-26 18:56:59,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
782
[WARNING|trainer.py:803] 2025-04-26 18:56:59,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:00,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
794
764
[WARNING|trainer.py:803] 2025-04-26 18:57:02,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
783
[WARNING|trainer.py:803] 2025-04-26 18:57:02,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
795
[WARNING|trainer.py:803] 2025-04-26 18:57:03,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:03,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
765
784
[WARNING|trainer.py:803] 2025-04-26 18:57:04,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
796
[WARNING|trainer.py:803] 2025-04-26 18:57:05,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:06,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
766
785
[WARNING|trainer.py:803] 2025-04-26 18:57:06,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
797
[WARNING|trainer.py:803] 2025-04-26 18:57:07,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:08,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
767
786
798
[WARNING|trainer.py:803] 2025-04-26 18:57:09,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:10,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:10,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
768
799
787
[WARNING|trainer.py:803] 2025-04-26 18:57:12,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:12,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:12,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
800
769
[WARNING|trainer.py:803] 2025-04-26 18:57:14,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
788
[WARNING|trainer.py:803] 2025-04-26 18:57:15,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:15,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
801
770
789
[WARNING|trainer.py:803] 2025-04-26 18:57:16,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:17,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:17,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
802
771
790
[WARNING|trainer.py:803] 2025-04-26 18:57:18,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:19,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:19,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
803
772
791
[WARNING|trainer.py:803] 2025-04-26 18:57:21,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:21,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:21,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
804
[WARNING|trainer.py:803] 2025-04-26 18:57:22,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
773
792
[WARNING|trainer.py:803] 2025-04-26 18:57:23,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
805
[WARNING|trainer.py:803] 2025-04-26 18:57:24,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:24,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
774
793
806
[WARNING|trainer.py:803] 2025-04-26 18:57:26,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:26,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:26,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
775
[WARNING|trainer.py:803] 2025-04-26 18:57:28,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
794
807
[WARNING|trainer.py:803] 2025-04-26 18:57:29,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:29,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
776
795
808
[WARNING|trainer.py:803] 2025-04-26 18:57:30,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:31,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:31,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
777
809
796
[WARNING|trainer.py:803] 2025-04-26 18:57:32,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:32,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:33,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
810
778
[WARNING|trainer.py:803] 2025-04-26 18:57:34,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
797
[WARNING|trainer.py:803] 2025-04-26 18:57:35,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:35,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
811
779
[WARNING|trainer.py:803] 2025-04-26 18:57:37,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
798
[WARNING|trainer.py:803] 2025-04-26 18:57:37,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
812
[WARNING|trainer.py:803] 2025-04-26 18:57:38,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:38,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
780
799
813
[WARNING|trainer.py:803] 2025-04-26 18:57:40,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:40,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:40,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
781
800
814
[WARNING|trainer.py:803] 2025-04-26 18:57:42,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:43,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:43,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
782
815
801
[WARNING|trainer.py:803] 2025-04-26 18:57:44,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:44,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:45,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
816
783
[WARNING|trainer.py:803] 2025-04-26 18:57:46,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:47,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
802
[WARNING|trainer.py:803] 2025-04-26 18:57:48,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
817
784
[WARNING|trainer.py:803] 2025-04-26 18:57:48,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:49,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
803
818
[WARNING|trainer.py:803] 2025-04-26 18:57:50,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:50,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
785
819
804
[WARNING|trainer.py:803] 2025-04-26 18:57:51,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:52,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:52,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
786
805
820
[WARNING|trainer.py:803] 2025-04-26 18:57:54,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:57:54,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:57:54,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
821
787
806
[WARNING|trainer.py:803] 2025-04-26 18:57:56,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:56,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:57:57,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
822
788
[WARNING|trainer.py:803] 2025-04-26 18:57:58,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
807
[WARNING|trainer.py:803] 2025-04-26 18:57:59,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:57:59,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
823
789
[WARNING|trainer.py:803] 2025-04-26 18:58:01,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
808
[WARNING|trainer.py:803] 2025-04-26 18:58:01,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:02,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
824
[WARNING|trainer.py:803] 2025-04-26 18:58:02,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
790
809
825
[WARNING|trainer.py:803] 2025-04-26 18:58:03,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:04,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:04,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
791
810
826
[WARNING|trainer.py:803] 2025-04-26 18:58:05,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:06,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:06,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
792
827
811
[WARNING|trainer.py:803] 2025-04-26 18:58:08,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:08,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:08,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
828
812
793
[WARNING|trainer.py:803] 2025-04-26 18:58:10,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:10,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:10,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
829
813
[WARNING|trainer.py:803] 2025-04-26 18:58:12,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
794
[WARNING|trainer.py:803] 2025-04-26 18:58:12,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
830
[WARNING|trainer.py:803] 2025-04-26 18:58:13,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:14,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
795
814
[WARNING|trainer.py:803] 2025-04-26 18:58:15,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
831
[WARNING|trainer.py:803] 2025-04-26 18:58:15,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:16,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
796
815
832
[WARNING|trainer.py:803] 2025-04-26 18:58:17,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:17,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:18,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
816
797
833
[WARNING|trainer.py:803] 2025-04-26 18:58:19,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:20,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:20,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
834
817
798
[WARNING|trainer.py:803] 2025-04-26 18:58:22,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:22,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:22,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
835
818
799
[WARNING|trainer.py:803] 2025-04-26 18:58:23,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:24,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:24,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
836
819
[WARNING|trainer.py:803] 2025-04-26 18:58:25,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
800
[WARNING|trainer.py:803] 2025-04-26 18:58:26,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:27,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
837
820
[WARNING|trainer.py:803] 2025-04-26 18:58:28,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:28,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
801
838
[WARNING|trainer.py:803] 2025-04-26 18:58:29,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
821
[WARNING|trainer.py:803] 2025-04-26 18:58:30,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:31,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
839
802
[WARNING|trainer.py:803] 2025-04-26 18:58:32,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:32,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
822
840
[WARNING|trainer.py:803] 2025-04-26 18:58:33,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
803
[WARNING|trainer.py:803] 2025-04-26 18:58:34,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:34,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
823
841
[WARNING|trainer.py:803] 2025-04-26 18:58:35,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
804
[WARNING|trainer.py:803] 2025-04-26 18:58:36,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:36,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
824
842
[WARNING|trainer.py:803] 2025-04-26 18:58:38,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:38,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
805
[WARNING|trainer.py:803] 2025-04-26 18:58:39,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
843
825
[WARNING|trainer.py:803] 2025-04-26 18:58:39,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:40,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
806
844
[WARNING|trainer.py:803] 2025-04-26 18:58:41,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
826
[WARNING|trainer.py:803] 2025-04-26 18:58:41,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:42,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
845
807
827
[WARNING|trainer.py:803] 2025-04-26 18:58:43,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:43,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:44,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
846
808
828
[WARNING|trainer.py:803] 2025-04-26 18:58:45,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:46,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:46,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
847
[WARNING|trainer.py:803] 2025-04-26 18:58:47,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
809
829
[WARNING|trainer.py:803] 2025-04-26 18:58:48,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:48,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
848
[WARNING|trainer.py:803] 2025-04-26 18:58:49,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
810
830
[WARNING|trainer.py:803] 2025-04-26 18:58:50,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
849
[WARNING|trainer.py:803] 2025-04-26 18:58:50,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:58:51,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
811
831
850
[WARNING|trainer.py:803] 2025-04-26 18:58:52,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:53,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:53,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
812
832
851
[WARNING|trainer.py:803] 2025-04-26 18:58:55,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:58:55,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:55,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
813
852
833
[WARNING|trainer.py:803] 2025-04-26 18:58:57,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:58:57,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:57,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
853
834
814
[WARNING|trainer.py:803] 2025-04-26 18:58:59,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:59,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:58:59,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
854
[WARNING|trainer.py:803] 2025-04-26 18:59:01,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
815
835
[WARNING|trainer.py:803] 2025-04-26 18:59:01,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:02,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
855
[WARNING|trainer.py:803] 2025-04-26 18:59:02,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
836
816
856
[WARNING|trainer.py:803] 2025-04-26 18:59:04,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:04,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:04,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
857
817
837
[WARNING|trainer.py:803] 2025-04-26 18:59:06,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:06,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:07,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
818
858
838
[WARNING|trainer.py:803] 2025-04-26 18:59:08,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:08,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:09,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
859
819
839
[WARNING|trainer.py:803] 2025-04-26 18:59:10,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:10,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:11,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
860
820
840
[WARNING|trainer.py:803] 2025-04-26 18:59:12,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:13,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:13,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
861
[WARNING|trainer.py:803] 2025-04-26 18:59:14,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
821
841
[WARNING|trainer.py:803] 2025-04-26 18:59:15,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
862
[WARNING|trainer.py:803] 2025-04-26 18:59:15,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:16,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
822
842
863
[WARNING|trainer.py:803] 2025-04-26 18:59:17,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:17,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:18,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
864
843
823
[WARNING|trainer.py:803] 2025-04-26 18:59:20,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:20,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:20,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
865
844
824
[WARNING|trainer.py:803] 2025-04-26 18:59:22,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:22,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:22,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
866
845
825
[WARNING|trainer.py:803] 2025-04-26 18:59:23,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:24,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:24,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
867
846
[WARNING|trainer.py:803] 2025-04-26 18:59:25,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
826
[WARNING|trainer.py:803] 2025-04-26 18:59:26,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
868
[WARNING|trainer.py:803] 2025-04-26 18:59:26,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:27,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
847
827
869
[WARNING|trainer.py:803] 2025-04-26 18:59:28,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:28,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:29,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
828
848
870
[WARNING|trainer.py:803] 2025-04-26 18:59:30,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:31,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:31,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
829
849
871
[WARNING|trainer.py:803] 2025-04-26 18:59:33,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:33,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:33,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
830
850
872
[WARNING|trainer.py:803] 2025-04-26 18:59:35,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:35,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:35,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
831
851
873
[WARNING|trainer.py:803] 2025-04-26 18:59:37,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:37,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:37,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
874
852
832
[WARNING|trainer.py:803] 2025-04-26 18:59:39,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:39,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:40,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
875
853
833
[WARNING|trainer.py:803] 2025-04-26 18:59:42,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:42,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:42,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
876
854
834
[WARNING|trainer.py:803] 2025-04-26 18:59:44,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:44,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 18:59:44,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
877
855
835
[WARNING|trainer.py:803] 2025-04-26 18:59:45,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:46,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:46,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
878
[WARNING|trainer.py:803] 2025-04-26 18:59:47,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
856
836
[WARNING|trainer.py:803] 2025-04-26 18:59:48,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
879
[WARNING|trainer.py:803] 2025-04-26 18:59:48,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:49,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
857
880
[WARNING|trainer.py:803] 2025-04-26 18:59:50,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
837
[WARNING|trainer.py:803] 2025-04-26 18:59:51,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 18:59:51,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
881
858
838
[WARNING|trainer.py:803] 2025-04-26 18:59:53,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:53,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:53,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
859
882
839
[WARNING|trainer.py:803] 2025-04-26 18:59:55,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:55,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:55,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
883
860
840
[WARNING|trainer.py:803] 2025-04-26 18:59:57,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 18:59:57,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 18:59:58,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
884
861
[WARNING|trainer.py:803] 2025-04-26 18:59:59,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
841
[WARNING|trainer.py:803] 2025-04-26 18:59:59,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
885
[WARNING|trainer.py:803] 2025-04-26 19:00:00,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:00,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
862
[WARNING|trainer.py:803] 2025-04-26 19:00:01,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
886
842
[WARNING|trainer.py:803] 2025-04-26 19:00:02,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:03,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
863
[WARNING|trainer.py:803] 2025-04-26 19:00:03,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
843
887
[WARNING|trainer.py:803] 2025-04-26 19:00:05,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
864
[WARNING|trainer.py:803] 2025-04-26 19:00:05,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:06,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
844
888
[WARNING|trainer.py:803] 2025-04-26 19:00:07,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:07,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
865
[WARNING|trainer.py:803] 2025-04-26 19:00:08,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
889
845
[WARNING|trainer.py:803] 2025-04-26 19:00:09,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:09,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
866
[WARNING|trainer.py:803] 2025-04-26 19:00:10,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
846
890
[WARNING|trainer.py:803] 2025-04-26 19:00:11,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:11,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
867
[WARNING|trainer.py:803] 2025-04-26 19:00:12,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
847
891
[WARNING|trainer.py:803] 2025-04-26 19:00:13,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:13,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
868
[WARNING|trainer.py:803] 2025-04-26 19:00:14,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
892
848
[WARNING|trainer.py:803] 2025-04-26 19:00:15,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
869
[WARNING|trainer.py:803] 2025-04-26 19:00:15,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
893
[WARNING|trainer.py:803] 2025-04-26 19:00:16,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
849
[WARNING|trainer.py:803] 2025-04-26 19:00:17,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
870
[WARNING|trainer.py:803] 2025-04-26 19:00:18,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
894
[WARNING|trainer.py:803] 2025-04-26 19:00:18,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:19,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
850
871
895
[WARNING|trainer.py:803] 2025-04-26 19:00:20,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:20,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:21,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
851
[WARNING|trainer.py:803] 2025-04-26 19:00:22,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
896
872
[WARNING|trainer.py:803] 2025-04-26 19:00:23,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:23,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
852
[WARNING|trainer.py:803] 2025-04-26 19:00:24,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
897
873
[WARNING|trainer.py:803] 2025-04-26 19:00:25,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:25,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
853
[WARNING|trainer.py:803] 2025-04-26 19:00:26,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
898
874
[WARNING|trainer.py:803] 2025-04-26 19:00:27,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:28,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
854
899
[WARNING|trainer.py:803] 2025-04-26 19:00:29,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
875
[WARNING|trainer.py:803] 2025-04-26 19:00:29,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
855
[WARNING|trainer.py:803] 2025-04-26 19:00:30,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
900
[WARNING|trainer.py:803] 2025-04-26 19:00:31,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:31,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
876
856
901
[WARNING|trainer.py:803] 2025-04-26 19:00:32,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:33,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 19:00:33,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
877
902
857
[WARNING|trainer.py:803] 2025-04-26 19:00:34,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:34,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:35,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
878
903
[WARNING|trainer.py:803] 2025-04-26 19:00:36,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
858
[WARNING|trainer.py:803] 2025-04-26 19:00:37,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
904
[WARNING|trainer.py:803] 2025-04-26 19:00:38,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
879
[WARNING|trainer.py:803] 2025-04-26 19:00:38,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:38,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
859
905
[WARNING|trainer.py:803] 2025-04-26 19:00:40,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
880
[WARNING|trainer.py:803] 2025-04-26 19:00:40,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
906
[WARNING|trainer.py:803] 2025-04-26 19:00:41,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
860
[WARNING|trainer.py:803] 2025-04-26 19:00:41,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:42,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
907
881
[WARNING|trainer.py:803] 2025-04-26 19:00:43,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:43,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
861
908
[WARNING|trainer.py:803] 2025-04-26 19:00:44,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:44,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
882
909
[WARNING|trainer.py:803] 2025-04-26 19:00:45,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
862
[WARNING|trainer.py:803] 2025-04-26 19:00:46,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:46,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
883
910
[WARNING|trainer.py:803] 2025-04-26 19:00:47,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
863
[WARNING|trainer.py:803] 2025-04-26 19:00:47,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
911
[WARNING|trainer.py:803] 2025-04-26 19:00:48,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
884
[WARNING|trainer.py:803] 2025-04-26 19:00:49,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
864
[WARNING|trainer.py:803] 2025-04-26 19:00:49,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
912
[WARNING|trainer.py:803] 2025-04-26 19:00:50,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:51,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
885
913
[WARNING|trainer.py:803] 2025-04-26 19:00:51,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
865
[WARNING|trainer.py:803] 2025-04-26 19:00:52,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:52,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
914
886
[WARNING|trainer.py:803] 2025-04-26 19:00:54,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:54,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
866
915
[WARNING|trainer.py:803] 2025-04-26 19:00:55,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:00:55,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
887
867
916
[WARNING|trainer.py:803] 2025-04-26 19:00:56,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:00:57,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:57,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
917
868
888
[WARNING|trainer.py:803] 2025-04-26 19:00:58,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:00:59,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:00:59,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
918
[WARNING|trainer.py:803] 2025-04-26 19:01:00,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
889
869
919
[WARNING|trainer.py:803] 2025-04-26 19:01:01,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:01,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:01,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
920
870
890
[WARNING|trainer.py:803] 2025-04-26 19:01:03,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:03,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:01:03,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
921
871
[WARNING|trainer.py:803] 2025-04-26 19:01:04,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
891
[WARNING|trainer.py:803] 2025-04-26 19:01:05,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
922
[WARNING|trainer.py:803] 2025-04-26 19:01:06,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:06,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
923
892
872
[WARNING|trainer.py:803] 2025-04-26 19:01:08,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:08,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:08,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
924
893
873
[WARNING|trainer.py:803] 2025-04-26 19:01:10,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:10,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:10,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
925
894
874
[WARNING|trainer.py:803] 2025-04-26 19:01:11,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:12,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:12,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
926
[WARNING|trainer.py:803] 2025-04-26 19:01:13,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
927
895
875
[WARNING|trainer.py:803] 2025-04-26 19:01:15,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:15,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:15,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
928
876
896
[WARNING|trainer.py:803] 2025-04-26 19:01:16,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:17,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:01:17,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
929
[WARNING|trainer.py:803] 2025-04-26 19:01:18,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
877
897
930
[WARNING|trainer.py:803] 2025-04-26 19:01:19,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:01:19,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:19,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
878
931
898
[WARNING|trainer.py:803] 2025-04-26 19:01:21,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:21,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:22,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
932
879
[WARNING|trainer.py:803] 2025-04-26 19:01:23,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:23,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
899
933
[WARNING|trainer.py:803] 2025-04-26 19:01:24,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:24,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
880
934
900
[WARNING|trainer.py:803] 2025-04-26 19:01:25,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:26,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:26,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
935
881
901
[WARNING|trainer.py:803] 2025-04-26 19:01:27,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:27,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:01:28,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
936
902
882
[WARNING|trainer.py:803] 2025-04-26 19:01:29,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:30,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:30,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
937
[WARNING|trainer.py:803] 2025-04-26 19:01:31,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
883
903
938
[WARNING|trainer.py:803] 2025-04-26 19:01:32,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:32,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:01:32,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
939
904
884
[WARNING|trainer.py:803] 2025-04-26 19:01:34,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:34,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:34,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
940
905
[WARNING|trainer.py:803] 2025-04-26 19:01:35,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
885
[WARNING|trainer.py:803] 2025-04-26 19:01:35,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
941
[WARNING|trainer.py:803] 2025-04-26 19:01:36,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
906
[WARNING|trainer.py:803] 2025-04-26 19:01:37,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:37,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
886
942
907
[WARNING|trainer.py:803] 2025-04-26 19:01:38,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:39,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:39,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
943
908
887
[WARNING|trainer.py:803] 2025-04-26 19:01:40,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:41,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 19:01:41,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
944
909
[WARNING|trainer.py:803] 2025-04-26 19:01:42,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:42,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
888
945
910
[WARNING|trainer.py:803] 2025-04-26 19:01:43,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:44,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:44,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
946
889
911
[WARNING|trainer.py:803] 2025-04-26 19:01:45,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:45,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:46,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
947
912
890
[WARNING|trainer.py:803] 2025-04-26 19:01:47,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
948
[WARNING|trainer.py:803] 2025-04-26 19:01:48,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:48,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:49,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
913
949
891
[WARNING|trainer.py:803] 2025-04-26 19:01:50,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:01:50,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:50,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
914
950
[WARNING|trainer.py:803] 2025-04-26 19:01:51,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
892
[WARNING|trainer.py:803] 2025-04-26 19:01:52,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:52,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
915
951
[WARNING|trainer.py:803] 2025-04-26 19:01:53,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
893
[WARNING|trainer.py:803] 2025-04-26 19:01:54,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
916
[WARNING|trainer.py:803] 2025-04-26 19:01:54,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
952
[WARNING|trainer.py:803] 2025-04-26 19:01:55,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:55,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
894
917
953
[WARNING|trainer.py:803] 2025-04-26 19:01:56,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:57,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:57,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
954
918
895
[WARNING|trainer.py:803] 2025-04-26 19:01:58,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:01:58,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:01:59,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
955
919
[WARNING|trainer.py:803] 2025-04-26 19:02:00,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:02:00,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
896
956
920
[WARNING|trainer.py:803] 2025-04-26 19:02:01,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:02:02,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:02,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
957
897
921
[WARNING|trainer.py:803] 2025-04-26 19:02:03,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:02:04,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:04,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
958
[WARNING|trainer.py:803] 2025-04-26 19:02:05,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
922
898
959
[WARNING|trainer.py:803] 2025-04-26 19:02:05,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:02:06,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:06,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
923
960
[WARNING|trainer.py:803] 2025-04-26 19:02:07,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
899
[WARNING|trainer.py:803] 2025-04-26 19:02:08,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
961
[WARNING|trainer.py:803] 2025-04-26 19:02:08,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
924
[WARNING|trainer.py:803] 2025-04-26 19:02:09,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
900
[WARNING|trainer.py:803] 2025-04-26 19:02:10,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
962
[WARNING|trainer.py:803] 2025-04-26 19:02:10,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
925
[WARNING|trainer.py:803] 2025-04-26 19:02:11,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
963
901
[WARNING|trainer.py:803] 2025-04-26 19:02:11,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:12,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:12,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
926
964
902
[WARNING|trainer.py:803] 2025-04-26 19:02:13,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:14,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:14,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
927
965
[WARNING|trainer.py:803] 2025-04-26 19:02:15,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:02:15,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
903
966
928
[WARNING|trainer.py:803] 2025-04-26 19:02:16,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:17,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:17,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
904
967
929
[WARNING|trainer.py:803] 2025-04-26 19:02:18,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:02:18,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:02:19,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
968
905
930
[WARNING|trainer.py:803] 2025-04-26 19:02:20,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:20,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:20,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
969
906
[WARNING|trainer.py:803] 2025-04-26 19:02:21,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
931
[WARNING|trainer.py:803] 2025-04-26 19:02:21,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:22,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
907
970
932
[WARNING|trainer.py:803] 2025-04-26 19:02:23,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:23,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
971
[WARNING|trainer.py:803] 2025-04-26 19:02:24,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
908
[WARNING|trainer.py:803] 2025-04-26 19:02:25,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
933
[WARNING|trainer.py:803] 2025-04-26 19:02:25,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
972
[WARNING|trainer.py:803] 2025-04-26 19:02:26,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
909
[WARNING|trainer.py:803] 2025-04-26 19:02:26,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
934
[WARNING|trainer.py:803] 2025-04-26 19:02:27,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
973
[WARNING|trainer.py:803] 2025-04-26 19:02:27,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
910
[WARNING|trainer.py:803] 2025-04-26 19:02:28,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:29,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
935
974
[WARNING|trainer.py:803] 2025-04-26 19:02:29,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:29,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
911
[WARNING|trainer.py:803] 2025-04-26 19:02:30,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
936
975
[WARNING|trainer.py:803] 2025-04-26 19:02:31,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
912
[WARNING|trainer.py:803] 2025-04-26 19:02:32,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
937
[WARNING|trainer.py:803] 2025-04-26 19:02:32,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
976
[WARNING|trainer.py:803] 2025-04-26 19:02:33,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
913
[WARNING|trainer.py:803] 2025-04-26 19:02:33,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
938
[WARNING|trainer.py:803] 2025-04-26 19:02:34,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
977
[WARNING|trainer.py:803] 2025-04-26 19:02:35,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
914
[WARNING|trainer.py:803] 2025-04-26 19:02:35,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
939
978
[WARNING|trainer.py:803] 2025-04-26 19:02:36,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:36,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:36,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
915
979
[WARNING|trainer.py:803] 2025-04-26 19:02:37,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
940
[WARNING|trainer.py:803] 2025-04-26 19:02:38,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:02:38,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
916
941
980
[WARNING|trainer.py:803] 2025-04-26 19:02:39,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:40,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
917
981
[WARNING|trainer.py:803] 2025-04-26 19:02:41,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
942
[WARNING|trainer.py:803] 2025-04-26 19:02:41,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
918
[WARNING|trainer.py:803] 2025-04-26 19:02:42,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
982
[WARNING|trainer.py:803] 2025-04-26 19:02:42,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
943
919
[WARNING|trainer.py:803] 2025-04-26 19:02:44,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
983
[WARNING|trainer.py:803] 2025-04-26 19:02:44,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
944
[WARNING|trainer.py:803] 2025-04-26 19:02:45,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
920
[WARNING|trainer.py:803] 2025-04-26 19:02:46,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:46,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
984
945
[WARNING|trainer.py:803] 2025-04-26 19:02:47,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
921
[WARNING|trainer.py:803] 2025-04-26 19:02:47,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
985
[WARNING|trainer.py:803] 2025-04-26 19:02:48,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
946
[WARNING|trainer.py:803] 2025-04-26 19:02:48,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
922
[WARNING|trainer.py:803] 2025-04-26 19:02:49,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:50,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
986
[WARNING|trainer.py:803] 2025-04-26 19:02:50,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
923
947
987
[WARNING|trainer.py:803] 2025-04-26 19:02:51,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:51,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:52,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
948
988
924
[WARNING|trainer.py:803] 2025-04-26 19:02:53,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:02:54,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:02:54,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
949
989
925
[WARNING|trainer.py:803] 2025-04-26 19:02:55,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:55,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:55,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
990
950
926
[WARNING|trainer.py:803] 2025-04-26 19:02:56,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:57,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:57,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
991
951
927
[WARNING|trainer.py:803] 2025-04-26 19:02:58,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:02:59,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:02:59,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
992
[WARNING|trainer.py:803] 2025-04-26 19:03:00,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
952
928
993
[WARNING|trainer.py:803] 2025-04-26 19:03:01,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:03:01,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:01,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
953
929
994
[WARNING|trainer.py:803] 2025-04-26 19:03:03,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:03:03,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:03,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
954
930
995
[WARNING|trainer.py:803] 2025-04-26 19:03:04,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:04,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:05,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
955
931
996
[WARNING|trainer.py:803] 2025-04-26 19:03:06,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:06,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:06,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
997
956
932
[WARNING|trainer.py:803] 2025-04-26 19:03:08,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:08,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:08,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
998
957
933
[WARNING|trainer.py:803] 2025-04-26 19:03:10,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:10,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:10,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
999
958
934
[WARNING|trainer.py:803] 2025-04-26 19:03:11,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:11,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:12,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1000
959
935
[WARNING|trainer.py:803] 2025-04-26 19:03:13,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:13,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:13,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1001
960
936
[WARNING|trainer.py:803] 2025-04-26 19:03:15,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:15,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:15,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1002
961
937
[WARNING|trainer.py:803] 2025-04-26 19:03:16,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:17,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1003
[WARNING|trainer.py:803] 2025-04-26 19:03:17,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
962
[WARNING|trainer.py:803] 2025-04-26 19:03:18,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
938
[WARNING|trainer.py:803] 2025-04-26 19:03:18,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:19,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1004
963
939
[WARNING|trainer.py:803] 2025-04-26 19:03:20,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:20,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1005
[WARNING|trainer.py:803] 2025-04-26 19:03:20,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
964
[WARNING|trainer.py:803] 2025-04-26 19:03:21,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
940
[WARNING|trainer.py:803] 2025-04-26 19:03:22,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1006
[WARNING|trainer.py:803] 2025-04-26 19:03:22,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
965
[WARNING|trainer.py:803] 2025-04-26 19:03:23,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
941
[WARNING|trainer.py:803] 2025-04-26 19:03:23,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1007
[WARNING|trainer.py:803] 2025-04-26 19:03:24,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
966
[WARNING|trainer.py:803] 2025-04-26 19:03:24,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1008
[WARNING|trainer.py:803] 2025-04-26 19:03:25,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
942
[WARNING|trainer.py:803] 2025-04-26 19:03:26,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
967
[WARNING|trainer.py:803] 2025-04-26 19:03:26,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1009
[WARNING|trainer.py:803] 2025-04-26 19:03:27,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
943
[WARNING|trainer.py:803] 2025-04-26 19:03:27,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
968
[WARNING|trainer.py:803] 2025-04-26 19:03:28,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1010
[WARNING|trainer.py:803] 2025-04-26 19:03:29,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
944
[WARNING|trainer.py:803] 2025-04-26 19:03:29,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
969
[WARNING|trainer.py:803] 2025-04-26 19:03:30,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1011
[WARNING|trainer.py:803] 2025-04-26 19:03:31,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
945
[WARNING|trainer.py:803] 2025-04-26 19:03:31,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1012
[WARNING|trainer.py:803] 2025-04-26 19:03:31,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
970
[WARNING|trainer.py:803] 2025-04-26 19:03:32,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
946
[WARNING|trainer.py:803] 2025-04-26 19:03:33,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1013
[WARNING|trainer.py:803] 2025-04-26 19:03:33,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
971
[WARNING|trainer.py:803] 2025-04-26 19:03:34,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:34,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
947
1014
972
[WARNING|trainer.py:803] 2025-04-26 19:03:36,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:36,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:03:36,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1015
948
973
[WARNING|trainer.py:803] 2025-04-26 19:03:37,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:38,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:38,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1016
949
974
[WARNING|trainer.py:803] 2025-04-26 19:03:39,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:39,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:03:40,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1017
[WARNING|trainer.py:803] 2025-04-26 19:03:41,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
950
975
[WARNING|trainer.py:803] 2025-04-26 19:03:41,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1018
[WARNING|trainer.py:803] 2025-04-26 19:03:42,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
951
[WARNING|trainer.py:803] 2025-04-26 19:03:43,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
976
1019
[WARNING|trainer.py:803] 2025-04-26 19:03:44,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:44,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:44,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
952
977
[WARNING|trainer.py:803] 2025-04-26 19:03:45,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1020
[WARNING|trainer.py:803] 2025-04-26 19:03:46,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
953
[WARNING|trainer.py:803] 2025-04-26 19:03:46,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
978
[WARNING|trainer.py:803] 2025-04-26 19:03:47,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1021
[WARNING|trainer.py:803] 2025-04-26 19:03:47,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
954
[WARNING|trainer.py:803] 2025-04-26 19:03:48,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
979
[WARNING|trainer.py:803] 2025-04-26 19:03:49,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:49,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1022
955
[WARNING|trainer.py:803] 2025-04-26 19:03:50,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
980
[WARNING|trainer.py:803] 2025-04-26 19:03:51,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1023
[WARNING|trainer.py:803] 2025-04-26 19:03:51,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
956
[WARNING|trainer.py:803] 2025-04-26 19:03:52,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
981
[WARNING|trainer.py:803] 2025-04-26 19:03:52,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:53,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1024
957
982
[WARNING|trainer.py:803] 2025-04-26 19:03:54,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:54,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:55,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1025
958
[WARNING|trainer.py:803] 2025-04-26 19:03:56,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:56,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
983
1026
959
[WARNING|trainer.py:803] 2025-04-26 19:03:57,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:57,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:03:58,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
984
960
1027
[WARNING|trainer.py:803] 2025-04-26 19:03:59,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:03:59,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:03:59,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1028
985
961
[WARNING|trainer.py:803] 2025-04-26 19:04:01,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:01,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:01,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1029
962
[WARNING|trainer.py:803] 2025-04-26 19:04:02,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
986
[WARNING|trainer.py:803] 2025-04-26 19:04:03,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1030
[WARNING|trainer.py:803] 2025-04-26 19:04:03,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
963
987
[WARNING|trainer.py:803] 2025-04-26 19:04:04,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:05,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1031
[WARNING|trainer.py:803] 2025-04-26 19:04:05,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
964
[WARNING|trainer.py:803] 2025-04-26 19:04:06,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
988
[WARNING|trainer.py:803] 2025-04-26 19:04:06,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1032
[WARNING|trainer.py:803] 2025-04-26 19:04:07,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:04:07,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
965
989
1033
[WARNING|trainer.py:803] 2025-04-26 19:04:08,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:08,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:09,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
966
990
1034
[WARNING|trainer.py:803] 2025-04-26 19:04:10,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:10,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:10,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
967
991
1035
[WARNING|trainer.py:803] 2025-04-26 19:04:11,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:12,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:12,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
968
1036
992
[WARNING|trainer.py:803] 2025-04-26 19:04:13,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:14,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:14,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
969
1037
993
[WARNING|trainer.py:803] 2025-04-26 19:04:15,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:16,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:16,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1038
970
994
[WARNING|trainer.py:803] 2025-04-26 19:04:17,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:17,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:17,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1039
971
[WARNING|trainer.py:803] 2025-04-26 19:04:19,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
995
[WARNING|trainer.py:803] 2025-04-26 19:04:19,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1040
[WARNING|trainer.py:803] 2025-04-26 19:04:20,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
972
[WARNING|trainer.py:803] 2025-04-26 19:04:20,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
996
1041
[WARNING|trainer.py:803] 2025-04-26 19:04:21,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:22,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:22,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
973
997
[WARNING|trainer.py:803] 2025-04-26 19:04:23,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1042
[WARNING|trainer.py:803] 2025-04-26 19:04:23,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
974
[WARNING|trainer.py:803] 2025-04-26 19:04:24,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:24,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1043
998
[WARNING|trainer.py:803] 2025-04-26 19:04:25,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:25,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
975
1044
999
[WARNING|trainer.py:803] 2025-04-26 19:04:27,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:27,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:27,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
976
1000
1045
[WARNING|trainer.py:803] 2025-04-26 19:04:29,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:29,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:29,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
977
1046
1001
[WARNING|trainer.py:803] 2025-04-26 19:04:30,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:31,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:31,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
978
1047
1002
[WARNING|trainer.py:803] 2025-04-26 19:04:32,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:33,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:33,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
979
1003
1048
[WARNING|trainer.py:803] 2025-04-26 19:04:34,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:35,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:35,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
980
1049
1004
[WARNING|trainer.py:803] 2025-04-26 19:04:36,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:36,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:37,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
981
1050
1005
[WARNING|trainer.py:803] 2025-04-26 19:04:38,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:38,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:38,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
982
1051
[WARNING|trainer.py:803] 2025-04-26 19:04:40,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1006
[WARNING|trainer.py:803] 2025-04-26 19:04:40,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:40,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1052
983
1007
[WARNING|trainer.py:803] 2025-04-26 19:04:41,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:42,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:42,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1053
1008
984
[WARNING|trainer.py:803] 2025-04-26 19:04:43,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:44,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1054
[WARNING|trainer.py:803] 2025-04-26 19:04:44,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1009
[WARNING|trainer.py:803] 2025-04-26 19:04:45,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
985
[WARNING|trainer.py:803] 2025-04-26 19:04:46,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1055
[WARNING|trainer.py:803] 2025-04-26 19:04:46,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:47,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1010
1056
986
[WARNING|trainer.py:803] 2025-04-26 19:04:48,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:48,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:04:48,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1011
1057
987
[WARNING|trainer.py:803] 2025-04-26 19:04:49,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:50,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:50,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1012
988
[WARNING|trainer.py:803] 2025-04-26 19:04:51,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1058
[WARNING|trainer.py:803] 2025-04-26 19:04:52,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:04:52,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1013
989
1059
[WARNING|trainer.py:803] 2025-04-26 19:04:53,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:53,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:53,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1014
990
1060
[WARNING|trainer.py:803] 2025-04-26 19:04:55,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:55,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:56,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1015
991
1061
[WARNING|trainer.py:803] 2025-04-26 19:04:57,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:57,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:04:57,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1016
1062
992
[WARNING|trainer.py:803] 2025-04-26 19:04:59,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:04:59,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:04:59,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1063
993
1017
[WARNING|trainer.py:803] 2025-04-26 19:05:00,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:01,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:01,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1064
994
1018
[WARNING|trainer.py:803] 2025-04-26 19:05:02,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:02,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1065
[WARNING|trainer.py:803] 2025-04-26 19:05:03,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
995
[WARNING|trainer.py:803] 2025-04-26 19:05:04,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1019
[WARNING|trainer.py:803] 2025-04-26 19:05:05,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1066
[WARNING|trainer.py:803] 2025-04-26 19:05:05,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
996
[WARNING|trainer.py:803] 2025-04-26 19:05:05,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:06,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1020
1067
997
[WARNING|trainer.py:803] 2025-04-26 19:05:07,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:08,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:08,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1021
1068
998
[WARNING|trainer.py:803] 2025-04-26 19:05:09,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:09,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:10,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
999
1022
1069
[WARNING|trainer.py:803] 2025-04-26 19:05:11,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:05:12,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:12,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1070
1000
1023
[WARNING|trainer.py:803] 2025-04-26 19:05:13,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:13,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:13,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1071
[WARNING|trainer.py:803] 2025-04-26 19:05:14,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1001
1024
1072
[WARNING|trainer.py:803] 2025-04-26 19:05:15,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:16,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:05:16,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1002
1073
1025
[WARNING|trainer.py:803] 2025-04-26 19:05:17,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:17,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:18,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1074
1003
1026
[WARNING|trainer.py:803] 2025-04-26 19:05:19,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:19,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:19,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1075
[WARNING|trainer.py:803] 2025-04-26 19:05:20,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1004
1027
[WARNING|trainer.py:803] 2025-04-26 19:05:21,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1076
[WARNING|trainer.py:803] 2025-04-26 19:05:22,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1005
[WARNING|trainer.py:803] 2025-04-26 19:05:22,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1028
1077
[WARNING|trainer.py:803] 2025-04-26 19:05:23,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:23,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:24,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1006
1029
1078
[WARNING|trainer.py:803] 2025-04-26 19:05:25,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:25,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:25,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1007
1030
1079
[WARNING|trainer.py:803] 2025-04-26 19:05:27,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:27,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:27,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1008
1031
1080
[WARNING|trainer.py:803] 2025-04-26 19:05:28,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:29,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:29,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1009
1032
1081
[WARNING|trainer.py:803] 2025-04-26 19:05:30,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:30,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:31,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1033
1010
1082
[WARNING|trainer.py:803] 2025-04-26 19:05:32,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:32,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:32,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1034
1083
1011
[WARNING|trainer.py:803] 2025-04-26 19:05:34,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:34,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:34,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1084
1012
1035
[WARNING|trainer.py:803] 2025-04-26 19:05:36,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:36,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:36,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1085
1013
1036
[WARNING|trainer.py:803] 2025-04-26 19:05:37,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:37,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1086
[WARNING|trainer.py:803] 2025-04-26 19:05:38,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:39,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1014
1037
[WARNING|trainer.py:803] 2025-04-26 19:05:39,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1087
[WARNING|trainer.py:803] 2025-04-26 19:05:40,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:40,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1015
1038
[WARNING|trainer.py:803] 2025-04-26 19:05:41,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1088
[WARNING|trainer.py:803] 2025-04-26 19:05:42,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1016
[WARNING|trainer.py:803] 2025-04-26 19:05:42,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1039
[WARNING|trainer.py:803] 2025-04-26 19:05:43,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1089
[WARNING|trainer.py:803] 2025-04-26 19:05:44,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:44,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1017
1040
1090
[WARNING|trainer.py:803] 2025-04-26 19:05:45,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:45,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:46,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1041
1091
1018
[WARNING|trainer.py:803] 2025-04-26 19:05:47,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:47,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:05:47,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1092
1042
1019
[WARNING|trainer.py:803] 2025-04-26 19:05:49,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:49,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:05:49,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1093
1043
1020
[WARNING|trainer.py:803] 2025-04-26 19:05:51,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:51,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:52,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1094
1044
[WARNING|trainer.py:803] 2025-04-26 19:05:53,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1021
[WARNING|trainer.py:803] 2025-04-26 19:05:53,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1095
[WARNING|trainer.py:803] 2025-04-26 19:05:54,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:54,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1045
1022
1096
[WARNING|trainer.py:803] 2025-04-26 19:05:56,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:05:56,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:56,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1046
1023
1097
[WARNING|trainer.py:803] 2025-04-26 19:05:57,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:58,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:05:58,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1047
1098
1024
[WARNING|trainer.py:803] 2025-04-26 19:05:59,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:05:59,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:00,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1099
1048
[WARNING|trainer.py:803] 2025-04-26 19:06:01,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1025
[WARNING|trainer.py:803] 2025-04-26 19:06:01,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1100
[WARNING|trainer.py:803] 2025-04-26 19:06:02,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1049
[WARNING|trainer.py:803] 2025-04-26 19:06:03,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1026
[WARNING|trainer.py:803] 2025-04-26 19:06:03,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1101
[WARNING|trainer.py:803] 2025-04-26 19:06:04,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1050
[WARNING|trainer.py:803] 2025-04-26 19:06:04,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1027
[WARNING|trainer.py:803] 2025-04-26 19:06:05,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1102
[WARNING|trainer.py:803] 2025-04-26 19:06:06,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1051
[WARNING|trainer.py:803] 2025-04-26 19:06:06,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1028
1103
[WARNING|trainer.py:803] 2025-04-26 19:06:07,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:06:07,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:08,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1052
1029
1104
[WARNING|trainer.py:803] 2025-04-26 19:06:09,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:09,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:09,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1105
1030
1053
[WARNING|trainer.py:803] 2025-04-26 19:06:11,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:06:11,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:11,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1106
1031
1054
[WARNING|trainer.py:803] 2025-04-26 19:06:12,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:06:13,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1107
[WARNING|trainer.py:803] 2025-04-26 19:06:13,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1032
[WARNING|trainer.py:803] 2025-04-26 19:06:14,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1055
[WARNING|trainer.py:803] 2025-04-26 19:06:14,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:15,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1108
1033
[WARNING|trainer.py:803] 2025-04-26 19:06:16,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1056
[WARNING|trainer.py:803] 2025-04-26 19:06:16,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:17,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1109
1034
[WARNING|trainer.py:803] 2025-04-26 19:06:18,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1057
[WARNING|trainer.py:803] 2025-04-26 19:06:18,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1110
[WARNING|trainer.py:803] 2025-04-26 19:06:18,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:19,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1035
1111
[WARNING|trainer.py:803] 2025-04-26 19:06:20,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1058
[WARNING|trainer.py:803] 2025-04-26 19:06:21,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:21,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1036
1112
1059
[WARNING|trainer.py:803] 2025-04-26 19:06:22,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:22,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:23,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1113
1037
[WARNING|trainer.py:803] 2025-04-26 19:06:24,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:24,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1114
1060
1038
[WARNING|trainer.py:803] 2025-04-26 19:06:25,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:25,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1115
[WARNING|trainer.py:803] 2025-04-26 19:06:26,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1061
[WARNING|trainer.py:803] 2025-04-26 19:06:26,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1039
[WARNING|trainer.py:803] 2025-04-26 19:06:27,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1116
[WARNING|trainer.py:803] 2025-04-26 19:06:27,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1062
[WARNING|trainer.py:803] 2025-04-26 19:06:28,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1040
[WARNING|trainer.py:803] 2025-04-26 19:06:29,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1117
[WARNING|trainer.py:803] 2025-04-26 19:06:29,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1063
[WARNING|trainer.py:803] 2025-04-26 19:06:30,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1041
[WARNING|trainer.py:803] 2025-04-26 19:06:30,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1118
[WARNING|trainer.py:803] 2025-04-26 19:06:31,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:31,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1064
1119
1042
[WARNING|trainer.py:803] 2025-04-26 19:06:32,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:33,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:33,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1065
1120
[WARNING|trainer.py:803] 2025-04-26 19:06:34,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1043
[WARNING|trainer.py:803] 2025-04-26 19:06:35,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:35,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1066
1121
[WARNING|trainer.py:803] 2025-04-26 19:06:36,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1044
[WARNING|trainer.py:803] 2025-04-26 19:06:36,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:06:37,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1122
1067
[WARNING|trainer.py:803] 2025-04-26 19:06:38,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:39,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1045
[WARNING|trainer.py:803] 2025-04-26 19:06:39,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1123
1068
1046
[WARNING|trainer.py:803] 2025-04-26 19:06:40,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:41,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1124
[WARNING|trainer.py:803] 2025-04-26 19:06:41,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:42,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1047
1069
1125
[WARNING|trainer.py:803] 2025-04-26 19:06:43,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:43,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:06:43,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1070
1126
1048
[WARNING|trainer.py:803] 2025-04-26 19:06:45,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:45,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:45,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1127
1071
1049
[WARNING|trainer.py:803] 2025-04-26 19:06:46,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:46,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1128
[WARNING|trainer.py:803] 2025-04-26 19:06:47,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1072
[WARNING|trainer.py:803] 2025-04-26 19:06:48,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:48,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1050
1129
[WARNING|trainer.py:803] 2025-04-26 19:06:49,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1073
[WARNING|trainer.py:803] 2025-04-26 19:06:49,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:50,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1130
1051
1074
[WARNING|trainer.py:803] 2025-04-26 19:06:51,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:51,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1131
[WARNING|trainer.py:803] 2025-04-26 19:06:52,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1052
[WARNING|trainer.py:803] 2025-04-26 19:06:52,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1075
[WARNING|trainer.py:803] 2025-04-26 19:06:53,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1132
[WARNING|trainer.py:803] 2025-04-26 19:06:53,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:06:54,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1053
1133
1076
[WARNING|trainer.py:803] 2025-04-26 19:06:55,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:55,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:06:55,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1054
1134
1077
[WARNING|trainer.py:803] 2025-04-26 19:06:57,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:57,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:57,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1135
1055
1078
[WARNING|trainer.py:803] 2025-04-26 19:06:58,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:06:59,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1136
[WARNING|trainer.py:803] 2025-04-26 19:06:59,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1056
[WARNING|trainer.py:803] 2025-04-26 19:07:00,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1079
[WARNING|trainer.py:803] 2025-04-26 19:07:00,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1137
[WARNING|trainer.py:803] 2025-04-26 19:07:01,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1057
[WARNING|trainer.py:803] 2025-04-26 19:07:02,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:02,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1138
1080
[WARNING|trainer.py:803] 2025-04-26 19:07:03,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:03,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1058
1139
1081
[WARNING|trainer.py:803] 2025-04-26 19:07:05,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:05,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:05,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1140
1059
1082
[WARNING|trainer.py:803] 2025-04-26 19:07:06,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:06,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:07,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1141
[WARNING|trainer.py:803] 2025-04-26 19:07:08,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1060
1083
1142
[WARNING|trainer.py:803] 2025-04-26 19:07:09,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:09,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:09,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1061
1084
1143
[WARNING|trainer.py:803] 2025-04-26 19:07:11,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:11,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:11,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1062
1144
1085
[WARNING|trainer.py:803] 2025-04-26 19:07:12,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:12,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:13,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1145
1063
1086
[WARNING|trainer.py:803] 2025-04-26 19:07:14,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:14,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:14,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1146
1087
[WARNING|trainer.py:803] 2025-04-26 19:07:15,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1064
1147
[WARNING|trainer.py:803] 2025-04-26 19:07:16,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:16,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:17,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1065
1088
1148
[WARNING|trainer.py:803] 2025-04-26 19:07:18,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:18,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:18,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1149
1066
1089
[WARNING|trainer.py:803] 2025-04-26 19:07:20,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:20,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:20,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1150
1090
[WARNING|trainer.py:803] 2025-04-26 19:07:21,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1067
1151
[WARNING|trainer.py:803] 2025-04-26 19:07:22,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:22,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:23,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1091
1152
1068
[WARNING|trainer.py:803] 2025-04-26 19:07:24,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:07:24,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:25,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1092
1153
[WARNING|trainer.py:803] 2025-04-26 19:07:26,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:26,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1069
1154
[WARNING|trainer.py:803] 2025-04-26 19:07:27,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:27,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1093
1070
1155
[WARNING|trainer.py:803] 2025-04-26 19:07:28,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:29,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:29,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1094
1071
1156
[WARNING|trainer.py:803] 2025-04-26 19:07:30,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:30,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:30,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1157
1095
1072
[WARNING|trainer.py:803] 2025-04-26 19:07:32,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:32,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:32,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1158
1073
1096
[WARNING|trainer.py:803] 2025-04-26 19:07:34,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:34,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:34,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1159
1074
[WARNING|trainer.py:803] 2025-04-26 19:07:35,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1097
[WARNING|trainer.py:803] 2025-04-26 19:07:36,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1160
[WARNING|trainer.py:803] 2025-04-26 19:07:36,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1075
[WARNING|trainer.py:803] 2025-04-26 19:07:36,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1098
1161
[WARNING|trainer.py:803] 2025-04-26 19:07:37,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:38,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:38,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1076
1162
1099
[WARNING|trainer.py:803] 2025-04-26 19:07:39,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:39,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:07:39,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1163
1077
1100
[WARNING|trainer.py:803] 2025-04-26 19:07:41,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:41,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:41,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1164
1078
[WARNING|trainer.py:803] 2025-04-26 19:07:42,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1101
[WARNING|trainer.py:803] 2025-04-26 19:07:43,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1165
[WARNING|trainer.py:803] 2025-04-26 19:07:44,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:44,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1079
1166
1102
[WARNING|trainer.py:803] 2025-04-26 19:07:45,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:46,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:46,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1167
1080
1103
[WARNING|trainer.py:803] 2025-04-26 19:07:47,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:47,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:48,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1168
1081
1104
[WARNING|trainer.py:803] 2025-04-26 19:07:49,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:49,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1169
[WARNING|trainer.py:803] 2025-04-26 19:07:49,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:50,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1082
1105
1170
[WARNING|trainer.py:803] 2025-04-26 19:07:51,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:51,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:52,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1106
1083
1171
[WARNING|trainer.py:803] 2025-04-26 19:07:53,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:07:53,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:53,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1107
1084
1172
[WARNING|trainer.py:803] 2025-04-26 19:07:54,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:55,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:55,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1173
1085
1108
[WARNING|trainer.py:803] 2025-04-26 19:07:57,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:57,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:07:57,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1086
1174
1109
[WARNING|trainer.py:803] 2025-04-26 19:07:58,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:58,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:07:59,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1087
1175
1110
[WARNING|trainer.py:803] 2025-04-26 19:08:00,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:00,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:00,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1176
1111
[WARNING|trainer.py:803] 2025-04-26 19:08:02,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1088
[WARNING|trainer.py:803] 2025-04-26 19:08:02,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1177
[WARNING|trainer.py:803] 2025-04-26 19:08:03,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1112
[WARNING|trainer.py:803] 2025-04-26 19:08:03,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1089
1178
[WARNING|trainer.py:803] 2025-04-26 19:08:04,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:04,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:05,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1113
1179
1090
[WARNING|trainer.py:803] 2025-04-26 19:08:06,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:06,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:06,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1114
1180
1091
[WARNING|trainer.py:803] 2025-04-26 19:08:07,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:08,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:08,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1115
1181
[WARNING|trainer.py:803] 2025-04-26 19:08:09,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1092
[WARNING|trainer.py:803] 2025-04-26 19:08:09,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1116
[WARNING|trainer.py:803] 2025-04-26 19:08:10,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1182
[WARNING|trainer.py:803] 2025-04-26 19:08:11,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:11,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1093
1183
1117
[WARNING|trainer.py:803] 2025-04-26 19:08:12,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:13,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:13,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1184
1094
1118
[WARNING|trainer.py:803] 2025-04-26 19:08:14,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:14,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:08:15,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1185
1095
1119
[WARNING|trainer.py:803] 2025-04-26 19:08:16,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:16,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:16,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1186
1096
[WARNING|trainer.py:803] 2025-04-26 19:08:18,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1120
[WARNING|trainer.py:803] 2025-04-26 19:08:18,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1187
[WARNING|trainer.py:803] 2025-04-26 19:08:19,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:19,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1097
1121
1188
[WARNING|trainer.py:803] 2025-04-26 19:08:20,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:20,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:21,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1098
1189
1122
[WARNING|trainer.py:803] 2025-04-26 19:08:22,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:23,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:23,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1099
[WARNING|trainer.py:803] 2025-04-26 19:08:24,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1190
1123
[WARNING|trainer.py:803] 2025-04-26 19:08:25,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1100
[WARNING|trainer.py:803] 2025-04-26 19:08:25,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1191
[WARNING|trainer.py:803] 2025-04-26 19:08:26,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1124
[WARNING|trainer.py:803] 2025-04-26 19:08:26,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:27,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1192
1101
1125
[WARNING|trainer.py:803] 2025-04-26 19:08:28,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:28,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1193
[WARNING|trainer.py:803] 2025-04-26 19:08:28,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:29,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1126
1102
1194
[WARNING|trainer.py:803] 2025-04-26 19:08:30,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:30,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:31,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1127
1103
1195
[WARNING|trainer.py:803] 2025-04-26 19:08:32,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:32,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:32,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1128
1104
1196
[WARNING|trainer.py:803] 2025-04-26 19:08:33,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:34,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:34,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1129
1197
1105
[WARNING|trainer.py:803] 2025-04-26 19:08:35,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:35,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:35,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1130
1198
1106
[WARNING|trainer.py:803] 2025-04-26 19:08:37,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:37,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:37,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1131
1199
1107
[WARNING|trainer.py:803] 2025-04-26 19:08:38,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:08:39,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:39,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1200
1132
[WARNING|trainer.py:803] 2025-04-26 19:08:40,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:40,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1108
1133
[WARNING|trainer.py:803] 2025-04-26 19:08:41,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1201
[WARNING|trainer.py:803] 2025-04-26 19:08:42,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1109
[WARNING|trainer.py:803] 2025-04-26 19:08:42,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1134
[WARNING|trainer.py:803] 2025-04-26 19:08:43,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:44,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1202
1110
1135
[WARNING|trainer.py:803] 2025-04-26 19:08:45,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:45,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:08:45,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1203
1111
1136
[WARNING|trainer.py:803] 2025-04-26 19:08:47,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:47,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:47,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1112
1204
1137
[WARNING|trainer.py:803] 2025-04-26 19:08:49,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:49,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:08:49,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1113
1138
1205
[WARNING|trainer.py:803] 2025-04-26 19:08:50,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:08:51,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:08:51,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1114
1139
[WARNING|trainer.py:803] 2025-04-26 19:08:52,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1206
[WARNING|trainer.py:803] 2025-04-26 19:08:52,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1115
[WARNING|trainer.py:803] 2025-04-26 19:08:53,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1140
[WARNING|trainer.py:803] 2025-04-26 19:08:54,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1207
[WARNING|trainer.py:803] 2025-04-26 19:08:54,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1116
[WARNING|trainer.py:803] 2025-04-26 19:08:55,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1141
[WARNING|trainer.py:803] 2025-04-26 19:08:56,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:56,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1208
1117
1142
[WARNING|trainer.py:803] 2025-04-26 19:08:57,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:08:58,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:08:58,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1209
1143
1118
[WARNING|trainer.py:803] 2025-04-26 19:08:59,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:00,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:00,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1210
1144
1119
[WARNING|trainer.py:803] 2025-04-26 19:09:01,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:01,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:01,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1145
1211
1120
[WARNING|trainer.py:803] 2025-04-26 19:09:03,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:03,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:04,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1146
1212
1121
[WARNING|trainer.py:803] 2025-04-26 19:09:05,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:05,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:05,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1147
1122
1213
[WARNING|trainer.py:803] 2025-04-26 19:09:07,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:07,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:07,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1148
[WARNING|trainer.py:803] 2025-04-26 19:09:08,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1214
1123
1149
[WARNING|trainer.py:803] 2025-04-26 19:09:09,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:10,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:10,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1124
1215
1150
[WARNING|trainer.py:803] 2025-04-26 19:09:11,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:11,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:12,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1125
1151
1216
[WARNING|trainer.py:803] 2025-04-26 19:09:13,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:14,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:14,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1126
1152
[WARNING|trainer.py:803] 2025-04-26 19:09:15,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1217
[WARNING|trainer.py:803] 2025-04-26 19:09:15,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1127
[WARNING|trainer.py:803] 2025-04-26 19:09:16,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1153
[WARNING|trainer.py:803] 2025-04-26 19:09:16,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1218
[WARNING|trainer.py:803] 2025-04-26 19:09:17,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1128
[WARNING|trainer.py:803] 2025-04-26 19:09:18,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1154
[WARNING|trainer.py:803] 2025-04-26 19:09:18,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1219
[WARNING|trainer.py:803] 2025-04-26 19:09:19,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1129
[WARNING|trainer.py:803] 2025-04-26 19:09:20,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:20,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1155
1130
[WARNING|trainer.py:803] 2025-04-26 19:09:21,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1220
[WARNING|trainer.py:803] 2025-04-26 19:09:22,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1156
[WARNING|trainer.py:803] 2025-04-26 19:09:22,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:22,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1131
1221
[WARNING|trainer.py:803] 2025-04-26 19:09:23,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1157
[WARNING|trainer.py:803] 2025-04-26 19:09:23,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1132
[WARNING|trainer.py:803] 2025-04-26 19:09:24,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1222
[WARNING|trainer.py:803] 2025-04-26 19:09:25,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1158
[WARNING|trainer.py:803] 2025-04-26 19:09:26,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1133
[WARNING|trainer.py:803] 2025-04-26 19:09:26,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:27,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1159
1223
1134
[WARNING|trainer.py:803] 2025-04-26 19:09:28,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:28,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:28,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1160
1135
[WARNING|trainer.py:803] 2025-04-26 19:09:29,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1224
[WARNING|trainer.py:803] 2025-04-26 19:09:30,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:30,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1161
1136
[WARNING|trainer.py:803] 2025-04-26 19:09:31,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1225
[WARNING|trainer.py:803] 2025-04-26 19:09:32,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1162
[WARNING|trainer.py:803] 2025-04-26 19:09:32,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:33,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1137
1226
1163
[WARNING|trainer.py:803] 2025-04-26 19:09:34,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:34,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:35,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1138
1164
1227
[WARNING|trainer.py:803] 2025-04-26 19:09:36,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:36,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:36,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1139
1165
[WARNING|trainer.py:803] 2025-04-26 19:09:37,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:38,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1228
1140
1166
[WARNING|trainer.py:803] 2025-04-26 19:09:39,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:39,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:40,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1229
1141
1167
[WARNING|trainer.py:803] 2025-04-26 19:09:41,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:41,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:41,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1142
1168
1230
[WARNING|trainer.py:803] 2025-04-26 19:09:43,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:43,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:43,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1143
1169
1231
[WARNING|trainer.py:803] 2025-04-26 19:09:44,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:45,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:45,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1144
1170
1232
[WARNING|trainer.py:803] 2025-04-26 19:09:46,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:09:47,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1145
[WARNING|trainer.py:803] 2025-04-26 19:09:47,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1171
[WARNING|trainer.py:803] 2025-04-26 19:09:48,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1233
[WARNING|trainer.py:803] 2025-04-26 19:09:48,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1146
[WARNING|trainer.py:803] 2025-04-26 19:09:49,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1172
[WARNING|trainer.py:803] 2025-04-26 19:09:50,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:50,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1234
1147
1173
[WARNING|trainer.py:803] 2025-04-26 19:09:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:09:52,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:52,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1148
1235
1174
[WARNING|trainer.py:803] 2025-04-26 19:09:53,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:54,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:09:54,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1149
1236
[WARNING|trainer.py:803] 2025-04-26 19:09:55,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1175
[WARNING|trainer.py:803] 2025-04-26 19:09:56,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:09:56,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1150
[WARNING|trainer.py:803] 2025-04-26 19:09:57,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1176
1237
[WARNING|trainer.py:803] 2025-04-26 19:09:58,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1151
[WARNING|trainer.py:803] 2025-04-26 19:09:58,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1177
[WARNING|trainer.py:803] 2025-04-26 19:09:59,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1238
[WARNING|trainer.py:803] 2025-04-26 19:09:59,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1152
[WARNING|trainer.py:803] 2025-04-26 19:10:00,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1178
[WARNING|trainer.py:803] 2025-04-26 19:10:00,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1239
[WARNING|trainer.py:803] 2025-04-26 19:10:01,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1153
[WARNING|trainer.py:803] 2025-04-26 19:10:02,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1179
[WARNING|trainer.py:803] 2025-04-26 19:10:02,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:03,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1154
1240
1180
[WARNING|trainer.py:803] 2025-04-26 19:10:04,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:04,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:04,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1155
1241
1181
[WARNING|trainer.py:803] 2025-04-26 19:10:06,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:06,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:06,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1156
1182
[WARNING|trainer.py:803] 2025-04-26 19:10:07,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1242
[WARNING|trainer.py:803] 2025-04-26 19:10:08,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1157
[WARNING|trainer.py:803] 2025-04-26 19:10:08,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:09,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1183
1243
[WARNING|trainer.py:803] 2025-04-26 19:10:10,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1158
[WARNING|trainer.py:803] 2025-04-26 19:10:11,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1184
[WARNING|trainer.py:803] 2025-04-26 19:10:11,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:12,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1159
1244
[WARNING|trainer.py:803] 2025-04-26 19:10:13,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1185
[WARNING|trainer.py:803] 2025-04-26 19:10:13,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1160
[WARNING|trainer.py:803] 2025-04-26 19:10:13,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1245
[WARNING|trainer.py:803] 2025-04-26 19:10:14,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1186
[WARNING|trainer.py:803] 2025-04-26 19:10:15,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1161
[WARNING|trainer.py:803] 2025-04-26 19:10:15,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:16,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1246
1187
1162
[WARNING|trainer.py:803] 2025-04-26 19:10:17,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:17,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:18,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1188
1247
1163
[WARNING|trainer.py:803] 2025-04-26 19:10:19,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:19,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:20,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1189
1164
1248
[WARNING|trainer.py:803] 2025-04-26 19:10:21,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:21,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:22,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1165
1190
1249
[WARNING|trainer.py:803] 2025-04-26 19:10:23,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:23,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:24,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1166
1191
1250
[WARNING|trainer.py:803] 2025-04-26 19:10:25,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:25,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:25,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1167
1192
[WARNING|trainer.py:803] 2025-04-26 19:10:27,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:27,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1251
1168
1193
[WARNING|trainer.py:803] 2025-04-26 19:10:28,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:28,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:29,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1169
1252
1194
[WARNING|trainer.py:803] 2025-04-26 19:10:30,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:30,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:30,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1170
1253
1195
[WARNING|trainer.py:803] 2025-04-26 19:10:32,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:32,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:32,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1171
1196
1254
[WARNING|trainer.py:803] 2025-04-26 19:10:34,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:34,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:34,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1172
1197
[WARNING|trainer.py:803] 2025-04-26 19:10:35,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1255
[WARNING|trainer.py:803] 2025-04-26 19:10:36,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:36,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1173
1198
[WARNING|trainer.py:803] 2025-04-26 19:10:37,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:38,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1256
1174
[WARNING|trainer.py:803] 2025-04-26 19:10:38,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1199
[WARNING|trainer.py:803] 2025-04-26 19:10:39,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:39,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1257
1200
[WARNING|trainer.py:803] 2025-04-26 19:10:40,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1175
[WARNING|trainer.py:803] 2025-04-26 19:10:41,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:41,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1258
1176
[WARNING|trainer.py:803] 2025-04-26 19:10:43,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1201
[WARNING|trainer.py:803] 2025-04-26 19:10:43,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:44,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1259
1177
[WARNING|trainer.py:803] 2025-04-26 19:10:45,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:45,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1202
1178
[WARNING|trainer.py:803] 2025-04-26 19:10:46,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1260
[WARNING|trainer.py:803] 2025-04-26 19:10:46,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:47,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1179
1203
1261
[WARNING|trainer.py:803] 2025-04-26 19:10:48,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:48,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:49,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1180
1204
[WARNING|trainer.py:803] 2025-04-26 19:10:50,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1262
[WARNING|trainer.py:803] 2025-04-26 19:10:51,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:10:51,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1181
[WARNING|trainer.py:803] 2025-04-26 19:10:52,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1205
1263
1182
[WARNING|trainer.py:803] 2025-04-26 19:10:53,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:53,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:54,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1264
1206
1183
[WARNING|trainer.py:803] 2025-04-26 19:10:55,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:55,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:10:55,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1184
1265
1207
[WARNING|trainer.py:803] 2025-04-26 19:10:57,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:57,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:57,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1185
1266
1208
[WARNING|trainer.py:803] 2025-04-26 19:10:59,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:10:59,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:10:59,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1186
1267
[WARNING|trainer.py:803] 2025-04-26 19:11:01,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1209
[WARNING|trainer.py:803] 2025-04-26 19:11:02,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:11:02,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1187
1268
[WARNING|trainer.py:803] 2025-04-26 19:11:03,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1210
[WARNING|trainer.py:803] 2025-04-26 19:11:04,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1188
[WARNING|trainer.py:803] 2025-04-26 19:11:04,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:05,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1269
1211
1189
[WARNING|trainer.py:803] 2025-04-26 19:11:06,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:11:06,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:06,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1270
1212
1190
[WARNING|trainer.py:803] 2025-04-26 19:11:08,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:09,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:09,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1271
1191
1213
[WARNING|trainer.py:803] 2025-04-26 19:11:10,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:11,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:11,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1272
1192
[WARNING|trainer.py:803] 2025-04-26 19:11:12,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:12,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1214
1193
[WARNING|trainer.py:803] 2025-04-26 19:11:13,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1273
[WARNING|trainer.py:803] 2025-04-26 19:11:14,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:14,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1215
1194
[WARNING|trainer.py:803] 2025-04-26 19:11:15,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1274
[WARNING|trainer.py:803] 2025-04-26 19:11:16,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:16,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1195
1216
[WARNING|trainer.py:803] 2025-04-26 19:11:18,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:18,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1275
1196
[WARNING|trainer.py:803] 2025-04-26 19:11:19,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1217
[WARNING|trainer.py:803] 2025-04-26 19:11:19,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1276
[WARNING|trainer.py:803] 2025-04-26 19:11:20,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1197
[WARNING|trainer.py:803] 2025-04-26 19:11:21,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:21,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1218
1277
1198
[WARNING|trainer.py:803] 2025-04-26 19:11:22,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:23,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:23,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1219
1278
1199
[WARNING|trainer.py:803] 2025-04-26 19:11:24,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:24,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:25,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1200
1220
1279
[WARNING|trainer.py:803] 2025-04-26 19:11:26,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:26,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:27,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1221
1201
1280
[WARNING|trainer.py:803] 2025-04-26 19:11:29,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:29,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:29,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1222
1281
1202
[WARNING|trainer.py:803] 2025-04-26 19:11:31,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:31,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:31,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1282
1223
1203
[WARNING|trainer.py:803] 2025-04-26 19:11:33,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:33,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:34,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1283
1204
1224
[WARNING|trainer.py:803] 2025-04-26 19:11:35,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:36,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:36,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1284
1205
1225
[WARNING|trainer.py:803] 2025-04-26 19:11:38,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:38,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:38,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1285
1206
1226
[WARNING|trainer.py:803] 2025-04-26 19:11:40,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:40,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:40,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1286
1207
1227
[WARNING|trainer.py:803] 2025-04-26 19:11:42,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:42,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:42,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1208
1287
1228
[WARNING|trainer.py:803] 2025-04-26 19:11:45,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:45,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:45,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1288
1209
1229
[WARNING|trainer.py:803] 2025-04-26 19:11:47,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:47,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:47,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1289
1210
[WARNING|trainer.py:803] 2025-04-26 19:11:49,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1230
[WARNING|trainer.py:803] 2025-04-26 19:11:49,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1290
[WARNING|trainer.py:803] 2025-04-26 19:11:50,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1211
[WARNING|trainer.py:803] 2025-04-26 19:11:51,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1231
[WARNING|trainer.py:803] 2025-04-26 19:11:51,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1291
[WARNING|trainer.py:803] 2025-04-26 19:11:52,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:53,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1212
1232
[WARNING|trainer.py:803] 2025-04-26 19:11:54,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1292
[WARNING|trainer.py:803] 2025-04-26 19:11:54,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:11:55,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1213
1233
1293
[WARNING|trainer.py:803] 2025-04-26 19:11:56,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:57,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:11:57,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1214
1294
1234
[WARNING|trainer.py:803] 2025-04-26 19:11:58,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:11:59,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:11:59,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1215
1295
1235
[WARNING|trainer.py:803] 2025-04-26 19:12:01,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:01,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:01,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1296
1216
1236
[WARNING|trainer.py:803] 2025-04-26 19:12:03,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:03,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:03,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1217
1297
1237
[WARNING|trainer.py:803] 2025-04-26 19:12:05,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:05,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:06,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1218
1298
[WARNING|trainer.py:803] 2025-04-26 19:12:07,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1238
[WARNING|trainer.py:803] 2025-04-26 19:12:07,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:08,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1299
1219
[WARNING|trainer.py:803] 2025-04-26 19:12:09,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:09,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1239
[WARNING|trainer.py:803] 2025-04-26 19:12:10,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1220
1300
[WARNING|trainer.py:803] 2025-04-26 19:12:11,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:12:12,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1240
[WARNING|trainer.py:803] 2025-04-26 19:12:13,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1221
1301
[WARNING|trainer.py:803] 2025-04-26 19:12:14,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1241
[WARNING|trainer.py:803] 2025-04-26 19:12:14,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1222
[WARNING|trainer.py:803] 2025-04-26 19:12:15,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1302
[WARNING|trainer.py:803] 2025-04-26 19:12:16,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:16,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1242
1303
[WARNING|trainer.py:803] 2025-04-26 19:12:18,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1223
[WARNING|trainer.py:803] 2025-04-26 19:12:18,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:12:18,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1243
1304
[WARNING|trainer.py:803] 2025-04-26 19:12:20,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1224
[WARNING|trainer.py:803] 2025-04-26 19:12:20,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:21,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1244
1305
[WARNING|trainer.py:803] 2025-04-26 19:12:22,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1225
[WARNING|trainer.py:803] 2025-04-26 19:12:22,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:23,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1306
1245
[WARNING|trainer.py:803] 2025-04-26 19:12:25,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1226
[WARNING|trainer.py:803] 2025-04-26 19:12:25,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:12:25,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1307
1246
[WARNING|trainer.py:803] 2025-04-26 19:12:27,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1227
[WARNING|trainer.py:803] 2025-04-26 19:12:27,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:27,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1308
[WARNING|trainer.py:803] 2025-04-26 19:12:29,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1247
1228
[WARNING|trainer.py:803] 2025-04-26 19:12:29,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1309
[WARNING|trainer.py:803] 2025-04-26 19:12:30,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:31,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1248
1229
1310
[WARNING|trainer.py:803] 2025-04-26 19:12:32,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:32,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:33,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1249
1311
[WARNING|trainer.py:803] 2025-04-26 19:12:34,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1230
[WARNING|trainer.py:803] 2025-04-26 19:12:35,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:35,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1250
1312
[WARNING|trainer.py:803] 2025-04-26 19:12:36,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1231
[WARNING|trainer.py:803] 2025-04-26 19:12:37,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:37,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1251
1313
1232
[WARNING|trainer.py:803] 2025-04-26 19:12:39,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:39,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:39,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1314
1252
1233
[WARNING|trainer.py:803] 2025-04-26 19:12:41,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:41,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:12:42,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1315
1253
1234
[WARNING|trainer.py:803] 2025-04-26 19:12:43,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:43,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:44,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1316
1254
1235
[WARNING|trainer.py:803] 2025-04-26 19:12:45,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:46,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:46,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1317
1255
[WARNING|trainer.py:803] 2025-04-26 19:12:48,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1236
[WARNING|trainer.py:803] 2025-04-26 19:12:48,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:48,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1318
1256
[WARNING|trainer.py:803] 2025-04-26 19:12:50,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1237
[WARNING|trainer.py:803] 2025-04-26 19:12:50,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:51,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1319
1257
[WARNING|trainer.py:803] 2025-04-26 19:12:52,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1238
[WARNING|trainer.py:803] 2025-04-26 19:12:52,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:53,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1320
1258
[WARNING|trainer.py:803] 2025-04-26 19:12:54,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1239
[WARNING|trainer.py:803] 2025-04-26 19:12:55,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:12:55,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1321
1259
1240
[WARNING|trainer.py:803] 2025-04-26 19:12:57,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:12:57,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:12:58,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1322
1260
[WARNING|trainer.py:803] 2025-04-26 19:12:59,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1241
[WARNING|trainer.py:803] 2025-04-26 19:13:00,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:00,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1323
1261
[WARNING|trainer.py:803] 2025-04-26 19:13:01,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:13:02,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1242
1324
[WARNING|trainer.py:803] 2025-04-26 19:13:03,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1262
[WARNING|trainer.py:803] 2025-04-26 19:13:03,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:04,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1243
1325
[WARNING|trainer.py:803] 2025-04-26 19:13:05,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1263
[WARNING|trainer.py:803] 2025-04-26 19:13:06,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:06,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1244
1326
[WARNING|trainer.py:803] 2025-04-26 19:13:07,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1264
[WARNING|trainer.py:803] 2025-04-26 19:13:08,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:08,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1245
1327
[WARNING|trainer.py:803] 2025-04-26 19:13:10,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:13:10,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1265
[WARNING|trainer.py:803] 2025-04-26 19:13:11,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1328
1246
[WARNING|trainer.py:803] 2025-04-26 19:13:12,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:12,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1266
1329
[WARNING|trainer.py:803] 2025-04-26 19:13:13,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1247
[WARNING|trainer.py:803] 2025-04-26 19:13:14,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1267
[WARNING|trainer.py:803] 2025-04-26 19:13:15,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1330
[WARNING|trainer.py:803] 2025-04-26 19:13:15,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:16,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1248
[WARNING|trainer.py:803] 2025-04-26 19:13:17,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1268
1331
[WARNING|trainer.py:803] 2025-04-26 19:13:18,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:18,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1249
[WARNING|trainer.py:803] 2025-04-26 19:13:19,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1332
1269
[WARNING|trainer.py:803] 2025-04-26 19:13:20,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:20,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1250
[WARNING|trainer.py:803] 2025-04-26 19:13:21,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1333
1270
[WARNING|trainer.py:803] 2025-04-26 19:13:22,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:23,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1251
1334
1271
[WARNING|trainer.py:803] 2025-04-26 19:13:24,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:24,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:25,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1252
1335
[WARNING|trainer.py:803] 2025-04-26 19:13:26,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1272
[WARNING|trainer.py:803] 2025-04-26 19:13:26,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:13:27,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1253
1336
[WARNING|trainer.py:803] 2025-04-26 19:13:28,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1273
[WARNING|trainer.py:803] 2025-04-26 19:13:29,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:29,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1337
1254
1274
[WARNING|trainer.py:803] 2025-04-26 19:13:31,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:31,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:31,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1338
1255
[WARNING|trainer.py:803] 2025-04-26 19:13:33,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:33,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1275
[WARNING|trainer.py:803] 2025-04-26 19:13:34,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1339
1256
[WARNING|trainer.py:803] 2025-04-26 19:13:35,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1276
[WARNING|trainer.py:803] 2025-04-26 19:13:35,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:36,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1340
1257
[WARNING|trainer.py:803] 2025-04-26 19:13:37,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1277
[WARNING|trainer.py:803] 2025-04-26 19:13:37,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:38,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1341
1258
[WARNING|trainer.py:803] 2025-04-26 19:13:39,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1278
[WARNING|trainer.py:803] 2025-04-26 19:13:40,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:40,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1342
[WARNING|trainer.py:803] 2025-04-26 19:13:41,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1259
1279
[WARNING|trainer.py:803] 2025-04-26 19:13:42,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1343
[WARNING|trainer.py:803] 2025-04-26 19:13:43,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:43,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1260
1280
1344
[WARNING|trainer.py:803] 2025-04-26 19:13:45,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:45,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:45,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1261
1281
[WARNING|trainer.py:803] 2025-04-26 19:13:47,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1345
[WARNING|trainer.py:803] 2025-04-26 19:13:47,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:48,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1262
1282
1346
[WARNING|trainer.py:803] 2025-04-26 19:13:49,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:50,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:13:50,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1263
1347
[WARNING|trainer.py:803] 2025-04-26 19:13:51,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1283
[WARNING|trainer.py:803] 2025-04-26 19:13:52,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:52,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1264
1348
[WARNING|trainer.py:803] 2025-04-26 19:13:53,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1284
[WARNING|trainer.py:803] 2025-04-26 19:13:54,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:13:55,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1265
1349
[WARNING|trainer.py:803] 2025-04-26 19:13:56,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:13:56,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1285
[WARNING|trainer.py:803] 2025-04-26 19:13:57,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1266
[WARNING|trainer.py:803] 2025-04-26 19:13:58,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1350
1286
[WARNING|trainer.py:803] 2025-04-26 19:14:00,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:00,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1267
[WARNING|trainer.py:803] 2025-04-26 19:14:01,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1287
1351
1268
[WARNING|trainer.py:803] 2025-04-26 19:14:02,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:03,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:03,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1288
[WARNING|trainer.py:803] 2025-04-26 19:14:04,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
1269
1352
[WARNING|trainer.py:803] 2025-04-26 19:14:05,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1289
[WARNING|trainer.py:803] 2025-04-26 19:14:06,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:07,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1270
1353
1290
[WARNING|trainer.py:803] 2025-04-26 19:14:08,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:08,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:09,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1271
1354
1291
[WARNING|trainer.py:803] 2025-04-26 19:14:10,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:11,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:11,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1272
1355
1292
[WARNING|trainer.py:803] 2025-04-26 19:14:13,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:13,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:13,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1273
1293
[WARNING|trainer.py:803] 2025-04-26 19:14:15,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1356
[WARNING|trainer.py:803] 2025-04-26 19:14:15,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:16,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1274
1294
[WARNING|trainer.py:803] 2025-04-26 19:14:17,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:18,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1357
[WARNING|trainer.py:803] 2025-04-26 19:14:19,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1275
1295
[WARNING|trainer.py:803] 2025-04-26 19:14:19,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:20,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1276
1358
1296
[WARNING|trainer.py:803] 2025-04-26 19:14:22,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:22,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:22,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1277
1359
1297
[WARNING|trainer.py:803] 2025-04-26 19:14:24,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:24,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:24,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1278
[WARNING|trainer.py:803] 2025-04-26 19:14:26,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1298
1360
[WARNING|trainer.py:803] 2025-04-26 19:14:27,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:27,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1279
[WARNING|trainer.py:803] 2025-04-26 19:14:28,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1299
1361
[WARNING|trainer.py:803] 2025-04-26 19:14:29,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:30,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1280
[WARNING|trainer.py:803] 2025-04-26 19:14:31,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1300
1362
[WARNING|trainer.py:803] 2025-04-26 19:14:32,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:32,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1281
[WARNING|trainer.py:803] 2025-04-26 19:14:33,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1301
1363
1282
[WARNING|trainer.py:803] 2025-04-26 19:14:35,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:35,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:35,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1302
[WARNING|trainer.py:803] 2025-04-26 19:14:37,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1283
1364
[WARNING|trainer.py:803] 2025-04-26 19:14:38,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1303
[WARNING|trainer.py:803] 2025-04-26 19:14:38,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:39,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1284
1365
[WARNING|trainer.py:803] 2025-04-26 19:14:40,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1304
[WARNING|trainer.py:803] 2025-04-26 19:14:41,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:41,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1285
1305
1366
[WARNING|trainer.py:803] 2025-04-26 19:14:43,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:43,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:44,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1286
1306
[WARNING|trainer.py:803] 2025-04-26 19:14:45,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1367
[WARNING|trainer.py:803] 2025-04-26 19:14:46,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:46,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1287
1307
1368
[WARNING|trainer.py:803] 2025-04-26 19:14:48,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:48,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:48,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1288
1308
[WARNING|trainer.py:803] 2025-04-26 19:14:50,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 19:14:50,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1369
1289
1309
[WARNING|trainer.py:803] 2025-04-26 19:14:52,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:14:52,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:52,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1370
1290
1310
[WARNING|trainer.py:803] 2025-04-26 19:14:54,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:14:54,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:14:55,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1291
1371
1311
[WARNING|trainer.py:803] 2025-04-26 19:14:56,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:14:57,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:57,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1372
1292
1312
[WARNING|trainer.py:803] 2025-04-26 19:14:59,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:59,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:14:59,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1293
1373
1313
[WARNING|trainer.py:803] 2025-04-26 19:15:01,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:01,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:02,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1294
1314
1374
[WARNING|trainer.py:803] 2025-04-26 19:15:03,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:04,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:04,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1295
1315
1375
[WARNING|trainer.py:803] 2025-04-26 19:15:06,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:06,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:06,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1296
1316
[WARNING|trainer.py:803] 2025-04-26 19:15:08,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1376
[WARNING|trainer.py:803] 2025-04-26 19:15:08,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:09,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1297
1317
[WARNING|trainer.py:803] 2025-04-26 19:15:10,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1377
[WARNING|trainer.py:803] 2025-04-26 19:15:11,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:15:11,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1298
1318
[WARNING|trainer.py:803] 2025-04-26 19:15:13,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:13,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1378
1299
[WARNING|trainer.py:803] 2025-04-26 19:15:15,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1319
[WARNING|trainer.py:803] 2025-04-26 19:15:15,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:16,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1300
1379
1320
[WARNING|trainer.py:803] 2025-04-26 19:15:17,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:17,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:18,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1301
1380
1321
[WARNING|trainer.py:803] 2025-04-26 19:15:20,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:20,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:21,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1302
1322
1381
[WARNING|trainer.py:803] 2025-04-26 19:15:22,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:23,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:23,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1303
1323
[WARNING|trainer.py:803] 2025-04-26 19:15:25,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1382
[WARNING|trainer.py:803] 2025-04-26 19:15:25,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:15:26,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1304
1324
[WARNING|trainer.py:803] 2025-04-26 19:15:27,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1383
[WARNING|trainer.py:803] 2025-04-26 19:15:28,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1305
[WARNING|trainer.py:803] 2025-04-26 19:15:28,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:15:29,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1325
1384
[WARNING|trainer.py:803] 2025-04-26 19:15:30,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:30,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1306
[WARNING|trainer.py:803] 2025-04-26 19:15:31,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1326
[WARNING|trainer.py:803] 2025-04-26 19:15:32,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1307
1385
[WARNING|trainer.py:803] 2025-04-26 19:15:34,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:34,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1327
[WARNING|trainer.py:803] 2025-04-26 19:15:35,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1308
1386
[WARNING|trainer.py:803] 2025-04-26 19:15:36,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1328
[WARNING|trainer.py:803] 2025-04-26 19:15:36,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:37,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1309
[WARNING|trainer.py:803] 2025-04-26 19:15:38,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1329
1387
[WARNING|trainer.py:803] 2025-04-26 19:15:39,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:39,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1310
[WARNING|trainer.py:803] 2025-04-26 19:15:40,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1330
[WARNING|trainer.py:803] 2025-04-26 19:15:41,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1311
1388
[WARNING|trainer.py:803] 2025-04-26 19:15:43,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1331
[WARNING|trainer.py:803] 2025-04-26 19:15:43,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:43,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1312
1389
[WARNING|trainer.py:803] 2025-04-26 19:15:45,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1332
[WARNING|trainer.py:803] 2025-04-26 19:15:45,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:46,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1313
1390
1333
[WARNING|trainer.py:803] 2025-04-26 19:15:47,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:48,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:48,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1314
[WARNING|trainer.py:803] 2025-04-26 19:15:49,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1334
1391
[WARNING|trainer.py:803] 2025-04-26 19:15:50,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:15:51,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1315
[WARNING|trainer.py:803] 2025-04-26 19:15:52,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1335
[WARNING|trainer.py:803] 2025-04-26 19:15:53,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1392
1316
[WARNING|trainer.py:803] 2025-04-26 19:15:54,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:15:54,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1336
[WARNING|trainer.py:803] 2025-04-26 19:15:55,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1317
1393
[WARNING|trainer.py:803] 2025-04-26 19:15:56,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1337
[WARNING|trainer.py:803] 2025-04-26 19:15:57,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:15:57,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1318
1394
1338
[WARNING|trainer.py:803] 2025-04-26 19:15:59,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:00,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:00,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1319
1339
[WARNING|trainer.py:803] 2025-04-26 19:16:01,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1395
[WARNING|trainer.py:803] 2025-04-26 19:16:02,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:03,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1320
1340
[WARNING|trainer.py:803] 2025-04-26 19:16:04,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:05,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1396
1321
1341
[WARNING|trainer.py:803] 2025-04-26 19:16:06,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:16:06,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:16:07,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1322
1342
1397
[WARNING|trainer.py:803] 2025-04-26 19:16:09,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:09,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:10,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1323
1343
[WARNING|trainer.py:803] 2025-04-26 19:16:11,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:11,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1398
1324
1344
[WARNING|trainer.py:803] 2025-04-26 19:16:13,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:14,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:14,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1399
1345
[WARNING|trainer.py:803] 2025-04-26 19:16:15,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1325
[WARNING|trainer.py:803] 2025-04-26 19:16:16,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:16,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1400
1346
[WARNING|trainer.py:803] 2025-04-26 19:16:18,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1326
[WARNING|trainer.py:803] 2025-04-26 19:16:18,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:18,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1401
[WARNING|trainer.py:803] 2025-04-26 19:16:19,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1347
1327
1402
[WARNING|trainer.py:803] 2025-04-26 19:16:21,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:16:21,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:21,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1328
1348
1403
[WARNING|trainer.py:803] 2025-04-26 19:16:23,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:23,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:23,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1329
1349
1404
[WARNING|trainer.py:803] 2025-04-26 19:16:25,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:25,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:25,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1330
1405
[WARNING|trainer.py:803] 2025-04-26 19:16:27,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:27,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1350
1406
1331
[WARNING|trainer.py:803] 2025-04-26 19:16:29,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:29,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:29,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1407
1332
1351
[WARNING|trainer.py:803] 2025-04-26 19:16:31,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:32,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:32,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1408
1333
[WARNING|trainer.py:803] 2025-04-26 19:16:34,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:34,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1352
1409
[WARNING|trainer.py:803] 2025-04-26 19:16:35,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:36,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1334
[WARNING|trainer.py:803] 2025-04-26 19:16:37,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1410
[WARNING|trainer.py:803] 2025-04-26 19:16:37,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1353
1335
[WARNING|trainer.py:803] 2025-04-26 19:16:38,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1411
[WARNING|trainer.py:803] 2025-04-26 19:16:39,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:40,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1354
1336
[WARNING|trainer.py:803] 2025-04-26 19:16:41,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1412
[WARNING|trainer.py:803] 2025-04-26 19:16:41,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:42,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1413
1355
1337
[WARNING|trainer.py:803] 2025-04-26 19:16:43,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:44,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:16:44,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1414
1338
1356
[WARNING|trainer.py:803] 2025-04-26 19:16:46,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:16:46,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:46,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1415
[WARNING|trainer.py:803] 2025-04-26 19:16:47,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1339
1357
1416
[WARNING|trainer.py:803] 2025-04-26 19:16:49,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:16:49,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:49,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1340
1417
[WARNING|trainer.py:803] 2025-04-26 19:16:51,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:51,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1358
1341
1418
[WARNING|trainer.py:803] 2025-04-26 19:16:52,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:16:53,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:53,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1359
1419
1342
[WARNING|trainer.py:803] 2025-04-26 19:16:55,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:55,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:56,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1420
1343
[WARNING|trainer.py:803] 2025-04-26 19:16:57,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1360
[WARNING|trainer.py:803] 2025-04-26 19:16:58,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1421
[WARNING|trainer.py:803] 2025-04-26 19:16:58,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:16:59,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1344
[WARNING|trainer.py:803] 2025-04-26 19:17:00,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1422
1361
[WARNING|trainer.py:803] 2025-04-26 19:17:01,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:01,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1345
1423
[WARNING|trainer.py:803] 2025-04-26 19:17:02,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1362
[WARNING|trainer.py:803] 2025-04-26 19:17:03,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:04,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1346
1424
[WARNING|trainer.py:803] 2025-04-26 19:17:05,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:05,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1363
1425
1347
[WARNING|trainer.py:803] 2025-04-26 19:17:07,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:07,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:07,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1426
1348
[WARNING|trainer.py:803] 2025-04-26 19:17:09,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:10,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1364
1427
[WARNING|trainer.py:803] 2025-04-26 19:17:11,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:11,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1349
1428
[WARNING|trainer.py:803] 2025-04-26 19:17:12,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1365
[WARNING|trainer.py:803] 2025-04-26 19:17:13,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:13,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1429
1350
[WARNING|trainer.py:803] 2025-04-26 19:17:15,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1366
[WARNING|trainer.py:803] 2025-04-26 19:17:16,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:16,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1430
[WARNING|trainer.py:803] 2025-04-26 19:17:17,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1367
1431
1351
[WARNING|trainer.py:803] 2025-04-26 19:17:19,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:19,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:19,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1432
1368
[WARNING|trainer.py:803] 2025-04-26 19:17:21,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:21,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1352
1433
[WARNING|trainer.py:803] 2025-04-26 19:17:22,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:23,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1369
1434
1353
[WARNING|trainer.py:803] 2025-04-26 19:17:25,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:25,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:25,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1435
1370
1354
[WARNING|trainer.py:803] 2025-04-26 19:17:27,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:27,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:28,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1436
[WARNING|trainer.py:803] 2025-04-26 19:17:29,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1371
1355
[WARNING|trainer.py:803] 2025-04-26 19:17:30,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1437
[WARNING|trainer.py:803] 2025-04-26 19:17:31,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:31,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1372
1438
[WARNING|trainer.py:803] 2025-04-26 19:17:32,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1356
[WARNING|trainer.py:803] 2025-04-26 19:17:33,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:33,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1439
1373
[WARNING|trainer.py:803] 2025-04-26 19:17:35,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:35,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1357
[WARNING|trainer.py:803] 2025-04-26 19:17:36,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1440
1374
[WARNING|trainer.py:803] 2025-04-26 19:17:37,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:38,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1441
1358
[WARNING|trainer.py:803] 2025-04-26 19:17:39,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:39,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1375
[WARNING|trainer.py:803] 2025-04-26 19:17:40,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1442
1359
[WARNING|trainer.py:803] 2025-04-26 19:17:41,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:42,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1443
1376
[WARNING|trainer.py:803] 2025-04-26 19:17:43,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:43,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1360
1444
1377
[WARNING|trainer.py:803] 2025-04-26 19:17:45,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:45,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:17:46,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1445
1361
[WARNING|trainer.py:803] 2025-04-26 19:17:47,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:48,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1446
1378
[WARNING|trainer.py:803] 2025-04-26 19:17:49,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:50,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1362
1447
[WARNING|trainer.py:803] 2025-04-26 19:17:51,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:17:51,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1379
[WARNING|trainer.py:803] 2025-04-26 19:17:52,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1448
1363
[WARNING|trainer.py:803] 2025-04-26 19:17:53,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:54,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1449
1380
[WARNING|trainer.py:803] 2025-04-26 19:17:55,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:17:56,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1364
1450
[WARNING|trainer.py:803] 2025-04-26 19:17:58,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:17:58,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1381
[WARNING|trainer.py:803] 2025-04-26 19:17:59,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1451
1365
[WARNING|trainer.py:803] 2025-04-26 19:18:00,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:00,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1382
1452
[WARNING|trainer.py:803] 2025-04-26 19:18:01,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:18:02,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1366
1453
[WARNING|trainer.py:803] 2025-04-26 19:18:03,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1383
[WARNING|trainer.py:803] 2025-04-26 19:18:04,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:04,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1454
1367
1384
[WARNING|trainer.py:803] 2025-04-26 19:18:06,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:06,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:06,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1455
1368
[WARNING|trainer.py:803] 2025-04-26 19:18:08,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:18:08,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1456
1385
[WARNING|trainer.py:803] 2025-04-26 19:18:10,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:10,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1457
1369
[WARNING|trainer.py:803] 2025-04-26 19:18:12,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1386
[WARNING|trainer.py:803] 2025-04-26 19:18:12,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:18:13,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1458
1370
[WARNING|trainer.py:803] 2025-04-26 19:18:14,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:14,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1387
1459
[WARNING|trainer.py:803] 2025-04-26 19:18:16,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1371
[WARNING|trainer.py:803] 2025-04-26 19:18:16,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:17,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1460
[WARNING|trainer.py:803] 2025-04-26 19:18:18,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1372
1388
1461
[WARNING|trainer.py:803] 2025-04-26 19:18:19,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:19,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:20,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1462
1373
1389
[WARNING|trainer.py:803] 2025-04-26 19:18:22,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:22,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:22,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1463
1374
[WARNING|trainer.py:803] 2025-04-26 19:18:24,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1390
[WARNING|trainer.py:803] 2025-04-26 19:18:25,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1464
[WARNING|trainer.py:803] 2025-04-26 19:18:25,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:26,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1375
1465
[WARNING|trainer.py:803] 2025-04-26 19:18:27,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1391
[WARNING|trainer.py:803] 2025-04-26 19:18:28,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:28,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1466
1376
[WARNING|trainer.py:803] 2025-04-26 19:18:30,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:30,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1392
1467
[WARNING|trainer.py:803] 2025-04-26 19:18:31,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:32,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1377
1468
[WARNING|trainer.py:803] 2025-04-26 19:18:33,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:18:34,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1393
1469
[WARNING|trainer.py:803] 2025-04-26 19:18:35,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:18:36,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1378
[WARNING|trainer.py:803] 2025-04-26 19:18:37,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1470
1394
[WARNING|trainer.py:803] 2025-04-26 19:18:38,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:18:38,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1379
1471
[WARNING|trainer.py:803] 2025-04-26 19:18:39,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:40,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1395
1472
[WARNING|trainer.py:803] 2025-04-26 19:18:41,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:42,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1380
[WARNING|trainer.py:803] 2025-04-26 19:18:43,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1473
[WARNING|trainer.py:803] 2025-04-26 19:18:44,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1396
1381
[WARNING|trainer.py:803] 2025-04-26 19:18:45,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1474
[WARNING|trainer.py:803] 2025-04-26 19:18:46,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:18:46,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1475
1397
1382
[WARNING|trainer.py:803] 2025-04-26 19:18:48,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:49,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 19:18:49,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
YesYes
1476
[WARNING|trainer.py:803] 2025-04-26 19:18:50,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1383
[WARNING|trainer.py:803] 2025-04-26 19:18:51,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1398
1477
[WARNING|trainer.py:803] 2025-04-26 19:18:52,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:52,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1384
1478
1399
[WARNING|trainer.py:803] 2025-04-26 19:18:53,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:54,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:18:54,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1479
1400
[WARNING|trainer.py:803] 2025-04-26 19:18:56,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1385
[WARNING|trainer.py:803] 2025-04-26 19:18:57,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:18:57,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1480
1401
[WARNING|trainer.py:803] 2025-04-26 19:18:58,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1386
[WARNING|trainer.py:803] 2025-04-26 19:18:59,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1481
[WARNING|trainer.py:803] 2025-04-26 19:19:00,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1402
[WARNING|trainer.py:803] 2025-04-26 19:19:00,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:01,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1482
1387
1403
[WARNING|trainer.py:803] 2025-04-26 19:19:03,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:03,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:03,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1483
1404
[WARNING|trainer.py:803] 2025-04-26 19:19:05,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:06,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1388
1484
[WARNING|trainer.py:803] 2025-04-26 19:19:07,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:19:07,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1405
[WARNING|trainer.py:803] 2025-04-26 19:19:08,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1485
1389
[WARNING|trainer.py:803] 2025-04-26 19:19:09,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1406
[WARNING|trainer.py:803] 2025-04-26 19:19:09,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:10,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1486
[WARNING|trainer.py:803] 2025-04-26 19:19:11,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1407
1390
[WARNING|trainer.py:803] 2025-04-26 19:19:12,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:12,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1487
1408
[WARNING|trainer.py:803] 2025-04-26 19:19:14,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1391
[WARNING|trainer.py:803] 2025-04-26 19:19:14,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1488
[WARNING|trainer.py:803] 2025-04-26 19:19:15,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:15,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1409
1489
[WARNING|trainer.py:803] 2025-04-26 19:19:17,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:17,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1392
1410
[WARNING|trainer.py:803] 2025-04-26 19:19:18,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:19:19,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1490
[WARNING|trainer.py:803] 2025-04-26 19:19:20,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1411
1393
[WARNING|trainer.py:803] 2025-04-26 19:19:21,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1491
[WARNING|trainer.py:803] 2025-04-26 19:19:22,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:22,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1412
[WARNING|trainer.py:803] 2025-04-26 19:19:23,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1492
1394
[WARNING|trainer.py:803] 2025-04-26 19:19:24,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1413
[WARNING|trainer.py:803] 2025-04-26 19:19:25,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1493
[WARNING|trainer.py:803] 2025-04-26 19:19:25,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:26,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1414
1494
1395
[WARNING|trainer.py:803] 2025-04-26 19:19:28,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:28,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:28,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1415
1495
[WARNING|trainer.py:803] 2025-04-26 19:19:30,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:19:30,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1396
1496
1416
[WARNING|trainer.py:803] 2025-04-26 19:19:32,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:19:32,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:32,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1497
1417
[WARNING|trainer.py:803] 2025-04-26 19:19:34,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:34,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1397
1498
[WARNING|trainer.py:803] 2025-04-26 19:19:36,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1418
[WARNING|trainer.py:803] 2025-04-26 19:19:36,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:36,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1499
1419
1398
[WARNING|trainer.py:803] 2025-04-26 19:19:38,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:39,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:39,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1500
1420
[WARNING|trainer.py:803] 2025-04-26 19:19:40,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1399
[WARNING|trainer.py:803] 2025-04-26 19:19:41,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:19:41,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1421
1501
[WARNING|trainer.py:803] 2025-04-26 19:19:43,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1400
[WARNING|trainer.py:803] 2025-04-26 19:19:43,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:19:44,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1422
1401
[WARNING|trainer.py:803] 2025-04-26 19:19:45,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:46,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1502
1423
[WARNING|trainer.py:803] 2025-04-26 19:19:47,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1402
[WARNING|trainer.py:803] 2025-04-26 19:19:47,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:48,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1424
1503
1403
[WARNING|trainer.py:803] 2025-04-26 19:19:49,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:50,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:50,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1425
1404
[WARNING|trainer.py:803] 2025-04-26 19:19:52,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1504
[WARNING|trainer.py:803] 2025-04-26 19:19:52,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:19:53,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1426
[WARNING|trainer.py:803] 2025-04-26 19:19:54,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1405
[WARNING|trainer.py:803] 2025-04-26 19:19:55,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1427
1505
[WARNING|trainer.py:803] 2025-04-26 19:19:56,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1406
[WARNING|trainer.py:803] 2025-04-26 19:19:56,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:19:57,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1428
[WARNING|trainer.py:803] 2025-04-26 19:19:58,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1407
1506
[WARNING|trainer.py:803] 2025-04-26 19:19:59,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1429
[WARNING|trainer.py:803] 2025-04-26 19:19:59,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:00,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1408
[WARNING|trainer.py:803] 2025-04-26 19:20:01,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1507
1430
[WARNING|trainer.py:803] 2025-04-26 19:20:02,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:20:02,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1409
1431
[WARNING|trainer.py:803] 2025-04-26 19:20:04,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:05,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1508
1410
[WARNING|trainer.py:803] 2025-04-26 19:20:06,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:06,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1432
[WARNING|trainer.py:803] 2025-04-26 19:20:07,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1411
1509
[WARNING|trainer.py:803] 2025-04-26 19:20:08,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1433
[WARNING|trainer.py:803] 2025-04-26 19:20:09,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:09,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1412
[WARNING|trainer.py:803] 2025-04-26 19:20:10,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1434
1510
[WARNING|trainer.py:803] 2025-04-26 19:20:11,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1413
[WARNING|trainer.py:803] 2025-04-26 19:20:12,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:12,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1435
[WARNING|trainer.py:803] 2025-04-26 19:20:13,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1414
1511
1436
[WARNING|trainer.py:803] 2025-04-26 19:20:15,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:15,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:16,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1415
1437
[WARNING|trainer.py:803] 2025-04-26 19:20:17,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1512
[WARNING|trainer.py:803] 2025-04-26 19:20:18,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:18,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1416
1438
[WARNING|trainer.py:803] 2025-04-26 19:20:19,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:20,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1513
1417
1439
[WARNING|trainer.py:803] 2025-04-26 19:20:21,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:21,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:22,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1418
1440
[WARNING|trainer.py:803] 2025-04-26 19:20:24,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1514
[WARNING|trainer.py:803] 2025-04-26 19:20:24,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:25,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1419
1441
[WARNING|trainer.py:803] 2025-04-26 19:20:26,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:27,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1515
1420
[WARNING|trainer.py:803] 2025-04-26 19:20:28,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:28,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1442
[WARNING|trainer.py:803] 2025-04-26 19:20:29,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1421
[WARNING|trainer.py:803] 2025-04-26 19:20:30,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1443
1516
[WARNING|trainer.py:803] 2025-04-26 19:20:31,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:31,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1422
[WARNING|trainer.py:803] 2025-04-26 19:20:32,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1444
[WARNING|trainer.py:803] 2025-04-26 19:20:33,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1423
1517
[WARNING|trainer.py:803] 2025-04-26 19:20:35,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:35,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1445
[WARNING|trainer.py:803] 2025-04-26 19:20:35,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1424
[WARNING|trainer.py:803] 2025-04-26 19:20:37,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1446
1518
[WARNING|trainer.py:803] 2025-04-26 19:20:38,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:38,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1425
[WARNING|trainer.py:803] 2025-04-26 19:20:39,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1447
[WARNING|trainer.py:803] 2025-04-26 19:20:40,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1519
1426
[WARNING|trainer.py:803] 2025-04-26 19:20:41,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:41,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1448
[WARNING|trainer.py:803] 2025-04-26 19:20:42,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1427
[WARNING|trainer.py:803] 2025-04-26 19:20:43,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1520
1449
[WARNING|trainer.py:803] 2025-04-26 19:20:44,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:44,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1428
[WARNING|trainer.py:803] 2025-04-26 19:20:45,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1450
1521
[WARNING|trainer.py:803] 2025-04-26 19:20:47,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1429
[WARNING|trainer.py:803] 2025-04-26 19:20:47,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:48,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1451
[WARNING|trainer.py:803] 2025-04-26 19:20:49,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1430
[WARNING|trainer.py:803] 2025-04-26 19:20:50,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1522
1452
[WARNING|trainer.py:803] 2025-04-26 19:20:51,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:20:51,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1431
[WARNING|trainer.py:803] 2025-04-26 19:20:52,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1453
[WARNING|trainer.py:803] 2025-04-26 19:20:53,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1523
1432
[WARNING|trainer.py:803] 2025-04-26 19:20:54,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:20:54,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1454
1433
[WARNING|trainer.py:803] 2025-04-26 19:20:55,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:20:56,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1524
1455
[WARNING|trainer.py:803] 2025-04-26 19:20:57,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1434
[WARNING|trainer.py:803] 2025-04-26 19:20:58,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:20:58,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1456
1435
[WARNING|trainer.py:803] 2025-04-26 19:21:00,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1525
[WARNING|trainer.py:803] 2025-04-26 19:21:01,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:21:01,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1457
[WARNING|trainer.py:803] 2025-04-26 19:21:02,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1436
[WARNING|trainer.py:803] 2025-04-26 19:21:03,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1526
1458
[WARNING|trainer.py:803] 2025-04-26 19:21:04,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1437
[WARNING|trainer.py:803] 2025-04-26 19:21:04,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:05,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1459
1438
1527
[WARNING|trainer.py:803] 2025-04-26 19:21:07,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:07,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:08,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1460
1439
[WARNING|trainer.py:803] 2025-04-26 19:21:09,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:09,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1461
1528
1440
[WARNING|trainer.py:803] 2025-04-26 19:21:11,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:11,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:21:12,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1462
1441
[WARNING|trainer.py:803] 2025-04-26 19:21:13,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1529
[WARNING|trainer.py:803] 2025-04-26 19:21:14,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:14,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1463
[WARNING|trainer.py:803] 2025-04-26 19:21:16,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1442
[WARNING|trainer.py:803] 2025-04-26 19:21:16,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1530
1464
[WARNING|trainer.py:803] 2025-04-26 19:21:17,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1443
[WARNING|trainer.py:803] 2025-04-26 19:21:18,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:19,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1465
1531
1444
[WARNING|trainer.py:803] 2025-04-26 19:21:20,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:21,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:21,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1466
1445
[WARNING|trainer.py:803] 2025-04-26 19:21:22,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:23,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1532
1467
[WARNING|trainer.py:803] 2025-04-26 19:21:24,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1446
[WARNING|trainer.py:803] 2025-04-26 19:21:24,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:21:25,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1468
1447
[WARNING|trainer.py:803] 2025-04-26 19:21:26,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1533
[WARNING|trainer.py:803] 2025-04-26 19:21:27,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:27,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1469
[WARNING|trainer.py:803] 2025-04-26 19:21:28,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1448
[WARNING|trainer.py:803] 2025-04-26 19:21:29,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1534
1470
[WARNING|trainer.py:803] 2025-04-26 19:21:30,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:31,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1449
1471
[WARNING|trainer.py:803] 2025-04-26 19:21:32,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:21:33,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1450
1535
[WARNING|trainer.py:803] 2025-04-26 19:21:34,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1472
[WARNING|trainer.py:803] 2025-04-26 19:21:34,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:35,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1451
[WARNING|trainer.py:803] 2025-04-26 19:21:36,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1473
1536
[WARNING|trainer.py:803] 2025-04-26 19:21:37,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1452
[WARNING|trainer.py:803] 2025-04-26 19:21:38,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:38,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1474
[WARNING|trainer.py:803] 2025-04-26 19:21:39,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1453
1537
[WARNING|trainer.py:803] 2025-04-26 19:21:41,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:41,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1475
1454
[WARNING|trainer.py:803] 2025-04-26 19:21:42,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:43,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1538
1476
1455
[WARNING|trainer.py:803] 2025-04-26 19:21:44,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:21:44,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:21:45,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1477
1456
[WARNING|trainer.py:803] 2025-04-26 19:21:46,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1539
[WARNING|trainer.py:803] 2025-04-26 19:21:47,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:48,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1478
1457
[WARNING|trainer.py:803] 2025-04-26 19:21:49,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:49,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1479
1540
[WARNING|trainer.py:803] 2025-04-26 19:21:51,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:21:51,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1458
[WARNING|trainer.py:803] 2025-04-26 19:21:52,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1480
[WARNING|trainer.py:803] 2025-04-26 19:21:53,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1541
1459
[WARNING|trainer.py:803] 2025-04-26 19:21:54,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:21:54,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1481
1460
[WARNING|trainer.py:803] 2025-04-26 19:21:55,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:21:56,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1542
1482
[WARNING|trainer.py:803] 2025-04-26 19:21:57,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1461
[WARNING|trainer.py:803] 2025-04-26 19:21:58,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:21:58,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1483
1462
1543
[WARNING|trainer.py:803] 2025-04-26 19:22:00,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:00,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:00,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1484
1463
[WARNING|trainer.py:803] 2025-04-26 19:22:02,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:03,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1544
[WARNING|trainer.py:803] 2025-04-26 19:22:04,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1485
1464
[WARNING|trainer.py:803] 2025-04-26 19:22:05,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:05,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1545
1465
1486
[WARNING|trainer.py:803] 2025-04-26 19:22:07,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:07,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:07,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1466
1487
[WARNING|trainer.py:803] 2025-04-26 19:22:09,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1546
[WARNING|trainer.py:803] 2025-04-26 19:22:10,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:10,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1467
1488
[WARNING|trainer.py:803] 2025-04-26 19:22:12,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:12,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1547
1468
1489
[WARNING|trainer.py:803] 2025-04-26 19:22:14,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:14,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:14,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1469
[WARNING|trainer.py:803] 2025-04-26 19:22:16,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1548
1490
[WARNING|trainer.py:803] 2025-04-26 19:22:17,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:17,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1470
[WARNING|trainer.py:803] 2025-04-26 19:22:18,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1491
1549
[WARNING|trainer.py:803] 2025-04-26 19:22:19,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1471
[WARNING|trainer.py:803] 2025-04-26 19:22:20,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:20,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1492
[WARNING|trainer.py:803] 2025-04-26 19:22:21,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1472
[WARNING|trainer.py:803] 2025-04-26 19:22:22,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1550
1493
[WARNING|trainer.py:803] 2025-04-26 19:22:23,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:23,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1473
1494
[WARNING|trainer.py:803] 2025-04-26 19:22:25,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:25,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1551
1474
[WARNING|trainer.py:803] 2025-04-26 19:22:26,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:27,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1495
[WARNING|trainer.py:803] 2025-04-26 19:22:28,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1475
1552
1496
[WARNING|trainer.py:803] 2025-04-26 19:22:29,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:30,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:30,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1476
1497
1553
[WARNING|trainer.py:803] 2025-04-26 19:22:32,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:32,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:32,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1477
1498
[WARNING|trainer.py:803] 2025-04-26 19:22:34,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:34,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1554
1478
[WARNING|trainer.py:803] 2025-04-26 19:22:36,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1499
[WARNING|trainer.py:803] 2025-04-26 19:22:36,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:37,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1479
1555
[WARNING|trainer.py:803] 2025-04-26 19:22:38,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1500
[WARNING|trainer.py:803] 2025-04-26 19:22:39,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:39,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1480
[WARNING|trainer.py:803] 2025-04-26 19:22:40,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1556
1501
1481
[WARNING|trainer.py:803] 2025-04-26 19:22:42,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:42,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:43,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1482
1557
1502
[WARNING|trainer.py:803] 2025-04-26 19:22:45,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:22:45,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:22:45,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1483
[WARNING|trainer.py:803] 2025-04-26 19:22:48,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1558
1503
[WARNING|trainer.py:803] 2025-04-26 19:22:48,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:49,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1484
[WARNING|trainer.py:803] 2025-04-26 19:22:50,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1559
1504
1485
[WARNING|trainer.py:803] 2025-04-26 19:22:52,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:52,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:52,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1486
1560
1505
[WARNING|trainer.py:803] 2025-04-26 19:22:55,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:55,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:22:55,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1487
1561
[WARNING|trainer.py:803] 2025-04-26 19:22:57,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:58,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1506
1488
[WARNING|trainer.py:803] 2025-04-26 19:22:59,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:22:59,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1489
1562
1507
[WARNING|trainer.py:803] 2025-04-26 19:23:01,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:01,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:02,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1490
1563
[WARNING|trainer.py:803] 2025-04-26 19:23:04,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:23:04,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1508
[WARNING|trainer.py:803] 2025-04-26 19:23:05,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1491
[WARNING|trainer.py:803] 2025-04-26 19:23:06,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1564
[WARNING|trainer.py:803] 2025-04-26 19:23:07,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1492
1509
[WARNING|trainer.py:803] 2025-04-26 19:23:09,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:09,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1565
1493
[WARNING|trainer.py:803] 2025-04-26 19:23:10,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:11,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1510
[WARNING|trainer.py:803] 2025-04-26 19:23:12,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1494
[WARNING|trainer.py:803] 2025-04-26 19:23:13,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1566
[WARNING|trainer.py:803] 2025-04-26 19:23:14,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1511
1495
[WARNING|trainer.py:803] 2025-04-26 19:23:15,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:15,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1567
1496
[WARNING|trainer.py:803] 2025-04-26 19:23:17,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:17,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1512
[WARNING|trainer.py:803] 2025-04-26 19:23:19,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1497
1568
[WARNING|trainer.py:803] 2025-04-26 19:23:20,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:20,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1513
1498
[WARNING|trainer.py:803] 2025-04-26 19:23:22,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:22,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1569
1499
[WARNING|trainer.py:803] 2025-04-26 19:23:24,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:23:24,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1514
[WARNING|trainer.py:803] 2025-04-26 19:23:25,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1500
1570
[WARNING|trainer.py:803] 2025-04-26 19:23:26,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:23:27,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1515
[WARNING|trainer.py:803] 2025-04-26 19:23:29,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1501
1571
[WARNING|trainer.py:803] 2025-04-26 19:23:30,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:30,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1516
[WARNING|trainer.py:803] 2025-04-26 19:23:32,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1502
1572
[WARNING|trainer.py:803] 2025-04-26 19:23:33,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:34,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1517
[WARNING|trainer.py:803] 2025-04-26 19:23:35,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1503
1573
[WARNING|trainer.py:803] 2025-04-26 19:23:36,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:37,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1518
[WARNING|trainer.py:803] 2025-04-26 19:23:39,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1504
1574
[WARNING|trainer.py:803] 2025-04-26 19:23:40,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:40,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1519
[WARNING|trainer.py:803] 2025-04-26 19:23:42,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1505
1575
[WARNING|trainer.py:803] 2025-04-26 19:23:43,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:23:43,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1520
[WARNING|trainer.py:803] 2025-04-26 19:23:45,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1576
1506
[WARNING|trainer.py:803] 2025-04-26 19:23:46,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:23:47,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1521
[WARNING|trainer.py:803] 2025-04-26 19:23:49,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1577
1507
[WARNING|trainer.py:803] 2025-04-26 19:23:50,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:50,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1522
1578
[WARNING|trainer.py:803] 2025-04-26 19:23:52,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1508
[WARNING|trainer.py:803] 2025-04-26 19:23:53,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:53,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1523
1579
[WARNING|trainer.py:803] 2025-04-26 19:23:56,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1509
[WARNING|trainer.py:803] 2025-04-26 19:23:56,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:23:57,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1524
1580
1510
[WARNING|trainer.py:803] 2025-04-26 19:23:59,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:23:59,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:24:00,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1581
1525
1511
[WARNING|trainer.py:803] 2025-04-26 19:24:03,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:03,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:24:03,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1582
1526
1512
[WARNING|trainer.py:803] 2025-04-26 19:24:06,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:06,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:06,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1583
1513
1527
[WARNING|trainer.py:803] 2025-04-26 19:24:10,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:10,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:10,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1584
1514
1528
[WARNING|trainer.py:803] 2025-04-26 19:24:13,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:13,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:13,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1515
1585
1529
[WARNING|trainer.py:803] 2025-04-26 19:24:16,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:24:16,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:17,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1516
1530
1586
[WARNING|trainer.py:803] 2025-04-26 19:24:20,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:20,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:20,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1531
1517
1587
[WARNING|trainer.py:803] 2025-04-26 19:24:24,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:24,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:24,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1588
1518
1532
[WARNING|trainer.py:803] 2025-04-26 19:24:27,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:27,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:27,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1589
1519
[WARNING|trainer.py:803] 2025-04-26 19:24:30,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1533
[WARNING|trainer.py:803] 2025-04-26 19:24:30,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:31,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1590
1520
[WARNING|trainer.py:803] 2025-04-26 19:24:33,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1534
[WARNING|trainer.py:803] 2025-04-26 19:24:34,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:24:34,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1591
1521
[WARNING|trainer.py:803] 2025-04-26 19:24:36,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:24:37,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1535
[WARNING|trainer.py:803] 2025-04-26 19:24:38,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1592
1522
[WARNING|trainer.py:803] 2025-04-26 19:24:40,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:41,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1536
[WARNING|trainer.py:803] 2025-04-26 19:24:41,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1593
1523
[WARNING|trainer.py:803] 2025-04-26 19:24:43,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1537
[WARNING|trainer.py:803] 2025-04-26 19:24:44,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:24:45,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1594
[WARNING|trainer.py:803] 2025-04-26 19:24:47,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1524
1538
[WARNING|trainer.py:803] 2025-04-26 19:24:48,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:48,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1595
[WARNING|trainer.py:803] 2025-04-26 19:24:50,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1525
1539
[WARNING|trainer.py:803] 2025-04-26 19:24:51,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:24:52,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1596
[WARNING|trainer.py:803] 2025-04-26 19:24:53,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1526
1540
[WARNING|trainer.py:803] 2025-04-26 19:24:55,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:24:55,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1597
[WARNING|trainer.py:803] 2025-04-26 19:24:56,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1527
1541
[WARNING|trainer.py:803] 2025-04-26 19:24:58,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:24:58,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1598
[WARNING|trainer.py:803] 2025-04-26 19:24:59,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1542
1528
[WARNING|trainer.py:803] 2025-04-26 19:25:02,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:25:02,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1599
[WARNING|trainer.py:803] 2025-04-26 19:25:03,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1529
1543
[WARNING|trainer.py:803] 2025-04-26 19:25:05,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:05,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1600
[WARNING|trainer.py:803] 2025-04-26 19:25:06,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1530
1544
[WARNING|trainer.py:803] 2025-04-26 19:25:08,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:08,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1601
[WARNING|trainer.py:803] 2025-04-26 19:25:09,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1531
1545
1602
[WARNING|trainer.py:803] 2025-04-26 19:25:12,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:12,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:13,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1546
1532
1603
[WARNING|trainer.py:803] 2025-04-26 19:25:15,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:15,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:25:16,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1547
1533
1604
[WARNING|trainer.py:803] 2025-04-26 19:25:19,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:25:19,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:19,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1548
1605
1534
[WARNING|trainer.py:803] 2025-04-26 19:25:22,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:25:22,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:22,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1549
1606
[WARNING|trainer.py:803] 2025-04-26 19:25:25,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1535
[WARNING|trainer.py:803] 2025-04-26 19:25:25,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:26,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1550
1607
1536
[WARNING|trainer.py:803] 2025-04-26 19:25:29,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:29,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:30,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1551
1608
[WARNING|trainer.py:803] 2025-04-26 19:25:32,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1537
[WARNING|trainer.py:803] 2025-04-26 19:25:33,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:25:33,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1552
1609
[WARNING|trainer.py:803] 2025-04-26 19:25:36,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1538
[WARNING|trainer.py:803] 2025-04-26 19:25:36,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:36,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1553
1610
[WARNING|trainer.py:803] 2025-04-26 19:25:39,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1539
[WARNING|trainer.py:803] 2025-04-26 19:25:39,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:40,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1554
1611
[WARNING|trainer.py:803] 2025-04-26 19:25:42,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1540
[WARNING|trainer.py:803] 2025-04-26 19:25:43,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:25:43,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1555
1612
[WARNING|trainer.py:803] 2025-04-26 19:25:46,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1541
[WARNING|trainer.py:803] 2025-04-26 19:25:46,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:47,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1556
1613
[WARNING|trainer.py:803] 2025-04-26 19:25:49,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:25:49,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1542
[WARNING|trainer.py:803] 2025-04-26 19:25:50,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1557
1614
[WARNING|trainer.py:803] 2025-04-26 19:25:52,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:25:52,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1543
[WARNING|trainer.py:803] 2025-04-26 19:25:53,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1615
1558
[WARNING|trainer.py:803] 2025-04-26 19:25:56,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:25:56,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1544
[WARNING|trainer.py:803] 2025-04-26 19:25:57,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1559
1616
[WARNING|trainer.py:803] 2025-04-26 19:25:59,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1545
[WARNING|trainer.py:803] 2025-04-26 19:25:59,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:26:00,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1560
1617
[WARNING|trainer.py:803] 2025-04-26 19:26:02,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1546
[WARNING|trainer.py:803] 2025-04-26 19:26:03,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:03,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1561
1618
[WARNING|trainer.py:803] 2025-04-26 19:26:06,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1547
[WARNING|trainer.py:803] 2025-04-26 19:26:06,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:07,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1619
1562
[WARNING|trainer.py:803] 2025-04-26 19:26:09,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1548
[WARNING|trainer.py:803] 2025-04-26 19:26:09,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:10,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1563
1549
1620
[WARNING|trainer.py:803] 2025-04-26 19:26:13,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:26:13,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:26:13,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1564
[WARNING|trainer.py:803] 2025-04-26 19:26:16,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1621
1550
[WARNING|trainer.py:803] 2025-04-26 19:26:17,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:17,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1565
1622
[WARNING|trainer.py:803] 2025-04-26 19:26:19,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1551
[WARNING|trainer.py:803] 2025-04-26 19:26:20,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:20,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1566
1623
1552
[WARNING|trainer.py:803] 2025-04-26 19:26:23,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:23,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:26:23,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1567
1624
1553
[WARNING|trainer.py:803] 2025-04-26 19:26:26,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:26,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:26:27,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1554
1568
1625
[WARNING|trainer.py:803] 2025-04-26 19:26:30,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:26:30,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:26:30,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1555
1626
1569
[WARNING|trainer.py:803] 2025-04-26 19:26:33,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:33,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:33,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1556
1570
1627
[WARNING|trainer.py:803] 2025-04-26 19:26:37,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:26:37,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:37,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1628
1557
1571
[WARNING|trainer.py:803] 2025-04-26 19:26:40,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:26:40,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:26:40,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1629
1558
1572
[WARNING|trainer.py:803] 2025-04-26 19:26:43,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:43,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:44,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1630
1559
1573
[WARNING|trainer.py:803] 2025-04-26 19:26:46,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:47,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:47,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1631
1560
1574
[WARNING|trainer.py:803] 2025-04-26 19:26:50,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:50,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:50,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1561
1632
1575
[WARNING|trainer.py:803] 2025-04-26 19:26:53,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:26:53,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:54,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1633
1562
1576
[WARNING|trainer.py:803] 2025-04-26 19:26:56,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:57,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:26:57,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1634
1563
1577
[WARNING|trainer.py:803] 2025-04-26 19:27:00,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:27:00,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:00,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1564
1635
1578
[WARNING|trainer.py:803] 2025-04-26 19:27:03,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:03,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:27:03,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1565
1579
1636
[WARNING|trainer.py:803] 2025-04-26 19:27:06,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:07,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:07,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1637
1566
1580
[WARNING|trainer.py:803] 2025-04-26 19:27:10,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:27:10,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:10,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1638
1581
1567
[WARNING|trainer.py:803] 2025-04-26 19:27:13,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:13,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:14,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1639
1582
1568
[WARNING|trainer.py:803] 2025-04-26 19:27:16,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:17,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:17,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1640
[WARNING|trainer.py:803] 2025-04-26 19:27:19,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1569
1583
[WARNING|trainer.py:803] 2025-04-26 19:27:21,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:21,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1641
[WARNING|trainer.py:803] 2025-04-26 19:27:23,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1570
1584
[WARNING|trainer.py:803] 2025-04-26 19:27:24,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:24,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1642
[WARNING|trainer.py:803] 2025-04-26 19:27:26,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1571
1585
[WARNING|trainer.py:803] 2025-04-26 19:27:27,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:27:27,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1643
[WARNING|trainer.py:803] 2025-04-26 19:27:30,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1572
1586
[WARNING|trainer.py:803] 2025-04-26 19:27:31,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:32,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1644
[WARNING|trainer.py:803] 2025-04-26 19:27:33,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1573
[WARNING|trainer.py:803] 2025-04-26 19:27:34,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1587
[WARNING|trainer.py:803] 2025-04-26 19:27:35,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1645
[WARNING|trainer.py:803] 2025-04-26 19:27:36,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1574
[WARNING|trainer.py:803] 2025-04-26 19:27:37,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1588
[WARNING|trainer.py:803] 2025-04-26 19:27:38,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1646
[WARNING|trainer.py:803] 2025-04-26 19:27:39,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1575
[WARNING|trainer.py:803] 2025-04-26 19:27:41,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1589
[WARNING|trainer.py:803] 2025-04-26 19:27:42,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1647
[WARNING|trainer.py:803] 2025-04-26 19:27:43,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1576
[WARNING|trainer.py:803] 2025-04-26 19:27:44,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1590
[WARNING|trainer.py:803] 2025-04-26 19:27:45,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1648
[WARNING|trainer.py:803] 2025-04-26 19:27:46,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1577
[WARNING|trainer.py:803] 2025-04-26 19:27:47,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1591
1649
[WARNING|trainer.py:803] 2025-04-26 19:27:48,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:27:48,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1578
1650
[WARNING|trainer.py:803] 2025-04-26 19:27:51,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:27:51,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1592
[WARNING|trainer.py:803] 2025-04-26 19:27:52,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1579
1651
[WARNING|trainer.py:803] 2025-04-26 19:27:54,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:54,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1593
[WARNING|trainer.py:803] 2025-04-26 19:27:55,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1652
1580
[WARNING|trainer.py:803] 2025-04-26 19:27:57,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:27:57,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1594
[WARNING|trainer.py:803] 2025-04-26 19:27:59,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1653
[WARNING|trainer.py:803] 2025-04-26 19:28:00,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1581
[WARNING|trainer.py:803] 2025-04-26 19:28:01,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1654
1595
[WARNING|trainer.py:803] 2025-04-26 19:28:02,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:28:03,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1582
[WARNING|trainer.py:803] 2025-04-26 19:28:04,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1655
[WARNING|trainer.py:803] 2025-04-26 19:28:05,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1596
[WARNING|trainer.py:803] 2025-04-26 19:28:06,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1656
1583
[WARNING|trainer.py:803] 2025-04-26 19:28:07,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:08,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1597
[WARNING|trainer.py:803] 2025-04-26 19:28:09,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1657
[WARNING|trainer.py:803] 2025-04-26 19:28:10,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1584
[WARNING|trainer.py:803] 2025-04-26 19:28:11,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1598
1658
[WARNING|trainer.py:803] 2025-04-26 19:28:12,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:13,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1585
1659
[WARNING|trainer.py:803] 2025-04-26 19:28:14,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1599
[WARNING|trainer.py:803] 2025-04-26 19:28:15,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:16,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1660
1586
[WARNING|trainer.py:803] 2025-04-26 19:28:18,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:28:18,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1600
[WARNING|trainer.py:803] 2025-04-26 19:28:19,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1661
[WARNING|trainer.py:803] 2025-04-26 19:28:21,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1587
[WARNING|trainer.py:803] 2025-04-26 19:28:21,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1601
1662
[WARNING|trainer.py:803] 2025-04-26 19:28:23,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:23,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1588
[WARNING|trainer.py:803] 2025-04-26 19:28:25,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1663
1602
[WARNING|trainer.py:803] 2025-04-26 19:28:26,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:28:26,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1589
1664
[WARNING|trainer.py:803] 2025-04-26 19:28:28,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:28:28,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1603
[WARNING|trainer.py:803] 2025-04-26 19:28:29,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1665
1590
[WARNING|trainer.py:803] 2025-04-26 19:28:31,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:31,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1604
1666
[WARNING|trainer.py:803] 2025-04-26 19:28:33,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:28:33,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1591
[WARNING|trainer.py:803] 2025-04-26 19:28:35,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1605
1667
[WARNING|trainer.py:803] 2025-04-26 19:28:36,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:36,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1592
1668
[WARNING|trainer.py:803] 2025-04-26 19:28:38,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:39,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1606
[WARNING|trainer.py:803] 2025-04-26 19:28:39,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1669
1593
[WARNING|trainer.py:803] 2025-04-26 19:28:41,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:28:42,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1607
1670
[WARNING|trainer.py:803] 2025-04-26 19:28:43,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:44,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1594
[WARNING|trainer.py:803] 2025-04-26 19:28:45,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1671
1608
[WARNING|trainer.py:803] 2025-04-26 19:28:46,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:28:47,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1595
1672
[WARNING|trainer.py:803] 2025-04-26 19:28:49,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:28:49,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1609
[WARNING|trainer.py:803] 2025-04-26 19:28:50,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1673
1596
[WARNING|trainer.py:803] 2025-04-26 19:28:51,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:28:52,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1674
1610
[WARNING|trainer.py:803] 2025-04-26 19:28:54,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:54,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1597
[WARNING|trainer.py:803] 2025-04-26 19:28:55,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1675
[WARNING|trainer.py:803] 2025-04-26 19:28:56,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1611
[WARNING|trainer.py:803] 2025-04-26 19:28:57,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1598
1676
[WARNING|trainer.py:803] 2025-04-26 19:28:59,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:28:59,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1612
[WARNING|trainer.py:803] 2025-04-26 19:29:00,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1677
1599
[WARNING|trainer.py:803] 2025-04-26 19:29:01,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:02,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1613
1678
[WARNING|trainer.py:803] 2025-04-26 19:29:04,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:04,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1600
[WARNING|trainer.py:803] 2025-04-26 19:29:05,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1679
1614
[WARNING|trainer.py:803] 2025-04-26 19:29:06,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:29:07,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1601
1680
[WARNING|trainer.py:803] 2025-04-26 19:29:09,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:09,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1615
[WARNING|trainer.py:803] 2025-04-26 19:29:11,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1681
1602
[WARNING|trainer.py:803] 2025-04-26 19:29:12,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:12,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1682
1616
[WARNING|trainer.py:803] 2025-04-26 19:29:14,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:29:14,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1603
[WARNING|trainer.py:803] 2025-04-26 19:29:15,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1683
[WARNING|trainer.py:803] 2025-04-26 19:29:17,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1617
[WARNING|trainer.py:803] 2025-04-26 19:29:18,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1604
1684
[WARNING|trainer.py:803] 2025-04-26 19:29:19,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:19,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1618
1605
1685
[WARNING|trainer.py:803] 2025-04-26 19:29:21,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:22,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:22,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1619
1686
[WARNING|trainer.py:803] 2025-04-26 19:29:25,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1606
[WARNING|trainer.py:803] 2025-04-26 19:29:25,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:29:26,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1687
[WARNING|trainer.py:803] 2025-04-26 19:29:27,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1620
1607
[WARNING|trainer.py:803] 2025-04-26 19:29:29,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1688
[WARNING|trainer.py:803] 2025-04-26 19:29:29,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:30,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1621
1689
1608
[WARNING|trainer.py:803] 2025-04-26 19:29:32,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:33,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:33,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1690
1622
1609
[WARNING|trainer.py:803] 2025-04-26 19:29:35,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:36,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:36,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1691
[WARNING|trainer.py:803] 2025-04-26 19:29:38,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1623
1610
[WARNING|trainer.py:803] 2025-04-26 19:29:39,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1692
[WARNING|trainer.py:803] 2025-04-26 19:29:40,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:29:41,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1624
[WARNING|trainer.py:803] 2025-04-26 19:29:42,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1693
1611
[WARNING|trainer.py:803] 2025-04-26 19:29:43,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:43,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1694
1612
1625
[WARNING|trainer.py:803] 2025-04-26 19:29:46,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:46,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:29:46,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1695
[WARNING|trainer.py:803] 2025-04-26 19:29:49,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1626
1613
[WARNING|trainer.py:803] 2025-04-26 19:29:50,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:50,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1696
[WARNING|trainer.py:803] 2025-04-26 19:29:51,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1614
1627
[WARNING|trainer.py:803] 2025-04-26 19:29:53,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1697
[WARNING|trainer.py:803] 2025-04-26 19:29:54,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:54,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1628
1615
1698
[WARNING|trainer.py:803] 2025-04-26 19:29:57,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:29:57,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:29:57,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1699
1629
[WARNING|trainer.py:803] 2025-04-26 19:29:59,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1616
[WARNING|trainer.py:803] 2025-04-26 19:30:00,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:01,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1700
[WARNING|trainer.py:803] 2025-04-26 19:30:02,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1630
1617
[WARNING|trainer.py:803] 2025-04-26 19:30:03,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1701
[WARNING|trainer.py:803] 2025-04-26 19:30:04,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:04,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1631
1702
[WARNING|trainer.py:803] 2025-04-26 19:30:07,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1618
[WARNING|trainer.py:803] 2025-04-26 19:30:07,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:07,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1703
1632
[WARNING|trainer.py:803] 2025-04-26 19:30:10,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1619
[WARNING|trainer.py:803] 2025-04-26 19:30:10,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:11,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1704
[WARNING|trainer.py:803] 2025-04-26 19:30:12,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1633
[WARNING|trainer.py:803] 2025-04-26 19:30:13,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1705
1620
[WARNING|trainer.py:803] 2025-04-26 19:30:14,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:30:15,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1706
1634
[WARNING|trainer.py:803] 2025-04-26 19:30:17,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:30:17,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1621
[WARNING|trainer.py:803] 2025-04-26 19:30:18,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1707
[WARNING|trainer.py:803] 2025-04-26 19:30:20,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1635
[WARNING|trainer.py:803] 2025-04-26 19:30:20,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1622
1708
[WARNING|trainer.py:803] 2025-04-26 19:30:22,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:22,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1636
1709
[WARNING|trainer.py:803] 2025-04-26 19:30:24,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1623
[WARNING|trainer.py:803] 2025-04-26 19:30:25,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:25,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1710
1637
[WARNING|trainer.py:803] 2025-04-26 19:30:27,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:30:27,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1624
[WARNING|trainer.py:803] 2025-04-26 19:30:28,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1711
[WARNING|trainer.py:803] 2025-04-26 19:30:30,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1638
[WARNING|trainer.py:803] 2025-04-26 19:30:31,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1625
1712
[WARNING|trainer.py:803] 2025-04-26 19:30:32,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:30:32,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1639
1713
[WARNING|trainer.py:803] 2025-04-26 19:30:34,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:30:34,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1626
[WARNING|trainer.py:803] 2025-04-26 19:30:35,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1640
1714
[WARNING|trainer.py:803] 2025-04-26 19:30:37,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:37,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1627
1715
[WARNING|trainer.py:803] 2025-04-26 19:30:39,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:40,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1641
[WARNING|trainer.py:803] 2025-04-26 19:30:41,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1716
1628
[WARNING|trainer.py:803] 2025-04-26 19:30:42,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:42,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1642
1717
[WARNING|trainer.py:803] 2025-04-26 19:30:44,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:44,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1629
[WARNING|trainer.py:803] 2025-04-26 19:30:46,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1718
[WARNING|trainer.py:803] 2025-04-26 19:30:47,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1643
[WARNING|trainer.py:803] 2025-04-26 19:30:48,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1630
1719
[WARNING|trainer.py:803] 2025-04-26 19:30:49,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:30:49,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1644
1720
[WARNING|trainer.py:803] 2025-04-26 19:30:51,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1631
[WARNING|trainer.py:803] 2025-04-26 19:30:52,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:30:52,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1721
1645
[WARNING|trainer.py:803] 2025-04-26 19:30:54,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:30:55,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1632
[WARNING|trainer.py:803] 2025-04-26 19:30:56,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1722
1646
[WARNING|trainer.py:803] 2025-04-26 19:30:57,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:30:58,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1633
1723
[WARNING|trainer.py:803] 2025-04-26 19:30:59,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:00,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1647
[WARNING|trainer.py:803] 2025-04-26 19:31:01,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1724
1634
[WARNING|trainer.py:803] 2025-04-26 19:31:02,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:03,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1648
1725
[WARNING|trainer.py:803] 2025-04-26 19:31:04,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:31:05,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1635
[WARNING|trainer.py:803] 2025-04-26 19:31:06,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1726
1649
[WARNING|trainer.py:803] 2025-04-26 19:31:07,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:07,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1727
1636
1650
[WARNING|trainer.py:803] 2025-04-26 19:31:10,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:10,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:10,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1728
[WARNING|trainer.py:803] 2025-04-26 19:31:12,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1651
1637
[WARNING|trainer.py:803] 2025-04-26 19:31:13,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:31:13,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1729
[WARNING|trainer.py:803] 2025-04-26 19:31:15,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1652
1638
[WARNING|trainer.py:803] 2025-04-26 19:31:16,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1730
[WARNING|trainer.py:803] 2025-04-26 19:31:16,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:31:17,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1653
1731
[WARNING|trainer.py:803] 2025-04-26 19:31:19,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1639
[WARNING|trainer.py:803] 2025-04-26 19:31:19,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:20,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1654
1732
[WARNING|trainer.py:803] 2025-04-26 19:31:22,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:22,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1640
[WARNING|trainer.py:803] 2025-04-26 19:31:23,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1655
1733
[WARNING|trainer.py:803] 2025-04-26 19:31:24,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:24,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1641
1734
1656
[WARNING|trainer.py:803] 2025-04-26 19:31:27,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:27,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:27,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1735
1657
1642
[WARNING|trainer.py:803] 2025-04-26 19:31:29,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:30,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:30,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1736
1658
[WARNING|trainer.py:803] 2025-04-26 19:31:32,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1643
[WARNING|trainer.py:803] 2025-04-26 19:31:33,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:33,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1737
1659
[WARNING|trainer.py:803] 2025-04-26 19:31:35,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:35,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1644
1738
[WARNING|trainer.py:803] 2025-04-26 19:31:37,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:31:37,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1660
[WARNING|trainer.py:803] 2025-04-26 19:31:38,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1739
1645
[WARNING|trainer.py:803] 2025-04-26 19:31:40,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:40,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1661
[WARNING|trainer.py:803] 2025-04-26 19:31:41,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1740
1646
[WARNING|trainer.py:803] 2025-04-26 19:31:42,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1662
[WARNING|trainer.py:803] 2025-04-26 19:31:43,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:44,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1741
[WARNING|trainer.py:803] 2025-04-26 19:31:45,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1663
1647
[WARNING|trainer.py:803] 2025-04-26 19:31:46,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1742
[WARNING|trainer.py:803] 2025-04-26 19:31:47,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:47,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1664
1743
[WARNING|trainer.py:803] 2025-04-26 19:31:49,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1648
[WARNING|trainer.py:803] 2025-04-26 19:31:50,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:50,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1665
1744
[WARNING|trainer.py:803] 2025-04-26 19:31:52,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1649
[WARNING|trainer.py:803] 2025-04-26 19:31:53,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:31:53,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1666
1745
[WARNING|trainer.py:803] 2025-04-26 19:31:55,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:31:55,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1650
[WARNING|trainer.py:803] 2025-04-26 19:31:56,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1667
1746
[WARNING|trainer.py:803] 2025-04-26 19:31:57,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:31:58,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1651
[WARNING|trainer.py:803] 2025-04-26 19:31:59,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1668
1747
[WARNING|trainer.py:803] 2025-04-26 19:32:00,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:00,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1652
[WARNING|trainer.py:803] 2025-04-26 19:32:02,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1748
1669
[WARNING|trainer.py:803] 2025-04-26 19:32:03,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:03,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1653
1670
[WARNING|trainer.py:803] 2025-04-26 19:32:05,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1749
[WARNING|trainer.py:803] 2025-04-26 19:32:06,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:06,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1654
1671
[WARNING|trainer.py:803] 2025-04-26 19:32:07,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1750
[WARNING|trainer.py:803] 2025-04-26 19:32:08,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:09,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1655
1672
1751
[WARNING|trainer.py:803] 2025-04-26 19:32:10,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:11,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:11,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1656
1673
1752
[WARNING|trainer.py:803] 2025-04-26 19:32:13,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:13,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:14,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1674
1657
1753
[WARNING|trainer.py:803] 2025-04-26 19:32:16,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:16,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:16,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1754
1675
1658
[WARNING|trainer.py:803] 2025-04-26 19:32:18,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:18,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:19,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1755
1676
1659
[WARNING|trainer.py:803] 2025-04-26 19:32:21,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:21,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:21,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1756
1677
1660
[WARNING|trainer.py:803] 2025-04-26 19:32:24,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:24,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:32:24,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1757
1678
[WARNING|trainer.py:803] 2025-04-26 19:32:26,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1661
[WARNING|trainer.py:803] 2025-04-26 19:32:27,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:27,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1758
1662
1679
[WARNING|trainer.py:803] 2025-04-26 19:32:29,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:30,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:30,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1759
[WARNING|trainer.py:803] 2025-04-26 19:32:31,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1680
1663
[WARNING|trainer.py:803] 2025-04-26 19:32:32,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:32:33,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1760
[WARNING|trainer.py:803] 2025-04-26 19:32:34,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1681
1664
[WARNING|trainer.py:803] 2025-04-26 19:32:35,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:32:35,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1761
[WARNING|trainer.py:803] 2025-04-26 19:32:36,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1682
1665
1762
[WARNING|trainer.py:803] 2025-04-26 19:32:38,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:38,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:39,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1683
1666
1763
[WARNING|trainer.py:803] 2025-04-26 19:32:40,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:41,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:41,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1764
1684
1667
[WARNING|trainer.py:803] 2025-04-26 19:32:43,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:43,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:44,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1765
1668
1685
[WARNING|trainer.py:803] 2025-04-26 19:32:46,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:46,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:46,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1766
1686
1669
[WARNING|trainer.py:803] 2025-04-26 19:32:49,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:49,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:49,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1767
1687
1670
[WARNING|trainer.py:803] 2025-04-26 19:32:51,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:32:52,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:32:52,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1768
1671
[WARNING|trainer.py:803] 2025-04-26 19:32:54,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1688
[WARNING|trainer.py:803] 2025-04-26 19:32:54,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:32:55,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1769
[WARNING|trainer.py:803] 2025-04-26 19:32:56,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1672
1689
[WARNING|trainer.py:803] 2025-04-26 19:32:57,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:32:57,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1770
[WARNING|trainer.py:803] 2025-04-26 19:32:59,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1673
1690
[WARNING|trainer.py:803] 2025-04-26 19:33:00,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:00,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1771
1674
[WARNING|trainer.py:803] 2025-04-26 19:33:01,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1691
[WARNING|trainer.py:803] 2025-04-26 19:33:02,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:33:03,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1772
[WARNING|trainer.py:803] 2025-04-26 19:33:04,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1675
[WARNING|trainer.py:803] 2025-04-26 19:33:05,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1692
1773
[WARNING|trainer.py:803] 2025-04-26 19:33:06,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:06,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1676
[WARNING|trainer.py:803] 2025-04-26 19:33:07,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1693
1774
[WARNING|trainer.py:803] 2025-04-26 19:33:08,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:09,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1677
[WARNING|trainer.py:803] 2025-04-26 19:33:10,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1694
1775
[WARNING|trainer.py:803] 2025-04-26 19:33:11,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:33:12,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1678
[WARNING|trainer.py:803] 2025-04-26 19:33:13,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1695
1776
[WARNING|trainer.py:803] 2025-04-26 19:33:14,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:14,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1679
[WARNING|trainer.py:803] 2025-04-26 19:33:15,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1777
1696
[WARNING|trainer.py:803] 2025-04-26 19:33:17,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:17,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1680
[WARNING|trainer.py:803] 2025-04-26 19:33:18,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1778
1697
[WARNING|trainer.py:803] 2025-04-26 19:33:19,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:20,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1681
1779
[WARNING|trainer.py:803] 2025-04-26 19:33:21,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:21,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1698
[WARNING|trainer.py:803] 2025-04-26 19:33:23,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1682
1780
[WARNING|trainer.py:803] 2025-04-26 19:33:23,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:24,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1699
[WARNING|trainer.py:803] 2025-04-26 19:33:25,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1683
1781
[WARNING|trainer.py:803] 2025-04-26 19:33:26,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:26,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1700
[WARNING|trainer.py:803] 2025-04-26 19:33:28,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1782
1684
[WARNING|trainer.py:803] 2025-04-26 19:33:29,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:29,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1701
[WARNING|trainer.py:803] 2025-04-26 19:33:31,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1783
1685
[WARNING|trainer.py:803] 2025-04-26 19:33:32,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:32,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1702
1784
[WARNING|trainer.py:803] 2025-04-26 19:33:34,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1686
[WARNING|trainer.py:803] 2025-04-26 19:33:34,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:35,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1703
1785
[WARNING|trainer.py:803] 2025-04-26 19:33:36,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1687
[WARNING|trainer.py:803] 2025-04-26 19:33:37,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:37,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1704
1786
[WARNING|trainer.py:803] 2025-04-26 19:33:39,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1688
[WARNING|trainer.py:803] 2025-04-26 19:33:40,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:40,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1705
[WARNING|trainer.py:803] 2025-04-26 19:33:42,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1787
1689
[WARNING|trainer.py:803] 2025-04-26 19:33:43,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:43,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1706
1788
[WARNING|trainer.py:803] 2025-04-26 19:33:44,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1690
[WARNING|trainer.py:803] 2025-04-26 19:33:45,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:46,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1707
1789
[WARNING|trainer.py:803] 2025-04-26 19:33:47,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1691
[WARNING|trainer.py:803] 2025-04-26 19:33:48,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:33:48,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1708
1790
[WARNING|trainer.py:803] 2025-04-26 19:33:50,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:50,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1692
[WARNING|trainer.py:803] 2025-04-26 19:33:51,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1709
1791
[WARNING|trainer.py:803] 2025-04-26 19:33:52,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:33:53,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1693
1710
[WARNING|trainer.py:803] 2025-04-26 19:33:54,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1792
[WARNING|trainer.py:803] 2025-04-26 19:33:55,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:33:55,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1694
1793
1711
[WARNING|trainer.py:803] 2025-04-26 19:33:57,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:33:58,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:33:58,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1695
1794
1712
[WARNING|trainer.py:803] 2025-04-26 19:34:00,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:00,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:00,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1795
1696
1713
[WARNING|trainer.py:803] 2025-04-26 19:34:03,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:03,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:03,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1796
1697
1714
[WARNING|trainer.py:803] 2025-04-26 19:34:05,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:06,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:06,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1797
1715
1698
[WARNING|trainer.py:803] 2025-04-26 19:34:08,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:34:08,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:09,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1798
1716
1699
[WARNING|trainer.py:803] 2025-04-26 19:34:11,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:11,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:11,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1799
1717
[WARNING|trainer.py:803] 2025-04-26 19:34:13,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1700
[WARNING|trainer.py:803] 2025-04-26 19:34:14,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:34:14,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1800
1718
[WARNING|trainer.py:803] 2025-04-26 19:34:16,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1701
[WARNING|trainer.py:803] 2025-04-26 19:34:16,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:17,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1801
[WARNING|trainer.py:803] 2025-04-26 19:34:18,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1719
1702
1802
[WARNING|trainer.py:803] 2025-04-26 19:34:19,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:20,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:20,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1720
1803
[WARNING|trainer.py:803] 2025-04-26 19:34:22,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1703
[WARNING|trainer.py:803] 2025-04-26 19:34:22,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:22,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1721
1804
1704
[WARNING|trainer.py:803] 2025-04-26 19:34:24,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:24,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:25,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1805
[WARNING|trainer.py:803] 2025-04-26 19:34:26,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1722
1705
[WARNING|trainer.py:803] 2025-04-26 19:34:27,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1806
[WARNING|trainer.py:803] 2025-04-26 19:34:27,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:28,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1807
1723
1706
[WARNING|trainer.py:803] 2025-04-26 19:34:30,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:30,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:30,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1808
[WARNING|trainer.py:803] 2025-04-26 19:34:31,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1724
1707
[WARNING|trainer.py:803] 2025-04-26 19:34:32,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1809
[WARNING|trainer.py:803] 2025-04-26 19:34:33,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:34,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1725
1810
1708
[WARNING|trainer.py:803] 2025-04-26 19:34:35,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:34:35,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:36,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1811
1726
[WARNING|trainer.py:803] 2025-04-26 19:34:37,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1709
[WARNING|trainer.py:803] 2025-04-26 19:34:38,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1812
[WARNING|trainer.py:803] 2025-04-26 19:34:38,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:39,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1727
1813
1710
[WARNING|trainer.py:803] 2025-04-26 19:34:40,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:41,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:41,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1814
1728
[WARNING|trainer.py:803] 2025-04-26 19:34:43,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1711
[WARNING|trainer.py:803] 2025-04-26 19:34:43,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:34:43,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1815
[WARNING|trainer.py:803] 2025-04-26 19:34:44,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1729
1712
[WARNING|trainer.py:803] 2025-04-26 19:34:46,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1816
[WARNING|trainer.py:803] 2025-04-26 19:34:46,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:34:46,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1730
1817
1713
[WARNING|trainer.py:803] 2025-04-26 19:34:48,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:34:48,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:34:49,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1818
1731
[WARNING|trainer.py:803] 2025-04-26 19:34:50,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1714
[WARNING|trainer.py:803] 2025-04-26 19:34:51,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1819
[WARNING|trainer.py:803] 2025-04-26 19:34:52,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:34:52,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1732
1715
1820
[WARNING|trainer.py:803] 2025-04-26 19:34:53,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:54,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:34:54,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1821
1733
1716
[WARNING|trainer.py:803] 2025-04-26 19:34:56,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:34:56,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:34:57,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1822
1734
[WARNING|trainer.py:803] 2025-04-26 19:34:58,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1717
[WARNING|trainer.py:803] 2025-04-26 19:34:59,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:34:59,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1823
[WARNING|trainer.py:803] 2025-04-26 19:35:00,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1735
1718
[WARNING|trainer.py:803] 2025-04-26 19:35:01,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:01,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1824
[WARNING|trainer.py:803] 2025-04-26 19:35:03,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1736
1719
1825
[WARNING|trainer.py:803] 2025-04-26 19:35:04,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:04,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:04,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1826
1720
1737
[WARNING|trainer.py:803] 2025-04-26 19:35:06,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:07,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:35:07,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1827
[WARNING|trainer.py:803] 2025-04-26 19:35:08,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1721
1738
1828
[WARNING|trainer.py:803] 2025-04-26 19:35:10,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:10,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:10,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1829
1739
1722
[WARNING|trainer.py:803] 2025-04-26 19:35:12,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:12,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:12,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1830
1740
[WARNING|trainer.py:803] 2025-04-26 19:35:14,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1723
[WARNING|trainer.py:803] 2025-04-26 19:35:15,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:15,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1831
[WARNING|trainer.py:803] 2025-04-26 19:35:17,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1741
1724
1832
[WARNING|trainer.py:803] 2025-04-26 19:35:18,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:18,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:18,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1833
1742
1725
[WARNING|trainer.py:803] 2025-04-26 19:35:20,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:20,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:20,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1834
[WARNING|trainer.py:803] 2025-04-26 19:35:22,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1743
1726
[WARNING|trainer.py:803] 2025-04-26 19:35:23,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:23,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1835
[WARNING|trainer.py:803] 2025-04-26 19:35:24,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1727
1836
1744
[WARNING|trainer.py:803] 2025-04-26 19:35:26,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:26,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:35:26,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1837
1745
1728
[WARNING|trainer.py:803] 2025-04-26 19:35:28,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:28,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:28,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1838
[WARNING|trainer.py:803] 2025-04-26 19:35:30,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1729
1746
1839
[WARNING|trainer.py:803] 2025-04-26 19:35:31,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:31,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:31,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1840
1730
1747
[WARNING|trainer.py:803] 2025-04-26 19:35:33,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:33,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:34,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1841
1731
[WARNING|trainer.py:803] 2025-04-26 19:35:35,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1748
[WARNING|trainer.py:803] 2025-04-26 19:35:36,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:37,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1842
[WARNING|trainer.py:803] 2025-04-26 19:35:37,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1732
1843
1749
[WARNING|trainer.py:803] 2025-04-26 19:35:39,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:39,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:39,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1733
1844
[WARNING|trainer.py:803] 2025-04-26 19:35:41,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1750
[WARNING|trainer.py:803] 2025-04-26 19:35:42,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:35:42,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1845
1734
1751
[WARNING|trainer.py:803] 2025-04-26 19:35:44,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:44,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:35:45,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1846
1735
[WARNING|trainer.py:803] 2025-04-26 19:35:46,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1752
[WARNING|trainer.py:803] 2025-04-26 19:35:47,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1847
[WARNING|trainer.py:803] 2025-04-26 19:35:47,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:35:48,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1736
1848
1753
[WARNING|trainer.py:803] 2025-04-26 19:35:49,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:50,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:50,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1849
1737
[WARNING|trainer.py:803] 2025-04-26 19:35:52,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1754
[WARNING|trainer.py:803] 2025-04-26 19:35:52,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:53,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1850
[WARNING|trainer.py:803] 2025-04-26 19:35:54,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1738
1755
[WARNING|trainer.py:803] 2025-04-26 19:35:55,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1851
[WARNING|trainer.py:803] 2025-04-26 19:35:55,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:35:56,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1739
1756
1852
[WARNING|trainer.py:803] 2025-04-26 19:35:58,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:58,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:35:58,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1853
1740
1757
[WARNING|trainer.py:803] 2025-04-26 19:36:00,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:00,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:01,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1854
1741
[WARNING|trainer.py:803] 2025-04-26 19:36:02,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1758
[WARNING|trainer.py:803] 2025-04-26 19:36:03,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:04,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1855
[WARNING|trainer.py:803] 2025-04-26 19:36:05,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1742
1759
1856
[WARNING|trainer.py:803] 2025-04-26 19:36:06,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:06,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:06,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1857
1743
1760
[WARNING|trainer.py:803] 2025-04-26 19:36:08,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:36:08,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:09,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1858
1744
[WARNING|trainer.py:803] 2025-04-26 19:36:10,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1761
[WARNING|trainer.py:803] 2025-04-26 19:36:11,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:36:11,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1859
[WARNING|trainer.py:803] 2025-04-26 19:36:12,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1745
1762
1860
[WARNING|trainer.py:803] 2025-04-26 19:36:14,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:14,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:14,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1861
1763
1746
[WARNING|trainer.py:803] 2025-04-26 19:36:16,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:16,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:16,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1862
1764
[WARNING|trainer.py:803] 2025-04-26 19:36:18,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1747
[WARNING|trainer.py:803] 2025-04-26 19:36:19,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:19,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1863
[WARNING|trainer.py:803] 2025-04-26 19:36:20,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1765
1864
1748
[WARNING|trainer.py:803] 2025-04-26 19:36:22,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:22,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:22,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1865
1766
1749
[WARNING|trainer.py:803] 2025-04-26 19:36:24,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:24,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:25,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1866
[WARNING|trainer.py:803] 2025-04-26 19:36:26,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1767
1867
1750
[WARNING|trainer.py:803] 2025-04-26 19:36:27,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:28,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:36:28,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1768
1868
1751
[WARNING|trainer.py:803] 2025-04-26 19:36:30,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:30,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:30,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1869
1769
[WARNING|trainer.py:803] 2025-04-26 19:36:32,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1752
[WARNING|trainer.py:803] 2025-04-26 19:36:32,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1870
[WARNING|trainer.py:803] 2025-04-26 19:36:33,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:36:34,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1770
1871
1753
[WARNING|trainer.py:803] 2025-04-26 19:36:35,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:36,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:36,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1872
1771
1754
[WARNING|trainer.py:803] 2025-04-26 19:36:38,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:38,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:38,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1873
1772
[WARNING|trainer.py:803] 2025-04-26 19:36:40,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1755
[WARNING|trainer.py:803] 2025-04-26 19:36:40,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1874
[WARNING|trainer.py:803] 2025-04-26 19:36:41,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:36:41,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1773
1875
1756
[WARNING|trainer.py:803] 2025-04-26 19:36:43,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:36:43,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:36:43,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1876
1774
[WARNING|trainer.py:803] 2025-04-26 19:36:45,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1757
[WARNING|trainer.py:803] 2025-04-26 19:36:46,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:36:46,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1877
[WARNING|trainer.py:803] 2025-04-26 19:36:47,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1775
1878
1758
[WARNING|trainer.py:803] 2025-04-26 19:36:49,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:36:49,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:49,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1879
1759
[WARNING|trainer.py:803] 2025-04-26 19:36:51,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1776
[WARNING|trainer.py:803] 2025-04-26 19:36:52,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:52,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1880
[WARNING|trainer.py:803] 2025-04-26 19:36:53,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1760
1777
1881
[WARNING|trainer.py:803] 2025-04-26 19:36:54,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:54,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:55,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1882
1761
1778
[WARNING|trainer.py:803] 2025-04-26 19:36:57,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:57,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:36:57,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1883
1779
1762
[WARNING|trainer.py:803] 2025-04-26 19:36:59,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:36:59,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:00,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1884
1780
1763
[WARNING|trainer.py:803] 2025-04-26 19:37:01,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:02,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:02,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1885
[WARNING|trainer.py:803] 2025-04-26 19:37:04,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1781
1764
[WARNING|trainer.py:803] 2025-04-26 19:37:05,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:05,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1886
[WARNING|trainer.py:803] 2025-04-26 19:37:06,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1782
1765
1887
[WARNING|trainer.py:803] 2025-04-26 19:37:07,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:07,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:08,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1888
1766
1783
[WARNING|trainer.py:803] 2025-04-26 19:37:10,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:10,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:10,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1889
[WARNING|trainer.py:803] 2025-04-26 19:37:11,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1767
1784
1890
[WARNING|trainer.py:803] 2025-04-26 19:37:12,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:13,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:13,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1891
1768
1785
[WARNING|trainer.py:803] 2025-04-26 19:37:15,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:15,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:15,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1892
1769
1786
[WARNING|trainer.py:803] 2025-04-26 19:37:18,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:18,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:18,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1893
[WARNING|trainer.py:803] 2025-04-26 19:37:19,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1770
1787
[WARNING|trainer.py:803] 2025-04-26 19:37:20,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1894
[WARNING|trainer.py:803] 2025-04-26 19:37:21,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:21,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1771
1895
1788
[WARNING|trainer.py:803] 2025-04-26 19:37:23,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:23,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:24,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1896
1772
[WARNING|trainer.py:803] 2025-04-26 19:37:25,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1789
[WARNING|trainer.py:803] 2025-04-26 19:37:26,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1897
[WARNING|trainer.py:803] 2025-04-26 19:37:26,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:27,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1773
1790
[WARNING|trainer.py:803] 2025-04-26 19:37:28,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1898
[WARNING|trainer.py:803] 2025-04-26 19:37:29,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:30,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1774
1899
1791
[WARNING|trainer.py:803] 2025-04-26 19:37:31,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:32,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:32,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1900
1792
1775
[WARNING|trainer.py:803] 2025-04-26 19:37:34,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:34,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:34,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1901
[WARNING|trainer.py:803] 2025-04-26 19:37:36,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1793
1776
[WARNING|trainer.py:803] 2025-04-26 19:37:37,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1902
[WARNING|trainer.py:803] 2025-04-26 19:37:37,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:38,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1794
1903
1777
[WARNING|trainer.py:803] 2025-04-26 19:37:39,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:40,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:40,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1904
1795
1778
[WARNING|trainer.py:803] 2025-04-26 19:37:42,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:42,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:42,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1905
[WARNING|trainer.py:803] 2025-04-26 19:37:44,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1779
1796
[WARNING|trainer.py:803] 2025-04-26 19:37:45,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:45,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1906
[WARNING|trainer.py:803] 2025-04-26 19:37:46,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1780
1797
1907
[WARNING|trainer.py:803] 2025-04-26 19:37:47,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:47,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:37:48,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1908
1781
1798
[WARNING|trainer.py:803] 2025-04-26 19:37:50,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:50,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:50,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1909
[WARNING|trainer.py:803] 2025-04-26 19:37:51,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1799
1782
1910
[WARNING|trainer.py:803] 2025-04-26 19:37:53,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:53,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:37:53,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1911
1800
1783
[WARNING|trainer.py:803] 2025-04-26 19:37:55,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:55,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:37:55,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1912
1801
[WARNING|trainer.py:803] 2025-04-26 19:37:57,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1784
[WARNING|trainer.py:803] 2025-04-26 19:37:57,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:37:58,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1913
1802
[WARNING|trainer.py:803] 2025-04-26 19:37:59,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:00,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1785
1914
[WARNING|trainer.py:803] 2025-04-26 19:38:01,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:01,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1803
[WARNING|trainer.py:803] 2025-04-26 19:38:02,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1915
[WARNING|trainer.py:803] 2025-04-26 19:38:03,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1786
1804
[WARNING|trainer.py:803] 2025-04-26 19:38:04,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1916
[WARNING|trainer.py:803] 2025-04-26 19:38:05,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:05,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1805
1787
1917
[WARNING|trainer.py:803] 2025-04-26 19:38:07,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:07,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:07,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1806
1918
1788
[WARNING|trainer.py:803] 2025-04-26 19:38:09,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:09,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:09,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1807
1919
[WARNING|trainer.py:803] 2025-04-26 19:38:11,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1789
[WARNING|trainer.py:803] 2025-04-26 19:38:11,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:12,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1808
1920
[WARNING|trainer.py:803] 2025-04-26 19:38:13,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:13,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1790
1921
1809
[WARNING|trainer.py:803] 2025-04-26 19:38:15,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:15,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:16,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1791
1810
1922
[WARNING|trainer.py:803] 2025-04-26 19:38:17,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:18,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:18,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1923
1811
1792
[WARNING|trainer.py:803] 2025-04-26 19:38:20,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:20,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:20,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1812
1924
1793
[WARNING|trainer.py:803] 2025-04-26 19:38:22,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:22,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:23,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1813
1925
[WARNING|trainer.py:803] 2025-04-26 19:38:24,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:24,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1794
1814
[WARNING|trainer.py:803] 2025-04-26 19:38:25,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1926
[WARNING|trainer.py:803] 2025-04-26 19:38:26,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:26,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1795
1815
1927
[WARNING|trainer.py:803] 2025-04-26 19:38:28,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:28,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:28,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1928
1796
1816
[WARNING|trainer.py:803] 2025-04-26 19:38:30,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:30,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:30,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1929
1817
[WARNING|trainer.py:803] 2025-04-26 19:38:32,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1797
[WARNING|trainer.py:803] 2025-04-26 19:38:32,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:33,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1930
[WARNING|trainer.py:803] 2025-04-26 19:38:34,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1818
1798
[WARNING|trainer.py:803] 2025-04-26 19:38:35,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1931
[WARNING|trainer.py:803] 2025-04-26 19:38:35,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:36,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1819
1932
[WARNING|trainer.py:803] 2025-04-26 19:38:37,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1799
[WARNING|trainer.py:803] 2025-04-26 19:38:38,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:38,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1820
1933
[WARNING|trainer.py:803] 2025-04-26 19:38:39,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:40,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1800
1821
1934
[WARNING|trainer.py:803] 2025-04-26 19:38:41,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:41,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:41,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1801
1822
1935
[WARNING|trainer.py:803] 2025-04-26 19:38:43,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:43,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:44,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1802
1936
[WARNING|trainer.py:803] 2025-04-26 19:38:45,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1823
[WARNING|trainer.py:803] 2025-04-26 19:38:45,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:46,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1937
1803
[WARNING|trainer.py:803] 2025-04-26 19:38:47,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:47,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1824
1938
[WARNING|trainer.py:803] 2025-04-26 19:38:49,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:49,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1804
1825
[WARNING|trainer.py:803] 2025-04-26 19:38:50,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1939
[WARNING|trainer.py:803] 2025-04-26 19:38:51,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:51,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1805
1940
1826
[WARNING|trainer.py:803] 2025-04-26 19:38:52,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:38:53,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:53,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1806
1941
1827
[WARNING|trainer.py:803] 2025-04-26 19:38:54,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:55,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:55,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1807
1942
[WARNING|trainer.py:803] 2025-04-26 19:38:56,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1828
[WARNING|trainer.py:803] 2025-04-26 19:38:57,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:38:57,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1808
1943
1829
[WARNING|trainer.py:803] 2025-04-26 19:38:59,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:38:59,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:38:59,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1944
1809
[WARNING|trainer.py:803] 2025-04-26 19:39:01,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:01,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1830
[WARNING|trainer.py:803] 2025-04-26 19:39:02,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1810
1945
[WARNING|trainer.py:803] 2025-04-26 19:39:03,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:03,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1831
1946
1811
[WARNING|trainer.py:803] 2025-04-26 19:39:04,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:05,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:05,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1832
1947
1812
[WARNING|trainer.py:803] 2025-04-26 19:39:06,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:07,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:07,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1833
1948
1813
[WARNING|trainer.py:803] 2025-04-26 19:39:09,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:09,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:09,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1949
1834
1814
[WARNING|trainer.py:803] 2025-04-26 19:39:11,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:11,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:11,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1950
1835
1815
[WARNING|trainer.py:803] 2025-04-26 19:39:13,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:13,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:13,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1951
1836
[WARNING|trainer.py:803] 2025-04-26 19:39:15,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1816
[WARNING|trainer.py:803] 2025-04-26 19:39:15,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:16,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1952
1837
1817
[WARNING|trainer.py:803] 2025-04-26 19:39:17,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:17,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:18,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1953
1838
1818
[WARNING|trainer.py:803] 2025-04-26 19:39:19,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:39:20,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:20,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1954
1839
[WARNING|trainer.py:803] 2025-04-26 19:39:21,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1819
[WARNING|trainer.py:803] 2025-04-26 19:39:22,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:22,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1955
1840
[WARNING|trainer.py:803] 2025-04-26 19:39:23,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1820
[WARNING|trainer.py:803] 2025-04-26 19:39:24,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:24,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1956
1841
[WARNING|trainer.py:803] 2025-04-26 19:39:25,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1821
[WARNING|trainer.py:803] 2025-04-26 19:39:26,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1957
[WARNING|trainer.py:803] 2025-04-26 19:39:26,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:27,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1842
1822
[WARNING|trainer.py:803] 2025-04-26 19:39:28,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:29,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1958
1843
[WARNING|trainer.py:803] 2025-04-26 19:39:30,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1823
[WARNING|trainer.py:803] 2025-04-26 19:39:31,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1959
[WARNING|trainer.py:803] 2025-04-26 19:39:31,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:39:32,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1844
1960
1824
[WARNING|trainer.py:803] 2025-04-26 19:39:33,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:39:34,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:34,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1845
1961
1825
[WARNING|trainer.py:803] 2025-04-26 19:39:35,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:36,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:36,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1962
1846
1826
[WARNING|trainer.py:803] 2025-04-26 19:39:38,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:38,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:38,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1963
1847
1827
[WARNING|trainer.py:803] 2025-04-26 19:39:40,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:39:40,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:39:40,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1964
1848
1828
[WARNING|trainer.py:803] 2025-04-26 19:39:42,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:42,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:42,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1965
1849
1829
[WARNING|trainer.py:803] 2025-04-26 19:39:44,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:44,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:44,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1966
1850
[WARNING|trainer.py:803] 2025-04-26 19:39:46,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1830
[WARNING|trainer.py:803] 2025-04-26 19:39:46,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:47,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1967
[WARNING|trainer.py:803] 2025-04-26 19:39:48,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1851
1831
[WARNING|trainer.py:803] 2025-04-26 19:39:49,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1968
[WARNING|trainer.py:803] 2025-04-26 19:39:49,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:50,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1832
1852
[WARNING|trainer.py:803] 2025-04-26 19:39:51,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:51,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1969
1833
1853
[WARNING|trainer.py:803] 2025-04-26 19:39:53,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:53,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1970
[WARNING|trainer.py:803] 2025-04-26 19:39:54,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:54,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1834
1854
1971
[WARNING|trainer.py:803] 2025-04-26 19:39:56,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:56,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:39:56,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1972
1835
1855
[WARNING|trainer.py:803] 2025-04-26 19:39:58,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:58,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:39:58,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1973
1836
1856
[WARNING|trainer.py:803] 2025-04-26 19:40:00,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:00,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:00,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1974
1837
[WARNING|trainer.py:803] 2025-04-26 19:40:02,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1857
[WARNING|trainer.py:803] 2025-04-26 19:40:02,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1975
[WARNING|trainer.py:803] 2025-04-26 19:40:02,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:03,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1838
1858
[WARNING|trainer.py:803] 2025-04-26 19:40:04,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1976
[WARNING|trainer.py:803] 2025-04-26 19:40:05,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:05,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1839
1859
1977
[WARNING|trainer.py:803] 2025-04-26 19:40:06,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:07,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:07,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1840
1978
1860
[WARNING|trainer.py:803] 2025-04-26 19:40:08,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:09,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:09,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1841
1861
1979
[WARNING|trainer.py:803] 2025-04-26 19:40:10,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:11,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:11,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1842
1980
1862
[WARNING|trainer.py:803] 2025-04-26 19:40:13,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:13,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:13,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1843
1981
1863
[WARNING|trainer.py:803] 2025-04-26 19:40:15,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:15,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:15,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1982
1864
1844
[WARNING|trainer.py:803] 2025-04-26 19:40:17,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:18,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:18,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
1983
1865
1845
[WARNING|trainer.py:803] 2025-04-26 19:40:19,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:40:20,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:20,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1984
1866
[WARNING|trainer.py:803] 2025-04-26 19:40:21,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1846
[WARNING|trainer.py:803] 2025-04-26 19:40:22,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:23,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1985
1867
[WARNING|trainer.py:803] 2025-04-26 19:40:24,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1847
[WARNING|trainer.py:803] 2025-04-26 19:40:24,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:25,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1986
1868
[WARNING|trainer.py:803] 2025-04-26 19:40:26,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1848
[WARNING|trainer.py:803] 2025-04-26 19:40:26,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1987
[WARNING|trainer.py:803] 2025-04-26 19:40:27,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1869
[WARNING|trainer.py:803] 2025-04-26 19:40:28,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1849
[WARNING|trainer.py:803] 2025-04-26 19:40:28,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:29,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1988
1870
[WARNING|trainer.py:803] 2025-04-26 19:40:30,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1850
[WARNING|trainer.py:803] 2025-04-26 19:40:31,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:31,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1989
1871
[WARNING|trainer.py:803] 2025-04-26 19:40:32,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:33,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1851
1990
[WARNING|trainer.py:803] 2025-04-26 19:40:34,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:34,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1872
1991
[WARNING|trainer.py:803] 2025-04-26 19:40:35,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1852
[WARNING|trainer.py:803] 2025-04-26 19:40:36,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:40:36,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1873
[WARNING|trainer.py:803] 2025-04-26 19:40:37,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1853
1992
[WARNING|trainer.py:803] 2025-04-26 19:40:38,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:38,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1874
[WARNING|trainer.py:803] 2025-04-26 19:40:39,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1993
1854
[WARNING|trainer.py:803] 2025-04-26 19:40:40,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:40,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1875
1994
[WARNING|trainer.py:803] 2025-04-26 19:40:41,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1855
[WARNING|trainer.py:803] 2025-04-26 19:40:42,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1876
[WARNING|trainer.py:803] 2025-04-26 19:40:43,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:43,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1995
1856
[WARNING|trainer.py:803] 2025-04-26 19:40:44,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1877
[WARNING|trainer.py:803] 2025-04-26 19:40:45,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:46,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1996
1857
[WARNING|trainer.py:803] 2025-04-26 19:40:47,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1878
[WARNING|trainer.py:803] 2025-04-26 19:40:47,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:40:48,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1858
1997
1879
[WARNING|trainer.py:803] 2025-04-26 19:40:49,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:49,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:50,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1998
1859
1880
[WARNING|trainer.py:803] 2025-04-26 19:40:51,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:51,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:52,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1999
1860
1881
[WARNING|trainer.py:803] 2025-04-26 19:40:53,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:54,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:54,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2000
1861
[WARNING|trainer.py:803] 2025-04-26 19:40:55,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1882
[WARNING|trainer.py:803] 2025-04-26 19:40:56,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:40:56,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2001
1862
[WARNING|trainer.py:803] 2025-04-26 19:40:57,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:40:58,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1883
2002
[WARNING|trainer.py:803] 2025-04-26 19:40:59,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1863
[WARNING|trainer.py:803] 2025-04-26 19:40:59,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:00,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1884
2003
1864
[WARNING|trainer.py:803] 2025-04-26 19:41:01,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:01,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:02,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2004
1885
[WARNING|trainer.py:803] 2025-04-26 19:41:03,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1865
[WARNING|trainer.py:803] 2025-04-26 19:41:04,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:04,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2005
1886
[WARNING|trainer.py:803] 2025-04-26 19:41:05,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1866
[WARNING|trainer.py:803] 2025-04-26 19:41:06,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:06,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2006
[WARNING|trainer.py:803] 2025-04-26 19:41:07,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1867
1887
2007
[WARNING|trainer.py:803] 2025-04-26 19:41:08,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:09,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:09,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1868
1888
2008
[WARNING|trainer.py:803] 2025-04-26 19:41:11,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:11,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:11,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1889
1869
2009
[WARNING|trainer.py:803] 2025-04-26 19:41:13,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:13,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:13,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1890
2010
1870
[WARNING|trainer.py:803] 2025-04-26 19:41:15,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:15,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:15,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2011
1891
1871
[WARNING|trainer.py:803] 2025-04-26 19:41:17,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:17,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:41:17,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2012
[WARNING|trainer.py:803] 2025-04-26 19:41:19,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1872
1892
[WARNING|trainer.py:803] 2025-04-26 19:41:19,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2013
[WARNING|trainer.py:803] 2025-04-26 19:41:20,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:20,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1873
1893
[WARNING|trainer.py:803] 2025-04-26 19:41:22,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:22,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2014
[WARNING|trainer.py:803] 2025-04-26 19:41:23,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1874
1894
[WARNING|trainer.py:803] 2025-04-26 19:41:24,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2015
[WARNING|trainer.py:803] 2025-04-26 19:41:24,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:25,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1875
1895
2016
[WARNING|trainer.py:803] 2025-04-26 19:41:26,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:41:26,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:26,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1876
1896
2017
[WARNING|trainer.py:803] 2025-04-26 19:41:28,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:28,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:28,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1877
1897
2018
[WARNING|trainer.py:803] 2025-04-26 19:41:30,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:30,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:31,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2019
1878
[WARNING|trainer.py:803] 2025-04-26 19:41:32,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:32,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1898
[WARNING|trainer.py:803] 2025-04-26 19:41:33,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2020
1879
[WARNING|trainer.py:803] 2025-04-26 19:41:34,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:34,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1899
2021
[WARNING|trainer.py:803] 2025-04-26 19:41:36,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1880
[WARNING|trainer.py:803] 2025-04-26 19:41:36,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:41:37,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1900
2022
1881
[WARNING|trainer.py:803] 2025-04-26 19:41:38,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:38,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:39,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1901
2023
1882
[WARNING|trainer.py:803] 2025-04-26 19:41:40,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:41,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:41:41,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1902
2024
[WARNING|trainer.py:803] 2025-04-26 19:41:42,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:42,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1883
2025
[WARNING|trainer.py:803] 2025-04-26 19:41:43,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1903
[WARNING|trainer.py:803] 2025-04-26 19:41:44,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:45,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1884
2026
[WARNING|trainer.py:803] 2025-04-26 19:41:46,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:41:46,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1904
[WARNING|trainer.py:803] 2025-04-26 19:41:47,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2027
1885
1905
[WARNING|trainer.py:803] 2025-04-26 19:41:48,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:48,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:49,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2028
1886
[WARNING|trainer.py:803] 2025-04-26 19:41:50,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1906
[WARNING|trainer.py:803] 2025-04-26 19:41:51,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2029
[WARNING|trainer.py:803] 2025-04-26 19:41:51,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:52,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1887
1907
2030
[WARNING|trainer.py:803] 2025-04-26 19:41:53,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:41:53,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:54,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1888
1908
2031
[WARNING|trainer.py:803] 2025-04-26 19:41:55,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:56,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:56,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1889
2032
1909
[WARNING|trainer.py:803] 2025-04-26 19:41:57,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:57,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:41:58,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2033
1890
1910
[WARNING|trainer.py:803] 2025-04-26 19:41:59,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:41:59,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:00,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2034
1891
1911
[WARNING|trainer.py:803] 2025-04-26 19:42:01,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:02,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:42:02,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2035
[WARNING|trainer.py:803] 2025-04-26 19:42:03,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1912
1892
[WARNING|trainer.py:803] 2025-04-26 19:42:04,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2036
[WARNING|trainer.py:803] 2025-04-26 19:42:04,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:05,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1913
1893
2037
[WARNING|trainer.py:803] 2025-04-26 19:42:06,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:06,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:07,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1894
1914
2038
[WARNING|trainer.py:803] 2025-04-26 19:42:08,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:09,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:09,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1915
2039
1895
[WARNING|trainer.py:803] 2025-04-26 19:42:11,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:11,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 19:42:11,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
:Yes
2040
1896
1916
[WARNING|trainer.py:803] 2025-04-26 19:42:13,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:13,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:13,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2041
1897
1917
[WARNING|trainer.py:803] 2025-04-26 19:42:15,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:15,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:15,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2042
[WARNING|trainer.py:803] 2025-04-26 19:42:17,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1918
1898
2043
[WARNING|trainer.py:803] 2025-04-26 19:42:18,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:18,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:19,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1919
1899
[WARNING|trainer.py:803] 2025-04-26 19:42:20,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2044
[WARNING|trainer.py:803] 2025-04-26 19:42:20,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:21,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1920
1900
2045
[WARNING|trainer.py:803] 2025-04-26 19:42:22,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:22,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:23,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1921
1901
2046
[WARNING|trainer.py:803] 2025-04-26 19:42:25,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:25,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:25,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2047
1902
1922
[WARNING|trainer.py:803] 2025-04-26 19:42:27,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:27,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:27,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2048
1903
1923
[WARNING|trainer.py:803] 2025-04-26 19:42:28,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:29,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2049
[WARNING|trainer.py:803] 2025-04-26 19:42:29,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:30,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1904
1924
2050
[WARNING|trainer.py:803] 2025-04-26 19:42:32,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:32,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:42:32,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1905
2051
1925
[WARNING|trainer.py:803] 2025-04-26 19:42:34,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:34,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:34,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2052
1906
1926
[WARNING|trainer.py:803] 2025-04-26 19:42:36,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:36,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:36,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2053
1907
[WARNING|trainer.py:803] 2025-04-26 19:42:38,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1927
[WARNING|trainer.py:803] 2025-04-26 19:42:38,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:39,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2054
1908
[WARNING|trainer.py:803] 2025-04-26 19:42:40,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1928
[WARNING|trainer.py:803] 2025-04-26 19:42:40,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:41,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2055
1909
[WARNING|trainer.py:803] 2025-04-26 19:42:42,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:42:42,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1929
2056
[WARNING|trainer.py:803] 2025-04-26 19:42:43,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:44,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1910
1930
[WARNING|trainer.py:803] 2025-04-26 19:42:44,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2057
[WARNING|trainer.py:803] 2025-04-26 19:42:45,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:45,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1911
[WARNING|trainer.py:803] 2025-04-26 19:42:47,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1931
2058
[WARNING|trainer.py:803] 2025-04-26 19:42:47,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:47,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1912
2059
[WARNING|trainer.py:803] 2025-04-26 19:42:49,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1932
[WARNING|trainer.py:803] 2025-04-26 19:42:49,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:50,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1913
2060
1933
[WARNING|trainer.py:803] 2025-04-26 19:42:51,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:51,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:52,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2061
1914
[WARNING|trainer.py:803] 2025-04-26 19:42:53,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1934
[WARNING|trainer.py:803] 2025-04-26 19:42:53,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2062
[WARNING|trainer.py:803] 2025-04-26 19:42:54,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:55,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1915
1935
[WARNING|trainer.py:803] 2025-04-26 19:42:55,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2063
[WARNING|trainer.py:803] 2025-04-26 19:42:56,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:42:57,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1916
1936
2064
[WARNING|trainer.py:803] 2025-04-26 19:42:58,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:42:58,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:42:58,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1917
1937
2065
[WARNING|trainer.py:803] 2025-04-26 19:43:00,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:00,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:00,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2066
1918
1938
[WARNING|trainer.py:803] 2025-04-26 19:43:02,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:03,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:03,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2067
1939
1919
[WARNING|trainer.py:803] 2025-04-26 19:43:04,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:05,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:05,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2068
[WARNING|trainer.py:803] 2025-04-26 19:43:06,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1940
1920
[WARNING|trainer.py:803] 2025-04-26 19:43:07,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2069
[WARNING|trainer.py:803] 2025-04-26 19:43:07,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:08,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1941
1921
2070
[WARNING|trainer.py:803] 2025-04-26 19:43:09,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:09,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:09,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2071
1942
1922
[WARNING|trainer.py:803] 2025-04-26 19:43:11,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:11,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:12,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2072
1943
1923
[WARNING|trainer.py:803] 2025-04-26 19:43:14,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:14,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:14,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2073
1944
1924
[WARNING|trainer.py:803] 2025-04-26 19:43:16,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:16,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:17,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2074
1945
[WARNING|trainer.py:803] 2025-04-26 19:43:18,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1925
[WARNING|trainer.py:803] 2025-04-26 19:43:18,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2075
[WARNING|trainer.py:803] 2025-04-26 19:43:19,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1946
[WARNING|trainer.py:803] 2025-04-26 19:43:20,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:20,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1926
2076
[WARNING|trainer.py:803] 2025-04-26 19:43:21,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1947
[WARNING|trainer.py:803] 2025-04-26 19:43:22,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:22,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1927
2077
[WARNING|trainer.py:803] 2025-04-26 19:43:24,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:24,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1948
2078
1928
[WARNING|trainer.py:803] 2025-04-26 19:43:25,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:25,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:26,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1949
2079
[WARNING|trainer.py:803] 2025-04-26 19:43:27,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1929
[WARNING|trainer.py:803] 2025-04-26 19:43:27,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:28,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2080
1950
1930
[WARNING|trainer.py:803] 2025-04-26 19:43:29,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:30,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:30,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2081
1951
1931
[WARNING|trainer.py:803] 2025-04-26 19:43:31,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:32,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:32,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2082
1952
[WARNING|trainer.py:803] 2025-04-26 19:43:33,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1932
[WARNING|trainer.py:803] 2025-04-26 19:43:34,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:34,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2083
[WARNING|trainer.py:803] 2025-04-26 19:43:36,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1933
1953
[WARNING|trainer.py:803] 2025-04-26 19:43:36,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:37,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2084
[WARNING|trainer.py:803] 2025-04-26 19:43:37,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1954
1934
[WARNING|trainer.py:803] 2025-04-26 19:43:39,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:39,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2085
[WARNING|trainer.py:803] 2025-04-26 19:43:40,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1955
1935
[WARNING|trainer.py:803] 2025-04-26 19:43:41,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:43:41,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2086
[WARNING|trainer.py:803] 2025-04-26 19:43:42,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1936
1956
[WARNING|trainer.py:803] 2025-04-26 19:43:43,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2087
[WARNING|trainer.py:803] 2025-04-26 19:43:43,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:44,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1937
1957
2088
[WARNING|trainer.py:803] 2025-04-26 19:43:45,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:43:45,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:46,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1938
2089
[WARNING|trainer.py:803] 2025-04-26 19:43:47,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:43:48,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1958
[WARNING|trainer.py:803] 2025-04-26 19:43:49,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1939
2090
[WARNING|trainer.py:803] 2025-04-26 19:43:50,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1959
[WARNING|trainer.py:803] 2025-04-26 19:43:50,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:43:50,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1940
2091
[WARNING|trainer.py:803] 2025-04-26 19:43:52,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1960
[WARNING|trainer.py:803] 2025-04-26 19:43:52,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:43:53,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1941
2092
[WARNING|trainer.py:803] 2025-04-26 19:43:54,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1961
[WARNING|trainer.py:803] 2025-04-26 19:43:54,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:43:55,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2093
1942
[WARNING|trainer.py:803] 2025-04-26 19:43:56,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:43:56,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1962
2094
[WARNING|trainer.py:803] 2025-04-26 19:43:57,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:43:58,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1943
[WARNING|trainer.py:803] 2025-04-26 19:43:58,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2095
1963
[WARNING|trainer.py:803] 2025-04-26 19:43:59,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:44:00,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1944
2096
[WARNING|trainer.py:803] 2025-04-26 19:44:01,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1964
[WARNING|trainer.py:803] 2025-04-26 19:44:01,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:02,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1945
2097
[WARNING|trainer.py:803] 2025-04-26 19:44:03,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:03,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1965
1946
[WARNING|trainer.py:803] 2025-04-26 19:44:04,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2098
[WARNING|trainer.py:803] 2025-04-26 19:44:05,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:05,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1966
[WARNING|trainer.py:803] 2025-04-26 19:44:06,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1947
2099
[WARNING|trainer.py:803] 2025-04-26 19:44:07,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:07,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1967
2100
1948
[WARNING|trainer.py:803] 2025-04-26 19:44:09,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:09,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2101
[WARNING|trainer.py:803] 2025-04-26 19:44:10,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:10,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1968
2102
1949
[WARNING|trainer.py:803] 2025-04-26 19:44:11,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:12,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:12,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2103
[WARNING|trainer.py:803] 2025-04-26 19:44:13,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2104
1969
1950
[WARNING|trainer.py:803] 2025-04-26 19:44:14,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2105
[WARNING|trainer.py:803] 2025-04-26 19:44:14,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:14,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:15,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1970
2106
1951
[WARNING|trainer.py:803] 2025-04-26 19:44:16,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:16,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2107
[WARNING|trainer.py:803] 2025-04-26 19:44:16,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1971
[WARNING|trainer.py:803] 2025-04-26 19:44:17,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2108
1952
[WARNING|trainer.py:803] 2025-04-26 19:44:18,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:18,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2109
[WARNING|trainer.py:803] 2025-04-26 19:44:19,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1972
[WARNING|trainer.py:803] 2025-04-26 19:44:19,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2110
[WARNING|trainer.py:803] 2025-04-26 19:44:20,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1953
[WARNING|trainer.py:803] 2025-04-26 19:44:20,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2111
[WARNING|trainer.py:803] 2025-04-26 19:44:21,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1973
[WARNING|trainer.py:803] 2025-04-26 19:44:22,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2112
[WARNING|trainer.py:803] 2025-04-26 19:44:22,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1954
[WARNING|trainer.py:803] 2025-04-26 19:44:23,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2113
[WARNING|trainer.py:803] 2025-04-26 19:44:23,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1974
[WARNING|trainer.py:803] 2025-04-26 19:44:24,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2114
[WARNING|trainer.py:803] 2025-04-26 19:44:24,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
1955
[WARNING|trainer.py:803] 2025-04-26 19:44:25,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2115
1975
[WARNING|trainer.py:803] 2025-04-26 19:44:26,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:44:26,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2116
[WARNING|trainer.py:803] 2025-04-26 19:44:26,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:27,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1956
2117
1976
[WARNING|trainer.py:803] 2025-04-26 19:44:28,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:28,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2118
[WARNING|trainer.py:803] 2025-04-26 19:44:28,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:29,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1957
2119
1977
[WARNING|trainer.py:803] 2025-04-26 19:44:30,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:30,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:30,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2120
[WARNING|trainer.py:803] 2025-04-26 19:44:31,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2121
1978
1958
[WARNING|trainer.py:803] 2025-04-26 19:44:33,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:33,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2122
[WARNING|trainer.py:803] 2025-04-26 19:44:33,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:34,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2123
1959
1979
[WARNING|trainer.py:803] 2025-04-26 19:44:35,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2124
[WARNING|trainer.py:803] 2025-04-26 19:44:35,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:35,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:36,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2125
1980
1960
[WARNING|trainer.py:803] 2025-04-26 19:44:37,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2126
[WARNING|trainer.py:803] 2025-04-26 19:44:38,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:38,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:38,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2127
1961
1981
[WARNING|trainer.py:803] 2025-04-26 19:44:39,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2128
[WARNING|trainer.py:803] 2025-04-26 19:44:40,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:40,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:40,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2129
1962
1982
[WARNING|trainer.py:803] 2025-04-26 19:44:42,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2130
[WARNING|trainer.py:803] 2025-04-26 19:44:42,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:42,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:43,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2131
1963
1983
[WARNING|trainer.py:803] 2025-04-26 19:44:44,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2132
[WARNING|trainer.py:803] 2025-04-26 19:44:44,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:44:44,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:44:45,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2133
1984
1964
[WARNING|trainer.py:803] 2025-04-26 19:44:46,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2134
[WARNING|trainer.py:803] 2025-04-26 19:44:47,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:47,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:47,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2135
1965
[WARNING|trainer.py:803] 2025-04-26 19:44:48,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1985
2136
[WARNING|trainer.py:803] 2025-04-26 19:44:49,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:49,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:49,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2137
1966
[WARNING|trainer.py:803] 2025-04-26 19:44:50,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1986
2138
[WARNING|trainer.py:803] 2025-04-26 19:44:51,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:51,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:51,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2139
[WARNING|trainer.py:803] 2025-04-26 19:44:53,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1967
1987
2140
[WARNING|trainer.py:803] 2025-04-26 19:44:54,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:54,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:54,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2141
[WARNING|trainer.py:803] 2025-04-26 19:44:55,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1968
2142
1988
[WARNING|trainer.py:803] 2025-04-26 19:44:56,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:56,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:44:56,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2143
[WARNING|trainer.py:803] 2025-04-26 19:44:57,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2144
1989
1969
[WARNING|trainer.py:803] 2025-04-26 19:44:58,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:44:59,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2145
[WARNING|trainer.py:803] 2025-04-26 19:44:59,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:44:59,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2146
1970
1990
[WARNING|trainer.py:803] 2025-04-26 19:45:00,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:45:01,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2147
[WARNING|trainer.py:803] 2025-04-26 19:45:01,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:02,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1971
2148
1991
[WARNING|trainer.py:803] 2025-04-26 19:45:03,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:03,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:45:03,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2149
1972
[WARNING|trainer.py:803] 2025-04-26 19:45:04,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2150
[WARNING|trainer.py:803] 2025-04-26 19:45:05,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1992
[WARNING|trainer.py:803] 2025-04-26 19:45:05,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2151
[WARNING|trainer.py:803] 2025-04-26 19:45:05,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1973
[WARNING|trainer.py:803] 2025-04-26 19:45:06,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2152
[WARNING|trainer.py:803] 2025-04-26 19:45:07,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1993
[WARNING|trainer.py:803] 2025-04-26 19:45:07,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2153
[WARNING|trainer.py:803] 2025-04-26 19:45:08,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1974
[WARNING|trainer.py:803] 2025-04-26 19:45:08,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2154
[WARNING|trainer.py:803] 2025-04-26 19:45:09,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:09,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1994
2155
1975
[WARNING|trainer.py:803] 2025-04-26 19:45:10,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:10,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2156
[WARNING|trainer.py:803] 2025-04-26 19:45:11,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:11,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1995
2157
1976
[WARNING|trainer.py:803] 2025-04-26 19:45:12,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:13,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2158
[WARNING|trainer.py:803] 2025-04-26 19:45:13,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:14,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2159
1996
1977
[WARNING|trainer.py:803] 2025-04-26 19:45:15,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:15,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2160
[WARNING|trainer.py:803] 2025-04-26 19:45:15,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:16,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2161
1978
1997
[WARNING|trainer.py:803] 2025-04-26 19:45:17,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2162
[WARNING|trainer.py:803] 2025-04-26 19:45:17,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:18,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:18,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2163
1979
1998
[WARNING|trainer.py:803] 2025-04-26 19:45:19,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2164
[WARNING|trainer.py:803] 2025-04-26 19:45:20,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:20,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:20,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2165
1999
1980
[WARNING|trainer.py:803] 2025-04-26 19:45:21,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2166
[WARNING|trainer.py:803] 2025-04-26 19:45:22,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:22,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:22,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2167
2000
[WARNING|trainer.py:803] 2025-04-26 19:45:24,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1981
2168
[WARNING|trainer.py:803] 2025-04-26 19:45:24,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:25,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:25,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2169
1982
[WARNING|trainer.py:803] 2025-04-26 19:45:26,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2001
2170
[WARNING|trainer.py:803] 2025-04-26 19:45:27,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:27,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:27,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2171
[WARNING|trainer.py:803] 2025-04-26 19:45:28,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2002
1983
2172
[WARNING|trainer.py:803] 2025-04-26 19:45:29,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:29,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:45:29,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2173
[WARNING|trainer.py:803] 2025-04-26 19:45:30,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
1984
2003
2174
[WARNING|trainer.py:803] 2025-04-26 19:45:31,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:31,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:31,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2175
[WARNING|trainer.py:803] 2025-04-26 19:45:33,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2176
2004
1985
[WARNING|trainer.py:803] 2025-04-26 19:45:34,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:45:34,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:34,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2177
[WARNING|trainer.py:803] 2025-04-26 19:45:35,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2178
1986
2005
[WARNING|trainer.py:803] 2025-04-26 19:45:36,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:36,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:36,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2179
[WARNING|trainer.py:803] 2025-04-26 19:45:37,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2006
2180
1987
[WARNING|trainer.py:803] 2025-04-26 19:45:38,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:38,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:38,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2181
[WARNING|trainer.py:803] 2025-04-26 19:45:39,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2007
2182
1988
[WARNING|trainer.py:803] 2025-04-26 19:45:40,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:45:40,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2183
[WARNING|trainer.py:803] 2025-04-26 19:45:41,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:42,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2008
2184
1989
[WARNING|trainer.py:803] 2025-04-26 19:45:42,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:43,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2185
[WARNING|trainer.py:803] 2025-04-26 19:45:43,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2009
[WARNING|trainer.py:803] 2025-04-26 19:45:44,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2186
[WARNING|trainer.py:803] 2025-04-26 19:45:45,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1990
[WARNING|trainer.py:803] 2025-04-26 19:45:45,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2187
[WARNING|trainer.py:803] 2025-04-26 19:45:46,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2010
[WARNING|trainer.py:803] 2025-04-26 19:45:46,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2188
[WARNING|trainer.py:803] 2025-04-26 19:45:47,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
1991
[WARNING|trainer.py:803] 2025-04-26 19:45:47,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2189
[WARNING|trainer.py:803] 2025-04-26 19:45:48,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2011
[WARNING|trainer.py:803] 2025-04-26 19:45:48,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2190
[WARNING|trainer.py:803] 2025-04-26 19:45:49,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
1992
[WARNING|trainer.py:803] 2025-04-26 19:45:49,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2191
[WARNING|trainer.py:803] 2025-04-26 19:45:50,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2012
[WARNING|trainer.py:803] 2025-04-26 19:45:50,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2192
[WARNING|trainer.py:803] 2025-04-26 19:45:51,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:52,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1993
2193
2013
[WARNING|trainer.py:803] 2025-04-26 19:45:52,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:53,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2194
[WARNING|trainer.py:803] 2025-04-26 19:45:53,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:45:54,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
1994
2195
2014
[WARNING|trainer.py:803] 2025-04-26 19:45:55,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:55,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2196
[WARNING|trainer.py:803] 2025-04-26 19:45:56,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:56,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2197
1995
2015
[WARNING|trainer.py:803] 2025-04-26 19:45:57,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:45:57,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2198
[WARNING|trainer.py:803] 2025-04-26 19:45:58,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:45:58,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2199
1996
2016
[WARNING|trainer.py:803] 2025-04-26 19:45:59,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2200
[WARNING|trainer.py:803] 2025-04-26 19:46:00,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:00,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:00,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2201
2017
1997
[WARNING|trainer.py:803] 2025-04-26 19:46:02,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2202
[WARNING|trainer.py:803] 2025-04-26 19:46:02,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:02,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:03,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2203
2018
1998
[WARNING|trainer.py:803] 2025-04-26 19:46:04,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2204
[WARNING|trainer.py:803] 2025-04-26 19:46:05,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:05,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:05,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2205
2019
1999
[WARNING|trainer.py:803] 2025-04-26 19:46:06,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:07,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2206
[WARNING|trainer.py:803] 2025-04-26 19:46:07,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:07,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2207
2020
2000
[WARNING|trainer.py:803] 2025-04-26 19:46:09,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2208
[WARNING|trainer.py:803] 2025-04-26 19:46:09,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:09,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:10,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2209
2021
2001
[WARNING|trainer.py:803] 2025-04-26 19:46:11,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:11,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2210
[WARNING|trainer.py:803] 2025-04-26 19:46:11,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:12,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2211
2022
2002
[WARNING|trainer.py:803] 2025-04-26 19:46:13,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:13,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2212
[WARNING|trainer.py:803] 2025-04-26 19:46:14,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:14,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2213
2023
2003
[WARNING|trainer.py:803] 2025-04-26 19:46:15,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2214
[WARNING|trainer.py:803] 2025-04-26 19:46:16,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:16,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:16,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2215
2024
2004
[WARNING|trainer.py:803] 2025-04-26 19:46:18,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2216
[WARNING|trainer.py:803] 2025-04-26 19:46:18,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:18,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:19,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2217
2025
2005
[WARNING|trainer.py:803] 2025-04-26 19:46:20,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:20,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2218
[WARNING|trainer.py:803] 2025-04-26 19:46:21,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:21,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2026
2219
2006
[WARNING|trainer.py:803] 2025-04-26 19:46:22,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:22,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2220
[WARNING|trainer.py:803] 2025-04-26 19:46:23,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:23,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2027
2221
2007
[WARNING|trainer.py:803] 2025-04-26 19:46:24,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:25,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:25,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2222
2028
[WARNING|trainer.py:803] 2025-04-26 19:46:26,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2223
2008
[WARNING|trainer.py:803] 2025-04-26 19:46:27,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:27,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:27,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2224
2029
[WARNING|trainer.py:803] 2025-04-26 19:46:28,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2009
2225
[WARNING|trainer.py:803] 2025-04-26 19:46:29,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:29,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:29,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2226
2030
2010
[WARNING|trainer.py:803] 2025-04-26 19:46:30,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2227
[WARNING|trainer.py:803] 2025-04-26 19:46:31,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:31,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:31,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2228
2031
2011
[WARNING|trainer.py:803] 2025-04-26 19:46:33,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2229
[WARNING|trainer.py:803] 2025-04-26 19:46:33,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:33,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:34,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2230
2032
2012
[WARNING|trainer.py:803] 2025-04-26 19:46:35,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:46:35,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2231
[WARNING|trainer.py:803] 2025-04-26 19:46:35,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:36,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2033
2232
2013
[WARNING|trainer.py:803] 2025-04-26 19:46:37,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:37,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2233
[WARNING|trainer.py:803] 2025-04-26 19:46:37,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2034
[WARNING|trainer.py:803] 2025-04-26 19:46:38,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2234
[WARNING|trainer.py:803] 2025-04-26 19:46:39,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2014
[WARNING|trainer.py:803] 2025-04-26 19:46:39,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2235
[WARNING|trainer.py:803] 2025-04-26 19:46:40,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2035
[WARNING|trainer.py:803] 2025-04-26 19:46:41,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2236
[WARNING|trainer.py:803] 2025-04-26 19:46:41,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2015
[WARNING|trainer.py:803] 2025-04-26 19:46:42,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2237
[WARNING|trainer.py:803] 2025-04-26 19:46:42,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2036
[WARNING|trainer.py:803] 2025-04-26 19:46:43,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2238
2016
[WARNING|trainer.py:803] 2025-04-26 19:46:43,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:46:44,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:44,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2239
[WARNING|trainer.py:803] 2025-04-26 19:46:45,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2037
2240
2017
[WARNING|trainer.py:803] 2025-04-26 19:46:46,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:46,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2241
[WARNING|trainer.py:803] 2025-04-26 19:46:46,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:47,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2038
2242
2018
[WARNING|trainer.py:803] 2025-04-26 19:46:48,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:48,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2243
[WARNING|trainer.py:803] 2025-04-26 19:46:49,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2039
[WARNING|trainer.py:803] 2025-04-26 19:46:49,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2244
2019
[WARNING|trainer.py:803] 2025-04-26 19:46:50,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:50,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:46:51,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2245
2040
[WARNING|trainer.py:803] 2025-04-26 19:46:52,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2246
[WARNING|trainer.py:803] 2025-04-26 19:46:52,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2020
[WARNING|trainer.py:803] 2025-04-26 19:46:53,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2247
[WARNING|trainer.py:803] 2025-04-26 19:46:53,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2041
[WARNING|trainer.py:803] 2025-04-26 19:46:54,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2248
[WARNING|trainer.py:803] 2025-04-26 19:46:54,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2021
[WARNING|trainer.py:803] 2025-04-26 19:46:55,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2249
[WARNING|trainer.py:803] 2025-04-26 19:46:55,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2042
[WARNING|trainer.py:803] 2025-04-26 19:46:56,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2250
2022
[WARNING|trainer.py:803] 2025-04-26 19:46:57,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:46:57,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2251
[WARNING|trainer.py:803] 2025-04-26 19:46:57,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2043
[WARNING|trainer.py:803] 2025-04-26 19:46:58,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2252
[WARNING|trainer.py:803] 2025-04-26 19:46:59,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2023
[WARNING|trainer.py:803] 2025-04-26 19:46:59,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2253
[WARNING|trainer.py:803] 2025-04-26 19:47:00,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2044
[WARNING|trainer.py:803] 2025-04-26 19:47:00,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2254
[WARNING|trainer.py:803] 2025-04-26 19:47:01,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2024
[WARNING|trainer.py:803] 2025-04-26 19:47:01,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2255
[WARNING|trainer.py:803] 2025-04-26 19:47:02,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2045
[WARNING|trainer.py:803] 2025-04-26 19:47:03,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2256
[WARNING|trainer.py:803] 2025-04-26 19:47:03,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2025
[WARNING|trainer.py:803] 2025-04-26 19:47:04,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2257
[WARNING|trainer.py:803] 2025-04-26 19:47:04,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:05,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2046
2258
[WARNING|trainer.py:803] 2025-04-26 19:47:05,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2026
[WARNING|trainer.py:803] 2025-04-26 19:47:06,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2259
[WARNING|trainer.py:803] 2025-04-26 19:47:06,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:07,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2047
2260
[WARNING|trainer.py:803] 2025-04-26 19:47:08,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:08,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2027
2261
[WARNING|trainer.py:803] 2025-04-26 19:47:09,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:09,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2048
2262
[WARNING|trainer.py:803] 2025-04-26 19:47:10,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:10,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2028
2263
[WARNING|trainer.py:803] 2025-04-26 19:47:11,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2049
[WARNING|trainer.py:803] 2025-04-26 19:47:11,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2264
[WARNING|trainer.py:803] 2025-04-26 19:47:12,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:47:12,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2029
2265
[WARNING|trainer.py:803] 2025-04-26 19:47:13,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:13,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2050
2266
[WARNING|trainer.py:803] 2025-04-26 19:47:14,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2030
[WARNING|trainer.py:803] 2025-04-26 19:47:14,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2267
[WARNING|trainer.py:803] 2025-04-26 19:47:15,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2051
[WARNING|trainer.py:803] 2025-04-26 19:47:15,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2268
[WARNING|trainer.py:803] 2025-04-26 19:47:16,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2031
[WARNING|trainer.py:803] 2025-04-26 19:47:16,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2269
[WARNING|trainer.py:803] 2025-04-26 19:47:17,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2052
[WARNING|trainer.py:803] 2025-04-26 19:47:17,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2270
[WARNING|trainer.py:803] 2025-04-26 19:47:18,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2032
[WARNING|trainer.py:803] 2025-04-26 19:47:18,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2271
[WARNING|trainer.py:803] 2025-04-26 19:47:19,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2053
[WARNING|trainer.py:803] 2025-04-26 19:47:20,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2272
2033
[WARNING|trainer.py:803] 2025-04-26 19:47:20,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:21,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2273
[WARNING|trainer.py:803] 2025-04-26 19:47:21,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2054
[WARNING|trainer.py:803] 2025-04-26 19:47:22,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2274
2034
[WARNING|trainer.py:803] 2025-04-26 19:47:22,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:47:23,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2275
[WARNING|trainer.py:803] 2025-04-26 19:47:23,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2055
[WARNING|trainer.py:803] 2025-04-26 19:47:24,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2276
2035
[WARNING|trainer.py:803] 2025-04-26 19:47:25,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:47:25,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2277
[WARNING|trainer.py:803] 2025-04-26 19:47:25,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2056
[WARNING|trainer.py:803] 2025-04-26 19:47:26,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2278
2036
[WARNING|trainer.py:803] 2025-04-26 19:47:27,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:27,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2279
[WARNING|trainer.py:803] 2025-04-26 19:47:27,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2057
[WARNING|trainer.py:803] 2025-04-26 19:47:28,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2280
[WARNING|trainer.py:803] 2025-04-26 19:47:29,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2037
[WARNING|trainer.py:803] 2025-04-26 19:47:29,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2281
[WARNING|trainer.py:803] 2025-04-26 19:47:30,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:47:30,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2058
2282
2038
[WARNING|trainer.py:803] 2025-04-26 19:47:31,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:31,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2283
[WARNING|trainer.py:803] 2025-04-26 19:47:32,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:32,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2059
2284
2039
[WARNING|trainer.py:803] 2025-04-26 19:47:33,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:34,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2285
[WARNING|trainer.py:803] 2025-04-26 19:47:34,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2060
[WARNING|trainer.py:803] 2025-04-26 19:47:35,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2286
2040
[WARNING|trainer.py:803] 2025-04-26 19:47:35,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:36,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2287
[WARNING|trainer.py:803] 2025-04-26 19:47:36,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2061
[WARNING|trainer.py:803] 2025-04-26 19:47:37,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2288
2041
[WARNING|trainer.py:803] 2025-04-26 19:47:37,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:38,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2289
[WARNING|trainer.py:803] 2025-04-26 19:47:38,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2062
[WARNING|trainer.py:803] 2025-04-26 19:47:39,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2290
[WARNING|trainer.py:803] 2025-04-26 19:47:40,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2042
[WARNING|trainer.py:803] 2025-04-26 19:47:40,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2291
[WARNING|trainer.py:803] 2025-04-26 19:47:41,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:41,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2063
2292
2043
[WARNING|trainer.py:803] 2025-04-26 19:47:42,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:42,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2293
[WARNING|trainer.py:803] 2025-04-26 19:47:43,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2064
[WARNING|trainer.py:803] 2025-04-26 19:47:43,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2294
[WARNING|trainer.py:803] 2025-04-26 19:47:44,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:44,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2044
2295
[WARNING|trainer.py:803] 2025-04-26 19:47:45,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:47:45,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2065
2296
2045
[WARNING|trainer.py:803] 2025-04-26 19:47:46,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:47:46,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2297
[WARNING|trainer.py:803] 2025-04-26 19:47:47,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2066
[WARNING|trainer.py:803] 2025-04-26 19:47:48,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2298
[WARNING|trainer.py:803] 2025-04-26 19:47:48,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2046
[WARNING|trainer.py:803] 2025-04-26 19:47:49,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2299
[WARNING|trainer.py:803] 2025-04-26 19:47:49,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2067
[WARNING|trainer.py:803] 2025-04-26 19:47:50,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2300
[WARNING|trainer.py:803] 2025-04-26 19:47:50,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2047
[WARNING|trainer.py:803] 2025-04-26 19:47:51,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2301
[WARNING|trainer.py:803] 2025-04-26 19:47:52,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2068
[WARNING|trainer.py:803] 2025-04-26 19:47:52,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2302
[WARNING|trainer.py:803] 2025-04-26 19:47:52,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2048
[WARNING|trainer.py:803] 2025-04-26 19:47:53,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2303
[WARNING|trainer.py:803] 2025-04-26 19:47:54,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2069
[WARNING|trainer.py:803] 2025-04-26 19:47:54,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2304
[WARNING|trainer.py:803] 2025-04-26 19:47:55,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2049
[WARNING|trainer.py:803] 2025-04-26 19:47:55,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2305
[WARNING|trainer.py:803] 2025-04-26 19:47:56,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2070
[WARNING|trainer.py:803] 2025-04-26 19:47:56,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2306
[WARNING|trainer.py:803] 2025-04-26 19:47:57,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2050
[WARNING|trainer.py:803] 2025-04-26 19:47:57,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2307
[WARNING|trainer.py:803] 2025-04-26 19:47:58,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2071
[WARNING|trainer.py:803] 2025-04-26 19:47:58,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2308
[WARNING|trainer.py:803] 2025-04-26 19:47:59,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2051
[WARNING|trainer.py:803] 2025-04-26 19:47:59,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2309
[WARNING|trainer.py:803] 2025-04-26 19:48:00,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2072
[WARNING|trainer.py:803] 2025-04-26 19:48:00,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2310
2052
[WARNING|trainer.py:803] 2025-04-26 19:48:01,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:02,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2311
[WARNING|trainer.py:803] 2025-04-26 19:48:02,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2073
[WARNING|trainer.py:803] 2025-04-26 19:48:03,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2312
2053
[WARNING|trainer.py:803] 2025-04-26 19:48:03,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:04,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2313
[WARNING|trainer.py:803] 2025-04-26 19:48:04,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:05,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2074
2314
2054
[WARNING|trainer.py:803] 2025-04-26 19:48:06,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:06,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2315
[WARNING|trainer.py:803] 2025-04-26 19:48:06,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:07,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2075
2316
2055
[WARNING|trainer.py:803] 2025-04-26 19:48:08,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:08,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:08,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2317
[WARNING|trainer.py:803] 2025-04-26 19:48:09,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2318
2076
2056
[WARNING|trainer.py:803] 2025-04-26 19:48:10,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:10,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:10,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2319
[WARNING|trainer.py:803] 2025-04-26 19:48:11,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2077
2320
2057
[WARNING|trainer.py:803] 2025-04-26 19:48:12,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:12,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:12,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2321
2078
[WARNING|trainer.py:803] 2025-04-26 19:48:14,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2322
2058
[WARNING|trainer.py:803] 2025-04-26 19:48:14,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:15,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:15,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2323
2079
[WARNING|trainer.py:803] 2025-04-26 19:48:16,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2324
2059
[WARNING|trainer.py:803] 2025-04-26 19:48:16,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:17,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:17,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2325
2080
[WARNING|trainer.py:803] 2025-04-26 19:48:18,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2326
2060
[WARNING|trainer.py:803] 2025-04-26 19:48:19,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:19,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:19,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2327
[WARNING|trainer.py:803] 2025-04-26 19:48:20,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2081
2061
2328
[WARNING|trainer.py:803] 2025-04-26 19:48:21,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:21,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:21,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2329
[WARNING|trainer.py:803] 2025-04-26 19:48:22,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2062
2330
2082
[WARNING|trainer.py:803] 2025-04-26 19:48:23,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:23,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:23,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2331
[WARNING|trainer.py:803] 2025-04-26 19:48:24,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2063
2332
2083
[WARNING|trainer.py:803] 2025-04-26 19:48:25,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:25,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:26,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2333
2064
[WARNING|trainer.py:803] 2025-04-26 19:48:26,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2334
2084
[WARNING|trainer.py:803] 2025-04-26 19:48:27,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:28,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:28,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2335
[WARNING|trainer.py:803] 2025-04-26 19:48:29,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2336
2065
2085
[WARNING|trainer.py:803] 2025-04-26 19:48:30,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:30,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2337
[WARNING|trainer.py:803] 2025-04-26 19:48:30,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:31,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2066
2338
[WARNING|trainer.py:803] 2025-04-26 19:48:32,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:32,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2086
2339
[WARNING|trainer.py:803] 2025-04-26 19:48:33,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:33,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2067
2340
[WARNING|trainer.py:803] 2025-04-26 19:48:34,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:34,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2341
2087
2068
[WARNING|trainer.py:803] 2025-04-26 19:48:35,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:35,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2342
[WARNING|trainer.py:803] 2025-04-26 19:48:36,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:36,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2343
2088
2069
[WARNING|trainer.py:803] 2025-04-26 19:48:37,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:48:37,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2344
[WARNING|trainer.py:803] 2025-04-26 19:48:38,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:38,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2089
2345
2070
[WARNING|trainer.py:803] 2025-04-26 19:48:39,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:39,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2346
[WARNING|trainer.py:803] 2025-04-26 19:48:40,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:48:40,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2347
2090
2071
[WARNING|trainer.py:803] 2025-04-26 19:48:41,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2348
[WARNING|trainer.py:803] 2025-04-26 19:48:42,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:42,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:42,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2349
2091
[WARNING|trainer.py:803] 2025-04-26 19:48:43,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2350
2072
[WARNING|trainer.py:803] 2025-04-26 19:48:44,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:44,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:48:44,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2351
2092
[WARNING|trainer.py:803] 2025-04-26 19:48:45,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2352
2073
[WARNING|trainer.py:803] 2025-04-26 19:48:46,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:48:47,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2353
[WARNING|trainer.py:803] 2025-04-26 19:48:47,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2093
[WARNING|trainer.py:803] 2025-04-26 19:48:48,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2354
[WARNING|trainer.py:803] 2025-04-26 19:48:48,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2074
[WARNING|trainer.py:803] 2025-04-26 19:48:49,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2355
[WARNING|trainer.py:803] 2025-04-26 19:48:49,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2094
[WARNING|trainer.py:803] 2025-04-26 19:48:50,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2356
[WARNING|trainer.py:803] 2025-04-26 19:48:50,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2075
[WARNING|trainer.py:803] 2025-04-26 19:48:51,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2357
[WARNING|trainer.py:803] 2025-04-26 19:48:51,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2095
[WARNING|trainer.py:803] 2025-04-26 19:48:52,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2358
[WARNING|trainer.py:803] 2025-04-26 19:48:52,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2076
[WARNING|trainer.py:803] 2025-04-26 19:48:53,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2359
[WARNING|trainer.py:803] 2025-04-26 19:48:54,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2096
[WARNING|trainer.py:803] 2025-04-26 19:48:54,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2360
[WARNING|trainer.py:803] 2025-04-26 19:48:55,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2077
[WARNING|trainer.py:803] 2025-04-26 19:48:55,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2361
[WARNING|trainer.py:803] 2025-04-26 19:48:56,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2097
[WARNING|trainer.py:803] 2025-04-26 19:48:56,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2362
[WARNING|trainer.py:803] 2025-04-26 19:48:57,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2078
[WARNING|trainer.py:803] 2025-04-26 19:48:57,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2363
[WARNING|trainer.py:803] 2025-04-26 19:48:58,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2098
[WARNING|trainer.py:803] 2025-04-26 19:48:58,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2364
[WARNING|trainer.py:803] 2025-04-26 19:48:59,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2079
[WARNING|trainer.py:803] 2025-04-26 19:48:59,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2365
[WARNING|trainer.py:803] 2025-04-26 19:49:00,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:00,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2099
2366
2080
[WARNING|trainer.py:803] 2025-04-26 19:49:01,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:01,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2367
[WARNING|trainer.py:803] 2025-04-26 19:49:02,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:02,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2100
2368
2081
[WARNING|trainer.py:803] 2025-04-26 19:49:03,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:49:04,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2369
2101
[WARNING|trainer.py:803] 2025-04-26 19:49:04,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:05,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:05,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2370
2102
[WARNING|trainer.py:803] 2025-04-26 19:49:06,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2082
2371
[WARNING|trainer.py:803] 2025-04-26 19:49:06,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2103
[WARNING|trainer.py:803] 2025-04-26 19:49:07,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:07,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2372
[WARNING|trainer.py:803] 2025-04-26 19:49:07,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2104
[WARNING|trainer.py:803] 2025-04-26 19:49:08,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2373
2083
[WARNING|trainer.py:803] 2025-04-26 19:49:09,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:09,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:09,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2105
2374
[WARNING|trainer.py:803] 2025-04-26 19:49:10,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:10,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2375
2084
2106
[WARNING|trainer.py:803] 2025-04-26 19:49:11,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:11,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:11,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2376
2107
[WARNING|trainer.py:803] 2025-04-26 19:49:12,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2377
[WARNING|trainer.py:803] 2025-04-26 19:49:13,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2085
2108
[WARNING|trainer.py:803] 2025-04-26 19:49:13,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2378
[WARNING|trainer.py:803] 2025-04-26 19:49:13,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:14,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:14,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2109
2379
[WARNING|trainer.py:803] 2025-04-26 19:49:15,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2086
[WARNING|trainer.py:803] 2025-04-26 19:49:15,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2380
2110
[WARNING|trainer.py:803] 2025-04-26 19:49:16,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:16,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:16,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2381
2111
[WARNING|trainer.py:803] 2025-04-26 19:49:17,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2382
[WARNING|trainer.py:803] 2025-04-26 19:49:18,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2087
2112
[WARNING|trainer.py:803] 2025-04-26 19:49:18,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:19,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2383
[WARNING|trainer.py:803] 2025-04-26 19:49:19,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2113
[WARNING|trainer.py:803] 2025-04-26 19:49:19,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2384
2088
[WARNING|trainer.py:803] 2025-04-26 19:49:20,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:21,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:21,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2114
2385
[WARNING|trainer.py:803] 2025-04-26 19:49:21,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:22,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2089
2386
2115
[WARNING|trainer.py:803] 2025-04-26 19:49:23,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:23,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:23,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2387
2116
[WARNING|trainer.py:803] 2025-04-26 19:49:24,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2388
[WARNING|trainer.py:803] 2025-04-26 19:49:24,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2117
2090
[WARNING|trainer.py:803] 2025-04-26 19:49:25,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2389
[WARNING|trainer.py:803] 2025-04-26 19:49:25,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:25,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:26,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2118
2390
[WARNING|trainer.py:803] 2025-04-26 19:49:27,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2091
[WARNING|trainer.py:803] 2025-04-26 19:49:27,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2391
2119
[WARNING|trainer.py:803] 2025-04-26 19:49:28,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:28,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:28,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2392
2120
[WARNING|trainer.py:803] 2025-04-26 19:49:29,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2092
[WARNING|trainer.py:803] 2025-04-26 19:49:29,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2393
2121
[WARNING|trainer.py:803] 2025-04-26 19:49:30,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:30,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2394
[WARNING|trainer.py:803] 2025-04-26 19:49:30,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2122
2093
[WARNING|trainer.py:803] 2025-04-26 19:49:31,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2395
[WARNING|trainer.py:803] 2025-04-26 19:49:32,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:32,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:32,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2123
2396
2094
[WARNING|trainer.py:803] 2025-04-26 19:49:33,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:33,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2397
2124
[WARNING|trainer.py:803] 2025-04-26 19:49:34,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:34,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:34,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2398
2125
2095
[WARNING|trainer.py:803] 2025-04-26 19:49:36,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:36,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2399
[WARNING|trainer.py:803] 2025-04-26 19:49:36,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2126
[WARNING|trainer.py:803] 2025-04-26 19:49:37,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2400
[WARNING|trainer.py:803] 2025-04-26 19:49:37,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2096
2127
[WARNING|trainer.py:803] 2025-04-26 19:49:38,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:38,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:38,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2128
2401
2097
[WARNING|trainer.py:803] 2025-04-26 19:49:40,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:40,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2129
[WARNING|trainer.py:803] 2025-04-26 19:49:40,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:41,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2402
2130
2098
[WARNING|trainer.py:803] 2025-04-26 19:49:42,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:49:42,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:43,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2131
2403
[WARNING|trainer.py:803] 2025-04-26 19:49:44,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2132
[WARNING|trainer.py:803] 2025-04-26 19:49:44,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2099
[WARNING|trainer.py:803] 2025-04-26 19:49:45,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:45,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2404
2133
[WARNING|trainer.py:803] 2025-04-26 19:49:46,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:46,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2100
2134
[WARNING|trainer.py:803] 2025-04-26 19:49:47,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2405
[WARNING|trainer.py:803] 2025-04-26 19:49:48,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2101
[WARNING|trainer.py:803] 2025-04-26 19:49:48,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2135
[WARNING|trainer.py:803] 2025-04-26 19:49:49,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2102
[WARNING|trainer.py:803] 2025-04-26 19:49:49,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2406
2136
[WARNING|trainer.py:803] 2025-04-26 19:49:50,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:49:50,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:50,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2103
2137
[WARNING|trainer.py:803] 2025-04-26 19:49:51,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2104
[WARNING|trainer.py:803] 2025-04-26 19:49:52,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2138
[WARNING|trainer.py:803] 2025-04-26 19:49:53,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:49:53,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2105
2139
[WARNING|trainer.py:803] 2025-04-26 19:49:54,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:54,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2106
2140
2407
[WARNING|trainer.py:803] 2025-04-26 19:49:55,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:56,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2107
[WARNING|trainer.py:803] 2025-04-26 19:49:56,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2141
[WARNING|trainer.py:803] 2025-04-26 19:49:56,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2108
[WARNING|trainer.py:803] 2025-04-26 19:49:57,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2408
2142
[WARNING|trainer.py:803] 2025-04-26 19:49:58,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:58,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:49:58,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2109
2143
[WARNING|trainer.py:803] 2025-04-26 19:49:59,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2409
2110
[WARNING|trainer.py:803] 2025-04-26 19:50:00,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:50:00,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2144
[WARNING|trainer.py:803] 2025-04-26 19:50:00,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2111
[WARNING|trainer.py:803] 2025-04-26 19:50:01,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2410
2145
[WARNING|trainer.py:803] 2025-04-26 19:50:02,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:02,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2112
[WARNING|trainer.py:803] 2025-04-26 19:50:02,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2146
2411
[WARNING|trainer.py:803] 2025-04-26 19:50:03,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2113
[WARNING|trainer.py:803] 2025-04-26 19:50:03,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:04,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2147
[WARNING|trainer.py:803] 2025-04-26 19:50:04,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2114
[WARNING|trainer.py:803] 2025-04-26 19:50:05,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2412
2148
[WARNING|trainer.py:803] 2025-04-26 19:50:05,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2115
[WARNING|trainer.py:803] 2025-04-26 19:50:06,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:50:06,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2149
[WARNING|trainer.py:803] 2025-04-26 19:50:07,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2116
2413
[WARNING|trainer.py:803] 2025-04-26 19:50:07,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2150
[WARNING|trainer.py:803] 2025-04-26 19:50:08,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:08,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2117
[WARNING|trainer.py:803] 2025-04-26 19:50:09,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2151
[WARNING|trainer.py:803] 2025-04-26 19:50:09,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2414
2118
[WARNING|trainer.py:803] 2025-04-26 19:50:10,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:10,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2152
[WARNING|trainer.py:803] 2025-04-26 19:50:11,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2119
2415
[WARNING|trainer.py:803] 2025-04-26 19:50:11,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2153
[WARNING|trainer.py:803] 2025-04-26 19:50:12,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:12,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2120
[WARNING|trainer.py:803] 2025-04-26 19:50:13,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2154
[WARNING|trainer.py:803] 2025-04-26 19:50:13,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2121
2416
[WARNING|trainer.py:803] 2025-04-26 19:50:14,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2155
[WARNING|trainer.py:803] 2025-04-26 19:50:14,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:15,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2122
[WARNING|trainer.py:803] 2025-04-26 19:50:15,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2156
2417
[WARNING|trainer.py:803] 2025-04-26 19:50:16,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2123
[WARNING|trainer.py:803] 2025-04-26 19:50:16,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:17,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2157
[WARNING|trainer.py:803] 2025-04-26 19:50:17,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2418
2124
[WARNING|trainer.py:803] 2025-04-26 19:50:18,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2158
[WARNING|trainer.py:803] 2025-04-26 19:50:18,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:18,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2125
[WARNING|trainer.py:803] 2025-04-26 19:50:19,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2159
2419
[WARNING|trainer.py:803] 2025-04-26 19:50:20,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2126
[WARNING|trainer.py:803] 2025-04-26 19:50:20,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:21,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2160
[WARNING|trainer.py:803] 2025-04-26 19:50:21,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2127
[WARNING|trainer.py:803] 2025-04-26 19:50:22,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2420
2161
[WARNING|trainer.py:803] 2025-04-26 19:50:22,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:23,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:23,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2128
2162
2421
[WARNING|trainer.py:803] 2025-04-26 19:50:24,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2129
[WARNING|trainer.py:803] 2025-04-26 19:50:24,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:24,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2163
[WARNING|trainer.py:803] 2025-04-26 19:50:25,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2130
[WARNING|trainer.py:803] 2025-04-26 19:50:25,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2422
2164
[WARNING|trainer.py:803] 2025-04-26 19:50:26,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:27,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:27,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2131
2165
[WARNING|trainer.py:803] 2025-04-26 19:50:28,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2423
2132
[WARNING|trainer.py:803] 2025-04-26 19:50:28,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2166
[WARNING|trainer.py:803] 2025-04-26 19:50:29,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:29,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:29,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2133
2167
[WARNING|trainer.py:803] 2025-04-26 19:50:30,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:31,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2134
2424
2168
[WARNING|trainer.py:803] 2025-04-26 19:50:32,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:50:32,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:32,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2135
2169
2425
[WARNING|trainer.py:803] 2025-04-26 19:50:33,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:33,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2136
[WARNING|trainer.py:803] 2025-04-26 19:50:34,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2170
[WARNING|trainer.py:803] 2025-04-26 19:50:34,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:35,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2137
2171
2426
[WARNING|trainer.py:803] 2025-04-26 19:50:36,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:36,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:36,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2138
2172
[WARNING|trainer.py:803] 2025-04-26 19:50:37,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2427
[WARNING|trainer.py:803] 2025-04-26 19:50:37,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2139
2173
[WARNING|trainer.py:803] 2025-04-26 19:50:38,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:38,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:39,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2140
2174
[WARNING|trainer.py:803] 2025-04-26 19:50:39,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2428
[WARNING|trainer.py:803] 2025-04-26 19:50:40,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2141
2175
[WARNING|trainer.py:803] 2025-04-26 19:50:40,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:50:41,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:41,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2142
2429
2176
[WARNING|trainer.py:803] 2025-04-26 19:50:42,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:42,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:42,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2143
2177
[WARNING|trainer.py:803] 2025-04-26 19:50:43,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2430
[WARNING|trainer.py:803] 2025-04-26 19:50:44,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2144
2178
[WARNING|trainer.py:803] 2025-04-26 19:50:45,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:45,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:45,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2145
2179
2431
[WARNING|trainer.py:803] 2025-04-26 19:50:46,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:46,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2146
2180
[WARNING|trainer.py:803] 2025-04-26 19:50:47,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:47,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:48,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2147
2181
2432
[WARNING|trainer.py:803] 2025-04-26 19:50:49,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:49,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:50:49,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2148
2182
[WARNING|trainer.py:803] 2025-04-26 19:50:50,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2433
[WARNING|trainer.py:803] 2025-04-26 19:50:50,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2149
2183
[WARNING|trainer.py:803] 2025-04-26 19:50:51,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:51,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:52,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2150
2184
2434
[WARNING|trainer.py:803] 2025-04-26 19:50:53,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:53,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2151
[WARNING|trainer.py:803] 2025-04-26 19:50:53,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2185
[WARNING|trainer.py:803] 2025-04-26 19:50:54,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:54,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2435
2152
2186
[WARNING|trainer.py:803] 2025-04-26 19:50:55,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:50:55,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:50:56,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2153
2187
2436
[WARNING|trainer.py:803] 2025-04-26 19:50:57,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:57,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:57,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2154
2188
[WARNING|trainer.py:803] 2025-04-26 19:50:58,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2437
[WARNING|trainer.py:803] 2025-04-26 19:50:58,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2155
2189
[WARNING|trainer.py:803] 2025-04-26 19:50:59,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:50:59,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:00,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2156
2190
2438
[WARNING|trainer.py:803] 2025-04-26 19:51:01,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:01,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2157
[WARNING|trainer.py:803] 2025-04-26 19:51:01,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2191
[WARNING|trainer.py:803] 2025-04-26 19:51:02,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2439
[WARNING|trainer.py:803] 2025-04-26 19:51:02,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2158
2192
[WARNING|trainer.py:803] 2025-04-26 19:51:03,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:03,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:04,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2159
2193
2440
[WARNING|trainer.py:803] 2025-04-26 19:51:04,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:05,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2160
[WARNING|trainer.py:803] 2025-04-26 19:51:05,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2194
[WARNING|trainer.py:803] 2025-04-26 19:51:06,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2161
[WARNING|trainer.py:803] 2025-04-26 19:51:06,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2441
2195
[WARNING|trainer.py:803] 2025-04-26 19:51:07,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:07,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:08,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2162
2196
[WARNING|trainer.py:803] 2025-04-26 19:51:09,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:09,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2163
2197
[WARNING|trainer.py:803] 2025-04-26 19:51:10,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:10,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2164
2198
[WARNING|trainer.py:803] 2025-04-26 19:51:11,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:12,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2165
2199
[WARNING|trainer.py:803] 2025-04-26 19:51:12,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:13,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2442
2166
2200
[WARNING|trainer.py:803] 2025-04-26 19:51:14,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:51:14,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:14,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2167
2201
2443
[WARNING|trainer.py:803] 2025-04-26 19:51:15,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:15,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2168
[WARNING|trainer.py:803] 2025-04-26 19:51:16,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2202
[WARNING|trainer.py:803] 2025-04-26 19:51:16,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2169
[WARNING|trainer.py:803] 2025-04-26 19:51:17,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2444
2203
[WARNING|trainer.py:803] 2025-04-26 19:51:18,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2170
[WARNING|trainer.py:803] 2025-04-26 19:51:18,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 19:51:18,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2204
[WARNING|trainer.py:803] 2025-04-26 19:51:19,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2445
2171
[WARNING|trainer.py:803] 2025-04-26 19:51:20,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2205
[WARNING|trainer.py:803] 2025-04-26 19:51:20,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:20,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2172
[WARNING|trainer.py:803] 2025-04-26 19:51:21,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2206
[WARNING|trainer.py:803] 2025-04-26 19:51:22,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2446
2173
[WARNING|trainer.py:803] 2025-04-26 19:51:22,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:23,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2207
[WARNING|trainer.py:803] 2025-04-26 19:51:23,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2174
[WARNING|trainer.py:803] 2025-04-26 19:51:24,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2208
[WARNING|trainer.py:803] 2025-04-26 19:51:24,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2175
[WARNING|trainer.py:803] 2025-04-26 19:51:25,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2447
2209
[WARNING|trainer.py:803] 2025-04-26 19:51:26,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:26,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2176
[WARNING|trainer.py:803] 2025-04-26 19:51:26,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2210
[WARNING|trainer.py:803] 2025-04-26 19:51:27,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2448
2177
[WARNING|trainer.py:803] 2025-04-26 19:51:28,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:28,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2211
[WARNING|trainer.py:803] 2025-04-26 19:51:28,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2178
[WARNING|trainer.py:803] 2025-04-26 19:51:29,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2212
[WARNING|trainer.py:803] 2025-04-26 19:51:29,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2179
[WARNING|trainer.py:803] 2025-04-26 19:51:30,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2213
[WARNING|trainer.py:803] 2025-04-26 19:51:31,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2180
[WARNING|trainer.py:803] 2025-04-26 19:51:31,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2214
[WARNING|trainer.py:803] 2025-04-26 19:51:32,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2181
[WARNING|trainer.py:803] 2025-04-26 19:51:33,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2215
2449
[WARNING|trainer.py:803] 2025-04-26 19:51:33,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2182
[WARNING|trainer.py:803] 2025-04-26 19:51:34,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:34,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2216
[WARNING|trainer.py:803] 2025-04-26 19:51:35,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2183
[WARNING|trainer.py:803] 2025-04-26 19:51:35,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2217
2450
[WARNING|trainer.py:803] 2025-04-26 19:51:36,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2184
[WARNING|trainer.py:803] 2025-04-26 19:51:37,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:37,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2218
[WARNING|trainer.py:803] 2025-04-26 19:51:37,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2185
[WARNING|trainer.py:803] 2025-04-26 19:51:38,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2451
2219
[WARNING|trainer.py:803] 2025-04-26 19:51:39,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:39,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2186
[WARNING|trainer.py:803] 2025-04-26 19:51:39,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2220
[WARNING|trainer.py:803] 2025-04-26 19:51:40,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2452
2187
[WARNING|trainer.py:803] 2025-04-26 19:51:41,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2221
[WARNING|trainer.py:803] 2025-04-26 19:51:41,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:51:42,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2188
[WARNING|trainer.py:803] 2025-04-26 19:51:42,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2453
2222
[WARNING|trainer.py:803] 2025-04-26 19:51:43,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2189
[WARNING|trainer.py:803] 2025-04-26 19:51:43,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:44,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:44,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2223
2454
2190
[WARNING|trainer.py:803] 2025-04-26 19:51:45,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:51:45,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:45,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2224
2191
[WARNING|trainer.py:803] 2025-04-26 19:51:46,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2455
[WARNING|trainer.py:803] 2025-04-26 19:51:47,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2225
2192
[WARNING|trainer.py:803] 2025-04-26 19:51:47,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:48,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:48,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2226
2193
[WARNING|trainer.py:803] 2025-04-26 19:51:49,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:49,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2227
2194
[WARNING|trainer.py:803] 2025-04-26 19:51:50,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:51,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2228
2195
2456
[WARNING|trainer.py:803] 2025-04-26 19:51:52,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:52,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:52,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2229
2196
[WARNING|trainer.py:803] 2025-04-26 19:51:53,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2457
[WARNING|trainer.py:803] 2025-04-26 19:51:53,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2230
2197
[WARNING|trainer.py:803] 2025-04-26 19:51:54,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:51:54,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:51:55,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2231
2198
2458
[WARNING|trainer.py:803] 2025-04-26 19:51:56,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:56,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:56,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2232
2199
[WARNING|trainer.py:803] 2025-04-26 19:51:57,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2459
[WARNING|trainer.py:803] 2025-04-26 19:51:57,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2233
2200
[WARNING|trainer.py:803] 2025-04-26 19:51:58,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:51:58,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:51:59,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2234
2201
[WARNING|trainer.py:803] 2025-04-26 19:52:00,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2460
[WARNING|trainer.py:803] 2025-04-26 19:52:00,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2235
2202
[WARNING|trainer.py:803] 2025-04-26 19:52:01,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:01,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:01,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2461
2236
2203
[WARNING|trainer.py:803] 2025-04-26 19:52:02,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:02,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:03,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2237
2204
[WARNING|trainer.py:803] 2025-04-26 19:52:04,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2462
[WARNING|trainer.py:803] 2025-04-26 19:52:04,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2238
2205
[WARNING|trainer.py:803] 2025-04-26 19:52:05,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:05,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:05,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2239
2206
[WARNING|trainer.py:803] 2025-04-26 19:52:06,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:07,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2240
2207
[WARNING|trainer.py:803] 2025-04-26 19:52:08,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:08,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2241
2208
2463
[WARNING|trainer.py:803] 2025-04-26 19:52:09,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:09,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2242
[WARNING|trainer.py:803] 2025-04-26 19:52:10,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2209
[WARNING|trainer.py:803] 2025-04-26 19:52:10,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:11,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2243
2210
[WARNING|trainer.py:803] 2025-04-26 19:52:12,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2244
[WARNING|trainer.py:803] 2025-04-26 19:52:12,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2211
[WARNING|trainer.py:803] 2025-04-26 19:52:13,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:13,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2245
2212
2464
[WARNING|trainer.py:803] 2025-04-26 19:52:14,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2246
[WARNING|trainer.py:803] 2025-04-26 19:52:15,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:15,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2213
[WARNING|trainer.py:803] 2025-04-26 19:52:16,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2247
[WARNING|trainer.py:803] 2025-04-26 19:52:16,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2214
[WARNING|trainer.py:803] 2025-04-26 19:52:17,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:17,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2248
2215
[WARNING|trainer.py:803] 2025-04-26 19:52:18,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2249
[WARNING|trainer.py:803] 2025-04-26 19:52:19,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2216
[WARNING|trainer.py:803] 2025-04-26 19:52:19,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2465
2250
[WARNING|trainer.py:803] 2025-04-26 19:52:20,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2217
[WARNING|trainer.py:803] 2025-04-26 19:52:21,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:21,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2251
[WARNING|trainer.py:803] 2025-04-26 19:52:21,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2218
[WARNING|trainer.py:803] 2025-04-26 19:52:22,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2252
[WARNING|trainer.py:803] 2025-04-26 19:52:23,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2219
[WARNING|trainer.py:803] 2025-04-26 19:52:23,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2253
[WARNING|trainer.py:803] 2025-04-26 19:52:24,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:25,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2220
2254
2466
[WARNING|trainer.py:803] 2025-04-26 19:52:26,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:26,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:26,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2221
2255
[WARNING|trainer.py:803] 2025-04-26 19:52:27,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:27,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2222
2256
[WARNING|trainer.py:803] 2025-04-26 19:52:28,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:28,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2257
2223
[WARNING|trainer.py:803] 2025-04-26 19:52:30,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:30,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2258
2224
2467
[WARNING|trainer.py:803] 2025-04-26 19:52:31,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:31,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:31,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2259
2225
[WARNING|trainer.py:803] 2025-04-26 19:52:32,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:32,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2260
2226
[WARNING|trainer.py:803] 2025-04-26 19:52:33,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:34,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2261
2227
[WARNING|trainer.py:803] 2025-04-26 19:52:35,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:35,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2262
2228
[WARNING|trainer.py:803] 2025-04-26 19:52:36,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2468
[WARNING|trainer.py:803] 2025-04-26 19:52:36,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2263
2229
[WARNING|trainer.py:803] 2025-04-26 19:52:37,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:37,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:38,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2264
2230
[WARNING|trainer.py:803] 2025-04-26 19:52:39,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:39,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2265
2231
[WARNING|trainer.py:803] 2025-04-26 19:52:40,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:40,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2266
2232
[WARNING|trainer.py:803] 2025-04-26 19:52:41,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2469
2267
[WARNING|trainer.py:803] 2025-04-26 19:52:42,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:42,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2233
[WARNING|trainer.py:803] 2025-04-26 19:52:42,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2268
[WARNING|trainer.py:803] 2025-04-26 19:52:43,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2234
[WARNING|trainer.py:803] 2025-04-26 19:52:44,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2269
[WARNING|trainer.py:803] 2025-04-26 19:52:44,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2235
[WARNING|trainer.py:803] 2025-04-26 19:52:45,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2270
[WARNING|trainer.py:803] 2025-04-26 19:52:46,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2236
[WARNING|trainer.py:803] 2025-04-26 19:52:46,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2470
2271
[WARNING|trainer.py:803] 2025-04-26 19:52:47,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:47,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2237
[WARNING|trainer.py:803] 2025-04-26 19:52:48,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2272
[WARNING|trainer.py:803] 2025-04-26 19:52:48,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2238
[WARNING|trainer.py:803] 2025-04-26 19:52:49,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2273
[WARNING|trainer.py:803] 2025-04-26 19:52:50,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2239
[WARNING|trainer.py:803] 2025-04-26 19:52:50,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2274
[WARNING|trainer.py:803] 2025-04-26 19:52:51,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2240
[WARNING|trainer.py:803] 2025-04-26 19:52:51,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2275
2471
[WARNING|trainer.py:803] 2025-04-26 19:52:52,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:53,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2241
[WARNING|trainer.py:803] 2025-04-26 19:52:53,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2276
[WARNING|trainer.py:803] 2025-04-26 19:52:54,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:54,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2242
2472
2277
[WARNING|trainer.py:803] 2025-04-26 19:52:55,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:55,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:52:55,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2243
2278
2473
[WARNING|trainer.py:803] 2025-04-26 19:52:56,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:57,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:52:57,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2244
2279
[WARNING|trainer.py:803] 2025-04-26 19:52:58,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2474
[WARNING|trainer.py:803] 2025-04-26 19:52:58,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2245
2280
[WARNING|trainer.py:803] 2025-04-26 19:52:59,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:52:59,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:52:59,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2246
2281
[WARNING|trainer.py:803] 2025-04-26 19:53:00,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:00,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2247
2282
2475
[WARNING|trainer.py:803] 2025-04-26 19:53:02,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:02,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:02,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2283
2248
2476
[WARNING|trainer.py:803] 2025-04-26 19:53:03,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:03,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2284
2249
[WARNING|trainer.py:803] 2025-04-26 19:53:04,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:04,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:04,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2285
2477
2250
[WARNING|trainer.py:803] 2025-04-26 19:53:06,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:06,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:06,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2286
2251
2478
[WARNING|trainer.py:803] 2025-04-26 19:53:07,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:07,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2287
2252
[WARNING|trainer.py:803] 2025-04-26 19:53:08,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:08,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:08,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2288
2253
2479
[WARNING|trainer.py:803] 2025-04-26 19:53:09,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:09,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:10,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2289
2254
[WARNING|trainer.py:803] 2025-04-26 19:53:11,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:11,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2480
2290
2255
[WARNING|trainer.py:803] 2025-04-26 19:53:12,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:53:12,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:12,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2291
2256
2481
[WARNING|trainer.py:803] 2025-04-26 19:53:13,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:13,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:53:13,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2292
2257
2482
[WARNING|trainer.py:803] 2025-04-26 19:53:14,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:15,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2293
2258
[WARNING|trainer.py:803] 2025-04-26 19:53:15,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:16,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:16,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2294
2259
[WARNING|trainer.py:803] 2025-04-26 19:53:17,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:17,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2295
2260
2483
[WARNING|trainer.py:803] 2025-04-26 19:53:18,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:18,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:19,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2296
2261
[WARNING|trainer.py:803] 2025-04-26 19:53:19,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:20,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2297
2262
[WARNING|trainer.py:803] 2025-04-26 19:53:21,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2484
[WARNING|trainer.py:803] 2025-04-26 19:53:21,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2298
2263
[WARNING|trainer.py:803] 2025-04-26 19:53:22,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:22,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:22,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2299
2485
2264
[WARNING|trainer.py:803] 2025-04-26 19:53:23,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2300
[WARNING|trainer.py:803] 2025-04-26 19:53:24,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:24,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2265
[WARNING|trainer.py:803] 2025-04-26 19:53:24,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2301
[WARNING|trainer.py:803] 2025-04-26 19:53:25,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2486
2266
[WARNING|trainer.py:803] 2025-04-26 19:53:26,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:26,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2302
[WARNING|trainer.py:803] 2025-04-26 19:53:26,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2267
[WARNING|trainer.py:803] 2025-04-26 19:53:27,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2487
2303
[WARNING|trainer.py:803] 2025-04-26 19:53:28,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2268
[WARNING|trainer.py:803] 2025-04-26 19:53:28,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:28,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2304
[WARNING|trainer.py:803] 2025-04-26 19:53:29,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2488
2269
[WARNING|trainer.py:803] 2025-04-26 19:53:30,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:30,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2305
[WARNING|trainer.py:803] 2025-04-26 19:53:30,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2270
[WARNING|trainer.py:803] 2025-04-26 19:53:31,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2489
2306
[WARNING|trainer.py:803] 2025-04-26 19:53:31,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:53:32,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2271
[WARNING|trainer.py:803] 2025-04-26 19:53:32,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2307
[WARNING|trainer.py:803] 2025-04-26 19:53:33,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2490
2272
[WARNING|trainer.py:803] 2025-04-26 19:53:33,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:34,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2308
[WARNING|trainer.py:803] 2025-04-26 19:53:34,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2273
[WARNING|trainer.py:803] 2025-04-26 19:53:35,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2491
2309
[WARNING|trainer.py:803] 2025-04-26 19:53:35,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2274
[WARNING|trainer.py:803] 2025-04-26 19:53:36,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:36,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2310
[WARNING|trainer.py:803] 2025-04-26 19:53:36,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2275
[WARNING|trainer.py:803] 2025-04-26 19:53:37,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2492
2311
[WARNING|trainer.py:803] 2025-04-26 19:53:38,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2276
[WARNING|trainer.py:803] 2025-04-26 19:53:38,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:38,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2312
[WARNING|trainer.py:803] 2025-04-26 19:53:39,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2493
2277
[WARNING|trainer.py:803] 2025-04-26 19:53:40,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2313
[WARNING|trainer.py:803] 2025-04-26 19:53:40,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:40,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2278
[WARNING|trainer.py:803] 2025-04-26 19:53:41,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2314
2494
[WARNING|trainer.py:803] 2025-04-26 19:53:41,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2279
[WARNING|trainer.py:803] 2025-04-26 19:53:42,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:42,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2315
[WARNING|trainer.py:803] 2025-04-26 19:53:43,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2280
[WARNING|trainer.py:803] 2025-04-26 19:53:43,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2316
2495
[WARNING|trainer.py:803] 2025-04-26 19:53:44,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2281
[WARNING|trainer.py:803] 2025-04-26 19:53:45,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:45,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2317
[WARNING|trainer.py:803] 2025-04-26 19:53:45,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2282
2496
[WARNING|trainer.py:803] 2025-04-26 19:53:46,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2318
[WARNING|trainer.py:803] 2025-04-26 19:53:47,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:47,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2283
[WARNING|trainer.py:803] 2025-04-26 19:53:47,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2319
2497
[WARNING|trainer.py:803] 2025-04-26 19:53:48,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2284
[WARNING|trainer.py:803] 2025-04-26 19:53:48,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:53:49,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2320
[WARNING|trainer.py:803] 2025-04-26 19:53:49,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2285
[WARNING|trainer.py:803] 2025-04-26 19:53:50,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2498
2321
[WARNING|trainer.py:803] 2025-04-26 19:53:50,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2286
[WARNING|trainer.py:803] 2025-04-26 19:53:51,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:53:51,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2322
[WARNING|trainer.py:803] 2025-04-26 19:53:52,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2499
2287
[WARNING|trainer.py:803] 2025-04-26 19:53:52,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2323
[WARNING|trainer.py:803] 2025-04-26 19:53:53,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:53:53,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2288
[WARNING|trainer.py:803] 2025-04-26 19:53:53,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2324
[WARNING|trainer.py:803] 2025-04-26 19:53:54,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2289
[WARNING|trainer.py:803] 2025-04-26 19:53:55,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2325
[WARNING|trainer.py:803] 2025-04-26 19:53:56,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2290
[WARNING|trainer.py:803] 2025-04-26 19:53:56,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2326
[WARNING|trainer.py:803] 2025-04-26 19:53:57,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2500
2291
[WARNING|trainer.py:803] 2025-04-26 19:53:57,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2327
[WARNING|trainer.py:803] 2025-04-26 19:53:58,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:53:58,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:53:58,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2292
2328
[WARNING|trainer.py:803] 2025-04-26 19:53:59,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2501
[WARNING|trainer.py:803] 2025-04-26 19:54:00,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2293
2329
[WARNING|trainer.py:803] 2025-04-26 19:54:00,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:01,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:01,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2294
2330
2502
[WARNING|trainer.py:803] 2025-04-26 19:54:02,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:02,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2295
[WARNING|trainer.py:803] 2025-04-26 19:54:03,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2331
[WARNING|trainer.py:803] 2025-04-26 19:54:03,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:03,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2296
2503
2332
[WARNING|trainer.py:803] 2025-04-26 19:54:04,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:04,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:05,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2297
2333
[WARNING|trainer.py:803] 2025-04-26 19:54:06,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:06,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2298
2504
2334
[WARNING|trainer.py:803] 2025-04-26 19:54:07,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:07,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:07,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2299
2335
[WARNING|trainer.py:803] 2025-04-26 19:54:08,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2505
[WARNING|trainer.py:803] 2025-04-26 19:54:09,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2300
2336
[WARNING|trainer.py:803] 2025-04-26 19:54:09,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:10,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:10,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2301
2506
2337
[WARNING|trainer.py:803] 2025-04-26 19:54:11,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:11,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:11,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2302
2338
[WARNING|trainer.py:803] 2025-04-26 19:54:12,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:12,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2303
2507
2339
[WARNING|trainer.py:803] 2025-04-26 19:54:13,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:14,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:14,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2304
2340
2508
[WARNING|trainer.py:803] 2025-04-26 19:54:15,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2305
[WARNING|trainer.py:803] 2025-04-26 19:54:15,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:15,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2341
[WARNING|trainer.py:803] 2025-04-26 19:54:16,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2306
[WARNING|trainer.py:803] 2025-04-26 19:54:16,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2509
2342
[WARNING|trainer.py:803] 2025-04-26 19:54:17,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:17,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2307
[WARNING|trainer.py:803] 2025-04-26 19:54:18,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2343
[WARNING|trainer.py:803] 2025-04-26 19:54:18,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2510
2308
[WARNING|trainer.py:803] 2025-04-26 19:54:19,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2344
[WARNING|trainer.py:803] 2025-04-26 19:54:19,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:20,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2309
[WARNING|trainer.py:803] 2025-04-26 19:54:20,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2345
2511
[WARNING|trainer.py:803] 2025-04-26 19:54:21,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2310
[WARNING|trainer.py:803] 2025-04-26 19:54:21,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:22,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2346
[WARNING|trainer.py:803] 2025-04-26 19:54:22,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2512
2311
[WARNING|trainer.py:803] 2025-04-26 19:54:23,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2347
[WARNING|trainer.py:803] 2025-04-26 19:54:23,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:23,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:24,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2312
2348
2513
[WARNING|trainer.py:803] 2025-04-26 19:54:25,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2313
[WARNING|trainer.py:803] 2025-04-26 19:54:25,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:26,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2349
[WARNING|trainer.py:803] 2025-04-26 19:54:26,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2314
[WARNING|trainer.py:803] 2025-04-26 19:54:26,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2350
2514
[WARNING|trainer.py:803] 2025-04-26 19:54:27,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2315
[WARNING|trainer.py:803] 2025-04-26 19:54:28,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:54:28,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2351
[WARNING|trainer.py:803] 2025-04-26 19:54:29,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2316
2515
[WARNING|trainer.py:803] 2025-04-26 19:54:29,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2352
[WARNING|trainer.py:803] 2025-04-26 19:54:30,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:30,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2317
[WARNING|trainer.py:803] 2025-04-26 19:54:30,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2353
2516
[WARNING|trainer.py:803] 2025-04-26 19:54:31,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2318
[WARNING|trainer.py:803] 2025-04-26 19:54:32,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:32,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2354
[WARNING|trainer.py:803] 2025-04-26 19:54:32,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2319
[WARNING|trainer.py:803] 2025-04-26 19:54:33,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2355
2517
[WARNING|trainer.py:803] 2025-04-26 19:54:34,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2320
[WARNING|trainer.py:803] 2025-04-26 19:54:34,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2356
[WARNING|trainer.py:803] 2025-04-26 19:54:35,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:35,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2321
[WARNING|trainer.py:803] 2025-04-26 19:54:35,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2357
[WARNING|trainer.py:803] 2025-04-26 19:54:36,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2322
[WARNING|trainer.py:803] 2025-04-26 19:54:37,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2518
2358
[WARNING|trainer.py:803] 2025-04-26 19:54:37,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:38,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:38,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2323
2359
[WARNING|trainer.py:803] 2025-04-26 19:54:39,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2519
[WARNING|trainer.py:803] 2025-04-26 19:54:39,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2324
2360
[WARNING|trainer.py:803] 2025-04-26 19:54:40,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:40,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:40,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2325
2361
[WARNING|trainer.py:803] 2025-04-26 19:54:41,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2520
[WARNING|trainer.py:803] 2025-04-26 19:54:42,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2326
2362
[WARNING|trainer.py:803] 2025-04-26 19:54:42,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:43,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2327
2363
2521
[WARNING|trainer.py:803] 2025-04-26 19:54:44,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:44,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2328
2364
[WARNING|trainer.py:803] 2025-04-26 19:54:45,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:45,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:45,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2329
2365
[WARNING|trainer.py:803] 2025-04-26 19:54:46,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:47,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2330
2366
[WARNING|trainer.py:803] 2025-04-26 19:54:48,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:48,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2331
2367
2522
[WARNING|trainer.py:803] 2025-04-26 19:54:49,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:49,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2332
2368
[WARNING|trainer.py:803] 2025-04-26 19:54:50,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:54:50,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:50,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2333
2369
[WARNING|trainer.py:803] 2025-04-26 19:54:52,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:52,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2334
2370
[WARNING|trainer.py:803] 2025-04-26 19:54:53,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:53,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2335
2371
[WARNING|trainer.py:803] 2025-04-26 19:54:54,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:54,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2336
2523
2372
[WARNING|trainer.py:803] 2025-04-26 19:54:55,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:55,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:54:56,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2337
2373
[WARNING|trainer.py:803] 2025-04-26 19:54:57,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:57,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2338
2374
[WARNING|trainer.py:803] 2025-04-26 19:54:58,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:58,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2339
2375
[WARNING|trainer.py:803] 2025-04-26 19:54:59,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:54:59,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2340
2376
[WARNING|trainer.py:803] 2025-04-26 19:55:00,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2524
[WARNING|trainer.py:803] 2025-04-26 19:55:01,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2341
2377
[WARNING|trainer.py:803] 2025-04-26 19:55:01,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:02,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:02,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2342
2378
[WARNING|trainer.py:803] 2025-04-26 19:55:03,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:03,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2343
2379
[WARNING|trainer.py:803] 2025-04-26 19:55:04,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:05,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2344
2380
[WARNING|trainer.py:803] 2025-04-26 19:55:06,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:06,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2345
2381
2525
[WARNING|trainer.py:803] 2025-04-26 19:55:07,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:07,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2346
2382
[WARNING|trainer.py:803] 2025-04-26 19:55:08,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:08,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:08,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2347
2383
[WARNING|trainer.py:803] 2025-04-26 19:55:10,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:10,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2348
2384
[WARNING|trainer.py:803] 2025-04-26 19:55:11,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:11,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2349
2385
[WARNING|trainer.py:803] 2025-04-26 19:55:12,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:12,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2350
2386
2526
[WARNING|trainer.py:803] 2025-04-26 19:55:13,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:13,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:14,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2351
2387
[WARNING|trainer.py:803] 2025-04-26 19:55:15,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:15,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2388
2352
[WARNING|trainer.py:803] 2025-04-26 19:55:16,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:16,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2389
2353
[WARNING|trainer.py:803] 2025-04-26 19:55:17,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:17,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2390
2354
[WARNING|trainer.py:803] 2025-04-26 19:55:19,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:19,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2391
2355
[WARNING|trainer.py:803] 2025-04-26 19:55:20,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:20,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2392
2356
[WARNING|trainer.py:803] 2025-04-26 19:55:21,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:21,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2393
2357
[WARNING|trainer.py:803] 2025-04-26 19:55:22,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:22,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2527
2394
2358
[WARNING|trainer.py:803] 2025-04-26 19:55:23,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:55:24,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:24,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2395
2359
[WARNING|trainer.py:803] 2025-04-26 19:55:25,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:25,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2396
2360
[WARNING|trainer.py:803] 2025-04-26 19:55:26,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:26,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2397
2361
2528
[WARNING|trainer.py:803] 2025-04-26 19:55:28,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:28,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2362
2398
[WARNING|trainer.py:803] 2025-04-26 19:55:28,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:55:29,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:29,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2363
2399
[WARNING|trainer.py:803] 2025-04-26 19:55:30,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:30,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2364
2400
[WARNING|trainer.py:803] 2025-04-26 19:55:32,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:32,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2365
2529
[WARNING|trainer.py:803] 2025-04-26 19:55:33,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:55:33,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2401
2366
[WARNING|trainer.py:803] 2025-04-26 19:55:34,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:55:34,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2367
[WARNING|trainer.py:803] 2025-04-26 19:55:35,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2402
2368
[WARNING|trainer.py:803] 2025-04-26 19:55:37,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:37,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2530
2369
[WARNING|trainer.py:803] 2025-04-26 19:55:38,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:38,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2403
2370
[WARNING|trainer.py:803] 2025-04-26 19:55:39,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:39,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2371
2404
[WARNING|trainer.py:803] 2025-04-26 19:55:41,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2372
[WARNING|trainer.py:803] 2025-04-26 19:55:41,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2531
[WARNING|trainer.py:803] 2025-04-26 19:55:42,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:42,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2373
2405
[WARNING|trainer.py:803] 2025-04-26 19:55:43,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:43,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2532
2374
[WARNING|trainer.py:803] 2025-04-26 19:55:44,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:55:45,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2406
2375
[WARNING|trainer.py:803] 2025-04-26 19:55:46,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:46,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2533
2376
[WARNING|trainer.py:803] 2025-04-26 19:55:47,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:47,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2377
2534
[WARNING|trainer.py:803] 2025-04-26 19:55:48,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:49,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2378
[WARNING|trainer.py:803] 2025-04-26 19:55:50,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2379
2407
2535
[WARNING|trainer.py:803] 2025-04-26 19:55:51,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:51,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:51,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2380
[WARNING|trainer.py:803] 2025-04-26 19:55:52,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2408
2536
2381
[WARNING|trainer.py:803] 2025-04-26 19:55:53,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:53,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:54,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2382
2537
[WARNING|trainer.py:803] 2025-04-26 19:55:55,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2409
2383
[WARNING|trainer.py:803] 2025-04-26 19:55:56,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:56,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:55:56,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2538
2384
2410
[WARNING|trainer.py:803] 2025-04-26 19:55:57,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:55:57,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2385
[WARNING|trainer.py:803] 2025-04-26 19:55:58,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:55:59,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2386
2539
2411
[WARNING|trainer.py:803] 2025-04-26 19:56:00,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:00,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:56:00,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2387
2540
[WARNING|trainer.py:803] 2025-04-26 19:56:01,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2388
2412
[WARNING|trainer.py:803] 2025-04-26 19:56:02,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:03,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:03,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2389
2541
[WARNING|trainer.py:803] 2025-04-26 19:56:04,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:04,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2413
2390
[WARNING|trainer.py:803] 2025-04-26 19:56:05,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:05,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2542
2391
[WARNING|trainer.py:803] 2025-04-26 19:56:06,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:07,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2414
2392
[WARNING|trainer.py:803] 2025-04-26 19:56:07,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:56:08,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2543
2393
2415
[WARNING|trainer.py:803] 2025-04-26 19:56:09,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:56:09,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:10,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2394
2544
[WARNING|trainer.py:803] 2025-04-26 19:56:10,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2395
[WARNING|trainer.py:803] 2025-04-26 19:56:11,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2416
[WARNING|trainer.py:803] 2025-04-26 19:56:12,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2396
[WARNING|trainer.py:803] 2025-04-26 19:56:12,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:13,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2397
2417
2545
[WARNING|trainer.py:803] 2025-04-26 19:56:14,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:56:15,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:15,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2398
2546
[WARNING|trainer.py:803] 2025-04-26 19:56:16,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2418
2399
[WARNING|trainer.py:803] 2025-04-26 19:56:17,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:17,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:17,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2400
2547
2419
[WARNING|trainer.py:803] 2025-04-26 19:56:19,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:19,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:56:19,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2401
2548
2420
[WARNING|trainer.py:803] 2025-04-26 19:56:21,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:56:21,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:21,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2549
2421
2402
[WARNING|trainer.py:803] 2025-04-26 19:56:23,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:23,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:56:24,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2550
2422
2403
[WARNING|trainer.py:803] 2025-04-26 19:56:25,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:26,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:26,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2551
[WARNING|trainer.py:803] 2025-04-26 19:56:27,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2423
2404
[WARNING|trainer.py:803] 2025-04-26 19:56:28,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:28,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2552
2405
[WARNING|trainer.py:803] 2025-04-26 19:56:30,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:56:31,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2424
2553
[WARNING|trainer.py:803] 2025-04-26 19:56:32,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2406
[WARNING|trainer.py:803] 2025-04-26 19:56:32,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:33,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2425
2554
[WARNING|trainer.py:803] 2025-04-26 19:56:34,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:56:35,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2426
2555
[WARNING|trainer.py:803] 2025-04-26 19:56:37,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 19:56:37,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2427
2407
2556
[WARNING|trainer.py:803] 2025-04-26 19:56:39,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:39,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:56:39,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2557
2408
[WARNING|trainer.py:803] 2025-04-26 19:56:41,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2428
[WARNING|trainer.py:803] 2025-04-26 19:56:41,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:41,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2409
2429
[WARNING|trainer.py:803] 2025-04-26 19:56:43,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:56:44,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2558
2410
[WARNING|trainer.py:803] 2025-04-26 19:56:45,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2430
[WARNING|trainer.py:803] 2025-04-26 19:56:45,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2559
[WARNING|trainer.py:803] 2025-04-26 19:56:46,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2411
[WARNING|trainer.py:803] 2025-04-26 19:56:47,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:48,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2431
2560
[WARNING|trainer.py:803] 2025-04-26 19:56:49,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2412
[WARNING|trainer.py:803] 2025-04-26 19:56:49,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:50,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2432
2561
[WARNING|trainer.py:803] 2025-04-26 19:56:51,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:52,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2413
2433
[WARNING|trainer.py:803] 2025-04-26 19:56:53,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2562
[WARNING|trainer.py:803] 2025-04-26 19:56:53,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:53,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2414
2563
[WARNING|trainer.py:803] 2025-04-26 19:56:55,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2434
[WARNING|trainer.py:803] 2025-04-26 19:56:56,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:56:56,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2415
[WARNING|trainer.py:803] 2025-04-26 19:56:57,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2435
[WARNING|trainer.py:803] 2025-04-26 19:56:58,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2564
2416
[WARNING|trainer.py:803] 2025-04-26 19:56:59,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2436
[WARNING|trainer.py:803] 2025-04-26 19:57:00,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:00,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2565
[WARNING|trainer.py:803] 2025-04-26 19:57:01,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2417
2437
[WARNING|trainer.py:803] 2025-04-26 19:57:02,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:02,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2566
2418
[WARNING|trainer.py:803] 2025-04-26 19:57:04,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2438
[WARNING|trainer.py:803] 2025-04-26 19:57:04,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:05,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2419
2439
[WARNING|trainer.py:803] 2025-04-26 19:57:06,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2567
[WARNING|trainer.py:803] 2025-04-26 19:57:07,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:08,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2420
2440
[WARNING|trainer.py:803] 2025-04-26 19:57:09,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:09,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2568
2421
[WARNING|trainer.py:803] 2025-04-26 19:57:11,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2441
[WARNING|trainer.py:803] 2025-04-26 19:57:11,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:12,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2569
2422
[WARNING|trainer.py:803] 2025-04-26 19:57:13,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:13,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2570
[WARNING|trainer.py:803] 2025-04-26 19:57:14,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2423
2571
[WARNING|trainer.py:803] 2025-04-26 19:57:16,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:16,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2442
2572
[WARNING|trainer.py:803] 2025-04-26 19:57:18,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:18,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2424
2573
[WARNING|trainer.py:803] 2025-04-26 19:57:19,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2443
[WARNING|trainer.py:803] 2025-04-26 19:57:20,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:21,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2425
2574
[WARNING|trainer.py:803] 2025-04-26 19:57:21,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:22,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2444
2575
2426
[WARNING|trainer.py:803] 2025-04-26 19:57:23,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 19:57:24,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:24,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2445
2427
[WARNING|trainer.py:803] 2025-04-26 19:57:25,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2576
[WARNING|trainer.py:803] 2025-04-26 19:57:26,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:27,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2446
2577
2428
[WARNING|trainer.py:803] 2025-04-26 19:57:28,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:29,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:29,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2429
2578
2447
[WARNING|trainer.py:803] 2025-04-26 19:57:31,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:31,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:32,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2430
2579
2448
[WARNING|trainer.py:803] 2025-04-26 19:57:34,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:34,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:34,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2431
2580
[WARNING|trainer.py:803] 2025-04-26 19:57:36,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:37,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2581
2432
[WARNING|trainer.py:803] 2025-04-26 19:57:39,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:39,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2449
2582
2433
[WARNING|trainer.py:803] 2025-04-26 19:57:40,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:41,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:41,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2450
2434
2583
[WARNING|trainer.py:803] 2025-04-26 19:57:43,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:43,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:44,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2435
2451
2584
[WARNING|trainer.py:803] 2025-04-26 19:57:46,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:46,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:46,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2436
2452
2585
[WARNING|trainer.py:803] 2025-04-26 19:57:48,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:48,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:48,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2437
2586
2453
[WARNING|trainer.py:803] 2025-04-26 19:57:50,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:50,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:57:50,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2454
2587
2438
[WARNING|trainer.py:803] 2025-04-26 19:57:52,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:53,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:53,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2439
2455
2588
[WARNING|trainer.py:803] 2025-04-26 19:57:55,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:57:55,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:57:55,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2440
2589
[WARNING|trainer.py:803] 2025-04-26 19:57:57,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:57:58,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2441
2456
2590
[WARNING|trainer.py:803] 2025-04-26 19:58:00,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:00,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:00,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2457
2591
[WARNING|trainer.py:803] 2025-04-26 19:58:02,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:02,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2458
[WARNING|trainer.py:803] 2025-04-26 19:58:04,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2592
[WARNING|trainer.py:803] 2025-04-26 19:58:05,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2442
2459
[WARNING|trainer.py:803] 2025-04-26 19:58:06,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:58:06,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2593
[WARNING|trainer.py:803] 2025-04-26 19:58:08,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2443
2460
2594
[WARNING|trainer.py:803] 2025-04-26 19:58:09,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:58:09,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:10,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2461
2444
2595
[WARNING|trainer.py:803] 2025-04-26 19:58:11,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:58:11,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:11,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
2445
2462
2596
[WARNING|trainer.py:803] 2025-04-26 19:58:14,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:14,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:14,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2446
2597
[WARNING|trainer.py:803] 2025-04-26 19:58:16,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:16,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2598
2463
[WARNING|trainer.py:803] 2025-04-26 19:58:18,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:19,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2599
2447
[WARNING|trainer.py:803] 2025-04-26 19:58:20,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:20,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2600
2448
[WARNING|trainer.py:803] 2025-04-26 19:58:22,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:22,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2464
[WARNING|trainer.py:803] 2025-04-26 19:58:24,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2601
[WARNING|trainer.py:803] 2025-04-26 19:58:25,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2602
[WARNING|trainer.py:803] 2025-04-26 19:58:27,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2449
2603
[WARNING|trainer.py:803] 2025-04-26 19:58:29,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2465
[WARNING|trainer.py:803] 2025-04-26 19:58:29,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:30,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2604
2450
[WARNING|trainer.py:803] 2025-04-26 19:58:32,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:58:32,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2605
2451
[WARNING|trainer.py:803] 2025-04-26 19:58:34,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2466
[WARNING|trainer.py:803] 2025-04-26 19:58:34,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:35,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2606
2452
[WARNING|trainer.py:803] 2025-04-26 19:58:36,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:37,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2607
[WARNING|trainer.py:803] 2025-04-26 19:58:38,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2453
[WARNING|trainer.py:803] 2025-04-26 19:58:39,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2608
2467
[WARNING|trainer.py:803] 2025-04-26 19:58:40,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2454
[WARNING|trainer.py:803] 2025-04-26 19:58:40,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2609
[WARNING|trainer.py:803] 2025-04-26 19:58:41,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:58:42,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2455
[WARNING|trainer.py:803] 2025-04-26 19:58:43,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2610
2468
[WARNING|trainer.py:803] 2025-04-26 19:58:46,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:46,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2456
[WARNING|trainer.py:803] 2025-04-26 19:58:48,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2457
[WARNING|trainer.py:803] 2025-04-26 19:58:51,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2611
2469
[WARNING|trainer.py:803] 2025-04-26 19:58:52,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:58:52,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2458
[WARNING|trainer.py:803] 2025-04-26 19:58:53,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2612
[WARNING|trainer.py:803] 2025-04-26 19:58:54,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2459
[WARNING|trainer.py:803] 2025-04-26 19:58:55,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2613
[WARNING|trainer.py:803] 2025-04-26 19:58:56,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2470
2460
[WARNING|trainer.py:803] 2025-04-26 19:58:57,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2614
[WARNING|trainer.py:803] 2025-04-26 19:58:58,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:58:58,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2461
2615
[WARNING|trainer.py:803] 2025-04-26 19:59:00,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:59:00,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2616
2462
[WARNING|trainer.py:803] 2025-04-26 19:59:02,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2471
[WARNING|trainer.py:803] 2025-04-26 19:59:02,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:03,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2617
[WARNING|trainer.py:803] 2025-04-26 19:59:04,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2472
[WARNING|trainer.py:803] 2025-04-26 19:59:05,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2618
2473
[WARNING|trainer.py:803] 2025-04-26 19:59:06,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2463
[WARNING|trainer.py:803] 2025-04-26 19:59:07,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2619
[WARNING|trainer.py:803] 2025-04-26 19:59:07,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:59:08,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2474
[WARNING|trainer.py:803] 2025-04-26 19:59:09,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2620
[WARNING|trainer.py:803] 2025-04-26 19:59:10,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2621
2464
2475
[WARNING|trainer.py:803] 2025-04-26 19:59:12,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:59:12,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:13,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2622
2476
[WARNING|trainer.py:803] 2025-04-26 19:59:14,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:15,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2477
2623
[WARNING|trainer.py:803] 2025-04-26 19:59:17,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:17,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2465
2478
[WARNING|trainer.py:803] 2025-04-26 19:59:18,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:19,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2624
[WARNING|trainer.py:803] 2025-04-26 19:59:20,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2479
2625
[WARNING|trainer.py:803] 2025-04-26 19:59:21,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 19:59:22,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2466
2480
[WARNING|trainer.py:803] 2025-04-26 19:59:24,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:24,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2626
[WARNING|trainer.py:803] 2025-04-26 19:59:25,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2481
[WARNING|trainer.py:803] 2025-04-26 19:59:26,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2627
[WARNING|trainer.py:803] 2025-04-26 19:59:27,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2482
2628
[WARNING|trainer.py:803] 2025-04-26 19:59:28,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2467
[WARNING|trainer.py:803] 2025-04-26 19:59:29,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:29,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2629
2483
[WARNING|trainer.py:803] 2025-04-26 19:59:31,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:59:32,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2630
[WARNING|trainer.py:803] 2025-04-26 19:59:33,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2484
2468
[WARNING|trainer.py:803] 2025-04-26 19:59:35,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:35,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2631
[WARNING|trainer.py:803] 2025-04-26 19:59:36,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2485
2632
[WARNING|trainer.py:803] 2025-04-26 19:59:37,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:38,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2633
2486
2469
[WARNING|trainer.py:803] 2025-04-26 19:59:40,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:40,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:40,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2634
2487
[WARNING|trainer.py:803] 2025-04-26 19:59:42,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:43,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2635
2488
[WARNING|trainer.py:803] 2025-04-26 19:59:44,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:59:45,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2470
2636
[WARNING|trainer.py:803] 2025-04-26 19:59:46,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2489
[WARNING|trainer.py:803] 2025-04-26 19:59:46,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:47,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2637
2490
[WARNING|trainer.py:803] 2025-04-26 19:59:48,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:49,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2638
[WARNING|trainer.py:803] 2025-04-26 19:59:50,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2471
2491
2639
[WARNING|trainer.py:803] 2025-04-26 19:59:51,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:51,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:59:52,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2472
2640
2492
[WARNING|trainer.py:803] 2025-04-26 19:59:53,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:54,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:54,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2473
2641
2493
[WARNING|trainer.py:803] 2025-04-26 19:59:55,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 19:59:56,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:56,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2474
2642
2494
[WARNING|trainer.py:803] 2025-04-26 19:59:58,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 19:59:58,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 19:59:58,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2495
2475
2643
[WARNING|trainer.py:803] 2025-04-26 20:00:01,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:01,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:02,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2476
2496
[WARNING|trainer.py:803] 2025-04-26 20:00:03,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:03,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2497
2477
[WARNING|trainer.py:803] 2025-04-26 20:00:05,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:05,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2644
2478
[WARNING|trainer.py:803] 2025-04-26 20:00:07,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2498
[WARNING|trainer.py:803] 2025-04-26 20:00:08,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:08,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2479
2499
[WARNING|trainer.py:803] 2025-04-26 20:00:10,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:00:10,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2645
2480
[WARNING|trainer.py:803] 2025-04-26 20:00:12,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:00:12,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2481
[WARNING|trainer.py:803] 2025-04-26 20:00:15,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2500
[WARNING|trainer.py:803] 2025-04-26 20:00:16,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2482
2646
[WARNING|trainer.py:803] 2025-04-26 20:00:17,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:17,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2501
[WARNING|trainer.py:803] 2025-04-26 20:00:18,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2483
2502
2647
[WARNING|trainer.py:803] 2025-04-26 20:00:20,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:21,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:21,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2503
2484
[WARNING|trainer.py:803] 2025-04-26 20:00:23,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:24,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2648
[WARNING|trainer.py:803] 2025-04-26 20:00:25,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2485
2504
[WARNING|trainer.py:803] 2025-04-26 20:00:26,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:26,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2505
2486
[WARNING|trainer.py:803] 2025-04-26 20:00:28,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2649
[WARNING|trainer.py:803] 2025-04-26 20:00:29,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:29,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2506
2487
[WARNING|trainer.py:803] 2025-04-26 20:00:30,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:31,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2507
2488
[WARNING|trainer.py:803] 2025-04-26 20:00:33,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:33,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2650
2489
2508
[WARNING|trainer.py:803] 2025-04-26 20:00:35,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:35,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:35,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2651
2490
[WARNING|trainer.py:803] 2025-04-26 20:00:36,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2509
[WARNING|trainer.py:803] 2025-04-26 20:00:37,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:37,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2652
[WARNING|trainer.py:803] 2025-04-26 20:00:38,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2491
2510
2653
[WARNING|trainer.py:803] 2025-04-26 20:00:40,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:00:40,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:40,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2511
2654
2492
[WARNING|trainer.py:803] 2025-04-26 20:00:42,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:42,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:42,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2655
2512
2493
[WARNING|trainer.py:803] 2025-04-26 20:00:44,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:44,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:45,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2656
2513
[WARNING|trainer.py:803] 2025-04-26 20:00:46,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2494
[WARNING|trainer.py:803] 2025-04-26 20:00:47,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2657
[WARNING|trainer.py:803] 2025-04-26 20:00:47,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:48,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2514
[WARNING|trainer.py:803] 2025-04-26 20:00:49,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2495
2658
[WARNING|trainer.py:803] 2025-04-26 20:00:50,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:50,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2515
2659
2496
[WARNING|trainer.py:803] 2025-04-26 20:00:51,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:52,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:52,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2660
2516
2497
[WARNING|trainer.py:803] 2025-04-26 20:00:54,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:00:54,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:00:54,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2661
[WARNING|trainer.py:803] 2025-04-26 20:00:56,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2517
2498
[WARNING|trainer.py:803] 2025-04-26 20:00:57,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2662
[WARNING|trainer.py:803] 2025-04-26 20:00:57,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:00:58,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2499
[WARNING|trainer.py:803] 2025-04-26 20:00:59,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2518
2663
[WARNING|trainer.py:803] 2025-04-26 20:01:00,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:00,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2519
2664
[WARNING|trainer.py:803] 2025-04-26 20:01:02,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:01:03,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2500
2665
[WARNING|trainer.py:803] 2025-04-26 20:01:04,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2520
[WARNING|trainer.py:803] 2025-04-26 20:01:05,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:05,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2501
[WARNING|trainer.py:803] 2025-04-26 20:01:07,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2521
[WARNING|trainer.py:803] 2025-04-26 20:01:08,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2502
2666
[WARNING|trainer.py:803] 2025-04-26 20:01:10,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:10,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2503
2667
[WARNING|trainer.py:803] 2025-04-26 20:01:12,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:12,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2522
2668
[WARNING|trainer.py:803] 2025-04-26 20:01:13,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2504
[WARNING|trainer.py:803] 2025-04-26 20:01:14,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:15,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2669
[WARNING|trainer.py:803] 2025-04-26 20:01:16,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2505
[WARNING|trainer.py:803] 2025-04-26 20:01:17,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2670
[WARNING|trainer.py:803] 2025-04-26 20:01:18,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2523
2506
[WARNING|trainer.py:803] 2025-04-26 20:01:19,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2671
[WARNING|trainer.py:803] 2025-04-26 20:01:19,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:20,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2507
2672
[WARNING|trainer.py:803] 2025-04-26 20:01:22,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:22,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2508
2673
[WARNING|trainer.py:803] 2025-04-26 20:01:24,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2524
[WARNING|trainer.py:803] 2025-04-26 20:01:24,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:01:25,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2509
[WARNING|trainer.py:803] 2025-04-26 20:01:26,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2674
[WARNING|trainer.py:803] 2025-04-26 20:01:28,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2510
[WARNING|trainer.py:803] 2025-04-26 20:01:29,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2511
2525
[WARNING|trainer.py:803] 2025-04-26 20:01:31,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:32,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2512
[WARNING|trainer.py:803] 2025-04-26 20:01:33,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2513
[WARNING|trainer.py:803] 2025-04-26 20:01:35,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2675
2526
2514
[WARNING|trainer.py:803] 2025-04-26 20:01:37,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:01:38,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:38,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2676
2515
[WARNING|trainer.py:803] 2025-04-26 20:01:40,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:01:40,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2677
[WARNING|trainer.py:803] 2025-04-26 20:01:42,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2516
2678
[WARNING|trainer.py:803] 2025-04-26 20:01:43,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:01:43,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2679
2517
[WARNING|trainer.py:803] 2025-04-26 20:01:45,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:01:45,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2680
[WARNING|trainer.py:803] 2025-04-26 20:01:47,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2527
[WARNING|trainer.py:803] 2025-04-26 20:01:48,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2518
2681
[WARNING|trainer.py:803] 2025-04-26 20:01:49,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:49,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2682
2519
[WARNING|trainer.py:803] 2025-04-26 20:01:51,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:51,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2528
2683
[WARNING|trainer.py:803] 2025-04-26 20:01:53,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:53,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2520
2684
[WARNING|trainer.py:803] 2025-04-26 20:01:54,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:01:55,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2521
2685
2529
[WARNING|trainer.py:803] 2025-04-26 20:01:57,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:01:57,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:01:57,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2686
[WARNING|trainer.py:803] 2025-04-26 20:02:00,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2687
2522
[WARNING|trainer.py:803] 2025-04-26 20:02:02,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2530
[WARNING|trainer.py:803] 2025-04-26 20:02:02,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2688
[WARNING|trainer.py:803] 2025-04-26 20:02:03,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:04,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2689
[WARNING|trainer.py:803] 2025-04-26 20:02:05,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2531
2690
[WARNING|trainer.py:803] 2025-04-26 20:02:07,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2523
[WARNING|trainer.py:803] 2025-04-26 20:02:08,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:02:08,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2532
2691
[WARNING|trainer.py:803] 2025-04-26 20:02:09,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:10,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2692
[WARNING|trainer.py:803] 2025-04-26 20:02:12,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2533
[WARNING|trainer.py:803] 2025-04-26 20:02:12,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2693
[WARNING|trainer.py:803] 2025-04-26 20:02:13,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2524
2534
[WARNING|trainer.py:803] 2025-04-26 20:02:14,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2694
[WARNING|trainer.py:803] 2025-04-26 20:02:15,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:02:15,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2695
2535
[WARNING|trainer.py:803] 2025-04-26 20:02:18,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:02:18,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2696
2536
[WARNING|trainer.py:803] 2025-04-26 20:02:20,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:20,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2525
2697
[WARNING|trainer.py:803] 2025-04-26 20:02:21,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2537
[WARNING|trainer.py:803] 2025-04-26 20:02:22,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:22,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2698
2538
[WARNING|trainer.py:803] 2025-04-26 20:02:24,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:02:24,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2699
[WARNING|trainer.py:803] 2025-04-26 20:02:26,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2526
2539
2700
[WARNING|trainer.py:803] 2025-04-26 20:02:27,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:27,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:02:28,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2540
2701
[WARNING|trainer.py:803] 2025-04-26 20:02:30,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:30,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2541
2702
[WARNING|trainer.py:803] 2025-04-26 20:02:32,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:32,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2542
2703
[WARNING|trainer.py:803] 2025-04-26 20:02:34,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:35,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2527
2704
2543
[WARNING|trainer.py:803] 2025-04-26 20:02:37,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:37,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:02:37,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2705
2544
[WARNING|trainer.py:803] 2025-04-26 20:02:40,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:02:40,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2528
2706
[WARNING|trainer.py:803] 2025-04-26 20:02:42,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:42,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2545
2707
[WARNING|trainer.py:803] 2025-04-26 20:02:44,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:02:45,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2546
[WARNING|trainer.py:803] 2025-04-26 20:02:46,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2529
2708
[WARNING|trainer.py:803] 2025-04-26 20:02:47,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:47,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2547
[WARNING|trainer.py:803] 2025-04-26 20:02:48,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2709
[WARNING|trainer.py:803] 2025-04-26 20:02:49,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2548
[WARNING|trainer.py:803] 2025-04-26 20:02:51,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2710
2530
[WARNING|trainer.py:803] 2025-04-26 20:02:52,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:52,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2549
[WARNING|trainer.py:803] 2025-04-26 20:02:53,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2711
[WARNING|trainer.py:803] 2025-04-26 20:02:54,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2550
2531
[WARNING|trainer.py:803] 2025-04-26 20:02:56,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2712
[WARNING|trainer.py:803] 2025-04-26 20:02:56,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:57,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2551
2532
[WARNING|trainer.py:803] 2025-04-26 20:02:58,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2713
[WARNING|trainer.py:803] 2025-04-26 20:02:59,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:02:59,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2552
2533
[WARNING|trainer.py:803] 2025-04-26 20:03:01,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2714
[WARNING|trainer.py:803] 2025-04-26 20:03:01,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:03:02,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2553
2534
2715
[WARNING|trainer.py:803] 2025-04-26 20:03:03,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:03:04,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:03:04,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2716
2554
2535
[WARNING|trainer.py:803] 2025-04-26 20:03:06,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:06,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:07,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2555
2717
2536
[WARNING|trainer.py:803] 2025-04-26 20:03:08,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:09,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:09,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2556
2718
2537
[WARNING|trainer.py:803] 2025-04-26 20:03:11,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:11,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:11,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2557
2538
2719
[WARNING|trainer.py:803] 2025-04-26 20:03:13,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:13,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:03:14,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2720
2539
[WARNING|trainer.py:803] 2025-04-26 20:03:16,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:03:16,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2558
[WARNING|trainer.py:803] 2025-04-26 20:03:18,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2721
2540
[WARNING|trainer.py:803] 2025-04-26 20:03:18,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:03:19,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2559
[WARNING|trainer.py:803] 2025-04-26 20:03:20,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2722
2541
[WARNING|trainer.py:803] 2025-04-26 20:03:21,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:21,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2560
2542
2723
[WARNING|trainer.py:803] 2025-04-26 20:03:23,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:23,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:23,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2561
2724
[WARNING|trainer.py:803] 2025-04-26 20:03:25,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2543
[WARNING|trainer.py:803] 2025-04-26 20:03:26,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:26,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2562
2725
[WARNING|trainer.py:803] 2025-04-26 20:03:27,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2544
[WARNING|trainer.py:803] 2025-04-26 20:03:28,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:29,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2563
2726
[WARNING|trainer.py:803] 2025-04-26 20:03:30,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:31,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2545
2727
[WARNING|trainer.py:803] 2025-04-26 20:03:32,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2564
[WARNING|trainer.py:803] 2025-04-26 20:03:33,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:34,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2546
[WARNING|trainer.py:803] 2025-04-26 20:03:35,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2728
2565
[WARNING|trainer.py:803] 2025-04-26 20:03:36,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:36,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2547
2729
[WARNING|trainer.py:803] 2025-04-26 20:03:37,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:03:38,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2566
2548
[WARNING|trainer.py:803] 2025-04-26 20:03:39,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2730
[WARNING|trainer.py:803] 2025-04-26 20:03:40,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:03:40,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2549
[WARNING|trainer.py:803] 2025-04-26 20:03:42,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2731
2567
[WARNING|trainer.py:803] 2025-04-26 20:03:43,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:43,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2550
2732
[WARNING|trainer.py:803] 2025-04-26 20:03:44,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:45,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2568
2551
[WARNING|trainer.py:803] 2025-04-26 20:03:46,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:47,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2733
[WARNING|trainer.py:803] 2025-04-26 20:03:47,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2569
2552
[WARNING|trainer.py:803] 2025-04-26 20:03:48,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2734
[WARNING|trainer.py:803] 2025-04-26 20:03:49,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2570
[WARNING|trainer.py:803] 2025-04-26 20:03:50,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:50,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2553
2735
2571
[WARNING|trainer.py:803] 2025-04-26 20:03:52,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:03:52,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:03:53,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2554
2572
2736
[WARNING|trainer.py:803] 2025-04-26 20:03:55,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:55,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:03:55,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2573
2555
2737
[WARNING|trainer.py:803] 2025-04-26 20:03:57,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:03:57,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:57,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2574
2556
2738
[WARNING|trainer.py:803] 2025-04-26 20:03:59,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:03:59,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:00,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2575
2557
2739
[WARNING|trainer.py:803] 2025-04-26 20:04:01,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:01,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:02,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2740
2576
[WARNING|trainer.py:803] 2025-04-26 20:04:04,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:04,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2558
2741
2577
[WARNING|trainer.py:803] 2025-04-26 20:04:06,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:07,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:04:07,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2559
2742
[WARNING|trainer.py:803] 2025-04-26 20:04:08,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2578
[WARNING|trainer.py:803] 2025-04-26 20:04:09,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:09,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2560
2743
[WARNING|trainer.py:803] 2025-04-26 20:04:11,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2579
[WARNING|trainer.py:803] 2025-04-26 20:04:11,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:12,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2561
2744
[WARNING|trainer.py:803] 2025-04-26 20:04:14,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:14,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2580
2562
[WARNING|trainer.py:803] 2025-04-26 20:04:15,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2745
[WARNING|trainer.py:803] 2025-04-26 20:04:16,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:16,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2581
2563
[WARNING|trainer.py:803] 2025-04-26 20:04:18,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2746
[WARNING|trainer.py:803] 2025-04-26 20:04:18,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:19,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2582
[WARNING|trainer.py:803] 2025-04-26 20:04:20,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2747
[WARNING|trainer.py:803] 2025-04-26 20:04:21,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2564
[WARNING|trainer.py:803] 2025-04-26 20:04:22,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2583
2748
[WARNING|trainer.py:803] 2025-04-26 20:04:23,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2565
[WARNING|trainer.py:803] 2025-04-26 20:04:24,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:04:24,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2584
2749
[WARNING|trainer.py:803] 2025-04-26 20:04:26,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:26,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2566
2585
[WARNING|trainer.py:803] 2025-04-26 20:04:27,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2750
[WARNING|trainer.py:803] 2025-04-26 20:04:28,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:28,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2586
2751
[WARNING|trainer.py:803] 2025-04-26 20:04:30,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2567
[WARNING|trainer.py:803] 2025-04-26 20:04:31,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:31,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2587
2752
[WARNING|trainer.py:803] 2025-04-26 20:04:33,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:04:33,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2568
[WARNING|trainer.py:803] 2025-04-26 20:04:34,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2753
2588
[WARNING|trainer.py:803] 2025-04-26 20:04:36,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:36,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2569
[WARNING|trainer.py:803] 2025-04-26 20:04:37,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2754
2589
[WARNING|trainer.py:803] 2025-04-26 20:04:38,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2570
[WARNING|trainer.py:803] 2025-04-26 20:04:39,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:04:39,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2755
2571
[WARNING|trainer.py:803] 2025-04-26 20:04:40,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2590
[WARNING|trainer.py:803] 2025-04-26 20:04:41,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:04:42,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2756
2572
[WARNING|trainer.py:803] 2025-04-26 20:04:43,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2591
[WARNING|trainer.py:803] 2025-04-26 20:04:43,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:04:44,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2757
2573
[WARNING|trainer.py:803] 2025-04-26 20:04:45,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:45,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2592
2574
[WARNING|trainer.py:803] 2025-04-26 20:04:47,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2758
[WARNING|trainer.py:803] 2025-04-26 20:04:47,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:48,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2593
2575
2759
[WARNING|trainer.py:803] 2025-04-26 20:04:50,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:04:50,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:50,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2594
2760
[WARNING|trainer.py:803] 2025-04-26 20:04:52,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2576
[WARNING|trainer.py:803] 2025-04-26 20:04:52,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:53,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2595
2761
[WARNING|trainer.py:803] 2025-04-26 20:04:54,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2577
[WARNING|trainer.py:803] 2025-04-26 20:04:55,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:55,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2596
2762
[WARNING|trainer.py:803] 2025-04-26 20:04:57,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:04:57,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2578
[WARNING|trainer.py:803] 2025-04-26 20:04:58,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2763
2597
[WARNING|trainer.py:803] 2025-04-26 20:04:59,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2579
[WARNING|trainer.py:803] 2025-04-26 20:05:00,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:00,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2598
2764
[WARNING|trainer.py:803] 2025-04-26 20:05:02,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:02,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2580
2599
[WARNING|trainer.py:803] 2025-04-26 20:05:03,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2765
[WARNING|trainer.py:803] 2025-04-26 20:05:04,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:04,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2581
2600
2766
[WARNING|trainer.py:803] 2025-04-26 20:05:06,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:05:06,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:06,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2582
2767
[WARNING|trainer.py:803] 2025-04-26 20:05:08,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2601
[WARNING|trainer.py:803] 2025-04-26 20:05:09,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:09,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2583
2768
2602
[WARNING|trainer.py:803] 2025-04-26 20:05:11,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:05:11,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:11,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2769
2584
2603
[WARNING|trainer.py:803] 2025-04-26 20:05:14,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:14,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:14,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2770
2604
2585
[WARNING|trainer.py:803] 2025-04-26 20:05:16,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:16,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:16,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2771
2586
2605
[WARNING|trainer.py:803] 2025-04-26 20:05:19,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:19,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:05:19,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2772
2606
2587
[WARNING|trainer.py:803] 2025-04-26 20:05:21,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:21,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:21,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2607
2773
2588
[WARNING|trainer.py:803] 2025-04-26 20:05:23,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:23,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:05:24,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2608
2774
[WARNING|trainer.py:803] 2025-04-26 20:05:26,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:26,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2589
[WARNING|trainer.py:803] 2025-04-26 20:05:27,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2609
2775
[WARNING|trainer.py:803] 2025-04-26 20:05:28,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:28,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2590
[WARNING|trainer.py:803] 2025-04-26 20:05:30,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2776
[WARNING|trainer.py:803] 2025-04-26 20:05:31,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2610
2591
[WARNING|trainer.py:803] 2025-04-26 20:05:32,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:32,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2777
[WARNING|trainer.py:803] 2025-04-26 20:05:33,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2592
2778
[WARNING|trainer.py:803] 2025-04-26 20:05:35,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:36,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2611
2593
2779
[WARNING|trainer.py:803] 2025-04-26 20:05:38,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:38,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:38,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2612
2594
2780
[WARNING|trainer.py:803] 2025-04-26 20:05:40,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:05:40,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:41,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2613
2595
2781
[WARNING|trainer.py:803] 2025-04-26 20:05:42,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:42,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:43,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2614
2596
2782
[WARNING|trainer.py:803] 2025-04-26 20:05:44,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:45,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:45,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2615
[WARNING|trainer.py:803] 2025-04-26 20:05:46,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2783
2597
[WARNING|trainer.py:803] 2025-04-26 20:05:48,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2616
[WARNING|trainer.py:803] 2025-04-26 20:05:48,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:49,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2784
2598
[WARNING|trainer.py:803] 2025-04-26 20:05:50,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2617
[WARNING|trainer.py:803] 2025-04-26 20:05:50,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:51,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2599
2785
[WARNING|trainer.py:803] 2025-04-26 20:05:52,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:52,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2618
[WARNING|trainer.py:803] 2025-04-26 20:05:54,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2786
2600
[WARNING|trainer.py:803] 2025-04-26 20:05:55,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:05:55,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2619
[WARNING|trainer.py:803] 2025-04-26 20:05:56,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2787
2601
[WARNING|trainer.py:803] 2025-04-26 20:05:57,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:05:58,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2620
[WARNING|trainer.py:803] 2025-04-26 20:05:58,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2788
2602
[WARNING|trainer.py:803] 2025-04-26 20:06:00,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2621
[WARNING|trainer.py:803] 2025-04-26 20:06:00,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:06:00,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2789
2603
[WARNING|trainer.py:803] 2025-04-26 20:06:02,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2622
[WARNING|trainer.py:803] 2025-04-26 20:06:03,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:03,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2790
2604
[WARNING|trainer.py:803] 2025-04-26 20:06:04,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:05,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2623
2791
[WARNING|trainer.py:803] 2025-04-26 20:06:06,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:06,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2605
[WARNING|trainer.py:803] 2025-04-26 20:06:08,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2792
2624
[WARNING|trainer.py:803] 2025-04-26 20:06:09,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2606
[WARNING|trainer.py:803] 2025-04-26 20:06:09,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:06:10,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2793
2625
2607
[WARNING|trainer.py:803] 2025-04-26 20:06:11,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:12,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:12,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2794
2608
2626
[WARNING|trainer.py:803] 2025-04-26 20:06:14,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:14,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:06:15,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2795
2609
2627
[WARNING|trainer.py:803] 2025-04-26 20:06:16,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:17,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:17,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2796
2628
[WARNING|trainer.py:803] 2025-04-26 20:06:19,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:19,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2610
2629
2797
[WARNING|trainer.py:803] 2025-04-26 20:06:21,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:21,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:06:21,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2630
2798
[WARNING|trainer.py:803] 2025-04-26 20:06:24,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:06:24,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2799
2611
2631
[WARNING|trainer.py:803] 2025-04-26 20:06:26,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:27,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:06:27,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2800
2632
2612
[WARNING|trainer.py:803] 2025-04-26 20:06:29,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:06:29,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:29,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2613
2633
2801
[WARNING|trainer.py:803] 2025-04-26 20:06:31,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:31,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:31,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2614
2634
2802
[WARNING|trainer.py:803] 2025-04-26 20:06:34,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:06:34,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:34,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2615
2635
2803
[WARNING|trainer.py:803] 2025-04-26 20:06:36,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:36,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:06:36,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2616
2636
2804
[WARNING|trainer.py:803] 2025-04-26 20:06:38,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:38,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:06:39,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2617
2637
2805
[WARNING|trainer.py:803] 2025-04-26 20:06:40,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:41,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:06:41,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2638
2618
2806
[WARNING|trainer.py:803] 2025-04-26 20:06:43,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:43,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:43,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2639
2619
[WARNING|trainer.py:803] 2025-04-26 20:06:45,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2807
[WARNING|trainer.py:803] 2025-04-26 20:06:45,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:46,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2640
[WARNING|trainer.py:803] 2025-04-26 20:06:47,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2620
2808
[WARNING|trainer.py:803] 2025-04-26 20:06:48,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:06:48,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2641
[WARNING|trainer.py:803] 2025-04-26 20:06:49,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2621
2809
[WARNING|trainer.py:803] 2025-04-26 20:06:50,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:06:50,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2642
2622
[WARNING|trainer.py:803] 2025-04-26 20:06:51,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2810
[WARNING|trainer.py:803] 2025-04-26 20:06:52,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:53,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2623
2811
[WARNING|trainer.py:803] 2025-04-26 20:06:55,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2643
[WARNING|trainer.py:803] 2025-04-26 20:06:55,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:06:56,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2812
[WARNING|trainer.py:803] 2025-04-26 20:06:58,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2624
[WARNING|trainer.py:803] 2025-04-26 20:06:59,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2813
2644
2625
[WARNING|trainer.py:803] 2025-04-26 20:07:00,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:07:01,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:01,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2814
[WARNING|trainer.py:803] 2025-04-26 20:07:03,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2626
[WARNING|trainer.py:803] 2025-04-26 20:07:04,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2815
2645
[WARNING|trainer.py:803] 2025-04-26 20:07:05,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2627
[WARNING|trainer.py:803] 2025-04-26 20:07:06,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:06,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2816
2628
[WARNING|trainer.py:803] 2025-04-26 20:07:07,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:07:08,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2817
2646
[WARNING|trainer.py:803] 2025-04-26 20:07:10,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2629
[WARNING|trainer.py:803] 2025-04-26 20:07:10,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:07:11,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2818
2630
[WARNING|trainer.py:803] 2025-04-26 20:07:12,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:13,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2647
2819
[WARNING|trainer.py:803] 2025-04-26 20:07:15,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:07:15,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2631
[WARNING|trainer.py:803] 2025-04-26 20:07:16,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2820
[WARNING|trainer.py:803] 2025-04-26 20:07:17,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2632
2648
[WARNING|trainer.py:803] 2025-04-26 20:07:18,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:19,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2821
2633
[WARNING|trainer.py:803] 2025-04-26 20:07:20,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:07:21,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2822
2634
[WARNING|trainer.py:803] 2025-04-26 20:07:22,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2649
[WARNING|trainer.py:803] 2025-04-26 20:07:23,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:23,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2823
2635
[WARNING|trainer.py:803] 2025-04-26 20:07:25,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:25,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2824
2636
[WARNING|trainer.py:803] 2025-04-26 20:07:27,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:07:28,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2650
[WARNING|trainer.py:803] 2025-04-26 20:07:29,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2825
2637
[WARNING|trainer.py:803] 2025-04-26 20:07:30,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:07:30,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2651
[WARNING|trainer.py:803] 2025-04-26 20:07:31,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2638
2826
[WARNING|trainer.py:803] 2025-04-26 20:07:32,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2652
[WARNING|trainer.py:803] 2025-04-26 20:07:32,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:33,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2639
2827
[WARNING|trainer.py:803] 2025-04-26 20:07:34,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2653
[WARNING|trainer.py:803] 2025-04-26 20:07:35,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:35,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2640
[WARNING|trainer.py:803] 2025-04-26 20:07:36,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2828
2654
[WARNING|trainer.py:803] 2025-04-26 20:07:37,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:07:37,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2641
[WARNING|trainer.py:803] 2025-04-26 20:07:38,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2829
2655
[WARNING|trainer.py:803] 2025-04-26 20:07:39,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:40,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2642
2656
[WARNING|trainer.py:803] 2025-04-26 20:07:41,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2830
[WARNING|trainer.py:803] 2025-04-26 20:07:42,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:42,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2657
2831
[WARNING|trainer.py:803] 2025-04-26 20:07:44,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:44,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2643
[WARNING|trainer.py:803] 2025-04-26 20:07:45,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2658
2832
[WARNING|trainer.py:803] 2025-04-26 20:07:47,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:07:47,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2659
2833
[WARNING|trainer.py:803] 2025-04-26 20:07:49,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2644
[WARNING|trainer.py:803] 2025-04-26 20:07:49,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2660
[WARNING|trainer.py:803] 2025-04-26 20:07:50,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:51,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2834
[WARNING|trainer.py:803] 2025-04-26 20:07:52,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2661
[WARNING|trainer.py:803] 2025-04-26 20:07:53,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2835
2645
[WARNING|trainer.py:803] 2025-04-26 20:07:54,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2662
[WARNING|trainer.py:803] 2025-04-26 20:07:55,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:55,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2836
[WARNING|trainer.py:803] 2025-04-26 20:07:57,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2663
[WARNING|trainer.py:803] 2025-04-26 20:07:58,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2837
2646
[WARNING|trainer.py:803] 2025-04-26 20:07:59,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:07:59,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2664
2838
[WARNING|trainer.py:803] 2025-04-26 20:08:01,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:02,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2665
2647
2839
[WARNING|trainer.py:803] 2025-04-26 20:08:03,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:04,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:08:04,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2840
[WARNING|trainer.py:803] 2025-04-26 20:08:07,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2648
[WARNING|trainer.py:803] 2025-04-26 20:08:08,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2666
2841
[WARNING|trainer.py:803] 2025-04-26 20:08:09,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:08:09,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2667
2842
[WARNING|trainer.py:803] 2025-04-26 20:08:11,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:11,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2649
[WARNING|trainer.py:803] 2025-04-26 20:08:12,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2668
2843
[WARNING|trainer.py:803] 2025-04-26 20:08:13,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:14,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2669
[WARNING|trainer.py:803] 2025-04-26 20:08:15,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2844
[WARNING|trainer.py:803] 2025-04-26 20:08:16,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2670
2650
[WARNING|trainer.py:803] 2025-04-26 20:08:17,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:18,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2845
[WARNING|trainer.py:803] 2025-04-26 20:08:19,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2671
2651
[WARNING|trainer.py:803] 2025-04-26 20:08:20,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:20,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2846
2652
[WARNING|trainer.py:803] 2025-04-26 20:08:21,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2672
[WARNING|trainer.py:803] 2025-04-26 20:08:22,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:22,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2847
2653
[WARNING|trainer.py:803] 2025-04-26 20:08:24,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:08:24,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2673
[WARNING|trainer.py:803] 2025-04-26 20:08:25,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2848
2654
[WARNING|trainer.py:803] 2025-04-26 20:08:26,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:26,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2655
2674
2849
[WARNING|trainer.py:803] 2025-04-26 20:08:28,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:08:28,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:08:28,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2656
2850
[WARNING|trainer.py:803] 2025-04-26 20:08:31,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:31,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2657
2851
[WARNING|trainer.py:803] 2025-04-26 20:08:33,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:33,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2658
2852
[WARNING|trainer.py:803] 2025-04-26 20:08:36,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:36,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2659
2675
2853
[WARNING|trainer.py:803] 2025-04-26 20:08:38,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:38,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:08:38,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2660
2676
[WARNING|trainer.py:803] 2025-04-26 20:08:40,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2854
[WARNING|trainer.py:803] 2025-04-26 20:08:40,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:41,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2661
2677
[WARNING|trainer.py:803] 2025-04-26 20:08:42,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2855
[WARNING|trainer.py:803] 2025-04-26 20:08:42,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:08:43,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2662
2678
[WARNING|trainer.py:803] 2025-04-26 20:08:44,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:44,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2856
2679
[WARNING|trainer.py:803] 2025-04-26 20:08:46,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2663
[WARNING|trainer.py:803] 2025-04-26 20:08:46,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:08:47,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2857
2680
[WARNING|trainer.py:803] 2025-04-26 20:08:48,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:48,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2664
2858
[WARNING|trainer.py:803] 2025-04-26 20:08:50,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2681
[WARNING|trainer.py:803] 2025-04-26 20:08:51,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:08:51,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2665
[WARNING|trainer.py:803] 2025-04-26 20:08:52,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2859
2682
[WARNING|trainer.py:803] 2025-04-26 20:08:53,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:08:54,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2683
2860
[WARNING|trainer.py:803] 2025-04-26 20:08:56,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:56,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2666
2684
2861
[WARNING|trainer.py:803] 2025-04-26 20:08:58,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:08:58,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:08:58,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2667
2685
[WARNING|trainer.py:803] 2025-04-26 20:09:00,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2862
[WARNING|trainer.py:803] 2025-04-26 20:09:00,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:09:01,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2668
[WARNING|trainer.py:803] 2025-04-26 20:09:02,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2686
2863
[WARNING|trainer.py:803] 2025-04-26 20:09:03,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:03,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2669
[WARNING|trainer.py:803] 2025-04-26 20:09:04,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2687
2864
[WARNING|trainer.py:803] 2025-04-26 20:09:05,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2670
[WARNING|trainer.py:803] 2025-04-26 20:09:06,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:06,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2688
2865
[WARNING|trainer.py:803] 2025-04-26 20:09:07,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2671
[WARNING|trainer.py:803] 2025-04-26 20:09:08,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2689
[WARNING|trainer.py:803] 2025-04-26 20:09:09,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:09:09,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2866
2672
[WARNING|trainer.py:803] 2025-04-26 20:09:10,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2690
[WARNING|trainer.py:803] 2025-04-26 20:09:11,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:12,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2867
2673
[WARNING|trainer.py:803] 2025-04-26 20:09:13,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2691
[WARNING|trainer.py:803] 2025-04-26 20:09:13,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:14,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2868
[WARNING|trainer.py:803] 2025-04-26 20:09:15,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2692
2674
[WARNING|trainer.py:803] 2025-04-26 20:09:16,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:17,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2869
2693
[WARNING|trainer.py:803] 2025-04-26 20:09:18,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:09:18,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2870
2694
[WARNING|trainer.py:803] 2025-04-26 20:09:20,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:21,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2871
2695
[WARNING|trainer.py:803] 2025-04-26 20:09:23,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:23,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2872
2696
[WARNING|trainer.py:803] 2025-04-26 20:09:25,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:26,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2675
2873
[WARNING|trainer.py:803] 2025-04-26 20:09:27,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2697
[WARNING|trainer.py:803] 2025-04-26 20:09:28,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:28,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2676
[WARNING|trainer.py:803] 2025-04-26 20:09:29,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2874
2698
[WARNING|trainer.py:803] 2025-04-26 20:09:30,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:30,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2677
[WARNING|trainer.py:803] 2025-04-26 20:09:31,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2699
2875
2678
[WARNING|trainer.py:803] 2025-04-26 20:09:32,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:33,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:33,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2700
2876
2679
[WARNING|trainer.py:803] 2025-04-26 20:09:35,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:09:35,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:35,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2680
2701
2877
[WARNING|trainer.py:803] 2025-04-26 20:09:37,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:37,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:37,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2878
2681
2702
[WARNING|trainer.py:803] 2025-04-26 20:09:40,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:40,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:09:40,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2879
2682
2703
[WARNING|trainer.py:803] 2025-04-26 20:09:42,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:09:42,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:43,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2683
2880
2704
[WARNING|trainer.py:803] 2025-04-26 20:09:44,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:09:45,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:45,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2684
[WARNING|trainer.py:803] 2025-04-26 20:09:46,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2881
2705
[WARNING|trainer.py:803] 2025-04-26 20:09:47,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:48,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2685
2882
[WARNING|trainer.py:803] 2025-04-26 20:09:49,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2706
[WARNING|trainer.py:803] 2025-04-26 20:09:50,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:50,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2686
2883
[WARNING|trainer.py:803] 2025-04-26 20:09:52,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2707
[WARNING|trainer.py:803] 2025-04-26 20:09:52,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:53,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2687
2884
[WARNING|trainer.py:803] 2025-04-26 20:09:54,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2708
[WARNING|trainer.py:803] 2025-04-26 20:09:55,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:09:55,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2688
[WARNING|trainer.py:803] 2025-04-26 20:09:56,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2885
2709
2689
[WARNING|trainer.py:803] 2025-04-26 20:09:57,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:09:58,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:09:58,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2886
2710
[WARNING|trainer.py:803] 2025-04-26 20:10:00,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2690
[WARNING|trainer.py:803] 2025-04-26 20:10:00,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:01,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2887
2691
[WARNING|trainer.py:803] 2025-04-26 20:10:02,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2711
[WARNING|trainer.py:803] 2025-04-26 20:10:03,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:03,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2888
2692
[WARNING|trainer.py:803] 2025-04-26 20:10:05,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2712
[WARNING|trainer.py:803] 2025-04-26 20:10:05,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:06,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2889
2693
[WARNING|trainer.py:803] 2025-04-26 20:10:07,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2713
[WARNING|trainer.py:803] 2025-04-26 20:10:07,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:08,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2694
2890
[WARNING|trainer.py:803] 2025-04-26 20:10:10,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:10,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2714
[WARNING|trainer.py:803] 2025-04-26 20:10:11,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2695
2891
[WARNING|trainer.py:803] 2025-04-26 20:10:12,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:12,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2715
[WARNING|trainer.py:803] 2025-04-26 20:10:13,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2696
2892
[WARNING|trainer.py:803] 2025-04-26 20:10:15,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:15,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2716
[WARNING|trainer.py:803] 2025-04-26 20:10:16,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2697
2893
[WARNING|trainer.py:803] 2025-04-26 20:10:17,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:17,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2717
[WARNING|trainer.py:803] 2025-04-26 20:10:18,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2698
2894
[WARNING|trainer.py:803] 2025-04-26 20:10:19,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:20,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2718
2699
[WARNING|trainer.py:803] 2025-04-26 20:10:21,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2895
[WARNING|trainer.py:803] 2025-04-26 20:10:21,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:22,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2719
2700
[WARNING|trainer.py:803] 2025-04-26 20:10:24,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2896
[WARNING|trainer.py:803] 2025-04-26 20:10:24,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:10:25,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2720
2701
2897
[WARNING|trainer.py:803] 2025-04-26 20:10:26,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:10:26,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:27,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2721
2702
2898
[WARNING|trainer.py:803] 2025-04-26 20:10:29,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:10:29,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:29,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2722
2703
2899
[WARNING|trainer.py:803] 2025-04-26 20:10:31,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:31,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:32,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2723
2704
2900
[WARNING|trainer.py:803] 2025-04-26 20:10:34,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:34,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:34,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2705
2901
2724
[WARNING|trainer.py:803] 2025-04-26 20:10:37,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:37,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:37,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2902
2725
2706
[WARNING|trainer.py:803] 2025-04-26 20:10:39,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:39,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:39,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2903
2726
2707
[WARNING|trainer.py:803] 2025-04-26 20:10:42,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:42,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:42,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2904
2727
2708
[WARNING|trainer.py:803] 2025-04-26 20:10:44,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:10:44,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:45,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2905
2728
2709
[WARNING|trainer.py:803] 2025-04-26 20:10:47,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:10:47,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:47,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2906
2710
2729
[WARNING|trainer.py:803] 2025-04-26 20:10:49,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:50,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:50,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2907
2711
2730
[WARNING|trainer.py:803] 2025-04-26 20:10:52,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:10:52,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:52,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2908
2712
2731
[WARNING|trainer.py:803] 2025-04-26 20:10:54,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:10:55,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:55,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2909
2713
2732
[WARNING|trainer.py:803] 2025-04-26 20:10:57,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:10:57,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:10:57,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2910
2714
[WARNING|trainer.py:803] 2025-04-26 20:10:59,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2733
[WARNING|trainer.py:803] 2025-04-26 20:11:00,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:00,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2911
[WARNING|trainer.py:803] 2025-04-26 20:11:01,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2715
2734
[WARNING|trainer.py:803] 2025-04-26 20:11:02,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:11:02,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2912
[WARNING|trainer.py:803] 2025-04-26 20:11:04,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2716
2735
[WARNING|trainer.py:803] 2025-04-26 20:11:05,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:05,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2913
[WARNING|trainer.py:803] 2025-04-26 20:11:06,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2717
2736
[WARNING|trainer.py:803] 2025-04-26 20:11:07,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:08,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2914
[WARNING|trainer.py:803] 2025-04-26 20:11:09,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2718
2737
[WARNING|trainer.py:803] 2025-04-26 20:11:10,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:10,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2915
[WARNING|trainer.py:803] 2025-04-26 20:11:11,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2719
2738
[WARNING|trainer.py:803] 2025-04-26 20:11:12,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:13,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2916
[WARNING|trainer.py:803] 2025-04-26 20:11:13,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2720
2739
2917
[WARNING|trainer.py:803] 2025-04-26 20:11:15,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:11:15,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:16,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2740
2721
2918
[WARNING|trainer.py:803] 2025-04-26 20:11:18,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:18,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:11:18,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2741
2722
2919
[WARNING|trainer.py:803] 2025-04-26 20:11:20,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:11:20,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:21,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2742
2723
2920
[WARNING|trainer.py:803] 2025-04-26 20:11:23,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:23,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:23,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2743
2724
2921
[WARNING|trainer.py:803] 2025-04-26 20:11:25,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:25,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:26,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2744
2725
2922
[WARNING|trainer.py:803] 2025-04-26 20:11:28,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:28,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:28,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2745
2923
2726
[WARNING|trainer.py:803] 2025-04-26 20:11:30,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:30,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:31,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2924
2746
2727
[WARNING|trainer.py:803] 2025-04-26 20:11:33,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:33,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:11:33,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2925
2747
2728
[WARNING|trainer.py:803] 2025-04-26 20:11:35,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:36,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:36,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2926
2748
2729
[WARNING|trainer.py:803] 2025-04-26 20:11:38,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:38,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:11:39,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2927
2749
[WARNING|trainer.py:803] 2025-04-26 20:11:40,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2730
[WARNING|trainer.py:803] 2025-04-26 20:11:41,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:41,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2928
[WARNING|trainer.py:803] 2025-04-26 20:11:42,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2750
2731
[WARNING|trainer.py:803] 2025-04-26 20:11:43,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:44,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2929
[WARNING|trainer.py:803] 2025-04-26 20:11:45,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2751
2732
[WARNING|trainer.py:803] 2025-04-26 20:11:46,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:46,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2930
[WARNING|trainer.py:803] 2025-04-26 20:11:47,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2752
2733
[WARNING|trainer.py:803] 2025-04-26 20:11:49,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:49,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2931
[WARNING|trainer.py:803] 2025-04-26 20:11:50,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2753
2734
[WARNING|trainer.py:803] 2025-04-26 20:11:51,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:51,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2932
[WARNING|trainer.py:803] 2025-04-26 20:11:52,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2754
2735
[WARNING|trainer.py:803] 2025-04-26 20:11:54,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2933
[WARNING|trainer.py:803] 2025-04-26 20:11:54,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:11:55,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2755
2736
2934
[WARNING|trainer.py:803] 2025-04-26 20:11:56,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:11:56,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:11:57,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2756
2737
2935
[WARNING|trainer.py:803] 2025-04-26 20:11:59,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:11:59,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:00,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2757
2738
2936
[WARNING|trainer.py:803] 2025-04-26 20:12:01,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:01,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:02,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2758
2739
2937
[WARNING|trainer.py:803] 2025-04-26 20:12:04,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:04,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:05,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2759
2740
2938
[WARNING|trainer.py:803] 2025-04-26 20:12:06,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:07,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:07,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2760
2741
2939
[WARNING|trainer.py:803] 2025-04-26 20:12:09,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:09,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:09,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2761
2742
2940
[WARNING|trainer.py:803] 2025-04-26 20:12:11,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:12,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:12,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2762
2941
2743
[WARNING|trainer.py:803] 2025-04-26 20:12:14,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:14,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:14,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2763
2942
2744
[WARNING|trainer.py:803] 2025-04-26 20:12:16,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:12:17,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:17,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2764
2943
2745
[WARNING|trainer.py:803] 2025-04-26 20:12:19,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:19,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:19,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2944
2765
2746
[WARNING|trainer.py:803] 2025-04-26 20:12:22,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:22,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:22,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2945
2766
2747
[WARNING|trainer.py:803] 2025-04-26 20:12:24,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:24,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:24,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2946
2767
2748
[WARNING|trainer.py:803] 2025-04-26 20:12:27,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:27,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:12:27,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
2947
2768
2749
[WARNING|trainer.py:803] 2025-04-26 20:12:29,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:12:29,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:30,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2948
2769
2750
[WARNING|trainer.py:803] 2025-04-26 20:12:32,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:32,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:32,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2949
2770
2751
[WARNING|trainer.py:803] 2025-04-26 20:12:34,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:35,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:35,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2950
2771
2752
[WARNING|trainer.py:803] 2025-04-26 20:12:37,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:37,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:37,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2951
2772
[WARNING|trainer.py:803] 2025-04-26 20:12:39,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2753
[WARNING|trainer.py:803] 2025-04-26 20:12:40,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:40,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2952
2773
[WARNING|trainer.py:803] 2025-04-26 20:12:41,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2754
[WARNING|trainer.py:803] 2025-04-26 20:12:42,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:12:42,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2953
[WARNING|trainer.py:803] 2025-04-26 20:12:44,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2774
2755
[WARNING|trainer.py:803] 2025-04-26 20:12:45,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:45,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2954
[WARNING|trainer.py:803] 2025-04-26 20:12:46,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2775
2756
[WARNING|trainer.py:803] 2025-04-26 20:12:47,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:48,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2955
[WARNING|trainer.py:803] 2025-04-26 20:12:49,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2776
2757
[WARNING|trainer.py:803] 2025-04-26 20:12:50,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:50,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2956
[WARNING|trainer.py:803] 2025-04-26 20:12:51,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2758
2777
2957
[WARNING|trainer.py:803] 2025-04-26 20:12:53,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:53,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:53,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2778
2759
2958
[WARNING|trainer.py:803] 2025-04-26 20:12:55,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:55,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:56,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2779
2760
2959
[WARNING|trainer.py:803] 2025-04-26 20:12:58,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:12:58,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:12:58,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2761
2780
2960
[WARNING|trainer.py:803] 2025-04-26 20:13:00,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:00,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:01,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2762
2781
2961
[WARNING|trainer.py:803] 2025-04-26 20:13:03,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:03,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:03,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2763
2782
2962
[WARNING|trainer.py:803] 2025-04-26 20:13:05,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:05,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:05,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2963
2764
2783
[WARNING|trainer.py:803] 2025-04-26 20:13:08,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:13:08,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:13:08,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2964
2784
2765
[WARNING|trainer.py:803] 2025-04-26 20:13:10,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:10,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:11,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2965
2785
2766
[WARNING|trainer.py:803] 2025-04-26 20:13:13,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:13,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:13,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2966
2767
2786
[WARNING|trainer.py:803] 2025-04-26 20:13:15,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:13:16,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:16,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2967
2787
2768
[WARNING|trainer.py:803] 2025-04-26 20:13:18,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:18,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:13:18,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2968
2788
2769
[WARNING|trainer.py:803] 2025-04-26 20:13:20,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:21,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:21,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2969
2789
2770
[WARNING|trainer.py:803] 2025-04-26 20:13:23,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:23,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:13:23,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2970
2790
2771
[WARNING|trainer.py:803] 2025-04-26 20:13:25,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:26,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:26,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2971
2791
2772
[WARNING|trainer.py:803] 2025-04-26 20:13:28,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:28,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:13:28,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2972
2773
2792
[WARNING|trainer.py:803] 2025-04-26 20:13:30,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:31,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:13:31,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2973
2774
2793
[WARNING|trainer.py:803] 2025-04-26 20:13:33,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:34,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:34,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2974
[WARNING|trainer.py:803] 2025-04-26 20:13:35,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2775
2794
[WARNING|trainer.py:803] 2025-04-26 20:13:36,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:36,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2975
[WARNING|trainer.py:803] 2025-04-26 20:13:38,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2795
2776
[WARNING|trainer.py:803] 2025-04-26 20:13:39,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:39,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2976
[WARNING|trainer.py:803] 2025-04-26 20:13:40,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2796
2777
[WARNING|trainer.py:803] 2025-04-26 20:13:41,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:41,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2977
[WARNING|trainer.py:803] 2025-04-26 20:13:43,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2778
2797
[WARNING|trainer.py:803] 2025-04-26 20:13:44,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:44,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2978
[WARNING|trainer.py:803] 2025-04-26 20:13:45,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2779
2798
[WARNING|trainer.py:803] 2025-04-26 20:13:46,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:13:47,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2979
[WARNING|trainer.py:803] 2025-04-26 20:13:47,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2780
2799
[WARNING|trainer.py:803] 2025-04-26 20:13:49,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2980
[WARNING|trainer.py:803] 2025-04-26 20:13:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:50,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2781
2800
[WARNING|trainer.py:803] 2025-04-26 20:13:51,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2981
[WARNING|trainer.py:803] 2025-04-26 20:13:52,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:13:52,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2782
2801
2982
[WARNING|trainer.py:803] 2025-04-26 20:13:54,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:54,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:55,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2783
2802
2983
[WARNING|trainer.py:803] 2025-04-26 20:13:57,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:57,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:13:57,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2784
2984
2803
[WARNING|trainer.py:803] 2025-04-26 20:13:59,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:00,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:00,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2785
2985
2804
[WARNING|trainer.py:803] 2025-04-26 20:14:02,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:02,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:02,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2986
2786
2805
[WARNING|trainer.py:803] 2025-04-26 20:14:04,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:14:04,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:05,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2987
2787
2806
[WARNING|trainer.py:803] 2025-04-26 20:14:07,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:14:07,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:14:07,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2988
2788
2807
[WARNING|trainer.py:803] 2025-04-26 20:14:09,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:10,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:10,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2989
2789
2808
[WARNING|trainer.py:803] 2025-04-26 20:14:12,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:12,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:14:12,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2990
2790
2809
[WARNING|trainer.py:803] 2025-04-26 20:14:14,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:15,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:15,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2991
2791
[WARNING|trainer.py:803] 2025-04-26 20:14:17,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2810
[WARNING|trainer.py:803] 2025-04-26 20:14:17,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:14:17,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2992
2792
[WARNING|trainer.py:803] 2025-04-26 20:14:19,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2811
[WARNING|trainer.py:803] 2025-04-26 20:14:20,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:20,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2993
[WARNING|trainer.py:803] 2025-04-26 20:14:21,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2793
2812
[WARNING|trainer.py:803] 2025-04-26 20:14:22,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:23,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2994
[WARNING|trainer.py:803] 2025-04-26 20:14:24,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2794
2813
[WARNING|trainer.py:803] 2025-04-26 20:14:25,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:25,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2995
[WARNING|trainer.py:803] 2025-04-26 20:14:26,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2795
2814
[WARNING|trainer.py:803] 2025-04-26 20:14:28,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:28,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2996
[WARNING|trainer.py:803] 2025-04-26 20:14:29,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2796
2815
[WARNING|trainer.py:803] 2025-04-26 20:14:30,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2997
[WARNING|trainer.py:803] 2025-04-26 20:14:30,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:31,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2797
2816
2998
[WARNING|trainer.py:803] 2025-04-26 20:14:33,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:33,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:14:34,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2798
2817
2999
[WARNING|trainer.py:803] 2025-04-26 20:14:35,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:36,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:14:36,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2799
2818
3000
[WARNING|trainer.py:803] 2025-04-26 20:14:38,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:38,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:39,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3001
2800
2819
[WARNING|trainer.py:803] 2025-04-26 20:14:40,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:41,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:41,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3002
[WARNING|trainer.py:803] 2025-04-26 20:14:42,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2801
2820
[WARNING|trainer.py:803] 2025-04-26 20:14:43,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3003
[WARNING|trainer.py:803] 2025-04-26 20:14:44,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:44,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3004
2802
2821
[WARNING|trainer.py:803] 2025-04-26 20:14:46,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:46,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:46,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3005
2803
[WARNING|trainer.py:803] 2025-04-26 20:14:48,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2822
[WARNING|trainer.py:803] 2025-04-26 20:14:49,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3006
[WARNING|trainer.py:803] 2025-04-26 20:14:49,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:50,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2804
2823
3007
[WARNING|trainer.py:803] 2025-04-26 20:14:51,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:52,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:14:52,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3008
2805
2824
[WARNING|trainer.py:803] 2025-04-26 20:14:54,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:14:54,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:54,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3009
[WARNING|trainer.py:803] 2025-04-26 20:14:55,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2806
2825
3010
[WARNING|trainer.py:803] 2025-04-26 20:14:57,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:57,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:14:57,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2807
3011
2826
[WARNING|trainer.py:803] 2025-04-26 20:14:59,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:59,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:14:59,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3012
2808
[WARNING|trainer.py:803] 2025-04-26 20:15:01,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2827
[WARNING|trainer.py:803] 2025-04-26 20:15:02,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3013
[WARNING|trainer.py:803] 2025-04-26 20:15:02,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:03,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2809
2828
3014
[WARNING|trainer.py:803] 2025-04-26 20:15:04,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:04,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:05,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3015
2810
2829
[WARNING|trainer.py:803] 2025-04-26 20:15:06,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:07,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:07,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3016
[WARNING|trainer.py:803] 2025-04-26 20:15:08,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2811
2830
3017
[WARNING|trainer.py:803] 2025-04-26 20:15:09,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:10,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:10,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3018
2812
2831
[WARNING|trainer.py:803] 2025-04-26 20:15:12,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:12,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:12,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3019
2813
[WARNING|trainer.py:803] 2025-04-26 20:15:14,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2832
[WARNING|trainer.py:803] 2025-04-26 20:15:15,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:15,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3020
[WARNING|trainer.py:803] 2025-04-26 20:15:16,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2814
2833
3021
[WARNING|trainer.py:803] 2025-04-26 20:15:17,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:17,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:18,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3022
2815
2834
[WARNING|trainer.py:803] 2025-04-26 20:15:19,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:20,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:20,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3023
[WARNING|trainer.py:803] 2025-04-26 20:15:21,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2816
2835
3024
[WARNING|trainer.py:803] 2025-04-26 20:15:22,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:23,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:15:23,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3025
2817
2836
[WARNING|trainer.py:803] 2025-04-26 20:15:25,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:25,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:15:25,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3026
[WARNING|trainer.py:803] 2025-04-26 20:15:27,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2818
2837
[WARNING|trainer.py:803] 2025-04-26 20:15:28,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3027
[WARNING|trainer.py:803] 2025-04-26 20:15:28,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:15:29,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2819
2838
3028
[WARNING|trainer.py:803] 2025-04-26 20:15:30,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:30,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:31,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3029
2820
2839
[WARNING|trainer.py:803] 2025-04-26 20:15:32,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:33,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:33,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3030
[WARNING|trainer.py:803] 2025-04-26 20:15:34,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2821
2840
3031
[WARNING|trainer.py:803] 2025-04-26 20:15:35,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:36,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:36,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3032
2822
2841
[WARNING|trainer.py:803] 2025-04-26 20:15:38,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:38,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:38,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3033
2823
2842
[WARNING|trainer.py:803] 2025-04-26 20:15:40,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:41,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:15:41,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3034
[WARNING|trainer.py:803] 2025-04-26 20:15:42,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2824
2843
3035
[WARNING|trainer.py:803] 2025-04-26 20:15:43,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:43,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:44,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3036
2844
2825
[WARNING|trainer.py:803] 2025-04-26 20:15:46,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:46,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:46,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3037
[WARNING|trainer.py:803] 2025-04-26 20:15:48,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2826
2845
3038
[WARNING|trainer.py:803] 2025-04-26 20:15:49,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:49,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:49,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3039
2846
2827
[WARNING|trainer.py:803] 2025-04-26 20:15:51,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:51,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:51,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3040
2828
[WARNING|trainer.py:803] 2025-04-26 20:15:53,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2847
[WARNING|trainer.py:803] 2025-04-26 20:15:54,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:15:54,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3041
[WARNING|trainer.py:803] 2025-04-26 20:15:55,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2829
2848
3042
[WARNING|trainer.py:803] 2025-04-26 20:15:56,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:57,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:57,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3043
2849
2830
[WARNING|trainer.py:803] 2025-04-26 20:15:59,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:15:59,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:15:59,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3044
[WARNING|trainer.py:803] 2025-04-26 20:16:01,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2831
2850
3045
[WARNING|trainer.py:803] 2025-04-26 20:16:02,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:02,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:02,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3046
2851
2832
[WARNING|trainer.py:803] 2025-04-26 20:16:04,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:04,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:04,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3047
2833
2852
[WARNING|trainer.py:803] 2025-04-26 20:16:06,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:07,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:07,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3048
[WARNING|trainer.py:803] 2025-04-26 20:16:08,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2834
2853
3049
[WARNING|trainer.py:803] 2025-04-26 20:16:10,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:10,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:10,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3050
2835
2854
[WARNING|trainer.py:803] 2025-04-26 20:16:12,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:12,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:12,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3051
[WARNING|trainer.py:803] 2025-04-26 20:16:14,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2836
2855
3052
[WARNING|trainer.py:803] 2025-04-26 20:16:15,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:15,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:15,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3053
2837
2856
[WARNING|trainer.py:803] 2025-04-26 20:16:17,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:17,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:16:18,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
3054
[WARNING|trainer.py:803] 2025-04-26 20:16:19,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2838
2857
[WARNING|trainer.py:803] 2025-04-26 20:16:20,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3055
[WARNING|trainer.py:803] 2025-04-26 20:16:20,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:21,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2839
3056
2858
[WARNING|trainer.py:803] 2025-04-26 20:16:22,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:16:23,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:16:23,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3057
2840
[WARNING|trainer.py:803] 2025-04-26 20:16:25,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2859
[WARNING|trainer.py:803] 2025-04-26 20:16:25,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:26,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3058
[WARNING|trainer.py:803] 2025-04-26 20:16:27,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2841
2860
3059
[WARNING|trainer.py:803] 2025-04-26 20:16:28,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:28,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:28,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3060
2842
2861
[WARNING|trainer.py:803] 2025-04-26 20:16:30,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:30,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:31,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3061
[WARNING|trainer.py:803] 2025-04-26 20:16:32,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2843
2862
[WARNING|trainer.py:803] 2025-04-26 20:16:33,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3062
[WARNING|trainer.py:803] 2025-04-26 20:16:33,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:34,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3063
2844
2863
[WARNING|trainer.py:803] 2025-04-26 20:16:36,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:36,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:36,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3064
[WARNING|trainer.py:803] 2025-04-26 20:16:37,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2845
2864
[WARNING|trainer.py:803] 2025-04-26 20:16:38,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3065
[WARNING|trainer.py:803] 2025-04-26 20:16:39,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:39,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2846
3066
2865
[WARNING|trainer.py:803] 2025-04-26 20:16:41,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:41,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:41,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3067
2847
2866
[WARNING|trainer.py:803] 2025-04-26 20:16:43,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:43,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:44,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3068
[WARNING|trainer.py:803] 2025-04-26 20:16:45,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2848
2867
3069
[WARNING|trainer.py:803] 2025-04-26 20:16:46,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:46,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:47,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3070
2849
2868
[WARNING|trainer.py:803] 2025-04-26 20:16:48,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:49,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:49,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3071
[WARNING|trainer.py:803] 2025-04-26 20:16:50,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2850
2869
[WARNING|trainer.py:803] 2025-04-26 20:16:51,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3072
[WARNING|trainer.py:803] 2025-04-26 20:16:52,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:16:52,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2851
3073
2870
[WARNING|trainer.py:803] 2025-04-26 20:16:54,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:16:54,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:54,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3074
2852
[WARNING|trainer.py:803] 2025-04-26 20:16:56,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2871
[WARNING|trainer.py:803] 2025-04-26 20:16:57,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3075
[WARNING|trainer.py:803] 2025-04-26 20:16:57,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:16:58,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2853
2872
3076
[WARNING|trainer.py:803] 2025-04-26 20:16:59,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:00,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:00,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3077
2854
2873
[WARNING|trainer.py:803] 2025-04-26 20:17:01,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:17:02,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:02,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3078
[WARNING|trainer.py:803] 2025-04-26 20:17:03,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2855
2874
3079
[WARNING|trainer.py:803] 2025-04-26 20:17:04,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:05,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:05,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2856
3080
2875
[WARNING|trainer.py:803] 2025-04-26 20:17:07,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:17:07,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:07,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3081
2857
[WARNING|trainer.py:803] 2025-04-26 20:17:09,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2876
[WARNING|trainer.py:803] 2025-04-26 20:17:10,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:10,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3082
[WARNING|trainer.py:803] 2025-04-26 20:17:11,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2858
2877
3083
[WARNING|trainer.py:803] 2025-04-26 20:17:12,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:13,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:13,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3084
2859
2878
[WARNING|trainer.py:803] 2025-04-26 20:17:15,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:17:15,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:17:15,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3085
[WARNING|trainer.py:803] 2025-04-26 20:17:16,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2879
2860
3086
[WARNING|trainer.py:803] 2025-04-26 20:17:18,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:18,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:18,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3087
2880
2861
[WARNING|trainer.py:803] 2025-04-26 20:17:20,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:17:20,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:20,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3088
[WARNING|trainer.py:803] 2025-04-26 20:17:22,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2881
2862
3089
[WARNING|trainer.py:803] 2025-04-26 20:17:23,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:23,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:24,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3090
2882
2863
[WARNING|trainer.py:803] 2025-04-26 20:17:25,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:26,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:26,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3091
2864
2883
[WARNING|trainer.py:803] 2025-04-26 20:17:27,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:28,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:28,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3092
[WARNING|trainer.py:803] 2025-04-26 20:17:29,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2865
2884
3093
[WARNING|trainer.py:803] 2025-04-26 20:17:31,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:31,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:31,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3094
2866
2885
[WARNING|trainer.py:803] 2025-04-26 20:17:33,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:33,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:34,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3095
[WARNING|trainer.py:803] 2025-04-26 20:17:35,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2867
2886
3096
[WARNING|trainer.py:803] 2025-04-26 20:17:36,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:36,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:37,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3097
2887
2868
[WARNING|trainer.py:803] 2025-04-26 20:17:38,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:39,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:39,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3098
[WARNING|trainer.py:803] 2025-04-26 20:17:40,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2888
2869
[WARNING|trainer.py:803] 2025-04-26 20:17:41,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3099
[WARNING|trainer.py:803] 2025-04-26 20:17:41,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:42,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2889
2870
3100
[WARNING|trainer.py:803] 2025-04-26 20:17:44,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:44,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:44,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2871
2890
3101
[WARNING|trainer.py:803] 2025-04-26 20:17:47,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:47,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:47,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2891
2872
3102
[WARNING|trainer.py:803] 2025-04-26 20:17:49,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:49,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:49,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
2892
2873
3103
[WARNING|trainer.py:803] 2025-04-26 20:17:52,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:52,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:17:52,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3104
2893
2874
[WARNING|trainer.py:803] 2025-04-26 20:17:54,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:17:54,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:17:54,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3105
2894
2875
[WARNING|trainer.py:803] 2025-04-26 20:17:56,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:57,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:17:57,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3106
2895
[WARNING|trainer.py:803] 2025-04-26 20:17:59,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2876
[WARNING|trainer.py:803] 2025-04-26 20:18:00,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:00,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3107
[WARNING|trainer.py:803] 2025-04-26 20:18:01,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2896
2877
[WARNING|trainer.py:803] 2025-04-26 20:18:02,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:02,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3108
[WARNING|trainer.py:803] 2025-04-26 20:18:03,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2897
2878
3109
[WARNING|trainer.py:803] 2025-04-26 20:18:05,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:05,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:05,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2879
2898
3110
[WARNING|trainer.py:803] 2025-04-26 20:18:07,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:07,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:08,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3111
2899
2880
[WARNING|trainer.py:803] 2025-04-26 20:18:10,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:10,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:10,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3112
2881
2900
[WARNING|trainer.py:803] 2025-04-26 20:18:12,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:13,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:13,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3113
[WARNING|trainer.py:803] 2025-04-26 20:18:14,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2882
2901
[WARNING|trainer.py:803] 2025-04-26 20:18:15,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:15,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3114
[WARNING|trainer.py:803] 2025-04-26 20:18:16,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2902
2883
3115
[WARNING|trainer.py:803] 2025-04-26 20:18:18,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:18,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:19,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2903
2884
3116
[WARNING|trainer.py:803] 2025-04-26 20:18:21,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:21,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:21,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2904
2885
3117
[WARNING|trainer.py:803] 2025-04-26 20:18:23,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:18:23,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:18:23,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2905
2886
3118
[WARNING|trainer.py:803] 2025-04-26 20:18:26,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:26,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:26,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2906
2887
3119
[WARNING|trainer.py:803] 2025-04-26 20:18:28,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:28,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:28,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3120
2907
2888
[WARNING|trainer.py:803] 2025-04-26 20:18:31,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 20:18:31,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:31,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3121
2908
2889
[WARNING|trainer.py:803] 2025-04-26 20:18:33,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:34,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:18:34,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2909
2890
3122
[WARNING|trainer.py:803] 2025-04-26 20:18:36,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:36,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:36,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3123
2910
2891
[WARNING|trainer.py:803] 2025-04-26 20:18:38,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:39,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:39,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3124
2911
2892
[WARNING|trainer.py:803] 2025-04-26 20:18:41,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:41,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:42,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3125
2912
2893
[WARNING|trainer.py:803] 2025-04-26 20:18:44,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:44,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:44,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3126
2913
[WARNING|trainer.py:803] 2025-04-26 20:18:46,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2894
[WARNING|trainer.py:803] 2025-04-26 20:18:46,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:18:47,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3127
2914
[WARNING|trainer.py:803] 2025-04-26 20:18:49,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2895
[WARNING|trainer.py:803] 2025-04-26 20:18:49,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:50,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3128
2915
[WARNING|trainer.py:803] 2025-04-26 20:18:51,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2896
[WARNING|trainer.py:803] 2025-04-26 20:18:52,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:52,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3129
[WARNING|trainer.py:803] 2025-04-26 20:18:53,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2916
2897
[WARNING|trainer.py:803] 2025-04-26 20:18:54,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3130
[WARNING|trainer.py:803] 2025-04-26 20:18:55,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:18:56,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2917
2898
[WARNING|trainer.py:803] 2025-04-26 20:18:57,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3131
[WARNING|trainer.py:803] 2025-04-26 20:18:57,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:18:58,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2918
2899
[WARNING|trainer.py:803] 2025-04-26 20:18:59,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3132
[WARNING|trainer.py:803] 2025-04-26 20:19:00,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:00,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2919
3133
2900
[WARNING|trainer.py:803] 2025-04-26 20:19:02,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:19:02,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:19:03,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3134
2920
[WARNING|trainer.py:803] 2025-04-26 20:19:04,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2901
[WARNING|trainer.py:803] 2025-04-26 20:19:05,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:05,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2921
3135
2902
[WARNING|trainer.py:803] 2025-04-26 20:19:07,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:19:07,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:19:08,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3136
2922
[WARNING|trainer.py:803] 2025-04-26 20:19:09,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2903
[WARNING|trainer.py:803] 2025-04-26 20:19:10,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:10,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3137
2923
[WARNING|trainer.py:803] 2025-04-26 20:19:12,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2904
[WARNING|trainer.py:803] 2025-04-26 20:19:12,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3138
[WARNING|trainer.py:803] 2025-04-26 20:19:13,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesNo
[WARNING|trainer.py:803] 2025-04-26 20:19:14,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2924
2905
[WARNING|trainer.py:803] 2025-04-26 20:19:15,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3139
[WARNING|trainer.py:803] 2025-04-26 20:19:15,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:16,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2925
2906
[WARNING|trainer.py:803] 2025-04-26 20:19:17,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3140
[WARNING|trainer.py:803] 2025-04-26 20:19:18,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:19,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2926
2907
[WARNING|trainer.py:803] 2025-04-26 20:19:20,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3141
[WARNING|trainer.py:803] 2025-04-26 20:19:21,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:21,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2927
3142
2908
[WARNING|trainer.py:803] 2025-04-26 20:19:22,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:23,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:23,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2928
3143
2909
[WARNING|trainer.py:803] 2025-04-26 20:19:25,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:25,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:26,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3144
2929
2910
[WARNING|trainer.py:803] 2025-04-26 20:19:28,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:28,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:19:28,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2930
3145
2911
[WARNING|trainer.py:803] 2025-04-26 20:19:30,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:31,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:31,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3146
2931
2912
[WARNING|trainer.py:803] 2025-04-26 20:19:33,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:19:33,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:33,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3147
2932
[WARNING|trainer.py:803] 2025-04-26 20:19:35,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2913
[WARNING|trainer.py:803] 2025-04-26 20:19:36,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:36,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3148
2933
[WARNING|trainer.py:803] 2025-04-26 20:19:37,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2914
[WARNING|trainer.py:803] 2025-04-26 20:19:38,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3149
[WARNING|trainer.py:803] 2025-04-26 20:19:39,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:39,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2934
3150
2915
[WARNING|trainer.py:803] 2025-04-26 20:19:41,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:41,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:41,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3151
2935
2916
[WARNING|trainer.py:803] 2025-04-26 20:19:43,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:43,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:19:44,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3152
[WARNING|trainer.py:803] 2025-04-26 20:19:45,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2936
2917
3153
[WARNING|trainer.py:803] 2025-04-26 20:19:46,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:19:46,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:47,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3154
2918
2937
[WARNING|trainer.py:803] 2025-04-26 20:19:48,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:19:49,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:49,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3155
[WARNING|trainer.py:803] 2025-04-26 20:19:50,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2919
2938
3156
[WARNING|trainer.py:803] 2025-04-26 20:19:51,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:19:51,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:52,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3157
2939
2920
[WARNING|trainer.py:803] 2025-04-26 20:19:54,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:54,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:54,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3158
[WARNING|trainer.py:803] 2025-04-26 20:19:55,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2940
2921
[WARNING|trainer.py:803] 2025-04-26 20:19:56,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3159
[WARNING|trainer.py:803] 2025-04-26 20:19:56,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:19:57,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2922
2941
3160
[WARNING|trainer.py:803] 2025-04-26 20:19:59,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:59,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:19:59,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3161
2942
2923
[WARNING|trainer.py:803] 2025-04-26 20:20:01,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:02,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:02,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3162
[WARNING|trainer.py:803] 2025-04-26 20:20:03,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2924
2943
3163
[WARNING|trainer.py:803] 2025-04-26 20:20:04,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:04,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:05,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3164
2925
2944
[WARNING|trainer.py:803] 2025-04-26 20:20:07,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:07,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:07,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3165
[WARNING|trainer.py:803] 2025-04-26 20:20:08,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2926
2945
3166
[WARNING|trainer.py:803] 2025-04-26 20:20:09,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:09,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:10,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3167
2927
2946
[WARNING|trainer.py:803] 2025-04-26 20:20:12,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:12,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:12,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3168
[WARNING|trainer.py:803] 2025-04-26 20:20:14,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2928
2947
[WARNING|trainer.py:803] 2025-04-26 20:20:15,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:15,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3169
[WARNING|trainer.py:803] 2025-04-26 20:20:15,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2948
2929
3170
[WARNING|trainer.py:803] 2025-04-26 20:20:17,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:17,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:17,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3171
[WARNING|trainer.py:803] 2025-04-26 20:20:19,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2949
2930
[WARNING|trainer.py:803] 2025-04-26 20:20:20,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:20,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3172
[WARNING|trainer.py:803] 2025-04-26 20:20:21,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2931
2950
3173
[WARNING|trainer.py:803] 2025-04-26 20:20:22,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:22,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:23,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3174
2932
2951
[WARNING|trainer.py:803] 2025-04-26 20:20:25,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:25,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:25,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3175
[WARNING|trainer.py:803] 2025-04-26 20:20:27,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2933
2952
3176
[WARNING|trainer.py:803] 2025-04-26 20:20:28,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:28,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:28,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2953
2934
3177
[WARNING|trainer.py:803] 2025-04-26 20:20:30,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:30,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:30,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3178
2954
2935
[WARNING|trainer.py:803] 2025-04-26 20:20:32,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:33,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:33,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3179
[WARNING|trainer.py:803] 2025-04-26 20:20:34,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2955
2936
3180
[WARNING|trainer.py:803] 2025-04-26 20:20:35,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:20:35,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:36,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3181
2956
2937
[WARNING|trainer.py:803] 2025-04-26 20:20:38,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:20:38,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:38,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3182
[WARNING|trainer.py:803] 2025-04-26 20:20:39,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2957
2938
3183
[WARNING|trainer.py:803] 2025-04-26 20:20:40,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:41,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:41,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3184
2958
2939
[WARNING|trainer.py:803] 2025-04-26 20:20:43,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:43,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:43,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3185
[WARNING|trainer.py:803] 2025-04-26 20:20:45,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2940
2959
3186
[WARNING|trainer.py:803] 2025-04-26 20:20:46,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:47,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2960
2941
3187
[WARNING|trainer.py:803] 2025-04-26 20:20:48,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:20:48,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:48,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3188
2961
2942
[WARNING|trainer.py:803] 2025-04-26 20:20:50,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:51,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:51,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3189
[WARNING|trainer.py:803] 2025-04-26 20:20:52,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2962
2943
3190
[WARNING|trainer.py:803] 2025-04-26 20:20:53,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:54,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:54,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3191
2963
2944
[WARNING|trainer.py:803] 2025-04-26 20:20:56,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:20:56,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:20:56,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3192
[WARNING|trainer.py:803] 2025-04-26 20:20:58,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2964
2945
3193
[WARNING|trainer.py:803] 2025-04-26 20:20:59,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:59,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:20:59,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3194
2946
2965
[WARNING|trainer.py:803] 2025-04-26 20:21:01,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:01,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:01,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3195
2947
[WARNING|trainer.py:803] 2025-04-26 20:21:03,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2966
[WARNING|trainer.py:803] 2025-04-26 20:21:04,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:21:04,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3196
[WARNING|trainer.py:803] 2025-04-26 20:21:05,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2967
2948
3197
[WARNING|trainer.py:803] 2025-04-26 20:21:06,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:06,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:07,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3198
2968
2949
[WARNING|trainer.py:803] 2025-04-26 20:21:09,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:09,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:09,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3199
[WARNING|trainer.py:803] 2025-04-26 20:21:10,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2969
2950
3200
[WARNING|trainer.py:803] 2025-04-26 20:21:12,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:12,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:12,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3201
2970
2951
[WARNING|trainer.py:803] 2025-04-26 20:21:14,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:14,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:14,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3202
[WARNING|trainer.py:803] 2025-04-26 20:21:16,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2952
2971
3203
[WARNING|trainer.py:803] 2025-04-26 20:21:17,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:17,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:17,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3204
2953
2972
[WARNING|trainer.py:803] 2025-04-26 20:21:19,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:19,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:19,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3205
[WARNING|trainer.py:803] 2025-04-26 20:21:21,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2954
2973
[WARNING|trainer.py:803] 2025-04-26 20:21:22,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3206
[WARNING|trainer.py:803] 2025-04-26 20:21:22,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:23,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2955
3207
2974
[WARNING|trainer.py:803] 2025-04-26 20:21:24,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:25,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:25,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3208
2956
2975
[WARNING|trainer.py:803] 2025-04-26 20:21:26,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:27,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:27,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3209
[WARNING|trainer.py:803] 2025-04-26 20:21:28,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2957
2976
3210
[WARNING|trainer.py:803] 2025-04-26 20:21:30,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:30,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:30,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3211
2958
2977
[WARNING|trainer.py:803] 2025-04-26 20:21:32,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:32,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:32,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3212
[WARNING|trainer.py:803] 2025-04-26 20:21:34,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2959
2978
3213
[WARNING|trainer.py:803] 2025-04-26 20:21:35,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:35,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:35,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3214
2960
2979
[WARNING|trainer.py:803] 2025-04-26 20:21:37,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:37,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:21:37,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3215
[WARNING|trainer.py:803] 2025-04-26 20:21:39,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2980
2961
3216
[WARNING|trainer.py:803] 2025-04-26 20:21:40,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:21:40,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:40,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3217
2981
2962
[WARNING|trainer.py:803] 2025-04-26 20:21:42,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:21:43,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:43,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3218
[WARNING|trainer.py:803] 2025-04-26 20:21:44,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
2982
2963
3219
[WARNING|trainer.py:803] 2025-04-26 20:21:45,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:45,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:21:46,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3220
2983
2964
[WARNING|trainer.py:803] 2025-04-26 20:21:47,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:48,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:48,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3221
[WARNING|trainer.py:803] 2025-04-26 20:21:49,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2984
2965
3222
[WARNING|trainer.py:803] 2025-04-26 20:21:50,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:51,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:51,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3223
2985
2966
[WARNING|trainer.py:803] 2025-04-26 20:21:53,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:53,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:21:53,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3224
[WARNING|trainer.py:803] 2025-04-26 20:21:54,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2986
2967
3225
[WARNING|trainer.py:803] 2025-04-26 20:21:55,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:56,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:21:56,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3226
2987
2968
[WARNING|trainer.py:803] 2025-04-26 20:21:58,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:58,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:21:58,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3227
[WARNING|trainer.py:803] 2025-04-26 20:22:00,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2988
2969
3228
[WARNING|trainer.py:803] 2025-04-26 20:22:01,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:01,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:02,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3229
2989
2970
[WARNING|trainer.py:803] 2025-04-26 20:22:03,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:03,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:03,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3230
[WARNING|trainer.py:803] 2025-04-26 20:22:05,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2990
2971
[WARNING|trainer.py:803] 2025-04-26 20:22:06,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3231
[WARNING|trainer.py:803] 2025-04-26 20:22:06,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:07,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2991
3232
2972
[WARNING|trainer.py:803] 2025-04-26 20:22:08,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:08,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:09,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3233
2992
[WARNING|trainer.py:803] 2025-04-26 20:22:10,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2973
[WARNING|trainer.py:803] 2025-04-26 20:22:11,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3234
[WARNING|trainer.py:803] 2025-04-26 20:22:11,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:22:12,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3235
2993
2974
[WARNING|trainer.py:803] 2025-04-26 20:22:14,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:14,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:14,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3236
[WARNING|trainer.py:803] 2025-04-26 20:22:15,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2994
2975
[WARNING|trainer.py:803] 2025-04-26 20:22:16,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3237
[WARNING|trainer.py:803] 2025-04-26 20:22:17,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:17,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2995
3238
2976
[WARNING|trainer.py:803] 2025-04-26 20:22:19,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:19,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:19,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3239
[WARNING|trainer.py:803] 2025-04-26 20:22:21,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2996
2977
[WARNING|trainer.py:803] 2025-04-26 20:22:21,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3240
[WARNING|trainer.py:803] 2025-04-26 20:22:22,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:22:22,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2997
3241
2978
[WARNING|trainer.py:803] 2025-04-26 20:22:24,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:24,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:24,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3242
2998
[WARNING|trainer.py:803] 2025-04-26 20:22:26,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2979
[WARNING|trainer.py:803] 2025-04-26 20:22:26,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3243
[WARNING|trainer.py:803] 2025-04-26 20:22:27,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:22:27,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2999
3244
2980
[WARNING|trainer.py:803] 2025-04-26 20:22:29,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:29,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:29,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3245
3000
2981
[WARNING|trainer.py:803] 2025-04-26 20:22:31,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:32,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:32,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3246
[WARNING|trainer.py:803] 2025-04-26 20:22:33,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3001
2982
3247
[WARNING|trainer.py:803] 2025-04-26 20:22:34,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:34,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:22:35,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3002
3248
[WARNING|trainer.py:803] 2025-04-26 20:22:36,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2983
[WARNING|trainer.py:803] 2025-04-26 20:22:36,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:37,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3249
3003
[WARNING|trainer.py:803] 2025-04-26 20:22:38,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:38,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2984
3250
[WARNING|trainer.py:803] 2025-04-26 20:22:40,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:40,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3004
3251
[WARNING|trainer.py:803] 2025-04-26 20:22:41,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
2985
[WARNING|trainer.py:803] 2025-04-26 20:22:41,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3005
3252
[WARNING|trainer.py:803] 2025-04-26 20:22:42,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:22:43,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:43,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3253
2986
3006
[WARNING|trainer.py:803] 2025-04-26 20:22:45,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:45,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:45,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3254
3007
[WARNING|trainer.py:803] 2025-04-26 20:22:46,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2987
[WARNING|trainer.py:803] 2025-04-26 20:22:47,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3255
[WARNING|trainer.py:803] 2025-04-26 20:22:47,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:48,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3008
3256
[WARNING|trainer.py:803] 2025-04-26 20:22:49,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
2988
[WARNING|trainer.py:803] 2025-04-26 20:22:50,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:22:50,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3009
3257
[WARNING|trainer.py:803] 2025-04-26 20:22:51,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:22:51,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
2989
3258
[WARNING|trainer.py:803] 2025-04-26 20:22:52,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3010
[WARNING|trainer.py:803] 2025-04-26 20:22:53,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:22:53,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3259
2990
3011
[WARNING|trainer.py:803] 2025-04-26 20:22:55,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:22:55,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:22:55,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3260
[WARNING|trainer.py:803] 2025-04-26 20:22:56,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3012
2991
3261
[WARNING|trainer.py:803] 2025-04-26 20:22:57,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:58,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:22:58,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3013
3262
2992
[WARNING|trainer.py:803] 2025-04-26 20:23:00,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:00,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:00,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3263
3014
[WARNING|trainer.py:803] 2025-04-26 20:23:02,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:02,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2993
3264
[WARNING|trainer.py:803] 2025-04-26 20:23:03,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3015
[WARNING|trainer.py:803] 2025-04-26 20:23:03,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:04,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
2994
3265
3016
[WARNING|trainer.py:803] 2025-04-26 20:23:05,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:05,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:06,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3266
2995
[WARNING|trainer.py:803] 2025-04-26 20:23:07,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3017
[WARNING|trainer.py:803] 2025-04-26 20:23:08,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:08,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3267
[WARNING|trainer.py:803] 2025-04-26 20:23:09,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3018
2996
3268
[WARNING|trainer.py:803] 2025-04-26 20:23:10,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:11,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:11,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3019
3269
2997
[WARNING|trainer.py:803] 2025-04-26 20:23:12,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:13,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:13,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3270
3020
[WARNING|trainer.py:803] 2025-04-26 20:23:14,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:15,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
2998
3271
[WARNING|trainer.py:803] 2025-04-26 20:23:16,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3021
[WARNING|trainer.py:803] 2025-04-26 20:23:16,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:17,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3272
2999
[WARNING|trainer.py:803] 2025-04-26 20:23:18,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3022
[WARNING|trainer.py:803] 2025-04-26 20:23:18,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3273
[WARNING|trainer.py:803] 2025-04-26 20:23:19,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:20,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3000
3023
3274
[WARNING|trainer.py:803] 2025-04-26 20:23:21,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:21,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:21,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3001
3024
3275
[WARNING|trainer.py:803] 2025-04-26 20:23:23,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:23,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:23,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3276
3025
3002
[WARNING|trainer.py:803] 2025-04-26 20:23:25,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:25,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:25,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3277
3026
3003
[WARNING|trainer.py:803] 2025-04-26 20:23:27,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:27,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:27,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3278
[WARNING|trainer.py:803] 2025-04-26 20:23:28,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3027
3004
3279
[WARNING|trainer.py:803] 2025-04-26 20:23:29,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:30,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:30,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3028
3005
3280
[WARNING|trainer.py:803] 2025-04-26 20:23:32,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:32,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:32,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3281
3029
3006
[WARNING|trainer.py:803] 2025-04-26 20:23:34,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:34,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:34,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3282
3030
3007
[WARNING|trainer.py:803] 2025-04-26 20:23:36,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:36,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:36,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3283
3031
3008
[WARNING|trainer.py:803] 2025-04-26 20:23:37,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:38,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:38,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3284
[WARNING|trainer.py:803] 2025-04-26 20:23:39,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3032
3009
3285
[WARNING|trainer.py:803] 2025-04-26 20:23:40,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:40,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:41,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3010
3033
3286
[WARNING|trainer.py:803] 2025-04-26 20:23:42,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:42,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:43,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3011
3034
3287
[WARNING|trainer.py:803] 2025-04-26 20:23:44,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:44,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:45,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3288
3012
3035
[WARNING|trainer.py:803] 2025-04-26 20:23:46,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:46,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:47,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3289
3013
[WARNING|trainer.py:803] 2025-04-26 20:23:48,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3036
[WARNING|trainer.py:803] 2025-04-26 20:23:48,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:49,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3290
3014
[WARNING|trainer.py:803] 2025-04-26 20:23:50,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3037
[WARNING|trainer.py:803] 2025-04-26 20:23:51,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3291
[WARNING|trainer.py:803] 2025-04-26 20:23:51,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:23:52,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3015
3038
3292
[WARNING|trainer.py:803] 2025-04-26 20:23:53,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:53,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:53,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3016
3039
3293
[WARNING|trainer.py:803] 2025-04-26 20:23:55,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:55,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:55,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3294
3017
3040
[WARNING|trainer.py:803] 2025-04-26 20:23:57,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:57,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:23:57,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3295
3018
3041
[WARNING|trainer.py:803] 2025-04-26 20:23:59,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:23:59,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:23:59,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3296
[WARNING|trainer.py:803] 2025-04-26 20:24:01,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3019
3042
3297
[WARNING|trainer.py:803] 2025-04-26 20:24:01,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:02,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:02,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3020
3043
3298
[WARNING|trainer.py:803] 2025-04-26 20:24:04,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:04,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:04,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3021
3299
3044
[WARNING|trainer.py:803] 2025-04-26 20:24:06,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:06,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:06,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3300
3022
3045
[WARNING|trainer.py:803] 2025-04-26 20:24:08,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:24:08,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3301
[WARNING|trainer.py:803] 2025-04-26 20:24:08,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:09,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3302
3023
3046
[WARNING|trainer.py:803] 2025-04-26 20:24:10,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:10,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3303
[WARNING|trainer.py:803] 2025-04-26 20:24:10,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:11,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3304
3024
3047
[WARNING|trainer.py:803] 2025-04-26 20:24:12,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:12,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3305
[WARNING|trainer.py:803] 2025-04-26 20:24:12,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:13,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3306
3025
3048
[WARNING|trainer.py:803] 2025-04-26 20:24:14,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:14,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:14,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3307
[WARNING|trainer.py:803] 2025-04-26 20:24:15,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3026
3308
3049
[WARNING|trainer.py:803] 2025-04-26 20:24:16,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:16,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:17,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3309
[WARNING|trainer.py:803] 2025-04-26 20:24:18,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3027
3050
3310
[WARNING|trainer.py:803] 2025-04-26 20:24:18,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:19,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:19,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3311
[WARNING|trainer.py:803] 2025-04-26 20:24:20,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3028
3051
3312
[WARNING|trainer.py:803] 2025-04-26 20:24:21,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:21,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:21,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3313
[WARNING|trainer.py:803] 2025-04-26 20:24:22,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3029
3052
3314
[WARNING|trainer.py:803] 2025-04-26 20:24:23,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:23,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:23,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3315
[WARNING|trainer.py:803] 2025-04-26 20:24:24,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3053
3030
3316
[WARNING|trainer.py:803] 2025-04-26 20:24:25,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:25,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:25,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3317
3054
[WARNING|trainer.py:803] 2025-04-26 20:24:26,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3031
3318
[WARNING|trainer.py:803] 2025-04-26 20:24:27,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:27,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:27,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3319
3055
3032
[WARNING|trainer.py:803] 2025-04-26 20:24:29,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3320
[WARNING|trainer.py:803] 2025-04-26 20:24:29,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:29,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:30,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3321
3056
3033
[WARNING|trainer.py:803] 2025-04-26 20:24:31,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3322
[WARNING|trainer.py:803] 2025-04-26 20:24:31,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:32,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:32,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3323
3057
3034
[WARNING|trainer.py:803] 2025-04-26 20:24:33,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3324
[WARNING|trainer.py:803] 2025-04-26 20:24:34,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:34,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:34,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3325
3058
3035
[WARNING|trainer.py:803] 2025-04-26 20:24:35,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3326
[WARNING|trainer.py:803] 2025-04-26 20:24:36,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:36,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:36,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3327
3059
3036
[WARNING|trainer.py:803] 2025-04-26 20:24:37,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3328
[WARNING|trainer.py:803] 2025-04-26 20:24:38,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:38,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:38,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3329
3060
3037
[WARNING|trainer.py:803] 2025-04-26 20:24:39,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3330
[WARNING|trainer.py:803] 2025-04-26 20:24:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:40,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:40,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3331
3061
3038
[WARNING|trainer.py:803] 2025-04-26 20:24:42,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3332
[WARNING|trainer.py:803] 2025-04-26 20:24:42,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:42,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:43,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3333
3062
3039
[WARNING|trainer.py:803] 2025-04-26 20:24:44,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3334
[WARNING|trainer.py:803] 2025-04-26 20:24:44,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:44,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:45,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3335
3063
3040
[WARNING|trainer.py:803] 2025-04-26 20:24:46,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:46,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3336
[WARNING|trainer.py:803] 2025-04-26 20:24:46,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:47,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3337
3064
3041
[WARNING|trainer.py:803] 2025-04-26 20:24:48,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:48,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3338
[WARNING|trainer.py:803] 2025-04-26 20:24:49,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:49,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3339
3065
3042
[WARNING|trainer.py:803] 2025-04-26 20:24:50,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:24:50,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3340
[WARNING|trainer.py:803] 2025-04-26 20:24:51,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:51,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3341
3066
3043
[WARNING|trainer.py:803] 2025-04-26 20:24:52,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:52,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3342
[WARNING|trainer.py:803] 2025-04-26 20:24:53,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:54,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3067
3343
3044
[WARNING|trainer.py:803] 2025-04-26 20:24:55,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:55,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3344
[WARNING|trainer.py:803] 2025-04-26 20:24:55,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:56,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3068
3345
3045
[WARNING|trainer.py:803] 2025-04-26 20:24:57,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:24:57,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:57,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3346
[WARNING|trainer.py:803] 2025-04-26 20:24:58,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3069
3347
3046
[WARNING|trainer.py:803] 2025-04-26 20:24:59,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:24:59,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:24:59,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3348
[WARNING|trainer.py:803] 2025-04-26 20:25:00,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3070
3349
3047
[WARNING|trainer.py:803] 2025-04-26 20:25:01,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:01,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:01,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3350
3071
[WARNING|trainer.py:803] 2025-04-26 20:25:02,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3048
3351
[WARNING|trainer.py:803] 2025-04-26 20:25:03,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:03,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:04,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3352
3072
[WARNING|trainer.py:803] 2025-04-26 20:25:05,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3049
3353
[WARNING|trainer.py:803] 2025-04-26 20:25:05,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:06,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:06,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3354
3073
3050
[WARNING|trainer.py:803] 2025-04-26 20:25:07,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3355
[WARNING|trainer.py:803] 2025-04-26 20:25:07,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:08,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:08,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3356
3074
3051
[WARNING|trainer.py:803] 2025-04-26 20:25:09,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3357
[WARNING|trainer.py:803] 2025-04-26 20:25:10,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:10,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:10,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3358
3075
3052
[WARNING|trainer.py:803] 2025-04-26 20:25:11,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3359
[WARNING|trainer.py:803] 2025-04-26 20:25:12,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:12,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:12,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3360
3076
3053
[WARNING|trainer.py:803] 2025-04-26 20:25:13,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3361
[WARNING|trainer.py:803] 2025-04-26 20:25:14,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:14,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:14,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3362
3077
3054
[WARNING|trainer.py:803] 2025-04-26 20:25:16,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3363
[WARNING|trainer.py:803] 2025-04-26 20:25:16,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:16,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:17,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3364
3078
3055
[WARNING|trainer.py:803] 2025-04-26 20:25:18,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3365
[WARNING|trainer.py:803] 2025-04-26 20:25:18,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:18,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:19,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3366
3079
3056
[WARNING|trainer.py:803] 2025-04-26 20:25:20,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:20,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3367
[WARNING|trainer.py:803] 2025-04-26 20:25:21,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:21,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3368
3080
3057
[WARNING|trainer.py:803] 2025-04-26 20:25:22,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:22,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3369
[WARNING|trainer.py:803] 2025-04-26 20:25:23,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:23,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3370
3081
3058
[WARNING|trainer.py:803] 2025-04-26 20:25:24,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:24,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3371
[WARNING|trainer.py:803] 2025-04-26 20:25:25,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:25,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3372
3082
3059
[WARNING|trainer.py:803] 2025-04-26 20:25:26,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:27,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3373
[WARNING|trainer.py:803] 2025-04-26 20:25:27,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:28,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3374
3083
3060
[WARNING|trainer.py:803] 2025-04-26 20:25:29,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:29,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3375
[WARNING|trainer.py:803] 2025-04-26 20:25:29,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:30,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3376
3084
3061
[WARNING|trainer.py:803] 2025-04-26 20:25:31,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:31,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3377
[WARNING|trainer.py:803] 2025-04-26 20:25:31,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:32,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3378
3085
3062
[WARNING|trainer.py:803] 2025-04-26 20:25:33,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:33,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3379
[WARNING|trainer.py:803] 2025-04-26 20:25:33,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:34,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3380
3086
3063
[WARNING|trainer.py:803] 2025-04-26 20:25:35,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:35,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:35,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3381
[WARNING|trainer.py:803] 2025-04-26 20:25:36,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3087
3382
3064
[WARNING|trainer.py:803] 2025-04-26 20:25:37,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:37,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:38,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3383
3088
[WARNING|trainer.py:803] 2025-04-26 20:25:39,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3384
3065
[WARNING|trainer.py:803] 2025-04-26 20:25:39,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:40,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:40,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3385
3089
[WARNING|trainer.py:803] 2025-04-26 20:25:41,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3066
3386
[WARNING|trainer.py:803] 2025-04-26 20:25:41,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:42,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:42,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3387
3090
[WARNING|trainer.py:803] 2025-04-26 20:25:43,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3067
3388
[WARNING|trainer.py:803] 2025-04-26 20:25:44,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:44,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:44,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3389
3091
3068
[WARNING|trainer.py:803] 2025-04-26 20:25:45,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3390
[WARNING|trainer.py:803] 2025-04-26 20:25:46,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:46,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:46,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3391
3092
3069
[WARNING|trainer.py:803] 2025-04-26 20:25:47,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3392
[WARNING|trainer.py:803] 2025-04-26 20:25:48,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:48,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:49,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3393
3093
[WARNING|trainer.py:803] 2025-04-26 20:25:50,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3070
3394
[WARNING|trainer.py:803] 2025-04-26 20:25:50,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:50,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:51,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3395
3094
3071
[WARNING|trainer.py:803] 2025-04-26 20:25:52,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:52,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3396
[WARNING|trainer.py:803] 2025-04-26 20:25:53,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:53,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3397
3095
3072
[WARNING|trainer.py:803] 2025-04-26 20:25:54,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:25:54,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3398
[WARNING|trainer.py:803] 2025-04-26 20:25:55,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:55,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3399
3096
3073
[WARNING|trainer.py:803] 2025-04-26 20:25:56,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:25:56,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3400
[WARNING|trainer.py:803] 2025-04-26 20:25:57,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:25:57,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3401
3097
3074
[WARNING|trainer.py:803] 2025-04-26 20:25:58,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:25:58,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3402
[WARNING|trainer.py:803] 2025-04-26 20:25:59,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:25:59,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3403
3098
3075
[WARNING|trainer.py:803] 2025-04-26 20:26:00,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:26:01,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3404
[WARNING|trainer.py:803] 2025-04-26 20:26:01,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:02,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3099
3405
3076
[WARNING|trainer.py:803] 2025-04-26 20:26:03,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:03,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3406
[WARNING|trainer.py:803] 2025-04-26 20:26:03,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:04,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3100
3407
3077
[WARNING|trainer.py:803] 2025-04-26 20:26:05,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:05,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3408
[WARNING|trainer.py:803] 2025-04-26 20:26:05,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:06,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3409
3078
[WARNING|trainer.py:803] 2025-04-26 20:26:07,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3101
3410
[WARNING|trainer.py:803] 2025-04-26 20:26:07,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:08,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:08,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3411
3079
[WARNING|trainer.py:803] 2025-04-26 20:26:09,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3412
[WARNING|trainer.py:803] 2025-04-26 20:26:10,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3102
[WARNING|trainer.py:803] 2025-04-26 20:26:10,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3413
[WARNING|trainer.py:803] 2025-04-26 20:26:11,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3080
[WARNING|trainer.py:803] 2025-04-26 20:26:11,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:12,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3414
[WARNING|trainer.py:803] 2025-04-26 20:26:13,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3103
3415
3081
[WARNING|trainer.py:803] 2025-04-26 20:26:13,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:26:14,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:26:14,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3416
[WARNING|trainer.py:803] 2025-04-26 20:26:15,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3417
3082
3104
[WARNING|trainer.py:803] 2025-04-26 20:26:16,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:26:16,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:16,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3418
[WARNING|trainer.py:803] 2025-04-26 20:26:17,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3083
3419
3105
[WARNING|trainer.py:803] 2025-04-26 20:26:18,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:18,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:26:18,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3420
[WARNING|trainer.py:803] 2025-04-26 20:26:19,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3084
3421
[WARNING|trainer.py:803] 2025-04-26 20:26:20,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3106
[WARNING|trainer.py:803] 2025-04-26 20:26:20,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3422
[WARNING|trainer.py:803] 2025-04-26 20:26:21,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:22,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3085
3423
[WARNING|trainer.py:803] 2025-04-26 20:26:22,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:23,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3424
3107
3086
[WARNING|trainer.py:803] 2025-04-26 20:26:24,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:24,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3425
[WARNING|trainer.py:803] 2025-04-26 20:26:25,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:25,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3426
3108
3087
[WARNING|trainer.py:803] 2025-04-26 20:26:26,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3427
[WARNING|trainer.py:803] 2025-04-26 20:26:26,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:27,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:27,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3428
3088
3109
[WARNING|trainer.py:803] 2025-04-26 20:26:28,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3429
[WARNING|trainer.py:803] 2025-04-26 20:26:29,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:29,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:29,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3430
3089
[WARNING|trainer.py:803] 2025-04-26 20:26:30,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3110
3431
[WARNING|trainer.py:803] 2025-04-26 20:26:31,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:26:31,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:31,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3432
3090
[WARNING|trainer.py:803] 2025-04-26 20:26:33,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3433
[WARNING|trainer.py:803] 2025-04-26 20:26:33,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3111
[WARNING|trainer.py:803] 2025-04-26 20:26:34,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:34,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3434
3091
[WARNING|trainer.py:803] 2025-04-26 20:26:35,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:35,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3435
3112
[WARNING|trainer.py:803] 2025-04-26 20:26:36,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:26:36,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3436
3092
[WARNING|trainer.py:803] 2025-04-26 20:26:37,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:26:37,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3437
3113
[WARNING|trainer.py:803] 2025-04-26 20:26:38,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3438
[WARNING|trainer.py:803] 2025-04-26 20:26:38,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3093
[WARNING|trainer.py:803] 2025-04-26 20:26:39,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:39,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3439
3114
[WARNING|trainer.py:803] 2025-04-26 20:26:40,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3094
3440
[WARNING|trainer.py:803] 2025-04-26 20:26:41,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:41,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:41,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3441
[WARNING|trainer.py:803] 2025-04-26 20:26:42,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3095
3442
3115
[WARNING|trainer.py:803] 2025-04-26 20:26:43,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:44,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:44,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3443
[WARNING|trainer.py:803] 2025-04-26 20:26:45,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3096
3444
3116
[WARNING|trainer.py:803] 2025-04-26 20:26:46,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:46,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3445
[WARNING|trainer.py:803] 2025-04-26 20:26:46,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:47,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3097
3446
[WARNING|trainer.py:803] 2025-04-26 20:26:48,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:48,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3117
3447
[WARNING|trainer.py:803] 2025-04-26 20:26:49,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:49,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3098
3448
[WARNING|trainer.py:803] 2025-04-26 20:26:50,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:50,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3449
3118
[WARNING|trainer.py:803] 2025-04-26 20:26:51,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3099
3450
[WARNING|trainer.py:803] 2025-04-26 20:26:52,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:52,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:52,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3451
3100
[WARNING|trainer.py:803] 2025-04-26 20:26:53,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3452
3119
[WARNING|trainer.py:803] 2025-04-26 20:26:54,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:55,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:26:55,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3453
[WARNING|trainer.py:803] 2025-04-26 20:26:56,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3454
3120
3101
[WARNING|trainer.py:803] 2025-04-26 20:26:57,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3455
[WARNING|trainer.py:803] 2025-04-26 20:26:57,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:26:57,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:26:58,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3456
[WARNING|trainer.py:803] 2025-04-26 20:26:59,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3457
3102
3121
[WARNING|trainer.py:803] 2025-04-26 20:27:00,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:00,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:27:00,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3458
[WARNING|trainer.py:803] 2025-04-26 20:27:01,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3459
3103
[WARNING|trainer.py:803] 2025-04-26 20:27:02,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3122
3460
[WARNING|trainer.py:803] 2025-04-26 20:27:03,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:27:03,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:03,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3461
[WARNING|trainer.py:803] 2025-04-26 20:27:05,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3104
3462
3123
[WARNING|trainer.py:803] 2025-04-26 20:27:06,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:06,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:27:06,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3463
[WARNING|trainer.py:803] 2025-04-26 20:27:07,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3105
3464
3124
[WARNING|trainer.py:803] 2025-04-26 20:27:08,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:08,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3465
[WARNING|trainer.py:803] 2025-04-26 20:27:09,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:09,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3466
3106
[WARNING|trainer.py:803] 2025-04-26 20:27:10,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3467
3125
[WARNING|trainer.py:803] 2025-04-26 20:27:11,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:11,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:11,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3468
[WARNING|trainer.py:803] 2025-04-26 20:27:12,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3469
3107
3126
[WARNING|trainer.py:803] 2025-04-26 20:27:13,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:13,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3470
[WARNING|trainer.py:803] 2025-04-26 20:27:14,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:14,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3471
3108
[WARNING|trainer.py:803] 2025-04-26 20:27:15,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3472
[WARNING|trainer.py:803] 2025-04-26 20:27:16,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3127
[WARNING|trainer.py:803] 2025-04-26 20:27:17,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3473
[WARNING|trainer.py:803] 2025-04-26 20:27:17,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3109
[WARNING|trainer.py:803] 2025-04-26 20:27:18,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3474
[WARNING|trainer.py:803] 2025-04-26 20:27:18,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3128
[WARNING|trainer.py:803] 2025-04-26 20:27:19,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3475
3110
[WARNING|trainer.py:803] 2025-04-26 20:27:20,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:20,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3476
[WARNING|trainer.py:803] 2025-04-26 20:27:21,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:21,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3477
3129
[WARNING|trainer.py:803] 2025-04-26 20:27:22,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3111
3478
[WARNING|trainer.py:803] 2025-04-26 20:27:22,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:23,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:23,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3479
3130
[WARNING|trainer.py:803] 2025-04-26 20:27:24,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3480
3112
[WARNING|trainer.py:803] 2025-04-26 20:27:25,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:25,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:25,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3481
[WARNING|trainer.py:803] 2025-04-26 20:27:26,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3131
3482
3113
[WARNING|trainer.py:803] 2025-04-26 20:27:27,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:28,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:27:28,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3483
[WARNING|trainer.py:803] 2025-04-26 20:27:29,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3484
3132
3114
[WARNING|trainer.py:803] 2025-04-26 20:27:30,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3485
[WARNING|trainer.py:803] 2025-04-26 20:27:30,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:30,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:31,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3486
3133
[WARNING|trainer.py:803] 2025-04-26 20:27:32,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3487
3115
[WARNING|trainer.py:803] 2025-04-26 20:27:32,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:33,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:33,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3488
3134
[WARNING|trainer.py:803] 2025-04-26 20:27:34,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3489
[WARNING|trainer.py:803] 2025-04-26 20:27:35,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3116
[WARNING|trainer.py:803] 2025-04-26 20:27:35,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3490
[WARNING|trainer.py:803] 2025-04-26 20:27:36,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:36,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3491
3135
3117
[WARNING|trainer.py:803] 2025-04-26 20:27:37,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3492
[WARNING|trainer.py:803] 2025-04-26 20:27:38,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:27:38,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:38,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3493
3136
[WARNING|trainer.py:803] 2025-04-26 20:27:40,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:40,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3494
3118
[WARNING|trainer.py:803] 2025-04-26 20:27:41,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3495
[WARNING|trainer.py:803] 2025-04-26 20:27:41,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3137
[WARNING|trainer.py:803] 2025-04-26 20:27:42,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3496
[WARNING|trainer.py:803] 2025-04-26 20:27:43,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:43,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3119
3497
[WARNING|trainer.py:803] 2025-04-26 20:27:44,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:44,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3138
3498
[WARNING|trainer.py:803] 2025-04-26 20:27:45,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:45,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3499
3120
[WARNING|trainer.py:803] 2025-04-26 20:27:46,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3500
[WARNING|trainer.py:803] 2025-04-26 20:27:47,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3139
[WARNING|trainer.py:803] 2025-04-26 20:27:47,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3501
[WARNING|trainer.py:803] 2025-04-26 20:27:48,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:27:48,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3121
3502
[WARNING|trainer.py:803] 2025-04-26 20:27:49,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:50,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3140
3503
[WARNING|trainer.py:803] 2025-04-26 20:27:51,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:51,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3504
[WARNING|trainer.py:803] 2025-04-26 20:27:52,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3122
3505
3141
[WARNING|trainer.py:803] 2025-04-26 20:27:53,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:53,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3506
[WARNING|trainer.py:803] 2025-04-26 20:27:53,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:54,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3507
3123
3142
[WARNING|trainer.py:803] 2025-04-26 20:27:55,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:27:55,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3508
[WARNING|trainer.py:803] 2025-04-26 20:27:56,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:27:56,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3509
3124
3143
[WARNING|trainer.py:803] 2025-04-26 20:27:57,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3510
[WARNING|trainer.py:803] 2025-04-26 20:27:58,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:58,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:27:58,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3511
[WARNING|trainer.py:803] 2025-04-26 20:27:59,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3512
3144
3125
[WARNING|trainer.py:803] 2025-04-26 20:28:01,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:01,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:01,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3513
[WARNING|trainer.py:803] 2025-04-26 20:28:02,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3514
3126
[WARNING|trainer.py:803] 2025-04-26 20:28:03,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3515
3145
[WARNING|trainer.py:803] 2025-04-26 20:28:03,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:04,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:04,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3516
[WARNING|trainer.py:803] 2025-04-26 20:28:05,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3517
3146
3127
[WARNING|trainer.py:803] 2025-04-26 20:28:06,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:06,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3518
[WARNING|trainer.py:803] 2025-04-26 20:28:07,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:07,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3519
3147
[WARNING|trainer.py:803] 2025-04-26 20:28:08,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3128
3520
[WARNING|trainer.py:803] 2025-04-26 20:28:09,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:09,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:09,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3521
[WARNING|trainer.py:803] 2025-04-26 20:28:10,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3148
3522
3129
[WARNING|trainer.py:803] 2025-04-26 20:28:11,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:11,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3523
[WARNING|trainer.py:803] 2025-04-26 20:28:12,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3149
[WARNING|trainer.py:803] 2025-04-26 20:28:12,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3524
[WARNING|trainer.py:803] 2025-04-26 20:28:13,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3130
[WARNING|trainer.py:803] 2025-04-26 20:28:13,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3525
[WARNING|trainer.py:803] 2025-04-26 20:28:14,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3150
[WARNING|trainer.py:803] 2025-04-26 20:28:15,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3526
[WARNING|trainer.py:803] 2025-04-26 20:28:15,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:16,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3131
3527
[WARNING|trainer.py:803] 2025-04-26 20:28:17,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3151
[WARNING|trainer.py:803] 2025-04-26 20:28:17,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3528
[WARNING|trainer.py:803] 2025-04-26 20:28:18,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:18,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3529
3132
3152
[WARNING|trainer.py:803] 2025-04-26 20:28:19,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3530
[WARNING|trainer.py:803] 2025-04-26 20:28:19,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:20,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:20,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x67d94b40] moov atom not found
[20:28:20] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
3531
3133
3153
[WARNING|trainer.py:803] 2025-04-26 20:28:21,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3532
[WARNING|trainer.py:803] 2025-04-26 20:28:22,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:22,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:22,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3533
3154
3134
[WARNING|trainer.py:803] 2025-04-26 20:28:23,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3534
[WARNING|trainer.py:803] 2025-04-26 20:28:24,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:24,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:24,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3535
3155
[WARNING|trainer.py:803] 2025-04-26 20:28:25,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3536
[WARNING|trainer.py:803] 2025-04-26 20:28:26,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3135
[WARNING|trainer.py:803] 2025-04-26 20:28:26,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3537
[WARNING|trainer.py:803] 2025-04-26 20:28:27,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3156
[WARNING|trainer.py:803] 2025-04-26 20:28:27,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3538
[WARNING|trainer.py:803] 2025-04-26 20:28:28,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3136
[WARNING|trainer.py:803] 2025-04-26 20:28:29,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3539
3157
[WARNING|trainer.py:803] 2025-04-26 20:28:29,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:30,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3540
[WARNING|trainer.py:803] 2025-04-26 20:28:30,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:31,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3541
3137
3158
[WARNING|trainer.py:803] 2025-04-26 20:28:32,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:32,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:32,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3542
[WARNING|trainer.py:803] 2025-04-26 20:28:33,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3543
3159
3138
[WARNING|trainer.py:803] 2025-04-26 20:28:34,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:34,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3544
[WARNING|trainer.py:803] 2025-04-26 20:28:34,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:35,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3545
3160
[WARNING|trainer.py:803] 2025-04-26 20:28:36,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3139
[WARNING|trainer.py:803] 2025-04-26 20:28:36,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3546
[WARNING|trainer.py:803] 2025-04-26 20:28:37,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:37,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3161
3547
[WARNING|trainer.py:803] 2025-04-26 20:28:38,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:38,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3548
3140
[WARNING|trainer.py:803] 2025-04-26 20:28:39,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3162
3549
[WARNING|trainer.py:803] 2025-04-26 20:28:40,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:40,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:41,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3550
3141
[WARNING|trainer.py:803] 2025-04-26 20:28:42,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3163
3551
[WARNING|trainer.py:803] 2025-04-26 20:28:42,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:43,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:43,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3552
[WARNING|trainer.py:803] 2025-04-26 20:28:44,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3164
3142
3553
[WARNING|trainer.py:803] 2025-04-26 20:28:45,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:28:45,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:45,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3554
3165
[WARNING|trainer.py:803] 2025-04-26 20:28:46,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3555
3143
[WARNING|trainer.py:803] 2025-04-26 20:28:47,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:47,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3556
[WARNING|trainer.py:803] 2025-04-26 20:28:47,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3166
[WARNING|trainer.py:803] 2025-04-26 20:28:48,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3557
[WARNING|trainer.py:803] 2025-04-26 20:28:49,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3144
[WARNING|trainer.py:803] 2025-04-26 20:28:49,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3558
[WARNING|trainer.py:803] 2025-04-26 20:28:50,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3167
[WARNING|trainer.py:803] 2025-04-26 20:28:50,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3559
[WARNING|trainer.py:803] 2025-04-26 20:28:51,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:51,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3560
3168
3145
[WARNING|trainer.py:803] 2025-04-26 20:28:52,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3561
[WARNING|trainer.py:803] 2025-04-26 20:28:53,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:53,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:53,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3562
3169
3146
[WARNING|trainer.py:803] 2025-04-26 20:28:55,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3563
[WARNING|trainer.py:803] 2025-04-26 20:28:55,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:55,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:28:56,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3564
3170
[WARNING|trainer.py:803] 2025-04-26 20:28:57,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:28:57,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3565
3147
[WARNING|trainer.py:803] 2025-04-26 20:28:58,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:28:58,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3171
3566
[WARNING|trainer.py:803] 2025-04-26 20:28:59,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:28:59,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3567
3148
3172
[WARNING|trainer.py:803] 2025-04-26 20:29:00,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3568
[WARNING|trainer.py:803] 2025-04-26 20:29:00,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:01,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:01,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3569
3149
3173
[WARNING|trainer.py:803] 2025-04-26 20:29:02,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3570
[WARNING|trainer.py:803] 2025-04-26 20:29:02,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:03,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:03,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3571
3150
3174
[WARNING|trainer.py:803] 2025-04-26 20:29:04,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3572
[WARNING|trainer.py:803] 2025-04-26 20:29:05,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:05,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:05,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3573
3151
3175
[WARNING|trainer.py:803] 2025-04-26 20:29:06,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3574
[WARNING|trainer.py:803] 2025-04-26 20:29:07,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:07,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:07,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3575
3152
3176
[WARNING|trainer.py:803] 2025-04-26 20:29:08,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3576
[WARNING|trainer.py:803] 2025-04-26 20:29:09,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:09,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:09,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3577
3153
3177
[WARNING|trainer.py:803] 2025-04-26 20:29:11,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3578
[WARNING|trainer.py:803] 2025-04-26 20:29:11,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:11,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:12,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3579
3154
3178
[WARNING|trainer.py:803] 2025-04-26 20:29:13,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3580
[WARNING|trainer.py:803] 2025-04-26 20:29:13,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:13,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:14,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3581
3155
3179
[WARNING|trainer.py:803] 2025-04-26 20:29:15,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3582
[WARNING|trainer.py:803] 2025-04-26 20:29:15,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:15,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:16,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3583
3156
3180
[WARNING|trainer.py:803] 2025-04-26 20:29:17,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:17,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3584
[WARNING|trainer.py:803] 2025-04-26 20:29:17,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:18,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3585
3157
3181
[WARNING|trainer.py:803] 2025-04-26 20:29:19,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:19,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3586
[WARNING|trainer.py:803] 2025-04-26 20:29:20,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:20,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3158
3587
3182
[WARNING|trainer.py:803] 2025-04-26 20:29:21,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:21,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:29:22,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3588
[WARNING|trainer.py:803] 2025-04-26 20:29:23,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3159
3589
3183
[WARNING|trainer.py:803] 2025-04-26 20:29:23,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:24,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:29:24,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3590
3160
[WARNING|trainer.py:803] 2025-04-26 20:29:25,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3184
3591
[WARNING|trainer.py:803] 2025-04-26 20:29:26,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:26,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:26,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3592
3161
[WARNING|trainer.py:803] 2025-04-26 20:29:27,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3185
3593
[WARNING|trainer.py:803] 2025-04-26 20:29:28,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:28,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:28,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3594
3162
3186
[WARNING|trainer.py:803] 2025-04-26 20:29:29,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3595
[WARNING|trainer.py:803] 2025-04-26 20:29:30,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:30,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:30,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3596
3163
3187
[WARNING|trainer.py:803] 2025-04-26 20:29:31,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3597
[WARNING|trainer.py:803] 2025-04-26 20:29:32,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:32,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:32,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3598
3164
3188
[WARNING|trainer.py:803] 2025-04-26 20:29:34,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3599
[WARNING|trainer.py:803] 2025-04-26 20:29:34,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:34,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:35,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3600
3189
3165
[WARNING|trainer.py:803] 2025-04-26 20:29:36,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3601
[WARNING|trainer.py:803] 2025-04-26 20:29:36,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:29:36,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:37,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3166
3602
3190
[WARNING|trainer.py:803] 2025-04-26 20:29:38,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:38,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:38,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3603
3167
[WARNING|trainer.py:803] 2025-04-26 20:29:39,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3191
3604
[WARNING|trainer.py:803] 2025-04-26 20:29:40,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:40,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:29:40,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3605
3168
[WARNING|trainer.py:803] 2025-04-26 20:29:41,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3192
3606
[WARNING|trainer.py:803] 2025-04-26 20:29:42,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:42,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:43,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3607
3169
[WARNING|trainer.py:803] 2025-04-26 20:29:44,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3193
3608
[WARNING|trainer.py:803] 2025-04-26 20:29:44,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:44,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:45,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3609
3170
3194
[WARNING|trainer.py:803] 2025-04-26 20:29:46,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3610
[WARNING|trainer.py:803] 2025-04-26 20:29:46,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:47,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:47,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3611
3171
3195
[WARNING|trainer.py:803] 2025-04-26 20:29:48,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:48,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3612
[WARNING|trainer.py:803] 2025-04-26 20:29:49,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:49,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3172
3613
3196
[WARNING|trainer.py:803] 2025-04-26 20:29:50,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:51,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:51,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3614
3173
[WARNING|trainer.py:803] 2025-04-26 20:29:52,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3615
3197
[WARNING|trainer.py:803] 2025-04-26 20:29:52,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:53,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:29:53,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3616
3174
[WARNING|trainer.py:803] 2025-04-26 20:29:54,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3198
3617
[WARNING|trainer.py:803] 2025-04-26 20:29:55,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:55,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:55,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3618
3175
3199
[WARNING|trainer.py:803] 2025-04-26 20:29:56,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3619
[WARNING|trainer.py:803] 2025-04-26 20:29:57,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:29:57,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:29:57,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3620
3176
3200
[WARNING|trainer.py:803] 2025-04-26 20:29:59,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:29:59,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3621
[WARNING|trainer.py:803] 2025-04-26 20:29:59,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:00,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3622
3177
3201
[WARNING|trainer.py:803] 2025-04-26 20:30:01,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:01,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:01,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3623
[WARNING|trainer.py:803] 2025-04-26 20:30:02,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3178
3202
3624
[WARNING|trainer.py:803] 2025-04-26 20:30:03,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:03,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:03,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3625
3203
[WARNING|trainer.py:803] 2025-04-26 20:30:04,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3179
3626
[WARNING|trainer.py:803] 2025-04-26 20:30:05,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:05,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:06,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3627
3204
3180
[WARNING|trainer.py:803] 2025-04-26 20:30:07,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:07,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3628
[WARNING|trainer.py:803] 2025-04-26 20:30:07,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:08,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3629
3205
3181
[WARNING|trainer.py:803] 2025-04-26 20:30:09,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:09,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3630
[WARNING|trainer.py:803] 2025-04-26 20:30:09,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:10,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3206
3631
3182
[WARNING|trainer.py:803] 2025-04-26 20:30:11,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:11,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:12,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3632
3207
[WARNING|trainer.py:803] 2025-04-26 20:30:13,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3183
3633
[WARNING|trainer.py:803] 2025-04-26 20:30:13,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:14,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:14,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3634
3208
[WARNING|trainer.py:803] 2025-04-26 20:30:15,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3184
3635
[WARNING|trainer.py:803] 2025-04-26 20:30:15,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:16,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:16,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3636
3209
3185
[WARNING|trainer.py:803] 2025-04-26 20:30:17,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3637
[WARNING|trainer.py:803] 2025-04-26 20:30:18,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:18,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:18,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3638
3210
3186
[WARNING|trainer.py:803] 2025-04-26 20:30:20,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:20,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:20,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3639
[WARNING|trainer.py:803] 2025-04-26 20:30:21,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3211
3640
3187
[WARNING|trainer.py:803] 2025-04-26 20:30:22,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:22,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:22,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3641
3212
[WARNING|trainer.py:803] 2025-04-26 20:30:23,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3188
3642
[WARNING|trainer.py:803] 2025-04-26 20:30:24,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:24,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:24,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3643
3213
3189
[WARNING|trainer.py:803] 2025-04-26 20:30:25,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:26,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3644
[WARNING|trainer.py:803] 2025-04-26 20:30:26,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:27,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3214
3645
3190
[WARNING|trainer.py:803] 2025-04-26 20:30:28,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:28,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3646
[WARNING|trainer.py:803] 2025-04-26 20:30:28,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3215
[WARNING|trainer.py:803] 2025-04-26 20:30:29,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3647
3191
[WARNING|trainer.py:803] 2025-04-26 20:30:30,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:30,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:30,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3648
3216
[WARNING|trainer.py:803] 2025-04-26 20:30:31,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3192
[WARNING|trainer.py:803] 2025-04-26 20:30:32,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3649
[WARNING|trainer.py:803] 2025-04-26 20:30:33,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:33,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3650
3217
[WARNING|trainer.py:803] 2025-04-26 20:30:34,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:34,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3193
3651
[WARNING|trainer.py:803] 2025-04-26 20:30:35,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3218
[WARNING|trainer.py:803] 2025-04-26 20:30:35,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3652
[WARNING|trainer.py:803] 2025-04-26 20:30:36,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3194
[WARNING|trainer.py:803] 2025-04-26 20:30:36,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3653
[WARNING|trainer.py:803] 2025-04-26 20:30:37,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3219
[WARNING|trainer.py:803] 2025-04-26 20:30:38,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:38,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3654
3195
[WARNING|trainer.py:803] 2025-04-26 20:30:39,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:39,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3220
3655
[WARNING|trainer.py:803] 2025-04-26 20:30:40,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:40,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3196
3656
[WARNING|trainer.py:803] 2025-04-26 20:30:41,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3221
[WARNING|trainer.py:803] 2025-04-26 20:30:41,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3657
[WARNING|trainer.py:803] 2025-04-26 20:30:42,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:42,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3197
3658
3222
[WARNING|trainer.py:803] 2025-04-26 20:30:43,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:43,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3659
[WARNING|trainer.py:803] 2025-04-26 20:30:44,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3198
[WARNING|trainer.py:803] 2025-04-26 20:30:44,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3660
[WARNING|trainer.py:803] 2025-04-26 20:30:45,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3223

Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:46,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:46,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3661
3199
[WARNING|trainer.py:803] 2025-04-26 20:30:47,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3224
[WARNING|trainer.py:803] 2025-04-26 20:30:47,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3662
[WARNING|trainer.py:803] 2025-04-26 20:30:48,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:48,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3663
3200
3225
[WARNING|trainer.py:803] 2025-04-26 20:30:49,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:30:49,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3664
[WARNING|trainer.py:803] 2025-04-26 20:30:50,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:50,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3201
3665
3226
[WARNING|trainer.py:803] 2025-04-26 20:30:51,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:51,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3666
[WARNING|trainer.py:803] 2025-04-26 20:30:52,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3202
[WARNING|trainer.py:803] 2025-04-26 20:30:53,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3227
3667
[WARNING|trainer.py:803] 2025-04-26 20:30:53,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:30:54,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:54,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3668
3203
3228
[WARNING|trainer.py:803] 2025-04-26 20:30:55,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:55,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3669
[WARNING|trainer.py:803] 2025-04-26 20:30:56,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:56,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3204
3670
3229
[WARNING|trainer.py:803] 2025-04-26 20:30:57,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:30:57,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:30:58,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3671
[WARNING|trainer.py:803] 2025-04-26 20:30:59,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3205
3230
3672
[WARNING|trainer.py:803] 2025-04-26 20:31:00,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:00,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:00,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3673
3206
3231
[WARNING|trainer.py:803] 2025-04-26 20:31:01,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3674
[WARNING|trainer.py:803] 2025-04-26 20:31:02,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:02,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:02,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3675
3232
3207
[WARNING|trainer.py:803] 2025-04-26 20:31:03,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3676
[WARNING|trainer.py:803] 2025-04-26 20:31:04,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:04,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:04,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3677
3233
3208
[WARNING|trainer.py:803] 2025-04-26 20:31:05,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:06,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3678
[WARNING|trainer.py:803] 2025-04-26 20:31:06,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:07,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3234
3679
3209
[WARNING|trainer.py:803] 2025-04-26 20:31:08,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:08,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:08,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3680
3235
[WARNING|trainer.py:803] 2025-04-26 20:31:09,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3210
3681
[WARNING|trainer.py:803] 2025-04-26 20:31:10,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:10,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:10,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3682
3236
3211
[WARNING|trainer.py:803] 2025-04-26 20:31:11,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:12,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3683
[WARNING|trainer.py:803] 2025-04-26 20:31:12,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:13,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3237
3684
3212
[WARNING|trainer.py:803] 2025-04-26 20:31:14,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:14,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:14,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3685
3238
[WARNING|trainer.py:803] 2025-04-26 20:31:15,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3213
3686
[WARNING|trainer.py:803] 2025-04-26 20:31:16,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:16,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:16,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3687
3239
3214
[WARNING|trainer.py:803] 2025-04-26 20:31:17,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3688
[WARNING|trainer.py:803] 2025-04-26 20:31:18,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:18,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:19,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3689
3240
3215
[WARNING|trainer.py:803] 2025-04-26 20:31:20,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:20,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3690
[WARNING|trainer.py:803] 2025-04-26 20:31:20,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:21,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3241
3691
3216
[WARNING|trainer.py:803] 2025-04-26 20:31:22,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:22,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:22,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3692
3242
[WARNING|trainer.py:803] 2025-04-26 20:31:23,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3217
3693
[WARNING|trainer.py:803] 2025-04-26 20:31:24,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:24,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:31:24,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3694
3243
[WARNING|trainer.py:803] 2025-04-26 20:31:25,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3218
[WARNING|trainer.py:803] 2025-04-26 20:31:26,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3695
[WARNING|trainer.py:803] 2025-04-26 20:31:26,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:27,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3244
3696
3219
[WARNING|trainer.py:803] 2025-04-26 20:31:28,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:28,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3697
[WARNING|trainer.py:803] 2025-04-26 20:31:28,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3245
[WARNING|trainer.py:803] 2025-04-26 20:31:29,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3698
3220
[WARNING|trainer.py:803] 2025-04-26 20:31:30,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:30,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:30,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3699
3246
[WARNING|trainer.py:803] 2025-04-26 20:31:31,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3221
3700
[WARNING|trainer.py:803] 2025-04-26 20:31:32,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:32,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:33,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3701
3247
3222
[WARNING|trainer.py:803] 2025-04-26 20:31:34,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:34,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3702
[WARNING|trainer.py:803] 2025-04-26 20:31:34,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3248
[WARNING|trainer.py:803] 2025-04-26 20:31:35,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3703
3223
[WARNING|trainer.py:803] 2025-04-26 20:31:36,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:36,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:37,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3704
3249
[WARNING|trainer.py:803] 2025-04-26 20:31:38,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3224
[WARNING|trainer.py:803] 2025-04-26 20:31:38,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3705
[WARNING|trainer.py:803] 2025-04-26 20:31:38,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:39,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3250
3706
3225
[WARNING|trainer.py:803] 2025-04-26 20:31:40,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:40,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3707
[WARNING|trainer.py:803] 2025-04-26 20:31:40,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3251
[WARNING|trainer.py:803] 2025-04-26 20:31:41,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3708
[WARNING|trainer.py:803] 2025-04-26 20:31:42,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3226
[WARNING|trainer.py:803] 2025-04-26 20:31:42,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:42,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3252
3709
[WARNING|trainer.py:803] 2025-04-26 20:31:43,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3227
[WARNING|trainer.py:803] 2025-04-26 20:31:43,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3710
[WARNING|trainer.py:803] 2025-04-26 20:31:44,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3253
[WARNING|trainer.py:803] 2025-04-26 20:31:45,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3711
[WARNING|trainer.py:803] 2025-04-26 20:31:45,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3228
[WARNING|trainer.py:803] 2025-04-26 20:31:46,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3254
[WARNING|trainer.py:803] 2025-04-26 20:31:46,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3712
[WARNING|trainer.py:803] 2025-04-26 20:31:47,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:47,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3713
3229
3255
[WARNING|trainer.py:803] 2025-04-26 20:31:48,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:48,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3714
[WARNING|trainer.py:803] 2025-04-26 20:31:49,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3230
[WARNING|trainer.py:803] 2025-04-26 20:31:50,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3715
3256
[WARNING|trainer.py:803] 2025-04-26 20:31:50,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:31:51,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:51,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3716
3231
[WARNING|trainer.py:803] 2025-04-26 20:31:52,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3257
3717
[WARNING|trainer.py:803] 2025-04-26 20:31:52,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:31:53,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:31:53,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3718
3232
3258
[WARNING|trainer.py:803] 2025-04-26 20:31:54,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:31:54,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3719
[WARNING|trainer.py:803] 2025-04-26 20:31:55,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3233
[WARNING|trainer.py:803] 2025-04-26 20:31:56,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3259
3720
[WARNING|trainer.py:803] 2025-04-26 20:31:56,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:31:57,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:31:57,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3721
3234
3260
[WARNING|trainer.py:803] 2025-04-26 20:31:58,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:31:58,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3722
[WARNING|trainer.py:803] 2025-04-26 20:31:59,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:00,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3235
3723
3261
[WARNING|trainer.py:803] 2025-04-26 20:32:00,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:01,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:01,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3724
3236
3262
[WARNING|trainer.py:803] 2025-04-26 20:32:02,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3725
[WARNING|trainer.py:803] 2025-04-26 20:32:02,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:03,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:03,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3726
3237
3263
[WARNING|trainer.py:803] 2025-04-26 20:32:04,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:05,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:05,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3727
[WARNING|trainer.py:803] 2025-04-26 20:32:06,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3238
3264
3728
[WARNING|trainer.py:803] 2025-04-26 20:32:07,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:07,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:07,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3729
3239
[WARNING|trainer.py:803] 2025-04-26 20:32:08,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3265
3730
[WARNING|trainer.py:803] 2025-04-26 20:32:09,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:09,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:09,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3731
3240
3266
[WARNING|trainer.py:803] 2025-04-26 20:32:10,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3732
[WARNING|trainer.py:803] 2025-04-26 20:32:11,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:11,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:11,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3733
3241
3267
[WARNING|trainer.py:803] 2025-04-26 20:32:13,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:13,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3734
[WARNING|trainer.py:803] 2025-04-26 20:32:13,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:14,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3242
3735
3268
[WARNING|trainer.py:803] 2025-04-26 20:32:15,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:15,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:15,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3736
3243
[WARNING|trainer.py:803] 2025-04-26 20:32:16,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3269
3737
[WARNING|trainer.py:803] 2025-04-26 20:32:17,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:17,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:17,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3738
3244
3270
[WARNING|trainer.py:803] 2025-04-26 20:32:19,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:19,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3739
[WARNING|trainer.py:803] 2025-04-26 20:32:19,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:20,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3245
3740
3271
[WARNING|trainer.py:803] 2025-04-26 20:32:21,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:21,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:21,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3741
3246
[WARNING|trainer.py:803] 2025-04-26 20:32:22,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3272
3742
[WARNING|trainer.py:803] 2025-04-26 20:32:23,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:23,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:23,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3743
3247
3273
[WARNING|trainer.py:803] 2025-04-26 20:32:25,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:25,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3744
[WARNING|trainer.py:803] 2025-04-26 20:32:25,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:26,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3248
3745
3274
[WARNING|trainer.py:803] 2025-04-26 20:32:27,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:27,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:27,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3746
3249
[WARNING|trainer.py:803] 2025-04-26 20:32:28,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3275
3747
[WARNING|trainer.py:803] 2025-04-26 20:32:29,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:29,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:29,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3748
3250
3276
[WARNING|trainer.py:803] 2025-04-26 20:32:31,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:31,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3749
[WARNING|trainer.py:803] 2025-04-26 20:32:31,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:32,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3251
3750
3277
[WARNING|trainer.py:803] 2025-04-26 20:32:33,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:33,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3751
[WARNING|trainer.py:803] 2025-04-26 20:32:33,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3252
[WARNING|trainer.py:803] 2025-04-26 20:32:34,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3278
3752
[WARNING|trainer.py:803] 2025-04-26 20:32:35,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:35,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:35,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3753
3253
3279
[WARNING|trainer.py:803] 2025-04-26 20:32:37,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:37,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3754
[WARNING|trainer.py:803] 2025-04-26 20:32:37,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:38,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3254
3755
3280
[WARNING|trainer.py:803] 2025-04-26 20:32:39,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:39,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3756
[WARNING|trainer.py:803] 2025-04-26 20:32:39,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3255
[WARNING|trainer.py:803] 2025-04-26 20:32:40,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3757
[WARNING|trainer.py:803] 2025-04-26 20:32:41,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3281
[WARNING|trainer.py:803] 2025-04-26 20:32:41,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:41,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3758
3256
[WARNING|trainer.py:803] 2025-04-26 20:32:43,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:32:43,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3282
3759
[WARNING|trainer.py:803] 2025-04-26 20:32:44,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3257
[WARNING|trainer.py:803] 2025-04-26 20:32:44,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3760
[WARNING|trainer.py:803] 2025-04-26 20:32:44,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3283
[WARNING|trainer.py:803] 2025-04-26 20:32:45,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3761
[WARNING|trainer.py:803] 2025-04-26 20:32:46,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3258
[WARNING|trainer.py:803] 2025-04-26 20:32:46,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:47,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3762
3284
[WARNING|trainer.py:803] 2025-04-26 20:32:47,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3259
[WARNING|trainer.py:803] 2025-04-26 20:32:48,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3763
[WARNING|trainer.py:803] 2025-04-26 20:32:48,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:32:49,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3285
3764
3260
[WARNING|trainer.py:803] 2025-04-26 20:32:50,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:50,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3765
[WARNING|trainer.py:803] 2025-04-26 20:32:50,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:51,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3286
3766
3261
[WARNING|trainer.py:803] 2025-04-26 20:32:52,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:52,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:52,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3767
3287
[WARNING|trainer.py:803] 2025-04-26 20:32:53,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3262
3768
[WARNING|trainer.py:803] 2025-04-26 20:32:54,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:54,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:32:55,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3769
3288
3263
[WARNING|trainer.py:803] 2025-04-26 20:32:56,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:56,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3770
[WARNING|trainer.py:803] 2025-04-26 20:32:56,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:32:57,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3771
3289
3264
[WARNING|trainer.py:803] 2025-04-26 20:32:58,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:32:58,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3772
[WARNING|trainer.py:803] 2025-04-26 20:32:58,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:32:59,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3290
3773
3265
[WARNING|trainer.py:803] 2025-04-26 20:33:00,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:00,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3774
[WARNING|trainer.py:803] 2025-04-26 20:33:01,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3291
[WARNING|trainer.py:803] 2025-04-26 20:33:01,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3775
3266
[WARNING|trainer.py:803] 2025-04-26 20:33:02,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:03,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:03,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3776
3292
[WARNING|trainer.py:803] 2025-04-26 20:33:04,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3267
3777
[WARNING|trainer.py:803] 2025-04-26 20:33:04,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:05,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:05,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3778
3293
3268
[WARNING|trainer.py:803] 2025-04-26 20:33:06,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:06,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3779
[WARNING|trainer.py:803] 2025-04-26 20:33:07,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:07,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3294
3780
3269
[WARNING|trainer.py:803] 2025-04-26 20:33:08,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:09,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:09,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3781
3295
[WARNING|trainer.py:803] 2025-04-26 20:33:10,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3270
3782
[WARNING|trainer.py:803] 2025-04-26 20:33:10,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:11,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:11,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3783
3296
3271
[WARNING|trainer.py:803] 2025-04-26 20:33:12,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:12,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3784
[WARNING|trainer.py:803] 2025-04-26 20:33:13,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:13,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3297
3785
3272
[WARNING|trainer.py:803] 2025-04-26 20:33:14,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:15,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:15,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3786
3298
[WARNING|trainer.py:803] 2025-04-26 20:33:16,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3273
3787
[WARNING|trainer.py:803] 2025-04-26 20:33:16,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:17,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:17,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3788
3299
3274
[WARNING|trainer.py:803] 2025-04-26 20:33:18,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:19,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3789
[WARNING|trainer.py:803] 2025-04-26 20:33:19,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:20,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3300
3790
3275
[WARNING|trainer.py:803] 2025-04-26 20:33:21,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:33:21,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:21,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3791
3301
[WARNING|trainer.py:803] 2025-04-26 20:33:22,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:33:22,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3276
3792
3302
[WARNING|trainer.py:803] 2025-04-26 20:33:23,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:23,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:23,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3793
3303
[WARNING|trainer.py:803] 2025-04-26 20:33:24,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3277
[WARNING|trainer.py:803] 2025-04-26 20:33:25,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3794
3304
[WARNING|trainer.py:803] 2025-04-26 20:33:25,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:25,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3795
[WARNING|trainer.py:803] 2025-04-26 20:33:26,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3278
3305
[WARNING|trainer.py:803] 2025-04-26 20:33:27,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3796
[WARNING|trainer.py:803] 2025-04-26 20:33:27,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:27,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3306
[WARNING|trainer.py:803] 2025-04-26 20:33:28,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3797
3279
[WARNING|trainer.py:803] 2025-04-26 20:33:28,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3307
[WARNING|trainer.py:803] 2025-04-26 20:33:29,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:29,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3798
[WARNING|trainer.py:803] 2025-04-26 20:33:30,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3308
[WARNING|trainer.py:803] 2025-04-26 20:33:30,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3280
3799
[WARNING|trainer.py:803] 2025-04-26 20:33:31,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:31,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:31,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3309
3800
[WARNING|trainer.py:803] 2025-04-26 20:33:32,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:32,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3281
3801
3310
[WARNING|trainer.py:803] 2025-04-26 20:33:33,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:34,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:34,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3802
3311
3282
[WARNING|trainer.py:803] 2025-04-26 20:33:35,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:35,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3803
[WARNING|trainer.py:803] 2025-04-26 20:33:35,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3312
[WARNING|trainer.py:803] 2025-04-26 20:33:36,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:36,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3804
3283
3313
[WARNING|trainer.py:803] 2025-04-26 20:33:37,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:37,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3805
[WARNING|trainer.py:803] 2025-04-26 20:33:38,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3314
[WARNING|trainer.py:803] 2025-04-26 20:33:38,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3806
3284
[WARNING|trainer.py:803] 2025-04-26 20:33:39,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3315
[WARNING|trainer.py:803] 2025-04-26 20:33:39,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:39,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3807
[WARNING|trainer.py:803] 2025-04-26 20:33:40,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:40,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3316
3285
3808
[WARNING|trainer.py:803] 2025-04-26 20:33:41,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:41,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:42,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3317
3809
[WARNING|trainer.py:803] 2025-04-26 20:33:43,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:43,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3286
3810
3318
[WARNING|trainer.py:803] 2025-04-26 20:33:44,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:44,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:44,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3811
3319
3287
[WARNING|trainer.py:803] 2025-04-26 20:33:45,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3812
[WARNING|trainer.py:803] 2025-04-26 20:33:45,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:46,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3320
[WARNING|trainer.py:803] 2025-04-26 20:33:46,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3813
[WARNING|trainer.py:803] 2025-04-26 20:33:46,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3288
3321
[WARNING|trainer.py:803] 2025-04-26 20:33:47,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3814
[WARNING|trainer.py:803] 2025-04-26 20:33:48,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:48,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:48,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3322
3815
3289
[WARNING|trainer.py:803] 2025-04-26 20:33:49,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:33:49,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3323
3816
[WARNING|trainer.py:803] 2025-04-26 20:33:50,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:50,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:50,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3817
3324
3290
[WARNING|trainer.py:803] 2025-04-26 20:33:51,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:52,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:33:52,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3818
3325
[WARNING|trainer.py:803] 2025-04-26 20:33:53,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3291
3819
[WARNING|trainer.py:803] 2025-04-26 20:33:53,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3326
[WARNING|trainer.py:803] 2025-04-26 20:33:54,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:54,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3820
[WARNING|trainer.py:803] 2025-04-26 20:33:54,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3327
[WARNING|trainer.py:803] 2025-04-26 20:33:55,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3292
3821
[WARNING|trainer.py:803] 2025-04-26 20:33:56,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:56,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:56,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3328
3822
[WARNING|trainer.py:803] 2025-04-26 20:33:57,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3293
[WARNING|trainer.py:803] 2025-04-26 20:33:57,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3329
3823
[WARNING|trainer.py:803] 2025-04-26 20:33:58,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:58,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:33:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3824
3330
3294
[WARNING|trainer.py:803] 2025-04-26 20:33:59,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:33:59,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3825
[WARNING|trainer.py:803] 2025-04-26 20:34:00,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3331
[WARNING|trainer.py:803] 2025-04-26 20:34:00,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3826
[WARNING|trainer.py:803] 2025-04-26 20:34:01,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3295
3332
[WARNING|trainer.py:803] 2025-04-26 20:34:02,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3827
[WARNING|trainer.py:803] 2025-04-26 20:34:02,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:02,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3333
[WARNING|trainer.py:803] 2025-04-26 20:34:03,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3828
3296
[WARNING|trainer.py:803] 2025-04-26 20:34:03,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:04,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3334
3829
[WARNING|trainer.py:803] 2025-04-26 20:34:04,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:05,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:05,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3830
3335
3297
[WARNING|trainer.py:803] 2025-04-26 20:34:06,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:06,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:06,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3831
3336
[WARNING|trainer.py:803] 2025-04-26 20:34:07,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3832
3298
[WARNING|trainer.py:803] 2025-04-26 20:34:07,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3337
[WARNING|trainer.py:803] 2025-04-26 20:34:08,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:08,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3833
[WARNING|trainer.py:803] 2025-04-26 20:34:09,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3338
[WARNING|trainer.py:803] 2025-04-26 20:34:09,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3834
3299
[WARNING|trainer.py:803] 2025-04-26 20:34:10,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:10,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:10,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3339
3835
[WARNING|trainer.py:803] 2025-04-26 20:34:11,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:11,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3300
3836
3340
[WARNING|trainer.py:803] 2025-04-26 20:34:12,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:34:12,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:12,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3837
3301
3341
[WARNING|trainer.py:803] 2025-04-26 20:34:13,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:13,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3838
[WARNING|trainer.py:803] 2025-04-26 20:34:14,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3302
3342
[WARNING|trainer.py:803] 2025-04-26 20:34:15,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:15,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3839
[WARNING|trainer.py:803] 2025-04-26 20:34:15,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3303
3343
[WARNING|trainer.py:803] 2025-04-26 20:34:16,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3840
[WARNING|trainer.py:803] 2025-04-26 20:34:16,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:16,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3304
[WARNING|trainer.py:803] 2025-04-26 20:34:17,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3344
3841
[WARNING|trainer.py:803] 2025-04-26 20:34:17,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:18,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3305
[WARNING|trainer.py:803] 2025-04-26 20:34:18,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3842
3345
[WARNING|trainer.py:803] 2025-04-26 20:34:19,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:19,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:19,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3306
3843
3346
[WARNING|trainer.py:803] 2025-04-26 20:34:20,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:20,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:20,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3844
3307
3347
[WARNING|trainer.py:803] 2025-04-26 20:34:21,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:21,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3845
[WARNING|trainer.py:803] 2025-04-26 20:34:22,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3308
3348
[WARNING|trainer.py:803] 2025-04-26 20:34:22,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3846
[WARNING|trainer.py:803] 2025-04-26 20:34:23,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:23,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3309
[WARNING|trainer.py:803] 2025-04-26 20:34:23,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3349
3847
[WARNING|trainer.py:803] 2025-04-26 20:34:24,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:24,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:24,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3310
3848
3350
[WARNING|trainer.py:803] 2025-04-26 20:34:25,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:25,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:25,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3849
3311
3351
[WARNING|trainer.py:803] 2025-04-26 20:34:26,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:26,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3850
[WARNING|trainer.py:803] 2025-04-26 20:34:27,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3312
3352
[WARNING|trainer.py:803] 2025-04-26 20:34:27,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3851
[WARNING|trainer.py:803] 2025-04-26 20:34:28,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:28,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3313
3353
[WARNING|trainer.py:803] 2025-04-26 20:34:29,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3852
[WARNING|trainer.py:803] 2025-04-26 20:34:29,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:29,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3314
[WARNING|trainer.py:803] 2025-04-26 20:34:30,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3853
3354
[WARNING|trainer.py:803] 2025-04-26 20:34:30,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:31,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:31,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3315
3854
3355
[WARNING|trainer.py:803] 2025-04-26 20:34:32,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:32,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3855
[WARNING|trainer.py:803] 2025-04-26 20:34:32,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3316
3356
[WARNING|trainer.py:803] 2025-04-26 20:34:33,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:33,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3856
[WARNING|trainer.py:803] 2025-04-26 20:34:33,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3317
3357
[WARNING|trainer.py:803] 2025-04-26 20:34:34,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3857
[WARNING|trainer.py:803] 2025-04-26 20:34:34,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:35,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3318
[WARNING|trainer.py:803] 2025-04-26 20:34:35,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3358
3858
[WARNING|trainer.py:803] 2025-04-26 20:34:36,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:36,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:36,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3319
3859
3359
[WARNING|trainer.py:803] 2025-04-26 20:34:37,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:37,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:37,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3860
3320
3360
[WARNING|trainer.py:803] 2025-04-26 20:34:38,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:38,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:38,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3861
3321
3361
[WARNING|trainer.py:803] 2025-04-26 20:34:39,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:40,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3862
[WARNING|trainer.py:803] 2025-04-26 20:34:40,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3322
3362
[WARNING|trainer.py:803] 2025-04-26 20:34:40,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3863
[WARNING|trainer.py:803] 2025-04-26 20:34:41,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:41,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3323
[WARNING|trainer.py:803] 2025-04-26 20:34:41,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3363
3864
[WARNING|trainer.py:803] 2025-04-26 20:34:42,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:42,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:43,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3324
3364
3865
[WARNING|trainer.py:803] 2025-04-26 20:34:43,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:44,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:44,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3866
3325
3365
[WARNING|trainer.py:803] 2025-04-26 20:34:45,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:45,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:45,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3867
3326
3366
[WARNING|trainer.py:803] 2025-04-26 20:34:46,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3868
[WARNING|trainer.py:803] 2025-04-26 20:34:46,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:46,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3327
3367
[WARNING|trainer.py:803] 2025-04-26 20:34:47,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3869
[WARNING|trainer.py:803] 2025-04-26 20:34:47,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:48,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3328
[WARNING|trainer.py:803] 2025-04-26 20:34:48,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3368
3870
[WARNING|trainer.py:803] 2025-04-26 20:34:49,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:49,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:49,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3329
3369
3871
[WARNING|trainer.py:803] 2025-04-26 20:34:50,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:50,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:50,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3330
3872
3370
[WARNING|trainer.py:803] 2025-04-26 20:34:51,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:51,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:51,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3331
3873
3371
[WARNING|trainer.py:803] 2025-04-26 20:34:53,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:53,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:53,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3874
3332
3372
[WARNING|trainer.py:803] 2025-04-26 20:34:54,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:54,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:54,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3875
3333
3373
[WARNING|trainer.py:803] 2025-04-26 20:34:55,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:34:55,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3876
[WARNING|trainer.py:803] 2025-04-26 20:34:55,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3334
3374
[WARNING|trainer.py:803] 2025-04-26 20:34:56,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3877
[WARNING|trainer.py:803] 2025-04-26 20:34:57,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:34:57,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3335
3375
[WARNING|trainer.py:803] 2025-04-26 20:34:57,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3878
[WARNING|trainer.py:803] 2025-04-26 20:34:58,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:34:58,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3336
[WARNING|trainer.py:803] 2025-04-26 20:34:58,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3376
3879
[WARNING|trainer.py:803] 2025-04-26 20:34:59,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:34:59,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:00,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3337
3377
3880
[WARNING|trainer.py:803] 2025-04-26 20:35:01,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:01,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:01,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3881
3338
3378
[WARNING|trainer.py:803] 2025-04-26 20:35:02,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:02,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:02,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3882
3339
3379
[WARNING|trainer.py:803] 2025-04-26 20:35:03,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:03,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:03,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3883
3340
3380
[WARNING|trainer.py:803] 2025-04-26 20:35:04,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3884
[WARNING|trainer.py:803] 2025-04-26 20:35:04,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:05,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3341
3381
[WARNING|trainer.py:803] 2025-04-26 20:35:05,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3885
[WARNING|trainer.py:803] 2025-04-26 20:35:06,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:06,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3342
[WARNING|trainer.py:803] 2025-04-26 20:35:06,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3382
3886
[WARNING|trainer.py:803] 2025-04-26 20:35:07,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:07,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:07,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3343
3383
3887
[WARNING|trainer.py:803] 2025-04-26 20:35:08,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:08,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:09,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3344
3888
3384
[WARNING|trainer.py:803] 2025-04-26 20:35:10,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:10,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:10,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3889
3345
3385
[WARNING|trainer.py:803] 2025-04-26 20:35:11,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:11,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:11,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3890
3346
3386
[WARNING|trainer.py:803] 2025-04-26 20:35:12,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:12,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:12,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3891
3347
3387
[WARNING|trainer.py:803] 2025-04-26 20:35:13,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3892
[WARNING|trainer.py:803] 2025-04-26 20:35:14,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:14,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3348
3388
[WARNING|trainer.py:803] 2025-04-26 20:35:14,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3893
[WARNING|trainer.py:803] 2025-04-26 20:35:15,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:15,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3349
3389
[WARNING|trainer.py:803] 2025-04-26 20:35:16,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3894
[WARNING|trainer.py:803] 2025-04-26 20:35:16,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:16,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:17,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3390
3350
3895
[WARNING|trainer.py:803] 2025-04-26 20:35:18,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:18,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:18,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3391
3351
3896
[WARNING|trainer.py:803] 2025-04-26 20:35:19,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:19,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:19,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3897
3392
3352
[WARNING|trainer.py:803] 2025-04-26 20:35:20,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:20,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:20,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3898
3393
3353
[WARNING|trainer.py:803] 2025-04-26 20:35:21,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:21,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:22,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3899
3394
3354
[WARNING|trainer.py:803] 2025-04-26 20:35:22,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3900
[WARNING|trainer.py:803] 2025-04-26 20:35:23,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:23,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3395
3355
[WARNING|trainer.py:803] 2025-04-26 20:35:24,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3901
[WARNING|trainer.py:803] 2025-04-26 20:35:24,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:24,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3396
3356
[WARNING|trainer.py:803] 2025-04-26 20:35:25,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3902
[WARNING|trainer.py:803] 2025-04-26 20:35:25,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:35:25,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3397
3357
[WARNING|trainer.py:803] 2025-04-26 20:35:26,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:27,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3903
[WARNING|trainer.py:803] 2025-04-26 20:35:27,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3398
3358
[WARNING|trainer.py:803] 2025-04-26 20:35:28,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3904
[WARNING|trainer.py:803] 2025-04-26 20:35:28,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:28,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3399
3359
[WARNING|trainer.py:803] 2025-04-26 20:35:29,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:29,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3905
[WARNING|trainer.py:803] 2025-04-26 20:35:29,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3400
3360
[WARNING|trainer.py:803] 2025-04-26 20:35:30,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:31,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3906
[WARNING|trainer.py:803] 2025-04-26 20:35:31,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3401
3361
[WARNING|trainer.py:803] 2025-04-26 20:35:31,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3907
[WARNING|trainer.py:803] 2025-04-26 20:35:32,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:35:32,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3402
3362
[WARNING|trainer.py:803] 2025-04-26 20:35:33,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3908
[WARNING|trainer.py:803] 2025-04-26 20:35:33,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:33,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3403
3363
[WARNING|trainer.py:803] 2025-04-26 20:35:34,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:34,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:35:35,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3909
3404
3364
[WARNING|trainer.py:803] 2025-04-26 20:35:35,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:36,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3910
[WARNING|trainer.py:803] 2025-04-26 20:35:36,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3405
3365
[WARNING|trainer.py:803] 2025-04-26 20:35:37,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:37,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3911
[WARNING|trainer.py:803] 2025-04-26 20:35:37,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3406
3366
[WARNING|trainer.py:803] 2025-04-26 20:35:38,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:38,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:38,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3912
3407
3367
[WARNING|trainer.py:803] 2025-04-26 20:35:39,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:40,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:40,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3913
3408
3368
[WARNING|trainer.py:803] 2025-04-26 20:35:41,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:41,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:41,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3914
3409
3369
[WARNING|trainer.py:803] 2025-04-26 20:35:42,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:42,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:42,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3915
3410
3370
[WARNING|trainer.py:803] 2025-04-26 20:35:43,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:44,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:44,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3916
3411
3371
[WARNING|trainer.py:803] 2025-04-26 20:35:45,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:45,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:45,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3917
3412
3372
[WARNING|trainer.py:803] 2025-04-26 20:35:46,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:46,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:46,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3918
3413
3373
[WARNING|trainer.py:803] 2025-04-26 20:35:47,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:48,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:48,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3919
3414
3374
[WARNING|trainer.py:803] 2025-04-26 20:35:49,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:49,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:49,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3920
3415
3375
[WARNING|trainer.py:803] 2025-04-26 20:35:50,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:50,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:50,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3921
3416
3376
[WARNING|trainer.py:803] 2025-04-26 20:35:51,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:52,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:52,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3922
3417
3377
[WARNING|trainer.py:803] 2025-04-26 20:35:53,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:53,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:35:53,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3923
3378
3418
[WARNING|trainer.py:803] 2025-04-26 20:35:54,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:54,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:35:54,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3924
3379
3419
[WARNING|trainer.py:803] 2025-04-26 20:35:56,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 20:35:56,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:35:56,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3925
3380
3420
[WARNING|trainer.py:803] 2025-04-26 20:35:57,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:57,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:57,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3381
3926
3421
[WARNING|trainer.py:803] 2025-04-26 20:35:58,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:35:58,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:35:58,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3927
3422
3382
[WARNING|trainer.py:803] 2025-04-26 20:36:00,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:00,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:00,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3928
3423
3383
[WARNING|trainer.py:803] 2025-04-26 20:36:01,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:01,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:01,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3929
3424
3384
[WARNING|trainer.py:803] 2025-04-26 20:36:02,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:02,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:02,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3930
3425
3385
[WARNING|trainer.py:803] 2025-04-26 20:36:03,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:04,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:04,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3931
3426
3386
[WARNING|trainer.py:803] 2025-04-26 20:36:05,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:05,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:05,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3932
3427
3387
[WARNING|trainer.py:803] 2025-04-26 20:36:06,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:06,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:06,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3933
3428
3388
[WARNING|trainer.py:803] 2025-04-26 20:36:07,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:07,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:08,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3429
3934
3389
[WARNING|trainer.py:803] 2025-04-26 20:36:09,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:09,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:09,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3430
3935
3390
[WARNING|trainer.py:803] 2025-04-26 20:36:10,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:10,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:10,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3431
3936
3391
[WARNING|trainer.py:803] 2025-04-26 20:36:11,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:11,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:12,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3937
3432
3392
[WARNING|trainer.py:803] 2025-04-26 20:36:13,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:13,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:13,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3938
3433
3393
[WARNING|trainer.py:803] 2025-04-26 20:36:14,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:14,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:14,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3939
3434
3394
[WARNING|trainer.py:803] 2025-04-26 20:36:15,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:15,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:16,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3940
3435
3395
[WARNING|trainer.py:803] 2025-04-26 20:36:17,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:17,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:17,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3941
3436
3396
[WARNING|trainer.py:803] 2025-04-26 20:36:18,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:18,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:18,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3942
3437
3397
[WARNING|trainer.py:803] 2025-04-26 20:36:19,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:19,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:20,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3438
3943
3398
[WARNING|trainer.py:803] 2025-04-26 20:36:21,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:21,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:21,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3439
3944
3399
[WARNING|trainer.py:803] 2025-04-26 20:36:22,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:22,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:22,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3440
3945
3400
[WARNING|trainer.py:803] 2025-04-26 20:36:23,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:23,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:24,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3441
3946
3401
[WARNING|trainer.py:803] 2025-04-26 20:36:25,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:25,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:25,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3442
3947
3402
[WARNING|trainer.py:803] 2025-04-26 20:36:26,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:26,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:26,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3443
3948
3403
[WARNING|trainer.py:803] 2025-04-26 20:36:27,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:27,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:28,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
3444
3949
3404
[WARNING|trainer.py:803] 2025-04-26 20:36:29,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:29,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:29,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3445
3950
3405
[WARNING|trainer.py:803] 2025-04-26 20:36:30,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:30,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:30,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3446
3951
3406
[WARNING|trainer.py:803] 2025-04-26 20:36:31,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:31,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:31,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3447
3952
3407
[WARNING|trainer.py:803] 2025-04-26 20:36:33,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:33,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:33,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3448
3408
3953
[WARNING|trainer.py:803] 2025-04-26 20:36:34,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:34,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:34,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3449
3409
3954
[WARNING|trainer.py:803] 2025-04-26 20:36:35,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:35,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:36,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3450
3410
3955
[WARNING|trainer.py:803] 2025-04-26 20:36:37,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:37,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:37,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3451
3411
3956
[WARNING|trainer.py:803] 2025-04-26 20:36:38,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:38,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:38,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3452
3412
3957
[WARNING|trainer.py:803] 2025-04-26 20:36:39,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:39,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:40,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3453
3413
3958
[WARNING|trainer.py:803] 2025-04-26 20:36:41,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:41,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:41,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3454
3414
3959
[WARNING|trainer.py:803] 2025-04-26 20:36:42,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:42,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:42,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3455
3415
3960
[WARNING|trainer.py:803] 2025-04-26 20:36:43,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:43,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:43,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3456
3416
3961
[WARNING|trainer.py:803] 2025-04-26 20:36:45,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:45,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:45,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3457
3417
3962
[WARNING|trainer.py:803] 2025-04-26 20:36:46,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:46,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:36:46,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3458
3963
3418
[WARNING|trainer.py:803] 2025-04-26 20:36:47,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:47,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:47,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3459
3964
3419
[WARNING|trainer.py:803] 2025-04-26 20:36:48,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:49,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:49,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3460
3420
3965
[WARNING|trainer.py:803] 2025-04-26 20:36:50,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:50,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:50,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3461
3966
3421
[WARNING|trainer.py:803] 2025-04-26 20:36:51,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:51,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:51,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3462
3967
3422
[WARNING|trainer.py:803] 2025-04-26 20:36:52,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:53,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:53,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3463
3423
3968
[WARNING|trainer.py:803] 2025-04-26 20:36:54,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:54,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:54,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3464
3424
3969
[WARNING|trainer.py:803] 2025-04-26 20:36:55,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:55,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:55,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3465
3425
3970
[WARNING|trainer.py:803] 2025-04-26 20:36:56,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:57,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:36:57,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3466
3426
3971
[WARNING|trainer.py:803] 2025-04-26 20:36:58,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:36:58,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:36:58,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3467
3427
3972
[WARNING|trainer.py:803] 2025-04-26 20:36:59,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:59,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:36:59,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3468
NoYes
3428
3973
[WARNING|trainer.py:803] 2025-04-26 20:37:00,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:01,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3469
[WARNING|trainer.py:803] 2025-04-26 20:37:01,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3429
3974
[WARNING|trainer.py:803] 2025-04-26 20:37:02,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:02,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:02,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3470
3430
3975
[WARNING|trainer.py:803] 2025-04-26 20:37:03,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:03,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3471
[WARNING|trainer.py:803] 2025-04-26 20:37:03,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3431
3976
[WARNING|trainer.py:803] 2025-04-26 20:37:04,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:05,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3472
[WARNING|trainer.py:803] 2025-04-26 20:37:05,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3432
3977
[WARNING|trainer.py:803] 2025-04-26 20:37:05,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3473
[WARNING|trainer.py:803] 2025-04-26 20:37:06,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:06,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3433
3978
[WARNING|trainer.py:803] 2025-04-26 20:37:07,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3474
[WARNING|trainer.py:803] 2025-04-26 20:37:07,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:07,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3434
3979
[WARNING|trainer.py:803] 2025-04-26 20:37:08,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3475
[WARNING|trainer.py:803] 2025-04-26 20:37:08,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:09,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3435
3980
[WARNING|trainer.py:803] 2025-04-26 20:37:09,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3476
[WARNING|trainer.py:803] 2025-04-26 20:37:10,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:10,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3436
3981
[WARNING|trainer.py:803] 2025-04-26 20:37:11,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3477
[WARNING|trainer.py:803] 2025-04-26 20:37:11,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:11,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3437
3982
[WARNING|trainer.py:803] 2025-04-26 20:37:12,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3478
[WARNING|trainer.py:803] 2025-04-26 20:37:12,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:12,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3438
3983
[WARNING|trainer.py:803] 2025-04-26 20:37:13,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3479
[WARNING|trainer.py:803] 2025-04-26 20:37:14,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:14,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3439
3984
[WARNING|trainer.py:803] 2025-04-26 20:37:14,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3480
[WARNING|trainer.py:803] 2025-04-26 20:37:15,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:15,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3440
3985
[WARNING|trainer.py:803] 2025-04-26 20:37:16,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3481
[WARNING|trainer.py:803] 2025-04-26 20:37:16,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:17,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3441
3986
[WARNING|trainer.py:803] 2025-04-26 20:37:17,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3482
[WARNING|trainer.py:803] 2025-04-26 20:37:18,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:18,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3442
3987
[WARNING|trainer.py:803] 2025-04-26 20:37:18,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3483
[WARNING|trainer.py:803] 2025-04-26 20:37:19,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:19,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3443
3988
[WARNING|trainer.py:803] 2025-04-26 20:37:20,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3484
[WARNING|trainer.py:803] 2025-04-26 20:37:20,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:20,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3989
3444
[WARNING|trainer.py:803] 2025-04-26 20:37:21,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3485
[WARNING|trainer.py:803] 2025-04-26 20:37:22,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:22,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3990
3445
[WARNING|trainer.py:803] 2025-04-26 20:37:22,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3486
[WARNING|trainer.py:803] 2025-04-26 20:37:23,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:23,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3991
3446
[WARNING|trainer.py:803] 2025-04-26 20:37:24,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3487
[WARNING|trainer.py:803] 2025-04-26 20:37:24,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:24,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3447
3992
[WARNING|trainer.py:803] 2025-04-26 20:37:25,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3488
[WARNING|trainer.py:803] 2025-04-26 20:37:26,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:26,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3993
3448
[WARNING|trainer.py:803] 2025-04-26 20:37:26,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3489
[WARNING|trainer.py:803] 2025-04-26 20:37:27,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:27,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3994
3449
[WARNING|trainer.py:803] 2025-04-26 20:37:27,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3490
[WARNING|trainer.py:803] 2025-04-26 20:37:28,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:28,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3995
3450
[WARNING|trainer.py:803] 2025-04-26 20:37:29,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3491
[WARNING|trainer.py:803] 2025-04-26 20:37:29,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:29,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3996
[WARNING|trainer.py:803] 2025-04-26 20:37:30,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3451
3492
[WARNING|trainer.py:803] 2025-04-26 20:37:31,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:31,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3997
[WARNING|trainer.py:803] 2025-04-26 20:37:31,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3452
3493
[WARNING|trainer.py:803] 2025-04-26 20:37:32,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:32,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3998
[WARNING|trainer.py:803] 2025-04-26 20:37:33,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3453
3494
[WARNING|trainer.py:803] 2025-04-26 20:37:33,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:33,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3999
[WARNING|trainer.py:803] 2025-04-26 20:37:34,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3454
3495
[WARNING|trainer.py:803] 2025-04-26 20:37:35,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:35,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4000
[WARNING|trainer.py:803] 2025-04-26 20:37:35,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3455
3496
[WARNING|trainer.py:803] 2025-04-26 20:37:36,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:36,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4001
[WARNING|trainer.py:803] 2025-04-26 20:37:37,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3456
3497
[WARNING|trainer.py:803] 2025-04-26 20:37:37,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:37,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:38,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4002
3457
3498
[WARNING|trainer.py:803] 2025-04-26 20:37:39,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:39,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:39,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4003
3458
3499
[WARNING|trainer.py:803] 2025-04-26 20:37:40,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:40,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:40,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4004
3459
3500
[WARNING|trainer.py:803] 2025-04-26 20:37:41,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:41,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:42,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4005
3460
3501
[WARNING|trainer.py:803] 2025-04-26 20:37:43,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:43,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:43,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4006
3461
3502
[WARNING|trainer.py:803] 2025-04-26 20:37:44,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:44,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:44,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4007
3462
3503
[WARNING|trainer.py:803] 2025-04-26 20:37:45,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:45,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:46,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4008
3463
3504
[WARNING|trainer.py:803] 2025-04-26 20:37:47,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:47,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:47,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4009
3464
3505
[WARNING|trainer.py:803] 2025-04-26 20:37:48,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:48,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:48,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4010
3465
3506
[WARNING|trainer.py:803] 2025-04-26 20:37:49,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:49,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:49,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4011
3466
3507
[WARNING|trainer.py:803] 2025-04-26 20:37:50,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:51,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:51,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4012
3467
3508
[WARNING|trainer.py:803] 2025-04-26 20:37:52,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:52,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:52,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4013
3468
3509
[WARNING|trainer.py:803] 2025-04-26 20:37:53,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:37:53,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4014[WARNING|trainer.py:803] 2025-04-26 20:37:53,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

:Yes
3469
3510
[WARNING|trainer.py:803] 2025-04-26 20:37:54,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:55,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:37:55,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4015
3470
3511
[WARNING|trainer.py:803] 2025-04-26 20:37:56,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:56,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:56,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4016
3471
3512
[WARNING|trainer.py:803] 2025-04-26 20:37:57,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:57,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:57,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4017
3472
3513
[WARNING|trainer.py:803] 2025-04-26 20:37:58,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:37:59,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:37:59,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4018
3473
3514
[WARNING|trainer.py:803] 2025-04-26 20:38:00,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:00,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:00,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3474
3515
4019
[WARNING|trainer.py:803] 2025-04-26 20:38:01,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:01,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:01,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3475
3516
4020
[WARNING|trainer.py:803] 2025-04-26 20:38:02,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:03,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:03,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3476
3517
4021
[WARNING|trainer.py:803] 2025-04-26 20:38:04,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:04,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:04,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3477
3518
4022
[WARNING|trainer.py:803] 2025-04-26 20:38:05,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:05,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3478
[WARNING|trainer.py:803] 2025-04-26 20:38:06,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3519
4023
[WARNING|trainer.py:803] 2025-04-26 20:38:06,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:06,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3479
3520
[WARNING|trainer.py:803] 2025-04-26 20:38:07,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4024
[WARNING|trainer.py:803] 2025-04-26 20:38:08,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:08,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3480
3521
[WARNING|trainer.py:803] 2025-04-26 20:38:08,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4025
[WARNING|trainer.py:803] 2025-04-26 20:38:09,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:09,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3481
3522
[WARNING|trainer.py:803] 2025-04-26 20:38:10,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:10,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4026
[WARNING|trainer.py:803] 2025-04-26 20:38:10,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3482
3523
[WARNING|trainer.py:803] 2025-04-26 20:38:11,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:12,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:12,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4027
3483
3524
[WARNING|trainer.py:803] 2025-04-26 20:38:13,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:13,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:13,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4028
3484
3525
[WARNING|trainer.py:803] 2025-04-26 20:38:14,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:14,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:14,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4029
3485
3526
[WARNING|trainer.py:803] 2025-04-26 20:38:15,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:15,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:16,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4030
3486
3527
[WARNING|trainer.py:803] 2025-04-26 20:38:17,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:17,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:17,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4031
3487
3528
[WARNING|trainer.py:803] 2025-04-26 20:38:18,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:18,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:18,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4032
3488
3529
[WARNING|trainer.py:803] 2025-04-26 20:38:19,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:19,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:19,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4033
3489
3530
[WARNING|trainer.py:803] 2025-04-26 20:38:20,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:21,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:21,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x8975f900] moov atom not found
[20:38:21] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
4034
3531
3490
[WARNING|trainer.py:803] 2025-04-26 20:38:22,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:22,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:22,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4035
3532
3491
[WARNING|trainer.py:803] 2025-04-26 20:38:23,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:23,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:23,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4036
3533
3492
[WARNING|trainer.py:803] 2025-04-26 20:38:24,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:25,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:25,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4037
3534
3493
[WARNING|trainer.py:803] 2025-04-26 20:38:26,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:26,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:26,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4038
3535
3494
[WARNING|trainer.py:803] 2025-04-26 20:38:27,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:27,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:27,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4039
3536
3495
[WARNING|trainer.py:803] 2025-04-26 20:38:28,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:28,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:29,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4040
3537
3496
[WARNING|trainer.py:803] 2025-04-26 20:38:30,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:30,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:30,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4041
3538
3497
[WARNING|trainer.py:803] 2025-04-26 20:38:31,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:31,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4042
[WARNING|trainer.py:803] 2025-04-26 20:38:31,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3539
3498
[WARNING|trainer.py:803] 2025-04-26 20:38:32,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:32,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:33,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4043
3540
3499
[WARNING|trainer.py:803] 2025-04-26 20:38:34,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:34,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:34,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4044
3541
3500
[WARNING|trainer.py:803] 2025-04-26 20:38:35,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:35,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:35,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4045
3542
3501
[WARNING|trainer.py:803] 2025-04-26 20:38:36,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:36,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:37,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4046
3543
3502
[WARNING|trainer.py:803] 2025-04-26 20:38:37,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:37,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:38,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4047
3544
3503
[WARNING|trainer.py:803] 2025-04-26 20:38:39,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:39,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:39,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3545
4048
3504
[WARNING|trainer.py:803] 2025-04-26 20:38:40,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:40,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:40,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3546
4049
3505
[WARNING|trainer.py:803] 2025-04-26 20:38:41,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:42,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:42,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3547
4050
3506
[WARNING|trainer.py:803] 2025-04-26 20:38:43,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:43,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:38:43,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3548
4051
3507
[WARNING|trainer.py:803] 2025-04-26 20:38:44,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:38:44,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:44,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3549
4052
3508
[WARNING|trainer.py:803] 2025-04-26 20:38:45,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:45,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:46,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3550
4053
3509
[WARNING|trainer.py:803] 2025-04-26 20:38:47,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:47,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:47,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3551
4054
3510
[WARNING|trainer.py:803] 2025-04-26 20:38:48,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:48,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:38:48,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3552
4055
3511
[WARNING|trainer.py:803] 2025-04-26 20:38:49,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:49,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:50,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3553
3512
4056
[WARNING|trainer.py:803] 2025-04-26 20:38:50,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:51,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:51,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3554
3513
4057
[WARNING|trainer.py:803] 2025-04-26 20:38:52,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:52,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3555
[WARNING|trainer.py:803] 2025-04-26 20:38:52,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3514
4058
[WARNING|trainer.py:803] 2025-04-26 20:38:53,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:53,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3556
[WARNING|trainer.py:803] 2025-04-26 20:38:54,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3515
4059
[WARNING|trainer.py:803] 2025-04-26 20:38:54,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:55,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3557
[WARNING|trainer.py:803] 2025-04-26 20:38:55,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3516
4060
[WARNING|trainer.py:803] 2025-04-26 20:38:56,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:56,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3558
[WARNING|trainer.py:803] 2025-04-26 20:38:56,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3517
4061
[WARNING|trainer.py:803] 2025-04-26 20:38:57,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:57,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3559
[WARNING|trainer.py:803] 2025-04-26 20:38:57,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3518
4062
[WARNING|trainer.py:803] 2025-04-26 20:38:58,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:38:59,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3560
[WARNING|trainer.py:803] 2025-04-26 20:38:59,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3519
4063
[WARNING|trainer.py:803] 2025-04-26 20:38:59,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:00,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3561
[WARNING|trainer.py:803] 2025-04-26 20:39:00,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3520
[WARNING|trainer.py:803] 2025-04-26 20:39:01,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4064
[WARNING|trainer.py:803] 2025-04-26 20:39:01,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3562
3521
[WARNING|trainer.py:803] 2025-04-26 20:39:02,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:02,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4065
[WARNING|trainer.py:803] 2025-04-26 20:39:02,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3563
3522
[WARNING|trainer.py:803] 2025-04-26 20:39:03,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:03,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4066
[WARNING|trainer.py:803] 2025-04-26 20:39:04,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3564
3523
[WARNING|trainer.py:803] 2025-04-26 20:39:04,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:05,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4067
[WARNING|trainer.py:803] 2025-04-26 20:39:05,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3565
3524
[WARNING|trainer.py:803] 2025-04-26 20:39:06,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:06,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4068
[WARNING|trainer.py:803] 2025-04-26 20:39:06,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3566
3525
[WARNING|trainer.py:803] 2025-04-26 20:39:07,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:07,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4069
[WARNING|trainer.py:803] 2025-04-26 20:39:08,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3567
3526
[WARNING|trainer.py:803] 2025-04-26 20:39:08,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:09,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4070
[WARNING|trainer.py:803] 2025-04-26 20:39:09,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3568
3527
[WARNING|trainer.py:803] 2025-04-26 20:39:10,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:10,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4071
[WARNING|trainer.py:803] 2025-04-26 20:39:10,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3569
3528
[WARNING|trainer.py:803] 2025-04-26 20:39:11,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:11,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4072
[WARNING|trainer.py:803] 2025-04-26 20:39:11,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3570
3529
[WARNING|trainer.py:803] 2025-04-26 20:39:12,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:39:12,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4073
[WARNING|trainer.py:803] 2025-04-26 20:39:13,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3571
3530
[WARNING|trainer.py:803] 2025-04-26 20:39:13,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:14,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4074
[WARNING|trainer.py:803] 2025-04-26 20:39:14,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3b860a80] moov atom not found
[20:39:14] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Latte/10231.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
3572
3531
[WARNING|trainer.py:803] 2025-04-26 20:39:15,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4075
[WARNING|trainer.py:803] 2025-04-26 20:39:15,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:15,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3573
3532
[WARNING|trainer.py:803] 2025-04-26 20:39:16,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:16,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4076
[WARNING|trainer.py:803] 2025-04-26 20:39:17,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3574
3533
[WARNING|trainer.py:803] 2025-04-26 20:39:17,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:18,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:18,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4077
3575
3534
[WARNING|trainer.py:803] 2025-04-26 20:39:19,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:19,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:19,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4078
3576
3535
[WARNING|trainer.py:803] 2025-04-26 20:39:20,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:20,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:20,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4079
3577
3536
[WARNING|trainer.py:803] 2025-04-26 20:39:21,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:22,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:22,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4080
3578
3537
[WARNING|trainer.py:803] 2025-04-26 20:39:23,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:23,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:23,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4081
3579
3538
[WARNING|trainer.py:803] 2025-04-26 20:39:24,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:24,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:24,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4082
3580
3539
[WARNING|trainer.py:803] 2025-04-26 20:39:25,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:25,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:26,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4083
3581
3540
[WARNING|trainer.py:803] 2025-04-26 20:39:27,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:27,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:27,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4084
3582
3541
[WARNING|trainer.py:803] 2025-04-26 20:39:28,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:28,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:28,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4085
3583
3542
[WARNING|trainer.py:803] 2025-04-26 20:39:29,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:29,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:29,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4086
3584
3543
[WARNING|trainer.py:803] 2025-04-26 20:39:31,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:31,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:31,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3585
3544
4087
[WARNING|trainer.py:803] 2025-04-26 20:39:32,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:32,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:32,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3586
3545
4088
[WARNING|trainer.py:803] 2025-04-26 20:39:33,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:33,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:34,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3587
3546
4089
[WARNING|trainer.py:803] 2025-04-26 20:39:35,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:35,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:35,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3588
3547
4090
[WARNING|trainer.py:803] 2025-04-26 20:39:36,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:36,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:36,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3589
3548
4091
[WARNING|trainer.py:803] 2025-04-26 20:39:37,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:37,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:38,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3549
3590
4092
[WARNING|trainer.py:803] 2025-04-26 20:39:39,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:39,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:39,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3550
3591
4093
[WARNING|trainer.py:803] 2025-04-26 20:39:40,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:40,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:40,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3551
3592
4094
[WARNING|trainer.py:803] 2025-04-26 20:39:41,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:41,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:42,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3552
3593
4095
[WARNING|trainer.py:803] 2025-04-26 20:39:42,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:42,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3553
3594
[WARNING|trainer.py:803] 2025-04-26 20:39:43,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4096
[WARNING|trainer.py:803] 2025-04-26 20:39:44,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:44,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3554
3595
[WARNING|trainer.py:803] 2025-04-26 20:39:44,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4097
[WARNING|trainer.py:803] 2025-04-26 20:39:45,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:45,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3555
3596
[WARNING|trainer.py:803] 2025-04-26 20:39:46,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:39:46,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4098
[WARNING|trainer.py:803] 2025-04-26 20:39:46,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3556
3597
[WARNING|trainer.py:803] 2025-04-26 20:39:47,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:39:48,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4099
[WARNING|trainer.py:803] 2025-04-26 20:39:48,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3557
3598
[WARNING|trainer.py:803] 2025-04-26 20:39:48,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:49,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:49,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4100
3558
3599
[WARNING|trainer.py:803] 2025-04-26 20:39:50,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:50,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:50,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4101
3559
3600
[WARNING|trainer.py:803] 2025-04-26 20:39:51,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:51,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:52,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4102
3560
3601
[WARNING|trainer.py:803] 2025-04-26 20:39:52,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:53,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:53,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4103
3561
3602
[WARNING|trainer.py:803] 2025-04-26 20:39:54,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:39:54,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:39:54,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4104
3562
3603
[WARNING|trainer.py:803] 2025-04-26 20:39:55,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:55,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4105
[WARNING|trainer.py:803] 2025-04-26 20:39:56,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3563
3604
[WARNING|trainer.py:803] 2025-04-26 20:39:56,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:57,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4106
[WARNING|trainer.py:803] 2025-04-26 20:39:57,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3564
3605
[WARNING|trainer.py:803] 2025-04-26 20:39:58,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:58,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4107
[WARNING|trainer.py:803] 2025-04-26 20:39:58,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3565
3606
[WARNING|trainer.py:803] 2025-04-26 20:39:59,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:39:59,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4108
[WARNING|trainer.py:803] 2025-04-26 20:40:00,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3566
3607
[WARNING|trainer.py:803] 2025-04-26 20:40:00,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:01,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4109
[WARNING|trainer.py:803] 2025-04-26 20:40:01,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3567
3608
[WARNING|trainer.py:803] 2025-04-26 20:40:02,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:02,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4110
[WARNING|trainer.py:803] 2025-04-26 20:40:02,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3568
3609
[WARNING|trainer.py:803] 2025-04-26 20:40:03,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:03,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4111
[WARNING|trainer.py:803] 2025-04-26 20:40:03,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3569
3610
[WARNING|trainer.py:803] 2025-04-26 20:40:04,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:04,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4112
[WARNING|trainer.py:803] 2025-04-26 20:40:05,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3570
3611
[WARNING|trainer.py:803] 2025-04-26 20:40:05,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:06,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4113
3571
[WARNING|trainer.py:803] 2025-04-26 20:40:06,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3612
[WARNING|trainer.py:803] 2025-04-26 20:40:07,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:07,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4114
3572
[WARNING|trainer.py:803] 2025-04-26 20:40:07,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3613
[WARNING|trainer.py:803] 2025-04-26 20:40:08,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:08,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4115
3573
[WARNING|trainer.py:803] 2025-04-26 20:40:09,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3614
[WARNING|trainer.py:803] 2025-04-26 20:40:09,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:10,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4116
3574
[WARNING|trainer.py:803] 2025-04-26 20:40:10,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3615
[WARNING|trainer.py:803] 2025-04-26 20:40:11,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:11,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4117
3575
[WARNING|trainer.py:803] 2025-04-26 20:40:11,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:12,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3616
[WARNING|trainer.py:803] 2025-04-26 20:40:12,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4118
3576
[WARNING|trainer.py:803] 2025-04-26 20:40:13,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:13,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3617
[WARNING|trainer.py:803] 2025-04-26 20:40:13,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4119
3577
[WARNING|trainer.py:803] 2025-04-26 20:40:14,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:15,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3618
[WARNING|trainer.py:803] 2025-04-26 20:40:15,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3578
4120
[WARNING|trainer.py:803] 2025-04-26 20:40:15,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3619
[WARNING|trainer.py:803] 2025-04-26 20:40:16,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:16,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4121
3579
[WARNING|trainer.py:803] 2025-04-26 20:40:17,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:17,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3620
[WARNING|trainer.py:803] 2025-04-26 20:40:17,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4122
3580
[WARNING|trainer.py:803] 2025-04-26 20:40:18,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3621
[WARNING|trainer.py:803] 2025-04-26 20:40:19,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:19,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3581
4123
[WARNING|trainer.py:803] 2025-04-26 20:40:19,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3622
[WARNING|trainer.py:803] 2025-04-26 20:40:20,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:20,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3582
4124
[WARNING|trainer.py:803] 2025-04-26 20:40:21,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3623
[WARNING|trainer.py:803] 2025-04-26 20:40:21,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:21,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4125
3583
[WARNING|trainer.py:803] 2025-04-26 20:40:22,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:22,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:23,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3624
4126
3584
[WARNING|trainer.py:803] 2025-04-26 20:40:23,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:24,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:24,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3625
4127
3585
[WARNING|trainer.py:803] 2025-04-26 20:40:25,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:25,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:25,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3626
4128
3586
[WARNING|trainer.py:803] 2025-04-26 20:40:26,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:26,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:26,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3627
4129
3587
[WARNING|trainer.py:803] 2025-04-26 20:40:27,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:28,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:28,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3628
4130
3588
[WARNING|trainer.py:803] 2025-04-26 20:40:29,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:29,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:29,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4131
3629
3589
[WARNING|trainer.py:803] 2025-04-26 20:40:30,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:30,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:30,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4132
3630
3590
[WARNING|trainer.py:803] 2025-04-26 20:40:31,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:32,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4133
[WARNING|trainer.py:803] 2025-04-26 20:40:32,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3631
3591
[WARNING|trainer.py:803] 2025-04-26 20:40:33,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:33,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4134
[WARNING|trainer.py:803] 2025-04-26 20:40:33,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3632
3592
[WARNING|trainer.py:803] 2025-04-26 20:40:34,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:34,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4135
[WARNING|trainer.py:803] 2025-04-26 20:40:34,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3633
3593
[WARNING|trainer.py:803] 2025-04-26 20:40:35,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:36,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4136
[WARNING|trainer.py:803] 2025-04-26 20:40:36,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3634
3594
[WARNING|trainer.py:803] 2025-04-26 20:40:36,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:37,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4137
[WARNING|trainer.py:803] 2025-04-26 20:40:37,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3635
3595
[WARNING|trainer.py:803] 2025-04-26 20:40:38,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:38,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4138
[WARNING|trainer.py:803] 2025-04-26 20:40:38,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3636
3596
[WARNING|trainer.py:803] 2025-04-26 20:40:39,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4139
[WARNING|trainer.py:803] 2025-04-26 20:40:39,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:40,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3597
3637
[WARNING|trainer.py:803] 2025-04-26 20:40:40,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4140
[WARNING|trainer.py:803] 2025-04-26 20:40:41,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:41,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3598
3638
[WARNING|trainer.py:803] 2025-04-26 20:40:42,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4141
[WARNING|trainer.py:803] 2025-04-26 20:40:42,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:42,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3599
[WARNING|trainer.py:803] 2025-04-26 20:40:43,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3639
4142
[WARNING|trainer.py:803] 2025-04-26 20:40:44,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:44,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3600
[WARNING|trainer.py:803] 2025-04-26 20:40:44,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3640
4143
[WARNING|trainer.py:803] 2025-04-26 20:40:45,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:45,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3601
[WARNING|trainer.py:803] 2025-04-26 20:40:45,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3641
4144
[WARNING|trainer.py:803] 2025-04-26 20:40:46,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:46,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3602
[WARNING|trainer.py:803] 2025-04-26 20:40:47,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3642
4145
[WARNING|trainer.py:803] 2025-04-26 20:40:47,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:40:48,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:48,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3603
3643
4146
[WARNING|trainer.py:803] 2025-04-26 20:40:49,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:49,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3604
[WARNING|trainer.py:803] 2025-04-26 20:40:49,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3644
4147
[WARNING|trainer.py:803] 2025-04-26 20:40:50,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:50,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3605
[WARNING|trainer.py:803] 2025-04-26 20:40:51,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3645
4148
[WARNING|trainer.py:803] 2025-04-26 20:40:51,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:52,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3606
[WARNING|trainer.py:803] 2025-04-26 20:40:52,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3646
4149
[WARNING|trainer.py:803] 2025-04-26 20:40:53,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3607
[WARNING|trainer.py:803] 2025-04-26 20:40:53,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:53,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3647
4150
[WARNING|trainer.py:803] 2025-04-26 20:40:54,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3608
[WARNING|trainer.py:803] 2025-04-26 20:40:55,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:55,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4151
3648
[WARNING|trainer.py:803] 2025-04-26 20:40:55,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:40:56,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3609
[WARNING|trainer.py:803] 2025-04-26 20:40:56,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4152
3649
[WARNING|trainer.py:803] 2025-04-26 20:40:57,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:57,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3610
[WARNING|trainer.py:803] 2025-04-26 20:40:57,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4153
3650
[WARNING|trainer.py:803] 2025-04-26 20:40:58,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:40:58,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3611
[WARNING|trainer.py:803] 2025-04-26 20:40:59,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4154
3651
[WARNING|trainer.py:803] 2025-04-26 20:40:59,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:40:59,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3612
4155
[WARNING|trainer.py:803] 2025-04-26 20:41:00,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3652
[WARNING|trainer.py:803] 2025-04-26 20:41:01,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:01,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4156
3613
[WARNING|trainer.py:803] 2025-04-26 20:41:01,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3653
[WARNING|trainer.py:803] 2025-04-26 20:41:02,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:02,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4157
3614
[WARNING|trainer.py:803] 2025-04-26 20:41:03,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3654
[WARNING|trainer.py:803] 2025-04-26 20:41:03,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:03,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4158
3615
[WARNING|trainer.py:803] 2025-04-26 20:41:04,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3655
[WARNING|trainer.py:803] 2025-04-26 20:41:05,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:05,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4159
3616
[WARNING|trainer.py:803] 2025-04-26 20:41:05,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3656
[WARNING|trainer.py:803] 2025-04-26 20:41:06,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:06,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4160
3617
[WARNING|trainer.py:803] 2025-04-26 20:41:07,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:07,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3657
4161
[WARNING|trainer.py:803] 2025-04-26 20:41:08,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:08,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3618
[WARNING|trainer.py:803] 2025-04-26 20:41:08,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3658
4162
[WARNING|trainer.py:803] 2025-04-26 20:41:09,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:09,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3619
[WARNING|trainer.py:803] 2025-04-26 20:41:10,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3659
[WARNING|trainer.py:803] 2025-04-26 20:41:10,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4163
[WARNING|trainer.py:803] 2025-04-26 20:41:11,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3620
[WARNING|trainer.py:803] 2025-04-26 20:41:11,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3660
4164
[WARNING|trainer.py:803] 2025-04-26 20:41:12,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:12,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3621
[WARNING|trainer.py:803] 2025-04-26 20:41:12,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3661
4165
[WARNING|trainer.py:803] 2025-04-26 20:41:13,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:13,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3622
[WARNING|trainer.py:803] 2025-04-26 20:41:14,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3662
4166
[WARNING|trainer.py:803] 2025-04-26 20:41:14,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3623
[WARNING|trainer.py:803] 2025-04-26 20:41:15,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:15,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3663
4167
[WARNING|trainer.py:803] 2025-04-26 20:41:16,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:16,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3624
[WARNING|trainer.py:803] 2025-04-26 20:41:16,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3664
4168
[WARNING|trainer.py:803] 2025-04-26 20:41:17,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:17,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3625
[WARNING|trainer.py:803] 2025-04-26 20:41:18,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3665
4169
[WARNING|trainer.py:803] 2025-04-26 20:41:18,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:19,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:19,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3626
3666
4170
[WARNING|trainer.py:803] 2025-04-26 20:41:20,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:41:20,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:20,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3627
3667
4171
[WARNING|trainer.py:803] 2025-04-26 20:41:21,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:41:22,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:22,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3628
4172
3668
[WARNING|trainer.py:803] 2025-04-26 20:41:23,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:23,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:23,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3629
4173
3669
[WARNING|trainer.py:803] 2025-04-26 20:41:24,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:41:24,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3630
4174
3670
[WARNING|trainer.py:803] 2025-04-26 20:41:25,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:41:26,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:26,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3631
4175
3671
[WARNING|trainer.py:803] 2025-04-26 20:41:27,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:27,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:27,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3632
4176
3672
[WARNING|trainer.py:803] 2025-04-26 20:41:28,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:28,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:29,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4177
3633
3673
[WARNING|trainer.py:803] 2025-04-26 20:41:29,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:29,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4178
[WARNING|trainer.py:803] 2025-04-26 20:41:30,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3634
3674
[WARNING|trainer.py:803] 2025-04-26 20:41:31,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:31,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4179
[WARNING|trainer.py:803] 2025-04-26 20:41:31,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3635
3675
[WARNING|trainer.py:803] 2025-04-26 20:41:32,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:32,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4180
[WARNING|trainer.py:803] 2025-04-26 20:41:33,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3636
[WARNING|trainer.py:803] 2025-04-26 20:41:33,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3676
[WARNING|trainer.py:803] 2025-04-26 20:41:33,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4181
[WARNING|trainer.py:803] 2025-04-26 20:41:34,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3637
3677
[WARNING|trainer.py:803] 2025-04-26 20:41:34,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:35,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4182
[WARNING|trainer.py:803] 2025-04-26 20:41:35,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3638
3678
[WARNING|trainer.py:803] 2025-04-26 20:41:36,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4183
[WARNING|trainer.py:803] 2025-04-26 20:41:36,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:36,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3639
[WARNING|trainer.py:803] 2025-04-26 20:41:37,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3679
4184
[WARNING|trainer.py:803] 2025-04-26 20:41:38,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:41:38,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3640
[WARNING|trainer.py:803] 2025-04-26 20:41:38,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3680
4185
[WARNING|trainer.py:803] 2025-04-26 20:41:39,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:39,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3641
[WARNING|trainer.py:803] 2025-04-26 20:41:40,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3681
4186
[WARNING|trainer.py:803] 2025-04-26 20:41:40,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:41,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3642
[WARNING|trainer.py:803] 2025-04-26 20:41:41,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3682
4187
[WARNING|trainer.py:803] 2025-04-26 20:41:42,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:42,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3643
[WARNING|trainer.py:803] 2025-04-26 20:41:42,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3683
4188
[WARNING|trainer.py:803] 2025-04-26 20:41:43,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:43,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:44,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3644
4189
3684
[WARNING|trainer.py:803] 2025-04-26 20:41:45,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:45,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:45,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3645
3685
4190
[WARNING|trainer.py:803] 2025-04-26 20:41:46,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:46,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3646
[WARNING|trainer.py:803] 2025-04-26 20:41:46,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4191
3686
[WARNING|trainer.py:803] 2025-04-26 20:41:47,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:48,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:48,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3647
4192
3687
[WARNING|trainer.py:803] 2025-04-26 20:41:49,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:49,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:49,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3648
4193
3688
[WARNING|trainer.py:803] 2025-04-26 20:41:50,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:50,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:50,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3649
4194
3689
[WARNING|trainer.py:803] 2025-04-26 20:41:51,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:52,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:52,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4195
3650
3690
[WARNING|trainer.py:803] 2025-04-26 20:41:53,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:53,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:41:53,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4196
3651
3691
[WARNING|trainer.py:803] 2025-04-26 20:41:54,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:54,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:54,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4197
3652
3692
[WARNING|trainer.py:803] 2025-04-26 20:41:55,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:56,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:56,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4198
3653
3693
[WARNING|trainer.py:803] 2025-04-26 20:41:57,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:57,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:57,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4199
3654
3694
[WARNING|trainer.py:803] 2025-04-26 20:41:58,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:41:58,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:41:58,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4200
3655
3695
[WARNING|trainer.py:803] 2025-04-26 20:41:59,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4201
[WARNING|trainer.py:803] 2025-04-26 20:42:00,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:00,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3656
3696
[WARNING|trainer.py:803] 2025-04-26 20:42:00,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4202
[WARNING|trainer.py:803] 2025-04-26 20:42:01,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:01,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:02,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3657
3697
4203
[WARNING|trainer.py:803] 2025-04-26 20:42:02,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:03,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:03,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4204
3658
3698
[WARNING|trainer.py:803] 2025-04-26 20:42:04,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:04,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4205
[WARNING|trainer.py:803] 2025-04-26 20:42:04,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3659
3699
[WARNING|trainer.py:803] 2025-04-26 20:42:05,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4206
[WARNING|trainer.py:803] 2025-04-26 20:42:05,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:05,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3660
3700
[WARNING|trainer.py:803] 2025-04-26 20:42:06,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4207
[WARNING|trainer.py:803] 2025-04-26 20:42:06,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:07,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:07,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4208
3661
3701
[WARNING|trainer.py:803] 2025-04-26 20:42:08,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:08,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:42:08,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4209
3662
3702
[WARNING|trainer.py:803] 2025-04-26 20:42:09,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4210
[WARNING|trainer.py:803] 2025-04-26 20:42:09,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:09,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3663
3703
[WARNING|trainer.py:803] 2025-04-26 20:42:10,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4211
[WARNING|trainer.py:803] 2025-04-26 20:42:11,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:11,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:11,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3664
4212
3704
[WARNING|trainer.py:803] 2025-04-26 20:42:12,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:12,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:12,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4213
3665
3705
[WARNING|trainer.py:803] 2025-04-26 20:42:13,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:13,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4214
[WARNING|trainer.py:803] 2025-04-26 20:42:14,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3666
[WARNING|trainer.py:803] 2025-04-26 20:42:14,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3706
4215
[WARNING|trainer.py:803] 2025-04-26 20:42:15,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:15,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3667
[WARNING|trainer.py:803] 2025-04-26 20:42:15,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4216
3707
[WARNING|trainer.py:803] 2025-04-26 20:42:16,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:16,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:17,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4217
3668
3708
[WARNING|trainer.py:803] 2025-04-26 20:42:18,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:18,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4218
[WARNING|trainer.py:803] 2025-04-26 20:42:18,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3669
3709
[WARNING|trainer.py:803] 2025-04-26 20:42:19,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4219
[WARNING|trainer.py:803] 2025-04-26 20:42:19,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:19,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3670
[WARNING|trainer.py:803] 2025-04-26 20:42:20,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4220
3710
[WARNING|trainer.py:803] 2025-04-26 20:42:20,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:21,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3671
4221
[WARNING|trainer.py:803] 2025-04-26 20:42:21,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3711
[WARNING|trainer.py:803] 2025-04-26 20:42:22,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:22,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4222
3672
[WARNING|trainer.py:803] 2025-04-26 20:42:22,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:23,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3712
4223
[WARNING|trainer.py:803] 2025-04-26 20:42:23,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3673
[WARNING|trainer.py:803] 2025-04-26 20:42:24,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:24,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4224
3713
[WARNING|trainer.py:803] 2025-04-26 20:42:24,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3674
[WARNING|trainer.py:803] 2025-04-26 20:42:25,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:25,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4225
[WARNING|trainer.py:803] 2025-04-26 20:42:26,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3714
[WARNING|trainer.py:803] 2025-04-26 20:42:26,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4226
3675
[WARNING|trainer.py:803] 2025-04-26 20:42:26,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:27,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3715
[WARNING|trainer.py:803] 2025-04-26 20:42:27,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4227
3676
[WARNING|trainer.py:803] 2025-04-26 20:42:28,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:28,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4228
3716
[WARNING|trainer.py:803] 2025-04-26 20:42:28,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3677
[WARNING|trainer.py:803] 2025-04-26 20:42:29,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:29,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4229
[WARNING|trainer.py:803] 2025-04-26 20:42:30,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3717
[WARNING|trainer.py:803] 2025-04-26 20:42:30,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3678
4230
[WARNING|trainer.py:803] 2025-04-26 20:42:31,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:42:31,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3718
[WARNING|trainer.py:803] 2025-04-26 20:42:31,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4231
3679
[WARNING|trainer.py:803] 2025-04-26 20:42:32,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:42:32,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:32,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4232
3719
3680
[WARNING|trainer.py:803] 2025-04-26 20:42:34,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:34,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4233
[WARNING|trainer.py:803] 2025-04-26 20:42:34,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3720
3681
[WARNING|trainer.py:803] 2025-04-26 20:42:35,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4234
[WARNING|trainer.py:803] 2025-04-26 20:42:35,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:35,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:36,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3721
3682
4235
[WARNING|trainer.py:803] 2025-04-26 20:42:37,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:37,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:37,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4236
3722
3683
[WARNING|trainer.py:803] 2025-04-26 20:42:38,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:38,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4237
[WARNING|trainer.py:803] 2025-04-26 20:42:38,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3723
3684
[WARNING|trainer.py:803] 2025-04-26 20:42:39,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4238
[WARNING|trainer.py:803] 2025-04-26 20:42:39,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:40,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:40,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3724
4239
3685
[WARNING|trainer.py:803] 2025-04-26 20:42:41,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:41,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:41,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4240
3725
3686
[WARNING|trainer.py:803] 2025-04-26 20:42:42,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:42,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4241
[WARNING|trainer.py:803] 2025-04-26 20:42:42,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3726
3687
[WARNING|trainer.py:803] 2025-04-26 20:42:43,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4242
[WARNING|trainer.py:803] 2025-04-26 20:42:44,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:44,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3727
[WARNING|trainer.py:803] 2025-04-26 20:42:44,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3688
4243
[WARNING|trainer.py:803] 2025-04-26 20:42:45,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:45,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:45,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4244
3728
3689
[WARNING|trainer.py:803] 2025-04-26 20:42:46,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:46,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:46,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4245
3729
3690
[WARNING|trainer.py:803] 2025-04-26 20:42:47,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4246
[WARNING|trainer.py:803] 2025-04-26 20:42:48,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:48,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3730
3691
[WARNING|trainer.py:803] 2025-04-26 20:42:48,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4247
[WARNING|trainer.py:803] 2025-04-26 20:42:49,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:42:49,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:49,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3731
4248
3692
[WARNING|trainer.py:803] 2025-04-26 20:42:50,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:42:51,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:51,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4249
3732
3693
[WARNING|trainer.py:803] 2025-04-26 20:42:52,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:52,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4250
[WARNING|trainer.py:803] 2025-04-26 20:42:52,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3733
3694
[WARNING|trainer.py:803] 2025-04-26 20:42:53,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4251
[WARNING|trainer.py:803] 2025-04-26 20:42:53,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:53,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3734
[WARNING|trainer.py:803] 2025-04-26 20:42:54,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4252
3695
[WARNING|trainer.py:803] 2025-04-26 20:42:54,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:55,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:55,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4253
3735
3696
[WARNING|trainer.py:803] 2025-04-26 20:42:56,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:42:56,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4254
[WARNING|trainer.py:803] 2025-04-26 20:42:56,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3736
3697
[WARNING|trainer.py:803] 2025-04-26 20:42:57,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4255
[WARNING|trainer.py:803] 2025-04-26 20:42:57,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:58,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3737
[WARNING|trainer.py:803] 2025-04-26 20:42:58,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3698
4256
[WARNING|trainer.py:803] 2025-04-26 20:42:59,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:42:59,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:42:59,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3738
4257
3699
[WARNING|trainer.py:803] 2025-04-26 20:43:00,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:00,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:00,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4258
3739
3700
[WARNING|trainer.py:803] 2025-04-26 20:43:01,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4259
[WARNING|trainer.py:803] 2025-04-26 20:43:01,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:02,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3740
3701
[WARNING|trainer.py:803] 2025-04-26 20:43:02,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4260
[WARNING|trainer.py:803] 2025-04-26 20:43:03,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:03,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:43:03,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3741
4261
3702
[WARNING|trainer.py:803] 2025-04-26 20:43:04,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:04,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:04,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4262
3742
3703
[WARNING|trainer.py:803] 2025-04-26 20:43:05,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:06,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4263
[WARNING|trainer.py:803] 2025-04-26 20:43:06,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3743
3704
[WARNING|trainer.py:803] 2025-04-26 20:43:07,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4264
[WARNING|trainer.py:803] 2025-04-26 20:43:07,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:07,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:08,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3744
3705
4265
[WARNING|trainer.py:803] 2025-04-26 20:43:09,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:09,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:09,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4266
3745
3706
[WARNING|trainer.py:803] 2025-04-26 20:43:10,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:10,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4267
[WARNING|trainer.py:803] 2025-04-26 20:43:10,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3746
3707
[WARNING|trainer.py:803] 2025-04-26 20:43:11,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4268
[WARNING|trainer.py:803] 2025-04-26 20:43:11,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:12,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3747
[WARNING|trainer.py:803] 2025-04-26 20:43:12,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4269
3708
[WARNING|trainer.py:803] 2025-04-26 20:43:13,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:13,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:13,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4270
3748
3709
[WARNING|trainer.py:803] 2025-04-26 20:43:14,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:14,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4271
[WARNING|trainer.py:803] 2025-04-26 20:43:14,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3749
[WARNING|trainer.py:803] 2025-04-26 20:43:15,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3710
4272
[WARNING|trainer.py:803] 2025-04-26 20:43:15,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:16,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3750
[WARNING|trainer.py:803] 2025-04-26 20:43:16,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4273
3711
[WARNING|trainer.py:803] 2025-04-26 20:43:17,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:17,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3751
[WARNING|trainer.py:803] 2025-04-26 20:43:17,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4274
3712
[WARNING|trainer.py:803] 2025-04-26 20:43:18,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:18,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4275
[WARNING|trainer.py:803] 2025-04-26 20:43:19,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3752
3713
[WARNING|trainer.py:803] 2025-04-26 20:43:19,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4276
[WARNING|trainer.py:803] 2025-04-26 20:43:20,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:20,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3753
[WARNING|trainer.py:803] 2025-04-26 20:43:20,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3714
4277
[WARNING|trainer.py:803] 2025-04-26 20:43:21,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:21,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:21,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3754
4278
3715
[WARNING|trainer.py:803] 2025-04-26 20:43:22,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:23,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4279
3755
[WARNING|trainer.py:803] 2025-04-26 20:43:23,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3716
[WARNING|trainer.py:803] 2025-04-26 20:43:24,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:24,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4280
3756
[WARNING|trainer.py:803] 2025-04-26 20:43:24,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:25,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3717
4281
[WARNING|trainer.py:803] 2025-04-26 20:43:25,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3757
[WARNING|trainer.py:803] 2025-04-26 20:43:26,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:26,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4282
[WARNING|trainer.py:803] 2025-04-26 20:43:26,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3718
[WARNING|trainer.py:803] 2025-04-26 20:43:27,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3758
4283
[WARNING|trainer.py:803] 2025-04-26 20:43:27,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:43:28,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3719
[WARNING|trainer.py:803] 2025-04-26 20:43:28,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4284
3759
[WARNING|trainer.py:803] 2025-04-26 20:43:29,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:29,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:29,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4285
3720
3760
[WARNING|trainer.py:803] 2025-04-26 20:43:30,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:30,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4286
[WARNING|trainer.py:803] 2025-04-26 20:43:31,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3721
[WARNING|trainer.py:803] 2025-04-26 20:43:31,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3761
4287
[WARNING|trainer.py:803] 2025-04-26 20:43:32,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:32,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:32,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3722
4288
3762
[WARNING|trainer.py:803] 2025-04-26 20:43:33,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:33,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:33,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4289
3723
3763
[WARNING|trainer.py:803] 2025-04-26 20:43:34,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4290
[WARNING|trainer.py:803] 2025-04-26 20:43:34,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:35,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3724
[WARNING|trainer.py:803] 2025-04-26 20:43:35,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3764
4291
[WARNING|trainer.py:803] 2025-04-26 20:43:36,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:36,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:36,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3725
4292
3765
[WARNING|trainer.py:803] 2025-04-26 20:43:37,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:37,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:38,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4293
3726
3766
[WARNING|trainer.py:803] 2025-04-26 20:43:38,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:39,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4294
[WARNING|trainer.py:803] 2025-04-26 20:43:39,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3727
3767
[WARNING|trainer.py:803] 2025-04-26 20:43:40,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4295
[WARNING|trainer.py:803] 2025-04-26 20:43:40,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:40,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3728
[WARNING|trainer.py:803] 2025-04-26 20:43:41,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3768
4296
[WARNING|trainer.py:803] 2025-04-26 20:43:41,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:42,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:42,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4297
3729
3769
[WARNING|trainer.py:803] 2025-04-26 20:43:43,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:43,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:43,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4298
3730
3770
[WARNING|trainer.py:803] 2025-04-26 20:43:44,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4299
[WARNING|trainer.py:803] 2025-04-26 20:43:44,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:44,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
3731
3771
[WARNING|trainer.py:803] 2025-04-26 20:43:45,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4300
[WARNING|trainer.py:803] 2025-04-26 20:43:46,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:46,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:46,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3732
3772
4301
[WARNING|trainer.py:803] 2025-04-26 20:43:47,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:47,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:47,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4302
3773
3733
[WARNING|trainer.py:803] 2025-04-26 20:43:48,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:48,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:48,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4303
3734
3774
[WARNING|trainer.py:803] 2025-04-26 20:43:49,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4304
[WARNING|trainer.py:803] 2025-04-26 20:43:50,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:50,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3775
3735
[WARNING|trainer.py:803] 2025-04-26 20:43:50,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4305
[WARNING|trainer.py:803] 2025-04-26 20:43:51,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:43:51,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:52,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3736
3776
4306
[WARNING|trainer.py:803] 2025-04-26 20:43:53,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:53,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:53,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4307
3777
3737
[WARNING|trainer.py:803] 2025-04-26 20:43:54,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:54,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4308
[WARNING|trainer.py:803] 2025-04-26 20:43:54,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3778
3738
[WARNING|trainer.py:803] 2025-04-26 20:43:55,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4309
[WARNING|trainer.py:803] 2025-04-26 20:43:55,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:55,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3779
[WARNING|trainer.py:803] 2025-04-26 20:43:56,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3739
4310
[WARNING|trainer.py:803] 2025-04-26 20:43:57,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:43:57,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:57,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4311
3780
3740
[WARNING|trainer.py:803] 2025-04-26 20:43:58,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:43:58,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:43:58,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4312
3781
3741
[WARNING|trainer.py:803] 2025-04-26 20:43:59,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4313
[WARNING|trainer.py:803] 2025-04-26 20:43:59,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:00,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3782
[WARNING|trainer.py:803] 2025-04-26 20:44:00,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3742
4314
[WARNING|trainer.py:803] 2025-04-26 20:44:01,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:01,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:01,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4315
3783
3743
[WARNING|trainer.py:803] 2025-04-26 20:44:02,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:02,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:02,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4316
3784
3744
[WARNING|trainer.py:803] 2025-04-26 20:44:03,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4317
[WARNING|trainer.py:803] 2025-04-26 20:44:04,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:04,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3785
[WARNING|trainer.py:803] 2025-04-26 20:44:04,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3745
4318
[WARNING|trainer.py:803] 2025-04-26 20:44:05,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:05,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:06,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3786
4319
3746
[WARNING|trainer.py:803] 2025-04-26 20:44:07,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:07,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:07,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4320
3787
3747
[WARNING|trainer.py:803] 2025-04-26 20:44:08,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:08,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4321
[WARNING|trainer.py:803] 2025-04-26 20:44:08,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3748
3788
[WARNING|trainer.py:803] 2025-04-26 20:44:09,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4322
[WARNING|trainer.py:803] 2025-04-26 20:44:09,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:09,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:10,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3749
3789
4323
[WARNING|trainer.py:803] 2025-04-26 20:44:11,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:11,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:11,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4324
3790
3750
[WARNING|trainer.py:803] 2025-04-26 20:44:12,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:12,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:12,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4325
3791
3751
[WARNING|trainer.py:803] 2025-04-26 20:44:13,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4326
[WARNING|trainer.py:803] 2025-04-26 20:44:13,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:44:14,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3792
[WARNING|trainer.py:803] 2025-04-26 20:44:14,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3752
4327
[WARNING|trainer.py:803] 2025-04-26 20:44:15,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:15,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:15,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3793
4328
3753
[WARNING|trainer.py:803] 2025-04-26 20:44:16,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:16,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:16,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4329
3794
3754
[WARNING|trainer.py:803] 2025-04-26 20:44:17,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:18,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4330
[WARNING|trainer.py:803] 2025-04-26 20:44:18,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3795
3755
[WARNING|trainer.py:803] 2025-04-26 20:44:18,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4331
[WARNING|trainer.py:803] 2025-04-26 20:44:19,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:19,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3756
3796
[WARNING|trainer.py:803] 2025-04-26 20:44:20,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4332
[WARNING|trainer.py:803] 2025-04-26 20:44:20,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:20,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:21,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3757
4333
3797
[WARNING|trainer.py:803] 2025-04-26 20:44:22,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:22,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:22,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4334
3798
3758
[WARNING|trainer.py:803] 2025-04-26 20:44:23,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4335
[WARNING|trainer.py:803] 2025-04-26 20:44:23,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:23,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3799
3759
[WARNING|trainer.py:803] 2025-04-26 20:44:24,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4336
[WARNING|trainer.py:803] 2025-04-26 20:44:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:25,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:25,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3800
4337
3760
[WARNING|trainer.py:803] 2025-04-26 20:44:26,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:26,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:26,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4338
3801
3761
[WARNING|trainer.py:803] 2025-04-26 20:44:27,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:27,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4339
[WARNING|trainer.py:803] 2025-04-26 20:44:27,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3802
3762
[WARNING|trainer.py:803] 2025-04-26 20:44:28,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4340
[WARNING|trainer.py:803] 2025-04-26 20:44:29,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:29,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3803
[WARNING|trainer.py:803] 2025-04-26 20:44:29,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4341
3763
[WARNING|trainer.py:803] 2025-04-26 20:44:30,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:30,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:30,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3804
4342
3764
[WARNING|trainer.py:803] 2025-04-26 20:44:31,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:31,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4343
3805
[WARNING|trainer.py:803] 2025-04-26 20:44:32,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3765
[WARNING|trainer.py:803] 2025-04-26 20:44:32,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:33,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4344
3806
[WARNING|trainer.py:803] 2025-04-26 20:44:33,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:33,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3766
4345
[WARNING|trainer.py:803] 2025-04-26 20:44:34,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3807
[WARNING|trainer.py:803] 2025-04-26 20:44:34,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:35,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4346
3767
[WARNING|trainer.py:803] 2025-04-26 20:44:35,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:36,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3808
4347
[WARNING|trainer.py:803] 2025-04-26 20:44:36,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3768
[WARNING|trainer.py:803] 2025-04-26 20:44:36,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:37,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4348
3809
[WARNING|trainer.py:803] 2025-04-26 20:44:37,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:38,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3769
[WARNING|trainer.py:803] 2025-04-26 20:44:38,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4349
3810
[WARNING|trainer.py:803] 2025-04-26 20:44:39,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:39,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4350
3770
[WARNING|trainer.py:803] 2025-04-26 20:44:39,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3811
[WARNING|trainer.py:803] 2025-04-26 20:44:40,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:40,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4351
[WARNING|trainer.py:803] 2025-04-26 20:44:40,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3771
[WARNING|trainer.py:803] 2025-04-26 20:44:41,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3812
4352
[WARNING|trainer.py:803] 2025-04-26 20:44:41,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:42,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3772
[WARNING|trainer.py:803] 2025-04-26 20:44:42,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4353
3813
[WARNING|trainer.py:803] 2025-04-26 20:44:43,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:43,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:43,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4354
3773
3814
[WARNING|trainer.py:803] 2025-04-26 20:44:44,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:44,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4355
[WARNING|trainer.py:803] 2025-04-26 20:44:44,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3774
3815
[WARNING|trainer.py:803] 2025-04-26 20:44:45,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4356
[WARNING|trainer.py:803] 2025-04-26 20:44:45,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:46,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:46,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3775
4357
3816
[WARNING|trainer.py:803] 2025-04-26 20:44:47,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:47,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:44:47,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4358
3817
3776
[WARNING|trainer.py:803] 2025-04-26 20:44:48,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4359
[WARNING|trainer.py:803] 2025-04-26 20:44:48,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:48,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3818
3777
[WARNING|trainer.py:803] 2025-04-26 20:44:49,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4360
[WARNING|trainer.py:803] 2025-04-26 20:44:50,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:50,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:50,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3819
4361
3778
[WARNING|trainer.py:803] 2025-04-26 20:44:51,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:51,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:44:51,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4362
3820
3779
[WARNING|trainer.py:803] 2025-04-26 20:44:52,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4363
[WARNING|trainer.py:803] 2025-04-26 20:44:52,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:52,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3821
[WARNING|trainer.py:803] 2025-04-26 20:44:53,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3780
4364
[WARNING|trainer.py:803] 2025-04-26 20:44:54,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:54,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:54,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3822
4365
3781
[WARNING|trainer.py:803] 2025-04-26 20:44:55,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:55,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4366
[WARNING|trainer.py:803] 2025-04-26 20:44:55,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3823
3782
[WARNING|trainer.py:803] 2025-04-26 20:44:56,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4367
[WARNING|trainer.py:803] 2025-04-26 20:44:56,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3824
[WARNING|trainer.py:803] 2025-04-26 20:44:57,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:57,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4368
3783
[WARNING|trainer.py:803] 2025-04-26 20:44:58,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:44:58,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3825
[WARNING|trainer.py:803] 2025-04-26 20:44:58,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4369
3784
[WARNING|trainer.py:803] 2025-04-26 20:44:59,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:44:59,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4370
3826
[WARNING|trainer.py:803] 2025-04-26 20:45:00,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3785
[WARNING|trainer.py:803] 2025-04-26 20:45:00,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:00,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4371
3827
[WARNING|trainer.py:803] 2025-04-26 20:45:01,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:01,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4372
3786
[WARNING|trainer.py:803] 2025-04-26 20:45:01,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3828
[WARNING|trainer.py:803] 2025-04-26 20:45:02,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4373
[WARNING|trainer.py:803] 2025-04-26 20:45:02,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:03,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3787
[WARNING|trainer.py:803] 2025-04-26 20:45:03,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3829
4374
[WARNING|trainer.py:803] 2025-04-26 20:45:04,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:04,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:04,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4375
3788
3830
[WARNING|trainer.py:803] 2025-04-26 20:45:05,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:05,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4376
[WARNING|trainer.py:803] 2025-04-26 20:45:05,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3831
3789
[WARNING|trainer.py:803] 2025-04-26 20:45:06,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4377
[WARNING|trainer.py:803] 2025-04-26 20:45:07,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:07,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3832
3790
[WARNING|trainer.py:803] 2025-04-26 20:45:07,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4378
[WARNING|trainer.py:803] 2025-04-26 20:45:08,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:08,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:08,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4379
3833
3791
[WARNING|trainer.py:803] 2025-04-26 20:45:09,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:09,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:09,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4380
3834
3792
[WARNING|trainer.py:803] 2025-04-26 20:45:10,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4381
[WARNING|trainer.py:803] 2025-04-26 20:45:10,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:11,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3835
3793
[WARNING|trainer.py:803] 2025-04-26 20:45:11,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4382
[WARNING|trainer.py:803] 2025-04-26 20:45:12,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:12,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3836
[WARNING|trainer.py:803] 2025-04-26 20:45:12,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4383
3794
[WARNING|trainer.py:803] 2025-04-26 20:45:13,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:13,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4384
[WARNING|trainer.py:803] 2025-04-26 20:45:13,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3837
3795
[WARNING|trainer.py:803] 2025-04-26 20:45:14,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:14,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4385
[WARNING|trainer.py:803] 2025-04-26 20:45:15,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3838
[WARNING|trainer.py:803] 2025-04-26 20:45:15,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3796
4386
[WARNING|trainer.py:803] 2025-04-26 20:45:16,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3839
[WARNING|trainer.py:803] 2025-04-26 20:45:16,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:16,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4387
3797
[WARNING|trainer.py:803] 2025-04-26 20:45:17,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:17,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3840
4388
[WARNING|trainer.py:803] 2025-04-26 20:45:18,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:18,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3798
[WARNING|trainer.py:803] 2025-04-26 20:45:18,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4389
3841
[WARNING|trainer.py:803] 2025-04-26 20:45:19,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:19,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3799
[WARNING|trainer.py:803] 2025-04-26 20:45:19,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4390
3842
[WARNING|trainer.py:803] 2025-04-26 20:45:20,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:20,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4391
[WARNING|trainer.py:803] 2025-04-26 20:45:21,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3800
3843
[WARNING|trainer.py:803] 2025-04-26 20:45:21,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4392
[WARNING|trainer.py:803] 2025-04-26 20:45:22,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:22,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3801
[WARNING|trainer.py:803] 2025-04-26 20:45:22,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4393
3844
[WARNING|trainer.py:803] 2025-04-26 20:45:23,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:23,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:23,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4394
3802
3845
[WARNING|trainer.py:803] 2025-04-26 20:45:24,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:25,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4395
[WARNING|trainer.py:803] 2025-04-26 20:45:25,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3803
3846
[WARNING|trainer.py:803] 2025-04-26 20:45:25,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4396
[WARNING|trainer.py:803] 2025-04-26 20:45:26,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:26,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3804
3847
[WARNING|trainer.py:803] 2025-04-26 20:45:26,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4397
[WARNING|trainer.py:803] 2025-04-26 20:45:27,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:27,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:27,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3805
4398
3848
[WARNING|trainer.py:803] 2025-04-26 20:45:28,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:28,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
[WARNING|trainer.py:803] 2025-04-26 20:45:29,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4399
3806
3849
[WARNING|trainer.py:803] 2025-04-26 20:45:29,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:30,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4400
[WARNING|trainer.py:803] 2025-04-26 20:45:30,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3807
3850
[WARNING|trainer.py:803] 2025-04-26 20:45:30,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4401
[WARNING|trainer.py:803] 2025-04-26 20:45:31,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:31,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3808
[WARNING|trainer.py:803] 2025-04-26 20:45:31,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3851
4402
[WARNING|trainer.py:803] 2025-04-26 20:45:32,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:32,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:32,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4403
3809
3852
[WARNING|trainer.py:803] 2025-04-26 20:45:34,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:34,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:34,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4404
3810
3853
[WARNING|trainer.py:803] 2025-04-26 20:45:35,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4405
[WARNING|trainer.py:803] 2025-04-26 20:45:35,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:35,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3811
3854
[WARNING|trainer.py:803] 2025-04-26 20:45:36,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4406
[WARNING|trainer.py:803] 2025-04-26 20:45:36,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:36,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:37,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3855
3812
4407
[WARNING|trainer.py:803] 2025-04-26 20:45:37,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:37,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:38,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4408
3813
3856
[WARNING|trainer.py:803] 2025-04-26 20:45:39,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:39,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:39,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4409
3857
3814
[WARNING|trainer.py:803] 2025-04-26 20:45:40,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4410
[WARNING|trainer.py:803] 2025-04-26 20:45:40,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:40,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3858
3815
[WARNING|trainer.py:803] 2025-04-26 20:45:41,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4411
[WARNING|trainer.py:803] 2025-04-26 20:45:41,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:41,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:42,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3859
3816
4412
[WARNING|trainer.py:803] 2025-04-26 20:45:43,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:43,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:43,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4413
3860
3817
[WARNING|trainer.py:803] 2025-04-26 20:45:44,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4414
[WARNING|trainer.py:803] 2025-04-26 20:45:44,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:44,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3861
3818
[WARNING|trainer.py:803] 2025-04-26 20:45:45,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4415
[WARNING|trainer.py:803] 2025-04-26 20:45:45,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:45,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:46,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3862
3819
4416
[WARNING|trainer.py:803] 2025-04-26 20:45:47,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:47,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:47,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4417
3863
3820
[WARNING|trainer.py:803] 2025-04-26 20:45:48,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:48,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4418
[WARNING|trainer.py:803] 2025-04-26 20:45:48,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3864
3821
[WARNING|trainer.py:803] 2025-04-26 20:45:49,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4419
[WARNING|trainer.py:803] 2025-04-26 20:45:49,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:49,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3865
3822
[WARNING|trainer.py:803] 2025-04-26 20:45:50,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4420
[WARNING|trainer.py:803] 2025-04-26 20:45:50,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:51,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:51,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3866
4421
3823
[WARNING|trainer.py:803] 2025-04-26 20:45:52,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:52,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4422
[WARNING|trainer.py:803] 2025-04-26 20:45:52,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3867
3824
[WARNING|trainer.py:803] 2025-04-26 20:45:53,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4423
[WARNING|trainer.py:803] 2025-04-26 20:45:53,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:53,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3868
3825
[WARNING|trainer.py:803] 2025-04-26 20:45:54,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4424
[WARNING|trainer.py:803] 2025-04-26 20:45:54,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:45:55,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:45:55,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3869
4425
3826
[WARNING|trainer.py:803] 2025-04-26 20:45:56,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:56,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:56,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4426
3870
3827
[WARNING|trainer.py:803] 2025-04-26 20:45:57,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:45:57,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4427
[WARNING|trainer.py:803] 2025-04-26 20:45:57,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3871
3828
[WARNING|trainer.py:803] 2025-04-26 20:45:58,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4428
[WARNING|trainer.py:803] 2025-04-26 20:45:58,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:45:59,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3872
[WARNING|trainer.py:803] 2025-04-26 20:45:59,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3829
4429
[WARNING|trainer.py:803] 2025-04-26 20:46:00,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:00,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:00,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4430
3830
3873
[WARNING|trainer.py:803] 2025-04-26 20:46:01,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4431
[WARNING|trainer.py:803] 2025-04-26 20:46:01,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:01,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3831
3874
[WARNING|trainer.py:803] 2025-04-26 20:46:02,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4432
[WARNING|trainer.py:803] 2025-04-26 20:46:02,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:02,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3832
[WARNING|trainer.py:803] 2025-04-26 20:46:03,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3875
4433
[WARNING|trainer.py:803] 2025-04-26 20:46:04,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:04,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:04,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4434
3833
3876
[WARNING|trainer.py:803] 2025-04-26 20:46:05,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:05,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4435
[WARNING|trainer.py:803] 2025-04-26 20:46:05,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3834
3877
[WARNING|trainer.py:803] 2025-04-26 20:46:06,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4436
[WARNING|trainer.py:803] 2025-04-26 20:46:06,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:07,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3835
[WARNING|trainer.py:803] 2025-04-26 20:46:07,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3878
4437
[WARNING|trainer.py:803] 2025-04-26 20:46:08,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:08,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3836
[WARNING|trainer.py:803] 2025-04-26 20:46:08,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4438
3879
[WARNING|trainer.py:803] 2025-04-26 20:46:09,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:09,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:09,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4439
3837
3880
[WARNING|trainer.py:803] 2025-04-26 20:46:10,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:10,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4440
[WARNING|trainer.py:803] 2025-04-26 20:46:10,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3838
[WARNING|trainer.py:803] 2025-04-26 20:46:11,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3881
4441
[WARNING|trainer.py:803] 2025-04-26 20:46:11,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3839
[WARNING|trainer.py:803] 2025-04-26 20:46:12,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:12,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4442
3882
[WARNING|trainer.py:803] 2025-04-26 20:46:13,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:13,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3840
4443
[WARNING|trainer.py:803] 2025-04-26 20:46:13,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3883
[WARNING|trainer.py:803] 2025-04-26 20:46:14,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:14,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4444
3841
[WARNING|trainer.py:803] 2025-04-26 20:46:15,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:15,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3884
4445
[WARNING|trainer.py:803] 2025-04-26 20:46:15,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3842
[WARNING|trainer.py:803] 2025-04-26 20:46:16,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:16,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4446
3885
[WARNING|trainer.py:803] 2025-04-26 20:46:17,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:17,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3843
4447
[WARNING|trainer.py:803] 2025-04-26 20:46:17,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3886
[WARNING|trainer.py:803] 2025-04-26 20:46:18,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:18,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4448
3844
[WARNING|trainer.py:803] 2025-04-26 20:46:19,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:19,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3887
4449
[WARNING|trainer.py:803] 2025-04-26 20:46:19,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3845
[WARNING|trainer.py:803] 2025-04-26 20:46:20,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:20,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4450
3888
[WARNING|trainer.py:803] 2025-04-26 20:46:21,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3846
[WARNING|trainer.py:803] 2025-04-26 20:46:21,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4451
[WARNING|trainer.py:803] 2025-04-26 20:46:21,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3889
[WARNING|trainer.py:803] 2025-04-26 20:46:22,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:22,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3847
4452
[WARNING|trainer.py:803] 2025-04-26 20:46:23,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:23,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:23,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3890
4453
3848
[WARNING|trainer.py:803] 2025-04-26 20:46:24,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:24,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4454
[WARNING|trainer.py:803] 2025-04-26 20:46:24,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3891
3849
[WARNING|trainer.py:803] 2025-04-26 20:46:25,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4455
[WARNING|trainer.py:803] 2025-04-26 20:46:26,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:26,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3892
[WARNING|trainer.py:803] 2025-04-26 20:46:26,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3850
4456
[WARNING|trainer.py:803] 2025-04-26 20:46:27,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:27,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:27,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3893
4457
3851
[WARNING|trainer.py:803] 2025-04-26 20:46:28,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:28,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:28,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4458
3894
3852
[WARNING|trainer.py:803] 2025-04-26 20:46:29,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4459
[WARNING|trainer.py:803] 2025-04-26 20:46:30,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:30,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3895
3853
[WARNING|trainer.py:803] 2025-04-26 20:46:30,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4460
[WARNING|trainer.py:803] 2025-04-26 20:46:31,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:31,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:31,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3896
3854
4461
[WARNING|trainer.py:803] 2025-04-26 20:46:32,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:32,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:32,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4462
3855
3897
[WARNING|trainer.py:803] 2025-04-26 20:46:33,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:33,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4463
[WARNING|trainer.py:803] 2025-04-26 20:46:34,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3856
3898
[WARNING|trainer.py:803] 2025-04-26 20:46:34,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4464
[WARNING|trainer.py:803] 2025-04-26 20:46:35,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:35,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3857
[WARNING|trainer.py:803] 2025-04-26 20:46:35,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3899
4465
[WARNING|trainer.py:803] 2025-04-26 20:46:36,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:36,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:36,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3858
4466
3900
[WARNING|trainer.py:803] 2025-04-26 20:46:37,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:37,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4467
[WARNING|trainer.py:803] 2025-04-26 20:46:38,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3859
3901
[WARNING|trainer.py:803] 2025-04-26 20:46:38,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4468
[WARNING|trainer.py:803] 2025-04-26 20:46:39,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3860
[WARNING|trainer.py:803] 2025-04-26 20:46:39,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:39,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4469
3902
[WARNING|trainer.py:803] 2025-04-26 20:46:40,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:40,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3861
[WARNING|trainer.py:803] 2025-04-26 20:46:41,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4470
[WARNING|trainer.py:803] 2025-04-26 20:46:41,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3903
[WARNING|trainer.py:803] 2025-04-26 20:46:41,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4471
3862
[WARNING|trainer.py:803] 2025-04-26 20:46:42,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:43,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:43,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4472
3904
3863
[WARNING|trainer.py:803] 2025-04-26 20:46:44,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:44,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4473
[WARNING|trainer.py:803] 2025-04-26 20:46:44,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3905
3864
[WARNING|trainer.py:803] 2025-04-26 20:46:45,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4474
[WARNING|trainer.py:803] 2025-04-26 20:46:45,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:45,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:46,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3865
4475
3906
[WARNING|trainer.py:803] 2025-04-26 20:46:47,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:47,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:47,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4476
3866
3907
[WARNING|trainer.py:803] 2025-04-26 20:46:48,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4477
[WARNING|trainer.py:803] 2025-04-26 20:46:48,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:48,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3867
[WARNING|trainer.py:803] 2025-04-26 20:46:49,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4478
3908
[WARNING|trainer.py:803] 2025-04-26 20:46:49,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3868
[WARNING|trainer.py:803] 2025-04-26 20:46:50,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:50,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4479
[WARNING|trainer.py:803] 2025-04-26 20:46:50,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3909
[WARNING|trainer.py:803] 2025-04-26 20:46:51,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4480
3869
[WARNING|trainer.py:803] 2025-04-26 20:46:51,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:52,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:52,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4481
3910
3870
[WARNING|trainer.py:803] 2025-04-26 20:46:53,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4482
[WARNING|trainer.py:803] 2025-04-26 20:46:53,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:53,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3871
3911
[WARNING|trainer.py:803] 2025-04-26 20:46:54,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4483
[WARNING|trainer.py:803] 2025-04-26 20:46:54,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:46:54,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:55,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4484
3872
3912
[WARNING|trainer.py:803] 2025-04-26 20:46:56,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:56,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4485
[WARNING|trainer.py:803] 2025-04-26 20:46:56,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3873
3913
[WARNING|trainer.py:803] 2025-04-26 20:46:57,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4486
[WARNING|trainer.py:803] 2025-04-26 20:46:57,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:46:57,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:46:58,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3874
4487
3914
[WARNING|trainer.py:803] 2025-04-26 20:46:59,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:46:59,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4488
[WARNING|trainer.py:803] 2025-04-26 20:46:59,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3875
3915
[WARNING|trainer.py:803] 2025-04-26 20:47:00,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4489
[WARNING|trainer.py:803] 2025-04-26 20:47:00,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:47:00,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3876
[WARNING|trainer.py:803] 2025-04-26 20:47:01,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4490
3916
[WARNING|trainer.py:803] 2025-04-26 20:47:01,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:02,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3877
4491
[WARNING|trainer.py:803] 2025-04-26 20:47:02,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3917
[WARNING|trainer.py:803] 2025-04-26 20:47:03,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:03,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4492
3878
[WARNING|trainer.py:803] 2025-04-26 20:47:04,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:04,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:04,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4493
3918
3879
[WARNING|trainer.py:803] 2025-04-26 20:47:05,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:05,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4494
[WARNING|trainer.py:803] 2025-04-26 20:47:05,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3919
3880
[WARNING|trainer.py:803] 2025-04-26 20:47:06,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4495
[WARNING|trainer.py:803] 2025-04-26 20:47:07,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:07,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:47:07,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4496
3881
3920
[WARNING|trainer.py:803] 2025-04-26 20:47:08,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:08,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:08,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4497
3882
3921
[WARNING|trainer.py:803] 2025-04-26 20:47:09,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4498
[WARNING|trainer.py:803] 2025-04-26 20:47:10,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:10,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3883
[WARNING|trainer.py:803] 2025-04-26 20:47:10,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3922
4499
[WARNING|trainer.py:803] 2025-04-26 20:47:11,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:11,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:11,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3884
4500
3923
[WARNING|trainer.py:803] 2025-04-26 20:47:12,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:12,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4501
3885
[WARNING|trainer.py:803] 2025-04-26 20:47:13,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:13,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4502
[WARNING|trainer.py:803] 2025-04-26 20:47:14,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3924
3886
[WARNING|trainer.py:803] 2025-04-26 20:47:14,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4503
[WARNING|trainer.py:803] 2025-04-26 20:47:15,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:15,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3925
[WARNING|trainer.py:803] 2025-04-26 20:47:15,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4504
3887
[WARNING|trainer.py:803] 2025-04-26 20:47:16,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:16,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:16,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4505
3888
3926
[WARNING|trainer.py:803] 2025-04-26 20:47:17,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4506
[WARNING|trainer.py:803] 2025-04-26 20:47:18,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:18,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3889
[WARNING|trainer.py:803] 2025-04-26 20:47:18,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3927
4507
[WARNING|trainer.py:803] 2025-04-26 20:47:19,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:19,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:19,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3890
4508
3928
[WARNING|trainer.py:803] 2025-04-26 20:47:21,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:21,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4509
[WARNING|trainer.py:803] 2025-04-26 20:47:21,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3891
[WARNING|trainer.py:803] 2025-04-26 20:47:22,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3929
4510
[WARNING|trainer.py:803] 2025-04-26 20:47:22,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3892
[WARNING|trainer.py:803] 2025-04-26 20:47:22,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:23,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4511
[WARNING|trainer.py:803] 2025-04-26 20:47:23,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3930
[WARNING|trainer.py:803] 2025-04-26 20:47:24,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3893
4512
[WARNING|trainer.py:803] 2025-04-26 20:47:24,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:47:24,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:25,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3931
4513
3894
[WARNING|trainer.py:803] 2025-04-26 20:47:26,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:26,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:26,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4514
3895
3932
[WARNING|trainer.py:803] 2025-04-26 20:47:27,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4515
[WARNING|trainer.py:803] 2025-04-26 20:47:27,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:27,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3896
[WARNING|trainer.py:803] 2025-04-26 20:47:28,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3933
4516
[WARNING|trainer.py:803] 2025-04-26 20:47:28,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:29,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:29,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4517
3897
3934
[WARNING|trainer.py:803] 2025-04-26 20:47:30,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:30,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4518
3898
[WARNING|trainer.py:803] 2025-04-26 20:47:30,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:31,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4519
[WARNING|trainer.py:803] 2025-04-26 20:47:31,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3935
3899
[WARNING|trainer.py:803] 2025-04-26 20:47:32,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:32,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4520
[WARNING|trainer.py:803] 2025-04-26 20:47:32,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3936
[WARNING|trainer.py:803] 2025-04-26 20:47:33,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3900
4521
[WARNING|trainer.py:803] 2025-04-26 20:47:33,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:34,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:34,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3937
4522
3901
[WARNING|trainer.py:803] 2025-04-26 20:47:35,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:35,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4523
[WARNING|trainer.py:803] 2025-04-26 20:47:35,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3938
3902
[WARNING|trainer.py:803] 2025-04-26 20:47:36,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4524
[WARNING|trainer.py:803] 2025-04-26 20:47:37,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:37,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:37,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3939
4525
3903
[WARNING|trainer.py:803] 2025-04-26 20:47:38,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:38,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4526
[WARNING|trainer.py:803] 2025-04-26 20:47:38,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3940
3904
[WARNING|trainer.py:803] 2025-04-26 20:47:39,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4527
[WARNING|trainer.py:803] 2025-04-26 20:47:40,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:40,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:40,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3941
4528
3905
[WARNING|trainer.py:803] 2025-04-26 20:47:41,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:41,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:41,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4529
3942
3906
[WARNING|trainer.py:803] 2025-04-26 20:47:42,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4530
[WARNING|trainer.py:803] 2025-04-26 20:47:43,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:43,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:43,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3907
4531
3943
[WARNING|trainer.py:803] 2025-04-26 20:47:44,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:44,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:44,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4532
3908
3944
[WARNING|trainer.py:803] 2025-04-26 20:47:46,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4533
[WARNING|trainer.py:803] 2025-04-26 20:47:46,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:46,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:47,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4534
3909
3945
[WARNING|trainer.py:803] 2025-04-26 20:47:48,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:48,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:48,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4535
3910
3946
[WARNING|trainer.py:803] 2025-04-26 20:47:49,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4536
[WARNING|trainer.py:803] 2025-04-26 20:47:49,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:49,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:50,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3911
4537
3947
[WARNING|trainer.py:803] 2025-04-26 20:47:51,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:51,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:51,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4538
3912
3948
[WARNING|trainer.py:803] 2025-04-26 20:47:52,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4539
[WARNING|trainer.py:803] 2025-04-26 20:47:52,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:52,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:53,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3913
4540
3949
[WARNING|trainer.py:803] 2025-04-26 20:47:54,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:54,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:54,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4541
3914
3950
[WARNING|trainer.py:803] 2025-04-26 20:47:55,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4542
[WARNING|trainer.py:803] 2025-04-26 20:47:55,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:55,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:56,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3915
3951
4543
[WARNING|trainer.py:803] 2025-04-26 20:47:57,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:57,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:47:57,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4544
3916
3952
[WARNING|trainer.py:803] 2025-04-26 20:47:58,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4545
[WARNING|trainer.py:803] 2025-04-26 20:47:59,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:47:59,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:47:59,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3917
3953
4546
[WARNING|trainer.py:803] 2025-04-26 20:48:00,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:00,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:00,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4547
3954
3918
[WARNING|trainer.py:803] 2025-04-26 20:48:01,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4548
[WARNING|trainer.py:803] 2025-04-26 20:48:02,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:02,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:02,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3955
3919
4549
[WARNING|trainer.py:803] 2025-04-26 20:48:03,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:03,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:03,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4550
3956
3920
[WARNING|trainer.py:803] 2025-04-26 20:48:04,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4551
[WARNING|trainer.py:803] 2025-04-26 20:48:05,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:05,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3957
[WARNING|trainer.py:803] 2025-04-26 20:48:05,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3921
4552
[WARNING|trainer.py:803] 2025-04-26 20:48:06,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:06,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:06,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4553
3958
3922
[WARNING|trainer.py:803] 2025-04-26 20:48:07,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4554
[WARNING|trainer.py:803] 2025-04-26 20:48:08,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:08,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3959
[WARNING|trainer.py:803] 2025-04-26 20:48:08,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3923
4555
[WARNING|trainer.py:803] 2025-04-26 20:48:09,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:09,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:09,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4556
3960
3924
[WARNING|trainer.py:803] 2025-04-26 20:48:11,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4557
[WARNING|trainer.py:803] 2025-04-26 20:48:11,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:11,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3961
[WARNING|trainer.py:803] 2025-04-26 20:48:12,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4558
3925
[WARNING|trainer.py:803] 2025-04-26 20:48:12,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:13,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:13,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4559
3962
3926
[WARNING|trainer.py:803] 2025-04-26 20:48:14,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:14,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4560
[WARNING|trainer.py:803] 2025-04-26 20:48:14,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3963
[WARNING|trainer.py:803] 2025-04-26 20:48:15,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4561
3927
[WARNING|trainer.py:803] 2025-04-26 20:48:15,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:16,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:16,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3964
4562
3928
[WARNING|trainer.py:803] 2025-04-26 20:48:17,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:17,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4563
[WARNING|trainer.py:803] 2025-04-26 20:48:17,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3965
[WARNING|trainer.py:803] 2025-04-26 20:48:18,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3929
4564
[WARNING|trainer.py:803] 2025-04-26 20:48:18,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:19,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:19,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3966
4565
3930
[WARNING|trainer.py:803] 2025-04-26 20:48:20,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:20,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4566
[WARNING|trainer.py:803] 2025-04-26 20:48:20,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3967
[WARNING|trainer.py:803] 2025-04-26 20:48:21,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3931
4567
[WARNING|trainer.py:803] 2025-04-26 20:48:21,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:22,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:22,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3968
4568
3932
[WARNING|trainer.py:803] 2025-04-26 20:48:23,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:23,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4569
[WARNING|trainer.py:803] 2025-04-26 20:48:23,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3969
3933
[WARNING|trainer.py:803] 2025-04-26 20:48:24,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4570
[WARNING|trainer.py:803] 2025-04-26 20:48:25,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:25,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:25,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3970
4571
3934
[WARNING|trainer.py:803] 2025-04-26 20:48:26,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:26,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4572
[WARNING|trainer.py:803] 2025-04-26 20:48:27,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3971
[WARNING|trainer.py:803] 2025-04-26 20:48:27,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4573
3935
[WARNING|trainer.py:803] 2025-04-26 20:48:28,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:28,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:28,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3972
4574
3936
[WARNING|trainer.py:803] 2025-04-26 20:48:29,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:29,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4575
[WARNING|trainer.py:803] 2025-04-26 20:48:30,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3973
[WARNING|trainer.py:803] 2025-04-26 20:48:30,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3937
4576
[WARNING|trainer.py:803] 2025-04-26 20:48:31,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:31,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:31,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3974
4577
3938
[WARNING|trainer.py:803] 2025-04-26 20:48:32,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:32,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4578
[WARNING|trainer.py:803] 2025-04-26 20:48:33,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3975
[WARNING|trainer.py:803] 2025-04-26 20:48:33,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3939
4579
[WARNING|trainer.py:803] 2025-04-26 20:48:34,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:34,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:34,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4580
3976
3940
[WARNING|trainer.py:803] 2025-04-26 20:48:35,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:36,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4581
[WARNING|trainer.py:803] 2025-04-26 20:48:36,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3977
[WARNING|trainer.py:803] 2025-04-26 20:48:37,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3941
4582
[WARNING|trainer.py:803] 2025-04-26 20:48:37,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:37,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:38,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3978
4583
3942
[WARNING|trainer.py:803] 2025-04-26 20:48:38,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:39,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4584
[WARNING|trainer.py:803] 2025-04-26 20:48:39,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3979
[WARNING|trainer.py:803] 2025-04-26 20:48:40,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4585
3943
[WARNING|trainer.py:803] 2025-04-26 20:48:40,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:48:41,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3980
[WARNING|trainer.py:803] 2025-04-26 20:48:41,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4586
[WARNING|trainer.py:803] 2025-04-26 20:48:42,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3944
[WARNING|trainer.py:803] 2025-04-26 20:48:42,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4587
3981
[WARNING|trainer.py:803] 2025-04-26 20:48:42,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:43,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4588
[WARNING|trainer.py:803] 2025-04-26 20:48:43,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3945
3982
[WARNING|trainer.py:803] 2025-04-26 20:48:44,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4589
[WARNING|trainer.py:803] 2025-04-26 20:48:44,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:44,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
3946
[WARNING|trainer.py:803] 2025-04-26 20:48:45,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4590
3983
[WARNING|trainer.py:803] 2025-04-26 20:48:46,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:46,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4591
[WARNING|trainer.py:803] 2025-04-26 20:48:46,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3947
[WARNING|trainer.py:803] 2025-04-26 20:48:47,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3984
4592
[WARNING|trainer.py:803] 2025-04-26 20:48:47,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:48,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3948
[WARNING|trainer.py:803] 2025-04-26 20:48:48,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4593
3985
[WARNING|trainer.py:803] 2025-04-26 20:48:49,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:49,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4594
[WARNING|trainer.py:803] 2025-04-26 20:48:49,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3949
[WARNING|trainer.py:803] 2025-04-26 20:48:50,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3986
4595
[WARNING|trainer.py:803] 2025-04-26 20:48:50,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3950
[WARNING|trainer.py:803] 2025-04-26 20:48:51,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:48:51,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4596
3987
[WARNING|trainer.py:803] 2025-04-26 20:48:52,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:52,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4597
3951
[WARNING|trainer.py:803] 2025-04-26 20:48:52,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3988
[WARNING|trainer.py:803] 2025-04-26 20:48:53,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:53,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4598
[WARNING|trainer.py:803] 2025-04-26 20:48:54,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3952
[WARNING|trainer.py:803] 2025-04-26 20:48:54,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4599
3989
[WARNING|trainer.py:803] 2025-04-26 20:48:55,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:55,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:55,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4600
3953
3990
[WARNING|trainer.py:803] 2025-04-26 20:48:56,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:48:56,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4601
[WARNING|trainer.py:803] 2025-04-26 20:48:57,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3954
[WARNING|trainer.py:803] 2025-04-26 20:48:57,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4602
3991
[WARNING|trainer.py:803] 2025-04-26 20:48:58,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:58,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:58,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4603
3955
3992
[WARNING|trainer.py:803] 2025-04-26 20:48:59,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:48:59,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4604
[WARNING|trainer.py:803] 2025-04-26 20:49:00,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3956
[WARNING|trainer.py:803] 2025-04-26 20:49:00,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4605
3993
[WARNING|trainer.py:803] 2025-04-26 20:49:01,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:01,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:01,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3957
4606
3994
[WARNING|trainer.py:803] 2025-04-26 20:49:02,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:02,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4607
[WARNING|trainer.py:803] 2025-04-26 20:49:03,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3958
[WARNING|trainer.py:803] 2025-04-26 20:49:03,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3995
4608
[WARNING|trainer.py:803] 2025-04-26 20:49:04,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:04,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
3959
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:04,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4609
3996
[WARNING|trainer.py:803] 2025-04-26 20:49:05,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:05,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4610
[WARNING|trainer.py:803] 2025-04-26 20:49:06,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
3960
[WARNING|trainer.py:803] 2025-04-26 20:49:06,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3997
4611
[WARNING|trainer.py:803] 2025-04-26 20:49:07,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:07,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3961
[WARNING|trainer.py:803] 2025-04-26 20:49:07,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4612
3998
[WARNING|trainer.py:803] 2025-04-26 20:49:08,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:09,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4613
3962
[WARNING|trainer.py:803] 2025-04-26 20:49:09,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:10,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3999
[WARNING|trainer.py:803] 2025-04-26 20:49:10,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4614
3963
[WARNING|trainer.py:803] 2025-04-26 20:49:10,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:11,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4615
4000
[WARNING|trainer.py:803] 2025-04-26 20:49:11,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:12,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3964
4616
[WARNING|trainer.py:803] 2025-04-26 20:49:12,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:13,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4001
[WARNING|trainer.py:803] 2025-04-26 20:49:13,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4617
3965
[WARNING|trainer.py:803] 2025-04-26 20:49:14,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:14,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4618
[WARNING|trainer.py:803] 2025-04-26 20:49:14,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4002
3966
[WARNING|trainer.py:803] 2025-04-26 20:49:15,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4619
[WARNING|trainer.py:803] 2025-04-26 20:49:15,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:16,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:16,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4003
4620
3967
[WARNING|trainer.py:803] 2025-04-26 20:49:17,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:17,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4621
[WARNING|trainer.py:803] 2025-04-26 20:49:17,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4004
[WARNING|trainer.py:803] 2025-04-26 20:49:18,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3968
4622
[WARNING|trainer.py:803] 2025-04-26 20:49:18,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:19,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:19,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4005
4623
3969
[WARNING|trainer.py:803] 2025-04-26 20:49:20,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:20,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 20:49:20,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4006
[WARNING|trainer.py:803] 2025-04-26 20:49:21,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3970
4625
[WARNING|trainer.py:803] 2025-04-26 20:49:21,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:22,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:22,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4007
4626
3971
[WARNING|trainer.py:803] 2025-04-26 20:49:23,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:23,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4627
[WARNING|trainer.py:803] 2025-04-26 20:49:23,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4008
[WARNING|trainer.py:803] 2025-04-26 20:49:24,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3972
4628
[WARNING|trainer.py:803] 2025-04-26 20:49:25,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:25,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:25,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4009
4629
3973
[WARNING|trainer.py:803] 2025-04-26 20:49:26,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:26,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4630
[WARNING|trainer.py:803] 2025-04-26 20:49:27,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4010
3974
[WARNING|trainer.py:803] 2025-04-26 20:49:27,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4631
[WARNING|trainer.py:803] 2025-04-26 20:49:28,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:28,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4011
[WARNING|trainer.py:803] 2025-04-26 20:49:28,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4632
3975
[WARNING|trainer.py:803] 2025-04-26 20:49:29,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4633
4012
[WARNING|trainer.py:803] 2025-04-26 20:49:30,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:30,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3976
[WARNING|trainer.py:803] 2025-04-26 20:49:31,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4634
[WARNING|trainer.py:803] 2025-04-26 20:49:31,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4013
[WARNING|trainer.py:803] 2025-04-26 20:49:32,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4635
3977
[WARNING|trainer.py:803] 2025-04-26 20:49:32,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:33,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4636
[WARNING|trainer.py:803] 2025-04-26 20:49:33,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4014
3978
[WARNING|trainer.py:803] 2025-04-26 20:49:34,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:34,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4637
[WARNING|trainer.py:803] 2025-04-26 20:49:34,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4015
[WARNING|trainer.py:803] 2025-04-26 20:49:35,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4638
3979
[WARNING|trainer.py:803] 2025-04-26 20:49:35,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:36,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:36,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4639
4016
3980
[WARNING|trainer.py:803] 2025-04-26 20:49:37,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:37,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4640
[WARNING|trainer.py:803] 2025-04-26 20:49:37,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4017
[WARNING|trainer.py:803] 2025-04-26 20:49:38,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
3981
[WARNING|trainer.py:803] 2025-04-26 20:49:38,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:39,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:39,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4642
4018
3982
[WARNING|trainer.py:803] 2025-04-26 20:49:40,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4643
[WARNING|trainer.py:803] 2025-04-26 20:49:40,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:40,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:41,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4019
4644
3983
[WARNING|trainer.py:803] 2025-04-26 20:49:42,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:42,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:42,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4645
4020
3984
[WARNING|trainer.py:803] 2025-04-26 20:49:43,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4646
[WARNING|trainer.py:803] 2025-04-26 20:49:43,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:43,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:44,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4647
3985
4021
[WARNING|trainer.py:803] 2025-04-26 20:49:45,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:45,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:45,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4648
3986
4022
[WARNING|trainer.py:803] 2025-04-26 20:49:46,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4649
[WARNING|trainer.py:803] 2025-04-26 20:49:47,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:47,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:47,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4650
3987
4023
[WARNING|trainer.py:803] 2025-04-26 20:49:48,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:48,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4651
[WARNING|trainer.py:803] 2025-04-26 20:49:48,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3988
[WARNING|trainer.py:803] 2025-04-26 20:49:49,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4024
4652
[WARNING|trainer.py:803] 2025-04-26 20:49:50,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:49:50,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:49:50,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3989
4653
4025
[WARNING|trainer.py:803] 2025-04-26 20:49:51,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:51,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4654
[WARNING|trainer.py:803] 2025-04-26 20:49:52,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3990
[WARNING|trainer.py:803] 2025-04-26 20:49:52,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4026
4655
[WARNING|trainer.py:803] 2025-04-26 20:49:52,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:53,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3991
[WARNING|trainer.py:803] 2025-04-26 20:49:53,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4656
4027
[WARNING|trainer.py:803] 2025-04-26 20:49:54,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:54,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4657
[WARNING|trainer.py:803] 2025-04-26 20:49:55,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
3992
[WARNING|trainer.py:803] 2025-04-26 20:49:55,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4658
4028
[WARNING|trainer.py:803] 2025-04-26 20:49:56,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:56,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3993
[WARNING|trainer.py:803] 2025-04-26 20:49:56,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4659
4029
[WARNING|trainer.py:803] 2025-04-26 20:49:57,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:49:57,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4660
3994
[WARNING|trainer.py:803] 2025-04-26 20:49:58,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:58,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4661
4030
[WARNING|trainer.py:803] 2025-04-26 20:49:59,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:49:59,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3995
4662
[WARNING|trainer.py:803] 2025-04-26 20:50:00,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:00,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4031
[WARNING|trainer.py:803] 2025-04-26 20:50:00,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4663
3996
[WARNING|trainer.py:803] 2025-04-26 20:50:01,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:01,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4664
4032
[WARNING|trainer.py:803] 2025-04-26 20:50:02,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:03,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
3997
[WARNING|trainer.py:803] 2025-04-26 20:50:03,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4665
4033
[WARNING|trainer.py:803] 2025-04-26 20:50:03,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:04,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4666
3998
[WARNING|trainer.py:803] 2025-04-26 20:50:04,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:05,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4667
4034
[WARNING|trainer.py:803] 2025-04-26 20:50:05,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
3999
[WARNING|trainer.py:803] 2025-04-26 20:50:06,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:06,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4668
[WARNING|trainer.py:803] 2025-04-26 20:50:06,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4035
[WARNING|trainer.py:803] 2025-04-26 20:50:07,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4669
4000
[WARNING|trainer.py:803] 2025-04-26 20:50:07,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:08,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:08,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4036
4670
4001
[WARNING|trainer.py:803] 2025-04-26 20:50:09,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:09,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4671
[WARNING|trainer.py:803] 2025-04-26 20:50:10,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4037
[WARNING|trainer.py:803] 2025-04-26 20:50:10,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4672
4002
[WARNING|trainer.py:803] 2025-04-26 20:50:10,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:11,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:11,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4038
4673
4003
[WARNING|trainer.py:803] 2025-04-26 20:50:12,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:12,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4674
[WARNING|trainer.py:803] 2025-04-26 20:50:13,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4039
[WARNING|trainer.py:803] 2025-04-26 20:50:13,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4675
4004
[WARNING|trainer.py:803] 2025-04-26 20:50:14,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:14,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:14,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4040
4676
4005
[WARNING|trainer.py:803] 2025-04-26 20:50:15,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:15,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4677
[WARNING|trainer.py:803] 2025-04-26 20:50:16,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4041
[WARNING|trainer.py:803] 2025-04-26 20:50:16,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4006
4678
[WARNING|trainer.py:803] 2025-04-26 20:50:17,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:17,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:17,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4042
4679
4007
[WARNING|trainer.py:803] 2025-04-26 20:50:18,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:18,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4680
[WARNING|trainer.py:803] 2025-04-26 20:50:19,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4043
[WARNING|trainer.py:803] 2025-04-26 20:50:19,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4681
4008
[WARNING|trainer.py:803] 2025-04-26 20:50:20,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:20,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:20,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4044
4682
4009
[WARNING|trainer.py:803] 2025-04-26 20:50:21,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:21,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4683
[WARNING|trainer.py:803] 2025-04-26 20:50:22,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4045
[WARNING|trainer.py:803] 2025-04-26 20:50:22,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4010
4684
[WARNING|trainer.py:803] 2025-04-26 20:50:23,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:23,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:23,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4046
4685
4011
[WARNING|trainer.py:803] 2025-04-26 20:50:24,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:24,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4686
[WARNING|trainer.py:803] 2025-04-26 20:50:25,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4047
[WARNING|trainer.py:803] 2025-04-26 20:50:25,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4012
4687
[WARNING|trainer.py:803] 2025-04-26 20:50:26,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:26,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:26,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4688
4048
4013
[WARNING|trainer.py:803] 2025-04-26 20:50:28,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:28,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4689
[WARNING|trainer.py:803] 2025-04-26 20:50:28,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4049
[WARNING|trainer.py:803] 2025-04-26 20:50:29,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4014
4690
[WARNING|trainer.py:803] 2025-04-26 20:50:29,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:30,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:30,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4050
4691
4015
[WARNING|trainer.py:803] 2025-04-26 20:50:31,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:31,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4692
[WARNING|trainer.py:803] 2025-04-26 20:50:31,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4051
[WARNING|trainer.py:803] 2025-04-26 20:50:32,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4016
4693
[WARNING|trainer.py:803] 2025-04-26 20:50:32,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:33,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:33,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4052
4694
4017
[WARNING|trainer.py:803] 2025-04-26 20:50:34,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:34,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4695
[WARNING|trainer.py:803] 2025-04-26 20:50:34,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4053
[WARNING|trainer.py:803] 2025-04-26 20:50:35,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4696
4018
[WARNING|trainer.py:803] 2025-04-26 20:50:35,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:36,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4054
[WARNING|trainer.py:803] 2025-04-26 20:50:36,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4697
4019
[WARNING|trainer.py:803] 2025-04-26 20:50:37,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:37,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4698
4055
[WARNING|trainer.py:803] 2025-04-26 20:50:37,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:38,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4699
4020
[WARNING|trainer.py:803] 2025-04-26 20:50:38,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:39,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4056
[WARNING|trainer.py:803] 2025-04-26 20:50:39,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4700
[WARNING|trainer.py:803] 2025-04-26 20:50:40,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4021
[WARNING|trainer.py:803] 2025-04-26 20:50:40,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4701
4057
[WARNING|trainer.py:803] 2025-04-26 20:50:41,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:41,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4702
[WARNING|trainer.py:803] 2025-04-26 20:50:41,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4022
[WARNING|trainer.py:803] 2025-04-26 20:50:42,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4058
4703
[WARNING|trainer.py:803] 2025-04-26 20:50:42,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:43,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:43,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4023
4704
4059
[WARNING|trainer.py:803] 2025-04-26 20:50:44,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:44,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4705
[WARNING|trainer.py:803] 2025-04-26 20:50:45,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4024
[WARNING|trainer.py:803] 2025-04-26 20:50:45,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4060
4706
[WARNING|trainer.py:803] 2025-04-26 20:50:46,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:50:46,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:46,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4025
4707
4061
[WARNING|trainer.py:803] 2025-04-26 20:50:47,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:47,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4708
[WARNING|trainer.py:803] 2025-04-26 20:50:48,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4026
[WARNING|trainer.py:803] 2025-04-26 20:50:48,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4062
4709
[WARNING|trainer.py:803] 2025-04-26 20:50:49,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:49,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:49,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4710
4027
4063
[WARNING|trainer.py:803] 2025-04-26 20:50:50,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:51,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4711
[WARNING|trainer.py:803] 2025-04-26 20:50:51,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4028
[WARNING|trainer.py:803] 2025-04-26 20:50:51,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4712
4064
[WARNING|trainer.py:803] 2025-04-26 20:50:52,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:52,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:53,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4713
4029
4065
[WARNING|trainer.py:803] 2025-04-26 20:50:53,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:54,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4714
[WARNING|trainer.py:803] 2025-04-26 20:50:54,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4030
[WARNING|trainer.py:803] 2025-04-26 20:50:54,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4066
4715
[WARNING|trainer.py:803] 2025-04-26 20:50:55,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:55,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:50:55,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4716
4031
4067
[WARNING|trainer.py:803] 2025-04-26 20:50:57,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:57,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4717
[WARNING|trainer.py:803] 2025-04-26 20:50:57,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4032
[WARNING|trainer.py:803] 2025-04-26 20:50:58,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4718
4068
[WARNING|trainer.py:803] 2025-04-26 20:50:58,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:50:59,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:50:59,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4719
4033
4069
[WARNING|trainer.py:803] 2025-04-26 20:51:00,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:00,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4720
[WARNING|trainer.py:803] 2025-04-26 20:51:00,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4034
[WARNING|trainer.py:803] 2025-04-26 20:51:01,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4721
4070
[WARNING|trainer.py:803] 2025-04-26 20:51:01,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:02,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:02,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4722
4035
4071
[WARNING|trainer.py:803] 2025-04-26 20:51:03,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:03,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4723
[WARNING|trainer.py:803] 2025-04-26 20:51:03,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4036
[WARNING|trainer.py:803] 2025-04-26 20:51:04,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4072
4724
[WARNING|trainer.py:803] 2025-04-26 20:51:04,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:05,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:51:05,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4725
4037
4073
[WARNING|trainer.py:803] 2025-04-26 20:51:06,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4726
[WARNING|trainer.py:803] 2025-04-26 20:51:06,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:06,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4038
4074
[WARNING|trainer.py:803] 2025-04-26 20:51:07,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4727
[WARNING|trainer.py:803] 2025-04-26 20:51:07,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:08,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:08,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4728
4039
4075
[WARNING|trainer.py:803] 2025-04-26 20:51:09,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4729
[WARNING|trainer.py:803] 2025-04-26 20:51:09,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:09,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4040
[WARNING|trainer.py:803] 2025-04-26 20:51:10,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4076
4730
[WARNING|trainer.py:803] 2025-04-26 20:51:11,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:51:11,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:11,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4731
4041
4077
[WARNING|trainer.py:803] 2025-04-26 20:51:12,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:12,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4732
[WARNING|trainer.py:803] 2025-04-26 20:51:12,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4042
[WARNING|trainer.py:803] 2025-04-26 20:51:13,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4078
4733
[WARNING|trainer.py:803] 2025-04-26 20:51:14,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:14,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:14,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4734
4043
4079
[WARNING|trainer.py:803] 2025-04-26 20:51:15,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4735
[WARNING|trainer.py:803] 2025-04-26 20:51:15,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:15,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4044
[WARNING|trainer.py:803] 2025-04-26 20:51:16,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4080
4736
[WARNING|trainer.py:803] 2025-04-26 20:51:17,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:17,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:17,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4737
4045
4081
[WARNING|trainer.py:803] 2025-04-26 20:51:18,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4738
[WARNING|trainer.py:803] 2025-04-26 20:51:18,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:19,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4046
[WARNING|trainer.py:803] 2025-04-26 20:51:19,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4082
4739
[WARNING|trainer.py:803] 2025-04-26 20:51:20,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:20,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:20,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4740
4047
4083
[WARNING|trainer.py:803] 2025-04-26 20:51:21,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4741
[WARNING|trainer.py:803] 2025-04-26 20:51:21,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:22,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:22,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4048
4742
4084
[WARNING|trainer.py:803] 2025-04-26 20:51:23,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:23,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:23,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4743
4049
4085
[WARNING|trainer.py:803] 2025-04-26 20:51:24,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4744
[WARNING|trainer.py:803] 2025-04-26 20:51:25,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:25,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:25,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4050
4086
4745
[WARNING|trainer.py:803] 2025-04-26 20:51:26,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:51:26,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:26,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4746
4051
4087
[WARNING|trainer.py:803] 2025-04-26 20:51:27,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4747
[WARNING|trainer.py:803] 2025-04-26 20:51:28,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:28,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:28,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4052
4748
4088
[WARNING|trainer.py:803] 2025-04-26 20:51:29,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:29,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4749
[WARNING|trainer.py:803] 2025-04-26 20:51:29,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4053
4089
[WARNING|trainer.py:803] 2025-04-26 20:51:30,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4750
[WARNING|trainer.py:803] 2025-04-26 20:51:31,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:31,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:31,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4054
4751
4090
[WARNING|trainer.py:803] 2025-04-26 20:51:32,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:32,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4752
[WARNING|trainer.py:803] 2025-04-26 20:51:33,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4055
4091
[WARNING|trainer.py:803] 2025-04-26 20:51:33,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4753
[WARNING|trainer.py:803] 2025-04-26 20:51:34,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:34,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:34,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4754
4056
4092
[WARNING|trainer.py:803] 2025-04-26 20:51:35,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:35,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:36,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4755
4057
4093
[WARNING|trainer.py:803] 2025-04-26 20:51:36,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4756
[WARNING|trainer.py:803] 2025-04-26 20:51:37,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:37,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:37,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4757
4058
4094
[WARNING|trainer.py:803] 2025-04-26 20:51:38,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:39,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4758
[WARNING|trainer.py:803] 2025-04-26 20:51:39,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4059
4095
[WARNING|trainer.py:803] 2025-04-26 20:51:39,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4759
[WARNING|trainer.py:803] 2025-04-26 20:51:40,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:40,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:40,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4760
4060
4096
[WARNING|trainer.py:803] 2025-04-26 20:51:41,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4761
[WARNING|trainer.py:803] 2025-04-26 20:51:42,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:42,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4061
[WARNING|trainer.py:803] 2025-04-26 20:51:43,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4097
4762
[WARNING|trainer.py:803] 2025-04-26 20:51:43,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:51:43,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:44,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4763
4062
4098
[WARNING|trainer.py:803] 2025-04-26 20:51:45,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4764
[WARNING|trainer.py:803] 2025-04-26 20:51:45,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:45,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 20:51:46,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4063
4099
4765
[WARNING|trainer.py:803] 2025-04-26 20:51:46,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:47,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:47,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4766
4100
4064
[WARNING|trainer.py:803] 2025-04-26 20:51:48,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4767
[WARNING|trainer.py:803] 2025-04-26 20:51:48,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:48,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:49,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4768
4065
4101
[WARNING|trainer.py:803] 2025-04-26 20:51:50,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:50,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:50,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4769
4066
4102
[WARNING|trainer.py:803] 2025-04-26 20:51:51,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4770
[WARNING|trainer.py:803] 2025-04-26 20:51:51,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:51,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:52,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4771
4103
4067
[WARNING|trainer.py:803] 2025-04-26 20:51:53,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:51:53,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:53,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4772
4104
4068
[WARNING|trainer.py:803] 2025-04-26 20:51:54,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4773
[WARNING|trainer.py:803] 2025-04-26 20:51:54,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:54,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:51:55,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4774
4105
4069
[WARNING|trainer.py:803] 2025-04-26 20:51:56,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:56,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4775
[WARNING|trainer.py:803] 2025-04-26 20:51:56,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4106
4070
[WARNING|trainer.py:803] 2025-04-26 20:51:57,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4776
[WARNING|trainer.py:803] 2025-04-26 20:51:57,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:58,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:58,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4107
4777
4071
[WARNING|trainer.py:803] 2025-04-26 20:51:59,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:59,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:51:59,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4778
4108
4072
[WARNING|trainer.py:803] 2025-04-26 20:52:00,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4779
[WARNING|trainer.py:803] 2025-04-26 20:52:00,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:52:00,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:52:01,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4073
4109
4780
[WARNING|trainer.py:803] 2025-04-26 20:52:02,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:02,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:02,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4781
4074
4110
[WARNING|trainer.py:803] 2025-04-26 20:52:03,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4782
[WARNING|trainer.py:803] 2025-04-26 20:52:03,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:52:03,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4075
4111
[WARNING|trainer.py:803] 2025-04-26 20:52:04,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4783
[WARNING|trainer.py:803] 2025-04-26 20:52:05,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:05,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:05,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4784
4112
4076
[WARNING|trainer.py:803] 2025-04-26 20:52:06,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:06,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4785
[WARNING|trainer.py:803] 2025-04-26 20:52:06,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4113
4077
[WARNING|trainer.py:803] 2025-04-26 20:52:07,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4786
[WARNING|trainer.py:803] 2025-04-26 20:52:08,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:08,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:08,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4787
4114
4078
[WARNING|trainer.py:803] 2025-04-26 20:52:09,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:09,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4788
[WARNING|trainer.py:803] 2025-04-26 20:52:10,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4115
[WARNING|trainer.py:803] 2025-04-26 20:52:10,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4079
4789
[WARNING|trainer.py:803] 2025-04-26 20:52:11,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:11,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:11,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4790
4116
4080
[WARNING|trainer.py:803] 2025-04-26 20:52:12,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:12,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
[WARNING|trainer.py:803] 2025-04-26 20:52:13,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4117
[WARNING|trainer.py:803] 2025-04-26 20:52:13,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4081
4792
[WARNING|trainer.py:803] 2025-04-26 20:52:14,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:14,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:14,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4118
4793
4082
[WARNING|trainer.py:803] 2025-04-26 20:52:15,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:15,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4794
[WARNING|trainer.py:803] 2025-04-26 20:52:16,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4119
[WARNING|trainer.py:803] 2025-04-26 20:52:17,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4083
4795
[WARNING|trainer.py:803] 2025-04-26 20:52:17,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:17,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:18,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4796
4120
4084
[WARNING|trainer.py:803] 2025-04-26 20:52:19,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:19,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4797
[WARNING|trainer.py:803] 2025-04-26 20:52:19,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4121
[WARNING|trainer.py:803] 2025-04-26 20:52:20,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4085
4798
[WARNING|trainer.py:803] 2025-04-26 20:52:20,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:21,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:21,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4122
4799
4086
[WARNING|trainer.py:803] 2025-04-26 20:52:22,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:22,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4800
[WARNING|trainer.py:803] 2025-04-26 20:52:22,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4123
[WARNING|trainer.py:803] 2025-04-26 20:52:23,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4087
4801
[WARNING|trainer.py:803] 2025-04-26 20:52:23,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:24,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:24,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4124
4802
4088
[WARNING|trainer.py:803] 2025-04-26 20:52:25,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:25,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4803
4125
[WARNING|trainer.py:803] 2025-04-26 20:52:25,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:26,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4804
[WARNING|trainer.py:803] 2025-04-26 20:52:26,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4089
[WARNING|trainer.py:803] 2025-04-26 20:52:27,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4126
4805
[WARNING|trainer.py:803] 2025-04-26 20:52:27,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:28,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4090
[WARNING|trainer.py:803] 2025-04-26 20:52:28,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4806
4127
[WARNING|trainer.py:803] 2025-04-26 20:52:29,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:29,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4807
[WARNING|trainer.py:803] 2025-04-26 20:52:29,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4091
[WARNING|trainer.py:803] 2025-04-26 20:52:30,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4128
4808
[WARNING|trainer.py:803] 2025-04-26 20:52:30,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:31,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4092
[WARNING|trainer.py:803] 2025-04-26 20:52:31,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4809
4129
[WARNING|trainer.py:803] 2025-04-26 20:52:32,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:32,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4810
[WARNING|trainer.py:803] 2025-04-26 20:52:32,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4093
4130
[WARNING|trainer.py:803] 2025-04-26 20:52:33,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4811
[WARNING|trainer.py:803] 2025-04-26 20:52:33,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:34,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:34,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4094
4812
4131
[WARNING|trainer.py:803] 2025-04-26 20:52:35,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:52:35,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4813
[WARNING|trainer.py:803] 2025-04-26 20:52:35,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4095
4132
[WARNING|trainer.py:803] 2025-04-26 20:52:36,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4814
[WARNING|trainer.py:803] 2025-04-26 20:52:36,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:37,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:37,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4815
4096
4133
[WARNING|trainer.py:803] 2025-04-26 20:52:38,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:52:38,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4816
[WARNING|trainer.py:803] 2025-04-26 20:52:38,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4134
4097
[WARNING|trainer.py:803] 2025-04-26 20:52:39,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4817
[WARNING|trainer.py:803] 2025-04-26 20:52:40,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:40,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:52:40,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4818
4135
4098
[WARNING|trainer.py:803] 2025-04-26 20:52:41,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:41,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4819
[WARNING|trainer.py:803] 2025-04-26 20:52:41,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4136
[WARNING|trainer.py:803] 2025-04-26 20:52:42,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4099
4820
[WARNING|trainer.py:803] 2025-04-26 20:52:43,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:43,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:43,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4821
4137
4100
[WARNING|trainer.py:803] 2025-04-26 20:52:44,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4822
[WARNING|trainer.py:803] 2025-04-26 20:52:44,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:45,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4138
[WARNING|trainer.py:803] 2025-04-26 20:52:45,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4823
4101
[WARNING|trainer.py:803] 2025-04-26 20:52:46,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:46,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:46,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4824
4139
4102
[WARNING|trainer.py:803] 2025-04-26 20:52:47,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:47,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4825
[WARNING|trainer.py:803] 2025-04-26 20:52:48,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4140
[WARNING|trainer.py:803] 2025-04-26 20:52:48,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4826
4103
[WARNING|trainer.py:803] 2025-04-26 20:52:49,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:49,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:49,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4827
4141
4104
[WARNING|trainer.py:803] 2025-04-26 20:52:50,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:50,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4828
[WARNING|trainer.py:803] 2025-04-26 20:52:51,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4142
[WARNING|trainer.py:803] 2025-04-26 20:52:51,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4829
4105
[WARNING|trainer.py:803] 2025-04-26 20:52:52,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:52,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:52,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4830
4143
4106
[WARNING|trainer.py:803] 2025-04-26 20:52:53,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:53,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4831
[WARNING|trainer.py:803] 2025-04-26 20:52:54,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4144
[WARNING|trainer.py:803] 2025-04-26 20:52:54,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4832
4107
[WARNING|trainer.py:803] 2025-04-26 20:52:55,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:55,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:55,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4833
4145
4108
[WARNING|trainer.py:803] 2025-04-26 20:52:56,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:56,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4834
[WARNING|trainer.py:803] 2025-04-26 20:52:57,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4146
[WARNING|trainer.py:803] 2025-04-26 20:52:57,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4835
4109
[WARNING|trainer.py:803] 2025-04-26 20:52:58,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:52:58,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:58,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4836
4147
4110
[WARNING|trainer.py:803] 2025-04-26 20:52:59,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:52:59,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4837
[WARNING|trainer.py:803] 2025-04-26 20:53:00,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4148
[WARNING|trainer.py:803] 2025-04-26 20:53:00,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4111
4838
[WARNING|trainer.py:803] 2025-04-26 20:53:01,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:01,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:01,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4839
4149
4112
[WARNING|trainer.py:803] 2025-04-26 20:53:02,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:53:02,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4840
[WARNING|trainer.py:803] 2025-04-26 20:53:03,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4150
[WARNING|trainer.py:803] 2025-04-26 20:53:03,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4113
4841
[WARNING|trainer.py:803] 2025-04-26 20:53:04,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:04,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:04,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4151
4842
4114
[WARNING|trainer.py:803] 2025-04-26 20:53:05,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:05,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4843
[WARNING|trainer.py:803] 2025-04-26 20:53:06,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4152
[WARNING|trainer.py:803] 2025-04-26 20:53:06,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4115
4844
[WARNING|trainer.py:803] 2025-04-26 20:53:07,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:07,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4153
[WARNING|trainer.py:803] 2025-04-26 20:53:07,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4845
4116
[WARNING|trainer.py:803] 2025-04-26 20:53:08,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:08,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4846
[WARNING|trainer.py:803] 2025-04-26 20:53:09,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4154
[WARNING|trainer.py:803] 2025-04-26 20:53:09,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4117
4847
[WARNING|trainer.py:803] 2025-04-26 20:53:10,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:10,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4155
[WARNING|trainer.py:803] 2025-04-26 20:53:10,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4848
4118
[WARNING|trainer.py:803] 2025-04-26 20:53:11,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:11,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4849
[WARNING|trainer.py:803] 2025-04-26 20:53:12,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4156
[WARNING|trainer.py:803] 2025-04-26 20:53:12,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4119
4850
[WARNING|trainer.py:803] 2025-04-26 20:53:13,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:13,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4157
[WARNING|trainer.py:803] 2025-04-26 20:53:13,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4851
[WARNING|trainer.py:803] 2025-04-26 20:53:14,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4120
[WARNING|trainer.py:803] 2025-04-26 20:53:14,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4852
4158
[WARNING|trainer.py:803] 2025-04-26 20:53:15,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:15,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4853
4121
[WARNING|trainer.py:803] 2025-04-26 20:53:16,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:16,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:16,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4159
4854
4122
[WARNING|trainer.py:803] 2025-04-26 20:53:17,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:17,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4855
4160
[WARNING|trainer.py:803] 2025-04-26 20:53:18,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:18,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4856
4123
[WARNING|trainer.py:803] 2025-04-26 20:53:19,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4161
[WARNING|trainer.py:803] 2025-04-26 20:53:19,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:20,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4857
[WARNING|trainer.py:803] 2025-04-26 20:53:20,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4124
[WARNING|trainer.py:803] 2025-04-26 20:53:20,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4858
4162
[WARNING|trainer.py:803] 2025-04-26 20:53:21,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:21,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4859
4125
[WARNING|trainer.py:803] 2025-04-26 20:53:22,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:22,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4163
[WARNING|trainer.py:803] 2025-04-26 20:53:23,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4860
4126
[WARNING|trainer.py:803] 2025-04-26 20:53:23,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:23,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4861
4164
[WARNING|trainer.py:803] 2025-04-26 20:53:24,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:24,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4862
[WARNING|trainer.py:803] 2025-04-26 20:53:25,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4127
4165
[WARNING|trainer.py:803] 2025-04-26 20:53:26,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:26,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4863
4128
[WARNING|trainer.py:803] 2025-04-26 20:53:26,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:27,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4864
4166
[WARNING|trainer.py:803] 2025-04-26 20:53:27,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:28,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4865
[WARNING|trainer.py:803] 2025-04-26 20:53:28,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4129
4167
[WARNING|trainer.py:803] 2025-04-26 20:53:29,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:29,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4866
4130
[WARNING|trainer.py:803] 2025-04-26 20:53:29,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:30,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4867
4168
[WARNING|trainer.py:803] 2025-04-26 20:53:30,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:31,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4868
[WARNING|trainer.py:803] 2025-04-26 20:53:31,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4131
4169
[WARNING|trainer.py:803] 2025-04-26 20:53:32,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:32,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4869
4132
[WARNING|trainer.py:803] 2025-04-26 20:53:32,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:33,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4870
4170
[WARNING|trainer.py:803] 2025-04-26 20:53:33,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:34,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4133
4871
[WARNING|trainer.py:803] 2025-04-26 20:53:34,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4171
[WARNING|trainer.py:803] 2025-04-26 20:53:35,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:35,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4872
4134
[WARNING|trainer.py:803] 2025-04-26 20:53:35,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:36,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4873
4172
[WARNING|trainer.py:803] 2025-04-26 20:53:36,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:37,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4135
[WARNING|trainer.py:803] 2025-04-26 20:53:37,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4874
4173
[WARNING|trainer.py:803] 2025-04-26 20:53:38,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:38,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4875
[WARNING|trainer.py:803] 2025-04-26 20:53:38,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4136
[WARNING|trainer.py:803] 2025-04-26 20:53:39,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4876
4174
[WARNING|trainer.py:803] 2025-04-26 20:53:39,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:40,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:40,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4137
4877
4175
[WARNING|trainer.py:803] 2025-04-26 20:53:41,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:41,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4878
4138
[WARNING|trainer.py:803] 2025-04-26 20:53:41,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:42,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4879
4176
[WARNING|trainer.py:803] 2025-04-26 20:53:42,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4139
[WARNING|trainer.py:803] 2025-04-26 20:53:43,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:43,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4880
4177
[WARNING|trainer.py:803] 2025-04-26 20:53:44,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:44,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4881
4140
[WARNING|trainer.py:803] 2025-04-26 20:53:44,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:45,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4882
4178
[WARNING|trainer.py:803] 2025-04-26 20:53:45,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4141
[WARNING|trainer.py:803] 2025-04-26 20:53:46,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:46,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4883
4179
[WARNING|trainer.py:803] 2025-04-26 20:53:47,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:47,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4884
4142
[WARNING|trainer.py:803] 2025-04-26 20:53:47,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:48,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4180
4885
[WARNING|trainer.py:803] 2025-04-26 20:53:48,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:49,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4143
[WARNING|trainer.py:803] 2025-04-26 20:53:49,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4886
4181
[WARNING|trainer.py:803] 2025-04-26 20:53:50,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:50,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4887
4144
[WARNING|trainer.py:803] 2025-04-26 20:53:50,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:51,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4182
4888
[WARNING|trainer.py:803] 2025-04-26 20:53:51,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:52,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4145
[WARNING|trainer.py:803] 2025-04-26 20:53:52,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4889
4183
[WARNING|trainer.py:803] 2025-04-26 20:53:53,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:53,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4890
[WARNING|trainer.py:803] 2025-04-26 20:53:53,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4146
[WARNING|trainer.py:803] 2025-04-26 20:53:54,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4184
4891
[WARNING|trainer.py:803] 2025-04-26 20:53:54,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:55,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4147
[WARNING|trainer.py:803] 2025-04-26 20:53:55,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4892
4185
[WARNING|trainer.py:803] 2025-04-26 20:53:56,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:56,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4893
[WARNING|trainer.py:803] 2025-04-26 20:53:56,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4148
[WARNING|trainer.py:803] 2025-04-26 20:53:57,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4186
4894
[WARNING|trainer.py:803] 2025-04-26 20:53:57,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:53:58,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:58,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4149
4895
4187
[WARNING|trainer.py:803] 2025-04-26 20:53:59,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:53:59,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4896
4150
[WARNING|trainer.py:803] 2025-04-26 20:54:00,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:00,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4897
4188
[WARNING|trainer.py:803] 2025-04-26 20:54:00,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4151
[WARNING|trainer.py:803] 2025-04-26 20:54:01,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:54:01,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4898
4189
[WARNING|trainer.py:803] 2025-04-26 20:54:02,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:02,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4899
4152
[WARNING|trainer.py:803] 2025-04-26 20:54:02,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:03,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4900
[WARNING|trainer.py:803] 2025-04-26 20:54:03,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4190
4153
[WARNING|trainer.py:803] 2025-04-26 20:54:04,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4901
[WARNING|trainer.py:803] 2025-04-26 20:54:04,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:05,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4191
[WARNING|trainer.py:803] 2025-04-26 20:54:05,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4902
4154
[WARNING|trainer.py:803] 2025-04-26 20:54:06,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:06,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:06,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4903
4192
4155
[WARNING|trainer.py:803] 2025-04-26 20:54:07,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4904
[WARNING|trainer.py:803] 2025-04-26 20:54:07,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:08,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4193
[WARNING|trainer.py:803] 2025-04-26 20:54:08,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4905
4156
[WARNING|trainer.py:803] 2025-04-26 20:54:09,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:09,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:09,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4906
4194
4157
[WARNING|trainer.py:803] 2025-04-26 20:54:10,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4907
[WARNING|trainer.py:803] 2025-04-26 20:54:10,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:11,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4195
[WARNING|trainer.py:803] 2025-04-26 20:54:11,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4908
4158
[WARNING|trainer.py:803] 2025-04-26 20:54:12,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:12,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4909
[WARNING|trainer.py:803] 2025-04-26 20:54:12,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4196
4159
[WARNING|trainer.py:803] 2025-04-26 20:54:13,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4910
[WARNING|trainer.py:803] 2025-04-26 20:54:13,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:14,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4197
[WARNING|trainer.py:803] 2025-04-26 20:54:14,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4911
4160
[WARNING|trainer.py:803] 2025-04-26 20:54:15,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:15,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:54:15,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4912
4198
4161
[WARNING|trainer.py:803] 2025-04-26 20:54:16,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4913
[WARNING|trainer.py:803] 2025-04-26 20:54:17,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:17,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4199
[WARNING|trainer.py:803] 2025-04-26 20:54:17,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4914
4162
[WARNING|trainer.py:803] 2025-04-26 20:54:18,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:18,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:18,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4915
4200
4163
[WARNING|trainer.py:803] 2025-04-26 20:54:19,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4916
[WARNING|trainer.py:803] 2025-04-26 20:54:20,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4201
[WARNING|trainer.py:803] 2025-04-26 20:54:20,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:20,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4917
4164
[WARNING|trainer.py:803] 2025-04-26 20:54:21,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4202
[WARNING|trainer.py:803] 2025-04-26 20:54:21,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4918
[WARNING|trainer.py:803] 2025-04-26 20:54:22,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:22,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4165
[WARNING|trainer.py:803] 2025-04-26 20:54:22,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4203
4919
[WARNING|trainer.py:803] 2025-04-26 20:54:23,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:23,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:23,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4920
4204
4166
[WARNING|trainer.py:803] 2025-04-26 20:54:24,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:25,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4921
[WARNING|trainer.py:803] 2025-04-26 20:54:25,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4205
4167
[WARNING|trainer.py:803] 2025-04-26 20:54:25,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4922
[WARNING|trainer.py:803] 2025-04-26 20:54:26,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:26,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4206
[WARNING|trainer.py:803] 2025-04-26 20:54:26,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4923
4168
[WARNING|trainer.py:803] 2025-04-26 20:54:27,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:27,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4207
[WARNING|trainer.py:803] 2025-04-26 20:54:28,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4924
[WARNING|trainer.py:803] 2025-04-26 20:54:28,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4169
[WARNING|trainer.py:803] 2025-04-26 20:54:28,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4925
4208
[WARNING|trainer.py:803] 2025-04-26 20:54:29,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:29,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:30,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4926
4170
4209
[WARNING|trainer.py:803] 2025-04-26 20:54:31,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:31,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4927
[WARNING|trainer.py:803] 2025-04-26 20:54:31,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4210
4171
[WARNING|trainer.py:803] 2025-04-26 20:54:32,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4928
[WARNING|trainer.py:803] 2025-04-26 20:54:32,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:32,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4211
[WARNING|trainer.py:803] 2025-04-26 20:54:33,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4929
4172
[WARNING|trainer.py:803] 2025-04-26 20:54:33,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:34,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4212
[WARNING|trainer.py:803] 2025-04-26 20:54:34,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4930
4173
[WARNING|trainer.py:803] 2025-04-26 20:54:34,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:35,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4931
4213
[WARNING|trainer.py:803] 2025-04-26 20:54:35,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:36,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:36,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4932
4174
4214
[WARNING|trainer.py:803] 2025-04-26 20:54:37,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:37,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4933
[WARNING|trainer.py:803] 2025-04-26 20:54:37,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4215
4175
[WARNING|trainer.py:803] 2025-04-26 20:54:38,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4934
[WARNING|trainer.py:803] 2025-04-26 20:54:38,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:38,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4216
[WARNING|trainer.py:803] 2025-04-26 20:54:39,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4935
4176
[WARNING|trainer.py:803] 2025-04-26 20:54:39,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:40,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:40,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4936
4217
4177
[WARNING|trainer.py:803] 2025-04-26 20:54:41,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:41,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4937
4218
[WARNING|trainer.py:803] 2025-04-26 20:54:41,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:42,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4178
4938
[WARNING|trainer.py:803] 2025-04-26 20:54:42,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4219
[WARNING|trainer.py:803] 2025-04-26 20:54:43,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:43,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4939
[WARNING|trainer.py:803] 2025-04-26 20:54:43,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4179
4220
[WARNING|trainer.py:803] 2025-04-26 20:54:44,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4940
[WARNING|trainer.py:803] 2025-04-26 20:54:44,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:44,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:45,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4180
4221
4941
[WARNING|trainer.py:803] 2025-04-26 20:54:46,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:46,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:46,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4942
4222
4181
[WARNING|trainer.py:803] 2025-04-26 20:54:47,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4943
[WARNING|trainer.py:803] 2025-04-26 20:54:47,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:47,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4223
[WARNING|trainer.py:803] 2025-04-26 20:54:48,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4182
4944
[WARNING|trainer.py:803] 2025-04-26 20:54:48,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4224
[WARNING|trainer.py:803] 2025-04-26 20:54:49,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:54:49,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4945
4183
[WARNING|trainer.py:803] 2025-04-26 20:54:49,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:50,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4225
4946
[WARNING|trainer.py:803] 2025-04-26 20:54:50,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:51,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:54:51,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4184
4947
4226
[WARNING|trainer.py:803] 2025-04-26 20:54:52,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:52,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:52,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4948
4227
4185
[WARNING|trainer.py:803] 2025-04-26 20:54:53,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4949
[WARNING|trainer.py:803] 2025-04-26 20:54:53,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:53,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4228
[WARNING|trainer.py:803] 2025-04-26 20:54:54,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4950
4186
[WARNING|trainer.py:803] 2025-04-26 20:54:54,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4229
[WARNING|trainer.py:803] 2025-04-26 20:54:55,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:55,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4951
[WARNING|trainer.py:803] 2025-04-26 20:54:55,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4187
[WARNING|trainer.py:803] 2025-04-26 20:54:56,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4230
4952
[WARNING|trainer.py:803] 2025-04-26 20:54:57,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:57,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:54:57,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4953
4231
4188
[WARNING|trainer.py:803] 2025-04-26 20:54:58,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:54:58,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4954
[WARNING|trainer.py:803] 2025-04-26 20:54:58,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4232
4189
[WARNING|trainer.py:803] 2025-04-26 20:54:59,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4955
[WARNING|trainer.py:803] 2025-04-26 20:54:59,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4233
[WARNING|trainer.py:803] 2025-04-26 20:55:00,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:00,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4956
[WARNING|trainer.py:803] 2025-04-26 20:55:00,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4190
4234
[WARNING|trainer.py:803] 2025-04-26 20:55:01,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4957
[WARNING|trainer.py:803] 2025-04-26 20:55:01,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:02,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:55:02,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4191
4235
4958
[WARNING|trainer.py:803] 2025-04-26 20:55:03,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:03,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:03,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4959
4236
4192
[WARNING|trainer.py:803] 2025-04-26 20:55:04,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4960
[WARNING|trainer.py:803] 2025-04-26 20:55:04,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:04,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4237
[WARNING|trainer.py:803] 2025-04-26 20:55:05,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4193
4961
[WARNING|trainer.py:803] 2025-04-26 20:55:05,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4238
[WARNING|trainer.py:803] 2025-04-26 20:55:06,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:06,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4962
4194
[WARNING|trainer.py:803] 2025-04-26 20:55:07,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4239
[WARNING|trainer.py:803] 2025-04-26 20:55:07,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4963
[WARNING|trainer.py:803] 2025-04-26 20:55:07,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:08,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:08,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4195
4964
4240
[WARNING|trainer.py:803] 2025-04-26 20:55:09,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:09,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:09,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4965
4241
4196
[WARNING|trainer.py:803] 2025-04-26 20:55:10,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4966
[WARNING|trainer.py:803] 2025-04-26 20:55:10,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:11,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4242
[WARNING|trainer.py:803] 2025-04-26 20:55:11,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4197
4967
[WARNING|trainer.py:803] 2025-04-26 20:55:12,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4243
[WARNING|trainer.py:803] 2025-04-26 20:55:12,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:12,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4968
4198
[WARNING|trainer.py:803] 2025-04-26 20:55:13,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:13,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4244
4969
[WARNING|trainer.py:803] 2025-04-26 20:55:14,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:14,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:14,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4199
4970
4245
[WARNING|trainer.py:803] 2025-04-26 20:55:15,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:15,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:15,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4971
4246
4200
[WARNING|trainer.py:803] 2025-04-26 20:55:16,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4972
[WARNING|trainer.py:803] 2025-04-26 20:55:17,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:17,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4247
4201
[WARNING|trainer.py:803] 2025-04-26 20:55:17,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4973
[WARNING|trainer.py:803] 2025-04-26 20:55:18,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:18,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4248
[WARNING|trainer.py:803] 2025-04-26 20:55:18,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4202
4974
[WARNING|trainer.py:803] 2025-04-26 20:55:19,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:19,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:19,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4249
4975
4203
[WARNING|trainer.py:803] 2025-04-26 20:55:20,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:20,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:20,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4976
4250
4204
[WARNING|trainer.py:803] 2025-04-26 20:55:21,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4977
[WARNING|trainer.py:803] 2025-04-26 20:55:22,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:22,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4251
4205
[WARNING|trainer.py:803] 2025-04-26 20:55:22,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4978
[WARNING|trainer.py:803] 2025-04-26 20:55:23,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:23,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4252
4206
[WARNING|trainer.py:803] 2025-04-26 20:55:23,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4979
[WARNING|trainer.py:803] 2025-04-26 20:55:24,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:24,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:24,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4253
4207
4980
[WARNING|trainer.py:803] 2025-04-26 20:55:25,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:25,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:25,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4981
4254
4208
[WARNING|trainer.py:803] 2025-04-26 20:55:26,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:26,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:27,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4982
4255
4209
[WARNING|trainer.py:803] 2025-04-26 20:55:27,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4983
[WARNING|trainer.py:803] 2025-04-26 20:55:28,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:28,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4256
4210
[WARNING|trainer.py:803] 2025-04-26 20:55:28,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4984
[WARNING|trainer.py:803] 2025-04-26 20:55:29,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:29,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4257
[WARNING|trainer.py:803] 2025-04-26 20:55:29,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4211
4985
[WARNING|trainer.py:803] 2025-04-26 20:55:30,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:30,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:30,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4258
4986
4212
[WARNING|trainer.py:803] 2025-04-26 20:55:31,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:32,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:32,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4987
4259
4213
[WARNING|trainer.py:803] 2025-04-26 20:55:33,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:33,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4988
[WARNING|trainer.py:803] 2025-04-26 20:55:33,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4260
4214
[WARNING|trainer.py:803] 2025-04-26 20:55:34,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4989
[WARNING|trainer.py:803] 2025-04-26 20:55:34,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:34,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4261
4215
[WARNING|trainer.py:803] 2025-04-26 20:55:35,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4990
[WARNING|trainer.py:803] 2025-04-26 20:55:35,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:35,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4262
[WARNING|trainer.py:803] 2025-04-26 20:55:36,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4991
4216
[WARNING|trainer.py:803] 2025-04-26 20:55:36,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:37,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:37,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4263
4992
4217
[WARNING|trainer.py:803] 2025-04-26 20:55:38,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:38,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4993
[WARNING|trainer.py:803] 2025-04-26 20:55:38,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4264
4218
[WARNING|trainer.py:803] 2025-04-26 20:55:39,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:39,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4994
[WARNING|trainer.py:803] 2025-04-26 20:55:39,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4265
4219
[WARNING|trainer.py:803] 2025-04-26 20:55:40,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4995
[WARNING|trainer.py:803] 2025-04-26 20:55:40,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:40,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4266
[WARNING|trainer.py:803] 2025-04-26 20:55:41,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4220
4996
[WARNING|trainer.py:803] 2025-04-26 20:55:41,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:42,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4267
[WARNING|trainer.py:803] 2025-04-26 20:55:42,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4997
4221
[WARNING|trainer.py:803] 2025-04-26 20:55:43,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:43,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:43,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4998
4268
4222
[WARNING|trainer.py:803] 2025-04-26 20:55:44,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:44,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4999
[WARNING|trainer.py:803] 2025-04-26 20:55:44,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4269
4223
[WARNING|trainer.py:803] 2025-04-26 20:55:45,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5000
[WARNING|trainer.py:803] 2025-04-26 20:55:45,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:45,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4270
4224
[WARNING|trainer.py:803] 2025-04-26 20:55:46,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5001
[WARNING|trainer.py:803] 2025-04-26 20:55:46,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:47,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4271
[WARNING|trainer.py:803] 2025-04-26 20:55:47,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4225
5002
[WARNING|trainer.py:803] 2025-04-26 20:55:47,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:48,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:55:48,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4272
5003
4226
[WARNING|trainer.py:803] 2025-04-26 20:55:49,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:49,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:49,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5004
4273
4227
[WARNING|trainer.py:803] 2025-04-26 20:55:50,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:50,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5005
[WARNING|trainer.py:803] 2025-04-26 20:55:50,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4274
4228
[WARNING|trainer.py:803] 2025-04-26 20:55:51,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5006
[WARNING|trainer.py:803] 2025-04-26 20:55:51,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:51,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4275
4229
[WARNING|trainer.py:803] 2025-04-26 20:55:52,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5007
[WARNING|trainer.py:803] 2025-04-26 20:55:52,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:53,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4276
[WARNING|trainer.py:803] 2025-04-26 20:55:53,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5008
4230
[WARNING|trainer.py:803] 2025-04-26 20:55:54,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:54,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:54,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4277
5009
4231
[WARNING|trainer.py:803] 2025-04-26 20:55:55,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:55:55,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5010
[WARNING|trainer.py:803] 2025-04-26 20:55:55,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4278
4232
[WARNING|trainer.py:803] 2025-04-26 20:55:56,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:56,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5011
[WARNING|trainer.py:803] 2025-04-26 20:55:56,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4279
4233
[WARNING|trainer.py:803] 2025-04-26 20:55:57,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5012
[WARNING|trainer.py:803] 2025-04-26 20:55:57,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4280
[WARNING|trainer.py:803] 2025-04-26 20:55:58,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:55:58,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4234
5013
[WARNING|trainer.py:803] 2025-04-26 20:55:58,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4281
[WARNING|trainer.py:803] 2025-04-26 20:55:59,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:55:59,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5014
4235
[WARNING|trainer.py:803] 2025-04-26 20:56:00,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:00,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4282
[WARNING|trainer.py:803] 2025-04-26 20:56:00,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5015
4236
[WARNING|trainer.py:803] 2025-04-26 20:56:01,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:01,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5016
[WARNING|trainer.py:803] 2025-04-26 20:56:01,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4283
4237
[WARNING|trainer.py:803] 2025-04-26 20:56:02,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5017
[WARNING|trainer.py:803] 2025-04-26 20:56:02,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:03,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4284
4238
[WARNING|trainer.py:803] 2025-04-26 20:56:03,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5018
[WARNING|trainer.py:803] 2025-04-26 20:56:03,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:04,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4285
[WARNING|trainer.py:803] 2025-04-26 20:56:04,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5019
4239
[WARNING|trainer.py:803] 2025-04-26 20:56:05,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:05,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:56:05,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4286
5020
4240
[WARNING|trainer.py:803] 2025-04-26 20:56:06,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:56:06,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5021
[WARNING|trainer.py:803] 2025-04-26 20:56:06,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4287
4241
[WARNING|trainer.py:803] 2025-04-26 20:56:07,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:07,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5022
[WARNING|trainer.py:803] 2025-04-26 20:56:07,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4288
4242
[WARNING|trainer.py:803] 2025-04-26 20:56:08,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5023
[WARNING|trainer.py:803] 2025-04-26 20:56:08,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4289
[WARNING|trainer.py:803] 2025-04-26 20:56:09,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:09,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4243
5024
[WARNING|trainer.py:803] 2025-04-26 20:56:10,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4290
[WARNING|trainer.py:803] 2025-04-26 20:56:10,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:10,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5025
4244
[WARNING|trainer.py:803] 2025-04-26 20:56:11,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:11,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4291
[WARNING|trainer.py:803] 2025-04-26 20:56:11,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5026
4245
[WARNING|trainer.py:803] 2025-04-26 20:56:12,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:12,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5027
[WARNING|trainer.py:803] 2025-04-26 20:56:12,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4292
4246
[WARNING|trainer.py:803] 2025-04-26 20:56:13,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:13,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5028
[WARNING|trainer.py:803] 2025-04-26 20:56:14,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4293
4247
[WARNING|trainer.py:803] 2025-04-26 20:56:14,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5029
[WARNING|trainer.py:803] 2025-04-26 20:56:15,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:15,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4294
[WARNING|trainer.py:803] 2025-04-26 20:56:15,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4248
5030
[WARNING|trainer.py:803] 2025-04-26 20:56:16,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:16,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4295
[WARNING|trainer.py:803] 2025-04-26 20:56:16,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5031
4249
[WARNING|trainer.py:803] 2025-04-26 20:56:17,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:17,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:17,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5032
4296
4250
[WARNING|trainer.py:803] 2025-04-26 20:56:18,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:18,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5033
[WARNING|trainer.py:803] 2025-04-26 20:56:19,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4297
4251
[WARNING|trainer.py:803] 2025-04-26 20:56:19,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5034
[WARNING|trainer.py:803] 2025-04-26 20:56:20,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:56:20,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4298
4252
[WARNING|trainer.py:803] 2025-04-26 20:56:20,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5035
[WARNING|trainer.py:803] 2025-04-26 20:56:21,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:21,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4299
[WARNING|trainer.py:803] 2025-04-26 20:56:21,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5036
4253
[WARNING|trainer.py:803] 2025-04-26 20:56:22,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:22,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:22,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4300
5037
4254
[WARNING|trainer.py:803] 2025-04-26 20:56:23,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:23,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5038
[WARNING|trainer.py:803] 2025-04-26 20:56:24,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4301
4255
[WARNING|trainer.py:803] 2025-04-26 20:56:24,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5039
[WARNING|trainer.py:803] 2025-04-26 20:56:25,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:25,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4302
4256
[WARNING|trainer.py:803] 2025-04-26 20:56:25,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5040
[WARNING|trainer.py:803] 2025-04-26 20:56:26,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:26,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4303
[WARNING|trainer.py:803] 2025-04-26 20:56:26,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5041
4257
[WARNING|trainer.py:803] 2025-04-26 20:56:27,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:27,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:27,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4304
5042
4258
[WARNING|trainer.py:803] 2025-04-26 20:56:28,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:28,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5043
[WARNING|trainer.py:803] 2025-04-26 20:56:29,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4305
4259
[WARNING|trainer.py:803] 2025-04-26 20:56:29,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:30,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5044
[WARNING|trainer.py:803] 2025-04-26 20:56:30,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4306
4260
[WARNING|trainer.py:803] 2025-04-26 20:56:30,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5045
[WARNING|trainer.py:803] 2025-04-26 20:56:31,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:31,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4307
[WARNING|trainer.py:803] 2025-04-26 20:56:31,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4261
5046
[WARNING|trainer.py:803] 2025-04-26 20:56:32,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:32,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:32,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4308
5047
4262
[WARNING|trainer.py:803] 2025-04-26 20:56:33,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:33,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:34,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5048
4309
4263
[WARNING|trainer.py:803] 2025-04-26 20:56:34,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:56:35,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5049
[WARNING|trainer.py:803] 2025-04-26 20:56:35,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4310
4264
[WARNING|trainer.py:803] 2025-04-26 20:56:35,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5050
[WARNING|trainer.py:803] 2025-04-26 20:56:36,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:36,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4311
4265
[WARNING|trainer.py:803] 2025-04-26 20:56:37,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5051
[WARNING|trainer.py:803] 2025-04-26 20:56:37,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:37,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4312
[WARNING|trainer.py:803] 2025-04-26 20:56:38,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4266
5052
[WARNING|trainer.py:803] 2025-04-26 20:56:38,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:39,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:39,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4313
5053
4267
[WARNING|trainer.py:803] 2025-04-26 20:56:39,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:40,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5054
[WARNING|trainer.py:803] 2025-04-26 20:56:40,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4314
4268
[WARNING|trainer.py:803] 2025-04-26 20:56:41,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:41,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5055
[WARNING|trainer.py:803] 2025-04-26 20:56:41,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4315
4269
[WARNING|trainer.py:803] 2025-04-26 20:56:42,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5056
[WARNING|trainer.py:803] 2025-04-26 20:56:42,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:42,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4316
[WARNING|trainer.py:803] 2025-04-26 20:56:43,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4270
5057
[WARNING|trainer.py:803] 2025-04-26 20:56:43,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:43,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:44,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4317
5058
4271
[WARNING|trainer.py:803] 2025-04-26 20:56:45,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:45,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:45,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5059
4318
4272
[WARNING|trainer.py:803] 2025-04-26 20:56:46,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:46,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5060
[WARNING|trainer.py:803] 2025-04-26 20:56:46,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4319
4273
[WARNING|trainer.py:803] 2025-04-26 20:56:47,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5061
[WARNING|trainer.py:803] 2025-04-26 20:56:47,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:47,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4320
4274
[WARNING|trainer.py:803] 2025-04-26 20:56:48,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5062
[WARNING|trainer.py:803] 2025-04-26 20:56:48,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:49,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:49,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4321
5063
4275
[WARNING|trainer.py:803] 2025-04-26 20:56:50,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:50,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:50,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5064
4322
4276
[WARNING|trainer.py:803] 2025-04-26 20:56:51,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:51,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5065
[WARNING|trainer.py:803] 2025-04-26 20:56:51,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4323
4277
[WARNING|trainer.py:803] 2025-04-26 20:56:52,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5066
[WARNING|trainer.py:803] 2025-04-26 20:56:52,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:52,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4324
4278
[WARNING|trainer.py:803] 2025-04-26 20:56:53,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5067
[WARNING|trainer.py:803] 2025-04-26 20:56:53,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:54,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4325
[WARNING|trainer.py:803] 2025-04-26 20:56:54,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5068
4279
[WARNING|trainer.py:803] 2025-04-26 20:56:55,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:55,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:55,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4326
5069
4280
[WARNING|trainer.py:803] 2025-04-26 20:56:56,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:56:56,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5070
[WARNING|trainer.py:803] 2025-04-26 20:56:56,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4327
4281
[WARNING|trainer.py:803] 2025-04-26 20:56:57,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5071
[WARNING|trainer.py:803] 2025-04-26 20:56:57,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:56:57,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4328
4282
[WARNING|trainer.py:803] 2025-04-26 20:56:58,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5072
[WARNING|trainer.py:803] 2025-04-26 20:56:58,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:56:59,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4329
[WARNING|trainer.py:803] 2025-04-26 20:56:59,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4283
5073
[WARNING|trainer.py:803] 2025-04-26 20:57:00,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:00,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:00,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4330
5074
4284
[WARNING|trainer.py:803] 2025-04-26 20:57:01,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:01,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5075
[WARNING|trainer.py:803] 2025-04-26 20:57:01,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4331
4285
[WARNING|trainer.py:803] 2025-04-26 20:57:02,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:02,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5076
[WARNING|trainer.py:803] 2025-04-26 20:57:02,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4332
4286
[WARNING|trainer.py:803] 2025-04-26 20:57:03,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5077
[WARNING|trainer.py:803] 2025-04-26 20:57:03,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:57:04,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4333
[WARNING|trainer.py:803] 2025-04-26 20:57:04,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4287
5078
[WARNING|trainer.py:803] 2025-04-26 20:57:04,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:05,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4334
[WARNING|trainer.py:803] 2025-04-26 20:57:05,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5079
4288
[WARNING|trainer.py:803] 2025-04-26 20:57:06,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:06,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:06,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4335
5080
4289
[WARNING|trainer.py:803] 2025-04-26 20:57:07,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:07,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5081
[WARNING|trainer.py:803] 2025-04-26 20:57:07,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4336
4290
[WARNING|trainer.py:803] 2025-04-26 20:57:08,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:08,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5082
[WARNING|trainer.py:803] 2025-04-26 20:57:09,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4337
4291
[WARNING|trainer.py:803] 2025-04-26 20:57:09,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5083
[WARNING|trainer.py:803] 2025-04-26 20:57:09,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:10,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4338
[WARNING|trainer.py:803] 2025-04-26 20:57:10,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4292
5084
[WARNING|trainer.py:803] 2025-04-26 20:57:11,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4339
[WARNING|trainer.py:803] 2025-04-26 20:57:11,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:11,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5085
4293
[WARNING|trainer.py:803] 2025-04-26 20:57:12,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:12,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4340
[WARNING|trainer.py:803] 2025-04-26 20:57:12,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5086
4294
[WARNING|trainer.py:803] 2025-04-26 20:57:13,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:13,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5087
4341
[WARNING|trainer.py:803] 2025-04-26 20:57:14,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4295
[WARNING|trainer.py:803] 2025-04-26 20:57:14,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:14,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5088
4342
[WARNING|trainer.py:803] 2025-04-26 20:57:15,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:15,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4296
5089
[WARNING|trainer.py:803] 2025-04-26 20:57:16,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4343
[WARNING|trainer.py:803] 2025-04-26 20:57:16,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:16,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5090
4297
[WARNING|trainer.py:803] 2025-04-26 20:57:17,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4344
[WARNING|trainer.py:803] 2025-04-26 20:57:17,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:17,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5091
4298
[WARNING|trainer.py:803] 2025-04-26 20:57:18,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:18,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5092
4345
[WARNING|trainer.py:803] 2025-04-26 20:57:19,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4299
[WARNING|trainer.py:803] 2025-04-26 20:57:19,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:19,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5093
4346
[WARNING|trainer.py:803] 2025-04-26 20:57:20,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4300
[WARNING|trainer.py:803] 2025-04-26 20:57:20,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5094
[WARNING|trainer.py:803] 2025-04-26 20:57:20,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4347
[WARNING|trainer.py:803] 2025-04-26 20:57:21,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:21,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5095
4301
[WARNING|trainer.py:803] 2025-04-26 20:57:22,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4348
[WARNING|trainer.py:803] 2025-04-26 20:57:22,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:22,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5096
4302
[WARNING|trainer.py:803] 2025-04-26 20:57:23,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:23,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4349
5097
[WARNING|trainer.py:803] 2025-04-26 20:57:24,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4303
[WARNING|trainer.py:803] 2025-04-26 20:57:24,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:24,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5098
4350
[WARNING|trainer.py:803] 2025-04-26 20:57:25,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4304
[WARNING|trainer.py:803] 2025-04-26 20:57:25,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:25,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5099
4351
[WARNING|trainer.py:803] 2025-04-26 20:57:26,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:26,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4305
5100
[WARNING|trainer.py:803] 2025-04-26 20:57:26,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4352
[WARNING|trainer.py:803] 2025-04-26 20:57:27,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:27,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5101
4306
[WARNING|trainer.py:803] 2025-04-26 20:57:28,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4353
[WARNING|trainer.py:803] 2025-04-26 20:57:28,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:57:29,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5102
4307
[WARNING|trainer.py:803] 2025-04-26 20:57:29,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4354
[WARNING|trainer.py:803] 2025-04-26 20:57:30,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5103
[WARNING|trainer.py:803] 2025-04-26 20:57:30,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4308
[WARNING|trainer.py:803] 2025-04-26 20:57:30,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:31,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4355
5104
[WARNING|trainer.py:803] 2025-04-26 20:57:31,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4309
[WARNING|trainer.py:803] 2025-04-26 20:57:32,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:32,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4356
5105
[WARNING|trainer.py:803] 2025-04-26 20:57:32,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:57:33,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4310
[WARNING|trainer.py:803] 2025-04-26 20:57:33,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4357
5106
[WARNING|trainer.py:803] 2025-04-26 20:57:34,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:34,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4311
[WARNING|trainer.py:803] 2025-04-26 20:57:34,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4358
5107
[WARNING|trainer.py:803] 2025-04-26 20:57:35,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:35,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:35,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4312
5108
4359
[WARNING|trainer.py:803] 2025-04-26 20:57:36,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:36,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:36,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4313
5109
4360
[WARNING|trainer.py:803] 2025-04-26 20:57:37,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:37,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:37,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5110
4314
4361
[WARNING|trainer.py:803] 2025-04-26 20:57:39,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:39,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:39,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5111
4315
4362
[WARNING|trainer.py:803] 2025-04-26 20:57:40,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:57:40,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5112
[WARNING|trainer.py:803] 2025-04-26 20:57:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4316
4363
[WARNING|trainer.py:803] 2025-04-26 20:57:41,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5113
[WARNING|trainer.py:803] 2025-04-26 20:57:41,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:41,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4317
4364
[WARNING|trainer.py:803] 2025-04-26 20:57:42,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5114
[WARNING|trainer.py:803] 2025-04-26 20:57:42,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:42,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4365
4318
[WARNING|trainer.py:803] 2025-04-26 20:57:43,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5115
[WARNING|trainer.py:803] 2025-04-26 20:57:43,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:44,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4366
[WARNING|trainer.py:803] 2025-04-26 20:57:44,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4319
5116
[WARNING|trainer.py:803] 2025-04-26 20:57:45,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:45,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4367
[WARNING|trainer.py:803] 2025-04-26 20:57:45,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4320
5117
[WARNING|trainer.py:803] 2025-04-26 20:57:46,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:46,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:46,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4368
5118
4321
[WARNING|trainer.py:803] 2025-04-26 20:57:47,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:57:47,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:57:47,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4369
5119
4322
[WARNING|trainer.py:803] 2025-04-26 20:57:48,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:48,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4370
[WARNING|trainer.py:803] 2025-04-26 20:57:49,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5120
4323
[WARNING|trainer.py:803] 2025-04-26 20:57:49,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:50,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4371
5121
[WARNING|trainer.py:803] 2025-04-26 20:57:50,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4324
[WARNING|trainer.py:803] 2025-04-26 20:57:51,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:51,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5122
4372
[WARNING|trainer.py:803] 2025-04-26 20:57:51,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4325
[WARNING|trainer.py:803] 2025-04-26 20:57:52,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:57:52,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5123
4373
[WARNING|trainer.py:803] 2025-04-26 20:57:52,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4326
[WARNING|trainer.py:803] 2025-04-26 20:57:53,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:53,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5124
4374
[WARNING|trainer.py:803] 2025-04-26 20:57:54,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4327
[WARNING|trainer.py:803] 2025-04-26 20:57:54,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:54,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5125
4375
[WARNING|trainer.py:803] 2025-04-26 20:57:55,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:55,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4328
[WARNING|trainer.py:803] 2025-04-26 20:57:55,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5126
4376
[WARNING|trainer.py:803] 2025-04-26 20:57:56,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:56,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4329
5127
[WARNING|trainer.py:803] 2025-04-26 20:57:57,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4377
[WARNING|trainer.py:803] 2025-04-26 20:57:57,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:57:57,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4330
5128
[WARNING|trainer.py:803] 2025-04-26 20:57:58,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4378
[WARNING|trainer.py:803] 2025-04-26 20:57:59,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:57:59,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5129
4331
[WARNING|trainer.py:803] 2025-04-26 20:57:59,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4379
[WARNING|trainer.py:803] 2025-04-26 20:58:00,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:58:00,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5130
[WARNING|trainer.py:803] 2025-04-26 20:58:00,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4332
4380
[WARNING|trainer.py:803] 2025-04-26 20:58:01,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5131
[WARNING|trainer.py:803] 2025-04-26 20:58:01,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:58:01,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4333
4381
[WARNING|trainer.py:803] 2025-04-26 20:58:02,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5132
[WARNING|trainer.py:803] 2025-04-26 20:58:02,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:03,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4334
4382
[WARNING|trainer.py:803] 2025-04-26 20:58:03,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5133
[WARNING|trainer.py:803] 2025-04-26 20:58:04,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:04,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4335
4383
[WARNING|trainer.py:803] 2025-04-26 20:58:04,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5134
[WARNING|trainer.py:803] 2025-04-26 20:58:05,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:05,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4336
[WARNING|trainer.py:803] 2025-04-26 20:58:05,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4384
5135
[WARNING|trainer.py:803] 2025-04-26 20:58:06,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:06,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:06,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4337
4385
5136
[WARNING|trainer.py:803] 2025-04-26 20:58:07,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:07,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:08,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4338
4386
5137
[WARNING|trainer.py:803] 2025-04-26 20:58:09,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:58:09,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:09,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4339
4387
5138
[WARNING|trainer.py:803] 2025-04-26 20:58:10,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:10,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:10,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4388
4340
5139
[WARNING|trainer.py:803] 2025-04-26 20:58:11,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:11,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:11,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4389
4341
5140
[WARNING|trainer.py:803] 2025-04-26 20:58:12,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:12,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:12,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4390
5141
4342
[WARNING|trainer.py:803] 2025-04-26 20:58:13,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:13,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:14,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5142
4391
4343
[WARNING|trainer.py:803] 2025-04-26 20:58:15,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:15,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:15,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4392
5143
4344
[WARNING|trainer.py:803] 2025-04-26 20:58:16,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:16,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:58:16,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5144
4393
4345
[WARNING|trainer.py:803] 2025-04-26 20:58:17,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:17,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5145
[WARNING|trainer.py:803] 2025-04-26 20:58:17,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4394
4346
[WARNING|trainer.py:803] 2025-04-26 20:58:18,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:18,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5146
4395
[WARNING|trainer.py:803] 2025-04-26 20:58:19,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4347
[WARNING|trainer.py:803] 2025-04-26 20:58:19,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:19,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5147
4396
[WARNING|trainer.py:803] 2025-04-26 20:58:20,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4348
[WARNING|trainer.py:803] 2025-04-26 20:58:20,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:21,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5148
4397
[WARNING|trainer.py:803] 2025-04-26 20:58:21,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4349
[WARNING|trainer.py:803] 2025-04-26 20:58:21,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5149
[WARNING|trainer.py:803] 2025-04-26 20:58:22,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4398
[WARNING|trainer.py:803] 2025-04-26 20:58:22,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:23,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4350
5150
[WARNING|trainer.py:803] 2025-04-26 20:58:23,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
4399
[WARNING|trainer.py:803] 2025-04-26 20:58:23,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:24,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4351
5151
[WARNING|trainer.py:803] 2025-04-26 20:58:24,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4400
[WARNING|trainer.py:803] 2025-04-26 20:58:25,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:25,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4352
5152
[WARNING|trainer.py:803] 2025-04-26 20:58:25,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4401
[WARNING|trainer.py:803] 2025-04-26 20:58:26,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:26,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4353
5153
[WARNING|trainer.py:803] 2025-04-26 20:58:26,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4402
[WARNING|trainer.py:803] 2025-04-26 20:58:27,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:27,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4354
5154
[WARNING|trainer.py:803] 2025-04-26 20:58:28,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4403
[WARNING|trainer.py:803] 2025-04-26 20:58:28,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:28,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5155
4355
[WARNING|trainer.py:803] 2025-04-26 20:58:29,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4404
[WARNING|trainer.py:803] 2025-04-26 20:58:29,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:29,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5156
4356
[WARNING|trainer.py:803] 2025-04-26 20:58:30,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4405
[WARNING|trainer.py:803] 2025-04-26 20:58:31,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:31,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5157
4357
[WARNING|trainer.py:803] 2025-04-26 20:58:31,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4406
[WARNING|trainer.py:803] 2025-04-26 20:58:32,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5158
[WARNING|trainer.py:803] 2025-04-26 20:58:32,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4358
[WARNING|trainer.py:803] 2025-04-26 20:58:32,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4407
[WARNING|trainer.py:803] 2025-04-26 20:58:33,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5159
[WARNING|trainer.py:803] 2025-04-26 20:58:33,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4359
[WARNING|trainer.py:803] 2025-04-26 20:58:34,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:34,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4408
5160
[WARNING|trainer.py:803] 2025-04-26 20:58:34,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4360
[WARNING|trainer.py:803] 2025-04-26 20:58:35,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:35,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4409
5161
[WARNING|trainer.py:803] 2025-04-26 20:58:35,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4361
[WARNING|trainer.py:803] 2025-04-26 20:58:36,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:36,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4410
5162
[WARNING|trainer.py:803] 2025-04-26 20:58:37,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4362
[WARNING|trainer.py:803] 2025-04-26 20:58:37,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:37,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5163
4411
[WARNING|trainer.py:803] 2025-04-26 20:58:38,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4363
[WARNING|trainer.py:803] 2025-04-26 20:58:38,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:58:38,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5164
4412
[WARNING|trainer.py:803] 2025-04-26 20:58:39,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:39,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4364
[WARNING|trainer.py:803] 2025-04-26 20:58:40,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5165
4413
[WARNING|trainer.py:803] 2025-04-26 20:58:40,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:41,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4365
[WARNING|trainer.py:803] 2025-04-26 20:58:41,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5166
4414
[WARNING|trainer.py:803] 2025-04-26 20:58:41,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:42,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4366
[WARNING|trainer.py:803] 2025-04-26 20:58:42,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5167
4415
[WARNING|trainer.py:803] 2025-04-26 20:58:43,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:43,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4367
[WARNING|trainer.py:803] 2025-04-26 20:58:43,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5168
4416
[WARNING|trainer.py:803] 2025-04-26 20:58:44,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:44,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4368
5169
[WARNING|trainer.py:803] 2025-04-26 20:58:44,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4417
[WARNING|trainer.py:803] 2025-04-26 20:58:45,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:58:45,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4369
5170
[WARNING|trainer.py:803] 2025-04-26 20:58:45,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4418
[WARNING|trainer.py:803] 2025-04-26 20:58:46,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:46,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5171
4370
[WARNING|trainer.py:803] 2025-04-26 20:58:47,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4419
[WARNING|trainer.py:803] 2025-04-26 20:58:47,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:58:47,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5172
4371
[WARNING|trainer.py:803] 2025-04-26 20:58:48,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4420
[WARNING|trainer.py:803] 2025-04-26 20:58:49,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:49,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5173
4372
[WARNING|trainer.py:803] 2025-04-26 20:58:49,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4421
[WARNING|trainer.py:803] 2025-04-26 20:58:50,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:50,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5174
4373
[WARNING|trainer.py:803] 2025-04-26 20:58:50,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4422
[WARNING|trainer.py:803] 2025-04-26 20:58:51,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5175
[WARNING|trainer.py:803] 2025-04-26 20:58:51,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4374
[WARNING|trainer.py:803] 2025-04-26 20:58:51,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4423
[WARNING|trainer.py:803] 2025-04-26 20:58:52,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5176
[WARNING|trainer.py:803] 2025-04-26 20:58:52,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:53,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4375
4424
[WARNING|trainer.py:803] 2025-04-26 20:58:53,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5177
[WARNING|trainer.py:803] 2025-04-26 20:58:53,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:54,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4376
4425
[WARNING|trainer.py:803] 2025-04-26 20:58:54,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5178
[WARNING|trainer.py:803] 2025-04-26 20:58:55,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:55,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4377
4426
[WARNING|trainer.py:803] 2025-04-26 20:58:55,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5179
[WARNING|trainer.py:803] 2025-04-26 20:58:56,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:58:56,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4378
4427
[WARNING|trainer.py:803] 2025-04-26 20:58:56,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5180
[WARNING|trainer.py:803] 2025-04-26 20:58:57,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:57,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4379
[WARNING|trainer.py:803] 2025-04-26 20:58:58,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4428
5181
[WARNING|trainer.py:803] 2025-04-26 20:58:58,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:58:58,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4380
[WARNING|trainer.py:803] 2025-04-26 20:58:59,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4429
5182
[WARNING|trainer.py:803] 2025-04-26 20:58:59,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:00,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4381
[WARNING|trainer.py:803] 2025-04-26 20:59:00,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4430
5183
[WARNING|trainer.py:803] 2025-04-26 20:59:01,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:01,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4382
[WARNING|trainer.py:803] 2025-04-26 20:59:01,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4431
5184
[WARNING|trainer.py:803] 2025-04-26 20:59:02,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:02,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4383
4432
[WARNING|trainer.py:803] 2025-04-26 20:59:02,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5185
[WARNING|trainer.py:803] 2025-04-26 20:59:03,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:03,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4384
4433
[WARNING|trainer.py:803] 2025-04-26 20:59:04,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5186
[WARNING|trainer.py:803] 2025-04-26 20:59:04,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:04,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4385
[WARNING|trainer.py:803] 2025-04-26 20:59:05,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4434
5187
[WARNING|trainer.py:803] 2025-04-26 20:59:05,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:06,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4386
[WARNING|trainer.py:803] 2025-04-26 20:59:06,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4435
5188
[WARNING|trainer.py:803] 2025-04-26 20:59:07,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:07,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4387
[WARNING|trainer.py:803] 2025-04-26 20:59:07,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4436
5189
[WARNING|trainer.py:803] 2025-04-26 20:59:08,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:08,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:08,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4388
4437
5190
[WARNING|trainer.py:803] 2025-04-26 20:59:09,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:09,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:09,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4389
4438
5191
[WARNING|trainer.py:803] 2025-04-26 20:59:10,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:10,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:10,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4390
4439
5192
[WARNING|trainer.py:803] 2025-04-26 20:59:11,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:11,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:12,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4440
4391
5193
[WARNING|trainer.py:803] 2025-04-26 20:59:13,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:13,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:13,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4441
5194
4392
[WARNING|trainer.py:803] 2025-04-26 20:59:14,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:14,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:14,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5195
4442
4393
[WARNING|trainer.py:803] 2025-04-26 20:59:15,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:15,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:15,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5196
4443
4394
[WARNING|trainer.py:803] 2025-04-26 20:59:16,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:16,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:16,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5197
4444
4395
[WARNING|trainer.py:803] 2025-04-26 20:59:17,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:59:17,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:17,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5198
4445
4396
[WARNING|trainer.py:803] 2025-04-26 20:59:18,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:19,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:19,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5199
4446
4397
[WARNING|trainer.py:803] 2025-04-26 20:59:20,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:20,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:20,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5200
4447
4398
[WARNING|trainer.py:803] 2025-04-26 20:59:21,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:59:21,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:21,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
   No
5201
4448
4399
[WARNING|trainer.py:803] 2025-04-26 20:59:22,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:22,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:22,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5202
4449
4400
[WARNING|trainer.py:803] 2025-04-26 20:59:23,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:23,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:23,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5203
4450
4401
[WARNING|trainer.py:803] 2025-04-26 20:59:24,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:24,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:25,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5204
4451
4402
[WARNING|trainer.py:803] 2025-04-26 20:59:26,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:26,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5205
[WARNING|trainer.py:803] 2025-04-26 20:59:26,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4452
4403
[WARNING|trainer.py:803] 2025-04-26 20:59:27,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:27,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5206
[WARNING|trainer.py:803] 2025-04-26 20:59:27,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4453
4404
[WARNING|trainer.py:803] 2025-04-26 20:59:28,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:28,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5207
[WARNING|trainer.py:803] 2025-04-26 20:59:28,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4454
4405
[WARNING|trainer.py:803] 2025-04-26 20:59:29,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:29,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5208
[WARNING|trainer.py:803] 2025-04-26 20:59:29,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4455
4406
[WARNING|trainer.py:803] 2025-04-26 20:59:30,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:30,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5209
[WARNING|trainer.py:803] 2025-04-26 20:59:31,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4456
4407
[WARNING|trainer.py:803] 2025-04-26 20:59:31,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:32,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5210
[WARNING|trainer.py:803] 2025-04-26 20:59:32,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4457
4408
[WARNING|trainer.py:803] 2025-04-26 20:59:32,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:33,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5211
4458
[WARNING|trainer.py:803] 2025-04-26 20:59:33,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4409
[WARNING|trainer.py:803] 2025-04-26 20:59:34,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 20:59:34,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
5212
4459
[WARNING|trainer.py:803] 2025-04-26 20:59:34,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4410
[WARNING|trainer.py:803] 2025-04-26 20:59:35,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5213
[WARNING|trainer.py:803] 2025-04-26 20:59:35,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4460
[WARNING|trainer.py:803] 2025-04-26 20:59:35,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4411
[WARNING|trainer.py:803] 2025-04-26 20:59:36,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5214
[WARNING|trainer.py:803] 2025-04-26 20:59:36,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4461
[WARNING|trainer.py:803] 2025-04-26 20:59:37,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4412
[WARNING|trainer.py:803] 2025-04-26 20:59:37,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:37,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5215
4462
[WARNING|trainer.py:803] 2025-04-26 20:59:38,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4413
[WARNING|trainer.py:803] 2025-04-26 20:59:38,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:39,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5216
4463
[WARNING|trainer.py:803] 2025-04-26 20:59:39,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4414
[WARNING|trainer.py:803] 2025-04-26 20:59:39,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:40,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5217
4464
[WARNING|trainer.py:803] 2025-04-26 20:59:40,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4415
[WARNING|trainer.py:803] 2025-04-26 20:59:41,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:41,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5218
4465
[WARNING|trainer.py:803] 2025-04-26 20:59:41,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4416
[WARNING|trainer.py:803] 2025-04-26 20:59:42,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 20:59:42,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5219
4466
[WARNING|trainer.py:803] 2025-04-26 20:59:43,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4417
[WARNING|trainer.py:803] 2025-04-26 20:59:43,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:43,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5220
4467
[WARNING|trainer.py:803] 2025-04-26 20:59:44,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4418
[WARNING|trainer.py:803] 2025-04-26 20:59:44,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 20:59:44,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5221
4468
[WARNING|trainer.py:803] 2025-04-26 20:59:45,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:45,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4419
[WARNING|trainer.py:803] 2025-04-26 20:59:46,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5222
4469
[WARNING|trainer.py:803] 2025-04-26 20:59:46,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:46,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4420
[WARNING|trainer.py:803] 2025-04-26 20:59:47,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5223
4470
[WARNING|trainer.py:803] 2025-04-26 20:59:47,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:48,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4421
[WARNING|trainer.py:803] 2025-04-26 20:59:48,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5224
4471
[WARNING|trainer.py:803] 2025-04-26 20:59:49,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:49,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4422
[WARNING|trainer.py:803] 2025-04-26 20:59:49,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5225
4472
[WARNING|trainer.py:803] 2025-04-26 20:59:50,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:50,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4423
[WARNING|trainer.py:803] 2025-04-26 20:59:50,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5226
4473
[WARNING|trainer.py:803] 2025-04-26 20:59:51,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:51,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4424
[WARNING|trainer.py:803] 2025-04-26 20:59:51,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5227
4474
[WARNING|trainer.py:803] 2025-04-26 20:59:52,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:52,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4425
[WARNING|trainer.py:803] 2025-04-26 20:59:53,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5228
4475
[WARNING|trainer.py:803] 2025-04-26 20:59:53,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:54,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4426
[WARNING|trainer.py:803] 2025-04-26 20:59:54,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5229
4476
[WARNING|trainer.py:803] 2025-04-26 20:59:55,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:55,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4427
[WARNING|trainer.py:803] 2025-04-26 20:59:55,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5230
4477
[WARNING|trainer.py:803] 2025-04-26 20:59:56,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:56,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4428
[WARNING|trainer.py:803] 2025-04-26 20:59:56,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5231
4478
[WARNING|trainer.py:803] 2025-04-26 20:59:57,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:57,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4429
[WARNING|trainer.py:803] 2025-04-26 20:59:57,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5232
4479
[WARNING|trainer.py:803] 2025-04-26 20:59:58,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 20:59:58,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4430
[WARNING|trainer.py:803] 2025-04-26 20:59:58,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5233
4480
[WARNING|trainer.py:803] 2025-04-26 20:59:59,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 20:59:59,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:00,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4431
5234
4481
[WARNING|trainer.py:803] 2025-04-26 21:00:00,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:01,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:01,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4432
5235
4482
[WARNING|trainer.py:803] 2025-04-26 21:00:02,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:02,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:02,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4433
5236
4483
[WARNING|trainer.py:803] 2025-04-26 21:00:03,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:03,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:03,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5237
4434
4484
[WARNING|trainer.py:803] 2025-04-26 21:00:04,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:04,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:04,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5238
4435
4485
[WARNING|trainer.py:803] 2025-04-26 21:00:05,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:05,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:06,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5239
4436
4486
[WARNING|trainer.py:803] 2025-04-26 21:00:06,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:07,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:07,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5240
4437
4487
[WARNING|trainer.py:803] 2025-04-26 21:00:08,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:08,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:08,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5241
4438
4488
[WARNING|trainer.py:803] 2025-04-26 21:00:09,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:09,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5242
[WARNING|trainer.py:803] 2025-04-26 21:00:09,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4439
4489
[WARNING|trainer.py:803] 2025-04-26 21:00:10,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:10,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5243
[WARNING|trainer.py:803] 2025-04-26 21:00:10,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4440
4490
[WARNING|trainer.py:803] 2025-04-26 21:00:11,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:11,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5244
[WARNING|trainer.py:803] 2025-04-26 21:00:11,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4441
4491
[WARNING|trainer.py:803] 2025-04-26 21:00:12,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:13,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:13,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5245
4442
4492
[WARNING|trainer.py:803] 2025-04-26 21:00:14,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:14,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:14,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5246
4443
4493
[WARNING|trainer.py:803] 2025-04-26 21:00:15,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:15,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:15,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5247
4444
4494
[WARNING|trainer.py:803] 2025-04-26 21:00:16,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:16,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5248
[WARNING|trainer.py:803] 2025-04-26 21:00:16,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4445
4495
[WARNING|trainer.py:803] 2025-04-26 21:00:17,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:17,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5249
[WARNING|trainer.py:803] 2025-04-26 21:00:18,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4446
4496
[WARNING|trainer.py:803] 2025-04-26 21:00:18,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5250
[WARNING|trainer.py:803] 2025-04-26 21:00:19,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:19,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4447
4497
[WARNING|trainer.py:803] 2025-04-26 21:00:19,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5251
[WARNING|trainer.py:803] 2025-04-26 21:00:20,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:20,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4448
4498
[WARNING|trainer.py:803] 2025-04-26 21:00:20,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5252
[WARNING|trainer.py:803] 2025-04-26 21:00:21,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:21,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4449
4499
[WARNING|trainer.py:803] 2025-04-26 21:00:22,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5253
[WARNING|trainer.py:803] 2025-04-26 21:00:22,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:22,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4450
[WARNING|trainer.py:803] 2025-04-26 21:00:23,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4500
5254
[WARNING|trainer.py:803] 2025-04-26 21:00:23,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:24,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4451
[WARNING|trainer.py:803] 2025-04-26 21:00:24,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4501
5255
[WARNING|trainer.py:803] 2025-04-26 21:00:25,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:25,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4452
[WARNING|trainer.py:803] 2025-04-26 21:00:25,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4502
5256
[WARNING|trainer.py:803] 2025-04-26 21:00:26,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:26,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4453
[WARNING|trainer.py:803] 2025-04-26 21:00:26,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4503
5257
[WARNING|trainer.py:803] 2025-04-26 21:00:27,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:27,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4454
[WARNING|trainer.py:803] 2025-04-26 21:00:27,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4504
5258
[WARNING|trainer.py:803] 2025-04-26 21:00:28,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:28,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4455
[WARNING|trainer.py:803] 2025-04-26 21:00:29,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4505
5259
[WARNING|trainer.py:803] 2025-04-26 21:00:29,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:30,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4456
[WARNING|trainer.py:803] 2025-04-26 21:00:30,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4506
5260
[WARNING|trainer.py:803] 2025-04-26 21:00:30,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:31,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4457
[WARNING|trainer.py:803] 2025-04-26 21:00:31,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4507
5261
[WARNING|trainer.py:803] 2025-04-26 21:00:32,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:32,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:32,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4458
5262
4508
[WARNING|trainer.py:803] 2025-04-26 21:00:33,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 21:00:33,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:33,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4459
5263
4509
[WARNING|trainer.py:803] 2025-04-26 21:00:34,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:34,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:34,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4460
5264
4510
[WARNING|trainer.py:803] 2025-04-26 21:00:35,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:36,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4461
5265
[WARNING|trainer.py:803] 2025-04-26 21:00:36,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4511
[WARNING|trainer.py:803] 2025-04-26 21:00:36,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:37,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4462
5266
[WARNING|trainer.py:803] 2025-04-26 21:00:37,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4512
[WARNING|trainer.py:803] 2025-04-26 21:00:38,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:38,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4463
5267
[WARNING|trainer.py:803] 2025-04-26 21:00:38,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4513
[WARNING|trainer.py:803] 2025-04-26 21:00:39,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:39,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5268
4464
[WARNING|trainer.py:803] 2025-04-26 21:00:39,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4514
[WARNING|trainer.py:803] 2025-04-26 21:00:40,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:40,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5269
4465
[WARNING|trainer.py:803] 2025-04-26 21:00:41,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4515
[WARNING|trainer.py:803] 2025-04-26 21:00:41,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:41,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5270
4466
[WARNING|trainer.py:803] 2025-04-26 21:00:42,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4516
[WARNING|trainer.py:803] 2025-04-26 21:00:42,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:42,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5271
4467
[WARNING|trainer.py:803] 2025-04-26 21:00:43,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4517
[WARNING|trainer.py:803] 2025-04-26 21:00:44,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:44,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5272
4468
[WARNING|trainer.py:803] 2025-04-26 21:00:44,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4518
[WARNING|trainer.py:803] 2025-04-26 21:00:45,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:00:45,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5273
4469
[WARNING|trainer.py:803] 2025-04-26 21:00:45,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4519
[WARNING|trainer.py:803] 2025-04-26 21:00:46,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5274
[WARNING|trainer.py:803] 2025-04-26 21:00:46,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4470
[WARNING|trainer.py:803] 2025-04-26 21:00:47,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:47,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4520
5275
[WARNING|trainer.py:803] 2025-04-26 21:00:47,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4471
[WARNING|trainer.py:803] 2025-04-26 21:00:48,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:48,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4521
5276
[WARNING|trainer.py:803] 2025-04-26 21:00:48,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4472
[WARNING|trainer.py:803] 2025-04-26 21:00:49,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:49,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4522
5277
[WARNING|trainer.py:803] 2025-04-26 21:00:50,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4473
[WARNING|trainer.py:803] 2025-04-26 21:00:50,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:50,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5278
4523
[WARNING|trainer.py:803] 2025-04-26 21:00:51,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4474
[WARNING|trainer.py:803] 2025-04-26 21:00:51,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:51,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5279
4524
[WARNING|trainer.py:803] 2025-04-26 21:00:52,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4475
[WARNING|trainer.py:803] 2025-04-26 21:00:52,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:53,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5280
4525
[WARNING|trainer.py:803] 2025-04-26 21:00:53,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4476
[WARNING|trainer.py:803] 2025-04-26 21:00:54,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:00:54,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5281
4526
[WARNING|trainer.py:803] 2025-04-26 21:00:54,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4477
[WARNING|trainer.py:803] 2025-04-26 21:00:55,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:55,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5282
4527
[WARNING|trainer.py:803] 2025-04-26 21:00:56,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4478
[WARNING|trainer.py:803] 2025-04-26 21:00:56,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5283
[WARNING|trainer.py:803] 2025-04-26 21:00:56,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4528
[WARNING|trainer.py:803] 2025-04-26 21:00:57,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:57,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4479
5284
[WARNING|trainer.py:803] 2025-04-26 21:00:57,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4529
[WARNING|trainer.py:803] 2025-04-26 21:00:58,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:00:58,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4480
5285
[WARNING|trainer.py:803] 2025-04-26 21:00:59,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4530
[WARNING|trainer.py:803] 2025-04-26 21:00:59,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:00:59,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4481
5286
[WARNING|trainer.py:803] 2025-04-26 21:01:00,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4531
[WARNING|trainer.py:803] 2025-04-26 21:01:00,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:00,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4482
5287
[WARNING|trainer.py:803] 2025-04-26 21:01:01,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4532
[WARNING|trainer.py:803] 2025-04-26 21:01:01,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:02,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4483
5288
[WARNING|trainer.py:803] 2025-04-26 21:01:02,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4533
[WARNING|trainer.py:803] 2025-04-26 21:01:03,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:03,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4484
5289
[WARNING|trainer.py:803] 2025-04-26 21:01:03,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4534
[WARNING|trainer.py:803] 2025-04-26 21:01:04,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:04,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4485
5290
[WARNING|trainer.py:803] 2025-04-26 21:01:04,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4535
[WARNING|trainer.py:803] 2025-04-26 21:01:05,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:05,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4486
5291
[WARNING|trainer.py:803] 2025-04-26 21:01:06,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4536
[WARNING|trainer.py:803] 2025-04-26 21:01:06,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:06,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5292
4487
[WARNING|trainer.py:803] 2025-04-26 21:01:07,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4537
[WARNING|trainer.py:803] 2025-04-26 21:01:07,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:07,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5293
4488
[WARNING|trainer.py:803] 2025-04-26 21:01:08,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4538
[WARNING|trainer.py:803] 2025-04-26 21:01:09,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:09,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5294
4489
[WARNING|trainer.py:803] 2025-04-26 21:01:09,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4539
[WARNING|trainer.py:803] 2025-04-26 21:01:10,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:01:10,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5295
4490
[WARNING|trainer.py:803] 2025-04-26 21:01:10,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4540
[WARNING|trainer.py:803] 2025-04-26 21:01:11,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:11,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5296
4491
[WARNING|trainer.py:803] 2025-04-26 21:01:12,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4541
[WARNING|trainer.py:803] 2025-04-26 21:01:12,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:12,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5297
4492
[WARNING|trainer.py:803] 2025-04-26 21:01:13,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4542
[WARNING|trainer.py:803] 2025-04-26 21:01:13,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:01:13,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5298
[WARNING|trainer.py:803] 2025-04-26 21:01:14,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4493
4543
[WARNING|trainer.py:803] 2025-04-26 21:01:14,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5299
[WARNING|trainer.py:803] 2025-04-26 21:01:15,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:15,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4494
4544
[WARNING|trainer.py:803] 2025-04-26 21:01:16,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5300
[WARNING|trainer.py:803] 2025-04-26 21:01:16,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:16,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4495
4545
[WARNING|trainer.py:803] 2025-04-26 21:01:17,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5301
[WARNING|trainer.py:803] 2025-04-26 21:01:17,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:17,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4496
4546
[WARNING|trainer.py:803] 2025-04-26 21:01:18,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5302
[WARNING|trainer.py:803] 2025-04-26 21:01:18,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:19,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4497
4547
[WARNING|trainer.py:803] 2025-04-26 21:01:19,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5303
[WARNING|trainer.py:803] 2025-04-26 21:01:20,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:20,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4498
4548
[WARNING|trainer.py:803] 2025-04-26 21:01:20,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5304
[WARNING|trainer.py:803] 2025-04-26 21:01:21,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:21,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4499
4549
[WARNING|trainer.py:803] 2025-04-26 21:01:21,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5305
[WARNING|trainer.py:803] 2025-04-26 21:01:22,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:22,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:23,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4550
4500
5306
[WARNING|trainer.py:803] 2025-04-26 21:01:23,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:23,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:24,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4551
4501
5307
[WARNING|trainer.py:803] 2025-04-26 21:01:25,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:25,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:01:25,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4552
4502
5308
[WARNING|trainer.py:803] 2025-04-26 21:01:26,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:26,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:26,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4553
4503
5309
[WARNING|trainer.py:803] 2025-04-26 21:01:27,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:27,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:27,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4554
4504
5310
[WARNING|trainer.py:803] 2025-04-26 21:01:28,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:28,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:28,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4555
5311
4505
[WARNING|trainer.py:803] 2025-04-26 21:01:29,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:29,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:30,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5312
4556
4506
[WARNING|trainer.py:803] 2025-04-26 21:01:31,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:31,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:31,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5313
4557
4507
[WARNING|trainer.py:803] 2025-04-26 21:01:32,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:32,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:32,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5314
4558
4508
[WARNING|trainer.py:803] 2025-04-26 21:01:33,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:01:33,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:01:33,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5315
4559
4509
[WARNING|trainer.py:803] 2025-04-26 21:01:34,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:34,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5316
[WARNING|trainer.py:803] 2025-04-26 21:01:34,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4560
4510
[WARNING|trainer.py:803] 2025-04-26 21:01:35,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:35,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5317
[WARNING|trainer.py:803] 2025-04-26 21:01:36,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4561
4511
[WARNING|trainer.py:803] 2025-04-26 21:01:36,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5318
[WARNING|trainer.py:803] 2025-04-26 21:01:37,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:37,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4562
4512
[WARNING|trainer.py:803] 2025-04-26 21:01:37,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5319
[WARNING|trainer.py:803] 2025-04-26 21:01:38,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:38,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4563
4513
[WARNING|trainer.py:803] 2025-04-26 21:01:38,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5320
[WARNING|trainer.py:803] 2025-04-26 21:01:39,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:39,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4564
4514
[WARNING|trainer.py:803] 2025-04-26 21:01:40,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5321
[WARNING|trainer.py:803] 2025-04-26 21:01:40,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:40,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4565
4515
[WARNING|trainer.py:803] 2025-04-26 21:01:41,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5322
[WARNING|trainer.py:803] 2025-04-26 21:01:41,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:42,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4566
4516
[WARNING|trainer.py:803] 2025-04-26 21:01:42,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5323
[WARNING|trainer.py:803] 2025-04-26 21:01:42,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:43,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4567
4517
[WARNING|trainer.py:803] 2025-04-26 21:01:43,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5324
[WARNING|trainer.py:803] 2025-04-26 21:01:44,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:44,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4568
4518
[WARNING|trainer.py:803] 2025-04-26 21:01:44,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5325
[WARNING|trainer.py:803] 2025-04-26 21:01:45,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:45,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4569
[WARNING|trainer.py:803] 2025-04-26 21:01:45,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4519
5326
[WARNING|trainer.py:803] 2025-04-26 21:01:46,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:46,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4570
[WARNING|trainer.py:803] 2025-04-26 21:01:47,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4520
5327
[WARNING|trainer.py:803] 2025-04-26 21:01:47,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:47,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4571
[WARNING|trainer.py:803] 2025-04-26 21:01:48,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4521
5328
[WARNING|trainer.py:803] 2025-04-26 21:01:48,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:49,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4572
[WARNING|trainer.py:803] 2025-04-26 21:01:49,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4522
5329
[WARNING|trainer.py:803] 2025-04-26 21:01:50,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:50,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4573
[WARNING|trainer.py:803] 2025-04-26 21:01:50,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5330
4523
[WARNING|trainer.py:803] 2025-04-26 21:01:51,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:51,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:51,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4574
5331
4524
[WARNING|trainer.py:803] 2025-04-26 21:01:52,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:52,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:01:52,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4575
5332
4525
[WARNING|trainer.py:803] 2025-04-26 21:01:53,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:53,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4576
[WARNING|trainer.py:803] 2025-04-26 21:01:53,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5333
4526
[WARNING|trainer.py:803] 2025-04-26 21:01:54,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:55,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4577
[WARNING|trainer.py:803] 2025-04-26 21:01:55,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5334
4527
[WARNING|trainer.py:803] 2025-04-26 21:01:55,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:56,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4578
[WARNING|trainer.py:803] 2025-04-26 21:01:56,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5335
4528
[WARNING|trainer.py:803] 2025-04-26 21:01:57,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:01:57,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4579
[WARNING|trainer.py:803] 2025-04-26 21:01:57,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5336
4529
[WARNING|trainer.py:803] 2025-04-26 21:01:58,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:58,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4580
[WARNING|trainer.py:803] 2025-04-26 21:01:58,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5337
4530
[WARNING|trainer.py:803] 2025-04-26 21:01:59,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:01:59,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4581
5338
[WARNING|trainer.py:803] 2025-04-26 21:01:59,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4531
[WARNING|trainer.py:803] 2025-04-26 21:02:00,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:00,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5339
4582
[WARNING|trainer.py:803] 2025-04-26 21:02:01,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4532
[WARNING|trainer.py:803] 2025-04-26 21:02:01,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:02:01,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5340
4583
[WARNING|trainer.py:803] 2025-04-26 21:02:02,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4533
[WARNING|trainer.py:803] 2025-04-26 21:02:02,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:03,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5341
4584
[WARNING|trainer.py:803] 2025-04-26 21:02:03,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4534
[WARNING|trainer.py:803] 2025-04-26 21:02:04,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:04,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5342
4585
[WARNING|trainer.py:803] 2025-04-26 21:02:04,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4535
[WARNING|trainer.py:803] 2025-04-26 21:02:05,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:05,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5343
[WARNING|trainer.py:803] 2025-04-26 21:02:05,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4586
4536
[WARNING|trainer.py:803] 2025-04-26 21:02:06,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:06,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5344
4587
[WARNING|trainer.py:803] 2025-04-26 21:02:07,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4537
[WARNING|trainer.py:803] 2025-04-26 21:02:07,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:07,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5345
4588
[WARNING|trainer.py:803] 2025-04-26 21:02:08,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4538
[WARNING|trainer.py:803] 2025-04-26 21:02:08,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:08,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5346
4589
[WARNING|trainer.py:803] 2025-04-26 21:02:09,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4539
[WARNING|trainer.py:803] 2025-04-26 21:02:09,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:10,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5347
4590
[WARNING|trainer.py:803] 2025-04-26 21:02:10,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4540
[WARNING|trainer.py:803] 2025-04-26 21:02:11,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:11,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5348
4591
[WARNING|trainer.py:803] 2025-04-26 21:02:11,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4541
[WARNING|trainer.py:803] 2025-04-26 21:02:12,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5349
[WARNING|trainer.py:803] 2025-04-26 21:02:12,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4592
[WARNING|trainer.py:803] 2025-04-26 21:02:12,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4542
[WARNING|trainer.py:803] 2025-04-26 21:02:13,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5350
[WARNING|trainer.py:803] 2025-04-26 21:02:13,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4593
[WARNING|trainer.py:803] 2025-04-26 21:02:14,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4543
[WARNING|trainer.py:803] 2025-04-26 21:02:14,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5351
[WARNING|trainer.py:803] 2025-04-26 21:02:14,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4594
[WARNING|trainer.py:803] 2025-04-26 21:02:15,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:15,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4544
5352
[WARNING|trainer.py:803] 2025-04-26 21:02:16,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4595
[WARNING|trainer.py:803] 2025-04-26 21:02:16,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:16,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4545
5353
[WARNING|trainer.py:803] 2025-04-26 21:02:17,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4596
[WARNING|trainer.py:803] 2025-04-26 21:02:17,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:17,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4546
5354
[WARNING|trainer.py:803] 2025-04-26 21:02:18,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4597
[WARNING|trainer.py:803] 2025-04-26 21:02:18,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:18,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4547
5355
[WARNING|trainer.py:803] 2025-04-26 21:02:19,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4598
[WARNING|trainer.py:803] 2025-04-26 21:02:19,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:20,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4548
5356
[WARNING|trainer.py:803] 2025-04-26 21:02:20,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4599
[WARNING|trainer.py:803] 2025-04-26 21:02:21,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:21,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4549
5357
[WARNING|trainer.py:803] 2025-04-26 21:02:21,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4600
[WARNING|trainer.py:803] 2025-04-26 21:02:22,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:22,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4550
5358
[WARNING|trainer.py:803] 2025-04-26 21:02:23,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4601
[WARNING|trainer.py:803] 2025-04-26 21:02:23,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:23,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5359
4551
[WARNING|trainer.py:803] 2025-04-26 21:02:24,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:02:24,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4602
[WARNING|trainer.py:803] 2025-04-26 21:02:24,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5360
4552
[WARNING|trainer.py:803] 2025-04-26 21:02:25,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:25,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4603
[WARNING|trainer.py:803] 2025-04-26 21:02:25,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5361
4553
[WARNING|trainer.py:803] 2025-04-26 21:02:26,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:27,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4604
[WARNING|trainer.py:803] 2025-04-26 21:02:27,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5362
4554
[WARNING|trainer.py:803] 2025-04-26 21:02:27,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:28,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:02:28,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4605
5363
4555
[WARNING|trainer.py:803] 2025-04-26 21:02:29,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:29,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5364
[WARNING|trainer.py:803] 2025-04-26 21:02:29,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4606
4556
[WARNING|trainer.py:803] 2025-04-26 21:02:30,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:02:30,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
5365
[WARNING|trainer.py:803] 2025-04-26 21:02:30,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4607
4557
[WARNING|trainer.py:803] 2025-04-26 21:02:31,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:31,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5366
[WARNING|trainer.py:803] 2025-04-26 21:02:31,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4608
4558
[WARNING|trainer.py:803] 2025-04-26 21:02:32,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:32,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5367
[WARNING|trainer.py:803] 2025-04-26 21:02:33,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4609
4559
[WARNING|trainer.py:803] 2025-04-26 21:02:33,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5368
[WARNING|trainer.py:803] 2025-04-26 21:02:34,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:34,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4610
4560
[WARNING|trainer.py:803] 2025-04-26 21:02:34,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5369
[WARNING|trainer.py:803] 2025-04-26 21:02:35,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:35,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4611
4561
[WARNING|trainer.py:803] 2025-04-26 21:02:35,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5370
[WARNING|trainer.py:803] 2025-04-26 21:02:36,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:36,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4612
4562
[WARNING|trainer.py:803] 2025-04-26 21:02:37,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5371
[WARNING|trainer.py:803] 2025-04-26 21:02:37,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:37,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4613
4563
[WARNING|trainer.py:803] 2025-04-26 21:02:38,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5372
[WARNING|trainer.py:803] 2025-04-26 21:02:38,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:39,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4614
[WARNING|trainer.py:803] 2025-04-26 21:02:39,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4564
5373
[WARNING|trainer.py:803] 2025-04-26 21:02:40,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:40,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4615
[WARNING|trainer.py:803] 2025-04-26 21:02:40,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4565
5374
[WARNING|trainer.py:803] 2025-04-26 21:02:41,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:41,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:41,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4566
4616
5375
[WARNING|trainer.py:803] 2025-04-26 21:02:42,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:42,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:42,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4567
5376
4617
[WARNING|trainer.py:803] 2025-04-26 21:02:43,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:44,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:44,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4568
5377
4618
[WARNING|trainer.py:803] 2025-04-26 21:02:44,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:45,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4569
[WARNING|trainer.py:803] 2025-04-26 21:02:45,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5378
4619
[WARNING|trainer.py:803] 2025-04-26 21:02:46,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:46,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4570
[WARNING|trainer.py:803] 2025-04-26 21:02:46,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5379
4620
[WARNING|trainer.py:803] 2025-04-26 21:02:47,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:47,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4571
5380
[WARNING|trainer.py:803] 2025-04-26 21:02:47,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4621
[WARNING|trainer.py:803] 2025-04-26 21:02:48,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:48,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5381
4572
[WARNING|trainer.py:803] 2025-04-26 21:02:49,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4622
[WARNING|trainer.py:803] 2025-04-26 21:02:49,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:49,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5382
4573
[WARNING|trainer.py:803] 2025-04-26 21:02:50,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4623
[WARNING|trainer.py:803] 2025-04-26 21:02:50,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:50,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5383
4574
[WARNING|trainer.py:803] 2025-04-26 21:02:51,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 21:02:52,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:52,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5384
4575
[WARNING|trainer.py:803] 2025-04-26 21:02:52,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4625
[WARNING|trainer.py:803] 2025-04-26 21:02:53,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:02:53,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5385
4576
[WARNING|trainer.py:803] 2025-04-26 21:02:53,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4626
[WARNING|trainer.py:803] 2025-04-26 21:02:54,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:54,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5386
4577
[WARNING|trainer.py:803] 2025-04-26 21:02:55,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4627
[WARNING|trainer.py:803] 2025-04-26 21:02:55,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:55,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5387
4578
[WARNING|trainer.py:803] 2025-04-26 21:02:56,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4628
[WARNING|trainer.py:803] 2025-04-26 21:02:56,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:02:56,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5388
4579
[WARNING|trainer.py:803] 2025-04-26 21:02:57,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4629
[WARNING|trainer.py:803] 2025-04-26 21:02:57,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5389
[WARNING|trainer.py:803] 2025-04-26 21:02:58,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4580
[WARNING|trainer.py:803] 2025-04-26 21:02:58,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:02:58,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4630
5390
[WARNING|trainer.py:803] 2025-04-26 21:02:59,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4581
[WARNING|trainer.py:803] 2025-04-26 21:02:59,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:03:00,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4631
5391
[WARNING|trainer.py:803] 2025-04-26 21:03:00,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4582
[WARNING|trainer.py:803] 2025-04-26 21:03:01,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:01,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4632
5392
[WARNING|trainer.py:803] 2025-04-26 21:03:01,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4583
[WARNING|trainer.py:803] 2025-04-26 21:03:02,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:02,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5393
4633
[WARNING|trainer.py:803] 2025-04-26 21:03:02,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4584
[WARNING|trainer.py:803] 2025-04-26 21:03:03,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:03,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5394
4634
[WARNING|trainer.py:803] 2025-04-26 21:03:04,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4585
[WARNING|trainer.py:803] 2025-04-26 21:03:04,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:04,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5395
4635
[WARNING|trainer.py:803] 2025-04-26 21:03:05,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4586
[WARNING|trainer.py:803] 2025-04-26 21:03:05,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:05,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5396
4636
[WARNING|trainer.py:803] 2025-04-26 21:03:06,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4587
[WARNING|trainer.py:803] 2025-04-26 21:03:06,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:07,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5397
4637
[WARNING|trainer.py:803] 2025-04-26 21:03:07,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:07,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4588
5398
[WARNING|trainer.py:803] 2025-04-26 21:03:08,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4638
[WARNING|trainer.py:803] 2025-04-26 21:03:08,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:09,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4589
5399
[WARNING|trainer.py:803] 2025-04-26 21:03:09,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4639
[WARNING|trainer.py:803] 2025-04-26 21:03:09,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:10,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4590
5400
[WARNING|trainer.py:803] 2025-04-26 21:03:10,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4640
[WARNING|trainer.py:803] 2025-04-26 21:03:11,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:11,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4591
5401
[WARNING|trainer.py:803] 2025-04-26 21:03:11,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
[WARNING|trainer.py:803] 2025-04-26 21:03:12,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:12,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4592
[WARNING|trainer.py:803] 2025-04-26 21:03:13,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5402
4642
[WARNING|trainer.py:803] 2025-04-26 21:03:13,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4593
[WARNING|trainer.py:803] 2025-04-26 21:03:13,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:03:14,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5403
4643
[WARNING|trainer.py:803] 2025-04-26 21:03:14,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4594
[WARNING|trainer.py:803] 2025-04-26 21:03:15,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:15,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5404
4644
[WARNING|trainer.py:803] 2025-04-26 21:03:15,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4595
[WARNING|trainer.py:803] 2025-04-26 21:03:16,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:16,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5405
4645
[WARNING|trainer.py:803] 2025-04-26 21:03:17,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4596
[WARNING|trainer.py:803] 2025-04-26 21:03:17,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:17,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4646
[WARNING|trainer.py:803] 2025-04-26 21:03:18,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5406
4597
[WARNING|trainer.py:803] 2025-04-26 21:03:19,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:19,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:19,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4647
5407
4598
[WARNING|trainer.py:803] 2025-04-26 21:03:20,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:20,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:20,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4648
5408
4599
[WARNING|trainer.py:803] 2025-04-26 21:03:21,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:21,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:21,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4649
4600
5409
[WARNING|trainer.py:803] 2025-04-26 21:03:22,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:23,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:23,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4650
4601
5410
[WARNING|trainer.py:803] 2025-04-26 21:03:23,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:24,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4651
[WARNING|trainer.py:803] 2025-04-26 21:03:24,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4602
5411
[WARNING|trainer.py:803] 2025-04-26 21:03:25,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:25,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4652
[WARNING|trainer.py:803] 2025-04-26 21:03:25,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4603
5412
[WARNING|trainer.py:803] 2025-04-26 21:03:26,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:26,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4653
[WARNING|trainer.py:803] 2025-04-26 21:03:26,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4604
5413
[WARNING|trainer.py:803] 2025-04-26 21:03:27,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:27,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4654
[WARNING|trainer.py:803] 2025-04-26 21:03:28,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4605
5414
[WARNING|trainer.py:803] 2025-04-26 21:03:28,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4655
[WARNING|trainer.py:803] 2025-04-26 21:03:29,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:29,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4606
5415
[WARNING|trainer.py:803] 2025-04-26 21:03:29,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4656
[WARNING|trainer.py:803] 2025-04-26 21:03:30,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4607
[WARNING|trainer.py:803] 2025-04-26 21:03:30,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:03:31,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5416
4657
[WARNING|trainer.py:803] 2025-04-26 21:03:31,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4608
[WARNING|trainer.py:803] 2025-04-26 21:03:31,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:32,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5417
4658
[WARNING|trainer.py:803] 2025-04-26 21:03:32,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4609
[WARNING|trainer.py:803] 2025-04-26 21:03:33,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:03:33,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5418
4659
[WARNING|trainer.py:803] 2025-04-26 21:03:33,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4610
[WARNING|trainer.py:803] 2025-04-26 21:03:34,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:34,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5419
[WARNING|trainer.py:803] 2025-04-26 21:03:35,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4660
4611
[WARNING|trainer.py:803] 2025-04-26 21:03:35,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:35,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:36,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5420
4661
4612
[WARNING|trainer.py:803] 2025-04-26 21:03:37,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:37,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:37,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4662
5421
4613
[WARNING|trainer.py:803] 2025-04-26 21:03:38,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:38,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:03:38,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4663
5422
4614
[WARNING|trainer.py:803] 2025-04-26 21:03:39,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:39,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:39,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4664
5423
4615
[WARNING|trainer.py:803] 2025-04-26 21:03:40,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:41,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:41,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4665
4616
5424
[WARNING|trainer.py:803] 2025-04-26 21:03:41,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:42,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4666
[WARNING|trainer.py:803] 2025-04-26 21:03:42,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4617
5425
[WARNING|trainer.py:803] 2025-04-26 21:03:43,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4667
[WARNING|trainer.py:803] 2025-04-26 21:03:43,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:43,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4618
5426
[WARNING|trainer.py:803] 2025-04-26 21:03:44,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4668
[WARNING|trainer.py:803] 2025-04-26 21:03:44,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:44,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4619
5427
[WARNING|trainer.py:803] 2025-04-26 21:03:45,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4669
[WARNING|trainer.py:803] 2025-04-26 21:03:45,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:46,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4620
[WARNING|trainer.py:803] 2025-04-26 21:03:46,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5428
4670
[WARNING|trainer.py:803] 2025-04-26 21:03:47,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4621
[WARNING|trainer.py:803] 2025-04-26 21:03:47,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:03:47,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5429
4671
[WARNING|trainer.py:803] 2025-04-26 21:03:48,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4622
[WARNING|trainer.py:803] 2025-04-26 21:03:48,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:49,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5430
4672
[WARNING|trainer.py:803] 2025-04-26 21:03:49,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4623
[WARNING|trainer.py:803] 2025-04-26 21:03:50,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:03:50,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4673
5431
[WARNING|trainer.py:803] 2025-04-26 21:03:50,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4624
[WARNING|trainer.py:803] 2025-04-26 21:03:51,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:51,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4674
[WARNING|trainer.py:803] 2025-04-26 21:03:51,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5432
4625
[WARNING|trainer.py:803] 2025-04-26 21:03:52,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:52,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4675
[WARNING|trainer.py:803] 2025-04-26 21:03:53,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5433
4626
[WARNING|trainer.py:803] 2025-04-26 21:03:53,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:54,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4676
[WARNING|trainer.py:803] 2025-04-26 21:03:54,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5434
4627
[WARNING|trainer.py:803] 2025-04-26 21:03:55,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:03:55,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4677
[WARNING|trainer.py:803] 2025-04-26 21:03:55,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5435
4628
[WARNING|trainer.py:803] 2025-04-26 21:03:56,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4678
[WARNING|trainer.py:803] 2025-04-26 21:03:56,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:56,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4629
5436
[WARNING|trainer.py:803] 2025-04-26 21:03:57,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4679
[WARNING|trainer.py:803] 2025-04-26 21:03:57,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:03:58,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4630
5437
[WARNING|trainer.py:803] 2025-04-26 21:03:58,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4680
[WARNING|trainer.py:803] 2025-04-26 21:03:59,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:03:59,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4631
5438
[WARNING|trainer.py:803] 2025-04-26 21:03:59,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4681
[WARNING|trainer.py:803] 2025-04-26 21:04:00,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:00,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4632
5439
[WARNING|trainer.py:803] 2025-04-26 21:04:01,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4682
[WARNING|trainer.py:803] 2025-04-26 21:04:01,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:01,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4633
5440
[WARNING|trainer.py:803] 2025-04-26 21:04:02,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4683
[WARNING|trainer.py:803] 2025-04-26 21:04:02,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4634
[WARNING|trainer.py:803] 2025-04-26 21:04:03,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:03,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5441
4684
[WARNING|trainer.py:803] 2025-04-26 21:04:03,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4635
[WARNING|trainer.py:803] 2025-04-26 21:04:04,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:04,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5442
4685
[WARNING|trainer.py:803] 2025-04-26 21:04:05,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4636
[WARNING|trainer.py:803] 2025-04-26 21:04:05,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:05,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5443
4686
[WARNING|trainer.py:803] 2025-04-26 21:04:06,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4637
[WARNING|trainer.py:803] 2025-04-26 21:04:07,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:04:07,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4687
[WARNING|trainer.py:803] 2025-04-26 21:04:07,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5444
4638
[WARNING|trainer.py:803] 2025-04-26 21:04:08,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:08,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4688
[WARNING|trainer.py:803] 2025-04-26 21:04:08,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5445
4639
[WARNING|trainer.py:803] 2025-04-26 21:04:09,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:09,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4689
[WARNING|trainer.py:803] 2025-04-26 21:04:09,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5446
4640
[WARNING|trainer.py:803] 2025-04-26 21:04:10,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:11,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4690
[WARNING|trainer.py:803] 2025-04-26 21:04:11,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4641
5447
[WARNING|trainer.py:803] 2025-04-26 21:04:11,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4691
[WARNING|trainer.py:803] 2025-04-26 21:04:12,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:12,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4642
5448
[WARNING|trainer.py:803] 2025-04-26 21:04:13,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4692
[WARNING|trainer.py:803] 2025-04-26 21:04:13,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:13,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4643
5449
[WARNING|trainer.py:803] 2025-04-26 21:04:14,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4693
[WARNING|trainer.py:803] 2025-04-26 21:04:14,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:14,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4644
5450
[WARNING|trainer.py:803] 2025-04-26 21:04:15,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:15,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4694
4645
[WARNING|trainer.py:803] 2025-04-26 21:04:16,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:16,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5451
4695
[WARNING|trainer.py:803] 2025-04-26 21:04:17,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4646
[WARNING|trainer.py:803] 2025-04-26 21:04:17,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:17,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5452
4696
[WARNING|trainer.py:803] 2025-04-26 21:04:18,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4647
[WARNING|trainer.py:803] 2025-04-26 21:04:19,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:19,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5453
4697
[WARNING|trainer.py:803] 2025-04-26 21:04:19,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4648
[WARNING|trainer.py:803] 2025-04-26 21:04:20,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:20,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4698
[WARNING|trainer.py:803] 2025-04-26 21:04:20,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5454
4649
[WARNING|trainer.py:803] 2025-04-26 21:04:21,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:21,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4699
[WARNING|trainer.py:803] 2025-04-26 21:04:21,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5455
4650
[WARNING|trainer.py:803] 2025-04-26 21:04:22,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:22,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4700
[WARNING|trainer.py:803] 2025-04-26 21:04:23,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5456
4651
[WARNING|trainer.py:803] 2025-04-26 21:04:23,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4701
[WARNING|trainer.py:803] 2025-04-26 21:04:24,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:24,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4652
5457
[WARNING|trainer.py:803] 2025-04-26 21:04:25,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4702
[WARNING|trainer.py:803] 2025-04-26 21:04:25,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:25,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4653
5458
[WARNING|trainer.py:803] 2025-04-26 21:04:26,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4703
[WARNING|trainer.py:803] 2025-04-26 21:04:26,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:26,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4654
5459
[WARNING|trainer.py:803] 2025-04-26 21:04:27,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4704
[WARNING|trainer.py:803] 2025-04-26 21:04:27,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:28,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4655
5460
[WARNING|trainer.py:803] 2025-04-26 21:04:28,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4705
[WARNING|trainer.py:803] 2025-04-26 21:04:29,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:29,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4656
[WARNING|trainer.py:803] 2025-04-26 21:04:29,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5461
4706
[WARNING|trainer.py:803] 2025-04-26 21:04:30,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4657
[WARNING|trainer.py:803] 2025-04-26 21:04:30,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:04:30,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5462
4707
[WARNING|trainer.py:803] 2025-04-26 21:04:31,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4658
[WARNING|trainer.py:803] 2025-04-26 21:04:31,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:32,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5463
4708
[WARNING|trainer.py:803] 2025-04-26 21:04:32,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4659
[WARNING|trainer.py:803] 2025-04-26 21:04:33,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:04:33,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4709
5464
[WARNING|trainer.py:803] 2025-04-26 21:04:33,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4660
[WARNING|trainer.py:803] 2025-04-26 21:04:34,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:34,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4710
5465
[WARNING|trainer.py:803] 2025-04-26 21:04:35,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4661
[WARNING|trainer.py:803] 2025-04-26 21:04:35,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:35,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4711
[WARNING|trainer.py:803] 2025-04-26 21:04:36,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5466
4662
[WARNING|trainer.py:803] 2025-04-26 21:04:36,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:37,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4712
[WARNING|trainer.py:803] 2025-04-26 21:04:37,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5467
4663
[WARNING|trainer.py:803] 2025-04-26 21:04:38,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:38,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4713
[WARNING|trainer.py:803] 2025-04-26 21:04:38,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5468
4664
[WARNING|trainer.py:803] 2025-04-26 21:04:39,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4714
[WARNING|trainer.py:803] 2025-04-26 21:04:39,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:39,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5469
4665
[WARNING|trainer.py:803] 2025-04-26 21:04:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4715
[WARNING|trainer.py:803] 2025-04-26 21:04:40,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:04:41,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5470
4666
[WARNING|trainer.py:803] 2025-04-26 21:04:41,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4716
[WARNING|trainer.py:803] 2025-04-26 21:04:42,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:42,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5471
4667
[WARNING|trainer.py:803] 2025-04-26 21:04:42,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4717
[WARNING|trainer.py:803] 2025-04-26 21:04:43,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:43,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4668
5472
[WARNING|trainer.py:803] 2025-04-26 21:04:43,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4718
[WARNING|trainer.py:803] 2025-04-26 21:04:44,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:44,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4669
[WARNING|trainer.py:803] 2025-04-26 21:04:45,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5473
4719
[WARNING|trainer.py:803] 2025-04-26 21:04:45,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:45,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4670
[WARNING|trainer.py:803] 2025-04-26 21:04:46,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5474
4720
[WARNING|trainer.py:803] 2025-04-26 21:04:47,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:47,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4671
[WARNING|trainer.py:803] 2025-04-26 21:04:47,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5475
4721
[WARNING|trainer.py:803] 2025-04-26 21:04:48,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:48,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4672
[WARNING|trainer.py:803] 2025-04-26 21:04:48,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5476
4722
[WARNING|trainer.py:803] 2025-04-26 21:04:49,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4673
[WARNING|trainer.py:803] 2025-04-26 21:04:49,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:04:49,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4723
5477
[WARNING|trainer.py:803] 2025-04-26 21:04:50,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4674
[WARNING|trainer.py:803] 2025-04-26 21:04:51,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:51,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4724
5478
[WARNING|trainer.py:803] 2025-04-26 21:04:51,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4675
[WARNING|trainer.py:803] 2025-04-26 21:04:52,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:52,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4725
5479
[WARNING|trainer.py:803] 2025-04-26 21:04:53,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4676
[WARNING|trainer.py:803] 2025-04-26 21:04:53,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:53,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4726
5480
[WARNING|trainer.py:803] 2025-04-26 21:04:54,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4677
[WARNING|trainer.py:803] 2025-04-26 21:04:54,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:04:54,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4727
5481
[WARNING|trainer.py:803] 2025-04-26 21:04:55,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4678
[WARNING|trainer.py:803] 2025-04-26 21:04:55,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4728
[WARNING|trainer.py:803] 2025-04-26 21:04:56,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:56,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5482
4679
[WARNING|trainer.py:803] 2025-04-26 21:04:56,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4729
[WARNING|trainer.py:803] 2025-04-26 21:04:57,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:57,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5483
4680
[WARNING|trainer.py:803] 2025-04-26 21:04:58,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4730
[WARNING|trainer.py:803] 2025-04-26 21:04:58,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:04:58,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5484
[WARNING|trainer.py:803] 2025-04-26 21:04:59,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4681
4731
[WARNING|trainer.py:803] 2025-04-26 21:05:00,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:00,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:00,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4682
5485
4732
[WARNING|trainer.py:803] 2025-04-26 21:05:01,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:01,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:01,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4683
5486
4733
[WARNING|trainer.py:803] 2025-04-26 21:05:02,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:02,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:02,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4684
4734
5487
[WARNING|trainer.py:803] 2025-04-26 21:05:03,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:04,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4685
[WARNING|trainer.py:803] 2025-04-26 21:05:04,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4735
5488
[WARNING|trainer.py:803] 2025-04-26 21:05:04,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:05,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4686
[WARNING|trainer.py:803] 2025-04-26 21:05:05,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4736
5489
[WARNING|trainer.py:803] 2025-04-26 21:05:06,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:06,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4687
[WARNING|trainer.py:803] 2025-04-26 21:05:06,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4737
5490
[WARNING|trainer.py:803] 2025-04-26 21:05:07,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:07,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4688
[WARNING|trainer.py:803] 2025-04-26 21:05:07,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4738
5491
[WARNING|trainer.py:803] 2025-04-26 21:05:08,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:08,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4689
4739
[WARNING|trainer.py:803] 2025-04-26 21:05:09,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5492
[WARNING|trainer.py:803] 2025-04-26 21:05:09,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:10,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4690
4740
[WARNING|trainer.py:803] 2025-04-26 21:05:10,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:10,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5493
[WARNING|trainer.py:803] 2025-04-26 21:05:11,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4691
4741
[WARNING|trainer.py:803] 2025-04-26 21:05:11,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:12,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5494
[WARNING|trainer.py:803] 2025-04-26 21:05:12,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4692
4742
[WARNING|trainer.py:803] 2025-04-26 21:05:13,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:13,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:13,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5495
4693
4743
[WARNING|trainer.py:803] 2025-04-26 21:05:14,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:14,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:14,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5496
4694
4744
[WARNING|trainer.py:803] 2025-04-26 21:05:15,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:15,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:15,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4695
5497
4745
[WARNING|trainer.py:803] 2025-04-26 21:05:16,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:17,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:17,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4696
5498
4746
[WARNING|trainer.py:803] 2025-04-26 21:05:18,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:18,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:18,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4697
4747
5499
[WARNING|trainer.py:803] 2025-04-26 21:05:19,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:19,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:19,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4698
4748
5500
[WARNING|trainer.py:803] 2025-04-26 21:05:20,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:20,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:20,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4699
4749
5501
[WARNING|trainer.py:803] 2025-04-26 21:05:21,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:21,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4700
[WARNING|trainer.py:803] 2025-04-26 21:05:22,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4750
5502
[WARNING|trainer.py:803] 2025-04-26 21:05:22,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:23,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4701
[WARNING|trainer.py:803] 2025-04-26 21:05:23,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4751
5503
[WARNING|trainer.py:803] 2025-04-26 21:05:24,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:24,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4702
[WARNING|trainer.py:803] 2025-04-26 21:05:24,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4752
5504
[WARNING|trainer.py:803] 2025-04-26 21:05:25,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:25,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4703
4753
[WARNING|trainer.py:803] 2025-04-26 21:05:25,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5505
[WARNING|trainer.py:803] 2025-04-26 21:05:26,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:26,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4704
4754
[WARNING|trainer.py:803] 2025-04-26 21:05:27,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5506
[WARNING|trainer.py:803] 2025-04-26 21:05:27,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:27,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4705
4755
[WARNING|trainer.py:803] 2025-04-26 21:05:28,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5507
[WARNING|trainer.py:803] 2025-04-26 21:05:28,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:29,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4706
4756
[WARNING|trainer.py:803] 2025-04-26 21:05:29,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5508
[WARNING|trainer.py:803] 2025-04-26 21:05:30,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:30,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4707
4757
[WARNING|trainer.py:803] 2025-04-26 21:05:30,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5509
[WARNING|trainer.py:803] 2025-04-26 21:05:31,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:31,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4708
4758
[WARNING|trainer.py:803] 2025-04-26 21:05:31,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5510
[WARNING|trainer.py:803] 2025-04-26 21:05:32,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:32,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4709
4759
[WARNING|trainer.py:803] 2025-04-26 21:05:33,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5511
[WARNING|trainer.py:803] 2025-04-26 21:05:33,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:33,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4710
4760
[WARNING|trainer.py:803] 2025-04-26 21:05:34,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5512
[WARNING|trainer.py:803] 2025-04-26 21:05:34,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:35,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4711
4761
[WARNING|trainer.py:803] 2025-04-26 21:05:35,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5513
[WARNING|trainer.py:803] 2025-04-26 21:05:36,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:36,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4712
4762
[WARNING|trainer.py:803] 2025-04-26 21:05:36,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5514
[WARNING|trainer.py:803] 2025-04-26 21:05:37,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:37,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4713
4763
[WARNING|trainer.py:803] 2025-04-26 21:05:38,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5515
[WARNING|trainer.py:803] 2025-04-26 21:05:38,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:38,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4714
4764
[WARNING|trainer.py:803] 2025-04-26 21:05:39,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5516
[WARNING|trainer.py:803] 2025-04-26 21:05:39,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:39,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4765
4715
[WARNING|trainer.py:803] 2025-04-26 21:05:40,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5517
[WARNING|trainer.py:803] 2025-04-26 21:05:40,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:40,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4766
4716
[WARNING|trainer.py:803] 2025-04-26 21:05:41,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:42,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:42,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5518
4767
4717
[WARNING|trainer.py:803] 2025-04-26 21:05:43,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:43,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:43,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5519
4768
4718
[WARNING|trainer.py:803] 2025-04-26 21:05:44,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:44,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:44,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5520
4769
4719
[WARNING|trainer.py:803] 2025-04-26 21:05:45,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:45,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:45,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5521
4770
4720
[WARNING|trainer.py:803] 2025-04-26 21:05:46,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:46,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:47,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4771
5522
4721
[WARNING|trainer.py:803] 2025-04-26 21:05:48,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:48,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:05:48,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4772
5523
4722
[WARNING|trainer.py:803] 2025-04-26 21:05:49,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:49,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:49,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4773
5524
4723
[WARNING|trainer.py:803] 2025-04-26 21:05:50,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:50,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:50,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4774
5525
4724
[WARNING|trainer.py:803] 2025-04-26 21:05:51,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:51,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:05:51,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4775
4725
5526
[WARNING|trainer.py:803] 2025-04-26 21:05:52,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:53,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:53,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4776
4726
5527
[WARNING|trainer.py:803] 2025-04-26 21:05:54,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:54,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:54,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4777
4727
5528
[WARNING|trainer.py:803] 2025-04-26 21:05:55,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:55,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:55,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4778
4728
5529
[WARNING|trainer.py:803] 2025-04-26 21:05:56,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:56,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:56,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4779
4729
5530
[WARNING|trainer.py:803] 2025-04-26 21:05:57,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:05:57,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4780
[WARNING|trainer.py:803] 2025-04-26 21:05:58,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4730
5531
[WARNING|trainer.py:803] 2025-04-26 21:05:58,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:05:59,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4781
[WARNING|trainer.py:803] 2025-04-26 21:05:59,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4731
5532
[WARNING|trainer.py:803] 2025-04-26 21:06:00,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:00,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4782
[WARNING|trainer.py:803] 2025-04-26 21:06:00,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4732
5533
[WARNING|trainer.py:803] 2025-04-26 21:06:01,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:01,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4783
[WARNING|trainer.py:803] 2025-04-26 21:06:01,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4733
5534
[WARNING|trainer.py:803] 2025-04-26 21:06:02,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:02,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4784
[WARNING|trainer.py:803] 2025-04-26 21:06:03,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4734
5535
[WARNING|trainer.py:803] 2025-04-26 21:06:03,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:04,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4785
[WARNING|trainer.py:803] 2025-04-26 21:06:04,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4735
5536
[WARNING|trainer.py:803] 2025-04-26 21:06:04,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:05,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4786
[WARNING|trainer.py:803] 2025-04-26 21:06:05,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4736
5537
[WARNING|trainer.py:803] 2025-04-26 21:06:06,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:06,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4787
[WARNING|trainer.py:803] 2025-04-26 21:06:06,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4737
[WARNING|trainer.py:803] 2025-04-26 21:06:07,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5538
[WARNING|trainer.py:803] 2025-04-26 21:06:07,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4788
4738
[WARNING|trainer.py:803] 2025-04-26 21:06:08,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:08,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:08,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5539
4789
4739
[WARNING|trainer.py:803] 2025-04-26 21:06:09,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:09,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:10,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4790
5540
4740
[WARNING|trainer.py:803] 2025-04-26 21:06:10,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:11,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:11,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
4741
5541
[WARNING|trainer.py:803] 2025-04-26 21:06:12,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:12,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4792
[WARNING|trainer.py:803] 2025-04-26 21:06:12,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4742
5542
[WARNING|trainer.py:803] 2025-04-26 21:06:13,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:13,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4793
4743
[WARNING|trainer.py:803] 2025-04-26 21:06:14,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:14,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5543
[WARNING|trainer.py:803] 2025-04-26 21:06:14,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4794
4744
[WARNING|trainer.py:803] 2025-04-26 21:06:15,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:15,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4795
5544
[WARNING|trainer.py:803] 2025-04-26 21:06:16,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4745
[WARNING|trainer.py:803] 2025-04-26 21:06:16,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:17,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4796
[WARNING|trainer.py:803] 2025-04-26 21:06:17,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5545
4746
[WARNING|trainer.py:803] 2025-04-26 21:06:18,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4797
[WARNING|trainer.py:803] 2025-04-26 21:06:18,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:18,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4747
5546
[WARNING|trainer.py:803] 2025-04-26 21:06:19,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4798
[WARNING|trainer.py:803] 2025-04-26 21:06:19,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:20,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4748
[WARNING|trainer.py:803] 2025-04-26 21:06:20,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5547
4799
[WARNING|trainer.py:803] 2025-04-26 21:06:21,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4749
[WARNING|trainer.py:803] 2025-04-26 21:06:21,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:21,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4800
[WARNING|trainer.py:803] 2025-04-26 21:06:22,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5548
4750
[WARNING|trainer.py:803] 2025-04-26 21:06:22,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:23,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4801
[WARNING|trainer.py:803] 2025-04-26 21:06:23,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5549
4751
[WARNING|trainer.py:803] 2025-04-26 21:06:24,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:24,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4802
[WARNING|trainer.py:803] 2025-04-26 21:06:24,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5550
4752
[WARNING|trainer.py:803] 2025-04-26 21:06:25,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4803
[WARNING|trainer.py:803] 2025-04-26 21:06:25,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:25,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5551
4753
[WARNING|trainer.py:803] 2025-04-26 21:06:26,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4804
[WARNING|trainer.py:803] 2025-04-26 21:06:27,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:27,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4754
5552
[WARNING|trainer.py:803] 2025-04-26 21:06:27,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4805
[WARNING|trainer.py:803] 2025-04-26 21:06:28,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:28,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4755
5553
[WARNING|trainer.py:803] 2025-04-26 21:06:28,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4806
[WARNING|trainer.py:803] 2025-04-26 21:06:29,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:29,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4756
5554
[WARNING|trainer.py:803] 2025-04-26 21:06:30,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4807
[WARNING|trainer.py:803] 2025-04-26 21:06:30,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:30,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4757
[WARNING|trainer.py:803] 2025-04-26 21:06:31,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5555
4808
[WARNING|trainer.py:803] 2025-04-26 21:06:31,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4758
[WARNING|trainer.py:803] 2025-04-26 21:06:32,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:32,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5556
4809
[WARNING|trainer.py:803] 2025-04-26 21:06:33,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4759
[WARNING|trainer.py:803] 2025-04-26 21:06:33,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:33,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5557
4810
[WARNING|trainer.py:803] 2025-04-26 21:06:34,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4760
[WARNING|trainer.py:803] 2025-04-26 21:06:34,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:34,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4811
5558
[WARNING|trainer.py:803] 2025-04-26 21:06:35,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4761
[WARNING|trainer.py:803] 2025-04-26 21:06:35,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:36,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4812
[WARNING|trainer.py:803] 2025-04-26 21:06:36,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5559
4762
[WARNING|trainer.py:803] 2025-04-26 21:06:37,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:37,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4813
[WARNING|trainer.py:803] 2025-04-26 21:06:37,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5560
4763
[WARNING|trainer.py:803] 2025-04-26 21:06:38,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4814
[WARNING|trainer.py:803] 2025-04-26 21:06:38,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:38,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5561
4764
[WARNING|trainer.py:803] 2025-04-26 21:06:39,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4815
[WARNING|trainer.py:803] 2025-04-26 21:06:40,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:40,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4765
5562
[WARNING|trainer.py:803] 2025-04-26 21:06:40,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4816
[WARNING|trainer.py:803] 2025-04-26 21:06:41,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:41,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4766
5563
[WARNING|trainer.py:803] 2025-04-26 21:06:41,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4817
[WARNING|trainer.py:803] 2025-04-26 21:06:42,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:42,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4767
[WARNING|trainer.py:803] 2025-04-26 21:06:43,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5564
4818
[WARNING|trainer.py:803] 2025-04-26 21:06:43,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:44,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4768
[WARNING|trainer.py:803] 2025-04-26 21:06:44,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5565
4819
[WARNING|trainer.py:803] 2025-04-26 21:06:44,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4769
[WARNING|trainer.py:803] 2025-04-26 21:06:45,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:45,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5566
4820
[WARNING|trainer.py:803] 2025-04-26 21:06:46,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4770
[WARNING|trainer.py:803] 2025-04-26 21:06:46,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:46,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4821
5567
[WARNING|trainer.py:803] 2025-04-26 21:06:47,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4771
[WARNING|trainer.py:803] 2025-04-26 21:06:47,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:47,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4822
5568
[WARNING|trainer.py:803] 2025-04-26 21:06:48,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4772
[WARNING|trainer.py:803] 2025-04-26 21:06:48,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:49,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4823
[WARNING|trainer.py:803] 2025-04-26 21:06:49,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5569
4773
[WARNING|trainer.py:803] 2025-04-26 21:06:50,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4824
[WARNING|trainer.py:803] 2025-04-26 21:06:50,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:06:50,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5570
4774
[WARNING|trainer.py:803] 2025-04-26 21:06:51,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4825
[WARNING|trainer.py:803] 2025-04-26 21:06:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:52,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5571
4775
[WARNING|trainer.py:803] 2025-04-26 21:06:52,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4826
[WARNING|trainer.py:803] 2025-04-26 21:06:53,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:53,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5572
4776
[WARNING|trainer.py:803] 2025-04-26 21:06:53,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4827
[WARNING|trainer.py:803] 2025-04-26 21:06:54,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:54,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5573
4777
[WARNING|trainer.py:803] 2025-04-26 21:06:54,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4828
[WARNING|trainer.py:803] 2025-04-26 21:06:55,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:55,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5574
[WARNING|trainer.py:803] 2025-04-26 21:06:56,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4778
4829
[WARNING|trainer.py:803] 2025-04-26 21:06:56,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:56,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:57,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5575
4779
4830
[WARNING|trainer.py:803] 2025-04-26 21:06:58,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:06:58,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:06:58,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4780
5576
4831
[WARNING|trainer.py:803] 2025-04-26 21:06:59,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:59,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:06:59,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4781
4832
5577
[WARNING|trainer.py:803] 2025-04-26 21:07:00,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:00,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:00,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4782
4833
5578
[WARNING|trainer.py:803] 2025-04-26 21:07:01,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:01,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4783
[WARNING|trainer.py:803] 2025-04-26 21:07:02,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4834
5579
[WARNING|trainer.py:803] 2025-04-26 21:07:02,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:03,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4784
4835
[WARNING|trainer.py:803] 2025-04-26 21:07:03,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5580
[WARNING|trainer.py:803] 2025-04-26 21:07:04,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:04,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4785
4836
[WARNING|trainer.py:803] 2025-04-26 21:07:04,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:05,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5581
[WARNING|trainer.py:803] 2025-04-26 21:07:05,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4786
4837
[WARNING|trainer.py:803] 2025-04-26 21:07:06,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:06,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5582
[WARNING|trainer.py:803] 2025-04-26 21:07:06,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4787
4838
[WARNING|trainer.py:803] 2025-04-26 21:07:07,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:07,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5583
[WARNING|trainer.py:803] 2025-04-26 21:07:07,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4788
4839
[WARNING|trainer.py:803] 2025-04-26 21:07:08,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:08,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:09,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5584
4789
4840
[WARNING|trainer.py:803] 2025-04-26 21:07:10,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:10,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:10,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5585
4790
4841
[WARNING|trainer.py:803] 2025-04-26 21:07:11,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:11,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:11,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4791
4842
5586
[WARNING|trainer.py:803] 2025-04-26 21:07:12,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:12,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:12,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4792
4843
5587
[WARNING|trainer.py:803] 2025-04-26 21:07:13,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:13,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:13,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4844
4793
5588
[WARNING|trainer.py:803] 2025-04-26 21:07:15,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:15,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:15,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4845
4794
5589
[WARNING|trainer.py:803] 2025-04-26 21:07:16,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:16,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4846
4795
[WARNING|trainer.py:803] 2025-04-26 21:07:16,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5590
[WARNING|trainer.py:803] 2025-04-26 21:07:17,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:17,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4847
4796
[WARNING|trainer.py:803] 2025-04-26 21:07:17,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5591
[WARNING|trainer.py:803] 2025-04-26 21:07:18,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:18,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4848
4797
[WARNING|trainer.py:803] 2025-04-26 21:07:19,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5592
[WARNING|trainer.py:803] 2025-04-26 21:07:19,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:19,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4849
4798
[WARNING|trainer.py:803] 2025-04-26 21:07:20,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5593
[WARNING|trainer.py:803] 2025-04-26 21:07:20,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:21,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4850
4799
[WARNING|trainer.py:803] 2025-04-26 21:07:21,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:22,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5594
[WARNING|trainer.py:803] 2025-04-26 21:07:22,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4851
4800
[WARNING|trainer.py:803] 2025-04-26 21:07:23,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:23,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5595
[WARNING|trainer.py:803] 2025-04-26 21:07:23,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4852
4801
[WARNING|trainer.py:803] 2025-04-26 21:07:24,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:24,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:24,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5596
4853
4802
[WARNING|trainer.py:803] 2025-04-26 21:07:25,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:25,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:25,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5597
4854
4803
[WARNING|trainer.py:803] 2025-04-26 21:07:26,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:26,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:27,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4855
5598
4804
[WARNING|trainer.py:803] 2025-04-26 21:07:28,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:28,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:28,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4856
5599
4805
[WARNING|trainer.py:803] 2025-04-26 21:07:29,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:29,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:29,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4857
5600
4806
[WARNING|trainer.py:803] 2025-04-26 21:07:30,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:30,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:30,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4858
4807
5601
[WARNING|trainer.py:803] 2025-04-26 21:07:31,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:31,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:32,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4859
4808
5602
[WARNING|trainer.py:803] 2025-04-26 21:07:32,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:33,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:33,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4860
4809
5603
[WARNING|trainer.py:803] 2025-04-26 21:07:34,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:34,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4861
[WARNING|trainer.py:803] 2025-04-26 21:07:34,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4810
5604
[WARNING|trainer.py:803] 2025-04-26 21:07:35,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:35,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4862
[WARNING|trainer.py:803] 2025-04-26 21:07:35,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4811
5605
[WARNING|trainer.py:803] 2025-04-26 21:07:36,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:36,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4863
[WARNING|trainer.py:803] 2025-04-26 21:07:37,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4812
5606
[WARNING|trainer.py:803] 2025-04-26 21:07:37,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:37,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4864
[WARNING|trainer.py:803] 2025-04-26 21:07:38,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4813
5607
[WARNING|trainer.py:803] 2025-04-26 21:07:38,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:39,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4865
[WARNING|trainer.py:803] 2025-04-26 21:07:39,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4814
5608
[WARNING|trainer.py:803] 2025-04-26 21:07:40,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:40,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4866
4815
[WARNING|trainer.py:803] 2025-04-26 21:07:40,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5609
[WARNING|trainer.py:803] 2025-04-26 21:07:41,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:41,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4867
4816
[WARNING|trainer.py:803] 2025-04-26 21:07:41,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5610
[WARNING|trainer.py:803] 2025-04-26 21:07:42,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:42,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4868
4817
[WARNING|trainer.py:803] 2025-04-26 21:07:43,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:43,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5611
[WARNING|trainer.py:803] 2025-04-26 21:07:43,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4869
4818
[WARNING|trainer.py:803] 2025-04-26 21:07:44,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:44,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5612
4870
[WARNING|trainer.py:803] 2025-04-26 21:07:45,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4819
[WARNING|trainer.py:803] 2025-04-26 21:07:45,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:07:45,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5613
4871
[WARNING|trainer.py:803] 2025-04-26 21:07:46,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4820
[WARNING|trainer.py:803] 2025-04-26 21:07:46,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:47,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5614
4872
[WARNING|trainer.py:803] 2025-04-26 21:07:47,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4821
[WARNING|trainer.py:803] 2025-04-26 21:07:48,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:48,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4873
[WARNING|trainer.py:803] 2025-04-26 21:07:48,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5615
4822
[WARNING|trainer.py:803] 2025-04-26 21:07:49,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:49,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4874
[WARNING|trainer.py:803] 2025-04-26 21:07:49,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5616
4823
[WARNING|trainer.py:803] 2025-04-26 21:07:50,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:50,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4875
[WARNING|trainer.py:803] 2025-04-26 21:07:51,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5617
4824
[WARNING|trainer.py:803] 2025-04-26 21:07:51,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:52,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4876
[WARNING|trainer.py:803] 2025-04-26 21:07:52,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5618
4825
[WARNING|trainer.py:803] 2025-04-26 21:07:53,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:53,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4877
[WARNING|trainer.py:803] 2025-04-26 21:07:53,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5619
4826
[WARNING|trainer.py:803] 2025-04-26 21:07:54,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4878
[WARNING|trainer.py:803] 2025-04-26 21:07:54,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:07:54,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4827
5620
[WARNING|trainer.py:803] 2025-04-26 21:07:55,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4879
[WARNING|trainer.py:803] 2025-04-26 21:07:55,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:55,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4828
5621
[WARNING|trainer.py:803] 2025-04-26 21:07:56,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4880
[WARNING|trainer.py:803] 2025-04-26 21:07:57,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:07:57,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4829
5622
[WARNING|trainer.py:803] 2025-04-26 21:07:57,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4881
[WARNING|trainer.py:803] 2025-04-26 21:07:58,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:07:58,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4830
[WARNING|trainer.py:803] 2025-04-26 21:07:58,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5623
4882
[WARNING|trainer.py:803] 2025-04-26 21:07:59,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4831
[WARNING|trainer.py:803] 2025-04-26 21:07:59,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:08:00,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5624
4883
[WARNING|trainer.py:803] 2025-04-26 21:08:00,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4832
[WARNING|trainer.py:803] 2025-04-26 21:08:01,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:01,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5625
4884
[WARNING|trainer.py:803] 2025-04-26 21:08:01,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4833
[WARNING|trainer.py:803] 2025-04-26 21:08:02,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:02,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4885
5626
[WARNING|trainer.py:803] 2025-04-26 21:08:02,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4834
[WARNING|trainer.py:803] 2025-04-26 21:08:03,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:03,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4886
5627
[WARNING|trainer.py:803] 2025-04-26 21:08:04,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4835
[WARNING|trainer.py:803] 2025-04-26 21:08:04,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:04,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4887
[WARNING|trainer.py:803] 2025-04-26 21:08:05,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5628
4836
[WARNING|trainer.py:803] 2025-04-26 21:08:05,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:06,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4888
[WARNING|trainer.py:803] 2025-04-26 21:08:06,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5629
4837
[WARNING|trainer.py:803] 2025-04-26 21:08:07,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:07,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4889
[WARNING|trainer.py:803] 2025-04-26 21:08:07,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5630
4838
[WARNING|trainer.py:803] 2025-04-26 21:08:08,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:08,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4890
[WARNING|trainer.py:803] 2025-04-26 21:08:08,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5631
4839
[WARNING|trainer.py:803] 2025-04-26 21:08:09,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4891
[WARNING|trainer.py:803] 2025-04-26 21:08:09,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:09,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4840
5632
[WARNING|trainer.py:803] 2025-04-26 21:08:10,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4892
[WARNING|trainer.py:803] 2025-04-26 21:08:11,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:11,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4841
5633
[WARNING|trainer.py:803] 2025-04-26 21:08:11,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4893
[WARNING|trainer.py:803] 2025-04-26 21:08:12,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:12,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4842
5634
[WARNING|trainer.py:803] 2025-04-26 21:08:12,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4894
[WARNING|trainer.py:803] 2025-04-26 21:08:13,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:13,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4843
[WARNING|trainer.py:803] 2025-04-26 21:08:14,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5635
4895
[WARNING|trainer.py:803] 2025-04-26 21:08:14,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4844
[WARNING|trainer.py:803] 2025-04-26 21:08:15,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:15,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5636
4896
[WARNING|trainer.py:803] 2025-04-26 21:08:15,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4845
[WARNING|trainer.py:803] 2025-04-26 21:08:16,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:08:16,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4897
5637
[WARNING|trainer.py:803] 2025-04-26 21:08:17,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4846
[WARNING|trainer.py:803] 2025-04-26 21:08:17,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:08:17,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4898
5638
[WARNING|trainer.py:803] 2025-04-26 21:08:18,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4847
[WARNING|trainer.py:803] 2025-04-26 21:08:18,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:18,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4899
5639
[WARNING|trainer.py:803] 2025-04-26 21:08:19,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4848
[WARNING|trainer.py:803] 2025-04-26 21:08:19,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:20,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4900
[WARNING|trainer.py:803] 2025-04-26 21:08:20,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5640
4849
[WARNING|trainer.py:803] 2025-04-26 21:08:21,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:08:21,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4901
[WARNING|trainer.py:803] 2025-04-26 21:08:21,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5641
4850
[WARNING|trainer.py:803] 2025-04-26 21:08:22,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4902
[WARNING|trainer.py:803] 2025-04-26 21:08:22,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:22,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5642
4851
[WARNING|trainer.py:803] 2025-04-26 21:08:23,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4903
[WARNING|trainer.py:803] 2025-04-26 21:08:24,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:24,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4852
5643
[WARNING|trainer.py:803] 2025-04-26 21:08:24,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4904
[WARNING|trainer.py:803] 2025-04-26 21:08:25,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:25,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4853
5644
[WARNING|trainer.py:803] 2025-04-26 21:08:25,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4905
[WARNING|trainer.py:803] 2025-04-26 21:08:26,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:26,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4854
5645
[WARNING|trainer.py:803] 2025-04-26 21:08:27,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4906
[WARNING|trainer.py:803] 2025-04-26 21:08:27,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:27,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4855
5646
[WARNING|trainer.py:803] 2025-04-26 21:08:28,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4907
[WARNING|trainer.py:803] 2025-04-26 21:08:28,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:29,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4856
[WARNING|trainer.py:803] 2025-04-26 21:08:29,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5647
4908
[WARNING|trainer.py:803] 2025-04-26 21:08:30,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:30,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4857
[WARNING|trainer.py:803] 2025-04-26 21:08:30,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5648
4909
[WARNING|trainer.py:803] 2025-04-26 21:08:31,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:31,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4858
[WARNING|trainer.py:803] 2025-04-26 21:08:31,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5649
4910
[WARNING|trainer.py:803] 2025-04-26 21:08:32,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4859
[WARNING|trainer.py:803] 2025-04-26 21:08:32,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:32,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4911
5650
[WARNING|trainer.py:803] 2025-04-26 21:08:33,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4860
[WARNING|trainer.py:803] 2025-04-26 21:08:34,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:08:34,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4912
5651
[WARNING|trainer.py:803] 2025-04-26 21:08:34,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4861
[WARNING|trainer.py:803] 2025-04-26 21:08:35,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:35,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4913
5652
[WARNING|trainer.py:803] 2025-04-26 21:08:36,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4862
[WARNING|trainer.py:803] 2025-04-26 21:08:36,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:36,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4914
5653
[WARNING|trainer.py:803] 2025-04-26 21:08:37,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4863
[WARNING|trainer.py:803] 2025-04-26 21:08:37,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:37,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4915
5654
[WARNING|trainer.py:803] 2025-04-26 21:08:38,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4864
[WARNING|trainer.py:803] 2025-04-26 21:08:38,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:39,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4916
[WARNING|trainer.py:803] 2025-04-26 21:08:39,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5655
4865
[WARNING|trainer.py:803] 2025-04-26 21:08:40,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4917
[WARNING|trainer.py:803] 2025-04-26 21:08:40,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:40,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5656
4866
[WARNING|trainer.py:803] 2025-04-26 21:08:41,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4918
[WARNING|trainer.py:803] 2025-04-26 21:08:41,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:41,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5657
4867
[WARNING|trainer.py:803] 2025-04-26 21:08:42,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4919
[WARNING|trainer.py:803] 2025-04-26 21:08:42,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:43,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5658
4868
[WARNING|trainer.py:803] 2025-04-26 21:08:43,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4920
[WARNING|trainer.py:803] 2025-04-26 21:08:44,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:44,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4869
5659
[WARNING|trainer.py:803] 2025-04-26 21:08:44,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4921
[WARNING|trainer.py:803] 2025-04-26 21:08:45,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:45,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4870
5660
[WARNING|trainer.py:803] 2025-04-26 21:08:45,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4922
[WARNING|trainer.py:803] 2025-04-26 21:08:46,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:46,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4871
[WARNING|trainer.py:803] 2025-04-26 21:08:47,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5661
4923
[WARNING|trainer.py:803] 2025-04-26 21:08:47,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:47,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4872
[WARNING|trainer.py:803] 2025-04-26 21:08:48,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5662
4924
[WARNING|trainer.py:803] 2025-04-26 21:08:48,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:49,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4873
[WARNING|trainer.py:803] 2025-04-26 21:08:49,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5663
4925
[WARNING|trainer.py:803] 2025-04-26 21:08:50,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4874
[WARNING|trainer.py:803] 2025-04-26 21:08:50,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:50,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5664
4926
[WARNING|trainer.py:803] 2025-04-26 21:08:51,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4875
[WARNING|trainer.py:803] 2025-04-26 21:08:51,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:51,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5665
4927
[WARNING|trainer.py:803] 2025-04-26 21:08:52,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4876
[WARNING|trainer.py:803] 2025-04-26 21:08:53,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:53,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5666
4928
[WARNING|trainer.py:803] 2025-04-26 21:08:53,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4877
[WARNING|trainer.py:803] 2025-04-26 21:08:54,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:08:54,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4929
5667
[WARNING|trainer.py:803] 2025-04-26 21:08:54,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4878
[WARNING|trainer.py:803] 2025-04-26 21:08:55,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:55,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4930
5668
[WARNING|trainer.py:803] 2025-04-26 21:08:55,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4879
[WARNING|trainer.py:803] 2025-04-26 21:08:56,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:56,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4931
[WARNING|trainer.py:803] 2025-04-26 21:08:57,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5669
4880
[WARNING|trainer.py:803] 2025-04-26 21:08:57,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:08:57,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4932
[WARNING|trainer.py:803] 2025-04-26 21:08:58,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5670
4881
[WARNING|trainer.py:803] 2025-04-26 21:08:58,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:08:59,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4933
[WARNING|trainer.py:803] 2025-04-26 21:08:59,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5671
4882
[WARNING|trainer.py:803] 2025-04-26 21:09:00,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4934
[WARNING|trainer.py:803] 2025-04-26 21:09:00,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:00,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5672
4883
[WARNING|trainer.py:803] 2025-04-26 21:09:01,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4935
[WARNING|trainer.py:803] 2025-04-26 21:09:01,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:01,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4884
5673
[WARNING|trainer.py:803] 2025-04-26 21:09:02,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4936
[WARNING|trainer.py:803] 2025-04-26 21:09:03,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:03,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4885
5674
[WARNING|trainer.py:803] 2025-04-26 21:09:03,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4937
[WARNING|trainer.py:803] 2025-04-26 21:09:04,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:04,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4886
[WARNING|trainer.py:803] 2025-04-26 21:09:04,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5675
4938
[WARNING|trainer.py:803] 2025-04-26 21:09:05,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:05,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4887
[WARNING|trainer.py:803] 2025-04-26 21:09:06,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5676
4939
[WARNING|trainer.py:803] 2025-04-26 21:09:06,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4888
[WARNING|trainer.py:803] 2025-04-26 21:09:06,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:07,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5677
4940
[WARNING|trainer.py:803] 2025-04-26 21:09:07,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4889
[WARNING|trainer.py:803] 2025-04-26 21:09:08,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:08,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
4941
5678
[WARNING|trainer.py:803] 2025-04-26 21:09:08,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4890
[WARNING|trainer.py:803] 2025-04-26 21:09:09,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:09,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4942
5679
[WARNING|trainer.py:803] 2025-04-26 21:09:10,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4891
[WARNING|trainer.py:803] 2025-04-26 21:09:10,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:10,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4943
[WARNING|trainer.py:803] 2025-04-26 21:09:11,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5680
4892
[WARNING|trainer.py:803] 2025-04-26 21:09:11,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:12,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4944
[WARNING|trainer.py:803] 2025-04-26 21:09:12,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5681
4893
[WARNING|trainer.py:803] 2025-04-26 21:09:13,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4945
[WARNING|trainer.py:803] 2025-04-26 21:09:13,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:13,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5682
4894
[WARNING|trainer.py:803] 2025-04-26 21:09:14,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4946
[WARNING|trainer.py:803] 2025-04-26 21:09:14,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:14,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5683
4895
[WARNING|trainer.py:803] 2025-04-26 21:09:15,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4947
[WARNING|trainer.py:803] 2025-04-26 21:09:15,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 21:09:15,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
NoYes
4896
5684
[WARNING|trainer.py:803] 2025-04-26 21:09:16,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4948
[WARNING|trainer.py:803] 2025-04-26 21:09:17,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:17,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4897
5685
[WARNING|trainer.py:803] 2025-04-26 21:09:17,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4949
[WARNING|trainer.py:803] 2025-04-26 21:09:18,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:09:18,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4898
[WARNING|trainer.py:803] 2025-04-26 21:09:18,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5686
4950
[WARNING|trainer.py:803] 2025-04-26 21:09:19,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:19,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4899
[WARNING|trainer.py:803] 2025-04-26 21:09:20,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5687
4951
[WARNING|trainer.py:803] 2025-04-26 21:09:20,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4900
[WARNING|trainer.py:803] 2025-04-26 21:09:21,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:21,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5688
4952
[WARNING|trainer.py:803] 2025-04-26 21:09:21,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4901
[WARNING|trainer.py:803] 2025-04-26 21:09:22,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:22,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5689
4953
[WARNING|trainer.py:803] 2025-04-26 21:09:22,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
4902
[WARNING|trainer.py:803] 2025-04-26 21:09:23,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:23,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4954
5690
[WARNING|trainer.py:803] 2025-04-26 21:09:24,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4903
[WARNING|trainer.py:803] 2025-04-26 21:09:24,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:24,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4955
5691
[WARNING|trainer.py:803] 2025-04-26 21:09:25,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4904
[WARNING|trainer.py:803] 2025-04-26 21:09:25,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:26,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
4956
[WARNING|trainer.py:803] 2025-04-26 21:09:26,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5692
4905
[WARNING|trainer.py:803] 2025-04-26 21:09:27,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:27,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4957
[WARNING|trainer.py:803] 2025-04-26 21:09:27,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5693
4906
[WARNING|trainer.py:803] 2025-04-26 21:09:28,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4958
[WARNING|trainer.py:803] 2025-04-26 21:09:28,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:28,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5694
4907
[WARNING|trainer.py:803] 2025-04-26 21:09:29,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4959
[WARNING|trainer.py:803] 2025-04-26 21:09:29,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:30,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5695
4908
[WARNING|trainer.py:803] 2025-04-26 21:09:30,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4960
[WARNING|trainer.py:803] 2025-04-26 21:09:31,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:31,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5696
4909
[WARNING|trainer.py:803] 2025-04-26 21:09:31,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4961
[WARNING|trainer.py:803] 2025-04-26 21:09:32,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x66148e80] moov atom not found
[21:09:32] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
[WARNING|trainer.py:803] 2025-04-26 21:09:32,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4910
5697
[WARNING|trainer.py:803] 2025-04-26 21:09:32,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4962
[WARNING|trainer.py:803] 2025-04-26 21:09:33,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:33,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4911
[WARNING|trainer.py:803] 2025-04-26 21:09:34,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5698
4963
[WARNING|trainer.py:803] 2025-04-26 21:09:34,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:09:34,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4912
[WARNING|trainer.py:803] 2025-04-26 21:09:35,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5699
4964
[WARNING|trainer.py:803] 2025-04-26 21:09:36,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:36,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4913
[WARNING|trainer.py:803] 2025-04-26 21:09:36,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5700
4965
[WARNING|trainer.py:803] 2025-04-26 21:09:37,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:37,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4914
[WARNING|trainer.py:803] 2025-04-26 21:09:37,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4966
[WARNING|trainer.py:803] 2025-04-26 21:09:38,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5701
4915
[WARNING|trainer.py:803] 2025-04-26 21:09:38,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4967
[WARNING|trainer.py:803] 2025-04-26 21:09:39,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:39,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4916
[WARNING|trainer.py:803] 2025-04-26 21:09:40,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4968
5702
[WARNING|trainer.py:803] 2025-04-26 21:09:40,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4917
[WARNING|trainer.py:803] 2025-04-26 21:09:41,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:41,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4969
[WARNING|trainer.py:803] 2025-04-26 21:09:41,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4918
[WARNING|trainer.py:803] 2025-04-26 21:09:42,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5703
4970
[WARNING|trainer.py:803] 2025-04-26 21:09:43,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:43,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:43,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4919
4971
[WARNING|trainer.py:803] 2025-04-26 21:09:44,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5704
[WARNING|trainer.py:803] 2025-04-26 21:09:44,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4920
4972
[WARNING|trainer.py:803] 2025-04-26 21:09:45,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:09:45,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:45,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4921
4973
5705
[WARNING|trainer.py:803] 2025-04-26 21:09:46,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:47,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4922
[WARNING|trainer.py:803] 2025-04-26 21:09:47,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4974
[WARNING|trainer.py:803] 2025-04-26 21:09:47,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5706
[WARNING|trainer.py:803] 2025-04-26 21:09:48,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4923
4975
[WARNING|trainer.py:803] 2025-04-26 21:09:49,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:49,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:49,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4924
4976
5707
[WARNING|trainer.py:803] 2025-04-26 21:09:50,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:50,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4925
[WARNING|trainer.py:803] 2025-04-26 21:09:50,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4977
[WARNING|trainer.py:803] 2025-04-26 21:09:51,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5708
[WARNING|trainer.py:803] 2025-04-26 21:09:51,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4926
4978
[WARNING|trainer.py:803] 2025-04-26 21:09:52,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:52,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:52,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4927
4979
5709
[WARNING|trainer.py:803] 2025-04-26 21:09:53,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:54,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4928
[WARNING|trainer.py:803] 2025-04-26 21:09:54,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4980
[WARNING|trainer.py:803] 2025-04-26 21:09:55,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:55,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5710
4929
4981
[WARNING|trainer.py:803] 2025-04-26 21:09:56,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:56,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:56,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4930
4982
5711
[WARNING|trainer.py:803] 2025-04-26 21:09:57,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:09:57,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4931
4983
[WARNING|trainer.py:803] 2025-04-26 21:09:58,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:58,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:09:58,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5712
4932
4984
[WARNING|trainer.py:803] 2025-04-26 21:09:59,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:00,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:00,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4933
4985
5713
[WARNING|trainer.py:803] 2025-04-26 21:10:01,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:01,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4934
4986
[WARNING|trainer.py:803] 2025-04-26 21:10:01,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:02,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:02,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4935
4987
5714
[WARNING|trainer.py:803] 2025-04-26 21:10:03,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:03,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:03,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4936
4988
5715
[WARNING|trainer.py:803] 2025-04-26 21:10:04,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:04,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4937
4989
[WARNING|trainer.py:803] 2025-04-26 21:10:05,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:06,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:06,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4938
4990
5716
[WARNING|trainer.py:803] 2025-04-26 21:10:07,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:07,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:07,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4939
4991
[WARNING|trainer.py:803] 2025-04-26 21:10:08,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5717
[WARNING|trainer.py:803] 2025-04-26 21:10:08,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4940
4992
[WARNING|trainer.py:803] 2025-04-26 21:10:09,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:09,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 21:10:09,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4941
4993
5718
[WARNING|trainer.py:803] 2025-04-26 21:10:10,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:10,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:11,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4942
4994
[WARNING|trainer.py:803] 2025-04-26 21:10:11,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5719
[WARNING|trainer.py:803] 2025-04-26 21:10:12,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4943
4995
[WARNING|trainer.py:803] 2025-04-26 21:10:12,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:13,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:13,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4944
4996
5720
[WARNING|trainer.py:803] 2025-04-26 21:10:14,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:14,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:14,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4945
4997
5721
[WARNING|trainer.py:803] 2025-04-26 21:10:15,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:15,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4946
4998
[WARNING|trainer.py:803] 2025-04-26 21:10:16,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:16,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:16,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4947
4999
5722
[WARNING|trainer.py:803] 2025-04-26 21:10:17,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:18,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4948
[WARNING|trainer.py:803] 2025-04-26 21:10:18,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5000
[WARNING|trainer.py:803] 2025-04-26 21:10:19,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:19,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5723
4949
5001
[WARNING|trainer.py:803] 2025-04-26 21:10:20,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:20,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:20,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4950
5002
5724
[WARNING|trainer.py:803] 2025-04-26 21:10:21,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:21,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4951
[WARNING|trainer.py:803] 2025-04-26 21:10:22,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5003
[WARNING|trainer.py:803] 2025-04-26 21:10:22,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:22,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4952
5725
5004
[WARNING|trainer.py:803] 2025-04-26 21:10:23,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:23,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:24,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4953
5005
5726
[WARNING|trainer.py:803] 2025-04-26 21:10:25,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:25,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4954
5006
[WARNING|trainer.py:803] 2025-04-26 21:10:25,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:26,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:26,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4955
5727
5007
[WARNING|trainer.py:803] 2025-04-26 21:10:27,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:27,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:27,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4956
5008
[WARNING|trainer.py:803] 2025-04-26 21:10:28,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5728
[WARNING|trainer.py:803] 2025-04-26 21:10:29,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4957
5009
[WARNING|trainer.py:803] 2025-04-26 21:10:29,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:29,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:30,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4958
5729
5010
[WARNING|trainer.py:803] 2025-04-26 21:10:31,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:31,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:31,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4959
5011
[WARNING|trainer.py:803] 2025-04-26 21:10:32,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5730
[WARNING|trainer.py:803] 2025-04-26 21:10:32,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4960
5012
[WARNING|trainer.py:803] 2025-04-26 21:10:33,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:33,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:33,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4961
5013
5731
[WARNING|trainer.py:803] 2025-04-26 21:10:34,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:34,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:35,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4962
5014
5732
[WARNING|trainer.py:803] 2025-04-26 21:10:35,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:36,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4963
5015
[WARNING|trainer.py:803] 2025-04-26 21:10:36,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:37,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:37,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4964
5733
5016
[WARNING|trainer.py:803] 2025-04-26 21:10:38,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:38,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:38,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4965
5017
5734
[WARNING|trainer.py:803] 2025-04-26 21:10:39,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:39,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4966
5018
[WARNING|trainer.py:803] 2025-04-26 21:10:40,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:40,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:40,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
4967
5735
5019
[WARNING|trainer.py:803] 2025-04-26 21:10:41,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:42,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 21:10:42,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
NoYes
4968
5020
[WARNING|trainer.py:803] 2025-04-26 21:10:43,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5736
[WARNING|trainer.py:803] 2025-04-26 21:10:43,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4969
5021
[WARNING|trainer.py:803] 2025-04-26 21:10:44,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:44,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:44,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4970
5022
5737
[WARNING|trainer.py:803] 2025-04-26 21:10:45,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:45,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4971
[WARNING|trainer.py:803] 2025-04-26 21:10:46,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5023
[WARNING|trainer.py:803] 2025-04-26 21:10:46,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5738
[WARNING|trainer.py:803] 2025-04-26 21:10:47,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4972
5024
[WARNING|trainer.py:803] 2025-04-26 21:10:47,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:47,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:48,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4973
5025
5739
[WARNING|trainer.py:803] 2025-04-26 21:10:49,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:49,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4974
[WARNING|trainer.py:803] 2025-04-26 21:10:49,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5026
[WARNING|trainer.py:803] 2025-04-26 21:10:50,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:50,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5740
4975
5027
[WARNING|trainer.py:803] 2025-04-26 21:10:51,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:51,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:51,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4976
5028
5741
[WARNING|trainer.py:803] 2025-04-26 21:10:52,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:52,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4977
[WARNING|trainer.py:803] 2025-04-26 21:10:53,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5029
[WARNING|trainer.py:803] 2025-04-26 21:10:53,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5742
[WARNING|trainer.py:803] 2025-04-26 21:10:54,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4978
5030
[WARNING|trainer.py:803] 2025-04-26 21:10:54,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:10:55,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:55,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4979
5031
5743
[WARNING|trainer.py:803] 2025-04-26 21:10:56,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:56,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4980
5032
[WARNING|trainer.py:803] 2025-04-26 21:10:56,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:10:57,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:10:57,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4981
5744
5033
[WARNING|trainer.py:803] 2025-04-26 21:10:58,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:58,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:10:58,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4982
5034
5745
[WARNING|trainer.py:803] 2025-04-26 21:10:59,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:00,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4983
5035
[WARNING|trainer.py:803] 2025-04-26 21:11:00,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:01,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:01,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4984
5036
5746
[WARNING|trainer.py:803] 2025-04-26 21:11:02,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:02,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4985
[WARNING|trainer.py:803] 2025-04-26 21:11:02,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5037
[WARNING|trainer.py:803] 2025-04-26 21:11:03,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:03,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5747
4986
5038
[WARNING|trainer.py:803] 2025-04-26 21:11:04,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:04,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:04,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4987
5039
5748
[WARNING|trainer.py:803] 2025-04-26 21:11:05,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:06,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4988
5040
[WARNING|trainer.py:803] 2025-04-26 21:11:06,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:07,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:07,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4989
5749
5041
[WARNING|trainer.py:803] 2025-04-26 21:11:08,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:08,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:08,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4990
5042
5750
[WARNING|trainer.py:803] 2025-04-26 21:11:09,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:09,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4991
5043
[WARNING|trainer.py:803] 2025-04-26 21:11:10,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:11:10,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:10,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4992
5751
5044
[WARNING|trainer.py:803] 2025-04-26 21:11:11,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:11,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:11:12,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4993
5045
[WARNING|trainer.py:803] 2025-04-26 21:11:12,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5752
[WARNING|trainer.py:803] 2025-04-26 21:11:13,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4994
5046
[WARNING|trainer.py:803] 2025-04-26 21:11:13,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:14,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:14,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4995
5753
5047
[WARNING|trainer.py:803] 2025-04-26 21:11:15,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:15,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:15,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
4996
5048
5754
[WARNING|trainer.py:803] 2025-04-26 21:11:16,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4997
[WARNING|trainer.py:803] 2025-04-26 21:11:16,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5049
[WARNING|trainer.py:803] 2025-04-26 21:11:17,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:17,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
4998
[WARNING|trainer.py:803] 2025-04-26 21:11:18,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5755
5050
[WARNING|trainer.py:803] 2025-04-26 21:11:18,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:19,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
4999
[WARNING|trainer.py:803] 2025-04-26 21:11:19,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5051
[WARNING|trainer.py:803] 2025-04-26 21:11:20,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5756
5000
[WARNING|trainer.py:803] 2025-04-26 21:11:20,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5052
[WARNING|trainer.py:803] 2025-04-26 21:11:21,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:11:21,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5001
[WARNING|trainer.py:803] 2025-04-26 21:11:21,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5757
5053
[WARNING|trainer.py:803] 2025-04-26 21:11:22,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:22,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5002
[WARNING|trainer.py:803] 2025-04-26 21:11:22,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5054
[WARNING|trainer.py:803] 2025-04-26 21:11:23,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5758
5003
[WARNING|trainer.py:803] 2025-04-26 21:11:24,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5055
[WARNING|trainer.py:803] 2025-04-26 21:11:24,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:24,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5004
[WARNING|trainer.py:803] 2025-04-26 21:11:25,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5759
5056
[WARNING|trainer.py:803] 2025-04-26 21:11:26,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5005
[WARNING|trainer.py:803] 2025-04-26 21:11:26,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:26,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5057
[WARNING|trainer.py:803] 2025-04-26 21:11:27,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5760
5006
[WARNING|trainer.py:803] 2025-04-26 21:11:27,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5058
[WARNING|trainer.py:803] 2025-04-26 21:11:28,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:28,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5007
[WARNING|trainer.py:803] 2025-04-26 21:11:28,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5761
5059
[WARNING|trainer.py:803] 2025-04-26 21:11:29,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:29,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5008
[WARNING|trainer.py:803] 2025-04-26 21:11:30,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5060
[WARNING|trainer.py:803] 2025-04-26 21:11:30,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5762
5009
[WARNING|trainer.py:803] 2025-04-26 21:11:31,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5061
[WARNING|trainer.py:803] 2025-04-26 21:11:31,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:32,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5010
[WARNING|trainer.py:803] 2025-04-26 21:11:32,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5763
5062
[WARNING|trainer.py:803] 2025-04-26 21:11:33,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:33,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5011
[WARNING|trainer.py:803] 2025-04-26 21:11:33,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5063
[WARNING|trainer.py:803] 2025-04-26 21:11:34,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5764
5012
[WARNING|trainer.py:803] 2025-04-26 21:11:34,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5064
[WARNING|trainer.py:803] 2025-04-26 21:11:35,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:35,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5013
[WARNING|trainer.py:803] 2025-04-26 21:11:35,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5765
5065
[WARNING|trainer.py:803] 2025-04-26 21:11:36,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5014
[WARNING|trainer.py:803] 2025-04-26 21:11:37,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:37,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5066
[WARNING|trainer.py:803] 2025-04-26 21:11:38,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5766
5015
[WARNING|trainer.py:803] 2025-04-26 21:11:38,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5067
[WARNING|trainer.py:803] 2025-04-26 21:11:38,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:39,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5016
[WARNING|trainer.py:803] 2025-04-26 21:11:39,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5068
5767
[WARNING|trainer.py:803] 2025-04-26 21:11:40,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5017
[WARNING|trainer.py:803] 2025-04-26 21:11:40,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:40,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5069
[WARNING|trainer.py:803] 2025-04-26 21:11:41,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5768
5018
[WARNING|trainer.py:803] 2025-04-26 21:11:41,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5070
[WARNING|trainer.py:803] 2025-04-26 21:11:42,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:42,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5019
[WARNING|trainer.py:803] 2025-04-26 21:11:43,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5769
5071
[WARNING|trainer.py:803] 2025-04-26 21:11:43,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:44,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5020
[WARNING|trainer.py:803] 2025-04-26 21:11:44,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5072
[WARNING|trainer.py:803] 2025-04-26 21:11:45,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5770
5021
[WARNING|trainer.py:803] 2025-04-26 21:11:45,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5073
[WARNING|trainer.py:803] 2025-04-26 21:11:46,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:46,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5022
[WARNING|trainer.py:803] 2025-04-26 21:11:46,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5074
5771
[WARNING|trainer.py:803] 2025-04-26 21:11:47,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5023
[WARNING|trainer.py:803] 2025-04-26 21:11:47,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:47,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5075
[WARNING|trainer.py:803] 2025-04-26 21:11:48,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5772
5024
[WARNING|trainer.py:803] 2025-04-26 21:11:49,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5076
[WARNING|trainer.py:803] 2025-04-26 21:11:49,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:49,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5025
[WARNING|trainer.py:803] 2025-04-26 21:11:50,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5077
5773
[WARNING|trainer.py:803] 2025-04-26 21:11:51,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:51,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5026
[WARNING|trainer.py:803] 2025-04-26 21:11:51,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5078
[WARNING|trainer.py:803] 2025-04-26 21:11:52,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5774
[WARNING|trainer.py:803] 2025-04-26 21:11:52,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5027
5079
[WARNING|trainer.py:803] 2025-04-26 21:11:53,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:53,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:53,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5028
5080
5775
[WARNING|trainer.py:803] 2025-04-26 21:11:54,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:11:55,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5029
5081
[WARNING|trainer.py:803] 2025-04-26 21:11:55,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:11:55,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:56,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5030
5776
5082
[WARNING|trainer.py:803] 2025-04-26 21:11:57,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:57,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5031
[WARNING|trainer.py:803] 2025-04-26 21:11:57,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5083
[WARNING|trainer.py:803] 2025-04-26 21:11:58,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5777
5032
[WARNING|trainer.py:803] 2025-04-26 21:11:58,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5084
[WARNING|trainer.py:803] 2025-04-26 21:11:59,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:11:59,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5033
[WARNING|trainer.py:803] 2025-04-26 21:11:59,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5778
5085
[WARNING|trainer.py:803] 2025-04-26 21:12:00,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:00,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5034
[WARNING|trainer.py:803] 2025-04-26 21:12:01,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5086
[WARNING|trainer.py:803] 2025-04-26 21:12:01,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5779
5035
[WARNING|trainer.py:803] 2025-04-26 21:12:02,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5087
[WARNING|trainer.py:803] 2025-04-26 21:12:02,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:03,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:03,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5036
5088
5780
[WARNING|trainer.py:803] 2025-04-26 21:12:04,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:04,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5037
5089
[WARNING|trainer.py:803] 2025-04-26 21:12:05,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:12:05,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:05,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5038
5781
5090
[WARNING|trainer.py:803] 2025-04-26 21:12:06,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:06,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:06,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5039
5091
[WARNING|trainer.py:803] 2025-04-26 21:12:07,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5782
[WARNING|trainer.py:803] 2025-04-26 21:12:08,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5040
5092
[WARNING|trainer.py:803] 2025-04-26 21:12:08,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:08,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:09,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5041
5093
5783
[WARNING|trainer.py:803] 2025-04-26 21:12:10,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:10,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5042
[WARNING|trainer.py:803] 2025-04-26 21:12:10,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5094
[WARNING|trainer.py:803] 2025-04-26 21:12:11,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:11,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5784
5043
5095
[WARNING|trainer.py:803] 2025-04-26 21:12:12,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:12,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:12,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5044
5096
5785
[WARNING|trainer.py:803] 2025-04-26 21:12:13,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:14,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5045
5097
[WARNING|trainer.py:803] 2025-04-26 21:12:14,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:14,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:15,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5046
5786
5098
[WARNING|trainer.py:803] 2025-04-26 21:12:16,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:16,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:16,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5047
5099
[WARNING|trainer.py:803] 2025-04-26 21:12:17,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:17,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5787
5048
5100
[WARNING|trainer.py:803] 2025-04-26 21:12:18,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:18,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:12:18,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5049
5101
5788
[WARNING|trainer.py:803] 2025-04-26 21:12:19,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:20,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5050
[WARNING|trainer.py:803] 2025-04-26 21:12:20,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5102
[WARNING|trainer.py:803] 2025-04-26 21:12:20,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5789
5051
[WARNING|trainer.py:803] 2025-04-26 21:12:21,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:21,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5103
[WARNING|trainer.py:803] 2025-04-26 21:12:22,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5052
5790
[WARNING|trainer.py:803] 2025-04-26 21:12:22,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5104
[WARNING|trainer.py:803] 2025-04-26 21:12:23,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:23,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5053
[WARNING|trainer.py:803] 2025-04-26 21:12:24,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:24,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5105
5791
5054
[WARNING|trainer.py:803] 2025-04-26 21:12:25,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:25,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:25,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5106
5055
5792
[WARNING|trainer.py:803] 2025-04-26 21:12:26,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:26,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:27,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5056
5107
[WARNING|trainer.py:803] 2025-04-26 21:12:28,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5793
[WARNING|trainer.py:803] 2025-04-26 21:12:28,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5057
5108
[WARNING|trainer.py:803] 2025-04-26 21:12:29,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:29,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:29,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5058
5794
5109
[WARNING|trainer.py:803] 2025-04-26 21:12:30,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5059
[WARNING|trainer.py:803] 2025-04-26 21:12:30,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:31,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5110
[WARNING|trainer.py:803] 2025-04-26 21:12:31,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5795
5060
[WARNING|trainer.py:803] 2025-04-26 21:12:32,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:32,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:32,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5111
5061
5796
[WARNING|trainer.py:803] 2025-04-26 21:12:33,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:34,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5112
5062
[WARNING|trainer.py:803] 2025-04-26 21:12:34,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:35,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:35,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5063
5113
5797
[WARNING|trainer.py:803] 2025-04-26 21:12:36,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:36,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:36,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5064
5114
5798
[WARNING|trainer.py:803] 2025-04-26 21:12:37,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:37,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5065
[WARNING|trainer.py:803] 2025-04-26 21:12:38,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5115
[WARNING|trainer.py:803] 2025-04-26 21:12:38,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:39,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5066
5799
5116
[WARNING|trainer.py:803] 2025-04-26 21:12:40,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:40,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5067
[WARNING|trainer.py:803] 2025-04-26 21:12:40,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5117
5800
[WARNING|trainer.py:803] 2025-04-26 21:12:41,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5068
[WARNING|trainer.py:803] 2025-04-26 21:12:42,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:42,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:12:42,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5118
5069
5801
[WARNING|trainer.py:803] 2025-04-26 21:12:43,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:12:43,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5119
[WARNING|trainer.py:803] 2025-04-26 21:12:44,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5070
[WARNING|trainer.py:803] 2025-04-26 21:12:44,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:44,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5802
5071
5120
[WARNING|trainer.py:803] 2025-04-26 21:12:45,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:46,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:12:46,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5072
5121
5803
[WARNING|trainer.py:803] 2025-04-26 21:12:47,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:47,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5073
[WARNING|trainer.py:803] 2025-04-26 21:12:47,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5122
[WARNING|trainer.py:803] 2025-04-26 21:12:48,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5074
[WARNING|trainer.py:803] 2025-04-26 21:12:48,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5804
5123
[WARNING|trainer.py:803] 2025-04-26 21:12:49,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:49,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5075
[WARNING|trainer.py:803] 2025-04-26 21:12:50,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5805
5124
[WARNING|trainer.py:803] 2025-04-26 21:12:50,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5076
[WARNING|trainer.py:803] 2025-04-26 21:12:51,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:51,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:52,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5125
5077
5806
[WARNING|trainer.py:803] 2025-04-26 21:12:53,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:53,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:12:53,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5126
5078
5807
[WARNING|trainer.py:803] 2025-04-26 21:12:54,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:54,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5079
5127
[WARNING|trainer.py:803] 2025-04-26 21:12:55,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:55,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:55,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5080
5808
5128
[WARNING|trainer.py:803] 2025-04-26 21:12:56,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:12:57,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:12:57,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5081
5129
5809
[WARNING|trainer.py:803] 2025-04-26 21:12:58,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5082
[WARNING|trainer.py:803] 2025-04-26 21:12:58,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:12:58,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5130
[WARNING|trainer.py:803] 2025-04-26 21:12:59,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5083
5810
[WARNING|trainer.py:803] 2025-04-26 21:12:59,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5131
[WARNING|trainer.py:803] 2025-04-26 21:13:00,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:00,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5084
[WARNING|trainer.py:803] 2025-04-26 21:13:01,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5811
[WARNING|trainer.py:803] 2025-04-26 21:13:01,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5132
5085
[WARNING|trainer.py:803] 2025-04-26 21:13:02,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:02,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:02,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5133
5086
5812
[WARNING|trainer.py:803] 2025-04-26 21:13:03,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:04,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:04,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5087
5134
5813
[WARNING|trainer.py:803] 2025-04-26 21:13:05,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:05,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5088
5135
[WARNING|trainer.py:803] 2025-04-26 21:13:06,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:06,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:06,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5089
5814
5136
[WARNING|trainer.py:803] 2025-04-26 21:13:07,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:07,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:08,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5090
5137
5815
[WARNING|trainer.py:803] 2025-04-26 21:13:08,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5091
[WARNING|trainer.py:803] 2025-04-26 21:13:09,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:09,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5138
[WARNING|trainer.py:803] 2025-04-26 21:13:10,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5092
[WARNING|trainer.py:803] 2025-04-26 21:13:10,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5816
5139
[WARNING|trainer.py:803] 2025-04-26 21:13:11,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5093
[WARNING|trainer.py:803] 2025-04-26 21:13:11,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:12,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:12,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5817
5140
5094
[WARNING|trainer.py:803] 2025-04-26 21:13:13,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:13,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:13,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5141
5095
5818
[WARNING|trainer.py:803] 2025-04-26 21:13:14,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:14,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5096
5142
[WARNING|trainer.py:803] 2025-04-26 21:13:15,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:16,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:16,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5097
5819
5143
[WARNING|trainer.py:803] 2025-04-26 21:13:17,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:17,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5098
[WARNING|trainer.py:803] 2025-04-26 21:13:17,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5144
[WARNING|trainer.py:803] 2025-04-26 21:13:18,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5820
5099
[WARNING|trainer.py:803] 2025-04-26 21:13:19,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:19,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5145
[WARNING|trainer.py:803] 2025-04-26 21:13:19,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5100
[WARNING|trainer.py:803] 2025-04-26 21:13:20,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5821
[WARNING|trainer.py:803] 2025-04-26 21:13:20,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5146
5101
[WARNING|trainer.py:803] 2025-04-26 21:13:21,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:21,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:22,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5147
5822
5102
[WARNING|trainer.py:803] 2025-04-26 21:13:23,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:23,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:23,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5148
5103
[WARNING|trainer.py:803] 2025-04-26 21:13:24,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5823
[WARNING|trainer.py:803] 2025-04-26 21:13:25,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5149
[WARNING|trainer.py:803] 2025-04-26 21:13:25,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5104
[WARNING|trainer.py:803] 2025-04-26 21:13:26,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:26,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5824
5150
5105
[WARNING|trainer.py:803] 2025-04-26 21:13:27,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:27,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:27,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5151
5825
5106
[WARNING|trainer.py:803] 2025-04-26 21:13:28,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:29,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:29,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5152
5107
[WARNING|trainer.py:803] 2025-04-26 21:13:30,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5826
[WARNING|trainer.py:803] 2025-04-26 21:13:30,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5153
[WARNING|trainer.py:803] 2025-04-26 21:13:31,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5108
[WARNING|trainer.py:803] 2025-04-26 21:13:31,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:32,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5154
5827
5109
[WARNING|trainer.py:803] 2025-04-26 21:13:32,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:33,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:33,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5155
5110
5828
[WARNING|trainer.py:803] 2025-04-26 21:13:34,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:34,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5156
[WARNING|trainer.py:803] 2025-04-26 21:13:34,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5111
[WARNING|trainer.py:803] 2025-04-26 21:13:35,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5829
[WARNING|trainer.py:803] 2025-04-26 21:13:36,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5157
[WARNING|trainer.py:803] 2025-04-26 21:13:36,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5112
[WARNING|trainer.py:803] 2025-04-26 21:13:37,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:37,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5158
5830
5113
[WARNING|trainer.py:803] 2025-04-26 21:13:38,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:38,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:38,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5159
5114
5831
[WARNING|trainer.py:803] 2025-04-26 21:13:39,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:40,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5160
[WARNING|trainer.py:803] 2025-04-26 21:13:40,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5115
[WARNING|trainer.py:803] 2025-04-26 21:13:41,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5832
5161
[WARNING|trainer.py:803] 2025-04-26 21:13:41,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5116
[WARNING|trainer.py:803] 2025-04-26 21:13:42,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:42,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5162
[WARNING|trainer.py:803] 2025-04-26 21:13:42,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5833
5117
[WARNING|trainer.py:803] 2025-04-26 21:13:43,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:44,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5163
[WARNING|trainer.py:803] 2025-04-26 21:13:44,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5118
5834
[WARNING|trainer.py:803] 2025-04-26 21:13:45,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:45,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5164
[WARNING|trainer.py:803] 2025-04-26 21:13:46,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5119
[WARNING|trainer.py:803] 2025-04-26 21:13:46,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:47,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5165
5835
5120
[WARNING|trainer.py:803] 2025-04-26 21:13:47,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:48,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:13:48,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5166
5121
5836
[WARNING|trainer.py:803] 2025-04-26 21:13:49,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:49,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:49,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5167
5122
[WARNING|trainer.py:803] 2025-04-26 21:13:50,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5837
[WARNING|trainer.py:803] 2025-04-26 21:13:51,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5168
[WARNING|trainer.py:803] 2025-04-26 21:13:51,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5123
[WARNING|trainer.py:803] 2025-04-26 21:13:52,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5838
5169
[WARNING|trainer.py:803] 2025-04-26 21:13:52,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5124
[WARNING|trainer.py:803] 2025-04-26 21:13:53,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:13:53,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5170
[WARNING|trainer.py:803] 2025-04-26 21:13:54,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5839
5125
[WARNING|trainer.py:803] 2025-04-26 21:13:54,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:55,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5171
[WARNING|trainer.py:803] 2025-04-26 21:13:55,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5126
[WARNING|trainer.py:803] 2025-04-26 21:13:56,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5840
5172
[WARNING|trainer.py:803] 2025-04-26 21:13:56,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:13:57,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5127
[WARNING|trainer.py:803] 2025-04-26 21:13:57,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5841
5173
[WARNING|trainer.py:803] 2025-04-26 21:13:58,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5128
[WARNING|trainer.py:803] 2025-04-26 21:13:58,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:13:59,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5174
[WARNING|trainer.py:803] 2025-04-26 21:13:59,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5842
5129
[WARNING|trainer.py:803] 2025-04-26 21:14:00,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:00,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:00,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5175
5130
[WARNING|trainer.py:803] 2025-04-26 21:14:01,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5843
5176
[WARNING|trainer.py:803] 2025-04-26 21:14:02,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:02,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5131
[WARNING|trainer.py:803] 2025-04-26 21:14:03,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5177
[WARNING|trainer.py:803] 2025-04-26 21:14:03,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5844
5132
[WARNING|trainer.py:803] 2025-04-26 21:14:04,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:04,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5178
[WARNING|trainer.py:803] 2025-04-26 21:14:05,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5845
5133
[WARNING|trainer.py:803] 2025-04-26 21:14:05,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5179
[WARNING|trainer.py:803] 2025-04-26 21:14:06,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:06,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5134
[WARNING|trainer.py:803] 2025-04-26 21:14:07,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5846
5180
[WARNING|trainer.py:803] 2025-04-26 21:14:07,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:08,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5135
[WARNING|trainer.py:803] 2025-04-26 21:14:08,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5847
5181
[WARNING|trainer.py:803] 2025-04-26 21:14:09,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5136
[WARNING|trainer.py:803] 2025-04-26 21:14:09,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:09,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5182
[WARNING|trainer.py:803] 2025-04-26 21:14:10,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5848
5137
[WARNING|trainer.py:803] 2025-04-26 21:14:11,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:11,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:11,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5183
5138
[WARNING|trainer.py:803] 2025-04-26 21:14:12,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5849
[WARNING|trainer.py:803] 2025-04-26 21:14:13,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5184
[WARNING|trainer.py:803] 2025-04-26 21:14:13,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5139
[WARNING|trainer.py:803] 2025-04-26 21:14:14,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:14,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5185
5850
5140
[WARNING|trainer.py:803] 2025-04-26 21:14:15,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:15,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:16,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5186
5141
5851
[WARNING|trainer.py:803] 2025-04-26 21:14:16,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:17,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5187
[WARNING|trainer.py:803] 2025-04-26 21:14:17,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5142
[WARNING|trainer.py:803] 2025-04-26 21:14:18,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5852
[WARNING|trainer.py:803] 2025-04-26 21:14:18,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5188
5143
[WARNING|trainer.py:803] 2025-04-26 21:14:19,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:19,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5189
[WARNING|trainer.py:803] 2025-04-26 21:14:20,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5853
5144
[WARNING|trainer.py:803] 2025-04-26 21:14:21,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:21,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:21,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5190
5145
5854
[WARNING|trainer.py:803] 2025-04-26 21:14:22,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:23,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5191
[WARNING|trainer.py:803] 2025-04-26 21:14:23,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5146
[WARNING|trainer.py:803] 2025-04-26 21:14:23,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5855
[WARNING|trainer.py:803] 2025-04-26 21:14:24,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5192
[WARNING|trainer.py:803] 2025-04-26 21:14:24,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5147
[WARNING|trainer.py:803] 2025-04-26 21:14:25,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:25,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5856
5193
5148
[WARNING|trainer.py:803] 2025-04-26 21:14:26,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:26,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:27,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5194
5857
5149
[WARNING|trainer.py:803] 2025-04-26 21:14:28,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:28,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:28,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5195
5150
[WARNING|trainer.py:803] 2025-04-26 21:14:29,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5858
[WARNING|trainer.py:803] 2025-04-26 21:14:29,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5196
5151
[WARNING|trainer.py:803] 2025-04-26 21:14:30,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:30,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:31,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5859
5197
5152
[WARNING|trainer.py:803] 2025-04-26 21:14:32,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:32,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:32,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5198
5153
5860
[WARNING|trainer.py:803] 2025-04-26 21:14:33,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:34,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:34,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5199
5154
[WARNING|trainer.py:803] 2025-04-26 21:14:35,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5861
[WARNING|trainer.py:803] 2025-04-26 21:14:35,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5200
[WARNING|trainer.py:803] 2025-04-26 21:14:35,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5155
[WARNING|trainer.py:803] 2025-04-26 21:14:36,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:36,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5862
5201
5156
[WARNING|trainer.py:803] 2025-04-26 21:14:37,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:37,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:38,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5202
5157
5863
[WARNING|trainer.py:803] 2025-04-26 21:14:39,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:39,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:39,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5203
5158
[WARNING|trainer.py:803] 2025-04-26 21:14:40,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5864
[WARNING|trainer.py:803] 2025-04-26 21:14:40,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5204
5159
[WARNING|trainer.py:803] 2025-04-26 21:14:41,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:14:42,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:42,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5205
5865
5160
[WARNING|trainer.py:803] 2025-04-26 21:14:43,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:43,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:43,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5206
5161
5866
[WARNING|trainer.py:803] 2025-04-26 21:14:44,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:45,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:45,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5207
5162
[WARNING|trainer.py:803] 2025-04-26 21:14:46,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5867
[WARNING|trainer.py:803] 2025-04-26 21:14:46,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5208
5163
[WARNING|trainer.py:803] 2025-04-26 21:14:47,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:47,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:48,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5209
5868
5164
[WARNING|trainer.py:803] 2025-04-26 21:14:49,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:49,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:49,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5210
5165
[WARNING|trainer.py:803] 2025-04-26 21:14:50,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5869
[WARNING|trainer.py:803] 2025-04-26 21:14:50,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5211
[WARNING|trainer.py:803] 2025-04-26 21:14:51,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5166
[WARNING|trainer.py:803] 2025-04-26 21:14:51,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:14:52,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5870
5212
5167
[WARNING|trainer.py:803] 2025-04-26 21:14:53,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:14:53,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:53,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5213
5871
5168
[WARNING|trainer.py:803] 2025-04-26 21:14:54,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:55,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:55,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5214
5169
5872
[WARNING|trainer.py:803] 2025-04-26 21:14:55,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:56,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5215
[WARNING|trainer.py:803] 2025-04-26 21:14:56,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5170
[WARNING|trainer.py:803] 2025-04-26 21:14:57,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5873
[WARNING|trainer.py:803] 2025-04-26 21:14:57,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5216
5171
[WARNING|trainer.py:803] 2025-04-26 21:14:58,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:14:58,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:14:59,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5217
5874
5172
[WARNING|trainer.py:803] 2025-04-26 21:15:00,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:00,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:00,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5218
5173
5875
[WARNING|trainer.py:803] 2025-04-26 21:15:01,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:02,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:02,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5219
5174
[WARNING|trainer.py:803] 2025-04-26 21:15:03,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5876
[WARNING|trainer.py:803] 2025-04-26 21:15:03,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5220
[WARNING|trainer.py:803] 2025-04-26 21:15:03,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5175
[WARNING|trainer.py:803] 2025-04-26 21:15:04,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:04,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5877
5221
5176
[WARNING|trainer.py:803] 2025-04-26 21:15:05,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:05,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:06,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5222
5177
5878
[WARNING|trainer.py:803] 2025-04-26 21:15:07,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:07,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:07,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5223
5178
5879
[WARNING|trainer.py:803] 2025-04-26 21:15:08,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:09,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5224
[WARNING|trainer.py:803] 2025-04-26 21:15:09,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5179
[WARNING|trainer.py:803] 2025-04-26 21:15:10,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:10,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5225
5880
5180
[WARNING|trainer.py:803] 2025-04-26 21:15:11,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:11,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:11,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5226
5181
5881
[WARNING|trainer.py:803] 2025-04-26 21:15:12,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:13,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5227
[WARNING|trainer.py:803] 2025-04-26 21:15:13,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5182
[WARNING|trainer.py:803] 2025-04-26 21:15:14,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5882
[WARNING|trainer.py:803] 2025-04-26 21:15:14,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5228
5183
[WARNING|trainer.py:803] 2025-04-26 21:15:15,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:15,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:15,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5229
5883
5184
[WARNING|trainer.py:803] 2025-04-26 21:15:17,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:17,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:17,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5230
5185
5884
[WARNING|trainer.py:803] 2025-04-26 21:15:18,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:18,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5231
[WARNING|trainer.py:803] 2025-04-26 21:15:19,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5186
[WARNING|trainer.py:803] 2025-04-26 21:15:19,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5885
[WARNING|trainer.py:803] 2025-04-26 21:15:20,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5232
5187
[WARNING|trainer.py:803] 2025-04-26 21:15:20,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:21,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:21,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5233
5886
5188
[WARNING|trainer.py:803] 2025-04-26 21:15:22,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:22,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:23,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5234
5189
[WARNING|trainer.py:803] 2025-04-26 21:15:23,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5887
[WARNING|trainer.py:803] 2025-04-26 21:15:24,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5235
[WARNING|trainer.py:803] 2025-04-26 21:15:24,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5190
[WARNING|trainer.py:803] 2025-04-26 21:15:25,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:25,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5236
5888
5191
[WARNING|trainer.py:803] 2025-04-26 21:15:26,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:26,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:27,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5237
5192
5889
[WARNING|trainer.py:803] 2025-04-26 21:15:28,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:28,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5238
[WARNING|trainer.py:803] 2025-04-26 21:15:28,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5193
[WARNING|trainer.py:803] 2025-04-26 21:15:29,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:29,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5239
5890
5194
[WARNING|trainer.py:803] 2025-04-26 21:15:30,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:30,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:31,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5240
5891
5195
[WARNING|trainer.py:803] 2025-04-26 21:15:32,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:32,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:32,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5241
5196
[WARNING|trainer.py:803] 2025-04-26 21:15:33,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5892
5242
[WARNING|trainer.py:803] 2025-04-26 21:15:34,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:34,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5197
[WARNING|trainer.py:803] 2025-04-26 21:15:34,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5243
[WARNING|trainer.py:803] 2025-04-26 21:15:35,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5893
5198
[WARNING|trainer.py:803] 2025-04-26 21:15:36,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:36,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5244
[WARNING|trainer.py:803] 2025-04-26 21:15:36,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5199
5894
[WARNING|trainer.py:803] 2025-04-26 21:15:37,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:15:38,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5245
[WARNING|trainer.py:803] 2025-04-26 21:15:38,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5200
[WARNING|trainer.py:803] 2025-04-26 21:15:39,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5895
5246
[WARNING|trainer.py:803] 2025-04-26 21:15:39,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5201
[WARNING|trainer.py:803] 2025-04-26 21:15:40,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:40,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5247
[WARNING|trainer.py:803] 2025-04-26 21:15:41,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5896
5202
[WARNING|trainer.py:803] 2025-04-26 21:15:41,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:15:42,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:42,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5248
5203
[WARNING|trainer.py:803] 2025-04-26 21:15:43,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5897
5249
[WARNING|trainer.py:803] 2025-04-26 21:15:43,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:44,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5204
[WARNING|trainer.py:803] 2025-04-26 21:15:44,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5250
[WARNING|trainer.py:803] 2025-04-26 21:15:45,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5898
5205
[WARNING|trainer.py:803] 2025-04-26 21:15:46,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:46,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5251
[WARNING|trainer.py:803] 2025-04-26 21:15:46,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5899
5206
[WARNING|trainer.py:803] 2025-04-26 21:15:47,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5252
[WARNING|trainer.py:803] 2025-04-26 21:15:48,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:48,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5207
[WARNING|trainer.py:803] 2025-04-26 21:15:48,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5900
5253
[WARNING|trainer.py:803] 2025-04-26 21:15:49,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:49,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5208
[WARNING|trainer.py:803] 2025-04-26 21:15:50,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5254
5901
[WARNING|trainer.py:803] 2025-04-26 21:15:50,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5209
[WARNING|trainer.py:803] 2025-04-26 21:15:51,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:51,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5255
[WARNING|trainer.py:803] 2025-04-26 21:15:52,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5210
5902
[WARNING|trainer.py:803] 2025-04-26 21:15:52,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5256
[WARNING|trainer.py:803] 2025-04-26 21:15:53,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:15:53,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5211
[WARNING|trainer.py:803] 2025-04-26 21:15:54,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5903
5257
[WARNING|trainer.py:803] 2025-04-26 21:15:55,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:15:55,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5212
[WARNING|trainer.py:803] 2025-04-26 21:15:55,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5258
5904
[WARNING|trainer.py:803] 2025-04-26 21:15:56,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5213
[WARNING|trainer.py:803] 2025-04-26 21:15:57,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:57,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5259
[WARNING|trainer.py:803] 2025-04-26 21:15:57,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5905
5214
[WARNING|trainer.py:803] 2025-04-26 21:15:58,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:15:59,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5260
[WARNING|trainer.py:803] 2025-04-26 21:15:59,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5215
[WARNING|trainer.py:803] 2025-04-26 21:15:59,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5906
5261
[WARNING|trainer.py:803] 2025-04-26 21:16:00,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:00,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5216
[WARNING|trainer.py:803] 2025-04-26 21:16:01,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5907
5262
[WARNING|trainer.py:803] 2025-04-26 21:16:01,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:02,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5217
[WARNING|trainer.py:803] 2025-04-26 21:16:02,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5263
[WARNING|trainer.py:803] 2025-04-26 21:16:03,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5908
5218
[WARNING|trainer.py:803] 2025-04-26 21:16:04,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:04,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5264
[WARNING|trainer.py:803] 2025-04-26 21:16:04,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5909
5219
[WARNING|trainer.py:803] 2025-04-26 21:16:05,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5265
[WARNING|trainer.py:803] 2025-04-26 21:16:06,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:06,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:06,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5220
5910
5266
[WARNING|trainer.py:803] 2025-04-26 21:16:07,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:08,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:08,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5221
5267
[WARNING|trainer.py:803] 2025-04-26 21:16:09,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5911
[WARNING|trainer.py:803] 2025-04-26 21:16:09,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5222
[WARNING|trainer.py:803] 2025-04-26 21:16:09,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5268
[WARNING|trainer.py:803] 2025-04-26 21:16:10,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5912
[WARNING|trainer.py:803] 2025-04-26 21:16:10,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5223
5269
[WARNING|trainer.py:803] 2025-04-26 21:16:11,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:11,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:12,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5913
5224
5270
[WARNING|trainer.py:803] 2025-04-26 21:16:13,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:13,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:13,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5225
5271
5914
[WARNING|trainer.py:803] 2025-04-26 21:16:14,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:14,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:15,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5226
5272
5915
[WARNING|trainer.py:803] 2025-04-26 21:16:16,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:16,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5227
[WARNING|trainer.py:803] 2025-04-26 21:16:16,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5273
[WARNING|trainer.py:803] 2025-04-26 21:16:17,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5916
[WARNING|trainer.py:803] 2025-04-26 21:16:17,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5228
5274
[WARNING|trainer.py:803] 2025-04-26 21:16:18,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:18,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:19,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5229
5917
5275
[WARNING|trainer.py:803] 2025-04-26 21:16:20,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:20,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:16:20,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5230
5276
5918
[WARNING|trainer.py:803] 2025-04-26 21:16:21,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:21,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:22,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5231
5277
5919
[WARNING|trainer.py:803] 2025-04-26 21:16:23,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:23,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5232
5278
[WARNING|trainer.py:803] 2025-04-26 21:16:23,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:24,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:24,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5920
5233
5279
[WARNING|trainer.py:803] 2025-04-26 21:16:25,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:25,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:26,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5234
5921
5280
[WARNING|trainer.py:803] 2025-04-26 21:16:27,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:27,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:27,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5235
5281
5922
[WARNING|trainer.py:803] 2025-04-26 21:16:28,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:28,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:29,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5236
5282
[WARNING|trainer.py:803] 2025-04-26 21:16:30,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:30,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5923
5237
5283
[WARNING|trainer.py:803] 2025-04-26 21:16:31,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:31,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:31,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5924
5238
5284
[WARNING|trainer.py:803] 2025-04-26 21:16:32,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:32,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:32,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5285
5239
5925
[WARNING|trainer.py:803] 2025-04-26 21:16:34,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:34,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:34,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5240
5286
5926
[WARNING|trainer.py:803] 2025-04-26 21:16:35,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:35,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5241
5287
[WARNING|trainer.py:803] 2025-04-26 21:16:36,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:37,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:37,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5927
5242
5288
[WARNING|trainer.py:803] 2025-04-26 21:16:38,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:38,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:38,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5928
5243
5289
[WARNING|trainer.py:803] 2025-04-26 21:16:39,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:39,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:40,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5244
5290
5929
[WARNING|trainer.py:803] 2025-04-26 21:16:41,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:41,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:41,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5245
5291
5930
[WARNING|trainer.py:803] 2025-04-26 21:16:42,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:42,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5246
[WARNING|trainer.py:803] 2025-04-26 21:16:43,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5292
[WARNING|trainer.py:803] 2025-04-26 21:16:44,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5931
[WARNING|trainer.py:803] 2025-04-26 21:16:44,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5247
5293
[WARNING|trainer.py:803] 2025-04-26 21:16:45,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:45,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:45,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5932
5248
5294
[WARNING|trainer.py:803] 2025-04-26 21:16:46,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:47,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:47,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5249
5295
5933
[WARNING|trainer.py:803] 2025-04-26 21:16:48,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:48,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:48,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5250
5296
5934
[WARNING|trainer.py:803] 2025-04-26 21:16:49,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:49,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:50,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5251
5297
5935
[WARNING|trainer.py:803] 2025-04-26 21:16:51,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:16:51,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5252
5298
[WARNING|trainer.py:803] 2025-04-26 21:16:51,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:52,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:52,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5936
5253
5299
[WARNING|trainer.py:803] 2025-04-26 21:16:53,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:53,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:54,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5254
5937
5300
[WARNING|trainer.py:803] 2025-04-26 21:16:55,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:55,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:55,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5255
5301
5938
[WARNING|trainer.py:803] 2025-04-26 21:16:56,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:16:56,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:57,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5256
5302
5939
[WARNING|trainer.py:803] 2025-04-26 21:16:58,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:16:58,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5257
[WARNING|trainer.py:803] 2025-04-26 21:16:58,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5303
[WARNING|trainer.py:803] 2025-04-26 21:16:59,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:16:59,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5940
5258
5304
[WARNING|trainer.py:803] 2025-04-26 21:17:00,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:01,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:01,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5259
5941
5305
[WARNING|trainer.py:803] 2025-04-26 21:17:02,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:02,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:02,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5260
5306
5942
[WARNING|trainer.py:803] 2025-04-26 21:17:03,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:03,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:04,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5261
5307
5943
[WARNING|trainer.py:803] 2025-04-26 21:17:05,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:05,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5262
5308
[WARNING|trainer.py:803] 2025-04-26 21:17:06,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:06,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:06,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5944
5263
5309
[WARNING|trainer.py:803] 2025-04-26 21:17:07,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:08,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:08,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5945
5264
5310
[WARNING|trainer.py:803] 2025-04-26 21:17:09,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:09,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:09,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5265
5311
5946
[WARNING|trainer.py:803] 2025-04-26 21:17:10,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:10,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:11,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5266
5312
5947
[WARNING|trainer.py:803] 2025-04-26 21:17:12,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:12,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5267
5313
[WARNING|trainer.py:803] 2025-04-26 21:17:12,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:13,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:13,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5948
5268
5314
[WARNING|trainer.py:803] 2025-04-26 21:17:14,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:15,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:15,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5949
5269
5315
[WARNING|trainer.py:803] 2025-04-26 21:17:16,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:16,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:16,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5316
5270
5950
[WARNING|trainer.py:803] 2025-04-26 21:17:17,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:17,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:18,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5317
5271
5951
[WARNING|trainer.py:803] 2025-04-26 21:17:19,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:19,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5318
5272
[WARNING|trainer.py:803] 2025-04-26 21:17:20,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:20,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:20,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5952
5319
5273
[WARNING|trainer.py:803] 2025-04-26 21:17:21,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:22,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:17:22,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5953
5274
5320
[WARNING|trainer.py:803] 2025-04-26 21:17:23,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:23,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:23,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5275
5321
5954
[WARNING|trainer.py:803] 2025-04-26 21:17:24,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:24,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:25,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5276
5322
5955
[WARNING|trainer.py:803] 2025-04-26 21:17:26,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:26,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5277
5323
[WARNING|trainer.py:803] 2025-04-26 21:17:26,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:27,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:27,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5956
5278
5324
[WARNING|trainer.py:803] 2025-04-26 21:17:28,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:28,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:29,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5279
5325
5957
[WARNING|trainer.py:803] 2025-04-26 21:17:30,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:30,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:30,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5326
5280
5958
[WARNING|trainer.py:803] 2025-04-26 21:17:31,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:31,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5327
[WARNING|trainer.py:803] 2025-04-26 21:17:32,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5281
[WARNING|trainer.py:803] 2025-04-26 21:17:33,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:33,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5959
5328
5282
[WARNING|trainer.py:803] 2025-04-26 21:17:34,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:34,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:34,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5960
5329
5283
[WARNING|trainer.py:803] 2025-04-26 21:17:35,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:17:35,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:35,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5330
5284
5961
[WARNING|trainer.py:803] 2025-04-26 21:17:37,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:37,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:37,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5331
5285
5962
[WARNING|trainer.py:803] 2025-04-26 21:17:38,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:38,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5332
[WARNING|trainer.py:803] 2025-04-26 21:17:39,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5286
[WARNING|trainer.py:803] 2025-04-26 21:17:39,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5963
[WARNING|trainer.py:803] 2025-04-26 21:17:40,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5333
5287
[WARNING|trainer.py:803] 2025-04-26 21:17:41,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:41,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:41,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5964
5334
5288
[WARNING|trainer.py:803] 2025-04-26 21:17:42,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:42,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:43,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5335
5965
5289
[WARNING|trainer.py:803] 2025-04-26 21:17:44,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:44,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:44,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5336
5290
5966
[WARNING|trainer.py:803] 2025-04-26 21:17:45,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:45,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5337
[WARNING|trainer.py:803] 2025-04-26 21:17:46,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5291
[WARNING|trainer.py:803] 2025-04-26 21:17:46,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5967
[WARNING|trainer.py:803] 2025-04-26 21:17:47,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5338
5292
[WARNING|trainer.py:803] 2025-04-26 21:17:47,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:48,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:48,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5968
5339
5293
[WARNING|trainer.py:803] 2025-04-26 21:17:49,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:49,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:50,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5340
5294
5969
[WARNING|trainer.py:803] 2025-04-26 21:17:51,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:51,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:17:51,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5341
5295
5970
[WARNING|trainer.py:803] 2025-04-26 21:17:52,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:52,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5342
[WARNING|trainer.py:803] 2025-04-26 21:17:53,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5296
[WARNING|trainer.py:803] 2025-04-26 21:17:53,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:54,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5971
5343
5297
[WARNING|trainer.py:803] 2025-04-26 21:17:55,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:55,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:17:55,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5344
5972
5298
[WARNING|trainer.py:803] 2025-04-26 21:17:56,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:56,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:17:57,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5345
5299
5973
[WARNING|trainer.py:803] 2025-04-26 21:17:57,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:17:58,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5346
[WARNING|trainer.py:803] 2025-04-26 21:17:58,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5300
[WARNING|trainer.py:803] 2025-04-26 21:17:59,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5974
5347
[WARNING|trainer.py:803] 2025-04-26 21:17:59,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5301
[WARNING|trainer.py:803] 2025-04-26 21:18:00,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:00,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5348
[WARNING|trainer.py:803] 2025-04-26 21:18:01,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5975
5302
[WARNING|trainer.py:803] 2025-04-26 21:18:02,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:02,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5349
[WARNING|trainer.py:803] 2025-04-26 21:18:02,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5303
5976
[WARNING|trainer.py:803] 2025-04-26 21:18:03,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5350
[WARNING|trainer.py:803] 2025-04-26 21:18:04,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:04,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5304
[WARNING|trainer.py:803] 2025-04-26 21:18:04,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5977
5351
[WARNING|trainer.py:803] 2025-04-26 21:18:05,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:06,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
5305
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:06,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5352
5978
[WARNING|trainer.py:803] 2025-04-26 21:18:06,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5306
[WARNING|trainer.py:803] 2025-04-26 21:18:07,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:07,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5353
[WARNING|trainer.py:803] 2025-04-26 21:18:08,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5979
5307
[WARNING|trainer.py:803] 2025-04-26 21:18:08,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:09,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5354
[WARNING|trainer.py:803] 2025-04-26 21:18:09,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5308
[WARNING|trainer.py:803] 2025-04-26 21:18:10,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5980
5355
[WARNING|trainer.py:803] 2025-04-26 21:18:11,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:11,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5309
[WARNING|trainer.py:803] 2025-04-26 21:18:11,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5356
5981
[WARNING|trainer.py:803] 2025-04-26 21:18:12,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5310
[WARNING|trainer.py:803] 2025-04-26 21:18:13,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 21:18:13,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5357
[WARNING|trainer.py:803] 2025-04-26 21:18:13,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5982
5311
[WARNING|trainer.py:803] 2025-04-26 21:18:14,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5358
[WARNING|trainer.py:803] 2025-04-26 21:18:15,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:18:15,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5312
[WARNING|trainer.py:803] 2025-04-26 21:18:15,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5983
5359
[WARNING|trainer.py:803] 2025-04-26 21:18:16,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:16,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5313
[WARNING|trainer.py:803] 2025-04-26 21:18:17,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5360
[WARNING|trainer.py:803] 2025-04-26 21:18:17,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5984
[WARNING|trainer.py:803] 2025-04-26 21:18:18,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5314
[WARNING|trainer.py:803] 2025-04-26 21:18:18,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5361
[WARNING|trainer.py:803] 2025-04-26 21:18:19,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5985
[WARNING|trainer.py:803] 2025-04-26 21:18:19,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5315
5362
[WARNING|trainer.py:803] 2025-04-26 21:18:20,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:18:20,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:21,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5316
5363
5986
[WARNING|trainer.py:803] 2025-04-26 21:18:22,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:22,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5317
[WARNING|trainer.py:803] 2025-04-26 21:18:22,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5364
[WARNING|trainer.py:803] 2025-04-26 21:18:23,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5987
[WARNING|trainer.py:803] 2025-04-26 21:18:23,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5318
5365
[WARNING|trainer.py:803] 2025-04-26 21:18:24,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:24,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:25,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5319
5988
5366
[WARNING|trainer.py:803] 2025-04-26 21:18:26,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:26,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:18:26,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5320
5367
5989
[WARNING|trainer.py:803] 2025-04-26 21:18:27,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:18:28,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:28,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5321
5368
[WARNING|trainer.py:803] 2025-04-26 21:18:29,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:29,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5990
5322
5369
[WARNING|trainer.py:803] 2025-04-26 21:18:30,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:30,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:30,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5323
5370
5991
[WARNING|trainer.py:803] 2025-04-26 21:18:31,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:32,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:32,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5324
5371
5992
[WARNING|trainer.py:803] 2025-04-26 21:18:33,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:33,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5325
5372
[WARNING|trainer.py:803] 2025-04-26 21:18:34,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:34,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:34,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5993
5326
5373
[WARNING|trainer.py:803] 2025-04-26 21:18:35,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:36,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:36,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5994
5327
5374
[WARNING|trainer.py:803] 2025-04-26 21:18:37,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:37,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:37,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5328
5375
5995
[WARNING|trainer.py:803] 2025-04-26 21:18:38,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:39,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:39,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5329
5376
5996
[WARNING|trainer.py:803] 2025-04-26 21:18:40,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:40,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5330
5377
[WARNING|trainer.py:803] 2025-04-26 21:18:41,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:41,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:41,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5997
5331
5378
[WARNING|trainer.py:803] 2025-04-26 21:18:42,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:43,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:43,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5332
5379
5998
[WARNING|trainer.py:803] 2025-04-26 21:18:44,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:44,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:44,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5333
5380
5999
[WARNING|trainer.py:803] 2025-04-26 21:18:45,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:45,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5334
5381
[WARNING|trainer.py:803] 2025-04-26 21:18:46,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:47,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:47,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6000
5335
5382
[WARNING|trainer.py:803] 2025-04-26 21:18:48,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:48,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:48,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5336
6001
5383
[WARNING|trainer.py:803] 2025-04-26 21:18:49,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:50,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:50,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5337
5384
6002
[WARNING|trainer.py:803] 2025-04-26 21:18:51,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:51,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:18:51,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5338
5385
[WARNING|trainer.py:803] 2025-04-26 21:18:52,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:52,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6003
5339
5386
[WARNING|trainer.py:803] 2025-04-26 21:18:53,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:54,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:54,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5340
6004
5387
[WARNING|trainer.py:803] 2025-04-26 21:18:55,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:18:55,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:18:55,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5341
5388
6005
[WARNING|trainer.py:803] 2025-04-26 21:18:56,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:57,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:57,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5342
5389
[WARNING|trainer.py:803] 2025-04-26 21:18:58,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6006
[WARNING|trainer.py:803] 2025-04-26 21:18:58,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5343
5390
[WARNING|trainer.py:803] 2025-04-26 21:18:59,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:18:59,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:18:59,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6007
5344
5391
[WARNING|trainer.py:803] 2025-04-26 21:19:00,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:00,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:01,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5345
6008
5392
[WARNING|trainer.py:803] 2025-04-26 21:19:02,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:02,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:02,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5346
5393
6009
[WARNING|trainer.py:803] 2025-04-26 21:19:03,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:03,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5347
[WARNING|trainer.py:803] 2025-04-26 21:19:04,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5394
[WARNING|trainer.py:803] 2025-04-26 21:19:05,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6010
[WARNING|trainer.py:803] 2025-04-26 21:19:05,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5348
5395
[WARNING|trainer.py:803] 2025-04-26 21:19:06,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:06,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:06,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6011
5349
5396
[WARNING|trainer.py:803] 2025-04-26 21:19:07,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:07,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:08,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5350
5397
6012
[WARNING|trainer.py:803] 2025-04-26 21:19:09,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:09,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:09,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5351
5398
6013
[WARNING|trainer.py:803] 2025-04-26 21:19:10,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:10,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5352
[WARNING|trainer.py:803] 2025-04-26 21:19:11,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5399
[WARNING|trainer.py:803] 2025-04-26 21:19:11,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6014
[WARNING|trainer.py:803] 2025-04-26 21:19:12,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5353
5400
[WARNING|trainer.py:803] 2025-04-26 21:19:13,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:13,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:13,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5354
6015
5401
[WARNING|trainer.py:803] 2025-04-26 21:19:14,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:14,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:15,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5355
6016
5402
[WARNING|trainer.py:803] 2025-04-26 21:19:16,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:16,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5356
[WARNING|trainer.py:803] 2025-04-26 21:19:16,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6017
5403
[WARNING|trainer.py:803] 2025-04-26 21:19:17,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
5357
[WARNING|trainer.py:803] 2025-04-26 21:19:18,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:18,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:18,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5404
6018
5358
[WARNING|trainer.py:803] 2025-04-26 21:19:19,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:19,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:20,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5405
5359
6019
[WARNING|trainer.py:803] 2025-04-26 21:19:21,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:21,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:21,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5406
5360
6020
[WARNING|trainer.py:803] 2025-04-26 21:19:22,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:22,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:23,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5407
5361
6021
[WARNING|trainer.py:803] 2025-04-26 21:19:24,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:24,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5362
5408
[WARNING|trainer.py:803] 2025-04-26 21:19:24,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:25,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:25,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6022
5363
5409
[WARNING|trainer.py:803] 2025-04-26 21:19:26,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:26,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:27,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5364
5410
6023
[WARNING|trainer.py:803] 2025-04-26 21:19:28,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:28,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:28,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5365
5411
[WARNING|trainer.py:803] 2025-04-26 21:19:29,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6024
[WARNING|trainer.py:803] 2025-04-26 21:19:30,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5366
5412
[WARNING|trainer.py:803] 2025-04-26 21:19:30,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:31,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:31,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6025
5367
5413
[WARNING|trainer.py:803] 2025-04-26 21:19:32,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:32,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5368
[WARNING|trainer.py:803] 2025-04-26 21:19:33,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6026
5414
[WARNING|trainer.py:803] 2025-04-26 21:19:33,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:34,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5369
[WARNING|trainer.py:803] 2025-04-26 21:19:34,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6027
[WARNING|trainer.py:803] 2025-04-26 21:19:35,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5415
5370
[WARNING|trainer.py:803] 2025-04-26 21:19:35,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:36,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:36,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6028
5416
5371
[WARNING|trainer.py:803] 2025-04-26 21:19:37,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:37,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:38,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5417
5372
6029
[WARNING|trainer.py:803] 2025-04-26 21:19:39,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:39,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:39,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5418
5373
[WARNING|trainer.py:803] 2025-04-26 21:19:40,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6030
[WARNING|trainer.py:803] 2025-04-26 21:19:40,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5419
5374
[WARNING|trainer.py:803] 2025-04-26 21:19:41,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:42,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:42,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6031
5420
5375
[WARNING|trainer.py:803] 2025-04-26 21:19:43,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:43,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:43,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6032
5421
5376
[WARNING|trainer.py:803] 2025-04-26 21:19:45,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:45,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:45,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5377
5422
6033
[WARNING|trainer.py:803] 2025-04-26 21:19:46,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:46,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:46,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5378
5423
[WARNING|trainer.py:803] 2025-04-26 21:19:47,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6034
[WARNING|trainer.py:803] 2025-04-26 21:19:48,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5379
5424
[WARNING|trainer.py:803] 2025-04-26 21:19:48,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:49,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:49,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6035
5380
5425
[WARNING|trainer.py:803] 2025-04-26 21:19:50,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:50,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:51,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5381
6036
5426
[WARNING|trainer.py:803] 2025-04-26 21:19:51,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:52,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5382
[WARNING|trainer.py:803] 2025-04-26 21:19:52,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6037
5427
[WARNING|trainer.py:803] 2025-04-26 21:19:53,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5383
[WARNING|trainer.py:803] 2025-04-26 21:19:53,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:54,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:54,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5428
6038
5384
[WARNING|trainer.py:803] 2025-04-26 21:19:55,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:19:55,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:19:56,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5429
5385
6039
[WARNING|trainer.py:803] 2025-04-26 21:19:57,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:57,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:57,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5430
5386
6040
[WARNING|trainer.py:803] 2025-04-26 21:19:58,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:19:58,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:19:59,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5431
5387
[WARNING|trainer.py:803] 2025-04-26 21:20:00,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6041
[WARNING|trainer.py:803] 2025-04-26 21:20:00,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5388
5432
[WARNING|trainer.py:803] 2025-04-26 21:20:00,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:01,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:01,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6042
5389
5433
[WARNING|trainer.py:803] 2025-04-26 21:20:02,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:02,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:03,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5390
5434
6043
[WARNING|trainer.py:803] 2025-04-26 21:20:04,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:04,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:04,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5391
5435
6044
[WARNING|trainer.py:803] 2025-04-26 21:20:05,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:06,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5392
[WARNING|trainer.py:803] 2025-04-26 21:20:06,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5436
[WARNING|trainer.py:803] 2025-04-26 21:20:07,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6045
[WARNING|trainer.py:803] 2025-04-26 21:20:07,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5393
[WARNING|trainer.py:803] 2025-04-26 21:20:08,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5437
[WARNING|trainer.py:803] 2025-04-26 21:20:08,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6046
5394
[WARNING|trainer.py:803] 2025-04-26 21:20:09,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5438
[WARNING|trainer.py:803] 2025-04-26 21:20:09,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:09,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5395
[WARNING|trainer.py:803] 2025-04-26 21:20:10,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6047
5439
[WARNING|trainer.py:803] 2025-04-26 21:20:11,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5396
[WARNING|trainer.py:803] 2025-04-26 21:20:11,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:12,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:12,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5440
6048
5397
[WARNING|trainer.py:803] 2025-04-26 21:20:13,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:13,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:14,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5441
6049
5398
[WARNING|trainer.py:803] 2025-04-26 21:20:15,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:15,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:15,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5442
5399
6050
[WARNING|trainer.py:803] 2025-04-26 21:20:16,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:16,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:17,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5443
5400
[WARNING|trainer.py:803] 2025-04-26 21:20:18,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:18,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6051
5444
5401
[WARNING|trainer.py:803] 2025-04-26 21:20:19,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:19,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:19,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6052
5445
5402
[WARNING|trainer.py:803] 2025-04-26 21:20:21,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:21,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:21,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5446
6053
5403
[WARNING|trainer.py:803] 2025-04-26 21:20:22,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:22,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:22,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5447
5404
6054
[WARNING|trainer.py:803] 2025-04-26 21:20:24,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:24,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:24,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5448
5405
6055
[WARNING|trainer.py:803] 2025-04-26 21:20:25,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:25,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:26,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5449
5406
6056
[WARNING|trainer.py:803] 2025-04-26 21:20:27,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:27,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:27,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5450
5407
6057
[WARNING|trainer.py:803] 2025-04-26 21:20:28,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:28,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:29,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5408
5451
6058
[WARNING|trainer.py:803] 2025-04-26 21:20:30,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:30,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5409
[WARNING|trainer.py:803] 2025-04-26 21:20:30,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5452
[WARNING|trainer.py:803] 2025-04-26 21:20:31,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:31,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6059
5410
5453
[WARNING|trainer.py:803] 2025-04-26 21:20:32,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:33,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:33,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6060
5411
5454
[WARNING|trainer.py:803] 2025-04-26 21:20:34,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:34,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:34,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6061
5412
5455
[WARNING|trainer.py:803] 2025-04-26 21:20:35,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:36,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:36,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6062
5413
5456
[WARNING|trainer.py:803] 2025-04-26 21:20:37,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:37,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:37,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6063
5414
5457
[WARNING|trainer.py:803] 2025-04-26 21:20:39,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:39,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:39,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5415
6064
5458
[WARNING|trainer.py:803] 2025-04-26 21:20:40,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:40,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:40,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5416
5459
6065
[WARNING|trainer.py:803] 2025-04-26 21:20:42,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:42,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5417
[WARNING|trainer.py:803] 2025-04-26 21:20:42,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5460
6066
[WARNING|trainer.py:803] 2025-04-26 21:20:43,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:43,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5461
5418
[WARNING|trainer.py:803] 2025-04-26 21:20:44,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:45,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6067
[WARNING|trainer.py:803] 2025-04-26 21:20:45,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5462
5419
[WARNING|trainer.py:803] 2025-04-26 21:20:46,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:46,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:46,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6068
5463
5420
[WARNING|trainer.py:803] 2025-04-26 21:20:47,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:48,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:48,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6069
5464
5421
[WARNING|trainer.py:803] 2025-04-26 21:20:49,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:49,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:49,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5422
5465
6070
[WARNING|trainer.py:803] 2025-04-26 21:20:51,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:51,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:20:51,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5423
5466
6071
[WARNING|trainer.py:803] 2025-04-26 21:20:52,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:20:52,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:53,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5467
5424
[WARNING|trainer.py:803] 2025-04-26 21:20:54,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:54,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6072
5425
5468
[WARNING|trainer.py:803] 2025-04-26 21:20:55,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:55,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:55,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6073
5469
5426
[WARNING|trainer.py:803] 2025-04-26 21:20:56,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:57,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6074
[WARNING|trainer.py:803] 2025-04-26 21:20:57,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5470
5427
[WARNING|trainer.py:803] 2025-04-26 21:20:58,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:20:58,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:20:58,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5471
6075
5428
[WARNING|trainer.py:803] 2025-04-26 21:21:00,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:00,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:00,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5472
5429
6076
[WARNING|trainer.py:803] 2025-04-26 21:21:01,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:02,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:02,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5473
5430
6077
[WARNING|trainer.py:803] 2025-04-26 21:21:03,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:03,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5474
[WARNING|trainer.py:803] 2025-04-26 21:21:03,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5431
[WARNING|trainer.py:803] 2025-04-26 21:21:04,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6078
[WARNING|trainer.py:803] 2025-04-26 21:21:05,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5475
[WARNING|trainer.py:803] 2025-04-26 21:21:05,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5432
[WARNING|trainer.py:803] 2025-04-26 21:21:06,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6079
[WARNING|trainer.py:803] 2025-04-26 21:21:06,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5476
[WARNING|trainer.py:803] 2025-04-26 21:21:07,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5433
[WARNING|trainer.py:803] 2025-04-26 21:21:07,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:08,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6080
5477
5434
[WARNING|trainer.py:803] 2025-04-26 21:21:09,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:09,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5478
[WARNING|trainer.py:803] 2025-04-26 21:21:09,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6081
5435
[WARNING|trainer.py:803] 2025-04-26 21:21:10,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:10,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5479
[WARNING|trainer.py:803] 2025-04-26 21:21:11,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6082
5436
[WARNING|trainer.py:803] 2025-04-26 21:21:12,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:12,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5480
[WARNING|trainer.py:803] 2025-04-26 21:21:12,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5437
[WARNING|trainer.py:803] 2025-04-26 21:21:13,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6083
5481
[WARNING|trainer.py:803] 2025-04-26 21:21:14,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:14,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5438
[WARNING|trainer.py:803] 2025-04-26 21:21:15,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5482
6084
[WARNING|trainer.py:803] 2025-04-26 21:21:15,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:16,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5439
[WARNING|trainer.py:803] 2025-04-26 21:21:16,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5483
[WARNING|trainer.py:803] 2025-04-26 21:21:17,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6085
5440
[WARNING|trainer.py:803] 2025-04-26 21:21:18,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:18,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5484
[WARNING|trainer.py:803] 2025-04-26 21:21:18,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6086
5441
[WARNING|trainer.py:803] 2025-04-26 21:21:19,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5485
[WARNING|trainer.py:803] 2025-04-26 21:21:20,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:20,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5442
[WARNING|trainer.py:803] 2025-04-26 21:21:21,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6087
[WARNING|trainer.py:803] 2025-04-26 21:21:21,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5486
[WARNING|trainer.py:803] 2025-04-26 21:21:22,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5443
[WARNING|trainer.py:803] 2025-04-26 21:21:22,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6088
5487
[WARNING|trainer.py:803] 2025-04-26 21:21:23,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:23,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
5444
[WARNING|trainer.py:803] 2025-04-26 21:21:24,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6089
5488
[WARNING|trainer.py:803] 2025-04-26 21:21:24,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:21:25,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5445
[WARNING|trainer.py:803] 2025-04-26 21:21:25,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6090
5489
[WARNING|trainer.py:803] 2025-04-26 21:21:26,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:21:26,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:27,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5446
6091
5490
[WARNING|trainer.py:803] 2025-04-26 21:21:27,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:28,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:21:28,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5447
5491
[WARNING|trainer.py:803] 2025-04-26 21:21:29,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6092
[WARNING|trainer.py:803] 2025-04-26 21:21:30,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5448
[WARNING|trainer.py:803] 2025-04-26 21:21:30,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5492
[WARNING|trainer.py:803] 2025-04-26 21:21:30,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6093
[WARNING|trainer.py:803] 2025-04-26 21:21:31,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5449
[WARNING|trainer.py:803] 2025-04-26 21:21:32,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5493
[WARNING|trainer.py:803] 2025-04-26 21:21:32,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6094
[WARNING|trainer.py:803] 2025-04-26 21:21:33,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5450
[WARNING|trainer.py:803] 2025-04-26 21:21:33,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5494
[WARNING|trainer.py:803] 2025-04-26 21:21:34,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:34,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6095
5451
5495
[WARNING|trainer.py:803] 2025-04-26 21:21:35,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:35,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:36,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5452
6096
5496
[WARNING|trainer.py:803] 2025-04-26 21:21:37,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:21:37,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:37,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5453
5497
6097
[WARNING|trainer.py:803] 2025-04-26 21:21:38,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:39,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:39,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5454
5498
[WARNING|trainer.py:803] 2025-04-26 21:21:40,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6098
[WARNING|trainer.py:803] 2025-04-26 21:21:40,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5455
5499
[WARNING|trainer.py:803] 2025-04-26 21:21:41,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:41,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:42,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6099
5456
5500
[WARNING|trainer.py:803] 2025-04-26 21:21:42,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:43,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:43,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6100
5457
5501
[WARNING|trainer.py:803] 2025-04-26 21:21:44,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:44,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:45,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5458
5502
6101
[WARNING|trainer.py:803] 2025-04-26 21:21:46,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:46,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:46,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5459
5503
6102
[WARNING|trainer.py:803] 2025-04-26 21:21:47,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:47,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:48,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5460
5504
6103
[WARNING|trainer.py:803] 2025-04-26 21:21:49,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:49,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5461
5505
[WARNING|trainer.py:803] 2025-04-26 21:21:50,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:50,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:50,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6104
5506
5462
[WARNING|trainer.py:803] 2025-04-26 21:21:51,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:52,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:52,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6105
5463
5507
[WARNING|trainer.py:803] 2025-04-26 21:21:53,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:53,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:21:53,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5508
5464
6106
[WARNING|trainer.py:803] 2025-04-26 21:21:55,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:55,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:55,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5509
5465
6107
[WARNING|trainer.py:803] 2025-04-26 21:21:56,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:56,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5510
[WARNING|trainer.py:803] 2025-04-26 21:21:57,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5466
[WARNING|trainer.py:803] 2025-04-26 21:21:58,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:21:58,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6108
5511
5467
[WARNING|trainer.py:803] 2025-04-26 21:21:59,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:21:59,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6109
[WARNING|trainer.py:803] 2025-04-26 21:21:59,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5512
5468
[WARNING|trainer.py:803] 2025-04-26 21:22:00,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:00,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:01,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5513
6110
5469
[WARNING|trainer.py:803] 2025-04-26 21:22:02,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:02,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:02,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5514
6111
5470
[WARNING|trainer.py:803] 2025-04-26 21:22:04,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:04,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:04,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5515
6112
5471
[WARNING|trainer.py:803] 2025-04-26 21:22:05,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:05,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:05,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5516
5472
6113
[WARNING|trainer.py:803] 2025-04-26 21:22:06,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:07,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5517
[WARNING|trainer.py:803] 2025-04-26 21:22:07,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5473
[WARNING|trainer.py:803] 2025-04-26 21:22:08,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6114
[WARNING|trainer.py:803] 2025-04-26 21:22:08,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5518
5474
[WARNING|trainer.py:803] 2025-04-26 21:22:09,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:09,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:10,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5519
6115
5475
[WARNING|trainer.py:803] 2025-04-26 21:22:11,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:22:11,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:11,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5520
6116
5476
[WARNING|trainer.py:803] 2025-04-26 21:22:12,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:13,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5521
[WARNING|trainer.py:803] 2025-04-26 21:22:13,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6117
5477
[WARNING|trainer.py:803] 2025-04-26 21:22:14,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5522
[WARNING|trainer.py:803] 2025-04-26 21:22:14,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:14,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5478
[WARNING|trainer.py:803] 2025-04-26 21:22:15,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6118
5523
[WARNING|trainer.py:803] 2025-04-26 21:22:16,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:17,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5479
[WARNING|trainer.py:803] 2025-04-26 21:22:17,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5524
[WARNING|trainer.py:803] 2025-04-26 21:22:17,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6119
[WARNING|trainer.py:803] 2025-04-26 21:22:18,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5480
[WARNING|trainer.py:803] 2025-04-26 21:22:18,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5525
[WARNING|trainer.py:803] 2025-04-26 21:22:19,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:20,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6120
5481
5526
[WARNING|trainer.py:803] 2025-04-26 21:22:20,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:20,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:21,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5482
6121
5527
[WARNING|trainer.py:803] 2025-04-26 21:22:22,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:22,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:22:22,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5483
5528
[WARNING|trainer.py:803] 2025-04-26 21:22:24,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6122
[WARNING|trainer.py:803] 2025-04-26 21:22:24,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5484
[WARNING|trainer.py:803] 2025-04-26 21:22:24,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5529
[WARNING|trainer.py:803] 2025-04-26 21:22:25,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6123
[WARNING|trainer.py:803] 2025-04-26 21:22:25,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5485
5530
[WARNING|trainer.py:803] 2025-04-26 21:22:26,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:22:27,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:27,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6124
5486
5531
[WARNING|trainer.py:803] 2025-04-26 21:22:28,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:28,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:22:28,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5532
5487
6125
[WARNING|trainer.py:803] 2025-04-26 21:22:30,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:30,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:30,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5533
5488
6126
[WARNING|trainer.py:803] 2025-04-26 21:22:31,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:31,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:32,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5534
5489
[WARNING|trainer.py:803] 2025-04-26 21:22:33,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:33,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6127
5535
5490
[WARNING|trainer.py:803] 2025-04-26 21:22:34,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:34,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:34,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6128
5536
5491
[WARNING|trainer.py:803] 2025-04-26 21:22:35,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:22:36,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:36,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5537
6129
5492
[WARNING|trainer.py:803] 2025-04-26 21:22:37,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:37,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:37,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5493
5538
6130
[WARNING|trainer.py:803] 2025-04-26 21:22:39,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:39,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:39,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5494
6131
5539
[WARNING|trainer.py:803] 2025-04-26 21:22:40,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:40,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:41,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5495
6132
5540
[WARNING|trainer.py:803] 2025-04-26 21:22:42,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:42,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5496
[WARNING|trainer.py:803] 2025-04-26 21:22:42,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:43,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6133
5541
5497
[WARNING|trainer.py:803] 2025-04-26 21:22:44,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:44,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:45,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6134
5542
5498
[WARNING|trainer.py:803] 2025-04-26 21:22:46,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:22:46,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:46,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5499
5543
6135
[WARNING|trainer.py:803] 2025-04-26 21:22:48,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:48,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:48,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5500
5544
6136
[WARNING|trainer.py:803] 2025-04-26 21:22:49,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:49,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:50,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5501
5545
[WARNING|trainer.py:803] 2025-04-26 21:22:51,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6137
[WARNING|trainer.py:803] 2025-04-26 21:22:51,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5502
[WARNING|trainer.py:803] 2025-04-26 21:22:52,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5546
[WARNING|trainer.py:803] 2025-04-26 21:22:52,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6138
5503
[WARNING|trainer.py:803] 2025-04-26 21:22:53,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:53,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:54,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5547
5504
6139
[WARNING|trainer.py:803] 2025-04-26 21:22:55,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:55,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:22:55,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5505
5548
6140
[WARNING|trainer.py:803] 2025-04-26 21:22:57,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:57,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5506
[WARNING|trainer.py:803] 2025-04-26 21:22:57,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5549
[WARNING|trainer.py:803] 2025-04-26 21:22:58,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:58,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6141
5507
5550
[WARNING|trainer.py:803] 2025-04-26 21:22:59,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:22:59,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:00,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6142
5508
5551
[WARNING|trainer.py:803] 2025-04-26 21:23:01,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:01,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:01,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5509
5552
6143
[WARNING|trainer.py:803] 2025-04-26 21:23:02,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:03,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:03,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5510
5553
6144
[WARNING|trainer.py:803] 2025-04-26 21:23:04,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:04,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5511
[WARNING|trainer.py:803] 2025-04-26 21:23:04,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5554
[WARNING|trainer.py:803] 2025-04-26 21:23:05,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6145
[WARNING|trainer.py:803] 2025-04-26 21:23:06,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5512
[WARNING|trainer.py:803] 2025-04-26 21:23:06,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5555
[WARNING|trainer.py:803] 2025-04-26 21:23:07,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6146
5513
[WARNING|trainer.py:803] 2025-04-26 21:23:07,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:08,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5556
[WARNING|trainer.py:803] 2025-04-26 21:23:08,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5514
[WARNING|trainer.py:803] 2025-04-26 21:23:09,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6147
5557
[WARNING|trainer.py:803] 2025-04-26 21:23:09,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:10,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5515
[WARNING|trainer.py:803] 2025-04-26 21:23:10,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:11,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6148
5558
5516
[WARNING|trainer.py:803] 2025-04-26 21:23:12,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:12,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:12,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5559
6149
5517
[WARNING|trainer.py:803] 2025-04-26 21:23:13,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:14,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:14,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5560
5518
6150
[WARNING|trainer.py:803] 2025-04-26 21:23:15,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:15,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:15,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5561
5519
6151
[WARNING|trainer.py:803] 2025-04-26 21:23:17,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:17,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5562
[WARNING|trainer.py:803] 2025-04-26 21:23:17,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5520
[WARNING|trainer.py:803] 2025-04-26 21:23:18,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:18,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6152
5563
5521
[WARNING|trainer.py:803] 2025-04-26 21:23:19,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:20,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:20,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6153
5564
5522
[WARNING|trainer.py:803] 2025-04-26 21:23:21,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:23:21,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:21,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5565
5523
6154
[WARNING|trainer.py:803] 2025-04-26 21:23:23,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:23,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:23:23,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5566
5524
6155
[WARNING|trainer.py:803] 2025-04-26 21:23:24,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:24,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:25,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5525
5567
6156
[WARNING|trainer.py:803] 2025-04-26 21:23:26,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:26,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5526
[WARNING|trainer.py:803] 2025-04-26 21:23:26,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5568
[WARNING|trainer.py:803] 2025-04-26 21:23:27,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6157
[WARNING|trainer.py:803] 2025-04-26 21:23:27,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5527
5569
[WARNING|trainer.py:803] 2025-04-26 21:23:28,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:29,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:29,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5528
5570
6158
[WARNING|trainer.py:803] 2025-04-26 21:23:30,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:30,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:30,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5529
5571
6159
[WARNING|trainer.py:803] 2025-04-26 21:23:32,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:32,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:23:32,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5530
5572
[WARNING|trainer.py:803] 2025-04-26 21:23:33,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6160
[WARNING|trainer.py:803] 2025-04-26 21:23:33,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5531
5573
[WARNING|trainer.py:803] 2025-04-26 21:23:34,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:34,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:35,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6161
5532
5574
[WARNING|trainer.py:803] 2025-04-26 21:23:36,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:36,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:36,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5533
6162
5575
[WARNING|trainer.py:803] 2025-04-26 21:23:37,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:37,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:38,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5534
5576
6163
[WARNING|trainer.py:803] 2025-04-26 21:23:39,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:39,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:39,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5535
5577
[WARNING|trainer.py:803] 2025-04-26 21:23:40,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6164
[WARNING|trainer.py:803] 2025-04-26 21:23:41,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5536
[WARNING|trainer.py:803] 2025-04-26 21:23:41,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5578
[WARNING|trainer.py:803] 2025-04-26 21:23:42,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6165
[WARNING|trainer.py:803] 2025-04-26 21:23:42,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5537
[WARNING|trainer.py:803] 2025-04-26 21:23:43,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5579
[WARNING|trainer.py:803] 2025-04-26 21:23:43,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6166
[WARNING|trainer.py:803] 2025-04-26 21:23:44,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5538
[WARNING|trainer.py:803] 2025-04-26 21:23:45,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5580
[WARNING|trainer.py:803] 2025-04-26 21:23:45,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:45,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6167
5581
5539
[WARNING|trainer.py:803] 2025-04-26 21:23:46,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:47,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:47,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6168
5582
5540
[WARNING|trainer.py:803] 2025-04-26 21:23:48,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:48,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:49,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6169
5583
5541
[WARNING|trainer.py:803] 2025-04-26 21:23:50,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:50,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:23:50,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6170
5584
5542
[WARNING|trainer.py:803] 2025-04-26 21:23:51,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:51,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:52,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6171
5585
[WARNING|trainer.py:803] 2025-04-26 21:23:53,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:53,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5543
5586
6172
[WARNING|trainer.py:803] 2025-04-26 21:23:54,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:55,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:23:55,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5544
5587
6173
[WARNING|trainer.py:803] 2025-04-26 21:23:56,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:56,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:56,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5545
5588
6174
[WARNING|trainer.py:803] 2025-04-26 21:23:57,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:23:58,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:23:58,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5546
5589
6175
[WARNING|trainer.py:803] 2025-04-26 21:23:59,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:23:59,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:00,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5590
5547
6176
[WARNING|trainer.py:803] 2025-04-26 21:24:01,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:01,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:01,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5591
6177
5548
[WARNING|trainer.py:803] 2025-04-26 21:24:02,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:24:03,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:03,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5592
5549
[WARNING|trainer.py:803] 2025-04-26 21:24:04,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6178
[WARNING|trainer.py:803] 2025-04-26 21:24:04,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5593
[WARNING|trainer.py:803] 2025-04-26 21:24:05,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
5550
[WARNING|trainer.py:803] 2025-04-26 21:24:05,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6179
[WARNING|trainer.py:803] 2025-04-26 21:24:06,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5594
5551
[WARNING|trainer.py:803] 2025-04-26 21:24:07,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:07,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:08,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5595
6180
5552
[WARNING|trainer.py:803] 2025-04-26 21:24:08,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:09,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:09,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5596
6181
5553
[WARNING|trainer.py:803] 2025-04-26 21:24:10,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:11,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:11,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5597
5554
[WARNING|trainer.py:803] 2025-04-26 21:24:11,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6182
[WARNING|trainer.py:803] 2025-04-26 21:24:12,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5598
[WARNING|trainer.py:803] 2025-04-26 21:24:12,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5555
[WARNING|trainer.py:803] 2025-04-26 21:24:13,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6183
5599
[WARNING|trainer.py:803] 2025-04-26 21:24:14,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:14,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5556
[WARNING|trainer.py:803] 2025-04-26 21:24:14,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6184
5600
[WARNING|trainer.py:803] 2025-04-26 21:24:15,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:16,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5557
[WARNING|trainer.py:803] 2025-04-26 21:24:16,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5601
[WARNING|trainer.py:803] 2025-04-26 21:24:17,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6185
[WARNING|trainer.py:803] 2025-04-26 21:24:17,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5558
[WARNING|trainer.py:803] 2025-04-26 21:24:18,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5602
[WARNING|trainer.py:803] 2025-04-26 21:24:18,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6186
[WARNING|trainer.py:803] 2025-04-26 21:24:19,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5559
[WARNING|trainer.py:803] 2025-04-26 21:24:19,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5603
[WARNING|trainer.py:803] 2025-04-26 21:24:20,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6187
[WARNING|trainer.py:803] 2025-04-26 21:24:20,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5560
5604
[WARNING|trainer.py:803] 2025-04-26 21:24:21,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:21,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:22,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5561
6188
5605
[WARNING|trainer.py:803] 2025-04-26 21:24:23,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:23,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:23,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5562
6189
5606
[WARNING|trainer.py:803] 2025-04-26 21:24:24,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:25,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:25,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5563
5607
6190
[WARNING|trainer.py:803] 2025-04-26 21:24:26,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:26,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:27,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5564
5608
6191
[WARNING|trainer.py:803] 2025-04-26 21:24:28,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:28,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:28,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5565
5609
[WARNING|trainer.py:803] 2025-04-26 21:24:29,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:29,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6192
5566
5610
[WARNING|trainer.py:803] 2025-04-26 21:24:30,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:31,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:31,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6193
5567
5611
[WARNING|trainer.py:803] 2025-04-26 21:24:32,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:32,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:32,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5568
5612
6194
[WARNING|trainer.py:803] 2025-04-26 21:24:34,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:34,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:34,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5613
5569
6195
[WARNING|trainer.py:803] 2025-04-26 21:24:35,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:35,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:36,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5614
5570
[WARNING|trainer.py:803] 2025-04-26 21:24:37,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:37,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6196
5615
5571
[WARNING|trainer.py:803] 2025-04-26 21:24:38,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:38,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:38,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5616
5572
6197
[WARNING|trainer.py:803] 2025-04-26 21:24:40,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:40,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:40,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5617
5573
6198
[WARNING|trainer.py:803] 2025-04-26 21:24:41,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:41,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:42,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5574
5618
6199
[WARNING|trainer.py:803] 2025-04-26 21:24:43,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:43,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:43,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5575
5619
6200
[WARNING|trainer.py:803] 2025-04-26 21:24:44,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:44,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5620
5576
[WARNING|trainer.py:803] 2025-04-26 21:24:45,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:46,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:46,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6201
5621
5577
[WARNING|trainer.py:803] 2025-04-26 21:24:47,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:47,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:47,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6202
5622
5578
[WARNING|trainer.py:803] 2025-04-26 21:24:49,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:49,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:49,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5623
5579
6203
[WARNING|trainer.py:803] 2025-04-26 21:24:50,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:50,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:50,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5624
5580
6204
[WARNING|trainer.py:803] 2025-04-26 21:24:52,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:52,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:52,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5625
5581
6205
[WARNING|trainer.py:803] 2025-04-26 21:24:53,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:53,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:24:53,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5626
5582
6206
[WARNING|trainer.py:803] 2025-04-26 21:24:55,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:24:55,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:55,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5627
5583
6207
[WARNING|trainer.py:803] 2025-04-26 21:24:56,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:56,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5628
[WARNING|trainer.py:803] 2025-04-26 21:24:57,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5584
[WARNING|trainer.py:803] 2025-04-26 21:24:57,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:24:58,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6208
5629
5585
[WARNING|trainer.py:803] 2025-04-26 21:24:59,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:24:59,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:00,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5630
6209
5586
[WARNING|trainer.py:803] 2025-04-26 21:25:00,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:01,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:01,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5631
6210
5587
[WARNING|trainer.py:803] 2025-04-26 21:25:02,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:02,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:03,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5632
6211
5588
[WARNING|trainer.py:803] 2025-04-26 21:25:03,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:04,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5633
[WARNING|trainer.py:803] 2025-04-26 21:25:04,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6212
5589
[WARNING|trainer.py:803] 2025-04-26 21:25:05,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5634
[WARNING|trainer.py:803] 2025-04-26 21:25:06,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:06,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6213
5590
[WARNING|trainer.py:803] 2025-04-26 21:25:06,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5635
[WARNING|trainer.py:803] 2025-04-26 21:25:07,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:07,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5591
[WARNING|trainer.py:803] 2025-04-26 21:25:08,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6214
5636
[WARNING|trainer.py:803] 2025-04-26 21:25:09,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:25:09,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:09,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5592
6215
5637
[WARNING|trainer.py:803] 2025-04-26 21:25:10,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:10,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:11,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5593
6216
5638
[WARNING|trainer.py:803] 2025-04-26 21:25:12,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:12,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:12,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5594
6217
5639
[WARNING|trainer.py:803] 2025-04-26 21:25:13,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:14,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:14,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5595
6218
5640
[WARNING|trainer.py:803] 2025-04-26 21:25:15,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:15,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:15,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5596
5641
6219
[WARNING|trainer.py:803] 2025-04-26 21:25:16,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:17,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:17,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5597
5642
6220
[WARNING|trainer.py:803] 2025-04-26 21:25:18,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:18,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5598
[WARNING|trainer.py:803] 2025-04-26 21:25:19,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5643
6221
[WARNING|trainer.py:803] 2025-04-26 21:25:19,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:20,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:20,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5599
5644
6222
[WARNING|trainer.py:803] 2025-04-26 21:25:21,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:21,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5600
[WARNING|trainer.py:803] 2025-04-26 21:25:22,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5645
[WARNING|trainer.py:803] 2025-04-26 21:25:22,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:23,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6223
5601
5646
[WARNING|trainer.py:803] 2025-04-26 21:25:24,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:24,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:24,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6224
5602
5647
[WARNING|trainer.py:803] 2025-04-26 21:25:25,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:26,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:26,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6225
5603
5648
[WARNING|trainer.py:803] 2025-04-26 21:25:27,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:27,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:27,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6226
5604
5649
[WARNING|trainer.py:803] 2025-04-26 21:25:28,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:28,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:29,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6227
5605
5650
[WARNING|trainer.py:803] 2025-04-26 21:25:30,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:30,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:30,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5606
6228
5651
[WARNING|trainer.py:803] 2025-04-26 21:25:31,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:32,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:32,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5607
5652
6229
[WARNING|trainer.py:803] 2025-04-26 21:25:33,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:33,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:33,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5608
5653
6230
[WARNING|trainer.py:803] 2025-04-26 21:25:34,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:35,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:35,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5609
5654
6231
[WARNING|trainer.py:803] 2025-04-26 21:25:36,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:36,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:36,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5610
5655
6232
[WARNING|trainer.py:803] 2025-04-26 21:25:37,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:37,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:38,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5656
5611
6233
[WARNING|trainer.py:803] 2025-04-26 21:25:39,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:39,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:39,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5612
5657
6234
[WARNING|trainer.py:803] 2025-04-26 21:25:40,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:40,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:41,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5613
5658
6235
[WARNING|trainer.py:803] 2025-04-26 21:25:42,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:42,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5614
[WARNING|trainer.py:803] 2025-04-26 21:25:43,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5659
6236
[WARNING|trainer.py:803] 2025-04-26 21:25:43,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:43,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5615
5660
[WARNING|trainer.py:803] 2025-04-26 21:25:44,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:45,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:45,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6237
5616
5661
[WARNING|trainer.py:803] 2025-04-26 21:25:46,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:46,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:46,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6238
5617
5662
[WARNING|trainer.py:803] 2025-04-26 21:25:48,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:48,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:48,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5618
6239
5663
[WARNING|trainer.py:803] 2025-04-26 21:25:49,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:49,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:49,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5619
5664
6240
[WARNING|trainer.py:803] 2025-04-26 21:25:51,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:51,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:51,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5620
5665
6241
[WARNING|trainer.py:803] 2025-04-26 21:25:52,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:52,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:53,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5666
5621
6242
[WARNING|trainer.py:803] 2025-04-26 21:25:54,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:54,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:54,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5667
5622
6243
[WARNING|trainer.py:803] 2025-04-26 21:25:55,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:25:55,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:25:56,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5668
5623
6244
[WARNING|trainer.py:803] 2025-04-26 21:25:57,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:57,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5669
[WARNING|trainer.py:803] 2025-04-26 21:25:57,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5624
6245
[WARNING|trainer.py:803] 2025-04-26 21:25:58,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:25:58,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:25:59,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5670
5625
6246
[WARNING|trainer.py:803] 2025-04-26 21:26:00,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:00,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:00,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5671
5626
6247
[WARNING|trainer.py:803] 2025-04-26 21:26:01,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:01,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5672
[WARNING|trainer.py:803] 2025-04-26 21:26:02,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5627
[WARNING|trainer.py:803] 2025-04-26 21:26:03,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6248
[WARNING|trainer.py:803] 2025-04-26 21:26:03,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5673
5628
[WARNING|trainer.py:803] 2025-04-26 21:26:04,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:04,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:04,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6249
5674
5629
[WARNING|trainer.py:803] 2025-04-26 21:26:05,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:06,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:06,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6250
5675
5630
[WARNING|trainer.py:803] 2025-04-26 21:26:07,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:07,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:07,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6251
5631
5676
[WARNING|trainer.py:803] 2025-04-26 21:26:08,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:09,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:09,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6252
5677
5632
[WARNING|trainer.py:803] 2025-04-26 21:26:10,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:10,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:10,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5633
6253
5678
[WARNING|trainer.py:803] 2025-04-26 21:26:12,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:12,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:12,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6254
5679
5634
[WARNING|trainer.py:803] 2025-04-26 21:26:13,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:13,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:13,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6255
5680
5635
[WARNING|trainer.py:803] 2025-04-26 21:26:14,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:15,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:15,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5681
6256
5636
[WARNING|trainer.py:803] 2025-04-26 21:26:16,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:16,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:16,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5682
6257
5637
[WARNING|trainer.py:803] 2025-04-26 21:26:17,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:18,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:18,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5683
5638
6258
[WARNING|trainer.py:803] 2025-04-26 21:26:19,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:19,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:19,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5684
5639
6259
[WARNING|trainer.py:803] 2025-04-26 21:26:20,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:21,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:21,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5685
5640
6260
[WARNING|trainer.py:803] 2025-04-26 21:26:22,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:22,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:23,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5686
5641
6261
[WARNING|trainer.py:803] 2025-04-26 21:26:23,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:24,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5687
[WARNING|trainer.py:803] 2025-04-26 21:26:24,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5642
[WARNING|trainer.py:803] 2025-04-26 21:26:25,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6262
[WARNING|trainer.py:803] 2025-04-26 21:26:25,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5688
5643
[WARNING|trainer.py:803] 2025-04-26 21:26:26,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:26,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:27,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6263
5689
5644
[WARNING|trainer.py:803] 2025-04-26 21:26:28,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:28,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:28,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6264
5690
5645
[WARNING|trainer.py:803] 2025-04-26 21:26:29,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:29,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:30,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5691
6265
5646
[WARNING|trainer.py:803] 2025-04-26 21:26:31,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:31,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:31,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5692
6266
5647
[WARNING|trainer.py:803] 2025-04-26 21:26:32,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:33,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:33,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5693
6267
5648
[WARNING|trainer.py:803] 2025-04-26 21:26:34,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:34,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:34,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5694
5649
6268
[WARNING|trainer.py:803] 2025-04-26 21:26:35,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:36,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5695
[WARNING|trainer.py:803] 2025-04-26 21:26:36,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5650
[WARNING|trainer.py:803] 2025-04-26 21:26:37,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6269
[WARNING|trainer.py:803] 2025-04-26 21:26:37,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5696
[WARNING|trainer.py:803] 2025-04-26 21:26:38,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5651
[WARNING|trainer.py:803] 2025-04-26 21:26:38,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x89738440] moov atom not found
[21:26:38] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
6270
[WARNING|trainer.py:803] 2025-04-26 21:26:39,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5697
[WARNING|trainer.py:803] 2025-04-26 21:26:39,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5652
[WARNING|trainer.py:803] 2025-04-26 21:26:40,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6271
[WARNING|trainer.py:803] 2025-04-26 21:26:40,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5698
5653
[WARNING|trainer.py:803] 2025-04-26 21:26:41,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:41,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:42,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5699
6272
5654
[WARNING|trainer.py:803] 2025-04-26 21:26:43,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:43,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:43,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5700
5655
6273
[WARNING|trainer.py:803] 2025-04-26 21:26:44,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:45,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:45,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5656
5701
6274
[WARNING|trainer.py:803] 2025-04-26 21:26:46,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:46,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:47,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5657
[WARNING|trainer.py:803] 2025-04-26 21:26:48,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6275
5702
5658
[WARNING|trainer.py:803] 2025-04-26 21:26:48,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:49,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:49,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6276
5659
5703
[WARNING|trainer.py:803] 2025-04-26 21:26:50,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:51,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:51,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6277
5660
[WARNING|trainer.py:803] 2025-04-26 21:26:52,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5704
[WARNING|trainer.py:803] 2025-04-26 21:26:52,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6278
5661
[WARNING|trainer.py:803] 2025-04-26 21:26:53,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:53,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:53,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6279
5705
5662
[WARNING|trainer.py:803] 2025-04-26 21:26:55,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:26:55,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:26:55,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6280
5663
5706
[WARNING|trainer.py:803] 2025-04-26 21:26:56,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:26:56,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:26:57,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5664
6281
[WARNING|trainer.py:803] 2025-04-26 21:26:58,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5707
[WARNING|trainer.py:803] 2025-04-26 21:26:58,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5665
[WARNING|trainer.py:803] 2025-04-26 21:26:59,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6282
[WARNING|trainer.py:803] 2025-04-26 21:26:59,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:00,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5666
5708
6283
[WARNING|trainer.py:803] 2025-04-26 21:27:01,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:01,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5667
[WARNING|trainer.py:803] 2025-04-26 21:27:02,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5709
[WARNING|trainer.py:803] 2025-04-26 21:27:02,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6284
5668
[WARNING|trainer.py:803] 2025-04-26 21:27:03,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:03,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:04,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6285
5710
5669
[WARNING|trainer.py:803] 2025-04-26 21:27:05,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:05,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:05,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5670
5711
6286
[WARNING|trainer.py:803] 2025-04-26 21:27:07,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:07,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:07,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5671
[WARNING|trainer.py:803] 2025-04-26 21:27:08,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5712
6287
5672
[WARNING|trainer.py:803] 2025-04-26 21:27:09,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:09,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:10,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6288
5673
5713
[WARNING|trainer.py:803] 2025-04-26 21:27:11,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:11,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:11,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6289
5674
[WARNING|trainer.py:803] 2025-04-26 21:27:12,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5714
[WARNING|trainer.py:803] 2025-04-26 21:27:13,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:13,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5675
6290
[WARNING|trainer.py:803] 2025-04-26 21:27:14,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:15,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5715
5676
6291
[WARNING|trainer.py:803] 2025-04-26 21:27:16,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:16,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:16,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5677
5716
6292
[WARNING|trainer.py:803] 2025-04-26 21:27:17,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:18,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5678
[WARNING|trainer.py:803] 2025-04-26 21:27:18,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6293
5717
[WARNING|trainer.py:803] 2025-04-26 21:27:19,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5679
[WARNING|trainer.py:803] 2025-04-26 21:27:20,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:20,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:20,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6294
5718
5680
[WARNING|trainer.py:803] 2025-04-26 21:27:21,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:22,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:22,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6295
5681
5719
[WARNING|trainer.py:803] 2025-04-26 21:27:23,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:23,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:24,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6296
5682
[WARNING|trainer.py:803] 2025-04-26 21:27:25,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:25,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5720
5683
6297
[WARNING|trainer.py:803] 2025-04-26 21:27:26,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:26,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:26,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5684
5721
6298
[WARNING|trainer.py:803] 2025-04-26 21:27:28,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:28,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:28,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5685
5722
6299
[WARNING|trainer.py:803] 2025-04-26 21:27:29,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:30,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5686
[WARNING|trainer.py:803] 2025-04-26 21:27:30,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6300
[WARNING|trainer.py:803] 2025-04-26 21:27:31,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5723
5687
[WARNING|trainer.py:803] 2025-04-26 21:27:32,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6301
[WARNING|trainer.py:803] 2025-04-26 21:27:32,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:32,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:33,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5688
6302
5724
[WARNING|trainer.py:803] 2025-04-26 21:27:34,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:34,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:34,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6303
5689
[WARNING|trainer.py:803] 2025-04-26 21:27:35,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:35,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6304
5725
5690
[WARNING|trainer.py:803] 2025-04-26 21:27:36,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:36,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6305
[WARNING|trainer.py:803] 2025-04-26 21:27:37,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:37,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5691
5726
6306
[WARNING|trainer.py:803] 2025-04-26 21:27:38,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:38,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:38,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6307
5692
[WARNING|trainer.py:803] 2025-04-26 21:27:40,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5727
[WARNING|trainer.py:803] 2025-04-26 21:27:40,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6308
5693
[WARNING|trainer.py:803] 2025-04-26 21:27:40,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:41,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6309
[WARNING|trainer.py:803] 2025-04-26 21:27:41,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5728
[WARNING|trainer.py:803] 2025-04-26 21:27:42,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5694
6310
[WARNING|trainer.py:803] 2025-04-26 21:27:43,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:43,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:43,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6311
5695
5729
[WARNING|trainer.py:803] 2025-04-26 21:27:44,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:44,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6312
[WARNING|trainer.py:803] 2025-04-26 21:27:45,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5696
[WARNING|trainer.py:803] 2025-04-26 21:27:45,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6313
[WARNING|trainer.py:803] 2025-04-26 21:27:46,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3b860a80] moov atom not found
[21:27:46] /github/workspace/src/video/video_reader.cc:83: ERROR opening: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, Invalid data found when processing input
Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4... sharegpt4v_instruct_gpt4-vision_cap100k
Traceback (most recent call last):
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 903, in __getitem__
    ret = self.video_get_item(data_item)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 809, in video_get_item
    sampled_video = self.load_video_fast(video_path)
  File "/home/wangjiarui/AIGV_2025/train/train_qa.py", line 688, in load_video_fast
    video_reader = VideoReader(video_path)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/decord/video_reader.py", line 57, in __init__
    raise RuntimeError("Error reading " + uri + "...")
RuntimeError: Error reading /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4...
Failed to load video: /home/wangjiarui/AIGV6K/Videos300/Pixverse/10297.mp4, the dataset is: sharegpt4v_instruct_gpt4-vision_cap100k
5730
[WARNING|trainer.py:803] 2025-04-26 21:27:46,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5697
6314
[WARNING|trainer.py:803] 2025-04-26 21:27:47,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:47,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:47,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6315
5698
5731
[WARNING|trainer.py:803] 2025-04-26 21:27:48,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6316
[WARNING|trainer.py:803] 2025-04-26 21:27:49,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:49,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5699
[WARNING|trainer.py:803] 2025-04-26 21:27:50,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6317
5732
[WARNING|trainer.py:803] 2025-04-26 21:27:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:51,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:51,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6318
5700
[WARNING|trainer.py:803] 2025-04-26 21:27:52,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:52,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6319
5733
[WARNING|trainer.py:803] 2025-04-26 21:27:53,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:53,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5701
6320
[WARNING|trainer.py:803] 2025-04-26 21:27:54,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:27:54,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5734
6321
[WARNING|trainer.py:803] 2025-04-26 21:27:55,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:27:55,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5702
6322
[WARNING|trainer.py:803] 2025-04-26 21:27:56,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:56,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5735
6323
[WARNING|trainer.py:803] 2025-04-26 21:27:57,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:27:57,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5703
6324
5736
[WARNING|trainer.py:803] 2025-04-26 21:27:58,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:27:58,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6325
[WARNING|trainer.py:803] 2025-04-26 21:27:59,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:00,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5704
6326
[WARNING|trainer.py:803] 2025-04-26 21:28:00,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5737
[WARNING|trainer.py:803] 2025-04-26 21:28:01,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6327
[WARNING|trainer.py:803] 2025-04-26 21:28:01,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:02,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5705
6328
5738
[WARNING|trainer.py:803] 2025-04-26 21:28:03,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:03,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6329
[WARNING|trainer.py:803] 2025-04-26 21:28:03,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:04,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5706
6330
5739
[WARNING|trainer.py:803] 2025-04-26 21:28:05,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:05,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6331
[WARNING|trainer.py:803] 2025-04-26 21:28:05,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5707
[WARNING|trainer.py:803] 2025-04-26 21:28:06,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6332
5740
[WARNING|trainer.py:803] 2025-04-26 21:28:07,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:07,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:08,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6333
5708
[WARNING|trainer.py:803] 2025-04-26 21:28:08,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5741
6334
[WARNING|trainer.py:803] 2025-04-26 21:28:09,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:09,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:10,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6335
5709
[WARNING|trainer.py:803] 2025-04-26 21:28:11,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5742
6336
[WARNING|trainer.py:803] 2025-04-26 21:28:11,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:11,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:12,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6337
5710
5743
[WARNING|trainer.py:803] 2025-04-26 21:28:13,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6338
[WARNING|trainer.py:803] 2025-04-26 21:28:13,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:14,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:14,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6339
5711
5744
[WARNING|trainer.py:803] 2025-04-26 21:28:15,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:15,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6340
[WARNING|trainer.py:803] 2025-04-26 21:28:16,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:16,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6341
5712
5745
[WARNING|trainer.py:803] 2025-04-26 21:28:17,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:17,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6342
[WARNING|trainer.py:803] 2025-04-26 21:28:18,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:18,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5713
6343
5746
[WARNING|trainer.py:803] 2025-04-26 21:28:20,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:20,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6344
[WARNING|trainer.py:803] 2025-04-26 21:28:20,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:21,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5714
6345
5747
[WARNING|trainer.py:803] 2025-04-26 21:28:22,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:22,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6346
[WARNING|trainer.py:803] 2025-04-26 21:28:22,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:23,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5715
6347
5748
[WARNING|trainer.py:803] 2025-04-26 21:28:24,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:24,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:24,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6348
5716
[WARNING|trainer.py:803] 2025-04-26 21:28:25,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5749
6349
[WARNING|trainer.py:803] 2025-04-26 21:28:26,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:26,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:26,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6350
5717
5750
[WARNING|trainer.py:803] 2025-04-26 21:28:27,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6351
[WARNING|trainer.py:803] 2025-04-26 21:28:28,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:28,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:29,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6352
5718
5751
[WARNING|trainer.py:803] 2025-04-26 21:28:30,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6353
[WARNING|trainer.py:803] 2025-04-26 21:28:30,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:30,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:31,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6354
5719
5752
[WARNING|trainer.py:803] 2025-04-26 21:28:32,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6355
[WARNING|trainer.py:803] 2025-04-26 21:28:32,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:32,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:33,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6356
5720
5753
[WARNING|trainer.py:803] 2025-04-26 21:28:34,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:34,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6357
[WARNING|trainer.py:803] 2025-04-26 21:28:34,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:35,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6358
5721
5754
[WARNING|trainer.py:803] 2025-04-26 21:28:36,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:36,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:36,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6359
[WARNING|trainer.py:803] 2025-04-26 21:28:37,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5722
6360
5755
[WARNING|trainer.py:803] 2025-04-26 21:28:38,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:38,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:38,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6361
[WARNING|trainer.py:803] 2025-04-26 21:28:40,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5756
5723
6362
[WARNING|trainer.py:803] 2025-04-26 21:28:41,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:41,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:41,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6363
5757
5724
[WARNING|trainer.py:803] 2025-04-26 21:28:42,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6364
[WARNING|trainer.py:803] 2025-04-26 21:28:42,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:43,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:43,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6365
5758
5725
[WARNING|trainer.py:803] 2025-04-26 21:28:44,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:44,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6366
[WARNING|trainer.py:803] 2025-04-26 21:28:45,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:45,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6367
5759
5726
[WARNING|trainer.py:803] 2025-04-26 21:28:46,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:47,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6368
[WARNING|trainer.py:803] 2025-04-26 21:28:47,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:48,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5760
6369
5727
[WARNING|trainer.py:803] 2025-04-26 21:28:49,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:49,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:49,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6370
5761
[WARNING|trainer.py:803] 2025-04-26 21:28:50,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6371
5728
[WARNING|trainer.py:803] 2025-04-26 21:28:51,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:28:51,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:51,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6372
5762
[WARNING|trainer.py:803] 2025-04-26 21:28:52,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5729
6373
[WARNING|trainer.py:803] 2025-04-26 21:28:53,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:53,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:53,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6374
5763
[WARNING|trainer.py:803] 2025-04-26 21:28:54,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5730
[WARNING|trainer.py:803] 2025-04-26 21:28:55,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6375
[WARNING|trainer.py:803] 2025-04-26 21:28:55,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:55,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6376
5764
5731
[WARNING|trainer.py:803] 2025-04-26 21:28:57,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:28:57,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6377
[WARNING|trainer.py:803] 2025-04-26 21:28:57,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:28:58,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6378
5765
5732
[WARNING|trainer.py:803] 2025-04-26 21:28:59,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:28:59,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6379
[WARNING|trainer.py:803] 2025-04-26 21:28:59,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:00,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5766
6380
5733
[WARNING|trainer.py:803] 2025-04-26 21:29:01,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:01,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6381
[WARNING|trainer.py:803] 2025-04-26 21:29:01,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5767
[WARNING|trainer.py:803] 2025-04-26 21:29:02,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6382
5734
[WARNING|trainer.py:803] 2025-04-26 21:29:03,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:03,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:03,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6383
5768
[WARNING|trainer.py:803] 2025-04-26 21:29:04,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6384
5735
[WARNING|trainer.py:803] 2025-04-26 21:29:05,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:05,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:06,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6385
5769
[WARNING|trainer.py:803] 2025-04-26 21:29:07,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:07,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6386
5736
[WARNING|trainer.py:803] 2025-04-26 21:29:08,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:08,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6387
5770
[WARNING|trainer.py:803] 2025-04-26 21:29:09,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:09,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5737
6388
[WARNING|trainer.py:803] 2025-04-26 21:29:10,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:29:10,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5771
6389
[WARNING|trainer.py:803] 2025-04-26 21:29:11,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:11,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5738
6390
[WARNING|trainer.py:803] 2025-04-26 21:29:12,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:12,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5772
6391
[WARNING|trainer.py:803] 2025-04-26 21:29:13,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5739
[WARNING|trainer.py:803] 2025-04-26 21:29:13,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6392
[WARNING|trainer.py:803] 2025-04-26 21:29:14,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5773
[WARNING|trainer.py:803] 2025-04-26 21:29:14,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6393
[WARNING|trainer.py:803] 2025-04-26 21:29:15,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5740
[WARNING|trainer.py:803] 2025-04-26 21:29:16,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6394
[WARNING|trainer.py:803] 2025-04-26 21:29:16,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5774
[WARNING|trainer.py:803] 2025-04-26 21:29:17,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6395
[WARNING|trainer.py:803] 2025-04-26 21:29:17,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5741
[WARNING|trainer.py:803] 2025-04-26 21:29:18,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6396
[WARNING|trainer.py:803] 2025-04-26 21:29:18,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5775
[WARNING|trainer.py:803] 2025-04-26 21:29:19,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6397
[WARNING|trainer.py:803] 2025-04-26 21:29:19,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5742
[WARNING|trainer.py:803] 2025-04-26 21:29:20,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:20,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6398
5776
[WARNING|trainer.py:803] 2025-04-26 21:29:21,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:21,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6399
5743
[WARNING|trainer.py:803] 2025-04-26 21:29:22,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:22,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6400
5777
[WARNING|trainer.py:803] 2025-04-26 21:29:23,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:23,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6401
5744
[WARNING|trainer.py:803] 2025-04-26 21:29:24,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:24,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5778
6402
[WARNING|trainer.py:803] 2025-04-26 21:29:25,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:26,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5745
6403
[WARNING|trainer.py:803] 2025-04-26 21:29:26,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5779
[WARNING|trainer.py:803] 2025-04-26 21:29:27,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6404
[WARNING|trainer.py:803] 2025-04-26 21:29:27,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:28,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5746
6405
[WARNING|trainer.py:803] 2025-04-26 21:29:29,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:29,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5780
6406
[WARNING|trainer.py:803] 2025-04-26 21:29:30,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:30,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5747
6407
[WARNING|trainer.py:803] 2025-04-26 21:29:31,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5781
[WARNING|trainer.py:803] 2025-04-26 21:29:31,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6408
[WARNING|trainer.py:803] 2025-04-26 21:29:32,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5748
[WARNING|trainer.py:803] 2025-04-26 21:29:32,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6409
[WARNING|trainer.py:803] 2025-04-26 21:29:33,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5782
[WARNING|trainer.py:803] 2025-04-26 21:29:33,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6410
[WARNING|trainer.py:803] 2025-04-26 21:29:34,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5749
[WARNING|trainer.py:803] 2025-04-26 21:29:34,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6411
[WARNING|trainer.py:803] 2025-04-26 21:29:35,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5783
[WARNING|trainer.py:803] 2025-04-26 21:29:36,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6412
[WARNING|trainer.py:803] 2025-04-26 21:29:36,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5750
[WARNING|trainer.py:803] 2025-04-26 21:29:37,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6413
[WARNING|trainer.py:803] 2025-04-26 21:29:37,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5784
[WARNING|trainer.py:803] 2025-04-26 21:29:38,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6414
5751
[WARNING|trainer.py:803] 2025-04-26 21:29:38,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:39,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6415
[WARNING|trainer.py:803] 2025-04-26 21:29:39,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5785
[WARNING|trainer.py:803] 2025-04-26 21:29:40,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6416
5752
[WARNING|trainer.py:803] 2025-04-26 21:29:40,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:41,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:41,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6417
5786
[WARNING|trainer.py:803] 2025-04-26 21:29:42,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
5753
6418
[WARNING|trainer.py:803] 2025-04-26 21:29:43,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:43,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:43,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6419
5787
5754
[WARNING|trainer.py:803] 2025-04-26 21:29:45,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6420
[WARNING|trainer.py:803] 2025-04-26 21:29:45,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:45,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:46,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6421
5788
5755
[WARNING|trainer.py:803] 2025-04-26 21:29:47,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6422
[WARNING|trainer.py:803] 2025-04-26 21:29:47,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:47,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:48,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6423
5789
5756
[WARNING|trainer.py:803] 2025-04-26 21:29:49,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:29:49,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6424
[WARNING|trainer.py:803] 2025-04-26 21:29:50,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:29:50,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5790
6425
5757
[WARNING|trainer.py:803] 2025-04-26 21:29:51,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:51,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6426
[WARNING|trainer.py:803] 2025-04-26 21:29:52,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:52,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5791
6427
5758
[WARNING|trainer.py:803] 2025-04-26 21:29:53,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:29:53,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:54,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6428
5792
[WARNING|trainer.py:803] 2025-04-26 21:29:55,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5759
6429
[WARNING|trainer.py:803] 2025-04-26 21:29:55,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:29:56,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:29:56,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6430
5793
5760
[WARNING|trainer.py:803] 2025-04-26 21:29:57,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6431
[WARNING|trainer.py:803] 2025-04-26 21:29:57,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:29:58,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:29:58,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6432
5794
5761
[WARNING|trainer.py:803] 2025-04-26 21:29:59,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:29:59,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6433
[WARNING|trainer.py:803] 2025-04-26 21:30:00,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:00,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6434
5795
5762
[WARNING|trainer.py:803] 2025-04-26 21:30:01,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:02,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:02,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6435
[WARNING|trainer.py:803] 2025-04-26 21:30:02,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5796
6436
5763
[WARNING|trainer.py:803] 2025-04-26 21:30:04,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:04,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:04,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6437
[WARNING|trainer.py:803] 2025-04-26 21:30:05,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5797
5764
6438
[WARNING|trainer.py:803] 2025-04-26 21:30:06,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:06,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:06,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6439
[WARNING|trainer.py:803] 2025-04-26 21:30:07,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5798
5765
6440
[WARNING|trainer.py:803] 2025-04-26 21:30:08,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:08,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:08,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6441
5799
5766
[WARNING|trainer.py:803] 2025-04-26 21:30:09,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6442
[WARNING|trainer.py:803] 2025-04-26 21:30:10,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:10,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:10,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6443
5800
5767
[WARNING|trainer.py:803] 2025-04-26 21:30:11,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6444
[WARNING|trainer.py:803] 2025-04-26 21:30:12,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:12,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:13,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6445
5801
5768
[WARNING|trainer.py:803] 2025-04-26 21:30:14,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6446
[WARNING|trainer.py:803] 2025-04-26 21:30:14,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:14,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:15,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6447
5769
5802
[WARNING|trainer.py:803] 2025-04-26 21:30:16,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:16,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6448
[WARNING|trainer.py:803] 2025-04-26 21:30:16,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:17,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6449
5803
5770
[WARNING|trainer.py:803] 2025-04-26 21:30:18,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:18,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:18,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6450
[WARNING|trainer.py:803] 2025-04-26 21:30:19,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6451
5771
5804
[WARNING|trainer.py:803] 2025-04-26 21:30:20,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:20,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:21,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6452
[WARNING|trainer.py:803] 2025-04-26 21:30:21,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5772
5805
6453
[WARNING|trainer.py:803] 2025-04-26 21:30:23,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:23,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:23,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6454
5806
[WARNING|trainer.py:803] 2025-04-26 21:30:24,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5773
6455
[WARNING|trainer.py:803] 2025-04-26 21:30:25,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:25,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:25,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6456
5774
5807
[WARNING|trainer.py:803] 2025-04-26 21:30:26,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6457
[WARNING|trainer.py:803] 2025-04-26 21:30:27,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:27,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:27,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6458
5808
5775
[WARNING|trainer.py:803] 2025-04-26 21:30:28,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6459
[WARNING|trainer.py:803] 2025-04-26 21:30:29,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:30:29,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:29,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6460
5809
5776
[WARNING|trainer.py:803] 2025-04-26 21:30:30,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:30:31,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6461
[WARNING|trainer.py:803] 2025-04-26 21:30:31,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:32,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6462
5810
5777
[WARNING|trainer.py:803] 2025-04-26 21:30:33,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:33,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:33,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6463
[WARNING|trainer.py:803] 2025-04-26 21:30:34,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5811
5778
6464
[WARNING|trainer.py:803] 2025-04-26 21:30:35,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:35,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:35,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6465
5812
[WARNING|trainer.py:803] 2025-04-26 21:30:36,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5779
6466
[WARNING|trainer.py:803] 2025-04-26 21:30:37,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:37,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:37,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6467
5813
[WARNING|trainer.py:803] 2025-04-26 21:30:38,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5780
6468
[WARNING|trainer.py:803] 2025-04-26 21:30:39,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:39,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:30:40,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6469
5814
[WARNING|trainer.py:803] 2025-04-26 21:30:41,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5781
[WARNING|trainer.py:803] 2025-04-26 21:30:41,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6470
[WARNING|trainer.py:803] 2025-04-26 21:30:42,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:42,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6471
5815
5782
[WARNING|trainer.py:803] 2025-04-26 21:30:43,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:43,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6472
[WARNING|trainer.py:803] 2025-04-26 21:30:44,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:44,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6473
5816
[WARNING|trainer.py:803] 2025-04-26 21:30:45,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:45,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5783
6474
[WARNING|trainer.py:803] 2025-04-26 21:30:46,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:46,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5817
6475
[WARNING|trainer.py:803] 2025-04-26 21:30:47,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:47,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5784
6476
[WARNING|trainer.py:803] 2025-04-26 21:30:48,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:48,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5818
6477
5785
[WARNING|trainer.py:803] 2025-04-26 21:30:50,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:50,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6478
[WARNING|trainer.py:803] 2025-04-26 21:30:50,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:51,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5819
6479
5786
[WARNING|trainer.py:803] 2025-04-26 21:30:52,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:30:52,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6480
[WARNING|trainer.py:803] 2025-04-26 21:30:52,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:53,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5820
6481
[WARNING|trainer.py:803] 2025-04-26 21:30:54,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5787
[WARNING|trainer.py:803] 2025-04-26 21:30:54,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6482
[WARNING|trainer.py:803] 2025-04-26 21:30:55,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:55,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5821
6483
5788
[WARNING|trainer.py:803] 2025-04-26 21:30:56,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:56,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6484
[WARNING|trainer.py:803] 2025-04-26 21:30:57,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:57,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6485
5822
5789
[WARNING|trainer.py:803] 2025-04-26 21:30:58,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:30:59,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:30:59,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6486
[WARNING|trainer.py:803] 2025-04-26 21:31:00,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5790
5823
6487
[WARNING|trainer.py:803] 2025-04-26 21:31:01,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:01,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:01,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6488
5824
[WARNING|trainer.py:803] 2025-04-26 21:31:02,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5791
6489
[WARNING|trainer.py:803] 2025-04-26 21:31:03,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:03,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:03,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6490
5792
5825
[WARNING|trainer.py:803] 2025-04-26 21:31:04,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6491
[WARNING|trainer.py:803] 2025-04-26 21:31:05,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:05,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:05,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6492
5793
5826
[WARNING|trainer.py:803] 2025-04-26 21:31:06,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6493
[WARNING|trainer.py:803] 2025-04-26 21:31:07,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:07,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:07,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6494
5794
[WARNING|trainer.py:803] 2025-04-26 21:31:09,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5827
6495
[WARNING|trainer.py:803] 2025-04-26 21:31:09,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:09,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:10,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6496
5795
5828
[WARNING|trainer.py:803] 2025-04-26 21:31:11,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6497
[WARNING|trainer.py:803] 2025-04-26 21:31:11,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:11,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:12,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6498
5796
5829
[WARNING|trainer.py:803] 2025-04-26 21:31:13,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:13,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6499
[WARNING|trainer.py:803] 2025-04-26 21:31:13,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:14,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5797
6500
5830
[WARNING|trainer.py:803] 2025-04-26 21:31:15,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:15,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:16,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6501
[WARNING|trainer.py:803] 2025-04-26 21:31:16,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5798
6502
5831
[WARNING|trainer.py:803] 2025-04-26 21:31:17,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:18,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:18,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6503
5799
[WARNING|trainer.py:803] 2025-04-26 21:31:19,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6504
5832
[WARNING|trainer.py:803] 2025-04-26 21:31:19,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:20,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:20,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6505
5800
[WARNING|trainer.py:803] 2025-04-26 21:31:21,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5833
6506
[WARNING|trainer.py:803] 2025-04-26 21:31:21,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:22,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:22,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6507
5801
[WARNING|trainer.py:803] 2025-04-26 21:31:23,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5834
6508
[WARNING|trainer.py:803] 2025-04-26 21:31:24,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:24,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:24,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6509
5802
5835
[WARNING|trainer.py:803] 2025-04-26 21:31:25,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:26,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6510
[WARNING|trainer.py:803] 2025-04-26 21:31:26,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:27,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6511
5803
5836
[WARNING|trainer.py:803] 2025-04-26 21:31:28,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:28,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6512
[WARNING|trainer.py:803] 2025-04-26 21:31:28,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:29,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6513
5804
5837
[WARNING|trainer.py:803] 2025-04-26 21:31:30,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:30,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6514
[WARNING|trainer.py:803] 2025-04-26 21:31:30,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:31,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5805
6515
5838
[WARNING|trainer.py:803] 2025-04-26 21:31:32,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:32,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6516
[WARNING|trainer.py:803] 2025-04-26 21:31:32,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5806
[WARNING|trainer.py:803] 2025-04-26 21:31:33,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6517
5839
[WARNING|trainer.py:803] 2025-04-26 21:31:34,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:34,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:35,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6518
5807
[WARNING|trainer.py:803] 2025-04-26 21:31:36,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5840
6519
[WARNING|trainer.py:803] 2025-04-26 21:31:36,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:37,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:37,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6520
5808
5841
[WARNING|trainer.py:803] 2025-04-26 21:31:38,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6521
[WARNING|trainer.py:803] 2025-04-26 21:31:38,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:31:39,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:39,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6522
5809
5842
[WARNING|trainer.py:803] 2025-04-26 21:31:40,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:40,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6523
[WARNING|trainer.py:803] 2025-04-26 21:31:41,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:41,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6524
5810
5843
[WARNING|trainer.py:803] 2025-04-26 21:31:42,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:42,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6525
[WARNING|trainer.py:803] 2025-04-26 21:31:43,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:43,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5811
6526
5844
[WARNING|trainer.py:803] 2025-04-26 21:31:44,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:45,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6527
[WARNING|trainer.py:803] 2025-04-26 21:31:45,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5812
[WARNING|trainer.py:803] 2025-04-26 21:31:46,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6528
5845
[WARNING|trainer.py:803] 2025-04-26 21:31:46,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:47,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6529
[WARNING|trainer.py:803] 2025-04-26 21:31:47,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5813
[WARNING|trainer.py:803] 2025-04-26 21:31:48,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6530
5846
[WARNING|trainer.py:803] 2025-04-26 21:31:48,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:49,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:49,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6531
5814
[WARNING|trainer.py:803] 2025-04-26 21:31:50,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5847
6532
[WARNING|trainer.py:803] 2025-04-26 21:31:51,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:51,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:51,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6533
5815
[WARNING|trainer.py:803] 2025-04-26 21:31:52,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5848
[WARNING|trainer.py:803] 2025-04-26 21:31:53,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6534
[WARNING|trainer.py:803] 2025-04-26 21:31:53,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:31:53,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6535
5816
[WARNING|trainer.py:803] 2025-04-26 21:31:55,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5849
[WARNING|trainer.py:803] 2025-04-26 21:31:55,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6536
[WARNING|trainer.py:803] 2025-04-26 21:31:55,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:31:56,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5817
6537
5850
[WARNING|trainer.py:803] 2025-04-26 21:31:57,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:57,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6538
[WARNING|trainer.py:803] 2025-04-26 21:31:58,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:31:58,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6539
5818
[WARNING|trainer.py:803] 2025-04-26 21:31:59,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:31:59,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5851
6540
[WARNING|trainer.py:803] 2025-04-26 21:32:00,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:00,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5819
6541
[WARNING|trainer.py:803] 2025-04-26 21:32:01,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:01,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5852
6542
[WARNING|trainer.py:803] 2025-04-26 21:32:02,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:02,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5820
6543
[WARNING|trainer.py:803] 2025-04-26 21:32:03,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:03,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5853
6544
[WARNING|trainer.py:803] 2025-04-26 21:32:04,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:04,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6545
5821
5854
[WARNING|trainer.py:803] 2025-04-26 21:32:06,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:32:06,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6546
[WARNING|trainer.py:803] 2025-04-26 21:32:06,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:07,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6547
5822
5855
[WARNING|trainer.py:803] 2025-04-26 21:32:08,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:32:08,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6548
[WARNING|trainer.py:803] 2025-04-26 21:32:08,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:09,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5823
6549
5856
[WARNING|trainer.py:803] 2025-04-26 21:32:10,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:32:10,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:10,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6550
[WARNING|trainer.py:803] 2025-04-26 21:32:11,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5824
6551
5857
[WARNING|trainer.py:803] 2025-04-26 21:32:12,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:12,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:12,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6552
5825
[WARNING|trainer.py:803] 2025-04-26 21:32:13,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6553
5858
[WARNING|trainer.py:803] 2025-04-26 21:32:14,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:14,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6554
[WARNING|trainer.py:803] 2025-04-26 21:32:15,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5826
[WARNING|trainer.py:803] 2025-04-26 21:32:15,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6555
5859
[WARNING|trainer.py:803] 2025-04-26 21:32:16,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:17,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:17,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6556
[WARNING|trainer.py:803] 2025-04-26 21:32:18,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5827
6557
5860
[WARNING|trainer.py:803] 2025-04-26 21:32:19,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:19,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:19,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6558
5828
[WARNING|trainer.py:803] 2025-04-26 21:32:20,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5861
6559
[WARNING|trainer.py:803] 2025-04-26 21:32:21,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:32:21,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:21,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6560
5829
[WARNING|trainer.py:803] 2025-04-26 21:32:22,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5862
6561
[WARNING|trainer.py:803] 2025-04-26 21:32:23,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:23,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:23,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6562
5830
[WARNING|trainer.py:803] 2025-04-26 21:32:24,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5863
6563
[WARNING|trainer.py:803] 2025-04-26 21:32:25,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:25,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:25,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6564
5831
[WARNING|trainer.py:803] 2025-04-26 21:32:27,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5864
6565
[WARNING|trainer.py:803] 2025-04-26 21:32:27,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:28,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:28,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6566
5832
[WARNING|trainer.py:803] 2025-04-26 21:32:29,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5865
6567
[WARNING|trainer.py:803] 2025-04-26 21:32:29,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:30,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:30,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6568
5833
5866
[WARNING|trainer.py:803] 2025-04-26 21:32:31,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6569
[WARNING|trainer.py:803] 2025-04-26 21:32:31,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:32,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:32,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6570
5834
5867
[WARNING|trainer.py:803] 2025-04-26 21:32:33,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6571
[WARNING|trainer.py:803] 2025-04-26 21:32:34,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:34,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:34,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6572
5835
5868
[WARNING|trainer.py:803] 2025-04-26 21:32:35,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6573
[WARNING|trainer.py:803] 2025-04-26 21:32:36,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:36,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:36,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6574
5836
5869
[WARNING|trainer.py:803] 2025-04-26 21:32:37,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6575
[WARNING|trainer.py:803] 2025-04-26 21:32:38,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:38,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:39,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6576
5870
5837
[WARNING|trainer.py:803] 2025-04-26 21:32:40,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6577
[WARNING|trainer.py:803] 2025-04-26 21:32:40,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:40,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:32:41,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6578
5838
5871
[WARNING|trainer.py:803] 2025-04-26 21:32:42,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6579
[WARNING|trainer.py:803] 2025-04-26 21:32:42,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:42,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:43,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6580
5872
5839
[WARNING|trainer.py:803] 2025-04-26 21:32:44,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:44,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:44,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6581
[WARNING|trainer.py:803] 2025-04-26 21:32:45,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5873
5840
6582
[WARNING|trainer.py:803] 2025-04-26 21:32:46,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:46,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:46,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6583
5874
[WARNING|trainer.py:803] 2025-04-26 21:32:47,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5841
6584
[WARNING|trainer.py:803] 2025-04-26 21:32:48,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:48,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:48,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6585
5875
5842
[WARNING|trainer.py:803] 2025-04-26 21:32:50,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6586
[WARNING|trainer.py:803] 2025-04-26 21:32:50,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:50,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:51,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6587
5876
5843
[WARNING|trainer.py:803] 2025-04-26 21:32:52,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6588
[WARNING|trainer.py:803] 2025-04-26 21:32:52,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:32:53,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:53,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6589
5877
5844
[WARNING|trainer.py:803] 2025-04-26 21:32:54,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6590
[WARNING|trainer.py:803] 2025-04-26 21:32:54,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:55,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:55,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6591
5878
5845
[WARNING|trainer.py:803] 2025-04-26 21:32:56,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:56,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6592
[WARNING|trainer.py:803] 2025-04-26 21:32:57,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:32:57,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6593
5879
5846
[WARNING|trainer.py:803] 2025-04-26 21:32:58,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:32:59,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6594
[WARNING|trainer.py:803] 2025-04-26 21:32:59,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:00,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6595
5880
5847
[WARNING|trainer.py:803] 2025-04-26 21:33:01,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:01,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:01,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6596
[WARNING|trainer.py:803] 2025-04-26 21:33:02,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6597
5881
5848
[WARNING|trainer.py:803] 2025-04-26 21:33:03,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:03,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6598
[WARNING|trainer.py:803] 2025-04-26 21:33:03,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:04,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6599
5882
5849
[WARNING|trainer.py:803] 2025-04-26 21:33:05,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:05,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:05,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6600
[WARNING|trainer.py:803] 2025-04-26 21:33:06,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6601
5883
5850
[WARNING|trainer.py:803] 2025-04-26 21:33:07,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:07,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:07,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6602
[WARNING|trainer.py:803] 2025-04-26 21:33:08,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5884
5851
6603
[WARNING|trainer.py:803] 2025-04-26 21:33:09,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:09,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:10,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6604
5885
[WARNING|trainer.py:803] 2025-04-26 21:33:11,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5852
6605
[WARNING|trainer.py:803] 2025-04-26 21:33:11,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:12,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:12,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6606
5886
[WARNING|trainer.py:803] 2025-04-26 21:33:13,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5853
6607
[WARNING|trainer.py:803] 2025-04-26 21:33:13,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:14,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:14,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
5887
5854
[WARNING|trainer.py:803] 2025-04-26 21:33:15,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6609
[WARNING|trainer.py:803] 2025-04-26 21:33:16,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:16,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:16,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6610
5855
[WARNING|trainer.py:803] 2025-04-26 21:33:17,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5888
6611
[WARNING|trainer.py:803] 2025-04-26 21:33:18,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:18,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:18,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6612
5856
[WARNING|trainer.py:803] 2025-04-26 21:33:20,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5889
6613
[WARNING|trainer.py:803] 2025-04-26 21:33:20,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:20,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:21,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6614
5857
[WARNING|trainer.py:803] 2025-04-26 21:33:22,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5890
6615
[WARNING|trainer.py:803] 2025-04-26 21:33:22,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:23,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:23,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6616
5858
5891
[WARNING|trainer.py:803] 2025-04-26 21:33:24,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6617
[WARNING|trainer.py:803] 2025-04-26 21:33:24,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:25,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:25,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6618
5859
5892
[WARNING|trainer.py:803] 2025-04-26 21:33:26,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6619
[WARNING|trainer.py:803] 2025-04-26 21:33:26,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:27,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:27,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6620
5860
5893
[WARNING|trainer.py:803] 2025-04-26 21:33:28,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6621
[WARNING|trainer.py:803] 2025-04-26 21:33:29,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:29,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:29,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6622
5861
5894
[WARNING|trainer.py:803] 2025-04-26 21:33:30,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6623
[WARNING|trainer.py:803] 2025-04-26 21:33:31,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:31,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:32,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6624
5862
5895
[WARNING|trainer.py:803] 2025-04-26 21:33:33,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6625
[WARNING|trainer.py:803] 2025-04-26 21:33:33,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:33,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:34,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6626
5863
5896
[WARNING|trainer.py:803] 2025-04-26 21:33:35,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6627
[WARNING|trainer.py:803] 2025-04-26 21:33:35,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:35,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:36,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6628
5864
5897
[WARNING|trainer.py:803] 2025-04-26 21:33:37,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6629
[WARNING|trainer.py:803] 2025-04-26 21:33:38,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:38,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:38,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6630
5898
5865
[WARNING|trainer.py:803] 2025-04-26 21:33:39,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6631
[WARNING|trainer.py:803] 2025-04-26 21:33:40,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:40,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:40,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6632
5899
5866
[WARNING|trainer.py:803] 2025-04-26 21:33:41,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6633
[WARNING|trainer.py:803] 2025-04-26 21:33:42,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:42,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:42,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6634
5900
5867
[WARNING|trainer.py:803] 2025-04-26 21:33:44,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6635
[WARNING|trainer.py:803] 2025-04-26 21:33:44,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:44,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:45,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6636
5901
5868
[WARNING|trainer.py:803] 2025-04-26 21:33:46,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:33:46,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6637
[WARNING|trainer.py:803] 2025-04-26 21:33:46,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:47,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5902
6638
5869
[WARNING|trainer.py:803] 2025-04-26 21:33:48,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:48,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6639
[WARNING|trainer.py:803] 2025-04-26 21:33:48,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5903
[WARNING|trainer.py:803] 2025-04-26 21:33:49,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6640
5870
[WARNING|trainer.py:803] 2025-04-26 21:33:50,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:50,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6641
[WARNING|trainer.py:803] 2025-04-26 21:33:51,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5904
[WARNING|trainer.py:803] 2025-04-26 21:33:51,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6642
5871
[WARNING|trainer.py:803] 2025-04-26 21:33:52,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:52,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:53,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6643
5905
[WARNING|trainer.py:803] 2025-04-26 21:33:54,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5872
6644
[WARNING|trainer.py:803] 2025-04-26 21:33:54,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:55,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:33:55,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6645
5906
[WARNING|trainer.py:803] 2025-04-26 21:33:56,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5873
[WARNING|trainer.py:803] 2025-04-26 21:33:56,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6646
[WARNING|trainer.py:803] 2025-04-26 21:33:57,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:33:57,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5907
6647
5874
[WARNING|trainer.py:803] 2025-04-26 21:33:58,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:33:58,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6648
[WARNING|trainer.py:803] 2025-04-26 21:33:59,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
5908
[WARNING|trainer.py:803] 2025-04-26 21:33:59,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6649
[WARNING|trainer.py:803] 2025-04-26 21:34:00,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5875
[WARNING|trainer.py:803] 2025-04-26 21:34:00,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6650
[WARNING|trainer.py:803] 2025-04-26 21:34:01,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5909
[WARNING|trainer.py:803] 2025-04-26 21:34:01,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6651
[WARNING|trainer.py:803] 2025-04-26 21:34:02,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5876
[WARNING|trainer.py:803] 2025-04-26 21:34:03,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:03,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6652
5910
[WARNING|trainer.py:803] 2025-04-26 21:34:04,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:04,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6653
5877
[WARNING|trainer.py:803] 2025-04-26 21:34:05,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:05,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6654
5911
[WARNING|trainer.py:803] 2025-04-26 21:34:06,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:06,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5878
6655
[WARNING|trainer.py:803] 2025-04-26 21:34:07,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:07,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5912
6656
[WARNING|trainer.py:803] 2025-04-26 21:34:08,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:08,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5879
6657
5913
[WARNING|trainer.py:803] 2025-04-26 21:34:09,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:09,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6658
[WARNING|trainer.py:803] 2025-04-26 21:34:10,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:10,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5880
6659
5914
[WARNING|trainer.py:803] 2025-04-26 21:34:11,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:34:11,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6660
[WARNING|trainer.py:803] 2025-04-26 21:34:12,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:13,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5881
6661
5915
[WARNING|trainer.py:803] 2025-04-26 21:34:14,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:34:14,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:14,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6662
[WARNING|trainer.py:803] 2025-04-26 21:34:15,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5916
5882
6663
[WARNING|trainer.py:803] 2025-04-26 21:34:16,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:16,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:16,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6664
5917
[WARNING|trainer.py:803] 2025-04-26 21:34:17,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5883
6665
[WARNING|trainer.py:803] 2025-04-26 21:34:18,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:34:18,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:18,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6666
5918
[WARNING|trainer.py:803] 2025-04-26 21:34:19,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5884
6667
[WARNING|trainer.py:803] 2025-04-26 21:34:20,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:20,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:20,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6668
5919
5885
[WARNING|trainer.py:803] 2025-04-26 21:34:21,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6669
[WARNING|trainer.py:803] 2025-04-26 21:34:22,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:22,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:23,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6670
5920
5886
[WARNING|trainer.py:803] 2025-04-26 21:34:24,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6671
[WARNING|trainer.py:803] 2025-04-26 21:34:24,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:34:24,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:25,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6672
5921
5887
[WARNING|trainer.py:803] 2025-04-26 21:34:26,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:26,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6673
[WARNING|trainer.py:803] 2025-04-26 21:34:26,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5922
[WARNING|trainer.py:803] 2025-04-26 21:34:27,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6674
[WARNING|trainer.py:803] 2025-04-26 21:34:28,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5888
[WARNING|trainer.py:803] 2025-04-26 21:34:28,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6675
[WARNING|trainer.py:803] 2025-04-26 21:34:29,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5923
[WARNING|trainer.py:803] 2025-04-26 21:34:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6676
[WARNING|trainer.py:803] 2025-04-26 21:34:30,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5889
[WARNING|trainer.py:803] 2025-04-26 21:34:30,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6677
[WARNING|trainer.py:803] 2025-04-26 21:34:31,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5924
[WARNING|trainer.py:803] 2025-04-26 21:34:32,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6678
[WARNING|trainer.py:803] 2025-04-26 21:34:32,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5890
[WARNING|trainer.py:803] 2025-04-26 21:34:33,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6679
[WARNING|trainer.py:803] 2025-04-26 21:34:33,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5925
[WARNING|trainer.py:803] 2025-04-26 21:34:34,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6680
[WARNING|trainer.py:803] 2025-04-26 21:34:34,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5891
[WARNING|trainer.py:803] 2025-04-26 21:34:35,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6681
[WARNING|trainer.py:803] 2025-04-26 21:34:35,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5926
[WARNING|trainer.py:803] 2025-04-26 21:34:36,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6682
[WARNING|trainer.py:803] 2025-04-26 21:34:36,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5892
[WARNING|trainer.py:803] 2025-04-26 21:34:37,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6683
5927
[WARNING|trainer.py:803] 2025-04-26 21:34:38,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:38,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:38,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6684
5893
5928
[WARNING|trainer.py:803] 2025-04-26 21:34:39,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6685
[WARNING|trainer.py:803] 2025-04-26 21:34:40,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:40,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:40,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6686
5894
5929
[WARNING|trainer.py:803] 2025-04-26 21:34:42,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6687
[WARNING|trainer.py:803] 2025-04-26 21:34:42,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:42,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:43,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6688
5895
5930
[WARNING|trainer.py:803] 2025-04-26 21:34:44,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6689
[WARNING|trainer.py:803] 2025-04-26 21:34:44,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:44,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:45,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6690
5931
5896
[WARNING|trainer.py:803] 2025-04-26 21:34:46,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:46,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6691
[WARNING|trainer.py:803] 2025-04-26 21:34:46,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:47,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5932
6692
5897
[WARNING|trainer.py:803] 2025-04-26 21:34:48,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:48,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6693
[WARNING|trainer.py:803] 2025-04-26 21:34:48,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5933
[WARNING|trainer.py:803] 2025-04-26 21:34:49,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6694
5898
[WARNING|trainer.py:803] 2025-04-26 21:34:50,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:50,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:50,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6695
5934
[WARNING|trainer.py:803] 2025-04-26 21:34:51,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5899
6696
[WARNING|trainer.py:803] 2025-04-26 21:34:52,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:52,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:34:53,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6697
5935
[WARNING|trainer.py:803] 2025-04-26 21:34:54,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5900
[WARNING|trainer.py:803] 2025-04-26 21:34:54,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6698
[WARNING|trainer.py:803] 2025-04-26 21:34:55,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:34:55,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5936
6699
[WARNING|trainer.py:803] 2025-04-26 21:34:56,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5901
[WARNING|trainer.py:803] 2025-04-26 21:34:56,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6700
[WARNING|trainer.py:803] 2025-04-26 21:34:57,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:34:57,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5937
6701
[WARNING|trainer.py:803] 2025-04-26 21:34:58,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5902
[WARNING|trainer.py:803] 2025-04-26 21:34:58,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6702
[WARNING|trainer.py:803] 2025-04-26 21:34:59,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5938
[WARNING|trainer.py:803] 2025-04-26 21:34:59,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6703
[WARNING|trainer.py:803] 2025-04-26 21:35:00,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5903
[WARNING|trainer.py:803] 2025-04-26 21:35:00,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6704
[WARNING|trainer.py:803] 2025-04-26 21:35:01,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5939
[WARNING|trainer.py:803] 2025-04-26 21:35:02,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6705
[WARNING|trainer.py:803] 2025-04-26 21:35:02,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5904
[WARNING|trainer.py:803] 2025-04-26 21:35:03,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:03,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6706
5940
[WARNING|trainer.py:803] 2025-04-26 21:35:04,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:04,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5905
6707
[WARNING|trainer.py:803] 2025-04-26 21:35:05,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5941
[WARNING|trainer.py:803] 2025-04-26 21:35:05,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6708
[WARNING|trainer.py:803] 2025-04-26 21:35:06,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5906
[WARNING|trainer.py:803] 2025-04-26 21:35:06,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6709
[WARNING|trainer.py:803] 2025-04-26 21:35:07,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5942
[WARNING|trainer.py:803] 2025-04-26 21:35:07,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6710
[WARNING|trainer.py:803] 2025-04-26 21:35:08,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5907
[WARNING|trainer.py:803] 2025-04-26 21:35:08,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6711
[WARNING|trainer.py:803] 2025-04-26 21:35:09,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5943
[WARNING|trainer.py:803] 2025-04-26 21:35:10,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6712
[WARNING|trainer.py:803] 2025-04-26 21:35:10,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5908
[WARNING|trainer.py:803] 2025-04-26 21:35:11,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6713
[WARNING|trainer.py:803] 2025-04-26 21:35:11,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5944
[WARNING|trainer.py:803] 2025-04-26 21:35:12,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:12,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6714
5909
[WARNING|trainer.py:803] 2025-04-26 21:35:13,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5945
[WARNING|trainer.py:803] 2025-04-26 21:35:13,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6715
[WARNING|trainer.py:803] 2025-04-26 21:35:14,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:14,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5910
6716
5946
[WARNING|trainer.py:803] 2025-04-26 21:35:15,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:15,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6717
[WARNING|trainer.py:803] 2025-04-26 21:35:16,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5911
[WARNING|trainer.py:803] 2025-04-26 21:35:16,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6718
5947
[WARNING|trainer.py:803] 2025-04-26 21:35:17,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:17,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:18,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6719
5912
[WARNING|trainer.py:803] 2025-04-26 21:35:19,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5948
6720
[WARNING|trainer.py:803] 2025-04-26 21:35:19,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:19,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:35:20,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6721
5913
5949
[WARNING|trainer.py:803] 2025-04-26 21:35:21,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:21,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6722
[WARNING|trainer.py:803] 2025-04-26 21:35:21,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:22,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5914
6723
5950
[WARNING|trainer.py:803] 2025-04-26 21:35:23,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:23,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6724
[WARNING|trainer.py:803] 2025-04-26 21:35:24,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5915
[WARNING|trainer.py:803] 2025-04-26 21:35:24,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6725
5951
[WARNING|trainer.py:803] 2025-04-26 21:35:25,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:35:25,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:26,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6726
5916
[WARNING|trainer.py:803] 2025-04-26 21:35:26,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5952
6727
[WARNING|trainer.py:803] 2025-04-26 21:35:27,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:27,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:28,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6728
5917
5953
[WARNING|trainer.py:803] 2025-04-26 21:35:29,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:29,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6729
[WARNING|trainer.py:803] 2025-04-26 21:35:29,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:30,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6730
5918
5954
[WARNING|trainer.py:803] 2025-04-26 21:35:31,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 21:35:31,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6731
[WARNING|trainer.py:803] 2025-04-26 21:35:31,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:32,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5919
6732
5955
[WARNING|trainer.py:803] 2025-04-26 21:35:33,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:33,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:33,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6733
[WARNING|trainer.py:803] 2025-04-26 21:35:34,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5920
5956
6734
[WARNING|trainer.py:803] 2025-04-26 21:35:35,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:35:35,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:35,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6735
5921
5957
[WARNING|trainer.py:803] 2025-04-26 21:35:37,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6736
[WARNING|trainer.py:803] 2025-04-26 21:35:37,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:37,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:38,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6737
5958
5922
[WARNING|trainer.py:803] 2025-04-26 21:35:39,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6738
[WARNING|trainer.py:803] 2025-04-26 21:35:39,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:39,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6739
5959
5923
[WARNING|trainer.py:803] 2025-04-26 21:35:41,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6740
[WARNING|trainer.py:803] 2025-04-26 21:35:41,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:41,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:42,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5960
6741
5924
[WARNING|trainer.py:803] 2025-04-26 21:35:43,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:35:43,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6742
[WARNING|trainer.py:803] 2025-04-26 21:35:43,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5961
[WARNING|trainer.py:803] 2025-04-26 21:35:44,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6743
5925
[WARNING|trainer.py:803] 2025-04-26 21:35:45,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:45,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:35:45,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6744
5962
[WARNING|trainer.py:803] 2025-04-26 21:35:47,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5926
6745
[WARNING|trainer.py:803] 2025-04-26 21:35:47,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:48,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:48,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6746
5963
5927
[WARNING|trainer.py:803] 2025-04-26 21:35:49,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:49,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6747
[WARNING|trainer.py:803] 2025-04-26 21:35:50,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:50,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5964
6748
5928
[WARNING|trainer.py:803] 2025-04-26 21:35:51,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:51,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6749
[WARNING|trainer.py:803] 2025-04-26 21:35:51,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5965
[WARNING|trainer.py:803] 2025-04-26 21:35:52,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6750
5929
[WARNING|trainer.py:803] 2025-04-26 21:35:53,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:53,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:53,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6751
5966
[WARNING|trainer.py:803] 2025-04-26 21:35:54,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5930
6752
[WARNING|trainer.py:803] 2025-04-26 21:35:55,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:55,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:56,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6753
5967
5931
[WARNING|trainer.py:803] 2025-04-26 21:35:57,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:35:57,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6754
[WARNING|trainer.py:803] 2025-04-26 21:35:57,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:35:58,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5968
6755
5932
[WARNING|trainer.py:803] 2025-04-26 21:35:59,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:35:59,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6756
[WARNING|trainer.py:803] 2025-04-26 21:35:59,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:00,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5969
6757
5933
[WARNING|trainer.py:803] 2025-04-26 21:36:01,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:01,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:01,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6758
5970
[WARNING|trainer.py:803] 2025-04-26 21:36:02,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5934
6759
[WARNING|trainer.py:803] 2025-04-26 21:36:03,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:03,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:03,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6760
5971
5935
[WARNING|trainer.py:803] 2025-04-26 21:36:04,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6761
[WARNING|trainer.py:803] 2025-04-26 21:36:05,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:05,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:06,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6762
5972
5936
[WARNING|trainer.py:803] 2025-04-26 21:36:07,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6763
[WARNING|trainer.py:803] 2025-04-26 21:36:07,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:07,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:08,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6764
5973
5937
[WARNING|trainer.py:803] 2025-04-26 21:36:09,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:09,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6765
[WARNING|trainer.py:803] 2025-04-26 21:36:09,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:10,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6766
5938
5974
[WARNING|trainer.py:803] 2025-04-26 21:36:11,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:11,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:11,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6767
5939
[WARNING|trainer.py:803] 2025-04-26 21:36:12,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6768
5975
[WARNING|trainer.py:803] 2025-04-26 21:36:13,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:13,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:13,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6769
5940
5976
[WARNING|trainer.py:803] 2025-04-26 21:36:14,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6770
[WARNING|trainer.py:803] 2025-04-26 21:36:15,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:15,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:16,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6771
5941
5977
[WARNING|trainer.py:803] 2025-04-26 21:36:17,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6772
[WARNING|trainer.py:803] 2025-04-26 21:36:17,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:17,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:18,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6773
5942
5978
[WARNING|trainer.py:803] 2025-04-26 21:36:19,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6774
[WARNING|trainer.py:803] 2025-04-26 21:36:19,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:19,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:20,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6775
5943
5979
[WARNING|trainer.py:803] 2025-04-26 21:36:21,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:21,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:21,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6776
[WARNING|trainer.py:803] 2025-04-26 21:36:22,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5944
5980
6777
[WARNING|trainer.py:803] 2025-04-26 21:36:23,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:23,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:23,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6778
5945
[WARNING|trainer.py:803] 2025-04-26 21:36:24,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5981
6779
[WARNING|trainer.py:803] 2025-04-26 21:36:25,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:25,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:25,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6780
5946
[WARNING|trainer.py:803] 2025-04-26 21:36:27,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5982
6781
[WARNING|trainer.py:803] 2025-04-26 21:36:27,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:27,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:28,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6782
5947
5983
[WARNING|trainer.py:803] 2025-04-26 21:36:29,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6783
[WARNING|trainer.py:803] 2025-04-26 21:36:29,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:30,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:30,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6784
5948
5984
[WARNING|trainer.py:803] 2025-04-26 21:36:31,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:31,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6785
[WARNING|trainer.py:803] 2025-04-26 21:36:32,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:32,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5949
6786
5985
[WARNING|trainer.py:803] 2025-04-26 21:36:33,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:33,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6787
[WARNING|trainer.py:803] 2025-04-26 21:36:34,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:36:34,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5950
6788
5986
[WARNING|trainer.py:803] 2025-04-26 21:36:35,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:35,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6789
[WARNING|trainer.py:803] 2025-04-26 21:36:36,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5951
[WARNING|trainer.py:803] 2025-04-26 21:36:36,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6790
[WARNING|trainer.py:803] 2025-04-26 21:36:37,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5987
[WARNING|trainer.py:803] 2025-04-26 21:36:38,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6791
[WARNING|trainer.py:803] 2025-04-26 21:36:38,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5952
[WARNING|trainer.py:803] 2025-04-26 21:36:39,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6792
[WARNING|trainer.py:803] 2025-04-26 21:36:39,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5988
[WARNING|trainer.py:803] 2025-04-26 21:36:40,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:40,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6793
5953
[WARNING|trainer.py:803] 2025-04-26 21:36:41,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:41,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5989
6794
[WARNING|trainer.py:803] 2025-04-26 21:36:42,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:36:42,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5954
6795
[WARNING|trainer.py:803] 2025-04-26 21:36:43,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:43,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6796
5990
5955
[WARNING|trainer.py:803] 2025-04-26 21:36:44,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:44,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6797
[WARNING|trainer.py:803] 2025-04-26 21:36:45,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:45,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5991
6798
5956
[WARNING|trainer.py:803] 2025-04-26 21:36:46,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:47,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6799
[WARNING|trainer.py:803] 2025-04-26 21:36:47,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5992
[WARNING|trainer.py:803] 2025-04-26 21:36:48,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6800
5957
[WARNING|trainer.py:803] 2025-04-26 21:36:48,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:49,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6801
[WARNING|trainer.py:803] 2025-04-26 21:36:49,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5993
[WARNING|trainer.py:803] 2025-04-26 21:36:50,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6802
5958
[WARNING|trainer.py:803] 2025-04-26 21:36:51,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:51,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:51,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6803
5994
[WARNING|trainer.py:803] 2025-04-26 21:36:52,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5959
6804
[WARNING|trainer.py:803] 2025-04-26 21:36:53,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:53,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:53,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6805
5995
5960
[WARNING|trainer.py:803] 2025-04-26 21:36:54,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6806
[WARNING|trainer.py:803] 2025-04-26 21:36:55,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:55,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:36:55,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6807
5996
5961
[WARNING|trainer.py:803] 2025-04-26 21:36:56,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:56,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6808
[WARNING|trainer.py:803] 2025-04-26 21:36:57,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:36:57,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5997
6809
5962
[WARNING|trainer.py:803] 2025-04-26 21:36:59,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:36:59,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6810
[WARNING|trainer.py:803] 2025-04-26 21:36:59,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5998
[WARNING|trainer.py:803] 2025-04-26 21:37:00,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6811
5963
[WARNING|trainer.py:803] 2025-04-26 21:37:00,978 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:01,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:01,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6812
5999
[WARNING|trainer.py:803] 2025-04-26 21:37:02,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5964
6813
[WARNING|trainer.py:803] 2025-04-26 21:37:02,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:03,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:03,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6814
6000
5965
[WARNING|trainer.py:803] 2025-04-26 21:37:04,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6815
[WARNING|trainer.py:803] 2025-04-26 21:37:04,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:05,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:05,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6816
6001
5966
[WARNING|trainer.py:803] 2025-04-26 21:37:06,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:06,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6817
[WARNING|trainer.py:803] 2025-04-26 21:37:07,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:07,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6818
6002
5967
[WARNING|trainer.py:803] 2025-04-26 21:37:08,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:09,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6819
[WARNING|trainer.py:803] 2025-04-26 21:37:09,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:09,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6820
5968
6003
[WARNING|trainer.py:803] 2025-04-26 21:37:11,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:11,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6821
[WARNING|trainer.py:803] 2025-04-26 21:37:11,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:12,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6004
6822
5969
[WARNING|trainer.py:803] 2025-04-26 21:37:13,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:13,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:13,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6823
[WARNING|trainer.py:803] 2025-04-26 21:37:14,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6005
5970
6824
[WARNING|trainer.py:803] 2025-04-26 21:37:15,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:15,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:15,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6825
6006
[WARNING|trainer.py:803] 2025-04-26 21:37:16,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5971
6826
[WARNING|trainer.py:803] 2025-04-26 21:37:17,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:17,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:17,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6827
6007
[WARNING|trainer.py:803] 2025-04-26 21:37:18,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5972
6828
[WARNING|trainer.py:803] 2025-04-26 21:37:19,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:19,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:19,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6829
6008
[WARNING|trainer.py:803] 2025-04-26 21:37:20,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5973
[WARNING|trainer.py:803] 2025-04-26 21:37:20,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6830
[WARNING|trainer.py:803] 2025-04-26 21:37:21,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:21,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6831
6009
[WARNING|trainer.py:803] 2025-04-26 21:37:23,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:23,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5974
6832
[WARNING|trainer.py:803] 2025-04-26 21:37:23,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6010
[WARNING|trainer.py:803] 2025-04-26 21:37:24,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6833
[WARNING|trainer.py:803] 2025-04-26 21:37:24,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:25,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5975
6834
6011
[WARNING|trainer.py:803] 2025-04-26 21:37:26,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:26,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6835
[WARNING|trainer.py:803] 2025-04-26 21:37:26,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5976
[WARNING|trainer.py:803] 2025-04-26 21:37:27,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6836
6012
[WARNING|trainer.py:803] 2025-04-26 21:37:28,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:28,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6837
[WARNING|trainer.py:803] 2025-04-26 21:37:29,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
5977
[WARNING|trainer.py:803] 2025-04-26 21:37:29,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6838
6013
[WARNING|trainer.py:803] 2025-04-26 21:37:30,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:30,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:30,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6839
5978
[WARNING|trainer.py:803] 2025-04-26 21:37:31,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6840
[WARNING|trainer.py:803] 2025-04-26 21:37:32,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6014
[WARNING|trainer.py:803] 2025-04-26 21:37:32,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:33,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6841
5979
[WARNING|trainer.py:803] 2025-04-26 21:37:33,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:34,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6015
6842
[WARNING|trainer.py:803] 2025-04-26 21:37:34,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:35,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6843
5980
6016
[WARNING|trainer.py:803] 2025-04-26 21:37:36,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:36,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6844
[WARNING|trainer.py:803] 2025-04-26 21:37:36,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:37,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6845
5981
6017
[WARNING|trainer.py:803] 2025-04-26 21:37:38,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:38,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6846
[WARNING|trainer.py:803] 2025-04-26 21:37:38,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:39,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
5982
6847
6018
[WARNING|trainer.py:803] 2025-04-26 21:37:40,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:40,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6848
[WARNING|trainer.py:803] 2025-04-26 21:37:40,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:41,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
5983
6849
6019
[WARNING|trainer.py:803] 2025-04-26 21:37:42,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:42,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:42,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6850
5984
[WARNING|trainer.py:803] 2025-04-26 21:37:43,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6020
6851
[WARNING|trainer.py:803] 2025-04-26 21:37:44,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:44,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:44,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6852
6021
5985
[WARNING|trainer.py:803] 2025-04-26 21:37:45,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6853
[WARNING|trainer.py:803] 2025-04-26 21:37:46,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:46,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:47,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6854
6022
[WARNING|trainer.py:803] 2025-04-26 21:37:48,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5986
6855
[WARNING|trainer.py:803] 2025-04-26 21:37:48,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:49,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:49,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6856
6023
5987
[WARNING|trainer.py:803] 2025-04-26 21:37:50,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6857
[WARNING|trainer.py:803] 2025-04-26 21:37:50,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:51,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:37:51,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6858
5988
6024
[WARNING|trainer.py:803] 2025-04-26 21:37:52,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6859
[WARNING|trainer.py:803] 2025-04-26 21:37:53,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:53,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:53,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6860
6025
5989
[WARNING|trainer.py:803] 2025-04-26 21:37:54,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6861
[WARNING|trainer.py:803] 2025-04-26 21:37:55,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:37:55,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:55,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6862
6026
5990
[WARNING|trainer.py:803] 2025-04-26 21:37:56,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6863
[WARNING|trainer.py:803] 2025-04-26 21:37:57,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:57,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:37:57,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6027
6864
5991
[WARNING|trainer.py:803] 2025-04-26 21:37:58,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:37:59,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6865
[WARNING|trainer.py:803] 2025-04-26 21:37:59,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:00,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6028
6866
5992
[WARNING|trainer.py:803] 2025-04-26 21:38:01,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:01,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6867
[WARNING|trainer.py:803] 2025-04-26 21:38:01,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:02,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6029
6868
5993
[WARNING|trainer.py:803] 2025-04-26 21:38:03,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:03,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:03,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6869
[WARNING|trainer.py:803] 2025-04-26 21:38:04,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6030
5994
6870
[WARNING|trainer.py:803] 2025-04-26 21:38:05,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:05,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:05,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6871
5995
[WARNING|trainer.py:803] 2025-04-26 21:38:06,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6031
6872
[WARNING|trainer.py:803] 2025-04-26 21:38:07,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:07,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:08,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6873
6032
5996
[WARNING|trainer.py:803] 2025-04-26 21:38:09,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6874
[WARNING|trainer.py:803] 2025-04-26 21:38:09,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:09,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:10,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6875
6033
5997
[WARNING|trainer.py:803] 2025-04-26 21:38:11,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:11,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6876
[WARNING|trainer.py:803] 2025-04-26 21:38:11,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:12,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6877
5998
6034
[WARNING|trainer.py:803] 2025-04-26 21:38:13,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:13,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:13,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6878
[WARNING|trainer.py:803] 2025-04-26 21:38:14,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
5999
6879
6035
[WARNING|trainer.py:803] 2025-04-26 21:38:15,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:15,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:15,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6880
6000
[WARNING|trainer.py:803] 2025-04-26 21:38:16,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6036
6881
[WARNING|trainer.py:803] 2025-04-26 21:38:17,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:17,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:18,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6882
6001
6037
[WARNING|trainer.py:803] 2025-04-26 21:38:19,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6883
[WARNING|trainer.py:803] 2025-04-26 21:38:19,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:19,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:20,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6884
6002
6038
[WARNING|trainer.py:803] 2025-04-26 21:38:21,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6885
[WARNING|trainer.py:803] 2025-04-26 21:38:21,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:21,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:22,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6886
6039
6003
[WARNING|trainer.py:803] 2025-04-26 21:38:23,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6887
[WARNING|trainer.py:803] 2025-04-26 21:38:23,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:24,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:24,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6040
6888
6004
[WARNING|trainer.py:803] 2025-04-26 21:38:25,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:25,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:25,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6889
[WARNING|trainer.py:803] 2025-04-26 21:38:26,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6041
6005
6890
[WARNING|trainer.py:803] 2025-04-26 21:38:27,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:27,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:28,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6891
6042
6006
[WARNING|trainer.py:803] 2025-04-26 21:38:29,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6892
[WARNING|trainer.py:803] 2025-04-26 21:38:29,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:29,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:30,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6893
6007
6043
[WARNING|trainer.py:803] 2025-04-26 21:38:31,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:31,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6894
[WARNING|trainer.py:803] 2025-04-26 21:38:31,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:32,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6895
6008
6044
[WARNING|trainer.py:803] 2025-04-26 21:38:33,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:33,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6896
[WARNING|trainer.py:803] 2025-04-26 21:38:34,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:34,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6009
6897
6045
[WARNING|trainer.py:803] 2025-04-26 21:38:35,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:35,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:35,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6898
6010
[WARNING|trainer.py:803] 2025-04-26 21:38:37,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6046
6899
[WARNING|trainer.py:803] 2025-04-26 21:38:37,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:37,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:38,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6900
6011
[WARNING|trainer.py:803] 2025-04-26 21:38:39,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6047
[WARNING|trainer.py:803] 2025-04-26 21:38:39,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6901
[WARNING|trainer.py:803] 2025-04-26 21:38:40,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:40,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6902
6012
[WARNING|trainer.py:803] 2025-04-26 21:38:41,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6048
6903
[WARNING|trainer.py:803] 2025-04-26 21:38:41,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:42,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:42,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6904
6013
6049
[WARNING|trainer.py:803] 2025-04-26 21:38:43,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:43,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6905
[WARNING|trainer.py:803] 2025-04-26 21:38:44,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:44,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6906
6014
[WARNING|trainer.py:803] 2025-04-26 21:38:45,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6050
[WARNING|trainer.py:803] 2025-04-26 21:38:45,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6907
[WARNING|trainer.py:803] 2025-04-26 21:38:46,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:38:46,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6908
6015
[WARNING|trainer.py:803] 2025-04-26 21:38:47,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:38:47,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6051
6909
[WARNING|trainer.py:803] 2025-04-26 21:38:48,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:38:48,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6016
6910
[WARNING|trainer.py:803] 2025-04-26 21:38:49,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:38:49,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6052
6911
[WARNING|trainer.py:803] 2025-04-26 21:38:50,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6017
[WARNING|trainer.py:803] 2025-04-26 21:38:50,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6912
[WARNING|trainer.py:803] 2025-04-26 21:38:51,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6053
[WARNING|trainer.py:803] 2025-04-26 21:38:51,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6913
[WARNING|trainer.py:803] 2025-04-26 21:38:52,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6018
[WARNING|trainer.py:803] 2025-04-26 21:38:52,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6914
[WARNING|trainer.py:803] 2025-04-26 21:38:53,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6054
[WARNING|trainer.py:803] 2025-04-26 21:38:53,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6915
[WARNING|trainer.py:803] 2025-04-26 21:38:54,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6019
[WARNING|trainer.py:803] 2025-04-26 21:38:54,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6916
[WARNING|trainer.py:803] 2025-04-26 21:38:55,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6055
[WARNING|trainer.py:803] 2025-04-26 21:38:55,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6917
[WARNING|trainer.py:803] 2025-04-26 21:38:56,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6020
[WARNING|trainer.py:803] 2025-04-26 21:38:56,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6918
[WARNING|trainer.py:803] 2025-04-26 21:38:57,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6056
[WARNING|trainer.py:803] 2025-04-26 21:38:57,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6919
[WARNING|trainer.py:803] 2025-04-26 21:38:58,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6021
[WARNING|trainer.py:803] 2025-04-26 21:38:58,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6920
[WARNING|trainer.py:803] 2025-04-26 21:38:59,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6057
[WARNING|trainer.py:803] 2025-04-26 21:38:59,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6921
[WARNING|trainer.py:803] 2025-04-26 21:39:00,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6022
[WARNING|trainer.py:803] 2025-04-26 21:39:00,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6922
[WARNING|trainer.py:803] 2025-04-26 21:39:01,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6058
[WARNING|trainer.py:803] 2025-04-26 21:39:01,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:02,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6923
6023
[WARNING|trainer.py:803] 2025-04-26 21:39:03,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6924
6059
[WARNING|trainer.py:803] 2025-04-26 21:39:03,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:04,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:04,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6925
6024
6060
[WARNING|trainer.py:803] 2025-04-26 21:39:05,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6926
[WARNING|trainer.py:803] 2025-04-26 21:39:05,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:05,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:06,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6927
6061
6025
[WARNING|trainer.py:803] 2025-04-26 21:39:07,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6928
[WARNING|trainer.py:803] 2025-04-26 21:39:07,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:07,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:08,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6929
6062
6026
[WARNING|trainer.py:803] 2025-04-26 21:39:09,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6930
[WARNING|trainer.py:803] 2025-04-26 21:39:09,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:09,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:10,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6931
6063
6027
[WARNING|trainer.py:803] 2025-04-26 21:39:11,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:11,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:11,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6932
[WARNING|trainer.py:803] 2025-04-26 21:39:12,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6064
6933
6028
[WARNING|trainer.py:803] 2025-04-26 21:39:13,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:13,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6934
[WARNING|trainer.py:803] 2025-04-26 21:39:13,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:14,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6935
6065
6029
[WARNING|trainer.py:803] 2025-04-26 21:39:15,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6936
[WARNING|trainer.py:803] 2025-04-26 21:39:15,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:16,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:16,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6937
6066
6030
[WARNING|trainer.py:803] 2025-04-26 21:39:17,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:17,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6938
[WARNING|trainer.py:803] 2025-04-26 21:39:18,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:18,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6067
6939
[WARNING|trainer.py:803] 2025-04-26 21:39:19,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6031
[WARNING|trainer.py:803] 2025-04-26 21:39:19,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6940
6068
[WARNING|trainer.py:803] 2025-04-26 21:39:20,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:20,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6941
[WARNING|trainer.py:803] 2025-04-26 21:39:21,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6032
[WARNING|trainer.py:803] 2025-04-26 21:39:21,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6942
[WARNING|trainer.py:803] 2025-04-26 21:39:22,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6069
[WARNING|trainer.py:803] 2025-04-26 21:39:22,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6943
[WARNING|trainer.py:803] 2025-04-26 21:39:23,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6033
[WARNING|trainer.py:803] 2025-04-26 21:39:23,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6944
[WARNING|trainer.py:803] 2025-04-26 21:39:24,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6070
[WARNING|trainer.py:803] 2025-04-26 21:39:24,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6945
[WARNING|trainer.py:803] 2025-04-26 21:39:25,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6034
[WARNING|trainer.py:803] 2025-04-26 21:39:25,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6946
[WARNING|trainer.py:803] 2025-04-26 21:39:26,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6071
[WARNING|trainer.py:803] 2025-04-26 21:39:27,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6947
[WARNING|trainer.py:803] 2025-04-26 21:39:27,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6035
[WARNING|trainer.py:803] 2025-04-26 21:39:28,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6948
[WARNING|trainer.py:803] 2025-04-26 21:39:28,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6072
[WARNING|trainer.py:803] 2025-04-26 21:39:29,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6949
[WARNING|trainer.py:803] 2025-04-26 21:39:29,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6036
[WARNING|trainer.py:803] 2025-04-26 21:39:30,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6950
6073
[WARNING|trainer.py:803] 2025-04-26 21:39:30,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:31,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6951
[WARNING|trainer.py:803] 2025-04-26 21:39:31,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6037
[WARNING|trainer.py:803] 2025-04-26 21:39:32,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6952
6074
[WARNING|trainer.py:803] 2025-04-26 21:39:32,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:33,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:33,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6953
6038
[WARNING|trainer.py:803] 2025-04-26 21:39:34,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6954
[WARNING|trainer.py:803] 2025-04-26 21:39:34,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6075
[WARNING|trainer.py:803] 2025-04-26 21:39:35,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6955
[WARNING|trainer.py:803] 2025-04-26 21:39:35,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6039
[WARNING|trainer.py:803] 2025-04-26 21:39:36,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6956
[WARNING|trainer.py:803] 2025-04-26 21:39:36,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6076
[WARNING|trainer.py:803] 2025-04-26 21:39:37,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6957
[WARNING|trainer.py:803] 2025-04-26 21:39:37,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6040
[WARNING|trainer.py:803] 2025-04-26 21:39:38,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:38,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6958
6077
[WARNING|trainer.py:803] 2025-04-26 21:39:39,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:39,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6041
6959
[WARNING|trainer.py:803] 2025-04-26 21:39:40,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:40,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6078
6960
[WARNING|trainer.py:803] 2025-04-26 21:39:41,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:41,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6042
6961
6079
[WARNING|trainer.py:803] 2025-04-26 21:39:42,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:42,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6962
[WARNING|trainer.py:803] 2025-04-26 21:39:43,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:43,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6043
6963
6080
[WARNING|trainer.py:803] 2025-04-26 21:39:44,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:44,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6964
[WARNING|trainer.py:803] 2025-04-26 21:39:45,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:45,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6044
6965
6081
[WARNING|trainer.py:803] 2025-04-26 21:39:46,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:46,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6966
[WARNING|trainer.py:803] 2025-04-26 21:39:47,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6045
[WARNING|trainer.py:803] 2025-04-26 21:39:48,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6967
6082
[WARNING|trainer.py:803] 2025-04-26 21:39:48,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:49,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:49,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6968
6046
[WARNING|trainer.py:803] 2025-04-26 21:39:50,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6969
6083
[WARNING|trainer.py:803] 2025-04-26 21:39:50,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:39:51,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6970
[WARNING|trainer.py:803] 2025-04-26 21:39:51,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:52,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6047
6971
6084
[WARNING|trainer.py:803] 2025-04-26 21:39:53,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:53,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6972
[WARNING|trainer.py:803] 2025-04-26 21:39:53,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6048
[WARNING|trainer.py:803] 2025-04-26 21:39:54,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6973
6085
[WARNING|trainer.py:803] 2025-04-26 21:39:55,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:39:55,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6974
[WARNING|trainer.py:803] 2025-04-26 21:39:55,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6049
[WARNING|trainer.py:803] 2025-04-26 21:39:56,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6975
[WARNING|trainer.py:803] 2025-04-26 21:39:57,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6086
[WARNING|trainer.py:803] 2025-04-26 21:39:57,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6976
[WARNING|trainer.py:803] 2025-04-26 21:39:58,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:39:58,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6050
6977
[WARNING|trainer.py:803] 2025-04-26 21:39:59,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:39:59,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6087
6978
[WARNING|trainer.py:803] 2025-04-26 21:40:00,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:00,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6051
6979
6088
[WARNING|trainer.py:803] 2025-04-26 21:40:01,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:01,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6980
[WARNING|trainer.py:803] 2025-04-26 21:40:02,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:02,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6052
6981
6089
[WARNING|trainer.py:803] 2025-04-26 21:40:03,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:40:03,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6982
[WARNING|trainer.py:803] 2025-04-26 21:40:04,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6053
[WARNING|trainer.py:803] 2025-04-26 21:40:04,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6983
6090
[WARNING|trainer.py:803] 2025-04-26 21:40:05,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:05,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:05,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6984
6054
[WARNING|trainer.py:803] 2025-04-26 21:40:06,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6985
6091
[WARNING|trainer.py:803] 2025-04-26 21:40:07,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:40:07,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:07,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6986
6055
[WARNING|trainer.py:803] 2025-04-26 21:40:08,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6987
6092
[WARNING|trainer.py:803] 2025-04-26 21:40:09,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:09,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6988
[WARNING|trainer.py:803] 2025-04-26 21:40:10,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6056
[WARNING|trainer.py:803] 2025-04-26 21:40:10,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6989
6093
[WARNING|trainer.py:803] 2025-04-26 21:40:11,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:11,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:40:12,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6990
6057
6094
[WARNING|trainer.py:803] 2025-04-26 21:40:13,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:13,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6991
[WARNING|trainer.py:803] 2025-04-26 21:40:13,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:14,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6058
6992
[WARNING|trainer.py:803] 2025-04-26 21:40:15,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:15,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6095
6993
[WARNING|trainer.py:803] 2025-04-26 21:40:16,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6059
[WARNING|trainer.py:803] 2025-04-26 21:40:16,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6994
[WARNING|trainer.py:803] 2025-04-26 21:40:17,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6096
[WARNING|trainer.py:803] 2025-04-26 21:40:17,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6995
[WARNING|trainer.py:803] 2025-04-26 21:40:18,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6060
[WARNING|trainer.py:803] 2025-04-26 21:40:18,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6996
[WARNING|trainer.py:803] 2025-04-26 21:40:18,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6097
[WARNING|trainer.py:803] 2025-04-26 21:40:19,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6997
6061
[WARNING|trainer.py:803] 2025-04-26 21:40:19,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:20,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6998
[WARNING|trainer.py:803] 2025-04-26 21:40:20,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6098
[WARNING|trainer.py:803] 2025-04-26 21:40:21,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6999
6062
[WARNING|trainer.py:803] 2025-04-26 21:40:22,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:22,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:22,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7000
6099
6063
[WARNING|trainer.py:803] 2025-04-26 21:40:23,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7001
[WARNING|trainer.py:803] 2025-04-26 21:40:24,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:24,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:24,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7002
6100
6064
[WARNING|trainer.py:803] 2025-04-26 21:40:26,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:26,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7003
[WARNING|trainer.py:803] 2025-04-26 21:40:26,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:27,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7004
6101
6065
[WARNING|trainer.py:803] 2025-04-26 21:40:28,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7005
[WARNING|trainer.py:803] 2025-04-26 21:40:28,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:28,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:29,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7006
6102
6066
[WARNING|trainer.py:803] 2025-04-26 21:40:30,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7007
[WARNING|trainer.py:803] 2025-04-26 21:40:30,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:30,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:31,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7008
6103
6067
[WARNING|trainer.py:803] 2025-04-26 21:40:32,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:40:32,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:32,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7009
[WARNING|trainer.py:803] 2025-04-26 21:40:33,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6068
7010
6104
[WARNING|trainer.py:803] 2025-04-26 21:40:34,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:34,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:34,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7011
[WARNING|trainer.py:803] 2025-04-26 21:40:35,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6069
7012
6105
[WARNING|trainer.py:803] 2025-04-26 21:40:36,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:36,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:36,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7013
[WARNING|trainer.py:803] 2025-04-26 21:40:37,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7014
6070
6106
[WARNING|trainer.py:803] 2025-04-26 21:40:38,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:38,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:38,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7015
[WARNING|trainer.py:803] 2025-04-26 21:40:39,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6071
7016
6107
[WARNING|trainer.py:803] 2025-04-26 21:40:40,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:40,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:40:40,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7017
[WARNING|trainer.py:803] 2025-04-26 21:40:41,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6072
7018
6108
[WARNING|trainer.py:803] 2025-04-26 21:40:42,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:42,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:42,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7019
6073
6109
[WARNING|trainer.py:803] 2025-04-26 21:40:43,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7020
[WARNING|trainer.py:803] 2025-04-26 21:40:44,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:44,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:44,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7021
6074
[WARNING|trainer.py:803] 2025-04-26 21:40:46,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6110
7022
[WARNING|trainer.py:803] 2025-04-26 21:40:46,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:46,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:47,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7023
6111
[WARNING|trainer.py:803] 2025-04-26 21:40:48,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6075
7024
[WARNING|trainer.py:803] 2025-04-26 21:40:48,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:49,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:49,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7025
6112
6076
[WARNING|trainer.py:803] 2025-04-26 21:40:50,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7026
[WARNING|trainer.py:803] 2025-04-26 21:40:50,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:50,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:51,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7027
6077
6113
[WARNING|trainer.py:803] 2025-04-26 21:40:52,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7028
[WARNING|trainer.py:803] 2025-04-26 21:40:52,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:52,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:53,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7029
6078
[WARNING|trainer.py:803] 2025-04-26 21:40:54,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6114
7030
[WARNING|trainer.py:803] 2025-04-26 21:40:54,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:55,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:40:55,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7031
6079
[WARNING|trainer.py:803] 2025-04-26 21:40:56,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6115
7032
[WARNING|trainer.py:803] 2025-04-26 21:40:56,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:57,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:40:57,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7033
6080
6116
[WARNING|trainer.py:803] 2025-04-26 21:40:58,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7034
[WARNING|trainer.py:803] 2025-04-26 21:40:58,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:59,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:40:59,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7035
6081
6117
[WARNING|trainer.py:803] 2025-04-26 21:41:00,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:00,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7036
[WARNING|trainer.py:803] 2025-04-26 21:41:01,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:01,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6082
7037
[WARNING|trainer.py:803] 2025-04-26 21:41:02,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6118
[WARNING|trainer.py:803] 2025-04-26 21:41:02,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7038
[WARNING|trainer.py:803] 2025-04-26 21:41:03,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:03,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7039
6083
[WARNING|trainer.py:803] 2025-04-26 21:41:04,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6119
7040
[WARNING|trainer.py:803] 2025-04-26 21:41:05,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:05,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:05,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7041
6084
[WARNING|trainer.py:803] 2025-04-26 21:41:06,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6120
7042
[WARNING|trainer.py:803] 2025-04-26 21:41:07,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:07,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:07,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7043
6085
[WARNING|trainer.py:803] 2025-04-26 21:41:09,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6121
7044
[WARNING|trainer.py:803] 2025-04-26 21:41:09,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:10,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:10,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7045
6086
[WARNING|trainer.py:803] 2025-04-26 21:41:11,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7046
[WARNING|trainer.py:803] 2025-04-26 21:41:11,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6122
[WARNING|trainer.py:803] 2025-04-26 21:41:12,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7047
[WARNING|trainer.py:803] 2025-04-26 21:41:12,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6087
[WARNING|trainer.py:803] 2025-04-26 21:41:13,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7048
6123
[WARNING|trainer.py:803] 2025-04-26 21:41:13,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:14,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:14,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7049
6088
[WARNING|trainer.py:803] 2025-04-26 21:41:15,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7050
[WARNING|trainer.py:803] 2025-04-26 21:41:15,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
 Yes
6124
[WARNING|trainer.py:803] 2025-04-26 21:41:16,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:16,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6089
7051
[WARNING|trainer.py:803] 2025-04-26 21:41:17,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:17,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7052
6125
6090
[WARNING|trainer.py:803] 2025-04-26 21:41:18,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7053
[WARNING|trainer.py:803] 2025-04-26 21:41:18,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:19,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:19,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7054
6126
6091
[WARNING|trainer.py:803] 2025-04-26 21:41:20,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7055
[WARNING|trainer.py:803] 2025-04-26 21:41:20,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:21,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:21,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7056
6127
[WARNING|trainer.py:803] 2025-04-26 21:41:22,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6092
7057
[WARNING|trainer.py:803] 2025-04-26 21:41:23,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:23,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:23,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7058
6128
6093
[WARNING|trainer.py:803] 2025-04-26 21:41:24,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7059
[WARNING|trainer.py:803] 2025-04-26 21:41:24,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:25,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:25,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7060
6129
6094
[WARNING|trainer.py:803] 2025-04-26 21:41:26,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7061
[WARNING|trainer.py:803] 2025-04-26 21:41:26,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:27,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:27,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7062
6130
6095
[WARNING|trainer.py:803] 2025-04-26 21:41:28,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7063
[WARNING|trainer.py:803] 2025-04-26 21:41:29,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:29,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:29,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7064
6131
6096
[WARNING|trainer.py:803] 2025-04-26 21:41:30,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:30,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7065
[WARNING|trainer.py:803] 2025-04-26 21:41:31,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:31,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7066
6132
6097
[WARNING|trainer.py:803] 2025-04-26 21:41:32,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:32,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7067
[WARNING|trainer.py:803] 2025-04-26 21:41:33,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:33,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7068
6133
6098
[WARNING|trainer.py:803] 2025-04-26 21:41:34,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:35,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7069
[WARNING|trainer.py:803] 2025-04-26 21:41:35,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:35,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6134
7070
6099
[WARNING|trainer.py:803] 2025-04-26 21:41:36,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:36,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7071
[WARNING|trainer.py:803] 2025-04-26 21:41:37,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:38,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7072
6135
6100
[WARNING|trainer.py:803] 2025-04-26 21:41:39,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7073
[WARNING|trainer.py:803] 2025-04-26 21:41:39,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:39,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:40,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7074
6136
6101
[WARNING|trainer.py:803] 2025-04-26 21:41:41,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7075
[WARNING|trainer.py:803] 2025-04-26 21:41:41,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:41,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:42,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7076
6137
6102
[WARNING|trainer.py:803] 2025-04-26 21:41:43,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:43,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7077
[WARNING|trainer.py:803] 2025-04-26 21:41:43,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:44,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7078
6138
6103
[WARNING|trainer.py:803] 2025-04-26 21:41:45,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:45,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:45,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7079
[WARNING|trainer.py:803] 2025-04-26 21:41:46,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7080
6139
6104
[WARNING|trainer.py:803] 2025-04-26 21:41:47,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:47,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7081
[WARNING|trainer.py:803] 2025-04-26 21:41:47,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:48,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7082
6105
6140
[WARNING|trainer.py:803] 2025-04-26 21:41:49,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:41:49,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7083
[WARNING|trainer.py:803] 2025-04-26 21:41:49,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:50,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7084
6106
6141
[WARNING|trainer.py:803] 2025-04-26 21:41:51,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 21:41:51,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7085
[WARNING|trainer.py:803] 2025-04-26 21:41:51,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:52,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7086
6107
6142
[WARNING|trainer.py:803] 2025-04-26 21:41:53,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:41:53,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7087
[WARNING|trainer.py:803] 2025-04-26 21:41:53,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:54,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7088
6108
6143
[WARNING|trainer.py:803] 2025-04-26 21:41:55,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7089
[WARNING|trainer.py:803] 2025-04-26 21:41:55,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:56,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:56,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6109
7090
6144
[WARNING|trainer.py:803] 2025-04-26 21:41:57,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:41:57,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7091
[WARNING|trainer.py:803] 2025-04-26 21:41:58,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:41:58,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7092
6110
6145
[WARNING|trainer.py:803] 2025-04-26 21:41:59,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:00,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7093
[WARNING|trainer.py:803] 2025-04-26 21:42:00,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6111
[WARNING|trainer.py:803] 2025-04-26 21:42:00,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7094
6146
[WARNING|trainer.py:803] 2025-04-26 21:42:01,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:02,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:02,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7095
6112
[WARNING|trainer.py:803] 2025-04-26 21:42:03,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7096
6147
[WARNING|trainer.py:803] 2025-04-26 21:42:03,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:04,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7097
[WARNING|trainer.py:803] 2025-04-26 21:42:04,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6113
[WARNING|trainer.py:803] 2025-04-26 21:42:05,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7098
6148
[WARNING|trainer.py:803] 2025-04-26 21:42:05,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:06,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7099
[WARNING|trainer.py:803] 2025-04-26 21:42:06,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:07,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6114
7100
6149
[WARNING|trainer.py:803] 2025-04-26 21:42:08,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:08,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7101
[WARNING|trainer.py:803] 2025-04-26 21:42:08,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:09,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6115
7102
6150
[WARNING|trainer.py:803] 2025-04-26 21:42:10,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:10,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7103
[WARNING|trainer.py:803] 2025-04-26 21:42:10,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6116
[WARNING|trainer.py:803] 2025-04-26 21:42:11,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7104
6151
[WARNING|trainer.py:803] 2025-04-26 21:42:12,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:12,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7105
[WARNING|trainer.py:803] 2025-04-26 21:42:12,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6117
[WARNING|trainer.py:803] 2025-04-26 21:42:13,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7106
6152
[WARNING|trainer.py:803] 2025-04-26 21:42:14,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:14,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7107
[WARNING|trainer.py:803] 2025-04-26 21:42:15,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:15,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7108
6118
6153
[WARNING|trainer.py:803] 2025-04-26 21:42:16,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:16,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7109
[WARNING|trainer.py:803] 2025-04-26 21:42:17,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:17,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7110
6119
6154
[WARNING|trainer.py:803] 2025-04-26 21:42:18,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7111
[WARNING|trainer.py:803] 2025-04-26 21:42:18,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:19,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:19,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7112
6120
6155
[WARNING|trainer.py:803] 2025-04-26 21:42:20,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7113
[WARNING|trainer.py:803] 2025-04-26 21:42:21,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:21,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:21,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7114
6121
6156
[WARNING|trainer.py:803] 2025-04-26 21:42:22,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7115
[WARNING|trainer.py:803] 2025-04-26 21:42:23,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:23,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:23,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7116
6157
[WARNING|trainer.py:803] 2025-04-26 21:42:24,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6122
7117
[WARNING|trainer.py:803] 2025-04-26 21:42:25,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:25,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:25,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7118
6123
[WARNING|trainer.py:803] 2025-04-26 21:42:26,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6158
7119
[WARNING|trainer.py:803] 2025-04-26 21:42:27,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:27,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:27,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7120
6124
6159
[WARNING|trainer.py:803] 2025-04-26 21:42:28,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7121
[WARNING|trainer.py:803] 2025-04-26 21:42:29,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:29,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:29,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7122
6160
[WARNING|trainer.py:803] 2025-04-26 21:42:30,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6125
7123
[WARNING|trainer.py:803] 2025-04-26 21:42:31,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:31,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:32,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7124
6161
[WARNING|trainer.py:803] 2025-04-26 21:42:33,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6126
7125
[WARNING|trainer.py:803] 2025-04-26 21:42:33,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:33,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:34,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7126
6162
[WARNING|trainer.py:803] 2025-04-26 21:42:35,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6127
7127
[WARNING|trainer.py:803] 2025-04-26 21:42:35,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:36,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:36,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7128
6163
6128
[WARNING|trainer.py:803] 2025-04-26 21:42:37,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7129
[WARNING|trainer.py:803] 2025-04-26 21:42:37,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:38,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:38,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7130
6164
6129
[WARNING|trainer.py:803] 2025-04-26 21:42:39,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7131
[WARNING|trainer.py:803] 2025-04-26 21:42:39,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:40,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:40,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7132
6165
6130
[WARNING|trainer.py:803] 2025-04-26 21:42:41,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7133
[WARNING|trainer.py:803] 2025-04-26 21:42:41,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:42,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:42,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7134
6166
6131
[WARNING|trainer.py:803] 2025-04-26 21:42:43,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:43,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7135
[WARNING|trainer.py:803] 2025-04-26 21:42:43,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:44,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7136
6167
6132
[WARNING|trainer.py:803] 2025-04-26 21:42:45,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:45,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7137
[WARNING|trainer.py:803] 2025-04-26 21:42:45,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:46,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7138
6168
6133
[WARNING|trainer.py:803] 2025-04-26 21:42:47,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:47,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7139
[WARNING|trainer.py:803] 2025-04-26 21:42:47,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:48,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6169
7140
6134
[WARNING|trainer.py:803] 2025-04-26 21:42:49,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:49,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:49,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7141
6170
[WARNING|trainer.py:803] 2025-04-26 21:42:50,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7142
6135
[WARNING|trainer.py:803] 2025-04-26 21:42:51,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:51,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7143
[WARNING|trainer.py:803] 2025-04-26 21:42:52,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6171
[WARNING|trainer.py:803] 2025-04-26 21:42:52,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7144
[WARNING|trainer.py:803] 2025-04-26 21:42:53,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6136
[WARNING|trainer.py:803] 2025-04-26 21:42:53,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7145
6172
[WARNING|trainer.py:803] 2025-04-26 21:42:54,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:42:54,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7146
[WARNING|trainer.py:803] 2025-04-26 21:42:55,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6137
[WARNING|trainer.py:803] 2025-04-26 21:42:55,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7147
6173
[WARNING|trainer.py:803] 2025-04-26 21:42:56,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:56,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:42:57,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7148
6138
[WARNING|trainer.py:803] 2025-04-26 21:42:58,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6174
7149
[WARNING|trainer.py:803] 2025-04-26 21:42:58,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:42:58,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:42:59,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7150
6139
6175
[WARNING|trainer.py:803] 2025-04-26 21:43:00,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7151
[WARNING|trainer.py:803] 2025-04-26 21:43:00,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:00,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:01,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7152
6140
[WARNING|trainer.py:803] 2025-04-26 21:43:02,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6176
7153
[WARNING|trainer.py:803] 2025-04-26 21:43:02,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:03,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:03,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7154
6177
6141
[WARNING|trainer.py:803] 2025-04-26 21:43:04,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7155
[WARNING|trainer.py:803] 2025-04-26 21:43:04,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:04,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:05,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7156
6178
6142
[WARNING|trainer.py:803] 2025-04-26 21:43:06,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7157
[WARNING|trainer.py:803] 2025-04-26 21:43:07,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 21:43:07,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:07,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7158
[WARNING|trainer.py:803] 2025-04-26 21:43:08,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6179
6143
7159
[WARNING|trainer.py:803] 2025-04-26 21:43:09,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:09,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:09,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7160
6144
[WARNING|trainer.py:803] 2025-04-26 21:43:10,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6180
7161
[WARNING|trainer.py:803] 2025-04-26 21:43:11,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:11,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:11,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7162
[WARNING|trainer.py:803] 2025-04-26 21:43:12,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6181
6145
7163
[WARNING|trainer.py:803] 2025-04-26 21:43:13,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:13,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:13,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7164
6146
[WARNING|trainer.py:803] 2025-04-26 21:43:14,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6182
7165
[WARNING|trainer.py:803] 2025-04-26 21:43:15,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:15,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:15,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7166
[WARNING|trainer.py:803] 2025-04-26 21:43:16,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6183
7167
6147
[WARNING|trainer.py:803] 2025-04-26 21:43:17,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:17,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:17,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7168
6184
[WARNING|trainer.py:803] 2025-04-26 21:43:18,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6148
7169
[WARNING|trainer.py:803] 2025-04-26 21:43:19,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:19,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:19,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7170
[WARNING|trainer.py:803] 2025-04-26 21:43:20,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6149
6185
7171
[WARNING|trainer.py:803] 2025-04-26 21:43:21,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:21,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:21,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7172
6186
[WARNING|trainer.py:803] 2025-04-26 21:43:23,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7173
6150
[WARNING|trainer.py:803] 2025-04-26 21:43:23,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:24,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:24,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7174
6187
[WARNING|trainer.py:803] 2025-04-26 21:43:25,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6151
7175
[WARNING|trainer.py:803] 2025-04-26 21:43:25,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:26,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:26,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7176
6188
[WARNING|trainer.py:803] 2025-04-26 21:43:27,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7177
6152
[WARNING|trainer.py:803] 2025-04-26 21:43:27,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:28,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7178
[WARNING|trainer.py:803] 2025-04-26 21:43:28,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6189
[WARNING|trainer.py:803] 2025-04-26 21:43:29,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7179
6153
[WARNING|trainer.py:803] 2025-04-26 21:43:29,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:30,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7180
[WARNING|trainer.py:803] 2025-04-26 21:43:30,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6190
[WARNING|trainer.py:803] 2025-04-26 21:43:31,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7181
6154
[WARNING|trainer.py:803] 2025-04-26 21:43:31,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:32,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7182
[WARNING|trainer.py:803] 2025-04-26 21:43:32,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6191
[WARNING|trainer.py:803] 2025-04-26 21:43:33,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7183
[WARNING|trainer.py:803] 2025-04-26 21:43:33,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6155
[WARNING|trainer.py:803] 2025-04-26 21:43:34,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7184
[WARNING|trainer.py:803] 2025-04-26 21:43:34,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6192
[WARNING|trainer.py:803] 2025-04-26 21:43:35,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7185
6156
[WARNING|trainer.py:803] 2025-04-26 21:43:36,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:36,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7186
[WARNING|trainer.py:803] 2025-04-26 21:43:36,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6193
[WARNING|trainer.py:803] 2025-04-26 21:43:37,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7187
6157
[WARNING|trainer.py:803] 2025-04-26 21:43:38,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:38,495 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:38,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7188
6194
[WARNING|trainer.py:803] 2025-04-26 21:43:39,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7189
6158
[WARNING|trainer.py:803] 2025-04-26 21:43:40,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:40,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7190
[WARNING|trainer.py:803] 2025-04-26 21:43:41,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6195
[WARNING|trainer.py:803] 2025-04-26 21:43:41,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7191
6159
[WARNING|trainer.py:803] 2025-04-26 21:43:42,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:42,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7192
[WARNING|trainer.py:803] 2025-04-26 21:43:42,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:43,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6196
7193
6160
[WARNING|trainer.py:803] 2025-04-26 21:43:44,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:44,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7194
[WARNING|trainer.py:803] 2025-04-26 21:43:45,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:45,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7195
6197
6161
[WARNING|trainer.py:803] 2025-04-26 21:43:46,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7196
[WARNING|trainer.py:803] 2025-04-26 21:43:47,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:47,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:47,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7197
6162
6198
[WARNING|trainer.py:803] 2025-04-26 21:43:48,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:48,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:49,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7198
[WARNING|trainer.py:803] 2025-04-26 21:43:49,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6199
7199
6163
[WARNING|trainer.py:803] 2025-04-26 21:43:50,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:50,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:51,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7200
[WARNING|trainer.py:803] 2025-04-26 21:43:52,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6200
7201
6164
[WARNING|trainer.py:803] 2025-04-26 21:43:52,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:53,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7202
[WARNING|trainer.py:803] 2025-04-26 21:43:53,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:54,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6201
7203
6165
[WARNING|trainer.py:803] 2025-04-26 21:43:55,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:55,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:43:55,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7204
[WARNING|trainer.py:803] 2025-04-26 21:43:56,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6166
7205
6202
[WARNING|trainer.py:803] 2025-04-26 21:43:56,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:57,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:43:57,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7206
[WARNING|trainer.py:803] 2025-04-26 21:43:58,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6167
6203
7207
[WARNING|trainer.py:803] 2025-04-26 21:43:59,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:43:59,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:43:59,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7208
6204
6168
[WARNING|trainer.py:803] 2025-04-26 21:44:00,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7209
[WARNING|trainer.py:803] 2025-04-26 21:44:01,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:01,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:01,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7210
6205
6169
[WARNING|trainer.py:803] 2025-04-26 21:44:02,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7211
[WARNING|trainer.py:803] 2025-04-26 21:44:02,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:03,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:03,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7212
6206
6170
[WARNING|trainer.py:803] 2025-04-26 21:44:04,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:44:04,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7213
[WARNING|trainer.py:803] 2025-04-26 21:44:04,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:05,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7214
6207
6171
[WARNING|trainer.py:803] 2025-04-26 21:44:06,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:06,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:06,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7215
[WARNING|trainer.py:803] 2025-04-26 21:44:07,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7216
6172
6208
[WARNING|trainer.py:803] 2025-04-26 21:44:08,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:08,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7217
[WARNING|trainer.py:803] 2025-04-26 21:44:08,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:09,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6173
7218
6209
[WARNING|trainer.py:803] 2025-04-26 21:44:10,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:10,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7219
[WARNING|trainer.py:803] 2025-04-26 21:44:11,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6174
[WARNING|trainer.py:803] 2025-04-26 21:44:11,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7220
6210
[WARNING|trainer.py:803] 2025-04-26 21:44:12,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:12,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:12,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7221
6175
[WARNING|trainer.py:803] 2025-04-26 21:44:13,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6211
7222
[WARNING|trainer.py:803] 2025-04-26 21:44:14,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:14,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:14,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7223
6176
6212
[WARNING|trainer.py:803] 2025-04-26 21:44:15,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7224
[WARNING|trainer.py:803] 2025-04-26 21:44:16,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:16,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:16,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7225
6213
6177
[WARNING|trainer.py:803] 2025-04-26 21:44:17,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7226
[WARNING|trainer.py:803] 2025-04-26 21:44:18,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:18,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:18,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7227
6214
6178
[WARNING|trainer.py:803] 2025-04-26 21:44:19,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7228
[WARNING|trainer.py:803] 2025-04-26 21:44:20,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:20,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 21:44:20,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7229
6215
6179
[WARNING|trainer.py:803] 2025-04-26 21:44:21,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7230
[WARNING|trainer.py:803] 2025-04-26 21:44:22,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:22,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:22,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7231
6216
[WARNING|trainer.py:803] 2025-04-26 21:44:23,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6180
7232
[WARNING|trainer.py:803] 2025-04-26 21:44:24,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:24,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:25,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6217
7233
[WARNING|trainer.py:803] 2025-04-26 21:44:26,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:26,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6181
7234
[WARNING|trainer.py:803] 2025-04-26 21:44:26,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:27,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6218
7235
[WARNING|trainer.py:803] 2025-04-26 21:44:27,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:28,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6182
7236
[WARNING|trainer.py:803] 2025-04-26 21:44:28,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6219
[WARNING|trainer.py:803] 2025-04-26 21:44:29,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7237
[WARNING|trainer.py:803] 2025-04-26 21:44:29,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:30,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6183
7238
6220
[WARNING|trainer.py:803] 2025-04-26 21:44:31,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:31,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7239
[WARNING|trainer.py:803] 2025-04-26 21:44:31,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6184
[WARNING|trainer.py:803] 2025-04-26 21:44:32,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7240
6221
[WARNING|trainer.py:803] 2025-04-26 21:44:32,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:33,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7241
[WARNING|trainer.py:803] 2025-04-26 21:44:33,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6185
[WARNING|trainer.py:803] 2025-04-26 21:44:34,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7242
6222
[WARNING|trainer.py:803] 2025-04-26 21:44:35,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:35,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:35,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7243
6186
[WARNING|trainer.py:803] 2025-04-26 21:44:36,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7244
6223
[WARNING|trainer.py:803] 2025-04-26 21:44:36,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:37,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:37,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7245
6187
6224
[WARNING|trainer.py:803] 2025-04-26 21:44:38,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7246
[WARNING|trainer.py:803] 2025-04-26 21:44:39,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:39,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:39,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7247
6225
6188
[WARNING|trainer.py:803] 2025-04-26 21:44:40,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7248
[WARNING|trainer.py:803] 2025-04-26 21:44:41,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:41,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:41,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7249
6226
6189
[WARNING|trainer.py:803] 2025-04-26 21:44:42,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7250
[WARNING|trainer.py:803] 2025-04-26 21:44:42,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:43,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:44:43,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7251
6227
6190
[WARNING|trainer.py:803] 2025-04-26 21:44:44,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:44,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7252
[WARNING|trainer.py:803] 2025-04-26 21:44:45,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:45,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6228
7253
6191
[WARNING|trainer.py:803] 2025-04-26 21:44:46,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:46,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7254
[WARNING|trainer.py:803] 2025-04-26 21:44:47,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6229
[WARNING|trainer.py:803] 2025-04-26 21:44:47,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7255
[WARNING|trainer.py:803] 2025-04-26 21:44:48,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6192
[WARNING|trainer.py:803] 2025-04-26 21:44:48,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7256
[WARNING|trainer.py:803] 2025-04-26 21:44:49,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6230
[WARNING|trainer.py:803] 2025-04-26 21:44:49,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7257
[WARNING|trainer.py:803] 2025-04-26 21:44:50,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6193
[WARNING|trainer.py:803] 2025-04-26 21:44:50,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7258
6231
[WARNING|trainer.py:803] 2025-04-26 21:44:51,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:51,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7259
[WARNING|trainer.py:803] 2025-04-26 21:44:52,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6194
[WARNING|trainer.py:803] 2025-04-26 21:44:52,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7260
6232
[WARNING|trainer.py:803] 2025-04-26 21:44:53,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:44:53,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7261
[WARNING|trainer.py:803] 2025-04-26 21:44:54,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6195
[WARNING|trainer.py:803] 2025-04-26 21:44:54,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6233
7262
[WARNING|trainer.py:803] 2025-04-26 21:44:55,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:55,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:56,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7263
6234
[WARNING|trainer.py:803] 2025-04-26 21:44:57,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7264
6196
[WARNING|trainer.py:803] 2025-04-26 21:44:57,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:44:58,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:44:58,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7265
6235
[WARNING|trainer.py:803] 2025-04-26 21:44:59,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7266
[WARNING|trainer.py:803] 2025-04-26 21:44:59,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6197
[WARNING|trainer.py:803] 2025-04-26 21:45:00,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7267
[WARNING|trainer.py:803] 2025-04-26 21:45:00,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6236
[WARNING|trainer.py:803] 2025-04-26 21:45:01,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:01,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7268
6198
[WARNING|trainer.py:803] 2025-04-26 21:45:02,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7269
[WARNING|trainer.py:803] 2025-04-26 21:45:02,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6237
[WARNING|trainer.py:803] 2025-04-26 21:45:03,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6199
[WARNING|trainer.py:803] 2025-04-26 21:45:03,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7270
[WARNING|trainer.py:803] 2025-04-26 21:45:04,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:04,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7271
6238
[WARNING|trainer.py:803] 2025-04-26 21:45:05,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:05,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6200
7272
[WARNING|trainer.py:803] 2025-04-26 21:45:06,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:06,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6239
7273
[WARNING|trainer.py:803] 2025-04-26 21:45:07,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:07,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6201
7274
6240
[WARNING|trainer.py:803] 2025-04-26 21:45:08,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:08,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7275
[WARNING|trainer.py:803] 2025-04-26 21:45:09,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:09,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7276
6202
6241
[WARNING|trainer.py:803] 2025-04-26 21:45:10,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:10,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7277
[WARNING|trainer.py:803] 2025-04-26 21:45:10,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:11,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6203
7278
6242
[WARNING|trainer.py:803] 2025-04-26 21:45:12,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:12,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:12,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7279
6204
[WARNING|trainer.py:803] 2025-04-26 21:45:13,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6243
7280
[WARNING|trainer.py:803] 2025-04-26 21:45:14,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:14,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:14,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7281
6205
6244
[WARNING|trainer.py:803] 2025-04-26 21:45:15,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7282
[WARNING|trainer.py:803] 2025-04-26 21:45:15,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:16,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:45:16,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7283
6206
6245
[WARNING|trainer.py:803] 2025-04-26 21:45:17,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:17,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7284
[WARNING|trainer.py:803] 2025-04-26 21:45:18,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:18,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7285
6207
6246
[WARNING|trainer.py:803] 2025-04-26 21:45:19,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:19,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:19,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7286
[WARNING|trainer.py:803] 2025-04-26 21:45:20,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6247
7287
6208
[WARNING|trainer.py:803] 2025-04-26 21:45:21,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:21,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7288
[WARNING|trainer.py:803] 2025-04-26 21:45:22,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6248
[WARNING|trainer.py:803] 2025-04-26 21:45:22,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7289
6209
[WARNING|trainer.py:803] 2025-04-26 21:45:23,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:23,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7290
[WARNING|trainer.py:803] 2025-04-26 21:45:24,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6249
[WARNING|trainer.py:803] 2025-04-26 21:45:24,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6210
7291
[WARNING|trainer.py:803] 2025-04-26 21:45:25,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:25,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:26,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7292
6250
6211
[WARNING|trainer.py:803] 2025-04-26 21:45:27,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7293
[WARNING|trainer.py:803] 2025-04-26 21:45:27,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:27,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:28,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7294
6251
6212
[WARNING|trainer.py:803] 2025-04-26 21:45:29,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:29,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7295
[WARNING|trainer.py:803] 2025-04-26 21:45:29,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:30,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6252
7296
6213
[WARNING|trainer.py:803] 2025-04-26 21:45:31,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:31,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7297
[WARNING|trainer.py:803] 2025-04-26 21:45:31,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6253
[WARNING|trainer.py:803] 2025-04-26 21:45:32,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7298
6214
[WARNING|trainer.py:803] 2025-04-26 21:45:32,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:33,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:33,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7299
6254
[WARNING|trainer.py:803] 2025-04-26 21:45:34,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7300
6215
[WARNING|trainer.py:803] 2025-04-26 21:45:34,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:35,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:35,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6255
7301
[WARNING|trainer.py:803] 2025-04-26 21:45:36,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:36,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6216
7302
[WARNING|trainer.py:803] 2025-04-26 21:45:37,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6256
[WARNING|trainer.py:803] 2025-04-26 21:45:37,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7303
[WARNING|trainer.py:803] 2025-04-26 21:45:38,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6217
[WARNING|trainer.py:803] 2025-04-26 21:45:38,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7304
6257
[WARNING|trainer.py:803] 2025-04-26 21:45:39,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:39,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7305
[WARNING|trainer.py:803] 2025-04-26 21:45:39,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6218
[WARNING|trainer.py:803] 2025-04-26 21:45:40,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7306
6258
[WARNING|trainer.py:803] 2025-04-26 21:45:41,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:41,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:41,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7307
6219
[WARNING|trainer.py:803] 2025-04-26 21:45:42,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7308
6259
[WARNING|trainer.py:803] 2025-04-26 21:45:42,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:43,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:43,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7309
6220
[WARNING|trainer.py:803] 2025-04-26 21:45:44,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6260
7310
[WARNING|trainer.py:803] 2025-04-26 21:45:45,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:45,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:45,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7311
6221
6261
[WARNING|trainer.py:803] 2025-04-26 21:45:46,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:45:46,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7312
[WARNING|trainer.py:803] 2025-04-26 21:45:47,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:47,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6222
7313
[WARNING|trainer.py:803] 2025-04-26 21:45:48,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6262
[WARNING|trainer.py:803] 2025-04-26 21:45:48,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7314
[WARNING|trainer.py:803] 2025-04-26 21:45:49,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:45:49,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6223
7315
6263
[WARNING|trainer.py:803] 2025-04-26 21:45:50,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:45:50,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7316
[WARNING|trainer.py:803] 2025-04-26 21:45:51,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6224
[WARNING|trainer.py:803] 2025-04-26 21:45:51,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7317
[WARNING|trainer.py:803] 2025-04-26 21:45:52,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6264
[WARNING|trainer.py:803] 2025-04-26 21:45:53,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7318
[WARNING|trainer.py:803] 2025-04-26 21:45:53,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6225
[WARNING|trainer.py:803] 2025-04-26 21:45:54,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7319
[WARNING|trainer.py:803] 2025-04-26 21:45:54,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6265
[WARNING|trainer.py:803] 2025-04-26 21:45:55,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7320
[WARNING|trainer.py:803] 2025-04-26 21:45:55,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6226
[WARNING|trainer.py:803] 2025-04-26 21:45:56,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:56,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7321
6266
6227
[WARNING|trainer.py:803] 2025-04-26 21:45:57,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:57,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7322
[WARNING|trainer.py:803] 2025-04-26 21:45:57,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:45:58,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6267
7323
6228
[WARNING|trainer.py:803] 2025-04-26 21:45:59,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:45:59,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7324
[WARNING|trainer.py:803] 2025-04-26 21:45:59,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6268
[WARNING|trainer.py:803] 2025-04-26 21:46:00,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7325
6229
[WARNING|trainer.py:803] 2025-04-26 21:46:01,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:01,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7326
[WARNING|trainer.py:803] 2025-04-26 21:46:01,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6269
[WARNING|trainer.py:803] 2025-04-26 21:46:02,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7327
6230
[WARNING|trainer.py:803] 2025-04-26 21:46:02,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:03,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:03,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7328
6270
[WARNING|trainer.py:803] 2025-04-26 21:46:04,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7329
6231
[WARNING|trainer.py:803] 2025-04-26 21:46:04,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:05,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:05,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7330
6271
6232
[WARNING|trainer.py:803] 2025-04-26 21:46:06,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7331
[WARNING|trainer.py:803] 2025-04-26 21:46:06,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:07,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:07,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7332
6233
6272
[WARNING|trainer.py:803] 2025-04-26 21:46:08,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7333
[WARNING|trainer.py:803] 2025-04-26 21:46:08,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:09,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:09,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7334
6234
6273
[WARNING|trainer.py:803] 2025-04-26 21:46:10,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:10,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7335
[WARNING|trainer.py:803] 2025-04-26 21:46:11,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:11,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6235
7336
6274
[WARNING|trainer.py:803] 2025-04-26 21:46:12,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:12,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7337
[WARNING|trainer.py:803] 2025-04-26 21:46:13,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6236
[WARNING|trainer.py:803] 2025-04-26 21:46:13,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7338
[WARNING|trainer.py:803] 2025-04-26 21:46:14,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6275
[WARNING|trainer.py:803] 2025-04-26 21:46:14,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7339
[WARNING|trainer.py:803] 2025-04-26 21:46:15,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6237
[WARNING|trainer.py:803] 2025-04-26 21:46:15,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7340
6276
[WARNING|trainer.py:803] 2025-04-26 21:46:16,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:16,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7341
[WARNING|trainer.py:803] 2025-04-26 21:46:17,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6238
[WARNING|trainer.py:803] 2025-04-26 21:46:17,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7342
6277
[WARNING|trainer.py:803] 2025-04-26 21:46:18,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:18,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7343
[WARNING|trainer.py:803] 2025-04-26 21:46:19,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6239
[WARNING|trainer.py:803] 2025-04-26 21:46:19,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7344
6278
[WARNING|trainer.py:803] 2025-04-26 21:46:20,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:20,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:21,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7345
6240
6279
[WARNING|trainer.py:803] 2025-04-26 21:46:22,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7346
[WARNING|trainer.py:803] 2025-04-26 21:46:22,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:22,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:23,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7347
6241
6280
[WARNING|trainer.py:803] 2025-04-26 21:46:24,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:24,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7348
[WARNING|trainer.py:803] 2025-04-26 21:46:24,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:25,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6242
7349
[WARNING|trainer.py:803] 2025-04-26 21:46:26,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:26,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6281
7350
[WARNING|trainer.py:803] 2025-04-26 21:46:27,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6243
[WARNING|trainer.py:803] 2025-04-26 21:46:27,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7351
[WARNING|trainer.py:803] 2025-04-26 21:46:27,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6282
[WARNING|trainer.py:803] 2025-04-26 21:46:28,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7352
6244
[WARNING|trainer.py:803] 2025-04-26 21:46:29,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:29,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7353
[WARNING|trainer.py:803] 2025-04-26 21:46:29,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6283
[WARNING|trainer.py:803] 2025-04-26 21:46:30,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7354
6245
[WARNING|trainer.py:803] 2025-04-26 21:46:30,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:31,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:31,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7355
6284
6246
[WARNING|trainer.py:803] 2025-04-26 21:46:32,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7356
[WARNING|trainer.py:803] 2025-04-26 21:46:32,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:33,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:33,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7357
6285
6247
[WARNING|trainer.py:803] 2025-04-26 21:46:34,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7358
[WARNING|trainer.py:803] 2025-04-26 21:46:34,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:34,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:35,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7359
6248
6286
[WARNING|trainer.py:803] 2025-04-26 21:46:36,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7360
[WARNING|trainer.py:803] 2025-04-26 21:46:36,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:37,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:37,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7361
6249
[WARNING|trainer.py:803] 2025-04-26 21:46:38,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7362
6287
[WARNING|trainer.py:803] 2025-04-26 21:46:38,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:39,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:39,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7363
6250
6288
[WARNING|trainer.py:803] 2025-04-26 21:46:40,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7364
[WARNING|trainer.py:803] 2025-04-26 21:46:40,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:41,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:41,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6251
7365
6289
[WARNING|trainer.py:803] 2025-04-26 21:46:42,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:42,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7366
[WARNING|trainer.py:803] 2025-04-26 21:46:43,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6252
[WARNING|trainer.py:803] 2025-04-26 21:46:43,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7367
[WARNING|trainer.py:803] 2025-04-26 21:46:44,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:44,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6290
7368
6253
[WARNING|trainer.py:803] 2025-04-26 21:46:45,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:45,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7369
[WARNING|trainer.py:803] 2025-04-26 21:46:46,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6291
[WARNING|trainer.py:803] 2025-04-26 21:46:46,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6254
7370
[WARNING|trainer.py:803] 2025-04-26 21:46:47,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:47,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:47,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7371
6255
6292
[WARNING|trainer.py:803] 2025-04-26 21:46:48,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7372
[WARNING|trainer.py:803] 2025-04-26 21:46:49,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:49,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:49,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7373
6256
6293
[WARNING|trainer.py:803] 2025-04-26 21:46:51,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7374
[WARNING|trainer.py:803] 2025-04-26 21:46:51,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:51,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:52,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7375
6257
6294
[WARNING|trainer.py:803] 2025-04-26 21:46:53,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:53,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7376
[WARNING|trainer.py:803] 2025-04-26 21:46:53,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:54,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6258
7377
6295
[WARNING|trainer.py:803] 2025-04-26 21:46:55,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:55,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:46:55,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7378
[WARNING|trainer.py:803] 2025-04-26 21:46:56,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6259
7379
6296
[WARNING|trainer.py:803] 2025-04-26 21:46:57,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:57,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:46:57,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7380
6260
[WARNING|trainer.py:803] 2025-04-26 21:46:58,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7381
6297
[WARNING|trainer.py:803] 2025-04-26 21:46:58,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:46:59,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:46:59,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7382
6261
[WARNING|trainer.py:803] 2025-04-26 21:47:00,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6298
7383
[WARNING|trainer.py:803] 2025-04-26 21:47:00,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:01,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:01,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7384
6262
[WARNING|trainer.py:803] 2025-04-26 21:47:02,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6299
7385
[WARNING|trainer.py:803] 2025-04-26 21:47:02,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:03,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:03,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7386
6263
6300
[WARNING|trainer.py:803] 2025-04-26 21:47:04,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7387
[WARNING|trainer.py:803] 2025-04-26 21:47:04,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:05,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:05,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6301
7388
6264
[WARNING|trainer.py:803] 2025-04-26 21:47:06,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:06,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:06,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7389
6302
[WARNING|trainer.py:803] 2025-04-26 21:47:07,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6265
[WARNING|trainer.py:803] 2025-04-26 21:47:07,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7390
6303
[WARNING|trainer.py:803] 2025-04-26 21:47:08,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:08,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7391
[WARNING|trainer.py:803] 2025-04-26 21:47:09,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6304
[WARNING|trainer.py:803] 2025-04-26 21:47:09,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6266
7392
[WARNING|trainer.py:803] 2025-04-26 21:47:10,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:10,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:10,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6305
7393
6267
[WARNING|trainer.py:803] 2025-04-26 21:47:11,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:11,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7394
6306
[WARNING|trainer.py:803] 2025-04-26 21:47:12,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:13,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:13,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7395
6307
6268
[WARNING|trainer.py:803] 2025-04-26 21:47:14,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7396
[WARNING|trainer.py:803] 2025-04-26 21:47:14,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:14,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6308
[WARNING|trainer.py:803] 2025-04-26 21:47:15,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7397
6269
[WARNING|trainer.py:803] 2025-04-26 21:47:15,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:16,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7398
6309
[WARNING|trainer.py:803] 2025-04-26 21:47:16,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:17,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:17,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7399
6310
6270
[WARNING|trainer.py:803] 2025-04-26 21:47:18,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7400
[WARNING|trainer.py:803] 2025-04-26 21:47:18,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:18,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6311
[WARNING|trainer.py:803] 2025-04-26 21:47:19,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7401
6271
[WARNING|trainer.py:803] 2025-04-26 21:47:19,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:20,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6312
7402
[WARNING|trainer.py:803] 2025-04-26 21:47:20,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:21,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:21,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7403
6313
6272
[WARNING|trainer.py:803] 2025-04-26 21:47:22,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:22,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7404
[WARNING|trainer.py:803] 2025-04-26 21:47:22,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6314
[WARNING|trainer.py:803] 2025-04-26 21:47:23,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7405
[WARNING|trainer.py:803] 2025-04-26 21:47:23,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6273
6315
[WARNING|trainer.py:803] 2025-04-26 21:47:24,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7406
[WARNING|trainer.py:803] 2025-04-26 21:47:24,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:24,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6316
[WARNING|trainer.py:803] 2025-04-26 21:47:25,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7407
6274
[WARNING|trainer.py:803] 2025-04-26 21:47:26,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:26,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6317
7408
[WARNING|trainer.py:803] 2025-04-26 21:47:26,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:27,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:27,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7409
6318
6275
[WARNING|trainer.py:803] 2025-04-26 21:47:28,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7410
[WARNING|trainer.py:803] 2025-04-26 21:47:28,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:28,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6319
[WARNING|trainer.py:803] 2025-04-26 21:47:29,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6276
7411
[WARNING|trainer.py:803] 2025-04-26 21:47:30,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6320
[WARNING|trainer.py:803] 2025-04-26 21:47:30,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:30,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7412
[WARNING|trainer.py:803] 2025-04-26 21:47:31,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6277
6321
[WARNING|trainer.py:803] 2025-04-26 21:47:31,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7413
[WARNING|trainer.py:803] 2025-04-26 21:47:32,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:32,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:33,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7414
6322
6278
[WARNING|trainer.py:803] 2025-04-26 21:47:34,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:34,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7415
[WARNING|trainer.py:803] 2025-04-26 21:47:34,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6323
[WARNING|trainer.py:803] 2025-04-26 21:47:35,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7416
6279
[WARNING|trainer.py:803] 2025-04-26 21:47:35,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6324
[WARNING|trainer.py:803] 2025-04-26 21:47:36,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:36,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7417
[WARNING|trainer.py:803] 2025-04-26 21:47:36,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:37,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6325
7418
6280
[WARNING|trainer.py:803] 2025-04-26 21:47:38,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:38,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
[WARNING|trainer.py:803] 2025-04-26 21:47:38,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7419
6326
[WARNING|trainer.py:803] 2025-04-26 21:47:39,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:39,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7420
6281
6327
[WARNING|trainer.py:803] 2025-04-26 21:47:40,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:40,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7421
[WARNING|trainer.py:803] 2025-04-26 21:47:40,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6328
[WARNING|trainer.py:803] 2025-04-26 21:47:41,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6282
7422
[WARNING|trainer.py:803] 2025-04-26 21:47:42,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:42,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:42,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6329
7423
[WARNING|trainer.py:803] 2025-04-26 21:47:43,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6283
[WARNING|trainer.py:803] 2025-04-26 21:47:43,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7424
6330
[WARNING|trainer.py:803] 2025-04-26 21:47:44,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:44,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:44,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7425
6331
6284
[WARNING|trainer.py:803] 2025-04-26 21:47:45,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7426
[WARNING|trainer.py:803] 2025-04-26 21:47:45,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:46,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6332
[WARNING|trainer.py:803] 2025-04-26 21:47:46,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7427
[WARNING|trainer.py:803] 2025-04-26 21:47:47,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6285
[WARNING|trainer.py:803] 2025-04-26 21:47:47,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7428
6333
[WARNING|trainer.py:803] 2025-04-26 21:47:48,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:48,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:48,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7429
6334
[WARNING|trainer.py:803] 2025-04-26 21:47:49,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6286
7430
[WARNING|trainer.py:803] 2025-04-26 21:47:50,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6335
[WARNING|trainer.py:803] 2025-04-26 21:47:50,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:50,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7431
[WARNING|trainer.py:803] 2025-04-26 21:47:51,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6336
7432
6287
[WARNING|trainer.py:803] 2025-04-26 21:47:52,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:52,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:47:52,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7433
6337
6288
[WARNING|trainer.py:803] 2025-04-26 21:47:53,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:53,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7434
6338
[WARNING|trainer.py:803] 2025-04-26 21:47:54,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:54,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7435
[WARNING|trainer.py:803] 2025-04-26 21:47:55,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6289
6339
[WARNING|trainer.py:803] 2025-04-26 21:47:55,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7436
[WARNING|trainer.py:803] 2025-04-26 21:47:56,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:56,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:56,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6340
7437
[WARNING|trainer.py:803] 2025-04-26 21:47:57,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:47:57,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7438
6290
6341
[WARNING|trainer.py:803] 2025-04-26 21:47:59,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:47:59,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:47:59,203 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7439
6342
[WARNING|trainer.py:803] 2025-04-26 21:48:00,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6291
7440
[WARNING|trainer.py:803] 2025-04-26 21:48:00,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:00,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6343
[WARNING|trainer.py:803] 2025-04-26 21:48:01,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7441
[WARNING|trainer.py:803] 2025-04-26 21:48:01,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6292
[WARNING|trainer.py:803] 2025-04-26 21:48:02,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7442
6344
[WARNING|trainer.py:803] 2025-04-26 21:48:02,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:03,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:03,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7443
6345
6293
[WARNING|trainer.py:803] 2025-04-26 21:48:04,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7444
[WARNING|trainer.py:803] 2025-04-26 21:48:04,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:04,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6346
[WARNING|trainer.py:803] 2025-04-26 21:48:05,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7445
6294
[WARNING|trainer.py:803] 2025-04-26 21:48:05,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:06,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6347
7446
[WARNING|trainer.py:803] 2025-04-26 21:48:06,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:07,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:07,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7447
6348
6295
[WARNING|trainer.py:803] 2025-04-26 21:48:08,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7448
[WARNING|trainer.py:803] 2025-04-26 21:48:08,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:08,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6349
[WARNING|trainer.py:803] 2025-04-26 21:48:09,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7449
6296
[WARNING|trainer.py:803] 2025-04-26 21:48:09,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6350
[WARNING|trainer.py:803] 2025-04-26 21:48:10,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:10,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7450
[WARNING|trainer.py:803] 2025-04-26 21:48:11,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:11,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7451
6351
6297
[WARNING|trainer.py:803] 2025-04-26 21:48:12,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:12,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:12,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7452
6352
[WARNING|trainer.py:803] 2025-04-26 21:48:13,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7453
6298
[WARNING|trainer.py:803] 2025-04-26 21:48:13,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6353
[WARNING|trainer.py:803] 2025-04-26 21:48:14,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:14,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7454
[WARNING|trainer.py:803] 2025-04-26 21:48:15,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:15,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7455
6299
6354
[WARNING|trainer.py:803] 2025-04-26 21:48:16,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:16,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:16,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7456
6355
[WARNING|trainer.py:803] 2025-04-26 21:48:17,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6300
7457
[WARNING|trainer.py:803] 2025-04-26 21:48:17,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6356
[WARNING|trainer.py:803] 2025-04-26 21:48:18,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:18,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7458
6301
[WARNING|trainer.py:803] 2025-04-26 21:48:19,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:19,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6357
[WARNING|trainer.py:803] 2025-04-26 21:48:19,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7459
6302
[WARNING|trainer.py:803] 2025-04-26 21:48:20,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:20,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7460
6358
[WARNING|trainer.py:803] 2025-04-26 21:48:21,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6303
[WARNING|trainer.py:803] 2025-04-26 21:48:21,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:21,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7461
6359
[WARNING|trainer.py:803] 2025-04-26 21:48:22,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:22,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7462
6304
[WARNING|trainer.py:803] 2025-04-26 21:48:23,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6360
[WARNING|trainer.py:803] 2025-04-26 21:48:23,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:23,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7463
6305
[WARNING|trainer.py:803] 2025-04-26 21:48:24,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:24,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6361
7464
[WARNING|trainer.py:803] 2025-04-26 21:48:25,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6306
[WARNING|trainer.py:803] 2025-04-26 21:48:25,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:25,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7465
6362
[WARNING|trainer.py:803] 2025-04-26 21:48:26,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:26,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6307
7466
[WARNING|trainer.py:803] 2025-04-26 21:48:27,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6363
[WARNING|trainer.py:803] 2025-04-26 21:48:27,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:27,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7467
6308
[WARNING|trainer.py:803] 2025-04-26 21:48:28,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6364
[WARNING|trainer.py:803] 2025-04-26 21:48:28,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7468
[WARNING|trainer.py:803] 2025-04-26 21:48:29,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:29,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6309
[WARNING|trainer.py:803] 2025-04-26 21:48:29,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7469
6365
[WARNING|trainer.py:803] 2025-04-26 21:48:30,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:31,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:31,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6310
7470
6366
[WARNING|trainer.py:803] 2025-04-26 21:48:31,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:32,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7471
[WARNING|trainer.py:803] 2025-04-26 21:48:32,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6311
6367
[WARNING|trainer.py:803] 2025-04-26 21:48:33,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:33,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7472
[WARNING|trainer.py:803] 2025-04-26 21:48:33,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6312
[WARNING|trainer.py:803] 2025-04-26 21:48:34,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6368
7473
[WARNING|trainer.py:803] 2025-04-26 21:48:34,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:35,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6313
[WARNING|trainer.py:803] 2025-04-26 21:48:35,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7474
6369
[WARNING|trainer.py:803] 2025-04-26 21:48:35,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:36,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7475
6314
[WARNING|trainer.py:803] 2025-04-26 21:48:36,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6370
[WARNING|trainer.py:803] 2025-04-26 21:48:37,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:37,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7476
6315
[WARNING|trainer.py:803] 2025-04-26 21:48:37,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:38,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6371
7477
[WARNING|trainer.py:803] 2025-04-26 21:48:38,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6316
[WARNING|trainer.py:803] 2025-04-26 21:48:39,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:39,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7478
6372
[WARNING|trainer.py:803] 2025-04-26 21:48:39,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:40,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6317
[WARNING|trainer.py:803] 2025-04-26 21:48:40,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7479
6373
[WARNING|trainer.py:803] 2025-04-26 21:48:41,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:41,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7480
6318
[WARNING|trainer.py:803] 2025-04-26 21:48:41,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6374
[WARNING|trainer.py:803] 2025-04-26 21:48:42,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:42,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7481
[WARNING|trainer.py:803] 2025-04-26 21:48:43,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6319
[WARNING|trainer.py:803] 2025-04-26 21:48:43,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6375
7482
[WARNING|trainer.py:803] 2025-04-26 21:48:43,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6320
[WARNING|trainer.py:803] 2025-04-26 21:48:44,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:44,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7483
6376
[WARNING|trainer.py:803] 2025-04-26 21:48:45,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:45,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7484
[WARNING|trainer.py:803] 2025-04-26 21:48:45,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6321
6377
[WARNING|trainer.py:803] 2025-04-26 21:48:46,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:46,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7485
[WARNING|trainer.py:803] 2025-04-26 21:48:47,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6322
6378
[WARNING|trainer.py:803] 2025-04-26 21:48:47,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7486
[WARNING|trainer.py:803] 2025-04-26 21:48:47,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:48,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6323
[WARNING|trainer.py:803] 2025-04-26 21:48:48,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7487
6379
[WARNING|trainer.py:803] 2025-04-26 21:48:49,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:49,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:49,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6324
7488
6380
[WARNING|trainer.py:803] 2025-04-26 21:48:50,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:50,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7489
[WARNING|trainer.py:803] 2025-04-26 21:48:51,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6325
6381
[WARNING|trainer.py:803] 2025-04-26 21:48:51,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7490
[WARNING|trainer.py:803] 2025-04-26 21:48:51,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:52,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6326
[WARNING|trainer.py:803] 2025-04-26 21:48:52,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6382
7491
[WARNING|trainer.py:803] 2025-04-26 21:48:53,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:53,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6327
[WARNING|trainer.py:803] 2025-04-26 21:48:53,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7492
6383
[WARNING|trainer.py:803] 2025-04-26 21:48:54,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:48:54,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7493
6328
[WARNING|trainer.py:803] 2025-04-26 21:48:55,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6384
[WARNING|trainer.py:803] 2025-04-26 21:48:55,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:48:55,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7494
6329
[WARNING|trainer.py:803] 2025-04-26 21:48:56,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:48:56,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6385
7495
[WARNING|trainer.py:803] 2025-04-26 21:48:57,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6330
[WARNING|trainer.py:803] 2025-04-26 21:48:57,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:57,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7496
6386
[WARNING|trainer.py:803] 2025-04-26 21:48:58,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:58,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6331
[WARNING|trainer.py:803] 2025-04-26 21:48:59,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7497
6387
[WARNING|trainer.py:803] 2025-04-26 21:48:59,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:48:59,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7498
6332
[WARNING|trainer.py:803] 2025-04-26 21:49:00,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6388
[WARNING|trainer.py:803] 2025-04-26 21:49:00,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7499
[WARNING|trainer.py:803] 2025-04-26 21:49:01,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6333
[WARNING|trainer.py:803] 2025-04-26 21:49:01,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:02,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7500
6389
[WARNING|trainer.py:803] 2025-04-26 21:49:02,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:03,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6334
[WARNING|trainer.py:803] 2025-04-26 21:49:03,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7501
6390
[WARNING|trainer.py:803] 2025-04-26 21:49:03,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:04,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:04,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6335
7502
6391
[WARNING|trainer.py:803] 2025-04-26 21:49:05,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:05,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6336
[WARNING|trainer.py:803] 2025-04-26 21:49:05,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7503
6392
[WARNING|trainer.py:803] 2025-04-26 21:49:06,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:06,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7504
6337
[WARNING|trainer.py:803] 2025-04-26 21:49:07,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6393
[WARNING|trainer.py:803] 2025-04-26 21:49:07,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:07,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7505
6338
[WARNING|trainer.py:803] 2025-04-26 21:49:08,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6394
[WARNING|trainer.py:803] 2025-04-26 21:49:08,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:09,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7506
6339
[WARNING|trainer.py:803] 2025-04-26 21:49:09,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:10,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6395
7507
[WARNING|trainer.py:803] 2025-04-26 21:49:10,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6340
[WARNING|trainer.py:803] 2025-04-26 21:49:11,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:11,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6396
7508
[WARNING|trainer.py:803] 2025-04-26 21:49:11,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6341
[WARNING|trainer.py:803] 2025-04-26 21:49:12,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:12,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6397
7509
[WARNING|trainer.py:803] 2025-04-26 21:49:13,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6342
[WARNING|trainer.py:803] 2025-04-26 21:49:13,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:13,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7510
6398
[WARNING|trainer.py:803] 2025-04-26 21:49:14,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:14,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6343
[WARNING|trainer.py:803] 2025-04-26 21:49:15,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7511
6399
[WARNING|trainer.py:803] 2025-04-26 21:49:15,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:16,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6344
[WARNING|trainer.py:803] 2025-04-26 21:49:16,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7512
6400
[WARNING|trainer.py:803] 2025-04-26 21:49:17,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:17,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6345
[WARNING|trainer.py:803] 2025-04-26 21:49:17,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7513
6401
[WARNING|trainer.py:803] 2025-04-26 21:49:18,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:18,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6346
7514
[WARNING|trainer.py:803] 2025-04-26 21:49:18,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6402
[WARNING|trainer.py:803] 2025-04-26 21:49:19,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:19,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7515
6347
[WARNING|trainer.py:803] 2025-04-26 21:49:20,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6403
[WARNING|trainer.py:803] 2025-04-26 21:49:20,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:21,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7516
6348
[WARNING|trainer.py:803] 2025-04-26 21:49:21,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:22,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6404
[WARNING|trainer.py:803] 2025-04-26 21:49:22,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7517
6349
[WARNING|trainer.py:803] 2025-04-26 21:49:23,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:23,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6405
[WARNING|trainer.py:803] 2025-04-26 21:49:23,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7518
6350
[WARNING|trainer.py:803] 2025-04-26 21:49:24,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:24,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6406
7519
[WARNING|trainer.py:803] 2025-04-26 21:49:25,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6351
[WARNING|trainer.py:803] 2025-04-26 21:49:25,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:25,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6407
7520
[WARNING|trainer.py:803] 2025-04-26 21:49:26,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6352
[WARNING|trainer.py:803] 2025-04-26 21:49:27,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:27,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7521
6408
[WARNING|trainer.py:803] 2025-04-26 21:49:27,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6353
[WARNING|trainer.py:803] 2025-04-26 21:49:28,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:28,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7522
6409
[WARNING|trainer.py:803] 2025-04-26 21:49:29,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6354
[WARNING|trainer.py:803] 2025-04-26 21:49:29,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:29,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7523
6410
[WARNING|trainer.py:803] 2025-04-26 21:49:30,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:30,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6355
[WARNING|trainer.py:803] 2025-04-26 21:49:31,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7524
6411
[WARNING|trainer.py:803] 2025-04-26 21:49:31,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:31,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6356
7525
[WARNING|trainer.py:803] 2025-04-26 21:49:32,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6412
[WARNING|trainer.py:803] 2025-04-26 21:49:32,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:33,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6357
7526
[WARNING|trainer.py:803] 2025-04-26 21:49:33,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6413
[WARNING|trainer.py:803] 2025-04-26 21:49:34,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:34,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7527
6358
[WARNING|trainer.py:803] 2025-04-26 21:49:34,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6414
[WARNING|trainer.py:803] 2025-04-26 21:49:35,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:35,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7528
6359
[WARNING|trainer.py:803] 2025-04-26 21:49:36,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6415
[WARNING|trainer.py:803] 2025-04-26 21:49:36,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:36,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7529
6360
[WARNING|trainer.py:803] 2025-04-26 21:49:37,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:37,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6416
[WARNING|trainer.py:803] 2025-04-26 21:49:38,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7530
6361
[WARNING|trainer.py:803] 2025-04-26 21:49:38,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:39,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6417
[WARNING|trainer.py:803] 2025-04-26 21:49:39,474 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7531
6362
[WARNING|trainer.py:803] 2025-04-26 21:49:40,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:49:40,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7532
6418
[WARNING|trainer.py:803] 2025-04-26 21:49:40,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6363
[WARNING|trainer.py:803] 2025-04-26 21:49:41,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:41,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7533
6419
[WARNING|trainer.py:803] 2025-04-26 21:49:42,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6364
[WARNING|trainer.py:803] 2025-04-26 21:49:42,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:42,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7534
[WARNING|trainer.py:803] 2025-04-26 21:49:43,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6420
[WARNING|trainer.py:803] 2025-04-26 21:49:43,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6365
7535
[WARNING|trainer.py:803] 2025-04-26 21:49:44,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:44,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6421
[WARNING|trainer.py:803] 2025-04-26 21:49:45,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6366
7536
[WARNING|trainer.py:803] 2025-04-26 21:49:45,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:46,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6422
[WARNING|trainer.py:803] 2025-04-26 21:49:46,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7537
6367
[WARNING|trainer.py:803] 2025-04-26 21:49:46,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:47,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:47,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6423
7538
6368
[WARNING|trainer.py:803] 2025-04-26 21:49:48,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:48,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:48,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6424
7539
6369
[WARNING|trainer.py:803] 2025-04-26 21:49:49,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:49:49,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7540
[WARNING|trainer.py:803] 2025-04-26 21:49:50,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6425
6370
[WARNING|trainer.py:803] 2025-04-26 21:49:50,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:50,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7541
6426
[WARNING|trainer.py:803] 2025-04-26 21:49:51,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6371
[WARNING|trainer.py:803] 2025-04-26 21:49:52,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:52,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7542
6427
[WARNING|trainer.py:803] 2025-04-26 21:49:52,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6372
[WARNING|trainer.py:803] 2025-04-26 21:49:53,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:49:53,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7543
6428
[WARNING|trainer.py:803] 2025-04-26 21:49:54,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:54,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6373
7544
[WARNING|trainer.py:803] 2025-04-26 21:49:54,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6429
[WARNING|trainer.py:803] 2025-04-26 21:49:55,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:49:55,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6374
7545
[WARNING|trainer.py:803] 2025-04-26 21:49:56,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6430
[WARNING|trainer.py:803] 2025-04-26 21:49:56,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:56,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6375
7546
[WARNING|trainer.py:803] 2025-04-26 21:49:57,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6431
[WARNING|trainer.py:803] 2025-04-26 21:49:58,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:58,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7547
6376
[WARNING|trainer.py:803] 2025-04-26 21:49:58,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:49:59,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6432
[WARNING|trainer.py:803] 2025-04-26 21:49:59,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7548
6377
[WARNING|trainer.py:803] 2025-04-26 21:50:00,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:50:00,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6433
[WARNING|trainer.py:803] 2025-04-26 21:50:00,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7549
6378
[WARNING|trainer.py:803] 2025-04-26 21:50:01,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:01,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6434
[WARNING|trainer.py:803] 2025-04-26 21:50:02,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7550
6379
[WARNING|trainer.py:803] 2025-04-26 21:50:02,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:02,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7551
6435
[WARNING|trainer.py:803] 2025-04-26 21:50:03,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6380
[WARNING|trainer.py:803] 2025-04-26 21:50:04,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:04,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7552
6436
[WARNING|trainer.py:803] 2025-04-26 21:50:04,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6381
[WARNING|trainer.py:803] 2025-04-26 21:50:05,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:05,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7553
6437
[WARNING|trainer.py:803] 2025-04-26 21:50:06,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:06,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6382
7554
[WARNING|trainer.py:803] 2025-04-26 21:50:06,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6438
[WARNING|trainer.py:803] 2025-04-26 21:50:07,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:07,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6383
7555
[WARNING|trainer.py:803] 2025-04-26 21:50:08,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6439
[WARNING|trainer.py:803] 2025-04-26 21:50:08,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:08,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7556
6384
[WARNING|trainer.py:803] 2025-04-26 21:50:09,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6440
[WARNING|trainer.py:803] 2025-04-26 21:50:10,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:10,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7557
6385
[WARNING|trainer.py:803] 2025-04-26 21:50:10,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:11,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6441
[WARNING|trainer.py:803] 2025-04-26 21:50:11,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7558
6386
[WARNING|trainer.py:803] 2025-04-26 21:50:12,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:12,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6442
7559
[WARNING|trainer.py:803] 2025-04-26 21:50:12,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6387
[WARNING|trainer.py:803] 2025-04-26 21:50:13,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:13,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6443
7560
[WARNING|trainer.py:803] 2025-04-26 21:50:14,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6388
[WARNING|trainer.py:803] 2025-04-26 21:50:14,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:14,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7561
6444
[WARNING|trainer.py:803] 2025-04-26 21:50:15,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6389
[WARNING|trainer.py:803] 2025-04-26 21:50:15,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:16,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7562
6445
[WARNING|trainer.py:803] 2025-04-26 21:50:16,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:17,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6390
7563
[WARNING|trainer.py:803] 2025-04-26 21:50:17,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6446
[WARNING|trainer.py:803] 2025-04-26 21:50:18,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:18,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6391
7564
[WARNING|trainer.py:803] 2025-04-26 21:50:18,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6447
[WARNING|trainer.py:803] 2025-04-26 21:50:19,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:19,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7565
6392
[WARNING|trainer.py:803] 2025-04-26 21:50:20,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6448
[WARNING|trainer.py:803] 2025-04-26 21:50:20,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:20,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7566
6393
[WARNING|trainer.py:803] 2025-04-26 21:50:21,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:21,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6449
[WARNING|trainer.py:803] 2025-04-26 21:50:22,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7567
6394
[WARNING|trainer.py:803] 2025-04-26 21:50:22,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:22,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6450
7568
[WARNING|trainer.py:803] 2025-04-26 21:50:23,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6395
[WARNING|trainer.py:803] 2025-04-26 21:50:23,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:24,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7569
6451
[WARNING|trainer.py:803] 2025-04-26 21:50:24,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6396
[WARNING|trainer.py:803] 2025-04-26 21:50:25,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:25,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7570
6452
[WARNING|trainer.py:803] 2025-04-26 21:50:25,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:26,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6397
7571
[WARNING|trainer.py:803] 2025-04-26 21:50:26,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6453
[WARNING|trainer.py:803] 2025-04-26 21:50:27,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:27,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6398
7572
[WARNING|trainer.py:803] 2025-04-26 21:50:28,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6454
[WARNING|trainer.py:803] 2025-04-26 21:50:28,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:28,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7573
6399
[WARNING|trainer.py:803] 2025-04-26 21:50:29,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6455
[WARNING|trainer.py:803] 2025-04-26 21:50:29,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:29,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7574
6400
[WARNING|trainer.py:803] 2025-04-26 21:50:30,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:31,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6456
[WARNING|trainer.py:803] 2025-04-26 21:50:31,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7575
6401
[WARNING|trainer.py:803] 2025-04-26 21:50:32,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:32,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6457
7576
[WARNING|trainer.py:803] 2025-04-26 21:50:32,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:33,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6402
[WARNING|trainer.py:803] 2025-04-26 21:50:33,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7577
6458
[WARNING|trainer.py:803] 2025-04-26 21:50:34,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6403
[WARNING|trainer.py:803] 2025-04-26 21:50:34,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:34,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7578
6459
[WARNING|trainer.py:803] 2025-04-26 21:50:35,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:35,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6404
7579
[WARNING|trainer.py:803] 2025-04-26 21:50:36,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6460
[WARNING|trainer.py:803] 2025-04-26 21:50:36,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:36,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6405
7580
[WARNING|trainer.py:803] 2025-04-26 21:50:37,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6461
[WARNING|trainer.py:803] 2025-04-26 21:50:38,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:38,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7581
6406
[WARNING|trainer.py:803] 2025-04-26 21:50:38,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:39,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6462
[WARNING|trainer.py:803] 2025-04-26 21:50:39,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7582
6407
[WARNING|trainer.py:803] 2025-04-26 21:50:40,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:40,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6463
[WARNING|trainer.py:803] 2025-04-26 21:50:40,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7583
6408
[WARNING|trainer.py:803] 2025-04-26 21:50:41,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:41,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7584
6464
[WARNING|trainer.py:803] 2025-04-26 21:50:42,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6409
[WARNING|trainer.py:803] 2025-04-26 21:50:42,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:42,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7585
6465
[WARNING|trainer.py:803] 2025-04-26 21:50:43,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6410
[WARNING|trainer.py:803] 2025-04-26 21:50:44,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:44,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7586
6466
[WARNING|trainer.py:803] 2025-04-26 21:50:44,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6411
[WARNING|trainer.py:803] 2025-04-26 21:50:45,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:45,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7587
6467
[WARNING|trainer.py:803] 2025-04-26 21:50:46,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:46,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6412
7588
[WARNING|trainer.py:803] 2025-04-26 21:50:46,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:47,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6468
[WARNING|trainer.py:803] 2025-04-26 21:50:47,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6413
7589
[WARNING|trainer.py:803] 2025-04-26 21:50:48,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:48,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6469
[WARNING|trainer.py:803] 2025-04-26 21:50:48,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7590
6414
[WARNING|trainer.py:803] 2025-04-26 21:50:49,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:50:50,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:50,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6470
7591
6415
[WARNING|trainer.py:803] 2025-04-26 21:50:50,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:51,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:51,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6471
7592
6416
[WARNING|trainer.py:803] 2025-04-26 21:50:52,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:52,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7593
[WARNING|trainer.py:803] 2025-04-26 21:50:52,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6472
6417
[WARNING|trainer.py:803] 2025-04-26 21:50:53,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:53,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7594
6473
[WARNING|trainer.py:803] 2025-04-26 21:50:54,110 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6418
[WARNING|trainer.py:803] 2025-04-26 21:50:54,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:54,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7595
6474
[WARNING|trainer.py:803] 2025-04-26 21:50:55,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:55,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6419
[WARNING|trainer.py:803] 2025-04-26 21:50:56,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7596
6475
[WARNING|trainer.py:803] 2025-04-26 21:50:56,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:50:57,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6420
7597
[WARNING|trainer.py:803] 2025-04-26 21:50:57,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6476
[WARNING|trainer.py:803] 2025-04-26 21:50:58,186 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:50:58,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7598
6421
[WARNING|trainer.py:803] 2025-04-26 21:50:58,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6477
[WARNING|trainer.py:803] 2025-04-26 21:50:59,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:50:59,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7599
6422
[WARNING|trainer.py:803] 2025-04-26 21:51:00,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:00,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6478
[WARNING|trainer.py:803] 2025-04-26 21:51:00,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7600
6423
[WARNING|trainer.py:803] 2025-04-26 21:51:01,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:01,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6479
[WARNING|trainer.py:803] 2025-04-26 21:51:02,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7601
6424
[WARNING|trainer.py:803] 2025-04-26 21:51:02,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:03,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6480
7602
[WARNING|trainer.py:803] 2025-04-26 21:51:03,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6425
[WARNING|trainer.py:803] 2025-04-26 21:51:04,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:04,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7603
6481
[WARNING|trainer.py:803] 2025-04-26 21:51:04,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6426
[WARNING|trainer.py:803] 2025-04-26 21:51:05,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:05,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7604
6482
[WARNING|trainer.py:803] 2025-04-26 21:51:06,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:06,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6427
[WARNING|trainer.py:803] 2025-04-26 21:51:06,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7605
6483
[WARNING|trainer.py:803] 2025-04-26 21:51:07,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:07,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6428
[WARNING|trainer.py:803] 2025-04-26 21:51:08,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7606
6484
[WARNING|trainer.py:803] 2025-04-26 21:51:08,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:09,084 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6429
7607
[WARNING|trainer.py:803] 2025-04-26 21:51:09,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6485
[WARNING|trainer.py:803] 2025-04-26 21:51:10,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:10,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7608
6430
[WARNING|trainer.py:803] 2025-04-26 21:51:10,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6486
[WARNING|trainer.py:803] 2025-04-26 21:51:11,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:11,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7609
6431
[WARNING|trainer.py:803] 2025-04-26 21:51:12,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6487
[WARNING|trainer.py:803] 2025-04-26 21:51:12,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:12,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7610
6432
[WARNING|trainer.py:803] 2025-04-26 21:51:13,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:13,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6488
[WARNING|trainer.py:803] 2025-04-26 21:51:14,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7611
6433
[WARNING|trainer.py:803] 2025-04-26 21:51:14,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:15,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6489
7612
[WARNING|trainer.py:803] 2025-04-26 21:51:15,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6434
[WARNING|trainer.py:803] 2025-04-26 21:51:16,143 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:16,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7613
6490
[WARNING|trainer.py:803] 2025-04-26 21:51:16,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6435
[WARNING|trainer.py:803] 2025-04-26 21:51:17,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:17,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7614
6491
[WARNING|trainer.py:803] 2025-04-26 21:51:18,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:18,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6436
[WARNING|trainer.py:803] 2025-04-26 21:51:18,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7615
6492
[WARNING|trainer.py:803] 2025-04-26 21:51:19,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:19,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6437
[WARNING|trainer.py:803] 2025-04-26 21:51:20,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7616
6493
[WARNING|trainer.py:803] 2025-04-26 21:51:20,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:21,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6438
[WARNING|trainer.py:803] 2025-04-26 21:51:21,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7617
6494
[WARNING|trainer.py:803] 2025-04-26 21:51:22,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:22,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6439
7618
[WARNING|trainer.py:803] 2025-04-26 21:51:22,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6495
[WARNING|trainer.py:803] 2025-04-26 21:51:23,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:23,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6440
7619
[WARNING|trainer.py:803] 2025-04-26 21:51:24,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6496
[WARNING|trainer.py:803] 2025-04-26 21:51:24,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:24,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7620
6441
[WARNING|trainer.py:803] 2025-04-26 21:51:25,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6497
[WARNING|trainer.py:803] 2025-04-26 21:51:26,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:26,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7621
6442
[WARNING|trainer.py:803] 2025-04-26 21:51:26,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:27,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6498
[WARNING|trainer.py:803] 2025-04-26 21:51:27,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7622
6443
[WARNING|trainer.py:803] 2025-04-26 21:51:28,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:28,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6499
[WARNING|trainer.py:803] 2025-04-26 21:51:28,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7623
6444
[WARNING|trainer.py:803] 2025-04-26 21:51:29,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:29,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6500
7624
[WARNING|trainer.py:803] 2025-04-26 21:51:30,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6445
[WARNING|trainer.py:803] 2025-04-26 21:51:30,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:30,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6501
7625
[WARNING|trainer.py:803] 2025-04-26 21:51:31,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6446
[WARNING|trainer.py:803] 2025-04-26 21:51:32,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:32,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7626
6502
[WARNING|trainer.py:803] 2025-04-26 21:51:32,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6447
[WARNING|trainer.py:803] 2025-04-26 21:51:33,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:33,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7627
6503
[WARNING|trainer.py:803] 2025-04-26 21:51:34,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6448
[WARNING|trainer.py:803] 2025-04-26 21:51:34,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:34,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7628
6504
[WARNING|trainer.py:803] 2025-04-26 21:51:35,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:35,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6449
[WARNING|trainer.py:803] 2025-04-26 21:51:36,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7629
6505
[WARNING|trainer.py:803] 2025-04-26 21:51:36,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:37,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6450
7630
[WARNING|trainer.py:803] 2025-04-26 21:51:37,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6506
[WARNING|trainer.py:803] 2025-04-26 21:51:38,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:38,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7631
6451
[WARNING|trainer.py:803] 2025-04-26 21:51:38,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6507
[WARNING|trainer.py:803] 2025-04-26 21:51:39,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:39,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7632
6452
[WARNING|trainer.py:803] 2025-04-26 21:51:40,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6508
[WARNING|trainer.py:803] 2025-04-26 21:51:40,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:40,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7633
[WARNING|trainer.py:803] 2025-04-26 21:51:41,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6453
[WARNING|trainer.py:803] 2025-04-26 21:51:41,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6509
[WARNING|trainer.py:803] 2025-04-26 21:51:42,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7634
6454
[WARNING|trainer.py:803] 2025-04-26 21:51:42,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:43,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6510
7635
[WARNING|trainer.py:803] 2025-04-26 21:51:43,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6455
[WARNING|trainer.py:803] 2025-04-26 21:51:44,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:44,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6511
7636
[WARNING|trainer.py:803] 2025-04-26 21:51:44,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:45,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6456
[WARNING|trainer.py:803] 2025-04-26 21:51:45,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6512
7637
[WARNING|trainer.py:803] 2025-04-26 21:51:46,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6457
[WARNING|trainer.py:803] 2025-04-26 21:51:46,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:46,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7638
6513
[WARNING|trainer.py:803] 2025-04-26 21:51:47,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:47,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:47,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6458
7639
6514
[WARNING|trainer.py:803] 2025-04-26 21:51:48,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:49,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:49,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6459
7640
6515
[WARNING|trainer.py:803] 2025-04-26 21:51:50,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:50,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:50,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7641
6460
6516
[WARNING|trainer.py:803] 2025-04-26 21:51:51,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:51:51,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7642
[WARNING|trainer.py:803] 2025-04-26 21:51:51,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6461
6517
[WARNING|trainer.py:803] 2025-04-26 21:51:52,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:52,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7643
[WARNING|trainer.py:803] 2025-04-26 21:51:53,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6462
6518
[WARNING|trainer.py:803] 2025-04-26 21:51:53,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:54,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7644
[WARNING|trainer.py:803] 2025-04-26 21:51:54,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6463
6519
[WARNING|trainer.py:803] 2025-04-26 21:51:55,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:55,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7645
[WARNING|trainer.py:803] 2025-04-26 21:51:55,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6464
6520
[WARNING|trainer.py:803] 2025-04-26 21:51:56,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7646
[WARNING|trainer.py:803] 2025-04-26 21:51:56,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:51:57,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6465
[WARNING|trainer.py:803] 2025-04-26 21:51:57,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6521
7647
[WARNING|trainer.py:803] 2025-04-26 21:51:58,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:51:58,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6466
[WARNING|trainer.py:803] 2025-04-26 21:51:58,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6522
7648
[WARNING|trainer.py:803] 2025-04-26 21:51:59,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:51:59,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6467
[WARNING|trainer.py:803] 2025-04-26 21:52:00,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6523
7649
[WARNING|trainer.py:803] 2025-04-26 21:52:00,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:01,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6468
[WARNING|trainer.py:803] 2025-04-26 21:52:01,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6524
7650
[WARNING|trainer.py:803] 2025-04-26 21:52:02,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:02,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:02,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6469
7651
6525
[WARNING|trainer.py:803] 2025-04-26 21:52:03,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:52:03,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:52:03,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6470
7652
6526
[WARNING|trainer.py:803] 2025-04-26 21:52:04,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:05,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:52:05,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6471
7653
6527
[WARNING|trainer.py:803] 2025-04-26 21:52:06,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:06,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:06,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6472
7654
6528
[WARNING|trainer.py:803] 2025-04-26 21:52:07,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:07,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6473
[WARNING|trainer.py:803] 2025-04-26 21:52:07,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7655
6529
[WARNING|trainer.py:803] 2025-04-26 21:52:08,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:08,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6474
[WARNING|trainer.py:803] 2025-04-26 21:52:09,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7656
6530
[WARNING|trainer.py:803] 2025-04-26 21:52:10,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:10,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7657
6475
[WARNING|trainer.py:803] 2025-04-26 21:52:10,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6531
[WARNING|trainer.py:803] 2025-04-26 21:52:11,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:11,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:11,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7658
6476
6532
[WARNING|trainer.py:803] 2025-04-26 21:52:12,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:12,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7659
[WARNING|trainer.py:803] 2025-04-26 21:52:13,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6477
6533
[WARNING|trainer.py:803] 2025-04-26 21:52:13,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:14,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7660
[WARNING|trainer.py:803] 2025-04-26 21:52:14,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6478
6534
[WARNING|trainer.py:803] 2025-04-26 21:52:15,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:15,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7661
[WARNING|trainer.py:803] 2025-04-26 21:52:15,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6479
6535
[WARNING|trainer.py:803] 2025-04-26 21:52:16,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:16,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7662
[WARNING|trainer.py:803] 2025-04-26 21:52:17,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6480
6536
[WARNING|trainer.py:803] 2025-04-26 21:52:17,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:18,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7663
[WARNING|trainer.py:803] 2025-04-26 21:52:18,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6481
6537
[WARNING|trainer.py:803] 2025-04-26 21:52:19,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7664
[WARNING|trainer.py:803] 2025-04-26 21:52:19,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:19,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6482
6538
[WARNING|trainer.py:803] 2025-04-26 21:52:20,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7665
[WARNING|trainer.py:803] 2025-04-26 21:52:20,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:21,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6483
[WARNING|trainer.py:803] 2025-04-26 21:52:21,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6539
7666
[WARNING|trainer.py:803] 2025-04-26 21:52:22,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:22,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6484
[WARNING|trainer.py:803] 2025-04-26 21:52:22,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6540
7667
[WARNING|trainer.py:803] 2025-04-26 21:52:23,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:23,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:23,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6485
6541
7668
[WARNING|trainer.py:803] 2025-04-26 21:52:24,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:25,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:25,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6486
6542
7669
[WARNING|trainer.py:803] 2025-04-26 21:52:26,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:26,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6487
[WARNING|trainer.py:803] 2025-04-26 21:52:26,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6543
7670
[WARNING|trainer.py:803] 2025-04-26 21:52:27,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:27,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:27,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6544
6488
7671
[WARNING|trainer.py:803] 2025-04-26 21:52:28,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:28,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:29,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6545
6489
7672
[WARNING|trainer.py:803] 2025-04-26 21:52:30,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:30,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:30,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6546
7673
6490
[WARNING|trainer.py:803] 2025-04-26 21:52:31,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:31,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:31,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7674
6547
6491
[WARNING|trainer.py:803] 2025-04-26 21:52:32,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:32,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:32,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7675
6548
6492
[WARNING|trainer.py:803] 2025-04-26 21:52:34,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:34,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:52:34,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7676
6549
6493
[WARNING|trainer.py:803] 2025-04-26 21:52:35,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:35,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:35,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7677
6550
6494
[WARNING|trainer.py:803] 2025-04-26 21:52:36,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:36,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7678
[WARNING|trainer.py:803] 2025-04-26 21:52:37,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6551
6495
[WARNING|trainer.py:803] 2025-04-26 21:52:37,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:38,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7679
[WARNING|trainer.py:803] 2025-04-26 21:52:38,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6552
6496
[WARNING|trainer.py:803] 2025-04-26 21:52:39,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:39,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7680
[WARNING|trainer.py:803] 2025-04-26 21:52:39,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6553
6497
[WARNING|trainer.py:803] 2025-04-26 21:52:40,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:40,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7681
[WARNING|trainer.py:803] 2025-04-26 21:52:41,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6554
[WARNING|trainer.py:803] 2025-04-26 21:52:41,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6498
[WARNING|trainer.py:803] 2025-04-26 21:52:41,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7682
6555
[WARNING|trainer.py:803] 2025-04-26 21:52:42,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:42,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6499
[WARNING|trainer.py:803] 2025-04-26 21:52:43,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7683
6556
[WARNING|trainer.py:803] 2025-04-26 21:52:43,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:44,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6500
[WARNING|trainer.py:803] 2025-04-26 21:52:44,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7684
6557
[WARNING|trainer.py:803] 2025-04-26 21:52:45,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:45,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6501
7685
[WARNING|trainer.py:803] 2025-04-26 21:52:45,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6558
[WARNING|trainer.py:803] 2025-04-26 21:52:46,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:46,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6502
7686
[WARNING|trainer.py:803] 2025-04-26 21:52:47,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6559
[WARNING|trainer.py:803] 2025-04-26 21:52:47,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:52:47,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6503
7687
[WARNING|trainer.py:803] 2025-04-26 21:52:48,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6560
[WARNING|trainer.py:803] 2025-04-26 21:52:49,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:49,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6504
7688
[WARNING|trainer.py:803] 2025-04-26 21:52:49,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6561
[WARNING|trainer.py:803] 2025-04-26 21:52:50,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:50,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7689
6505
[WARNING|trainer.py:803] 2025-04-26 21:52:51,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6562
[WARNING|trainer.py:803] 2025-04-26 21:52:51,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:51,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7690
6506
[WARNING|trainer.py:803] 2025-04-26 21:52:52,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6563
[WARNING|trainer.py:803] 2025-04-26 21:52:52,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:53,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7691
6507
[WARNING|trainer.py:803] 2025-04-26 21:52:53,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6564
[WARNING|trainer.py:803] 2025-04-26 21:52:54,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:54,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7692
[WARNING|trainer.py:803] 2025-04-26 21:52:54,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6508
6565
[WARNING|trainer.py:803] 2025-04-26 21:52:55,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:55,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7693
[WARNING|trainer.py:803] 2025-04-26 21:52:56,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6509
6566
[WARNING|trainer.py:803] 2025-04-26 21:52:56,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:52:57,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7694
[WARNING|trainer.py:803] 2025-04-26 21:52:57,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6510
6567
[WARNING|trainer.py:803] 2025-04-26 21:52:58,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:52:58,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7695
[WARNING|trainer.py:803] 2025-04-26 21:52:58,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6511
6568
[WARNING|trainer.py:803] 2025-04-26 21:52:59,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:52:59,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7696
[WARNING|trainer.py:803] 2025-04-26 21:53:00,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6512
6569
[WARNING|trainer.py:803] 2025-04-26 21:53:00,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:01,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7697
[WARNING|trainer.py:803] 2025-04-26 21:53:01,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6513
6570
[WARNING|trainer.py:803] 2025-04-26 21:53:02,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:02,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7698
[WARNING|trainer.py:803] 2025-04-26 21:53:02,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6514
6571
[WARNING|trainer.py:803] 2025-04-26 21:53:03,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7699
[WARNING|trainer.py:803] 2025-04-26 21:53:03,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:04,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6515
[WARNING|trainer.py:803] 2025-04-26 21:53:04,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6572
7700
[WARNING|trainer.py:803] 2025-04-26 21:53:05,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:05,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6516
[WARNING|trainer.py:803] 2025-04-26 21:53:05,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6573
7701
[WARNING|trainer.py:803] 2025-04-26 21:53:06,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:06,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6517
[WARNING|trainer.py:803] 2025-04-26 21:53:07,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6574
7702
[WARNING|trainer.py:803] 2025-04-26 21:53:07,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:07,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6518
[WARNING|trainer.py:803] 2025-04-26 21:53:08,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6575
7703
[WARNING|trainer.py:803] 2025-04-26 21:53:09,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:09,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:09,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6519
6576
7704
[WARNING|trainer.py:803] 2025-04-26 21:53:10,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:10,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:10,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6520
6577
7705
[WARNING|trainer.py:803] 2025-04-26 21:53:11,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:11,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:12,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6521
7706
6578
[WARNING|trainer.py:803] 2025-04-26 21:53:13,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:13,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:13,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7707
6522
6579
[WARNING|trainer.py:803] 2025-04-26 21:53:14,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:14,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:14,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7708
6523
6580
[WARNING|trainer.py:803] 2025-04-26 21:53:15,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:15,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:15,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7709
6524
6581
[WARNING|trainer.py:803] 2025-04-26 21:53:16,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:17,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:17,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7710
6525
6582
[WARNING|trainer.py:803] 2025-04-26 21:53:18,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:18,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7711
[WARNING|trainer.py:803] 2025-04-26 21:53:18,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6526
6583
[WARNING|trainer.py:803] 2025-04-26 21:53:19,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7712
[WARNING|trainer.py:803] 2025-04-26 21:53:19,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:19,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6527
6584
[WARNING|trainer.py:803] 2025-04-26 21:53:20,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7713
[WARNING|trainer.py:803] 2025-04-26 21:53:21,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:21,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6528
[WARNING|trainer.py:803] 2025-04-26 21:53:21,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6585
7714
[WARNING|trainer.py:803] 2025-04-26 21:53:22,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:22,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:22,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6529
6586
7715
[WARNING|trainer.py:803] 2025-04-26 21:53:23,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:23,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:24,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6530
7716
6587
[WARNING|trainer.py:803] 2025-04-26 21:53:25,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:25,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:25,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6531
7717
6588
[WARNING|trainer.py:803] 2025-04-26 21:53:26,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:26,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:26,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7718
6532
6589
[WARNING|trainer.py:803] 2025-04-26 21:53:27,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:27,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7719
[WARNING|trainer.py:803] 2025-04-26 21:53:27,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6533
6590
[WARNING|trainer.py:803] 2025-04-26 21:53:28,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:29,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7720
[WARNING|trainer.py:803] 2025-04-26 21:53:29,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6534
6591
[WARNING|trainer.py:803] 2025-04-26 21:53:30,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:30,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7721
[WARNING|trainer.py:803] 2025-04-26 21:53:30,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6535
6592
[WARNING|trainer.py:803] 2025-04-26 21:53:31,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7722
[WARNING|trainer.py:803] 2025-04-26 21:53:31,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:32,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6536
6593
[WARNING|trainer.py:803] 2025-04-26 21:53:32,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7723
[WARNING|trainer.py:803] 2025-04-26 21:53:33,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:33,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6537
[WARNING|trainer.py:803] 2025-04-26 21:53:33,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6594
7724
[WARNING|trainer.py:803] 2025-04-26 21:53:34,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:34,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6538
[WARNING|trainer.py:803] 2025-04-26 21:53:34,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6595
7725
[WARNING|trainer.py:803] 2025-04-26 21:53:35,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:35,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6539
[WARNING|trainer.py:803] 2025-04-26 21:53:36,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7726
6596
[WARNING|trainer.py:803] 2025-04-26 21:53:36,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:37,307 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:37,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6540
7727
6597
[WARNING|trainer.py:803] 2025-04-26 21:53:38,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:38,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:38,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6541
7728
6598
[WARNING|trainer.py:803] 2025-04-26 21:53:39,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:39,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6542
7729
[WARNING|trainer.py:803] 2025-04-26 21:53:39,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6599
[WARNING|trainer.py:803] 2025-04-26 21:53:40,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:40,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7730
6543
[WARNING|trainer.py:803] 2025-04-26 21:53:41,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6600
[WARNING|trainer.py:803] 2025-04-26 21:53:41,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:42,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7731
6544
[WARNING|trainer.py:803] 2025-04-26 21:53:42,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6601
[WARNING|trainer.py:803] 2025-04-26 21:53:43,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:43,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7732
6545
[WARNING|trainer.py:803] 2025-04-26 21:53:43,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:44,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6602
7733
[WARNING|trainer.py:803] 2025-04-26 21:53:44,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6546
[WARNING|trainer.py:803] 2025-04-26 21:53:45,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:45,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6603
7734
[WARNING|trainer.py:803] 2025-04-26 21:53:45,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6547
[WARNING|trainer.py:803] 2025-04-26 21:53:46,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:46,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7735
6604
[WARNING|trainer.py:803] 2025-04-26 21:53:47,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6548
[WARNING|trainer.py:803] 2025-04-26 21:53:47,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:47,885 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7736
6605
[WARNING|trainer.py:803] 2025-04-26 21:53:48,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:48,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6549
[WARNING|trainer.py:803] 2025-04-26 21:53:49,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7737
6606
[WARNING|trainer.py:803] 2025-04-26 21:53:49,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:50,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6550
7738
[WARNING|trainer.py:803] 2025-04-26 21:53:50,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6607
[WARNING|trainer.py:803] 2025-04-26 21:53:51,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:53:51,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7739
6551
[WARNING|trainer.py:803] 2025-04-26 21:53:51,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
[WARNING|trainer.py:803] 2025-04-26 21:53:52,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:52,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7740
6552
[WARNING|trainer.py:803] 2025-04-26 21:53:53,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:53:53,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6609
[WARNING|trainer.py:803] 2025-04-26 21:53:53,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7741
6553
[WARNING|trainer.py:803] 2025-04-26 21:53:54,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:54,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6610
[WARNING|trainer.py:803] 2025-04-26 21:53:55,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7742
6554
[WARNING|trainer.py:803] 2025-04-26 21:53:55,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:56,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6611
7743
[WARNING|trainer.py:803] 2025-04-26 21:53:56,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6555
[WARNING|trainer.py:803] 2025-04-26 21:53:57,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:57,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7744
6612
[WARNING|trainer.py:803] 2025-04-26 21:53:57,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6556
[WARNING|trainer.py:803] 2025-04-26 21:53:58,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:53:58,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7745
6613
[WARNING|trainer.py:803] 2025-04-26 21:53:59,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:53:59,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6557
[WARNING|trainer.py:803] 2025-04-26 21:53:59,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7746
6614
[WARNING|trainer.py:803] 2025-04-26 21:54:00,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:00,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6558
[WARNING|trainer.py:803] 2025-04-26 21:54:01,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7747
6615
[WARNING|trainer.py:803] 2025-04-26 21:54:01,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:01,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6559
7748
[WARNING|trainer.py:803] 2025-04-26 21:54:02,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6616
[WARNING|trainer.py:803] 2025-04-26 21:54:03,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:03,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7749
6560
[WARNING|trainer.py:803] 2025-04-26 21:54:03,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6617
[WARNING|trainer.py:803] 2025-04-26 21:54:04,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:04,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7750
[WARNING|trainer.py:803] 2025-04-26 21:54:05,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6561
6618
[WARNING|trainer.py:803] 2025-04-26 21:54:05,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:05,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7751
[WARNING|trainer.py:803] 2025-04-26 21:54:06,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6562
[WARNING|trainer.py:803] 2025-04-26 21:54:06,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6619
7752
[WARNING|trainer.py:803] 2025-04-26 21:54:07,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:07,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6563
[WARNING|trainer.py:803] 2025-04-26 21:54:07,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6620
7753
[WARNING|trainer.py:803] 2025-04-26 21:54:08,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:08,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6564
[WARNING|trainer.py:803] 2025-04-26 21:54:09,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7754
6621
[WARNING|trainer.py:803] 2025-04-26 21:54:09,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:10,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:10,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6565
7755
6622
[WARNING|trainer.py:803] 2025-04-26 21:54:11,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:11,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:11,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6566
7756
6623
[WARNING|trainer.py:803] 2025-04-26 21:54:12,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:12,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7757
[WARNING|trainer.py:803] 2025-04-26 21:54:12,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6567
6624
[WARNING|trainer.py:803] 2025-04-26 21:54:13,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:13,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7758
[WARNING|trainer.py:803] 2025-04-26 21:54:14,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6568
6625
[WARNING|trainer.py:803] 2025-04-26 21:54:14,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:15,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7759
[WARNING|trainer.py:803] 2025-04-26 21:54:15,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6569
6626
[WARNING|trainer.py:803] 2025-04-26 21:54:16,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7760
[WARNING|trainer.py:803] 2025-04-26 21:54:16,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:16,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6570
[WARNING|trainer.py:803] 2025-04-26 21:54:17,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6627
7761
[WARNING|trainer.py:803] 2025-04-26 21:54:17,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:18,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6571
[WARNING|trainer.py:803] 2025-04-26 21:54:18,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6628
7762
[WARNING|trainer.py:803] 2025-04-26 21:54:19,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:19,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:19,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6572
7763
6629
[WARNING|trainer.py:803] 2025-04-26 21:54:20,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:20,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:20,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6573
7764
6630
[WARNING|trainer.py:803] 2025-04-26 21:54:21,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:21,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7765
[WARNING|trainer.py:803] 2025-04-26 21:54:22,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6574
6631
[WARNING|trainer.py:803] 2025-04-26 21:54:23,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:23,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7766
[WARNING|trainer.py:803] 2025-04-26 21:54:23,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6575
6632
[WARNING|trainer.py:803] 2025-04-26 21:54:24,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7767
[WARNING|trainer.py:803] 2025-04-26 21:54:24,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:24,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6576
[WARNING|trainer.py:803] 2025-04-26 21:54:25,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6633
7768
[WARNING|trainer.py:803] 2025-04-26 21:54:25,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:26,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6577
[WARNING|trainer.py:803] 2025-04-26 21:54:26,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6634
7769
[WARNING|trainer.py:803] 2025-04-26 21:54:27,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:27,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:27,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6578
7770
6635
[WARNING|trainer.py:803] 2025-04-26 21:54:28,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:28,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:54:28,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6579
7771
6636
[WARNING|trainer.py:803] 2025-04-26 21:54:29,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:29,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:30,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6580
7772
6637
[WARNING|trainer.py:803] 2025-04-26 21:54:31,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:31,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:31,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7773
6581
6638
[WARNING|trainer.py:803] 2025-04-26 21:54:32,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:32,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7774
[WARNING|trainer.py:803] 2025-04-26 21:54:32,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6582
6639
[WARNING|trainer.py:803] 2025-04-26 21:54:33,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7775
[WARNING|trainer.py:803] 2025-04-26 21:54:33,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:34,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6583
6640
[WARNING|trainer.py:803] 2025-04-26 21:54:34,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7776
[WARNING|trainer.py:803] 2025-04-26 21:54:35,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:35,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6584
[WARNING|trainer.py:803] 2025-04-26 21:54:35,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6641
7777
[WARNING|trainer.py:803] 2025-04-26 21:54:36,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:36,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:37,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6585
6642
7778
[WARNING|trainer.py:803] 2025-04-26 21:54:37,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:38,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:38,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7779
6586
6643
[WARNING|trainer.py:803] 2025-04-26 21:54:39,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:39,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:39,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7780
6587
6644
[WARNING|trainer.py:803] 2025-04-26 21:54:40,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:40,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:40,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7781
6588
6645
[WARNING|trainer.py:803] 2025-04-26 21:54:41,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:41,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7782
[WARNING|trainer.py:803] 2025-04-26 21:54:42,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6589
6646
[WARNING|trainer.py:803] 2025-04-26 21:54:42,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7783
[WARNING|trainer.py:803] 2025-04-26 21:54:43,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:43,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6590
6647
[WARNING|trainer.py:803] 2025-04-26 21:54:44,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7784
[WARNING|trainer.py:803] 2025-04-26 21:54:44,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:44,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6591
[WARNING|trainer.py:803] 2025-04-26 21:54:45,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6648
7785
[WARNING|trainer.py:803] 2025-04-26 21:54:46,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:46,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:46,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6592
6649
7786
[WARNING|trainer.py:803] 2025-04-26 21:54:47,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:47,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:47,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6593
6650
7787
[WARNING|trainer.py:803] 2025-04-26 21:54:48,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:48,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:54:48,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7788
6594
6651
[WARNING|trainer.py:803] 2025-04-26 21:54:49,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:50,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:50,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7789
6595
6652
[WARNING|trainer.py:803] 2025-04-26 21:54:51,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:51,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:51,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7790
6596
6653
[WARNING|trainer.py:803] 2025-04-26 21:54:52,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:52,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7791
[WARNING|trainer.py:803] 2025-04-26 21:54:52,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6597
6654
[WARNING|trainer.py:803] 2025-04-26 21:54:53,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7792
[WARNING|trainer.py:803] 2025-04-26 21:54:54,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:54,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6598
6655
[WARNING|trainer.py:803] 2025-04-26 21:54:54,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7793
[WARNING|trainer.py:803] 2025-04-26 21:54:55,385 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:55,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6599
[WARNING|trainer.py:803] 2025-04-26 21:54:55,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6656
7794
[WARNING|trainer.py:803] 2025-04-26 21:54:56,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:56,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:57,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6600
6657
7795
[WARNING|trainer.py:803] 2025-04-26 21:54:58,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:58,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:58,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6601
6658
7796
[WARNING|trainer.py:803] 2025-04-26 21:54:59,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:54:59,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:54:59,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6602
7797
6659
[WARNING|trainer.py:803] 2025-04-26 21:55:00,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:00,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:00,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7798
6603
6660
[WARNING|trainer.py:803] 2025-04-26 21:55:01,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:02,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:02,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7799
6604
6661
[WARNING|trainer.py:803] 2025-04-26 21:55:03,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:03,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:03,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7800
6605
6662
[WARNING|trainer.py:803] 2025-04-26 21:55:04,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:04,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:04,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7801
6606
6663
[WARNING|trainer.py:803] 2025-04-26 21:55:05,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:06,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:06,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7802
6607
6664
[WARNING|trainer.py:803] 2025-04-26 21:55:07,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:07,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:07,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6608
6665
7803
[WARNING|trainer.py:803] 2025-04-26 21:55:08,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:08,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:09,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6609
6666
7804
[WARNING|trainer.py:803] 2025-04-26 21:55:10,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:10,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6610
[WARNING|trainer.py:803] 2025-04-26 21:55:10,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6667
7805
[WARNING|trainer.py:803] 2025-04-26 21:55:11,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:11,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6611
6668
[WARNING|trainer.py:803] 2025-04-26 21:55:12,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:12,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7806
[WARNING|trainer.py:803] 2025-04-26 21:55:12,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6612
6669
[WARNING|trainer.py:803] 2025-04-26 21:55:13,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:13,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:14,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7807
6613
6670
[WARNING|trainer.py:803] 2025-04-26 21:55:15,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:15,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:15,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7808
6614
6671
[WARNING|trainer.py:803] 2025-04-26 21:55:16,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:16,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:16,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6615
6672
7809
[WARNING|trainer.py:803] 2025-04-26 21:55:17,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:18,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:18,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6616
6673
7810
[WARNING|trainer.py:803] 2025-04-26 21:55:19,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:19,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:19,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6617
6674
7811
[WARNING|trainer.py:803] 2025-04-26 21:55:20,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:20,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6618
6675
[WARNING|trainer.py:803] 2025-04-26 21:55:21,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7812
[WARNING|trainer.py:803] 2025-04-26 21:55:21,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:22,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6619
6676
[WARNING|trainer.py:803] 2025-04-26 21:55:22,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:23,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:23,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7813
6620
6677
[WARNING|trainer.py:803] 2025-04-26 21:55:24,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:24,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:24,683 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7814
6621
6678
[WARNING|trainer.py:803] 2025-04-26 21:55:25,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:25,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:25,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7815
6679
6622
[WARNING|trainer.py:803] 2025-04-26 21:55:27,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:27,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:27,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6680
6623
7816
[WARNING|trainer.py:803] 2025-04-26 21:55:28,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:28,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:28,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6681
6624
7817
[WARNING|trainer.py:803] 2025-04-26 21:55:29,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:30,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:30,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6682
6625
7818
[WARNING|trainer.py:803] 2025-04-26 21:55:31,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:31,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:31,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6683
6626
7819
[WARNING|trainer.py:803] 2025-04-26 21:55:32,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:32,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6627
6684
[WARNING|trainer.py:803] 2025-04-26 21:55:33,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7820
[WARNING|trainer.py:803] 2025-04-26 21:55:34,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:34,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6685
6628
[WARNING|trainer.py:803] 2025-04-26 21:55:34,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:35,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:35,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7821
6686
6629
[WARNING|trainer.py:803] 2025-04-26 21:55:36,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:36,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:36,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7822
6687
6630
[WARNING|trainer.py:803] 2025-04-26 21:55:37,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:37,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:38,052 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6688
6631
7823
[WARNING|trainer.py:803] 2025-04-26 21:55:39,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:39,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:39,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6689
6632
7824
[WARNING|trainer.py:803] 2025-04-26 21:55:40,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:40,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6690
6633
[WARNING|trainer.py:803] 2025-04-26 21:55:41,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7825
[WARNING|trainer.py:803] 2025-04-26 21:55:41,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:42,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6691
6634
[WARNING|trainer.py:803] 2025-04-26 21:55:42,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7826
[WARNING|trainer.py:803] 2025-04-26 21:55:43,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6692
6635
[WARNING|trainer.py:803] 2025-04-26 21:55:43,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:44,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7827
[WARNING|trainer.py:803] 2025-04-26 21:55:44,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6693
6636
[WARNING|trainer.py:803] 2025-04-26 21:55:45,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:45,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:46,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6694
7828
6637
[WARNING|trainer.py:803] 2025-04-26 21:55:47,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:47,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:47,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6695
6638
7829
[WARNING|trainer.py:803] 2025-04-26 21:55:48,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:48,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6696
[WARNING|trainer.py:803] 2025-04-26 21:55:49,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6639
7830
[WARNING|trainer.py:803] 2025-04-26 21:55:49,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:50,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6697
[WARNING|trainer.py:803] 2025-04-26 21:55:50,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6640
[WARNING|trainer.py:803] 2025-04-26 21:55:51,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:51,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7831
6698
6641
[WARNING|trainer.py:803] 2025-04-26 21:55:52,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:52,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:55:52,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7832
6699
6642
[WARNING|trainer.py:803] 2025-04-26 21:55:53,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:53,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:54,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7833
6700
6643
[WARNING|trainer.py:803] 2025-04-26 21:55:55,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:55:55,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:55,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6701
7834
6644
[WARNING|trainer.py:803] 2025-04-26 21:55:56,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:55:56,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:56,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6702
6645
7835
[WARNING|trainer.py:803] 2025-04-26 21:55:57,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:55:58,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:58,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6703
6646
7836
[WARNING|trainer.py:803] 2025-04-26 21:55:59,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:55:59,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:55:59,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6704
6647
7837
[WARNING|trainer.py:803] 2025-04-26 21:56:00,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:00,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:00,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6705
6648
7838
[WARNING|trainer.py:803] 2025-04-26 21:56:01,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:02,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6706
[WARNING|trainer.py:803] 2025-04-26 21:56:02,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6649
7839
[WARNING|trainer.py:803] 2025-04-26 21:56:03,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:03,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6707
6650
[WARNING|trainer.py:803] 2025-04-26 21:56:03,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7840
[WARNING|trainer.py:803] 2025-04-26 21:56:04,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:04,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6708
6651
[WARNING|trainer.py:803] 2025-04-26 21:56:05,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7841
[WARNING|trainer.py:803] 2025-04-26 21:56:05,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:06,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6709
6652
[WARNING|trainer.py:803] 2025-04-26 21:56:06,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:07,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:07,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6710
6653
7842
[WARNING|trainer.py:803] 2025-04-26 21:56:08,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:08,774 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:08,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6711
6654
7843
[WARNING|trainer.py:803] 2025-04-26 21:56:09,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:10,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6712
[WARNING|trainer.py:803] 2025-04-26 21:56:10,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6655
[WARNING|trainer.py:803] 2025-04-26 21:56:11,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:11,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6713
7844
6656
[WARNING|trainer.py:803] 2025-04-26 21:56:12,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:12,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:12,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6714
7845
6657
[WARNING|trainer.py:803] 2025-04-26 21:56:13,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:13,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:14,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6715
6658
7846
[WARNING|trainer.py:803] 2025-04-26 21:56:15,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:15,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6716
[WARNING|trainer.py:803] 2025-04-26 21:56:15,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6659
[WARNING|trainer.py:803] 2025-04-26 21:56:16,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7847
[WARNING|trainer.py:803] 2025-04-26 21:56:16,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6717
6660
[WARNING|trainer.py:803] 2025-04-26 21:56:17,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:17,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7848
[WARNING|trainer.py:803] 2025-04-26 21:56:18,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6718
6661
[WARNING|trainer.py:803] 2025-04-26 21:56:18,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:19,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:19,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7849
6719
6662
[WARNING|trainer.py:803] 2025-04-26 21:56:20,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:20,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:20,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6720
6663
7850
[WARNING|trainer.py:803] 2025-04-26 21:56:21,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:22,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:22,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6721
6664
[WARNING|trainer.py:803] 2025-04-26 21:56:23,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7851
[WARNING|trainer.py:803] 2025-04-26 21:56:23,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6722
6665
[WARNING|trainer.py:803] 2025-04-26 21:56:24,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:24,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:24,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6723
7852
6666
[WARNING|trainer.py:803] 2025-04-26 21:56:25,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:26,147 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6724
[WARNING|trainer.py:803] 2025-04-26 21:56:26,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7853
6667
[WARNING|trainer.py:803] 2025-04-26 21:56:27,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:27,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6725
[WARNING|trainer.py:803] 2025-04-26 21:56:27,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6668
7854
[WARNING|trainer.py:803] 2025-04-26 21:56:28,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6726
[WARNING|trainer.py:803] 2025-04-26 21:56:29,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:29,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6669
7855
[WARNING|trainer.py:803] 2025-04-26 21:56:29,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6727
[WARNING|trainer.py:803] 2025-04-26 21:56:30,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:30,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6670
7856
[WARNING|trainer.py:803] 2025-04-26 21:56:31,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6728
[WARNING|trainer.py:803] 2025-04-26 21:56:31,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:32,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6671
[WARNING|trainer.py:803] 2025-04-26 21:56:32,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7857
6729
[WARNING|trainer.py:803] 2025-04-26 21:56:33,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:33,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6672
[WARNING|trainer.py:803] 2025-04-26 21:56:33,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7858
6730
[WARNING|trainer.py:803] 2025-04-26 21:56:34,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:34,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6673
[WARNING|trainer.py:803] 2025-04-26 21:56:35,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7859
6731
[WARNING|trainer.py:803] 2025-04-26 21:56:35,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6674
[WARNING|trainer.py:803] 2025-04-26 21:56:36,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:36,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7860
6732
[WARNING|trainer.py:803] 2025-04-26 21:56:36,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6675
[WARNING|trainer.py:803] 2025-04-26 21:56:37,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:37,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7861
6733
[WARNING|trainer.py:803] 2025-04-26 21:56:38,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6676
[WARNING|trainer.py:803] 2025-04-26 21:56:39,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:39,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6734
7862
[WARNING|trainer.py:803] 2025-04-26 21:56:39,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6677
[WARNING|trainer.py:803] 2025-04-26 21:56:40,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:40,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6735
7863
[WARNING|trainer.py:803] 2025-04-26 21:56:40,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6678
[WARNING|trainer.py:803] 2025-04-26 21:56:41,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:41,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6736
[WARNING|trainer.py:803] 2025-04-26 21:56:42,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7864
6679
[WARNING|trainer.py:803] 2025-04-26 21:56:43,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:43,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6737
[WARNING|trainer.py:803] 2025-04-26 21:56:43,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6680
7865
[WARNING|trainer.py:803] 2025-04-26 21:56:44,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6738
[WARNING|trainer.py:803] 2025-04-26 21:56:44,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:45,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6681
7866
[WARNING|trainer.py:803] 2025-04-26 21:56:45,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6739
[WARNING|trainer.py:803] 2025-04-26 21:56:46,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:46,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6682
7867
[WARNING|trainer.py:803] 2025-04-26 21:56:46,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6740
[WARNING|trainer.py:803] 2025-04-26 21:56:47,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:47,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6683
[WARNING|trainer.py:803] 2025-04-26 21:56:48,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6741
[WARNING|trainer.py:803] 2025-04-26 21:56:48,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7868
6684
[WARNING|trainer.py:803] 2025-04-26 21:56:49,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:49,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6742
[WARNING|trainer.py:803] 2025-04-26 21:56:50,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7869
6685
[WARNING|trainer.py:803] 2025-04-26 21:56:50,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:51,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6743
[WARNING|trainer.py:803] 2025-04-26 21:56:51,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6686
[WARNING|trainer.py:803] 2025-04-26 21:56:52,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7870
6744
[WARNING|trainer.py:803] 2025-04-26 21:56:52,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:56:53,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6687
[WARNING|trainer.py:803] 2025-04-26 21:56:53,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7871
6745
[WARNING|trainer.py:803] 2025-04-26 21:56:54,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:56:54,686 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6688
[WARNING|trainer.py:803] 2025-04-26 21:56:54,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7872
6746
[WARNING|trainer.py:803] 2025-04-26 21:56:55,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6689
[WARNING|trainer.py:803] 2025-04-26 21:56:56,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:56,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7873
6747
[WARNING|trainer.py:803] 2025-04-26 21:56:56,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6690
[WARNING|trainer.py:803] 2025-04-26 21:56:57,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:56:57,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7874
6748
[WARNING|trainer.py:803] 2025-04-26 21:56:58,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6691
[WARNING|trainer.py:803] 2025-04-26 21:56:58,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:56:58,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6749
[WARNING|trainer.py:803] 2025-04-26 21:56:59,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6692
[WARNING|trainer.py:803] 2025-04-26 21:57:00,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7875
6750
[WARNING|trainer.py:803] 2025-04-26 21:57:00,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:01,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6693
[WARNING|trainer.py:803] 2025-04-26 21:57:01,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7876
6751
[WARNING|trainer.py:803] 2025-04-26 21:57:02,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:02,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6694
[WARNING|trainer.py:803] 2025-04-26 21:57:02,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6752
7877
[WARNING|trainer.py:803] 2025-04-26 21:57:03,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6695
[WARNING|trainer.py:803] 2025-04-26 21:57:03,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:04,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6753
[WARNING|trainer.py:803] 2025-04-26 21:57:04,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7878
6696
[WARNING|trainer.py:803] 2025-04-26 21:57:05,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:05,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6754
[WARNING|trainer.py:803] 2025-04-26 21:57:06,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7879
6697
[WARNING|trainer.py:803] 2025-04-26 21:57:06,579 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6755
[WARNING|trainer.py:803] 2025-04-26 21:57:07,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:07,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6698
[WARNING|trainer.py:803] 2025-04-26 21:57:07,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7880
6756
[WARNING|trainer.py:803] 2025-04-26 21:57:08,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:08,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:09,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6699
7881
6757
[WARNING|trainer.py:803] 2025-04-26 21:57:10,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:10,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:10,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6700
6758
7882
[WARNING|trainer.py:803] 2025-04-26 21:57:11,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:11,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6701
[WARNING|trainer.py:803] 2025-04-26 21:57:11,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6759
[WARNING|trainer.py:803] 2025-04-26 21:57:12,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:57:13,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7883
6702
6760
[WARNING|trainer.py:803] 2025-04-26 21:57:13,943 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:13,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:14,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6703
6761
7884
[WARNING|trainer.py:803] 2025-04-26 21:57:15,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 21:57:15,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6704
[WARNING|trainer.py:803] 2025-04-26 21:57:16,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6762
[WARNING|trainer.py:803] 2025-04-26 21:57:16,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7885
[WARNING|trainer.py:803] 2025-04-26 21:57:17,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6705
6763
[WARNING|trainer.py:803] 2025-04-26 21:57:17,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:17,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:18,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7886
6706
6764
[WARNING|trainer.py:803] 2025-04-26 21:57:19,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:19,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:19,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6707
7887
6765
[WARNING|trainer.py:803] 2025-04-26 21:57:20,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:20,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:21,014 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6708
7888
6766
[WARNING|trainer.py:803] 2025-04-26 21:57:21,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:22,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:22,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6709
7889
6767
[WARNING|trainer.py:803] 2025-04-26 21:57:23,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:23,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:23,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6710
7890
6768
[WARNING|trainer.py:803] 2025-04-26 21:57:24,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:24,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:24,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6711
6769
7891
[WARNING|trainer.py:803] 2025-04-26 21:57:25,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:26,325 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6712
[WARNING|trainer.py:803] 2025-04-26 21:57:26,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6770
[WARNING|trainer.py:803] 2025-04-26 21:57:27,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7892
6713
[WARNING|trainer.py:803] 2025-04-26 21:57:27,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:28,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6771
[WARNING|trainer.py:803] 2025-04-26 21:57:28,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7893
6714
[WARNING|trainer.py:803] 2025-04-26 21:57:29,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6772
[WARNING|trainer.py:803] 2025-04-26 21:57:29,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:29,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6715
[WARNING|trainer.py:803] 2025-04-26 21:57:30,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6773
7894
[WARNING|trainer.py:803] 2025-04-26 21:57:31,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:31,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6716
[WARNING|trainer.py:803] 2025-04-26 21:57:31,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6774
[WARNING|trainer.py:803] 2025-04-26 21:57:32,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7895
[WARNING|trainer.py:803] 2025-04-26 21:57:33,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6717
[WARNING|trainer.py:803] 2025-04-26 21:57:33,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6775
[WARNING|trainer.py:803] 2025-04-26 21:57:33,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7896
[WARNING|trainer.py:803] 2025-04-26 21:57:34,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6718
6776
[WARNING|trainer.py:803] 2025-04-26 21:57:35,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:35,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:35,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7897
6719
6777
[WARNING|trainer.py:803] 2025-04-26 21:57:36,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:36,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:36,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6720
6778
7898
[WARNING|trainer.py:803] 2025-04-26 21:57:38,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:38,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:38,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6721
6779
7899
[WARNING|trainer.py:803] 2025-04-26 21:57:39,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:39,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6722
[WARNING|trainer.py:803] 2025-04-26 21:57:39,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6780
7900
[WARNING|trainer.py:803] 2025-04-26 21:57:40,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:40,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6723
6781
[WARNING|trainer.py:803] 2025-04-26 21:57:41,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:57:42,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:42,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7901
6724
6782
[WARNING|trainer.py:803] 2025-04-26 21:57:43,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:43,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:43,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6725
7902
6783
[WARNING|trainer.py:803] 2025-04-26 21:57:44,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:44,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:44,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6726
6784
7903
[WARNING|trainer.py:803] 2025-04-26 21:57:46,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:46,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:46,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6785
6727
7904
[WARNING|trainer.py:803] 2025-04-26 21:57:47,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:47,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6786
6728
[WARNING|trainer.py:803] 2025-04-26 21:57:48,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7905
[WARNING|trainer.py:803] 2025-04-26 21:57:48,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:48,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6787
6729
[WARNING|trainer.py:803] 2025-04-26 21:57:49,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:50,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:50,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7906
6788
6730
[WARNING|trainer.py:803] 2025-04-26 21:57:51,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:51,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:51,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7907
6789
6731
[WARNING|trainer.py:803] 2025-04-26 21:57:52,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:52,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:52,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7908
6790
6732
[WARNING|trainer.py:803] 2025-04-26 21:57:54,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:54,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:54,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7909
6791
6733
[WARNING|trainer.py:803] 2025-04-26 21:57:55,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:55,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:55,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6792
6734
7910
[WARNING|trainer.py:803] 2025-04-26 21:57:56,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:56,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:57,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6735
6793
7911
[WARNING|trainer.py:803] 2025-04-26 21:57:58,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:58,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:57:58,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6736
6794
7912
[WARNING|trainer.py:803] 2025-04-26 21:57:59,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:57:59,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:57:59,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6795
6737
7913
[WARNING|trainer.py:803] 2025-04-26 21:58:00,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:00,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:01,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6796
6738
7914
[WARNING|trainer.py:803] 2025-04-26 21:58:02,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:02,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6797
6739
[WARNING|trainer.py:803] 2025-04-26 21:58:02,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7915
[WARNING|trainer.py:803] 2025-04-26 21:58:03,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:03,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6798
6740
[WARNING|trainer.py:803] 2025-04-26 21:58:04,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:04,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:04,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7916
6799
6741
[WARNING|trainer.py:803] 2025-04-26 21:58:05,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:06,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:06,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6800
7917
6742
[WARNING|trainer.py:803] 2025-04-26 21:58:07,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:07,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:07,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6801
6743
7918
[WARNING|trainer.py:803] 2025-04-26 21:58:08,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:08,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6802
[WARNING|trainer.py:803] 2025-04-26 21:58:09,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6744
7919
[WARNING|trainer.py:803] 2025-04-26 21:58:10,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:10,282 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6803
6745
[WARNING|trainer.py:803] 2025-04-26 21:58:10,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:11,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:11,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6804
7920
6746
[WARNING|trainer.py:803] 2025-04-26 21:58:12,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:12,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:12,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6805
6747
7921
[WARNING|trainer.py:803] 2025-04-26 21:58:14,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:14,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6806
[WARNING|trainer.py:803] 2025-04-26 21:58:14,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6748
[WARNING|trainer.py:803] 2025-04-26 21:58:15,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:15,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6807
7922
6749
[WARNING|trainer.py:803] 2025-04-26 21:58:16,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:16,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:16,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6808
6750
7923
[WARNING|trainer.py:803] 2025-04-26 21:58:17,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:18,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:18,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6809
6751
[WARNING|trainer.py:803] 2025-04-26 21:58:19,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:19,600 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7924
6810
6752
[WARNING|trainer.py:803] 2025-04-26 21:58:20,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:20,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:20,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6811
7925
6753
[WARNING|trainer.py:803] 2025-04-26 21:58:21,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:21,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:22,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6812
7926
6754
[WARNING|trainer.py:803] 2025-04-26 21:58:23,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:23,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:23,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6813
6755
[WARNING|trainer.py:803] 2025-04-26 21:58:24,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7927
[WARNING|trainer.py:803] 2025-04-26 21:58:24,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6814
6756
[WARNING|trainer.py:803] 2025-04-26 21:58:25,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:25,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7928
[WARNING|trainer.py:803] 2025-04-26 21:58:26,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6815
6757
[WARNING|trainer.py:803] 2025-04-26 21:58:26,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:27,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:27,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7929
6816
6758
[WARNING|trainer.py:803] 2025-04-26 21:58:28,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:28,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6817
[WARNING|trainer.py:803] 2025-04-26 21:58:28,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7930
6759
[WARNING|trainer.py:803] 2025-04-26 21:58:29,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:29,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6818
[WARNING|trainer.py:803] 2025-04-26 21:58:30,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7931
6760
[WARNING|trainer.py:803] 2025-04-26 21:58:30,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:31,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6819
[WARNING|trainer.py:803] 2025-04-26 21:58:31,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6761
7932
[WARNING|trainer.py:803] 2025-04-26 21:58:32,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6820
[WARNING|trainer.py:803] 2025-04-26 21:58:32,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:33,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6762
[WARNING|trainer.py:803] 2025-04-26 21:58:33,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7933
6821
[WARNING|trainer.py:803] 2025-04-26 21:58:34,069 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6763
[WARNING|trainer.py:803] 2025-04-26 21:58:34,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:58:34,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6822
[WARNING|trainer.py:803] 2025-04-26 21:58:35,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7934
6764
[WARNING|trainer.py:803] 2025-04-26 21:58:36,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:36,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6823
[WARNING|trainer.py:803] 2025-04-26 21:58:36,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6765
7935
[WARNING|trainer.py:803] 2025-04-26 21:58:37,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6824
[WARNING|trainer.py:803] 2025-04-26 21:58:38,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:38,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6766
7936
[WARNING|trainer.py:803] 2025-04-26 21:58:38,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6825
[WARNING|trainer.py:803] 2025-04-26 21:58:39,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:39,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6767
[WARNING|trainer.py:803] 2025-04-26 21:58:40,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6826
7937
[WARNING|trainer.py:803] 2025-04-26 21:58:40,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6768
[WARNING|trainer.py:803] 2025-04-26 21:58:41,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:41,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6827
[WARNING|trainer.py:803] 2025-04-26 21:58:41,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7938
6769
[WARNING|trainer.py:803] 2025-04-26 21:58:42,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:42,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6828
[WARNING|trainer.py:803] 2025-04-26 21:58:43,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7939
6770
[WARNING|trainer.py:803] 2025-04-26 21:58:43,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6829
[WARNING|trainer.py:803] 2025-04-26 21:58:44,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:44,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6771
7940
[WARNING|trainer.py:803] 2025-04-26 21:58:45,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6830
[WARNING|trainer.py:803] 2025-04-26 21:58:45,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:46,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6772
[WARNING|trainer.py:803] 2025-04-26 21:58:46,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7941
6831
[WARNING|trainer.py:803] 2025-04-26 21:58:47,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:47,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6773
[WARNING|trainer.py:803] 2025-04-26 21:58:47,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6832
7942
[WARNING|trainer.py:803] 2025-04-26 21:58:48,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6774
[WARNING|trainer.py:803] 2025-04-26 21:58:49,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:49,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6833
[WARNING|trainer.py:803] 2025-04-26 21:58:49,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7943
6775
[WARNING|trainer.py:803] 2025-04-26 21:58:50,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:50,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6834
[WARNING|trainer.py:803] 2025-04-26 21:58:51,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7944
6776
[WARNING|trainer.py:803] 2025-04-26 21:58:51,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6835
[WARNING|trainer.py:803] 2025-04-26 21:58:52,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:52,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6777
[WARNING|trainer.py:803] 2025-04-26 21:58:52,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6836
7945
[WARNING|trainer.py:803] 2025-04-26 21:58:53,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:54,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6778
[WARNING|trainer.py:803] 2025-04-26 21:58:54,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6837
7946
[WARNING|trainer.py:803] 2025-04-26 21:58:55,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:58:55,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6779
[WARNING|trainer.py:803] 2025-04-26 21:58:55,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6838
[WARNING|trainer.py:803] 2025-04-26 21:58:56,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7947
[WARNING|trainer.py:803] 2025-04-26 21:58:56,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6780
6839
[WARNING|trainer.py:803] 2025-04-26 21:58:57,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:57,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:58,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6781
7948
6840
[WARNING|trainer.py:803] 2025-04-26 21:58:59,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:58:59,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:58:59,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6782
7949
6841
[WARNING|trainer.py:803] 2025-04-26 21:59:00,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:00,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:00,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6783
6842
7950
[WARNING|trainer.py:803] 2025-04-26 21:59:01,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:02,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6784
[WARNING|trainer.py:803] 2025-04-26 21:59:02,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6843
[WARNING|trainer.py:803] 2025-04-26 21:59:02,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:03,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7951
6785
6844
[WARNING|trainer.py:803] 2025-04-26 21:59:04,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:04,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:04,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6786
6845
7952
[WARNING|trainer.py:803] 2025-04-26 21:59:05,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:05,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6787
[WARNING|trainer.py:803] 2025-04-26 21:59:06,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6846
[WARNING|trainer.py:803] 2025-04-26 21:59:06,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7953
[WARNING|trainer.py:803] 2025-04-26 21:59:07,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6788
6847
[WARNING|trainer.py:803] 2025-04-26 21:59:07,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:08,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7954
[WARNING|trainer.py:803] 2025-04-26 21:59:08,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6789
6848
[WARNING|trainer.py:803] 2025-04-26 21:59:09,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:09,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:09,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6790
7955
6849
[WARNING|trainer.py:803] 2025-04-26 21:59:10,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:11,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:11,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6791
7956
6850
[WARNING|trainer.py:803] 2025-04-26 21:59:12,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:12,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:12,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6792
6851
7957
[WARNING|trainer.py:803] 2025-04-26 21:59:13,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:13,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:13,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6793
6852
[WARNING|trainer.py:803] 2025-04-26 21:59:15,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:15,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7958
6794
6853
[WARNING|trainer.py:803] 2025-04-26 21:59:16,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:16,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:16,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7959
6795
6854
[WARNING|trainer.py:803] 2025-04-26 21:59:17,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:17,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:17,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6796
6855
7960
[WARNING|trainer.py:803] 2025-04-26 21:59:19,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:19,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:19,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6797
6856
7961
[WARNING|trainer.py:803] 2025-04-26 21:59:20,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:20,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:20,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6798
6857
7962
[WARNING|trainer.py:803] 2025-04-26 21:59:21,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:21,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6858
6799
[WARNING|trainer.py:803] 2025-04-26 21:59:22,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7963
[WARNING|trainer.py:803] 2025-04-26 21:59:23,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 21:59:23,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6800
6859
[WARNING|trainer.py:803] 2025-04-26 21:59:23,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:24,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:24,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6860
6801
7964
[WARNING|trainer.py:803] 2025-04-26 21:59:25,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:25,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:25,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6861
6802
7965
[WARNING|trainer.py:803] 2025-04-26 21:59:26,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:27,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:27,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6862
6803
7966
[WARNING|trainer.py:803] 2025-04-26 21:59:28,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:28,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6863
[WARNING|trainer.py:803] 2025-04-26 21:59:28,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6804
[WARNING|trainer.py:803] 2025-04-26 21:59:29,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:29,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7967
6864
6805
[WARNING|trainer.py:803] 2025-04-26 21:59:30,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:30,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:30,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6865
6806
7968
[WARNING|trainer.py:803] 2025-04-26 21:59:32,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:32,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:32,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6866
6807
7969
[WARNING|trainer.py:803] 2025-04-26 21:59:33,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:33,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6867
[WARNING|trainer.py:803] 2025-04-26 21:59:33,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6808
7970
[WARNING|trainer.py:803] 2025-04-26 21:59:34,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:34,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6868
[WARNING|trainer.py:803] 2025-04-26 21:59:35,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6809
7971
[WARNING|trainer.py:803] 2025-04-26 21:59:36,006 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:36,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6869
6810
[WARNING|trainer.py:803] 2025-04-26 21:59:36,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:37,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7972
[WARNING|trainer.py:803] 2025-04-26 21:59:37,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6870
6811
[WARNING|trainer.py:803] 2025-04-26 21:59:38,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:38,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7973
[WARNING|trainer.py:803] 2025-04-26 21:59:38,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6871
6812
[WARNING|trainer.py:803] 2025-04-26 21:59:39,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:39,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7974
[WARNING|trainer.py:803] 2025-04-26 21:59:40,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6872
6813
[WARNING|trainer.py:803] 2025-04-26 21:59:40,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:41,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:41,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7975
6873
6814
[WARNING|trainer.py:803] 2025-04-26 21:59:42,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:42,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:42,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6874
6815
7976
[WARNING|trainer.py:803] 2025-04-26 21:59:43,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:44,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6875
[WARNING|trainer.py:803] 2025-04-26 21:59:44,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6816
7977
[WARNING|trainer.py:803] 2025-04-26 21:59:45,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:45,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6876
[WARNING|trainer.py:803] 2025-04-26 21:59:45,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6817
[WARNING|trainer.py:803] 2025-04-26 21:59:46,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7978
[WARNING|trainer.py:803] 2025-04-26 21:59:46,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6877
6818
[WARNING|trainer.py:803] 2025-04-26 21:59:47,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:47,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:48,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6878
7979
6819
[WARNING|trainer.py:803] 2025-04-26 21:59:49,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:49,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:49,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6879
6820
7980
[WARNING|trainer.py:803] 2025-04-26 21:59:50,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:50,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6880
[WARNING|trainer.py:803] 2025-04-26 21:59:51,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6821
[WARNING|trainer.py:803] 2025-04-26 21:59:51,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7981
[WARNING|trainer.py:803] 2025-04-26 21:59:52,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6881
6822
[WARNING|trainer.py:803] 2025-04-26 21:59:52,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:53,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7982
[WARNING|trainer.py:803] 2025-04-26 21:59:53,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6882
6823
[WARNING|trainer.py:803] 2025-04-26 21:59:54,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:54,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:54,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6883
7983
6824
[WARNING|trainer.py:803] 2025-04-26 21:59:55,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 21:59:55,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 21:59:56,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6884
7984
6825
[WARNING|trainer.py:803] 2025-04-26 21:59:57,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:57,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 21:59:57,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6885
6826
7985
[WARNING|trainer.py:803] 2025-04-26 21:59:58,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 21:59:58,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6886
[WARNING|trainer.py:803] 2025-04-26 21:59:58,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6827
7986
[WARNING|trainer.py:803] 2025-04-26 21:59:59,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:00,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6887
[WARNING|trainer.py:803] 2025-04-26 22:00:00,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6828
[WARNING|trainer.py:803] 2025-04-26 22:00:00,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:01,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6888
7987
6829
[WARNING|trainer.py:803] 2025-04-26 22:00:02,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:02,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:02,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6889
6830
7988
[WARNING|trainer.py:803] 2025-04-26 22:00:03,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:03,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6890
[WARNING|trainer.py:803] 2025-04-26 22:00:04,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6831
7989
[WARNING|trainer.py:803] 2025-04-26 22:00:04,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:05,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6891
[WARNING|trainer.py:803] 2025-04-26 22:00:05,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6832
[WARNING|trainer.py:803] 2025-04-26 22:00:06,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7990
[WARNING|trainer.py:803] 2025-04-26 22:00:06,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6892
6833
[WARNING|trainer.py:803] 2025-04-26 22:00:07,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:07,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:07,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6893
7991
6834
[WARNING|trainer.py:803] 2025-04-26 22:00:08,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:09,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:09,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6894
6835
[WARNING|trainer.py:803] 2025-04-26 22:00:10,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7992
[WARNING|trainer.py:803] 2025-04-26 22:00:10,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6895
6836
[WARNING|trainer.py:803] 2025-04-26 22:00:11,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:11,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:11,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6896
7993
6837
[WARNING|trainer.py:803] 2025-04-26 22:00:12,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:13,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:13,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6897
6838
7994
[WARNING|trainer.py:803] 2025-04-26 22:00:14,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:14,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6898
[WARNING|trainer.py:803] 2025-04-26 22:00:14,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6839
7995
[WARNING|trainer.py:803] 2025-04-26 22:00:15,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:15,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6899
6840
[WARNING|trainer.py:803] 2025-04-26 22:00:16,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:16,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:17,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6900
7996
6841
[WARNING|trainer.py:803] 2025-04-26 22:00:18,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:18,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:18,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6901
6842
7997
[WARNING|trainer.py:803] 2025-04-26 22:00:19,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:19,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6902
6843
[WARNING|trainer.py:803] 2025-04-26 22:00:20,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:20,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7998
[WARNING|trainer.py:803] 2025-04-26 22:00:20,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6903
6844
[WARNING|trainer.py:803] 2025-04-26 22:00:21,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:21,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6904
[WARNING|trainer.py:803] 2025-04-26 22:00:22,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7999
6845
[WARNING|trainer.py:803] 2025-04-26 22:00:23,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:23,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6905
[WARNING|trainer.py:803] 2025-04-26 22:00:23,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6846
[WARNING|trainer.py:803] 2025-04-26 22:00:24,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8000
6906
[WARNING|trainer.py:803] 2025-04-26 22:00:24,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:25,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6847
[WARNING|trainer.py:803] 2025-04-26 22:00:25,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6907
[WARNING|trainer.py:803] 2025-04-26 22:00:26,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8001
6848
[WARNING|trainer.py:803] 2025-04-26 22:00:26,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:27,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6908
[WARNING|trainer.py:803] 2025-04-26 22:00:27,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:28,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6849
8002
6909
[WARNING|trainer.py:803] 2025-04-26 22:00:28,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:28,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:29,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6850
8003
6910
[WARNING|trainer.py:803] 2025-04-26 22:00:30,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:30,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6851
[WARNING|trainer.py:803] 2025-04-26 22:00:30,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6911
8004
[WARNING|trainer.py:803] 2025-04-26 22:00:31,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:31,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6852
[WARNING|trainer.py:803] 2025-04-26 22:00:32,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6912
[WARNING|trainer.py:803] 2025-04-26 22:00:32,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8005
[WARNING|trainer.py:803] 2025-04-26 22:00:33,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6853
6913
[WARNING|trainer.py:803] 2025-04-26 22:00:33,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:34,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8006
[WARNING|trainer.py:803] 2025-04-26 22:00:34,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6854
6914
[WARNING|trainer.py:803] 2025-04-26 22:00:35,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:35,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:35,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6915
8007
6855
[WARNING|trainer.py:803] 2025-04-26 22:00:36,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:36,804 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:36,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6916
6856
8008
[WARNING|trainer.py:803] 2025-04-26 22:00:38,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:38,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6917
[WARNING|trainer.py:803] 2025-04-26 22:00:38,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6857
[WARNING|trainer.py:803] 2025-04-26 22:00:39,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8009
[WARNING|trainer.py:803] 2025-04-26 22:00:39,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6918
6858
[WARNING|trainer.py:803] 2025-04-26 22:00:40,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:40,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8010
[WARNING|trainer.py:803] 2025-04-26 22:00:40,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6919
6859
[WARNING|trainer.py:803] 2025-04-26 22:00:41,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:41,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8011
[WARNING|trainer.py:803] 2025-04-26 22:00:42,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6920
6860
[WARNING|trainer.py:803] 2025-04-26 22:00:42,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:43,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:43,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6921
8012
6861
[WARNING|trainer.py:803] 2025-04-26 22:00:44,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:44,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6922
[WARNING|trainer.py:803] 2025-04-26 22:00:44,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8013
6862
[WARNING|trainer.py:803] 2025-04-26 22:00:45,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:45,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6923
[WARNING|trainer.py:803] 2025-04-26 22:00:46,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8014
6863
[WARNING|trainer.py:803] 2025-04-26 22:00:46,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:47,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6924
[WARNING|trainer.py:803] 2025-04-26 22:00:47,340 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8015
6864
[WARNING|trainer.py:803] 2025-04-26 22:00:48,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:48,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6925
[WARNING|trainer.py:803] 2025-04-26 22:00:48,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8016
6865
[WARNING|trainer.py:803] 2025-04-26 22:00:49,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6926
[WARNING|trainer.py:803] 2025-04-26 22:00:49,823 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:49,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8017
6866
[WARNING|trainer.py:803] 2025-04-26 22:00:50,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6927
[WARNING|trainer.py:803] 2025-04-26 22:00:51,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:51,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6867
[WARNING|trainer.py:803] 2025-04-26 22:00:51,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8018
6928
[WARNING|trainer.py:803] 2025-04-26 22:00:52,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:52,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6868
[WARNING|trainer.py:803] 2025-04-26 22:00:53,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8019
6929
[WARNING|trainer.py:803] 2025-04-26 22:00:53,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6869
[WARNING|trainer.py:803] 2025-04-26 22:00:54,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:00:54,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6930
[WARNING|trainer.py:803] 2025-04-26 22:00:55,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8020
6870
[WARNING|trainer.py:803] 2025-04-26 22:00:55,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6931
[WARNING|trainer.py:803] 2025-04-26 22:00:56,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:00:56,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8021
6871
[WARNING|trainer.py:803] 2025-04-26 22:00:56,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6932
[WARNING|trainer.py:803] 2025-04-26 22:00:57,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:57,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:58,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6872
8022
6933
[WARNING|trainer.py:803] 2025-04-26 22:00:59,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:00:59,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:00:59,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6873
6934
8023
[WARNING|trainer.py:803] 2025-04-26 22:01:00,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:01:00,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6874
[WARNING|trainer.py:803] 2025-04-26 22:01:00,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6935
8024
[WARNING|trainer.py:803] 2025-04-26 22:01:01,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:01,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6875
6936
[WARNING|trainer.py:803] 2025-04-26 22:01:02,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:03,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8025
[WARNING|trainer.py:803] 2025-04-26 22:01:03,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6876
6937
[WARNING|trainer.py:803] 2025-04-26 22:01:04,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:04,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:04,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8026
6877
6938
[WARNING|trainer.py:803] 2025-04-26 22:01:05,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:05,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:05,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6939
6878
8027
[WARNING|trainer.py:803] 2025-04-26 22:01:07,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:01:07,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:07,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6940
6879
8028
[WARNING|trainer.py:803] 2025-04-26 22:01:08,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:08,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:08,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6941
6880
8029
[WARNING|trainer.py:803] 2025-04-26 22:01:09,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:09,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:09,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6942
6881
8030
[WARNING|trainer.py:803] 2025-04-26 22:01:10,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:10,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6943
[WARNING|trainer.py:803] 2025-04-26 22:01:11,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6882
8031
[WARNING|trainer.py:803] 2025-04-26 22:01:12,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:12,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6944
[WARNING|trainer.py:803] 2025-04-26 22:01:12,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6883
[WARNING|trainer.py:803] 2025-04-26 22:01:13,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8032
[WARNING|trainer.py:803] 2025-04-26 22:01:13,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6945
6884
[WARNING|trainer.py:803] 2025-04-26 22:01:14,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:14,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8033
6946
[WARNING|trainer.py:803] 2025-04-26 22:01:14,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6885
[WARNING|trainer.py:803] 2025-04-26 22:01:15,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:15,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6947
[WARNING|trainer.py:803] 2025-04-26 22:01:16,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8034
6886
[WARNING|trainer.py:803] 2025-04-26 22:01:17,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:17,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6948
[WARNING|trainer.py:803] 2025-04-26 22:01:17,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8035
6887
[WARNING|trainer.py:803] 2025-04-26 22:01:18,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:18,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6949
[WARNING|trainer.py:803] 2025-04-26 22:01:18,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8036
6888
[WARNING|trainer.py:803] 2025-04-26 22:01:19,493 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:19,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6950
[WARNING|trainer.py:803] 2025-04-26 22:01:20,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
6889
[WARNING|trainer.py:803] 2025-04-26 22:01:20,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6951
[WARNING|trainer.py:803] 2025-04-26 22:01:21,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:21,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:21,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8038
6890
6952
[WARNING|trainer.py:803] 2025-04-26 22:01:22,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:01:22,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:23,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8039
6891
6953
[WARNING|trainer.py:803] 2025-04-26 22:01:24,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:01:24,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:24,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6892
6954
8040
[WARNING|trainer.py:803] 2025-04-26 22:01:25,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:25,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:25,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6893
6955
8041
[WARNING|trainer.py:803] 2025-04-26 22:01:26,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:26,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:27,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6956
6894
8042
[WARNING|trainer.py:803] 2025-04-26 22:01:28,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:28,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6957
[WARNING|trainer.py:803] 2025-04-26 22:01:28,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6895
8043
[WARNING|trainer.py:803] 2025-04-26 22:01:29,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:29,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6958
[WARNING|trainer.py:803] 2025-04-26 22:01:30,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6896
8044
[WARNING|trainer.py:803] 2025-04-26 22:01:30,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:31,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6959
[WARNING|trainer.py:803] 2025-04-26 22:01:31,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6897
[WARNING|trainer.py:803] 2025-04-26 22:01:31,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8045
[WARNING|trainer.py:803] 2025-04-26 22:01:32,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6960
6898
[WARNING|trainer.py:803] 2025-04-26 22:01:32,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:33,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6961
[WARNING|trainer.py:803] 2025-04-26 22:01:33,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8046
6899
[WARNING|trainer.py:803] 2025-04-26 22:01:34,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6962
[WARNING|trainer.py:803] 2025-04-26 22:01:34,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:34,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6900
8047
[WARNING|trainer.py:803] 2025-04-26 22:01:35,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6963
[WARNING|trainer.py:803] 2025-04-26 22:01:36,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:36,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6901
[WARNING|trainer.py:803] 2025-04-26 22:01:36,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8048
6964
[WARNING|trainer.py:803] 2025-04-26 22:01:37,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:37,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6902
[WARNING|trainer.py:803] 2025-04-26 22:01:38,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6965
8049
[WARNING|trainer.py:803] 2025-04-26 22:01:38,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6903
[WARNING|trainer.py:803] 2025-04-26 22:01:39,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:39,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6966
[WARNING|trainer.py:803] 2025-04-26 22:01:40,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8050
6904
[WARNING|trainer.py:803] 2025-04-26 22:01:40,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:40,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6967
[WARNING|trainer.py:803] 2025-04-26 22:01:41,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8051
6905
[WARNING|trainer.py:803] 2025-04-26 22:01:41,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6968
[WARNING|trainer.py:803] 2025-04-26 22:01:42,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:42,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6906
8052
[WARNING|trainer.py:803] 2025-04-26 22:01:43,151 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6969
[WARNING|trainer.py:803] 2025-04-26 22:01:43,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:43,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6907
8053
[WARNING|trainer.py:803] 2025-04-26 22:01:44,388 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6970
[WARNING|trainer.py:803] 2025-04-26 22:01:45,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:45,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6908
[WARNING|trainer.py:803] 2025-04-26 22:01:45,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8054
6971
[WARNING|trainer.py:803] 2025-04-26 22:01:46,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:46,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6909
[WARNING|trainer.py:803] 2025-04-26 22:01:46,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8055
6972
[WARNING|trainer.py:803] 2025-04-26 22:01:47,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:47,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6910
[WARNING|trainer.py:803] 2025-04-26 22:01:48,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8056
6973
[WARNING|trainer.py:803] 2025-04-26 22:01:48,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:49,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6911
[WARNING|trainer.py:803] 2025-04-26 22:01:49,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8057
6974
[WARNING|trainer.py:803] 2025-04-26 22:01:50,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6912
[WARNING|trainer.py:803] 2025-04-26 22:01:50,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:01:50,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6975
8058
[WARNING|trainer.py:803] 2025-04-26 22:01:51,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6913
[WARNING|trainer.py:803] 2025-04-26 22:01:51,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:52,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6976
8059
[WARNING|trainer.py:803] 2025-04-26 22:01:52,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6914
[WARNING|trainer.py:803] 2025-04-26 22:01:53,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:01:53,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6977
8060
[WARNING|trainer.py:803] 2025-04-26 22:01:53,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6915
[WARNING|trainer.py:803] 2025-04-26 22:01:54,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:54,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6978
[WARNING|trainer.py:803] 2025-04-26 22:01:55,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8061
6916
[WARNING|trainer.py:803] 2025-04-26 22:01:55,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6979
[WARNING|trainer.py:803] 2025-04-26 22:01:56,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:01:56,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6917
8062
[WARNING|trainer.py:803] 2025-04-26 22:01:56,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6980
[WARNING|trainer.py:803] 2025-04-26 22:01:57,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:01:57,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6918
[WARNING|trainer.py:803] 2025-04-26 22:01:58,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8063
6981
[WARNING|trainer.py:803] 2025-04-26 22:01:58,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6919
[WARNING|trainer.py:803] 2025-04-26 22:01:59,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:01:59,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6982
8064
[WARNING|trainer.py:803] 2025-04-26 22:02:00,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6920
[WARNING|trainer.py:803] 2025-04-26 22:02:00,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:00,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6983
[WARNING|trainer.py:803] 2025-04-26 22:02:01,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8065
6921
[WARNING|trainer.py:803] 2025-04-26 22:02:01,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:02,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6984
[WARNING|trainer.py:803] 2025-04-26 22:02:02,485 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8066
6922
[WARNING|trainer.py:803] 2025-04-26 22:02:03,035 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6985
[WARNING|trainer.py:803] 2025-04-26 22:02:03,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:02:03,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6923
8067
[WARNING|trainer.py:803] 2025-04-26 22:02:04,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6986
[WARNING|trainer.py:803] 2025-04-26 22:02:04,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:05,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6924
[WARNING|trainer.py:803] 2025-04-26 22:02:05,507 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8068
6987
[WARNING|trainer.py:803] 2025-04-26 22:02:06,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:06,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6925
[WARNING|trainer.py:803] 2025-04-26 22:02:06,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6988
8069
[WARNING|trainer.py:803] 2025-04-26 22:02:07,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6926
[WARNING|trainer.py:803] 2025-04-26 22:02:07,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:08,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6989
8070
[WARNING|trainer.py:803] 2025-04-26 22:02:08,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6927
[WARNING|trainer.py:803] 2025-04-26 22:02:09,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:09,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6990
[WARNING|trainer.py:803] 2025-04-26 22:02:10,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8071
6928
[WARNING|trainer.py:803] 2025-04-26 22:02:10,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6991
[WARNING|trainer.py:803] 2025-04-26 22:02:11,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:11,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8072
6929
[WARNING|trainer.py:803] 2025-04-26 22:02:11,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6992
[WARNING|trainer.py:803] 2025-04-26 22:02:12,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:02:12,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:02:12,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6930
8073
6993
[WARNING|trainer.py:803] 2025-04-26 22:02:13,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:14,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:14,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6931
6994
[WARNING|trainer.py:803] 2025-04-26 22:02:15,057 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8074
[WARNING|trainer.py:803] 2025-04-26 22:02:15,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6932
6995
[WARNING|trainer.py:803] 2025-04-26 22:02:16,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:16,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:16,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6933
8075
6996
[WARNING|trainer.py:803] 2025-04-26 22:02:17,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:17,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:17,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6934
6997
8076
[WARNING|trainer.py:803] 2025-04-26 22:02:18,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:19,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6935
[WARNING|trainer.py:803] 2025-04-26 22:02:19,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6998
8077
[WARNING|trainer.py:803] 2025-04-26 22:02:20,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:20,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6936
[WARNING|trainer.py:803] 2025-04-26 22:02:20,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6999
8078
[WARNING|trainer.py:803] 2025-04-26 22:02:21,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:21,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6937
[WARNING|trainer.py:803] 2025-04-26 22:02:22,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7000
8079
[WARNING|trainer.py:803] 2025-04-26 22:02:22,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:22,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6938
7001
[WARNING|trainer.py:803] 2025-04-26 22:02:23,346 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:23,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8080
[WARNING|trainer.py:803] 2025-04-26 22:02:24,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6939
7002
[WARNING|trainer.py:803] 2025-04-26 22:02:24,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:25,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:02:25,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6940
8081
7003
[WARNING|trainer.py:803] 2025-04-26 22:02:26,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:26,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:02:26,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6941
7004
8082
[WARNING|trainer.py:803] 2025-04-26 22:02:27,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:27,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6942
[WARNING|trainer.py:803] 2025-04-26 22:02:28,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7005
8083
[WARNING|trainer.py:803] 2025-04-26 22:02:28,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6943
[WARNING|trainer.py:803] 2025-04-26 22:02:29,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:29,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7006
[WARNING|trainer.py:803] 2025-04-26 22:02:30,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8084
6944
[WARNING|trainer.py:803] 2025-04-26 22:02:30,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7007
[WARNING|trainer.py:803] 2025-04-26 22:02:30,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:31,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6945
8085
[WARNING|trainer.py:803] 2025-04-26 22:02:31,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7008
[WARNING|trainer.py:803] 2025-04-26 22:02:32,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:32,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6946
[WARNING|trainer.py:803] 2025-04-26 22:02:32,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7009
8086
[WARNING|trainer.py:803] 2025-04-26 22:02:33,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6947
[WARNING|trainer.py:803] 2025-04-26 22:02:34,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:02:34,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7010
[WARNING|trainer.py:803] 2025-04-26 22:02:35,028 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8087
6948
[WARNING|trainer.py:803] 2025-04-26 22:02:35,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7011
[WARNING|trainer.py:803] 2025-04-26 22:02:36,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:36,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8088
6949
[WARNING|trainer.py:803] 2025-04-26 22:02:36,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7012
[WARNING|trainer.py:803] 2025-04-26 22:02:37,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:37,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8089
6950
[WARNING|trainer.py:803] 2025-04-26 22:02:37,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7013
[WARNING|trainer.py:803] 2025-04-26 22:02:38,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:38,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:39,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6951
7014
[WARNING|trainer.py:803] 2025-04-26 22:02:40,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8090
[WARNING|trainer.py:803] 2025-04-26 22:02:40,419 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6952
7015
[WARNING|trainer.py:803] 2025-04-26 22:02:41,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:41,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8091
[WARNING|trainer.py:803] 2025-04-26 22:02:41,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6953
7016
[WARNING|trainer.py:803] 2025-04-26 22:02:42,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:42,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:42,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6954
8092
7017
[WARNING|trainer.py:803] 2025-04-26 22:02:43,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:43,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:44,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
6955
8093
7018
[WARNING|trainer.py:803] 2025-04-26 22:02:45,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:45,412 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6956
[WARNING|trainer.py:803] 2025-04-26 22:02:45,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7019
8094
[WARNING|trainer.py:803] 2025-04-26 22:02:46,301 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6957
[WARNING|trainer.py:803] 2025-04-26 22:02:46,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:46,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7020
8095
[WARNING|trainer.py:803] 2025-04-26 22:02:47,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:47,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6958
[WARNING|trainer.py:803] 2025-04-26 22:02:48,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7021
8096
[WARNING|trainer.py:803] 2025-04-26 22:02:48,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:49,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6959
[WARNING|trainer.py:803] 2025-04-26 22:02:49,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7022
[WARNING|trainer.py:803] 2025-04-26 22:02:50,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8097
[WARNING|trainer.py:803] 2025-04-26 22:02:50,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6960
7023
[WARNING|trainer.py:803] 2025-04-26 22:02:50,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:02:51,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:02:51,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6961
8098
7024
[WARNING|trainer.py:803] 2025-04-26 22:02:52,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:52,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:02:52,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6962
7025
8099
[WARNING|trainer.py:803] 2025-04-26 22:02:53,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:54,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:54,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6963
7026
8100
[WARNING|trainer.py:803] 2025-04-26 22:02:55,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:55,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6964
[WARNING|trainer.py:803] 2025-04-26 22:02:55,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7027
[WARNING|trainer.py:803] 2025-04-26 22:02:56,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8101
[WARNING|trainer.py:803] 2025-04-26 22:02:56,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6965
7028
[WARNING|trainer.py:803] 2025-04-26 22:02:57,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:02:57,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:57,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6966
7029
8102
[WARNING|trainer.py:803] 2025-04-26 22:02:58,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:59,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:02:59,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6967
7030
[WARNING|trainer.py:803] 2025-04-26 22:03:00,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8103
[WARNING|trainer.py:803] 2025-04-26 22:03:00,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
6968
7031
[WARNING|trainer.py:803] 2025-04-26 22:03:00,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:01,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:01,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6969
8104
7032
[WARNING|trainer.py:803] 2025-04-26 22:03:02,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:02,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:02,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6970
7033
8105
[WARNING|trainer.py:803] 2025-04-26 22:03:03,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:04,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6971
[WARNING|trainer.py:803] 2025-04-26 22:03:04,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7034
[WARNING|trainer.py:803] 2025-04-26 22:03:05,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8106
[WARNING|trainer.py:803] 2025-04-26 22:03:05,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6972
7035
[WARNING|trainer.py:803] 2025-04-26 22:03:06,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:06,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6973
[WARNING|trainer.py:803] 2025-04-26 22:03:06,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8107
7036
[WARNING|trainer.py:803] 2025-04-26 22:03:07,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:07,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6974
[WARNING|trainer.py:803] 2025-04-26 22:03:08,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7037
[WARNING|trainer.py:803] 2025-04-26 22:03:08,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8108
6975
[WARNING|trainer.py:803] 2025-04-26 22:03:09,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:09,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7038
[WARNING|trainer.py:803] 2025-04-26 22:03:10,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6976
[WARNING|trainer.py:803] 2025-04-26 22:03:10,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8109
7039
[WARNING|trainer.py:803] 2025-04-26 22:03:11,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:11,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:11,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6977
7040
8110
[WARNING|trainer.py:803] 2025-04-26 22:03:12,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:13,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6978
[WARNING|trainer.py:803] 2025-04-26 22:03:13,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7041
[WARNING|trainer.py:803] 2025-04-26 22:03:14,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8111
[WARNING|trainer.py:803] 2025-04-26 22:03:14,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
6979
7042
[WARNING|trainer.py:803] 2025-04-26 22:03:14,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:15,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:15,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6980
8112
7043
[WARNING|trainer.py:803] 2025-04-26 22:03:16,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:16,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:16,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6981
7044
8113
[WARNING|trainer.py:803] 2025-04-26 22:03:17,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:18,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6982
[WARNING|trainer.py:803] 2025-04-26 22:03:18,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7045
[WARNING|trainer.py:803] 2025-04-26 22:03:19,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6983
[WARNING|trainer.py:803] 2025-04-26 22:03:19,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8114
7046
[WARNING|trainer.py:803] 2025-04-26 22:03:20,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:20,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
6984
[WARNING|trainer.py:803] 2025-04-26 22:03:20,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7047
8115
[WARNING|trainer.py:803] 2025-04-26 22:03:21,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6985
[WARNING|trainer.py:803] 2025-04-26 22:03:21,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:22,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7048
[WARNING|trainer.py:803] 2025-04-26 22:03:22,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6986
[WARNING|trainer.py:803] 2025-04-26 22:03:23,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8116
7049
[WARNING|trainer.py:803] 2025-04-26 22:03:23,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:24,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6987
[WARNING|trainer.py:803] 2025-04-26 22:03:24,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7050
8117
[WARNING|trainer.py:803] 2025-04-26 22:03:25,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6988
[WARNING|trainer.py:803] 2025-04-26 22:03:25,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:25,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7051
[WARNING|trainer.py:803] 2025-04-26 22:03:26,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6989
[WARNING|trainer.py:803] 2025-04-26 22:03:26,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8118
7052
[WARNING|trainer.py:803] 2025-04-26 22:03:27,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:27,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
6990
[WARNING|trainer.py:803] 2025-04-26 22:03:28,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7053
8119
[WARNING|trainer.py:803] 2025-04-26 22:03:28,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6991
[WARNING|trainer.py:803] 2025-04-26 22:03:29,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:29,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7054
[WARNING|trainer.py:803] 2025-04-26 22:03:30,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8120
6992
[WARNING|trainer.py:803] 2025-04-26 22:03:30,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7055
[WARNING|trainer.py:803] 2025-04-26 22:03:31,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:31,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6993
[WARNING|trainer.py:803] 2025-04-26 22:03:31,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8121
7056
[WARNING|trainer.py:803] 2025-04-26 22:03:32,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6994
[WARNING|trainer.py:803] 2025-04-26 22:03:33,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:33,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7057
[WARNING|trainer.py:803] 2025-04-26 22:03:33,904 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8122
6995
[WARNING|trainer.py:803] 2025-04-26 22:03:34,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7058
[WARNING|trainer.py:803] 2025-04-26 22:03:34,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:35,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6996
[WARNING|trainer.py:803] 2025-04-26 22:03:35,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8123
7059
[WARNING|trainer.py:803] 2025-04-26 22:03:36,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:36,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
6997
[WARNING|trainer.py:803] 2025-04-26 22:03:36,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7060
[WARNING|trainer.py:803] 2025-04-26 22:03:37,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8124
6998
[WARNING|trainer.py:803] 2025-04-26 22:03:38,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7061
[WARNING|trainer.py:803] 2025-04-26 22:03:38,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:38,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
6999
[WARNING|trainer.py:803] 2025-04-26 22:03:39,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8125
7062
[WARNING|trainer.py:803] 2025-04-26 22:03:40,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:40,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7000
[WARNING|trainer.py:803] 2025-04-26 22:03:40,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7063
8126
[WARNING|trainer.py:803] 2025-04-26 22:03:41,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7001
[WARNING|trainer.py:803] 2025-04-26 22:03:41,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:03:42,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7064
[WARNING|trainer.py:803] 2025-04-26 22:03:42,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8127
7002
[WARNING|trainer.py:803] 2025-04-26 22:03:43,159 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7065
[WARNING|trainer.py:803] 2025-04-26 22:03:43,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:43,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7003
[WARNING|trainer.py:803] 2025-04-26 22:03:44,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7066
8128
[WARNING|trainer.py:803] 2025-04-26 22:03:45,128 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7004
[WARNING|trainer.py:803] 2025-04-26 22:03:45,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:45,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7067
[WARNING|trainer.py:803] 2025-04-26 22:03:46,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8129
7005
[WARNING|trainer.py:803] 2025-04-26 22:03:46,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7068
[WARNING|trainer.py:803] 2025-04-26 22:03:47,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:47,709 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:48,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7006
7069
8130
[WARNING|trainer.py:803] 2025-04-26 22:03:48,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:49,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7007
[WARNING|trainer.py:803] 2025-04-26 22:03:49,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7070
[WARNING|trainer.py:803] 2025-04-26 22:03:50,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8131
[WARNING|trainer.py:803] 2025-04-26 22:03:50,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7008
7071
[WARNING|trainer.py:803] 2025-04-26 22:03:51,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:51,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:51,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7009
7072
8132
[WARNING|trainer.py:803] 2025-04-26 22:03:52,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:03:53,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:53,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7010
7073
[WARNING|trainer.py:803] 2025-04-26 22:03:54,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:54,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8133
7011
7074
[WARNING|trainer.py:803] 2025-04-26 22:03:55,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:55,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:55,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7012
7075
8134
[WARNING|trainer.py:803] 2025-04-26 22:03:56,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:56,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:03:56,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7013
7076
[WARNING|trainer.py:803] 2025-04-26 22:03:57,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8135
[WARNING|trainer.py:803] 2025-04-26 22:03:58,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7014
7077
[WARNING|trainer.py:803] 2025-04-26 22:03:58,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:03:59,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:03:59,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7015
8136
7078
[WARNING|trainer.py:803] 2025-04-26 22:04:00,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:00,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:04:00,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7016
7079
8137
[WARNING|trainer.py:803] 2025-04-26 22:04:01,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:01,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7017
7080
[WARNING|trainer.py:803] 2025-04-26 22:04:02,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:02,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 22:04:03,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8138
7018
7081
[WARNING|trainer.py:803] 2025-04-26 22:04:04,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:04,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:04,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7019
7082
8139
[WARNING|trainer.py:803] 2025-04-26 22:04:05,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:05,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:05,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7020
7083
8140
[WARNING|trainer.py:803] 2025-04-26 22:04:06,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:04:06,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7021
7084
[WARNING|trainer.py:803] 2025-04-26 22:04:07,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:07,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:08,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
7022
7085
8141
[WARNING|trainer.py:803] 2025-04-26 22:04:09,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:09,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:09,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7023
7086
8142
[WARNING|trainer.py:803] 2025-04-26 22:04:10,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:10,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7024
7087
[WARNING|trainer.py:803] 2025-04-26 22:04:11,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:11,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:11,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7025
7088
8143
[WARNING|trainer.py:803] 2025-04-26 22:04:12,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:13,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:13,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7026
7089
8144
[WARNING|trainer.py:803] 2025-04-26 22:04:14,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:04:14,290 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7027
7090
[WARNING|trainer.py:803] 2025-04-26 22:04:14,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:15,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:15,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8145
7091
7028
[WARNING|trainer.py:803] 2025-04-26 22:04:16,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:16,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:16,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7092
7029
8146
[WARNING|trainer.py:803] 2025-04-26 22:04:18,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:18,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7093
[WARNING|trainer.py:803] 2025-04-26 22:04:18,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7030
[WARNING|trainer.py:803] 2025-04-26 22:04:19,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:19,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8147
7094
7031
[WARNING|trainer.py:803] 2025-04-26 22:04:20,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:20,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:20,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7095
7032
8148
[WARNING|trainer.py:803] 2025-04-26 22:04:21,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:21,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:22,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7096
7033
[WARNING|trainer.py:803] 2025-04-26 22:04:22,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8149
[WARNING|trainer.py:803] 2025-04-26 22:04:23,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7097
7034
[WARNING|trainer.py:803] 2025-04-26 22:04:23,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:24,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:24,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7098
7035
8150
[WARNING|trainer.py:803] 2025-04-26 22:04:25,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:25,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:25,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7099
7036
[WARNING|trainer.py:803] 2025-04-26 22:04:26,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8151
[WARNING|trainer.py:803] 2025-04-26 22:04:26,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7100
7037
[WARNING|trainer.py:803] 2025-04-26 22:04:27,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:27,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:28,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7101
8152
7038
[WARNING|trainer.py:803] 2025-04-26 22:04:29,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:29,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:29,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7102
7039
8153
[WARNING|trainer.py:803] 2025-04-26 22:04:30,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:30,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7103
7040
[WARNING|trainer.py:803] 2025-04-26 22:04:31,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:31,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7104
[WARNING|trainer.py:803] 2025-04-26 22:04:32,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8154
7041
[WARNING|trainer.py:803] 2025-04-26 22:04:32,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:32,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7105
[WARNING|trainer.py:803] 2025-04-26 22:04:33,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7042
8155
[WARNING|trainer.py:803] 2025-04-26 22:04:34,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7106
[WARNING|trainer.py:803] 2025-04-26 22:04:34,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:04:34,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7043
[WARNING|trainer.py:803] 2025-04-26 22:04:35,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8156
7107
[WARNING|trainer.py:803] 2025-04-26 22:04:35,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7044
[WARNING|trainer.py:803] 2025-04-26 22:04:36,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:36,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7108
[WARNING|trainer.py:803] 2025-04-26 22:04:37,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8157
7045
[WARNING|trainer.py:803] 2025-04-26 22:04:37,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7109
[WARNING|trainer.py:803] 2025-04-26 22:04:38,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:38,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7046
[WARNING|trainer.py:803] 2025-04-26 22:04:38,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8158
7110
[WARNING|trainer.py:803] 2025-04-26 22:04:39,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:39,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7047
[WARNING|trainer.py:803] 2025-04-26 22:04:40,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7111
[WARNING|trainer.py:803] 2025-04-26 22:04:40,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8159
7048
[WARNING|trainer.py:803] 2025-04-26 22:04:41,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:41,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7112
[WARNING|trainer.py:803] 2025-04-26 22:04:42,064 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7049
8160
[WARNING|trainer.py:803] 2025-04-26 22:04:42,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7113
[WARNING|trainer.py:803] 2025-04-26 22:04:43,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:43,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7050
[WARNING|trainer.py:803] 2025-04-26 22:04:43,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7114
8161
[WARNING|trainer.py:803] 2025-04-26 22:04:44,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7051
[WARNING|trainer.py:803] 2025-04-26 22:04:45,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:45,189 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7115
[WARNING|trainer.py:803] 2025-04-26 22:04:45,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7052
8162
[WARNING|trainer.py:803] 2025-04-26 22:04:46,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7116
[WARNING|trainer.py:803] 2025-04-26 22:04:47,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:47,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7053
[WARNING|trainer.py:803] 2025-04-26 22:04:47,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8163
7117
[WARNING|trainer.py:803] 2025-04-26 22:04:48,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7054
[WARNING|trainer.py:803] 2025-04-26 22:04:48,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:48,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7118
[WARNING|trainer.py:803] 2025-04-26 22:04:49,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8164
7055
[WARNING|trainer.py:803] 2025-04-26 22:04:50,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7119
[WARNING|trainer.py:803] 2025-04-26 22:04:50,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:50,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7056
[WARNING|trainer.py:803] 2025-04-26 22:04:51,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8165
7120
[WARNING|trainer.py:803] 2025-04-26 22:04:52,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:04:52,424 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7057
[WARNING|trainer.py:803] 2025-04-26 22:04:52,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7121
8166
[WARNING|trainer.py:803] 2025-04-26 22:04:53,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7058
[WARNING|trainer.py:803] 2025-04-26 22:04:53,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:54,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7122
[WARNING|trainer.py:803] 2025-04-26 22:04:54,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8167
7059
[WARNING|trainer.py:803] 2025-04-26 22:04:54,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7123
[WARNING|trainer.py:803] 2025-04-26 22:04:55,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:55,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:56,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7060
7124
8168
[WARNING|trainer.py:803] 2025-04-26 22:04:57,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:57,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7061
[WARNING|trainer.py:803] 2025-04-26 22:04:57,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7125
[WARNING|trainer.py:803] 2025-04-26 22:04:58,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8169
[WARNING|trainer.py:803] 2025-04-26 22:04:58,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7062
7126
[WARNING|trainer.py:803] 2025-04-26 22:04:59,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:04:59,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:04:59,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7063
7127
8170
[WARNING|trainer.py:803] 2025-04-26 22:05:00,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:01,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7064
[WARNING|trainer.py:803] 2025-04-26 22:05:01,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7128
[WARNING|trainer.py:803] 2025-04-26 22:05:02,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8171
[WARNING|trainer.py:803] 2025-04-26 22:05:02,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7065
7129
[WARNING|trainer.py:803] 2025-04-26 22:05:03,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:03,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:03,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7066
7130
8172
[WARNING|trainer.py:803] 2025-04-26 22:05:04,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:04,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7067
[WARNING|trainer.py:803] 2025-04-26 22:05:05,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7131
[WARNING|trainer.py:803] 2025-04-26 22:05:05,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8173
[WARNING|trainer.py:803] 2025-04-26 22:05:06,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7068
7132
[WARNING|trainer.py:803] 2025-04-26 22:05:06,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:07,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:07,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7069
8174
7133
[WARNING|trainer.py:803] 2025-04-26 22:05:08,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:08,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:08,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7070
7134
[WARNING|trainer.py:803] 2025-04-26 22:05:09,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8175
[WARNING|trainer.py:803] 2025-04-26 22:05:09,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7071
7135
[WARNING|trainer.py:803] 2025-04-26 22:05:10,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:10,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:11,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7072
8176
7136
[WARNING|trainer.py:803] 2025-04-26 22:05:12,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:12,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:12,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7073
7137
8177
[WARNING|trainer.py:803] 2025-04-26 22:05:13,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:13,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7074
7138
[WARNING|trainer.py:803] 2025-04-26 22:05:14,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:14,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:14,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8178
7075
7139
[WARNING|trainer.py:803] 2025-04-26 22:05:15,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:15,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:15,996 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7076
7140
8179
[WARNING|trainer.py:803] 2025-04-26 22:05:17,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:17,236 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:17,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7077
7141
[WARNING|trainer.py:803] 2025-04-26 22:05:18,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8180
[WARNING|trainer.py:803] 2025-04-26 22:05:18,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7078
7142
[WARNING|trainer.py:803] 2025-04-26 22:05:19,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:19,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:19,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7079
7143
8181
[WARNING|trainer.py:803] 2025-04-26 22:05:20,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:20,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:21,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7080
7144
8182
[WARNING|trainer.py:803] 2025-04-26 22:05:22,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:22,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7081
7145
[WARNING|trainer.py:803] 2025-04-26 22:05:22,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:23,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:23,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8183
7082
7146
[WARNING|trainer.py:803] 2025-04-26 22:05:24,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:24,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:24,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7083
7147
8184
[WARNING|trainer.py:803] 2025-04-26 22:05:26,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:26,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7084
7148
[WARNING|trainer.py:803] 2025-04-26 22:05:26,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:27,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerYes
[WARNING|trainer.py:803] 2025-04-26 22:05:27,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8185
7085
7149
[WARNING|trainer.py:803] 2025-04-26 22:05:28,362 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:28,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:28,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7086
7150
8186
[WARNING|trainer.py:803] 2025-04-26 22:05:29,793 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:29,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7087
7151
[WARNING|trainer.py:803] 2025-04-26 22:05:30,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:31,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:31,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7088
8187
7152
[WARNING|trainer.py:803] 2025-04-26 22:05:32,296 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:32,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:32,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7089
7153
8188
[WARNING|trainer.py:803] 2025-04-26 22:05:33,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:33,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7090
[WARNING|trainer.py:803] 2025-04-26 22:05:34,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7154
8189
[WARNING|trainer.py:803] 2025-04-26 22:05:34,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:34,886 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7091
7155
[WARNING|trainer.py:803] 2025-04-26 22:05:35,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:36,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:36,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8190
7092
7156
[WARNING|trainer.py:803] 2025-04-26 22:05:37,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:37,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:37,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7093
7157
8191
[WARNING|trainer.py:803] 2025-04-26 22:05:38,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:38,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7094
7158
[WARNING|trainer.py:803] 2025-04-26 22:05:39,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:39,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:39,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7095
8192
7159
[WARNING|trainer.py:803] 2025-04-26 22:05:41,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:41,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:41,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7096
7160
8193
[WARNING|trainer.py:803] 2025-04-26 22:05:42,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:42,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7097
7161
[WARNING|trainer.py:803] 2025-04-26 22:05:42,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:43,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:43,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8194
7098
7162
[WARNING|trainer.py:803] 2025-04-26 22:05:44,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:44,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:45,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7099
7163
8195
[WARNING|trainer.py:803] 2025-04-26 22:05:46,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:46,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:46,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7100
7164
[WARNING|trainer.py:803] 2025-04-26 22:05:47,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8196
[WARNING|trainer.py:803] 2025-04-26 22:05:47,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7101
7165
[WARNING|trainer.py:803] 2025-04-26 22:05:48,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:48,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:48,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7102
8197
7166
[WARNING|trainer.py:803] 2025-04-26 22:05:49,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:50,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:50,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7103
7167
[WARNING|trainer.py:803] 2025-04-26 22:05:51,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:51,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8198
7104
7168
[WARNING|trainer.py:803] 2025-04-26 22:05:52,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:52,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:52,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7105
7169
8199
[WARNING|trainer.py:803] 2025-04-26 22:05:53,763 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:53,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:54,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7170
7106
8200
[WARNING|trainer.py:803] 2025-04-26 22:05:55,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:55,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7171
7107
[WARNING|trainer.py:803] 2025-04-26 22:05:55,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:56,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:05:56,324 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7172
7108
8201
[WARNING|trainer.py:803] 2025-04-26 22:05:57,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:05:57,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:05:57,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7173
7109
[WARNING|trainer.py:803] 2025-04-26 22:05:58,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8202
[WARNING|trainer.py:803] 2025-04-26 22:05:58,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7174
7110
[WARNING|trainer.py:803] 2025-04-26 22:05:59,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:05:59,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:00,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7175
7111
8203
[WARNING|trainer.py:803] 2025-04-26 22:06:01,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:01,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:01,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7176
7112
[WARNING|trainer.py:803] 2025-04-26 22:06:02,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8204
[WARNING|trainer.py:803] 2025-04-26 22:06:02,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7177
7113
[WARNING|trainer.py:803] 2025-04-26 22:06:03,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:03,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:03,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7178
8205
7114
[WARNING|trainer.py:803] 2025-04-26 22:06:04,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:05,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:05,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7179
7115
8206
[WARNING|trainer.py:803] 2025-04-26 22:06:06,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:06,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7180
7116
[WARNING|trainer.py:803] 2025-04-26 22:06:06,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:07,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:07,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7181
8207
7117
[WARNING|trainer.py:803] 2025-04-26 22:06:08,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:08,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:08,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7182
7118
8208
[WARNING|trainer.py:803] 2025-04-26 22:06:09,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7183
[WARNING|trainer.py:803] 2025-04-26 22:06:10,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:10,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7119
[WARNING|trainer.py:803] 2025-04-26 22:06:11,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7184
[WARNING|trainer.py:803] 2025-04-26 22:06:11,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8209
7120
[WARNING|trainer.py:803] 2025-04-26 22:06:12,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:12,335 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7185
[WARNING|trainer.py:803] 2025-04-26 22:06:12,737 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7121
8210
[WARNING|trainer.py:803] 2025-04-26 22:06:13,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7186
[WARNING|trainer.py:803] 2025-04-26 22:06:13,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:14,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7122
[WARNING|trainer.py:803] 2025-04-26 22:06:14,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8211
7187
[WARNING|trainer.py:803] 2025-04-26 22:06:15,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7123
[WARNING|trainer.py:803] 2025-04-26 22:06:15,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:15,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7188
[WARNING|trainer.py:803] 2025-04-26 22:06:16,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8212
7124
[WARNING|trainer.py:803] 2025-04-26 22:06:17,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:17,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7189
[WARNING|trainer.py:803] 2025-04-26 22:06:17,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7125
[WARNING|trainer.py:803] 2025-04-26 22:06:18,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8213
7190
[WARNING|trainer.py:803] 2025-04-26 22:06:19,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:19,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7126
[WARNING|trainer.py:803] 2025-04-26 22:06:19,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7191
[WARNING|trainer.py:803] 2025-04-26 22:06:20,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8214
7127
[WARNING|trainer.py:803] 2025-04-26 22:06:20,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:21,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7192
[WARNING|trainer.py:803] 2025-04-26 22:06:21,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7128
[WARNING|trainer.py:803] 2025-04-26 22:06:22,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8215
7193
[WARNING|trainer.py:803] 2025-04-26 22:06:22,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:22,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7129
[WARNING|trainer.py:803] 2025-04-26 22:06:23,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7194
8216
[WARNING|trainer.py:803] 2025-04-26 22:06:24,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7130
[WARNING|trainer.py:803] 2025-04-26 22:06:24,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:24,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7195
[WARNING|trainer.py:803] 2025-04-26 22:06:25,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7131
[WARNING|trainer.py:803] 2025-04-26 22:06:25,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8217
7196
[WARNING|trainer.py:803] 2025-04-26 22:06:26,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:26,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7132
[WARNING|trainer.py:803] 2025-04-26 22:06:27,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7197
[WARNING|trainer.py:803] 2025-04-26 22:06:27,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8218
7133
[WARNING|trainer.py:803] 2025-04-26 22:06:28,256 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7198
[WARNING|trainer.py:803] 2025-04-26 22:06:28,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:29,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7134
[WARNING|trainer.py:803] 2025-04-26 22:06:29,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8219
7199
[WARNING|trainer.py:803] 2025-04-26 22:06:30,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:30,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:30,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7135
7200
[WARNING|trainer.py:803] 2025-04-26 22:06:31,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8220
[WARNING|trainer.py:803] 2025-04-26 22:06:31,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7136
7201
[WARNING|trainer.py:803] 2025-04-26 22:06:32,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:32,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:33,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7137
8221
7202
[WARNING|trainer.py:803] 2025-04-26 22:06:34,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:34,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:34,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7138
7203
[WARNING|trainer.py:803] 2025-04-26 22:06:35,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8222
[WARNING|trainer.py:803] 2025-04-26 22:06:35,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7139
7204
[WARNING|trainer.py:803] 2025-04-26 22:06:36,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:36,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:36,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7140
8223
7205
[WARNING|trainer.py:803] 2025-04-26 22:06:37,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:38,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:38,246 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7141
7206
8224
[WARNING|trainer.py:803] 2025-04-26 22:06:39,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:39,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7142
7207
[WARNING|trainer.py:803] 2025-04-26 22:06:39,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:40,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:40,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7143
8225
7208
[WARNING|trainer.py:803] 2025-04-26 22:06:41,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:41,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:42,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7144
7209
8226
[WARNING|trainer.py:803] 2025-04-26 22:06:42,915 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:43,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7145
7210
[WARNING|trainer.py:803] 2025-04-26 22:06:43,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:44,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:44,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7146
8227
7211
[WARNING|trainer.py:803] 2025-04-26 22:06:45,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:45,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:45,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7147
7212
8228
[WARNING|trainer.py:803] 2025-04-26 22:06:46,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:46,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7148
[WARNING|trainer.py:803] 2025-04-26 22:06:47,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7213
[WARNING|trainer.py:803] 2025-04-26 22:06:47,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:48,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7149
8229
7214
[WARNING|trainer.py:803] 2025-04-26 22:06:49,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:49,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:49,497 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7150
7215
8230
[WARNING|trainer.py:803] 2025-04-26 22:06:50,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:50,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7151
[WARNING|trainer.py:803] 2025-04-26 22:06:51,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7216
[WARNING|trainer.py:803] 2025-04-26 22:06:51,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8231
[WARNING|trainer.py:803] 2025-04-26 22:06:51,981 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7152
7217
[WARNING|trainer.py:803] 2025-04-26 22:06:52,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:06:52,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:53,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7153
7218
8232
[WARNING|trainer.py:803] 2025-04-26 22:06:54,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:06:54,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7154
[WARNING|trainer.py:803] 2025-04-26 22:06:54,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7219
[WARNING|trainer.py:803] 2025-04-26 22:06:55,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8233
7155
[WARNING|trainer.py:803] 2025-04-26 22:06:55,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7220
[WARNING|trainer.py:803] 2025-04-26 22:06:56,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:06:56,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7156
[WARNING|trainer.py:803] 2025-04-26 22:06:56,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8234
7221
[WARNING|trainer.py:803] 2025-04-26 22:06:57,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7157
[WARNING|trainer.py:803] 2025-04-26 22:06:58,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:06:58,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7222
[WARNING|trainer.py:803] 2025-04-26 22:06:59,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8235
7158
[WARNING|trainer.py:803] 2025-04-26 22:06:59,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7223
[WARNING|trainer.py:803] 2025-04-26 22:07:00,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:07:00,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7159
[WARNING|trainer.py:803] 2025-04-26 22:07:00,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7224
8236
[WARNING|trainer.py:803] 2025-04-26 22:07:01,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7160
[WARNING|trainer.py:803] 2025-04-26 22:07:01,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:02,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7225
[WARNING|trainer.py:803] 2025-04-26 22:07:02,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8237
7161
[WARNING|trainer.py:803] 2025-04-26 22:07:03,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7226
[WARNING|trainer.py:803] 2025-04-26 22:07:03,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:04,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7162
[WARNING|trainer.py:803] 2025-04-26 22:07:04,470 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8238
7227
[WARNING|trainer.py:803] 2025-04-26 22:07:05,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7163
[WARNING|trainer.py:803] 2025-04-26 22:07:05,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:05,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7228
[WARNING|trainer.py:803] 2025-04-26 22:07:06,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8239
7164
[WARNING|trainer.py:803] 2025-04-26 22:07:06,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7229
[WARNING|trainer.py:803] 2025-04-26 22:07:07,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:07:07,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7165
[WARNING|trainer.py:803] 2025-04-26 22:07:08,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8240
7230
[WARNING|trainer.py:803] 2025-04-26 22:07:08,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:09,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7166
[WARNING|trainer.py:803] 2025-04-26 22:07:09,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7231
[WARNING|trainer.py:803] 2025-04-26 22:07:10,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8241
7167
[WARNING|trainer.py:803] 2025-04-26 22:07:10,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7232
[WARNING|trainer.py:803] 2025-04-26 22:07:11,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:11,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7168
[WARNING|trainer.py:803] 2025-04-26 22:07:11,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8242
7233
[WARNING|trainer.py:803] 2025-04-26 22:07:12,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:12,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7169
[WARNING|trainer.py:803] 2025-04-26 22:07:13,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7234
8243
[WARNING|trainer.py:803] 2025-04-26 22:07:13,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7170
[WARNING|trainer.py:803] 2025-04-26 22:07:14,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:07:14,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7235
[WARNING|trainer.py:803] 2025-04-26 22:07:15,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7171
8244
[WARNING|trainer.py:803] 2025-04-26 22:07:15,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7236
[WARNING|trainer.py:803] 2025-04-26 22:07:16,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:16,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7172
[WARNING|trainer.py:803] 2025-04-26 22:07:16,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7237
8245
[WARNING|trainer.py:803] 2025-04-26 22:07:17,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7173
[WARNING|trainer.py:803] 2025-04-26 22:07:18,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:18,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7238
[WARNING|trainer.py:803] 2025-04-26 22:07:18,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8246
7174
[WARNING|trainer.py:803] 2025-04-26 22:07:19,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:19,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7239
[WARNING|trainer.py:803] 2025-04-26 22:07:20,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7175
[WARNING|trainer.py:803] 2025-04-26 22:07:20,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8247
7240
[WARNING|trainer.py:803] 2025-04-26 22:07:21,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7176
[WARNING|trainer.py:803] 2025-04-26 22:07:21,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:21,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7241
[WARNING|trainer.py:803] 2025-04-26 22:07:22,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8248
7177
[WARNING|trainer.py:803] 2025-04-26 22:07:23,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7242
[WARNING|trainer.py:803] 2025-04-26 22:07:23,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:07:23,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7178
[WARNING|trainer.py:803] 2025-04-26 22:07:24,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8249
7243
[WARNING|trainer.py:803] 2025-04-26 22:07:25,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:25,358 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7179
[WARNING|trainer.py:803] 2025-04-26 22:07:25,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7244
[WARNING|trainer.py:803] 2025-04-26 22:07:26,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8250
7180
[WARNING|trainer.py:803] 2025-04-26 22:07:26,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:27,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7245
[WARNING|trainer.py:803] 2025-04-26 22:07:27,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7181
[WARNING|trainer.py:803] 2025-04-26 22:07:28,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8251
7246
[WARNING|trainer.py:803] 2025-04-26 22:07:28,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:29,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7182
[WARNING|trainer.py:803] 2025-04-26 22:07:29,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7247
[WARNING|trainer.py:803] 2025-04-26 22:07:29,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8252
7183
[WARNING|trainer.py:803] 2025-04-26 22:07:30,596 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7248
[WARNING|trainer.py:803] 2025-04-26 22:07:31,061 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:31,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7184
[WARNING|trainer.py:803] 2025-04-26 22:07:31,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8253
7249
[WARNING|trainer.py:803] 2025-04-26 22:07:32,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7185
[WARNING|trainer.py:803] 2025-04-26 22:07:32,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:33,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7250
[WARNING|trainer.py:803] 2025-04-26 22:07:33,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8254
7186
[WARNING|trainer.py:803] 2025-04-26 22:07:34,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:34,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7251
[WARNING|trainer.py:803] 2025-04-26 22:07:34,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7187
[WARNING|trainer.py:803] 2025-04-26 22:07:35,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8255
7252
[WARNING|trainer.py:803] 2025-04-26 22:07:36,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7188
[WARNING|trainer.py:803] 2025-04-26 22:07:36,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:36,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7253
[WARNING|trainer.py:803] 2025-04-26 22:07:37,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8256
7189
[WARNING|trainer.py:803] 2025-04-26 22:07:38,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:38,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7254
[WARNING|trainer.py:803] 2025-04-26 22:07:38,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7190
[WARNING|trainer.py:803] 2025-04-26 22:07:39,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8257
7255
[WARNING|trainer.py:803] 2025-04-26 22:07:39,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:40,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7191
[WARNING|trainer.py:803] 2025-04-26 22:07:40,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7256
[WARNING|trainer.py:803] 2025-04-26 22:07:41,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8258
7192
[WARNING|trainer.py:803] 2025-04-26 22:07:41,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7257
[WARNING|trainer.py:803] 2025-04-26 22:07:42,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:42,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7193
[WARNING|trainer.py:803] 2025-04-26 22:07:43,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8259
7258
[WARNING|trainer.py:803] 2025-04-26 22:07:43,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:43,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7194
[WARNING|trainer.py:803] 2025-04-26 22:07:44,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7259
[WARNING|trainer.py:803] 2025-04-26 22:07:44,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8260
7195
[WARNING|trainer.py:803] 2025-04-26 22:07:45,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7260
[WARNING|trainer.py:803] 2025-04-26 22:07:45,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:46,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7196
[WARNING|trainer.py:803] 2025-04-26 22:07:46,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8261
7261
[WARNING|trainer.py:803] 2025-04-26 22:07:47,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:47,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7197
[WARNING|trainer.py:803] 2025-04-26 22:07:47,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7262
8262
[WARNING|trainer.py:803] 2025-04-26 22:07:48,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7198
[WARNING|trainer.py:803] 2025-04-26 22:07:49,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:49,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7263
[WARNING|trainer.py:803] 2025-04-26 22:07:49,920 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7199
[WARNING|trainer.py:803] 2025-04-26 22:07:50,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8263
7264
[WARNING|trainer.py:803] 2025-04-26 22:07:51,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:07:51,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7200
[WARNING|trainer.py:803] 2025-04-26 22:07:51,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7265
[WARNING|trainer.py:803] 2025-04-26 22:07:52,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8264
7201
[WARNING|trainer.py:803] 2025-04-26 22:07:52,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:07:53,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7266
[WARNING|trainer.py:803] 2025-04-26 22:07:53,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7202
[WARNING|trainer.py:803] 2025-04-26 22:07:54,108 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8265
7267
[WARNING|trainer.py:803] 2025-04-26 22:07:54,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:07:55,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7203
[WARNING|trainer.py:803] 2025-04-26 22:07:55,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7268
8266
[WARNING|trainer.py:803] 2025-04-26 22:07:56,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7204
[WARNING|trainer.py:803] 2025-04-26 22:07:56,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:56,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7269
[WARNING|trainer.py:803] 2025-04-26 22:07:57,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7205
[WARNING|trainer.py:803] 2025-04-26 22:07:57,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8267
7270
[WARNING|trainer.py:803] 2025-04-26 22:07:58,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:07:58,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7206
[WARNING|trainer.py:803] 2025-04-26 22:07:58,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7271
8268
[WARNING|trainer.py:803] 2025-04-26 22:07:59,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7207
[WARNING|trainer.py:803] 2025-04-26 22:08:00,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:00,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7272
[WARNING|trainer.py:803] 2025-04-26 22:08:00,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7208
[WARNING|trainer.py:803] 2025-04-26 22:08:01,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8269
7273
[WARNING|trainer.py:803] 2025-04-26 22:08:02,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:02,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7209
[WARNING|trainer.py:803] 2025-04-26 22:08:02,667 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7274
8270
[WARNING|trainer.py:803] 2025-04-26 22:08:03,512 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:03,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7210
[WARNING|trainer.py:803] 2025-04-26 22:08:04,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7275
[WARNING|trainer.py:803] 2025-04-26 22:08:04,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:05,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8271
7211
7276
[WARNING|trainer.py:803] 2025-04-26 22:08:05,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:803] 2025-04-26 22:08:05,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:08:06,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7212
7277
8272
[WARNING|trainer.py:803] 2025-04-26 22:08:07,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:08:07,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7213
[WARNING|trainer.py:803] 2025-04-26 22:08:07,723 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7278
[WARNING|trainer.py:803] 2025-04-26 22:08:08,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8273
[WARNING|trainer.py:803] 2025-04-26 22:08:08,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7214
7279
[WARNING|trainer.py:803] 2025-04-26 22:08:09,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:09,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:10,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7215
8274
7280
[WARNING|trainer.py:803] 2025-04-26 22:08:10,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:11,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:11,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7216
7281
8275
[WARNING|trainer.py:803] 2025-04-26 22:08:12,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:12,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7217
[WARNING|trainer.py:803] 2025-04-26 22:08:12,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7282
[WARNING|trainer.py:803] 2025-04-26 22:08:13,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:13,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7218
8276
7283
[WARNING|trainer.py:803] 2025-04-26 22:08:14,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:14,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:15,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7219
7284
8277
[WARNING|trainer.py:803] 2025-04-26 22:08:15,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:16,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7220
7285
[WARNING|trainer.py:803] 2025-04-26 22:08:16,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:17,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:17,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7221
8278
7286
[WARNING|trainer.py:803] 2025-04-26 22:08:18,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:18,624 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:18,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7222
7287
8279
[WARNING|trainer.py:803] 2025-04-26 22:08:19,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:19,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7223
7288
[WARNING|trainer.py:803] 2025-04-26 22:08:20,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:20,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:08:21,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7224
8280
7289
[WARNING|trainer.py:803] 2025-04-26 22:08:22,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:22,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:22,456 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7225
7290
8281
[WARNING|trainer.py:803] 2025-04-26 22:08:23,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:23,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7226
7291
[WARNING|trainer.py:803] 2025-04-26 22:08:24,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:24,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:24,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7227
8282
7292
[WARNING|trainer.py:803] 2025-04-26 22:08:25,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:25,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:26,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7228
7293
8283
[WARNING|trainer.py:803] 2025-04-26 22:08:27,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:08:27,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7229
7294
[WARNING|trainer.py:803] 2025-04-26 22:08:27,919 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:08:28,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:28,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7230
8284
7295
[WARNING|trainer.py:803] 2025-04-26 22:08:29,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:29,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7231
[WARNING|trainer.py:803] 2025-04-26 22:08:29,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7296
8285
[WARNING|trainer.py:803] 2025-04-26 22:08:30,796 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7232
[WARNING|trainer.py:803] 2025-04-26 22:08:31,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:31,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7297
[WARNING|trainer.py:803] 2025-04-26 22:08:32,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7233
[WARNING|trainer.py:803] 2025-04-26 22:08:32,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8286
7298
[WARNING|trainer.py:803] 2025-04-26 22:08:33,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:33,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7234
[WARNING|trainer.py:803] 2025-04-26 22:08:33,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7299
8287
[WARNING|trainer.py:803] 2025-04-26 22:08:34,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7235
[WARNING|trainer.py:803] 2025-04-26 22:08:34,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:35,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7300
[WARNING|trainer.py:803] 2025-04-26 22:08:35,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7236
[WARNING|trainer.py:803] 2025-04-26 22:08:36,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8288
7301
[WARNING|trainer.py:803] 2025-04-26 22:08:36,951 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:37,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7237
[WARNING|trainer.py:803] 2025-04-26 22:08:37,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7302
8289
[WARNING|trainer.py:803] 2025-04-26 22:08:38,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7238
[WARNING|trainer.py:803] 2025-04-26 22:08:38,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:39,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7303
[WARNING|trainer.py:803] 2025-04-26 22:08:39,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7239
8290
[WARNING|trainer.py:803] 2025-04-26 22:08:40,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7304
[WARNING|trainer.py:803] 2025-04-26 22:08:40,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:40,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7240
[WARNING|trainer.py:803] 2025-04-26 22:08:41,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8291
7305
[WARNING|trainer.py:803] 2025-04-26 22:08:41,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7241
[WARNING|trainer.py:803] 2025-04-26 22:08:42,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:42,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7306
[WARNING|trainer.py:803] 2025-04-26 22:08:43,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7242
8292
[WARNING|trainer.py:803] 2025-04-26 22:08:43,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7307
[WARNING|trainer.py:803] 2025-04-26 22:08:44,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:08:44,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7243
[WARNING|trainer.py:803] 2025-04-26 22:08:45,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7308
[WARNING|trainer.py:803] 2025-04-26 22:08:45,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8293
7244
[WARNING|trainer.py:803] 2025-04-26 22:08:46,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:46,638 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7309
[WARNING|trainer.py:803] 2025-04-26 22:08:46,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7245
[WARNING|trainer.py:803] 2025-04-26 22:08:47,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8294
7310
[WARNING|trainer.py:803] 2025-04-26 22:08:48,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:48,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7246
[WARNING|trainer.py:803] 2025-04-26 22:08:48,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7311
[WARNING|trainer.py:803] 2025-04-26 22:08:49,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8295
7247
[WARNING|trainer.py:803] 2025-04-26 22:08:50,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:08:50,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7312
[WARNING|trainer.py:803] 2025-04-26 22:08:50,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7248
[WARNING|trainer.py:803] 2025-04-26 22:08:51,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8296
7313
[WARNING|trainer.py:803] 2025-04-26 22:08:51,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7249
[WARNING|trainer.py:803] 2025-04-26 22:08:52,371 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:52,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7314
[WARNING|trainer.py:803] 2025-04-26 22:08:53,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7250
8297
[WARNING|trainer.py:803] 2025-04-26 22:08:53,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7315
[WARNING|trainer.py:803] 2025-04-26 22:08:54,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:54,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7251
[WARNING|trainer.py:803] 2025-04-26 22:08:55,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8298
7316
[WARNING|trainer.py:803] 2025-04-26 22:08:55,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7252
[WARNING|trainer.py:803] 2025-04-26 22:08:56,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:56,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7317
[WARNING|trainer.py:803] 2025-04-26 22:08:56,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8299
7253
[WARNING|trainer.py:803] 2025-04-26 22:08:57,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:08:57,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7318
[WARNING|trainer.py:803] 2025-04-26 22:08:58,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7254
[WARNING|trainer.py:803] 2025-04-26 22:08:58,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8300
7319
[WARNING|trainer.py:803] 2025-04-26 22:08:59,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:08:59,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7255
[WARNING|trainer.py:803] 2025-04-26 22:09:00,161 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7320
[WARNING|trainer.py:803] 2025-04-26 22:09:00,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8301
7256
[WARNING|trainer.py:803] 2025-04-26 22:09:01,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:01,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7321
[WARNING|trainer.py:803] 2025-04-26 22:09:01,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7257
[WARNING|trainer.py:803] 2025-04-26 22:09:02,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8302
7322
[WARNING|trainer.py:803] 2025-04-26 22:09:03,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:03,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7258
[WARNING|trainer.py:803] 2025-04-26 22:09:03,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7323
8303
[WARNING|trainer.py:803] 2025-04-26 22:09:04,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7259
[WARNING|trainer.py:803] 2025-04-26 22:09:05,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:09:05,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7324
[WARNING|trainer.py:803] 2025-04-26 22:09:05,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7260
8304
[WARNING|trainer.py:803] 2025-04-26 22:09:06,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7325
[WARNING|trainer.py:803] 2025-04-26 22:09:06,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:06,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7261
[WARNING|trainer.py:803] 2025-04-26 22:09:07,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8305
7326
[WARNING|trainer.py:803] 2025-04-26 22:09:08,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7262
[WARNING|trainer.py:803] 2025-04-26 22:09:08,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:08,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7327
[WARNING|trainer.py:803] 2025-04-26 22:09:09,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8306
7263
[WARNING|trainer.py:803] 2025-04-26 22:09:10,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:10,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7328
[WARNING|trainer.py:803] 2025-04-26 22:09:10,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7264
8307
[WARNING|trainer.py:803] 2025-04-26 22:09:11,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7329
[WARNING|trainer.py:803] 2025-04-26 22:09:11,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:12,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7265
[WARNING|trainer.py:803] 2025-04-26 22:09:12,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7330
8308
[WARNING|trainer.py:803] 2025-04-26 22:09:13,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7266
[WARNING|trainer.py:803] 2025-04-26 22:09:13,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:09:13,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7331
[WARNING|trainer.py:803] 2025-04-26 22:09:14,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8309
7267
[WARNING|trainer.py:803] 2025-04-26 22:09:15,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7332
[WARNING|trainer.py:803] 2025-04-26 22:09:15,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:15,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7268
[WARNING|trainer.py:803] 2025-04-26 22:09:16,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8310
7333
[WARNING|trainer.py:803] 2025-04-26 22:09:16,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:17,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7269
[WARNING|trainer.py:803] 2025-04-26 22:09:17,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7334
8311
[WARNING|trainer.py:803] 2025-04-26 22:09:18,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7270
[WARNING|trainer.py:803] 2025-04-26 22:09:18,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:09:18,984 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7335
[WARNING|trainer.py:803] 2025-04-26 22:09:19,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8312
7271
[WARNING|trainer.py:803] 2025-04-26 22:09:20,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7336
[WARNING|trainer.py:803] 2025-04-26 22:09:20,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:20,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7272
[WARNING|trainer.py:803] 2025-04-26 22:09:21,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8313
7337
[WARNING|trainer.py:803] 2025-04-26 22:09:22,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:22,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7273
[WARNING|trainer.py:803] 2025-04-26 22:09:22,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7338
[WARNING|trainer.py:803] 2025-04-26 22:09:23,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8314
7274
[WARNING|trainer.py:803] 2025-04-26 22:09:23,798 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7339
[WARNING|trainer.py:803] 2025-04-26 22:09:24,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:24,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7275
[WARNING|trainer.py:803] 2025-04-26 22:09:25,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8315
7340
[WARNING|trainer.py:803] 2025-04-26 22:09:25,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:25,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7276
[WARNING|trainer.py:803] 2025-04-26 22:09:26,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7341
8316
[WARNING|trainer.py:803] 2025-04-26 22:09:27,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7277
[WARNING|trainer.py:803] 2025-04-26 22:09:27,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:27,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7342
[WARNING|trainer.py:803] 2025-04-26 22:09:28,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8317
7278
[WARNING|trainer.py:803] 2025-04-26 22:09:28,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7343
[WARNING|trainer.py:803] 2025-04-26 22:09:29,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:29,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7279
[WARNING|trainer.py:803] 2025-04-26 22:09:29,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8318
7344
[WARNING|trainer.py:803] 2025-04-26 22:09:30,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:31,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7280
[WARNING|trainer.py:803] 2025-04-26 22:09:31,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7345
8319
[WARNING|trainer.py:803] 2025-04-26 22:09:32,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7281
[WARNING|trainer.py:803] 2025-04-26 22:09:32,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:32,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7346
[WARNING|trainer.py:803] 2025-04-26 22:09:33,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8320
7282
[WARNING|trainer.py:803] 2025-04-26 22:09:33,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7347
[WARNING|trainer.py:803] 2025-04-26 22:09:34,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:34,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7283
[WARNING|trainer.py:803] 2025-04-26 22:09:35,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8321
7348
[WARNING|trainer.py:803] 2025-04-26 22:09:35,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:36,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7284
[WARNING|trainer.py:803] 2025-04-26 22:09:36,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7349
[WARNING|trainer.py:803] 2025-04-26 22:09:36,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8322
7285
[WARNING|trainer.py:803] 2025-04-26 22:09:37,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:37,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7350
[WARNING|trainer.py:803] 2025-04-26 22:09:38,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7286
8323
[WARNING|trainer.py:803] 2025-04-26 22:09:38,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7351
[WARNING|trainer.py:803] 2025-04-26 22:09:39,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:39,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7287
[WARNING|trainer.py:803] 2025-04-26 22:09:40,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8324
7352
[WARNING|trainer.py:803] 2025-04-26 22:09:40,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7288
[WARNING|trainer.py:803] 2025-04-26 22:09:41,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:41,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7353
[WARNING|trainer.py:803] 2025-04-26 22:09:41,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8325
7289
[WARNING|trainer.py:803] 2025-04-26 22:09:42,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:43,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7354
[WARNING|trainer.py:803] 2025-04-26 22:09:43,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7290
8326
[WARNING|trainer.py:803] 2025-04-26 22:09:43,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7355
[WARNING|trainer.py:803] 2025-04-26 22:09:44,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:44,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7291
[WARNING|trainer.py:803] 2025-04-26 22:09:45,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8327
7356
[WARNING|trainer.py:803] 2025-04-26 22:09:45,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7292
[WARNING|trainer.py:803] 2025-04-26 22:09:46,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:46,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7357
[WARNING|trainer.py:803] 2025-04-26 22:09:46,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8328
7293
[WARNING|trainer.py:803] 2025-04-26 22:09:47,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:47,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7358
[WARNING|trainer.py:803] 2025-04-26 22:09:48,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7294
8329
[WARNING|trainer.py:803] 2025-04-26 22:09:48,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7359
[WARNING|trainer.py:803] 2025-04-26 22:09:49,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:09:49,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7295
[WARNING|trainer.py:803] 2025-04-26 22:09:50,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8330
7360
[WARNING|trainer.py:803] 2025-04-26 22:09:50,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7296
[WARNING|trainer.py:803] 2025-04-26 22:09:51,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:51,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7361
[WARNING|trainer.py:803] 2025-04-26 22:09:51,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8331
7297
[WARNING|trainer.py:803] 2025-04-26 22:09:52,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7362
[WARNING|trainer.py:803] 2025-04-26 22:09:53,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:53,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7298
[WARNING|trainer.py:803] 2025-04-26 22:09:53,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8332
7363
[WARNING|trainer.py:803] 2025-04-26 22:09:54,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:54,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7299
[WARNING|trainer.py:803] 2025-04-26 22:09:55,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7364
8333
[WARNING|trainer.py:803] 2025-04-26 22:09:55,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7300
[WARNING|trainer.py:803] 2025-04-26 22:09:56,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:09:56,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7365
[WARNING|trainer.py:803] 2025-04-26 22:09:56,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7301
8334
[WARNING|trainer.py:803] 2025-04-26 22:09:57,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7366
[WARNING|trainer.py:803] 2025-04-26 22:09:58,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:09:58,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7302
[WARNING|trainer.py:803] 2025-04-26 22:09:58,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8335
7367
[WARNING|trainer.py:803] 2025-04-26 22:09:59,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7303
[WARNING|trainer.py:803] 2025-04-26 22:10:00,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:00,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7368
[WARNING|trainer.py:803] 2025-04-26 22:10:00,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8336
7304
[WARNING|trainer.py:803] 2025-04-26 22:10:01,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:01,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7369
[WARNING|trainer.py:803] 2025-04-26 22:10:01,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7305
8337
[WARNING|trainer.py:803] 2025-04-26 22:10:02,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7370
[WARNING|trainer.py:803] 2025-04-26 22:10:03,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:03,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7306
[WARNING|trainer.py:803] 2025-04-26 22:10:03,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7371
8338
[WARNING|trainer.py:803] 2025-04-26 22:10:04,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7307
[WARNING|trainer.py:803] 2025-04-26 22:10:05,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:05,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7372
[WARNING|trainer.py:803] 2025-04-26 22:10:05,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8339
7308
[WARNING|trainer.py:803] 2025-04-26 22:10:06,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7373
[WARNING|trainer.py:803] 2025-04-26 22:10:06,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:06,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7309
[WARNING|trainer.py:803] 2025-04-26 22:10:07,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8340
7374
[WARNING|trainer.py:803] 2025-04-26 22:10:08,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7310
[WARNING|trainer.py:803] 2025-04-26 22:10:08,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:08,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7375
[WARNING|trainer.py:803] 2025-04-26 22:10:09,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8341
7311
[WARNING|trainer.py:803] 2025-04-26 22:10:10,051 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7376
[WARNING|trainer.py:803] 2025-04-26 22:10:10,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:10,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7312
[WARNING|trainer.py:803] 2025-04-26 22:10:11,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8342
7377
[WARNING|trainer.py:803] 2025-04-26 22:10:11,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:12,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7313
[WARNING|trainer.py:803] 2025-04-26 22:10:12,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7378
[WARNING|trainer.py:803] 2025-04-26 22:10:13,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8343
7314
[WARNING|trainer.py:803] 2025-04-26 22:10:13,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:14,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7379
[WARNING|trainer.py:803] 2025-04-26 22:10:14,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7315
8344
[WARNING|trainer.py:803] 2025-04-26 22:10:14,974 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7380
[WARNING|trainer.py:803] 2025-04-26 22:10:15,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:15,673 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7316
[WARNING|trainer.py:803] 2025-04-26 22:10:16,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8345
7381
[WARNING|trainer.py:803] 2025-04-26 22:10:16,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7317
[WARNING|trainer.py:803] 2025-04-26 22:10:17,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:17,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7382
[WARNING|trainer.py:803] 2025-04-26 22:10:18,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8346
7318
[WARNING|trainer.py:803] 2025-04-26 22:10:18,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7383
[WARNING|trainer.py:803] 2025-04-26 22:10:19,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:19,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7319
[WARNING|trainer.py:803] 2025-04-26 22:10:19,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8347
7384
[WARNING|trainer.py:803] 2025-04-26 22:10:20,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:20,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7320
[WARNING|trainer.py:803] 2025-04-26 22:10:21,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7385
8348
[WARNING|trainer.py:803] 2025-04-26 22:10:21,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7321
[WARNING|trainer.py:803] 2025-04-26 22:10:22,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:22,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7386
[WARNING|trainer.py:803] 2025-04-26 22:10:23,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8349
7322
[WARNING|trainer.py:803] 2025-04-26 22:10:23,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7387
[WARNING|trainer.py:803] 2025-04-26 22:10:24,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:24,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7323
[WARNING|trainer.py:803] 2025-04-26 22:10:24,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7388
8350
[WARNING|trainer.py:803] 2025-04-26 22:10:25,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:26,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7324
[WARNING|trainer.py:803] 2025-04-26 22:10:26,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7389
[WARNING|trainer.py:803] 2025-04-26 22:10:26,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8351
[WARNING|trainer.py:803] 2025-04-26 22:10:27,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7325
7390
[WARNING|trainer.py:803] 2025-04-26 22:10:27,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:28,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:28,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7326
8352
7391
[WARNING|trainer.py:803] 2025-04-26 22:10:29,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:29,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:29,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7327
7392
8353
[WARNING|trainer.py:803] 2025-04-26 22:10:30,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:31,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7328
[WARNING|trainer.py:803] 2025-04-26 22:10:31,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7393
[WARNING|trainer.py:803] 2025-04-26 22:10:31,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:32,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8354
7329
7394
[WARNING|trainer.py:803] 2025-04-26 22:10:33,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:33,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:33,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7330
7395
8355
[WARNING|trainer.py:803] 2025-04-26 22:10:34,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:10:34,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:34,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7331
7396
[WARNING|trainer.py:803] 2025-04-26 22:10:35,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8356
[WARNING|trainer.py:803] 2025-04-26 22:10:36,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7332
7397
[WARNING|trainer.py:803] 2025-04-26 22:10:36,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:36,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:37,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8357
7333
7398
[WARNING|trainer.py:803] 2025-04-26 22:10:38,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:38,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:38,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7334
7399
8358
[WARNING|trainer.py:803] 2025-04-26 22:10:39,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:39,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7335
[WARNING|trainer.py:803] 2025-04-26 22:10:40,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7400
[WARNING|trainer.py:803] 2025-04-26 22:10:40,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8359
[WARNING|trainer.py:803] 2025-04-26 22:10:40,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7336
7401
[WARNING|trainer.py:803] 2025-04-26 22:10:41,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:41,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:42,162 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7337
8360
7402
[WARNING|trainer.py:803] 2025-04-26 22:10:43,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:43,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:43,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7338
7403
8361
[WARNING|trainer.py:803] 2025-04-26 22:10:44,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:44,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7339
[WARNING|trainer.py:803] 2025-04-26 22:10:45,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7404
[WARNING|trainer.py:803] 2025-04-26 22:10:45,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8362
[WARNING|trainer.py:803] 2025-04-26 22:10:45,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7340
7405
[WARNING|trainer.py:803] 2025-04-26 22:10:46,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:46,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:47,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7341
7406
8363
[WARNING|trainer.py:803] 2025-04-26 22:10:48,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:48,446 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:48,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7342
7407
8364
[WARNING|trainer.py:803] 2025-04-26 22:10:49,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:49,700 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7343
7408
[WARNING|trainer.py:803] 2025-04-26 22:10:50,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:50,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:50,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7344
8365
7409
[WARNING|trainer.py:803] 2025-04-26 22:10:52,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:52,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:52,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7345
7410
8366
[WARNING|trainer.py:803] 2025-04-26 22:10:53,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:53,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7346
[WARNING|trainer.py:803] 2025-04-26 22:10:53,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7411
8367
[WARNING|trainer.py:803] 2025-04-26 22:10:54,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:10:54,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7347
7412
[WARNING|trainer.py:803] 2025-04-26 22:10:55,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:55,772 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:10:55,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7348
8368
7413
[WARNING|trainer.py:803] 2025-04-26 22:10:57,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:10:57,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:57,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7349
7414
8369
[WARNING|trainer.py:803] 2025-04-26 22:10:58,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:10:58,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7350
[WARNING|trainer.py:803] 2025-04-26 22:10:58,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7415
[WARNING|trainer.py:803] 2025-04-26 22:10:59,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8370
[WARNING|trainer.py:803] 2025-04-26 22:10:59,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7351
7416
[WARNING|trainer.py:803] 2025-04-26 22:11:00,571 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:00,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:01,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7352
8371
7417
[WARNING|trainer.py:803] 2025-04-26 22:11:02,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:02,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:02,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7353
7418
8372
[WARNING|trainer.py:803] 2025-04-26 22:11:03,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:03,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
7354
7419
[WARNING|trainer.py:803] 2025-04-26 22:11:04,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:04,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:04,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7355
8373
7420
[WARNING|trainer.py:803] 2025-04-26 22:11:05,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:05,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:06,055 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7356
7421
8374
[WARNING|trainer.py:803] 2025-04-26 22:11:06,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:07,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7357
[WARNING|trainer.py:803] 2025-04-26 22:11:07,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7422
[WARNING|trainer.py:803] 2025-04-26 22:11:08,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:08,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7358
8375
7423
[WARNING|trainer.py:803] 2025-04-26 22:11:09,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:09,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:09,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7359
7424
8376
[WARNING|trainer.py:803] 2025-04-26 22:11:10,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:11,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7360
7425
[WARNING|trainer.py:803] 2025-04-26 22:11:11,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:11,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8377
[WARNING|trainer.py:803] 2025-04-26 22:11:12,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7361
7426
[WARNING|trainer.py:803] 2025-04-26 22:11:13,025 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:13,185 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:13,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7362
8378
7427
[WARNING|trainer.py:803] 2025-04-26 22:11:14,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:14,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:14,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7363
7428
8379
[WARNING|trainer.py:803] 2025-04-26 22:11:15,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:15,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7364
7429
[WARNING|trainer.py:803] 2025-04-26 22:11:16,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:16,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:17,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8380
7365
7430
[WARNING|trainer.py:803] 2025-04-26 22:11:18,097 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:18,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:18,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7366
7431
8381
[WARNING|trainer.py:803] 2025-04-26 22:11:19,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:19,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7367
[WARNING|trainer.py:803] 2025-04-26 22:11:19,906 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7432
[WARNING|trainer.py:803] 2025-04-26 22:11:20,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8382
[WARNING|trainer.py:803] 2025-04-26 22:11:20,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7368
7433
[WARNING|trainer.py:803] 2025-04-26 22:11:21,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:21,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:22,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7369
8383
7434
[WARNING|trainer.py:803] 2025-04-26 22:11:23,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:23,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:23,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7370
7435
8384
[WARNING|trainer.py:803] 2025-04-26 22:11:24,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:24,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7371
7436
[WARNING|trainer.py:803] 2025-04-26 22:11:25,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:25,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:25,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7372
8385
7437
[WARNING|trainer.py:803] 2025-04-26 22:11:26,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:26,952 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:27,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7373
7438
8386
[WARNING|trainer.py:803] 2025-04-26 22:11:28,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:28,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7374
7439
[WARNING|trainer.py:803] 2025-04-26 22:11:28,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:29,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:29,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7375
7440
8387
[WARNING|trainer.py:803] 2025-04-26 22:11:30,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:30,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:31,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7376
7441
8388
[WARNING|trainer.py:803] 2025-04-26 22:11:31,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:32,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7377
7442
[WARNING|trainer.py:803] 2025-04-26 22:11:32,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:33,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:33,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7378
8389
7443
[WARNING|trainer.py:803] 2025-04-26 22:11:34,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:34,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:34,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7379
7444
8390
[WARNING|trainer.py:803] 2025-04-26 22:11:35,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:35,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7380
7445
[WARNING|trainer.py:803] 2025-04-26 22:11:36,427 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:36,903 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:37,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7381
8391
7446
[WARNING|trainer.py:803] 2025-04-26 22:11:38,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:38,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:38,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7382
7447
8392
[WARNING|trainer.py:803] 2025-04-26 22:11:39,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:39,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7383
7448
[WARNING|trainer.py:803] 2025-04-26 22:11:40,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:40,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:40,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8393
7384
7449
[WARNING|trainer.py:803] 2025-04-26 22:11:41,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:41,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:41,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7385
7450
8394
[WARNING|trainer.py:803] 2025-04-26 22:11:43,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:43,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:43,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7386
7451
8395
[WARNING|trainer.py:803] 2025-04-26 22:11:44,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:44,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7452
7387
[WARNING|trainer.py:803] 2025-04-26 22:11:45,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:45,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:45,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8396
7453
7388
[WARNING|trainer.py:803] 2025-04-26 22:11:46,902 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:46,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:47,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7454
7389
8397
[WARNING|trainer.py:803] 2025-04-26 22:11:48,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:48,297 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7455
[WARNING|trainer.py:803] 2025-04-26 22:11:48,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7390
[WARNING|trainer.py:803] 2025-04-26 22:11:49,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8398
[WARNING|trainer.py:803] 2025-04-26 22:11:49,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7456
7391
[WARNING|trainer.py:803] 2025-04-26 22:11:50,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:50,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:50,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7457
7392
8399
[WARNING|trainer.py:803] 2025-04-26 22:11:51,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:52,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:11:52,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7458
7393
8400
[WARNING|trainer.py:803] 2025-04-26 22:11:53,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:53,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7459
7394
[WARNING|trainer.py:803] 2025-04-26 22:11:53,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:54,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8401
[WARNING|trainer.py:803] 2025-04-26 22:11:54,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7460
7395
[WARNING|trainer.py:803] 2025-04-26 22:11:55,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:55,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:55,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8402
7461
7396
[WARNING|trainer.py:803] 2025-04-26 22:11:56,818 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:56,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:57,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7462
7397
8403
[WARNING|trainer.py:803] 2025-04-26 22:11:58,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:11:58,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:11:58,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7463
7398
8404
[WARNING|trainer.py:803] 2025-04-26 22:11:59,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:11:59,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7464
[WARNING|trainer.py:803] 2025-04-26 22:11:59,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7399
8405
[WARNING|trainer.py:803] 2025-04-26 22:12:00,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:00,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7465
7400
[WARNING|trainer.py:803] 2025-04-26 22:12:01,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:01,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8406
7466
[WARNING|trainer.py:803] 2025-04-26 22:12:02,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7401
[WARNING|trainer.py:803] 2025-04-26 22:12:02,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:02,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7467
[WARNING|trainer.py:803] 2025-04-26 22:12:03,393 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8407
7402
[WARNING|trainer.py:803] 2025-04-26 22:12:04,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:04,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7468
[WARNING|trainer.py:803] 2025-04-26 22:12:04,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8408
7403
[WARNING|trainer.py:803] 2025-04-26 22:12:05,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:05,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7469
[WARNING|trainer.py:803] 2025-04-26 22:12:05,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7404
8409
[WARNING|trainer.py:803] 2025-04-26 22:12:06,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7470
[WARNING|trainer.py:803] 2025-04-26 22:12:07,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:07,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7405
[WARNING|trainer.py:803] 2025-04-26 22:12:07,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8410
7471
[WARNING|trainer.py:803] 2025-04-26 22:12:08,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7406
[WARNING|trainer.py:803] 2025-04-26 22:12:08,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:09,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8411
7472
[WARNING|trainer.py:803] 2025-04-26 22:12:09,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7407
[WARNING|trainer.py:803] 2025-04-26 22:12:10,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:10,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7473
[WARNING|trainer.py:803] 2025-04-26 22:12:10,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8412
7408
[WARNING|trainer.py:803] 2025-04-26 22:12:11,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:11,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7474
[WARNING|trainer.py:803] 2025-04-26 22:12:12,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7409
8413
[WARNING|trainer.py:803] 2025-04-26 22:12:12,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7475
[WARNING|trainer.py:803] 2025-04-26 22:12:13,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:13,292 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7410
8414
[WARNING|trainer.py:803] 2025-04-26 22:12:13,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7476
[WARNING|trainer.py:803] 2025-04-26 22:12:14,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:14,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7411
[WARNING|trainer.py:803] 2025-04-26 22:12:15,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8415
7477
[WARNING|trainer.py:803] 2025-04-26 22:12:15,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:16,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7412
[WARNING|trainer.py:803] 2025-04-26 22:12:16,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7478
8416
[WARNING|trainer.py:803] 2025-04-26 22:12:17,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7413
[WARNING|trainer.py:803] 2025-04-26 22:12:17,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:17,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7479
[WARNING|trainer.py:803] 2025-04-26 22:12:18,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8417
7414
[WARNING|trainer.py:803] 2025-04-26 22:12:18,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:19,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7480
[WARNING|trainer.py:803] 2025-04-26 22:12:19,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7415
8418
[WARNING|trainer.py:803] 2025-04-26 22:12:20,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7481
[WARNING|trainer.py:803] 2025-04-26 22:12:20,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:20,743 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7416
[WARNING|trainer.py:803] 2025-04-26 22:12:21,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8419
7482
[WARNING|trainer.py:803] 2025-04-26 22:12:21,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:22,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7417
[WARNING|trainer.py:803] 2025-04-26 22:12:22,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8420
7483
[WARNING|trainer.py:803] 2025-04-26 22:12:23,224 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:23,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7418
[WARNING|trainer.py:803] 2025-04-26 22:12:23,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7484
8421
[WARNING|trainer.py:803] 2025-04-26 22:12:24,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
AnswerNo
7419
[WARNING|trainer.py:803] 2025-04-26 22:12:24,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:25,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7485
[WARNING|trainer.py:803] 2025-04-26 22:12:25,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8422
7420
[WARNING|trainer.py:803] 2025-04-26 22:12:26,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7486
[WARNING|trainer.py:803] 2025-04-26 22:12:26,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:26,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7421
[WARNING|trainer.py:803] 2025-04-26 22:12:27,413 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8423
7487
[WARNING|trainer.py:803] 2025-04-26 22:12:28,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:28,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7422
[WARNING|trainer.py:803] 2025-04-26 22:12:28,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7488
8424
[WARNING|trainer.py:803] 2025-04-26 22:12:29,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7423
[WARNING|trainer.py:803] 2025-04-26 22:12:29,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:29,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7489
8425
[WARNING|trainer.py:803] 2025-04-26 22:12:30,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7424
[WARNING|trainer.py:803] 2025-04-26 22:12:31,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:31,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7490
[WARNING|trainer.py:803] 2025-04-26 22:12:31,884 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8426
7425
[WARNING|trainer.py:803] 2025-04-26 22:12:32,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7491
[WARNING|trainer.py:803] 2025-04-26 22:12:32,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:33,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8427
7426
[WARNING|trainer.py:803] 2025-04-26 22:12:33,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7492
[WARNING|trainer.py:803] 2025-04-26 22:12:34,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:34,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7427
[WARNING|trainer.py:803] 2025-04-26 22:12:34,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8428
7493
[WARNING|trainer.py:803] 2025-04-26 22:12:35,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:35,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:35,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7428
7494
8429
[WARNING|trainer.py:803] 2025-04-26 22:12:36,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:37,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7429
[WARNING|trainer.py:803] 2025-04-26 22:12:37,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7495
[WARNING|trainer.py:803] 2025-04-26 22:12:38,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8430
[WARNING|trainer.py:803] 2025-04-26 22:12:38,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7430
7496
[WARNING|trainer.py:803] 2025-04-26 22:12:38,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:39,291 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:39,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7431
8431
7497
[WARNING|trainer.py:803] 2025-04-26 22:12:40,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:40,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:40,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7432
8432
7498
[WARNING|trainer.py:803] 2025-04-26 22:12:41,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:41,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:42,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7433
7499
8433
[WARNING|trainer.py:803] 2025-04-26 22:12:43,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:43,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:43,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7434
7500
8434
[WARNING|trainer.py:803] 2025-04-26 22:12:44,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:44,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7435
[WARNING|trainer.py:803] 2025-04-26 22:12:44,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7501
8435
[WARNING|trainer.py:803] 2025-04-26 22:12:45,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7436
[WARNING|trainer.py:803] 2025-04-26 22:12:46,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:46,341 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7502
[WARNING|trainer.py:803] 2025-04-26 22:12:46,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8436
7437
[WARNING|trainer.py:803] 2025-04-26 22:12:47,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:47,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:48,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7503
7438
8437
[WARNING|trainer.py:803] 2025-04-26 22:12:48,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:49,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:49,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7504
7439
8438
[WARNING|trainer.py:803] 2025-04-26 22:12:50,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:50,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:12:50,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7505
7440
8439
[WARNING|trainer.py:803] 2025-04-26 22:12:51,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:51,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7441
[WARNING|trainer.py:803] 2025-04-26 22:12:52,261 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7506
8440
[WARNING|trainer.py:803] 2025-04-26 22:12:53,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:53,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7442
7507
[WARNING|trainer.py:803] 2025-04-26 22:12:53,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:54,288 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8441
[WARNING|trainer.py:803] 2025-04-26 22:12:54,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7443
7508
[WARNING|trainer.py:803] 2025-04-26 22:12:55,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:55,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:12:55,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7444
8442
7509
[WARNING|trainer.py:803] 2025-04-26 22:12:56,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:12:56,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7445
[WARNING|trainer.py:803] 2025-04-26 22:12:57,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8443
7510
[WARNING|trainer.py:803] 2025-04-26 22:12:58,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:12:58,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7446
[WARNING|trainer.py:803] 2025-04-26 22:12:58,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8444
7511
[WARNING|trainer.py:803] 2025-04-26 22:12:59,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7447
[WARNING|trainer.py:803] 2025-04-26 22:12:59,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:00,071 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:00,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7512
8445
7448
[WARNING|trainer.py:803] 2025-04-26 22:13:01,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:01,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:01,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7513
7449
8446
[WARNING|trainer.py:803] 2025-04-26 22:13:02,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:03,016 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:03,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7450
7514
8447
[WARNING|trainer.py:803] 2025-04-26 22:13:04,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:04,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:04,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7451
7515
8448
[WARNING|trainer.py:803] 2025-04-26 22:13:05,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:05,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7452
[WARNING|trainer.py:803] 2025-04-26 22:13:05,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7516
8449
[WARNING|trainer.py:803] 2025-04-26 22:13:06,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:07,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7453
[WARNING|trainer.py:803] 2025-04-26 22:13:07,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7517
[WARNING|trainer.py:803] 2025-04-26 22:13:08,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8450
7454
[WARNING|trainer.py:803] 2025-04-26 22:13:08,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:09,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7518
[WARNING|trainer.py:803] 2025-04-26 22:13:09,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7455
8451
[WARNING|trainer.py:803] 2025-04-26 22:13:09,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7519
[WARNING|trainer.py:803] 2025-04-26 22:13:10,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:10,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7456
8452
[WARNING|trainer.py:803] 2025-04-26 22:13:11,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:11,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7520
[WARNING|trainer.py:803] 2025-04-26 22:13:12,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7457
8453
[WARNING|trainer.py:803] 2025-04-26 22:13:12,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:13,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7521
[WARNING|trainer.py:803] 2025-04-26 22:13:13,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7458
8454
[WARNING|trainer.py:803] 2025-04-26 22:13:14,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:14,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7522
7459
[WARNING|trainer.py:803] 2025-04-26 22:13:14,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:15,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8455
[WARNING|trainer.py:803] 2025-04-26 22:13:15,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7460
7523
[WARNING|trainer.py:803] 2025-04-26 22:13:16,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:16,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:17,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8456
7461
7524
[WARNING|trainer.py:803] 2025-04-26 22:13:17,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:18,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8457
[WARNING|trainer.py:803] 2025-04-26 22:13:18,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7462
7525
[WARNING|trainer.py:803] 2025-04-26 22:13:19,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:19,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7463
[WARNING|trainer.py:803] 2025-04-26 22:13:19,969 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8458
7526
[WARNING|trainer.py:803] 2025-04-26 22:13:20,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:20,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7464
[WARNING|trainer.py:803] 2025-04-26 22:13:21,378 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8459
7527
[WARNING|trainer.py:803] 2025-04-26 22:13:21,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:22,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7465
[WARNING|trainer.py:803] 2025-04-26 22:13:22,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8460
[WARNING|trainer.py:803] 2025-04-26 22:13:23,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7528
7466
[WARNING|trainer.py:803] 2025-04-26 22:13:23,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:24,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8461
[WARNING|trainer.py:803] 2025-04-26 22:13:24,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7529
7467
[WARNING|trainer.py:803] 2025-04-26 22:13:25,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:25,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:25,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8462
7468
7530
[WARNING|trainer.py:803] 2025-04-26 22:13:26,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:26,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:27,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7469
8463
7531
[WARNING|trainer.py:803] 2025-04-26 22:13:28,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:28,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:28,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7470
8464
7532
[WARNING|trainer.py:803] 2025-04-26 22:13:29,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:29,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7471
[WARNING|trainer.py:803] 2025-04-26 22:13:30,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8465
7533
[WARNING|trainer.py:803] 2025-04-26 22:13:30,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7472
[WARNING|trainer.py:803] 2025-04-26 22:13:31,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:31,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8466
[WARNING|trainer.py:803] 2025-04-26 22:13:31,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7534
7473
[WARNING|trainer.py:803] 2025-04-26 22:13:32,758 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:32,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:33,210 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7535
8467
7474
[WARNING|trainer.py:803] 2025-04-26 22:13:34,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:34,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:34,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7536
8468
7475
[WARNING|trainer.py:803] 2025-04-26 22:13:35,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:35,633 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:35,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7476
7537
8469
[WARNING|trainer.py:803] 2025-04-26 22:13:36,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:37,021 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:37,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7477
7538
8470
[WARNING|trainer.py:803] 2025-04-26 22:13:38,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:38,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:38,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7478
7539
8471
[WARNING|trainer.py:803] 2025-04-26 22:13:39,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:39,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7479
[WARNING|trainer.py:803] 2025-04-26 22:13:40,124 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7540
[WARNING|trainer.py:803] 2025-04-26 22:13:40,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8472
7480
[WARNING|trainer.py:803] 2025-04-26 22:13:41,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:41,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7541
[WARNING|trainer.py:803] 2025-04-26 22:13:42,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8473
7481
[WARNING|trainer.py:803] 2025-04-26 22:13:42,732 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:43,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:43,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7542
7482
8474
[WARNING|trainer.py:803] 2025-04-26 22:13:44,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:44,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:44,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7543
7483
8475
[WARNING|trainer.py:803] 2025-04-26 22:13:45,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:45,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:46,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7484
7544
8476
[WARNING|trainer.py:803] 2025-04-26 22:13:47,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:47,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7485
[WARNING|trainer.py:803] 2025-04-26 22:13:47,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7545
8477
[WARNING|trainer.py:803] 2025-04-26 22:13:48,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:48,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7486
[WARNING|trainer.py:803] 2025-04-26 22:13:49,041 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7546
[WARNING|trainer.py:803] 2025-04-26 22:13:49,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8478
[WARNING|trainer.py:803] 2025-04-26 22:13:49,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7487
[WARNING|trainer.py:803] 2025-04-26 22:13:50,498 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7547
[WARNING|trainer.py:803] 2025-04-26 22:13:50,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8479
7488
[WARNING|trainer.py:803] 2025-04-26 22:13:51,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7548
[WARNING|trainer.py:803] 2025-04-26 22:13:52,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:52,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7489
8480
[WARNING|trainer.py:803] 2025-04-26 22:13:52,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:53,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7549
[WARNING|trainer.py:803] 2025-04-26 22:13:53,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7490
[WARNING|trainer.py:803] 2025-04-26 22:13:54,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8481
[WARNING|trainer.py:803] 2025-04-26 22:13:54,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7550
7491
[WARNING|trainer.py:803] 2025-04-26 22:13:55,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:13:55,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8482
[WARNING|trainer.py:803] 2025-04-26 22:13:55,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7551
7492
[WARNING|trainer.py:803] 2025-04-26 22:13:56,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:56,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:13:57,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8483
7493
7552
[WARNING|trainer.py:803] 2025-04-26 22:13:57,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:58,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:58,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8484
7494
7553
[WARNING|trainer.py:803] 2025-04-26 22:13:59,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:13:59,605 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:13:59,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7495
8485
7554
[WARNING|trainer.py:803] 2025-04-26 22:14:00,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:00,887 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:01,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7496
8486
7555
[WARNING|trainer.py:803] 2025-04-26 22:14:02,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:02,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7497
[WARNING|trainer.py:803] 2025-04-26 22:14:02,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8487
7556
[WARNING|trainer.py:803] 2025-04-26 22:14:03,377 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7498
[WARNING|trainer.py:803] 2025-04-26 22:14:04,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:04,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8488
7557
[WARNING|trainer.py:803] 2025-04-26 22:14:04,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7499
[WARNING|trainer.py:803] 2025-04-26 22:14:05,402 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:05,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:05,860 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7558
8489
7500
[WARNING|trainer.py:803] 2025-04-26 22:14:06,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:06,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:07,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7559
8490
7501
[WARNING|trainer.py:803] 2025-04-26 22:14:08,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:08,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:08,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7560
8491
7502
[WARNING|trainer.py:803] 2025-04-26 22:14:09,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:09,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:10,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7561
8492
7503
[WARNING|trainer.py:803] 2025-04-26 22:14:10,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:11,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7562
[WARNING|trainer.py:803] 2025-04-26 22:14:11,534 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8493
7504
[WARNING|trainer.py:803] 2025-04-26 22:14:12,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:12,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7563
[WARNING|trainer.py:803] 2025-04-26 22:14:12,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8494
7505
[WARNING|trainer.py:803] 2025-04-26 22:14:13,642 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7564
[WARNING|trainer.py:803] 2025-04-26 22:14:14,206 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:14,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8495
7506
[WARNING|trainer.py:803] 2025-04-26 22:14:15,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7565
[WARNING|trainer.py:803] 2025-04-26 22:14:15,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:15,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7507
8496
[WARNING|trainer.py:803] 2025-04-26 22:14:16,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7566
[WARNING|trainer.py:803] 2025-04-26 22:14:17,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:17,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7508
8497
[WARNING|trainer.py:803] 2025-04-26 22:14:17,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7567
[WARNING|trainer.py:803] 2025-04-26 22:14:18,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:18,620 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7509
[WARNING|trainer.py:803] 2025-04-26 22:14:19,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8498
7568
[WARNING|trainer.py:803] 2025-04-26 22:14:19,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:20,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7510
[WARNING|trainer.py:803] 2025-04-26 22:14:20,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8499
7569
[WARNING|trainer.py:803] 2025-04-26 22:14:21,391 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:21,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7511
[WARNING|trainer.py:803] 2025-04-26 22:14:22,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8500
7570
[WARNING|trainer.py:803] 2025-04-26 22:14:22,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:23,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:23,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7512
8501
7571
[WARNING|trainer.py:803] 2025-04-26 22:14:24,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:24,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7513
[WARNING|trainer.py:803] 2025-04-26 22:14:24,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7572
8502
[WARNING|trainer.py:803] 2025-04-26 22:14:25,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:26,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7514
[WARNING|trainer.py:803] 2025-04-26 22:14:26,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7573
8503
[WARNING|trainer.py:803] 2025-04-26 22:14:27,107 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:27,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7515
[WARNING|trainer.py:803] 2025-04-26 22:14:27,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7574
[WARNING|trainer.py:803] 2025-04-26 22:14:28,513 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8504
[WARNING|trainer.py:803] 2025-04-26 22:14:28,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7516
[WARNING|trainer.py:803] 2025-04-26 22:14:29,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7575
[WARNING|trainer.py:803] 2025-04-26 22:14:29,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8505
[WARNING|trainer.py:803] 2025-04-26 22:14:30,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7517
7576
[WARNING|trainer.py:803] 2025-04-26 22:14:31,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:14:31,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:31,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8506
7518
7577
[WARNING|trainer.py:803] 2025-04-26 22:14:32,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:32,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:33,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7519
8507
7578
[WARNING|trainer.py:803] 2025-04-26 22:14:34,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:34,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:34,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7520
8508
7579
[WARNING|trainer.py:803] 2025-04-26 22:14:35,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:35,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:35,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7521
8509
7580
[WARNING|trainer.py:803] 2025-04-26 22:14:37,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:37,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:37,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7522
8510
7581
[WARNING|trainer.py:803] 2025-04-26 22:14:38,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:38,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:38,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7523
8511
7582
[WARNING|trainer.py:803] 2025-04-26 22:14:40,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:40,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:40,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7524
8512
7583
[WARNING|trainer.py:803] 2025-04-26 22:14:41,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:41,593 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:41,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7525
7584
8513
[WARNING|trainer.py:803] 2025-04-26 22:14:42,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:43,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:43,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7585
7526
8514
[WARNING|trainer.py:803] 2025-04-26 22:14:44,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:44,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:44,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7527
7586
8515
[WARNING|trainer.py:803] 2025-04-26 22:14:45,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:45,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:46,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7528
7587
8516
[WARNING|trainer.py:803] 2025-04-26 22:14:47,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:47,348 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:47,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7588
7529
8517
[WARNING|trainer.py:803] 2025-04-26 22:14:48,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:48,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7589
7530
[WARNING|trainer.py:803] 2025-04-26 22:14:49,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:50,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8518
[WARNING|trainer.py:803] 2025-04-26 22:14:50,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7590
7531
[WARNING|trainer.py:803] 2025-04-26 22:14:50,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:51,488 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:51,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8519
7591
7532
[WARNING|trainer.py:803] 2025-04-26 22:14:52,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:52,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:53,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8520
7592
7533
[WARNING|trainer.py:803] 2025-04-26 22:14:54,181 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:54,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:14:54,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7593
8521
7534
[WARNING|trainer.py:803] 2025-04-26 22:14:55,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:55,781 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:55,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7594
7535
8522
[WARNING|trainer.py:803] 2025-04-26 22:14:57,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:57,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:14:57,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7595
7536
8523
[WARNING|trainer.py:803] 2025-04-26 22:14:58,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:14:58,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:14:58,819 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7596
7537
8524
[WARNING|trainer.py:803] 2025-04-26 22:14:59,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:00,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:00,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7597
7538
8525
[WARNING|trainer.py:803] 2025-04-26 22:15:01,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:01,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7598
[WARNING|trainer.py:803] 2025-04-26 22:15:02,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7539
8526
[WARNING|trainer.py:803] 2025-04-26 22:15:02,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:03,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7599
[WARNING|trainer.py:803] 2025-04-26 22:15:03,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7540
[WARNING|trainer.py:803] 2025-04-26 22:15:04,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8527
[WARNING|trainer.py:803] 2025-04-26 22:15:04,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7600
7541
[WARNING|trainer.py:803] 2025-04-26 22:15:05,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:05,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8528
[WARNING|trainer.py:803] 2025-04-26 22:15:05,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7601
[WARNING|trainer.py:803] 2025-04-26 22:15:06,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7542
[WARNING|trainer.py:803] 2025-04-26 22:15:07,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8529
[WARNING|trainer.py:803] 2025-04-26 22:15:07,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7602
7543
[WARNING|trainer.py:803] 2025-04-26 22:15:08,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:08,486 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8530
[WARNING|trainer.py:803] 2025-04-26 22:15:08,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7603
7544
[WARNING|trainer.py:803] 2025-04-26 22:15:09,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:09,893 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8531
[WARNING|trainer.py:803] 2025-04-26 22:15:10,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7604
7545
[WARNING|trainer.py:803] 2025-04-26 22:15:11,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:11,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:11,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8532
7605
7546
[WARNING|trainer.py:803] 2025-04-26 22:15:12,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:12,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:13,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8533
7606
7547
[WARNING|trainer.py:803] 2025-04-26 22:15:14,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:14,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:14,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8534
7607
7548
[WARNING|trainer.py:803] 2025-04-26 22:15:15,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:15,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:15,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7608
8535
7549
[WARNING|trainer.py:803] 2025-04-26 22:15:17,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:17,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:17,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7609
7550
8536
[WARNING|trainer.py:803] 2025-04-26 22:15:18,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:18,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:18,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7610
7551
8537
[WARNING|trainer.py:803] 2025-04-26 22:15:20,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:20,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:20,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7611
7552
8538
[WARNING|trainer.py:803] 2025-04-26 22:15:21,519 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:21,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:21,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7612
7553
8539
[WARNING|trainer.py:803] 2025-04-26 22:15:22,891 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:23,111 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:23,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7613
7554
8540
[WARNING|trainer.py:803] 2025-04-26 22:15:24,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:24,496 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:24,929 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7614
7555
8541
[WARNING|trainer.py:803] 2025-04-26 22:15:26,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:26,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:26,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7615
7556
8542
[WARNING|trainer.py:803] 2025-04-26 22:15:27,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:27,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:28,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7557
7616
8543
[WARNING|trainer.py:803] 2025-04-26 22:15:29,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:29,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:29,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7558
7617
8544
[WARNING|trainer.py:803] 2025-04-26 22:15:30,356 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:30,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7559
[WARNING|trainer.py:803] 2025-04-26 22:15:31,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7618
[WARNING|trainer.py:803] 2025-04-26 22:15:31,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8545
[WARNING|trainer.py:803] 2025-04-26 22:15:32,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7560
[WARNING|trainer.py:803] 2025-04-26 22:15:32,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7619
[WARNING|trainer.py:803] 2025-04-26 22:15:33,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8546
[WARNING|trainer.py:803] 2025-04-26 22:15:33,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7561
[WARNING|trainer.py:803] 2025-04-26 22:15:33,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7620
[WARNING|trainer.py:803] 2025-04-26 22:15:34,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8547
[WARNING|trainer.py:803] 2025-04-26 22:15:34,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7562
7621
[WARNING|trainer.py:803] 2025-04-26 22:15:35,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:35,815 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8548
7563
[WARNING|trainer.py:803] 2025-04-26 22:15:36,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7622
[WARNING|trainer.py:803] 2025-04-26 22:15:37,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:37,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7564
8549
[WARNING|trainer.py:803] 2025-04-26 22:15:37,856 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7623
[WARNING|trainer.py:803] 2025-04-26 22:15:38,544 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:38,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7565
[WARNING|trainer.py:803] 2025-04-26 22:15:39,272 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8550
7624
[WARNING|trainer.py:803] 2025-04-26 22:15:39,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:40,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7566
8551
[WARNING|trainer.py:803] 2025-04-26 22:15:40,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:41,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7625
[WARNING|trainer.py:803] 2025-04-26 22:15:41,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7567
[WARNING|trainer.py:803] 2025-04-26 22:15:42,243 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8552
[WARNING|trainer.py:803] 2025-04-26 22:15:42,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7626
[WARNING|trainer.py:803] 2025-04-26 22:15:43,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7568
[WARNING|trainer.py:803] 2025-04-26 22:15:43,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8553
[WARNING|trainer.py:803] 2025-04-26 22:15:44,122 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7627
7569
[WARNING|trainer.py:803] 2025-04-26 22:15:44,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:45,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8554
[WARNING|trainer.py:803] 2025-04-26 22:15:45,481 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7628
7570
[WARNING|trainer.py:803] 2025-04-26 22:15:46,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:46,619 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:46,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8555
7629
7571
[WARNING|trainer.py:803] 2025-04-26 22:15:47,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:48,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:48,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8556
7630
7572
[WARNING|trainer.py:803] 2025-04-26 22:15:49,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:49,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:49,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8557
7631
7573
[WARNING|trainer.py:803] 2025-04-26 22:15:50,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:50,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:51,062 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7632
8558
7574
[WARNING|trainer.py:803] 2025-04-26 22:15:52,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:52,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:52,482 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8559
7633
7575
[WARNING|trainer.py:803] 2025-04-26 22:15:53,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:53,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:15:53,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8560
7634
7576
[WARNING|trainer.py:803] 2025-04-26 22:15:55,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:55,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:55,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7577
8561
7635
[WARNING|trainer.py:803] 2025-04-26 22:15:56,808 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:56,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:56,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7578
7636
8562
[WARNING|trainer.py:803] 2025-04-26 22:15:58,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:15:58,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:15:58,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7637
7579
8563
[WARNING|trainer.py:803] 2025-04-26 22:15:59,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:59,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:15:59,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7638
7580
8564
[WARNING|trainer.py:803] 2025-04-26 22:16:01,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:01,135 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:01,510 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7581
7639
8565
[WARNING|trainer.py:803] 2025-04-26 22:16:02,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:02,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:03,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7582
7640
8566
[WARNING|trainer.py:803] 2025-04-26 22:16:03,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:04,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7583
7641
[WARNING|trainer.py:803] 2025-04-26 22:16:04,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8567
[WARNING|trainer.py:803] 2025-04-26 22:16:05,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:05,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7584
[WARNING|trainer.py:803] 2025-04-26 22:16:06,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7642
8568
[WARNING|trainer.py:803] 2025-04-26 22:16:06,789 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:06,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7585
7643
[WARNING|trainer.py:803] 2025-04-26 22:16:07,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8569
[WARNING|trainer.py:803] 2025-04-26 22:16:08,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:08,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7644
7586
[WARNING|trainer.py:803] 2025-04-26 22:16:09,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8570
[WARNING|trainer.py:803] 2025-04-26 22:16:09,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:09,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7587
7645
[WARNING|trainer.py:803] 2025-04-26 22:16:10,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8571
[WARNING|trainer.py:803] 2025-04-26 22:16:11,188 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:11,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7588
7646
[WARNING|trainer.py:803] 2025-04-26 22:16:12,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:12,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:12,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8572
7589
7647
[WARNING|trainer.py:803] 2025-04-26 22:16:13,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:13,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:14,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8573
7590
7648
[WARNING|trainer.py:803] 2025-04-26 22:16:14,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:15,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:15,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8574
7591
7649
[WARNING|trainer.py:803] 2025-04-26 22:16:16,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:16,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:16,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8575
7592
7650
[WARNING|trainer.py:803] 2025-04-26 22:16:17,852 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:18,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:18,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8576
7593
7651
[WARNING|trainer.py:803] 2025-04-26 22:16:19,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:19,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:19,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8577
7594
7652
[WARNING|trainer.py:803] 2025-04-26 22:16:20,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:21,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:21,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8578
7595
7653
[WARNING|trainer.py:803] 2025-04-26 22:16:22,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:22,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:22,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8579
7596
7654
[WARNING|trainer.py:803] 2025-04-26 22:16:23,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:23,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:24,363 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7597
8580
7655
[WARNING|trainer.py:803] 2025-04-26 22:16:25,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:25,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:25,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7598
8581
7656
[WARNING|trainer.py:803] 2025-04-26 22:16:26,681 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:27,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:27,254 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7599
8582
7657
[WARNING|trainer.py:803] 2025-04-26 22:16:28,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:28,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:28,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7600
8583
7658
[WARNING|trainer.py:803] 2025-04-26 22:16:29,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7601
[WARNING|trainer.py:803] 2025-04-26 22:16:30,240 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:30,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7659
8584
[WARNING|trainer.py:803] 2025-04-26 22:16:31,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7602
[WARNING|trainer.py:803] 2025-04-26 22:16:31,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:31,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:32,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7660
8585
7603
[WARNING|trainer.py:803] 2025-04-26 22:16:33,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:33,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:33,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7661
8586
7604
[WARNING|trainer.py:803] 2025-04-26 22:16:34,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:34,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:35,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7662
8587
7605
[WARNING|trainer.py:803] 2025-04-26 22:16:36,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:36,352 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:36,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7663
8588
7606
[WARNING|trainer.py:803] 2025-04-26 22:16:37,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:37,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:38,253 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7664
8589
7607
[WARNING|trainer.py:803] 2025-04-26 22:16:39,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:39,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:39,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7665
8590
7608
[WARNING|trainer.py:803] 2025-04-26 22:16:40,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:40,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:41,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7666
8591
7609
[WARNING|trainer.py:803] 2025-04-26 22:16:42,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:42,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:16:42,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7667
7610
8592
[WARNING|trainer.py:803] 2025-04-26 22:16:43,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:43,914 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:43,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7668
7611
8593
[WARNING|trainer.py:803] 2025-04-26 22:16:45,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:45,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:45,454 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7612
7669
8594
[WARNING|trainer.py:803] 2025-04-26 22:16:46,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:46,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:16:47,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7613
7670
8595
[WARNING|trainer.py:803] 2025-04-26 22:16:48,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:48,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:48,545 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7614
7671
8596
[WARNING|trainer.py:803] 2025-04-26 22:16:49,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:49,778 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:50,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7615
7672
8597
[WARNING|trainer.py:803] 2025-04-26 22:16:51,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:51,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7616
[WARNING|trainer.py:803] 2025-04-26 22:16:51,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7673
8598
[WARNING|trainer.py:803] 2025-04-26 22:16:52,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:52,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:16:53,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7674
7617
8599
[WARNING|trainer.py:803] 2025-04-26 22:16:54,068 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:54,137 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:54,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7675
7618
8600
[WARNING|trainer.py:803] 2025-04-26 22:16:55,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:16:55,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7676
[WARNING|trainer.py:803] 2025-04-26 22:16:56,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7619
8601
[WARNING|trainer.py:803] 2025-04-26 22:16:56,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:57,121 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7677
[WARNING|trainer.py:803] 2025-04-26 22:16:57,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7620
8602
[WARNING|trainer.py:803] 2025-04-26 22:16:58,447 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:16:58,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7678
[WARNING|trainer.py:803] 2025-04-26 22:16:59,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7621
8603
[WARNING|trainer.py:803] 2025-04-26 22:16:59,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:00,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:00,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7679
7622
8604
[WARNING|trainer.py:803] 2025-04-26 22:17:01,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:01,543 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:01,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7680
7623
8605
[WARNING|trainer.py:803] 2025-04-26 22:17:02,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:02,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:03,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7681
7624
8606
[WARNING|trainer.py:803] 2025-04-26 22:17:04,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:04,428 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:04,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7682
7625
8607
[WARNING|trainer.py:803] 2025-04-26 22:17:05,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:05,940 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:06,221 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7626
7683
8608
[WARNING|trainer.py:803] 2025-04-26 22:17:07,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:07,381 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:07,749 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7684
7627
8609
[WARNING|trainer.py:803] 2025-04-26 22:17:08,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:08,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:09,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7628
7685
8610
[WARNING|trainer.py:803] 2025-04-26 22:17:10,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:10,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:10,589 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7629
7686
8611
[WARNING|trainer.py:803] 2025-04-26 22:17:11,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:11,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:12,140 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7630
7687
8612
[WARNING|trainer.py:803] 2025-04-26 22:17:13,219 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:13,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:13,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7631
7688
8613
[WARNING|trainer.py:803] 2025-04-26 22:17:14,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:14,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:15,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7632
7689
8614
[WARNING|trainer.py:803] 2025-04-26 22:17:16,048 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:16,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7633
7690
[WARNING|trainer.py:803] 2025-04-26 22:17:16,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:17,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8615
[WARNING|trainer.py:803] 2025-04-26 22:17:17,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7634
7691
[WARNING|trainer.py:803] 2025-04-26 22:17:18,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8616
[WARNING|trainer.py:803] 2025-04-26 22:17:18,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:19,149 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7635
7692
[WARNING|trainer.py:803] 2025-04-26 22:17:19,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8617
[WARNING|trainer.py:803] 2025-04-26 22:17:20,459 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:20,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7636
[WARNING|trainer.py:803] 2025-04-26 22:17:21,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7693
8618
[WARNING|trainer.py:803] 2025-04-26 22:17:21,918 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:22,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7637
[WARNING|trainer.py:803] 2025-04-26 22:17:22,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7694
[WARNING|trainer.py:803] 2025-04-26 22:17:23,323 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8619
[WARNING|trainer.py:803] 2025-04-26 22:17:23,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7638
[WARNING|trainer.py:803] 2025-04-26 22:17:24,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7695
[WARNING|trainer.py:803] 2025-04-26 22:17:24,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8620
[WARNING|trainer.py:803] 2025-04-26 22:17:25,215 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7639
[WARNING|trainer.py:803] 2025-04-26 22:17:25,621 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7696
[WARNING|trainer.py:803] 2025-04-26 22:17:26,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8621
[WARNING|trainer.py:803] 2025-04-26 22:17:26,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7640
[WARNING|trainer.py:803] 2025-04-26 22:17:27,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7697
[WARNING|trainer.py:803] 2025-04-26 22:17:27,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8622
7641
[WARNING|trainer.py:803] 2025-04-26 22:17:28,298 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:28,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7698
[WARNING|trainer.py:803] 2025-04-26 22:17:29,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8623
7642
[WARNING|trainer.py:803] 2025-04-26 22:17:29,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:30,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7699
[WARNING|trainer.py:803] 2025-04-26 22:17:30,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8624
7643
[WARNING|trainer.py:803] 2025-04-26 22:17:31,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:31,484 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7700
[WARNING|trainer.py:803] 2025-04-26 22:17:31,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8625
7644
[WARNING|trainer.py:803] 2025-04-26 22:17:32,609 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:32,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7701
[WARNING|trainer.py:803] 2025-04-26 22:17:33,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8626
7645
[WARNING|trainer.py:803] 2025-04-26 22:17:34,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:34,527 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7702
[WARNING|trainer.py:803] 2025-04-26 22:17:34,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8627
7646
[WARNING|trainer.py:803] 2025-04-26 22:17:35,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:35,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7703
[WARNING|trainer.py:803] 2025-04-26 22:17:36,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8628
7647
[WARNING|trainer.py:803] 2025-04-26 22:17:37,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:37,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7704
[WARNING|trainer.py:803] 2025-04-26 22:17:37,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8629
7648
[WARNING|trainer.py:803] 2025-04-26 22:17:38,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:38,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7705
8630
[WARNING|trainer.py:803] 2025-04-26 22:17:39,265 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7649
[WARNING|trainer.py:803] 2025-04-26 22:17:40,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:40,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8631
7706
[WARNING|trainer.py:803] 2025-04-26 22:17:40,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7650
[WARNING|trainer.py:803] 2025-04-26 22:17:41,405 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:41,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8632
7707
[WARNING|trainer.py:803] 2025-04-26 22:17:42,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7651
[WARNING|trainer.py:803] 2025-04-26 22:17:42,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:42,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8633
7708
[WARNING|trainer.py:803] 2025-04-26 22:17:43,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:44,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7652
[WARNING|trainer.py:803] 2025-04-26 22:17:44,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8634
7709
[WARNING|trainer.py:803] 2025-04-26 22:17:45,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:45,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7653
[WARNING|trainer.py:803] 2025-04-26 22:17:45,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8635
7710
[WARNING|trainer.py:803] 2025-04-26 22:17:46,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:47,002 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:47,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7654
8636
7711
[WARNING|trainer.py:803] 2025-04-26 22:17:48,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:48,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:48,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7655
8637
7712
[WARNING|trainer.py:803] 2025-04-26 22:17:49,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:49,846 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:50,232 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7656
8638
7713
[WARNING|trainer.py:803] 2025-04-26 22:17:51,201 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:17:51,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:51,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7657
8639
7714
[WARNING|trainer.py:803] 2025-04-26 22:17:52,607 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:52,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:53,037 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8640
7658
7715
[WARNING|trainer.py:803] 2025-04-26 22:17:54,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:54,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:17:54,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8641
7659
7716
[WARNING|trainer.py:803] 2025-04-26 22:17:55,584 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:55,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:55,769 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8642
7717
7660
[WARNING|trainer.py:803] 2025-04-26 22:17:57,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:57,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:57,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7718
8643
7661
[WARNING|trainer.py:803] 2025-04-26 22:17:58,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:17:58,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:17:58,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7719
8644
7662
[WARNING|trainer.py:803] 2025-04-26 22:17:59,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:00,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:00,241 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7720
8645
7663
[WARNING|trainer.py:803] 2025-04-26 22:18:01,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:01,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:01,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7721
8646
7664
[WARNING|trainer.py:803] 2025-04-26 22:18:02,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:03,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:03,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7722
8647
7665
[WARNING|trainer.py:803] 2025-04-26 22:18:04,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:04,440 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:04,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7723
8648
7666
[WARNING|trainer.py:803] 2025-04-26 22:18:05,632 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:05,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7724
[WARNING|trainer.py:803] 2025-04-26 22:18:06,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8649
7667
[WARNING|trainer.py:803] 2025-04-26 22:18:06,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:07,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7725
[WARNING|trainer.py:803] 2025-04-26 22:18:07,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8650
[WARNING|trainer.py:803] 2025-04-26 22:18:08,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7668
[WARNING|trainer.py:803] 2025-04-26 22:18:08,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7726
8651
[WARNING|trainer.py:803] 2025-04-26 22:18:09,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:09,786 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:10,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7669
7727
8652
[WARNING|trainer.py:803] 2025-04-26 22:18:10,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:11,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7670
[WARNING|trainer.py:803] 2025-04-26 22:18:11,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7728
8653
[WARNING|trainer.py:803] 2025-04-26 22:18:12,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:12,570 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7671
[WARNING|trainer.py:803] 2025-04-26 22:18:13,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7729
8654
[WARNING|trainer.py:803] 2025-04-26 22:18:13,844 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:13,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7672
[WARNING|trainer.py:803] 2025-04-26 22:18:14,453 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7730
8655
[WARNING|trainer.py:803] 2025-04-26 22:18:15,276 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:15,344 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7673
7731
[WARNING|trainer.py:803] 2025-04-26 22:18:15,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8656
[WARNING|trainer.py:803] 2025-04-26 22:18:16,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:16,745 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7674
[WARNING|trainer.py:803] 2025-04-26 22:18:17,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7732
8657
[WARNING|trainer.py:803] 2025-04-26 22:18:18,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:18,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:18,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7733
7675
8658
[WARNING|trainer.py:803] 2025-04-26 22:18:19,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:19,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7734
[WARNING|trainer.py:803] 2025-04-26 22:18:20,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7676
8659
[WARNING|trainer.py:803] 2025-04-26 22:18:20,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:21,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7735
[WARNING|trainer.py:803] 2025-04-26 22:18:21,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7677
8660
[WARNING|trainer.py:803] 2025-04-26 22:18:22,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:22,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:22,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7736
7678
8661
[WARNING|trainer.py:803] 2025-04-26 22:18:23,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:24,091 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:24,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7737
7679
8662
[WARNING|trainer.py:803] 2025-04-26 22:18:25,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:25,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7738
[WARNING|trainer.py:803] 2025-04-26 22:18:25,807 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7680
8663
[WARNING|trainer.py:803] 2025-04-26 22:18:26,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:27,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7739
[WARNING|trainer.py:803] 2025-04-26 22:18:27,220 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7681
8664
[WARNING|trainer.py:803] 2025-04-26 22:18:27,916 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7740
[WARNING|trainer.py:803] 2025-04-26 22:18:28,562 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:28,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7682
[WARNING|trainer.py:803] 2025-04-26 22:18:29,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8665
7741
[WARNING|trainer.py:803] 2025-04-26 22:18:30,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:30,163 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7683
[WARNING|trainer.py:803] 2025-04-26 22:18:30,665 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8666
7742
[WARNING|trainer.py:803] 2025-04-26 22:18:31,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:31,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:32,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7684
8667
7743
[WARNING|trainer.py:803] 2025-04-26 22:18:33,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:33,132 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:33,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7685
8668
7744
[WARNING|trainer.py:803] 2025-04-26 22:18:34,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:34,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:34,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7686
8669
7745
[WARNING|trainer.py:803] 2025-04-26 22:18:35,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:36,120 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:36,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7687
8670
7746
[WARNING|trainer.py:803] 2025-04-26 22:18:37,435 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:37,653 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:37,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7688
8671
7747
[WARNING|trainer.py:803] 2025-04-26 22:18:38,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:39,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:39,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7689
7748
8672
[WARNING|trainer.py:803] 2025-04-26 22:18:40,397 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:40,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:40,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7690
7749
8673
[WARNING|trainer.py:803] 2025-04-26 22:18:41,907 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:41,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:42,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7691
7750
8674
[WARNING|trainer.py:803] 2025-04-26 22:18:43,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:43,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:43,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7751
7692
8675
[WARNING|trainer.py:803] 2025-04-26 22:18:44,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:44,896 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:45,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7752
7693
8676
[WARNING|trainer.py:803] 2025-04-26 22:18:46,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:46,394 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:46,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7753
8677
7694
[WARNING|trainer.py:803] 2025-04-26 22:18:47,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:47,963 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:48,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7754
8678
7695
[WARNING|trainer.py:803] 2025-04-26 22:18:48,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:49,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:49,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7755
8679
7696
[WARNING|trainer.py:803] 2025-04-26 22:18:50,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:18:50,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7756
[WARNING|trainer.py:803] 2025-04-26 22:18:51,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8680
7697
[WARNING|trainer.py:803] 2025-04-26 22:18:51,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:18:52,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7757
[WARNING|trainer.py:803] 2025-04-26 22:18:52,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8681
[WARNING|trainer.py:803] 2025-04-26 22:18:53,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7698
[WARNING|trainer.py:803] 2025-04-26 22:18:53,728 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7758
[WARNING|trainer.py:803] 2025-04-26 22:18:54,086 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8682
[WARNING|trainer.py:803] 2025-04-26 22:18:54,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7699
7759
[WARNING|trainer.py:803] 2025-04-26 22:18:55,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:55,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8683
[WARNING|trainer.py:803] 2025-04-26 22:18:56,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7700
7760
[WARNING|trainer.py:803] 2025-04-26 22:18:56,729 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:18:57,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8684
[WARNING|trainer.py:803] 2025-04-26 22:18:57,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7701
7761
[WARNING|trainer.py:803] 2025-04-26 22:18:58,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:58,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:18:58,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8685
7702
7762
[WARNING|trainer.py:803] 2025-04-26 22:18:59,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:00,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:00,222 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8686
7703
7763
[WARNING|trainer.py:803] 2025-04-26 22:19:01,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:01,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:01,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8687
7764
7704
[WARNING|trainer.py:803] 2025-04-26 22:19:02,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:03,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:03,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8688
7765
7705
[WARNING|trainer.py:803] 2025-04-26 22:19:04,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:04,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:04,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8689
7766
7706
[WARNING|trainer.py:803] 2025-04-26 22:19:05,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:05,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:05,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8690
7767
7707
[WARNING|trainer.py:803] 2025-04-26 22:19:07,180 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:07,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:07,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8691
7768
7708
[WARNING|trainer.py:803] 2025-04-26 22:19:08,608 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:08,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:08,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7769
8692
7709
[WARNING|trainer.py:803] 2025-04-26 22:19:10,039 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:10,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:10,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7770
8693
7710
[WARNING|trainer.py:803] 2025-04-26 22:19:11,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:11,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:11,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7771
8694
7711
[WARNING|trainer.py:803] 2025-04-26 22:19:12,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:13,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:13,278 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7772
7712
8695
[WARNING|trainer.py:803] 2025-04-26 22:19:14,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:14,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:14,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7773
7713
8696
[WARNING|trainer.py:803] 2025-04-26 22:19:15,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:16,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:16,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7774
7714
8697
[WARNING|trainer.py:803] 2025-04-26 22:19:17,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:17,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:17,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7775
7715
8698
[WARNING|trainer.py:803] 2025-04-26 22:19:18,465 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:19,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
7776
[WARNING|trainer.py:803] 2025-04-26 22:19:19,038 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8699
7716
[WARNING|trainer.py:803] 2025-04-26 22:19:19,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7777
[WARNING|trainer.py:803] 2025-04-26 22:19:20,418 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:20,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7717
8700
[WARNING|trainer.py:803] 2025-04-26 22:19:21,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7778
[WARNING|trainer.py:803] 2025-04-26 22:19:21,869 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:22,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8701
[WARNING|trainer.py:803] 2025-04-26 22:19:22,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7718
[WARNING|trainer.py:803] 2025-04-26 22:19:22,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7779
8702
[WARNING|trainer.py:803] 2025-04-26 22:19:23,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:23,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:24,050 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7719
8703
7780
[WARNING|trainer.py:803] 2025-04-26 22:19:24,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:25,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:25,343 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8704
7720
7781
[WARNING|trainer.py:803] 2025-04-26 22:19:26,275 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:26,387 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8705
[WARNING|trainer.py:803] 2025-04-26 22:19:26,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7721
[WARNING|trainer.py:803] 2025-04-26 22:19:27,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7782
8706
[WARNING|trainer.py:803] 2025-04-26 22:19:27,838 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:28,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:28,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7722
8707
7783
[WARNING|trainer.py:803] 2025-04-26 22:19:29,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:29,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:29,616 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8708
7723
7784
[WARNING|trainer.py:803] 2025-04-26 22:19:30,693 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:30,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8709
[WARNING|trainer.py:803] 2025-04-26 22:19:31,029 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7724
7785
[WARNING|trainer.py:803] 2025-04-26 22:19:31,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8710
[WARNING|trainer.py:803] 2025-04-26 22:19:32,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:32,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:32,799 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7725
8711
7786
[WARNING|trainer.py:803] 2025-04-26 22:19:33,688 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:33,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:33,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8712
7726
7787
[WARNING|trainer.py:803] 2025-04-26 22:19:35,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:35,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8713
[WARNING|trainer.py:803] 2025-04-26 22:19:35,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7727
7788
[WARNING|trainer.py:803] 2025-04-26 22:19:36,127 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8714
[WARNING|trainer.py:803] 2025-04-26 22:19:36,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:36,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7728
[WARNING|trainer.py:803] 2025-04-26 22:19:37,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7789
8715
[WARNING|trainer.py:803] 2025-04-26 22:19:37,898 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:38,141 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:38,379 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7729
8716
7790
[WARNING|trainer.py:803] 2025-04-26 22:19:39,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:39,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:39,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8717
7730
7791
[WARNING|trainer.py:803] 2025-04-26 22:19:40,602 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:40,610 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8718
[WARNING|trainer.py:803] 2025-04-26 22:19:41,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7731
7792
[WARNING|trainer.py:803] 2025-04-26 22:19:41,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8719
[WARNING|trainer.py:803] 2025-04-26 22:19:42,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:42,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7732
[WARNING|trainer.py:803] 2025-04-26 22:19:42,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7793
8720
[WARNING|trainer.py:803] 2025-04-26 22:19:43,407 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:43,850 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:43,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7733
8721
7794
[WARNING|trainer.py:803] 2025-04-26 22:19:44,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:45,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:45,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8722
7734
7795
[WARNING|trainer.py:803] 2025-04-26 22:19:46,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:46,194 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8723
[WARNING|trainer.py:803] 2025-04-26 22:19:46,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7735
7796
[WARNING|trainer.py:803] 2025-04-26 22:19:47,347 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:47,597 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8724
[WARNING|trainer.py:803] 2025-04-26 22:19:48,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7736
[WARNING|trainer.py:803] 2025-04-26 22:19:48,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7797
8725
[WARNING|trainer.py:803] 2025-04-26 22:19:48,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:49,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:49,601 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7737
8726
7798
[WARNING|trainer.py:803] 2025-04-26 22:19:50,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:50,740 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:19:50,990 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8727
7738
7799
[WARNING|trainer.py:803] 2025-04-26 22:19:51,830 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:51,881 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8728
[WARNING|trainer.py:803] 2025-04-26 22:19:52,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7739
[WARNING|trainer.py:803] 2025-04-26 22:19:52,938 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7800
8729
[WARNING|trainer.py:803] 2025-04-26 22:19:53,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:53,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7740
[WARNING|trainer.py:803] 2025-04-26 22:19:54,003 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8730
[WARNING|trainer.py:803] 2025-04-26 22:19:54,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7801
[WARNING|trainer.py:803] 2025-04-26 22:19:55,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7741
8731
[WARNING|trainer.py:803] 2025-04-26 22:19:55,598 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:19:56,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:56,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8732
7802
7742
[WARNING|trainer.py:803] 2025-04-26 22:19:57,376 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:19:57,411 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:57,531 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8733
7743
[WARNING|trainer.py:803] 2025-04-26 22:19:58,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7803
8734
[WARNING|trainer.py:803] 2025-04-26 22:19:58,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:19:59,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7744
[WARNING|trainer.py:803] 2025-04-26 22:19:59,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8735
7804
[WARNING|trainer.py:803] 2025-04-26 22:20:00,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:00,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7745
8736
[WARNING|trainer.py:803] 2025-04-26 22:20:01,134 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:01,716 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:01,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7805
8737
7746
[WARNING|trainer.py:803] 2025-04-26 22:20:02,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:02,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:03,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8738
7747
7806
[WARNING|trainer.py:803] 2025-04-26 22:20:04,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8739
[WARNING|trainer.py:803] 2025-04-26 22:20:04,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:04,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7748
[WARNING|trainer.py:803] 2025-04-26 22:20:05,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8740
7807
[WARNING|trainer.py:803] 2025-04-26 22:20:05,985 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:06,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:06,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8741
7749
7808
[WARNING|trainer.py:803] 2025-04-26 22:20:07,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:07,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8742
7750
[WARNING|trainer.py:803] 2025-04-26 22:20:08,152 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:08,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8743
[WARNING|trainer.py:803] 2025-04-26 22:20:08,864 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7809
7751
[WARNING|trainer.py:803] 2025-04-26 22:20:09,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8744
[WARNING|trainer.py:803] 2025-04-26 22:20:10,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:10,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:10,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7752
8745
7810
[WARNING|trainer.py:803] 2025-04-26 22:20:11,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:11,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:11,868 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8746
7753
7811
[WARNING|trainer.py:803] 2025-04-26 22:20:12,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:13,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8747
[WARNING|trainer.py:803] 2025-04-26 22:20:13,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7754
[WARNING|trainer.py:803] 2025-04-26 22:20:14,139 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8748
7812
[WARNING|trainer.py:803] 2025-04-26 22:20:14,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7755
[WARNING|trainer.py:803] 2025-04-26 22:20:15,274 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:15,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8749
[WARNING|trainer.py:803] 2025-04-26 22:20:16,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7813
[WARNING|trainer.py:803] 2025-04-26 22:20:16,386 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7756
8750
[WARNING|trainer.py:803] 2025-04-26 22:20:17,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:17,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:17,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8751
7814
7757
[WARNING|trainer.py:803] 2025-04-26 22:20:18,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:18,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:18,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8752
7758
7815
[WARNING|trainer.py:803] 2025-04-26 22:20:19,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8753
[WARNING|trainer.py:803] 2025-04-26 22:20:20,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:20,564 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:20,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7759
8754
7816
[WARNING|trainer.py:803] 2025-04-26 22:20:21,768 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:22,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8755
[WARNING|trainer.py:803] 2025-04-26 22:20:22,342 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7760
[WARNING|trainer.py:803] 2025-04-26 22:20:23,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:23,182 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7817
8756
7761
[WARNING|trainer.py:803] 2025-04-26 22:20:24,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:24,218 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8757
[WARNING|trainer.py:803] 2025-04-26 22:20:24,558 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7818
7762
[WARNING|trainer.py:803] 2025-04-26 22:20:25,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8758
[WARNING|trainer.py:803] 2025-04-26 22:20:25,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:25,968 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:26,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7763
8759
7819
[WARNING|trainer.py:803] 2025-04-26 22:20:27,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:27,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:27,557 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8760
7764
7820
[WARNING|trainer.py:803] 2025-04-26 22:20:28,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:28,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8761
7765
[WARNING|trainer.py:803] 2025-04-26 22:20:29,366 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:29,765 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8762
[WARNING|trainer.py:803] 2025-04-26 22:20:30,146 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7821
7766
[WARNING|trainer.py:803] 2025-04-26 22:20:30,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:31,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8763
[WARNING|trainer.py:803] 2025-04-26 22:20:31,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:31,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7767
7822
8764
[WARNING|trainer.py:803] 2025-04-26 22:20:32,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:32,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:33,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8765
7768
7823
[WARNING|trainer.py:803] 2025-04-26 22:20:34,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:34,270 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8766
7769
[WARNING|trainer.py:803] 2025-04-26 22:20:34,992 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:35,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8767
[WARNING|trainer.py:803] 2025-04-26 22:20:35,721 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7824
7770
[WARNING|trainer.py:803] 2025-04-26 22:20:36,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8768
[WARNING|trainer.py:803] 2025-04-26 22:20:36,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:37,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:37,541 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7825
7771
8769
[WARNING|trainer.py:803] 2025-04-26 22:20:38,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:38,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:38,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8770
7772
7826
[WARNING|trainer.py:803] 2025-04-26 22:20:39,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:39,982 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8771
[WARNING|trainer.py:803] 2025-04-26 22:20:40,022 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7773
[WARNING|trainer.py:803] 2025-04-26 22:20:40,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7827
8772
[WARNING|trainer.py:803] 2025-04-26 22:20:41,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:41,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7774
[WARNING|trainer.py:803] 2025-04-26 22:20:41,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8773
[WARNING|trainer.py:803] 2025-04-26 22:20:42,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:43,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7828
8774
7775
[WARNING|trainer.py:803] 2025-04-26 22:20:43,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:44,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:44,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8775
7776
7829
[WARNING|trainer.py:803] 2025-04-26 22:20:45,277 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8776
[WARNING|trainer.py:803] 2025-04-26 22:20:45,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:45,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7777
[WARNING|trainer.py:803] 2025-04-26 22:20:46,396 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7830
8777
[WARNING|trainer.py:803] 2025-04-26 22:20:47,030 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:47,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:47,523 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7778
8778
[WARNING|trainer.py:803] 2025-04-26 22:20:48,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:48,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7831
8779
7779
[WARNING|trainer.py:803] 2025-04-26 22:20:49,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:49,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:49,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8780
7780
7832
[WARNING|trainer.py:803] 2025-04-26 22:20:50,993 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8781
[WARNING|trainer.py:803] 2025-04-26 22:20:51,216 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:51,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7781
[WARNING|trainer.py:803] 2025-04-26 22:20:52,056 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7833
8782
[WARNING|trainer.py:803] 2025-04-26 22:20:52,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:20:52,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:53,150 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7782
8783
7834
[WARNING|trainer.py:803] 2025-04-26 22:20:54,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:54,349 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:20:54,645 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8784
7783
[WARNING|trainer.py:803] 2025-04-26 22:20:55,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:55,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7835
8785
7784
[WARNING|trainer.py:803] 2025-04-26 22:20:56,374 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:56,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:56,917 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8786
7836
7785
[WARNING|trainer.py:803] 2025-04-26 22:20:57,773 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:20:58,026 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8787
[WARNING|trainer.py:803] 2025-04-26 22:20:58,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7837
[WARNING|trainer.py:803] 2025-04-26 22:20:58,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7786
8788
[WARNING|trainer.py:803] 2025-04-26 22:20:59,606 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:59,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:20:59,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8789
7787
7838
[WARNING|trainer.py:803] 2025-04-26 22:21:01,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:01,199 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8790
[WARNING|trainer.py:803] 2025-04-26 22:21:01,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7788
[WARNING|trainer.py:803] 2025-04-26 22:21:02,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7839
8791
[WARNING|trainer.py:803] 2025-04-26 22:21:02,592 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7789
[WARNING|trainer.py:803] 2025-04-26 22:21:03,286 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:03,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8792
[WARNING|trainer.py:803] 2025-04-26 22:21:03,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7840
[WARNING|trainer.py:803] 2025-04-26 22:21:04,529 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7790
8793
[WARNING|trainer.py:803] 2025-04-26 22:21:04,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:05,475 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:05,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7841
8794
7791
[WARNING|trainer.py:803] 2025-04-26 22:21:06,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:06,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:06,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8795
7792
[WARNING|trainer.py:803] 2025-04-26 22:21:07,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7842
8796
[WARNING|trainer.py:803] 2025-04-26 22:21:08,365 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:08,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7793
[WARNING|trainer.py:803] 2025-04-26 22:21:08,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8797
[WARNING|trainer.py:803] 2025-04-26 22:21:09,760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7843
[WARNING|trainer.py:803] 2025-04-26 22:21:10,102 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7794
8798
[WARNING|trainer.py:803] 2025-04-26 22:21:10,588 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:11,202 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:11,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8799
7795
7844
[WARNING|trainer.py:803] 2025-04-26 22:21:12,368 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:12,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8800
[WARNING|trainer.py:803] 2025-04-26 22:21:12,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7796
[WARNING|trainer.py:803] 2025-04-26 22:21:13,506 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7845
8801
[WARNING|trainer.py:803] 2025-04-26 22:21:14,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:14,469 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:14,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7797
8802
[WARNING|trainer.py:803] 2025-04-26 22:21:15,540 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7846
[WARNING|trainer.py:803] 2025-04-26 22:21:15,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8803
7798
[WARNING|trainer.py:803] 2025-04-26 22:21:16,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:16,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:16,977 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8804
7799
7847
[WARNING|trainer.py:803] 2025-04-26 22:21:18,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8805
[WARNING|trainer.py:803] 2025-04-26 22:21:18,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:18,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7800
[WARNING|trainer.py:803] 2025-04-26 22:21:19,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7848
8806
[WARNING|trainer.py:803] 2025-04-26 22:21:19,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:20,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:20,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8807
7801
7849
[WARNING|trainer.py:803] 2025-04-26 22:21:21,538 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:21,636 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8808
[WARNING|trainer.py:803] 2025-04-26 22:21:22,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7802
[WARNING|trainer.py:803] 2025-04-26 22:21:22,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8809
7850
[WARNING|trainer.py:803] 2025-04-26 22:21:23,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:23,812 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8810
[WARNING|trainer.py:803] 2025-04-26 22:21:24,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7803
[WARNING|trainer.py:803] 2025-04-26 22:21:24,932 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8811
7851
[WARNING|trainer.py:803] 2025-04-26 22:21:25,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:26,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:26,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7804
8812
[WARNING|trainer.py:803] 2025-04-26 22:21:27,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:27,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8813
7852
7805
[WARNING|trainer.py:803] 2025-04-26 22:21:28,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:28,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8814
[WARNING|trainer.py:803] 2025-04-26 22:21:28,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7853
[WARNING|trainer.py:803] 2025-04-26 22:21:29,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8815
7806
[WARNING|trainer.py:803] 2025-04-26 22:21:30,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:30,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:30,680 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8816
7854
7807
[WARNING|trainer.py:803] 2025-04-26 22:21:31,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:31,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8817
[WARNING|trainer.py:803] 2025-04-26 22:21:32,392 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7855
[WARNING|trainer.py:803] 2025-04-26 22:21:32,858 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8818
7808
[WARNING|trainer.py:803] 2025-04-26 22:21:33,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:34,099 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:34,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8819
7856
7809
[WARNING|trainer.py:803] 2025-04-26 22:21:35,208 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:35,287 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8820
[WARNING|trainer.py:803] 2025-04-26 22:21:36,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7857
[WARNING|trainer.py:803] 2025-04-26 22:21:36,375 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8821
7810
[WARNING|trainer.py:803] 2025-04-26 22:21:36,964 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:37,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7858
[WARNING|trainer.py:803] 2025-04-26 22:21:37,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8822
[WARNING|trainer.py:803] 2025-04-26 22:21:38,518 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7811
[WARNING|trainer.py:803] 2025-04-26 22:21:38,692 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8823
[WARNING|trainer.py:803] 2025-04-26 22:21:39,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7859
[WARNING|trainer.py:803] 2025-04-26 22:21:39,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8824
[WARNING|trainer.py:803] 2025-04-26 22:21:40,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7812
[WARNING|trainer.py:803] 2025-04-26 22:21:40,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7860
8825
[WARNING|trainer.py:803] 2025-04-26 22:21:41,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:41,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:42,019 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7813
8826
7861
[WARNING|trainer.py:803] 2025-04-26 22:21:42,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:43,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:43,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8827
7814
7862
[WARNING|trainer.py:803] 2025-04-26 22:21:44,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8828
[WARNING|trainer.py:803] 2025-04-26 22:21:44,766 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:45,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:45,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7815
8829
7863
[WARNING|trainer.py:803] 2025-04-26 22:21:46,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:46,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:46,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8830
7816
7864
[WARNING|trainer.py:803] 2025-04-26 22:21:47,731 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8831
[WARNING|trainer.py:803] 2025-04-26 22:21:48,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:48,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:48,811 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8832
7817
7865
[WARNING|trainer.py:803] 2025-04-26 22:21:49,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:50,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8833
[WARNING|trainer.py:803] 2025-04-26 22:21:50,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7818
7866
[WARNING|trainer.py:803] 2025-04-26 22:21:51,067 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8834
[WARNING|trainer.py:803] 2025-04-26 22:21:51,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:51,866 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:21:52,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8835
7819
7867
[WARNING|trainer.py:803] 2025-04-26 22:21:53,313 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:53,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8836
[WARNING|trainer.py:803] 2025-04-26 22:21:53,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:54,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7820
8837
7868
[WARNING|trainer.py:803] 2025-04-26 22:21:55,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:21:55,511 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8838
[WARNING|trainer.py:803] 2025-04-26 22:21:55,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7821
[WARNING|trainer.py:803] 2025-04-26 22:21:56,591 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7869
8839
[WARNING|trainer.py:803] 2025-04-26 22:21:57,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:57,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:21:57,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8840
7822
[WARNING|trainer.py:803] 2025-04-26 22:21:58,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7870
[WARNING|trainer.py:803] 2025-04-26 22:21:58,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8841
[WARNING|trainer.py:803] 2025-04-26 22:21:59,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:21:59,905 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8842
7823
7871
[WARNING|trainer.py:803] 2025-04-26 22:22:00,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:00,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8843
[WARNING|trainer.py:803] 2025-04-26 22:22:01,520 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7824
[WARNING|trainer.py:803] 2025-04-26 22:22:02,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7872
8844
[WARNING|trainer.py:803] 2025-04-26 22:22:02,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:03,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:03,304 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8845
7825
7873
[WARNING|trainer.py:803] 2025-04-26 22:22:04,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:04,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8846
[WARNING|trainer.py:803] 2025-04-26 22:22:04,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7826
[WARNING|trainer.py:803] 2025-04-26 22:22:05,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7874
8847
[WARNING|trainer.py:803] 2025-04-26 22:22:06,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:06,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:06,655 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8848
7827
[WARNING|trainer.py:803] 2025-04-26 22:22:07,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:07,961 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7875
8849
[WARNING|trainer.py:803] 2025-04-26 22:22:08,895 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:08,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8850
7828
7876
[WARNING|trainer.py:803] 2025-04-26 22:22:10,023 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:10,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8851
[WARNING|trainer.py:803] 2025-04-26 22:22:10,639 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:11,131 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7829
8852
7877
[WARNING|trainer.py:803] 2025-04-26 22:22:11,994 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:12,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:12,398 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8853
7830
7878
[WARNING|trainer.py:803] 2025-04-26 22:22:13,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:13,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8854
[WARNING|trainer.py:803] 2025-04-26 22:22:14,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:14,568 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8855
7831
7879
[WARNING|trainer.py:803] 2025-04-26 22:22:15,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:15,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8856
[WARNING|trainer.py:803] 2025-04-26 22:22:15,931 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7832
[WARNING|trainer.py:803] 2025-04-26 22:22:16,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8857
7880
[WARNING|trainer.py:803] 2025-04-26 22:22:17,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:17,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:18,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8858
7833
7881
[WARNING|trainer.py:803] 2025-04-26 22:22:18,991 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:19,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8859
[WARNING|trainer.py:803] 2025-04-26 22:22:19,684 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7834
[WARNING|trainer.py:803] 2025-04-26 22:22:20,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8860
7882
[WARNING|trainer.py:803] 2025-04-26 22:22:20,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:21,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8861
[WARNING|trainer.py:803] 2025-04-26 22:22:21,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7835
[WARNING|trainer.py:803] 2025-04-26 22:22:22,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8862
[WARNING|trainer.py:803] 2025-04-26 22:22:22,585 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7883
[WARNING|trainer.py:803] 2025-04-26 22:22:23,383 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7836
8863
[WARNING|trainer.py:803] 2025-04-26 22:22:23,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:24,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:24,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8864
7837
7884
[WARNING|trainer.py:803] 2025-04-26 22:22:25,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:25,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8865
[WARNING|trainer.py:803] 2025-04-26 22:22:26,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:26,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7838
7885
8866
[WARNING|trainer.py:803] 2025-04-26 22:22:27,805 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:27,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:27,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8867
7839
7886
[WARNING|trainer.py:803] 2025-04-26 22:22:29,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8868
[WARNING|trainer.py:803] 2025-04-26 22:22:29,436 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:29,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:30,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7840
8869
7887
[WARNING|trainer.py:803] 2025-04-26 22:22:31,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:31,285 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8870
7841
[WARNING|trainer.py:803] 2025-04-26 22:22:31,848 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:32,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:32,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7888
8871
[WARNING|trainer.py:803] 2025-04-26 22:22:33,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:33,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8872
7842
7889
[WARNING|trainer.py:803] 2025-04-26 22:22:34,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:34,832 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8873
[WARNING|trainer.py:803] 2025-04-26 22:22:34,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:35,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7890
7843
8874
[WARNING|trainer.py:803] 2025-04-26 22:22:36,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:36,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:36,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8875
7891
[WARNING|trainer.py:803] 2025-04-26 22:22:37,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8876
7844
[WARNING|trainer.py:803] 2025-04-26 22:22:38,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:39,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:39,058 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8877
7892
7845
[WARNING|trainer.py:803] 2025-04-26 22:22:40,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8878
[WARNING|trainer.py:803] 2025-04-26 22:22:40,514 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:40,635 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:41,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7893
8879
7846
[WARNING|trainer.py:803] 2025-04-26 22:22:42,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:42,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8880
[WARNING|trainer.py:803] 2025-04-26 22:22:42,761 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:43,473 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8881
7894
7847
[WARNING|trainer.py:803] 2025-04-26 22:22:44,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:44,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:44,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8882
7848
[WARNING|trainer.py:803] 2025-04-26 22:22:45,690 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7895
8883
[WARNING|trainer.py:803] 2025-04-26 22:22:46,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:46,537 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:46,810 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8884
7849
7896
[WARNING|trainer.py:803] 2025-04-26 22:22:47,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8885
[WARNING|trainer.py:803] 2025-04-26 22:22:48,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:48,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:48,983 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8886
7897
7850
[WARNING|trainer.py:803] 2025-04-26 22:22:50,089 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:50,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8887
[WARNING|trainer.py:803] 2025-04-26 22:22:50,471 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:51,211 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7898
8888
7851
[WARNING|trainer.py:803] 2025-04-26 22:22:52,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:52,336 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:52,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8889
7899
[WARNING|trainer.py:803] 2025-04-26 22:22:53,480 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8890
7852
[WARNING|trainer.py:803] 2025-04-26 22:22:54,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:22:54,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:22:54,734 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8891
7900
7853
[WARNING|trainer.py:803] 2025-04-26 22:22:55,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:22:55,901 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8892
[WARNING|trainer.py:803] 2025-04-26 22:22:56,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:56,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7901
8893
7854
[WARNING|trainer.py:803] 2025-04-26 22:22:57,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:57,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8894
[WARNING|trainer.py:803] 2025-04-26 22:22:58,268 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7902
[WARNING|trainer.py:803] 2025-04-26 22:22:59,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7855
8895
[WARNING|trainer.py:803] 2025-04-26 22:22:59,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:22:59,922 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:00,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8896
7903
7856
[WARNING|trainer.py:803] 2025-04-26 22:23:01,217 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:01,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:01,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8897
[WARNING|trainer.py:803] 2025-04-26 22:23:02,426 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7857
8898
7904
[WARNING|trainer.py:803] 2025-04-26 22:23:03,279 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:03,533 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:03,755 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8899
7858
7905
[WARNING|trainer.py:803] 2025-04-26 22:23:04,685 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:04,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8900
[WARNING|trainer.py:803] 2025-04-26 22:23:05,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7859
[WARNING|trainer.py:803] 2025-04-26 22:23:05,814 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8901
7906
[WARNING|trainer.py:803] 2025-04-26 22:23:06,617 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:06,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:07,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8902
7860
[WARNING|trainer.py:803] 2025-04-26 22:23:08,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:08,305 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7907
8903
7861
[WARNING|trainer.py:803] 2025-04-26 22:23:09,175 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:09,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8904
[WARNING|trainer.py:803] 2025-04-26 22:23:09,783 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7908
[WARNING|trainer.py:803] 2025-04-26 22:23:10,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8905
7862
[WARNING|trainer.py:803] 2025-04-26 22:23:10,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:11,449 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:11,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7909
8906
7863
[WARNING|trainer.py:803] 2025-04-26 22:23:12,476 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:12,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8907
[WARNING|trainer.py:803] 2025-04-26 22:23:13,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7910
[WARNING|trainer.py:803] 2025-04-26 22:23:13,666 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7864
8908
[WARNING|trainer.py:803] 2025-04-26 22:23:14,314 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:14,795 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:14,825 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8909
7911
[WARNING|trainer.py:803] 2025-04-26 22:23:15,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:16,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7865
8910
[WARNING|trainer.py:803] 2025-04-26 22:23:16,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7912
[WARNING|trainer.py:803] 2025-04-26 22:23:17,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8911
7866
[WARNING|trainer.py:803] 2025-04-26 22:23:17,746 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:18,174 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:18,359 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8912
7913
7867
[WARNING|trainer.py:803] 2025-04-26 22:23:19,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:19,389 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8913
[WARNING|trainer.py:803] 2025-04-26 22:23:20,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7914
[WARNING|trainer.py:803] 2025-04-26 22:23:20,441 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8914
[WARNING|trainer.py:803] 2025-04-26 22:23:21,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:21,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7868
8915
7915
[WARNING|trainer.py:803] 2025-04-26 22:23:22,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:22,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:22,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8916
7869
[WARNING|trainer.py:803] 2025-04-26 22:23:23,733 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8917
7916
[WARNING|trainer.py:803] 2025-04-26 22:23:24,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:24,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:24,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8918
7870
7917
[WARNING|trainer.py:803] 2025-04-26 22:23:25,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8919
[WARNING|trainer.py:803] 2025-04-26 22:23:26,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:26,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:27,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7871
8920
7918
[WARNING|trainer.py:803] 2025-04-26 22:23:28,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:28,226 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8921
[WARNING|trainer.py:803] 2025-04-26 22:23:28,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7872
[WARNING|trainer.py:803] 2025-04-26 22:23:29,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7919
8922
[WARNING|trainer.py:803] 2025-04-26 22:23:29,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:30,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7873
[WARNING|trainer.py:803] 2025-04-26 22:23:30,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8923
[WARNING|trainer.py:803] 2025-04-26 22:23:31,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:31,775 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7920
8924
7874
[WARNING|trainer.py:803] 2025-04-26 22:23:32,618 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:32,942 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:32,949 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8925
7921
[WARNING|trainer.py:803] 2025-04-26 22:23:34,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8926
7875
[WARNING|trainer.py:803] 2025-04-26 22:23:34,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:35,115 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:35,452 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8927
7922
[WARNING|trainer.py:803] 2025-04-26 22:23:36,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7876
8928
[WARNING|trainer.py:803] 2025-04-26 22:23:37,093 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:37,213 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:37,515 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8929
7923
7877
[WARNING|trainer.py:803] 2025-04-26 22:23:38,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8930
[WARNING|trainer.py:803] 2025-04-26 22:23:38,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:38,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:39,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7878
8931
7924
[WARNING|trainer.py:803] 2025-04-26 22:23:40,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:40,853 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8932
[WARNING|trainer.py:803] 2025-04-26 22:23:41,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7879
[WARNING|trainer.py:803] 2025-04-26 22:23:42,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7925
8933
[WARNING|trainer.py:803] 2025-04-26 22:23:42,508 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:42,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:43,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8934
7880
7926
[WARNING|trainer.py:803] 2025-04-26 22:23:44,309 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:44,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8935
[WARNING|trainer.py:803] 2025-04-26 22:23:44,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7881
[WARNING|trainer.py:803] 2025-04-26 22:23:45,434 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8936
[WARNING|trainer.py:803] 2025-04-26 22:23:46,231 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7927
[WARNING|trainer.py:803] 2025-04-26 22:23:46,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8937
[WARNING|trainer.py:803] 2025-04-26 22:23:47,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7882
[WARNING|trainer.py:803] 2025-04-26 22:23:47,719 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7928
8938
[WARNING|trainer.py:803] 2025-04-26 22:23:48,013 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:48,720 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:48,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8939
7883
7929
[WARNING|trainer.py:803] 2025-04-26 22:23:49,960 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8940
[WARNING|trainer.py:803] 2025-04-26 22:23:50,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:50,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:51,117 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7930
8941
7884
[WARNING|trainer.py:803] 2025-04-26 22:23:52,299 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:52,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8942
[WARNING|trainer.py:803] 2025-04-26 22:23:52,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7931
[WARNING|trainer.py:803] 2025-04-26 22:23:53,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7885
8943
[WARNING|trainer.py:803] 2025-04-26 22:23:53,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:23:54,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:54,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8944
7932
7886
[WARNING|trainer.py:803] 2025-04-26 22:23:55,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8945
[WARNING|trainer.py:803] 2025-04-26 22:23:56,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:23:56,338 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:23:56,689 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8946
7933
7887
[WARNING|trainer.py:803] 2025-04-26 22:23:57,782 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:57,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8947
[WARNING|trainer.py:803] 2025-04-26 22:23:58,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7934
[WARNING|trainer.py:803] 2025-04-26 22:23:58,890 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8948
7888
[WARNING|trainer.py:803] 2025-04-26 22:23:59,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:23:59,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:00,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8949
7889
7935
[WARNING|trainer.py:803] 2025-04-26 22:24:01,083 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8950
[WARNING|trainer.py:803] 2025-04-26 22:24:01,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:01,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:02,235 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7890
8951
7936
[WARNING|trainer.py:803] 2025-04-26 22:24:03,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:03,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:03,502 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8952
7891
[WARNING|trainer.py:803] 2025-04-26 22:24:04,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7937
8953
[WARNING|trainer.py:803] 2025-04-26 22:24:05,076 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:05,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:05,580 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8954
7892
7938
[WARNING|trainer.py:803] 2025-04-26 22:24:06,663 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8955
[WARNING|trainer.py:803] 2025-04-26 22:24:07,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:07,293 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:07,764 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7893
8956
7939
[WARNING|trainer.py:803] 2025-04-26 22:24:08,822 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:08,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8957
[WARNING|trainer.py:803] 2025-04-26 22:24:09,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:09,995 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7940
8958
7894
[WARNING|trainer.py:803] 2025-04-26 22:24:11,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:11,114 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:11,247 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8959
[WARNING|trainer.py:803] 2025-04-26 22:24:12,166 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7895
7941
8960
[WARNING|trainer.py:803] 2025-04-26 22:24:13,042 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:13,112 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:13,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8961
7896
7942
[WARNING|trainer.py:803] 2025-04-26 22:24:14,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8962
[WARNING|trainer.py:803] 2025-04-26 22:24:14,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:14,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:15,499 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8963
7897
7943
[WARNING|trainer.py:803] 2025-04-26 22:24:16,586 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:16,676 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8964
[WARNING|trainer.py:803] 2025-04-26 22:24:16,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:17,702 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7944
7898
8965
[WARNING|trainer.py:803] 2025-04-26 22:24:18,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:18,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:18,809 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8966
7899
[WARNING|trainer.py:803] 2025-04-26 22:24:19,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8967
7945
[WARNING|trainer.py:803] 2025-04-26 22:24:20,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:21,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:21,142 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8968
7900
7946
[WARNING|trainer.py:803] 2025-04-26 22:24:22,178 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:22,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8969
[WARNING|trainer.py:803] 2025-04-26 22:24:22,784 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:24:23,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7901
8970
7947
[WARNING|trainer.py:803] 2025-04-26 22:24:24,311 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:24,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:24,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8971
7902
[WARNING|trainer.py:803] 2025-04-26 22:24:25,547 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8972
7948
[WARNING|trainer.py:803] 2025-04-26 22:24:26,197 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:26,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:26,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8973
7903
7949
[WARNING|trainer.py:803] 2025-04-26 22:24:27,747 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:27,875 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8974
[WARNING|trainer.py:803] 2025-04-26 22:24:28,406 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:28,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7904
8975
7950
[WARNING|trainer.py:803] 2025-04-26 22:24:30,007 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:30,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:30,438 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8976
7905
[WARNING|trainer.py:803] 2025-04-26 22:24:31,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7951
[WARNING|trainer.py:803] 2025-04-26 22:24:31,674 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8977
[WARNING|trainer.py:803] 2025-04-26 22:24:32,466 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:32,555 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7906
8978
[WARNING|trainer.py:803] 2025-04-26 22:24:33,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7952
[WARNING|trainer.py:803] 2025-04-26 22:24:33,790 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8979
7907
[WARNING|trainer.py:803] 2025-04-26 22:24:34,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:34,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8980
[WARNING|trainer.py:803] 2025-04-26 22:24:35,430 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7953
[WARNING|trainer.py:803] 2025-04-26 22:24:35,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7908
8981
[WARNING|trainer.py:803] 2025-04-26 22:24:36,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:37,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:37,148 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8982
7954
7909
[WARNING|trainer.py:803] 2025-04-26 22:24:38,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:38,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8983
[WARNING|trainer.py:803] 2025-04-26 22:24:38,707 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7955
[WARNING|trainer.py:803] 2025-04-26 22:24:39,450 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8984
7910
[WARNING|trainer.py:803] 2025-04-26 22:24:40,257 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:40,521 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:40,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8985
7956
7911
[WARNING|trainer.py:803] 2025-04-26 22:24:41,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:41,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8986
[WARNING|trainer.py:803] 2025-04-26 22:24:42,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:42,828 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7957
8987
7912
[WARNING|trainer.py:803] 2025-04-26 22:24:43,724 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:43,965 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:44,010 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8988
7913
[WARNING|trainer.py:803] 2025-04-26 22:24:45,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8989
7958
[WARNING|trainer.py:803] 2025-04-26 22:24:45,682 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:46,177 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:46,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8990
7914
7959
[WARNING|trainer.py:803] 2025-04-26 22:24:47,354 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:47,423 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8991
[WARNING|trainer.py:803] 2025-04-26 22:24:47,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7915
[WARNING|trainer.py:803] 2025-04-26 22:24:48,439 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8992
7960
[WARNING|trainer.py:803] 2025-04-26 22:24:49,160 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:49,504 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8993
[WARNING|trainer.py:803] 2025-04-26 22:24:49,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7916
[WARNING|trainer.py:803] 2025-04-26 22:24:50,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7961
8994
[WARNING|trainer.py:803] 2025-04-26 22:24:51,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:24:51,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:51,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8995
7917
7962
[WARNING|trainer.py:803] 2025-04-26 22:24:52,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:52,953 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8996
[WARNING|trainer.py:803] 2025-04-26 22:24:53,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:53,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8997
7918
7963
[WARNING|trainer.py:803] 2025-04-26 22:24:54,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:55,125 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:24:55,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8998
7919
[WARNING|trainer.py:803] 2025-04-26 22:24:56,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8999
7964
[WARNING|trainer.py:803] 2025-04-26 22:24:56,841 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:57,260 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:24:57,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
9000
7920
[WARNING|trainer.py:803] 2025-04-26 22:24:58,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7965
Results saved to /home/wangjiarui/InternVL/internvl_chat/AIGV60K_weights/qa_fast/qa_results.csv
Accuracy: 0.787
[WARNING|trainer.py:803] 2025-04-26 22:24:58,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:24:59,204 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7921
7966
[2025-04-26 22:25:00,538] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING|trainer.py:803] 2025-04-26 22:25:01,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:01,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7967
7922
[WARNING|trainer.py:803] 2025-04-26 22:25:03,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:03,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
7968
7923
[WARNING|trainer.py:803] 2025-04-26 22:25:05,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:05,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7969
[2025-04-26 22:25:06,450] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
7924
[WARNING|trainer.py:803] 2025-04-26 22:25:07,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:25:07,670 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7970
7925
[WARNING|trainer.py:803] 2025-04-26 22:25:08,696 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[WARNING|trainer.py:803] 2025-04-26 22:25:09,332 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7971
7926
[WARNING|trainer.py:803] 2025-04-26 22:25:10,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:10,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7972
[WARNING|trainer.py:803] 2025-04-26 22:25:11,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[2025-04-26 22:25:12,145] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
7927
7973
[WARNING|trainer.py:803] 2025-04-26 22:25:13,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:13,656 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7928
7974
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[WARNING|trainer.py:803] 2025-04-26 22:25:15,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:15,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7929
7975
[WARNING|trainer.py:803] 2025-04-26 22:25:16,725 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:17,072 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7930
[2025-04-26 22:25:17,667] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
7976
[WARNING|trainer.py:803] 2025-04-26 22:25:18,539 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:19,138 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7931
[WARNING|trainer.py:803] 2025-04-26 22:25:19,987 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7977
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
[WARNING|trainer.py:803] 2025-04-26 22:25:20,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7932
7978
[WARNING|trainer.py:803] 2025-04-26 22:25:22,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:22,859 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7933
7979
[WARNING|trainer.py:803] 2025-04-26 22:25:24,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:24,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7934
[WARNING|trainer.py:803] 2025-04-26 22:25:25,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7980
[WARNING|trainer.py:803] 2025-04-26 22:25:26,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7935
7981
[WARNING|trainer.py:803] 2025-04-26 22:25:28,054 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:28,770 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7936
7982
[WARNING|trainer.py:803] 2025-04-26 22:25:29,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:30,399 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7937
7983
[WARNING|trainer.py:803] 2025-04-26 22:25:31,787 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:32,408 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7938
7984
[WARNING|trainer.py:803] 2025-04-26 22:25:33,478 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:25:34,040 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7939
7985
[WARNING|trainer.py:803] 2025-04-26 22:25:35,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:25:36,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7940
[WARNING|trainer.py:803] 2025-04-26 22:25:37,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7986
[WARNING|trainer.py:803] 2025-04-26 22:25:37,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7941
[WARNING|trainer.py:803] 2025-04-26 22:25:38,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7987
7942
[WARNING|trainer.py:803] 2025-04-26 22:25:39,954 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:40,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7988
7943
[WARNING|trainer.py:803] 2025-04-26 22:25:42,009 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:42,587 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7989
7944
[WARNING|trainer.py:803] 2025-04-26 22:25:43,658 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:44,345 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7990
[WARNING|trainer.py:803] 2025-04-26 22:25:45,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7945
[WARNING|trainer.py:803] 2025-04-26 22:25:46,751 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7991
7946
[WARNING|trainer.py:803] 2025-04-26 22:25:47,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:48,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
7992
7947
[WARNING|trainer.py:803] 2025-04-26 22:25:49,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:50,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7993
7948
[WARNING|trainer.py:803] 2025-04-26 22:25:51,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:52,264 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7949
7994
[WARNING|trainer.py:803] 2025-04-26 22:25:53,910 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:25:53,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7995
7950
[WARNING|trainer.py:803] 2025-04-26 22:25:55,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:25:55,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7996
7951
[WARNING|trainer.py:803] 2025-04-26 22:25:57,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:25:57,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7997
7952
[WARNING|trainer.py:803] 2025-04-26 22:25:59,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:00,020 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7998
7953
[WARNING|trainer.py:803] 2025-04-26 22:26:01,652 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:01,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7999
7954
[WARNING|trainer.py:803] 2025-04-26 22:26:03,445 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:03,726 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7955
8000
[WARNING|trainer.py:803] 2025-04-26 22:26:05,679 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:05,739 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7956
8001
[WARNING|trainer.py:803] 2025-04-26 22:26:07,312 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:07,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7957
8002
[WARNING|trainer.py:803] 2025-04-26 22:26:09,046 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:09,816 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7958
8003
[WARNING|trainer.py:803] 2025-04-26 22:26:11,489 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:26:11,634 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7959
8004
[WARNING|trainer.py:803] 2025-04-26 22:26:13,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:13,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8005
7960
[WARNING|trainer.py:803] 2025-04-26 22:26:14,933 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:26:15,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8006
7961
[WARNING|trainer.py:803] 2025-04-26 22:26:16,505 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:16,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8007
7962
[WARNING|trainer.py:803] 2025-04-26 22:26:18,432 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:18,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
7963
8008
[WARNING|trainer.py:803] 2025-04-26 22:26:20,322 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:26:20,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8009
7964
[WARNING|trainer.py:803] 2025-04-26 22:26:22,281 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:22,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8010
7965
[WARNING|trainer.py:803] 2025-04-26 22:26:23,873 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:24,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8011
7966
[WARNING|trainer.py:803] 2025-04-26 22:26:25,487 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:26,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8012
[WARNING|trainer.py:803] 2025-04-26 22:26:27,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7967
8013
[WARNING|trainer.py:803] 2025-04-26 22:26:28,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:28,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7968
8014
[WARNING|trainer.py:803] 2025-04-26 22:26:30,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:30,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8015
7969
[WARNING|trainer.py:803] 2025-04-26 22:26:31,870 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:31,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8016
7970
[WARNING|trainer.py:803] 2025-04-26 22:26:33,468 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:26:33,524 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8017
7971
[WARNING|trainer.py:803] 2025-04-26 22:26:35,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:35,238 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7972
8018
[WARNING|trainer.py:803] 2025-04-26 22:26:36,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:37,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7973
8019
[WARNING|trainer.py:803] 2025-04-26 22:26:38,566 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:38,736 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
7974
8020
[WARNING|trainer.py:803] 2025-04-26 22:26:40,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:40,803 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7975
8021
[WARNING|trainer.py:803] 2025-04-26 22:26:41,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:26:42,326 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7976
8022
[WARNING|trainer.py:803] 2025-04-26 22:26:44,005 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:44,350 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7977
8023
[WARNING|trainer.py:803] 2025-04-26 22:26:45,730 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:26:46,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7978
8024
[WARNING|trainer.py:803] 2025-04-26 22:26:47,703 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:47,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7979
8025
[WARNING|trainer.py:803] 2025-04-26 22:26:49,741 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:49,742 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8026
7980
[WARNING|trainer.py:803] 2025-04-26 22:26:51,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:51,771 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8027
7981
[WARNING|trainer.py:803] 2025-04-26 22:26:53,266 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:26:53,556 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8028
7982
[WARNING|trainer.py:803] 2025-04-26 22:26:54,854 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:55,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8029
7983
[WARNING|trainer.py:803] 2025-04-26 22:26:56,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:57,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8030
7984
[WARNING|trainer.py:803] 2025-04-26 22:26:58,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:26:58,756 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8031
[WARNING|trainer.py:803] 2025-04-26 22:26:59,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7985
[WARNING|trainer.py:803] 2025-04-26 22:27:00,776 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8032
[WARNING|trainer.py:803] 2025-04-26 22:27:01,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7986
8033
[WARNING|trainer.py:803] 2025-04-26 22:27:02,640 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:03,327 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
7987
8034
[WARNING|trainer.py:803] 2025-04-26 22:27:04,672 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:05,090 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8035
7988
[WARNING|trainer.py:803] 2025-04-26 22:27:06,650 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:06,727 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8036
7989
[WARNING|trainer.py:803] 2025-04-26 22:27:08,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:08,409 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
7990
[WARNING|trainer.py:803] 2025-04-26 22:27:10,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:10,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8038
7991
[WARNING|trainer.py:803] 2025-04-26 22:27:11,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:27:12,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8039
[WARNING|trainer.py:803] 2025-04-26 22:27:13,337 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
7992
8040
[WARNING|trainer.py:803] 2025-04-26 22:27:14,748 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:15,100 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8041
7993
[WARNING|trainer.py:803] 2025-04-26 22:27:16,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:16,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8042
7994
[WARNING|trainer.py:803] 2025-04-26 22:27:18,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:18,613 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8043
7995
[WARNING|trainer.py:803] 2025-04-26 22:27:19,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:20,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8044
7996
[WARNING|trainer.py:803] 2025-04-26 22:27:21,697 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:22,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8045
[WARNING|trainer.py:803] 2025-04-26 22:27:23,351 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
7997
[WARNING|trainer.py:803] 2025-04-26 22:27:24,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8046
7998
[WARNING|trainer.py:803] 2025-04-26 22:27:25,713 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:26,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8047
7999
[WARNING|trainer.py:803] 2025-04-26 22:27:27,443 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:28,233 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8048
[WARNING|trainer.py:803] 2025-04-26 22:27:29,165 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8000
8049
[WARNING|trainer.py:803] 2025-04-26 22:27:30,526 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:31,027 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8001
8050
[WARNING|trainer.py:803] 2025-04-26 22:27:32,599 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:32,647 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8051
8002
[WARNING|trainer.py:803] 2025-04-26 22:27:34,400 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:34,627 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8052
8003
[WARNING|trainer.py:803] 2025-04-26 22:27:36,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:36,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8053
8004
[WARNING|trainer.py:803] 2025-04-26 22:27:37,578 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:27:37,998 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8054
8005
[WARNING|trainer.py:803] 2025-04-26 22:27:39,158 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:39,851 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8055
8006
[WARNING|trainer.py:803] 2025-04-26 22:27:40,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8056
[WARNING|trainer.py:803] 2025-04-26 22:27:41,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:42,239 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8007
8057
[WARNING|trainer.py:803] 2025-04-26 22:27:43,417 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:43,847 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8008
8058
[WARNING|trainer.py:803] 2025-04-26 22:27:45,361 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:45,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8059
8009
[WARNING|trainer.py:803] 2025-04-26 22:27:47,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:47,271 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8060
8010
[WARNING|trainer.py:803] 2025-04-26 22:27:48,516 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:48,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8061
8011
[WARNING|trainer.py:803] 2025-04-26 22:27:50,214 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:27:50,457 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8062
8012
[WARNING|trainer.py:803] 2025-04-26 22:27:51,946 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:52,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8013
8063
[WARNING|trainer.py:803] 2025-04-26 22:27:53,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:53,762 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8064
8014
[WARNING|trainer.py:803] 2025-04-26 22:27:55,382 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:27:55,395 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8015
8065
[WARNING|trainer.py:803] 2025-04-26 22:27:56,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:27:57,176 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8016
8066
[WARNING|trainer.py:803] 2025-04-26 22:27:58,582 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:27:58,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8017
8067
[WARNING|trainer.py:803] 2025-04-26 22:28:00,200 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:00,490 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8068
8018
[WARNING|trainer.py:803] 2025-04-26 22:28:02,153 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:28:02,167 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8019
8069
[WARNING|trainer.py:803] 2025-04-26 22:28:03,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:28:03,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8070
8020
[WARNING|trainer.py:803] 2025-04-26 22:28:05,671 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:28:05,861 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8071
8021
[WARNING|trainer.py:803] 2025-04-26 22:28:07,318 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:07,401 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8072
8022
[WARNING|trainer.py:803] 2025-04-26 22:28:09,092 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:28:09,464 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8073
8023
[WARNING|trainer.py:803] 2025-04-26 22:28:10,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:11,328 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8024
8074
[WARNING|trainer.py:803] 2025-04-26 22:28:12,980 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:12,997 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8025
8075
[WARNING|trainer.py:803] 2025-04-26 22:28:14,802 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:15,133 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8026
8076
[WARNING|trainer.py:803] 2025-04-26 22:28:16,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:16,800 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8027
8077
[WARNING|trainer.py:803] 2025-04-26 22:28:18,258 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:18,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8028
8078
[WARNING|trainer.py:803] 2025-04-26 22:28:19,820 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:19,927 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8029
8079
[WARNING|trainer.py:803] 2025-04-26 22:28:21,410 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:21,535 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8030
8080
[WARNING|trainer.py:803] 2025-04-26 22:28:23,154 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:23,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8031
8081
[WARNING|trainer.py:803] 2025-04-26 22:28:24,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:25,280 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8032
8082
[WARNING|trainer.py:803] 2025-04-26 22:28:26,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:27,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8033
8083
[WARNING|trainer.py:803] 2025-04-26 22:28:28,172 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:28,722 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8034
8084
[WARNING|trainer.py:803] 2025-04-26 22:28:29,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:30,308 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8035
8085
[WARNING|trainer.py:803] 2025-04-26 22:28:31,491 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:32,145 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8036
[WARNING|trainer.py:803] 2025-04-26 22:28:33,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8086
[WARNING|trainer.py:803] 2025-04-26 22:28:34,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8037
[WARNING|trainer.py:803] 2025-04-26 22:28:35,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8087
8038
[WARNING|trainer.py:803] 2025-04-26 22:28:36,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:36,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8088
8039
[WARNING|trainer.py:803] 2025-04-26 22:28:37,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:38,230 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8089
8040
[WARNING|trainer.py:803] 2025-04-26 22:28:39,283 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:40,012 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8041
8090
[WARNING|trainer.py:803] 2025-04-26 22:28:41,668 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:41,801 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8042
8091
[WARNING|trainer.py:803] 2025-04-26 22:28:43,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:43,437 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8043
8092
[WARNING|trainer.py:803] 2025-04-26 22:28:44,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:45,187 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8044
8093
[WARNING|trainer.py:803] 2025-04-26 22:28:46,705 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:46,813 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8045
8094
[WARNING|trainer.py:803] 2025-04-26 22:28:48,331 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:28:48,603 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8095
8046
[WARNING|trainer.py:803] 2025-04-26 22:28:50,065 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:50,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8096
8047
[WARNING|trainer.py:803] 2025-04-26 22:28:51,614 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:52,372 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8097
8048
[WARNING|trainer.py:803] 2025-04-26 22:28:53,269 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:28:54,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8098
8049
[WARNING|trainer.py:803] 2025-04-26 22:28:55,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:55,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8099
8050
[WARNING|trainer.py:803] 2025-04-26 22:28:56,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:28:57,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8100
8051
[WARNING|trainer.py:803] 2025-04-26 22:28:58,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
[WARNING|trainer.py:803] 2025-04-26 22:28:59,273 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8101
8052
[WARNING|trainer.py:803] 2025-04-26 22:29:00,695 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:00,900 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8053
8102
[WARNING|trainer.py:803] 2025-04-26 22:29:02,414 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:29:03,073 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8054
[WARNING|trainer.py:803] 2025-04-26 22:29:04,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8103
8055
[WARNING|trainer.py:803] 2025-04-26 22:29:05,123 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:29:05,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8104
8056
[WARNING|trainer.py:803] 2025-04-26 22:29:07,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:07,191 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8057
8105
[WARNING|trainer.py:803] 2025-04-26 22:29:08,834 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:09,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8058
8106
[WARNING|trainer.py:803] 2025-04-26 22:29:10,528 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
[WARNING|trainer.py:803] 2025-04-26 22:29:11,171 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8059
[WARNING|trainer.py:803] 2025-04-26 22:29:12,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8107
8060
[WARNING|trainer.py:803] 2025-04-26 22:29:13,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:29:13,573 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8108
8061
[WARNING|trainer.py:803] 2025-04-26 22:29:15,205 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:15,259 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8062
8109
[WARNING|trainer.py:803] 2025-04-26 22:29:17,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:17,321 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8063
8110
[WARNING|trainer.py:803] 2025-04-26 22:29:18,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:19,284 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8064
8111
[WARNING|trainer.py:803] 2025-04-26 22:29:20,501 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:21,267 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8065
[WARNING|trainer.py:803] 2025-04-26 22:29:22,315 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8112
8066
[WARNING|trainer.py:803] 2025-04-26 22:29:23,330 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:24,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8113
8067
[WARNING|trainer.py:803] 2025-04-26 22:29:25,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:25,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8068
8114
[WARNING|trainer.py:803] 2025-04-26 22:29:27,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:27,455 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8069
8115
[WARNING|trainer.py:803] 2025-04-26 22:29:29,196 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:29,552 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8070
8116
[WARNING|trainer.py:803] 2025-04-26 22:29:30,934 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:31,623 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8071
[WARNING|trainer.py:803] 2025-04-26 22:29:32,615 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8117
8072
[WARNING|trainer.py:803] 2025-04-26 22:29:33,622 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:29:34,364 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8118
8073
[WARNING|trainer.py:803] 2025-04-26 22:29:35,714 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:36,207 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8119
8074
[WARNING|trainer.py:803] 2025-04-26 22:29:37,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:38,317 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8120
8075
[WARNING|trainer.py:803] 2025-04-26 22:29:39,821 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:29:40,467 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8121
8076
[WARNING|trainer.py:803] 2025-04-26 22:29:41,845 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:42,168 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8077
8122
[WARNING|trainer.py:803] 2025-04-26 22:29:43,738 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:29:43,924 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8078
8123
[WARNING|trainer.py:803] 2025-04-26 22:29:45,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:29:46,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8079
[WARNING|trainer.py:803] 2025-04-26 22:29:46,894 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8124
8080
[WARNING|trainer.py:803] 2025-04-26 22:29:48,060 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:48,595 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8125
8081
[WARNING|trainer.py:803] 2025-04-26 22:29:50,032 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:50,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8126
8082
[WARNING|trainer.py:803] 2025-04-26 22:29:52,044 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:52,431 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8127
8083
[WARNING|trainer.py:803] 2025-04-26 22:29:53,936 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:29:54,082 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8084
8128
[WARNING|trainer.py:803] 2025-04-26 22:29:55,659 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:29:56,193 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8085
8129
[WARNING|trainer.py:803] 2025-04-26 22:29:57,530 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:29:58,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8086
[WARNING|trainer.py:803] 2025-04-26 22:29:59,503 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8130
[WARNING|trainer.py:803] 2025-04-26 22:30:00,517 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8087
[WARNING|trainer.py:803] 2025-04-26 22:30:01,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8131
8088
[WARNING|trainer.py:803] 2025-04-26 22:30:02,628 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:30:03,017 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8132
8089
[WARNING|trainer.py:803] 2025-04-26 22:30:04,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:30:04,662 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8133
8090
[WARNING|trainer.py:803] 2025-04-26 22:30:06,882 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:30:07,209 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8091
8134
[WARNING|trainer.py:803] 2025-04-26 22:30:08,872 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:08,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8092
8135
[WARNING|trainer.py:803] 2025-04-26 22:30:10,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:10,863 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8093
8136
[WARNING|trainer.py:803] 2025-04-26 22:30:12,306 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:12,878 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8094
8137
[WARNING|trainer.py:803] 2025-04-26 22:30:14,078 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8095
[WARNING|trainer.py:803] 2025-04-26 22:30:14,865 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:30:15,553 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8138
8096
[WARNING|trainer.py:803] 2025-04-26 22:30:17,001 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:30:17,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8097
8139
[WARNING|trainer.py:803] 2025-04-26 22:30:18,753 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:30:18,923 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8098
8140
[WARNING|trainer.py:803] 2025-04-26 22:30:20,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:20,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8099
8141
[WARNING|trainer.py:803] 2025-04-26 22:30:22,477 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:22,956 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8100
[WARNING|trainer.py:803] 2025-04-26 22:30:24,130 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8142
[WARNING|trainer.py:803] 2025-04-26 22:30:25,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8101
[WARNING|trainer.py:803] 2025-04-26 22:30:26,255 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8143
[WARNING|trainer.py:803] 2025-04-26 22:30:27,234 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8102
[WARNING|trainer.py:803] 2025-04-26 22:30:28,251 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8144
[WARNING|trainer.py:803] 2025-04-26 22:30:29,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8103
[WARNING|trainer.py:803] 2025-04-26 22:30:30,369 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8145
[WARNING|trainer.py:803] 2025-04-26 22:30:31,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8104
[WARNING|trainer.py:803] 2025-04-26 22:30:32,390 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8146
[WARNING|trainer.py:803] 2025-04-26 22:30:33,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8105
[WARNING|trainer.py:803] 2025-04-26 22:30:34,403 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8147
[WARNING|trainer.py:803] 2025-04-26 22:30:35,444 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8106
[WARNING|trainer.py:803] 2025-04-26 22:30:36,433 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8148
[WARNING|trainer.py:803] 2025-04-26 22:30:37,549 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8107
[WARNING|trainer.py:803] 2025-04-26 22:30:38,416 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8149
[WARNING|trainer.py:803] 2025-04-26 22:30:39,548 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8108
[WARNING|trainer.py:803] 2025-04-26 22:30:40,532 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8150
[WARNING|trainer.py:803] 2025-04-26 22:30:41,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8109
[WARNING|trainer.py:803] 2025-04-26 22:30:42,669 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8151
[WARNING|trainer.py:803] 2025-04-26 22:30:43,797 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8110
[WARNING|trainer.py:803] 2025-04-26 22:30:44,708 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8152
8111
[WARNING|trainer.py:803] 2025-04-26 22:30:45,959 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:30:46,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8153
8112
[WARNING|trainer.py:803] 2025-04-26 22:30:47,925 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:30:48,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8154
[WARNING|trainer.py:803] 2025-04-26 22:30:49,909 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8113
[WARNING|trainer.py:803] 2025-04-26 22:30:50,791 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8155
[WARNING|trainer.py:803] 2025-04-26 22:30:51,945 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8114
[WARNING|trainer.py:803] 2025-04-26 22:30:52,827 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8156
[WARNING|trainer.py:803] 2025-04-26 22:30:53,908 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8115
[WARNING|trainer.py:803] 2025-04-26 22:30:54,880 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8157
[WARNING|trainer.py:803] 2025-04-26 22:30:55,824 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8116
[WARNING|trainer.py:803] 2025-04-26 22:30:56,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8158
[WARNING|trainer.py:803] 2025-04-26 22:30:57,785 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8117
8159
[WARNING|trainer.py:803] 2025-04-26 22:30:59,063 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:30:59,857 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8118
8160
[WARNING|trainer.py:803] 2025-04-26 22:31:01,126 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:01,829 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8119
8161
[WARNING|trainer.py:803] 2025-04-26 22:31:03,169 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:03,757 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8120
8162
[WARNING|trainer.py:803] 2025-04-26 22:31:05,198 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:31:05,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8121
8163
[WARNING|trainer.py:803] 2025-04-26 22:31:07,250 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:07,939 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8122
8164
[WARNING|trainer.py:803] 2025-04-26 22:31:09,310 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:09,973 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8123
8165
[WARNING|trainer.py:803] 2025-04-26 22:31:11,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:31:12,079 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8124
8166
[WARNING|trainer.py:803] 2025-04-26 22:31:13,630 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:13,976 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8125
8167
[WARNING|trainer.py:803] 2025-04-26 22:31:15,625 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:15,912 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8126
8168
[WARNING|trainer.py:803] 2025-04-26 22:31:17,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:17,957 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8127
8169
[WARNING|trainer.py:803] 2025-04-26 22:31:19,649 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:20,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8128
8170
[WARNING|trainer.py:803] 2025-04-26 22:31:21,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:22,045 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8129
8171
[WARNING|trainer.py:803] 2025-04-26 22:31:24,008 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:24,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8172
8130
[WARNING|trainer.py:803] 2025-04-26 22:31:26,195 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:31:26,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8173
8131
[WARNING|trainer.py:803] 2025-04-26 22:31:28,229 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:31:28,451 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8174
8132
[WARNING|trainer.py:803] 2025-04-26 22:31:30,225 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:31:30,461 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8175
8133
[WARNING|trainer.py:803] 2025-04-26 22:31:32,546 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:32,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8176
8134
[WARNING|trainer.py:803] 2025-04-26 22:31:34,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:31:34,839 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8177
8135
[WARNING|trainer.py:803] 2025-04-26 22:31:36,612 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:36,889 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8178
8136
[WARNING|trainer.py:803] 2025-04-26 22:31:38,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:38,975 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8179
8137
[WARNING|trainer.py:803] 2025-04-26 22:31:40,744 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:41,015 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8180
8138
[WARNING|trainer.py:803] 2025-04-26 22:31:42,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:31:43,170 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8181
8139
[WARNING|trainer.py:803] 2025-04-26 22:31:44,871 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:45,223 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8182
8140
[WARNING|trainer.py:803] 2025-04-26 22:31:46,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:47,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8183
8141
[WARNING|trainer.py:803] 2025-04-26 22:31:49,118 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:49,329 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8184
8142
[WARNING|trainer.py:803] 2025-04-26 22:31:51,353 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:51,494 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8185
8143
[WARNING|trainer.py:803] 2025-04-26 22:31:53,339 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:53,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8186
8144
[WARNING|trainer.py:803] 2025-04-26 22:31:55,643 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:55,792 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8145
8187
[WARNING|trainer.py:803] 2025-04-26 22:31:57,837 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:31:57,874 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8188
8146
[WARNING|trainer.py:803] 2025-04-26 22:31:59,777 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:31:59,840 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8189
8147
[WARNING|trainer.py:803] 2025-04-26 22:32:01,675 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:32:01,950 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8190
8148
[WARNING|trainer.py:803] 2025-04-26 22:32:03,550 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:04,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8191
8149
[WARNING|trainer.py:803] 2025-04-26 22:32:05,712 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:32:06,101 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8192
8150
[WARNING|trainer.py:803] 2025-04-26 22:32:08,036 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:32:08,458 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8193
8151
[WARNING|trainer.py:803] 2025-04-26 22:32:10,024 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:10,604 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8194
8152
[WARNING|trainer.py:803] 2025-04-26 22:32:11,986 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:12,759 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
8195
8153
[WARNING|trainer.py:803] 2025-04-26 22:32:14,103 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:14,806 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8196
8154
[WARNING|trainer.py:803] 2025-04-26 22:32:16,173 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:16,897 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8197
8155
[WARNING|trainer.py:803] 2025-04-26 22:32:18,157 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:32:18,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8198
8156
[WARNING|trainer.py:803] 2025-04-26 22:32:20,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:32:21,031 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8199
8157
[WARNING|trainer.py:803] 2025-04-26 22:32:22,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:22,971 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8200
8158
[WARNING|trainer.py:803] 2025-04-26 22:32:24,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:32:24,962 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8201
8159
[WARNING|trainer.py:803] 2025-04-26 22:32:26,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:27,049 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8160
8202
[WARNING|trainer.py:803] 2025-04-26 22:32:28,999 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:29,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8161
8203
[WARNING|trainer.py:803] 2025-04-26 22:32:30,883 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:31,059 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8162
8204
[WARNING|trainer.py:803] 2025-04-26 22:32:33,053 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:32:33,252 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8163
8205
[WARNING|trainer.py:803] 2025-04-26 22:32:35,119 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:32:35,263 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8164
8206
[WARNING|trainer.py:803] 2025-04-26 22:32:37,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:37,442 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8165
8207
[WARNING|trainer.py:803] 2025-04-26 22:32:39,262 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:39,626 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8166
8208
[WARNING|trainer.py:803] 2025-04-26 22:32:41,227 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:41,657 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8167
8209
[WARNING|trainer.py:803] 2025-04-26 22:32:43,183 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:43,780 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8168
8210
[WARNING|trainer.py:803] 2025-04-26 22:32:45,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:32:45,958 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8169
8211
[WARNING|trainer.py:803] 2025-04-26 22:32:47,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:47,926 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8170
8212
[WARNING|trainer.py:803] 2025-04-26 22:32:49,320 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:49,921 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8171
8213
[WARNING|trainer.py:803] 2025-04-26 22:32:51,242 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:32:51,944 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8172
8214
[WARNING|trainer.py:803] 2025-04-26 22:32:53,462 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:32:54,129 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8173
8215
[WARNING|trainer.py:803] 2025-04-26 22:32:55,460 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:32:56,179 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8174
[WARNING|trainer.py:803] 2025-04-26 22:32:57,370 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8216
[WARNING|trainer.py:803] 2025-04-26 22:32:58,334 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8175
8217
[WARNING|trainer.py:803] 2025-04-26 22:32:59,661 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:00,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8176
[WARNING|trainer.py:803] 2025-04-26 22:33:01,717 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8218
[WARNING|trainer.py:803] 2025-04-26 22:33:02,767 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8177
[WARNING|trainer.py:803] 2025-04-26 22:33:03,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8219
8178
[WARNING|trainer.py:803] 2025-04-26 22:33:04,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:05,637 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8220
8179
[WARNING|trainer.py:803] 2025-04-26 22:33:07,080 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:07,687 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8221
8180
[WARNING|trainer.py:803] 2025-04-26 22:33:09,248 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:33:09,754 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8222
8181
[WARNING|trainer.py:803] 2025-04-26 22:33:11,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:11,704 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8182
8223
[WARNING|trainer.py:803] 2025-04-26 22:33:13,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:13,664 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8224
8183
[WARNING|trainer.py:803] 2025-04-26 22:33:15,788 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:15,970 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8225
8184
[WARNING|trainer.py:803] 2025-04-26 22:33:18,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:18,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8226
8185
[WARNING|trainer.py:803] 2025-04-26 22:33:20,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:20,156 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8227
8186
[WARNING|trainer.py:803] 2025-04-26 22:33:22,294 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:22,425 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8228
8187
[WARNING|trainer.py:803] 2025-04-26 22:33:24,303 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:33:24,648 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8188
8229
[WARNING|trainer.py:803] 2025-04-26 22:33:26,551 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:26,559 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8189
8230
[WARNING|trainer.py:803] 2025-04-26 22:33:28,522 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:33:28,641 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8190
8231
[WARNING|trainer.py:803] 2025-04-26 22:33:30,448 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:30,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8191
8232
[WARNING|trainer.py:803] 2025-04-26 22:33:32,574 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:33:32,660 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8233
8192
[WARNING|trainer.py:803] 2025-04-26 22:33:34,752 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:34,972 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8234
8193
[WARNING|trainer.py:803] 2025-04-26 22:33:36,831 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:33:36,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8194
8235
[WARNING|trainer.py:803] 2025-04-26 22:33:38,948 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:38,988 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8195
8236
[WARNING|trainer.py:803] 2025-04-26 22:33:41,043 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:41,164 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8196
8237
[WARNING|trainer.py:803] 2025-04-26 22:33:43,018 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:43,245 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8197
8238
[WARNING|trainer.py:803] 2025-04-26 22:33:44,967 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:45,249 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8239
8198
[WARNING|trainer.py:803] 2025-04-26 22:33:47,295 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:47,421 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8240
8199
[WARNING|trainer.py:803] 2025-04-26 22:33:49,384 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:49,483 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8241
8200
[WARNING|trainer.py:803] 2025-04-26 22:33:51,463 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:33:51,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8242
8201
[WARNING|trainer.py:803] 2025-04-26 22:33:53,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:33:53,701 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8243
8202
[WARNING|trainer.py:803] 2025-04-26 22:33:55,581 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:55,849 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8244
8203
[WARNING|trainer.py:803] 2025-04-26 22:33:57,563 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:33:57,888 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

No
8245
8204
[WARNING|trainer.py:803] 2025-04-26 22:33:59,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:34:00,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8246
8205
[WARNING|trainer.py:803] 2025-04-26 22:34:01,594 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:02,116 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8247
8206
[WARNING|trainer.py:803] 2025-04-26 22:34:03,855 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:34:04,237 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8248
8207
[WARNING|trainer.py:803] 2025-04-26 22:34:06,034 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:34:06,357 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8249
8208
[WARNING|trainer.py:803] 2025-04-26 22:34:08,047 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:08,429 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8250
8209
[WARNING|trainer.py:803] 2025-04-26 22:34:10,190 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:10,525 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:No
8251
8210
[WARNING|trainer.py:803] 2025-04-26 22:34:12,289 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:34:12,654 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8252
8211
[WARNING|trainer.py:803] 2025-04-26 22:34:14,492 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:14,691 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8253
8212
[WARNING|trainer.py:803] 2025-04-26 22:34:16,577 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:34:16,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8254
8213
[WARNING|trainer.py:803] 2025-04-26 22:34:18,554 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:18,735 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8255
8214
[WARNING|trainer.py:803] 2025-04-26 22:34:20,876 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:20,935 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8256
8215
[WARNING|trainer.py:803] 2025-04-26 22:34:22,843 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:23,000 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8257
8216
[WARNING|trainer.py:803] 2025-04-26 22:34:24,955 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:25,144 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8217
8258
[WARNING|trainer.py:803] 2025-04-26 22:34:27,316 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:27,319 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8259
8218
[WARNING|trainer.py:803] 2025-04-26 22:34:29,244 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:29,651 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8260
8219
[WARNING|trainer.py:803] 2025-04-26 22:34:31,536 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:31,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8261
8220
[WARNING|trainer.py:803] 2025-04-26 22:34:33,631 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:33,937 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8262
8221
[WARNING|trainer.py:803] 2025-04-26 22:34:35,611 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:36,077 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8263
8222
[WARNING|trainer.py:803] 2025-04-26 22:34:37,879 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:34:38,360 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8264
8223
[WARNING|trainer.py:803] 2025-04-26 22:34:40,136 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:34:40,367 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
8265
8224
[WARNING|trainer.py:803] 2025-04-26 22:34:42,192 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:42,472 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8266
8225
[WARNING|trainer.py:803] 2025-04-26 22:34:44,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:44,779 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8267
8226
[WARNING|trainer.py:803] 2025-04-26 22:34:46,380 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:46,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8268
8227
[WARNING|trainer.py:803] 2025-04-26 22:34:48,750 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:48,979 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8228
8269
[WARNING|trainer.py:803] 2025-04-26 22:34:50,947 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:34:51,004 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8270
8229
[WARNING|trainer.py:803] 2025-04-26 22:34:52,877 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:34:53,212 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8271
8230
[WARNING|trainer.py:803] 2025-04-26 22:34:54,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:34:55,355 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8272
8231
[WARNING|trainer.py:803] 2025-04-26 22:34:56,862 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:34:57,228 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8273
8232
[WARNING|trainer.py:803] 2025-04-26 22:34:58,966 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:34:59,415 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8274
8233
[WARNING|trainer.py:803] 2025-04-26 22:35:00,911 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.

Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:01,542 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8275
8234
[WARNING|trainer.py:803] 2025-04-26 22:35:02,842 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:03,644 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8276
8235
[WARNING|trainer.py:803] 2025-04-26 22:35:05,070 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:05,817 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8277
[WARNING|trainer.py:803] 2025-04-26 22:35:07,155 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8236
[WARNING|trainer.py:803] 2025-04-26 22:35:08,011 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8278
8237
[WARNING|trainer.py:803] 2025-04-26 22:35:09,333 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:10,105 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8279
8238
[WARNING|trainer.py:803] 2025-04-26 22:35:11,420 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:35:12,113 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8280
8239
[WARNING|trainer.py:803] 2025-04-26 22:35:13,590 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:14,184 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8281
8240
[WARNING|trainer.py:803] 2025-04-26 22:35:15,646 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
[WARNING|trainer.py:803] 2025-04-26 22:35:16,300 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8282
8241
[WARNING|trainer.py:803] 2025-04-26 22:35:17,715 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:35:18,404 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8283
8242
[WARNING|trainer.py:803] 2025-04-26 22:35:19,833 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:35:20,500 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8284
8243
[WARNING|trainer.py:803] 2025-04-26 22:35:21,892 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:22,565 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8285
8244
[WARNING|trainer.py:803] 2025-04-26 22:35:24,033 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:35:24,572 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8286
8245
[WARNING|trainer.py:803] 2025-04-26 22:35:25,989 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:35:26,629 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8287
8246
[WARNING|trainer.py:803] 2025-04-26 22:35:28,098 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:28,583 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8288
8247
[WARNING|trainer.py:803] 2025-04-26 22:35:30,373 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
[WARNING|trainer.py:803] 2025-04-26 22:35:30,867 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoYes
8289
8248
[WARNING|trainer.py:803] 2025-04-26 22:35:32,479 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
NoNo
[WARNING|trainer.py:803] 2025-04-26 22:35:32,941 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
YesYes
8290
8249
[rank0]:[E426 22:35:34.569973427 ProcessGroupNCCL.cpp:616] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=13920, OpType=ALLREDUCE, NumelIn=52297728, NumelOut=52297728, Timeout(ms)=600000) ran for 600060 milliseconds before timing out.
[rank0]:[E426 22:35:34.570961530 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 0] Exception (either an error or timeout) detected by watchdog at work: 13920, last enqueued NCCL work: 13920, last completed NCCL work: 13919.
[WARNING|trainer.py:803] 2025-04-26 22:35:34,567 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[rank0]:[E426 22:35:34.004621339 ProcessGroupNCCL.cpp:1834] [PG ID 1 PG GUID 1 Rank 0] Timeout at NCCL work: 13920, last enqueued NCCL work: 13920, last completed NCCL work: 13919.
[rank0]:[E426 22:35:34.004648571 ProcessGroupNCCL.cpp:630] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E426 22:35:34.004653621 ProcessGroupNCCL.cpp:636] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E426 22:35:34.005814513 ProcessGroupNCCL.cpp:1595] [PG ID 1 PG GUID 1 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=13920, OpType=ALLREDUCE, NumelIn=52297728, NumelOut=52297728, Timeout(ms)=600000) ran for 600060 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f052e96c446 in /home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f04e41c4a92 in /home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f04e41cbed3 in /home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f04e41cd93d in /home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x145c0 (0x7f052ee085c0 in /home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/lib/libtorch.so)
frame #5: <unknown function> + 0x94ac3 (0x7f053338bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x126850 (0x7f053341d850 in /lib/x86_64-linux-gnu/libc.so.6)

[WARNING|trainer.py:803] 2025-04-26 22:35:34,930 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
8291
8250
[WARNING|trainer.py:803] 2025-04-26 22:35:36,569 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
[WARNING|trainer.py:803] 2025-04-26 22:35:37,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
:Yes
W0426 22:35:37.608392 784474 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 784563 closing signal SIGTERM
W0426 22:35:37.611305 784474 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 784564 closing signal SIGTERM
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
E0426 22:35:41.451914 784474 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 0 (pid: 784562) of binary: /home/wangjiarui/.conda/envs/intern25/bin/python
Traceback (most recent call last):
  File "/home/wangjiarui/.conda/envs/intern25/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangjiarui/.conda/envs/intern25/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
=======================================================
train/train_qa.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-26_22:35:37
  host      : amax
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 784562)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 784562
=======================================================
/home/wangjiarui/.conda/envs/intern25/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 21 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '